From patchwork Sat Jun 19 11:34:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akira Tsukamoto X-Patchwork-Id: 12332935 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.2 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1372C2B9F4 for ; Sat, 19 Jun 2021 11:35:08 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4FD27610A1 for ; Sat, 19 Jun 2021 11:35:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4FD27610A1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date: Message-ID:From:References:To:Subject:Reply-To:Cc:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=aB4drN5HK1uaTl2tLe2bLCZS2/tJADh3xbmzPhZIoTc=; b=EkYlxneYh4ZAbvYXbCLb/AK46B Atu69HnrLy+SCUG27p7lhJqWHVjPn+lnoN2sQIZ3/rsrloTQutZ/sERRz+KJg0IaMe0tWnxXEQwVG by7vRXYLqbGT91pUZf2zdwCfY64NF+42LtVer+pOc25OlEp+m5wtrCLDXoVE4eGnkIr9Jg54dk6xI JB4FBq/TjMfLEm0eCAlhD5q73K0pBUhp6vgXUnZhfBLshU1KU7Bd5Tt/I7j+VMH7ymwHpdiMQsXc2 XDW0mv1BmtvyHCZ3TIQQGiuL8ZIGK3JVyuSQKRLAZArSsiOULshav9p6KkPwY5n/VE0PgNWT6KWAD EPUUITVA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1luZFU-00GtMA-Hu; Sat, 19 Jun 2021 11:34:52 +0000 Received: from mail-pf1-x430.google.com ([2607:f8b0:4864:20::430]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1luZFR-00GtLt-Nl for linux-riscv@lists.infradead.org; Sat, 19 Jun 2021 11:34:51 +0000 Received: by mail-pf1-x430.google.com with SMTP id y4so2848289pfi.9 for ; Sat, 19 Jun 2021 04:34:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language:content-transfer-encoding; bh=PxDFQ3I0YJog9pgbJh/SJmwo/SIulZ+XOOwzT95hgRI=; b=JCEUHJYya5T1Q1Qm4yidgPbR6DoCK/e1s3YMSHu9YwPL7XanlcDTu0x18rQmElD5Pt 8SucHreHwpRnYPcNe4VO5S1CwbIaCH+BgGm0CgtuM39boN1Wt1X8/3cyHrqcRpGKzI7M /tX1roPGmUW9XWKk/xzERyih4NTXVUbZWXY6kgjEQPkwwEa9Nrdl1KaGh64+h2/4Hekx WdSFpVP6w/wySTQAlWvYW1rywBdJqsSRSKX8NQzfQwgirerzOgr2moLNsupTbSfxca0t LBiwb5mfo0Z2f2QY3NgTNHRdXe/ZrbYr5ZWIZ2sO+CZ10hkjYyd1GKeJR43dJLKEgGOh jPRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=PxDFQ3I0YJog9pgbJh/SJmwo/SIulZ+XOOwzT95hgRI=; b=c+9gcG65yIroz+JqozVoybi6Pffz1sN5ByX5LFw+C6VM0CJYiwkOulj0sPxvK+iyoh oLg46kBXZ38RIIO0QQ0RbBHKtiw3s9UuaUl2F0gR8ual+zv3soOUe4gTvOuu8OALvUgF 86Vh8Fph24J4UUxJTnQBjhkO1UXiP8nQN4vDGuU+/Kf4a3dDJTgzmMCriKNdhjOmfXjt Z79YBcviGKQxY/DLzhq/4fNBhRlfDlHDJ/1Jy6e9Xzmh/M82n1JipDpqpc53cSwX4akK 4KpZ549UIFznEzCpHvnqfvKG6RYWPSFxH5eDsLbHRd95RsQTUEk63I6XWZ04QMTmbVuz 2CcA== X-Gm-Message-State: AOAM531cEP/tvkPSMs75SHxOHpH9ydhc+zRZwPWpT2JwkwwwcVu48xMD gKeDBN+i8VodwaeokSTPsZF3chSm6V1Gzg== X-Google-Smtp-Source: ABdhPJyjbBaZ6iRflKyZAFZkA3uGjZ7yFTU9YjiKzEk2ueC6ocCxhQ2H8r+fJMcBq+8pOIoGE4tv5Q== X-Received: by 2002:a63:d815:: with SMTP id b21mr14649944pgh.321.1624102488505; Sat, 19 Jun 2021 04:34:48 -0700 (PDT) Received: from [192.168.1.153] (87.19.178.217.shared.user.transix.jp. [217.178.19.87]) by smtp.gmail.com with ESMTPSA id oc10sm9585530pjb.44.2021.06.19.04.34.46 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 19 Jun 2021 04:34:48 -0700 (PDT) Subject: [PATCH 1/5] riscv: __asm_to/copy_from_user: delete existing code To: Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org References: <5a5c07ac-8c11-79d3-46a3-a255d4148f76@gmail.com> From: Akira Tsukamoto Message-ID: <1b454390-cee5-a414-509b-3adb25ec0330@gmail.com> Date: Sat, 19 Jun 2021 20:34:43 +0900 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <5a5c07ac-8c11-79d3-46a3-a255d4148f76@gmail.com> Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210619_043449_831931_DE250BD0 X-CRM114-Status: UNSURE ( 8.10 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This is to make the diff easier to read, since the diff on assembler is horrible to read. Signed-off-by: Akira Tsukamoto --- arch/riscv/lib/uaccess.S | 40 ---------------------------------------- 1 file changed, 40 deletions(-) diff --git a/arch/riscv/lib/uaccess.S b/arch/riscv/lib/uaccess.S index fceaeb18cc64..da9536e1e9cb 100644 --- a/arch/riscv/lib/uaccess.S +++ b/arch/riscv/lib/uaccess.S @@ -19,50 +19,10 @@ ENTRY(__asm_copy_from_user) li t6, SR_SUM csrs CSR_STATUS, t6 - add a3, a1, a2 - /* Use word-oriented copy only if low-order bits match */ - andi t0, a0, SZREG-1 - andi t1, a1, SZREG-1 - bne t0, t1, 2f - - addi t0, a1, SZREG-1 - andi t1, a3, ~(SZREG-1) - andi t0, t0, ~(SZREG-1) - /* - * a3: terminal address of source region - * t0: lowest XLEN-aligned address in source - * t1: highest XLEN-aligned address in source - */ - bgeu t0, t1, 2f - bltu a1, t0, 4f -1: - fixup REG_L, t2, (a1), 10f - fixup REG_S, t2, (a0), 10f - addi a1, a1, SZREG - addi a0, a0, SZREG - bltu a1, t1, 1b -2: - bltu a1, a3, 5f - -3: /* Disable access to user memory */ csrc CSR_STATUS, t6 li a0, 0 ret -4: /* Edge case: unalignment */ - fixup lbu, t2, (a1), 10f - fixup sb, t2, (a0), 10f - addi a1, a1, 1 - addi a0, a0, 1 - bltu a1, t0, 4b - j 1b -5: /* Edge case: remainder */ - fixup lbu, t2, (a1), 10f - fixup sb, t2, (a0), 10f - addi a1, a1, 1 - addi a0, a0, 1 - bltu a1, a3, 5b - j 3b ENDPROC(__asm_copy_to_user) ENDPROC(__asm_copy_from_user) EXPORT_SYMBOL(__asm_copy_to_user) From patchwork Sat Jun 19 11:35:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akira Tsukamoto X-Patchwork-Id: 12332937 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.2 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F949C2B9F4 for ; Sat, 19 Jun 2021 11:36:02 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 009B2610A1 for ; Sat, 19 Jun 2021 11:36:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 009B2610A1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date: Message-ID:From:References:To:Subject:Reply-To:Cc:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=NRl4UVj48FssZM8/lhhhbU8A+hWIOQP197DioxuwFtM=; b=OzeEKwpir8yQRRv6db3Dqfwxhu XLv3Z61QLuqaaf1GutaTk+S2iMyBTqiOmtzcdIFDim0Khgmpt2aV6aNhpDk1jpFH/BIwaJ1YBoQVU dgAmF12jrvOkXCD+c5jc6dX8lS8+7V0pwSmg6lGIiukNlslzGYBp1XVmt+Yn/wVkv4tcD1ecvHYtj PxlZeJkcHKwu3p4wzwLAYdLQTlxI9QuQCIIA7B4gQcpc0KejAVHgcZoOGHmd4havBVscltSrpbR3y jBcANrh5uOEWVzQUNUiqO9Yt9+vRf/eSLJEH05iL35cqQPUPoP3n8YCI4GH+5mf+++QuPepBh83hs YvE6iv0A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1luZGO-00GtQI-6V; Sat, 19 Jun 2021 11:35:48 +0000 Received: from mail-pl1-x631.google.com ([2607:f8b0:4864:20::631]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1luZGL-00GtPs-8N for linux-riscv@lists.infradead.org; Sat, 19 Jun 2021 11:35:46 +0000 Received: by mail-pl1-x631.google.com with SMTP id i4so2318007plt.12 for ; Sat, 19 Jun 2021 04:35:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language:content-transfer-encoding; bh=wa9WYCmkHZzaHb+kJYIYFh7H1ZMev+84Ux7Wa9pyeNg=; b=TJBsklKGaYkX1f/McIdSy1IF3SIo35dUxuNO7swhZTvyZof36F8W01SA8kRnDXecKN HkMTrtXpR2HO7f3Qzrk0xvXmIvrUtnuFB1xGZNIATyq6IYG8U/m+vgTWvj8lWZQB51f8 8TD29wasl+lyHj7cbWBcElmKxhU3bGPkI3VlvDNqq+97koi1Pa0xernpGaZu4uLFj/a0 bq37rnz6DEulxlLZd6K+AAR5kD7kMs9Tq5S1a1TjN163pg4fCgqpKkcZNHSTqovkA9b2 u07axWAWlkepuJX7ueNSde4imkwkyeRNCcH4wNjyIJZfKXFR10S+e2mXKxCSZDdcFfyE fWwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=wa9WYCmkHZzaHb+kJYIYFh7H1ZMev+84Ux7Wa9pyeNg=; b=O05iVl2XRDNIpYZ/tAz3Axd4jXUsxnyPrwSMCp+m0eJuoY/XwLETw0QPOVB8Sh7k3B ZM18km096xdXsYxhZfFTCpzOpeToQ8YJgy8dYk3u1Oo7arSTEJGd+n8FskauNBwc+XgO Fdtu3HcEcjsA2vp/4icH3ZgklFDWGFP3aQAbd5VsdTKOaahJBvTkjVjR92ou5UAQARB7 06PoPH1NMxSa88kgC5M6oqdFlE4aibWlPtgHGpb4C1qnpHZtKKVK5R5IdRpcDDpGRmU9 QWLvCInabUV62OhK70wl0gZn1i5dJjZa8ZVjip5kVm+eOMl/Vr2nCyuu10EomVUR8+ug Lz1g== X-Gm-Message-State: AOAM531GVgO3I1IodOyqGXT8kA1PEJESvuCNA5/q+N7dG09jHNRRZkno p+igbmv40DKhzZIyiSSJBz7/epesaZpnbA== X-Google-Smtp-Source: ABdhPJwxj/8EC0yVjHOiWLUTU5TkRA/v04eM3BFH5QLYC5JPAJ4o72KhVR7pXhkcoyt3Zhwk4Js/sQ== X-Received: by 2002:a17:90a:4cc2:: with SMTP id k60mr4411538pjh.83.1624102544292; Sat, 19 Jun 2021 04:35:44 -0700 (PDT) Received: from [192.168.1.153] (87.19.178.217.shared.user.transix.jp. [217.178.19.87]) by smtp.gmail.com with ESMTPSA id p6sm11481026pjh.24.2021.06.19.04.35.42 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 19 Jun 2021 04:35:43 -0700 (PDT) Subject: [PATCH 2/5] riscv: __asm_to/copy_from_user: Adding byte copy first To: Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org References: <5a5c07ac-8c11-79d3-46a3-a255d4148f76@gmail.com> From: Akira Tsukamoto Message-ID: <98088fde-cdb2-6410-8d11-7c8ca984d3c3@gmail.com> Date: Sat, 19 Jun 2021 20:35:39 +0900 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <5a5c07ac-8c11-79d3-46a3-a255d4148f76@gmail.com> Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210619_043545_357989_8603B2AA X-CRM114-Status: GOOD ( 10.44 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Typical load and store loop. It is used for mainly copying the remainder in one byte at a time. Signed-off-by: Akira Tsukamoto --- arch/riscv/lib/uaccess.S | 34 ++++++++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/arch/riscv/lib/uaccess.S b/arch/riscv/lib/uaccess.S index da9536e1e9cb..be1810077f9a 100644 --- a/arch/riscv/lib/uaccess.S +++ b/arch/riscv/lib/uaccess.S @@ -19,9 +19,39 @@ ENTRY(__asm_copy_from_user) li t6, SR_SUM csrs CSR_STATUS, t6 + /* Save for return value */ + mv t5, a2 + + /* + * Register allocation for code below: + * a0 - start of uncopied dst + * a1 - start of uncopied src + * a2 - size + * t0 - end of uncopied dst + */ + add t0, a0, a2 + bgtu a0, t0, 5f + +.Lbyte_copy_tail: + /* + * Byte copy anything left. + * + * a0 - start of remaining dst + * a1 - start of remaining src + * t0 - end of remaining dst + */ + bgeu a0, t0, 5f +4: + fixup lb a5, 0(a1), 10f + addi a1, a1, 1 /* src */ + fixup sb a5, 0(a0), 10f + addi a0, a0, 1 /* dst */ + bltu a0, t0, 4b /* t0 - end of dst */ + +5: /* Disable access to user memory */ csrc CSR_STATUS, t6 - li a0, 0 + li a0, 0 ret ENDPROC(__asm_copy_to_user) ENDPROC(__asm_copy_from_user) @@ -77,7 +107,7 @@ EXPORT_SYMBOL(__clear_user) 10: /* Disable access to user memory */ csrs CSR_STATUS, t6 - mv a0, a2 + mv a0, t5 ret 11: csrs CSR_STATUS, t6 From patchwork Sat Jun 19 11:36:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akira Tsukamoto X-Patchwork-Id: 12332939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.2 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0BA9C2B9F4 for ; Sat, 19 Jun 2021 11:37:11 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A5C71610A1 for ; Sat, 19 Jun 2021 11:37:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A5C71610A1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date: Message-ID:From:References:To:Subject:Reply-To:Cc:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=JMKuer9VVNRgo8DZx2urdDnkjbwoFnIYqIgrmcqEUxw=; b=nzJw1pEcYtqNYiS2BBwx+cwFa0 Op3xgExW5wW0tLKSAfS4aavS8rqaimi5ONPoZj7h70T6g5/kIe8XrGTin5nzY0YQdR134Ll5A2bIP PpAVZ/424pJUq1mqzwxMPYAynMFwhzDszFUvRwNmmaWlrYYb5wxu168adlPP9/7N1ctTb0MJD9PI5 wAtVNs+FTGzg7zAofHXNPLqc0T65ZqU3zKW+NSUdsAP+fIgA2vp/VKLkC3tfVUZ54EqgG8+uv3XTR 4h8Bd/nZZTQFviGJV/Mnl8Xy1EK4qBdS1qGXE1tQdr2rczZW6GhpDPpQXjwd5QH/oM9KYICDpY/0Q kN81uV+g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1luZHW-00GtZO-DO; Sat, 19 Jun 2021 11:36:58 +0000 Received: from mail-pl1-x62c.google.com ([2607:f8b0:4864:20::62c]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1luZHT-00GtYV-LP for linux-riscv@lists.infradead.org; Sat, 19 Jun 2021 11:36:56 +0000 Received: by mail-pl1-x62c.google.com with SMTP id v12so6024792plo.10 for ; Sat, 19 Jun 2021 04:36:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language:content-transfer-encoding; bh=tWMnQuK+9BIZ4cUDgnnI5UsvvwctYskK0oxgmeF7wt8=; b=coJxx5qMalRKncnE5J452fL+slVPFJmTv7X1W8/2E+ZIh9VGPK+NTvs8E4FwnCU6vd KQnTqW7h7hU3ExBYdrgu+Sm37nhSuDMdsPPVqHEQbAf/FBFGQ7bJRNPeoQVDmvKwib50 HoGQW8mVLNy5sVg40UvRa+q17iIKRDtDwwkoaY8fUk+MGFoMszhv1uT1goaTFT2U1BIp XZpuTf4QaoURvGZLS1rr5Kpbg4bG5FhccQnfA/5Mq6vIEDAJ/YXJgwmUV5yLflYSCNFu tMY43h1Ewkjt9gfS4Wfi6DQ2y47CP8H/dkQHQN2RYdjYpF3BJXU9JBcKcdLe669ntX5S t4HQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=tWMnQuK+9BIZ4cUDgnnI5UsvvwctYskK0oxgmeF7wt8=; b=ZAKaOM6E2oVXpjmCikzG5xf0Si3yHkG8zvgOuziT5fPykI7ei5h1YvJtiEm91VpiIT TJgEa1LCa9tq7EyoUuPkvS3uux4cihhgCeOgzYYVkYXXskRMP0Z+6e10aaLhAlqmpcJx yE0ByczqBqP8uJItBZ5Dqrh7urgIl5MP4CzyIyH8j5q0E1VY3oA0yAS5snLvCFI4w+f8 0Eu1vVhxwCykNrjjtrQqfG5A8HSF6C4zqN+DkRNtoddXXr97ezXnDcyzviKSPgFfJYNS FyznipWkbkzB+F87eQU4A1tEqoHpChb8owfiaH/i9vXd4/nd3T+ZydEVaN6gmOh0r1q8 xorA== X-Gm-Message-State: AOAM532rGdyW8J+PDEpqOyuS+ei8xCePOlhlIQiK+Oz1/C/Gm8sxhW3C Nd/TTq4Zsty9HOZW5Hj+IzNUlAwSKuVf6w== X-Google-Smtp-Source: ABdhPJyvEGPpuyqfanHruuFDbzULMZeI2N4HC4CNwz2NULTWpGj39RawbBaqLAMjOnRjtnFdZ6o6aQ== X-Received: by 2002:a17:90b:503:: with SMTP id r3mr16224862pjz.195.1624102614543; Sat, 19 Jun 2021 04:36:54 -0700 (PDT) Received: from [192.168.1.153] (87.19.178.217.shared.user.transix.jp. [217.178.19.87]) by smtp.gmail.com with ESMTPSA id o16sm10411752pfk.129.2021.06.19.04.36.52 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 19 Jun 2021 04:36:54 -0700 (PDT) Subject: [PATCH 3/5] riscv: __asm_to/copy_from_user: Copy until dst is aligned To: Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org References: <5a5c07ac-8c11-79d3-46a3-a255d4148f76@gmail.com> From: Akira Tsukamoto Message-ID: Date: Sat, 19 Jun 2021 20:36:49 +0900 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <5a5c07ac-8c11-79d3-46a3-a255d4148f76@gmail.com> Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210619_043655_762766_88C2F003 X-CRM114-Status: GOOD ( 10.05 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org First copy in bytes until reaches the first word aligned boundary in destination memory address. For speeding up the copy, trying to avoid both the unaligned memory access and byte access are the key. This is the preparation before the bulk aligned word copy. Signed-off-by: Akira Tsukamoto --- arch/riscv/lib/uaccess.S | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/arch/riscv/lib/uaccess.S b/arch/riscv/lib/uaccess.S index be1810077f9a..4906b5ca91c3 100644 --- a/arch/riscv/lib/uaccess.S +++ b/arch/riscv/lib/uaccess.S @@ -32,6 +32,34 @@ ENTRY(__asm_copy_from_user) add t0, a0, a2 bgtu a0, t0, 5f + /* + * Use byte copy only if too small. + */ + li a3, 8*SZREG /* size must be larger than size in word_copy */ + bltu a2, a3, .Lbyte_copy_tail + + /* + * Copy first bytes until dst is align to word boundary. + * a0 - start of dst + * t1 - start of aligned dst + */ + addi t1, a0, SZREG-1 + andi t1, t1, ~(SZREG-1) + /* dst is already aligned, skip */ + beq a0, t1, .Lskip_first_bytes +1: + /* a5 - one byte for copying data */ + fixup lb a5, 0(a1), 10f + addi a1, a1, 1 /* src */ + fixup sb a5, 0(a0), 10f + addi a0, a0, 1 /* dst */ + bltu a0, t1, 1b /* t1 - start of aligned dst */ + +.Lskip_first_bytes: + +.Lword_copy: +.Lshift_copy: + .Lbyte_copy_tail: /* * Byte copy anything left. From patchwork Sat Jun 19 11:37:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akira Tsukamoto X-Patchwork-Id: 12332941 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.2 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A18A8C2B9F4 for ; Sat, 19 Jun 2021 11:38:09 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 58C26601FF for ; Sat, 19 Jun 2021 11:38:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 58C26601FF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date: Message-ID:From:References:To:Subject:Reply-To:Cc:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=dQOdtXqLsVD2ZTZVNc8ugl8bpR8DugOBqpm6vq3aCuk=; b=oxqDbXvx7siU3l2o78bzubu/qw KaqxBqz13Obl09cyTLDS1g5qBkJyAXpe4MtvbN8+Jnu8lo/ZRxKCWdUvtzZ99m37jHWTrfP9PkNb3 sP+XMC9/WcDpJUnBHSMdACe8Ur65REApkOfKSrx1slb7seXnkfyOO96uKcx5z3pU2bsdz4j+dlO87 L8iAG4yUIi/C5C+tm68UnZkk1akyfhdUiWgGuNh8ddy+i5V6dsI3CNdgjpuUHR8q3Z0zs8BUsp7pO vJM0x1eCRDM8DSShrrzf/0Pw5LAzsZCfvzllcRN+wc7SfXnVo+AVQSFVnLTZbF3sqJXudkyzCg3tr ZZQLZE5A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1luZIR-00Gtm2-EO; Sat, 19 Jun 2021 11:37:55 +0000 Received: from mail-pf1-x431.google.com ([2607:f8b0:4864:20::431]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1luZIO-00Gthp-KG for linux-riscv@lists.infradead.org; Sat, 19 Jun 2021 11:37:54 +0000 Received: by mail-pf1-x431.google.com with SMTP id y15so4005580pfl.4 for ; Sat, 19 Jun 2021 04:37:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language:content-transfer-encoding; bh=vXLVQsXj6FC6nXeevSc8Ao3YS/Kg37+h24F/TyUkG4c=; b=FpmHzw9ecJn1mlRFA0MovE301HRBZjiQTpyNM4UrqH7L1sr0MtHSIs4gMoYM64SXkN ugIN3cXt0xSYfsWzTvL5SwsrQYTMxx9cuwqlHUJcMObsfHC2PTn+0nUg8bLC9LLah8/h jd5vM7Rr/yd6x1nrtNj04B5d5WNHX5xB4fTwCkT/wEGNcoOSjd4eilrs2aTdixgr1J3o iRgepBp/UD5LlB+mQ9qsKNeA+dD+7WMOPfa2WQH6zEcXyXskk+VIx7Z4H+ganGPeFDAE W3nFKtaKCjch35SuC/1gWpPZlkifhiogDi4S/eXIxF0hFv+mvsSj6ca8t2F+bgSQCvTo q66w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=vXLVQsXj6FC6nXeevSc8Ao3YS/Kg37+h24F/TyUkG4c=; b=ccS7cAqsRFHl/s5D5FvwNUfx+5KgjT4SqvqvN8awsjhhc4UrPfKYQVvsi8mfz8XRoX F8b3vU8K/9oyxvauv1Ei4WB2WICmSxVGSTwygfhF94kv0v2FgQW2DlnCbUVYgUhULUQf kCu7IV9nXqpj8wYaoFapcL3YqSloeMV51fXwd59ugMsSzsxzDaBlsyDq/KX1Z2v3gRoy wl12s1dnaAQCj0s+3dVwqJKztwvYjwyMaY5i39FYBhIJH5XNJPkGDHZBQLuiONevqBai Z9m/hZ4IDFc5YgAGCXsLylObedV8EYj2vhzqmKSZDk2MlyJLjptq3YTHe6Q9dasrKvn8 UIIg== X-Gm-Message-State: AOAM5313pjFCR8OBa/16I4IpXBxCPNfVVYL09snXlu8xPAkTrVmuDVIC zYY14n5dcuVAlZ5lzR+/3U+Vhua9kyjnuQ== X-Google-Smtp-Source: ABdhPJy1L+2Pe5g/52F2AiehAyM0Rvkeekl8RrJbMqiPvZQYDx/zPa/hkZ+9SkONG1a2Dxt8DGvRTg== X-Received: by 2002:a62:788b:0:b029:300:21b3:d630 with SMTP id t133-20020a62788b0000b029030021b3d630mr7860968pfc.77.1624102671655; Sat, 19 Jun 2021 04:37:51 -0700 (PDT) Received: from [192.168.1.153] (87.19.178.217.shared.user.transix.jp. [217.178.19.87]) by smtp.gmail.com with ESMTPSA id d13sm10218023pfn.136.2021.06.19.04.37.50 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 19 Jun 2021 04:37:51 -0700 (PDT) Subject: [PATCH 4/5] riscv: __asm_to/copy_from_user: Bulk copy while shifting To: Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org References: <5a5c07ac-8c11-79d3-46a3-a255d4148f76@gmail.com> From: Akira Tsukamoto Message-ID: <767aa7e1-0cd0-1e60-3081-77b4c07d14ff@gmail.com> Date: Sat, 19 Jun 2021 20:37:46 +0900 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <5a5c07ac-8c11-79d3-46a3-a255d4148f76@gmail.com> Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210619_043752_739668_DF0C48ED X-CRM114-Status: GOOD ( 15.13 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The destination address is aligned now, but often time the source address is not in an aligned boundary. To reduce the unaligned memory access, it reads the data from source in aligned boundaries, which will cause the data to have an offset, and then combines the data in the next iteration by fixing offset with shifting before writing to destination. The majority of the improving copy speed comes from this shift copy. Signed-off-by: Akira Tsukamoto --- arch/riscv/lib/uaccess.S | 60 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/arch/riscv/lib/uaccess.S b/arch/riscv/lib/uaccess.S index 4906b5ca91c3..e2e57551fc76 100644 --- a/arch/riscv/lib/uaccess.S +++ b/arch/riscv/lib/uaccess.S @@ -56,10 +56,70 @@ ENTRY(__asm_copy_from_user) bltu a0, t1, 1b /* t1 - start of aligned dst */ .Lskip_first_bytes: + /* + * Now dst is aligned. + * Use shift-copy if src is misaligned. + * Use word-copy if both src and dst are aligned because + * can not use shift-copy which do not require shifting + */ + /* a1 - start of src */ + andi a3, a1, SZREG-1 + bnez a3, .Lshift_copy .Lword_copy: .Lshift_copy: + /* + * Word copy with shifting. + * For misaligned copy we still perform aligned word copy, but + * we need to use the value fetched from the previous iteration and + * do some shifts. + * This is safe because reading less than a word size. + * + * a0 - start of aligned dst + * a1 - start of src + * a3 - a1 & mask:(SZREG-1) + * t0 - end of uncopied dst + * t1 - end of aligned dst + */ + /* calculating aligned word boundary for dst */ + andi t1, t0, ~(SZREG-1) + /* Converting unaligned src to aligned arc */ + andi a1, a1, ~(SZREG-1) + + /* + * Calculate shifts + * t3 - prev shift + * t4 - current shift + */ + slli t3, a3, LGREG + li a5, SZREG*8 + sub t4, a5, t3 + + /* Load the first word to combine with seceond word */ + fixup REG_L a5, 0(a1), 10f + +3: + /* Main shifting copy + * + * a0 - start of aligned dst + * a1 - start of aligned src + * t1 - end of aligned dst + */ + + /* At least one iteration will be executed */ + srl a4, a5, t3 + fixup REG_L a5, SZREG(a1), 10f + addi a1, a1, SZREG + sll a2, a5, t4 + or a2, a2, a4 + fixup REG_S a2, 0(a0), 10f + addi a0, a0, SZREG + bltu a0, t1, 3b + + /* Revert src to original unaligned value */ + add a1, a1, a3 + .Lbyte_copy_tail: /* * Byte copy anything left. From patchwork Sat Jun 19 11:43:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Akira Tsukamoto X-Patchwork-Id: 12332943 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.2 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1F9EC2B9F4 for ; Sat, 19 Jun 2021 11:43:31 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7076C600D1 for ; Sat, 19 Jun 2021 11:43:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7076C600D1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date: Message-ID:From:References:To:Subject:Reply-To:Cc:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=lmJ7dapmshvqkN6AT6Wd4S0MDJi9s0S5BU2urm58N/c=; b=a9I7fj7r1lpWU5uoSAMb3aCZiE 76IzQiqXCJjiwvd66/HOzFTt6d4qD7I7cMyQSgBN2Wg/IB/+a78BoIHOlZBXbfltw4F3le49ZUJnj poXb1b1lBeMCmQsjOz2BnVqJBDDm6ZOrpqB+dC42aVZQs3PywIuFpasGOfjCKTPyEmDOo3sl+neei r5P35+pNvqIJQcYDNhdbqbgsH1lYfLtHPqt/NPdaHZAEXjrYfCn6T4JBwiGgBA/pkZaczjCmUQpjO SnhyIKsXTlBN6PqQdYI0N7LvEGa3mpHunA3+11AXJeM0wJXtyjFVPReJaq/79svqLikLxRw8yRJio bbjQ4BUw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1luZNc-00GuTA-C0; Sat, 19 Jun 2021 11:43:16 +0000 Received: from mail-pf1-x429.google.com ([2607:f8b0:4864:20::429]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1luZNZ-00GuSk-2r for linux-riscv@lists.infradead.org; Sat, 19 Jun 2021 11:43:14 +0000 Received: by mail-pf1-x429.google.com with SMTP id a127so9819668pfa.10 for ; Sat, 19 Jun 2021 04:43:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language:content-transfer-encoding; bh=OgpqTQ8TRc+evfamjKBsHRPlfB6iAzq8hyJY+nFPML4=; b=prFuScye+Kf39AgSxdKfHdVrPldWc7GbN49YMICgpbe6UvkMdZvX4dfPKyI6NMAg3X U88AY3CbsGqFET6OTfoCKwLy1NOF0ruLiLVcKFzUqeNwUfXuV2fUEscciFv6ZkLievX7 L8K19EcBbfh5NVmBui30qHIR8CH2qx8No5v4OIJSHsZPjuQGodqZjE2yEToft3CIzR0A 3egG60yLsv+AwcaNExycHzFw9lE1iBJItU1IGeEaiGgZ2OhDlMDbNfT+Sfk23EwUr9jd yQHdumfCXhAdgws+Ycs0ijY2yD/356LnTCzn1693OGttfQLPfa8gX40LixYcn+HPWTKi PGAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=OgpqTQ8TRc+evfamjKBsHRPlfB6iAzq8hyJY+nFPML4=; b=uL2PX/qs6j4n82W0T1EaH/IpWxRc8wvchqhFI+BEL8CBkeH7EHaNNzs91QrZKaEbG8 jE/fj5cnHtCk7ZBsgmH72cUUeNiNSzHOmdboqcg1C3EGtn6V1IQrZmRm6w1GJj3mPwGe N+6zPJFyFMqBfG+PhAPoDmRuhQ0B1ZgOeu3o1jsz76T0ol0PCosdcE3qXVosAFMo3pH5 8R7x45dEUp2Jzwp8kPUGcaMk22YqLjt3pW1U7ZGYXoDSXx2mZgXShpV8cKNQYV+3xUDz 32sNqU+IZgwPF5TVbWW6OrkSEBOmuktfiXqzGit2mgLAU6PeCEmGy7U5ODIciw+W4PyH tggQ== X-Gm-Message-State: AOAM533kM2+FVSy65mY5pAihtMsC07yfhyfLFRCoiXUBX1zxOuRAIodz vzbYQSdxkvioK0yFzRCO/hXpz3x6f3nsEA== X-Google-Smtp-Source: ABdhPJzX5hLzSu58HgsENCBFDIdHp/AgPEgHWVxVqjft49bZ5+oO01jOZTN9Fqc5OuYGQs/tuODEnA== X-Received: by 2002:a63:308:: with SMTP id 8mr10453545pgd.194.1624102991565; Sat, 19 Jun 2021 04:43:11 -0700 (PDT) Received: from [192.168.1.153] (87.19.178.217.shared.user.transix.jp. [217.178.19.87]) by smtp.gmail.com with ESMTPSA id l201sm10806599pfd.183.2021.06.19.04.43.09 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 19 Jun 2021 04:43:11 -0700 (PDT) Subject: [PATCH 5/5] riscv: __asm_to/copy_from_user: Bulk copy when both src, dst are aligned To: Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org References: <5a5c07ac-8c11-79d3-46a3-a255d4148f76@gmail.com> From: Akira Tsukamoto Message-ID: <4637f0f2-2da9-1056-37bf-17c0861b6bff@gmail.com> Date: Sat, 19 Jun 2021 20:43:06 +0900 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <5a5c07ac-8c11-79d3-46a3-a255d4148f76@gmail.com> Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210619_044313_201196_9390E863 X-CRM114-Status: UNSURE ( 9.35 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In the lucky situation that the both source and destination address are on the aligned boundary, perform load and store with register size to copy the data. Without the unrolling, it will reduce the speed since the next store instruction for the same register using from the load will stall the pipeline. Signed-off-by: Akira Tsukamoto --- arch/riscv/lib/uaccess.S | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/arch/riscv/lib/uaccess.S b/arch/riscv/lib/uaccess.S index e2e57551fc76..bceb0629e440 100644 --- a/arch/riscv/lib/uaccess.S +++ b/arch/riscv/lib/uaccess.S @@ -67,6 +67,39 @@ ENTRY(__asm_copy_from_user) bnez a3, .Lshift_copy .Lword_copy: + /* + * Both src and dst are aligned, unrolled word copy + * + * a0 - start of aligned dst + * a1 - start of aligned src + * a3 - a1 & mask:(SZREG-1) + * t0 - end of aligned dst + */ + addi t0, t0, -(8*SZREG-1) /* not to over run */ +2: + fixup REG_L a4, 0(a1), 10f + fixup REG_L a5, SZREG(a1), 10f + fixup REG_L a6, 2*SZREG(a1), 10f + fixup REG_L a7, 3*SZREG(a1), 10f + fixup REG_L t1, 4*SZREG(a1), 10f + fixup REG_L t2, 5*SZREG(a1), 10f + fixup REG_L t3, 6*SZREG(a1), 10f + fixup REG_L t4, 7*SZREG(a1), 10f + fixup REG_S a4, 0(a0), 10f + fixup REG_S a5, SZREG(a0), 10f + fixup REG_S a6, 2*SZREG(a0), 10f + fixup REG_S a7, 3*SZREG(a0), 10f + fixup REG_S t1, 4*SZREG(a0), 10f + fixup REG_S t2, 5*SZREG(a0), 10f + fixup REG_S t3, 6*SZREG(a0), 10f + fixup REG_S t4, 7*SZREG(a0), 10f + addi a0, a0, 8*SZREG + addi a1, a1, 8*SZREG + bltu a0, t0, 2b + + addi t0, t0, 8*SZREG-1 /* revert to original value */ + j .Lbyte_copy_tail + .Lshift_copy: /*