From patchwork Wed Jan 13 07:08:18 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Pinski X-Patchwork-Id: 8022731 Return-Path: X-Original-To: patchwork-linux-arm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id AF8E6BEEE5 for ; Wed, 13 Jan 2016 07:11:00 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id D17812040F for ; Wed, 13 Jan 2016 07:10:59 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.9]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 130FE203FB for ; Wed, 13 Jan 2016 07:10:59 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1aJFYb-0005Uh-Pk; Wed, 13 Jan 2016 07:09:25 +0000 Received: from mail-ig0-x231.google.com ([2607:f8b0:4001:c05::231]) by bombadil.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1aJFXz-0005CT-FS for linux-arm-kernel@lists.infradead.org; Wed, 13 Jan 2016 07:08:48 +0000 Received: by mail-ig0-x231.google.com with SMTP id ik10so165409012igb.1 for ; Tue, 12 Jan 2016 23:08:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cavium-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=v0cwOjcpdko06a1JGMKnSDvXLj/c7Rp4d5KHfNBOzY4=; b=02m6joDSNJoHSghSXSifF8cUojMOUYkJ/uUGjaXuf/biVaeOjvf9L1NWuhyZkHpu1m AyzEuY8hmOv+lsNzRlGRTmdA26Nd9lmaJVZp6/nh8dH2OcFT5l5Uh2c5KiPknfqEjfg8 0Rp8ApLb8KpDcIFv+lwFpJbC0a+5FwM2JFP1idJlr7z9RobLyR1ZdfvoCnmNm6z1Pvrj 1BbtdzP/pO1j7mriKXE8Wk3Y7z8V+1lWcB94DH1y8E9QxqDt3n/GhvMovfd3NbVy9ema 1t/4SPf1m7r3fcxXsD/4Js+3Gf731jFXsA1JEkFfjJprE48UpQNPAhNCA+Y0arSTLZ4E 18jw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=v0cwOjcpdko06a1JGMKnSDvXLj/c7Rp4d5KHfNBOzY4=; b=IaiLXWJ6c53Io/lHUKLEV95+7FNwZXA2LHC6v8i4XxV2PmUGIS/IipatlpZ1scgeut VnyzDXP0KE7NwVmcUeqTqRH2468av6AzOVzP2ZQ3qdgRqmhrET3WQhAVV0C8xLHg5u8M oeU89kZTNCIZ010k6jXIYeJBAI7EcYObNAY5qvkxcL1leBsUBdt1EDCr8WoiDTXbfz3Y /BfTxY8/MTFHY7ZNMOLbrsVGTIuXoe8D0WxWfam58eijUJ5Fcmx66Dq5lTK0GTsXTrAd fHbVC0133RXlEVD1BMy7wELsZzxcj02K590J3tt3PVFZSZ4VPy/hqcnXQ3BW9rdeAteh Os3Q== X-Gm-Message-State: ALoCoQkJuV1CMERGJDBpA2BkRob+mvASSLxqe1456yo3FUL4hHCFNQbbdhp3osCFqBJaRfkIuhWUGUBJJbbunECHU+akhip34w== X-Received: by 10.50.85.13 with SMTP id d13mr22681801igz.32.1452668906589; Tue, 12 Jan 2016 23:08:26 -0800 (PST) Received: from localhost.localdomain ([64.2.3.194]) by smtp.gmail.com with ESMTPSA id y143sm537817iod.35.2016.01.12.23.08.24 (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 12 Jan 2016 23:08:25 -0800 (PST) Received: from localhost.localdomain (apinskidesktop [127.0.0.1]) by localhost.localdomain (8.14.3/8.14.3/Debian-9.4) with ESMTP id u0D78NgK003622 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES256-SHA bits=256 verify=NO); Tue, 12 Jan 2016 23:08:23 -0800 Received: (from apinski@localhost) by localhost.localdomain (8.14.3/8.14.3/Submit) id u0D78Nmk003621; Tue, 12 Jan 2016 23:08:23 -0800 From: Andrew Pinski To: pinskia@gmail.com, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Will Deacon Subject: [PATCH 4/5] ARM64: Patch in prefetching for copy_page is requested Date: Tue, 12 Jan 2016 23:08:18 -0800 Message-Id: <1452668899-3553-5-git-send-email-apinski@cavium.com> X-Mailer: git-send-email 1.7.2.5 In-Reply-To: <1452668899-3553-1-git-send-email-apinski@cavium.com> References: <1452668899-3553-1-git-send-email-apinski@cavium.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20160112_230847_683419_E6A08BC0 X-CRM114-Status: GOOD ( 11.17 ) X-Spam-Score: -2.6 (--) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Andrew Pinski MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Spam-Status: No, score=-4.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,RP_MATCHES_RCVD,T_DKIM_INVALID,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On ThunderX T88 pass 1 and pass 2, there is no hardware prefetching so we need to patch in software prefetching. Prefetching improves this code by 60% over the original code and 2x over the code without prefetching. Meaured by using the benchmark code at https://github.com/apinski-cavium/copy_page_benchmark Signed-off-by: Andrew Pinski --- arch/arm64/lib/copy_page.S | 17 +++++++++++++++++ 1 files changed, 17 insertions(+), 0 deletions(-) diff --git a/arch/arm64/lib/copy_page.S b/arch/arm64/lib/copy_page.S index dfb0316..876e882 100644 --- a/arch/arm64/lib/copy_page.S +++ b/arch/arm64/lib/copy_page.S @@ -18,6 +18,8 @@ #include #include #include +#include +#include /* @@ -28,6 +30,15 @@ * x1 - src */ ENTRY(copy_page) +alternative_if_not ARM64_NEEDS_PREFETCH_128 + nop + nop +alternative_else + # Prefetch two cache lines ahead. + prfm pldl1strm, [x1, #128] + prfm pldl1strm, [x1, #256] +alternative_endif + ldp x2, x3, [x1] ldp x4, x5, [x1, #16] ldp x6, x7, [x1, #32] @@ -42,6 +53,12 @@ ENTRY(copy_page) 1: subs x18, x18, #128 +alternative_if_not ARM64_NEEDS_PREFETCH_128 + nop +alternative_else + prfm pldl1strm, [x1, #384] +alternative_endif + stnp x2, x3, [x0] ldp x2, x3, [x1] stnp x4, x5, [x0, #16]