From patchwork Wed Nov 22 09:28:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Huang Shijie X-Patchwork-Id: 13464320 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 27423C61D9B for ; Wed, 22 Nov 2023 09:30:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=+fas+HeOkc0uBHsshgI3xNo1L3lyDPDLQjmpJtXgfH8=; b=kHYc/7OE5d4HKU btqwJOIZPeL06QrTZgMY3oWcCq2mR0Qs/vR5gzsTlOa4a4aDyhqq/gtoIUew+8G0BO+RJb+TrCqgO C3WP6wArQYsPyGZzRk9iPMAf5hC6005YWDk8OUg+1PNfIGehabLLdA7/a3N2nCcKkhQQsTP/+rXXO CdhC7pf9trRALZMUHShVafEAnlROLHLC8CvZrmRZvmHJKVCDbicQbIZuCb9iI2pMx0S0ZQ4xkwmbc yZJeoqw4zCbOJMfc23QV8AjiAsZzetvMbTvwh2CmlBjFJR73724QQ5ZazhyfLXw6HaN30fEg/i7xz bd4rEQBdoD3k7wE0LaPA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r5jYL-001Csc-2g; Wed, 22 Nov 2023 09:29:49 +0000 Received: from mail-bn1nam02on20710.outbound.protection.outlook.com ([2a01:111:f400:7eb2::710] helo=NAM02-BN1-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r5jYG-001Cox-2e for linux-arm-kernel@lists.infradead.org; Wed, 22 Nov 2023 09:29:46 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=E8TJsLuph98IcU3hPomrl9oeoPm58jX5q7aty1zWSj5cqt4lhkSGsQO/vFHWVWDXw7Ayluips3y86r+SU9fNb/b8lJTstSI3X+yOkjKQHbSAhJKQMudb/zp7y+MYUrX2CxKwkO8/G3HuD2r4Ec07vld1Jp6WBYHWmEMKwMge1M2iwWv8TyPlOc7Sf4Wvf8cCDKzmNb7yu2LPCxYQ6ekH04nx0uS7JKHxf2uATvSnbm6Qizt2HQ/hURW1hYXyP0QwL5j6aAEn/y7YvUWozxShfQphtMHUvL/2I8GecWJkCDL9nkzvEQhPP8r4LQXgyd/JT1CWIpvDeSs++rBdF6fDlw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=x+DWlggQhCfDusBUkGypzmxAR/FoqzJEi9ZprSfUN0I=; b=Le1RtiOAQIbPpiZE+wG6UHEhueHts9mGg6xgGiHsHuJiuEglJ6eCqQy8gSwna7LQrsSF3NhgEwyr8fikwDfphVmpaRPDBjpOTy28ppWITIvTVS5KKLzw+9+fMmQ7YGiYtkemEugo113V0IMDHU9X7cRddcLMIPvndbTi+u30DnkwVd0dk5UDVqzy2ZJ11Qvnp7gAAbyzmR7h1t5miw6wpP83IHC4ScJqN8FVQhni2YKKEcxftqg1VNGQKSxpCzgWXh42313GABvIwKrU2Vg102hUhGg7l5Ht5s1FylC6VbU51Ru/qty8kYJWwj1xTGvm2J/S7eT0B12UCQQGl55roQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=x+DWlggQhCfDusBUkGypzmxAR/FoqzJEi9ZprSfUN0I=; b=i36imAqF9Jc6ZepsxaSZxOLPdeOmXwh5ZsHqg+M6ikZTdzi7UQ7SiLzBWfO4DKasOfe7JIqCspmXVUWkU3F9TJVGBVdeGrpMkw32g9FE6i7I2DGeZZ7Pkt/O/SVcyvFv5taCyWJJvA9yXKVI52YooqPwZ6T1Z4GuO3l+U4qzwp8= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from PH0PR01MB7975.prod.exchangelabs.com (2603:10b6:510:26d::15) by CO1PR01MB6760.prod.exchangelabs.com (2603:10b6:303:f2::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7025.18; Wed, 22 Nov 2023 09:29:39 +0000 Received: from PH0PR01MB7975.prod.exchangelabs.com ([fe80::3f45:6905:e017:3b77]) by PH0PR01MB7975.prod.exchangelabs.com ([fe80::3f45:6905:e017:3b77%7]) with mapi id 15.20.7002.027; Wed, 22 Nov 2023 09:29:39 +0000 From: Huang Shijie To: catalin.marinas@arm.com Cc: will@kernel.org, mark.rutland@arm.com, suzuki.poulose@arm.com, broonie@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, anshuman.khandual@arm.com, robh@kernel.org, oliver.upton@linux.dev, maz@kernel.org, patches@amperecomputing.com, Huang Shijie Subject: [PATCH 1/4] extable: add __sort_main_extable Date: Wed, 22 Nov 2023 17:28:52 +0800 Message-Id: <20231122092855.4440-2-shijie@os.amperecomputing.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231122092855.4440-1-shijie@os.amperecomputing.com> References: <20231122092855.4440-1-shijie@os.amperecomputing.com> X-ClientProxiedBy: CH0PR03CA0247.namprd03.prod.outlook.com (2603:10b6:610:e5::12) To PH0PR01MB7975.prod.exchangelabs.com (2603:10b6:510:26d::15) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH0PR01MB7975:EE_|CO1PR01MB6760:EE_ X-MS-Office365-Filtering-Correlation-Id: 4328448c-3e67-4d2e-65f5-08dbeb3d8ccc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: QXVusi7mRcgn1LjazqDbgUtZJpzUm7XH0suVrfiIuxoewpf5UO02ntrPTec4uZYGom57iDIlu7b/KwyBRf+IcD95jGXD+RGM5PLpIkat6O9ZMCfJ47uJT2h9Xul5GTP1sgwQKKqybuRNBCmMIKJkp89l5NE06U79wjVzzeHfufpmCn+lJU0dhKFjhNLbFK3RZvfRID5nZ/f2JPdt8t4OQovv/tAfHP9SXSO0613sGNfGpv25KfmZcvO9oMi0WBkEIqilBRH6MpanDXY3zUZ6NPWXgdD5stHghQD11kjN9Fp0lf5fEtQ8A042P1z8diFS8BgVUtDFgZNMD727ZIWCfuWz006oC+sJCTGjEGElWvVlm9Jq1FgixJTI008DurhPr5Izqpwk42G5e69jB7983csbAgFmA9sZbI9AlTcG3ni6Wz5qgY8wcNb3EhZt1Kdxe8sbhyLNWiS4gxDN90LDZ+cRemctdcZg4JEE29PYney4vAMpYz2K6Aqz2Rt9mN8QHOoe6HRfF57PQsr+LYLz2N4dNyKKwzu/ogYvIMTUJ5zP4YRlTPVPno1am9pEUXhsgOdHxGBcI1mDcrXkrp+XDiw8iCB5xlJKj51OazM93tcaXIk0IGXeeqNSLIzOI3Hr X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH0PR01MB7975.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230031)(39850400004)(396003)(136003)(366004)(376002)(346002)(230922051799003)(451199024)(186009)(64100799003)(1800799012)(41300700001)(2906002)(86362001)(7416002)(5660300002)(38350700005)(6512007)(2616005)(1076003)(26005)(478600001)(83380400001)(6486002)(107886003)(6506007)(52116002)(6666004)(38100700002)(66946007)(316002)(4326008)(8676002)(66556008)(8936002)(6916009)(66476007);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 1RzdCUcNKvGfBTKrBp3ZcMNl4wK3wN/7GIMULMm5NbO5JORCcQ6te1WUy6KIkCEUFYGMpTa3Gup7JWRBvrsgj5bnx1zIdoROZEVxupD7W54y5X721nNp9dCzAn9Ye968Dt8ZyfiLYwO+EfmN7O5NHNuSHB04qlk62fWxVRvbHQLXDo02xbgvEbQPRD0S1PLNZ7DIWKMElrX+GPMofr0N1graSLBkp2VJgUS2lyCX5cjyhvVaZIqI+3sISLJGD2SkB61J7eX06cKvCBaUXkEIWOEZQkXW/DBi/uILBsHPOUgSYQYaYGVpt5ZKCdsfiKSVWoLyIflNY7iGDs9yLBqy7sjAIevTJ2wfAi+I3wPTBmhoO3qeBCQZc2q0vUNuhlFYFFhM7b/72RqRlr82hauhnjwVY0/4mOGWJLaR/ocJg+G2IO24JJSnYxQ1WxRprbRTFtiDgT6byBUN4WKSoV6oRWiR2ur/vKNxWfWL+BK44MG6IA1Ke6zkb7t8S1UTZdvuaHL6rGvIBgvGH4Y85+AMSa5HqAX/9a0rQPERS7AZysZ/MUib6l92VKNBLmUEkF8rTCPCavqWSLarcd+FTuSjlSsZb0rejPKQSKjX53g+0P3NKXV/ArhD16xvS44Q/lH5VzkIAdzhsRKvDWAHJgScYmTFMyhBEi/wajxnNWt/6UnjEIS5m0trrgtqicc1XtwFFqQHO7BbaqwyyraBttnc2sE7yci+Wrr6nvroXas5RSCBAde0URPAXJZk6TLQbEbCS8dmh+zEllQBABImfIT52YOW7p/FeN2lznVa9PK6MmFlKVaQzJZ3UOEFV5a9r8+ZGVnduYxPyw061gPVGHMfQE2nBf9a2X3gQGz4jKTDNPBxY16+gA7VdJKCTch5RE8a2MWS1yhXbFK9LyW+YRF8WVOSr/zTr697aQx51aHjekTZSpNvmsBkc/PS9qN1N4Zftb1JIssKmQ/wdUwmdEeYV5QcBuwPsQo1n2qDkJedLnarAF+VxdXuMCiuaMOu2diqHPh22ZOYDwpS9CaLgFRBcwZAd7ZYXaUXmYOU02ZQ0OM4JXHdAKrg39ydy4GX7wf6IHEXS9rsuLEIfXE5EPnPMNv0MeJd+7D/WOXrByFnV8HKE+eKUTv3LADa5WRV5erm567e+V4U4bu2o0v/H0AX7Ucw5nAfxUEanKFIKIRtbLyNyCSrBGUhDkjeJbzcWJ9Ohd3TbNFmGBWFD0NZOfqLyvmwy6k1Nedne1HoKMA0zXtJVBk2bT7qloxENkyrdCRWj2EuaeaoML9hCKkByqDzabWyeDCP/fZDfYW06gW9QpYAhIgWLnFpx2LIlC2mRL3nNzxeqzu2T18sa0oV/qoBLoOWaJ7yEMu5PiebqS27bxKq++e/zuXdmeHJwrcfEUZEdDAC2JCMTf1cmsFIuWfldJAawt8swsewAZqgwYed8SNVYTbf6U36i15oi66reTUBlkmY3+fXYO5lnNIGYPhVLp9m308tsZ5p96iYoF9rhIHir9xuPDisoLQCTh20JC33n0HBDsh8eV2Q27scazuHSMjdYoPGhPwSs+99+71JHjhM9xvR45XExwDbPjji4CCyCLTbqSUqu/fA+FJMtn5nX3FQHNq8htA89jyRhDaoWr8= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4328448c-3e67-4d2e-65f5-08dbeb3d8ccc X-MS-Exchange-CrossTenant-AuthSource: PH0PR01MB7975.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Nov 2023 09:29:39.3289 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: DVqP2sfgyoJ+rHBDZmyM0sHtwgQoqZNVe6RPXNnmlVy5THcXWkK89pnAlVC8y7cWZ8MRxbTMSfaMKMhkmTSrFjsYOaQuXTOVGKYf/GjnCF9gpFg/U28sVygqdaP/TeJ6 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR01MB6760 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231122_012944_865930_D3A23CDC X-CRM114-Status: GOOD ( 13.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The AmpereOne chip(arm64) may change the kernel exception table at boot time, so it needs to sort the kernel extable table during the kernel boot. Introduce __sort_main_extable which is used to sort the kernel exception table. Signed-off-by: Huang Shijie --- include/linux/extable.h | 2 ++ kernel/extable.c | 8 +++++++- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/include/linux/extable.h b/include/linux/extable.h index 4ab9e78f313b..ef70ec3633b0 100644 --- a/include/linux/extable.h +++ b/include/linux/extable.h @@ -15,6 +15,8 @@ search_extable(const struct exception_table_entry *base, void sort_extable(struct exception_table_entry *start, struct exception_table_entry *finish); void sort_main_extable(void); +void __sort_main_extable(void); + void trim_init_extable(struct module *m); /* Given an address, look for it in the exception tables */ diff --git a/kernel/extable.c b/kernel/extable.c index 71f482581cab..0fbe0ccb1c3a 100644 --- a/kernel/extable.c +++ b/kernel/extable.c @@ -32,13 +32,19 @@ extern struct exception_table_entry __stop___ex_table[]; /* Cleared by build time tools if the table is already sorted. */ u32 __initdata __visible main_extable_sort_needed = 1; +void __sort_main_extable(void) +{ + if (&__stop___ex_table > &__start___ex_table) + sort_extable(__start___ex_table, __stop___ex_table); +} + /* Sort the kernel's built-in exception table */ void __init sort_main_extable(void) { if (main_extable_sort_needed && &__stop___ex_table > &__start___ex_table) { pr_notice("Sorting __ex_table...\n"); - sort_extable(__start___ex_table, __stop___ex_table); + __sort_main_extable(); } } From patchwork Wed Nov 22 09:28:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Huang Shijie X-Patchwork-Id: 13464321 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C3552C61D9B for ; Wed, 22 Nov 2023 09:30:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=mfy1DlBov25oZCqpPXseBkSHgLM4BlSYhzSjNErmNQc=; b=lo+DQxo51gm1xg Mgci+KK7IKbaGbsKNY3iP0ZytHwWRcZLqs7WS17DOcbRfkdzMz6pvTc+XBtwVk0zzFDZoPFwipJyc aTZoDrbcQLugkC0pVEBIQe9ahnDfKMm6arxsvRptYdOJbKrDjPY31HKkaCvbejfttaz5+H9iqs8E+ jr4ObHlz/1KapEapP9zOva9+bBzxPA4gNWTNykrhPzBHjFX+kaLQ3yjlc8hpWfP7t1wZ0nqLLVVf2 Hf5Rzpgb7Ya3n6wR5op12vfiHgYdjMCW49xtJmiezs2pJ44eh894kosrKn8HEtG0JDE9uofevzJFL A/Zg5zV53Dn0AJSHYQXg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r5jYT-001Cw1-0Y; Wed, 22 Nov 2023 09:29:57 +0000 Received: from mail-bn1nam02on20730.outbound.protection.outlook.com ([2a01:111:f400:7eb2::730] helo=NAM02-BN1-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r5jYQ-001CsA-2w for linux-arm-kernel@lists.infradead.org; Wed, 22 Nov 2023 09:29:56 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LLlwMUdvihAQH93fCLa5GcBj+KWJSpJ8HxQuYss1ggzTkX+S6apvRVgY7V54JIk4m8j7UecDqyBxWomfQ1ioq5RTo3KzalKhoQMydgUncyKqiefvyYye46E5Egy7L7KFHrlJFGmzHryqH4GKFSOMNSSff3iUDT7Srt6ZUrMF7z010y+fFlWBR4MvOC8MveH7Ebd1v+gY+4KfBQ43y8KDBVT3H0yDg8mxzklBi0lJaXD2TR19wptvw0H5TG3vM+wg9UkavuGqvDP7kwMXSeqJLXuHu3SrSJGYmuwibkTHNkJsDwZew3DQ+5bqf39Sglq+HyE9Cy0mYUd4/qnd81vxqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=SUn//U9TfDz72Pv19tjJUG4gFENPgF7D5GIy7JZ2GPE=; b=RY0g4WF3DlKXqSw0pAMlMdRHj8WND5BEzD6kBdvz5mxTz87ipOOF9+haHbXJp3s9vB4m4eQPhpaxeuRmbLJJEI98C0NfgvIbMpRFec5K3r6nQaYfbCtnrNU0moPNvP3HhkO/n7iZzp4oYYWIbc0nINrI8qBSmEXb4DphpMiPcSJHtzLOiZHmiHup1/sj0J3bMceDO+zDsj3kmN9rEBa+PejWTwcqr3ZhWcuhLLvrX1LZcof670GDmX/Nb1UBlUxMjrFwTGlYA5fCbL+7eyITODa3Orqr3aHkv8iP6N7ctAPXKu7jUgGfmTLjtuMB6DWmJ/N0GjyWwi7K1IQF9cnFJg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SUn//U9TfDz72Pv19tjJUG4gFENPgF7D5GIy7JZ2GPE=; b=mLRaBt4NF4bEZbFvCblOULGm3iRwFLc19WHq8sB81H/lIC5XSKed91GWUIURUFbszwzVoGfFJUwuCnUolT9P/oWroa04TUfOXPTXF90gPo5KOeAYv1m5REFcqthvAzuPD7OZfmmGIN4Wj+iH37RvF0bBoGVZLC14kP7G51zsEaQ= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from PH0PR01MB7975.prod.exchangelabs.com (2603:10b6:510:26d::15) by CO1PR01MB6760.prod.exchangelabs.com (2603:10b6:303:f2::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7025.18; Wed, 22 Nov 2023 09:29:44 +0000 Received: from PH0PR01MB7975.prod.exchangelabs.com ([fe80::3f45:6905:e017:3b77]) by PH0PR01MB7975.prod.exchangelabs.com ([fe80::3f45:6905:e017:3b77%7]) with mapi id 15.20.7002.027; Wed, 22 Nov 2023 09:29:44 +0000 From: Huang Shijie To: catalin.marinas@arm.com Cc: will@kernel.org, mark.rutland@arm.com, suzuki.poulose@arm.com, broonie@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, anshuman.khandual@arm.com, robh@kernel.org, oliver.upton@linux.dev, maz@kernel.org, patches@amperecomputing.com, Huang Shijie Subject: [PATCH 2/4] arm64: alternative: handle the kernel exception table Date: Wed, 22 Nov 2023 17:28:53 +0800 Message-Id: <20231122092855.4440-3-shijie@os.amperecomputing.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231122092855.4440-1-shijie@os.amperecomputing.com> References: <20231122092855.4440-1-shijie@os.amperecomputing.com> X-ClientProxiedBy: CH0PR03CA0247.namprd03.prod.outlook.com (2603:10b6:610:e5::12) To PH0PR01MB7975.prod.exchangelabs.com (2603:10b6:510:26d::15) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH0PR01MB7975:EE_|CO1PR01MB6760:EE_ X-MS-Office365-Filtering-Correlation-Id: aa21e5d5-c202-4ad2-8287-08dbeb3d8f90 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: QCp8t67fD1sIeBn/PpE8bjsJvqFUMAmsLAmiy/bjbC7mQ5s+40tdnsNEYiawqEmjOYUO2gu0dNFIEU0YAPE7I6CaOc7Wr04F2q88bPmKYxlnNSeO1X2QKeOYowKp9eY4oyCCx7E6JqTTd3jU+VZtjjTy4gjI4JaZuzDjogIfCL+J3vfnhBSpnlMNbx6LZJor+ADs0utlBzPOaSQP0Q1gJ6wbXL9pueV9pU+R1OqF8ITX7TAZNDg6cXRntVDV1fB47k8C4dxXa35mfpRmcLFkI/4H9EYBi7dVi5XsIsF7pXuZQ9iOkkE2DXkQ8xfXW3uzb6a9TyYdJNM60GaWAUZcGBDT2c73pOQ5+I2qjoZaqu3LLc9sqUAu9WOeNOn/p7DV4X3vm/DEeMMUQyPzlcPJN+1meOqPrygbW0FCs6J7NK0COSlNHNnkTgPosB7fLlX62YDrRtSsosQ2AOo8PYorBa2B3dAxbTVjW5l1l40gvPKZlUnFPhN8nRr1bgDuCJDy0aOWI3VFWErCfLwZQkSpbup3bMnaSWP6BfClPaEUsHVlgOleJiTNFIg3f/LnQoJwEj1s6UK0zq1YTl9wlwdF/DjCtRGFDVJuefbXzcp+US0= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH0PR01MB7975.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230031)(39850400004)(396003)(136003)(366004)(376002)(346002)(230922051799003)(451199024)(186009)(64100799003)(1800799012)(41300700001)(2906002)(86362001)(7416002)(5660300002)(38350700005)(6512007)(2616005)(1076003)(26005)(478600001)(6486002)(107886003)(6506007)(52116002)(6666004)(38100700002)(66946007)(316002)(4326008)(8676002)(66556008)(8936002)(6916009)(66476007);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: EWKGUL0VhRMabPmxy3qFWoWmeDPfiAeD1DTmts7nn1+6LTWpMf/Vw4L4aiqPR7N/10lTliK7F+JtwbqPvD2Z7i0p99KA7e/SirEbeZRqIMzf8i5sdNc0gtWJAKnHb16Hr1rpzwyQVN7qBaA/M/ceqbz6n0PNBRFca21zXqU7dy+VMe0hmlhkTf4DmofOlvucPBUznZ5qtm69/jYAaW1ydR3/7MVsDssoGAQqXB3Vsmn6bFSguGk8g9cSr3pIoRvA+EMSU3pgkEXnRfLpGiXXEyUyeqC3w0NxnweC9H0YT2MjKbB2dA5DCINLajpyxwFenFCP/7GWIkML0ffXytJARCsIA+zXxs907p/9m0cBHWgCn+RiBGsqlHreK1sNc4GSFkxRlDNV+XzqDyM2OlJuMGyGy4zuLwTTKM+U38iRpP9uPXd5Rq1BmhThj8tfqotJ6us4g8hqti6RVxS1pOUNqi42ar6BNWvfq/LUyEafkQDtRWbmFp4B3TVPXngHvGTKA1o3ODUTxLkqmuYZWN3C1as/gedi8Qzsk6hyWhOqY7T/Plyy1J2CJZQPXzSOcm7dOz/4yTgrFowLVrgpGouLcVdajbOZbbH0mVowxd/N230Jx4hCCL83Jo9i0rxoTDzIn2/ClaKvtheTwW9pu9+UlHk0ij30fHMoBGZwnd1/1ZeayGKkXgKTbwroIIIXDnp8Wa2IGEoSJymdX/4CSWwosSLZUhptVMGeE/vAQr69CCdyyXy1OxY5DqhLqfzLSNCfl5h3VUqZps3B4RKYrODdINPtzrG4mqH3Bky0fm/FdUv/Wltp+bxEKl+hWAMaoQGpZhpTvCWXzAtjt1T5bsSkqe8u35oC2WryA8627hU1V7YFcquIvF4ArgY+mI1Dzyn1Ag8WpiZDYiH60616ZBmKAtpZlWnNd7oEEwCfdKLWUos0zW5x90UmYghGgubhi3zvnYQW7F6BwWh827xC2eiE5P3VhqXVg7MQptWB1mqozsKAxu5X88foovrKGnGJl8pOhOb8Pip7fdE1V21oQlcOEi1AB0EPWiD+PqrJWokKp5LTNTOQffLDailPr+QVUwJ7xsVeYuDSf4qnViXJx1CZPaKwOrAn6nFx9iSTF7AwjNRfxdXWUfxmteTostKRX8zeo10mFhSQIZu2hQeebDARq2QABpHR9v/4dvAqlgrJ2xdLBlhf64wTAWolVTv8gpDAVYWdaZm687Ok0cy0V47Wxbco6dKs0ESnorgFReeabb8bK66FTdd7RimxPKktTmGGRfWiu+MfmwgqxlTcHMsAhKC8IKw9JZ5Wknie1A+NH8P0pewpbmPthUKhIGPIPTTTphWstLeHtw/83qzLNuIYmWWBdL2/RXPTS5LU7lL8f2hEOEe1Uzsu7ssunGG7ZAl3ZO+dvf4kwusOzFtFAXsz9bw9uzQBJd4Z5yR6nPJ5bC890CmT7gnE+Wgjz3fynr3RxYuYe1ADuGCV89W56ZbAbCvuI6lVII25/dxA9sfsak+JvMgvZhAFMpTD2fyVaeT0CX6NnnPbPjBErWP7eP53H0pYQCrepsWdoX1GUBXiE5ePPsQnT4gxvqzjN5WWVq6uI98NaVOJZJnwK4Tr1NuJ4kaCADZtuCLOq1yzVX5MHMc= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: aa21e5d5-c202-4ad2-8287-08dbeb3d8f90 X-MS-Exchange-CrossTenant-AuthSource: PH0PR01MB7975.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Nov 2023 09:29:43.9654 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: yo83bd2K29ZBU2doqssWgpeoT6cTPDq3FugHG7cfvsJ19fkgMg8riBQ5K+3w529K72K4I8cYkTz8UjBNpF7UB0Kyf/EQupacCvE1QrwNLlGqor0ScJHVbQ3XpeNKQOn/ X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR01MB6760 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231122_012954_952537_98CEF458 X-CRM114-Status: GOOD ( 12.45 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In arch/arm64/lib/copy_templates.S, some macros involve kernel exception table, such as ldp1/stp1/strh1/str1,etc. Current alternative_if does not support to embed them. This patch adds the support to embed them into alternative_if: 1.) Modify the @insn to the right address, 2.) Sort the kernel exception table. Signed-off-by: Huang Shijie --- arch/arm64/kernel/alternative.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/arch/arm64/kernel/alternative.c b/arch/arm64/kernel/alternative.c index 8ff6610af496..4c73f9cc9a85 100644 --- a/arch/arm64/kernel/alternative.c +++ b/arch/arm64/kernel/alternative.c @@ -19,6 +19,7 @@ #include #include #include +#include #define __ALT_PTR(a, f) ((void *)&(a)->f + (a)->f) #define ALT_ORIG_PTR(a) __ALT_PTR(a, orig_offset) @@ -101,6 +102,22 @@ static __always_inline u32 get_alt_insn(struct alt_instr *alt, __le32 *insnptr, return insn; } +/* Check the kernel exception table */ +static void check_extable(__le32 *origptr, __le32 *updptr, __le32 *replptr) +{ + struct exception_table_entry *e; + unsigned long addr = (unsigned long)replptr; + + e = (struct exception_table_entry *)search_kernel_exception_table(addr); + if (e) { + /* Modify the @insn to the right address */ + e->insn = cpu_to_le32((int)((long)(origptr) - (long)(&e->insn))); + + /* Sort the kernel exception table */ + __sort_main_extable(); + } +} + static noinstr void patch_alternative(struct alt_instr *alt, __le32 *origptr, __le32 *updptr, int nr_inst) { @@ -112,6 +129,7 @@ static noinstr void patch_alternative(struct alt_instr *alt, u32 insn; insn = get_alt_insn(alt, origptr + i, replptr + i); + check_extable(origptr + i, updptr + i, replptr + i); updptr[i] = cpu_to_le32(insn); } } From patchwork Wed Nov 22 09:28:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Huang Shijie X-Patchwork-Id: 13464323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2C324C61D9D for ; Wed, 22 Nov 2023 09:30:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=PMisb67JjY/QRuPlK0eT1NO4BxvRdHfJKbQPJ20HHSI=; b=Q3osVGLBQiX7uy JN8tsNxgP3FlwacUSKQpaulII3xh40sqlGlo+yee7WIxxX1dlj050At74pP/pENyWmVVnPdaTiaM1 ZthI0l1vOzPpXd2GnOa6+Ll+FzIKa0PxQZcrlzpi6WBqb+4ZZHJ12q0Sz7fHn1yfUG1Mod1bbeFcD /CYNaj+MnrSLdIn2SA3iYC3H7om//d3/BuCRzGI8DkMc2W06abycpcWWIBNNUJ8EoI7UkCNl+xp1d SEN/xwjQb2S2s7drlkVa2te1C0VVUzDTwXqPInG83D05+vlzHNr9MKha1x0BnuIi1lLAJfn3nYAuJ mjnkIfECWWyASoEfB/xw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r5jYZ-001Czl-1V; Wed, 22 Nov 2023 09:30:03 +0000 Received: from mail-bn1nam02on20730.outbound.protection.outlook.com ([2a01:111:f400:7eb2::730] helo=NAM02-BN1-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r5jYS-001CsA-1g for linux-arm-kernel@lists.infradead.org; Wed, 22 Nov 2023 09:29:57 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LnHI9ph2qoiZ73ZxRc4pKB49m2ttJbAKfYDTOGDgE+0j69ywZCo7WoSnuD1s94JatM7EU67ZMTujqeY3faowWYEUcy1ST74HQeraB6KIRNosVhpC8VwcU1ju3DJmbmEBUoaOt2WZK4FpwqShIEaixE8m1qsLYmCBhNLx/ahHHE5RtwAeCK/N3PeSAc8xJJzlKhv0iAMGN6gUrYzjYYvRZDXthcPMwp4P+OMg9Mm9emLyZoiQ2BMi/1XOKuDkqE0wTwjJyvCJVnAAH1Ub7LymrLzsOZIerc8lIT2QaQ635Af1f51kpyyWudI3E1l/7NcLshG3NJWkt0Z0Ddirh+VDzg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vtkB6JlqAx6Ig7BPg1WUx9jtZreCP+mVpAxMP6FHeHQ=; b=XRq0OZi4zUMtNIqP8YEjhJyPEFXyikK9/PfkV+qMzuAOxRzcsW2RKi66zpbk1Reewaga/lt4U7skqBlQyxkGqil+P1lRl6arVvit3S+FfHWrvhnugP5PhMewesf8c1kuxD4vdTNvaLL7UpDKJ0Wl5hEVK1MFEH+hYem7gy3GdNF7zSPrJWd/Uz3Bd/ghjnlhyA116qUtraalvQqA5CWGDlAXlmCaVt8rJYbRNDvaOY5RLHlaJOxzrK7fS9l1vZj5Qk/1vdRUsrppRvAxrV+E+W8omQV7k0p1pW1iI79QHLu0c780n9AEljHWz/uV0Dcu6VJVdvOUB8Ga58MpwvBcIQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vtkB6JlqAx6Ig7BPg1WUx9jtZreCP+mVpAxMP6FHeHQ=; b=cl9wSKf81PxAZ9hd3qtGyp+Q6rsScpnOzePzc+qpQyuGnfXv1JyA6xxp08CoVOgSYE+dG6sVLHUFuiDNNMjT9p5ImdysKB02i+m1EFpRm/rHkK1rjXzGrI06iu1TCB47b7BjTAOWLWGa3r3i3ghsx5+dBY+mVUso8WiXp5A7yYM= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from PH0PR01MB7975.prod.exchangelabs.com (2603:10b6:510:26d::15) by CO1PR01MB6760.prod.exchangelabs.com (2603:10b6:303:f2::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7025.18; Wed, 22 Nov 2023 09:29:48 +0000 Received: from PH0PR01MB7975.prod.exchangelabs.com ([fe80::3f45:6905:e017:3b77]) by PH0PR01MB7975.prod.exchangelabs.com ([fe80::3f45:6905:e017:3b77%7]) with mapi id 15.20.7002.027; Wed, 22 Nov 2023 09:29:48 +0000 From: Huang Shijie To: catalin.marinas@arm.com Cc: will@kernel.org, mark.rutland@arm.com, suzuki.poulose@arm.com, broonie@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, anshuman.khandual@arm.com, robh@kernel.org, oliver.upton@linux.dev, maz@kernel.org, patches@amperecomputing.com, Huang Shijie Subject: [PATCH 3/4] arm64: copy_template.S: add loop_for_copy_128_bytes macro Date: Wed, 22 Nov 2023 17:28:54 +0800 Message-Id: <20231122092855.4440-4-shijie@os.amperecomputing.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231122092855.4440-1-shijie@os.amperecomputing.com> References: <20231122092855.4440-1-shijie@os.amperecomputing.com> X-ClientProxiedBy: CH0PR03CA0247.namprd03.prod.outlook.com (2603:10b6:610:e5::12) To PH0PR01MB7975.prod.exchangelabs.com (2603:10b6:510:26d::15) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH0PR01MB7975:EE_|CO1PR01MB6760:EE_ X-MS-Office365-Filtering-Correlation-Id: 9de6066c-1b17-4f10-9f1b-08dbeb3d9254 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: m2+YSYKB8XofwUlyfPO/EVNGLciuj4D8+pYStyj5RvPu350rzVw7Yx6V4p042+fNfCf63DcGjgcb0MLd7GGYCNMuopYDyljsWhkU0urJT6rwu0n1q1UUk/yYKohUOwga38qkNqDvL7zh3l4XvrRExvj1VOk6NHwQEHlbqv0xcQTiyKkHJiuuZ8xy6rcb8LEdCj/ySMNDD2FDfsYdLGjGeNVHKFduqcpz4rHxfZ4Q1lkrjCsIzrmEfMA8mLRjWgLCwuZxr54JYPfSHdfEwYgKJlBeJVaTCPujXjEq+L4VxUTaOBWBAGwjR8irlb9vIMggRpk8sCqe3l5yXsoRyf3YYhn8LUU3WqpoGtjBD2zQ+FA/rX9Ap9JVn9W7r7B81f8mvF30L2hEJ6mZ4udc+oBT2K3j6jRdlcQiQNXcOJP8cXbtgloOtagzAVS7ut6lMMkhKq76fOMyn1YMs47s0W3HQGsQnjcaHdg/W530w3tDuYDRfB7u74zU1rKKK8kZQuV+gIw49DbuHdJ0PSV1SvxIqDrfsZatJ4Qd6RcZ5GeZUTHcU7tGhrBQN/sd7JzyDBsergz5jN0uWLG88xmcFyO1K3ir0GjgwzN/P8UmvGDnjhAREAZ1ppRf5y+AFY62muJJ X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH0PR01MB7975.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230031)(39850400004)(396003)(136003)(366004)(376002)(346002)(230922051799003)(451199024)(186009)(64100799003)(1800799012)(41300700001)(2906002)(86362001)(7416002)(5660300002)(38350700005)(6512007)(2616005)(1076003)(26005)(478600001)(83380400001)(6486002)(107886003)(6506007)(52116002)(6666004)(38100700002)(66946007)(316002)(4326008)(8676002)(66556008)(8936002)(6916009)(66476007);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 8MNkqwJJbfzyWUnWwcSMDBgv79lVeVSEYAiANMIPiypHto8alOaf/wdYNqeVZknBlBJJ5YHsZsmyLn22Sxo9nLJkSfxZkKOd+vIOjgnQz1DIGuouEN1ULDxQtLMoM4GvrdGmU7/nTqAnzNHmFhDhU+48pzYcoGv8RCdCgCbY1eZiKdYjewoyg8oOtdsKFI1DtJBZOs6c+9703yPmcUCB0FiDcJ+k9Z0Trb6iU7SEwSlEDOY8pjjZN2Zf5257+xzbzpuSlxbHVGSuXKOBIT+b79vYKs6JDNNZ0xorGtgh8Wm2p2UOQQx2Uz6T2M6+CVgzf2mUH1HkWXd+b4izKCStcCNzTkaNG8S65cBhNYk6FB6Ni9l6lWwpkkhqrFflTfyQcWA4nWMvcvFHTLQIhYrqCFWdPDDJ4WLAKAcMKC6lbDKZhSgVkwL6lqS73dezRJJnvjRFo/d1v0AU6x7CyINKA/cGmaDmS5yKA7GnC79hw+zJIzyGWgoQQqqyf+3+CJ8gwv/PgXFOGGMzRn6uV+dPutczU+OqSXduaS3YE4EnKAmhVSJh0HFhcYMdg8+vwPYeZp6HHIvlCQChbfSSo2UaNsgPgkZWozCA9sRmKwXVCKRQW30blSPuZsmcsVl1Px78mu0D6+3mrv+1ZvhbZ0aUGc65pfgt49LL4vZKgPBFMPm62cgjtJ8LeVO5uhAnwpicg7hOnkiFZVMd9z6Rd43dM4/L4Pz+wSNKpvXiB3YWKzKa6k6765waO8jyaerV9D/oJLuyNmMUUdjgbt4TwKP37PKuYaDSu+0Jdza2ahk0j9cZAsKScpjiVFa/VFhqozVLjMmHNPS0/pCp9RBAk+7z+G5H3ifPBYzFi2nHfSo1WzDfzbAIWmV1aNJFaiXXHgRd1TEaKfpSaY/BCqTRPqC3UkN9dcgjNDXfqAaI8u8dhjc8DC2+n42nPHE8IXNwEkhotJbSMMRlPHJSGXGMZ4DuehBuHZyeUB+6nf3lJtkgAR3han8Kz8hmq6PyME7ckt+jl5DVQS/wyYyvWHFkUUgzeY9PGswHH/FrrZR53lkH8Rk509cp4byNLbf6elCmwtfVzYgce5CAZ8rE+tKKMTUcAZObU8tftz5JseyxDHRA8tsI0FOTcHR3HO5MfQpE8EpMS8q3b1IMMPJCgsN+9+6Ax46cHlvkElcZB4L1OACb2iu9tE8RSsy6bpeRju+QsT1f6jbqzNhVvN8Ci40IgM9WJ58Tgf3/TN+bFGTORBdsgnDo6+85PWn9uEv0XlXaw/UECrgeNC1jQYhhWZndGIVUyp3th4QkuQI1t+rbeEptKkUHfZme8Eok6oxcff58vSyDErh1SLM1t1K/ZrMvTo9moKT6TLvXTSsS3qEDGVW9rF+EWmiEa66LHUd5gQnIqOmh28GjmSDW6nCnhGzK4EY6+KGzwrMQJQsM0b9K1g/QJ37B81UNksiObcg7c3v4PtokJj27m9W6zv9ma3MB87VRo7/7QhYcMyYwcY+nuKtMkWumRgbnebjrqDlPeYkzN3r0w1ZczwCi5BPT07pREKlq5QibgVfv8A1WwZKL7a9IYmigFCtbwh9pUqE5pueNI+LhKJM4usA6RpcALLz9OHbjLw== X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9de6066c-1b17-4f10-9f1b-08dbeb3d9254 X-MS-Exchange-CrossTenant-AuthSource: PH0PR01MB7975.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Nov 2023 09:29:48.7083 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 1WOACMTPFMsr1ZRP3NMuIddXCHOyDZdS7HnpFUDbsufymSDzIGZjwsSqhKfMLlBK3jb2Kd6g8m+wvB3S1mNoqZnnE9f+IAtmCahJQ4FE1YVkw60qbSxbD9XKLAgqbRsi X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR01MB6760 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231122_012956_561200_C03EB7D8 X-CRM114-Status: GOOD ( 11.24 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add the loop_for_copy_128_bytes macro, to make the code clean. And make preparation for the next patch. Signed-off-by: Huang Shijie --- arch/arm64/lib/copy_template.S | 58 ++++++++++++++++++---------------- 1 file changed, 31 insertions(+), 27 deletions(-) diff --git a/arch/arm64/lib/copy_template.S b/arch/arm64/lib/copy_template.S index 488df234c49a..79b32569260c 100644 --- a/arch/arm64/lib/copy_template.S +++ b/arch/arm64/lib/copy_template.S @@ -10,6 +10,36 @@ * files/head:/src/aarch64/ */ +.macro loop_for_copy_128_bytes extra_ops + /* pre-get 64 bytes data. */ + ldp1 A_l, A_h, src, #16 + ldp1 B_l, B_h, src, #16 + ldp1 C_l, C_h, src, #16 + ldp1 D_l, D_h, src, #16 +1: + \extra_ops + /* + * interlace the load of next 64 bytes data block with store of the last + * loaded 64 bytes data. + */ + stp1 A_l, A_h, dst, #16 + ldp1 A_l, A_h, src, #16 + stp1 B_l, B_h, dst, #16 + ldp1 B_l, B_h, src, #16 + stp1 C_l, C_h, dst, #16 + ldp1 C_l, C_h, src, #16 + stp1 D_l, D_h, dst, #16 + ldp1 D_l, D_h, src, #16 + subs count, count, #64 + b.ge 1b + stp1 A_l, A_h, dst, #16 + stp1 B_l, B_h, dst, #16 + stp1 C_l, C_h, dst, #16 + stp1 D_l, D_h, dst, #16 + + tst count, #0x3f + b.ne .Ltail63 +.endm /* * Copy a buffer from src to dest (alignment handled by the hardware) @@ -151,31 +181,5 @@ D_h .req x14 */ .p2align L1_CACHE_SHIFT .Lcpy_body_large: - /* pre-get 64 bytes data. */ - ldp1 A_l, A_h, src, #16 - ldp1 B_l, B_h, src, #16 - ldp1 C_l, C_h, src, #16 - ldp1 D_l, D_h, src, #16 -1: - /* - * interlace the load of next 64 bytes data block with store of the last - * loaded 64 bytes data. - */ - stp1 A_l, A_h, dst, #16 - ldp1 A_l, A_h, src, #16 - stp1 B_l, B_h, dst, #16 - ldp1 B_l, B_h, src, #16 - stp1 C_l, C_h, dst, #16 - ldp1 C_l, C_h, src, #16 - stp1 D_l, D_h, dst, #16 - ldp1 D_l, D_h, src, #16 - subs count, count, #64 - b.ge 1b - stp1 A_l, A_h, dst, #16 - stp1 B_l, B_h, dst, #16 - stp1 C_l, C_h, dst, #16 - stp1 D_l, D_h, dst, #16 - - tst count, #0x3f - b.ne .Ltail63 + loop_for_copy_128_bytes .Lexitfunc: From patchwork Wed Nov 22 09:28:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Huang Shijie X-Patchwork-Id: 13464322 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CD8BDC61D9B for ; Wed, 22 Nov 2023 09:30:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=V83tgn6RnptufLL+Z1wKIz/LYsHWtvM69sWgHc6vDfM=; b=Y3CcF+OceJsO3i 9Hs6DEWGVxwnR016MYNNkjkxumC4hFzlRuoXsFoS/HzZGcp2QY1pUXOhpus6oV62F/uQcaS5rbU9L CzDGvoVuZP69eK+fY6A/6cIFWx9gSLZQaP4xrPWcJ+vtIOTKJTGPK13kGntKIRQSNKGWrieooLH8Q aJ/fjerWolrzzHO6ZGn4T7vgtM+4unkEFX7IYiZtbMsI5LE0VHeb2BPAEWkJ6DK8TJzhyS3ioDykm OlXkDTaxE4ySZL0xdjoTI7X6TBZ8PxuJoETIUr/gjdMUhYxCbZDD47/3Fe6afKRTo2tIuFchlA36b BEqIgNT/iMgWcmcUKZkg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r5jYa-001D0t-0y; Wed, 22 Nov 2023 09:30:04 +0000 Received: from mail-bn1nam02on20730.outbound.protection.outlook.com ([2a01:111:f400:7eb2::730] helo=NAM02-BN1-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r5jYU-001CsA-0m for linux-arm-kernel@lists.infradead.org; Wed, 22 Nov 2023 09:29:59 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Skg+I5Te1f1G/Hh4L+QL04KGogE5GLMlOvcNFqf0maUJl+KNU45It5tlzt/+8Km3O8qKX0SYt/hACM5PPW2EzJ6vW1+zqIaqh3TJ4biiMo5AKGyHR06w4v9shjAkNO8wuyyq5Y6XvLfcB8kCWb1xCkR0dKsQNj6wHzCNjNDuOl1tFUKQBUNFVNlLfSpiMYU11BtuUOPcfnqoRT9ezYmZg/uzC7A6H9MqQWR2bF4KOaJ2v0sCJwdc2GWKPDJHXwv14sMIyAl09dfPuANZIneB7qbF/dmCGdKuYK/EBmQvBevHZNZfXCYq4FsHZdHwDYZ+yZk/NXTBLPACZWIFPXLICA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Qvz1xzjMMv4Pv3SIfIGoOTOkrpJuBpE2oKIozxq3/dE=; b=Y2/E9xoegL42B+mCsGoKywyAiG3RAD3Z3jE31kUIDVUEoYLVvJfnAznn7P9dROd2zFZVeq6gjsttznTYsQzcDgnxBh15dvQ0xm8o14HZVgg95qZtCUL+wI7a2CxfJa2oUWnRHxftjO40LBqFpBB88hQT//RpUP5WHglAtk6YjHWZ6/+6FNcQVwIh94a2UgLuSZWp30+Cbv8fbSx1Gq0HOoUHRkVZUXuwyk6UaDPK6h9l/g0YZ7VOOTVX3LYy+oE1kq85KZTlVeyG3eSHm6GmAwyaIyBBj/u4blJ5jjnAJ64I+6rX1qHan+FQztdzvfLX+mSKyY4xGcniA/vZ/sHa/g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Qvz1xzjMMv4Pv3SIfIGoOTOkrpJuBpE2oKIozxq3/dE=; b=Va6NvMrOWCoNyvt+Q+dZnf0c+5llouw7C7cW/sRINvF5OCQs5bwZWeYzH98JYNIsNfwkkxZzHLBZMzMmaAfU1YiJMJJ+3ZiHv8j/Foecdz9bkeBx8CexeDjDyXWAKI5K15k/9bBE3q1cLKlZOxVSOFamkVwrTUgDWN2+O1lBxQA= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from PH0PR01MB7975.prod.exchangelabs.com (2603:10b6:510:26d::15) by CO1PR01MB6760.prod.exchangelabs.com (2603:10b6:303:f2::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7025.18; Wed, 22 Nov 2023 09:29:53 +0000 Received: from PH0PR01MB7975.prod.exchangelabs.com ([fe80::3f45:6905:e017:3b77]) by PH0PR01MB7975.prod.exchangelabs.com ([fe80::3f45:6905:e017:3b77%7]) with mapi id 15.20.7002.027; Wed, 22 Nov 2023 09:29:53 +0000 From: Huang Shijie To: catalin.marinas@arm.com Cc: will@kernel.org, mark.rutland@arm.com, suzuki.poulose@arm.com, broonie@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, anshuman.khandual@arm.com, robh@kernel.org, oliver.upton@linux.dev, maz@kernel.org, patches@amperecomputing.com, Huang Shijie Subject: [PATCH 4/4] arm64: add software prefetches for AmpereOne Date: Wed, 22 Nov 2023 17:28:55 +0800 Message-Id: <20231122092855.4440-5-shijie@os.amperecomputing.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20231122092855.4440-1-shijie@os.amperecomputing.com> References: <20231122092855.4440-1-shijie@os.amperecomputing.com> X-ClientProxiedBy: CH0PR03CA0247.namprd03.prod.outlook.com (2603:10b6:610:e5::12) To PH0PR01MB7975.prod.exchangelabs.com (2603:10b6:510:26d::15) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: PH0PR01MB7975:EE_|CO1PR01MB6760:EE_ X-MS-Office365-Filtering-Correlation-Id: 6e0a9d07-f217-4f25-097e-08dbeb3d9527 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5o8NkJfBHJ/wde5KDjcTyIUs0tOj/P1YVCR/umXEBMEWz74rzano3bFoAR8a3VxNrkKX2J6OfbHQNqGejrNfu8jG58eNo8FSFjg38dISKZUykJ1EyXr7gU4IHKc/hJUbr768Cqy+Lpc3uNGkbPEFjuI8C9UsLBKaAUBa/l5+1r4kcGCcoQD8T29JYbChPcE6xrrQt1fZu9dljEtsa4DIRMFgNoy211D0+Sz4eev9uZ1UqQ5vI1tWxtxt1VlVm/jr+Z4dfrj6QgR8I75bFo5iNU3G2Es+lwFt1TeqEXK2wnhxsQwDlC6lIFVEYz07TYvnMV2F7y3jEVsx+TDUhcDB0J9Sryk5zAEuwEn+tv0SOSf7MkF070UU86Mk5dLE2XoIBMui3VoRbJmcHFbEI3OxnkP2BGyQBmtXNxNXPg/g7Ke9VdY8wz9vqpAFjT8oU8xSC2Ep7CVYeDW+JEXVkUWGO2b+8cQym+erbJHGi/US6vGAE2zCXNNHSs/JFPZQh0fKgqG6nxo+DNiQnbVnLpIOcH6H4QOnhxWPV/CAuKR6PQ7EShQ3g9GmORL/7h0CZwbRSGP3u3DiCTS+2nEejBJqwnvRFJ2io4vKjOn+lCg/RbTMdDyhFGaHwD2RZYL4ScMi X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH0PR01MB7975.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230031)(39850400004)(396003)(136003)(366004)(376002)(346002)(230922051799003)(451199024)(186009)(64100799003)(1800799012)(41300700001)(2906002)(86362001)(7416002)(5660300002)(38350700005)(6512007)(2616005)(1076003)(26005)(478600001)(6486002)(107886003)(6506007)(52116002)(38100700002)(66946007)(316002)(4326008)(8676002)(66556008)(8936002)(6916009)(66476007);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: YlRLB5IhmcVUud23x1PxCPadLd3M30PmghkRahcxH4WJihnBDHhds5WwICx9xBNKB24CgWVtTgPk8IGQfHkzPcemYmSsCwlZqdRDwRbwYAO2SUgwm6FJwzT8alR13RnE37xZqL0I4uIz1XQHozvjHuuQhVrfxZvbruYtWHVbL+T1lBr1onq7UzzsC5NLZswLcQQHjlu2yEbW9ZdjZFMN30ERox2aFU32ZiCtTVUiF/r/KPXY3Nqz3VlnSUOZ4eUyLq4YmXHnIA0uj1FCT7FFhyNxYTNwmqxYkIem4dQ3NClo6na4sRcQpndcr2o6ZeYty829Ghf20N25DWq8JMwsaGh66X/+kQZ/PSR6ZS7Ec41uA+PFXDV50E+yvKN6VGBUJ9UbTZy2yLDN4jVrHOMYobtW8rOST8OvhaWJT5HMNjESTmw2iD0Jead8EFw82hqtFCGPvn/0joj8pk84kIRQzD7rlZxgnPmAu1ohFYM+fNnh1MfpMwjvFt/5Nos5bPC4/bcbSjYWUxVjypqa4fO5AAMOurb8KdzhWg0lIgkX6pgMazF7Vv6EOIobRoDZRhY8qH5qj30IrVSHUqs+kesVgiH24cu6lcsTxXsaYL8XDjTqiaGm/S+8AsnzRU7251eED2X06nKw67V6WS81ZLlZLpoSmC0ZBSGJyxg9Ry1/k/Ba8qTQsemLvuJs8zhD/ctbHSufdYYOHIAg4oE5hXvx4LKDJal4OwmA1qhmgjUkYS5pDv5daC9Ygiwere2IXJj/lWX5/16nKj/FgkX38CBhyTrpFHxpUpGL0vHYk/IkB/DMrhqNfUNl+M6fuaFgLWLyvtLeqi3tO3HVHRS8PpGiKM5DKgMY+iRonxDFEKXP7KN8/tsBB5el7Lo3LU/WlPAP7pungZm94ugfI7n7ePfPn7LMf5Zhc80HhFbETaxUtv+0R3nmVkSpSSyckZ6wBlnd9HnjqnaLoDVfhUKMPC6J4tTxF6cTN/v4ewAHXOYLFjg5Ht5uQfsqTOItV0kohbGalwBT0lmiiOpZng9hfqJ1Q+QNZKQ8FxiXKZzO4hpmXtBi7QNgURiUWE/L4rsj4PddX8v45z9/qlCAEmK2THLn4MsfRs2XD4ADieuNWn7wQItM4AkLfk0f6KYG1J02u58lHH5c3SaUB29akbf8cUX0/sT67o9FMegREVzqgDn4U093sL/+KS+JTHfL/TV6lyp/vCeHGhGZ06DiAzHwFV0YHwtncQCWkZfOJhVJlySdRBM62lc6raX6Ypq0tduErkJ9ze6umt3WasFztyn7CvtvJdy7b4/Kk8C5T91gMwmBcmnolbZssf7M+y56QSLdyS72Ki5rKrQNn5ilbC1oSPRQ7N+7mgCDo+uRfPSmX1gXutu9ri6Ickwic2gRi7gUkH+wLG6xeu7FA0bOyiHYqIXyyrVCYjWgaKMAMBDeo9ezwFusOcwKoN+DW2xJ9nVSrgJjZi0xx+bZGqTOLlP9tove+1XK+bLw9g3NWn3Z6ni3JZNXOgV3jGcyUgWxlFaE6M+To0VHpAxjqNEcGPH5hJB6YtsWZflzKtGdhcaAXLUgfHK2lzytvIve5K+Ekc4XABYFeNc8ByRYXQC3r5EnSw9Ecq3xLeHMPmRQpmiflXev1Fg= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6e0a9d07-f217-4f25-097e-08dbeb3d9527 X-MS-Exchange-CrossTenant-AuthSource: PH0PR01MB7975.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Nov 2023 09:29:53.3077 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ZRoMnSk7nItgxt/v/2aNmDCN4Cll5bqU2bcTASzOyAKKS/5EQhIpSnk8oIjP2qtjg2hXCKxOZtF3bRgenp5/XjuyP78YYonhMrCpmPo+V7LNNSMTTCBkoVtRlkU469L0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR01MB6760 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231122_012958_281603_D71CCD69 X-CRM114-Status: GOOD ( 13.65 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org 0) Background: We found that AmpereOne benefits from aggressive prefetches when using 4K page size. 1) This patch: 1.1) adds new WORKAROUND_AMPERE_AC03_PREFETCH capability. 1.2) uses MIDR_AMPERE1 to filter the processor. 1.3) uses alternative_if to alternative the code for AmpereOne. 1.4) adds software prefetches for the specific loop. Also add a macro add_prefetch. 2) Test result: In hugetlb or tmpfs, We can get big seqential read performance improvement up to 1.3x ~ 1.4x. Signed-off-by: Huang Shijie --- arch/arm64/Kconfig.platforms | 7 +++++++ arch/arm64/kernel/cpu_errata.c | 9 +++++++++ arch/arm64/lib/copy_template.S | 31 +++++++++++++++++++++++++++++++ arch/arm64/tools/cpucaps | 1 + 4 files changed, 48 insertions(+) diff --git a/arch/arm64/Kconfig.platforms b/arch/arm64/Kconfig.platforms index 6069120199bb..74ab8bea0019 100644 --- a/arch/arm64/Kconfig.platforms +++ b/arch/arm64/Kconfig.platforms @@ -8,6 +8,13 @@ config ARCH_ACTIONS help This enables support for the Actions Semiconductor S900 SoC family. +config ARCH_AMPEREONE + bool "AmpereOne Platforms" + help + This enables support for the ARMv8 based AmpereOne chipsets. + AmpereOne is the next generation of Cloud Native Processors from + Ampere. + config ARCH_SUNXI bool "Allwinner sunxi 64-bit SoC Family" select ARCH_HAS_RESET_CONTROLLER diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c index 5706e74c5578..c0060d3086d0 100644 --- a/arch/arm64/kernel/cpu_errata.c +++ b/arch/arm64/kernel/cpu_errata.c @@ -744,6 +744,15 @@ const struct arm64_cpu_capabilities arm64_errata[] = { .capability = ARM64_WORKAROUND_AMPERE_AC03_CPU_38, ERRATA_MIDR_ALL_VERSIONS(MIDR_AMPERE1), }, +#endif +#if defined(CONFIG_ARCH_AMPEREONE) && defined(CONFIG_ARM64_4K_PAGES) + { + .desc = "Optimization for AmpereOne chip", + .capability = ARM64_WORKAROUND_AMPERE_AC03_PREFETCH, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = is_affected_midr_range, + .midr_range = MIDR_ALL_VERSIONS(MIDR_AMPERE1) + }, #endif { } diff --git a/arch/arm64/lib/copy_template.S b/arch/arm64/lib/copy_template.S index 79b32569260c..b707c3ec6820 100644 --- a/arch/arm64/lib/copy_template.S +++ b/arch/arm64/lib/copy_template.S @@ -41,6 +41,18 @@ b.ne .Ltail63 .endm +#if defined(CONFIG_ARCH_AMPEREONE) && defined(CONFIG_ARM64_4K_PAGES) +.macro add_prefetch + /* + * Add prefetch two cache lines by prfm to optimize the + * performance. The 2K offset is the best offset which + * we get from the tests. + */ + prfm pldl2keep, [src, #2048] + prfm pldl2keep, [src, #2112] +.endm +#endif + /* * Copy a buffer from src to dest (alignment handled by the hardware) * @@ -156,6 +168,13 @@ D_h .req x14 b .Lexitfunc .Lcpy_over64: +#if defined(CONFIG_ARCH_AMPEREONE) && defined(CONFIG_ARM64_4K_PAGES) +alternative_if ARM64_WORKAROUND_AMPERE_AC03_PREFETCH + cmp count, #PAGE_SIZE + b.ge .Lcpy_over_pagesize +alternative_else_nop_endif +#endif + subs count, count, #128 b.ge .Lcpy_body_large /* @@ -182,4 +201,16 @@ D_h .req x14 .p2align L1_CACHE_SHIFT .Lcpy_body_large: loop_for_copy_128_bytes + +#if defined(CONFIG_ARCH_AMPEREONE) && defined(CONFIG_ARM64_4K_PAGES) + b .Lexitfunc + + .p2align L1_CACHE_SHIFT +.Lcpy_over_pagesize: +alternative_if ARM64_WORKAROUND_AMPERE_AC03_PREFETCH + subs count, count, #128 + loop_for_copy_128_bytes add_prefetch +alternative_else_nop_endif +#endif + .Lexitfunc: diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index dea3dc89234b..13e197abf249 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -100,3 +100,4 @@ WORKAROUND_NVIDIA_CARMEL_CNP WORKAROUND_QCOM_FALKOR_E1003 WORKAROUND_REPEAT_TLBI WORKAROUND_SPECULATIVE_AT +WORKAROUND_AMPERE_AC03_PREFETCH