From patchwork Mon Nov 18 18:16:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 13878941 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 30DB2D591AD for ; Mon, 18 Nov 2024 18:18:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=TnSYudl8LgJnfeoPjXZCrrq6bAGwcihoW/+IO3jYjJw=; b=lN/u0UuE9hZlHtjXQXHNgZ1IDU NW2T7fcvrYZceW1Nm07LoEKG70BYAG4G2ZdM7ghyNRKkjRpM18VvHB9YnlKfNEvkLjfnplRm2m/9Y +NI8IpWJLVlWc848eLxEJ3PAoFvFsGlQFeka5RiHte/j1IcYXuh4rCbMc/caOILlD9LZ2Ru9X+t8d MfHM2DV7rXo8OA38lVn9e/AGOmQBHfKET2j7PdH3sIbcxE2QrY82aQ6PhpVrP/olRvhWhYvUjS7Fg Bu4hVYQaiOA0Me7YiBSiBtmSjpwhJrkAJ8Sf8MO9dYlLgET+rWI2Q7aCVih6PR1UwmUXiZNlAkVjN rYIdbxYg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tD6Ki-0000000AJom-0Qz7; Mon, 18 Nov 2024 18:18:44 +0000 Received: from mail-northcentralusazlp170120005.outbound.protection.outlook.com ([2a01:111:f403:c105::5] helo=CH5PR02CU005.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tD6Jl-0000000AJco-1gd9 for linux-arm-kernel@lists.infradead.org; Mon, 18 Nov 2024 18:17:46 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ejEGfq2MewT4jq3JNJiBitwAzxwHyqQHlBgse3723dx5Nu94JH3KDZk1YVUMq8zvCn8RvygTY/1VxVUnWeERLH4DjnU9pbKPdNKF0JUzKBDAwDHO4KU6RWbEhZZ5be0fPAMQEFzo22s/FnJngh33p1Njke/ALLYEC1M0MEuEjnWpCjKbouzKi/GGDOv57v0nj1xLek5LeSglKsUbzK/q9ujWmGC/lcziLRhofzk5ewgbDpc4OkKeEZVea5ObkrJ8rP9uAFxpBrGx6JPKJ4QXsNeTVpr7eukwWpIDo8ar6c5coaOwDl4qKYPko3TOPO0Re+7+1WQFSz3uOSHqjsBgig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TnSYudl8LgJnfeoPjXZCrrq6bAGwcihoW/+IO3jYjJw=; b=JP//7XOWhnqHzND2FTt2Cg7gjt7UMpKj8aCndkcLDcW3IXQgxekmAYnmec52wlHnWTBN5W8E/JWpwOTABf3Hd2Bq9XFqynTvRHCQxdCvHRLX3nWISGwS/JHHrblFnRMroMuP+ylq3DPgT5z7rbEfJZdVsaxNw2/n99cz0kpu9erJAoK7gl+V8LqgdkQKImHW6iCu7guzobyetbmKoROeetRKAxVaZvl28hsR2IEnoVVCTbLb+9BAvLmyqU6MfmDfvDOKHqLw7FBv9VqxfA6XSRg24OC5U4+6VYYbaCtAHJE/OM6w+idnsgYkfpl+XoWAb6fCy6olhgyHYR8bp57tiQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TnSYudl8LgJnfeoPjXZCrrq6bAGwcihoW/+IO3jYjJw=; b=XIVlTWxdRQux3DoGWx5wQWTLBvkZaCHPN6l8+1w9Oioz8ZWVfA4c7i0Vw98AVU1vv97sH/CTXzdGHKeZiYdsaKAc3NcUx63xi6ypr03GNXz/Hv618lx3yawAGsQnqihT5ArToTWGBx9HHxyctGEJYHx4kZUXo0J9PhTzVYY+7pc= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by CYYPR01MB8601.prod.exchangelabs.com (2603:10b6:930:c5::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8114.20; Mon, 18 Nov 2024 18:17:40 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460%4]) with mapi id 15.20.8182.013; Mon, 18 Nov 2024 18:17:40 +0000 From: Yang Shi To: catalin.marinas@arm.com, will@kernel.org Cc: cl@gentwo.org, scott@os.amperecomputing.com, yang@os.amperecomputing.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 1/3] arm64: cpufeature: detect FEAT_BBM level 2 Date: Mon, 18 Nov 2024 10:16:08 -0800 Message-ID: <20241118181711.962576-2-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20241118181711.962576-1-yang@os.amperecomputing.com> References: <20241118181711.962576-1-yang@os.amperecomputing.com> X-ClientProxiedBy: CH0PR13CA0038.namprd13.prod.outlook.com (2603:10b6:610:b2::13) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|CYYPR01MB8601:EE_ X-MS-Office365-Filtering-Correlation-Id: 25208b6d-855a-4742-c88a-08dd07fd49b2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|52116014|376014|366016|38350700014; X-Microsoft-Antispam-Message-Info: x0MJm+zd3HCcLtuEQ8BCjbULl1BCHKhNYIYgoDvCy08jP7vx90YBrVZlfIlBw9+a0fD7bpYnC7oC+lRDvUs0khRgowyNT0WcgGY5x02mnrta+ia/rEY4InpFXyaiaSIHVt4hLsLiPQtSMQNMyGgynLIrbJzzyjqni2TmTEv/90yfELvkVIr/rfBcxLnmcW/Kkx7cR7xnsS/BxRKZoQ699xigoM0x4H/kPYXv67EXQWAF3NgzsxanJVJhhXwvKpteG7AEnkmeqZo7TCg4AvsVBH9/m73v2rZmjVOCkKHkpC7aEgh7r5DrsfGZtOJdETIZH9zkBpn+huXVKOD5EH2pEnOOVFGBcsZgxIeDhVISvyBcnEo95g8ZKN4vrEG8+L8cKDeYhM5KRMOWz81WOUVH6ko+uo2AonfCq7ySP+bGtf1y50sRYBuREghr5w4ra2zHafQbaXPeU34B7ioNV++AoIK3/pRSXII9M640FJpoM/ndBV8SZE1uERWVlS4S2DDTUrewcqqRea9F/NLdrOzqXWx9JZl2TYbqTTNIn2ZkUxSRAEVxOAjyUSE+TRZAN/RJ+hlEhCvoPnwvs0jMT+nvJTi1xuBV2e6KGzTimxj8/sjdzRmO+EmJUk0ZqfG+bdC/GN1a1tuka+v11kCQ2MADe7w2mKZXqB9EittyCa8j6dg0+cmUXshPPaDCXssUPUZe1HA1iwkRUUyaVj8H0VrQOeralj4RWU4GYIUETHOd6n1GLfF4DvDaftL4T1ABG2mz6AnhyGV20EGuWZLvMdyMIDO70WO0GPfiYS7Lg0WtYrDTVGI6BlNnTh45LpotwGLSIq/BLNnBszygsYJOa7qzieDt/Ivuzh6CcmIsxHi1mLc2zb7mO4HSTEWv6pzSXkKgLv/ddNLNvPoHi3x2Vrox8M3ulDgIgdzGyi9DGJXkfK066Sn2JFE7cgTox8r9yZQld27hxNuatKqmE8BJgzJKO84ZyZ8rn5x+3vCjM5SeFqfNSm/n+y0wnMnEvh/DHJnsNG3LEri/uSERBJhbnYS4hZ9w80Go8+vQTjoYNbfQCNJMyLUGPQFMayT+W7pqvY/P5NYb75MTKho3qbjEym3bN7kTV1OgVg2UflJhdqqA5I3YQt2cqyISEuYaUyLVNIGjaweXEC0NjSzX9FMBtPt2xwyqJHGyGRioBUA18hW5OjG5OlqNmp8WtFSW3WM+x990fUJ6hGZNcg5KrHq/z0vKcZ/qc2bQIf97pLPsz+JLyQE8/AGnF5ejieYPcR9lR0er7MieIJFfB4HzOcuOoAKhWFe01EJCbXoPBqbwKGUgOJOUjQg54BaHB/MuyFtmmtkHtnfJYpl2RGB43ON1U15NUcauo74L/8uQdmV6MBb19jo= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(52116014)(376014)(366016)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: HA+NuMCEDTERAKgZcZ0rsi33zbb1DxY6NU4bqPRQST4rksJ9PeEGGBVHk4+1TC8J22+ngWq7GmfDgne9WOJH2mN6WImg4aWwesRVJjlNr/E51IE2CeW9VNzjNbBEHUZv6uo327D0a02WY/NugznQa1TLssWfu/hHk1S1aH7ZrSjZyJ1NVHzzGIqklGEFy04o+9e1qUHUWKrhCqdfaXKHUik6qcS80Q7JOroKTZZLMubtUZPz02iUeyriyVjrOJi6iZLlGyEbUW0n8ESoQ45CjF+4SJIMzpSPn45HqLcHzK44Y9j2sJd/MTU7ThwK8O46XbuH5Fh+t0x9eRSYHo05VnByb0fEwoF5jVMKqOotA/CEc97g1cLsIt3qeii9X2EnHm6DfvcFbsykRTzp5ZuFMdNdKV9RFQAOk1yVLdh//vvyzyvPBksFDJsFAWcHfgs23YVaTBAWNxvZ1Docw/9BAhzpoLYH9VIKQKV6InDm7rxfKzn0LSfzix0jBr3derMiSW0jkpeFHAruUzrdNh3/9dcT2mnoeCs6HFkpg4kbGKo+LdVFPTZpp1HJVXoMEqfJ1bfp13FNqoBuA/HcFvf7RmyDLvZI09PmE3+Vb8pYv8OjytfJtbcDzklGcKA2HjsfSuYgFhXcZRyTn/+9tSOcAKmbrJAU1owNZeA2Iqj26Na3Iwu9gFY5V7P5HOpNOQ3MX4gKf4+BC+kP4EQhQkHfioXE1bd2RtHdQuLmOGwC1kejtHqsSQqtF7yjcHSVnuJiU5IzVRVZvI9GHbhfXYpXVFY8UDgHT2w40Jw24nau4FiA1fQEKxpGKU4Dnc0z7HsChvu8zDlNLUEFhxzvUvgQt5B7wWQjLdgpJLv08iHr6Bm9DXHamlKhHsNgd0sVoD6vU98Ewb1AJhanQyEZnsbO0zBnhykmDKmy2M1RZqAjNYSIs+W/S+5K9Le6R01FU/spql1scaZJY2zyuypP0k7Qi5H8nSqsAlNsMaxwCEp6yb/+dODioEeNCjUkWmyu+LChRXrqkmod5EoGWrJc7GkFj1l9xNa7/s3tLtEOnKoqy9ODd4iDcyyQU9s3q61CkY0PXT2ocG4zHnVIGISHcZADmGX95Dry+qLIDdu+0M0mkgPg4sbjQyTTzg+iE7FfC5jKIrOmX46jY29jQCUTWjpfJFbZVuq2f1qa9Q5crHCCFMo85flCznUoc53eLswb4mSIe6CSPbpojNlLeR8Q8aeiPOrzzD4DI9yF11juc2+3ws0HZHR8Vy/EqhstSyajYG9yoFaWecoXX+f/mrrfJD2WPIRNz/Vn24jPjCywGNguTQH3UFv6XzB+UyWO6BgGmKVLq0kzpvGNqrcPXZzaGoKo4QKiRFnpW5ct1UjxPkr2zP3DKKWP7V2N+OSUY59WjqjHZofJ0O9kG3YufAdRb7jV2Z9J5pgsKqrziTAJd1bgrpVmIWPDAvS/U7AA3+mlUnXW3cnkkMgX21pD9JNfC+RgxezP8Lq8MW/WKdxysIJb2xY10OyXLJ1ogNJHkjzO7ZuuxrjqQ9ro1BU3kb9QFezcgkVy40cvmLQCauuMzQcYmHTla+WNQaRryJA+lXdIKTuyjjbnjmUZQKGizsM9rEPyOHaGd9X1eD4WYqVs8spqhJg= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 25208b6d-855a-4742-c88a-08dd07fd49b2 X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Nov 2024 18:17:40.2416 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: +zQL03NfoA4XCaxHG5FB8ePWSM/2nH1dPxIGm3onmF+CncmVYeK57TFOdmLfHhiGUPvzc4KJKNCUxDlAuUbLu791WhWVmGTugkT5EpU3V+k= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYYPR01MB8601 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241118_101745_456212_4ABA386A X-CRM114-Status: GOOD ( 11.42 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org FEAT_BBM level 2 is useful to split large page table without breaking the page table entry. The following patch will use to improve performance. Detect it in cpufeature and use BOOT_CPU feature for now, if the late CPU cores have conflict, kernel will not bring them up. Signed-off-by: Yang Shi --- arch/arm64/include/asm/cpufeature.h | 15 +++++++++++++++ arch/arm64/kernel/cpufeature.c | 11 +++++++++++ arch/arm64/tools/cpucaps | 1 + 3 files changed, 27 insertions(+) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 3d261cc123c1..c7ca5f9f88bb 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -838,6 +838,21 @@ static inline bool system_supports_poe(void) alternative_has_cap_unlikely(ARM64_HAS_S1POE); } +static inline bool system_supports_bbmlv2(void) +{ + return cpus_have_final_boot_cap(ARM64_HAS_BBMLV2); +} + +static inline bool bbmlv2_available(void) +{ + u64 mmfr2; + u32 bbm; + + mmfr2 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR2_EL1); + bbm = cpuid_feature_extract_unsigned_field(mmfr2, ID_AA64MMFR2_EL1_BBM_SHIFT); + return bbm == ID_AA64MMFR2_EL1_BBM_2; +} + int do_emulate_mrs(struct pt_regs *regs, u32 sys_reg, u32 rt); bool try_emulate_mrs(struct pt_regs *regs, u32 isn); diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 718728a85430..cb916747cd31 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1866,6 +1866,11 @@ static bool has_lpa2(const struct arm64_cpu_capabilities *entry, int scope) } #endif +static bool has_bbmlv2(const struct arm64_cpu_capabilities *entry, int scope) +{ + return bbmlv2_available(); +} + #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 #define KPTI_NG_TEMP_VA (-(1UL << PMD_SHIFT)) @@ -2890,6 +2895,12 @@ static const struct arm64_cpu_capabilities arm64_features[] = { ARM64_CPUID_FIELDS(ID_AA64MMFR3_EL1, S1POE, IMP) }, #endif + { + .desc = "BBM Level 2", + .capability = ARM64_HAS_BBMLV2, + .type = ARM64_CPUCAP_BOOT_CPU_FEATURE, + .matches = has_bbmlv2, + }, {}, }; diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index eedb5acc21ed..175b7eb42b0b 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -14,6 +14,7 @@ HAS_ADDRESS_AUTH_ARCH_QARMA5 HAS_ADDRESS_AUTH_IMP_DEF HAS_AMU_EXTN HAS_ARMv8_4_TTL +HAS_BBMLV2 HAS_CACHE_DIC HAS_CACHE_IDC HAS_CNP From patchwork Mon Nov 18 18:16:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 13878943 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C073AD591AC for ; Mon, 18 Nov 2024 18:20:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=fwhb2/6B0cSjLz3HSByelLgDMMOQl4Nq5+24O+ATC/E=; b=WZiUnyYJcMf1EQX4DRxXmfE603 eaLa2bHoAIc8vEvOJOHZckH/iOU/Hc6mIWvN2kPqiBD/UwexQ9VWr/GKX1eym4Aj3we1+wxCwSBpp J3x73gnMVJ4Mbi/H3yeF9YjChN6isDBjFhyrBCFOdlyftBh8pZh9A+W2ttkCbESVoIDmzXtLG9+GJ MjL4++5TsIFtE5XviTQtH35aRRj/kzWslUUE2Zd7ZYI/bO09KoVChE60kCzp/vsPELA83KLUOJvfU MBcVtoZ6T3gqYw9OBXcb4qcqJU7S46rCdkBHHEwpAAwAjF+U1EnQfWWKKn+IDUE/Z4h6Eb/QXvmJV 05uhYDgg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tD6MV-0000000AK86-1VJl; Mon, 18 Nov 2024 18:20:35 +0000 Received: from mail-northcentralusazlp170120005.outbound.protection.outlook.com ([2a01:111:f403:c105::5] helo=CH5PR02CU005.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tD6Jo-0000000AJco-0Q1A for linux-arm-kernel@lists.infradead.org; Mon, 18 Nov 2024 18:17:49 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Xcw8zp5M7kSVWFZg6qN3SNnkcZ55+GQuD3lvxFS5I+zfLTYrBG3Ih1uEp+AOJnGlPx+0VFnIZv5ns5//yvTcbNJJBz5vVZ5uvow2lOImLas7FusQXqMR33K2FchQLYqVeOVMwM7IKyAHspgIOBIJ/m4PhidhgghYs0DCThaJxQQ+wvkZq4b+SucCMNjX+lxwZMYAaPNIh/20X5pqsoAralolQc0PimHjWRu3C9pWxOOGM3XRgV/8c9iVqYh0pJKHJFa/VuxSLzXud8XQleLwr9+hgEAg3FYiceG2ykQ6UzMtTNdnCi+lOfMfcRO+tJuNzPoyEeeYRXVXranlKwTcDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fwhb2/6B0cSjLz3HSByelLgDMMOQl4Nq5+24O+ATC/E=; b=KtIwM5iNa1GgMkI1wU/+hgoG3Ap7KIkcXMf0tIliITTPoz8+HWPu026jIwdRvBtEx/W0D9z2hq6jDo2jb4zmnGyLphOMX9gZEHgO9G7EB6yIWAUtJdffxBB8EiqlPNJByzgzv2eEcaQRSTCcm+zpqoWAQ+XvqlxfSl05SkylT72p+nAkdcQ+99NEKoMTwlZFr1GWQsFYwZbwqAbqnuGrDqZg9Opw6JWKLbEUxZOg8XRrwGvNSzqBV3ZIIvpi6NBCc08fRv5GnHPFspF2q9jUHFYvbc01DTtG7YALlBhYqcMKQ5JlIJWs5yw3sSExzUMxG1g8lIQEj1bxir9PeGZv9A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fwhb2/6B0cSjLz3HSByelLgDMMOQl4Nq5+24O+ATC/E=; b=TZl6G2ikiXTIYp8/ih7PktVn6oDU9ZImzRrZqgGG0OHyjI8QkIYGs7kpP9lAeNYALF1ofYwMUODUpNe2inGnSAn1XYHur5iXWKJYoOVN+OmBrnGZqPOvnsnJd6kC8A1JLlxbuNkBFWyEp/tU6CamzZpJxaEeIVjhAp3IDy2KdFo= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by CYYPR01MB8601.prod.exchangelabs.com (2603:10b6:930:c5::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8114.20; Mon, 18 Nov 2024 18:17:41 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460%4]) with mapi id 15.20.8182.013; Mon, 18 Nov 2024 18:17:41 +0000 From: Yang Shi To: catalin.marinas@arm.com, will@kernel.org Cc: cl@gentwo.org, scott@os.amperecomputing.com, yang@os.amperecomputing.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/3] arm64: mm: support large block mapping when rodata=full Date: Mon, 18 Nov 2024 10:16:09 -0800 Message-ID: <20241118181711.962576-3-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20241118181711.962576-1-yang@os.amperecomputing.com> References: <20241118181711.962576-1-yang@os.amperecomputing.com> X-ClientProxiedBy: CH0PR13CA0038.namprd13.prod.outlook.com (2603:10b6:610:b2::13) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|CYYPR01MB8601:EE_ X-MS-Office365-Filtering-Correlation-Id: d9ae7aa2-ba32-4f81-11a9-08dd07fd4a25 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|52116014|376014|366016|38350700014; X-Microsoft-Antispam-Message-Info: NNdD+qhKXN8K9huuDaac2zdVldt7oYEdRkIhtlztDDHzvjN38q/FwEitsYywDBgagsoVaRWhsiO2r4qvsxRT7F7Uv/YWWTtWF5+ZEW83/GouCYWvchrxDVSUHKWcL1GBKpwu2WnfzNkxv3jVGrWVHX6xh9aANJEIDE30cWEVc9Qytfx2BZuhqZbwrwiS6XocMNBzaShTI65qdluV5+LvZt1F2FldKLKA8cPqEPNXopQlSXh/XtpRNRCNOSWBFmU29O2t1T2q0JJJ5MhZiFcMNnkvVLYQZxasxD5swioLJYnh8AFYmIw9DBgTKMQln3QF7cFLq/8bio4Vnyiy2I9+BjFkiVvjlWNGhys7oFrB/kedhwMq+qJgWto58b00+6qvh7x3PFiSCs0uVL7twe+6ojEvzwFsEPOGjD6CV6g6cxjE2EKcGcjGfH59BZMk40Dm4V85098xiuqeZJ5BXMxgshz3loLg/9EXrVXzpDOOAWCJD30j8dsRXhGpST0J0FTKG2Z272yXy36VRKRXunlJOjua0uTuCTBVfJMktXd77SeWDwmf/Uzk9+aTo2TRGjKmtqOoDD4xGMv9pkpes2Cxi4vUFLxtp/fkBd3dbI4GebwU5LldLUCOeaH/J3Sy8o1KjrWXvnwpm0bVIgJNMa3VW/ROlvq/AxljwWfiDWC1diof+caBAaot2gi9piGMyW0LWacXmV3vnnQd/UX4zUzd1HA068mdfhVbCvcQLvSnnLqN/2uEtYqqYyyqqBGY7ZWubImGtuq4slKz+eSw7Gs4SN5AUbIRhR8ViO3gMo/NqpP5gknh/nl4tFIscx5xID+cImBQWv2N/H0UD8KHPEZYDL5GhTLOYyO4Rhdp8JH3NIMiedTXUOnPU9B6h1nFlLDGhd9ndqYJ7OOmj+ZXcyGxXQx548hjnKef+v5aMr8O8+mF1UJ1z/TIg1eH2O5Koa9wv0Mj9YcTGbPFgmfHb5EVAhlmGRkRZbWXd+8D4wnFgL2bw1mo45QrUgMV09in+IVRT5oAHbmvL/H+UBShbjlULcIhdKcRIjxWkbmd7OjnpJnw3GHGEzY3r/hxki0pP2MsbYjTe7qe+w4cwkoWLYvenSUuA1/ECPQRc6m2vJdZFlb+wI79NfMEY1N2PNncYUlPWmIkGdTecxUM8FG/A1lk/Wkboc9LdbICWoh1sNe2mmOq9vBGQ97YIdg/SmZO+IlAPYac/YhjUKNvaJqt1kE6H9BcEPqMNXo8jPfaE6dBvwnTdsfJ8Yj3SnSkSA+kC8l1SFS+nQ56NtaBPcxZMn9oIbmPG0Nspbwl0n91Qa3ZVhC+A9Wdz/r/J6kU5O6mXSv1MSlHLhz8+cPtpRPj2sFkotSF+JpsWQ447zDIi8QGgEU= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(52116014)(376014)(366016)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: CqXmBIjl/UmbgIdv40EP/zh6Feruc1hDlA52H7UsUH73b/DWOopun6lUDhMhYap/kR8pK26N3M7U/WyJvc6ZQSUgHf1iEMEwpPxhfwjFMurlPFQ6eE0HxP4352U3dl9bdUQmETSjnRQpinjz5lKndAXykATIzv+i1QYJIKDuNEDyiPLZWjMrlSXE9WYdxkHkWD2bXUvOdgbTfmbCkIkBLB0uW2rvq0JLPX9PM7wM8/tiX0hqX8zDJhRFnESjLSr+0g62Ymw1KcZgxVRQEw0b0V13wM0plspbD6ZDUDftq7ri8Lti8RGQZ/S2bsXZktRZPBGs9piEwhC5hDm8VKqO9chKP61bJpRpgGkbO3/eVsYIMOUtQxUvVBDTxKMXlq8y9X3y2Wft2o/86zMp2GEeCsFPAOkd5j05tMssI5qNVwiwccs0TKtMVJT3rX+rZsENC7mj1snI9qHvUgIInhZkpd+zfWwsrFofZ+jb+KMa80NaZgIWRcwmRFoUl+uX+FHadMsuujja+qNXKbL27uLVWeyauZCwW0idBAJpNlvEyjD/hoQGJbRMw3k3INo3Z6XCAvaOdDNbUImY/d01leFJCH4AOyCyqb1Lxl76tezenwRHzWgl/DS5sXN4Dt936fwM/84G3rCp6NG9xfQdMtK9iXCEncBitQ/vTszJwTq/9zqrHTSv8CloiArKnRIUHlVrrqsJbBUEOl/zJzDuxlZe9bmQAHaHRWVHV93GN7VZ0gCSynIE/AMFBm0/ElzXLKCStiCW9J29x7KQXNN0ZRVhbEL1FSaVCxe24rEn8XMioavU4Qn33VXt7/Nb/lTxWtf6bDCQA/MN2jKuVa3kCwVQw2D/qEBKw9veAfK+WgsIVkYjyhNieiOveg/n4FDgY5Xg3qOUwPzypByzICSOTS42nnITDeu9IRPKDxaL1Y+yttFP3cwsUZY/rZWbfvqgww2xSifoKDZXkvCAf+A+q+nBz6dhBg6LrWBBmMh+4EUJsaNiNa9z3Ne3+nrDnPwYH4mTLuLrxy0a44kODdSp3Rg83zxP5ix+LziX0vvjQkwbXFxKKzviLdrYFs+CVj9kixChvTXPAKc5GbGCHqbzy64Gt2MyNioAQYZHf2cDPEPpgFS9Vx83sx42L3t9YbrgxN3R8Ffj66Fto08jrKs9r+hzBC3K5HA/a2Rl8gVwQYTl/7O9j6/r/XYTMVgcGlR2kphLTT6pvGKIOjOlL11h7g5jDciw/v0w/ICoMKxAiV5qdR7D+Ypn7Tlpw45trFvEhcSLzIEE+r6Tpgl7p9AmPojIoDRzP4fua72wGecnM2SSY0bE1M+vslLZbdFWKsYFg9Ck0pmouBbFAme2G6QEdPRcyNApoezF0YEdM7471ethZAAMLzTYf80ZpL1RhyGxwkx5cn5c+ElupFSBFQCoLvMiWZyNMofP61ZeW3W5yTN+25KePm3l+E6gkdh1AoVYMZNOnLAaHGfkZhTNWIWF600FcZ0RAx1sPLx3/ARZLKPsNm9nXSQWAqKekim8vJENd2zPuWQSgEqQ2GCYGDI1M6nHOn8yGqpuwUdxQCDNb3rndhb77AxW9Hf296PYBV4JiRvrsv6t00Fq/rGGCGOBjn9B7lsSHroiHVMEyheiCo41uzo= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: d9ae7aa2-ba32-4f81-11a9-08dd07fd4a25 X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Nov 2024 18:17:41.0000 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: eWbMCbKLBaxUyVJZf+iDIn2/+PFtQhzA9cwV5S7GBdPajz2RtkSTvAimry7oDwliuEcOyaIwJX22CnenPTifwWsMnbIRqHBShv4tbENq8kk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYYPR01MB8601 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241118_101748_142225_E90C2E8D X-CRM114-Status: GOOD ( 23.15 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When rodata=full is specified, kernel linear mapping has to be mapped at PTE level since large page table can't be split due to break-before-make rule on ARM64. This resulted in a couple of problems: - performance degradation - more TLB pressure - memory waste for kernel page table With FEAT_BBM level 2 support, splitting large block page table to smaller ones doesn't need to make the page table entry invalid anymore. This allows kernel split large block mapping on the fly. Add kernel page table split support and use large block mapping by default when FEAT_BBM level 2 is supported for rodata=full. When changing permissions for kernel linear mapping, the page table will be split to PTE level. The machine without FEAT_BBM level 2 will fallback to have kernel linear mapping PTE-mapped when rodata=full. With this we saw significant performance boost with some benchmarks and much less memory consumption on my AmpereOne machine (192 cores, 1P) with 256GB memory. * Memory use after boot Before: MemTotal: 258988984 kB MemFree: 254821700 kB After: MemTotal: 259505132 kB MemFree: 255410264 kB Around 500MB more memory are free to use. The larger the machine, the more memory saved. * Memcached We saw performance degradation when running Memcached benchmark with rodata=full vs rodata=on. Our profiling pointed to kernel TLB pressure. With this patchset we saw ops/sec is increased by around 3.5%, P99 latency is reduced by around 9.6%. The gain mainly came from reduced kernel TLB misses. The kernel TLB MPKI is reduced by 28.5%. The benchmark data is now on par with rodata=on too. * Disk encryption (dm-crypt) benchmark Ran fio benchmark with the below command on a 128G ramdisk (ext4) with disk encryption (by dm-crypt). fio --directory=/data --random_generator=lfsr --norandommap --randrepeat 1 \ --status-interval=999 --rw=write --bs=4k --loops=1 --ioengine=sync \ --iodepth=1 --numjobs=1 --fsync_on_close=1 --group_reporting --thread \ --name=iops-test-job --eta-newline=1 --size 100G The IOPS is increased by 90% - 150% (the variance is high, but the worst number of good case is around 90% more than the best number of bad case). The bandwidth is increased and the avg clat is reduced proportionally. * Sequential file read Read 100G file sequentially on XFS (xfs_io read with page cache populated). The bandwidth is increased by 150%. Keep using PTE mapping when pagealloc debug is enabled. It is not worth the complexity. Kfence can be converted to use page block mapping later. Signed-off-by: Yang Shi --- arch/arm64/include/asm/pgtable.h | 7 +- arch/arm64/mm/mmu.c | 31 +++++- arch/arm64/mm/pageattr.c | 173 +++++++++++++++++++++++++++++-- 3 files changed, 202 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index c329ea061dc9..473c133ce10c 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -750,7 +750,7 @@ static inline bool in_swapper_pgdir(void *addr) ((unsigned long)swapper_pg_dir & PAGE_MASK); } -static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) +static inline void __set_pmd_nosync(pmd_t *pmdp, pmd_t pmd) { #ifdef __PAGETABLE_PMD_FOLDED if (in_swapper_pgdir(pmdp)) { @@ -760,6 +760,11 @@ static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) #endif /* __PAGETABLE_PMD_FOLDED */ WRITE_ONCE(*pmdp, pmd); +} + +static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) +{ + __set_pmd_nosync(pmdp, pmd); if (pmd_valid(pmd)) { dsb(ishst); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index e55b02fbddc8..09ccb4f8964a 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -620,6 +620,18 @@ static inline void arm64_kfence_map_pool(phys_addr_t kfence_pool, pgd_t *pgdp) { #endif /* CONFIG_KFENCE */ +static inline bool force_pte_mapping(void) +{ + /* + * Can't use cpufeature API to determine whether BBM level 2 + * is supported or not since cpufeature have not been + * finalized yet. + */ + return (rodata_full && !bbmlv2_available()) || + debug_pagealloc_enabled() || + arm64_kfence_can_set_direct_map(); +} + static void __init map_mem(pgd_t *pgdp) { static const u64 direct_map_end = _PAGE_END(VA_BITS_MIN); @@ -645,9 +657,21 @@ static void __init map_mem(pgd_t *pgdp) early_kfence_pool = arm64_kfence_alloc_pool(); - if (can_set_direct_map()) + if (force_pte_mapping()) flags |= NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; + /* + * With FEAT_BBM level 2 we can split large block mapping without + * making it invalid. So kernel linear mapping can be mapped with + * large block instead of PTE level. + * + * Need to break cont for CONT_MAPPINGS when changing permission, + * and need to inspect the adjacent page table entries to make + * them cont again later. It sounds not worth the complexity. + */ + if (rodata_full) + flags |= NO_CONT_MAPPINGS; + /* * Take care not to create a writable alias for the * read-only text and rodata sections of the kernel image. @@ -1342,9 +1366,12 @@ int arch_add_memory(int nid, u64 start, u64 size, VM_BUG_ON(!mhp_range_allowed(start, size, true)); - if (can_set_direct_map()) + if (force_pte_mapping()) flags |= NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; + if (rodata_full) + flags |= NO_CONT_MAPPINGS; + __create_pgd_mapping(swapper_pg_dir, start, __phys_to_virt(start), size, params->pgprot, __pgd_pgtable_alloc, flags); diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 0e270a1c51e6..dbb2b709d184 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -45,6 +45,145 @@ static int change_page_range(pte_t *ptep, unsigned long addr, void *data) return 0; } +static int __split_linear_mapping_pmd(pud_t *pudp, + unsigned long vaddr, unsigned long end) +{ + pmd_t *pmdp; + unsigned long next; + + pmdp = pmd_offset(pudp, vaddr); + + do { + next = pmd_addr_end(vaddr, end); + + if (pmd_leaf(pmdp_get(pmdp))) { + struct page *pte_page; + unsigned long pfn = pmd_pfn(pmdp_get(pmdp)); + pgprot_t prot = pmd_pgprot(pmdp_get(pmdp)); + pte_t *ptep_new; + int i; + + pte_page = alloc_page(GFP_KERNEL); + if (!pte_page) + return -ENOMEM; + + prot = __pgprot(pgprot_val(prot) | PTE_TYPE_PAGE); + ptep_new = (pte_t *)page_address(pte_page); + for (i = 0; i < PTRS_PER_PTE; ++i, ++ptep_new) + __set_pte_nosync(ptep_new, + pfn_pte(pfn + i, prot)); + + dsb(ishst); + isb(); + + set_pmd(pmdp, pfn_pmd(page_to_pfn(pte_page), + __pgprot(PMD_TYPE_TABLE))); + } + } while (pmdp++, vaddr = next, vaddr != end); + + return 0; +} + +static int __split_linear_mapping_pud(p4d_t *p4dp, + unsigned long vaddr, unsigned long end) +{ + pud_t *pudp; + unsigned long next; + int ret; + + pudp = pud_offset(p4dp, vaddr); + + do { + next = pud_addr_end(vaddr, end); + + if (pud_leaf(pudp_get(pudp))) { + struct page *pmd_page; + unsigned long pfn = pud_pfn(pudp_get(pudp)); + pgprot_t prot = pud_pgprot(pudp_get(pudp)); + pmd_t *pmdp_new; + int i; + unsigned int step; + + pmd_page = alloc_page(GFP_KERNEL); + if (!pmd_page) + return -ENOMEM; + + pmdp_new = (pmd_t *)page_address(pmd_page); + for (i = 0; i < PTRS_PER_PMD; ++i, ++pmdp_new) { + step = (i * PMD_SIZE) >> PAGE_SHIFT; + __set_pmd_nosync(pmdp_new, + pfn_pmd(pfn + step, prot)); + } + + dsb(ishst); + isb(); + + set_pud(pudp, pfn_pud(page_to_pfn(pmd_page), + __pgprot(PUD_TYPE_TABLE))); + } + + ret = __split_linear_mapping_pmd(pudp, vaddr, next); + if (ret) + return ret; + } while (pudp++, vaddr = next, vaddr != end); + + return 0; +} + +static int __split_linear_mapping_p4d(pgd_t *pgdp, + unsigned long vaddr, unsigned long end) +{ + p4d_t *p4dp; + unsigned long next; + int ret; + + p4dp = p4d_offset(pgdp, vaddr); + + do { + next = p4d_addr_end(vaddr, end); + + ret = __split_linear_mapping_pud(p4dp, vaddr, next); + if (ret) + return ret; + } while (p4dp++, vaddr = next, vaddr != end); + + return 0; +} + +static int __split_linear_mapping_pgd(pgd_t *pgdp, + unsigned long vaddr, + unsigned long end) +{ + unsigned long next; + int ret = 0; + + mmap_write_lock(&init_mm); + + do { + next = pgd_addr_end(vaddr, end); + ret = __split_linear_mapping_p4d(pgdp, vaddr, next); + if (ret) + break; + } while (pgdp++, vaddr = next, vaddr != end); + + mmap_write_unlock(&init_mm); + + return ret; +} + +static int split_linear_mapping(unsigned long start, unsigned long end) +{ + int ret; + + if (!system_supports_bbmlv2()) + return 0; + + ret = __split_linear_mapping_pgd(pgd_offset_k(start), start, end); + flush_tlb_kernel_range(start, end); + + return ret; +} + /* * This function assumes that the range is mapped with PAGE_SIZE pages. */ @@ -70,8 +209,9 @@ static int change_memory_common(unsigned long addr, int numpages, unsigned long start = addr; unsigned long size = PAGE_SIZE * numpages; unsigned long end = start + size; + unsigned long l_start; struct vm_struct *area; - int i; + int i, ret; if (!PAGE_ALIGNED(addr)) { start &= PAGE_MASK; @@ -108,7 +248,12 @@ static int change_memory_common(unsigned long addr, int numpages, if (rodata_full && (pgprot_val(set_mask) == PTE_RDONLY || pgprot_val(clear_mask) == PTE_RDONLY)) { for (i = 0; i < area->nr_pages; i++) { - __change_memory_common((u64)page_address(area->pages[i]), + l_start = (u64)page_address(area->pages[i]); + ret = split_linear_mapping(l_start, l_start + PAGE_SIZE); + if (WARN_ON_ONCE(ret)) + return ret; + + __change_memory_common(l_start, PAGE_SIZE, set_mask, clear_mask); } } @@ -164,6 +309,9 @@ int set_memory_valid(unsigned long addr, int numpages, int enable) int set_direct_map_invalid_noflush(struct page *page) { + unsigned long l_start; + int ret; + struct page_change_data data = { .set_mask = __pgprot(0), .clear_mask = __pgprot(PTE_VALID), @@ -172,13 +320,21 @@ int set_direct_map_invalid_noflush(struct page *page) if (!can_set_direct_map()) return 0; + l_start = (unsigned long)page_address(page); + ret = split_linear_mapping(l_start, l_start + PAGE_SIZE); + if (WARN_ON_ONCE(ret)) + return ret; + return apply_to_page_range(&init_mm, - (unsigned long)page_address(page), - PAGE_SIZE, change_page_range, &data); + l_start, PAGE_SIZE, change_page_range, + &data); } int set_direct_map_default_noflush(struct page *page) { + unsigned long l_start; + int ret; + struct page_change_data data = { .set_mask = __pgprot(PTE_VALID | PTE_WRITE), .clear_mask = __pgprot(PTE_RDONLY), @@ -187,9 +343,14 @@ int set_direct_map_default_noflush(struct page *page) if (!can_set_direct_map()) return 0; + l_start = (unsigned long)page_address(page); + ret = split_linear_mapping(l_start, l_start + PAGE_SIZE); + if (WARN_ON_ONCE(ret)) + return ret; + return apply_to_page_range(&init_mm, - (unsigned long)page_address(page), - PAGE_SIZE, change_page_range, &data); + l_start, PAGE_SIZE, change_page_range, + &data); } #ifdef CONFIG_DEBUG_PAGEALLOC From patchwork Mon Nov 18 18:16:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 13878944 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CBCC6D591AC for ; Mon, 18 Nov 2024 18:21:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=CMy6CyGxIl2yyFuR4S4pWWa5p+rMXlhUtz651DwcqwQ=; b=zwyf0RhnDCVVhD05Dm5HtfGtf2 7McAtYeBZEPWsKsg/jNJGTv9kjJfU2zODvb101gJGAazkqjzG6YtJp+k6Ki+MRfsqZlgFGspD2hFo zy+zCN9La9scVaLDfsFvTUKtAmpv6EWjOQ/eu8JOt2zjsu6wM73hKi1isS9DfmHmX8jF/5M5TNxVS EA4S9Z7CAwDaHonfmgllYRc4lm9Ew7Wq1TJjbJbjbXpB8759piJyMEGMxrUouZxZ4mr8MQHw+mfD/ BrOCcRfUx2xRpjJS7Ll7/EpFUV6oB8zZFDcuusCbTUqswtmHfxu7CEdUsS/vLFJkag/D/JpR9NnK0 UVWZTuuw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tD6NP-0000000AKFl-07FQ; Mon, 18 Nov 2024 18:21:31 +0000 Received: from mail-mw2nam10on20715.outbound.protection.outlook.com ([2a01:111:f403:2412::715] helo=NAM10-MW2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tD6K0-0000000AJeq-00F4 for linux-arm-kernel@lists.infradead.org; Mon, 18 Nov 2024 18:18:01 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Hi7zeUyZqqupLF2Eo13vUeAYErRz/DDn8XpdwzPIKT//WjNAsXsL4qFo+x/xSZVY7cQuMENtOpwWQ1nOdSEYniAkox7NP+yKyW0SB+AFfH3BAZ4gQZ713nBNVg9Oq/kLfny6+rPj+a9ctCFBSC64AQckZH0V/9Wu0hnxchYHdu7lhGJWg0uJ+oCao3bO0x7blK9TNwAbEMhx4fElTaFR6zNCtqYywDsc9kG3lMfraZY5eb1CezmhfS/2V+k4LDTD6bBdrB9/35YvuaJUWmLZIzowHVpHZE+jxkKYd15nLjj7yuciyCSD1Anqb7bgvJkwFyCr97Lbpk8AlmRBfQ6m5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=CMy6CyGxIl2yyFuR4S4pWWa5p+rMXlhUtz651DwcqwQ=; b=U9TDf9H+JuKlyCMcCGFkitK8uqFAkNnqHQ+LC+8aD/PjfmjYXfMyfg/DIAv2TrTOaRUgYj2BwzJ9WoWVFGA0i68q0+RpaOFq+3umbTMkUETGj8jMIopN96FCGRrMshNBnbj8cOBqIlUBKX8ARZlNNb1OT3pUAOeycxYpC594aa3//paUCUd1cnNoFBBkx2SkUel20LMAIqGhsfW2cGqxUCLS86T0NDIbrKEzBvDKwujhavhbIi5U7VAcRQou14XU6PBuSHb00euNfOfy8gJNznd8eTRZ9Ohkz0v/WkF0uoD1WpCJe5zOmPBxGt8FyRNixZdiB5OQx1cT3Rs9+/hBJA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CMy6CyGxIl2yyFuR4S4pWWa5p+rMXlhUtz651DwcqwQ=; b=JYLYKSNz6fD0PyStP2uk0RcgebJJRAacERu9FX7RtjCpccxSgRDJSe3WFVGDaw9fRD/P14nMTu+2D/8I1Kl3CNbux4ckzLugJr8onIa1bTr8I5DqiSHf9hlE5oTDEargi9uEgM2cuHfSdqLl+JPydBK5X5AS9blXTNFX20i33tU= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by CO1PR01MB6694.prod.exchangelabs.com (2603:10b6:303:f5::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8182.13; Mon, 18 Nov 2024 18:17:41 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460%4]) with mapi id 15.20.8182.013; Mon, 18 Nov 2024 18:17:41 +0000 From: Yang Shi To: catalin.marinas@arm.com, will@kernel.org Cc: cl@gentwo.org, scott@os.amperecomputing.com, yang@os.amperecomputing.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/3] arm64: cpufeature: workaround AmpereOne FEAT_BBM level 2 Date: Mon, 18 Nov 2024 10:16:10 -0800 Message-ID: <20241118181711.962576-4-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20241118181711.962576-1-yang@os.amperecomputing.com> References: <20241118181711.962576-1-yang@os.amperecomputing.com> X-ClientProxiedBy: CH0PR13CA0038.namprd13.prod.outlook.com (2603:10b6:610:b2::13) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|CO1PR01MB6694:EE_ X-MS-Office365-Filtering-Correlation-Id: e78287f5-fcf0-4a65-e06a-08dd07fd4a99 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|52116014|376014|1800799024|366016|38350700014; X-Microsoft-Antispam-Message-Info: P1gMl1sy1BiyWADik0Ii2kIoNw3t4O8mH8WT49zKqcICZVBIEwH0cwFHAJQP2h3vU0lK4qQsplUdpyhuivL7MEWYOllIhG+6fI3ln97CFzi+b9S43EUEDQ+sluhk07xTExiNhif3Y/T65zYTGbHIlM1cKjcwBOSPgiBr+1gEirn2LTVqj57+Zc62TFGw0+2kMEfe1zb2LRDF8xCtnbhMlmDfv8m7Tm+OkHK3tlAL4Zzf4PalYxIjGDysqNWNhyJJV+K1GmaFshhWt9X4mvhVX1kVsqPlOaAd+bhAtqoPWWV7HfA7r8Qoqgovzj0tc1PBEEejBVnRzg4j4ZSVS3YmZZI7fe21DJfeO5+tllydJ/runcbBJN+svuGcZeaLkMpVrGHSPv7IAWUJ5pWjsfpoJLfOl3dpvB7LAHt+A2NLVA7XE5i0eTBxzmKKSal+pbwOVee3cueKrsRYBggDpLeW4o7I9XY/TaMsB/tzIsVpOwfMmr0fXThsx4NYXD1+117hJLixZ7J2nowGtyHZZS71O9C7CbkQw+fz0PgcbHQHmqrrY924PwVGZvxCBl+yO/4wLdJ7BbBiU8RUPbT+6v0jjDGLUWyOcMAGkxE2KsQnnm/DgiyNrb/nq1QOiu6oeyouRCjUzQZ7Qd+xR9DrNh0FpI1uDk3V/uR4Eisn1irAmR17MnFqk8mBwnAyEcoqZyMbPVkWaWYT6OgN6xoStLfFpQV8k0aagBQxn6wiC7bRJvW0uqx28daowZqMLtZqd3zVFRwWnu2hNYohK0HazgOMl5qKFYePznPGRppkq2K2Aq8SDxISsxlcBGkaA9wB4UH/Q5V4nrLBqcxsw31524O0kzrgaalLKxEKcG5CBhn1dRSJN5GBJIxFYf5Ftlrrrdmp0uv6qMlnDfCuRqMnJGW6jAEuuhCNq3lYjiF+lbnuTIFL58rMjaR1IKumC6OvE7HKQUyQME2iPFlgvr4Be+5vlBTXvki1dPIMV9g3iO6Yod4ZkRezh5ikVm2aRaCtlPAInTm9OmWvqAaCs8r+GVC/OU9Pz0i2DhE1g8t3iiqE2BjEZgn0GMFzI24dBOTEjx2GR6DOmxPb0gLexH5X3JtXaubkj1XpN1OdeLY0eS9yQLuNGMZC5F7yA07/A1iRTavugnK+3j5QEmHxZM8ZEoT3phzSFQVucGf5vYSRCG1pcKepzOJZfJ126ipKD7eOiUgLLv4h6gRE2befFCrgQN/dv7MHz0MWAiZtruK2TmykHfqm1oxnE5FvVYuqX0veygaDOX1rBxzkwAGLVUlMh2CvmxGgQvw2d/h8/cqV+YLDsTkL8a8lQup/5iUwh3h4/QVn/VPUoonNIwJIqRN5nxVAsYjXktVsuilSunPs2FgmxjQ= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(52116014)(376014)(1800799024)(366016)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 9X/xPXVQ9qv6qgJ2Vp/i1qBP3x9nIfwPp+GeKpsyxHe8+/lUHTYc4g77+hnpUjm+sFyw8jleT1pgL7k5IHFaRQt0PL25yjdt0adD5y3BEr+BUvaf4tOVvTwglPGB5jO4hNGNl310wIt+WsdNdPdniLGfsCDtWVJu26Xely2eya3YySnQBIAyIPcxisaVSRQAKu8ym8JO2GdxMURpjqdJVsTvCh8qauO1uhdD3C+6cOS8blufHCmCaNypCNST8YtWoDPt9WeTwUDvXb9eaaZLmprmPBr41jtQOSN8ZkOTvQ93hoXWETv0+b/ofbmFZtvwNnXrXw+JwciKcz6KJPM6p2CR/kxXebHOIZUUkfaLRKd8dwehDf6i2YZnx0Sju7wuABupPi5kwC6PgzmviGY8zPVXb5HxYSWLHpIH1hxYrrbghFHBVWvcTYUgr/O/RECFRDbtdGB7RczQytRIFZvjVtpz9pdKeMgahryRpe8MXhfK3zgkNvEr6AaFfQhFfl8sN1EXnAB+RYPeqmn237QnvOuAz5QxbgAKqnuA2flcr5xP2062uX4oF++l0Ji98iFuQ7jV5gCAYvbfLMbOgNhPuNVfZF2aYtqVUI3f09ShqKqTMiRJ/SA6tny0RZScQPIFib5/vjrHgBZL/GW3QDZGQm+gGn25JLhdAd4MuQDebTuuIK6WWLDYkoEkV+nr7Z7sp6Rt2JDBhacmGs95OpihHXFYy7F+sMdMi8Mbhwczk1ooYP5OH9qg3O/V8/P0OZ5V4LTU+As/WHXdv0eQOOKN624C5PpNXaLRI9Y3zltUJ48i8EIrd0yi7likBZJLyYnIXJazKl9ap90XHMgz+N2zb/Wy4Y8/RtQeRFVR2rl941tx/6zCTRe5Kz43txfWW1CacrjRwMs5h2Rpy5bqfPMEm86cc3f/MzT74eVnFaPn2JviPGp0dbfAZbkqX9jGVArJohMKhwQ1xV/+bsttEcDtevDjLVoFFvDOi+WJLRXyKGGP1GGZqJ4Gvg6fD5ebH3F7XBKORcfz/jR3n1YrH6mPgmLXyZobXQAd0m2cQm1n1FBIcH1m1hx5D0ZCrGa9PbCEoT9FRJXHHQFuyBwzkMVwiOHwmmQemB+l5bYDbddK+Ppj8NtG12b8ecczagGwfqSx5H7e/fUH7oGo6iduz0B7wPgfXOUaZV5mxenNgltqCfiI8bUvNojIyTFfhsCpkQg4B+PqLTUNEAeMfpUKWStBKgKH1NYvd9RF2K6vngaQaPSS5XkZ5lbXGOFboiHSZ5EnNFbPPyfrQC1nmZ9aNj2hOpiMhFOeu52NZjyO+VIZULwWFYle5u5KSqc60OvDXBgJHQnG5T309UVnz+aOdoKarxziMzU7mIUMytetmpH6MaOFNx9g/y7MNlVoHUx/IzN1UhBZC7evlSz59qUKYJ46znExdE+WmpH8QOiTCQzMxDEP5SOfLTA9SzQnDsDbg0lkFmZq5624bDnnDv7VGcKVU132uZFIwWPptC7vv222kNb0V37VBBv3FJj/tdAWXfUKiWMWVVVnr/YRlw71dtWyyWCJ7nat4sZ4hrJgCuyJ6SRTGo9p6wy0Z8iKuyeV3Bcn0IHzkdJZnMCSgSTCviB+VnTp85lfpY5tnXhCYqEdMHE= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: e78287f5-fcf0-4a65-e06a-08dd07fd4a99 X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Nov 2024 18:17:41.7273 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7hQsEjbUiLQHeNv3BEQsI7d+IbcUReWIfRKKcg4RKdI4P36VUchiI/6dUAiCSWixfE8rKzo/TKdeuHU81wVIXcSuP8aFRqezXzFTiB8bT04= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO1PR01MB6694 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241118_101800_036015_5D09970F X-CRM114-Status: GOOD ( 10.20 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org FEAT_BBM level 2 is not advertised on AmpereOne because of a bug when collapsing stage 2 mappings from smaller to larger translations. That doesn't impact splitting stage 1 mappings (whether stage 2 is enabled or not), so workaround it by detecting CPUID. Signed-off-by: Yang Shi Reviewed-by: Christoph Lameter --- arch/arm64/include/asm/cpufeature.h | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index c7ca5f9f88bb..d9b20eb43d31 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -847,10 +847,19 @@ static inline bool bbmlv2_available(void) { u64 mmfr2; u32 bbm; + static const struct midr_range ampereone[] = { + MIDR_ALL_VERSIONS(MIDR_AMPERE1), + MIDR_ALL_VERSIONS(MIDR_AMPERE1A), + {} + }; mmfr2 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR2_EL1); bbm = cpuid_feature_extract_unsigned_field(mmfr2, ID_AA64MMFR2_EL1_BBM_SHIFT); - return bbm == ID_AA64MMFR2_EL1_BBM_2; + if ((bbm == ID_AA64MMFR2_EL1_BBM_2) || + is_midr_in_range_list(read_cpuid_id(), ampereone)) + return true; + + return false; } int do_emulate_mrs(struct pt_regs *regs, u32 sys_reg, u32 rt);