From patchwork Tue Nov 15 14:38:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13043775 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B1B19C433FE for ; Tue, 15 Nov 2022 14:40:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=kgQTI4eKBrgJ9ormddnF+8iANVyMJYmPRn3QUF0qSDs=; b=2SPOkyG3DbslGm IBjl9cHETM7Q/CZKNoAAyXDov5e9MGh8JBarDg+cbuWCvI5+YG1bbekUT/lwVl11DW7FOqcmXVyGr LDKGtxlKqBJimKkd9/GudbWlKxbHn1T7kQVl5Ls/otpR3abQqf5wdJ80j1octQf3PCG3XtNKvFPMA SX5qhKyHolGcBj7WEeEk47/4znfm00vhwXEjxLseZWoTKCDwip1klWfJAMiSy2Rwf2CyBuw1DtWkW EPvScDNdWCwR02WYAWt431xEh7zyzYpFuNp5zz8HD5OIc//u8jm++JwBTzdGdXI15HD4msAr7/1c4 QsGF5yRsmglUIGe7wqCw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oux5j-00Bx8B-J6; Tue, 15 Nov 2022 14:39:11 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oux5F-00BwlB-Gi for linux-arm-kernel@lists.infradead.org; Tue, 15 Nov 2022 14:38:43 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 35693B81988; Tue, 15 Nov 2022 14:38:40 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6198AC4347C; Tue, 15 Nov 2022 14:38:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668523118; bh=LRkGMQibuVisbNEjQKl+MFAp64QlTTGv+gkVXHH91i0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=utPS8U/1aSmBZ1yWtIIntVhggVwjigrx79cWm++clzs1c/7q0P4D3Z/wwrkPcJBsN AoLS3uP33DsJqexb9ftz9iO+g0MZBZ0O1XJ+OfX2df0y1zrY2fIafYCnvaB+LD7Km0 A56tee1q17t1/wvyQU3lG4VkASi1/eLeTaW+GZJ4TyMeWEZYo99nd4seh3h/dC6Y0v sH5n4aYq+cSY29qaMVuTo05CqJe0CAv87uax69w7gj9NZCRu7YBdeZC1dzDyJMdHeL PspMmxYShsUeengzUWrr4v2RLZVZ+KQzCQsYYofWGvstwbydFpJcy0MvY+1HdY/DV1 ayb5yqzZSJt6Q== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Mark Rutland , Anshuman Khandual , Joey Gouly Subject: [PATCH 3/3] arm64: mm: Add feature override support for LVA and E0PD Date: Tue, 15 Nov 2022 15:38:24 +0100 Message-Id: <20221115143824.2798908-4-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221115143824.2798908-1-ardb@kernel.org> References: <20221115143824.2798908-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6918; i=ardb@kernel.org; h=from:subject; bh=LRkGMQibuVisbNEjQKl+MFAp64QlTTGv+gkVXHH91i0=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjc6RfAl6LYlgVNAQOTY+eeEpFtd80ifMgO06tuG9j d8j38aiJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY3OkXwAKCRDDTyI5ktmPJAXQC/ 40jKtNc9/881TdpPnyOqZVELOAvC1KDXW0AjVPXVkjsW5oMv9R0QTq+99doyViUJygJJnt3DoFm1sD GtuGeimc2+3UXsYVGmS5hi1E9d2YRHEYmL+MoXmGKtLfIwiRuJx8a2rfgxamCzs+SunUNlSaGV0U7/ wDke2wsTWSZDtBIPhmHcQDnFxGWgbxWyKB2CLcKqW0APwMZnWIhBCeU+h6jaiIf/+rd8R8dOxfPicY SvnOErgAIXglJZl6W8qXtUDc8NgDVOyBO0Ov8ylTK9U+r32FQ9HXhHa0v90xrD0H/+KWV9NG1JCMDY vdzSHsOuAGrfo2HHgnRUZ3RveX7ZQAMYlyyjkItT4a5Cq0K6olfYYBcegr+g38pe018Jy6VIqLGLa9 hj7A6V8Z9gLagYVBnEbDEFT62oGXqBcZ7jVCn+uK/lBZfIO8Nvx9hX0nPUItIul5K/AEb128BN1/+w 2JV0JQ5EMFWBDO3kroMI8NNSF7cLo8VuUHuF9rW87b92c= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221115_063841_916893_3C45CBB2 X-CRM114-Status: GOOD ( 22.92 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add support for overriding the VARange and E0PD fields of the MMFR2 CPU ID register. This permits the associated features to be overridden early enough for the boot code that creates the kernel mapping to take it into account. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 17 ++++++++++------- arch/arm64/include/asm/cpufeature.h | 1 + arch/arm64/kernel/cpufeature.c | 6 +++++- arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kernel/pi/idreg-override.c | 8 +++++++- arch/arm64/kernel/pi/map_kernel.c | 4 ++++ 6 files changed, 28 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index e5957a53be3983ac..941082cfb788151a 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -604,18 +604,21 @@ alternative_endif .endm /* - * Offset ttbr1 to allow for 48-bit kernel VAs set with 52-bit PTRS_PER_PGD. + * If the kernel is built for 52-bit virtual addressing but the hardware only + * supports 48 bits, we cannot program the pgdir address into TTBR1 directly, + * but we have to add an offset so that the TTBR1 address corresponds with the + * pgdir entry that covers the lowest 48-bit addressable VA. + * * orr is used as it can cover the immediate value (and is idempotent). - * In future this may be nop'ed out when dealing with 52-bit kernel VAs. * ttbr: Value of ttbr to set, modified. */ .macro offset_ttbr1, ttbr, tmp #ifdef CONFIG_ARM64_VA_BITS_52 - mrs_s \tmp, SYS_ID_AA64MMFR2_EL1 - and \tmp, \tmp, #(0xf << ID_AA64MMFR2_EL1_VARange_SHIFT) - cbnz \tmp, .Lskipoffs_\@ - orr \ttbr, \ttbr, #TTBR1_BADDR_4852_OFFSET -.Lskipoffs_\@ : + mrs \tmp, tcr_el1 + and \tmp, \tmp, #TCR_T1SZ_MASK + cmp \tmp, #TCR_T1SZ(VA_BITS_MIN) + orr \tmp, \ttbr, #TTBR1_BADDR_4852_OFFSET + csel \ttbr, \tmp, \ttbr, eq #endif .endm diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 4b5c639a5a0a7fab..7aa9cd4fc67f7c61 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -911,6 +911,7 @@ static inline unsigned int get_vmid_bits(u64 mmfr1) struct arm64_ftr_reg *get_arm64_ftr_reg(u32 sys_id); extern struct arm64_ftr_override id_aa64mmfr1_override; +extern struct arm64_ftr_override id_aa64mmfr2_override; extern struct arm64_ftr_override id_aa64pfr0_override; extern struct arm64_ftr_override id_aa64pfr1_override; extern struct arm64_ftr_override id_aa64zfr0_override; diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index b44aece5024c3e2d..469d8b31487e88b6 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -637,6 +637,7 @@ static const struct arm64_ftr_bits ftr_raz[] = { __ARM64_FTR_REG_OVERRIDE(#id, id, table, &no_override) struct arm64_ftr_override id_aa64mmfr1_override; +struct arm64_ftr_override id_aa64mmfr2_override; struct arm64_ftr_override id_aa64pfr0_override; struct arm64_ftr_override id_aa64pfr1_override; struct arm64_ftr_override id_aa64zfr0_override; @@ -703,7 +704,8 @@ static const struct __ftr_reg_entry { ARM64_FTR_REG(SYS_ID_AA64MMFR0_EL1, ftr_id_aa64mmfr0), ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR1_EL1, ftr_id_aa64mmfr1, &id_aa64mmfr1_override), - ARM64_FTR_REG(SYS_ID_AA64MMFR2_EL1, ftr_id_aa64mmfr2), + ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR2_EL1, ftr_id_aa64mmfr2, + &id_aa64mmfr2_override), /* Op1 = 0, CRn = 1, CRm = 2 */ ARM64_FTR_REG(SYS_ZCR_EL1, ftr_zcr), @@ -1605,6 +1607,8 @@ bool kaslr_requires_kpti(void) */ if (IS_ENABLED(CONFIG_ARM64_E0PD)) { u64 mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); + mmfr2 &= ~id_aa64mmfr2_override.mask; + mmfr2 |= id_aa64mmfr2_override.val; if (cpuid_feature_extract_unsigned_field(mmfr2, ID_AA64MMFR2_EL1_E0PD_SHIFT)) return false; diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 5bd878f414d85366..6626f95f7ead0682 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -46,6 +46,7 @@ PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); PROVIDE(__pi_id_aa64isar1_override = id_aa64isar1_override); PROVIDE(__pi_id_aa64isar2_override = id_aa64isar2_override); PROVIDE(__pi_id_aa64mmfr1_override = id_aa64mmfr1_override); +PROVIDE(__pi_id_aa64mmfr2_override = id_aa64mmfr2_override); PROVIDE(__pi_id_aa64pfr0_override = id_aa64pfr0_override); PROVIDE(__pi_id_aa64pfr1_override = id_aa64pfr1_override); PROVIDE(__pi_id_aa64smfr0_override = id_aa64smfr0_override); diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index 662c3d21e150e7f9..3be2f887e6cae29f 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -139,12 +139,17 @@ DEFINE_OVERRIDE(6, sw_features, "arm64_sw", arm64_sw_feature_override, FIELD("nowxn", ARM64_SW_FEATURE_OVERRIDE_NOWXN), {}); +DEFINE_OVERRIDE(7, mmfr2, "id_aa64mmfr2", id_aa64mmfr2_override, + FIELD("varange", ID_AA64MMFR2_EL1_VARange_SHIFT), + FIELD("e0pd", ID_AA64MMFR2_EL1_E0PD_SHIFT), + {}); + /* * regs[] is populated by R_AARCH64_PREL32 directives invisible to the compiler * so it cannot be static or const, or the compiler might try to use constant * propagation on the values. */ -asmlinkage s32 regs[7] __initdata = { [0 ... ARRAY_SIZE(regs) - 1] = S32_MAX }; +asmlinkage s32 regs[8] __initdata = { [0 ... ARRAY_SIZE(regs) - 1] = S32_MAX }; static struct arm64_ftr_override * __init reg_override(int i) { @@ -170,6 +175,7 @@ static const struct { { "nokaslr", "arm64_sw.nokaslr=1" }, { "rodata=off", "arm64_sw.rodataoff=1 arm64_sw.nowxn=1" }, { "arm64.nowxn", "arm64_sw.nowxn=1" }, + { "arm64.nolva", "id_aa64mmfr2.varange=0" }, }; static int __init find_field(const char *cmdline, char *opt, int len, diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index 3504e3266b02f636..c3edd207e3c031a2 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -118,6 +118,8 @@ static bool __init arm64_early_this_cpu_has_e0pd(void) return false; mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); + mmfr2 &= ~id_aa64mmfr2_override.mask; + mmfr2 |= id_aa64mmfr2_override.val; return cpuid_feature_extract_unsigned_field(mmfr2, ID_AA64MMFR2_EL1_E0PD_SHIFT); } @@ -127,6 +129,8 @@ static bool __init arm64_early_this_cpu_has_lva(void) u64 mmfr2; mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); + mmfr2 &= ~id_aa64mmfr2_override.mask; + mmfr2 |= id_aa64mmfr2_override.val; return cpuid_feature_extract_unsigned_field(mmfr2, ID_AA64MMFR2_EL1_VARange_SHIFT); }