From patchwork Tue Sep 12 14:16:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381803 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 89BB4CA0EEC for ; Tue, 12 Sep 2023 14:20:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=rB4qKVxW0HUr7z4abZoNLUtg/Jzt0iYuZQEdccXahEc=; b=fo/VRRkLdHOyR7Bl0XXFDOJtdq db/MpOTRjqRJPRa7NhN64ar4WTE5APC/Z40Us0S1NLIbC0zXX5G1qzsS1byjdlko+lhmzPPRqxIrf pZ4OBHwvIR5G5ITWldVql1lGjdeXFFmHkg4siGSFmvREtwMAMOy/wk+yEulUljmp/hSjWUa0ygvj0 2HVQuwHxGHz3e786/mth2XPAQk+jCJOUyi5QaV6XD5XZX7hld80i2fP+G44jiKGWBD7AXCfh06N7G 5n4opkwDokvCzT1O4AFnbJ3Dj1E+yLmaR9b2MTRjPKlnSbnUucwJjV5IxAAMgRC1IMRHAKZvWbRgF rIj1XEBg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4FX-003YBm-37; Tue, 12 Sep 2023 14:20:20 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4EK-003X9j-2z for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:15 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-d802ecb5883so3748470276.3 for ; Tue, 12 Sep 2023 07:19:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528343; x=1695133143; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7g3rQc5ohntznFdEpK61zpmbym33bi0tjP5Qy9B6T6w=; b=BTyhEZLPABJy3ov636XWkCV2z+1cLRNfaGcrcaoJYnuArIDdHLCdzk9yPqDeXCpFJ/ k2xfNkuK/W1ZawJHPlkphfX8aN3wYNaCMAHb6YMKILhELAOl3zo312VfEHe65DjXg0LA BLNYgm/2nKKhbHgj0sEPRIpmYNxTZrErvCExxQHzb6n5BZdXrF0VpFoQqiirT2jvff9v 2AfC9ww4hLT+iAzE3xuHo5gBO5htLCfYAEyl6faCKOh6EfZrCOqNS2WBX+Y6LYeyJWQT fx9BBjm1QBx2i5HrsRA9KsdAACjTXyzaOkxsg1tr1CfzBdLqEyYykMzeQbDIvSYxVFol ZO3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528343; x=1695133143; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7g3rQc5ohntznFdEpK61zpmbym33bi0tjP5Qy9B6T6w=; b=njkYYw4XDDZaf4BR/ze9pOkECPXJ97AI2jQWchRmwEVc2oIsm+60B/TqT3uBMliHX1 BpMyXQPqGm9M17Li7FJAdKS0YOpd656afHo4JTyMU2PLj8JnszbUXkFKsakMqdjN1fvP crnzWe5iO78OjWpQT8ytXXB991bPPhM3/3KjnbKKwkoh+bo6gEami6wK+EBuqjRv0YLP uLQXNfEMrAmHC5czhnLcvTvIiEU5jq3ibyd4hRGIWy8c7oaJ6ddepfz3u1Ol4151Y6Q2 kER5uSAdrkm0T+vklJois1vbVwTnj6W2um3ofqvgcwzX+R28Mg+DcGYLxRvHxVVOs7Gn FPMw== X-Gm-Message-State: AOJu0Yxeg5u4gxgUIpMUXQS546EC0eVZFLyeSSi/HHjKgPEVh84NEab3 SFWXTXjCp8gp3fqT50r3vI4BMe8GlfS8WTLkmSAYD0NCPHhOkpHG/Z2feO1o4G1yYoAm/jyXd1B +4x8f4C0hduOpnWhXidSLxCMEvfKv+vvaN9SwvWivCen4igTSvV9GIaRcDRNDksZuR886VvPH3z Y= X-Google-Smtp-Source: AGHT+IFfwkCsrpwwd99WKR7AW03+TpYam0LEMMgkUE98Z2CPsVrxOKylh4DfoGi+4q+jP/9x+i4GAm5g X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:adc3:0:b0:d77:f7c3:37db with SMTP id d3-20020a25adc3000000b00d77f7c337dbmr302647ybe.8.1694528342614; Tue, 12 Sep 2023 07:19:02 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:34 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6960; i=ardb@kernel.org; h=from:subject; bh=2dJHCIiitpBTpEMWV5ZMZiGNeeCX6/OF7FQfNtGJZlY=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6MiNa0es+BmvHtFul/fM8ZzNVDlt7SXdl8Yvply1W FC50JS9o5SFQYyDQVZMkUVg9t93O09PlKp1niULM4eVCWQIAxenAEzk33SGP9wN4QxFrTYvKqJn WjveeTRlp2QGs8OutMNc66amvPu3YjYjw4lzYiKnhU2WFD+NfmLY8F2hdHc4Z/oby2fL3FIXXnL zYAEA X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-107-ardb@google.com> Subject: [PATCH v4 44/61] arm64: mm: Add feature override support for LVA From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071905_000869_7334F874 X-CRM114-Status: GOOD ( 23.20 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add support for overriding the VARange field of the MMFR2 CPU ID register. This permits the associated LVA feature to be overridden early enough for the boot code that creates the kernel mapping to take it into account. Given that LPA2 implies LVA, disabling the latter should disable the former as well. So override the ID_AA64MMFR0.TGran field of the current page size as well if it advertises support for 52-bit addressing. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 17 ++++++----- arch/arm64/include/asm/cpufeature.h | 4 +++ arch/arm64/kernel/cpufeature.c | 8 +++-- arch/arm64/kernel/image-vars.h | 2 ++ arch/arm64/kernel/pi/idreg-override.c | 31 ++++++++++++++++++++ 5 files changed, 53 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index beb53bbd8c19..0710c17800a4 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -576,18 +576,21 @@ alternative_endif .endm /* - * Offset ttbr1 to allow for 48-bit kernel VAs set with 52-bit PTRS_PER_PGD. + * If the kernel is built for 52-bit virtual addressing but the hardware only + * supports 48 bits, we cannot program the pgdir address into TTBR1 directly, + * but we have to add an offset so that the TTBR1 address corresponds with the + * pgdir entry that covers the lowest 48-bit addressable VA. + * * orr is used as it can cover the immediate value (and is idempotent). - * In future this may be nop'ed out when dealing with 52-bit kernel VAs. * ttbr: Value of ttbr to set, modified. */ .macro offset_ttbr1, ttbr, tmp #ifdef CONFIG_ARM64_VA_BITS_52 - mrs_s \tmp, SYS_ID_AA64MMFR2_EL1 - and \tmp, \tmp, #(0xf << ID_AA64MMFR2_EL1_VARange_SHIFT) - cbnz \tmp, .Lskipoffs_\@ - orr \ttbr, \ttbr, #TTBR1_BADDR_4852_OFFSET -.Lskipoffs_\@ : + mrs \tmp, tcr_el1 + and \tmp, \tmp, #TCR_T1SZ_MASK + cmp \tmp, #TCR_T1SZ(VA_BITS_MIN) + orr \tmp, \ttbr, #TTBR1_BADDR_4852_OFFSET + csel \ttbr, \tmp, \ttbr, eq #endif .endm diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 4ffbd08f6b77..351e0b39f0bb 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -912,7 +912,9 @@ static inline unsigned int get_vmid_bits(u64 mmfr1) s64 arm64_ftr_safe_value(const struct arm64_ftr_bits *ftrp, s64 new, s64 cur); struct arm64_ftr_reg *get_arm64_ftr_reg(u32 sys_id); +extern struct arm64_ftr_override id_aa64mmfr0_override; extern struct arm64_ftr_override id_aa64mmfr1_override; +extern struct arm64_ftr_override id_aa64mmfr2_override; extern struct arm64_ftr_override id_aa64pfr0_override; extern struct arm64_ftr_override id_aa64pfr1_override; extern struct arm64_ftr_override id_aa64zfr0_override; @@ -982,6 +984,8 @@ static inline bool cpu_has_lva(void) u64 mmfr2; mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); + mmfr2 &= ~id_aa64mmfr2_override.mask; + mmfr2 |= id_aa64mmfr2_override.val; return cpuid_feature_extract_unsigned_field(mmfr2, ID_AA64MMFR2_EL1_VARange_SHIFT); } diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 3778a7384c62..a2c320ed36b3 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -664,7 +664,9 @@ static const struct arm64_ftr_bits ftr_raz[] = { #define ARM64_FTR_REG(id, table) \ __ARM64_FTR_REG_OVERRIDE(#id, id, table, &no_override) +struct arm64_ftr_override id_aa64mmfr0_override; struct arm64_ftr_override id_aa64mmfr1_override; +struct arm64_ftr_override id_aa64mmfr2_override; struct arm64_ftr_override id_aa64pfr0_override; struct arm64_ftr_override id_aa64pfr1_override; struct arm64_ftr_override id_aa64zfr0_override; @@ -728,10 +730,12 @@ static const struct __ftr_reg_entry { &id_aa64isar2_override), /* Op1 = 0, CRn = 0, CRm = 7 */ - ARM64_FTR_REG(SYS_ID_AA64MMFR0_EL1, ftr_id_aa64mmfr0), + ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR0_EL1, ftr_id_aa64mmfr0, + &id_aa64mmfr0_override), ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR1_EL1, ftr_id_aa64mmfr1, &id_aa64mmfr1_override), - ARM64_FTR_REG(SYS_ID_AA64MMFR2_EL1, ftr_id_aa64mmfr2), + ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR2_EL1, ftr_id_aa64mmfr2, + &id_aa64mmfr2_override), ARM64_FTR_REG(SYS_ID_AA64MMFR3_EL1, ftr_id_aa64mmfr3), /* Op1 = 0, CRn = 1, CRm = 2 */ diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index d312f4176a2c..c5eeb33a6280 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -36,7 +36,9 @@ PROVIDE(__pi___memset = __pi_memset); PROVIDE(__pi_id_aa64isar1_override = id_aa64isar1_override); PROVIDE(__pi_id_aa64isar2_override = id_aa64isar2_override); +PROVIDE(__pi_id_aa64mmfr0_override = id_aa64mmfr0_override); PROVIDE(__pi_id_aa64mmfr1_override = id_aa64mmfr1_override); +PROVIDE(__pi_id_aa64mmfr2_override = id_aa64mmfr2_override); PROVIDE(__pi_id_aa64pfr0_override = id_aa64pfr0_override); PROVIDE(__pi_id_aa64pfr1_override = id_aa64pfr1_override); PROVIDE(__pi_id_aa64smfr0_override = id_aa64smfr0_override); diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index 5857e4efad59..062aeae68936 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -63,6 +63,35 @@ static const struct ftr_set_desc mmfr1 __prel64_initconst = { }, }; + +static bool __init mmfr2_varange_filter(u64 val) +{ + int __maybe_unused feat; + + if (val) + return false; + +#ifdef CONFIG_ARM64_LPA2 + feat = cpuid_feature_extract_signed_field(read_sysreg(id_aa64mmfr0_el1), + ID_AA64MMFR0_EL1_TGRAN_SHIFT); + if (feat >= ID_AA64MMFR0_EL1_TGRAN_LPA2) { + id_aa64mmfr0_override.val |= + (ID_AA64MMFR0_EL1_TGRAN_LPA2 - 1) << ID_AA64MMFR0_EL1_TGRAN_SHIFT; + id_aa64mmfr0_override.mask |= 0xfU << ID_AA64MMFR0_EL1_TGRAN_SHIFT; + } +#endif + return true; +} + +static const struct ftr_set_desc mmfr2 __prel64_initconst = { + .name = "id_aa64mmfr2", + .override = &id_aa64mmfr2_override, + .fields = { + FIELD("varange", ID_AA64MMFR2_EL1_VARange_SHIFT, mmfr2_varange_filter), + {} + }, +}; + static bool __init pfr0_sve_filter(u64 val) { /* @@ -173,6 +202,7 @@ static const union { prel64_t reg_prel; } regs[] __prel64_initconst = { { .reg = &mmfr1 }, + { .reg = &mmfr2 }, { .reg = &pfr0 }, { .reg = &pfr1 }, { .reg = &isar1 }, @@ -198,6 +228,7 @@ static const struct { { "arm64.nomte", "id_aa64pfr1.mte=0" }, { "nokaslr", "arm64_sw.nokaslr=1" }, { "rodata=off", "arm64_sw.rodataoff=1" }, + { "arm64.nolva", "id_aa64mmfr2.varange=0" }, }; static int __init parse_hexdigit(const char *p, u64 *v)