From patchwork Tue Jan 23 14:53:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527730 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CF1F3C47258 for ; Tue, 23 Jan 2024 16:01:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=KOLhNhWsOwLdz4JC9VVQf7eA+OLgJjhX6smgtOuhlMo=; b=UStK71UucTAgw2kVtfsOJAityg mSlc1th/0EW6f6sFeSvwUvkNkoWYWI2fXuzFSvkPGx7DWsfO2WrHrlfbwGrMsp95URcbk/tVJL4ja AXBDMkhJAzLFSPGHnp2QJ3kEIdHFUzuz+9bCdq/pfB9pzxIqTXya8or4QAZPEGEFrkTF5Zh6wM4mt 0JI1xeDFk59/6MZQfjAVDSdu35NzqYlXYlP7x65UPiL408GfYUNJhXwt/wRUK581OROG43gg2vs5u jfHHmJuW9a+2xnkoFKEGym/U6qiMVrLmkHHd58WyTbVWPyusRqEG10MXKmRvd67rmI4moccqCjIvn sYlljoUw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSJCV-00HAuX-2D; Tue, 23 Jan 2024 16:00:35 +0000 Received: from mail-yw1-f201.google.com ([209.85.128.201]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBi-00GuGq-0N for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:58 +0000 Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5ff7eabfd94so64238087b3.3 for ; Tue, 23 Jan 2024 06:55:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021738; x=1706626538; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OgqzMMo0YQZO+a6NJtgIIV7c1m+h9plq6R7e39zthig=; b=dBbSNJCzupociiGOuUTvpK71PpCCL04H8mPa36Qx3ej/r9Psd2JhJfkXCViH0SdCRV /6HeoYqlWevValE56UwzgWsVQlIMue900hHy0B/BnEyePYVahuDF7S76pPe3sFmkq4WZ IKJtdtCl67U5C1w3VZ8q8uYyhnVVhf24hs0QUgrEcrHIHwRgH3OL5B81YTJwlPXIXVdE WqPe0SAnhx2Osyn1pNCrf3GF7Bm3Z9FmdhpdSddWsMvXOHqeoXLWUCuDZKGhZ+/E93jo Z6+O9z9AyWnJpLY3K9peCghB6+Vx9bQdenbUTXGJC8gMTYX7UtYDAksia1m7fon3nwNG +oFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021738; x=1706626538; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OgqzMMo0YQZO+a6NJtgIIV7c1m+h9plq6R7e39zthig=; b=NVmHd7GYM9hvqZf9ssDJQtb3f6o3n1Bz1kDlh3mDERdgnMhUSQJCc/slu1RlSD4Aln gV5J5LzSzZdZ7QTZQiW83qyCzZhyfAIqYILM3tMw7r6KhZC8woH8dQmQ0H3o5b7R0eaR UWOkjOVqYuS+wyZcz4XXBeFW4iRssY7wnE5hbBKTQw4uttgr/ONCi1b3xHPYfomBOSKH Fio74LBN+goAefpc/2c1QTAF7XGjux150Nv47vHCpLgWgA+zlp/Y+yGcI3goyjG2QtPo mlMtrDnFOh1TGUCj4H9XkKGnT5NmMQPM5EcqpFYJ+V0I4QvtZX3oY5oQGYfAgAODNovN mcGQ== X-Gm-Message-State: AOJu0YzFHAS1BIT7bK0diistyhC5v3skxIfwlC+EGJQQ5jYqYuKkZ7Ke yOhTb/z+MfjHnccm6EX6I8oGuMp82x68Y7lpKg8+hrZB6pEecBsySZ7fEBE3mE+8hICvXKwQ6ub Tlu4ntIDuCKT/kAW7FPn002DWQ99uoI4E01aAtmlMwCz/SR/KP8aJP2o1exAVLkDOOzqUI1dbcW Yikgiup3XrfjpG3OXapwBtqXPux2pjUkJrERDgZBnu X-Google-Smtp-Source: AGHT+IG6EJZj1nGwAxXWz6GUceBz/Z7FoJqwShrluVL6GXzOOc9Qg9WNISK/HPhrNa0pbPfw8tUPGk8d X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a81:5709:0:b0:5f8:bd56:176f with SMTP id l9-20020a815709000000b005f8bd56176fmr2778872ywb.2.1706021738410; Tue, 23 Jan 2024 06:55:38 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:32 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=6997; i=ardb@kernel.org; h=from:subject; bh=13TyWlMjR5z8U4YiJso59gvqkGS2/GpmFRI04ZiVl+M=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pXdV1w/f/7nydKd5//LtF0/P+nJC3iLITdZ/8o300 C3FmXf+dZSyMIhxMMiKKbIIzP77bufpiVK1zrNkYeawMoEMYeDiFICJaOQwMkyo+f974ke17LNJ aZ8XCvuzSS5zaJaResrB6jmTuV2rPpbhn51OSPmeHLuExJvJSyVXeRaf2PTmYOrlL/v5HlzzmPU njQEA X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-85-ardb+git@google.com> Subject: [PATCH v7 33/50] arm64: mm: Add feature override support for LVA From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_065542_188997_4A1DCEF8 X-CRM114-Status: GOOD ( 22.07 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Add support for overriding the VARange field of the MMFR2 CPU ID register. This permits the associated LVA feature to be overridden early enough for the boot code that creates the kernel mapping to take it into account. Given that LPA2 implies LVA, disabling the latter should disable the former as well. So override the ID_AA64MMFR0.TGran field of the current page size as well if it advertises support for 52-bit addressing. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 17 ++++++----- arch/arm64/include/asm/cpufeature.h | 4 +++ arch/arm64/kernel/cpufeature.c | 8 +++-- arch/arm64/kernel/image-vars.h | 2 ++ arch/arm64/kernel/pi/idreg-override.c | 31 ++++++++++++++++++++ 5 files changed, 53 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 6a467c694039..68a99b116256 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -576,18 +576,21 @@ alternative_endif .endm /* - * Offset ttbr1 to allow for 48-bit kernel VAs set with 52-bit PTRS_PER_PGD. + * If the kernel is built for 52-bit virtual addressing but the hardware only + * supports 48 bits, we cannot program the pgdir address into TTBR1 directly, + * but we have to add an offset so that the TTBR1 address corresponds with the + * pgdir entry that covers the lowest 48-bit addressable VA. + * * orr is used as it can cover the immediate value (and is idempotent). - * In future this may be nop'ed out when dealing with 52-bit kernel VAs. * ttbr: Value of ttbr to set, modified. */ .macro offset_ttbr1, ttbr, tmp #ifdef CONFIG_ARM64_VA_BITS_52 - mrs_s \tmp, SYS_ID_AA64MMFR2_EL1 - and \tmp, \tmp, #(0xf << ID_AA64MMFR2_EL1_VARange_SHIFT) - cbnz \tmp, .Lskipoffs_\@ - orr \ttbr, \ttbr, #TTBR1_BADDR_4852_OFFSET -.Lskipoffs_\@ : + mrs \tmp, tcr_el1 + and \tmp, \tmp, #TCR_T1SZ_MASK + cmp \tmp, #TCR_T1SZ(VA_BITS_MIN) + orr \tmp, \ttbr, #TTBR1_BADDR_4852_OFFSET + csel \ttbr, \tmp, \ttbr, eq #endif .endm diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 4f4dc5496ee3..a2ac31aecdd9 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -906,7 +906,9 @@ static inline unsigned int get_vmid_bits(u64 mmfr1) s64 arm64_ftr_safe_value(const struct arm64_ftr_bits *ftrp, s64 new, s64 cur); struct arm64_ftr_reg *get_arm64_ftr_reg(u32 sys_id); +extern struct arm64_ftr_override id_aa64mmfr0_override; extern struct arm64_ftr_override id_aa64mmfr1_override; +extern struct arm64_ftr_override id_aa64mmfr2_override; extern struct arm64_ftr_override id_aa64pfr0_override; extern struct arm64_ftr_override id_aa64pfr1_override; extern struct arm64_ftr_override id_aa64zfr0_override; @@ -1000,6 +1002,8 @@ static inline bool cpu_has_lva(void) u64 mmfr2; mmfr2 = read_sysreg_s(SYS_ID_AA64MMFR2_EL1); + mmfr2 &= ~id_aa64mmfr2_override.mask; + mmfr2 |= id_aa64mmfr2_override.val; return cpuid_feature_extract_unsigned_field(mmfr2, ID_AA64MMFR2_EL1_VARange_SHIFT); } diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 8eb8c7f7b317..ed9670d8360c 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -655,7 +655,9 @@ static const struct arm64_ftr_bits ftr_raz[] = { #define ARM64_FTR_REG(id, table) \ __ARM64_FTR_REG_OVERRIDE(#id, id, table, &no_override) +struct arm64_ftr_override id_aa64mmfr0_override; struct arm64_ftr_override id_aa64mmfr1_override; +struct arm64_ftr_override id_aa64mmfr2_override; struct arm64_ftr_override id_aa64pfr0_override; struct arm64_ftr_override id_aa64pfr1_override; struct arm64_ftr_override id_aa64zfr0_override; @@ -719,10 +721,12 @@ static const struct __ftr_reg_entry { &id_aa64isar2_override), /* Op1 = 0, CRn = 0, CRm = 7 */ - ARM64_FTR_REG(SYS_ID_AA64MMFR0_EL1, ftr_id_aa64mmfr0), + ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR0_EL1, ftr_id_aa64mmfr0, + &id_aa64mmfr0_override), ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR1_EL1, ftr_id_aa64mmfr1, &id_aa64mmfr1_override), - ARM64_FTR_REG(SYS_ID_AA64MMFR2_EL1, ftr_id_aa64mmfr2), + ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR2_EL1, ftr_id_aa64mmfr2, + &id_aa64mmfr2_override), ARM64_FTR_REG(SYS_ID_AA64MMFR3_EL1, ftr_id_aa64mmfr3), /* Op1 = 1, CRn = 0, CRm = 0 */ diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index 2b9d702abe0f..ff81f809a240 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -38,7 +38,9 @@ PROVIDE(__pi___memset = __pi_memset); PROVIDE(__pi_id_aa64isar1_override = id_aa64isar1_override); PROVIDE(__pi_id_aa64isar2_override = id_aa64isar2_override); +PROVIDE(__pi_id_aa64mmfr0_override = id_aa64mmfr0_override); PROVIDE(__pi_id_aa64mmfr1_override = id_aa64mmfr1_override); +PROVIDE(__pi_id_aa64mmfr2_override = id_aa64mmfr2_override); PROVIDE(__pi_id_aa64pfr0_override = id_aa64pfr0_override); PROVIDE(__pi_id_aa64pfr1_override = id_aa64pfr1_override); PROVIDE(__pi_id_aa64smfr0_override = id_aa64smfr0_override); diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index 1884bd936c0d..aad399796e81 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -59,6 +59,35 @@ static const struct ftr_set_desc mmfr1 __prel64_initconst = { }, }; + +static bool __init mmfr2_varange_filter(u64 val) +{ + int __maybe_unused feat; + + if (val) + return false; + +#ifdef CONFIG_ARM64_LPA2 + feat = cpuid_feature_extract_signed_field(read_sysreg(id_aa64mmfr0_el1), + ID_AA64MMFR0_EL1_TGRAN_SHIFT); + if (feat >= ID_AA64MMFR0_EL1_TGRAN_LPA2) { + id_aa64mmfr0_override.val |= + (ID_AA64MMFR0_EL1_TGRAN_LPA2 - 1) << ID_AA64MMFR0_EL1_TGRAN_SHIFT; + id_aa64mmfr0_override.mask |= 0xfU << ID_AA64MMFR0_EL1_TGRAN_SHIFT; + } +#endif + return true; +} + +static const struct ftr_set_desc mmfr2 __prel64_initconst = { + .name = "id_aa64mmfr2", + .override = &id_aa64mmfr2_override, + .fields = { + FIELD("varange", ID_AA64MMFR2_EL1_VARange_SHIFT, mmfr2_varange_filter), + {} + }, +}; + static bool __init pfr0_sve_filter(u64 val) { /* @@ -167,6 +196,7 @@ static const struct ftr_set_desc sw_features __prel64_initconst = { static const PREL64(const struct ftr_set_desc, reg) regs[] __prel64_initconst = { { &mmfr1 }, + { &mmfr2 }, { &pfr0 }, { &pfr1 }, { &isar1 }, @@ -192,6 +222,7 @@ static const struct { { "arm64.nomte", "id_aa64pfr1.mte=0" }, { "nokaslr", "arm64_sw.nokaslr=1" }, { "rodata=off", "arm64_sw.rodataoff=1" }, + { "arm64.nolva", "id_aa64mmfr2.varange=0" }, }; static int __init parse_hexdigit(const char *p, u64 *v)