From patchwork Tue Sep 12 14:16:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381866 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AF183CA0EF4 for ; Tue, 12 Sep 2023 15:30:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=foZ2q1NTIr4JKQ1c11WeGQ1AVrD5XC/pLfjPgaHfIUw=; b=p/KCwgxw7J3UBAQ78YlYVXgJ08 F/v1nF6+wrDFIA+ovcrnCvRkU9YODRGqhpw12nJ6Bhq54Ru9YxFKgUmXebDg8YpwIdBk7jDEDwhZz Jfb6ibqDtVKUGC3+1wTmiYohUDDHUMlZ4q8uv8LQV0ajak112y4TBY9ovpZ6DktUKZ41bF91Bhp84 RICTL1VHsE4LeXRri8LJi8j1bF0i0cyXkznDNExCBT1b/vHxf1/aFCdm1lPD+yCtocm5PkunyX2iY 8+9Wa/K0PwI3gd9ProNcI2nvq2ohNZ9ypNPtksouUCBypnsLmAAsGYWjWWJD/RFj5e9ggJkslF71Y p48YkHfA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg5LR-003ihM-0r; Tue, 12 Sep 2023 15:30:29 +0000 Received: from mail-wm1-f74.google.com ([209.85.128.74]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4EZ-003XMd-0n for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:29 +0000 Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-402493d2997so41148955e9.0 for ; Tue, 12 Sep 2023 07:19:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528357; x=1695133157; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6kpHMh1fxJX0WosZvnbK8dUUkz2qB4AwpNaVnaT03lM=; b=CTEPN2PKwmuXw14K43zldkyz5xCD7Lf+pDc9sB9zBlrzjyyq/5MQBmTKTQFUnTzfvU 4Puva8sJjIX6isBrevZYhwMlswDWAOAaJuQckQk4pZKXz8WqIcscJ/QYPPUnpLmw3/0S K1zbW9PyEKuotrBhUHYA8SIsVktbG4lZlWHyf2xvJlj0j2o7I9FL82iZwwpVp0+T3Rpe GrzNp1xa/lgs1jXWg+Fb4zrjHi9oEXOkX59lbTrPgWExrBkSy7psuSgdEYygVn4HpAC8 BpIw2RlbQMDSksaouik3kL/wM352AGD/0aeROv06+LSTJKJR3EplH8/ylAyt+WH5ToId ycsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528357; x=1695133157; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6kpHMh1fxJX0WosZvnbK8dUUkz2qB4AwpNaVnaT03lM=; b=InFr6nBrcUHjbjffsG+dyWTXhLedzg1Bne+bq1GMc7fXKyJJNiDFlqfQvdPcadqrB3 ombOtwNdSa8V0WOfOtUEx0/QrIxGcG+uBwIjlFMqaalg6HsnQwCV0R8/SiVB9XQ4x0dP gaK+ui6ShlufF/bKpB4TzsZqvIDHmeEsRERPYsgoZ9GJpfGD8HW/FKGhP4jm2o9JeD0c 9YaleZ18/8sOX4jR74DPgLNtGs3K/wHTEy0HYNuRcj8X0Mjdhh1iZM9E1/bvfZSKgkIP CCCEQpWYfVf5YdMIc3dKsbhlq/m/o0vcrzRIcdytv/2tDmW0ASx7Fst5wGWJuhdOqTuJ GXag== X-Gm-Message-State: AOJu0YyvqvIa2djIZNafrOFP0w4vUp9LG8LHP0iY7N3fWidSmfw1cbyM BD4+1HR9PqnGjiedtZLPzJR9DVySyBqJFXCt9nQOW9GScnC2BXaWrChCY44ZvnxsovwFXCu6mhs aFXJm+mOMNulGFslbb77XVok0aoQzghYx6KYyDcSldQ2cMT2eTLiPB3ZC68ENOtRYTkWbh+mBBM Q= X-Google-Smtp-Source: AGHT+IHmr98bQ0q5U401wQcQr4IP3HtQdtRQM1fdsEnvbniJkta9Di2RPLFxXmznY6Ni5GeK2Vcb6cT0 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a7b:cd96:0:b0:3fb:bcec:72da with SMTP id y22-20020a7bcd96000000b003fbbcec72damr214342wmj.5.1694528356893; Tue, 12 Sep 2023 07:19:16 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:40 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=13274; i=ardb@kernel.org; h=from:subject; bh=KJROpUVFPieDH4rah4e246WaYyoYeAT8rn8KGExAweE=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6NTr11mVF+csOZn2pKLsRKrHBSm9hRcff2HkVgkXO vBheu7KjlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjCRvasYGb6U8IgG6mtuuOGy ho/z2LqyB4xlgTN4zinq8Mj8Xlybtojhr8CMzZJB+Uu7Q/q3fHw2v2vHtI9beByZP1/Pm7do0yW lBD4A X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-113-ardb@google.com> Subject: [PATCH v4 50/61] arm64: Enable LPA2 at boot if supported by the system From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071919_326921_3C6CD054 X-CRM114-Status: GOOD ( 33.41 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel Update the early kernel mapping code to take 52-bit virtual addressing into account based on the LPA2 feature. This is a bit more involved than LVA (which is supported with 64k pages only), given that some page table descriptor bits change meaning in this case. To keep the handling in asm to a minimum, the initial ID map is still created with 48-bit virtual addressing, which implies that the kernel image must be loaded into 48-bit addressable physical memory. This is currently required by the boot protocol, even though we happen to support placement outside of that for LVA/64k based configurations. Enabling LPA2 involves more than setting TCR.T1SZ to a lower value, there is also a DS bit in TCR that needs to be set, and which changes the meaning of bits [9:8] in all page table descriptors. Since we cannot enable DS and every live page table descriptor at the same time, let's pivot through another temporary mapping. This avoids the need to reintroduce manipulations of the page tables with the MMU and caches disabled. To permit the LPA2 feature to be overridden on the kernel command line, which may be necessary to work around silicon errata, or to deal with mismatched features on heterogeneous SoC designs, test for CPU feature overrides first, and only then enable LPA2. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 8 ++- arch/arm64/include/asm/cpufeature.h | 18 +++++ arch/arm64/include/asm/memory.h | 4 ++ arch/arm64/kernel/head.S | 8 +++ arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kernel/pi/map_kernel.c | 70 +++++++++++++++++++- arch/arm64/kernel/pi/map_range.c | 11 ++- arch/arm64/kernel/pi/pi.h | 4 +- arch/arm64/mm/init.c | 2 +- arch/arm64/mm/mmu.c | 6 +- arch/arm64/mm/proc.S | 3 + 11 files changed, 124 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 55e8731844cf..d5e139ce0820 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -581,11 +581,17 @@ alternative_endif * but we have to add an offset so that the TTBR1 address corresponds with the * pgdir entry that covers the lowest 48-bit addressable VA. * + * Note that this trick is only used for LVA/64k pages - LPA2/4k pages uses an + * additional paging level, and on LPA2/16k pages, we would end up with a root + * level table with only 2 entries, which is suboptimal in terms of TLB + * utilization, so there we fall back to 47 bits of translation if LPA2 is not + * supported. + * * orr is used as it can cover the immediate value (and is idempotent). * ttbr: Value of ttbr to set, modified. */ .macro offset_ttbr1, ttbr, tmp -#ifdef CONFIG_ARM64_VA_BITS_52 +#if defined(CONFIG_ARM64_VA_BITS_52) && !defined(CONFIG_ARM64_LPA2) mrs \tmp, tcr_el1 and \tmp, \tmp, #TCR_T1SZ_MASK cmp \tmp, #TCR_T1SZ(VA_BITS_MIN) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 351e0b39f0bb..5eed4a12c625 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -990,6 +990,24 @@ static inline bool cpu_has_lva(void) ID_AA64MMFR2_EL1_VARange_SHIFT); } +static inline bool cpu_has_lpa2(void) +{ +#ifdef CONFIG_ARM64_LPA2 + u64 mmfr0; + int feat; + + mmfr0 = read_sysreg(id_aa64mmfr0_el1); + mmfr0 &= ~id_aa64mmfr0_override.mask; + mmfr0 |= id_aa64mmfr0_override.val; + feat = cpuid_feature_extract_signed_field(mmfr0, + ID_AA64MMFR0_EL1_TGRAN_SHIFT); + + return feat >= ID_AA64MMFR0_EL1_TGRAN_LPA2; +#else + return false; +#endif +} + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index f6277f66388f..17ec3068c5f7 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -54,7 +54,11 @@ #define FIXADDR_TOP (-UL(SZ_8M)) #if VA_BITS > 48 +#ifdef CONFIG_ARM64_16K_PAGES +#define VA_BITS_MIN (47) +#else #define VA_BITS_MIN (48) +#endif #else #define VA_BITS_MIN (VA_BITS) #endif diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index e25351addfd0..405e9bce8c73 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -89,6 +89,7 @@ SYM_CODE_START(primary_entry) mov sp, x1 mov x29, xzr adrp x0, init_idmap_pg_dir + mov x1, xzr bl __pi_create_init_idmap /* @@ -473,9 +474,16 @@ SYM_FUNC_END(__enable_mmu) #ifdef CONFIG_ARM64_VA_BITS_52 SYM_FUNC_START(__cpu_secondary_check52bitva) +#ifndef CONFIG_ARM64_LPA2 mrs_s x0, SYS_ID_AA64MMFR2_EL1 and x0, x0, ID_AA64MMFR2_EL1_VARange_MASK cbnz x0, 2f +#else + mrs x0, id_aa64mmfr0_el1 + sbfx x0, x0, #ID_AA64MMFR0_EL1_TGRAN_SHIFT, 4 + cmp x0, #ID_AA64MMFR0_EL1_TGRAN_LPA2 + b.ge 2f +#endif update_early_cpu_boot_status \ CPU_STUCK_IN_KERNEL | CPU_STUCK_REASON_52_BIT_VA, x0, x1 diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index c5eeb33a6280..9472fc3d1303 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -52,6 +52,7 @@ PROVIDE(__pi__ctype = _ctype); PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed); PROVIDE(__pi_init_idmap_pg_dir = init_idmap_pg_dir); +PROVIDE(__pi_init_idmap_pg_end = init_idmap_pg_end); PROVIDE(__pi_init_pg_dir = init_pg_dir); PROVIDE(__pi_init_pg_end = init_pg_end); PROVIDE(__pi_swapper_pg_dir = swapper_pg_dir); diff --git a/arch/arm64/kernel/pi/map_kernel.c b/arch/arm64/kernel/pi/map_kernel.c index 18a8d4249136..591394befa5f 100644 --- a/arch/arm64/kernel/pi/map_kernel.c +++ b/arch/arm64/kernel/pi/map_kernel.c @@ -128,11 +128,64 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) } /* Copy the root page table to its final location */ - memcpy((void *)swapper_pg_dir + va_offset, init_pg_dir, PGD_SIZE); + memcpy((void *)swapper_pg_dir + va_offset, init_pg_dir, PAGE_SIZE); dsb(ishst); idmap_cpu_replace_ttbr1(swapper_pg_dir); } +static void noinline __section(".idmap.text") set_ttbr0_for_lpa2(u64 ttbr) +{ + u64 sctlr = read_sysreg(sctlr_el1); + u64 tcr = read_sysreg(tcr_el1) | TCR_DS; + + asm(" msr sctlr_el1, %0 ;" + " isb ;" + " msr ttbr0_el1, %1 ;" + " msr tcr_el1, %2 ;" + " isb ;" + " tlbi vmalle1 ;" + " dsb nsh ;" + " isb ;" + " msr sctlr_el1, %3 ;" + " isb ;" + :: "r"(sctlr & ~SCTLR_ELx_M), "r"(ttbr), "r"(tcr), "r"(sctlr)); +} + +static void __init remap_idmap_for_lpa2(void) +{ + /* clear the bits that change meaning once LPA2 is turned on */ + pteval_t mask = PTE_SHARED; + + /* + * We have to clear bits [9:8] in all block or page descriptors in the + * initial ID map, as otherwise they will be (mis)interpreted as + * physical address bits once we flick the LPA2 switch (TCR.DS). Since + * we cannot manipulate live descriptors in that way without creating + * potential TLB conflicts, let's create another temporary ID map in a + * LPA2 compatible fashion, and update the initial ID map while running + * from that. + */ + create_init_idmap(init_pg_dir, mask); + dsb(ishst); + set_ttbr0_for_lpa2((u64)init_pg_dir); + + /* + * Recreate the initial ID map with the same granularity as before. + * Don't bother with the FDT, we no longer need it after this. + */ + memset(init_idmap_pg_dir, 0, + (u64)init_idmap_pg_dir - (u64)init_idmap_pg_end); + + create_init_idmap(init_idmap_pg_dir, mask); + dsb(ishst); + + /* switch back to the updated initial ID map */ + set_ttbr0_for_lpa2((u64)init_idmap_pg_dir); + + /* wipe the temporary ID map from memory */ + memset(init_pg_dir, 0, (u64)init_pg_end - (u64)init_pg_dir); +} + static void map_fdt(u64 fdt) { static u8 ptes[INIT_IDMAP_FDT_SIZE] __initdata __aligned(PAGE_SIZE); @@ -155,6 +208,7 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) u64 va_base, pa_base = (u64)&_text; u64 kaslr_offset = pa_base % MIN_KIMG_ALIGN; int root_level = 4 - CONFIG_PGTABLE_LEVELS; + int va_bits = VA_BITS; int chosen; map_fdt((u64)fdt); @@ -166,8 +220,15 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) chosen = fdt_path_offset(fdt, chosen_str); init_feature_override(boot_status, fdt, chosen); - if (VA_BITS > VA_BITS_MIN && cpu_has_lva()) - sysreg_clear_set(tcr_el1, TCR_T1SZ_MASK, TCR_T1SZ(VA_BITS)); + if (IS_ENABLED(CONFIG_ARM64_64K_PAGES) && !cpu_has_lva()) { + va_bits = VA_BITS_MIN; + } else if (IS_ENABLED(CONFIG_ARM64_LPA2) && !cpu_has_lpa2()) { + va_bits = VA_BITS_MIN; + root_level++; + } + + if (va_bits > VA_BITS_MIN) + sysreg_clear_set(tcr_el1, TCR_T1SZ_MASK, TCR_T1SZ(va_bits)); /* * The virtual KASLR displacement modulo 2MiB is decided by the @@ -185,6 +246,9 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) kaslr_offset |= kaslr_seed & ~(MIN_KIMG_ALIGN - 1); } + if (IS_ENABLED(CONFIG_ARM64_LPA2) && va_bits > VA_BITS_MIN) + remap_idmap_for_lpa2(); + va_base = KIMAGE_VADDR + kaslr_offset; map_kernel(kaslr_offset, va_base - pa_base, root_level); } diff --git a/arch/arm64/kernel/pi/map_range.c b/arch/arm64/kernel/pi/map_range.c index 79e4f6a2efe1..5410b2cac590 100644 --- a/arch/arm64/kernel/pi/map_range.c +++ b/arch/arm64/kernel/pi/map_range.c @@ -87,14 +87,19 @@ void __init map_range(u64 *pte, u64 start, u64 end, u64 pa, pgprot_t prot, } } -asmlinkage u64 __init create_init_idmap(pgd_t *pg_dir) +asmlinkage u64 __init create_init_idmap(pgd_t *pg_dir, pteval_t clrmask) { u64 ptep = (u64)pg_dir + PAGE_SIZE; + pgprot_t text_prot = PAGE_KERNEL_ROX; + pgprot_t data_prot = PAGE_KERNEL; + + pgprot_val(text_prot) &= ~clrmask; + pgprot_val(data_prot) &= ~clrmask; map_range(&ptep, (u64)_stext, (u64)__initdata_begin, (u64)_stext, - PAGE_KERNEL_ROX, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); + text_prot, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); map_range(&ptep, (u64)__initdata_begin, (u64)_end, (u64)__initdata_begin, - PAGE_KERNEL, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); + data_prot, IDMAP_ROOT_LEVEL, (pte_t *)pg_dir, false, 0); return ptep; } diff --git a/arch/arm64/kernel/pi/pi.h b/arch/arm64/kernel/pi/pi.h index a8fe79c9b111..93d7457dd10d 100644 --- a/arch/arm64/kernel/pi/pi.h +++ b/arch/arm64/kernel/pi/pi.h @@ -17,7 +17,7 @@ static inline void *prel64_to_pointer(const prel64_t *offset) extern bool dynamic_scs_is_enabled; -extern pgd_t init_idmap_pg_dir[]; +extern pgd_t init_idmap_pg_dir[], init_idmap_pg_end[]; void init_feature_override(u64 boot_status, const void *fdt, int chosen); u64 kaslr_early_init(void *fdt, int chosen); @@ -27,4 +27,4 @@ int scs_patch(const u8 eh_frame[], int size); void map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, int level, pte_t *tbl, bool may_use_cont, u64 va_offset); -asmlinkage u64 create_init_idmap(pgd_t *pgd); +asmlinkage u64 create_init_idmap(pgd_t *pgd, pteval_t clrmask); diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 8a0f8604348b..11c577358d3a 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -361,7 +361,7 @@ void __init arm64_memblock_init(void) * physical address of PAGE_OFFSET, we have to *subtract* from it. */ if (IS_ENABLED(CONFIG_ARM64_VA_BITS_52) && (vabits_actual != 52)) - memstart_addr -= _PAGE_OFFSET(48) - _PAGE_OFFSET(52); + memstart_addr -= _PAGE_OFFSET(vabits_actual) - _PAGE_OFFSET(52); /* * Apply the memory limit if it was set. Since the kernel may be loaded diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 2212289c9a02..ed18cc4cea0d 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -582,8 +582,12 @@ static void __init map_mem(pgd_t *pgdp) * entries at any level are being shared between the linear region and * the vmalloc region. Check whether this is true for the PGD level, in * which case it is guaranteed to be true for all other levels as well. + * (Unless we are running with support for LPA2, in which case the + * entire reduced VA space is covered by a single pgd_t which will have + * been populated without the PXNTable attribute by the time we get here.) */ - BUILD_BUG_ON(pgd_index(direct_map_end - 1) == pgd_index(direct_map_end)); + BUILD_BUG_ON(pgd_index(direct_map_end - 1) == pgd_index(direct_map_end) && + pgd_index(_PAGE_OFFSET(VA_BITS_MIN)) != PTRS_PER_PGD - 1); early_kfence_pool = arm64_kfence_alloc_pool(); diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index d60366144031..f32e4b087ef8 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -489,6 +489,9 @@ SYM_FUNC_START(__cpu_setup) mov x9, #64 - VA_BITS alternative_if ARM64_HAS_VA52 tcr_set_t1sz tcr, x9 +#ifdef CONFIG_ARM64_LPA2 + orr tcr, tcr, #TCR_DS +#endif alternative_else_nop_endif #endif