From patchwork Thu Nov 17 13:24:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13046887 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8ADFEC4332F for ; Thu, 17 Nov 2022 13:27:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=FpnqzHhemDlF1WHLrFMxP1yGz8HrXQCmnZ/NRd+zAy4=; b=YUtGc2wT0Kvp/K 8E8jzt+MqfS15I4NLtduVJ4Cs86+mgGbwzrNdisax0KyfJ8xrPVnrzQqFjtlFt6vGFWpcDieJbAPW e/+VmMyaBHgbfD8WlOfwQgxHsxX74Rz4VCT1c8GkbZ7HFJc85ULbo8zWUsQU0IRziYWRxbBufz1ce QnFXCMJ94p3HzW0eMr6xzfooVNXYuY8yLI4WdWotEW6HCXw3B0mDFni4CYznHCwRenY0UfdsRmwFS 5I1kyecREDQZjKCHHks2UUG2h+1qv3hj+cslDIrIbcjal1vCTFqNcQrIxFa7bpwk5PncAN9TU226V QLadb3cWE5MCTfopDbgA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oveuZ-00EFsj-BQ; Thu, 17 Nov 2022 13:26:36 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ovesm-00EESM-5N for linux-arm-kernel@lists.infradead.org; Thu, 17 Nov 2022 13:24:46 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 066F261DDB; Thu, 17 Nov 2022 13:24:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5929FC43145; Thu, 17 Nov 2022 13:24:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1668691482; bh=JVg2NwngbQcTNpL08is0iO4FiyT5nKKVP/xhLJGt3PM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cm65h20lYxVHfhzb+QYo6So+zY8dPQTbZRCmxbyXXzya5fKJn/AU/LeePOO8ozPe4 TL/QFCv/zez/iIpDutn2Lvt8BD9tTzDTRcmxCAo80/W8mr0+4vity8LO0kwwifLjZA v8WYXevPldYhtBrSl2C4Apc+ILv7LQDr+G4icFeXWjEJlwNRwEkEdHpHP4tUhxeQJA ZwFhuA58nlxoVovnOH83wBcv21G9LK/5f61RMwll3AXEBKTU6pkAXEmTE1TRXfhuGk ACBCNCZUL9dQLzboCkcrfhNyT10un9WIDmpDjvN4Y0IUYnkB+S7pp3vyv7W8gBSA6M uDGNX8P9tsukQ== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Marc Zyngier , Will Deacon , Mark Rutland , Kees Cook , Catalin Marinas , Mark Brown , Anshuman Khandual , Richard Henderson Subject: [RFC PATCH 5/7] arm64: mm: Add LPA2 support to phys<->pte conversion routines Date: Thu, 17 Nov 2022 14:24:21 +0100 Message-Id: <20221117132423.1252942-6-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221117132423.1252942-1-ardb@kernel.org> References: <20221117132423.1252942-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4773; i=ardb@kernel.org; h=from:subject; bh=JVg2NwngbQcTNpL08is0iO4FiyT5nKKVP/xhLJGt3PM=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjdjYD0TnPQVELbMMOqX21V1WkHqIuy1B8Fu+o2UR1 TmvspSaJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY3Y2AwAKCRDDTyI5ktmPJGQHC/ 9i5duAo0lJubGfvFNZoUq8LcaQvgBzyv6Kr23VWEWNSmILfFIW0SFFTquKc1iAtED+X68/CSG9yi9F w2Jpd9xc5E/a88IcWsWfSalbd7hZ27xq4fQPbxDrK8xkYD65b8tWobV0B3RIDf9tJVIdKwWelAqj+R fzF0/h+kIN0ogBcGsnZ9z8bWAF/7viTG0+hH+WD33mSdxveif3PZ3JnJEVo6s5FVaKRgL5lDX+hsFM Alo+nWGFNnRBt84qzse6q485ksAcAR7RO42SmmIgHSNdqG0SGZybHP2su+qUy1qS6CI3f4iK5ulnCy ux6NNvkWIYvz7Rp6gPZXsHPy7VrzdNDN1t+MSY1qPyjfxQBgHyhSREolIqqgILqY8qDcNPUYdSCcXr jZENMV3AvTI+jqvzthswgv9viFiAl3KSoOT+4z7KS7LdGHTxHU+wMN1V0LvKe+f8ciirzGzg68O0JX bvl/Jk052gY59hwHlf2yIvm8UQ8oxxj1hE8btc/3EUdm8= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221117_052444_423195_8EDFC3F6 X-CRM114-Status: GOOD ( 15.70 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In preparation for enabling LPA2 support, introduce the mask values for converting between physical addresses and their representations in a page table descriptor. While at it, move pte_to_phys into its only user, which gets invoked when system-wide alternatives are applied, which means we can rely on a boot-time alternative here. For LPA2, the PTE_ADDR_MASK contains two non-adjacent sequences of zero bits, which means it no longer fits into the immediate field of an ordinary ALU instruction. So let's redefine it to include the bits in between as well, and only use it when converting from physical address to PTE representation, where the distinction does not matter. Also update the name accordingly to emphasize this. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 16 ++-------------- arch/arm64/include/asm/pgtable-hwdef.h | 10 +++++++--- arch/arm64/include/asm/pgtable.h | 5 +++-- arch/arm64/mm/proc.S | 10 ++++++++++ 4 files changed, 22 insertions(+), 19 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 9fa62f102c1c94e9..44a801e1dc4bf027 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -644,25 +644,13 @@ alternative_endif .macro phys_to_pte, pte, phys #ifdef CONFIG_ARM64_PA_BITS_52 - /* - * We assume \phys is 64K aligned and this is guaranteed by only - * supporting this configuration with 64K pages. - */ - orr \pte, \phys, \phys, lsr #36 - and \pte, \pte, #PTE_ADDR_MASK + orr \pte, \phys, \phys, lsr #PTE_ADDR_HIGH_SHIFT + and \pte, \pte, #PHYS_TO_PTE_ADDR_MASK #else mov \pte, \phys #endif .endm - .macro pte_to_phys, phys, pte - and \phys, \pte, #PTE_ADDR_MASK -#ifdef CONFIG_ARM64_PA_BITS_52 - orr \phys, \phys, \phys, lsl #PTE_ADDR_HIGH_SHIFT - and \phys, \phys, GENMASK_ULL(PHYS_MASK_SHIFT - 1, PAGE_SHIFT) -#endif - .endm - /* * tcr_clear_errata_bits - Clear TCR bits that trigger an errata on this CPU. */ diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index c4ad7fbb12c5c07a..b91fe4781b066d54 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -155,13 +155,17 @@ #define PTE_PXN (_AT(pteval_t, 1) << 53) /* Privileged XN */ #define PTE_UXN (_AT(pteval_t, 1) << 54) /* User XN */ -#define PTE_ADDR_LOW (((_AT(pteval_t, 1) << (48 - PAGE_SHIFT)) - 1) << PAGE_SHIFT) +#define PTE_ADDR_LOW (((_AT(pteval_t, 1) << (50 - PAGE_SHIFT)) - 1) << PAGE_SHIFT) #ifdef CONFIG_ARM64_PA_BITS_52 +#ifdef CONFIG_ARM64_64K_PAGES #define PTE_ADDR_HIGH (_AT(pteval_t, 0xf) << 12) -#define PTE_ADDR_MASK (PTE_ADDR_LOW | PTE_ADDR_HIGH) #define PTE_ADDR_HIGH_SHIFT 36 +#define PHYS_TO_PTE_ADDR_MASK (PTE_ADDR_LOW | PTE_ADDR_HIGH) #else -#define PTE_ADDR_MASK PTE_ADDR_LOW +#define PTE_ADDR_HIGH (_AT(pteval_t, 0x3) << 8) +#define PTE_ADDR_HIGH_SHIFT 42 +#define PHYS_TO_PTE_ADDR_MASK GENMASK_ULL(49, 8) +#endif #endif /* diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index daedd6172227f0ca..666db7173d0f9b66 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -76,15 +76,16 @@ extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)]; #ifdef CONFIG_ARM64_PA_BITS_52 static inline phys_addr_t __pte_to_phys(pte_t pte) { + pte_val(pte) &= ~PTE_MAYBE_SHARED; return (pte_val(pte) & PTE_ADDR_LOW) | ((pte_val(pte) & PTE_ADDR_HIGH) << PTE_ADDR_HIGH_SHIFT); } static inline pteval_t __phys_to_pte_val(phys_addr_t phys) { - return (phys | (phys >> PTE_ADDR_HIGH_SHIFT)) & PTE_ADDR_MASK; + return (phys | (phys >> PTE_ADDR_HIGH_SHIFT)) & PHYS_TO_PTE_ADDR_MASK; } #else -#define __pte_to_phys(pte) (pte_val(pte) & PTE_ADDR_MASK) +#define __pte_to_phys(pte) (pte_val(pte) & PTE_ADDR_LOW) #define __phys_to_pte_val(phys) (phys) #endif diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 02818fa6aded3218..c747a2ef478cabec 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -208,6 +208,16 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) .pushsection ".idmap.text", "awx" + .macro pte_to_phys, phys, pte + and \phys, \pte, #PTE_ADDR_LOW +#ifdef CONFIG_ARM64_PA_BITS_52 +alternative_if ARM64_HAS_LVA + orr \phys, \phys, \pte, lsl #PTE_ADDR_HIGH_SHIFT + and \phys, \phys, GENMASK_ULL(PHYS_MASK_SHIFT - 1, PAGE_SHIFT) +alternative_else_nop_endif +#endif + .endm + .macro kpti_mk_tbl_ng, type, num_entries add end_\type\()p, cur_\type\()p, #\num_entries * 8 .Ldo_\type: