From patchwork Tue Sep 12 14:16:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13381808 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0C0E0CA0EEE for ; Tue, 12 Sep 2023 14:21:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=I0XFTidGzf0zrd5URN20wuQo75JMETZ711xNiQKeifw=; b=xCWKPxk1HnVPGwCvo4Eb5hkmdb oL3N99fLzcB1ik+lVvck+VpupV8rwuj35WKYDiYRoonh0FcWPLf3bil80LfazFStK+9oiUNKmQ0W4 e97j37WwJaIo134wWnFdnqVdZO3xzMlXXAedO7G7fXSni2zv1a7g2Qvx8VZDEfulcummpV8JDxCRP 6yNBQniiqF9rnCRPbQGShgRQU7om2jhZEqibc3xshW9P1rQg3Ay8Ib0qCk6cJRwgT/zp+Q1W1EV/d +JHE8X1NTM6pEun8fOe/+Qj3h8zptfG6T4KR971L81BjgcYueGt/8gn+bh6J1SDIP0z72qslG9Y1c 7axEPHBA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qg4Fe-003YGr-1C; Tue, 12 Sep 2023 14:20:26 +0000 Received: from mail-wr1-x44a.google.com ([2a00:1450:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qg4ER-003XF0-1f for linux-arm-kernel@lists.infradead.org; Tue, 12 Sep 2023 14:19:20 +0000 Received: by mail-wr1-x44a.google.com with SMTP id ffacd0b85a97d-31f46ccee0fso2672502f8f.1 for ; Tue, 12 Sep 2023 07:19:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694528349; x=1695133149; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=axKk0zNAoFzdIdM0Ez/IIWZeXThEIBY1gj3iUdXv6TE=; b=A/Qd3XKTbMolDo0nr7XdgawkrtDlcg1+U5TE7fYMwhQi2RY2grym5hOX2DyFhG7iXs NR6fMS0/noHrDzX+OHPzOrbunS7UTMxbptTlVFdn3EJmi/g6SRKWIceYHKwxW3AB5G5m vSAYUPtcMMOejzd3ySNgNe77qdRnHOf6sEm0jNqXVxhQslFg4SHo6cmxwUzTV/NCx1Ns 3k9mlejlJUhSUrpjxU0c2FvrQyAJS3wwCRsFx6X2K7WjVGKC+jvqfW87afl75vpTHzFM /PMv5DtfMsYoj3Ia9ROcNPXmj8J0bXVB38mnP0F7iTAo+GcZBvln/Pu0WgEa2p2FHmQJ o8Gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694528349; x=1695133149; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=axKk0zNAoFzdIdM0Ez/IIWZeXThEIBY1gj3iUdXv6TE=; b=GoGT+0MVAq4h+HzD8eIF5vd3upk4YKfeNPoAX2zzioLWXr/IVMCyryr5aP44HQHjEQ +NqGtKdWmpZrl7N3tTjJfjrrY3ZaGzc0BZ0ems28zBpFFanYf48ZUL55xeLeDxeCRfdI lBpFs6ZD47N4lgMn7ducv20CLmFSjTU82N81B8je4mm2HzHueXRjO7IPqzSzvGTViFGC F6y7nneGRLbNiyZAs5hm39aZYC9N/NR+IyDn0WJDlwDUBZ3aZCoRJzhRkKd3yx39uMwN jT0//P8utgNoYvXLC41NQNrlDjd7EaYdrdVtWZSz4Zje5JbSSmKko7snXVFHLegJsiWP Tz5w== X-Gm-Message-State: AOJu0YzWYivQS8YlSWpexwLSI/qxAKln9+PrrS0SofIYi6wlssHarM/G zoXVTDl5rVvlriaV2GTffQ8T3mmSqbJy+s+KpFh9fOlV9GaymiQm//ZePrc4ujC3IVAj8celltY rLU85bkn7l6IxdogOhS9ZXxMTusolLpNykaKcm5Aim4U9Mdu9QnKm+wXLQWwqFhNgrnq9Ykw3+n 8= X-Google-Smtp-Source: AGHT+IHa+Vrf71HijJ8v/ye1eoxXTqJKQIc8+FlI5nhZSuVWo/Z4EJHG7TngdNIO2AhwuuvsamOWKu2v X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6000:100a:b0:31d:c732:8d4 with SMTP id a10-20020a056000100a00b0031dc73208d4mr53471wrx.3.1694528349354; Tue, 12 Sep 2023 07:19:09 -0700 (PDT) Date: Tue, 12 Sep 2023 14:16:37 +0000 In-Reply-To: <20230912141549.278777-63-ardb@google.com> Mime-Version: 1.0 References: <20230912141549.278777-63-ardb@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=4645; i=ardb@kernel.org; h=from:subject; bh=9BsuuvReeuLNSahj33TapR0erGyCQo/IE359oNWbL7g=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIZWh6LjepWbuFW1twiXuJ8Teih3bceSLzQ37B7Ks/mka+ S+Yf2/vKGVhEONgkBVTZBGY/ffdztMTpWqdZ8nCzGFlAhnCwMUpABPRLGH4nxdWO319y6XqdYVG i21KN4hwpK0PbVlkPduLZS7LzmOX2xkZdh3Yz/TnXfHM7zXXDb90G/E/Nbrg5Hku52yCckqyt78 EDwA= X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230912141549.278777-110-ardb@google.com> Subject: [PATCH v4 47/61] arm64: mm: Add LPA2 support to phys<->pte conversion routines From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook , Joey Gouly X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230912_071911_682796_BC0825C9 X-CRM114-Status: GOOD ( 16.42 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel In preparation for enabling LPA2 support, introduce the mask values for converting between physical addresses and their representations in a page table descriptor. While at it, move the pte_to_phys asm macro into its only user, so that we can freely modify it to use its input value register as a temp register. For LPA2, the PTE_ADDR_MASK contains two non-adjacent sequences of zero bits, which means it no longer fits into the immediate field of an ordinary ALU instruction. So let's redefine it to include the bits in between as well, and only use it when converting from physical address to PTE representation, where the distinction does not matter. Also update the name accordingly to emphasize this. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 16 ++-------------- arch/arm64/include/asm/pgtable-hwdef.h | 10 +++++++--- arch/arm64/include/asm/pgtable.h | 5 +++-- arch/arm64/mm/proc.S | 8 ++++++++ 4 files changed, 20 insertions(+), 19 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 0710c17800a4..55e8731844cf 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -612,25 +612,13 @@ alternative_endif .macro phys_to_pte, pte, phys #ifdef CONFIG_ARM64_PA_BITS_52 - /* - * We assume \phys is 64K aligned and this is guaranteed by only - * supporting this configuration with 64K pages. - */ - orr \pte, \phys, \phys, lsr #36 - and \pte, \pte, #PTE_ADDR_MASK + orr \pte, \phys, \phys, lsr #PTE_ADDR_HIGH_SHIFT + and \pte, \pte, #PHYS_TO_PTE_ADDR_MASK #else mov \pte, \phys #endif .endm - .macro pte_to_phys, phys, pte - and \phys, \pte, #PTE_ADDR_MASK -#ifdef CONFIG_ARM64_PA_BITS_52 - orr \phys, \phys, \phys, lsl #PTE_ADDR_HIGH_SHIFT - and \phys, \phys, GENMASK_ULL(PHYS_MASK_SHIFT - 1, PAGE_SHIFT) -#endif - .endm - /* * tcr_clear_errata_bits - Clear TCR bits that trigger an errata on this CPU. */ diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index b770f98fc0b5..4426f48f2ae0 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -155,13 +155,17 @@ #define PTE_PXN (_AT(pteval_t, 1) << 53) /* Privileged XN */ #define PTE_UXN (_AT(pteval_t, 1) << 54) /* User XN */ -#define PTE_ADDR_LOW (((_AT(pteval_t, 1) << (48 - PAGE_SHIFT)) - 1) << PAGE_SHIFT) +#define PTE_ADDR_LOW (((_AT(pteval_t, 1) << (50 - PAGE_SHIFT)) - 1) << PAGE_SHIFT) #ifdef CONFIG_ARM64_PA_BITS_52 +#ifdef CONFIG_ARM64_64K_PAGES #define PTE_ADDR_HIGH (_AT(pteval_t, 0xf) << 12) -#define PTE_ADDR_MASK (PTE_ADDR_LOW | PTE_ADDR_HIGH) #define PTE_ADDR_HIGH_SHIFT 36 +#define PHYS_TO_PTE_ADDR_MASK (PTE_ADDR_LOW | PTE_ADDR_HIGH) #else -#define PTE_ADDR_MASK PTE_ADDR_LOW +#define PTE_ADDR_HIGH (_AT(pteval_t, 0x3) << 8) +#define PTE_ADDR_HIGH_SHIFT 42 +#define PHYS_TO_PTE_ADDR_MASK GENMASK_ULL(49, 8) +#endif #endif /* diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 1858abd51b8f..2d1294b532f4 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -80,15 +80,16 @@ extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)]; #ifdef CONFIG_ARM64_PA_BITS_52 static inline phys_addr_t __pte_to_phys(pte_t pte) { + pte_val(pte) &= ~PTE_MAYBE_SHARED; return (pte_val(pte) & PTE_ADDR_LOW) | ((pte_val(pte) & PTE_ADDR_HIGH) << PTE_ADDR_HIGH_SHIFT); } static inline pteval_t __phys_to_pte_val(phys_addr_t phys) { - return (phys | (phys >> PTE_ADDR_HIGH_SHIFT)) & PTE_ADDR_MASK; + return (phys | (phys >> PTE_ADDR_HIGH_SHIFT)) & PHYS_TO_PTE_ADDR_MASK; } #else -#define __pte_to_phys(pte) (pte_val(pte) & PTE_ADDR_MASK) +#define __pte_to_phys(pte) (pte_val(pte) & PTE_ADDR_LOW) #define __phys_to_pte_val(phys) (phys) #endif diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 12305fbefe1f..31be0d16241d 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -205,6 +205,14 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) .pushsection ".idmap.text", "a" + .macro pte_to_phys, phys, pte + and \phys, \pte, #PTE_ADDR_LOW +#ifdef CONFIG_ARM64_PA_BITS_52 + and \pte, \pte, #PTE_ADDR_HIGH + orr \phys, \phys, \pte, lsl #PTE_ADDR_HIGH_SHIFT +#endif + .endm + .macro kpti_mk_tbl_ng, type, num_entries add end_\type\()p, cur_\type\()p, #\num_entries * 8 .Ldo_\type: