From patchwork Tue Jan 23 14:53:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13527555 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BE831C47258 for ; Tue, 23 Jan 2024 14:57:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=VBh6X38Vlu66A5QeuWR4q7zr243smSAyOEj3cpAH3aA=; b=DgHjEDApe69zjpK4I0TXv3EDih LNU9PWyRubEzY4GxFt7+N8YZ5ywFkEZzCsc0PyX18QLEBCpvnr/AnVp+Nu1Nj7zaqiisuYPABMOOc d+HFL3Ky5KDhIumfrCSjTx4M5b9Pdd0lb36unHgKCxTOcJm9i+rKR7zWO8lThVxG+GsUC/fSGMyC7 aQSEwYOcZkX1HgaBD/vlyYe2aDH/LxLM9fHEa9NrnSlazBVDWGebirwEyGVAeV9+R2GRF/zUI/gOy eiHXxfmhlK4vTUNa5tkc1ns4fAVatdlG+urma2so6bYRl4ULtquSM+6hwWe26yaMKLpTg94js2ZNL 4xI9eR5w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rSICm-00Gurj-2O; Tue, 23 Jan 2024 14:56:48 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rSIBt-00GuO2-2V for linux-arm-kernel@bombadil.infradead.org; Tue, 23 Jan 2024 14:55:53 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=E4wrsZgepcPmsNgQem185zgjGPeGrdho/+cjw/u+q8w=; b=HbGdudrbO62iuksk3/QVBS+/x0 Sb9Uubh0kFTAzj2d8cX1WJix+gOwuvkb0YkBCtNxtQIkUQW317oW6R5apPlsv7D5U/h0KmbYaxcBM VItWzVsqFlJ6mr+yu24OK6ujiW1/dwKs9VLIFmdeX8vNZANkwDdLOYXm2vdWAjbeYLYI/Nab3tenF IftebdquYJQKaUMfytD0UXgsXEoDiS+atjb5O/vB1sVjgDG3lNAYJcvMij+z67LeGTURLbCnKWQd5 p+kaLhYTGXYJAX8Ka+MCEH9O1WiZCMq9vzutiBYnWmelcYqUgGveQYLugcX35jZddlTB8KCVEE7Ek BI4jNWJA==; Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rSIBp-00000003vXt-3IgG for linux-arm-kernel@lists.infradead.org; Tue, 23 Jan 2024 14:55:51 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-dbdb46770d7so6226098276.1 for ; Tue, 23 Jan 2024 06:55:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706021747; x=1706626547; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=E4wrsZgepcPmsNgQem185zgjGPeGrdho/+cjw/u+q8w=; b=mFisMEDkh3q/TNu815oJukyBOhL1LqvQ4GmD+XBPjXw1hzHY7EDl95Pfeskx/Ac+7h Dg3oHzBwD68AslRoI+MxgbxctKCTZnGUag7KCanCaWmaYoMg5F1VpKGv1gSoDOLc4d2M hwxZM0DjmcSCjDxxOjTFXKa6DsIdYnNjeXBWVmqzpdPVJ2Y7d/RnY3bFAntRHo0AEkoS NzJdhYUkwjPIEifFHdDAos8CZejvaTW2bAlvsAzNLMRpfpXK4dnmhaq1yB6fXGB10IRC hAE8hkAxwEep9eEwocaRjxWawk4EXFNRlIDDE4iMyEqerhFNs0cHYppzO83kz7ddETLY v09Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706021747; x=1706626547; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=E4wrsZgepcPmsNgQem185zgjGPeGrdho/+cjw/u+q8w=; b=q/6Ddg0ItDfs+PwaYK5OoM5dT1K6xhqxmSJ19+5ofnT7t4WHdQ5wS+dwKE4pZgClm0 Mar6296XSqoBZEp/YBzzwNW+izb9oPgD58OnasxvaOzOwHmob0WRUwd+4s8sQFKuSK8v 8HNHzEwqY6J21B1lYQXj8Y009GxdICPCpr3BBwUDPLmaeoRt6SPA0QPcT8lLSq9H1qtz pV5QiGLiMljFdN99nlyljra7fhL/GPI08RAn0U6oExozVRyrID/9V7fxNaXpQfu3Dj4l x6RLszBI0gq0MFFDNF4/yCSwl5mJqMSVtuejyiNRW8dh5TquJkT27SQu/3htRFmBp1bj oTsA== X-Gm-Message-State: AOJu0Yyy5f6mGxDkuCZmGYSseFhBKnESn+pEHGEi7B2V3C60Hq5eFmdk dc2Zefn5ndxh0I7JsmbT2J0h7saSRnp9ofAmxoiB+pp13V3zateW9lBeYwNdUh4htgr0+gfZas4 Vy/130p721yfj6hx7aq+fgaSwwhcNXyu+VXOw35EFkNnqxC24vkRHby2hpHxRWCv1FnAZdK13ZW 2iYaXEZxO8lJQMhmUFZxtRZdFdWcabQSD7UY5oTkqu X-Google-Smtp-Source: AGHT+IFvSuEtlYS8p2iotss0uTEq41hV2hT/c9lhW+lmYKUjt2wpzhhVniQkaSC3dQZ7YpEAUZUHMPWu X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a25:8152:0:b0:dbc:c697:63bd with SMTP id j18-20020a258152000000b00dbcc69763bdmr2613904ybm.0.1706021747661; Tue, 23 Jan 2024 06:55:47 -0800 (PST) Date: Tue, 23 Jan 2024 15:53:36 +0100 In-Reply-To: <20240123145258.1462979-52-ardb+git@google.com> Mime-Version: 1.0 References: <20240123145258.1462979-52-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=4645; i=ardb@kernel.org; h=from:subject; bh=b423qQkbUAHh+y9gBE+yrDBhYvk6YjkRX7q73L9IZXA=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIXX9pc/bay/JTZyh7F8/8794gVqCDNvJw+yTP9598HHCe rbqv9/YOkpZGMQ4GGTFFFkEZv99t/P0RKla51myMHNYmUCGMHBxCsBEBD4z/BVZGmU5X7L5ysn9 Gn8rnQ5YHP3FychVeL90qs5LuZC15QcZ/llOanebuIj9t4/Y4+d6rAsO5NcddTk7qWras4/LPz0 +eZwZAA== X-Mailer: git-send-email 2.43.0.429.g432eaa2c6b-goog Message-ID: <20240123145258.1462979-89-ardb+git@google.com> Subject: [PATCH v7 37/50] arm64: mm: Add LPA2 support to phys<->pte conversion routines From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240123_145549_958241_90DCA0D0 X-CRM114-Status: GOOD ( 15.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel In preparation for enabling LPA2 support, introduce the mask values for converting between physical addresses and their representations in a page table descriptor. While at it, move the pte_to_phys asm macro into its only user, so that we can freely modify it to use its input value register as a temp register. For LPA2, the PTE_ADDR_MASK contains two non-adjacent sequences of zero bits, which means it no longer fits into the immediate field of an ordinary ALU instruction. So let's redefine it to include the bits in between as well, and only use it when converting from physical address to PTE representation, where the distinction does not matter. Also update the name accordingly to emphasize this. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 16 ++-------------- arch/arm64/include/asm/pgtable-hwdef.h | 10 +++++++--- arch/arm64/include/asm/pgtable.h | 5 +++-- arch/arm64/mm/proc.S | 8 ++++++++ 4 files changed, 20 insertions(+), 19 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 68a99b116256..7eedcb36ebe0 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -612,25 +612,13 @@ alternative_endif .macro phys_to_pte, pte, phys #ifdef CONFIG_ARM64_PA_BITS_52 - /* - * We assume \phys is 64K aligned and this is guaranteed by only - * supporting this configuration with 64K pages. - */ - orr \pte, \phys, \phys, lsr #36 - and \pte, \pte, #PTE_ADDR_MASK + orr \pte, \phys, \phys, lsr #PTE_ADDR_HIGH_SHIFT + and \pte, \pte, #PHYS_TO_PTE_ADDR_MASK #else mov \pte, \phys #endif .endm - .macro pte_to_phys, phys, pte - and \phys, \pte, #PTE_ADDR_MASK -#ifdef CONFIG_ARM64_PA_BITS_52 - orr \phys, \phys, \phys, lsl #PTE_ADDR_HIGH_SHIFT - and \phys, \phys, GENMASK_ULL(PHYS_MASK_SHIFT - 1, PAGE_SHIFT) -#endif - .endm - /* * tcr_clear_errata_bits - Clear TCR bits that trigger an errata on this CPU. */ diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index b770f98fc0b5..4426f48f2ae0 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -155,13 +155,17 @@ #define PTE_PXN (_AT(pteval_t, 1) << 53) /* Privileged XN */ #define PTE_UXN (_AT(pteval_t, 1) << 54) /* User XN */ -#define PTE_ADDR_LOW (((_AT(pteval_t, 1) << (48 - PAGE_SHIFT)) - 1) << PAGE_SHIFT) +#define PTE_ADDR_LOW (((_AT(pteval_t, 1) << (50 - PAGE_SHIFT)) - 1) << PAGE_SHIFT) #ifdef CONFIG_ARM64_PA_BITS_52 +#ifdef CONFIG_ARM64_64K_PAGES #define PTE_ADDR_HIGH (_AT(pteval_t, 0xf) << 12) -#define PTE_ADDR_MASK (PTE_ADDR_LOW | PTE_ADDR_HIGH) #define PTE_ADDR_HIGH_SHIFT 36 +#define PHYS_TO_PTE_ADDR_MASK (PTE_ADDR_LOW | PTE_ADDR_HIGH) #else -#define PTE_ADDR_MASK PTE_ADDR_LOW +#define PTE_ADDR_HIGH (_AT(pteval_t, 0x3) << 8) +#define PTE_ADDR_HIGH_SHIFT 42 +#define PHYS_TO_PTE_ADDR_MASK GENMASK_ULL(49, 8) +#endif #endif /* diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 522c21348ae8..61de7b1516bc 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -80,15 +80,16 @@ extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)]; #ifdef CONFIG_ARM64_PA_BITS_52 static inline phys_addr_t __pte_to_phys(pte_t pte) { + pte_val(pte) &= ~PTE_MAYBE_SHARED; return (pte_val(pte) & PTE_ADDR_LOW) | ((pte_val(pte) & PTE_ADDR_HIGH) << PTE_ADDR_HIGH_SHIFT); } static inline pteval_t __phys_to_pte_val(phys_addr_t phys) { - return (phys | (phys >> PTE_ADDR_HIGH_SHIFT)) & PTE_ADDR_MASK; + return (phys | (phys >> PTE_ADDR_HIGH_SHIFT)) & PHYS_TO_PTE_ADDR_MASK; } #else -#define __pte_to_phys(pte) (pte_val(pte) & PTE_ADDR_MASK) +#define __pte_to_phys(pte) (pte_val(pte) & PTE_ADDR_LOW) #define __phys_to_pte_val(phys) (phys) #endif diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 7c46f8cfd6ae..d03434b7bca5 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -205,6 +205,14 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) .pushsection ".idmap.text", "a" + .macro pte_to_phys, phys, pte + and \phys, \pte, #PTE_ADDR_LOW +#ifdef CONFIG_ARM64_PA_BITS_52 + and \pte, \pte, #PTE_ADDR_HIGH + orr \phys, \phys, \pte, lsl #PTE_ADDR_HIGH_SHIFT +#endif + .endm + .macro kpti_mk_tbl_ng, type, num_entries add end_\type\()p, cur_\type\()p, #\num_entries * 8 .Ldo_\type: