From patchwork Thu Jun 9 17:43:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12875870 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3A022C433EF for ; Thu, 9 Jun 2022 17:44:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=GPBxwLkaMGmtSi6M5kV66ykM6H8nv5U1RPutjU2w21g=; b=1RD8UfA8N9GhAk 6cQbpWAtibmEF3U3AZNwuCFcX6OdZS9ha6jHIgBXsxP2PFzESQ5J+T/RPXV9Hy5NNkD3n4oktt+bQ xEP378UTIK2bTjb024n21TrkFrHnDIlQXLrrQImywCKAk4FpE+BhjQysSpX5pdmWAa1CBvEoC3cip nypIKyv5sDt7mECQFl7cLvaEHey1d8bTyKis+oiAYg8oyaIOwDwfLU8b4N9UYxHWzadypB2P3I/9P LEYPri2x/tTV9OP3oZ7S0higyo51FZ0bpeW1lo/jjbQfq2IOm8pX9p5cL2L5zLHtxKKM3ncpq4y0c FJsYEYwUaWxT4wgTiJIw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nzMC8-003GVY-8p; Thu, 09 Jun 2022 17:43:44 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nzMBw-003GPV-NU for linux-arm-kernel@lists.infradead.org; Thu, 09 Jun 2022 17:43:34 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 0395761BED; Thu, 9 Jun 2022 17:43:32 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DAFD2C341C0; Thu, 9 Jun 2022 17:43:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1654796611; bh=rSWIgoS95v9XbMumsDTJrZZcltFcUDi3tRgBJdAZlMo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AO6LQ63gntoHLtEH/MX0eRRWw/IMys1z4oCVg3m2W/ciw4sxntkO9TirUzlTgobhY IQGUMfT6HA+c1JPs+piAurrZuM6PMOPBX1DBv5D2/iakVjx8FxoMDWQ3wS0Qp0F1VW sVE1ILfz+xKs3ZQJTJPlaRYxiPxW3iRb/0PlK4f8mIL2FmuyjEHUTZxElNgheXIHF3 bU0LvolkyvhijBhetjBKQCk20oi/644k0vrT18qANveZVzoRj896XNEX/h4vg1TiY+ sKxWKcA5j4ddLW0SuQtCoYl9oV0Kz7KZ9O0Q4mh6FbW39iRimNrzgnSakRShz6Xpv2 CzUPF26HaW2Og== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , will@kernel.org, maz@kernel.org, mark.rutland@arm.com, catalin.marinas@arm.com, keescook@chromium.org Subject: [PATCH v5 1/2] arm64: kpti-ng: simplify page table traversal logic Date: Thu, 9 Jun 2022 19:43:19 +0200 Message-Id: <20220609174320.4035379-2-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220609174320.4035379-1-ardb@kernel.org> References: <20220609174320.4035379-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5177; h=from:subject; bh=rSWIgoS95v9XbMumsDTJrZZcltFcUDi3tRgBJdAZlMo=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBiojE0SsuM40QdYDfYmwbNQLJBBkJq58a5uorv5Nif oVR2AHOJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYqIxNAAKCRDDTyI5ktmPJJ43C/ 4tPBSHhhfclzWLEPHoQVBWzclNPzEe+80z3lJINkwPbC+XbxmgaWIzc/9JbcSQNXI8XzYxezBGzhLp DA2d0nTDESU9VUr7VrsOl/hp0bR31Lf6K/Fy10aymZT6rq5ZI9Vhe2Ah8kmIme7i+DccTQIIw8A/iW Fnpug0YvRiEF6QYF0kUEQYToDuz4U5RXMpdg18z+vb4SdmUpkVxR1B9nZR35gNZv4Bftikv+ZmUbom G8Rt4s5FopVKXyjONKpsRu/oLXABjTKZvSEU7IR/XOXO8vDpq+h7m/ZCMuc2aQx1WhV+sCgH7moVa6 7Vxqd45bJIuRvIRNPIvJFdD9fnUNPFQ/vF67xwyGcnEvpjOSoWcWriiKGwF6Y5HH+iF6zm+3UgSP2A Efd/FUX4Umt2yGTI0eFM4qFzlm0Sh40GaI580j+Th6meg9W8MW6FzcMjFuStQtw6EaKCgv5aHehXJ4 vrTP6QYTWXSNB6BqZPic6kg+uiy+ktmbY/TPPv5ic0iZI= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220609_104332_902321_B5BE414F X-CRM114-Status: GOOD ( 14.90 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Simplify the KPTI G-to-nG asm helper code by: - pulling the 'table bit' test into the get/put macros so we can combine them and incorporate the entire loop; - moving the 'table bit' test after the update of bit #11 so we no longer need separate next_xxx and skip_xxx labels; - redefining the pmd/pud register aliases and the next_pmd/next_pud labels instead of branching to them if the number of configured page table levels is less than 3 or 4, respectively. No functional change intended, except for the fact that we now descend into a next level table after setting bit #11 on its descriptor but this should make no difference in practice. While at it, switch to .L prefixed local labels so they don't clutter up the symbol tables, kallsyms, etc, and clean up the indentation for legibility. Signed-off-by: Ard Biesheuvel Reviewed-by: Mark Rutland Tested-by: Mark Rutland --- arch/arm64/mm/proc.S | 100 +++++++------------- 1 file changed, 36 insertions(+), 64 deletions(-) diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 50bbed947bec..660887152dba 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -202,19 +202,25 @@ SYM_FUNC_END(idmap_cpu_replace_ttbr1) #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 .pushsection ".idmap.text", "awx" - .macro __idmap_kpti_get_pgtable_ent, type - dc cvac, cur_\()\type\()p // Ensure any existing dirty + .macro kpti_mk_tbl_ng, type, num_entries + add end_\type\()p, cur_\type\()p, #\num_entries * 8 +.Ldo_\type: + dc cvac, cur_\type\()p // Ensure any existing dirty dmb sy // lines are written back before - ldr \type, [cur_\()\type\()p] // loading the entry - tbz \type, #0, skip_\()\type // Skip invalid and - tbnz \type, #11, skip_\()\type // non-global entries - .endm - - .macro __idmap_kpti_put_pgtable_ent_ng, type + ldr \type, [cur_\type\()p] // loading the entry + tbz \type, #0, .Lnext_\type // Skip invalid and + tbnz \type, #11, .Lnext_\type // non-global entries orr \type, \type, #PTE_NG // Same bit for blocks and pages - str \type, [cur_\()\type\()p] // Update the entry and ensure + str \type, [cur_\type\()p] // Update the entry and ensure dmb sy // that it is visible to all dc civac, cur_\()\type\()p // CPUs. + .ifnc \type, pte + tbnz \type, #1, .Lderef_\type + .endif +.Lnext_\type: + add cur_\type\()p, cur_\type\()p, #8 + cmp cur_\type\()p, end_\type\()p + b.ne .Ldo_\type .endm /* @@ -235,10 +241,8 @@ SYM_FUNC_START(idmap_kpti_install_ng_mappings) pgd .req x7 cur_pudp .req x8 end_pudp .req x9 - pud .req x10 cur_pmdp .req x11 end_pmdp .req x12 - pmd .req x13 cur_ptep .req x14 end_ptep .req x15 pte .req x16 @@ -265,16 +269,8 @@ SYM_FUNC_START(idmap_kpti_install_ng_mappings) /* Everybody is enjoying the idmap, so we can rewrite swapper. */ /* PGD */ - mov cur_pgdp, swapper_pa - add end_pgdp, cur_pgdp, #(PTRS_PER_PGD * 8) -do_pgd: __idmap_kpti_get_pgtable_ent pgd - tbnz pgd, #1, walk_puds -next_pgd: - __idmap_kpti_put_pgtable_ent_ng pgd -skip_pgd: - add cur_pgdp, cur_pgdp, #8 - cmp cur_pgdp, end_pgdp - b.ne do_pgd + mov cur_pgdp, swapper_pa + kpti_mk_tbl_ng pgd, PTRS_PER_PGD /* Publish the updated tables and nuke all the TLBs */ dsb sy @@ -291,59 +287,35 @@ skip_pgd: str wzr, [flag_ptr] ret +.Lderef_pgd: /* PUD */ -walk_puds: - .if CONFIG_PGTABLE_LEVELS > 3 + .if CONFIG_PGTABLE_LEVELS > 3 + pud .req x10 pte_to_phys cur_pudp, pgd - add end_pudp, cur_pudp, #(PTRS_PER_PUD * 8) -do_pud: __idmap_kpti_get_pgtable_ent pud - tbnz pud, #1, walk_pmds -next_pud: - __idmap_kpti_put_pgtable_ent_ng pud -skip_pud: - add cur_pudp, cur_pudp, 8 - cmp cur_pudp, end_pudp - b.ne do_pud - b next_pgd - .else /* CONFIG_PGTABLE_LEVELS <= 3 */ - mov pud, pgd - b walk_pmds -next_pud: - b next_pgd + kpti_mk_tbl_ng pud, PTRS_PER_PUD + b .Lnext_pgd + .else /* CONFIG_PGTABLE_LEVELS <= 3 */ + pud .req pgd + .set .Lnext_pud, .Lnext_pgd .endif +.Lderef_pud: /* PMD */ -walk_pmds: - .if CONFIG_PGTABLE_LEVELS > 2 + .if CONFIG_PGTABLE_LEVELS > 2 + pmd .req x13 pte_to_phys cur_pmdp, pud - add end_pmdp, cur_pmdp, #(PTRS_PER_PMD * 8) -do_pmd: __idmap_kpti_get_pgtable_ent pmd - tbnz pmd, #1, walk_ptes -next_pmd: - __idmap_kpti_put_pgtable_ent_ng pmd -skip_pmd: - add cur_pmdp, cur_pmdp, #8 - cmp cur_pmdp, end_pmdp - b.ne do_pmd - b next_pud - .else /* CONFIG_PGTABLE_LEVELS <= 2 */ - mov pmd, pud - b walk_ptes -next_pmd: - b next_pud + kpti_mk_tbl_ng pmd, PTRS_PER_PMD + b .Lnext_pud + .else /* CONFIG_PGTABLE_LEVELS <= 2 */ + pmd .req pgd + .set .Lnext_pmd, .Lnext_pgd .endif +.Lderef_pmd: /* PTE */ -walk_ptes: pte_to_phys cur_ptep, pmd - add end_ptep, cur_ptep, #(PTRS_PER_PTE * 8) -do_pte: __idmap_kpti_get_pgtable_ent pte - __idmap_kpti_put_pgtable_ent_ng pte -skip_pte: - add cur_ptep, cur_ptep, #8 - cmp cur_ptep, end_ptep - b.ne do_pte - b next_pmd + kpti_mk_tbl_ng pte, PTRS_PER_PTE + b .Lnext_pmd .unreq cpu .unreq num_cpus From patchwork Thu Jun 9 17:43:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 12875871 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 03248C433EF for ; Thu, 9 Jun 2022 17:45:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=mV45fJTKda6bku+kQcWl/5nTlbDKSgmWBkpm3oaTepI=; b=Zrgb5TTly+4EuY 7kzHdUNyWkJZyCRDDUN9cWAfhYn1ZmIw2fkCPCC7diFsxCReWzT48QGowy+Pk1RlOL/eXBCcNejFg 0s2pH6+z7SRvEeUIRDVLrzAwyYQeJNDF3que69Fg8VbxOLa5hw2SG2PwCuxBG6RFbIv7n2rseqID/ vhDUek76sOiK6r1JbRbJx76tprGfeGWltCl8DuH/ZH2zBVSogitW8tBkeBUbE+ICHrz8gNZDCZ1+R LzDxJ8fs1885rQyPKtYxBWX1PmL7IZfISjEdTrlGuYaSTs1jtlHaoyeZSDs0XPUB8VJcP53jPRXt7 vroMifPQSr6WMbzKtjhA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nzMCI-003GZQ-Dl; Thu, 09 Jun 2022 17:43:54 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nzMBy-003GQV-GP for linux-arm-kernel@lists.infradead.org; Thu, 09 Jun 2022 17:43:36 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 012EC61C50; Thu, 9 Jun 2022 17:43:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D46CDC341C4; Thu, 9 Jun 2022 17:43:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1654796613; bh=yuBCgQ7JVzg9JoSS3VEGVSYEqno0ekbrEjhJigOcO0w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GExIikjNF+cQ3XYNYxPIjuz4aiKI90ad79/VPwDSmuokre1Tc7DweAjyc8k1SfbfH RGFTQL3CvEUpwbtStd3jAx6N4XIWBr3Zx7w5zzyuTHp5ummZExpQOQ0r6oZE6HmJWf 13qR8lk0IPTanJvsP1BrGJVurZpWO9FenJ0megCY//fNUljXpAqddiNjHBuzrLS1Iq Hd+tyI5UZ7caBfrv7tuHHtTF0y+dP2FqcC11g0sZoaPazfJOQRap6eTqbX59gglgHU LoTP1pUuU4u2Q140ltxqHRgbYSiVsVeosnYqfxUT1HKiTRjBYO6nt3Ir+b2Z/IYPJt JkFSPlCvvfzgw== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , will@kernel.org, maz@kernel.org, mark.rutland@arm.com, catalin.marinas@arm.com, keescook@chromium.org Subject: [PATCH v5 2/2] arm64: mm: install KPTI nG mappings with MMU enabled Date: Thu, 9 Jun 2022 19:43:20 +0200 Message-Id: <20220609174320.4035379-3-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220609174320.4035379-1-ardb@kernel.org> References: <20220609174320.4035379-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=10959; h=from:subject; bh=yuBCgQ7JVzg9JoSS3VEGVSYEqno0ekbrEjhJigOcO0w=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBiojE2A/rJKVziMXVaQKp4hvdFO2avQemgsR72jfri 6gxv/+CJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYqIxNgAKCRDDTyI5ktmPJDwlC/ 9CMnxOuCnbQU0GjIGhww/Y73J1MwB1Yf+BmsoCaKJ8cNDJZm/8m4LWrW1giWGQaORcwqPnkV4l+5PJ etr5vFpZSelHXRc530mh4qyKU9tty1e+1g32Iu7hDU984Kaf5tPcDo62RsFtL5I6uhwhL96qza0u9J MYOhrdN5zmESsGHK8hvisaFuo1hbLGCUPvh60szsaJdB42rXLwcQRCs88Ds16OEog+65NRQ1HXUSpF eI6QCewudPfvZUJhKY4JpE9E3KFcb+q4LHxrfaVF0tu4KhMZZz/LhPktBJkq0iuxKYQdUeDvn+uOPI Gv2Aaby5D/EhGFGXdCU1vFDK+FkthWzhS/bH6ocHecKJ6Un7b9xQGnOI576xXR8CDCfHdi538p/tqr UJ3otyq2kzwHs+BZdrqY9KfngHzgWl6ya+Sgds3UOsdyMNf4WLEniPwcNXkKx73WglhmHHUzByoB21 8/OCda6URiWe+2RFoPyVbmYU9LZ7Gu4Wv8TRJGRQSL0d8= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220609_104334_697750_B201E4B5 X-CRM114-Status: GOOD ( 29.80 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In cases where we unmap the kernel while running in user space, we rely on ASIDs to distinguish the minimal trampoline from the full kernel mapping, and this means we must use non-global attributes for those mappings, to ensure they are scoped by ASID and will not hit in the TLB inadvertently. We only do this when needed, as this is generally more costly in terms of TLB pressure, and so we boot without these non-global attributes, and apply them to all existing kernel mappings once all CPUs are up and we know whether or not the non-global attributes are needed. At this point, we cannot simply unmap and remap the entire address space, so we have to update all existing block and page descriptors in place. Currently, we go through a lot of trouble to perform these updates with the MMU and caches off, to avoid violating break before make (BBM) rules imposed by the architecture. Since we make changes to page tables that are not covered by the ID map, we gain access to those descriptors by disabling translations altogether. This means that the stores to memory are issued with device attributes, and require extra care in terms of coherency, which is costly. We also rely on the ID map to access a shared flag, which requires the ID map to be executable and writable at the same time, which is another thing we'd prefer to avoid. So let's switch to an approach where we replace the kernel mapping with a minimal mapping of a few pages that can be used for a minimal, ad-hoc fixmap that we can use to map each page table in turn as we traverse the hierarchy. Signed-off-by: Ard Biesheuvel --- arch/arm64/kernel/cpufeature.c | 54 ++++++++++++- arch/arm64/mm/mmu.c | 7 ++ arch/arm64/mm/proc.S | 81 +++++++++++++------- 3 files changed, 113 insertions(+), 29 deletions(-) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 42ea2bd856c6..c2a64c9e451e 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1645,14 +1645,34 @@ static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry, } #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 +#define KPTI_NG_TEMP_VA (-(1UL << PMD_SHIFT)) + +extern +void create_kpti_ng_temp_pgd(pgd_t *pgdir, phys_addr_t phys, unsigned long virt, + phys_addr_t size, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), int flags); + +static phys_addr_t kpti_ng_temp_alloc; + +static phys_addr_t kpti_ng_pgd_alloc(int shift) +{ + kpti_ng_temp_alloc -= PAGE_SIZE; + return kpti_ng_temp_alloc; +} + static void __nocfi kpti_install_ng_mappings(const struct arm64_cpu_capabilities *__unused) { - typedef void (kpti_remap_fn)(int, int, phys_addr_t); + typedef void (kpti_remap_fn)(int, int, phys_addr_t, unsigned long); extern kpti_remap_fn idmap_kpti_install_ng_mappings; kpti_remap_fn *remap_fn; int cpu = smp_processor_id(); + int levels = CONFIG_PGTABLE_LEVELS; + int order = order_base_2(levels); + u64 kpti_ng_temp_pgd_pa = 0; + pgd_t *kpti_ng_temp_pgd; + u64 alloc = 0; if (__this_cpu_read(this_cpu_vector) == vectors) { const char *v = arm64_get_bp_hardening_vector(EL1_VECTOR_KPTI); @@ -1670,12 +1690,40 @@ kpti_install_ng_mappings(const struct arm64_cpu_capabilities *__unused) remap_fn = (void *)__pa_symbol(function_nocfi(idmap_kpti_install_ng_mappings)); + if (!cpu) { + alloc = __get_free_pages(GFP_ATOMIC | __GFP_ZERO, order); + kpti_ng_temp_pgd = (pgd_t *)(alloc + (levels - 1) * PAGE_SIZE); + kpti_ng_temp_alloc = kpti_ng_temp_pgd_pa = __pa(kpti_ng_temp_pgd); + + // + // Create a minimal page table hierarchy that permits us to map + // the swapper page tables temporarily as we traverse them. + // + // The physical pages are laid out as follows: + // + // +--------+-/-------+-/------ +-\\--------+ + // : PTE[] : | PMD[] : | PUD[] : || PGD[] : + // +--------+-\-------+-\------ +-//--------+ + // ^ + // The first page is mapped into this hierarchy at a PMD_SHIFT + // aligned virtual address, so that we can manipulate the PTE + // level entries while the mapping is active. The first entry + // covers the PTE[] page itself, the remaining entries are free + // to be used as a ad-hoc fixmap. + // + create_kpti_ng_temp_pgd(kpti_ng_temp_pgd, __pa(alloc), + KPTI_NG_TEMP_VA, PAGE_SIZE, PAGE_KERNEL, + kpti_ng_pgd_alloc, 0); + } + cpu_install_idmap(); - remap_fn(cpu, num_online_cpus(), __pa_symbol(swapper_pg_dir)); + remap_fn(cpu, num_online_cpus(), kpti_ng_temp_pgd_pa, KPTI_NG_TEMP_VA); cpu_uninstall_idmap(); - if (!cpu) + if (!cpu) { + free_pages(alloc, order); arm64_use_ng_mappings = true; + } } #else static void diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index be4d6c3f5692..c5563ff990da 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -388,6 +388,13 @@ static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, } while (pgdp++, addr = next, addr != end); } +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 +extern __alias(__create_pgd_mapping) +void create_kpti_ng_temp_pgd(pgd_t *pgdir, phys_addr_t phys, unsigned long virt, + phys_addr_t size, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), int flags); +#endif + static phys_addr_t __pgd_pgtable_alloc(int shift) { void *ptr = (void *)__get_free_page(GFP_PGTABLE_KERNEL); diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 660887152dba..972ce8d7f2c5 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -200,20 +201,19 @@ SYM_FUNC_END(idmap_cpu_replace_ttbr1) .popsection #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 + +#define KPTI_NG_PTE_FLAGS (PTE_ATTRINDX(MT_NORMAL) | SWAPPER_PTE_FLAGS) + .pushsection ".idmap.text", "awx" .macro kpti_mk_tbl_ng, type, num_entries add end_\type\()p, cur_\type\()p, #\num_entries * 8 .Ldo_\type: - dc cvac, cur_\type\()p // Ensure any existing dirty - dmb sy // lines are written back before - ldr \type, [cur_\type\()p] // loading the entry + ldr \type, [cur_\type\()p] // Load the entry tbz \type, #0, .Lnext_\type // Skip invalid and tbnz \type, #11, .Lnext_\type // non-global entries orr \type, \type, #PTE_NG // Same bit for blocks and pages - str \type, [cur_\type\()p] // Update the entry and ensure - dmb sy // that it is visible to all - dc civac, cur_\()\type\()p // CPUs. + str \type, [cur_\type\()p] // Update the entry .ifnc \type, pte tbnz \type, #1, .Lderef_\type .endif @@ -223,8 +223,29 @@ SYM_FUNC_END(idmap_cpu_replace_ttbr1) b.ne .Ldo_\type .endm + /* + * Dereference the current table entry and map it into the temporary + * fixmap slot associated with the current level. + */ + .macro kpti_map_pgtbl, type, level + str xzr, [temp_pte, #8 * (\level + 1)] // break before make + dsb nshst + add pte, temp_pte, #PAGE_SIZE * (\level + 1) + lsr pte, pte, #12 + tlbi vaae1, pte + dsb nsh + isb + + phys_to_pte pte, cur_\type\()p + add cur_\type\()p, temp_pte, #PAGE_SIZE * (\level + 1) + orr pte, pte, pte_flags + str pte, [temp_pte, #8 * (\level + 1)] + dsb nshst + .endm + /* - * void __kpti_install_ng_mappings(int cpu, int num_cpus, phys_addr_t swapper) + * void __kpti_install_ng_mappings(int cpu, int num_secondaries, phys_addr_t temp_pgd, + * unsigned long temp_pte_va) * * Called exactly once from stop_machine context by each CPU found during boot. */ @@ -232,8 +253,10 @@ __idmap_kpti_flag: .long 1 SYM_FUNC_START(idmap_kpti_install_ng_mappings) cpu .req w0 + temp_pte .req x0 num_cpus .req w1 - swapper_pa .req x2 + pte_flags .req x1 + temp_pgd_phys .req x2 swapper_ttb .req x3 flag_ptr .req x4 cur_pgdp .req x5 @@ -246,9 +269,10 @@ SYM_FUNC_START(idmap_kpti_install_ng_mappings) cur_ptep .req x14 end_ptep .req x15 pte .req x16 + valid .req x17 + mov x5, x3 // preserve temp_pte arg mrs swapper_ttb, ttbr1_el1 - restore_ttbr1 swapper_ttb adr flag_ptr, __idmap_kpti_flag cbnz cpu, __idmap_kpti_secondary @@ -260,28 +284,28 @@ SYM_FUNC_START(idmap_kpti_install_ng_mappings) eor w17, w17, num_cpus cbnz w17, 1b - /* We need to walk swapper, so turn off the MMU. */ - pre_disable_mmu_workaround - mrs x17, sctlr_el1 - bic x17, x17, #SCTLR_ELx_M - msr sctlr_el1, x17 + /* Switch to the temporary page tables on this CPU only */ + __idmap_cpu_set_reserved_ttbr1 x8, x9 + offset_ttbr1 temp_pgd_phys, x8 + msr ttbr1_el1, temp_pgd_phys isb + mov temp_pte, x5 + mov pte_flags, #KPTI_NG_PTE_FLAGS + /* Everybody is enjoying the idmap, so we can rewrite swapper. */ /* PGD */ - mov cur_pgdp, swapper_pa + adrp cur_pgdp, swapper_pg_dir + kpti_map_pgtbl pgd, 0 kpti_mk_tbl_ng pgd, PTRS_PER_PGD - /* Publish the updated tables and nuke all the TLBs */ - dsb sy - tlbi vmalle1is - dsb ish - isb + /* Ensure all the updated entries are visible to secondary CPUs */ + dsb ishst - /* We're done: fire up the MMU again */ - mrs x17, sctlr_el1 - orr x17, x17, #SCTLR_ELx_M - set_sctlr_el1 x17 + /* We're done: fire up swapper_pg_dir again */ + __idmap_cpu_set_reserved_ttbr1 x8, x9 + msr ttbr1_el1, swapper_ttb + isb /* Set the flag to zero to indicate that we're all done */ str wzr, [flag_ptr] @@ -292,6 +316,7 @@ SYM_FUNC_START(idmap_kpti_install_ng_mappings) .if CONFIG_PGTABLE_LEVELS > 3 pud .req x10 pte_to_phys cur_pudp, pgd + kpti_map_pgtbl pud, 1 kpti_mk_tbl_ng pud, PTRS_PER_PUD b .Lnext_pgd .else /* CONFIG_PGTABLE_LEVELS <= 3 */ @@ -304,6 +329,7 @@ SYM_FUNC_START(idmap_kpti_install_ng_mappings) .if CONFIG_PGTABLE_LEVELS > 2 pmd .req x13 pte_to_phys cur_pmdp, pud + kpti_map_pgtbl pmd, 2 kpti_mk_tbl_ng pmd, PTRS_PER_PMD b .Lnext_pud .else /* CONFIG_PGTABLE_LEVELS <= 2 */ @@ -314,12 +340,15 @@ SYM_FUNC_START(idmap_kpti_install_ng_mappings) .Lderef_pmd: /* PTE */ pte_to_phys cur_ptep, pmd + kpti_map_pgtbl pte, 3 kpti_mk_tbl_ng pte, PTRS_PER_PTE b .Lnext_pmd .unreq cpu + .unreq temp_pte .unreq num_cpus - .unreq swapper_pa + .unreq pte_flags + .unreq temp_pgd_phys .unreq cur_pgdp .unreq end_pgdp .unreq pgd @@ -332,6 +361,7 @@ SYM_FUNC_START(idmap_kpti_install_ng_mappings) .unreq cur_ptep .unreq end_ptep .unreq pte + .unreq valid /* Secondary CPUs end up here */ __idmap_kpti_secondary: @@ -351,7 +381,6 @@ __idmap_kpti_secondary: cbnz w16, 1b /* All done, act like nothing happened */ - offset_ttbr1 swapper_ttb, x16 msr ttbr1_el1, swapper_ttb isb ret