From patchwork Thu Dec 19 16:44:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 13915309 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9DD23E77184 for ; Thu, 19 Dec 2024 16:48:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-Type: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=aEX8PRpkFrAcxu9PrNYJZGjzYKd+o1ls5Xhqvv0zRi8=; b=wBK5WbzPBOh1SwViHnLlLAtCVc L22058Of2YlF/6GG8Rdtu6oSeOSF4/7HaSiwm2U3U7gmdJnNv+wvuAtmFGps4UJCKw6SULmYyHTTX UE/goOcT24a1wKbjRhTj77kyyF8pcbH+huvU1DkTJHaoU/mvirrUXhhhHoNf0hAzQB6HElqArVjbo wQ4V0rLqr2hS83hyGpCsmG7Z/olTtwGg40wnYJY1+taDOZ7ngAssysj+dmaXjGIoUEbX3hXdfoyjD Uu0aI3NguoxGm06Tk8nhWzSYaC4dj2rIDI6YmbsJcq7t+PuND8Z071RGli7WsfyalwXCdbe9tAkfl dtUqwUdA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tOJgl-00000002Ppy-2KIf; Thu, 19 Dec 2024 16:47:51 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tOJfB-00000002PF4-3PMO; Thu, 19 Dec 2024 16:46:15 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3235D1480; Thu, 19 Dec 2024 08:46:39 -0800 (PST) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2E9D63F58B; Thu, 19 Dec 2024 08:46:07 -0800 (PST) From: Kevin Brodsky To: linux-mm@kvack.org Cc: Kevin Brodsky , Andrew Morton , Catalin Marinas , Dave Hansen , Linus Walleij , Andy Lutomirski , Peter Zijlstra , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , linux-alpha@vger.kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-kernel@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-parisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-um@lists.infradead.org, loongarch@lists.linux.dev, x86@kernel.org Subject: [PATCH 00/10] Account page tables at all levels Date: Thu, 19 Dec 2024 16:44:15 +0000 Message-ID: <20241219164425.2277022-1-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241219_084613_940621_021F15A4 X-CRM114-Status: GOOD ( 15.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We currently have a pair of ctor/dtor calls for lower page table levels, up to PUD. At PTE and PMD level, these handle split locks, if supported. Additionally, the helpers ensure correct accounting of page table pages to the corresponding process. This series takes that principle to its logical conclusion: account all page table pages, at all levels and on all architectures (see caveat below), through suitable ctor/dtor calls. This means concretely: * Ensuring that the existing pagetable_{pte,pmd,pud}_[cd]tor are called on all architectures. * Introduce pagetable_{p4d,pgd}_[cd]tor and call them at P4D/PGD level. The primary motivation for this series is not page accounting, though. P4D/PGD-level pages represent a tiny proportion of the memory used by a process. Rather, the appeal comes from the introduction of a single, generic place where construction/destruction hooks can be called for all page table pages at all levels. This will come in handy for protecting page tables using kpkeys [1]. Peter Zijlstra suggested this approach [2] to avoid handling this in arch code. With this series, __pagetable_ctor() and __pagetable_dtor() (introduced in patch 1) should be called when page tables are allocated/freed at any level on any architecture. Note however that only P*D that consist of one or more regular pages are handled. This excludes: * All P*D allocated from a kmem_cache (or kmalloc). * P*D that are not allocated via GFP (only an issue on sparc). The table at the end of this email gives more details for each architecture. Patches in details: * Patch 1 factors out the common implementation of all pagetable_*_[cd]tor. * Patch 2-4: PMD/PUD; add missing calls to pagetable_{pmd,pud}_[cd]tor on various architectures. * Patch 5-7: P4D; move most arch to using generic alloc/free functions at P4D level, and then have them call pagetable_p4d_[cd]tor. * Patch 8-10: PGD; same principle at PGD level. The patches were build-tested on all architectures (thanks Linus Walleij for triggering the LKP CI for me!), and boot-tested on arm64 and x86_64. - Kevin [1] https://lore.kernel.org/linux-hardening/20241206101110.1646108-1-kevin.brodsky@arm.com/ [2] https://lore.kernel.org/linux-hardening/20241210122355.GN8562@noisy.programming.kicks-ass.net/ Acked-by: Dave Hansen --- Overview of the situation on all arch after this series is applied: +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | arch | #include | Complete ctor/dtor | ctor/dtor | Notes | | | | calls up to p4d level | at pgd level | | +===============+=========================+=======================+==============+====================================+ | alpha | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | arc | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | arm | Y | Y | Y/N | kmalloc at pgd level if LPAE | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | arm64 | Y | Y | Y/N | kmem_cache if pgd not page-sized | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | csky | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | hexagon | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | loongarch | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | m68k (Sun3) | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | m68k (others) | N | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | microblaze | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | mips | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | nios2 | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | openrisc | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | parisc | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | powerpc | N | Y/N | N | kmem_cache at: | | | | | | - pgd level | | | | | | - pud level in 64-bit | | | | | | - pmd level in 64-bit on !book3s | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | riscv | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | s390 | N | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | sh | Y | N | N | kmem_cache at pmd/pgd level | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | sparc | N | N | N | 32-bit: special memory | | | | | | 64-bit: kmem_cache above pte level | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | um | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | x86 | Y | Y | Y/N | kmem_cache at pgd level if PAE | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | xtensa | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ Cc: Andrew Morton Cc: Catalin Marinas Cc: Dave Hansen Cc: Linus Walleij Cc: Andy Lutomirski Cc: Peter Zijlstra Cc: "Mike Rapoport (IBM)" Cc: Ryan Roberts Cc: Thomas Gleixner Cc: Will Deacon Cc: Matthew Wilcox Cc: linux-alpha@vger.kernel.org Cc: linux-arch@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org Cc: linux-csky@vger.kernel.org Cc: linux-hexagon@vger.kernel.org Cc: linux-kernel@vger.kernel.org Cc: linux-m68k@lists.linux-m68k.org Cc: linux-mips@vger.kernel.org Cc: linux-openrisc@vger.kernel.org Cc: linux-parisc@vger.kernel.org Cc: linux-riscv@lists.infradead.org Cc: linux-s390@vger.kernel.org Cc: linux-snps-arc@lists.infradead.org Cc: linux-um@lists.infradead.org Cc: loongarch@lists.linux.dev Cc: x86@kernel.org --- Kevin Brodsky (10): mm: Move common parts of pagetable_*_[cd]tor to helpers parisc: mm: Ensure pagetable_pmd_[cd]tor are called m68k: mm: Add calls to pagetable_pmd_[cd]tor s390/mm: Add calls to pagetable_pud_[cd]tor riscv: mm: Skip pgtable level check in {pud,p4d}_alloc_one asm-generic: pgalloc: Provide generic p4d_{alloc_one,free} mm: Introduce ctor/dtor at P4D level ARM: mm: Rename PGD helpers asm-generic: pgalloc: Provide generic __pgd_{alloc,free} mm: Introduce ctor/dtor at PGD level arch/alpha/mm/init.c | 2 +- arch/arc/include/asm/pgalloc.h | 9 +-- arch/arm/mm/pgd.c | 16 +++-- arch/arm64/include/asm/pgalloc.h | 17 ------ arch/arm64/mm/pgd.c | 4 +- arch/csky/include/asm/pgalloc.h | 2 +- arch/hexagon/include/asm/pgalloc.h | 2 +- arch/loongarch/mm/pgtable.c | 7 +-- arch/m68k/include/asm/mcf_pgalloc.h | 2 + arch/m68k/include/asm/motorola_pgalloc.h | 6 +- arch/m68k/include/asm/sun3_pgalloc.h | 2 +- arch/m68k/mm/motorola.c | 31 ++++++++-- arch/microblaze/include/asm/pgalloc.h | 7 +-- arch/mips/include/asm/pgalloc.h | 6 -- arch/mips/mm/pgtable.c | 8 +-- arch/nios2/mm/pgtable.c | 3 +- arch/openrisc/include/asm/pgalloc.h | 6 +- arch/parisc/include/asm/pgalloc.h | 39 ++++-------- arch/riscv/include/asm/pgalloc.h | 46 ++------------ arch/s390/include/asm/pgalloc.h | 33 +++++++--- arch/um/kernel/mem.c | 7 +-- arch/x86/include/asm/pgalloc.h | 18 ------ arch/x86/mm/pgtable.c | 27 +++++---- arch/xtensa/include/asm/pgalloc.h | 2 +- include/asm-generic/pgalloc.h | 76 +++++++++++++++++++++++- include/linux/mm.h | 64 +++++++++++++------- 26 files changed, 234 insertions(+), 208 deletions(-) base-commit: 78d4f34e2115b517bcbfe7ec0d018bbbb6f9b0b8