From patchwork Thu Dec 19 16:44:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Brodsky X-Patchwork-Id: 13915287 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35116E77184 for ; Thu, 19 Dec 2024 16:46:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AA3C86B007B; Thu, 19 Dec 2024 11:46:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A2C666B0082; Thu, 19 Dec 2024 11:46:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8A5956B0083; Thu, 19 Dec 2024 11:46:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 68CDB6B007B for ; Thu, 19 Dec 2024 11:46:14 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 12C19121797 for ; Thu, 19 Dec 2024 16:46:14 +0000 (UTC) X-FDA: 82912285260.22.FC2AD5F Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf03.hostedemail.com (Postfix) with ESMTP id 3087A2000C for ; Thu, 19 Dec 2024 16:45:57 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=none; spf=pass (imf03.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734626750; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references; bh=aEX8PRpkFrAcxu9PrNYJZGjzYKd+o1ls5Xhqvv0zRi8=; b=rjbvrdtGJTrm+9CSLxFKidYYMBPFRzJtOYI2GujE+AgJPjWL5/GJGvwb4EgWY+Ju+BiSwF DIdVylUKmj/5DOOU9D4vjmL7280kOyLx2tl0ga97pwMQYDPnhOnTv7PgYUTvhHnK3Lb+qC uTHdMljqEZAA0givuBPPNDZeWVupe00= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734626750; a=rsa-sha256; cv=none; b=wpe1eOJ5oF1tKWB8noOSfbFWyH2M9Y/G+drgI3/dg1cGksr0V9diWSdTK4oy3xSgLXf8/+ 7RF7W6JHk6xpS6MuvRWXzVOi2WTQcWZE5d7xxgDDXy8PFG3NqizolkqpjEZ0re7c1yA+TG /PFoZN+MBjXKyrQG3sZFFsJIjCwHPtU= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=none; spf=pass (imf03.hostedemail.com: domain of kevin.brodsky@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=kevin.brodsky@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3235D1480; Thu, 19 Dec 2024 08:46:39 -0800 (PST) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2E9D63F58B; Thu, 19 Dec 2024 08:46:07 -0800 (PST) From: Kevin Brodsky To: linux-mm@kvack.org Cc: Kevin Brodsky , Andrew Morton , Catalin Marinas , Dave Hansen , Linus Walleij , Andy Lutomirski , Peter Zijlstra , "Mike Rapoport (IBM)" , Ryan Roberts , Thomas Gleixner , Will Deacon , Matthew Wilcox , linux-alpha@vger.kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-kernel@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-parisc@vger.kernel.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-um@lists.infradead.org, loongarch@lists.linux.dev, x86@kernel.org Subject: [PATCH 00/10] Account page tables at all levels Date: Thu, 19 Dec 2024 16:44:15 +0000 Message-ID: <20241219164425.2277022-1-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 3087A2000C X-Stat-Signature: uqkkg9xozi94sdj84yt185fedkcdyjj1 X-Rspam-User: X-HE-Tag: 1734626757-928542 X-HE-Meta: U2FsdGVkX19xzMgo39Kf9AGkosmDv3s4huqHgGZ4EljB03CN8C4dgGzFin/uGhqEVMKDd+6yBP2FQiWRJ2sZmMrMpBKWB8zeX1JTORt7NNC7iR7d4AlbNiUujhGeNcFW6/0E9yFZhJ7Ql9MNuMZe5dXizdc6vlj889EdON4jztzL3MSBmgpZdSJVy+CNkucVDuB6LovENLXreSTzg1w12V66cIvuON5kx0W2I4defcRDeaRb0mmLlpnzRI+u2VLmei37nCoS9yCwfDN5cRTm0ieJ1l9S+gT79yZ4fdvLWSX2aOPUySomuFa7AD8ntT/gRwypiyAforsDGbVUzK6pPbq6Hyj3Tov+w+tjwJFVdDA7Lgq2A3Ivg0gYpm684DSBgswBE79XHTavAfg0Nh3avEH7YTI7T44yNVUE80JjYOS0xwwCuIhyqJXqLKwkPUFLhWr0eBhVrvWpYclzLNehwR8cw1l61t7f4cIktmSwk8x2BzO5pz1ft1P8k6nCf4rmMmRXSmSQvlmogwT1xVkHlzH4Rn3l6wfMYKQbAkGxUaWW0ONjAUmLBYOwwRGC22oDobM1UOREEagUObrYWQ0tgrm9kQ+loXeeE95bxOzM6Byda41lnjAOfDcqMvKH9bHuIqAbYIDhwcFLsIeMfrVtunqAwzGRazlseUG9Ou/herxIWnJPov+8iUat/ibESMnCF9B9ZZsBYA7AvDzOLPGRozlRmK9xjekQBoY2ws4UdZt7FbQfsW5YBDuV6x+//xaygVPKX/dvtJ8mNVsdmCz4vkxK0BgYqE6WQYPX4b2/Gd2GE+UPW4MaNTJk85N6ctkEiIFph7ovqJ9Ckp3wscq3mgwNWNodI1YXWvOeKknl4Q8GSHZpbV5hBKxkGraTd1mPN81XC6KrtB/UMk98JXeQazQShrLTW0iFeZ/BuF9N14KAN4M6Xo1eGfh+JmoLO6FtDrm2H+N0zlBaZEm2EXl /pCRSNbW 6AAq0IM0TCcL4IEIhOf3KDa1vIOHEwAvjO9m2Qx9oO3e38JXoIwcvlEEi8c/AjTZxYXQwY7WXCWUgIGsdq1kEmYAv3JnWyAfLx4qay1JOm8qZUkb5HwgVSWbK2pNiKaf9xaCufXLai5YdszrXV0MtvLDhTssrfoaz3QKKR8+zmJbZgBQ6n7U03J8atKVFn3ikMFtr4KfVg2xcdLwHofM2xUIg4Zz7+HtR/JNbcGeXs4nBQTMgWfGkp51jjNLHbjxq10H03paUrrEHP9M6QWVHczbd0D8sXaK20Jxn0idVtNtk1FrGGyk5QXhnmVbTPJkLLqhbuheCQZnGcokHzpzvMND4Pd4GeF7JV48rDFq1NqGTz6azwcCRsF1BzC885HEEq84ER1ufynDWXUbOcLaSeqmv0Sz7JjKTAwpR+T3uc+WKYtRp2qKXiTmYAqumWOljozADukfZUI2o5CN38U9MMp6VZXG3PJ/wJMtlQT5nzCdb0N6/r1GVIqt5hAH1OjUR+eahh9SNsd6X31+2EaV/uuOwVpmxWWb4wJk4qfuUq/mOwMCPG5sJAR5dQoszOPgbj8/mPZefUmjjXb3/lVmW6WrVBQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We currently have a pair of ctor/dtor calls for lower page table levels, up to PUD. At PTE and PMD level, these handle split locks, if supported. Additionally, the helpers ensure correct accounting of page table pages to the corresponding process. This series takes that principle to its logical conclusion: account all page table pages, at all levels and on all architectures (see caveat below), through suitable ctor/dtor calls. This means concretely: * Ensuring that the existing pagetable_{pte,pmd,pud}_[cd]tor are called on all architectures. * Introduce pagetable_{p4d,pgd}_[cd]tor and call them at P4D/PGD level. The primary motivation for this series is not page accounting, though. P4D/PGD-level pages represent a tiny proportion of the memory used by a process. Rather, the appeal comes from the introduction of a single, generic place where construction/destruction hooks can be called for all page table pages at all levels. This will come in handy for protecting page tables using kpkeys [1]. Peter Zijlstra suggested this approach [2] to avoid handling this in arch code. With this series, __pagetable_ctor() and __pagetable_dtor() (introduced in patch 1) should be called when page tables are allocated/freed at any level on any architecture. Note however that only P*D that consist of one or more regular pages are handled. This excludes: * All P*D allocated from a kmem_cache (or kmalloc). * P*D that are not allocated via GFP (only an issue on sparc). The table at the end of this email gives more details for each architecture. Patches in details: * Patch 1 factors out the common implementation of all pagetable_*_[cd]tor. * Patch 2-4: PMD/PUD; add missing calls to pagetable_{pmd,pud}_[cd]tor on various architectures. * Patch 5-7: P4D; move most arch to using generic alloc/free functions at P4D level, and then have them call pagetable_p4d_[cd]tor. * Patch 8-10: PGD; same principle at PGD level. The patches were build-tested on all architectures (thanks Linus Walleij for triggering the LKP CI for me!), and boot-tested on arm64 and x86_64. - Kevin [1] https://lore.kernel.org/linux-hardening/20241206101110.1646108-1-kevin.brodsky@arm.com/ [2] https://lore.kernel.org/linux-hardening/20241210122355.GN8562@noisy.programming.kicks-ass.net/ Acked-by: Dave Hansen --- Overview of the situation on all arch after this series is applied: +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | arch | #include | Complete ctor/dtor | ctor/dtor | Notes | | | | calls up to p4d level | at pgd level | | +===============+=========================+=======================+==============+====================================+ | alpha | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | arc | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | arm | Y | Y | Y/N | kmalloc at pgd level if LPAE | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | arm64 | Y | Y | Y/N | kmem_cache if pgd not page-sized | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | csky | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | hexagon | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | loongarch | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | m68k (Sun3) | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | m68k (others) | N | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | microblaze | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | mips | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | nios2 | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | openrisc | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | parisc | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | powerpc | N | Y/N | N | kmem_cache at: | | | | | | - pgd level | | | | | | - pud level in 64-bit | | | | | | - pmd level in 64-bit on !book3s | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | riscv | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | s390 | N | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | sh | Y | N | N | kmem_cache at pmd/pgd level | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | sparc | N | N | N | 32-bit: special memory | | | | | | 64-bit: kmem_cache above pte level | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | um | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | x86 | Y | Y | Y/N | kmem_cache at pgd level if PAE | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ | xtensa | Y | Y | Y | | +---------------+-------------------------+-----------------------+--------------+------------------------------------+ Cc: Andrew Morton Cc: Catalin Marinas Cc: Dave Hansen Cc: Linus Walleij Cc: Andy Lutomirski Cc: Peter Zijlstra Cc: "Mike Rapoport (IBM)" Cc: Ryan Roberts Cc: Thomas Gleixner Cc: Will Deacon Cc: Matthew Wilcox Cc: linux-alpha@vger.kernel.org Cc: linux-arch@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org Cc: linux-csky@vger.kernel.org Cc: linux-hexagon@vger.kernel.org Cc: linux-kernel@vger.kernel.org Cc: linux-m68k@lists.linux-m68k.org Cc: linux-mips@vger.kernel.org Cc: linux-openrisc@vger.kernel.org Cc: linux-parisc@vger.kernel.org Cc: linux-riscv@lists.infradead.org Cc: linux-s390@vger.kernel.org Cc: linux-snps-arc@lists.infradead.org Cc: linux-um@lists.infradead.org Cc: loongarch@lists.linux.dev Cc: x86@kernel.org --- Kevin Brodsky (10): mm: Move common parts of pagetable_*_[cd]tor to helpers parisc: mm: Ensure pagetable_pmd_[cd]tor are called m68k: mm: Add calls to pagetable_pmd_[cd]tor s390/mm: Add calls to pagetable_pud_[cd]tor riscv: mm: Skip pgtable level check in {pud,p4d}_alloc_one asm-generic: pgalloc: Provide generic p4d_{alloc_one,free} mm: Introduce ctor/dtor at P4D level ARM: mm: Rename PGD helpers asm-generic: pgalloc: Provide generic __pgd_{alloc,free} mm: Introduce ctor/dtor at PGD level arch/alpha/mm/init.c | 2 +- arch/arc/include/asm/pgalloc.h | 9 +-- arch/arm/mm/pgd.c | 16 +++-- arch/arm64/include/asm/pgalloc.h | 17 ------ arch/arm64/mm/pgd.c | 4 +- arch/csky/include/asm/pgalloc.h | 2 +- arch/hexagon/include/asm/pgalloc.h | 2 +- arch/loongarch/mm/pgtable.c | 7 +-- arch/m68k/include/asm/mcf_pgalloc.h | 2 + arch/m68k/include/asm/motorola_pgalloc.h | 6 +- arch/m68k/include/asm/sun3_pgalloc.h | 2 +- arch/m68k/mm/motorola.c | 31 ++++++++-- arch/microblaze/include/asm/pgalloc.h | 7 +-- arch/mips/include/asm/pgalloc.h | 6 -- arch/mips/mm/pgtable.c | 8 +-- arch/nios2/mm/pgtable.c | 3 +- arch/openrisc/include/asm/pgalloc.h | 6 +- arch/parisc/include/asm/pgalloc.h | 39 ++++-------- arch/riscv/include/asm/pgalloc.h | 46 ++------------ arch/s390/include/asm/pgalloc.h | 33 +++++++--- arch/um/kernel/mem.c | 7 +-- arch/x86/include/asm/pgalloc.h | 18 ------ arch/x86/mm/pgtable.c | 27 +++++---- arch/xtensa/include/asm/pgalloc.h | 2 +- include/asm-generic/pgalloc.h | 76 +++++++++++++++++++++++- include/linux/mm.h | 64 +++++++++++++------- 26 files changed, 234 insertions(+), 208 deletions(-) base-commit: 78d4f34e2115b517bcbfe7ec0d018bbbb6f9b0b8