From patchwork Mon Feb 27 17:57:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13154035 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD31CC64ED6 for ; Mon, 27 Feb 2023 17:57:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 11CEF6B0082; Mon, 27 Feb 2023 12:57:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F2D876B0087; Mon, 27 Feb 2023 12:57:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DBD936B0088; Mon, 27 Feb 2023 12:57:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CDBCF6B0082 for ; Mon, 27 Feb 2023 12:57:48 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 713EBAAC4C for ; Mon, 27 Feb 2023 17:57:48 +0000 (UTC) X-FDA: 80513829816.22.F255621 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf22.hostedemail.com (Postfix) with ESMTP id B9F81C0017 for ; Mon, 27 Feb 2023 17:57:45 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=KGN3qiGA; dmarc=none; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677520666; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=3tdRFaZeQcW/4jPn25orVr3lG3sjhHxz5IbjVZonYE0=; b=V/GxnKeG09pdfVu9vTRAs/QvHZVN634PzVeQFwbqRaus39qbt2FctzJNEqIsdJx+iSPtH3 dAqrM2HLSTzrl+w4UICSFE27KsbtGRQOo7OR97dtmlVPVCKPFGaWChXbxeavfVsxSiBUkn p/pOghw0hncaQ84IfIIea/1LTrzr+t0= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=KGN3qiGA; dmarc=none; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677520666; a=rsa-sha256; cv=none; b=VOEIGkcTIuNsrWhJe0d0H5jatrt0RdHAkRHxyH0Uwi+QsCnzpvZUrrAPLmbGP6y8QXXyob 9Fbb/fKrvboq9OoOIQgZc6eomgZqb9ZGyerhPyvBMKZ0Or4pcYxOSnprjhJXpytamO0y56 xdK9PDlGIqsD1zf6aN+tCtIvgt5taKk= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:In-Reply-To:References; bh=3tdRFaZeQcW/4jPn25orVr3lG3sjhHxz5IbjVZonYE0=; b=KGN3qiGAedqU9G+2dxTsB5Atye +9rGfP48swyhdH/2v9S9G36W1666kQlY39X1xKpIQVB4Nf5gvlyZu4XcG2N9FFaVi1wH7MsvlHBu/ OWYeMQGgCGLIBjSNjkV2c8d++14737v1xrVKFfPH4Te5ff+1FCXMgg6qvS55m4oK6QH5xH8ffxFkX QCgPY66AqyiM6Pqspa7xWxEpp8gwq44s3xw8nZ87uG5obywzC0f7FWDMF4nZhM1AhHwN81RjmwyPq grLNAzPe4ycwmu5h7ZPwhFoQQ1os1lIZMpFr33jxdfS50zk+4eCjOylVBZ3dILo0ZCQ/lpeFp1bgT pCxhOHTg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pWhks-000IWn-UM; Mon, 27 Feb 2023 17:57:43 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org, linux-arch@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org Subject: [PATCH v2 00/30] New page table range API Date: Mon, 27 Feb 2023 17:57:11 +0000 Message-Id: <20230227175741.71216-1-willy@infradead.org> X-Mailer: git-send-email 2.37.1 MIME-Version: 1.0 X-Rspamd-Queue-Id: B9F81C0017 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: mddn8jss455875i5s7ahngp1wwq4pagr X-HE-Tag: 1677520665-663053 X-HE-Meta: U2FsdGVkX1+Fn2jwSMc/KR82z1wYWBHVuxkWb9V5VO/bxo91ut1oW0I6b2Gs76Cv9cZxPbfITuNuJAK3ayrI2FEbARPja7VJZ8J9Vc04+ZA04pvSLTnt56M1Di9K/qHSC9nPkqN02nYRJpdyGFAQMyy/sA67vfqYLE/Qwb3N4hjvCOc4GdHd0oOAXUnHBeRW/+q3b7rkU0oDx8RKpC9LsD4vJbPHgrpS7t45R9+zglTSZn7vLuPXt/59xHRKZzpmwgbC8OkCobY9NHl3giDDpyK2U9RcBn9OOpo9XRiMoHJRrrUOamX20jHNHrbnQYVJlcZ8BliENX25ivhNN0vP3Aa7n9l7o2JCRMZIsYNoXtHr4NWt5FMFWtLh/ZTGDHjIVA10Gva7CaE9//Nwhq//lD5/hGVL5SjsdFoIbI1qJE7pwmvy0ZOJD/6eI8YEE9rUi28P58SJ7eacYA3u1/+FmJBhFhYlFy0tYb3t7kjdAHu0MY3OP+SkF2/zRrnLcHWlaRnxWIeBmOcPW90cEGWzUPidUZz0dM2mmDiLtCCQxx3h75KMAP/vNcR4SqjM86P0yCDHkpPXhfrIDQBME+z5/BWDb77hIl4EKyoE6VmH8O+mVplTQ+IyssEhrRa1wT2xEaRZoKc/zwaULr7XpnjlACQPhkJx7oB++0lQDvVoKFBNn0Ld4g6KJa78cjGQvS97gzw9cnqj6OlW4Vt4BKIAvm13ujJshhzpDKi9OqsJhhPJHRUZIZlz6fd6zhI0wXf1i4cdunFUCY1a+zA9U6zqItGW080CvPKFlWZG/1tTPvgU9xrP7hBUTkHnEQV8E/pj8faXCgTogeHnQEnL3tPWtO94XBARS7KsKLQ/LZeqh8J6DkgA5NURE96KMphI5lFGSnjhnEReIkfgc9URyzHnfLCvZ1E8dS3BhKT4XOPnU1Mo8vkdfYhLAT0qdgLtmnjWNxt8uDNO3oPX4hteDLT zmptuuqJ vPsLGsBt4zb4FyqJ8RNwBOG86D1tbcKwJuTr3MnDlG+u+d/HLldPHBwTxczBvKjQAMW/QraL/tn4+b5D8eWgre4Lq153Q4dIpJNuESSHfPr1j7WuseozdgZ1GVLANp/wPdwYpkpBOVJY4MrOEdE0U4e6xMRfEhuXwegMxxN7p9xFwNi30Oylm4RYxqVr7UkWVwMN8or8Y5vWDvaKP6YCCkDLcoYRtmCBzXYBZde/PFYmx53Vg95sSVqpfMw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patchset changes the API used by the MM to set up page table entries. The four APIs are: set_ptes(mm, addr, ptep, pte, nr) update_mmu_cache_range(vma, addr, ptep, nr) flush_dcache_folio(folio) flush_icache_pages(vma, page, nr) flush_dcache_folio() isn't technically new, but no architecture implemented it, so I've done that for you. The old APIs remain around but are mostly implemented by calling the new interfaces. The new APIs are based around setting up N page table entries at once. The N entries belong to the same PMD, the same folio and the same VMA, so ptep++ is a legitimate operation, and locking is taken care of for you. Some architectures can do a better job of it than just a loop, but I have hesitated to make too deep a change to architectures I don't understand well. One thing I have changed in every architecture is that PG_arch_1 is now a per-folio bit instead of a per-page bit. This was something that would have to happen eventually, and it makes sense to do it now rather than iterate over every page involved in a cache flush and figure out if it needs to happen. The point of all this is better performance, and Fengwei Yin has measured improvement on x86. I suspect you'll see improvement on your architecture too. Try the new will-it-scale test mentioned here: https://lore.kernel.org/linux-mm/20230206140639.538867-5-fengwei.yin@intel.com/ You'll need to run it on an XFS filesystem and have CONFIG_TRANSPARENT_HUGEPAGE set. For testing, I've only run the code on x86. If an x86->foo compiler exists in Debian, I've built defconfig. I'm relying on the buildbots to tell me what I missed, and people who actually have the hardware to tell me if it actually works. I'd like to get this into the MM tree soon after the current merge window closes, so quick feedback would be appreciated. Matthew Wilcox (Oracle) (26): mm: Convert page_table_check_pte_set() to page_table_check_ptes_set() mm: Add generic flush_icache_pages() and documentation mm: Add folio_flush_mapping() mm: Remove ARCH_IMPLEMENTS_FLUSH_DCACHE_FOLIO alpha: Implement the new page table range API arc: Implement the new page table range API arm64: Implement the new page table range API csky: Implement the new page table range API hexagon: Implement the new page table range API ia64: Implement the new page table range API loongarch: Implement the new page table range API m68k: Implement the new page table range API microblaze: Implement the new page table range API mips: Implement the new page table range API nios2: Implement the new page table range API openrisc: Implement the new page table range API parisc: Implement the new page table range API powerpc: Implement the new page table range API riscv: Implement the new page table range API s390: Implement the new page table range API superh: Implement the new page table range API sparc32: Implement the new page table range API sparc64: Implement the new page table range API um: Implement the new page table range API x86: Implement the new page table range API xtensa: Implement the new page table range API Yin Fengwei (4): filemap: Add filemap_map_folio_range() rmap: add folio_add_file_rmap_range() mm: Convert do_set_pte() to set_pte_range() filemap: Batch PTE mappings Documentation/core-api/cachetlb.rst | 35 +++---- Documentation/filesystems/locking.rst | 2 +- arch/alpha/include/asm/cacheflush.h | 10 ++ arch/alpha/include/asm/pgtable.h | 18 +++- arch/arc/include/asm/cacheflush.h | 7 +- arch/arc/include/asm/pgtable-bits-arcv2.h | 20 +++- arch/arc/mm/cache.c | 61 +++++++----- arch/arc/mm/tlb.c | 18 ++-- arch/arm/include/asm/cacheflush.h | 24 +++-- arch/arm/include/asm/pgtable.h | 5 +- arch/arm/include/asm/tlbflush.h | 13 ++- arch/arm/mm/copypage-v4mc.c | 5 +- arch/arm/mm/copypage-v6.c | 5 +- arch/arm/mm/copypage-xscale.c | 5 +- arch/arm/mm/dma-mapping.c | 24 ++--- arch/arm/mm/fault-armv.c | 14 +-- arch/arm/mm/flush.c | 99 +++++++++++-------- arch/arm/mm/mm.h | 2 +- arch/arm/mm/mmu.c | 14 ++- arch/arm64/include/asm/cacheflush.h | 4 +- arch/arm64/include/asm/pgtable.h | 25 +++-- arch/arm64/mm/flush.c | 36 +++---- arch/csky/abiv1/cacheflush.c | 32 ++++--- arch/csky/abiv1/inc/abi/cacheflush.h | 2 + arch/csky/abiv2/cacheflush.c | 30 +++--- arch/csky/abiv2/inc/abi/cacheflush.h | 10 +- arch/csky/include/asm/pgtable.h | 21 +++- arch/hexagon/include/asm/cacheflush.h | 7 +- arch/hexagon/include/asm/pgtable.h | 16 +++- arch/ia64/hp/common/sba_iommu.c | 26 ++--- arch/ia64/include/asm/cacheflush.h | 14 ++- arch/ia64/include/asm/pgtable.h | 14 ++- arch/ia64/mm/init.c | 29 ++++-- arch/loongarch/include/asm/cacheflush.h | 2 + arch/loongarch/include/asm/pgtable.h | 30 ++++-- arch/m68k/include/asm/cacheflush_mm.h | 26 +++-- arch/m68k/include/asm/pgtable_mm.h | 21 +++- arch/m68k/mm/motorola.c | 2 +- arch/microblaze/include/asm/cacheflush.h | 8 ++ arch/microblaze/include/asm/pgtable.h | 17 +++- arch/microblaze/include/asm/tlbflush.h | 4 +- arch/mips/include/asm/cacheflush.h | 32 ++++--- arch/mips/include/asm/pgtable.h | 36 ++++--- arch/mips/mm/c-r4k.c | 5 +- arch/mips/mm/cache.c | 56 +++++------ arch/mips/mm/init.c | 17 ++-- arch/nios2/include/asm/cacheflush.h | 6 +- arch/nios2/include/asm/pgtable.h | 27 ++++-- arch/nios2/mm/cacheflush.c | 61 ++++++------ arch/openrisc/include/asm/cacheflush.h | 8 +- arch/openrisc/include/asm/pgtable.h | 27 +++++- arch/openrisc/mm/cache.c | 12 ++- arch/parisc/include/asm/cacheflush.h | 14 ++- arch/parisc/include/asm/pgtable.h | 28 ++++-- arch/parisc/kernel/cache.c | 101 ++++++++++++++------ arch/powerpc/include/asm/book3s/pgtable.h | 10 +- arch/powerpc/include/asm/cacheflush.h | 14 ++- arch/powerpc/include/asm/kvm_ppc.h | 10 +- arch/powerpc/include/asm/nohash/pgtable.h | 13 +-- arch/powerpc/include/asm/pgtable.h | 6 ++ arch/powerpc/mm/book3s64/hash_utils.c | 11 ++- arch/powerpc/mm/cacheflush.c | 81 ++-------------- arch/powerpc/mm/nohash/e500_hugetlbpage.c | 3 +- arch/powerpc/mm/pgtable.c | 51 +++++----- arch/riscv/include/asm/cacheflush.h | 19 ++-- arch/riscv/include/asm/pgtable.h | 26 +++-- arch/riscv/mm/cacheflush.c | 11 +-- arch/s390/include/asm/pgtable.h | 34 +++++-- arch/sh/include/asm/cacheflush.h | 21 ++-- arch/sh/include/asm/pgtable.h | 6 +- arch/sh/include/asm/pgtable_32.h | 16 +++- arch/sh/mm/cache-j2.c | 4 +- arch/sh/mm/cache-sh4.c | 26 +++-- arch/sh/mm/cache-sh7705.c | 26 +++-- arch/sh/mm/cache.c | 54 ++++++----- arch/sh/mm/kmap.c | 3 +- arch/sparc/include/asm/cacheflush_32.h | 9 +- arch/sparc/include/asm/cacheflush_64.h | 18 ++-- arch/sparc/include/asm/pgtable_32.h | 15 ++- arch/sparc/include/asm/pgtable_64.h | 25 ++++- arch/sparc/kernel/smp_64.c | 56 +++++++---- arch/sparc/mm/init_32.c | 13 ++- arch/sparc/mm/init_64.c | 78 ++++++++------- arch/sparc/mm/tlb.c | 5 +- arch/um/include/asm/pgtable.h | 15 ++- arch/x86/include/asm/pgtable.h | 21 +++- arch/xtensa/include/asm/cacheflush.h | 9 +- arch/xtensa/include/asm/pgtable.h | 24 +++-- arch/xtensa/mm/cache.c | 83 +++++++++------- include/asm-generic/cacheflush.h | 5 + include/linux/cacheflush.h | 4 +- include/linux/mm.h | 3 +- include/linux/page_table_check.h | 14 +-- include/linux/pagemap.h | 26 ++++- include/linux/rmap.h | 2 + mm/filemap.c | 111 +++++++++++++--------- mm/memory.c | 27 +++--- mm/page_table_check.c | 14 +-- mm/rmap.c | 60 +++++++++--- mm/util.c | 2 +- 100 files changed, 1433 insertions(+), 838 deletions(-)