From patchwork Mon Jul 12 19:01:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12371881 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4AA5C07E99 for ; Mon, 12 Jul 2021 19:02:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5177E6121E for ; Mon, 12 Jul 2021 19:02:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5177E6121E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 57A3F6B0095; Mon, 12 Jul 2021 15:02:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 52AF76B0096; Mon, 12 Jul 2021 15:02:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3CBD66B0098; Mon, 12 Jul 2021 15:02:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0023.hostedemail.com [216.40.44.23]) by kanga.kvack.org (Postfix) with ESMTP id 125436B0095 for ; Mon, 12 Jul 2021 15:02:59 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 25597230FC for ; Mon, 12 Jul 2021 19:02:58 +0000 (UTC) X-FDA: 78354858036.18.CCFB3A7 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id 76AE51005B30 for ; Mon, 12 Jul 2021 19:02:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:In-Reply-To:References; bh=IVmHYOAXTI19qlhMZur37PZfYtnXt/URUV67Hwi1f7M=; b=ljsTVykPu0NKoRx0Qif/upo39b pNyoxZap3Z9yUs1ArecBeTCy3qU2LiRB3UVIYeKa9gzqw+FiAuGeAXYgVzNQOZV5DRaEhGboQayve EDOh1c8k3pZfGz4EW/KqRShUYCNbFYuEnhsCePUPhDuqCYi0REvJAv1CHexcxmt+lkboX3vpKhvnm gcQ3WNdKmFkRr7G6OniTRWjtgzsY9X/dMFjCLX5LYUzJWKAFcdeHPLwioR7HOy3jbSGAEp0U2fqsH cDHH1GU+10nj20nsAKwy46ak5AYLUK3XNbFDZezhANy1wXnsu/+d0dfEHfPIK0bPKfTBX5u4p0vmC 2G0LhAOQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1m31Bt-000L5O-U5; Mon, 12 Jul 2021 19:02:10 +0000 From: "Matthew Wilcox (Oracle)" To: akpm@linux-foundation.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v13a 00/32] Memory folios Date: Mon, 12 Jul 2021 20:01:32 +0100 Message-Id: <20210712190204.80979-1-willy@infradead.org> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 76AE51005B30 X-Stat-Signature: 9eipwff5azmsco6o9nix94igpany6p4o Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ljsTVykP; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1626116577-888962 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Managing memory in 4KiB pages is a serious overhead. Many benchmarks benefit from managing memory in larger chunks. As an example, an earlier iteration of this idea which used compound pages (and wasn't particularly tuned) got a 7% performance boost when compiling the kernel. Using compound pages or THPs exposes a weakness of our type system. Functions are often unprepared for compound pages to be passed to them, and may only act on PAGE_SIZE chunks. Even functions which are aware of compound pages may expect a head page, and do the wrong thing if passed a tail page. We also waste a lot of instructions ensuring that we're not looking at a tail page. Almost every call to PageFoo() contains one or more hidden calls to compound_head(). This also happens for get_page(), put_page() and many more functions. This patch series uses a new type, the struct folio, to manage memory. It provides some basic infrastructure that's worthwhile in its own right, shrinking the kernel by about 6kB of text. -- 8< -- This is the first batch of patches for the next merge window. They are identical to the ones sent yesterday to linux-kernel and the build bots didn't complain about any of these. They have been extensively reviewed. Please apply. Matthew Wilcox (Oracle) (32): mm: Convert get_page_unless_zero() to return bool mm: Introduce struct folio mm: Add folio_pgdat(), folio_zone() and folio_zonenum() mm/vmstat: Add functions to account folio statistics mm/debug: Add VM_BUG_ON_FOLIO() and VM_WARN_ON_ONCE_FOLIO() mm: Add folio reference count functions mm: Add folio_put() mm: Add folio_get() mm: Add folio_try_get_rcu() mm: Add folio flag manipulation functions mm/lru: Add folio LRU functions mm: Handle per-folio private data mm/filemap: Add folio_index(), folio_file_page() and folio_contains() mm/filemap: Add folio_next_index() mm/filemap: Add folio_pos() and folio_file_pos() mm/util: Add folio_mapping() and folio_file_mapping() mm/filemap: Add folio_unlock() mm/filemap: Add folio_lock() mm/filemap: Add folio_lock_killable() mm/filemap: Add __folio_lock_async() mm/filemap: Add folio_wait_locked() mm/filemap: Add __folio_lock_or_retry() mm/swap: Add folio_rotate_reclaimable() mm/filemap: Add folio_end_writeback() mm/writeback: Add folio_wait_writeback() mm/writeback: Add folio_wait_stable() mm/filemap: Add folio_wait_bit() mm/filemap: Add folio_wake_bit() mm/filemap: Convert page wait queues to be folios mm/filemap: Add folio private_2 functions fs/netfs: Add folio fscache functions mm: Add folio_mapped() Documentation/core-api/mm-api.rst | 4 + Documentation/filesystems/netfs_library.rst | 2 + fs/afs/write.c | 9 +- fs/cachefiles/rdwr.c | 16 +- fs/io_uring.c | 2 +- include/linux/huge_mm.h | 15 - include/linux/mm.h | 165 +++++++-- include/linux/mm_inline.h | 85 +++-- include/linux/mm_types.h | 77 ++++ include/linux/mmdebug.h | 20 + include/linux/netfs.h | 77 ++-- include/linux/page-flags.h | 247 +++++++++---- include/linux/page_ref.h | 158 +++++++- include/linux/pagemap.h | 390 +++++++++++--------- include/linux/swap.h | 7 +- include/linux/vmstat.h | 107 ++++++ mm/Makefile | 2 +- mm/filemap.c | 329 +++++++++-------- mm/folio-compat.c | 43 +++ mm/internal.h | 1 + mm/memory.c | 8 +- mm/page-writeback.c | 72 ++-- mm/page_io.c | 4 +- mm/swap.c | 30 +- mm/swapfile.c | 8 +- mm/util.c | 59 +-- 26 files changed, 1356 insertions(+), 581 deletions(-) create mode 100644 mm/folio-compat.c