From patchwork Sat Feb 17 02:25:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13561219 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D95CC48260 for ; Sat, 17 Feb 2024 02:26:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A5E226B00AE; Fri, 16 Feb 2024 21:26:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A0EA66B00AF; Fri, 16 Feb 2024 21:26:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 889F36B00B0; Fri, 16 Feb 2024 21:26:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 76CD06B00AE for ; Fri, 16 Feb 2024 21:26:08 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 2737B1A014C for ; Sat, 17 Feb 2024 02:26:08 +0000 (UTC) X-FDA: 81799706016.14.11C3867 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf26.hostedemail.com (Postfix) with ESMTP id 8A03E140012 for ; Sat, 17 Feb 2024 02:26:06 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=QmlP8Yff; spf=none (imf26.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708136766; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=21JMIPgwOOuKR2eeBq0L1sNsizlI1+DP/HMzZ2lCV+A=; b=VxLmf8utm5XuuYbszPY8FxJ35Dj+lsvpdpwUvxqPawm+XW77H+kQOLtPG8NsHMqzGrbtzr YO56r6OrlQlGTRhot5oHyqbMYxmGPb1u/K0eaMvezyJ0M328wUEQjjFhylBTaA4dXkJbVt 5/tzTT0E0W7CNlWj9549whxqGaOghZE= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=QmlP8Yff; spf=none (imf26.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708136766; a=rsa-sha256; cv=none; b=JdrCbcSRAtlsJHcsa1B/XdHkLZzpJb/6xwEyrlCgkraoSYX+OsFj6WvRKxHLbnXM22QiEO FLRabPDw0fvs3/Yv+uEXlLeGTViatJNsj4Y5exLGh5eaOZr3llctTI6ELeZGmArp38plTQ yDNfKpiaon4LBBdOiqR10BYw9iIq7xw= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=21JMIPgwOOuKR2eeBq0L1sNsizlI1+DP/HMzZ2lCV+A=; b=QmlP8Yff9ZiFb4bAExFaWdYrmc vRivX9/H+y7nNEG5APYPNHaUDvu+SGtbMKSoK0b2uw91dYrUGaBioiCVmc/mWzSYI7ZSr53kqVURr DGgIm+DRXvSi2TjlMze82pPIRSOlnzOXpxG+wBgH+OuuC3o4G5OILC2oVgR1x7tgS+UWN63xt/nUJ MVzsygl/uIyLwMHCvMhzSCxyMPnIrkByIWYzCJKIOKZDjBjLwnD2QaEHbGsmS0IRAQfADduSWJTBm g5dr8pdz1Du0WFOU9lNHzGmTtklrq68ACahpSXvtAno4GdFdNeFAuv3Yhybk6TJSV8+a8ddHmvD3b iNaNMqDA==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rbAOi-00000006HDL-3jz6; Sat, 17 Feb 2024 02:25:48 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 08/18] mm: use __page_cache_release() in folios_put() Date: Sat, 17 Feb 2024 02:25:34 +0000 Message-ID: <20240217022546.1496101-9-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217022546.1496101-1-willy@infradead.org> References: <20240217022546.1496101-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8A03E140012 X-Rspam-User: X-Stat-Signature: 3rpy8m8zy6ksqfxucsy7sx7zeps9f1c7 X-Rspamd-Server: rspam01 X-HE-Tag: 1708136766-187192 X-HE-Meta: U2FsdGVkX18Tmh36Y3pk65pTM208wdgHPTyVTePpUNC59GW2Z15QkWmxkF/LzrXjfdTDI4B0uqNhULPl0q6vUl9OoCOjAlpyjqUKe8uuJT0zHUNy7G5uOdTUFnSkXoY8uQudjd9PoE7Gz8nJyH5Fgl+RnD5U3EiuyDLpXLaPEkxy4ftTo5aUBcSw2EUKPhxr+rXL+uOy/q6DCHFl5xyfcmDGP4rFQ1Rn838dcpoc1DZSfuuVqZozBZpnh4shXuMiGYyXKqzfabWMXRJ3jCRSt8xR6pdlQF1M97sN9tCknYI9Lq9SSoS5mB2C2WIZtV79hxuBQ4EC70HJ8g20qYniis88kZKTmSs2lPsydyZhWfXwUfGnHRkSyQ2DCAX3m9iLXx6TCjjpZWCdzpgwVAOGXLscPeKwof2hb4KHTV9qoCQH4VkZNImDBcAcs0QCLs9A6kvXX1XTBWYKyCwvZX/MP5xBTlaCv6OG/WpI5g9QLK5CqTSQlreseBOzY7h49e1/j+alQTvZB6Jj0Dw9RwOH/2RcDLaUJzsejGWXkpUioXiM4KkV/t/vbiY4FXFcR3wPNh4X4UDFDfPd+ALJjgBFzdaElFUqGhSm/6ly4Tyi0+mZlFtL3cQwv1HZWWD/go0qbUzjeXD0q/lyY7OMEol2YgH2HSd9uE1vvwyINE6vzeVorneCEH+vHXenOu0m02kGGDeTGBGNqfcLIiE9jbIGhMrqit+NKdHwXz+tEnRXc4frZUFVDYgYL3myMvtgH78MyD1a5j4Z2EvNR+tz0IdyLOpaRqjhzfzlT3sPXEnH0jgkOH0B5UO2hKQ3jfhFLwF2vCIFh9YvV3LlgLUsTm2RPYHvn+2wOONj5r8pTjJ7ukmBORqNuKKychH1vwdf6R4IkQvHB3+5Fjl5vyyGfHB9RjHdmVI9YjZHcalts/3nhA0KcwUbYZr+zLpKl0+5fBysEdiKaKzUl++7v3oNfLf IcfOX1Mj d44OgtdOehAzoLdPwdhNuTvf73cTmWPmEPGWywxZ3xKKNl5r+uPV7+EUNho3FkQNYvU5RzpZOQzTl6v9owin6zqrMh2BDuMLejLsxd7CY/kAsVvv268SakY5j2NRkNxPz0lX4eCJxAwz2uw/WFbbSKkbgUY2UDzkLy+XCl0gRIXcrDcX5CBlx+7XpIsP+Vw+OspYjsdJ0fBbbGacKyMWNvVm5p+aHpJHhU04aT2lCrPg89z7TfN1W+HvPLtLVuXgydxF569mvueSRyZzjfxMUa1ciPA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Pass a pointer to the lruvec so we can take advantage of the folio_lruvec_relock_irqsave(). Adjust the calling convention of folio_lruvec_relock_irqsave() to suit and add a page_cache_release() wrapper. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/memcontrol.h | 16 +++++----- mm/swap.c | 62 ++++++++++++++++++-------------------- 2 files changed, 37 insertions(+), 41 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 46d9abb20761..8a0e8972a3d3 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1705,18 +1705,18 @@ static inline struct lruvec *folio_lruvec_relock_irq(struct folio *folio, return folio_lruvec_lock_irq(folio); } -/* Don't lock again iff page's lruvec locked */ -static inline struct lruvec *folio_lruvec_relock_irqsave(struct folio *folio, - struct lruvec *locked_lruvec, unsigned long *flags) +/* Don't lock again iff folio's lruvec locked */ +static inline void folio_lruvec_relock_irqsave(struct folio *folio, + struct lruvec **lruvecp, unsigned long *flags) { - if (locked_lruvec) { - if (folio_matches_lruvec(folio, locked_lruvec)) - return locked_lruvec; + if (*lruvecp) { + if (folio_matches_lruvec(folio, *lruvecp)) + return; - unlock_page_lruvec_irqrestore(locked_lruvec, *flags); + unlock_page_lruvec_irqrestore(*lruvecp, *flags); } - return folio_lruvec_lock_irqsave(folio, flags); + *lruvecp = folio_lruvec_lock_irqsave(folio, flags); } #ifdef CONFIG_CGROUP_WRITEBACK diff --git a/mm/swap.c b/mm/swap.c index fffdb48cfbf9..21c2df0f7928 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -74,22 +74,21 @@ static DEFINE_PER_CPU(struct cpu_fbatches, cpu_fbatches) = { .lock = INIT_LOCAL_LOCK(lock), }; -/* - * This path almost never happens for VM activity - pages are normally freed - * in batches. But it gets used by networking - and for compound pages. - */ -static void __page_cache_release(struct folio *folio) +static void __page_cache_release(struct folio *folio, struct lruvec **lruvecp, + unsigned long *flagsp) { if (folio_test_lru(folio)) { - struct lruvec *lruvec; - unsigned long flags; - - lruvec = folio_lruvec_lock_irqsave(folio, &flags); - lruvec_del_folio(lruvec, folio); + folio_lruvec_relock_irqsave(folio, lruvecp, flagsp); + lruvec_del_folio(*lruvecp, folio); __folio_clear_lru_flags(folio); - unlock_page_lruvec_irqrestore(lruvec, flags); } - /* See comment on folio_test_mlocked in folios_put() */ + + /* + * In rare cases, when truncation or holepunching raced with + * munlock after VM_LOCKED was cleared, Mlocked may still be + * found set here. This does not indicate a problem, unless + * "unevictable_pgs_cleared" appears worryingly large. + */ if (unlikely(folio_test_mlocked(folio))) { long nr_pages = folio_nr_pages(folio); @@ -99,9 +98,23 @@ static void __page_cache_release(struct folio *folio) } } +/* + * This path almost never happens for VM activity - pages are normally freed + * in batches. But it gets used by networking - and for compound pages. + */ +static void page_cache_release(struct folio *folio) +{ + struct lruvec *lruvec = NULL; + unsigned long flags; + + __page_cache_release(folio, &lruvec, &flags); + if (lruvec) + unlock_page_lruvec_irqrestore(lruvec, flags); +} + static void __folio_put_small(struct folio *folio) { - __page_cache_release(folio); + page_cache_release(folio); mem_cgroup_uncharge(folio); free_unref_page(&folio->page, 0); } @@ -115,7 +128,7 @@ static void __folio_put_large(struct folio *folio) * be called for hugetlb (it has a separate hugetlb_cgroup.) */ if (!folio_test_hugetlb(folio)) - __page_cache_release(folio); + page_cache_release(folio); destroy_large_folio(folio); } @@ -216,7 +229,7 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) if (move_fn != lru_add_fn && !folio_test_clear_lru(folio)) continue; - lruvec = folio_lruvec_relock_irqsave(folio, lruvec, &flags); + folio_lruvec_relock_irqsave(folio, &lruvec, &flags); move_fn(lruvec, folio); folio_set_lru(folio); @@ -996,24 +1009,7 @@ void folios_put(struct folio_batch *folios) continue; } - if (folio_test_lru(folio)) { - lruvec = folio_lruvec_relock_irqsave(folio, lruvec, - &flags); - lruvec_del_folio(lruvec, folio); - __folio_clear_lru_flags(folio); - } - - /* - * In rare cases, when truncation or holepunching raced with - * munlock after VM_LOCKED was cleared, Mlocked may still be - * found set here. This does not indicate a problem, unless - * "unevictable_pgs_cleared" appears worryingly large. - */ - if (unlikely(folio_test_mlocked(folio))) { - __folio_clear_mlocked(folio); - zone_stat_sub_folio(folio, NR_MLOCK); - count_vm_event(UNEVICTABLE_PGCLEARED); - } + __page_cache_release(folio, &lruvec, &flags); if (j != i) folios->folios[j] = folio;