From patchwork Wed Dec 13 21:58:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13491921 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A482BC4332F for ; Wed, 13 Dec 2023 21:58:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 105678D005D; Wed, 13 Dec 2023 16:58:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0B5158D0049; Wed, 13 Dec 2023 16:58:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E97DC8D005D; Wed, 13 Dec 2023 16:58:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D55D58D0049 for ; Wed, 13 Dec 2023 16:58:52 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id A050D14034A for ; Wed, 13 Dec 2023 21:58:52 +0000 (UTC) X-FDA: 81563160504.08.5ABDDDB Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf03.hostedemail.com (Postfix) with ESMTP id D9AC12000E for ; Wed, 13 Dec 2023 21:58:49 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=hA1xqBKo; dmarc=none; spf=none (imf03.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702504730; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VTPS5o/byYmwk9OFQloTDmuvZsrWNfGkucOclSw3UC8=; b=kmV1l8R9Z14c2im1cEAO7iubNaI1n7hpX9f/b7XlKoLOWkUDbSf8BIZJSe05bjhRa/EPeQ ZLb6DePwWELVcaHl0NOBW0WVGvhJLeceE8WMVMa4IbawN5A8Vn+HLi1nejmYKqe4KePxwg Pp4sArm8eDVnGoZRf22KqnsMck5Tml0= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=hA1xqBKo; dmarc=none; spf=none (imf03.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702504730; a=rsa-sha256; cv=none; b=T4AlIdPHnZ7RTwIFHjokiDLLFblGBie0p09fnMfW9ppeq1FN+BkBGfoNJua1tVZ/sWkicX 4EO9FfmknBomBGBjdRECk5fCzeH+1Vnw5Ck9GKBCagcudCXqG3aI/OLJybUJton98gr0ps Ut7BKEENn+UhFyHCwEXPioOcnYxsqq0= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=VTPS5o/byYmwk9OFQloTDmuvZsrWNfGkucOclSw3UC8=; b=hA1xqBKoOwSweuidhOfjd7ZwYA 945uPxkf57pPk+RF3mrI5yK1gZ4cxzjByp1BONsfFTdWV6Mn2hDFINV25IzYUL6qhBBzdqh03U1FH wMZ6bMeRC09ehQWcFoinRbH8Ce1GCp/6KJkbkPyfwqf7W79XQWdwwhVtjycgU92k8ONd4C9M/QDsf k599Q/Ah/6rmHBOxp4LUmDy5emXtR0mQsgkjU6u+cFKlX5QPKjJYUN5wXypSRix7txT+zow01aFAk tM3T0076dyn6NTZ9E8VDtQrfjg3gSH/QcMoD0D4a0NVzbhoUHIo7hCIsEzUfF6r1pXtTOfv7y3I3g LH/h4s0w==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1rDXFd-002oiJ-7r; Wed, 13 Dec 2023 21:58:45 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH 01/13] mm: Return the folio from __read_swap_cache_async() Date: Wed, 13 Dec 2023 21:58:30 +0000 Message-Id: <20231213215842.671461-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20231213215842.671461-1-willy@infradead.org> References: <20231213215842.671461-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: D9AC12000E X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: xqm87gxmzrsz3kxqpy76r3fexie1g3fn X-HE-Tag: 1702504729-670823 X-HE-Meta: U2FsdGVkX1/T45pdR3SEaHlDqBOcrJDaLw1RykWn6viZL+AVcZmBwAtBl9k4B+k2Rur6FdevhXJodxuH6CmgQq+3pTWfpVIiszgvPA2gCtozZQq42bZEYQKCYH83PhI3gAaNWyCCOL9XWDzMPPsGcOPJVrd2K/ukqtKR/QqHm8MKNbnmGhCy1FyiKp66FpNdWa5PK3BzmNVg+8X0/ViZuRPCPPRBM6gllLZX31YTIAMlC9LtMmV1DioJd9AeewV3eTy/JH07OnWB0X+BC0N15SMaDXlgViA6QLCFIuFDu+mS5NU2FNc/btGGzVC9uzakef+qdkhzDzibWtvOy5M9H6CzIacYqfvQ3/Ap/1u0kK4v3bYCdVyxmxZ6nZheGbIDg7t4olsKNEd6NY7IxMZv0kf9kasxy3t9WqlqypxpVy7K2dDJb8A9WUEmxEjrYsQG+4WVPI2Yz+VErb7iGwtzD4wheIrKUlon3rOcAFy74uvjX8RA1WoAoMycgeFqz+FBx5agj+3z8Tc+H+kHYtWRBxF6E5QjjkmAnshxMMV+voJTm41nymvbPQ/ReZqkvPLxQIZCDHvrJPlPlo1Q4XJ6wpqBeNd0Y8tX5Tske0tzvHQm1Zrwep1DVXy3GNrPj7NeZP7ytV75LaB+3a+nvk5/SWOT0yNg4KJK24Z6p08A5OadGT/8HJsK2xnF1LTwoLtSxvRVRFQ8vO8r8gokiVosyan4PtidmAbFOrk1JJUcasZhSIfCUi397Z/foQBtbV1PWG3GPm9OFhvPCh8ZTnzA8V/yCtDLPINAHbXG5bDVQndWEJm0mykOAH5JsoiXzkx1zS1IybWMNpEg7SeQKDrfO40ceEJhC8TzGdQtkmomGMU+hYSEq8OtO8NjM5TCNtS8MPRkMIJhlyORmSieEoo6TiZqPoEYtkp6nLvwm8WuknL+iaF9Hi0xt/1Q80NVdnNRk4aSDC2pe9WxEM33opk BMYXxWZZ iKnsE7IjQHeJA92LBpjVrfCagp/ZFs1E/3g5mu6k7VWlhdkF/jMzMcQ3FKn1jmytupUKFmKNNE0HsnO6g9QIWIK7f/A32v10+ziQQyK4cpzXVXwsJko7ECnfwBvnv2Youemqt2LFFZ+WWEWh67bKy+keYyl6jZvcBqSHvv16jY5Z0wNcbwgbXIt9M0XJmxHhLcPFliZT4qISipibIK6bsfsj2kOLBS7L91wYU X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move the folio->page conversion into the callers that actually want that. Most of the callers are happier with the folio anyway. If the page_allocated boolean is set, the folio allocated is of order-0, so it is safe to pass the page directly to swap_readpage(). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/zswap.h | 4 +-- mm/swap.h | 7 ++-- mm/swap_state.c | 75 ++++++++++++++++++++----------------------- mm/zswap.c | 48 +++++++++++++-------------- 4 files changed, 64 insertions(+), 70 deletions(-) diff --git a/include/linux/zswap.h b/include/linux/zswap.h index 08c240e16a01..e88572d4c720 100644 --- a/include/linux/zswap.h +++ b/include/linux/zswap.h @@ -34,7 +34,7 @@ void zswap_swapon(int type); void zswap_swapoff(int type); void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg); void zswap_lruvec_state_init(struct lruvec *lruvec); -void zswap_page_swapin(struct page *page); +void zswap_folio_swapin(struct folio *folio); #else struct zswap_lruvec_state {}; @@ -54,7 +54,7 @@ static inline void zswap_swapon(int type) {} static inline void zswap_swapoff(int type) {} static inline void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg) {} static inline void zswap_lruvec_state_init(struct lruvec *lruvec) {} -static inline void zswap_page_swapin(struct page *page) {} +static inline void zswap_folio_swapin(struct folio *folio) {} #endif #endif /* _LINUX_ZSWAP_H */ diff --git a/mm/swap.h b/mm/swap.h index c0dc73e10e91..a60ab1cfcaf2 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -49,10 +49,9 @@ struct page *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, struct vm_area_struct *vma, unsigned long addr, struct swap_iocb **plug); -struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, - struct mempolicy *mpol, pgoff_t ilx, - bool *new_page_allocated, - bool skip_if_exists); +struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_flags, + struct mempolicy *mpol, pgoff_t ilx, bool *new_page_allocated, + bool skip_if_exists); struct page *swap_cluster_readahead(swp_entry_t entry, gfp_t flag, struct mempolicy *mpol, pgoff_t ilx); struct page *swapin_readahead(swp_entry_t entry, gfp_t flag, diff --git a/mm/swap_state.c b/mm/swap_state.c index 7d775d0b1312..d4e25d9b5dc6 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -419,14 +419,12 @@ struct folio *filemap_get_incore_folio(struct address_space *mapping, return folio; } -struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, - struct mempolicy *mpol, pgoff_t ilx, - bool *new_page_allocated, - bool skip_if_exists) +struct folio *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, + struct mempolicy *mpol, pgoff_t ilx, bool *new_page_allocated, + bool skip_if_exists) { struct swap_info_struct *si; struct folio *folio; - struct page *page; void *shadow = NULL; *new_page_allocated = false; @@ -443,10 +441,8 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, */ folio = filemap_get_folio(swap_address_space(entry), swp_offset(entry)); - if (!IS_ERR(folio)) { - page = folio_file_page(folio, swp_offset(entry)); - goto got_page; - } + if (!IS_ERR(folio)) + goto got_folio; /* * Just skip read ahead for unused swap slot. @@ -460,7 +456,7 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, goto fail_put_swap; /* - * Get a new page to read into from swap. Allocate it now, + * Get a new folio to read into from swap. Allocate it now, * before marking swap_map SWAP_HAS_CACHE, when -EEXIST will * cause any racers to loop around until we add it to cache. */ @@ -496,13 +492,13 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, * stumble across a swap_map entry whose SWAP_HAS_CACHE * has not yet been cleared. Or race against another * __read_swap_cache_async(), which has set SWAP_HAS_CACHE - * in swap_map, but not yet added its page to swap cache. + * in swap_map, but not yet added its folio to swap cache. */ schedule_timeout_uninterruptible(1); } /* - * The swap entry is ours to swap in. Prepare the new page. + * The swap entry is ours to swap in. Prepare the new folio. */ __folio_set_locked(folio); @@ -523,10 +519,9 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, /* Caller will initiate read into locked folio */ folio_add_lru(folio); *new_page_allocated = true; - page = &folio->page; -got_page: +got_folio: put_swap_device(si); - return page; + return folio; fail_unlock: put_swap_folio(folio, entry); @@ -554,16 +549,16 @@ struct page *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, bool page_allocated; struct mempolicy *mpol; pgoff_t ilx; - struct page *page; + struct folio *folio; mpol = get_vma_policy(vma, addr, 0, &ilx); - page = __read_swap_cache_async(entry, gfp_mask, mpol, ilx, + folio = __read_swap_cache_async(entry, gfp_mask, mpol, ilx, &page_allocated, false); mpol_cond_put(mpol); if (page_allocated) - swap_readpage(page, false, plug); - return page; + swap_readpage(&folio->page, false, plug); + return folio_file_page(folio, swp_offset(entry)); } static unsigned int __swapin_nr_pages(unsigned long prev_offset, @@ -648,7 +643,7 @@ static unsigned long swapin_nr_pages(unsigned long offset) struct page *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, struct mempolicy *mpol, pgoff_t ilx) { - struct page *page; + struct folio *folio; unsigned long entry_offset = swp_offset(entry); unsigned long offset = entry_offset; unsigned long start_offset, end_offset; @@ -673,31 +668,31 @@ struct page *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask, blk_start_plug(&plug); for (offset = start_offset; offset <= end_offset ; offset++) { /* Ok, do the async read-ahead now */ - page = __read_swap_cache_async( + folio = __read_swap_cache_async( swp_entry(swp_type(entry), offset), gfp_mask, mpol, ilx, &page_allocated, false); - if (!page) + if (!folio) continue; if (page_allocated) { - swap_readpage(page, false, &splug); + swap_readpage(&folio->page, false, &splug); if (offset != entry_offset) { - SetPageReadahead(page); + folio_set_readahead(folio); count_vm_event(SWAP_RA); } } - put_page(page); + folio_put(folio); } blk_finish_plug(&plug); swap_read_unplug(splug); lru_add_drain(); /* Push any new pages onto the LRU now */ skip: /* The page was likely read above, so no need for plugging here */ - page = __read_swap_cache_async(entry, gfp_mask, mpol, ilx, + folio = __read_swap_cache_async(entry, gfp_mask, mpol, ilx, &page_allocated, false); if (unlikely(page_allocated)) - swap_readpage(page, false, NULL); - zswap_page_swapin(page); - return page; + swap_readpage(&folio->page, false, NULL); + zswap_folio_swapin(folio); + return folio_file_page(folio, swp_offset(entry)); } int init_swap_address_space(unsigned int type, unsigned long nr_pages) @@ -815,7 +810,7 @@ static struct page *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, { struct blk_plug plug; struct swap_iocb *splug = NULL; - struct page *page; + struct folio *folio; pte_t *pte = NULL, pentry; unsigned long addr; swp_entry_t entry; @@ -848,18 +843,18 @@ static struct page *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, continue; pte_unmap(pte); pte = NULL; - page = __read_swap_cache_async(entry, gfp_mask, mpol, ilx, + folio = __read_swap_cache_async(entry, gfp_mask, mpol, ilx, &page_allocated, false); - if (!page) + if (!folio) continue; if (page_allocated) { - swap_readpage(page, false, &splug); + swap_readpage(&folio->page, false, &splug); if (i != ra_info.offset) { - SetPageReadahead(page); + folio_set_readahead(folio); count_vm_event(SWAP_RA); } } - put_page(page); + folio_put(folio); } if (pte) pte_unmap(pte); @@ -867,13 +862,13 @@ static struct page *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, swap_read_unplug(splug); lru_add_drain(); skip: - /* The page was likely read above, so no need for plugging here */ - page = __read_swap_cache_async(targ_entry, gfp_mask, mpol, targ_ilx, + /* The folio was likely read above, so no need for plugging here */ + folio = __read_swap_cache_async(targ_entry, gfp_mask, mpol, targ_ilx, &page_allocated, false); if (unlikely(page_allocated)) - swap_readpage(page, false, NULL); - zswap_page_swapin(page); - return page; + swap_readpage(&folio->page, false, NULL); + zswap_folio_swapin(folio); + return folio_file_page(folio, swp_offset(entry)); } /** diff --git a/mm/zswap.c b/mm/zswap.c index d49aad0359d2..08b5212aa6b8 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -368,12 +368,12 @@ void zswap_lruvec_state_init(struct lruvec *lruvec) atomic_long_set(&lruvec->zswap_lruvec_state.nr_zswap_protected, 0); } -void zswap_page_swapin(struct page *page) +void zswap_folio_swapin(struct folio *folio) { struct lruvec *lruvec; - if (page) { - lruvec = folio_lruvec(page_folio(page)); + if (folio) { + lruvec = folio_lruvec(folio); atomic_long_inc(&lruvec->zswap_lruvec_state.nr_zswap_protected); } } @@ -1379,14 +1379,14 @@ static int zswap_enabled_param_set(const char *val, * writeback code **********************************/ /* - * Attempts to free an entry by adding a page to the swap cache, - * decompressing the entry data into the page, and issuing a - * bio write to write the page back to the swap device. + * Attempts to free an entry by adding a folio to the swap cache, + * decompressing the entry data into the folio, and issuing a + * bio write to write the folio back to the swap device. * - * This can be thought of as a "resumed writeback" of the page + * This can be thought of as a "resumed writeback" of the folio * to the swap device. We are basically resuming the same swap * writeback path that was intercepted with the zswap_store() - * in the first place. After the page has been decompressed into + * in the first place. After the folio has been decompressed into * the swap cache, the compressed version stored by zswap can be * freed. */ @@ -1394,7 +1394,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, struct zswap_tree *tree) { swp_entry_t swpentry = entry->swpentry; - struct page *page; + struct folio *folio; struct mempolicy *mpol; struct scatterlist input, output; struct crypto_acomp_ctx *acomp_ctx; @@ -1413,18 +1413,18 @@ static int zswap_writeback_entry(struct zswap_entry *entry, return -ENOMEM; } - /* try to allocate swap cache page */ + /* try to allocate swap cache folio */ mpol = get_task_policy(current); - page = __read_swap_cache_async(swpentry, GFP_KERNEL, mpol, + folio = __read_swap_cache_async(swpentry, GFP_KERNEL, mpol, NO_INTERLEAVE_INDEX, &page_was_allocated, true); - if (!page) { + if (!folio) { ret = -ENOMEM; goto fail; } - /* Found an existing page, we raced with load/swapin */ + /* Found an existing folio, we raced with load/swapin */ if (!page_was_allocated) { - put_page(page); + folio_put(folio); ret = -EEXIST; goto fail; } @@ -1434,12 +1434,12 @@ static int zswap_writeback_entry(struct zswap_entry *entry, * concurrent swapping to and from the slot. Verify that the * swap entry hasn't been invalidated and recycled behind our * backs (our zswap_entry reference doesn't prevent that), to - * avoid overwriting a new swap page with old compressed data. + * avoid overwriting a new swap folio with old compressed data. */ spin_lock(&tree->lock); if (zswap_rb_search(&tree->rbroot, swp_offset(entry->swpentry)) != entry) { spin_unlock(&tree->lock); - delete_from_swap_cache(page_folio(page)); + delete_from_swap_cache(folio); ret = -ENOMEM; goto fail; } @@ -1459,7 +1459,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, mutex_lock(acomp_ctx->mutex); sg_init_one(&input, src, entry->length); sg_init_table(&output, 1); - sg_set_page(&output, page, PAGE_SIZE, 0); + sg_set_page(&output, &folio->page, PAGE_SIZE, 0); acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, dlen); ret = crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait); dlen = acomp_ctx->req->dlen; @@ -1473,20 +1473,20 @@ static int zswap_writeback_entry(struct zswap_entry *entry, BUG_ON(ret); BUG_ON(dlen != PAGE_SIZE); - /* page is up to date */ - SetPageUptodate(page); + /* folio is up to date */ + folio_mark_uptodate(folio); /* move it to the tail of the inactive list after end_writeback */ - SetPageReclaim(page); + folio_set_reclaim(folio); - if (!PageLRU(page)) { + if (!folio_test_lru(folio)) { /* drain lru cache to help folio_rotate_reclaimable() */ lru_add_drain(); } /* start writeback */ - __swap_writepage(page, &wbc); - put_page(page); + __swap_writepage(&folio->page, &wbc); + folio_put(folio); return ret; @@ -1495,7 +1495,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, kfree(tmp); /* - * If we get here because the page is already in swapcache, a + * If we get here because the folio is already in swapcache, a * load may be happening concurrently. It is safe and okay to * not free the entry. It is also okay to return !0. */