From patchwork Mon Aug 8 19:33:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939038 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F655C00140 for ; Mon, 8 Aug 2022 19:34:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DAC508E0003; Mon, 8 Aug 2022 15:34:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D0EFD8E0001; Mon, 8 Aug 2022 15:34:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE9C78E0002; Mon, 8 Aug 2022 15:34:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 8930C6B0071 for ; Mon, 8 Aug 2022 15:34:49 -0400 (EDT) Received: from smtpin31.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 5DC57AB32B for ; Mon, 8 Aug 2022 19:34:49 +0000 (UTC) X-FDA: 79777427898.31.C814228 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf06.hostedemail.com (Postfix) with ESMTP id 4D6DD18014A for ; Mon, 8 Aug 2022 19:34:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=1bTiisbYRFIZBii2Ka+0vHR7SjqaAbpzvQ4SCW085yw=; b=EV0zKL1X+4VWE9IgGj4n/w9IIP B9/BHxaLiNlsIu30PCiU2x5O0athJN8uXtFFL43wBOB1qphwcAjtcJ1oLQkEqi6eSxWaFoIqAbeWN 1g9N+Xm8Xs+MvXVq1IVf40NIzyPqCoOxyy+Y4j3+Bv2UiGz0N2s6qNt9Y4HyKMmPzgdkMQtLn/9W9 GQcF3fRXYW3/z43LbWAwWvA1ns1j44zlh5KDss6Vfzbal3uiXF7LcWGMi3hCYQI2zbu2icgpEtc52 Nual1UgFcXFxAzkK6YLOtR6OuEteYTdeCeZqCpOjCt/PkiRkOAWDAVkTUgRLWdzST2qdhq3/iXxs4 ymAF6m7Q==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8WO-00EAs2-Me; Mon, 08 Aug 2022 19:34:43 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 01/59] mm: Fix VM_BUG_ON in __delete_from_swap_cache() Date: Mon, 8 Aug 2022 20:33:29 +0100 Message-Id: <20220808193430.3378317-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987289; a=rsa-sha256; cv=none; b=jqLqzLkm27d1cthCoPxkyoudA5AvbExXfDY+b6UY7jvvEpnNcEzYJrmYlyVqed+Dm1QGM2 7N7tq462MMx5cQZgSeyLkcZu0wAnrT77N9tx+I7bJqwBhKj1jKg/gcHZmCHdk5k5K4cwRa zyWHnzNfute8b9tpMGlXlRdHL8MoBis= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=EV0zKL1X; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987289; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1bTiisbYRFIZBii2Ka+0vHR7SjqaAbpzvQ4SCW085yw=; b=Ue+9t05TPWvNCjUIHTFVvTit7UCZKqdRfp4DhVeUbl/IbftezMbQOw1iwvVDEZN027Aj3a Xr9ONhjQ+6Ucygx3RV5TNhVaVXShp/xWONq2yBstRnHosOvxE+qDxSzmUtun7gPJfoThqc V7FcxLjDOIlAfi897ycw3ZGWOQRiVOY= X-Rspamd-Queue-Id: 4D6DD18014A Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=EV0zKL1X; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: keg43adq5bgaj5go48ixkpupeqss9bue X-HE-Tag: 1659987288-550186 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Commit ceff9d3354e9 changed the VM_BUG_ON() to dump the folio we're storing instead of the entry we retrieved. This was a mistake; the entry we retrieved is the more interesting page to dump. Signed-off-by: Matthew Wilcox (Oracle) --- mm/swap_state.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/swap_state.c b/mm/swap_state.c index e166051566f4..41afa6d45b23 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -151,7 +151,7 @@ void __delete_from_swap_cache(struct folio *folio, for (i = 0; i < nr; i++) { void *entry = xas_store(&xas, shadow); - VM_BUG_ON_FOLIO(entry != folio, folio); + VM_BUG_ON_PAGE(entry != folio, entry); set_page_private(folio_page(folio, i), 0); xas_next(&xas); } From patchwork Mon Aug 8 19:33:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939051 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E294AC00140 for ; Mon, 8 Aug 2022 19:35:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7B88094000B; Mon, 8 Aug 2022 15:35:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 73F9D940007; Mon, 8 Aug 2022 15:35:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5E0C794000B; Mon, 8 Aug 2022 15:35:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4948D940007 for ; Mon, 8 Aug 2022 15:35:12 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 322031C6449 for ; Mon, 8 Aug 2022 19:35:12 +0000 (UTC) X-FDA: 79777428864.01.745BE35 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id CB0A12017C for ; Mon, 8 Aug 2022 19:35:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=s4LgfjgMSKwodnJCulgKaAjoW6t9ZjF90g5FC669hGQ=; b=t2bXWmVScpPPjMd+JapjtU+qWL SRzNQkk6dpcKCvWIg5HeaC1qiplG03fkYj1DGdIdsA/a19UDoDYBTfEgfbedbVAhl61HYBhbFlD+e 3C/A8Xe4NT6cOQ9xLykyuR6OViT1thsiXdq1AfQH82Rt2j5e1cmvs4MGVFHiYPJtLrLIGZaJIgtCN ecyXGtMhBBGLfaSICuKtXzlPBSyL3+jHn7ZP2ZDcRce7dQ8+NBW+u7zfB/BL0kokwOzuCKxqnQhH4 zGiA7wGAjBRjJiihqbkKTzq6/1N14uOlhcxEECAFBfOZ7UzmjDoY8L13qUaliQ2iC9za2NWHX/pbU i1In/vFQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8WR-00EAs8-GG; Mon, 08 Aug 2022 19:34:45 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com, stable@vger.kernel.org, William Kucharski Subject: [PATCH 02/59] shmem: Update folio if shmem_replace_page() updates the page Date: Mon, 8 Aug 2022 20:33:30 +0100 Message-Id: <20220808193430.3378317-3-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987311; a=rsa-sha256; cv=none; b=ES6CxCXvO/9ZkG+87QZ7Mu2NujfXN+E30WpaM8v+ixIbpo4KnHHZ4EMEb7F7/lgNNLWkSz RI6iLEdypukjMNqpQL5cudFGci6BifflhqL6pU0ygKM8uaK7K/P/5BE0GCF/JVcfXygM2d YMyu2bPW1Xs9FDHY1bXZeF1f9hQSFow= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=t2bXWmVS; dmarc=none; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987311; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=s4LgfjgMSKwodnJCulgKaAjoW6t9ZjF90g5FC669hGQ=; b=a8cmZD4D6Lv7wmam/t7UY77sfh9d47FpbKhvEvSHveTFB+dsyNwitXalJQM9IEPQKsZjRc eqyPslsskphbb1vTJZ6t4JbIRJ22aNu0Bm1EzDLTHOSupVYfIBcZUFR2FoQVIBl7Si9Yp1 ovKvrjHy0W3qdFS8EYTk0AbdbQ87tKE= X-Rspamd-Server: rspam10 X-Stat-Signature: x5xege1h1qgjsoyuo9jbg9u5mwczuoc7 Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=t2bXWmVS; dmarc=none; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Rspamd-Queue-Id: CB0A12017C X-HE-Tag: 1659987311-636029 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If we allocate a new page, we need to make sure that our folio matches that new page. If we don't, we store the wrong folio in the shmem page cache which will lead to data corruption. This problem will be solved by changing shmem_replace_page() to shmem_replace_folio(), but this patch is the minimal fix. Fixes: da08e9b79323 ("mm/shmem: convert shmem_swapin_page() to shmem_swapin_folio()") Cc: stable@vger.kernel.org Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: William Kucharski --- mm/shmem.c | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/shmem.c b/mm/shmem.c index e975fcd9d2e1..4ae43cffeda3 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1780,6 +1780,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, if (shmem_should_replace_folio(folio, gfp)) { error = shmem_replace_page(&page, gfp, info, index); + folio = page_folio(page); if (error) goto failed; } From patchwork Mon Aug 8 19:33:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939040 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C077C25B08 for ; Mon, 8 Aug 2022 19:34:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 597838E0001; Mon, 8 Aug 2022 15:34:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 56CE26B0078; Mon, 8 Aug 2022 15:34:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA84F8E0002; Mon, 8 Aug 2022 15:34:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B61256B0071 for ; Mon, 8 Aug 2022 15:34:49 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 8B209160151 for ; Mon, 8 Aug 2022 19:34:49 +0000 (UTC) X-FDA: 79777427898.06.21C19E9 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf28.hostedemail.com (Postfix) with ESMTP id 9F27FC0150 for ; Mon, 8 Aug 2022 19:34:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=BqtDNBokS9uyBPWuNbawImIMA3ufUINE/LSTsHWrTgg=; b=hvE8wJBb3sQG/Jr4pyXYhefJrZ TJFA5M9i+hEogxJ9sUOPrnKD/VKaMTebPe6/58JZ6XuH27rw/rSJhApmbaMmIPkD36iUcsYBCZfps T5ofRlk/FggKS85KZqE/7JN2OvHYwCZZhBUa4YbW12VJMzHF1Ld600X7lKvqLEZFuq3l+DtvJzarx 2Zr2LCWZRYr75V13zYSs1+sTc+S1NPrRppwDeolBP0fGnNYaHLkq7I9wHSinD+NZ9xPWDcDtx7kNo 9FzftWSn0TxJL7NVo+KtNwADb7ifJX7Z2ezTBLUigltUPd2LBFirV5p57jcViD3S44n6GBQl/3+Fk aDQCY7IA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8WT-00EAsI-E2; Mon, 08 Aug 2022 19:34:45 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 03/59] vmscan: Check folio_test_private(), not folio_get_private() Date: Mon, 8 Aug 2022 20:33:31 +0100 Message-Id: <20220808193430.3378317-4-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987289; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BqtDNBokS9uyBPWuNbawImIMA3ufUINE/LSTsHWrTgg=; b=erIZsZYYm4QqgfuNc+GaoS6WLwbDirLWydzSlpQssgeeCQ8sKuJGu4kiP104xCK4q07SB5 MRUtS1aUGHPaY6nmvUm8KsYedIwlK/I5C5sY4Dju1SoRmAjY9C/CE/Q0mqmnUT01cDTQr6 dZQaMtYjtxRSa33hBQmQ7hSa6Qulcjw= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=hvE8wJBb; dmarc=none; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987289; a=rsa-sha256; cv=none; b=3Zuu8tBvT2Z7W6REnQ0DpB5ILj1+ItiHH5T5Jyf1ZrjjqDsEr5uY7LRNpuROFIw9JkJLfT JgMM75GdXwYeYpr7m8vBbl5BV0Cusivgw0iGcc/tB2WvjW9EQs6bcDbhwO0IQsWQzbFOTp sQP9lonwN7sLRa/41Cx4yqqNBrmEjR8= X-Rspamd-Queue-Id: 9F27FC0150 X-Rspamd-Server: rspam03 X-Rspam-User: Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=hvE8wJBb; dmarc=none; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Stat-Signature: t719xu4xqth83qr66adfkds6eraz1oku X-HE-Tag: 1659987288-185663 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: These two predicates are the same for file pages, but are not the same for anonymous pages. Reported-by: Hugh Dickins Fixes: 07f67a8dedc0 ("mm/vmscan: convert shrink_active_list() to use a folio") Signed-off-by: Matthew Wilcox (Oracle) --- mm/vmscan.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index b2b1431352dc..382dbe97329f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2550,8 +2550,8 @@ static void shrink_active_list(unsigned long nr_to_scan, } if (unlikely(buffer_heads_over_limit)) { - if (folio_get_private(folio) && folio_trylock(folio)) { - if (folio_get_private(folio)) + if (folio_test_private(folio) && folio_trylock(folio)) { + if (folio_test_private(folio)) filemap_release_folio(folio, 0); folio_unlock(folio); } From patchwork Mon Aug 8 19:33:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939041 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8B08C00140 for ; Mon, 8 Aug 2022 19:34:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D0DC6B0071; Mon, 8 Aug 2022 15:34:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7BA448E0002; Mon, 8 Aug 2022 15:34:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4C1D86B007B; Mon, 8 Aug 2022 15:34:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E187B8E0005 for ; Mon, 8 Aug 2022 15:34:49 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id BAA6316018F for ; Mon, 8 Aug 2022 19:34:49 +0000 (UTC) X-FDA: 79777427898.10.E24542D Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf21.hostedemail.com (Postfix) with ESMTP id 2625F1C005F for ; Mon, 8 Aug 2022 19:34:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=dhyA1+k3/oD84JmGsDqBjS76RHZcTcJfSFZtNWcKtjE=; b=slDTpdWtWHSi46Ha+7R2BdVaa0 bZxXpzLdUZDTK3vYj9NG7G1HHd2fMwWiuFghIWjaxQF5+XY9IzSc7Ygct4kzwChYhuhI+xsVWMKwA q07e3nGq2wNjo1MYxOlRkyv8RxiUUPsJ0deV/MK1FCA4waaMMdjRVAZK8zIP43GpmT7uJXga7LNZI O3OpIhaGU8WEZHRt9f2Y3wt15ATiEIJcX+w/uLVejmqJgHmSlZMXwl0JWf9owmSVwbDF5CaQ/wJaJ VJtinATFHRM/W0A5nRlq790ciMGb4A/imQTdrZeLIbRs+JaRHva1MO7v6WUxqpNK2FgLgfEwh7J2t bS963gPQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8WT-00EAsK-Qo; Mon, 08 Aug 2022 19:34:45 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 04/59] mm/vmscan: Fix a lot of comments Date: Mon, 8 Aug 2022 20:33:32 +0100 Message-Id: <20220808193430.3378317-5-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987289; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dhyA1+k3/oD84JmGsDqBjS76RHZcTcJfSFZtNWcKtjE=; b=RvPUqAxeYK1y7lRFYbarg+ZPT1eTS9qoPmUfMy430mG8RQ+Qd6/qBa/uPjY3WCFi9KIN0r gwv6bl041qnpuIE5AzjLZA+rUzLLK35jQv/us8l7U1yePZ+4lTyHlNfi2XYq1fRVCdrfQb a09ho5fehhhMlV2NDt8zDFVB3fGaOg0= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=slDTpdWt; dmarc=none; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987289; a=rsa-sha256; cv=none; b=gisoZbGswKjOHn1Qk8STNOqgCEQ7rSECqGZfFSx1eVoSUjIhXiruXmlTrjaAJhL8eHfBgp iSevdCjCnZy+PxttqvW02v/G3X5ATsJvbLwkZRxhRxZ46nXByTzXsTRbe1Ajs6LJXOVLcH zsLoRzB3UqgLg5odQmTvxACB6x4Uye8= Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=slDTpdWt; dmarc=none; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspamd-Server: rspam02 X-Stat-Signature: dzem4x1t6h95or1cyity553fc7w3mo8b X-Rspamd-Queue-Id: 2625F1C005F X-Rspam-User: X-HE-Tag: 1659987288-776485 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A lot of comments mention pages when they should say folios. Fix them up. Signed-off-by: Matthew Wilcox (Oracle) --- mm/vmscan.c | 252 ++++++++++++++++++++++++++-------------------------- 1 file changed, 125 insertions(+), 127 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 382dbe97329f..93b1087fc969 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -85,7 +85,7 @@ struct scan_control { unsigned long anon_cost; unsigned long file_cost; - /* Can active pages be deactivated as part of reclaim? */ + /* Can active folios be deactivated as part of reclaim? */ #define DEACTIVATE_ANON 1 #define DEACTIVATE_FILE 2 unsigned int may_deactivate:2; @@ -95,10 +95,10 @@ struct scan_control { /* Writepage batching in laptop mode; RECLAIM_WRITE */ unsigned int may_writepage:1; - /* Can mapped pages be reclaimed? */ + /* Can mapped folios be reclaimed? */ unsigned int may_unmap:1; - /* Can pages be swapped as part of reclaim? */ + /* Can folios be swapped as part of reclaim? */ unsigned int may_swap:1; /* Proactive reclaim invoked by userspace through memory.reclaim */ @@ -123,7 +123,7 @@ struct scan_control { /* There is easily reclaimable cold cache in the current node */ unsigned int cache_trim_mode:1; - /* The file pages on the current node are dangerously low */ + /* The file folios on the current node are dangerously low */ unsigned int file_is_tiny:1; /* Always discard instead of demoting to lower tier memory */ @@ -135,7 +135,7 @@ struct scan_control { /* Scan (total_size >> priority) pages at once */ s8 priority; - /* The highest zone to isolate pages for reclaim from */ + /* The highest zone to isolate folios for reclaim from */ s8 reclaim_idx; /* This context's GFP mask */ @@ -443,7 +443,7 @@ static bool cgroup_reclaim(struct scan_control *sc) * * The normal page dirty throttling mechanism in balance_dirty_pages() is * completely broken with the legacy memcg and direct stalling in - * shrink_page_list() is used for throttling instead, which lacks all the + * shrink_folio_list() is used for throttling instead, which lacks all the * niceties such as fairness, adaptive pausing, bandwidth proportional * allocation and configurability. * @@ -564,9 +564,9 @@ static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg, } /* - * This misses isolated pages which are not accounted for to save counters. + * This misses isolated folios which are not accounted for to save counters. * As the data only determines if reclaim or compaction continues, it is - * not expected that isolated pages will be a dominating factor. + * not expected that isolated folios will be a dominating factor. */ unsigned long zone_reclaimable_pages(struct zone *zone) { @@ -1039,9 +1039,9 @@ void drop_slab(void) static inline int is_page_cache_freeable(struct folio *folio) { /* - * A freeable page cache page is referenced only by the caller - * that isolated the page, the page cache and optional buffer - * heads at page->private. + * A freeable page cache folio is referenced only by the caller + * that isolated the folio, the page cache and optional filesystem + * private data at folio->private. */ return folio_ref_count(folio) - folio_test_private(folio) == 1 + folio_nr_pages(folio); @@ -1081,8 +1081,8 @@ static bool skip_throttle_noprogress(pg_data_t *pgdat) return true; /* - * If there are a lot of dirty/writeback pages then do not - * throttle as throttling will occur when the pages cycle + * If there are a lot of dirty/writeback folios then do not + * throttle as throttling will occur when the folios cycle * towards the end of the LRU if still under writeback. */ for (i = 0; i < MAX_NR_ZONES; i++) { @@ -1125,7 +1125,7 @@ void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason) * short. Failing to make progress or waiting on writeback are * potentially long-lived events so use a longer timeout. This is shaky * logic as a failure to make progress could be due to anything from - * writeback to a slow device to excessive references pages at the tail + * writeback to a slow device to excessive referenced folios at the tail * of the inactive LRU. */ switch(reason) { @@ -1171,8 +1171,8 @@ void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason) } /* - * Account for pages written if tasks are throttled waiting on dirty - * pages to clean. If enough pages have been cleaned since throttling + * Account for folios written if tasks are throttled waiting on dirty + * folios to clean. If enough folios have been cleaned since throttling * started then wakeup the throttled tasks. */ void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio, @@ -1198,18 +1198,18 @@ void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio, /* possible outcome of pageout() */ typedef enum { - /* failed to write page out, page is locked */ + /* failed to write folio out, folio is locked */ PAGE_KEEP, - /* move page to the active list, page is locked */ + /* move folio to the active list, folio is locked */ PAGE_ACTIVATE, - /* page has been sent to the disk successfully, page is unlocked */ + /* folio has been sent to the disk successfully, folio is unlocked */ PAGE_SUCCESS, - /* page is clean and locked */ + /* folio is clean and locked */ PAGE_CLEAN, } pageout_t; /* - * pageout is called by shrink_page_list() for each dirty page. + * pageout is called by shrink_folio_list() for each dirty folio. * Calls ->writepage(). */ static pageout_t pageout(struct folio *folio, struct address_space *mapping, @@ -1283,7 +1283,7 @@ static pageout_t pageout(struct folio *folio, struct address_space *mapping, } /* - * Same as remove_mapping, but if the page is removed from the mapping, it + * Same as remove_mapping, but if the folio is removed from the mapping, it * gets returned with a refcount of 0. */ static int __remove_mapping(struct address_space *mapping, struct folio *folio, @@ -1299,34 +1299,34 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio, spin_lock(&mapping->host->i_lock); xa_lock_irq(&mapping->i_pages); /* - * The non racy check for a busy page. + * The non racy check for a busy folio. * * Must be careful with the order of the tests. When someone has - * a ref to the page, it may be possible that they dirty it then - * drop the reference. So if PageDirty is tested before page_count - * here, then the following race may occur: + * a ref to the folio, it may be possible that they dirty it then + * drop the reference. So if the dirty flag is tested before the + * refcount here, then the following race may occur: * * get_user_pages(&page); * [user mapping goes away] * write_to(page); - * !PageDirty(page) [good] - * SetPageDirty(page); - * put_page(page); - * !page_count(page) [good, discard it] + * !folio_test_dirty(folio) [good] + * folio_set_dirty(folio); + * folio_put(folio); + * !refcount(folio) [good, discard it] * * [oops, our write_to data is lost] * * Reversing the order of the tests ensures such a situation cannot - * escape unnoticed. The smp_rmb is needed to ensure the page->flags - * load is not satisfied before that of page->_refcount. + * escape unnoticed. The smp_rmb is needed to ensure the folio->flags + * load is not satisfied before that of folio->_refcount. * - * Note that if SetPageDirty is always performed via set_page_dirty, + * Note that if the dirty flag is always set via folio_mark_dirty, * and thus under the i_pages lock, then this ordering is not required. */ refcount = 1 + folio_nr_pages(folio); if (!folio_ref_freeze(folio, refcount)) goto cannot_free; - /* note: atomic_cmpxchg in page_ref_freeze provides the smp_rmb */ + /* note: atomic_cmpxchg in folio_ref_freeze provides the smp_rmb */ if (unlikely(folio_test_dirty(folio))) { folio_ref_unfreeze(folio, refcount); goto cannot_free; @@ -1355,7 +1355,7 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio, * back. * * We also don't store shadows for DAX mappings because the - * only page cache pages found in these are zero pages + * only page cache folios found in these are zero pages * covering holes, and because we don't want to mix DAX * exceptional entries and shadow exceptional entries in the * same address_space. @@ -1423,14 +1423,14 @@ void folio_putback_lru(struct folio *folio) folio_put(folio); /* drop ref from isolate */ } -enum page_references { - PAGEREF_RECLAIM, - PAGEREF_RECLAIM_CLEAN, - PAGEREF_KEEP, - PAGEREF_ACTIVATE, +enum folio_references { + FOLIOREF_RECLAIM, + FOLIOREF_RECLAIM_CLEAN, + FOLIOREF_KEEP, + FOLIOREF_ACTIVATE, }; -static enum page_references folio_check_references(struct folio *folio, +static enum folio_references folio_check_references(struct folio *folio, struct scan_control *sc) { int referenced_ptes, referenced_folio; @@ -1445,11 +1445,11 @@ static enum page_references folio_check_references(struct folio *folio, * Let the folio, now marked Mlocked, be moved to the unevictable list. */ if (vm_flags & VM_LOCKED) - return PAGEREF_ACTIVATE; + return FOLIOREF_ACTIVATE; /* rmap lock contention: rotate */ if (referenced_ptes == -1) - return PAGEREF_KEEP; + return FOLIOREF_KEEP; if (referenced_ptes) { /* @@ -1469,34 +1469,34 @@ static enum page_references folio_check_references(struct folio *folio, folio_set_referenced(folio); if (referenced_folio || referenced_ptes > 1) - return PAGEREF_ACTIVATE; + return FOLIOREF_ACTIVATE; /* * Activate file-backed executable folios after first usage. */ if ((vm_flags & VM_EXEC) && folio_is_file_lru(folio)) - return PAGEREF_ACTIVATE; + return FOLIOREF_ACTIVATE; - return PAGEREF_KEEP; + return FOLIOREF_KEEP; } /* Reclaim if clean, defer dirty folios to writeback */ if (referenced_folio && folio_is_file_lru(folio)) - return PAGEREF_RECLAIM_CLEAN; + return FOLIOREF_RECLAIM_CLEAN; - return PAGEREF_RECLAIM; + return FOLIOREF_RECLAIM; } -/* Check if a page is dirty or under writeback */ +/* Check if a folio is dirty or under writeback */ static void folio_check_dirty_writeback(struct folio *folio, bool *dirty, bool *writeback) { struct address_space *mapping; /* - * Anonymous pages are not handled by flushers and must be written + * Anonymous folios are not handled by flushers and must be written * from reclaim context. Do not stall reclaim based on them. - * MADV_FREE anonymous pages are put into inactive file list too. + * MADV_FREE anonymous folios are put into inactive file list too. * They could be mistakenly treated as file lru. So further anon * test is needed. */ @@ -1538,24 +1538,24 @@ static struct page *alloc_demote_page(struct page *page, unsigned long node) } /* - * Take pages on @demote_list and attempt to demote them to - * another node. Pages which are not demoted are left on - * @demote_pages. + * Take folios on @demote_folios and attempt to demote them to + * another node. Folios which are not demoted are left on + * @demote_folios. */ -static unsigned int demote_page_list(struct list_head *demote_pages, +static unsigned int demote_folio_list(struct list_head *demote_folios, struct pglist_data *pgdat) { int target_nid = next_demotion_node(pgdat->node_id); unsigned int nr_succeeded; - if (list_empty(demote_pages)) + if (list_empty(demote_folios)) return 0; if (target_nid == NUMA_NO_NODE) return 0; /* Demotion ignores all cpuset and mempolicy settings */ - migrate_pages(demote_pages, alloc_demote_page, NULL, + migrate_pages(demote_folios, alloc_demote_page, NULL, target_nid, MIGRATE_ASYNC, MR_DEMOTION, &nr_succeeded); @@ -1584,17 +1584,15 @@ static bool may_enter_fs(struct folio *folio, gfp_t gfp_mask) } /* - * shrink_page_list() returns the number of reclaimed pages + * shrink_folio_list() returns the number of reclaimed pages */ -static unsigned int shrink_page_list(struct list_head *page_list, - struct pglist_data *pgdat, - struct scan_control *sc, - struct reclaim_stat *stat, - bool ignore_references) +static unsigned int shrink_folio_list(struct list_head *folio_list, + struct pglist_data *pgdat, struct scan_control *sc, + struct reclaim_stat *stat, bool ignore_references) { - LIST_HEAD(ret_pages); - LIST_HEAD(free_pages); - LIST_HEAD(demote_pages); + LIST_HEAD(ret_folios); + LIST_HEAD(free_folios); + LIST_HEAD(demote_folios); unsigned int nr_reclaimed = 0; unsigned int pgactivate = 0; bool do_demote_pass; @@ -1605,16 +1603,16 @@ static unsigned int shrink_page_list(struct list_head *page_list, do_demote_pass = can_demote(pgdat->node_id, sc); retry: - while (!list_empty(page_list)) { + while (!list_empty(folio_list)) { struct address_space *mapping; struct folio *folio; - enum page_references references = PAGEREF_RECLAIM; + enum folio_references references = FOLIOREF_RECLAIM; bool dirty, writeback; unsigned int nr_pages; cond_resched(); - folio = lru_to_folio(page_list); + folio = lru_to_folio(folio_list); list_del(&folio->lru); if (!folio_trylock(folio)) @@ -1733,7 +1731,7 @@ static unsigned int shrink_page_list(struct list_head *page_list, folio_unlock(folio); folio_wait_writeback(folio); /* then go back and try same folio again */ - list_add_tail(&folio->lru, page_list); + list_add_tail(&folio->lru, folio_list); continue; } } @@ -1742,13 +1740,13 @@ static unsigned int shrink_page_list(struct list_head *page_list, references = folio_check_references(folio, sc); switch (references) { - case PAGEREF_ACTIVATE: + case FOLIOREF_ACTIVATE: goto activate_locked; - case PAGEREF_KEEP: + case FOLIOREF_KEEP: stat->nr_ref_keep += nr_pages; goto keep_locked; - case PAGEREF_RECLAIM: - case PAGEREF_RECLAIM_CLEAN: + case FOLIOREF_RECLAIM: + case FOLIOREF_RECLAIM_CLEAN: ; /* try to reclaim the folio below */ } @@ -1758,7 +1756,7 @@ static unsigned int shrink_page_list(struct list_head *page_list, */ if (do_demote_pass && (thp_migration_supported() || !folio_test_large(folio))) { - list_add(&folio->lru, &demote_pages); + list_add(&folio->lru, &demote_folios); folio_unlock(folio); continue; } @@ -1785,7 +1783,7 @@ static unsigned int shrink_page_list(struct list_head *page_list, */ if (!folio_entire_mapcount(folio) && split_folio_to_list(folio, - page_list)) + folio_list)) goto activate_locked; } if (!add_to_swap(folio)) { @@ -1793,7 +1791,7 @@ static unsigned int shrink_page_list(struct list_head *page_list, goto activate_locked_split; /* Fallback to swap normal pages */ if (split_folio_to_list(folio, - page_list)) + folio_list)) goto activate_locked; #ifdef CONFIG_TRANSPARENT_HUGEPAGE count_vm_event(THP_SWPOUT_FALLBACK); @@ -1805,7 +1803,7 @@ static unsigned int shrink_page_list(struct list_head *page_list, } else if (folio_test_swapbacked(folio) && folio_test_large(folio)) { /* Split shmem folio */ - if (split_folio_to_list(folio, page_list)) + if (split_folio_to_list(folio, folio_list)) goto keep_locked; } @@ -1870,7 +1868,7 @@ static unsigned int shrink_page_list(struct list_head *page_list, goto activate_locked; } - if (references == PAGEREF_RECLAIM_CLEAN) + if (references == FOLIOREF_RECLAIM_CLEAN) goto keep_locked; if (!may_enter_fs(folio, sc->gfp_mask)) goto keep_locked; @@ -1983,13 +1981,13 @@ static unsigned int shrink_page_list(struct list_head *page_list, nr_reclaimed += nr_pages; /* - * Is there need to periodically free_page_list? It would + * Is there need to periodically free_folio_list? It would * appear not as the counts should be low */ if (unlikely(folio_test_large(folio))) destroy_large_folio(folio); else - list_add(&folio->lru, &free_pages); + list_add(&folio->lru, &free_folios); continue; activate_locked_split: @@ -2017,29 +2015,29 @@ static unsigned int shrink_page_list(struct list_head *page_list, keep_locked: folio_unlock(folio); keep: - list_add(&folio->lru, &ret_pages); + list_add(&folio->lru, &ret_folios); VM_BUG_ON_FOLIO(folio_test_lru(folio) || folio_test_unevictable(folio), folio); } - /* 'page_list' is always empty here */ + /* 'folio_list' is always empty here */ /* Migrate folios selected for demotion */ - nr_reclaimed += demote_page_list(&demote_pages, pgdat); - /* Folios that could not be demoted are still in @demote_pages */ - if (!list_empty(&demote_pages)) { - /* Folios which weren't demoted go back on @page_list for retry: */ - list_splice_init(&demote_pages, page_list); + nr_reclaimed += demote_folio_list(&demote_folios, pgdat); + /* Folios that could not be demoted are still in @demote_folios */ + if (!list_empty(&demote_folios)) { + /* Folios which weren't demoted go back on @folio_list for retry: */ + list_splice_init(&demote_folios, folio_list); do_demote_pass = false; goto retry; } pgactivate = stat->nr_activate[0] + stat->nr_activate[1]; - mem_cgroup_uncharge_list(&free_pages); + mem_cgroup_uncharge_list(&free_folios); try_to_unmap_flush(); - free_unref_page_list(&free_pages); + free_unref_page_list(&free_folios); - list_splice(&ret_pages, page_list); + list_splice(&ret_folios, folio_list); count_vm_events(PGACTIVATE, pgactivate); if (plug) @@ -2048,7 +2046,7 @@ static unsigned int shrink_page_list(struct list_head *page_list, } unsigned int reclaim_clean_pages_from_list(struct zone *zone, - struct list_head *folio_list) + struct list_head *folio_list) { struct scan_control sc = { .gfp_mask = GFP_KERNEL, @@ -2076,7 +2074,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone, * change in the future. */ noreclaim_flag = memalloc_noreclaim_save(); - nr_reclaimed = shrink_page_list(&clean_folios, zone->zone_pgdat, &sc, + nr_reclaimed = shrink_folio_list(&clean_folios, zone->zone_pgdat, &sc, &stat, true); memalloc_noreclaim_restore(noreclaim_flag); @@ -2135,7 +2133,7 @@ static __always_inline void update_lru_sizes(struct lruvec *lruvec, * * returns how many pages were moved onto *@dst. */ -static unsigned long isolate_lru_pages(unsigned long nr_to_scan, +static unsigned long isolate_lru_folios(unsigned long nr_to_scan, struct lruvec *lruvec, struct list_head *dst, unsigned long *nr_scanned, struct scan_control *sc, enum lru_list lru) @@ -2242,8 +2240,8 @@ static unsigned long isolate_lru_pages(unsigned long nr_to_scan, * * Context: * - * (1) Must be called with an elevated refcount on the page. This is a - * fundamental difference from isolate_lru_pages() (which is called + * (1) Must be called with an elevated refcount on the folio. This is a + * fundamental difference from isolate_lru_folios() (which is called * without a stable reference). * (2) The lru_lock must not be held. * (3) Interrupts must be enabled. @@ -2315,13 +2313,13 @@ static int too_many_isolated(struct pglist_data *pgdat, int file, } /* - * move_pages_to_lru() moves folios from private @list to appropriate LRU list. + * move_folios_to_lru() moves folios from private @list to appropriate LRU list. * On return, @list is reused as a list of folios to be freed by the caller. * * Returns the number of pages moved to the given lruvec. */ -static unsigned int move_pages_to_lru(struct lruvec *lruvec, - struct list_head *list) +static unsigned int move_folios_to_lru(struct lruvec *lruvec, + struct list_head *list) { int nr_pages, nr_moved = 0; LIST_HEAD(folios_to_free); @@ -2341,7 +2339,7 @@ static unsigned int move_pages_to_lru(struct lruvec *lruvec, /* * The folio_set_lru needs to be kept here for list integrity. * Otherwise: - * #0 move_pages_to_lru #1 release_pages + * #0 move_folios_to_lru #1 release_pages * if (!folio_put_testzero()) * if (folio_put_testzero()) * !lru //skip lru_lock @@ -2398,11 +2396,11 @@ static int current_may_throttle(void) * shrink_inactive_list() is a helper for shrink_node(). It returns the number * of reclaimed pages */ -static unsigned long -shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec, - struct scan_control *sc, enum lru_list lru) +static unsigned long shrink_inactive_list(unsigned long nr_to_scan, + struct lruvec *lruvec, struct scan_control *sc, + enum lru_list lru) { - LIST_HEAD(page_list); + LIST_HEAD(folio_list); unsigned long nr_scanned; unsigned int nr_reclaimed = 0; unsigned long nr_taken; @@ -2429,7 +2427,7 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec, spin_lock_irq(&lruvec->lru_lock); - nr_taken = isolate_lru_pages(nr_to_scan, lruvec, &page_list, + nr_taken = isolate_lru_folios(nr_to_scan, lruvec, &folio_list, &nr_scanned, sc, lru); __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, nr_taken); @@ -2444,10 +2442,10 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec, if (nr_taken == 0) return 0; - nr_reclaimed = shrink_page_list(&page_list, pgdat, sc, &stat, false); + nr_reclaimed = shrink_folio_list(&folio_list, pgdat, sc, &stat, false); spin_lock_irq(&lruvec->lru_lock); - move_pages_to_lru(lruvec, &page_list); + move_folios_to_lru(lruvec, &folio_list); __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken); item = current_is_kswapd() ? PGSTEAL_KSWAPD : PGSTEAL_DIRECT; @@ -2458,16 +2456,16 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec, spin_unlock_irq(&lruvec->lru_lock); lru_note_cost(lruvec, file, stat.nr_pageout); - mem_cgroup_uncharge_list(&page_list); - free_unref_page_list(&page_list); + mem_cgroup_uncharge_list(&folio_list); + free_unref_page_list(&folio_list); /* - * If dirty pages are scanned that are not queued for IO, it + * If dirty folios are scanned that are not queued for IO, it * implies that flushers are not doing their job. This can - * happen when memory pressure pushes dirty pages to the end of + * happen when memory pressure pushes dirty folios to the end of * the LRU before the dirty limits are breached and the dirty * data has expired. It can also happen when the proportion of - * dirty pages grows not through writes but through memory + * dirty folios grows not through writes but through memory * pressure reclaiming all the clean cache. And in some cases, * the flushers simply cannot keep up with the allocation * rate. Nudge the flusher threads in case they are asleep. @@ -2526,7 +2524,7 @@ static void shrink_active_list(unsigned long nr_to_scan, spin_lock_irq(&lruvec->lru_lock); - nr_taken = isolate_lru_pages(nr_to_scan, lruvec, &l_hold, + nr_taken = isolate_lru_folios(nr_to_scan, lruvec, &l_hold, &nr_scanned, sc, lru); __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, nr_taken); @@ -2586,8 +2584,8 @@ static void shrink_active_list(unsigned long nr_to_scan, */ spin_lock_irq(&lruvec->lru_lock); - nr_activate = move_pages_to_lru(lruvec, &l_active); - nr_deactivate = move_pages_to_lru(lruvec, &l_inactive); + nr_activate = move_folios_to_lru(lruvec, &l_active); + nr_deactivate = move_folios_to_lru(lruvec, &l_inactive); /* Keep all free folios in l_active list */ list_splice(&l_inactive, &l_active); @@ -2603,7 +2601,7 @@ static void shrink_active_list(unsigned long nr_to_scan, nr_deactivate, nr_rotated, sc->priority, file); } -static unsigned int reclaim_page_list(struct list_head *page_list, +static unsigned int reclaim_folio_list(struct list_head *folio_list, struct pglist_data *pgdat) { struct reclaim_stat dummy_stat; @@ -2617,9 +2615,9 @@ static unsigned int reclaim_page_list(struct list_head *page_list, .no_demotion = 1, }; - nr_reclaimed = shrink_page_list(page_list, pgdat, &sc, &dummy_stat, false); - while (!list_empty(page_list)) { - folio = lru_to_folio(page_list); + nr_reclaimed = shrink_folio_list(folio_list, pgdat, &sc, &dummy_stat, false); + while (!list_empty(folio_list)) { + folio = lru_to_folio(folio_list); list_del(&folio->lru); folio_putback_lru(folio); } @@ -2649,11 +2647,11 @@ unsigned long reclaim_pages(struct list_head *folio_list) continue; } - nr_reclaimed += reclaim_page_list(&node_folio_list, NODE_DATA(nid)); + nr_reclaimed += reclaim_folio_list(&node_folio_list, NODE_DATA(nid)); nid = folio_nid(lru_to_folio(folio_list)); } while (!list_empty(folio_list)); - nr_reclaimed += reclaim_page_list(&node_folio_list, NODE_DATA(nid)); + nr_reclaimed += reclaim_folio_list(&node_folio_list, NODE_DATA(nid)); memalloc_noreclaim_restore(noreclaim_flag); @@ -2683,13 +2681,13 @@ static unsigned long shrink_list(enum lru_list lru, unsigned long nr_to_scan, * but large enough to avoid thrashing the aggregate readahead window. * * Both inactive lists should also be large enough that each inactive - * page has a chance to be referenced again before it is reclaimed. + * folio has a chance to be referenced again before it is reclaimed. * * If that fails and refaulting is observed, the inactive list grows. * - * The inactive_ratio is the target ratio of ACTIVE to INACTIVE pages + * The inactive_ratio is the target ratio of ACTIVE to INACTIVE folios * on this LRU, maintained by the pageout code. An inactive_ratio - * of 3 means 3:1 or 25% of the pages are kept on the inactive list. + * of 3 means 3:1 or 25% of the folios are kept on the inactive list. * * total target max * memory ratio inactive @@ -2732,8 +2730,8 @@ enum scan_balance { * Determine how aggressively the anon and file LRU lists should be * scanned. * - * nr[0] = anon inactive pages to scan; nr[1] = anon active pages to scan - * nr[2] = file inactive pages to scan; nr[3] = file active pages to scan + * nr[0] = anon inactive folios to scan; nr[1] = anon active folios to scan + * nr[2] = file inactive folios to scan; nr[3] = file active folios to scan */ static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc, unsigned long *nr) @@ -2748,7 +2746,7 @@ static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc, unsigned long ap, fp; enum lru_list lru; - /* If we have no swap space, do not bother scanning anon pages. */ + /* If we have no swap space, do not bother scanning anon folios. */ if (!sc->may_swap || !can_reclaim_anon_pages(memcg, pgdat->node_id, sc)) { scan_balance = SCAN_FILE; goto out; From patchwork Mon Aug 8 19:33:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939042 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 564E7C25B08 for ; Mon, 8 Aug 2022 19:34:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C95486B0075; Mon, 8 Aug 2022 15:34:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BFB456B0074; Mon, 8 Aug 2022 15:34:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 829DD8E0005; Mon, 8 Aug 2022 15:34:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 10A576B0071 for ; Mon, 8 Aug 2022 15:34:50 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id DB4E7A0B7B for ; Mon, 8 Aug 2022 19:34:49 +0000 (UTC) X-FDA: 79777427898.04.072E1AD Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf24.hostedemail.com (Postfix) with ESMTP id 8487E18017A for ; Mon, 8 Aug 2022 19:34:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=D9OxPNakbQ7dXKQCeoqgLhnSNcTIRcyWrbUHuZ9BacA=; b=NPX6uQWh6ILJI7341MGeUAJfzE 7z59gMsRaWJdTSStyg8Xj1WMPn0riCNnTiZzhYTE4uVgF9xjxhI4iUqx4pgwphxr+h46GI5pSJrm+ KydgmHP3tHKwd7PaIbx5fpgh8/DJSOoB8gpVawNVfYGCWVCI8PFfEF1TVp7+VI0/I/SviXcpZvM1O TmgvnQ0oymn1NolYAOfTpMZKk92llD4JNhZl4IBCPtz51qQKoFvKcZtQ1UVUy0ClzqyZsdhkTbepn FRrVzKWizmGk5ffAv0JL/zvinkBFYlJQ0IwD37OjS5kWeohE6wfhDcg9x4YTRWkcI9R6AB1jZm0kr BxggQb1A==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8WU-00EAsQ-3Z; Mon, 08 Aug 2022 19:34:46 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 05/59] mm: Add the first tail page to struct folio Date: Mon, 8 Aug 2022 20:33:33 +0100 Message-Id: <20220808193430.3378317-6-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=NPX6uQWh; spf=none (imf24.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987289; a=rsa-sha256; cv=none; b=swZ531NxIZyE0EBeOdFp0QoJR0SCdi14jC408GO5/F8L4qb+qgSQa5I9Ea2WE4bpP1qYdt 6OiSQdC3zcPYIbv4osmlSdnSY58k5GfXKDDf1Ihs6FbKQayyLHuKo/EYvbDcn0zHW77EAA 9DastaM8gZz5udt+f/L7gDwhz3rKQt4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987289; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=D9OxPNakbQ7dXKQCeoqgLhnSNcTIRcyWrbUHuZ9BacA=; b=F1EF6DWk4OtLHq9TTP1wGgXVji5yHm5xKTyIHh+3YernjFTjB3ZCeU/W+vx7IHqeU25CCN zKvR1pPG2b8uRy947/IK+/4i5Z0b+S7DAGTjab34gJJ1kCGkUruQ0Jmr8RTW5AT5DIBBR1 rvE78ouUD5XtrsSH6oLzIgbdc2eTOfs= X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 8487E18017A X-Rspam-User: Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=NPX6uQWh; spf=none (imf24.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Stat-Signature: yci7y4rjursjsyyu79knznibe8o3wbq9 X-HE-Tag: 1659987289-688124 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some of the static checkers get confused by extracting the page from the folio and referring to fields in the first tail page. Adding these fields to struct folio lets us avoid doing that. It has the risk that people will refer to those fields without checking that the folio is actually a large folio, so prefix them with underscores and document the preferred function to use instead. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm_types.h | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index cf97f3884fda..8a9ee9d24973 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -244,6 +244,13 @@ struct page { * @_refcount: Do not access this member directly. Use folio_ref_count() * to find how many references there are to this folio. * @memcg_data: Memory Control Group data. + * @_flags_1: For large folios, additional page flags. + * @__head: Points to the folio. Do not use. + * @_folio_dtor: Which destructor to use for this folio. + * @_folio_order: Do not use directly, call folio_order(). + * @_total_mapcount: Do not use directly, call folio_entire_mapcount(). + * @_pincount: Do not use directly, call folio_maybe_dma_pinned(). + * @_folio_nr_pages: Do not use directly, call folio_nr_pages(). * * A folio is a physically, virtually and logically contiguous set * of bytes. It is a power-of-two in size, and it is aligned to that @@ -282,9 +289,17 @@ struct folio { }; struct page page; }; + unsigned long _flags_1; + unsigned long __head; + unsigned char _folio_dtor; + unsigned char _folio_order; + atomic_t _total_mapcount; + atomic_t _pincount; +#ifdef CONFIG_64BIT + unsigned int _folio_nr_pages; +#endif }; -static_assert(sizeof(struct page) == sizeof(struct folio)); #define FOLIO_MATCH(pg, fl) \ static_assert(offsetof(struct page, pg) == offsetof(struct folio, fl)) FOLIO_MATCH(flags, flags); @@ -299,6 +314,19 @@ FOLIO_MATCH(_refcount, _refcount); FOLIO_MATCH(memcg_data, memcg_data); #endif #undef FOLIO_MATCH +#define FOLIO_MATCH(pg, fl) \ + static_assert(offsetof(struct folio, fl) == \ + offsetof(struct page, pg) + sizeof(struct page)) +FOLIO_MATCH(flags, _flags_1); +FOLIO_MATCH(compound_head, __head); +FOLIO_MATCH(compound_dtor, _folio_dtor); +FOLIO_MATCH(compound_order, _folio_order); +FOLIO_MATCH(compound_mapcount, _total_mapcount); +FOLIO_MATCH(compound_pincount, _pincount); +#ifdef CONFIG_64BIT +FOLIO_MATCH(compound_nr, _folio_nr_pages); +#endif +#undef FOLIO_MATCH static inline atomic_t *folio_mapcount_ptr(struct folio *folio) { From patchwork Mon Aug 8 19:33:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939039 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48C1EC25B0C for ; Mon, 8 Aug 2022 19:34:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1C0506B0073; Mon, 8 Aug 2022 15:34:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EE17C8E0006; Mon, 8 Aug 2022 15:34:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C49386B0075; Mon, 8 Aug 2022 15:34:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A16438E0001 for ; Mon, 8 Aug 2022 15:34:49 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7427D1C5CB5 for ; Mon, 8 Aug 2022 19:34:49 +0000 (UTC) X-FDA: 79777427898.22.88B7375 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf22.hostedemail.com (Postfix) with ESMTP id 22874C012E for ; Mon, 8 Aug 2022 19:34:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=PiHBVuKwdKoB1+T6CMjhocZsD9aMxQ9a1/As9MIyMJs=; b=GCLusaoJrOn7UVUelevIz+vrvc A2KdNtTS8pXefXkAFre00lDAjhncwu7N1+lUzdHH9Jq4v88c13kHPW2BHvMl5l4b6ljmWgbt+HDJu ny4Rihjo2tGg1RmXPy63d3GpAxfyp/JuwIdcVo8hkDaKiEnYweF5a8QFtLvqNgbUafpl1P7Z3FY4g 02SYF+ZaNjdjlqCxNoREKNAXqN7Cia+QGOZuDlTemYVQQmaM/BcFDhhvR8+ZL7BNiN5j6too9CjkU QhieHsyB9u2BcsEF4YsdRnmxdTDXGuBFC1gfb/HetAecwvwsPfmAlWDSUlebVtVsCOJMZtP1z+mas ecq7Zu7g==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8WU-00EAsW-CV; Mon, 08 Aug 2022 19:34:46 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 06/59] mm: Reimplement folio_order() and folio_nr_pages() Date: Mon, 8 Aug 2022 20:33:34 +0100 Message-Id: <20220808193430.3378317-7-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987289; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PiHBVuKwdKoB1+T6CMjhocZsD9aMxQ9a1/As9MIyMJs=; b=UHNNjckD3mbMM+8D6pEOri3fZmL0NRjOFCop2qkyQEo1MlPdC4LYbzcLd+cA2RTcQ+nVY8 XeQURLmjP1SoI+fHrvh8Bq4xEUdZ1XILDpuiedxObffc5h2oqIrTD2Op4+Z50Y2SfL9PDs hrK8l+tJbq6AYqZJMIdhUzFBZ3j0MYE= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=GCLusaoJ; dmarc=none; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987289; a=rsa-sha256; cv=none; b=dV37PXJHeO1F6Hy5aq8xXjmmuQGRunrdZDN3VqB98PmJvEL7EC93pwLtS2NBLsVAaWIOXD kVxZV2Af+HupwjJHuZ8MxXRQbLmlhqDEKQ+pNabPJ8OU8WrdMSGpz7EH9A8whVbhdK9gkj 4K1UJBX0WHx7XwRZ/6cej5sNtmzALFo= Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=GCLusaoJ; dmarc=none; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspamd-Server: rspam02 X-Stat-Signature: cuawmcae65dpqoukwozxrsmc4fyjbmc3 X-Rspamd-Queue-Id: 22874C012E X-Rspam-User: X-HE-Tag: 1659987287-771957 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Instead of calling compound_order() and compound_nr_pages(), use the folio directly. Saves 1905 bytes from mm/filemap.o due to folio_test_large() now being a cheaper check than PageHead(). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 18e01474cf6b..7ee34dcf1aa3 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -697,7 +697,9 @@ static inline unsigned int compound_order(struct page *page) */ static inline unsigned int folio_order(struct folio *folio) { - return compound_order(&folio->page); + if (!folio_test_large(folio)) + return 0; + return folio->_folio_order; } #include @@ -1590,7 +1592,13 @@ static inline void set_page_links(struct page *page, enum zone_type zone, */ static inline long folio_nr_pages(struct folio *folio) { - return compound_nr(&folio->page); + if (!folio_test_large(folio)) + return 1; +#ifdef CONFIG_64BIT + return folio->_folio_nr_pages; +#else + return 1L << folio->_folio_order; +#endif } /** From patchwork Mon Aug 8 19:33:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939044 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 199D1C00140 for ; Mon, 8 Aug 2022 19:34:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7B8058E0002; Mon, 8 Aug 2022 15:34:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 71C4E8E0007; Mon, 8 Aug 2022 15:34:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5451F8E0002; Mon, 8 Aug 2022 15:34:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 3496C8E0006 for ; Mon, 8 Aug 2022 15:34:51 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 022ECAB373 for ; Mon, 8 Aug 2022 19:34:50 +0000 (UTC) X-FDA: 79777427982.14.EC5ACC3 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf16.hostedemail.com (Postfix) with ESMTP id A7BFC180162 for ; Mon, 8 Aug 2022 19:34:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=25EELXvgziRYyg6Tw20CeFvEjR4VSR3Jcb5bQsXu2dE=; b=DkNcC5txygi3U5yQZ29aksV1wp pevV1udR7jsw2kKSZl++UUSLmpAGMNH4f72eWrltfKoBBhnJZNeUjiLxMGDHHuUQIbR2L1iJG/3u0 +LxS3QQdZBLJuMucpZVYx53OOR8HdhWK4oVgfhlw0fIJ0/eCdswsAZOQ3EqxfXnFyVw25g8g8UsWD B8BqDLTsICgFK35M42if16akN7uf5OYMP1CevvKS04VRAdK+TwNW+ogPyDlD9/HuPBADBZphWa34A YecP32YY53xlRctBGFTSWry9Wr2T1BXedKdHy/ti8cYRDbswPPEVP6uZ4fgAY34XVmLxzPcQyzSLE ciSplqRA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8WU-00EAsc-JF; Mon, 08 Aug 2022 19:34:46 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 07/59] mm: Add split_folio() Date: Mon, 8 Aug 2022 20:33:35 +0100 Message-Id: <20220808193430.3378317-8-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987290; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=25EELXvgziRYyg6Tw20CeFvEjR4VSR3Jcb5bQsXu2dE=; b=np0oCRkR+W+RBHyhGtTNGmnEEF2fgRS9v7LVzhH7gPHu6gP5QzhzA3Ul7A3DjOo9RKTMcx sEGUoUqucnquJC8wmVWG6NegmEZ7RODtGNwCxXet+Dd0kBetQA3YBp6Db7rlb+oGE9IZyn NgIQK8nj7l508KGkNicm09xB18jYehM= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=DkNcC5tx; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987290; a=rsa-sha256; cv=none; b=l1b1q/OiEJsiu7XiXh22STRORR502kRRVkLln6gfWGnZbNRvcMtI6N3jVJZ1wMLcu/hdHc rM3V2oApy1LluVVynjCtCTNqXe7cq1tWp/twmH3BjDL1UMuCZ0b6uhhb9zjT9HN+jSSrmN +wbC7z6IAMKtBaTvatqONahgBMqNOvk= X-Rspamd-Queue-Id: A7BFC180162 Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=DkNcC5tx; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: u3ru9aacryakwt35dez4dx6fm58i3mj7 X-HE-Tag: 1659987290-918756 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This wrapper removes a need to use split_huge_page(&folio->page). Convert two callers. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/huge_mm.h | 5 +++++ mm/shmem.c | 2 +- mm/truncate.c | 2 +- 3 files changed, 7 insertions(+), 2 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 768e5261fdae..aa0a427284aa 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -435,6 +435,11 @@ static inline int split_folio_to_list(struct folio *folio, return split_huge_page_to_list(&folio->page, list); } +static inline int split_folio(struct folio *folio) +{ + return split_folio_to_list(folio, NULL); +} + /* * archs that select ARCH_WANTS_THP_SWAP but don't support THP_SWP due to * limitations in the implementation like arm64 MTE can override this to diff --git a/mm/shmem.c b/mm/shmem.c index 4ae43cffeda3..de267a23248d 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -629,7 +629,7 @@ static unsigned long shmem_unused_huge_shrink(struct shmem_sb_info *sbinfo, goto move_back; } - ret = split_huge_page(&folio->page); + ret = split_folio(folio); folio_unlock(folio); folio_put(folio); diff --git a/mm/truncate.c b/mm/truncate.c index 0b0708bf935f..c0be77e5c008 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -240,7 +240,7 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end) folio_invalidate(folio, offset, length); if (!folio_test_large(folio)) return true; - if (split_huge_page(&folio->page) == 0) + if (split_folio(folio) == 0) return true; if (folio_test_dirty(folio)) return false; From patchwork Mon Aug 8 19:33:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939045 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C202C25B0C for ; Mon, 8 Aug 2022 19:35:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BD7A58E0007; Mon, 8 Aug 2022 15:34:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B39018E0006; Mon, 8 Aug 2022 15:34:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F8938E0008; Mon, 8 Aug 2022 15:34:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 662798E0006 for ; Mon, 8 Aug 2022 15:34:51 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 2FFB4C0134 for ; Mon, 8 Aug 2022 19:34:51 +0000 (UTC) X-FDA: 79777427982.12.F182324 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf22.hostedemail.com (Postfix) with ESMTP id CFB25C0136 for ; Mon, 8 Aug 2022 19:34:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=eQz2L5Sd+WJhCeFxF2qMEcGTWQTSZ58GjI77k+ag9os=; b=NG/xZwcT26rMmkefvclqeWiLrG DR4rtALAmr0jiYQKYswxrBCbefGHBB4xYJSY0mU8mBcg8xQjJE3HCanMuE6E6BasxAeeW3xm+o4jg csoF9xPvvmfsUMS6GEpRceVQFW2s7pbQX6LN4TX/EJGBMBiWmkYWnRugOH7CtKQi7Il/l5RX/kg4M IVC90t9R+Jz2Sba5FhJCIOqte3y4+VuaKJTEo1Nm0msl/c7GFIMMFlWeRi4DEzQZceZ00aNFbh8iL QbGqgL+hpvMUNXQIJ9W1JY0Sce2ZCz47+O7OArHNzsMwh1sQZxgioKn1Gs+qw+dr6mjgOyGprTtza z7eg3RhA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8WX-00EAsi-BC; Mon, 08 Aug 2022 19:34:49 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 08/59] mm: Add folio_add_lru_vma() Date: Mon, 8 Aug 2022 20:33:36 +0100 Message-Id: <20220808193430.3378317-9-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987290; a=rsa-sha256; cv=none; b=VNvMERLMRiiiaBnORGyiC68VyJhylQ9KwaGkg7RTW/aNIDKsNaeamGQSp+haDRCEgcxs9G 1wwaTWycOHQM920Tek3lnnT18M+uCNsAyVmQVhrZT03Hhitb5jLLIfD6ey2nuFINT2/Tyb XxWgjgR0UHRtDOnBwlEmnz8GUFoOrng= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="NG/xZwcT"; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987290; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eQz2L5Sd+WJhCeFxF2qMEcGTWQTSZ58GjI77k+ag9os=; b=2AfMopFHPPcC8pPXpwXxmXLNaZcpL5cqmlZ4UfkAXtfiEnRXs5OCiNg8UGDcdqQLX5FoSM Pn57IvespwfizeggqHS73GRFvyMCV8p6a1ifCTE17VM7p1X98LdGNFqtmUwte+esDnP8kB gDJ9gqKeONsibHbBtedkoO1f5AIDT1g= X-Rspamd-Queue-Id: CFB25C0136 Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="NG/xZwcT"; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: xtyg6uibnb149dikdp6gt418e98hqdo6 X-HE-Tag: 1659987290-326578 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert lru_cache_add_inactive_or_unevictable() to folio_add_lru_vma() and add a compatibility wrapper. --- include/linux/swap.h | 10 +++++----- mm/folio-compat.c | 6 ++++++ mm/swap.c | 19 +++++++++---------- 3 files changed, 20 insertions(+), 15 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 43150b9bbc5c..333d5588dc2d 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -375,11 +375,11 @@ extern unsigned long totalreserve_pages; /* linux/mm/swap.c */ -extern void lru_note_cost(struct lruvec *lruvec, bool file, - unsigned int nr_pages); -extern void lru_note_cost_folio(struct folio *); -extern void folio_add_lru(struct folio *); -extern void lru_cache_add(struct page *); +void lru_note_cost(struct lruvec *lruvec, bool file, unsigned int nr_pages); +void lru_note_cost_folio(struct folio *); +void folio_add_lru(struct folio *); +void folio_add_lru_vma(struct folio *, struct vm_area_struct *); +void lru_cache_add(struct page *); void mark_page_accessed(struct page *); void folio_mark_accessed(struct folio *); diff --git a/mm/folio-compat.c b/mm/folio-compat.c index 458618c7302c..e1e23b4947d7 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -88,6 +88,12 @@ void lru_cache_add(struct page *page) } EXPORT_SYMBOL(lru_cache_add); +void lru_cache_add_inactive_or_unevictable(struct page *page, + struct vm_area_struct *vma) +{ + folio_add_lru_vma(page_folio(page), vma); +} + int add_to_page_cache_lru(struct page *page, struct address_space *mapping, pgoff_t index, gfp_t gfp) { diff --git a/mm/swap.c b/mm/swap.c index 9cee7f6a3809..6525011b715e 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -493,22 +493,21 @@ void folio_add_lru(struct folio *folio) EXPORT_SYMBOL(folio_add_lru); /** - * lru_cache_add_inactive_or_unevictable - * @page: the page to be added to LRU - * @vma: vma in which page is mapped for determining reclaimability + * folio_add_lru_vma() - Add a folio to the appropate LRU list for this VMA. + * @folio: The folio to be added to the LRU. + * @vma: VMA in which the folio is mapped. * - * Place @page on the inactive or unevictable LRU list, depending on its - * evictability. + * If the VMA is mlocked, @folio is added to the unevictable list. + * Otherwise, it is treated the same way as folio_add_lru(). */ -void lru_cache_add_inactive_or_unevictable(struct page *page, - struct vm_area_struct *vma) +void folio_add_lru_vma(struct folio *folio, struct vm_area_struct *vma) { - VM_BUG_ON_PAGE(PageLRU(page), page); + VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); if (unlikely((vma->vm_flags & (VM_LOCKED | VM_SPECIAL)) == VM_LOCKED)) - mlock_new_page(page); + mlock_new_page(&folio->page); else - lru_cache_add(page); + folio_add_lru(folio); } /* From patchwork Mon Aug 8 19:33:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939047 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7AB99C00140 for ; Mon, 8 Aug 2022 19:35:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C9A0E8E0008; Mon, 8 Aug 2022 15:34:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BAF5E8E0006; Mon, 8 Aug 2022 15:34:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 93D5A8E0008; Mon, 8 Aug 2022 15:34:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7030E8E0006 for ; Mon, 8 Aug 2022 15:34:54 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 5BC424050F for ; Mon, 8 Aug 2022 19:34:54 +0000 (UTC) X-FDA: 79777428108.16.4203E9F Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id BA11E12015D for ; Mon, 8 Aug 2022 19:34:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Br2UbecTvmiQlxiPKbP5mNziAK8h3pPAFMipE73VnEQ=; b=YszMjZqf48Yq4THQBJ8ConKVwj sdE8m69Cf/W4RdQdDdBOcW1muQkR4q5hcVIJf1w7bKaBmaJ/XBsgSr/qB6T62ysMesa4YRr1geSoq +IkqkJhw3bcUWt5aFczCASqvsP1JE2c0W+nL70BaijUh7/tslqtnS0YPZ6PHdS2gHbC+hyRKok+cn aRp9Jx3sfk6K/3xeKTsoQtuvjfDYPqIfXuQ6/RZoFbOB3cMIC6wfNsm1wUuQBJNvxbc+z9vklpCuh dZnUrCSEDQ6+Epv317UwYbdMCwDfsQeDfRA1GMG6OGmfNZEG6jVC7kfeFeL5464GmVZKbbQmLXSDG kIz70FUg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8WX-00EAsv-Iz; Mon, 08 Aug 2022 19:34:49 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 09/59] shmem: Convert shmem_writepage() to use a folio throughout Date: Mon, 8 Aug 2022 20:33:37 +0100 Message-Id: <20220808193430.3378317-10-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987293; a=rsa-sha256; cv=none; b=zEgtqXOfjPX9FgCcuh6/D5Z0hiAocELdCHHtC4q8I4fFma141cTJc9H3sOTigLTmDkOMV1 b8ikkiVvR1lxZaVy6u/r9wC027zXY+4HaYecFGAVhNoe20IM4FqG7SwmW6pa0M/DQQQzxe vhCty3p1YlggbfQWU826D8FKJUJaurE= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=YszMjZqf; dmarc=none; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987293; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Br2UbecTvmiQlxiPKbP5mNziAK8h3pPAFMipE73VnEQ=; b=gjECgSeA0wI1BR+YdAWay6jUi5kZsobFIBa1YiczhBgcwfcIlfxuriEmGvfXrOuQHvpAUW FLfAWvZNxMyJdK79ZWRFo8KuWYLMmR90XGnF9Fh2R9ZvmCbuQh+bAJc3vGQdFduHC+O8ab cL3Hl+AYjK+RnyNiWkj5r2bsRh5c578= X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: BA11E12015D Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=YszMjZqf; dmarc=none; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: johoi889j6y7z6k5q6akqwka53un8nu7 X-HE-Tag: 1659987293-799848 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Even though we will split any large folio that comes in, write the code to handle large folios so as to not leave a trap for whoever tries to handle large folios in the swap cache. Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 47 ++++++++++++++++++++++++----------------------- 1 file changed, 24 insertions(+), 23 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index de267a23248d..bcd3644eb9c3 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1328,17 +1328,18 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) * "force", drivers/gpu/drm/i915/gem/i915_gem_shmem.c gets huge pages, * and its shmem_writeback() needs them to be split when swapping. */ - if (PageTransCompound(page)) { + if (folio_test_large(folio)) { /* Ensure the subpages are still dirty */ - SetPageDirty(page); + folio_test_set_dirty(folio); if (split_huge_page(page) < 0) goto redirty; - ClearPageDirty(page); + folio = page_folio(page); + folio_clear_dirty(folio); } - BUG_ON(!PageLocked(page)); - mapping = page->mapping; - index = page->index; + BUG_ON(!folio_test_locked(folio)); + mapping = folio->mapping; + index = folio->index; inode = mapping->host; info = SHMEM_I(inode); if (info->flags & VM_LOCKED) @@ -1361,15 +1362,15 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) /* * This is somewhat ridiculous, but without plumbing a SWAP_MAP_FALLOC * value into swapfile.c, the only way we can correctly account for a - * fallocated page arriving here is now to initialize it and write it. + * fallocated folio arriving here is now to initialize it and write it. * - * That's okay for a page already fallocated earlier, but if we have + * That's okay for a folio already fallocated earlier, but if we have * not yet completed the fallocation, then (a) we want to keep track - * of this page in case we have to undo it, and (b) it may not be a + * of this folio in case we have to undo it, and (b) it may not be a * good idea to continue anyway, once we're pushing into swap. So - * reactivate the page, and let shmem_fallocate() quit when too many. + * reactivate the folio, and let shmem_fallocate() quit when too many. */ - if (!PageUptodate(page)) { + if (!folio_test_uptodate(folio)) { if (inode->i_private) { struct shmem_falloc *shmem_falloc; spin_lock(&inode->i_lock); @@ -1385,9 +1386,9 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) if (shmem_falloc) goto redirty; } - clear_highpage(page); - flush_dcache_page(page); - SetPageUptodate(page); + folio_zero_range(folio, 0, folio_size(folio)); + flush_dcache_folio(folio); + folio_mark_uptodate(folio); } swap = folio_alloc_swap(folio); @@ -1396,7 +1397,7 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) /* * Add inode to shmem_unuse()'s list of swapped-out inodes, - * if it's not already there. Do it now before the page is + * if it's not already there. Do it now before the folio is * moved to swap cache, when its pagelock no longer protects * the inode from eviction. But don't unlock the mutex until * we've incremented swapped, because shmem_unuse_inode() will @@ -1406,7 +1407,7 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) if (list_empty(&info->swaplist)) list_add(&info->swaplist, &shmem_swaplist); - if (add_to_swap_cache(page, swap, + if (add_to_swap_cache(&folio->page, swap, __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN, NULL) == 0) { spin_lock_irq(&info->lock); @@ -1415,21 +1416,21 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) spin_unlock_irq(&info->lock); swap_shmem_alloc(swap); - shmem_delete_from_page_cache(page, swp_to_radix_entry(swap)); + shmem_delete_from_page_cache(&folio->page, swp_to_radix_entry(swap)); mutex_unlock(&shmem_swaplist_mutex); - BUG_ON(page_mapped(page)); - swap_writepage(page, wbc); + BUG_ON(folio_mapped(folio)); + swap_writepage(&folio->page, wbc); return 0; } mutex_unlock(&shmem_swaplist_mutex); - put_swap_page(page, swap); + put_swap_page(&folio->page, swap); redirty: - set_page_dirty(page); + folio_mark_dirty(folio); if (wbc->for_reclaim) - return AOP_WRITEPAGE_ACTIVATE; /* Return with page locked */ - unlock_page(page); + return AOP_WRITEPAGE_ACTIVATE; /* Return with folio locked */ + folio_unlock(folio); return 0; } From patchwork Mon Aug 8 19:33:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939046 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B93E4C25B08 for ; Mon, 8 Aug 2022 19:35:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4FEEC8E0006; Mon, 8 Aug 2022 15:34:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3BF77940007; Mon, 8 Aug 2022 15:34:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1E8708E0009; Mon, 8 Aug 2022 15:34:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id F176F8E0006 for ; Mon, 8 Aug 2022 15:34:58 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id CCD2E160113 for ; Mon, 8 Aug 2022 19:34:58 +0000 (UTC) X-FDA: 79777428276.10.DA5C423 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf16.hostedemail.com (Postfix) with ESMTP id 7AA4D18015B for ; Mon, 8 Aug 2022 19:34:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=rep9f/ggXUAP4MIsL/faNmIcRn2PrPB6iaRqYgvpAik=; b=AU0bLLNol8P9a1LDvlbedxxRJo 9QDC2TDiMLfpEZYYAj9MCNsJtcDQdzG/3vhZEZOCZtIPrhFsxPHIYhZ1CAZrmKdVwXGvF0sT2wETZ 6FL60zURQSvOzGnxpiqAD7aAGHcOGudjDrD06pHy+b1NouyAjwT+HpSUmICYGoezIrSVtEL+HXULn 1TJ9C5dGx/WxTjVspzhhFv7EIK2Rr2KYRaCHw+M3Eqm3/vbjcEaX4TER0uKlGvfMGL+seqhzckWsT tSwzcrj4y0/86cXhQ5MnDe1SVjHMXiKDWYwja5XxQmB7D+92o5m6uG3ZYGPeZ5cvahdm+asaks7qW 8iLLeGXA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8WX-00EAt2-RR; Mon, 08 Aug 2022 19:34:52 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 10/59] shmem: Convert shmem_delete_from_page_cache() to take a folio Date: Mon, 8 Aug 2022 20:33:38 +0100 Message-Id: <20220808193430.3378317-11-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987298; a=rsa-sha256; cv=none; b=fyvSdvXnO+qiStsqfzSGEPOtBlmdNetjILHX/YqWLNJEBY3mdz5clqUE3X0+VL9uruiUIq G0koXl9OFuvegEMoUwB7Vdiss7+KupcOP7WeDDQQ3sci5xoxLwrv1XbeYRm76cxhUlfTFU CyBIli6Z+OsIQy8H76i3UQl86nDLy48= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=AU0bLLNo; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987298; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rep9f/ggXUAP4MIsL/faNmIcRn2PrPB6iaRqYgvpAik=; b=sD1ZaKnJwrp7zTzqIFds1mkbUNqI/C1f4dl1m2qXy5YfBB5PmjYlysRO41qnwbZzl1ZYQx z5x6jIaF+UiWQGihcKnAYcRax8moBYaWlvOTN62C9GMhLK4Lk3cW72WpAXd8AOrFhdqsPU clYhfo0hyXRh3SoiQOr7nson49gViS8= X-Stat-Signature: 6ryii6qzoz5b9puxwp8jz7d5az683smg X-Rspamd-Queue-Id: 7AA4D18015B Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=AU0bLLNo; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1659987298-449902 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove the assertion that the page is not Compound as this function now handles large folios correctly. Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index bcd3644eb9c3..f561f6e7f53b 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -763,23 +763,22 @@ static int shmem_add_to_page_cache(struct folio *folio, } /* - * Like delete_from_page_cache, but substitutes swap for page. + * Like delete_from_page_cache, but substitutes swap for @folio. */ -static void shmem_delete_from_page_cache(struct page *page, void *radswap) +static void shmem_delete_from_page_cache(struct folio *folio, void *radswap) { - struct address_space *mapping = page->mapping; + struct address_space *mapping = folio->mapping; + long nr = folio_nr_pages(folio); int error; - VM_BUG_ON_PAGE(PageCompound(page), page); - xa_lock_irq(&mapping->i_pages); - error = shmem_replace_entry(mapping, page->index, page, radswap); - page->mapping = NULL; - mapping->nrpages--; - __dec_lruvec_page_state(page, NR_FILE_PAGES); - __dec_lruvec_page_state(page, NR_SHMEM); + error = shmem_replace_entry(mapping, folio->index, folio, radswap); + folio->mapping = NULL; + mapping->nrpages -= nr; + __lruvec_stat_mod_folio(folio, NR_FILE_PAGES, -nr); + __lruvec_stat_mod_folio(folio, NR_SHMEM, -nr); xa_unlock_irq(&mapping->i_pages); - put_page(page); + folio_put(folio); BUG_ON(error); } @@ -1416,7 +1415,7 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) spin_unlock_irq(&info->lock); swap_shmem_alloc(swap); - shmem_delete_from_page_cache(&folio->page, swp_to_radix_entry(swap)); + shmem_delete_from_page_cache(folio, swp_to_radix_entry(swap)); mutex_unlock(&shmem_swaplist_mutex); BUG_ON(folio_mapped(folio)); From patchwork Mon Aug 8 19:33:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939049 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6903CC00140 for ; Mon, 8 Aug 2022 19:35:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 94669940008; Mon, 8 Aug 2022 15:35:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 658DD940009; Mon, 8 Aug 2022 15:35:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 433DE940007; Mon, 8 Aug 2022 15:35:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 27BD4940008 for ; Mon, 8 Aug 2022 15:35:00 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id ED4D0AB3BE for ; Mon, 8 Aug 2022 19:34:59 +0000 (UTC) X-FDA: 79777428318.21.BBE31B4 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf19.hostedemail.com (Postfix) with ESMTP id 8D7861A013A for ; Mon, 8 Aug 2022 19:34:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=pVvOgQThtf/C4s3gexRZB6amKOirvomPfXjR1TGMYTc=; b=P7EPqs8dMvSCZhvW2tx4R2WjJb 7Tc2LWmti+ZaHRiXEF8ZFiBgi4J6MpCbnoEddA1B+rHGRp1pWaDyLyfzEMqQPoAg3l5NrUAy9J1qm q1hM1fpMOv8o19brdrtomlSUvHj1zieJYm1VSskR4nQaPlpWz2rLplu6XA8jGxSHCWpT3Ion3OpBO mh+KbxT4NST8mwFcvzKMKw5fe83bL+D6hPr/x5cpDztgrk470AUfWkCt0gT50aWfAkVMEOVJM8D2a xX0tJReh6jov9jc8C66TvpyDWHd/SyoFBK27MVAX8e7eYdJQh5G+0U1i1muDygylTPX5+3RwJq9cO HWfwQi6w==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8Wa-00EAtG-Nk; Mon, 08 Aug 2022 19:34:55 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 11/59] shmem: Convert shmem_replace_page() to use folios throughout Date: Mon, 8 Aug 2022 20:33:39 +0100 Message-Id: <20220808193430.3378317-12-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987299; a=rsa-sha256; cv=none; b=NTE7mzasUDjX3AjMkbSTztHbLUZg6I1NfsE2+kX8uTYKrtaAH8h8iqlgBCQ5QVJEw4KgKY OktIY1wHEJbiAAq/p9ORChVJKFyKlO6DbuxHMm/wTvyz6AgVQnzNHeTzCPurm3woAnpEC0 nBZ+4GPvc9mNK/O/SQYQMFg5g5rv51w= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=P7EPqs8d; spf=none (imf19.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987299; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pVvOgQThtf/C4s3gexRZB6amKOirvomPfXjR1TGMYTc=; b=8X92qxovFOYyq0pTHbxAPkvN7AhNsPNpY8asp/XRwIOE51G+6EooZY4e3DnJnTMf1z3CPV O8szfyx3wLlkqvyJWMtvkGEqcWqwQ5w28U8/TGVvwhgmKQhOb29XSlf7Z5Wl/Yjc2QkOwM UmjYenrYM8oYtb94kGOPP1gBx6PnJEw= X-Rspamd-Queue-Id: 8D7861A013A Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=P7EPqs8d; spf=none (imf19.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: map7hsj71z1bdznu33n7e6bmf6ncme4c X-HE-Tag: 1659987299-82013 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce folio_set_swap_entry() to abstract how both folio->private and swp_entry_t work. Use swap_address_space() directly instead of indirecting through folio_mapping(). Include an assertion that the old folio is not large as we only allocate a single-page folio to replace it. Use folio_put_refs() instead of calling folio_put() twice. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/swap.h | 5 ++++ mm/shmem.c | 63 +++++++++++++++++++++----------------------- 2 files changed, 35 insertions(+), 33 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 333d5588dc2d..afcb76bbd141 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -351,6 +351,11 @@ static inline swp_entry_t folio_swap_entry(struct folio *folio) return entry; } +static inline void folio_set_swap_entry(struct folio *folio, swp_entry_t entry) +{ + folio->private = (void *)entry.val; +} + /* linux/mm/workingset.c */ void workingset_age_nonresident(struct lruvec *lruvec, unsigned long nr_pages); void *workingset_eviction(struct folio *folio, struct mem_cgroup *target_memcg); diff --git a/mm/shmem.c b/mm/shmem.c index f561f6e7f53b..eec32307984d 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1560,12 +1560,6 @@ static struct folio *shmem_alloc_folio(gfp_t gfp, return folio; } -static struct page *shmem_alloc_page(gfp_t gfp, - struct shmem_inode_info *info, pgoff_t index) -{ - return &shmem_alloc_folio(gfp, info, index)->page; -} - static struct folio *shmem_alloc_and_acct_folio(gfp_t gfp, struct inode *inode, pgoff_t index, bool huge) { @@ -1617,49 +1611,47 @@ static bool shmem_should_replace_folio(struct folio *folio, gfp_t gfp) static int shmem_replace_page(struct page **pagep, gfp_t gfp, struct shmem_inode_info *info, pgoff_t index) { - struct page *oldpage, *newpage; struct folio *old, *new; struct address_space *swap_mapping; swp_entry_t entry; pgoff_t swap_index; int error; - oldpage = *pagep; - entry.val = page_private(oldpage); + old = page_folio(*pagep); + entry = folio_swap_entry(old); swap_index = swp_offset(entry); - swap_mapping = page_mapping(oldpage); + swap_mapping = swap_address_space(entry); /* * We have arrived here because our zones are constrained, so don't * limit chance of success by further cpuset and node constraints. */ gfp &= ~GFP_CONSTRAINT_MASK; - newpage = shmem_alloc_page(gfp, info, index); - if (!newpage) + VM_BUG_ON_FOLIO(folio_test_large(old), old); + new = shmem_alloc_folio(gfp, info, index); + if (!new) return -ENOMEM; - get_page(newpage); - copy_highpage(newpage, oldpage); - flush_dcache_page(newpage); + folio_get(new); + folio_copy(new, old); + flush_dcache_folio(new); - __SetPageLocked(newpage); - __SetPageSwapBacked(newpage); - SetPageUptodate(newpage); - set_page_private(newpage, entry.val); - SetPageSwapCache(newpage); + __folio_set_locked(new); + __folio_set_swapbacked(new); + folio_mark_uptodate(new); + folio_set_swap_entry(new, entry); + folio_set_swapcache(new); /* * Our caller will very soon move newpage out of swapcache, but it's * a nice clean interface for us to replace oldpage by newpage there. */ xa_lock_irq(&swap_mapping->i_pages); - error = shmem_replace_entry(swap_mapping, swap_index, oldpage, newpage); + error = shmem_replace_entry(swap_mapping, swap_index, old, new); if (!error) { - old = page_folio(oldpage); - new = page_folio(newpage); mem_cgroup_migrate(old, new); - __inc_lruvec_page_state(newpage, NR_FILE_PAGES); - __dec_lruvec_page_state(oldpage, NR_FILE_PAGES); + __lruvec_stat_mod_folio(new, NR_FILE_PAGES, 1); + __lruvec_stat_mod_folio(old, NR_FILE_PAGES, -1); } xa_unlock_irq(&swap_mapping->i_pages); @@ -1669,18 +1661,17 @@ static int shmem_replace_page(struct page **pagep, gfp_t gfp, * both PageSwapCache and page_private after getting page lock; * but be defensive. Reverse old to newpage for clear and free. */ - oldpage = newpage; + old = new; } else { - lru_cache_add(newpage); - *pagep = newpage; + folio_add_lru(new); + *pagep = &new->page; } - ClearPageSwapCache(oldpage); - set_page_private(oldpage, 0); + folio_clear_swapcache(old); + old->private = NULL; - unlock_page(oldpage); - put_page(oldpage); - put_page(oldpage); + folio_unlock(old); + folio_put_refs(old, 2); return error; } @@ -2362,6 +2353,12 @@ static struct inode *shmem_get_inode(struct super_block *sb, struct inode *dir, } #ifdef CONFIG_USERFAULTFD +static struct page *shmem_alloc_page(gfp_t gfp, + struct shmem_inode_info *info, pgoff_t index) +{ + return &shmem_alloc_folio(gfp, info, index)->page; +} + int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, struct vm_area_struct *dst_vma, From patchwork Mon Aug 8 19:33:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939048 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14E36C25B0C for ; Mon, 8 Aug 2022 19:35:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 65A2C94000A; Mon, 8 Aug 2022 15:35:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 54652940008; Mon, 8 Aug 2022 15:35:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 32453940009; Mon, 8 Aug 2022 15:35:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0D8D1940007 for ; Mon, 8 Aug 2022 15:35:00 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id DF72CA0856 for ; Mon, 8 Aug 2022 19:34:59 +0000 (UTC) X-FDA: 79777428318.14.E3051E1 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf11.hostedemail.com (Postfix) with ESMTP id 94E7F4018B for ; Mon, 8 Aug 2022 19:34:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=rZ9tiYok3CmkdoKC4u5YlyBJYtd4IGvFvqCkTZC9NJA=; b=BvEe5ydiWzp+RgzNiYuOUZc6yY uzSuDXLkGRrkzmWlOx1FU+DpJfpTt8MkHHA8+FUVtwUJyfkIiTZJj6o9kW4uEorHyZ8ZazVlFlbM0 HQa4ac/GeNbXXkJLVKt9nPTUjKAtans4XomqDgolxbGi1Ku+iUzmR5MoeC1vbwrjd2c5OrDEdlFA9 i/Fc9bWHyBvhf2ASkrEj5Y88kYKuucqk3ydyREtodqWSqLBw92V0w9sT0XwxHbbXwLIVySW5BQjxd Di3kYAEy4oyNWM42Sf8BddztG99dyckmtTct+jZugi188GAxj2djv2MtNkgQlPni4oha8SACUWIvy KrDf57lA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8Wd-00EAtQ-Eu; Mon, 08 Aug 2022 19:34:58 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 12/59] mm/swapfile: Remove page_swapcount() Date: Mon, 8 Aug 2022 20:33:40 +0100 Message-Id: <20220808193430.3378317-13-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987299; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rZ9tiYok3CmkdoKC4u5YlyBJYtd4IGvFvqCkTZC9NJA=; b=aLlPXoJXR3y11DB6v0jHeiwMITW+juHuakR/yCw4djxyP6YcoXkABv+ebBuMVvmAx4IdnG Z3M33W5h35gEP9teiikIsC4tNzOItUcc3y7B0xHK+uSwawaRJ58j4CWHSR/j+iquwWwvfw epip63LI1Q8PfBQGDrwNZjTZBDt39Wc= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=BvEe5ydi; dmarc=none; spf=none (imf11.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987299; a=rsa-sha256; cv=none; b=h1yyjq7JbjXJvDjQ+Vxo+hp5OAzJJE8+cnjGSsP2IgvjPDwwoNTwCmqS/G6JIG4IRA0UUZ 6ke2OrOeXQMQOjQDCfC3KO0j0gx1SNWoq7YsfegdupChMLOLzL5/bjcWcGRDAwo6WM8uKF mBbAs/7qprDS7fyC7+HcTfi4O3NlnDY= X-Rspamd-Queue-Id: 94E7F4018B Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=BvEe5ydi; dmarc=none; spf=none (imf11.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: yjgo1iuckc9xpmogqogp3cg5gmgnttoa X-HE-Tag: 1659987299-757892 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: By restructuring folio_swapped(), it can use swap_swapcount() instead of page_swapcount(). It's even a little more efficient. Signed-off-by: Matthew Wilcox (Oracle) --- mm/swapfile.c | 46 +++++++++++++--------------------------------- 1 file changed, 13 insertions(+), 33 deletions(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index 1fdccd2f1422..c042fd71de02 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1427,30 +1427,6 @@ void swapcache_free_entries(swp_entry_t *entries, int n) spin_unlock(&p->lock); } -/* - * How many references to page are currently swapped out? - * This does not give an exact answer when swap count is continued, - * but does include the high COUNT_CONTINUED flag to allow for that. - */ -static int page_swapcount(struct page *page) -{ - int count = 0; - struct swap_info_struct *p; - struct swap_cluster_info *ci; - swp_entry_t entry; - unsigned long offset; - - entry.val = page_private(page); - p = _swap_info_get(entry); - if (p) { - offset = swp_offset(entry); - ci = lock_cluster_or_swap_info(p, offset); - count = swap_count(p->swap_map[offset]); - unlock_cluster_or_swap_info(p, ci); - } - return count; -} - int __swap_count(swp_entry_t entry) { struct swap_info_struct *si; @@ -1465,11 +1441,16 @@ int __swap_count(swp_entry_t entry) return count; } +/* + * How many references to @entry are currently swapped out? + * This does not give an exact answer when swap count is continued, + * but does include the high COUNT_CONTINUED flag to allow for that. + */ static int swap_swapcount(struct swap_info_struct *si, swp_entry_t entry) { - int count = 0; pgoff_t offset = swp_offset(entry); struct swap_cluster_info *ci; + int count; ci = lock_cluster_or_swap_info(si, offset); count = swap_count(si->swap_map[offset]); @@ -1570,17 +1551,16 @@ static bool swap_page_trans_huge_swapped(struct swap_info_struct *si, static bool folio_swapped(struct folio *folio) { - swp_entry_t entry; - struct swap_info_struct *si; + swp_entry_t entry = folio_swap_entry(folio); + struct swap_info_struct *si = _swap_info_get(entry); + + if (!si) + return false; if (!IS_ENABLED(CONFIG_THP_SWAP) || likely(!folio_test_large(folio))) - return page_swapcount(&folio->page) != 0; + return swap_swapcount(si, entry) != 0; - entry = folio_swap_entry(folio); - si = _swap_info_get(entry); - if (si) - return swap_page_trans_huge_swapped(si, entry); - return false; + return swap_page_trans_huge_swapped(si, entry); } /* From patchwork Mon Aug 8 19:33:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939050 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0C2FC25B08 for ; Mon, 8 Aug 2022 19:35:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E711C940009; Mon, 8 Aug 2022 15:35:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D33D8940007; Mon, 8 Aug 2022 15:35:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B45DF94000B; Mon, 8 Aug 2022 15:35:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 85300940007 for ; Mon, 8 Aug 2022 15:35:00 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 520F9C0134 for ; Mon, 8 Aug 2022 19:35:00 +0000 (UTC) X-FDA: 79777428360.19.85D41AC Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf17.hostedemail.com (Postfix) with ESMTP id D2B694005B for ; Mon, 8 Aug 2022 19:34:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=oGSPn8NwOXRP4TYNP3MMOCC5rGBSouXy25gKhFOlJYA=; b=XaFplew6Su9BsNcWpanv3xjw9I xgaQ4nzViWClBwtlGAJuBGFEEsoqBlrjwtidTmFiRZHx1WivSlAZq8LQbaAoeYnLDWlzxNdpCWIiO MHS1BFh1zXNh7Gi4k+E8FkELzFLOuIndBbvst+9AxP2bJwjpHB4Rr/B108FNk5W2Tx05MJO7Z5Dsn RaC8tB9aZA1de/GDWZcXRq5u9zlEaEbK90s51O5BinMs87Ogr1rd+gfXToxHbnhd1yg3zKuxTkZ3q EYFwkut5yC7qQYR330kcvs7QfQeo/0ouDFZZtzjpRyaKpg6EwubhDP0UoJQUlKbC7S73kpe5vAbOR 3GtaClNg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8Wg-00EAti-B0; Mon, 08 Aug 2022 19:34:58 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 13/59] mm/swapfile: Convert try_to_free_swap() to folio_free_swap() Date: Mon, 8 Aug 2022 20:33:41 +0100 Message-Id: <20220808193430.3378317-14-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987299; a=rsa-sha256; cv=none; b=GLqL5v3y0oR178isLo4zR+Y0ryCc9E8RaWtcK+jKriXxfCCWX/KCK9t+pYwOF4+c6Sz3bM /Bp1Sw0Meb9vUH1pl4yft3wRI6yGNoDlqJTgwmqZLJwJAgjfkwixFtm0cOtAdcBSy35+sb C6Ctl7mRJhJo7mtI/x1yAdUgN2l+z5o= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=XaFplew6; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987299; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oGSPn8NwOXRP4TYNP3MMOCC5rGBSouXy25gKhFOlJYA=; b=2yzJ4Zzufm8vm5kYlk8lbUTbdDb3mEFN/5SchXNp3px3x2D2R1lcj1dbQ60xhUKWdknIO1 UUTzpFp4XyBnxXkqNAb7paGHZ/4QGjonPpOhaKV3IpzZlnW0nRUmVAUdGQsvJfIdmZpO+2 9Gg0Kj7kejEdK4j52OXJ04vIuMz0ZfQ= X-Rspamd-Server: rspam06 X-Rspam-User: X-Stat-Signature: eyyqmix44e931k6o7je4tutf7f74zqtp X-Rspamd-Queue-Id: D2B694005B Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=XaFplew6; spf=none (imf17.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1659987299-698971 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add kernel-doc for folio_free_swap() and make it return bool. Add a try_to_free_swap() compatibility wrapper. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/swap.h | 6 ++++++ mm/folio-compat.c | 7 +++++++ mm/swapfile.c | 32 ++++++++++++++++++-------------- mm/vmscan.c | 2 +- 4 files changed, 32 insertions(+), 15 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index afcb76bbd141..4595cbc1cb02 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -486,6 +486,7 @@ static inline long get_nr_swap_pages(void) extern void si_swapinfo(struct sysinfo *); swp_entry_t folio_alloc_swap(struct folio *folio); +bool folio_free_swap(struct folio *folio); extern void put_swap_page(struct page *page, swp_entry_t entry); extern swp_entry_t get_swap_page_of_type(int); extern int get_swap_pages(int n, swp_entry_t swp_entries[], int entry_size); @@ -602,6 +603,11 @@ static inline swp_entry_t folio_alloc_swap(struct folio *folio) return entry; } +static inline bool folio_free_swap(struct folio *folio) +{ + return false; +} + static inline int add_swap_extent(struct swap_info_struct *sis, unsigned long start_page, unsigned long nr_pages, sector_t start_block) diff --git a/mm/folio-compat.c b/mm/folio-compat.c index e1e23b4947d7..06d47f00609b 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -146,3 +146,10 @@ void putback_lru_page(struct page *page) { folio_putback_lru(page_folio(page)); } + +#ifdef CONFIG_SWAP +int try_to_free_swap(struct page *page) +{ + return folio_free_swap(page_folio(page)); +} +#endif diff --git a/mm/swapfile.c b/mm/swapfile.c index c042fd71de02..880871f4c6d4 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1563,43 +1563,47 @@ static bool folio_swapped(struct folio *folio) return swap_page_trans_huge_swapped(si, entry); } -/* - * If swap is getting full, or if there are no more mappings of this page, - * then try_to_free_swap is called to free its swap space. +/** + * folio_free_swap() - Free the swap space used for this folio. + * @folio: The folio to remove. + * + * If swap is getting full, or if there are no more mappings of this folio, + * then call folio_free_swap to free its swap space. + * + * Return: true if we were able to release the swap space. */ -int try_to_free_swap(struct page *page) +bool folio_free_swap(struct folio *folio) { - struct folio *folio = page_folio(page); VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); if (!folio_test_swapcache(folio)) - return 0; + return false; if (folio_test_writeback(folio)) - return 0; + return false; if (folio_swapped(folio)) - return 0; + return false; /* * Once hibernation has begun to create its image of memory, - * there's a danger that one of the calls to try_to_free_swap() + * there's a danger that one of the calls to folio_free_swap() * - most probably a call from __try_to_reclaim_swap() while * hibernation is allocating its own swap pages for the image, * but conceivably even a call from memory reclaim - will free - * the swap from a page which has already been recorded in the - * image as a clean swapcache page, and then reuse its swap for + * the swap from a folio which has already been recorded in the + * image as a clean swapcache folio, and then reuse its swap for * another page of the image. On waking from hibernation, the - * original page might be freed under memory pressure, then + * original folio might be freed under memory pressure, then * later read back in from swap, now with the wrong data. * * Hibernation suspends storage while it is writing the image * to disk so check that here. */ if (pm_suspended_storage()) - return 0; + return false; delete_from_swap_cache(folio); folio_set_dirty(folio); - return 1; + return true; } /* diff --git a/mm/vmscan.c b/mm/vmscan.c index 93b1087fc969..d3e26712dbc1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2004,7 +2004,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, if (folio_test_swapcache(folio) && (mem_cgroup_swap_full(&folio->page) || folio_test_mlocked(folio))) - try_to_free_swap(&folio->page); + folio_free_swap(folio); VM_BUG_ON_FOLIO(folio_test_active(folio), folio); if (!folio_test_mlocked(folio)) { int type = folio_is_file_lru(folio); From patchwork Mon Aug 8 19:33:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939052 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D957BC25B08 for ; Mon, 8 Aug 2022 19:35:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3251C94000C; Mon, 8 Aug 2022 15:35:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1E98D940007; Mon, 8 Aug 2022 15:35:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E1F0B94000C; Mon, 8 Aug 2022 15:35:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C87DB940007 for ; Mon, 8 Aug 2022 15:35:12 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 75E4F80119 for ; Mon, 8 Aug 2022 19:35:12 +0000 (UTC) X-FDA: 79777428864.20.7147994 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf16.hostedemail.com (Postfix) with ESMTP id 208F8180169 for ; Mon, 8 Aug 2022 19:35:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=EtUfHusf3N70jYSPPPixSjqlm8bwcBxVI5pl2nsQ5N4=; b=vCA0PVi0e1XGXwZ108Fa/Pq36z 6lYCFDsOJJdZJ29IBlu+5HB/ZTMPfJ8rmgKFMIxsYXwF02+G52M85FG6P7ZX0JXg2gLTEVZkYdJBu d1Ds0J03q1A5RaaoOw6S721P9GwjxQEmDvKuMQhQu9E5dBhSf8UIM4eiylhwXUYPh5DfkRM8c/hK0 AWSZB5sl4uoAGSPiNyaJ/wKXLXSxUQ+Jf5DnR1uUPEIiNdFHxdNIPoNP3g1dUorlKO9F00QQFueak WrMtzZRCMG2W2CdOnrV3K7wRbnK9TzEBERpWwkRcnY4+sEVmcwEmr49wihsgbMGM1UUNjG009lOZ1 XialFtLw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8Wg-00EAtr-Fa; Mon, 08 Aug 2022 19:34:58 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 14/59] mm/swap: Convert __read_swap_cache_async() to use a folio Date: Mon, 8 Aug 2022 20:33:42 +0100 Message-Id: <20220808193430.3378317-15-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987312; a=rsa-sha256; cv=none; b=hIx8BwUU1ZcFgda43Jv2BsY562lJkRy6H6W484S8TWdAyZ4rXc/D57EkQngwS9Gim0JUTl G+CT81IMCLrwogSqQ3S1/9IvwzLMXbCV2TmYZngktwjB9m0JwJtTpwEx70DQbSHY/Xc1tf 7b9ISLSSL8D3YyRv25KLYHA1G28xMyY= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=vCA0PVi0; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987312; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EtUfHusf3N70jYSPPPixSjqlm8bwcBxVI5pl2nsQ5N4=; b=ZKDeyR3c1Swi4LKqCxBfspsW6ARDoguZG8s4sGv9fk65CswUzgzKMMptzUVQC+8qv7yQoz 6vSpa/WrUnHVFT8VThM7wcxx4L2Uha9sIsQfP5WsSoD5sQJWZJgPzecey0VsvegMXzF8ft 0iazNCPO+3buQYbQfn5x4UnOmn9Mcsk= X-Rspamd-Server: rspam10 X-Stat-Signature: gqhn58c3ysijjbcoqxdaxsxa3fpxfiyt Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=vCA0PVi0; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Rspamd-Queue-Id: 208F8180169 X-HE-Tag: 1659987311-340910 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove a few hidden (and one visible) calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/swap_state.c | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/mm/swap_state.c b/mm/swap_state.c index 41afa6d45b23..b1e181fc5268 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -411,7 +411,7 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, bool *new_page_allocated) { struct swap_info_struct *si; - struct page *page; + struct folio *folio; void *shadow = NULL; *new_page_allocated = false; @@ -426,11 +426,11 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, si = get_swap_device(entry); if (!si) return NULL; - page = find_get_page(swap_address_space(entry), - swp_offset(entry)); + folio = filemap_get_folio(swap_address_space(entry), + swp_offset(entry)); put_swap_device(si); - if (page) - return page; + if (folio) + return folio_file_page(folio, swp_offset(entry)); /* * Just skip read ahead for unused swap slot. @@ -448,8 +448,8 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, * before marking swap_map SWAP_HAS_CACHE, when -EEXIST will * cause any racers to loop around until we add it to cache. */ - page = alloc_page_vma(gfp_mask, vma, addr); - if (!page) + folio = vma_alloc_folio(gfp_mask, 0, vma, addr, false); + if (!folio) return NULL; /* @@ -459,7 +459,7 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, if (!err) break; - put_page(page); + folio_put(folio); if (err != -EEXIST) return NULL; @@ -477,30 +477,30 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, * The swap entry is ours to swap in. Prepare the new page. */ - __SetPageLocked(page); - __SetPageSwapBacked(page); + __folio_set_locked(folio); + __folio_set_swapbacked(folio); - if (mem_cgroup_swapin_charge_page(page, NULL, gfp_mask, entry)) + if (mem_cgroup_swapin_charge_page(&folio->page, NULL, gfp_mask, entry)) goto fail_unlock; /* May fail (-ENOMEM) if XArray node allocation failed. */ - if (add_to_swap_cache(page, entry, gfp_mask & GFP_RECLAIM_MASK, &shadow)) + if (add_to_swap_cache(&folio->page, entry, gfp_mask & GFP_RECLAIM_MASK, &shadow)) goto fail_unlock; mem_cgroup_swapin_uncharge_swap(entry); if (shadow) - workingset_refault(page_folio(page), shadow); + workingset_refault(folio, shadow); - /* Caller will initiate read into locked page */ - lru_cache_add(page); + /* Caller will initiate read into locked folio */ + folio_add_lru(folio); *new_page_allocated = true; - return page; + return &folio->page; fail_unlock: - put_swap_page(page, entry); - unlock_page(page); - put_page(page); + put_swap_page(&folio->page, entry); + folio_unlock(folio); + folio_put(folio); return NULL; } From patchwork Mon Aug 8 19:33:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939080 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E724CC00140 for ; Mon, 8 Aug 2022 19:35:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC210940018; Mon, 8 Aug 2022 15:35:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C23056B0075; Mon, 8 Aug 2022 15:35:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A2FE9940018; Mon, 8 Aug 2022 15:35:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 869886B0074 for ; Mon, 8 Aug 2022 15:35:46 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 607BC80122 for ; Mon, 8 Aug 2022 19:35:46 +0000 (UTC) X-FDA: 79777430292.28.8C2CE49 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id 5EBAC2017B for ; Mon, 8 Aug 2022 19:35:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=idEOnUugnwQ9RQyH3KgSQbPxcJC+h+rJ4btW2BC2IeM=; b=EU2EeJVeqbhS75dv0RdAiI2wcu 4R6kglbfPviSFfo7uoYqY/lobW6CtFPWuGEQLmQ8JJ/8v0OLKNdy77Hs3idoS6TsygbdGxmGZy9k1 9XQh+bKOAWwLba/FXeCyW/go26e+Tv3eh7Y+rm7JGwZ0YVAVnbl/5L7sJbaEeMa1K/qTPg/e5LESN HQI2ABc0ky/W1Rezz0g6oTPGASp9pttM33xKVTZD8yW75WJnVkfp3s5Aqj5r5Oaj2odPX8maFFvJV OS9l+cLugNjFyVWd8Fhfy1m7Q5z9MCMFO1cR25eSvDnHHpsh8d93mUv7bBCtfqt2p6za+f6oidYp4 QOruIAiQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8Wp-00EAua-O8; Mon, 08 Aug 2022 19:35:10 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 15/59] mm/swap: Convert add_to_swap_cache() to take a folio Date: Mon, 8 Aug 2022 20:33:43 +0100 Message-Id: <20220808193430.3378317-16-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987345; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=idEOnUugnwQ9RQyH3KgSQbPxcJC+h+rJ4btW2BC2IeM=; b=bXlphIQ38zTMKMl6kDSHeqjJlnbY+9cnmfyoPBh/rzTzBv8dCl8qZ9dbfMfr/L7Cs7ifFP B/QAPu9FGm7KSOAe9JujSuGVMCzN/hH4BTfeFZQQsadb9NwQ/U25JHleZJD6qtHxjbdZCz 9tfNVDjmSG+Gvn9DXevO9M07eDUZKbw= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=EU2EeJVe; dmarc=none; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987345; a=rsa-sha256; cv=none; b=b9wR5vbc+OtjO1c9hoBW5aVh+sDQry9c2Io8rchnFWjk91wGK3SZRXrjRS4ZWZujRwNqF1 0nKHQKuHox+Lj0HbgYkE1tjz87KC16cCiK3cz6Ka9pBPGTPp5oW3ldcmL8GX2uH+51i5M7 M269KkWXb0CLV8nWQz2sYdJYamFVJ5Q= X-Rspamd-Queue-Id: 5EBAC2017B X-Rspamd-Server: rspam03 X-Rspam-User: Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=EU2EeJVe; dmarc=none; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Stat-Signature: 6c8iia818xitkw7txjpwbt4rqkmuowr4 X-HE-Tag: 1659987345-817919 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With all callers using folios, we can convert add_to_swap_cache() to take a folio and use it throughout. Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 2 +- mm/swap.h | 4 ++-- mm/swap_state.c | 34 +++++++++++++++++----------------- 3 files changed, 20 insertions(+), 20 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index eec32307984d..49c3f59b5b76 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1406,7 +1406,7 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) if (list_empty(&info->swaplist)) list_add(&info->swaplist, &shmem_swaplist); - if (add_to_swap_cache(&folio->page, swap, + if (add_to_swap_cache(folio, swap, __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN, NULL) == 0) { spin_lock_irq(&info->lock); diff --git a/mm/swap.h b/mm/swap.h index 17936e068c1c..0e023765e110 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -34,7 +34,7 @@ extern struct address_space *swapper_spaces[]; void show_swap_cache_info(void); bool add_to_swap(struct folio *folio); void *get_shadow_from_swap_cache(swp_entry_t entry); -int add_to_swap_cache(struct page *page, swp_entry_t entry, +int add_to_swap_cache(struct folio *folio, swp_entry_t entry, gfp_t gfp, void **shadowp); void __delete_from_swap_cache(struct folio *folio, swp_entry_t entry, void *shadow); @@ -124,7 +124,7 @@ static inline void *get_shadow_from_swap_cache(swp_entry_t entry) return NULL; } -static inline int add_to_swap_cache(struct page *page, swp_entry_t entry, +static inline int add_to_swap_cache(struct folio *folio, swp_entry_t entry, gfp_t gfp_mask, void **shadowp) { return -1; diff --git a/mm/swap_state.c b/mm/swap_state.c index b1e181fc5268..ecf1accc2fb1 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -85,21 +85,21 @@ void *get_shadow_from_swap_cache(swp_entry_t entry) * add_to_swap_cache resembles filemap_add_folio on swapper_space, * but sets SwapCache flag and private instead of mapping and index. */ -int add_to_swap_cache(struct page *page, swp_entry_t entry, +int add_to_swap_cache(struct folio *folio, swp_entry_t entry, gfp_t gfp, void **shadowp) { struct address_space *address_space = swap_address_space(entry); pgoff_t idx = swp_offset(entry); - XA_STATE_ORDER(xas, &address_space->i_pages, idx, compound_order(page)); - unsigned long i, nr = thp_nr_pages(page); + XA_STATE_ORDER(xas, &address_space->i_pages, idx, folio_order(folio)); + unsigned long i, nr = folio_nr_pages(folio); void *old; - VM_BUG_ON_PAGE(!PageLocked(page), page); - VM_BUG_ON_PAGE(PageSwapCache(page), page); - VM_BUG_ON_PAGE(!PageSwapBacked(page), page); + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + VM_BUG_ON_FOLIO(folio_test_swapcache(folio), folio); + VM_BUG_ON_FOLIO(!folio_test_swapbacked(folio), folio); - page_ref_add(page, nr); - SetPageSwapCache(page); + folio_ref_add(folio, nr); + folio_set_swapcache(folio); do { xas_lock_irq(&xas); @@ -107,19 +107,19 @@ int add_to_swap_cache(struct page *page, swp_entry_t entry, if (xas_error(&xas)) goto unlock; for (i = 0; i < nr; i++) { - VM_BUG_ON_PAGE(xas.xa_index != idx + i, page); + VM_BUG_ON_FOLIO(xas.xa_index != idx + i, folio); old = xas_load(&xas); if (xa_is_value(old)) { if (shadowp) *shadowp = old; } - set_page_private(page + i, entry.val + i); - xas_store(&xas, page); + set_page_private(folio_page(folio, i), entry.val + i); + xas_store(&xas, folio); xas_next(&xas); } address_space->nrpages += nr; - __mod_node_page_state(page_pgdat(page), NR_FILE_PAGES, nr); - __mod_lruvec_page_state(page, NR_SWAPCACHE, nr); + __node_stat_mod_folio(folio, NR_FILE_PAGES, nr); + __lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr); unlock: xas_unlock_irq(&xas); } while (xas_nomem(&xas, gfp)); @@ -127,8 +127,8 @@ int add_to_swap_cache(struct page *page, swp_entry_t entry, if (!xas_error(&xas)) return 0; - ClearPageSwapCache(page); - page_ref_sub(page, nr); + folio_clear_swapcache(folio); + folio_ref_sub(folio, nr); return xas_error(&xas); } @@ -194,7 +194,7 @@ bool add_to_swap(struct folio *folio) /* * Add it to the swap cache. */ - err = add_to_swap_cache(&folio->page, entry, + err = add_to_swap_cache(folio, entry, __GFP_HIGH|__GFP_NOMEMALLOC|__GFP_NOWARN, NULL); if (err) /* @@ -484,7 +484,7 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, goto fail_unlock; /* May fail (-ENOMEM) if XArray node allocation failed. */ - if (add_to_swap_cache(&folio->page, entry, gfp_mask & GFP_RECLAIM_MASK, &shadow)) + if (add_to_swap_cache(folio, entry, gfp_mask & GFP_RECLAIM_MASK, &shadow)) goto fail_unlock; mem_cgroup_swapin_uncharge_swap(entry); From patchwork Mon Aug 8 19:33:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939053 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90975C00140 for ; Mon, 8 Aug 2022 19:35:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B3AE294000D; Mon, 8 Aug 2022 15:35:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AC25D940007; Mon, 8 Aug 2022 15:35:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8EDF094000D; Mon, 8 Aug 2022 15:35:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 742A1940007 for ; Mon, 8 Aug 2022 15:35:13 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 51E90AB3BE for ; Mon, 8 Aug 2022 19:35:13 +0000 (UTC) X-FDA: 79777428906.05.E884985 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf04.hostedemail.com (Postfix) with ESMTP id D9A464016F for ; Mon, 8 Aug 2022 19:35:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=r0t1Hz+hsni9kEv0k6vlGiW0j65xLGBEaD2IUw319+E=; b=ERzxlbeksApbqO950flYMexPd+ EQhOkMkhXCriRi0Qug26esVKvG4duk7yUCoQnzpDam43cLTK2Z+ggrLEtaA0BQvTMHQmhqMg/XBXk mo8k44aDarP4sWUD2hmnyGgH2VUFjhTuZ8XV4KCQGcGmgukv3AVk/Vfqc0RhGlDpRDI37hu5H/Kz+ aPF+HtMuSgE5MsLmFrMmORUeWfoef8qLcZMvGcIOB54KbKKlu4bzQPOKtuMrXzL1hZSjSIfkdQQeG 5qNQSNORaZGnFj0VSa+ydwOG+bWM8e3gNzWyjGUKzD8CuUV3eYbVS9YF8rTcw1rVwiSlrZzQO3Ofk GtX4ighg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8Ws-00EAv3-U4; Mon, 08 Aug 2022 19:35:11 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 16/59] mm/swap: Convert put_swap_page() to put_swap_folio() Date: Mon, 8 Aug 2022 20:33:44 +0100 Message-Id: <20220808193430.3378317-17-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987312; a=rsa-sha256; cv=none; b=gxnSrYis3jmyNfWbVoM39N5CgN97EjhczJb+bNTTJNdh0ecO9kxzIbubHNEe0Lp+uWiBqD x7ghCYr4Y8ML4IXoEIbAJ8iOf5LUBVA+ZFR+PWpbG7L2LGZQQ4hCQkTIXBAWIA6/MYciXM wCSnbZ+rpKsMjkwNY0XPQOUvc7DFLew= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ERzxlbek; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987312; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=r0t1Hz+hsni9kEv0k6vlGiW0j65xLGBEaD2IUw319+E=; b=UyfRbP/0yK3GqjF+8dG/ZykD5MNZy9uEOSkDzTvQxg7KmH/mSM2fcc5P+v18f289bV2jQx zfMXF15aHWhD5Z+r9K+dn5vAKsaA87kMzCp3VZ66HnkDyWjCSYJftBHaSpPcOuvAu9Q6r1 T75fmyQSBkS9aGyuL9rfvN9Bu9W8Mtk= X-Stat-Signature: znoogubge4yyjrfhkm33dduhgz14bfng X-Rspamd-Queue-Id: D9A464016F Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ERzxlbek; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1659987312-964351 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With all callers now using a folio, we can convert this function. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/swap.h | 4 ++-- mm/shmem.c | 2 +- mm/swap_slots.c | 2 +- mm/swap_state.c | 6 +++--- mm/swapfile.c | 4 ++-- mm/vmscan.c | 2 +- 6 files changed, 10 insertions(+), 10 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 4595cbc1cb02..f16c9af6bf32 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -487,7 +487,7 @@ static inline long get_nr_swap_pages(void) extern void si_swapinfo(struct sysinfo *); swp_entry_t folio_alloc_swap(struct folio *folio); bool folio_free_swap(struct folio *folio); -extern void put_swap_page(struct page *page, swp_entry_t entry); +void put_swap_folio(struct folio *folio, swp_entry_t entry); extern swp_entry_t get_swap_page_of_type(int); extern int get_swap_pages(int n, swp_entry_t swp_entries[], int entry_size); extern int add_swap_count_continuation(swp_entry_t, gfp_t); @@ -572,7 +572,7 @@ static inline void swap_free(swp_entry_t swp) { } -static inline void put_swap_page(struct page *page, swp_entry_t swp) +static inline void put_swap_folio(struct folio *folio, swp_entry_t swp) { } diff --git a/mm/shmem.c b/mm/shmem.c index 49c3f59b5b76..4693edb33648 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1424,7 +1424,7 @@ static int shmem_writepage(struct page *page, struct writeback_control *wbc) } mutex_unlock(&shmem_swaplist_mutex); - put_swap_page(&folio->page, swap); + put_swap_folio(folio, swap); redirty: folio_mark_dirty(folio); if (wbc->for_reclaim) diff --git a/mm/swap_slots.c b/mm/swap_slots.c index 10b94d64cc25..0bec1f705f8e 100644 --- a/mm/swap_slots.c +++ b/mm/swap_slots.c @@ -343,7 +343,7 @@ swp_entry_t folio_alloc_swap(struct folio *folio) get_swap_pages(1, &entry, 1); out: if (mem_cgroup_try_charge_swap(folio, entry)) { - put_swap_page(&folio->page, entry); + put_swap_folio(folio, entry); entry.val = 0; } return entry; diff --git a/mm/swap_state.c b/mm/swap_state.c index ecf1accc2fb1..ea354efd3735 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -218,7 +218,7 @@ bool add_to_swap(struct folio *folio) return true; fail: - put_swap_page(&folio->page, entry); + put_swap_folio(folio, entry); return false; } @@ -237,7 +237,7 @@ void delete_from_swap_cache(struct folio *folio) __delete_from_swap_cache(folio, entry, NULL); xa_unlock_irq(&address_space->i_pages); - put_swap_page(&folio->page, entry); + put_swap_folio(folio, entry); folio_ref_sub(folio, folio_nr_pages(folio)); } @@ -498,7 +498,7 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, return &folio->page; fail_unlock: - put_swap_page(&folio->page, entry); + put_swap_folio(folio, entry); folio_unlock(folio); folio_put(folio); return NULL; diff --git a/mm/swapfile.c b/mm/swapfile.c index 880871f4c6d4..186511a8ef4f 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1328,7 +1328,7 @@ void swap_free(swp_entry_t entry) /* * Called after dropping swapcache to decrease refcnt to swap entries. */ -void put_swap_page(struct page *page, swp_entry_t entry) +void put_swap_folio(struct folio *folio, swp_entry_t entry) { unsigned long offset = swp_offset(entry); unsigned long idx = offset / SWAPFILE_CLUSTER; @@ -1337,7 +1337,7 @@ void put_swap_page(struct page *page, swp_entry_t entry) unsigned char *map; unsigned int i, free_entries = 0; unsigned char val; - int size = swap_entry_size(thp_nr_pages(page)); + int size = swap_entry_size(folio_nr_pages(folio)); si = _swap_info_get(entry); if (!si) diff --git a/mm/vmscan.c b/mm/vmscan.c index d3e26712dbc1..ac7f6f77e28a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1339,7 +1339,7 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio, shadow = workingset_eviction(folio, target_memcg); __delete_from_swap_cache(folio, swap, shadow); xa_unlock_irq(&mapping->i_pages); - put_swap_page(&folio->page, swap); + put_swap_folio(folio, swap); } else { void (*free_folio)(struct folio *); From patchwork Mon Aug 8 19:33:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939054 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32F63C25B0C for ; Mon, 8 Aug 2022 19:35:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9188394000E; Mon, 8 Aug 2022 15:35:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8A1A5940007; Mon, 8 Aug 2022 15:35:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5BBCA94000E; Mon, 8 Aug 2022 15:35:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 347F8940007 for ; Mon, 8 Aug 2022 15:35:16 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0D39A8016B for ; Mon, 8 Aug 2022 19:35:16 +0000 (UTC) X-FDA: 79777429032.17.DBB6363 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf02.hostedemail.com (Postfix) with ESMTP id 561AF80171 for ; Mon, 8 Aug 2022 19:35:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=m1WsqyDeTWbIGRO38ZCMMnf2bX8TTKK3IemcukWSpSE=; b=P2Uo/wYjj2ydFTTeakzx3Z0qT8 JBFslqdjbOzQ//I2TsN82dDXrmBNbdo38OaIesUdzUDFIv/FEfvGy0/9xT7ov7WrVR8DdZCt7Ce4w gDbsPDCiN8nj5e7AXlrOS6QVUVHSILXep8DAZR+RxIkIytzL1bADW0uZf2AeaOn7uKq1N3AJv5trY jYyIr0pUvCSOlzqPiBG9kSOniZqRYDP/NiFfLRtbM/eVJk6RwAf0puLWEIuuv4QNpMd44Jr0saTey pj3u0snoqdKKVgjwOuhP87CQKrtYKE4yoFsUn1iHSwUOwCdDNRrK2r0OTywNNWoE31K+VUdYd5V1p a253HEYQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8Wt-00EAvD-54; Mon, 08 Aug 2022 19:35:11 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 17/59] mm: Convert do_swap_page() to use a folio Date: Mon, 8 Aug 2022 20:33:45 +0100 Message-Id: <20220808193430.3378317-18-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987315; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=m1WsqyDeTWbIGRO38ZCMMnf2bX8TTKK3IemcukWSpSE=; b=40MSiyxZ/HdjGbsljfGOfPP2MJx7RKsCRSc2kcE3Ik/kUYLw3flPkjvlq0nB8fl3RNAmZy bjM4QMZcm3Z00Wd17BPWrvEbetY9gp5favUaWv7/H399br6tutB1PNGs12wTzFzAKGcRF+ BWGzT2XiY3k9UQNPluMByqwecthtiUs= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="P2Uo/wYj"; dmarc=none; spf=none (imf02.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987315; a=rsa-sha256; cv=none; b=ji60WPg3TkfX4PWQifGO8foop/NbH+FmzdnrZu62OujJNL3B6RHVaqRUGtvUYBhgYxGIu2 Rni10HIP0QVKJ7YGcyPe+KTWwrXyQEIINpoC/Ly+PCKZaXHPozQNgaOIHqiEcX1PUmDJJu BtAT6SpJqUmKoh/tcUxGbjry866Fnk4= Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="P2Uo/wYj"; dmarc=none; spf=none (imf02.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspamd-Server: rspam02 X-Stat-Signature: 4mbu1tpuugsq6oes4nrzndc6wukfzukf X-Rspamd-Queue-Id: 561AF80171 X-Rspam-User: X-HE-Tag: 1659987315-788816 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Removes quite a lot of calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 57 +++++++++++++++++++++++++++++++---------------------- 1 file changed, 33 insertions(+), 24 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 4ba73f5aa8bb..f172b148e29b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3718,6 +3718,7 @@ static vm_fault_t handle_pte_marker(struct vm_fault *vmf) vm_fault_t do_swap_page(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; + struct folio *folio; struct page *page = NULL, *swapcache; struct swap_info_struct *si = NULL; rmap_t rmap_flags = RMAP_NONE; @@ -3762,19 +3763,23 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) page = lookup_swap_cache(entry, vma, vmf->address); swapcache = page; + if (page) + folio = page_folio(page); if (!page) { if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && __swap_count(entry) == 1) { /* skip swapcache */ - page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, - vmf->address); - if (page) { - __SetPageLocked(page); - __SetPageSwapBacked(page); + folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, + vma, vmf->address, false); + page = &folio->page; + if (folio) { + __folio_set_locked(folio); + __folio_set_swapbacked(folio); if (mem_cgroup_swapin_charge_page(page, - vma->vm_mm, GFP_KERNEL, entry)) { + vma->vm_mm, GFP_KERNEL, + entry)) { ret = VM_FAULT_OOM; goto out_page; } @@ -3782,20 +3787,21 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) shadow = get_shadow_from_swap_cache(entry); if (shadow) - workingset_refault(page_folio(page), - shadow); + workingset_refault(folio, shadow); - lru_cache_add(page); + folio_add_lru(folio); /* To provide entry to swap_readpage() */ - set_page_private(page, entry.val); + folio_set_swap_entry(folio, entry); swap_readpage(page, true, NULL); - set_page_private(page, 0); + folio->private = NULL; } } else { page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vmf); swapcache = page; + if (page) + folio = page_folio(page); } if (!page) { @@ -3838,7 +3844,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * swapcache, we need to check that the page's swap has not * changed. */ - if (unlikely(!PageSwapCache(page) || + if (unlikely(!folio_test_swapcache(folio) || page_private(page) != entry.val)) goto out_page; @@ -3853,6 +3859,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) page = swapcache; goto out_page; } + folio = page_folio(page); /* * If we want to map a page that's in the swapcache writable, we @@ -3861,7 +3868,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * pagevecs if required. */ if ((vmf->flags & FAULT_FLAG_WRITE) && page == swapcache && - !PageKsm(page) && !PageLRU(page)) + !folio_test_ksm(folio) && !folio_test_lru(folio)) lru_add_drain(); } @@ -3875,7 +3882,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) goto out_nomap; - if (unlikely(!PageUptodate(page))) { + if (unlikely(!folio_test_uptodate(folio))) { ret = VM_FAULT_SIGBUS; goto out_nomap; } @@ -3888,14 +3895,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * check after taking the PT lock and making sure that nobody * concurrently faulted in this page and set PG_anon_exclusive. */ - BUG_ON(!PageAnon(page) && PageMappedToDisk(page)); - BUG_ON(PageAnon(page) && PageAnonExclusive(page)); + BUG_ON(!folio_test_anon(folio) && folio_test_mappedtodisk(folio)); + BUG_ON(folio_test_anon(folio) && PageAnonExclusive(page)); /* * Check under PT lock (to protect against concurrent fork() sharing * the swap entry concurrently) for certainly exclusive pages. */ - if (!PageKsm(page)) { + if (!folio_test_ksm(folio)) { /* * Note that pte_swp_exclusive() == false for architectures * without __HAVE_ARCH_PTE_SWP_EXCLUSIVE. @@ -3907,7 +3914,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * swapcache -> certainly exclusive. */ exclusive = true; - } else if (exclusive && PageWriteback(page) && + } else if (exclusive && folio_test_writeback(folio) && data_race(si->flags & SWP_STABLE_WRITES)) { /* * This is tricky: not all swap backends support @@ -3950,7 +3957,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * exposing them to the swapcache or because the swap entry indicates * exclusivity. */ - if (!PageKsm(page) && (exclusive || page_count(page) == 1)) { + if (!folio_test_ksm(folio) && + (exclusive || folio_ref_count(folio) == 1)) { if (vmf->flags & FAULT_FLAG_WRITE) { pte = maybe_mkwrite(pte_mkdirty(pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE; @@ -3970,16 +3978,17 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) /* ksm created a completely new copy */ if (unlikely(page != swapcache && swapcache)) { page_add_new_anon_rmap(page, vma, vmf->address); - lru_cache_add_inactive_or_unevictable(page, vma); + folio_add_lru_vma(folio, vma); } else { page_add_anon_rmap(page, vma, vmf->address, rmap_flags); } - VM_BUG_ON(!PageAnon(page) || (pte_write(pte) && !PageAnonExclusive(page))); + VM_BUG_ON(!folio_test_anon(folio) || + (pte_write(pte) && !PageAnonExclusive(page))); set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); arch_do_swap_page(vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); - unlock_page(page); + folio_unlock(folio); if (page != swapcache && swapcache) { /* * Hold the lock to avoid the swap entry to be reused @@ -4011,9 +4020,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) out_nomap: pte_unmap_unlock(vmf->pte, vmf->ptl); out_page: - unlock_page(page); + folio_unlock(folio); out_release: - put_page(page); + folio_put(folio); if (page != swapcache && swapcache) { unlock_page(swapcache); put_page(swapcache); From patchwork Mon Aug 8 19:33:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939056 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2408BC00140 for ; Mon, 8 Aug 2022 19:35:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 76A76940011; Mon, 8 Aug 2022 15:35:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6FF8F94000F; Mon, 8 Aug 2022 15:35:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 40C57940011; Mon, 8 Aug 2022 15:35:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 25B4294000F for ; Mon, 8 Aug 2022 15:35:19 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id F09AD80122 for ; Mon, 8 Aug 2022 19:35:18 +0000 (UTC) X-FDA: 79777429116.07.D0DEE24 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id A194F120063 for ; Mon, 8 Aug 2022 19:35:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=pL+3uDREeON3UijNk2fSt/B1Lgs/D8FgAX6uutYNL7w=; b=C58w4GYGaZ3mWkFOKJ0LvL71Ap mSwBSMjCiwrQLyADowLxO/EdAxuTKdttnYkdTwlHkCTN9gfzSN3/0UGG/bprMtrjowIRh62dLwfWv Nwky9fYsCpi5LMJrkcNJfPbpUV8x7mZW3yLkKRtKgHha8/KQkU5x1aPEv45pa9atugC9VuohKdM9z bUt3uu5cGqdYMfIN2J6A3cVBvtwI5dn/EmBwju3H7l90NVcH/mgWfK0Gxme+f4WaLDd0P0B7jzcB0 InPoGmzDYKPwBKFWg4qw6kzBkvIvPD0huUlltuCJIUSIiAt/YQtpxqVj4zldfb7RbtPtfYzGOnxFa YFTKCWng==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8Wt-00EAvM-QM; Mon, 08 Aug 2022 19:35:14 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 18/59] mm: Convert do_swap_page()'s swapcache variable to a folio Date: Mon, 8 Aug 2022 20:33:46 +0100 Message-Id: <20220808193430.3378317-19-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987318; a=rsa-sha256; cv=none; b=jvOOsUbE7nXWP8pRGsF7vM0wd9a3syBBn7aPmowgJy6/3gjrcN2kNt2DYXNv5wp7/fa53l W8BlCwzySiDsR4lGYIwwFOwzuCPiEDBsMyD3Z33OMPQ1/3jucVOg7oJ5mjC8lIWLzbrz/Y psM9GFvrgQ918MkNJQ0jSGxDtJDxt5Y= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=C58w4GYG; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987318; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pL+3uDREeON3UijNk2fSt/B1Lgs/D8FgAX6uutYNL7w=; b=Gbk9WtUZIreEyqb/QT0UVJBkagl8K0y9AG/2zqLdro6uMRE5nkahIHC1qUR3unwPT+HYnP vIZyCeXvT48N9Odba46ZO+dXxKKsstGW03DYHYBpnB/a8SG4/RZyMydlYEEDecIBySU87+ ZgR9mMZ1SR2KygCmy1gfVpakSooyscs= X-Stat-Signature: 7hnpwcodpkucbr9m7ua5ms7xerk6t938 X-Rspamd-Queue-Id: A194F120063 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=C58w4GYG; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1659987318-90888 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The 'swapcache' variable is used to track whether the page is from the swapcache or not. It can do this equally well by being the folio of the page rather than the page itself, and this saves a number of calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index f172b148e29b..471102f0cbf2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3718,8 +3718,8 @@ static vm_fault_t handle_pte_marker(struct vm_fault *vmf) vm_fault_t do_swap_page(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; - struct folio *folio; - struct page *page = NULL, *swapcache; + struct folio *swapcache, *folio = NULL; + struct page *page; struct swap_info_struct *si = NULL; rmap_t rmap_flags = RMAP_NONE; bool exclusive = false; @@ -3762,11 +3762,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out; page = lookup_swap_cache(entry, vma, vmf->address); - swapcache = page; if (page) folio = page_folio(page); + swapcache = folio; - if (!page) { + if (!folio) { if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && __swap_count(entry) == 1) { /* skip swapcache */ @@ -3799,12 +3799,12 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } else { page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vmf); - swapcache = page; if (page) folio = page_folio(page); + swapcache = folio; } - if (!page) { + if (!folio) { /* * Back out if somebody else faulted in this pte * while we released the pte lock. @@ -3856,10 +3856,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) page = ksm_might_need_to_copy(page, vma, vmf->address); if (unlikely(!page)) { ret = VM_FAULT_OOM; - page = swapcache; goto out_page; } folio = page_folio(page); + swapcache = folio; /* * If we want to map a page that's in the swapcache writable, we @@ -3867,7 +3867,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * owner. Try removing the extra reference from the local LRU * pagevecs if required. */ - if ((vmf->flags & FAULT_FLAG_WRITE) && page == swapcache && + if ((vmf->flags & FAULT_FLAG_WRITE) && folio == swapcache && !folio_test_ksm(folio) && !folio_test_lru(folio)) lru_add_drain(); } @@ -3908,7 +3908,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * without __HAVE_ARCH_PTE_SWP_EXCLUSIVE. */ exclusive = pte_swp_exclusive(vmf->orig_pte); - if (page != swapcache) { + if (folio != swapcache) { /* * We have a fresh page that is not exposed to the * swapcache -> certainly exclusive. @@ -3976,7 +3976,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vmf->orig_pte = pte; /* ksm created a completely new copy */ - if (unlikely(page != swapcache && swapcache)) { + if (unlikely(folio != swapcache && swapcache)) { page_add_new_anon_rmap(page, vma, vmf->address); folio_add_lru_vma(folio, vma); } else { @@ -3989,7 +3989,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) arch_do_swap_page(vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); folio_unlock(folio); - if (page != swapcache && swapcache) { + if (folio != swapcache && swapcache) { /* * Hold the lock to avoid the swap entry to be reused * until we take the PT lock for the pte_same() check @@ -3998,8 +3998,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * so that the swap count won't change under a * parallel locked swapcache. */ - unlock_page(swapcache); - put_page(swapcache); + folio_unlock(swapcache); + folio_put(swapcache); } if (vmf->flags & FAULT_FLAG_WRITE) { @@ -4023,9 +4023,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) folio_unlock(folio); out_release: folio_put(folio); - if (page != swapcache && swapcache) { - unlock_page(swapcache); - put_page(swapcache); + if (folio != swapcache && swapcache) { + folio_unlock(swapcache); + folio_put(swapcache); } if (si) put_swap_device(si); From patchwork Mon Aug 8 19:33:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939055 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F19EC25B0C for ; Mon, 8 Aug 2022 19:35:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 30370940010; Mon, 8 Aug 2022 15:35:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 236B0940007; Mon, 8 Aug 2022 15:35:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0B0F294000F; Mon, 8 Aug 2022 15:35:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E9195940007 for ; Mon, 8 Aug 2022 15:35:18 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id BE90E1C6453 for ; Mon, 8 Aug 2022 19:35:18 +0000 (UTC) X-FDA: 79777429116.17.1EA17A9 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf09.hostedemail.com (Postfix) with ESMTP id 62374140025 for ; Mon, 8 Aug 2022 19:35:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=UpApTrl+fGVezBdm8XccGS5Ph+9HSF3fVBpF2a3IHhc=; b=m/MCIClVd9YKLBTs0iYMWITuQp nF1/mvjxtrVRj7MXhpwuSD9niVqsTrDrHcuqjn1FDCv0rwOhJaRytiLRnepbjUKI+LdUSeTgTkSbQ gJazzZeoX39E04DrWsg5KES5uFPrMoAsftfudHVXwRctbHWI492QR99p9K90E3JXCU9urgC0tQtin G9VMiZGKwcLP3qJc9seGO7UD2Nq1yq2aGZfDkwwiiTLKPhYYamsSXqUMsEzMmHzhpFS8E809ggW5e 564bvZQGQ22SNJRUIMUQqSjfBwigvA8tvM22GlFEHmSE/Tl5ciHhznv9JBesPsTBIc1E7q+LoMbDN lEaMRy+g==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8Ww-00EAvZ-6V; Mon, 08 Aug 2022 19:35:14 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 19/59] memcg: Convert mem_cgroup_swapin_charge_page() to mem_cgroup_swapin_charge_folio() Date: Mon, 8 Aug 2022 20:33:47 +0100 Message-Id: <20220808193430.3378317-20-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987318; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UpApTrl+fGVezBdm8XccGS5Ph+9HSF3fVBpF2a3IHhc=; b=b8BbVl5DRmXwcS2rrT5Svanz0lJTwJ3VFxVd9yMznNjni50cnQ7xz96f2QBlSdqTgXRvoh KPVb+4K4BY4R2r64prVdrSLdhmG8xlBfw+Q7wr4xmYH+CAjCqZ9/YJDRdoCLe1x+LdVdH9 hWpgUBq4xAPBr1b2sf2pdux7ogdM/Fo= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="m/MCIClV"; dmarc=none; spf=none (imf09.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987318; a=rsa-sha256; cv=none; b=CYbR/aPKIagbDkxMUnJvfWBzgq7gV4Gt1Q7Xr9kisSVutr2M6OcR8YuFQh08Bvri2fj8t+ W0ZHtMouipx5Lg3NKAWOwEaD5U1ZsGb4472Lqud3C94UbX1QRtG419JwE8Z+nCTnCPkQh/ smqxveA9qyIwV9uDQrkd3OizxRHy+/o= X-Rspamd-Queue-Id: 62374140025 Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="m/MCIClV"; dmarc=none; spf=none (imf09.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: 4q1nde3irxk7d3jbb6t5ykrgscyfjptu X-HE-Tag: 1659987318-931329 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: All callers now have a folio, so pass it in here and remove an unnecessary call to page_folio(). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/memcontrol.h | 4 ++-- mm/memcontrol.c | 13 ++++++------- mm/memory.c | 2 +- mm/swap_state.c | 2 +- 4 files changed, 10 insertions(+), 11 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 4d31ce55b1c0..5d7e834f37ba 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -689,7 +689,7 @@ static inline int mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, return __mem_cgroup_charge(folio, mm, gfp); } -int mem_cgroup_swapin_charge_page(struct page *page, struct mm_struct *mm, +int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm, gfp_t gfp, swp_entry_t entry); void mem_cgroup_swapin_uncharge_swap(swp_entry_t entry); @@ -1227,7 +1227,7 @@ static inline int mem_cgroup_charge(struct folio *folio, return 0; } -static inline int mem_cgroup_swapin_charge_page(struct page *page, +static inline int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm, gfp_t gfp, swp_entry_t entry) { return 0; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index b69979c9ced5..f65833efe90d 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6813,21 +6813,20 @@ int __mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, gfp_t gfp) } /** - * mem_cgroup_swapin_charge_page - charge a newly allocated page for swapin - * @page: page to charge + * mem_cgroup_swapin_charge_folio - Charge a newly allocated folio for swapin. + * @folio: folio to charge. * @mm: mm context of the victim * @gfp: reclaim mode - * @entry: swap entry for which the page is allocated + * @entry: swap entry for which the folio is allocated * - * This function charges a page allocated for swapin. Please call this before - * adding the page to the swapcache. + * This function charges a folio allocated for swapin. Please call this before + * adding the folio to the swapcache. * * Returns 0 on success. Otherwise, an error code is returned. */ -int mem_cgroup_swapin_charge_page(struct page *page, struct mm_struct *mm, +int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm, gfp_t gfp, swp_entry_t entry) { - struct folio *folio = page_folio(page); struct mem_cgroup *memcg; unsigned short id; int ret; diff --git a/mm/memory.c b/mm/memory.c index 471102f0cbf2..23b164bf3c70 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3777,7 +3777,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) __folio_set_locked(folio); __folio_set_swapbacked(folio); - if (mem_cgroup_swapin_charge_page(page, + if (mem_cgroup_swapin_charge_folio(folio, vma->vm_mm, GFP_KERNEL, entry)) { ret = VM_FAULT_OOM; diff --git a/mm/swap_state.c b/mm/swap_state.c index ea354efd3735..a7e0438902dd 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -480,7 +480,7 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, __folio_set_locked(folio); __folio_set_swapbacked(folio); - if (mem_cgroup_swapin_charge_page(&folio->page, NULL, gfp_mask, entry)) + if (mem_cgroup_swapin_charge_folio(folio, NULL, gfp_mask, entry)) goto fail_unlock; /* May fail (-ENOMEM) if XArray node allocation failed. */ From patchwork Mon Aug 8 19:33:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939057 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96AB9C25B0C for ; Mon, 8 Aug 2022 19:35:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D10A5940012; Mon, 8 Aug 2022 15:35:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C4B5C940007; Mon, 8 Aug 2022 15:35:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 93E9A940013; Mon, 8 Aug 2022 15:35:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 5CC91940007 for ; Mon, 8 Aug 2022 15:35:19 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 2E28EAB3BE for ; Mon, 8 Aug 2022 19:35:19 +0000 (UTC) X-FDA: 79777429158.03.A7E4033 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf21.hostedemail.com (Postfix) with ESMTP id B75341C005A for ; Mon, 8 Aug 2022 19:35:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=6cHy2CgAmT8QbiJcz9fEbNPRrvcgWUDZeC1E/6gPUr8=; b=JVtd4uChBzXwrhrGzyBkoGveSd fHHis0Db7zAnYwh9OyrPpUVtbnW/MZRc7HoePmXc06xGXfeng0GPEF7MlxTKllU0toMAqVg2WEekW tJXzg4GI62VbOP5L8rgjWiSiheergTTgc0CUFOJ5k0PC9H15wrMtu8dE1N87cFtHfKiXvb26JxUdl rpYP4RIHMeWPq7YKrRvWCnafOEXtK1LdDVJF27RgcIlrfpI/v73uUbHWR4WfyL1K1sVFyW7hMdHR6 SqTQC8IqxDWa1LjSZJfn5MZLVwz1cnSqsDvvnYdQA4TxfRN6TdQwPtpOeGwrLjIsdL1vpML71yF5S 6S9ljnCw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8Ww-00EAvg-CT; Mon, 08 Aug 2022 19:35:17 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 20/59] shmem: Convert shmem_mfill_atomic_pte() to use a folio Date: Mon, 8 Aug 2022 20:33:48 +0100 Message-Id: <20220808193430.3378317-21-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=JVtd4uCh; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987318; a=rsa-sha256; cv=none; b=OJLrQJ/CzTG5THFRY7N4ubtDhRw3AYJODrwJgoJKWYBmsxuzsXaXeafEHQxpEoQa1022eB oHMr83G17SLv+vejn1SQIO25gJ+36r33S1khy6u5jB82Is4FlU8doCXE4Y1woQM6I48Eof UTsn8NXPuC0TE1Rf2b+AQAXokSjhCdc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987318; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6cHy2CgAmT8QbiJcz9fEbNPRrvcgWUDZeC1E/6gPUr8=; b=nG6vACgjkC8hK741kC/7SS/t1lqDTaO8kK2R52iHloUSY7zZoDFQABol8GZdVnL8g9y05B FwT4TL/J6nTnmxkk4t4SoMfuCu8u2HbccIs+Fs8Tn9FeHKMFO4NCyl8654ewrp+fdOyfBL ukPQanZqPF6l0cK96MiM4bqX361nxN8= X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: B75341C005A X-Rspam-User: Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=JVtd4uCh; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Stat-Signature: txmaf589zr5johhc58tmth89idph3c3s X-HE-Tag: 1659987318-594460 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Assert that this is a single-page folio as there are several assumptions in here that it's exactly PAGE_SIZE bytes large. Saves several calls to compound_head() and removes the last caller of shmem_alloc_page(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 45 +++++++++++++++++++-------------------------- 1 file changed, 19 insertions(+), 26 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 4693edb33648..e7fd1dfb2895 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2353,12 +2353,6 @@ static struct inode *shmem_get_inode(struct super_block *sb, struct inode *dir, } #ifdef CONFIG_USERFAULTFD -static struct page *shmem_alloc_page(gfp_t gfp, - struct shmem_inode_info *info, pgoff_t index) -{ - return &shmem_alloc_folio(gfp, info, index)->page; -} - int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, struct vm_area_struct *dst_vma, @@ -2374,7 +2368,6 @@ int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, pgoff_t pgoff = linear_page_index(dst_vma, dst_addr); void *page_kaddr; struct folio *folio; - struct page *page; int ret; pgoff_t max_off; @@ -2393,53 +2386,53 @@ int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, if (!*pagep) { ret = -ENOMEM; - page = shmem_alloc_page(gfp, info, pgoff); - if (!page) + folio = shmem_alloc_folio(gfp, info, pgoff); + if (!folio) goto out_unacct_blocks; if (!zeropage) { /* COPY */ - page_kaddr = kmap_atomic(page); + page_kaddr = kmap_local_folio(folio, 0); ret = copy_from_user(page_kaddr, (const void __user *)src_addr, PAGE_SIZE); - kunmap_atomic(page_kaddr); + kunmap_local(page_kaddr); /* fallback to copy_from_user outside mmap_lock */ if (unlikely(ret)) { - *pagep = page; + *pagep = &folio->page; ret = -ENOENT; /* don't free the page */ goto out_unacct_blocks; } - flush_dcache_page(page); + flush_dcache_folio(folio); } else { /* ZEROPAGE */ - clear_user_highpage(page, dst_addr); + clear_user_highpage(&folio->page, dst_addr); } } else { - page = *pagep; + folio = page_folio(*pagep); + VM_BUG_ON_FOLIO(folio_test_large(folio), folio); *pagep = NULL; } - VM_BUG_ON(PageLocked(page)); - VM_BUG_ON(PageSwapBacked(page)); - __SetPageLocked(page); - __SetPageSwapBacked(page); - __SetPageUptodate(page); + VM_BUG_ON(folio_test_locked(folio)); + VM_BUG_ON(folio_test_swapbacked(folio)); + __folio_set_locked(folio); + __folio_set_swapbacked(folio); + __folio_mark_uptodate(folio); ret = -EFAULT; max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); if (unlikely(pgoff >= max_off)) goto out_release; - folio = page_folio(page); ret = shmem_add_to_page_cache(folio, mapping, pgoff, NULL, gfp & GFP_RECLAIM_MASK, dst_mm); if (ret) goto out_release; ret = mfill_atomic_install_pte(dst_mm, dst_pmd, dst_vma, dst_addr, - page, true, wp_copy); + &folio->page, true, wp_copy); if (ret) goto out_delete_from_cache; @@ -2449,13 +2442,13 @@ int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, shmem_recalc_inode(inode); spin_unlock_irq(&info->lock); - unlock_page(page); + folio_unlock(folio); return 0; out_delete_from_cache: - delete_from_page_cache(page); + filemap_remove_folio(folio); out_release: - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); out_unacct_blocks: shmem_inode_unacct_blocks(inode, 1); return ret; From patchwork Mon Aug 8 19:33:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939058 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED86EC00140 for ; Mon, 8 Aug 2022 19:35:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 216F194000F; Mon, 8 Aug 2022 15:35:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0B40F940007; Mon, 8 Aug 2022 15:35:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D0D1794000F; Mon, 8 Aug 2022 15:35:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 7E346940012 for ; Mon, 8 Aug 2022 15:35:19 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 680FBA046F for ; Mon, 8 Aug 2022 19:35:19 +0000 (UTC) X-FDA: 79777429158.21.A15C711 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf08.hostedemail.com (Postfix) with ESMTP id F1492160172 for ; Mon, 8 Aug 2022 19:35:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=m7JOJji/bhVJxkubfkFOhgpm23cyZZ+v/2pPI0vnz+o=; b=rM9ymd/lXI8jTym5/BlNuAHJrQ ghpNY7LTpfUKeyS+RFSkf+OQiE4GtZGL78H+aste515lcE0Q9QjITSoGEEVoshdac4RXqFx7neM+i ut2Burm5HEA/YwWNNViEo25jPFZPPf0oTK2RiFlCR90X3l1vNh1USlsCYfA8F8c9vhe/FusHw1V5H wp35ybgLi/KOnQKNb8xBwFj3ZlG2+g8UbHAJb5LTs1gX1V4uji7qLPJu5AED6E9v5ZtEjci4rkYQR y65Tn7/tas+kNWpsUG3diww0aD62ldAINd0SrApnJvyMsZZ6S1PvRhL+clbNh+Db/+XVQqyaQc3bv 68Dr794Q==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8Wz-00EAw3-B7; Mon, 08 Aug 2022 19:35:17 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 21/59] shmem: Convert shmem_replace_page() to shmem_replace_folio() Date: Mon, 8 Aug 2022 20:33:49 +0100 Message-Id: <20220808193430.3378317-22-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987319; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=m7JOJji/bhVJxkubfkFOhgpm23cyZZ+v/2pPI0vnz+o=; b=NcqXlhgezL/6JKGum/sOK5T8hUgVDK7umk70+nonGOorMNWelzH7ptRPn6dtZHIGHQXkoU hzZiRomIQBYxAdqLqj/CRAUVhslSQcY1VCyU3G3rtBGu9LImY+bX49eKk0KDOfPC/xJOBP 0fA3vtefQ+CooOMokQImnBaZQw2mHEw= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="rM9ymd/l"; dmarc=none; spf=none (imf08.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987319; a=rsa-sha256; cv=none; b=NlO2gRKPMgL06rfar7UYmWGex+We911CVNZk5YPXOzkZJmV41cwIpCHAZ1yUmDRnEUnn3C mTFJSZFeQyKwb2wPo4t18woLENtb8IiYsTuH7BJ9+RqnP1Laipw4csU+B4HvcRz1TITlWl ScjTHE2N2pVnxktSWEmNe/njYwF615I= X-Rspamd-Queue-Id: F1492160172 X-Rspamd-Server: rspam03 X-Rspam-User: Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="rM9ymd/l"; dmarc=none; spf=none (imf08.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Stat-Signature: e331orp1gee597617ga33s7atb6skka3 X-HE-Tag: 1659987318-957682 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The caller has a folio, so convert the calling convention and rename the function. Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index e7fd1dfb2895..ac2b2ebfc9c9 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1608,7 +1608,7 @@ static bool shmem_should_replace_folio(struct folio *folio, gfp_t gfp) return folio_zonenum(folio) > gfp_zone(gfp); } -static int shmem_replace_page(struct page **pagep, gfp_t gfp, +static int shmem_replace_folio(struct folio **foliop, gfp_t gfp, struct shmem_inode_info *info, pgoff_t index) { struct folio *old, *new; @@ -1617,7 +1617,7 @@ static int shmem_replace_page(struct page **pagep, gfp_t gfp, pgoff_t swap_index; int error; - old = page_folio(*pagep); + old = *foliop; entry = folio_swap_entry(old); swap_index = swp_offset(entry); swap_mapping = swap_address_space(entry); @@ -1664,7 +1664,7 @@ static int shmem_replace_page(struct page **pagep, gfp_t gfp, old = new; } else { folio_add_lru(new); - *pagep = &new->page; + *foliop = new; } folio_clear_swapcache(old); @@ -1770,8 +1770,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, arch_swap_restore(swap, folio); if (shmem_should_replace_folio(folio, gfp)) { - error = shmem_replace_page(&page, gfp, info, index); - folio = page_folio(page); + error = shmem_replace_folio(&folio, gfp, info, index); if (error) goto failed; } From patchwork Mon Aug 8 19:33:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939059 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 485ECC25B08 for ; Mon, 8 Aug 2022 19:35:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4D20F940013; Mon, 8 Aug 2022 15:35:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 459AC940007; Mon, 8 Aug 2022 15:35:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D448940013; Mon, 8 Aug 2022 15:35:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 12563940007 for ; Mon, 8 Aug 2022 15:35:22 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id DDF06C02CA for ; Mon, 8 Aug 2022 19:35:21 +0000 (UTC) X-FDA: 79777429242.15.8F192CA Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id 8FC5F120156 for ; Mon, 8 Aug 2022 19:35:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=LV3i0kkJqsfpyICvVTbaOoAHHMn/zS4yYuJXoR6jl3w=; b=qb3aalkjk7ODd9Iawlzw+ph270 fGFsa7NqGAQv22X/lavHDC21wQwXuLkbupnvs5tVElSBTJuU8B1v+dQgvVrf8IzrQewo2vbQ7RoSo Slvd2kohH6MNGYlu/Fn+ytVJ6++zDGJInr342r+sjTd2RAsHf59CCsH4yEwU+CTTszVkrR+CaPOrr DWvc7sBuxqGr7FM4BA7PNapEUVQmL1pI6da+HCYwl25sIXsyaSacMow5qPQxW10qOpYdzm7HOsFBs o1ac4Jd/YNHXxVl2DDVzvlvaUANXvhD7d/FHWTt1VQMeV1XpGianQRaBoYyuBqJjB5WfF8YRpzgjk 7bY7D1YQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8Wz-00EAwe-K7; Mon, 08 Aug 2022 19:35:17 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 22/59] swap: Add swap_cache_get_folio() Date: Mon, 8 Aug 2022 20:33:50 +0100 Message-Id: <20220808193430.3378317-23-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987321; a=rsa-sha256; cv=none; b=kJTXh6sbMRyydNs3gxO9nXjQSCcy7uFV4V6CjNJwhC4kT5YortWe5H9Kd92HZd+5uGhGFr lPz8/yvqjucQIYPVDb5hrWcmADsdR/34r6GmlJqY88qceCMchFBhkqQbPgWGALLdp1aJEH dSGxFS4dA24MYvohJl08hq4Lqsm2UHc= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=qb3aalkj; dmarc=none; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987321; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LV3i0kkJqsfpyICvVTbaOoAHHMn/zS4yYuJXoR6jl3w=; b=D0xfa7ODpE7ojc8U7ZZOMwT1lQ6tm1bEL7/krE8QAXe3GT+F3vJPP29HZ3XEbOJ62nwavO j16Lxmw2euhZOVjhLOScEIWAUNmaZLY+RHkfrEjPZ8MsHycfP00N+5iEOzBtf6rgXoQrOc XidGc6kgPaMHMI9xEWJ7rophwsVeLCM= X-Rspamd-Server: rspam10 X-Stat-Signature: gwrw456rg9qkmpynot9e3nt6qidespep Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=qb3aalkj; dmarc=none; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Rspamd-Queue-Id: 8FC5F120156 X-HE-Tag: 1659987321-793174 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert lookup_swap_cache() into swap_cache_get_folio() and add a lookup_swap_cache() wrapper around it. Signed-off-by: Matthew Wilcox (Oracle) --- mm/swap.h | 2 ++ mm/swap_state.c | 32 +++++++++++++++++++++----------- 2 files changed, 23 insertions(+), 11 deletions(-) diff --git a/mm/swap.h b/mm/swap.h index 0e023765e110..f70ff34dab82 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -41,6 +41,8 @@ void __delete_from_swap_cache(struct folio *folio, void delete_from_swap_cache(struct folio *folio); void clear_shadow_from_swap_cache(int type, unsigned long begin, unsigned long end); +struct folio *swap_cache_get_folio(swp_entry_t entry, + struct vm_area_struct *vma, unsigned long addr); struct page *lookup_swap_cache(swp_entry_t entry, struct vm_area_struct *vma, unsigned long addr); diff --git a/mm/swap_state.c b/mm/swap_state.c index a7e0438902dd..b96bf4ec8b5b 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -317,24 +317,24 @@ static inline bool swap_use_vma_readahead(void) } /* - * Lookup a swap entry in the swap cache. A found page will be returned + * Lookup a swap entry in the swap cache. A found folio will be returned * unlocked and with its refcount incremented - we rely on the kernel - * lock getting page table operations atomic even if we drop the page + * lock getting page table operations atomic even if we drop the folio * lock before returning. */ -struct page *lookup_swap_cache(swp_entry_t entry, struct vm_area_struct *vma, - unsigned long addr) +struct folio *swap_cache_get_folio(swp_entry_t entry, + struct vm_area_struct *vma, unsigned long addr) { - struct page *page; + struct folio *folio; struct swap_info_struct *si; si = get_swap_device(entry); if (!si) return NULL; - page = find_get_page(swap_address_space(entry), swp_offset(entry)); + folio = filemap_get_folio(swap_address_space(entry), swp_offset(entry)); put_swap_device(si); - if (page) { + if (folio) { bool vma_ra = swap_use_vma_readahead(); bool readahead; @@ -342,10 +342,10 @@ struct page *lookup_swap_cache(swp_entry_t entry, struct vm_area_struct *vma, * At the moment, we don't support PG_readahead for anon THP * so let's bail out rather than confusing the readahead stat. */ - if (unlikely(PageTransCompound(page))) - return page; + if (unlikely(folio_test_large(folio))) + return folio; - readahead = TestClearPageReadahead(page); + readahead = folio_test_clear_readahead(folio); if (vma && vma_ra) { unsigned long ra_val; int win, hits; @@ -366,7 +366,17 @@ struct page *lookup_swap_cache(swp_entry_t entry, struct vm_area_struct *vma, } } - return page; + return folio; +} + +struct page *lookup_swap_cache(swp_entry_t entry, struct vm_area_struct *vma, + unsigned long addr) +{ + struct folio *folio = swap_cache_get_folio(entry, vma, addr); + + if (!folio) + return NULL; + return folio_file_page(folio, swp_offset(entry)); } /** From patchwork Mon Aug 8 19:33:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939085 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72583C25B08 for ; Mon, 8 Aug 2022 19:36:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C32C940026; Mon, 8 Aug 2022 15:35:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2FA6F940025; Mon, 8 Aug 2022 15:35:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 06613940026; Mon, 8 Aug 2022 15:35:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D6EF4940023 for ; Mon, 8 Aug 2022 15:35:54 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B6794120A27 for ; Mon, 8 Aug 2022 19:35:54 +0000 (UTC) X-FDA: 79777430628.27.2CA6DDC Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP id 4DA4E100168 for ; Mon, 8 Aug 2022 19:35:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=G5e4Fs2y2NAeyNJ3BltO4rVtKhFKBhR+x5bxL9J9noc=; b=Qx9lhni4ZwF5wwGzxh0ge9aUxQ SJDLyA8wPmxPcFyTpFrJa+LOaBPTH7rBX0uyV+9qsO8RqqWEwklSnBIXLCnb0sHPJXUTHgrexVYca m3ZaxBCQthq39B0xWZnas2YZufVWCaAudqPCx/ATWhOxflch1WwEipLR45S07UnZIAM9MhEdADU5J +b2nk0hvFgqHSCEH+1fhBZ3krd0VVpd35W0NtF2crW9Jj5jk8fYZEGV55uAzpFlcENtwrD7X64L4s /fjKbcdrsSgsa0ZSHjCn/jkHnDr+ueWxxD8OdbSp7UL+6P+3U/ZGe90bXHOsTMzQ87qsDVqq5KP6Q GdRBzetA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8Wz-00EAwk-Qs; Mon, 08 Aug 2022 19:35:20 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 23/59] shmem: Eliminate struct page from shmem_swapin_folio() Date: Mon, 8 Aug 2022 20:33:51 +0100 Message-Id: <20220808193430.3378317-24-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987354; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=G5e4Fs2y2NAeyNJ3BltO4rVtKhFKBhR+x5bxL9J9noc=; b=Cd70iaSKPKM5cBAXQYRE+Vk3OcLgyrPAFkTWDN0WZjZ57bZoruHQNkrTDMnxOGINvA6hmh u4rZiEqA5e42KyMi8uc96Ru5sDGBR2oDtawIN9xG3XX2rtN8ILhQNZQtA4W4hJALJBuvPn 7G2n9ivqG/kr1UGyZkqZsPTVHUx31BI= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Qx9lhni4; dmarc=none; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987354; a=rsa-sha256; cv=none; b=EdC7magHvQj9837gAIV9plB47xPsPP4Ync2Bncrz5ZaTmkkey47PEZKAGUiYtlNWbfCWJi J6a6Yn9FBj+3x2TsC/0sO35LSRECr0wSWS4NlAyprtlpKMF0NHyfiEMNxzQEVrw74lN+T3 sV2eeUrdkUgP7kC3g7m0/Hy9lhW/hpg= X-Rspamd-Queue-Id: 4DA4E100168 X-Rspamd-Server: rspam03 X-Rspam-User: Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Qx9lhni4; dmarc=none; spf=none (imf05.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Stat-Signature: supawkjcmoxm4rmyhj8hoao9bekrn947 X-HE-Tag: 1659987354-220022 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert shmem_swapin() to return a folio and use swap_cache_get_folio(), removing all uses of struct page in this function. Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index ac2b2ebfc9c9..a83b49a6e1d8 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1486,7 +1486,7 @@ static void shmem_pseudo_vma_destroy(struct vm_area_struct *vma) mpol_cond_put(vma->vm_policy); } -static struct page *shmem_swapin(swp_entry_t swap, gfp_t gfp, +static struct folio *shmem_swapin(swp_entry_t swap, gfp_t gfp, struct shmem_inode_info *info, pgoff_t index) { struct vm_area_struct pvma; @@ -1499,7 +1499,9 @@ static struct page *shmem_swapin(swp_entry_t swap, gfp_t gfp, page = swap_cluster_readahead(swap, gfp, &vmf); shmem_pseudo_vma_destroy(&pvma); - return page; + if (!page) + return NULL; + return page_folio(page); } /* @@ -1719,7 +1721,6 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, struct address_space *mapping = inode->i_mapping; struct shmem_inode_info *info = SHMEM_I(inode); struct mm_struct *charge_mm = vma ? vma->vm_mm : NULL; - struct page *page; struct folio *folio = NULL; swp_entry_t swap; int error; @@ -1732,8 +1733,8 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, return -EIO; /* Look it up and read it in.. */ - page = lookup_swap_cache(swap, NULL, 0); - if (!page) { + folio = swap_cache_get_folio(swap, NULL, 0); + if (!folio) { /* Or update major stats only when swapin succeeds?? */ if (fault_type) { *fault_type |= VM_FAULT_MAJOR; @@ -1741,13 +1742,12 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, count_memcg_event_mm(charge_mm, PGMAJFAULT); } /* Here we actually start the io */ - page = shmem_swapin(swap, gfp, info, index); - if (!page) { + folio = shmem_swapin(swap, gfp, info, index); + if (!folio) { error = -ENOMEM; goto failed; } } - folio = page_folio(page); /* We have to do this with folio locked to prevent races */ folio_lock(folio); From patchwork Mon Aug 8 19:33:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939060 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90BD1C00140 for ; Mon, 8 Aug 2022 19:35:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 435D9940014; Mon, 8 Aug 2022 15:35:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3BF9F940007; Mon, 8 Aug 2022 15:35:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 21747940014; Mon, 8 Aug 2022 15:35:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D6BA3940007 for ; Mon, 8 Aug 2022 15:35:22 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B9293160151 for ; Mon, 8 Aug 2022 19:35:22 +0000 (UTC) X-FDA: 79777429284.07.95AC2AB Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf06.hostedemail.com (Postfix) with ESMTP id 246F6180144 for ; Mon, 8 Aug 2022 19:35:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=24iH33ADTkEgyrdj6fTvSCpcUmC+dSo2ObVv3bR1zOA=; b=Lp6/lTP+0DOTqcBx+CoyMwx5c0 3DglMmmMs3RDAY5hfVp8x/PwLaupBFULiV0sXaFR6/MR2N0SIkr7/x8QATA+Ra49WXHKbnp7lMcft WzejgA2kRcRU/Aa0CetLjEtDO7GDe7Ovt/g4SUTbiNJfCvvVBZig3EMbzrza6SGS50kvwnFyjXPad DqQD9QIW8SOcwpVMiWvfVIp9sSPu4oNcnzXN0/yiS8/svz6pMxfAiK5S+qFPiUAlvTK4rBMabGd0X gEU7WvMa3Dl07vVXLc8AEa/2kngW9kvTm2GWOYLISgSBZMbXTUnzdAk9HKVwTk1PAuu0/8YP6cFEe 2v8GTPFg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8X2-00EAx3-Lb; Mon, 08 Aug 2022 19:35:20 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 24/59] shmem: Convert shmem_getpage_gfp() to shmem_get_folio_gfp() Date: Mon, 8 Aug 2022 20:33:52 +0100 Message-Id: <20220808193430.3378317-25-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="Lp6/lTP+"; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987322; a=rsa-sha256; cv=none; b=mGyWOsYJyT/vkMH2xva49cj27pB3wWG+M7h+RyWjFZp8ilkPoFy4AN+EG7AyktBvq13RFm V8VMgGYkT0Xh57JS08bgs3k4R6LlaYOC3Ajxuklom0Rp9RXUGzoVm1qjhGsztSH+qCUtlI +OSGzyHPWJLRnC+sQfalHhD2rsN6mCU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987322; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=24iH33ADTkEgyrdj6fTvSCpcUmC+dSo2ObVv3bR1zOA=; b=UW1Ac2roAfJIznFgoo9zs1FzhC/TZwzdwhywSYGuaJ7/Lq+MUddbCBE7gMO8Nwvuteq0lR 5sKAwQHke+B0AmHtX45PYxkMBUWBQU4wBCBURltZYH+JUWjc1Atu597nR0+ollKS7L8iim wT1YG3F67RYWTT+Veptgt+lC9awQhRU= X-Stat-Signature: nzdgfakb3faaikonwhktew1kuguce19i X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 246F6180144 Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="Lp6/lTP+"; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1659987321-880564 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a shmem_getpage_gfp() wrapper for compatibility with current users. Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 70 ++++++++++++++++++++++++++++++++---------------------- 1 file changed, 41 insertions(+), 29 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index a83b49a6e1d8..e41214dcb137 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -139,17 +139,6 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, struct folio **foliop, enum sgp_type sgp, gfp_t gfp, struct vm_area_struct *vma, vm_fault_t *fault_type); -static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, - struct page **pagep, enum sgp_type sgp, - gfp_t gfp, struct vm_area_struct *vma, - struct vm_fault *vmf, vm_fault_t *fault_type); - -int shmem_getpage(struct inode *inode, pgoff_t index, - struct page **pagep, enum sgp_type sgp) -{ - return shmem_getpage_gfp(inode, index, pagep, sgp, - mapping_gfp_mask(inode->i_mapping), NULL, NULL, NULL); -} static inline struct shmem_sb_info *SHMEM_SB(struct super_block *sb) { @@ -1595,7 +1584,7 @@ static struct folio *shmem_alloc_and_acct_folio(gfp_t gfp, struct inode *inode, /* * When a page is moved from swapcache to shmem filecache (either by the - * usual swapin of shmem_getpage_gfp(), or by the less common swapoff of + * usual swapin of shmem_get_folio_gfp(), or by the less common swapoff of * shmem_unuse_inode()), it may have been read in earlier from swap, in * ignorance of the mapping it belongs to. If that mapping has special * constraints (like the gma500 GEM driver, which requires RAM below 4GB), @@ -1810,7 +1799,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, } /* - * shmem_getpage_gfp - find page in cache, or get from swap, or allocate + * shmem_get_folio_gfp - find page in cache, or get from swap, or allocate * * If we allocate a new one we do not mark it dirty. That's up to the * vm. If we swap it in we mark it dirty since we also free the swap @@ -1819,10 +1808,10 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, * vma, vmf, and fault_type are only supplied by shmem_fault: * otherwise they are NULL. */ -static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, - struct page **pagep, enum sgp_type sgp, gfp_t gfp, - struct vm_area_struct *vma, struct vm_fault *vmf, - vm_fault_t *fault_type) +static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, + struct folio **foliop, enum sgp_type sgp, gfp_t gfp, + struct vm_area_struct *vma, struct vm_fault *vmf, + vm_fault_t *fault_type) { struct address_space *mapping = inode->i_mapping; struct shmem_inode_info *info = SHMEM_I(inode); @@ -1862,7 +1851,7 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, if (error == -EEXIST) goto repeat; - *pagep = &folio->page; + *foliop = folio; return error; } @@ -1872,7 +1861,7 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, folio_mark_accessed(folio); if (folio_test_uptodate(folio)) goto out; - /* fallocated page */ + /* fallocated folio */ if (sgp != SGP_READ) goto clear; folio_unlock(folio); @@ -1880,10 +1869,10 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, } /* - * SGP_READ: succeed on hole, with NULL page, letting caller zero. - * SGP_NOALLOC: fail on hole, with NULL page, letting caller fail. + * SGP_READ: succeed on hole, with NULL folio, letting caller zero. + * SGP_NOALLOC: fail on hole, with NULL folio, letting caller fail. */ - *pagep = NULL; + *foliop = NULL; if (sgp == SGP_READ) return 0; if (sgp == SGP_NOALLOC) @@ -1916,7 +1905,7 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, if (error != -ENOSPC) goto unlock; /* - * Try to reclaim some space by splitting a huge page + * Try to reclaim some space by splitting a large folio * beyond i_size on the filesystem. */ while (retry--) { @@ -1952,9 +1941,9 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, if (folio_test_pmd_mappable(folio) && DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE) < - hindex + HPAGE_PMD_NR - 1) { + folio_next_index(folio) - 1) { /* - * Part of the huge page is beyond i_size: subject + * Part of the large folio is beyond i_size: subject * to shrink under memory pressure. */ spin_lock(&sbinfo->shrinklist_lock); @@ -1971,14 +1960,14 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, } /* - * Let SGP_FALLOC use the SGP_WRITE optimization on a new page. + * Let SGP_FALLOC use the SGP_WRITE optimization on a new folio. */ if (sgp == SGP_FALLOC) sgp = SGP_WRITE; clear: /* - * Let SGP_WRITE caller clear ends if write does not fill page; - * but SGP_FALLOC on a page fallocated earlier must initialize + * Let SGP_WRITE caller clear ends if write does not fill folio; + * but SGP_FALLOC on a folio fallocated earlier must initialize * it now, lest undo on failure cancel our earlier guarantee. */ if (sgp != SGP_WRITE && !folio_test_uptodate(folio)) { @@ -2004,7 +1993,7 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, goto unlock; } out: - *pagep = folio_page(folio, index - hindex); + *foliop = folio; return 0; /* @@ -2034,6 +2023,29 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, return error; } +static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, + struct page **pagep, enum sgp_type sgp, + gfp_t gfp, struct vm_area_struct *vma, + struct vm_fault *vmf, vm_fault_t *fault_type) +{ + struct folio *folio = NULL; + int ret = shmem_get_folio_gfp(inode, index, &folio, sgp, gfp, vma, + vmf, fault_type); + + if (folio) + *pagep = folio_file_page(folio, index); + else + *pagep = NULL; + return ret; +} + +int shmem_getpage(struct inode *inode, pgoff_t index, + struct page **pagep, enum sgp_type sgp) +{ + return shmem_getpage_gfp(inode, index, pagep, sgp, + mapping_gfp_mask(inode->i_mapping), NULL, NULL, NULL); +} + /* * This is like autoremove_wake_function, but it removes the wait queue * entry unconditionally - even if something else had already woken the From patchwork Mon Aug 8 19:33:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939061 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAD8BC25B08 for ; Mon, 8 Aug 2022 19:35:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D5EBB940015; Mon, 8 Aug 2022 15:35:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CEE39940016; Mon, 8 Aug 2022 15:35:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A2B9C940015; Mon, 8 Aug 2022 15:35:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 868F0940007 for ; Mon, 8 Aug 2022 15:35:25 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 6393AC019A for ; Mon, 8 Aug 2022 19:35:25 +0000 (UTC) X-FDA: 79777429410.10.8175288 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf23.hostedemail.com (Postfix) with ESMTP id 01C0D140035 for ; Mon, 8 Aug 2022 19:35:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=4/bBVrQwW9UQGE0RAa+OZQItuDYpAzdCXqP4/yS9MfU=; b=bIH93aETHmHh/KLf0RkLP3J0Yz ZpMuvKXgXe6vHBrKnopUl3n/bZJsk0z/TntoiVnUkp7vjX9iuTad2ChAq0zR7v1kYPQAoazXFSiN9 rRgNtEpc/1TU6QbjdEiODEGgUZa1+ctR5Vuiz9MjGF7h0NoWTx+OwDFCmN/MqHttLsw842FfjOiMG UFNjYWCoU+TEbUtzH+ufZXKFE0T/7gOpQhWRJxDuelxfcXQ4ZvE7lz5LvaZFOL9pVfRobQlzIgIbb BubpyJjIKCm0yTkRKHTGAoBADXAZ53e9q1f9KJwZQt9R7A9ZLYWbvInnuC96pA6oqqTkgeLwFWr1Q 0OK/tmxg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8X2-00EAx8-TV; Mon, 08 Aug 2022 19:35:20 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 25/59] shmem: Convert shmem_fault() to use shmem_get_folio_gfp() Date: Mon, 8 Aug 2022 20:33:53 +0100 Message-Id: <20220808193430.3378317-26-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987325; a=rsa-sha256; cv=none; b=237L/bDNwXUW7x8aLHBe4HjOTMyvtYmTTUrttfgS/1KSKSjvlfrYfkiQ1H1hWJbZRWf5/d EtJO23QtXxzzVPa/RKPJPBBZ0FZGlkXPSdtyAYaUQluQ1XuKb2Kidvx7pOT5pOEwgQpl4/ IIZ7XoB4tClny78wzQkhq6RDUPlAmXg= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=bIH93aET; dmarc=none; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987325; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4/bBVrQwW9UQGE0RAa+OZQItuDYpAzdCXqP4/yS9MfU=; b=DMyQrd0LUSDisnnlSUafdkzA72DjbmczCGFYGUkzQiJKM32TDT2ozY1Va43RMtWDVjfHnt nvqKYu56fk0Egu7geSJmFNr/LYuhqsZgLhNztGG2uNKwWkaA6p44+F8CmoLr2eLIHV8GGF u7kEiwSi3SNMew6L3dKPZzseu5Jo64g= X-Rspamd-Server: rspam10 X-Stat-Signature: cycgojo5ergpyudj345octdragcf6hr6 Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=bIH93aET; dmarc=none; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Rspamd-Queue-Id: 01C0D140035 X-HE-Tag: 1659987324-928346 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: No particular advantage for this function, but necessary to remove shmem_getpage_gfp(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/mm/shmem.c b/mm/shmem.c index e41214dcb137..f9654008950e 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2063,6 +2063,7 @@ static vm_fault_t shmem_fault(struct vm_fault *vmf) struct vm_area_struct *vma = vmf->vma; struct inode *inode = file_inode(vma->vm_file); gfp_t gfp = mapping_gfp_mask(inode->i_mapping); + struct folio *folio; int err; vm_fault_t ret = VM_FAULT_LOCKED; @@ -2125,10 +2126,11 @@ static vm_fault_t shmem_fault(struct vm_fault *vmf) spin_unlock(&inode->i_lock); } - err = shmem_getpage_gfp(inode, vmf->pgoff, &vmf->page, SGP_CACHE, + err = shmem_get_folio_gfp(inode, vmf->pgoff, &folio, SGP_CACHE, gfp, vma, vmf, &ret); if (err) return vmf_error(err); + vmf->page = folio_file_page(folio, vmf->pgoff); return ret; } From patchwork Mon Aug 8 19:33:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939062 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 129A2C00140 for ; Mon, 8 Aug 2022 19:35:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 21B5A940016; Mon, 8 Aug 2022 15:35:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 15422940007; Mon, 8 Aug 2022 15:35:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E2458940017; Mon, 8 Aug 2022 15:35:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id BB742940007 for ; Mon, 8 Aug 2022 15:35:25 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 9AA8916017C for ; Mon, 8 Aug 2022 19:35:25 +0000 (UTC) X-FDA: 79777429410.16.CC8BBF4 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf10.hostedemail.com (Postfix) with ESMTP id 4886EC002C for ; Mon, 8 Aug 2022 19:35:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=q8gnnWajV66diQ2wfjTJSfiMq1VpYbZiFpN3K+ei4MU=; b=m8F9zBYlm3VQCVcTs85/wO2tcR RLayHQm28n7zDH3+Zm08FxrzFNy8OKF3iwz5W1HEIlOKMEywmd8S96PQHIG1eq97fr4b1TfuceXxE V+DHq26gM/PZQrWOMWiQUPbk6L5HDmQfP3gyjssKIRcHuuqaS9DTcc1Ncapo4y89oIXdgL8Gm8QSg nrR+NQwizoV5HY0nIW2lQHnPRnxD9WnOL1kQ5jsJZbmDgivLHztOBSFl7H0c78cCo29/QgIBt/TNi Jus6TS+EGNN28sj9v1q6wBRNzrzBr3NvaFeB8uT9Z0kNaVAJGnnrbWYW1yt52plAHi06qCYrnFTgH sqNv4SjQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8X3-00EAxE-2q; Mon, 08 Aug 2022 19:35:23 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 26/59] shmem: Convert shmem_read_mapping_page_gfp() to use shmem_get_folio_gfp() Date: Mon, 8 Aug 2022 20:33:54 +0100 Message-Id: <20220808193430.3378317-27-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987325; a=rsa-sha256; cv=none; b=TXInvlRBjoX4ZnRSIWuiOaH4R2oT0uHl6fQwP5T5+0dLhVO8KdrkthBiYSkwd5kn3tonSp mOSrbXq4zg8u8ivprwsIPZzK+ATgbu0TPp9AlqGiLJJ22LJFHkEkfWFDi8TmLLM3g+E1Rl edZECebaLNITX3+FSt0mNW1XaXpMw1Y= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=m8F9zBYl; spf=none (imf10.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987325; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=q8gnnWajV66diQ2wfjTJSfiMq1VpYbZiFpN3K+ei4MU=; b=zUsPP1VYPtrAT0JsTuBLDg87aPUUmONlONeHBrotK2PSjf8tTs/tAu7Ahnkapbe56rdgZZ N6AKx1rogECsWI3gOGFjj5FkYkENvAk+78Sgj+gpLJwj7TLGeBhQVMkAuA8odhhrjQMLhV WsRxEepDhPg84PgdJ1/kFHNfOpQ2la4= X-Rspamd-Queue-Id: 4886EC002C Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=m8F9zBYl; spf=none (imf10.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: qptriodeaw1cnck8ueam7mugt8d7f746 X-HE-Tag: 1659987325-46902 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Saves a couple of calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index f9654008950e..eaaadb7d31b7 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -4258,18 +4258,20 @@ struct page *shmem_read_mapping_page_gfp(struct address_space *mapping, { #ifdef CONFIG_SHMEM struct inode *inode = mapping->host; + struct folio *folio; struct page *page; int error; BUG_ON(!shmem_mapping(mapping)); - error = shmem_getpage_gfp(inode, index, &page, SGP_CACHE, + error = shmem_get_folio_gfp(inode, index, &folio, SGP_CACHE, gfp, NULL, NULL, NULL); if (error) return ERR_PTR(error); - unlock_page(page); + folio_unlock(folio); + page = folio_file_page(folio, index); if (PageHWPoison(page)) { - put_page(page); + folio_put(folio); return ERR_PTR(-EIO); } From patchwork Mon Aug 8 19:33:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939063 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B896C25B08 for ; Mon, 8 Aug 2022 19:35:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9B4F8940007; Mon, 8 Aug 2022 15:35:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 93CEF940017; Mon, 8 Aug 2022 15:35:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 74640940007; Mon, 8 Aug 2022 15:35:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 29066940017 for ; Mon, 8 Aug 2022 15:35:26 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 0C30D120133 for ; Mon, 8 Aug 2022 19:35:26 +0000 (UTC) X-FDA: 79777429452.13.BDD1A7A Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf08.hostedemail.com (Postfix) with ESMTP id 951E2160172 for ; Mon, 8 Aug 2022 19:35:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=gaYJc1Y+x++0gmr0tEmhrs1/PndL40dj2HR9/lZk7rM=; b=MBKrfPSbqEeh9a7k7UViMg+8DY okNJ+VS4qfOSJoof73tp1d9oHZWYj11O89S1QEqqlsMJfGhl12PUNwFRSFasKUIbaj2lPig58oh+/ nWKMgf1nPUPXd4qOt4GzHDRpXp8kW5g4r7E4kW9wOa1n70AYOH0NdH4+L5aOO7PSyGz5AwZy1Mehg N1pJKX3L5cmDm5B5ekzH9pOOslxNpBAWF35vE1+B6RrNoqOH+tjz8r59a3Bl7aV9Ftq+2G9hYKRtz wt8hIuM9LyAugLtK1IK0dnRiSz4U+3wSpZ72nBKkirmK2a1YSRQ0vKoD8cM0AE0Y/EqehGBW1APu/ u4HWGjzA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8X5-00EAxT-VG; Mon, 08 Aug 2022 19:35:24 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 27/59] shmem: Add shmem_get_folio() Date: Mon, 8 Aug 2022 20:33:55 +0100 Message-Id: <20220808193430.3378317-28-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987325; a=rsa-sha256; cv=none; b=COA//NbdyMVSH0e1+JfCTiU07oPmmkUfZ57jdNm2/I6y0ByWVZs8IYsL0dAg10M8kaa2GJ N/fd82xMXx9FHwsZ04XU+aHcexjdC7mWadY+EX5LU4jA42jvkjI+FFptX4i7CneRjX/eYM Sv6wzgoS2r63ktE+5IzMk4XpRfubDEw= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=MBKrfPSb; spf=none (imf08.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987325; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gaYJc1Y+x++0gmr0tEmhrs1/PndL40dj2HR9/lZk7rM=; b=DMc6fsUtcfO6vXziADYLBf3eUkrbBddXSAQvlF1HwgDPlBe6Vmvj3jiR0VlMJmXhGAYI5m MU6XpVsrNI5npdBQLTR9wAQ5APFjeCblpU8Tw/kPpxPD5QhxHPdFCgifGQJ4im7tl11IMZ hiavOgoG0uwBPs6Fm12B10H00kxhiUg= X-Rspamd-Queue-Id: 951E2160172 X-Rspam-User: X-Rspamd-Server: rspam11 Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=MBKrfPSb; spf=none (imf08.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Stat-Signature: txyjhzpn5mp18o448njksxa8m4d8htrn X-HE-Tag: 1659987325-940087 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With no remaining callers of shmem_getpage_gfp(), add shmem_get_folio() and reimplement shmem_getpage() as a call to shmem_get_folio(). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/shmem_fs.h | 2 ++ mm/shmem.c | 23 ++++++++++------------- 2 files changed, 12 insertions(+), 13 deletions(-) diff --git a/include/linux/shmem_fs.h b/include/linux/shmem_fs.h index 1b6c4013f691..0549929261f2 100644 --- a/include/linux/shmem_fs.h +++ b/include/linux/shmem_fs.h @@ -118,6 +118,8 @@ enum sgp_type { extern int shmem_getpage(struct inode *inode, pgoff_t index, struct page **pagep, enum sgp_type sgp); +int shmem_get_folio(struct inode *inode, pgoff_t index, struct folio **foliop, + enum sgp_type sgp); static inline struct page *shmem_read_mapping_page( struct address_space *mapping, pgoff_t index) diff --git a/mm/shmem.c b/mm/shmem.c index eaaadb7d31b7..75d9304b964e 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2023,14 +2023,18 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, return error; } -static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, - struct page **pagep, enum sgp_type sgp, - gfp_t gfp, struct vm_area_struct *vma, - struct vm_fault *vmf, vm_fault_t *fault_type) +int shmem_get_folio(struct inode *inode, pgoff_t index, struct folio **foliop, + enum sgp_type sgp) +{ + return shmem_get_folio_gfp(inode, index, foliop, sgp, + mapping_gfp_mask(inode->i_mapping), NULL, NULL, NULL); +} + +int shmem_getpage(struct inode *inode, pgoff_t index, + struct page **pagep, enum sgp_type sgp) { struct folio *folio = NULL; - int ret = shmem_get_folio_gfp(inode, index, &folio, sgp, gfp, vma, - vmf, fault_type); + int ret = shmem_get_folio(inode, index, &folio, sgp); if (folio) *pagep = folio_file_page(folio, index); @@ -2039,13 +2043,6 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, return ret; } -int shmem_getpage(struct inode *inode, pgoff_t index, - struct page **pagep, enum sgp_type sgp) -{ - return shmem_getpage_gfp(inode, index, pagep, sgp, - mapping_gfp_mask(inode->i_mapping), NULL, NULL, NULL); -} - /* * This is like autoremove_wake_function, but it removes the wait queue * entry unconditionally - even if something else had already woken the From patchwork Mon Aug 8 19:33:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939065 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01C12C25B08 for ; Mon, 8 Aug 2022 19:35:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1C49694001A; Mon, 8 Aug 2022 15:35:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1529C940017; Mon, 8 Aug 2022 15:35:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF9BB94001A; Mon, 8 Aug 2022 15:35:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A695D940018 for ; Mon, 8 Aug 2022 15:35:28 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 83CF61C5CB5 for ; Mon, 8 Aug 2022 19:35:28 +0000 (UTC) X-FDA: 79777429536.19.F834E5B Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf11.hostedemail.com (Postfix) with ESMTP id 1FA8840183 for ; Mon, 8 Aug 2022 19:35:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=axbdm1vZfpnkHvpQc3L6SkUUiWdMrUO7JQX8stMwqQY=; b=cyHBpYEYVFxGzFBSnqzbmQcJBm M5JGh5xBVfo1fn71yP/OMx72DQ4mGiIPIyiKJ5X0DwmNc6hxXPAe5pzFs2h4jYB1ndhcNfzhLukbB sD7gr67a0JStv5wQe8tQmGVe17tam0uKZ1E17Tv3dFPQcFneDBLCZIJ9PcPLAPG4Jcnyej7DrG1Ow Xj+s/zxUhu18rFPBxYEeK20ySglqee+OltSpel82m2Wfrq3GqQFvzSMjmAWnqzUOwJyl+qq3riUpb Eks3CQygja08Zi/ieWxhhjxY4rV17v68x6aXggzsTPQlENBJUvfLzSS3AzLR0gvmnZXUoxJSHJM8y cv6bSFGQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8X6-00EAxd-4u; Mon, 08 Aug 2022 19:35:24 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 28/59] shmem: Convert shmem_get_partial_folio() to use shmem_get_folio() Date: Mon, 8 Aug 2022 20:33:56 +0100 Message-Id: <20220808193430.3378317-29-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987328; a=rsa-sha256; cv=none; b=xh1rwx0e6TTotlRW99+VFnfwrdwYfzYy76RlAYESr6jb//6OwgZxtVEKKRzS9Tu6qLkAWm EkkM4QDcAw/FJJ4xjCtR9bkwXZR7Yvj3aQz5DH5PLaHtKCCZk9naHD2x+Oich+PEq5Nxyw W+DqPf1bMiyk53esJBTO0tM+lay85qc= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=cyHBpYEY; spf=none (imf11.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987328; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=axbdm1vZfpnkHvpQc3L6SkUUiWdMrUO7JQX8stMwqQY=; b=PnBGlNOgtspZj5OKy5PrQMKNmI0Q+ByigpaC8H04XUhpNd3TJPK9YXrMuc0AR54LuaofRf IOrTie83av/B8FSwQIBADJHIbvADXyI0izQzPJWavdXBWc8if8eZMyuMyPpX+YCKdD12wd OLJp1Ae3NQk20YNA6YInY4Nw+vrwYWQ= X-Rspamd-Server: rspam06 X-Rspam-User: X-Stat-Signature: kon3bjfrqsdqkeyibwnr6uiurz8u7hfo X-Rspamd-Queue-Id: 1FA8840183 Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=cyHBpYEY; spf=none (imf11.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1659987327-823398 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Get rid of an unnecessary folio->page->folio conversion. Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 75d9304b964e..b26ab3167040 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -874,10 +874,9 @@ void shmem_unlock_mapping(struct address_space *mapping) static struct folio *shmem_get_partial_folio(struct inode *inode, pgoff_t index) { struct folio *folio; - struct page *page; /* - * At first avoid shmem_getpage(,,,SGP_READ): that fails + * At first avoid shmem_get_folio(,,,SGP_READ): that fails * beyond i_size, and reports fallocated pages as holes. */ folio = __filemap_get_folio(inode->i_mapping, index, @@ -888,9 +887,9 @@ static struct folio *shmem_get_partial_folio(struct inode *inode, pgoff_t index) * But read a page back from swap if any of it is within i_size * (although in some cases this is just a waste of time). */ - page = NULL; - shmem_getpage(inode, index, &page, SGP_READ); - return page ? page_folio(page) : NULL; + folio = NULL; + shmem_get_folio(inode, index, &folio, SGP_READ); + return folio; } /* From patchwork Mon Aug 8 19:33:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939066 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53295C00140 for ; Mon, 8 Aug 2022 19:35:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 55C06940017; Mon, 8 Aug 2022 15:35:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 45DFB94001B; Mon, 8 Aug 2022 15:35:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F3188940018; Mon, 8 Aug 2022 15:35:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D3EAB940017 for ; Mon, 8 Aug 2022 15:35:28 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A2FA91A0BE3 for ; Mon, 8 Aug 2022 19:35:28 +0000 (UTC) X-FDA: 79777429536.09.1226299 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf21.hostedemail.com (Postfix) with ESMTP id 05DDD1C0198 for ; Mon, 8 Aug 2022 19:35:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=p/s7zN93eAE1jGt/6wsL0yiCckkaIiWrMsY7IVplJbc=; b=oWn4fTyBqdtA9Is6IiO9a7lCt4 TM968kd/J292cDXPKpFZDfodfSQoDPBtdu4vmakLudO24c1NFdCCrygpEn6++SjVpymASGGHjooQW WUCdPUOCC2QGyff0+KGhp3lNp/YyktVrw8MMELci43w0n9lxost4O6pW2fj0r0Ze/xh37z5ij0CsI 0vPbci5N5hB7GcVF1CN504EsSLlPmWZFhUdc4nxi2U6+4t8mQTlnwHa7Rqk7xADOzdOrCHgPJtIbI atjoOaMafMfAfQXL7OMMIFui0ERQicVWAYqASXOXkVjn51KwRJMjjhSfRY1eckdGSrOfvkYxzDpZp qdi3SgoQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8X6-00EAxl-Fw; Mon, 08 Aug 2022 19:35:24 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 29/59] shmem: Convert shmem_write_begin() to use shmem_get_folio() Date: Mon, 8 Aug 2022 20:33:57 +0100 Message-Id: <20220808193430.3378317-30-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=oWn4fTyB; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987328; a=rsa-sha256; cv=none; b=g8FBlNUXu44Ll2GdSGYRkfDYZq7/OUNTspipffEdsjNohjf8x+TbunjpwycxTR6mTKQY4n VeLtXtiCqRn0RmRtAC6RWjqMU+9p9kQWi/4JwP4C+X/6VKMPNG7T/F2TpxC5G6BMN9Yead S4SpGoaQ1GtG3Xaq4PyeNrsWgahYFoc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987328; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=p/s7zN93eAE1jGt/6wsL0yiCckkaIiWrMsY7IVplJbc=; b=QF8ehIgwlatPVdK1NQwWdFMjNRzteS8v/xZYFcN2++eh0dhPQE11iYC90KciVItWdHa3Bg /m+cDl6wP6i9ISnd+S7UQKwfyc/6OKAtqhixm4OdW/DBZvCtw8OzCW9Ch85sksSYx8mw4n GV1NQtwViK3wuVdhRkRRefY3xuXOIA0= X-Stat-Signature: asipgqt79t63fkspiwa5q6bnpkhqgjay X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 05DDD1C0198 Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=oWn4fTyB; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1659987327-418696 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use a folio throughout this function, saving a couple of calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index b26ab3167040..45212001effd 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2482,6 +2482,7 @@ shmem_write_begin(struct file *file, struct address_space *mapping, struct inode *inode = mapping->host; struct shmem_inode_info *info = SHMEM_I(inode); pgoff_t index = pos >> PAGE_SHIFT; + struct folio *folio; int ret = 0; /* i_rwsem is held by caller */ @@ -2493,14 +2494,15 @@ shmem_write_begin(struct file *file, struct address_space *mapping, return -EPERM; } - ret = shmem_getpage(inode, index, pagep, SGP_WRITE); + ret = shmem_get_folio(inode, index, &folio, SGP_WRITE); if (ret) return ret; + *pagep = folio_file_page(folio, index); if (PageHWPoison(*pagep)) { - unlock_page(*pagep); - put_page(*pagep); + folio_unlock(folio); + folio_put(folio); *pagep = NULL; return -EIO; } From patchwork Mon Aug 8 19:33:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939064 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C36A6C00140 for ; Mon, 8 Aug 2022 19:35:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AA058940019; Mon, 8 Aug 2022 15:35:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D972940017; Mon, 8 Aug 2022 15:35:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8522F940018; Mon, 8 Aug 2022 15:35:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 6CD10940017 for ; Mon, 8 Aug 2022 15:35:28 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 55DF7C0121 for ; Mon, 8 Aug 2022 19:35:28 +0000 (UTC) X-FDA: 79777429536.03.58B3A95 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf16.hostedemail.com (Postfix) with ESMTP id EC7A118016B for ; Mon, 8 Aug 2022 19:35:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=bIVlghjHX5ZoI79ilxenPX7R4/CW8hGR+dhD1yFJKb4=; b=OsOiOsF8EmL+6IGFE8rriYSeQU OJd1fu4CKbwWHXf2AfReObqnRd8s9certrnAkpX1j0IFSWWefMX336Es7/hV9nNs3ROd7QfSZ753h OMnAdP6aPhzJmq8I6haBRbisQ0CLU/iLW8/XnsrTAl2ezSrYZghwKuHlqyHxHmAlxIOmwyh2fTtYX IqJZ7A4bRIwJza0umd2WsBme6QmWY3UZIPAcX9UYEb+lcl1xL+aNljTsO7dxy63CG3BF7pJfO6SfS y7f5TXstzyQbWA9GXYrkL+LykkGcHnahOZ6vq+/IFnLfeAdPE3VoLnKLzwGWl0vHhap0O01XomP7H 4ho0jK/w==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8X6-00EAxt-P2; Mon, 08 Aug 2022 19:35:24 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 30/59] shmem: Convert shmem_file_read_iter() to use shmem_get_folio() Date: Mon, 8 Aug 2022 20:33:58 +0100 Message-Id: <20220808193430.3378317-31-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987328; a=rsa-sha256; cv=none; b=ruPYL0HHnxY5GKEKQGPTKnCqesbvKAldtRxFCo4pAwi6AeXszADWZCSPhd+bxpTpauYCtP ONYLTgtF0ieXFQmw/Fh1SAMH5QWUj2mfwQY/6vN8EdqfVBlwmri02K9JjP8w2oeVVgA2T7 mYb7R1Li7QzHhwwXVtbPOEtbntjvzM4= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=OsOiOsF8; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987328; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bIVlghjHX5ZoI79ilxenPX7R4/CW8hGR+dhD1yFJKb4=; b=WafseohRYBocpiqexv4pVEyPG5kJuwjGcstEhQpaiYAvPzVYdQMJo6Azh9uhNPw4FQpe+C gYQlAceuc9rkpOTNpo1F6JYdP75L1K3qF5ojHJVsR9Qk1uTlF+BvRTsu1jzXtvCkC1wTtn InhKRthGyBYoMmIpS9hcXhQqZHj8To4= X-Rspamd-Queue-Id: EC7A118016B Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=OsOiOsF8; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: tsrg6qrsjkggkgj5fzi1athhbkpsbg8c X-HE-Tag: 1659987327-311854 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use a folio throughout, saving five calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 45212001effd..51a1934f8353 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2561,6 +2561,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) offset = *ppos & ~PAGE_MASK; for (;;) { + struct folio *folio = NULL; struct page *page = NULL; pgoff_t end_index; unsigned long nr, ret; @@ -2575,17 +2576,18 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) break; } - error = shmem_getpage(inode, index, &page, SGP_READ); + error = shmem_get_folio(inode, index, &folio, SGP_READ); if (error) { if (error == -EINVAL) error = 0; break; } - if (page) { - unlock_page(page); + if (folio) { + folio_unlock(folio); + page = folio_file_page(folio, index); if (PageHWPoison(page)) { - put_page(page); + folio_put(folio); error = -EIO; break; } @@ -2601,14 +2603,14 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) if (index == end_index) { nr = i_size & ~PAGE_MASK; if (nr <= offset) { - if (page) - put_page(page); + if (folio) + folio_put(folio); break; } } nr -= offset; - if (page) { + if (folio) { /* * If users can be writing to this page using arbitrary * virtual addresses, take care about potential aliasing @@ -2620,13 +2622,13 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) * Mark the page accessed if we read the beginning. */ if (!offset) - mark_page_accessed(page); + folio_mark_accessed(folio); /* * Ok, we have the page, and it's up-to-date, so * now we can copy it to user space... */ ret = copy_page_to_iter(page, offset, nr, to); - put_page(page); + folio_put(folio); } else if (iter_is_iovec(to)) { /* From patchwork Mon Aug 8 19:33:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939089 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82563C25B0C for ; Mon, 8 Aug 2022 19:36:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 56F888E0003; Mon, 8 Aug 2022 15:36:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4814994000C; Mon, 8 Aug 2022 15:36:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2AD2D8E000C; Mon, 8 Aug 2022 15:36:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 106AB8E0003 for ; Mon, 8 Aug 2022 15:36:02 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id E81BAC0126 for ; Mon, 8 Aug 2022 19:36:01 +0000 (UTC) X-FDA: 79777430922.02.DD5AEF2 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf10.hostedemail.com (Postfix) with ESMTP id 9E08EC015E for ; Mon, 8 Aug 2022 19:36:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=D77zS7INJVOvYz/CP+vtDGyyfpLH7LZUg0F9gzXZvLY=; b=s7rZApVf2Kbm/Lnu0ELGxlHGmt RXHe5vgdsozolMWSknEbAmtJhbUndGxycNtgBYiL2s8PY6UvR2YlxTCTCuCOWdxdjbE7c+dZhXA6p UApJEsNhewFlNfBBNOZBC33TiYPWFeRuREG40GqM/3YU3OoQs6CU5H9eZQfLe8UOWx9IUAst44J4z 3NbYURaGxSqxQO73CFeG6IwzcS5O+t2FpQd1LaIYl3Gkwjxlzc7TVtpBU+mxymHGCKVjOPDvJWgmY Y2J1QPKOgv75xtjPFUa5xli3vwIWyZ7cIc4W4fIeG/NecPY15tL4oqPp1Ruaw+x2uUHLOmrdJFQJW QRrpVpBw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8X6-00EAy2-U5; Mon, 08 Aug 2022 19:35:25 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 31/59] shmem: Convert shmem_fallocate() to use a folio Date: Mon, 8 Aug 2022 20:33:59 +0100 Message-Id: <20220808193430.3378317-32-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=s7rZApVf; spf=none (imf10.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987361; a=rsa-sha256; cv=none; b=WhINbTeMG+tGQ8xOMe0pgjdE9jVhSMP/K7WGvTKH4dZ26QboXBahltOSam74zhQBQPOjYk kizdyvfRHozfJtqc1TKTUUUKBRg0zkDcxzDMeHr8jKiuE58wL6cDr71fZG++AarwiGhoUj ATR0vuvu4QSYg0sQQ5QqJOuckyCq1ug= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987361; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=D77zS7INJVOvYz/CP+vtDGyyfpLH7LZUg0F9gzXZvLY=; b=OaYlcdc2ivvDpB7Qg/8rky+enU8CbJ2VLA1p+tY3Yq+UlFAcEGz3a4TMc95MiyZgaoWIHT tnKzCUGeFp1OPWwOCaIUoTywyTRL95Lc0IMm5qMm0zk4NaXlesVSs29NjveVdaS99HJOyt 5eb8BxIHd23JhyFjdRhJ1Ba1aLdbUeU= X-Stat-Signature: fwpfi4d5wseo6waana7xf794tk3wfz5t X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 9E08EC015E Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=s7rZApVf; spf=none (imf10.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1659987361-234266 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Call shmem_get_folio() and use the folio APIs instead of the page APIs. Saves several calls to compound_head() and removes assumptions about the size of a large folio. Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 36 +++++++++++++++++------------------- 1 file changed, 17 insertions(+), 19 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 51a1934f8353..fda56776b510 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2771,7 +2771,7 @@ static long shmem_fallocate(struct file *file, int mode, loff_t offset, info->fallocend = end; for (index = start; index < end; ) { - struct page *page; + struct folio *folio; /* * Good, the fallocate(2) manpage permits EINTR: we may have @@ -2782,10 +2782,11 @@ static long shmem_fallocate(struct file *file, int mode, loff_t offset, else if (shmem_falloc.nr_unswapped > shmem_falloc.nr_falloced) error = -ENOMEM; else - error = shmem_getpage(inode, index, &page, SGP_FALLOC); + error = shmem_get_folio(inode, index, &folio, + SGP_FALLOC); if (error) { info->fallocend = undo_fallocend; - /* Remove the !PageUptodate pages we added */ + /* Remove the !uptodate folios we added */ if (index > start) { shmem_undo_range(inode, (loff_t)start << PAGE_SHIFT, @@ -2794,37 +2795,34 @@ static long shmem_fallocate(struct file *file, int mode, loff_t offset, goto undone; } - index++; /* * Here is a more important optimization than it appears: - * a second SGP_FALLOC on the same huge page will clear it, - * making it PageUptodate and un-undoable if we fail later. + * a second SGP_FALLOC on the same large folio will clear it, + * making it uptodate and un-undoable if we fail later. */ - if (PageTransCompound(page)) { - index = round_up(index, HPAGE_PMD_NR); - /* Beware 32-bit wraparound */ - if (!index) - index--; - } + index = folio_next_index(folio); + /* Beware 32-bit wraparound */ + if (!index) + index--; /* * Inform shmem_writepage() how far we have reached. * No need for lock or barrier: we have the page lock. */ - if (!PageUptodate(page)) + if (!folio_test_uptodate(folio)) shmem_falloc.nr_falloced += index - shmem_falloc.next; shmem_falloc.next = index; /* - * If !PageUptodate, leave it that way so that freeable pages + * If !uptodate, leave it that way so that freeable folios * can be recognized if we need to rollback on error later. - * But set_page_dirty so that memory pressure will swap rather - * than free the pages we are allocating (and SGP_CACHE pages + * But mark it dirty so that memory pressure will swap rather + * than free the folios we are allocating (and SGP_CACHE folios * might still be clean: we now need to mark those dirty too). */ - set_page_dirty(page); - unlock_page(page); - put_page(page); + folio_mark_dirty(folio); + folio_unlock(folio); + folio_put(folio); cond_resched(); } From patchwork Mon Aug 8 19:34:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939068 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 158AAC00140 for ; Mon, 8 Aug 2022 19:35:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BCA018E0009; Mon, 8 Aug 2022 15:35:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8F5CB8E000B; Mon, 8 Aug 2022 15:35:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5E0518E0001; Mon, 8 Aug 2022 15:35:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2715A8E0009 for ; Mon, 8 Aug 2022 15:35:30 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 0BDD01A0AB5 for ; Mon, 8 Aug 2022 19:35:30 +0000 (UTC) X-FDA: 79777429620.11.67292AA Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf14.hostedemail.com (Postfix) with ESMTP id 76489100165 for ; Mon, 8 Aug 2022 19:35:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=oKWgL+GAkV2UorbTCCg0q6V99pHLaC7KYMyhuwUPzbw=; b=coB/DCXR1XSOKI087AYObYsUEn hOhujcbfUV+G8xGpbBWMP4YXPsIEuCGJXayYLUAInoVvIwxh6Ulf/wfGZTZ5Jpzj6lbzqrN+89qz5 IIVm13B2bmVqLedYANLO4OfihvSukVtFOgwdcUTBkQQVOeNIQX6kbIM52pp2S35P+HfHSZ85hUNCx tPG33uJZpokZ+2abY7NdGbqgwLApskKABEbxPLs387gP69zoJpCAUY34AtXkO1f0+IDbWgDBgn7vK yD0L1tRVmxRmKMRwPJFca1fFRkf7lpBy5taihFVP0mxSWjcE94egKwRaL/fuwp2eSqzholmk+h1qc 5obMnTOQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8X9-00EAyB-Ly; Mon, 08 Aug 2022 19:35:27 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 32/59] shmem: Convert shmem_symlink() to use a folio Date: Mon, 8 Aug 2022 20:34:00 +0100 Message-Id: <20220808193430.3378317-33-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="coB/DCXR"; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987329; a=rsa-sha256; cv=none; b=v2+e8FI7m2Fruonj4TZ+rSuDXnMQXhSV92EC/cIzoDVgAo700w1arBhCHxe+idKFTUAyK1 I3J3sxpdxpKoxKv/oTzKsHj0XVs++m4BbROwWhz3KUQ5GfbzHLPDmH13/iIPf11xCT1OtO HoHSkczCF3+/IrIrqny7pPIuiaz9/mU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987329; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oKWgL+GAkV2UorbTCCg0q6V99pHLaC7KYMyhuwUPzbw=; b=UBFMUMPpT15x5sMGqqVMEH1GU0IulaO/Q6AnNszvzhm2sfmFm/Z+PgYlG+yZsmZyM5o88m bXQ8L2BPeNWGVA+Gv8knzvWpwYTuGUg0O72WXVQ2saNzgfhojVVo0luUGhp2oBzaulOaEN LRsTrK/bXL6xJR6KxLQ3BibEn6ouNfg= X-Stat-Signature: 1tc1dt6gokx6ugufia3fwpizde9h1tcq X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 76489100165 Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="coB/DCXR"; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1659987329-343873 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: While symlinks will always be < PAGE_SIZE, using the folio APIs gets rid of unnecessary calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index fda56776b510..9af2292e257b 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -3076,7 +3076,7 @@ static int shmem_symlink(struct user_namespace *mnt_userns, struct inode *dir, int error; int len; struct inode *inode; - struct page *page; + struct folio *folio; len = strlen(symname) + 1; if (len > PAGE_SIZE) @@ -3104,18 +3104,18 @@ static int shmem_symlink(struct user_namespace *mnt_userns, struct inode *dir, inode->i_op = &shmem_short_symlink_operations; } else { inode_nohighmem(inode); - error = shmem_getpage(inode, 0, &page, SGP_WRITE); + error = shmem_get_folio(inode, 0, &folio, SGP_WRITE); if (error) { iput(inode); return error; } inode->i_mapping->a_ops = &shmem_aops; inode->i_op = &shmem_symlink_inode_operations; - memcpy(page_address(page), symname, len); - SetPageUptodate(page); - set_page_dirty(page); - unlock_page(page); - put_page(page); + memcpy(folio_address(folio), symname, len); + folio_mark_uptodate(folio); + folio_mark_dirty(folio); + folio_unlock(folio); + folio_put(folio); } dir->i_size += BOGO_DIRENT_SIZE; dir->i_ctime = dir->i_mtime = current_time(dir); From patchwork Mon Aug 8 19:34:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939067 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CF09C25B08 for ; Mon, 8 Aug 2022 19:35:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8F5B68E000A; Mon, 8 Aug 2022 15:35:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 78C768E0009; Mon, 8 Aug 2022 15:35:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 40E928E000B; Mon, 8 Aug 2022 15:35:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1BA638E0001 for ; Mon, 8 Aug 2022 15:35:30 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id EEB29160146 for ; Mon, 8 Aug 2022 19:35:29 +0000 (UTC) X-FDA: 79777429578.23.FC7C63C Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id 90086120156 for ; Mon, 8 Aug 2022 19:35:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=epuyNg9S7J8yV/mFDJ3WzX6EqgqxvHf8/oT33SymrtM=; b=IhJwxJ4XvNBKu9fSekCU3r/aZa mEi2/zN6hud5K8Fn8WPhg+6lw2pSJCNUTBtk3h3fUHfyp2haJh+D1XT8+6tfNxAcMzN2dQXN6Hoy/ FUM7VfMcZtXu6QH18I4g/w7Zhs5mG3aSMS1RyxrWDnYM2DGI7wGF0/KAVdByelN6ntTOe6pE3muAw AVBf66zBKDkUcH11SttCl2oWWNa0z3ILc/AEBIqPs8l2ITcNH8Wu/TmPHos1NFzjRDe7RdM9yC8X4 O41TX+w7qdYTv98c1YvWFH8KDtpv8XkDweeiMchONjeSke3PFtPUJDUeIFwKj4RAQZMvM4unmlERq 3MRWMBDw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XA-00EAyM-2Q; Mon, 08 Aug 2022 19:35:28 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 33/59] shmem: Convert shmem_get_link() to use a folio Date: Mon, 8 Aug 2022 20:34:01 +0100 Message-Id: <20220808193430.3378317-34-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987329; a=rsa-sha256; cv=none; b=Vh1m2IDFn7mUiPnPQGjGGIla5dbJwd0Fo33lHPgXhZTAE53oTvainKZzNkiL4gvoGn0zT5 KbY8TRI+SBCshI5lMTcyHpo8shVYhGMvNZ/eXkk+uyOHDSKeTZetrQ8FS7pHcgRjX0RFtL VptCUY54/nEVIgqwQmzK8UwYx6bgg30= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=IhJwxJ4X; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987329; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=epuyNg9S7J8yV/mFDJ3WzX6EqgqxvHf8/oT33SymrtM=; b=nUkmYwY8zk8Bd2s0HAAuptGqEsFUIkWjvasJ2iMq43tL4GI1wS9qtOvA1ArS0mL80xiHhi ANcb5qZ/65d7py8vxdP8agNSuMM9kfCytSGcuGlyCvrnV7ub1C35oX34zharcSUz0wyAxp Fv4kT/MxpPBchTdbfLi/uej/fLTi1R8= X-Rspamd-Queue-Id: 90086120156 X-Rspam-User: X-Rspamd-Server: rspam11 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=IhJwxJ4X; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Stat-Signature: 9418fsrph1mxre6noxwwqbmfb3su6hpu X-HE-Tag: 1659987329-942884 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Symlinks will never use a large folio, but using the folio API removes a lot of unnecessary folio->page->folio conversions. Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 33 +++++++++++++++++---------------- 1 file changed, 17 insertions(+), 16 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index 9af2292e257b..128970a74aa2 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -3126,40 +3126,41 @@ static int shmem_symlink(struct user_namespace *mnt_userns, struct inode *dir, static void shmem_put_link(void *arg) { - mark_page_accessed(arg); - put_page(arg); + folio_mark_accessed(arg); + folio_put(arg); } static const char *shmem_get_link(struct dentry *dentry, struct inode *inode, struct delayed_call *done) { - struct page *page = NULL; + struct folio *folio = NULL; int error; + if (!dentry) { - page = find_get_page(inode->i_mapping, 0); - if (!page) + folio = filemap_get_folio(inode->i_mapping, 0); + if (!folio) return ERR_PTR(-ECHILD); - if (PageHWPoison(page) || - !PageUptodate(page)) { - put_page(page); + if (PageHWPoison(&folio->page) || + !folio_test_uptodate(folio)) { + folio_put(folio); return ERR_PTR(-ECHILD); } } else { - error = shmem_getpage(inode, 0, &page, SGP_READ); + error = shmem_get_folio(inode, 0, &folio, SGP_READ); if (error) return ERR_PTR(error); - if (!page) + if (!folio) return ERR_PTR(-ECHILD); - if (PageHWPoison(page)) { - unlock_page(page); - put_page(page); + if (PageHWPoison(&folio->page)) { + folio_unlock(folio); + folio_put(folio); return ERR_PTR(-ECHILD); } - unlock_page(page); + folio_unlock(folio); } - set_delayed_call(done, shmem_put_link, page); - return page_address(page); + set_delayed_call(done, shmem_put_link, folio); + return folio_address(folio); } #ifdef CONFIG_TMPFS_XATTR From patchwork Mon Aug 8 19:34:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939069 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F46DC25B08 for ; Mon, 8 Aug 2022 19:35:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E86178E0001; Mon, 8 Aug 2022 15:35:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DFAA28E0003; Mon, 8 Aug 2022 15:35:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ACB9B8E0001; Mon, 8 Aug 2022 15:35:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3DF7A8E000A for ; Mon, 8 Aug 2022 15:35:30 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 1DC5C120114 for ; Mon, 8 Aug 2022 19:35:30 +0000 (UTC) X-FDA: 79777429620.05.8F29661 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf26.hostedemail.com (Postfix) with ESMTP id B94CC14002C for ; Mon, 8 Aug 2022 19:35:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=UPtozOF/DITQ9DrzcZ6fY1WTW9Teyt71UWkDor+71lg=; b=R4wxYc9KR/ACDSPjUfAHWRGAAb YWC01rWjmBgQigS9cu/s+iCQ+/WtUPCAXDslEUenIMtlV5YzMUascowVtKfT2mkOmX4evQPEAksQ4 R44lr3gq2RiNGhMxn+gOuZ4LjfOYglwZNqSe15CWmJQsHZ1lkWUhWVKBZUlaoA5czVZWoh5NWHuHm NpsijOO9Bm81WIv26sULpEYHFZ/ITDpPAwVO/Roel4UDMJSBIouBu2VSiAKTu6vG/jk53b9Ia5QE8 IlNyyfFoD2FygSOellJeZeQgOeJVmnZsMvbCoI00VrCRJfEMDug80+kHSEVXNfgExuF0OTENK7+s0 vTAKDKTw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XA-00EAyX-A3; Mon, 08 Aug 2022 19:35:28 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 34/59] khugepaged: Call shmem_get_folio() Date: Mon, 8 Aug 2022 20:34:02 +0100 Message-Id: <20220808193430.3378317-35-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987329; a=rsa-sha256; cv=none; b=1AsANufYT161BsiWMgmBdV8KAwqjboEPBVZYYYRPDhYU4KaKNOWieXtClGLIcYg7chy0k0 nrQsqlLYl5FDLLSOx6OAWbYPphS1TFCbpUMPOqDax1sTBFfWA2fvrfuG9l9c4I0ygbyE7N snU8h8WOwi0BqI7EYvUVg40ZPiv0ftc= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=R4wxYc9K; dmarc=none; spf=none (imf26.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987329; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UPtozOF/DITQ9DrzcZ6fY1WTW9Teyt71UWkDor+71lg=; b=fN+crzuJUflrHjH3C5qFUHQVbHDz11b3lfIIITHGA5c2Nz9eSYyTwPsuOVFqn78l+cTmGo gaqQksBAn4iWUMlrzjMIDqIAkidAJ+9chH4piHZ0NoYu3g7aUKHYnpJn0GXlw1bU5vNUKk xGYbsEX2+RjVZdGTbE1JX41/RXci7xY= X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: B94CC14002C Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=R4wxYc9K; dmarc=none; spf=none (imf26.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: qpm95k3tyhcrg84n94zzo85tyhywnkek X-HE-Tag: 1659987329-477025 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: shmem_getpage() is being removed, so call its replacement and find the precise page ourselves. Signed-off-by: Matthew Wilcox (Oracle) --- mm/khugepaged.c | 7 +++++-- mm/shmem.c | 4 ++-- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 01f71786d530..6cf4bbd22fc9 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1682,13 +1682,16 @@ static void collapse_file(struct mm_struct *mm, } if (xa_is_value(page) || !PageUptodate(page)) { + struct folio *folio; + xas_unlock_irq(&xas); /* swap in or instantiate fallocated page */ - if (shmem_getpage(mapping->host, index, &page, - SGP_NOALLOC)) { + if (shmem_get_folio(mapping->host, index, + &folio, SGP_NOALLOC)) { result = SCAN_FAIL; goto xa_unlocked; } + page = folio_file_page(folio, index); } else if (trylock_page(page)) { get_page(page); xas_unlock_irq(&xas); diff --git a/mm/shmem.c b/mm/shmem.c index 128970a74aa2..da7ce4039430 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -3141,7 +3141,7 @@ static const char *shmem_get_link(struct dentry *dentry, folio = filemap_get_folio(inode->i_mapping, 0); if (!folio) return ERR_PTR(-ECHILD); - if (PageHWPoison(&folio->page) || + if (PageHWPoison(folio_page(folio, 0)) || !folio_test_uptodate(folio)) { folio_put(folio); return ERR_PTR(-ECHILD); @@ -3152,7 +3152,7 @@ static const char *shmem_get_link(struct dentry *dentry, return ERR_PTR(error); if (!folio) return ERR_PTR(-ECHILD); - if (PageHWPoison(&folio->page)) { + if (PageHWPoison(folio_page(folio, 0))) { folio_unlock(folio); folio_put(folio); return ERR_PTR(-ECHILD); From patchwork Mon Aug 8 19:34:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939070 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F47CC00140 for ; Mon, 8 Aug 2022 19:35:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8542F8E000B; Mon, 8 Aug 2022 15:35:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 78F898E0003; Mon, 8 Aug 2022 15:35:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5BB8A8E000B; Mon, 8 Aug 2022 15:35:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 44CFF8E0003 for ; Mon, 8 Aug 2022 15:35:32 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 2A0371A0944 for ; Mon, 8 Aug 2022 19:35:32 +0000 (UTC) X-FDA: 79777429704.27.03D9AFC Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf21.hostedemail.com (Postfix) with ESMTP id B81B11C018F for ; Mon, 8 Aug 2022 19:35:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=f9YAhEtoiQZH7wQXDCCvyo8IXQQUIYKAUUwcaMyCAKI=; b=rpfSjgs89aXZPDwRfXQuVO+ezl 3v6gN6HyNokt8c8G1+kUrsqh3aM5gQ+NTbz623+N1lgX8MuGxEGD7h2am0TIx7S00bWtJUqCbLyWY FFHsRN7u6Qpv8IEMumQwoabxs/730JGsCHdxXoujZshSMg4eMmDtduhAUNZXWGimQShHkeekjsMgt heOXlT6A+3BcPnsgybyi3vBAe4GQ09Q6hWMkSyOWkGUpv9jVkoj8liWTCPsXHSc3e1To1JWFFxJUV Tfki+VhZsmCchVtUJTGWnVMNKEOfZFsFMalYHwmeOQNEoby/On3ZhDYe9KqMF4yDwKMef0ve+QkT0 480un1aA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XA-00EAye-GE; Mon, 08 Aug 2022 19:35:28 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 35/59] userfaultfd: Convert mcontinue_atomic_pte() to use a folio Date: Mon, 8 Aug 2022 20:34:03 +0100 Message-Id: <20220808193430.3378317-36-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987331; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=f9YAhEtoiQZH7wQXDCCvyo8IXQQUIYKAUUwcaMyCAKI=; b=TgqxPdnefPWe/t0jhYcVRnMxJyTXiUGd+xqX/0RTCilHpyFhoroxxN0NRFSYwCcaCY41gY zn5GLsmbjMWqobLFSoH3JkGin944ebgO4YZ3HTbOgs84RXqNWMSl/ohCI1ouP2lx8xf6l5 WFw8Ge3B/KanN29zx23WxQxcfXFCQ6o= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=rpfSjgs8; dmarc=none; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987331; a=rsa-sha256; cv=none; b=yxvIqBwUy5//ocJ3rGvjH/nkB+et7/ekG6r0PJfFc6ZMLsM9g6VIBiug/kBkok+Sr6HCUt F9UZO0ICgIIXoL4lEK8fQh7FvCIdAyY4WWR9ksRYiVxqD8TtUli/rvJZcoerLRqpFbH6nF lPSKZR3xpAUsvGcgYTIpMPF6ntZWigw= X-Rspamd-Queue-Id: B81B11C018F X-Rspamd-Server: rspam03 X-Rspam-User: Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=rpfSjgs8; dmarc=none; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Stat-Signature: m9ouxqqbifrd6e4grgsu9xacic44533q X-HE-Tag: 1659987331-250880 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: shmem_getpage() is being replaced by shmem_get_folio() so use a folio throughout this function. Saves several calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/userfaultfd.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 07d3befc80e4..823f2da7aba5 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -243,20 +243,22 @@ static int mcontinue_atomic_pte(struct mm_struct *dst_mm, { struct inode *inode = file_inode(dst_vma->vm_file); pgoff_t pgoff = linear_page_index(dst_vma, dst_addr); + struct folio *folio; struct page *page; int ret; - ret = shmem_getpage(inode, pgoff, &page, SGP_NOALLOC); - /* Our caller expects us to return -EFAULT if we failed to find page. */ + ret = shmem_get_folio(inode, pgoff, &folio, SGP_NOALLOC); + /* Our caller expects us to return -EFAULT if we failed to find folio */ if (ret == -ENOENT) ret = -EFAULT; if (ret) goto out; - if (!page) { + if (!folio) { ret = -EFAULT; goto out; } + page = folio_file_page(folio, pgoff); if (PageHWPoison(page)) { ret = -EIO; goto out_release; @@ -267,13 +269,13 @@ static int mcontinue_atomic_pte(struct mm_struct *dst_mm, if (ret) goto out_release; - unlock_page(page); + folio_unlock(folio); ret = 0; out: return ret; out_release: - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); goto out; } From patchwork Mon Aug 8 19:34:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939088 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97A64C00140 for ; Mon, 8 Aug 2022 19:36:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9BAFF940028; Mon, 8 Aug 2022 15:35:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 93C0F940023; Mon, 8 Aug 2022 15:35:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7B652940028; Mon, 8 Aug 2022 15:35:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 646E5940023 for ; Mon, 8 Aug 2022 15:35:59 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 8B48D160196 for ; Mon, 8 Aug 2022 19:35:58 +0000 (UTC) X-FDA: 79777430796.12.E8C3D12 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id D5DB412015B for ; Mon, 8 Aug 2022 19:35:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=HR/mBIOdxosWBhe7YLzmGPGSClLtJ9GPmZ0+j22jXvQ=; b=vklXHJH1TP3wfj48pNk8cBN0I8 7pmfyedhrJq3VtjNuqmGtVG9s59pBPc+79gJsXls8NQc/NiCoudAeC15SyQ4Czh7xGEiLL7uZaqQK 7xCz7g5nf5DSQ7vcUds/c/yTKtDPUn+mLejOLTXtWfwu8rPaXrXkkfBMDcpoPtGjmqm0ZCyKsA5p6 p3qvGbgb0EaZ2YVPK+Ben/HOp8WUkyEGEu9n15p+a0oTcvnaVRPjXxvuGOOWK39BbRM8XESZMHTjm 9C8Fmt+MBQE0qeZLi6tE4NF2ZoHl4GuGjSCrKfgdZBsL7uUBKot6QAehEFc3j7BkYY/NwfZz6ID2f WIebdp5Q==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XA-00EAym-L9; Mon, 08 Aug 2022 19:35:30 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 36/59] shmem: Remove shmem_getpage() Date: Mon, 8 Aug 2022 20:34:04 +0100 Message-Id: <20220808193430.3378317-37-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=vklXHJH1; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987358; a=rsa-sha256; cv=none; b=mBISAdleu0DWoiY43vop3sdXVthXQBl4RcXwcT6qQSyiGTRSW/dsmCIT3nFFeu4pZ+Q50j Cyn0grpHXS+R3vALaZnx9Yk/HNFLrL6Y8/lyxYKPm5dk/0hPqPBM1ROdRd5iWzJ3YtirdZ Ikz2W/6tpdg5ppIQO/DdGocE4rg0TY8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987358; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HR/mBIOdxosWBhe7YLzmGPGSClLtJ9GPmZ0+j22jXvQ=; b=cwPbZKN5tb+1xo3ctWFJZpKZ2es6szUFCkYGt/NDAaQcnH2pdO4ZJV2xZKsYclKhx5SHo+ wbd9bB85S+0bSH6jJtb9XiXvSZqYrZOkE+x8X8IRi5pKAHrbMg/2dkcHI1/ac5QNQ46NeD GIjQh4teQ7pncnTbd0ULqjRkoiTO1MY= X-Stat-Signature: dd3taebjer6sx8p5x1iwg4u7jd8rewj3 X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: D5DB412015B Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=vklXHJH1; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1659987357-662743 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With all callers removed, remove this wrapper function. The flags are now mysteriously called SGP, but I think we can live with that. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/shmem_fs.h | 4 +--- mm/shmem.c | 15 +-------------- 2 files changed, 2 insertions(+), 17 deletions(-) diff --git a/include/linux/shmem_fs.h b/include/linux/shmem_fs.h index 0549929261f2..5c8cc23fcefe 100644 --- a/include/linux/shmem_fs.h +++ b/include/linux/shmem_fs.h @@ -107,7 +107,7 @@ extern unsigned long shmem_swap_usage(struct vm_area_struct *vma); extern unsigned long shmem_partial_swap_usage(struct address_space *mapping, pgoff_t start, pgoff_t end); -/* Flag allocation requirements to shmem_getpage */ +/* Flag allocation requirements to shmem_get_folio */ enum sgp_type { SGP_READ, /* don't exceed i_size, don't allocate page */ SGP_NOALLOC, /* similar, but fail on hole or use fallocated page */ @@ -116,8 +116,6 @@ enum sgp_type { SGP_FALLOC, /* like SGP_WRITE, but make existing page Uptodate */ }; -extern int shmem_getpage(struct inode *inode, pgoff_t index, - struct page **pagep, enum sgp_type sgp); int shmem_get_folio(struct inode *inode, pgoff_t index, struct folio **foliop, enum sgp_type sgp); diff --git a/mm/shmem.c b/mm/shmem.c index da7ce4039430..3cdfb061b2ff 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -179,7 +179,7 @@ static inline int shmem_reacct_size(unsigned long flags, /* * ... whereas tmpfs objects are accounted incrementally as * pages are allocated, in order to allow large sparse files. - * shmem_getpage reports shmem_acct_block failure as -ENOSPC not -ENOMEM, + * shmem_get_folio reports shmem_acct_block failure as -ENOSPC not -ENOMEM, * so that a failure on a sparse tmpfs mapping will give SIGBUS not OOM. */ static inline int shmem_acct_block(unsigned long flags, long pages) @@ -2029,19 +2029,6 @@ int shmem_get_folio(struct inode *inode, pgoff_t index, struct folio **foliop, mapping_gfp_mask(inode->i_mapping), NULL, NULL, NULL); } -int shmem_getpage(struct inode *inode, pgoff_t index, - struct page **pagep, enum sgp_type sgp) -{ - struct folio *folio = NULL; - int ret = shmem_get_folio(inode, index, &folio, sgp); - - if (folio) - *pagep = folio_file_page(folio, index); - else - *pagep = NULL; - return ret; -} - /* * This is like autoremove_wake_function, but it removes the wait queue * entry unconditionally - even if something else had already woken the From patchwork Mon Aug 8 19:34:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939090 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48631C00140 for ; Mon, 8 Aug 2022 19:36:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DD2E6940019; Mon, 8 Aug 2022 15:36:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D5E5D94000C; Mon, 8 Aug 2022 15:36:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B85F4940019; Mon, 8 Aug 2022 15:36:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A423D94000C for ; Mon, 8 Aug 2022 15:36:02 -0400 (EDT) Received: from smtpin31.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 84217A0856 for ; Mon, 8 Aug 2022 19:36:02 +0000 (UTC) X-FDA: 79777430964.31.C879AE7 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf14.hostedemail.com (Postfix) with ESMTP id 2A39A100054 for ; Mon, 8 Aug 2022 19:36:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=pI4aU5Niep/H4leC+h0AKItoqsaUlWZ93voTFp62Ql4=; b=SO/NCllTl+45tpF8hjtXwSaW/I 4sIOllg3eWSKa7fvEqfCz4a60f6v7FzdL15pda6mDYwmgMYehdqxxbhhb6sdFhF+mmuMtZDl95Gxz hHjy24Ri7dymukknWgsK/nCkoTxXJ1oqvh/B9G7uY+6UrBfdPB7LjAx2GpcX7ahJfAVb9baX5VpLG srE4qoN1P2FpQMHuhQoyfgIqsqYyqyfQ79vlvuCOy06YTmVql1kyHErttDuKhK4i2OEU/5Wjl+jo5 LVoLbjUJlgwjSVjXe0DC5HL262lf78bKM9HmmZOCTb9cZxNj/v6KGvk9ZB/0kYUdSMohnIi+f5l6N DwFJMaSw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XC-00EAz0-Ln; Mon, 08 Aug 2022 19:35:30 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 37/59] swapfile: Convert try_to_unuse() to use a folio Date: Mon, 8 Aug 2022 20:34:05 +0100 Message-Id: <20220808193430.3378317-38-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987362; a=rsa-sha256; cv=none; b=cQP0s4+uWoKRBKJzzNRg3HqelvORW3WgbWRqJwkyXTpTcWsrxYE15ERCbS7sS5vrwkhlh7 mK1anPTXuGoaQUYl064byQZcDPBY58dkFLA40V8l/92EMNSr4bB67EPkpfCQXcGbDaLZiw Id5waBG+4E6VaoDCI5+Fi71TBm1ofKc= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="SO/NCllT"; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987362; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pI4aU5Niep/H4leC+h0AKItoqsaUlWZ93voTFp62Ql4=; b=7jD5TDXxHEKPJ34LCA6ZoXWbdvTPgCHJkpPIS6C7+1PpjKNfRI3g8JN42KFdTJlbyKpWwj R1kviLb1jnFrLsHv/UiSMJhju/eI85PWMpbUyQyG/IYClmvLRV/uI19BsFTuZg+sZVV0Im zRVDdRXjH5VKa9swdzL1DIN5s1mXGN0= X-Rspamd-Queue-Id: 2A39A100054 X-Rspam-User: X-Rspamd-Server: rspam11 Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="SO/NCllT"; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Stat-Signature: qe6uq33zd9d7x11jo3kkpu8ou3mqcp7h X-HE-Tag: 1659987362-934193 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Saves five calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/swapfile.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index 186511a8ef4f..df85eb73f34e 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -2026,7 +2026,7 @@ static int try_to_unuse(unsigned int type) struct list_head *p; int retval = 0; struct swap_info_struct *si = swap_info[type]; - struct page *page; + struct folio *folio; swp_entry_t entry; unsigned int i; @@ -2076,21 +2076,21 @@ static int try_to_unuse(unsigned int type) (i = find_next_to_unuse(si, i)) != 0) { entry = swp_entry(type, i); - page = find_get_page(swap_address_space(entry), i); - if (!page) + folio = filemap_get_folio(swap_address_space(entry), i); + if (!folio) continue; /* - * It is conceivable that a racing task removed this page from - * swap cache just before we acquired the page lock. The page + * It is conceivable that a racing task removed this folio from + * swap cache just before we acquired the page lock. The folio * might even be back in swap cache on another swap area. But - * that is okay, try_to_free_swap() only removes stale pages. + * that is okay, folio_free_swap() only removes stale folios. */ - lock_page(page); - wait_on_page_writeback(page); - try_to_free_swap(page); - unlock_page(page); - put_page(page); + folio_lock(folio); + folio_wait_writeback(folio); + folio_free_swap(folio); + folio_unlock(folio); + folio_put(folio); } /* From patchwork Mon Aug 8 19:34:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939071 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E53EC25B08 for ; Mon, 8 Aug 2022 19:35:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 524128E0003; Mon, 8 Aug 2022 15:35:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 40D89940018; Mon, 8 Aug 2022 15:35:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1C2908E000C; Mon, 8 Aug 2022 15:35:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id EF2E48E0003 for ; Mon, 8 Aug 2022 15:35:32 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id CEEF9401EC for ; Mon, 8 Aug 2022 19:35:32 +0000 (UTC) X-FDA: 79777429704.17.B2EDC78 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id 792B020178 for ; Mon, 8 Aug 2022 19:35:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=YNJmY6dWypJzW3KXnsYA0Vz2JLUMi482l/ybv6oKdxY=; b=LUVQCCcZHeoMkW+/2WWOXAG1Hx 4PLsRsxqdLYJIHo+rW8gpJYqgY8IUxL2S/Ppqnh4mc5Fg0wjLJ+Hyy+MPFGfyysByLWWBGT9zq4ja oKOu0BRI/078eQDGpF1so+O+vxOLACVv0bc596tcG/0V8YJWrdD8YZD1BSKRgMdAGOya2cNiByJQw wuBoYgHGFOD9OBmlpDFJaM9SpH1JOA/uR71CXCSn/6Grf8Hv2yiDUGxeYGju502EgrOxdNvsIsuYL n45iLeZWtR56/gvkygvF0JQqci4WoX1kkOjtf101YnIlZfLQoFqgN75suCj8HChGtqb26axxLIqoh EexpcOrA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XC-00EAz8-Uz; Mon, 08 Aug 2022 19:35:31 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 38/59] swapfile: Convert __try_to_reclaim_swap() to use a folio Date: Mon, 8 Aug 2022 20:34:06 +0100 Message-Id: <20220808193430.3378317-39-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987332; a=rsa-sha256; cv=none; b=F/7i1MSg2mLN974W96JzcaX3CMzLOWk8t49Cyx3VR0sWlqEkd/BQ3QVO7tzhKVygJ4Fubo FBiW8kAgcd+w2KWPkLGGSr1kkOIThyBgB/CSSdvF6Dzft1zp5lm3yyZAUtw6Y6jEbw3CoZ FfzLPknXEu2q9RkSNrUhJjbk0ayzzX4= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=LUVQCCcZ; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987332; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YNJmY6dWypJzW3KXnsYA0Vz2JLUMi482l/ybv6oKdxY=; b=GLLpVs9PHaZ3+nyKj/6OHfyu9fs4tp7Ye6JMfktig891235XGtme0FfaiVbrNjVsiFNAdC L0Bc37HSLxwlBluIZ4dwhbm/YRHZq2hcHInCAGJEP+H7N3RcG6G2OGkBR1UhIt9e90Oo5j TzkCVMifU0VTw7S3862Iip78aVOLypc= X-Rspamd-Server: rspam06 X-Rspam-User: X-Stat-Signature: rywwfhyy51c4h7hwz688ghh9q8uzd7xz X-Rspamd-Queue-Id: 792B020178 Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=LUVQCCcZ; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1659987332-978735 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Saves five calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/swapfile.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index df85eb73f34e..ce538c3f9161 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -128,27 +128,27 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si, unsigned long offset, unsigned long flags) { swp_entry_t entry = swp_entry(si->type, offset); - struct page *page; + struct folio *folio; int ret = 0; - page = find_get_page(swap_address_space(entry), offset); - if (!page) + folio = filemap_get_folio(swap_address_space(entry), offset); + if (!folio) return 0; /* * When this function is called from scan_swap_map_slots() and it's - * called by vmscan.c at reclaiming pages. So, we hold a lock on a page, + * called by vmscan.c at reclaiming folios. So we hold a folio lock * here. We have to use trylock for avoiding deadlock. This is a special - * case and you should use try_to_free_swap() with explicit lock_page() + * case and you should use folio_free_swap() with explicit folio_lock() * in usual operations. */ - if (trylock_page(page)) { + if (folio_trylock(folio)) { if ((flags & TTRS_ANYWAY) || - ((flags & TTRS_UNMAPPED) && !page_mapped(page)) || - ((flags & TTRS_FULL) && mem_cgroup_swap_full(page))) - ret = try_to_free_swap(page); - unlock_page(page); + ((flags & TTRS_UNMAPPED) && !folio_mapped(folio)) || + ((flags & TTRS_FULL) && mem_cgroup_swap_full(&folio->page))) + ret = folio_free_swap(folio); + folio_unlock(folio); } - put_page(page); + folio_put(folio); return ret; } From patchwork Mon Aug 8 19:34:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939072 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BACD7C00140 for ; Mon, 8 Aug 2022 19:35:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EBD8B94001B; Mon, 8 Aug 2022 15:35:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E45F8940018; Mon, 8 Aug 2022 15:35:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B85A094001B; Mon, 8 Aug 2022 15:35:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 83741940018 for ; Mon, 8 Aug 2022 15:35:33 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 6410F1A0AB5 for ; Mon, 8 Aug 2022 19:35:33 +0000 (UTC) X-FDA: 79777429746.10.A334C11 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf06.hostedemail.com (Postfix) with ESMTP id F3F3B180145 for ; Mon, 8 Aug 2022 19:35:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=RwARIESPVJRvPjdBwRi4bFYQLWzQ2fXgslMpG4V504o=; b=NMM0eNj9D+5PD6yqkSBOcBx89h csMALxGWup2UgYZOBZteKiv3rXf9f2mDYglGaWHwIp0Qi80rJPifGCZjZSQhNNg9Llt+emlTidCRU wRIoePzg2ykG2QuW/SaPiFLRWrrxMX2SX7wAeL1i7lO2UFv2PtpieK7lwu7amn5d91oCr9KHa5XEb HpA7cc/UF/IJsTejJuMf5jnhXNzvWM5C7KSXyYSxJl+wqjAP4qA2+I+XPiU7dg69IiF2xECwJHRp2 xbop1QvyNEC2hKlEQpv33D6+PwEnDBzHx7/9nCudUxen51g43HgKwwDdxxzE1iHpnCkknCbqRZa5A 39DMHpfw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XD-00EAzF-8a; Mon, 08 Aug 2022 19:35:31 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 39/59] swapfile: Convert unuse_pte_range() to use a folio Date: Mon, 8 Aug 2022 20:34:07 +0100 Message-Id: <20220808193430.3378317-40-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987333; a=rsa-sha256; cv=none; b=aFEXB9lrgza6FqDr6OwqAiYcubvGGdW0zxA8Urs0oCCc/JVk3NcwHrISEK0Bz8TzvgpAlg 0v6IOMS/v64PSxCvxAd3QhQ8F7REqfFDdmmYoyT9dHzauRr9dLt5n4z7vioRLLMk1ksl5i qFkvVlnvZgmbWIOQXT48+wk51Zr4P0g= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=NMM0eNj9; dmarc=none; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987333; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RwARIESPVJRvPjdBwRi4bFYQLWzQ2fXgslMpG4V504o=; b=55MZD02n7GQlf3neIzrOMs64I0tLInC8M4etkWtXje/YMjYXctvhJAFfL0+/BRC7i/b2pE atJUAHhvp97/e3K9GWVsUHsvr2nnC+STt67kb6dkmVZqeVX3uwIp3eaZS7rgm7tAmsYxIZ /oHzHTVXrfTIP7GUSr/GO/EArul5GtM= X-Rspamd-Server: rspam10 X-Stat-Signature: 1ema1cprohnqyja7eigrcujd9bogudbk Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=NMM0eNj9; dmarc=none; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Rspamd-Queue-Id: F3F3B180145 X-HE-Tag: 1659987332-636235 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Delay fetching the precise page from the folio until we're in unuse_pte(). Saves many calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/swapfile.c | 33 +++++++++++++++++++-------------- 1 file changed, 19 insertions(+), 14 deletions(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index ce538c3f9161..9ee42a12cffc 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1754,8 +1754,9 @@ static inline int pte_same_as_swp(pte_t pte, pte_t swp_pte) * force COW, vm_page_prot omits write permission from any private vma. */ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, - unsigned long addr, swp_entry_t entry, struct page *page) + unsigned long addr, swp_entry_t entry, struct folio *folio) { + struct page *page = folio_file_page(folio, swp_offset(entry)); struct page *swapcache; spinlock_t *ptl; pte_t *pte, new_pte; @@ -1827,17 +1828,18 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, unsigned long end, unsigned int type) { - struct page *page; swp_entry_t entry; pte_t *pte; struct swap_info_struct *si; - unsigned long offset; int ret = 0; volatile unsigned char *swap_map; si = swap_info[type]; pte = pte_offset_map(pmd, addr); do { + struct folio *folio; + unsigned long offset; + if (!is_swap_pte(*pte)) continue; @@ -1848,8 +1850,9 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, offset = swp_offset(entry); pte_unmap(pte); swap_map = &si->swap_map[offset]; - page = lookup_swap_cache(entry, vma, addr); - if (!page) { + folio = swap_cache_get_folio(entry, vma, addr); + if (!folio) { + struct page *page; struct vm_fault vmf = { .vma = vma, .address = addr, @@ -1859,25 +1862,27 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, &vmf); + if (page) + folio = page_folio(page); } - if (!page) { + if (!folio) { if (*swap_map == 0 || *swap_map == SWAP_MAP_BAD) goto try_next; return -ENOMEM; } - lock_page(page); - wait_on_page_writeback(page); - ret = unuse_pte(vma, pmd, addr, entry, page); + folio_lock(folio); + folio_wait_writeback(folio); + ret = unuse_pte(vma, pmd, addr, entry, folio); if (ret < 0) { - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); goto out; } - try_to_free_swap(page); - unlock_page(page); - put_page(page); + folio_free_swap(folio); + folio_unlock(folio); + folio_put(folio); try_next: pte = pte_offset_map(pmd, addr); } while (pte++, addr += PAGE_SIZE, addr != end); From patchwork Mon Aug 8 19:34:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939073 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BEE4C25B08 for ; Mon, 8 Aug 2022 19:35:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9DCDB94001C; Mon, 8 Aug 2022 15:35:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8A0AD940018; Mon, 8 Aug 2022 15:35:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 719A794001C; Mon, 8 Aug 2022 15:35:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 5B571940018 for ; Mon, 8 Aug 2022 15:35:36 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3BC3540D84 for ; Mon, 8 Aug 2022 19:35:36 +0000 (UTC) X-FDA: 79777429872.15.2273100 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf15.hostedemail.com (Postfix) with ESMTP id C75B1A0050 for ; Mon, 8 Aug 2022 19:35:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=IcQHBc7Dn05IEjQ1QkZofVBHDYD9coNpNeAa/u7bGVY=; b=T25EZ42yinyKx+QV8R7zXPhdl8 Jk1o8WzfTKlTVsULC4hm1l8lgpMWHJewEPaeiqvhq9odTtlD3PIOotrA1poMd0j7JohcliXtiAXPY qKLd15WnqhMzkk6MQ+fML9tyNVzAa6NmR2GrBxeEUhPX6gC3pIant8E0HrSCAG20sPc1Op6KBUA0F Ga7PmFe45LB9YjMtsboVmMXdpIUnOfHoYUA3sruWfY94PHtLBMxo9N0W/gllLbIdsRu5jVrEUEeDE 0Ixfb6F6GHE+nj9BCHigetBsB4J+W/7wKO4VBQqJTJSCcJyJPkY/u5Vk5wILeZRDwZm8+C9Jvhj6A yVurSbFQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XD-00EAzM-Ez; Mon, 08 Aug 2022 19:35:31 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 40/59] mm: Convert do_swap_page() to use swap_cache_get_folio() Date: Mon, 8 Aug 2022 20:34:08 +0100 Message-Id: <20220808193430.3378317-41-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=T25EZ42y; spf=none (imf15.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987335; a=rsa-sha256; cv=none; b=Jaucz7vT68bTgwDzMb/YmXdzS1nvDxc+LR4RYvqa/7bfnwcQGiVWr67iHj4MCSzyMXanQd YAVnVjZtRciUssstKBcLh0vynkCI/COidEyeuZc2pTS7TZLzx1qBS76TgFDRsKeeYvMgfM 1GgPfgVje0lnwHgXNBOjYGF91IgE+8k= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987335; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IcQHBc7Dn05IEjQ1QkZofVBHDYD9coNpNeAa/u7bGVY=; b=VVjeJ+iCtLZ2h0AtFXbKf6mtvik/7f2hfFswpMJLycGDzXag+rnt+sXvrVmksKcumy35li 7546vsG6Avg+75cL6t+i3mVKde2kR1wFOtjPki2YTzQYIHgOfce52qkNBJ7ZwQMPDTGCfp PTKNxYH+TCoYyyJRaPCXTrB53DP5Pg8= X-Stat-Signature: x7biiss3zt47mr1hrqgpxhoea65q87zz X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: C75B1A0050 Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=T25EZ42y; spf=none (imf15.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1659987335-56060 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Saves a folio->page->folio conversion. Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 23b164bf3c70..5465037c237c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3761,9 +3761,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (unlikely(!si)) goto out; - page = lookup_swap_cache(entry, vma, vmf->address); - if (page) - folio = page_folio(page); + folio = swap_cache_get_folio(entry, vma, vmf->address); + if (folio) + page = folio_file_page(folio, swp_offset(entry)); swapcache = folio; if (!folio) { From patchwork Mon Aug 8 19:34:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939074 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6043CC00140 for ; Mon, 8 Aug 2022 19:35:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7BCED94001D; Mon, 8 Aug 2022 15:35:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5BDB1940018; Mon, 8 Aug 2022 15:35:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 19E5094001D; Mon, 8 Aug 2022 15:35:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E4341940018 for ; Mon, 8 Aug 2022 15:35:36 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C92D81C63D4 for ; Mon, 8 Aug 2022 19:35:36 +0000 (UTC) X-FDA: 79777429872.02.A28968E Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf22.hostedemail.com (Postfix) with ESMTP id E5AB8C0136 for ; Mon, 8 Aug 2022 19:35:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=wIlw5FNX3gdHLQFFPi6WZtVvC9jRyLN+M3rxcAEhghc=; b=ofwoH4hX44AnLQ+NA0aU5i2fnY Fng9H3/TW0vQk9TfKTg/Dil44Pb7EFLdQ1LEg6dRSkq3RHi+FMBl7CdvzPm2CzmxdtEKXvFdJhaQx H0RgoMJ5YjqgrnMf48dk28Huun5QcgYSIZPuM/eSWWEUpwQ1m9ZpdgE91LyVyXFGWaKrZIJvMbwdo xpQxavurKdRrjlB86Xoe5as1P0r6glNoabfJVjaXvboeLcddRGncXH+Y5XA3sdSrSN4edfuQY3EMd Uz0kjvci7xFI/ixs/cKsnMPAhLsHLvyRcu0kkmCPsYJ9vzIjuL922HDGeNSywff3Z8pwNbWZBFZLk jMoFuDcQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XD-00EAzU-So; Mon, 08 Aug 2022 19:35:34 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 41/59] mm: Remove lookup_swap_cache() Date: Mon, 8 Aug 2022 20:34:09 +0100 Message-Id: <20220808193430.3378317-42-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987335; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wIlw5FNX3gdHLQFFPi6WZtVvC9jRyLN+M3rxcAEhghc=; b=M1tcx+xTsPqTNGn5UFOQZBmTatfUmUAf/J+WbB38epO5NpQz9PWfRNQIgBqoBUAax4gt3L 0AEEnWBa4QQyCplCLdt5+ToDS6jBGhkKW54doA5qYffizquJpG22xDybCJQGXVs/AN3DJ7 ukMH8VOXzLKwJMBdZzI5gYaWJFpJULA= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ofwoH4hX; dmarc=none; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987336; a=rsa-sha256; cv=none; b=VRV3j9R4VMSDheXEwl70TWpEEewjSOZ5lzJ+LLamRgI2PbiCAZD7iFFg4ljL33+w/rRgTN SSVmaGKP7U2LYRwga34eZ93PN5kyQrCS2gvc8cRp4tpg8d7bzZLv9+n3XYEIrAVYYiGvEn DgL3BkJqRrYdfeDwTZHF8k2RGLgMXEI= Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ofwoH4hX; dmarc=none; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspamd-Server: rspam02 X-Stat-Signature: 65ec8pm9fqw1qi4n36ca8x1uh4hikhfs X-Rspamd-Queue-Id: E5AB8C0136 X-Rspam-User: X-HE-Tag: 1659987335-110237 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: All callers have now been converted to swap_cache_get_folio(), so we can remove this wrapper. Signed-off-by: Matthew Wilcox (Oracle) --- mm/memcontrol.c | 2 +- mm/swap.h | 10 ---------- mm/swap_state.c | 12 +----------- 3 files changed, 2 insertions(+), 22 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f65833efe90d..3b0698faf1ee 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5561,7 +5561,7 @@ static struct page *mc_handle_swap_pte(struct vm_area_struct *vma, return NULL; /* - * Because lookup_swap_cache() updates some statistics counter, + * Because swap_cache_get_folio() updates some statistics counter, * we call find_get_page() with swapper_space directly. */ page = find_get_page(swap_address_space(ent), swp_offset(ent)); diff --git a/mm/swap.h b/mm/swap.h index f70ff34dab82..9551d7815c37 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -43,9 +43,6 @@ void clear_shadow_from_swap_cache(int type, unsigned long begin, unsigned long end); struct folio *swap_cache_get_folio(swp_entry_t entry, struct vm_area_struct *vma, unsigned long addr); -struct page *lookup_swap_cache(swp_entry_t entry, - struct vm_area_struct *vma, - unsigned long addr); struct page *find_get_incore_page(struct address_space *mapping, pgoff_t index); struct page *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, @@ -103,13 +100,6 @@ static inline int swap_writepage(struct page *p, struct writeback_control *wbc) return 0; } -static inline struct page *lookup_swap_cache(swp_entry_t swp, - struct vm_area_struct *vma, - unsigned long addr) -{ - return NULL; -} - static inline struct page *find_get_incore_page(struct address_space *mapping, pgoff_t index) { diff --git a/mm/swap_state.c b/mm/swap_state.c index b96bf4ec8b5b..4af135a7b53c 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -369,16 +369,6 @@ struct folio *swap_cache_get_folio(swp_entry_t entry, return folio; } -struct page *lookup_swap_cache(swp_entry_t entry, struct vm_area_struct *vma, - unsigned long addr) -{ - struct folio *folio = swap_cache_get_folio(entry, vma, addr); - - if (!folio) - return NULL; - return folio_file_page(folio, swp_offset(entry)); -} - /** * find_get_incore_page - Find and get a page from the page or swap caches. * @mapping: The address_space to search. @@ -430,7 +420,7 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask, int err; /* * First check the swap cache. Since this is normally - * called after lookup_swap_cache() failed, re-calling + * called after swap_cache_get_folio() failed, re-calling * that would confuse statistics. */ si = get_swap_device(entry); From patchwork Mon Aug 8 19:34:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939075 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF2EAC25B08 for ; Mon, 8 Aug 2022 19:35:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 681B094001E; Mon, 8 Aug 2022 15:35:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 608BC940018; Mon, 8 Aug 2022 15:35:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 40CD994001E; Mon, 8 Aug 2022 15:35:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2B60E940018 for ; Mon, 8 Aug 2022 15:35:39 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id EEB04C0134 for ; Mon, 8 Aug 2022 19:35:38 +0000 (UTC) X-FDA: 79777429956.15.927D276 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf06.hostedemail.com (Postfix) with ESMTP id 9D5C218014C for ; Mon, 8 Aug 2022 19:35:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=6A6aPUSwkycrtSCt4V834moUq4uc1T4cliZhXfXil0c=; b=Ws3ErNENid8S22qJWB1Ij+zhwT dZ6xj0u4U9EKmxYfIvr4I5LwcX5amoAjgnuSFTWLGooFRNjAq24fRGPaIir9qAgIIiCt9kBaV4mp/ uvNNMrnBMkQeHsXqLXrrqAnH1TlsAcFqeQeg1UdWHbp05E0rXrLTGww8uYfuUDe4l/5c2pLJwaDfK H2N7gS+zfzf3BKJcVsfa/jEO41+9peeXEvy6rpcf6TxQekKXoLcQO4OCE5rvnlX4wfq4I2dMDa8XG a06qQrSyV64YEEB/TQDt1nAdSVdxLtIXFxDlXT6VbTSFkzXXTJWh1W1ONnhcXzpvfFrJo1LNt1kkG uB+7rzYw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XG-00EAzj-LV; Mon, 08 Aug 2022 19:35:34 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 42/59] swap_state: Convert free_swap_cache() to use a folio Date: Mon, 8 Aug 2022 20:34:10 +0100 Message-Id: <20220808193430.3378317-43-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987338; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6A6aPUSwkycrtSCt4V834moUq4uc1T4cliZhXfXil0c=; b=K1f0zM8hNPNYkXDjSbisbMXUQs3sT5ZQkyxdDsiR6fCzrkvOda+o0S4USCosXq2xoEp/ZL 4eYWZXvDKAl3isQptg2IjuhmtZESKo1fgkxreP3sr5uJcmxulDgPEZQAmJGn+DR/nOaEaP Wqj8yolJvKeOwUk251mob88Go24Ftmc= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Ws3ErNEN; dmarc=none; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987338; a=rsa-sha256; cv=none; b=0QBGcuy+PxHPv1UmXzhnVjYeJpFvGPwrD2tsALs/8rCYliJrGLIRfSRc7AHmVeQFczlvys KUQVaNzpydNt8fl/GaeZTprbbqpeytYCm2lrTsZlXdANq3LzZ3/Cy5HaFnNjFfP4lqtA1U JKm3SnnY+0qVkdJTdAwe2id5iyJtYxc= Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Ws3ErNEN; dmarc=none; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspamd-Server: rspam02 X-Stat-Signature: z1eeohddop7c14fdrqmidujf3s8uz76j X-Rspamd-Queue-Id: 9D5C218014C X-Rspam-User: X-HE-Tag: 1659987338-768351 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Saves several calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/swap_state.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/mm/swap_state.c b/mm/swap_state.c index 4af135a7b53c..438d0676c5be 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -272,16 +272,19 @@ void clear_shadow_from_swap_cache(int type, unsigned long begin, /* * If we are the only user, then try to free up the swap cache. * - * Its ok to check for PageSwapCache without the page lock + * Its ok to check the swapcache flag without the folio lock * here because we are going to recheck again inside - * try_to_free_swap() _with_ the lock. + * folio_free_swap() _with_ the lock. * - Marcelo */ void free_swap_cache(struct page *page) { - if (PageSwapCache(page) && !page_mapped(page) && trylock_page(page)) { - try_to_free_swap(page); - unlock_page(page); + struct folio *folio = page_folio(page); + + if (folio_test_swapcache(folio) && !folio_mapped(folio) && + folio_trylock(folio)) { + folio_free_swap(folio); + folio_unlock(folio); } } From patchwork Mon Aug 8 19:34:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939092 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D517C00140 for ; Mon, 8 Aug 2022 19:36:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9678294001B; Mon, 8 Aug 2022 15:36:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8F00E94000C; Mon, 8 Aug 2022 15:36:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 56BC294001B; Mon, 8 Aug 2022 15:36:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 2D73894001D for ; Mon, 8 Aug 2022 15:36:04 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C06FB80333 for ; Mon, 8 Aug 2022 19:36:03 +0000 (UTC) X-FDA: 79777431006.04.B2A22DF Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf31.hostedemail.com (Postfix) with ESMTP id 69F1D20161 for ; Mon, 8 Aug 2022 19:36:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=rTpYsQFP2H5ZjV6RoJPpO4sBbPRbz8lr5sbkwA+Wh7o=; b=dGsPDVUY60HMcu/Y+Ugm2T8cyn W6Yk4rvPB2B5nUTXEu3EXvnqViG+64qZ2P1U45ywHn2p8ZxlPig4Qxvi/mtSkx0JuBrSE+PllyyA3 UNbdEZp+RWDIb5GxFUJm9P7CF/lytzkZ4ZomW4H3YBEI5F/xNI1r2M6j9MeXzSmda+Qzu+MQcnvvZ lgD5akxNR0rQ5f0xwPF5sfaBxtYMmIsMywc+UPHXfyeWyK/L/5WZO94689Dk1JhIESI7SFf6uEnUX 7+qJ9wrMfdzOnzGIWXzBfiPReWh1rJyIkG5DSmmxEBbn+9qV2ADjrL6BUYnBYqcNdxus4djVwSlO8 fs2cizjg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XG-00EAzt-Qg; Mon, 08 Aug 2022 19:35:37 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 43/59] swap: Convert swap_writepage() to use a folio Date: Mon, 8 Aug 2022 20:34:11 +0100 Message-Id: <20220808193430.3378317-44-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=dGsPDVUY; spf=none (imf31.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987363; a=rsa-sha256; cv=none; b=V0nWrMWTwuR+JEzjuF9L/H18CDi2+UcEAXBqxdO0Vt+Y2TOOBhf5jMGCiDa+f530wHJWfu lLKgm0IrtA1cyxkbpvoSuthGLb9mHQY/Q6Oh0pcd2xjdt3gKpEOh9MzomZyrM3JU/7XQf9 xvUDxCHBn8M10wR1c0XE5K3vovGcTUw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987363; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rTpYsQFP2H5ZjV6RoJPpO4sBbPRbz8lr5sbkwA+Wh7o=; b=nw5VyLMQxttUyfEnJ1XmegADxciah5QYEvNjTtcKkmpywczYIGi5TdLYQwvMXNqE5CRZ1n avzjKo/mNK0Li4zTI1jMwvRN5sLragwB15RdNHgn79VVbl9btNF9VMuD/ihpPW/kkMXrhj rG1ZRIUIStGpgzs1NgvPU9Dtcp3t1dk= X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 69F1D20161 X-Rspam-User: Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=dGsPDVUY; spf=none (imf31.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Stat-Signature: 5hdxwugg5u1a5r63gkhnm966dpjchw85 X-HE-Tag: 1659987363-995164 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Removes many calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/page_io.c | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/mm/page_io.c b/mm/page_io.c index 68318134dc92..2a3e3e0a6497 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -180,29 +180,30 @@ int generic_swapfile_activate(struct swap_info_struct *sis, */ int swap_writepage(struct page *page, struct writeback_control *wbc) { + struct folio *folio = page_folio(page); int ret = 0; - if (try_to_free_swap(page)) { - unlock_page(page); + if (folio_free_swap(folio)) { + folio_unlock(folio); goto out; } /* * Arch code may have to preserve more data than just the page * contents, e.g. memory tags. */ - ret = arch_prepare_to_swap(page); + ret = arch_prepare_to_swap(&folio->page); if (ret) { - set_page_dirty(page); - unlock_page(page); + folio_mark_dirty(folio); + folio_unlock(folio); goto out; } - if (frontswap_store(page) == 0) { - set_page_writeback(page); - unlock_page(page); - end_page_writeback(page); + if (frontswap_store(&folio->page) == 0) { + folio_start_writeback(folio); + folio_unlock(folio); + folio_end_writeback(folio); goto out; } - ret = __swap_writepage(page, wbc, end_swap_bio_write); + ret = __swap_writepage(&folio->page, wbc, end_swap_bio_write); out: return ret; } From patchwork Mon Aug 8 19:34:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939076 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F597C00140 for ; Mon, 8 Aug 2022 19:35:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1C9B6940021; Mon, 8 Aug 2022 15:35:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0B1F094001F; Mon, 8 Aug 2022 15:35:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD281940020; Mon, 8 Aug 2022 15:35:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B5712940018 for ; Mon, 8 Aug 2022 15:35:39 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 62FC240D66 for ; Mon, 8 Aug 2022 19:35:39 +0000 (UTC) X-FDA: 79777429998.29.EEDCB93 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf10.hostedemail.com (Postfix) with ESMTP id 1D1DCC015C for ; Mon, 8 Aug 2022 19:35:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=42N2P8VBy7GhhphvSyGSriPslQU9frJYiGFx8k5syLc=; b=aKGK+gN5Ovn9kVdtMfJwkr5Hsg fyDeYSi9CBsWMLV9ivm3UiN2xgQqnO1GqckblJCVIKJJwHksMdDyzBjJd7JuU7LxOBYA/1UaG7ds8 GQ7r+4aXVuEZnFm6WSle15Rkj2087NFH3jBGrqH2AXqUeVJ1E9QQrfjKqh3nwiZLOrdfbNZVgFeH4 lIKVWNMR8h/UqjXjCuF6RMHgiZWWjgtGeO5RuHLtKzpbrtEd4rCw6+zrvDpbqn1mxxin9tc4OH4wS EkvQzochDcyX5hjUDA4c0qGq2DIJd/ufYDUIqQPgBwm+1BCTRGxq0cMLhfUS9BprEca+hdjg6H+kK GDRftyYQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XJ-00EB07-JR; Mon, 08 Aug 2022 19:35:37 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 44/59] mm: Convert do_wp_page() to use a folio Date: Mon, 8 Aug 2022 20:34:12 +0100 Message-Id: <20220808193430.3378317-45-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987339; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=42N2P8VBy7GhhphvSyGSriPslQU9frJYiGFx8k5syLc=; b=Abt4yhUeLTyfnAaKi4ElD2RzSzL40BTQB8NnRSqQLYP0UPMDbh5qnsV7TI/YicReDJ3tKt d36M1XvcFO/YndEI6ihSTfk54yRMFkciL1+xYUpo30OcPRlRcy8XT9Ute68ZfC9+cPebd+ BmGZ4Afu2KokPzBKQCCeknpEJgPPcVo= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=aKGK+gN5; dmarc=none; spf=none (imf10.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987339; a=rsa-sha256; cv=none; b=RIKpLx+ze2JS1VZhu5uxpT/D0ZlsYuTuwxSi7bEjpLci0cfBBzwl1WPJkNh6r+W+sq1Q15 O61C5kyfhyuCSecoiVkXY8Ti6OEvL8DWCb8dFE43r9ZznkmlOuXuiTuetNoguWjAwCip7W udcMdIUbFSYkjQpMHtSJTpzitLuoKyc= X-Rspamd-Queue-Id: 1D1DCC015C X-Rspamd-Server: rspam03 X-Rspam-User: Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=aKGK+gN5; dmarc=none; spf=none (imf10.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Stat-Signature: h4seinhrf8ykizb6x4mxb3jczai6giay X-HE-Tag: 1659987338-289521 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Saves many calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 40 ++++++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 5465037c237c..43432b877447 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3362,6 +3362,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) { const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE; struct vm_area_struct *vma = vmf->vma; + struct folio *folio; VM_BUG_ON(unshare && (vmf->flags & FAULT_FLAG_WRITE)); VM_BUG_ON(!unshare && !(vmf->flags & FAULT_FLAG_WRITE)); @@ -3408,48 +3409,47 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) * Take out anonymous pages first, anonymous shared vmas are * not dirty accountable. */ - if (PageAnon(vmf->page)) { - struct page *page = vmf->page; - + folio = page_folio(vmf->page); + if (folio_test_anon(folio)) { /* * If the page is exclusive to this process we must reuse the * page without further checks. */ - if (PageAnonExclusive(page)) + if (PageAnonExclusive(vmf->page)) goto reuse; /* - * We have to verify under page lock: these early checks are - * just an optimization to avoid locking the page and freeing + * We have to verify under folio lock: these early checks are + * just an optimization to avoid locking the folio and freeing * the swapcache if there is little hope that we can reuse. * - * PageKsm() doesn't necessarily raise the page refcount. + * KSM doesn't necessarily raise the folio refcount. */ - if (PageKsm(page) || page_count(page) > 3) + if (folio_test_ksm(folio) || folio_ref_count(folio) > 3) goto copy; - if (!PageLRU(page)) + if (!folio_test_lru(folio)) /* * Note: We cannot easily detect+handle references from - * remote LRU pagevecs or references to PageLRU() pages. + * remote LRU pagevecs or references to LRU folios. */ lru_add_drain(); - if (page_count(page) > 1 + PageSwapCache(page)) + if (folio_ref_count(folio) > 1 + folio_test_swapcache(folio)) goto copy; - if (!trylock_page(page)) + if (!folio_trylock(folio)) goto copy; - if (PageSwapCache(page)) - try_to_free_swap(page); - if (PageKsm(page) || page_count(page) != 1) { - unlock_page(page); + if (folio_test_swapcache(folio)) + folio_free_swap(folio); + if (folio_test_ksm(folio) || folio_ref_count(folio) != 1) { + folio_unlock(folio); goto copy; } /* - * Ok, we've got the only page reference from our mapping - * and the page is locked, it's dark out, and we're wearing + * Ok, we've got the only folio reference from our mapping + * and the folio is locked, it's dark out, and we're wearing * sunglasses. Hit it. */ - page_move_anon_rmap(page, vma); - unlock_page(page); + page_move_anon_rmap(vmf->page, vma); + folio_unlock(folio); reuse: if (unlikely(unshare)) { pte_unmap_unlock(vmf->pte, vmf->ptl); From patchwork Mon Aug 8 19:34:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939077 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91BCEC25B08 for ; Mon, 8 Aug 2022 19:35:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3F9F494001F; Mon, 8 Aug 2022 15:35:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1C579940020; Mon, 8 Aug 2022 15:35:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC00B940018; Mon, 8 Aug 2022 15:35:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id CD98A94001F for ; Mon, 8 Aug 2022 15:35:39 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id ADBB31A01A9 for ; Mon, 8 Aug 2022 19:35:39 +0000 (UTC) X-FDA: 79777429998.05.0DFE2FC Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf02.hostedemail.com (Postfix) with ESMTP id 64C1A80169 for ; Mon, 8 Aug 2022 19:35:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=FPbPinexUBKsHg2vN+cvLFrHcGFh99O3FGVUIwS4RZY=; b=bkPkNS18cspTZZqhoCEQFZmS4O Xy5aV1N4sYKZeJQDzD7BNplQTLTaALIAE9BN3UgaWMnTN2zMRN7jSGQYzVHhuEZmdkGq48k8GZYJF dR9F2RbFMw5fwOmM+Wq2JpsIwPkpaXLFrBKcZxH+s1F1fzoaXX2zsK6ZVIpSCkG5GokjdoO6F37Nf gJlyw2IE5NM0NJCNNzy61AvXflqt8zoiMsgHDWnCF8FbYm2WeJkoCcwNKD7tFmVyqxrSGtQlo9yGI OVnqK1xMx0qE4fGnWOAMzVDUMttJ4dbOzp/tcvt5AJS+qANr7YUJ9UXF6rTrv3oPT/0fbX/WDD1n5 duxDlsCw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XJ-00EB0F-R6; Mon, 08 Aug 2022 19:35:37 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 45/59] huge_memory: Convert do_huge_pmd_wp_page() to use a folio Date: Mon, 8 Aug 2022 20:34:13 +0100 Message-Id: <20220808193430.3378317-46-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987339; a=rsa-sha256; cv=none; b=GR0QLazpwSaoeSJpB7PljX8KCa3eF08Q8PNGWn/SDEDgtdyc53FeFgy+by1ZrpsdCA/E3Q WUq85XyomOJUgDY2rtXMStdNnmjZSWQMH610poGzLapYUj9VRjp8nWL16zehF0V9KXZeAG +epZ5tRsGckawjE54l/V5ZeTkPuLMds= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987339; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FPbPinexUBKsHg2vN+cvLFrHcGFh99O3FGVUIwS4RZY=; b=MKi70wNua2QbrRW1DA4diBg0EOJ1uOrKDWOtsGLH3Y+zw5DJsw1E45icoB2KSOyeYPmsGp q2160/0L0jI8aWUKFFpRuuT2TxowH4F+hXkPflj1k3BScm5jrkdYtuhq+XnuUWI4yjDwSz ruST5CazE7O+w+GmP4miFZz7uk4TqE4= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=bkPkNS18; spf=none (imf02.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Stat-Signature: an4j46iohcganbb9apdz7ymff9zgh3sg X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 64C1A80169 Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=bkPkNS18; spf=none (imf02.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1659987339-575222 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Removes many calls to compound_head(). Does not remove the assumption that a folio may not be larger than a PMD. Signed-off-by: Matthew Wilcox (Oracle) --- mm/huge_memory.c | 35 +++++++++++++++++++---------------- 1 file changed, 19 insertions(+), 16 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 8a7c1b344abe..7b998f2083aa 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1313,6 +1313,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf) { const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE; struct vm_area_struct *vma = vmf->vma; + struct folio *folio; struct page *page; unsigned long haddr = vmf->address & HPAGE_PMD_MASK; pmd_t orig_pmd = vmf->orig_pmd; @@ -1334,46 +1335,48 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf) } page = pmd_page(orig_pmd); + folio = page_folio(page); VM_BUG_ON_PAGE(!PageHead(page), page); /* Early check when only holding the PT lock. */ if (PageAnonExclusive(page)) goto reuse; - if (!trylock_page(page)) { - get_page(page); + if (!folio_trylock(folio)) { + folio_get(folio); spin_unlock(vmf->ptl); - lock_page(page); + folio_lock(folio); spin_lock(vmf->ptl); if (unlikely(!pmd_same(*vmf->pmd, orig_pmd))) { spin_unlock(vmf->ptl); - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); return 0; } - put_page(page); + folio_put(folio); } /* Recheck after temporarily dropping the PT lock. */ if (PageAnonExclusive(page)) { - unlock_page(page); + folio_unlock(folio); goto reuse; } /* - * See do_wp_page(): we can only reuse the page exclusively if there are - * no additional references. Note that we always drain the LRU - * pagevecs immediately after adding a THP. + * See do_wp_page(): we can only reuse the folio exclusively if + * there are no additional references. Note that we always drain + * the LRU pagevecs immediately after adding a THP. */ - if (page_count(page) > 1 + PageSwapCache(page) * thp_nr_pages(page)) + if (folio_ref_count(folio) > + 1 + folio_test_swapcache(folio) * folio_nr_pages(folio)) goto unlock_fallback; - if (PageSwapCache(page)) - try_to_free_swap(page); - if (page_count(page) == 1) { + if (folio_test_swapcache(folio)) + folio_free_swap(folio); + if (folio_ref_count(folio) == 1) { pmd_t entry; page_move_anon_rmap(page, vma); - unlock_page(page); + folio_unlock(folio); reuse: if (unlikely(unshare)) { spin_unlock(vmf->ptl); @@ -1388,7 +1391,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf) } unlock_fallback: - unlock_page(page); + folio_unlock(folio); spin_unlock(vmf->ptl); fallback: __split_huge_pmd(vma, vmf->pmd, vmf->address, false, NULL); From patchwork Mon Aug 8 19:34:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939078 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18CB0C00140 for ; Mon, 8 Aug 2022 19:35:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9185C6B0073; Mon, 8 Aug 2022 15:35:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 879F06B0074; Mon, 8 Aug 2022 15:35:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 60589940022; Mon, 8 Aug 2022 15:35:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 270C8940018 for ; Mon, 8 Aug 2022 15:35:40 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id E61F6C019A for ; Mon, 8 Aug 2022 19:35:39 +0000 (UTC) X-FDA: 79777429998.26.F99FDC2 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf21.hostedemail.com (Postfix) with ESMTP id 929051C005A for ; Mon, 8 Aug 2022 19:35:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=9mOmmVtZpFIWWo3nfTdSfePzyVXKHrnvdlc0DSY2URw=; b=ulk5fJbGld4YTUhYPsK819aM/u deJFxCZtd1dX1BFAElWhl5ceiliGwft3tZfNoj5jxlHh3pIRUub9zpG7T9WFVNmw3kgn5DOYZA8KW C7pMZBr34zIx9pLrRxOWnWXpCH+mZGcExmXx4sg95xv14DYgHU+zS4a14gJDWwJeMLLyobZEOgwBt 1yOQwz7RiloU0kUaYaiF6tBmlBx/b465+PDv+ODeHZQveRVnWU6p+CP8n+jLkUpZYiMu3jJuB+ET5 5S1qsuMa6oIBGmzcO/sgQ0j/Ycb846XBa0hk7KcyvhZDr/AOe1JqvlYicZxTyGBnK/bF0Z0zpEtIp 7lg6CzSA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XK-00EB0M-0n; Mon, 08 Aug 2022 19:35:38 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 46/59] madvise: Convert madvise_free_pte_range() to use a folio Date: Mon, 8 Aug 2022 20:34:14 +0100 Message-Id: <20220808193430.3378317-47-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987339; a=rsa-sha256; cv=none; b=Z+zW1lJwyOHwCh2d6ykRQ4s5+TmxGdrsYyK/7STxI1ksk1n+r4zGHh/jfEAmS/AA+pwkAN rTmwsiFJcCoNBX7GDzpLesXYHroHyKCq3OI4rlI3q3hF4GwsAlZrQGW8HCjUWq6co0GDG0 IY18tGjk5+FHjrAEEhE/kqMURFZOn7o= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987339; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9mOmmVtZpFIWWo3nfTdSfePzyVXKHrnvdlc0DSY2URw=; b=KjQ1Y5CMmtkn+Fepp1pvWfyLPVkdFQIiztk8EQ2oIJef5TM2+s26Kq3Aqie1lJfS0eDqVW 5SsZK/0iRUqvN5zy2TRgGW6I0PcpHvH6LSaEr3ttYrWR/UWzBqyiPSD0G0iDjYJiwtFF4B aJuWPro5YJvZypqcIbw3Rl2DkTy6x8M= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ulk5fJbG; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Stat-Signature: bazihtk5kpiarngozxpwer9firn8ub5r X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 929051C005A Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=ulk5fJbG; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1659987339-529066 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Saves a lot of calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/madvise.c | 49 +++++++++++++++++++++++++------------------------ 1 file changed, 25 insertions(+), 24 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index 5f0f0948a50e..05de79d269e8 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -597,6 +597,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, struct vm_area_struct *vma = walk->vma; spinlock_t *ptl; pte_t *orig_pte, *pte, ptent; + struct folio *folio; struct page *page; int nr_swap = 0; unsigned long next; @@ -641,56 +642,56 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, page = vm_normal_page(vma, addr, ptent); if (!page || is_zone_device_page(page)) continue; + folio = page_folio(page); /* - * If pmd isn't transhuge but the page is THP and + * If pmd isn't transhuge but the folio is large and * is owned by only this process, split it and * deactivate all pages. */ - if (PageTransCompound(page)) { - if (page_mapcount(page) != 1) + if (folio_test_large(folio)) { + if (folio_mapcount(folio) != 1) goto out; - get_page(page); - if (!trylock_page(page)) { - put_page(page); + folio_get(folio); + if (!folio_trylock(folio)) { + folio_put(folio); goto out; } pte_unmap_unlock(orig_pte, ptl); - if (split_huge_page(page)) { - unlock_page(page); - put_page(page); + if (split_folio(folio)) { + folio_unlock(folio); + folio_put(folio); orig_pte = pte_offset_map_lock(mm, pmd, addr, &ptl); goto out; } - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); orig_pte = pte = pte_offset_map_lock(mm, pmd, addr, &ptl); pte--; addr -= PAGE_SIZE; continue; } - VM_BUG_ON_PAGE(PageTransCompound(page), page); - - if (PageSwapCache(page) || PageDirty(page)) { - if (!trylock_page(page)) + if (folio_test_swapcache(folio) || folio_test_dirty(folio)) { + if (!folio_trylock(folio)) continue; /* - * If page is shared with others, we couldn't clear - * PG_dirty of the page. + * If folio is shared with others, we mustn't clear + * the folio's dirty flag. */ - if (page_mapcount(page) != 1) { - unlock_page(page); + if (folio_mapcount(folio) != 1) { + folio_unlock(folio); continue; } - if (PageSwapCache(page) && !try_to_free_swap(page)) { - unlock_page(page); + if (folio_test_swapcache(folio) && + !folio_free_swap(folio)) { + folio_unlock(folio); continue; } - ClearPageDirty(page); - unlock_page(page); + folio_clear_dirty(folio); + folio_unlock(folio); } if (pte_young(ptent) || pte_dirty(ptent)) { @@ -708,7 +709,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, set_pte_at(mm, addr, pte, ptent); tlb_remove_tlb_entry(tlb, pte, addr); } - mark_page_lazyfree(page); + mark_page_lazyfree(&folio->page); } out: if (nr_swap) { From patchwork Mon Aug 8 19:34:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939079 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D43AC25B08 for ; Mon, 8 Aug 2022 19:35:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C4DB66B0071; Mon, 8 Aug 2022 15:35:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BDB5C940020; Mon, 8 Aug 2022 15:35:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8A1B7940018; Mon, 8 Aug 2022 15:35:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 6DDC46B0071 for ; Mon, 8 Aug 2022 15:35:43 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 32B10A0C08 for ; Mon, 8 Aug 2022 19:35:43 +0000 (UTC) X-FDA: 79777430166.07.D834104 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf04.hostedemail.com (Postfix) with ESMTP id AEA1A40174 for ; Mon, 8 Aug 2022 19:35:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=cDP+DNYsXXA+kb85wo7Nrqn0aeI4BRiMp8pyQ5b78Q0=; b=nYRW3KJ1Oxn4+Dx/WPEAhmgGK9 ReSZ6tsOYiI5pbwD3OWnD8EiCD8hwUcV9tkALmTMz6hJbk25z3Eyf9egXIc2MsxSBhir8+6v9Bvez QZHyMcpluDVdN6EBWtLPxD0Cv4DX4PK4CU/ioPomBduccgv8IKs3PT3ylHyXR21+1N++AOyEg7cQu QQJBgHbf+ndtwANduak3v3GvxAWXHCaDMrcfhwtfs0gex27Hn8PKJn+6OikiebTUcHrwRzmF2+uXo 7vfuwOMs+pXSDaZ18uD6igmYE4xABB1DdztbIu4vb+NoBO1X5EbyeDaxDHfjAbL22ETjss8bGnwxD DfrJJnWg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XK-00EB0S-84; Mon, 08 Aug 2022 19:35:38 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 47/59] uprobes: Use folios more widely in __replace_page() Date: Mon, 8 Aug 2022 20:34:15 +0100 Message-Id: <20220808193430.3378317-48-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987342; a=rsa-sha256; cv=none; b=ApB4oAxS224u0AaXBILaOfBD1DfecZLQMu8MzoSKfL7L2pgggLbDlmNaKq0i1lmSnpxA3w 77x7FOKwiDRl0/FeWwD/+lrvyNLCw/TioHpX1t8M94HoC58qnPyVRMveYvQYlhz8mJfm+/ Zlj4PvdsvrRDn9ODWivHF4b2dYPGKyM= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=nYRW3KJ1; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987342; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cDP+DNYsXXA+kb85wo7Nrqn0aeI4BRiMp8pyQ5b78Q0=; b=L1FmjzmfHpfVgvYTOhoAMZwzzUNpgiDPMXz25qoUPVX8eQbC4ydXX5Bsz3QhbsRI6Z8oYT t/jcdTxo+o5TfnrNwUEbGUe4QTZrMaM4/VnKKZclL//vCy2zpN9Rty5tSJPPe6k6a7DQxq JlebZc67cOEIe8EvjuVpbWWns1BDtqA= X-Rspamd-Server: rspam06 X-Rspam-User: X-Stat-Signature: xrs8ufwayt1as6i6h47irpqd8g1hqaix X-Rspamd-Queue-Id: AEA1A40174 Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=nYRW3KJ1; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1659987342-772289 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove a few hidden calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- kernel/events/uprobes.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 2eaa327f8158..9722c4587c48 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -19,7 +19,7 @@ #include #include /* anon_vma_prepare */ #include /* set_pte_at_notify */ -#include /* try_to_free_swap */ +#include /* folio_free_swap */ #include /* user_enable_single_step */ #include /* notifier mechanism */ #include "../../mm/internal.h" /* munlock_vma_page */ @@ -154,8 +154,9 @@ static loff_t vaddr_to_offset(struct vm_area_struct *vma, unsigned long vaddr) static int __replace_page(struct vm_area_struct *vma, unsigned long addr, struct page *old_page, struct page *new_page) { + struct folio *old_folio = page_folio(old_page); struct mm_struct *mm = vma->vm_mm; - DEFINE_FOLIO_VMA_WALK(pvmw, page_folio(old_page), vma, addr, 0); + DEFINE_FOLIO_VMA_WALK(pvmw, old_folio, vma, addr, 0); int err; struct mmu_notifier_range range; @@ -169,8 +170,8 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, return err; } - /* For try_to_free_swap() below */ - lock_page(old_page); + /* For folio_free_swap() below */ + folio_lock(old_folio); mmu_notifier_invalidate_range_start(&range); err = -EAGAIN; @@ -186,7 +187,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, /* no new page, just dec_mm_counter for old_page */ dec_mm_counter(mm, MM_ANONPAGES); - if (!PageAnon(old_page)) { + if (!folio_test_anon(old_folio)) { dec_mm_counter(mm, mm_counter_file(old_page)); inc_mm_counter(mm, MM_ANONPAGES); } @@ -198,15 +199,15 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, mk_pte(new_page, vma->vm_page_prot)); page_remove_rmap(old_page, vma, false); - if (!page_mapped(old_page)) - try_to_free_swap(old_page); + if (!folio_mapped(old_folio)) + folio_free_swap(old_folio); page_vma_mapped_walk_done(&pvmw); - put_page(old_page); + folio_put(old_folio); err = 0; unlock: mmu_notifier_invalidate_range_end(&range); - unlock_page(old_page); + folio_unlock(old_folio); return err; } From patchwork Mon Aug 8 19:34:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939094 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96D5FC00140 for ; Mon, 8 Aug 2022 19:36:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F0BEE8E000D; Mon, 8 Aug 2022 15:36:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E94948E000B; Mon, 8 Aug 2022 15:36:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE6D48E000D; Mon, 8 Aug 2022 15:36:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B93DB8E000A for ; Mon, 8 Aug 2022 15:36:17 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8CB32120114 for ; Mon, 8 Aug 2022 19:36:17 +0000 (UTC) X-FDA: 79777431594.29.48262BA Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf02.hostedemail.com (Postfix) with ESMTP id 4B5FE8016F for ; Mon, 8 Aug 2022 19:36:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=YMk9TEVmL8q7WxjorjEAwIc7bPZKGMaknIib7W5PcUw=; b=tmCCUT9YbVZ+85cH89OGN/1JHg PU5Pp8iCiz0NhcWlH3MMtCpoDhXVEPaM16dvP+dI7QIGL3XoyAZPDuiNj2YfWHLjtN3RfFqBLs/wE 1kLVv0FmULIB3Xjns9wMMe7NQEnAwDPYhthmS9FsDeJZCfnBs+hsfGLi5O7LGYFR4PTHOmM4xo0d0 gXq5nlY+O4th0tTWqlgtyUeMOPwxUmyGT6LZipt/gvDthbCiOaRLTsjH0Uq33ilosskTPPnMa++C8 zAvN1Lj5M7TSM36+Q9nJnKrloRipTjVWyRW/l497SxJyzTlYDJb41Y/BcNwuaINDXWB2OhTaG5fMx 6HLWcYkA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XK-00EB0a-Fh; Mon, 08 Aug 2022 19:35:41 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 48/59] ksm: Use a folio in replace_page() Date: Mon, 8 Aug 2022 20:34:16 +0100 Message-Id: <20220808193430.3378317-49-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987377; a=rsa-sha256; cv=none; b=WTfBJUGWaRhNUdCfu1y/h6GC7qJvldupF5HzRJaVK7yuIzNhfoaFNVJnwDq6ONkwf7tql+ X5DZHH5mnbAve5/qkDrr+vS1Kl2L96Eq6xDeiAzjm+UYagVz6gSN2ZvpTDJIQyRW8y+/us 4OK7K90eIBNEkDIeuM6F7JZ3SYf4wng= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987377; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YMk9TEVmL8q7WxjorjEAwIc7bPZKGMaknIib7W5PcUw=; b=clxDKaOl3OW0frAxpAky3hF77ePHRnOJybCWu4KbqUsPyovwX1RNS2UGaD1qHz/8jnWOkj UCykfj0obtZroQhYn+aXW+ezYxMxTBeQ8LvzcP6rRzfT62V3s1Cy9RyL5Tk9fooLsqgMrV /oR10Fnv8l/8nw+elgunRGa3UoFamws= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=tmCCUT9Y; spf=none (imf02.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Stat-Signature: becbk85figsed179d9u14bige3ifyxym X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 4B5FE8016F Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=tmCCUT9Y; spf=none (imf02.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1659987377-672992 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace three calls to compound_head() with one. Signed-off-by: Matthew Wilcox (Oracle) --- mm/ksm.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/ksm.c b/mm/ksm.c index 42ab153335a2..322652e7e6fe 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1133,6 +1133,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, struct page *kpage, pte_t orig_pte) { struct mm_struct *mm = vma->vm_mm; + struct folio *folio; pmd_t *pmd; pte_t *ptep; pte_t newpte; @@ -1191,10 +1192,11 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, ptep_clear_flush(vma, addr, ptep); set_pte_at_notify(mm, addr, ptep, newpte); + folio = page_folio(page); page_remove_rmap(page, vma, false); - if (!page_mapped(page)) - try_to_free_swap(page); - put_page(page); + if (!folio_mapped(folio)) + folio_free_swap(folio); + folio_put(folio); pte_unmap_unlock(ptep, ptl); err = 0; From patchwork Mon Aug 8 19:34:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939095 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6145EC25B08 for ; Mon, 8 Aug 2022 19:36:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 29C8D8E000A; Mon, 8 Aug 2022 15:36:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2194D940009; Mon, 8 Aug 2022 15:36:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DCFCF8E000A; Mon, 8 Aug 2022 15:36:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C2E3E8E000B for ; Mon, 8 Aug 2022 15:36:17 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A7F6F401B8 for ; Mon, 8 Aug 2022 19:36:17 +0000 (UTC) X-FDA: 79777431594.12.8071EE6 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf14.hostedemail.com (Postfix) with ESMTP id 4F332100161 for ; Mon, 8 Aug 2022 19:36:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=fv2y1iAyO39DZlMXbrHe8clJyhsWTUGrGxo2kThV8U0=; b=djbMfEYLrnrTQElSsmXbk45aoY 6d1Po+FjQDEY99uGyxNsVFla1Ej33NX62qRTl9RiuKeaT3EZJQcjhdXR2IS58AGE2D2I7wZsncAe+ /Nc52AvJ7bnQNZKOjB5sh34ub0ovMCO2wIDnguyusL3pXGgPxVszJzOrOErQwjn1xCLTbK/Sz2XW8 e8WonXKtZxrQtyLz5ktbP1mI7xNEB1U+NDW/PRhE70mjATm0Uo94US+EOFIz3pHBaTxC+BuuGKTtV +IIvTGsFU/ct08J+FoBx+enqJsjuQb15qOmu51G4JAC78b99ye3FRDjM6ODhp4hWME1uJdyeCb4vH GgN3miHA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XN-00EB13-A7; Mon, 08 Aug 2022 19:35:43 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 49/59] mm: Convert do_swap_page() to use folio_free_swap() Date: Mon, 8 Aug 2022 20:34:17 +0100 Message-Id: <20220808193430.3378317-50-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987377; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fv2y1iAyO39DZlMXbrHe8clJyhsWTUGrGxo2kThV8U0=; b=M5exNFzOl5A7gUdmylywe8m0hvTOB/49dUz2n4fOEOTQes6u8F1nJZwDF434/WgoWw19tL tW0fAG7amVebkdNWsYiOnlikMsSkA5szb3IDAOkr0GPatl5JT1EYSOPYHaRHj6CNTsybkh wPOS9BEZLRRm7pm6/97NiXTG8uIpRNA= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=djbMfEYL; dmarc=none; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987377; a=rsa-sha256; cv=none; b=mbbARyvOLr+NEuufPKhPw6EjlJVwYW3WXs1wOqg2tjZKURB2BzMLA0LmR3lG7/4EcZPp1t a4ARj3oEz8A8GAFEJpzDwJPlTJp65Nr43INSL/7NwhOcyJ6QTPtUf2MGrTh0ZLDEgZbFc9 aKOhH2ZOvtQjKIf2Mb5z28L2e264fG8= X-Rspamd-Queue-Id: 4F332100161 X-Rspamd-Server: rspam03 X-Rspam-User: Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=djbMfEYL; dmarc=none; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Stat-Signature: f5xr99rc1dixxxjnxdgbds1sd36kgcj1 X-HE-Tag: 1659987377-342735 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Also convert should_try_to_free_swap() to use a folio. This removes a few calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 43432b877447..5b440045d306 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3635,14 +3635,14 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf) return 0; } -static inline bool should_try_to_free_swap(struct page *page, +static inline bool should_try_to_free_swap(struct folio *folio, struct vm_area_struct *vma, unsigned int fault_flags) { - if (!PageSwapCache(page)) + if (!folio_test_swapcache(folio)) return false; - if (mem_cgroup_swap_full(page) || (vma->vm_flags & VM_LOCKED) || - PageMlocked(page)) + if (mem_cgroup_swap_full(&folio->page) || (vma->vm_flags & VM_LOCKED) || + folio_test_mlocked(folio)) return true; /* * If we want to map a page that's in the swapcache writable, we @@ -3650,8 +3650,8 @@ static inline bool should_try_to_free_swap(struct page *page, * user. Try freeing the swapcache to get rid of the swapcache * reference only in case it's likely that we'll be the exlusive user. */ - return (fault_flags & FAULT_FLAG_WRITE) && !PageKsm(page) && - page_count(page) == 2; + return (fault_flags & FAULT_FLAG_WRITE) && !folio_test_ksm(folio) && + folio_ref_count(folio) == 2; } static vm_fault_t pte_marker_clear(struct vm_fault *vmf) @@ -3944,8 +3944,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * yet. */ swap_free(entry); - if (should_try_to_free_swap(page, vma, vmf->flags)) - try_to_free_swap(page); + if (should_try_to_free_swap(folio, vma, vmf->flags)) + folio_free_swap(folio); inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); dec_mm_counter_fast(vma->vm_mm, MM_SWAPENTS); From patchwork Mon Aug 8 19:34:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939081 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64F8EC25B08 for ; Mon, 8 Aug 2022 19:35:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8A5D0940020; Mon, 8 Aug 2022 15:35:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7B7606B0075; Mon, 8 Aug 2022 15:35:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 59362940020; Mon, 8 Aug 2022 15:35:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3E7A16B0074 for ; Mon, 8 Aug 2022 15:35:48 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 21F02AB3BE for ; Mon, 8 Aug 2022 19:35:48 +0000 (UTC) X-FDA: 79777430376.17.753ECC3 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf06.hostedemail.com (Postfix) with ESMTP id BCEF918014A for ; Mon, 8 Aug 2022 19:35:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=jfGX7dEfasv1wxZafBF6k8KgKDVoKzhQTUTPCWkB5MU=; b=LdNR+XxwBRzk1hYyIkJ6jceizO j59G3KYh2caYg0MnoJK9qiz3RmoTL1wjnyvniaSt5zGRM5tNKbVAxlmfHJOtNeuvpoSTs053AL+yR WUVcjzKzeVNfLoDR5LtG65weKj0ecd1w7wLm8gS5fyzjA5LAUXybpKMaLgMuCe/x5nMW0vAml4UWP Hc4sED+SoWYvDs7X9nkgCTWYTnBww+hiahtpDp5nnV/J1i2KfGQyL8oIEjSIZWSO+V93C87QETOtT tKCMxG5QjBGWY2LjbEWlqMpXuMNwaRcuv03TZZeqh2a8zkV9TCJSBckNqAgED6c11x9Mf9DJO4tR2 Tk1bEUTA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XQ-00EB1o-6P; Mon, 08 Aug 2022 19:35:44 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 50/59] memcg: Convert mem_cgroup_swap_full() to take a folio Date: Mon, 8 Aug 2022 20:34:18 +0100 Message-Id: <20220808193430.3378317-51-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=LdNR+Xxw; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987347; a=rsa-sha256; cv=none; b=Wkmr4dUO/yI1Q+wkrQ++xlhdk4aMrO6WgxvRa0oV3RJleT9kYPgPxkCFM5lbG+cuJWGT5r q0NPf8wzX/kWCJSp3J4/0VlAp6foVJU0MJkO5tpjOm2+m9kr35Vcwxz9Q9iMbcCDu3rM2w tu9aBc3azT2QcWFZ98FUzxwj9pj8IIc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987347; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jfGX7dEfasv1wxZafBF6k8KgKDVoKzhQTUTPCWkB5MU=; b=AmTj5qMvAqp5skKaueied3jinmD3A1LB+HNqsR866mclWNgc83avPQPClZM0M9xe+CCmW/ fA2zIfXGdldmMl0k09UYMl/cOtAV35JVNZ5sI3q1wg1GdQ7iW+2NgCv2rKuIXhYhC/l7x4 N8RFMKG7Cz1tLgjFdzCGh7kqe5onYe4= X-Stat-Signature: 7iyu6ahtqo3kfepw1k1w7bpmf81eph58 X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: BCEF918014A Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=LdNR+Xxw; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1659987347-765045 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: All callers now have a folio, so convert the function to take a folio. Saves a couple of calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/swap.h | 4 ++-- mm/memcontrol.c | 6 +++--- mm/memory.c | 2 +- mm/swapfile.c | 2 +- mm/vmscan.c | 3 +-- 5 files changed, 8 insertions(+), 9 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index f16c9af6bf32..8178ec471fe3 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -688,7 +688,7 @@ static inline void mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_p } extern long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg); -extern bool mem_cgroup_swap_full(struct page *page); +extern bool mem_cgroup_swap_full(struct folio *folio); #else static inline void mem_cgroup_swapout(struct folio *folio, swp_entry_t entry) { @@ -710,7 +710,7 @@ static inline long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg) return get_nr_swap_pages(); } -static inline bool mem_cgroup_swap_full(struct page *page) +static inline bool mem_cgroup_swap_full(struct folio *folio) { return vm_swap_full(); } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 3b0698faf1ee..557579f43e7b 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -7375,18 +7375,18 @@ long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg) return nr_swap_pages; } -bool mem_cgroup_swap_full(struct page *page) +bool mem_cgroup_swap_full(struct folio *folio) { struct mem_cgroup *memcg; - VM_BUG_ON_PAGE(!PageLocked(page), page); + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); if (vm_swap_full()) return true; if (cgroup_memory_noswap || !cgroup_subsys_on_dfl(memory_cgrp_subsys)) return false; - memcg = page_memcg(page); + memcg = folio_memcg(folio); if (!memcg) return false; diff --git a/mm/memory.c b/mm/memory.c index 5b440045d306..b6f2ccda4f45 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3641,7 +3641,7 @@ static inline bool should_try_to_free_swap(struct folio *folio, { if (!folio_test_swapcache(folio)) return false; - if (mem_cgroup_swap_full(&folio->page) || (vma->vm_flags & VM_LOCKED) || + if (mem_cgroup_swap_full(folio) || (vma->vm_flags & VM_LOCKED) || folio_test_mlocked(folio)) return true; /* diff --git a/mm/swapfile.c b/mm/swapfile.c index 9ee42a12cffc..74929a8cbf88 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -144,7 +144,7 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si, if (folio_trylock(folio)) { if ((flags & TTRS_ANYWAY) || ((flags & TTRS_UNMAPPED) && !folio_mapped(folio)) || - ((flags & TTRS_FULL) && mem_cgroup_swap_full(&folio->page))) + ((flags & TTRS_FULL) && mem_cgroup_swap_full(folio))) ret = folio_free_swap(folio); folio_unlock(folio); } diff --git a/mm/vmscan.c b/mm/vmscan.c index ac7f6f77e28a..85af57bbfd81 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2002,8 +2002,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, activate_locked: /* Not a candidate for swapping, so reclaim swap space. */ if (folio_test_swapcache(folio) && - (mem_cgroup_swap_full(&folio->page) || - folio_test_mlocked(folio))) + (mem_cgroup_swap_full(folio) || folio_test_mlocked(folio))) folio_free_swap(folio); VM_BUG_ON_FOLIO(folio_test_active(folio), folio); if (!folio_test_mlocked(folio)) { From patchwork Mon Aug 8 19:34:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939084 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFABCC00140 for ; Mon, 8 Aug 2022 19:35:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4AB696B0074; Mon, 8 Aug 2022 15:35:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 40F5E6B0075; Mon, 8 Aug 2022 15:35:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F08F9940023; Mon, 8 Aug 2022 15:35:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C6BB9940023 for ; Mon, 8 Aug 2022 15:35:51 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 9FBC78015D for ; Mon, 8 Aug 2022 19:35:51 +0000 (UTC) X-FDA: 79777430502.12.82476E5 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf25.hostedemail.com (Postfix) with ESMTP id 36C1BA0164 for ; Mon, 8 Aug 2022 19:35:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=OsFrXHvB8cYITjSSiLVE8LTYqmXFfAMGQNN9DW2H+TI=; b=Vy/LP8K3DIV3TPPNdUJ+0BCZwD yVP+NVZToRNK464d01Ig6gV3ZCN9FewXFXseN0uaVXUBQWYa8yTJTUyeZ4RC4gfzXbrL5tpgX6/S1 2E2z+Cxhh0wNGxBEp4Gbjfg5WCa8XP9KqKZw+mFXBdwLfgivkJs1l4u7QY3awsSP+wMhzFsIF6XtY 7Bl5XNiMiNNmvMR9+8+OxQu3laJXgpiW+b0MZ9wmmzvZqyvSeNNQZo9TIS6mQV/f3KQMKIUGauBkm CHK0h5y2AkXfBREkAdxwUUcaaAM21emvpTP0fwRvblBjZDi4P8dW/S+7uqtzqiNc1ekQkZ4gEutSt FV/QwOKA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XQ-00EB1w-Ed; Mon, 08 Aug 2022 19:35:46 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 51/59] mm: Remove try_to_free_swap() Date: Mon, 8 Aug 2022 20:34:19 +0100 Message-Id: <20220808193430.3378317-52-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987351; a=rsa-sha256; cv=none; b=hgZYe5PT6Gg6L0dE5CGjIjTbRSrg0q/z8pnvv0m7+QHhkikzTpb5HpWy6konMwssjN/j1k GfmUsS5ItKwR+RN/GmdpkruvlZCpGge8tpvglB58Q3o87kMRC4biAdWOncxGtnYtHxFUeE Sqs553uavftfmfwd2kH140BbVruvdsA= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="Vy/LP8K3"; dmarc=none; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987351; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OsFrXHvB8cYITjSSiLVE8LTYqmXFfAMGQNN9DW2H+TI=; b=xlwuCVXyhLUt4f9kXsAP90FkGoHCd0aWStu/1mIoQSLSqInEpiEHTAaAi/rwZRcRaxVXOb gHTFSa8swesi8UVmeeZCt4YEwT0hkSLJYWK2KL3No8HLV42BBEb0iNyXTZN4G155QYv1Gn AvuywCvmfT+wjAgMUDJLm46TorYPvE8= X-Rspamd-Server: rspam10 X-Stat-Signature: 63rbbozw37jnpaja75nid5q6aaqh8i7q Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="Vy/LP8K3"; dmarc=none; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Rspamd-Queue-Id: 36C1BA0164 X-HE-Tag: 1659987351-997479 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: All callers have now been converted to folio_free_swap() and we can remove this wrapper. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/swap.h | 6 ------ mm/folio-compat.c | 7 ------- mm/memory.c | 2 +- 3 files changed, 1 insertion(+), 14 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 8178ec471fe3..3be59affca63 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -506,7 +506,6 @@ extern int __swp_swapcount(swp_entry_t entry); extern int swp_swapcount(swp_entry_t entry); extern struct swap_info_struct *page_swap_info(struct page *); extern struct swap_info_struct *swp_swap_info(swp_entry_t entry); -extern int try_to_free_swap(struct page *); struct backing_dev_info; extern int init_swap_address_space(unsigned int type, unsigned long nr_pages); extern void exit_swap_address_space(unsigned int type); @@ -591,11 +590,6 @@ static inline int swp_swapcount(swp_entry_t entry) return 0; } -static inline int try_to_free_swap(struct page *page) -{ - return 0; -} - static inline swp_entry_t folio_alloc_swap(struct folio *folio) { swp_entry_t entry; diff --git a/mm/folio-compat.c b/mm/folio-compat.c index 06d47f00609b..e1e23b4947d7 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -146,10 +146,3 @@ void putback_lru_page(struct page *page) { folio_putback_lru(page_folio(page)); } - -#ifdef CONFIG_SWAP -int try_to_free_swap(struct page *page) -{ - return folio_free_swap(page_folio(page)); -} -#endif diff --git a/mm/memory.c b/mm/memory.c index b6f2ccda4f45..76b0e67399a1 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3838,7 +3838,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (swapcache) { /* - * Make sure try_to_free_swap or swapoff did not release the + * Make sure folio_free_swap() or swapoff did not release the * swapcache from under us. The page pin, and pte_same test * below, are not enough to exclude that. Even if it is still * swapcache, we need to check that the page's swap has not From patchwork Mon Aug 8 19:34:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939082 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED2A6C00140 for ; Mon, 8 Aug 2022 19:35:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 45DCE940022; Mon, 8 Aug 2022 15:35:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2AD956B0075; Mon, 8 Aug 2022 15:35:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 12BCE940022; Mon, 8 Aug 2022 15:35:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id B84F56B0074 for ; Mon, 8 Aug 2022 15:35:48 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 94D98C019A for ; Mon, 8 Aug 2022 19:35:48 +0000 (UTC) X-FDA: 79777430376.25.2CEEEB4 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf12.hostedemail.com (Postfix) with ESMTP id 53BF54016A for ; Mon, 8 Aug 2022 19:35:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=SLtFH9Ktf5rT30cGnE4FvYlOFxqPubA96OX3PYhEr0k=; b=LZSXiNHaQoAM/EL87HPlcxHQIl ZvFGoywLZzGZcwUZIc5A4O8ZHkwNAZktHo7COrkBjuQU1k/Mjjd//eHQvaHuD7poUMQ3xsHThqF0T r6CZavquDHThoDRv7a54xZ6S1YmJU2iYVD7a4DhkmmFwh1c1cbIt2cMsr9S1txJ8Cp+oBKV0OIwCb DwRc2iodVVjpKbNH6G5tm280cdNH7JMk3/6X3GyYvGL1TziZTsGc27eenuuY+c/gKTIi02hL3T6cF YQtTAvv0GI/rujya1gTRTfRluQftH7tQ/BMRm/W7fT5n7Q2RNp+NqcvgfDO3eZzQluTCW+98WO3+h NDdXqesQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XS-00EB2F-R1; Mon, 08 Aug 2022 19:35:46 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 52/59] rmap: Convert page_move_anon_rmap() to use a folio Date: Mon, 8 Aug 2022 20:34:20 +0100 Message-Id: <20220808193430.3378317-53-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987348; a=rsa-sha256; cv=none; b=C5nRWOcqFOIpw+eNVgZCNR0dgAShoZQs0ouT8esnx7gFOQIj7faXjj7i5P8LCzTzW5/8bE 2DJlBkSMx4Uo/PoaoROf7qCN/8uJYh5mhvXVGtwgYl6jlrIBHNy1NKn0p46E8B2W47E3+R hd7JI9VXexuOji14QQZE79eL9R8meBE= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=LZSXiNHa; spf=none (imf12.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987348; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SLtFH9Ktf5rT30cGnE4FvYlOFxqPubA96OX3PYhEr0k=; b=jEqRnlY+IpmtSz+S1PXd5aqLWMbUAy0njO8vztj92tAYPEoiTxm6SMXLfKTatZbrpp7Lus Rx1u2oId88bY0fyULBORpSZ6cs43GFvA579VyuX4Wxkbc4uoAU6OBdCKOEq/5Ln/p779cs l8djCKU1RRwOf8Uc2iF2cJPOnqSos7o= X-Stat-Signature: ae4oh5aags38wy1b8dw9x13uay9m6c3w X-Rspamd-Queue-Id: 53BF54016A Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=LZSXiNHa; spf=none (imf12.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1659987348-808133 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Removes one call to compound_head() and a reference to page->mapping. Signed-off-by: Matthew Wilcox (Oracle) --- mm/rmap.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index edc06c52bc82..aaab1c1078b4 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1098,22 +1098,20 @@ int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff, */ void page_move_anon_rmap(struct page *page, struct vm_area_struct *vma) { - struct anon_vma *anon_vma = vma->anon_vma; - struct page *subpage = page; - - page = compound_head(page); + void *anon_vma = vma->anon_vma; + struct folio *folio = page_folio(page); - VM_BUG_ON_PAGE(!PageLocked(page), page); + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_VMA(!anon_vma, vma); - anon_vma = (void *) anon_vma + PAGE_MAPPING_ANON; + anon_vma += PAGE_MAPPING_ANON; /* * Ensure that anon_vma and the PAGE_MAPPING_ANON bit are written * simultaneously, so a concurrent reader (eg folio_referenced()'s * folio_test_anon()) will not see one without the other. */ - WRITE_ONCE(page->mapping, (struct address_space *) anon_vma); - SetPageAnonExclusive(subpage); + WRITE_ONCE(folio->mapping, anon_vma); + SetPageAnonExclusive(page); } /** From patchwork Mon Aug 8 19:34:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939083 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5D85C25B08 for ; Mon, 8 Aug 2022 19:35:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F0EAF940024; Mon, 8 Aug 2022 15:35:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E93A96B0078; Mon, 8 Aug 2022 15:35:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C975C940024; Mon, 8 Aug 2022 15:35:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A98446B0074 for ; Mon, 8 Aug 2022 15:35:51 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 87F1E802D9 for ; Mon, 8 Aug 2022 19:35:51 +0000 (UTC) X-FDA: 79777430502.29.C9E995D Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf06.hostedemail.com (Postfix) with ESMTP id 2068718014D for ; Mon, 8 Aug 2022 19:35:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=T5pqUNJVuk3AY3ms43palyFIe+URqU42hbhfDsQmVx0=; b=X6kRMSPgZir19leBvkWGS6yBTo 2XC4KVJSozHPTg4eLhTQduFm6Yub3LJxyXthRSWaT0NeD161XkT2fi8hZMaMU4ljGaCL9XHtlWqXx WD4e5CcR2SmQCvV5RuKr3Mb8EICfqL7bi3hk7RwUPiupuu1MLyobRFtbMhI4Q4fVu4bM7ZAPCditI e707+TsiaESID5Iup7Oxgr2tkXu8mBykhZSGijjuMB+zqLrYXRETnI3Ga36pLmLB+A6rAOi5YCsfr S+U2Vs02XkL9g/iomud497GNAE6JZreuhqXEZv/jaMbFpmPi5jln11aDO4zDSsAGYZLxoQm/oe+Mb Y8OKlzOA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XT-00EB2O-1q; Mon, 08 Aug 2022 19:35:47 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 53/59] migrate: Convert __unmap_and_move() to use folios Date: Mon, 8 Aug 2022 20:34:21 +0100 Message-Id: <20220808193430.3378317-54-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987351; a=rsa-sha256; cv=none; b=3L/EjBdkQWCZQGJq65H57xvIlzhhLYtbT+4DI7Ns2qFd6mWOnoMyYCSTENbTIo2SkoygS1 bVqz8iCjx4TMFZJFfCebJXGdGUB5p3O4ya2VCZ3IldfexNfrnuYo4IFytv2vhHtHguAQY6 Uz+vwwiLhmgeCguWxRsBJdIE8uPA5Lw= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=X6kRMSPg; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987351; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=T5pqUNJVuk3AY3ms43palyFIe+URqU42hbhfDsQmVx0=; b=uEov/7ATZsVxLEYjorXPr4HDKMruCA48OMbiWgDAjMC8CH/qtbZbyW4xqc9g+jIV9Ji2Ov 4cydaRc+JLQdnuGmuKkoU3nVIWo9y4oE2Fi5Eppjud45VBEZq9OKq+HtCQxEe6XSzXS0vf p1krfQbkJiVHRvaVAjbMctLei5tIx8s= X-Rspamd-Queue-Id: 2068718014D Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=X6kRMSPg; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 56gxx3ygzmph8x7qon314eqjk5ygw3o4 X-HE-Tag: 1659987350-48507 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Removes a lot of calls to compound_head(). Also remove a VM_BUG_ON that can never trigger as the PageAnon bit is the bottom bit of page->mapping. Signed-off-by: Matthew Wilcox (Oracle) --- mm/migrate.c | 75 ++++++++++++++++++++++++++-------------------------- 1 file changed, 37 insertions(+), 38 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 6a1597c92261..3cd1fdb5f572 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -976,17 +976,15 @@ static int move_to_new_folio(struct folio *dst, struct folio *src, return rc; } -static int __unmap_and_move(struct page *page, struct page *newpage, +static int __unmap_and_move(struct folio *src, struct folio *dst, int force, enum migrate_mode mode) { - struct folio *folio = page_folio(page); - struct folio *dst = page_folio(newpage); int rc = -EAGAIN; bool page_was_mapped = false; struct anon_vma *anon_vma = NULL; - bool is_lru = !__PageMovable(page); + bool is_lru = !__PageMovable(&src->page); - if (!trylock_page(page)) { + if (!folio_trylock(src)) { if (!force || mode == MIGRATE_ASYNC) goto out; @@ -1006,10 +1004,10 @@ static int __unmap_and_move(struct page *page, struct page *newpage, if (current->flags & PF_MEMALLOC) goto out; - lock_page(page); + folio_lock(src); } - if (PageWriteback(page)) { + if (folio_test_writeback(src)) { /* * Only in the case of a full synchronous migration is it * necessary to wait for PageWriteback. In the async case, @@ -1026,12 +1024,12 @@ static int __unmap_and_move(struct page *page, struct page *newpage, } if (!force) goto out_unlock; - wait_on_page_writeback(page); + folio_wait_writeback(src); } /* - * By try_to_migrate(), page->mapcount goes down to 0 here. In this case, - * we cannot notice that anon_vma is freed while we migrates a page. + * By try_to_migrate(), src->mapcount goes down to 0 here. In this case, + * we cannot notice that anon_vma is freed while we migrate a page. * This get_anon_vma() delays freeing anon_vma pointer until the end * of migration. File cache pages are no problem because of page_lock() * File Caches may use write_page() or lock_page() in migration, then, @@ -1043,22 +1041,22 @@ static int __unmap_and_move(struct page *page, struct page *newpage, * because that implies that the anon page is no longer mapped * (and cannot be remapped so long as we hold the page lock). */ - if (PageAnon(page) && !PageKsm(page)) - anon_vma = page_get_anon_vma(page); + if (folio_test_anon(src) && !folio_test_ksm(src)) + anon_vma = page_get_anon_vma(&src->page); /* * Block others from accessing the new page when we get around to * establishing additional references. We are usually the only one - * holding a reference to newpage at this point. We used to have a BUG - * here if trylock_page(newpage) fails, but would like to allow for - * cases where there might be a race with the previous use of newpage. + * holding a reference to dst at this point. We used to have a BUG + * here if folio_trylock(dst) fails, but would like to allow for + * cases where there might be a race with the previous use of dst. * This is much like races on refcount of oldpage: just don't BUG(). */ - if (unlikely(!trylock_page(newpage))) + if (unlikely(!folio_trylock(dst))) goto out_unlock; if (unlikely(!is_lru)) { - rc = move_to_new_folio(dst, folio, mode); + rc = move_to_new_folio(dst, src, mode); goto out_unlock_both; } @@ -1066,7 +1064,7 @@ static int __unmap_and_move(struct page *page, struct page *newpage, * Corner case handling: * 1. When a new swap-cache page is read into, it is added to the LRU * and treated as swapcache but it has no rmap yet. - * Calling try_to_unmap() against a page->mapping==NULL page will + * Calling try_to_unmap() against a src->mapping==NULL page will * trigger a BUG. So handle it here. * 2. An orphaned page (see truncate_cleanup_page) might have * fs-private metadata. The page can be picked up due to memory @@ -1074,57 +1072,56 @@ static int __unmap_and_move(struct page *page, struct page *newpage, * invisible to the vm, so the page can not be migrated. So try to * free the metadata, so the page can be freed. */ - if (!page->mapping) { - VM_BUG_ON_PAGE(PageAnon(page), page); - if (page_has_private(page)) { - try_to_free_buffers(folio); + if (!src->mapping) { + if (folio_test_private(src)) { + try_to_free_buffers(src); goto out_unlock_both; } - } else if (page_mapped(page)) { + } else if (folio_mapped(src)) { /* Establish migration ptes */ - VM_BUG_ON_PAGE(PageAnon(page) && !PageKsm(page) && !anon_vma, - page); - try_to_migrate(folio, 0); + VM_BUG_ON_FOLIO(folio_test_anon(src) && + !folio_test_ksm(src) && !anon_vma, src); + try_to_migrate(src, 0); page_was_mapped = true; } - if (!page_mapped(page)) - rc = move_to_new_folio(dst, folio, mode); + if (!folio_mapped(src)) + rc = move_to_new_folio(dst, src, mode); /* - * When successful, push newpage to LRU immediately: so that if it + * When successful, push dst to LRU immediately: so that if it * turns out to be an mlocked page, remove_migration_ptes() will - * automatically build up the correct newpage->mlock_count for it. + * automatically build up the correct dst->mlock_count for it. * * We would like to do something similar for the old page, when * unsuccessful, and other cases when a page has been temporarily * isolated from the unevictable LRU: but this case is the easiest. */ if (rc == MIGRATEPAGE_SUCCESS) { - lru_cache_add(newpage); + folio_add_lru(dst); if (page_was_mapped) lru_add_drain(); } if (page_was_mapped) - remove_migration_ptes(folio, - rc == MIGRATEPAGE_SUCCESS ? dst : folio, false); + remove_migration_ptes(src, + rc == MIGRATEPAGE_SUCCESS ? dst : src, false); out_unlock_both: - unlock_page(newpage); + folio_unlock(dst); out_unlock: /* Drop an anon_vma reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); - unlock_page(page); + folio_unlock(src); out: /* - * If migration is successful, decrease refcount of the newpage, + * If migration is successful, decrease refcount of dst, * which will not free the page because new page owner increased * refcounter. */ if (rc == MIGRATEPAGE_SUCCESS) - put_page(newpage); + folio_put(dst); return rc; } @@ -1140,6 +1137,7 @@ static int unmap_and_move(new_page_t get_new_page, enum migrate_reason reason, struct list_head *ret) { + struct folio *dst, *src = page_folio(page); int rc = MIGRATEPAGE_SUCCESS; struct page *newpage = NULL; @@ -1157,9 +1155,10 @@ static int unmap_and_move(new_page_t get_new_page, newpage = get_new_page(page, private); if (!newpage) return -ENOMEM; + dst = page_folio(newpage); newpage->private = 0; - rc = __unmap_and_move(page, newpage, force, mode); + rc = __unmap_and_move(src, dst, force, mode); if (rc == MIGRATEPAGE_SUCCESS) set_page_owner_migrate_reason(newpage, reason); From patchwork Mon Aug 8 19:34:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939087 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A47F9C25B08 for ; Mon, 8 Aug 2022 19:36:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B89D4940027; Mon, 8 Aug 2022 15:35:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AE95D940023; Mon, 8 Aug 2022 15:35:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 91C7B940028; Mon, 8 Aug 2022 15:35:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 573E3940027 for ; Mon, 8 Aug 2022 15:35:55 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 0F4201A0B61 for ; Mon, 8 Aug 2022 19:35:55 +0000 (UTC) X-FDA: 79777430670.04.083FE29 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf04.hostedemail.com (Postfix) with ESMTP id 924D64016F for ; Mon, 8 Aug 2022 19:35:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=HZ8b1C8NuYghJrw/IzYZl0rPHNemrZPs9QNaQ3LLfIg=; b=eftnJqU0jWeWiL+O1chLeJkF+G WxhYyjJf3BvEzFo7wXpaVLt6qZI8MmXmCPYEJigckj4zJJBQ6TnDgQ0zMgB6wbVoD9lhKMxlLebDB MZbhI4jr/Ki/wHLP2+06PLDPotysWK5j0b/YhXXyoIMYnJ9jrmrN0zVZ6odYjoQo3WQIYuAfet5VI F3fzEjEuhy1vAmZ7pnP/OgL3VitVeQqVqoLJQmrDe6qq1nVO/Llg5oa3eDMlwXU2RyWgDEtIXllSi BwhtiDbMIIVaV2fFgkYmsxWBJ6FO+EZWPZXp3iSTN5YWhi8jV9vbPDwKV9GPOEoWJOp1Ig40Q0aPL 2gM1T2oQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XT-00EB2V-7o; Mon, 08 Aug 2022 19:35:49 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 54/59] migrate: Convert unmap_and_move_huge_page() to use folios Date: Mon, 8 Aug 2022 20:34:22 +0100 Message-Id: <20220808193430.3378317-55-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987354; a=rsa-sha256; cv=none; b=CkCVbwIq6JjkHaCjin9/VcSjyFkRNR7dqyeV0P3ZNfJjBf5DZkvi4Q6damaMPPOGI1zROg UQv+CxjgB2ej5sq/JwFMjmT6/jR5NyyQZXM3yqfimoZcXI1myHkWfHFJhfR0RgXU36NzR3 4V5YHBHWr0CVuqjp1qowmCMvMGX2UGo= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=eftnJqU0; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987354; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HZ8b1C8NuYghJrw/IzYZl0rPHNemrZPs9QNaQ3LLfIg=; b=coxjdFCRiqCt0MxuJPdvGPjhf0p7D6yp1XMrYlVLjkzDX0lUbzngIvIuH+wWPbcN/2rGi9 yHjQULkRHy8mNZbRQLvO9mtC0wHLiVbJycjE6vUjgHJVyQYRuaSr8DimYS9fP2n2iGxyjt pPz8gKlRCdTP4Hvewe6YcQfR0ZHU/2U= X-Rspamd-Server: rspam06 X-Rspam-User: X-Stat-Signature: imo36qah9b15umcmtqdk88kp87oxsp8x X-Rspamd-Queue-Id: 924D64016F Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=eftnJqU0; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1659987354-347712 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Saves several calls to compound_head() and removes a couple of uses of page->lru. Signed-off-by: Matthew Wilcox (Oracle) --- mm/migrate.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 3cd1fdb5f572..7b338ddd011a 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1244,11 +1244,11 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, * kicking migration. */ if (!hugepage_migration_supported(page_hstate(hpage))) { - list_move_tail(&hpage->lru, ret); + list_move_tail(&src->lru, ret); return -ENOSYS; } - if (page_count(hpage) == 1) { + if (folio_ref_count(src) == 1) { /* page was freed from under us. So we are done. */ putback_active_hugepage(hpage); return MIGRATEPAGE_SUCCESS; @@ -1259,7 +1259,7 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, return -ENOMEM; dst = page_folio(new_hpage); - if (!trylock_page(hpage)) { + if (!folio_trylock(src)) { if (!force) goto out; switch (mode) { @@ -1269,29 +1269,29 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, default: goto out; } - lock_page(hpage); + folio_lock(src); } /* * Check for pages which are in the process of being freed. Without - * page_mapping() set, hugetlbfs specific move page routine will not + * folio_mapping() set, hugetlbfs specific move page routine will not * be called and we could leak usage counts for subpools. */ - if (hugetlb_page_subpool(hpage) && !page_mapping(hpage)) { + if (hugetlb_page_subpool(hpage) && !folio_mapping(src)) { rc = -EBUSY; goto out_unlock; } - if (PageAnon(hpage)) - anon_vma = page_get_anon_vma(hpage); + if (folio_test_anon(src)) + anon_vma = page_get_anon_vma(&src->page); - if (unlikely(!trylock_page(new_hpage))) + if (unlikely(!folio_trylock(dst))) goto put_anon; - if (page_mapped(hpage)) { + if (folio_mapped(src)) { enum ttu_flags ttu = 0; - if (!PageAnon(hpage)) { + if (!folio_test_anon(src)) { /* * In shared mappings, try_to_unmap could potentially * call huge_pmd_unshare. Because of this, take @@ -1312,7 +1312,7 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, i_mmap_unlock_write(mapping); } - if (!page_mapped(hpage)) + if (!folio_mapped(src)) rc = move_to_new_folio(dst, src, mode); if (page_was_mapped) @@ -1320,7 +1320,7 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, rc == MIGRATEPAGE_SUCCESS ? dst : src, false); unlock_put_anon: - unlock_page(new_hpage); + folio_unlock(dst); put_anon: if (anon_vma) @@ -1332,12 +1332,12 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, } out_unlock: - unlock_page(hpage); + folio_unlock(src); out: if (rc == MIGRATEPAGE_SUCCESS) putback_active_hugepage(hpage); else if (rc != -EAGAIN) - list_move_tail(&hpage->lru, ret); + list_move_tail(&src->lru, ret); /* * If migration was not successful and there's a freeing callback, use From patchwork Mon Aug 8 19:34:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939086 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87D1DC00140 for ; Mon, 8 Aug 2022 19:36:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6F6A9940025; Mon, 8 Aug 2022 15:35:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 69C6F940023; Mon, 8 Aug 2022 15:35:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1E8D6940023; Mon, 8 Aug 2022 15:35:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E383C940025 for ; Mon, 8 Aug 2022 15:35:54 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B8ADA1C6449 for ; Mon, 8 Aug 2022 19:35:54 +0000 (UTC) X-FDA: 79777430628.24.C46AC16 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf30.hostedemail.com (Postfix) with ESMTP id 627D380042 for ; Mon, 8 Aug 2022 19:35:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=z8X6vYTk9MzT6MttdzTAM6ZnacOtFDqMaLC1FwKGRRA=; b=tdN6KxPnckqwQG2VJjkjthlz5j D4P+APR3PKBe+dThU5zh0BukbUnCMs4YcPeYy0cIuWP8aVTsMsVkfKjT+xtR9HjmmffLrWE6ped7X Sr/XN2GYk0n3AsR1XxgUjqmnVE3QVKXOzNBiNFd3U86A2XYu3HaAUcz5LzzWEXIJCfXF2dZmum6aH Hq2ypitHF0zHzAb+Bl8kkWp2GsmXquqKXWDN17YClrAMnUBBBGpeAkGYta1oFc9uofyvEzhCzGKsW LX9LF2wwmDIz/QgLwK1iuK1/nYmY3EXGsJFDBhpGBYyynfSbW9fuKX/FertT/JZistUp0QwJ1VeOo ZYtTbWdg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XW-00EB2y-2F; Mon, 08 Aug 2022 19:35:50 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 55/59] huge_memory: Convert split_huge_page_to_list() to use a folio Date: Mon, 8 Aug 2022 20:34:23 +0100 Message-Id: <20220808193430.3378317-56-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987354; a=rsa-sha256; cv=none; b=WrHO++stnkC4PfqPatQWK122G8/CggoUUN0gJT/woSsb7Qfej3BcA3Btpn0TELWQmQi/9u IBgUIQyOAJF+fL1QvyHJyLqIPuMlXA3zvSrDF9ryCRb1m4UOq4ExEn0iYIYR6DPDDHyThS NNdX9X7yV/vW74bqsnfZShj/wUknXr0= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=tdN6KxPn; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987354; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=z8X6vYTk9MzT6MttdzTAM6ZnacOtFDqMaLC1FwKGRRA=; b=Gx4r41dpTP4XPpECl3CCf6MXD2WdmlmvDVuaBUbAjhA8bU5QjwXsvi44yQZz577Vi/FTYA Fjb3XZEtogF9pPZPGMbko4jccF0KHLTNrJndmI2Wt0DrL3b8UoqNSpsyxtpqcvzsaK6L3g CuLa9k3gHleJwcEksH3N3RD4PMUOYRE= X-Rspamd-Queue-Id: 627D380042 Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=tdN6KxPn; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 1gcn5e8qrw4nuoxny7kysbxtkzr8h33d X-HE-Tag: 1659987354-329745 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Saves many calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/huge_memory.c | 49 ++++++++++++++++++++++++------------------------ 1 file changed, 24 insertions(+), 25 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 7b998f2083aa..431a3b7078c7 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2592,27 +2592,26 @@ bool can_split_folio(struct folio *folio, int *pextra_pins) int split_huge_page_to_list(struct page *page, struct list_head *list) { struct folio *folio = page_folio(page); - struct page *head = &folio->page; - struct deferred_split *ds_queue = get_deferred_split_queue(head); - XA_STATE(xas, &head->mapping->i_pages, head->index); + struct deferred_split *ds_queue = get_deferred_split_queue(&folio->page); + XA_STATE(xas, &folio->mapping->i_pages, folio->index); struct anon_vma *anon_vma = NULL; struct address_space *mapping = NULL; int extra_pins, ret; pgoff_t end; bool is_hzp; - VM_BUG_ON_PAGE(!PageLocked(head), head); - VM_BUG_ON_PAGE(!PageCompound(head), head); + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); - is_hzp = is_huge_zero_page(head); - VM_WARN_ON_ONCE_PAGE(is_hzp, head); + is_hzp = is_huge_zero_page(&folio->page); + VM_WARN_ON_ONCE_FOLIO(is_hzp, folio); if (is_hzp) return -EBUSY; - if (PageWriteback(head)) + if (folio_test_writeback(folio)) return -EBUSY; - if (PageAnon(head)) { + if (folio_test_anon(folio)) { /* * The caller does not necessarily hold an mmap_lock that would * prevent the anon_vma disappearing so we first we take a @@ -2621,7 +2620,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) * is taken to serialise against parallel split or collapse * operations. */ - anon_vma = page_get_anon_vma(head); + anon_vma = page_get_anon_vma(&folio->page); if (!anon_vma) { ret = -EBUSY; goto out; @@ -2630,7 +2629,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) mapping = NULL; anon_vma_lock_write(anon_vma); } else { - mapping = head->mapping; + mapping = folio->mapping; /* Truncated ? */ if (!mapping) { @@ -2638,7 +2637,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) goto out; } - xas_split_alloc(&xas, head, compound_order(head), + xas_split_alloc(&xas, folio, folio_order(folio), mapping_gfp_mask(mapping) & GFP_RECLAIM_MASK); if (xas_error(&xas)) { ret = xas_error(&xas); @@ -2653,7 +2652,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) * but on 32-bit, i_size_read() takes an irq-unsafe seqlock, * which cannot be nested inside the page tree lock. So note * end now: i_size itself may be changed at any moment, but - * head page lock is good enough to serialize the trimming. + * folio lock is good enough to serialize the trimming. */ end = DIV_ROUND_UP(i_size_read(mapping->host), PAGE_SIZE); if (shmem_mapping(mapping)) @@ -2669,38 +2668,38 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) goto out_unlock; } - unmap_page(head); + unmap_page(&folio->page); /* block interrupt reentry in xa_lock and spinlock */ local_irq_disable(); if (mapping) { /* - * Check if the head page is present in page cache. - * We assume all tail are present too, if head is there. + * Check if the folio is present in page cache. + * We assume all tail are present too, if folio is there. */ xas_lock(&xas); xas_reset(&xas); - if (xas_load(&xas) != head) + if (xas_load(&xas) != folio) goto fail; } /* Prevent deferred_split_scan() touching ->_refcount */ spin_lock(&ds_queue->split_queue_lock); - if (page_ref_freeze(head, 1 + extra_pins)) { - if (!list_empty(page_deferred_list(head))) { + if (folio_ref_freeze(folio, 1 + extra_pins)) { + if (!list_empty(page_deferred_list(&folio->page))) { ds_queue->split_queue_len--; - list_del(page_deferred_list(head)); + list_del(page_deferred_list(&folio->page)); } spin_unlock(&ds_queue->split_queue_lock); if (mapping) { - int nr = thp_nr_pages(head); + int nr = folio_nr_pages(folio); - xas_split(&xas, head, thp_order(head)); - if (PageSwapBacked(head)) { - __mod_lruvec_page_state(head, NR_SHMEM_THPS, + xas_split(&xas, folio, folio_order(folio)); + if (folio_test_swapbacked(folio)) { + __lruvec_stat_mod_folio(folio, NR_SHMEM_THPS, -nr); } else { - __mod_lruvec_page_state(head, NR_FILE_THPS, + __lruvec_stat_mod_folio(folio, NR_FILE_THPS, -nr); filemap_nr_thps_dec(mapping); } From patchwork Mon Aug 8 19:34:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939096 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C848C00140 for ; Mon, 8 Aug 2022 19:36:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 98CC38E0002; Mon, 8 Aug 2022 15:36:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9151A8E0001; Mon, 8 Aug 2022 15:36:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 78E7E8E0002; Mon, 8 Aug 2022 15:36:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 6568F8E0001 for ; Mon, 8 Aug 2022 15:36:25 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 46833160168 for ; Mon, 8 Aug 2022 19:36:25 +0000 (UTC) X-FDA: 79777431930.17.39F0EE8 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf22.hostedemail.com (Postfix) with ESMTP id EE599C012E for ; Mon, 8 Aug 2022 19:36:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=M1IcjVrPwbJ90c9OBDp6bP2D6HcUQ3yjRyZFQBHUAWE=; b=sQI6ByvPmhKRinNlyzulNxry+l c3bh5HiQz5RtvNVqidf3bK2no331JgaQuN/jRuWxI9Kd3JK48/AzvK6p9f2RumRUtS7bOuzAIVJdC x1adtKAJNoXjaisFQHzFuWnQQKZIKkZklbGrYdavwh26SieC8oYMovjp1J3QcJZknPI2eDPzPiKHv d96/yy6TRKHzvTvMFJxflTpakV+Zcbq8ZfZ1ExnoW2DxnnQrB/BbFU5vJFfTFHhedoA0YU6uBrFGS Eeh7uMOOPHbxMV82/zLXG68oItvf8wb1EaOpod2amP5JyotIKkcVD68jAel/GYBtkvT1HlbDgYf8v A812j10g==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XW-00EB3P-8l; Mon, 08 Aug 2022 19:35:53 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 56/59] huge_memory: Convert unmap_page() to unmap_folio() Date: Mon, 8 Aug 2022 20:34:24 +0100 Message-Id: <20220808193430.3378317-57-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987385; a=rsa-sha256; cv=none; b=dkB2NTNNaWWcCmhsDib7Fq5eQLGoxTXNCQD6oUytfm/vkH94ZMD/2ghVcNpdmRqaFx7hvQ kBn4Y7hLKac9lwjxphEknBxivgIWyWmuP+POF1q4wOpvXkLskQEakp9HqoNkwvU31b/7dV 0xM48aoK14enHajyZKpNfJcTfvO1I8M= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=sQI6ByvP; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987385; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=M1IcjVrPwbJ90c9OBDp6bP2D6HcUQ3yjRyZFQBHUAWE=; b=ONseVrAF6uRhXVE5f3ovN2R0ni/lv4T6wcB9khQBgbHxxFw4C5Kq5EmNw/eW9eJd3yqgoO KzLYif300kiUAvsufnNXB1FNo/YM5suVM/drg4tmXm2BODIRHiNWZSmZg9RBifVws7/t8j yeVzhuTKrVwlR37LkWKsE/bimLYPxzQ= X-Rspamd-Server: rspam06 X-Rspam-User: X-Stat-Signature: 5zxnmrfmr5ax9jw1cm7mwztwigq6d8fs X-Rspamd-Queue-Id: EE599C012E Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=sQI6ByvP; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1659987384-743539 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove a folio->page->folio conversion. Signed-off-by: Matthew Wilcox (Oracle) --- mm/huge_memory.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 431a3b7078c7..2b0f8787c7ed 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2326,13 +2326,12 @@ void vma_adjust_trans_huge(struct vm_area_struct *vma, } } -static void unmap_page(struct page *page) +static void unmap_folio(struct folio *folio) { - struct folio *folio = page_folio(page); enum ttu_flags ttu_flags = TTU_RMAP_LOCKED | TTU_SPLIT_HUGE_PMD | TTU_SYNC; - VM_BUG_ON_PAGE(!PageHead(page), page); + VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); /* * Anon pages need migration entries to preserve them, but file @@ -2349,7 +2348,7 @@ static void remap_page(struct folio *folio, unsigned long nr) { int i = 0; - /* If unmap_page() uses try_to_migrate() on file, remove this check */ + /* If unmap_folio() uses try_to_migrate() on file, remove this check */ if (!folio_test_anon(folio)) return; for (;;) { @@ -2399,7 +2398,7 @@ static void __split_huge_page_tail(struct page *head, int tail, * for example lock_page() which set PG_waiters. * * Note that for mapped sub-pages of an anonymous THP, - * PG_anon_exclusive has been cleared in unmap_page() and is stored in + * PG_anon_exclusive has been cleared in unmap_folio() and is stored in * the migration entry instead from where remap_page() will restore it. * We can still have PG_anon_exclusive set on effectively unmapped and * unreferenced sub-pages of an anonymous THP: we can simply drop @@ -2660,7 +2659,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) } /* - * Racy check if we can split the page, before unmap_page() will + * Racy check if we can split the page, before unmap_folio() will * split PMDs */ if (!can_split_folio(folio, &extra_pins)) { @@ -2668,7 +2667,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) goto out_unlock; } - unmap_page(&folio->page); + unmap_folio(folio); /* block interrupt reentry in xa_lock and spinlock */ local_irq_disable(); From patchwork Mon Aug 8 19:34:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939097 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C1CAC00140 for ; Mon, 8 Aug 2022 19:36:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DD2558E0005; Mon, 8 Aug 2022 15:36:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D5B1E8E0001; Mon, 8 Aug 2022 15:36:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD4DD8E0005; Mon, 8 Aug 2022 15:36:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A83058E0001 for ; Mon, 8 Aug 2022 15:36:28 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 8A6171C60D3 for ; Mon, 8 Aug 2022 19:36:28 +0000 (UTC) X-FDA: 79777432056.17.E161F6C Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf20.hostedemail.com (Postfix) with ESMTP id 303901C002E for ; Mon, 8 Aug 2022 19:36:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=XUv3giDWg4UmrVVtaNHEwd8Y/d+wpN+coo8NOgPtOmg=; b=U3E2UJoB4J3l4EvxwiwzpEL7gs 0Mt61DbacENBLiLwmQef6St3tSn1A/3qcdAVKNx5NpoD7f/VZwMBmTPi2yZXp75we4f4MAoVaBuQw x3nI3fXKnRYI47orPUzNVXSnHsXKLCP+yzyf+DUXuL5rGcfZt7kGYRWN9RC8XSoBqdOjFeceQXOpM 0cdB9FcQ5iBpWTSXzvvIq9AU2ojBxL+ClrFFrunPkmz5HxDyRVOH70igLyh7kPYtzNlurVFnb6lBv GKQ6zxzMUKK2auoBTPPh/+sPse/R0mMlfT1nslPDgJYZHJ3uxuWCMc7UjtrlvUYFCWWtyWyNgPtIc 7B8RAQzA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8XZ-00EB3s-9A; Mon, 08 Aug 2022 19:35:53 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 57/59] mm: Convert page_get_anon_vma() to folio_get_anon_vma() Date: Mon, 8 Aug 2022 20:34:25 +0100 Message-Id: <20220808193430.3378317-58-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=U3E2UJoB; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987388; a=rsa-sha256; cv=none; b=bCS2LMaV1w5/892CfC8h1mcNeVzmXx9WjgkC9PFJWxcHw1GP9KUZrWNercsIZLvRG0sUNn eAWgikEsQ6B9Q/rmPaSKQbdXyrNmgif3aBNBkLrGzv/KS1ipCCaMYzPtmLtl+9wWYOp+Ay 3XeiwdUzT0gH/9nJpGtnzCT9yShqPYU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987388; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XUv3giDWg4UmrVVtaNHEwd8Y/d+wpN+coo8NOgPtOmg=; b=OZj6io22a/8P8aozuPrKjvBtHfGgFVo8Dgcya3n0COcNUY5xf60pMDmK5QKro7kBWwF0bV /7bEmTnDCmluoduiUdkQhNAIySrCjb9n++ngrg+yhPs9W5oE3t50yQgzLIr0g5W8qU2cmt 91LEXrDxY1/lborVAYo8rxolRwb48No= X-Stat-Signature: gsj77pp54wb4eqi5dioi4gbbhjicz88r X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 303901C002E Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=U3E2UJoB; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1659987388-150030 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With all callers now passing in a folio, rename the function and convert all callers. Removes a couple of calls to compound_head() and a reference to page->mapping. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/rmap.h | 2 +- mm/huge_memory.c | 2 +- mm/migrate.c | 6 +++--- mm/rmap.c | 14 +++++++------- 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index bf80adca980b..4fde1cf5a5e8 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -163,7 +163,7 @@ static inline void anon_vma_merge(struct vm_area_struct *vma, unlink_anon_vmas(next); } -struct anon_vma *page_get_anon_vma(struct page *page); +struct anon_vma *folio_get_anon_vma(struct folio *folio); /* RMAP flags, currently only relevant for some anon rmap operations. */ typedef int __bitwise rmap_t; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 2b0f8787c7ed..44a843f12fb3 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2619,7 +2619,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) * is taken to serialise against parallel split or collapse * operations. */ - anon_vma = page_get_anon_vma(&folio->page); + anon_vma = folio_get_anon_vma(folio); if (!anon_vma) { ret = -EBUSY; goto out; diff --git a/mm/migrate.c b/mm/migrate.c index 7b338ddd011a..c688319ffd46 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1035,14 +1035,14 @@ static int __unmap_and_move(struct folio *src, struct folio *dst, * File Caches may use write_page() or lock_page() in migration, then, * just care Anon page here. * - * Only page_get_anon_vma() understands the subtleties of + * Only folio_get_anon_vma() understands the subtleties of * getting a hold on an anon_vma from outside one of its mms. * But if we cannot get anon_vma, then we won't need it anyway, * because that implies that the anon page is no longer mapped * (and cannot be remapped so long as we hold the page lock). */ if (folio_test_anon(src) && !folio_test_ksm(src)) - anon_vma = page_get_anon_vma(&src->page); + anon_vma = folio_get_anon_vma(src); /* * Block others from accessing the new page when we get around to @@ -1283,7 +1283,7 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, } if (folio_test_anon(src)) - anon_vma = page_get_anon_vma(&src->page); + anon_vma = folio_get_anon_vma(src); if (unlikely(!folio_trylock(dst))) goto put_anon; diff --git a/mm/rmap.c b/mm/rmap.c index aaab1c1078b4..dcb856150295 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -486,16 +486,16 @@ void __init anon_vma_init(void) * if there is a mapcount, we can dereference the anon_vma after observing * those. */ -struct anon_vma *page_get_anon_vma(struct page *page) +struct anon_vma *folio_get_anon_vma(struct folio *folio) { struct anon_vma *anon_vma = NULL; unsigned long anon_mapping; rcu_read_lock(); - anon_mapping = (unsigned long)READ_ONCE(page->mapping); + anon_mapping = (unsigned long)READ_ONCE(folio->mapping); if ((anon_mapping & PAGE_MAPPING_FLAGS) != PAGE_MAPPING_ANON) goto out; - if (!page_mapped(page)) + if (!folio_mapped(folio)) goto out; anon_vma = (struct anon_vma *) (anon_mapping - PAGE_MAPPING_ANON); @@ -505,13 +505,13 @@ struct anon_vma *page_get_anon_vma(struct page *page) } /* - * If this page is still mapped, then its anon_vma cannot have been + * If this folio is still mapped, then its anon_vma cannot have been * freed. But if it has been unmapped, we have no security against the * anon_vma structure being freed and reused (for another anon_vma: * SLAB_TYPESAFE_BY_RCU guarantees that - so the atomic_inc_not_zero() * above cannot corrupt). */ - if (!page_mapped(page)) { + if (!folio_mapped(folio)) { rcu_read_unlock(); put_anon_vma(anon_vma); return NULL; @@ -523,11 +523,11 @@ struct anon_vma *page_get_anon_vma(struct page *page) } /* - * Similar to page_get_anon_vma() except it locks the anon_vma. + * Similar to folio_get_anon_vma() except it locks the anon_vma. * * Its a little more complex as it tries to keep the fast path to a single * atomic op -- the trylock. If we fail the trylock, we fall back to getting a - * reference like with page_get_anon_vma() and then block on the mutex + * reference like with folio_get_anon_vma() and then block on the mutex * on !rwc->try_lock case. */ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, From patchwork Mon Aug 8 19:34:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939093 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C70EEC25B08 for ; Mon, 8 Aug 2022 19:36:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A52668E000C; Mon, 8 Aug 2022 15:36:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 997B08E000B; Mon, 8 Aug 2022 15:36:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 805128E000C; Mon, 8 Aug 2022 15:36:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 5E33B8E000B for ; Mon, 8 Aug 2022 15:36:07 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 43072A07C3 for ; Mon, 8 Aug 2022 19:36:07 +0000 (UTC) X-FDA: 79777431174.17.B9ACEBA Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf21.hostedemail.com (Postfix) with ESMTP id DE4611C019B for ; Mon, 8 Aug 2022 19:36:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=8WkGVJRQXE1KkIueq4OgDng9X2hBjBJEb6XOPFGsvk4=; b=Ka9i9FISo3HgeA7Lkd7UtZ7bkK 5epbltelWf08t5QPFQrWYVRJKX2zpqR7dFWo694K//Dbryl7Ycqwv7DzwQgl02BFzy1zygw4cRpvN JK5luR3et6a9JnGAZbkTIFBFP19I3LRAWBoYNKlRtB9Gdn+neYBdUyJXgUs5gdxeYMVamJONkmGBq XN3lx6GoXHs1Fb8WuFpFCYIqg+/xKdj5AmhqtWPePcA8hk9Mq2jQREMwHCP5QB5y/EGX3C0E+ciw1 RdSgccIe4KH93wowRRpXmviDxLrVrog3eqIM7f6yYft+gi1cElXUQUsBDPp7wFuxkdzhjn+3VWJKD Ckh32qCg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8Xc-00EB41-Ac; Mon, 08 Aug 2022 19:35:56 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 58/59] rmap: Remove page_unlock_anon_vma_read() Date: Mon, 8 Aug 2022 20:34:26 +0100 Message-Id: <20220808193430.3378317-59-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987366; a=rsa-sha256; cv=none; b=NQyCMmKS+y023D557KFzqm22tKnE+YE7m9REInRgD6elUwEKlg5h4kFozI7QfnOE+r+e1H /OqEr3zt2y5mg76nVzJ1+I/7/NWj7WDhNcJ2UAax0ZFzjySQuLUvGltXGwzF82uLCqwYEo PNDrYyHsuUzNDpTJsvooF6+RN/81O4E= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Ka9i9FIS; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987366; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8WkGVJRQXE1KkIueq4OgDng9X2hBjBJEb6XOPFGsvk4=; b=szrYesnreAZrdMt/+xyY+tcBRRF3ab7RWUhg69c0do9ysPl3oSvMF6iCZhu2cfgbcoC0bD OU9ZndiJiAJoLpniL+L+/L9tU8Mkea8dtFgtjQmCMxVmUVzBjeUDgU3V93eLV1/l4WSTsP I0xupy9qfkIViIJ+1ghAe6h7vcvFD2g= X-Rspamd-Queue-Id: DE4611C019B Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Ka9i9FIS; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: c3m3g6kwpc64koht7p6m5eaaa66f15sa X-HE-Tag: 1659987366-688033 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This was simply an alias for anon_vma_unlock_read() since 2011. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/rmap.h | 5 ----- mm/memory-failure.c | 2 +- mm/rmap.c | 5 ----- 3 files changed, 1 insertion(+), 11 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 4fde1cf5a5e8..1c6cc96925a4 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -402,13 +402,8 @@ struct rmap_walk_control { void rmap_walk(struct folio *folio, struct rmap_walk_control *rwc); void rmap_walk_locked(struct folio *folio, struct rmap_walk_control *rwc); - -/* - * Called by memory-failure.c to kill processes. - */ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, struct rmap_walk_control *rwc); -void page_unlock_anon_vma_read(struct anon_vma *anon_vma); #else /* !CONFIG_MMU */ diff --git a/mm/memory-failure.c b/mm/memory-failure.c index 9a7a228ad04a..794b111b327f 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -521,7 +521,7 @@ static void collect_procs_anon(struct page *page, struct list_head *to_kill, } } read_unlock(&tasklist_lock); - page_unlock_anon_vma_read(av); + anon_vma_unlock_read(av); } /* diff --git a/mm/rmap.c b/mm/rmap.c index dcb856150295..d025ad00e63c 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -599,11 +599,6 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, return anon_vma; } -void page_unlock_anon_vma_read(struct anon_vma *anon_vma) -{ - anon_vma_unlock_read(anon_vma); -} - #ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH /* * Flush TLB entries for recently unmapped pages from remote CPUs. It is From patchwork Mon Aug 8 19:34:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12939091 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 478A0C25B0C for ; Mon, 8 Aug 2022 19:36:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 56F7A94001D; Mon, 8 Aug 2022 15:36:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4CEB194000C; Mon, 8 Aug 2022 15:36:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D5F594001B; Mon, 8 Aug 2022 15:36:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0A4FD94000C for ; Mon, 8 Aug 2022 15:36:04 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D2141AB342 for ; Mon, 8 Aug 2022 19:36:03 +0000 (UTC) X-FDA: 79777431006.23.C3B8D74 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id 7A6512017B for ; Mon, 8 Aug 2022 19:35:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=OLpZzsA000SaHs8Mq1xkTy2luxKoWvALbfqtapq3Q7E=; b=grVSQRA32OAFZdUIELht4WgN9O T3sdE+VqH+4j88dUp/VhMR3Fbp4ZFuDYQrQfpz8p95CsuLh2YPpd3JHOMwqC6wphAy2js9kvQzfWf y2SrB5IV7gNevozFrGQBnnQgl8/TjXYtYLqBIXEW/kp5mNJvqN4+ERnkQ41qSxowLayCiduyOKvXx Ru5Vu6D1dfj6c9F6yqmxx+57W/ZR+87DXfUYxTUw/T9w8GZjNQYQSClA6oeQipMlIICAClGc2DHT7 D0djr6+TO5cFPZcaaBT6ogHlDqkqMgMzsaSRTM4xNNf68KyMTEwHY/eBbXC98RFKq06q8efv4Eiiu Vh69OQHQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1oL8Xc-00EB4J-Sg; Mon, 08 Aug 2022 19:35:57 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , hughd@google.com Subject: [PATCH 59/59] uprobes: Use new_folio in __replace_page() Date: Mon, 8 Aug 2022 20:34:27 +0100 Message-Id: <20220808193430.3378317-60-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20220808193430.3378317-1-willy@infradead.org> References: <20220808193430.3378317-1-willy@infradead.org> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659987358; a=rsa-sha256; cv=none; b=1+YO+zr9efjSBypkaD23M8oIAlFX5DflHy76E6J7ji1yiNowf6Fxd8hzQa5nVw07RO0OBG 3MdaP92jqk/02HZMso7dRk7KKeEBXV9A4XMaYnrr+5NbVdj+S7d8fxn2W0GLoi96RokU3C hGLf7ePfMfKtRT2Cq3MMS4mMYsB9bcw= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=grVSQRA3; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659987358; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OLpZzsA000SaHs8Mq1xkTy2luxKoWvALbfqtapq3Q7E=; b=1W0+aELxaRNLbPrfjvc5AmhaqiagNHlzAE04KL7Nxfg1f6lhNsKDceguGB4ClEEva2OHKn l1w+4d4bJAUcnoOf2eDg2yjx137TubWh/hicxTZQP6lNezreHUOZ5hmPMdd1HdL5g2dZdh rEj4Z1xtNvKrWPKkd7oXDFMO3NsPs8E= X-Rspamd-Server: rspam06 X-Rspam-User: X-Stat-Signature: oyj1h1e5rdez69frx6itprafyupyzsm8 X-Rspamd-Queue-Id: 7A6512017B Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=grVSQRA3; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none X-HE-Tag: 1659987358-894022 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Saves several calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- kernel/events/uprobes.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 9722c4587c48..7179298808c2 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -155,6 +155,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, struct page *old_page, struct page *new_page) { struct folio *old_folio = page_folio(old_page); + struct folio *new_folio; struct mm_struct *mm = vma->vm_mm; DEFINE_FOLIO_VMA_WALK(pvmw, old_folio, vma, addr, 0); int err; @@ -164,8 +165,8 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, addr + PAGE_SIZE); if (new_page) { - err = mem_cgroup_charge(page_folio(new_page), vma->vm_mm, - GFP_KERNEL); + new_folio = page_folio(new_page); + err = mem_cgroup_charge(new_folio, vma->vm_mm, GFP_KERNEL); if (err) return err; } @@ -180,9 +181,9 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, VM_BUG_ON_PAGE(addr != pvmw.address, old_page); if (new_page) { - get_page(new_page); + folio_get(new_folio); page_add_new_anon_rmap(new_page, vma, addr); - lru_cache_add_inactive_or_unevictable(new_page, vma); + folio_add_lru_vma(new_folio, vma); } else /* no new page, just dec_mm_counter for old_page */ dec_mm_counter(mm, MM_ANONPAGES);