From patchwork Thu Feb 6 07:20:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13962462 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF8ACC02196 for ; Thu, 6 Feb 2025 07:21:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 71D0D28000A; Thu, 6 Feb 2025 02:21:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6CE18280004; Thu, 6 Feb 2025 02:21:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 51FA028000A; Thu, 6 Feb 2025 02:21:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2D55F280004 for ; Thu, 6 Feb 2025 02:21:17 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id DE5B51C88C1 for ; Thu, 6 Feb 2025 07:21:16 +0000 (UTC) X-FDA: 83088673752.16.D7BAC29 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.8]) by imf08.hostedemail.com (Postfix) with ESMTP id D0562160002 for ; Thu, 6 Feb 2025 07:21:14 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=k9UjCEjT; spf=pass (imf08.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738826475; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kJtkvR6syFxv+hOQ8eJ4pefrzelQlHkLW8z1Vmr+FFg=; b=BIJ0Xjvrm1IYTpqqBTvnvf8RUThEfgIjsPulmAGeYE/+z6C3aSqP2pZNQUTaDMND2NLe+1 DPPvqDraykc0cyX48R4EDEM/NPrzlYrMCwskEWe7Lk/eZ7NlVnVcmlGpNzmWLJfqSxxLop a9/u6drAi5/rYkzgq4Z7HJihEOk5sQM= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=k9UjCEjT; spf=pass (imf08.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.8 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738826475; a=rsa-sha256; cv=none; b=NW51PvIht0UvhZ0yqVIFRWp+zzCWjc0Oaq4Lk/y+6DpTIkY+lRmbBtT3Mh2Q+QgTe6wz99 6CyY3aqFbzvJtT8EkwmyoWq1w6bNe73BtkuVy0QD6Ntfbp2BDa1jE/fiN1OSWNNCpuaBq/ pk3e40jr2nzECg7iN8pBZ4K8ZkHiWHc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1738826475; x=1770362475; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=MDL8+i/ID50qONhYxICYmIy+BnPTX+KVFghJNhMgSNU=; b=k9UjCEjT0DDDGUCgi96yIqzrdqao8RiySDqwjMkuoIBuZsElD11nxiNv PQ5Krxu3//z5MjcmmnQmxY393/XzSBCtDgq4HDUC9hjU9zC5HSZHOhUa2 zXr6gs8DyNgaL2OlIYhZP9L5I1bLSNGBQNSulmqFYZSfKbAyKX7UaK0h2 3Y/qZ1c3pq82Kko9ImFP+qxDrSwOpcqEnEgrf15cGKGDaf/ZzHE20mXOP ucZZpKMqgL2nbJYX3P879FzkS5s5anwb4Bi9OkCaQcZrI0XRZpxruOv8U 22hrfV+sVicIm98Efzav51bVlGyxkOZY3ZYb3GkThWT7Vv1abE5Btkbjb g==; X-CSE-ConnectionGUID: cazEGVrOT/axDSFKnyrlqA== X-CSE-MsgGUID: FVpNsXVyQR6I/BfvwxyPPQ== X-IronPort-AV: E=McAfee;i="6700,10204,11336"; a="56962716" X-IronPort-AV: E=Sophos;i="6.13,263,1732608000"; d="scan'208";a="56962716" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by fmvoesa102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2025 23:21:05 -0800 X-CSE-ConnectionGUID: AgAlP4i4QmilcDGyzbBDYg== X-CSE-MsgGUID: w4+5RT7XT0WkI456g626+A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.12,224,1728975600"; d="scan'208";a="112022642" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.115]) by orviesa008.jf.intel.com with ESMTP; 05 Feb 2025 23:21:05 -0800 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosry.ahmed@linux.dev, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v6 13/16] mm: zswap: Restructure & simplify zswap_store() to make it amenable for batching. Date: Wed, 5 Feb 2025 23:20:59 -0800 Message-Id: <20250206072102.29045-14-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> References: <20250206072102.29045-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: D0562160002 X-Stat-Signature: 8usenn89dxderzpurobtqqskqnxrf1af X-Rspam-User: X-HE-Tag: 1738826474-998044 X-HE-Meta: U2FsdGVkX19kA8XNM6iO4B1kyJzD1AGdfOfU3n+ooGvzGWWnZYJ/m0sv+yFqj2SlKh2AaeWvn+o2cmWLvK9FLEdx89PCn9wPYT8gasKTNz2qtqMcaiTAYJ03V406fjoNvDFrb2NqFmvKXEPCXmSp2vpHAHdPg5GhZeR/tnKmVhm0JZIZAP0Cx1jr6v6VeWFdA7/OVkJb159Esrih4ksJxPi0Swx4t/sEP25ySnQyuQgeeWsWPfPySibVmHR3QQfGLEDGBSYBNQCxFz/sEjuA1tTUvvr2iuWJRB9+uXkzYvZR5Qsa+Z2cYY0NEoY7S+HgSv01E9P6kN/9lNt1P5Vs4fd0mqtllfOi4dwMWiP11yP7WVxNem80EUlqWAHinWqQaj8cSANLQSPVxuDfGKc3xgIJn84n0l6jOoUJPvandCjwJKTYnX0BHB7JjVxxcc3CuYigugn0Po6J7V+P/6kzJuBbl2G57/J3nHjsU5BtBwNNjg175o10W4HBoP4+aKewqVbarIo94BMLwxi5lKDVbsZehPAxw0ih/+KNJKcFhjkvYAGhBzF3iaS29V+j7RFn6sunQ7l7UcQRLHj5cWjtiFnhtXl/R+aJ69kMMY1xy7/dKPXCL8YtmzlH6LgJyjE7P45Mkd3RDaxQXY4ArjoV6PRzws+ZIQdYX57X5f7ZUNeHc49EhrEJIkq/rTtY2S4zWA8bxjFJ0TRJU0uJcY1ronPtl6ENe8TBtxjNKzJD6pG7q1u7h6sIHB3aDkmnIX5nNbizsd/HRPhQVdL3b4nF5aNhiR3FKLieD+1RofZf1S/FNCx8Nmxx0XxvPH6eQXoopAWTIJAxqBryFBawpGwoQimwEqOpy/2CdEjWd6XGWz2maEDqjyw3MchTCPqTxJCgII9Tmv9wn95tYzlTGcSvIidCQporaxsViNZXJzfDCd3iCX6JGaDpy98yHSd3EtjTFbs9h7gJ1LeUwOM09bW a2eLyVj5 DX3wAPQ+L1Axw9TtjgLW998sdLx8uQXByjMSaen6C2zC3rl5CuKfuloFLhZgzm1uwCK37FWgNqKZIFcmBHpaCEOvuDOHInXHYMWuoWlzDaLKxBf4WmsHc1lFArFZLvvAoED01rWnEQWpktzeeeWlmseeZqpLvzHrVg7UTFUk1Tr61MSDH0fxxWHYsblf1Xqj1edA/BzEy9TQ9+gkN5Upm1qMC3/xm6D1dONJEZAyw7vZPZMw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch introduces zswap_store_folio() that implements all the computes done earlier in zswap_store_page() for a single-page, for all the pages in a folio. This allows us to move the loop over the folio's pages from zswap_store() to zswap_store_folio(). zswap_store_folio() starts by allocating all zswap entries required to store the folio. Next, it calls zswap_compress() for all pages in the folio. Finally, it adds the entries to the xarray and LRU, charges zswap memory and increments zswap stats. The error handling and cleanup required for all failure scenarios that can occur while storing a folio in zswap are now consolidated to a "store_folio_failed" label in zswap_store_folio(). These changes facilitate developing support for compress batching in zswap_store_folio(). Signed-off-by: Kanchana P Sridhar --- mm/zswap.c | 164 ++++++++++++++++++++++++++++++++--------------------- 1 file changed, 98 insertions(+), 66 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index dc7d1ff04b22..af682bf0f690 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1509,81 +1509,116 @@ static void shrink_worker(struct work_struct *w) * main API **********************************/ -static bool zswap_store_page(struct page *page, - struct obj_cgroup *objcg, - struct zswap_pool *pool) +/* + * Store all pages in a folio. + * + * The error handling from all failure points is consolidated to the + * "store_folio_failed" label, based on the initialization of the zswap entries' + * handles to ERR_PTR(-EINVAL) at allocation time, and the fact that the + * entry's handle is subsequently modified only upon a successful zpool_malloc() + * after the page is compressed. + */ +static bool zswap_store_folio(struct folio *folio, + struct obj_cgroup *objcg, + struct zswap_pool *pool) { - swp_entry_t page_swpentry = page_swap_entry(page); - struct zswap_entry *entry, *old; + long index, from_index = 0, nr_pages = folio_nr_pages(folio); + struct zswap_entry **entries = NULL; + int node_id = folio_nid(folio); - /* allocate entry */ - entry = zswap_entry_cache_alloc(GFP_KERNEL, page_to_nid(page)); - if (!entry) { - zswap_reject_kmemcache_fail++; + entries = kmalloc(nr_pages * sizeof(*entries), GFP_KERNEL); + if (!entries) return false; - } - if (!zswap_compress(page, entry, pool)) - goto compress_failed; + for (index = 0; index < nr_pages; ++index) { + entries[index] = zswap_entry_cache_alloc(GFP_KERNEL, node_id); - old = xa_store(swap_zswap_tree(page_swpentry), - swp_offset(page_swpentry), - entry, GFP_KERNEL); - if (xa_is_err(old)) { - int err = xa_err(old); + if (!entries[index]) { + zswap_reject_kmemcache_fail++; + nr_pages = index; + goto store_folio_failed; + } - WARN_ONCE(err != -ENOMEM, "unexpected xarray error: %d\n", err); - zswap_reject_alloc_fail++; - goto store_failed; + entries[index]->handle = (unsigned long)ERR_PTR(-EINVAL); } - /* - * We may have had an existing entry that became stale when - * the folio was redirtied and now the new version is being - * swapped out. Get rid of the old. - */ - if (old) - zswap_entry_free(old); + for (index = 0; index < nr_pages; ++index) { + struct page *page = folio_page(folio, index); - /* - * The entry is successfully compressed and stored in the tree, there is - * no further possibility of failure. Grab refs to the pool and objcg, - * charge zswap memory, and increment zswap_stored_pages. - * The opposite actions will be performed by zswap_entry_free() - * when the entry is removed from the tree. - */ - zswap_pool_get(pool); - if (objcg) { - obj_cgroup_get(objcg); - obj_cgroup_charge_zswap(objcg, entry->length); + if (!zswap_compress(page, entries[index], pool)) + goto store_folio_failed; } - atomic_long_inc(&zswap_stored_pages); - /* - * We finish initializing the entry while it's already in xarray. - * This is safe because: - * - * 1. Concurrent stores and invalidations are excluded by folio lock. - * - * 2. Writeback is excluded by the entry not being on the LRU yet. - * The publishing order matters to prevent writeback from seeing - * an incoherent entry. - */ - entry->pool = pool; - entry->swpentry = page_swpentry; - entry->objcg = objcg; - entry->referenced = true; - if (entry->length) { - INIT_LIST_HEAD(&entry->lru); - zswap_lru_add(&zswap_list_lru, entry); + for (index = 0; index < nr_pages; ++index) { + swp_entry_t page_swpentry = page_swap_entry(folio_page(folio, index)); + struct zswap_entry *old, *entry = entries[index]; + + old = xa_store(swap_zswap_tree(page_swpentry), + swp_offset(page_swpentry), + entry, GFP_KERNEL); + if (xa_is_err(old)) { + int err = xa_err(old); + + WARN_ONCE(err != -ENOMEM, "unexpected xarray error: %d\n", err); + zswap_reject_alloc_fail++; + from_index = index; + goto store_folio_failed; + } + + /* + * We may have had an existing entry that became stale when + * the folio was redirtied and now the new version is being + * swapped out. Get rid of the old. + */ + if (old) + zswap_entry_free(old); + + /* + * The entry is successfully compressed and stored in the tree, there is + * no further possibility of failure. Grab refs to the pool and objcg, + * charge zswap memory, and increment zswap_stored_pages. + * The opposite actions will be performed by zswap_entry_free() + * when the entry is removed from the tree. + */ + zswap_pool_get(pool); + if (objcg) { + obj_cgroup_get(objcg); + obj_cgroup_charge_zswap(objcg, entry->length); + } + atomic_long_inc(&zswap_stored_pages); + + /* + * We finish initializing the entry while it's already in xarray. + * This is safe because: + * + * 1. Concurrent stores and invalidations are excluded by folio lock. + * + * 2. Writeback is excluded by the entry not being on the LRU yet. + * The publishing order matters to prevent writeback from seeing + * an incoherent entry. + */ + entry->pool = pool; + entry->swpentry = page_swpentry; + entry->objcg = objcg; + entry->referenced = true; + if (entry->length) { + INIT_LIST_HEAD(&entry->lru); + zswap_lru_add(&zswap_list_lru, entry); + } } + kfree(entries); return true; -store_failed: - zpool_free(pool->zpool, entry->handle); -compress_failed: - zswap_entry_cache_free(entry); +store_folio_failed: + for (index = from_index; index < nr_pages; ++index) { + if (!IS_ERR_VALUE(entries[index]->handle)) + zpool_free(pool->zpool, entries[index]->handle); + + zswap_entry_cache_free(entries[index]); + } + + kfree(entries); return false; } @@ -1595,7 +1630,6 @@ bool zswap_store(struct folio *folio) struct mem_cgroup *memcg = NULL; struct zswap_pool *pool; bool ret = false; - long index; VM_WARN_ON_ONCE(!folio_test_locked(folio)); VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); @@ -1629,12 +1663,9 @@ bool zswap_store(struct folio *folio) mem_cgroup_put(memcg); } - for (index = 0; index < nr_pages; ++index) { - struct page *page = folio_page(folio, index); + if (!zswap_store_folio(folio, objcg, pool)) + goto put_pool; - if (!zswap_store_page(page, objcg, pool)) - goto put_pool; - } if (objcg) count_objcg_events(objcg, ZSWPOUT, nr_pages); @@ -1661,6 +1692,7 @@ bool zswap_store(struct folio *folio) pgoff_t offset = swp_offset(swp); struct zswap_entry *entry; struct xarray *tree; + long index; for (index = 0; index < nr_pages; ++index) { tree = swap_zswap_tree(swp_entry(type, offset + index));