From patchwork Thu Feb 1 15:49:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13541307 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23884C48291 for ; Thu, 1 Feb 2024 15:50:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AFCE16B0099; Thu, 1 Feb 2024 10:50:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A86E46B009A; Thu, 1 Feb 2024 10:50:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 926646B009B; Thu, 1 Feb 2024 10:50:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 7CC956B0099 for ; Thu, 1 Feb 2024 10:50:24 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 468C740DCF for ; Thu, 1 Feb 2024 15:50:24 +0000 (UTC) X-FDA: 81743671968.11.7DDAF1E Received: from out-171.mta1.migadu.com (out-171.mta1.migadu.com [95.215.58.171]) by imf29.hostedemail.com (Postfix) with ESMTP id 7C06B120025 for ; Thu, 1 Feb 2024 15:50:21 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf29.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.171 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706802621; a=rsa-sha256; cv=none; b=rM+GxTCZbnJcRbQPBhr1Q5cvbR2XBmm2mwllsEBJbh7lFsS+e6L2+VICoROyLdVkGSsXpc 4tMGY8+pCMIoOUizB4XZHp0iK1ZqjMpd5ZZt6h057VAM/zsGdJF2eFc8co930OwewXGsmx KTC80uAY0MBCJPBwVnB3gPz10wZBBuA= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf29.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.171 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706802621; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gRXNvrwRwU8rXbqOKQDS9J1LBJUthFhHZOynRdqSwrM=; b=pz7pCuze+X9xGBxkAl3syhHxqM+LCstkp9OjdvzDcubyS23IcaHEi2Xt72IVCjWXn4+4Hz 91r1u0va5SMQHpvygfu2GI2TamC3RuQMXsSfOT7t8UkqLCgnXZKZZzUsS6bsiRqjJK5DUm rHUMovfg+qqG6AGzY3jRbjCxGXofo2Y= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Thu, 01 Feb 2024 15:49:01 +0000 Subject: [PATCH 1/6] mm/zswap: add more comments in shrink_memcg_cb() MIME-Version: 1.0 Message-Id: <20240201-b4-zswap-invalidate-entry-v1-1-56ed496b6e55@bytedance.com> References: <20240201-b4-zswap-invalidate-entry-v1-0-56ed496b6e55@bytedance.com> In-Reply-To: <20240201-b4-zswap-invalidate-entry-v1-0-56ed496b6e55@bytedance.com> To: Nhat Pham , Johannes Weiner , Andrew Morton , Yosry Ahmed Cc: linux-kernel@vger.kernel.org, Yosry Ahmed , Chengming Zhou , Johannes Weiner , linux-mm@kvack.org X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 7C06B120025 X-Stat-Signature: nt1tb8kbwji9aea3zcyfxddp3k7hh5ra X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam-User: X-Rspam: Yes X-HE-Tag: 1706802621-637694 X-HE-Meta: U2FsdGVkX189vmxnWefXEE2BTXNcUbitOBF9P0aAKXCkmvSscPoKiWhKbE6+4krtDEa5MC8LUBAzkgZcftMi1rrf0LhiLdOp/gFxQLZrFVhL2RPmsWM7GwB0qIp5pcFlKHmfpPTe4ta7AkX7g3Ksyb+M886+G5dtLt3AJCxMlSI0e3+RdurM49U4V0OKpUX5cNxQINamshqJ7Zc+cGT3bW51NjBQwtVVamLaXl5ftxhdXXBjg8YaIN3fnf2Ek+Gkq5i7ENW9pX/92jWb9i7TDyCoBh2rqUlZK6nDVqPHudP6jpXzx2b5Sfc/LNcVgG5oaqPnlYYJZilAbRu5mjDUSqDqe9ZSc9kh5jr15GQRP3w6oDlE9sFnLbWefLeJkXEt3pZTUA5uzH9zsrNSuqaSfZl8Ut/9P1tgZMjxuR17ym5LGXy2u1Hf/5uEb1dwW/mYT7aEruX7xtYc/qayBksUkgrrTU7lyQqJxiWX/WmHH/H4SjuO9AbmPlLqyQzPt89Aqe212U7k/CJ1DY2bAbliQZEFsq+9APyyslOCteJu6Kmgp2TKW/3BnqMGeizJOFEAJXUUczzRfQ4Csj2kAlt3BoEeCzc7S0tF4r9tfAD6uilFq4tb0IvEgA8rWKOdU/Ug/v1axFOeaSM3AlgysxRp9d99d3XXRbIbL6FL1IQiynJsNSh/YXf76mNGrcEnETVVDDzB18sMdnM140zAFlSNprX/XFahl8wiDDR6AQ2bB3nCTiBump38nra7YgZhoKwmBQLa+DP8GfvvsK3Tjb5rV/+viwuJlW2ygWvJtZh6hFtk8ZYHEHGhTbUDpaN8LbcsjjdOisH8gmZU0HRjXvD7T8VZIaBdZQHu5cmmuGBEE6OsmImkRuuRFV4+Xanszvl0FZpOD50WPnYmOE1l6LUW0tEcG1jMij/n9eLMIFNdyo8KWh2XLQmThD5nrKdeSi6wrtdqvKgv23q0GDqSzTt N7uflCx9 VncM669EOhHCNrc4v+ecSN1agWTkJXnFPlFNSoZbU+iW7nWCNna5UfylYfgLCZMQd6TwPnluSU+R0qkecEBwQSHVxMTNGC0ai3W2ib34h436Rcwzk45SVjKCDigMeLXo9GaMLLpEPNgPBEf1hKJcwSBZDt8Hr5/DbSxvhzB1i56eqX4Q9CYfmjHOkIq5NrkU4PgxadPnWDM+b/fPYwIo4SrDv6IJaDF42mNh+VHrIFFGH+FJceXIavAq9+LsmdtwQrrYgWIIGHYiAGNC5RL0Nnz7ilVsZ4qOink91QZ7Dk4gb2kRtIqW2fQtXBYb3mR47JH3LwxhqrSKw5OGYNr88ONqHFGCoqH1en017i4udNdk8wBgiWhOGWbZJYSvQ41O0k53OBANPWt+7jc6WO1incPz2Q2Bij7Mjjs8mnXRu7ZU5ewTM3/p0KvQazxBz05QYJ46i8FbN96ecrSiSvG6U0njwmhr3gX425fxEaAxmu1JP7xQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add more comments in shrink_memcg_cb() to describe the deref dance which is implemented to fix race problem between lru writeback and swapoff, and the reason why we rotate the entry at the beginning. Also fix the stale comments in zswap_writeback_entry(), and add more comments to state that we only deref the tree after we get the swapcache reference. Suggested-by: Yosry Ahmed Suggested-by: Johannes Weiner Signed-off-by: Chengming Zhou Signed-off-by: Johannes Weiner Acked-by: Yosry Ahmed Reviewed-by: Nhat Pham --- mm/zswap.c | 43 ++++++++++++++++++++++++++----------------- 1 file changed, 26 insertions(+), 17 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 4aea03285532..735f1a6ef336 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1207,10 +1207,12 @@ static int zswap_writeback_entry(struct zswap_entry *entry, /* * folio is locked, and the swapcache is now secured against - * concurrent swapping to and from the slot. Verify that the - * swap entry hasn't been invalidated and recycled behind our - * backs (our zswap_entry reference doesn't prevent that), to - * avoid overwriting a new swap folio with old compressed data. + * concurrent swapping to and from the slot, and concurrent + * swapoff so we can safely dereference the zswap tree here. + * Verify that the swap entry hasn't been invalidated and recycled + * behind our backs, to avoid overwriting a new swap folio with + * old compressed data. Only when this is successful can the entry + * be dereferenced. */ tree = swap_zswap_tree(swpentry); spin_lock(&tree->lock); @@ -1263,22 +1265,29 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o int writeback_result; /* - * Rotate the entry to the tail before unlocking the LRU, - * so that in case of an invalidation race concurrent - * reclaimers don't waste their time on it. + * As soon as we drop the LRU lock, the entry can be freed by + * a concurrent invalidation. This means the following: * - * If writeback succeeds, or failure is due to the entry - * being invalidated by the swap subsystem, the invalidation - * will unlink and free it. + * 1. We extract the swp_entry_t to the stack, allowing + * zswap_writeback_entry() to pin the swap entry and + * then validate the zwap entry against that swap entry's + * tree using pointer value comparison. Only when that + * is successful can the entry be dereferenced. * - * Temporary failures, where the same entry should be tried - * again immediately, almost never happen for this shrinker. - * We don't do any trylocking; -ENOMEM comes closest, - * but that's extremely rare and doesn't happen spuriously - * either. Don't bother distinguishing this case. + * 2. Usually, objects are taken off the LRU for reclaim. In + * this case this isn't possible, because if reclaim fails + * for whatever reason, we have no means of knowing if the + * entry is alive to put it back on the LRU. * - * But since they do exist in theory, the entry cannot just - * be unlinked, or we could leak it. Hence, rotate. + * So rotate it before dropping the lock. If the entry is + * written back or invalidated, the free path will unlink + * it. For failures, rotation is the right thing as well. + * + * Temporary failures, where the same entry should be tried + * again immediately, almost never happen for this shrinker. + * We don't do any trylocking; -ENOMEM comes closest, + * but that's extremely rare and doesn't happen spuriously + * either. Don't bother distinguishing this case. */ list_move_tail(item, &l->list); From patchwork Thu Feb 1 15:49:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13541308 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D470C4828F for ; Thu, 1 Feb 2024 15:50:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C353E6B009A; Thu, 1 Feb 2024 10:50:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B70336B009B; Thu, 1 Feb 2024 10:50:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A11506B009C; Thu, 1 Feb 2024 10:50:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8B2076B009A for ; Thu, 1 Feb 2024 10:50:26 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 634761409E3 for ; Thu, 1 Feb 2024 15:50:26 +0000 (UTC) X-FDA: 81743672052.28.451D919 Received: from out-175.mta1.migadu.com (out-175.mta1.migadu.com [95.215.58.175]) by imf22.hostedemail.com (Postfix) with ESMTP id 89C04C0024 for ; Thu, 1 Feb 2024 15:50:24 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=none; spf=pass (imf22.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.175 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706802624; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KvE903wmurJBYqeBdkfmIVsoeRCjMmKspY25ZMwQskg=; b=Prpg00mQdBQkHE8QH2gj4b9MokNL2kTvPvysrPM2/R0dolBeMrtddLdA2fmTUgGSVwMZIW 6nEL+aPj1JPLU/jfOq0Y8q+yTOiXCyYa5QXG3ps9i6t9ttyKnZC6nQd5P0JDzeghCL7C6h CiZEJ8e1Yw+RI6188mx+5D75Q7tGWGc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706802624; a=rsa-sha256; cv=none; b=fwtOhN184WpldTkSMbzFDpEjrt0ECvOn9h/7/yj9PcePvuDU/LqFBlrgM7bRa6J0ZNjhAa QlRMuw1B/GZdz73NhxNikYtjePXL3IvFmP6VL+E83l8AgC7jccJNVZ7TlJDaBciS7U5sbm vMZEjpVtIqfRj6P0rlv81GZuDuP7M5Y= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=none; spf=pass (imf22.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.175 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Thu, 01 Feb 2024 15:49:02 +0000 Subject: [PATCH 2/6] mm/zswap: invalidate zswap entry when swap entry free MIME-Version: 1.0 Message-Id: <20240201-b4-zswap-invalidate-entry-v1-2-56ed496b6e55@bytedance.com> References: <20240201-b4-zswap-invalidate-entry-v1-0-56ed496b6e55@bytedance.com> In-Reply-To: <20240201-b4-zswap-invalidate-entry-v1-0-56ed496b6e55@bytedance.com> To: Nhat Pham , Johannes Weiner , Andrew Morton , Yosry Ahmed Cc: linux-kernel@vger.kernel.org, Yosry Ahmed , Chengming Zhou , Johannes Weiner , linux-mm@kvack.org X-Migadu-Flow: FLOW_OUT X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam-User: X-Rspamd-Queue-Id: 89C04C0024 X-Rspamd-Server: rspam11 X-Stat-Signature: fj5fsij66hm5yfpsbfwge7tfinud9ukk X-Rspam: Yes X-HE-Tag: 1706802624-327872 X-HE-Meta: U2FsdGVkX19HMPdrSSfbvPrOg8HMVIqM3sq942MdDZw8vNq0xE9k8Wv1vhZHV630yBLrJu+w7XUE50qlKCc1ZunN9ucMPYp8ByWRYqw7zFh0SvzQ20Po2I2xrCUvSD/SKx29ElgdKa0IWyJ0IBy50dC0h3dY4mr6HDS2ISoNIhaJq1EDVDDz47sNZJ95wFBjkMD6hV+XzD3qvNYqubV+or2U5XOSUFH1HJTwvUvo0B4JSWDU3XXXEMq7Tx7TC71Z4dBFKTq0RjvhCWBizrgBIjg64aIgkCIlJvwKknqK2IfxTpxc9Au5hllIBN9ceXj6Kw0QTsGeFvHhFZ8qvRSFVevUvDIt3gqigUoQt7VgqvTgx8afPVTILQh809LJhUb51pAGHYdXDFE3Kic2jlwImiYzI4pUkwi8NkZsG12iEaUl2oAo+KpJeVAlSIi3jhFKBLe5GzfA1pzk1raupZ0Lhnvp/6aOFawGgQR84ZUe3kEZuYoXlTXyoZWymPi7s2ji9Hf2IZr+xhoeJ/MPoxLNtAXy3W4GuLMyt+z28IQwxw6neJEsuE3IbOOuiOVbokoQ1JwjYMv/dLgw61xt7s90wjmBlO42yBaQh+4GJUXVIQw2hJjnvZgTEYJ37RAtgfW2oIk7E/FeNoVUmyATJ7p7FfxEirhw2yBdSO4q8oTL5ByTEVS5M/gkxQtbXKE5ncLFdfEi31uthmShNrzVo7DzoRRmnzL53DIxBdzsaCmKnJ0eAfsre6ixluhfjb+OpAroXFH2hY5XB/kCEMkQgcxtu4RpIcFAXEXVA2sBU45y8RYZ7wi4Zh5Lc0GQNcZ7AxYDAgbJi+v//CMfhqxrpQz8eGxQuo525VZTU3hJcF7DWBFb3p39CKTq4PBQGd8QnQgIPwfo99iAG7m0195VLTY1vGKxiWCIYurGD+INihgBwxZNNpxZvbLN1VBQMJ8GIQ2d00P/nEpUxQ47Eq0twnA tNPfeQkL CQazUMun2PpkRfBBfuhmsNqOWgQ3eCYguZiUiMEoUH9ADVtesJiI6hLJDRBFoTURh18Bp6QMWpRXy/Ossjx+6lXfNYbFizFmYfp7BXrYtxJ0yY18xyden6JxI3e6C+5IX2fQrMhxIX2KxgRWzqMFTd01leeUmnlVpb5XBJiD5R9cMjsx3sarC870IEGYT9lnqbxsE7tQyIpf1GSx8Pga4MD17x+5F/iYXPettELocrC2a5zfsjFaOlA5N4WwM+ujXYwmKKqulbeK6rBSi38JmVYHnE2nhc0yIlzbgo+vicEtcmZ9TwmH+3Pgts0JKf8FzQYmYpFC1WWTTnw99p6xRifs7QfxeRp2rQikg1CWsAkEBxrg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: During testing I found there are some times the zswap_writeback_entry() return -ENOMEM, which is not we expected: bpftrace -e 'kr:zswap_writeback_entry {@[(int32)retval]=count()}' @[-12]: 1563 @[0]: 277221 The reason is that __read_swap_cache_async() return NULL because swapcache_prepare() failed. The reason is that we won't invalidate zswap entry when swap entry freed to the per-cpu pool, these zswap entries are still on the zswap tree and lru list. This patch moves the invalidation ahead to when swap entry freed to the per-cpu pool, since there is no any benefit to leave trashy zswap entry on the tree and lru list. With this patch: bpftrace -e 'kr:zswap_writeback_entry {@[(int32)retval]=count()}' @[0]: 259744 Note: large folio can't have zswap entry for now, so don't bother to add zswap entry invalidation in the large folio swap free path. Signed-off-by: Chengming Zhou Acked-by: Johannes Weiner Reviewed-by: Nhat Pham --- include/linux/zswap.h | 4 ++-- mm/swap_slots.c | 2 ++ mm/swapfile.c | 1 - mm/zswap.c | 5 +++-- 4 files changed, 7 insertions(+), 5 deletions(-) diff --git a/include/linux/zswap.h b/include/linux/zswap.h index 91895ce1fdbc..341aea490070 100644 --- a/include/linux/zswap.h +++ b/include/linux/zswap.h @@ -29,7 +29,7 @@ struct zswap_lruvec_state { bool zswap_store(struct folio *folio); bool zswap_load(struct folio *folio); -void zswap_invalidate(int type, pgoff_t offset); +void zswap_invalidate(swp_entry_t swp); int zswap_swapon(int type, unsigned long nr_pages); void zswap_swapoff(int type); void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg); @@ -50,7 +50,7 @@ static inline bool zswap_load(struct folio *folio) return false; } -static inline void zswap_invalidate(int type, pgoff_t offset) {} +static inline void zswap_invalidate(swp_entry_t swp) {} static inline int zswap_swapon(int type, unsigned long nr_pages) { return 0; diff --git a/mm/swap_slots.c b/mm/swap_slots.c index 0bec1f705f8e..d24cdea26daa 100644 --- a/mm/swap_slots.c +++ b/mm/swap_slots.c @@ -273,6 +273,8 @@ void free_swap_slot(swp_entry_t entry) { struct swap_slots_cache *cache; + zswap_invalidate(entry); + cache = raw_cpu_ptr(&swp_slots); if (likely(use_swap_slot_cache && cache->slots_ret)) { spin_lock_irq(&cache->free_lock); diff --git a/mm/swapfile.c b/mm/swapfile.c index 0580bb3e34d7..65b49db89b36 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -744,7 +744,6 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset, swap_slot_free_notify = NULL; while (offset <= end) { arch_swap_invalidate_page(si->type, offset); - zswap_invalidate(si->type, offset); if (swap_slot_free_notify) swap_slot_free_notify(si->bdev, offset); offset++; diff --git a/mm/zswap.c b/mm/zswap.c index 735f1a6ef336..d8bb0e06e2b0 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1738,9 +1738,10 @@ bool zswap_load(struct folio *folio) return true; } -void zswap_invalidate(int type, pgoff_t offset) +void zswap_invalidate(swp_entry_t swp) { - struct zswap_tree *tree = swap_zswap_tree(swp_entry(type, offset)); + pgoff_t offset = swp_offset(swp); + struct zswap_tree *tree = swap_zswap_tree(swp); struct zswap_entry *entry; spin_lock(&tree->lock); From patchwork Thu Feb 1 15:49:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13541309 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4AC42C4828D for ; Thu, 1 Feb 2024 15:50:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C0E306B009B; Thu, 1 Feb 2024 10:50:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B96D76B009C; Thu, 1 Feb 2024 10:50:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A379F6B009D; Thu, 1 Feb 2024 10:50:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8E9E36B009B for ; Thu, 1 Feb 2024 10:50:30 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 240E9A1B7E for ; Thu, 1 Feb 2024 15:50:30 +0000 (UTC) X-FDA: 81743672220.19.8A80932 Received: from out-182.mta1.migadu.com (out-182.mta1.migadu.com [95.215.58.182]) by imf27.hostedemail.com (Postfix) with ESMTP id 408814000F for ; Thu, 1 Feb 2024 15:50:28 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=none; spf=pass (imf27.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.182 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706802628; a=rsa-sha256; cv=none; b=VJrWKffzmfgIYjMYqn8Wai+2W4Eg3BwwC+Qzf/hjSryeu/Www7NrUvBJm4n6ILUwVZSOWb 27qYNlshM9e5A3lbHG7S/gD/DqBYdFtE7jGb07rwjQcBsvUMaamx2s2jfE8gEGQQYyB1VI WPtNdd5aF1yaWdW3mr4aheO1iymOC5A= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=none; spf=pass (imf27.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.182 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706802628; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ChF6yAyrxR93Q+ibVPgtMoofnZ58v2EsFurbhlA0kyE=; b=W1M61InrmgsFWCiuRIsTJMRLsLlgJsmakKa+TsMfgA2Zug34fkdFvNljDMVy0CkdkYAc+s yjqu77nX+FOoKoiwyC+w/fOZcpUF8wlZXd2V3V2SgvmKgEKTG+ukmdj8h70E5i3xemJ5YU qIKovwgr6kuWoY/wAEw9ne8YxipOH5s= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Thu, 01 Feb 2024 15:49:03 +0000 Subject: [PATCH 3/6] mm/zswap: stop lru list shrinking when encounter warm region MIME-Version: 1.0 Message-Id: <20240201-b4-zswap-invalidate-entry-v1-3-56ed496b6e55@bytedance.com> References: <20240201-b4-zswap-invalidate-entry-v1-0-56ed496b6e55@bytedance.com> In-Reply-To: <20240201-b4-zswap-invalidate-entry-v1-0-56ed496b6e55@bytedance.com> To: Nhat Pham , Johannes Weiner , Andrew Morton , Yosry Ahmed Cc: linux-kernel@vger.kernel.org, Yosry Ahmed , Chengming Zhou , Johannes Weiner , linux-mm@kvack.org X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 408814000F X-Stat-Signature: wqjwuc4w6hed4t1yz51gmu9xp57nx5oc X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam: Yes X-HE-Tag: 1706802628-805623 X-HE-Meta: U2FsdGVkX19r0efKgd9MEtr2HhoqSDZZ3uqKIUcnx4abEjWYMx4NPoQayR1YLO4J44Z2QllhVM0siO/TFDBwpDKHaZUREBCj7xhlUIbdk5BhZl+h2Is5LQzv7gicecd/SnqxKj+IGOpQVz10sq7NhMI67lZBVfZZOId7Rst/4ZzaH1Rcn7w7ucs+sL106eHxcNDz3XzQjrsL2DRrsUyxqmtamvykFh3hWhar0HOYPH/wrBTJhCPa3Yp6X7/WKwokY51WPfYVKowfIwD1Ey3h+RZmpXeKN0Z5mwVSCxIRt2/riuQEzaD5SksjT74jX+7dgrQI+mnu6AzrJdwLfE4gupIKkTjzt75YdreABolSEhxXELdMH3fUDcNLBD0Gabo8Pe7uUNNSd4y9H0L+tgIVv9BUlcFx3/5buJmyD6a6NAQGeT+KXUpUCBP/fj9Gi0Nj37Rq6uByzi0nzS3tML99Z6TVBfE7qemvGIAN0gWrJl0UFRivmgh1GLwCeZGEqvEC0Hj5Aos+Tw/Vi8osBlbSSotDZ5q8a8bIvzRdCIZDuM1voOtCP3sWwinmTuIp20yO/+Aw305am4APXTxoIGScqQgEkMYgwVprDOizHimv9LgKV/58dq7a82jI9LhB/DKDEXhyR3UYksehk1VRyA+CCtDWqzz+bfxfYvP/SOmRG1EWJ95a2VgMsffVP+HWjzcZLbBE9MfmiNXO0uY4cnR2OqxKtd+e3UQTC6M23Gqm2k32eoBLUt0vZgKBR9hDVB2r4F+xxxFChoh3XbicpvBh4BT9hcBdROtUd4yOy/+MqUxn6SScXRxV7YdP5h3PlzjTrLf3X4ncNSLxSUIo32SCSNPImozNLOPAiH2QczVMdo28klrN12tH+6ePAICifc6VTMqhT9/xyMckkS1Y7hHUMw0Y3gpiFPNguCkIKpX+Xvkub1TW5ybi3fRVyH79iR90WWalOJgd3lk88GmPjsE WSU/LAvG XLTCQ6i/59kJ1Ti8WNuqbj6iV27gnHNiUErcFY4/WwcIcZzcADR36ZDTHkVtwIFyI8AXWWnShUVrKOIYKK2waILBtMGkVkuupKcoeWuVJijjPNILvvgmSQvDouJOKaEHeyawc7OZnhFgckhadZelQTkkcangJegwlje3PFvtvN5Fz9LH6GsuFT976LHYdDTK3O5eTgmcP3ZVXVbuaso//Uz3Bavuyv5qLy/mS1IW3hp8S54ec6KW/MNg5o5vX7MGQKxmGnt0TvtKBKziIU3hdRUgPa4Cm2PDy46C9Gtgi1mXuuTFollB5rGCeWWA0q1EN60JFk0P0N/sb+x6spx4ry2sn/tUgH3DisDms X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When the shrinker encounter an existing folio in swap cache, it means we are shrinking into the warmer region. We should terminate shrinking if we're in the dynamic shrinker context. This patch add LRU_STOP to support this, to avoid overshrinking. Signed-off-by: Chengming Zhou Acked-by: Johannes Weiner Acked-by: Nhat Pham Reviewed-by: Yosry Ahmed --- include/linux/list_lru.h | 1 + mm/list_lru.c | 3 +++ mm/zswap.c | 4 +++- 3 files changed, 7 insertions(+), 1 deletion(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index f2882a820690..5633e970144b 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -24,6 +24,7 @@ enum lru_status { LRU_SKIP, /* item cannot be locked, skip */ LRU_RETRY, /* item not freeable. May drop the lock internally, but has to return locked. */ + LRU_STOP, /* stop lru list walking */ }; struct list_lru_one { diff --git a/mm/list_lru.c b/mm/list_lru.c index 61f3b6b1134f..3fd64736bc45 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -243,6 +243,9 @@ __list_lru_walk_one(struct list_lru *lru, int nid, int memcg_idx, */ assert_spin_locked(&nlru->lock); goto restart; + case LRU_STOP: + assert_spin_locked(&nlru->lock); + goto out; default: BUG(); } diff --git a/mm/zswap.c b/mm/zswap.c index d8bb0e06e2b0..4381b7a2d4d6 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1315,8 +1315,10 @@ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_o * into the warmer region. We should terminate shrinking (if we're in the dynamic * shrinker context). */ - if (writeback_result == -EEXIST && encountered_page_in_swapcache) + if (writeback_result == -EEXIST && encountered_page_in_swapcache) { + ret = LRU_STOP; *encountered_page_in_swapcache = true; + } } else { zswap_written_back_pages++; } From patchwork Thu Feb 1 15:49:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13541310 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA8B9C4828D for ; Thu, 1 Feb 2024 15:50:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 530B16B009C; Thu, 1 Feb 2024 10:50:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4B9056B009D; Thu, 1 Feb 2024 10:50:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 35A0E6B009E; Thu, 1 Feb 2024 10:50:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 26D7E6B009C for ; Thu, 1 Feb 2024 10:50:36 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id DEC261205B9 for ; Thu, 1 Feb 2024 15:50:35 +0000 (UTC) X-FDA: 81743672430.18.02AD9C6 Received: from out-171.mta1.migadu.com (out-171.mta1.migadu.com [95.215.58.171]) by imf08.hostedemail.com (Postfix) with ESMTP id 06414160017 for ; Thu, 1 Feb 2024 15:50:33 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=none; spf=pass (imf08.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.171 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706802634; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=I6+sO9JMMA3RtqRqlu5dPcLpUdnxswT8oHMiHI0DBBE=; b=bVGF4n/tg9uzSOMf6O4m6Ykcs8vmEKNkSGLjlckWWM/YAs7PeEf6B7q+ojwOFyaSvr6eYd kffpmypT/hLB99+up2ay9SGtfMT5SqdGP1uLWEAKForYJdFjketMYoPxkHM5kpWFpM9D8p /tgenjSTCOnr92U0/buXcN4RfSmCKOQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706802634; a=rsa-sha256; cv=none; b=y4G7nJ5SW4yDlg6WeKgJYNY45YIQNt/1cWX7ghOVRCdcm3tlk6lc1/0GvLQBPLYWFGwSxO jJutRrqBVEo9qjghk47rRCVM15d0MVyriOE2cIPsiWr/yNFvkmkik5VUaO25vVI5oxTW8g GXN9draAh+3Jnfaq69sWxLyzdVtta64= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=none; spf=pass (imf08.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.171 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Thu, 01 Feb 2024 15:49:04 +0000 Subject: [PATCH 4/6] mm/zswap: remove duplicate_entry debug value MIME-Version: 1.0 Message-Id: <20240201-b4-zswap-invalidate-entry-v1-4-56ed496b6e55@bytedance.com> References: <20240201-b4-zswap-invalidate-entry-v1-0-56ed496b6e55@bytedance.com> In-Reply-To: <20240201-b4-zswap-invalidate-entry-v1-0-56ed496b6e55@bytedance.com> To: Nhat Pham , Johannes Weiner , Andrew Morton , Yosry Ahmed Cc: linux-kernel@vger.kernel.org, Yosry Ahmed , Chengming Zhou , Johannes Weiner , linux-mm@kvack.org X-Migadu-Flow: FLOW_OUT X-Stat-Signature: ehq1x9i349tgopjo74j986iyian87d8u X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 06414160017 X-Rspam-User: X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam: Yes X-HE-Tag: 1706802633-595577 X-HE-Meta: U2FsdGVkX19BOsRZ7m7NcRUey1GSYOZbYT80QKIw5vTKO0adityILdt5lY6fPU7EbzQYC4/fDGe+qBUZmdw7hl09Cgezm5EJUxgGKo2ciMsb4BiD9ukEzOfaUbtPlPBDPyEMYmytUDlo+Yb8su560v/4hFgJShVEMkPQlV6ivix4cr1/0H2gI2hY9TDCHgnmZHquOVSfMhQ9W4pYMD2ixSqksMilc6Mcnlu/IzSKefW67uFhPzsHM0yb19BAuVFeVFxoUVLkvDjydiz87OAl6T3EG54jGUsejGQdZ7+OAj64GoAPVAL5wfyIqHAGQLMPKe15/IP2tJr/Kx4AHA6eYmKX5e8ynOitEwD/A//H3ODG71CmQnzJHUpxiHK4A60ZB10T2X+2+f0HF0wTgyXqjTOYvOpBnl8JDEZriSwL5ppUi5E4i4vQcacz0ybHLOJVpYfklZWSFUhARkps29DLOIqgvg3kLAuQ5XpwdKgWEH1DEwK2WutKWS9EQg8NGc/SaTd8wByG7m2bgyCcfgvpZmPGdlJLA/HlFp05vKISoY/GD8FPHIQZcvftJFKJDc2+4nl9IEp9nTpKiq9gpR1Q13oPvNjOxGz5+cJSYDKNFxvd4E+2xWq/gJldSieyHjr7CZ0lzPVAT0NvWRL5nn4mT5tatoc2DUl2s3ln1653dgJnr/uDi3i2kUI5GCosU47O0SiJfRURkJyAAFVq/D+a1HUeDl4SSUHVhz2i3bx5KQOUp0wYoVBSZpm1DmYV0Ur2iykECceVh/BIS5fRoct4CD1YvzewWcRLxt1euvjkN+yihxYbX0QQYWZyuQhyRDeqW5tT3ltE53qyAf47+6wa9XGgXqH8VvwBAfOo64Jei6LEpMAjBlMGb6p6jSyLELZLge08Hd0BkPYlaJGLjkJJ3Ph2J1w6rKHvLDgDFQmQUKHTWhCG6/fAAV8mpt0kwFj0tgsBTp79KXHu7dI4Noz ++z4Zncs wnpDVcOdCge8FAIJxVDyl1Zbwx4KG8wLXR1EHzFgWKSQRVeEu/pam1I3H68gGcRARiteNlj6KKLD4tL/MMWajPz5Ee2FrREvr1xmBD4wrh6pqGW1W9G99knhSPxfFgrkm56c+WuxcfcI09cDbArSc5iayuTzvAqQ9h2xNOqxfHgjtkuzkdc2YFxHOLVP3j5+ldvhpNGASnB/XIRGt/mqatLpo7pVVvAOKfy4g8qe30hd45scfVFugJ/Rzt7E0GBM6/p0VphgRpE5wB8/hld/IZ4SeFXfZ3rm+UhglwZFgrpxOFkqmM20JIvC53+NLHbkVhs98t/bzqZMRdYldNfeHDIVOW89oEAZeGym77lm9zaGYSm2wg8S3VR67E0yEikaG03/nirQrrJwIDYeYKH17ejJsreYoJ+U5kFVq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: cat /sys/kernel/debug/zswap/duplicate_entry 2086447 When testing, the duplicate_entry value is very high, but no warning message in the kernel log. From the comment of duplicate_entry "Duplicate store was encountered (rare)", it seems something goes wrong. Actually it's incremented in the beginning of zswap_store(), which found its zswap entry has already on the tree. And this is a normal case, since the folio could leave zswap entry on the tree after swapin, later it's dirtied and swapout/zswap_store again, found its original zswap entry. (Maybe we can reuse it instead of invalidating it?) So duplicate_entry should be only incremented in the real bug case, which already have "WARN_ON(1)", it looks redundant to count bug case, so this patch just remove it. Signed-off-by: Chengming Zhou Acked-by: Johannes Weiner Acked-by: Yosry Ahmed Reviewed-by: Nhat Pham --- mm/zswap.c | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 4381b7a2d4d6..3fbb7e2c8b8d 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -71,8 +71,6 @@ static u64 zswap_reject_compress_poor; static u64 zswap_reject_alloc_fail; /* Store failed because the entry metadata could not be allocated (rare) */ static u64 zswap_reject_kmemcache_fail; -/* Duplicate store was encountered (rare) */ -static u64 zswap_duplicate_entry; /* Shrinker work queue */ static struct workqueue_struct *shrink_wq; @@ -1571,10 +1569,8 @@ bool zswap_store(struct folio *folio) */ spin_lock(&tree->lock); entry = zswap_rb_search(&tree->rbroot, offset); - if (entry) { + if (entry) zswap_invalidate_entry(tree, entry); - zswap_duplicate_entry++; - } spin_unlock(&tree->lock); objcg = get_obj_cgroup_from_folio(folio); if (objcg && !obj_cgroup_may_zswap(objcg)) { @@ -1661,7 +1657,6 @@ bool zswap_store(struct folio *folio) */ while (zswap_rb_insert(&tree->rbroot, entry, &dupentry) == -EEXIST) { WARN_ON(1); - zswap_duplicate_entry++; zswap_invalidate_entry(tree, dupentry); } if (entry->length) { @@ -1822,8 +1817,6 @@ static int zswap_debugfs_init(void) zswap_debugfs_root, &zswap_reject_compress_poor); debugfs_create_u64("written_back_pages", 0444, zswap_debugfs_root, &zswap_written_back_pages); - debugfs_create_u64("duplicate_entry", 0444, - zswap_debugfs_root, &zswap_duplicate_entry); debugfs_create_u64("pool_total_size", 0444, zswap_debugfs_root, &zswap_pool_total_size); debugfs_create_atomic_t("stored_pages", 0444, From patchwork Thu Feb 1 15:49:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13541311 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3A8AC4828D for ; Thu, 1 Feb 2024 15:50:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7EF256B009D; Thu, 1 Feb 2024 10:50:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 79F2E6B009E; Thu, 1 Feb 2024 10:50:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 63FE96B009F; Thu, 1 Feb 2024 10:50:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 520BC6B009D for ; Thu, 1 Feb 2024 10:50:40 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 0FFC0120999 for ; Thu, 1 Feb 2024 15:50:40 +0000 (UTC) X-FDA: 81743672640.06.D3DA2A8 Received: from out-177.mta1.migadu.com (out-177.mta1.migadu.com [95.215.58.177]) by imf13.hostedemail.com (Postfix) with ESMTP id 2B26620008 for ; Thu, 1 Feb 2024 15:50:37 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf13.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.177 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706802638; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gvHyoQn3NFb3I4D2AD0ZUO5HMDP07d8j0a01bzSQah8=; b=JtaRokmoxBAhsUVgMGBnLSaPYL3y7r2zojrvZO220JTjLf2Dx3f5IkuFhibWsFiiMP8xiR /J2iciWHQyApvAyCmwmRrKJueHInkPuqZ5VbAtgHAIo4YecneKco5kQePIEyp2oG0c5VaK 0FkPCLs65L6bUqb6pmqPGq3wiNkBEew= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine); spf=pass (imf13.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.177 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706802638; a=rsa-sha256; cv=none; b=hPg9g8YfKazKfWSN+J6UYN/h4Hphn5rZBuybXPsaW7cYWwzTmLlEGRUQMjaKKy8VxBcInG SfY2nKSxNn9kEqxdNhs8uiNdbzqC4V0Q7dh83oi4uAC1yeQILIS660Gb+tyn0/sm5PB2ih eskMD5WXVELiIRwdvprydXRPY9RmPaU= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Thu, 01 Feb 2024 15:49:05 +0000 Subject: [PATCH 5/6] mm/zswap: only support zswap_exclusive_loads_enabled MIME-Version: 1.0 Message-Id: <20240201-b4-zswap-invalidate-entry-v1-5-56ed496b6e55@bytedance.com> References: <20240201-b4-zswap-invalidate-entry-v1-0-56ed496b6e55@bytedance.com> In-Reply-To: <20240201-b4-zswap-invalidate-entry-v1-0-56ed496b6e55@bytedance.com> To: Nhat Pham , Johannes Weiner , Andrew Morton , Yosry Ahmed Cc: linux-kernel@vger.kernel.org, Yosry Ahmed , Chengming Zhou , Johannes Weiner , linux-mm@kvack.org X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Queue-Id: 2B26620008 X-Rspamd-Server: rspam12 X-Stat-Signature: 4bbk61mhm9p9cki54n1ownmmk69m7f6c X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam: Yes X-HE-Tag: 1706802637-878207 X-HE-Meta: U2FsdGVkX1834SaVPdWRIoTg8klRDg9uqMGkruJT8Iy8A/B0bTy6TluXwJtowC9dhR7VmdPpQtBdOo/QwSXegnr3YjjmCZFeHuU+ASG2/ao4R/m9QVKO2nyRw1mIht76e1dFI4ikZh6Nl0hDMHyyb3uKtzG5DGZNffrO4ghJhfYLHW+tpIPLGFqAUreQ/Lk0JwJCaYD/6JJqlcVloZSvRFQ67Sq85pAnWUhVDi5tnvYTTWGf7kg+kvc2vLuTu8yNG1Wdf5AqLE2v23tGPC4rxSTIeZyrQkf+SM0Jg+0ambKiAh5wAKCN3MmYl82Mdjrbq/X5IdUAzROZFdKGokFKNJI90CFhABVNh+y16zPtDPb24izXFffsF3uiya0G/1ZP4yM41du60IRbSL39lGuGX5lF10uup5N/t6JdO1lLQ+EnW37Vij6APRWrmZbSU71WYEsY7EgglRgEgtCmyIS1kL/qQd0Kkl0tUcp9FfTPKd1WfnIAfLFzZj44Sl61drKJ+iQm6A+BFGSTki0jDuepBKSHMoez2ByUsiX7rLqeIoEecF93oTNYjLhES/2AuE2Sjk2WJb8uM4b82taQ6I605kI/o84HWqG1hRSOG/Tl93fad6SKEBoBd02dZ+6yhHeLxGGhdZ0lM1lmlLb5YYNic3UiTtJM7UHhAwnm2uR683A8YyouNPYTdwGVDGbyj36WbR31yvXw0qf8987cJUU3QRRD0yb+1gB8G/qHR49WNnqDhczMZXfPGzcs7z2X5eyvxjSsHnaYhf8v4hGmXt4raEalCCy1YEL8x1oPuItcOYi5i6kONch82STKCz3G8vO2MXusdQppu2x0O9S4+4ljU8wkkG+ErSvu2XbJPgVMj4yidBaqtRuVe4TCVEwUinSeiuZ/uC9MolU1I8z8XxBo8RJldUK9UbQmum7ADU+rbFpaTM3c8FbUmuxdo8spxYPVHhgx6EhWPSBecbK9amh E3R5OPCo egqypdUQfDjbk5AY5PHSm45XgYXJI7EEtOjtMN7jV0yk9M9CqJOz2YLsnL1bnLAETD6Q8cd9EWtPSV/8Z/u6tpkuv2hACxKlPTarH2yLKsOW1jmMC8x3vH4fPvzuPVBCRsDMsTrBGsCysReQ1j7ohbtCbdABTv3PKOYhimeGWF1AWh4+nGJdxlzBaQTlmCOiMaXegMBwNFpKSF1EgZC4Am/HrMW5GXOKLkZ3sKtkHqhQsSgnE9Vl6RVIh2V0t4vAsa5Qy/AVtz4styjZa8ldjriKbq7B1zryLl/OwkSkjLzrCNn/8v2cDlgqzvonL/cXtrtAAbLe3FoSHZdW6nJ77EZ/b/wFIOQIE3v6OWG9uHNHpEQCnEhcwgSq0W5o0NoaluAexHs+9OdxxGxGwzMHp3x6Xd6md8SoNBnHP X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The !zswap_exclusive_loads_enabled mode will leave compressed copy in the zswap tree and lru list after the folio swapin. There are some disadvantages in this mode: 1. It's a waste of memory since there are two copies of data, one is folio, the other one is compressed data in zswap. And it's unlikely the compressed data is useful in the near future. 2. If that folio is dirtied, the compressed data must be not useful, but we don't know and don't invalidate the trashy memory in zswap. 3. It's not reclaimable from zswap shrinker since zswap_writeback_entry() will always return -EEXIST and terminate the shrinking process. On the other hand, the only downside of zswap_exclusive_loads_enabled is a little more cpu usage/latency when compression, and the same if the folio is removed from swapcache or dirtied. Not sure if we should accept the above disadvantages in the case of !zswap_exclusive_loads_enabled, so send this out for disscusion. Signed-off-by: Chengming Zhou Acked-by: Johannes Weiner Acked-by: Yosry Ahmed Reviewed-by: Nhat Pham --- mm/Kconfig | 16 ---------------- mm/zswap.c | 14 +++----------- 2 files changed, 3 insertions(+), 27 deletions(-) diff --git a/mm/Kconfig b/mm/Kconfig index ffc3a2ba3a8c..673b35629074 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -45,22 +45,6 @@ config ZSWAP_DEFAULT_ON The selection made here can be overridden by using the kernel command line 'zswap.enabled=' option. -config ZSWAP_EXCLUSIVE_LOADS_DEFAULT_ON - bool "Invalidate zswap entries when pages are loaded" - depends on ZSWAP - help - If selected, exclusive loads for zswap will be enabled at boot, - otherwise it will be disabled. - - If exclusive loads are enabled, when a page is loaded from zswap, - the zswap entry is invalidated at once, as opposed to leaving it - in zswap until the swap entry is freed. - - This avoids having two copies of the same page in memory - (compressed and uncompressed) after faulting in a page from zswap. - The cost is that if the page was never dirtied and needs to be - swapped out again, it will be re-compressed. - config ZSWAP_SHRINKER_DEFAULT_ON bool "Shrink the zswap pool on memory pressure" depends on ZSWAP diff --git a/mm/zswap.c b/mm/zswap.c index 3fbb7e2c8b8d..cbf379abb6c7 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -139,10 +139,6 @@ static bool zswap_non_same_filled_pages_enabled = true; module_param_named(non_same_filled_pages_enabled, zswap_non_same_filled_pages_enabled, bool, 0644); -static bool zswap_exclusive_loads_enabled = IS_ENABLED( - CONFIG_ZSWAP_EXCLUSIVE_LOADS_DEFAULT_ON); -module_param_named(exclusive_loads, zswap_exclusive_loads_enabled, bool, 0644); - /* Number of zpools in zswap_pool (empirically determined for scalability) */ #define ZSWAP_NR_ZPOOLS 32 @@ -1722,16 +1718,12 @@ bool zswap_load(struct folio *folio) count_objcg_event(entry->objcg, ZSWPIN); spin_lock(&tree->lock); - if (zswap_exclusive_loads_enabled) { - zswap_invalidate_entry(tree, entry); - folio_mark_dirty(folio); - } else if (entry->length) { - zswap_lru_del(&entry->pool->list_lru, entry); - zswap_lru_add(&entry->pool->list_lru, entry); - } + zswap_invalidate_entry(tree, entry); zswap_entry_put(entry); spin_unlock(&tree->lock); + folio_mark_dirty(folio); + return true; } From patchwork Thu Feb 1 15:49:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chengming Zhou X-Patchwork-Id: 13541312 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EAF21C4828F for ; Thu, 1 Feb 2024 15:50:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7C8936B0098; Thu, 1 Feb 2024 10:50:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7525B6B009E; Thu, 1 Feb 2024 10:50:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5CB276B00A0; Thu, 1 Feb 2024 10:50:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 45ACE6B009E for ; Thu, 1 Feb 2024 10:50:44 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 044AD1A0E60 for ; Thu, 1 Feb 2024 15:50:43 +0000 (UTC) X-FDA: 81743672808.29.55542E6 Received: from out-171.mta1.migadu.com (out-171.mta1.migadu.com [95.215.58.171]) by imf01.hostedemail.com (Postfix) with ESMTP id 1CCED4000F for ; Thu, 1 Feb 2024 15:50:41 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.171 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706802642; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=J0ZWT+pBrGoRn+2Kb2hwCOno34Uzt8vCj8esbw8lBgQ=; b=5h7F7Dq0lsoSi9zMN9A0k14d8d8DDdGuk9xLZxLIBB5IscmMNK9ADy3TqsLUpQnJa93A8C +VP+oOnv9P+3e98+xxNWMiAjTJNRlx7d6qWwYyY9sJ5Jzk75xFLMZ3ZW4l2W8I95/SPXAC hJGAs9hkM1O2FA6VrDb01uCSkoj955E= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706802642; a=rsa-sha256; cv=none; b=j9TD8Jf5oCRDxLWwzUTDJqhF5rne7oq91txqhm8cmRVYKMqemOZLI4MnN16Cx6bqu444B/ fmkwEB4wP2+iMKMLRkL0M4uAB70k2U7iQl3uJXFEr0Y97WPTD1Nx0E1xA90S5ek8CS5PDf y44Fq1A0ugkJZArssuufmaum0Zfswc4= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.171 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=bytedance.com (policy=quarantine) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Chengming Zhou Date: Thu, 01 Feb 2024 15:49:06 +0000 Subject: [PATCH 6/6] mm/zswap: zswap entry doesn't need refcount anymore MIME-Version: 1.0 Message-Id: <20240201-b4-zswap-invalidate-entry-v1-6-56ed496b6e55@bytedance.com> References: <20240201-b4-zswap-invalidate-entry-v1-0-56ed496b6e55@bytedance.com> In-Reply-To: <20240201-b4-zswap-invalidate-entry-v1-0-56ed496b6e55@bytedance.com> To: Nhat Pham , Johannes Weiner , Andrew Morton , Yosry Ahmed Cc: linux-kernel@vger.kernel.org, Yosry Ahmed , Chengming Zhou , Johannes Weiner , linux-mm@kvack.org X-Migadu-Flow: FLOW_OUT X-Rspamd-Pre-Result: action=add header; module=dmarc; Action set by DMARC X-Rspam-User: X-Rspamd-Queue-Id: 1CCED4000F X-Rspamd-Server: rspam11 X-Stat-Signature: 4ekpokstixm787rysgt35kq4qhnm157y X-Rspam: Yes X-HE-Tag: 1706802641-473290 X-HE-Meta: U2FsdGVkX19JwD8TZvTHZITSmHIb9TvMPLs8Y9rFUTCqmqQiZDy0C8jcA/ShYXCxjAH/okaSt5Z3is7oelhKqyFa+O+8OwZF8zDzLTVYS7IeGlwiqYjKWnr03yfaCU81kCi8WeXruZq3kF/GSRyBtiVnKgkn3lSbgOjykpPvK2Os/enJ1Sj882qzSJdu6TI6/EsznKT894OIFIbWtDzAoKVfOH+npiln4ZEvELgHyCA8Jw0f1W32mTH6YIUO6dWBJh+TkAMK6U8OpTBy5czwknxSTBIIBUIOt/Veqrq/ueVmwIOsZUfPoWi98tKi0zYF4VfftCqRfsKo9cEdWw1CdKYWhvOBAoIdn3esl1c3lZOaXnnVzzYGPqZjbeoXLKsN4gb1/P403LiEn2SeHCxT+Y7XBWw4poFlKfFZQjbHT+1+rfmFJYsECJ/l5+zcklhlR/xukPq/t8C0xmnUvtenIrOEriRVOdEG0M5V2s9eD151wsUGHDbsQMhmyw/fmKj7JOyh1xetJ7VD0LpbjU+fYzpaX0DQ2Ul52hKABxgpReCdCRMf2x955CPhkmzpiBbuGhOCu/sjw0PqCL+2cAKk1G6ljkwM8reHBO01G/cEf4tVcH0OWlaG1pL1fFxF3QFAdcj3R/dTBqBDYKYpLjXeDv8YgabTYuyrR7uKGK/MdGzx7WpqPiEWjKL0NyFRTNrTcfWL/dHOATCLVIx74UxHYf88YSUePVBoSHOam3Wnz7Ub2lMCrtguKl9FlD/vEsLe7sNtjgvCb2XIu03VtIMCnNAFyhyee43SkUaSD0SUms26Uyhf58RxZax0KuofSAjE5mNB6JChQTxWDO7HEp7Pst2LJ9Psi0A+rDtqucP6MVpRV/+VlfurEpWSorzjSv5qPPB92wI6QIq7mWIPihUVlOAyo1oTqHjMdjG9929C6iACWCoLLDnablM9p/wdgaWH+LHVhGvgzC7IuLnc9hD QsYLaggt KGp0mfuMOQAzlASU5nL5YpOAAM5uyKx03iqLiv3UoTi6A2GMBkPrSWnjj24TUlBvueBIvG6XGTlbXpIAKxF3db79+qBYrN8NZG7VrBStIMZ5QJjNjZnnjE9BSg+O/bOSdhtKkw7HvRjB1mID44qyInH4983mJS963xAHFklgIMqkD4ZZHu/NcVA2jW9D+NfTr4fFkBN7JHcZh04pEkfpY3+QGYahQCySpYFueXFuAwNYazQS/WP2OkXUN0hiMCwsVJt7EcKP4RKv5xG/oPf+F4SVwMdtxTnGNXvki4+uzRw8iN/Po/IYjzoLkM62US7WSmz3YlCXTI6ox8IreaEyi6meAAt1p4CfcI3tembkwsKbi6IA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Since we don't need to leave zswap entry on the zswap tree anymore, we should remove it from tree once we find it from the tree. Then after using it, we can directly free it, no concurrent path can find it from tree. Only the shrinker can see it from lru list, which will also double check under tree lock, so no race problem. So we don't need refcount in zswap entry anymore and don't need to take the spinlock for the second time to invalidate it. The side effect is that zswap_entry_free() maybe not happen in tree spinlock, but it's ok since nothing need to be protected by the lock. Signed-off-by: Chengming Zhou Acked-by: Johannes Weiner Reviewed-by: Nhat Pham --- mm/zswap.c | 63 +++++++++++--------------------------------------------------- 1 file changed, 11 insertions(+), 52 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index cbf379abb6c7..cd67f7f6b302 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -193,12 +193,6 @@ struct zswap_pool { * * rbnode - links the entry into red-black tree for the appropriate swap type * swpentry - associated swap entry, the offset indexes into the red-black tree - * refcount - the number of outstanding reference to the entry. This is needed - * to protect against premature freeing of the entry by code - * concurrent calls to load, invalidate, and writeback. The lock - * for the zswap_tree structure that contains the entry must - * be held while changing the refcount. Since the lock must - * be held, there is no reason to also make refcount atomic. * length - the length in bytes of the compressed page data. Needed during * decompression. For a same value filled page length is 0, and both * pool and lru are invalid and must be ignored. @@ -211,7 +205,6 @@ struct zswap_pool { struct zswap_entry { struct rb_node rbnode; swp_entry_t swpentry; - int refcount; unsigned int length; struct zswap_pool *pool; union { @@ -222,11 +215,6 @@ struct zswap_entry { struct list_head lru; }; -/* - * The tree lock in the zswap_tree struct protects a few things: - * - the rbtree - * - the refcount field of each entry in the tree - */ struct zswap_tree { struct rb_root rbroot; spinlock_t lock; @@ -890,14 +878,10 @@ static int zswap_rb_insert(struct rb_root *root, struct zswap_entry *entry, return 0; } -static bool zswap_rb_erase(struct rb_root *root, struct zswap_entry *entry) +static void zswap_rb_erase(struct rb_root *root, struct zswap_entry *entry) { - if (!RB_EMPTY_NODE(&entry->rbnode)) { - rb_erase(&entry->rbnode, root); - RB_CLEAR_NODE(&entry->rbnode); - return true; - } - return false; + rb_erase(&entry->rbnode, root); + RB_CLEAR_NODE(&entry->rbnode); } /********************************* @@ -911,7 +895,6 @@ static struct zswap_entry *zswap_entry_cache_alloc(gfp_t gfp, int nid) entry = kmem_cache_alloc_node(zswap_entry_cache, gfp, nid); if (!entry) return NULL; - entry->refcount = 1; RB_CLEAR_NODE(&entry->rbnode); return entry; } @@ -954,33 +937,15 @@ static void zswap_entry_free(struct zswap_entry *entry) zswap_update_total_size(); } -/* caller must hold the tree lock */ -static void zswap_entry_get(struct zswap_entry *entry) -{ - WARN_ON_ONCE(!entry->refcount); - entry->refcount++; -} - -/* caller must hold the tree lock */ -static void zswap_entry_put(struct zswap_entry *entry) -{ - WARN_ON_ONCE(!entry->refcount); - if (--entry->refcount == 0) { - WARN_ON_ONCE(!RB_EMPTY_NODE(&entry->rbnode)); - zswap_entry_free(entry); - } -} - /* - * If the entry is still valid in the tree, drop the initial ref and remove it - * from the tree. This function must be called with an additional ref held, - * otherwise it may race with another invalidation freeing the entry. + * The caller hold the tree lock and search the entry from the tree, + * so it must be on the tree, remove it from the tree and free it. */ static void zswap_invalidate_entry(struct zswap_tree *tree, struct zswap_entry *entry) { - if (zswap_rb_erase(&tree->rbroot, entry)) - zswap_entry_put(entry); + zswap_rb_erase(&tree->rbroot, entry); + zswap_entry_free(entry); } /********************************* @@ -1219,7 +1184,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, } /* Safe to deref entry after the entry is verified above. */ - zswap_entry_get(entry); + zswap_rb_erase(&tree->rbroot, entry); spin_unlock(&tree->lock); zswap_decompress(entry, &folio->page); @@ -1228,10 +1193,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, if (entry->objcg) count_objcg_event(entry->objcg, ZSWPWB); - spin_lock(&tree->lock); - zswap_invalidate_entry(tree, entry); - zswap_entry_put(entry); - spin_unlock(&tree->lock); + zswap_entry_free(entry); /* folio is up to date */ folio_mark_uptodate(folio); @@ -1702,7 +1664,7 @@ bool zswap_load(struct folio *folio) spin_unlock(&tree->lock); return false; } - zswap_entry_get(entry); + zswap_rb_erase(&tree->rbroot, entry); spin_unlock(&tree->lock); if (entry->length) @@ -1717,10 +1679,7 @@ bool zswap_load(struct folio *folio) if (entry->objcg) count_objcg_event(entry->objcg, ZSWPIN); - spin_lock(&tree->lock); - zswap_invalidate_entry(tree, entry); - zswap_entry_put(entry); - spin_unlock(&tree->lock); + zswap_entry_free(entry); folio_mark_dirty(folio);