From patchwork Mon Jan 13 17:57:32 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13937879 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C437CC02183 for ; Mon, 13 Jan 2025 18:00:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 51FC36B00A8; Mon, 13 Jan 2025 13:00:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4A63B6B00A9; Mon, 13 Jan 2025 13:00:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2FB946B00AA; Mon, 13 Jan 2025 13:00:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id F40286B00A8 for ; Mon, 13 Jan 2025 13:00:44 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A2BB7AE601 for ; Mon, 13 Jan 2025 18:00:44 +0000 (UTC) X-FDA: 83003194008.03.1951357 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf16.hostedemail.com (Postfix) with ESMTP id 93DC0180019 for ; Mon, 13 Jan 2025 18:00:42 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=RTvSnOG9; spf=pass (imf16.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736791242; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FrVAA4WZZ90HWrqJV3lVdkYD6VffUgtaPFosI/u8elE=; b=vVNB6aeHibVY5cf3hCX7NbLJ1lA2xmwzY9Zq1oqTsPsBFfCPovED/+QZTQoKVs7BtwBIu5 5uQBrPVxUPxpufbKj+fCxbCtUF0mDiV2PavVNkpAJKi5J+DBMZz0fS+FAFp1xfp71Pf0h6 LoispkO1BEDtlq5pwWNd6Y4hjWQ2Q4E= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=RTvSnOG9; spf=pass (imf16.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736791242; a=rsa-sha256; cv=none; b=nFYjZJby14RUZEeTkAL7oTLq/Ga+HaR10yECrMRONuIaJrWIFcw3Gpve+dqOQmx9HZ6eI9 LXXIvg7KXj8eUSlnbsecuGlzKoAXDKMRHZKBv0YsagTUdTHwM84gEDa3F6jhoQVl/Wb3gL OelXHi8hhZz1VeVJYGPMItLHEfOHUIk= Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-216395e151bso56013045ad.0 for ; Mon, 13 Jan 2025 10:00:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736791241; x=1737396041; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=FrVAA4WZZ90HWrqJV3lVdkYD6VffUgtaPFosI/u8elE=; b=RTvSnOG9fkA9F67bMevGlpUWmTjfehUPehU87QhL56ODkRGBtlD3sk+qKFxUqnb6ob WkRNTPNUESSVbw4t/krefgOi2u1U0DwCEXkaz96StZt7Kz2g4/Aksf6B8P8s3tEPV36B ggXhobs1D/OOIhevMVS1u3KIDoVWM0gBHA/aM3iXbTNnJt4WQjbGhMn1R8/Ei6rmcurI yfmMkwO1b0g0oeIcezbVNotuGJVwAS8e2VzKIJst7OQAOyOlylx6baskHR/voDGkS/Uz VehuL5XW6+dIMDqoVtPZ4FZsb/4fudHeK38Pxi0cDTJd9R3BwVnJXAxA6WHqm9YX0Iup af2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736791241; x=1737396041; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=FrVAA4WZZ90HWrqJV3lVdkYD6VffUgtaPFosI/u8elE=; b=ur9VsfwgdUFEwaGSxQcEz1U31OATSMCT/tRelX/gMnzkobZOMKpIRCQEgeiNdSvdp1 LmO0pcnDKrljsurxpVRBXKDdgd6rLxGPQZ/SOBf8FbsZ9afsyNNRLLCAuACcdLQVxEo6 W5rqBgd99Os7OKH+6MpSwtzSxUKn68d4I6WlozVOSh0+21VFCqHnrEQt+hCI/9/4Lqjb ZwMesl5yU0tRMiYFNbNAjlSuAvkgjbN+pC8VDt8rbn3PmejrlgDyfiStgHRpkHAklID+ O1kreBUU3SzdVu6dbbspaPyHSqYl37rWkai/VKBrbOknFiZAN6jtvjSoeAHk+z2ReBS+ dqRQ== X-Gm-Message-State: AOJu0Yz8FZ7Qzd2AoSmwvvtC8qaCzx4iibpLVNxdyA/cYHaknjLLn9a4 b1TwigjOFghb18x0zmNvnhxbPmGzpij3BtnpLZveKTWeqe16XQoNqOXfjf4Sbig= X-Gm-Gg: ASbGncu3Sf/qnDd2EoQeJqEaxxc8EZPMA6LciPuJEOBCEmZ8aY03KBz0fZH8F72w1n7 5vnRK1mf8PnFKx6mWQtuex62VsmISU+c1+2eZ8gzhGVXQHVvc/WFoo+b0roPEaRUSHlQH/p0hOi jPYBc3ukfzUKlgySy80zpV1SPu3viqmtKxOb3JW57SIn+OF29a2aBy8HL3zaFav7GZywnACbPqa yybfpefP8EhlAuNb30ppaYr/NLz+9eP7TSQdPI2s+oldUGSC+ww8SN/aBJRGXiSNkecaCwtqr1f Xw== X-Google-Smtp-Source: AGHT+IHICrhgTxcI0KQLiwilMoiewVem2y8BV2E+Xf0iSPUlg9PW0V2LTkmSTxZnAq9iJkrodfAO5Q== X-Received: by 2002:a17:903:2b10:b0:215:a56f:1e50 with SMTP id d9443c01a7336-21ad9ee7348mr157415045ad.8.1736791240589; Mon, 13 Jan 2025 10:00:40 -0800 (PST) Received: from KASONG-MC4.tencent.com ([115.171.41.132]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21a9f21aba7sm57023635ad.113.2025.01.13.10.00.37 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 13 Jan 2025 10:00:40 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Barry Song , Ryan Roberts , Hugh Dickins , Yosry Ahmed , "Huang, Ying" , Baoquan He , Nhat Pham , Johannes Weiner , Kalesh Singh , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v4 13/13] mm, swap_slots: remove slot cache for freeing path Date: Tue, 14 Jan 2025 01:57:32 +0800 Message-ID: <20250113175732.48099-14-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250113175732.48099-1-ryncsn@gmail.com> References: <20250113175732.48099-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: 93DC0180019 X-Rspamd-Server: rspam12 X-Stat-Signature: zqqn7i1eqhxrhzycfk7nhojpa7utt1ak X-Rspam-User: X-HE-Tag: 1736791242-661048 X-HE-Meta: U2FsdGVkX1+nc2HUPsd8BZELrnurEi5NSjEGhzSc2giPqiTlvA0yV26v1P/fWuXXAVgWRw4UXhK+/Wm4y+P0BUMDkfNxmRdxI+ZkIIZtNWcym/5eWAFU0pyTIp2+m7qY/nFX8PnD976cl7AiXmwZAH1OEMuWAb2PMEzwF7OcpOFbVFUF0JaFVf4d6AYJ0JnCiITEXxOkbQUgK0DXGMOUkD1MJW9Td9av5aXvss4C5XOJPuXKNpvf9U6TCM0UamnW0FInsv1kRARqCUIksNTnwwSN7o+BB1f6j5PaR2Z1CSr7FyKCTENy2M12NBuI+ow7rq8B/WTDmBZRV0K1/Y19gHzqpVzQCM2kiymuGhBoe6iDREKex8bMtKBjUYTxMnhWntA4XPNmBNxwwVeUoGkVil+wIXVYMqCewPRf+U+9Jf78VbgBnyrLlbDj0XXrqf/U0iOT48UfysFL+/5+VsIPckiWBhjlBSpb+xXDq6YxcR3eVk6hDeyxr8E+DqW5eYBUx6Cvif7LmX41SA1QR61vFnAjuiNCx8XkxfmdgIGs4M/S+JHrP5Hwzrbwib+ueWKTs1cVIDVfST6Mj0qW7muj+flBGjKb2dZQ3NDEmoZVWIo9oR0NAd9TokLFK1kYCpC6TOvY6NZGi7pAjhSiYt3IhVS0RHp8gCXC3fwww8cby4nBGSHezQLhjZRzY1Yqw8fswUhDpl8w9LV5Cjv9qYF88aGyfS9mVRIlxEFzIGxeTFk5kG74IvOzcGcwGNI/L06XKp4w/2Sj7BEa51i4eFQrtUbaDPpBBtpaps1zXe/DHiinCCHWY1bv279Pi+uOWqONwmXZHqtmW8cVflcQaAtQsab9J+Ksb7W2bRaOb77EzsUX3+whUabbnuRDHvUFhAdigII0BIwgApDiNlYJS35MyU8E08rLKF8gMOA6iorJg1sUDPKqPMBIY1KO6HcJO/vkBRnJHrRr7NFa8djLhWu W1VF6TVb JO2VFo+MiFyWGiNWqp+bRW4J0sQXTjAUD+RaJR1Auo11YRTg98ZRR9N6I4iIna39WXhIB2D5/oCccY9G/Q3RNecmlHDM45DP1P2LUcHFXFz7qqfarHHT1CHOc7mnJKwRu5l5eAg87UrHnVtiIYdQdG/oFtvh1T0yX38vlUkOmcR01q6ifXG1JXw0rbeQO5S92jVNjzGlRhO7QxXgr7h1kuxDzMGZDRuoMiMeQR2cL8JfTKVWaBXfoJjH36I9q3/3B2e25zxom1uBt0A2QRxa3pfcmXxTe506rSs2r49V8hNpti0aVfYQK9b9xMwbCpUkdouISWw7nHHlyHTtkhHb9aP8peawa6gV81rNOEZ91xQuicyfJTIhUpE2kPIae/AluqT4c8BZiSIO5Cf6yoYiURzCpXrAiVsSIXcwjBUSYQU2D4M8wHovnKJWSc32nX1RJefnXYNMA/MOTDOP2JPP19VzXVpf4Yd8yAe7WmdAb0zfCTyRd16tXAgOJTosL3b44VcGe/gj1v5+ZNPWqXmGrLfhN9JyuDpzTIpXgmC8pzBlUkkZLnEmf86eeuv+u+Lbw+KGXgVvC3wsORIekdrdItkhOqPMwNk9jcJuSPe+9u5PhAj8peS4HrB3nZw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song The slot cache for freeing path is mostly for reducing the overhead of si->lock. As we have basically eliminated the si->lock usage for freeing path, it can be removed. This helps simplify the code, and avoids swap entries from being hold in cache upon freeing. The delayed freeing of entries have been causing trouble for further optimizations for zswap [1] and in theory will also cause more fragmentation, and extra overhead. Test with build linux kernel showed both performance and fragmentation is better without the cache: tiem make -j96 / 768M memcg, 4K pages, 10G ZRAM, avg of 4 test run:: Before: Sys time: 36047.78, Real time: 472.43 After: (-7.6% sys time, -7.3% real time) Sys time: 33314.76, Real time: 437.67 time make -j96 / 1152M memcg, 64K mTHP, 10G ZRAM, avg of 4 test run: Before: Sys time: 46859.04, Real time: 562.63 hugepages-64kB/stats/swpout: 1783392 hugepages-64kB/stats/swpout_fallback: 240875 After: (-23.3% sys time, -21.3% real time) Sys time: 35958.87, Real time: 442.69 hugepages-64kB/stats/swpout: 1866267 hugepages-64kB/stats/swpout_fallback: 158330 Sequential SWAP should be also slightly faster, tests didn't show a measurable difference though, at least no regression: Swapin 4G zero page on ZRAM (time in us): Before (avg. 1923756) 1912391 1927023 1927957 1916527 1918263 1914284 1934753 1940813 1921791 After (avg. 1922290): 1919101 1925743 1916810 1917007 1923930 1935152 1917403 1923549 1921913 Link: https://lore.kernel.org/all/CAMgjq7ACohT_uerSz8E_994ZZCv709Zor+43hdmesW_59W1BWw@mail.gmail.com/[1] Suggested-by: Chris Li Signed-off-by: Kairui Song --- include/linux/swap_slots.h | 3 -- mm/swap_slots.c | 78 +++++---------------------------- mm/swapfile.c | 89 +++++++++++++++----------------------- 3 files changed, 44 insertions(+), 126 deletions(-) diff --git a/include/linux/swap_slots.h b/include/linux/swap_slots.h index 15adfb8c813a..840aec3523b2 100644 --- a/include/linux/swap_slots.h +++ b/include/linux/swap_slots.h @@ -16,15 +16,12 @@ struct swap_slots_cache { swp_entry_t *slots; int nr; int cur; - spinlock_t free_lock; /* protects slots_ret, n_ret */ - swp_entry_t *slots_ret; int n_ret; }; void disable_swap_slots_cache_lock(void); void reenable_swap_slots_cache_unlock(void); void enable_swap_slots_cache(void); -void free_swap_slot(swp_entry_t entry); extern bool swap_slot_cache_enabled; diff --git a/mm/swap_slots.c b/mm/swap_slots.c index 13ab3b771409..9c7c171df7ba 100644 --- a/mm/swap_slots.c +++ b/mm/swap_slots.c @@ -43,17 +43,15 @@ static DEFINE_MUTEX(swap_slots_cache_mutex); /* Serialize swap slots cache enable/disable operations */ static DEFINE_MUTEX(swap_slots_cache_enable_mutex); -static void __drain_swap_slots_cache(unsigned int type); +static void __drain_swap_slots_cache(void); #define use_swap_slot_cache (swap_slot_cache_active && swap_slot_cache_enabled) -#define SLOTS_CACHE 0x1 -#define SLOTS_CACHE_RET 0x2 static void deactivate_swap_slots_cache(void) { mutex_lock(&swap_slots_cache_mutex); swap_slot_cache_active = false; - __drain_swap_slots_cache(SLOTS_CACHE|SLOTS_CACHE_RET); + __drain_swap_slots_cache(); mutex_unlock(&swap_slots_cache_mutex); } @@ -72,7 +70,7 @@ void disable_swap_slots_cache_lock(void) if (swap_slot_cache_initialized) { /* serialize with cpu hotplug operations */ cpus_read_lock(); - __drain_swap_slots_cache(SLOTS_CACHE|SLOTS_CACHE_RET); + __drain_swap_slots_cache(); cpus_read_unlock(); } } @@ -113,7 +111,7 @@ static bool check_cache_active(void) static int alloc_swap_slot_cache(unsigned int cpu) { struct swap_slots_cache *cache; - swp_entry_t *slots, *slots_ret; + swp_entry_t *slots; /* * Do allocation outside swap_slots_cache_mutex @@ -125,28 +123,19 @@ static int alloc_swap_slot_cache(unsigned int cpu) if (!slots) return -ENOMEM; - slots_ret = kvcalloc(SWAP_SLOTS_CACHE_SIZE, sizeof(swp_entry_t), - GFP_KERNEL); - if (!slots_ret) { - kvfree(slots); - return -ENOMEM; - } - mutex_lock(&swap_slots_cache_mutex); cache = &per_cpu(swp_slots, cpu); - if (cache->slots || cache->slots_ret) { + if (cache->slots) { /* cache already allocated */ mutex_unlock(&swap_slots_cache_mutex); kvfree(slots); - kvfree(slots_ret); return 0; } if (!cache->lock_initialized) { mutex_init(&cache->alloc_lock); - spin_lock_init(&cache->free_lock); cache->lock_initialized = true; } cache->nr = 0; @@ -160,19 +149,16 @@ static int alloc_swap_slot_cache(unsigned int cpu) */ mb(); cache->slots = slots; - cache->slots_ret = slots_ret; mutex_unlock(&swap_slots_cache_mutex); return 0; } -static void drain_slots_cache_cpu(unsigned int cpu, unsigned int type, - bool free_slots) +static void drain_slots_cache_cpu(unsigned int cpu, bool free_slots) { struct swap_slots_cache *cache; - swp_entry_t *slots = NULL; cache = &per_cpu(swp_slots, cpu); - if ((type & SLOTS_CACHE) && cache->slots) { + if (cache->slots) { mutex_lock(&cache->alloc_lock); swapcache_free_entries(cache->slots + cache->cur, cache->nr); cache->cur = 0; @@ -183,20 +169,9 @@ static void drain_slots_cache_cpu(unsigned int cpu, unsigned int type, } mutex_unlock(&cache->alloc_lock); } - if ((type & SLOTS_CACHE_RET) && cache->slots_ret) { - spin_lock_irq(&cache->free_lock); - swapcache_free_entries(cache->slots_ret, cache->n_ret); - cache->n_ret = 0; - if (free_slots && cache->slots_ret) { - slots = cache->slots_ret; - cache->slots_ret = NULL; - } - spin_unlock_irq(&cache->free_lock); - kvfree(slots); - } } -static void __drain_swap_slots_cache(unsigned int type) +static void __drain_swap_slots_cache(void) { unsigned int cpu; @@ -224,13 +199,13 @@ static void __drain_swap_slots_cache(unsigned int type) * There are no slots on such cpu that need to be drained. */ for_each_online_cpu(cpu) - drain_slots_cache_cpu(cpu, type, false); + drain_slots_cache_cpu(cpu, false); } static int free_slot_cache(unsigned int cpu) { mutex_lock(&swap_slots_cache_mutex); - drain_slots_cache_cpu(cpu, SLOTS_CACHE | SLOTS_CACHE_RET, true); + drain_slots_cache_cpu(cpu, true); mutex_unlock(&swap_slots_cache_mutex); return 0; } @@ -269,39 +244,6 @@ static int refill_swap_slots_cache(struct swap_slots_cache *cache) return cache->nr; } -void free_swap_slot(swp_entry_t entry) -{ - struct swap_slots_cache *cache; - - /* Large folio swap slot is not covered. */ - zswap_invalidate(entry); - - cache = raw_cpu_ptr(&swp_slots); - if (likely(use_swap_slot_cache && cache->slots_ret)) { - spin_lock_irq(&cache->free_lock); - /* Swap slots cache may be deactivated before acquiring lock */ - if (!use_swap_slot_cache || !cache->slots_ret) { - spin_unlock_irq(&cache->free_lock); - goto direct_free; - } - if (cache->n_ret >= SWAP_SLOTS_CACHE_SIZE) { - /* - * Return slots to global pool. - * The current swap_map value is SWAP_HAS_CACHE. - * Set it to 0 to indicate it is available for - * allocation in global pool - */ - swapcache_free_entries(cache->slots_ret, cache->n_ret); - cache->n_ret = 0; - } - cache->slots_ret[cache->n_ret++] = entry; - spin_unlock_irq(&cache->free_lock); - } else { -direct_free: - swapcache_free_entries(&entry, 1); - } -} - swp_entry_t folio_alloc_swap(struct folio *folio) { swp_entry_t entry; diff --git a/mm/swapfile.c b/mm/swapfile.c index 793b2fd1a2a8..b3154e52cb45 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -53,14 +53,15 @@ static bool swap_count_continued(struct swap_info_struct *, pgoff_t, unsigned char); static void free_swap_count_continuations(struct swap_info_struct *); -static void swap_entry_range_free(struct swap_info_struct *si, swp_entry_t entry, - unsigned int nr_pages); +static void swap_entry_range_free(struct swap_info_struct *si, + struct swap_cluster_info *ci, + swp_entry_t entry, unsigned int nr_pages); static void swap_range_alloc(struct swap_info_struct *si, unsigned int nr_entries); static bool folio_swapcache_freeable(struct folio *folio); static struct swap_cluster_info *lock_cluster(struct swap_info_struct *si, unsigned long offset); -static void unlock_cluster(struct swap_cluster_info *ci); +static inline void unlock_cluster(struct swap_cluster_info *ci); static DEFINE_SPINLOCK(swap_lock); static unsigned int nr_swapfiles; @@ -261,10 +262,9 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si, folio_ref_sub(folio, nr_pages); folio_set_dirty(folio); - /* Only sinple page folio can be backed by zswap */ - if (nr_pages == 1) - zswap_invalidate(entry); - swap_entry_range_free(si, entry, nr_pages); + ci = lock_cluster(si, offset); + swap_entry_range_free(si, ci, entry, nr_pages); + unlock_cluster(ci); ret = nr_pages; out_unlock: folio_unlock(folio); @@ -1128,8 +1128,10 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset, * Use atomic clear_bit operations only on zeromap instead of non-atomic * bitmap_clear to prevent adjacent bits corruption due to simultaneous writes. */ - for (i = 0; i < nr_entries; i++) + for (i = 0; i < nr_entries; i++) { clear_bit(offset + i, si->zeromap); + zswap_invalidate(swp_entry(si->type, offset + i)); + } if (si->flags & SWP_BLKDEV) swap_slot_free_notify = @@ -1434,9 +1436,9 @@ static unsigned char __swap_entry_free(struct swap_info_struct *si, ci = lock_cluster(si, offset); usage = __swap_entry_free_locked(si, offset, 1); - unlock_cluster(ci); if (!usage) - free_swap_slot(entry); + swap_entry_range_free(si, ci, swp_entry(si->type, offset), 1); + unlock_cluster(ci); return usage; } @@ -1464,13 +1466,10 @@ static bool __swap_entries_free(struct swap_info_struct *si, } for (i = 0; i < nr; i++) WRITE_ONCE(si->swap_map[offset + i], SWAP_HAS_CACHE); + if (!has_cache) + swap_entry_range_free(si, ci, entry, nr); unlock_cluster(ci); - if (!has_cache) { - for (i = 0; i < nr; i++) - zswap_invalidate(swp_entry(si->type, offset + i)); - swap_entry_range_free(si, entry, nr); - } return has_cache; fallback: @@ -1490,15 +1489,13 @@ static bool __swap_entries_free(struct swap_info_struct *si, * Drop the last HAS_CACHE flag of swap entries, caller have to * ensure all entries belong to the same cgroup. */ -static void swap_entry_range_free(struct swap_info_struct *si, swp_entry_t entry, - unsigned int nr_pages) +static void swap_entry_range_free(struct swap_info_struct *si, + struct swap_cluster_info *ci, + swp_entry_t entry, unsigned int nr_pages) { unsigned long offset = swp_offset(entry); unsigned char *map = si->swap_map + offset; unsigned char *map_end = map + nr_pages; - struct swap_cluster_info *ci; - - ci = lock_cluster(si, offset); /* It should never free entries across different clusters */ VM_BUG_ON(ci != offset_to_cluster(si, offset + nr_pages - 1)); @@ -1518,7 +1515,6 @@ static void swap_entry_range_free(struct swap_info_struct *si, swp_entry_t entry free_cluster(si, ci); else partial_free_cluster(si, ci); - unlock_cluster(ci); } static void cluster_swap_free_nr(struct swap_info_struct *si, @@ -1526,28 +1522,13 @@ static void cluster_swap_free_nr(struct swap_info_struct *si, unsigned char usage) { struct swap_cluster_info *ci; - DECLARE_BITMAP(to_free, BITS_PER_LONG) = { 0 }; - int i, nr; + unsigned long end = offset + nr_pages; ci = lock_cluster(si, offset); - while (nr_pages) { - nr = min(BITS_PER_LONG, nr_pages); - for (i = 0; i < nr; i++) { - if (!__swap_entry_free_locked(si, offset + i, usage)) - bitmap_set(to_free, i, 1); - } - if (!bitmap_empty(to_free, BITS_PER_LONG)) { - unlock_cluster(ci); - for_each_set_bit(i, to_free, BITS_PER_LONG) - free_swap_slot(swp_entry(si->type, offset + i)); - if (nr == nr_pages) - return; - bitmap_clear(to_free, 0, BITS_PER_LONG); - ci = lock_cluster(si, offset); - } - offset += nr; - nr_pages -= nr; - } + do { + if (!__swap_entry_free_locked(si, offset, usage)) + swap_entry_range_free(si, ci, swp_entry(si->type, offset), 1); + } while (++offset < end); unlock_cluster(ci); } @@ -1588,18 +1569,12 @@ void put_swap_folio(struct folio *folio, swp_entry_t entry) return; ci = lock_cluster(si, offset); - if (size > 1 && swap_is_has_cache(si, offset, size)) { - unlock_cluster(ci); - swap_entry_range_free(si, entry, size); - return; - } - for (int i = 0; i < size; i++, entry.val++) { - if (!__swap_entry_free_locked(si, offset + i, SWAP_HAS_CACHE)) { - unlock_cluster(ci); - free_swap_slot(entry); - if (i == size - 1) - return; - lock_cluster(si, offset); + if (swap_is_has_cache(si, offset, size)) + swap_entry_range_free(si, ci, entry, size); + else { + for (int i = 0; i < size; i++, entry.val++) { + if (!__swap_entry_free_locked(si, offset + i, SWAP_HAS_CACHE)) + swap_entry_range_free(si, ci, entry, 1); } } unlock_cluster(ci); @@ -1608,6 +1583,7 @@ void put_swap_folio(struct folio *folio, swp_entry_t entry) void swapcache_free_entries(swp_entry_t *entries, int n) { int i; + struct swap_cluster_info *ci; struct swap_info_struct *si = NULL; if (n <= 0) @@ -1615,8 +1591,11 @@ void swapcache_free_entries(swp_entry_t *entries, int n) for (i = 0; i < n; ++i) { si = _swap_info_get(entries[i]); - if (si) - swap_entry_range_free(si, entries[i], 1); + if (si) { + ci = lock_cluster(si, swp_offset(entries[i])); + swap_entry_range_free(si, ci, entries[i], 1); + unlock_cluster(ci); + } } }