From patchwork Tue Jan 30 01:36:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13536686 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6EBF2C47DA9 for ; Tue, 30 Jan 2024 01:43:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 121916B00C0; Mon, 29 Jan 2024 20:42:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0CE576B00C2; Mon, 29 Jan 2024 20:42:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E8B866B00C1; Mon, 29 Jan 2024 20:42:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D50C16B00BF for ; Mon, 29 Jan 2024 20:42:40 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B23C7A0170 for ; Tue, 30 Jan 2024 01:42:40 +0000 (UTC) X-FDA: 81734278080.01.3D09ED4 Received: from mail-qt1-f181.google.com (mail-qt1-f181.google.com [209.85.160.181]) by imf14.hostedemail.com (Postfix) with ESMTP id E94F5100010 for ; Tue, 30 Jan 2024 01:42:38 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=U6D5kgSz; spf=pass (imf14.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.181 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706578959; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jHRUZ67yNEKZS9FrhnY4ruPkQ8T6m4mDy3O5Fk7spbc=; b=TxR46PhEK3vlA6TnDhdC94lsJUaPn32S6xieFWsJIcKXP/PA4/wG2R8T31mCxqGk/DFS9J g9frjhPAdGMmZEIYCTBAgKgSiY2MKn5QJi0ITZMDLiLKi4w4XnePmBnsAc6xlDzcCS2jMU /7TUoB8z4Kg/V6DalhJYiWftyNPTQCs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706578959; a=rsa-sha256; cv=none; b=CK82ZMl8KAouyYKMWDe8Vaw7v3ri0VkWQoPYiqdoROTj8C38JukM0l2KaTyZP+EDt1SDhI DDyqZN81QigzdX3HmoNGdXt5Ug/8KZWAwHe/hAjSQ87yt9a6SmuemvQU3aWoWqlyIapsON +8GpSmcbysymaOo9qM9mdtbjOTrEQUQ= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=U6D5kgSz; spf=pass (imf14.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.181 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org Received: by mail-qt1-f181.google.com with SMTP id d75a77b69052e-42aa93264b1so6379101cf.2 for ; Mon, 29 Jan 2024 17:42:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1706578958; x=1707183758; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jHRUZ67yNEKZS9FrhnY4ruPkQ8T6m4mDy3O5Fk7spbc=; b=U6D5kgSzOnI4c4609Ra0cphIvgGZymUNmF9vLAwkCPfHqfb+n11WxDEO0ld1y3OPoZ 1YzT+3wfEhtrh0IkLRMAC7Atk3EYctbQ6aNALsOHBCcCjL5qasptVVTXaewj0l26i0KS rwBKlwtosZYmEdHLbAY3/RK7rnUKWWiNUO58yHY8eGhZ+8DuNkmUs/ijen5PihIytegK keSbPL2/7JgnnXC8wo8C9Vec8cSWFXT/J82Q5gxxHixd29iCmH4tj8PfeY4pqBj5pMDv vS4fDX2Nf+zHozO0jgwSVlyquH2NBllx3UEgUHJW9kvk+v0QzmALT43upgbFcWt+TKnx 9/Yg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706578958; x=1707183758; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jHRUZ67yNEKZS9FrhnY4ruPkQ8T6m4mDy3O5Fk7spbc=; b=f1zkon1VRIPv233rXpxYhDeWQZdn9m0jbvyHkcfxKriT6lcEOepxB5Zo5mFf2jcvxg hgj/PMl8rI9MVkbGAMTaDjtXeOY7PmS6sLSIh3OND529JbauvJvY5iZqteJ8+dXmZMWG P5Dv9XERBrAEh6Pc5J4Ii725kLgEDqW51KpEDXsHL6xQ+6Zq7YPNZMdVFAfjSS+pwz1u 2qYN86GM7RaNtWNO3EK9P+NmojM4eF+HeutWLRtx0BJi9pnmjfIifpz6+8oKdgbWYiDB bb7MKrfO41U1PLvS+N8jZvRQDMJHXs0dG8YeYWLT8O5OQP3pF2iaoGgOb479mQHKcf5f lE4A== X-Gm-Message-State: AOJu0YzM3V/CqfMXOSSAu3Ie2Dvs+gZ0nvZodxg+jlPYHEINlISJNUTk CF0GDIUIzG3F0dqqci8mGXUcIBl7bHJ53MC6ZswS8eYOQYP7+TwAsyRc/cviKcY= X-Google-Smtp-Source: AGHT+IFMteiVW0eCHxuGhuCuBCoCg6cS0vgJoS1KnkVEaCB7DQT4/zN8xIaB+z0TYxs1jXneazTs4Q== X-Received: by 2002:ac8:5bcc:0:b0:42a:9c39:9727 with SMTP id b12-20020ac85bcc000000b0042a9c399727mr5682189qtb.83.1706578958090; Mon, 29 Jan 2024 17:42:38 -0800 (PST) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id bl4-20020a05622a244400b004260b65b4f7sm1369468qtb.97.2024.01.29.17.42.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Jan 2024 17:42:37 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Nhat Pham , Yosry Ahmed , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 20/20] mm: zswap: function ordering: shrink_memcg_cb Date: Mon, 29 Jan 2024 20:36:56 -0500 Message-ID: <20240130014208.565554-21-hannes@cmpxchg.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240130014208.565554-1-hannes@cmpxchg.org> References: <20240130014208.565554-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: E94F5100010 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 5uid4w8ftwe4mh3n6ukosnbfrw17kyi4 X-HE-Tag: 1706578958-881184 X-HE-Meta: U2FsdGVkX18F5LJ/D8ZqcccbVc+VVWq5J/nq1qTpLkTBiythndE5J7H2OObMc5c5FjgBX7cOIcE+I5rMDM2GWMQo8Xvj3h+TLgNiwD2jgG/DqzjA7aCW6WjA0GHk8uAcDtHg/lekb9cjz5cRrxq5ZXTrNtvkICbC8BPpMcf7cHiUH1vS+sRztsC+1ntW24SdgyUvMCHUr+hcbD+5ChroaJXN9UsUsLdm76C6Ff/2zctpjBjaf72NOhA0C8iuqzMJ8NQNE7VEIaJqwi5YNoLDCI6eUgQtJlEhHoeHySLDtB17L2iHjtodh/rKR7PMlu/9s6ZtsUtdkly789WtgaKbjbrabxX5PQfdyLAKMU5x4y1MaZM3SjTGNdPr2QT2UDwpWnUrlBi+PJBzldaWphO69cyHcGwGSePqKArK7JPY5O0bfuosoi78DGnAfQTlIUmUz5nJNxoqBGbxA0B0HSOJR4+huRvIE4JR1KYBQfxGK2s+/Nmo6tFDv495VquIxLvrm/k3Es1IlSjdgArDZ7Mh67RPl2ZnR0W0t7XxHDm6RPIYmNDtUIb6/l6HQfg3/SJjNqjHjPVlU9HrWJWSPnMW74OYqEJsHZsvunO78nxjwl25ZXqA2Lj9oGysVF1yxmx2IVROj9F0Z0b3RROiV9lQ+VOi1KpEM2jhZZ0NVOD7/MeuZwQ0Xn+vnTO5gDg2svmli8nql+VrwNVkmqBn5wWbtwsxddkqBwTGoStOkFbtl3WRi+kWXHwwvVq6YkBam16EY2DBKLMTUJGWzAhIG1VgWZPIegm9oSPLi6RnChSDSduK24Tm3yQkNFfZdJndOQMxC5NYNnJs7KbMliTKwmfaXj/gtsPKvHgyIP8uwqoKhfP9uNWv2cRV7lfVPEm3jYCLGjEIL+du0cyLpT/f2Q7mLX+GqxDcESRqu8Q6hx5M8myOVM1Di8gW79XSgiz1x7AgF8P1FPbnDrquaHGIbA7 zKB+lbiL NQ0CccPrLHlDF5mLgPS4K58GRKWTTY0q/uuiN3Nc4W13nbBJh187ZOlAH2kNfucr7RTqJJn+buYHR3g5gmwrE6BvlGfekii3IO2lmPq1wfAhQciORJ3k4jS4N6ljaTowq7llMhXQETGMIOgEDBZXbX6EqI196kT7qndmta7s6QXyyVNjWG9c6yPxN7FuP699XgxiU2DGCg0ztDgSV/fY8XKfJKMX9T+IZLmTMVxpxVgMu0o4ZIjhvWhrEcnDQakPcMd424O0VdBfj3PZ9po/PCBndW7G39UaA1uTSStzZ+ttAdsWDc6+kaWQi1S3Ho3bbkmb5YBdQExoMLVdDPikctXFT4OG4lK0cs01dvnuSXtZsvfCQ1kycTakFQteqP9HU8nDouy6AoY304zrKMiekL+0nXfb2y7pqdC2jd18M/TcuJbSWG7caLSjoruWS6yFPvRLsnuP6w3n/TVPQYItLcJv5PQRn6wUPKsuPuo+z7KmBTds= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: shrink_memcg_cb() is called by the shrinker and is based on zswap_writeback_entry(). Move it in between. Save one fwd decl. Signed-off-by: Johannes Weiner Reviewed-by: Nhat Pham --- mm/zswap.c | 125 ++++++++++++++++++++++++++--------------------------- 1 file changed, 61 insertions(+), 64 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 0cb3437d47eb..4aea03285532 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1254,7 +1254,67 @@ static int zswap_writeback_entry(struct zswap_entry *entry, * shrinker functions **********************************/ static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_one *l, - spinlock_t *lock, void *arg); + spinlock_t *lock, void *arg) +{ + struct zswap_entry *entry = container_of(item, struct zswap_entry, lru); + bool *encountered_page_in_swapcache = (bool *)arg; + swp_entry_t swpentry; + enum lru_status ret = LRU_REMOVED_RETRY; + int writeback_result; + + /* + * Rotate the entry to the tail before unlocking the LRU, + * so that in case of an invalidation race concurrent + * reclaimers don't waste their time on it. + * + * If writeback succeeds, or failure is due to the entry + * being invalidated by the swap subsystem, the invalidation + * will unlink and free it. + * + * Temporary failures, where the same entry should be tried + * again immediately, almost never happen for this shrinker. + * We don't do any trylocking; -ENOMEM comes closest, + * but that's extremely rare and doesn't happen spuriously + * either. Don't bother distinguishing this case. + * + * But since they do exist in theory, the entry cannot just + * be unlinked, or we could leak it. Hence, rotate. + */ + list_move_tail(item, &l->list); + + /* + * Once the lru lock is dropped, the entry might get freed. The + * swpentry is copied to the stack, and entry isn't deref'd again + * until the entry is verified to still be alive in the tree. + */ + swpentry = entry->swpentry; + + /* + * It's safe to drop the lock here because we return either + * LRU_REMOVED_RETRY or LRU_RETRY. + */ + spin_unlock(lock); + + writeback_result = zswap_writeback_entry(entry, swpentry); + + if (writeback_result) { + zswap_reject_reclaim_fail++; + ret = LRU_RETRY; + + /* + * Encountering a page already in swap cache is a sign that we are shrinking + * into the warmer region. We should terminate shrinking (if we're in the dynamic + * shrinker context). + */ + if (writeback_result == -EEXIST && encountered_page_in_swapcache) + *encountered_page_in_swapcache = true; + } else { + zswap_written_back_pages++; + } + + spin_lock(lock); + return ret; +} static unsigned long zswap_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc) @@ -1354,69 +1414,6 @@ static void zswap_alloc_shrinker(struct zswap_pool *pool) pool->shrinker->seeks = DEFAULT_SEEKS; } -static enum lru_status shrink_memcg_cb(struct list_head *item, struct list_lru_one *l, - spinlock_t *lock, void *arg) -{ - struct zswap_entry *entry = container_of(item, struct zswap_entry, lru); - bool *encountered_page_in_swapcache = (bool *)arg; - swp_entry_t swpentry; - enum lru_status ret = LRU_REMOVED_RETRY; - int writeback_result; - - /* - * Rotate the entry to the tail before unlocking the LRU, - * so that in case of an invalidation race concurrent - * reclaimers don't waste their time on it. - * - * If writeback succeeds, or failure is due to the entry - * being invalidated by the swap subsystem, the invalidation - * will unlink and free it. - * - * Temporary failures, where the same entry should be tried - * again immediately, almost never happen for this shrinker. - * We don't do any trylocking; -ENOMEM comes closest, - * but that's extremely rare and doesn't happen spuriously - * either. Don't bother distinguishing this case. - * - * But since they do exist in theory, the entry cannot just - * be unlinked, or we could leak it. Hence, rotate. - */ - list_move_tail(item, &l->list); - - /* - * Once the lru lock is dropped, the entry might get freed. The - * swpentry is copied to the stack, and entry isn't deref'd again - * until the entry is verified to still be alive in the tree. - */ - swpentry = entry->swpentry; - - /* - * It's safe to drop the lock here because we return either - * LRU_REMOVED_RETRY or LRU_RETRY. - */ - spin_unlock(lock); - - writeback_result = zswap_writeback_entry(entry, swpentry); - - if (writeback_result) { - zswap_reject_reclaim_fail++; - ret = LRU_RETRY; - - /* - * Encountering a page already in swap cache is a sign that we are shrinking - * into the warmer region. We should terminate shrinking (if we're in the dynamic - * shrinker context). - */ - if (writeback_result == -EEXIST && encountered_page_in_swapcache) - *encountered_page_in_swapcache = true; - } else { - zswap_written_back_pages++; - } - - spin_lock(lock); - return ret; -} - static int shrink_memcg(struct mem_cgroup *memcg) { struct zswap_pool *pool;