From patchwork Mon Mar 25 23:50:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13603182 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4EE2CC54E64 for ; Mon, 25 Mar 2024 23:50:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C209B6B0083; Mon, 25 Mar 2024 19:50:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BA9CA6B0087; Mon, 25 Mar 2024 19:50:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 961906B0085; Mon, 25 Mar 2024 19:50:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 827AC6B0082 for ; Mon, 25 Mar 2024 19:50:25 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 485C21401F6 for ; Mon, 25 Mar 2024 23:50:25 +0000 (UTC) X-FDA: 81937208010.25.564209B Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf26.hostedemail.com (Postfix) with ESMTP id AB8C0140009 for ; Mon, 25 Mar 2024 23:50:22 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=mojwaJjZ; spf=pass (imf26.hostedemail.com: domain of 3vQ0CZgoKCBgMCGFMy5A214CC492.0CA96BIL-AA8Jy08.CF4@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3vQ0CZgoKCBgMCGFMy5A214CC492.0CA96BIL-AA8Jy08.CF4@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711410622; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=u9fvbqW1pLtPErlE8l2YKQeiKLLE5wFNh7avewbFYWs=; b=d4NECfLax4oQGq4phswaj2RnYanYsHVPjdGHZG2vZ+rGGmFbSJOm4SsFpTHCrmq9UL/04v rFokEu7lksPsqY9V0PefClVDtYcnOVMN6dmVTIVD4v93votUqw9GHejnc8ye1VlbJjD1MK MH0VbVspNwDoM53YpmsNl/k3tqzgysA= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=mojwaJjZ; spf=pass (imf26.hostedemail.com: domain of 3vQ0CZgoKCBgMCGFMy5A214CC492.0CA96BIL-AA8Jy08.CF4@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3vQ0CZgoKCBgMCGFMy5A214CC492.0CA96BIL-AA8Jy08.CF4@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711410622; a=rsa-sha256; cv=none; b=solxQS1W7E8+3hJyv8xJtmBnPNYcPNuCOYIi2si7bCwFtdddzbwc2P9UhLrcdoitnhgIzO hAT8y1Csj0SjR6coLdxV8ygoEoh2yYslt9Fq69KN+8mwfoDaBs2i3MKgBXKMHeO858hPOH eO+xS/lUvGqwJXxp3uXrklRMPHwhebg= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc6b26ce0bbso10012761276.1 for ; Mon, 25 Mar 2024 16:50:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711410622; x=1712015422; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=u9fvbqW1pLtPErlE8l2YKQeiKLLE5wFNh7avewbFYWs=; b=mojwaJjZYrglg3ygnQOqVg8JsZFM5R6TJKdA8wrySVT9K28BdWAy72KbTaeMmVg43r NzrE/el2niauCzMoIbthxyguIF9AwPD5rObsV4yCbwJeIQIDXNEvvivu6a35wTkyBAjn iFOMKjMb1AgGjEZ9QWyPfbO6Wfu7a/SVydr5YXUm+qJeUlSiuUksvYkltppQ2AFvpwht +yn2wOrHyvJ+ha1FHO3v0gfVHG02R7eJwG0d/9RhyNvma3IPH1VJGldozzpI+OcX17iZ QXx3g1SLYotsAAu54GUr13NkyQE7Kq5obTQxwbY7GZHPav3cG2omPkDhsjpBWa1AaAga Q+/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711410622; x=1712015422; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=u9fvbqW1pLtPErlE8l2YKQeiKLLE5wFNh7avewbFYWs=; b=spEE5uqSjXw1SqbKnDL1e+NKH9sOJIsNCaj1OEe12Tui79FajkrYmJMcgDpBJzSPsj lKbiAJfHSOg7DpSE7s8YyeibbTdT28k1Kr8otMzzwAFMwcudSeETvi4gtAAsDg6GkBhl Xj9N7il8jO4aJRHsoVCKdkmznDMeX8x0TotcNYYTRjXeAM8CUwccjhNEQS0XCOIhUbWf Wp/f5QDBzcPYSHvgJMbT0lKNZN5mPAX5V2p/a04B+FbS9nmpSmRIygAm1cz/duiIlh1S wBIUg3SWpD2kP0IAqFIxlF8mzSzBA56DEgmi7yiG6zV+7tmQgcgrWfSkcrhr3BXX5B+5 BvOw== X-Forwarded-Encrypted: i=1; AJvYcCUWfFL5k5G+fMeDryrLp60fh35GUsdKJrK3CGecFRwqFs/Srs5UB8vJVxfBVI/WG9dUuXFjc+4xkkqBNaKEPZpRbBo= X-Gm-Message-State: AOJu0YwkOs4apyb0yVNPmFTPJXAJhxBItTEJMs9fUSubMSQOePy1c6i+ +AUakKEVsjwrVvngOP8JcZ9M9FREhGarox0EMtOQ4gYFiVy9fp8iqNPeAxzs3y55F6t588Xqiz3 WH/c/DoiqbI6ZmCBbrQ== X-Google-Smtp-Source: AGHT+IH+DXWV+vX4OULsfH+Uyq0gzOaLllxrZEK/FYDaAYjBCM9JZ11SOX35Q3oKMnkT3VL0MGvjY9eZ8pU6LA5R X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:2491:b0:dcb:e982:4e40 with SMTP id ds17-20020a056902249100b00dcbe9824e40mr2590953ybb.12.1711410621894; Mon, 25 Mar 2024 16:50:21 -0700 (PDT) Date: Mon, 25 Mar 2024 23:50:09 +0000 In-Reply-To: <20240325235018.2028408-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20240325235018.2028408-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.396.g6e790dbe36-goog Message-ID: <20240325235018.2028408-2-yosryahmed@google.com> Subject: [RFC PATCH 1/9] mm: zswap: always shrink in zswap_store() if zswap_pool_reached_full From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: AB8C0140009 X-Rspam-User: X-Stat-Signature: msfzyii39afs7bmgai4hrma4ez5p7nt7 X-Rspamd-Server: rspam01 X-HE-Tag: 1711410622-4350 X-HE-Meta: U2FsdGVkX1+wDq4+IMvddYGPpiC+r6JYSicxwlX0aBExE5QDjMBJj1HZffIRim+0v49riyEsCHCzn3ksz/2hjQsixVjiBo+Zj8pIUTXmiRXeshBRCm/23qPFP/kQOMSWuYyiX/FuxgiHyY3J1GSbu23sQxEy3+L9j8vql0GUdVaqJBCJu2W20+IxhuTVd9N49t9k+WXGqhhkzQRX0HjgIseO7Nv9OTycY/NXu+8Y5gHWrzlkZE65uoaCZafGCzFRW6REaag7vv2ppU5vvfsc1ZJ36rJBlZstHfDjcED4L2n1N+4YmzguSRGJg9wt8bGHi6iIdXIGnjFk4PCVTuU7cYRpfoUX8jm9iQz7vUBJyL0pemijr20TtYzjbJvRQw0bdLD6XRMx/S+mEJFhBN+k6mDCXh1GEhQyXzVEYmcZp0fqoczD51VMcBh9Z7YkG9XO+KRYOb5Ar1N2VfZreBFCtC7y6Gp0bMDm6S4o05R9UZ/bAuR7YQy0LZyDTQsWsHj6eloID1aDtoozjftrEs6/e3/o7HHae2LK8hU2l+ydH3XUw+QLiuauWqMhmwu80OvGf1NuwFwOTsL0kG1xADSwoSqpysvq3HX2F+2Ft4A/6cMeh5K5eZfhdlmudg5A5o5uFgGERrSK5DJY3uFlH/yOiGWi6BONJHullN6T7mQFP6dhakeQEBG0QFlFyCGXna4Bx96vp9yS9m+yFcdND0H6fiY+FSyduGa1tPM0RdSgqo1ytQkKmK1xALHpnE8KszPNrYKvTqxgxCWWqkEchIKQhp0LX3Hb7Dc6BtsGDg2hjezjdZpPIgSeKaPJSCELwEa4biO/4+LzgLHvd9piil3iWdcyp4Lf3YDR7tJZCsyf0X83DCDNmHr/LQzQEzp8pXiMkbvVK/ADtKFxpyv3Zy2s8tiy81BQv9/D7XfWL55d8FdZCJkluMdGCnU+NT7v88RaCR7yK6QOICil7GJMms+ NWZ5s80t qKvBi/NAb1EpDO8Xj/ga5xAxxOfONEjUMXl0VKcyaC7bOlxcdj/Iov+LwXxAgUp89p3qRbmdVEqMsnZho+pTuSZMSQJWk7d/MtvKK8U5hoI0wQ0ZnLlMLPUXpvFp2tNJgMn2EN4KfawLqKTTPljD5k3jmURHz5eQpRnAruGDwb3fudKEsDaCJdco72urDWPiseZLO+/jnjBoAh+f0QIrBAazuW+7k+mHVfJVgOgrHIJMdackaP5gPbiwuoaaAGnifWh9Iuog6wdQm3NRY4TuCjaZ1Ee9rNsU1AULixmBXEQX3fNlXvl3BccUSyATjnSFp8Lz9xc4lhq3Zazkmtl5DVyY+kq2GGxksf4c6L0Ok4SeO7Skvg4aDKnnFVDqpPTejDdN4KfEI2iaLlF3QGwiQ36eBwd924/sMf0GmNtBNU7UyFyU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The cleanup code in zswap_store() is not pretty, particularly the 'shrink' label at the bottom that ends up jumping between cleanup labels. Instead of having a dedicated label to shrink the pool, just use zswap_pool_reached_full directly to figure out if the pool needs shrinking. zswap_pool_reached_full should be true if and only if the pool needs shrinking. The only caveat is that the value of zswap_pool_reached_full may be changed by concurrent zswap_store() calls between checking the limit and testing zswap_pool_reached_full in the cleanup code. This is fine because: - If zswap_pool_reached_full was true during limit checking then became false during the cleanup code, then someone else already took care of shrinking the pool and there is no need to queue the worker. That would be a good change. - If zswap_pool_reached_full was false during limit checking then became true during the cleanup code, then someone else hit the limit meanwhile. In this case, both threads will try to queue the worker, but it never gets queued more than once anyway. Also, calling queue_work() multiple times when the limit is hit could already happen today, so this isn't a significant change in any way. Signed-off-by: Yosry Ahmed Reviewed-by: Nhat Pham Reviewed-by: Chengming Zhou Acked-by: Johannes Weiner --- mm/zswap.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index c4979c76d58e3..1cf3ab4b22e64 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1429,12 +1429,12 @@ bool zswap_store(struct folio *folio) if (cur_pages >= max_pages) { zswap_pool_limit_hit++; zswap_pool_reached_full = true; - goto shrink; + goto reject; } if (zswap_pool_reached_full) { if (cur_pages > zswap_accept_thr_pages()) - goto shrink; + goto reject; else zswap_pool_reached_full = false; } @@ -1540,6 +1540,8 @@ bool zswap_store(struct folio *folio) zswap_entry_cache_free(entry); reject: obj_cgroup_put(objcg); + if (zswap_pool_reached_full) + queue_work(shrink_wq, &zswap_shrink_work); check_old: /* * If the zswap store fails or zswap is disabled, we must invalidate the @@ -1550,10 +1552,6 @@ bool zswap_store(struct folio *folio) if (entry) zswap_entry_free(entry); return false; - -shrink: - queue_work(shrink_wq, &zswap_shrink_work); - goto reject; } bool zswap_load(struct folio *folio) From patchwork Mon Mar 25 23:50:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13603183 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6AAE3CD11BF for ; Mon, 25 Mar 2024 23:50:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AC4116B0085; Mon, 25 Mar 2024 19:50:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9ADC46B0087; Mon, 25 Mar 2024 19:50:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 802E96B0088; Mon, 25 Mar 2024 19:50:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 5F0286B0085 for ; Mon, 25 Mar 2024 19:50:26 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 39C5A40157 for ; Mon, 25 Mar 2024 23:50:26 +0000 (UTC) X-FDA: 81937208052.20.98FE569 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf25.hostedemail.com (Postfix) with ESMTP id 6A7CFA0011 for ; Mon, 25 Mar 2024 23:50:24 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=R1JBf600; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 3vw0CZgoKCBoOEIHO07C436EE6B4.2ECB8DKN-CCAL02A.EH6@flex--yosryahmed.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3vw0CZgoKCBoOEIHO07C436EE6B4.2ECB8DKN-CCAL02A.EH6@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711410624; a=rsa-sha256; cv=none; b=RHZye6qW16MnjzI9XHtw3j/EKbUz52kXU2TwR7sMGomPPH00EeDCpSmWxwOussL7yie1vD Nra2mE9bc3D8s72yJp7Z7VWCMNCxfFol32plsZ2otCOJt7rw3g3ZLoKZXmVhyHaaGKZ8mZ LhSpPnB7ELE08Y0MIHzhbJf3GMq8/mU= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=R1JBf600; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 3vw0CZgoKCBoOEIHO07C436EE6B4.2ECB8DKN-CCAL02A.EH6@flex--yosryahmed.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3vw0CZgoKCBoOEIHO07C436EE6B4.2ECB8DKN-CCAL02A.EH6@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711410624; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ltPcwa62HBgtxDU26adJs5Ek9NcA4wCnN/irpaDy45s=; b=ZrH0I8bGhGfLbitPAU51Ivrww5pH0Gu1j1YpmcQduw2aG5vm+Amp99XWgHgZAQbRvRv6DM SwRH2qUovEbUf2DwDxunOlPQbydaKSsrQuLoEG51LANCWGpXlj0YJLz07blBKBR7+v/XYQ C6xqMTQGJvhflDl2gLmLniIRrBeT0LM= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-60cbba6f571so93109107b3.1 for ; Mon, 25 Mar 2024 16:50:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711410623; x=1712015423; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ltPcwa62HBgtxDU26adJs5Ek9NcA4wCnN/irpaDy45s=; b=R1JBf600OGOWrvvrk3j88awfQZH+5VhixQpUc08U4RyW0OgBJD+v8ZKA4rmzLwHF6F DNO3ODQaHBy6041eiJqjYnXOWCS/SNPbXt7BZPgJEv61yhm4eUFlArxn3nifHOKhiDqk n/68CwS2aVbDUOHbms2ocKkDGUAhdO1S50xCVC4gvsUTreWmS1sG0lG4f41FkE5QVzWk O6itxxdjxaeqkooYplxwfP+OyRLiyrsZ3Pia9QZ1b1Ka802b63witUSBcZMcm3GkQ5um SlfRbX+b314SUQRD1bcsRMy58xfpvB9HbI0r+VAjhhJOkL913fOKpPxutW1zNKykMLLg qdyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711410623; x=1712015423; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ltPcwa62HBgtxDU26adJs5Ek9NcA4wCnN/irpaDy45s=; b=n7xLJ/mwi2w66gZih/WeroCIy+8sl9z8skb9Sg09v+BDvKxbXPz4BYNamY0d3RTH5p huHpfumlRuYqAc93wLlgTPpTbyqhslDOLSeyCpVIsxEPxNw39oq5gfRGwfCSPZp7PFCs DP0Yow1YStpQN53wjyx+0MDahr+n2zR3fP51fjWNx/mDwwUFbeLUFREXNuX2Qwl7wUYY 8sVP5QWoU1ovN4LtOmhu/z2OQOOeGytW2BWhcFkVoFR/M59wZu3qYsqbhsktQvCW8dcN zPXznH5sE8eNERBmU3qNg0xBq0bdoKLA1GYLc5y1VgAhOWn8ZHZMD7Q5Men89e0qeiaD pHYQ== X-Forwarded-Encrypted: i=1; AJvYcCURH+o6/efJHZkmQqORqMFkeE+6bZSKkr7ykSsSu8bwmVJC+WK3Ep581vBnVdGIYCllCXTbBWdZ81rpvDiFrxQTtRY= X-Gm-Message-State: AOJu0YyK96yVDyEI8XmxBABAlz6ScdWSSTkgylw7pt0adsHdaZzWFWNF tyrnbJJAB8cXx+uv5WawCYU1QWJ8pxbPEwZVAx0rXsOAw3diSbMADYmblwjcReuIC3SslyJuZg1 PKuOuvzy6qFFlzzglxw== X-Google-Smtp-Source: AGHT+IETdRLWcKvQU7ALPH/FxGMfk6VVMBGbdetkJernCRp2EMtMjfpphTVvqJ9nLnh8UXcxjIFAV/8JU9lCEqEU X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a0d:eb04:0:b0:610:d0c9:696b with SMTP id u4-20020a0deb04000000b00610d0c9696bmr2096495ywe.8.1711410623435; Mon, 25 Mar 2024 16:50:23 -0700 (PDT) Date: Mon, 25 Mar 2024 23:50:10 +0000 In-Reply-To: <20240325235018.2028408-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20240325235018.2028408-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.396.g6e790dbe36-goog Message-ID: <20240325235018.2028408-3-yosryahmed@google.com> Subject: [RFC PATCH 2/9] mm: zswap: refactor storing to the tree out of zswap_store() From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 6A7CFA0011 X-Stat-Signature: 518h3xeyaknmdaok6bjag18y7sekw7wt X-HE-Tag: 1711410624-487062 X-HE-Meta: U2FsdGVkX19bRK8avoBZjEAfCTArpO05bwH2ZvqRAFj8GXiH+9UIW3NctzI6FY9Y4A0q0lNk4A8aI6J8WBecqPKEER+fHytgLr2RVgTa8WGmLinED7ne/M84YF7EYeb3kKgj8N9NMGM1NlIb2IOmfNzfA6eKW3VIqJbrDbAF5UBrlXzvaVxEQVg2MVpO0FvxLwj2ILVO2Cm8lxMR1er4Icej7wcxJXbEXhRCS+1lU6qq//17R7HTA8kDEK1fhP6p/Ie5pewoTiR9/YmjDBVBOlzbEiYcoPxxuQzbx67BWS/uwnQyPIzLHIl+d6OWbvUdknSH4Z5/2Bd5GGIE6uEYurmMZWeJ3f76nOw2rwamYvnw2hCg3qlESzfd7w25+Gt2r5d8f75Cf2YaknMjPuTrUOu0DNYnN0yfJqYW6+0AuUdL/TKwfyovlFODyTwDlWkSIDClakMIrECBbqWXfjjtJs9adlocce4r/CsS3nQ65ayR4yWlJhzwNp0II5nEov/h0YFDEfIjkHoVb+VPujG0bSQBRcNn03ut07yqVO3e6drbiuwivVYWeDNlxQHmr1o2fMJQXgm/NWO9NkY89An4DuL85iL5eoFqG9n+YXz31xiSLUB81pjh+noIFOFSQC3ALPBbIeYhRK8/thP1IC+Rkw4TML9feOntZURu0m3RnM9r9JLPlEfISilsZFhjzBk8fqKEOQTcYIXUX8naSU1pTHBXDd+6ZWuwAt6240X+XxnTB4FLKkpIJcuVdh1nGZ5gyw7Vk+qivSLvaOOXQAvUmL/EKfa+k9yt8mNx4J6DGEF8Fxcb/EWl43LSHnc9LFaEnaTdwd5IxbQ6blLqCRugGluH8Lqe0XaJ4d5r+rPvI872jGVN9qWg6q9DZCztRQE2QZiyLOI+HWU2l26KgYbVQj27wO8OFzNmFWrdgcmRZ0CLkFjItCHJ3PGA1mUrjvl2ar7tBmOZE4qFDYWD9ks ucb8Bmfs +LxTiZ7AcJoCW8PLCtvG8B57wVi22A3gV1OeFPG5Fdu4OKOXY0gpVgoiiVKBaeDr9R6TOtRAqY6X0LFTCFMLCfL4gq8Bp8EkpQ8ZKyq01i392NDkAXnIxZt6wbGiZVY5nuD1tFpMshVSus98RUPpdrl3o3xb2YBAkp/3QocthIjJnHuTm/M5Vz3s0ZyMZQaTKGons+hhksaP6Sd1NWNnKV+w0IQZ5XyBCK/fJEzp5Ru/1ZM7cTCS5Pw5OI8fsFUBw/btkdPJ8UUbp67I0Itw/E2z+OUA4MEFaStstkWd3gHsEM29xU5qyl4W0n8fWTN9QN6Z4epop+SYTGzYXmGYBfnZ9744OdqIOpqHTVUm6UcvQwLVOvLHnZWa8nxXTBlCWps5+R3RTvS3fHCtn35kdGMGIKv9bjfb29+o48XtDxr4wDyF8XVCgW36oyFb3tgk68UgRq4m1cEuohh9sxEeFtDSSTuj+FX4n6iEpP12MvLrV2bkMF1WAyxM8Xl8NiU97zEEwNK8crDnrV2VVJEErDvJPkH+LeH+jmK0vGmOEmP/cyPbPQZHC8PE6vsQZhLPcmBXDEME/rDgKdOpX6q7M5oe1yhyh18Z6VnDb2H7Noll/TXGM4gPkSiZ1uNX2DtBkKBs/dyORNMuoD/9yPx5WWBbbcpxoxN7RijDrrFoVd9TGLiDb7A0gMIos6w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Refactor the code that attempts storing to the xarray, handling erros, and freeing stale entries into a helper. This will be reused in a following patch to free other types of tree elements as well. Signed-off-by: Yosry Ahmed --- mm/zswap.c | 42 ++++++++++++++++++++++++++---------------- 1 file changed, 26 insertions(+), 16 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 1cf3ab4b22e64..ff1975afb7e3d 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -827,6 +827,30 @@ static void zswap_entry_free(struct zswap_entry *entry) atomic_dec(&zswap_stored_pages); } +/********************************* +* zswap tree functions +**********************************/ +static int zswap_tree_store(struct xarray *tree, pgoff_t offset, void *new) +{ + void *old; + int err; + + old = xa_store(tree, offset, new, GFP_KERNEL); + err = xa_is_err(old); + if (err) { + WARN_ONCE(err != -ENOMEM, "unexpected xarray error: %d\n", err); + zswap_reject_alloc_fail++; + } else if (old) { + /* + * We may have had an existing entry that became stale when + * the folio was redirtied and now the new version is being + * swapped out. Get rid of the old. + */ + zswap_entry_free(old); + } + return err; +} + /********************************* * compressed storage functions **********************************/ @@ -1396,10 +1420,10 @@ bool zswap_store(struct folio *folio) swp_entry_t swp = folio->swap; pgoff_t offset = swp_offset(swp); struct xarray *tree = swap_zswap_tree(swp); - struct zswap_entry *entry, *old; struct obj_cgroup *objcg = NULL; struct mem_cgroup *memcg = NULL; unsigned long max_pages, cur_pages; + struct zswap_entry *entry; VM_WARN_ON_ONCE(!folio_test_locked(folio)); VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); @@ -1485,22 +1509,8 @@ bool zswap_store(struct folio *folio) entry->swpentry = swp; entry->objcg = objcg; - old = xa_store(tree, offset, entry, GFP_KERNEL); - if (xa_is_err(old)) { - int err = xa_err(old); - - WARN_ONCE(err != -ENOMEM, "unexpected xarray error: %d\n", err); - zswap_reject_alloc_fail++; + if (zswap_tree_store(tree, offset, entry)) goto store_failed; - } - - /* - * We may have had an existing entry that became stale when - * the folio was redirtied and now the new version is being - * swapped out. Get rid of the old. - */ - if (old) - zswap_entry_free(old); if (objcg) { obj_cgroup_charge_zswap(objcg, entry->length); From patchwork Mon Mar 25 23:50:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13603184 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6A20C54E58 for ; Mon, 25 Mar 2024 23:50:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 236816B0087; Mon, 25 Mar 2024 19:50:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 197E16B0088; Mon, 25 Mar 2024 19:50:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F2CB76B0089; Mon, 25 Mar 2024 19:50:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id CC7E66B0087 for ; Mon, 25 Mar 2024 19:50:27 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 759F9C018F for ; Mon, 25 Mar 2024 23:50:27 +0000 (UTC) X-FDA: 81937208094.17.7F792B9 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf05.hostedemail.com (Postfix) with ESMTP id B6978100003 for ; Mon, 25 Mar 2024 23:50:25 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=T2sWVUu7; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3wA0CZgoKCBsPFJIP18D547FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3wA0CZgoKCBsPFJIP18D547FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711410625; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Dd0Yn1LXqAraa72sbpCkcMKeXtgN9BdSqsW0A73YyKg=; b=Tu2yefUqA+Tl2PXF80YwJrPsjxIAYZhqGKRcFIi6Z+7wV8DbFLKwv1q+pO5U0WtF5fHmd4 BLDqlwEDNpyRb3Szx3SSmnt0dE+vnVda8fPqeLciDcHjPlyOT2PfEjK+JbdRcX4cl0bvn6 LnxJ+qpsASnTtE7WU78fDOQY0vK032Q= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=T2sWVUu7; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3wA0CZgoKCBsPFJIP18D547FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3wA0CZgoKCBsPFJIP18D547FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711410625; a=rsa-sha256; cv=none; b=cFXKlkpolkR3UGVW8UfvNWYEZHkcziD9thekZW25sa+lwzaHLXuXtePMwWzftdphXZut2I XrGDuakewa+p5+jmYrpRLyupegUq2cLdcQ723C4qpfNZmvrsLv3zYUb0T/AeiwKRKiyZHI 1p2agqed2H10Vqtry0rlJ2ERGyldBEQ= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60ccc3cfa39so74340967b3.2 for ; Mon, 25 Mar 2024 16:50:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711410625; x=1712015425; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Dd0Yn1LXqAraa72sbpCkcMKeXtgN9BdSqsW0A73YyKg=; b=T2sWVUu7zSyUjXaWBN+FAqeAg/syrm/k5XzKkQLRbQOg6ewW8HDhSbluuGuQAfxOa/ xpNX3ITy9+f2lBxZYmiyg1Vmie86xlbpIFv1T1EGYSWWgqXlpJQW93znOnD1wNjphNlz B7dGdP8jE0GELA4LLbdkuNFLvecJU8WhdC0ZCeEQb+sN0AEd3G0pCuWpnf0pgCco22hJ 525NMZFxN8Hq7QmCVzy1PypFwKBPO5XWTAbXympmgGwPUezAWWKNmDOwbXapg36ruYOM 08pLYchz7mA24p3ifAD5LE65MYpNpe6sd4hpuzYqLmlrSiBw+X7OgGlMUfWiW8haA3zr HpOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711410625; x=1712015425; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Dd0Yn1LXqAraa72sbpCkcMKeXtgN9BdSqsW0A73YyKg=; b=r+H99OAsAXh2VvORJXtbykS0hHnfzBDdqw3+JRocHQtqhCtVISXzhy56S7tHTIsL3s 9I/QUS6rTEu2sEuSHSQl2XBwOLx9vXKdf8h2PCUQnnPm5Un3/mymUdf4mf2+LNoY64wH ashSWuGpl2F7Zt74aVqMlzkonuMxJrr5VLOgTEg0e/BZE/U7iOFxNnYImkrg2iIAlpAP 4GJr3ZC8bVydQITbbZU2ZQp/+YH6xm7vm0g3vrOWdOe87gXC1B3L6avdgrWH0wDLArDw I5j55wwmAbzRUeFYY/e0zei/2k2v3kfQbwpKlIFxSkvv5tSvdnOSMO7+nZaWvk8Xhll6 5D2Q== X-Forwarded-Encrypted: i=1; AJvYcCXbVtIu3hWlnjXDzHqq7edIxz3PAEWkU+Y29EouDr8GxLS7wnfMR9WRRIrRiPHXHyzmTjdTsV3oRKWLjLArzPYJmx4= X-Gm-Message-State: AOJu0YzyH3aqRFnvI+hZECrDuC6yM5l/9Ge5R4AptIxGAydBAKrvUGif 46t9tJP8drFwqqonwTulLymWQ+7Ayt5HGBpvPyIjG8l9kBxu+l7C6qr+ZPsfyZWeNdb9eerkBR1 sz55VE4oWUfe8WgOiZQ== X-Google-Smtp-Source: AGHT+IHM4PjIMqVasHvxcW/NHELy1n3EZw0GsOSiULCSozqmucZP5yLYhHluyhz301rMxFhkk5EZKcifeTwjvilx X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a0d:fe44:0:b0:611:a84c:fef2 with SMTP id o65-20020a0dfe44000000b00611a84cfef2mr165827ywf.10.1711410624906; Mon, 25 Mar 2024 16:50:24 -0700 (PDT) Date: Mon, 25 Mar 2024 23:50:11 +0000 In-Reply-To: <20240325235018.2028408-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20240325235018.2028408-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.396.g6e790dbe36-goog Message-ID: <20240325235018.2028408-4-yosryahmed@google.com> Subject: [RFC PATCH 3/9] mm: zswap: refactor limit checking from zswap_store() From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: B6978100003 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: mr7ufd1qkcni7g96js6jb7gn4s1gz1t7 X-HE-Tag: 1711410625-478988 X-HE-Meta: U2FsdGVkX1+JOMrz557j+SoKrrD1RNAk+TBr2BJR6VfgMPbbyixZym7PWRMfMz4svTMPcrIWs7ZY4vLsMaV1xZlD4lzpjz+0m5/MyrS3CiUejXOo7KWcbx0AZouwwNYV2umtW5+deyOP5skyKUkm4SNO+um9riObw6UKk5pBwuBKalFigCX9xnjTuZ0A5Wh8KWheHKBRmVMSjFBPlUn4DE1UwFH3p06xFfDwyB/CkcGB6XgCDkEo3aic0mmwd4bxz6znvBB9ULEr7ZfQsY/UXk6MKec9SwpOxsLOo4Kg+ZcooZL8ZqJkbdNEoJY7fh7UMt8ID92zDyFWI4xXgzm4fUjC2Da4nvF5gcPgQoFEA2oK33lWG3oLNjdpAd2ZvacY1I9YeA/acOTQuk87i+QGzR7JQa+xgV+qsC/P3v7Wu7VeT6HJlsr/+DohTTZT/AQUM4NQ5Rfjgb0WrqHKrPcqkbSGFEVx6MeT/LA8C2AbRPs3vNyIcDQxFdUYvUlg640UV/YYt2JykXc6KTPJc1hpGCvibdWWN9vbK4vlni6w6lKEeslkkl5WLMvs+7wEpRG2aAJ2xp201+xWjPkf0PyXPjgc1PsvtFOZQhLGgFujmoMEHiJ6TrnbPgOP/703NFsfBPT64nJbnhubINDsBRZFzyz75bCUmY6vWJGiaQYcYxdwZsP1ZgM4Bvwz1MQEbbitQtMG+637Cszz90BwuKypOOU+Z40//VqxzYQ6WXS3d78JiToNXJ4QzOF/g7OZUdzqiA7Y2wwgv8xQxRffrMvFPIocMHg3f9SAE5T00xKANd22sVePgUsQlCwn7fzDdU6WBkj6GgbMEagTlGcXP59HJrXS7Hl4ajGulVx3/O139H0t80YjYbnPxqSZT64zrUOeo/vABzU3VLqOOuBLTie5eLXXHDCq0u31BOYcsNeLghcDOgvSCLG7OzT4Tggy6tmAqwA3u9itsVYkInOb+HB XEkr2hdU pU03+1h6IKBdsjIeQkSTh324sSXTkE5R2zkRG0a1uyDWoq/Kgr3OW1W0oKe6ssZNwdRUINPJcGWwF5aXoWDpxw+QyPaBJN/7RHpO272FS0iOZqto/9Kyl4WGWJvIIOdYxEmHy4ntivPbfKL3w0tL5lZfndO5tkfUC3s4/2rRHOVdQp5IdS1F2OSpVw51COU7kpeO4oFCOn9BKF1WbH0jQITMeXoeLL4QLlR77upkl3fm74hSPrMxvEUDNQND0g+u2gcOzSP2XyRJ9JUdImKEuPmEgqh3XNZepy5qReTW0GTZevJAGCaOYrckbfo50Pd15d/2ojlFhN5Cspo7R0zYAiy8emIMPC4/UOiWOPpEznKymALHg3roXzoo9XmaUoPIkPkaaQKg/93zDDBTfZKA0sCLWIND/6dPEQwaRmK/ygj10js+x4xr5g1yBifTKw9VZQmmaR8em2q7uAM3HdY1+lM6qj8HK43JThGCuUO6jE6okxPulxuaYFY574zI13X5qvA2mbkY9CA+UOq9sMk7IUxfdONq3zf6hz1nRyHSEvTgE8wiTpCbL19EbEP/PPaSOrVgfFX/hggjuzxPe1RC+yTfaTD/LVjTrcMH9N9Z8EqLPAv+KyEid3BeRyoIk7lKmzgbHFRRWQzkM14IP1MCnF0wZgAGPVNwvPdNhUz1fWVa2eevApB9OVkRzLD47wSzc+yDiqbAInF/k0TC/dSNVwy/boA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Refactor limit and acceptance threshold checking outside of zswap_store(). This code will be moved around in a following patch, so it would be cleaner to move a function call around. Signed-off-by: Yosry Ahmed --- mm/zswap.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index ff1975afb7e3d..6b890c8590ef7 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1415,6 +1415,21 @@ static void zswap_fill_page(void *ptr, unsigned long value) memset_l(page, value, PAGE_SIZE / sizeof(unsigned long)); } +static bool zswap_check_limit(void) +{ + unsigned long cur_pages = zswap_total_pages(); + unsigned long thr = zswap_accept_thr_pages(); + unsigned long max_pages = zswap_max_pages(); + + if (cur_pages >= max_pages) { + zswap_pool_limit_hit++; + zswap_pool_reached_full = true; + } else if (zswap_pool_reached_full && cur_pages <= thr) { + zswap_pool_reached_full = false; + } + return !zswap_pool_reached_full; +} + bool zswap_store(struct folio *folio) { swp_entry_t swp = folio->swap; @@ -1422,7 +1437,6 @@ bool zswap_store(struct folio *folio) struct xarray *tree = swap_zswap_tree(swp); struct obj_cgroup *objcg = NULL; struct mem_cgroup *memcg = NULL; - unsigned long max_pages, cur_pages; struct zswap_entry *entry; VM_WARN_ON_ONCE(!folio_test_locked(folio)); @@ -1446,22 +1460,8 @@ bool zswap_store(struct folio *folio) mem_cgroup_put(memcg); } - /* Check global limits */ - cur_pages = zswap_total_pages(); - max_pages = zswap_max_pages(); - - if (cur_pages >= max_pages) { - zswap_pool_limit_hit++; - zswap_pool_reached_full = true; + if (!zswap_check_limit()) goto reject; - } - - if (zswap_pool_reached_full) { - if (cur_pages > zswap_accept_thr_pages()) - goto reject; - else - zswap_pool_reached_full = false; - } /* allocate entry */ entry = zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(folio)); From patchwork Mon Mar 25 23:50:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13603185 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F39FC54E58 for ; Mon, 25 Mar 2024 23:50:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 805BD6B0088; Mon, 25 Mar 2024 19:50:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 78C7D6B0089; Mon, 25 Mar 2024 19:50:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5DE9C6B008A; Mon, 25 Mar 2024 19:50:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 425C76B0088 for ; Mon, 25 Mar 2024 19:50:29 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 110A4A0A99 for ; Mon, 25 Mar 2024 23:50:29 +0000 (UTC) X-FDA: 81937208178.08.5253B8A Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf24.hostedemail.com (Postfix) with ESMTP id 4CF3918000E for ; Mon, 25 Mar 2024 23:50:27 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=lS941FnH; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3wg0CZgoKCB0RHLKR3AF769HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3wg0CZgoKCB0RHLKR3AF769HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711410627; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+3t/0JKEdL4mJ/vvKrn6t1zV3oNIlUMv4pczMFevQsk=; b=fVIzdyn5bfTX8Mna9sPMpuHilYqOcBCeq0TX60x7TqjZW6HXIhMMLLSMEZFkzcvKSbBlup 3VVo0gLnOXGQirvI2pzD2BiPKUV+O3iDo1K+AtOrlTnSzTHJx33UZgxpNtVPbCgDuNtLcG kPyMjIQSpu8q+WWjD4wbAHHA27YQ+CE= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=lS941FnH; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3wg0CZgoKCB0RHLKR3AF769HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3wg0CZgoKCB0RHLKR3AF769HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711410627; a=rsa-sha256; cv=none; b=LwAUjzwmhewoZYsg2Oh0CGfp4Bdc/3Gjt2LSn8VsfGX8fRNtv/kpMa5ZuJwEEWrnNwcwHl i336HJNNBhS53miKo4FM+aFsUgRpFTgQ6zzakrTUgixdmHU0n8t7pAw9PvKOQoOqfU5T/s aVygn4k0n+WOQObW+VOdWYZIFYH3/sA= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc693399655so9148470276.1 for ; Mon, 25 Mar 2024 16:50:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711410626; x=1712015426; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+3t/0JKEdL4mJ/vvKrn6t1zV3oNIlUMv4pczMFevQsk=; b=lS941FnHlnffa2C6nL1yI0YqZiv44baydsU7kR+IOCzGJ2ewU5rxiG2nDmLIrPVMyb WgKM3A65rfR5JyfCPun+2lHXwfVnJa0QmSXFZp1N84gmrDbKQCZFpg93QtNZZPB5urhs IpMzMziE2cYCSaG88vPw5SH3I0C7gWZwlWxLzRFJxhhGdtopqKIicFnQonTBnkqm6mio FkolHPEd5ud4VU3koHEHfrQebm6F3b4Tg9p21YVJo3dVSutYPR1IsO9sbn5I2Z0g0Y5L SYxB1DGZqm4KkwLotECl4z1aW+SupHbafuRg+WuecgyZh3Xu9njbcXdECgPl7SziSzTn 7ANA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711410626; x=1712015426; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+3t/0JKEdL4mJ/vvKrn6t1zV3oNIlUMv4pczMFevQsk=; b=t7C7yB/BjjnoD7LFjGQw5/jKT6KUrtSfbgZwejLSrmukcc/syWm0hQg65r2377FBqD P5ZVlHv/prm/6ba3EeeRQ9n1sKgWVtChr3BAWOJpYck0oGSyL7eV/nzcBS4lH6XT8u3N v7H7H9PcFrLvfW5JZz7azkTpMT24or8ixFHG63Nsxdu5XhV0Nmz0UeOQ8og7hUKCYuQo n3QKZhM6cteDY5XYKcR89XK5/M6jSA2jC7X9P0HK1OoWPPrgjxj61AtG0SLpd44kx8E7 kwXL79pZSdPJt/Gm/I73X/IiTK9xNxNp+fcLDvUuUS5vtcATG1QTWZVOjpnuXMvY/daa nlkA== X-Forwarded-Encrypted: i=1; AJvYcCXUhk+EMynjott5IAjy76OaV57w0+0NvS5NXM9U8nJJ9JTIgh/oNT2oZlZUtWMfnPbupuN5vaxRjfxOHwI9tx7eC4E= X-Gm-Message-State: AOJu0YwLUMY8uEXg0qDclRPS4JhvOUKxuQ6sw2a6Eu+CXOxdwaFUgQHc Dzi0zX1s5HjrYlVXyE10m5N6DUjKWT/gdB2XT6h1x0fVbiP56XrPYczgHkfTqUveHYO2900R+/X +mW4N9SFjCUfg88MOng== X-Google-Smtp-Source: AGHT+IGK+u8xbKSjmKMyfk167gHIuEXTDEJgFDPVRTLSrDn75a8RPhmRcxr7pMVWHrDhD2ZGkE5Banp7zX8fMMUb X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:18cd:b0:dc6:fec4:1c26 with SMTP id ck13-20020a05690218cd00b00dc6fec41c26mr2628277ybb.1.1711410626525; Mon, 25 Mar 2024 16:50:26 -0700 (PDT) Date: Mon, 25 Mar 2024 23:50:12 +0000 In-Reply-To: <20240325235018.2028408-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20240325235018.2028408-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.396.g6e790dbe36-goog Message-ID: <20240325235018.2028408-5-yosryahmed@google.com> Subject: [RFC PATCH 4/9] mm: zswap: move more same-filled pages checks outside of zswap_store() From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspam-User: X-Stat-Signature: kobauhgfqryh1ahm6m5hfu7d4zanntxs X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 4CF3918000E X-HE-Tag: 1711410627-700690 X-HE-Meta: U2FsdGVkX1+sQ+v3VQYsu5Rxly7GWcYGtv0yVukn2vveaJzhqsyHFPtP3UA9TeHmeXUvHOTV3HftF5kKkFr50rWtLTsTXoGsZupgOGVR3t10AAPlYuvdIbbCfbsxvXp5cQICxkKJ6uNqsjpA1lYJKmhaPZN2BJxHwN8kKjupfePynhREPekWFqJsYTQL9FYKLEd99aLnQRpgKy7MefVeUkzc96xT/4cTjbtesY1GUqsr/ZkHFZIN5Y9OXdiw1ZR92HHl/a5AW5TDuttq9R2b/aeAbJ3nV3+VbQOe4P2j+LGsI1LQ2s67SwA0xuCRaVLa2ik62RLodJRXPhiKIf781z89hvR5jL4PWKwdISDV/pJVnyN40i/+BYV1njKAAWK8rLfXyqDtYpKvRcGSuZ1Ses2zuUFi6E5pd8OR7ZgFjsWRL2MtqLgud/baBLHM7sAMQt6DqvsGHdMLR6KK8Hwqb6+MxERB7RMFf2RM8ateC2/PGVOF8xwYbIf4lb6hRr7tXnYYci9X9+BF0zZr5i3cn+tiZgJkAhA9QmqimLvF+x2Cu16C3+hULsYiMoaWd67RoFXndzrF3eeHw5/kNw+8LZCpV6N1bO+QnfrI+jlJ09XJVTGNasu/uOjV45o2lfcx+n2qjKCgoUgiEgSYbpxcRjmJw7pAL2T+oJ4wAezJYDZxO2FJkpYhZf+Vkw5tVBouUFSv/MPo59/fysiEVOVQ/kGG3yys2qTDHEmG4+mqkT0INF//6AQ2Io5Q/Qny1OvghzdyEOgtBo1Ou4gvIXHWBy9yAjy/xx4m9qNfGRJKcfjZ3T3B+zgYCLiIBgvAw7IP5ZUN4WJknJIXFZJtLcIodUZh+0i8z9rToIKoBWOIPS6mWb3OoCa1/GLpLvblItb+7zIxW41YMDTq5Yce7x6EpnYDCNvT5uK7We+gZmVkqtedrLiZApqkwCFprFDWUbspojte19h2VLeWawc7uBT 4GuiWx6V 1rp7DGxwDSggBbYhMskUQ4MlKma61/OCVAp6I/dUO4dX6+V0ZbCU5p85H2/sHn0g4/IoAZHJ/DvJJFFK8m657Y7s8yAXKz8viw8eeBCZ+G8g3VpANb+2BC64EYLZ8jVmQ1bRP8VB/pp7jcVAmT960W62x6RDKdgiobmAvFXfHfrfwcwXKVsXP5Kgs9FzYxAoWGJYGs9ljQGcmVBWb9b1BM0XhdruU96yngO/QjV8bnNCncLyrbkJLdXexk2jweZY9c3Hup4ITtKvtK1sQVwedR7qiowS1ZTpOEoJNfy0TAAe42CVhrCTM8L1iw2P0ooASHHdKpRHRC3j2eZMJOBc8pKJNLjmGe3fn9VfU4Zx4wCaPFkyj7UIBYyrY+Oi0VwZ8u4OiDHCQYTL5l8EIrcYPxmRDoZ6i0uD14cqiNTV8nB1/rKHyZ0wCaQ/1NqJ9+QxDDZQnaG65L3+m2sHF+abG/7d23i0RqeTIflOUoOFBt2nBvgDvBSY5ZnDaiNYnqVvXi3FHMglVTwolRa7gCXWhUloBfOsTls+NTuj9sXMBHzxfLNt+qNGHC5V4JcGcFRGT8/8JaezqSloCyA5g7rn1uXL5GmqH7M7mCwLAfmnt4S1F6wYB79b8dBqL/ajGhs4GaDr656uI63tksIOMSnIDHqUUTraU9Kwld2PdoW071juJAE5Bc3PO3HEF2w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently, zswap_store() check zswap_same_filled_pages_enabled, kmaps the folio, then calls zswap_is_page_same_filled() to check the folio contents. Move this logic into zswap_is_page_same_filled() as well (and rename it to use 'folio' while we are at it). This makes zswap_store() cleaner, and makes following changes to that logic contained within the helper. Signed-off-by: Yosry Ahmed Reviewed-by: Nhat Pham Reviewed-by: Chengming Zhou --- mm/zswap.c | 45 ++++++++++++++++++++++++--------------------- 1 file changed, 24 insertions(+), 21 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 6b890c8590ef7..498a6c5839bef 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1385,26 +1385,36 @@ static void shrink_worker(struct work_struct *w) } while (zswap_total_pages() > thr); } -static int zswap_is_page_same_filled(void *ptr, unsigned long *value) +static bool zswap_is_folio_same_filled(struct folio *folio, unsigned long *value) { unsigned long *page; unsigned long val; unsigned int pos, last_pos = PAGE_SIZE / sizeof(*page) - 1; + bool ret; - page = (unsigned long *)ptr; + if (!zswap_same_filled_pages_enabled) + return false; + + page = kmap_local_folio(folio, 0); val = page[0]; - if (val != page[last_pos]) - return 0; + if (val != page[last_pos]) { + ret = false; + goto out; + } for (pos = 1; pos < last_pos; pos++) { - if (val != page[pos]) - return 0; + if (val != page[pos]) { + ret = false; + goto out; + } } *value = val; - - return 1; + ret = true; +out: + kunmap_local(page); + return ret; } static void zswap_fill_page(void *ptr, unsigned long value) @@ -1438,6 +1448,7 @@ bool zswap_store(struct folio *folio) struct obj_cgroup *objcg = NULL; struct mem_cgroup *memcg = NULL; struct zswap_entry *entry; + unsigned long value; VM_WARN_ON_ONCE(!folio_test_locked(folio)); VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); @@ -1470,19 +1481,11 @@ bool zswap_store(struct folio *folio) goto reject; } - if (zswap_same_filled_pages_enabled) { - unsigned long value; - u8 *src; - - src = kmap_local_folio(folio, 0); - if (zswap_is_page_same_filled(src, &value)) { - kunmap_local(src); - entry->length = 0; - entry->value = value; - atomic_inc(&zswap_same_filled_pages); - goto insert_entry; - } - kunmap_local(src); + if (zswap_is_folio_same_filled(folio, &value)) { + entry->length = 0; + entry->value = value; + atomic_inc(&zswap_same_filled_pages); + goto insert_entry; } if (!zswap_non_same_filled_pages_enabled) From patchwork Mon Mar 25 23:50:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13603186 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09A85C54E58 for ; Mon, 25 Mar 2024 23:50:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 36FF96B0089; Mon, 25 Mar 2024 19:50:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F6EB6B0092; Mon, 25 Mar 2024 19:50:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1716E6B0093; Mon, 25 Mar 2024 19:50:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id EFCE26B0089 for ; Mon, 25 Mar 2024 19:50:30 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id C8C49A0A99 for ; Mon, 25 Mar 2024 23:50:30 +0000 (UTC) X-FDA: 81937208220.01.19321BD Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf19.hostedemail.com (Postfix) with ESMTP id 0F1ED1A0003 for ; Mon, 25 Mar 2024 23:50:28 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="PDAk0Ax/"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 3xA0CZgoKCB8TJNMT5CH98BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--yosryahmed.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3xA0CZgoKCB8TJNMT5CH98BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711410629; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6rS+9vqlodSMLG0LOLi3GYviKuQ0ce9GmEUN+BZASUg=; b=FXjVVYlnsER9GnTvluX43ufNmmEWnb/XbuCJWjjEH+Fg+U6kCw6q7mKODPjdbLFryQ9gS+ XvdBOeyjAczq/2VbOR61PcI2gyyUoLpo3jnFcKuqf/PFrAALQ5h/cQ2lXmJxwPVEFXtDJq 79bnabGNJFpA9/w5l8Bh2lIPyPo1d3I= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="PDAk0Ax/"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 3xA0CZgoKCB8TJNMT5CH98BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--yosryahmed.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3xA0CZgoKCB8TJNMT5CH98BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711410629; a=rsa-sha256; cv=none; b=YyAUic/rr5wcMS7IViXn25Am3WSWAy1raTMYZpr5WKNxTWF8TupqVUZGC3JTOimv3ihmqJ UtmyaA0HTLxYKxYeLU5FlGektQbWoaVjFRevTCh6HOulovpqBRIkkDOu986qlyLaL4aa28 tW7JxqJE1Dlh15gthd78o3l56Sekz9Q= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc6b269686aso7529481276.1 for ; Mon, 25 Mar 2024 16:50:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711410628; x=1712015428; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6rS+9vqlodSMLG0LOLi3GYviKuQ0ce9GmEUN+BZASUg=; b=PDAk0Ax/obFn0Eji4GCSh2vtwHDZkMbCR0YW7RRUe3gPeqCGJoujJGUfXteVVFrQfK GTP3EqjQKO59CAnCGGeYnI+bdwRqa0eLxv/zpXxaN3Ga7Qk3HnSVLNlNtvWOzdRKhZDe HRHw+0jYwmkjzqaZjlkoqWhVCsEFvsGirxbc7YtLEluFGFg2k4C4KIABP/kCoIVNMxkR zoYLpY4R9CIZwn2eJwc1XJXCU6haMNkNN/uGgCV1lCsy4BZw79rk6EOunbU+QKOH4k8g LXBoiUsqUrJ5dq6jigq1To2X1ejC9FmQt4sz+g/3fg9KXeIx+eDXt9JK4hP52Vz84AhB a/Dw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711410628; x=1712015428; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6rS+9vqlodSMLG0LOLi3GYviKuQ0ce9GmEUN+BZASUg=; b=JfvRpn5QUHcyTGiq14OMFMpuiF9TE4tPvAmRlQfC44KErw9we/IrBL5epQlCijSUCG xgEvQwzkR6sbJUikc9WBqhdk27hh/3jp3YH/6A1khPr4tGupvoqxYfUoM6xvYgdtlfJF p3puIpcdGv73fp9rPt5JlMtYEJ00Zgjrf49Cp+OWtfDR7GGfuxU66V6GOOEcWLCxSjFS oGmefMcFAlPpsw5z4Dhr3a4eK4zLj366LCQz8hJTioSpJR7JyxGCC6XxJRAmbZ4bWfr4 9RkWE3mfSf9u3H9aHdt7uQRO7QtDZtvRNRi2v4a8GlK3hr5jVRz2mPOYmaNus0KhanGO npZw== X-Forwarded-Encrypted: i=1; AJvYcCXFWa0vv2Bgo8S7mB9bF0TN0kG3zo6zcWNZ+SIFxGweSdvaJnIuBUjyL1orH9jWOcBWcRWSzyEzP3GNFUFBSsYMSBQ= X-Gm-Message-State: AOJu0Yy4qa3xGFN8Py3Y5RWBKxmcChAmD9xQTyhRcjg2WOMu2Vc+Ooxl MNbz0S2WUmm2ZPsSUxMufkXPoBQl8s1mOsgkkumtaPCvyNa8hGsHUQK+cc7fyNQP4Xa9H9AR9m6 7cV6LxwSbnLwuEHzgFQ== X-Google-Smtp-Source: AGHT+IGKv5EFMnzQrdwPt+dy51PJVNwrFBzsXHDzxlAniqnpd5xrXPYCvCMKdfAD9j2nV5K8TiJsS4lrUK/HiNL3 X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:e82:b0:dc6:dfd9:d423 with SMTP id dg2-20020a0569020e8200b00dc6dfd9d423mr371033ybb.3.1711410628291; Mon, 25 Mar 2024 16:50:28 -0700 (PDT) Date: Mon, 25 Mar 2024 23:50:13 +0000 In-Reply-To: <20240325235018.2028408-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20240325235018.2028408-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.396.g6e790dbe36-goog Message-ID: <20240325235018.2028408-6-yosryahmed@google.com> Subject: [RFC PATCH 5/9] mm: zswap: remove zswap_same_filled_pages_enabled From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 0F1ED1A0003 X-Stat-Signature: q6z51ct4exmspg1e5f7e6aqy6z6nxo8t X-Rspam-User: X-HE-Tag: 1711410628-81093 X-HE-Meta: U2FsdGVkX19cX4uTTOklzy4uX0JYP8T6GaRbl8i69JPrHSGxs5y2me8DGq/jwbyT5z9fYuoVsjLUca0C3xe2jobkVeRuZ5Pz3cq/5o1A8qZvS2vvffaOm0Q2mjRzbnXB4G+KRGauNL/AWGkeyk7Qg4INQnKO+B8hvmjCZlr3DSx7XzYQq23DExGe6YEwuq0cR8NBmqmNLkMFPlxdH/qVjPSFPYr8BfE9CW9zvS7uG3UuuKrZMHIGdhFa6aY095vhAc1LWUtY4UVtbGZhp+43601Pl2zUO1IRniRXniaz0wMwbYiCmVVYoHp9T2gqUYSYJarB8oMuMmJZSIhXRNHBjg0c/R/4yIEJUEPamyCcxEmOBWY1VkLrqajLw1xqJkjtqZ9uy9K8aarO3er9S39x3jq6RMC0c3vdKZVG0umZVrQKl6SiyCdBY7NWJimT3DIx78XoM5gZEukfYzIiXRXaWYOzzsA7NZRDOq3RrMtWv14Hea8zz80JSdPnRRpNbYS8C+vzIGJRxLHfMuvr74zpUCURKofad0Tm+jKX6KDuMTnX3c7fP9ap4dfWPW4B2aQ/vRlrgIzNsV9HJzzhIlGzTua9gcIYVSCxKAL1R9DBv8cD/thBNYOsjYhgNfuoHGHKUtQ0fCpfaiB1am9z2tTfc0ITkQ/2/9zq7/eW+G2s+NbD7U6W0xwZI7c33iKgTkGTeN9bk0dG0WcS/LJZ4Tl0Kj1UaHlDLR6cZ34bckfk5fA0UnI9SIbjZx44mNSfMDkawNPZyYuhxwhH4wT+B49jhzoItz0QKSpU5ctmao9qUtSn88z6zeD+9QJLXinm2yCHftSZBdMwCX2sKqo5/GP2F3dguZT8vYWezLkYlVJPsDvj5tyIxod1KzeAhDmKNp3k0Ny5QoGR2qRTrnJnnizF+bB9qJOCPLo1EqEVbxybfJgGERp5Viov0XQdUJPyNhwq61MR8tEfav57A+3j4ry +FaxjmU0 in67FR1/3oIlV7Mbn08t6ZuxZEmxibmx01giNfp4LjWW+Zqkq0lrTSceRdj6VnGOGElngDGgMQZXaJg5X2RCkfejZm+P+fMZlKlMFUna7kpnItqioPO74wHZjUP7TnwP5+PAZVZPJ9ZUValyxKFLqZlc4Vgc+3BduQhbS+ZwFwsI49REezKuzA9j72vcASqbrRFDYTt1nKFRu6iF3k1EFkaQ1isnwL+ayX6zOgFNy2PB0DOjqZuTP/4V2dA1LygWu/YM4Q3RCB+LrTwF63kyP+N6Z6szHh/yD2i/RVsKkc7ylcZUdpdIMu82UjybT97W5G9jEbcB6Y2+Ezqm5kko/lyqHaUse5joHGAhiKsYw9HzBR4tqpPAfFCjRXtLWRQFqqO6JuBREfAoCIA9TKS7D+fXkoPS0V0b53foHyg2krkVqMjbMPS1qGWQCF4Lu4MycbX+ddYJa2dg0Iqw1HRys4f2uGUg07twjAROdK3wM5fDE9Wp41pMws1/D9qkhgzFteMMRmNRa89u9qHUA7/EE/ajW0Qk0ekICS2jGt3jlA9lHimoVIe2cmVCl+cx02H7mM+iM+0ye+fI39ofz0b5KAEtf2Ew8y80C6H8sCT5HG6h4Gskb7sTbOpn22KloVyjic/8lAxeXTHe+smXJH2J0lH5C/q3Di1PIRbUqwgzz4mbi0ABcbDD9trVo6g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There is no logical reason to refuse storing same-filled pages more efficiently and opt for compression. Remove the userspace knob. Signed-off-by: Yosry Ahmed Reviewed-by: Nhat Pham Reviewed-by: Chengming Zhou Acked-by: Johannes Weiner --- mm/zswap.c | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 498a6c5839bef..0fc27ae950c74 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -123,14 +123,6 @@ static unsigned int zswap_accept_thr_percent = 90; /* of max pool size */ module_param_named(accept_threshold_percent, zswap_accept_thr_percent, uint, 0644); -/* - * Enable/disable handling same-value filled pages (enabled by default). - * If disabled every page is considered non-same-value filled. - */ -static bool zswap_same_filled_pages_enabled = true; -module_param_named(same_filled_pages_enabled, zswap_same_filled_pages_enabled, - bool, 0644); - /* Enable/disable handling non-same-value filled pages (enabled by default) */ static bool zswap_non_same_filled_pages_enabled = true; module_param_named(non_same_filled_pages_enabled, zswap_non_same_filled_pages_enabled, @@ -1392,9 +1384,6 @@ static bool zswap_is_folio_same_filled(struct folio *folio, unsigned long *value unsigned int pos, last_pos = PAGE_SIZE / sizeof(*page) - 1; bool ret; - if (!zswap_same_filled_pages_enabled) - return false; - page = kmap_local_folio(folio, 0); val = page[0]; From patchwork Mon Mar 25 23:50:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13603187 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56FC7CD11BF for ; Mon, 25 Mar 2024 23:50:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 36EBB6B0093; Mon, 25 Mar 2024 19:50:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2D1856B0095; Mon, 25 Mar 2024 19:50:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 124966B0096; Mon, 25 Mar 2024 19:50:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E3C506B0093 for ; Mon, 25 Mar 2024 19:50:32 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 8A4071A0324 for ; Mon, 25 Mar 2024 23:50:32 +0000 (UTC) X-FDA: 81937208304.20.8B362BD Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf13.hostedemail.com (Postfix) with ESMTP id B8C5B2000C for ; Mon, 25 Mar 2024 23:50:30 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=efl+5ABz; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3xQ0CZgoKCCAUKONU6DIA9CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3xQ0CZgoKCCAUKONU6DIA9CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711410630; a=rsa-sha256; cv=none; b=IKIW/htTgCrmT0L5CSS3NzhWf3k9kr1FNkf943AE9y+0Nrzzb/vymEXPQfizRkVGMT43IL /CVcMJYdfXdj61ewZHF9QJqsEr4jwL6ZVliOrcZf6VOH585veuKIjyREmW7ixVP+Q7tt3Q qv09zuumb4qAKpRQcERSeIZrQBm535g= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=efl+5ABz; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3xQ0CZgoKCCAUKONU6DIA9CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3xQ0CZgoKCCAUKONU6DIA9CKKCHA.8KIHEJQT-IIGR68G.KNC@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711410630; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AaCCk+fj5n3OUjEVk3zw3Mi2iE3P+5rpOrMtgAMtN6o=; b=h8HMWNfTFwdOdtkCl51Ow2lPh6iGhtcVSn034qelAzYY4YqDwUgfaztcLT3tv5NxvqKc1m U+uHZGK1KI9Otu+m+nlHbaAUh4PApDiDVzmscdPgjXoffe2aaB3C9WArIamxNg0VrCwhSS P677QbhkKWT3XWCSUI4bfwQ12VrNtpI= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60ff1816749so76925737b3.3 for ; Mon, 25 Mar 2024 16:50:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711410630; x=1712015430; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=AaCCk+fj5n3OUjEVk3zw3Mi2iE3P+5rpOrMtgAMtN6o=; b=efl+5ABzGzdcBZdj+Bc5nRkUHaZ6aG9qldmAOTAhldyZW9Wv8bNvns093FZToDII6y fxSFCAIS63VRxIT/241TXqrzZGcarX5j2D1ayXRzOPTO889+JztgEolu/UCiMd9EYg23 aRiTUljTpdr6FozERCyIVSTm1Rerso0LQh0J5TuOf90I+9CqIwnO8pTusen80dreqsg9 D9c8zfpjtP91kk2SVZwEkxF4i/6k7KB849drAYSW2FLfWo6+mJnHU10yTbdAitf9m28e JyEaYFML9Sb22DYLLRB9OWgQayndoi7zfXssMGqhwmkZSQ4b7LLyCQCky9GwJlJlH4o0 CArA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711410630; x=1712015430; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AaCCk+fj5n3OUjEVk3zw3Mi2iE3P+5rpOrMtgAMtN6o=; b=X4c8Yt9/MJF0cc8dlEMzWViZ+UEgfUGtGbzG3AC/x2LNe4Mpdc6GBzfQsdbTmO/od4 i94gWrAdJ/JuxyYKJFWB/eXKoF+8gtx7dWEF8yIHztwSwJ3usJl0Ij6zHzc+zDihyc1X 3cQbrxWmFGBHpcHSHwqBESZphrw6J/PGvD63j3NX33iLfWnAzKtC5Qwh62I9XIPXI2YN CqmOdR3PJhuBC+rK6A0Y/sjaTh106RWQc5XMFxGUcQrjtHtEmW5XaWXm1IEP8LqURdaI ehofzTvZrf9LhdjtCnZdqcxmNkA6ItTk6g62CkjlaxmY7jH6g8+fGed31HURtp+h53/r aBtw== X-Forwarded-Encrypted: i=1; AJvYcCWk1XLwi2ZDaPVsHC/ve7sB3w/X6+gaiPCB86jzIkfN1qKfjQ6LJ2CHGYCeiR39FEDy+4ulKltbt/vndUYZ6G+uuLs= X-Gm-Message-State: AOJu0YwFmSpo8bJOHD+70ntgi4z9OiSOW8U1w0bYqaKH5x2KBRqfmUx0 //b+LsO0edgaKpIQVddUDPvsNyFGf1HZ4s02MPY+d4NK9Lq/n64KXqMF2c457Dvunf4Y3DBBPlf N192tfuWc6VR/y5yZjw== X-Google-Smtp-Source: AGHT+IEOvQVqxv2Y0ouP0PCwiv8cpiRAswb4b3lmfWQXPAhEkV469xAOUnqSRda9DyCx6d0T4Sypd2qE7f/9L3CU X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:705:b0:dcd:59a5:7545 with SMTP id k5-20020a056902070500b00dcd59a57545mr451342ybt.10.1711410629980; Mon, 25 Mar 2024 16:50:29 -0700 (PDT) Date: Mon, 25 Mar 2024 23:50:14 +0000 In-Reply-To: <20240325235018.2028408-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20240325235018.2028408-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.396.g6e790dbe36-goog Message-ID: <20240325235018.2028408-7-yosryahmed@google.com> Subject: [RFC PATCH 6/9] mm: zswap: drop support for non-zero same-filled pages handling From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: B8C5B2000C X-Stat-Signature: qyad8zc7ok6tm66n6r868k6qk7xeonaw X-HE-Tag: 1711410630-8131 X-HE-Meta: U2FsdGVkX19eaX3gawRTqQ8RF4gRLz+mRfsvSSMfTGmeVlzeRfX4u+9kYwJFBrQIZfwh4N1kLq1Sokyy25LgeYM0KLoh3haCLXz+z50ePGiQAqGAYHJsNjbSyG2y4rsMN63+jFNuZT1fEPUVEoR+VUDNR2HLA5oDHUHKtjWY9uBbxxEyLH3doWZo/qS/Ac93L26TFuW7UluQT6Ic4+Gqxy0HagXuBJONaKqDZPPt5qnr/IQ2tHHViE6lE+c9EL6YfsHmPfrKTXXnElbUmOXiTfIV800Yg9HOKE3qmWpotBBuplZNMyjDp3sojanBlY5rRNcn3NE4PryKMd1uVZFade18ANL3WzmZ7OSR3iKV93GZfs37QmNLQk326PKi+jei5R2WkQLqOebJTOOYNfDi9x5spWvYyLQeTbEla7ReEkAwPSDs1V6yyGgsjA7KcK5W9A8AFr0y5Kx1zxuUaXYI88ie0KQnm1GPojlEQmppmHCiMEUCV/rtaB+n8D/ezoymRKFORAiwKfBG3k4Sre7Dgs3mz3DdXf1DWV2jRAPJvl3JVj09Hh0CiIBiinl3FL+hwC/2cTjs46/1adoLiyxK1xXKrPSPbKB+D632xUV/2KbO+Y1z1RP0bkgBwCvVZj29AMsKg1O72YadNA1Nr9R8ktjyFoip47qx3ul5CUyTpTD0JQq63q84hn68wMBGE3YmtDzWEKP4On5EzHN0Q0dG3hWHf3Cbge+HvlFQalOpZoxNSkBU4uR9PzGZyyIc0IJytjNN/60yBoKak/o82EbV/6c5/WVWZM0h8oX9XegWLsBSHj51V0sRMZ7bU5Xkjs5WKga29PMTiibyjQxKJBWWMDur8XXObt9Sly7lEBhVlgq4Re//IBe3QL0adVMErAj1wj26v/VP7lGF5FRBzTCEQsn+LvMWwitVn/BJwCd8iWcwZZtizQy7tr7wx/0Fh1BiznLgKX4LIamDbzaPkvF w+zec/JS Rwo9EEXYfEScoID0dbsWmlQSDYq2m4NM1eTBOBkLW1qvMCBaF0Wwq9OEVBlpUB1uL+lTmqLcHkM3oc/nEv00Zgm/uppPeshlEXeZgqFmzZhgJz3HEA5YtmJYsNLxYE7lV03FxfVbo8VeImqkRXAU05VnuXaHQy8reAGGGH0YeKHfmqfcabBdGV2zruBMD1pxMssIFIxfzWNs0XKMRh4U6aJq6wWi9y4qIdE4B49HQ5T/lS+ZA3V5P7ELsiHC/onVU524iw7nixp4EcHJXVnhYlIpRVrVyUnoHwlEtoUZqL8U95tLu3hgFlevMf+4Gn5U6U45AB3ZYwtZJNdim+Niy484EiK03o3tLR2hSANACXD+5mpft+2Jqgcf1NLU2SR25R6nqoJnln1mMQQik+sJlH/KfgQLrlgHSAM4g5VzAvPwoc8d+sC72Qr8G9iZjaIfVePOTD6YO+Olwh+t1QOGJYIKlzgpkTruFqjPcsc/9hqm1LNUsv5H1k9Yp8UNfbOD4okUlmlXjK8usnUn76FS0l/jIt0PtApQtu0TjYtZj2pwwPH0JEj0JIUQgAgPigWj/UNJc7SxNKGGDCJJN/+nqQBnHv/aXuH2dTtRUWKUC/KrcIQ5BN6Km0z1iGv+BxKi9DBgeB0Z8aezwrp4MixB+icI9Ql328FcFftXjaZw5HUflRo8K6d57W1BFzLLatG9XAtweduL3Rk4usRrBxWxPbIokr9OqfQ72/fuAjXMoT3NrcDaIwfxGPeFrRg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The current same-filled pages handling supports pages filled with any repeated word-sized pattern. However, in practice, most of these should be zero pages anyway. Other patterns should be nearly as common. Drop the support for non-zero same-filled pages, but keep the names of knobs exposed to userspace as "same_filled", which isn't entirely inaccurate. This yields some nice code simplification and enables a following patch that eliminates the need to allocate struct zswap_entry for those pages completely. There is also a very small performance improvement observed over 50 runs of kernel build test (kernbench) comparing the mean build time on a skylake machine when building the kernel in a cgroup v1 container with a 3G limit: base patched % diff real 70.167 69.915 -0.359% user 2953.068 2956.147 +0.104% sys 2612.811 2594.718 -0.692% This probably comes from more optimized operations like memchr_inv() and clear_highpage(). Note that the percentage of zero-filled pages during this test was only around 1.5% on average, and was not affected by this patch. Practical workloads could have a larger proportion of such pages (e.g. Johannes observed around 10% [1]), so the performance improvement should be larger. [1]https://lore.kernel.org/linux-mm/20240320210716.GH294822@cmpxchg.org/ Signed-off-by: Yosry Ahmed Reviewed-by: Chengming Zhou Reviewed-by: Nhat Pham --- mm/zswap.c | 76 ++++++++++++++---------------------------------------- 1 file changed, 20 insertions(+), 56 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 0fc27ae950c74..413d9242cf500 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -44,8 +44,8 @@ **********************************/ /* The number of compressed pages currently stored in zswap */ atomic_t zswap_stored_pages = ATOMIC_INIT(0); -/* The number of same-value filled pages currently stored in zswap */ -static atomic_t zswap_same_filled_pages = ATOMIC_INIT(0); +/* The number of zero-filled pages currently stored in zswap */ +static atomic_t zswap_zero_filled_pages = ATOMIC_INIT(0); /* * The statistics below are not protected from concurrent access for @@ -123,9 +123,9 @@ static unsigned int zswap_accept_thr_percent = 90; /* of max pool size */ module_param_named(accept_threshold_percent, zswap_accept_thr_percent, uint, 0644); -/* Enable/disable handling non-same-value filled pages (enabled by default) */ -static bool zswap_non_same_filled_pages_enabled = true; -module_param_named(non_same_filled_pages_enabled, zswap_non_same_filled_pages_enabled, +/* Enable/disable handling non-zero-filled pages (enabled by default) */ +static bool zswap_non_zero_filled_pages_enabled = true; +module_param_named(non_same_filled_pages_enabled, zswap_non_zero_filled_pages_enabled, bool, 0644); /* Number of zpools in zswap_pool (empirically determined for scalability) */ @@ -187,11 +187,10 @@ static struct shrinker *zswap_shrinker; * * swpentry - associated swap entry, the offset indexes into the red-black tree * length - the length in bytes of the compressed page data. Needed during - * decompression. For a same value filled page length is 0, and both + * decompression. For a zero-filled page length is 0, and both * pool and lru are invalid and must be ignored. * pool - the zswap_pool the entry's data is in * handle - zpool allocation handle that stores the compressed page data - * value - value of the same-value filled pages which have same content * objcg - the obj_cgroup that the compressed memory is charged to * lru - handle to the pool's lru used to evict pages. */ @@ -199,10 +198,7 @@ struct zswap_entry { swp_entry_t swpentry; unsigned int length; struct zswap_pool *pool; - union { - unsigned long handle; - unsigned long value; - }; + unsigned long handle; struct obj_cgroup *objcg; struct list_head lru; }; @@ -805,7 +801,7 @@ static struct zpool *zswap_find_zpool(struct zswap_entry *entry) static void zswap_entry_free(struct zswap_entry *entry) { if (!entry->length) - atomic_dec(&zswap_same_filled_pages); + atomic_dec(&zswap_zero_filled_pages); else { zswap_lru_del(&zswap_list_lru, entry); zpool_free(zswap_find_zpool(entry), entry->handle); @@ -1377,43 +1373,17 @@ static void shrink_worker(struct work_struct *w) } while (zswap_total_pages() > thr); } -static bool zswap_is_folio_same_filled(struct folio *folio, unsigned long *value) +static bool zswap_is_folio_zero_filled(struct folio *folio) { - unsigned long *page; - unsigned long val; - unsigned int pos, last_pos = PAGE_SIZE / sizeof(*page) - 1; + unsigned long *kaddr; bool ret; - page = kmap_local_folio(folio, 0); - val = page[0]; - - if (val != page[last_pos]) { - ret = false; - goto out; - } - - for (pos = 1; pos < last_pos; pos++) { - if (val != page[pos]) { - ret = false; - goto out; - } - } - - *value = val; - ret = true; -out: - kunmap_local(page); + kaddr = kmap_local_folio(folio, 0); + ret = !memchr_inv(kaddr, 0, PAGE_SIZE); + kunmap_local(kaddr); return ret; } -static void zswap_fill_page(void *ptr, unsigned long value) -{ - unsigned long *page; - - page = (unsigned long *)ptr; - memset_l(page, value, PAGE_SIZE / sizeof(unsigned long)); -} - static bool zswap_check_limit(void) { unsigned long cur_pages = zswap_total_pages(); @@ -1437,7 +1407,6 @@ bool zswap_store(struct folio *folio) struct obj_cgroup *objcg = NULL; struct mem_cgroup *memcg = NULL; struct zswap_entry *entry; - unsigned long value; VM_WARN_ON_ONCE(!folio_test_locked(folio)); VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); @@ -1470,14 +1439,13 @@ bool zswap_store(struct folio *folio) goto reject; } - if (zswap_is_folio_same_filled(folio, &value)) { + if (zswap_is_folio_zero_filled(folio)) { entry->length = 0; - entry->value = value; - atomic_inc(&zswap_same_filled_pages); + atomic_inc(&zswap_zero_filled_pages); goto insert_entry; } - if (!zswap_non_same_filled_pages_enabled) + if (!zswap_non_zero_filled_pages_enabled) goto freepage; /* if entry is successfully added, it keeps the reference */ @@ -1532,7 +1500,7 @@ bool zswap_store(struct folio *folio) store_failed: if (!entry->length) - atomic_dec(&zswap_same_filled_pages); + atomic_dec(&zswap_zero_filled_pages); else { zpool_free(zswap_find_zpool(entry), entry->handle); put_pool: @@ -1563,7 +1531,6 @@ bool zswap_load(struct folio *folio) struct page *page = &folio->page; struct xarray *tree = swap_zswap_tree(swp); struct zswap_entry *entry; - u8 *dst; VM_WARN_ON_ONCE(!folio_test_locked(folio)); @@ -1573,11 +1540,8 @@ bool zswap_load(struct folio *folio) if (entry->length) zswap_decompress(entry, page); - else { - dst = kmap_local_page(page); - zswap_fill_page(dst, entry->value); - kunmap_local(dst); - } + else + clear_highpage(page); count_vm_event(ZSWPIN); if (entry->objcg) @@ -1679,7 +1643,7 @@ static int zswap_debugfs_init(void) debugfs_create_atomic_t("stored_pages", 0444, zswap_debugfs_root, &zswap_stored_pages); debugfs_create_atomic_t("same_filled_pages", 0444, - zswap_debugfs_root, &zswap_same_filled_pages); + zswap_debugfs_root, &zswap_zero_filled_pages); return 0; } From patchwork Mon Mar 25 23:50:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13603188 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FA3FC54E58 for ; Mon, 25 Mar 2024 23:50:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D0EB06B0095; Mon, 25 Mar 2024 19:50:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C951D6B0096; Mon, 25 Mar 2024 19:50:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A98606B0098; Mon, 25 Mar 2024 19:50:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 902C66B0095 for ; Mon, 25 Mar 2024 19:50:34 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 725CAA09EA for ; Mon, 25 Mar 2024 23:50:34 +0000 (UTC) X-FDA: 81937208388.09.013F8C8 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf03.hostedemail.com (Postfix) with ESMTP id A213C20015 for ; Mon, 25 Mar 2024 23:50:32 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ZQ+G1DnV; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3xw0CZgoKCCIWMQPW8FKCBEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3xw0CZgoKCCIWMQPW8FKCBEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711410632; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UL7FjpWvNZHo/XDshUAVHcqtRsHSd5BUf04PTKRvwLE=; b=ixUVQA/1ZbNMtApsAowMWGjDNXo7geSNeixSH3W2l8rP5c45KKmo54LItw3R0nM/mRKI2X j7iRVKfrkNq8MhGTifKs3IiYAjSiL5VGPdpYPb++7pc4TgV9ipSxaSWe48jvqxthfLfbga UCpgUJIAmgES1G6Yot2O249+K656oCc= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ZQ+G1DnV; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3xw0CZgoKCCIWMQPW8FKCBEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3xw0CZgoKCCIWMQPW8FKCBEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711410632; a=rsa-sha256; cv=none; b=lwJq5Dn3PA8XRzPo3qwC3n01hQpcZrBIym+M7vReIrnIyhlpl6t/d5Qw3hy99Hx3086NZr aSaY9hWNecR3sWyWzoot+BedBKU8n10wgFDBwdYAmH8CaKXgmsr4JBk4MdQVlmBCB9WNxI XXXEm65nZXqwtHGXcLFacRXoSxlsAUE= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc6b2682870so8003243276.0 for ; Mon, 25 Mar 2024 16:50:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711410632; x=1712015432; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UL7FjpWvNZHo/XDshUAVHcqtRsHSd5BUf04PTKRvwLE=; b=ZQ+G1DnVp+IqoLKJu82U7VVOAW1+CIcWdVHrrQtcYEGtIwhwE9u3QHD8OB/DF3fQrs 3GVVl1Zr7407AG/j1Y8Z0YLZlgi2VWD/u+IjymCukWU+iucpYr1oPg2HuOTBDWuwB0jo ORjztwp2DzDhh0m5jfm5RqTK71L7ZGTbT/BmQEH5HM5Hstghsm+8XSsyc1PcbbZhUZ70 rkyTLnZjcjW/FWZ9tM29H/3jlOvqGm5DhPXsHCimBs6MHAJurrcWK+cWZI85rYIJ6O7l MuiheonKIXWpFPkBZhXqaxQ7VPxSHsyeFbB10EmsoEqfGi4+yTvG1o9WUwPTUQ0rFxE9 Hglw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711410632; x=1712015432; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UL7FjpWvNZHo/XDshUAVHcqtRsHSd5BUf04PTKRvwLE=; b=EA5G/IfXNk5dP0yXaXgmnXWlu3MEVzN7oGCGOkLlciMfeTf5ccJGu6KB4HuPjsPY3C nLoT3Lsbn7m7j/4k+Q3PsYxQgGOsgoZqL/IzOCSFsAEpJKwrPdwGNvm4F2NjssPdEs9b 1uFkq/eLAwNzGRjMPs5V+8S+6lf5kja4c9+8edIM42bSnJKuBVyBiqtrmMhvNLts8DH5 IvCVO6qtzHEi6HV0zrFkKla2aHQhIb6VaSuHkn7rYYN4XoRO7WwIHP/MFzZwtIbBSx+y EOpkuFW1tyfiRzO7neJW5/Z0LRfj3LxqbIygrfCWUPLO51R+h5XuCIWNZCnj2p63Q2bc Ufzg== X-Forwarded-Encrypted: i=1; AJvYcCVfulzCCf5KioCg3nrcnrq6o/r7rcYEi5C9ASMg7SwhriZJcwQlJaspTQUbSBmQ8OszY74olQARiDdYFW2VCHloRr8= X-Gm-Message-State: AOJu0YzXG+6m3skJI7YizB37bFR1GVfCGzCnhpN6A2LnsTxYin1ee2eF ysjszWjJmJhLqldVqR+BubC5vzNo8GYBjWA76rrzc1+I+NKYIxvOfacPX8mlWL1ju/hJIz/wBmZ +27wRFhBlWEEw+mQNuw== X-Google-Smtp-Source: AGHT+IG+TLfwzbh3JCRDIgWmOJOjvnZ/+V3sJUz+/QQZSrlaneyk9M9ZmZQb1PVEN8h1tDdQ5SE3NK8iK22/6mCW X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:1507:b0:dc6:e8a7:fdba with SMTP id q7-20020a056902150700b00dc6e8a7fdbamr2612541ybu.4.1711410631787; Mon, 25 Mar 2024 16:50:31 -0700 (PDT) Date: Mon, 25 Mar 2024 23:50:15 +0000 In-Reply-To: <20240325235018.2028408-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20240325235018.2028408-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.396.g6e790dbe36-goog Message-ID: <20240325235018.2028408-8-yosryahmed@google.com> Subject: [RFC PATCH 7/9] mm: zswap: store zero-filled pages without a zswap_entry From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: A213C20015 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: qaskzk6tb45hhb541ugd4tqz7tftudrm X-HE-Tag: 1711410632-963789 X-HE-Meta: U2FsdGVkX1+uu4rgMH3PYw4ew+GfKKlul61IKBZrjTpdHNkdChVpPiei2hOi3hov8ewBkThRB8u+KW3MskWiJ+ekaMs9CMDTQqomNbwQWmySJvsHkCy6p9pECAeKuBz7T2vEqk7UiEKOaxmX1k6JhUgPtWclWUVoHdNcRKX5GkOfDLYep3yZi9E6TAsxEF80L1AkOQlfWLwEAN4TnOffTimKc1j8qngDQswYuzEDyZtebSBFWgzBVsz/2gTSL+IUWFBO4EZJOxb7A32q8eU9Q8TmfGrGxB25aBGgXsw3I5PGQq14RT74UWraBIJfv8heQuJL9jY9BsQ0tEYSLpiSwPiS/SDSv38pYUGZ7sqJ8VCLgCTP7lbPfFZD+0Y9UDNmbwhUXTSS3Xtq2H0BS/04sP7Jq70TpfomPoMR6qqFVtK+/OzQ3MWfchfBnfAS50TBCpA459AkFnT6qiFnwcQeQIO/aLRGGi2M5Asq3jZ2c5PylukRnrv55W2kIWMHEUz8DwtKw7KGiDIV76Czv3S0h2KzX441HFCFzkoweIJwgRkNuDxElGU6TKkMxhszBfpRTaxitQH1QNp9plyiHZv/zQCKqlDITwB4m4O2opvt15m1aQQBDMGKodpuxSIpKyLJ54XZAvVBUfdEcQbjF//KKK7E4IKUQyVUExp5YQ50W5ZpVUwQlw5DETdMj++f+rSeDsI8WQJ1z0Wq90Lv9RY6Eya+yPApuvUXAyrJ9K0XNp0XGwmK901rgboiQFLWJWElSB3UuVMUvE2cfG1FrN69i9bAAlZ7QQ7+c2xGv3ilwohC0DyZQ3XK1pV2Cii4lAvS8RmhERP5Pdvjs4OK9gy6zZzuXpNJ0KzGIzvrtZxNJ4HMbt84Q1D4MQA+TOydGKCF97L0yd6msRUghBSQzLMRjorSP60XNRNrIq2kacSXUCW3KFn4ZRK+8pJYPjUwcUY6pocA/HlMSrVwQb2Gmmp oseWce/2 +TwLSMtj0a/kgmDVWZgQxRcVo0X1VQkvNH8DNDxIWcsW9sZHlCu28SihsjynYBz5akofBoso6TAHHDpPTmzL9xw+/r452d8knY2QnRwKjdQFLIb6AVGqrSjjx+IonZ1ur70a5svY9oU7NhAqcY90nPPrxYfbw4pQfOKwWEqW36e5AVEKGD8SNv0AzADEWyv9Mc8QYDXrZeNUzCC3ubCa/JPOlMXA6d3bGqbeRIEJJRKWz6m416jhzI2dOuF6PJvK6twDMuGS2C5uVwRUJchz//hTSae8UJcJT5iRJcO1HopecR8pirXmTLD83He1Lw4KVolBjyC+mUVKiuQ5v127kZ2jWXpLaahulISj1WuJ9smcHamoBHkYdBvKZGZrDUhbk5cQyRkICvAynCf5rJa8sa8QBUSfaqrvqbH/v4ojYm63+Rd05adEzIGop7ZWzqJEDp4MeNse4XiL2mmh79evX9Rt3mJ1ljWgL3HGdYQFWycAa7wkclZYG7NLkCI8XSWjVOHfYva+f33WvQyI6p9kDDSaWpm+wbdsoDstSBQh6w/pWvgncZrVr8LOQjmXvmYWtXiXnfI2ltCYlERgu2avexOcIf6VsSgvgGXF3SlhGtabZdnipqSY+9nU7siJqT7ebj7droEh+QGxr+15zpd3vu8/fZrSijKED61onFJ0nIm2bfRrBO4QiLU//SCu6NsDAWmsPVQPiUtKT+ja6vK+WUUKkjfj3iN6pgWvxGf1iSy0RcWpLUWY1KnvH/aIHDeqNjMeD6kh8uJ7DJb5lhU6C6c0DL0stn2Pz9cloDMN86rAW65YZ+iiMPa38WA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: After the rbtree to xarray conversion, and dropping zswap_entry.refcount and zswap_entry.value, the only members of zswap_entry utilized by zero-filled pages are zswap_entry.length (always 0) and zswap_entry.objcg. Store the objcg pointer directly in the xarray as a tagged pointer and avoid allocating a zswap_entry completely for zero-filled pages. This simplifies the code as we no longer need to special case zero-length cases. We are also able to further separate the zero-filled pages handling logic and completely isolate them within store/load helpers. Handling tagged xarray pointers is handled in these two helpers, as well as the newly introduced helper for freeing tree elements, zswap_tree_free_element(). There is also a small performance improvement observed over 50 runs of kernel build test (kernbench) comparing the mean build time on a skylake machine when building the kernel in a cgroup v1 container with a 3G limit. This is on top of the improvement from dropping support for non-zero same-filled pages: base patched % diff real 69.915 69.757 -0.229% user 2956.147 2955.244 -0.031% sys 2594.718 2575.747 -0.731% This probably comes from avoiding the zswap_entry allocation and cleanup/freeing for zero-filled pages. Note that the percentage of zero-filled pages during this test was only around 1.5% on average. Practical workloads could have a larger proportion of such pages (e.g. Johannes observed around 10% [1]), so the performance improvement should be larger. This change also saves a small amount of memory due to less allocated zswap_entry's. In the kernel build test above, we save around 2M of slab usage when we swap out 3G to zswap. [1]https://lore.kernel.org/linux-mm/20240320210716.GH294822@cmpxchg.org/ Signed-off-by: Yosry Ahmed Reviewed-by: Chengming Zhou --- mm/zswap.c | 137 ++++++++++++++++++++++++++++++----------------------- 1 file changed, 78 insertions(+), 59 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 413d9242cf500..efc323bab2f22 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -183,12 +183,11 @@ static struct shrinker *zswap_shrinker; * struct zswap_entry * * This structure contains the metadata for tracking a single compressed - * page within zswap. + * page within zswap, it does not track zero-filled pages. * * swpentry - associated swap entry, the offset indexes into the red-black tree * length - the length in bytes of the compressed page data. Needed during - * decompression. For a zero-filled page length is 0, and both - * pool and lru are invalid and must be ignored. + * decompression. * pool - the zswap_pool the entry's data is in * handle - zpool allocation handle that stores the compressed page data * objcg - the obj_cgroup that the compressed memory is charged to @@ -794,30 +793,35 @@ static struct zpool *zswap_find_zpool(struct zswap_entry *entry) return entry->pool->zpools[hash_ptr(entry, ilog2(ZSWAP_NR_ZPOOLS))]; } -/* - * Carries out the common pattern of freeing and entry's zpool allocation, - * freeing the entry itself, and decrementing the number of stored pages. - */ static void zswap_entry_free(struct zswap_entry *entry) { - if (!entry->length) - atomic_dec(&zswap_zero_filled_pages); - else { - zswap_lru_del(&zswap_list_lru, entry); - zpool_free(zswap_find_zpool(entry), entry->handle); - zswap_pool_put(entry->pool); - } + zswap_lru_del(&zswap_list_lru, entry); + zpool_free(zswap_find_zpool(entry), entry->handle); + zswap_pool_put(entry->pool); if (entry->objcg) { obj_cgroup_uncharge_zswap(entry->objcg, entry->length); obj_cgroup_put(entry->objcg); } zswap_entry_cache_free(entry); - atomic_dec(&zswap_stored_pages); } /********************************* * zswap tree functions **********************************/ +static void zswap_tree_free_element(void *elem) +{ + if (!elem) + return; + + if (xa_pointer_tag(elem)) { + obj_cgroup_put(xa_untag_pointer(elem)); + atomic_dec(&zswap_zero_filled_pages); + } else { + zswap_entry_free((struct zswap_entry *)elem); + } + atomic_dec(&zswap_stored_pages); +} + static int zswap_tree_store(struct xarray *tree, pgoff_t offset, void *new) { void *old; @@ -834,7 +838,7 @@ static int zswap_tree_store(struct xarray *tree, pgoff_t offset, void *new) * the folio was redirtied and now the new version is being * swapped out. Get rid of the old. */ - zswap_entry_free(old); + zswap_tree_free_element(old); } return err; } @@ -1089,7 +1093,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, if (entry->objcg) count_objcg_event(entry->objcg, ZSWPWB); - zswap_entry_free(entry); + zswap_tree_free_element(entry); /* folio is up to date */ folio_mark_uptodate(folio); @@ -1373,6 +1377,33 @@ static void shrink_worker(struct work_struct *w) } while (zswap_total_pages() > thr); } +/********************************* +* zero-filled functions +**********************************/ +#define ZSWAP_ZERO_FILLED_TAG 1UL + +static int zswap_store_zero_filled(struct xarray *tree, pgoff_t offset, + struct obj_cgroup *objcg) +{ + int err = zswap_tree_store(tree, offset, + xa_tag_pointer(objcg, ZSWAP_ZERO_FILLED_TAG)); + + if (!err) + atomic_inc(&zswap_zero_filled_pages); + return err; +} + +static bool zswap_load_zero_filled(void *elem, struct page *page, + struct obj_cgroup **objcg) +{ + if (!xa_pointer_tag(elem)) + return false; + + clear_highpage(page); + *objcg = xa_untag_pointer(elem); + return true; +} + static bool zswap_is_folio_zero_filled(struct folio *folio) { unsigned long *kaddr; @@ -1432,22 +1463,21 @@ bool zswap_store(struct folio *folio) if (!zswap_check_limit()) goto reject; - /* allocate entry */ + if (zswap_is_folio_zero_filled(folio)) { + if (zswap_store_zero_filled(tree, offset, objcg)) + goto reject; + goto stored; + } + + if (!zswap_non_zero_filled_pages_enabled) + goto reject; + entry = zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(folio)); if (!entry) { zswap_reject_kmemcache_fail++; goto reject; } - if (zswap_is_folio_zero_filled(folio)) { - entry->length = 0; - atomic_inc(&zswap_zero_filled_pages); - goto insert_entry; - } - - if (!zswap_non_zero_filled_pages_enabled) - goto freepage; - /* if entry is successfully added, it keeps the reference */ entry->pool = zswap_pool_current_get(); if (!entry->pool) @@ -1465,17 +1495,14 @@ bool zswap_store(struct folio *folio) if (!zswap_compress(folio, entry)) goto put_pool; -insert_entry: entry->swpentry = swp; entry->objcg = objcg; if (zswap_tree_store(tree, offset, entry)) goto store_failed; - if (objcg) { + if (objcg) obj_cgroup_charge_zswap(objcg, entry->length); - count_objcg_event(objcg, ZSWPOUT); - } /* * We finish initializing the entry while it's already in xarray. @@ -1487,25 +1514,21 @@ bool zswap_store(struct folio *folio) * The publishing order matters to prevent writeback from seeing * an incoherent entry. */ - if (entry->length) { - INIT_LIST_HEAD(&entry->lru); - zswap_lru_add(&zswap_list_lru, entry); - } + INIT_LIST_HEAD(&entry->lru); + zswap_lru_add(&zswap_list_lru, entry); - /* update stats */ +stored: + if (objcg) + count_objcg_event(objcg, ZSWPOUT); atomic_inc(&zswap_stored_pages); count_vm_event(ZSWPOUT); return true; store_failed: - if (!entry->length) - atomic_dec(&zswap_zero_filled_pages); - else { - zpool_free(zswap_find_zpool(entry), entry->handle); + zpool_free(zswap_find_zpool(entry), entry->handle); put_pool: - zswap_pool_put(entry->pool); - } + zswap_pool_put(entry->pool); freepage: zswap_entry_cache_free(entry); reject: @@ -1518,9 +1541,7 @@ bool zswap_store(struct folio *folio) * possibly stale entry which was previously stored at this offset. * Otherwise, writeback could overwrite the new data in the swapfile. */ - entry = xa_erase(tree, offset); - if (entry) - zswap_entry_free(entry); + zswap_tree_free_element(xa_erase(tree, offset)); return false; } @@ -1531,26 +1552,27 @@ bool zswap_load(struct folio *folio) struct page *page = &folio->page; struct xarray *tree = swap_zswap_tree(swp); struct zswap_entry *entry; + struct obj_cgroup *objcg; + void *elem; VM_WARN_ON_ONCE(!folio_test_locked(folio)); - entry = xa_erase(tree, offset); - if (!entry) + elem = xa_erase(tree, offset); + if (!elem) return false; - if (entry->length) + if (!zswap_load_zero_filled(elem, page, &objcg)) { + entry = elem; + objcg = entry->objcg; zswap_decompress(entry, page); - else - clear_highpage(page); + } count_vm_event(ZSWPIN); - if (entry->objcg) - count_objcg_event(entry->objcg, ZSWPIN); - - zswap_entry_free(entry); + if (objcg) + count_objcg_event(objcg, ZSWPIN); + zswap_tree_free_element(elem); folio_mark_dirty(folio); - return true; } @@ -1558,11 +1580,8 @@ void zswap_invalidate(swp_entry_t swp) { pgoff_t offset = swp_offset(swp); struct xarray *tree = swap_zswap_tree(swp); - struct zswap_entry *entry; - entry = xa_erase(tree, offset); - if (entry) - zswap_entry_free(entry); + zswap_tree_free_element(xa_erase(tree, offset)); } int zswap_swapon(int type, unsigned long nr_pages) From patchwork Mon Mar 25 23:50:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13603189 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5335CD11BF for ; Mon, 25 Mar 2024 23:50:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B38F66B0098; Mon, 25 Mar 2024 19:50:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AC35B6B0099; Mon, 25 Mar 2024 19:50:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9158B6B009A; Mon, 25 Mar 2024 19:50:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 797276B0098 for ; Mon, 25 Mar 2024 19:50:36 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 1BB9AA0A08 for ; Mon, 25 Mar 2024 23:50:36 +0000 (UTC) X-FDA: 81937208472.30.2DD47DF Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf10.hostedemail.com (Postfix) with ESMTP id 49322C0002 for ; Mon, 25 Mar 2024 23:50:34 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Q2lfIGdj; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3yQ0CZgoKCCQYOSRYAHMEDGOOGLE.COMLINUX-MMKVACK.ORG@flex--yosryahmed.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3yQ0CZgoKCCQYOSRYAHMEDGOOGLE.COMLINUX-MMKVACK.ORG@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711410634; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KO2q76LvMLcDayey/RN5wyIAuuRCIUh2utVXFGvhTms=; b=u0v5K9wXVGuULG9uhTs984iJZXwFm5gXv2EQ6+9cYRjMHucwd6Z+U56KPCXPe9AaMdSl3a BIbBQol3xi9lav7DQn5bWpbxOhxxClYqHfnkOCMwjT7/O4+KR0klGPu3aFJwGpEVN7nj6x k/iRrmRhKCw5MG+vrnYqI8dUAcGlpw8= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Q2lfIGdj; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of 3yQ0CZgoKCCQYOSRYAHMEDGOOGLE.COMLINUX-MMKVACK.ORG@flex--yosryahmed.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3yQ0CZgoKCCQYOSRYAHMEDGOOGLE.COMLINUX-MMKVACK.ORG@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711410634; a=rsa-sha256; cv=none; b=SvGNNqD96tRvV6feKc6+iSNYQS6Q5+tKglrutwqCDw8GyClk0/yL2xGHXNfin8LzbEubLf FQwEGJpVh0Tw+n1in5lxAzrmpomTs0x+/Y8THyL61/Fq5uXixYhOnI0tDho+SInxt8X4fs /nXv81olaZtZi009b9nKISbp5Tw2P2I= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-60a0815e3f9so75221607b3.2 for ; Mon, 25 Mar 2024 16:50:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711410633; x=1712015433; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=KO2q76LvMLcDayey/RN5wyIAuuRCIUh2utVXFGvhTms=; b=Q2lfIGdjr+xdTnmwbi5d/3bsHWP+q6oPx90yaOAzKd0YpQ97893muEXb6IhakQ9dUY bBGmcPFlTlCBSI1xE1Nmht2gYEpI3TO0hlTr5h3s0cvqMv5NMqsu0KuwzOirDVKbdFrn OfCArANEYmXC8iEIxgOq3LFXk5KUns3KFgEv7mLJRdtUtU6R8aqX/7zCwuxuXOi7lb1g Jc8NrToqKMlfFc6Mqb5++Guz5vYbc/R4b8Fxyiyst8Vz4XddExfPGwkIcARa/Cpx3Q0B lSeVbAV/cjmz5HCewcLJiEW2jMgq5KKV4ISH8fielCYfSKx/dREDKYMlmSjVhDf6L24t LaUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711410633; x=1712015433; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KO2q76LvMLcDayey/RN5wyIAuuRCIUh2utVXFGvhTms=; b=PvzD6liIbPattBTQLIS3iQ7grXI9uYONDUA5osh/GaZ0KNzAQtOC/+EcB4qYR4Z8vp uP9Dm7ijfWBf9EPfOWIyXZ7qV5UyyU4+1FQyCO5M6o84g6k+bQqlCdjbS/1jHNbVKlxR co2UGzWvT3VSXsnwFaanQdjKPqH9+LerBy389e4IFpfGBzrtfUoCwETlOOhQcT3PegHc 9yBbO6WdiR60at2y8ENeWZvS13DwrvOkYRD7lZWQmAw6agkA843KK0S60KfhnLo1n9ue TyZSXkVYleEleYJiSH5hVO58KDj4fgDTFtwG1AowkgsmiwxEI/ai0ICtoH4q7i5LFFrR 2xmw== X-Forwarded-Encrypted: i=1; AJvYcCXRJO05mQQLCjoladDIxeHEselyZAFXZkhxqMhWIufpfubjuqeOstugNaP5yjJrc8ttscVM7qDzPfC//Jf17i2vvKA= X-Gm-Message-State: AOJu0Ywg3J4OgQzxj8crljk3wi4Zx9cKxX9gQcZSC7Bs5cMEYh5+FfHp mCmBVtjwOCxasinJ8vXj0Dx5o75zyv3rnJE9uEHmgDUv3TLES3PAilF7b9IhMESPXLtuVd5Xdwv pMh9WB1gG2LpixGEX+Q== X-Google-Smtp-Source: AGHT+IGsngKICRKq/pDsaMChol7XL6fK76guiT9EmRwqzuuG62psC+Y/KoJk23mBwa/MlcuS5UwOswLoFZ1CnMT3 X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:1241:b0:dcc:4785:b51e with SMTP id t1-20020a056902124100b00dcc4785b51emr370308ybu.12.1711410633551; Mon, 25 Mar 2024 16:50:33 -0700 (PDT) Date: Mon, 25 Mar 2024 23:50:16 +0000 In-Reply-To: <20240325235018.2028408-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20240325235018.2028408-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.396.g6e790dbe36-goog Message-ID: <20240325235018.2028408-9-yosryahmed@google.com> Subject: [RFC PATCH 8/9] mm: zswap: do not check the global limit for zero-filled pages From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspam-User: X-Stat-Signature: fpont1q9b85gzkco1ngpthckbhumd57d X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 49322C0002 X-HE-Tag: 1711410634-345662 X-HE-Meta: U2FsdGVkX1+4fnTffUI5k3Y5ZJMYyQIJCknYzMAPeKm8QwFBNW7kwW6dhf9h0fkUV7t236aValu1djKQd+n0AqJ+rP67catIGwgXjl8vrj24wHhB4e6ZMQZBmQuXrssDeB8z6BaRGtjhe1HpWgM4qY2fnH8AasqHNf//us47VFIYw5lE3lPCxetAYV+6mq+NdyoX0IZfB7SRmW9pivmH20KR58o/4Stu6Swxv5c79h122qzga5avUbY+03WF6F3rdBZeOX7hdCEDBic75cPcDnvM7cgCYP7mPJ4+x/zrQXZoxKNe/mZGPJSzZN0NdS81HARCLVsGuSG6TS31nst78ajrMt/8cK7vvqjoiSUEc1bvjAnJFCVb6BEwHZ4WP7ebhq9a2s9TcrU42imDmmRPzt05ulEXDaSsm5cxEkSSWfUyJVJ0on7aIxhePO0N+Uru67TWIonQJUNsX3foITcXa5yI9qMwncNiyvIeFMzF4AYV9Kxn6lXiJGoNDbeSyphnGmVJtBC4b530souGXAyFHPU7e2YysqCJGwkMRBvTq9lO3IBvhElDjR0bC/45XRurzM9MoQl+E3ChYEKLN0mxn9NQqo7KHv/foXoih57Azi09UtfUly4cB0urNI9oCIxwFKGh7eLkwfOwEtvsBlQS7vp6P+d3mJOX98ewftit+ZAVVIWdPuNApFt8VgIVRp7/xqTT7cMgi4FAM3vmuaQDE0M356k5aqc/kyScsfk2Y7KklVUhYrIpV5PGpyD3jZvdWu3cdK9rYgwNIUFGOcpbznlzFTpoN9LW1WX2ZEOonJVeuN+7xuE+D7gHYPSQQRGT7VS0f/dpLfd/+QUZVgzhFXDwkCYR3YxqlGSB2MCwSfsawdGHrdon9tVEC1sGub20k+sj0ndDERv43TjT1Vh1I1z1O8UE/O8/gRRBQf3diWBAK3vSXJKMa+UxfGo71auq4jZ1Qi6usOHdjD5OLpT gdyNefvO sa3f8L0L7GroLUT7schkyaLx9lfAACzCft58bl+VNsPfOn5Wi6OUio22iJf3cs3dpmUBKdcemtqcxvNNirAtTZYABJGOPHr2rH4HRH+Ki+te8VSqTcdCOdwKZBhYG8LVwroWLDWGPHAVNp+bU47WO5/FujP2g3rJ5AwJFxAfeymNuMbiE/Sw+xH/T18LieIKTGeYoEirjuD4dXnOCAXwA2SbhIvE85uPMPFsJ33/3meO9es5wa6EH/lWimBeF718gMl0EaKWdOKXu2I+1J7Vilxp3zlFVTkA0UFX29L1tFG6pgQIxlDNWgLxXlVIKbXzFTscXFz85MK9spm9K0ctIP4DwmEoaWHznm2MQTc0HqdgvqcrfIss6SyHXPSLteWkeu4OTtDQPP0ukXICzUvyKyHoGJJdhMcI1L6+LO0EVuzvRNRZJTMXUa9cDnz2GWNmJ3bcRHoU2EzHMop/7F+qt7jtbmMAqrmHHIzIQVEgDcvTLTfIhcKrLwz0Cb1WSoTAGvRFwAMAccG0r332bmIMsl9nY+qzMqycmXcPpjBLvhQqQ+eWT6wKXhHzM+c9LOApuEOk6foKLyCXAPgc28TzNNVCDwY23UhncpspfOBGlXsN8SwzOFjgxvXAHSh06/i/9EEeHUbZM0VP0dnE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000122, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When storing zero-filled pages, there is no point of checking the global zswap limit. These pages do not consume any memory that contributes toward the limit. Move the limit checking after zero-filled pages are handled. This avoids having zero-filled pages skip zswap and go to disk swap if the limit is hit. It also avoids queueing the shrink worker, which may end up being unnecessary if the zswap usage goes down on its own before another store is attempted. Ignoring the memcg limits as well for zero-filled pages is more controversial. Those limits are more a matter of per-workload policy. Some workloads disable zswap completely by setting memory.zswap.max = 0, and those workloads could start observing some zswap activity even after disabling zswap. Although harmless, this could cause confusion to userspace. Remain conservative and keep respecting those limits. Signed-off-by: Yosry Ahmed Reviewed-by: Chengming Zhou --- mm/zswap.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index efc323bab2f22..9357328d940af 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1460,9 +1460,6 @@ bool zswap_store(struct folio *folio) mem_cgroup_put(memcg); } - if (!zswap_check_limit()) - goto reject; - if (zswap_is_folio_zero_filled(folio)) { if (zswap_store_zero_filled(tree, offset, objcg)) goto reject; @@ -1472,6 +1469,9 @@ bool zswap_store(struct folio *folio) if (!zswap_non_zero_filled_pages_enabled) goto reject; + if (!zswap_check_limit()) + goto reject; + entry = zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(folio)); if (!entry) { zswap_reject_kmemcache_fail++; From patchwork Mon Mar 25 23:50:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13603190 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1507BC54E64 for ; Mon, 25 Mar 2024 23:50:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 398826B0099; Mon, 25 Mar 2024 19:50:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 322536B009A; Mon, 25 Mar 2024 19:50:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0FCC16B009B; Mon, 25 Mar 2024 19:50:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DECCE6B0099 for ; Mon, 25 Mar 2024 19:50:37 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B0026A03C9 for ; Mon, 25 Mar 2024 23:50:37 +0000 (UTC) X-FDA: 81937208514.19.FCFF2E6 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf22.hostedemail.com (Postfix) with ESMTP id EF5F1C0007 for ; Mon, 25 Mar 2024 23:50:35 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JBMZOzew; spf=pass (imf22.hostedemail.com: domain of 3yw0CZgoKCCYaQUTaCJOGFIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3yw0CZgoKCCYaQUTaCJOGFIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711410636; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=z6nu3U2cdQmEtB2lnDGjl91RMjz6aut26G90+PFcykM=; b=6Y9v8bMfmC0AuCAVcX0eVhLu5EFq57N2dYKyNIH7W0WTiQdpeQYjb/nDobUXbe1gpYn3D/ K9UQyr/51p9QuzepX8kduXi9x0g/ULt9Uvec8gjdYLmijJXqSynDCZKRPCn35334zjFaMS 6Vg4c0Yf2wPntNYXsoCTRGu+QHKTpQ0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711410636; a=rsa-sha256; cv=none; b=XbDJb8IWttUVXnAv/yVEo1c2s+YngQ+8eN5czcugfnS4P16yPShwEcGVjnRE2/ZcddiLgu dcgRZkd6sEULL+cFHahkfWVZXkH5xNbxjRQYspvLYRGbZ15SKzZZMPmCjSVOTxhnhQaOvW pqMBZr20T2VcAxkFXE2GQqrgud3zB8o= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JBMZOzew; spf=pass (imf22.hostedemail.com: domain of 3yw0CZgoKCCYaQUTaCJOGFIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3yw0CZgoKCCYaQUTaCJOGFIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dbf618042daso7713071276.0 for ; Mon, 25 Mar 2024 16:50:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711410635; x=1712015435; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=z6nu3U2cdQmEtB2lnDGjl91RMjz6aut26G90+PFcykM=; b=JBMZOzewLcGqH4BfisGsBnLO4DpJ4UWl4/75QE1luth2Tn0uL0V/7ijkVYHfQar+lb +rir/3cGD3AO32BpIU47pgW7kC4PqnpGqy9YwvDsGYEWaLsugQHIBHDB5R0fiKGyHtwO vHkjHMlEh4lYALkobMnvMHUxOIhI4l5NPD9aqn8u4JKv9NcoysPHqT2DDGxTjx4Drj4F vUvq1evGm/SrQXXGMJyk0nQ/h3ryV3+27yA8ZQ5e6oQdYN4VCecuBSoTdM7ZIHf/xFEA qHC8Nzv4PcvwNIJlVV2mQ+hsak90rJomiPG6c55okZXLxGc9t3C0ujiim5R1ZVVVgXMD GMtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711410635; x=1712015435; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=z6nu3U2cdQmEtB2lnDGjl91RMjz6aut26G90+PFcykM=; b=CX/qE3aGI1rGIfjmn71w48EL6HmquaTSn1V0Z8N8HZv4pabFZlQLIefqj8Ah7WHJBb eHTOX8TxVjbvidp+x+Op3v/O7cnft3S9laEBID+Tv9lw75AwAqPysrvs52UQEmjpGyVO EWBKkrO4YN3lM+6Lmor+qIz0jORTAN23ILpXdVZbFznyhuhDoK+MwK12XQUwMgvls/aT 29H5JHPdhAsZh28LtZADVctcYn1DURHjIp7ts+vbz+zwALGW+IMBTfsbLbapoBtzgFB3 MoPSvv/uShhYgGvuto+gq+XzSbScnKu/bs5PTTEF9OsjhREnRDgB/7mnHm9YJ5uzMqbh wFnA== X-Forwarded-Encrypted: i=1; AJvYcCVXDbU6XgNB/2VoeSJ6MsjLsS7dbbwnV6l3UCnp9oKp+g8p/GVtVYPMslAfRsSIi5kL/8S7cKUzYWQ8m88wXSk6g3M= X-Gm-Message-State: AOJu0Yy7Ns0JBl++vF80MGeb0+xmqEtoG+ejouy69u2UG3jGBMpMfyiH QZx1UX7FvfhummfqJ5O1JLHzdUK3MnbUlPq+Ah01F82QdopCJBRRw09O0rDE+cD3gEE7TtN6rfy Os2PpdLP9tmbiyPIGRw== X-Google-Smtp-Source: AGHT+IHhwuSMeZ8KbkpdVbgoG6k+kjBoalv5THVzyluQgGq+kifTjQF/ACTCwAHFlLZp6s7t/niB6q7ZIp/Hk5oK X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:218d:b0:dda:c565:1025 with SMTP id dl13-20020a056902218d00b00ddac5651025mr317614ybb.2.1711410635103; Mon, 25 Mar 2024 16:50:35 -0700 (PDT) Date: Mon, 25 Mar 2024 23:50:17 +0000 In-Reply-To: <20240325235018.2028408-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20240325235018.2028408-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.396.g6e790dbe36-goog Message-ID: <20240325235018.2028408-10-yosryahmed@google.com> Subject: [RFC PATCH 9/9] mm: zswap: use zswap_entry_free() for partially initialized entries From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: EF5F1C0007 X-Rspam-User: X-Stat-Signature: dwb798bi6bbd19nbjxxh7oy7c78wr3jc X-Rspamd-Server: rspam03 X-HE-Tag: 1711410635-884521 X-HE-Meta: U2FsdGVkX1+3W64ABLJwacUhx0FHWXBDZqi0lz7JV6QUbM7S+hDRZ+cTq+f4XF+x2lgcVEIXHAF/UmRPL8qFqU+2KG80/wLtOFot9q0Gy8DiMT4nsVl23rHgaAPevyAKdHolecc3Db08owZ7VW6wMtNx/Iyur+G4kpx5VH9VmHs4bdpGAH4bKrpEJUtsntiU3WXaeWgEsWz6w0gndBhUM7QeLBYINcIJAIjybJ1a/nAdKNyvuX0FnZ72pR0fKppMoMvvuWUHoMBGy2lgbQGAbCcpUZCqEtSmWqHcH+qk5rK+rVtajcoebo/rkZJgzU6LIgKxUmz+q33Zf9snkQptMT5p6XnvOItJINkMZaZMVUh5nSA1vHqE+N+FQNMcXFsUm9AY0TPrXtDa1ihYPLT0cv/I6kEbyAFpI03+pQxBD9UfBunDgtFpAe7XfnOA7aXqixxrnk6dzCLubYjQC3avvmfMTLiN/Uf3mpu8qT33yJv3qBKAGej1M8MtqpGYyqlT2KcB6Oo4vI+puVznzEWNnwp1fSZ1cLRR37GVV13mIfuQPg1dVJbFLoHUNtpkKfEmCiwVrQf7A6neM3C84SVL6OlkPPVrOslEweULZsWI+xyOZL9AFVQsuTYw/UNkS53hOh7/ulTmjNPVEr0fqITvMsN0/fdCC3bgQfa/zeCek5Z1SBLu7R6dtgQIEyKPX26PP3eNz0T35yCyyBvyUmBB4ueNv2Z9Rl1WaWUEiioX2mejatRm36a+1odpY6I+unmGaC2DVUybWDpJRy7bEVxMClqGOuSFzcmfkZJQp+rt+1t7FpPiK87GKsfO7pIPCYIhJmdrneHaC+Wbx7JOKZUJdCmaT5HHc5GHIR+BAaGu/A/Z8r8dcNnz77VkLCRvfaswCaptnRouqUOgFKszawnAZbPGhkb7fxWAcpRVlaxA2ZgQHKZPuL7pnqKwyMkABWF7FCesovOEBqJqKdYEVjN yP7WP783 /Ck4pw68BwTm/oDH7gjw/ypjUSD+ZxleC/3ldhwWxUs2iXBH/fyHTiH4jRkjoL+V9TzXfgFvqzlU1zOh1mpw44ufufLQfIJ0py0LXtqNZFPm4nMgwEb/YgQICNU5S7ChD7GMcOkqemeGQXlc8HAIZ5B7oIE9gXVPqdWbKQmjRp+0w28MdnKd8yoKieavk3ivZb5DhO9ESXZ611slOPOCUi4zEQcHU5SGOwBnJ/4zkDTB/zHwQaS2yRf5BwoawnVac4w7vGSLEvFEsYazvnzjdYMATB5tmnhWKPumxo9SFF2TsWd8/diEPlA2yfHSI81PX9i5Nl0hFp7e3GNVeGPzTmEYPGjTGOa5ku6Egx6MI5rr4xEiSEZan1fFSQGUjXGn5nXAFCt3/V77urd6hdmFnGfSD+mjYM5vgEyeCX1bj+UVZfhUqOfTUfSgvKSHretlVFwenRaNNvKLs+MMSBX1yX24DUVKs041Y6mmS0FGFJ5DGtMTzeFEgsck71bqILxDfpRfFPwYtqziXTkX7UjRGWqRXCrq1YkDbJHfYYh/swONFvx7aEr3tR0+D7K4Yc/nXhAo1EimBj93T2g9Lc+ouvjAE9qlyNWOe73uh14AODsAucYE6EeMWUcBGJDbb5SgopGMgkMfloWrKfvCnZWu0bUWYF0K7iC1WiQ6ZkVrMuaLyu0A= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: zswap_entry_free() performs four types of cleanups before freeing a zswap_entry: - Deletes the entry from the LRU. - Frees compressed memory. - Puts the pool reference. - Uncharges the compressed memory and puts the objcg. zswap_entry_free() always expects a fully initialized entry. Allow zswap_entry_free() to handle partially initialized entries by making it possible to identify what cleanups are needed as follows: - Allocate entries with __GFP_ZERO and initialize zswap_entry.lru when the entry is allocated. Points are NULL and length is zero upon initialization. - Use zswap_entry.length to identify if there is compressed memory to free. This is possible now that zero-filled pages are handled separately, so a length of zero means we did not successfully compress the page. - Only initialize entry->objcg after the memory is charged in zswap_store(). With this in place, use zswap_entry_free() in the failure path of zswap_store() to cleanup partially initialized entries. This simplifies the cleanup code in zswap_store(). While we are at it, rename the remaining cleanup labels to more meaningful names. Signed-off-by: Yosry Ahmed --- mm/zswap.c | 62 ++++++++++++++++++++++++++---------------------------- 1 file changed, 30 insertions(+), 32 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 9357328d940af..c50f9df230ca3 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -774,12 +774,13 @@ void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg) **********************************/ static struct kmem_cache *zswap_entry_cache; -static struct zswap_entry *zswap_entry_cache_alloc(gfp_t gfp, int nid) +static struct zswap_entry *zswap_entry_cache_alloc(int nid) { struct zswap_entry *entry; - entry = kmem_cache_alloc_node(zswap_entry_cache, gfp, nid); - if (!entry) - return NULL; + entry = kmem_cache_alloc_node(zswap_entry_cache, + GFP_KERNEL | __GFP_ZERO, nid); + if (entry) + INIT_LIST_HEAD(&entry->lru); return entry; } @@ -795,9 +796,12 @@ static struct zpool *zswap_find_zpool(struct zswap_entry *entry) static void zswap_entry_free(struct zswap_entry *entry) { - zswap_lru_del(&zswap_list_lru, entry); - zpool_free(zswap_find_zpool(entry), entry->handle); - zswap_pool_put(entry->pool); + if (!list_empty(&entry->lru)) + zswap_lru_del(&zswap_list_lru, entry); + if (entry->length) + zpool_free(zswap_find_zpool(entry), entry->handle); + if (entry->pool) + zswap_pool_put(entry->pool); if (entry->objcg) { obj_cgroup_uncharge_zswap(entry->objcg, entry->length); obj_cgroup_put(entry->objcg); @@ -1447,7 +1451,7 @@ bool zswap_store(struct folio *folio) return false; if (!zswap_enabled) - goto check_old; + goto erase_old; /* Check cgroup limits */ objcg = get_obj_cgroup_from_folio(folio); @@ -1455,54 +1459,52 @@ bool zswap_store(struct folio *folio) memcg = get_mem_cgroup_from_objcg(objcg); if (shrink_memcg(memcg)) { mem_cgroup_put(memcg); - goto reject; + goto put_objcg; } mem_cgroup_put(memcg); } if (zswap_is_folio_zero_filled(folio)) { if (zswap_store_zero_filled(tree, offset, objcg)) - goto reject; + goto put_objcg; goto stored; } if (!zswap_non_zero_filled_pages_enabled) - goto reject; + goto put_objcg; if (!zswap_check_limit()) - goto reject; + goto put_objcg; - entry = zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(folio)); + entry = zswap_entry_cache_alloc(folio_nid(folio)); if (!entry) { zswap_reject_kmemcache_fail++; - goto reject; + goto put_objcg; } - /* if entry is successfully added, it keeps the reference */ entry->pool = zswap_pool_current_get(); if (!entry->pool) - goto freepage; + goto free_entry; if (objcg) { memcg = get_mem_cgroup_from_objcg(objcg); if (memcg_list_lru_alloc(memcg, &zswap_list_lru, GFP_KERNEL)) { mem_cgroup_put(memcg); - goto put_pool; + goto free_entry; } mem_cgroup_put(memcg); } if (!zswap_compress(folio, entry)) - goto put_pool; - - entry->swpentry = swp; - entry->objcg = objcg; + goto free_entry; if (zswap_tree_store(tree, offset, entry)) - goto store_failed; + goto free_entry; - if (objcg) + if (objcg) { obj_cgroup_charge_zswap(objcg, entry->length); + entry->objcg = objcg; + } /* * We finish initializing the entry while it's already in xarray. @@ -1514,7 +1516,7 @@ bool zswap_store(struct folio *folio) * The publishing order matters to prevent writeback from seeing * an incoherent entry. */ - INIT_LIST_HEAD(&entry->lru); + entry->swpentry = swp; zswap_lru_add(&zswap_list_lru, entry); stored: @@ -1525,17 +1527,13 @@ bool zswap_store(struct folio *folio) return true; -store_failed: - zpool_free(zswap_find_zpool(entry), entry->handle); -put_pool: - zswap_pool_put(entry->pool); -freepage: - zswap_entry_cache_free(entry); -reject: +free_entry: + zswap_entry_free(entry); +put_objcg: obj_cgroup_put(objcg); if (zswap_pool_reached_full) queue_work(shrink_wq, &zswap_shrink_work); -check_old: +erase_old: /* * If the zswap store fails or zswap is disabled, we must invalidate the * possibly stale entry which was previously stored at this offset.