From patchwork Sat Jun 8 15:53:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Takero Funaki X-Patchwork-Id: 13691030 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7EDB5C27C6E for ; Sat, 8 Jun 2024 15:53:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CB0AB6B0089; Sat, 8 Jun 2024 11:53:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C39536B0092; Sat, 8 Jun 2024 11:53:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A65D46B0093; Sat, 8 Jun 2024 11:53:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 85FDC6B0089 for ; Sat, 8 Jun 2024 11:53:31 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 196EBA103B for ; Sat, 8 Jun 2024 15:53:31 +0000 (UTC) X-FDA: 82208166222.21.94D20B3 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by imf24.hostedemail.com (Postfix) with ESMTP id 3318D180003 for ; Sat, 8 Jun 2024 15:53:28 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=G8j3CB7h; spf=pass (imf24.hostedemail.com: domain of flintglass@gmail.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=flintglass@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717862009; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hZHxiUjeMqD6vZkOfwjSbyKs/eS2zlbVa8rN6xjbDxU=; b=HfNnNg+XBKvwhCC3JqPH6k0Jnw0ljEc2NAZxDCIJEmjWUXkGIrZ1k5WgjRg/cJKa3aBYXQ 8TL2/WcOjG1ofdoHSUWmCszKA3L7wDRUWaPcdQh9eQofmSww0GfsKpaPjuv2DlF1I74gwQ nh/KI/B5W7UtLWnGk8owIQWMKrveR2A= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=G8j3CB7h; spf=pass (imf24.hostedemail.com: domain of flintglass@gmail.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=flintglass@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717862009; a=rsa-sha256; cv=none; b=xrev0RGHTSsX9CRZj0KdqU52zZN96UUxtK2eglMbMv4jGVQF6RmAjPCqPTk66EmlZh9qFh AXb2Yyri97VXahnJZrVJ7BlFAwrpPlU/Vh4axfPNj/rXyW7zj70Hvl699CNUPwTw7E9mz/ pp9sY77FS12wYe/9iA34KzcOtTxZxpk= Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-704313fa830so104824b3a.3 for ; Sat, 08 Jun 2024 08:53:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717862008; x=1718466808; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hZHxiUjeMqD6vZkOfwjSbyKs/eS2zlbVa8rN6xjbDxU=; b=G8j3CB7hBpK3u2pdHJ4Pd3pAQDOFM7tEg2o0X+GWc8+zGPDzBE5MMZ9f625UdwNo7E dOGuCdHu2fzmKLJzbxw/bIjBGLJZpajQVGUT6Grf/AnlkNYtsnrefwB4pdNrc0L+FtJ4 wqONc6NQaX1JKVnVV4FRT14UI/837TaGTahOg7Nc9DRtP6T7Aaw+LlIdPimg8fYQ8UrS UZfn/Yw8ncsU8dvwIw9I3zls4fZG1GZiIU0Biwv8HmF3n1G5lFXXekU1Xp6BK1+qGCym cpnRZbqZN8KSuqrY0BmaTluaamqbmr69xPXh9GD5Ul1b2gyNMk42oZC14Fs/NMLgMaCJ xing== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717862008; x=1718466808; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hZHxiUjeMqD6vZkOfwjSbyKs/eS2zlbVa8rN6xjbDxU=; b=LQwPJB6Sn8o1pwR6vZEIs9AS9a7jDk0gi+Ad/9q6XnnVadBPR4OhHZUW7Y7JobJs4y phZEX3sgA+KUNohrmEPDYhIvJH6/lkVPLOTVWpWeTTI1gXFXqVaF+DFNx2TeWqyE2Eb4 FPwpw2XRC1RMfGHc7dywG9YwqqBgZ5RVp38agXdZ80hZpYIXjk6FioIAyb1bgUakeEmd hOoRujMa+XFqLDFRBRbAcbAHJhW3dmgVg72nsF6pGdK7XKaGYfwv0vkScqfv5bw0jk+B BRd+7UwnuBFWCfOHbndk357nxB+Uav86ZKnYpSW4ltykDG9+QtaMDZLmP0F/IXWXPkMp 598w== X-Forwarded-Encrypted: i=1; AJvYcCU05QxGYNeHeCYfiZOkRm33aNjiePJW6wkW7zbx4xGcULqtunZEW1K1ciF4AlueAoCbZq6lPhd018tTyCzeMCl7EtQ= X-Gm-Message-State: AOJu0YyCDIZNpfiQ3Jn/uqVAISP4fi9l8nQ92kwSDsTakpV/w/OdehOj 0MHOHTfwgf1nu+kEzr4umJj+cseAzpiSsIIjeDZCfxWYtuNItfSIyTUgMpU/ X-Google-Smtp-Source: AGHT+IGe4Ii+KwF0gPByGAcULNxz091Evf59hb1FQhfH6o9f5mRo+uFoQZvIIBQ3ItmZA7vqYSzMtQ== X-Received: by 2002:a05:6a21:32a3:b0:1b5:cf9c:2936 with SMTP id adf61e73a8af0-1b5cf9c2b30mr1003625637.39.1717862007919; Sat, 08 Jun 2024 08:53:27 -0700 (PDT) Received: from cbuild.srv.usb0.net (uw2.srv.usb0.net. [185.197.30.200]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-703fd4d9d8fsm4335209b3a.149.2024.06.08.08.53.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 08 Jun 2024 08:53:27 -0700 (PDT) From: Takero Funaki To: Johannes Weiner , Yosry Ahmed , Nhat Pham , Chengming Zhou , Jonathan Corbet , Andrew Morton , Domenico Cerasuolo Cc: Takero Funaki , linux-mm@kvack.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v1 1/3] mm: zswap: fix global shrinker memcg iteration Date: Sat, 8 Jun 2024 15:53:08 +0000 Message-ID: <20240608155316.451600-2-flintglass@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240608155316.451600-1-flintglass@gmail.com> References: <20240608155316.451600-1-flintglass@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 3318D180003 X-Stat-Signature: wgmmincjnkazonpzaiyfdtxo6wzku65o X-HE-Tag: 1717862008-171655 X-HE-Meta: U2FsdGVkX19YDX8pM6aMtTGjVpsDGOP3UVw2Hb3zZtALUqlx3C72yFubFBXfXdrIg9hsFt/fTqCcV5JUzLngYAmPKRwEoy88JDvCTz0nQ+rU6E7e2av6oTARAtRxbX3qXzPqrultrXE26Q7Z9or+0dSnNnwFOR8IFXxBm4i7DitK2TKAGl/n9DXcwR6mSBUhVYKHjHP77youFqDmosuaMhjTtNS+v1gOYW1ovhpxgJLMi9rCLTkeVLjuM5mbKyjRxhItz8xGMqpAIku5Q7dKTS5XsnSxQJ1HqkUqIkf+h4yTowZg5F/XiOM9WYxcMKwbb6o+2ooYdH8ycLvijphFhQRYfUXC6WIp5DiJfmtNP3o3vf/ApoVopSbH1M9MdGI2vh75JOZ+HYEINe4vPi6Y7dx8V59tf9bhFvoPoqSTYT26TW7AiZb8xRKXVPWY0Ps7Yacu39Y/Nc3S//4j9ejCSiST00nFcv44AmzdP3kyRjrirGdfYs293CAe2cNSy4HPl4m0O3D4GM1QvpABYi2V05sQLMWkXena8xY+NLdx4iDye8oFYTZ9K8PRacI8hHw9J8khstxqu5wwr6GTBDPgBrpcrEVj2GOCtFR2myTpLwFDKbb/Bhznlc8nBy/277KddBFR9tasEVoeQ8yKWFq4RNDV+kKde1vFvN310qpBllxbwOar4h9YCx9SQol0nZb6gyVHf0twB/FfUOdjUinwrFj07pxpKiGXRNaxVJmVZGKBEcZVskolvydq0FhGwanBDmXVs6fyWTVM2gkBCxCNgdP36glTUMYJ3bmyJVSkBd5GE2ZcQJCwoK8848AyRbdwgM1xToEFZDd7/6QXSq18MJtkEmVnCx41Q2HT+eE5bHIDPA87+RYSMWToQLlTWP0QmhaMLYA4lMlz5R4fSZyfMRP2UCFaszpDHLi8g9uBkOnC8vQPADswzcMtb4d+xKbANKZrOo8bk9a8gq2QT3s +xkq1l6g UMmyckSegDJzGAotBxd4f8VKjsy+oRwkxDE4qdjmsQriiTjWXA1kbZwa97DNZxEPobptC3570qHDhQeHIBYSeud0wmzGgiC5xU8VXttlcKKwGHJ3ZaeAIv60tUOAFsdiQcEcJIX6I0Nv9bHApefTvl5GlwQOW731BXqo6coF2ADchE0rt6kr00VzAoCttmGHx8te5IEz/Bm81KC/0yucJgl6yt/se4Atz1Qm+2tkZnTgt6sMtUQ2dR0olGIvFdVHttmQzvL5aNKtqxy613Kaof10kh9lUeydNwWkhrQRSRdqyvFmGS33neXqynaqV2DRaq9RONp9jltP34KjuQ7Sp/YcPvO17RtVhDodgyPppl3ZJ0K+n1VPBSe1tvzZkoEaAgXQPu3XHuS5leS5tvtOD/r3b7bNcVI3KeNBJODOsQELVepdjhPLPZfoaN23tLJD8GbJg X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch fixes an issue where the zswap global shrinker stopped iterating through the memcg tree. The problem was that `shrink_worker()` would stop iterating when a memcg was being offlined and restart from the tree root. Now, it properly handles the offlining memcg and continues shrinking with the next memcg. This patch also modified handing of the lock for offlined memcg cleaner to adapt the change in the iteration, and avoid negligibly rare skipping of a memcg from shrink iteration. Fixes: a65b0e7607cc ("zswap: make shrinking memcg-aware") Signed-off-by: Takero Funaki --- mm/zswap.c | 87 ++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 68 insertions(+), 19 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 80c634acb8d5..d720a42069b6 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -827,12 +827,27 @@ void zswap_folio_swapin(struct folio *folio) } } +/* + * This function should be called when a memcg is being offlined. + * + * Since the global shrinker shrink_worker() may hold a reference + * of the memcg, we must check and release the reference in + * zswap_next_shrink. + * + * shrink_worker() must handle the case where this function releases + * the reference of memcg being shrunk. + */ void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg) { /* lock out zswap shrinker walking memcg tree */ spin_lock(&zswap_shrink_lock); - if (zswap_next_shrink == memcg) - zswap_next_shrink = mem_cgroup_iter(NULL, zswap_next_shrink, NULL); + + if (READ_ONCE(zswap_next_shrink) == memcg) { + /* put back reference and advance the cursor */ + memcg = mem_cgroup_iter(NULL, memcg, NULL); + WRITE_ONCE(zswap_next_shrink, memcg); + } + spin_unlock(&zswap_shrink_lock); } @@ -1401,25 +1416,44 @@ static int shrink_memcg(struct mem_cgroup *memcg) static void shrink_worker(struct work_struct *w) { - struct mem_cgroup *memcg; + struct mem_cgroup *memcg = NULL; + struct mem_cgroup *next_memcg; int ret, failures = 0; unsigned long thr; /* Reclaim down to the accept threshold */ thr = zswap_accept_thr_pages(); - /* global reclaim will select cgroup in a round-robin fashion. */ + /* global reclaim will select cgroup in a round-robin fashion. + * + * We save iteration cursor memcg into zswap_next_shrink, + * which can be modified by the offline memcg cleaner + * zswap_memcg_offline_cleanup(). + * + * Since the offline cleaner is called only once, we cannot abandone + * offline memcg reference in zswap_next_shrink. + * We can rely on the cleaner only if we get online memcg under lock. + * If we get offline memcg, we cannot determine the cleaner will be + * called later. We must put it before returning from this function. + */ do { +iternext: spin_lock(&zswap_shrink_lock); - zswap_next_shrink = mem_cgroup_iter(NULL, zswap_next_shrink, NULL); - memcg = zswap_next_shrink; + next_memcg = READ_ONCE(zswap_next_shrink); + + if (memcg != next_memcg) { + /* + * Ours was released by offlining. + * Use the saved memcg reference. + */ + memcg = next_memcg; + } else { + /* advance cursor */ + memcg = mem_cgroup_iter(NULL, memcg, NULL); + WRITE_ONCE(zswap_next_shrink, memcg); + } /* - * We need to retry if we have gone through a full round trip, or if we - * got an offline memcg (or else we risk undoing the effect of the - * zswap memcg offlining cleanup callback). This is not catastrophic - * per se, but it will keep the now offlined memcg hostage for a while. - * * Note that if we got an online memcg, we will keep the extra * reference in case the original reference obtained by mem_cgroup_iter * is dropped by the zswap memcg offlining callback, ensuring that the @@ -1434,16 +1468,25 @@ static void shrink_worker(struct work_struct *w) } if (!mem_cgroup_tryget_online(memcg)) { - /* drop the reference from mem_cgroup_iter() */ - mem_cgroup_iter_break(NULL, memcg); - zswap_next_shrink = NULL; + /* + * It is an offline memcg which we cannot shrink + * until its pages are reparented. + * + * Since we cannot determine if the offline cleaner has + * been already called or not, the offline memcg must be + * put back unconditonally. We cannot abort the loop while + * zswap_next_shrink has a reference of this offline memcg. + */ spin_unlock(&zswap_shrink_lock); - - if (++failures == MAX_RECLAIM_RETRIES) - break; - - goto resched; + goto iternext; } + /* + * We got an extra memcg reference before unlocking. + * The cleaner cannot free it using zswap_next_shrink. + * + * Our memcg can be offlined after we get online memcg here. + * In this case, the cleaner is waiting the lock just behind us. + */ spin_unlock(&zswap_shrink_lock); ret = shrink_memcg(memcg); @@ -1457,6 +1500,12 @@ static void shrink_worker(struct work_struct *w) resched: cond_resched(); } while (zswap_total_pages() > thr); + + /* + * We can still hold the original memcg reference. + * The reference is stored in zswap_next_shrink, and then reused + * by the next shrink_worker(). + */ } /********************************* From patchwork Sat Jun 8 15:53:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Takero Funaki X-Patchwork-Id: 13691031 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2FC81C25B76 for ; Sat, 8 Jun 2024 15:53:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B2AD76B0092; Sat, 8 Jun 2024 11:53:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A67BC6B0093; Sat, 8 Jun 2024 11:53:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7F48B6B0095; Sat, 8 Jun 2024 11:53:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 5C7B76B0092 for ; Sat, 8 Jun 2024 11:53:33 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 0F7ED1205DB for ; Sat, 8 Jun 2024 15:53:33 +0000 (UTC) X-FDA: 82208166306.07.550C440 Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) by imf24.hostedemail.com (Postfix) with ESMTP id 3A6AA180003 for ; Sat, 8 Jun 2024 15:53:31 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ip2FzWr5; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf24.hostedemail.com: domain of flintglass@gmail.com designates 209.85.215.176 as permitted sender) smtp.mailfrom=flintglass@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717862011; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+SvMQN/q38nefiYyRiVmPEMKlu3DU/JjYA/6JNmM/yw=; b=HJwbLGLe5l+TGV25yOSJTJkm0xajNNunes6NL+pVlK832m81ty6VT2Gr11tFj0S+wWn6bH 2SHTPnQ3h+uCJeQz3qei2A+ywRxIyG4YD1/LvWSipyLFz6Ey9O/JW/SURlE91I+hPDNJHu oTlI8Aubj54RGDA8vggC+2SqwKTTc/w= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ip2FzWr5; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf24.hostedemail.com: domain of flintglass@gmail.com designates 209.85.215.176 as permitted sender) smtp.mailfrom=flintglass@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717862011; a=rsa-sha256; cv=none; b=753ByVHdhJQs2KSAgud8hhrRtDxGaYacq2wN8OBxbCccOm9qpsHDxM0H8XFqX8h5IfyMAI vqOrrKRiGRyqzZlxYBxbXmr8BQHgd56PQPcaUDcb9CYFEKf8E56Hv2JMRwMMnGjM4y7afc nTvctH0XJxMnP91EVxrjtonqN70xPbM= Received: by mail-pg1-f176.google.com with SMTP id 41be03b00d2f7-681ad081695so2394741a12.3 for ; Sat, 08 Jun 2024 08:53:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717862010; x=1718466810; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+SvMQN/q38nefiYyRiVmPEMKlu3DU/JjYA/6JNmM/yw=; b=ip2FzWr5ly+v87n0gOeR6rR9m3yKtjLnImcWrwtxTtfy6+oIAwESVY+FMv7Lt/FDG/ HG+H0Cjwm3ivhRMQBXd348MBoOgiX5wQf7Rxe48awzeA8eqUV12PweyW3RWuYg2xJZU/ Vo9Csb3j5WUYMEl2AeedugULMaT3EcUX9FY5GpBd+zRnbx+Pwmuj7x1rksQzHxG1MrTw 8aq/AQ5cPxfh7TONSo6GYOMH97jTyn/wjEPs3sA1nioF/7ZJW6XxdZKnjx7ixmO/YMUf HUuOMLc7GfCoGY4hEKkWWwUIBoAuBfd2yfqoXHvJ19yQreRnGUUYMaOJjztZ5ABSeoeV ckEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717862010; x=1718466810; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+SvMQN/q38nefiYyRiVmPEMKlu3DU/JjYA/6JNmM/yw=; b=JO+WSPO8vpy+siRHb2wGXKjN2U8s9ZjpKCCTlQE7d2P8Ake6VwqTJVF82I8k4U4L6v Rq92HAulQQNLwtUtUZh4ZYkpQDD6/Xvu2VzNVFtABuM7zAF3onAKl9P0R9qC8dbiRTtF 5FW8Fb/ISr2aQYrGn1U0iYIWvGlmRRxXIJG/tp/cG8HlKHqYYJpgaPPfmis8zpqPgyzK SVcNUdL62x6Di71nQxyazaS9nNdNql3voNkyNdNtEzk+8NG8rArZ9PnMHsThlZ4bqgGM pWveg6goZHiDSyxaO3VSjVnZWNzY1pUQ0fYX9OOH81u45oPSuFN9mTMAx0dQPDT0Y1ij rsBg== X-Forwarded-Encrypted: i=1; AJvYcCVxwp8sZuxG7zg4Xv6wr3PUem398N09XeD6APYqWyMuR6+9RlBQHRqAc0a3S/KWOmsmWnnO3YX+KnfYAVuLI57l9gI= X-Gm-Message-State: AOJu0Yx2BsfX9yQvTvGwQLa413TdUJL25fffkM0O/AGSTDhx+btGoFvI xhLZ4MPR94sDRahTHld5ZOa0eY9CjU36xRPE0696qbaaT+ra2LW/ X-Google-Smtp-Source: AGHT+IEG/zCdA3pqdzro3I+Qtktx3F9FKRDjkiSf63QDjZuWrfOfFPcvyEL5zE403+r39r2Uach56A== X-Received: by 2002:a05:6a20:12d6:b0:1b2:b220:2db6 with SMTP id adf61e73a8af0-1b2f969ed25mr5659976637.6.1717862009830; Sat, 08 Jun 2024 08:53:29 -0700 (PDT) Received: from cbuild.srv.usb0.net (uw2.srv.usb0.net. [185.197.30.200]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-703fd4d9d8fsm4335209b3a.149.2024.06.08.08.53.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 08 Jun 2024 08:53:29 -0700 (PDT) From: Takero Funaki To: Johannes Weiner , Yosry Ahmed , Nhat Pham , Chengming Zhou , Jonathan Corbet , Andrew Morton , Domenico Cerasuolo Cc: Takero Funaki , linux-mm@kvack.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v1 2/3] mm: zswap: fix global shrinker error handling logic Date: Sat, 8 Jun 2024 15:53:09 +0000 Message-ID: <20240608155316.451600-3-flintglass@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240608155316.451600-1-flintglass@gmail.com> References: <20240608155316.451600-1-flintglass@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 3A6AA180003 X-Stat-Signature: jh39fe6krfkdo9nn9sx3dpk8ncjnz5ro X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1717862011-453476 X-HE-Meta: U2FsdGVkX1+TAPCu6M809fWMfKW5lqSv/EdzzNSrBQ//Ho+iNr6LxOzxGiklnEF7nN0qg4o+Lk0FzFWMCte1YxAPPVumr7FptjgFCXZ4mjzwoNuoxYdYi1sMOU95m0UXownFJR4tH+wXEczfao3iJxh0coAYADL+U5hJKjqBQma6hHKeryckSQefptgFfKt2m/nRb4bkrxW1AE7zfgGBTVCDyiI/M7qUGGvDIL1Y/RIaA/z5jYyUaAnF+KttIQYF+asBBph1HFOiCiey2KpRJF9H8b5HLZPkoiFbD1yt3Q3lm9L6LUQ4kbm6DP3Z296j0h0HlYusyKfdfewS5ZXwOee4P3TyLAMuwQ6gc7vEk4FHEjHqQ64a0nWxlrbavqB7w7hFDxlldjUQh+maSKCK/2ro4QTwkE825X/H9n/d6BKWd/kZXWFd9qTYmRoMeLE5hR7bzVZqEGZDZC2L8tvi1eUQwXfjyVQSRb5Lu2IKmz2A0uyo9j5Q+/61V3Fiw9cC4GTzNnlz+zULde/R6awjDmq7C717ofTa5J9S4O/EAOBMoJ1wCer0Z8kWxVMg0V5mtJyzjiOiSo/AVrSQ3bSJo0agNZByqU0X9J6qct1eDyQKwR3e104GPepJLqJPXg3Y7hdBlpBFh3oW4rUBkRcU1uF2H1KEE8UIKdY9kNE9h4+WK7BNBD7su3NiYoot2jUt/rdI2+6qBCvhEWMZCr9khJHUz/VFWnqsSW/ULca6EWVoztWqYnNeqoce4ct7X9gWon8vlQ9h3IggdCcXN9EauL7S4Y18B3wU4dbOwV0cH4dGQy97/Fntzwr2NzVGDw9xb2WZfoNctFKUIHYki2FkVC5agq1wVW5r4DegrkRzL7Q2EFvGrna9VoHT16NahfiYVtaQtgKIOFfqUfORnEnxrVTRs9rhT56utIOmGnSMdK7VQxukhS3PJF6Qf3ovijE195I58AmsyOxZKQ/vG6c XXeSQD9x btjBBOX9zMAlruGwiNbyUIGIT8ayT1JILORnhXIC/eaodYZsxOLIUUHqmL6aCTWUuAb5aARZgRMXYC05FZgEJ08s3a7LcExKB5zKeVu7Rlac9kDg5hQjWJ3wXIqa0XT9gdpIMvTBcvJZEkcztoV8DwLEdCQHz8DRnxjCCHPtbgNfy7VLlOWJo7oAV3c+ap3hyOr+QiCDCxvKe/hAlYupKMQ5EdVEIEAdl0LEH6mFYQGP65cK+Jwl47PVO99cXm17lDc7VkcKnAoykm39R71yZo8tUu9hL0uYug6BJVdIje3vGyoSUJRr8/DXAitxyPzuoyavam3WVkJyAMjgCi2nBFAQpczYrZLJJyqIZR4PoCq0qKD/zM8F5Eeg62OdULcrEvXfPSL1UmvZgK9Xq5p9/1RoOqdHIkC04HJ9zdF2T2Ww3IPaymINPxHvNqQJmCVzNalBJ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch fixes zswap global shrinker that did not shrink zpool as expected. The issue it addresses is that `shrink_worker()` did not distinguish between unexpected errors and expected error codes that should be skipped, such as when there is no stored page in a memcg. This led to the shrinking process being aborted on the expected error codes. The shrinker should ignore these cases and skip to the next memcg. However, skipping all memcgs presents another problem. To address this, this patch tracks progress while walking the memcg tree and checks for progress once the tree walk is completed. To handle the empty memcg case, the helper function `shrink_memcg()` is modified to check if the memcg is empty and then return -ENOENT. Fixes: a65b0e7607cc ("zswap: make shrinking memcg-aware") Signed-off-by: Takero Funaki --- mm/zswap.c | 31 ++++++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index d720a42069b6..1a90f434f247 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1393,7 +1393,7 @@ static struct shrinker *zswap_alloc_shrinker(void) static int shrink_memcg(struct mem_cgroup *memcg) { - int nid, shrunk = 0; + int nid, shrunk = 0, stored = 0; if (!mem_cgroup_zswap_writeback_enabled(memcg)) return -EINVAL; @@ -1408,9 +1408,16 @@ static int shrink_memcg(struct mem_cgroup *memcg) for_each_node_state(nid, N_NORMAL_MEMORY) { unsigned long nr_to_walk = 1; + if (!list_lru_count_one(&zswap_list_lru, nid, memcg)) + continue; + ++stored; shrunk += list_lru_walk_one(&zswap_list_lru, nid, memcg, &shrink_memcg_cb, NULL, &nr_to_walk); } + + if (!stored) + return -ENOENT; + return shrunk ? 0 : -EAGAIN; } @@ -1418,12 +1425,18 @@ static void shrink_worker(struct work_struct *w) { struct mem_cgroup *memcg = NULL; struct mem_cgroup *next_memcg; - int ret, failures = 0; + int ret, failures = 0, progress; unsigned long thr; /* Reclaim down to the accept threshold */ thr = zswap_accept_thr_pages(); + /* + * We might start from the last memcg. + * That is not a failure. + */ + progress = 1; + /* global reclaim will select cgroup in a round-robin fashion. * * We save iteration cursor memcg into zswap_next_shrink, @@ -1461,9 +1474,12 @@ static void shrink_worker(struct work_struct *w) */ if (!memcg) { spin_unlock(&zswap_shrink_lock); - if (++failures == MAX_RECLAIM_RETRIES) + + /* tree walk completed but no progress */ + if (!progress && ++failures == MAX_RECLAIM_RETRIES) break; + progress = 0; goto resched; } @@ -1493,10 +1509,15 @@ static void shrink_worker(struct work_struct *w) /* drop the extra reference */ mem_cgroup_put(memcg); - if (ret == -EINVAL) - break; + /* not a writeback candidate memcg */ + if (ret == -EINVAL || ret == -ENOENT) + continue; + if (ret && ++failures == MAX_RECLAIM_RETRIES) break; + + ++progress; + /* reschedule as we performed some IO */ resched: cond_resched(); } while (zswap_total_pages() > thr); From patchwork Sat Jun 8 15:53:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Takero Funaki X-Patchwork-Id: 13691032 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E480C27C5F for ; Sat, 8 Jun 2024 15:53:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 86C9C6B0095; Sat, 8 Jun 2024 11:53:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7A5406B0096; Sat, 8 Jun 2024 11:53:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5851E6B0098; Sat, 8 Jun 2024 11:53:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 33AB16B0095 for ; Sat, 8 Jun 2024 11:53:35 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id A595A120160 for ; Sat, 8 Jun 2024 15:53:34 +0000 (UTC) X-FDA: 82208166348.10.C5B6925 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf16.hostedemail.com (Postfix) with ESMTP id D3661180004 for ; Sat, 8 Jun 2024 15:53:32 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=UeKZFZ3U; spf=pass (imf16.hostedemail.com: domain of flintglass@gmail.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=flintglass@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717862012; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HrIr32TNVXDKH7oG1xumlOKq/cguM0V8sG+9FXVj9vY=; b=4XKhis/2lsaxnayB/VfZVW8h1kUdEOWksIRLv9Vq8id+LPbVqt7z9lQiWJpKS6wwV9nC21 p0Da2avHx424gOfh5bVFIIBnuXK/V72jD+ou6YuxcTYOAu7i0ml1Vqfj0WOydrSUiK6ZX8 3lVGrZKKFHtW+RheOQfkxrRTv6CIl44= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717862012; a=rsa-sha256; cv=none; b=m9cSdmZidgQB+qa1bqTEkT92GIwLsSvWz0rVhtSIJnM+DIID9Q06V+G7wUk7AjU9fwWTG0 80JTw4dRnG1EizaaT6DG8bVSaiaFS46u5KX0FF+1HYjQ7WGJ5I/Kdt3m7MNv5fvFBwj1rw xNjTdEIHQGd957MdzSl1mITGidLNxhI= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=UeKZFZ3U; spf=pass (imf16.hostedemail.com: domain of flintglass@gmail.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=flintglass@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-701b0b0be38so3010200b3a.0 for ; Sat, 08 Jun 2024 08:53:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717862012; x=1718466812; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HrIr32TNVXDKH7oG1xumlOKq/cguM0V8sG+9FXVj9vY=; b=UeKZFZ3U10RLVCdgkeECyhVEeSnruyQMWAfT0ZWf+FVTVB9ddXenZ5fcVeXY9fch5T gMqib5ccXd+7InENt5oC04I5LjWC7W2JOHCzRf8SHYuDiobTEWN874LpU5pTVSscm7hz V1IOX4Pr31jWS3+rTphL+uT0KQxbK4xC42Dq+QC6ubk3+VoqC+XFU8AKYuQMcrVEpbM6 1U7HI1UMr4LNVxKHIKZpqKu/Zs400C7UlDNwSO9VKF89Uz5XvnLrN4srxJwHwDaLqZZg NcR0T7JYV7bdIUMEkonuvSXSjS0lQPZWPXkH/QGNW16I4lyX874YjYc3h7sZjLYGeZDU 8Y6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717862012; x=1718466812; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HrIr32TNVXDKH7oG1xumlOKq/cguM0V8sG+9FXVj9vY=; b=dUsd0fvxLnvnMct0B6lcSNeEJfrhUnjzbthyL1FdgG0NmX+u6TKQJO0ROHsNNwAkfN SUSkgm8usOP7iX4YG0fVWe1uRWK+CB6yiFYPSSh4NqUOJTVTyoO/YE5v+JYFgNJ68XUq xY94jirOSgTdCBEnn0B7YP0gsCfrnkoaiB55QkpfcJWrgNEAgiMt/Pz1fqGhxqWzqpm2 2fMsGi1h+DiyW43llHlTfZwRcnNUnVXpIsqmNPqiqHRg2VPAu6LAx+wbvhub1ajBTbOB ZKPRyizS3ssd2GsqRBQ8F1C8agpACeK3KkKR5OIyjCO0LwHk13Vp80waTbH0Nurfiln+ Vnxw== X-Forwarded-Encrypted: i=1; AJvYcCXr05amVwVXD/nJFBiHDT99Y7hlXub6utmV5Tv9rnxkhSIoghwJYb46oU88gdbA268xCihLxhKhrid3ZplKHQVfOs0= X-Gm-Message-State: AOJu0Ywdx/+Yl3mYqHxNfFJOY1FWgx64j7ytCgKCs9edeCYnaUfPqi9v cHQLlKWJlzzXDt588oETn/JvYEEOS+0pMfuRT+Ql3S0NRd4iydFP X-Google-Smtp-Source: AGHT+IEteUQgfUsrWSdHev4f9MAtLQvgT4O27ZIatD/ynyxVmEqXJoDhlhnvAnSdhI3u3ENtRXai/Q== X-Received: by 2002:a05:6a20:43a2:b0:1b4:2a8:629 with SMTP id adf61e73a8af0-1b402a8081dmr4737912637.53.1717862011509; Sat, 08 Jun 2024 08:53:31 -0700 (PDT) Received: from cbuild.srv.usb0.net (uw2.srv.usb0.net. [185.197.30.200]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-703fd4d9d8fsm4335209b3a.149.2024.06.08.08.53.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 08 Jun 2024 08:53:31 -0700 (PDT) From: Takero Funaki To: Johannes Weiner , Yosry Ahmed , Nhat Pham , Chengming Zhou , Jonathan Corbet , Andrew Morton , Domenico Cerasuolo Cc: Takero Funaki , linux-mm@kvack.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v1 3/3] mm: zswap: proactive shrinking before pool size limit is hit Date: Sat, 8 Jun 2024 15:53:10 +0000 Message-ID: <20240608155316.451600-4-flintglass@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240608155316.451600-1-flintglass@gmail.com> References: <20240608155316.451600-1-flintglass@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: D3661180004 X-Rspam-User: X-Stat-Signature: x1zp5931psu3bgng8f66spfhm9demutp X-HE-Tag: 1717862012-101743 X-HE-Meta: U2FsdGVkX1/q8IaT6MBVzYcsFrKvhJmDAtbqtrpB233GsPb1+iml8O3N5+pOcbLwmLraHkXjpY0r+FTaGc+HqVcCEEp0A4Mmk5hk3QYwRuws8EuViqSlIlTv46dDFlwjHYF1pg3uTqm1/svFpwTP2D5Ui0ptVSLbua8R7oPEWo/EJvZHUHVZpN3++RqpNktt3i0PvlEix8a6ak4H+8BVR9X8Rukbjc16ODKqTSG74jWm5p7Npz0ks0hCHPH2ienkIusLnhquDgsg4EacX48fUblPI8MpOOprxT6vWVT9vR6zs4NaChzKNXAhPvXs/RxwS+h07xk+qEu8hNZNjaoM3UsfldC9KLiYum7WUKrauMANpTez1ZfoyRf0JQgZNW141qsaNZoJD6gW06Z+BQJzrd3Yt5vmoFt6sVqaFzz6SRAHoF/XS/nbmFTbQs2PHjxlmgLExD8qTeKc8zmzZXNU9woe4nfcWDW3puzVdpC7oCJQ15bP6djd6kmaJU2U+Tyu5Isa8fFfgDTBV9xuqkNYJ8E9Rr7obl3UrDbf7cRhVp0mdplUUA2DyIqnc9sqMizhwAsT4CWwfhOLVscj2quuHFGhpr/Xq/CXl58LhCsg2n/drpKyZatn8py9mi9ewVlao3lsS2Sp+GLlqLSzOSlA6+mRrqzHbuheOannDRNudaQhj7aoKl8FvxOMHYl4vTIycHCGAvtcFF4TrR9cXkzs3SgabUAC72mxZpFn+qSdSJitkWd5pKOZDhems9RtSDl6QSpkcAaGRVTZT4l1lS9E6B8+3N5TqRlOY69MJGWqVoMeX5UNo3w5ZuxEvIZxZqmNWIUCoZEY6oNb/UxDBH7QbIRlUTucvpDxIn4LDfxPFKN6tmK54eXyDQpfx+ruqDI1NPq3dxYCMVsQUwHc8l1fsLXuokHi4sOpcEYllHuvbyse26gGGTEosp/x3j11OvR68FZ4bioxy/a/D9cXxfO /XIj4OSX wKVLYb+Lc9Qpg8mpCovHW0/Ux4YTYdeJLE0jBUW7vjVhyoQz8D/UZP8XeyL9X7zwegZI3d9eGHhIDzp47AFL6qcNA2gTAM0klm+6UaXfEuhpk1kS4EXjdbQ1WAJ42C/gXQ5AR2J8LsAu4yJeMvBxS/6F47wolg4XLAgiWdDUup6gTqcL5ypCmTqYTpZhbP+Zyi2TpZFhptPubHYq0oG0I6b5iSV4/yZy+qkfnItB4xoVXsiKBLpMp2ZbRqA2+BWxMHRVUeKgnMmKpqA2DBBAgjemO/fNuZ/B3btLhM7cboJ7o5HM7AX21v9Q0E31z4fHJ8IGSHoVWrJl/eJ9OIxXkVZ+lhoquNizeetJfJrT0ZyK1U2sWpKfa69Y/XhEEcShU+nNna/Fxzo70J3DdPS9wfnFcEqk1Bthg3KJCSx+6fMh/NoBzqSni64je9m2vMYsxpl/i X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch implements proactive shrinking of zswap pool before the max pool size limit is reached. This also changes zswap to accept new pages while the shrinker is running. To prevent zswap from rejecting new pages and incurring latency when zswap is full, this patch queues the global shrinker by a pool usage threshold between 100% and accept_thr_percent, instead of the max pool size. The pool size will be controlled between 90% to 91% for the default accept_thr_percent=90. Since the current global shrinker continues to shrink until accept_thr_percent, we do not need to maintain the hysteresis variable tracking the pool limit overage in zswap_store(). Before this patch, zswap rejected pages while the shrinker is running without incrementing zswap_pool_limit_hit counter. It could be a reason why zswap writethrough new pages before writeback old pages. With this patch, zswap accepts new pages while shrinking, and zswap increments the counter when and only when zswap rejects pages by the max pool size. Now, reclaims smaller than the proactive shrinking amount finish instantly and trigger background shrinking. Admins can check if new pages are buffered by zswap by monitoring the pool_limit_hit counter. The name of sysfs tunable accept_thr_percent is unchanged as it is still the stop condition of the shrinker. The respective documentation is updated to describe the new behavior. Signed-off-by: Takero Funaki --- Documentation/admin-guide/mm/zswap.rst | 17 ++++---- mm/zswap.c | 54 ++++++++++++++++---------- 2 files changed, 42 insertions(+), 29 deletions(-) diff --git a/Documentation/admin-guide/mm/zswap.rst b/Documentation/admin-guide/mm/zswap.rst index 3598dcd7dbe7..a1d8f167a27a 100644 --- a/Documentation/admin-guide/mm/zswap.rst +++ b/Documentation/admin-guide/mm/zswap.rst @@ -111,18 +111,17 @@ checked if it is a same-value filled page before compressing it. If true, the compressed length of the page is set to zero and the pattern or same-filled value is stored. -To prevent zswap from shrinking pool when zswap is full and there's a high -pressure on swap (this will result in flipping pages in and out zswap pool -without any real benefit but with a performance drop for the system), a -special parameter has been introduced to implement a sort of hysteresis to -refuse taking pages into zswap pool until it has sufficient space if the limit -has been hit. To set the threshold at which zswap would start accepting pages -again after it became full, use the sysfs ``accept_threshold_percent`` -attribute, e. g.:: +To prevent zswap from rejecting new pages and incurring latency when zswap is +full, zswap initiates a worker called global shrinker that proactively evicts +some pages from the pool to swap devices while the pool is reaching the limit. +The global shrinker continues to evict pages until there is sufficient space to +accept new pages. To control how many pages should remain in the pool, use the +sysfs ``accept_threshold_percent`` attribute as a percentage of the max pool +size, e. g.:: echo 80 > /sys/module/zswap/parameters/accept_threshold_percent -Setting this parameter to 100 will disable the hysteresis. +Setting this parameter to 100 will disable the proactive shrinking. Some users cannot tolerate the swapping that comes with zswap store failures and zswap writebacks. Swapping can be disabled entirely (without disabling diff --git a/mm/zswap.c b/mm/zswap.c index 1a90f434f247..e957bfdeaf70 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -71,8 +71,6 @@ static u64 zswap_reject_kmemcache_fail; /* Shrinker work queue */ static struct workqueue_struct *shrink_wq; -/* Pool limit was hit, we need to calm down */ -static bool zswap_pool_reached_full; /********************************* * tunables @@ -118,7 +116,10 @@ module_param_cb(zpool, &zswap_zpool_param_ops, &zswap_zpool_type, 0644); static unsigned int zswap_max_pool_percent = 20; module_param_named(max_pool_percent, zswap_max_pool_percent, uint, 0644); -/* The threshold for accepting new pages after the max_pool_percent was hit */ +/* + * The percentage of pool size that the global shrinker keeps in memory. + * It does not protect old pages from the dynamic shrinker. + */ static unsigned int zswap_accept_thr_percent = 90; /* of max pool size */ module_param_named(accept_threshold_percent, zswap_accept_thr_percent, uint, 0644); @@ -539,6 +540,20 @@ static unsigned long zswap_accept_thr_pages(void) return zswap_max_pages() * zswap_accept_thr_percent / 100; } +/* + * Returns threshold to start proactive global shrinking. + */ +static inline unsigned long zswap_shrink_start_pages(void) +{ + /* + * Shrinker will evict pages to the accept threshold. + * We add 1% to not schedule shrinker too frequently + * for small swapout. + */ + return zswap_max_pages() * + min(100, zswap_accept_thr_percent + 1) / 100; +} + unsigned long zswap_total_pages(void) { struct zswap_pool *pool; @@ -556,21 +571,6 @@ unsigned long zswap_total_pages(void) return total; } -static bool zswap_check_limits(void) -{ - unsigned long cur_pages = zswap_total_pages(); - unsigned long max_pages = zswap_max_pages(); - - if (cur_pages >= max_pages) { - zswap_pool_limit_hit++; - zswap_pool_reached_full = true; - } else if (zswap_pool_reached_full && - cur_pages <= zswap_accept_thr_pages()) { - zswap_pool_reached_full = false; - } - return zswap_pool_reached_full; -} - /********************************* * param callbacks **********************************/ @@ -1577,6 +1577,8 @@ bool zswap_store(struct folio *folio) struct obj_cgroup *objcg = NULL; struct mem_cgroup *memcg = NULL; unsigned long value; + unsigned long cur_pages; + bool need_global_shrink = false; VM_WARN_ON_ONCE(!folio_test_locked(folio)); VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); @@ -1599,8 +1601,17 @@ bool zswap_store(struct folio *folio) mem_cgroup_put(memcg); } - if (zswap_check_limits()) + cur_pages = zswap_total_pages(); + + if (cur_pages >= zswap_max_pages()) { + zswap_pool_limit_hit++; + need_global_shrink = true; goto reject; + } + + /* schedule shrink for incoming pages */ + if (cur_pages >= zswap_shrink_start_pages()) + queue_work(shrink_wq, &zswap_shrink_work); /* allocate entry */ entry = zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(folio)); @@ -1643,6 +1654,9 @@ bool zswap_store(struct folio *folio) WARN_ONCE(err != -ENOMEM, "unexpected xarray error: %d\n", err); zswap_reject_alloc_fail++; + + /* reduce entry in array */ + need_global_shrink = true; goto store_failed; } @@ -1692,7 +1706,7 @@ bool zswap_store(struct folio *folio) zswap_entry_cache_free(entry); reject: obj_cgroup_put(objcg); - if (zswap_pool_reached_full) + if (need_global_shrink) queue_work(shrink_wq, &zswap_shrink_work); check_old: /*