From patchwork Wed Jul 24 19:02:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kinsey Ho X-Patchwork-Id: 13741274 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07471C3DA61 for ; Wed, 24 Jul 2024 19:02:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3E9186B008A; Wed, 24 Jul 2024 15:02:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 371756B008C; Wed, 24 Jul 2024 15:02:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1777D6B0092; Wed, 24 Jul 2024 15:02:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id EB3D66B008A for ; Wed, 24 Jul 2024 15:02:27 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 9C2E6807A6 for ; Wed, 24 Jul 2024 19:02:27 +0000 (UTC) X-FDA: 82375567134.05.93C4671 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf25.hostedemail.com (Postfix) with ESMTP id BB2D4A0027 for ; Wed, 24 Jul 2024 19:02:25 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="op/YRiLR"; spf=pass (imf25.hostedemail.com: domain of 3wE-hZggKCCEHFKPBVELDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--kinseyho.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3wE-hZggKCCEHFKPBVELDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--kinseyho.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721847721; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/TQiG1UXUGUFW+lVBdf0Q/W1aoqYwGoLbWZS7cahHzw=; b=G6zu1MGTqGSXYWdJMrE547iGq/zeJg5xeHUM18eX2007zhW/kWaoJv0s5fVDgzRxX9Qtvt /WdgqM3XMiPst4cUa/HSQlWN+syqdLD+A+oRaFUeIDBxwqiB78LmTZKOdfUlAL9aOmlSgq CQhriZmwi+qtSVH+BpTDF0KlWk7JQmQ= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="op/YRiLR"; spf=pass (imf25.hostedemail.com: domain of 3wE-hZggKCCEHFKPBVELDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--kinseyho.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3wE-hZggKCCEHFKPBVELDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--kinseyho.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721847721; a=rsa-sha256; cv=none; b=3hYZmNVML/3rSPVnepBIcTFkAQ85LaConHHGwAL4gt6gHZi7OqemrKGR4U2dg0BvaaVzfZ 0dCjDxECl0IJ9/zY3qoJd/1jgksqJXr/wx15kJ6DFetxGAn1RvmpOXsCCMVkunUgeKAVql MC4/IZ5Pj1jsW4Ej6W67oC1eGjno4Pk= Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-1fc5e651bcdso815605ad.3 for ; Wed, 24 Jul 2024 12:02:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1721847744; x=1722452544; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/TQiG1UXUGUFW+lVBdf0Q/W1aoqYwGoLbWZS7cahHzw=; b=op/YRiLRGAahUToKNvxkZDLgwLaxhndYx6THbByF5Wc64YglkEq2p1TmnvnUGFPFOa rUACb5tHV6xXlztiPkQOdisBw8jeRRe/wTolqe59O2qI0R9osyVEW1zi00dS/UasbSyB CgujoCnfiyh7Sft9bDgvtENWH56R5qJhB6grQoK/v9liHyNpijhtVXp5I/hIh8+mlOgk 9V6s7L9giOG/r8lb+ntgCSqC4hw23g6/MX3ZKRhDSR/EQaLBVR6yHDB++IQNkqk2qeM+ 7XtcFIoalQ/6eFZYuQ4jFf0KCW+hJgSA487nabefPQK02BSRV83E6W0Xkn74Pypy0UvU cilw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721847744; x=1722452544; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/TQiG1UXUGUFW+lVBdf0Q/W1aoqYwGoLbWZS7cahHzw=; b=pA9f63Ji8Y2dmya123T6sfufdP7VwYfn+rvCsy2zloelDGMT5SIqZgnrYNngVzO1Fc dXsWuBQa2Uw4fDBOnxNLofk1IJePESZzYMAUXuiRVzl4ovK7V1nyU1/8cxhkxx+zIjdt BdFoXbPKGxn0jWQx6tXkhoM01W81gdYjXKzPEoAx5Zuaa2fAvlLjsu9gaL8EEHw/ytMp qWa+ez57c/nCBLDjhQX/NfuW4EE1Ho+PuWCxwZYm0jNzP4X5IL3tQL9Rkr4TFwCOXMGn GE+4Gbof5SAC2Ds8on+k/KcuDRrFbONKyhBW8LGLrhfZn74mxC3Lb9SwHEQ4vyJpllZ0 xkXA== X-Gm-Message-State: AOJu0YxMDoiERu9ZkHwS1JFTdbsdfHL1xuMjLN3T3ZrSYEw+EGdjBBxD 9LQh+Cx/cIMq/3bEOIdafNkvejEt11RkfFXMSwQ/9vfGQOQ5zhExhQRuu2I43hlLzJQ4+oiaS3N D2BM+KgNRRQ== X-Google-Smtp-Source: AGHT+IE+tnQU31Bbj4dJVy+3PQYbdu9GD4nWCJk68x0wq+h9pms1Ho5g0zih9cGcyJ9eyRiAWRuy7HbM8eN7gg== X-Received: from kinseyct.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:46b]) (user=kinseyho job=sendgmr) by 2002:a17:903:24d:b0:1fc:369b:c1b5 with SMTP id d9443c01a7336-1fed3531a98mr17415ad.5.1721847744302; Wed, 24 Jul 2024 12:02:24 -0700 (PDT) Date: Wed, 24 Jul 2024 19:02:13 +0000 In-Reply-To: <20240724190214.1108049-1-kinseyho@google.com> Mime-Version: 1.0 References: <20240724190214.1108049-1-kinseyho@google.com> X-Mailer: git-send-email 2.45.2.1089.g2a221341d9-goog Message-ID: <20240724190214.1108049-4-kinseyho@google.com> Subject: [PATCH mm-unstable v1 3/4] mm: restart if multiple traversals raced From: Kinsey Ho To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Roman Gushchin , Kinsey Ho X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: BB2D4A0027 X-Stat-Signature: yw7tkptro46hr5cfejng1tb6e534ig1a X-Rspam-User: X-HE-Tag: 1721847745-363923 X-HE-Meta: U2FsdGVkX1/r3BNMJNN08emhNkHKXNX1EnOoGWG6IMYY8fRfPPZEv0MZv7wowh0i3PE6HTRmCRt+xj28OAuz/DIhVIdF5ymjX0sMsOatGh8f8wz6C4cx14lUgscUyB831esi4hHqTpvBIh4hiK8OkqO66Idj43+0UrzeUumTZTac3X4p/2B+BvSEBY1E+vg8wdK/etAcEUGZCQIAwLyICAIBvw8lEb2smRffOTz2PurWWC2q8Vt9laqxzyVTEGy+3WYSL5o1ZYGf1n9JsZLVFGOd8lJCZJiuXfjMs6EMVTV5knRkp5T9Dqz41Wl5Cptxnlf2z1Av5yHHvb8vUK+4SqLN5ucd7nvUlPs1+eQSl5TbFHP9QhlNuZdZrZW9fiXeb1vIqon6DvVQtBi4MQB/aL3UOciKUauJD0qw6IORzTlotpascFJY+3LgzzRsVWntrfxB2twT7qzX9igXka7V1Zt3RzK+U4qdCY0aqBPIewErL69+aFTEzca5mxBkajXwi6vVMCebiD8ZMEXeGRGjJtT7itEHzEX1q3QP4Bb0D99as/K8uVyCzem8BwLR6vZ+elM4quAdlnl33jyzNn3khxsg1xp3JgO+6OhL8H7IM0+EZ2e3SjISwejZ3ZyeZTt3dbjrwbv9b+LELTYv0dTm3zrF0ramgbQsZKwJ4M9Ulc8+OPvujcDCwFUoUkcEdgxUUc2xZohvK3+/DSPXTdNimzM8oFzoDPWbhHSHs46WAtIXOiDE7A3RWSSAOv/Wl9CILteU/VFUdE94ufZeR/3AGxx6dZxNOjKt8x4SQPLDdsmhJHIR+bhjbljPIQdkFpJNUXGeXYTdY3VwPs++xs7Uf/rCCYcFPw5xS5HiUTtSKJdNlkDNu22mVWkD7XVbAsPq8LBq6cnhYz2X6uH+RUOayVBnV1ZA7w+dk9YcIp5c1UqCkjKGBNmfdKXUMgWZnFBXd0t5sPcILJ4IQugsXwT UI4hLq/6 pa/Pom/hpBnydswKNx+O157iWky2Ggd8+4+VEu3EYT7Sr9qnWNpNM4Pmet0rpoliFG3sDr+SxmlMflG/pfbzuFQ07v/GzuHTDjYULfCEVqwD7YLRQw2M0ThJniyUq11Ert18KwLjdJ5ESqvQsQEhuY6pbM4hgcK/ar5KMZzPkh0tdBGjYP05fYAsia1Q6NKvchD5v2oA1uB7j6lXvlUHdBo6PqRf9ZyGFYu5uYg7QXNLSHGbFUCEY8bQsoU9WNbbrGCokIScZIDKj3UrWY1yfFH9zYZg3+khXyRnl5j00mf0q4vNgoyn/b3NcmFNygl5/eCS9iy5iE8PXpczuTemQmVfAAL4neEpiIegcgIjcaYLXAFdkOkgaGyvjiz8sXJxE6Zcqhk+JtQQCNZp4AYXk5VRvxxdkZwQWmkKzI8amV7402D7p8aIOpZgoTRgZD08j7ncTpCDR7a/6KjPQ3t3jrbG0wg5SqagXsLxPq1CUY8VZdHKt5EWBOchk9ek6thuxRnxbPGBed6grZyM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently, if multiple reclaimers raced on the same position, the reclaimers which detect the race will still reclaim from the same memcg. Instead, the reclaimers which detect the race should move on to the next memcg in the hierarchy. So, in the case where multiple traversals race, jump back to the start of the mem_cgroup_iter() function to find the next memcg in the hierarchy to reclaim from. Signed-off-by: Kinsey Ho --- include/linux/memcontrol.h | 4 ++-- mm/memcontrol.c | 14 ++++++++++---- 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 4cbab85e2e56..2b354abe6d48 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -57,7 +57,7 @@ enum memcg_memory_event { struct mem_cgroup_reclaim_cookie { pg_data_t *pgdat; - unsigned int generation; + int generation; }; #ifdef CONFIG_MEMCG @@ -77,7 +77,7 @@ struct lruvec_stats; struct mem_cgroup_reclaim_iter { struct mem_cgroup __rcu *position; /* scan generation, increased every round-trip */ - unsigned int generation; + atomic_t generation; }; /* diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f672bc47c6b5..4314a2b8848d 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1005,18 +1005,20 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup *root, rcu_read_lock(); restart: if (reclaim) { + int gen; struct mem_cgroup_per_node *mz; mz = root->nodeinfo[reclaim->pgdat->node_id]; iter = &mz->iter; + gen = atomic_read(&iter->generation); /* * On start, join the current reclaim iteration cycle. * Exit when a concurrent walker completes it. */ if (!prev) - reclaim->generation = iter->generation; - else if (reclaim->generation != iter->generation) + reclaim->generation = gen; + else if (reclaim->generation != gen) goto out_unlock; pos = rcu_dereference(iter->position); @@ -1050,10 +1052,14 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup *root, * thread, so check that the value hasn't changed since we read * it to avoid reclaiming from the same cgroup twice. */ - (void)cmpxchg(&iter->position, pos, memcg); + if (cmpxchg(&iter->position, pos, memcg) != pos) { + if (css && css != &root->css) + css_put(css); + goto restart; + } if (!memcg) { - iter->generation++; + atomic_inc(&iter->generation); /* * Reclaimers share the hierarchy walk, and a