From patchwork Thu Sep 5 00:30:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Kinsey Ho X-Patchwork-Id: 13791568 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F965CD4F4C for ; Thu, 5 Sep 2024 00:32:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BC54D6B015C; Wed, 4 Sep 2024 20:31:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B4D776B015D; Wed, 4 Sep 2024 20:31:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9534D6B0160; Wed, 4 Sep 2024 20:31:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 6FD976B015C for ; Wed, 4 Sep 2024 20:31:58 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 32FD312015F for ; Thu, 5 Sep 2024 00:31:58 +0000 (UTC) X-FDA: 82528807116.10.520A1DB Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf03.hostedemail.com (Postfix) with ESMTP id 5623E2000C for ; Thu, 5 Sep 2024 00:31:56 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="3oNAI/Gs"; spf=pass (imf03.hostedemail.com: domain of 3-vvYZggKCJM75AF1L4B3BB381.zB985AHK-997Ixz7.BE3@flex--kinseyho.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3-vvYZggKCJM75AF1L4B3BB381.zB985AHK-997Ixz7.BE3@flex--kinseyho.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725496219; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=C+NIQ3hruAHgC9xMj3fhofnWDiiLf7+Cp7wyaHwiSJ4=; b=CQAX2GSUITgLiWjP71AU2XBYVbi7eULUokHjJDgHJ+XlNUJGaMPCwWJc5AVsbtk1dRfHGn g+0a6azpijY/hmuFwI2LIGQXeOU4Qe298ELL+RWjqdHP6qK7iqS11eMHrPv2phRvEC4D7S Ozr8yV+9Yr6ieH4Y9fEYi9DvHGUox0I= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725496219; a=rsa-sha256; cv=none; b=MUxqMbEVrBHzwaIYHfer4Kp0zs8DAKnE2GapWpU0TB0WfO4OEDTHhLHjXRZ/7mTjmgzDcw CnGbz+olmJq/a228hHC4mD/i5PYAbdGumSo63O7m2QPiMIkb/5mE3EXnG2ruPnZ4eu42BH ABE+eQh3rCCBScy8J5sHkILilDwa3S0= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="3oNAI/Gs"; spf=pass (imf03.hostedemail.com: domain of 3-vvYZggKCJM75AF1L4B3BB381.zB985AHK-997Ixz7.BE3@flex--kinseyho.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3-vvYZggKCJM75AF1L4B3BB381.zB985AHK-997Ixz7.BE3@flex--kinseyho.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-7cd98f27becso285676a12.3 for ; Wed, 04 Sep 2024 17:31:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725496315; x=1726101115; darn=kvack.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=C+NIQ3hruAHgC9xMj3fhofnWDiiLf7+Cp7wyaHwiSJ4=; b=3oNAI/GsiGZkEU7P1lA0+DrQarS7McVWhwn3I2YN4S4QAPmtUL/BMGfaGOlG2ItnaO yscG5FGMI/jfq7vVdyDtCyLQJabLA7FhCv7TtL8n4b9eB8SpDL1zTtgP//Vd4OKL8rfv bjprAbwrxgGyVSle7wbJOclyepdE4qWfERJ7eyoFqFVVsb9UO4Dq2Gibbg5N4bZL5CQw PRzQzXllEnLYIBZmAuEeDfBvhiMlKdKnzdvNIOybvoPxo+SAwt+a4BvSCHm5DztBDwJZ 0PY5E0xHJKA8SzJRvbGm576Ajt1b9IC9RhGYrjvw9cN2Tg+oZzBz3+U1awgsDx7UvRq0 k0+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725496315; x=1726101115; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=C+NIQ3hruAHgC9xMj3fhofnWDiiLf7+Cp7wyaHwiSJ4=; b=vHT+M0VxjvFJxhOZloBCjS7vKowoda+a2Vnz2OvIAb5dD+Y3mtz6ik6m2sgmUMZlFt /GUKF4GArLCr4au62i0oro86l1gz5r3EFAoM5ymn1bZdRTJgvlNLig9aG1I530fnchLL imlzVeHVKMLKTn2PrydOMLNkJGCW2oZpVDpL3Hi4P3zz/8T5Z6TxSlpFflwSP88ApfEF G5BTZzbSTPMzhjTcsNtbEKvQMJh7PhksFJ31wa6tHdvrqlboKOhxrCwEn0vUtRyBtOUX SwMjWB1s1332oZ4RtdOGeaPfxQCi8gXiiltuYXZ+B6Is0kVF9QUFGkziRfwDLMh/XQbb PVAg== X-Gm-Message-State: AOJu0Yxaoe3zWyboJFZ6JUQOJCmZ235qEKlh6EglKKITQ0+mTkLGd+5+ lfFsz9Vw3lIGx0jbnmEnXEI6q+wqeUytsvwHf0T5AobzGjx8Ga4u0vj+UB+6avl+sZNfrKJtYUv hmZg5vb1OYQ== X-Google-Smtp-Source: AGHT+IFE9E8HfCvWJ7p3tjF6dU79+RYxA1wZ2PaXCVQ7+B7BuvDX9ZOeiHESUdZETs6+j99tFDtqjKU0xWP7eQ== X-Received: from kinseyct.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:46b]) (user=kinseyho job=sendgmr) by 2002:a63:1c16:0:b0:7cd:8558:9151 with SMTP id 41be03b00d2f7-7d4c1094218mr69531a12.10.1725496314864; Wed, 04 Sep 2024 17:31:54 -0700 (PDT) Date: Thu, 5 Sep 2024 00:30:53 +0000 In-Reply-To: <20240905003058.1859929-1-kinseyho@google.com> Mime-Version: 1.0 References: <20240905003058.1859929-1-kinseyho@google.com> X-Mailer: git-send-email 2.46.0.469.g59c65b2a67-goog Message-ID: <20240905003058.1859929-5-kinseyho@google.com> Subject: [PATCH mm-unstable v4 4/5] mm: restart if multiple traversals raced From: Kinsey Ho To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Yosry Ahmed , Roman Gushchin , Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Tejun Heo , Zefan Li , mkoutny@suse.com, "T . J . Mercier" , Hugh Dickins , Kinsey Ho , syzbot+e099d407346c45275ce9@syzkaller.appspotmail.com X-Stat-Signature: ijc5jn6sejfd31697uk8yz8e9yzt5qe7 X-Rspamd-Queue-Id: 5623E2000C X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1725496316-742609 X-HE-Meta: U2FsdGVkX1+QJE6uTKr9jJMj5Ab/xUPMH0qsU+7jFa3YfqaO5T4yLEd6krrh+rdSoEY4Mukdz/njHeOEOOxvq1mnoGQA6XQYrhoET9rntD/oIVu0V3hVK6UeEo8vg7FbHNe/5hR7G+HHGk1Xz7rpirfwZOjolRtlEL+9d4ydOI4U+Kw4O1khQSqVIG1K+uKBWd3iO2U0RyKiVLHlC5WIMmFetQnpas4jHomeAQUudDlhcUhNVKIkJ7urDaYkcQj4Huz6Aj+J+oA36xxM4kHFNQlunGq7Da+CvAM3RyuHizSdZhEEc93UzOjlJqPayaJ2rIMUXCMh0oR9kxQqisSwDaIHqmK4xDZz9KvzXPuVzMwWOEwzqF3AC75u0LsAcLKslMKYeDoKofp+/jiAV86Xa7rjwkZfVvfQzBk6xBbHKkdNdSTbKhDI7UAdzVygHeleKjLwee77u2nvzHv1w+iz+nxYhNixtake0ivYgeQqRWN1Sc1Qi23CxE4B5lXyOPhyIWuYbYf7McerFxSTNBdiqFEvuy9f6Yl7SuFoCPZIRtiqV0oSmzpymBDoUriiaV7TvROwQe5slQDR/dmpGnSS+C8V9t6WcfzydPBwbUjM3gVWdmnCZE1WlMwvgsBiKoSiBdyhPE6dEWpOZmZMXznHB4nqnhlU09zbBIs345YtLedMV6U8BKCd3dgrjez5oZsDviP2Gv/isbHM0xsEhZGV6YhTA+r0DAV+aV4jF3vpXfSs+ZMqtY3PiwpCEPLCAk7DLaUIPlT0M6bxDwUVrwUE1ATYrEXlDh91+mkzcHPfwBtNmXpnG2avSA6BJagvswMjhTU+XabsMe03FNi6NLFlLrlwbnfih1gnNiCiCK4SXj26bXNtUeXGgEqCyiSE1g/MXMblUsQ/sbOvLh6K2Q394FbaIlR2Xj3lmor2n+RlDLlABipq7Zcksqp2IOO47/Kp8Mr2s68GjBQw1jOSbA6 DmLHWr9e ZIt4llwA93ixfsxClr4H+ZD+MydQSy+5x9APb8MnHv5LyQWrbfdIBJ0AXJq34y69Hw0Uw5pXVLkJmBccJGN2KmZn0EI1T9db9BJBUpncKKzSdH7NAsarTDvogp50fLFAiA/KSnMr8JwyDI+yuTeyKFmKU98br59kp2fvcksMZjIYVkLyHqZEnOWmlsCYOCm68RLkkBxWMLlQ5IZBn/9wsCGFlWMFJudb8NLWNAuG8RXhMWV8EsU4cNKL3tYXJ7xR6NCITa6oy5Vgv1qiALmj2s4Zw1qtuI+rTcRpYtOKQnEW4W/6mrF6UW2yDEge05IdWM9alb61vDDGueqe0Ssd6vOFXZBqXJVSdqQWckhRhd+C3Y75InQUpGyVpcaUiVeY9JbHFy6aXSX2ZEl4e8P9F2oKKU7BaNI0HX7Hh+zC1+IbZbmLwRGrr0irwnIcmY1dth9WDeEeQ3fFdJ8M0eZc8xcGjb/zWaKFxi+ec0/i1VgoMXtKUr3aDOiNn89c6C9T1DdO17TZTBxl54QzWJnS2XUPwRhPG1iiRcq8YyttJFUPIkZCzHwO0jxIMMBBRHzQvk+HzPoq0B07qdBOZzvkGfte7wL4GnOFAjSkX0vgLIHOKLPPH4n3MRD5LzukQqjMYXLtWRXoTgLjt0KNq1IuCteX/xDPpfbUBMtLrqkKO2u50fSA5IWCQlgFEy+7rXKO68lGJcS1+LMpbM5FIqjEtDlISGSMgeU0zkedvYCeZluqPugIkQ9rUWHo3AhuLayHaKbWvMjx/Jmbv2dV/6NbyuF3hyhV2eh/I9bDQ87+ocRyF0NLs4FI2k2kPUiyi0q55mCPZbXH114peast/btCAsFvBA0a1NB44QfqeHdMwKukFpG/4FWLIe3kA5gbpyOBx4Aa6 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently, if multiple reclaimers raced on the same position, the reclaimers which detect the race will still reclaim from the same memcg. Instead, the reclaimers which detect the race should move on to the next memcg in the hierarchy. So, in the case where multiple traversals race, jump back to the start of the mem_cgroup_iter() function to find the next memcg in the hierarchy to reclaim from. Reported-by: syzbot+e099d407346c45275ce9@syzkaller.appspotmail.com Closes: https://lore.kernel.org/000000000000817cf10620e20d33@google.com/ Signed-off-by: Kinsey Ho Reviewed-by: T.J. Mercier Cc: Johannes Weiner Cc: Michal Hocko Cc: Michal Koutný Cc: Muchun Song Cc: Roman Gushchin Cc: Shakeel Butt Cc: Tejun Heo Cc: Yosry Ahmed Cc: Zefan Li Cc: Hugh Dickins --- include/linux/memcontrol.h | 4 ++-- mm/memcontrol.c | 26 +++++++++++++++++--------- 2 files changed, 19 insertions(+), 11 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index fe05fdb92779..2ef94c74847d 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -57,7 +57,7 @@ enum memcg_memory_event { struct mem_cgroup_reclaim_cookie { pg_data_t *pgdat; - unsigned int generation; + int generation; }; #ifdef CONFIG_MEMCG @@ -78,7 +78,7 @@ struct lruvec_stats; struct mem_cgroup_reclaim_iter { struct mem_cgroup *position; /* scan generation, increased every round-trip */ - unsigned int generation; + atomic_t generation; }; /* diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c24ef6a106e0..ce5d066393c9 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -988,8 +988,8 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup *root, struct mem_cgroup_reclaim_cookie *reclaim) { struct mem_cgroup_reclaim_iter *iter; - struct cgroup_subsys_state *css = NULL; - struct mem_cgroup *memcg = NULL; + struct cgroup_subsys_state *css; + struct mem_cgroup *memcg; struct mem_cgroup *pos = NULL; if (mem_cgroup_disabled()) @@ -1000,19 +1000,23 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup *root, rcu_read_lock(); restart: + memcg = NULL; + if (reclaim) { + int gen; struct mem_cgroup_per_node *mz; mz = root->nodeinfo[reclaim->pgdat->node_id]; iter = &mz->iter; + gen = atomic_read(&iter->generation); /* * On start, join the current reclaim iteration cycle. * Exit when a concurrent walker completes it. */ if (!prev) - reclaim->generation = iter->generation; - else if (reclaim->generation != iter->generation) + reclaim->generation = gen; + else if (reclaim->generation != gen) goto out_unlock; pos = READ_ONCE(iter->position); @@ -1020,8 +1024,7 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup *root, pos = prev; } - if (pos) - css = &pos->css; + css = pos ? &pos->css : NULL; for (;;) { css = css_next_descendant_pre(css, &root->css); @@ -1035,21 +1038,26 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup *root, * and kicking, and don't take an extra reference. */ if (css == &root->css || css_tryget(css)) { - memcg = mem_cgroup_from_css(css); break; } } + memcg = mem_cgroup_from_css(css); + if (reclaim) { /* * The position could have already been updated by a competing * thread, so check that the value hasn't changed since we read * it to avoid reclaiming from the same cgroup twice. */ - (void)cmpxchg(&iter->position, pos, memcg); + if (cmpxchg(&iter->position, pos, memcg) != pos) { + if (css && css != &root->css) + css_put(css); + goto restart; + } if (!memcg) { - iter->generation++; + atomic_inc(&iter->generation); /* * Reclaimers share the hierarchy walk, and a