From patchwork Thu Nov 16 02:24:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13457531 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF4ACC072A2 for ; Thu, 16 Nov 2023 02:24:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 10B786B03E4; Wed, 15 Nov 2023 21:24:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 01EF86B03E6; Wed, 15 Nov 2023 21:24:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DB5606B03E8; Wed, 15 Nov 2023 21:24:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id BE42A6B03E4 for ; Wed, 15 Nov 2023 21:24:23 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 9D0B912035C for ; Thu, 16 Nov 2023 02:24:23 +0000 (UTC) X-FDA: 81462223206.01.EC43A1D Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf15.hostedemail.com (Postfix) with ESMTP id E2E85A001B for ; Thu, 16 Nov 2023 02:24:21 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=PqiSAEOI; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf15.hostedemail.com: domain of 3VX1VZQoKCNYQGKJQ29E658GG8D6.4GEDAFMP-EECN24C.GJ8@flex--yosryahmed.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3VX1VZQoKCNYQGKJQ29E658GG8D6.4GEDAFMP-EECN24C.GJ8@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700101461; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hOK1UyBshtkV5dW8DFB6ZrNswsHIuDTjDTN/Td8b6tw=; b=UHhqFjpaB/9argHwrQ1fcIHS5Y6guI31FKz7ja681bSVB1cxKViYB/LQeTG2FKrEkpgU4j lqXP5bEGLiwsB+6YWtax37KZH/j1MGRcml7gB6bfgX4EMKLwiIIrqYXVq7s4LFM8tXKtRV HOB//tvMqjPKwMAcftaEQV9G/If6kqs= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=PqiSAEOI; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf15.hostedemail.com: domain of 3VX1VZQoKCNYQGKJQ29E658GG8D6.4GEDAFMP-EECN24C.GJ8@flex--yosryahmed.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3VX1VZQoKCNYQGKJQ29E658GG8D6.4GEDAFMP-EECN24C.GJ8@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700101461; a=rsa-sha256; cv=none; b=5ZM628Ef9jv9CMQxPui0VHQ04FZztIfTn4gUy9Y4YbIDY1p5VvW2tvXLY+zkiYZpzcqbF+ dDzeVNL5ZwUzU72p6eqOw9TyUV2JVDyJh73Ix/wtZ1Obd9RiT5eZ8fvun74DG7eOLkkjzV 7FoQ5txigLcE5LOsyJWBoUtSdnaOi14= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-5b59662ff67so12690827b3.0 for ; Wed, 15 Nov 2023 18:24:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1700101461; x=1700706261; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=hOK1UyBshtkV5dW8DFB6ZrNswsHIuDTjDTN/Td8b6tw=; b=PqiSAEOIVgg45ICYwB1sjTBd/BJhJllQ0+irGCVRQgCTE17qd1GNcDfKZr3POUmbqP U6J7W0sPBl+u0rs4tDJJwq8oVyqVr/lhL8q6CLo5JlOmBFpffmoa5bsd8laGWnjtPU3E 7CVzew83aCpADZaGsb6Zu01wew2MQH0hqNu/Je21o/mWtgd9h8/+sEt9K8u5D9XPQg/W PDDwDEBsNGn7KZOZB7MgXC9uQQKOO37NZVSziURQasqny6288m9WXuttKJp4Y2NgzNpL EVQN4Uvv2iCNMlJW9B/i7+X8T0BiSFFvWg2Hnhdv9LEPjhiBj6Jrv5FBwq+simcpYB0F 1IpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700101461; x=1700706261; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hOK1UyBshtkV5dW8DFB6ZrNswsHIuDTjDTN/Td8b6tw=; b=op+WHveIoEzefFdG9Mpq9I+W9BWKmX3aSGbJDcXnX6J2UgBPcwDC1gw+7s7rK/h9KR Bnoc2MAo0NryIA3qT/+VFk7gvWC5lJ4VM0xHaqYnnhABEt8GXoyi3x1oEFEBevXMfRh7 alBPua8NmWrVGyqeAEL06eUjhXFCT6lco1veexI7V392CFZgcga27nl7xXkNwW8oKuTz NVk9IEbbp2/zaWhYq9AUUFfeY23jVWNNPx139HF3As3aIR15Tfbc776xDPZEoKaVrOg9 Utzk0v8e44fk36PtqNkx2WtEXbKWIFU6em0/T/nWoyTcfNZCVEeo/fF5bzZk6MYeIwF5 1TYw== X-Gm-Message-State: AOJu0Ywjho6wBq4c/N4v3ge4RYcOuRG9UDDViTUoekumRxXFcBup+P3W JwrVxpU1w7f2vi1KdAYEWfcAOHIFTCNRC/DW X-Google-Smtp-Source: AGHT+IEQtgXmf9iSCqMERLu3YFDRxYLIfFN9r8bkSUo7k+VLrtBcNBEUIUl5/rQbpV2uMfGCYzqdDt8J6V5tLAR0 X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a0d:d716:0:b0:5ae:75db:92c5 with SMTP id z22-20020a0dd716000000b005ae75db92c5mr12091ywd.2.1700101461069; Wed, 15 Nov 2023 18:24:21 -0800 (PST) Date: Thu, 16 Nov 2023 02:24:09 +0000 In-Reply-To: <20231116022411.2250072-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20231116022411.2250072-1-yosryahmed@google.com> X-Mailer: git-send-email 2.43.0.rc0.421.g78406f8d94-goog Message-ID: <20231116022411.2250072-5-yosryahmed@google.com> Subject: [PATCH v3 4/5] mm: workingset: move the stats flush into workingset_test_recent() From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Ivan Babrou , Tejun Heo , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Waiman Long , kernel-team@cloudflare.com, Wei Xu , Greg Thelen , Domenico Cerasuolo , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: E2E85A001B X-Stat-Signature: zjeq9ie1fxrn39j5k7mrzyubousg1suo X-HE-Tag: 1700101461-46699 X-HE-Meta: U2FsdGVkX1+SgxouWWVCC1yxEeyvKpyWE7hK3qzprDzHlDuTUAtxMo5wcqH3br9UgnnA9uwTR/+9LPOLf6CLA8wX8Kcj8NJCbR6ysqSAjQkHE87tja8/AHFuSaPzZpC80Dm8BwxsMXKbtJXfvaFyjcJD5+U+SZOKr8yaDX7i00+LwER37Antr68rI9nZ5edYGTU/n6am73QGARnox/nto07ZNguQp1/Soc55KN9mV/gT5FJ1ZFvOQKtfSVFoiIvZ2vOla8W2p8jOjzJLE4ZmRVBboBpGW8QljidL+T1BJK9FxeCFvcgEjU8dKk+HrbEe1qNwzimaGjGlGwxWi0IywpHtYWS9TLPxvPgzxKhOKGOFmc9824sq2BzQ4gQ9pfM3qkh6AjwZwMMfBh86OEXsFPuj54ZUCz2skvimIF9yM+Wond5DHCcQzGf9RRBQ8UNyHMCnJvOg6bWXhpW+UBjF7FKTTapyvoYs4aHljT18jGDVLKWcczxMENAXa86b0xJG9vCk4T2PJ7UIi5Glkxg4KgN0Ah7P+oyuwboeFMjcYAA8dIkiYwm/eVLLhiOA8Ck06vZc+mLaL9KSQEfiZ83rM1oyQMxoj6XDd4a5h4jmIZzulZJzwpVf2rCgiCzfQyZG11QCtKdJ5nGhHpqRGA353n/aYVy004oguU5yqB/GHv6TzuTgMmYOPfEKfwrx4vJSxe1oVRzhmwyUlUz5vtMNSdHMGPz9w541keMNpfalUQWbA198EBoFxPT8dutFhoKWRjofMy91h0ZC/o7o2ep55cNX/G8ts40yMKqSpMf3yRvr8eOaI0tbNADj/8XN10I8sVJ5d4b3vxhZ8x2epN4SfFy2itk6ogVDVzjThh2MchhFqRQvxFhvEqq+4P4/+OwaSqs8G0Naw9MYCAyMvzdgwdQrunqy3s63ZUhvf4dgLkoxRwpxtqLn9Wx74eFUN+Ug93jNTVX/b8xJFNMvPdH yOW/wL3I fG/XpOWlUeyywMfUR+3l/PcFGlW195ShaD4zNUX/0zZAO0ez5vFCQ1PwDVdcn6TLlUjqBWRVGEvAOlmxL5BybkDeR0ie6+fV5YqN40T272vN34ahTseuXyl2BP8y9WeUjoFs/NS2SUq7IyBwYQUC5dp6xsxjPdtdnD9CtLgxGeON5MfoH0kVXwbaVUVJWSPa/5xHje3ZxhFSwPORQkc3l243sx5SXNMTRctjIE8id5JRjdpwpFAq9udfh+HDCjffQN7/1AumRLEC3+wRqlpX4dhJsbHJoW9m5kkZMkrlvfXY1tP4VZ6dUcB/kXFxYW7Gj5xfyNp3B4b/R5L4pwStN6jrpFAYEERIV+AXyQaw76EJzEaWq011Dqmo8I8jzn8ClEN+km7sNN5OLy5K2MxvUt6M1Tc0TWPJ9V+TPMU9uSn9DHIJQf/3o8PqBueeKL7NswGYi2wTrvFgE8SUvTFdm0QjC3tIEtm4fXg4cwvUZNAZUdGmNXBKyk8rsRhJwiWtpEhYFSlo+xRMals2e+ohUaoqYcRb4Xi7l66vk88ZsIXlmLAZFpb7RQMITIi9/q3hw0A9LyEf3FhtUFurN3TfB+YK3ZEWSoVfKjHMaKxcb7mL+b9PXsSTe2e9Xuh2xVfiWmZQHJSsXtO/vfATvbLprRpQ3jV8GmZLHjglsGZ4Idbm++GKUJHmAmOI73vlWnc0/8gbJmG+2Y3Upr1Qtgt1dmot3gQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.031912, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The workingset code flushes the stats in workingset_refault() to get accurate stats of the eviction memcg. In preparation for more scoped flushed and passing the eviction memcg to the flush call, move the call to workingset_test_recent() where we have a pointer to the eviction memcg. The flush call is sleepable, and cannot be made in an rcu read section. Hence, minimize the rcu read section by also moving it into workingset_test_recent(). Furthermore, instead of holding the rcu read lock throughout workingset_test_recent(), only hold it briefly to get a ref on the eviction memcg. This allows us to make the flush call after we get the eviction memcg. As for workingset_refault(), nothing else there appears to be protected by rcu. The memcg of the faulted folio (which is not necessarily the same as the eviction memcg) is protected by the folio lock, which is held from all callsites. Add a VM_BUG_ON() to make sure this doesn't change from under us. No functional change intended. Signed-off-by: Yosry Ahmed Tested-by: Domenico Cerasuolo --- mm/workingset.c | 36 ++++++++++++++++++++++++------------ 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/mm/workingset.c b/mm/workingset.c index b192e44a0e7cc..a573be6c59fd9 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -425,8 +425,16 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset) struct pglist_data *pgdat; unsigned long eviction; - if (lru_gen_enabled()) - return lru_gen_test_recent(shadow, file, &eviction_lruvec, &eviction, workingset); + rcu_read_lock(); + + if (lru_gen_enabled()) { + bool recent = lru_gen_test_recent(shadow, file, + &eviction_lruvec, &eviction, workingset); + + rcu_read_unlock(); + return recent; + } + unpack_shadow(shadow, &memcgid, &pgdat, &eviction, workingset); eviction <<= bucket_order; @@ -448,8 +456,16 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset) * configurations instead. */ eviction_memcg = mem_cgroup_from_id(memcgid); - if (!mem_cgroup_disabled() && !eviction_memcg) + if (!mem_cgroup_disabled() && + (!eviction_memcg || !mem_cgroup_tryget(eviction_memcg))) { + rcu_read_unlock(); return false; + } + + rcu_read_unlock(); + + /* Flush stats (and potentially sleep) outside the RCU read section */ + mem_cgroup_flush_stats_ratelimited(); eviction_lruvec = mem_cgroup_lruvec(eviction_memcg, pgdat); refault = atomic_long_read(&eviction_lruvec->nonresident_age); @@ -493,6 +509,7 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset) } } + mem_cgroup_put(eviction_memcg); return refault_distance <= workingset_size; } @@ -519,19 +536,16 @@ void workingset_refault(struct folio *folio, void *shadow) return; } - /* Flush stats (and potentially sleep) before holding RCU read lock */ - mem_cgroup_flush_stats_ratelimited(); - - rcu_read_lock(); - /* * The activation decision for this folio is made at the level * where the eviction occurred, as that is where the LRU order * during folio reclaim is being determined. * * However, the cgroup that will own the folio is the one that - * is actually experiencing the refault event. + * is actually experiencing the refault event. Make sure the folio is + * locked to guarantee folio_memcg() stability throughout. */ + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); nr = folio_nr_pages(folio); memcg = folio_memcg(folio); pgdat = folio_pgdat(folio); @@ -540,7 +554,7 @@ void workingset_refault(struct folio *folio, void *shadow) mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); if (!workingset_test_recent(shadow, file, &workingset)) - goto out; + return; folio_set_active(folio); workingset_age_nonresident(lruvec, nr); @@ -556,8 +570,6 @@ void workingset_refault(struct folio *folio, void *shadow) lru_note_cost_refault(folio); mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + file, nr); } -out: - rcu_read_unlock(); } /**