From patchwork Tue Oct 10 03:21:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13414780 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6851CD68E4 for ; Tue, 10 Oct 2023 03:21:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6D6526B020B; Mon, 9 Oct 2023 23:21:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6630B6B020D; Mon, 9 Oct 2023 23:21:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B2F66B020E; Mon, 9 Oct 2023 23:21:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 330576B020B for ; Mon, 9 Oct 2023 23:21:30 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 0B729B4CE7 for ; Tue, 10 Oct 2023 03:21:30 +0000 (UTC) X-FDA: 81328101540.05.ABADC68 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf11.hostedemail.com (Postfix) with ESMTP id 3DC8240015 for ; Tue, 10 Oct 2023 03:21:28 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=F0dkyk3y; spf=pass (imf11.hostedemail.com: domain of 3N8MkZQoKCIA2swv2elqihksskpi.gsqpmry1-qqozego.svk@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3N8MkZQoKCIA2swv2elqihksskpi.gsqpmry1-qqozego.svk@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696908088; a=rsa-sha256; cv=none; b=hut5uoMpdwxrLmUWxQZtnPUlz8UIgcCvaDlyAUqQzaKJ3GHaZQDiCgXqRVuxln/bY0rPfl /rgcLC1HkK1kIp4MfSnJZ1rVi1voQ9NZJTbNhyRh5XO/AV+FBSAJYTMFo7asN7YPq1DjzE D/gtIeJlPfGojh62LHETuR+KEFAoEOk= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=F0dkyk3y; spf=pass (imf11.hostedemail.com: domain of 3N8MkZQoKCIA2swv2elqihksskpi.gsqpmry1-qqozego.svk@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3N8MkZQoKCIA2swv2elqihksskpi.gsqpmry1-qqozego.svk@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696908088; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2uIMTmtGdqUKunPgdDdu073V0kaqxVUtvtbHday0wrc=; b=nF5BLllLAPtVty9QcmlQ1nYe2swmAZEbXtgFhnDQGq2FhgsCtZKhu7ivNRG2icwkKX+o8b 2CpMM9Omluic5TdLQ1Gajqpxlo/R/nnwjISrz0srLllZbfmBRLDTKpfkZlxs0lRBxgQtKc rxl8kyIH1ssFg9q8b+xz0h0cASf57Xs= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5a7b53a48e5so10550827b3.1 for ; Mon, 09 Oct 2023 20:21:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1696908087; x=1697512887; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2uIMTmtGdqUKunPgdDdu073V0kaqxVUtvtbHday0wrc=; b=F0dkyk3ygiXSbRY8kq1sG+ZR6Ik5RNwrFMNXMRtUbGSYzBbXivlr63jxKyz9OJXWlA E90kaZsJYtpUYe2Ba8huxh43npX9C4o9x2svbhdTSt2d0AUp8Ty0cZ3eeLmFBifMikrY PR4ZeAeL8yKrlu4sKjFgd2+8VddmR050yVkBWvgGSX2lA4c+JnriNcYhZX8kqMgwohkP 0QvoBDGTHeItdHsR7rRxkPMS2CVTWf3W2dTA6gqOrLR9SDQ5WGfKAmci7ekcGXSGLSsi mDi7o5N6r339fOOmk4mTV/hbdS+Ol0PoNaPLE0VBYdzGIv0RL3bDZ0EgdRoTE+YT6fgI 5bEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696908087; x=1697512887; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2uIMTmtGdqUKunPgdDdu073V0kaqxVUtvtbHday0wrc=; b=DN5SaCyfCfR9F3Lg/VH8ZcORfRneMGtR5blMS8FhIVztcsgtGAwIJY+CUxzKqPo3HE JIOENhQROq/eTI2X45dEja0t0gOMc8APFt6LOmj/3M1yogAUwrllxZbH1QAoxdDBCWo+ W4erlRSLPGvjFjGBVdQ8HEqd6GnLaPrkg8pFHqcmQcJJh4xkpJ1HKjj/bok7F7+aR5d9 hN4JV+3bFcn59hIP3txz4F84PsfSmy8ffDi+3y8JoEs8n1TZ7PNbFGWFsWqinEX+FwX8 wZBCaawkVdrXd+9I9Uyxnu9wYP7w8OoKCXwD4ubFQJ0uSRcjfTfX4erWFW9oltMHzmkm f8VQ== X-Gm-Message-State: AOJu0YwGab71RmN5PpuuIkyby0M4uCQIpe0JDH2TBlEy4zIhDkDGCH3E Ne8BFOad+vOhodsMKw+4Q4Nh/gyJ6CYmDlMS X-Google-Smtp-Source: AGHT+IH27StA7m2woHqh/tcnCp3uYnoxUCBEP5s+DN8GVWI1rrXovlh8AYsn+eA2pINcXrMN7noHvFZw+UL9M22y X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a81:c44c:0:b0:586:50cf:e13f with SMTP id s12-20020a81c44c000000b0058650cfe13fmr286393ywj.1.1696908087335; Mon, 09 Oct 2023 20:21:27 -0700 (PDT) Date: Tue, 10 Oct 2023 03:21:15 +0000 In-Reply-To: <20231010032117.1577496-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20231010032117.1577496-1-yosryahmed@google.com> X-Mailer: git-send-email 2.42.0.609.gbb76f46606-goog Message-ID: <20231010032117.1577496-5-yosryahmed@google.com> Subject: [PATCH v2 4/5] mm: workingset: move the stats flush into workingset_test_recent() From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Ivan Babrou , Tejun Heo , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Waiman Long , kernel-team@cloudflare.com, Wei Xu , Greg Thelen , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 3DC8240015 X-Stat-Signature: hxbsjxeso7sjuex9q7cgo641ioe5915s X-Rspam-User: X-HE-Tag: 1696908088-848472 X-HE-Meta: U2FsdGVkX18Q3C91slTCb6PXtcUjsNDJfUMNRHQDUXhrd6nhf+Wsz7tTtB/XcsVrcFwdAafUfwSInHTKCUqQlQ0SWuyvYQ4f2zUs7JoSoO6MkQICwnVSdKKiiz01hCLTEATV/rdmWNUQTM5c7+zz57QPjUF5lhCqErbp+uo7vDNTTcYsmbCj4HWKBZo7Oy3kCgwc2FZX0DNvo87oCT/EiP9eTyhXJgFI3AIfh2cpOPIWVvVofmIx+MRKKBwkIid629K3Q0Ygbx0L4nsgHGNKnSUS0HZr4ABhk0fhpoXcpQTOIXdzV/5vd5aJdg1qjQ3wiYfawbbbtr/p1+gGnez0+Rj1177NI9tLJG0GrtJH6zjx2gvc1NokoOUBd8ua4OXZyuGcQfB6dUy9o5yz0ZBR8t1aQJNgCWpYg8oYAC1OBo8f+DWhSLZzGkyEFYCPbwWgwYpjiCKfowJ/jBO+AYOrXOJQlDWxZ9Rx+lvseSn/QTwlBGuCnPifQy8Ea4vv5xepPaMqBYLaWFc2SF6BJ5ljCUGwU+Xn9EFXobpA7W0MiEfg5GG1RgySQER+7xSPcaz+mGx7z9sTcpYLsDqj3H2NNDEP/JFOvy8UsaNz9PsiZPYopncoKNmVXnq52sM1j2l0T/2Nf6TeZiRZbAGtVPnMkyfjCCo+PdGnsQ2Ggzy6kvdwoV1N+bSaQI/30gOJX2vV9ldZWyYMm9veSjgHJ538KvCFy+fVtbxvFDtHVjWFEnIbZJLgJtz4u3Ksc4q4nl0xng4ebW70CXhuXIb9Dz5DnfP7R6xUj9CCFtERIWvg0QIh7h515reyvLTHURa15rOFbz3WhcekcrcU7nNcHEBVTPOauTB9JPuYzpfHMmWpMvBKneyA32+ysEZsvFZrKf59r1f1N3BNn1sKHZJO4KtzHrXpyXRhM7or/wUafZHbPmr8Nferd1tJn7RDePU2x5Lalb61343h29Xvig3AvSR imHYRG3s s/eVKnMUvls2WhKUJ0XlH5O/XFzPiBJ1UiJMap3DI366nQkrjUuxzgxc37T5r/qGU7/YXAEwH2dml4bF29YuwZ3bSzsRjlBLCseqsMQ3MXGi7hpfaYp9gRade01x78dCvNIhgpnU6JX6qpprVxPnSauFQcy0h65GfmcBV31HKAcTqyrPjqJJdZ3vN8M9+Zur9kBpMDIkBuKRqymyDNQQb3qpdDdesh2F8GpDKTlqWxd4GMUVUZJ0iE3lel+fcpXAT+SWeHD42Mx4PzmfQWYl2EbxcoQExv2QoRxkbfCzTtY17d+aPbFsPugS/a/YLThjggNrs07B7FUg02vdu9aECIkS53K1vYrmjFVfGSbBRqlia1dw0+6K5TITaZlkeQlrPqbOE6z0FQZG05Gw3BvY4UgX3LiKP0GKhFniQ4YSubDjBj6JjQaYUU41sdNx42sEHDVzIMHgi4WVly+dGHEgazhqy5dVon+Jn9qfMU+gXt4dgILoJB7mJkLcLYziWMBiOOEuL+mVDcE/h5aYVcNHw5U5ditaJ2G/ROBSH2swXWMNjzrGJZPJ9Vv2DgjzXDcGgDwoGBNLm5oG1EN6E/fdz7rMRAMFZR3AzFRAou0IV2CAOe09vz2JOQvMUIAWfP/fe9LTMTv2h9sTO0nfq9ASvDLNCph8r/3psou1nzT8d6xzCu6H6W8M2U25TIQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000068, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The workingset code flushes the stats in workingset_refault() to get accurate stats of the eviction memcg. In preparation for more scoped flushed and passing the eviction memcg to the flush call, move the call to workingset_test_recent() where we have a pointer to the eviction memcg. The flush call is sleepable, and cannot be made in an rcu read section. Hence, minimize the rcu read section by also moving it into workingset_test_recent(). Furthermore, instead of holding the rcu read lock throughout workingset_test_recent(), only hold it briefly to get a ref on the eviction memcg. This allows us to make the flush call after we get the eviction memcg. As for workingset_refault(), nothing else there appears to be protected by rcu. The memcg of the faulted folio (which is not necessarily the same as the eviction memcg) is protected by the folio lock, which is held from all callsites. Add a VM_BUG_ON() to make sure this doesn't change from under us. No functional change intended. Signed-off-by: Yosry Ahmed --- mm/workingset.c | 36 ++++++++++++++++++++++++------------ 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/mm/workingset.c b/mm/workingset.c index b192e44a0e7c..a573be6c59fd 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -425,8 +425,16 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset) struct pglist_data *pgdat; unsigned long eviction; - if (lru_gen_enabled()) - return lru_gen_test_recent(shadow, file, &eviction_lruvec, &eviction, workingset); + rcu_read_lock(); + + if (lru_gen_enabled()) { + bool recent = lru_gen_test_recent(shadow, file, + &eviction_lruvec, &eviction, workingset); + + rcu_read_unlock(); + return recent; + } + unpack_shadow(shadow, &memcgid, &pgdat, &eviction, workingset); eviction <<= bucket_order; @@ -448,8 +456,16 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset) * configurations instead. */ eviction_memcg = mem_cgroup_from_id(memcgid); - if (!mem_cgroup_disabled() && !eviction_memcg) + if (!mem_cgroup_disabled() && + (!eviction_memcg || !mem_cgroup_tryget(eviction_memcg))) { + rcu_read_unlock(); return false; + } + + rcu_read_unlock(); + + /* Flush stats (and potentially sleep) outside the RCU read section */ + mem_cgroup_flush_stats_ratelimited(); eviction_lruvec = mem_cgroup_lruvec(eviction_memcg, pgdat); refault = atomic_long_read(&eviction_lruvec->nonresident_age); @@ -493,6 +509,7 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset) } } + mem_cgroup_put(eviction_memcg); return refault_distance <= workingset_size; } @@ -519,19 +536,16 @@ void workingset_refault(struct folio *folio, void *shadow) return; } - /* Flush stats (and potentially sleep) before holding RCU read lock */ - mem_cgroup_flush_stats_ratelimited(); - - rcu_read_lock(); - /* * The activation decision for this folio is made at the level * where the eviction occurred, as that is where the LRU order * during folio reclaim is being determined. * * However, the cgroup that will own the folio is the one that - * is actually experiencing the refault event. + * is actually experiencing the refault event. Make sure the folio is + * locked to guarantee folio_memcg() stability throughout. */ + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); nr = folio_nr_pages(folio); memcg = folio_memcg(folio); pgdat = folio_pgdat(folio); @@ -540,7 +554,7 @@ void workingset_refault(struct folio *folio, void *shadow) mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); if (!workingset_test_recent(shadow, file, &workingset)) - goto out; + return; folio_set_active(folio); workingset_age_nonresident(lruvec, nr); @@ -556,8 +570,6 @@ void workingset_refault(struct folio *folio, void *shadow) lru_note_cost_refault(folio); mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + file, nr); } -out: - rcu_read_unlock(); } /**