From patchwork Wed Nov 29 03:21:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13472160 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7756AC4167B for ; Wed, 29 Nov 2023 03:22:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A1866B039A; Tue, 28 Nov 2023 22:22:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 02B7B6B039B; Tue, 28 Nov 2023 22:22:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE3A56B039C; Tue, 28 Nov 2023 22:22:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id CB0176B039A for ; Tue, 28 Nov 2023 22:22:10 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id A0321803EC for ; Wed, 29 Nov 2023 03:22:10 +0000 (UTC) X-FDA: 81509543220.25.E708DA7 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf22.hostedemail.com (Postfix) with ESMTP id C161FC000E for ; Wed, 29 Nov 2023 03:22:08 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=TWfSuGDn; spf=pass (imf22.hostedemail.com: domain of 3YK5mZQoKCIkB154Bnuzrqt11tyr.p1zyv07A-zzx8npx.14t@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3YK5mZQoKCIkB154Bnuzrqt11tyr.p1zyv07A-zzx8npx.14t@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701228128; a=rsa-sha256; cv=none; b=lx33ZHwztdsyBKSFOXgbLcyDxifL74Azok2zTiMlFr8mur/yo0rTtRhVwGnHUE0x8Fqrra WKBRPOFMf/jvnrAQwuRtOij1jcSl0fUB7qekXf/jSfNHdQw4/z09F3tBPpzMMhgLy5F/A0 fjjdGF+Nwcbb5BzjGH6iCycUF+4WsXA= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=TWfSuGDn; spf=pass (imf22.hostedemail.com: domain of 3YK5mZQoKCIkB154Bnuzrqt11tyr.p1zyv07A-zzx8npx.14t@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3YK5mZQoKCIkB154Bnuzrqt11tyr.p1zyv07A-zzx8npx.14t@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701228128; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GMYQlWUldBV9LWasymxz6cRCo6mSsi6IzDXKk+8BatU=; b=3SXSuumxoZQbeM3HCtlKLhtzC1S0NjVE5ClhNfxzgIJJ9OuojQL/0NfNOCONlNnvDQnVgk WZgab4ZmTpLkpO/NbYBgBCCIefh5PMD6QrC/DAWfnZqFMYxU3yl12Opa5937MrIrjp5PlN alwRtgLdHGoIuiDz4uUaA/zbGQKDtBs= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5ca61d84dc3so82006097b3.0 for ; Tue, 28 Nov 2023 19:22:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701228128; x=1701832928; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GMYQlWUldBV9LWasymxz6cRCo6mSsi6IzDXKk+8BatU=; b=TWfSuGDnIudKGEo7ES/f3xTNHZBfrdmPbmcvvg5LlEzgecuVAjraGlfh9QVh70QuxQ 1ZjvWo7QC0T+FjqSu1nqCmrTvmaQ67KSN9j21vn9C+Q26k09bmrexcmzjatQwWMVTrw9 VRnHuNTl2JAOTrpn/YH6vTRLlDRrSwJhhUJE5HTdLtzjbT1yCXPySKya9ZDA/GIRwHaY jGJ2PZHWUgDPRRborFcbpGnwg8kk02SSfQAV40Hud3hdUTipq5RpzEppPISgorLgDEWA AZrE1uCS077fWeWANBk5BJP8huEX2r1y1kQdwBwJFIFcY5x+/RfTa3PBDOi2GspKarrK U8bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701228128; x=1701832928; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GMYQlWUldBV9LWasymxz6cRCo6mSsi6IzDXKk+8BatU=; b=jS80VDX+T3vDvwdlo9e5ajAeaXcTzSNp7yJN7xeR/6pThbFPLLDhhxaJ3ldrJvcVMe BTu+gT6rzWhJoh4w5GFHhRzf+q6KIYy6uA4krMmnPsvv8PvcZTgoO9BzhsX9w9AWAy9+ n/0zykAi4bp7w5YaFl7hLYg1yAP7AOgkSiQTBDbR5gF5kxBy2IW5gWg4ah9dZVkWPZ6X wHc9b3uSvJhWKs+HyiQRHJAZg2kddx41DsZiT1DagpSyJyXgoo3cnC4YBpYzB5H8G0mb xbiEmC3zkNCqkAcvU+zdVLD9u1oE6GqC8WKIVJTvC6Xuz8YPhWMplsbzbHfLY/2++hax xbVg== X-Gm-Message-State: AOJu0YyzwMi5nfSubHXVIL1y7r7csqoBD+rks3A2YOD6kQ5Ih4ziJLBC Y+OpPc0hSuPKhTpkQafXBzti+kAjxW2DOaeW X-Google-Smtp-Source: AGHT+IF6lUJvoUePL6awLPUorDhmrelziQ66K/nBRxKZkO4u8ApkLFgb5tzVr7krTn6SjMgmlcBKg7J4Ix+mgnzs X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:690c:f84:b0:5ca:ad72:2d78 with SMTP id df4-20020a05690c0f8400b005caad722d78mr634136ywb.8.1701228128017; Tue, 28 Nov 2023 19:22:08 -0800 (PST) Date: Wed, 29 Nov 2023 03:21:52 +0000 In-Reply-To: <20231129032154.3710765-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20231129032154.3710765-1-yosryahmed@google.com> X-Mailer: git-send-email 2.43.0.rc1.413.gea7ed67945-goog Message-ID: <20231129032154.3710765-5-yosryahmed@google.com> Subject: [mm-unstable v4 4/5] mm: workingset: move the stats flush into workingset_test_recent() From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Ivan Babrou , Tejun Heo , " =?utf-8?q?Michal_Koutn=C3=BD?= " , Waiman Long , kernel-team@cloudflare.com, Wei Xu , Greg Thelen , Domenico Cerasuolo , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: C161FC000E X-Stat-Signature: y9rhh4issawb11sfdghwf4r7yoy64peh X-Rspam-User: X-HE-Tag: 1701228128-570355 X-HE-Meta: U2FsdGVkX1/fJ4Qb3EDZjgxmOpQVjfG9vSzS70SgLC/2LhVpRMcsqjizUqpm9hLZ0ujqTxhy5W9phscbThsHdUhnPdbct9LKUfoAIZu1l6IKFCNBZj0DPEeVEvqMEJL5+WGJcHrZGIBHb70/dHmDK7jZmw39Y6QazVc1bRoMN7GQfa91NbQ3+91NNQz7KAf1WD0/nrOgr7BoLpc2APGLM6HFQRmn1vziJ3+XTPPSMqQVFZtoJtfK8Sd4fuCao3e+uovjrn3yWWkXzY2+mcKhCqWM8k/amUFQR5TrmUAha3ppx4WWymOSyV5EgwbZe26pBq3c+wDmNsU6h6bYhOBVgnRsu3ouyv0BRDELJ/1DU7RKDZg4PKRES8iPN1ZYO21JoBOz+jitEM1h4sn4fvU6e4hktvROw1ql1H1ckldgPMobh/Qjte//LGuh8x0dWcT+qp/6jWD7UPnrYtl8upqlMaikXD2NsCoS9w9jwFO/wOqrYV0o5TKjSlKYvaO+ZhxRncJygQz/vqCwOBU0w1k4ueDOo0OUYzNBsPIwUeA12qLSM4BA8UkolXKg4+aWnH9N1rIsp8/0mvPeFCQVgVqk9Otw4lVnyYQmi+noIEjEA1acHsyyHcTomriwZSBN/Qy7co8Y/0C/LCuBgx+x5Z5ilNDUg1Zop1kDylyJ0OB4xAThxSEnlQfwceflpveV744z2pa+b6OTykcGVamavI50sPyl8jFXP9evI/ccKskvuCYpmDrl+nxGpMsAwWrKI9dGjBHFUB0KYZsxx5asiPrRcXcauJKhgRSAX6jyS9sprJyW/2zBbmnrzMBFtZB56jzSyHtLme5WEbCOByDR66PAODopeCVdyaxaa6PWLAGhYke+WAZTfShyI1H4YDWa0c5Ye2ovPGYEiHR8vgpKaeW0EQmLjqM6u57T/b4C9XlOJBkYhCFCE2SEqeJZODTTxMNmW+TS/FjOZXZtL9Cirdy 5ibrcQ8w YxG0g6rk4O/cubSebRdAU8xsnWmSk1WE0mlA/it7G1Fbpvrr+6B9fLVUqxiSntID7t2LRvQqojYuBWtpp5iWDh098vOOCSzX1boXA0xekPZbgyk+S5P3aIH0VRqONuvzgCLsWf2lZtmYW/ukmGMeP92oyHjiM+rsfL3j3JSZYdOXn8m476/5dr669dysLvHf57acNMuzZB7+vcKL/AGaBxt6R1d6crB64eg7/Z3kIAkCsAQAL8E5y1JrAHBgM39HW6rHJfj5Q/BGeqBolZCuqTSVUyuoKKEYrMEYelStTj0zF63t7YT/UQc0R4+LFJ6nGo+JPMYZBSYcxAXjoKCxCM1kfq9/OyLEcnF/2M2/s0tVdMRKGLjQ40Q98QVPcMdYG0w0fbJQYEExVs9/cu3UozTVUdWqsV4biRcjdA0hfCtPFvdyNomGYONzyiW5I4Y5MPIz8atpbXinmrwtTmhvUIfaBcIDdwXC9LW+wx/wfSWXVfjv7x8H0StRrXuAHBTMuvDGxTZWgy4PNJFqSdcA0MuyR+oXkQxcoQEb4gpsIxDx1YfvodLMPWfVkQQqmbrEoJbdbUe01vXDTjzMLl4VFI/cG1FVczyveYsCSxdkWJYZhqmlor9Izunqx4gBRtYoBGr6r4wtWaqxBd8IsV4PPA02axqVuE4rOBJCNlI2j6sYL52vSpEJFujPhOx25e/RxW7J5yhLF9+oJYaDWzuHMpgfb3Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.002704, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The workingset code flushes the stats in workingset_refault() to get accurate stats of the eviction memcg. In preparation for more scoped flushed and passing the eviction memcg to the flush call, move the call to workingset_test_recent() where we have a pointer to the eviction memcg. The flush call is sleepable, and cannot be made in an rcu read section. Hence, minimize the rcu read section by also moving it into workingset_test_recent(). Furthermore, instead of holding the rcu read lock throughout workingset_test_recent(), only hold it briefly to get a ref on the eviction memcg. This allows us to make the flush call after we get the eviction memcg. As for workingset_refault(), nothing else there appears to be protected by rcu. The memcg of the faulted folio (which is not necessarily the same as the eviction memcg) is protected by the folio lock, which is held from all callsites. Add a VM_BUG_ON() to make sure this doesn't change from under us. No functional change intended. Signed-off-by: Yosry Ahmed Tested-by: Domenico Cerasuolo Acked-by: Shakeel Butt --- mm/workingset.c | 36 ++++++++++++++++++++++++------------ 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/mm/workingset.c b/mm/workingset.c index c17d45c6f29b0..dce41577a49d2 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -425,8 +425,16 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset) struct pglist_data *pgdat; unsigned long eviction; - if (lru_gen_enabled()) - return lru_gen_test_recent(shadow, file, &eviction_lruvec, &eviction, workingset); + rcu_read_lock(); + + if (lru_gen_enabled()) { + bool recent = lru_gen_test_recent(shadow, file, + &eviction_lruvec, &eviction, workingset); + + rcu_read_unlock(); + return recent; + } + unpack_shadow(shadow, &memcgid, &pgdat, &eviction, workingset); eviction <<= bucket_order; @@ -448,8 +456,16 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset) * configurations instead. */ eviction_memcg = mem_cgroup_from_id(memcgid); - if (!mem_cgroup_disabled() && !eviction_memcg) + if (!mem_cgroup_disabled() && + (!eviction_memcg || !mem_cgroup_tryget(eviction_memcg))) { + rcu_read_unlock(); return false; + } + + rcu_read_unlock(); + + /* Flush stats (and potentially sleep) outside the RCU read section */ + mem_cgroup_flush_stats_ratelimited(); eviction_lruvec = mem_cgroup_lruvec(eviction_memcg, pgdat); refault = atomic_long_read(&eviction_lruvec->nonresident_age); @@ -493,6 +509,7 @@ bool workingset_test_recent(void *shadow, bool file, bool *workingset) } } + mem_cgroup_put(eviction_memcg); return refault_distance <= workingset_size; } @@ -519,19 +536,16 @@ void workingset_refault(struct folio *folio, void *shadow) return; } - /* Flush stats (and potentially sleep) before holding RCU read lock */ - mem_cgroup_flush_stats_ratelimited(); - - rcu_read_lock(); - /* * The activation decision for this folio is made at the level * where the eviction occurred, as that is where the LRU order * during folio reclaim is being determined. * * However, the cgroup that will own the folio is the one that - * is actually experiencing the refault event. + * is actually experiencing the refault event. Make sure the folio is + * locked to guarantee folio_memcg() stability throughout. */ + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); nr = folio_nr_pages(folio); memcg = folio_memcg(folio); pgdat = folio_pgdat(folio); @@ -540,7 +554,7 @@ void workingset_refault(struct folio *folio, void *shadow) mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); if (!workingset_test_recent(shadow, file, &workingset)) - goto out; + return; folio_set_active(folio); workingset_age_nonresident(lruvec, nr); @@ -556,8 +570,6 @@ void workingset_refault(struct folio *folio, void *shadow) lru_note_cost_refault(folio); mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + file, nr); } -out: - rcu_read_unlock(); } /**