From patchwork Thu Mar 30 19:17:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13194751 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73053C77B6E for ; Thu, 30 Mar 2023 19:18:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0EE12900003; Thu, 30 Mar 2023 15:18:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0770A6B0078; Thu, 30 Mar 2023 15:18:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E3320900003; Thu, 30 Mar 2023 15:18:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D480C6B0074 for ; Thu, 30 Mar 2023 15:18:09 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id AB05E140F75 for ; Thu, 30 Mar 2023 19:18:09 +0000 (UTC) X-FDA: 80626525098.14.C2F442C Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf15.hostedemail.com (Postfix) with ESMTP id 7775DA0016 for ; Thu, 30 Mar 2023 19:18:06 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=tY0E3oyb; spf=pass (imf15.hostedemail.com: domain of 3beAlZAoKCOwmcgfmOVaSRUccUZS.QcaZWbil-aaYjOQY.cfU@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3beAlZAoKCOwmcgfmOVaSRUccUZS.QcaZWbil-aaYjOQY.cfU@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680203886; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ECd9n7cK9XN/T+IJ39MyPXAm65DzgHuyZCFnDGQgZIE=; b=t0CUaTfO/2nb6E2qYVKNzEzhQgl/A3ISjMcddWXt2Mhsuu5jx0/DpX2BjLnp+d/D8BU8QA uQcXFlyYMF0whHHtgFjro9t0psZ59FoaGNsUTbgqy+4xhlDaIPbM732/COQIz9XezPkdA+ P56zfSju5Jqc/DAvp0vGHYU+G7Dr4cM= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=tY0E3oyb; spf=pass (imf15.hostedemail.com: domain of 3beAlZAoKCOwmcgfmOVaSRUccUZS.QcaZWbil-aaYjOQY.cfU@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3beAlZAoKCOwmcgfmOVaSRUccUZS.QcaZWbil-aaYjOQY.cfU@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680203886; a=rsa-sha256; cv=none; b=EuqNWQYC8DJQyh84xUoqvxfadouiMgfamwSH5ZZhPp5Tztry1rWR86pDInkju0wVUhwz6X Nszh0aSLoX5JwIgHgm2zxF3MGXMuqbV3sKBwdurstPtlIo9stWZHGHOemJmomcdgaa9A3t L3Z+LFYq0S3zZgkYea1C8FBBHelFGC8= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5458dde029bso199618027b3.13 for ; Thu, 30 Mar 2023 12:18:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680203885; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ECd9n7cK9XN/T+IJ39MyPXAm65DzgHuyZCFnDGQgZIE=; b=tY0E3oybgVZIuKKOOKBykm8/TKbf2scexLJVT3DkzJcaMo8pgAVQKd31VkYqY1QVPJ mHo+yh2Z2e/UWhIAS6NyKjgzsMFfM3Q7bRmRT9xetX5hDAZEg0lz49jvXxYxOEOA0GHz 2iOR7gvTKjdYv9QT9UofKbbkFs9vSWVQ0TKS3904q473sNNcjSfvya1m1B2ab+CL/f+M 4qtYTT2pypw3Co5Bobg3nKNc5an7wdNYyXFt6rny9eUWrmoVflUhZJQGBV9r+u9oNXbf thlMA7X4kUIdycQp/zBF4S4OoaK6pqFmZjXA4rakf85jUKtTz9rXXIDv36itoKwLwzmv vt/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680203885; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ECd9n7cK9XN/T+IJ39MyPXAm65DzgHuyZCFnDGQgZIE=; b=Eqwg2QQON2zQDkRB8VfQrXvDCuOU5O0bKgQEn/t9dQzxSAuiP/Kx4/gWxUj/VcHXgH jVAxPn/ZAaCMq/AryMARDQt2vmaXddvGL3nr74LbqH1zzl0Yx+mE1sL/dG/bUIPW1jnP WRHr+qElAYetPha5pXZyirt0plThVhsyWT9CRPcxlPSbS41/RW+WMCjnPkc+IHnbaUxS MmDUQ7zIkAnB4g4Q3fH2tC+050WaA08V096o/t8OuEb3K1V+DFtlsr67XnoAc4wtjBig HcklgrriWQbZwKX12UwQcOipsQXIIzzCfbmCbNAgF2khW5KDF5rL/cX7W4gFknipfXmC Mrpw== X-Gm-Message-State: AAQBX9cnsb6GBjvpLxEkv2kbsSC1rCyGJgteFKMoIgBA6qfmtvXo5D4r s6GgpvYh0bhrxKfciNI0RLyz39mJl/3KTz2J X-Google-Smtp-Source: AKy350aEWgmUGoq47vtgVPSLObUE1OHXFIc1JYGvZjZHSdRsfuGe4z8jTEoHLVbNJzm7tfhcDcAxLa1st977h/QM X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a25:d1d8:0:b0:b6d:fc53:c5c0 with SMTP id i207-20020a25d1d8000000b00b6dfc53c5c0mr4650822ybg.1.1680203885608; Thu, 30 Mar 2023 12:18:05 -0700 (PDT) Date: Thu, 30 Mar 2023 19:17:54 +0000 In-Reply-To: <20230330191801.1967435-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230330191801.1967435-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230330191801.1967435-2-yosryahmed@google.com> Subject: [PATCH v3 1/8] cgroup: rename cgroup_rstat_flush_"irqsafe" to "atomic" From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Stat-Signature: azdf9n3myzfaquedr519fp593h73ii1z X-Rspam-User: X-Rspamd-Queue-Id: 7775DA0016 X-Rspamd-Server: rspam06 X-HE-Tag: 1680203886-228548 X-HE-Meta: U2FsdGVkX1/g3VujgbgELHasFD1wZny0kyOf2/CSFbHSI38ykTF6LeIkEOmSjvsBHXRU6F+4DkYBQsx1ivDIH4aHiGHKA1VUIuFmcMGqWNwYAjd4hH6QDh/mgw1FE488hMIakyUpaAp7UJvN5EN9NUTjAhZFpIZ7Dv/Yu6suDrn0DGQXV1SWlVBRB7f7cvQQuf8QmHe+EXWJRR+38e6jN/y5nWaiAu0gwu/Jw4uhqrVaMdH3S+4qXtdYy/KVlq3594OV5UaE24didNCYfEyRX7wtbmIlDdEo9EkKyH+a1LNrkwGlC5ZClQ2baw244iIG0G9HjbI2LRrEui8qTtb3rKchiQ+TqmWQWIwn5WgIugJBRtLB3wBCiDW+D/pxLl0cKsTSEfNs98ovRb17KxBVyK59aLICdwXcWutLjYmRCBU0Eim25kJfwAfragKWrUHZNc3OEz+lAO3QhoKFJdoMPBSFkBKw6Ob6GI4lZLyW14X8o4oIAV/7jBpQMXFTcLFeJIXjQvfWmpJEQPQ/50sHVxeBDf/wDvmvIsCPHMVG9f9uW/mzWrzSfE8ISbOSkFuwxE6sevcBxu1Q+Ch6ay3alsrXXqUCsXu2baZ1PtBfFZeh69oY0W+Vs+LHSvg0HZYIyVfV8QH/BZ5S0NT+Zq/eeCtsyAs12ilskLRtcCriWMLDuj91+pGqpNHyYxjNtLI25P8E/EUHyG5sWz8GWpDZF3hb5xUTdhNk3VvQ3dpEoQzCc4nNu5ZY+Mx9KWfP1s1ljc1YqCs3yx3LsVH+BtmZJHBqP4oKz8CKOy4wpKszJSf+gNiOxLEyjqIaM7XEUvVcJnfVcsGASgwEb8vEZGTl3T5dHoPfF12afqq/7M+QJY9A5kBRqywmdQPWPFZeqID/PLukFF9PBBbCPBx8gTsxX4Lwl+znjOfYofwNi4VQQFHVNleWbDEGoQcowc+rG0CGv/vvZe3CuCYV0X6Sh73 ODa5TAY+ wRHrSiH+ygF5+HuxGMLIlaSDjDhm23dbDu/+x2QDtV3UQTXmQ3MWRahIp8R8L+9kSQxO16gRbxpqq/mqBb320agLswY5WJafVuL4EDLFkVaW0Rn0Fe7n3zmQ+28UcnczaDIRjemT4LKFh58rJYRCBqGZHPiyLJxonpA1ShOScH4Klz5AeXYj1kAfUUUDNvqXvFQdaWJdXQjjAOgdUgavwSTcFx+vbTSeJvDRdzP60l5AJI40eoss+1ztGaSC6VCDga4iFqyQlDq+sR3ztQoo/b/jNGDHYOihJ5bdaijoenrTckv6KDeP68cG2IjIduQ7XKf2veEH1ZV0paQHkQPqatXvFhX9dWgT3dzc3WLmf2eCKKVJf4vW4OBZbCsZB/gIkcXk/ygs5TxjG3HDHONrnPPGVyJEN2RWWgMg/zxo7Tz+Bf2hrBV8/oc/5WMd2952EHF3TjjynLuFvk3ENTpaRquN/YfZK+NnnOMgFZDebY1iqL3fVGsQafo9GIe8YbMRRPxxWeFlsgRWl0HmJosDlIHxAnn7DDFhMHLB5PlHq6vjD74cQSgjt1lDP29oA85xa5hAQfj0/xc4l0AXib1U/Yz+1fBGgQ1qHrtrzIAr9q7pdcNe8VCezDZKee+apiYfMPoDkpZbcRGe7LOCfBSzhWWBhdQVb4/YOYD+iwpIKWJKFx3xSuJisgBTtRNGZ0gxbSrq7e/oHDx6Y09YBwpZcpGNIUfRqYiBpEGg0+oQZAbRZ66ki68+FHioVFsmI+oUtG5evuhrSdkNiEUo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: cgroup_rstat_flush_irqsafe() can be a confusing name. It may read as "irqs are disabled throughout", which is what the current implementation does (currently under discussion [1]), but is not the intention. The intention is that this function is safe to call from atomic contexts. Name it as such. Suggested-by: Johannes Weiner Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner --- include/linux/cgroup.h | 2 +- kernel/cgroup/rstat.c | 4 ++-- mm/memcontrol.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h index 3410aecffdb4..885f5395fcd0 100644 --- a/include/linux/cgroup.h +++ b/include/linux/cgroup.h @@ -692,7 +692,7 @@ static inline void cgroup_path_from_kernfs_id(u64 id, char *buf, size_t buflen) */ void cgroup_rstat_updated(struct cgroup *cgrp, int cpu); void cgroup_rstat_flush(struct cgroup *cgrp); -void cgroup_rstat_flush_irqsafe(struct cgroup *cgrp); +void cgroup_rstat_flush_atomic(struct cgroup *cgrp); void cgroup_rstat_flush_hold(struct cgroup *cgrp); void cgroup_rstat_flush_release(void); diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 831f1f472bb8..d3252b0416b6 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -241,12 +241,12 @@ __bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp) } /** - * cgroup_rstat_flush_irqsafe - irqsafe version of cgroup_rstat_flush() + * cgroup_rstat_flush_atomic- atomic version of cgroup_rstat_flush() * @cgrp: target cgroup * * This function can be called from any context. */ -void cgroup_rstat_flush_irqsafe(struct cgroup *cgrp) +void cgroup_rstat_flush_atomic(struct cgroup *cgrp) { unsigned long flags; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 5abffe6f8389..0205e58ea430 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -642,7 +642,7 @@ static void __mem_cgroup_flush_stats(void) return; flush_next_time = jiffies_64 + 2*FLUSH_TIME; - cgroup_rstat_flush_irqsafe(root_mem_cgroup->css.cgroup); + cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup); atomic_set(&stats_flush_threshold, 0); spin_unlock_irqrestore(&stats_flush_lock, flag); } From patchwork Thu Mar 30 19:17:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13194752 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7259C77B71 for ; Thu, 30 Mar 2023 19:18:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 31F656B0074; Thu, 30 Mar 2023 15:18:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 273FA900004; Thu, 30 Mar 2023 15:18:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0ED466B007B; Thu, 30 Mar 2023 15:18:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id F0BAE6B0074 for ; Thu, 30 Mar 2023 15:18:10 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 7A7F4AC957 for ; Thu, 30 Mar 2023 19:18:10 +0000 (UTC) X-FDA: 80626525140.04.58E2655 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf10.hostedemail.com (Postfix) with ESMTP id 92C0EC000E for ; Thu, 30 Mar 2023 19:18:08 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=QmYQM+j1; spf=pass (imf10.hostedemail.com: domain of 3b-AlZAoKCO4oeihoQXcUTWeeWbU.SecbYdkn-ccalQSa.ehW@flex--yosryahmed.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3b-AlZAoKCO4oeihoQXcUTWeeWbU.SecbYdkn-ccalQSa.ehW@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680203888; a=rsa-sha256; cv=none; b=KLyIJnv87/3+CxrjtToaxG9DoPxqs4LDi/XvLY3Zsp3Dp+LhuogDaDBXWRf4cbBGt0Mixi OPMNa0R79hsLFPJaL7n6A5XUW91apVSIOPXE4y4zo6rp8kgyuUSECv5quoOaqHZ8dNi/mo o38bhC+SaPQZdDY1+63SJmLXqzH9qj4= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=QmYQM+j1; spf=pass (imf10.hostedemail.com: domain of 3b-AlZAoKCO4oeihoQXcUTWeeWbU.SecbYdkn-ccalQSa.ehW@flex--yosryahmed.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3b-AlZAoKCO4oeihoQXcUTWeeWbU.SecbYdkn-ccalQSa.ehW@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680203888; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LXlZsArwPDY7OuU2BxzeoXG1UXSoglEAfUQ5EzzMXG4=; b=wa9kXfwIQc9FnTprlPYPlfyyTPq4FAWGqbviLztbYKvFtqVciTNjfBYfZA55rBshbOc9GB MaUwc+3IagWP4ahL337RSmbL2GvsAQd1xbcxNHJo5vbSLvNbpqeeK51Y0gstIEhocymHBe IbKQVMVR0Auw68Dm4NVp4aY1Ujd+dbs= Received: by mail-pl1-f201.google.com with SMTP id z14-20020a170903018e00b001a1a2bee58dso11594082plg.3 for ; Thu, 30 Mar 2023 12:18:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680203887; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=LXlZsArwPDY7OuU2BxzeoXG1UXSoglEAfUQ5EzzMXG4=; b=QmYQM+j13cdbjBiZy3Y+dzkJUgGOVHTdr9lvDbSE859fYaJtq+8zmH02RDaKbdpvap CqejHKuoW/4IBz5mIHOf6Y8AHa6GMb/hCZ7zMABh1AePaLkgpcjAyzU7xGg8rQCDHyuT xd/GhSCAHuv9H/zwirCUR2wyGX+B6mkWwB9WVRwD19bK9qyHzg7+GCXy42vkNiRXXZdr kN4hOyhVE2VLeXDBEBRHB8fSA0RE0BkgU+jq32E4Jq+3xIhGwrpX/kGZ92PdxYBPFxyB 8OgXFYA1yB1rgGnRYxUqB9CL2/ZWFsevIT5f1UxupCOTzAHu4BNZEW/BqlkhNKD16qxi CoQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680203887; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=LXlZsArwPDY7OuU2BxzeoXG1UXSoglEAfUQ5EzzMXG4=; b=UgMOcJBTavaK42IzACau6kbGeqWq+Lh/TZeK+SLPdvh518cY2rz44V2i8hCQrD1PcI 3TiCNrUjCw0eDv6/RtJBN5MefqOZCTXwamoMgLDw89UAhSFY3+nXAbQlBumZ3cajYS5D mQaZMPzRABanfsPnBobEJ0xkKy9sZCNgvj5BAsJ7RMkKwPiKSGffWDiFiA8lpBMX9uz1 lIotCTEX5dukTC1RNPbP07Xu9vV3ONsm9vAkn4SEdqXXPqanJF50zCp/x3lyuBR9euQ9 WmDbNdlEuqoFALQ/7BljaispSvkDu+dtNqFi7tooMKQWPQwWW3KR4DnX74NO09Canzq6 wPiA== X-Gm-Message-State: AAQBX9dA1H9xxfryIf0rupzvZsi3w60aIDWPyjmURMuau3LNXOy8TG1m 8E8MDX45J/e3c/m/HmLp6MFVwu7j0P1Fd52w X-Google-Smtp-Source: AKy350acDpkb2oKWRJtDPrvhJr6G7w3aWGLRFehCl/mgbnA6nqxhni7pGTt+2Eb5S7HVMulqi96jSmdvt00VgGJt X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:90a:ec11:b0:237:9cbe:22ad with SMTP id l17-20020a17090aec1100b002379cbe22admr7641616pjy.5.1680203887352; Thu, 30 Mar 2023 12:18:07 -0700 (PDT) Date: Thu, 30 Mar 2023 19:17:55 +0000 In-Reply-To: <20230330191801.1967435-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230330191801.1967435-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230330191801.1967435-3-yosryahmed@google.com> Subject: [PATCH v3 2/8] memcg: rename mem_cgroup_flush_stats_"delayed" to "ratelimited" From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed , Michal Hocko X-Rspam-User: X-Rspamd-Queue-Id: 92C0EC000E X-Rspamd-Server: rspam01 X-Stat-Signature: 854euski8y4h7w1m9abe47zm9u8r7jdm X-HE-Tag: 1680203888-253626 X-HE-Meta: U2FsdGVkX1/vwKFF1V5Ke330zjdtN0xK2XsAkskz4rSQ3O2ayZNIJgGVBagMBjijPd9eCpZ1b/3fmVXPQxk/SK4Q+ZoVQytFlj3Oh84ekPDiHwWHYIaJd+TDfE4IRosJ3+fxmDuny2shcHGbxErZPKPLol7gw+PT6oP1a4G7XXRpgMM+PfrUJU9+c91YFDXign1vrRt/jAcoDYAR7tmM44fEG0AwkdMkUYDnXrTu4u0YPiTZ6Ma5nEw+anKUsAhRu9qfmA+r7bZ1c5R89bNtRar7Vj0OjhjxNPMK0DhW9cV8odP9iN+YS79LWgisdlN5n2YPMnx6PICCZuYPgdG1rmv+d+AdQgiX5FL8+rQ8hX/4T7NfjyfcEzBwd+LsnXTDhAl1bei5ERtu/YnDm1Iuw4Jyxj7YPuBW2YKP8SX8uyzn+mHwEoYbAO5Rl5sVpQ7fz7MeAkF5OAQ+7HbBL3UT/7CTX8a9dS1WVYmz/PyZupzIWDRTfNVntrCwX+LQFTqyhaOjnzgT07eaUgpTo8UxtspBCiiW+nYkqgOOlhKldA1gy93LiDd/YBzT4qq/Z41v3tb1K/jm3jEc6eAAULsKGLKtgLMbbMdMCLIELr8m4ttxvR+T1ioe6Hwnb/OdNL8pp3hMbD7lw1f5DyLzP89Z3zxU7FMToKq6ARPCqAP0aKRZdK6VpRdNTW1rutTwKXGaUdHjjqJn1S35tY845Lc+z4rqdKMCrzK7d46LXDKFjxGk+V2aadqLX9cMHwfcWe3deb5V+DgHUeF/05oFD9HSxUGOFyjxPHdtcc/74CPxO0ko/DR+nckBxWzAAA3x23NBysy9rRbbHOcI7gRFoGExdUADF/fYPrn905X3xXQFRRQd7f47wpBlOLG8+DZbW4ZyYHGRI9iNBt0So9qlRoBelynf2GJgwpbXSJv4xaSaMj1Ye3ol3wxiJAd9LqeMmOL8aHcC9QSoNMur+fWkFH8 iZtt78Il PmdKbNGCZaRl9tyx2qEXAa3SLiVtGvRG1fJwWVHG6XrXxhxMhp5DtoaNmVVkDVieorE1xK/+hf8HNQcwBGVrUP75d3kzajKi4Lky9KB185EmXMr7+Lx7xjWOIEvwtUFnY9QVVT6oD4s1CE+py3W/Y5choPoson2ytzK8ND4eLcM5MkT44FC7WErPf+5BLDGhoZHeh1B1b3MswMcySm5T7lBAbUsFjR5Rbv9lZ+I/neaOfM9bjSzJnVNScxb1Ezbt0LeCxvnzgSBborOv3RnD1B4HO5yjaPmyoBokiwdOHINRKLa0CYZGuONeSxRaZKzzvkBGr3NeDnrgHwZ3gd+yjdrEPF7ZaWcYovT26ehc3MdDaZVRMq7vSoP4tZlt31R4TBZlHgJPtKObq+rf+kcIZYzMIshDOH6R6e9hXUidp4UjG/iOc0LB27N4Phu/uvVLSTwxdyH9yHM0PcuQRyIHTWkf1fvO+F8aM+Qj6PaXQWJoVbUy+bJgs6YbjgXBGu4/Z5lVxu2m4abtxT7MXe/eED3MK/t2C/8gk/oTaHGILP0SFaDwxsDzOmLwYwslkTgPoRPkC+Rop39nK1EygctEtie+d2l3FTDWhmhIDhZakUxkGsOrCNKUOw7sHLvjXkqJPgNZvp149R4HE1lcV3j+iAZFfTtfBeNPum966gjCfzHDSfUrVWhLAsnB82rWoReRpSKpFeiKjDTtBNETxrHmHj4BI8IPpXKf3GYlGP1L0GmixpRCgxH89aga+s7lsr/GzSixz5got2+PDISCXfpM2RkTqzn24zjpGQIuQH9nzXSUIi7x8sCJ597aya8LGM2pbGTv7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: mem_cgroup_flush_stats_delayed() suggests his is using a delayed_work, but this is actually sometimes flushing directly from the callsite. What it's doing is ratelimited calls. A better name would be mem_cgroup_flush_stats_ratelimited(). Suggested-by: Johannes Weiner Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Michal Hocko --- include/linux/memcontrol.h | 4 ++-- mm/memcontrol.c | 2 +- mm/workingset.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b6eda2ab205d..ac3f3b3a45e2 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1037,7 +1037,7 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, } void mem_cgroup_flush_stats(void); -void mem_cgroup_flush_stats_delayed(void); +void mem_cgroup_flush_stats_ratelimited(void); void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val); @@ -1535,7 +1535,7 @@ static inline void mem_cgroup_flush_stats(void) { } -static inline void mem_cgroup_flush_stats_delayed(void) +static inline void mem_cgroup_flush_stats_ratelimited(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 0205e58ea430..c3b6aae78901 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -653,7 +653,7 @@ void mem_cgroup_flush_stats(void) __mem_cgroup_flush_stats(); } -void mem_cgroup_flush_stats_delayed(void) +void mem_cgroup_flush_stats_ratelimited(void) { if (time_after64(jiffies_64, flush_next_time)) mem_cgroup_flush_stats(); diff --git a/mm/workingset.c b/mm/workingset.c index 00c6f4d9d9be..af862c6738c3 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -462,7 +462,7 @@ void workingset_refault(struct folio *folio, void *shadow) mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); - mem_cgroup_flush_stats_delayed(); + mem_cgroup_flush_stats_ratelimited(); /* * Compare the distance to the existing workingset size. We * don't activate pages that couldn't stay resident even if From patchwork Thu Mar 30 19:17:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13194753 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86B48C761A6 for ; Thu, 30 Mar 2023 19:18:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3D55C900005; Thu, 30 Mar 2023 15:18:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 35FB1900004; Thu, 30 Mar 2023 15:18:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13F01900005; Thu, 30 Mar 2023 15:18:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E36B7900004 for ; Thu, 30 Mar 2023 15:18:11 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id BCAB941103 for ; Thu, 30 Mar 2023 19:18:11 +0000 (UTC) X-FDA: 80626525182.16.9B348E3 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf13.hostedemail.com (Postfix) with ESMTP id 03DF420013 for ; Thu, 30 Mar 2023 19:18:09 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Rb2fsB8a; spf=pass (imf13.hostedemail.com: domain of 3ceAlZAoKCPAqgkjqSZeWVYggYdW.Ugedafmp-eecnSUc.gjY@flex--yosryahmed.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3ceAlZAoKCPAqgkjqSZeWVYggYdW.Ugedafmp-eecnSUc.gjY@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680203890; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aVus0GFA9nzxOZ5lNIH1Nx8kHcLEa/aLkX98gl9HKOI=; b=NSkahq5EeEW0kiLf6xABY4/lvGeqZTYxwmE5s1boWUYxuQ1YZ0zHRyA/7ohilErFwZvOw1 fmSu2oB0D5KmcvAwJlsTKZVwlA56Aw9sRbff8Gh/cilOOw7RiHd+7+ldcH+9wRnb+v6hcL R67EWdmMpYx3WKlGotWUPI/vlglNH0M= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Rb2fsB8a; spf=pass (imf13.hostedemail.com: domain of 3ceAlZAoKCPAqgkjqSZeWVYggYdW.Ugedafmp-eecnSUc.gjY@flex--yosryahmed.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3ceAlZAoKCPAqgkjqSZeWVYggYdW.Ugedafmp-eecnSUc.gjY@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680203890; a=rsa-sha256; cv=none; b=qeeHWosvM4Kw5XbDO2sogXT2C+sx4qavE89Ij3nRmJdTQmE5V+xRo1VDzQM5UdOlNCRIP4 Csqjc4S9PXE6OzyY8oP5pFP4z0fqDvvplMTR1UBbR9eJxXXghXa45JJ1yjTHik5aENrgFK kdtKswh9DK8etotP/57Ns0T7N3xlvc0= Received: by mail-yb1-f202.google.com with SMTP id y144-20020a253296000000b00b69ce0e6f2dso19770063yby.18 for ; Thu, 30 Mar 2023 12:18:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680203889; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=aVus0GFA9nzxOZ5lNIH1Nx8kHcLEa/aLkX98gl9HKOI=; b=Rb2fsB8aWLuAh/6UfhLacuvSldBOfVahXww4db+vgAZH6d8hHK5iS/Pb3yBuHu8zXu 8OHs3K+JgNd5BWKL8HmUqSNmeBMWOUfM93BgScQIJGiFyUTuS/OhqqsgB8B/zESgECWW id0La031n/vRfUHiqMH5ptV4ZUHLtuC1awDopegwUjfP8gYPSzkD9GboR6Sv+BT5F0il 4cP0p/kpaTUv9ECKBi7X6nDyHboM9b8Z+lLZ6iyDmkLsMIt+cWstg1DQo4e/ZnrIN56I iRBi8QgWvfbGxJTMXxv2zZuY7CwzGwzqEOEYe4XvyV9MRzo4MibTgBPKLkzWKa272DY9 IgGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680203889; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=aVus0GFA9nzxOZ5lNIH1Nx8kHcLEa/aLkX98gl9HKOI=; b=lOtma1IMzvJ534QWEu1QcZtBzPMu+4wMi249ol7zhXZUDa6JZ8zUgd4xVYhWMve32Z Kpcm1C9gLtzXK76EuiIjRyq9t1P35AZ+wPmMACATuGblB4lZVN/LDHKq9O0Wvq97XRQy q11dF2qwSTy14GZtTIOt+dSOf5FM7DUe0QUtKqU0fUF+iTUveJ2r1nJcUBbTuw+2vOfm YscMVZ2ALx/jwfVkWhpLIXDr+h72NbTmSGuSIa9OhdSYG0xXVIojbpI0hEsC5Ovf6/0w KvM/c2LFYgRv5j6n31ZkKHUrdl5XraliiZol4nsoTs1Q6+mjg6QzkQ9zFkUR5zegS5L9 Qmig== X-Gm-Message-State: AAQBX9c7mG/PDa7VIkj5c/c+OM+DGJ92k8Hqsr4/c8oSds9ejxv6Vqrh yJscRmd2Vvl3SV3EoNqqehRyoBtNJm6MF+FR X-Google-Smtp-Source: AKy350az4CClmAfjDaxG6/AcT2u/aalMEwRTuNYj9qh25Mudi/3feszrufxHB4k9is1ESXC81UQXzGZVI+EC5ExV X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:10c3:b0:b79:4826:e8e3 with SMTP id w3-20020a05690210c300b00b794826e8e3mr10930499ybu.1.1680203889210; Thu, 30 Mar 2023 12:18:09 -0700 (PDT) Date: Thu, 30 Mar 2023 19:17:56 +0000 In-Reply-To: <20230330191801.1967435-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230330191801.1967435-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230330191801.1967435-4-yosryahmed@google.com> Subject: [PATCH v3 3/8] memcg: do not flush stats in irq context From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed , Michal Hocko X-Rspamd-Queue-Id: 03DF420013 X-Stat-Signature: 1yrzanjzyhqcsupf5pj8nq9x94phqpeq X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1680203889-9800 X-HE-Meta: U2FsdGVkX1/OjXkKg0R1orkShi+A19tencPmKNIx5w65m7UtVw6+1kO3LCigovNsV+ZNRBJtfj2tKkQp+4wxfaclh105zLf4i3wVsalwJOYZ194qMidHQt4zLmLctMKCXW66dtckUOJ9uZGxr8q8YlCupgoQ7/vLmW4SjUJp6ItBK9SftpGEDqK0uekA/N5hN9HDFtganfryrvUL99SA+VVtkihvp1+bwd0o19ucNfyuWfatM6vw03QFkrOhcnYWjXDjWAUq+2u3TLmkx97cT33Se8QWNgEe3tCSA8yNz6jIDyLO5Z/5v9VjFSBKPdCTo4Sv0kXcQRF/4ihnmhe3t64v8MBtLkIOXj1WOIVA7iZBY4ukU7qRrhfdAV6OBDqbi3wRtf8uNBsOSUh9/INhWUNj9mqWzGRLNXyNCz0QTiEllIhsWfxcaCwhcr+f5kte+mPIMNBwQnkPLgELTh6Dw6I82LoEiV1zcLG2ANuRhT7yQkq3okqY5sVGgkAqWUNrMa+9qixk2aZTXfxXCK7sEMG+LyDTXiwQGFgH7j8alsJoUzsjSgh99UTLcGNFi5Qql0Ngug8zPyHgRfmDOGBE9oIQ1h5uEAMRFBM2jRJ5bRc9V9XkokO5NRkxBnrFwF56tHOxWdwPxYAeBBPVieEAHZLSq8VQbJIK3LQvLYO17D3i+YVpbDwufAhG+QGv9y2kXURdoRBu0nOuEk52KgDOQ4Yx3vUt7aovCAbtOrDWtg8wdb3Ief+Oq9NFcBpNurP4DResXIuuGIWgiBjk6DqsC1qiNgATDCdtp42QuOIXZckkD8ylF0yIZhZivHAsrgYEJca5vtblarL2wVhz4wkLmgJ4jBJFtCxhW6tVYMtl4KZ5hKw6ApdCKhn8HI/sxCC73sxPD0ByXRsh0AqW5T82sFXOVwPmPBTtMCfznprzalMvDXKeFJX0reMlzssHSO8fqQvhrg6SDDMTg8SX6+z sh1TJDOE nYfKlstogeHumNDvTiDYvMUtXdhtjBHH1W/vaB+tk/1swwkY0PRX7q7mi2AfBz5EqRUiqTKpkK5mzNrEDYlVHT+mYUChfCCJSWFpglBUbZ6owhbvP0ZSqKWhQmz0tvw6OoJ9B8JAU0BQiLHiu3A5eB8XthZJpFlgbU95Gcnn9y4A7IFglP9XzM8jedJO7KdGA0vYvoJsnC0TRuiGhDJ2LB7XgVt9AAq7gpA8qZLuV8Ryo4H2cH78UylLauAT2fwpYcykP24vDoQULL3WPj/o3A+WSyV/3q0tZJaU1keyJRjBCa2idhNk2zIvGLpBJhoG63aMnlpyGGhXmVVKwqRIcD/FruXQQwV9s6mVCWybdAtDNlDJJFySvID1Sspdftqxnw1ISmLAlyWOGPljiCpcPN7J3ekcPumRoznb+knGv6Xesm7hkycLC82ZgNwm+sY+urpHqOp/mYJ1WcFJ1G+2CGOZflFE26hWYRNC0HsSQBb4muzA5onHZ/8YJpJBciWnNYA8e5LaxcO0tdmYJgNWJDxkkJqp6w7yVC0ZcuMBQ8oHnpoxGe7eLQc68CDp6AIeb+D2dHNIYwHlEM6l9OPL3AkLfjLdLa+F7XgKSVsmFsQdkJHonJKC6TJvdTqYCPcAX2c6VSIOjiaWz/RTKyifmqYedcGB7T4RnWTxiLRfmBUPGG4UtMArNwPq+X+gaKB47DxwfjcMFxpImc3Xw2fEVz4HGTA6ilPJr8kjxFo6nUgkzrXY+6QWOeXVoHpXJwk07f7mvNcdIiC2OekXWwjdWFX77Jw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, the only context in which we can invoke an rstat flush from irq context is through mem_cgroup_usage() on the root memcg when called from memcg_check_events(). An rstat flush is an expensive operation that should not be done in irq context, so do not flush stats and use the stale stats in this case. Arguably, usage threshold events are not reliable on the root memcg anyway since its usage is ill-defined. Suggested-by: Johannes Weiner Suggested-by: Shakeel Butt Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Michal Hocko --- mm/memcontrol.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c3b6aae78901..ff39f78f962e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3669,7 +3669,21 @@ static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) unsigned long val; if (mem_cgroup_is_root(memcg)) { - mem_cgroup_flush_stats(); + /* + * We can reach here from irq context through: + * uncharge_batch() + * |--memcg_check_events() + * |--mem_cgroup_threshold() + * |--__mem_cgroup_threshold() + * |--mem_cgroup_usage + * + * rstat flushing is an expensive operation that should not be + * done from irq context; use stale stats in this case. + * Arguably, usage threshold events are not reliable on the root + * memcg anyway since its usage is ill-defined. + */ + if (in_task()) + mem_cgroup_flush_stats(); val = memcg_page_state(memcg, NR_FILE_PAGES) + memcg_page_state(memcg, NR_ANON_MAPPED); if (swap) From patchwork Thu Mar 30 19:17:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13194754 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69F50C77B6D for ; Thu, 30 Mar 2023 19:18:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 339C2900004; Thu, 30 Mar 2023 15:18:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2E973280001; Thu, 30 Mar 2023 15:18:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0ED32900006; Thu, 30 Mar 2023 15:18:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E9CB8900004 for ; Thu, 30 Mar 2023 15:18:13 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id C2227AC957 for ; Thu, 30 Mar 2023 19:18:13 +0000 (UTC) X-FDA: 80626525266.28.61E00B3 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf12.hostedemail.com (Postfix) with ESMTP id EFDAB40023 for ; Thu, 30 Mar 2023 19:18:11 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=X2RFp1pZ; spf=pass (imf12.hostedemail.com: domain of 3cuAlZAoKCPErhlkrTafXWZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--yosryahmed.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3cuAlZAoKCPErhlkrTafXWZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680203892; a=rsa-sha256; cv=none; b=QCUR8nHQQOs7dJIBRTH2QIsnN4uAm87NERso1v0/iI1FgBwkRYNzexwEEQH6/SJBNpdm/I lCig0ONPe3lYxhMeiECiJbFORdX2Svt85vD/1aduJwSQKNb+HP+kEllC1xIy2Q1KT3kcgS DxDG2X5Nd9z40nJufN2O0P5CB04n4uM= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=X2RFp1pZ; spf=pass (imf12.hostedemail.com: domain of 3cuAlZAoKCPErhlkrTafXWZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--yosryahmed.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3cuAlZAoKCPErhlkrTafXWZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680203892; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4nnUf1Bi8zb8HMhH9DtFsVEWdtAjMz82Sf34ZimV0fg=; b=SRnktgfIhbD9mKRWK+QI4wxKb5oqISfXOG+1wlzGpYY9g5O0srB+QbkXnpHT9cQQ0RJMqb lVySssD5all2iPP8e3OV8FPmGSDOhAJmpY0AxZ/iYFUtVFU0RHgc5wc2hLDSAGw6nR95uJ CuvHN1LwCpX4NXHMBt5jKkGVsHzsVrg= Received: by mail-pl1-f201.google.com with SMTP id l1-20020a170903244100b001a0468b4afcso11726622pls.12 for ; Thu, 30 Mar 2023 12:18:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680203891; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4nnUf1Bi8zb8HMhH9DtFsVEWdtAjMz82Sf34ZimV0fg=; b=X2RFp1pZWOYnRsIh7IWjsVsM+KEFc9WJSHxF/6WWA0h+C4aKaj5fmmxIF9JIbSgACd wSFJlPVZHSShfit9sIv2gEMkHiLDYI1aj0ZjNxN+KDFEUuJ5hrS0dAiMoZtgG8tE01Qw Z6IuI1YwDe0K3jJKl09PQ+88WTXCXmLj3mXjZyc3NpqNvz62t2kYidmfKl22OQzA3sQQ EAjXhE1Jh+vHizPfZUZ3LQ+IzSkqjVL6en02vxZp22lS6vBDlsf8gGPBHsHSCXP5Ac15 JluiS/D70cGu14FmTesFiFf/4cN9gJ7flhGTivB535PauxJnWfGkMyHN0wJ36oB1Mtk7 YlVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680203891; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4nnUf1Bi8zb8HMhH9DtFsVEWdtAjMz82Sf34ZimV0fg=; b=PFBfqTO5YeTSEPgcUGX5+BHZxZniz7uQb0o05SdLWK7Txw3lsFaq9osqYhAesQ4ooJ /FasqmzcDGkiLxrCuomb3Vwb7uVS9CytwwCtcufyupBUl7riVps2o1V6n7sMpmBUUwmZ FWiDup+5i/czRna+nI7cN6ycHfFWSdIM8xX8s0gH748qDUWvS1fvRiJHQejkDMK6uv2b WMIU06M3H9Cndhzu1IqoITrDiIHHsa0lRyvgjkNeAhKuF0V8fWnNt7s5Tq5FWZQgqLyh +yKVX5KQilAW67KOqrq/4Xjb6Fk/527fUMGKEcNLfUVfZWhgFJIe7xFD8FJ+49nSte4D a+Fg== X-Gm-Message-State: AAQBX9d3JPe7/TDpklJp1Jw23TqaWw4fxPRmuuy9A8fjcdNqmBloXS7q Lfp1EeW98gAdThTJbqqYf9WbbjmZ/Yn2lN+J X-Google-Smtp-Source: AKy350ZmAXD9xXrcG5sJYCYyMiAcFBYVD5KvS63XKanxHd2/adSILGt8yBb9ajmjtAKFUTblNAGcd8BbimVIOaMZ X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6a00:2286:b0:627:e6d5:ba2d with SMTP id f6-20020a056a00228600b00627e6d5ba2dmr13290706pfe.6.1680203890966; Thu, 30 Mar 2023 12:18:10 -0700 (PDT) Date: Thu, 30 Mar 2023 19:17:57 +0000 In-Reply-To: <20230330191801.1967435-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230330191801.1967435-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230330191801.1967435-5-yosryahmed@google.com> Subject: [PATCH v3 4/8] memcg: replace stats_flush_lock with an atomic From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed , Michal Hocko X-Rspam-User: X-Rspamd-Queue-Id: EFDAB40023 X-Rspamd-Server: rspam01 X-Stat-Signature: fkqcdey1y1t3ueo3ppemp4zf6ox8ttzt X-HE-Tag: 1680203891-72137 X-HE-Meta: U2FsdGVkX18VU1r1IRSA6C5iHqqMJX0V7J8whux0a6ZJ+hIr9HJoQIiWUgM3hNBBvJXUhsWH10QPCdYH5aoJNaUdGxDjWO1IoyG5V9iiQ5/Ly+gh6P0tCQZq/NE42D2xszhpV0DgbypkffTaqpfMaQdNcpjjrZOA1G9yPoQXO2/RHQ0A/M1yhexugQLq2ycz3ml7EdLnQHCKvAi43DoxBqVlD8kqWOhPmAF//Slp22lyttzh390qtFjOCoWPB6ZpsI0FXAdoUyGlP1vOc8lLofq0oEdyQR2j8p92hahmcqqeX+mhqXNI9k886PaVXzvM2z3b+46XCNUsOrSahzZhhFzieuPutXDaqFLV1zKmvvKpK6TAu+XkUOh3yYf55XUbXN0kZVK6GlhDBU54iEsy00TPV+MlmjNEmOAqHIJm3ZTfmQDT7IcRXqBHyhiDt6x38V/9HzxBBHwy4hfzfT+d8s7Un/3Ni79rFE2G3YNPeqILzmDDLFPBuRXLCoPKmcbaRqjpHEIo5kuKWlnNlWvKvlXTyVm2W45b/jJEjEcMhoYwook8ZeTr3uCzgM+4Kkw7CNNjzKfdkoh4Gvl0xACMPYY8nhNviBANOgqK15VjerqtjT4XN9fpFYM62Db7n3431YUo0l0k2owjXluLU5dNgwstF937ciSc43UuiyeSZP9rV0vzEhw2dJitor2EEWujnuiDCjf/3hivoeMK/fxqPQYZxyhtpK+W8AgkUpjLWSfbk+x/YEtxvqPJ4Xpz5REkYzUkxiyiB+myhg+Y5HstLonnWhEpJUAW5OfG8sQBHPmkleV9BxxTfBesVYGD5rZRkDMBOnMt0gwJK9BSE3WMRYXC4u/jYLrj7AcX6XzFh0kN0OqzdksPuadIMuHm/gJszQmpNxYttAMeenQfyRN9w3ZV2vmfZd8ssi3KoaPe6cAINFH68FmHVL9WfQoLblVS1b/47kVmp8mxy32oVEp ouwuIAp9 AOGa3dYoSHh/JmFNqGbEJ6VwiPdN+R7U7Qp06Xs2XceR+M5anh7ZlbiNh26/qetw3qtI+QuJMSdfZDtexj20lrSpDAZVD7EugKc/NjKZhLP7MMJt6CaOjadz6f57wGbANyH3gjoJoKWkR5J4Vd1gQ81+asw0vubimwVohLDsOcl9Q3LqBUvRgg/zPgooXYZYzEcxq0GTH9AXaUbKSKRAVwVgYO6JYiwaa+iFKnZ4RuncrBITjP/1ZuiDxLP5U5AVEtPIFqHmGUwD4jTql9O/+w+6dqS9m31u77CygsKpKedCKWDUv1qF4nTA62SLckL11tRbwNJCNXUzXztm00dgOJht3LJ24QUmdV2uj2heAjGSBAxHq0HX9K4Z5r5QgxUQkp8mpYizu/7wbv3Xcxt5f0n8b/f+vizFrXNoth1+vyCjTOJznK8UhIxpBYdxWWeoosRs9n6FGCq7sTquNeLA4tdS3apOC9j+1OApSTYT9kECFOf0NYzk5BoE8KEjslORNq0UMuehKqqNVIPzYbXNQXdilfHSJ/Y5tQqi+PrGLiz711fedEUBjLeXit+XJ7W3sZh7ArGUYWXttgvq2tBMVlENOBARz3HiMJeHhj8rc1iKvw5s6sALWJTY67AXUTyGFdwOXJA3GHKY3YmgqeD1MupsP6+T/Cg6CFcblhQ/BeFJeOA7mbLwLEOaT7UELpHsR30+RyzXLeD3tPqTVYF0n50OCEG7H29Abx8IqQpPDbGXiP0RcemBTmXKGYtSOntrRnIq6S45qea6YPy3yUYxwTHVwautjzlOcUGoGDbg8tZJiX2MiXW4pvfC0UqS0rwDxV7kV9hLQzhJy0cINhCMgrCjUGP3kx4NMHe4f+qHTcaCojhAdeqdqsb/RuPcpHlBsXs08 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: As Johannes notes in [1], stats_flush_lock is currently used to: (a) Protect updated to stats_flush_threshold. (b) Protect updates to flush_next_time. (c) Serializes calls to cgroup_rstat_flush() based on those ratelimits. However: 1. stats_flush_threshold is already an atomic 2. flush_next_time is not atomic. The writer is locked, but the reader is lockless. If the reader races with a flush, you could see this: if (time_after(jiffies, flush_next_time)) spin_trylock() flush_next_time = now + delay flush() spin_unlock() spin_trylock() flush_next_time = now + delay flush() spin_unlock() which means we already can get flushes at a higher frequency than FLUSH_TIME during races. But it isn't really a problem. The reader could also see garbled partial updates if the compiler decides to split the write, so it needs at least READ_ONCE and WRITE_ONCE protection. 3. Serializing cgroup_rstat_flush() calls against the ratelimit factors is currently broken because of the race in 2. But the race is actually harmless, all we might get is the occasional earlier flush. If there is no delta, the flush won't do much. And if there is, the flush is justified. So the lock can be removed all together. However, the lock also served the purpose of preventing a thundering herd problem for concurrent flushers, see [2]. Use an atomic instead to serve the purpose of unifying concurrent flushers. [1]https://lore.kernel.org/lkml/20230323172732.GE739026@cmpxchg.org/ [2]https://lore.kernel.org/lkml/20210716212137.1391164-2-shakeelb@google.com/ Signed-off-by: Yosry Ahmed Acked-by: Johannes Weiner Acked-by: Shakeel Butt Acked-by: Michal Hocko --- mm/memcontrol.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index ff39f78f962e..65750f8b8259 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -585,8 +585,8 @@ mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz) */ static void flush_memcg_stats_dwork(struct work_struct *w); static DECLARE_DEFERRABLE_WORK(stats_flush_dwork, flush_memcg_stats_dwork); -static DEFINE_SPINLOCK(stats_flush_lock); static DEFINE_PER_CPU(unsigned int, stats_updates); +static atomic_t stats_flush_ongoing = ATOMIC_INIT(0); static atomic_t stats_flush_threshold = ATOMIC_INIT(0); static u64 flush_next_time; @@ -636,15 +636,19 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) static void __mem_cgroup_flush_stats(void) { - unsigned long flag; - - if (!spin_trylock_irqsave(&stats_flush_lock, flag)) + /* + * We always flush the entire tree, so concurrent flushers can just + * skip. This avoids a thundering herd problem on the rstat global lock + * from memcg flushers (e.g. reclaim, refault, etc). + */ + if (atomic_read(&stats_flush_ongoing) || + atomic_xchg(&stats_flush_ongoing, 1)) return; - flush_next_time = jiffies_64 + 2*FLUSH_TIME; + WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME); cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup); atomic_set(&stats_flush_threshold, 0); - spin_unlock_irqrestore(&stats_flush_lock, flag); + atomic_set(&stats_flush_ongoing, 0); } void mem_cgroup_flush_stats(void) @@ -655,7 +659,7 @@ void mem_cgroup_flush_stats(void) void mem_cgroup_flush_stats_ratelimited(void) { - if (time_after64(jiffies_64, flush_next_time)) + if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) mem_cgroup_flush_stats(); } From patchwork Thu Mar 30 19:17:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13194755 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3543CC77B62 for ; Thu, 30 Mar 2023 19:18:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D4BD6280002; Thu, 30 Mar 2023 15:18:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C8690280001; Thu, 30 Mar 2023 15:18:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD94A280002; Thu, 30 Mar 2023 15:18:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 85EC8280001 for ; Thu, 30 Mar 2023 15:18:15 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 49BCB81172 for ; Thu, 30 Mar 2023 19:18:15 +0000 (UTC) X-FDA: 80626525350.12.E20BB27 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) by imf25.hostedemail.com (Postfix) with ESMTP id 871E5A0018 for ; Thu, 30 Mar 2023 19:18:13 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=nHw0e79h; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 3dOAlZAoKCPMtjnmtVchZYbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--yosryahmed.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3dOAlZAoKCPMtjnmtVchZYbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680203893; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4K13XuzFhLt0ZIgNSFirN/NTpinGGuhSsjDOtu88ZFQ=; b=MaijTmFqJlEm2cL+Di5RKFX2pOg7CwhT9reB0dRvMDpgapRfryhX7SAE85B0LVfxc8A+1X 1qpMwLctL7umR+Q74P3xZbXKAwqGUKh3jkVDx40RwlnA6Rd2WgL5rWz//KLl95wB2RD92E 7VE6U6/ryHzve2Y/iJ+71LBvcf57zyc= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=nHw0e79h; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 3dOAlZAoKCPMtjnmtVchZYbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--yosryahmed.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3dOAlZAoKCPMtjnmtVchZYbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680203893; a=rsa-sha256; cv=none; b=OtHcAgf+DxMcpMBKBAymtpdRCxNrfbZVndeOP7BXH9D3QqgdNIv4O1K1sz7b9PIaRytJTv TH0dj9b/resZx2SF4/DGCIt3L4e9Yhc4cbDf13SxdgGmLf7mcQNejDMATG3+KEPFHTieMH zoVZKPJ+uByKQ9MPnQ0DWDKCxmXXDqM= Received: by mail-pg1-f201.google.com with SMTP id 9-20020a630009000000b0051393797707so1577937pga.5 for ; Thu, 30 Mar 2023 12:18:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680203893; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4K13XuzFhLt0ZIgNSFirN/NTpinGGuhSsjDOtu88ZFQ=; b=nHw0e79hZFaw3nf4LyIXmmhTYfA1FbBbf2dimkl5At5Psg3CjvNgu6oG6xad78uoN/ S2j2UYV8EipalJrwt2zNvbI6g73PyAp4x9X0MELejGCTtg493QkbzRVplqF8mo+gd+ZT v2g6y/Uq/dc/IcDqjVuG6qTAXC3lltOSedgQENPqcdYx34VNoyr/anJbPmkUtw6k7KGZ kbHnvz3Vd5/Dm/bxM/NXhZiy/XES3eJpoxargQYJd3pDWMStv8K0/vXl6hkWv2MnHLzC WyPbqSXfAwOToFWCyu1C3jP4ndGUE9RCrti/toBLaYuvsaopTHlhvDYChxTi2n0kxppd O1Nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680203893; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4K13XuzFhLt0ZIgNSFirN/NTpinGGuhSsjDOtu88ZFQ=; b=HSu62o+BDC0W8mY+QTegfHWpBAITYrdfKmDNphFVXvAaTfOXA1ui8CssCoZGJvIBJ2 BePjNeRQqPiKkXPWd89HQlq2yiohxC6vxF4NOTsk9rhi8SgJsCIRUuAAvHwpMEdiQM+0 ILAfaLcn29Tj5YPS9etvRH1L0RCUeZavT02YZ+CjMQxu8fWEc7iv+DSP1wXtRaarxXxW CenQg4lgV51dCyj36SvP94gHJH9QV4pPjbvdZyHhZ22KqD3tfa+WWQuMeSSjW6CUWuEX 9sdXo1/uiG7LG3aSSVSOHhREB44XZiDc7+LO+/QyEqIMF9XKHABBPlUtCOKHHpsZWvhY 18Rg== X-Gm-Message-State: AAQBX9cARrEPodUZ616uqNFHU/T2y4iWEKpmpomvGcG7iJuDe42Uoo/4 B0gFIvYeSqpNmuhKBPEXlD8l8Z1aByFCN5L5 X-Google-Smtp-Source: AKy350YSzgQBtWYiTKsbkJ1oQdlD62/Xn6W7mjRdu2TN+RxVPyaHwSjuEqzrgKulPryFVMSvh1+rAuiwX+ctKyIl X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a63:1914:0:b0:50b:de95:af77 with SMTP id z20-20020a631914000000b0050bde95af77mr2125406pgl.1.1680203892860; Thu, 30 Mar 2023 12:18:12 -0700 (PDT) Date: Thu, 30 Mar 2023 19:17:58 +0000 In-Reply-To: <20230330191801.1967435-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230330191801.1967435-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230330191801.1967435-6-yosryahmed@google.com> Subject: [PATCH v3 5/8] memcg: sleep during flushing stats in safe contexts From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed , Michal Hocko X-Rspamd-Queue-Id: 871E5A0018 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: ffc6nn4mz5dohjokf89jxaukjmxdem4x X-HE-Tag: 1680203893-181529 X-HE-Meta: U2FsdGVkX18VsG1vJIEmlOniY0YK0H/f6d58Aw3Dl/PpscCMkHYxsg03BT/0bJZbHeMqViooHuLvlK3+OLMDViFF6IuMjJQn2EvJ6qEZNBqGWVQzNFQjWHgTsKS2y1wyd7vGYK9DtPSVYHCA9gacqysgLyOvov435eWrGfds4o+fnEc+5yEYK3a2u5k5d57zI4Ix2yiw2bVD5tYgyN62nuD/IvM587zLnLDLlPjoyjrMTFnwnmUPRtv5RQ+5SJHK4MIewKkETj2ALAFmadn+Wt/DrORbi7zc+VJr1inqijEghN1Q0u7Irge56xjcf0IfYpWiidFdPjuW2pXz9wSAwqlS9R/6X8Qxlp9KEbjwlndsDzmojuVzU8eyKlhak083SGDnpu+TJwl+6JyU9zA7C1pgtr+bn5CrEnnJWGz8ZQ0Aif9lpM30swsEw9Py0q2CPbSiJCIqAg8XiHFWig7QdtdtD9ejg0nyQ3UXd3MBEhwVpzAjOglyvB3bMtH/WaV+p6aHFgMotaA5P1BQqtCSD21Rh/M6YdhUznAeBGb9THMC7Y+i9bCwNxlRBdHWL0+wTO8v4qhQoaOe1a6AbO+g7dyeK81MU9rJBk5jxamBJmtsDiTevTkq0SFzd08Anndl7KcBN+ROCbhexQRf7S2pEf9HHU5nCXW2+kx61ip568MT2bgZeIz8+UB8Kenc+BXTGb2quctLkQR7BCrr5EJTPuXOCpz7i64a2l0v8d9dP/vEn1U8+aa/FahSH1gmAVX4ZmpoiFnftH0s4xONR2/w+W6LI3KHGbjzsTuKM3a498KtGIZnF558EORRBZfKyBKOej+PF2jDde77kvjQfONyHQKAr8Rx+4iF1n/1V74eVuPXqhz+7dkCmS+VlyZFXX5wt6CdIfjhS9+hMFJPbtKwIBWOxcs7a2ZBu70jUVpgWNBV6/mEq1e/uL6gAqXyErnJVBH8/6ikM/gR8JSKh2k XPYnox3H 48iLneYrsNTOEiFixZneKm6Ceca4vQnAlf+NIrH7CgK8PMkvkwfrrc1koP84H0HowlKW4WqkcKG2GR295oR9J6223+XQack3ZtR/ms17Cd04vuCYmDEt7WNG1iupb1POyVpNYuIRLHj0A0Fm8HiY7A7dD6d92T9RA2btOGHTcpoYyp5Q38x6ztH8lMuRnYPFQi653Zic7uGx7bc3FjUSqf773CnhNI0xqQUME1YmcyJ1+nl0gZxQw92CCOKsjHEHfGE+tJ9gokNU9+AYDqVvPc5i8LeVMepWWEBDi+AAkBhPep7Sdbg2g5GhD54V5o7Ym6V0oFbXiMM3ZOrTGCTFP6vySMkbeSEtUOHCZTGmuw/wTiRJgc2WJ6hU/+vCgZEBYJaf8L6/Gb31yJNdIYYfdf99KuX3etB3oe89Tv4Z1FdqB73pYIAlRw5hFamPr6cGWRoebi399LTA2QAaqUxQ/GkqLujL89UDN3hVXtTSzTG+GreGKAbTmUOFcI0RAjyF64rQbpQtPEpOs0hubYasjfa5/kOhTEd2C6BH+GF4Q7afMcrfbHDveQC6sw7USzgEG5zob3H3jDXmei26YZrKjhPVfI32UW//aP8dD1JkvqjqxFa7eENqrKdKE4/7fAIQPxkz4+7YLVcNNJ0dhUDNyAx/ZQxO32cfPpKFCiK2xgmxwQK8aFY/xuc10kjVttACByg9LT8cLDHYlXm18auefg0/5PA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, all contexts that flush memcg stats do so with sleeping not allowed. Some of these contexts are perfectly safe to sleep in, such as reading cgroup files from userspace or the background periodic flusher. Flushing is an expensive operation that scales with the number of cpus and the number of cgroups in the system, so avoid doing it atomically where possible. Refactor the code to make mem_cgroup_flush_stats() non-atomic (aka sleepable), and provide a separate atomic version. The atomic version is used in reclaim, refault, writeback, and in mem_cgroup_usage(). All other code paths are left to use the non-atomic version. This includes callbacks for userspace reads and the periodic flusher. Since refault is the only caller of mem_cgroup_flush_stats_ratelimited(), change it to mem_cgroup_flush_stats_atomic_ratelimited(). Reclaim and refault code paths are modified to do non-atomic flushing in separate later patches -- so it will eventually be changed back to mem_cgroup_flush_stats_ratelimited(). Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Michal Hocko Acked-by: Johannes Weiner Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 9 ++++++-- mm/memcontrol.c | 45 ++++++++++++++++++++++++++++++-------- mm/vmscan.c | 2 +- mm/workingset.c | 2 +- 4 files changed, 45 insertions(+), 13 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index ac3f3b3a45e2..b424ba3ebd09 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1037,7 +1037,8 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, } void mem_cgroup_flush_stats(void); -void mem_cgroup_flush_stats_ratelimited(void); +void mem_cgroup_flush_stats_atomic(void); +void mem_cgroup_flush_stats_atomic_ratelimited(void); void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val); @@ -1535,7 +1536,11 @@ static inline void mem_cgroup_flush_stats(void) { } -static inline void mem_cgroup_flush_stats_ratelimited(void) +static inline void mem_cgroup_flush_stats_atomic(void) +{ +} + +static inline void mem_cgroup_flush_stats_atomic_ratelimited(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 65750f8b8259..a2ce3aa10d94 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -634,7 +634,7 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) } } -static void __mem_cgroup_flush_stats(void) +static void do_flush_stats(bool atomic) { /* * We always flush the entire tree, so concurrent flushers can just @@ -646,26 +646,46 @@ static void __mem_cgroup_flush_stats(void) return; WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME); - cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup); + + if (atomic) + cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup); + else + cgroup_rstat_flush(root_mem_cgroup->css.cgroup); + atomic_set(&stats_flush_threshold, 0); atomic_set(&stats_flush_ongoing, 0); } +static bool should_flush_stats(void) +{ + return atomic_read(&stats_flush_threshold) > num_online_cpus(); +} + void mem_cgroup_flush_stats(void) { - if (atomic_read(&stats_flush_threshold) > num_online_cpus()) - __mem_cgroup_flush_stats(); + if (should_flush_stats()) + do_flush_stats(false); } -void mem_cgroup_flush_stats_ratelimited(void) +void mem_cgroup_flush_stats_atomic(void) +{ + if (should_flush_stats()) + do_flush_stats(true); +} + +void mem_cgroup_flush_stats_atomic_ratelimited(void) { if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats_atomic(); } static void flush_memcg_stats_dwork(struct work_struct *w) { - __mem_cgroup_flush_stats(); + /* + * Always flush here so that flushing in latency-sensitive paths is + * as cheap as possible. + */ + do_flush_stats(false); queue_delayed_work(system_unbound_wq, &stats_flush_dwork, FLUSH_TIME); } @@ -3685,9 +3705,12 @@ static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) * done from irq context; use stale stats in this case. * Arguably, usage threshold events are not reliable on the root * memcg anyway since its usage is ill-defined. + * + * Additionally, other call paths through memcg_check_events() + * disable irqs, so make sure we are flushing stats atomically. */ if (in_task()) - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats_atomic(); val = memcg_page_state(memcg, NR_FILE_PAGES) + memcg_page_state(memcg, NR_ANON_MAPPED); if (swap) @@ -4610,7 +4633,11 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages, struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css); struct mem_cgroup *parent; - mem_cgroup_flush_stats(); + /* + * wb_writeback() takes a spinlock and calls + * wb_over_bg_thresh()->mem_cgroup_wb_stats(). Do not sleep. + */ + mem_cgroup_flush_stats_atomic(); *pdirty = memcg_page_state(memcg, NR_FILE_DIRTY); *pwriteback = memcg_page_state(memcg, NR_WRITEBACK); diff --git a/mm/vmscan.c b/mm/vmscan.c index 9c1c5e8b24b8..a9511ccb936f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2845,7 +2845,7 @@ static void prepare_scan_count(pg_data_t *pgdat, struct scan_control *sc) * Flush the memory cgroup stats, so that we read accurate per-memcg * lruvec stats for heuristics. */ - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats_atomic(); /* * Determine the scan balance between anon and file LRUs. diff --git a/mm/workingset.c b/mm/workingset.c index af862c6738c3..dab0c362b9e3 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -462,7 +462,7 @@ void workingset_refault(struct folio *folio, void *shadow) mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); - mem_cgroup_flush_stats_ratelimited(); + mem_cgroup_flush_stats_atomic_ratelimited(); /* * Compare the distance to the existing workingset size. We * don't activate pages that couldn't stay resident even if From patchwork Thu Mar 30 19:17:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13194756 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 017EBC77B60 for ; Thu, 30 Mar 2023 19:18:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 471D9280003; Thu, 30 Mar 2023 15:18:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3FB9B280001; Thu, 30 Mar 2023 15:18:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1D8DB280003; Thu, 30 Mar 2023 15:18:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id EDE25280001 for ; Thu, 30 Mar 2023 15:18:17 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C08EE81169 for ; Thu, 30 Mar 2023 19:18:17 +0000 (UTC) X-FDA: 80626525434.18.D16CC02 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf21.hostedemail.com (Postfix) with ESMTP id EB98D1C000F for ; Thu, 30 Mar 2023 19:18:15 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=EVT1tk0h; spf=pass (imf21.hostedemail.com: domain of 3duAlZAoKCPUvlpovXejbadlldib.Zljifkru-jjhsXZh.lod@flex--yosryahmed.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3duAlZAoKCPUvlpovXejbadlldib.Zljifkru-jjhsXZh.lod@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680203896; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zjUVlGzaNGh8M5BDppplh7Z2E1z5drNgoNePIuD8ZZo=; b=lHSrkDEdcMBMgcu67V9SoOCMYJ+6mYdlALV4J/leHN+wj+JIQof38QzAAkPwUK3u0VgWkc BGwzIowoO9wxXxGIxTxTpXu2cji5KRle67o8rv1ykwpeUzBD4IUo7+4Cg5Wcn7fqjt0CVg JbTkzo8AD4BUQXrjmNfkf8q3id+W0+4= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=EVT1tk0h; spf=pass (imf21.hostedemail.com: domain of 3duAlZAoKCPUvlpovXejbadlldib.Zljifkru-jjhsXZh.lod@flex--yosryahmed.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3duAlZAoKCPUvlpovXejbadlldib.Zljifkru-jjhsXZh.lod@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680203896; a=rsa-sha256; cv=none; b=SbU3yDIEQ+AL4WsGUCi6x2e4Vf+MrAadrXEs2XbsLUpC9VpmkhMD9Br25cSMzIy4JsZuPz wrPMYWpm+/0ooFZorrcDelxwONEPa7/6on/yb/fSmXAGYAfU2hKqfJAkjRR30rqM4L1BKM 89VXEDBHt/x9o2jQbD8NDkOCbOiu3+4= Received: by mail-pf1-f202.google.com with SMTP id i192-20020a6287c9000000b0062a43acb7faso9111008pfe.8 for ; Thu, 30 Mar 2023 12:18:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680203895; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zjUVlGzaNGh8M5BDppplh7Z2E1z5drNgoNePIuD8ZZo=; b=EVT1tk0hlY3K/RTYvcQHMKPpwQPVXOnpdSgQM68BKQtZmfmON2APgqvMOvMMFyRoss Dmd2H4VUZwWGIfs6kzYLS/yY0LBdZ4oH1VkvSRWDGcSOJqUREz+4yAA6vO0jGdRbUllx xh0AWez/MaDxogJM7rwIORKSy2FNGdepATzkqraYhiO3RCB8HnWcqe3zSlq35IxkzxtC FV1PJ6neoxu760xZAlTGk8nkupYvKz5UcVjEtp57Ggrbl9Lds+WScYdzag5BG/1gW5Yv DUw8gcccaLuVJ49JXssrnmfFrYddyvv++tF0t2hffuUaTqszDGsJ7ZLQoChqmjJGkrqP YfTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680203895; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zjUVlGzaNGh8M5BDppplh7Z2E1z5drNgoNePIuD8ZZo=; b=zSMsbuVz9xiyetHcS2v+belY5+KGQALCCrBwGOnxHOld2Jst3VogPGjd4Zf5UYI6DM a7jWf/4V/rhnQj6AFtdHOqhZVuUhO532E3+wD1bwDXcVL6KsDagVo9zLKYjKhzoNsXZT 8kYBaGAVVJm1gHkjtGWvVl4seedXf3VBxgbKtdQhMVJlPg81bSXcWyv/NyVTcKlP4Cu9 v26ry2Fkx5CuapMtF01dTtIp0RNUbmOb8JgMYx1t5DMHHYhJXh99vy5yHpgsU/0mX30S 5VI5N9iQ5fuD+5nUnsFN0JB6tiQb5ihiQnbRT2bIGxy1Fry2HR53+yMOG/cHgRolwG2H HgSg== X-Gm-Message-State: AAQBX9dk/lCkiAQU1JvRKS/ySg2ZsqUmVnYcD9BYR93uQTqnhmWNeut4 ikVdlKTt9Wbuxx4NseN3FIB2AK98nb4e/t7h X-Google-Smtp-Source: AKy350YjU2Y2MgqNkqgZoIOPOW0+HOYv1MqQ0pV2FUPZVEGY798zCaQjIBhDtJyF0YZxiEsyw9QKlZMU8q/qm34J X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:902:6b44:b0:1a2:6e4d:7832 with SMTP id g4-20020a1709026b4400b001a26e4d7832mr2680127plt.1.1680203894775; Thu, 30 Mar 2023 12:18:14 -0700 (PDT) Date: Thu, 30 Mar 2023 19:17:59 +0000 In-Reply-To: <20230330191801.1967435-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230330191801.1967435-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230330191801.1967435-7-yosryahmed@google.com> Subject: [PATCH v3 6/8] workingset: memcg: sleep when flushing stats in workingset_refault() From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed , Michal Hocko X-Rspamd-Queue-Id: EB98D1C000F X-Stat-Signature: txuccs7586baj5r3d3kkisxtcnhcknzj X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1680203895-230380 X-HE-Meta: U2FsdGVkX18BWEgioZiUpfvp0h5PVD9rYWw139EdqGfTrr2UptFOAey0FSRM8ytOMyGjsj2e+0UBc8X3OU9HayMYNYJ7M0NEAet1PzWVJK2nuWw07eKG0EoiREh0g8PRbu1qlLltyoR0Zoi0IXSsT+dm9Xm8//3C4lyFMRwjE1xyXVg3BtVFvBE4I7gmEpCwtncZhbLoQT4X/hT9D6jqn1xd7YA2dk4Rj+HYprqIQ4hU3yzoQHQqMhfpcpohH22v7ja8iyohSn6j71jMni0efv2XRljEh99igMIIHWgD8sbyEgc34hMS2YePVbIECcBDScvL1L9hhvWETMrEckMTYxR9knhBNlLOFBJ0ARJyDu9U5iZyrUQ82Ce03KWRfctzW/lQ+kocyA3bVUzYzzMM1FoTDWo1ZSRjMPHvA2Ks8AhgyOJ5cD0ToVKh1Yvy1m49Ohcw2uPqfpK1Mwq/n7Xy1XO5NXKma1FBcId3ZEHsn7sXyGVnBfTJ9H2/uWGYAYrvIMaif6bW9cT404/VoC+cEyuz4P/79kX6Tr0hdgiXCp8ohl0H4YMtDtVdipS6zPsAuswI+Euy+X+9EDiRNE/FTDXZvlojKEq6ZaxsMMLWl5OaDdISeefHl/bovP8tCYOwGB3t2I3aJDhLPmwZoL6vUVxq62mT9tR+x2SdCUZ9qAD2jvw0qVLDuJxDvNfYWF1neB1oPS8risX6wsLPtYS7lrDirsGouQZdQT7O6fznXhdsFAFGIJWrnxvH4kAzA4Ra19tF8hTCKkYlg0jRa6HLW5OAlSvOKPhN1bMto4q3liJpq6TL5uhHPVlXTBBJR3F7IBlZeJdW1yKmtmIqD0Xw6pLvyWL3gEcYLpMsijqr38Zax/iAqOKN58R/javANqt0IwL75+qGj6HDahKoiJzlLTcHyLc0wmiVUJdsyqIUuzR7XFK3NQlqLF3gbvyw4VM/xV8hBBaqTJyP/5QXbHk tbA1IpEF bgwynuX8jGDTKBAyVSR1Y9QPhjmm7i+19q9/10Qbye4qvvqLgo6T1P/Qy0OyDCwU8bOjTK2NpMro7spgpldVjg8HgEiX1AESwBLtQs4MZzeZOZHqMgm0aF0H3OXznaoUZtY4TeXcaZM18OQApRd2aR+U3Lo6/zFJex4Sj6IVYTGTCMpo4SM5gfN3PkwCTl3LFWry4KuQtK8KQ7KHtFwJb1fzckS5wq255nPexEDoXVci1EHzjlTugmerx/nuGwWgOdZO/ysYTLJ3WrRFE6s3I8hLoTs0FRdkXcjekQDUPY8o80PdLZeOFxXfxUmLQeIwY8gYccn2wPTq7uc23v0foraI7vp04vZflgyj0QBECfbNPcnPvpCZ5SQUKPYPJdkF4agOZex0f7lBd5EtHPbahk4zSgw9EALVMQ64u0J7U9Hh7aA6YvAOHJolFGXX87/z0tPhSTCw8iikDyVPEgzg2IK5ldFDYQlPwC5kkO/KLGY/8cYcLf2FkSGpD/dz/Ge4nRNlBZPh129kY4UdB+UOhmT+4BXRbk33IUiwLWQa3VctG5m8rSQ1WdPeWzfha79t/osZtwdouRfLvZa6WqgyULFu6kd7hDcc6p+U41H/lcPPob4ZchAcif5WNzvcNv5XkgGHFzTHfucrVzmZFjxpisuKNVJsyNivQVZqsLQZ3YsPyMN91I2IJg0M8n3y6DTrDT1hNFG5yRAXHYZYKsS28Dfdl5bIfLX9/4QtRdP1hVfkyyPuIYmPRxDBqHHFYTsJFK2O6I2DzkaS4uVfHOqvde1/NnLUv6z77W0k8jKjD/naTJLs4cdw7704htOOKBuW+Q7Sr X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In workingset_refault(), we call mem_cgroup_flush_stats_atomic_ratelimited() to read accurate stats within an RCU read section and with sleeping disallowed. Move the call above the RCU read section to make it non-atomic. Flushing is an expensive operation that scales with the number of cpus and the number of cgroups in the system, so avoid doing it atomically where possible. Since workingset_refault() is the only caller of mem_cgroup_flush_stats_atomic_ratelimited(), just make it non-atomic, and rename it to mem_cgroup_flush_stats_ratelimited(). Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Michal Hocko --- include/linux/memcontrol.h | 4 ++-- mm/memcontrol.c | 4 ++-- mm/workingset.c | 5 +++-- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b424ba3ebd09..a4bc3910a2eb 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1038,7 +1038,7 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, void mem_cgroup_flush_stats(void); void mem_cgroup_flush_stats_atomic(void); -void mem_cgroup_flush_stats_atomic_ratelimited(void); +void mem_cgroup_flush_stats_ratelimited(void); void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val); @@ -1540,7 +1540,7 @@ static inline void mem_cgroup_flush_stats_atomic(void) { } -static inline void mem_cgroup_flush_stats_atomic_ratelimited(void) +static inline void mem_cgroup_flush_stats_ratelimited(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index a2ce3aa10d94..361c0bbf7283 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -673,10 +673,10 @@ void mem_cgroup_flush_stats_atomic(void) do_flush_stats(true); } -void mem_cgroup_flush_stats_atomic_ratelimited(void) +void mem_cgroup_flush_stats_ratelimited(void) { if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) - mem_cgroup_flush_stats_atomic(); + mem_cgroup_flush_stats(); } static void flush_memcg_stats_dwork(struct work_struct *w) diff --git a/mm/workingset.c b/mm/workingset.c index dab0c362b9e3..3025beee9b34 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -406,6 +406,9 @@ void workingset_refault(struct folio *folio, void *shadow) unpack_shadow(shadow, &memcgid, &pgdat, &eviction, &workingset); eviction <<= bucket_order; + /* Flush stats (and potentially sleep) before holding RCU read lock */ + mem_cgroup_flush_stats_ratelimited(); + rcu_read_lock(); /* * Look up the memcg associated with the stored ID. It might @@ -461,8 +464,6 @@ void workingset_refault(struct folio *folio, void *shadow) lruvec = mem_cgroup_lruvec(memcg, pgdat); mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); - - mem_cgroup_flush_stats_atomic_ratelimited(); /* * Compare the distance to the existing workingset size. We * don't activate pages that couldn't stay resident even if From patchwork Thu Mar 30 19:18:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13194757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA24AC6FD1D for ; Thu, 30 Mar 2023 19:18:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E834B280004; Thu, 30 Mar 2023 15:18:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DE4B1280001; Thu, 30 Mar 2023 15:18:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BC2F6280004; Thu, 30 Mar 2023 15:18:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A9FC7280001 for ; Thu, 30 Mar 2023 15:18:19 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 705BA120B0F for ; Thu, 30 Mar 2023 19:18:19 +0000 (UTC) X-FDA: 80626525518.27.9B0E4BD Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf28.hostedemail.com (Postfix) with ESMTP id 9FDC9C0011 for ; Thu, 30 Mar 2023 19:18:17 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ltNePRXu; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of 3eOAlZAoKCPcxnrqxZgldcfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--yosryahmed.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3eOAlZAoKCPcxnrqxZgldcfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680203897; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xcUaqJ6BFfuwa7nRJSvBlAwhhxeRf5mk+29EPOCHz/Q=; b=Lb19UDqhfDy/a8OaiYD5WpJMczSegSigAyiuTXnXBiOTVsrPOmgLwl7v3gEvem3x2328iF YIG7JCMwoIbYKiGclof8ZZJ1a15cs2ilHm9br8ZosCrSXzP+KWNElSk8bJzf8mnoX/7KXU 2pFALjh/zlEYu4lbytBtLWLbLEJFK/E= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ltNePRXu; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of 3eOAlZAoKCPcxnrqxZgldcfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--yosryahmed.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3eOAlZAoKCPcxnrqxZgldcfnnfkd.bnlkhmtw-lljuZbj.nqf@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680203897; a=rsa-sha256; cv=none; b=2taSBBTxe2QVrj5d+dOfFEtVxcDHCjnwshjp5xI+QbeuzSZttj1cpPgm1mD7lEQNONsqEp 1VZO0GREsfV2YyUMFUy8K2MTOrFXSw91gVZj8BjLvxgf05K1k7dPIgoA0FGPT8vgLegX95 xccr/Rz8JHg2PMtIS7QJ8jVAwnz3X38= Received: by mail-pg1-f202.google.com with SMTP id s3-20020a632c03000000b0050300a8089aso5681150pgs.0 for ; Thu, 30 Mar 2023 12:18:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680203896; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xcUaqJ6BFfuwa7nRJSvBlAwhhxeRf5mk+29EPOCHz/Q=; b=ltNePRXuZIePvYweAMHUj9ZiAeC5qavTrptvs7HWUZy4ZQjHtls0D9eRHEg0pqkaqA 2rj8RSL+AK/a3iTAV1slsfA/XO8i87DoFdpL6M03j+wstTxD0gNLM1Eo1e2c6UpyLkfA 02M3+F6NXEWgNHJDDpxLzGC2sGDBHCNs+LO2yZgTc79ihpz2oe9Gkrj2oQSZRb8nP/pk DTjpgFJ4LNEzmr5tCkDpR0cOQrENPev1JFpwtylRIUOchlXAi5bgsrxJgKXBgxobydtr bGBrN9gUtdaDKVSRv2j9cJxRBN0C3BLXf8CobNS0sZxfbCzcE+BakwfvTig0jcnVMYxC dA2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680203896; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xcUaqJ6BFfuwa7nRJSvBlAwhhxeRf5mk+29EPOCHz/Q=; b=cbFew21ud2DxnzwWktWnvefOzzqZGY0G92qYiyzkN03uzseEP7ZurMckNbxth/MR8u fD6/xQxajXGxHGEJaKvJ4tzQaj03ifPHEMwJePvBdzz9+iOsuEuiajtoIqNeDFLQogeo LeRYKZMizzo80Wy4VbYtUOeo80OmQvDClIvAqVqix27j/V7RvCXuq7C3n21lzOb0et19 gJ8oDmFreFS1L/EQT1SUiv33gBrGJUpN762rtD5ZNAuZTPH13kixyPOhZ8+gqr7ZjeS/ Vh4qkqSr0LsdJLI+a5mGMJb0xFCOLnzD+Ng2Dx+UX2qNBjOdXdFEqMKrY5P2EPrjlfAh /nbQ== X-Gm-Message-State: AAQBX9fyD0AUZMF+BipWnszQfOW5ckerHo0lGkCf6Vltt7PSAu5e0vJH gG0wd0r0HQB5//dMvXGWziEUVabfF1vmhhC4 X-Google-Smtp-Source: AKy350bGhjPkePduCVIwuMcmgqQ6a76256SvlUw7LUz9ChuwvjB5lA7RHnWs+sTxxUB3GpOtDLTlb1QCGdSZixrZ X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a63:40c1:0:b0:4fd:5105:eb93 with SMTP id n184-20020a6340c1000000b004fd5105eb93mr6667473pga.3.1680203896567; Thu, 30 Mar 2023 12:18:16 -0700 (PDT) Date: Thu, 30 Mar 2023 19:18:00 +0000 In-Reply-To: <20230330191801.1967435-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230330191801.1967435-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230330191801.1967435-8-yosryahmed@google.com> Subject: [PATCH v3 7/8] vmscan: memcg: sleep when flushing stats during reclaim From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed , Michal Hocko X-Rspamd-Queue-Id: 9FDC9C0011 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 9niu5quakad8wfmt9u611bjrxn44tqzx X-HE-Tag: 1680203897-354730 X-HE-Meta: U2FsdGVkX1/rqmq5sSmMQofwRF8lCgWiMPNGqEKN+EW+G2Losky0xxYtcOi978LwsfSHxSJuY/Bcf5qvYYwW3m72UmXiE0r+G52TbNiMog/pAtzp3hBCAAlC5AO8Mbc52Ywnqg7EEs99SPbtn3KUltA7Kk4B0BVSbWi8XVQdj0VvOyUcU4MjOXwRCiKXQNobFxk6U/WY04mL0Gg5vuYv88Lg322cO/nzMSqVfHohSx+I639RW1SZMT6qFb7invx+cgIfE4QxOufaRL2Z7oR1xv+QgIadXeTQhNumQeik0yu4LmPu5tQNDwn+/PqZhorwB4hHGq9x5L74DiFOq7zYw6cz4jtXnnXbkHSQ6Y7GfCJOctyBt+Ql5PsV36IKiH9EgEVUdOaX22E7jAhTIYKUMCUlYKsMGDluu5zo0pCnhlzsTq3bka7tjsP0I2zCiIOw3iFBDrmxT6DdQNelWSLSFwGDQbtFwOl7iUEp0q5hNyLM0Z+JQt0z5cp7o0YhPOYAJfh971xUdnd9Js0EMu+ZlrVqCSASBXyx5rnaS/Peti5adboW5MToIWLMva2k6z/0JzwS5H9Z3Ip4Z4snly3PWkHg76v6TY2BQFqfwORojhse5lUxsDMj9OjCxqtVofphU8nCmiV+zm1N8SctkxjPeSC30y2xOxJO9yz4TG0ZpdaGywFXnvaKzRa03VW20yiAtDe97cCd5GgzJODAaikzO+XpkwYQI3ZvmnGH3CxsSI/vnADhR1f2Afs38JiMaM9gFMcW9xXwMPtYVP26vtbpElaC+GgHk2VAOjN9WiJE55ujuw3Ir2NL2D2Js7+U4dLliWGwB3eH6VqfDaefEHdT/a1PGTkRYvtIz8itpdkFq0EFj4c270HRvTtKw5rhDicCqWnwbzMufM4yCuJmgBUpRKc6ABpjory/DbkWkwQLv8fcGOfbo/K0aGEmjur+IgmYG6ec0TBlV0IXRnI3XA+ 2EqrP+Xm 2L/pZhDvcyqm61kRjvxHT8gKrAgG0zkGY1TQ56dxBS41IUsGIUTjrao0NH5GWNg3BupHcVOKeYiHhOSZqmPFkFrx0NZwpukqO0g0FhajOyN3z99cNIQ3yHrlCp70QKSMf2ubVcD3cZ87qiGWSqvFrCFLtanoc1e5znSHa97tWAE1io59r2ol9IOkRVZ+W/FKin3Te2hCXxhZb+b1uz7ZPgGmoK+ObhzCFnsgvNHQIUcl2n2SNjPnjcjL7IO4fzILcQ1LwbkCghRJHc6uJStdzVN0FvH0nsJWrqwIAcQ8i2cTCeXMXNhXiI/D9+gV6RpAfRh+j59epnU3WrkwUK+LjRgnbDnham+GOoUJbhrPE38Iy16gWQOMG2+5yRwUfoAbiOCL0m5n0S2aJELmg/gV38hYgcVNlK/Xq2TwPX82NeHVxshPa8y6ZT9lamVQxTa3mYzVzB+gTSKVLvmysMvkzE8OS799pxgHL0ZM5A50gejyVHoLujTvcCH3XAKZ5j52G/nktFb2hZy4Or00tf5oY/b8gJJF/hXb5gxzJFEeTDvBCvAD4PYGPkd2KlUcwWbSwAWpgDTXjjVm2JBSrPFC1L3aLVq1EaRcwNX+tCBZMBCZ0kZgIBgIfa+tn4dbuNmVSAv3EiT6y795Id1i5EIvGO8If4ABUfsPkeTCt8OolbcA4qqmYmgVPSdqRAq0Ck1pf11EFcJ6/4F7PZc31/mxyIV+YTJLZPui+Vq4TvXjNV7ca7xkV+9sepG6DhI+OeG5GI+IP1gDG7fdhU/V/hH6ya5KJHQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Memory reclaim is a sleepable context. Flushing is an expensive operaiton that scales with the number of cpus and the number of cgroups in the system, so avoid doing it atomically unnecessarily. This can slow down reclaim code if flushing stats is taking too long, but there is already multiple cond_resched()'s in reclaim code. Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Michal Hocko --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index a9511ccb936f..9c1c5e8b24b8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2845,7 +2845,7 @@ static void prepare_scan_count(pg_data_t *pgdat, struct scan_control *sc) * Flush the memory cgroup stats, so that we read accurate per-memcg * lruvec stats for heuristics. */ - mem_cgroup_flush_stats_atomic(); + mem_cgroup_flush_stats(); /* * Determine the scan balance between anon and file LRUs. From patchwork Thu Mar 30 19:18:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13194758 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CDAD7C761A6 for ; Thu, 30 Mar 2023 19:18:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9EF60280005; Thu, 30 Mar 2023 15:18:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 97B28280001; Thu, 30 Mar 2023 15:18:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 70737280005; Thu, 30 Mar 2023 15:18:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 57532280001 for ; Thu, 30 Mar 2023 15:18:21 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2A25A80E5A for ; Thu, 30 Mar 2023 19:18:21 +0000 (UTC) X-FDA: 80626525602.14.9487D63 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf12.hostedemail.com (Postfix) with ESMTP id 5E1714000A for ; Thu, 30 Mar 2023 19:18:19 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=PUv9MA0n; spf=pass (imf12.hostedemail.com: domain of 3euAlZAoKCPkzptszbinfehpphmf.dpnmjovy-nnlwbdl.psh@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3euAlZAoKCPkzptszbinfehpphmf.dpnmjovy-nnlwbdl.psh@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680203899; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WbNC07VGnJucfMQdaVru/HWxeCTMcNM6hFkgQ/MCeFU=; b=XnXtzwcU8A6bD8z+4/eUFGzNVfdiDWiAXl87rqCwXcSRVi16tnB6lb7wYXisL3s12m48bU 7T1qt/0b7i4cJRoERC1n5zBUQ1GQFRwOosMGfDlouzIVN4m8EoXQUf4hOlpykqATv0/kO8 HPHDhgH3Y4QkYtSQdEgs56j55mwsvPw= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=PUv9MA0n; spf=pass (imf12.hostedemail.com: domain of 3euAlZAoKCPkzptszbinfehpphmf.dpnmjovy-nnlwbdl.psh@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3euAlZAoKCPkzptszbinfehpphmf.dpnmjovy-nnlwbdl.psh@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680203899; a=rsa-sha256; cv=none; b=PzB72aNG6uV4CVKpxtmqj8wxxaHMTL2O0cELNfdaZ4sNKxyWyoi337g/JyH7t0wGxHkeec JrzIwGBCW3nDoRLR7GcSDHxOE5N7djk/p+k8KWX3gVDtFyRd0UwIRNHRmDuwaAQEq6MLAV bYPilN/xvhbTW13I+NIw2XLi6lZSY7g= Received: by mail-yb1-f201.google.com with SMTP id g5-20020a25a485000000b009419f64f6afso19688928ybi.2 for ; Thu, 30 Mar 2023 12:18:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680203898; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=WbNC07VGnJucfMQdaVru/HWxeCTMcNM6hFkgQ/MCeFU=; b=PUv9MA0n+Z8fm7ErrGvHySjECm+JmVMmb7yIewlfjR/61XxilLPkk2HWdbFMTfUOaL VEd5S3Db2lj6fZaM1EVkiIBt/SsoMNdbAydQDGkFf9C+290T1e1uwhZ4tZ9DFf8Phq0h P4OVDvENTCX+3IndK3Cvs6OkRD4foG1c4Yew7PAGOLNrfjOHkmrszhM9+d8Istzt+wK/ krUK3yorRZOqS1BM/g9ye1FRB5jRNaXtQwUxDYcXcRNeFj0C93RhkVtRFak+8MvkC/U9 MW/LMGQgSZbJ8G2s7XbIxfyCgArJtDyQ9vNCBWNE1utNTCLeMAHTcanEcYFbU2EbD5hm feog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680203898; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WbNC07VGnJucfMQdaVru/HWxeCTMcNM6hFkgQ/MCeFU=; b=v5ql9A5ZCzG1g8SwEhDNmklF+HpjHby4rH9ykeq1eDG5ogdFSIdPKAHddSYkCtkcKu 4nSuk9SGZbHKj1TScqbX5/S97idkGhsm+wgyeyUHiZPEzx9L/Mvwj83bkMkReHPdV3jE Q9NPput/kPeYJo36F8yvu7ra+Yf5TZBgXnTU/VVv5MDJQR5VDLFct2Cnw5MhC+Fx2gl0 zTCTIepvq8UaMlYvPOzgkNeViteYUMD47SILTHgzRvFRFCLvZUsuO75/aLyoXZIU76hB BM46SVIaKQ2FjBMZzREJFeoalN+8q/eARg/GzmU317XbYQsZ9aNRASi+PNxI71VxMCh8 YTHA== X-Gm-Message-State: AAQBX9eOUppM0MWbOjnHx0Cx/sM2fXIz6tlFOtGbjCy6W32swaXdlZDt kmBRfxV4ehg0BqJH5WqH42MpPp9rGpY4kx69 X-Google-Smtp-Source: AKy350Z/bGsj2O9Q2OanxyiTVpU0KIn4cOhMHnfEJjh1llR4Xo1YrB6Sk31TDPAteoghtoX9TuixuaLotnLFVi3d X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a25:12d4:0:b0:b6e:d788:eba4 with SMTP id 203-20020a2512d4000000b00b6ed788eba4mr12664999ybs.6.1680203898630; Thu, 30 Mar 2023 12:18:18 -0700 (PDT) Date: Thu, 30 Mar 2023 19:18:01 +0000 In-Reply-To: <20230330191801.1967435-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230330191801.1967435-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230330191801.1967435-9-yosryahmed@google.com> Subject: [PATCH v3 8/8] memcg: do not modify rstat tree for zero updates From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed , Michal Hocko X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 5E1714000A X-Stat-Signature: w7dapencpr4dr48sj6rzddp15q486kmh X-Rspam-User: X-HE-Tag: 1680203899-358933 X-HE-Meta: U2FsdGVkX19MDVGu4M+y40EWvcv57DN3zgzW3QJwJbxW5pUXR4KUOs6K2jqg+gUUILBhIp60264BhENaWOoDj1w991YyxJ7crlv66nmykHVOC2LyOtvdZuents7i1ds6QL6pN6cMXxNMDlajEoOR9ZkMfC279EJdtUoOHSE3nug/iu527IeMEZ+gAVn1qjWIciJhMiRUS3sIjWljLYKhi5rfDM8xWhYUKv1Fb7oAgzqUN7drG2A2i0JXcNwqcEawvfwrQo3HD0A8sZ/jU9U/lNEO09SUyX8uFyh7tuQ88jV2t3jZRFWjjx3JYajmLzWupO+4ZT5Ox6OnNxWdXTK8SKKQVq3GsYoxg+s3/5oH3byGv5xxR+nGGFNF/zqx2gIHibNA7aVgEaIXGJNxbZoxM2Fm7r5j/aOMB+zJP0XCi4RFVNT+w0QHA/UbNQk6m9OVHJogInsKHD0WZxNfcqSjBxQz5gqa2dkhqmMGqAsePyl0ohT0OfE0GZEQPZYYeg54hBlpwf5haWt82ARxWefkLFKs5Z3eI+egupVyhiN05DrzVD3ol+KRj4AupvgZcHTZOha5pB7uZI9iw/qlXhOofIbg1flEGGPMmtPlXG1qwF8aSFTjGO8hnzqutQHRt1s4X1Iiz8YB3Mf4+kgRRLMid7vQ8YIpq5Y7u+/b0KWwo51YdlmwtlEaw/BZcv/blA0Irvp9+EnfnQenIlykhhU3LmAJHj18oJHtNZ/MrQdf9AS/ufisNWocLgkp+wYhmbZpuSugwZWosAXmKv0uJBevxI0c82AKcodqHqZLv1Sm2G1IBuZRz/Wo1NZHM09dcOzwctsVej2RcXwsT90pkgyDe6AgtkXbmHuZzFFQlydkxivE3zq0MkZSIDJMn9ConUvWql0uvuHqtbWCGS06ZV0Lp0vPZceWISa6jacFCh/+rCAWtACrL6U1ApyoXu2iTqUYrUb+Z5UGOfdJV533TQu jOdUs3nH nsH7QXSKwLNYwMGO8u2DO6g34TlDV4Os9Eh2FqL7r35SxUrvozoMxdGc+aeKNds7gOkPQjDe1Cne724H75bKp4y1CuNeaOKjn8xMHTav22bicUAs658fWKcahuGG2FHcFjfTe5LnkFndVEF7+dbOuGX49UODptkdLTr0V1w4x1WaY8n4zMl5/EBEUCt1hS8e1tuqfPHD3PVQbSSMRPzHVSuYFD9ZW/7Bo4wBHHxx+U2rFKMiBuiz++H63rPyB5xtQ79IiC8hGOMOIg9KmYABq6uS9J6QXa25VrauKEFbpVUBB+EdhM5ZVW+YLW6isubSGoBKgdcSAcItFvCijMFeEARxy7vNwaLxcFK9cWvY0w9CRV87iehgExzn9HiXwZU7INiEcTc7WL1zu/7erpxEgF+zQ/DpuFgigeF9gjkRYUog/cNu8QSoHHONaRNEaDlsebZpqw6Q/Z8PUpEGdmsWgKz9LRcOIy4Fp9Mazx+pvj/tE+bOF56zBZQ9X+0XQWnMhBVEgc1Jf6FGnq6xV3tG7tBGgiw+jwJryBhSEHqfNfkKBRd5ZHfyj0cEHg6wIrB+AQ/1l1/yZ6wwT0DalDJ2cQrGmLIdWa5T6mpVj3NfGuhWriQxuPUOutuf9qrAn1a4e6MRLcSfCkqxFITmvgwTywhfIXFzmlxCjgKJOM1RsHWLJmTAd5OPRcctWTihatmPSZLoURjF60SED9yAQ7sez6uE1WwOLKoSHUn+NHO/buwfmaWpuUvpn/Xjp2eM4gIMI/i2gYTcfnf4cNWnrsrJ/ywRKSQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In some situations, we may end up calling memcg_rstat_updated() with a value of 0, which means the stat was not actually updated. An example is if we fail to reclaim any pages in shrink_folio_list(). Do not add the cgroup to the rstat updated tree in this case, to avoid unnecessarily flushing it. Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Michal Hocko Reviewed-by: Michal Koutný --- mm/memcontrol.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 361c0bbf7283..a63ee2efa780 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -618,6 +618,9 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) { unsigned int x; + if (!val) + return; + cgroup_rstat_updated(memcg->css.cgroup, smp_processor_id()); x = __this_cpu_add_return(stats_updates, abs(val));