From patchwork Tue Mar 28 22:16:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13191613 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29CE2C6FD18 for ; Tue, 28 Mar 2023 22:16:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A82BF6B0071; Tue, 28 Mar 2023 18:16:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A33866B0072; Tue, 28 Mar 2023 18:16:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8FAC66B0074; Tue, 28 Mar 2023 18:16:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7FA396B0071 for ; Tue, 28 Mar 2023 18:16:54 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 5006C1C5F63 for ; Tue, 28 Mar 2023 22:16:54 +0000 (UTC) X-FDA: 80619717948.19.1759B5A Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf25.hostedemail.com (Postfix) with ESMTP id 6B102A0018 for ; Tue, 28 Mar 2023 22:16:52 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ckuVrFZp; spf=pass (imf25.hostedemail.com: domain of 3UmcjZAoKCNUPFJIP18D547FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yosryahmed.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3UmcjZAoKCNUPFJIP18D547FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680041812; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=cwQWCEo9ePRYrBfd54RTGHI4ReD6zTB2CQA9ky8D7Xo=; b=p0EyGjRycKkCZWrSPphNAPkYZiYFVLxLGyeX0UVSAT+E+/t3vIE3eCOt/nj0lSIzY66laH lJo0jkLXhE1qwozt/xh1K4/z5HeADKk/X+Mk3BZmGN+Zn3ETcdl7YYtksx212O+i3Xfxjg Sw8cOKyjoUkOB3b1mg+Ay+Ax00nw7Jk= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=ckuVrFZp; spf=pass (imf25.hostedemail.com: domain of 3UmcjZAoKCNUPFJIP18D547FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yosryahmed.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3UmcjZAoKCNUPFJIP18D547FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680041812; a=rsa-sha256; cv=none; b=in9fX6+0t/htTWnSSQRJ1n/0wmVwuxNkAJ3D0E9MDg+sovVeKwJ9Mk6vt5/I4x8cfxBO0s XguKCPiu5EQxqLBqxFASVqpdWaKaZDe2FdyfNdqQK4qddrvcyPaTNCINmTuPsU3ym0r/W0 wrrWqVrDr5tS1hATC/NkjFMXzhoPvj8= Received: by mail-pj1-f73.google.com with SMTP id kx3-20020a17090b228300b0023cfd09ed94so6710331pjb.4 for ; Tue, 28 Mar 2023 15:16:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680041811; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=cwQWCEo9ePRYrBfd54RTGHI4ReD6zTB2CQA9ky8D7Xo=; b=ckuVrFZpRMbKkYaT641zugwGgqS/ki1HcHRGVRwRjkqbpYmJn53n3BJxlm4PtefNMZ V7Lx0t0VCx9xJbkbk0054d9U/+6MMsQejk2NSwnt5D35xCm5Hq5pZm5SJK5CIZfjCnht V0pY409txNDdSi4+78Xf8fYrfgteKRSatE0Yv5rBDd+5j6WpeDHZJpDO5IiqBfLgwy3G 9G3wSwe0z1jWNeULV+cFUkJOWs1knwD7pW4WcXwfaanarWpk2Ozk+fACPDNx1NcCWRdH tAvDGN/2QSuKylRwD/br3VcJA7SF9Tuz8RLoH1VBIr+l7Bie6Py+vND+8JOUEY43Ncvz wX4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680041811; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=cwQWCEo9ePRYrBfd54RTGHI4ReD6zTB2CQA9ky8D7Xo=; b=qfArFJ1OQQXS6rbcxxybK8dQNFpA5L7n8//q5dgNJrZJGaimPLAgdWpRH3zPyXo2Uz m/FTNAaW2gXEJBKpibYhtpMuCKb7xxsZ4TFwuPHHO1QDh2uD1li5O6EvLbYXyHfw/mWE FjxpZAlhMRKJHbh1SCPYp1z2uMGxoNhQ/iNpVqLdP2q9gXMI7AkP2mGmoQxucfL4IWbb 9g4DFM96pOkT/n2/ljIyijvPBjNZp4m4tJAb8E9hOdQHYo4802iUAKXoaiRJLjxj505o QFz0ctldvTshQf1Vbzt4nCBL40soIl96Qs2s6md7JP2dTk7Za0ZExzy9YkNj9mtocmis YzYQ== X-Gm-Message-State: AAQBX9e+9eg9wulMvR168yl6fbkVbRZV+SF/n8TtRfNmatf5KECtiIL/ KjqyiFXoaC2VDueWsD3Bn4Os1RR+8Mhq5PyR X-Google-Smtp-Source: AKy350a0LftA8+YBeO7HxI5a3hE9RhuyDxnrWJZbioOsZo1jxLgNNr2gQblYzMvl1vhkIgmDGcTTrBWiLU1iOxlk X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a63:d34e:0:b0:50f:8c32:79ee with SMTP id u14-20020a63d34e000000b0050f8c3279eemr4775173pgi.4.1680041810879; Tue, 28 Mar 2023 15:16:50 -0700 (PDT) Date: Tue, 28 Mar 2023 22:16:35 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328221644.803272-1-yosryahmed@google.com> Subject: [PATCH v2 0/9] memcg: make rstat flushing irq and sleep From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 6B102A0018 X-Stat-Signature: 5pnygx47x8pf3fbacnt13kzbh3855g8r X-Rspam-User: X-HE-Tag: 1680041812-284561 X-HE-Meta: U2FsdGVkX18VgY8DqGI5A1CYP9uXDl0dpS8H19/qC3aRNksuNA0Kx5KJmce/B85rkcFVs2DLNlx+ggS3Rh8SG/ZZiYlv4CPwKiGYz8qVZWA5/UKGzI6b9YURD309c4Qz9oScOaYbPG/F8voyMC5mZTTG/b67OC0IqeALsG97FrVSE4OEvnqdRQtA6AA5vinywkVlm+dYlXXLReaJKJBJEGE/eLPZPcWETcrRynhjdrk5K2ONn9x3+bGg2lA21Twj4/qym1B+Fv/1ndg5A+ljzyloE2suhbPFETS+FSP/XZat6AmnMd1GCtQOb9vBMDlokBTiy5VbkGttiLuV5gk66fuykQom2Z/L9YpVQ4D5ODW49Qvtoz821TnMuF00snV1Y9qvFi+g5NmluRTm4q74bX/Bm6mdycr55Q2j1vAzae0Yl6cHI4d4nJd41uA8NbrqCEaiGzgVEmKhJPuY0WVYAYRHYue5hbI1v5Lsk4yBdUTUE2bDtge0Y5ABLM3YiAmkjCYQa1T75/x1tK6h0lUQ/dGUMw9hmX3en6nGFBkxxq7jsIHJ1tubRClmiPORgQMxI0dtZbmC/hIiH3LOHMBl6MX8dQ3S2tpTm6MOSOp4bV4UtdmqDHjDpmwUFelFI4vZ82nUlgiIwNrXKBjIc3RnpU34vlPsJl/wn+AEZhLsxzezh2c6r5wpzs9oTxsuhOmAamaCMB3p6JMuhnfhMLqkU1sYVVqGkxdbR9ln/EbsVeORSMcb1+h3Xj4m36cUJ3QIN5DPchxdPQShRX/XGNxu6wDc5kaN7c7imReW6o8O21+8JG7gpt5Y0T0fj/085thTll3x/+E6aPwCx67uDo5/YwnOjRhOv8KE46ePoV5gG+JtT+B2V+GlPRRashpUJXxrOjaKHiwkF0vgsxEm2MvRiLD+KeTeZMR0XTCpUPZ0f4cdDSVFyMgyvETyDwD13pd8bSvXOCqOwfKtysww4Py V3w9B97q j3J/4VnST4+Jyh/9uT0BJ6eCK3ukhsV/fGrxIpkkB8S1C+EnUxsAJglx+NrCCS54x7r4r3AlDv5bffW0qFfMvY78SZVJ5uiPGOGR9WRZEyWNsxcg/adxzrbOI0CV2PNv0GQli1CCzAXjOuJ+kH0whcgwgvn+znx2mIvNF2B54yB/7Yln+CjZfzUiOWV3Hmk9XXisCx2vdESm3pfY3IfYh3hUTzip3nK7oI4vNeHhOf3Keep4vuq9AvxGG1V1/JKSodSWgnr4uU3StTawIepcX50xhsKi4W7jBUqweyV/rO+3YYisBE206m3QCZooqIg+8sXAi01TgsYFfbPAHdZKL+vrqcCpEI5hVDH/rGoDyXkSkjKNug15vOhneJJ5TcoVIra9j69F/zPzdTR7U4z2sXzx4b7X7c423nSYI8Ke4AbfWVHzMqZk6SJ3ySmMyrkXMbf/GConXi//Q92AS2I9JA3Nwjh6mkeKrsgW1h/EBF77zCI6sEfrVlUUhqD0JsFD1JvD58QPS833KTWJ+hlM9zoiPR7nzYustjIDDM9QdduVjdvLqCH2oxwvcikj5BXhhHc2FOMKXdj1R3oYdCZ89U41HXxCzrl5oi8NqDM8LItlZ+eVk5PSM005WfCboW67kzS/m X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Patches 1 and 2 are cleanups requested during reviews of prior versions of this series. Patch 3 makes sure we never try to flush from within an irq context, and patch 4 adds a WARN_ON_ONCE() to make sure we catch any violations. Patches 5 to 8 introduce separate variants of mem_cgroup_flush_stats() for atomic and non-atomic flushing, and make sure we only flush the stats atomically when necessary. Patch 9 is a slightly tangential optimization that limits the work done by rstat flushing in some scenarios. v1 -> v2: - Added more context in patch 4's commit log. - Added atomic_read() before atomic_xchg() in patch 5 to avoid needlessly locking the cache line (Shakeel). - Refactored patch 6: added a common helper, do_flush_stats(), for mem_cgroup_flush_stats{_atomic}() (Johannes). - Renamed mem_cgroup_flush_stats_ratelimited() to mem_cgroup_flush_stats_atomic_ratelimited() in patch 6. It is restored in patch 7, but this maintains consistency (Johannes). - Added line break to keep the lock section visually separated in patch 7 (Johannes). RFC -> v1: - Dropped patch 1 that attempted to make the global rstat lock a non-irq lock, will follow up on that separetly (Shakeel). - Dropped stats_flush_lock entirely, replaced by an atomic (Johannes). - Renamed cgroup_rstat_flush_irqsafe() to cgroup_rstat_flush_atomic() instead of removing it (Johannes). - Added a patch to rename mem_cgroup_flush_stats_delayed() to mem_cgroup_flush_stats_ratelimited() (Johannes). - Separate APIs for flushing memcg stats in atomic and non-atomic contexts instead of a boolean argument (Johannes). - Added patches 3 & 4 to make sure we never flush from irq context (Shakeel & Johannes). Yosry Ahmed (9): cgroup: rename cgroup_rstat_flush_"irqsafe" to "atomic" memcg: rename mem_cgroup_flush_stats_"delayed" to "ratelimited" memcg: do not flush stats in irq context cgroup: rstat: add WARN_ON_ONCE() if flushing outside task context memcg: replace stats_flush_lock with an atomic memcg: sleep during flushing stats in safe contexts workingset: memcg: sleep when flushing stats in workingset_refault() vmscan: memcg: sleep when flushing stats during reclaim memcg: do not modify rstat tree for zero updates include/linux/cgroup.h | 2 +- include/linux/memcontrol.h | 9 ++++- kernel/cgroup/rstat.c | 6 ++- mm/memcontrol.c | 78 ++++++++++++++++++++++++++++++-------- mm/workingset.c | 5 ++- 5 files changed, 78 insertions(+), 22 deletions(-)