From patchwork Thu Jul 20 07:08:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13319865 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 892DEEB64DD for ; Thu, 20 Jul 2023 07:08:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 23C942800BC; Thu, 20 Jul 2023 03:08:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1C79728004C; Thu, 20 Jul 2023 03:08:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 018202800BC; Thu, 20 Jul 2023 03:08:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id DF1C528004C for ; Thu, 20 Jul 2023 03:08:32 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id AD5E7160188 for ; Thu, 20 Jul 2023 07:08:32 +0000 (UTC) X-FDA: 81031112064.12.1393554 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) by imf17.hostedemail.com (Postfix) with ESMTP id EE7DE40002 for ; Thu, 20 Jul 2023 07:08:30 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=PyMiTiRs; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3bd24ZAoKCDgsimlsUbgYXaiiafY.Wigfchor-ggepUWe.ila@flex--yosryahmed.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3bd24ZAoKCDgsimlsUbgYXaiiafY.Wigfchor-ggepUWe.ila@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689836911; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=P4XNYMJYgxf4Y2ysW943aMZDD9DgQegnQSW9aAxie4w=; b=BeiTQ8SQkYUBH3KClBabLzGH9aaEwHDj0dSCiQxB3pD8FTT1LFfhnm46DvYpTfIU7MQ2fC 9JomcbCkehd08385pHWQps+gV7Ei6bNVk9vPG+QvsUM7YMeCS5Rcel+LEtBmwZAOK1HEoQ BZYMsEzROuEwmQ23QaCbT++fzerYyDs= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=PyMiTiRs; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3bd24ZAoKCDgsimlsUbgYXaiiafY.Wigfchor-ggepUWe.ila@flex--yosryahmed.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3bd24ZAoKCDgsimlsUbgYXaiiafY.Wigfchor-ggepUWe.ila@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689836911; a=rsa-sha256; cv=none; b=yCw5WywLe9ucXVSqTHkQo3YtQwGZV0WZ8nkMtvISoptiKksAOqzC3h9lJEfNrgOv5CM4ei vIq11HyM+QtlCr+0BitwEsXqzmWicU3tcxbuli/dC+GrOBBVfmsKKqKCv4ytbUdt0mmJdx KaIs0J4w/vJgpZbGbTk5JA16Fc0SbT8= Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-55c7bb27977so397067a12.0 for ; Thu, 20 Jul 2023 00:08:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689836910; x=1692428910; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=P4XNYMJYgxf4Y2ysW943aMZDD9DgQegnQSW9aAxie4w=; b=PyMiTiRslPlYxeAzNGyf+s2RMn2Y5D+io4F49/M2XRwJrS8TwV/LhLqLj0BM+kLdbs SJdh95igG91J+W+YW3QhUBoovRNlfDdy0rtENHO9Q0tdlMq6Mi6nZDp8wPmseQG3GEz7 yTNtdLVRNpjOoL79dp3VDPbMRHNV99200JwxoCivq9oH22XjolvYivx1g2ceZNC2Pn7s Da/CFyreh5mYpyR/lWRRdvMG1hkWdPWRhELE3hNZjAEKxkN7vhsHgeLNCSqpeMvOeAHq lRZ2ErdDGOl8n4urJ5Mk+4WBu+P+b25Mk8sxetPQU+cK2DCaPQPxX5kv/nJSdD/CPPwf W7UA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689836910; x=1692428910; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=P4XNYMJYgxf4Y2ysW943aMZDD9DgQegnQSW9aAxie4w=; b=aNWjO2xe2GtmYqaN0mxqA/8G44A7ox5aLf4RxPZtVNJjg4A1KewZonm8j8Oj89e6YQ X0JzZP/kF/VrkyAEv7ehGLuQqEaelcfHEwp1Jgcn0TDaUyK1E/y7LEGE5wbFD8hLhmAM v0uS63EN+LFcIB4ZXXljsRKtu91pC5mowZ6bReOjaCYYIDYu98UdlpfAvJJA3v3q2RhY L3DErwGJBRPlKB+AOo0T+CcK/EbjMl8Z6SOzNYmbVjODcMifBelth0wS7I2dU9mFvaPn JlRtHv7P5/WpP149Vq6e6X7LWqcU9pN3F65DH2P6UhBnzha9emm/95jC1o8d++nDQWxw tFIA== X-Gm-Message-State: ABy/qLZKSBGE3B3wVTBhwAswPCMZEPn/sH4uslY55JOQSvlZvQM+ibg0 yua9/KJu/u6n3MpE0nHKdVPH43m32Oihu3/b X-Google-Smtp-Source: APBJJlEGR67b7I5wM02HN8lsWXpPeepUUbVvmNMwkaq7frNphjV2zKpBSorV/PgnNk3Mo7NS/0jW1PWGUeUjmfjP X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a63:3e8f:0:b0:563:3b08:f869 with SMTP id l137-20020a633e8f000000b005633b08f869mr22818pga.2.1689836909753; Thu, 20 Jul 2023 00:08:29 -0700 (PDT) Date: Thu, 20 Jul 2023 07:08:18 +0000 In-Reply-To: <20230720070825.992023-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230720070825.992023-1-yosryahmed@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230720070825.992023-2-yosryahmed@google.com> Subject: [RFC PATCH 1/8] memcg: refactor updating memcg->moving_account From: Yosry Ahmed To: Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt Cc: Muchun Song , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Yu Zhao , Luis Chamberlain , Kees Cook , Iurii Zaikin , "T.J. Mercier" , Greg Thelen , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: EE7DE40002 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: qf19eh9wacrcu5e4i5tz15fp19h5o71t X-HE-Tag: 1689836910-38944 X-HE-Meta: U2FsdGVkX18qy/ONYdVDai92C0kdKngMhVuROLySVndHtHPkSCNIjeMEeNJj6rMeS6DxDh42Co8ckLCXDdiskkxSR4pDwy7OiSTakdhYdXc8+AgI02QFXuIe5k6sk9R033gg4Kq/vrMdLm3lNt7FLVcOqjG5pi/G4HdrGzMr2C4Cu4NTGFqqaqefNYfW73N/cT3VgScrxsCKPF0MaGOwB6ciKWyV5SbfqpZ4Iu4S/7VzCZQ9B+dTPZDIeAtbXXeBmWW8QvVsh1LuUitZiwcY5wVEscE7wBWAFQ+3tFP5UIZwvzy/yGuLy3Zv5/YwfNHrHVT1mPFEn7nJ3oVqq1DOAIogptTwyoXdlrv4iBcWDZ0iRSgKiHOs5d+rn4niJMKcmw0fZS0cjTw5lwvSPp6yM33EZkgAxCncj9V4Yy96QjIZfOmgpHmucFwpBzxtOr8QI9Fl0HZ+4NT6zJlY4GeIFoZHRBoWVIlJH1SJ0GeBjYcaXlYdvZTv7sXrmRrJzimvCrMlnP/l8iH/N2MJzkVfhWQsGqkoNaXHkY0JC4b9cLvrevpWOtMR0XcD/B/KkT/qmkJQhGCyTK44LkH3T4ca5BPdV//n/V1Ls+05yQuM3G0a1Cewf8hkz/OEhgMpcuUagSf47HuVbVHtG/G8XvRYXDYVgtPIGSbXnX9zr73A/ARymEOwF1XddHPHwBSv9hsZWE2XqH8TFTXat0VwylSUcHXDaAxq+teX+hViZ8s/WKSTODys+y1J+Gh5T7DYQUiBjp4cM7tXkTuCK6TExDXASpZtuej5zsANv4Wi16/8NjxfrGmtbmIdegVVV77hhRwq1VcCacubQAtjLbi45m8Hsrz1GAl8agyEKRWSFEIsRb1aTAyvUhNSIOXga1suWIpXkXj3Mhy0kc/3JJu6QvcqNXyqHDjR1QnIXdkxW60o+inT9pT28cbDMS/eH9t7RDBxSHPfm+UFKVwHmg3zC78 ZKl16J8G pDiggzZ5v59GyD6KBV+hMgusQq7gb3/jgQremsPHwfCtWZGcDjacPd0iiP8CTEUNxab5WMoogAJZ+ZlnwvjU2oqE/4otC4jLXCfmW6oQogaUcG8yEs14IH96tlWQcZAYs4pKY/e3aUeJlKMPkvSmU85lXLhy+5Sk7r3UkNJk8PkZk9+SYHS/3iPmZNiWPSWT+imldvUFp1A8GRjfzQ26irmhwt19fqhdQ6uThB9oRf73ogDxuNwoFy0KpZ//zcgg/prIca5zxbnmtyJtrwofLbIG5286esRFQWhfTocVh5+n6NxC2Mujnu5YYGyzboKXzVmxF6q52d0M6s4JlgLMZKCA+qNh66S1KyGIrQyp2xuMPn4zDCKZKUWXgKEEOh/nYmbzZguco+yvCM9qNfp4NTb7iWdpCHkKTZ7FdJcPe6Is3IhLVqH7rt14KJ0Z9D7mvZ0h41yP3YB8aFzWKpZ7H1b8ZyofYI+QuVvTgkEbySLO4eYTNrGE8O8w2C9ugE7ivjRWK6mzN7armRNF9dfCimkMnb9CMV+WSv0j54MCDxFC1sFWgOhQoYXso3521Guqq+r9a4vKeyYbP4165KqY0BAdjxpalEqzgnwMaxYLRrQm+o9DHVG/h/MMiRrpHCdhPp+foCBNwXuJv5J/njQL0y34RFHXZfcJe0hhIQs7EbXB8R8Z8OUaxiZL6EQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: memcg->moving_account is used to signal that a memcg move is taking place, so that folio_memcg_lock() would start acquiring the per-memcg move lock instead of just initiating an rcu read section. Refactor incrementing and decrementing memcg->moving_account, together with rcu synchornization and the elaborate comment into helpers, to allow for reuse by incoming patches. Signed-off-by: Yosry Ahmed --- mm/memcontrol.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e8ca4bdcb03c..ffdb848f4003 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6305,16 +6305,26 @@ static const struct mm_walk_ops charge_walk_ops = { .pmd_entry = mem_cgroup_move_charge_pte_range, }; -static void mem_cgroup_move_charge(void) +static void mem_cgroup_start_move_charge(struct mem_cgroup *memcg) { - lru_add_drain_all(); /* * Signal folio_memcg_lock() to take the memcg's move_lock * while we're moving its pages to another memcg. Then wait * for already started RCU-only updates to finish. */ - atomic_inc(&mc.from->moving_account); + atomic_inc(&memcg->moving_account); synchronize_rcu(); +} + +static void mem_cgroup_end_move_charge(struct mem_cgroup *memcg) +{ + atomic_dec(&memcg->moving_account); +} + +static void mem_cgroup_move_charge(void) +{ + lru_add_drain_all(); + mem_cgroup_start_move_charge(mc.from); retry: if (unlikely(!mmap_read_trylock(mc.mm))) { /* @@ -6334,7 +6344,7 @@ static void mem_cgroup_move_charge(void) */ walk_page_range(mc.mm, 0, ULONG_MAX, &charge_walk_ops, NULL); mmap_read_unlock(mc.mm); - atomic_dec(&mc.from->moving_account); + mem_cgroup_end_move_charge(mc.from); } static void mem_cgroup_move_task(void) From patchwork Thu Jul 20 07:08:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13319866 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9A45EB64DC for ; Thu, 20 Jul 2023 07:08:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 416762800BD; Thu, 20 Jul 2023 03:08:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3280528004C; Thu, 20 Jul 2023 03:08:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0DF0A2800BD; Thu, 20 Jul 2023 03:08:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E1DF028004C for ; Thu, 20 Jul 2023 03:08:34 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B3AFEC0196 for ; Thu, 20 Jul 2023 07:08:34 +0000 (UTC) X-FDA: 81031112148.23.23DDD99 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf20.hostedemail.com (Postfix) with ESMTP id C46F61C002B for ; Thu, 20 Jul 2023 07:08:32 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=1DoBmPRp; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of 3b924ZAoKCDoukonuWdiaZckkcha.Ykihejqt-iigrWYg.knc@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3b924ZAoKCDoukonuWdiaZckkcha.Ykihejqt-iigrWYg.knc@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689836912; a=rsa-sha256; cv=none; b=vcn0sYAYqmWL3TlL5XU91iwBYPexBRJU9hDdp+gmp7qtN4z10KVnm9hH8rjaKuZBZm+Vx2 Sr0RggkSpREl8VuoXCiI08r/q/7JftOIdlYf7JhQ7kYetJx78vKoYjbQIFPSfVotlaGniB 3XL0mo/5B9XCC+dKF61tOKhixgBmPKY= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=1DoBmPRp; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of 3b924ZAoKCDoukonuWdiaZckkcha.Ykihejqt-iigrWYg.knc@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3b924ZAoKCDoukonuWdiaZckkcha.Ykihejqt-iigrWYg.knc@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689836912; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GTtTsHYHZoypkh/ZbOFrb+97cNaIZGVMYPTBvEUZ8QA=; b=ZnZZ+Eak7ZOZUwox6s1SQzHzqHqUq/Ovm5mGoDwiJZhMxOgy8WYVIz23CwH1bRfYm/4DjK g5EAjS0U31hXUt7USKLeQeUJPSYKhaWa3mYyeSLMEXyd9ceWgEgIyJAsPkWwcn0jaEPRmH +YoywdwL6pSIJiKZbt4SqAW5hEQDt+s= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-c4cda9d3823so421545276.1 for ; Thu, 20 Jul 2023 00:08:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689836912; x=1692428912; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GTtTsHYHZoypkh/ZbOFrb+97cNaIZGVMYPTBvEUZ8QA=; b=1DoBmPRpqoKxp5AI1ATOavPZeo0Whgkd5Tc59Swf0g3JyV+Sf4czsw5vt5hRV/iIWU m0gxZI0ZpwmKuzcQe6n8ABa+pR3RNfGEwG4M79yda3Vq21aVBJ/g7QPw4WwY0KIttAPP qZs11I8hXcrNiTzcuVsx+5sSznkYJ5ijPhs+K2QY4AKhJ+1OPsJskvfvc2riuWl1X0pr fjY+rxKO1ypRGpwy/gVhyEiUbivpXGQsHDrV1rAgXPICZpfqdKSn1Khg2rXXV4x7hO0B V/yFis6Q3bBvFsNeipAISnXIhdf4e5y1csYuoWKJL9e1bvfMGj/AyzmZSyj99NVp9/vT 61xw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689836912; x=1692428912; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GTtTsHYHZoypkh/ZbOFrb+97cNaIZGVMYPTBvEUZ8QA=; b=a4YzMxGVWfpEax1R8jZUHA2BRB2CMESwvF2LbNQQKwTNJcGrjRPJy4HhxApayUWZQo uZmcViRA2x/h9zTX+iIsvPDnzDP3NkwcJxmQTsQ68oz6tCfmLKRnWhc00je9YcPX9HAT DIFTKEIXYlx/kQDH3v0FyLHvJZaP+ipeCfuedC98qPPkxybTrzFtA+6ucR7Gb4Igx0iL Nua7srf4L2fPjvMiOeOATJHM21tiNlZOH93p0X0uqkYPqHfqJXvBbuRJlm7FUaVJwnno Cj/rcRKHJlvU6SvNovcGh267TyPYAGHwR2mdja8p1MJx+Sja765PIo7uq/k586rgEzXC B94A== X-Gm-Message-State: ABy/qLY458EkQzyd3xzSN8mOER1gh3qZityHUYEuM43+2mpf5AAGiRj2 gxfB1a0+kbDiNe2gzQh2V85yeDSMi7/d6TXD X-Google-Smtp-Source: APBJJlFLKOPz8aT2ynaMGRvS7LvS60oh3DFyuJzygEiZuz8hLCEZV4fcd2R5Eiopcv2P65Ha4e5fn/ebih6t3IaU X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a25:53c4:0:b0:c6e:fe1a:3657 with SMTP id h187-20020a2553c4000000b00c6efe1a3657mr38214ybb.3.1689836911707; Thu, 20 Jul 2023 00:08:31 -0700 (PDT) Date: Thu, 20 Jul 2023 07:08:19 +0000 In-Reply-To: <20230720070825.992023-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230720070825.992023-1-yosryahmed@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230720070825.992023-3-yosryahmed@google.com> Subject: [RFC PATCH 2/8] mm: vmscan: add lruvec_for_each_list() helper From: Yosry Ahmed To: Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt Cc: Muchun Song , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Yu Zhao , Luis Chamberlain , Kees Cook , Iurii Zaikin , "T.J. Mercier" , Greg Thelen , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, Yosry Ahmed X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: C46F61C002B X-Stat-Signature: 7srz1jsjdekf6p1csss5ty6njobu49a6 X-HE-Tag: 1689836912-176847 X-HE-Meta: U2FsdGVkX191Ie5XKPQHf0SPdzoJOs1nkgd1DI+IZz3s8qDA8PqKZcNdsxyZaNnLrzFsSj7JUFQz+V1VS7i+5plfG0IcMJHHC7nnz346CEi7YrdkIJ4/4re/R8+FB/7Jo2Lv25KFX5rDqxhbQ9dIDBC+88z0uKFJq5Z6Ws1LIfCWDbwwp/YfY7onqwURSfuJrg58eF5gd9V4Yahuj6L4HDWkGA2IKEUbinIfo3zV/pXhrtyb642zwwR7DDFLx9DjT3qYNZlTdZqGBjSnjFYi9k3TWplAtmzdKaA6yJjMFFA16irKn0O6RD7EMvvioWGJj+Wg+z0pZh8kKH73qcSwqXGe8qCtPJuEy50ugIW9Db9Xomq+0FkH5QMwmBxgHL2hDhWTHPb7NQMtP8bq/xD3ATYiAIyM91zWn/WvxUP1DL6cWiVUkagWjaVr6jE/6h+Ct6ji/lRpP6opgM+e+Ov5qM2/YA7kOV8X630MVC2m9N8QXe9Vbq/At/QA/O0YOd1Hhk8F/zf6JG1F5HLWuYgiPd7JgsujHCi4d0hbmwi1W08LuJPJ8hPAJnV4Vqh5TzbT4cj2gL+o1WOJ6FWQcIUxapjQmX5Pi7bmB2u5MY6OCnDWgSZ8t7hIZ9BLvy4dFOtOJSYtZFeDT7qO1+02JczpsqLFolxoj26DvfUCWXvDGpyKv5ql6AEOL3JsaxCE+FyYhqLDHffRXFfeirSDocjZDYU7hvfAPcdGPjU7rSwc/Hhpjvv4prLbpbX1HMzfVSMHOLyM9CKDoepWANdRYFcJ1w+mtVNL59UIlFKpfuUC/xhohNZoDbwASngXJ+8RwgJWQyaWnd9ZgfteoudGvNQC0E0zdfTvUE0FrRHluuB5Mxly2U0PhyqaucWUT0ucWFnuaP1qmetgoCncdPZLTx8CVf//JF5Ij+PWGGY0rLsUqYqesvKrzLBvXUHhgQeE7cJytvmVpCDA2+D95mARK7t Q7DWBPHP mC6xWsICDA2u4x4imrizFH3d23J8QSVojFn/vwRuYLFyEUieF/Vq/MW6HmjAaedlfsHxReJq/La7BkyteZdBvajZp2l7T/MqhkdYDEqURTkfHw1gqHJNvAftOdu7JqcOHYmhRtB9ksUj85iSDY1woDMVbw3h5/K+Tjj0+31YE2r4wMqCpu/6I8JUFVtrMqe2hybaa5bAboDsWPQaSuWAFHF5wr5XOLCA6lak7ujdUqVUDGRM15BKgt66l06GcwfZ4FG53SHVFH+CeKFCvGwNeqxoRSMlI3Wn9OwBXEeshHQUu9uF6wq08ZYbDrvdbtDCNeT4yLY/jxPne/4qYryc+MoDD0NVd0eWXdLLerR/7S9wQxyBBwPEhHQFp1A2kiSuqap79WBwWjcNks3OjWsLaB6VZrWsFzpLPeutaVCP2AVxlSk/8YoXH5qxhdVBT+WQCYBRyttCDpvoPfG67ZEb4wzXoAch8Ci71LUFNzWX43wNF5I2UKU/DqP4NJZjHNADJvRHQhiaA1Av5jSu7nU2fstwZwAdyMD5ZjUCQqqum5I/Vm18+feufxobwEH1UGTncnA0S/X/cRAazNtmhnRGwz+3K0Ci1XcBkxAvh85Rm8Jnd3uAyWIjWA3GImHhw3IWR3NKFO+64PHFqubxPpb1BypDTZL4wh7NQKGzmFZZh4IIIYwstaEItBw8gyg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This helper is used to provide a callback to be called for each lruvec list. This abstracts different lruvec implementations (MGLRU vs. classic LRUs). The helper is used by a following commit to iterate all folios in all LRUs lists for memcg recharging. Signed-off-by: Yosry Ahmed --- include/linux/swap.h | 8 ++++++++ mm/vmscan.c | 28 ++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+) diff --git a/include/linux/swap.h b/include/linux/swap.h index 456546443f1f..c0621deceb03 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -406,6 +406,14 @@ extern void lru_cache_add_inactive_or_unevictable(struct page *page, struct vm_area_struct *vma); /* linux/mm/vmscan.c */ +typedef bool (*lruvec_list_fn_t)(struct lruvec *lruvec, + struct list_head *list, + enum lru_list lru, + void *arg); +extern void lruvec_for_each_list(struct lruvec *lruvec, + lruvec_list_fn_t fn, + void *arg); + extern unsigned long zone_reclaimable_pages(struct zone *zone); extern unsigned long try_to_free_pages(struct zonelist *zonelist, int order, gfp_t gfp_mask, nodemask_t *mask); diff --git a/mm/vmscan.c b/mm/vmscan.c index 1080209a568b..e7956000a3b6 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -6254,6 +6254,34 @@ static void lru_gen_shrink_node(struct pglist_data *pgdat, struct scan_control * #endif /* CONFIG_LRU_GEN */ +/* + * lruvec_for_each_list - make a callback for every folio list in the lruvec + * @lruvec: the lruvec to iterate lists in + * @fn: the callback to make for each list, iteration stops if it returns true + * @arg: argument to pass to @fn + */ +void lruvec_for_each_list(struct lruvec *lruvec, lruvec_list_fn_t fn, void *arg) +{ + enum lru_list lru; + +#ifdef CONFIG_LRU_GEN + if (lru_gen_enabled()) { + int gen, type, zone; + + for_each_gen_type_zone(gen, type, zone) { + lru = type * LRU_INACTIVE_FILE; + if (fn(lruvec, &lruvec->lrugen.folios[gen][type][zone], + lru, arg)) + break; + } + } else +#endif + for_each_evictable_lru(lru) { + if (fn(lruvec, &lruvec->lists[lru], lru, arg)) + break; + } +} + static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) { unsigned long nr[NR_LRU_LISTS]; From patchwork Thu Jul 20 07:08:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13319867 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DBF9EB64DA for ; Thu, 20 Jul 2023 07:08:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C2D7D2800BE; Thu, 20 Jul 2023 03:08:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BB2C028004C; Thu, 20 Jul 2023 03:08:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 991E22800BE; Thu, 20 Jul 2023 03:08:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7CBA028004C for ; Thu, 20 Jul 2023 03:08:36 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 50B5B1A0196 for ; Thu, 20 Jul 2023 07:08:36 +0000 (UTC) X-FDA: 81031112232.09.FBC4C38 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf30.hostedemail.com (Postfix) with ESMTP id 5CE6580004 for ; Thu, 20 Jul 2023 07:08:34 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="e/D65lKv"; spf=pass (imf30.hostedemail.com: domain of 3cd24ZAoKCDwwmqpwYfkcbemmejc.amkjglsv-kkitYai.mpe@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3cd24ZAoKCDwwmqpwYfkcbemmejc.amkjglsv-kkitYai.mpe@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689836914; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zkE9erfuSG8gEvl0NErmWP/HjsAjs5BO1W50P95ENGk=; b=upbVGbtc+qemgwp3Dmk8tOWpXcq0lWDCsC6QLwp1enndOsncHLXor08nBPC6B4sfmVr/V/ PNi8rXT8amiu4X8YaoxhkDk8XVq9w9iN3fBvJhXaa+tUaDzMVBgmhTbclb2fd0M2NRAwfe c67omu8B+7Gm9U7mutaR5LkSrHaQxpI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689836914; a=rsa-sha256; cv=none; b=1QiBHJAw0VdlU969XDZvTxvhFafRY9KX71JN4iYHDDLG2Ramjfn2j8Zs4tkhu6UvULwCFI IipJhCyuLLLCSvsHncmBreYV+Yj/rfUwi2DOUJUj0MigGDaoSHvZeTDXXvSM+tLRtYQCvQ QmEpvl4ApN7LA9XXiJcPnUsseezH/r0= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="e/D65lKv"; spf=pass (imf30.hostedemail.com: domain of 3cd24ZAoKCDwwmqpwYfkcbemmejc.amkjglsv-kkitYai.mpe@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3cd24ZAoKCDwwmqpwYfkcbemmejc.amkjglsv-kkitYai.mpe@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-c64521ac8d6so469250276.1 for ; Thu, 20 Jul 2023 00:08:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689836913; x=1692428913; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zkE9erfuSG8gEvl0NErmWP/HjsAjs5BO1W50P95ENGk=; b=e/D65lKvyv50V3yr9BFRknnH79sSEuD/+1vUrcA4yvr7h7CImtQrd+boASPoSkcO0L 6nRrg9RNuJbGc3QdJ+P9rn/E7NPWcOSDBxQOvrkM1p7NFo6krBFtBVk5c7HmdKOK4Es2 PuB6i+F5kcVT9zlYpRauNGwgIdsKA8qks2YqdR/pIFwq1jYHE090fjgDVGsSN7ZNarF3 4Cq9H5nWz2ZZjH50Yn64E7tOX0T+9lLwqWlGsXWl0d6eVKSMhor5hN/I8U314hOaUpAt UvHj1RrFPkxtGpaYcmmpPta/OEHLEEHI1epsszfPSw/hGh7pMPrxFrxoMgzBHB89kmi2 mFqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689836913; x=1692428913; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zkE9erfuSG8gEvl0NErmWP/HjsAjs5BO1W50P95ENGk=; b=OC1sqoF53PTX44QrvFrEigfNvchNfP4KmPCFrWVGkI9u7LaH1NLwO6HPZYpaDn0Hzf 8MtZ6ElhAdS6hJd723p796vtD/0ZyXzcM39P8/x1r6/lscwhAUbeoaKuelwuNFbMhcOB neviobDF1SvTNU9G6Txem/tUellQaUz3hWRuuJ/06PfZmBEo7cdU4VrtsJE/KshuEBCd 8fQGI9y1e6Nga5bO/DZsRQCIsZ2nxycJdNtFohxE/iYLAxN2CxaraMbKdrLSZanxPl6K 4y1NpGq2LxHSVtAL9z+f5hSCv30wOhhSH6c966LMBudlTVhRIhQjHA/EAeyFKhYahK9J BQIg== X-Gm-Message-State: ABy/qLYl3F3LURSduulG3zAcFI2SWT5OnN6Aa8qEbnMiDB5Rz0/a6sWL tnaWaA1guI4ctX35qCVuiEe6g5LK2WxUYrSr X-Google-Smtp-Source: APBJJlF1XLkdIp+rrL6VnpxoZHQAd5Y1MvxoDSvMdyFfa3VhNOkSbCTVEqhKIhjXU4YLSRrvLIXqVgBHET8DWPA4 X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a25:a021:0:b0:cf2:9e82:a5b5 with SMTP id x30-20020a25a021000000b00cf29e82a5b5mr32616ybh.8.1689836913480; Thu, 20 Jul 2023 00:08:33 -0700 (PDT) Date: Thu, 20 Jul 2023 07:08:20 +0000 In-Reply-To: <20230720070825.992023-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230720070825.992023-1-yosryahmed@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230720070825.992023-4-yosryahmed@google.com> Subject: [RFC PATCH 3/8] memcg: recharge mapped folios when a memcg is offlined From: Yosry Ahmed To: Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt Cc: Muchun Song , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Yu Zhao , Luis Chamberlain , Kees Cook , Iurii Zaikin , "T.J. Mercier" , Greg Thelen , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: 5CE6580004 X-Rspam-User: X-Stat-Signature: r1eb1xgfkq6pepxd9ohy66ni6tutss5x X-Rspamd-Server: rspam03 X-HE-Tag: 1689836914-209663 X-HE-Meta: U2FsdGVkX1+PqSHV7ugzDUDNK/qgx8ZF/2fdjWTFTyrsQ3ooWgcl0UK1vyFOOESZp5FltZpilYXirddhlFtMIy3dCxFwh7K7YTLWE96p1VUU/aY5MhbCKyp8M9jq49/3o+Fl3YV+hztx1aZn2Dsm0m6bhWFBgr6Q1OnfV2kx8kRzl3jn9XwFsUNQncbXWEKp1lJ9uP8fK2laf0ameFRx+YpLSeo5LPgsp0miX4q9vTvbDCKF3/w/s02NibIFfZdWUPmX2BYMRlAorGnzB++oD3EofU3k+Ak2LW0Idi5Q1D0kMIDPxM3EJ89cLTRcsVETU13qD0USeRKOE+w3hzkFBfQu4R9aQt92phsqinLwhTBfO7OH2w5jFxr2eVrPJqJowsCtku2ieA+nXAIFjaEZuAz1U4IBW50u4x5IBFtpEIaScNXIQjDeDSxthe4QAd/dmol0tkttultSnOZAOVSWodWgHQKjqY6h07mNblWuR2PFA7xErqdMGNiDey0JkWXuoErVszwvUSI2husMmQv9kViSB1g8fLDDpDFjiK4YuhwLndhmHKYb2y9i2b5vhAaDwSjFObG4PTAGE+dLt1tnkExGBab784h6n92eT4o/Ba/SqwYMEG4n7v6sNtENP55zpNCQEPCfTdDWTrqo5vk7/zzvUg9rnxh4UxD+BCJ2IVMq8ooWYhx7rZn0LuamXKsmNk8MgrBvcFqPuUlRQF2NfcELOkWyXFjTH2LB/koo7LbEp5cPH+tooQiKVwr4ynRAxyac+uAFAeECSzEW8C55IPCbYG6b7MoJqHeFbxlv6sX0b7ROZYwgCu5c4sz6Cda1T0BaCk9RaAvHTfDkqquzixhWFD1JqsQ+rgEKKI6I4+V/2L6KhMlcpLPGgPFIArqcORIZvXnC5XK/yT1fKMWbEHobF8s/ehHegJE26DK7K4kine0hgbCKspoaUvsPwtHG5bxNPSvXUmTIWGPirES LmNAKLWm d6GYsSAf9JyWImK6Z7+phvU0V3m2KwlF+e8G74IjhD/SzBgAUyMg4oph8fI7BscR8e2YJAyAC+6d83TcClFRTeQaAKYzPo7kt2MvvWEsFoDBH2I61p/Dpg8uCUMDNkxSCIkcLgTswVfoG9TL1cTHPjKHmlMJ5mRR7zpKBAt33kB0uOYTA/ANIPpIv/zuJDLy09BxSaKvIxcBXjDfvNlDs+gQvs6TY1JcMTt7OPYEP3IdUPpZJaXFX2YnP+5IrHKW03QtlxOih55pDMtXcKWRFU/ybYUzcHmBaBvx869mD67FDAc0TN1WPInkTJwEGvjAvTXi4CoUMmsnLtXV2ELxhBBb/MqgWqWVnP5QougVQ/ovoB+qsWxc1UjIGLl9douUIaG0hiMkggRuA9Cu9Bp3kw9uneI5HMaTR8YsR6nuMe8tnM7284R7n/XikdoXbhBynVM60aHcI9ckFNQgs0t/BVWqCzEiV6i1OBSxd5HGvEUAiNXpK7ADW3SBQgeED40wd/NuuLXCwG8hCkwE76yy3TmJ0oSlik+4Bw8Y+ccb5CtairggEwreXkPRlVRZxQVXOVP8dOM1cER16Kbo7NN+ySlzxwu/KL9WcTY+OUYheUY0QpaMQ7IZ8JiErNA+9UDW+MgMCbSVcd8TYjzE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a cgroup is removed by userspace and its memcg goes offline, attempt to recharge the pages charged to the memcg to other memcgs that are actually using the folios. Recharging is done in an asynchronous worker as it is an expensive operation and needs to acquire multiple locks. Recharge targets are identified by walking the rmap and checking the memcgs of the processes mapping the folio, if any. We avoid an OOM kill if we fail to charge the folio, to avoid inducing an OOM kill at a seemingly random point in time in the target memcg. If we fail for any reason (e.g. could not isolate all folios, could not lock folio, target memcg reached its limit, etc), we reschedule the worker to rety. Signed-off-by: Yosry Ahmed --- include/linux/memcontrol.h | 6 + mm/memcontrol.c | 230 +++++++++++++++++++++++++++++++++++++ 2 files changed, 236 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 5818af8eca5a..b41d69685ead 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -326,6 +326,12 @@ struct mem_cgroup { struct lru_gen_mm_list mm_list; #endif + /* async recharge of mapped folios for offline memcgs */ + struct { + struct delayed_work dwork; + int retries; + } recharge_mapped_work; + struct mem_cgroup_per_node *nodeinfo[]; }; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index ffdb848f4003..a46bc8f000c8 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -96,6 +96,8 @@ static bool cgroup_memory_nobpf __ro_after_init; static DECLARE_WAIT_QUEUE_HEAD(memcg_cgwb_frn_waitq); #endif +static struct workqueue_struct *memcg_recharge_wq; + /* Whether legacy memory+swap accounting is active */ static bool do_memsw_account(void) { @@ -5427,6 +5429,8 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) return -ENOMEM; } +static void memcg_recharge_mapped_folios(struct mem_cgroup *memcg); + static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) { struct mem_cgroup *memcg = mem_cgroup_from_css(css); @@ -5455,6 +5459,8 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) drain_all_stock(memcg); mem_cgroup_id_put(memcg); + + memcg_recharge_mapped_folios(memcg); } static void mem_cgroup_css_released(struct cgroup_subsys_state *css) @@ -5487,6 +5493,7 @@ static void mem_cgroup_css_free(struct cgroup_subsys_state *css) vmpressure_cleanup(&memcg->vmpressure); cancel_work_sync(&memcg->high_work); + cancel_delayed_work_sync(&memcg->recharge_mapped_work.dwork); mem_cgroup_remove_from_trees(memcg); free_shrinker_info(memcg); mem_cgroup_free(memcg); @@ -6367,6 +6374,219 @@ static void mem_cgroup_move_task(void) } #endif +/* Returns true if recharging is successful */ +static bool mem_cgroup_recharge_folio(struct folio *folio, + struct mem_cgroup *new_memcg) +{ + struct mem_cgroup *old_memcg = folio_memcg(folio); + gfp_t gfp = GFP_KERNEL | __GFP_RETRY_MAYFAIL; + long nr_pages = folio_nr_pages(folio); + int err = -1; + + if (!new_memcg) + goto out; + + err = try_charge(new_memcg, gfp, nr_pages); + if (err) + goto out; + + err = mem_cgroup_move_account(&folio->page, folio_test_large(folio), + old_memcg, new_memcg); + cancel_charge(err ? new_memcg : old_memcg, nr_pages); +out: + return err == 0; +} + +struct folio_memcg_rmap_recharge_arg { + bool recharged; +}; + +static bool folio_memcg_rmap_recharge_one(struct folio *folio, + struct vm_area_struct *vma, unsigned long address, void *arg) +{ + struct folio_memcg_rmap_recharge_arg *recharge_arg = arg; + DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, 0); + struct mem_cgroup *memcg; + + /* + * page_vma_mapped_walk() is only needed to grab any pte lock to + * serialize with page_remove_rmap(), as folio_mapped() must remain + * stable during the move. + */ + recharge_arg->recharged = false; + while (page_vma_mapped_walk(&pvmw)) { + memcg = get_mem_cgroup_from_mm(vma->vm_mm); + if (mem_cgroup_recharge_folio(folio, memcg)) + recharge_arg->recharged = true; + mem_cgroup_put(memcg); + page_vma_mapped_walk_done(&pvmw); + break; + } + + /* stop the rmap walk if we were successful */ + return !recharge_arg->recharged; +} + +/* Returns true if recharging is successful */ +static bool folio_memcg_rmap_recharge(struct folio *folio) +{ + struct folio_memcg_rmap_recharge_arg arg = { .recharged = false }; + struct rmap_walk_control rwc = { + .rmap_one = folio_memcg_rmap_recharge_one, + .arg = (void *)&arg, + .anon_lock = folio_lock_anon_vma_read, + .try_lock = true, + }; + + rmap_walk(folio, &rwc); + return arg.recharged; +} + +static unsigned long lruvec_nr_local_mapped_pages(struct lruvec *lruvec) +{ + return lruvec_page_state_local(lruvec, NR_ANON_MAPPED) + + lruvec_page_state_local(lruvec, NR_FILE_MAPPED); +} + +static unsigned long memcg_nr_local_mapped_pages(struct mem_cgroup *memcg) +{ + return memcg_page_state_local(memcg, NR_ANON_MAPPED) + + memcg_page_state_local(memcg, NR_FILE_MAPPED); +} + +static bool memcg_recharge_lruvec_list(struct lruvec *lruvec, + struct list_head *list, + enum lru_list lru, + void *arg) +{ + int isolated_idx = NR_ISOLATED_ANON + is_file_lru(lru); + struct mem_cgroup *memcg = lruvec_memcg(lruvec); + unsigned long *nr_recharged = arg; + unsigned long nr_staged = 0; + LIST_HEAD(folios_skipped); + LIST_HEAD(folios_staged); + struct folio *folio; + + /* Are we done with mapped pages on this node? */ + if (!lruvec_nr_local_mapped_pages(lruvec)) + return true; + + /* + * Iterating the LRUs here is tricky, because we + * usually cannot iterate the entire list with the + * lock held, and the LRU can change once we release it. + * + * What we try to do is isolate as many folios as we can + * without hogging the lock or the cpu. We need to move + * all the folios we iterate off the list to try to + * avoid re-visiting them on retries. + * + * The folios we are interested in are moved to + * @folios_staged, and other folios are moved to + * @folios_skipped. Before releasing the lock, we splice + * @folios_skipped back into the beginning of the LRU. + * This essentially rotates the LRU, similar to reclaim, + * as lru_to_folio() fetches folios from the end of the + * LRU. + */ + spin_lock_irq(&lruvec->lru_lock); + while (!list_empty(list) && !need_resched() && + !spin_is_contended(&lruvec->lru_lock)) { + folio = lru_to_folio(list); + if (!folio_mapped(folio)) { + list_move(&folio->lru, &folios_skipped); + continue; + } + + if (unlikely(!folio_try_get(folio))) { + list_move(&folio->lru, &folios_skipped); + continue; + } + + if (!folio_test_clear_lru(folio)) { + list_move(&folio->lru, &folios_skipped); + folio_put(folio); + continue; + } + + lruvec_del_folio(lruvec, folio); + list_add(&folio->lru, &folios_staged); + nr_staged += folio_nr_pages(folio); + } + list_splice(&folios_skipped, list); + spin_unlock_irq(&lruvec->lru_lock); + + mod_lruvec_state(lruvec, isolated_idx, nr_staged); + mem_cgroup_start_move_charge(memcg); + while (!list_empty(&folios_staged)) { + folio = lru_to_folio(&folios_staged); + list_del(&folio->lru); + + if (!folio_trylock(folio)) { + folio_putback_lru(folio); + continue; + } + + if (folio_memcg_rmap_recharge(folio)) + *nr_recharged += folio_nr_pages(folio); + + folio_unlock(folio); + folio_putback_lru(folio); + cond_resched(); + } + mem_cgroup_end_move_charge(memcg); + mod_lruvec_state(lruvec, isolated_idx, -nr_staged); + return false; +} + +static void memcg_do_recharge_mapped_folios(struct work_struct *work) +{ + struct delayed_work *dwork = to_delayed_work(work); + struct mem_cgroup *memcg = container_of(dwork, struct mem_cgroup, + recharge_mapped_work.dwork); + unsigned long nr_recharged = 0; + struct lruvec *lruvec; + int nid; + + lru_add_drain_all(); + for_each_node_state(nid, N_MEMORY) { + lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(nid)); + lruvec_for_each_list(lruvec, memcg_recharge_lruvec_list, + &nr_recharged); + } + + /* Are we done with all mapped folios? */ + if (!memcg_nr_local_mapped_pages(memcg)) + return; + + /* Did we make progress? reset retries */ + if (nr_recharged > 0) + memcg->recharge_mapped_work.retries = 0; + + /* Exponentially increase delay before each retry (from 1ms to ~32s) */ + if (memcg->recharge_mapped_work.retries < MAX_RECLAIM_RETRIES) + queue_delayed_work(memcg_recharge_wq, + &memcg->recharge_mapped_work.dwork, + 1 << memcg->recharge_mapped_work.retries++); +} + +static void memcg_recharge_mapped_folios(struct mem_cgroup *memcg) +{ + /* + * We need to initialize the work here, even if we are not going to + * queue it, such that cancel_delayed_work_sync() in + * mem_cgroup_css_free() does not complain. + */ + INIT_DELAYED_WORK(&memcg->recharge_mapped_work.dwork, + memcg_do_recharge_mapped_folios); + + if (memcg_recharge_wq && memcg_nr_local_mapped_pages(memcg)) { + memcg->recharge_mapped_work.retries = 0; + queue_delayed_work(memcg_recharge_wq, + &memcg->recharge_mapped_work.dwork, 0); + } +} + #ifdef CONFIG_LRU_GEN static void mem_cgroup_attach(struct cgroup_taskset *tset) { @@ -7904,3 +8124,13 @@ static int __init mem_cgroup_swap_init(void) subsys_initcall(mem_cgroup_swap_init); #endif /* CONFIG_SWAP */ + +static int __init memcg_recharge_wq_init(void) +{ + if (mem_cgroup_disabled()) + return 0; + memcg_recharge_wq = alloc_workqueue("memcg_recharge", WQ_UNBOUND, 0); + WARN_ON(!memcg_recharge_wq); + return 0; +} +subsys_initcall(memcg_recharge_wq_init); From patchwork Thu Jul 20 07:08:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13319868 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA395EB64DC for ; Thu, 20 Jul 2023 07:08:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 80B232800BF; Thu, 20 Jul 2023 03:08:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 792CE28004C; Thu, 20 Jul 2023 03:08:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5E4DA2800BF; Thu, 20 Jul 2023 03:08:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4046028004C for ; Thu, 20 Jul 2023 03:08:38 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 0A021A018F for ; Thu, 20 Jul 2023 07:08:38 +0000 (UTC) X-FDA: 81031112316.01.7C413C5 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf03.hostedemail.com (Postfix) with ESMTP id 3D60E20016 for ; Thu, 20 Jul 2023 07:08:36 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=buokeDrF; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3c924ZAoKCD43txw3fmrjilttlqj.htrqnsz2-rrp0fhp.twl@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3c924ZAoKCD43txw3fmrjilttlqj.htrqnsz2-rrp0fhp.twl@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689836916; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pwRzTa1HWBGDz2y6kKmJLG3qmZZBS7SfdYMDl6S6pYU=; b=ZokrgKPjM44spE/2tXHz2NUGsVqFTglxZlspmaIhJsMG5CxUvXhdxF6QUOrrfMX1nCaZ6G Z3gdmsrmy3ngG5b/5NWoMcFFs9Uj0T3TvcWk14OKw9lXgv2K6ks+3AIP+Ka+PoEDVeJOeM Ycyj9kHjD+oVo+0PdSjEZ9evgcgW/us= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=buokeDrF; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3c924ZAoKCD43txw3fmrjilttlqj.htrqnsz2-rrp0fhp.twl@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3c924ZAoKCD43txw3fmrjilttlqj.htrqnsz2-rrp0fhp.twl@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689836916; a=rsa-sha256; cv=none; b=uRoDMREdItNgA9fJK5MwC7ISew9tb6ERilMYW+z5Y17jpToeVeGnMW5zUxi65oZtW22hli R7EjlrMaBiYzSRL9Ax8k33W7KGJQRYZjnc32xBDqV8vKb8Pm8nuJVQ+8u8Gll8xB8KA1ra hfoMoatKVxwxqFSkD+WORMOWq6Ph8jM= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-5618857518dso5324307b3.2 for ; Thu, 20 Jul 2023 00:08:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689836915; x=1692428915; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=pwRzTa1HWBGDz2y6kKmJLG3qmZZBS7SfdYMDl6S6pYU=; b=buokeDrFKzaUQNx7mhOgxc7yhdstiZpTj86UXwGCh3nTqt6qszmfeuRufu8Lj7sJoB FZ8d+MffXHna2mt12Y1ia8+ZNaiYEL4KKqiWl5QyksN4EKDtA5MmamlaJzubK1L4iKoj JnIOyZVRWPOZEDrLS+Z7GpFAMpF6YowvDF+xXDYfmorPwNBAYyVmDpwyqMhca813Xnoe RKGYMhZTq7+fZsmgRNH21NnT88RSM7deT4LXZ9ISfVkZGDcZnSBTWZjY2y9bjfmsBNsN UTtLHPVn8pgWNVg/M6uRpXBrkyjKtRhGZuuMQmP3rtFreh0ZALGBo7Ltyu1x30E1hL2z 2vZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689836915; x=1692428915; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pwRzTa1HWBGDz2y6kKmJLG3qmZZBS7SfdYMDl6S6pYU=; b=KKTbDwnFRswcL5n14bc733/YCrQUY3DWSq1LuAvA8xya3gHWmttBouJocFzc1aeVoz HEcDbKvt1V3gJRHxuDfL+iyNmzPzE/RbTvqSjHvcz1TpzqVSiEr+xhL2T90uAXpNODf2 fLC3QsmW7XfDMERswj/p7jhgouGtRn/afJNUSz/pBbAn2Lb+K1xqIEJ6TvihPLilrV+t If5Pm24Ns+JcGvEXMlxtJIsjtP8lj7XS6bmbeR+Lmmf+DQHU4lIkCnsSd40vjveHWNOB VhltqgcATvx/SQT+ckrjUhZvwN+y81U7xcHJJZhGyjFZ+hxiMrAvVDfE/jzRHH22wjvx YPng== X-Gm-Message-State: ABy/qLYUW2qwB04iCPiojYGF92HJEaj4NQyk/Bwpt5FfxxZ1u0+xE2vr 78dEX0ufbtPMI3/HsmVyVqvN+iPuMvKBnHRE X-Google-Smtp-Source: APBJJlE07xGsF+T8VwpNf+vzMb7xCWgMW2wkAUXSoba/cYRX8PQTLWxD8xY/UgGAeecgR6Yrkm/cfEiUn0RQpzi8 X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a81:8d51:0:b0:579:fa4c:1f22 with SMTP id w17-20020a818d51000000b00579fa4c1f22mr224500ywj.6.1689836915238; Thu, 20 Jul 2023 00:08:35 -0700 (PDT) Date: Thu, 20 Jul 2023 07:08:21 +0000 In-Reply-To: <20230720070825.992023-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230720070825.992023-1-yosryahmed@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230720070825.992023-5-yosryahmed@google.com> Subject: [RFC PATCH 4/8] memcg: support deferred memcg recharging From: Yosry Ahmed To: Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt Cc: Muchun Song , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Yu Zhao , Luis Chamberlain , Kees Cook , Iurii Zaikin , "T.J. Mercier" , Greg Thelen , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: 3D60E20016 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: ripd55wnfbiwxhejk7nubcecyzocspmn X-HE-Tag: 1689836916-799144 X-HE-Meta: U2FsdGVkX18SIDQ3w9MfsX/PUc9Qdu0kXCpFH3u2bJsaT8S9jvSikmNK7uOZHecL9HkIkfyVILUYcdOpHl2EW0/ihSLurWx/cRxFk3jz6ICvuX/6nYi5kKGxwCAKg8ZTfiqBH2LvB4BUxaxabAi90Ik/sWXWMrufNIunaf5hcL/KqrF8h9O61eMBwn748rJLs4awBuvVRbWRwZNFpssupsS6aYQTNXaIcP9umHe43THYePXFuXmdZmmZutdymfn6COHIvtNoZjpJZZMTdVMsMayUW9KJnhUWIaLVDjprAX56Y98Szhl8zi+aIczELnSoJNT1qYvWtHyG3FanOtUyVzHpxMYwjPkXdHxEMpGCXSHtV5RJopgu8e85Fb+7MKU3+V5/SRI2fjF0JE1MSdWzwBk0/cO2ycWpnfsfWW7MHPuS6JDAlQVZuNJQGyzwy4o9c/dM+n95lvIyM5toWynDAg1MsbVErMwgcNVRqHBOHCLOKSCCcura4o65JbmM81MJDoHeE7Uu3alFF2/nnWB3Fy8+PcDc1XhGenfjYWIRxlaMqttphFQGncNevty+m5JRpJ9pnHyqOlkkQSOVQjLFFteAXYAUb+2y2ID3vBFg3VS7zUc6tnR9LrKnsnov0uiOtFtfav8ioUZbH1V+t64s5i4vb6v7udEzDXsKa4Wp7j1S7ZAYBfu0A+craBfTjZ7mRaGM5wLWYvhO+I760Z+j05lz8ZOt/xl7v13nZwgRdilYjE/rdRacSbh8z6w3NTtZYO+ja+tRIQg0K+Dsl5MLq1M2ZNuw9ybl7pkgR866YJ3GLsEXVSC3BdX+FFZPpI6bMZLBfVtd0GpE7Iy6Ap+/efhVLs2CY9Aem37nxjRvaeKLsSAgtc+Pdwavw7z/vtzlMWR1e+kc6ivtB0DqWez+Q7JFwh8BqN8an52V+D4G1Mga2ucWAei+t5DhIXgff+xrMdAM++NcUs3faI5JRvV FPgeKfQl r4wgJpYMdt68Fzfq4EGn92YJWrltpLnEvTnwlATwkGMLwBIpu7x9tvqbZuMkd/eMl75ctJ0JR8/mpDZRY2ETOj0e6g8nTu93wyWtxplBOIDNpHZtmrcJU9hgRN7GW0+LZoSzVQAYcouSoytGO94oRDD0NKziCC5XzWRQoroz09F4LCsIc4AAADA+p7vkIcDeRt95K360d6ljDMUkKoJxB8at9syHM2vxiB+VL1xYT18cSnBhnIMeC6tnQyKqPx2LSWgEg1z3lxvbfmvhmRJtGsdL6D9RJ9CapRy25uApbN2XN4+Yltic3FFEYd15xTs5oW9u8fylfqQl3QG1X8kfOUaxMhOKHBccIxNg4vyktsN4N+5ibQnw4Y2ynjoFJmM3TD8V8fbz5uw3mZ8Ykp/kraDCBNHyfW+csxcmAgSGOuIrpB/9Yn3UFjeARM8TfVNqmCx/qMxy6RSnwXs0k2NBe2thYVrGpeS1Qia+B+3XO6v6VMlZeffpqdiAbE5BvGUtywSFBJQTsE37M+b7WdH6WScwWwR3/xxHS1ig46yrLpuSQBDpX7b6366oWPE1iyUyUE7dFDM5jH0720RAJokBYYqqr8r3lW7VQ9uUnlVulk6AGfwBrPkofCuACJljteHlWYVRInnH1qGwF2e8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The previous patch added support for memcg recharging for mapped folios when a memcg goes offline. For unmapped folios, it is not straighforward to find a rightful memcg to recharge the folio to. Hence, add support for deferred recharging. Deferred recharging provides a hook that can be added in data access paths: folio_memcg_deferred_recharge(). folio_memcg_deferred_recharge() will check if the memcg that the folio is charged to is offline. If so, it will queue an asynchronous worker to attempt to recharge the folio to the memcg of the process accessing the folio. An asynchronous worker is used for 2 reasons: (a) Avoid expensive operations on the data access path. (b) Acquring some locks (e.g. folio lock, lruvec lock) is not safe to do from all contexts. Deferring recharging will not cause an OOM kill in the target memcg. If recharging fails for any reason, the worker reschedules itself to retry, unless the folio is freed or the target memcg goes offline. Signed-off-by: Yosry Ahmed --- include/linux/memcontrol.h | 6 ++ mm/memcontrol.c | 125 +++++++++++++++++++++++++++++++++++-- 2 files changed, 126 insertions(+), 5 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b41d69685ead..59b653d4a76e 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -956,6 +956,8 @@ void mem_cgroup_print_oom_group(struct mem_cgroup *memcg); void folio_memcg_lock(struct folio *folio); void folio_memcg_unlock(struct folio *folio); +void folio_memcg_deferred_recharge(struct folio *folio); + void __mod_memcg_state(struct mem_cgroup *memcg, int idx, int val); /* try to stablize folio_memcg() for all the pages in a memcg */ @@ -1461,6 +1463,10 @@ static inline void mem_cgroup_unlock_pages(void) rcu_read_unlock(); } +static inline void folio_memcg_deferred_recharge(struct folio *folio) +{ +} + static inline void mem_cgroup_handle_over_high(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index a46bc8f000c8..cf9fb51ecfcc 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6398,6 +6398,7 @@ static bool mem_cgroup_recharge_folio(struct folio *folio, } struct folio_memcg_rmap_recharge_arg { + struct mem_cgroup *memcg; bool recharged; }; @@ -6415,10 +6416,12 @@ static bool folio_memcg_rmap_recharge_one(struct folio *folio, */ recharge_arg->recharged = false; while (page_vma_mapped_walk(&pvmw)) { - memcg = get_mem_cgroup_from_mm(vma->vm_mm); + memcg = recharge_arg->memcg ?: + get_mem_cgroup_from_mm(vma->vm_mm); if (mem_cgroup_recharge_folio(folio, memcg)) recharge_arg->recharged = true; - mem_cgroup_put(memcg); + if (!recharge_arg->memcg) + mem_cgroup_put(memcg); page_vma_mapped_walk_done(&pvmw); break; } @@ -6428,9 +6431,13 @@ static bool folio_memcg_rmap_recharge_one(struct folio *folio, } /* Returns true if recharging is successful */ -static bool folio_memcg_rmap_recharge(struct folio *folio) +static bool folio_memcg_rmap_recharge(struct folio *folio, + struct mem_cgroup *memcg) { - struct folio_memcg_rmap_recharge_arg arg = { .recharged = false }; + struct folio_memcg_rmap_recharge_arg arg = { + .recharged = false, + .memcg = memcg, + }; struct rmap_walk_control rwc = { .rmap_one = folio_memcg_rmap_recharge_one, .arg = (void *)&arg, @@ -6527,7 +6534,7 @@ static bool memcg_recharge_lruvec_list(struct lruvec *lruvec, continue; } - if (folio_memcg_rmap_recharge(folio)) + if (folio_memcg_rmap_recharge(folio, NULL)) *nr_recharged += folio_nr_pages(folio); folio_unlock(folio); @@ -6587,6 +6594,114 @@ static void memcg_recharge_mapped_folios(struct mem_cgroup *memcg) } } +/* Result is only stable if @folio is locked */ +static bool should_do_deferred_recharge(struct folio *folio) +{ + struct mem_cgroup *memcg; + bool ret; + + rcu_read_lock(); + memcg = folio_memcg_rcu(folio); + ret = memcg && !!(memcg->css.flags & CSS_DYING); + rcu_read_unlock(); + + return ret; +} + +struct deferred_recharge_work { + struct folio *folio; + struct mem_cgroup *memcg; + struct work_struct work; +}; + +static void folio_memcg_do_deferred_recharge(struct work_struct *work) +{ + struct deferred_recharge_work *recharge_work = container_of(work, + struct deferred_recharge_work, work); + struct folio *folio = recharge_work->folio; + struct mem_cgroup *new = recharge_work->memcg; + struct mem_cgroup *old; + + /* We are holding the last ref to the folio, let it be freed */ + if (unlikely(folio_ref_count(folio) == 1)) + goto out; + + if (!folio_isolate_lru(folio)) + goto out; + + if (unlikely(!folio_trylock(folio))) + goto out_putback; + + /* @folio was already recharged since the worker was queued? */ + if (unlikely(!should_do_deferred_recharge(folio))) + goto out_unlock; + + /* @folio was already recharged to @new and it already went offline? */ + old = folio_memcg(folio); + if (unlikely(old == new)) + goto out_unlock; + + /* + * folio_mapped() must remain stable during the move. If the folio is + * mapped, we must use rmap recharge to serialize against unmapping. + * Otherwise, if the folio is unmapped, the folio lock is held so this + * should prevent faults against the pagecache or swapcache to map it. + */ + mem_cgroup_start_move_charge(old); + if (folio_mapped(folio)) + folio_memcg_rmap_recharge(folio, new); + else + mem_cgroup_recharge_folio(folio, new); + mem_cgroup_end_move_charge(old); + +out_unlock: + folio_unlock(folio); +out_putback: + folio_putback_lru(folio); +out: + folio_put(folio); + mem_cgroup_put(new); + kfree(recharge_work); +} + +/* + * Queue deferred work to recharge @folio to current's memcg if needed. + */ +void folio_memcg_deferred_recharge(struct folio *folio) +{ + struct deferred_recharge_work *recharge_work = NULL; + struct mem_cgroup *memcg = NULL; + + /* racy check, the async worker will check again with @folio locked */ + if (likely(!should_do_deferred_recharge(folio))) + return; + + if (unlikely(!memcg_recharge_wq)) + return; + + if (unlikely(!folio_try_get(folio))) + return; + + memcg = get_mem_cgroup_from_mm(current->mm); + if (!memcg) + goto fail; + + recharge_work = kmalloc(sizeof(*recharge_work), GFP_ATOMIC); + if (!recharge_work) + goto fail; + + /* we hold refs to both the folio and the memcg we are charging to */ + recharge_work->folio = folio; + recharge_work->memcg = memcg; + INIT_WORK(&recharge_work->work, folio_memcg_do_deferred_recharge); + queue_work(memcg_recharge_wq, &recharge_work->work); + return; +fail: + kfree(recharge_work); + mem_cgroup_put(memcg); + folio_put(folio); +} + #ifdef CONFIG_LRU_GEN static void mem_cgroup_attach(struct cgroup_taskset *tset) { From patchwork Thu Jul 20 07:08:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13319869 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DDBBC001DE for ; Thu, 20 Jul 2023 07:08:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 992A92800C0; Thu, 20 Jul 2023 03:08:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8CAD228004C; Thu, 20 Jul 2023 03:08:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 682302800C0; Thu, 20 Jul 2023 03:08:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 48D7D28004C for ; Thu, 20 Jul 2023 03:08:40 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 239AD1A019D for ; Thu, 20 Jul 2023 07:08:40 +0000 (UTC) X-FDA: 81031112400.09.7957363 Received: from mail-oo1-f74.google.com (mail-oo1-f74.google.com [209.85.161.74]) by imf07.hostedemail.com (Postfix) with ESMTP id 313354000F for ; Thu, 20 Jul 2023 07:08:38 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=QdsbdWEv; spf=pass (imf07.hostedemail.com: domain of 3dd24ZAoKCEA0qut0cjogfiqqing.eqonkpwz-oomxcem.qti@flex--yosryahmed.bounces.google.com designates 209.85.161.74 as permitted sender) smtp.mailfrom=3dd24ZAoKCEA0qut0cjogfiqqing.eqonkpwz-oomxcem.qti@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689836918; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OzwAcynUjm4TDDZtYUdTlt8GRef6hCkFmXsjGPYLvQA=; b=X/DgpT2K1fQNCG3YhL8LIcTvAwcMPRVBDbbeC1aUEl+CLBj8nlwaqTVrrNCthWy163kk+1 M0bWuYMEoH/9zpOV6+pCrnv62Db9tdhaKFkgR2fseAHkkvvWOi2KD5jDqHERHvNx4hUApA iORdO3/BTRmyTIRYMN6Cnm5qgygC/uQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689836918; a=rsa-sha256; cv=none; b=DbYP8bUpUE4RwroFdphspgTuLFYDu+RGF4Hat+sRo8Qk+pd5Pkb8UjvRbCriK8H/B52YXY Cq25vlwMDeIyA29vKbsJo3N9MTLT6kk/ObKpWy6FWpZ38Mhu4auWaee9FqiKjtBwDppfDV yLXf+I/5iZd3ONFn3HWnRffKctc4xXw= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=QdsbdWEv; spf=pass (imf07.hostedemail.com: domain of 3dd24ZAoKCEA0qut0cjogfiqqing.eqonkpwz-oomxcem.qti@flex--yosryahmed.bounces.google.com designates 209.85.161.74 as permitted sender) smtp.mailfrom=3dd24ZAoKCEA0qut0cjogfiqqing.eqonkpwz-oomxcem.qti@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-oo1-f74.google.com with SMTP id 006d021491bc7-565f0c7c243so828484eaf.2 for ; Thu, 20 Jul 2023 00:08:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689836917; x=1692428917; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OzwAcynUjm4TDDZtYUdTlt8GRef6hCkFmXsjGPYLvQA=; b=QdsbdWEvmo4m7dFreE18whaI10HNxA5U6Eq4dlh5z46nHRuNKvgnmtyQrlecaO4sV8 pWMQ+xztiQx/wc1YGWhMYzOCtuiM6PjW82kZXGkYJ68Xwsx5KrsGhi4OgbRK80gIgcB7 clLYTavc+M9b93VW/QkD9WL4K9KunD/+0BOFjNktnoseyObLt2ro/bWXeUjfsH//HPvv iKo/LdVH7XZs+1rJF1M/AwxhXSl+9z13Axu/a0fHLc9te/mN1I0tCp0Us23Ispa/XUDp ufe5XWIgsZtbi6tiF99R36YBIvSUA+Db8luFMXb3WtMXUvnpGh0pFJrrPbseVvr7ZJUC YeoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689836917; x=1692428917; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OzwAcynUjm4TDDZtYUdTlt8GRef6hCkFmXsjGPYLvQA=; b=F/jep/c1PVlsDZKbWEqiQ0c76esEbbyOR5Xk52UvDC5C4IwHNp3FX5zeciOfMiqfAQ 3PApAk3zY+mF+s3eeMTcxVpPgUFjkip0cdzlsLISmlQRMxLO1BYXuPeIT5DG8E+NGMmR PTG/c6Gv7W1gPoKEj7wO+SFqOgBLy3JxoOAmTBfVooXW7BSw5d72Hzr8QeoCAgnd5hec qcdK5vUjXkVzj9ui4oj3FLqy63GccScxqmqPuQHQgBgciiCT5C0bUTXwdR7JJLGvAsit 8YAhXA879d2ek3TbU+CMuDs1ygLayL0XLJMgEkVKil916FL/VOh+qoFkqrvKfAW2gXQy lZdA== X-Gm-Message-State: ABy/qLaFObdHuGGEcqZyh2ZVWUwt+i+IQfw282P6xU24PMdPOpLkgxUs Xhqz2q5s618PHYr65iCBrrvFnZjaVg+l2oNC X-Google-Smtp-Source: APBJJlHxP5P9yoiXFXdFsa+fxsrJ91qoDIJe3rq084GoRyXYn6ED56dX4GksMdKRSpbhY7hotG9fq78xxc+urVcp X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6820:34a:b0:563:4841:891f with SMTP id m10-20020a056820034a00b005634841891fmr2470502ooe.0.1689836917359; Thu, 20 Jul 2023 00:08:37 -0700 (PDT) Date: Thu, 20 Jul 2023 07:08:22 +0000 In-Reply-To: <20230720070825.992023-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230720070825.992023-1-yosryahmed@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230720070825.992023-6-yosryahmed@google.com> Subject: [RFC PATCH 5/8] memcg: recharge folios when accessed or dirtied From: Yosry Ahmed To: Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt Cc: Muchun Song , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Yu Zhao , Luis Chamberlain , Kees Cook , Iurii Zaikin , "T.J. Mercier" , Greg Thelen , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: 313354000F X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 7goh9xmqcqgq84rb3qgyb5raard1apiw X-HE-Tag: 1689836918-405576 X-HE-Meta: U2FsdGVkX1+ZHXD/Ff3gMZbM6uwCaxELavyM+u/iFRls05BdnfPePOEwaI845gF5zPagzca8JP9mu0Rap3UsCXFqamsUwoMX391TdW2g9NunRB0MZkTL1Z9sgtWyat04JV4PeddQaXbPAM0giQquBFRBeRm4Pi5oIKq9GyFav21vscdFOemO5mFXSdzva79zKlOrxrpWKx6GUttSTIQGfxvHiyerXM/0keQgWotG51XHHgGc5oaIuwsRQKWOXrRy9Yt0B2j66USvfLvEn6BXDV1SlEKJDDHk69MBRIj/9jpek3qI5NgRfuiNAG9yrbMspgrfZWB5IbCwpI0xF/ZAzdnh75Aor/WFGYfOTUaFYc/sEVC4r7VQZeIwfVgiFi3oyBHmQlTu2gX3afgJ41WwC4TVvTtn40/yr9ze0ZHYfi2oDbUTC/SnKuTF8oM767yYjqh47Bm4ATL+PL5i2AARIxJBRLBCXk1LNmWeklvPEoFhDVe0U0uah58gJ9UyWkJHu34lLwETeOASgZ2iLkzw5j2/xss6GtnHSKNYQVSv1cEW5tdD9EDyOQq0DfXXJtG/ZOkEEH+0XYXrHeCa7AD7aTVdMJY135JrChyy4qbRPw3p3KTXZ/ih7zdK/h8W97DZmz289JpG+nYOysO0RHHTJBiu0l8yc/69mHJOm85FPInddLjLCbKJuwnTWYudAp+0GPG4+OxI4jyHN1G3uGaJtUdWRHy1aE32tb/7/g9sqHKNnaVxgLWKQpe0wdwk7yK+WlmV3QfV3LcT4NHA/H1kDFCaY5qe8o5iBTUE60mJkO6+QYkm6MfrrVQYAfFyJhc12QiDcAbE84zwoOgofWbcRePL+P/jxSEzYnWUUnkiVNt4lR3EWrQMAvXkw5XUrv8rYCw3u5VtbIr0Fgz45ftybflyLjwjsMk3d3Z5jtXK2LAhW5aYrSmwuEjxEP5fVOPNTeg43v1b71JPnf3nSHg ZGiG+oRh Aut4Dm1hvQlsJT2LjP96PfN/uuJFSRJBHGTg/IH3jd054tnVTynb3cRZarSQiNOmGL8qFforWqRy8LvOORi4Sq4Qpwmj2JDBJVqtysO4YoX7/cyr8u2nOPg1JKSTBQP2xfftZJU4nAeQ7+qL7RD/Lr42A/qXGsjRapsoLNAnaFIvIKN9IHSVq2Lebu+lZp0qDgbQ9VmiXB2L9ESa2sQPMxOiBrOjfkO+A/s13YhzPDjQNKh4qdZYfyRfieeQLPB+tygecajp0Nh1dNUyyptGJLg5jWyyBndfonvXs+ZbOpluJyl8L/ZezV2xhVLzJAke1iWYCZTM0PL78TQrX5sAQ5GkWI6qyAbzCaOBTqgisK0umCH3wHWnbx+IoPfxU+/2qjQxojYSUbaGF50Typ4MOesR17nCWWijY6qQgvuRTTw2D3MTlI9JTlQzq45cwAb/8Siv71A4R/xL1qEpHkhr5QiEcYh1Lu1ojzyO8VHhG1+ayiGaCfB6RHlrbhgVFH2FQpLI50zNeG+anX5RxDG/KwJI2jjQpmctqTJCS5iNBt2L9yjy97d9a2BDvrp+LOhH8Fqqu+eQB/GksOmLS+7cTlFhH1ARUUfEyHWb31dA/4mqDKUxweAmVQdjeIsE2wrXMI4ZZ+JwbMpPvfEM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The previous patch provided support for deferred recharging of folios when their memcgs go offline. This patch adds recharging hooks to folio_mark_accessed() and folio_mark_dirty(). This should cover a variety of code paths where folios are accessed by userspace. The hook, folio_memcg_deferred_recharge() only checks if the folio is charged to an offline memcg in the common fast path (i.e checks folio->memcg_data). If yes, an asynchronous worker is queued to do the actual work. Signed-off-by: Yosry Ahmed --- mm/page-writeback.c | 2 ++ mm/swap.c | 2 ++ 2 files changed, 4 insertions(+) diff --git a/mm/page-writeback.c b/mm/page-writeback.c index d3f42009bb70..a644530d98c7 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -2785,6 +2785,8 @@ bool folio_mark_dirty(struct folio *folio) { struct address_space *mapping = folio_mapping(folio); + folio_memcg_deferred_recharge(folio); + if (likely(mapping)) { /* * readahead/folio_deactivate could remain diff --git a/mm/swap.c b/mm/swap.c index cd8f0150ba3a..296c0b87c967 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -457,6 +457,8 @@ static void folio_inc_refs(struct folio *folio) */ void folio_mark_accessed(struct folio *folio) { + folio_memcg_deferred_recharge(folio); + if (lru_gen_enabled()) { folio_inc_refs(folio); return; From patchwork Thu Jul 20 07:08:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13319870 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D41CEB64DC for ; Thu, 20 Jul 2023 07:08:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 59A9A2800C1; Thu, 20 Jul 2023 03:08:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4AE7728004C; Thu, 20 Jul 2023 03:08:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3004F2800C1; Thu, 20 Jul 2023 03:08:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 168BC28004C for ; Thu, 20 Jul 2023 03:08:42 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D3396801A7 for ; Thu, 20 Jul 2023 07:08:41 +0000 (UTC) X-FDA: 81031112442.29.13D65C6 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf28.hostedemail.com (Postfix) with ESMTP id 11FA8C0006 for ; Thu, 20 Jul 2023 07:08:39 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=oBM+gyrv; spf=pass (imf28.hostedemail.com: domain of 3d924ZAoKCEI2swv2elqihksskpi.gsqpmry1-qqozego.svk@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3d924ZAoKCEI2swv2elqihksskpi.gsqpmry1-qqozego.svk@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689836920; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YhI2s0e+GIGqReCnHtRMQPrzSnIqjbw1XBQvBPYjKY0=; b=UalIofI8VJRD7tk9+FUreVDaWdcTvr6FmJYbnnlJ6s3Xdr80LS/LAr02gkPHATXgiLcPd2 WUaJnlQzGw5UnKIhWgJJO0nPj7/gc464kExxpf2IPsWIiqxa6kUQcE6JGFxBedZlJO4yi4 TU/brGufLmJ6C9bcIJ7GJCLGZ0h8Ucg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689836920; a=rsa-sha256; cv=none; b=ekEgAVlN7cE4vrExkUMt9uNRBCRGZSKz1E03Ln6R/0NuuGdQfRqLfVWPPpv1RAH3vjGV3u TcSh8M1Y8THxIzhjFXaIPlGTOBQopxtFNSxrRfunbaSPn4eqeE499hQhy1X08nBICr55pc BETeCe2I+qmNwLBgJmJ18wlqj2AiuKg= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=oBM+gyrv; spf=pass (imf28.hostedemail.com: domain of 3d924ZAoKCEI2swv2elqihksskpi.gsqpmry1-qqozego.svk@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3d924ZAoKCEI2swv2elqihksskpi.gsqpmry1-qqozego.svk@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-cac213f9264so458663276.3 for ; Thu, 20 Jul 2023 00:08:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689836919; x=1692428919; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=YhI2s0e+GIGqReCnHtRMQPrzSnIqjbw1XBQvBPYjKY0=; b=oBM+gyrvc4kZmfzCDqqW014mddb+6ng6aUJcvm1PGzKPmmatk8vBmxnaO+jvawg8Cc cf0h8c+uRGKv+PpUjZOjju465YCs/RMkGS7FOfpfBg905u8KA48PZPv9pv5l5xU64hAu RRIbJwf2dG4G9KtiFyA18EerxfR/ghn9NkJLcaUskUA8yTC4t3T0XaugdSZxINO7vhKx oM7NXvmkzb7JDGBaiO8CTn60src/z33GmAEJWb+ZSh6IMofXorSGDhYJD6lnOVdfNi/O 3BRMWtylTLjUhnMgpCj7SaSMPCTWyEH1d0Cr6gcddMeYKbX7Wxwr5jKmhdVPpppLRUDQ nHOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689836919; x=1692428919; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YhI2s0e+GIGqReCnHtRMQPrzSnIqjbw1XBQvBPYjKY0=; b=fRcKK/GjIUWKOLtqdMJ3SySbwJf2LXFRYcSjexycwWWSXFKV42QAuC79Ug42WjnH/q dwgPrsnXlFRZMfyo6cWy+zVVFmfgcYCE3ESZW9OuFETaIAyuHGcsFkyQBvv5KmPcMhRa FcIglJhgiT+iF8hZo41/V/InBpxjL8Msmpzb2pJsiA2xIxOjU5Q3YyTgIKj2RVIBvdMf RX+z+T0Sun4MSI0HURtRGRyEKvnjSIU3sxaSJGrD8LpoxVTBs9VNuBpecfiq8xoeLIrm niNob31Z//o/rX2ba+f2ZjHND9wdUj5vHO4Glt3RBvLkpSdfg3nT2LyNCnr6p24OT2lH ZJxA== X-Gm-Message-State: ABy/qLYkMEY1rdU4Q1w4QXo+MrePHWsyuwW7+Eleo3KjPUbrd8WWa6pI T+SrATbSmETZd6ToSj/WK4+TCFq7s0M3L6FD X-Google-Smtp-Source: APBJJlF2n+4mQLRSUDirJE55LlRkKhJrzGh/hC5rHcN/nPwLMBMZVZTXUQayuWTk1KX2FFq+vwWoN66wVm+m0JP7 X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a25:ab04:0:b0:bfe:ea69:91b4 with SMTP id u4-20020a25ab04000000b00bfeea6991b4mr36455ybi.4.1689836919167; Thu, 20 Jul 2023 00:08:39 -0700 (PDT) Date: Thu, 20 Jul 2023 07:08:23 +0000 In-Reply-To: <20230720070825.992023-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230720070825.992023-1-yosryahmed@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230720070825.992023-7-yosryahmed@google.com> Subject: [RFC PATCH 6/8] memcg: add stats for offline memcgs recharging From: Yosry Ahmed To: Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt Cc: Muchun Song , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Yu Zhao , Luis Chamberlain , Kees Cook , Iurii Zaikin , "T.J. Mercier" , Greg Thelen , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: 11FA8C0006 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 4ubi74f97cy9sj9i4uwndgfuqkjsgis4 X-HE-Tag: 1689836919-560650 X-HE-Meta: U2FsdGVkX18Z9t3qdv5Et/2d0bGpJg8k7oJ6IS9MeDleD2cTuzalc0dU3FnB5HZAwNZg6m1iE1srSqD/CbYC1FIu58wex8qT65I110hNgPOj6jz9qumUV07GLJ5ykDbCurdPGdiudLUwX2fM5MUhPky39LiVgG5wTSx/ogW6XXCO1ZO9gIdE3PEflREdBCo2HOVTj5OCKCDFecGpezYbU6YrsQb+BQr9l5QwGPklDbwhtoa2S42wf52LZ2DfAG6md21Gdulig3QPLkSotWFgjfMJq/BMtTXp+UPQQsHv1G031I5qWHRBvSjRw1yKGw0fOGIejrJa6jbQ0qJX0uu1WiUZcEOpYHBUAw89HM6Dg0/7TLmGJumEDn3+WM8GOmk8GMarcxtxNJn1JNjL/xJJGovV4rfFX/imjg0C6hfC2Wl2SDa2PibpQBIdZeFs1fuzalerZnGOe3sJRk4uSk66Jrr6+rObtODLJSeX/SdRJM6s8S/+Q3nggsV8kSYSPsD2uA/lffJpxEJvpbyOh8ZL3JyVniM2fKlcy9LhSB11UNpSSTjTz5PJxMRxleJYJPAnj0X59mfwocJJC6lnvqsxjwPTYuLssY3ndPFurp0EGh844TG2YxeJdEal3fl4iXNgF6DtXw0I1PMz2eWoFVFe9DdakYAe8N12A1g41D93Q572w3n8qFr6UoWXEQ5SpAz1onKPA54Ltgjwv2+erngQ9hG3jvgSqwDXs8eOlb0Fm8G7eeHWXdBYZ2syGQGGxjomDGr0E+8onT+Mmqam7NltdRjHnLyXeC8D4nWLcZLKR6kJPBdwBHRc2ZYGI/afQaq4ZPa5G3nnZ3hiAH7pVfnOkoNEtUZl+uxE/G71Kdcy7NXXmrz4Kblhw0kOm9O4qmBa2MQiAJVQIk23x9dZnZnahxpvCq6rIko086RDbrgeCETq3Ic5g987IJbwVN52pxcfRIh40fJOyQ79iDDMbpm HGnOqBD0 8rA+Z8omD8Sm/XCukT0zlrHzi49KnKraykyeE5diR6n1QgO556G+7kUiaXo20TaWx7i9wh61eolHs/53BJ70EIkOFIqDYg8EX5YBkqh4Szovd3gQmOodt73nVQCDZKG8NJ0p+Ech3bMqtSuQObTHrINwVw7f7mATEuZoziBbQUKJQZMjKKDcVPC+gjjhj5MxFKFvnmFghBpnH91RbjvpRlpeNUp/jBo6mlStHDjnJOI1d3lojkhyG+JCYWhLJmJKW4uAaIZVh/muVSg/T0cDWl7NoWqc6dwsFV8uywngE4LpGNoCrpjDIVn5cUhTAJC78D6yAJk+iNwf9TBpCA4q6K0WpirS/k80T0RY1u+fTZXWbtY1qORNnw/p/AA5XYTLnr3G2tin9QrDsw6DVsOvs6cPAzvelli5XCKrjbOOBXBsoHxwqRAx6wFcq2sG6rOvTh3aUHH9GaoGowEQXwZlevafun+v08ZYo80ZJSTwnV1ViCCZuIK/65t4BAzFVA3lKw9CYJCl3Saj/1WqjKLN7LHDSUdr4AO31PcJkEZ4SDo7xZlQXeLMEZPZibUp8b9meBOUvRW72C3LAkgu1QoHK5UDXnOA8lJ7KXSx42fp6q5nHZBbmxfggzk9EfraqC1RnqVHSRq/FUk+9KR0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add vm events for scanning pages for recharge, successfully recharging pages, and cancelling a recharge due to failure to charge the target memcg. Signed-off-by: Yosry Ahmed --- include/linux/vm_event_item.h | 5 +++++ mm/memcontrol.c | 6 ++++++ mm/vmstat.c | 6 +++++- 3 files changed, 16 insertions(+), 1 deletion(-) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 8abfa1240040..cd80c00c50c2 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -60,6 +60,11 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, PAGEOUTRUN, PGROTATED, DROP_PAGECACHE, DROP_SLAB, OOM_KILL, +#ifdef CONFIG_MEMCG + RECHARGE_PGSCANNED, + RECHARGE_PGMOVED, + RECHARGE_PGCANCELLED, +#endif #ifdef CONFIG_NUMA_BALANCING NUMA_PTE_UPDATES, NUMA_HUGE_PTE_UPDATES, diff --git a/mm/memcontrol.c b/mm/memcontrol.c index cf9fb51ecfcc..2fe9c6f1be80 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6394,6 +6394,8 @@ static bool mem_cgroup_recharge_folio(struct folio *folio, old_memcg, new_memcg); cancel_charge(err ? new_memcg : old_memcg, nr_pages); out: + count_vm_events(err ? RECHARGE_PGCANCELLED : RECHARGE_PGMOVED, + nr_pages); return err == 0; } @@ -6469,6 +6471,7 @@ static bool memcg_recharge_lruvec_list(struct lruvec *lruvec, int isolated_idx = NR_ISOLATED_ANON + is_file_lru(lru); struct mem_cgroup *memcg = lruvec_memcg(lruvec); unsigned long *nr_recharged = arg; + unsigned long nr_scanned = 0; unsigned long nr_staged = 0; LIST_HEAD(folios_skipped); LIST_HEAD(folios_staged); @@ -6505,6 +6508,7 @@ static bool memcg_recharge_lruvec_list(struct lruvec *lruvec, continue; } + nr_scanned += folio_nr_pages(folio); if (unlikely(!folio_try_get(folio))) { list_move(&folio->lru, &folios_skipped); continue; @@ -6543,6 +6547,7 @@ static bool memcg_recharge_lruvec_list(struct lruvec *lruvec, } mem_cgroup_end_move_charge(memcg); mod_lruvec_state(lruvec, isolated_idx, -nr_staged); + count_vm_events(RECHARGE_PGSCANNED, nr_scanned); return false; } @@ -6679,6 +6684,7 @@ void folio_memcg_deferred_recharge(struct folio *folio) if (unlikely(!memcg_recharge_wq)) return; + count_vm_events(RECHARGE_PGSCANNED, folio_nr_pages(folio)); if (unlikely(!folio_try_get(folio))) return; diff --git a/mm/vmstat.c b/mm/vmstat.c index b731d57996c5..e425a1aa7890 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1303,7 +1303,11 @@ const char * const vmstat_text[] = { "drop_pagecache", "drop_slab", "oom_kill", - +#ifdef CONFIG_MEMCG + "recharge_pgs_scanned", + "recharge_pgs_moved", + "recharge_pgs_cancelled", +#endif #ifdef CONFIG_NUMA_BALANCING "numa_pte_updates", "numa_huge_pte_updates", From patchwork Thu Jul 20 07:08:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13319871 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE42BEB64DD for ; Thu, 20 Jul 2023 07:08:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F35312800C2; Thu, 20 Jul 2023 03:08:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E98F728004C; Thu, 20 Jul 2023 03:08:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C9B3E2800C2; Thu, 20 Jul 2023 03:08:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9F4B128004C for ; Thu, 20 Jul 2023 03:08:43 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 741E61A019D for ; Thu, 20 Jul 2023 07:08:43 +0000 (UTC) X-FDA: 81031112526.15.65DD1D5 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf20.hostedemail.com (Postfix) with ESMTP id B28821C001D for ; Thu, 20 Jul 2023 07:08:41 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="sV87N/Hx"; spf=pass (imf20.hostedemail.com: domain of 3eN24ZAoKCEM3txw3fmrjilttlqj.htrqnsz2-rrp0fhp.twl@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3eN24ZAoKCEM3txw3fmrjilttlqj.htrqnsz2-rrp0fhp.twl@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689836921; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=n6O0Lt5KhyslMSHzCGrHaCXx7KAwNPu5E55KNetzzTM=; b=vB+dP8ID25WlWjN+mwdAG4sh4YQDRUY6L2NHy8+ZKw8GfP2bESP23QOrfeTJhPYWStAz5r CnO93at83sXxPlbuq5h04jkUio9uh1lUkM16HdUY9OHDlUtUunDPIbZIFwiXb8z8h+1gdG fNDh/1W81k5tSh0ZZrMVznraWvtaptM= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="sV87N/Hx"; spf=pass (imf20.hostedemail.com: domain of 3eN24ZAoKCEM3txw3fmrjilttlqj.htrqnsz2-rrp0fhp.twl@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3eN24ZAoKCEM3txw3fmrjilttlqj.htrqnsz2-rrp0fhp.twl@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689836921; a=rsa-sha256; cv=none; b=hJLFA2sYFnT0Cg8Llb7BjCDAg9++FsKxWk5PfVqBgyUC2+/TGAgVFk0amneW9UVzfqvUoh ssCVx35kKbqOj3b9rLYAILoCEs8LLaNCpNOH4aSN8oL8ExT2daeWVyRFrmF1n3QPjbwZ7M DI9lzzn0q/G5k0beUUfF0PaUkqnWY1Q= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-c64ef5bde93so454564276.0 for ; Thu, 20 Jul 2023 00:08:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689836921; x=1692428921; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=n6O0Lt5KhyslMSHzCGrHaCXx7KAwNPu5E55KNetzzTM=; b=sV87N/HxxcZHHL1alJO5nBf8HzjYMdw6nkcR38Ddq2rUuAf28cTRr0rb709fkiRMW+ qEy3FdhFn01ikwkfx7y9qLguXlaMHGLFmuRtJXH+L+mW9RyHbcOJWlAUGrGs6aAu/INw 41Y2qTn8C7MjL6RvRw/y0YgIGxiJWL68zL7cWNRAYxyzZ5Cms/cFb5dUzlTJ6iCOM/ec 43odK5EQY91sLY+01W8AhKo4cVLmfc3JuvWr3teNDoKfkk1Lyj7dpYXltbO4KTOFr6c4 DheP8D0OnAwjx0EXPzUp5uV1Ui6HMxgha12C+iK56/iEaIj2+ON6C0S7v2ijUdptonk9 M9ZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689836921; x=1692428921; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=n6O0Lt5KhyslMSHzCGrHaCXx7KAwNPu5E55KNetzzTM=; b=S/j4NnRrMCLZGOWAQSiVibUDRHmfdSVZu7+4O2uZB4/fbAmLSRPRU/jK8JB/G8AIbV eBKZTPsm4O+X7oqt4KPOY4OgwFXtJZ0bc4qbcMcgViw/kB5Lqh3skXGf994x964tabu+ 5MZchWKrb3CNU7ClDwLCNLIjAurSwa8QdsYPD3POCPiDkJPnqUvkrKxajD6l9GTY0NcW 923svQCB/r+XSkuwOsiRrqEMEN/T8HbvQrse+NSTWq/JbJA9bRKZu1vE94HeC77A2MEm dkHHXZDzhdFgbosSC2tQeo6QZeOx/CIuQEVGErMH2BbnMXhruqOjvwHohRlyQTmkKC2f /gsQ== X-Gm-Message-State: ABy/qLaBeIiaGKlE46+dDKI5ndXHTYFDc7cdoHOokgWUQmwaO8i1EWYh 5VLO0OiOEWkRJh+fK6GHIyrU56D/ME3jlifW X-Google-Smtp-Source: APBJJlGMoGTZrbru1fVb9JslthbAiF2wsIl2HsR1Hk3nalG2jl6ctXDHaNYmeI9gMwOHtknP/aKqVV0pkkfu/CUf X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a5b:b8b:0:b0:c4c:f97e:421a with SMTP id l11-20020a5b0b8b000000b00c4cf97e421amr39950ybq.4.1689836920855; Thu, 20 Jul 2023 00:08:40 -0700 (PDT) Date: Thu, 20 Jul 2023 07:08:24 +0000 In-Reply-To: <20230720070825.992023-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230720070825.992023-1-yosryahmed@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230720070825.992023-8-yosryahmed@google.com> Subject: [RFC PATCH 7/8] memcg: add sysctl and config option to control memory recharging From: Yosry Ahmed To: Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt Cc: Muchun Song , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Yu Zhao , Luis Chamberlain , Kees Cook , Iurii Zaikin , "T.J. Mercier" , Greg Thelen , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: B28821C001D X-Rspam-User: X-Stat-Signature: j9aosmdmnpn6a77ssw778tocyk79pk6r X-Rspamd-Server: rspam01 X-HE-Tag: 1689836921-993582 X-HE-Meta: U2FsdGVkX1+XaNBZvg5S4eckjThk8sglxMRGVCFp9Fy8CiUHvfjUwjN8JcxbiO10sTLg6a/bEJA93NIJ6e3XsrvacjoTr2tkQkHplZJ0TFSFDLtmE3yhH8CUPYSbdzDY8WwMFNoalumnDYKT8EHHDgkn236ulwcjwyrXCyoLe2TL05dlHNBHf2Pfzxa16VicN4ajaIerFuwUCuFpqX8CNzfpUloZp6RshPtbzv1ZyInOek5I84uVGB/jkBnjb5aUJ8Zl/ssXQ4dBPsI0gghn3ZaqfM2LbttFCO8prkDLHSdmeN+tGuALApSPpYfjYTQmbFi9xdZkozQiYpyT0lLbCbu0FGCBEpt4pB0lEYUwN4Yl/QrJ7vZ2rB8GK8u4/n0+MXORDgmmt1ucv4jB3Fk2IHeF1M2HYRIUyfd32QbgUWlQkxzzpTKJzM4WcyZY6eKQUJmpT5fAyN8xagxpQytowTW6uPIAeLxVvEtCzD2uBRhV94kcMYwtLfWXipIo0SgGoB4ct3u6ec9de2dUNqN8Eu83R6SFMRwchhBQt0TvDtmJJ02hJAtltv/4SNKychakCQTIVE65SMXdyN1tU3gYyFR4hFUhqn1AJnu/g8ctjIu14NxIWjgK8j6oMGYDl+Oys867df1u92LBi37Qme+xy8HhBq+ZrLKcSzgxDZ/sTuwBgSQds08dcT044HpzdRCsGLmTYe7F+tOtBfj4kc507nZ3KWETPdVoCU+PhihOLy4/A+t2WgDOxYbuLRcDK/1QS7Fai6fhx9TDMmQPZOfHG3KoXdotc/VIX/j40J8vGgWw7raWpPySEgyhUvCmSm3hyUtQrT+MsVhbtORGEFfFKSvq5ZaLW6ZMPl0SBKBqlAhaJoCG9wBIafDpQaNQ/Pti7MNh+r40kNdrvKV+3jp9OlPOPBlvN3719cUVTqGSKGxKw2CFUuKlMMPkKBai7IvuMJXZMoZ5X0V7QlqVxsI eIifyX59 cs1Vw5rhZFcPcaX0JM20zKiQY+C3CJr36apIhfVRl/UfqMqxiQuwYu0V9hejjn19BCI4BNlLhWECzxiGIIKHKIC3bSqcYz59Ody/oXf3dpcRwGzJALmhT4Gp2Lp/RRRkdUFDoOdDBvAF0Ft78nlG9S9sN3btR0Ai8n0yyUHEMkiU9GPZq3B4k1CnOYiX+6FlpxoRtxYJVenfVKXTCewnC6/vEZH49qa/1pMrqHJwzJ5lpeJ9YrC1NzMxYSvRQU/0j3+ASsmwEAWyHyGtNHXid/ZGsvnf+EP1g3t36yMJw/oTZuJKttmg3j09Pou+osgyMe4lv6iPSc6kN8CTF1OBEKZkNC1O/jqkO6f3d7rVu9a42g3oO8tQJE787jpYVznPZ+WtDFGHjbQwuHGevc9TyEZyJuo3FvQKNwKoJ6rZrXgZOd4FO20UqhXZVJ0wnYP55nv7ppdECnmTWk2sqwWuxCj6bCLMm4pp96mgMdRQzIsdWruB7ZjwWg+pxKJ6Xv/olPOMbUwObv98Y6YPKJci+WnV71Ng2cLMjdAl7LGMAkFAzyV2i5XqKo0X3KVV4B+8W+wC6c+1cAUm+niDYGw2ojLw7Cfo2P3GaLz3HdIpQQz59ybl/mGdo+3dwb2y7RIiQotIpMHIAyiKIdwk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a sysctl to enable/disable memory recharging for offline memcgs. Add a config option to control whether or not it is enabled by default. Signed-off-by: Yosry Ahmed --- include/linux/memcontrol.h | 2 ++ kernel/sysctl.c | 11 +++++++++++ mm/Kconfig | 12 ++++++++++++ mm/memcontrol.c | 9 ++++++++- 4 files changed, 33 insertions(+), 1 deletion(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 59b653d4a76e..ae9f09ee90cb 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -60,6 +60,8 @@ struct mem_cgroup_reclaim_cookie { #ifdef CONFIG_MEMCG +extern int sysctl_recharge_offline_memcgs; + #define MEM_CGROUP_ID_SHIFT 16 #define MEM_CGROUP_ID_MAX USHRT_MAX diff --git a/kernel/sysctl.c b/kernel/sysctl.c index 354a2d294f52..1735d1d95652 100644 --- a/kernel/sysctl.c +++ b/kernel/sysctl.c @@ -2249,6 +2249,17 @@ static struct ctl_table vm_table[] = { .extra2 = (void *)&mmap_rnd_compat_bits_max, }, #endif +#ifdef CONFIG_MEMCG + { + .procname = "recharge_offline_memcgs", + .data = &sysctl_recharge_offline_memcgs, + .maxlen = sizeof(sysctl_recharge_offline_memcgs), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = SYSCTL_ONE, + }, +#endif /* CONFIG_MEMCG */ { } }; diff --git a/mm/Kconfig b/mm/Kconfig index 09130434e30d..9462c4b598d9 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1236,6 +1236,18 @@ config LOCK_MM_AND_FIND_VMA bool depends on !STACK_GROWSUP +config MEMCG_RECHARGE_OFFLINE_ENABLED + bool "Recharge memory charged to offline memcgs" + depends on MEMCG + help + When a memory cgroup is removed by userspace, try to recharge any + memory still charged to it to avoid having it live on as an offline + memcg. Offline memcgs potentially consume memory and limit scalability + of some operations. + + This option enables the above behavior by default. It can be override + at runtime through /proc/sys/vm/recharge_offline_memcgs. + source "mm/damon/Kconfig" endmenu diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 2fe9c6f1be80..25cdb17eaaa3 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -96,6 +96,9 @@ static bool cgroup_memory_nobpf __ro_after_init; static DECLARE_WAIT_QUEUE_HEAD(memcg_cgwb_frn_waitq); #endif +int sysctl_recharge_offline_memcgs __read_mostly = IS_ENABLED( + CONFIG_MEMCG_RECHARGE_OFFLINE_ENABLED); + static struct workqueue_struct *memcg_recharge_wq; /* Whether legacy memory+swap accounting is active */ @@ -6592,7 +6595,8 @@ static void memcg_recharge_mapped_folios(struct mem_cgroup *memcg) INIT_DELAYED_WORK(&memcg->recharge_mapped_work.dwork, memcg_do_recharge_mapped_folios); - if (memcg_recharge_wq && memcg_nr_local_mapped_pages(memcg)) { + if (sysctl_recharge_offline_memcgs && + memcg_recharge_wq && memcg_nr_local_mapped_pages(memcg)) { memcg->recharge_mapped_work.retries = 0; queue_delayed_work(memcg_recharge_wq, &memcg->recharge_mapped_work.dwork, 0); @@ -6605,6 +6609,9 @@ static bool should_do_deferred_recharge(struct folio *folio) struct mem_cgroup *memcg; bool ret; + if (!sysctl_recharge_offline_memcgs) + return false; + rcu_read_lock(); memcg = folio_memcg_rcu(folio); ret = memcg && !!(memcg->css.flags & CSS_DYING); From patchwork Thu Jul 20 07:08:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13319872 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2BABEB64DC for ; Thu, 20 Jul 2023 07:08:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 944512800C3; Thu, 20 Jul 2023 03:08:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8097528004C; Thu, 20 Jul 2023 03:08:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 65C252800C3; Thu, 20 Jul 2023 03:08:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 4257828004C for ; Thu, 20 Jul 2023 03:08:45 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 171DAA0180 for ; Thu, 20 Jul 2023 07:08:45 +0000 (UTC) X-FDA: 81031112610.05.536A0D7 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf07.hostedemail.com (Postfix) with ESMTP id 3B2A64001D for ; Thu, 20 Jul 2023 07:08:43 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=byOeZzDw; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 3et24ZAoKCEU5vzy5hotlknvvnsl.jvtspu14-ttr2hjr.vyn@flex--yosryahmed.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3et24ZAoKCEU5vzy5hotlknvvnsl.jvtspu14-ttr2hjr.vyn@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689836923; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VyMIM9+tcYiRGMTsBGjvi0vU3Mwws8djANktEDDglwA=; b=emVdC7yGj2CHgY7JWn2UXgGkOehOXTvsJqCSwQZfq/GnaUrTsMDu1anDTA++yZfJQHjpVg JE64LjSIUF7ibqhpvZe/aqLeyRCCvUEO6CB31xTCCS4dMVnVXZI/qh0IEk0UerY7tpnARy QlpFo3mQ7yHAz7YhndksXsY82yDlAr8= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=byOeZzDw; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 3et24ZAoKCEU5vzy5hotlknvvnsl.jvtspu14-ttr2hjr.vyn@flex--yosryahmed.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3et24ZAoKCEU5vzy5hotlknvvnsl.jvtspu14-ttr2hjr.vyn@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689836923; a=rsa-sha256; cv=none; b=mGHW+A/CLPmHgNvdFv9xc0QFXx5DMICBYDzY/orkyhd0Ai+zaSrdDoiz+TgDZGAws2oM0w mUCBOhda4E8NEhnbMwayuIsw/c0KjsmImZEB3vwuqu2QuaWPLzMkzpit+2zgVwg5wLIeOi LfoXti6L9t4/p3GN/SYTSzb6NeBr3R0= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-cfdebfe9c14so327062276.3 for ; Thu, 20 Jul 2023 00:08:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689836922; x=1692428922; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=VyMIM9+tcYiRGMTsBGjvi0vU3Mwws8djANktEDDglwA=; b=byOeZzDwSe+TlCjYRKBoXqqWywqR6C1yG/A40dOXCXNDbvhvWImB25mBWajRKzlpsd 7pw0yGkSnP78uE/qIJ91xiFtjZ8WsotdIkdUAUo9pCwMIjy6NapBzMWDaD5jSQ5WRKqG Se0fctUKe/3aEu5ubRL9pul76zHCCtINkNBEyKURGH2aFpknJ55HycotBhqzgpzYPQKE UJtv9JLEiNXgfyyCUh834Zn94be/WG5kodBP7zmNrBOk5cu/mZ3dShiUkQYtjSSYO1Rs 4gRgPSThLVz2L65vzOc7OySwGSHuufaREX+F7eqZX0F2Ko4WmwxK9yLR5upAijtlIKWO +cGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689836922; x=1692428922; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VyMIM9+tcYiRGMTsBGjvi0vU3Mwws8djANktEDDglwA=; b=AIoiGScqYWZdYaIeoBAEaXROjiPaxsRcj+UIs941p8BDew7bUBgEyG+U9sbJxU7kCF aERz/e5wzQqhCYSvnkgzTARYZZzbthWgmzqqK2iXxQPY1ym8ZfywqKCk1Frd4Yu/LIBn z8Y73ZqR98FmxAuoMK6iNqEHyXSHJexufycetTq4bJmV2FlifvIcC1Hm9S2CILlzVBMg JrtYYOL395Od3ujvUoJo+MHYemrBrAztF72oSmroyN581lmtwHJktJxTeHhfgTOm9275 mLN9LcPFgeISTnKCbuBsoQessb/u6f2tDlMjqGZLr/AUF4ErrfqZ7s9+ic8gDzpgAsDk d/Sw== X-Gm-Message-State: ABy/qLZRJ4y1GJ9SunY1FwFy6jlw1wkvYKoE16ag8eBGnNtkgiXLBKaY TrWKq9W/RzBrFOOuK5W42opFcP0M/x0h/UtZ X-Google-Smtp-Source: APBJJlGnnUAk0gRTSVnJDIUOzDDJab7OFbTWyp59V1sIEHIaW1b9MjwRCYZFrCmirYEKnPFJ8IXeXLXbdqJdCfyP X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:1709:b0:cab:e42c:876b with SMTP id by9-20020a056902170900b00cabe42c876bmr40935ybb.3.1689836922401; Thu, 20 Jul 2023 00:08:42 -0700 (PDT) Date: Thu, 20 Jul 2023 07:08:25 +0000 In-Reply-To: <20230720070825.992023-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230720070825.992023-1-yosryahmed@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230720070825.992023-9-yosryahmed@google.com> Subject: [RFC PATCH 8/8] selftests: cgroup: test_memcontrol: add a selftest for memcg recharging From: Yosry Ahmed To: Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt Cc: Muchun Song , "Matthew Wilcox (Oracle)" , Tejun Heo , Zefan Li , Yu Zhao , Luis Chamberlain , Kees Cook , Iurii Zaikin , "T.J. Mercier" , Greg Thelen , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: 3B2A64001D X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: yh3xxa8cpcc7xtg8w45mijrgbznfwt8c X-HE-Tag: 1689836923-692455 X-HE-Meta: U2FsdGVkX18dlYhRR/BZjYd/VDzgafbaiiFGXk28ARbAblq4M6Z+/g1QjODWDsNo8z0/a2QX94SPIMll/XUij9ssqobzz3+IdwWE+cL6xkp3AkN8kVlAT+qvYD6I/eGnIl+DSl0MdVXBEqwMxO47HAVw2IMuoS4UmTRH3cpNLmDQjkk8gSKNn0zBVTlgsI/5mmYG2XsLPJY+ToZ7QH1Jj7CtCTjCOlv9bD826hlM9xTUiZAos7AAC7i7Pl5nCzFEAUHOq9Y1Zj6wA/swv1p4WqlFcxsRWQNtqBpc7Kyyp+oobvF5F1XjuyfKYeegxjzJ/wWwKfO2jiOj/fEgRlt8o9ps5dAxJamMuaklrWOvpbpHml76utBGvu5zjBeHCWHd4UjzUYw0F5GRsr6kKL1OHYL/CQFWYZKYajx9RCkBe04MoAYKOvCG1Mopa9ow4D9G6Oz1r063E0JVReofDkRrsUbG3dvWUf/voAkxftgloi12nWjvY0JHqVuJ393k/iJxdnanYhmpBeq6kP/0aPbORsmyvhf3rOd/MwyR24YA/FBlhCYWd3S+615/97Vs7VkbbMKqgpQFxskFD3AEBpjYuTiJ5OTfo7MzHtcmXYI/0jA2vZjSe/OxdGvavgRiVz7GKHlkztdTrzkOc6/cK+xxhzBtrLgbPkAgXEfE2T4Y2zthOCKO59lSGfPm89bvL3+gHeElod/CGhV0yQjAE/Cqa/QEAVLMcwSPY1IXTgfq4N1wJYEdwX9V3us/Mk2JFKZlZ0Snwp0fH8+q5S8warSnLtY4B4dxz3/Bpu5i8PmdglcA2KAG7Rm7KQ18UyUtxKf2bcuvy+RoMGjDkc7Mxf6Etkv7KQJRlltml9n5I65apBddeUC+/z4eIbcJJCK+0Lchx1p1opACDFgDt7Hdv8IAl9/qrBHLvF4BizLVwuasFhMftWewPIiumOoGXc3Y05AIr3DZUyPz2oKFlrWGpVx kf5azP38 7iyIAg7+w7rsnt/K14KF0z0p40pdNUEgsrCwxsEXV9kfi/ZMGOd7dZBdrGniIPwHMyt4lwDDPtnYpWV/kujWuON54GpA3zUu8vBi+Oh+8bRu5g6CFn1hSnkJjMOAGDES0Qbj3JdpTqhxNFmAlF6zoJdbL0Yjjak00gvp2bkCCyWimczIgwB242EINUzi7qr8rwv7TJGjEvmvd2AlyiI8REq48BxE7wHYB+LtorGPfWePcEGJqmfuIN3D+bJeVRF9Wr5h8cj2ni0G9YBNR29OCqJYV5b9rMwFujIbGCRxaKj6J7loOcAi5gEMyhdpn8c28tjA+fOP6SCuO2zoCCFFeHx32HeEo5rumYM5BHsS3fUk2ZFjFyZ0Ijde6SVHAlWUCMqX+q4m41dcYmf6o8K/GdrMo4xqQSNTXe8ymqMNaDxC+b3VnWXefglybfj2bi/km7YY4Ql4RB45Sx8WBM9qB4MaN6hNvbgrGQPkCJJSOQgOn0uvjJrlFZ+9zaPSeAa5qEI8aGG01AGWLC5JhjKcBgtC4vrfYLVI6Kt06Z1rXWsJvEfyriquKKF+opSVjJW035WyFlEkD2+i8OwndfeCyieLAuxTVYnFfIZHYFzRGsgOFw4aulpSRs0ZVmhDYP4SmbR/pGh6Fa+RPe0e2QksiRQ87+hgHp7DD9R7/9bxDpmCpTItpyGr7fb5WrA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.007979, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a memcg is removed, any mapped pages charged to it are recharged to the memcg of the process(es) mapping them. Any remaining pages are recharged using deferred recharge on the next time they are accessed or ditied. Add a selftest that exercises these paths for shmem and normal files: - A page is recharged on offlining if it is already mapped into the address space of a process in a different memcg. - A page is recharged after offlining when written to by a process in a different memcg (if the write results in dirtying the page). - A page is recharged after offlining when read by a process in a different memcg. - A page is recharged after offlining when mapped by a process in a different memcg. Signed-off-by: Yosry Ahmed --- tools/testing/selftests/cgroup/cgroup_util.c | 14 + tools/testing/selftests/cgroup/cgroup_util.h | 1 + .../selftests/cgroup/test_memcontrol.c | 310 ++++++++++++++++++ 3 files changed, 325 insertions(+) diff --git a/tools/testing/selftests/cgroup/cgroup_util.c b/tools/testing/selftests/cgroup/cgroup_util.c index e8bbbdb77e0d..e853b2a4db77 100644 --- a/tools/testing/selftests/cgroup/cgroup_util.c +++ b/tools/testing/selftests/cgroup/cgroup_util.c @@ -519,6 +519,20 @@ int is_swap_enabled(void) return cnt > 1; } + +int is_memcg_recharging_enabled(void) +{ + char buf[10]; + bool enabled; + + if (read_text("/proc/sys/vm/recharge_offline_memcgs", + buf, sizeof(buf)) <= 0) + return -1; + + enabled = strtol(buf, NULL, 10); + return enabled; +} + int set_oom_adj_score(int pid, int score) { char path[PATH_MAX]; diff --git a/tools/testing/selftests/cgroup/cgroup_util.h b/tools/testing/selftests/cgroup/cgroup_util.h index c92df4e5d395..10c0fa36bfd7 100644 --- a/tools/testing/selftests/cgroup/cgroup_util.h +++ b/tools/testing/selftests/cgroup/cgroup_util.h @@ -49,6 +49,7 @@ extern int get_temp_fd(void); extern int alloc_pagecache(int fd, size_t size); extern int alloc_anon(const char *cgroup, void *arg); extern int is_swap_enabled(void); +extern int is_memcg_recharging_enabled(void); extern int set_oom_adj_score(int pid, int score); extern int cg_wait_for_proc_count(const char *cgroup, int count); extern int cg_killall(const char *cgroup); diff --git a/tools/testing/selftests/cgroup/test_memcontrol.c b/tools/testing/selftests/cgroup/test_memcontrol.c index c7c9572003a8..4e1ea93e0a54 100644 --- a/tools/testing/selftests/cgroup/test_memcontrol.c +++ b/tools/testing/selftests/cgroup/test_memcontrol.c @@ -17,6 +17,8 @@ #include #include #include +#include +#include #include "../kselftest.h" #include "cgroup_util.h" @@ -1287,6 +1289,313 @@ static int test_memcg_oom_group_score_events(const char *root) return ret; } +/* Map 50M from the beginning of a file */ +static int map_fd_50M_noexit(const char *cgroup, void *arg) +{ + size_t size = MB(50); + int ppid = getppid(); + int fd = (long)arg; + char *memory; + + memory = mmap(NULL, size, PROT_READ, MAP_SHARED | MAP_POPULATE, fd, 0); + if (memory == MAP_FAILED) { + fprintf(stderr, "error: mmap, errno %d\n", errno); + return -1; + } + + while (getppid() == ppid) + sleep(1); + + munmap(memory, size); + return 0; +} + +/* + * Write 50M to the beginning of a file. + * The file is sync'ed first to make sure any dirty pages are laundered before + * we dirty them again. + */ +static int write_fd_50M(const char *cgroup, void *arg) +{ + size_t size = MB(50); + int fd = (long)arg; + char buf[PAGE_SIZE]; + int i; + + fsync(fd); + lseek(fd, 0, SEEK_SET); + for (i = 0; i < size; i += sizeof(buf)) + write(fd, buf, sizeof(buf)); + + return 0; +} + +/* See write_fd_50M() */ +static int write_fd_50M_noexit(const char *cgroup, void *arg) +{ + int ppid = getppid(); + + write_fd_50M(cgroup, arg); + + while (getppid() == ppid) + sleep(1); + + return 0; +} + +/* Read 50M from the beginning of a file */ +static int read_fd_50M_noexit(const char *cgroup, void *arg) +{ + size_t size = MB(50); + int ppid = getppid(); + int fd = (long)arg; + char buf[PAGE_SIZE]; + int i; + + lseek(fd, 0, SEEK_SET); + for (i = 0; i < size; i += sizeof(buf)) + read(fd, buf, sizeof(buf)); + + while (getppid() == ppid) + sleep(1); + + return 0; +} + +#define TEST_RECHARGE_DIR "/test-recharge" + +static int __test_memcg_recharge(const char *root, char *stat_name) +{ + char *parent = NULL, *child1 = NULL, *child2 = NULL; + long stat, prev, pstat, current; + int ret = KSFT_FAIL; + char file_path[256]; + int i, pid; + struct { + int fd; + int (*before_fn)(const char *cgroup, void *arg); + int (*after_fn)(const char *cgroup, void *arg); + } test_files[] = { + /* test recharge for already mapped file */ + { + .before_fn = map_fd_50M_noexit, + }, + /* test recharge on new mapping after offline */ + { + .after_fn = map_fd_50M_noexit, + }, + /* test recharge on write after offline */ + { + .after_fn = write_fd_50M_noexit, + }, + /* test recharge on read after offline */ + { + .after_fn = read_fd_50M_noexit, + } + }; + + parent = cg_name(root, "parent"); + if (!parent) + goto cleanup; + + if (cg_create(parent)) + goto cleanup; + + if (cg_write(parent, "cgroup.subtree_control", "+memory")) + goto cleanup; + + child1 = cg_name(parent, "child1"); + if (!child1) + goto cleanup; + + if (cg_create(child1)) + goto cleanup; + + child2 = cg_name(parent, "child2"); + if (!child2) + goto cleanup; + + if (cg_create(child2)) + goto cleanup; + + for (i = 0; i < ARRAY_SIZE(test_files); i++) { + long target = MB(50) * (i+1); /* 50MB per file */ + int fd; + + snprintf(file_path, sizeof(file_path), "%s/file%d", + TEST_RECHARGE_DIR, i); + + fd = open(file_path, O_CREAT | O_RDWR); + if (fd < 0) + goto cleanup; + + test_files[i].fd = fd; + if (cg_run(child1, write_fd_50M, (void *)(long) fd)) + goto cleanup; + + stat = 0; + do { + sleep(1); + prev = stat; + stat = cg_read_key_long(child1, "memory.stat", + stat_name); + } while (stat < target && stat > prev); + + if (stat < target) { + fprintf(stderr, "error: child1 %s %ld < %ld", + stat_name, stat, target); + goto cleanup; + } + + current = cg_read_long(child1, "memory.current"); + if (current < target) { + fprintf(stderr, "error: child1 current %ld < %ld", + current, target); + goto cleanup; + } + + if (test_files[i].before_fn) { + pid = cg_run_nowait(child2, test_files[i].before_fn, + (void *)(long)fd); + if (pid < 0) + goto cleanup; + /* make sure before_fn() finishes executing before offlining */ + sleep(1); + } + } + + current = cg_read_long(child2, "memory.current"); + if (current > MB(1)) { + fprintf(stderr, "error: child2 current %ld > 1M\n", current); + goto cleanup; + } + + stat = cg_read_key_long(child2, "memory.stat", stat_name); + if (stat > 0) { + fprintf(stderr, "error: child2 %s %ld > 0\n", + stat_name, stat); + goto cleanup; + } + + if (cg_destroy(child1) < 0) + goto cleanup; + + for (i = 0; i < ARRAY_SIZE(test_files); i++) { + long target = MB(50) * (i+1); + int fd = test_files[i].fd; + + if (test_files[i].after_fn) { + pid = cg_run_nowait(child2, test_files[i].after_fn, + (void *)(long)fd); + if (pid < 0) + goto cleanup; + } + + stat = 0; + do { + sleep(1); + prev = stat; + stat = cg_read_key_long(child2, "memory.stat", + stat_name); + } while (stat < target && stat > prev); + + if (stat < target) { + fprintf(stderr, "error: child2 %s %ld < %ld\n", + stat_name, stat, target); + goto cleanup; + } + + current = cg_read_long(child2, "memory.current"); + if (current < target) { + fprintf(stderr, "error: child2 current %ld < %ld\n", + current, target); + goto cleanup; + } + } + + pstat = cg_read_key_long(parent, "memory.stat", stat_name); + if (stat < pstat) { + fprintf(stderr, "error: recharged %s (%ld) < total (%ld)\n", + stat_name, stat, pstat); + goto cleanup; + } + + ret = KSFT_PASS; +cleanup: + if (child2) { + cg_destroy(child2); + free(child2); + } + if (child1) { + cg_destroy(child1); + free(child1); + } + if (parent) { + cg_destroy(parent); + free(parent); + } + for (i = 0; i < ARRAY_SIZE(test_files); i++) { + close(test_files[i].fd); + snprintf(file_path, sizeof(file_path), "%s/file%d", + TEST_RECHARGE_DIR, i); + remove(file_path); + } + return ret; +} + +static int test_memcg_recharge(const char *root) +{ + int i, ret = KSFT_PASS; + struct { + char *mount_type, *stat_name; + } test_setups[] = { + /* test both shmem & normal files */ + { + .mount_type = "tmpfs", + .stat_name = "shmem", + }, + { + .stat_name = "file", + } + }; + + if (!is_memcg_recharging_enabled()) + return KSFT_SKIP; + + if (unshare(CLONE_NEWNS) < 0) + return KSFT_FAIL; + + if (mount(NULL, "/", "", MS_REC | MS_PRIVATE, NULL) < 0) + return KSFT_FAIL; + + for (i = 0; i < ARRAY_SIZE(test_setups); i++) { + int setup_ret = KSFT_FAIL; + char *mount_type = test_setups[i].mount_type; + char *stat_name = test_setups[i].stat_name; + + if (mkdir(TEST_RECHARGE_DIR, 0777) < 0) + goto next; + + if (mount_type && + mount(NULL, TEST_RECHARGE_DIR, mount_type, 0, NULL) < 0) + goto next; + + setup_ret = __test_memcg_recharge(root, stat_name); + +next: + if (mount_type) + umount(TEST_RECHARGE_DIR); + remove(TEST_RECHARGE_DIR); + + if (setup_ret == KSFT_FAIL) { + ret = KSFT_FAIL; + break; + } + } + umount("/"); + return ret; +} + #define T(x) { x, #x } struct memcg_test { int (*fn)(const char *root); @@ -1306,6 +1615,7 @@ struct memcg_test { T(test_memcg_oom_group_leaf_events), T(test_memcg_oom_group_parent_events), T(test_memcg_oom_group_score_events), + T(test_memcg_recharge), }; #undef T