From patchwork Fri Sep 20 22:11:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kaiyang Zhao X-Patchwork-Id: 13808711 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A161CF9C6B for ; Fri, 20 Sep 2024 22:12:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8F26E6B0082; Fri, 20 Sep 2024 18:12:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 87BFC6B0085; Fri, 20 Sep 2024 18:12:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 71C436B0088; Fri, 20 Sep 2024 18:12:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 569956B0082 for ; Fri, 20 Sep 2024 18:12:49 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 017EC1406BC for ; Fri, 20 Sep 2024 22:12:48 +0000 (UTC) X-FDA: 82586517258.21.248FD3F Received: from mail-qv1-f51.google.com (mail-qv1-f51.google.com [209.85.219.51]) by imf12.hostedemail.com (Postfix) with ESMTP id 3AE4D4000C for ; Fri, 20 Sep 2024 22:12:47 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=cs.cmu.edu header.s=google-2021 header.b=XyHY4HRR; dmarc=pass (policy=none) header.from=cs.cmu.edu; spf=pass (imf12.hostedemail.com: domain of kaiyang2@andrew.cmu.edu designates 209.85.219.51 as permitted sender) smtp.mailfrom=kaiyang2@andrew.cmu.edu ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726870282; a=rsa-sha256; cv=none; b=AdJO/PLUl5oI7uz9suR50tJ8TDlga4zomj2QC8hgDJqeVeVEvZBKHg2Gc8TFY2wIUu1JMF 982f2eubvBsg67OzxCJ2WlHdLLsTg5BePJ5ai9k7xkQoeemXueuAGcTOUK624ytHLFZv9u 9UOsiun6nkIz41j+L3mu1Mn9iNUklFY= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=cs.cmu.edu header.s=google-2021 header.b=XyHY4HRR; dmarc=pass (policy=none) header.from=cs.cmu.edu; spf=pass (imf12.hostedemail.com: domain of kaiyang2@andrew.cmu.edu designates 209.85.219.51 as permitted sender) smtp.mailfrom=kaiyang2@andrew.cmu.edu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726870282; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mjQIYISgv8bXMzl4SQq6QWJhA5lMgQlc/Hdxb7kdcNw=; b=z2owilUMkoF+nYEOwB5QYl7AE8DC3qzoX/OVYr3ZkGyLI+x67DNFI+fFq1Or7AYOQP/QZQ O2pR8Kf8LllOUuLICeIoGLebaCmgFYKevoCQqTSQsMfpoGOGrDBQLy6hZfleSOt5DKjqHl 15yZdchUumBx85JcPHTF5vDbutqQS9M= Received: by mail-qv1-f51.google.com with SMTP id 6a1803df08f44-6c35357cdacso17239326d6.0 for ; Fri, 20 Sep 2024 15:12:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.cmu.edu; s=google-2021; t=1726870366; x=1727475166; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mjQIYISgv8bXMzl4SQq6QWJhA5lMgQlc/Hdxb7kdcNw=; b=XyHY4HRRpbrLrAl+aNc8S1kO6UEep46/f6DZxqoGpCkt84M+pOWKNbYTqdirDd/ZCe BHthymhF3Jl/3HUb3rlOh+X3VyT+CiFTXeUdO+STLoShigLP4M4RJkp+jtVNBG9xokuv I8gbm73svG9zLBINO91Nz96PwLJepEqgaatUJryh6Brep+YXOkzKy1njcqKyTKQ+p1IX gueo9TrBnOktZE1aR0ufFcGZ3+zTSxsKtQJbt+82YnRlyzXNQ0u5ICKsRhFw/b1xfjb/ H+wvZLuQfFG/ckoyGdN5OkTZBmm6KqBc9ekPI31H3w2OSo0AL1+ePshsEOSC6i8AnL4L aueQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726870366; x=1727475166; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mjQIYISgv8bXMzl4SQq6QWJhA5lMgQlc/Hdxb7kdcNw=; b=DntuObMtJyIY5fCdVXcVvAHcN+L7bnr74mUqvqL6C0aU0CFGyIWYwCtiJO7k2rPG7B YyPC9bBOOPwo7q/fYkZbq9xn88vqts/mR1IklOrgiodnLLWZAxlj5sJ2jkEsXSKwQDYX TwtUTzmjnkirVBDjSIE6562NUxE8z5c8NLLS8vl/PCmhMt/6N8/RtA61dtUwxFbrSp0F eteGF41HcKikFBhnCd26ooZxE9vgmvAHWvOGd35iRAO8MlW5at6VheSoavMYbZ5BrF/0 dn2fdAU2et1CrIh3p+/WB6bNnEswyEaCuaRnXpP7v+2Gu/eK8cyBJG4M+di7ew/TKUT+ Z6tg== X-Gm-Message-State: AOJu0Yyvf3ZEQx2nhfSWBMolHkgGzIzbVzHIACITaRsg0lmcjbhtxH3m JytJEmmGDXCcS3N46bnRMuUZAnAYiPDhQ0oWDriv76Us11S8hAMTb98zw6TCr92lfXra/W2hknm rdEvQOb6nHKcG5KUEmQImkueGaALn2w/PSccYPNKwwx2uXuPSrGQ+4E+B6wV6W4xkN9JxIS8pPb 6DDXDWmXgWYLawLyPz691CmuEMkGBMjvd0LLX30w== X-Google-Smtp-Source: AGHT+IES2HVe0X98tZeGiUhTQNrbSdJWRIaPaFJBiEj5ef4ucxT4/xboRx0SNgqoj2W0gV0qAMe+fw== X-Received: by 2002:a0c:fac3:0:b0:6c7:c658:5f40 with SMTP id 6a1803df08f44-6c7c658610bmr26601826d6.14.1726870365846; Fri, 20 Sep 2024 15:12:45 -0700 (PDT) Received: from localhost (pool-74-98-231-160.pitbpa.fios.verizon.net. [74.98.231.160]) by smtp.gmail.com with UTF8SMTPSA id 6a1803df08f44-6c75e557c01sm22904026d6.101.2024.09.20.15.12.44 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 20 Sep 2024 15:12:45 -0700 (PDT) From: kaiyang2@cs.cmu.edu To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, mhocko@kernel.org, nehagholkar@meta.com, abhishekd@meta.com, hannes@cmpxchg.org, weixugc@google.com, rientjes@google.com, Kaiyang Zhao Subject: [RFC PATCH 1/4] Add get_cgroup_local_usage for estimating the top-tier memory usage Date: Fri, 20 Sep 2024 22:11:48 +0000 Message-ID: <20240920221202.1734227-2-kaiyang2@cs.cmu.edu> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240920221202.1734227-1-kaiyang2@cs.cmu.edu> References: <20240920221202.1734227-1-kaiyang2@cs.cmu.edu> MIME-Version: 1.0 X-Rspamd-Queue-Id: 3AE4D4000C X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 4scesxsnzyanogqwfgasdjixs45tdk6i X-HE-Tag: 1726870367-282275 X-HE-Meta: U2FsdGVkX18+D1kv2FEOgR1/GOlcZt4nw9JC26oyxC/hU08tMQOB64U7EnrePlHKD0/hng34Pv1Oq3g1o2M+GU6rLgvpLaXrizcqFSoF7ISPMwOBiq1gsg7GsTa2zgYT7NCEEU6OwNY+ZoGUheBPMPEh1LvV16m2658sRRJlJm02VUCpnmMixTkzsakqBcH8HRQRpGZKlcplzYhs4YQ0q4T6oUblo/Me1bPjHpfaRx6j7zepBZB5vUBD9IH6r9pT14Fcq8o5UR4GEpJaqNeZi6236Uqrm+szrpHHbxl1B3kGVXLmMW6xrnjxFbTfLmgYOTR3qX6Dg8Lx2Xy2JPwdOFn5kHO7JNWS/ZZ06C/aMCEXwF2+TNTL5roHtOdd0rA+SnPmQc9busC7aviR6Gzypo/1l5XoLPWEIWU7f8CKaCCi6IcEjFWt3xR8aeFSgv/y9V6AG1Uz6Khdnyll7jABx2IbrpYJoQbaxf+/bwXNcNPxSm5wPZGJ4bTMazkHaCt407NiUIlfThOjf0q5ycXvQMG3GSKffjJ6o73Atgek/B6EyfiHN7ONBT+BN3+sAfC8iIo5uWFNRcwkCGTnhXT1d7M87na8lH/CvxSCPOb5Zy3fq7tJeVdRwVPNABMVDYdIsTvvzHGSs1VfCSRe+4SNn+3VatNbUbtudyecJgEm02rg0qbsjVMhU5sAkzW4f5kJJ6iuCs6xYD/UVZLnTkGLcwmqYV53X7KAxLCoLz438gVAsGzJS0LACeiiVMIdiXYAtbZC3uE+nAXbsKKw9B0IekgpqVxW+71Lf8fuX5X19OeSbsH85p1ghpq8t06tNnTRk95DoU/XJQfOqA0fkOjCQ4pQOIq7aszR/rMhvKZf5m3kOkdHu3oEvyGUMT8vd+ueylF1/gVo6reH6VTXqsn3aW3SfPW0k9EJYIogCgfo6jQKD+3H+s8H/VVCIFgw8VjoAAtVgxZW9EMhaEQBg4/ qPGuBTEI +iBsUjQPCJhzhIWkODefPV97lPKTmWi+2nS/nK7MrmLduCOtjIFKoqFQ2CxbZAO/efnQ/DoMztZmynq+8/88utyXC3ld+EtN4aW9D/xKl+7kblyuSVXyZSB857DbSB6f/z9tBUPCP9X18+2Imdr9nvU16r0UFcHEPgt93i/LgQajThIlX53I0cUi1KXAiYGw/kME24n9Tww24uylI7Hjz+8cLeENlkcdep/HmD5sQMVPTCzrGlDlVT+MlkQl9BLxcartzbP4SCASpLvQG8giuaw/WUJswYfpRrAxqMk4VK0pGjAVt2ZEYw3XK8R5iFJq2Uz1SvpS5Tc6nKycDtgJCgJgX0T7gK8xd9P5/ho21I0vnEIoh9oR1FWgW+NCxr5s8c4a2DR43rhoQJtTqbjRpCKDD5hCajTXI7edCowxlQ4pdPkvhgOOqPirbLw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.002013, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kaiyang Zhao Approximate the usage of top-tier memory of a cgroup by its anon, file, shmem and slab sizes in the top-tier. Signed-off-by: Kaiyang Zhao --- include/linux/memcontrol.h | 2 ++ mm/memcontrol.c | 24 ++++++++++++++++++++++++ 2 files changed, 26 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 34d2da05f2f1..94aba4498fca 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -648,6 +648,8 @@ static inline bool mem_cgroup_unprotected(struct mem_cgroup *target, memcg == target; } +unsigned long get_cgroup_local_usage(struct mem_cgroup *memcg, bool flush); + static inline bool mem_cgroup_below_low(struct mem_cgroup *target, struct mem_cgroup *memcg) { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f19a58c252f0..20b715441332 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -855,6 +855,30 @@ unsigned long memcg_events_local(struct mem_cgroup *memcg, int event) return READ_ONCE(memcg->vmstats->events_local[i]); } +/* Usage is in pages. */ +unsigned long get_cgroup_local_usage(struct mem_cgroup *memcg, bool flush) +{ + struct lruvec *lruvec; + const int local_nid = 0; + + if (!memcg) + return 0; + + if (flush) + mem_cgroup_flush_stats_ratelimited(memcg); + + lruvec = mem_cgroup_lruvec(memcg, NODE_DATA(local_nid)); + unsigned long anon = lruvec_page_state(lruvec, NR_ANON_MAPPED); + unsigned long file = lruvec_page_state(lruvec, NR_FILE_PAGES); + unsigned long shmem = lruvec_page_state(lruvec, NR_SHMEM); + /* Slab size are in bytes */ + unsigned long slab = + lruvec_page_state(lruvec, NR_SLAB_RECLAIMABLE_B) / PAGE_SIZE + + lruvec_page_state(lruvec, NR_SLAB_UNRECLAIMABLE_B) / PAGE_SIZE; + + return anon + file + shmem + slab; +} + struct mem_cgroup *mem_cgroup_from_task(struct task_struct *p) { /* From patchwork Fri Sep 20 22:11:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kaiyang Zhao X-Patchwork-Id: 13808712 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41332CF9C68 for ; Fri, 20 Sep 2024 22:12:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8A6BB6B0088; Fri, 20 Sep 2024 18:12:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 856346B0089; Fri, 20 Sep 2024 18:12:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5E4E46B008A; Fri, 20 Sep 2024 18:12:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 360216B0088 for ; Fri, 20 Sep 2024 18:12:51 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id DC56C1C4AC2 for ; Fri, 20 Sep 2024 22:12:50 +0000 (UTC) X-FDA: 82586517300.19.089B57F Received: from mail-qv1-f42.google.com (mail-qv1-f42.google.com [209.85.219.42]) by imf21.hostedemail.com (Postfix) with ESMTP id 1AA0D1C000A for ; Fri, 20 Sep 2024 22:12:48 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=cs.cmu.edu header.s=google-2021 header.b=c91fx0Y7; dmarc=pass (policy=none) header.from=cs.cmu.edu; spf=pass (imf21.hostedemail.com: domain of kaiyang2@andrew.cmu.edu designates 209.85.219.42 as permitted sender) smtp.mailfrom=kaiyang2@andrew.cmu.edu ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726870275; a=rsa-sha256; cv=none; b=WEnb5sL2qHN7vyoVTlwWpiees14xFMIWwmRNmelczc/OD7/NufyRNLdjOoUHk+zRsX0pPk IHN+vbWtcQ1HiD8NqW+enCofn5mvndbGBrEwx6EWjHwvUFEr90dTN7dwl3Sqln7J/aBc/s sk1yKq9JUSGJ5qWQL1ackyMPM689R40= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=cs.cmu.edu header.s=google-2021 header.b=c91fx0Y7; dmarc=pass (policy=none) header.from=cs.cmu.edu; spf=pass (imf21.hostedemail.com: domain of kaiyang2@andrew.cmu.edu designates 209.85.219.42 as permitted sender) smtp.mailfrom=kaiyang2@andrew.cmu.edu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726870275; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uf3jgwkumzU+UPaFgIeEZ3O7SSYAkqNrO+ZAvwxaUDo=; b=WCFwfNKUxmtmpu0g6ANrG2zI34TRZOmC+6CG+xrdmITyDdERDd8kH8bNqd0kHV5X+b8BF4 xG7F3TSgnXoei95yI2fg79/abeO38bmXbGzdsgne1fou+NsgyM46zYb1SaVupHsQ+Mw73H bSBbn003xJSGC7i/S+lQ5S6tMgFrmYs= Received: by mail-qv1-f42.google.com with SMTP id 6a1803df08f44-6c34dd6c21aso15934706d6.2 for ; Fri, 20 Sep 2024 15:12:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.cmu.edu; s=google-2021; t=1726870368; x=1727475168; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uf3jgwkumzU+UPaFgIeEZ3O7SSYAkqNrO+ZAvwxaUDo=; b=c91fx0Y7mcCTt54D79SGCPYyOK25ZORlBT9jxBwE0kdLXcTruc5MCn0PzGRNwscaCH YHofiUkEkARfUOr3OfZVKtf30vU5ci5sNxXbHaWa/v/yRpI6DQO/bucrnHWIDVdUPIXZ 7lfoc7wpcUaHDsNx5JgjdURUlKLaPxTu13oH4tJYvmJWjQ8E/dDqjo+Fgy+jAxZa7jrx rgt+sGlasv9Y3rj+t0rL7bQfkyBQAV0zKP93NgMdC3m5khiOiFo/TOsz71Ec10r+hif1 IzGn2eH2RSYqL1I+hhfEa50nPf3aKRXE/Y56PGF1kXxm6ksGYNIfAvJsaFeXzb+N4Mfa 0aEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726870368; x=1727475168; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uf3jgwkumzU+UPaFgIeEZ3O7SSYAkqNrO+ZAvwxaUDo=; b=NNo84Y9qHlK077dKYoy6oHgFGloov5j8cktsTBXDPNGblNH1tbmf6TmATKpBNxgTiN T9nR5oCYGw7VUhgYAnoeG8HcKHw33odBeyEDUNvWKa5a+GgznhfUKU4Oze1u7inAGfY4 Yk3sGhcWYXyw1crDlb0RsTBKNGTlFIpVv6a5iQaM3plQs2ITmU17z/w5JbBEiDC0q/Jh rv2/Bi7M/wCO8CCCVdz5C9f7W5ZiR+cZYjERk/UJEnfFopaKXfuA3pOetN+9tu6bBO70 oH6l/BkOVuzyEpe7CNfFPAWsNLoijiYfv6DUyiP0miwSuo34Cb6koq+AwXihCtKfxzKC qlXw== X-Gm-Message-State: AOJu0YwFEQQFeLKzVa5OBbiXG8zt+5KqWdUZi0EXjPew+2u0+U7qX0I+ YEd0k1HaJGWyHJp7EzIct7FW5t0+kQc0v0Ugj6/6yjE0S6c1/gg0X9phOetx3JqGZUGa9sxNEHi TfVXyVjz7izP1JNqTWQygN3RvRhrqHUZMjzpJT96DwEXMlkg3wMGBU1Neel3d1ZNZTN5SPCX/Q9 Uo/XuLq02+J/aW6cP5VJXGVdr05U/PypXMDkhWYQ== X-Google-Smtp-Source: AGHT+IEyE6YRvGHqfK+QKR9XTMQex28jIRLr/J4ytHxbuqKZ8XfKVCXNbZWUk9cnyNyCrbUZvhYABQ== X-Received: by 2002:a05:6214:5781:b0:6c3:5496:3e06 with SMTP id 6a1803df08f44-6c7bc6a4c39mr80018546d6.10.1726870367936; Fri, 20 Sep 2024 15:12:47 -0700 (PDT) Received: from localhost (pool-74-98-231-160.pitbpa.fios.verizon.net. [74.98.231.160]) by smtp.gmail.com with UTF8SMTPSA id 6a1803df08f44-6c75e57a4acsm23134796d6.116.2024.09.20.15.12.47 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 20 Sep 2024 15:12:47 -0700 (PDT) From: kaiyang2@cs.cmu.edu To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, mhocko@kernel.org, nehagholkar@meta.com, abhishekd@meta.com, hannes@cmpxchg.org, weixugc@google.com, rientjes@google.com, Kaiyang Zhao Subject: [RFC PATCH 2/4] calculate memory.low for the local node and track its usage Date: Fri, 20 Sep 2024 22:11:49 +0000 Message-ID: <20240920221202.1734227-3-kaiyang2@cs.cmu.edu> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240920221202.1734227-1-kaiyang2@cs.cmu.edu> References: <20240920221202.1734227-1-kaiyang2@cs.cmu.edu> MIME-Version: 1.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 1AA0D1C000A X-Stat-Signature: oise978btf918z9rr9hjrw7gw9fgo6ju X-Rspam-User: X-HE-Tag: 1726870368-923418 X-HE-Meta: U2FsdGVkX19LMFrYU9yiJq49dCYYf7pR4THFNMkOa2HBAjyH6xChKPkS3qDmliemBpfAThBqyTQd+4ZSbBTckoSEi4wzhvhx8KPwtOfqEqFQm+UNULxbYen6knD7hDOvg2oeCPYg4+jqJs19g9gBxoREn6JnMLwlePIv3DfprMOLZks2n8YzdfxWsu0FmEtUiDwb7QAlKqxo9JKQbaQ7io7rIk2eVJMm3LIsJl0tE97auiXLYnf8laVJGJngmQBgDZE94DP4wYasU9FjPYyLJr8YfsJqd8VL+OZAYy43J+GpACwO2XdPdIXjro9okf2B/p46oPSJnKL06NzH33i+5I/hJpzQ44Vybfrg+3l/Y2dZyg+y2wms7MwCfPLX0h+07BsD2G0NDSUj+DsSfXfJwpHVag3iaeussFVLhtDpy9/sH4ilYO5RCAPjA4djW4yKSY7d+8nIUyhhoNgq5lOcSMsicwg6SEZseWME2qYuZbAVLPCQtTaXq/i5c/E2g53Lp6JU17FDPJanUTHDtDA+pnRSXfiMVggKZ4aNc6Riu3Cw1IRRGhOeG+e7H5TO/XJBOm7/Y/p2ftL8+2xRh3Cl9Lfu2IIOg1WCmE5Znm3+EUF1cPE8byAlqfl4CswGF2vXdxITO60RpxSeCns5ooCvUbxWNxNkUeXC9/12HLl0Jz8BPaptBXmgwrrQMbuSr9/dy3xM8l5YhUKWIG7eosdAlsqDxx+8b7H64VvQDaUbLYHW1HoKACEi63+qaQNjGq+7VFdUqmtvlp4n6akRM71cPmLWS1dR/h7xzg6Wzb2JKruRacTbmu0fFf2L8Nlzmdw97d2s/unCBvx4qDReudJsR+GJlwIBB2nhT1S7t2kXMW5ksjLM2ZLqckPHbWC0wxHFOZKr3XMZfigVFyePsTxYS4lJLQuNGflSL2e7R3z8v4vWSERN4aqaSArQTD9kowD5MBNpV17dm2r+LGeJjUl yQVT3bDL g62GPzE7nuoFIwt/clsnXvqSaI/yp51FNhLbQYLnB1NA7HehulUsIwgJRt9SKiCaTAK3PxvBd0AFz4qRrOEfPLDJFI8pwbchstCrViRnFs6jCf6b9fTXNcjYOrDRCGLeBk6jZXVTZg7aybZXg5sczaIGy8YrVuK+aeU1VbnnH44pOHhf5die11ePdvab2Oo9eeUw7j4+d92CFwAMg2yzM6sRHWA5gYJ9qFtJjnOUW1XlYQfOHQcBMK2VRKUtULmgGHUR/nl7ybQ2qD2jNWdGnv5SzWcCSOrXmMU/uuf9QVaXZOsoHWMFTgyV+uKsklOicy1iJ3aujpTD06fk3yF0nvOEgkrGhHxNOL2xiDzTDxXCCehOz5Sr5dGWSBjUmrRreuAn5uDjRLTVm28dfXs3Rj5C+roDehQCk/lFa8XF6nzCO5Ks= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kaiyang Zhao Add a memory.low for the top-tier node (locallow) and track its usage. locallow is set by scaling low by the ratio of node 0 capacity and node 0 + node 1 capacity. Signed-off-by: Kaiyang Zhao --- include/linux/page_counter.h | 16 ++++++++--- mm/hugetlb_cgroup.c | 4 +-- mm/memcontrol.c | 42 ++++++++++++++++++++++------- mm/page_counter.c | 52 ++++++++++++++++++++++++++++-------- 4 files changed, 88 insertions(+), 26 deletions(-) diff --git a/include/linux/page_counter.h b/include/linux/page_counter.h index 79dbd8bc35a7..aa56c93415ef 100644 --- a/include/linux/page_counter.h +++ b/include/linux/page_counter.h @@ -13,6 +13,7 @@ struct page_counter { * memcg->memory.usage is a hot member of struct mem_cgroup. */ atomic_long_t usage; + struct mem_cgroup *memcg; /* memcg that owns this counter */ CACHELINE_PADDING(_pad1_); /* effective memory.min and memory.min usage tracking */ @@ -25,6 +26,10 @@ struct page_counter { atomic_long_t low_usage; atomic_long_t children_low_usage; + unsigned long elocallow; + atomic_long_t locallow_usage; + atomic_long_t children_locallow_usage; + unsigned long watermark; /* Latest cg2 reset watermark */ unsigned long local_watermark; @@ -36,6 +41,7 @@ struct page_counter { bool protection_support; unsigned long min; unsigned long low; + unsigned long locallow; unsigned long high; unsigned long max; struct page_counter *parent; @@ -52,12 +58,13 @@ struct page_counter { */ static inline void page_counter_init(struct page_counter *counter, struct page_counter *parent, - bool protection_support) + bool protection_support, struct mem_cgroup *memcg) { counter->usage = (atomic_long_t)ATOMIC_LONG_INIT(0); counter->max = PAGE_COUNTER_MAX; counter->parent = parent; counter->protection_support = protection_support; + counter->memcg = memcg; } static inline unsigned long page_counter_read(struct page_counter *counter) @@ -72,7 +79,8 @@ bool page_counter_try_charge(struct page_counter *counter, struct page_counter **fail); void page_counter_uncharge(struct page_counter *counter, unsigned long nr_pages); void page_counter_set_min(struct page_counter *counter, unsigned long nr_pages); -void page_counter_set_low(struct page_counter *counter, unsigned long nr_pages); +void page_counter_set_low(struct page_counter *counter, unsigned long nr_pages, + unsigned long nr_pages_local); static inline void page_counter_set_high(struct page_counter *counter, unsigned long nr_pages) @@ -99,11 +107,11 @@ static inline void page_counter_reset_watermark(struct page_counter *counter) #ifdef CONFIG_MEMCG void page_counter_calculate_protection(struct page_counter *root, struct page_counter *counter, - bool recursive_protection); + bool recursive_protection, int is_local); #else static inline void page_counter_calculate_protection(struct page_counter *root, struct page_counter *counter, - bool recursive_protection) {} + bool recursive_protection, int is_local) {} #endif #endif /* _LINUX_PAGE_COUNTER_H */ diff --git a/mm/hugetlb_cgroup.c b/mm/hugetlb_cgroup.c index d8d0e665caed..0e07a7a1d5b8 100644 --- a/mm/hugetlb_cgroup.c +++ b/mm/hugetlb_cgroup.c @@ -114,10 +114,10 @@ static void hugetlb_cgroup_init(struct hugetlb_cgroup *h_cgroup, } page_counter_init(hugetlb_cgroup_counter_from_cgroup(h_cgroup, idx), - fault_parent, false); + fault_parent, false, NULL); page_counter_init( hugetlb_cgroup_counter_from_cgroup_rsvd(h_cgroup, idx), - rsvd_parent, false); + rsvd_parent, false, NULL); limit = round_down(PAGE_COUNTER_MAX, pages_per_huge_page(&hstates[idx])); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 20b715441332..d7c5fff12105 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1497,6 +1497,9 @@ static void memcg_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) vm_event_name(memcg_vm_event_stat[i]), memcg_events(memcg, memcg_vm_event_stat[i])); } + + seq_buf_printf(s, "local_usage %lu\n", + get_cgroup_local_usage(memcg, true)); } static void memory_stat_format(struct mem_cgroup *memcg, struct seq_buf *s) @@ -3597,8 +3600,8 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) if (parent) { WRITE_ONCE(memcg->swappiness, mem_cgroup_swappiness(parent)); - page_counter_init(&memcg->memory, &parent->memory, true); - page_counter_init(&memcg->swap, &parent->swap, false); + page_counter_init(&memcg->memory, &parent->memory, true, memcg); + page_counter_init(&memcg->swap, &parent->swap, false, NULL); #ifdef CONFIG_MEMCG_V1 WRITE_ONCE(memcg->oom_kill_disable, READ_ONCE(parent->oom_kill_disable)); page_counter_init(&memcg->kmem, &parent->kmem, false); @@ -3607,8 +3610,8 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css) } else { init_memcg_stats(); init_memcg_events(); - page_counter_init(&memcg->memory, NULL, true); - page_counter_init(&memcg->swap, NULL, false); + page_counter_init(&memcg->memory, NULL, true, memcg); + page_counter_init(&memcg->swap, NULL, false, NULL); #ifdef CONFIG_MEMCG_V1 page_counter_init(&memcg->kmem, NULL, false); page_counter_init(&memcg->tcpmem, NULL, false); @@ -3677,7 +3680,7 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) memcg1_css_offline(memcg); page_counter_set_min(&memcg->memory, 0); - page_counter_set_low(&memcg->memory, 0); + page_counter_set_low(&memcg->memory, 0, 0); zswap_memcg_offline_cleanup(memcg); @@ -3748,7 +3751,7 @@ static void mem_cgroup_css_reset(struct cgroup_subsys_state *css) page_counter_set_max(&memcg->tcpmem, PAGE_COUNTER_MAX); #endif page_counter_set_min(&memcg->memory, 0); - page_counter_set_low(&memcg->memory, 0); + page_counter_set_low(&memcg->memory, 0, 0); page_counter_set_high(&memcg->memory, PAGE_COUNTER_MAX); memcg1_soft_limit_reset(memcg); page_counter_set_high(&memcg->swap, PAGE_COUNTER_MAX); @@ -4051,6 +4054,12 @@ static ssize_t memory_min_write(struct kernfs_open_file *of, return nbytes; } +static int memory_locallow_show(struct seq_file *m, void *v) +{ + return seq_puts_memcg_tunable(m, + READ_ONCE(mem_cgroup_from_seq(m)->memory.locallow)); +} + static int memory_low_show(struct seq_file *m, void *v) { return seq_puts_memcg_tunable(m, @@ -4061,7 +4070,8 @@ static ssize_t memory_low_write(struct kernfs_open_file *of, char *buf, size_t nbytes, loff_t off) { struct mem_cgroup *memcg = mem_cgroup_from_css(of_css(of)); - unsigned long low; + struct sysinfo si; + unsigned long low, locallow, local_capacity, total_capacity; int err; buf = strstrip(buf); @@ -4069,7 +4079,15 @@ static ssize_t memory_low_write(struct kernfs_open_file *of, if (err) return err; - page_counter_set_low(&memcg->memory, low); + /* Hardcoded 0 for local node and 1 for remote. */ + si_meminfo_node(&si, 0); + local_capacity = si.totalram; /* In pages. */ + total_capacity = local_capacity; + si_meminfo_node(&si, 1); + total_capacity += si.totalram; + locallow = low * local_capacity / total_capacity; + + page_counter_set_low(&memcg->memory, low, locallow); return nbytes; } @@ -4394,6 +4412,11 @@ static struct cftype memory_files[] = { .seq_show = memory_low_show, .write = memory_low_write, }, + { + .name = "locallow", + .flags = CFTYPE_NOT_ON_ROOT, + .seq_show = memory_locallow_show, + }, { .name = "high", .flags = CFTYPE_NOT_ON_ROOT, @@ -4483,7 +4506,8 @@ void mem_cgroup_calculate_protection(struct mem_cgroup *root, if (!root) root = root_mem_cgroup; - page_counter_calculate_protection(&root->memory, &memcg->memory, recursive_protection); + page_counter_calculate_protection(&root->memory, &memcg->memory, + recursive_protection, false); } static int charge_memcg(struct folio *folio, struct mem_cgroup *memcg, diff --git a/mm/page_counter.c b/mm/page_counter.c index b249d15af9dd..97205aafab46 100644 --- a/mm/page_counter.c +++ b/mm/page_counter.c @@ -18,8 +18,10 @@ static bool track_protection(struct page_counter *c) return c->protection_support; } +extern unsigned long get_cgroup_local_usage(struct mem_cgroup *memcg, bool flush); + static void propagate_protected_usage(struct page_counter *c, - unsigned long usage) + unsigned long usage, unsigned long local_usage) { unsigned long protected, old_protected; long delta; @@ -44,6 +46,15 @@ static void propagate_protected_usage(struct page_counter *c, if (delta) atomic_long_add(delta, &c->parent->children_low_usage); } + + protected = min(local_usage, READ_ONCE(c->locallow)); + old_protected = atomic_long_read(&c->locallow_usage); + if (protected != old_protected) { + old_protected = atomic_long_xchg(&c->locallow_usage, protected); + delta = protected - old_protected; + if (delta) + atomic_long_add(delta, &c->parent->children_locallow_usage); + } } /** @@ -63,7 +74,8 @@ void page_counter_cancel(struct page_counter *counter, unsigned long nr_pages) atomic_long_set(&counter->usage, new); } if (track_protection(counter)) - propagate_protected_usage(counter, new); + propagate_protected_usage(counter, new, + get_cgroup_local_usage(counter->memcg, false)); } /** @@ -83,7 +95,8 @@ void page_counter_charge(struct page_counter *counter, unsigned long nr_pages) new = atomic_long_add_return(nr_pages, &c->usage); if (protection) - propagate_protected_usage(c, new); + propagate_protected_usage(c, new, + get_cgroup_local_usage(counter->memcg, false)); /* * This is indeed racy, but we can live with some * inaccuracy in the watermark. @@ -151,7 +164,8 @@ bool page_counter_try_charge(struct page_counter *counter, goto failed; } if (protection) - propagate_protected_usage(c, new); + propagate_protected_usage(c, new, + get_cgroup_local_usage(counter->memcg, false)); /* see comment on page_counter_charge */ if (new > READ_ONCE(c->local_watermark)) { @@ -238,7 +252,8 @@ void page_counter_set_min(struct page_counter *counter, unsigned long nr_pages) WRITE_ONCE(counter->min, nr_pages); for (c = counter; c; c = c->parent) - propagate_protected_usage(c, atomic_long_read(&c->usage)); + propagate_protected_usage(c, atomic_long_read(&c->usage), + get_cgroup_local_usage(counter->memcg, false)); } /** @@ -248,14 +263,17 @@ void page_counter_set_min(struct page_counter *counter, unsigned long nr_pages) * * The caller must serialize invocations on the same counter. */ -void page_counter_set_low(struct page_counter *counter, unsigned long nr_pages) +void page_counter_set_low(struct page_counter *counter, unsigned long nr_pages, + unsigned long nr_pages_local) { struct page_counter *c; WRITE_ONCE(counter->low, nr_pages); + WRITE_ONCE(counter->locallow, nr_pages_local); for (c = counter; c; c = c->parent) - propagate_protected_usage(c, atomic_long_read(&c->usage)); + propagate_protected_usage(c, atomic_long_read(&c->usage), + get_cgroup_local_usage(counter->memcg, false)); } /** @@ -421,9 +439,9 @@ static unsigned long effective_protection(unsigned long usage, */ void page_counter_calculate_protection(struct page_counter *root, struct page_counter *counter, - bool recursive_protection) + bool recursive_protection, int is_local) { - unsigned long usage, parent_usage; + unsigned long usage, parent_usage, local_usage, parent_local_usage; struct page_counter *parent = counter->parent; /* @@ -437,16 +455,19 @@ void page_counter_calculate_protection(struct page_counter *root, return; usage = page_counter_read(counter); - if (!usage) + local_usage = get_cgroup_local_usage(counter->memcg, true); + if (!usage || !local_usage) return; if (parent == root) { counter->emin = READ_ONCE(counter->min); counter->elow = READ_ONCE(counter->low); + counter->elocallow = READ_ONCE(counter->locallow); return; } parent_usage = page_counter_read(parent); + parent_local_usage = get_cgroup_local_usage(parent->memcg, true); WRITE_ONCE(counter->emin, effective_protection(usage, parent_usage, READ_ONCE(counter->min), @@ -454,7 +475,16 @@ void page_counter_calculate_protection(struct page_counter *root, atomic_long_read(&parent->children_min_usage), recursive_protection)); - WRITE_ONCE(counter->elow, effective_protection(usage, parent_usage, + if (is_local) + WRITE_ONCE(counter->elocallow, + effective_protection(local_usage, parent_local_usage, + READ_ONCE(counter->locallow), + READ_ONCE(parent->elocallow), + atomic_long_read(&parent->children_locallow_usage), + recursive_protection)); + else + WRITE_ONCE(counter->elow, + effective_protection(usage, parent_usage, READ_ONCE(counter->low), READ_ONCE(parent->elow), atomic_long_read(&parent->children_low_usage), From patchwork Fri Sep 20 22:11:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kaiyang Zhao X-Patchwork-Id: 13808713 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C6DBCF9C69 for ; Fri, 20 Sep 2024 22:12:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 090B56B008C; Fri, 20 Sep 2024 18:12:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 03F126B0092; Fri, 20 Sep 2024 18:12:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DFBA86B0093; Fri, 20 Sep 2024 18:12:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id BE7746B008C for ; Fri, 20 Sep 2024 18:12:55 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5B4D4C06E7 for ; Fri, 20 Sep 2024 22:12:55 +0000 (UTC) X-FDA: 82586517510.18.19B4B34 Received: from mail-qv1-f53.google.com (mail-qv1-f53.google.com [209.85.219.53]) by imf22.hostedemail.com (Postfix) with ESMTP id 6E9BDC0007 for ; Fri, 20 Sep 2024 22:12:53 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=cs.cmu.edu header.s=google-2021 header.b=UnAhzWel; spf=pass (imf22.hostedemail.com: domain of kaiyang2@andrew.cmu.edu designates 209.85.219.53 as permitted sender) smtp.mailfrom=kaiyang2@andrew.cmu.edu; dmarc=pass (policy=none) header.from=cs.cmu.edu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726870258; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dua3EzvkneWM3TAvRq2/3u51msVEEUUg6Zhqmm/qfGM=; b=3Oa6LKQfcsEW+XH2uUMxVBZI/md/rX5gQzfppM2HY3Otm5wcDD32SUlB1XA399iHwMfekH xEJs+u4WiV/W5htQdPXZ1RvMAB3/pktNTB43Bh8FUR3V9k1TE4rIlXLO8zrLL97jhqXKLW Jfg/XFKy6A2gN2Iy9e/8qenslEx0NfU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726870258; a=rsa-sha256; cv=none; b=VQekYlAHTSTQMRo+ZTHWb829ptNO8+BFthDMftZBDwPGIp/mU75zc07T3/T+9ylfsCtbPH cY12jkEQgKxKxU9d/vr/7FF4dt85U8hqq6K5rza2Cpt4SVHTXSu2lPFn8SgpMm101kCgYr mGnzLiVZQ7f9cBy/QIvoO4e7U9DMR+c= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=cs.cmu.edu header.s=google-2021 header.b=UnAhzWel; spf=pass (imf22.hostedemail.com: domain of kaiyang2@andrew.cmu.edu designates 209.85.219.53 as permitted sender) smtp.mailfrom=kaiyang2@andrew.cmu.edu; dmarc=pass (policy=none) header.from=cs.cmu.edu Received: by mail-qv1-f53.google.com with SMTP id 6a1803df08f44-6c34c02ff1cso15810566d6.2 for ; Fri, 20 Sep 2024 15:12:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.cmu.edu; s=google-2021; t=1726870372; x=1727475172; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dua3EzvkneWM3TAvRq2/3u51msVEEUUg6Zhqmm/qfGM=; b=UnAhzWela+QLM4lEg4U+dD50PEXA0sTNAK+vnJTjlW2iy1I8a4NS02yaiyvhDeP0dm wYqKAJ5e2x+FB6p8cxLb5FG94LY1Vn22qAoRypiUl7EiWIdToL67fA4beVBiMGmkCcLK 2I0DB4ytCuqd6wSxb9p+eQx4L0+8lEOUHM96iviClQvcnI6SRS/irC4nA3CAivI5CeMX uIfhZwY1txgnLD4waZD598BPW5qq6MXypRS40b9KLpV7rEhQ8C65UijsCuI6hGqRH5l1 NGzL+ZzvlGo2xfOWGTV6z3CgPldJJ3Yzf7Z/65vPXaXWHrmT/KXWSFqxkdhQuKbM3s3J f6OA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726870372; x=1727475172; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dua3EzvkneWM3TAvRq2/3u51msVEEUUg6Zhqmm/qfGM=; b=p3kcoO5uqiaLmwvkbj4xfT6kFv9ss7uDxi7Ae1VdqE9/CXHnWeFQqVjudLHQ1CYthR fBUfub0eyGyLrWk0/TB0Eden+UarTSJkyaU4wRHjZ9vVSU/LAZgiiQKvyZ8grwB7lj6+ ffo0mvHYysDjTw5p3QfyLMvpbD4jIEssJE/atkgQrz3gZhfPu+3nRtQ4J+Z/awUMfeFS eHjJbHCZPNBrXa3hxNE6SOJWiLT4juushD+FpKwG/y1tDHRHoZpFg1tFBuLQXqUf6/Uw TjZ7jAO3lLsnuXPr4fSp2EaQ00bizPyl+dIQ9dimLDMY208XRZ19h9+riPMbMB7bVQv1 gaew== X-Gm-Message-State: AOJu0YwJ8S0l8QXTz8k74KsJsVPw5/qkD0uN0arbMvnCly9wXUHqwE/t 9nKLK0GDOJ7nXKv4N8jmGi9zWAfHjC6DUFPAd0HzBT4ena2y0Uey5X9gexhwwHP6qtx8kvMhn5M eKp10jym50KhAUusrx1x+A46Fyu6G1pTHjSESkFQHLEWaoNLo1CDTzz+qgIH7kp0dqHHPgBx6u1 qLSSLOGziVmQW/E1YLbzLWI+0Q3JOL8EIKjNkyjA== X-Google-Smtp-Source: AGHT+IFt0V61eP797QLmbHgiSqqaJMC8ZiLd28uIJxGM3kNR5lwGLx+JECFMYHwxRLmmCyhdNYC+KA== X-Received: by 2002:a05:6214:5c4a:b0:6c5:32a5:567a with SMTP id 6a1803df08f44-6c7bd486c6bmr56855866d6.1.1726870372363; Fri, 20 Sep 2024 15:12:52 -0700 (PDT) Received: from localhost (pool-74-98-231-160.pitbpa.fios.verizon.net. [74.98.231.160]) by smtp.gmail.com with UTF8SMTPSA id 6a1803df08f44-6c75e557e58sm23236106d6.81.2024.09.20.15.12.51 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 20 Sep 2024 15:12:52 -0700 (PDT) From: kaiyang2@cs.cmu.edu To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, mhocko@kernel.org, nehagholkar@meta.com, abhishekd@meta.com, hannes@cmpxchg.org, weixugc@google.com, rientjes@google.com, Kaiyang Zhao Subject: [RFC PATCH 3/4] use memory.low local node protection for local node reclaim Date: Fri, 20 Sep 2024 22:11:50 +0000 Message-ID: <20240920221202.1734227-4-kaiyang2@cs.cmu.edu> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240920221202.1734227-1-kaiyang2@cs.cmu.edu> References: <20240920221202.1734227-1-kaiyang2@cs.cmu.edu> MIME-Version: 1.0 X-Stat-Signature: rxerhrq7fbcd5socaf1ky1s9runfeike X-Rspamd-Queue-Id: 6E9BDC0007 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1726870373-121524 X-HE-Meta: U2FsdGVkX18kJ9v+2rOElhLkfMySS+KYl0uKPaXLNs8UkUi4XTkr+mKMj5Q/ophuiWrc+TNdHeWTXsj3B06tyjrY1JTUmC+WahcuvnNbapLTOIYMXDulzszGW3Hc1Rjdd5/j8JYRgKqkL/It0+dDnvoQrKCD+pq2CsEzXegY0OFaiYHf4TCwTMhQ6/86/ohbuqGLCVxkY40PhZZiSKLvhK4Upywso6T9h7T0MGME9C2G/v0mISEsRofITNa6dCfFFag3yuOBRjtL3KrZsFfn0MhMHBve6swdvOEBBEyuPb2niLFOVMT/0DyknQd0XG8p4f5aCbXHadwRjs2E4fAh86e21GnA8zOIHONBIswfMokkrFhlIdkLTIbbeGZDFTthOx5BaC1q0e28hKuDmH/tZG+ZU3Rd2BAwjM9KR2UlG2Ye0DBG04VXZ4YLz1plN5DLT7Nxo/m06/4R8dmJW7kL+1BmMXOBIhWX7PE9HpcJWpwSj8TGIjm6uKZf0+i0Ng/9sAn+/IEESp3ZxP91KYg9ZhcsMEzw/sTqEVyaBS8/xKJ5m4SxsJnojgb5+WvF57gLfl7L6xVC6j4SYORkGZ3FSqea052K97HHnGuqQvyrf0Y7v+gHLL+wdOhc6euGytX8ylpTaO7W871oZsAypWrTYt9KsU38KU/DcaO3sqWJ8KeaGax+M+wTe1sIbUf11Ga5BIfJ7iTkdrzDYml4Oi9K/P+qtx5KrPxsULt2+UmK5i0d4k2Z3bkjvudnWtxcyqBu7g2pZQtC8wV68mRG1EM1h9jAZ7a4d97Br5HE38H7vTiogzwqwH+7PgaEpXAmjW1FbmScd7cO1XKTfk/ntsXcn1fIH6oUY9EvoP4ZuyHK2JtjzqzwMgE5IjkeLPp9p28wGMQ77wiz00hF5Z/dWF54olMazojTNB/lckmbXWrcfvDTLgvE5wt3HjrmOpxqQ12tKSxbfCwdGIQImVofivd Xt9FGNcl i0pGPqbc3J49EtfMnMRiUAxK8YWWTx46+CsG9sP6HBATUjwVYAVFa5ZIONzCWl/T7Al38IiTl7YtS+jpuNVZ7RDS1YgOaHP7RIzGKRvJ6aqkiYOOwLKwqjdt59E6vYyEMspQ+rCYJiaRUOWfT0aBLIIef7t1CAKxaV0yziRN5jTGU4L4eXOKRrpmhP+OEGYkxbj2EK8TbBLEYCuP/iKNKVa0o1cjfpnrIGu7ds7/1B3RN/sYgm5Ph4hhQm5nqndmW3qx4qAWaQ0NpiS63KTnGk5Q4LwchC4BUnmFk1I+wuedRwn6K/FR5mhNvBS/BSRF8g1Hczehou1DQCh42duo9LJTl59tvnC2Z7pk4oZQ1Gu+5G0/fwWdPqN8QaAfCcEO4Uu/YbP5WxbHoKD4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kaiyang Zhao When reclaim targets the top-tier node usage by the root memcg, apply local memory.low protection instead of global protection. Signed-off-by: Kaiyang Zhao --- include/linux/memcontrol.h | 23 ++++++++++++++--------- mm/memcontrol.c | 4 ++-- mm/vmscan.c | 19 ++++++++++++++----- 3 files changed, 30 insertions(+), 16 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 94aba4498fca..256912b91922 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -586,9 +586,9 @@ static inline bool mem_cgroup_disabled(void) static inline void mem_cgroup_protection(struct mem_cgroup *root, struct mem_cgroup *memcg, unsigned long *min, - unsigned long *low) + unsigned long *low, unsigned long *locallow) { - *min = *low = 0; + *min = *low = *locallow = 0; if (mem_cgroup_disabled()) return; @@ -631,10 +631,11 @@ static inline void mem_cgroup_protection(struct mem_cgroup *root, *min = READ_ONCE(memcg->memory.emin); *low = READ_ONCE(memcg->memory.elow); + *locallow = READ_ONCE(memcg->memory.elocallow); } void mem_cgroup_calculate_protection(struct mem_cgroup *root, - struct mem_cgroup *memcg); + struct mem_cgroup *memcg, int is_local); static inline bool mem_cgroup_unprotected(struct mem_cgroup *target, struct mem_cgroup *memcg) @@ -651,13 +652,17 @@ static inline bool mem_cgroup_unprotected(struct mem_cgroup *target, unsigned long get_cgroup_local_usage(struct mem_cgroup *memcg, bool flush); static inline bool mem_cgroup_below_low(struct mem_cgroup *target, - struct mem_cgroup *memcg) + struct mem_cgroup *memcg, int is_local) { if (mem_cgroup_unprotected(target, memcg)) return false; - return READ_ONCE(memcg->memory.elow) >= - page_counter_read(&memcg->memory); + if (is_local) + return READ_ONCE(memcg->memory.elocallow) >= + get_cgroup_local_usage(memcg, true); + else + return READ_ONCE(memcg->memory.elow) >= + page_counter_read(&memcg->memory); } static inline bool mem_cgroup_below_min(struct mem_cgroup *target, @@ -1159,13 +1164,13 @@ static inline void memcg_memory_event_mm(struct mm_struct *mm, static inline void mem_cgroup_protection(struct mem_cgroup *root, struct mem_cgroup *memcg, unsigned long *min, - unsigned long *low) + unsigned long *low, unsigned long *locallow) { *min = *low = 0; } static inline void mem_cgroup_calculate_protection(struct mem_cgroup *root, - struct mem_cgroup *memcg) + struct mem_cgroup *memcg, int is_local) { } @@ -1175,7 +1180,7 @@ static inline bool mem_cgroup_unprotected(struct mem_cgroup *target, return true; } static inline bool mem_cgroup_below_low(struct mem_cgroup *target, - struct mem_cgroup *memcg) + struct mem_cgroup *memcg, int is_local) { return false; } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index d7c5fff12105..61718ba998fe 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4495,7 +4495,7 @@ struct cgroup_subsys memory_cgrp_subsys = { * of a top-down tree iteration, not for isolated queries. */ void mem_cgroup_calculate_protection(struct mem_cgroup *root, - struct mem_cgroup *memcg) + struct mem_cgroup *memcg, int is_local) { bool recursive_protection = cgrp_dfl_root.flags & CGRP_ROOT_MEMORY_RECURSIVE_PROT; @@ -4507,7 +4507,7 @@ void mem_cgroup_calculate_protection(struct mem_cgroup *root, root = root_mem_cgroup; page_counter_calculate_protection(&root->memory, &memcg->memory, - recursive_protection, false); + recursive_protection, is_local); } static int charge_memcg(struct folio *folio, struct mem_cgroup *memcg, diff --git a/mm/vmscan.c b/mm/vmscan.c index ce471d686a88..a2681d52fc5f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2377,6 +2377,7 @@ static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc, enum scan_balance scan_balance; unsigned long ap, fp; enum lru_list lru; + int is_local = (pgdat->node_id == 0) && root_reclaim(sc); /* If we have no swap space, do not bother scanning anon folios. */ if (!sc->may_swap || !can_reclaim_anon_pages(memcg, pgdat->node_id, sc)) { @@ -2457,12 +2458,14 @@ static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc, for_each_evictable_lru(lru) { bool file = is_file_lru(lru); unsigned long lruvec_size; - unsigned long low, min; + unsigned long low, min, locallow; unsigned long scan; lruvec_size = lruvec_lru_size(lruvec, lru, sc->reclaim_idx); mem_cgroup_protection(sc->target_mem_cgroup, memcg, - &min, &low); + &min, &low, &locallow); + if (is_local) + low = locallow; if (min || low) { /* @@ -2494,7 +2497,12 @@ static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc, * again by how much of the total memory used is under * hard protection. */ - unsigned long cgroup_size = mem_cgroup_size(memcg); + unsigned long cgroup_size; + + if (is_local) + cgroup_size = get_cgroup_local_usage(memcg, true); + else + cgroup_size = mem_cgroup_size(memcg); unsigned long protection; /* memory.low scaling, make sure we retry before OOM */ @@ -5869,6 +5877,7 @@ static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc) }; struct mem_cgroup_reclaim_cookie *partial = &reclaim; struct mem_cgroup *memcg; + int is_local = (pgdat->node_id == 0) && root_reclaim(sc); /* * In most cases, direct reclaimers can do partial walks @@ -5896,7 +5905,7 @@ static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc) */ cond_resched(); - mem_cgroup_calculate_protection(target_memcg, memcg); + mem_cgroup_calculate_protection(target_memcg, memcg, is_local); if (mem_cgroup_below_min(target_memcg, memcg)) { /* @@ -5904,7 +5913,7 @@ static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc) * If there is no reclaimable memory, OOM. */ continue; - } else if (mem_cgroup_below_low(target_memcg, memcg)) { + } else if (mem_cgroup_below_low(target_memcg, memcg, is_local)) { /* * Soft protection. * Respect the protection only as long as From patchwork Fri Sep 20 22:11:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kaiyang Zhao X-Patchwork-Id: 13808714 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97C65CF9C68 for ; Fri, 20 Sep 2024 22:12:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 041496B0093; Fri, 20 Sep 2024 18:12:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F0BF16B0095; Fri, 20 Sep 2024 18:12:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D69476B0096; Fri, 20 Sep 2024 18:12:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B07DA6B0093 for ; Fri, 20 Sep 2024 18:12:57 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 5BC28403C7 for ; Fri, 20 Sep 2024 22:12:57 +0000 (UTC) X-FDA: 82586517594.08.79E5E4F Received: from mail-qv1-f48.google.com (mail-qv1-f48.google.com [209.85.219.48]) by imf22.hostedemail.com (Postfix) with ESMTP id 8F010C0003 for ; Fri, 20 Sep 2024 22:12:55 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=cs.cmu.edu header.s=google-2021 header.b=EF9YUGB2; spf=pass (imf22.hostedemail.com: domain of kaiyang2@andrew.cmu.edu designates 209.85.219.48 as permitted sender) smtp.mailfrom=kaiyang2@andrew.cmu.edu; dmarc=pass (policy=none) header.from=cs.cmu.edu ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726870260; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TftwzflgOcr42nYSagxxE5NRGUcR+2wBAeOf9BrcIxs=; b=Q46ZT9ATXwuLZME8ZKvequ6lGo7fJyF2xlICVE5k0AIEmURMrbibs3jgLlL5aKDD9WLw7n FFhGA+mEVEsTAjE0/fA9ccGV9M3LUr2S9NYO9bR8JtXoyld2jpsO0HU61m6Dzjzoop9J6B LK1aW1jBRhXy+bm60qzcrGXir+3V/mU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726870260; a=rsa-sha256; cv=none; b=dRUChLCIQWinWlzvHYVp/9X8YayruZ/0RoqneyxNg+OdRAhc/28cMk8BrjwbKdiWmBmnCI RGhj9uzegirCGozBDY1Jo+asSO20252Y0LZ7PVjCgpThMXFjmcf0XSMfGCA6vmbkZpLHAD xaBAD2j9ID8sNN5ML9wrYFp+3RPHzhE= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=cs.cmu.edu header.s=google-2021 header.b=EF9YUGB2; spf=pass (imf22.hostedemail.com: domain of kaiyang2@andrew.cmu.edu designates 209.85.219.48 as permitted sender) smtp.mailfrom=kaiyang2@andrew.cmu.edu; dmarc=pass (policy=none) header.from=cs.cmu.edu Received: by mail-qv1-f48.google.com with SMTP id 6a1803df08f44-6c3552ce7faso20946026d6.1 for ; Fri, 20 Sep 2024 15:12:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.cmu.edu; s=google-2021; t=1726870374; x=1727475174; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TftwzflgOcr42nYSagxxE5NRGUcR+2wBAeOf9BrcIxs=; b=EF9YUGB2J8pKIZyneCDbP1g7/Se7mqiDelUq3LY7MeJuphOHry6JFySFlEscQy7jsU cUET3kemSTSCu9hff1YLxSmPKffcyxPNymVzCHZcD4yE3iM+U+XuEiRN9ySJhGeyXiZ8 NKuGtkSDLTI85COGg8buYbpmhPHcAW0jSUCgccXEufjDMYEk/fT7ks/eTY+MROLAEAj/ rg0x3gh4cwViTwH+e7W3mxStDlwqfnUrhf8VX2c1A8J0NTwjegbOoB1Iy4FIMQaKeGwf KFJHJM4JsPH2D4j/0ReuerKJDgj9dB0CSR8tBMV3hCl9Bndc+vA0XNAOlrtjfOFW6SBu 92qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726870374; x=1727475174; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TftwzflgOcr42nYSagxxE5NRGUcR+2wBAeOf9BrcIxs=; b=hSAQLtZUV7J+L4tL0m17DO0ngjSuskcu2c9+o8l6l2Oqa1FLOgvm3ViDz39w52UXHY XH4CmFk+nxOCMgqslMIVENBoLOybLimnmGbr0AWuEFL79DfoFeGk/b0iAP2PNfqsfWd6 d3OqIXINvooz3EwaS4j4SkJqoLQvPZR6Wsfj0i93Rap+54ZqMoEGsplSXcn0xvmnnNcZ d4BnvPPKXVS3VX4/MuB6/3IQ3ioPel0KjBxo/qwPK1rW5E5QXd2GWveMowcqnxbjb1YZ bTlYB96Q2P4OH9+rLbb2r17lo0XrAtZ9E94pE65oWpX66UeCbHTblTiuJRZzTvnQnMqq Z5jA== X-Gm-Message-State: AOJu0Yy3N6iZDFEZL4vD0VOyFVq4ZTia5pZ9agGjnn09xV0IfuUAREZE DyIfITprk7YOE/nib4SjT5UjZy+aJZM2rQ8qx82KNnTLuxodkBbZ6p3TL+rUwAay3lgJcf3tdMH XDEtdelh/r+S+wZYs1gDbweDmZcMxvv9wbraXzHX5yZPqCq+R3iTCgZlSUJ3sd7MBjeV669/ixU DCctcXWXaDMhp+r3Kg6IBaSgGLWasDXV4Nyyy2rQ== X-Google-Smtp-Source: AGHT+IHikCGCO6wkPfPxYljfcKXf0g1rNsuHaxXCKqPX29rklo2k6jfuFPREyFGdHgTgjVFsYh9NRQ== X-Received: by 2002:a05:6214:419d:b0:6c5:1f00:502f with SMTP id 6a1803df08f44-6c7bb99bc41mr64018646d6.2.1726870374456; Fri, 20 Sep 2024 15:12:54 -0700 (PDT) Received: from localhost (pool-74-98-231-160.pitbpa.fios.verizon.net. [74.98.231.160]) by smtp.gmail.com with UTF8SMTPSA id 6a1803df08f44-6c75e557c5csm23290296d6.82.2024.09.20.15.12.53 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 20 Sep 2024 15:12:54 -0700 (PDT) From: kaiyang2@cs.cmu.edu To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: roman.gushchin@linux.dev, shakeel.butt@linux.dev, muchun.song@linux.dev, akpm@linux-foundation.org, mhocko@kernel.org, nehagholkar@meta.com, abhishekd@meta.com, hannes@cmpxchg.org, weixugc@google.com, rientjes@google.com, Kaiyang Zhao Subject: [RFC PATCH 4/4] reduce NUMA balancing scan size of cgroups over their local memory.low Date: Fri, 20 Sep 2024 22:11:51 +0000 Message-ID: <20240920221202.1734227-5-kaiyang2@cs.cmu.edu> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240920221202.1734227-1-kaiyang2@cs.cmu.edu> References: <20240920221202.1734227-1-kaiyang2@cs.cmu.edu> MIME-Version: 1.0 X-Stat-Signature: 3yck7y77qouzy1zms9nih33e7mi7ex7q X-Rspamd-Queue-Id: 8F010C0003 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1726870375-255091 X-HE-Meta: U2FsdGVkX1/mAaHKQ/32Xi172iG6lThAJe1Ue5Nf7uD7Fg9ro43EgqfjwDHb4HlrXn1M700mQ8hpq2G3Tc2RBapCkuVyZ3k+MGyVFUtqXwWMc9orG9DDA3DrlqL3L3ahYwBUZbFClX5+5dixRfHfZfCYNFjcK9w9Wyw5WFMr9JXFcPR+VNdTUVpAa7C83NNlBzEl79NcMhSH3LIKouAPPkl53bBTu8YRkzF7nM4Cz6ijCgnIWRy9K+8TPYo45p5tPnFeQE4CBn3W3GQ5NTjNcHUzc3EB7vn5nqjkspbzpi35JxgEHkIiJvqmkCdIG9yuM+t3PrjKYryOQPdIaVv/OH1X3rtld7B+8So0aYJwPX4tJROQ6LvXZxuHFeGDK/psQIPj7u+XBSxe7CBuIglghV/A8lREv3K6KqfHYcMTjyMIv36+2Lobag5d9cSeVIi2BVrGOny788hT3zg9DAZmZBJNd1TIvpXmmy24sPJyGkvtsXp8sYxVcZ9uWpY90WIC2iskeVMdoAXE12ErVu/XCED6JLpDUNG/Cb2X5A4hG7YbwOsuG1Ojb+qmsSIhn/o5QqF3hcSrD2NY3dDAHUInCTWTnjZuhM0uqDRA4SQ2rlf5X3Fy3XaWhhcQz3qcsfjSXbzbOLb3nbuDgJ1Ll2JY7l+5TRQaIXW9sSBduEN2kE/hAEk0iauZQwDv7spERcKck6xU+vVGwljjZIotdaoYNh/mEZg2DUCV3SHNGBzLM4q43ajhv4zFfUaCmuzDX2pEfYg8dQZ/2SzDaDxV/Rqc6QUtW+qVyWadzcVlJrpEJz5OeovLUf9oOReL9PWnk9e2yzf9Js8ZcKq99Yarwy99Gnk6heHLyJ73KeKKh9Wp+lQCntgPBb4eo2x53TtpOSYhTb/tCmDD+12Gu0FC69fxL8CpbAUQn490ZS3wr5n8btivJeVpdqvGpPvxQHC/LUGXQCkz+15naDHFgNCvk7H P86Y4jqk rxR/p84utT1KQ+7hs0Tywqa344py6LXYm3rplWAsav0K4VxmgUTv5epDMr6BZKP89eCljj5J67LphdHKBnl54pR8bDH7S5+bSoILm6F+Ruehj8VSeNqp8cqddSjsPL+dK2x8Nfd/36LbkSLMQ/bGlBNdI8SvCH3FntfVcQsAwsUx+yy9xBK+gbqfqj96/k9RTVJQQmwr8f22Lt/p4y7Yf5rS9SDekZp33qopVTeEYSu51Vi2gaC6C+FXFQwe914iIRuzLsb1muij+WrUpCharK8YVrlNKUtCk3BbkmDu1WwZ50xFmZKF9y5sKfMhlhCAIsWZeP0gU825uu74qYhjFy598SmWX5sv4ByT6wTSfAd+gqTKCGxv/BQ8PRL5kbEk17X/bPYRU5xNrqzk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kaiyang Zhao When the top-tier node has less free memory than the promotion watermark, reduce the scan size of cgroups that are over their local memory.low proportional to their overage. In this case, the top-tier memory usage of the cgroup should be reduced, and demotion is working towards the goal. A smaller scan size should cause a slower rate of promotion for the cgroup so as to not working against demotion. A mininum of 1/16th of sysctl_numa_balancing_scan_size is still allowed for such cgroups because identifying hot pages trapped in slow-tier is still a worthy goal in this case (although a secondary objective). 16 is arbitrary and may need tuning. Signed-off-by: Kaiyang Zhao --- kernel/sched/fair.c | 54 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 49 insertions(+), 5 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index a1b756f927b2..1737b2369f56 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1727,14 +1727,21 @@ static inline bool cpupid_valid(int cpupid) * advantage of fast memory capacity, all recently accessed slow * memory pages will be migrated to fast memory node without * considering hot threshold. + * This is also used for detecting memory pressure and decide whether + * limitting promotion scan size is needed, for which we don't requrie + * more free pages than the promo watermark. */ -static bool pgdat_free_space_enough(struct pglist_data *pgdat) +static bool pgdat_free_space_enough(struct pglist_data *pgdat, + bool require_extra) { int z; unsigned long enough_wmark; - enough_wmark = max(1UL * 1024 * 1024 * 1024 >> PAGE_SHIFT, - pgdat->node_present_pages >> 4); + if (require_extra) + enough_wmark = max(1UL * 1024 * 1024 * 1024 >> PAGE_SHIFT, + pgdat->node_present_pages >> 4); + else + enough_wmark = 0; for (z = pgdat->nr_zones - 1; z >= 0; z--) { struct zone *zone = pgdat->node_zones + z; @@ -1846,7 +1853,7 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio, unsigned int latency, th, def_th; pgdat = NODE_DATA(dst_nid); - if (pgdat_free_space_enough(pgdat)) { + if (pgdat_free_space_enough(pgdat, true)) { /* workload changed, reset hot threshold */ pgdat->nbp_threshold = 0; return true; @@ -3214,10 +3221,14 @@ static void task_numa_work(struct callback_head *work) struct vm_area_struct *vma; unsigned long start, end; unsigned long nr_pte_updates = 0; - long pages, virtpages; + long pages, virtpages, min_scan_pages; struct vma_iterator vmi; bool vma_pids_skipped; bool vma_pids_forced = false; + struct pglist_data *pgdat = NODE_DATA(0); /* hardcoded node 0 */ + struct mem_cgroup *memcg; + unsigned long cgroup_size, cgroup_locallow; + const long min_scan_pages_fraction = 16; /* 1/16th of the scan size */ SCHED_WARN_ON(p != container_of(work, struct task_struct, numa_work)); @@ -3262,6 +3273,39 @@ static void task_numa_work(struct callback_head *work) pages = sysctl_numa_balancing_scan_size; pages <<= 20 - PAGE_SHIFT; /* MB in pages */ + + min_scan_pages = pages; + min_scan_pages /= min_scan_pages_fraction; + + memcg = get_mem_cgroup_from_current(); + /* + * Reduce the scan size when the local node is under pressure + * (WMARK_PROMO is not satisfied), + * proportional to a cgroup's overage of local memory guarantee. + * 10% over: 68% of scan size + * 20% over: 48% of scan size + * 50% over: 20% of scan size + * 100% over: 6% of scan size + */ + if (likely(memcg)) { + if (!pgdat_free_space_enough(pgdat, false)) { + cgroup_size = get_cgroup_local_usage(memcg, false); + /* + * Protection needs refreshing, but reclaim on the cgroup + * should have refreshed recently. + */ + cgroup_locallow = READ_ONCE(memcg->memory.elocallow); + if (cgroup_size > cgroup_locallow) { + /* 1/x^4 */ + for (int i = 0; i < 4; i++) + pages = pages * cgroup_locallow / (cgroup_size + 1); + /* Lower bound to min_scan_pages. */ + pages = max(pages, min_scan_pages); + } + } + css_put(&memcg->css); + } + virtpages = pages * 8; /* Scan up to this much virtual space */ if (!pages) return;