From patchwork Mon Sep 19 18:06:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 12980905 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A06DECAAA1 for ; Mon, 19 Sep 2022 18:06:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 75F416B0071; Mon, 19 Sep 2022 14:06:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 70F4380008; Mon, 19 Sep 2022 14:06:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B02B80007; Mon, 19 Sep 2022 14:06:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4AAC66B0071 for ; Mon, 19 Sep 2022 14:06:54 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 127D31403CD for ; Mon, 19 Sep 2022 18:06:54 +0000 (UTC) X-FDA: 79929615948.21.29F674D Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf24.hostedemail.com (Postfix) with ESMTP id 8E91718000C for ; Mon, 19 Sep 2022 18:06:53 +0000 (UTC) Received: by mail-pl1-f173.google.com with SMTP id c24so6950722plo.3 for ; Mon, 19 Sep 2022 11:06:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date; bh=C+GNb1CRkpsaWy3nNaUeEZwBpKEJtnfXNeVMo/vFMIU=; b=fJXZ9vt1XMtBtJhooHOuHVO6gpZg/2Uq4qGa9T5h11pcGRnzjkEqyCrZLqmAkJO3mt 6jZ6Kmcm5Xrl4n/4Ha9OVyvbf7KSeSHSaBIOVNkl/Y+DFmj3H/uy0xNC+pLhyrze7MCr dwJzSTDmkot2vsTjksuK2VD4SWXd5wJcjGqt1tPITOlZ7VGEfcr6qO3cRxkn0CYilHgB T55kdJjZVfwysg0KjtfHwmZN+5pQSnz3gMRiZsYhWXzRHLhezJGrkX5qv07Q/7gmikk6 B/FPhC5n0u2iEpfA1o6oGoGvCu3B2YW8UjFu0aCFVVNQytVv//tVdMVPcPIP8TNm7bdz I93w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date; bh=C+GNb1CRkpsaWy3nNaUeEZwBpKEJtnfXNeVMo/vFMIU=; b=XLgLL10sXwFjMsfQuj+cPj9A6FavX6FhgrQPVDcIqNmY7pwud3d+JX2SeIe7S3ZIm6 insgp5E7ukS0SnzlkTMyUlXx9hIgK9v2TC+DuZeHx2KHi/oE9cE1CAnPlC7esC5CrDKT mxrh7Rah0LZRH+Po3BSmmCaicZqXgJNt9og1XtLbPz2cqqmmGYkj/MYusg3wx0k6shbQ DvitrPohFuCwj/GUe9HNjZ392+hN1vcYHpqyiErMzJ5W9E2VrLiiWDrWF5Hc3F0Sz/ac 5zNtGH73KoZ9pd46Ys94v7LypthzutWEBxJyyrKpkfzqESinAJ+DFrqRXn1hZg27JacZ DKDg== X-Gm-Message-State: ACrzQf1eSO3zmYw/0F+gPLwwYZN3yJy0KszZvn5RHtk0+jHcCvop4ElZ gjp49f4asbkbgZCCV5djCZc= X-Google-Smtp-Source: AMsMyM779q0TTvhK4qsuqZ791YA4zyjY0m9M+JWLvcM1gMkroLtT/c+VgF5ZePOZ7fH3LVOzJk9gXw== X-Received: by 2002:a17:90a:be10:b0:202:cdf2:56a1 with SMTP id a16-20020a17090abe1000b00202cdf256a1mr31606922pjs.41.1663610812577; Mon, 19 Sep 2022 11:06:52 -0700 (PDT) Received: from KASONG-MB0.tencent.com ([115.171.41.135]) by smtp.gmail.com with ESMTPSA id u21-20020a632355000000b0041c30def5e8sm14176654pgm.33.2022.09.19.11.06.49 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 19 Sep 2022 11:06:52 -0700 (PDT) From: Kairui Song To: cgroups@vger.kernel.org, linux-mm@kvack.org Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 1/2] mm: memcontrol: use memcg_kmem_enabled in count_objcg_event Date: Tue, 20 Sep 2022 02:06:33 +0800 Message-Id: <20220919180634.45958-2-ryncsn@gmail.com> X-Mailer: git-send-email 2.35.2 In-Reply-To: <20220919180634.45958-1-ryncsn@gmail.com> References: <20220919180634.45958-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663610813; a=rsa-sha256; cv=none; b=sDhL7StRjevh+6ThUU3mKRVZK6Dfljc5cNEhjFEI/I34v2jxVXYkchrQ/PO4Fk2qXWVUEF +O1eOhGL9Z6tsUD0qat3FH2XtWN+0+WYALaHoZeClQTv4wgQsloLOYP7RQGj5Qr1KhXwZD 4SHRBOs1bGtzolhEuEIxubcs4zeYTPI= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=fJXZ9vt1; spf=pass (imf24.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663610813; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=C+GNb1CRkpsaWy3nNaUeEZwBpKEJtnfXNeVMo/vFMIU=; b=PfULqaZNf6MZ099xpgI4MBCjUNzdwEwBtic4PliRbhL+XSiLMJQOI9388iz/I0wGA31CSq ev6BbJ15WVRSmPuWIegxmsnWJniw4ZzTiK4raAgPjpU1huaYp/T+UtVzPFWGU9hHohWRng T8W4Q1j2OPHpKYXcdD+VwVgvsIQo+H4= X-Stat-Signature: 7cr4fajrt1wdhhcm6q9djft99ap7tjec X-Rspam-User: Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=fJXZ9vt1; spf=pass (imf24.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 8E91718000C X-HE-Tag: 1663610813-904815 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kairui Song There are currently two helpers for checking if cgroup kmem accounting is enabled: - mem_cgroup_kmem_disabled - memcg_kmem_enabled mem_cgroup_kmem_disabled is a simple helper that returns true if cgroup.memory=nokmem is specified, otherwise returns false. memcg_kmem_enabled is a bit different, it returns true if cgroup.memory=nokmem is not specified and there was at least one non-root memory control enabled cgroup ever created. This help improve performance when kmem accounting was not actually activated. And it's optimized with static branch. The usage of mem_cgroup_kmem_disabled is for sub-systems that need to preallocate data for kmem accounting since they could be initialized before kmem accounting is activated. But count_objcg_event doesn't need that, so using memcg_kmem_enabled is better here. Signed-off-by: Kairui Song Acked-by: Shakeel Butt Acked-by: Roman Gushchin Acked-by: Muchun Song --- include/linux/memcontrol.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 6257867fbf95..e6d3d5870d6f 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1779,7 +1779,7 @@ static inline void count_objcg_event(struct obj_cgroup *objcg, { struct mem_cgroup *memcg; - if (mem_cgroup_kmem_disabled()) + if (!memcg_kmem_enabled()) return; rcu_read_lock(); From patchwork Mon Sep 19 18:06:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 12980906 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C74CECAAA1 for ; Mon, 19 Sep 2022 18:06:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DF03580007; Mon, 19 Sep 2022 14:06:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D9F116B0073; Mon, 19 Sep 2022 14:06:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C188980007; Mon, 19 Sep 2022 14:06:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B34F56B0072 for ; Mon, 19 Sep 2022 14:06:57 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 928458017E for ; Mon, 19 Sep 2022 18:06:57 +0000 (UTC) X-FDA: 79929616074.05.25502AB Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf18.hostedemail.com (Postfix) with ESMTP id 30F131C0006 for ; Mon, 19 Sep 2022 18:06:57 +0000 (UTC) Received: by mail-pj1-f43.google.com with SMTP id y11so427417pjv.4 for ; Mon, 19 Sep 2022 11:06:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date; bh=BmyIE6bjX+SmejLTpdpcM4Os0QidwRyPSwq1uGxWGR4=; b=IR2ENBZsA/cBit/3urmvxegAS9PRC1wzHTzogKeHFPUPE1gGA5gMyqjxue9fw/rGPG 06Afz3Uz5FYaEmI5TyDKksHG8XgWmBvNsR3lQxSc1VZ7Z5so228BHeD3Zr5QR4SFjAZu M8pV46fXT5l5TVU/dvQGhG7fEeogG0mImARtNJFksmGQC0ZoI3A6U4TP2xNdZmXujCL9 aXCSqWoUSyvzEEGRno68RWgWwpE/dRTtzwkexkBFjMSYdcSQ8DciCDzzMlJts36O4u35 HxZefgRj+/xOwLuBlX8rFAwkjMDEgQ+1ZRNBEtr2p1bOOU02oOsjzv/qMPd9VxGvRnRE g1Xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date; bh=BmyIE6bjX+SmejLTpdpcM4Os0QidwRyPSwq1uGxWGR4=; b=AuJ5aQFneZgiBAi0iuGsQcUDgq7m4lfgF9qCrhRWpDcYs3f5oGSzj/BZ+pwX/9cC8g rzr0aihdW1x6GkemfZnyXrccrpbQ+5wWgpgjdGoWzYD8JK229xKOWNbTMfyFFjUH1NAf ztDUD+Ej+SrpjnMPTITpyOxGJf/YralBgH8y1Cfc3ASOwWxqaimhfGwj9ouEELy4k9Iy f4CZCu322LRoDu/BzFkh/95FBdGFkjOBDxY5H75IzjULbr3tEYjCOGobDbpzPO3QTQCd 815zQMLmo2shQdkcMtmZ58qCh+qUKoM9lNMZSC0Y2Sy7gEjNNGUz2qAZwF/MDjW4pU3O qV6Q== X-Gm-Message-State: ACrzQf1lDoStaopvL1DSg4P8wwEO3a4P9/qrPy911nJqMJtFEJYJFipc g4K/oHGGD22Wf3eG4C+El/A= X-Google-Smtp-Source: AMsMyM626ee05frqAji9sXQtkEGcWw4FomA2JSV51nq3aHWZ5KW02UHaGhNgNZCzBINRJGwW3+0CiQ== X-Received: by 2002:a17:902:ec83:b0:178:39e5:abee with SMTP id x3-20020a170902ec8300b0017839e5abeemr985357plg.84.1663610816240; Mon, 19 Sep 2022 11:06:56 -0700 (PDT) Received: from KASONG-MB0.tencent.com ([115.171.41.135]) by smtp.gmail.com with ESMTPSA id u21-20020a632355000000b0041c30def5e8sm14176654pgm.33.2022.09.19.11.06.52 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 19 Sep 2022 11:06:55 -0700 (PDT) From: Kairui Song To: cgroups@vger.kernel.org, linux-mm@kvack.org Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , linux-kernel@vger.kernel.org, Kairui Song , Michal Hocko Subject: [PATCH v2 2/2] mm: memcontrol: make cgroup_memory_noswap a static key Date: Tue, 20 Sep 2022 02:06:34 +0800 Message-Id: <20220919180634.45958-3-ryncsn@gmail.com> X-Mailer: git-send-email 2.35.2 In-Reply-To: <20220919180634.45958-1-ryncsn@gmail.com> References: <20220919180634.45958-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663610817; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BmyIE6bjX+SmejLTpdpcM4Os0QidwRyPSwq1uGxWGR4=; b=N9yCcJvEXrdOQIKdSVbaD5P7xPHyeUyZ5qBULgW9Zhnq5xfyxDqT8TPpDmxvgIsL3CT+UG eK59SsoYfdNWvCSakULWEm3e+d4Covz3kxlInFWvp2wxLx9ggiv2Edty075qqDSStZ29ib Hs8NenW2w8NaMYCfNClXq/mwOqcL870= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=IR2ENBZs; spf=pass (imf18.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663610817; a=rsa-sha256; cv=none; b=GTGoF/NIiOoQ53bK7Dxr4FDTfNmlYjpd8R6deZo3wttOGAO91mxnzcKPhPgmHG9muPE4a0 XE7RGkrntlVsI9uMKdZ8Y6Z8+SKVfeL7zZaiOUvgZJJks6NL5ujJgV7mNNmUox6QAbpMUi +a+TvS3usTs/3asv2WpMkiOpsk1bin8= X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 30F131C0006 X-Rspam-User: Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=IR2ENBZs; spf=pass (imf18.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: 4cq51mg8fc19t9txxzj9xxr573x3pz3w X-HE-Tag: 1663610817-94068 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kairui Song cgroup_memory_noswap is used in many hot path, so make it a static key to lower the kernel overhead. Using 8G of ZRAM as SWAP, benchmark using `perf stat -d -d -d --repeat 100` with the following code snip in a non-root cgroup: #include #include #include #include #define MB 1024UL * 1024UL int main(int argc, char **argv){ void *p = mmap(NULL, 8000 * MB, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); memset(p, 0xff, 8000 * MB); madvise(p, 8000 * MB, MADV_PAGEOUT); memset(p, 0xff, 8000 * MB); return 0; } Before: 7,021.43 msec task-clock # 0.967 CPUs utilized ( +- 0.03% ) 4,010 context-switches # 573.853 /sec ( +- 0.01% ) 0 cpu-migrations # 0.000 /sec 2,052,057 page-faults # 293.661 K/sec ( +- 0.00% ) 12,616,546,027 cycles # 1.805 GHz ( +- 0.06% ) (39.92%) 156,823,666 stalled-cycles-frontend # 1.25% frontend cycles idle ( +- 0.10% ) (40.25%) 310,130,812 stalled-cycles-backend # 2.47% backend cycles idle ( +- 4.39% ) (40.73%) 18,692,516,591 instructions # 1.49 insn per cycle # 0.01 stalled cycles per insn ( +- 0.04% ) (40.75%) 4,907,447,976 branches # 702.283 M/sec ( +- 0.05% ) (40.30%) 13,002,578 branch-misses # 0.26% of all branches ( +- 0.08% ) (40.48%) 7,069,786,296 L1-dcache-loads # 1.012 G/sec ( +- 0.03% ) (40.32%) 649,385,847 L1-dcache-load-misses # 9.13% of all L1-dcache accesses ( +- 0.07% ) (40.10%) 1,485,448,688 L1-icache-loads # 212.576 M/sec ( +- 0.15% ) (39.49%) 31,628,457 L1-icache-load-misses # 2.13% of all L1-icache accesses ( +- 0.40% ) (39.57%) 6,667,311 dTLB-loads # 954.129 K/sec ( +- 0.21% ) (39.50%) 5,668,555 dTLB-load-misses # 86.40% of all dTLB cache accesses ( +- 0.12% ) (39.03%) 765 iTLB-loads # 109.476 /sec ( +- 21.81% ) (39.44%) 4,370,351 iTLB-load-misses # 214320.09% of all iTLB cache accesses ( +- 1.44% ) (39.86%) 149,207,254 L1-dcache-prefetches # 21.352 M/sec ( +- 0.13% ) (40.27%) 7.25869 +- 0.00203 seconds time elapsed ( +- 0.03% ) After: 6,576.16 msec task-clock # 0.953 CPUs utilized ( +- 0.10% ) 4,020 context-switches # 605.595 /sec ( +- 0.01% ) 0 cpu-migrations # 0.000 /sec 2,052,056 page-faults # 309.133 K/sec ( +- 0.00% ) 11,967,619,180 cycles # 1.803 GHz ( +- 0.36% ) (38.76%) 161,259,240 stalled-cycles-frontend # 1.38% frontend cycles idle ( +- 0.27% ) (36.58%) 253,605,302 stalled-cycles-backend # 2.16% backend cycles idle ( +- 4.45% ) (34.78%) 19,328,171,892 instructions # 1.65 insn per cycle # 0.01 stalled cycles per insn ( +- 0.10% ) (31.46%) 5,213,967,902 branches # 785.461 M/sec ( +- 0.18% ) (30.68%) 12,385,170 branch-misses # 0.24% of all branches ( +- 0.26% ) (34.13%) 7,271,687,822 L1-dcache-loads # 1.095 G/sec ( +- 0.12% ) (35.29%) 649,873,045 L1-dcache-load-misses # 8.93% of all L1-dcache accesses ( +- 0.11% ) (41.41%) 1,950,037,608 L1-icache-loads # 293.764 M/sec ( +- 0.33% ) (43.11%) 31,365,566 L1-icache-load-misses # 1.62% of all L1-icache accesses ( +- 0.39% ) (45.89%) 6,767,809 dTLB-loads # 1.020 M/sec ( +- 0.47% ) (48.42%) 6,339,590 dTLB-load-misses # 95.43% of all dTLB cache accesses ( +- 0.50% ) (46.60%) 736 iTLB-loads # 110.875 /sec ( +- 1.79% ) (48.60%) 4,314,836 iTLB-load-misses # 518653.73% of all iTLB cache accesses ( +- 0.63% ) (42.91%) 144,950,156 L1-dcache-prefetches # 21.836 M/sec ( +- 0.37% ) (41.39%) 6.89935 +- 0.00703 seconds time elapsed ( +- 0.10% ) The performance is clearly better. There is no significant hotspot improvement according to perf report, as there are quite a few callers of memcg_swap_enabled and do_memsw_account (which calls memcg_swap_enabled). Many pieces of minor optimizations resulted in lower overhead for the branch predictor, and bettter performance. Acked-by: Michal Hocko Signed-off-by: Kairui Song Acked-by: Shakeel Butt Acked-by: Roman Gushchin Acked-by: Muchun Song --- mm/memcontrol.c | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index b69979c9ced5..5bb89c745233 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -90,9 +90,18 @@ static bool cgroup_memory_nokmem __ro_after_init; /* Whether the swap controller is active */ #ifdef CONFIG_MEMCG_SWAP -static bool cgroup_memory_noswap __ro_after_init; +static bool cgroup_memory_noswap __initdata; + +static DEFINE_STATIC_KEY_FALSE(memcg_swap_enabled_key); +static inline bool memcg_swap_enabled(void) +{ + return static_branch_likely(&memcg_swap_enabled_key); +} #else -#define cgroup_memory_noswap 1 +static inline bool memcg_swap_enabled(void) +{ + return false; +} #endif #ifdef CONFIG_CGROUP_WRITEBACK @@ -102,7 +111,7 @@ static DECLARE_WAIT_QUEUE_HEAD(memcg_cgwb_frn_waitq); /* Whether legacy memory+swap accounting is active */ static bool do_memsw_account(void) { - return !cgroup_subsys_on_dfl(memory_cgrp_subsys) && !cgroup_memory_noswap; + return !cgroup_subsys_on_dfl(memory_cgrp_subsys) && memcg_swap_enabled(); } #define THRESHOLDS_EVENTS_TARGET 128 @@ -7267,7 +7276,7 @@ void mem_cgroup_swapout(struct folio *folio, swp_entry_t entry) if (!mem_cgroup_is_root(memcg)) page_counter_uncharge(&memcg->memory, nr_entries); - if (!cgroup_memory_noswap && memcg != swap_memcg) { + if (memcg_swap_enabled() && memcg != swap_memcg) { if (!mem_cgroup_is_root(swap_memcg)) page_counter_charge(&swap_memcg->memsw, nr_entries); page_counter_uncharge(&memcg->memsw, nr_entries); @@ -7319,7 +7328,7 @@ int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry) memcg = mem_cgroup_id_get_online(memcg); - if (!cgroup_memory_noswap && !mem_cgroup_is_root(memcg) && + if (memcg_swap_enabled() && !mem_cgroup_is_root(memcg) && !page_counter_try_charge(&memcg->swap, nr_pages, &counter)) { memcg_memory_event(memcg, MEMCG_SWAP_MAX); memcg_memory_event(memcg, MEMCG_SWAP_FAIL); @@ -7351,7 +7360,7 @@ void __mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_pages) rcu_read_lock(); memcg = mem_cgroup_from_id(id); if (memcg) { - if (!cgroup_memory_noswap && !mem_cgroup_is_root(memcg)) { + if (memcg_swap_enabled() && !mem_cgroup_is_root(memcg)) { if (cgroup_subsys_on_dfl(memory_cgrp_subsys)) page_counter_uncharge(&memcg->swap, nr_pages); else @@ -7367,7 +7376,7 @@ long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg) { long nr_swap_pages = get_nr_swap_pages(); - if (cgroup_memory_noswap || !cgroup_subsys_on_dfl(memory_cgrp_subsys)) + if (!memcg_swap_enabled() || !cgroup_subsys_on_dfl(memory_cgrp_subsys)) return nr_swap_pages; for (; memcg != root_mem_cgroup; memcg = parent_mem_cgroup(memcg)) nr_swap_pages = min_t(long, nr_swap_pages, @@ -7384,7 +7393,7 @@ bool mem_cgroup_swap_full(struct page *page) if (vm_swap_full()) return true; - if (cgroup_memory_noswap || !cgroup_subsys_on_dfl(memory_cgrp_subsys)) + if (!memcg_swap_enabled() || !cgroup_subsys_on_dfl(memory_cgrp_subsys)) return false; memcg = page_memcg(page); @@ -7692,6 +7701,8 @@ static int __init mem_cgroup_swap_init(void) if (cgroup_memory_noswap) return 0; + static_branch_enable(&memcg_swap_enabled_key); + WARN_ON(cgroup_add_dfl_cftypes(&memory_cgrp_subsys, swap_files)); WARN_ON(cgroup_add_legacy_cftypes(&memory_cgrp_subsys, memsw_files)); #if defined(CONFIG_MEMCG_KMEM) && defined(CONFIG_ZSWAP)