From patchwork Mon Dec 12 00:37:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13070694 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A179BC4167B for ; Mon, 12 Dec 2022 00:37:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 42E378E0005; Sun, 11 Dec 2022 19:37:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3DE668E0002; Sun, 11 Dec 2022 19:37:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 280008E0005; Sun, 11 Dec 2022 19:37:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 18AFA8E0002 for ; Sun, 11 Dec 2022 19:37:46 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D8726A9175 for ; Mon, 12 Dec 2022 00:37:45 +0000 (UTC) X-FDA: 80231791290.19.511E918 Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) by imf24.hostedemail.com (Postfix) with ESMTP id 3A13118000D for ; Mon, 12 Dec 2022 00:37:42 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=bWMeQ5bn; spf=pass (imf24.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.48 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670805462; a=rsa-sha256; cv=none; b=VBn86p2LHFdw++HZw+ncKinUA03N8KP/a9d8A+JQyj+5T/mRRKShelF1cJjYfNEeWeyMSb uN4QX/aXRVkQGIQAGlN7bBQkErKrC740pVBsxTRprR6v46s5TXvAr7p2ezXF8qC2dM0dIR 7vn3ldpeHewDpfUXgrBMh78Y8v8ga88= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=bWMeQ5bn; spf=pass (imf24.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.48 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670805462; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EWked+wJERqcTSi79JV5oNIuOa0tfY0vnwLs5gY8MS8=; b=pjqcb7D6ST1k21gaBEu2JLipEurtMMFMgniHAzPRGYdzKQJBKpki0Me4GXGQl5DpgOCcGQ JM2zqvs5LlVG5mtgruZ9asj74cohWRWlAWcLfRmwjkXtj6dcyLQKGEMUycjFasHKOdlf71 /RRlz/lKHfoNBXqa9nPMA1Y4f8v9Qao= Received: by mail-pj1-f48.google.com with SMTP id q17-20020a17090aa01100b002194cba32e9so13887507pjp.1 for ; Sun, 11 Dec 2022 16:37:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EWked+wJERqcTSi79JV5oNIuOa0tfY0vnwLs5gY8MS8=; b=bWMeQ5bnX0TAF6bOdYkP/oOKV5jf+3Uc15vF0wU9CfGJjw8kfYoU9EPZGM+bc1iKJx ky0g+H15r5zMpolW5OyrMmBrnfZpvmLsGwkUUacNNe8GZ4BzalwNbsYnV8siK0KSl7lv 9/STfhGYSN5Bjms4C9NOCDfn0fZIaDlTYFfh4UM012DUHv+Y/RfbqnWW06UsZxe8BGa4 +qnpdhR49of03EqQmxuVAUhq0r8nqWyKiAn1qvq5SLlOTbQKqnSOoCbL3MnLuPcYWUUF p3LOJ5YBnQDwUEvNREGZPpa+SVWD17iJwLObXxC6Z/3sjrjshYcCezzIqtZ1d8aoDxMB UFyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EWked+wJERqcTSi79JV5oNIuOa0tfY0vnwLs5gY8MS8=; b=Z2SMrO/KhM4ITNa9Sjy7I4OVoNHPGjfYusbsQmdpj6cRBJnaVNrQzHOSWWimbdC4QJ E3NdPozIrF3W2ThTQwWtQwL0SYhMvj7CaJZ59vGtZWz/DdmJlFlu8X7VZfL3MyDB7Lop uo8PO/rwEs/pcsHfySlD0+ux8/2BbiKYV9pzuflTH9jvCCJ8+Zst5y3M3Ou28aiihs2U KbWgJqP6Ky/bLxr/z+Kfjqxh/eixp3M7G82hQiKHHg8n0H8f1mJYqtSxdmvyJbmwJuzz IFVr6E34OZnFfZF8uzOYcK8eLGV2QDmqu2c2wNcqhF9Oe5l1Y4J9ztIC43bExnzNYq5t kzOw== X-Gm-Message-State: ANoB5pliisRykl3THN3byx1eY+stuh0Oc7Vh8L0TeI1xrZ1cSWDErGyo L8ep/tE0vM7qDEsXj2qiGcw= X-Google-Smtp-Source: AA0mqf7ELqiSpWsfHtvtdVOpa+IdGBSzwZN2i+Zxsp57q2bOLFd/Qst7Jv0nZP60GjGAejI5hY2y/g== X-Received: by 2002:a17:902:728d:b0:189:6293:e01a with SMTP id d13-20020a170902728d00b001896293e01amr13335786pll.12.1670805461055; Sun, 11 Dec 2022 16:37:41 -0800 (PST) Received: from vultr.guest ([2001:19f0:7002:8c7:5400:4ff:fe3d:656a]) by smtp.gmail.com with ESMTPSA id w9-20020a170902e88900b00177fb862a87sm4895960plg.20.2022.12.11.16.37.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 11 Dec 2022 16:37:40 -0800 (PST) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next 1/9] mm: Introduce active vm item Date: Mon, 12 Dec 2022 00:37:03 +0000 Message-Id: <20221212003711.24977-2-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20221212003711.24977-1-laoar.shao@gmail.com> References: <20221212003711.24977-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 3A13118000D X-Stat-Signature: ypwu3dzwhmc99m3en8yxmahgprmaqgpr X-HE-Tag: 1670805462-21523 X-HE-Meta: U2FsdGVkX1/HGkApgF9O4iSViRnMv5lj2UvQdpa5FapJGRJbbaywFTX60zNbAT6SfYt2nmptYtXgmrKcs/CuO2cGFXdGZ/b8NlJVFQ852yonHYaPhS+xxJkTdfSS8TdHIwP97XVdmymlFd/xLIUy8zOi1b/YyMY4tXbjeG/tLglEhTO+26FXO3Ylrne8Xc9cw8zigeTjETjicDn7bVhmuPguYhM01v4sUrNhBsnquHk2+rNAivLW44IR8w877GHNwC0Dk6sjk+8PzjVQjbqVQRJAMfv5Hy5BsMn1M1nosEjQTDdFBXTw5QUxHIcnbJRSNFo8vQyCJWiss3T2cmWOmyo77zXbYkevR2jCn7iFcN7Q8y1lwmP72XeXmktwjv64p8nYjr8ra7IlQphY2aLw4gdk3etetzQfcC5hRmu4WiR7V0M4W7vrYY6nHt2vNst98Zf321lMP7qacFajcPDndC5Sy+3ueyVglDuOSZetZMvpIppWHTbch2FyT5aPeCdUBoZUYOh7HHr4Hz9xQRyorD4IOgr3ZPNmwWoR7jDwDFA2Y1MQ0FXSOAWEwG6SfmgEb6F8fIlJ9R6dVZPf4arje2fTsxP2eX6j0DrYEVVt/6q9mhlWTm20nkU4Wjn17aoOGo3fwMWkgdz9SSiCZbXNKHQKTd7wxAQOrEFFwbHE4sbgm8Uiu/qMwZdoGCjXf7lPTWCpuSbKhEg17S2VZugduPx4cNFYf3fm5wk8sSh7tP60+jJ2rIDKQjJvfeb9tv5E1jZXA5Ffe4uhoT+qNybvPJCZXIF2u43lXAbMiwkfh0nY5hU3OR+XpRn+GW65wgbGKYWIj7c3g5rYG62sR6P/ktiYWxGxoPRZ0ZT66rnLufhdzrg1Rs4qbPG2GpP+GlXQILmTNfncUEeDckrzq9ZTKOAISzH4IShfv64j5WZFPEI4LjqDPXT6o90T/puBJrtW7p3PubMiz7fXk6B+B66 SFyt+mTv LOfUquwiQEXxkrh9Voycq2hRFpepDwHKWH/TIALHc4AFHxniXxTm8XWSA5fub1zwMyQvKpJawiu6dYi8Z3YSvdl+qd5y+wn3X5vOAi6dALD4BuwNMGxFyC+XnPPr7L9nrme+JABnYzVA71ZB8bwr1lNot6LcU8rVxyx1DUbkY2BDgCVKAtUMH+9ny4cBmW7iLySFVBvtvYVJ3S5bQPlhOolq36Mp0ptqxcgna+QNnPsh6ql4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A new page extension active_vm is introduced in this patch. It can be enabled and disabled by setting CONFIG_ACTIVE_VM at compile time or a boot parameter 'active_vm' at run time. In the followup patches, we will use it to account specific memory allocation for page, slab and percpu memory. Signed-off-by: Yafang Shao --- include/linux/active_vm.h | 23 +++++++++++++++++++++++ mm/Kconfig | 8 ++++++++ mm/Makefile | 1 + mm/active_vm.c | 33 +++++++++++++++++++++++++++++++++ mm/active_vm.h | 8 ++++++++ mm/page_ext.c | 4 ++++ 6 files changed, 77 insertions(+) create mode 100644 include/linux/active_vm.h create mode 100644 mm/active_vm.c create mode 100644 mm/active_vm.h diff --git a/include/linux/active_vm.h b/include/linux/active_vm.h new file mode 100644 index 000000000000..899e578e94fa --- /dev/null +++ b/include/linux/active_vm.h @@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __INCLUDE_ACTIVE_VM_H +#define __INCLUDE_ACTIVE_VM_H + +#ifdef CONFIG_ACTIVE_VM +#include + +extern struct static_key_true active_vm_disabled; + +static inline bool active_vm_enabled(void) +{ + if (static_branch_likely(&active_vm_disabled)) + return false; + + return true; +} +#else +static inline bool active_vm_enabled(void) +{ + return false; +} +#endif /* CONFIG_ACTIVE_VM */ +#endif /* __INCLUDE_ACTIVE_VM_H */ diff --git a/mm/Kconfig b/mm/Kconfig index 57e1d8c5b505..ba1087e4afff 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1150,6 +1150,14 @@ config LRU_GEN_STATS This option has a per-memcg and per-node memory overhead. # } +config ACTIVE_VM + bool "To track memory size of active VM item" + default y + depends on PAGE_EXTENSION + help + Allow scope-based memory accouting for specific memory, e.g. the + system-wide BPF memory usage. + source "mm/damon/Kconfig" endmenu diff --git a/mm/Makefile b/mm/Makefile index 8e105e5b3e29..347dcff061d5 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -138,3 +138,4 @@ obj-$(CONFIG_IO_MAPPING) += io-mapping.o obj-$(CONFIG_HAVE_BOOTMEM_INFO_NODE) += bootmem_info.o obj-$(CONFIG_GENERIC_IOREMAP) += ioremap.o obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o +obj-$(CONFIG_ACTIVE_VM) += active_vm.o diff --git a/mm/active_vm.c b/mm/active_vm.c new file mode 100644 index 000000000000..60849930a7d3 --- /dev/null +++ b/mm/active_vm.c @@ -0,0 +1,33 @@ +// SPDX-License-Identifier: GPL-2.0 +#include + +static bool __active_vm_enabled __initdata = + IS_ENABLED(CONFIG_ACTIVE_VM); + +DEFINE_STATIC_KEY_TRUE(active_vm_disabled); +EXPORT_SYMBOL(active_vm_disabled); + +static int __init early_active_vm_param(char *buf) +{ + return strtobool(buf, &__active_vm_enabled); +} + +early_param("active_vm", early_active_vm_param); + +static bool __init need_active_vm(void) +{ + return __active_vm_enabled; +} + +static void __init init_active_vm(void) +{ + if (!__active_vm_enabled) + return; + + static_branch_disable(&active_vm_disabled); +} + +struct page_ext_operations active_vm_ops = { + .need = need_active_vm, + .init = init_active_vm, +}; diff --git a/mm/active_vm.h b/mm/active_vm.h new file mode 100644 index 000000000000..72978955833e --- /dev/null +++ b/mm/active_vm.h @@ -0,0 +1,8 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __MM_ACTIVE_VM_H +#define __MM_ACTIVE_VM_H + +#ifdef CONFIG_ACTIVE_VM +extern struct page_ext_operations active_vm_ops; +#endif /* CONFIG_ACTIVE_VM */ +#endif /* __MM_ACTIVE_VM_H */ diff --git a/mm/page_ext.c b/mm/page_ext.c index ddf1968560f0..3a3a91bc9e06 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -10,6 +10,7 @@ #include #include #include +#include "active_vm.h" /* * struct page extension @@ -84,6 +85,9 @@ static struct page_ext_operations *page_ext_ops[] __initdata = { #ifdef CONFIG_PAGE_TABLE_CHECK &page_table_check_ops, #endif +#ifdef CONFIG_ACTIVE_VM + &active_vm_ops, +#endif }; unsigned long page_ext_size = sizeof(struct page_ext); From patchwork Mon Dec 12 00:37:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13070695 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5AAE4C4332F for ; Mon, 12 Dec 2022 00:37:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B5B078E0006; Sun, 11 Dec 2022 19:37:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B318B8E0002; Sun, 11 Dec 2022 19:37:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9AB9C8E0006; Sun, 11 Dec 2022 19:37:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8BE4E8E0002 for ; Sun, 11 Dec 2022 19:37:48 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 5B8841C5C32 for ; Mon, 12 Dec 2022 00:37:48 +0000 (UTC) X-FDA: 80231791416.04.194263E Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf09.hostedemail.com (Postfix) with ESMTP id BC5FF14000F for ; Mon, 12 Dec 2022 00:37:46 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=aylXhwHR; spf=pass (imf09.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670805466; a=rsa-sha256; cv=none; b=BrIaoZeKLcmphBFHG+usdJkAb15qrGNmuaoK5Tfh167fqady3MSuWCMeOMt+T3G3S6mjMd yfNPAEXIundVtX2cqAQMSi7l1pumUKyzHM7VbkDw8xVeB+oEQNGppFzFDcepslO+WzDQQs 8Aw+slniXdt9+B6CMXJXgwIA8A/miTg= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=aylXhwHR; spf=pass (imf09.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670805466; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rGtC3SN6Ifn7NlhGu9Dw24wgt0iqbWERqtVVRikZ7FU=; b=5rsEHrEw8Vdf0hy7h8zZT1TCET4MVIWCG3mA5NzI4luLNV0gd5hnDwoD+r5hRBmi6mEKGu m9cK9WaHyHVifVnH29uU2m7CSGL8VbdKYzlyr/Pg14hQZmfkfITfgBe33n77ftRWldjbh8 dIr3c3V9KXlBheTUWZPag03m5HZztk4= Received: by mail-pl1-f180.google.com with SMTP id d3so10410187plr.10 for ; Sun, 11 Dec 2022 16:37:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rGtC3SN6Ifn7NlhGu9Dw24wgt0iqbWERqtVVRikZ7FU=; b=aylXhwHRB/AnaqV0nbRReAnU29I+73XtbQYit2HtT6Nf4V02lXAmWcUcXfAMaKKMyW ongHgiHPo/1Yo/hBjs1Yonph+tyD/hp39THh63O0jAOOrpoVfVW/gFXkc1QVjqQ1sYiL Q5Pobduy9ttgieGJeoJ3ggbEV9QYNbSZ2mX4u1ANqmQlGpY9GKRtcV7baGzZ6FgIVx+r +kOxzFrlGcZCgssVQNCEtuJh9I0VFURrNHQW8x1ib2kSYXZO8Wpiu8uGtc2/lvGfclsb 8VSviT9bHd5lIOEooS7oy7agNzaiKd6m4bpfeDCtz+Mex01Uu0FR1DKPDPgjyDTOXIqa XzEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rGtC3SN6Ifn7NlhGu9Dw24wgt0iqbWERqtVVRikZ7FU=; b=uKTUarUhYTOPRtWqf1VU7R45uzIhtXXl4XCOV0nhK5oCRf8SYf0/tKAhAfCHn20Oiv +cuDrjK5GRXvclqoIft3VWsq4SQG7/x4E74QwMit4KckA/gD2MbcFjYZw2z+MagzB43s fhR9e2yF8mfmNwLeq9BGdy2YYDzJbzECL2ql4MWPEpQmo7VnW5IBURxe40EhstsydE2H ujndknipX0wMcpqDabEsOlg7j3yyjvYikdbj9o99bnXrnTqCMSpvdavjd91SUQp2bMim 1A1j060RooNrfQ/TLwjlJwrePfyYEjpetBj9JQ+xxSYyvA05JyQz4yIhr4y+lwVi7hdn EIPg== X-Gm-Message-State: ANoB5pkcfU/Afw+J42sw1vopDYPe9oy9OmY9B7uvAYqU67Lzm7spm/tk HLluGKxiVGe9YNJa7G1PvKA= X-Google-Smtp-Source: AA0mqf5OlxYfuHjLp+MFFLeSNr64rxHfH89FJoaBzG1fbT222yfAv8sK9MnrXVllA3XEBIYOCtCxTg== X-Received: by 2002:a05:6a20:a681:b0:ac:1266:bda with SMTP id ba1-20020a056a20a68100b000ac12660bdamr19043512pzb.7.1670805466209; Sun, 11 Dec 2022 16:37:46 -0800 (PST) Received: from vultr.guest ([2001:19f0:7002:8c7:5400:4ff:fe3d:656a]) by smtp.gmail.com with ESMTPSA id w9-20020a170902e88900b00177fb862a87sm4895960plg.20.2022.12.11.16.37.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 11 Dec 2022 16:37:45 -0800 (PST) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next 2/9] mm: Allow using active vm in all contexts Date: Mon, 12 Dec 2022 00:37:04 +0000 Message-Id: <20221212003711.24977-3-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20221212003711.24977-1-laoar.shao@gmail.com> References: <20221212003711.24977-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: BC5FF14000F X-Stat-Signature: g9ye74e4x9gpu485i758zwrw5siuohmc X-HE-Tag: 1670805466-374049 X-HE-Meta: U2FsdGVkX1+r5Eu9eFfPaXM0eaY8jN3m2Kypr61Sn8ggqeM2izFKzSNX2SXpuO5IoHxYcQuK8pPf8rF1wBAPGzhlv2+FXT5yFdmQkrRPJ2LKdf+65SVC3XSHznyem/9pbZp37IoXEQvNttJCjH3+uyMml5Uyv7Mcz9Yq2sBrDgP0mL1sEL4GPudLLpaAexXc19bhHc1gD0s5G4baHGQpcHvUWiew7uDlbjZTDmWNQCzRX1KZTF5ieYqKKs9Lu6t7lDb8P+pKtPoLNfjKbbpXytpP8ltJeQOH+RgZ/ObR0HHdvIyVg/K9xLqHx79pTVssQuqFMva2ksz3ZfcB2NVcpLfrxQpMrJvCh9LdbW5Au9CFgl63wn7qlYtKGX43AQBRKFd3aV9KgfrZFmK/JQD2XGWdQQpfEZy4pX5ijQWWOz/EYtjRrt5Hum5jH4c5+MowTtaG4y/WbImuwVYcYb7CuYEXSA+rlR+VRBhyr/4vEEEvvqjw73aOLo5w7sNNe3vYDV9h/8rhD6ZVlxcP43Qj3Dwss+Z+oAmjyZVDeoth5M5IMHCXTTU2jSbppeeSbjZ/XDiB5H2z4f/HeyZv8AUdldbSnFTWGAq21I6JQ44FuQYkm5l5K5A6+P844dRTJ8zc6wQ+OXXaKoTZ7wkGNA1wrA2B/F7HTQ5nrC1DRfgCRDlOLfAF2nF1JZx02qaN1+Yzrmol8+Tokfy742ITSpSO0QDLLitUYYHMdmfctdOHbOj15nHOVflYN9+7igaZPnIsUD03Filn/i85p/d4AZ+7sUpxMYXS2Do8vEeLd+cgIq7aSZGM8oG3inGWkQc5SitQPLs3GexonbPh2GOBERifvCCH+n8jsn/bmx0y+S3xaqQdCjyPtFApV/soTsjRxNummsSF9zxzqLBCsrte+tgws1DMOm7Tb/EwqKFlorlefoWyGbUYKDQZ0i+/6X55O9mVIGLv1TQvIkCyv3A/nb2 cknmCzjX Z5DpmyxfMxTJkyHuxWPQo6EwWyfi52NZkwi7QWmgSMgsuto5mYY8Mv7p/zNJ7ZHC3XWgm4d+mzoU8KYwlZEL4D/ex60c4p3BCW11wO/OpWSTaibfNjnXLO5u5SV43qnbis55Zp9BjjXI48+sQ2gx2ungqAB5V+kUc2RPxBOJ8WQXWgzWbVxevoMi+MYXAFRgQtmdLF4bzsMn4MyehpDPnRB74N4Krbm17X8Vzmt5PKTADd3g= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We can use active vm in task, softirq and irq to account specific memory usage. Two new helpers active_vm_{add,sub} are introduced. A dummy item is introduced here, which will be removed when we introduce real item. Signed-off-by: Yafang Shao --- include/linux/active_vm.h | 50 +++++++++++++++++++++++++++++++++++++++ include/linux/sched.h | 5 ++++ kernel/fork.c | 4 ++++ mm/active_vm.c | 23 ++++++++++++++++++ mm/active_vm.h | 38 +++++++++++++++++++++++++++++ 5 files changed, 120 insertions(+) diff --git a/include/linux/active_vm.h b/include/linux/active_vm.h index 899e578e94fa..21f9aaca12c4 100644 --- a/include/linux/active_vm.h +++ b/include/linux/active_vm.h @@ -4,6 +4,9 @@ #ifdef CONFIG_ACTIVE_VM #include +#include +#include +#include extern struct static_key_true active_vm_disabled; @@ -14,10 +17,57 @@ static inline bool active_vm_enabled(void) return true; } + +enum active_vm_item { + DUMMY_ITEM = 1, + NR_ACTIVE_VM_ITEM = DUMMY_ITEM, +}; + +struct active_vm_stat { + long stat[NR_ACTIVE_VM_ITEM]; +}; + +DECLARE_PER_CPU(struct active_vm_stat, active_vm_stats); +DECLARE_PER_CPU(int, irq_active_vm_item); +DECLARE_PER_CPU(int, soft_active_vm_item); + +static inline int +active_vm_item_set(int item) +{ + int old_item; + + if (in_irq()) { + old_item = this_cpu_read(irq_active_vm_item); + this_cpu_write(irq_active_vm_item, item); + } else if (in_softirq()) { + old_item = this_cpu_read(soft_active_vm_item); + this_cpu_write(soft_active_vm_item, item); + } else { + old_item = current->active_vm_item; + current->active_vm_item = item; + } + + return old_item; +} + +long active_vm_item_sum(int item); + #else static inline bool active_vm_enabled(void) { return false; } + +static inline int +active_vm_item_set(int item) +{ + return 0; +} + +static inline long active_vm_item_sum(int item) +{ + return 0; +} + #endif /* CONFIG_ACTIVE_VM */ #endif /* __INCLUDE_ACTIVE_VM_H */ diff --git a/include/linux/sched.h b/include/linux/sched.h index ffb6eb55cd13..05acefd383d4 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1441,6 +1441,11 @@ struct task_struct { struct mem_cgroup *active_memcg; #endif +#ifdef CONFIG_ACTIVE_VM + /* Used for scope-based memory accounting */ + int active_vm_item; +#endif + #ifdef CONFIG_BLK_CGROUP struct request_queue *throttle_queue; #endif diff --git a/kernel/fork.c b/kernel/fork.c index 08969f5aa38d..590d949ff131 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1043,6 +1043,10 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node) tsk->active_memcg = NULL; #endif +#ifdef CONFIG_ACTIVE_VM + tsk->active_vm_item = 0; +#endif + #ifdef CONFIG_CPU_SUP_INTEL tsk->reported_split_lock = 0; #endif diff --git a/mm/active_vm.c b/mm/active_vm.c index 60849930a7d3..541b2ba22da9 100644 --- a/mm/active_vm.c +++ b/mm/active_vm.c @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 #include +#include static bool __active_vm_enabled __initdata = IS_ENABLED(CONFIG_ACTIVE_VM); @@ -31,3 +32,25 @@ struct page_ext_operations active_vm_ops = { .need = need_active_vm, .init = init_active_vm, }; + +DEFINE_PER_CPU(int, irq_active_vm_item); +DEFINE_PER_CPU(int, soft_active_vm_item); +EXPORT_PER_CPU_SYMBOL(irq_active_vm_item); +EXPORT_PER_CPU_SYMBOL(soft_active_vm_item); +DEFINE_PER_CPU(struct active_vm_stat, active_vm_stats); +EXPORT_PER_CPU_SYMBOL(active_vm_stats); + +long active_vm_item_sum(int item) +{ + struct active_vm_stat *this; + long sum = 0; + int cpu; + + WARN_ON_ONCE(item <= 0); + for_each_online_cpu(cpu) { + this = &per_cpu(active_vm_stats, cpu); + sum += this->stat[item - 1]; + } + + return sum; +} diff --git a/mm/active_vm.h b/mm/active_vm.h index 72978955833e..1df088d768ef 100644 --- a/mm/active_vm.h +++ b/mm/active_vm.h @@ -3,6 +3,44 @@ #define __MM_ACTIVE_VM_H #ifdef CONFIG_ACTIVE_VM +#include + extern struct page_ext_operations active_vm_ops; + +static inline int active_vm_item(void) +{ + if (in_irq()) + return this_cpu_read(irq_active_vm_item); + + if (in_softirq()) + return this_cpu_read(soft_active_vm_item); + + return current->active_vm_item; +} + +static inline void active_vm_item_add(int item, long delta) +{ + WARN_ON_ONCE(item <= 0); + this_cpu_add(active_vm_stats.stat[item - 1], delta); +} + +static inline void active_vm_item_sub(int item, long delta) +{ + WARN_ON_ONCE(item <= 0); + this_cpu_sub(active_vm_stats.stat[item - 1], delta); +} +#else /* CONFIG_ACTIVE_VM */ +static inline int active_vm_item(void) +{ + return 0; +} + +static inline void active_vm_item_add(int item, long delta) +{ +} + +static inline void active_vm_item_sub(int item, long delta) +{ +} #endif /* CONFIG_ACTIVE_VM */ #endif /* __MM_ACTIVE_VM_H */ From patchwork Mon Dec 12 00:37:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13070696 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 174CBC4332F for ; Mon, 12 Dec 2022 00:37:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7898A8E0007; Sun, 11 Dec 2022 19:37:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 73A248E0002; Sun, 11 Dec 2022 19:37:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5DBA48E0007; Sun, 11 Dec 2022 19:37:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4B9AA8E0002 for ; Sun, 11 Dec 2022 19:37:54 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 19E1F1A06BE for ; Mon, 12 Dec 2022 00:37:54 +0000 (UTC) X-FDA: 80231791668.04.97E19AF Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf17.hostedemail.com (Postfix) with ESMTP id 7C4C940006 for ; Mon, 12 Dec 2022 00:37:52 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=blDmjwOG; spf=pass (imf17.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670805472; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/FpiiW8tlCQuBLdcxwgK57oWmDCwhuuS+u14RJYhlmk=; b=upu5sSnKi3BPvbcXLzDe6prUtTugZyy/qlG0SMPmC+WuFoQTOFQtqQaCKEJxIj0/nF6LrP kllw8KdtsGntAkrFr1xCZkqF87Nt9A/VFKFVwG1cd/01H1NGkuPPqVnOJna1qmfGgTVijc wS2qRzFPBhZTBjuTVZRLhO4eqrrAcZI= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=blDmjwOG; spf=pass (imf17.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670805472; a=rsa-sha256; cv=none; b=w8lDRqoCl3yaVRGP/sDb9M3+C8BhlQN2L4Q+fUB3hArM/EXzlOgf+Vf+Sryr3MEFtho5td Itwn7zD0zgq8xKnIy9YcVIFF4byh56gctzoJFbeZOnvRIxiOkegotolTIYbyoS6YKU0Mr6 nyyVkx1Zv7XURELGPrG++3jpi1A/7ok= Received: by mail-pl1-f174.google.com with SMTP id 4so4460066plj.3 for ; Sun, 11 Dec 2022 16:37:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/FpiiW8tlCQuBLdcxwgK57oWmDCwhuuS+u14RJYhlmk=; b=blDmjwOG5yBpOKGDu4OqOJfK6YDC2tIgsaA++wSa0spxRtLKgIr7tNI1/nRT1GOLtp kmICcOuS0tuAzHuhsh3L4tSLzOkTgZDXK8KlHTw1Zulv2p2rFKaK5DBXCtOzTWLLXLXM nFdtzntL+7qlYziM7IZ6Z9Eu20pf9537iCQMLsJ0nKLZ5Z4nBKTo3jKnKyTXIkZWEMWK uToyEoyYinHFpycXiVUdruUq96vWChVYpvMHF5dpYLdnhpth42d/gxX097KBpmpTJCFt EHawh4kk5/jSb54tQVSkfcXVjAF6va6Pb8fF0SIuMb5dMRpI7dzXrhcTgE9Jtr+vFpZy 8VbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/FpiiW8tlCQuBLdcxwgK57oWmDCwhuuS+u14RJYhlmk=; b=HW5bzaMqMaCRNKYn2Q+mjPNov5hlKUn8A0h8o8vPoxS52QYb52XmoE9XFzf3dUi8YU yiUi9EO71bZqFaiAnAkQhp6ZpYeupj0vKanlNy+D/pCGvINIdeTmLZURIpAdoT/z5KA7 kdotjKMEpbfFEVJ3gS3EYNFi8tHUbg8Djox9hCHrm35uWJqd32ovIUPI1SLBi/9HaYts snwzfsxxbMvpgQP6jvHYsCzvwVZUaYm2RSWNg2tBYKbS6rndBDngrhGcETvB2BHnTWe9 mMoW8KY0O2p4QLrxADCP2ow8nsVJGGN+Xu5vzp0qmnoYW0PPrzd3boG3q3nrd84PwBLb 348Q== X-Gm-Message-State: ANoB5pkJLxIySQH+dWC9sWEavikvZOfoasUujn+Xt2EYmMuqTIpNrqWp EesIeqOGjWl8wA+XzD6RAsk= X-Google-Smtp-Source: AA0mqf6sGfx5c19Rrfw5tyzmw9AMZXV+iZ/hNtnw3UdNnuW5oRiZL8CBPsG6dFQMVoMUC2lhjL16RA== X-Received: by 2002:a17:902:b58c:b0:189:184d:f0ff with SMTP id a12-20020a170902b58c00b00189184df0ffmr13651625pls.11.1670805471482; Sun, 11 Dec 2022 16:37:51 -0800 (PST) Received: from vultr.guest ([2001:19f0:7002:8c7:5400:4ff:fe3d:656a]) by smtp.gmail.com with ESMTPSA id w9-20020a170902e88900b00177fb862a87sm4895960plg.20.2022.12.11.16.37.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 11 Dec 2022 16:37:50 -0800 (PST) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next 3/9] mm: percpu: Account active vm for percpu Date: Mon, 12 Dec 2022 00:37:05 +0000 Message-Id: <20221212003711.24977-4-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20221212003711.24977-1-laoar.shao@gmail.com> References: <20221212003711.24977-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 7C4C940006 X-Stat-Signature: qgeh9na5afozwacaz1u534id4hupe4qr X-Rspam-User: X-HE-Tag: 1670805472-475186 X-HE-Meta: U2FsdGVkX19ImJ8OSX7KNCam6h/5+MqcA3i/kQsVDVOgscFT1vpSchd/hVKkDK0lN3MUnEJedi1BOiVzdvttzzHSYlIABUWegOl5Ds58eV/aJdlHIAB8HesVVbqw1MFDE/Fr3RZ7r5jCKyZQSJNIrahBBITFojF+fmnmuNJtqlTY/yufquN4AxQdWKZTEpiIR32BxpKtPOP7m7OtnFH2N0WozgMVAny8vgA+ChLlyL05A+coUo08S3DeHTzZBdgdrgRAjQr7iHfgguEfCKxVjbD2VfEEsaReKWc3Z5I3UfVarKW8vL+sAgkebjLlgaQvEMklldQPCyituibrvv5zMXaAIO5agBuae423vcNRwiQ7KXEPsM86yyt6UD2Kz0cM07Itm3hUqu2BZL4hWGE/n/NryQgUC4WM2hFtGwEZxMyuo1rDnwqZyMRrbrfXeHXrO6HbI4k0FWdv94z5XHPC5NH6kCYObsMGvq/Vxg24tb54fM/4haHG8uF4zq72YLc0p7Y9OquXvVNl3I4zUYuL23kivt3tKtIxuklGYDrX9sTEdOSIWlZdK3x3GmA8avxGeI22FOAYn8oFqMpEkczcLDaoqVg+L/ppjs7ERpXA4HycFmfCbps0M2a/yMsh9N7XjQcHR2jZj1eo/ZlZYySeqwpBAMwbZgE1bJ9UCkw+XgWgKjfLSjDMbryiftWSP54yWV+AXDs6V9PlLov0SaFE2+NM3oVjwHUXLcMIUdTa8QikY4lLIzDlRMBJau60THNHsbVqZIUHQpJPVjQ/FSsTMKiPueTaFbIxNdf1pLtM9yFxhAzHiEELIdU+B98pBN+kjEi8N6IskAHk4L3bIXMLR0oQXppBrUtVIMYpzJKxZwSUoiUpS0l1F6yjkry7WBkDn+I/yIqm3poiJ4XLov5w+PEk18S/Y9BWue94uE4gnfyLypXX2Wj33woS5mNjj1avjN8n3mNqLpZ8gxoPnGm RcWBRzmd YnLAJylc81hw2p3D42lGoORTzV2QqrJ+LvaWga3d6Fs0yJJN9qUiXunnwK65waVdA9TDdLZsb4y8DVG6kGDsuwfc5alIynGOhhW1Ll6sMaBH0VNwL6XIVSJKDthOzDqQbWYuyi0oPygNmDRjFNwpzIigeShD3g0t9LmG4qTd/4sAhtPhSrFXWbptzX4WqnknvL8ibMY7bALO2/4iY0b8FyJP8WuYYinXZGGBpeWBPdAgsjFM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Account percpu allocation when active vm item is set. The percpu memory is accounted at percpu alloc and unaccount at percpu free. To record which part of percpu chunk is enabled with active vm, we have to allocate extra memory for this percpu chunk to record the active vm information. This extra memory will be freed when this percpu chunk is freed. Signed-off-by: Yafang Shao --- mm/percpu-internal.h | 3 +++ mm/percpu.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+) diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index 70b1ea23f4d2..f56e236a2cf3 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -63,6 +63,9 @@ struct pcpu_chunk { int nr_pages; /* # of pages served by this chunk */ int nr_populated; /* # of populated pages */ int nr_empty_pop_pages; /* # of empty populated pages */ +#ifdef CONFIG_ACTIVE_VM + int *active_vm; /* vector of activem vm items */ +#endif unsigned long populated[]; /* populated bitmap */ }; diff --git a/mm/percpu.c b/mm/percpu.c index 27697b2429c2..05858981ed4a 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -98,6 +98,7 @@ #include #include "percpu-internal.h" +#include "active_vm.h" /* * The slots are sorted by the size of the biggest continuous free area. @@ -1398,6 +1399,9 @@ static struct pcpu_chunk * __init pcpu_alloc_first_chunk(unsigned long tmp_addr, #ifdef CONFIG_MEMCG_KMEM /* first chunk is free to use */ chunk->obj_cgroups = NULL; +#endif +#ifdef CONFIG_ACTIVE_VM + chunk->active_vm = NULL; #endif pcpu_init_md_blocks(chunk); @@ -1476,6 +1480,14 @@ static struct pcpu_chunk *pcpu_alloc_chunk(gfp_t gfp) } #endif +#ifdef CONFIG_ACTIVE_VM + if (active_vm_enabled()) { + chunk->active_vm = pcpu_mem_zalloc(pcpu_chunk_map_bits(chunk) * + sizeof(int), gfp); + if (!chunk->active_vm) + goto active_vm_fail; + } +#endif pcpu_init_md_blocks(chunk); /* init metadata */ @@ -1483,6 +1495,12 @@ static struct pcpu_chunk *pcpu_alloc_chunk(gfp_t gfp) return chunk; +#ifdef CONFIG_ACTIVE_VM +active_vm_fail: +#ifdef CONFIG_MEMCG_KMEM + pcpu_mem_free(chunk->obj_cgroups); +#endif +#endif #ifdef CONFIG_MEMCG_KMEM objcg_fail: pcpu_mem_free(chunk->md_blocks); @@ -1501,6 +1519,9 @@ static void pcpu_free_chunk(struct pcpu_chunk *chunk) { if (!chunk) return; +#ifdef CONFIG_ACTIVE_VM + pcpu_mem_free(chunk->active_vm); +#endif #ifdef CONFIG_MEMCG_KMEM pcpu_mem_free(chunk->obj_cgroups); #endif @@ -1890,6 +1911,17 @@ static void __percpu *pcpu_alloc(size_t size, size_t align, bool reserved, pcpu_memcg_post_alloc_hook(objcg, chunk, off, size); +#ifdef CONFIG_ACTIVE_VM + if (active_vm_enabled() && chunk->active_vm && (gfp & __GFP_ACCOUNT)) { + int item = active_vm_item(); + + if (item > 0) { + chunk->active_vm[off >> PCPU_MIN_ALLOC_SHIFT] = item; + active_vm_item_add(item, size); + } + } +#endif + return ptr; fail_unlock: @@ -2283,6 +2315,17 @@ void free_percpu(void __percpu *ptr) pcpu_memcg_free_hook(chunk, off, size); +#ifdef CONFIG_ACTIVE_VM + if (active_vm_enabled() && chunk->active_vm) { + int item = chunk->active_vm[off >> PCPU_MIN_ALLOC_SHIFT]; + + if (item > 0) { + active_vm_item_sub(item, size); + chunk->active_vm[off >> PCPU_MIN_ALLOC_SHIFT] = 0; + } + } +#endif + /* * If there are more than one fully free chunks, wake up grim reaper. * If the chunk is isolated, it may be in the process of being From patchwork Mon Dec 12 00:37:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13070697 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15845C4167B for ; Mon, 12 Dec 2022 00:38:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A78C38E0008; Sun, 11 Dec 2022 19:37:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A1FD38E0002; Sun, 11 Dec 2022 19:37:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C1E18E0008; Sun, 11 Dec 2022 19:37:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 7C8F88E0002 for ; Sun, 11 Dec 2022 19:37:59 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 4ADE7A0330 for ; Mon, 12 Dec 2022 00:37:59 +0000 (UTC) X-FDA: 80231791878.10.B9DDC58 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf24.hostedemail.com (Postfix) with ESMTP id ACF1E180012 for ; Mon, 12 Dec 2022 00:37:57 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=R1Dq44NI; spf=pass (imf24.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670805477; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QHvieJOPfKU228A/FcGpl6NQ2QtETLo/uwnKW3/KnGM=; b=Q3Ie6EMb/PEZzvuY2alPHVlp/auEdrZXrE/so5U2wAlmUvF4YrtyNjcBNig2z3FgvFGDYp DnTfL9249eGwdWYOlDIgK02ohWm8s63uOJ+4etKAfM9zlDLmce9wbzZr4mX/S8dr8t0FDp 5FGMvyKsdJc9h4fbc+YM+IOvHVCCOh0= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=R1Dq44NI; spf=pass (imf24.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670805477; a=rsa-sha256; cv=none; b=RCdEkAwber/C9abOeeEFDf8tVzrK5PydKTmIoCtKL+4yGwaOVkDrCLTp13aw+6+nLSLzbG D+8G4LS0l4bLDozyS/Cn/anS8OcPkIMmuSqJxOw5wArclH+cxnn914mR3JHfk+++WafMms /YzCUVP+NyzLYaPv/LFyaMBRAoHaHuc= Received: by mail-pj1-f43.google.com with SMTP id t17so10285326pjo.3 for ; Sun, 11 Dec 2022 16:37:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QHvieJOPfKU228A/FcGpl6NQ2QtETLo/uwnKW3/KnGM=; b=R1Dq44NIQGVO0+qq21tEHH6CncuZ3dTgE2cbw8HLO8saI8/5z0r/X0CECMdjYOgRv5 SMjSPHWZ8K2mAGRes/5tIu7FBuiyDmZfqOd6zZtvSmpU2LipHlsLhWkPY4mEV5JoKNRi BpaWYokf6VLHMRv8IXvOqTUQ1BqQZlb6F1ZsovRUuujPt0efJ9O2cbw3ryYciICij6gR W2NclCv2pqtr7MdFxnuVO1kO83DIlfndrqXivZo6F/9XzBbyauW/h1+LcLsNpSphAhP/ WgRcDWX/f/2LzKBVFNKMEqblIw2fLaqSAgkzDYeF1+tSuC9j2136W3PmA2kqcTyA5rAL 3iEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QHvieJOPfKU228A/FcGpl6NQ2QtETLo/uwnKW3/KnGM=; b=Ng9+gIBfuw7kPYm9bqC50ImpqDtZbl8EzbbDqofelvJRMcfCnPNc7dwuj1tIn7ucyL kzqn5viX+CXXdwddqgZfrHZKCPrX+/cJMLwg+t3Le7RjGYpnqdLbQlJGDsa9HLGQo/40 BxkpbmkDWtg5/7jxn3kWASxL0x9N4k5d6fFEEBjTH43oHFNryicTzpmqe+egl/B1q9Rz 6Y6RLVFDNCh3qoiOQ2pXcjKkOLYAP2dO0tSikmVolidWgYbhQFY50GbmjL0AtktrOTT+ yRfYO1UfSYdaLl3A46Dx4A1CkcszXcchm+0nET/F8lVt1HPHC82anNyn4zuY7yhKhscU /X0w== X-Gm-Message-State: ANoB5pkNgG2ZUNbcW+mUFf4+mRGlHsxkKGvTaj3Ji15jqrc777QdtKmj 4YkPQz0iBKOTvQqRPrGlVLw= X-Google-Smtp-Source: AA0mqf6GDUpbXr/OdqXwQUyMcN2xVFgzoCfs01LkdidF7uxzxZaISdRKNaWHUuHKNdtKC3oYE3Ycsg== X-Received: by 2002:a17:902:c14c:b0:189:9733:59d3 with SMTP id 12-20020a170902c14c00b00189973359d3mr14030488plj.29.1670805476623; Sun, 11 Dec 2022 16:37:56 -0800 (PST) Received: from vultr.guest ([2001:19f0:7002:8c7:5400:4ff:fe3d:656a]) by smtp.gmail.com with ESMTPSA id w9-20020a170902e88900b00177fb862a87sm4895960plg.20.2022.12.11.16.37.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 11 Dec 2022 16:37:55 -0800 (PST) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next 4/9] mm: slab: Account active vm for slab Date: Mon, 12 Dec 2022 00:37:06 +0000 Message-Id: <20221212003711.24977-5-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20221212003711.24977-1-laoar.shao@gmail.com> References: <20221212003711.24977-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: ACF1E180012 X-Stat-Signature: shy88rantrry7ezbd4eh76rwgjipsys7 X-HE-Tag: 1670805477-296656 X-HE-Meta: U2FsdGVkX1/xOkQdsN9oGJ2CEoYNuJvWeNJGZ4LmdwVRlawi+97zMlB7Mn6yHQG/jGsEjnm9d3OQY6uyapL4hNlhRGg0MY6tE99wGXlDtcIft4LZhdAq8dV7CJhVSoGbkJ3lPvJ0h9UznnkFqBvPKhItAbFxFlPhCw6NT0RHuTdHE71I8y4YvG58zxn3YMm7YoHo8SbjkeDEG9gSaRgDS9Lojs8H+SlPdOUdim2RW6gvL2zdDsWf8aOwpKCswVsBdYJJQGtMoQgBa8bIqg11NBSeNFklaBlKjDt6C8FvF61w/S2AToiomA27oeFbQcc9WgbgMuzFnNqPAW6Tkn2YVOwbA5i1rk3XAyVK37peV7EBMxOMKE4uRABaTig3Gymm5OR55KkwVxjgsV1ThJIJpnEISkEint7RrMCqfSXXY2f1bv9kvUY+hmIOIvKjZOgs9mURzBns2lnUU60/M4RydLxS/J7N8eyxgY1r2tZEcYBmTVLGBI1qT/udrLo+BSKH75zLOn/uaH4xGQK42WkzQb8xcDbeYjEl9yj63lMOaFaHH6U3X/HoWvZyRBqrUYtK1jDHVJ+eVc5Yf5ACQGC6wl2KhliFl2YMjnWr4lm2zK3Y4cvUI54oFmS+dhbZ1wwBqnt7ymlD5JVITYwIvMbFy+ygCQWidX7NSs8I4tiTE2VsjHV7+bh1WZrjYi8a8HTncDnXWkFTPAnaKLdIv7ZKVgNYa1IqXONjLdYfMUyJg+mK0h2QSJNPh9U+ljmfbkrrpzeXc0oYhG+wD9xGAH79zomm9i6Ad6Jc0YamT6QsgVga6xfL7PoA9Wc7M59DSpLWSjmBAhJFB1DnpgdHeOpHkrl3vmpkqYe2tZuLyA1bvf/p4blF+uTlPtaH5Sg2emRNOWon4bt7d7Q31ODdn2VsbY8XcEc5cC0vJCpx+doJ74u4otRz2bZ/erbaZNzd5BlXSE/nJZhWZGTsrfyBpX4 4oxYYtKg BMnkH8wiu1N7uEaKhMz61n+2DCstD3foHKDMS9HwUMu2qmkhfYge0P5K/o1Z7jc8w4U/wvnSbln3BU5X2aqNmfWZrE5B6Qy2jxDeBW3R9Jo7TgKKfe4lk21engPsj/HquXzaXAYqT8TqGvT2wdjGucPBEPE7mPyJdY3dZTNpk6GYN+7lHRFgEObsh3DLqodjNiVbKidSko7ryrTG3BDTU0973sBcEHp9DeFI2PvOjLwmVFcs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a slab object is allocated, we will mark this object in this slab and check it at slab object freeing. That said we need extra memory to store the information of each object in a slab. The information of each object in a slab can be stored in the new introduced page extension active_vm, so a new member is added into struct active_vm. Signed-off-by: Yafang Shao --- mm/active_vm.c | 111 +++++++++++++++++++++++++++++++++++++++++++++++++ mm/active_vm.h | 16 +++++++ mm/slab.h | 7 ++++ mm/slub.c | 2 + 4 files changed, 136 insertions(+) diff --git a/mm/active_vm.c b/mm/active_vm.c index 541b2ba22da9..ee38047a4adc 100644 --- a/mm/active_vm.c +++ b/mm/active_vm.c @@ -1,6 +1,11 @@ // SPDX-License-Identifier: GPL-2.0 +#include #include #include +#include + +#include "active_vm.h" +#include "slab.h" static bool __active_vm_enabled __initdata = IS_ENABLED(CONFIG_ACTIVE_VM); @@ -28,7 +33,12 @@ static void __init init_active_vm(void) static_branch_disable(&active_vm_disabled); } +struct active_vm { + int *slab_data; /* for slab */ +}; + struct page_ext_operations active_vm_ops = { + .size = sizeof(struct active_vm), .need = need_active_vm, .init = init_active_vm, }; @@ -54,3 +64,104 @@ long active_vm_item_sum(int item) return sum; } + +static int *active_vm_from_slab(struct page_ext *page_ext) +{ + struct active_vm *av; + + if (static_branch_likely(&active_vm_disabled)) + return NULL; + + av = (void *)(page_ext) + active_vm_ops.offset; + return READ_ONCE(av->slab_data); +} + +void active_vm_slab_free(struct slab *slab) +{ + struct page_ext *page_ext; + struct active_vm *av; + struct page *page; + + page = slab_page(slab); + page_ext = page_ext_get(page); + if (!page_ext) + return; + + av = (void *)(page_ext) + active_vm_ops.offset; + kfree(av->slab_data); + av->slab_data = NULL; + page_ext_put(page_ext); +} + +static bool active_vm_slab_cmpxchg(struct page_ext *page_ext, int *new) +{ + struct active_vm *av; + + av = (void *)(page_ext) + active_vm_ops.offset; + return cmpxchg(&av->slab_data, NULL, new) == NULL; +} + +void active_vm_slab_add(struct kmem_cache *s, gfp_t flags, size_t size, void **p) +{ + struct page_ext *page_ext; + struct slab *slab; + struct page *page; + int *vec; + int item; + int off; + int i; + + item = active_vm_item(); + for (i = 0; i < size; i++) { + slab = virt_to_slab(p[i]); + page = slab_page(slab); + page_ext = page_ext_get(page); + + if (!page_ext) + continue; + + off = obj_to_index(s, slab, p[i]); + vec = active_vm_from_slab(page_ext); + if (!vec) { + vec = kcalloc_node(objs_per_slab(s, slab), sizeof(int), + flags & ~__GFP_ACCOUNT, slab_nid(slab)); + if (!vec) { + page_ext_put(page_ext); + continue; + } + + if (!active_vm_slab_cmpxchg(page_ext, vec)) { + kfree(vec); + vec = active_vm_from_slab(page_ext); + } + } + + vec[off] = item; + active_vm_item_add(item, obj_full_size(s)); + page_ext_put(page_ext); + } +} + +void active_vm_slab_sub(struct kmem_cache *s, struct slab *slab, void **p, int cnt) +{ + struct page *page = slab_page(slab); + struct page_ext *page_ext = page_ext_get(page); + int *vec; + int off; + int i; + + if (!page_ext) + return; + + for (i = 0; i < cnt; i++) { + vec = active_vm_from_slab(page_ext); + if (vec) { + off = obj_to_index(s, slab, p[i]); + if (vec[off] > 0) { + active_vm_item_sub(vec[off], obj_full_size(s)); + vec[off] = 0; + } + } + } + page_ext_put(page_ext); +} diff --git a/mm/active_vm.h b/mm/active_vm.h index 1df088d768ef..cf80b35412c5 100644 --- a/mm/active_vm.h +++ b/mm/active_vm.h @@ -4,8 +4,12 @@ #ifdef CONFIG_ACTIVE_VM #include +#include extern struct page_ext_operations active_vm_ops; +void active_vm_slab_add(struct kmem_cache *s, gfp_t flags, size_t size, void **p); +void active_vm_slab_sub(struct kmem_cache *s, struct slab *slab, void **p, int cnt); +void active_vm_slab_free(struct slab *slab); static inline int active_vm_item(void) { @@ -42,5 +46,17 @@ static inline void active_vm_item_add(int item, long delta) static inline void active_vm_item_sub(int item, long delta) { } + +static inline void active_vm_slab_add(struct kmem_cache *s, gfp_t flags, size_t size, void **p) +{ +} + +static inline void active_vm_slab_sub(struct kmem_cache *s, struct slab *slab, void **p, int cnt) +{ +} + +static inline void active_vm_slab_free(struct slab *slab) +{ +} #endif /* CONFIG_ACTIVE_VM */ #endif /* __MM_ACTIVE_VM_H */ diff --git a/mm/slab.h b/mm/slab.h index 0202a8c2f0d2..e8a4c16c29cb 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -232,6 +232,8 @@ struct kmem_cache { #include #include #include +#include +#include "active_vm.h" /* * State of the slab allocator. @@ -644,6 +646,9 @@ static __always_inline void unaccount_slab(struct slab *slab, int order, if (memcg_kmem_enabled()) memcg_free_slab_cgroups(slab); + if (active_vm_enabled()) + active_vm_slab_free(slab); + mod_node_page_state(slab_pgdat(slab), cache_vmstat_idx(s), -(PAGE_SIZE << order)); } @@ -742,6 +747,8 @@ static inline void slab_post_alloc_hook(struct kmem_cache *s, kmsan_slab_alloc(s, p[i], flags); } + if (active_vm_enabled() && (flags & __GFP_ACCOUNT) && active_vm_item() > 0) + active_vm_slab_add(s, flags, size, p); memcg_slab_post_alloc_hook(s, objcg, flags, size, p); } diff --git a/mm/slub.c b/mm/slub.c index 157527d7101b..21bd714c1182 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -45,6 +45,7 @@ #include #include "internal.h" +#include "active_vm.h" /* * Lock order: @@ -3654,6 +3655,7 @@ static __always_inline void slab_free(struct kmem_cache *s, struct slab *slab, unsigned long addr) { memcg_slab_free_hook(s, slab, p, cnt); + active_vm_slab_sub(s, slab, p, cnt); /* * With KASAN enabled slab_free_freelist_hook modifies the freelist * to remove objects, whose reuse must be delayed. From patchwork Mon Dec 12 00:37:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13070698 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA50CC4332F for ; Mon, 12 Dec 2022 00:38:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 56777900002; Sun, 11 Dec 2022 19:38:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 517718E0002; Sun, 11 Dec 2022 19:38:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3B81A900002; Sun, 11 Dec 2022 19:38:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 29B598E0002 for ; Sun, 11 Dec 2022 19:38:04 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id F2CE080823 for ; Mon, 12 Dec 2022 00:38:03 +0000 (UTC) X-FDA: 80231792046.22.F59C95C Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf17.hostedemail.com (Postfix) with ESMTP id 58E2040007 for ; Mon, 12 Dec 2022 00:38:02 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=P7L3A8LS; spf=pass (imf17.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670805482; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lNKI7Uo9ceKwdJtylngktQ/nCvprH/we2VJP8jXRS4g=; b=OGud9XKKwBFYQlgZOE6poLXAYZ/kad38ZxYHL/B842D7orIh+gveFyEx0KzVGg9km1uA49 wxVDIwYYSr8ExhF91L5qmgEYaJ6HlyStbRKS3ak1xkio0yRBl6mrw7HL9j+yng+EKwTVcJ 0VxUXLGCkXfAZmx6Cz5UrognNeJpP2k= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=P7L3A8LS; spf=pass (imf17.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670805482; a=rsa-sha256; cv=none; b=C2NMdTFVxpyI10268NqKKqsd6lg8suQ9llLS8O+4BPtRjcX3ta6TpkiExRDOIrhpIbiD7N DnInXtHl0ez4PGJa6G0G9+pWqPHQEjtkOjCSXK8I6Q9sKqXzwiF2is04KKR8JHQg6XTVsQ e5CV2emhfhCCOdMTjuRT9Id0QS74I3c= Received: by mail-pl1-f174.google.com with SMTP id 4so4460397plj.3 for ; Sun, 11 Dec 2022 16:38:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lNKI7Uo9ceKwdJtylngktQ/nCvprH/we2VJP8jXRS4g=; b=P7L3A8LSQ6EuBchoo9o72G0HR9beI6ptksZv7pGabGWc21e5bYToqayU6KK7gpv+KW LNO60GB0maoylCFzxuy7ZqsXKuIBcgvGt+apx8Tmq/mfpL7XtcE8QC5AuyQB0bUSs4CV t+qTZt3ykBrEDsJ1dX7EGsQMucv4KFNDm9W2KRsa9nhIYqiyhZdcTkwfdZP9jue72YQF xGaOwfN8haEgmbtswwr6QpNh82Qg3XEOAMM94W24qrBtb/wUM72Uhhj+8XRzzxb1Zvkn tyjFOn5ZuO/uYxaYuu58vwbl3foO3tgwR9ZUZGqgmrQiON9ADCHjs8LsgoZ4WmuB+Zj+ YhpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lNKI7Uo9ceKwdJtylngktQ/nCvprH/we2VJP8jXRS4g=; b=lU/p8VpYcaGQn0lvz5eHnK2S7rmWSSpoeSSgyEAyIXDOm7SuEZ3yoEhsmOX/D8jQ4C sj4Y9K0TW0iNGEa07ri6/3Z9YqkwoZ1mrOmRAfDnFCT3jj1gQjs9vXH3wnD0dTI0VE3Y QFCaiR4AQlF3Kyup8V1wExBzOAE00MydTyiJqqFCmfnmZwK4D1dHCyJJh95WcMWob11D kef3zOArxMaJ30cX3LeP0GvrO32glbduDaxIC5+aFdl6ZAb3MN6S33jWSES72AXC+R/j IU36/SWOJR5vITvD088BEuTrVGaYI+hGkhc0y4XA5O4WhPkXOh2KE05hXtQCDn6CuP+a qdmg== X-Gm-Message-State: ANoB5plyg/8/nlGhtsB7j/XsgkZQ5jYlwC7f8iDmtViz+j1yM5se3SG+ fPkrtWQPyuVb3oifjHmNTbs= X-Google-Smtp-Source: AA0mqf6gVkVR/BLhx3chzrF6SJRugCJ3u+HzeGV/gLvVMYMtlhxlpXYv9UYQhE7PhTOyWjPolnJ7Mw== X-Received: by 2002:a17:902:7c8c:b0:185:4421:72ed with SMTP id y12-20020a1709027c8c00b00185442172edmr14327503pll.50.1670805481847; Sun, 11 Dec 2022 16:38:01 -0800 (PST) Received: from vultr.guest ([2001:19f0:7002:8c7:5400:4ff:fe3d:656a]) by smtp.gmail.com with ESMTPSA id w9-20020a170902e88900b00177fb862a87sm4895960plg.20.2022.12.11.16.37.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 11 Dec 2022 16:38:01 -0800 (PST) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next 5/9] mm: Account active vm for page Date: Mon, 12 Dec 2022 00:37:07 +0000 Message-Id: <20221212003711.24977-6-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20221212003711.24977-1-laoar.shao@gmail.com> References: <20221212003711.24977-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 58E2040007 X-Stat-Signature: ij7knf5uqg91hhhgcecfpnopycgfjxri X-Rspam-User: X-HE-Tag: 1670805482-401613 X-HE-Meta: U2FsdGVkX18boijMDf+WenRpgF1bUG1s5epg27Hf8xWMFjF/OSAlUtCqvAD9Ud3uLV0mzHecVQLBeeGqo6MyYkJM3Ev/kcNnMOpt9bGzTd+QrPOFMFqPUBRv027FMzt++OYQyq9RRMCwnxBbsngmlrOLsZxzFVQb+iE+YNWMMpuU9Fiyw+JyiTNSMLVKaVYFGFN7423B7lSBWMBC4BMffbu1eWd5OizJVYWvGkWItw90h4dPJW1KHGRteWRJPABTk+Qg+9vWTL2TlYxW0aonViIFWV18ppHshwkZv/ZsToU+5TkRPE4Ikd8KTWyhrT+jtn094yC4EcomLngUg9izEH5DpeycC+4b3elWhrC15cHYyJ4zRtf3RcqBPzS2c8V7NTULusYDg+SACCDmIfDBDsjF3VSlp8cdUIJXisQSrbOTuj68nyB/wu/o1cAer3OPchnjgFuGjkoget5ijXp7ELI0x3pS1Mgmc5txGSnnvLKS5m0E+v/q+qtlBUQavhfJO7NsfWabXYHCaba6T4+7pjOUNdH50A73Xao2QPsDeJA/E9Az7felNs8OyE0Gr6VQ0RbCfaWPlbmk0GIt9iwRww2O7ZMAyle1y3JkyV3vLw2kQwmIAot+8oioQZ/qXjYslf2PlZiXV4kyZKfKLxq7wiLGeWDyjElViMX1h4G6BZiNWcNVSA4zG3CgoP99uLCHooqlsBbPi0A+hu5iWimcVbqWc0r90zkT0lZin5+snMu6wdLZ4fj8bB/6OwUDOZ5jTeIx6WzNIaXB6V4d5HK0/lNvDRHsfhCxW0uNQUsLgxq2e8Xc1jMPgJXv9ymatH4vGaSMtgKQqL+S16gumzIthoe9pg3+CNOWcwGs4tp/F55uP+mOLtTLSuMhagl65jmkVWv319Z6avkKp7WhqK+EMR1H9WHCw/Wb2lbfvRUzj0h+B7byBnU+TD22CRwFrzPI6azi1SovMyrhRBFTAlW WdUA+O8S o6w1+nsaXP01Mzj+SuPr/tlff4PP3LxycCl/1jBKhvipT9H1aC8WxMPigaN6IaUxmy/0XMRKguxR4TgPqX2xjJrg4WzuwXz2nFkmGvgjjaW1cSJ/D5Sptvx5eG8c1dwaEWFtCCh2KCuzzAROCW4Is8UmGX1yWFki2IkHDxpJNnVPYrIeR4BueGI1Tidkhzcj6X2ZzbwycY3rQV7S8L8+BcZIiFaRh/Y7XG074xiCxIFPfop0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Account active vm for page allocation and unaccount then page is freed. We can reuse the slab_data in struct active_vm to store the information of page allocation. Signed-off-by: Yafang Shao --- include/linux/page_ext.h | 1 + mm/active_vm.c | 38 +++++++++++++++++++++++++++++++++++++- mm/active_vm.h | 12 ++++++++++++ mm/page_alloc.c | 14 ++++++++++++++ 4 files changed, 64 insertions(+), 1 deletion(-) diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h index 22be4582faae..5d02f939d5df 100644 --- a/include/linux/page_ext.h +++ b/include/linux/page_ext.h @@ -5,6 +5,7 @@ #include #include #include +#include struct pglist_data; struct page_ext_operations { diff --git a/mm/active_vm.c b/mm/active_vm.c index ee38047a4adc..a06987639e00 100644 --- a/mm/active_vm.c +++ b/mm/active_vm.c @@ -34,7 +34,10 @@ static void __init init_active_vm(void) } struct active_vm { - int *slab_data; /* for slab */ + union { + int *slab_data; /* for slab */ + unsigned long page_data; /* for page */ + } }; struct page_ext_operations active_vm_ops = { @@ -165,3 +168,36 @@ void active_vm_slab_sub(struct kmem_cache *s, struct slab *slab, void **p, int c } page_ext_put(page_ext); } + +void page_set_active_vm(struct page *page, unsigned int item, unsigned int order) +{ + struct page_ext *page_ext = page_ext_get(page); + struct active_vm *av; + + if (unlikely(!page_ext)) + return; + + av = (void *)(page_ext) + active_vm_ops.offset; + WARN_ON_ONCE(av->page_data != 0); + av->page_data = item; + page_ext_put(page_ext); + active_vm_item_add(item, PAGE_SIZE << order); +} + +void page_test_clear_active_vm(struct page *page, unsigned int order) +{ + struct page_ext *page_ext = page_ext_get(page); + struct active_vm *av; + + if (unlikely(!page_ext)) + return; + + av = (void *)(page_ext) + active_vm_ops.offset; + if (av->page_data <= 0) + goto out; + + active_vm_item_sub(av->page_data, PAGE_SIZE << order); + av->page_data = 0; +out: + page_ext_put(page_ext); +} diff --git a/mm/active_vm.h b/mm/active_vm.h index cf80b35412c5..1ff27b0b5dbe 100644 --- a/mm/active_vm.h +++ b/mm/active_vm.h @@ -10,6 +10,8 @@ extern struct page_ext_operations active_vm_ops; void active_vm_slab_add(struct kmem_cache *s, gfp_t flags, size_t size, void **p); void active_vm_slab_sub(struct kmem_cache *s, struct slab *slab, void **p, int cnt); void active_vm_slab_free(struct slab *slab); +void page_set_active_vm(struct page *page, unsigned int item, unsigned int order); +void page_test_clear_active_vm(struct page *page, unsigned int order); static inline int active_vm_item(void) { @@ -33,6 +35,7 @@ static inline void active_vm_item_sub(int item, long delta) WARN_ON_ONCE(item <= 0); this_cpu_sub(active_vm_stats.stat[item - 1], delta); } + #else /* CONFIG_ACTIVE_VM */ static inline int active_vm_item(void) { @@ -58,5 +61,14 @@ static inline void active_vm_slab_sub(struct kmem_cache *s, struct slab *slab, v static inline void active_vm_slab_free(struct slab *slab) { } + +static inline void page_set_active_vm(struct page *page, int item, + unsigned int order) +{ +} + +static inline void page_test_clear_active_vm(struct page *page, unsigned int order) +{ +} #endif /* CONFIG_ACTIVE_VM */ #endif /* __MM_ACTIVE_VM_H */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 6e60657875d3..deac544e9050 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -76,6 +76,8 @@ #include #include #include +#include +#include #include #include #include @@ -83,6 +85,7 @@ #include "shuffle.h" #include "page_reporting.h" #include "swap.h" +#include "active_vm.h" /* Free Page Internal flags: for internal, non-pcp variants of free_pages(). */ typedef int __bitwise fpi_t; @@ -1449,6 +1452,10 @@ static __always_inline bool free_pages_prepare(struct page *page, page->mapping = NULL; if (memcg_kmem_enabled() && PageMemcgKmem(page)) __memcg_kmem_uncharge_page(page, order); + + if (active_vm_enabled()) + page_test_clear_active_vm(page, order); + if (check_free && free_page_is_bad(page)) bad++; if (bad) @@ -5577,6 +5584,13 @@ struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, page = NULL; } + if (active_vm_enabled() && (gfp & __GFP_ACCOUNT) && page) { + int active_vm = active_vm_item(); + + if (active_vm > 0) + page_set_active_vm(page, active_vm, order); + } + trace_mm_page_alloc(page, order, alloc_gfp, ac.migratetype); kmsan_alloc_page(page, order, alloc_gfp); From patchwork Mon Dec 12 00:37:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13070699 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49D0AC4332F for ; Mon, 12 Dec 2022 00:38:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E196F8E0009; Sun, 11 Dec 2022 19:38:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DC9508E0002; Sun, 11 Dec 2022 19:38:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C4CC58E0009; Sun, 11 Dec 2022 19:38:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B5CD88E0002 for ; Sun, 11 Dec 2022 19:38:09 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 92AE01A04E2 for ; Mon, 12 Dec 2022 00:38:09 +0000 (UTC) X-FDA: 80231792298.01.A4666F5 Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) by imf23.hostedemail.com (Postfix) with ESMTP id E839D140004 for ; Mon, 12 Dec 2022 00:38:07 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=QwHnrYU5; spf=pass (imf23.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.52 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670805488; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=b7mLIzH9QGa/Vp7L1frOgw2MpIN2USRO4iU649EtnEc=; b=5a0cNpFpVLhLzPEo80939//xwFn4A5r9Y9XyuOFis7tsOdItXwhYm95VdlLlJ2FaD4iCBT bcR4M3t+BfKFrqiwC7b0Il3aW98UPnHaPFjfvWld/jp80jjUDbaxtok6V1HkeO/D9c0Qa9 oLklsBofqrMEYe9Xh7zcbrz7ejgN/2o= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=QwHnrYU5; spf=pass (imf23.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.52 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670805488; a=rsa-sha256; cv=none; b=znxpo2nAfDisnzsgjyLPb9hAdi3OejlYs+TRM37sKDb3AioFY6KhyUYfINAIsxov4/FPdr JRQfADo86fsn7PAE0hpNmMSeeC5WkOv4EeGqAuKPKWKG09y0eiE7nVpekXfpK+asxEDt6a Y7q+1XP1CjxOzBPv8P/FYuWe2AnZ1rg= Received: by mail-pj1-f52.google.com with SMTP id u15-20020a17090a3fcf00b002191825cf02so10476019pjm.2 for ; Sun, 11 Dec 2022 16:38:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=b7mLIzH9QGa/Vp7L1frOgw2MpIN2USRO4iU649EtnEc=; b=QwHnrYU5dlXPi4mKazfawl3O2vLOVNWdtMU7R0MIhOaiv6hPJkTBBHO24XgIPdO/Gf 9na56WI++Cb+a/hnv0W5d43A4c9pZsDP0T6C2ZZl5O2bTxtqze9EQPUi7+XEuVneB6jT kticj3IJQ4XUgA+Z1YDssULNPsBFCtJzYxLaHCITVnk6zPI61pwxHeuvswRzdZilRwc0 G8WrVsH3guWmfGM4IyiM+dMvuNvxlqlKIxYxIRRqC/yJpCDsrbeCXRRYbki05brj76EA queFVstoRXm+39NEjEKp+U/sBWNt4j/jCl8AAQzPQO6p0E07EggCIzSsaOAdHCHnNxpH bSiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=b7mLIzH9QGa/Vp7L1frOgw2MpIN2USRO4iU649EtnEc=; b=Qtr9L68D2WZ9A8fMhVqesLJyZwzM4hxtpHYPG9Ylni1z/wHH5YCZBNkLE4ir8xWM9C dqZLLIeC455P/nyR+uV+YN4w/Cyp/aYwhPt8cl15FCzT7oHMZ84Wjvm37sJmlxp5CaoU HwrR/e47z0Z/fTODpTKO2AV+vDiqGFkNoboa1vMX1YxKYD58MsMFuhY4cJVFjjzUOqeF x7z3iBVn1jne4UO9rvZ9fMKQT8Gc8WxCGxS4KxMZaUvMRC1xm2jnylnFN7tGsz/7+sTQ AqLlIxt1OrucJ1R5vE1pT6cnWCIDBuOICCDV/T/AzC8CU80hLayCujQw29UQlYJofLVR LVrg== X-Gm-Message-State: ANoB5pmf7UpIj0oJsratbjKnUGfgqhQHShLPh2YN+Evw3OV4EGgKgXvH BOX93ohi7oP15nIkzw1AJ0I= X-Google-Smtp-Source: AA0mqf5z6xRPI9ZAMyGtUx6mm4rODaDI03F7bbR1btk/8oT+AjpT7zlHoWOWr51yN5JyyqN1xQqw+g== X-Received: by 2002:a17:902:ab54:b0:189:6bda:e98f with SMTP id ij20-20020a170902ab5400b001896bdae98fmr13586506plb.58.1670805486992; Sun, 11 Dec 2022 16:38:06 -0800 (PST) Received: from vultr.guest ([2001:19f0:7002:8c7:5400:4ff:fe3d:656a]) by smtp.gmail.com with ESMTPSA id w9-20020a170902e88900b00177fb862a87sm4895960plg.20.2022.12.11.16.38.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 11 Dec 2022 16:38:06 -0800 (PST) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next 6/9] bpf: Introduce new helpers bpf_ringbuf_pages_{alloc,free} Date: Mon, 12 Dec 2022 00:37:08 +0000 Message-Id: <20221212003711.24977-7-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20221212003711.24977-1-laoar.shao@gmail.com> References: <20221212003711.24977-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Stat-Signature: cuqszgee3bojqjqbmtnqxhpsx4m5sfbx X-Rspam-User: X-Rspamd-Queue-Id: E839D140004 X-Rspamd-Server: rspam06 X-HE-Tag: 1670805487-879977 X-HE-Meta: U2FsdGVkX198Oh/8GzXUByH746boG87Ok5IiAon6cN7nJY+oDDCVj22xfsA629cJssKUE3BVyelJUN/7DdGMVpl1OQ0UZxGrgZN4dMw9xlEzEwsXWZ76u4SOvcbymrizVOvRjI9dZn3tsZcb+hKcajwS6rU9MRhW1aV1zQwmzIb7aXvHfRwYM80LPp+mEQVek6RdeER7AneQuKojEw/QQS3ueF0ehsaUkKbcMINc7UUq78bgP03eDBn5I7Gffll88KxoLvY7clqxljiWoS1e1cFu8UnuSRogxxde6EgQsjqMeCeXQ6yomgS8REwMbdlx3ZWFwRpDNyH5k/7BsWLdSbeoQdUrRSf8KGC110baf+MH/JHBEiTeR1sfP9c9xr0jAwSnyy6s0Uk79aHKgi/JNTmU3tp6zb4ThQ8Gu2YVgY3H4VSM2+D43RpR2CS+QiVC//2o7lHehXRh+S0mOSacRrdbG6kKa+rVzFUJQuKLUQkmByV8bHae7BCfJN94xRBMJufw/w6z57UZzqLJBqZ2vxLyxei5yZb2DPWFeFqWZkfmxNKBPgvLqnoJDvTzEM0cE69mT3YSuisYInd0cKHZwt3LgvxVbVGlrb8oqy0Xi378LNs1EEztGIqFPomcp19xag+5z6Iarw6Ax6cy9N823mJpHIaeuwm/M2rAmNgcp6vCtlqrNpgdcyZGzdHhPGJRjNlHiFTjNncaOleUtXv1zV9AeEzsykNL5u65pCpz+yslh8zZOWP0TbD97U5LyDoiw5IQOaLTuAs1oCophxPmloiG+ObWNpU1j9PKplOhujHxGx2tAou5LrLwHKdsLgei0uCJR5I2HN0w4aD0HfuD2mxLs0R06ruaem6T6rRTj78b5rNkRJ+IXEe6DnlmtMa1F23AWaM3qAEfCfHpbzAOF8uY6PQtcQoudvDOl+0eubVLIIxi3A5rWV99FyXDwsosNGpoxAu4yUoTfZBxYJp WwsTJ7k3 f5aHBvjSj/GAtzzm2drwztgZxNcWlI9urVHF5vFNPoqF8EKV+FOTAjRxbNJGL7eiDsY73TxnqKMnYtwZrm9adPoGIUD0MQ3tTGabfCMIVURgAiNROACPjC2+HIde0y8/tb0fOA3Q7P3FXwdIol2FlJRbu1vp021x0AA+gGvpo30tl1/5JepkaFy2L5+zAuATAMIpbpUgxOMx7T8y3OfVMIn5YtUbyEfOTGq7XW1b9v52hFXo0sf9YtV3gzqkOPbx7dg9JhCTxZOO/LT8ZBnrCxkRh7w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allocate pages related memory into the new helper bpf_ringbuf_pages_alloc(), then it can be handled as a single unit. Suggested-by: Andrii Nakryiko Signed-off-by: Yafang Shao Acked-by: Andrii Nakryiko --- kernel/bpf/ringbuf.c | 71 +++++++++++++++++++++++++++++--------------- 1 file changed, 47 insertions(+), 24 deletions(-) diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c index 80f4b4d88aaf..3264bf509c68 100644 --- a/kernel/bpf/ringbuf.c +++ b/kernel/bpf/ringbuf.c @@ -92,6 +92,48 @@ struct bpf_ringbuf_hdr { u32 pg_off; }; +static void bpf_ringbuf_pages_free(struct page **pages, int nr_pages) +{ + int i; + + for (i = 0; i < nr_pages; i++) + __free_page(pages[i]); + bpf_map_area_free(pages); +} + +static struct page **bpf_ringbuf_pages_alloc(int nr_meta_pages, + int nr_data_pages, int numa_node, + const gfp_t flags) +{ + int nr_pages = nr_meta_pages + nr_data_pages; + struct page **pages, *page; + int array_size; + int i; + + array_size = (nr_meta_pages + 2 * nr_data_pages) * sizeof(*pages); + pages = bpf_map_area_alloc(array_size, numa_node); + if (!pages) + goto err; + + for (i = 0; i < nr_pages; i++) { + page = alloc_pages_node(numa_node, flags, 0); + if (!page) { + nr_pages = i; + goto err_free_pages; + } + pages[i] = page; + if (i >= nr_meta_pages) + pages[nr_data_pages + i] = page; + } + + return pages; + +err_free_pages: + bpf_ringbuf_pages_free(pages, nr_pages); +err: + return NULL; +} + static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) { const gfp_t flags = GFP_KERNEL_ACCOUNT | __GFP_RETRY_MAYFAIL | @@ -99,10 +141,8 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) int nr_meta_pages = RINGBUF_PGOFF + RINGBUF_POS_PAGES; int nr_data_pages = data_sz >> PAGE_SHIFT; int nr_pages = nr_meta_pages + nr_data_pages; - struct page **pages, *page; struct bpf_ringbuf *rb; - size_t array_size; - int i; + struct page **pages; /* Each data page is mapped twice to allow "virtual" * continuous read of samples wrapping around the end of ring @@ -121,22 +161,11 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) * when mmap()'ed in user-space, simplifying both kernel and * user-space implementations significantly. */ - array_size = (nr_meta_pages + 2 * nr_data_pages) * sizeof(*pages); - pages = bpf_map_area_alloc(array_size, numa_node); + pages = bpf_ringbuf_pages_alloc(nr_meta_pages, nr_data_pages, + numa_node, flags); if (!pages) return NULL; - for (i = 0; i < nr_pages; i++) { - page = alloc_pages_node(numa_node, flags, 0); - if (!page) { - nr_pages = i; - goto err_free_pages; - } - pages[i] = page; - if (i >= nr_meta_pages) - pages[nr_data_pages + i] = page; - } - rb = vmap(pages, nr_meta_pages + 2 * nr_data_pages, VM_MAP | VM_USERMAP, PAGE_KERNEL); if (rb) { @@ -146,10 +175,6 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) return rb; } -err_free_pages: - for (i = 0; i < nr_pages; i++) - __free_page(pages[i]); - bpf_map_area_free(pages); return NULL; } @@ -219,12 +244,10 @@ static void bpf_ringbuf_free(struct bpf_ringbuf *rb) * to unmap rb itself with vunmap() below */ struct page **pages = rb->pages; - int i, nr_pages = rb->nr_pages; + int nr_pages = rb->nr_pages; vunmap(rb); - for (i = 0; i < nr_pages; i++) - __free_page(pages[i]); - bpf_map_area_free(pages); + bpf_ringbuf_pages_free(pages, nr_pages); } static void ringbuf_map_free(struct bpf_map *map) From patchwork Mon Dec 12 00:37:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13070700 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88951C4332F for ; Mon, 12 Dec 2022 00:38:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 25B528E0002; Sun, 11 Dec 2022 19:38:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1E303940007; Sun, 11 Dec 2022 19:38:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0AA438E000A; Sun, 11 Dec 2022 19:38:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id EF29C8E0002 for ; Sun, 11 Dec 2022 19:38:14 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C2931407CA for ; Mon, 12 Dec 2022 00:38:14 +0000 (UTC) X-FDA: 80231792508.02.C194F79 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf16.hostedemail.com (Postfix) with ESMTP id 27274180003 for ; Mon, 12 Dec 2022 00:38:12 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=TWn91mBr; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf16.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670805493; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+Gn9WpBDEDaKUZZ8QU8b0wRBxnaqWXKyRBa/KIthXAc=; b=zDlhaf199X8IB0Sj/E9tmwV9Q3ahoZOVZr3LuAkKAv4ujVIXIwKVYozFP7ymxcbY5qdGUb UkE2ikvqM7LnI9nTq+8m6IsXPbwYdF0nA9E81hR46itdZIT4pKAC/mLDpiPZwc0J64FiXh EX45QHuKLd9nA5XSQ9KIDVym43PFu/I= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=TWn91mBr; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf16.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670805493; a=rsa-sha256; cv=none; b=Yh7Mn97gi35dn4N+rxwnc8U/vCdU5VlQq0OpCGO3tCPFC0c/p+/jTtsibf0deWwJqVx0gi WQjJg22REjZKBjv9dQiH0C0Eh68Npdd3PmXmIkuNj2VZDh+8ZUIA/ROfuwKMBsulN4fhKA tDw5d9C0H/M6VUiNhm7i1M/Dlcs/B0w= Received: by mail-pl1-f180.google.com with SMTP id g10so10410712plo.11 for ; Sun, 11 Dec 2022 16:38:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+Gn9WpBDEDaKUZZ8QU8b0wRBxnaqWXKyRBa/KIthXAc=; b=TWn91mBr7aKw/ejdDKR8YL7ViEiJcmwoPO13rsfGRzSRZE90vZVGrc6m64SxoME0rZ E9fh4oE0UPZADwjceaLGA4JaZzdz/IgvhYVJ9gkZbU1JRYUvBeN0VVWTtH9Aj110I2Tr 4fM0+PLVoB0u4oFlIsTnJqhY6GjykrCYTJW1LQaZ754Nr8tOqlv2pMi6C+kKF2+EIKno kHiEl3JnREg8e5EjK8s+1wLeoI9rT4QwKporT6WDsvXLGG3UZoLK8SJnszfUYecDX4VX kEpaWZ1S1SrCVl8ShKGHFUKaEAiSFq6nWM/byd0bd7dRnm2NTps4eMiA3pf3enuIC8qT 4edQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+Gn9WpBDEDaKUZZ8QU8b0wRBxnaqWXKyRBa/KIthXAc=; b=Ro7nnWFieWSjG8C9PcCmWBaQJLXYePYDuyESBePQ6s6PQ5Y4UnxNeCB4MkORCxFZG+ 3RvJ8oTvoYU93gdbzfPVBwtmRPCYuR7OMUw+W9fcoPWDZPjoyz0ZOdFsaKRSWnSeHiBT yfzgNnqT2g/HX/htVzNgTm8NhGmgsfEW396CZ3QYm/rGSrz4X+u1qZ4bGVQKkRX5YFLE ecyknSkhu2Rb9Bk+OoM6DIJYMiZmqX3Dqe5hrIVldhlw3l5OC40Est0jOqyjkBK5afdJ XpGNfvFND+NY2IZvbfF1v/pRjCyVmAizfFj7zSN8BBCdglzUTV8A0+vFmlMa7YTu2MrL Qj9w== X-Gm-Message-State: ANoB5plMpvUOHPdpPksCLiWwVcboyJ5gByGh0x8agNE9QVP1wRYyNHBA 6jjsYiPH8/YbNvenWOlOnw0= X-Google-Smtp-Source: AA0mqf4aXVqdAlSTQEp8cHa2whCkSMepIO+xNKgkOlasHFVjqEnA9uC4JbLlGFLM9jtxp3TZIyO5fg== X-Received: by 2002:a17:902:ed85:b0:185:441e:2d9f with SMTP id e5-20020a170902ed8500b00185441e2d9fmr12716619plj.54.1670805492169; Sun, 11 Dec 2022 16:38:12 -0800 (PST) Received: from vultr.guest ([2001:19f0:7002:8c7:5400:4ff:fe3d:656a]) by smtp.gmail.com with ESMTPSA id w9-20020a170902e88900b00177fb862a87sm4895960plg.20.2022.12.11.16.38.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 11 Dec 2022 16:38:11 -0800 (PST) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next 7/9] bpf: Use bpf_map_kzalloc in arraymap Date: Mon, 12 Dec 2022 00:37:09 +0000 Message-Id: <20221212003711.24977-8-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20221212003711.24977-1-laoar.shao@gmail.com> References: <20221212003711.24977-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 27274180003 X-Stat-Signature: 31hatixqc859xtaurgm5ts3gcno4dct4 X-HE-Tag: 1670805492-658246 X-HE-Meta: U2FsdGVkX1/6fD3OZw5MltfqIY1mOy9TWZFdIj/JNIYFwZDiZdF9S6g9xdKKy9hbn+JR0I0qDZTVN2/oN3UIWDVkqEiY7vtkLuXdrHaQIDQkxwff6Rm71E4mKWdVP+pFbQwTzfTq6s3L9wRiWHAVsYsXnxkpXQ0mCwviD+5kpFY8TUAIdidqKS4SgITIrASytaErwsuU/PqTDuJC8UVmc+mGgOrigokNLdTJqGMEP9+BDZ0iYUkiMxhIyKTYo7XyoFBi57o+iOgdlFINjuafHK0vKgp89MWNGoJuHf3ZAM0cYnAdOsLIAAOz2ee+fKE7XXk+rLO5Yo1ETAP1GLGhSm4ZhIrPhYyYif4nrRNBPA+0YqxODnju1pnVL3mqMOAnvg3p75NhHM52zKy9aw9Y46lpCqFvOEFuRnPobOisMRXopqIkRGn+VVCLZ9cFSUWwQNLvO8tzpcRYymPZqkmLJHYuCOzCNdsL8xfhUkPcqyTnlhqeWm4psRfGloMkbc5YmjXJweBcAVuKoMtx8sIUaQCbEIDdTInJj1hZTwpOhVckpQJk0PAB4l3CnrmtrJ8OzVozVrD7q+RriVj+ixnJ1vW+iSs2b0D4Kd6snUtRWCLv+TrVHhTLa3TA/u+MIC3CQz6fF0aQZ4OyPzQbHrqeyHe5eALS8RX96Hr8pjtvPFLZ+64cXfwhqWDplGI8VtXDZajUv2sntlKGgq3jAD8IO5tOsRgkdGxPCLPj3F8lWoTmhZHfHAIJUzfy5gYxyqA/cqvKsukcjf5BDsW4Zul7+k0jb5+1Tc+cy/ib8YDADadNzRZRL4aemnn9X6IFKm1P/TozoUrDm3lJBCyPuGtTNMpdr8Q8SOhqLrzfPxOlfqvD4wK8Z3zgZFN/Iv2UjwnWzhesuCMcX9rie2SJa+xSsXhHfiu8M7h9Bi76kyY/Jq/W7kKtOSRTxIknaYulv8RHWrAgE4jHcYaDpQZ6Otf ak2nT7dS HSmgvWMKb2o++k9QMu0pU0TVLpJut6j/VWIBL5wX1L3ZfW1qPZL+08k/6FZB7YOmrzIV139H2GJZXuyOwafSTwdU/i0fPm81SYcqkrYyZkdHVKgGP1cvq2LTV88ZGFRmQjM/zLRceKeB9ALpnq4bGII4yfa9okX94ZpOEqMTTUV9D3hTMKP04Ca3CsIPB3fFfLVDWlutCIuOKq8rYHXV7ndl7oP3RvnJZJSGQ3IGZbT1C4SRbGCitTEZUvagKZkKMjYGN033UA90iil5S9RGFT+MeD0I2kTEnW3Vu X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allocates memory after map creation, then we can use the generic helper bpf_map_kzalloc() instead of the open-coded kzalloc(). Signed-off-by: Yafang Shao --- kernel/bpf/arraymap.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index 484706959556..e64a4178d92d 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -1102,20 +1102,20 @@ static struct bpf_map *prog_array_map_alloc(union bpf_attr *attr) struct bpf_array_aux *aux; struct bpf_map *map; - aux = kzalloc(sizeof(*aux), GFP_KERNEL_ACCOUNT); - if (!aux) + map = array_map_alloc(attr); + if (IS_ERR(map)) return ERR_PTR(-ENOMEM); + aux = bpf_map_kzalloc(map, sizeof(*aux), GFP_KERNEL); + if (!aux) { + array_map_free(map); + return ERR_PTR(-ENOMEM); + } + INIT_WORK(&aux->work, prog_array_map_clear_deferred); INIT_LIST_HEAD(&aux->poke_progs); mutex_init(&aux->poke_mutex); - map = array_map_alloc(attr); - if (IS_ERR(map)) { - kfree(aux); - return map; - } - container_of(map, struct bpf_array, map)->aux = aux; aux->map = map; From patchwork Mon Dec 12 00:37:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13070701 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC582C4332F for ; Mon, 12 Dec 2022 00:38:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4CA87940008; Sun, 11 Dec 2022 19:38:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 47A8B940007; Sun, 11 Dec 2022 19:38:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3426A940008; Sun, 11 Dec 2022 19:38:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2136E940007 for ; Sun, 11 Dec 2022 19:38:20 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id EDAD7AAC82 for ; Mon, 12 Dec 2022 00:38:19 +0000 (UTC) X-FDA: 80231792718.10.A881967 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf29.hostedemail.com (Postfix) with ESMTP id 541B9120006 for ; Mon, 12 Dec 2022 00:38:18 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=eI2+IZqP; spf=pass (imf29.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670805498; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wzAnvaTBJovo8xlms0y3igjQiNiSLX0cTsxqHTkuwSM=; b=jOYCj7gL0ti2Axq2fd2ewjOx3nde++LUmqRNGUi78KUjuZSauJ8Kyw1nAhdVn8RVZytfXN P1rL9fDjlH0NEBgfLe7ZOVot0PLzHQvcPXYokkedA6Jyhbzwd2K4Onur3pudXYQBLGslTS XkhD5HZZJuD3/YuqaBYyM0jKxXvrKkM= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=eI2+IZqP; spf=pass (imf29.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670805498; a=rsa-sha256; cv=none; b=C0+Egdke/5RtatmbIYyqetAi8Fk8LLpGpWYmRLG+W2u8vXu3xF7WcyM9HGXVGnNU8NIgE0 wd2Oqw/HRMque6SEL3LbpVhIiGd/PW81MRQR86Nf4p+v8DCv3nI5V3LRgiffLQ2y1Wx6ID yYYXMU7nNjkYfeflMat7bYO0SOWaLBg= Received: by mail-pl1-f182.google.com with SMTP id jn7so10401423plb.13 for ; Sun, 11 Dec 2022 16:38:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wzAnvaTBJovo8xlms0y3igjQiNiSLX0cTsxqHTkuwSM=; b=eI2+IZqPnNU+fWYDGLSbjTEgzHl4BXtLMuGhA0MtZ/EVxW2o2LK1qM8+6vA0Qk6JD9 avFzijOjoV482Nc2xWtNub++FUgX2aQQeh8wyK4R2ltGlzHEFPz6p9oSU4CilG47spe/ wAMMsCWcJf7Yiupe6+CCTwSTzn1Ef6ow/O1PRSmqvhYuuo38Ez8SBGxe0QeJtxeFn7ik 3WfYXwhlyHCNbsTQXRzne+6AJytjw2lHIwee4IH7b82XSUiHCjtp7YGkBm8oxARfv6cJ aVUBb5N8UcXGeog32P/+1cJdhozcbt/PmipuOqt0dJBNY3XIGjYd4my+0OEFVlpRZdI2 BMaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wzAnvaTBJovo8xlms0y3igjQiNiSLX0cTsxqHTkuwSM=; b=4HMVrJ/MVG7h1BrAM5+VESbLhJZ4UOmd5PGbGkfZf0ga4W/R0TK9ExnSKLF/XRizs9 N5uvn4NjHgk+5WkcS6KfQVRMhWEuzZPOux0AcKh1A7nDnBMyEjsQ+6KD13i14fwyQjXc wDxtj3Y4t+DF3cdeHtMHu10GNMCBdY70/4eb1G8mwZo2L3IY5vY4hCJQXHJfG7OjWEpN LwvVAkEiqUYuMzAhZDiZ3VO+AUy7QshQ+vVx3QBhYSvj8T2+rZFsHmKSHglEQMPG3uTn Dbm6Vcjt4jEYIj32gu/BNTbBX4fkIPg3Lt3TJA+FU32MdFOW5BKz/7tghGcyq1gFd+MP XxWQ== X-Gm-Message-State: ANoB5pnH5o7vbXb8xxOy3U7dga7KpPtI56UEUeXp5qyO1zhrFGWokxcd h8HFsTzcUM5nsNla7ARcDKg= X-Google-Smtp-Source: AA0mqf4zVo0bbhsURr5t16X4g/g03JHmX3vwRlJEnZVQmsaxmoghlhj+eGYNGx0eCT7QsIDY4UOTuw== X-Received: by 2002:a05:6a21:329a:b0:a4:829f:a43f with SMTP id yt26-20020a056a21329a00b000a4829fa43fmr22545343pzb.2.1670805497345; Sun, 11 Dec 2022 16:38:17 -0800 (PST) Received: from vultr.guest ([2001:19f0:7002:8c7:5400:4ff:fe3d:656a]) by smtp.gmail.com with ESMTPSA id w9-20020a170902e88900b00177fb862a87sm4895960plg.20.2022.12.11.16.38.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 11 Dec 2022 16:38:16 -0800 (PST) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next 8/9] bpf: Use bpf_map_kvcalloc in bpf_local_storage Date: Mon, 12 Dec 2022 00:37:10 +0000 Message-Id: <20221212003711.24977-9-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20221212003711.24977-1-laoar.shao@gmail.com> References: <20221212003711.24977-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 541B9120006 X-Stat-Signature: b6pnsn5yk8dypg9r9gqwoqxti4y9fc1i X-HE-Tag: 1670805498-362245 X-HE-Meta: U2FsdGVkX1+X6BWCZF2Ztd1XIv+sdS4x0yHlwMoLZgbfwAS5Z03EXeyfiLi/2TEwtsF0qFh6Ce3tNcYDRhqdP/O6wqsnNnFP8kV0GgOKjvP9aX/xRDeWU9raKPPqg8H3oKyKK0/1mK00WW2VpTnA/rY2+4tW3YEHePfE1zfTGiNqN6cFElEGkmqcIfM4vADZE5tVSPsAJSU33JEQxAhZd9DJOGxENwu6gyTwQVuh7mitJeLD8Zb1zPWD4VE4djq1BrtMCQKTgPZkwkfy6oRaLyThW5CtcfIWmRw0fpB85+tw4P+LrXz4T7/hXTydhUmwzy4lcQkw6cgbx22biSNVWzoXTtOaCQxt0Kni1h854QAlo7h4hc8sRIyctRXk0Y5zqV0xtQtsJ6T+tC7DKDXqb/B6eNxKNqVqFNIsSOuMhBsZqC45gUMf7ch0im3Pu8lqUVsQaQL4jNjBJnBskcaOP22Dr80JKi/W1/eoIfr7qukLIqBZhoIska2iCcGd9o8DCMWh9SYW+Ldd9SEaRDgscg7GqYqpPXEslJlLw+PfjIs2JQdnfLdAwfTwbLpwtYhyKt0t1IVtbjCMBzCgHAe3k/Cpe/PmrLqPPZ9Q23Bm85IhDR+fGQw8yPs1m8Z8WicqRu0PDoPg+XGXckoKi1B+qDTN5kfmSVeAP/hUgqYoiTi4m2rsUnPhzlwcK9Dsb+0I1DLXitXtVIURxogEQlsvd7Z/XaiR8VWszpDSbqFacM+NYg90wGP0T+LP389YcYq/mnNKV5GSkbgu67e6H7CvdiapfGMextupjUlYMo1d22f3XXzO3Od2FWBdvV6ne2ZUT98k290/tUeYqwiDP6Manqm6VS8rHjqwvUKacDtck2saL0mDNjVKyuvAWWuJj237mbWayaJFERkcVRD72npc+7Eu/ClN8aQbGxm+qMWkixFKYmQavVyNLCc3z5xep0vp8/OrZEDvWdSYM90wIGc QgWLPjBd I+hgnkO3mNf2DZfeWty3/fmNQjFgMdDjBp2stBoAkoR143EOURnYEDUdZp+wuFTKahziepsPiIJY45GRzMJWzlb/QrKvgfevxhc1r7cF8Uh8fPy9GQXgZs70e9T32Gcf4lP+wY/sF6ifBAMhZCDFZi2UDTp9JEA66YITGkZjvKqilglTUARWX+/fPCd1rHeMjqbBH4u3OlEDu4RDtUI9g8Mn633re49BseURwFMpWvM1s1PRUPmfDouRitd1lrSTZyc9OM6wEOqHKMB6bZDyhD4TBHc4ujWv4jALd X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce new helper bpf_map_kvcalloc() for this memory allocation. Signed-off-by: Yafang Shao --- include/linux/bpf.h | 8 ++++++++ kernel/bpf/bpf_local_storage.c | 4 ++-- kernel/bpf/syscall.c | 15 +++++++++++++++ 3 files changed, 25 insertions(+), 2 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 3de24cfb7a3d..80aca1ef8cc5 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1873,6 +1873,8 @@ struct bpf_prog *bpf_prog_get_curr_or_next(u32 *id); void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, int node); void *bpf_map_kzalloc(const struct bpf_map *map, size_t size, gfp_t flags); +void *bpf_map_kvcalloc(struct bpf_map *map, size_t n, size_t size, + gfp_t flags); void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, size_t align, gfp_t flags); #else @@ -1889,6 +1891,12 @@ bpf_map_kzalloc(const struct bpf_map *map, size_t size, gfp_t flags) return kzalloc(size, flags); } +static inline void * +bpf_map_kvcalloc(struct bpf_map *map, size_t n, size_t size, gfp_t flags) +{ + return kvcalloc(n, size, flags); +} + static inline void __percpu * bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, size_t align, gfp_t flags) diff --git a/kernel/bpf/bpf_local_storage.c b/kernel/bpf/bpf_local_storage.c index b39a46e8fb08..0e43cecfdc07 100644 --- a/kernel/bpf/bpf_local_storage.c +++ b/kernel/bpf/bpf_local_storage.c @@ -568,8 +568,8 @@ static struct bpf_local_storage_map *__bpf_local_storage_map_alloc(union bpf_att nbuckets = max_t(u32, 2, nbuckets); smap->bucket_log = ilog2(nbuckets); - smap->buckets = kvcalloc(sizeof(*smap->buckets), nbuckets, - GFP_USER | __GFP_NOWARN | __GFP_ACCOUNT); + smap->buckets = bpf_map_kvcalloc(&smap->map, sizeof(*smap->buckets), + nbuckets, GFP_USER | __GFP_NOWARN); if (!smap->buckets) { bpf_map_area_free(smap); return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 35972afb6850..c38875d6aea4 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -470,6 +470,21 @@ void *bpf_map_kzalloc(const struct bpf_map *map, size_t size, gfp_t flags) return ptr; } +void *bpf_map_kvcalloc(struct bpf_map *map, size_t n, size_t size, + gfp_t flags) +{ + struct mem_cgroup *memcg, *old_memcg; + void *ptr; + + memcg = bpf_map_get_memcg(map); + old_memcg = set_active_memcg(memcg); + ptr = kvcalloc(n, size, flags | __GFP_ACCOUNT); + set_active_memcg(old_memcg); + mem_cgroup_put(memcg); + + return ptr; +} + void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, size_t align, gfp_t flags) { From patchwork Mon Dec 12 00:37:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13070702 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19C13C4332F for ; Mon, 12 Dec 2022 00:38:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 95DE8940009; Sun, 11 Dec 2022 19:38:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 90E14940007; Sun, 11 Dec 2022 19:38:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7AEEB940009; Sun, 11 Dec 2022 19:38:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 676A4940007 for ; Sun, 11 Dec 2022 19:38:25 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 37CDC1206F4 for ; Mon, 12 Dec 2022 00:38:25 +0000 (UTC) X-FDA: 80231792970.16.A13A7EC Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) by imf26.hostedemail.com (Postfix) with ESMTP id 98C7D140002 for ; Mon, 12 Dec 2022 00:38:23 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=gM322d7V; spf=pass (imf26.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.49 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670805503; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PuT3qM92ko5qNO6YfTKcWz9SzFB1kaIJ+qSW7zfvd0I=; b=y2yyCnPg95XydSViJf+Rpxz+xIhjgUNobMwFvQ1bgWxLtm3iiK9QHoyLxGtu2Ia765pL8M JwuRnjjyU3cxHG/rROdHRB51hkp1KoOJIvqvv9tboUPQA4xJTgS7lBctQG9D8FpMQbeyr2 vKobnfX/XK2KDlJwucF6A44iCTT5Lx4= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=gM322d7V; spf=pass (imf26.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.49 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670805503; a=rsa-sha256; cv=none; b=LQsZuZoQN8tdGYyE9DEbLprcH0MvzdE/4NsO0kislV7NRu1QUjvTQcGtJatVE02S9dIFRy TUBd/UmfZpHXrqhDIIa6Mmrjutt2MJbgdYg4165yfbidHypgIHbdar19QE560wdpw5u8qb OXdd2XgBHqdUzgBFUIe3JhupQnpM7QI= Received: by mail-pj1-f49.google.com with SMTP id t11-20020a17090a024b00b0021932afece4so13871756pje.5 for ; Sun, 11 Dec 2022 16:38:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PuT3qM92ko5qNO6YfTKcWz9SzFB1kaIJ+qSW7zfvd0I=; b=gM322d7VHc70+Y+yPUkReUGDHOAeBFOhMgXzKuk0JkfN6IOXVglSZdhh4zPFpN6mUO +OqZNeJ1Wp4vk1UoHHU19GMs3op5YQixNEJ3LUHzTMwg5j8jEY6LqFzaEAXvRxENB/Py H0nGejN4Fi62x3SVbNBuFCdHJLjwJKpvQC85hWLBAGXxE05mN4E+Rn+pLN3umlHx2etJ 0XlrD/WA5q3xQdbAEsl6N0eMr9rk1tGNN01jprIg5FztL6ELOdSLoV/5DSl0LNFakvNk SXcYpwXhexJxJlHYJ+6IvSy2tT2Nr2xFSlPwUyoSs307vM2Ux7y2ejgY4VlFtD7zQ2qM 3cXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PuT3qM92ko5qNO6YfTKcWz9SzFB1kaIJ+qSW7zfvd0I=; b=vbJCrAILVdjpFgi7RSWb6TBEakM3DjqMD15uEMni5kiOnSzKv2j2W5qUy8+d7zrC19 4yz+53X4eHvDF9rAFh6FqR1LNTD4ExDNb9rELN1UpprzrbP2YWsHn6SDlTY/OZ5xC/yI hKku38nmUa2dqJMgQ0fym6g3Zc2rHyLVhmnAWZCjKUORfzJOGd4DN0tPlpalKg4UVI3M iACXfplypTwpPRTCFVSLsjka8UOXj0YXJ0gC+zxA6DvqmSqAnwn9bBwDDnyWD1mnaEeO oE49EpPh5h6K9Lut4zAYtKKIFJAO0n1+QWmBGIQ25t8W83ofJ6ilRpCEilRzZg4AvuQ8 bqeg== X-Gm-Message-State: ANoB5pl4Anqka3OREzzihnv2t54d3z1NaMPHC9lEWM0KSkaA1LuOSMuX bmXVtXD0FngDiDNA0xzNEp4= X-Google-Smtp-Source: AA0mqf7HtLLdgXzJmiQSUymEYADGmF93YA+PMZ7qB6+IJuIV4O0kni0uKc6ddVPvmjdEaBO4DiKkaQ== X-Received: by 2002:a17:902:ea03:b0:18e:1549:9131 with SMTP id s3-20020a170902ea0300b0018e15499131mr9666025plg.23.1670805502513; Sun, 11 Dec 2022 16:38:22 -0800 (PST) Received: from vultr.guest ([2001:19f0:7002:8c7:5400:4ff:fe3d:656a]) by smtp.gmail.com with ESMTPSA id w9-20020a170902e88900b00177fb862a87sm4895960plg.20.2022.12.11.16.38.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 11 Dec 2022 16:38:21 -0800 (PST) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next 9/9] bpf: Use active vm to account bpf map memory usage Date: Mon, 12 Dec 2022 00:37:11 +0000 Message-Id: <20221212003711.24977-10-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20221212003711.24977-1-laoar.shao@gmail.com> References: <20221212003711.24977-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 98C7D140002 X-Rspam-User: X-Stat-Signature: 153bbzx5unmnsmgzewp5x7xmzb4hw6zk X-HE-Tag: 1670805503-364453 X-HE-Meta: U2FsdGVkX1/t1zutt2a/Tm4KOggbOvy24mKFTliGPZzswlqkTpAbqt5BVoG+6bAlFPctSqKzMRJboeOz9BPKSyjPa1IMFUPnV44iYUgJyRGh0gKFZ41EsTOaLY61at+vF/U85GtYfwiaXYPM5LuXfzZvV0fRmsvdLH2VQA+cb08V7HGCsEBks6VML4WTQvwoK9HttYf/Q1cYUTY/dO0mbijM7RtiR/cr/eQzr5KMIH5F2KIX1dH6BM+unkw7P4Sxtib/sAyqGyKbpScYEXZa1fqO2NoqGXCykEj7cVrUZeOslivCPMPKSGUsDM63Nu4/dAoqCjOBulN9MC3o52U4xBMY1AS4RMsVQxpxdcy5DNb2qTSDCf+5nSx3dajm0X5N1H8eCO1pVDPhZD335YdmK57e1o//FgLkAHDPUcRjM+iwcNqV3eUfUXm97bU162yUafg+VV7+gCX2TyzY7Yl6g29cPjzvsZwM4uVwIHYeFIWRTYyH20MroH2bzU/p416d3mh9PGo6k9lS7i6rXCGbJIlKNRI7z4SExqfqiVkxLEZEnA4/jv6NLRyYOZqGdvpUJb2H5pmDrpdSf+Kdha6xsos5Hb9drnfvlS0cweYJHLNvEdhqYgkFfQmdTUb688hqKaDvpuOoBTCd4ii7e6aHONtGvMF2j36Vljdrm8hILG/9ThT5Z6QgKpHYddqV0MHidt8aRbVrxG0Jumx/S3mOYnoc2fqEfTKsKMEHCVEO/bt2un4545yuntxyl3/sg+TAKj45eOjxPG8u7HuRGbvg89mj24uQHtuMzG28lujF/dzHUfkkz98lh7t+iciUaHmiypARynDtqMdYgNMIFlTFH/bS8Sh9VDqpI8LyLGvD//dFPZhycYhs+47LgIbo0PjwFPcC/XC3H30GMeRtZs8NPkWaIMcCVdOOR8VRb08EosqFCG9AQzkJX8JlHdEN8jcgv1FZnNvH2tAoKw1wuxh 9M0+uZxV 1RlcVRp9D7ALsB4jN4+UAwrO8sEOPbPsTk8w8BtOcrIkGL6SjRGVsGSrej7srwd66HUAsIfRmH6MaN3XGbqKG1/09OQMBxQnCzGF2mP0S1WMe70PY/kOzSQngcyhkrvbBf1hq8zEF1+E1KnHGqdME0ySfE6r8EeDXEYno9HXbpXURXJrgoelwblNGIwhO4VbscpAgdj1eSlfuuAzlKLHUFJg4sfEnlZNS7+yhhr/yva8tyEk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The memory allocation via bpf_map_*alloc or bpf memalloc are accounted with active vm. We only need to annotate the allocation. These memory will automatically unaccount when they are freed. Signed-off-by: Yafang Shao --- fs/proc/meminfo.c | 3 +++ include/linux/active_vm.h | 10 +++++----- kernel/bpf/memalloc.c | 5 +++++ kernel/bpf/ringbuf.c | 8 ++++++-- kernel/bpf/syscall.c | 25 +++++++++++++++++++++++-- 5 files changed, 42 insertions(+), 9 deletions(-) diff --git a/fs/proc/meminfo.c b/fs/proc/meminfo.c index 440960110a42..efe1fbd6a80e 100644 --- a/fs/proc/meminfo.c +++ b/fs/proc/meminfo.c @@ -16,6 +16,7 @@ #ifdef CONFIG_CMA #include #endif +#include #include #include "internal.h" @@ -159,6 +160,8 @@ static int meminfo_proc_show(struct seq_file *m, void *v) arch_report_meminfo(m); + seq_printf(m, "BPF: %8lu kB\n", + active_vm_item_sum(ACTIVE_VM_BPF) >> 10); return 0; } diff --git a/include/linux/active_vm.h b/include/linux/active_vm.h index 21f9aaca12c4..e26edfb3654e 100644 --- a/include/linux/active_vm.h +++ b/include/linux/active_vm.h @@ -2,6 +2,11 @@ #ifndef __INCLUDE_ACTIVE_VM_H #define __INCLUDE_ACTIVE_VM_H +enum active_vm_item { + ACTIVE_VM_BPF = 1, + NR_ACTIVE_VM_ITEM = ACTIVE_VM_BPF, +}; + #ifdef CONFIG_ACTIVE_VM #include #include @@ -18,11 +23,6 @@ static inline bool active_vm_enabled(void) return true; } -enum active_vm_item { - DUMMY_ITEM = 1, - NR_ACTIVE_VM_ITEM = DUMMY_ITEM, -}; - struct active_vm_stat { long stat[NR_ACTIVE_VM_ITEM]; }; diff --git a/kernel/bpf/memalloc.c b/kernel/bpf/memalloc.c index ebcc3dd0fa19..403ae0d83241 100644 --- a/kernel/bpf/memalloc.c +++ b/kernel/bpf/memalloc.c @@ -7,6 +7,8 @@ #include #include #include +#include +#include /* Any context (including NMI) BPF specific memory allocator. * @@ -165,11 +167,13 @@ static void alloc_bulk(struct bpf_mem_cache *c, int cnt, int node) { struct mem_cgroup *memcg = NULL, *old_memcg; unsigned long flags; + int old_active_vm; void *obj; int i; memcg = get_memcg(c); old_memcg = set_active_memcg(memcg); + old_active_vm = active_vm_item_set(ACTIVE_VM_BPF); for (i = 0; i < cnt; i++) { /* * free_by_rcu is only manipulated by irq work refill_work(). @@ -209,6 +213,7 @@ static void alloc_bulk(struct bpf_mem_cache *c, int cnt, int node) if (IS_ENABLED(CONFIG_PREEMPT_RT)) local_irq_restore(flags); } + active_vm_item_set(old_active_vm); set_active_memcg(old_memcg); mem_cgroup_put(memcg); } diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c index 3264bf509c68..7575f078eb34 100644 --- a/kernel/bpf/ringbuf.c +++ b/kernel/bpf/ringbuf.c @@ -11,6 +11,7 @@ #include #include #include +#include #define RINGBUF_CREATE_FLAG_MASK (BPF_F_NUMA_NODE) @@ -107,16 +108,18 @@ static struct page **bpf_ringbuf_pages_alloc(int nr_meta_pages, { int nr_pages = nr_meta_pages + nr_data_pages; struct page **pages, *page; + int old_active_vm; int array_size; int i; + old_active_vm = active_vm_item_set(ACTIVE_VM_BPF); array_size = (nr_meta_pages + 2 * nr_data_pages) * sizeof(*pages); pages = bpf_map_area_alloc(array_size, numa_node); if (!pages) goto err; for (i = 0; i < nr_pages; i++) { - page = alloc_pages_node(numa_node, flags, 0); + page = alloc_pages_node(numa_node, flags | __GFP_ACCOUNT, 0); if (!page) { nr_pages = i; goto err_free_pages; @@ -125,12 +128,13 @@ static struct page **bpf_ringbuf_pages_alloc(int nr_meta_pages, if (i >= nr_meta_pages) pages[nr_data_pages + i] = page; } - + active_vm_item_set(old_active_vm); return pages; err_free_pages: bpf_ringbuf_pages_free(pages, nr_pages); err: + active_vm_item_set(old_active_vm); return NULL; } diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index c38875d6aea4..92572d4a09fb 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -35,6 +35,8 @@ #include #include #include +#include +#include #define IS_FD_ARRAY(map) ((map)->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY || \ (map)->map_type == BPF_MAP_TYPE_CGROUP_ARRAY || \ @@ -312,11 +314,14 @@ static void *__bpf_map_area_alloc(u64 size, int numa_node, bool mmapable) const gfp_t gfp = __GFP_NOWARN | __GFP_ZERO | __GFP_ACCOUNT; unsigned int flags = 0; unsigned long align = 1; + int old_active_vm; void *area; + void *ptr; if (size >= SIZE_MAX) return NULL; + old_active_vm = active_vm_item_set(ACTIVE_VM_BPF); /* kmalloc()'ed memory can't be mmap()'ed */ if (mmapable) { BUG_ON(!PAGE_ALIGNED(size)); @@ -325,13 +330,17 @@ static void *__bpf_map_area_alloc(u64 size, int numa_node, bool mmapable) } else if (size <= (PAGE_SIZE << PAGE_ALLOC_COSTLY_ORDER)) { area = kmalloc_node(size, gfp | GFP_USER | __GFP_NORETRY, numa_node); - if (area != NULL) + if (area != NULL) { + active_vm_item_set(old_active_vm); return area; + } } - return __vmalloc_node_range(size, align, VMALLOC_START, VMALLOC_END, + ptr = __vmalloc_node_range(size, align, VMALLOC_START, VMALLOC_END, gfp | GFP_KERNEL | __GFP_RETRY_MAYFAIL, PAGE_KERNEL, flags, numa_node, __builtin_return_address(0)); + active_vm_item_set(old_active_vm); + return ptr; } void *bpf_map_area_alloc(u64 size, int numa_node) @@ -445,11 +454,14 @@ void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, int node) { struct mem_cgroup *memcg, *old_memcg; + int old_active_vm; void *ptr; memcg = bpf_map_get_memcg(map); old_memcg = set_active_memcg(memcg); + old_active_vm = active_vm_item_set(ACTIVE_VM_BPF); ptr = kmalloc_node(size, flags | __GFP_ACCOUNT, node); + active_vm_item_set(old_active_vm); set_active_memcg(old_memcg); mem_cgroup_put(memcg); @@ -459,11 +471,14 @@ void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, void *bpf_map_kzalloc(const struct bpf_map *map, size_t size, gfp_t flags) { struct mem_cgroup *memcg, *old_memcg; + int old_active_vm; void *ptr; memcg = bpf_map_get_memcg(map); old_memcg = set_active_memcg(memcg); + old_active_vm = active_vm_item_set(ACTIVE_VM_BPF); ptr = kzalloc(size, flags | __GFP_ACCOUNT); + active_vm_item_set(old_active_vm); set_active_memcg(old_memcg); mem_cgroup_put(memcg); @@ -474,11 +489,14 @@ void *bpf_map_kvcalloc(struct bpf_map *map, size_t n, size_t size, gfp_t flags) { struct mem_cgroup *memcg, *old_memcg; + int old_active_vm; void *ptr; memcg = bpf_map_get_memcg(map); old_memcg = set_active_memcg(memcg); + old_active_vm = active_vm_item_set(ACTIVE_VM_BPF); ptr = kvcalloc(n, size, flags | __GFP_ACCOUNT); + active_vm_item_set(old_active_vm); set_active_memcg(old_memcg); mem_cgroup_put(memcg); @@ -490,10 +508,13 @@ void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, { struct mem_cgroup *memcg, *old_memcg; void __percpu *ptr; + int old_active_vm; memcg = bpf_map_get_memcg(map); old_memcg = set_active_memcg(memcg); + old_active_vm = active_vm_item_set(ACTIVE_VM_BPF); ptr = __alloc_percpu_gfp(size, align, flags | __GFP_ACCOUNT); + active_vm_item_set(old_active_vm); set_active_memcg(old_memcg); mem_cgroup_put(memcg);