From patchwork Thu Jan 12 15:53:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13098267 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD26EC61DB3 for ; Thu, 12 Jan 2023 15:53:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 685F68E0003; Thu, 12 Jan 2023 10:53:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 524F28E0001; Thu, 12 Jan 2023 10:53:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 39E408E0003; Thu, 12 Jan 2023 10:53:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 2289B8E0001 for ; Thu, 12 Jan 2023 10:53:37 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id EF6C4A094A for ; Thu, 12 Jan 2023 15:53:36 +0000 (UTC) X-FDA: 80346592032.03.69F5537 Received: from mail-qt1-f172.google.com (mail-qt1-f172.google.com [209.85.160.172]) by imf27.hostedemail.com (Postfix) with ESMTP id 51CE84001B for ; Thu, 12 Jan 2023 15:53:35 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=S4nLopOs; spf=pass (imf27.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.172 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673538815; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OeqvHlFMdqUrdKwM3stpBCHG4LxLYAKLvc5cbQKfD2M=; b=jQa9clZunA8mgnGUIQOjN3I6NVejK67nQjjYQIUrmJrgYemoRslO79XSUgL4SGapOcDAM8 0NKiEepkH/CWxQNGzFzKw5jspcOIjpPUOCV/C8U07amuT6Vf8n/pRJF5nowT2C0DoyMjIa fB7UOm0fb7cGWwG5fW/75Exra+Vk1fA= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=S4nLopOs; spf=pass (imf27.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.172 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673538815; a=rsa-sha256; cv=none; b=4QwkKBhueXc5O9SN6EOA0YGUVWOsES6T8S4CoRpiiD4VtCTMFSM+MNNHaFY0wJXpDHvxbG isW2K2uB/gg1myg0L+z+B6uXM7InZcj3GyUIFxsr4xH7J7acWhClqlNUVegXctL88OcZby tjQEm7LLRgfX6iWgCltVdhjvCEqCL/8= Received: by mail-qt1-f172.google.com with SMTP id jr10so9499939qtb.7 for ; Thu, 12 Jan 2023 07:53:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OeqvHlFMdqUrdKwM3stpBCHG4LxLYAKLvc5cbQKfD2M=; b=S4nLopOsJYZ3zGIfDTyNZtVttayMeAqM2OAGWJeYz+rJMZVqUtVZ0qeVdz0pGGdEYp RzXgIUgsGXuNW8ou5ahVgC9fDgvc/45xslOlfcZli/PwOeMzS+n1kz+fmj3tjM/kLFtc nb4GiTTMVuRQ1ZWPYviO0Uist+d648cnZJPgSW46ZSvXWmr8vzq9TZ6460qWupRfItmj QKGQLPzzCq/dWgpurqXSuDZFF9iwxaklOhCOG10RsQ45JSoQ++tBJCXDzXdGERYUaad8 OIIXmK+OLBSqpMmBhOFcKxnWNUTcRJNYwEm/1BdWY4tAKYsaLSIrvLhwbcTxUT0bgNvL vtiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OeqvHlFMdqUrdKwM3stpBCHG4LxLYAKLvc5cbQKfD2M=; b=kGSDMRG1sUv6+4UzJ/9N42pDyYICT/8b5fXYQ4xkaOqMEh0X7NSv1ZzbGfdT91RKg4 tI5r8UxXCh0pY3/A0Xl1T+S1rSa1X49GYsH+zFSMNuuOLESURaPTyBN06hsWvYH6x9Iy TIOU/aWeUnNH8gUiAA9A/zaD9UANCpMZyeJXof+zEl2KC72DQ7bvG/31+AWwpT5auAjK iy509amkI59HZaen4eq1DHW4sVyOCOdoaJWSGfDeBaNKFmXjAwqNqRSTQrDMHvFFT3+y FFyJ4qX9FibDASv1ZZXPtqpunqIEXnB+YdHs6rVmGbfYWaFzckGSQfl0dbeKcmaHk1o4 oFnQ== X-Gm-Message-State: AFqh2koIIn56zg3YIN5HabJfxspy0gpnlBUrqOhHZP5p+lvmHnnZpRPK 9xD9gTNms4PkOZOVbYH2nl0= X-Google-Smtp-Source: AMrXdXsWmJxHSdOEBgnVEWIwQ0w62GOgHxRERmvpZuOPy7H7cm6CcasScRmQoJH25sfjpEi+fxKE+Q== X-Received: by 2002:ac8:12ca:0:b0:3a8:2d6:521e with SMTP id b10-20020ac812ca000000b003a802d6521emr15089349qtj.37.1673538814596; Thu, 12 Jan 2023 07:53:34 -0800 (PST) Received: from vultr.guest ([173.199.122.241]) by smtp.gmail.com with ESMTPSA id l17-20020ac848d1000000b003ab43dabfb1sm9280836qtr.55.2023.01.12.07.53.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jan 2023 07:53:33 -0800 (PST) From: Yafang Shao To: 42.hyeyoo@gmail.com, vbabka@suse.cz, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, roman.gushchin@linux.dev Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao , Vasily Averin Subject: [RFC PATCH bpf-next v2 01/11] mm: percpu: count memcg relevant memory only when kmemcg is enabled Date: Thu, 12 Jan 2023 15:53:16 +0000 Message-Id: <20230112155326.26902-2-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230112155326.26902-1-laoar.shao@gmail.com> References: <20230112155326.26902-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: 5gpyj984o5wnnshi5rwk9z8basziigmg X-Rspamd-Queue-Id: 51CE84001B X-HE-Tag: 1673538815-560762 X-HE-Meta: U2FsdGVkX1+Lq1ThZJa/5leXkVRbzLoNYM4tetwuSr12IRHWKNSFx+HhZS6bSnS1yJ2bttbKZAxhQ8QeJnJ6iL0ElgxLbZcISwLX4F5wk4LePjcpNgN7ZKg867vi66OxaeF+2MUmwHBDGTXJzu7ZZVBPTdu+8xLtVH27WbZzFkjm1tUescpc4Pt27WzWfdpTYPPT7Y+jWXSYBPE6wTR1eYlUrCaonuwHm6Js9ZYdx7F6vgkHW8w8xWnMPGHjl8yHul/nSoQ3mKC7Chpn7OfDiXll97mvKrs1a1ss7LPbAG262kpL/IKuLgswc/0T1SQLNs2jrnKpUkGDQKcemA3j7/ROhTesoerO+GMfLzGc2SC89JPv5NeMxDXNU2kMtBhsxa0t2dMS/h79orEUdhFyTNBXmnZ0B3hm/uE4DcOv/ViXAl9Tgmxy3+xB18575aqD/dWP/dXlbj6dDQI0E0Uuwzbs8hXwsfd8KDj8bMloACkwcc53JTlH6rpyKSXyjUMZV194xbuWT518tR9pRq4ICpG6ZUmjqoIBb2ok7Kd8HD56FcsNyxsc1EYnbEeIeUX9i1xGuSL/L/4Ic69UWf6h+Gbxm2igjF2Vlx7HIoog3g8wRmAtxXIaf40p140dr3VuchdtK72dAU2A+NCtBz7FWpotfcekt5KVTypaujwO1AaYhVGcaCnvjhBTW1I90PxNvbr3f6z9MuJeXlTWQJjwtU8zjZds0net75ysHob8uNF07rBQH0De7EAZ8KT2FRwg6Ur69bWXKXIp6pwW+N/jinusPxW62U+WKYSEZS+Vng4QXjJ8hyvD9aZxh1+h3KoF0+yfdMmNHITWs+IYzGwPohW118mU+NkMgUidIcvNhw/fRRx3wwLEQ7p5zhCis/Ls2rpJJ8sJzoEaqTjuDxUdYqBPN3AFa2OUP8PCNHHFtmGgchOHXBLRysHAGv5qpErzwhhpGUKKyznCdalo+Mg UIcpqOiQ +Q9ZZn742n6pU66Rf1tApeCVdceo9FTtNh8dpCgw70Au7AR12N5FFsfSJFMU4FISdokTSEL62s96v+ZRM4js9CC95MTNnMFJhvg+XsOwmJGKQqT9bonA0bm6iRUuffwsycKkNOXPZXOOoWtyWVgeWf0Hz3v9jhlQKGEH0FBLINU7mc6C6VgAWGRtPahZvpXOC8+ropKIdDIAl2SVlYytG9/c1TuBfrzTYtiRa5FWRdWiNFKA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The extra space which is used to store the obj_cgroup membership is only valid when kmemcg is enabled. The kmemcg can be disabled via the kernel parameter "cgroup.memory=nokmem" at runtime. This helper is also used in non-memcg code, for example the tracepoint, so we should fix it. Signed-off-by: Yafang Shao Cc: Roman Gushchin Cc: Vasily Averin --- mm/percpu-internal.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index 70b1ea2..2a95b1f 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -4,6 +4,7 @@ #include #include +#include /* * pcpu_block_md is the metadata block struct. @@ -125,7 +126,8 @@ static inline size_t pcpu_obj_full_size(size_t size) size_t extra_size = 0; #ifdef CONFIG_MEMCG_KMEM - extra_size += size / PCPU_MIN_ALLOC_SIZE * sizeof(struct obj_cgroup *); + if (!mem_cgroup_kmem_disabled()) + extra_size += size / PCPU_MIN_ALLOC_SIZE * sizeof(struct obj_cgroup *); #endif return size * num_possible_cpus() + extra_size; From patchwork Thu Jan 12 15:53:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13098266 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FD44C63797 for ; Thu, 12 Jan 2023 15:53:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A7E108E0005; Thu, 12 Jan 2023 10:53:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A07158E0001; Thu, 12 Jan 2023 10:53:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 837038E0005; Thu, 12 Jan 2023 10:53:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 67FF28E0001 for ; Thu, 12 Jan 2023 10:53:38 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 4D1E11C5D21 for ; Thu, 12 Jan 2023 15:53:38 +0000 (UTC) X-FDA: 80346592116.21.77BDB84 Received: from mail-qt1-f169.google.com (mail-qt1-f169.google.com [209.85.160.169]) by imf22.hostedemail.com (Postfix) with ESMTP id A3DF7C0004 for ; Thu, 12 Jan 2023 15:53:36 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Cs86hcYw; spf=pass (imf22.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.169 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673538816; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0GttG9cGo2fGw7Ie6+QYs6E6KpCITpSOtaS4vLsszc8=; b=5oqFgQ50uDbcEUBYdSVaxszQ18dWtrqtTK7abYeNb14xGhI1c3RqbSY7n9vFyoeNWmPkdh 57z1heMhR4WVug7ZMKL4f2XhSSnkEp6oU94UxfBGNOa+8MQFkQyJ+7XtD5F91UhWGeB7Ph cX5WMDhq2q43iIDXRJLcnQrKqjn0/ME= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Cs86hcYw; spf=pass (imf22.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.169 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673538816; a=rsa-sha256; cv=none; b=f4DIxO+vI0/HEIK3AA1dJH6WlQ3ocMTObKnWmpAby+k3IAcQ8TqVYPBXqNR5o8EEX99D8b 1tsZTdQ++HrHkNZvjOjc/BkQt8LXsAdaFu0/yOfOvkAONzZDdqWDGSf3lrESQIT15F+V2G rfiw98OluyW6fAqSoKbXfs4yOJEiGQ8= Received: by mail-qt1-f169.google.com with SMTP id jr10so9500000qtb.7 for ; Thu, 12 Jan 2023 07:53:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0GttG9cGo2fGw7Ie6+QYs6E6KpCITpSOtaS4vLsszc8=; b=Cs86hcYwXW1kyT1Ar3qnL76ZCbnEz0PuLFPljmDEP02SjXoY1Z3RjcF2o1cZcrfit5 QsRJZD+pHkF5Lp65YJU8iZVYq4d7OoijI3kadHA/0fI4edzur6B4RGyBR3E4vF662I/j wNEecnuesci1bDHq9WwQtoP5yUf7oWgsfjkZY6J0nUR1rgflYRRAS4sWd2NT04xkGNd1 biHCt1RQu1AIGF++m/rRURQXtxdvBzCNjbZDBLpYJd7CRbWoeaX/X1UlYq/ECAWaQ3II //PX23Rf28b9O8k3iB6GI8CVbc4v8x/BmlLTWUreFOpaZbasteZ4LBpLbnySid+akRmc d17g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0GttG9cGo2fGw7Ie6+QYs6E6KpCITpSOtaS4vLsszc8=; b=ikRncnEveqR0GCEhnGiTJzX8+oniQnBSm58PGSwLVrIWQ+npPA/sqXYuNAkl61y1DB vZGUIDPfi9WR4ZKalADQZdK83KnAKd9iPpMrfaK75EkDSpPk5Ir5xjsF4iH6qTmINgaN w5EM0KBd7sGXGF944nig6mwEDiPzjDsyoTlKNkQnz+94qkMrjafgltUvADS3VH1zoqjQ Orru1SbNC/8wFmQPee5sXZAjMEJODNy9ng1Z3TNB/Qa22l+30GhSXfiMWRW9i4Fqj/5f 78Twv0HiIj8ES6K0cUv518rGEDFFztNU0kzytgbaiAUcowkcvFGTBmgOQRE8cvCRW3CA cqrA== X-Gm-Message-State: AFqh2kr3yeR7h3cqC5ewa8ce00kA8XmfWjQw9BergYU6EeH8PxytROUp nB6olnoRnS+OOlnEafO4jKU= X-Google-Smtp-Source: AMrXdXujjvIkIMxOnezoY+bqzXiyRl7Otg+3lH6xNrFMGBGSGIg4NYiYb1pdv7pwlHe5tHHIls2pwQ== X-Received: by 2002:ac8:7409:0:b0:3b1:691f:3d7a with SMTP id p9-20020ac87409000000b003b1691f3d7amr5867423qtq.68.1673538815891; Thu, 12 Jan 2023 07:53:35 -0800 (PST) Received: from vultr.guest ([173.199.122.241]) by smtp.gmail.com with ESMTPSA id l17-20020ac848d1000000b003ab43dabfb1sm9280836qtr.55.2023.01.12.07.53.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jan 2023 07:53:35 -0800 (PST) From: Yafang Shao To: 42.hyeyoo@gmail.com, vbabka@suse.cz, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, roman.gushchin@linux.dev Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next v2 02/11] mm: percpu: introduce percpu_size() Date: Thu, 12 Jan 2023 15:53:17 +0000 Message-Id: <20230112155326.26902-3-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230112155326.26902-1-laoar.shao@gmail.com> References: <20230112155326.26902-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Stat-Signature: apoc37gxz8773m5traswnhxtr55zcrkf X-Rspam-User: X-Rspamd-Queue-Id: A3DF7C0004 X-Rspamd-Server: rspam06 X-HE-Tag: 1673538816-802064 X-HE-Meta: U2FsdGVkX19uVNIYFm1RXmyDRz9elphZGKNd8XjvlNjKaNnS4LM/SEBg6EUUEcCpf7E6CIYVUhtQ8FSQyjRsIT6l6gzkUSLebTgpcaYOKAPMWYR4swqOSbZCx2XE56bX5Oopf6cNAgeSxH50ioPp4QQ8PJy13zPG66i8chXHSz3SBaMDxIguy8fi1JIn3rGiP4CdH9jMAg4B16ub04E+7jWEK76EV2Fy9XwknYVI5zFf8+oTKTpF+rL0rAvuxX+9Pn/dE2E4hOJ+xSaOTupZFeYJm75hOwdn1rACfLa2rwHF6YVjhMn7fmct7PFejINXurpTW0QXRTOyxbkP9jfwqSR5KEa9yJb0BsTS2tpH+UnOl7aLP9nHqXv6KQkKdXhS8Hs0DkicsZgQSWBtoUQg9MusOSpUedv+ZgAPAI2LFtCQrMnXSpgl3xUQ2Tjk15XOfkdoAk3XjK1JEbGQQ/WNOyH6vzDB94b8z9p5V6h+b9K61Y/BsUxyVrzY7VwAMQrgTfSSWTrlOwn6UmNmoSRg5h7o5xZ/3bNyXAuw19lSEu1wHffkxMQ2L2e9Y7peCI/Q3p12cdGuTsuSKricwtod+GVlOiShUkZAJeu/VMeoMak768PXHiYzrjgX9beTEbEpJu9bzQMHsxIAaleE6Tm9R4ubQ6aqVvJYNbGuAumY7Nrro+LzMuzl59eGArFSt99A5LBSp2lKku+tSaJWg9U3m1POdUfXBAN1fd3aNCM/AqUyhxX6obmfJEALF5cTGTiIyXwrwIk+eC+IctY88Zw0ulFlfEs/em+VG3iG2HsPsrVtUF2i4k+A9eDpIlE6tSFR0rf3uzvCVzB+VGRtPj3EsLT4QncITXj8+XswGixuD6KV+wKa2rxGYXFjlOlTebAJBYGsCUkaaYlG6Eh0PGtTOXx/cmCUfvvU5lBxuwYw/3Kl9eVnaZs1nZQ1ixO6speIlOOnCHsJGiN/lQ7WbZF Xvj5SLl0 4l4+I5/bc98Xysr0IkTHEufRUiS896mXB10EPZ/dSBZjFL6voE/fnCxP0GnOeNQkuwcm6m//0mIHaxG/pgsQFS42z8U5vXIf8+yGf+wHMdSKR4uHlckixWSd++L4lnOt8N8BbaeKUzglEYUC4+euJGNNR/rq/rV8SfFhOjp/sOp+LTRW6Aka8ktCEr5w7SEg1S4D0PTnpQ82ALk7bEcDP4pBbDAp608BvHnq8a7WeNdKNYHKroOlDXZtbNkR4v4fKqDPhX9adpFfSmuJySG9bx5JcNZ2HwA62wHPW X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce a new helper percpu_size() to report full size of underlying allocation of a percpu address. Signed-off-by: Yafang Shao --- include/linux/percpu.h | 1 + mm/percpu.c | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+) diff --git a/include/linux/percpu.h b/include/linux/percpu.h index 1338ea2..7be4234 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -137,5 +137,6 @@ extern int __init pcpu_page_first_chunk(size_t reserved_size, __alignof__(type)) extern unsigned long pcpu_nr_pages(void); +extern size_t percpu_size(void __percpu *ptr); #endif /* __LINUX_PERCPU_H */ diff --git a/mm/percpu.c b/mm/percpu.c index acd78da..5580688 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -2302,6 +2302,41 @@ void free_percpu(void __percpu *ptr) } EXPORT_SYMBOL_GPL(free_percpu); +/** + * percpu_size - report full size of underlying allocation of percpu addr + * @ptr: pointer to percpu area + * + * CONTEXT: + * Can be called from atomic context. + */ +size_t percpu_size(void __percpu *ptr) +{ + int bit_off, bits, end, off, size; + struct pcpu_chunk *chunk; + unsigned long flags; + void *addr; + + if (!ptr) + return 0; + + addr = __pcpu_ptr_to_addr(ptr); + + spin_lock_irqsave(&pcpu_lock, flags); + chunk = pcpu_chunk_addr_search(addr); + off = addr - chunk->base_addr; + bit_off = off / PCPU_MIN_ALLOC_SIZE; + + /* find end index */ + end = find_next_bit(chunk->bound_map, pcpu_chunk_map_bits(chunk), + bit_off + 1); + spin_unlock_irqrestore(&pcpu_lock, flags); + + bits = end - bit_off; + size = bits * PCPU_MIN_ALLOC_SIZE; + + return pcpu_obj_full_size(size); +} + bool __is_kernel_percpu_address(unsigned long addr, unsigned long *can_addr) { #ifdef CONFIG_SMP From patchwork Thu Jan 12 15:53:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13098268 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2171C54EBC for ; Thu, 12 Jan 2023 15:53:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3796A8E0006; Thu, 12 Jan 2023 10:53:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B4928E0001; Thu, 12 Jan 2023 10:53:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0B84C8E0006; Thu, 12 Jan 2023 10:53:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E00018E0001 for ; Thu, 12 Jan 2023 10:53:39 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id BD9A81A0276 for ; Thu, 12 Jan 2023 15:53:39 +0000 (UTC) X-FDA: 80346592158.09.443C2A0 Received: from mail-qt1-f179.google.com (mail-qt1-f179.google.com [209.85.160.179]) by imf03.hostedemail.com (Postfix) with ESMTP id 2FC8020015 for ; Thu, 12 Jan 2023 15:53:38 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=ALz+sfZN; spf=pass (imf03.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.179 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673538818; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mVUzU5Lm9v4TwCr8MR19Sbma13E60mByaOAa7yrtf24=; b=688NJjWP16YvegOf5HrtGlhbrfBhydtuyo/xBnZMRdXiG8AUfxAoo2GHY6pv2ppRLEzAI3 ehoJsB3IcUrSWjKMxC+Bl2idtst1Y/H2dw2O779jAmH1CCE+tJalONqz11O+fY93rjvpg4 ZwLhlQzQ/6UTeMVP93POaiY3VVQh6mg= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=ALz+sfZN; spf=pass (imf03.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.179 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673538818; a=rsa-sha256; cv=none; b=p1ggCaxMZV3DAPPdbHYea2qWX/JVmghqv7Hlf7ujm3rVLR9D/BIG0L8pM9OGxWhXR0YpXr EpSFkZCdfPbOS9Q05hXw8CX606I6MyBcyuub7zw5LkDjJTAPVONrPmMsP0Ujh4hJKdz/OP ZZ556omZbNPhpy7VKIeWiKdNkYboJZU= Received: by mail-qt1-f179.google.com with SMTP id h21so16647020qta.12 for ; Thu, 12 Jan 2023 07:53:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mVUzU5Lm9v4TwCr8MR19Sbma13E60mByaOAa7yrtf24=; b=ALz+sfZNfS4qifRdKVjJ8liTZOnDRU+va/zE0oxhTHHmiy4uVJp/MTYMO3s8HKBBX7 kfY7l06ClrnXzuJ7SwOeoV5XP8m4rE/PpAwk2M/ze8SKWK3++rl0/fpfL9a1+7NncPo/ SsTYC3AIuXIEEPnYcGdNOIgGgXh6MnLjg7cFUrFJw3+s5Pzcdcwez0u3SEm8SWfVa8SI lV3T8Y6TP2pAb72E04aXg+cxHgxvtBaUv2MNbXhHDcG968NZCThz/0JKKSqupM5LkS5s bzBoNfaoCpdlDpOHwABYaSzg706CqymnvMJ07sqq4vcd/D+PIpKMKMaWwx7WypE2pfis NrUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mVUzU5Lm9v4TwCr8MR19Sbma13E60mByaOAa7yrtf24=; b=JO966VsE0mkgsL9EdehFnqDcCVTgBJTtNrcIhd+G1rBz43q0uHwA6+w8ccuoObPdv+ eV4L6RYbyfr27yFoe6Hf5L3sXqDRcrWRK+DqkJtWrvN54pOeigtGb7Ryyr3ufElle2Z9 R+b/KXturnD1Xcfri1biCxaXM+b9XrC3jJikOD5BUdiaepfT2AtM0efc0ahAwCrreHc6 n1B9PZmS86+lHPjjIWSrkAWpgSQiJBz5pKc9l3Xw846mXzajcqxFDVvuI3w0wecSGoDh wMuB1ijmD9fjwjZePFLEE0rZ11NN72vYfpoGJ0Micu041oxM9GFjSYWwzVQa9KcWJhlH y8eQ== X-Gm-Message-State: AFqh2kpituNoLvsMtForchyRQCcuDDkX4VyqAHfBU+Mz0Y1Q0qF//X8W 3Oc5qaUXDcFXhj8xQNEGij0= X-Google-Smtp-Source: AMrXdXsF0qX10TMukl4qhalYqOJc8cvWNp4i4Otvd5G6MW0t1u1Ue6UiuE5Z2o4FUsT/NSZGV/wCPQ== X-Received: by 2002:ac8:6c7:0:b0:3a7:f424:d1bb with SMTP id j7-20020ac806c7000000b003a7f424d1bbmr14664614qth.21.1673538817402; Thu, 12 Jan 2023 07:53:37 -0800 (PST) Received: from vultr.guest ([173.199.122.241]) by smtp.gmail.com with ESMTPSA id l17-20020ac848d1000000b003ab43dabfb1sm9280836qtr.55.2023.01.12.07.53.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jan 2023 07:53:36 -0800 (PST) From: Yafang Shao To: 42.hyeyoo@gmail.com, vbabka@suse.cz, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, roman.gushchin@linux.dev Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next v2 03/11] mm: slab: rename obj_full_size() Date: Thu, 12 Jan 2023 15:53:18 +0000 Message-Id: <20230112155326.26902-4-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230112155326.26902-1-laoar.shao@gmail.com> References: <20230112155326.26902-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Stat-Signature: 7qzsmmrex5m5r4ej9ai87mzeuxi41sf9 X-Rspam-User: X-Rspamd-Queue-Id: 2FC8020015 X-Rspamd-Server: rspam06 X-HE-Tag: 1673538818-663883 X-HE-Meta: U2FsdGVkX1+RyHTLlQ5/cBakdIe7Mu0WQk697Wl8ha1v8A7aKtCbo3Qdw2ykdbLHzKjWEsBL2zllON8SIghhcxEB1EZiL+n4n0k2CZ53ILLlG6ArjNRlQSTPyhMVrlPIVDapaqJV6hdmCURKFxlaH4bN4XsW8zzMkY1n+xmNXoSV3eNkP5LKotWeto6kp9ggoL5ovN937O4YIgQA8uguSThabtHzTGXGFWLepkViwSKcZLrSFKpCcSlupgKUGAbdlvU5i11MznF2MksHX8Fag+QQRiJyz+pX1iCy+ZxJC/iw6fH5DpA3rzKKWIIV4krpJ7EQyPL4A47yhgLZ6+TuuSR4ZGmIODldOwjGU9dOJLGOtSu7BQMBIpn72UN/Rmi2B6zcQKb2t4YDIBK14TmeDPEEhynko+jTkHL2Jalo9D2PNsUBW3G7hstL+MQ0ZbeQBfllByDhhpEkOOWIuFYeM0JR1gy2zs697n59niTuBrv5B1UFcttCA+iv948Qw3TtlZmVa3WDU1e2Hb1Dy+J4D+5MnxlDMtIKStQ5AAHZyPYK2JRMGo40pFj9CvTrwCz6HsOC9v7QnxPGFm4alx5NJ2SqiGMhVbauICvKHFaWl6Rm7GrZ8WEUEL8VsddXGZdgChP/zusp6Sed2QilTnFQjC5NTzuheDRml6GA6rt+wZkD1BKnRURLXD3S5ME/haYfHQSADXIkqkyGR8huIxPkgkY5GW+BXjj95ZTVg3tH6xGZjI30Jk2YjFFnlCNTbXsV3bBx56bz/kqU9xcyf2PIdoUL0JuctJaAcYz8h2YunD5LG+S5XqNshNk5NhCCRq9l6P8OZygIprN+tQnUm6KXDgBv7Jar35RF7egPr+zHPq2uX8bwnGBrvmJD+AY63g3ei7OyZgRKs0ogeXZxbwYs5NiRsevHFnp4k+KxlkQFvOPefR2d3ytncgjMv6HHhrOtZJfbVUQ33dFquiEcsfB iTMRUuMd 5PzrbtYy9HJaDDp5hdsPiKfQhyiySmWy3A3oktH3100SoiXGSgCJLmqEu63h+NITf2nahHricDNF0+diNf4QrhZBp4R01VviXOPPgAbpkeMoxLER5sYh1c8U5sdHIiFGo8BKcVgCAmlnx5kaLC+m5tnE3qmKG9xVJcYkklCkEgbSs8LeI8ojyhXD0KVBEhc1ZWj5MoFamltVM03R0HMVo9BSrQVHA7gYh6yOz61Cf2dz5qEc3/hmt4rRj+thFUAxHEI9gFMjnlDRziw2KfLosBq4o0q9fl4mi92NP X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The helper obj_full_size() is a little misleading, because it is only valid when kmemcg is enabled. Meanwhile it is only used when kmemcg is enabled currently, so we just need to rename it to a more meaningful name. Signed-off-by: Yafang Shao Cc: Roman Gushchin --- mm/slab.h | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index 7cc4329..35e0b3b 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -467,7 +467,10 @@ static inline void memcg_free_slab_cgroups(struct slab *slab) slab->memcg_data = 0; } -static inline size_t obj_full_size(struct kmem_cache *s) +/* + * This helper is only valid when kmemcg isn't disabled. + */ +static inline size_t obj_kmemcg_size(struct kmem_cache *s) { /* * For each accounted object there is an extra space which is used @@ -508,7 +511,7 @@ static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, goto out; } - if (obj_cgroup_charge(objcg, flags, objects * obj_full_size(s))) + if (obj_cgroup_charge(objcg, flags, objects * obj_kmemcg_size(s))) goto out; *objcgp = objcg; @@ -537,7 +540,7 @@ static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, if (!slab_objcgs(slab) && memcg_alloc_slab_cgroups(slab, s, flags, false)) { - obj_cgroup_uncharge(objcg, obj_full_size(s)); + obj_cgroup_uncharge(objcg, obj_kmemcg_size(s)); continue; } @@ -545,9 +548,9 @@ static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, obj_cgroup_get(objcg); slab_objcgs(slab)[off] = objcg; mod_objcg_state(objcg, slab_pgdat(slab), - cache_vmstat_idx(s), obj_full_size(s)); + cache_vmstat_idx(s), obj_kmemcg_size(s)); } else { - obj_cgroup_uncharge(objcg, obj_full_size(s)); + obj_cgroup_uncharge(objcg, obj_kmemcg_size(s)); } } obj_cgroup_put(objcg); @@ -576,9 +579,9 @@ static inline void memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab, continue; objcgs[off] = NULL; - obj_cgroup_uncharge(objcg, obj_full_size(s)); + obj_cgroup_uncharge(objcg, obj_kmemcg_size(s)); mod_objcg_state(objcg, slab_pgdat(slab), cache_vmstat_idx(s), - -obj_full_size(s)); + -obj_kmemcg_size(s)); obj_cgroup_put(objcg); } } From patchwork Thu Jan 12 15:53:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13098269 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6FFC2C54EBD for ; Thu, 12 Jan 2023 15:53:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B8E498E0007; Thu, 12 Jan 2023 10:53:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B1A188E0001; Thu, 12 Jan 2023 10:53:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 96B0E8E0007; Thu, 12 Jan 2023 10:53:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 78C798E0001 for ; Thu, 12 Jan 2023 10:53:41 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 3C619140616 for ; Thu, 12 Jan 2023 15:53:41 +0000 (UTC) X-FDA: 80346592242.26.E9B0392 Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) by imf27.hostedemail.com (Postfix) with ESMTP id 9D2084001B for ; Thu, 12 Jan 2023 15:53:39 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=mN5ObGrh; spf=pass (imf27.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.176 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673538819; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1PdPmTP/6nNwfuUbfoIGYl12UKrMMnWaGWGE68CKK00=; b=e/X9g8rv6luzZQbG2V/+UaPA/6FMJoXZss6s/zVsZPxChxjAe5DOjC0Zz3hl55EevZDJvJ pT4CwnDxW8D0dt44oPr2Wg7KZ8RXwp40zD2hfrGxbfU0S9q3q4X8eXruZgZigpasp3LXoJ 8RmUPU1R9qeZ0D/uj/GUUuO+qhcgke0= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=mN5ObGrh; spf=pass (imf27.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.176 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673538819; a=rsa-sha256; cv=none; b=U4LtMHpiqAQ/DmnVs2saTm0B1yGFNHUHRnKSnaA+Bi/9mPWqBWuwCjV+slHwmtFXiqeVRO nTWP15tOtT9lVbgH67V6m/FQDlU/QfzTZ/TwjwH3ykFC0557rcZis8ogPA67DzhBAarn5K q3/RilTnha7JOl/5mPTXuHirwR7h994= Received: by mail-qt1-f176.google.com with SMTP id a25so9626982qto.10 for ; Thu, 12 Jan 2023 07:53:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1PdPmTP/6nNwfuUbfoIGYl12UKrMMnWaGWGE68CKK00=; b=mN5ObGrhWrxMJkdoooArLIKC8XD0Tvow9VTzY+Dt+bM8q/RM6rw039QTNBaDeGeO3W jc5ev4ZPmIqrRhwhGdLdkzTzN8gZKgSqZNiEFBB8OAHgukMGMx+f/InoklehDY6tX4bB CUnIWUqLRKGvCWlsJSI5IP1MkFCBw1ZTcSf2C/sDXTI6RYUmopTDbtNDJP+JyheZzkFf TzLt3sLAhYi+rdZI7qQ1UV42ayBjgKZFUszSk4A+zhvgOxP4u/jn2S3xHtTb2g+P1y58 CnyYcLq2SXGRAvEUlWv58eL9NelCvX7CcHE8aby/6geeLc5bxLV+FACVmTuHiL8tHB3/ 36HA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1PdPmTP/6nNwfuUbfoIGYl12UKrMMnWaGWGE68CKK00=; b=ZcclO9lud6g2TNySSaDnrOd54Q0kBgCMLWrtkcE7R93iF0rITa9C7yynQYhpVqV3Av QUcHRtrEeRd4COLHoRFxlheDdqiHJBv2BqiMENASbh+DLSTc6o11zjUn7udK2bkaXiar sDnNlxHEULbiniPB+9NFA5W+czStozwrXGO2d7x8uNVBnYuS5ZHohqJL8/5PevKjSfdQ 0cG6voK3IDaK8FZPSORgxpKHbkPVLY5eM0cBKBxTaR1cycDL2RV55xtnNPsdc+xfF45x 2Tl8zH7EVTqaUwu1xAfPGcYX8Tm9ntnWX7QlAtkrlMTga1sWmfPBDJ7J1qhsVJiYB39P jseQ== X-Gm-Message-State: AFqh2kq+YcRj6SbhN7snyp+NdDWgbvUVG4x5v7JQucv8Rza8m3bwM4ct HwGq0/3YUfh6VB3akdpbzDA= X-Google-Smtp-Source: AMrXdXvcvv5QnOw+ucO2xICHIrhkKnumytKrPNexoUtZMl0/bJueZhRxVfPyCqg8K84Xgk088KoBmg== X-Received: by 2002:a05:622a:1741:b0:3a8:2716:ac2d with SMTP id l1-20020a05622a174100b003a82716ac2dmr149575319qtk.56.1673538818785; Thu, 12 Jan 2023 07:53:38 -0800 (PST) Received: from vultr.guest ([173.199.122.241]) by smtp.gmail.com with ESMTPSA id l17-20020ac848d1000000b003ab43dabfb1sm9280836qtr.55.2023.01.12.07.53.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jan 2023 07:53:38 -0800 (PST) From: Yafang Shao To: 42.hyeyoo@gmail.com, vbabka@suse.cz, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, roman.gushchin@linux.dev Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next v2 04/11] mm: slab: introduce ksize_full() Date: Thu, 12 Jan 2023 15:53:19 +0000 Message-Id: <20230112155326.26902-5-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230112155326.26902-1-laoar.shao@gmail.com> References: <20230112155326.26902-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 9D2084001B X-Stat-Signature: 83ds3ntiqqgwrc1sz8zwib6t7gpp5knb X-Rspam-User: X-HE-Tag: 1673538819-810261 X-HE-Meta: U2FsdGVkX1+YLBsT4gacMGnyI91VERmlxc6QnlZtYZb+sVYrtONSCEmi2D7RgjNrnoQ/C/p3jsbwR+uSvPS+whbG49ImhlC8+kOm992PHC8DszEu40ZO7imDqjJJumu6AxFCjScfB7+2wI0oL31rIusnoCONnVao3qTJugspzcelibGfVw03LyFWLiKCBr4DKU38s1nV5QuMlYw4DZkiH94Kd77cNIsKC1gRw91/TZ/FJ6bvotD1D4B0J2j3xq17MTOgPCO/eHvKPV73V4s45ewJdEG6OkWGB8xVs0aARi1yWr55N876u4VRRRRW09/5IH4jKKAq13sWBKwDrf3ElA4Ac6F7T5h/Dmgnv5YmNqwnOgejtMVNHZ8GCbqt4K+zrB0e8e+pDiQMabNqcqxVsDVYh+WmWG18FtEtETo/OVaUpxPtZ08RtOguj/7mjPKmroRmX35KKkbeyRzf0D93vKKTLiuc/rZRg5dCFin9UNh1QWNhObnlTPhxSLixDVI4AOJdRUbV159jcNFYC3bByDGdcBUZA8VZf5z1+TpGnedRqp1oBqNhC/flFXYzaAgM0LdyTGpk2SybCx/oAXKL29upub1YgvhzSLscf5jd/q+ql4XgMCSmZbBFxsZonjrYoZLvGW+WyQEtG5e+vrEBFuNhy75PiNPEpKoK4dt8B59ZV+T1xepg+TJiKS18FdqTbYy9eSkJwMgA3UfXgmfuuenrveEM5cCxK60GxcW/VJ9aHT6CNkc/4tGEcaBXL866uLm11aB+S0v52KR7spiswkOFkahRmGQ3LffSuF5+t4og9CRO/4CQ8jpX/ptn7122m8ni50Jygd1Rzzb6iLEA6TVM7nb6HtIsOMf8cPf2tUCacNAsI2Tdf0q4EFwxu1VyXVilbMT6YZ0XLLYfXCnBHrjMSaYa8El1eWbnbeysPQIZIAscEYCoFe/IoWvNQ30Y9EUQ85Rss3OZg5tVk/e kWCogU9E kwstrUzWikjzjJsKimxjQ/lRD7T2xDMxIIh2dhqj58GhAHBLGq+rZc/4tzhkNQDLTM8QrLw1+YBEzBaM/B441peJ9B+5ZWE+iK2m8b3JLMkRjPJeViMYhu3DGvvwnqLRR6rTTH+QS67W9nrHwgzrMhe2xh2Iu6/fN9WSr7Ke5t4AUTxN7iHS019bvtNiN1CUjckHNR6ZTIJfknMr9J2LM+wfaUcs6SlPgqL6IBZ4WJcNyGtxmzjBQagRPsKNVnbkvU4O6MAVhajlhXo2cnghVV9GKqIVO9grQ9pyC X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When the object is charged to kmemcg, it will alloc extra memory to store the kmemcg ownership, so introduce a new helper to include this memory size. The reason we introduce a new helper other than changing the current helper ksize() is that the allocation of the kmemcg ownership is a nested allocation, which is independent of the original allocation. Some user may relays on ksize() to get the layout of this slab, so we'd better not changing it. Signed-off-by: Yafang Shao --- mm/slab.h | 2 +- mm/slab_common.c | 52 ++++++++++++++++++++++++++++++++++++---------------- mm/slob.c | 2 +- 3 files changed, 38 insertions(+), 18 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index 35e0b3b..e07ae90 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -681,7 +681,7 @@ static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x) #endif /* CONFIG_SLOB */ -size_t __ksize(const void *objp); +size_t ___ksize(const void *objp, bool full); static inline size_t slab_ksize(const struct kmem_cache *s) { diff --git a/mm/slab_common.c b/mm/slab_common.c index 1cba98a..4f1e2bc 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1021,21 +1021,11 @@ void kfree(const void *object) } EXPORT_SYMBOL(kfree); -/** - * __ksize -- Report full size of underlying allocation - * @object: pointer to the object - * - * This should only be used internally to query the true size of allocations. - * It is not meant to be a way to discover the usable size of an allocation - * after the fact. Instead, use kmalloc_size_roundup(). Using memory beyond - * the originally requested allocation size may trigger KASAN, UBSAN_BOUNDS, - * and/or FORTIFY_SOURCE. - * - * Return: size of the actual memory used by @object in bytes - */ -size_t __ksize(const void *object) +size_t ___ksize(const void *object, bool full) { + size_t kmemcg_size = 0; struct folio *folio; + struct slab *slab; if (unlikely(object == ZERO_SIZE_PTR)) return 0; @@ -1054,7 +1044,27 @@ size_t __ksize(const void *object) skip_orig_size_check(folio_slab(folio)->slab_cache, object); #endif - return slab_ksize(folio_slab(folio)->slab_cache); + slab = folio_slab(folio); + if (memcg_kmem_enabled() && full && slab_objcgs(slab)) + kmemcg_size = sizeof(struct obj_cgroup *); + return slab_ksize(slab->slab_cache) + kmemcg_size; +} + +/** + * __ksize -- Report full size of underlying allocation + * @object: pointer to the object + * + * This should only be used internally to query the true size of allocations. + * It is not meant to be a way to discover the usable size of an allocation + * after the fact. Instead, use kmalloc_size_roundup(). Using memory beyond + * the originally requested allocation size may trigger KASAN, UBSAN_BOUNDS, + * and/or FORTIFY_SOURCE. + * + * Return: size of the actual memory used by @object in bytes + */ +size_t __ksize(const void *object) +{ + return ___ksize(object, false); } void *kmalloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size) @@ -1428,7 +1438,7 @@ void kfree_sensitive(const void *p) } EXPORT_SYMBOL(kfree_sensitive); -size_t ksize(const void *objp) +size_t _ksize(const void *objp, bool full) { /* * We need to first check that the pointer to the object is valid. @@ -1448,10 +1458,20 @@ size_t ksize(const void *objp) if (unlikely(ZERO_OR_NULL_PTR(objp)) || !kasan_check_byte(objp)) return 0; - return kfence_ksize(objp) ?: __ksize(objp); + return kfence_ksize(objp) ?: ___ksize(objp, full); } EXPORT_SYMBOL(ksize); +size_t ksize(const void *objp) +{ + return _ksize(objp, false); +} + +size_t ksize_full(const void *objp) +{ + return _ksize(objp, true); +} + /* Tracepoints definitions. */ EXPORT_TRACEPOINT_SYMBOL(kmalloc); EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc); diff --git a/mm/slob.c b/mm/slob.c index fe567fcf..8c46bdc 100644 --- a/mm/slob.c +++ b/mm/slob.c @@ -579,7 +579,7 @@ size_t kmalloc_size_roundup(size_t size) EXPORT_SYMBOL(kmalloc_size_roundup); /* can't use ksize for kmem_cache_alloc memory, only kmalloc */ -size_t __ksize(const void *block) +size_t ___ksize(const void *block, bool full) { struct folio *folio; unsigned int align; From patchwork Thu Jan 12 15:53:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13098278 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BACCC54EBC for ; Thu, 12 Jan 2023 16:00:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 017DD8E0002; Thu, 12 Jan 2023 11:00:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F0A128E0001; Thu, 12 Jan 2023 11:00:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD1CA8E0002; Thu, 12 Jan 2023 11:00:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CD24B8E0001 for ; Thu, 12 Jan 2023 11:00:10 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 33B1F1208B3 for ; Thu, 12 Jan 2023 16:00:10 +0000 (UTC) X-FDA: 80346608580.22.7DED1FF Received: from mail-vk1-f175.google.com (mail-vk1-f175.google.com [209.85.221.175]) by imf03.hostedemail.com (Postfix) with ESMTP id 6197A2001A for ; Thu, 12 Jan 2023 16:00:08 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=IZZcCnX8; spf=pass (imf03.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.221.175 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673539208; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=e5Uabr6rMB0Xp+uFEKRqPdRCxhNWiBeN0NwYKLeAixU=; b=YtokODqOJc/xZxjaPdua0z51abB6bsyzKHU60M8aMptuBa5wKrCm3s+fuQxuzw6hqmcswk vmO2PzKzMAjDk7cItHe5LGkw3XOj1eRKt4JEpjRFXNwyCtGg2IPi8z92Ik2ZDs3pSFuUf5 NZ+phStNYi8mooEAJh/vwBepvsWEeTw= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=IZZcCnX8; spf=pass (imf03.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.221.175 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673539208; a=rsa-sha256; cv=none; b=SqvLmAIUSxx7GYemJGLperALTkCgEZqQUMsDm14B87JHASu0uTSRS2dEaW0zNS0mEAW4x8 NTCM6hhkrKvM6edY8VeGzE+AsqW/ncZEwkVm7VL2kKGBMhD68TVKDRsiHf6SBJByTsGJrI MvS9z+YJa9cSxl8TX+Vo8jP0xcLHKbI= Received: by mail-vk1-f175.google.com with SMTP id v81so8944712vkv.5 for ; Thu, 12 Jan 2023 08:00:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=e5Uabr6rMB0Xp+uFEKRqPdRCxhNWiBeN0NwYKLeAixU=; b=IZZcCnX8wvwbpWe4gRFRcNAA8cfuZZ0wsr6T2tk5JqzyUrWoECk/4yqCErfaMsazk6 N7DKy/pwRR8tik0IPz2om2ZlUpPenM3WYjJKSGh/XM0KWou/YEj178TroQUGudA3Gr/0 A+wXOR5mzYBFFtuLD/Xyosv98vs84h0GPrJUuURrbn0BVmnGez0uawzzxWR/bhJ3jYWF oncdliudHrevq3hyfaG8LM68B4V9wn4EXhKt8v+tGioF2P21ja/ntPiVXqNgI3SYQF1m hXMe2o3dK6RgqFDyGs5TPySYbn19i6IgD2uZok0X+D/OLeRK3WA7uktdtx5xsJDT4zhY ZcUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=e5Uabr6rMB0Xp+uFEKRqPdRCxhNWiBeN0NwYKLeAixU=; b=wk73exWFW5b9xhqs6PC8U/8GSgQe9MB9IZT6A9RcpWTdCNbm8HxfldZJx/ksQ8ZxTm YErUBV6Q5pKtoSWWEopGICW4U1BOz0me7U74+7Rwf0wklhLtkABO5VmOQDH4c0rgAa9J 8jNAjheBUYtStb1oTFJARrz9vfYKNhK5/h4PtTiUr/lLkq+xrpH7TAwKJBGvstVMDMOa nBaRVPz7TtvFLAcrzaENe4cPvwmiLbciOXHkhwFmjoT3mgSuiyggNfINU9ki6vt6i+8z c8pgNSkydjCiMtgiq2cFMHrixTCJMV6r3i17+zedOHks8bp7KyDYY4JC/zgOoq5KZ+sB 0nvg== X-Gm-Message-State: AFqh2kpIr0HZQGyWudoQN0XHGG3vyf6OkrD71JdT0Cx5DVboJyL3Yboa I/sQ8ecE216vkhYRNyiIAjJZNdusQeaLcj1KarU= X-Google-Smtp-Source: AMrXdXteMaFBBBox4hoGYeP7OQ1QSvsy+19+ts6oymG9XgnQ4GKqRTiAV+vZ7JEEP0SB0OSyUI7vXQ== X-Received: by 2002:ac8:5197:0:b0:3ac:940c:6a6c with SMTP id c23-20020ac85197000000b003ac940c6a6cmr22208861qtn.34.1673538820190; Thu, 12 Jan 2023 07:53:40 -0800 (PST) Received: from vultr.guest ([173.199.122.241]) by smtp.gmail.com with ESMTPSA id l17-20020ac848d1000000b003ab43dabfb1sm9280836qtr.55.2023.01.12.07.53.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jan 2023 07:53:39 -0800 (PST) From: Yafang Shao To: 42.hyeyoo@gmail.com, vbabka@suse.cz, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, roman.gushchin@linux.dev Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next v2 05/11] mm: vmalloc: introduce vsize() Date: Thu, 12 Jan 2023 15:53:20 +0000 Message-Id: <20230112155326.26902-6-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230112155326.26902-1-laoar.shao@gmail.com> References: <20230112155326.26902-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 6197A2001A X-Stat-Signature: 47q8fkbkbm8kr9toxt7x5zzpqnfuamo7 X-HE-Tag: 1673539208-866253 X-HE-Meta: U2FsdGVkX1+0ju327erm4h8QJorWjsmjHjJ/t09F3j/mAXCIQ/81wOQtCST1yweV1dDEsd0RT2bvQBD+5jm+6qvlPQeSSJbkzQiW73cRUUKVymqdWcWWcB36o/XYXVgENKwam0g3uCnWH7lWoRyTihWaJl0uls/3ckCUPLDhXnoLyOv+lTHQrrtCmet1hqtOQ/E9lK2ZAUUB9OtFsRekHdl+gvf+baf4S8dGdCiWmx1pXWo1oJLbvUKPbvecznGsGupea/Uby6D8pLqRpbg/6UyMI66DnP/z3+GTKSE1ceWHa39flbrba4zKKyx9vmekE5xVvrBEZNpnO9Pld7zztH161/RentZbCOscxtnO3DnR+j0W8q711w/V7rtb/sND9yuRomFjjYo4P384LCrHgha8NJ8Bx7Tu29l4RDBbZ7T+F5br38P504pJlBDnUfLT8DXXzyiMvLt7ZkQWrw5drspwx2V+LRxy4tcoZYxO5s5xCU8RdHuyS/gr/arQaRAKJVB6RWErk5HHer6WRUKlFFr49hhcROyoMEWBSvck6Dx3rVdx9dLaS/L2yz6NCzY7R0QTwluPiht74q4TyZ1gqk5voNX9QP/+tVFtNkrrAFBc1LKBMnDWLJT1vkPBxm0IV5uC8AR2lFs8SKATOsipOP70jgr0nZHnAkNrfbW3fTQxwZB8RJ0TQK6yceqOKGApSfSo/whe9p7GTb6lL5gb1jXVk6X6wnXC2IgbdPy/DP0SU44V36QKuN3Qgve44Kfar9l898Ny/nZ1S7dXzLCavWijm8H1ddRUkLuk225W+j+OfZa7WWPP2Hl7pgqs0CJqDNAfOTnvxoojtxinzpW96zZcKe0BrYZsgs4LheA3I0WH1ylET36BiODUXBRSQJshajbeG3PTIk/aItLZIkj4togXmSDwcEACG0W79lfK96UrtPNoNTc3+HkVWubTK9RTpaCXeur9Et1osCJKb0q BXs2PHfT mTi3XOdskMJERTF7Dxp2ESVlJA9ATO8vArFus7O0O3k6qEnhNGAMKNht24vB1CYjn1rRgBZxdD5mNJ9I4THQ8pyvQSP3LgGzozrBDAL+acIe/Mo2IF32AqpfWyScKKoGndilRvqjufVNOBY4j9/vP8TMgP+fr0D0Lk0UyeV/hoBIwcam9BiAKvICBD2ruqNA1a1nBYr6JVy/8uQnuBCW2731kThNgdljPlCVX0lCyCQopF2o= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce a helper to report full size of underlying allocation of a vmalloc'ed address. Signed-off-by: Yafang Shao --- include/linux/vmalloc.h | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 096d48a..52f925f 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -297,4 +297,19 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets, static inline bool vmalloc_dump_obj(void *object) { return false; } #endif +/* Report full size of underlying allocation of a vmalloc'ed addr */ +static inline size_t vsize(const void *addr) +{ + struct vm_struct *area; + + if (!addr) + return 0; + + area = find_vm_area(addr); + if (unlikely(!area)) + return 0; + + return area->size; +} + #endif /* _LINUX_VMALLOC_H */ From patchwork Thu Jan 12 15:53:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13098270 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 098A0C67871 for ; Thu, 12 Jan 2023 15:53:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 993C38E0008; Thu, 12 Jan 2023 10:53:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 91C558E0001; Thu, 12 Jan 2023 10:53:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D21B8E0008; Thu, 12 Jan 2023 10:53:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 51C718E0001 for ; Thu, 12 Jan 2023 10:53:44 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 056C5AB2E0 for ; Thu, 12 Jan 2023 15:53:44 +0000 (UTC) X-FDA: 80346592368.02.5223A63 Received: from mail-qt1-f169.google.com (mail-qt1-f169.google.com [209.85.160.169]) by imf26.hostedemail.com (Postfix) with ESMTP id 56BE8140009 for ; Thu, 12 Jan 2023 15:53:42 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=YodB935w; spf=pass (imf26.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.169 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673538822; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uihgwx1T6nkcpHBqU4nGrP1KTt8QoN0fwfvzg1wucPQ=; b=7O6eUXKdyQMR37ogxvMqikJtwNqdSqhQzM7rJO76iYcB+2CubBqb0aakwyxMeaHjAOnPnJ vm9sdFXJUpIC298mnGEJj7XzfJ/i1ZqB9GtNbBMlKP88FmybkWjfinIAKHZve4BtAWPCWO fgIRIuWyeBrF0Qq5ljjAHbiAXcrcM88= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=YodB935w; spf=pass (imf26.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.169 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673538822; a=rsa-sha256; cv=none; b=7auRiocTwMjbfFwWnIzc4DuoTM+xFd1VTVXHyMATH2YT8gzrVK3BjTwDC/tsoxq6E16sSK lHSN5yuNvodOoYfYKFQU37+yv2vuaYffb0HsIO6GsGVskbv/REU6Q8EotJAtqdyNfu7cVQ +ZqYWAueMc10yvgVmppN8k1ED7joTrY= Received: by mail-qt1-f169.google.com with SMTP id s5so12327087qtx.6 for ; Thu, 12 Jan 2023 07:53:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uihgwx1T6nkcpHBqU4nGrP1KTt8QoN0fwfvzg1wucPQ=; b=YodB935w9n5znFspbhXFeVSzFSE5JsGjdXO3T7QxM4mYUliNC+TPtRxw2JuS3qxGjU I2i8BfBS5PcscctdPUhc//YULRIwAZeg1mS5UNvHjm9u/Zq0t8pH/dyxAZz5micRnLjo e9u6Pz9RSOvxKChnvdrDNq5sufGnV3XuasdTaZw30pseNUm8v1DjX0A8FUR+KM2BkclZ qGgHThyqikGWYyWV81ZlED6FUAeDFbhe+Z7dvttH3V4gRa3OZMZhslzzSITLkPBaTLoy eMSdtv+WJhmL/l2hEXvVt+Aczvbpaz13WvZDnwvZLHETZHye1E7tiOLO2wLcwF+0sVFp wgmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uihgwx1T6nkcpHBqU4nGrP1KTt8QoN0fwfvzg1wucPQ=; b=PT/nmQdo+0pBtcJVD0RX8t66oP3Rgst7QDQrz2eawgBnSGeutpyus9r4koHodCHUAs BDIuQcjYDEogb+It72WvfPCbIifovOzkL3js05NCbYU3B8UAdPq3KOk6x846jeifk710 b+eRd4heS4te67D1pm0w0HbQ8uOE9mCScKszgsqerltTHzQZmyYF8Up4yq+MOAbaYqGx 33P0IN7oO8gVgDwpxX8KCyo/FtZ9CH/dkK1eAwrHddFcLFD92jtXtGxSySGEnOukTVBR o5JJwmFZXufnxuA/WnSRsOha5hzPmQMKWa0hBHQdhxlwrUPMGv+j5rObRlEA7+NHXZfk XPHA== X-Gm-Message-State: AFqh2kotB+/xr1qs6VXH7EFJzLajmTj5op2gX5UWmqbmNspYYS1AZ4tO m2jXBOx8ntKl9jVIt2XAj9M= X-Google-Smtp-Source: AMrXdXs0Up2USZyI0o43Vk0+bBMZSWgXv+eJ2HsxBP+9FPA9u+0LO7bceUcOwMvSq9Zd58JbXWCPBg== X-Received: by 2002:a05:622a:5a98:b0:3ab:8c3f:328b with SMTP id fz24-20020a05622a5a9800b003ab8c3f328bmr94053633qtb.4.1673538821529; Thu, 12 Jan 2023 07:53:41 -0800 (PST) Received: from vultr.guest ([173.199.122.241]) by smtp.gmail.com with ESMTPSA id l17-20020ac848d1000000b003ab43dabfb1sm9280836qtr.55.2023.01.12.07.53.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jan 2023 07:53:40 -0800 (PST) From: Yafang Shao To: 42.hyeyoo@gmail.com, vbabka@suse.cz, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, roman.gushchin@linux.dev Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next v2 06/11] mm: util: introduce kvsize() Date: Thu, 12 Jan 2023 15:53:21 +0000 Message-Id: <20230112155326.26902-7-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230112155326.26902-1-laoar.shao@gmail.com> References: <20230112155326.26902-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 56BE8140009 X-Stat-Signature: jigmjwzbo6yxp1yob4n1qe5bq6a1gixk X-HE-Tag: 1673538822-127676 X-HE-Meta: U2FsdGVkX18ICzsXobZg8cDp5WQ51vH/SfSZ8E74bhSTM/7LwC08TdYhYkP+E2W/cUv8qjUqh77jbdZHpfpVEmxM9+mbxjZsNIZhJnaGIYDFa8MWH4k6lEdCQ2Bb2nUmm1+XOKvSMFbTqxSitMcpMAGlmH8x3Cw7XylmbxvDXskJaOx8Hpy3prnEbCZQQ8AjSadU8TnOks8TcaiV0dEbNLOBxdCqsMM/G6uAwQyUj7lf4UGnztInbR1hKTLkxPC11wkvq35bWqV7KnKIchalDxB+aADl9VCB9HA86Qb30u97PN8G1PRZQn235Aw2tvM9O2yqPCeg5nc8B2vcd/xIS0x84kfxOYeD/hAn3tyllJWP+1pittJh548mySmepLx8HO3IErGYVl7VXshYwpiWLBMUjTKACGpgqjXq0dG4kk+w0xtVe3tQvnZ2OO/pMirqmV83g3pk0Ke21+GVdzjb6zbyEMgFItSkgYuKFnQ8xbZpwK0z1MTZo0I+WSHBvdXELoqtdoxXa++87IaNEDkF5ky/uCkTSBVLs6i25XGryafJlOf0Q/eto+tNUKFw4oXgnr/jmKTkXka4hst+Q1i6fYBQvS8YKl6YZ/8otSyVyRhtF0Imu/rJeGQFKiqKnlsjM16aS+N7ndqNoMJdV7XwSq937zRORr/sYZi8hcxYzAI8g/JqFdGzV9yJsPgV6mcpRS36+AXp0sl1LO/YEt5eh3oVIeGhDKReNzybChVoozAY6rqn8a+W9oa/8muA/1WYOyCbnXHkiiTYUY1XR1b39B+DS1wQh5r+Bnk1TV2cf+s4H5OIR67Un7YOCgIuvYhkpv8fDpwCRDN2irRVX5E8W1WThGC90V7Tc7AbcyLuoaBoo3sZDgd+rc2VibgMASk//mDYTKMX1p7a2nU24iFO7L8IuNEwK7NNrvhxUWozxFA4woBkhZV1I5PbvoNB00jOAtBDm1uL1+ZC8wh2Brl hlUpXrE3 s7s3BbjL1BSDJVXBCcFhAIlwqDFTQX8en3ylaBA1VxB3Sq3YniXDEK746S2MuKqSL//YuK4nDrze/AX0Xgf2puIEz/Y1a3CJgOUxuI3SNTM4f/5GlcE78A0+4DimdFutviQQwsmYuwbrjEPqatBAvQz8V5IRH+u9P+FekTxjX0yPPccUZlz3Fw+idRheBP6BuwkgJD2i2HS3iDbe6cAN/NHo5HfsUM5IcNFMj4cuQB/ywsI8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce a new help kvsize() to report full size of underlying allocation of a kmalloc'ed addr or vmalloc'ed addr. Signed-off-by: Yafang Shao --- include/linux/slab.h | 10 ++++++++++ mm/util.c | 15 +++++++++++++++ 2 files changed, 25 insertions(+) diff --git a/include/linux/slab.h b/include/linux/slab.h index 45af703..740ddf7 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -226,6 +226,15 @@ struct kmem_cache *kmem_cache_create_usercopy(const char *name, */ size_t ksize(const void *objp); +/** + * ksize_full - Report full size of each accounted objp + * @objp: pointer to the object + * + * The difference between ksize() and ksize_full() is that ksize_full() + * includes the extra space which is used to store obj_cgroup membership. + */ +size_t ksize_full(const void *objp); + #ifdef CONFIG_PRINTK bool kmem_valid_obj(void *object); void kmem_dump_obj(void *object); @@ -762,6 +771,7 @@ static inline __alloc_size(1, 2) void *kvcalloc(size_t n, size_t size, gfp_t fla extern void *kvrealloc(const void *p, size_t oldsize, size_t newsize, gfp_t flags) __realloc_size(3); +extern size_t kvsize(const void *addr); extern void kvfree(const void *addr); extern void kvfree_sensitive(const void *addr, size_t len); diff --git a/mm/util.c b/mm/util.c index b56c92f..d6be4f94 100644 --- a/mm/util.c +++ b/mm/util.c @@ -610,6 +610,21 @@ void *kvmalloc_node(size_t size, gfp_t flags, int node) EXPORT_SYMBOL(kvmalloc_node); /** + * kvsize() - Report full size of underlying allocation of adddr + * @addr: Pointer to kmalloc'ed or vmalloc'ed memory + * + * kvsize reports full size of underlying allocation of a kmalloc'ed addr + * or a vmalloc'ed addr. + */ +size_t kvsize(const void *addr) +{ + if (is_vmalloc_addr(addr)) + return vsize(addr); + + return ksize_full(addr); +} + +/** * kvfree() - Free memory. * @addr: Pointer to allocated memory. * From patchwork Thu Jan 12 15:53:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13098271 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C358C61DB3 for ; Thu, 12 Jan 2023 15:53:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C04B28E0009; Thu, 12 Jan 2023 10:53:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B8C9D8E0001; Thu, 12 Jan 2023 10:53:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B9088E0009; Thu, 12 Jan 2023 10:53:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 853B98E0001 for ; Thu, 12 Jan 2023 10:53:45 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 50134C0169 for ; Thu, 12 Jan 2023 15:53:45 +0000 (UTC) X-FDA: 80346592410.15.4EAD2DF Received: from mail-qt1-f173.google.com (mail-qt1-f173.google.com [209.85.160.173]) by imf27.hostedemail.com (Postfix) with ESMTP id 95C9540017 for ; Thu, 12 Jan 2023 15:53:43 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=V5AonCl6; spf=pass (imf27.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.173 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673538823; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TBH19B7tt8c2ZqgTL5MGs4kQ+EgYdVrIXT4FgqCf58E=; b=qOybszHICS/ACsljKiZr4jBCp4YQPd41aIsGXIXiL6nIzR4wcFjc4KhzdTjrF3M9/W2qjZ jY3rider7CM76EHnGAckMMiNAlStwiyOuK0fmdIfvE71HGnfWZYbbLKum2blA7zmZGCu2f 5AJVEtdcSAeVDP5crtf3ZOIJJcfAvfQ= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=V5AonCl6; spf=pass (imf27.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.173 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673538823; a=rsa-sha256; cv=none; b=283YeEHjbQWBA4TKL0taCNDTQWxwJCHeOX35pKjOIuWlbZDsqSi2Vf1i0T5XPlvYdhKXlQ WAibHmDKGLnwUdFJyfkDhUA7dnYC7qRYJkc8wZE5E+SLbMDIe+WyCtNUqoVpEA++2v9kTi PzsDJ6A2CIMV7M1qxksLOG4L5N3N2o4= Received: by mail-qt1-f173.google.com with SMTP id bp44so16890214qtb.0 for ; Thu, 12 Jan 2023 07:53:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TBH19B7tt8c2ZqgTL5MGs4kQ+EgYdVrIXT4FgqCf58E=; b=V5AonCl6GtoPHEQDYaUYetOY7qBor5eB4wJnYjHi6v81IJ6LFl63Vb7qTQ6fZ62GmJ FZ3VHrJc1gdN1TOHjP2hoB35KssqBdVaaZbEQZ/E5zivkI0c5eo7XBU1gZCICr7Fpmmx tK8mbPvFPAdW5NE9w07N1Eta2988UZDI9ppynyPtdFQ5OYtkpv0D4+CkEJpBYNYN+qyQ uuqqCAyOCmPlXT/hg59VPHxvJMezaeIia6wBeOewTU0JIGj73v/uGM99APwnJyPb1MAa 6tlQS2nILNzaj0VNoUnSKxnsveyfjiyJpOyMNF4QnV3ia2uczbf/7pWYoLPwNbm+KQcS i0Fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TBH19B7tt8c2ZqgTL5MGs4kQ+EgYdVrIXT4FgqCf58E=; b=YHPapSTWUsRjces3+WTY67Aw1MovJC2TssyrBQUH21XqyY4mBVjLO+z2JKxuEpnnmP Df1wbtogLEEZ4x2hTrJwmeQ22Tg/z64lwHNRfN2zLMET5Ifl/q+NHwkWF8aGB557Xbmz ubMzjrvOG1Yv13nR/PRFshQLnFAMJ9uiLtR7G+xzNhH0KwKk+EqVcJmPQD23PIDz/+Lo pt/7tckU2FQ1C93hGp4SdNKR/cn6uo3RfmavEueFug8MmI5KhbnQqD0wxPx52D/bFb7B vU1RIumC5ebB/5NxYdsRVyrke3vVpVDkKf0iFcCq+1LNa14h9xtt9L4KZdlra1A8dx+O tXDw== X-Gm-Message-State: AFqh2kovNyaenagDLR89+mrpqZ5hSk0OC+hDzSu/1k90rNgncGFQ/lA9 xNsIjz9/dwems3DdnDDlM/w= X-Google-Smtp-Source: AMrXdXunMQbXCe3PjTlqR622vY/D62Blbxfq9VYi3TCPRza8jAekLqfz3F3c2jSj1Xlfyg52SWyWbw== X-Received: by 2002:ac8:7957:0:b0:3ad:903d:3ed4 with SMTP id r23-20020ac87957000000b003ad903d3ed4mr19548488qtt.59.1673538823178; Thu, 12 Jan 2023 07:53:43 -0800 (PST) Received: from vultr.guest ([173.199.122.241]) by smtp.gmail.com with ESMTPSA id l17-20020ac848d1000000b003ab43dabfb1sm9280836qtr.55.2023.01.12.07.53.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jan 2023 07:53:42 -0800 (PST) From: Yafang Shao To: 42.hyeyoo@gmail.com, vbabka@suse.cz, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, roman.gushchin@linux.dev Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next v2 07/11] bpf: introduce new helpers bpf_ringbuf_pages_{alloc,free} Date: Thu, 12 Jan 2023 15:53:22 +0000 Message-Id: <20230112155326.26902-8-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230112155326.26902-1-laoar.shao@gmail.com> References: <20230112155326.26902-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: pukwiuxcz884ye5ycyz59nsx75eypf4u X-Rspamd-Queue-Id: 95C9540017 X-HE-Tag: 1673538823-95030 X-HE-Meta: U2FsdGVkX19e1tcqEQhrfIDrMGay7gYkY3tKtRIWZgA8rh2bWNn+MrEZxbZtkcRi+3Y0zI6vAlLr8MuLmrMd2GBjYC94r3E0SmG7rGFx2lvWddQ74MmOm+PEP9oL7HSUu7KtOhOuZYz1eK0TdhIC1qBGsUuY+bGt/4QAjCh7EhQYjX5r29EwvMCGX/GH5GgTeuUDzr3d4oJ4SGn4uKkzYMEDGFKGIWMooGqMZN/eZEnC/T8mkCldXhQUaE8tzTPx1KGJ90zbNmTynTvbgkQkVDLEobyxoDmWIVHx80R2hyFPPhI+qqKZX/4Jw+B5uzHpn4o2oMwsYYvILzu0cC+0iK02/I/pW4I6eCE1/tEKNgoWPtV+/BbB0Rknnpy2HbOSAPX24mKz/2EENH60IQtYrwyHSg6NosTh7lWIlvQVsCvc6prFitCdlzHC4peXSfqhCjeb19xRGasl+NUtMm50I9y6mvTFA23dKGgQ9PXvCO6n6VckQHTABXUWq1zl79uVr24/DlYcPEBDIHfJUKdaJwwOMDzSwLQleB8Q3y39/gCF7bQa0/qu+E/3aOLYCPb2Vc6ux0zc3RJ5uNxjqgNnjwThdRZ9N0Ge+OFLGHUgw/O+2JKwtk2YClXH0fEJpZnMJVrGnSWUX24NEHLVYFP8zbmSRltoTmhNkfiNDPbhDVRB54c7mfTFBLaLqYHTW00GQVL5imjTmwvXjGftC/F79ibyxOtY7s6WGgDKXkELL059N8mhIRaN82y07lSmYAEUN3hlBtKlU1gNZCR7uGy17/ZviqHiaxETKKn753SNUU6f5r7MzgPW3R8uJL8YWTzpL8ggtqjfM+keT4/ImgW1biR4iikhetcjz2g5XQtWr29AGq7Tkz+o6guD5wXxuplFR1yISgUA2sQogETRcbFXONm2KIoVFVNlmutGP4lAqYTR5I+5GDuSU1VQ0wi9SDU9i1UArZqbzKIStqW5VUe T035wk6h m6ClkIRJxwxcUdqu+wzGZkMn7DJslGA2PJQ/HSJvYECm6VDlFuf3ExwrFV/JzpejCKAC7CLRfTt063TsEab8oF91AFhYMdVhmtEWd0KqbcXwT/LAZMA76OquqdWsi1PeY5SDe2d6yb2lnW8jL5qn5wGv51FtMIpHnnIaaKlNAoVLl/BK9VByfuxujuqA2dEf68L0RcUbpRPSTMQRxDqA8pHhBd68UFFFZwUYNnd+tHaINlNGuwdVIOE+f/Fxww9vAiHrzgc5bgY+VyAJULwsDUFbcuth0dmM/UoJr X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allocate pages related memory into the new helper bpf_ringbuf_pages_alloc(), then it can be handled as a single unit. Suggested-by: Andrii Nakryiko Signed-off-by: Yafang Shao Acked-by: Andrii Nakryiko --- kernel/bpf/ringbuf.c | 71 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 47 insertions(+), 24 deletions(-) diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c index 80f4b4d..3264bf5 100644 --- a/kernel/bpf/ringbuf.c +++ b/kernel/bpf/ringbuf.c @@ -92,6 +92,48 @@ struct bpf_ringbuf_hdr { u32 pg_off; }; +static void bpf_ringbuf_pages_free(struct page **pages, int nr_pages) +{ + int i; + + for (i = 0; i < nr_pages; i++) + __free_page(pages[i]); + bpf_map_area_free(pages); +} + +static struct page **bpf_ringbuf_pages_alloc(int nr_meta_pages, + int nr_data_pages, int numa_node, + const gfp_t flags) +{ + int nr_pages = nr_meta_pages + nr_data_pages; + struct page **pages, *page; + int array_size; + int i; + + array_size = (nr_meta_pages + 2 * nr_data_pages) * sizeof(*pages); + pages = bpf_map_area_alloc(array_size, numa_node); + if (!pages) + goto err; + + for (i = 0; i < nr_pages; i++) { + page = alloc_pages_node(numa_node, flags, 0); + if (!page) { + nr_pages = i; + goto err_free_pages; + } + pages[i] = page; + if (i >= nr_meta_pages) + pages[nr_data_pages + i] = page; + } + + return pages; + +err_free_pages: + bpf_ringbuf_pages_free(pages, nr_pages); +err: + return NULL; +} + static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) { const gfp_t flags = GFP_KERNEL_ACCOUNT | __GFP_RETRY_MAYFAIL | @@ -99,10 +141,8 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) int nr_meta_pages = RINGBUF_PGOFF + RINGBUF_POS_PAGES; int nr_data_pages = data_sz >> PAGE_SHIFT; int nr_pages = nr_meta_pages + nr_data_pages; - struct page **pages, *page; struct bpf_ringbuf *rb; - size_t array_size; - int i; + struct page **pages; /* Each data page is mapped twice to allow "virtual" * continuous read of samples wrapping around the end of ring @@ -121,22 +161,11 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) * when mmap()'ed in user-space, simplifying both kernel and * user-space implementations significantly. */ - array_size = (nr_meta_pages + 2 * nr_data_pages) * sizeof(*pages); - pages = bpf_map_area_alloc(array_size, numa_node); + pages = bpf_ringbuf_pages_alloc(nr_meta_pages, nr_data_pages, + numa_node, flags); if (!pages) return NULL; - for (i = 0; i < nr_pages; i++) { - page = alloc_pages_node(numa_node, flags, 0); - if (!page) { - nr_pages = i; - goto err_free_pages; - } - pages[i] = page; - if (i >= nr_meta_pages) - pages[nr_data_pages + i] = page; - } - rb = vmap(pages, nr_meta_pages + 2 * nr_data_pages, VM_MAP | VM_USERMAP, PAGE_KERNEL); if (rb) { @@ -146,10 +175,6 @@ static struct bpf_ringbuf *bpf_ringbuf_area_alloc(size_t data_sz, int numa_node) return rb; } -err_free_pages: - for (i = 0; i < nr_pages; i++) - __free_page(pages[i]); - bpf_map_area_free(pages); return NULL; } @@ -219,12 +244,10 @@ static void bpf_ringbuf_free(struct bpf_ringbuf *rb) * to unmap rb itself with vunmap() below */ struct page **pages = rb->pages; - int i, nr_pages = rb->nr_pages; + int nr_pages = rb->nr_pages; vunmap(rb); - for (i = 0; i < nr_pages; i++) - __free_page(pages[i]); - bpf_map_area_free(pages); + bpf_ringbuf_pages_free(pages, nr_pages); } static void ringbuf_map_free(struct bpf_map *map) From patchwork Thu Jan 12 15:53:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13098272 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E46FC54EBD for ; Thu, 12 Jan 2023 15:53:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C79E68E000A; Thu, 12 Jan 2023 10:53:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BB5B08E0001; Thu, 12 Jan 2023 10:53:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B9398E000A; Thu, 12 Jan 2023 10:53:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8127C8E0001 for ; Thu, 12 Jan 2023 10:53:47 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 39716160305 for ; Thu, 12 Jan 2023 15:53:47 +0000 (UTC) X-FDA: 80346592494.26.84ACF71 Received: from mail-qt1-f171.google.com (mail-qt1-f171.google.com [209.85.160.171]) by imf10.hostedemail.com (Postfix) with ESMTP id 86B32C000F for ; Thu, 12 Jan 2023 15:53:45 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=aU2Esvnu; spf=pass (imf10.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.171 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673538825; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=E3K14TM0boLxDmRFoAT7RRQ6Ph6mAKiCi/aS411tfbA=; b=ofNO37a2GVKbTpW0obykx5eG5GnYVacAisLcarhSb/fEo/VJD6SNHrV5/AMa/GTNOFo3Zx lKtqdjRpiiDiqFLzJxkW0/PWI5smS3E2bosY1Rlvei5CAqSmo1jtayqt2G5jeaWi76hYnG Ed330DCjp9qKm2Ss20UC78qAFSvT2cc= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=aU2Esvnu; spf=pass (imf10.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.171 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673538825; a=rsa-sha256; cv=none; b=Qmy8m7fJhQn9FrBsg9/1se9B0j/gpWI2uV4wo3CPSk6iDJ4W2MAkK4ImL+nbTA/fxDawJ9 k8BgmM614xxvxcrDO6eqRZmmQTj9RpC4Jc1VIfqhQkXcArNCGd/iYCRfQEjby30W2Ozlc8 vFfKDHtvi+BBd4OMlp8oX0O7dt7EXdE= Received: by mail-qt1-f171.google.com with SMTP id x7so6685460qtv.13 for ; Thu, 12 Jan 2023 07:53:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=E3K14TM0boLxDmRFoAT7RRQ6Ph6mAKiCi/aS411tfbA=; b=aU2EsvnuuPe3ZSnD50zuV9E4JG6cC0vrWHgi5vg4ikr91yqv2ZGfpwHTgtuioVbpBM yyzQKV9/BjIWz7exRjSTq4RarPTJ/Rlay11Cq2e4N063c8OsXOEaMwQ+s4OEoF4KoPch oWIcu/UkJwCVTSCFUjbild5uruartTUjGA2B+vrw7cJgwLrO1u6b2EFAn8IJTkNBc5OO UlJE64nia/i0LkkrFI92NE/BnSEWTF9cfEphiDZKIJzaE3S1HPXo98teauQs5sn57MMQ ceIE7gIKfpLLQKA7fWNK1yOatwIAlDmcEH49jqOD3eVtCUdCrdcwfOk8ScblBs3TJux0 xs7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=E3K14TM0boLxDmRFoAT7RRQ6Ph6mAKiCi/aS411tfbA=; b=7S1FLAWmTMt2DTfscuDly3Ebe3GxHDP5kZHixUfdpzLmPefh8hvensAkWORtMOV3h3 LNM7HZ2HWeS568cVASimX3IM1yr/K3zenorla5nTESGX6Sp4PLQ20DNH5cmEQlLjIAJP 0etGORjW1+nCS7aNXwb7tl59XpJO8K8lo04hUyUzerN7Rlmjs3yg19+397C9FkirPlJv 4YbvUM38yHpeIN7pLjutFgK03fIkHW7d6fVpjzsi+0ulub/wMSqFwplPvWe0699u/5eB SuMa3LOWGWyyKyLpWDGsSNyHp+sYO1xT7Kly7uS2qJVMkpHA1n8wg6Tl533DscIJSevL eTAA== X-Gm-Message-State: AFqh2koyiaupbnx643LAM1IKsupZvxGUXPpagsukrFaAFoTDutD2LfCz zE1TtgLQqRu/OgyMhwMBDq4= X-Google-Smtp-Source: AMrXdXvViXPTwmFpafLglW8xYZpT4nbyGcr8EcVCNuoHszbaRFi7jBpgklWOBB6m5tYn5p20XICckQ== X-Received: by 2002:ac8:528c:0:b0:395:396f:a519 with SMTP id s12-20020ac8528c000000b00395396fa519mr101217124qtn.0.1673538824580; Thu, 12 Jan 2023 07:53:44 -0800 (PST) Received: from vultr.guest ([173.199.122.241]) by smtp.gmail.com with ESMTPSA id l17-20020ac848d1000000b003ab43dabfb1sm9280836qtr.55.2023.01.12.07.53.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jan 2023 07:53:43 -0800 (PST) From: Yafang Shao To: 42.hyeyoo@gmail.com, vbabka@suse.cz, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, roman.gushchin@linux.dev Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next v2 08/11] bpf: use bpf_map_kzalloc in arraymap Date: Thu, 12 Jan 2023 15:53:23 +0000 Message-Id: <20230112155326.26902-9-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230112155326.26902-1-laoar.shao@gmail.com> References: <20230112155326.26902-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 86B32C000F X-Stat-Signature: pzkitboj5oc5xogabuzp9cxaspbdrrbb X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1673538825-776751 X-HE-Meta: U2FsdGVkX1/9oxY/dZZvBIcxhu3f46J4CDXO4qW8oywsvZpqFRMYTz+tFJhJ1KfJRFb94mFv/ox4q5Sf1lm/i18d29R7qJ5MbrNW3r4kGZwcZD+61HCJD4bkyEsqKAAW2pd7L4O9ATw2EoJEu1ojYUKfBVrmIwa+FRcBK+RjZ4z+ZKwIgj1ZkLAu237g1R0pfwvAGgL6HDqDzuNcHlLa7zioQzL87ZirEmhCOjLIwBcYOcD05EIF9D88m1eUJslIw4gHq5kYBeWaGAsDFNmXnnOnhpUU6pUpqwXe6sCZwnccizhQL+1RlN8SmVfBjUuDCE7NeJgfr79liFlxnLv4EesrdHoKV1+I3FGb/2CYccIxSbKY8nVRfp56Xk/sEA7U8LPrlJFYLCqIkrbx0l08o3p5VgtF6SzbzmfST5yUT/TIFUWqAJyxDTfcgWeFTEmI18KC37MwdWZWSARzOsIRyfS9Rlh15MjtnUUbGLsPL3mEUEmRsJ5ZYl/qSVXA1SSen/Ax/1vxeuHO54Eohx7V/Vt9v5oAUrOaRwsN2W5oy8NDpgGqJ+rvOCO6mLVsARVzFXdDSlAKaXwJerZuzG0V4/mtrWd2MOpKrr4budTtxk/DChNTa6iGmCBUx1aWjGeRU5ZLEX4paSEtRTVbdzWocNUHR4BtfwkcdMWhPe0dgZIkrYuyjMsxJ8CCarFsO5wQOtPyMOY98FQAhiRL7hJYavw13aXzBpZ+pxPaJhg2HphZKO3eLqO9aIGjiP8K8+m6ekkTlfxMJvEIFxcgjLdyvQLR0XhsaiSbnVC0XIIKQ/pNPHqccxJlngGZBljQpu9hzbnw61rLtXRXqzecIWJ+xu2eDwzNMyO2ZyjX6yMru/u8AEwbE55DA3Z8fwK9+pBH9t5EyzbMk3dbdinf1OSSZtXkUhc6YTc03pQzIbuQVTguZNfp2Mx6SYVyhDaHv0CPs5TXStfCv3VqsMyW06G nUTU/Rku 2OxnXN9/NHiN7/xlmvfRAX/dIomvjn2VgpHau7nu2vbob4/fVM+8kJFmdysDgTa807K/8WSL9I8Lbx08jmNnKFXUM90Wz1VDD/HcE+BxBdohRuO10ZUhsAACTq5JgdTcSyHcjH3P+dWblfRiP3orZ7hYQKvpj6JuHu1g48yTzn0+gy8Yh37E8Povg6GTlECtjKMvMc+DU7Ne/fbh7ACAzlHen9EdphPGnQ41OjDJgxmYFenj+DxUK4A2uS51Aqn98wDRJEjx6IW//AodA9Enq2tp6k5trV1BDBd27 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allocates memory after map creation, then we can use the generic helper bpf_map_kzalloc() instead of the open-coded kzalloc(). Signed-off-by: Yafang Shao --- kernel/bpf/arraymap.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index 4847069..e64a417 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -1102,20 +1102,20 @@ static struct bpf_map *prog_array_map_alloc(union bpf_attr *attr) struct bpf_array_aux *aux; struct bpf_map *map; - aux = kzalloc(sizeof(*aux), GFP_KERNEL_ACCOUNT); - if (!aux) + map = array_map_alloc(attr); + if (IS_ERR(map)) return ERR_PTR(-ENOMEM); + aux = bpf_map_kzalloc(map, sizeof(*aux), GFP_KERNEL); + if (!aux) { + array_map_free(map); + return ERR_PTR(-ENOMEM); + } + INIT_WORK(&aux->work, prog_array_map_clear_deferred); INIT_LIST_HEAD(&aux->poke_progs); mutex_init(&aux->poke_mutex); - map = array_map_alloc(attr); - if (IS_ERR(map)) { - kfree(aux); - return map; - } - container_of(map, struct bpf_array, map)->aux = aux; aux->map = map; From patchwork Thu Jan 12 15:53:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13098273 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D5AA8C63797 for ; Thu, 12 Jan 2023 15:53:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F3ADB8E000B; Thu, 12 Jan 2023 10:53:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E73AC8E0001; Thu, 12 Jan 2023 10:53:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C9EC88E000B; Thu, 12 Jan 2023 10:53:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A02218E0001 for ; Thu, 12 Jan 2023 10:53:48 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 66297120BCA for ; Thu, 12 Jan 2023 15:53:48 +0000 (UTC) X-FDA: 80346592536.19.10A5A26 Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) by imf29.hostedemail.com (Postfix) with ESMTP id B8F0F120002 for ; Thu, 12 Jan 2023 15:53:46 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=UwdPEQKp; spf=pass (imf29.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.176 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673538826; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rS5gOGm/FV0EomE+G/HxAF0nnvFi+9pGVhYFw6UMCCU=; b=qHZLP9n+x55bPvhpOx2gwuDqHUmHoRgbOn8cYs6lSBRF7+Bd03elC+YTLNAqPC+Bhy+CIw 9/tYYRfP7pywylVHFp7u7qb92h4p/5ySuX7D295kVb4BMuIheX9gqAU6bmaBIwzNMUQuMH 1R4TLyWTST76G93Ob5M/ucb8Gu/Uqz8= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=UwdPEQKp; spf=pass (imf29.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.176 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673538826; a=rsa-sha256; cv=none; b=YCMuAqqvJxl1Whk7OC0plHWwy+3d7f3ah2gZOZHoqE0SVXi1KQ9q045lyXdE3Wvhlvl0zb L0T2sWo/kwd11anSXDQVJXScyYH9OyLIsGBsOGsVtikonzXdzAOL/Kyw6omg8cLoh2iz54 okhFFyF2tjC+nlVuOmz6bjaE2He0Azs= Received: by mail-qt1-f176.google.com with SMTP id e22so4306061qts.1 for ; Thu, 12 Jan 2023 07:53:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rS5gOGm/FV0EomE+G/HxAF0nnvFi+9pGVhYFw6UMCCU=; b=UwdPEQKpeaenmGBhuznEUg2Sgbzj2CbVaC5pOx/wXR0lBhCD+Kmr+Ukcq32XrTVLeX QD5J4wEyXv36kjjM5LJZD2ViCBneCDE+D38KzvcyswB6pmGebWYMatgB/TlTiTBbJVJt 4X/EnkVd1Js9zemKZm9qMUdchkr7bKUQEeIoIcwG+HebxpzFEJ7wnAtwcbpJMOpR3QxL UZcwi5dRYZOX6riW+E4KupYoGQEPruyPGDOYSOkjE/yH8L2ypWd/VEi6SQRHHnZdlRUW IVP5tx2gBV77V9Fjfr9na9W1tsh3ny+CcrEIlk/NIilzOE1vy8vTwWzy6zZ8H7H4nh2D 69RA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rS5gOGm/FV0EomE+G/HxAF0nnvFi+9pGVhYFw6UMCCU=; b=yEufzjCTFT4hv+Cej1DFcvh7IMKk2E+hMXZP/NHNQcuVCzjJrLSSS4j/re0/PzySyd Na1k5LcJBmpbQIE5U2oxFUJ2Ri8yp0PYeMp0MZuYNWP8J3BVDYzR9jUyDQ91KxhkIkK9 iwkjyjHzIp5vR8x5VNhvjgtILkea8cwqJHaakQS2RRN+oFeBpplD9WfNMth12BWE4HQV igPJhiBt7mP4iZzwM+ZoQIIScBl/EHwzHPfEQMuKHm0JDPcpr/v/Is8nyuxmxaCHDaJY GByddRnbDx7yWwaIC5uJrqRJWS+vG7S5rNT9PGiLNr6mOR1BTXeBUWUSq6HztvVSIJW1 NhUA== X-Gm-Message-State: AFqh2kp8NAwD+g5+25NQMhCecXVpdp+4p+KQXQtNAV1nWAbPjkCqO/Uk GK6qd0QUnv36ugV58RNGRb8= X-Google-Smtp-Source: AMrXdXs89lOju7eofF5NQ1SL2Vit25d1eDBdT+g9j1z0eBLF0PzlschIVu6iMcAjd8ub9ViVM50vww== X-Received: by 2002:ac8:5e90:0:b0:3a7:ea9b:5627 with SMTP id r16-20020ac85e90000000b003a7ea9b5627mr13990618qtx.13.1673538825926; Thu, 12 Jan 2023 07:53:45 -0800 (PST) Received: from vultr.guest ([173.199.122.241]) by smtp.gmail.com with ESMTPSA id l17-20020ac848d1000000b003ab43dabfb1sm9280836qtr.55.2023.01.12.07.53.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jan 2023 07:53:45 -0800 (PST) From: Yafang Shao To: 42.hyeyoo@gmail.com, vbabka@suse.cz, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, roman.gushchin@linux.dev Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next v2 09/11] bpf: use bpf_map_kvcalloc in bpf_local_storage Date: Thu, 12 Jan 2023 15:53:24 +0000 Message-Id: <20230112155326.26902-10-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230112155326.26902-1-laoar.shao@gmail.com> References: <20230112155326.26902-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: B8F0F120002 X-Stat-Signature: qbrkbr4bg7nimbr3fcrh6zktze9ww6io X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1673538826-568721 X-HE-Meta: U2FsdGVkX1/MJ01hPm268CboSd7hykt+6zLPFMiyJg+rKU0PxX5w736CA7OCDw95ueDPgm3C0fgWu3Y0SL74gtp/wVbTonQqGSkcBHGzEYPKa7hgsuYyPydxXXLNXcNcnuIgALQ6MPgmyYG0z9455/HrqjQVCTBsaQVGUBlUKbYawts5IhzvIUj71VvDzY34qPPrWl8q8ze278Cr0U/FuLRvcC4JSTqrJwUrsa8FXhtlTTDT5Rlq3eZ9kq7/FgbAsGdXjdJ52YeCJtshUyS1PxLOeSq/+P8Z0ZyepP1W6Wz4+J+iU4kBpDJmzm0o5BEcbdIMaEEHKuJi+4qUlsVpiNhGoUc+pWrU1qVBrSU5mxy+D4n2M1gSMCsVWNgFrNRCbTB2/xHuEdENZjKPIlSYHLybBkxNvlgQeWtW+pJJOvWNhmGbSvWRgm/DnXZjgxEMNHI3+I7beAVgNzbj3DMus2J8mPvh3DRYo0OH02TqznpV+eAtV9QA39qUtNVTloH65172uFQ/c73sk+KAoxUtcPDZJjCvkK8WhOlpyAEDxjqn1cyOdaMscmD5vk6/OlCRZvBJ5vjstIAvm2P73teBEuGIsMhhfnmxcCz12xGFD6XLLan5+cmnfUXK7GszJVodQoYKWjNYg8gqA93MjpqgOy48O8lTK3/U76+5Cq3Rnadr0EwfUkhejZeWDwRnl34DWpX2xbPKm0Ihuy68haJxOhvNdu/c2niu3Z0JJKHh65IqlNYlE8uXQFhI1zYY53jL9Haf6H9+2wfIOgjXjdC6OhG7fvvrYBzp77/l85rLOYnov6/lVVhhm3VYKbxOAGZFiJQZYc2+FyodA8gXHtXxoExb/R9oEX4BUs2aSAQ2yZJioTAbdr3nwl4SMQLafwI54ZR0Ejb+4UisJu63HqUEZei1YhhU2De6TZ9a2AmsM+mGePOWMTvW+M8ACQbcPC6dSW53K2H3bV6O2EkM+l5 gCbf54YY BLtZBmEniA28IuBrVkiK1UieE4o2YAno4T+t+q7QH2Q86Fw0N0ZkvK+qZajqgEvxC9ZRRcANgAU3mgwqW+9YdMirgfiuDcNsiTyCWhXOokPL297SGnS/6eaFSe5aQGqiRmWAt/YfW1bcw7b+rpeSBLgULONhTjZHQmQ7U9xLEt/h9k3hJsbaNtwJsJsjbHP44B5yRvgNkeI4ifcvE7cjFj+/LJGudbND7PvXzGJGWYGB99SezBr94/5r6gQko9kXFU9XVjJHOivucbT2tlJYZnBFcJA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce new helper bpf_map_kvcalloc() for this memory allocation. Signed-off-by: Yafang Shao --- include/linux/bpf.h | 8 ++++++++ kernel/bpf/bpf_local_storage.c | 4 ++-- kernel/bpf/syscall.c | 15 +++++++++++++++ 3 files changed, 25 insertions(+), 2 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index ae7771c..fb14cc6 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1873,6 +1873,8 @@ int generic_map_delete_batch(struct bpf_map *map, void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, int node); void *bpf_map_kzalloc(const struct bpf_map *map, size_t size, gfp_t flags); +void *bpf_map_kvcalloc(struct bpf_map *map, size_t n, size_t size, + gfp_t flags); void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, size_t align, gfp_t flags); #else @@ -1889,6 +1891,12 @@ void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, return kzalloc(size, flags); } +static inline void * +bpf_map_kvcalloc(struct bpf_map *map, size_t n, size_t size, gfp_t flags) +{ + return kvcalloc(n, size, flags); +} + static inline void __percpu * bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, size_t align, gfp_t flags) diff --git a/kernel/bpf/bpf_local_storage.c b/kernel/bpf/bpf_local_storage.c index 373c3c2..35f4138 100644 --- a/kernel/bpf/bpf_local_storage.c +++ b/kernel/bpf/bpf_local_storage.c @@ -568,8 +568,8 @@ static struct bpf_local_storage_map *__bpf_local_storage_map_alloc(union bpf_att nbuckets = max_t(u32, 2, nbuckets); smap->bucket_log = ilog2(nbuckets); - smap->buckets = kvcalloc(sizeof(*smap->buckets), nbuckets, - GFP_USER | __GFP_NOWARN | __GFP_ACCOUNT); + smap->buckets = bpf_map_kvcalloc(&smap->map, sizeof(*smap->buckets), + nbuckets, GFP_USER | __GFP_NOWARN); if (!smap->buckets) { bpf_map_area_free(smap); return ERR_PTR(-ENOMEM); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 35ffd80..9e266e8 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -470,6 +470,21 @@ void *bpf_map_kzalloc(const struct bpf_map *map, size_t size, gfp_t flags) return ptr; } +void *bpf_map_kvcalloc(struct bpf_map *map, size_t n, size_t size, + gfp_t flags) +{ + struct mem_cgroup *memcg, *old_memcg; + void *ptr; + + memcg = bpf_map_get_memcg(map); + old_memcg = set_active_memcg(memcg); + ptr = kvcalloc(n, size, flags | __GFP_ACCOUNT); + set_active_memcg(old_memcg); + mem_cgroup_put(memcg); + + return ptr; +} + void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, size_t align, gfp_t flags) { From patchwork Thu Jan 12 15:53:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13098274 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91DFEC61DB3 for ; Thu, 12 Jan 2023 15:53:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 85AF68E000C; Thu, 12 Jan 2023 10:53:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7E6078E0001; Thu, 12 Jan 2023 10:53:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C1928E000C; Thu, 12 Jan 2023 10:53:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 3D81C8E0001 for ; Thu, 12 Jan 2023 10:53:50 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 17AE7160778 for ; Thu, 12 Jan 2023 15:53:50 +0000 (UTC) X-FDA: 80346592620.29.4566B7E Received: from mail-qt1-f173.google.com (mail-qt1-f173.google.com [209.85.160.173]) by imf23.hostedemail.com (Postfix) with ESMTP id 52295140015 for ; Thu, 12 Jan 2023 15:53:48 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=FvZrTql4; spf=pass (imf23.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.173 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673538828; a=rsa-sha256; cv=none; b=8KbylCi4Zve9b3674od3RkSNGbEpHlGE2jdtmbpIVFcLIMsBaHQRCD74vQRo3yLWq2EKO1 +9A32eAckfpN98E+mv0CGsNR08A3dSQvw4caSSnEOy5jzFZR0XUBYjhgedSvK8D50QL3fE ASRQ3uwZjJOHcw/qgJ7PNaH5oJzDoFk= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=FvZrTql4; spf=pass (imf23.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.173 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673538828; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NL12hQzfY2vYmbpWyvvoyioaohRiBjaClaYN//mRPYQ=; b=IqTL/9Jj71KYgzvs9A6JAJNTa8Wn1e47IMw2E4OO7GQPuQNH4KlRfTqh5LLRF3yVI63pIs 72Ef2L9FpboxorfTIb550e6b0gi3jFUmX3qDsVkEsnrrnkEsG+VKaEgT/UOx40qn5/OpxY N60TgmWp2sUMEKoRzsC6qQOf7lurWXQ= Received: by mail-qt1-f173.google.com with SMTP id e22so4306147qts.1 for ; Thu, 12 Jan 2023 07:53:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NL12hQzfY2vYmbpWyvvoyioaohRiBjaClaYN//mRPYQ=; b=FvZrTql4zacpEwvmFGOwf9iV0y/EKQVGFKFf9nq0POhyw8MxgeuS8VG92UdznVgtJV 4H/D98zVL/EKDRWUVUv/bTaooQENTOj5PpMz29hVWkYQ1ZRPLQnUY0Sd6PEvXnxMvOev CeMRpuj/IXWH4Vm8MrrLicNtBXkYMSZtgn4X8EjFb01T/oX9RWsGAhyPBJqs1g7zORKA /3WMr8O0TX4mM2Dlnrjcl6ZOmHlUO6+0cXRwaE9beomhlC3WKe/kL9xgoJWOlFRTBqkT 2f+qA2OwYHqfQ9ANF6cayhBlOmcn6ezYYrkSk1V5VljoG+j79m8UkN5b+XxuanHgZFCt XUew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NL12hQzfY2vYmbpWyvvoyioaohRiBjaClaYN//mRPYQ=; b=ZSAxu1Sq0qrgJgx3G5+U3TBNDH39f+cfLvXsCYz7AIyWajUO9qYtwLOKJy1jFNLZSu NYuqohSGO9R6R7HMP127iNEeyBewBnrBfAY1SZa+/IWPaSF/WwPSYwrcu7zNFVgt4rEM lpCeAINiP48Hj4iQcQCZxWQyRGiWxpJWWTYCVY4WUUr4gDff0LB12tPzOpZK1cbOFwod xer0Z1qqb/9/u/qIB4NnEQJS5AMTT9F5t0WAdsOFol550bD3bSsdykytc1PwoUlCwS4K 2SesnqID7iNbDCgW6NoktDP69SvTDb8TTvCJWx+fzLKPAoPQVC3kpi6X5WiEBEsYQzgL X8xg== X-Gm-Message-State: AFqh2krucwutzlYunimGUndj/dj3Mv+M3xK7wbB375YgU33szeZPZjsu 7y1N+C+uh5lqNMUyQ/8owLM= X-Google-Smtp-Source: AMrXdXsWItcapcAoDWCDnJzKExTlnOFYmnkVbKpZGZioHjBrHENK8zzquTPTX1PHjAt5R5HXxVzEMA== X-Received: by 2002:ac8:5386:0:b0:3a4:f758:fc6d with SMTP id x6-20020ac85386000000b003a4f758fc6dmr103306816qtp.46.1673538827319; Thu, 12 Jan 2023 07:53:47 -0800 (PST) Received: from vultr.guest ([173.199.122.241]) by smtp.gmail.com with ESMTPSA id l17-20020ac848d1000000b003ab43dabfb1sm9280836qtr.55.2023.01.12.07.53.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jan 2023 07:53:46 -0800 (PST) From: Yafang Shao To: 42.hyeyoo@gmail.com, vbabka@suse.cz, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, roman.gushchin@linux.dev Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next v2 10/11] bpf: add and use bpf map free helpers Date: Thu, 12 Jan 2023 15:53:25 +0000 Message-Id: <20230112155326.26902-11-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230112155326.26902-1-laoar.shao@gmail.com> References: <20230112155326.26902-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 52295140015 X-Rspamd-Server: rspam01 X-Stat-Signature: ngo548uu14nb1o5h3uh1eineu7r4ybxw X-HE-Tag: 1673538828-41720 X-HE-Meta: U2FsdGVkX198VqHWA6VlIOn7J1uWII0oJmeNWGoLJmMjXBmW34rq/pjli7vUHaipnrTfu2kq/E+MWbFYrKbHDQBsPXlIKaEykyTUfZzQG2BtUdXhLfqfuQQ3Is7KmS/F7J8Z56pndg5FAe9sB9ahFJUNPCCzdLCGQTM4LQKrcTB2ZIhot7tedAHJxrQcLrJk47FqicKNeOlzEcLYv+SGsG9JRw9zOt5fy36F1Zasr1QYa9z4OjBIIN1BSyMzMOtSUp2d65vMTruVjTv9pvKM6IvV7QeJlGugeSCrllJHqlOSZF+TAR4zTaeprZNATLaKzR5rdkH4+o3uHVFCJyqmWSuVeiLY2MUaMDe2ViP8KIIOmmmuisDDVty3HukGIzMXFLjmNmG5nVClitb2ZU7RuaJ0FoOOSLH6gA95RJpUHynS1QsKL594G5nmbgBBCITEy26BuSlrN8wLpwEYKI/uGtKlfgvRajI0Zl+eWcGDucshlZL/0GSx4pgN/mDX4Ep1YATbIDU/qR3oF4o1EzZhY09X5kOulp32rxPozEzGP7zZphTFdPs0sg4sIh5pp51+Z/klQOM11zNtTzAj8vEXkqiLX0XfeZygXNZpNm9+PgNvv6s6CP/uOs2JUYb6+LmRK39sH7ZXY9S99jMh4Pf450neSpzD9kT7CFVa5OuxCK7GMMCZ4aRsZYp9TZ+dl0NMDXMoSZCAWgvQWTf3p0QVMb8srnCehbB+yc6kvtnY/tZLn2+qiEVQf5pWWyxdxDeLW1XKudjU2KIzrADDd5Gr+CA4vnxeFuREGlQUudfETlUDAnOPpQDgEH3bi3PQWMwIUvz8sIwrr93QR/iBCRLFC5kwPjYPIRz60wiho2gnngPBtjwZ8u6BBzZeLazWOfOTPmEHivhe1Io63xCcPUBK4XmsWmAn84NpyjjIzeOWENFk5IMKrxYYneQBBaE0xbMDVeF9KiaIOWM1C7DLt4p eEUQRls8 BMO1/yTFvQS3M6pqR0X5L+4WkEO/ylb7wvhwPDrbfekZx9guee8TXeJaXd4ycIhxdNtklROKA2pGb7+DjUreFmb/zqAI9yN1mWDViYOpWt8UA79jUrlpUmH4Wi1p2Kcj1cnX1M27B4ry47s/uu7OyoUS6rXs+7wWcLzoFrT/c1rkCEFULw3LhpBa2Gg8+zfPh3BS1I9uDF1X+BCZJ7lwUzVQZsbb49bZp/45Y83918jQ+4HU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some new helpers are introduced to free bpf memory, instead of using the generic free helpers. Then we can do something in these new helpers to track the free of bpf memory in the future. Signed-off-by: Yafang Shao --- include/linux/bpf.h | 19 +++++++++++++++++++ kernel/bpf/arraymap.c | 4 ++-- kernel/bpf/bpf_cgrp_storage.c | 2 +- kernel/bpf/bpf_inode_storage.c | 2 +- kernel/bpf/bpf_local_storage.c | 20 ++++++++++---------- kernel/bpf/bpf_task_storage.c | 2 +- kernel/bpf/cpumap.c | 13 ++++++------- kernel/bpf/devmap.c | 10 ++++++---- kernel/bpf/hashtab.c | 8 ++++---- kernel/bpf/helpers.c | 2 +- kernel/bpf/local_storage.c | 12 ++++++------ kernel/bpf/lpm_trie.c | 14 +++++++------- net/core/bpf_sk_storage.c | 4 ++-- net/core/sock_map.c | 2 +- net/xdp/xskmap.c | 2 +- 15 files changed, 68 insertions(+), 48 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index fb14cc6..17c218e 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1869,6 +1869,24 @@ int generic_map_delete_batch(struct bpf_map *map, struct bpf_map *bpf_map_get_curr_or_next(u32 *id); struct bpf_prog *bpf_prog_get_curr_or_next(u32 *id); + +static inline void bpf_map_kfree(const void *ptr) +{ + kfree(ptr); +} + +static inline void bpf_map_kvfree(const void *ptr) +{ + kvfree(ptr); +} + +static inline void bpf_map_free_percpu(void __percpu *ptr) +{ + free_percpu(ptr); +} + +#define bpf_map_kfree_rcu(ptr, rhf...) kvfree_rcu(ptr, ## rhf) + #ifdef CONFIG_MEMCG_KMEM void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, int node); @@ -1877,6 +1895,7 @@ void *bpf_map_kvcalloc(struct bpf_map *map, size_t n, size_t size, gfp_t flags); void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, size_t align, gfp_t flags); + #else static inline void * bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index e64a417..d218bf0 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -24,7 +24,7 @@ static void bpf_array_free_percpu(struct bpf_array *array) int i; for (i = 0; i < array->map.max_entries; i++) { - free_percpu(array->pptrs[i]); + bpf_map_free_percpu(array->pptrs[i]); cond_resched(); } } @@ -1132,7 +1132,7 @@ static void prog_array_map_free(struct bpf_map *map) list_del_init(&elem->list); kfree(elem); } - kfree(aux); + bpf_map_kfree(aux); fd_array_map_free(map); } diff --git a/kernel/bpf/bpf_cgrp_storage.c b/kernel/bpf/bpf_cgrp_storage.c index 6cdf6d9..d0ac4eb 100644 --- a/kernel/bpf/bpf_cgrp_storage.c +++ b/kernel/bpf/bpf_cgrp_storage.c @@ -64,7 +64,7 @@ void bpf_cgrp_storage_free(struct cgroup *cgroup) rcu_read_unlock(); if (free_cgroup_storage) - kfree_rcu(local_storage, rcu); + bpf_map_kfree_rcu(local_storage, rcu); } static struct bpf_local_storage_data * diff --git a/kernel/bpf/bpf_inode_storage.c b/kernel/bpf/bpf_inode_storage.c index 05f4c66..0297f36 100644 --- a/kernel/bpf/bpf_inode_storage.c +++ b/kernel/bpf/bpf_inode_storage.c @@ -78,7 +78,7 @@ void bpf_inode_storage_free(struct inode *inode) rcu_read_unlock(); if (free_inode_storage) - kfree_rcu(local_storage, rcu); + bpf_map_kfree_rcu(local_storage, rcu); } static void *bpf_fd_inode_storage_lookup_elem(struct bpf_map *map, void *key) diff --git a/kernel/bpf/bpf_local_storage.c b/kernel/bpf/bpf_local_storage.c index 35f4138..2fd79a1 100644 --- a/kernel/bpf/bpf_local_storage.c +++ b/kernel/bpf/bpf_local_storage.c @@ -93,9 +93,9 @@ void bpf_local_storage_free_rcu(struct rcu_head *rcu) */ local_storage = container_of(rcu, struct bpf_local_storage, rcu); if (rcu_trace_implies_rcu_gp()) - kfree(local_storage); + bpf_map_kfree(local_storage); else - kfree_rcu(local_storage, rcu); + bpf_map_kfree_rcu(local_storage, rcu); } static void bpf_selem_free_rcu(struct rcu_head *rcu) @@ -104,9 +104,9 @@ static void bpf_selem_free_rcu(struct rcu_head *rcu) selem = container_of(rcu, struct bpf_local_storage_elem, rcu); if (rcu_trace_implies_rcu_gp()) - kfree(selem); + bpf_map_kfree(selem); else - kfree_rcu(selem, rcu); + bpf_map_kfree_rcu(selem, rcu); } /* local_storage->lock must be held and selem->local_storage == local_storage. @@ -162,7 +162,7 @@ static bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_stor if (use_trace_rcu) call_rcu_tasks_trace(&selem->rcu, bpf_selem_free_rcu); else - kfree_rcu(selem, rcu); + bpf_map_kfree_rcu(selem, rcu); return free_local_storage; } @@ -191,7 +191,7 @@ static void __bpf_selem_unlink_storage(struct bpf_local_storage_elem *selem, call_rcu_tasks_trace(&local_storage->rcu, bpf_local_storage_free_rcu); else - kfree_rcu(local_storage, rcu); + bpf_map_kfree_rcu(local_storage, rcu); } } @@ -358,7 +358,7 @@ int bpf_local_storage_alloc(void *owner, return 0; uncharge: - kfree(storage); + bpf_map_kfree(storage); mem_uncharge(smap, owner, sizeof(*storage)); return err; } @@ -402,7 +402,7 @@ struct bpf_local_storage_data * err = bpf_local_storage_alloc(owner, smap, selem, gfp_flags); if (err) { - kfree(selem); + bpf_map_kfree(selem); mem_uncharge(smap, owner, smap->elem_size); return ERR_PTR(err); } @@ -496,7 +496,7 @@ struct bpf_local_storage_data * raw_spin_unlock_irqrestore(&local_storage->lock, flags); if (selem) { mem_uncharge(smap, owner, smap->elem_size); - kfree(selem); + bpf_map_kfree(selem); } return ERR_PTR(err); } @@ -713,6 +713,6 @@ void bpf_local_storage_map_free(struct bpf_map *map, */ synchronize_rcu(); - kvfree(smap->buckets); + bpf_map_kvfree(smap->buckets); bpf_map_area_free(smap); } diff --git a/kernel/bpf/bpf_task_storage.c b/kernel/bpf/bpf_task_storage.c index 1e48605..7287b02 100644 --- a/kernel/bpf/bpf_task_storage.c +++ b/kernel/bpf/bpf_task_storage.c @@ -91,7 +91,7 @@ void bpf_task_storage_free(struct task_struct *task) rcu_read_unlock(); if (free_task_storage) - kfree_rcu(local_storage, rcu); + bpf_map_kfree_rcu(local_storage, rcu); } static void *bpf_pid_task_storage_lookup_elem(struct bpf_map *map, void *key) diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c index e0b2d01..3470c13 100644 --- a/kernel/bpf/cpumap.c +++ b/kernel/bpf/cpumap.c @@ -164,8 +164,8 @@ static void put_cpu_map_entry(struct bpf_cpu_map_entry *rcpu) /* The queue should be empty at this point */ __cpu_map_ring_cleanup(rcpu->queue); ptr_ring_cleanup(rcpu->queue, NULL); - kfree(rcpu->queue); - kfree(rcpu); + bpf_map_kfree(rcpu->queue); + bpf_map_kfree(rcpu); } } @@ -484,11 +484,11 @@ static int __cpu_map_load_bpf_program(struct bpf_cpu_map_entry *rcpu, free_ptr_ring: ptr_ring_cleanup(rcpu->queue, NULL); free_queue: - kfree(rcpu->queue); + bpf_map_kfree(rcpu->queue); free_bulkq: - free_percpu(rcpu->bulkq); + bpf_map_free_percpu(rcpu->bulkq); free_rcu: - kfree(rcpu); + bpf_map_kfree(rcpu); return NULL; } @@ -502,8 +502,7 @@ static void __cpu_map_entry_free(struct rcu_head *rcu) * find this entry. */ rcpu = container_of(rcu, struct bpf_cpu_map_entry, rcu); - - free_percpu(rcpu->bulkq); + bpf_map_free_percpu(rcpu->bulkq); /* Cannot kthread_stop() here, last put free rcpu resources */ put_cpu_map_entry(rcpu); } diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index d01e4c5..be10c47 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -218,7 +218,7 @@ static void dev_map_free(struct bpf_map *map) if (dev->xdp_prog) bpf_prog_put(dev->xdp_prog); dev_put(dev->dev); - kfree(dev); + bpf_map_kfree(dev); } } @@ -234,7 +234,7 @@ static void dev_map_free(struct bpf_map *map) if (dev->xdp_prog) bpf_prog_put(dev->xdp_prog); dev_put(dev->dev); - kfree(dev); + bpf_map_kfree(dev); } bpf_map_area_free(dtab->netdev_map); @@ -791,12 +791,14 @@ static void *dev_map_hash_lookup_elem(struct bpf_map *map, void *key) static void __dev_map_entry_free(struct rcu_head *rcu) { struct bpf_dtab_netdev *dev; + struct bpf_dtab *dtab; dev = container_of(rcu, struct bpf_dtab_netdev, rcu); if (dev->xdp_prog) bpf_prog_put(dev->xdp_prog); dev_put(dev->dev); - kfree(dev); + dtab = dev->dtab; + bpf_map_kfree(dev); } static int dev_map_delete_elem(struct bpf_map *map, void *key) @@ -881,7 +883,7 @@ static struct bpf_dtab_netdev *__dev_map_alloc_node(struct net *net, err_put_dev: dev_put(dev->dev); err_out: - kfree(dev); + bpf_map_kfree(dev); return ERR_PTR(-EINVAL); } diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 5aa2b55..1047788 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -266,7 +266,7 @@ static void htab_free_elems(struct bpf_htab *htab) pptr = htab_elem_get_ptr(get_htab_elem(htab, i), htab->map.key_size); - free_percpu(pptr); + bpf_map_free_percpu(pptr); cond_resched(); } free_elems: @@ -584,7 +584,7 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr) if (htab->use_percpu_counter) percpu_counter_destroy(&htab->pcount); for (i = 0; i < HASHTAB_MAP_LOCK_COUNT; i++) - free_percpu(htab->map_locked[i]); + bpf_map_free_percpu(htab->map_locked[i]); bpf_map_area_free(htab->buckets); bpf_mem_alloc_destroy(&htab->pcpu_ma); bpf_mem_alloc_destroy(&htab->ma); @@ -1511,14 +1511,14 @@ static void htab_map_free(struct bpf_map *map) prealloc_destroy(htab); } - free_percpu(htab->extra_elems); + bpf_map_free_percpu(htab->extra_elems); bpf_map_area_free(htab->buckets); bpf_mem_alloc_destroy(&htab->pcpu_ma); bpf_mem_alloc_destroy(&htab->ma); if (htab->use_percpu_counter) percpu_counter_destroy(&htab->pcount); for (i = 0; i < HASHTAB_MAP_LOCK_COUNT; i++) - free_percpu(htab->map_locked[i]); + bpf_map_free_percpu(htab->map_locked[i]); lockdep_unregister_key(&htab->lockdep_key); bpf_map_area_free(htab); } diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 458db2d..49b0040 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -1383,7 +1383,7 @@ void bpf_timer_cancel_and_free(void *val) */ if (this_cpu_read(hrtimer_running) != t) hrtimer_cancel(&t->timer); - kfree(t); + bpf_map_kfree(t); } BPF_CALL_2(bpf_kptr_xchg, void *, map_value, void *, ptr) diff --git a/kernel/bpf/local_storage.c b/kernel/bpf/local_storage.c index e90d9f6..ed5cf5b 100644 --- a/kernel/bpf/local_storage.c +++ b/kernel/bpf/local_storage.c @@ -174,7 +174,7 @@ static int cgroup_storage_update_elem(struct bpf_map *map, void *key, check_and_init_map_value(map, new->data); new = xchg(&storage->buf, new); - kfree_rcu(new, rcu); + bpf_map_kfree_rcu(new, rcu); return 0; } @@ -526,7 +526,7 @@ struct bpf_cgroup_storage *bpf_cgroup_storage_alloc(struct bpf_prog *prog, return storage; enomem: - kfree(storage); + bpf_map_kfree(storage); return ERR_PTR(-ENOMEM); } @@ -535,8 +535,8 @@ static void free_shared_cgroup_storage_rcu(struct rcu_head *rcu) struct bpf_cgroup_storage *storage = container_of(rcu, struct bpf_cgroup_storage, rcu); - kfree(storage->buf); - kfree(storage); + bpf_map_kfree(storage->buf); + bpf_map_kfree(storage); } static void free_percpu_cgroup_storage_rcu(struct rcu_head *rcu) @@ -544,8 +544,8 @@ static void free_percpu_cgroup_storage_rcu(struct rcu_head *rcu) struct bpf_cgroup_storage *storage = container_of(rcu, struct bpf_cgroup_storage, rcu); - free_percpu(storage->percpu_buf); - kfree(storage); + bpf_map_free_percpu(storage->percpu_buf); + bpf_map_kfree(storage); } void bpf_cgroup_storage_free(struct bpf_cgroup_storage *storage) diff --git a/kernel/bpf/lpm_trie.c b/kernel/bpf/lpm_trie.c index d833496..b2bee07 100644 --- a/kernel/bpf/lpm_trie.c +++ b/kernel/bpf/lpm_trie.c @@ -379,7 +379,7 @@ static int trie_update_elem(struct bpf_map *map, trie->n_entries--; rcu_assign_pointer(*slot, new_node); - kfree_rcu(node, rcu); + bpf_map_kfree_rcu(node, rcu); goto out; } @@ -421,8 +421,8 @@ static int trie_update_elem(struct bpf_map *map, if (new_node) trie->n_entries--; - kfree(new_node); - kfree(im_node); + bpf_map_kfree(new_node); + bpf_map_kfree(im_node); } spin_unlock_irqrestore(&trie->lock, irq_flags); @@ -503,8 +503,8 @@ static int trie_delete_elem(struct bpf_map *map, void *_key) else rcu_assign_pointer( *trim2, rcu_access_pointer(parent->child[0])); - kfree_rcu(parent, rcu); - kfree_rcu(node, rcu); + bpf_map_kfree_rcu(parent, rcu); + bpf_map_kfree_rcu(node, rcu); goto out; } @@ -518,7 +518,7 @@ static int trie_delete_elem(struct bpf_map *map, void *_key) rcu_assign_pointer(*trim, rcu_access_pointer(node->child[1])); else RCU_INIT_POINTER(*trim, NULL); - kfree_rcu(node, rcu); + bpf_map_kfree_rcu(node, rcu); out: spin_unlock_irqrestore(&trie->lock, irq_flags); @@ -602,7 +602,7 @@ static void trie_free(struct bpf_map *map) continue; } - kfree(node); + bpf_map_kfree(node); RCU_INIT_POINTER(*slot, NULL); break; } diff --git a/net/core/bpf_sk_storage.c b/net/core/bpf_sk_storage.c index bb378c3..7b6d7fd 100644 --- a/net/core/bpf_sk_storage.c +++ b/net/core/bpf_sk_storage.c @@ -64,7 +64,7 @@ void bpf_sk_storage_free(struct sock *sk) rcu_read_unlock(); if (free_sk_storage) - kfree_rcu(sk_storage, rcu); + bpf_map_kfree_rcu(sk_storage, rcu); } static void bpf_sk_storage_map_free(struct bpf_map *map) @@ -203,7 +203,7 @@ int bpf_sk_storage_clone(const struct sock *sk, struct sock *newsk) } else { ret = bpf_local_storage_alloc(newsk, smap, copy_selem, GFP_ATOMIC); if (ret) { - kfree(copy_selem); + bpf_map_kfree(copy_selem); atomic_sub(smap->elem_size, &newsk->sk_omem_alloc); bpf_map_put(map); diff --git a/net/core/sock_map.c b/net/core/sock_map.c index 22fa2c5..059e55c 100644 --- a/net/core/sock_map.c +++ b/net/core/sock_map.c @@ -888,7 +888,7 @@ static void sock_hash_free_elem(struct bpf_shtab *htab, struct bpf_shtab_elem *elem) { atomic_dec(&htab->count); - kfree_rcu(elem, rcu); + bpf_map_kfree_rcu(elem, rcu); } static void sock_hash_delete_from_link(struct bpf_map *map, struct sock *sk, diff --git a/net/xdp/xskmap.c b/net/xdp/xskmap.c index 771d0fa..1cb24b1 100644 --- a/net/xdp/xskmap.c +++ b/net/xdp/xskmap.c @@ -33,7 +33,7 @@ static struct xsk_map_node *xsk_map_node_alloc(struct xsk_map *map, static void xsk_map_node_free(struct xsk_map_node *node) { bpf_map_put(&node->map->map); - kfree(node); + bpf_map_kfree(node); } static void xsk_map_sock_add(struct xdp_sock *xs, struct xsk_map_node *node) From patchwork Thu Jan 12 15:53:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13098275 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 802A8C54EBC for ; Thu, 12 Jan 2023 15:53:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 06C918E000D; Thu, 12 Jan 2023 10:53:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F0FAE8E0001; Thu, 12 Jan 2023 10:53:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D15E48E000D; Thu, 12 Jan 2023 10:53:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B58B28E0001 for ; Thu, 12 Jan 2023 10:53:51 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 8932C1C5F3A for ; Thu, 12 Jan 2023 15:53:51 +0000 (UTC) X-FDA: 80346592662.17.51427BF Received: from mail-qt1-f170.google.com (mail-qt1-f170.google.com [209.85.160.170]) by imf17.hostedemail.com (Postfix) with ESMTP id C09634000E for ; Thu, 12 Jan 2023 15:53:49 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=PFL20qpJ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.170 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673538829; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZHBLuOt22MmYduNrNXpPncXZaLhT6xO/3AZAghLO+28=; b=w8ETdJsTIKnkdxK4v6IF+Hx1klkjfi9GBk331jMXyfCVWPcy2qEb64iD+hbSo7MJMQ6HN7 J2svdwpSmBmAD3X3wVEoaUB+2X3cXjOuCP23naEtc3rhODMfM8UMhXjBsA+0AfUrzrSC4d K6atGE5K2FT2MRKlMbu2E4/C8KP2BX4= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=PFL20qpJ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.160.170 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673538829; a=rsa-sha256; cv=none; b=5vvNskLxy3ZJIsNGkf6DNfxL7kInwggVcnCtR8C/Lh58tseTo0F+e3C/ZLA8AVCWtOWpw5 T935+9aC1uu68SqaC/lRU6OpFGSdxBTxhXqBobYO5txD8+bp4R/WRPc9Ri1CiHop2b3iky R1i0rIouGh3vLaGC+Qzm7VGjFqdtxys= Received: by mail-qt1-f170.google.com with SMTP id j15so11295832qtv.4 for ; Thu, 12 Jan 2023 07:53:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZHBLuOt22MmYduNrNXpPncXZaLhT6xO/3AZAghLO+28=; b=PFL20qpJup5OHKv9UPkTO7fFgClOp8U9dXkFVAS69SL27zMiYGmBTuMYEO0GgGywQ0 V193glQ5i21Tw/cAkUxdMyN8zuEXZkM/mm8oX8ZYLoPWz0EWoeX1NqhCNUKoQ0wEjWYY CsQojYbPs+HGgxdT6CxkpJqTasLIyBCw7BrnDBw49VpR/0pX1xavKhdlPIH2loiOhbbZ K4g3kNIRNj3Qg/ZcZSZ2GY7YuAxuiZSwMEh+zl1GIQz2O8ge6wnA+0C4RLes/fgHs3U5 VXhsJZBUy4FhEd+GR2eoW8qt3YKLUTFhpXRwZ4MpqkzMq7f7Dt2wJHqm5Ss1//i4Dm1A eJqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZHBLuOt22MmYduNrNXpPncXZaLhT6xO/3AZAghLO+28=; b=YW3H/63A3sEO/rAM7Agobljc6OUHq1mXJMc1KTOHo2gcvKJEXSfvYS+j08tc1iqM+w kGDzV/0aFW6bxTIBvwXGaKvSN/46t4JlJXiqRWp/kOEx6e+f76ybUG7sWefb0D0Z6BSU mtOktOTH9Vy1fJQYlWIkIbB86wHYXX6IYIhfgHvx3XSq4u2MjmivbWdMdMy8kaMZRBqG Mf1IyYVCgrlOYpXXLcw4ce5Tqhh91VgG3Mr52L+6XWdWUDYHARpDa89jKPxTXTZmIyGL xsh5HiME8q5J8y22YDSaxubu9rRDs/mbCSdd9ifxnUS7sUc3z3/Ka/1vUqjtAh9sJEEx YV3A== X-Gm-Message-State: AFqh2kqgarEFH+G9P9wn1uSIggbot0eslJN1Wrk0oi8/mWdW3WvuiRNC YloUIzIpD7m03p873ybyha0= X-Google-Smtp-Source: AMrXdXtch+HYePOFAPhpj0/byz/XWiJclPWZEbWn/oV7C6FmuW8LQxsnkoeOAzPoqykxf8NIdgRnPQ== X-Received: by 2002:ac8:4601:0:b0:3af:7bf6:d209 with SMTP id p1-20020ac84601000000b003af7bf6d209mr13302883qtn.45.1673538828928; Thu, 12 Jan 2023 07:53:48 -0800 (PST) Received: from vultr.guest ([173.199.122.241]) by smtp.gmail.com with ESMTPSA id l17-20020ac848d1000000b003ab43dabfb1sm9280836qtr.55.2023.01.12.07.53.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jan 2023 07:53:48 -0800 (PST) From: Yafang Shao To: 42.hyeyoo@gmail.com, vbabka@suse.cz, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, roman.gushchin@linux.dev Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [RFC PATCH bpf-next v2 11/11] bpf: introduce bpf memory statistics Date: Thu, 12 Jan 2023 15:53:26 +0000 Message-Id: <20230112155326.26902-12-laoar.shao@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230112155326.26902-1-laoar.shao@gmail.com> References: <20230112155326.26902-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: C09634000E X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 977byg7zgtzru5yd1hh36kqgqp6r3hkk X-HE-Tag: 1673538829-150617 X-HE-Meta: U2FsdGVkX18xJNLFRCGI3yw0CpivhUEZObg0EtE9TmrVGOwLRtlgDg3CjBx5cphz+kuuGXKuMlHFDJQdhr/kqaqw6cvUft01hbU0TPs3NAWXIFiJBZom8rovlijTuzase40lFLS2OgQJIkSWzrWkpZV3MjZ5OYkz0+foWNpNN/VyZPEk040GpV/9qkJmYzsKg9FU/Oa7mF4dUwSBCPBrWcensVaVDrEB3nFxomzunjBmkL2D0+dvF5SJKmHij684fzHQx/5LMHRr8Gic9plKoq1puobYG/2YAdqWLo0bo+r8+BhU+JgaiCekQqu2fvvxRe/RAyZ2c9Eyg8Q+7ub1L2yXtDYB17YUV13x3vphwdPGoKmb3VlroHumg2tA94B8fDsZ6btSj6ZpJQ6EE/m8cmEVM0QtEdY/2bQU2xy6iZFxE9IZX2+gbRj8uLDzWRYIfEuX0CBVoQDleNYpfB0WCeg6HieW8wV2yhxZLjvknln6rzTZDz20Kr67zCe7uxDR9T3E5ahw35VMNGQZwO/P4ZB2TpelAPQ3wwbr63my0W98fFDF3R5UTVjAi1vP6aecxbXMvv1yQjxT9VLVX+H1gHtPsYhdW0rjrPl9g51YIPh6GEwSAAC2B0P7qaR45RNVTOHkCeXmrixZY7MbwbUhSLaR0f016h3M/MQGhSIlP/aximVL2SPmeC0ATY3m0+hmrftfELf+x98w+231Thr+PjBF58X/WJiQWRZlOiqH1Ya5KipX5IhvqI2YuPY9ASeLheZofh+B7jsOwqOUR6dA3uHgSL1/jTVp47oyMcasXtBPzR92LxPVr9iHpej3m6ax/prMIOcgt0RYwIcHV3q8ySbP30acgKkAG5TGZaKK4dzdFjWk8v4P0qkobzvmlKOJMFkfG8ipaRrs7ysK/XINDxNGDzgxy7bPwEWloGKJD7B1ykP3rztYJRixwheEHy1Swpp01I1j+n4ujeJJTxb tgrAu3s7 UjVKtDgXQr7R6YZXkjTAAOx+Ny3QJqGdPRCa52jmc9Lww2ory4XGCwdhm2M7W/mPV4Fd960OswMSE3la9l2vXfTIdbRKgrA3sjKXc2uhi4lp36owZVm+Elxz/4+x2yFMV1t1w0gHJFntc3Ye6qY5C6Z2zZhz9jkMy3Nat0XtPtpc2ZXnblDdEgslgN5B86Pl1i35rJuLbqp+PV5d8PuC39P6z6zKL39nUjoJe45U3kYucwNQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It introduces a new percpu global variable to store bpf memory statistic. It will adds this percpu variable at bpf memory allocation and subs this percpu variable at bpf memory freeing. A new item "BPF" is added into /proc/meminfo to show the bpf memory statistic. Pls. note that there're some deferred freeing, for example the kfree_rcu(), vfree_deferred(). For these deferred freeing, the in-flight memory may be not freed immediately after we subs them from the bpf memory statistic. But it won't take long time to free them, so this behavior is acceptible. Below is the output, $ grep BPF /proc/meminfo BPF: 358 kB Signed-off-by: Yafang Shao --- fs/proc/meminfo.c | 4 +++ include/linux/bpf.h | 92 +++++++++++++++++++++++++++++++++++++++++++++++---- kernel/bpf/memalloc.c | 19 ++++++++++- kernel/bpf/ringbuf.c | 4 +++ kernel/bpf/syscall.c | 39 ++++++++++++++++++++-- 5 files changed, 149 insertions(+), 9 deletions(-) diff --git a/fs/proc/meminfo.c b/fs/proc/meminfo.c index 4409601..5b67331 100644 --- a/fs/proc/meminfo.c +++ b/fs/proc/meminfo.c @@ -16,6 +16,7 @@ #ifdef CONFIG_CMA #include #endif +#include #include #include "internal.h" @@ -159,6 +160,9 @@ static int meminfo_proc_show(struct seq_file *m, void *v) arch_report_meminfo(m); + seq_printf(m, "BPF: %8lu kB\n", + bpf_mem_stat_sum() >> 10); + return 0; } diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 17c218e..add307e 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1869,23 +1869,71 @@ int generic_map_delete_batch(struct bpf_map *map, struct bpf_map *bpf_map_get_curr_or_next(u32 *id); struct bpf_prog *bpf_prog_get_curr_or_next(u32 *id); +struct bpf_mem_stat { + long stat; +}; + +DECLARE_PER_CPU(struct bpf_mem_stat, bpfmm); + +static inline void bpf_mem_stat_add(size_t cnt) +{ + this_cpu_add(bpfmm.stat, cnt); +} + +static inline void bpf_mem_stat_sub(size_t cnt) +{ + this_cpu_sub(bpfmm.stat, cnt); +} + +static inline long bpf_mem_stat_sum(void) +{ + struct bpf_mem_stat *this; + long sum = 0; + int cpu; + + for_each_online_cpu(cpu) { + this = &per_cpu(bpfmm, cpu); + sum += this->stat; + } + + return sum; +} static inline void bpf_map_kfree(const void *ptr) { + size_t sz = ksize_full(ptr); + + if (sz) + bpf_mem_stat_sub(sz); kfree(ptr); } static inline void bpf_map_kvfree(const void *ptr) { + size_t sz = kvsize(ptr); + + if (sz) + bpf_mem_stat_sub(sz); kvfree(ptr); } static inline void bpf_map_free_percpu(void __percpu *ptr) { + size_t sz = percpu_size(ptr); + + if (sz) + bpf_mem_stat_sub(sz); free_percpu(ptr); } -#define bpf_map_kfree_rcu(ptr, rhf...) kvfree_rcu(ptr, ## rhf) +#define bpf_map_kfree_rcu(ptr, rhf...) \ +do { \ + size_t sz = kvsize(ptr); \ + \ + if (sz) \ + bpf_mem_stat_sub(sz); \ + kvfree_rcu(ptr, ## rhf); \ +} while (0) #ifdef CONFIG_MEMCG_KMEM void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, @@ -1901,26 +1949,54 @@ void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, int node) { - return kmalloc_node(size, flags, node); + void *ptr; + size_t sz; + + ptr = kmalloc_node(size, flags, node); + sz = ksize_full(ptr); + if (sz) + bpf_mem_stat_add(sz); + return ptr; } static inline void * bpf_map_kzalloc(const struct bpf_map *map, size_t size, gfp_t flags) { - return kzalloc(size, flags); + void *ptr; + size_t sz; + + ptr = kzalloc(size, flags); + sz = ksize_full(ptr); + if (sz) + bpf_mem_stat_add(sz); + return ptr; } static inline void * bpf_map_kvcalloc(struct bpf_map *map, size_t n, size_t size, gfp_t flags) { - return kvcalloc(n, size, flags); + void *ptr; + size_t sz; + + ptr = kvcalloc(n, size, flags); + sz = kvsize(ptr); + if (sz) + bpf_mem_stat_add(sz); + return ptr; } static inline void __percpu * bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, size_t align, gfp_t flags) { - return __alloc_percpu_gfp(size, align, flags); + void *ptr; + size_t sz; + + ptr = __alloc_percpu_gfp(size, align, flags); + sz = percpu_size(ptr); + if (sz) + bpf_mem_stat_add(sz); + return ptr; } #endif @@ -2461,6 +2537,11 @@ static inline void bpf_prog_inc_misses_counter(struct bpf_prog *prog) static inline void bpf_cgrp_storage_free(struct cgroup *cgroup) { } + +static inline long bpf_mem_stat_sum(void) +{ + return 0; +} #endif /* CONFIG_BPF_SYSCALL */ void __bpf_free_used_btfs(struct bpf_prog_aux *aux, @@ -2886,5 +2967,4 @@ static inline bool type_is_alloc(u32 type) { return type & MEM_ALLOC; } - #endif /* _LINUX_BPF_H */ diff --git a/kernel/bpf/memalloc.c b/kernel/bpf/memalloc.c index ebcc3dd..4e35f287 100644 --- a/kernel/bpf/memalloc.c +++ b/kernel/bpf/memalloc.c @@ -129,6 +129,8 @@ static void *__alloc(struct bpf_mem_cache *c, int node) * want here. */ gfp_t flags = GFP_NOWAIT | __GFP_NOWARN | __GFP_ACCOUNT; + void *ptr; + size_t sz; if (c->percpu_size) { void **obj = kmalloc_node(c->percpu_size, flags, node); @@ -140,10 +142,18 @@ static void *__alloc(struct bpf_mem_cache *c, int node) return NULL; } obj[1] = pptr; + sz = ksize_full(obj); + sz += percpu_size(pptr); + if (sz) + bpf_mem_stat_add(sz); return obj; } - return kmalloc_node(c->unit_size, flags, node); + ptr = kmalloc_node(c->unit_size, flags, node); + sz = ksize_full(ptr); + if (sz) + bpf_mem_stat_add(sz); + return ptr; } static struct mem_cgroup *get_memcg(const struct bpf_mem_cache *c) @@ -215,12 +225,19 @@ static void alloc_bulk(struct bpf_mem_cache *c, int cnt, int node) static void free_one(struct bpf_mem_cache *c, void *obj) { + size_t sz = ksize_full(obj); + if (c->percpu_size) { + sz += percpu_size(((void **)obj)[1]); + if (sz) + bpf_mem_stat_sub(sz); free_percpu(((void **)obj)[1]); kfree(obj); return; } + if (sz) + bpf_mem_stat_sub(sz); kfree(obj); } diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c index 3264bf5..766c2f1 100644 --- a/kernel/bpf/ringbuf.c +++ b/kernel/bpf/ringbuf.c @@ -96,6 +96,7 @@ static void bpf_ringbuf_pages_free(struct page **pages, int nr_pages) { int i; + bpf_mem_stat_sub(nr_pages * PAGE_SIZE); for (i = 0; i < nr_pages; i++) __free_page(pages[i]); bpf_map_area_free(pages); @@ -126,9 +127,12 @@ static struct page **bpf_ringbuf_pages_alloc(int nr_meta_pages, pages[nr_data_pages + i] = page; } + bpf_mem_stat_add(nr_pages * PAGE_SIZE); return pages; err_free_pages: + if (nr_pages) + bpf_mem_stat_add(nr_pages * PAGE_SIZE); bpf_ringbuf_pages_free(pages, nr_pages); err: return NULL; diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 9e266e8..6ca2ceb 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -46,6 +46,7 @@ #define BPF_OBJ_FLAG_MASK (BPF_F_RDONLY | BPF_F_WRONLY) +DEFINE_PER_CPU(struct bpf_mem_stat, bpfmm); DEFINE_PER_CPU(int, bpf_prog_active); static DEFINE_IDR(prog_idr); static DEFINE_SPINLOCK(prog_idr_lock); @@ -336,16 +337,34 @@ static void *__bpf_map_area_alloc(u64 size, int numa_node, bool mmapable) void *bpf_map_area_alloc(u64 size, int numa_node) { - return __bpf_map_area_alloc(size, numa_node, false); + size_t sz; + void *ptr; + + ptr = __bpf_map_area_alloc(size, numa_node, false); + sz = kvsize(ptr); + if (sz) + bpf_mem_stat_add(sz); + return ptr; } void *bpf_map_area_mmapable_alloc(u64 size, int numa_node) { - return __bpf_map_area_alloc(size, numa_node, true); + size_t sz; + void *ptr; + + ptr = __bpf_map_area_alloc(size, numa_node, true); + sz = kvsize(ptr); + if (sz) + bpf_mem_stat_add(sz); + return ptr; } void bpf_map_area_free(void *area) { + size_t sz = kvsize(area); + + if (sz) + bpf_mem_stat_sub(sz); kvfree(area); } @@ -446,12 +465,16 @@ void *bpf_map_kmalloc_node(const struct bpf_map *map, size_t size, gfp_t flags, { struct mem_cgroup *memcg, *old_memcg; void *ptr; + size_t sz; memcg = bpf_map_get_memcg(map); old_memcg = set_active_memcg(memcg); ptr = kmalloc_node(size, flags | __GFP_ACCOUNT, node); set_active_memcg(old_memcg); mem_cgroup_put(memcg); + sz = ksize_full(ptr); + if (sz) + bpf_mem_stat_add(sz); return ptr; } @@ -460,12 +483,16 @@ void *bpf_map_kzalloc(const struct bpf_map *map, size_t size, gfp_t flags) { struct mem_cgroup *memcg, *old_memcg; void *ptr; + size_t sz; memcg = bpf_map_get_memcg(map); old_memcg = set_active_memcg(memcg); ptr = kzalloc(size, flags | __GFP_ACCOUNT); set_active_memcg(old_memcg); mem_cgroup_put(memcg); + sz = ksize_full(ptr); + if (sz) + bpf_mem_stat_add(sz); return ptr; } @@ -475,12 +502,16 @@ void *bpf_map_kvcalloc(struct bpf_map *map, size_t n, size_t size, { struct mem_cgroup *memcg, *old_memcg; void *ptr; + size_t sz; memcg = bpf_map_get_memcg(map); old_memcg = set_active_memcg(memcg); ptr = kvcalloc(n, size, flags | __GFP_ACCOUNT); set_active_memcg(old_memcg); mem_cgroup_put(memcg); + sz = kvsize(ptr); + if (sz) + bpf_mem_stat_add(sz); return ptr; } @@ -490,12 +521,16 @@ void __percpu *bpf_map_alloc_percpu(const struct bpf_map *map, size_t size, { struct mem_cgroup *memcg, *old_memcg; void __percpu *ptr; + size_t sz; memcg = bpf_map_get_memcg(map); old_memcg = set_active_memcg(memcg); ptr = __alloc_percpu_gfp(size, align, flags | __GFP_ACCOUNT); set_active_memcg(old_memcg); mem_cgroup_put(memcg); + sz = percpu_size(ptr); + if (sz) + bpf_mem_stat_add(sz); return ptr; }