From patchwork Thu Feb 2 01:41:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13125208 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44200C636CD for ; Thu, 2 Feb 2023 01:42:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DCD056B007D; Wed, 1 Feb 2023 20:42:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D56946B0088; Wed, 1 Feb 2023 20:42:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B82A66B0089; Wed, 1 Feb 2023 20:42:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A770D6B007D for ; Wed, 1 Feb 2023 20:42:50 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 837291A0C31 for ; Thu, 2 Feb 2023 01:42:50 +0000 (UTC) X-FDA: 80420652900.06.36FB999 Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by imf07.hostedemail.com (Postfix) with ESMTP id BAF8740007 for ; Thu, 2 Feb 2023 01:42:48 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=qw4zWHEG; spf=pass (imf07.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675302168; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IlsPDQQrn+oExLAwKU+9a981T51GRXTZu9kwUUI2F68=; b=FMkIt/G9nM+M8qV2mu9OIPAZ91t9MILCqBWFd6AtiQzQkm5YvZCGkSWpnet8RptOF9KyNl 9O/wnNjWo+I2+uoqoDJmbjp1kIfACdkul+NSPVxaaWlQ6VU5aRsMjECGDpYa7aMxBixD4o xG8Pf44K7jYsbpYaDPH5a0deImDJcFM= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=qw4zWHEG; spf=pass (imf07.hostedemail.com: domain of laoar.shao@gmail.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=laoar.shao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675302168; a=rsa-sha256; cv=none; b=rhinQce3QgmsR37xyhIJu/TYqyfinJ9TkawikIUjP2lm6KKxoJRuO9RuQRMQCZCJvi7YGr WhiHDLkSdxuxJ+NYpvqQexYn2fUIIOwFComl/4J8/rZVMvX4FHQcmeZkxaM+iKxzYijLId OkMsbeV91cMluJ2DDKJ+OVR9SyKq8YU= Received: by mail-pj1-f42.google.com with SMTP id t12-20020a17090aae0c00b00229f4cff534so4600159pjq.1 for ; Wed, 01 Feb 2023 17:42:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IlsPDQQrn+oExLAwKU+9a981T51GRXTZu9kwUUI2F68=; b=qw4zWHEG5rlxLuRA2Mk2zQxmQoWx+8vAJfsjWhcoXTLwNOskwzmVzveBoO/Ik8mj/K jYB40qB1HcmjfgW4nXG7OjV/04e3FIRK51MW/KIInIHJEk0gTzEruSbVfzBqCj0+BiUm A4jK+3jrJFm1TCQvZF4dDc0Hsr3bf+i2XLlq1IWjcxdzswxaf3xPKKRBt1mT5zyO29ep 0rbTQVBit/K5XjnUmlEiG9J8i37h23wW7+oTDwbuVwt7QV8z4O43KsfzUKSHPq/6HiHS Ug6SgpdJ17MIdoCo8XPN3c51YsQgozW6lYKoWgLSp2jmPM9kcpakKcbGoAantIgGh4uQ iC3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IlsPDQQrn+oExLAwKU+9a981T51GRXTZu9kwUUI2F68=; b=ond/aud/rO4xqnFOKTEd/Vue7NndkXL9cVvAzSbCxFKd5a5HvyOwuRFIqsYkuYNUdL qRFv6cfvLFvZw1TmO1DJ9kiQnZdeW/BniqA2NYUI1g/evaW6RyS27NPqUksDlEAjDaRB 21RGXCgrBw78QLlKmg30FqmCtsLCz9C50BTZG0sf1ZI8U07jrsLS2qIeGt/+gPzzKEr5 Vqp7YLkH4Y68vNsNxK8bpgMvuoyt/y2tgbfrq90FijlyxuysLs6/MUqfyfrLt6H/dFrB EM4Xuf8+KHruTnL68qORgpkkIS95NAlJ1RGebo/5+TBwi08LRSYhTEpNh89FzgIkteGg 3IfA== X-Gm-Message-State: AO0yUKVFTdd4VSdPl1yChOmoqjdjdWsJ+Awy81LbeMkMXC08UbQ1stte zUpJdXBFhiEX1bnhct1rTCk= X-Google-Smtp-Source: AK7set/OuunuF82xfnkKr4JO7JphAerYxqJCUhDt5Rs18OlwYb7xCNJ/ZlBTM0ydmhVnZGVpcgln1Q== X-Received: by 2002:a05:6a20:158e:b0:b8:54a2:b8d1 with SMTP id h14-20020a056a20158e00b000b854a2b8d1mr5750054pzj.57.1675302167711; Wed, 01 Feb 2023 17:42:47 -0800 (PST) Received: from vultr.guest ([2001:19f0:7001:3f48:5400:4ff:fe4a:8c8b]) by smtp.gmail.com with ESMTPSA id t191-20020a6381c8000000b004e8f7f23c4bsm6594205pgd.76.2023.02.01.17.42.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Feb 2023 17:42:47 -0800 (PST) From: Yafang Shao To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, vbabka@suse.cz, urezki@gmail.com Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Subject: [PATCH bpf-next 7/7] bpf: hashtab memory usage Date: Thu, 2 Feb 2023 01:41:58 +0000 Message-Id: <20230202014158.19616-8-laoar.shao@gmail.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230202014158.19616-1-laoar.shao@gmail.com> References: <20230202014158.19616-1-laoar.shao@gmail.com> MIME-Version: 1.0 X-Stat-Signature: 8zz5cj16s1btpnbdq9i1fz6nae4pmdux X-Rspam-User: X-Rspamd-Queue-Id: BAF8740007 X-Rspamd-Server: rspam06 X-HE-Tag: 1675302168-505164 X-HE-Meta: U2FsdGVkX1+rcuLdtgsouEx9Qs/WIwfnI6crvodnQf+AvT73mRdCh/yBnazafO1wMgW6JuZPbGUjB7deHEDNrfHWT0E7uJdyGGX8sbJat8JV/4kVvlYlHZuJk1yPDu9I1AnD13TLC6BIYZe2VtrrZRHLd26uFF0q8b4ug2RjIXllqAlsn2q7g8egEbL+lDhUzZ4iBEvzji9GbxsmGuOjU8XKFgDdrdz3sIq4Yt9uRBYf0rB4BOSuH2DHS7gIyWjMSPZ2e9GMFnCcaBDQ8/Uuq85vdIXVCztC0BpSw0MU+Ia6kzcsIegFz7pw3UTZ/6Srk8DYd/Vf75fs4417UJ2lvFZ9jPEghwG2D1jt563HrtsVpWj59ASZjUJMMfz+w2YvPXfRpyHXSNX5+LTk7L/5Cn+IR4IfjY+kVWrzkTA0Pq/ZFoZ8OuI8SqUqfjOrj4Vs4f2qFAxsVshpiPxEGUtXXihswlC39JaZrUu3yZHVFPZ+1tD05Fl40S1Fs6ao+Rx6NVF2e2lyF7xpjoyL+qo+zcFvqOBYu71+XrE48bR+mzPHvH0Fxrt1Hu9VggEwWSFIp4XDS5f6T7kJ1S2tQS6vn53LZM1uMKPCqrLdPOoWRHqMg/4UMedraIvc27PXnxLWXKwEVRcHsp5ZNWcqwKOrLmsypUC+EoKEPjD9NBmdc0k6OA5F7SpdN5LTLVhrfVaCqGvpqMEBYjb+Yxom3IQpPIRp3VunfWtSlHvOiN72ZjifLgwDzGiFrN46Rp7B4FJmTvW1JJqWwTXRYxuLZCzlucTlxoYNGCbATVsn1d61I2BoU3gVJVE6BQ9tnTSjGGJ9eSsSyVK7i9wn8XjCwPRpY/wKO6kDMxxKmiF/QiLTiiRY/+Y4dxor41sKXrA1BvD4N6UA3FVL98Or3iXOMkpsv/B5IBlTEqLqVHVV1HE4sgp20Wnz/O38fmJC+dRPvZ8coIYVzXoZZ3D+RnJE8cV l1FBccFs DW7RsqZq7kpfakLH7Im4w9DD52jLKyiY8iqPOfz5PO7UsDqffDzPCS7Nr/FfNkiorpONHfbifq89s7rcjSuzzeTbbeerLvRuMSbud30U+qQBHiyNfoPX7EUbfgszjimu162gY8CeobeZ4sZ3ZBUM59OGIsnfa035+FmqELvdL/AE0LR5zdyczI/3PEa/nh29JO8UjR74UBgjC2SSDEdnL9CYcvH1dRyosjdQCElia8VXXE6hIRzu8JJKJUWwn3FnIpluF2YiHAb70+UEwOlVu9YMU5mmcal1TPdJf2+gghPKKllIyGS546toq0rICeSDSuP79H1wd83uy45bKnSXLMrbAwmIU9BfoL4rAKHBJWIxPai8W0d3H+BZ3/heVwTkc4bs/XJMhWlm+szGMjQ5X8+nVKsszDYiyD6wQwqzdqb0vXvOL7jAIAnwt4fx2/btAPWZUh6edGEzK3rtT0AuOlvv2qkWwBMizP+cB+wDPwHqjfCfF4snB0MAlRmaCeI7Xsil5A9mT7JBHdKiUisnxpIxofhvm8tqYA6Pu6+BMqK6ugrw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Get htab memory usage from the htab pointers we have allocated. Some small pointers are ignored as their size are quite small compared with the total size. The result as follows, - before this change 1: hash name count_map flags 0x0 <<<< prealloc key 16B value 24B max_entries 1048576 memlock 41943040B 2: hash name count_map flags 0x1 <<<< non prealloc, fully set key 16B value 24B max_entries 1048576 memlock 41943040B 3: hash name count_map flags 0x1 <<<< non prealloc, non set key 16B value 24B max_entries 1048576 memlock 41943040B The memlock is always a fixed number whatever it is preallocated or not, and whatever the allocated elements number is. - after this change 1: hash name count_map flags 0x0 <<<< prealloc key 16B value 24B max_entries 1048576 memlock 109064464B 2: hash name count_map flags 0x1 <<<< non prealloc, fully set key 16B value 24B max_entries 1048576 memlock 117464320B 3: hash name count_map flags 0x1 <<<< non prealloc, non set key 16B value 24B max_entries 1048576 memlock 16797952B The memlock now is hashtab actually allocated. At worst, the difference can be 10x, for example, - before this change 4: hash name count_map flags 0x0 key 4B value 4B max_entries 1048576 memlock 8388608B - after this change 4: hash name count_map flags 0x0 key 4B value 4B max_entries 1048576 memlock 83898640B Signed-off-by: Yafang Shao --- kernel/bpf/hashtab.c | 80 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 79 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 66bded1..cba540b 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -273,6 +273,25 @@ static void htab_free_elems(struct bpf_htab *htab) bpf_map_area_free(htab->elems); } +static unsigned long htab_prealloc_elems_size(struct bpf_htab *htab) +{ + unsigned long size = 0; + int i; + + if (!htab_is_percpu(htab)) + return kvsize(htab->elems); + + for (i = 0; i < htab->map.max_entries; i++) { + void __percpu *pptr; + + pptr = htab_elem_get_ptr(get_htab_elem(htab, i), + htab->map.key_size); + size += percpu_size(pptr); + } + size += kvsize(htab->elems); + return size; +} + /* The LRU list has a lock (lru_lock). Each htab bucket has a lock * (bucket_lock). If both locks need to be acquired together, the lock * order is always lru_lock -> bucket_lock and this only happens in @@ -864,6 +883,16 @@ static void htab_elem_free(struct bpf_htab *htab, struct htab_elem *l) bpf_mem_cache_free(&htab->ma, l); } +static unsigned long htab_elem_size(struct bpf_htab *htab, struct htab_elem *l) +{ + unsigned long size = 0; + + if (htab->map.map_type == BPF_MAP_TYPE_PERCPU_HASH) + size += bpf_mem_cache_elem_size(&htab->pcpu_ma, l->ptr_to_pptr); + + return size + bpf_mem_cache_elem_size(&htab->ma, l); +} + static void htab_put_fd_value(struct bpf_htab *htab, struct htab_elem *l) { struct bpf_map *map = &htab->map; @@ -899,7 +928,6 @@ static void dec_elem_count(struct bpf_htab *htab) atomic_dec(&htab->count); } - static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l) { htab_put_fd_value(htab, l); @@ -1457,6 +1485,31 @@ static void delete_all_elements(struct bpf_htab *htab) migrate_enable(); } +static unsigned long htab_non_prealloc_elems_size(struct bpf_htab *htab) +{ + unsigned long size = 0; + unsigned long count; + int i; + + rcu_read_lock(); + for (i = 0; i < htab->n_buckets; i++) { + struct hlist_nulls_head *head = select_bucket(htab, i); + struct hlist_nulls_node *n; + struct htab_elem *l; + + hlist_nulls_for_each_entry(l, n, head, hash_node) { + size = htab_elem_size(htab, l); + goto out; + } + } +out: + rcu_read_unlock(); + count = htab->use_percpu_counter ? percpu_counter_sum(&htab->pcount) : + atomic_read(&htab->count); + + return size * count; +} + static void htab_free_malloced_timers(struct bpf_htab *htab) { int i; @@ -1523,6 +1576,26 @@ static void htab_map_free(struct bpf_map *map) bpf_map_area_free(htab); } +/* Get the htab memory usage from pointers we have already allocated. + * Some minor pointers are igored as their size are quite small compared + * with the total size. + */ +static unsigned long htab_mem_usage(const struct bpf_map *map) +{ + struct bpf_htab *htab = container_of(map, struct bpf_htab, map); + unsigned long size = 0; + + if (!htab_is_prealloc(htab)) + size += htab_non_prealloc_elems_size(htab); + else + size += htab_prealloc_elems_size(htab); + size += percpu_size(htab->extra_elems); + size += kvsize(htab->buckets); + size += bpf_mem_alloc_size(&htab->pcpu_ma); + size += bpf_mem_alloc_size(&htab->ma); + return size; +} + static void htab_map_seq_show_elem(struct bpf_map *map, void *key, struct seq_file *m) { @@ -2191,6 +2264,7 @@ static int bpf_for_each_hash_elem(struct bpf_map *map, bpf_callback_t callback_f .map_seq_show_elem = htab_map_seq_show_elem, .map_set_for_each_callback_args = map_set_for_each_callback_args, .map_for_each_callback = bpf_for_each_hash_elem, + .map_mem_usage = htab_mem_usage, BATCH_OPS(htab), .map_btf_id = &htab_map_btf_ids[0], .iter_seq_info = &iter_seq_info, @@ -2212,6 +2286,7 @@ static int bpf_for_each_hash_elem(struct bpf_map *map, bpf_callback_t callback_f .map_seq_show_elem = htab_map_seq_show_elem, .map_set_for_each_callback_args = map_set_for_each_callback_args, .map_for_each_callback = bpf_for_each_hash_elem, + .map_mem_usage = htab_mem_usage, BATCH_OPS(htab_lru), .map_btf_id = &htab_map_btf_ids[0], .iter_seq_info = &iter_seq_info, @@ -2363,6 +2438,7 @@ static void htab_percpu_map_seq_show_elem(struct bpf_map *map, void *key, .map_seq_show_elem = htab_percpu_map_seq_show_elem, .map_set_for_each_callback_args = map_set_for_each_callback_args, .map_for_each_callback = bpf_for_each_hash_elem, + .map_mem_usage = htab_mem_usage, BATCH_OPS(htab_percpu), .map_btf_id = &htab_map_btf_ids[0], .iter_seq_info = &iter_seq_info, @@ -2382,6 +2458,7 @@ static void htab_percpu_map_seq_show_elem(struct bpf_map *map, void *key, .map_seq_show_elem = htab_percpu_map_seq_show_elem, .map_set_for_each_callback_args = map_set_for_each_callback_args, .map_for_each_callback = bpf_for_each_hash_elem, + .map_mem_usage = htab_mem_usage, BATCH_OPS(htab_lru_percpu), .map_btf_id = &htab_map_btf_ids[0], .iter_seq_info = &iter_seq_info, @@ -2519,6 +2596,7 @@ static void htab_of_map_free(struct bpf_map *map) .map_fd_sys_lookup_elem = bpf_map_fd_sys_lookup_elem, .map_gen_lookup = htab_of_map_gen_lookup, .map_check_btf = map_check_no_btf, + .map_mem_usage = htab_mem_usage, BATCH_OPS(htab), .map_btf_id = &htab_map_btf_ids[0], };