From patchwork Thu Jun 23 00:32:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 12891590 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82A65C433EF for ; Thu, 23 Jun 2022 00:32:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236540AbiFWAcl (ORCPT ); Wed, 22 Jun 2022 20:32:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48378 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235713AbiFWAck (ORCPT ); Wed, 22 Jun 2022 20:32:40 -0400 Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 501FE41635 for ; Wed, 22 Jun 2022 17:32:39 -0700 (PDT) Received: by mail-pj1-x102c.google.com with SMTP id x1-20020a17090abc8100b001ec7f8a51f5so989741pjr.0 for ; Wed, 22 Jun 2022 17:32:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ucyEHucarrsO8YiUMSd/dUDVTGwsVMH1bAyNaSQGLDU=; b=QuZkDvieHGiVTgq1UeX4M5a79u/SlEPdzfzjLDR8ZyaeqZ/00xE97+zYNZjfrNWs6W Dz/R7WVXJvGt5X6t5CnGwQ3T7PX95AYpUCM7gcRHs2sPX14faxR0sSF7aX1/l3IGlP8D xYlukeMFK0P2sErH/8BrPvz1paXenZuju7JWaYa98WXWMjoFnFQSO87hkvkNy51xB0tE PAjkdp/LCFpfnqAtaPh0SsYtNeW4Zp7LxLfG1RaArv6DOM/pgDFx+tPk3KjGM/wE8Spv zzr4hS0CtU2GLlC3xQg3lECbnN8WysqOY7/JBzSOewyDE4zMaC7EugcwiWp7iPJhDojJ UNMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ucyEHucarrsO8YiUMSd/dUDVTGwsVMH1bAyNaSQGLDU=; b=0Varba7UHdxYmT0uYL4A8K8wOcG7hGo0EcCgN7OgUKCW1kRVsHNaGOMHtbMnE16kXF 1/So+YUQhapG6JNPgOgZO7GiGHtwEk6zuf2ByST7YVgvtxkfRhHnc0tzmloO25i17791 fL/Kn7/eTLz/yQPajjn92auqVqWlOVkb0Qswt6jYdR7DQ8Y3eth9xuhtb3te08xBehuI CPMa+nwPBREeQJBSvRXrbJ4IeHmHounhgF/d5RhFWg4aV+ri88C8WpLmtCURQnjLZq9k Wht56nvgzr1gBhsRALdOB3ZiXMNPSQ/ppq1YtsqzgJtGiJEnMQtTrFNJ8MLpNAxWg0sU /IWw== X-Gm-Message-State: AJIora//QMGwlqquKi6an4DXaLtJpzUmWOR47kMar7932+40F8i8keur Efmclo/cv2sGgUOiFb2oSmsxPj2DtRQ= X-Google-Smtp-Source: AGRyM1vTDd73BWdb8GSGlIQXMGr1biUbC40rBU/7eFay/JEPrmVjkCZyeRXkbQgFN/OHCHFsEO54wA== X-Received: by 2002:a17:902:8e86:b0:168:d6d6:660f with SMTP id bg6-20020a1709028e8600b00168d6d6660fmr35766190plb.35.1655944358608; Wed, 22 Jun 2022 17:32:38 -0700 (PDT) Received: from macbook-pro-3.dhcp.thefacebook.com ([2620:10d:c090:400::5:29cb]) by smtp.gmail.com with ESMTPSA id j25-20020a62e919000000b0051baeb06c0bsm2545178pfh.168.2022.06.22.17.32.37 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 22 Jun 2022 17:32:38 -0700 (PDT) From: Alexei Starovoitov To: davem@davemloft.net Cc: daniel@iogearbox.net, andrii@kernel.org, tj@kernel.org, kafai@fb.com, bpf@vger.kernel.org, kernel-team@fb.com Subject: [PATCH bpf-next 1/5] bpf: Introduce any context BPF specific memory allocator. Date: Wed, 22 Jun 2022 17:32:26 -0700 Message-Id: <20220623003230.37497-2-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220623003230.37497-1-alexei.starovoitov@gmail.com> References: <20220623003230.37497-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Tracing BPF programs can attach to kprobe and fentry. Hence they run in unknown context where calling plain kmalloc() might not be safe. Front-end kmalloc() with per-cpu per-bucket cache of free elements. Refill this cache asynchronously from irq_work. BPF programs always run with migration disabled. It's safe to allocate from cache of the current cpu with irqs disabled. Free-ing is always done into bucket of the current cpu as well. irq_work trims extra free elements from buckets with kfree and refills them with kmalloc, so global kmalloc logic takes care of freeing objects allocated by one cpu and freed on another. struct bpf_mem_alloc supports two modes: - When size != 0 create kmem_cache and bpf_mem_cache for each cpu. This is typical bpf hash map use case when all elements have equal size. - When size == 0 allocate 11 bpf_mem_cache-s for each cpu, then rely on kmalloc/kfree. Max allocation size is 4096 in this case. This is bpf_dynptr and bpf_kptr use case. Signed-off-by: Alexei Starovoitov --- include/linux/bpf_mem_alloc.h | 26 ++ kernel/bpf/Makefile | 2 +- kernel/bpf/memalloc.c | 512 ++++++++++++++++++++++++++++++++++ 3 files changed, 539 insertions(+), 1 deletion(-) create mode 100644 include/linux/bpf_mem_alloc.h create mode 100644 kernel/bpf/memalloc.c diff --git a/include/linux/bpf_mem_alloc.h b/include/linux/bpf_mem_alloc.h new file mode 100644 index 000000000000..804733070f8d --- /dev/null +++ b/include/linux/bpf_mem_alloc.h @@ -0,0 +1,26 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright (c) 2022 Meta Platforms, Inc. and affiliates. */ +#ifndef _BPF_MEM_ALLOC_H +#define _BPF_MEM_ALLOC_H +#include + +struct bpf_mem_cache; +struct bpf_mem_caches; + +struct bpf_mem_alloc { + struct bpf_mem_caches __percpu *caches; + struct bpf_mem_cache __percpu *cache; +}; + +int bpf_mem_alloc_init(struct bpf_mem_alloc *ma, int size); +void bpf_mem_alloc_destroy(struct bpf_mem_alloc *ma); + +/* kmalloc/kfree equivalent: */ +void *bpf_mem_alloc(struct bpf_mem_alloc *ma, size_t size); +void bpf_mem_free(struct bpf_mem_alloc *ma, void *ptr); + +/* kmem_cache_alloc/free equivalent: */ +void *bpf_mem_cache_alloc(struct bpf_mem_alloc *ma); +void bpf_mem_cache_free(struct bpf_mem_alloc *ma, void *ptr); + +#endif /* _BPF_MEM_ALLOC_H */ diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile index 057ba8e01e70..11fb9220909b 100644 --- a/kernel/bpf/Makefile +++ b/kernel/bpf/Makefile @@ -13,7 +13,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o obj-${CONFIG_BPF_LSM} += bpf_inode_storage.o obj-$(CONFIG_BPF_SYSCALL) += disasm.o obj-$(CONFIG_BPF_JIT) += trampoline.o -obj-$(CONFIG_BPF_SYSCALL) += btf.o +obj-$(CONFIG_BPF_SYSCALL) += btf.o memalloc.o obj-$(CONFIG_BPF_JIT) += dispatcher.o ifeq ($(CONFIG_NET),y) obj-$(CONFIG_BPF_SYSCALL) += devmap.o diff --git a/kernel/bpf/memalloc.c b/kernel/bpf/memalloc.c new file mode 100644 index 000000000000..01b041563fe1 --- /dev/null +++ b/kernel/bpf/memalloc.c @@ -0,0 +1,512 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (c) 2022 Meta Platforms, Inc. and affiliates. */ +#include +#include +#include +#include +#include +#include + +/* Any context (including NMI) BPF specific memory allocator. + * + * Tracing BPF programs can attach to kprobe and fentry. Hence they + * run in unknown context where calling plain kmalloc() might not be safe. + * + * Front-end kmalloc() with per-cpu per-bucket cache of free elements. + * Refill this cache asynchronously from irq_work. + * + * CPU_0 buckets + * 16 32 64 96 128 196 256 512 1024 2048 4096 + * ... + * CPU_N buckets + * 16 32 64 96 128 196 256 512 1024 2048 4096 + * + * The buckets are prefilled at the start. + * BPF programs always run with migration disabled. + * It's safe to allocate from cache of the current cpu with irqs disabled. + * Free-ing is always done into bucket of the current cpu as well. + * irq_work trims extra free elements from buckets with kfree + * and refills them with kmalloc, so global kmalloc logic takes care + * of freeing objects allocated by one cpu and freed on another. + * + * Every allocated objected is padded with extra 8 bytes that contains + * struct llist_node. + */ + +/* similar to kmalloc, but sizeof == 8 bucket is gone */ +static u8 size_index[24] __ro_after_init = { + 3, /* 8 */ + 3, /* 16 */ + 4, /* 24 */ + 4, /* 32 */ + 5, /* 40 */ + 5, /* 48 */ + 5, /* 56 */ + 5, /* 64 */ + 1, /* 72 */ + 1, /* 80 */ + 1, /* 88 */ + 1, /* 96 */ + 6, /* 104 */ + 6, /* 112 */ + 6, /* 120 */ + 6, /* 128 */ + 2, /* 136 */ + 2, /* 144 */ + 2, /* 152 */ + 2, /* 160 */ + 2, /* 168 */ + 2, /* 176 */ + 2, /* 184 */ + 2 /* 192 */ +}; + +static int bpf_mem_cache_idx(size_t size) +{ + if (size <= 192) { + if (!size) + return -1; + + return size_index[(size - 1) / 8] - 1; + } else { + if (size > PAGE_SIZE) + return -1; + + return fls(size - 1) - 1; + } +} + +#define NUM_CACHES 11 + +struct bpf_mem_cache { + /* per-cpu list of free objects of size 'unit_size'. + * All accesses are done with preemption disabled + * with __llist_add() and __llist_del_first(). + */ + struct llist_head free_llist; + + /* NMI only free list. + * All accesses are NMI-safe llist_add() and llist_del_first(). + * + * Each allocated object is either on free_llist or on free_llist_nmi. + * One cpu can allocate it from NMI by doing llist_del_first() from + * free_llist_nmi, while another might free it back from non-NMI by + * doing llist_add() into free_llist. + */ + struct llist_head free_llist_nmi; + /* kmem_cache != NULL when bpf_mem_alloc was created for specific + * element size. + */ + struct kmem_cache *kmem_cache; + struct irq_work refill_work; + struct mem_cgroup *memcg; + int unit_size; + /* count of objects in free_llist */ + int free_cnt; + /* count of objects in free_llist_nmi */ + atomic_t free_cnt_nmi; + /* flag to refill nmi list too */ + bool refill_nmi_list; +}; + +struct bpf_mem_caches { + struct bpf_mem_cache cache[NUM_CACHES]; +}; + +static struct llist_node notrace *__llist_del_first(struct llist_head *head) +{ + struct llist_node *entry, *next; + + entry = head->first; + if (!entry) + return NULL; + next = entry->next; + head->first = next; + return entry; +} + +#define BATCH 48 +#define LOW_WATERMARK 32 +#define HIGH_WATERMARK 96 +/* Assuming the average number of elements per bucket is 64, when all buckets + * are used the total memory will be: 64*16*32 + 64*32*32 + 64*64*32 + ... + + * 64*4096*32 ~ 20Mbyte + */ + +/* extra macro useful for testing by randomizing in_nmi condition */ +#define bpf_in_nmi() in_nmi() + +static void *__alloc(struct bpf_mem_cache *c, int node) +{ + /* Allocate, but don't deplete atomic reserves that typical + * GFP_ATOMIC would do. irq_work runs on this cpu and kmalloc + * will allocate from the current numa node which is what we + * want here. + */ + gfp_t flags = GFP_ATOMIC | __GFP_NOMEMALLOC | __GFP_NOWARN | __GFP_ACCOUNT; + + if (c->kmem_cache) + return kmem_cache_alloc_node(c->kmem_cache, flags, node); + + return kmalloc_node(c->unit_size, flags, node); +} +/* Mostly runs from irq_work except __init phase. */ +static void alloc_bulk(struct bpf_mem_cache *c, int cnt, int node) +{ + struct mem_cgroup *old_memcg; + unsigned long flags; + void *obj; + int i; + + old_memcg = set_active_memcg(c->memcg); + for (i = 0; i < cnt; i++) { + obj = __alloc(c, node); + if (!obj) + break; + if (IS_ENABLED(CONFIG_PREEMPT_RT)) + /* In RT irq_work runs in per-cpu kthread, so we have + * to disable interrupts to avoid race with + * bpf_mem_alloc/free. Note the read of free_cnt in + * bpf_mem_refill is racy in RT. It's ok to do. + */ + local_irq_save(flags); + __llist_add(obj, &c->free_llist); + c->free_cnt++; + if (IS_ENABLED(CONFIG_PREEMPT_RT)) + local_irq_restore(flags); + } + set_active_memcg(old_memcg); +} + +/* Refill NMI specific llist. Mostly runs from irq_work except __init phase. */ +static void alloc_bulk_nmi(struct bpf_mem_cache *c, int cnt, int node) +{ + struct mem_cgroup *old_memcg; + void *obj; + int i; + + old_memcg = set_active_memcg(c->memcg); + for (i = 0; i < cnt; i++) { + obj = __alloc(c, node); + if (!obj) + break; + llist_add(obj, &c->free_llist_nmi); + atomic_inc(&c->free_cnt_nmi); + } + set_active_memcg(old_memcg); +} + +static void __free(struct bpf_mem_cache *c, void *obj) +{ + if (c->kmem_cache) + kmem_cache_free(c->kmem_cache, obj); + else + kfree(obj); +} + +static void free_bulk(struct bpf_mem_cache *c) +{ + struct llist_node *llnode; + unsigned long flags; + int cnt; + + do { + if (IS_ENABLED(CONFIG_PREEMPT_RT)) + local_irq_save(flags); + llnode = __llist_del_first(&c->free_llist); + if (llnode) + cnt = --c->free_cnt; + else + cnt = 0; + if (IS_ENABLED(CONFIG_PREEMPT_RT)) + local_irq_restore(flags); + __free(c, llnode); + } while (cnt > (HIGH_WATERMARK + LOW_WATERMARK) / 2); +} + +static void free_bulk_nmi(struct bpf_mem_cache *c) +{ + struct llist_node *llnode; + int cnt; + + do { + llnode = llist_del_first(&c->free_llist_nmi); + if (llnode) + cnt = atomic_dec_return(&c->free_cnt_nmi); + else + cnt = 0; + __free(c, llnode); + } while (cnt > (HIGH_WATERMARK + LOW_WATERMARK) / 2); +} + +static void bpf_mem_refill(struct irq_work *work) +{ + struct bpf_mem_cache *c = container_of(work, struct bpf_mem_cache, refill_work); + int cnt; + + cnt = c->free_cnt; + if (cnt < LOW_WATERMARK) + /* irq_work runs on this cpu and kmalloc will allocate + * from the current numa node which is what we want here. + */ + alloc_bulk(c, BATCH, NUMA_NO_NODE); + else if (cnt > HIGH_WATERMARK) + free_bulk(c); + + if (!c->refill_nmi_list) + /* don't refill NMI specific freelist + * until alloc/free from NMI. + */ + return; + cnt = atomic_read(&c->free_cnt_nmi); + if (cnt < LOW_WATERMARK) + alloc_bulk_nmi(c, BATCH, NUMA_NO_NODE); + else if (cnt > HIGH_WATERMARK) + free_bulk_nmi(c); + c->refill_nmi_list = false; +} + +static void notrace irq_work_raise(struct bpf_mem_cache *c, bool in_nmi) +{ + c->refill_nmi_list = in_nmi; + irq_work_queue(&c->refill_work); +} + +static void prefill_mem_cache(struct bpf_mem_cache *c, int cpu) +{ + init_irq_work(&c->refill_work, bpf_mem_refill); + /* To avoid consuming memory assume that 1st run of bpf + * prog won't be doing more than 4 map_update_elem from + * irq disabled region + */ + alloc_bulk(c, c->unit_size < 256 ? 4 : 1, cpu_to_node(cpu)); + + /* NMI progs are rare. Assume they have one map_update + * per prog at the very beginning. + */ + alloc_bulk_nmi(c, 1, cpu_to_node(cpu)); +} + +/* When size != 0 create kmem_cache and bpf_mem_cache for each cpu. + * This is typical bpf hash map use case when all elements have equal size. + * + * When size == 0 allocate 11 bpf_mem_cache-s for each cpu, then rely on + * kmalloc/kfree. Max allocation size is 4096 in this case. + * This is bpf_dynptr and bpf_kptr use case. + */ +int bpf_mem_alloc_init(struct bpf_mem_alloc *ma, int size) +{ + static u16 sizes[NUM_CACHES] = {96, 192, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096}; + struct bpf_mem_caches *cc, __percpu *pcc; + struct bpf_mem_cache *c, __percpu *pc; + struct kmem_cache *kmem_cache; + struct mem_cgroup *memcg; + char buf[32]; + int cpu, i; + + if (size) { + pc = __alloc_percpu_gfp(sizeof(*pc), 8, GFP_KERNEL); + if (!pc) + return -ENOMEM; + size += 8; /* room for llist_node */ + snprintf(buf, sizeof(buf), "bpf-%u", size); + kmem_cache = kmem_cache_create(buf, size, 8, 0, NULL); + if (!kmem_cache) { + free_percpu(pc); + return -ENOMEM; + } + memcg = get_mem_cgroup_from_mm(current->mm); + for_each_possible_cpu(cpu) { + c = per_cpu_ptr(pc, cpu); + c->kmem_cache = kmem_cache; + c->unit_size = size; + c->memcg = memcg; + prefill_mem_cache(c, cpu); + } + ma->cache = pc; + return 0; + } + + pcc = __alloc_percpu_gfp(sizeof(*cc), 8, GFP_KERNEL); + if (!pcc) + return -ENOMEM; + memcg = get_mem_cgroup_from_mm(current->mm); + for_each_possible_cpu(cpu) { + cc = per_cpu_ptr(pcc, cpu); + for (i = 0; i < NUM_CACHES; i++) { + c = &cc->cache[i]; + c->unit_size = sizes[i]; + c->memcg = memcg; + prefill_mem_cache(c, cpu); + } + } + ma->caches = pcc; + return 0; +} + +static void drain_mem_cache(struct bpf_mem_cache *c) +{ + struct llist_node *llnode; + + for (;;) { + llnode = llist_del_first(&c->free_llist_nmi); + if (!llnode) + break; + if (c->kmem_cache) + kmem_cache_free(c->kmem_cache, llnode); + else + kfree(llnode); + } + for (;;) { + llnode = __llist_del_first(&c->free_llist); + if (!llnode) + break; + if (c->kmem_cache) + kmem_cache_free(c->kmem_cache, llnode); + else + kfree(llnode); + } +} + +void bpf_mem_alloc_destroy(struct bpf_mem_alloc *ma) +{ + struct bpf_mem_caches *cc; + struct bpf_mem_cache *c; + int cpu, i; + + if (ma->cache) { + for_each_possible_cpu(cpu) { + c = per_cpu_ptr(ma->cache, cpu); + drain_mem_cache(c); + } + /* kmem_cache and memcg are the same across cpus */ + kmem_cache_destroy(c->kmem_cache); + mem_cgroup_put(c->memcg); + free_percpu(ma->cache); + ma->cache = NULL; + } + if (ma->caches) { + for_each_possible_cpu(cpu) { + cc = per_cpu_ptr(ma->caches, cpu); + for (i = 0; i < NUM_CACHES; i++) { + c = &cc->cache[i]; + drain_mem_cache(c); + } + } + mem_cgroup_put(c->memcg); + free_percpu(ma->caches); + ma->caches = NULL; + } +} + +/* notrace is necessary here and in other functions to make sure + * bpf programs cannot attach to them and cause llist corruptions. + */ +static void notrace *unit_alloc(struct bpf_mem_cache *c) +{ + bool in_nmi = bpf_in_nmi(); + struct llist_node *llnode; + unsigned long flags; + int cnt = 0; + + if (unlikely(in_nmi)) { + llnode = llist_del_first(&c->free_llist_nmi); + if (llnode) + cnt = atomic_dec_return(&c->free_cnt_nmi); + } else { + /* Disable irqs to prevent the following race: + * bpf_prog_A + * bpf_mem_alloc + * preemption or irq -> bpf_prog_B + * bpf_mem_alloc + */ + local_irq_save(flags); + llnode = __llist_del_first(&c->free_llist); + if (llnode) + cnt = --c->free_cnt; + local_irq_restore(flags); + } + WARN_ON(cnt < 0); + + if (cnt < LOW_WATERMARK) + irq_work_raise(c, in_nmi); + return llnode; +} + +/* Though 'ptr' object could have been allocated on a different cpu + * add it to the free_llist of the current cpu. + * Let kfree() logic deal with it when it's later called from irq_work. + */ +static void notrace unit_free(struct bpf_mem_cache *c, void *ptr) +{ + struct llist_node *llnode = ptr - 8; + bool in_nmi = bpf_in_nmi(); + unsigned long flags; + int cnt; + + BUILD_BUG_ON(sizeof(struct llist_node) > 8); + + if (unlikely(in_nmi)) { + llist_add(llnode, &c->free_llist_nmi); + cnt = atomic_inc_return(&c->free_cnt_nmi); + } else { + local_irq_save(flags); + __llist_add(llnode, &c->free_llist); + cnt = ++c->free_cnt; + local_irq_restore(flags); + } + WARN_ON(cnt <= 0); + + if (cnt > HIGH_WATERMARK) + /* free few objects from current cpu into global kmalloc pool */ + irq_work_raise(c, in_nmi); +} + +/* Called from BPF program or from sys_bpf syscall. + * In both cases migration is disabled. + */ +void notrace *bpf_mem_alloc(struct bpf_mem_alloc *ma, size_t size) +{ + int idx; + void *ret; + + if (!size) + return ZERO_SIZE_PTR; + + idx = bpf_mem_cache_idx(size + 8); + if (idx < 0) + return NULL; + + ret = unit_alloc(this_cpu_ptr(ma->caches)->cache + idx); + return !ret ? NULL : ret + 8; +} + +void notrace bpf_mem_free(struct bpf_mem_alloc *ma, void *ptr) +{ + int idx; + + if (!ptr) + return; + + idx = bpf_mem_cache_idx(__ksize(ptr - 8)); + if (idx < 0) + return; + + unit_free(this_cpu_ptr(ma->caches)->cache + idx, ptr); +} + +void notrace *bpf_mem_cache_alloc(struct bpf_mem_alloc *ma) +{ + void *ret; + + ret = unit_alloc(this_cpu_ptr(ma->cache)); + return !ret ? NULL : ret + 8; +} + +void notrace bpf_mem_cache_free(struct bpf_mem_alloc *ma, void *ptr) +{ + if (!ptr) + return; + + unit_free(this_cpu_ptr(ma->cache), ptr); +} From patchwork Thu Jun 23 00:32:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 12891591 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4BEF2C433EF for ; Thu, 23 Jun 2022 00:32:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237818AbiFWAco (ORCPT ); Wed, 22 Jun 2022 20:32:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48420 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235713AbiFWAcn (ORCPT ); Wed, 22 Jun 2022 20:32:43 -0400 Received: from mail-pf1-x431.google.com (mail-pf1-x431.google.com [IPv6:2607:f8b0:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E40A241635 for ; Wed, 22 Jun 2022 17:32:42 -0700 (PDT) Received: by mail-pf1-x431.google.com with SMTP id x4so11228647pfq.2 for ; Wed, 22 Jun 2022 17:32:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9T9NR4nKlMACX3pxYCOfVfWAzLl4l8l5nSD0Zg/WaoU=; b=pyzTNfUB8mPg3XTU36LXo5D980GccTgJD/KYvoii7Sro0X0gY7GqCtE3pl40Kye2Qr unHDj9dxMALP1/BgCj7tCoKRbLOzrStCOAwlcG3UcfJ7fAj5TMTnKlHUbPD4i+F/uClR kPsxJUpkgF2gdHS5w55+0RbMcFlqNGL3MDjVinJ6OHiFa7cDh+MAZoIzBMUUHCvAGnam +ymboXyzI5vTjl37P+TYRwOjldY9vUFGKS4lzkWgB6OKCfCkBi4s7yEe4CisA/EUmbD6 rr6ZHEULWByOOx0BdRAI81Dk6bPb2cKV6+J8WCoouYOXSCL8eegC6oCRwmnwK8dLDQT9 BRtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9T9NR4nKlMACX3pxYCOfVfWAzLl4l8l5nSD0Zg/WaoU=; b=JszxqLIN4VqRIf4zJHuox+AZNnfMNBZpSmsS9jDn4Le4dAjVnd0kf7x7CMu2qEoNzn pBm8Dxsqcstxi8KXs1ZWqmVpUg8LMAaDhJjQUgoaIkqeUPVDDYrAhWNCaU0aL5vO5ivz K+DOE3GNe7VVDI/4NHMf08O6IUptO5/kVWl44DDYRw4UuKIL0ah7gCMYfTmDJiJplWF6 wsH6ItqF70ho8T6Yr/3N5bgKyE1W/BmzQ592dy0ypfJbj0tj9hqv5o1G6Fy8VOjPD83G zvLNmahpMc7y21046X+T8Cyekj4sVwbUjjJPWcvDqFz2kz2EGLqdd7lwZ0efbNgbmqe5 q8iw== X-Gm-Message-State: AJIora/NDEEO68hS7SkJYby5knh0BEiMTaTWmIXViWHF3ixdz3TltfI6 9HhIrVDwA/L+qkdDpZsBh6Y= X-Google-Smtp-Source: AGRyM1tWu/A2obwa/13PSfMz3DGE6J0rTR7f/gaJAT3x7rYk4yylYTjSMLywyXNyU2w3g3lC3xJXAg== X-Received: by 2002:a05:6a00:234f:b0:525:1f7c:f2bf with SMTP id j15-20020a056a00234f00b005251f7cf2bfmr20455740pfj.14.1655944362381; Wed, 22 Jun 2022 17:32:42 -0700 (PDT) Received: from macbook-pro-3.dhcp.thefacebook.com ([2620:10d:c090:400::5:29cb]) by smtp.gmail.com with ESMTPSA id mv24-20020a17090b199800b001d954837197sm389339pjb.22.2022.06.22.17.32.40 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 22 Jun 2022 17:32:41 -0700 (PDT) From: Alexei Starovoitov To: davem@davemloft.net Cc: daniel@iogearbox.net, andrii@kernel.org, tj@kernel.org, kafai@fb.com, bpf@vger.kernel.org, kernel-team@fb.com Subject: [PATCH bpf-next 2/5] bpf: Convert hash map to bpf_mem_alloc. Date: Wed, 22 Jun 2022 17:32:27 -0700 Message-Id: <20220623003230.37497-3-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220623003230.37497-1-alexei.starovoitov@gmail.com> References: <20220623003230.37497-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Convert bpf hash map to use bpf memory allocator. Signed-off-by: Alexei Starovoitov --- kernel/bpf/hashtab.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 17fb69c0e0dc..62b7b6e6751b 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -14,6 +14,7 @@ #include "percpu_freelist.h" #include "bpf_lru_list.h" #include "map_in_map.h" +#include #define HTAB_CREATE_FLAG_MASK \ (BPF_F_NO_PREALLOC | BPF_F_NO_COMMON_LRU | BPF_F_NUMA_NODE | \ @@ -92,6 +93,7 @@ struct bucket { struct bpf_htab { struct bpf_map map; + struct bpf_mem_alloc ma; struct bucket *buckets; void *elems; union { @@ -554,6 +556,10 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr) htab_init_buckets(htab); + err = bpf_mem_alloc_init(&htab->ma, htab->elem_size); + if (err) + goto free_map_locked; + if (prealloc) { err = prealloc_init(htab); if (err) @@ -577,6 +583,7 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *attr) for (i = 0; i < HASHTAB_MAP_LOCK_COUNT; i++) free_percpu(htab->map_locked[i]); bpf_map_area_free(htab->buckets); + bpf_mem_alloc_destroy(&htab->ma); free_htab: lockdep_unregister_key(&htab->lockdep_key); kfree(htab); @@ -853,7 +860,7 @@ static void htab_elem_free(struct bpf_htab *htab, struct htab_elem *l) if (htab->map.map_type == BPF_MAP_TYPE_PERCPU_HASH) free_percpu(htab_elem_get_ptr(l, htab->map.key_size)); check_and_free_fields(htab, l); - kfree(l); + bpf_mem_cache_free(&htab->ma, l); } static void htab_elem_free_rcu(struct rcu_head *head) @@ -977,9 +984,7 @@ static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key, l_new = ERR_PTR(-E2BIG); goto dec_count; } - l_new = bpf_map_kmalloc_node(&htab->map, htab->elem_size, - GFP_ATOMIC | __GFP_NOWARN, - htab->map.numa_node); + l_new = bpf_mem_cache_alloc(&htab->ma); if (!l_new) { l_new = ERR_PTR(-ENOMEM); goto dec_count; @@ -998,7 +1003,7 @@ static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key, pptr = bpf_map_alloc_percpu(&htab->map, size, 8, GFP_ATOMIC | __GFP_NOWARN); if (!pptr) { - kfree(l_new); + bpf_mem_cache_free(&htab->ma, l_new); l_new = ERR_PTR(-ENOMEM); goto dec_count; } @@ -1493,6 +1498,7 @@ static void htab_map_free(struct bpf_map *map) bpf_map_free_kptr_off_tab(map); free_percpu(htab->extra_elems); bpf_map_area_free(htab->buckets); + bpf_mem_alloc_destroy(&htab->ma); for (i = 0; i < HASHTAB_MAP_LOCK_COUNT; i++) free_percpu(htab->map_locked[i]); lockdep_unregister_key(&htab->lockdep_key); From patchwork Thu Jun 23 00:32:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 12891592 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DE1EC43334 for ; Thu, 23 Jun 2022 00:32:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235713AbiFWAcv (ORCPT ); Wed, 22 Jun 2022 20:32:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48470 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356387AbiFWAcs (ORCPT ); Wed, 22 Jun 2022 20:32:48 -0400 Received: from mail-pf1-x431.google.com (mail-pf1-x431.google.com [IPv6:2607:f8b0:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C23AE41637 for ; Wed, 22 Jun 2022 17:32:46 -0700 (PDT) Received: by mail-pf1-x431.google.com with SMTP id d17so8595536pfq.9 for ; Wed, 22 Jun 2022 17:32:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=dmkm8LUqLMPRLLj04Upjo1iLNrY0YeL6wYt31D9wn9c=; b=ouJqR47932CNc3bXd7J1Tx/spyOSdMx9ufdDtiPkGqsSyAlY5rWMgqkbF2EPpTxhdG eo+AyZZEnCkSV702OURVg1ljjakvjNBtw9JEFVI3ZZmUIE0jM30fEL96JhgSTm6B9mNo tnzkPtXl/MuGoOqopLE8FaMI3JvInEAaB6fBqi/u4LA8Z3KZz6iFYMxD78glhb8nLV9X Mvcz4x1uRDt7FBhHneFlkbFYUr9JNGkd/2RfyOPv27zb5T5MeZsh/+BEZSRKlv0P3CtE wbDIKx5QEHCo7q8FzSb6XjjI3N3AUym58N+V9+KF2D9PCsPA4vj9OdR2a2MDVSW6scqm mkHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=dmkm8LUqLMPRLLj04Upjo1iLNrY0YeL6wYt31D9wn9c=; b=M4NHlmdz674km80jenVf5V1Q4+TTWx/XvtK5QnmSG32sE/L+ZmB/o9d2cKUmR08sGo wY5v6Jcrq8vid0kuSGBdqGPI234tMt/6YvaY3jbbBvmprmaZ0l76xl120cmGQPBSAPv5 8tNjY+rvzbKZ9MMtanLajcPDz4Wez3Ij8jn4evHGE4hxRiILPn+IK7tLzXRgR1t2I3b9 7j3lPD9opjJ0lXPvRxPABp7BSb2frI2h7684RLoVeo4nRt37rbqGCbDfeyCvyWNhQ0ry NR5iz+7hGygX1cru11NO5QV1XF/mmDkIlmHMUkfT3y8yz0/KJLrcNUJvbGjg3RkYMFw9 odxw== X-Gm-Message-State: AJIora/dA5ccxo5+g/fRmkuRd5Oz7K8yXfKderSBhAb1Yq3Z4akP73Qs doG/GUayC6+RjTfbs/P6GR8= X-Google-Smtp-Source: AGRyM1uhWkUIzzuRn5UBWakegGOzsNVEpb17YQYBJAyFWMQiP+sc0iRlsSz8z4KdxnSdjHW3kxVzjw== X-Received: by 2002:a05:6a00:150a:b0:525:3030:fe41 with SMTP id q10-20020a056a00150a00b005253030fe41mr14568266pfu.37.1655944366083; Wed, 22 Jun 2022 17:32:46 -0700 (PDT) Received: from macbook-pro-3.dhcp.thefacebook.com ([2620:10d:c090:400::5:29cb]) by smtp.gmail.com with ESMTPSA id u8-20020a17090a410800b001ecb5a6385bsm372959pjf.36.2022.06.22.17.32.44 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 22 Jun 2022 17:32:45 -0700 (PDT) From: Alexei Starovoitov To: davem@davemloft.net Cc: daniel@iogearbox.net, andrii@kernel.org, tj@kernel.org, kafai@fb.com, bpf@vger.kernel.org, kernel-team@fb.com Subject: [PATCH bpf-next 3/5] selftests/bpf: Improve test coverage of test_maps Date: Wed, 22 Jun 2022 17:32:28 -0700 Message-Id: <20220623003230.37497-4-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220623003230.37497-1-alexei.starovoitov@gmail.com> References: <20220623003230.37497-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Make test_maps more stressful with more parallelism in update/delete/lookup/walk including different value sizes. Signed-off-by: Alexei Starovoitov --- tools/testing/selftests/bpf/test_maps.c | 38 ++++++++++++++++--------- 1 file changed, 24 insertions(+), 14 deletions(-) diff --git a/tools/testing/selftests/bpf/test_maps.c b/tools/testing/selftests/bpf/test_maps.c index cbebfaa7c1e8..d1ffc76814d9 100644 --- a/tools/testing/selftests/bpf/test_maps.c +++ b/tools/testing/selftests/bpf/test_maps.c @@ -264,10 +264,11 @@ static void test_hashmap_percpu(unsigned int task, void *data) close(fd); } +#define VALUE_SIZE 3 static int helper_fill_hashmap(int max_entries) { int i, fd, ret; - long long key, value; + long long key, value[VALUE_SIZE] = {}; fd = bpf_map_create(BPF_MAP_TYPE_HASH, NULL, sizeof(key), sizeof(value), max_entries, &map_opts); @@ -276,8 +277,8 @@ static int helper_fill_hashmap(int max_entries) "err: %s, flags: 0x%x\n", strerror(errno), map_opts.map_flags); for (i = 0; i < max_entries; i++) { - key = i; value = key; - ret = bpf_map_update_elem(fd, &key, &value, BPF_NOEXIST); + key = i; value[0] = key; + ret = bpf_map_update_elem(fd, &key, value, BPF_NOEXIST); CHECK(ret != 0, "can't update hashmap", "err: %s\n", strerror(ret)); @@ -288,8 +289,8 @@ static int helper_fill_hashmap(int max_entries) static void test_hashmap_walk(unsigned int task, void *data) { - int fd, i, max_entries = 1000; - long long key, value, next_key; + int fd, i, max_entries = 10000; + long long key, value[VALUE_SIZE], next_key; bool next_key_valid = true; fd = helper_fill_hashmap(max_entries); @@ -297,7 +298,7 @@ static void test_hashmap_walk(unsigned int task, void *data) for (i = 0; bpf_map_get_next_key(fd, !i ? NULL : &key, &next_key) == 0; i++) { key = next_key; - assert(bpf_map_lookup_elem(fd, &key, &value) == 0); + assert(bpf_map_lookup_elem(fd, &key, value) == 0); } assert(i == max_entries); @@ -305,9 +306,9 @@ static void test_hashmap_walk(unsigned int task, void *data) assert(bpf_map_get_next_key(fd, NULL, &key) == 0); for (i = 0; next_key_valid; i++) { next_key_valid = bpf_map_get_next_key(fd, &key, &next_key) == 0; - assert(bpf_map_lookup_elem(fd, &key, &value) == 0); - value++; - assert(bpf_map_update_elem(fd, &key, &value, BPF_EXIST) == 0); + assert(bpf_map_lookup_elem(fd, &key, value) == 0); + value[0]++; + assert(bpf_map_update_elem(fd, &key, value, BPF_EXIST) == 0); key = next_key; } @@ -316,8 +317,8 @@ static void test_hashmap_walk(unsigned int task, void *data) for (i = 0; bpf_map_get_next_key(fd, !i ? NULL : &key, &next_key) == 0; i++) { key = next_key; - assert(bpf_map_lookup_elem(fd, &key, &value) == 0); - assert(value - 1 == key); + assert(bpf_map_lookup_elem(fd, &key, value) == 0); + assert(value[0] - 1 == key); } assert(i == max_entries); @@ -1371,16 +1372,16 @@ static void __run_parallel(unsigned int tasks, static void test_map_stress(void) { + run_parallel(100, test_hashmap_walk, NULL); run_parallel(100, test_hashmap, NULL); run_parallel(100, test_hashmap_percpu, NULL); run_parallel(100, test_hashmap_sizes, NULL); - run_parallel(100, test_hashmap_walk, NULL); run_parallel(100, test_arraymap, NULL); run_parallel(100, test_arraymap_percpu, NULL); } -#define TASKS 1024 +#define TASKS 100 #define DO_UPDATE 1 #define DO_DELETE 0 @@ -1432,6 +1433,8 @@ static void test_update_delete(unsigned int fn, void *data) int fd = ((int *)data)[0]; int i, key, value, err; + if (fn & 1) + test_hashmap_walk(fn, NULL); for (i = fn; i < MAP_SIZE; i += TASKS) { key = value = i; @@ -1455,7 +1458,7 @@ static void test_update_delete(unsigned int fn, void *data) static void test_map_parallel(void) { - int i, fd, key = 0, value = 0; + int i, fd, key = 0, value = 0, j = 0; int data[2]; fd = bpf_map_create(BPF_MAP_TYPE_HASH, NULL, sizeof(key), sizeof(value), @@ -1466,6 +1469,7 @@ static void test_map_parallel(void) exit(1); } +again: /* Use the same fd in children to add elements to this map: * child_0 adds key=0, key=1024, key=2048, ... * child_1 adds key=1, key=1025, key=2049, ... @@ -1502,6 +1506,12 @@ static void test_map_parallel(void) key = -1; assert(bpf_map_get_next_key(fd, NULL, &key) < 0 && errno == ENOENT); assert(bpf_map_get_next_key(fd, &key, &key) < 0 && errno == ENOENT); + + key = 0; + bpf_map_delete_elem(fd, &key); + if (j++ < 5) + goto again; + close(fd); } static void test_map_rdonly(void) From patchwork Thu Jun 23 00:32:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 12891593 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BBBBC433EF for ; Thu, 23 Jun 2022 00:32:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351013AbiFWAcv (ORCPT ); Wed, 22 Jun 2022 20:32:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48504 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245202AbiFWAcv (ORCPT ); Wed, 22 Jun 2022 20:32:51 -0400 Received: from mail-pl1-x62a.google.com (mail-pl1-x62a.google.com [IPv6:2607:f8b0:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7801D41635 for ; Wed, 22 Jun 2022 17:32:50 -0700 (PDT) Received: by mail-pl1-x62a.google.com with SMTP id y6so16791438plg.0 for ; Wed, 22 Jun 2022 17:32:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=hINQ/yHfHd8fEcG2GvNFV4wbT1UEVwTut/D2io8x27E=; b=aZL/BW+jGtcjIHjCl7AO1q13jaC1ToE4efM9IgZIpt85g2hO2a3Asjl49WHQ0lGb1n zw1NTf78iAOhOpnfe/8TtT+mWla9vb/vLMdgZXJta1lHZxbkwrw/9KZG4blsGC5iaGtT QuEZGXm140ZEMuU1pzbeXXMjE52TWpQ6z1DPzLUicpIKHSY2HO4pe29u9n6HRytwsQS5 JT8UbdfKskXmECsiH5fUfB5UNBlbGKPDZ2ClTeFxke37YSjO269wptkvjyHKp983ETdi Dboq2B6+lMiPOJdeQbN9njEyfeYd08bnyaqnuYfHb0ZNk/qY9LXbMwSBdAGC0hrwad6Q UAuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hINQ/yHfHd8fEcG2GvNFV4wbT1UEVwTut/D2io8x27E=; b=fv/JgDpZ5EIxJIQKcJTHbWaDT8ZCFsVwQ2ZInWcYw0cvm8FwNRsmKL+9TEVrk97FlX yZQ+WMPR2YuOk7g3r9OxsMtlnYysqT0kQkaX6zmv0JF4WZAWuwmcpUN9NIIwMBboG5LK bK7lJ8+w/99RjW3zMUKHPk6kLXAhLE66PtdNumRxmC8NS+K2VOwu2VS5iiH6FRVXNzRX fhynewhTTWePhtIDGPDyeROuvFJv3iRkl9lUU06zaG6mmtM7FsffeKBpGcg8G36dQYZr 3L6xLxzwpsvPFVO9KiiJ3g0AXU0EGBBf1E9uraTaDiNNf1GLCighInBKWrihU6PpWAzT vBhA== X-Gm-Message-State: AJIora+98qHtuTQzHFPXOvhr7PiQOmSSrvWd9fx4k+p4FQmMkI/t1e+N DK9LVfmk0CZHCVejo99kOLw= X-Google-Smtp-Source: AGRyM1vF5n6k5yd8it0PIFmuxNUMmL64pcFcWXmUGEIGH72f2D5RFekUTbj8+u3ej6uH1od8cNIaYQ== X-Received: by 2002:a17:90b:17cb:b0:1ec:9d52:46f7 with SMTP id me11-20020a17090b17cb00b001ec9d5246f7mr1058082pjb.221.1655944369920; Wed, 22 Jun 2022 17:32:49 -0700 (PDT) Received: from macbook-pro-3.dhcp.thefacebook.com ([2620:10d:c090:400::5:29cb]) by smtp.gmail.com with ESMTPSA id d12-20020a170902e14c00b001624dab05edsm11216447pla.8.2022.06.22.17.32.48 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 22 Jun 2022 17:32:49 -0700 (PDT) From: Alexei Starovoitov To: davem@davemloft.net Cc: daniel@iogearbox.net, andrii@kernel.org, tj@kernel.org, kafai@fb.com, bpf@vger.kernel.org, kernel-team@fb.com Subject: [PATCH bpf-next 4/5] samples/bpf: Reduce syscall overhead in map_perf_test. Date: Wed, 22 Jun 2022 17:32:29 -0700 Message-Id: <20220623003230.37497-5-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220623003230.37497-1-alexei.starovoitov@gmail.com> References: <20220623003230.37497-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Make map_perf_test for preallocated and non-preallocated hash map spend more time inside bpf program to focus performance analysis on the speed of update/lookup/delete operations performed by bpf program. It makes 'perf report' of bpf_mem_alloc look like: 11.76% map_perf_test [k] _raw_spin_lock_irqsave 11.26% map_perf_test [k] htab_map_update_elem 9.70% map_perf_test [k] _raw_spin_lock 9.47% map_perf_test [k] htab_map_delete_elem 8.57% map_perf_test [k] memcpy_erms 5.58% map_perf_test [k] alloc_htab_elem 4.09% map_perf_test [k] __htab_map_lookup_elem 3.44% map_perf_test [k] syscall_exit_to_user_mode 3.13% map_perf_test [k] lookup_nulls_elem_raw 3.05% map_perf_test [k] migrate_enable 3.04% map_perf_test [k] memcmp 2.67% map_perf_test [k] unit_free 2.39% map_perf_test [k] lookup_elem_raw Signed-off-by: Alexei Starovoitov --- samples/bpf/map_perf_test_kern.c | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/samples/bpf/map_perf_test_kern.c b/samples/bpf/map_perf_test_kern.c index 8773f22b6a98..d5af4df9d403 100644 --- a/samples/bpf/map_perf_test_kern.c +++ b/samples/bpf/map_perf_test_kern.c @@ -108,11 +108,14 @@ int stress_hmap(struct pt_regs *ctx) u32 key = bpf_get_current_pid_tgid(); long init_val = 1; long *value; + int i; - bpf_map_update_elem(&hash_map, &key, &init_val, BPF_ANY); - value = bpf_map_lookup_elem(&hash_map, &key); - if (value) - bpf_map_delete_elem(&hash_map, &key); + for (i = 0; i < 30; i++) { + bpf_map_update_elem(&hash_map, &key, &init_val, BPF_ANY); + value = bpf_map_lookup_elem(&hash_map, &key); + if (value) + bpf_map_delete_elem(&hash_map, &key); + } return 0; } @@ -137,11 +140,14 @@ int stress_hmap_alloc(struct pt_regs *ctx) u32 key = bpf_get_current_pid_tgid(); long init_val = 1; long *value; + int i; - bpf_map_update_elem(&hash_map_alloc, &key, &init_val, BPF_ANY); - value = bpf_map_lookup_elem(&hash_map_alloc, &key); - if (value) - bpf_map_delete_elem(&hash_map_alloc, &key); + for (i = 0; i < 30; i++) { + bpf_map_update_elem(&hash_map_alloc, &key, &init_val, BPF_ANY); + value = bpf_map_lookup_elem(&hash_map_alloc, &key); + if (value) + bpf_map_delete_elem(&hash_map_alloc, &key); + } return 0; } From patchwork Thu Jun 23 00:32:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 12891594 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C62D5C43334 for ; Thu, 23 Jun 2022 00:32:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1359223AbiFWAc6 (ORCPT ); Wed, 22 Jun 2022 20:32:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48670 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1358574AbiFWAcz (ORCPT ); Wed, 22 Jun 2022 20:32:55 -0400 Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5E7961124 for ; Wed, 22 Jun 2022 17:32:54 -0700 (PDT) Received: by mail-pj1-x102f.google.com with SMTP id h9-20020a17090a648900b001ecb8596e43so992296pjj.5 for ; Wed, 22 Jun 2022 17:32:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=hireJk5IsliacX/pLET2ULAtcVeuHiddhIKbm7z+wFU=; b=fQECmFTOgP/m6aKXdsyg+p1Ubg3Vr3sG7vpfMmJwraTLKZo9C5ozVYkrj1HBau2OkI lYcnWh3GvzwjzbmVFTnZ/5e9lg3l7u/UkcdUMR0N58Sz0ej4XIIfVL09fALxaO4ctN8d wHvfnAmHu4LXYucvsPaVxLQdupNcQANZPX8Q7jPZ3+th+j7XXSG1z5ycHngxup0W6ol6 +8I8HBRXfFWC9TiGmJsDg7CULJ0nN+rnt8eDqnBC2sP+Sr41s479kXbftoPmNw+xvnHW rHb60HQlIqQzs9WubetNBhRmPlNvBaJIZyuS81hJoRT/PIi7QYMk4675WtKcTPxR8KMP qbGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hireJk5IsliacX/pLET2ULAtcVeuHiddhIKbm7z+wFU=; b=ZFLI1pymEJogN6ko9jFPdvzEYy9o+0vgz2ujHoSHs7G/M1+hq3JHXdixXjz7vXQduy okNo83aGcWHovqpFa9i0y6OR/Ejk9z9ina6bPXNdUASMRtog0Nz9fS9xovBbYKm+pzqF zjKR5tU/LVnhSErCjG8kx25JwZn4IfuMR0jmyQe+K7uMrSaA63FHTpI9kOmWaBwsJ7KH 7XtN+I9dY7NurKhGlsGgwDJvkQhdlt1clhskkRKjyIJtQ7sLjsuaWayHKzVDmnoquL+0 rDQGZ9roodXK3t/J4mm6k/CvbRO4+FzhejWzUYwrmXBL0Io/mTYgmF6aPPZbIn25377u t1SA== X-Gm-Message-State: AJIora/Da/VcrEcKac6cXzlZvpXUZ6yeJruSollOzEuSmAlt5PLB+XM6 5EhaHXmGp1H4ykZWrXsgKiw= X-Google-Smtp-Source: AGRyM1uHQzUnabatIn/SGdVv6GubGphcZqva7td5H86oeR/cyrbArqi1w8nd2zNNcEOJ0LP5KNgJUA== X-Received: by 2002:a17:902:a618:b0:168:9ef0:fb82 with SMTP id u24-20020a170902a61800b001689ef0fb82mr37047704plq.144.1655944373798; Wed, 22 Jun 2022 17:32:53 -0700 (PDT) Received: from macbook-pro-3.dhcp.thefacebook.com ([2620:10d:c090:400::5:29cb]) by smtp.gmail.com with ESMTPSA id b9-20020a17090a550900b001eaec814132sm4373788pji.3.2022.06.22.17.32.52 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 22 Jun 2022 17:32:53 -0700 (PDT) From: Alexei Starovoitov To: davem@davemloft.net Cc: daniel@iogearbox.net, andrii@kernel.org, tj@kernel.org, kafai@fb.com, bpf@vger.kernel.org, kernel-team@fb.com Subject: [PATCH bpf-next 5/5] bpf: Relax the requirement to use preallocated hash maps in tracing progs. Date: Wed, 22 Jun 2022 17:32:30 -0700 Message-Id: <20220623003230.37497-6-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220623003230.37497-1-alexei.starovoitov@gmail.com> References: <20220623003230.37497-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net From: Alexei Starovoitov Since bpf hash map was converted to use bpf_mem_alloc it is safe to use from tracing programs and in RT kernels. But per-cpu hash map is still using dynamic allocation for per-cpu map values, hence keep the warning for this map type. In the future alloc_percpu_gfp can be front-end-ed with bpf_mem_cache and this restriction will be completely lifted. perf_event (NMI) bpf programs have to use preallocated hash maps, because free_htab_elem() is using call_rcu which might crash if re-entered. Sleepable bpf programs have to use preallocated hash maps, because life time of the map elements is not protected by rcu_read_lock/unlock. This restriction can be lifted in the future as well. Signed-off-by: Alexei Starovoitov --- kernel/bpf/verifier.c | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index a20d7736a5b2..90d70304c6f1 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -12566,10 +12566,12 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, * For programs attached to PERF events this is mandatory as the * perf NMI can hit any arbitrary code sequence. * - * All other trace types using preallocated hash maps are unsafe as - * well because tracepoint or kprobes can be inside locked regions - * of the memory allocator or at a place where a recursion into the - * memory allocator would see inconsistent state. + * All other trace types using non-preallocated per-cpu hash maps are + * unsafe as well because tracepoint or kprobes can be inside locked + * regions of the per-cpu memory allocator or at a place where a + * recursion into the per-cpu memory allocator would see inconsistent + * state. Non per-cpu hash maps are using bpf_mem_alloc-tor which is + * safe to use from kprobe/fentry and in RT. * * On RT enabled kernels run-time allocation of all trace type * programs is strictly prohibited due to lock type constraints. On @@ -12579,15 +12581,26 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, */ if (is_tracing_prog_type(prog_type) && !is_preallocated_map(map)) { if (prog_type == BPF_PROG_TYPE_PERF_EVENT) { + /* perf_event bpf progs have to use preallocated hash maps + * because non-prealloc is still relying on call_rcu to free + * elements. + */ verbose(env, "perf_event programs can only use preallocated hash map\n"); return -EINVAL; } - if (IS_ENABLED(CONFIG_PREEMPT_RT)) { - verbose(env, "trace type programs can only use preallocated hash map\n"); - return -EINVAL; + if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH || + (map->inner_map_meta && + map->inner_map_meta->map_type == BPF_MAP_TYPE_PERCPU_HASH)) { + if (IS_ENABLED(CONFIG_PREEMPT_RT)) { + verbose(env, + "trace type programs can only use preallocated per-cpu hash map\n"); + return -EINVAL; + } + WARN_ONCE(1, "trace type BPF program uses run-time allocation\n"); + verbose(env, + "trace type programs with run-time allocated per-cpu hash maps are unsafe." + " Switch to preallocated hash maps.\n"); } - WARN_ONCE(1, "trace type BPF program uses run-time allocation\n"); - verbose(env, "trace type programs with run-time allocated hash maps are unsafe. Switch to preallocated hash maps.\n"); } if (map_value_has_spin_lock(map)) {