From patchwork Mon Jan 27 16:21:55 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Hwang X-Patchwork-Id: 13951607 X-Patchwork-Delegate: bpf@iogearbox.net Received: from out-172.mta1.migadu.com (out-172.mta1.migadu.com [95.215.58.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C1E9F14E2E8 for ; Mon, 27 Jan 2025 16:22:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737994953; cv=none; b=Toxi4PAbNHUQKTMEHQubKifqIhJs6e3cMUdecpRCYmfKSJxToR/c9j3A+mzXje2lACwY5hoHJqH5aexa4iQBU9f7CJXE4sU2lkvB7EHUF9u+lqltk91QIoVMrAZcAf9yOW4fEoLofOp2FaClfVbeZGMmULyS3SFkOVCbykD4lbM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737994953; c=relaxed/simple; bh=7DbjlrbGZB02APpDJdzLCYA2uOUpF1xjr7UTQyNEbaI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ho88Lz+cGM+2GdX2CZlSfTbN6d4e9br1EOjKn5BUNM467ThrbpiOOb+CNL8SQim8i5RrCNjVEorQBtIBbH9TQeqU3Rv7njgArr4GhL9cRGx2KI0hDeG9fRJ9drzset1PdNVT6Di4aoKQIjzDygH+ni0XbGbZ5ebXRiIo+i39z3E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=CvoM9Yhb; arc=none smtp.client-ip=95.215.58.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="CvoM9Yhb" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1737994948; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2sEzi/ol8biiXzqNX4u2Cupxa8A39NoWvtgT6jwYkrM=; b=CvoM9YhbslOakSp45eu2gnTKqccXJZGsRNmfLnr8VA0UkTnzVLWCG13AxAkajL81rjyPp1 LYhboXaixM0/WYrRVF+ZUTH39KZg7k+0cUxnGLIgsoa0kDmPjuHJvYSbF6BwFBUGEjhDlX t12dA6ov3xsxhT+s4pAtn2rnNcl7sjM= From: Leon Hwang To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, yonghong.song@linux.dev, song@kernel.org, eddyz87@gmail.com, qmo@kernel.org, dxu@dxuuu.xyz, leon.hwang@linux.dev, kernel-patches-bot@fb.com Subject: [PATCH bpf-next 1/4] bpf: Introduce global percpu data Date: Tue, 28 Jan 2025 00:21:55 +0800 Message-ID: <20250127162158.84906-2-leon.hwang@linux.dev> In-Reply-To: <20250127162158.84906-1-leon.hwang@linux.dev> References: <20250127162158.84906-1-leon.hwang@linux.dev> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Patchwork-Delegate: bpf@iogearbox.net This patch introduces global percpu data, inspired by commit 6316f78306c1 ("Merge branch 'support-global-data'"). It enables the definition of global percpu variables in BPF, similar to the DEFINE_PER_CPU() macro in the kernel[0]. For example, in BPF, it is able to define a global percpu variable like this: int percpu_data SEC(".percpu"); With this patch, tools like retsnoop[1] and bpflbr[2] can simplify their BPF code for handling LBRs. The code can be updated from static struct perf_branch_entry lbrs[1][MAX_LBR_ENTRIES] SEC(".data.lbrs"); to static struct perf_branch_entry lbrs[MAX_LBR_ENTRIES] SEC(".percpu.lbrs"); This eliminates the need to retrieve the CPU ID using the bpf_get_smp_processor_id() helper. Additionally, by reusing global percpu data map, sharing information between tail callers and callees or freplace callers and callees becomes simpler compared to reusing percpu_array maps. Links: [0] https://github.com/torvalds/linux/blob/fbfd64d25c7af3b8695201ebc85efe90be28c5a3/include/linux/percpu-defs.h#L114 [1] https://github.com/anakryiko/retsnoop [2] https://github.com/Asphaltt/bpflbr Signed-off-by: Leon Hwang --- kernel/bpf/arraymap.c | 39 ++++++++++++++++++++++++++++++++++++- kernel/bpf/verifier.c | 45 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 83 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index eb28c0f219ee4..f8c60d8331975 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -249,6 +249,40 @@ static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key) return this_cpu_ptr(array->pptrs[index & array->index_mask]); } +static int percpu_array_map_direct_value_addr(const struct bpf_map *map, + u64 *imm, u32 off) +{ + struct bpf_array *array = container_of(map, struct bpf_array, map); + + if (map->max_entries != 1) + return -EOPNOTSUPP; + if (off >= map->value_size) + return -EINVAL; + if (!bpf_jit_supports_percpu_insn()) + return -EOPNOTSUPP; + + *imm = (u64) array->pptrs[0]; + return 0; +} + +static int percpu_array_map_direct_value_meta(const struct bpf_map *map, + u64 imm, u32 *off) +{ + struct bpf_array *array = container_of(map, struct bpf_array, map); + u64 base = (u64) array->pptrs[0]; + u64 range = array->elem_size; + + if (map->max_entries != 1) + return -EOPNOTSUPP; + if (imm < base || imm >= base + range) + return -ENOENT; + if (!bpf_jit_supports_percpu_insn()) + return -EOPNOTSUPP; + + *off = imm - base; + return 0; +} + /* emit BPF instructions equivalent to C code of percpu_array_map_lookup_elem() */ static int percpu_array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf) { @@ -534,7 +568,8 @@ static int array_map_check_btf(const struct bpf_map *map, /* One exception for keyless BTF: .bss/.data/.rodata map */ if (btf_type_is_void(key_type)) { - if (map->map_type != BPF_MAP_TYPE_ARRAY || + if ((map->map_type != BPF_MAP_TYPE_ARRAY && + map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) || map->max_entries != 1) return -EINVAL; @@ -815,6 +850,8 @@ const struct bpf_map_ops percpu_array_map_ops = { .map_get_next_key = array_map_get_next_key, .map_lookup_elem = percpu_array_map_lookup_elem, .map_gen_lookup = percpu_array_map_gen_lookup, + .map_direct_value_addr = percpu_array_map_direct_value_addr, + .map_direct_value_meta = percpu_array_map_direct_value_meta, .map_update_elem = array_map_update_elem, .map_delete_elem = array_map_delete_elem, .map_lookup_percpu_elem = percpu_array_map_lookup_percpu_elem, diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 9971c03adfd5d..9d99497c2b94c 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6810,6 +6810,8 @@ static int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val, u64 addr; int err; + if (map->map_type != BPF_MAP_TYPE_ARRAY) + return -EINVAL; err = map->ops->map_direct_value_addr(map, &addr, off); if (err) return err; @@ -7322,6 +7324,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn /* if map is read-only, track its contents as scalars */ if (tnum_is_const(reg->var_off) && bpf_map_is_rdonly(map) && + map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY && map->ops->map_direct_value_addr) { int map_off = off + reg->var_off.value; u64 val = 0; @@ -9128,6 +9131,11 @@ static int check_reg_const_str(struct bpf_verifier_env *env, return -EACCES; } + if (map->map_type != BPF_MAP_TYPE_ARRAY) { + verbose(env, "only array map supports direct string value access\n"); + return -EINVAL; + } + err = check_map_access(env, regno, reg->off, map->value_size - reg->off, false, ACCESS_HELPER); @@ -10802,6 +10810,11 @@ static int check_bpf_snprintf_call(struct bpf_verifier_env *env, return -EINVAL; num_args = data_len_reg->var_off.value / 8; + if (fmt_map->map_type != BPF_MAP_TYPE_ARRAY) { + verbose(env, "only array map supports snprintf\n"); + return -EINVAL; + } + /* fmt being ARG_PTR_TO_CONST_STR guarantees that var_off is const * and map_direct_value_addr is set. */ @@ -21381,6 +21394,38 @@ static int do_misc_fixups(struct bpf_verifier_env *env) goto next_insn; } +#ifdef CONFIG_SMP + if (insn->code == (BPF_LD | BPF_IMM | BPF_DW) && + (insn->src_reg == BPF_PSEUDO_MAP_VALUE || + insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE)) { + struct bpf_map *map; + + aux = &env->insn_aux_data[i + delta]; + map = env->used_maps[aux->map_index]; + if (map->map_type != BPF_MAP_TYPE_PERCPU_ARRAY) + goto next_insn; + + /* Reuse the original ld_imm64 insn. And add one + * mov64_percpu_reg insn. + */ + + insn_buf[0] = insn[1]; + insn_buf[1] = BPF_MOV64_PERCPU_REG(insn->dst_reg, insn->dst_reg); + cnt = 2; + + i++; + new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt); + if (!new_prog) + return -ENOMEM; + + delta += cnt - 1; + env->prog = prog = new_prog; + insn = new_prog->insnsi + i + delta; + + goto next_insn; + } +#endif + if (insn->code != (BPF_JMP | BPF_CALL)) goto next_insn; if (insn->src_reg == BPF_PSEUDO_CALL) From patchwork Mon Jan 27 16:21:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Hwang X-Patchwork-Id: 13951608 X-Patchwork-Delegate: bpf@iogearbox.net Received: from out-172.mta1.migadu.com (out-172.mta1.migadu.com [95.215.58.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 103C513633F for ; Mon, 27 Jan 2025 16:22:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737994963; cv=none; b=tPCuzXFN/KSYNIqROjplDLqd1xtIKOppIKDq26BQLa79ODYsOhCaxfQ4lT/Jmd9LUQyFb6NYZiHfwMLCpp6KIWBZYRrlAOML++lJleqYZqyJHjQsVNuGm/HATjpdgqzT26s2g6+ogyOF9RJdKzUrFKyHFwUPmRvLaqptaCNmuLQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737994963; c=relaxed/simple; bh=i8q4pp3vYJftdaixIrFKiNGEucKRjoM1RYEoTXF643Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qNQ4PuwvDDvUoH8aHyuMQ1CHVFzDbn4/YxajPMVd83mNi0foa80+2Xb9ExEdd2VvCCubpxbB0kw2QS7X2LNPVYbLmMDyxViUhczr38TdDt4PG2e5iNOJ6dqAFwqArj7NRks+U56OZnNftQEw8T+li9SgvTb3N9ixH4/BXpvk+94= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=IVOwsksS; arc=none smtp.client-ip=95.215.58.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="IVOwsksS" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1737994959; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AYx9gYzzX0cf0KN5Ej/iK3YYsmPfm49EYmVrld1ACsU=; b=IVOwsksS4FtXSN06W9zynt0OhmDqIwNZh6EK1ytv1v9gsk9h5JO2ltr368ymTHvPAKhlvm DE8l4FYkizFyXOCzgTuA6YPTXmUSNRAF1XyC+BbA2VgXeST+4h3ByIVPzNP9y1YlrN41LA oTzIC0eemxOZWKZHLtpcMzIAFKLVguk= From: Leon Hwang To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, yonghong.song@linux.dev, song@kernel.org, eddyz87@gmail.com, qmo@kernel.org, dxu@dxuuu.xyz, leon.hwang@linux.dev, kernel-patches-bot@fb.com Subject: [PATCH bpf-next 2/4] bpf, libbpf: Support global percpu data Date: Tue, 28 Jan 2025 00:21:56 +0800 Message-ID: <20250127162158.84906-3-leon.hwang@linux.dev> In-Reply-To: <20250127162158.84906-1-leon.hwang@linux.dev> References: <20250127162158.84906-1-leon.hwang@linux.dev> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Patchwork-Delegate: bpf@iogearbox.net This patch introduces support for global percpu data in libbpf. A new section named ".percpu" is added, similar to the existing ".data" section. Internal maps are created for ".percpu" sections, which are then initialized and populated accordingly. The changes include: * Introduction of the ".percpu" section in libbpf. * Creation of internal maps for percpu data. * Initialization and population of these maps. This enhancement allows BPF programs to efficiently manage and access percpu global data, improving performance for use cases that require percpu buffer. Signed-off-by: Leon Hwang --- tools/lib/bpf/libbpf.c | 172 ++++++++++++++++++++++++++++++++--------- 1 file changed, 135 insertions(+), 37 deletions(-) diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 194809da51725..6da6004c5c84d 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -516,6 +516,7 @@ struct bpf_struct_ops { }; #define DATA_SEC ".data" +#define PERCPU_DATA_SEC ".percpu" #define BSS_SEC ".bss" #define RODATA_SEC ".rodata" #define KCONFIG_SEC ".kconfig" @@ -530,6 +531,7 @@ enum libbpf_map_type { LIBBPF_MAP_BSS, LIBBPF_MAP_RODATA, LIBBPF_MAP_KCONFIG, + LIBBPF_MAP_PERCPU_DATA, }; struct bpf_map_def { @@ -562,6 +564,7 @@ struct bpf_map { __u32 btf_value_type_id; __u32 btf_vmlinux_value_type_id; enum libbpf_map_type libbpf_type; + void *data; void *mmaped; struct bpf_struct_ops *st_ops; struct bpf_map *inner_map; @@ -640,6 +643,7 @@ enum sec_type { SEC_DATA, SEC_RODATA, SEC_ST_OPS, + SEC_PERCPU_DATA, }; struct elf_sec_desc { @@ -1923,13 +1927,24 @@ static bool map_is_mmapable(struct bpf_object *obj, struct bpf_map *map) return false; } +static void map_copy_data(struct bpf_map *map, const void *data) +{ + bool is_percpu_data = map->libbpf_type == LIBBPF_MAP_PERCPU_DATA; + size_t data_sz = map->def.value_size; + + if (data) + memcpy(is_percpu_data ? map->data : map->mmaped, data, data_sz); +} + static int bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type, const char *real_name, int sec_idx, void *data, size_t data_sz) { + bool is_percpu_data = type == LIBBPF_MAP_PERCPU_DATA; struct bpf_map_def *def; struct bpf_map *map; size_t mmap_sz; + size_t elem_sz; int err; map = bpf_object__add_map(obj); @@ -1948,7 +1963,8 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type, } def = &map->def; - def->type = BPF_MAP_TYPE_ARRAY; + def->type = is_percpu_data ? BPF_MAP_TYPE_PERCPU_ARRAY + : BPF_MAP_TYPE_ARRAY; def->key_size = sizeof(int); def->value_size = data_sz; def->max_entries = 1; @@ -1958,29 +1974,53 @@ bpf_object__init_internal_map(struct bpf_object *obj, enum libbpf_map_type type, /* failures are fine because of maps like .rodata.str1.1 */ (void) map_fill_btf_type_info(obj, map); - if (map_is_mmapable(obj, map)) - def->map_flags |= BPF_F_MMAPABLE; + pr_debug("map '%s' (global %sdata): at sec_idx %d, offset %zu, flags %x.\n", + map->name, is_percpu_data ? "percpu " : "", map->sec_idx, + map->sec_offset, def->map_flags); - pr_debug("map '%s' (global data): at sec_idx %d, offset %zu, flags %x.\n", - map->name, map->sec_idx, map->sec_offset, def->map_flags); + if (is_percpu_data) { + elem_sz = roundup(data_sz, 8); - mmap_sz = bpf_map_mmap_sz(map); - map->mmaped = mmap(NULL, mmap_sz, PROT_READ | PROT_WRITE, - MAP_SHARED | MAP_ANONYMOUS, -1, 0); - if (map->mmaped == MAP_FAILED) { - err = -errno; - map->mmaped = NULL; - pr_warn("failed to alloc map '%s' content buffer: %s\n", map->name, errstr(err)); - zfree(&map->real_name); - zfree(&map->name); - return err; - } + map->data = malloc(elem_sz); + if (!map->data) { + err = -ENOMEM; + pr_warn("map '%s': failed to alloc content buffer: %s\n", + map->name, errstr(err)); + goto free_name; + } - if (data) - memcpy(map->mmaped, data, data_sz); + if (data) { + memcpy(map->data, data, data_sz); + if (data_sz != elem_sz) + memset(map->data + data_sz, 0, elem_sz - data_sz); + } else { + memset(map->data, 0, elem_sz); + } + } else { + if (map_is_mmapable(obj, map)) + def->map_flags |= BPF_F_MMAPABLE; + + mmap_sz = bpf_map_mmap_sz(map); + map->mmaped = mmap(NULL, mmap_sz, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_ANONYMOUS, -1, 0); + if (map->mmaped == MAP_FAILED) { + err = -errno; + map->mmaped = NULL; + pr_warn("map '%s': failed to alloc content buffer: %s\n", + map->name, errstr(err)); + goto free_name; + } + + map_copy_data(map, data); + } pr_debug("map %td is \"%s\"\n", map - obj->maps, map->name); return 0; + +free_name: + zfree(&map->real_name); + zfree(&map->name); + return err; } static int bpf_object__init_global_data_maps(struct bpf_object *obj) @@ -2015,6 +2055,13 @@ static int bpf_object__init_global_data_maps(struct bpf_object *obj) sec_desc->data->d_buf, sec_desc->data->d_size); break; + case SEC_PERCPU_DATA: + sec_name = elf_sec_name(obj, elf_sec_by_idx(obj, sec_idx)); + err = bpf_object__init_internal_map(obj, LIBBPF_MAP_PERCPU_DATA, + sec_name, sec_idx, + sec_desc->data->d_buf, + sec_desc->data->d_size); + break; case SEC_BSS: sec_name = elf_sec_name(obj, elf_sec_by_idx(obj, sec_idx)); err = bpf_object__init_internal_map(obj, LIBBPF_MAP_BSS, @@ -3934,6 +3981,11 @@ static int bpf_object__elf_collect(struct bpf_object *obj) sec_desc->sec_type = SEC_RODATA; sec_desc->shdr = sh; sec_desc->data = data; + } else if (strcmp(name, PERCPU_DATA_SEC) == 0 || + str_has_pfx(name, PERCPU_DATA_SEC)) { + sec_desc->sec_type = SEC_PERCPU_DATA; + sec_desc->shdr = sh; + sec_desc->data = data; } else if (strcmp(name, STRUCT_OPS_SEC) == 0 || strcmp(name, STRUCT_OPS_LINK_SEC) == 0 || strcmp(name, "?" STRUCT_OPS_SEC) == 0 || @@ -4453,6 +4505,7 @@ static bool bpf_object__shndx_is_data(const struct bpf_object *obj, case SEC_BSS: case SEC_DATA: case SEC_RODATA: + case SEC_PERCPU_DATA: return true; default: return false; @@ -4478,6 +4531,8 @@ bpf_object__section_to_libbpf_map_type(const struct bpf_object *obj, int shndx) return LIBBPF_MAP_DATA; case SEC_RODATA: return LIBBPF_MAP_RODATA; + case SEC_PERCPU_DATA: + return LIBBPF_MAP_PERCPU_DATA; default: return LIBBPF_MAP_UNSPEC; } @@ -4795,7 +4850,7 @@ static int map_fill_btf_type_info(struct bpf_object *obj, struct bpf_map *map) /* * LLVM annotates global data differently in BTF, that is, - * only as '.data', '.bss' or '.rodata'. + * only as '.data', '.bss', '.percpu' or '.rodata'. */ if (!bpf_map__is_internal(map)) return -ENOENT; @@ -5125,23 +5180,54 @@ static int bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map) { enum libbpf_map_type map_type = map->libbpf_type; - int err, zero = 0; + bool is_percpu_data = map_type == LIBBPF_MAP_PERCPU_DATA; + int err = 0, zero = 0; + void *data = NULL; + int num_cpus, i; + size_t data_sz; + size_t elem_sz; size_t mmap_sz; + data_sz = map->def.value_size; + if (is_percpu_data) { + num_cpus = libbpf_num_possible_cpus(); + if (num_cpus < 0) { + err = libbpf_err_errno(num_cpus); + pr_warn("map '%s': failed to get num_cpus: %s\n", + bpf_map__name(map), errstr(err)); + return err; + } + + elem_sz = roundup(data_sz, 8); + data_sz = elem_sz * num_cpus; + data = malloc(data_sz); + if (!data) { + err = -ENOMEM; + pr_warn("map '%s': failed to malloc memory: %s\n", + bpf_map__name(map), errstr(err)); + return err; + } + + for (i = 0; i < num_cpus; i++) + memcpy(data + i * elem_sz, map->data, elem_sz); + } else { + data = map->mmaped; + } + if (obj->gen_loader) { bpf_gen__map_update_elem(obj->gen_loader, map - obj->maps, - map->mmaped, map->def.value_size); + data, data_sz); if (map_type == LIBBPF_MAP_RODATA || map_type == LIBBPF_MAP_KCONFIG) bpf_gen__map_freeze(obj->gen_loader, map - obj->maps); - return 0; + goto free_data; } - err = bpf_map_update_elem(map->fd, &zero, map->mmaped, 0); + err = bpf_map_update_elem(map->fd, &zero, data, 0); if (err) { err = -errno; pr_warn("map '%s': failed to set initial contents: %s\n", bpf_map__name(map), errstr(err)); - return err; + goto free_data; } /* Freeze .rodata and .kconfig map as read-only from syscall side. */ @@ -5151,7 +5237,7 @@ bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map) err = -errno; pr_warn("map '%s': failed to freeze as read-only: %s\n", bpf_map__name(map), errstr(err)); - return err; + goto free_data; } } @@ -5178,7 +5264,7 @@ bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map) err = -errno; pr_warn("map '%s': failed to re-mmap() contents: %s\n", bpf_map__name(map), errstr(err)); - return err; + goto free_data; } map->mmaped = mmaped; } else if (map->mmaped) { @@ -5186,7 +5272,10 @@ bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map) map->mmaped = NULL; } - return 0; +free_data: + if (is_percpu_data) + free(data); + return err; } static void bpf_map__destroy(struct bpf_map *map); @@ -8120,7 +8209,9 @@ static int bpf_object__sanitize_maps(struct bpf_object *obj) struct bpf_map *m; bpf_object__for_each_map(m, obj) { - if (!bpf_map__is_internal(m)) + if (!bpf_map__is_internal(m) || + /* percpu data map is internal and not-mmapable. */ + m->libbpf_type == LIBBPF_MAP_PERCPU_DATA) continue; if (!kernel_supports(obj, FEAT_ARRAY_MMAP)) m->def.map_flags &= ~BPF_F_MMAPABLE; @@ -9041,6 +9132,8 @@ static void bpf_map__destroy(struct bpf_map *map) if (map->mmaped && map->mmaped != map->obj->arena_data) munmap(map->mmaped, bpf_map_mmap_sz(map)); map->mmaped = NULL; + if (map->data) + zfree(&map->data); if (map->st_ops) { zfree(&map->st_ops->data); @@ -10132,14 +10225,18 @@ int bpf_map__fd(const struct bpf_map *map) static bool map_uses_real_name(const struct bpf_map *map) { - /* Since libbpf started to support custom .data.* and .rodata.* maps, - * their user-visible name differs from kernel-visible name. Users see - * such map's corresponding ELF section name as a map name. - * This check distinguishes .data/.rodata from .data.* and .rodata.* - * maps to know which name has to be returned to the user. + /* Since libbpf started to support custom .data.*, .percpu.* and + * .rodata.* maps, their user-visible name differs from kernel-visible + * name. Users see such map's corresponding ELF section name as a map + * name. This check distinguishes .data/.percpu/.rodata from .data.*, + * .percpu.* and .rodata.* maps to know which name has to be returned to + * the user. */ if (map->libbpf_type == LIBBPF_MAP_DATA && strcmp(map->real_name, DATA_SEC) != 0) return true; + if (map->libbpf_type == LIBBPF_MAP_PERCPU_DATA && + strcmp(map->real_name, PERCPU_DATA_SEC) != 0) + return true; if (map->libbpf_type == LIBBPF_MAP_RODATA && strcmp(map->real_name, RODATA_SEC) != 0) return true; return false; @@ -10348,7 +10445,8 @@ int bpf_map__set_initial_value(struct bpf_map *map, if (map->obj->loaded || map->reused) return libbpf_err(-EBUSY); - if (!map->mmaped || map->libbpf_type == LIBBPF_MAP_KCONFIG) + if ((!map->mmaped && !map->data) || + map->libbpf_type == LIBBPF_MAP_KCONFIG) return libbpf_err(-EINVAL); if (map->def.type == BPF_MAP_TYPE_ARENA) @@ -10358,7 +10456,7 @@ int bpf_map__set_initial_value(struct bpf_map *map, if (size != actual_sz) return libbpf_err(-EINVAL); - memcpy(map->mmaped, data, size); + map_copy_data(map, data); return 0; } @@ -10370,7 +10468,7 @@ void *bpf_map__initial_value(const struct bpf_map *map, size_t *psize) return map->st_ops->data; } - if (!map->mmaped) + if (!map->mmaped && !map->data) return NULL; if (map->def.type == BPF_MAP_TYPE_ARENA) @@ -10378,7 +10476,7 @@ void *bpf_map__initial_value(const struct bpf_map *map, size_t *psize) else *psize = map->def.value_size; - return map->mmaped; + return map->libbpf_type == LIBBPF_MAP_PERCPU_DATA ? map->data : map->mmaped; } bool bpf_map__is_internal(const struct bpf_map *map) From patchwork Mon Jan 27 16:21:57 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Hwang X-Patchwork-Id: 13951609 X-Patchwork-Delegate: bpf@iogearbox.net Received: from out-181.mta1.migadu.com (out-181.mta1.migadu.com [95.215.58.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DDD9115573F for ; Mon, 27 Jan 2025 16:22:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737994968; cv=none; b=VOTIMNV+Qwnuq3AR63+7fzCsWvvGPxnDPANR0JpnSFMVZ/LyRIvwP+mFkT/O3YzNg2E91C8R2lwf+N/Wv6OufJ9dKXrpQyC5Jjw7oR1auVJqA+UO3cqg6AdQ8k0oCyxBG9jl4Zy3E2i0dTygso1nD4LvCntpdtqJ35P1G92NIMU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737994968; c=relaxed/simple; bh=f/cLWhg2L3G/CxGQlXAn2p5Hyiqy8nRAzbJSPsejB3o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oR4cwRdx+j0ZeOoCOpOK2CHAQAsl5ztkuq1jMjamFl1Oob1o/+IlmfOtBKqTZHUYNYHhJYiLb6fBBR9jWO1JMelaD14QwA7cRW2TpbxCcoF/HyeWDMci0W0e6KjQb/7TfYpRU5ZKjo21uK79aPWOeYCrgfK7ilRyULhWFyRQvfo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=HF6SHI1b; arc=none smtp.client-ip=95.215.58.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="HF6SHI1b" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1737994964; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HdimOJaKKRKEu4ZlEyRbQ+3cZhT739BXfiHVCtmbZsM=; b=HF6SHI1bQdbzTmuOygkKBITwF/N1HD62bokJLc9pxZClXfcolue1DcD+1Lt28+BB6ZHMgg 4hRQF3fJhqEszMeHJUBNNTaSE737OXVYxAIxvVWFWdc4yGHEISaxRBC8MSdgZehWQeQzmI uhK/8r6aElbT3QcGL2YEWutVexNAn/g= From: Leon Hwang To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, yonghong.song@linux.dev, song@kernel.org, eddyz87@gmail.com, qmo@kernel.org, dxu@dxuuu.xyz, leon.hwang@linux.dev, kernel-patches-bot@fb.com Subject: [PATCH bpf-next 3/4] bpf, bpftool: Generate skeleton for global percpu data Date: Tue, 28 Jan 2025 00:21:57 +0800 Message-ID: <20250127162158.84906-4-leon.hwang@linux.dev> In-Reply-To: <20250127162158.84906-1-leon.hwang@linux.dev> References: <20250127162158.84906-1-leon.hwang@linux.dev> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Patchwork-Delegate: bpf@iogearbox.net This patch enhances bpftool to generate skeletons for global percpu variables. The generated skeleton includes a dedicated structure for percpu data, allowing users to initialize and access percpu variables efficiently. Changes: 1. skeleton structure: For global percpu variables, the skeleton now includes a nested structure, e.g.: struct test_global_percpu_data { struct bpf_object_skeleton *skeleton; struct bpf_object *obj; struct { struct bpf_map *bss; struct bpf_map *percpu; } maps; // ... struct test_global_percpu_data__percpu { int percpu_data; } *percpu; // ... }; * The "struct test_global_percpu_data__percpu" points to initialized data, which is actually "maps.percpu->data". * Before loading the skeleton, updating the "struct test_global_percpu_data__percpu" modifies the initial value of the corresponding global percpu variables. * After loading the skeleton, accessing or updating this struct is no longer meaningful. Instead, users must interact with the global percpu variables via the "maps.percpu" map. 2. code changes: * Added support for ".percpu" sections in bpftool's map identification logic. * Modified skeleton generation to handle percpu data maps appropriately. * Updated libbpf to make "percpu" pointing to "maps.percpu->data". Signed-off-by: Leon Hwang --- tools/bpf/bpftool/gen.c | 13 +++++++++---- tools/lib/bpf/libbpf.c | 3 +++ tools/lib/bpf/libbpf.h | 1 + 3 files changed, 13 insertions(+), 4 deletions(-) diff --git a/tools/bpf/bpftool/gen.c b/tools/bpf/bpftool/gen.c index 5a4d3240689ed..975775683ca12 100644 --- a/tools/bpf/bpftool/gen.c +++ b/tools/bpf/bpftool/gen.c @@ -92,7 +92,7 @@ static void get_header_guard(char *guard, const char *obj_name, const char *suff static bool get_map_ident(const struct bpf_map *map, char *buf, size_t buf_sz) { - static const char *sfxs[] = { ".data", ".rodata", ".bss", ".kconfig" }; + static const char *sfxs[] = { ".data", ".rodata", ".percpu", ".bss", ".kconfig" }; const char *name = bpf_map__name(map); int i, n; @@ -117,7 +117,7 @@ static bool get_map_ident(const struct bpf_map *map, char *buf, size_t buf_sz) static bool get_datasec_ident(const char *sec_name, char *buf, size_t buf_sz) { - static const char *pfxs[] = { ".data", ".rodata", ".bss", ".kconfig" }; + static const char *pfxs[] = { ".data", ".rodata", ".percpu", ".bss", ".kconfig" }; int i, n; /* recognize hard coded LLVM section name */ @@ -263,7 +263,9 @@ static bool is_mmapable_map(const struct bpf_map *map, char *buf, size_t sz) return true; } - if (!bpf_map__is_internal(map) || !(bpf_map__map_flags(map) & BPF_F_MMAPABLE)) + if (!bpf_map__is_internal(map) || + (!(bpf_map__map_flags(map) & BPF_F_MMAPABLE) && + bpf_map__type(map) != BPF_MAP_TYPE_PERCPU_ARRAY)) return false; if (!get_map_ident(map, buf, sz)) @@ -903,7 +905,10 @@ codegen_maps_skeleton(struct bpf_object *obj, size_t map_cnt, bool mmaped, bool i, bpf_map__name(map), ident); /* memory-mapped internal maps */ if (mmaped && is_mmapable_map(map, ident, sizeof(ident))) { - printf("\tmap->mmaped = (void **)&obj->%s;\n", ident); + if (bpf_map__type(map) == BPF_MAP_TYPE_PERCPU_ARRAY) + printf("\tmap->data = (void **)&obj->%s;\n", ident); + else + printf("\tmap->mmaped = (void **)&obj->%s;\n", ident); } if (populate_links && bpf_map__type(map) == BPF_MAP_TYPE_STRUCT_OPS) { diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 6da6004c5c84d..dafb419bc5b86 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -13915,6 +13915,7 @@ static int populate_skeleton_maps(const struct bpf_object *obj, struct bpf_map **map = map_skel->map; const char *name = map_skel->name; void **mmaped = map_skel->mmaped; + void **data = map_skel->data; *map = bpf_object__find_map_by_name(obj, name); if (!*map) { @@ -13925,6 +13926,8 @@ static int populate_skeleton_maps(const struct bpf_object *obj, /* externs shouldn't be pre-setup from user code */ if (mmaped && (*map)->libbpf_type != LIBBPF_MAP_KCONFIG) *mmaped = (*map)->mmaped; + if (data && (*map)->libbpf_type == LIBBPF_MAP_PERCPU_DATA) + *data = (*map)->data; } return 0; } diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h index 3020ee45303a0..c49d6e44b5630 100644 --- a/tools/lib/bpf/libbpf.h +++ b/tools/lib/bpf/libbpf.h @@ -1701,6 +1701,7 @@ struct bpf_map_skeleton { const char *name; struct bpf_map **map; void **mmaped; + void **data; struct bpf_link **link; }; From patchwork Mon Jan 27 16:21:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Hwang X-Patchwork-Id: 13951610 X-Patchwork-Delegate: bpf@iogearbox.net Received: from out-173.mta1.migadu.com (out-173.mta1.migadu.com [95.215.58.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 060A415442D for ; Mon, 27 Jan 2025 16:22:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737994974; cv=none; b=TU+e6B4D8TWvIbQfMJyTPLXJFqGBg9GQUfG7MZ6b9Gaen7tk3iZWFKCP6SmqpRT1/ezEj+0/N1W4e1SgnRathTyfdzwlUO0AqosvqXg3pJoZAg+lQ/t6OC94zk6g5AZM57dUsOOv65zV0kcJnQyublFgd4huTofN1414LUc1fQk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737994974; c=relaxed/simple; bh=zXF+iIWfXeqpjqcalFixwr56iIRI0Gz3CXU1eGflHTQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kCl4EtEM/q6jv+86g0v/h8XrBW/0eicMF/RgZqWydOcdI35O/3GJM4HlKnAkBNV+CfBzuog4zN9lHUsZ7ulz1ZpdQnXpch1bWwzxbZzTsp8IIiu8kBQAKdT9V1/Y9LfT8UBJNXO4VvS1h5n0Tk2elBOZi+Hh/fLHw2NvCpueTY0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=Eyp/pzIK; arc=none smtp.client-ip=95.215.58.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="Eyp/pzIK" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1737994970; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1J1mQJZm9dOclTH351NApfaEMSuhjnTGy0CaWvBwMus=; b=Eyp/pzIK7hJXyZMAFDD0dRxuXyAb8qFSWsQZoIj3x+xFMK6wCmk55ZFshcw4v592RzwtQJ ijhfy6qW4cxqIAFeK+FsApIwssbzDTbfelUMNTn2liqcHFYa8V4BQbhHEfZOQsZFtTV4Id dYXwSqjypu7yBfFOxn3GwYyLy6L+VY4= From: Leon Hwang To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, yonghong.song@linux.dev, song@kernel.org, eddyz87@gmail.com, qmo@kernel.org, dxu@dxuuu.xyz, leon.hwang@linux.dev, kernel-patches-bot@fb.com Subject: [PATCH bpf-next 4/4] selftests/bpf: Add a case to test global percpu data Date: Tue, 28 Jan 2025 00:21:58 +0800 Message-ID: <20250127162158.84906-5-leon.hwang@linux.dev> In-Reply-To: <20250127162158.84906-1-leon.hwang@linux.dev> References: <20250127162158.84906-1-leon.hwang@linux.dev> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Patchwork-Delegate: bpf@iogearbox.net If the arch, like s390x, does not support percpu insn, this case won't test global percpu data by checking -EOPNOTSUPP when load prog. The following APIs have been tested for global percpu data: 1. bpf_map__set_initial_value() 2. bpf_map__initial_value() 3. generated percpu struct pointer that points to internal map's data 4. bpf_map__lookup_elem() for global percpu data map cd tools/testing/selftests/bpf; ./test_progs -t global_percpu_data 124 global_percpu_data_init:OK Summary: 1/0 PASSED, 0 SKIPPED, 0 FAILED Signed-off-by: Leon Hwang --- .../bpf/prog_tests/global_data_init.c | 89 ++++++++++++++++++- .../bpf/progs/test_global_percpu_data.c | 21 +++++ 2 files changed, 109 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/test_global_percpu_data.c diff --git a/tools/testing/selftests/bpf/prog_tests/global_data_init.c b/tools/testing/selftests/bpf/prog_tests/global_data_init.c index 8466332d7406f..a5d0890444f67 100644 --- a/tools/testing/selftests/bpf/prog_tests/global_data_init.c +++ b/tools/testing/selftests/bpf/prog_tests/global_data_init.c @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 #include +#include "test_global_percpu_data.skel.h" void test_global_data_init(void) { @@ -8,7 +9,7 @@ void test_global_data_init(void) __u8 *buff = NULL, *newval = NULL; struct bpf_object *obj; struct bpf_map *map; - __u32 duration = 0; + __u32 duration = 0; size_t sz; obj = bpf_object__open_file(file, NULL); @@ -60,3 +61,89 @@ void test_global_data_init(void) free(newval); bpf_object__close(obj); } + +void test_global_percpu_data_init(void) +{ + struct test_global_percpu_data *skel = NULL; + u64 *percpu_data = NULL; + struct bpf_map *map; + size_t init_data_sz; + char buff[128] = {}; + int init_value = 2; + int key, value_sz; + int prog_fd, err; + int *init_data; + int num_cpus; + + LIBBPF_OPTS(bpf_test_run_opts, topts, + .data_in = buff, + .data_size_in = sizeof(buff), + .repeat = 1, + ); + + num_cpus = libbpf_num_possible_cpus(); + if (!ASSERT_GT(num_cpus, 0, "libbpf_num_possible_cpus")) + return; + + percpu_data = calloc(num_cpus, sizeof(*percpu_data)); + if (!ASSERT_FALSE(percpu_data == NULL, "calloc percpu_data")) + return; + + value_sz = sizeof(*percpu_data) * num_cpus; + memset(percpu_data, 0, value_sz); + + skel = test_global_percpu_data__open(); + if (!ASSERT_OK_PTR(skel, "test_global_percpu_data__open")) + goto out; + + ASSERT_EQ(skel->percpu->percpu_data, -1, "skel->percpu->percpu_data"); + + map = skel->maps.percpu; + err = bpf_map__set_initial_value(map, &init_value, + sizeof(init_value)); + if (!ASSERT_OK(err, "bpf_map__set_initial_value")) + goto out; + + init_data = bpf_map__initial_value(map, &init_data_sz); + if (!ASSERT_OK_PTR(init_data, "bpf_map__initial_value")) + goto out; + + ASSERT_EQ(*init_data, init_value, "initial_value"); + ASSERT_EQ(init_data_sz, sizeof(init_value), "initial_value size"); + + if (!ASSERT_EQ((void *) init_data, (void *) skel->percpu, "skel->percpu")) + goto out; + ASSERT_EQ(skel->percpu->percpu_data, init_value, "skel->percpu->percpu_data"); + + err = test_global_percpu_data__load(skel); + if (err == -EOPNOTSUPP) + goto out; + if (!ASSERT_OK(err, "test_global_percpu_data__load")) + goto out; + + ASSERT_EQ(bpf_map__type(map), BPF_MAP_TYPE_PERCPU_ARRAY, + "bpf_map__type"); + + prog_fd = bpf_program__fd(skel->progs.update_percpu_data); + err = bpf_prog_test_run_opts(prog_fd, &topts); + ASSERT_OK(err, "update_percpu_data"); + ASSERT_EQ(topts.retval, 0, "update_percpu_data retval"); + + key = 0; + err = bpf_map__lookup_elem(map, &key, sizeof(key), percpu_data, + value_sz, 0); + if (!ASSERT_OK(err, "bpf_map__lookup_elem")) + goto out; + + if (!ASSERT_LT(skel->bss->curr_cpu, num_cpus, "curr_cpu")) + goto out; + ASSERT_EQ((int) percpu_data[skel->bss->curr_cpu], 1, "percpu_data"); + if (num_cpus > 1) + ASSERT_EQ((int) percpu_data[(skel->bss->curr_cpu+1)%num_cpus], + init_value, "init_value"); + +out: + test_global_percpu_data__destroy(skel); + if (percpu_data) + free(percpu_data); +} diff --git a/tools/testing/selftests/bpf/progs/test_global_percpu_data.c b/tools/testing/selftests/bpf/progs/test_global_percpu_data.c new file mode 100644 index 0000000000000..731c3214b0bb4 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_global_percpu_data.c @@ -0,0 +1,21 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright Leon Hwang */ + +#include +#include + +#include + +int percpu_data SEC(".percpu") = -1; +int curr_cpu; + +SEC("tc") +int update_percpu_data(struct __sk_buff *skb) +{ + curr_cpu = bpf_get_smp_processor_id(); + percpu_data = 1; + + return TC_ACT_OK; +} + +char _license[] SEC("license") = "GPL";