From patchwork Fri Mar 29 18:47:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13610929 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3D9361C0DC0 for ; Fri, 29 Mar 2024 18:47:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711738066; cv=none; b=PFyFEqyXqiwi0/FA+YCyeNPQGbIo6X+FGfTo6lchR3Dse4XTO7xg1Tn/99W8dtxlUSF6msRunFm9QMTHkhYE/rXdt14Ut7pgKp159Wb+o8seGCQdZV+8D+jppaOfb1zWnztHGN1YgWHwyqw6U7MEadNSSbHQHCWCmUVQkI5yBrE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711738066; c=relaxed/simple; bh=sqFVimtWpb/l/+AVDR8nH4xVlwtbkUFCwrJyPNbEutg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cDeokqjl7asjyJ+1gPmVWTpEvyKK2EJPTqm03fESdIZyX2xhBMZGX0j0AKJ9VOssC0SKTWHr8WU4jfAyZKqTjX9Zty0hPtYunqKuAn30pcTYovJ98Z9k/FG+MywB/EiKmZw6RyLlXdqJXgH7gYWYz4qXVsyjpeB0r2tkCRhvWKo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=IJRBAH40; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="IJRBAH40" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DA389C433F1; Fri, 29 Mar 2024 18:47:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1711738066; bh=sqFVimtWpb/l/+AVDR8nH4xVlwtbkUFCwrJyPNbEutg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IJRBAH40xkFzicUmTZTdESCNOp6kQWKRe2HCUvaglY5ysujeECen8eV+UHUQetuhl OL2HXzRU8jPohywG6x8ClOl4zd/CH4ui8Ts26FSUb+KXvLcmp6nrF2z+xJa1ZeQ8jl aB8K9uzyNwyfFoCCHDDP1St6+8VsUlEVeN3ABMdH1Vvzw14XRnglZe5dZE0cKwWRxa PVdULGcnfUF+kUmuu4Z9tlpfQi2pJ5L51Ivwrij7eLlzAY4KrUZ6/io1elC3eRkqxr p/4Lt9GFHo4/3DOtxc7VNZLllUnCqRpQxNOZY/bfOWiF52dKad37+7HdSCmbhQg22Y s9Nq4dfwLuBgw== From: Andrii Nakryiko To: bpf@vger.kernel.org, ast@kernel.org, daniel@iogearbox.net, martin.lau@kernel.org Cc: andrii@kernel.org, kernel-team@meta.com Subject: [PATCH bpf-next 1/4] bpf: add internal-only per-CPU LDX instructions Date: Fri, 29 Mar 2024 11:47:37 -0700 Message-ID: <20240329184740.4084786-2-andrii@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240329184740.4084786-1-andrii@kernel.org> References: <20240329184740.4084786-1-andrii@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Add BPF instructions for working with per-CPU data. These instructions are internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now. Two different instructions are added. One, with BPF_MEM_PERCPU opcode, performs memory dereferencing of a per-CPU "address" (which is actually an offset). This one is useful when inlined logic needs to load data stored in per-CPU storage (bpf_get_smp_processor_id() is one such example). Another, with BPF_ADDR_PERCPU opcode, performs a resolution of a per-CPU address (offset) stored in a register. This one is useful anywhere where per-CPU data is not read, but rather is returned to user as just absolute raw memory pointer (useful in bpf_map_lookup_elem() helper inlinings, for example). BPF disassembler is also taught to recognize them to support dumping final BPF assembly code (non-JIT'ed version). Add arch-specific way for BPF JITs to mark support for this instructions. This patch also adds support for these instructions in x86-64 BPF JIT. Signed-off-by: Andrii Nakryiko --- arch/x86/net/bpf_jit_comp.c | 29 +++++++++++++++++++++++++++++ include/linux/filter.h | 27 +++++++++++++++++++++++++++ kernel/bpf/core.c | 5 +++++ kernel/bpf/disasm.c | 33 ++++++++++++++++++++++++++------- 4 files changed, 87 insertions(+), 7 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 3b639d6f2f54..610bbedaae70 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1910,6 +1910,30 @@ st: if (is_imm8(insn->off)) } break; + /* internal-only per-cpu zero-extending memory load */ + case BPF_LDX | BPF_MEM_PERCPU | BPF_B: + case BPF_LDX | BPF_MEM_PERCPU | BPF_H: + case BPF_LDX | BPF_MEM_PERCPU | BPF_W: + case BPF_LDX | BPF_MEM_PERCPU | BPF_DW: + insn_off = insn->off; + EMIT1(0x65); /* gs segment modifier */ + emit_ldx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off); + break; + + /* internal-only load-effective-address-of per-cpu offset */ + case BPF_LDX | BPF_ADDR_PERCPU | BPF_DW: { + u32 off = (u32)(void *)&this_cpu_off; + + /* mov , (if necessary) */ + EMIT_mov(dst_reg, src_reg); + + /* add , gs:[] */ + EMIT2(0x65, add_1mod(0x48, dst_reg)); + EMIT3(0x03, add_1reg(0x04, dst_reg), 0x25); + EMIT(off, 4); + + break; + } case BPF_STX | BPF_ATOMIC | BPF_W: case BPF_STX | BPF_ATOMIC | BPF_DW: if (insn->imm == (BPF_AND | BPF_FETCH) || @@ -3365,6 +3389,11 @@ bool bpf_jit_supports_subprog_tailcalls(void) return true; } +bool bpf_jit_supports_percpu_insns(void) +{ + return true; +} + void bpf_jit_free(struct bpf_prog *prog) { if (prog->jited) { diff --git a/include/linux/filter.h b/include/linux/filter.h index 44934b968b57..85ffaa238bc1 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -75,6 +75,14 @@ struct ctl_table_header; /* unused opcode to mark special load instruction. Same as BPF_MSH */ #define BPF_PROBE_MEM32 0xa0 +/* unused opcode to mark special zero-extending per-cpu load instruction. */ +#define BPF_MEM_PERCPU 0xc0 + +/* unused opcode to mark special load-effective-address-of instruction for + * a given per-CPU offset + */ +#define BPF_ADDR_PERCPU 0xe0 + /* unused opcode to mark call to interpreter with arguments */ #define BPF_CALL_ARGS 0xe0 @@ -318,6 +326,24 @@ static inline bool insn_is_cast_user(const struct bpf_insn *insn) .off = OFF, \ .imm = 0 }) +/* Per-CPU zero-extending memory load (internal-only) */ +#define BPF_LDX_MEM_PERCPU(SIZE, DST, SRC, OFF) \ + ((struct bpf_insn) { \ + .code = BPF_LDX | BPF_SIZE(SIZE) | BPF_MEM_PERCPU,\ + .dst_reg = DST, \ + .src_reg = SRC, \ + .off = OFF, \ + .imm = 0 }) + +/* Load effective address of a given per-CPU offset */ +#define BPF_LDX_ADDR_PERCPU(DST, SRC, OFF) \ + ((struct bpf_insn) { \ + .code = BPF_LDX | BPF_DW | BPF_ADDR_PERCPU, \ + .dst_reg = DST, \ + .src_reg = SRC, \ + .off = OFF, \ + .imm = 0 }) + /* Memory store, *(uint *) (dst_reg + off16) = src_reg */ #define BPF_STX_MEM(SIZE, DST, SRC, OFF) \ @@ -970,6 +996,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog); void bpf_jit_compile(struct bpf_prog *prog); bool bpf_jit_needs_zext(void); bool bpf_jit_supports_subprog_tailcalls(void); +bool bpf_jit_supports_percpu_insns(void); bool bpf_jit_supports_kfunc_call(void); bool bpf_jit_supports_far_kfunc_call(void); bool bpf_jit_supports_exceptions(void); diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index ab400cdd7d7a..73f7183f3285 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -2945,6 +2945,11 @@ bool __weak bpf_jit_supports_subprog_tailcalls(void) return false; } +bool __weak bpf_jit_supports_percpu_insns(void) +{ + return false; +} + bool __weak bpf_jit_supports_kfunc_call(void) { return false; diff --git a/kernel/bpf/disasm.c b/kernel/bpf/disasm.c index bd2e2dd04740..37732ed4be3f 100644 --- a/kernel/bpf/disasm.c +++ b/kernel/bpf/disasm.c @@ -13,6 +13,13 @@ static const char * const func_id_str[] = { }; #undef __BPF_FUNC_STR_FN +#ifndef BPF_MEM_PERCPU +#define BPF_MEM_PERCPU 0xc0 +#endif +#ifndef BPF_ADDR_PERCPU +#define BPF_ADDR_PERCPU 0xe0 +#endif + static const char *__func_get_name(const struct bpf_insn_cbs *cbs, const struct bpf_insn *insn, char *buff, size_t len) @@ -178,6 +185,7 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs, { const bpf_insn_print_t verbose = cbs->cb_print; u8 class = BPF_CLASS(insn->code); + u8 mode = BPF_MODE(insn->code); if (class == BPF_ALU || class == BPF_ALU64) { if (BPF_OP(insn->code) == BPF_END) { @@ -269,16 +277,27 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs, verbose(cbs->private_data, "BUG_st_%02x\n", insn->code); } } else if (class == BPF_LDX) { - if (BPF_MODE(insn->code) != BPF_MEM && BPF_MODE(insn->code) != BPF_MEMSX) { + switch (BPF_MODE(insn->code)) { + case BPF_ADDR_PERCPU: + verbose(cbs->private_data, "(%02x) r%d = &(void __percpu *)(r%d %+d)\n", + insn->code, insn->dst_reg, + insn->src_reg, insn->off); + break; + case BPF_MEM: + case BPF_MEMSX: + case BPF_MEM_PERCPU: + verbose(cbs->private_data, "(%02x) r%d = *(%s%s *)(r%d %+d)\n", + insn->code, insn->dst_reg, + mode == BPF_MEM || mode == BPF_MEM_PERCPU ? + bpf_ldst_string[BPF_SIZE(insn->code) >> 3] : + bpf_ldsx_string[BPF_SIZE(insn->code) >> 3], + mode == BPF_MEM_PERCPU ? " __percpu" : "", + insn->src_reg, insn->off); + break; + default: verbose(cbs->private_data, "BUG_ldx_%02x\n", insn->code); return; } - verbose(cbs->private_data, "(%02x) r%d = *(%s *)(r%d %+d)\n", - insn->code, insn->dst_reg, - BPF_MODE(insn->code) == BPF_MEM ? - bpf_ldst_string[BPF_SIZE(insn->code) >> 3] : - bpf_ldsx_string[BPF_SIZE(insn->code) >> 3], - insn->src_reg, insn->off); } else if (class == BPF_LD) { if (BPF_MODE(insn->code) == BPF_ABS) { verbose(cbs->private_data, "(%02x) r0 = *(%s *)skb[%d]\n", From patchwork Fri Mar 29 18:47:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13610930 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9904B225A8 for ; Fri, 29 Mar 2024 18:47:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711738069; cv=none; b=tlMaWmagQsbGvhtOYcvtUNKJ4rygQuAjeJubPJjdEkeEkQTbNz4oYpNDlLG6lt0T2Gj6SODsk66N5TjEiEh5EcXsxCoC7akYcc2UsQAF3qlzdYOVDxmAoJOyIh3FWiB32fLS6l/O8CDcpfVOLOY+YnM2HujMCGe8r/9/k1Z5z+o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711738069; c=relaxed/simple; bh=bKWtYC2Zn55ptgsWc3tn3d/Y9QuPa0RDYZ/pP2P65kQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IEoGJG1j5CZgnXZFK0Ev3xsDg04EIOdYf294kKkcOR33P1Wy/5wn6caaf7lwlIL/rvsy6RdNvzNqCFdEd7BNeSd16B0csdoONik3Rlx16hA+k9iLidNS6eg2K9HylkXa+oYU+3c2bNT0rIFeJsQak6T+afp39O8ml2vsH5KrfJ0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=gBROYdZu; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="gBROYdZu" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 19FE5C433C7; Fri, 29 Mar 2024 18:47:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1711738069; bh=bKWtYC2Zn55ptgsWc3tn3d/Y9QuPa0RDYZ/pP2P65kQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gBROYdZuyZ5vJgkKX6foaJ6DlkpXwwmBGyNfm/NnShTJu3WHVqj9uVbWz3f2MsAgJ pLpTzcobK/DtTEV1eVIo5haaplNDl7J1MBrHEhutb9Zx84SzoTjCadGrF/EeWOz5dn jxaKJZfCSs7Xb6VOPk5pIlfQ76Bajtb9IoWAelLr8LWN+DCd47OFRPb9j8MzR6VEyW bTJ+iilJgTTBoUvAT67qEtePEzX5B18LQZpI8HIcjiAw8Vak+jTc0hiU3t+qApw6a2 Abq8k9TYeQWKsEPZi0fdWhGEu3RyNlDCaF502FaCDwF1fG0Y9uOeANu2yj5QCLIgr9 wt10+LuyF0IQw== From: Andrii Nakryiko To: bpf@vger.kernel.org, ast@kernel.org, daniel@iogearbox.net, martin.lau@kernel.org Cc: andrii@kernel.org, kernel-team@meta.com Subject: [PATCH bpf-next 2/4] bpf: inline bpf_get_smp_processor_id() helper Date: Fri, 29 Mar 2024 11:47:38 -0700 Message-ID: <20240329184740.4084786-3-andrii@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240329184740.4084786-1-andrii@kernel.org> References: <20240329184740.4084786-1-andrii@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net If BPF JIT supports per-CPU LDX instructions, inline bpf_get_smp_processor_id() to eliminate unnecessary function calls. Signed-off-by: Andrii Nakryiko --- kernel/bpf/verifier.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index edb650667f44..24caec8b200d 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -20072,6 +20072,23 @@ static int do_misc_fixups(struct bpf_verifier_env *env) goto next_insn; } + /* Implement bpf_get_smp_processor_id() inline. */ + if (insn->imm == BPF_FUNC_get_smp_processor_id && + prog->jit_requested && bpf_jit_supports_percpu_insns()) { + insn_buf[0] = BPF_MOV32_IMM(BPF_REG_0, (u32)(long)&pcpu_hot.cpu_number); + insn_buf[1] = BPF_LDX_MEM_PERCPU(BPF_W, BPF_REG_0, BPF_REG_0, 0); + cnt = 2; + + new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt); + if (!new_prog) + return -ENOMEM; + + delta += cnt - 1; + env->prog = prog = new_prog; + insn = new_prog->insnsi + i + delta; + goto next_insn; + } + /* Implement bpf_get_func_arg inline. */ if (prog_type == BPF_PROG_TYPE_TRACING && insn->imm == BPF_FUNC_get_func_arg) { From patchwork Fri Mar 29 18:47:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13610931 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC6463B1AC for ; Fri, 29 Mar 2024 18:47:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711738072; cv=none; b=mIoReL4graQmkluAN2u1vXbJOrLmSMdiUz84QosO6RcYO5j+9e72+LlkiRBEAtLMa+6By+1et0ZN8abWV3Pk687UiETf7wJJNVV5TVY8uQfKdYrYkeSaC9NbyVHZTZfaMbtDDk5/peYKvCTq3G/SLD0eHOtqIlSRIIzDH607jOg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711738072; c=relaxed/simple; bh=/WLgrEbfVw01tVbYMD6LgpIF8WYcDH3vzzknKD8nRWk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lv5UzmZj7YmvklNi0tTsr32dyCrw7Eo7Po42nN/6UBcabLTJCB3y/VaG0GFbyLa2xGC/D2Un9HTQcOqdakdUJkxOPaIW8fewOMmLS8rY44x7Kr93GIKNtjuNMEkFBHrEjQ7luo014JnoNQcibJJYET0suNYX0nCIu3vRVlmop3c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Q2bG/juA; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Q2bG/juA" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 38641C433C7; Fri, 29 Mar 2024 18:47:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1711738072; bh=/WLgrEbfVw01tVbYMD6LgpIF8WYcDH3vzzknKD8nRWk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Q2bG/juAB5EAY69bTukQ233H6Q/YDAap3RFfKvSajyg3A0/nd2Pd93m3sHQQWNbzq kk2Yww5ZlM4tE+CYRQocSwLMQrGVeeXkdUh5k3pjFJ4KkxZhuP0pR7MDSuw62fWrvT qcWOq0uhIK8SnJj418r0/lz2Hmc7z7k8UEqKnwthmvW1KMF1D4Tm4/BsqJVYUSPjkl F6SnWGE3haByZclEmLiq3j2ZenNklv/USN4A+wRFwgqHzvNhDaL/PkAiIrB8QK9DMt VyBX/7/lhpFk7eyta7IJ3Lycs+nBAE3Z7XCkB4B1X9dkNjybqPoTfEhc++vhJfHHfk raQiCwxaGi36Q== From: Andrii Nakryiko To: bpf@vger.kernel.org, ast@kernel.org, daniel@iogearbox.net, martin.lau@kernel.org Cc: andrii@kernel.org, kernel-team@meta.com Subject: [PATCH bpf-next 3/4] bpf: inline bpf_map_lookup_elem() for PERCPU_ARRAY maps Date: Fri, 29 Mar 2024 11:47:39 -0700 Message-ID: <20240329184740.4084786-4-andrii@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240329184740.4084786-1-andrii@kernel.org> References: <20240329184740.4084786-1-andrii@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Using new per-CPU BPF instructions implement inlining for per-CPU ARRAY map lookup helper, if BPF JIT support is present. Signed-off-by: Andrii Nakryiko --- kernel/bpf/arraymap.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index 13358675ff2e..557661b96cf2 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -246,6 +246,38 @@ static void *percpu_array_map_lookup_elem(struct bpf_map *map, void *key) return this_cpu_ptr(array->pptrs[index & array->index_mask]); } +/* emit BPF instructions equivalent to C code of percpu_array_map_lookup_elem() */ +static int percpu_array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf) +{ + struct bpf_array *array = container_of(map, struct bpf_array, map); + struct bpf_insn *insn = insn_buf; + + if (!bpf_jit_supports_percpu_insns()) + return -EOPNOTSUPP; + + if (map->map_flags & BPF_F_INNER_MAP) + return -EOPNOTSUPP; + + BUILD_BUG_ON(offsetof(struct bpf_array, map) != 0); + *insn++ = BPF_ALU64_IMM(BPF_ADD, BPF_REG_1, offsetof(struct bpf_array, pptrs)); + + *insn++ = BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_2, 0); + if (!map->bypass_spec_v1) { + *insn++ = BPF_JMP_IMM(BPF_JGE, BPF_REG_0, map->max_entries, 6); + *insn++ = BPF_ALU32_IMM(BPF_AND, BPF_REG_0, array->index_mask); + } else { + *insn++ = BPF_JMP_IMM(BPF_JGE, BPF_REG_0, map->max_entries, 5); + } + + *insn++ = BPF_ALU64_IMM(BPF_LSH, BPF_REG_0, 3); + *insn++ = BPF_ALU64_REG(BPF_ADD, BPF_REG_0, BPF_REG_1); + *insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0); + *insn++ = BPF_LDX_ADDR_PERCPU(BPF_REG_0, BPF_REG_0, 0); + *insn++ = BPF_JMP_IMM(BPF_JA, 0, 0, 1); + *insn++ = BPF_MOV64_IMM(BPF_REG_0, 0); + return insn - insn_buf; +} + static void *percpu_array_map_lookup_percpu_elem(struct bpf_map *map, void *key, u32 cpu) { struct bpf_array *array = container_of(map, struct bpf_array, map); @@ -776,6 +808,7 @@ const struct bpf_map_ops percpu_array_map_ops = { .map_free = array_map_free, .map_get_next_key = array_map_get_next_key, .map_lookup_elem = percpu_array_map_lookup_elem, + .map_gen_lookup = percpu_array_map_gen_lookup, .map_update_elem = array_map_update_elem, .map_delete_elem = array_map_delete_elem, .map_lookup_percpu_elem = percpu_array_map_lookup_percpu_elem, From patchwork Fri Mar 29 18:47:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrii Nakryiko X-Patchwork-Id: 13610932 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B04E23B2A6 for ; Fri, 29 Mar 2024 18:47:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711738075; cv=none; b=ntEqAbfTOz/6Uk++ja4T9qnmefRfH5jSN3A5G/kZOmzqHbPv8HWihwd8n4icHTBTG5jsbXUoMNYVqBkrMOl4A+r+cNozWlzy4QBo5SitHcdPsnZTkIE9258NgmZ4o7/UfX6tPU9ZjfBbXeqcCA5YoLoT6aRBYEDmxzo0ok47Ums= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711738075; c=relaxed/simple; bh=mEz9VApgbx0FD0OunFg6ZVIEXnoTzB7glx9OXzO+gGY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=j3/vkTRJmsJ5TyK4Kl7fEfNviZwPPy8Un70edsntcWmJvx1GEHSgTAOHT2vaflEJWiph7UCEUS2rCvx5OvWHuKicDhAaRGMzclx19KAnMfT0JOdBWtmm59qtkN6Aw4HmJy2I+k+lT3Nnys5/avLi5FyZKv5NuO5y9qlNQ0+tnlA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=IhgaQDlS; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="IhgaQDlS" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6E124C433C7; Fri, 29 Mar 2024 18:47:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1711738075; bh=mEz9VApgbx0FD0OunFg6ZVIEXnoTzB7glx9OXzO+gGY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IhgaQDlS4VyX36xz9ebrrkKizWnzctbqo0jMmP5/H0bAEKIjjWLGtFh/xwFsQFd5o yUH11WsZRQWMDcYzFJWc2sBE+F1qJhA3efLak8ktRc8oyJdMfvxjntwqLuWLK6v5kf WJ7KetnfXRWX+DsjqVI7HkPs0z0POR5SE4uWxTNr0d4QikxOeoR15+E1DrG029rnB4 G8FvFAe4gcGJaox5i+pdxwM7nYw49SAspsiHwlw7mYGa3iLQVS/RAPPsh+7yCnqBMk ZBjPHi5b7/YfR0n0Eg7DuioS6I+T64uVnzZ/9mbujRbvQ2Ql/AA8afj6fTrq/nVH+l IfSGCSzqDl41Q== From: Andrii Nakryiko To: bpf@vger.kernel.org, ast@kernel.org, daniel@iogearbox.net, martin.lau@kernel.org Cc: andrii@kernel.org, kernel-team@meta.com Subject: [PATCH bpf-next 4/4] bpf: inline bpf_map_lookup_elem() helper for PERCPU_HASH map Date: Fri, 29 Mar 2024 11:47:40 -0700 Message-ID: <20240329184740.4084786-5-andrii@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240329184740.4084786-1-andrii@kernel.org> References: <20240329184740.4084786-1-andrii@kernel.org> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Using new per-CPU BPF instruction, partially inline bpf_map_lookup_elem() helper for per-CPU hashmap BPF map. Just like for normal HASH map, we still generate a call into __htab_map_lookup_elem(), but after that we resolve per-CPU element address using a new instruction, saving on extra functions calls. Signed-off-by: Andrii Nakryiko --- kernel/bpf/hashtab.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index e81059faae63..74950f373bab 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -2308,6 +2308,26 @@ static void *htab_percpu_map_lookup_elem(struct bpf_map *map, void *key) return NULL; } +/* inline bpf_map_lookup_elem() call for per-CPU hashmap */ +static int htab_percpu_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf) +{ + struct bpf_insn *insn = insn_buf; + + if (!bpf_jit_supports_percpu_insns()) + return -EOPNOTSUPP; + + BUILD_BUG_ON(!__same_type(&__htab_map_lookup_elem, + (void *(*)(struct bpf_map *map, void *key))NULL)); + *insn++ = BPF_EMIT_CALL(__htab_map_lookup_elem); + *insn++ = BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 3); + *insn++ = BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, + offsetof(struct htab_elem, key) + map->key_size); + *insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0); + *insn++ = BPF_LDX_ADDR_PERCPU(BPF_REG_0, BPF_REG_0, 0); + + return insn - insn_buf; +} + static void *htab_percpu_map_lookup_percpu_elem(struct bpf_map *map, void *key, u32 cpu) { struct htab_elem *l; @@ -2436,6 +2456,7 @@ const struct bpf_map_ops htab_percpu_map_ops = { .map_free = htab_map_free, .map_get_next_key = htab_map_get_next_key, .map_lookup_elem = htab_percpu_map_lookup_elem, + .map_gen_lookup = htab_percpu_map_gen_lookup, .map_lookup_and_delete_elem = htab_percpu_map_lookup_and_delete_elem, .map_update_elem = htab_percpu_map_update_elem, .map_delete_elem = htab_map_delete_elem,