From patchwork Tue Sep 12 23:31:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13382306 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2E2D117C2 for ; Tue, 12 Sep 2023 23:32:18 +0000 (UTC) Received: from mail-wm1-x342.google.com (mail-wm1-x342.google.com [IPv6:2a00:1450:4864:20::342]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 577AA1702 for ; Tue, 12 Sep 2023 16:32:18 -0700 (PDT) Received: by mail-wm1-x342.google.com with SMTP id 5b1f17b1804b1-402cc6b8bedso70543195e9.1 for ; Tue, 12 Sep 2023 16:32:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1694561536; x=1695166336; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vPZKosC1Aaqq1ZcoJJGdd++MUM7bZOkhdXp+eqhi/Ms=; b=XUC7dwoo2hlKTSxmbP0nD7jSAALeYDKmiD+S6e34SA2LM63FClO/MSxuhYss7CME5i 80XXK7fwbeUnQAciyrrSkpzxZpSTB/0EfJtuNZW+OpjJK0iZb3p+QOraVdJnJQais/hO DbGKLhJBjobca1jv9QPHpbpPETGD1twZUN8OHj+qBW9HCAByEUakgknpM5NTCsVRGNit rqu9RJChxOGXHLrtX4jj7dpgMC2t2CZZHH8a98heRtdWwYEBay/jSdNT7dJZ8/Fx3ecu IiMOP102rDQuzYzdXQnZAqHZTIXqfne5hSHmxDVd8piqalGPLDozz3s/96+V2Htil1fb v80w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694561536; x=1695166336; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vPZKosC1Aaqq1ZcoJJGdd++MUM7bZOkhdXp+eqhi/Ms=; b=HLEs8+pI62WPCJgMVzaNBTZHfG+AOCPV5qDIDpvhO5CoI4i9QQPuE3Tm2JliXjE5Gc b6x1v2mx+Vfcdp0xfc0Sbj23+G9o5x2N34+jiFNsGBQvFAM7S4aaPs2ENkWtAI+Iz0+o KlN2u3RpLLqP497MTRljTSGyKEpTXMe+ewlWZmRk6kl/u5RgRG/IsIB/wP+OnwFFAOe3 aGtboZ58k6zjl82LXzptwXdwDGYOXokMejoAxc2mjNMOSpCnQvo3WbtLCoiQY0jrwZ+T TxpBNKI6ciowvOCdPfQNvUv8oa4vwjk0JNgMYG5pfqfk5EwTGKNuIqCuzyvYUFCQ8FNP u7gQ== X-Gm-Message-State: AOJu0Ywznbdj7OW+Y/I0V3Sb8t7gWyVa6RwISCw0kQy1aSeLJi1YPYid +TZmHzbs1J3uylE2KdAaHbeR+GKl2IFo4Q== X-Google-Smtp-Source: AGHT+IHc2/AQMlrd34SFVs4/2wo36w+taUmrlBQh83Bf4I6VuhBwuQTTJCHlKC139iQpdjUZhIpi+g== X-Received: by 2002:adf:ce10:0:b0:31f:98b4:4b62 with SMTP id p16-20020adfce10000000b0031f98b44b62mr694546wrn.37.1694561536608; Tue, 12 Sep 2023 16:32:16 -0700 (PDT) Received: from localhost ([212.203.98.42]) by smtp.gmail.com with ESMTPSA id q14-20020aa7cc0e000000b00521f4ee396fsm6441032edt.12.2023.09.12.16.32.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 16:32:16 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet , Puranjay Mohan Subject: [PATCH bpf-next v3 01/17] bpf: Use bpf_is_subprog to check for subprogs Date: Wed, 13 Sep 2023 01:31:58 +0200 Message-ID: <20230912233214.1518551-2-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230912233214.1518551-1-memxor@gmail.com> References: <20230912233214.1518551-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2998; i=memxor@gmail.com; h=from:subject; bh=W8BM/AO+gni3m/B/jvBaUMVNoZ0nNjaba66GzNpnA7s=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBlAPSsfnbQMNH/+vH2tFSbHwDgFBVLZjA6dauyO ZKAKEHNVV+JAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZQD0rAAKCRBM4MiGSL8R ylnMEAC92ZqvQ3lBmXiYlrfG8mKbak0p2L9ba9QCMNJi1oJojSc/4Unijjw3Ue1qyTH6MvL4khj 46NAWwDhmI1muB2ABcNF27aZKCWPrxZB8qPUXouUz1b5I6sY0eVfWFfz07iDuWu7ZHLpM1XqeJL wwpCen/zIlKOr2WdF6ybDxuGvPFUiNxmDniSxCRp1ftHOrmwQOJZ05NyQof6IKOE5iWN2+hZXtH PGzSHzOPXmsHURJScSNIwsKb07/Rp4RPYrsoZ+gBQLrS7cicshqa+8jUCZwWhlS2eU3SNycN1Vw m5+NRlRnA8JF1Nlc7HvQp0hpk0QlYVzjNBcfimFO4Dcw0g8fzSWC2H7n9Mbjh+O5xPxX6CusYXL XtDQbS4+SEQyVZMAjwbWkR4p9jCEFJ4PgKWPyiBENYAFewaUCGikQ0BdLfpW9OyfCLhRxk2BZP7 /KTtqrTwp8jrurlzvdmIv2D5p5LZI7SggZ9XRMlYsQOViDtw3SINbc1ryEBUD4p6gd7gS5YTlvK Xo9HsRjLpF1cakyVa1dikD8hjN9fgZcR4Kaqzzgm1PO+I4U/CTOisUoWCd/f4jeHJj/VEzb0UwJ sBM6yNl4hcfbQf8K5IjKVzfop3hi+SPpWFAyX1GvzF7pBBe4tXc8V4a1hYJ2gwQHA2B1FYgAlHE E076KilcQxbK/PQ== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net We would like to know whether a bpf_prog corresponds to the main prog or one of the subprogs. The current JIT implementations simply check this using the func_idx in bpf_prog->aux->func_idx. When the index is 0, it belongs to the main program, otherwise it corresponds to some subprogram. This will also be necessary to halt exception propagation while walking the stack when an exception is thrown, so we add a simple helper function to check this, named bpf_is_subprog, and convert existing JIT implementations to also make use of it. Signed-off-by: Kumar Kartikeya Dwivedi --- arch/arm64/net/bpf_jit_comp.c | 2 +- arch/s390/net/bpf_jit_comp.c | 2 +- arch/x86/net/bpf_jit_comp.c | 2 +- include/linux/bpf.h | 5 +++++ 4 files changed, 8 insertions(+), 3 deletions(-) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 150d1c6543f7..7d4af64e3982 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -288,7 +288,7 @@ static bool is_lsi_offset(int offset, int scale) static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) { const struct bpf_prog *prog = ctx->prog; - const bool is_main_prog = prog->aux->func_idx == 0; + const bool is_main_prog = !bpf_is_subprog(prog); const u8 r6 = bpf2a64[BPF_REG_6]; const u8 r7 = bpf2a64[BPF_REG_7]; const u8 r8 = bpf2a64[BPF_REG_8]; diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c index de2fb12120d2..eeb42e5cd7d6 100644 --- a/arch/s390/net/bpf_jit_comp.c +++ b/arch/s390/net/bpf_jit_comp.c @@ -556,7 +556,7 @@ static void bpf_jit_prologue(struct bpf_jit *jit, struct bpf_prog *fp, EMIT6_PCREL_RILC(0xc0040000, 0, jit->prologue_plt); jit->prologue_plt_ret = jit->prg; - if (fp->aux->func_idx == 0) { + if (!bpf_is_subprog(fp)) { /* Initialize the tail call counter in the main program. */ /* xc STK_OFF_TCCNT(4,%r15),STK_OFF_TCCNT(%r15) */ _EMIT6(0xd703f000 | STK_OFF_TCCNT, 0xf000 | STK_OFF_TCCNT); diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 2846c21d75bf..a0d03503b3cb 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1049,7 +1049,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image emit_prologue(&prog, bpf_prog->aux->stack_depth, bpf_prog_was_classic(bpf_prog), tail_call_reachable, - bpf_prog->aux->func_idx != 0); + bpf_is_subprog(bpf_prog)); push_callee_regs(&prog, callee_regs_used); ilen = prog - temp; diff --git a/include/linux/bpf.h b/include/linux/bpf.h index b9e573159432..9171b0b6a590 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -3194,4 +3194,9 @@ static inline gfp_t bpf_memcg_flags(gfp_t flags) return flags; } +static inline bool bpf_is_subprog(const struct bpf_prog *prog) +{ + return prog->aux->func_idx != 0; +} + #endif /* _LINUX_BPF_H */ From patchwork Tue Sep 12 23:31:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13382307 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8561B17C2 for ; Tue, 12 Sep 2023 23:32:20 +0000 (UTC) Received: from mail-lf1-x141.google.com (mail-lf1-x141.google.com [IPv6:2a00:1450:4864:20::141]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C9D8110FE for ; Tue, 12 Sep 2023 16:32:19 -0700 (PDT) Received: by mail-lf1-x141.google.com with SMTP id 2adb3069b0e04-50078e52537so10500709e87.1 for ; Tue, 12 Sep 2023 16:32:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1694561538; x=1695166338; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nQyCKvejR05FLvij5SV3MltntkSCOrFj//tSa1rbDlg=; b=T+1C/U4aC5a2DAeYtkTGZqY9j2Vpj5P8dxOaLsa/dj+IW/vIFRKi2JIg8vmcIalaHk Tezw8RA6n+uc7f2Usm8k2BDQaKFpuQKtfuoREfVQ8WvDFdY1UthMALiPWSmRujrIlr2J eh5VWYWgh32aHLpQf6kzcTn8XYlihglwsDqYVBbUNmVtYED1Wiel5Z19Vy+kA8tqLlDW dUCm/QKaa7cSEpUEA9Ikarc3B2wVCgEMJtxskSt7i9GbwPekKZVBk0wfdEBSPd5pWCV/ 9e5GgTgEiKA+ePDMLCU7In8y5tnf0Pa72vgoepUS531yICn9AKqxsgZTSjgc4suLhljH oMWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694561538; x=1695166338; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nQyCKvejR05FLvij5SV3MltntkSCOrFj//tSa1rbDlg=; b=j4QKA1CVu+DCG8q+VR2g6TgDCVm5zvSknfyGrLyceVZR8fTrEzzOscIuUyu/h1vgr7 xrlcAEpS0CImjoNDyQ+QY3XnHHReySFcjuYU/NG6fSeg+ULUuSzKn2ULWPUh2j8eUQKP yoUp0JPin1piovs0Fs2UPJEDvYi3dIVf+dIXadmlmF0IXnTf3XxYa9SAMUtXa8IeaEF/ BjbDXcC3waNG2jiacQuHB6gGhge/fds+Ecj1hrnyKCOL2NQdFQeTMOM58uirOnO8kUU7 aGFA8jl0o81lzomtZphHWhVEjmK53L1uXEZe+5ZqncP/bgFwg4qLn9SDQJgh41AgPIwe SaKw== X-Gm-Message-State: AOJu0Yxhzc9+2rz3PEocrHq/2X51EQ26eLrss8Hlr7NwXqghMMNW1UYI l+C0QiDoVw4+8MaRiBeNmNt8G7+XG2Fn3g== X-Google-Smtp-Source: AGHT+IGNHVA1tb1cN2M7//o9tjAvSHCrlI27ttuh+KuGqzY1Qw9dzoC+euDpX2UoA++9K5MfpGinuw== X-Received: by 2002:a19:2d52:0:b0:4fd:d213:dfd4 with SMTP id t18-20020a192d52000000b004fdd213dfd4mr650167lft.20.1694561537503; Tue, 12 Sep 2023 16:32:17 -0700 (PDT) Received: from localhost ([212.203.98.42]) by smtp.gmail.com with ESMTPSA id f20-20020a170906825400b0099bc2d1429csm7460356ejx.72.2023.09.12.16.32.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 16:32:17 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet , Puranjay Mohan Subject: [PATCH bpf-next v3 02/17] arch/x86: Implement arch_bpf_stack_walk Date: Wed, 13 Sep 2023 01:31:59 +0200 Message-ID: <20230912233214.1518551-3-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230912233214.1518551-1-memxor@gmail.com> References: <20230912233214.1518551-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4040; i=memxor@gmail.com; h=from:subject; bh=sgqrFU8/IE1hJC02IhP+Kq/56a74YibDEH8nsdcFXWw=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBlAPSsX9VVxxLHLIMs4BgGQ8qbLOSOKDyPbyywJ d2bm7Zw9VmJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZQD0rAAKCRBM4MiGSL8R yk9sD/9tJQFU9+cn3Jdat0Jz9oQa7oy8Vym4WdieonMdqPoTsmuBXOXiYIA8eFxJYRtc+rCIrDa 0iZNErqZSMVJd3S6YrZ/THVgiuXl46htKIlBtlYuqEOFvHZoDFwYuL6+BIIdVnzQXzruxrsLKXp H8pmLuLrpVvu7KiILkDKwk4D+OLjxXDe8mWtBWi6plkpJWAoq2+9NxDZMqvG3cIZrZ6sT+Sjsg7 YIe+dPqIiuVtJrJTEhPVe5sjwwp2Z+T2Gm0Htm7C2DwxXOOsJpexyNG5t2bVaNdWnSm9LEFpN/J hqpTW9vfttlW7vMy8eEoVdFOIaeVAoqS4F3QMtwW2YIbgMnKng/AG83WNAB0hJXPvSvMnM/Lo+L D+LK5zdUwnN0S+UATx0iV+OC+Lml1xonnc4Dbmic2zsSfNuVzGQM5xHol3uKXP4UhyiDY4ybRuf xU6ycpG3rJxrHht7aEaKW++JwZX89CmA3qxjFXDrVr8wcHeuvZxIfjiKXIf5ZY6gGUkcAmd/6SA NB77Dabctx5i1pYCVSw5fW6QIvdOTgyZwTv5vpSd4rHsmQPABumU042Mf7gdoqAfN00NJF12jcY dBSytTHkTh97zExEO00b/EuTjfG9hpT7u3KaiejR+psoNXhQtXblEneAux3uBa+mXPdge+jFQXX QJ6C8GFHshSdNTw== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net The plumbing for offline unwinding when we throw an exception in programs would require walking the stack, hence introduce a new arch_bpf_stack_walk function. This is provided when the JIT supports exceptions, i.e. bpf_jit_supports_exceptions is true. The arch-specific code is really minimal, hence it should be straightforward to extend this support to other architectures as well, as it reuses the logic of arch_stack_walk, but allowing access to unwind_state data. Once the stack pointer and frame pointer are known for the main subprog during the unwinding, we know the stack layout and location of any callee-saved registers which must be restored before we return back to the kernel. This handling will be added in the subsequent patches. Note that while we primarily unwind through BPF frames, which are effectively CONFIG_UNWINDER_FRAME_POINTER, we still need one of this or CONFIG_UNWINDER_ORC to be able to unwind through the bpf_throw frame from which we begin walking the stack. We also require both sp and bp (stack and frame pointers) from the unwind_state structure, which are only available when one of these two options are enabled. Signed-off-by: Kumar Kartikeya Dwivedi --- arch/x86/net/bpf_jit_comp.c | 28 ++++++++++++++++++++++++++++ include/linux/filter.h | 2 ++ kernel/bpf/core.c | 9 +++++++++ 3 files changed, 39 insertions(+) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index a0d03503b3cb..d0c24b5a6abb 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -16,6 +16,7 @@ #include #include #include +#include static u8 *emit_code(u8 *ptr, u32 bytes, unsigned int len) { @@ -2933,3 +2934,30 @@ void bpf_jit_free(struct bpf_prog *prog) bpf_prog_unlock_free(prog); } + +bool bpf_jit_supports_exceptions(void) +{ + /* We unwind through both kernel frames (starting from within bpf_throw + * call) and BPF frames. Therefore we require one of ORC or FP unwinder + * to be enabled to walk kernel frames and reach BPF frames in the stack + * trace. + */ + return IS_ENABLED(CONFIG_UNWINDER_ORC) || IS_ENABLED(CONFIG_UNWINDER_FRAME_POINTER); +} + +void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie) +{ +#if defined(CONFIG_UNWINDER_ORC) || defined(CONFIG_UNWINDER_FRAME_POINTER) + struct unwind_state state; + unsigned long addr; + + for (unwind_start(&state, current, NULL, NULL); !unwind_done(&state); + unwind_next_frame(&state)) { + addr = unwind_get_return_address(&state); + if (!addr || !consume_fn(cookie, (u64)addr, (u64)state.sp, (u64)state.bp)) + break; + } + return; +#endif + WARN(1, "verification of programs using bpf_throw should have failed\n"); +} diff --git a/include/linux/filter.h b/include/linux/filter.h index 761af6b3cf2b..9fd8f0dc4077 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -912,6 +912,8 @@ bool bpf_jit_needs_zext(void); bool bpf_jit_supports_subprog_tailcalls(void); bool bpf_jit_supports_kfunc_call(void); bool bpf_jit_supports_far_kfunc_call(void); +bool bpf_jit_supports_exceptions(void); +void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie); bool bpf_helper_changes_pkt_data(void *func); static inline bool bpf_dump_raw_ok(const struct cred *cred) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 95599df82ee4..c4ac084f2767 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -2914,6 +2914,15 @@ int __weak bpf_arch_text_invalidate(void *dst, size_t len) return -ENOTSUPP; } +bool __weak bpf_jit_supports_exceptions(void) +{ + return false; +} + +void __weak arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie) +{ +} + #ifdef CONFIG_BPF_SYSCALL static int __init bpf_global_ma_init(void) { From patchwork Tue Sep 12 23:32:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13382308 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0749017C2 for ; Tue, 12 Sep 2023 23:32:20 +0000 (UTC) Received: from mail-wr1-x442.google.com (mail-wr1-x442.google.com [IPv6:2a00:1450:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0577F10FF for ; Tue, 12 Sep 2023 16:32:20 -0700 (PDT) Received: by mail-wr1-x442.google.com with SMTP id ffacd0b85a97d-31c6d17aec4so6107237f8f.1 for ; Tue, 12 Sep 2023 16:32:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1694561538; x=1695166338; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5fOfWa2tmL8t6EH3saX0Z0ggyM4Di2cdRmBbwAqkfyk=; b=qPPRishCaidTLj6eKgXHgLU6KajK913VGvCq3p+ySUsemjX8u3t1C0VJ/4cIaoNdEE JutnnRQe6kTpuQKW+eHoa5JqYW9UcxWg9J3hpWDqibxwi3EPhK79MKJjIP+fR63sg4DS xiJincs/vGG/GjxqP/8sw782fo4xaivrCVNxzA/R6B/tEroWXjj0k+ELgEVgVbaR6xO6 WamoTQZqogS2EtRJy2Nn+xpqoe37VSaYCwbdrLK+PndA3i9EbRlFVXUwxcbe19mF6bD+ xb8tN2x6AaqbYqTF8rDQBvcsidW/ggDMMw1sJL7yr3aS94IbTBCuymgBGaBkQOCStiTn kttw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694561538; x=1695166338; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5fOfWa2tmL8t6EH3saX0Z0ggyM4Di2cdRmBbwAqkfyk=; b=vCAUHb+A6LEPEsAfQFG1IyMiIVo+o+lT9kE9beQZQb0f8R9KMRYjIb2iK6QSCl2rxn 7meIQAZj4TIxYO0V2ltDA/mA56a9d+I+Hth6NwbaVLB2Fl/mvUg/HMfXyxzR6Yvwt9QW gNTezeuQz2f0iJIZfBUTS7Qe1D+6AK3Kuqger/us4qk+lKfXNAosKoZ85jBV6ZSz7eJs Jqh2AVq5v7go0twMRn3WHcL5/YPXlhl6p/tj2QEMBlCBp+sd+vjorhuP7PQWPcuiIfBr G8HscjClo8h0cJp/VkMC8eDL4sHkA8vOS9HYQVeUv6voKJfYHI3LrecWmwcdcyfrIiuA dqdQ== X-Gm-Message-State: AOJu0Yzfz4wcj4hrhiSr6uORIWou9Vn1jymLlmX3ySi06jzmaKfcsqlf XMr/cySUAe/zvC2FOfAbKK3XyEf9tf9yGg== X-Google-Smtp-Source: AGHT+IHknOJTnoJd4b1Sq1Rj5m7sOTA1uPpiGYA337JJFHfK70Y6vShIU3tPSaBO5Ada6WLZIBSfkg== X-Received: by 2002:adf:d4c1:0:b0:31f:a277:4cde with SMTP id w1-20020adfd4c1000000b0031fa2774cdemr700068wrk.43.1694561538254; Tue, 12 Sep 2023 16:32:18 -0700 (PDT) Received: from localhost ([212.203.98.42]) by smtp.gmail.com with ESMTPSA id u12-20020aa7d98c000000b0052a1d98618bsm6506825eds.54.2023.09.12.16.32.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 16:32:18 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet , Puranjay Mohan Subject: [PATCH bpf-next v3 03/17] bpf: Implement support for adding hidden subprogs Date: Wed, 13 Sep 2023 01:32:00 +0200 Message-ID: <20230912233214.1518551-4-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230912233214.1518551-1-memxor@gmail.com> References: <20230912233214.1518551-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=8798; i=memxor@gmail.com; h=from:subject; bh=taTmdIM29si9oEtx177zT6xSGvloeVryMsABiMpFttI=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBlAPSs97mBFCvOHPHvSPjInht5snikqcS62KUx5 kYrP9VSjeWJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZQD0rAAKCRBM4MiGSL8R ytIbEACfpLni0II7Kn8DVhAWl/ug3O069hXn+ypUEKbcs8x9/JYUyerctHPtuc+85HJjWL2JyFN kmuX6bFwDs2f6eOTcCSxnVzaP2dmZhwCv3jgnVGUOcVDKfzGRXUB0NMaivXrtV5bwTiN6aqydbZ ndhSMum/nVFPu/VR9EbRtB8SV7dpbgHw7A9IoeBBIAiCs6HXm6Bk+BH0oS5SA6A9QUE9zQF+s5Q Fs+9J5OJL9Mq4u4Te3SswHzCc/LfRwy6kKjcj9o4uemTT3pmOU5qWiGiRnG5Ub8girxvZS2ibl9 4Q9Sf2qDxAKfRNd/seGPq7v+LQEkjy0chp8XkZgvpuBOB+bE14xasIs9Ieem8TyK84DV4hMWPOs OF+UpOwr1xSbNZcx78Q3DmCWloYKhBVwl8eV1CnWqXzZ53S62gp5yWh7EvEouyhwYDo5X8LaeeP tme255mjd8H8n1hTIKlBr1y/NnF82LjAV1Dz/neWeq37dc9RULPsx7dWvGLHc/xI9Ec69veTI+e joaRghkW7mJTPvyYCjHTRUgZi6XCyjFKXbasXRoyD3cIr6xxM4w3RXm8NEor7bU435WP3hw+oQ3 D5DrHsAAruI/oNQSPGhyvAmTeb8k1wfNGdYXubxKWAGBikNMUM8qMs/3AdYmO6uWfY8gof2pq6H AR+5Eusi5N7CmHw== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net Introduce support in the verifier for generating a subprogram and include it as part of a BPF program dynamically after the do_check phase is complete. The first user will be the next patch which generates default exception callbacks if none are set for the program. The phase of invocation will be do_misc_fixups. Note that this is an internal verifier function, and should be used with instruction blocks which uphold the invariants stated in check_subprogs. Since these subprogs are always appended to the end of the instruction sequence of the program, it becomes relatively inexpensive to do the related adjustments to the subprog_info of the program. Only the fake exit subprogram is shifted forward, making room for our new subprog. This is useful to insert a new subprogram, get it JITed, and obtain its function pointer. The next patch will use this functionality to insert a default exception callback which will be invoked after unwinding the stack. Note that these added subprograms are invisible to userspace, and never reported in BPF_OBJ_GET_INFO_BY_ID etc. For now, only a single subprogram is supported, but more can be easily supported in the future. To this end, two function counts are introduced now, the existing func_cnt, and real_func_cnt, the latter including hidden programs. This allows us to conver the JIT code to use the real_func_cnt for management of resources while syscall path continues working with existing func_cnt. Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf.h | 1 + include/linux/bpf_verifier.h | 3 ++- kernel/bpf/core.c | 12 ++++++------ kernel/bpf/syscall.c | 2 +- kernel/bpf/verifier.c | 36 +++++++++++++++++++++++++++++++++--- 5 files changed, 43 insertions(+), 11 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 9171b0b6a590..c3667e95af59 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1389,6 +1389,7 @@ struct bpf_prog_aux { u32 stack_depth; u32 id; u32 func_cnt; /* used by non-func prog as the number of func progs */ + u32 real_func_cnt; /* includes hidden progs, only used for JIT and freeing progs */ u32 func_idx; /* 0 for non-func prog, the index in func array for func prog */ u32 attach_btf_id; /* in-kernel BTF type id to attach to */ u32 ctx_arg_info_size; diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index a3236651ec64..3c2a8636ab29 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -588,6 +588,7 @@ struct bpf_verifier_env { u32 used_map_cnt; /* number of used maps */ u32 used_btf_cnt; /* number of used BTF objects */ u32 id_gen; /* used to generate unique reg IDs */ + u32 hidden_subprog_cnt; /* number of hidden subprogs */ bool explore_alu_limits; bool allow_ptr_leaks; bool allow_uninit_stack; @@ -598,7 +599,7 @@ struct bpf_verifier_env { struct bpf_insn_aux_data *insn_aux_data; /* array of per-insn state */ const struct bpf_line_info *prev_linfo; struct bpf_verifier_log log; - struct bpf_subprog_info subprog_info[BPF_MAX_SUBPROGS + 1]; + struct bpf_subprog_info subprog_info[BPF_MAX_SUBPROGS + 2]; /* max + 2 for the fake and exception subprogs */ union { struct bpf_idmap idmap_scratch; struct bpf_idset idset_scratch; diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index c4ac084f2767..840ba952702d 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -212,7 +212,7 @@ void bpf_prog_fill_jited_linfo(struct bpf_prog *prog, const struct bpf_line_info *linfo; void **jited_linfo; - if (!prog->aux->jited_linfo) + if (!prog->aux->jited_linfo || prog->aux->func_idx > prog->aux->func_cnt) /* Userspace did not provide linfo */ return; @@ -539,7 +539,7 @@ static void bpf_prog_kallsyms_del_subprogs(struct bpf_prog *fp) { int i; - for (i = 0; i < fp->aux->func_cnt; i++) + for (i = 0; i < fp->aux->real_func_cnt; i++) bpf_prog_kallsyms_del(fp->aux->func[i]); } @@ -589,7 +589,7 @@ bpf_prog_ksym_set_name(struct bpf_prog *prog) sym = bin2hex(sym, prog->tag, sizeof(prog->tag)); /* prog->aux->name will be ignored if full btf name is available */ - if (prog->aux->func_info_cnt) { + if (prog->aux->func_info_cnt && prog->aux->func_idx < prog->aux->func_info_cnt) { type = btf_type_by_id(prog->aux->btf, prog->aux->func_info[prog->aux->func_idx].type_id); func_name = btf_name_by_offset(prog->aux->btf, type->name_off); @@ -1208,7 +1208,7 @@ int bpf_jit_get_func_addr(const struct bpf_prog *prog, if (!extra_pass) addr = NULL; else if (prog->aux->func && - off >= 0 && off < prog->aux->func_cnt) + off >= 0 && off < prog->aux->real_func_cnt) addr = (u8 *)prog->aux->func[off]->bpf_func; else return -EINVAL; @@ -2721,7 +2721,7 @@ static void bpf_prog_free_deferred(struct work_struct *work) #endif if (aux->dst_trampoline) bpf_trampoline_put(aux->dst_trampoline); - for (i = 0; i < aux->func_cnt; i++) { + for (i = 0; i < aux->real_func_cnt; i++) { /* We can just unlink the subprog poke descriptor table as * it was originally linked to the main program and is also * released along with it. @@ -2729,7 +2729,7 @@ static void bpf_prog_free_deferred(struct work_struct *work) aux->func[i]->aux->poke_tab = NULL; bpf_jit_free(aux->func[i]); } - if (aux->func_cnt) { + if (aux->real_func_cnt) { kfree(aux->func); bpf_prog_unlock_free(aux->prog); } else { diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 6a692f3bea15..85c1d908f70f 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2749,7 +2749,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) * period before we can tear down JIT memory since symbols * are already exposed under kallsyms. */ - __bpf_prog_put_noref(prog, prog->aux->func_cnt); + __bpf_prog_put_noref(prog, prog->aux->real_func_cnt); return err; free_prog_sec: free_uid(prog->aux->user); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 18e673c0ac15..39548e326d53 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -15210,7 +15210,8 @@ static void adjust_btf_func(struct bpf_verifier_env *env) if (!aux->func_info) return; - for (i = 0; i < env->subprog_cnt; i++) + /* func_info is not available for hidden subprogs */ + for (i = 0; i < env->subprog_cnt - env->hidden_subprog_cnt; i++) aux->func_info[i].insn_off = env->subprog_info[i].start; } @@ -18151,7 +18152,8 @@ static int jit_subprogs(struct bpf_verifier_env *env) * the call instruction, as an index for this list */ func[i]->aux->func = func; - func[i]->aux->func_cnt = env->subprog_cnt; + func[i]->aux->func_cnt = env->subprog_cnt - env->hidden_subprog_cnt; + func[i]->aux->real_func_cnt = env->subprog_cnt; } for (i = 0; i < env->subprog_cnt; i++) { old_bpf_func = func[i]->bpf_func; @@ -18197,7 +18199,8 @@ static int jit_subprogs(struct bpf_verifier_env *env) prog->aux->extable = func[0]->aux->extable; prog->aux->num_exentries = func[0]->aux->num_exentries; prog->aux->func = func; - prog->aux->func_cnt = env->subprog_cnt; + prog->aux->func_cnt = env->subprog_cnt - env->hidden_subprog_cnt; + prog->aux->real_func_cnt = env->subprog_cnt; bpf_prog_jit_attempt_done(prog); return 0; out_free: @@ -18433,6 +18436,33 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, return 0; } +/* The function requires that first instruction in 'patch' is insnsi[prog->len - 1] */ +static __maybe_unused int add_hidden_subprog(struct bpf_verifier_env *env, struct bpf_insn *patch, int len) +{ + struct bpf_subprog_info *info = env->subprog_info; + int cnt = env->subprog_cnt; + struct bpf_prog *prog; + + /* We only reserve one slot for hidden subprogs in subprog_info. */ + if (env->hidden_subprog_cnt) { + verbose(env, "verifier internal error: only one hidden subprog supported\n"); + return -EFAULT; + } + /* We're not patching any existing instruction, just appending the new + * ones for the hidden subprog. Hence all of the adjustment operations + * in bpf_patch_insn_data are no-ops. + */ + prog = bpf_patch_insn_data(env, env->prog->len - 1, patch, len); + if (!prog) + return -ENOMEM; + env->prog = prog; + info[cnt + 1].start = info[cnt].start; + info[cnt].start = prog->len - len + 1; + env->subprog_cnt++; + env->hidden_subprog_cnt++; + return 0; +} + /* Do various post-verification rewrites in a single program pass. * These rewrites simplify JIT and interpreter implementations. */ From patchwork Tue Sep 12 23:32:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13382310 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D92E917C2 for ; Tue, 12 Sep 2023 23:32:21 +0000 (UTC) Received: from mail-ed1-x544.google.com (mail-ed1-x544.google.com [IPv6:2a00:1450:4864:20::544]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2505710FE for ; Tue, 12 Sep 2023 16:32:21 -0700 (PDT) Received: by mail-ed1-x544.google.com with SMTP id 4fb4d7f45d1cf-51a52a7d859so789451a12.0 for ; Tue, 12 Sep 2023 16:32:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1694561539; x=1695166339; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QJ21yuVoz7wiPdHAcyvrcUWYh9xWXZBQttp6kj1ZBwo=; b=sivt6lPyZY/n7oI3pZfZSTHjCzg+boSIy8uiJbX9UjvW0+sKy4lSXKrOGpWLCPOjze 8AmIsomeG1w37AO3cp7yRC/bjEvNdNzVwEYlBgIOVf9j1bVRKaLsDOKw0mfIl2aMkquS ZxRS13x4MzuquR5haWgIWGOcnpemmwQl5IjUKA3l4QyHjva7ulkiEVaTFiaQXeY6ITou eri8LWWfkOmmaCoD6968u9sdAfS9Sp4S2RkgarokApLm3K8Lje0T3DB66cg4VnxgqZCr UQJ5gGSbT74VPSDkZEMBhrcHSAbwe9O41bghzGQJ2xkOJniVBSrRb0QA4+9yps0ZtbqK 7zKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694561539; x=1695166339; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QJ21yuVoz7wiPdHAcyvrcUWYh9xWXZBQttp6kj1ZBwo=; b=ibVg88nEiiCix+i45JpXAspV0N407RgL4AioXO6i+bc4wJXaUgSccx2+kkctztwJXR K/EPk+jctQq3GKkxPb63WScNVVBjUn165jmdm4wp8l4/z7soyWhN2XjLIbLOYWEW4y+B LHFXKoWoZ7EpJRD9ZGN7yG+14veb0lmUbeC1zXPFo5DRS9jKi1eYg5OLQ66SAoYxr4PJ 8c5VkPWqWgloX1ZKrHO/WwNZgSVaRypG9xEXQD+HG6G+Z41D5oYQ2rs8LpKA9RSG5mBb su/h861933cS3si3MEdLxD2PsSVJJYQA8EBSvUi7rwaouTLs4ndC3VSneKBHp9dip9Sy fBzQ== X-Gm-Message-State: AOJu0YxnDMBH1df9xKeZLERqbE0NX92l4PtkNf2a3iqnCXb2txq+M5IU t5X8fZQLv4/gLi4dNEuaDA16HmA1vGbfVA== X-Google-Smtp-Source: AGHT+IHkDC7og1nV7UDcQ/MLZZrddrLeltuD0oMJ9XUE1jTDYrCd3cbpbkQCQO53PMkvX2iJdDWVSw== X-Received: by 2002:aa7:d8d0:0:b0:523:b37e:b83b with SMTP id k16-20020aa7d8d0000000b00523b37eb83bmr1621715eds.13.1694561538998; Tue, 12 Sep 2023 16:32:18 -0700 (PDT) Received: from localhost ([212.203.98.42]) by smtp.gmail.com with ESMTPSA id m21-20020a50ef15000000b005232ea6a330sm6470428eds.2.2023.09.12.16.32.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 16:32:18 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet , Puranjay Mohan Subject: [PATCH bpf-next v3 04/17] bpf: Implement BPF exceptions Date: Wed, 13 Sep 2023 01:32:01 +0200 Message-ID: <20230912233214.1518551-5-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230912233214.1518551-1-memxor@gmail.com> References: <20230912233214.1518551-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=25872; i=memxor@gmail.com; h=from:subject; bh=haq9dFiLNXTn+v6xsXQzWfZbt4kHIpUdng66Xk64EOI=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBlAPSszYVUD8edRz1EGJF9jwcyYQROOHCHkWQ37 +lXeM/x6haJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZQD0rAAKCRBM4MiGSL8R ypQfEACZpIv+eSsut61HjyrpdwhZfenGSJzSWqYIpYORDpF5nGFjUxBcSOWbDfl5xVMBvJ6XQlo t7jiQ0darTIHxGJTFv/rhyZ3h2imvbtDdKYvhDMgPWexJbnPgExODWQ+YTVG9TA81VOprTqYehD NXCbmKuFR0F28oHTnPC5ZZYYGUrMWCDlt4j5aAYny9FukH7mWdeR1G+HXHRTqYv912d+KxTMq7E GzR8CkpsR58K6BeTB6G62aZQ6TuYxaDF2qxDJm4GtYc8KFu7IwFGPYLcONPh66k1JtA8/aJq1IE +Yts/B1A/q8nKWh8NGNGn9vORM6zjYQkEbV7FfP3BbOx3bPFiu7+9rUY9pPdNwyOCfZoTCCnZyd DwOVlQpgXG1bTR3GihjBHVt3JzyoJt4yN22txGjqLelAHQ8k0zvwdw2OS3EQj0RQWqeuo23zEm3 CrIM89KWcxINB01Jhd9CjW/0mdVXJCdcjecK7RQNBw6WJr0Az3PWLtbD66m7InDCSn5b2ut4Kd7 aruuLhVm/l5sflUY8/AMHoS9JQnqZdGCkesk9/eQIqZvco7vvx2YCRsbKdCXU9S+4hK+ftqH7n6 R0MWAMjC3goaMFhMMgEmwJ8p36lbsaeoRa1ojScwsy10xucSc9+bG+qwTxLHmOKAVoJJWl1hs1p a7EKOrNJUdMyHjA== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net This patch implements BPF exceptions, and introduces a bpf_throw kfunc to allow programs to throw exceptions during their execution at runtime. A bpf_throw invocation is treated as an immediate termination of the program, returning back to its caller within the kernel, unwinding all stack frames. This allows the program to simplify its implementation, by testing for runtime conditions which the verifier has no visibility into, and assert that they are true. In case they are not, the program can simply throw an exception from the other branch. BPF exceptions are explicitly *NOT* an unlikely slowpath error handling primitive, and this objective has guided design choices of the implementation of the them within the kernel (with the bulk of the cost for unwinding the stack offloaded to the bpf_throw kfunc). The implementation of this mechanism requires use of add_hidden_subprog mechanism introduced in the previous patch, which generates a couple of instructions to move R1 to R0 and exit. The JIT then rewrites the prologue of this subprog to take the stack pointer and frame pointer as inputs and reset the stack frame, popping all callee-saved registers saved by the main subprog. The bpf_throw function then walks the stack at runtime, and invokes this exception subprog with the stack and frame pointers as parameters. Reviewers must take note that currently the main program is made to save all callee-saved registers on x86_64 during entry into the program. This is because we must do an equivalent of a lightweight context switch when unwinding the stack, therefore we need the callee-saved registers of the caller of the BPF program to be able to return with a sane state. Note that we have to additionally handle r12, even though it is not used by the program, because when throwing the exception the program makes an entry into the kernel which could clobber r12 after saving it on the stack. To be able to preserve the value we received on program entry, we push r12 and restore it from the generated subprogram when unwinding the stack. For now, bpf_throw invocation fails when lingering resources or locks exist in that path of the program. In a future followup, bpf_throw will be extended to perform frame-by-frame unwinding to release lingering resources for each stack frame, removing this limitation. Signed-off-by: Kumar Kartikeya Dwivedi --- arch/x86/net/bpf_jit_comp.c | 89 ++++++++++++-- include/linux/bpf.h | 3 + include/linux/bpf_verifier.h | 4 + include/linux/filter.h | 6 + kernel/bpf/core.c | 2 +- kernel/bpf/helpers.c | 38 ++++++ kernel/bpf/verifier.c | 116 +++++++++++++++--- .../testing/selftests/bpf/bpf_experimental.h | 16 +++ 8 files changed, 247 insertions(+), 27 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index d0c24b5a6abb..84005f2114e0 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -18,6 +18,8 @@ #include #include +static bool all_callee_regs_used[4] = {true, true, true, true}; + static u8 *emit_code(u8 *ptr, u32 bytes, unsigned int len) { if (len == 1) @@ -256,6 +258,14 @@ struct jit_context { /* Number of bytes that will be skipped on tailcall */ #define X86_TAIL_CALL_OFFSET (11 + ENDBR_INSN_SIZE) +static void push_r12(u8 **pprog) +{ + u8 *prog = *pprog; + + EMIT2(0x41, 0x54); /* push r12 */ + *pprog = prog; +} + static void push_callee_regs(u8 **pprog, bool *callee_regs_used) { u8 *prog = *pprog; @@ -271,6 +281,14 @@ static void push_callee_regs(u8 **pprog, bool *callee_regs_used) *pprog = prog; } +static void pop_r12(u8 **pprog) +{ + u8 *prog = *pprog; + + EMIT2(0x41, 0x5C); /* pop r12 */ + *pprog = prog; +} + static void pop_callee_regs(u8 **pprog, bool *callee_regs_used) { u8 *prog = *pprog; @@ -292,7 +310,8 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used) * while jumping to another program */ static void emit_prologue(u8 **pprog, u32 stack_depth, bool ebpf_from_cbpf, - bool tail_call_reachable, bool is_subprog) + bool tail_call_reachable, bool is_subprog, + bool is_exception_cb) { u8 *prog = *pprog; @@ -312,8 +331,22 @@ static void emit_prologue(u8 **pprog, u32 stack_depth, bool ebpf_from_cbpf, /* Keep the same instruction layout. */ EMIT2(0x66, 0x90); /* nop2 */ } - EMIT1(0x55); /* push rbp */ - EMIT3(0x48, 0x89, 0xE5); /* mov rbp, rsp */ + /* Exception callback receives FP as third parameter */ + if (is_exception_cb) { + EMIT3(0x48, 0x89, 0xF4); /* mov rsp, rsi */ + EMIT3(0x48, 0x89, 0xD5); /* mov rbp, rdx */ + /* The main frame must have exception_boundary as true, so we + * first restore those callee-saved regs from stack, before + * reusing the stack frame. + */ + pop_callee_regs(&prog, all_callee_regs_used); + pop_r12(&prog); + /* Reset the stack frame. */ + EMIT3(0x48, 0x89, 0xEC); /* mov rsp, rbp */ + } else { + EMIT1(0x55); /* push rbp */ + EMIT3(0x48, 0x89, 0xE5); /* mov rbp, rsp */ + } /* X86_TAIL_CALL_OFFSET is here */ EMIT_ENDBR(); @@ -472,7 +505,8 @@ static void emit_return(u8 **pprog, u8 *ip) * goto *(prog->bpf_func + prologue_size); * out: */ -static void emit_bpf_tail_call_indirect(u8 **pprog, bool *callee_regs_used, +static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog, + u8 **pprog, bool *callee_regs_used, u32 stack_depth, u8 *ip, struct jit_context *ctx) { @@ -522,7 +556,12 @@ static void emit_bpf_tail_call_indirect(u8 **pprog, bool *callee_regs_used, offset = ctx->tail_call_indirect_label - (prog + 2 - start); EMIT2(X86_JE, offset); /* je out */ - pop_callee_regs(&prog, callee_regs_used); + if (bpf_prog->aux->exception_boundary) { + pop_callee_regs(&prog, all_callee_regs_used); + pop_r12(&prog); + } else { + pop_callee_regs(&prog, callee_regs_used); + } EMIT1(0x58); /* pop rax */ if (stack_depth) @@ -546,7 +585,8 @@ static void emit_bpf_tail_call_indirect(u8 **pprog, bool *callee_regs_used, *pprog = prog; } -static void emit_bpf_tail_call_direct(struct bpf_jit_poke_descriptor *poke, +static void emit_bpf_tail_call_direct(struct bpf_prog *bpf_prog, + struct bpf_jit_poke_descriptor *poke, u8 **pprog, u8 *ip, bool *callee_regs_used, u32 stack_depth, struct jit_context *ctx) @@ -575,7 +615,13 @@ static void emit_bpf_tail_call_direct(struct bpf_jit_poke_descriptor *poke, emit_jump(&prog, (u8 *)poke->tailcall_target + X86_PATCH_SIZE, poke->tailcall_bypass); - pop_callee_regs(&prog, callee_regs_used); + if (bpf_prog->aux->exception_boundary) { + pop_callee_regs(&prog, all_callee_regs_used); + pop_r12(&prog); + } else { + pop_callee_regs(&prog, callee_regs_used); + } + EMIT1(0x58); /* pop rax */ if (stack_depth) EMIT3_off32(0x48, 0x81, 0xC4, round_up(stack_depth, 8)); @@ -1050,8 +1096,20 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image emit_prologue(&prog, bpf_prog->aux->stack_depth, bpf_prog_was_classic(bpf_prog), tail_call_reachable, - bpf_is_subprog(bpf_prog)); - push_callee_regs(&prog, callee_regs_used); + bpf_is_subprog(bpf_prog), bpf_prog->aux->exception_cb); + /* Exception callback will clobber callee regs for its own use, and + * restore the original callee regs from main prog's stack frame. + */ + if (bpf_prog->aux->exception_boundary) { + /* We also need to save r12, which is not mapped to any BPF + * register, as we throw after entry into the kernel, which may + * overwrite r12. + */ + push_r12(&prog); + push_callee_regs(&prog, all_callee_regs_used); + } else { + push_callee_regs(&prog, callee_regs_used); + } ilen = prog - temp; if (rw_image) @@ -1648,13 +1706,15 @@ st: if (is_imm8(insn->off)) case BPF_JMP | BPF_TAIL_CALL: if (imm32) - emit_bpf_tail_call_direct(&bpf_prog->aux->poke_tab[imm32 - 1], + emit_bpf_tail_call_direct(bpf_prog, + &bpf_prog->aux->poke_tab[imm32 - 1], &prog, image + addrs[i - 1], callee_regs_used, bpf_prog->aux->stack_depth, ctx); else - emit_bpf_tail_call_indirect(&prog, + emit_bpf_tail_call_indirect(bpf_prog, + &prog, callee_regs_used, bpf_prog->aux->stack_depth, image + addrs[i - 1], @@ -1907,7 +1967,12 @@ st: if (is_imm8(insn->off)) seen_exit = true; /* Update cleanup_addr */ ctx->cleanup_addr = proglen; - pop_callee_regs(&prog, callee_regs_used); + if (bpf_prog->aux->exception_boundary) { + pop_callee_regs(&prog, all_callee_regs_used); + pop_r12(&prog); + } else { + pop_callee_regs(&prog, callee_regs_used); + } EMIT1(0xC9); /* leave */ emit_return(&prog, image + addrs[i - 1] + (prog - temp)); break; diff --git a/include/linux/bpf.h b/include/linux/bpf.h index c3667e95af59..16740ee82082 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1410,6 +1410,8 @@ struct bpf_prog_aux { bool sleepable; bool tail_call_reachable; bool xdp_has_frags; + bool exception_cb; + bool exception_boundary; /* BTF_KIND_FUNC_PROTO for valid attach_btf_id */ const struct btf_type *attach_func_proto; /* function name for valid attach_btf_id */ @@ -1432,6 +1434,7 @@ struct bpf_prog_aux { int cgroup_atype; /* enum cgroup_bpf_attach_type */ struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE]; char name[BPF_OBJ_NAME_LEN]; + unsigned int (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp); #ifdef CONFIG_SECURITY void *security; #endif diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 3c2a8636ab29..da21a3ec5027 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -541,7 +541,9 @@ struct bpf_subprog_info { bool has_tail_call; bool tail_call_reachable; bool has_ld_abs; + bool is_cb; bool is_async_cb; + bool is_exception_cb; }; struct bpf_verifier_env; @@ -589,6 +591,7 @@ struct bpf_verifier_env { u32 used_btf_cnt; /* number of used BTF objects */ u32 id_gen; /* used to generate unique reg IDs */ u32 hidden_subprog_cnt; /* number of hidden subprogs */ + int exception_callback_subprog; bool explore_alu_limits; bool allow_ptr_leaks; bool allow_uninit_stack; @@ -596,6 +599,7 @@ struct bpf_verifier_env { bool bypass_spec_v1; bool bypass_spec_v4; bool seen_direct_write; + bool seen_exception; struct bpf_insn_aux_data *insn_aux_data; /* array of per-insn state */ const struct bpf_line_info *prev_linfo; struct bpf_verifier_log log; diff --git a/include/linux/filter.h b/include/linux/filter.h index 9fd8f0dc4077..389e550a6a25 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -1129,6 +1129,7 @@ const char *__bpf_address_lookup(unsigned long addr, unsigned long *size, bool is_bpf_text_address(unsigned long addr); int bpf_get_kallsym(unsigned int symnum, unsigned long *value, char *type, char *sym); +struct bpf_prog *bpf_prog_ksym_find(unsigned long addr); static inline const char * bpf_address_lookup(unsigned long addr, unsigned long *size, @@ -1196,6 +1197,11 @@ static inline int bpf_get_kallsym(unsigned int symnum, unsigned long *value, return -ERANGE; } +static inline struct bpf_prog *bpf_prog_ksym_find(unsigned long addr) +{ + return NULL; +} + static inline const char * bpf_address_lookup(unsigned long addr, unsigned long *size, unsigned long *off, char **modname, char *sym) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 840ba952702d..7849b9cca749 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -733,7 +733,7 @@ bool is_bpf_text_address(unsigned long addr) return ret; } -static struct bpf_prog *bpf_prog_ksym_find(unsigned long addr) +struct bpf_prog *bpf_prog_ksym_find(unsigned long addr) { struct bpf_ksym *ksym = bpf_ksym_find(addr); diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index b0a9834f1051..78e8f4de6750 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -2449,6 +2449,43 @@ __bpf_kfunc void bpf_rcu_read_unlock(void) rcu_read_unlock(); } +struct bpf_throw_ctx { + struct bpf_prog_aux *aux; + u64 sp; + u64 bp; + int cnt; +}; + +static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp) +{ + struct bpf_throw_ctx *ctx = cookie; + struct bpf_prog *prog; + + if (!is_bpf_text_address(ip)) + return !ctx->cnt; + prog = bpf_prog_ksym_find(ip); + ctx->cnt++; + if (bpf_is_subprog(prog)) + return true; + ctx->aux = prog->aux; + ctx->sp = sp; + ctx->bp = bp; + return false; +} + +__bpf_kfunc void bpf_throw(u64 cookie) +{ + struct bpf_throw_ctx ctx = {}; + + arch_bpf_stack_walk(bpf_stack_walker, &ctx); + WARN_ON_ONCE(!ctx.aux); + if (ctx.aux) + WARN_ON_ONCE(!ctx.aux->exception_boundary); + WARN_ON_ONCE(!ctx.bp); + WARN_ON_ONCE(!ctx.cnt); + ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp); +} + __diag_pop(); BTF_SET8_START(generic_btf_ids) @@ -2478,6 +2515,7 @@ BTF_ID_FLAGS(func, bpf_cgroup_from_id, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_task_under_cgroup, KF_RCU) #endif BTF_ID_FLAGS(func, bpf_task_from_pid, KF_ACQUIRE | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_throw) BTF_SET8_END(generic_btf_ids) static const struct btf_kfunc_id_set generic_kfunc_set = { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 39548e326d53..9baa6f187b38 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -543,6 +543,7 @@ static bool is_dynptr_ref_function(enum bpf_func_id func_id) } static bool is_callback_calling_kfunc(u32 btf_id); +static bool is_bpf_throw_kfunc(struct bpf_insn *insn); static bool is_callback_calling_function(enum bpf_func_id func_id) { @@ -1748,7 +1749,9 @@ static int copy_verifier_state(struct bpf_verifier_state *dst_state, return -ENOMEM; dst_state->jmp_history_cnt = src->jmp_history_cnt; - /* if dst has more stack frames then src frame, free them */ + /* if dst has more stack frames then src frame, free them, this is also + * necessary in case of exceptional exits using bpf_throw. + */ for (i = src->curframe + 1; i <= dst_state->curframe; i++) { free_func_state(dst_state->frame[i]); dst_state->frame[i] = NULL; @@ -2868,7 +2871,7 @@ static int check_subprogs(struct bpf_verifier_env *env) if (i == subprog_end - 1) { /* to avoid fall-through from one subprog into another * the last insn of the subprog should be either exit - * or unconditional jump back + * or unconditional jump back or bpf_throw call */ if (code != (BPF_JMP | BPF_EXIT) && code != (BPF_JMP32 | BPF_JA) && @@ -5661,6 +5664,27 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx) for (; i < subprog_end; i++) { int next_insn, sidx; + if (bpf_pseudo_kfunc_call(insn + i) && !insn[i].off) { + bool err = false; + + if (!is_bpf_throw_kfunc(insn + i)) + continue; + if (subprog[idx].is_cb) + err = true; + for (int c = 0; c < frame && !err; c++) { + if (subprog[ret_prog[c]].is_cb) { + err = true; + break; + } + } + if (!err) + continue; + verbose(env, + "bpf_throw kfunc (insn %d) cannot be called from callback subprog %d\n", + i, idx); + return -EINVAL; + } + if (!bpf_pseudo_call(insn + i) && !bpf_pseudo_func(insn + i)) continue; /* remember insn and function to return to */ @@ -8919,6 +8943,7 @@ static int __check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn * callbacks */ if (set_callee_state_cb != set_callee_state) { + env->subprog_info[subprog].is_cb = true; if (bpf_pseudo_kfunc_call(insn) && !is_callback_calling_kfunc(insn->imm)) { verbose(env, "verifier bug: kfunc %s#%d not marked as callback-calling\n", @@ -9308,7 +9333,8 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) verbose(env, "to caller at %d:\n", *insn_idx); print_verifier_state(env, caller, true); } - /* clear everything in the callee */ + /* clear everything in the callee. In case of exceptional exits using + * bpf_throw, this will be done by copy_verifier_state for extra frames. */ free_func_state(callee); state->frame[state->curframe--] = NULL; return 0; @@ -9432,17 +9458,17 @@ record_func_key(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, return 0; } -static int check_reference_leak(struct bpf_verifier_env *env) +static int check_reference_leak(struct bpf_verifier_env *env, bool exception_exit) { struct bpf_func_state *state = cur_func(env); bool refs_lingering = false; int i; - if (state->frameno && !state->in_callback_fn) + if (!exception_exit && state->frameno && !state->in_callback_fn) return 0; for (i = 0; i < state->acquired_refs; i++) { - if (state->in_callback_fn && state->refs[i].callback_ref != state->frameno) + if (!exception_exit && state->in_callback_fn && state->refs[i].callback_ref != state->frameno) continue; verbose(env, "Unreleased reference id=%d alloc_insn=%d\n", state->refs[i].id, state->refs[i].insn_idx); @@ -9697,7 +9723,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn switch (func_id) { case BPF_FUNC_tail_call: - err = check_reference_leak(env); + err = check_reference_leak(env, false); if (err) { verbose(env, "tail_call would lead to reference leak\n"); return err; @@ -10332,6 +10358,7 @@ enum special_kfunc_type { KF_bpf_dynptr_clone, KF_bpf_percpu_obj_new_impl, KF_bpf_percpu_obj_drop_impl, + KF_bpf_throw, }; BTF_SET_START(special_kfunc_set) @@ -10354,6 +10381,7 @@ BTF_ID(func, bpf_dynptr_slice_rdwr) BTF_ID(func, bpf_dynptr_clone) BTF_ID(func, bpf_percpu_obj_new_impl) BTF_ID(func, bpf_percpu_obj_drop_impl) +BTF_ID(func, bpf_throw) BTF_SET_END(special_kfunc_set) BTF_ID_LIST(special_kfunc_list) @@ -10378,6 +10406,7 @@ BTF_ID(func, bpf_dynptr_slice_rdwr) BTF_ID(func, bpf_dynptr_clone) BTF_ID(func, bpf_percpu_obj_new_impl) BTF_ID(func, bpf_percpu_obj_drop_impl) +BTF_ID(func, bpf_throw) static bool is_kfunc_ret_null(struct bpf_kfunc_call_arg_meta *meta) { @@ -10695,6 +10724,12 @@ static bool is_callback_calling_kfunc(u32 btf_id) return btf_id == special_kfunc_list[KF_bpf_rbtree_add_impl]; } +static bool is_bpf_throw_kfunc(struct bpf_insn *insn) +{ + return bpf_pseudo_kfunc_call(insn) && insn->off == 0 && + insn->imm == special_kfunc_list[KF_bpf_throw]; +} + static bool is_rbtree_lock_required_kfunc(u32 btf_id) { return is_bpf_rbtree_api_kfunc(btf_id); @@ -11480,6 +11515,15 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, } } + if (meta.func_id == special_kfunc_list[KF_bpf_throw]) { + if (!bpf_jit_supports_exceptions()) { + verbose(env, "JIT does not support calling kfunc %s#%d\n", + func_name, meta.func_id); + return -ENOTSUPP; + } + env->seen_exception = true; + } + for (i = 0; i < CALLER_SAVED_REGS; i++) mark_reg_not_init(env, regs, caller_saved[i]); @@ -14525,7 +14569,7 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn) * gen_ld_abs() may terminate the program at runtime, leading to * reference leak. */ - err = check_reference_leak(env); + err = check_reference_leak(env, false); if (err) { verbose(env, "BPF_LD_[ABS|IND] cannot be mixed with socket references\n"); return err; @@ -16539,6 +16583,7 @@ static int do_check(struct bpf_verifier_env *env) int prev_insn_idx = -1; for (;;) { + bool exception_exit = false; struct bpf_insn *insn; u8 class; int err; @@ -16753,12 +16798,17 @@ static int do_check(struct bpf_verifier_env *env) return -EINVAL; } } - if (insn->src_reg == BPF_PSEUDO_CALL) + if (insn->src_reg == BPF_PSEUDO_CALL) { err = check_func_call(env, insn, &env->insn_idx); - else if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) + } else if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) { err = check_kfunc_call(env, insn, &env->insn_idx); - else + if (!err && is_bpf_throw_kfunc(insn)) { + exception_exit = true; + goto process_bpf_exit_full; + } + } else { err = check_helper_call(env, insn, &env->insn_idx); + } if (err) return err; @@ -16788,7 +16838,7 @@ static int do_check(struct bpf_verifier_env *env) verbose(env, "BPF_EXIT uses reserved fields\n"); return -EINVAL; } - +process_bpf_exit_full: if (env->cur_state->active_lock.ptr && !in_rbtree_lock_required_cb(env)) { verbose(env, "bpf_spin_unlock is missing\n"); @@ -16807,10 +16857,23 @@ static int do_check(struct bpf_verifier_env *env) * function, for which reference_state must * match caller reference state when it exits. */ - err = check_reference_leak(env); + err = check_reference_leak(env, exception_exit); if (err) return err; + /* The side effect of the prepare_func_exit + * which is being skipped is that it frees + * bpf_func_state. Typically, process_bpf_exit + * will only be hit with outermost exit. + * copy_verifier_state in pop_stack will handle + * freeing of any extra bpf_func_state left over + * from not processing all nested function + * exits. We also skip return code checks as + * they are not needed for exceptional exits. + */ + if (exception_exit) + goto process_bpf_exit; + if (state->curframe) { /* exit from nested function */ err = prepare_func_exit(env, &env->insn_idx); @@ -18113,6 +18176,9 @@ static int jit_subprogs(struct bpf_verifier_env *env) } func[i]->aux->num_exentries = num_exentries; func[i]->aux->tail_call_reachable = env->subprog_info[i].tail_call_reachable; + func[i]->aux->exception_cb = env->subprog_info[i].is_exception_cb; + if (!i) + func[i]->aux->exception_boundary = env->seen_exception; func[i] = bpf_int_jit_compile(func[i]); if (!func[i]->jited) { err = -ENOTSUPP; @@ -18201,6 +18267,8 @@ static int jit_subprogs(struct bpf_verifier_env *env) prog->aux->func = func; prog->aux->func_cnt = env->subprog_cnt - env->hidden_subprog_cnt; prog->aux->real_func_cnt = env->subprog_cnt; + prog->aux->bpf_exception_cb = (void *)func[env->exception_callback_subprog]->bpf_func; + prog->aux->exception_boundary = func[0]->aux->exception_boundary; bpf_prog_jit_attempt_done(prog); return 0; out_free: @@ -18437,7 +18505,7 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, } /* The function requires that first instruction in 'patch' is insnsi[prog->len - 1] */ -static __maybe_unused int add_hidden_subprog(struct bpf_verifier_env *env, struct bpf_insn *patch, int len) +static int add_hidden_subprog(struct bpf_verifier_env *env, struct bpf_insn *patch, int len) { struct bpf_subprog_info *info = env->subprog_info; int cnt = env->subprog_cnt; @@ -18481,6 +18549,26 @@ static int do_misc_fixups(struct bpf_verifier_env *env) struct bpf_map *map_ptr; int i, ret, cnt, delta = 0; + if (env->seen_exception && !env->exception_callback_subprog) { + struct bpf_insn patch[] = { + env->prog->insnsi[insn_cnt - 1], + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }; + + ret = add_hidden_subprog(env, patch, ARRAY_SIZE(patch)); + if (ret < 0) + return ret; + prog = env->prog; + insn = prog->insnsi; + + env->exception_callback_subprog = env->subprog_cnt - 1; + /* Don't update insn_cnt, as add_hidden_subprog always appends insns */ + env->subprog_info[env->exception_callback_subprog].is_cb = true; + env->subprog_info[env->exception_callback_subprog].is_async_cb = true; + env->subprog_info[env->exception_callback_subprog].is_exception_cb = true; + } + for (i = 0; i < insn_cnt; i++, insn++) { /* Make divide-by-zero exceptions impossible. */ if (insn->code == (BPF_ALU64 | BPF_MOD | BPF_X) || diff --git a/tools/testing/selftests/bpf/bpf_experimental.h b/tools/testing/selftests/bpf/bpf_experimental.h index 4494eaa9937e..333b54a86e3a 100644 --- a/tools/testing/selftests/bpf/bpf_experimental.h +++ b/tools/testing/selftests/bpf/bpf_experimental.h @@ -162,4 +162,20 @@ extern void bpf_percpu_obj_drop_impl(void *kptr, void *meta) __ksym; /* Convenience macro to wrap over bpf_obj_drop_impl */ #define bpf_percpu_obj_drop(kptr) bpf_percpu_obj_drop_impl(kptr, NULL) +/* Description + * Throw a BPF exception from the program, immediately terminating its + * execution and unwinding the stack. The supplied 'cookie' parameter + * will be the return value of the program when an exception is thrown. + * + * Note that throwing an exception with lingering resources (locks, + * references, etc.) will lead to a verification error. + * + * Note that callbacks *cannot* call this helper. + * Returns + * Never. + * Throws + * An exception with the specified 'cookie' value. + */ +extern void bpf_throw(u64 cookie) __ksym; + #endif From patchwork Tue Sep 12 23:32:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13382309 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1489F1429F for ; Tue, 12 Sep 2023 23:32:22 +0000 (UTC) Received: from mail-ej1-x643.google.com (mail-ej1-x643.google.com [IPv6:2a00:1450:4864:20::643]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6DEB210FF for ; Tue, 12 Sep 2023 16:32:21 -0700 (PDT) Received: by mail-ej1-x643.google.com with SMTP id a640c23a62f3a-99bdcade7fbso769046366b.1 for ; Tue, 12 Sep 2023 16:32:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1694561540; x=1695166340; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FnInjG/GiUGZnd4FDblN9l9WbO13rPNJBMBFSUVUPls=; b=i0ygThk33J2tbEvT1QbJweiD5mBITQJ1ZVHAo2mIplBrPhuJ2l+imvO+w24yNe26Eh yIMq++2/dXxJ8A2rM75lwi6vxAvUnJY+c9Z7yQX00D5ldHDO9+4d2ZC30sXvdlf+CGEF Dd03e8L4Ku/1zeu80ZSM50vyZcLjjHwADeAJdWPr43i7PjdBc/drkSKt+ddgeatzDQH6 Q/H8WuTu9ePGGCZKGBYoWYA5Hxi4H4Ll7iw/JqDfcBEy3NQg72xXkimnMU+iCKw/eOlj mT8NnQ+ZxFTDXKM4gzlpzWSKhBkl4VEthJGnnK3m/YDV5FhAhS7RTzV0+EofHvNmhVao 2JAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694561540; x=1695166340; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FnInjG/GiUGZnd4FDblN9l9WbO13rPNJBMBFSUVUPls=; b=wTP6g2ZzwizT1UPxt8L9MS0DyHqPFPE4y+2gjf+2lHu1JWXp0kLclYv/QPuFLI3K0V jgEvwU9xYu1iclmH0qA+AW/M5By5YMHJHs4t09W0vondxn+ZfZteCZwZHJeFjhGvZ05m u7+XRdbwER4q/rUHVjbTxg6+LQ0ySduTLWYeYfSNAo8rFsQ6Ptw/uxE4C/n6IJdL2McC zeVpy9dbpAYgAK4kqa4+EE6ErKZkj1i60C7EDuDnMviufCc+VzDfR7qU5BYYrmPf9B2I OxALZQaZ2AN66Lncckz1BiHwC4/ROlhG/KoKNebtWK5WMwsX+CZnAAeLNVZBDf1Sc8oq 5VeA== X-Gm-Message-State: AOJu0YyxyEPbvX451C9BCIy2Z7jTH0pg95pxfIS3NIN3ZIY4q5noz2LB wzHfogWfgLmJkEG+Y9LgPhpEAv8fvxIihw== X-Google-Smtp-Source: AGHT+IEDHXBr49p4haN+dJO2ObWGfLaD+m2z1Tf5uRp0SZkey96d3yc65OaW5oacSsM9/ICs5oWmnw== X-Received: by 2002:a17:907:7714:b0:9a9:e53d:5d59 with SMTP id kw20-20020a170907771400b009a9e53d5d59mr435155ejc.25.1694561539712; Tue, 12 Sep 2023 16:32:19 -0700 (PDT) Received: from localhost ([212.203.98.42]) by smtp.gmail.com with ESMTPSA id oy25-20020a170907105900b0099d0c0bb92bsm7430481ejb.80.2023.09.12.16.32.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 16:32:19 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet , Puranjay Mohan Subject: [PATCH bpf-next v3 05/17] bpf: Refactor check_btf_func and split into two phases Date: Wed, 13 Sep 2023 01:32:02 +0200 Message-ID: <20230912233214.1518551-6-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230912233214.1518551-1-memxor@gmail.com> References: <20230912233214.1518551-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=7961; i=memxor@gmail.com; h=from:subject; bh=E2xIvz5e13nheKE0OeG1Qehip8VtJjWYOdIDnjR2D4k=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBlAPSs53rV+MnMc4PA1jYNqb6XVsGGNZe6U5n9o aLTEH/dMQKJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZQD0rAAKCRBM4MiGSL8R ygMND/48xFUhIxLjOMH3fHgkL9Yz6jpiZphZy+Dzh67/gmjktFT49+F78I4pL8/AVYmmHk99p0Y FISoFsb9HV1RoRYAwmrzCbGZ4RQkqBf4QbC9QzIbH5B02UZcDv8+bUJwyBlYKkoLJVpoP+BxiL2 +Xny4s9DyL0Ci2k8eD6lPZHdn+6K8DssODR0aYE+On7QmkqxLGQ1LC50NlPLZdLhg9nlT/mG2ff L7MSWCEQRp+0rR3GtPSGCLZ3+jzCFLMBVw5czVrvb7HV8ohrFSklC6r6Kjjc/UGW5r5GfPC6ZPr ycOV7yGJ+IlYlHrqz56vdyHZvfaZWmohOT/odGWYaTS9RqY7J+/RJtsHpFdllqIBN1qI6hmwCB3 MIO5H/J6VuxwP3wTdkDsiuNMaLmFIKYFRRaM/3OjfENUAh0ouL1x5JirLGc+3oKS1KdP6QNYPBw 8oLiNrKVcas3aviiGBf8sr7S48D1RGN/vGxm2UZUHonZNQrZdYi3ZTaKCqV4QIuwfzf0/+oQrE+ MMmytdWitH/5L1GGO3nht2oFBM1UR3uIyfj5xPEvdTlgyqzzWi4Gy5p1Kv/Oem3dECrt7ckp8GI LvES+uakiLRidlcBW2IiE3bZ44GJzoysEiEKPmf4leHej1rz9XsdqyxYOoeqLHSATaDMrq5FtQp pW/gXqh8dkR2WGA== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net This patch splits the check_btf_info's check_btf_func check into two separate phases. The first phase sets up the BTF and prepares func_info, but does not perform any validation of required invariants for subprogs just yet. This is left to the second phase, which happens where check_btf_info executes currently, and performs the line_info and CO-RE relocation. The reason to perform this split is to obtain the userspace supplied func_info information before we perform the add_subprog call, where we would now require finding and adding subprogs that may not have a bpf_pseudo_call or bpf_pseudo_func instruction in the program. We require this as we want to enable userspace to supply exception callbacks that can override the default hidden subprogram generated by the verifier (which performs a hardcoded action). In such a case, the exception callback may never be referenced in an instruction, but will still be suitably annotated (by way of BTF declaration tags). For finding this exception callback, we would require the program's BTF information, and the supplied func_info information which maps BTF type IDs to subprograms. Since the exception callback won't actually be referenced through instructions, later checks in check_cfg and do_check_subprogs will not verify the subprog. This means that add_subprog needs to add them in the add_subprog_and_kfunc phase before we move forward, which is why the BTF and func_info are required at that point. Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 128 +++++++++++++++++++++++++++++++++--------- 1 file changed, 100 insertions(+), 28 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 9baa6f187b38..ec767ae08c2b 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -15115,20 +15115,18 @@ static int check_abnormal_return(struct bpf_verifier_env *env) #define MIN_BPF_FUNCINFO_SIZE 8 #define MAX_FUNCINFO_REC_SIZE 252 -static int check_btf_func(struct bpf_verifier_env *env, - const union bpf_attr *attr, - bpfptr_t uattr) +static int check_btf_func_early(struct bpf_verifier_env *env, + const union bpf_attr *attr, + bpfptr_t uattr) { - const struct btf_type *type, *func_proto, *ret_type; - u32 i, nfuncs, urec_size, min_size; u32 krec_size = sizeof(struct bpf_func_info); + const struct btf_type *type, *func_proto; + u32 i, nfuncs, urec_size, min_size; struct bpf_func_info *krecord; - struct bpf_func_info_aux *info_aux = NULL; struct bpf_prog *prog; const struct btf *btf; - bpfptr_t urecord; u32 prev_offset = 0; - bool scalar_return; + bpfptr_t urecord; int ret = -ENOMEM; nfuncs = attr->func_info_cnt; @@ -15138,11 +15136,6 @@ static int check_btf_func(struct bpf_verifier_env *env, return 0; } - if (nfuncs != env->subprog_cnt) { - verbose(env, "number of funcs in func_info doesn't match number of subprogs\n"); - return -EINVAL; - } - urec_size = attr->func_info_rec_size; if (urec_size < MIN_BPF_FUNCINFO_SIZE || urec_size > MAX_FUNCINFO_REC_SIZE || @@ -15160,9 +15153,6 @@ static int check_btf_func(struct bpf_verifier_env *env, krecord = kvcalloc(nfuncs, krec_size, GFP_KERNEL | __GFP_NOWARN); if (!krecord) return -ENOMEM; - info_aux = kcalloc(nfuncs, sizeof(*info_aux), GFP_KERNEL | __GFP_NOWARN); - if (!info_aux) - goto err_free; for (i = 0; i < nfuncs; i++) { ret = bpf_check_uarg_tail_zero(urecord, krec_size, urec_size); @@ -15201,11 +15191,6 @@ static int check_btf_func(struct bpf_verifier_env *env, goto err_free; } - if (env->subprog_info[i].start != krecord[i].insn_off) { - verbose(env, "func_info BTF section doesn't match subprog layout in BPF program\n"); - goto err_free; - } - /* check type_id */ type = btf_type_by_id(btf, krecord[i].type_id); if (!type || !btf_type_is_func(type)) { @@ -15213,12 +15198,80 @@ static int check_btf_func(struct bpf_verifier_env *env, krecord[i].type_id); goto err_free; } - info_aux[i].linkage = BTF_INFO_VLEN(type->info); func_proto = btf_type_by_id(btf, type->type); if (unlikely(!func_proto || !btf_type_is_func_proto(func_proto))) /* btf_func_check() already verified it during BTF load */ goto err_free; + + prev_offset = krecord[i].insn_off; + bpfptr_add(&urecord, urec_size); + } + + prog->aux->func_info = krecord; + prog->aux->func_info_cnt = nfuncs; + return 0; + +err_free: + kvfree(krecord); + return ret; +} + +static int check_btf_func(struct bpf_verifier_env *env, + const union bpf_attr *attr, + bpfptr_t uattr) +{ + const struct btf_type *type, *func_proto, *ret_type; + u32 i, nfuncs, urec_size, min_size; + u32 krec_size = sizeof(struct bpf_func_info); + struct bpf_func_info *krecord; + struct bpf_func_info_aux *info_aux = NULL; + struct bpf_prog *prog; + const struct btf *btf; + bpfptr_t urecord; + u32 prev_offset = 0; + bool scalar_return; + int ret = -ENOMEM; + + nfuncs = attr->func_info_cnt; + if (!nfuncs) { + if (check_abnormal_return(env)) + return -EINVAL; + return 0; + } + if (nfuncs != env->subprog_cnt) { + verbose(env, "number of funcs in func_info doesn't match number of subprogs\n"); + return -EINVAL; + } + + urec_size = attr->func_info_rec_size; + + prog = env->prog; + btf = prog->aux->btf; + + urecord = make_bpfptr(attr->func_info, uattr.is_kernel); + min_size = min_t(u32, krec_size, urec_size); + + krecord = prog->aux->func_info; + info_aux = kcalloc(nfuncs, sizeof(*info_aux), GFP_KERNEL | __GFP_NOWARN); + if (!info_aux) + return -ENOMEM; + + for (i = 0; i < nfuncs; i++) { + /* check insn_off */ + ret = -EINVAL; + + if (env->subprog_info[i].start != krecord[i].insn_off) { + verbose(env, "func_info BTF section doesn't match subprog layout in BPF program\n"); + goto err_free; + } + + /* Already checked type_id */ + type = btf_type_by_id(btf, krecord[i].type_id); + info_aux[i].linkage = BTF_INFO_VLEN(type->info); + /* Already checked func_proto */ + func_proto = btf_type_by_id(btf, type->type); + ret_type = btf_type_skip_modifiers(btf, func_proto->type, NULL); scalar_return = btf_type_is_small_int(ret_type) || btf_is_any_enum(ret_type); @@ -15235,13 +15288,10 @@ static int check_btf_func(struct bpf_verifier_env *env, bpfptr_add(&urecord, urec_size); } - prog->aux->func_info = krecord; - prog->aux->func_info_cnt = nfuncs; prog->aux->func_info_aux = info_aux; return 0; err_free: - kvfree(krecord); kfree(info_aux); return ret; } @@ -15459,9 +15509,9 @@ static int check_core_relo(struct bpf_verifier_env *env, return err; } -static int check_btf_info(struct bpf_verifier_env *env, - const union bpf_attr *attr, - bpfptr_t uattr) +static int check_btf_info_early(struct bpf_verifier_env *env, + const union bpf_attr *attr, + bpfptr_t uattr) { struct btf *btf; int err; @@ -15481,6 +15531,24 @@ static int check_btf_info(struct bpf_verifier_env *env, } env->prog->aux->btf = btf; + err = check_btf_func_early(env, attr, uattr); + if (err) + return err; + return 0; +} + +static int check_btf_info(struct bpf_verifier_env *env, + const union bpf_attr *attr, + bpfptr_t uattr) +{ + int err; + + if (!attr->func_info_cnt && !attr->line_info_cnt) { + if (check_abnormal_return(env)) + return -EINVAL; + return 0; + } + err = check_btf_func(env, attr, uattr); if (err) return err; @@ -19990,6 +20058,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3 if (!env->explored_states) goto skip_full_check; + ret = check_btf_info_early(env, attr, uattr); + if (ret < 0) + goto skip_full_check; + ret = add_subprog_and_kfunc(env); if (ret < 0) goto skip_full_check; From patchwork Tue Sep 12 23:32:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13382311 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EA5381429F for ; Tue, 12 Sep 2023 23:32:22 +0000 (UTC) Received: from mail-ed1-x544.google.com (mail-ed1-x544.google.com [IPv6:2a00:1450:4864:20::544]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3C59F10FE for ; Tue, 12 Sep 2023 16:32:22 -0700 (PDT) Received: by mail-ed1-x544.google.com with SMTP id 4fb4d7f45d1cf-52eed139ec2so6313590a12.2 for ; Tue, 12 Sep 2023 16:32:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1694561540; x=1695166340; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1WLolWLG60Kjojm7uamT/JxAmYWhvrYdhW6u3XhIoUw=; b=IUpjBrA2XokbgFi7RspmLRH9dJxDgq/d+GKxuTtgiHixCdIE2NyCcJphkGLbm1kREV vACXZ5HRYILSMjAPt0jfLgE2e66R4199m2bOhwpKHWv56kHOvCMPHQcqTG6t7ueseZ/8 HmtQWCD9kvDjy8Zjk/BLEtAwTJ/h7Q5aeMW2RYWhIcRH8x/mYiNiEngitKwDjbpSU1C5 AL5ip070xwNBvMDB16+5RIcYAYX2Xc87KaCcWZD5xcOdful2Dpfqm4m8t+MIXV0TXxJM +zvpb2eCHTMxnbCzImtjLkTrucSD9OBxEhd5Nh9sD1v4HCzZ/oOOeCmyZqODQw3SQ0LY 689w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694561540; x=1695166340; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1WLolWLG60Kjojm7uamT/JxAmYWhvrYdhW6u3XhIoUw=; b=KZFDDt17aNtb5Tpw2AYroXIUOZsKQdsesurU+G8aFIHwfiRSXuy7sxbTERnoBXpA4+ DWeyn7evC7C6HqTQ5ZCPU/6aXYhDlJ80JiW2+q1ff3cHj4g8eYPAPOYxQ6oPC28FGF/l lYd+FhXGhNE8shp8fPRzt5Ut2cQBqmQE740/9AlsWlzQ5KNg6vOuuEa6TvWKPC7wjPd4 guSjxnR3nOuaKxbL9VHX5aavMPIvP2xJls7h0HKfysJl6IHPuKAHZckklcuY9F7gA2Tt 5KkvkgIFZSk5xQWVjPTwEYEX2m+6Zz6oLXUANp47Jxpmm4KIgTZ1y3J8SP+XIb3KDRK7 JvDQ== X-Gm-Message-State: AOJu0YwymtttjktR/EQOE2m3or4uL7LL4jgBO5yAZdlbGuils/UObrX/ BECCgfcD3UT336pWrv5nBtW/54PEnnCQ0g== X-Google-Smtp-Source: AGHT+IEHAuFU2vLP9Tbx7iPEmYWHdrrWlMQ3HPmCUsKMW2km2m3f2Ytts9JIXC13edVINCmKohainw== X-Received: by 2002:a17:906:d8:b0:9ad:8ab4:523d with SMTP id 24-20020a17090600d800b009ad8ab4523dmr526618eji.69.1694561540470; Tue, 12 Sep 2023 16:32:20 -0700 (PDT) Received: from localhost ([212.203.98.42]) by smtp.gmail.com with ESMTPSA id oy25-20020a170907105900b0099d0c0bb92bsm7430493ejb.80.2023.09.12.16.32.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 16:32:20 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet , Puranjay Mohan Subject: [PATCH bpf-next v3 06/17] bpf: Add support for custom exception callbacks Date: Wed, 13 Sep 2023 01:32:03 +0200 Message-ID: <20230912233214.1518551-7-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230912233214.1518551-1-memxor@gmail.com> References: <20230912233214.1518551-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=14967; i=memxor@gmail.com; h=from:subject; bh=TDfzw634HmkH1qVCVePwzkM2jE1F2uiZx1L4alpQjTM=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBlAPSsL7t77AucDPyTTLAKmcVekU7OJFloBpBK7 VZvoOFgh1+JAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZQD0rAAKCRBM4MiGSL8R ysemEACNPNAYCB2cQN4hROiLygjE42W9w/aqhRPKy8Ww3cJBHiOzPVZtKezS60WBA5omivAKIcp QXwwUszS4XP52QnhZeazOyQF7+h3Q7FYx4o32WvsvqI/RxW/UGr1JH8rdE0FkaJPSag8hGvY75D yq46pF21tm//Wclsfd+bge00k7nnPIkGM3RziuJzbOcuwRYJVdsf41NdRjxXCJA1vgztgp9Qz7H jOi4X4Z6AUuf4lRVQi0PzZnl7hdZtjiPvs6E2blf7ztlQRppYXwww0u+8FSNm78cOjqcS1WHb4U Gk4vq0wnNsPahuuqrst6zGbDiLzQpka2FkTsplwmsibg1ubYsdqn6/eI+pUFRKanxYEIKCHZkMt dh73ZNfLOhKW5/+rAe89rzrxNzKSR0zb1Llk1UhbECL/sQeW91gWVoR9kEqc6g5Fzvc1oBJOhOg 6PiVaCHhSdl8xejym+xKimzA+oullUOPh98Kl7MbdFhK24cpIVVhGxmiL6YqfrkkAFkVujEDMHq YO2NIQl0YGnb6+UemTQ8gDF6T1zBrFJnUJUFTPNEU++/G0tLQ4hFlyZlbrOm9X3E76gfSypMe/v z0RqfFFelcddrhhvW5w2eXjReubXqS3u4WsDgnfWiTXBZpLyxfd15UFbtzzBEXtAZ075clPFVA5 4XAGYJxYM+REDPQ== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net By default, the subprog generated by the verifier to handle a thrown exception hardcodes a return value of 0. To allow user-defined logic and modification of the return value when an exception is thrown, introduce the 'exception_callback:' declaration tag, which marks a callback as the default exception handler for the program. The format of the declaration tag is 'exception_callback:', where is the name of the exception callback. Each main program can be tagged using this BTF declaratiion tag to associate it with an exception callback. In case the tag is absent, the default callback is used. As such, the exception callback cannot be modified at runtime, only set during verification. Allowing modification of the callback for the current program execution at runtime leads to issues when the programs begin to nest, as any per-CPU state maintaing this information will have to be saved and restored. We don't want it to stay in bpf_prog_aux as this takes a global effect for all programs. An alternative solution is spilling the callback pointer at a known location on the program stack on entry, and then passing this location to bpf_throw as a parameter. However, since exceptions are geared more towards a use case where they are ideally never invoked, optimizing for this use case and adding to the complexity has diminishing returns. Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf.h | 4 +- include/linux/bpf_verifier.h | 1 + kernel/bpf/btf.c | 29 +++-- kernel/bpf/verifier.c | 113 ++++++++++++++++-- .../testing/selftests/bpf/bpf_experimental.h | 31 ++++- 5 files changed, 160 insertions(+), 18 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 16740ee82082..30063a760b5a 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -2422,9 +2422,11 @@ int btf_check_subprog_arg_match(struct bpf_verifier_env *env, int subprog, int btf_check_subprog_call(struct bpf_verifier_env *env, int subprog, struct bpf_reg_state *regs); int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog, - struct bpf_reg_state *reg); + struct bpf_reg_state *reg, bool is_ex_cb); int btf_check_type_match(struct bpf_verifier_log *log, const struct bpf_prog *prog, struct btf *btf, const struct btf_type *t); +const char *btf_find_decl_tag_value(const struct btf *btf, const struct btf_type *pt, + int comp_idx, const char *tag_key); struct bpf_prog *bpf_prog_by_id(u32 id); struct bpf_link *bpf_link_by_id(u32 id); diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index da21a3ec5027..94ec766432f5 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -300,6 +300,7 @@ struct bpf_func_state { bool in_callback_fn; struct tnum callback_ret_range; bool in_async_callback_fn; + bool in_exception_callback_fn; /* The following fields should be last. See copy_func_state() */ int acquired_refs; diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 187b57276fec..f93e835d90af 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -3310,10 +3310,10 @@ static int btf_find_kptr(const struct btf *btf, const struct btf_type *t, return BTF_FIELD_FOUND; } -static const char *btf_find_decl_tag_value(const struct btf *btf, - const struct btf_type *pt, - int comp_idx, const char *tag_key) +const char *btf_find_decl_tag_value(const struct btf *btf, const struct btf_type *pt, + int comp_idx, const char *tag_key) { + const char *value = NULL; int i; for (i = 1; i < btf_nr_types(btf); i++) { @@ -3327,9 +3327,14 @@ static const char *btf_find_decl_tag_value(const struct btf *btf, continue; if (strncmp(__btf_name_by_offset(btf, t->name_off), tag_key, len)) continue; - return __btf_name_by_offset(btf, t->name_off) + len; + /* Prevent duplicate entries for same type */ + if (value) + return ERR_PTR(-EEXIST); + value = __btf_name_by_offset(btf, t->name_off) + len; } - return NULL; + if (!value) + return ERR_PTR(-ENOENT); + return value; } static int @@ -3347,7 +3352,7 @@ btf_find_graph_root(const struct btf *btf, const struct btf_type *pt, if (t->size != sz) return BTF_FIELD_IGNORE; value_type = btf_find_decl_tag_value(btf, pt, comp_idx, "contains:"); - if (!value_type) + if (IS_ERR(value_type)) return -EINVAL; node_field_name = strstr(value_type, ":"); if (!node_field_name) @@ -6954,7 +6959,7 @@ int btf_check_subprog_call(struct bpf_verifier_env *env, int subprog, * (either PTR_TO_CTX or SCALAR_VALUE). */ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog, - struct bpf_reg_state *regs) + struct bpf_reg_state *regs, bool is_ex_cb) { struct bpf_verifier_log *log = &env->log; struct bpf_prog *prog = env->prog; @@ -7011,7 +7016,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog, tname, nargs, MAX_BPF_FUNC_REG_ARGS); return -EINVAL; } - /* check that function returns int */ + /* check that function returns int, exception cb also requires this */ t = btf_type_by_id(btf, t->type); while (btf_type_is_modifier(t)) t = btf_type_by_id(btf, t->type); @@ -7060,6 +7065,14 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog, i, btf_type_str(t), tname); return -EINVAL; } + /* We have already ensured that the callback returns an integer, just + * like all global subprogs. We need to determine it only has a single + * scalar argument. + */ + if (is_ex_cb && (nargs != 1 || regs[BPF_REG_1].type != SCALAR_VALUE)) { + bpf_log(log, "exception cb only supports single integer argument\n"); + return -EINVAL; + } return 0; } diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index ec767ae08c2b..ec3f22312516 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -2457,6 +2457,68 @@ static int add_subprog(struct bpf_verifier_env *env, int off) return env->subprog_cnt - 1; } +static int bpf_find_exception_callback_insn_off(struct bpf_verifier_env *env) +{ + struct bpf_prog_aux *aux = env->prog->aux; + struct btf *btf = aux->btf; + const struct btf_type *t; + u32 main_btf_id, id; + const char *name; + int ret, i; + + /* Non-zero func_info_cnt implies valid btf */ + if (!aux->func_info_cnt) + return 0; + main_btf_id = aux->func_info[0].type_id; + + t = btf_type_by_id(btf, main_btf_id); + if (!t) { + verbose(env, "invalid btf id for main subprog in func_info\n"); + return -EINVAL; + } + + name = btf_find_decl_tag_value(btf, t, -1, "exception_callback:"); + if (IS_ERR(name)) { + ret = PTR_ERR(name); + /* If there is no tag present, there is no exception callback */ + if (ret == -ENOENT) + ret = 0; + else if (ret == -EEXIST) + verbose(env, "multiple exception callback tags for main subprog\n"); + return ret; + } + + ret = btf_find_by_name_kind(btf, name, BTF_KIND_FUNC); + if (ret < 0) { + verbose(env, "exception callback '%s' could not be found in BTF\n", name); + return ret; + } + id = ret; + t = btf_type_by_id(btf, id); + if (btf_func_linkage(t) != BTF_FUNC_GLOBAL) { + verbose(env, "exception callback '%s' must have global linkage\n", name); + return -EINVAL; + } + ret = 0; + for (i = 0; i < aux->func_info_cnt; i++) { + if (aux->func_info[i].type_id != id) + continue; + ret = aux->func_info[i].insn_off; + /* Further func_info and subprog checks will also happen + * later, so assume this is the right insn_off for now. + */ + if (!ret) { + verbose(env, "invalid exception callback insn_off in func_info: 0\n"); + ret = -EINVAL; + } + } + if (!ret) { + verbose(env, "exception callback type id not found in func_info\n"); + ret = -EINVAL; + } + return ret; +} + #define MAX_KFUNC_DESCS 256 #define MAX_KFUNC_BTFS 256 @@ -2796,8 +2858,8 @@ bpf_jit_find_kfunc_model(const struct bpf_prog *prog, static int add_subprog_and_kfunc(struct bpf_verifier_env *env) { struct bpf_subprog_info *subprog = env->subprog_info; + int i, ret, insn_cnt = env->prog->len, ex_cb_insn; struct bpf_insn *insn = env->prog->insnsi; - int i, ret, insn_cnt = env->prog->len; /* Add entry function. */ ret = add_subprog(env, 0); @@ -2823,6 +2885,26 @@ static int add_subprog_and_kfunc(struct bpf_verifier_env *env) return ret; } + ret = bpf_find_exception_callback_insn_off(env); + if (ret < 0) + return ret; + ex_cb_insn = ret; + + /* If ex_cb_insn > 0, this means that the main program has a subprog + * marked using BTF decl tag to serve as the exception callback. + */ + if (ex_cb_insn) { + ret = add_subprog(env, ex_cb_insn); + if (ret < 0) + return ret; + for (i = 1; i < env->subprog_cnt; i++) { + if (env->subprog_info[i].start != ex_cb_insn) + continue; + env->exception_callback_subprog = i; + break; + } + } + /* Add a fake 'exit' subprog which could simplify subprog iteration * logic. 'subprog_cnt' should not be increased. */ @@ -5707,6 +5789,10 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx) /* async callbacks don't increase bpf prog stack size unless called directly */ if (!bpf_pseudo_call(insn + i)) continue; + if (subprog[sidx].is_exception_cb) { + verbose(env, "insn %d cannot call exception cb directly\n", i); + return -EINVAL; + } } i = next_insn; idx = sidx; @@ -5728,8 +5814,13 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx) * tail call counter throughout bpf2bpf calls combined with tailcalls */ if (tail_call_reachable) - for (j = 0; j < frame; j++) + for (j = 0; j < frame; j++) { + if (subprog[ret_prog[j]].is_exception_cb) { + verbose(env, "cannot tail call within exception cb\n"); + return -EINVAL; + } subprog[ret_prog[j]].tail_call_reachable = true; + } if (subprog[0].tail_call_reachable) env->prog->aux->tail_call_reachable = true; @@ -14630,7 +14721,7 @@ static int check_return_code(struct bpf_verifier_env *env) const bool is_subprog = frame->subprogno; /* LSM and struct_ops func-ptr's return type could be "void" */ - if (!is_subprog) { + if (!is_subprog || frame->in_exception_callback_fn) { switch (prog_type) { case BPF_PROG_TYPE_LSM: if (prog->expected_attach_type == BPF_LSM_CGROUP) @@ -14678,7 +14769,7 @@ static int check_return_code(struct bpf_verifier_env *env) return 0; } - if (is_subprog) { + if (is_subprog && !frame->in_exception_callback_fn) { if (reg->type != SCALAR_VALUE) { verbose(env, "At subprogram exit the register R0 is not a scalar value (%s)\n", reg_type_str(env, reg->type)); @@ -19334,7 +19425,7 @@ static void free_states(struct bpf_verifier_env *env) } } -static int do_check_common(struct bpf_verifier_env *env, int subprog) +static int do_check_common(struct bpf_verifier_env *env, int subprog, bool is_ex_cb) { bool pop_log = !(env->log.level & BPF_LOG_LEVEL2); struct bpf_verifier_state *state; @@ -19365,7 +19456,7 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog) regs = state->frame[state->curframe]->regs; if (subprog || env->prog->type == BPF_PROG_TYPE_EXT) { - ret = btf_prepare_func_args(env, subprog, regs); + ret = btf_prepare_func_args(env, subprog, regs, is_ex_cb); if (ret) goto out; for (i = BPF_REG_1; i <= BPF_REG_5; i++) { @@ -19381,6 +19472,12 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog) regs[i].id = ++env->id_gen; } } + if (is_ex_cb) { + state->frame[0]->in_exception_callback_fn = true; + env->subprog_info[subprog].is_cb = true; + env->subprog_info[subprog].is_async_cb = true; + env->subprog_info[subprog].is_exception_cb = true; + } } else { /* 1st arg to a function */ regs[BPF_REG_1].type = PTR_TO_CTX; @@ -19445,7 +19542,7 @@ static int do_check_subprogs(struct bpf_verifier_env *env) continue; env->insn_idx = env->subprog_info[i].start; WARN_ON_ONCE(env->insn_idx == 0); - ret = do_check_common(env, i); + ret = do_check_common(env, i, env->exception_callback_subprog == i); if (ret) { return ret; } else if (env->log.level & BPF_LOG_LEVEL) { @@ -19462,7 +19559,7 @@ static int do_check_main(struct bpf_verifier_env *env) int ret; env->insn_idx = 0; - ret = do_check_common(env, 0); + ret = do_check_common(env, 0, false); if (!ret) env->prog->aux->stack_depth = env->subprog_info[0].stack_depth; return ret; diff --git a/tools/testing/selftests/bpf/bpf_experimental.h b/tools/testing/selftests/bpf/bpf_experimental.h index 333b54a86e3a..9a87170524ce 100644 --- a/tools/testing/selftests/bpf/bpf_experimental.h +++ b/tools/testing/selftests/bpf/bpf_experimental.h @@ -165,7 +165,16 @@ extern void bpf_percpu_obj_drop_impl(void *kptr, void *meta) __ksym; /* Description * Throw a BPF exception from the program, immediately terminating its * execution and unwinding the stack. The supplied 'cookie' parameter - * will be the return value of the program when an exception is thrown. + * will be the return value of the program when an exception is thrown, + * and the default exception callback is used. Otherwise, if an exception + * callback is set using the '__exception_cb(callback)' declaration tag + * on the main program, the 'cookie' parameter will be the callback's only + * input argument. + * + * Thus, in case of default exception callback, 'cookie' is subjected to + * constraints on the program's return value (as with R0 on exit). + * Otherwise, the return value of the marked exception callback will be + * subjected to the same checks. * * Note that throwing an exception with lingering resources (locks, * references, etc.) will lead to a verification error. @@ -178,4 +187,24 @@ extern void bpf_percpu_obj_drop_impl(void *kptr, void *meta) __ksym; */ extern void bpf_throw(u64 cookie) __ksym; +/* This macro must be used to mark the exception callback corresponding to the + * main program. For example: + * + * int exception_cb(u64 cookie) { + * return cookie; + * } + * + * SEC("tc") + * __exception_cb(exception_cb) + * int main_prog(struct __sk_buff *ctx) { + * ... + * return TC_ACT_OK; + * } + * + * Here, exception callback for the main program will be 'exception_cb'. Note + * that this attribute can only be used once, and multiple exception callbacks + * specified for the main program will lead to verification error. + */ +#define __exception_cb(name) __attribute__((btf_decl_tag("exception_callback:" #name))) + #endif From patchwork Tue Sep 12 23:32:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13382312 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EDE001429F for ; Tue, 12 Sep 2023 23:32:23 +0000 (UTC) Received: from mail-lj1-x241.google.com (mail-lj1-x241.google.com [IPv6:2a00:1450:4864:20::241]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3720310FE for ; Tue, 12 Sep 2023 16:32:23 -0700 (PDT) Received: by mail-lj1-x241.google.com with SMTP id 38308e7fff4ca-2bfb1167277so9828191fa.2 for ; Tue, 12 Sep 2023 16:32:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1694561541; x=1695166341; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=K6fOZXSAiZjm2cSMS2cSkgEH3WnXEVvnoYLQHmSdkEw=; b=FX2IXR1mgJSexOYP8d1N0iCT87w8vgnsS9tKFxvma+2b9vNL++NVmak7A4Ssg6D8RP ZihGOT9Uy5dEj4YyhEFFZ+EM1qYdiqdyhKAx7zeqX4xDQGnKUy5gm90gLIsXwUyNZpQ1 wMjLEMW0fsktwFpB69kte6/yVqql4qjZjZH1f17h7uizyRiqKoZWKiBin8dTqnunsh46 y0Ighm+kdsl42MeQ1usIksXWOcLEUxH24TvqChegZvyZM9Nnr8/t7BnANuWDY5qo8db2 Bx24/XJT5f9MeEBetVrSEFLcknCvwCOLssITeKtJ6BelaRF5czn9mQ/p0AfBGJhnqv49 TqJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694561541; x=1695166341; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=K6fOZXSAiZjm2cSMS2cSkgEH3WnXEVvnoYLQHmSdkEw=; b=Y45E/v8f9N6iEJrLDYN7iTJFtcjdCngfAdNITHXH4jC/sWV6JOnBQiSuE1Zf/XSPam NbDpxTjHsROOWgXfpb82oOPZDIYJ1peY7vyIf/HtkAz9PcJd5PC3+75X1uvgOAQJT2G2 tDcX2YSUp6HO4AkLwp2MJl95MPlTbTgNnBEHluhDtLNtZagddKVNT5toH5mk6whhi9vB sVK6MLwlVR/sXPdBZfXSpghgJlZbmSObLoWExpfVBced67Se4MP3raXuKr1R2dxt+OhQ CI5rITGfqmi/pecpQaMJWEX+XP20Quu4S07Q0x/Eycd+nlrcIuXOkrUt6RgbzBYUctqh P2Yg== X-Gm-Message-State: AOJu0YwmKQVBVhVlL/TIZRsFWFuO3pxcP9vTa2AwFAJWB8gXIz/z4X0j 4v96sXE08t68Iyb2VhVUFxI0Z2HCuo0fDA== X-Google-Smtp-Source: AGHT+IECRtOEGE6Pc4SrptqysMKVfxW4SXncDzXfaMr18UQjetnzVOj3yQA29KotoHZLMLW8H5xPMw== X-Received: by 2002:a2e:8206:0:b0:2b9:ee3e:2412 with SMTP id w6-20020a2e8206000000b002b9ee3e2412mr923831ljg.22.1694561541217; Tue, 12 Sep 2023 16:32:21 -0700 (PDT) Received: from localhost ([212.203.98.42]) by smtp.gmail.com with ESMTPSA id oz13-20020a170906cd0d00b0098951bb4dc3sm7446788ejb.184.2023.09.12.16.32.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 16:32:21 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet , Puranjay Mohan Subject: [PATCH bpf-next v3 07/17] bpf: Perform CFG walk for exception callback Date: Wed, 13 Sep 2023 01:32:04 +0200 Message-ID: <20230912233214.1518551-8-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230912233214.1518551-1-memxor@gmail.com> References: <20230912233214.1518551-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1909; i=memxor@gmail.com; h=from:subject; bh=KMEC7C7g+3JkHUGpCQErdFOWVieZ1zfyS4CL+h7sjbw=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBlAPStFiDWp5kvthNHypcBbyK+NuFN02qSh4BXx oi/B9OqD1uJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZQD0rQAKCRBM4MiGSL8R yqdzD/4vQmvKR2caqfDxCdUhOpuWr03ecOG+K5gkL9Lrk0IJ8fHHYzKR0r26E6D2EODlxC0aNlF uWWkFhDaFppbgRVw4lREov7qksVtG0cVGij+Hjg1VWw/yVD9ebQanP3IcdHAQhJu/LqsxVNyHXp PLQIaDrYBsiRwQG4Lsthn7xZcmxN66EN0gn1kDmnKiNJAiMk6h9+IANwKHnbPyC3JNqvofDInDq o+QVra8QRPCyZvqoE1/ucxQyG1O+WFNLxDMEtAb7g522nP+p7TLaVH3JYCfT9oCGPjLEi1ABTNU de/RRxOyf6Gczmy8ZR5tFebC2+TqR1j42SAYZ4yapYBkK/0cL0Kr3ZeWkCFaPe6GxzY/kziwvvz aI/PGGsFjWEAqsmb0Zi2BvX4wSu4M93/rFi/c2FXaHpRXTSWUw5m1stho5nIJ1/cE8yRwi9Iozj HYkPfYKT1DTx1oKWsFXCf6k/qGpq2xedj00Y6kED7sXhfTLphbSwEVbVrcbPWHpjBMDi5MRV2I7 SQhpo6S5bUkt78FwdUtIK/qJUfJDWbvf0rwaBcvQ0/TYUUl6lry/D1JXk2crARZVHfKMJ0+guj/ Qs7EDYJGO00OfkVhz36ULg044eh6+TLp57GiUq4uwFe3NyFlvDgepaVaRv8lFB+nGwwGMRVj9fw FGhGSRo0YOF26Gw== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net Since exception callbacks are not referenced using bpf_pseudo_func and bpf_pseudo_call instructions, check_cfg traversal will never explore instructions of the exception callback. Even after adding the subprog, the program will then fail with a 'unreachable insn' error. We thus need to begin walking from the start of the exception callback again in check_cfg after a complete CFG traversal finishes, so as to explore the CFG rooted at the exception callback. Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index ec3f22312516..863e4e6c4616 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -15126,8 +15126,8 @@ static int check_cfg(struct bpf_verifier_env *env) { int insn_cnt = env->prog->len; int *insn_stack, *insn_state; - int ret = 0; - int i; + int ex_insn_beg, i, ret = 0; + bool ex_done = false; insn_state = env->cfg.insn_state = kvcalloc(insn_cnt, sizeof(int), GFP_KERNEL); if (!insn_state) @@ -15143,6 +15143,7 @@ static int check_cfg(struct bpf_verifier_env *env) insn_stack[0] = 0; /* 0 is the first instruction */ env->cfg.cur_stack = 1; +walk_cfg: while (env->cfg.cur_stack > 0) { int t = insn_stack[env->cfg.cur_stack - 1]; @@ -15169,6 +15170,16 @@ static int check_cfg(struct bpf_verifier_env *env) goto err_free; } + if (env->exception_callback_subprog && !ex_done) { + ex_insn_beg = env->subprog_info[env->exception_callback_subprog].start; + + insn_state[ex_insn_beg] = DISCOVERED; + insn_stack[0] = ex_insn_beg; + env->cfg.cur_stack = 1; + ex_done = true; + goto walk_cfg; + } + for (i = 0; i < insn_cnt; i++) { if (insn_state[i] != EXPLORED) { verbose(env, "unreachable insn %d\n", i); From patchwork Tue Sep 12 23:32:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13382313 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A71A31429F for ; Tue, 12 Sep 2023 23:32:24 +0000 (UTC) Received: from mail-ed1-x543.google.com (mail-ed1-x543.google.com [IPv6:2a00:1450:4864:20::543]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0E02510FE for ; Tue, 12 Sep 2023 16:32:24 -0700 (PDT) Received: by mail-ed1-x543.google.com with SMTP id 4fb4d7f45d1cf-52f33659d09so4426740a12.1 for ; Tue, 12 Sep 2023 16:32:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1694561542; x=1695166342; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DEqNKEyyn2bquryM443pcbyNh0rQI9bn3pNb7P20yv0=; b=oaOz9eSie9Xm1ywgaVUbg2t2qcY7Jry0fUSl8oWKDbavVvLA3HdvO0xIdFbRXfzaIo ti4I3jngnFG7h59XYVgPwRP6TqANLa7/cbUNkr4YTGMs+8R3E1huwP/uXya7SWbkvD9S PJivGlPK+cDLNi2BzJMCi2PIIglCb703D4mczNcbBtd1x1TLVIb7WzI/ThKDX0jgfkpm RMhcm0cWJ31RyFBmIKvx9BLyPfixf2bPpG81CPJ8XUQ1X5q07RwpNYqW6crVp5EJzMhq 5lCBCQjbPSEq6TTi/DjmQiS1FwKYjOeRaXd6z+dP0sK777qSdFHd4TkZojXhGyK54kUs s+bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694561542; x=1695166342; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DEqNKEyyn2bquryM443pcbyNh0rQI9bn3pNb7P20yv0=; b=V0RW/7QGOmrGRKqbq5LaRewBjQ+lrXrwpigonS0OwQbK6gK2cvXbfgSPF5GWaxG7RE vlEzoebK4ROQzxzb1GvVrLeKqNd8tEIFTv+mfcXMFUT7mrI3z9mVpdvc2YdD8q/y742D ykTglRJRpvqK75cdP/KpqfaXnVreImsYSZhHX4pyPZ0d8cJGkeXNmLuwfUY96bSOs4KE JSN9PcAS8vELOcSO4I6ap2zDmpfSsqOZAZZB1hJ8KjkkM3/fU1BGR/dE6yafetK10avq H3qPQCDu1BedP02GCtNWtxH0yopnGrpTNVuaUo4M3zGjQl2zyqLwu5IBuuYk60+jO1lH l+2w== X-Gm-Message-State: AOJu0Yz0wb2qBYyxZMgcM4PSine9vqieJiIm1NpJ33NJDqvaQEx3lhB6 Iun9p/VEmEdxn1looYbk3eA7CHEAgAC9nQ== X-Google-Smtp-Source: AGHT+IEjuI+DgEaQ5vfu0OMk+2aXX3NkLWQ46JOTGJYDiVITOt0CuonVNB/qRFZr3f8nEDYdwTUXbQ== X-Received: by 2002:a50:fb89:0:b0:52b:daff:f6e9 with SMTP id e9-20020a50fb89000000b0052bdafff6e9mr809317edq.17.1694561542086; Tue, 12 Sep 2023 16:32:22 -0700 (PDT) Received: from localhost ([212.203.98.42]) by smtp.gmail.com with ESMTPSA id eq18-20020a056402299200b0052fe2faf44fsm10411edb.75.2023.09.12.16.32.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 16:32:21 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet , Puranjay Mohan Subject: [PATCH bpf-next v3 08/17] bpf: Treat first argument as return value for bpf_throw Date: Wed, 13 Sep 2023 01:32:05 +0200 Message-ID: <20230912233214.1518551-9-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230912233214.1518551-1-memxor@gmail.com> References: <20230912233214.1518551-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4675; i=memxor@gmail.com; h=from:subject; bh=NYrdPgP3ltPdMBCETLJI7uiGC1Lo4GXt4r58HXYpCNQ=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBlAPSty5CANEROfhpiVp2x8LTYd+djg+BPI4cGM /Fln6GI34GJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZQD0rQAKCRBM4MiGSL8R ygrxEAClHTzrnHzCLB9+1DynctsmmYCg7U0t27lQNc16RbEtYmrI23s0xg2jndvJ+uQ35q8G5W0 cwH1N2I3C59dQIbpWhTYV1WqB2DzUIBQ9k28CqZVA7Zkx+dWDah1NYNOXHgoUfbCCu3qoyVP3rh RdSHMHGp4YsaKdj4pimUtOx3XTgSXxD41YL/j9+B37KJV73wLKARb0pXtoUdJV1Zg2Ntbc6+RR6 et8pi+ywUOaQ80WvF1rtQtQkoTKxaeNVvhn8b61LgHOCXyabzWkYEuLTRr7RHx2cXCUnADArGvg B/OHzE272JRCHDn/O5FiAiMxPPHBKdttGEuDgnuHZ1iEiUoKBBnBGzEt2V+V8CJEsJ2l4/aZQgI WU46IjQMLOxYp5K54sJmTtg5TNO5URNucZeObdSf0pAnAVFHz6XI24j5C1OHQAAx5hWuH77KejZ Ji+huxJ3b1LanwUrM6bXWbhFNqxlYGyzvYVevWUGHQpHzINsEhoWC7WPpfggfTf1rW5XL+NyuaB DdZiW4hgP23ukbgbmJOs6jpfAuNftExBRTeMZaa5jFLmq+SX1fYHJluvQBLaJlCLw3ogT3ODoQ5 EMYbyrDCvQw71l+1lbm79DF/lvplEVnlFJSdWORj0J33BsB0uqaTXF9sds7Qf7eQSCaTN+ViuPg fCLcsEX1wlBbloA== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net In case of the default exception callback, change the behavior of bpf_throw, where the passed cookie value is no longer ignored, but is instead the return value of the default exception callback. As such, we need to place restrictions on the value being passed into bpf_throw in such a case, only allowing those permitted by the check_return_code function. Thus, bpf_throw can now control the return value of the program from each call site without having the user install a custom exception callback just to override the return value when an exception is thrown. We also modify the hidden subprog instructions to now move BPF_REG_1 to BPF_REG_0, so as to set the return value before exit in the default callback. Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 37 ++++++++++++++++++++++++------------- 1 file changed, 24 insertions(+), 13 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 863e4e6c4616..0ba32b626320 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -11485,6 +11485,8 @@ static int fetch_kfunc_meta(struct bpf_verifier_env *env, return 0; } +static int check_return_code(struct bpf_verifier_env *env, int regno); + static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, int *insn_idx_p) { @@ -11613,6 +11615,15 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, return -ENOTSUPP; } env->seen_exception = true; + + /* In the case of the default callback, the cookie value passed + * to bpf_throw becomes the return value of the program. + */ + if (!env->exception_callback_subprog) { + err = check_return_code(env, BPF_REG_1); + if (err < 0) + return err; + } } for (i = 0; i < CALLER_SAVED_REGS; i++) @@ -14709,7 +14720,7 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn) return 0; } -static int check_return_code(struct bpf_verifier_env *env) +static int check_return_code(struct bpf_verifier_env *env, int regno) { struct tnum enforce_attach_type_range = tnum_unknown; const struct bpf_prog *prog = env->prog; @@ -14743,22 +14754,22 @@ static int check_return_code(struct bpf_verifier_env *env) * of bpf_exit, which means that program wrote * something into it earlier */ - err = check_reg_arg(env, BPF_REG_0, SRC_OP); + err = check_reg_arg(env, regno, SRC_OP); if (err) return err; - if (is_pointer_value(env, BPF_REG_0)) { - verbose(env, "R0 leaks addr as return value\n"); + if (is_pointer_value(env, regno)) { + verbose(env, "R%d leaks addr as return value\n", regno); return -EACCES; } - reg = cur_regs(env) + BPF_REG_0; + reg = cur_regs(env) + regno; if (frame->in_async_callback_fn) { /* enforce return zero from async callbacks like timer */ if (reg->type != SCALAR_VALUE) { - verbose(env, "In async callback the register R0 is not a known value (%s)\n", - reg_type_str(env, reg->type)); + verbose(env, "In async callback the register R%d is not a known value (%s)\n", + regno, reg_type_str(env, reg->type)); return -EINVAL; } @@ -14771,8 +14782,8 @@ static int check_return_code(struct bpf_verifier_env *env) if (is_subprog && !frame->in_exception_callback_fn) { if (reg->type != SCALAR_VALUE) { - verbose(env, "At subprogram exit the register R0 is not a scalar value (%s)\n", - reg_type_str(env, reg->type)); + verbose(env, "At subprogram exit the register R%d is not a scalar value (%s)\n", + regno, reg_type_str(env, reg->type)); return -EINVAL; } return 0; @@ -14854,8 +14865,8 @@ static int check_return_code(struct bpf_verifier_env *env) } if (reg->type != SCALAR_VALUE) { - verbose(env, "At program exit the register R0 is not a known value (%s)\n", - reg_type_str(env, reg->type)); + verbose(env, "At program exit the register R%d is not a known value (%s)\n", + regno, reg_type_str(env, reg->type)); return -EINVAL; } @@ -17053,7 +17064,7 @@ static int do_check(struct bpf_verifier_env *env) continue; } - err = check_return_code(env); + err = check_return_code(env, BPF_REG_0); if (err) return err; process_bpf_exit: @@ -18722,7 +18733,7 @@ static int do_misc_fixups(struct bpf_verifier_env *env) if (env->seen_exception && !env->exception_callback_subprog) { struct bpf_insn patch[] = { env->prog->insnsi[insn_cnt - 1], - BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_MOV64_REG(BPF_REG_0, BPF_REG_1), BPF_EXIT_INSN(), }; From patchwork Tue Sep 12 23:32:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13382314 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 80B761429F for ; Tue, 12 Sep 2023 23:32:25 +0000 (UTC) Received: from mail-ed1-x541.google.com (mail-ed1-x541.google.com [IPv6:2a00:1450:4864:20::541]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E0E8910FE for ; Tue, 12 Sep 2023 16:32:24 -0700 (PDT) Received: by mail-ed1-x541.google.com with SMTP id 4fb4d7f45d1cf-52a3ff5f0abso7989079a12.1 for ; Tue, 12 Sep 2023 16:32:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1694561543; x=1695166343; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CX5W3ta3gU0ie2rO52b7pq5F2OK5nUY1Jzz7hdFnd8g=; b=a/BTfN6kC/rS4GudjOtm+ikSjtZPfvmqQJhC9N0eqjgQGkzLblhvWsr3PVk2IDAIsY vgIjMMgUsX0NYjA1nHC+Aa1K0TkSmdJlMGGwsqjdiyp+heYoq1j9s5Jgje4S86I/R5Hs yvJLJDhGHHv5/sxbOmj8m0yNm1eh+rXCur5WI3UZRx/Ek6z9xyl4u4q95TmmBwzU+MH1 GCYoPNmyCalx/vkQxPz6juKVWD3HQ3y9YNSzGveUVlPVoERVItoJ+OdFIxLX+okSs29R xVNYBGxUo2XwZmTQIVNuXgUTARoiQquBlOWit1lkRMXmzkv0KkL63SWB/0ZRf5AY43tC 9lyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694561543; x=1695166343; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CX5W3ta3gU0ie2rO52b7pq5F2OK5nUY1Jzz7hdFnd8g=; b=w649VUthJH+5aOawFOvN5A1mziQUbdZxZsXWbg7em3JlVdH0nlBXRfeOOwmCpJKYNj PVphRmjERrzoslgrl6X1Qrld7rzunCQz50IhsXm0qoEZejmianvD0EJdXAl4f5rz8/4H GyOuIHylTIbWiZ6sEIlLHvaMl5sIr8kg8kDrPCyZi+AUydM673N5vp8jsSIfwUHUxz+K FrhJOv5os2nnbW7TqFB6LYGB2kBFYy1gUY+e+3EU9ovI4gWYJd+EVcvcfx83fCRCK7pb 2vVU3Bq7UBeizhgvOAyJYN6xmIBNRiVMqEnqUFgJ7pdm/8rq2622PehGxhsI9GIhGygi 4DSQ== X-Gm-Message-State: AOJu0Ywo7U9waUz6EzfkQzqBGx3Td1YO+HBsV3bmx8DKISyX4vRODbMd +yzPz43g+snel8zf/sWuagfFeuit9mKmfw== X-Google-Smtp-Source: AGHT+IFcAdOr+jGbCuOQfQY+xvaDT+sj3Shn01gwlyolFgLuNGnZeI/kYCR0zpF0D5HO0S+24/iwxw== X-Received: by 2002:a05:6402:1841:b0:525:469a:fc38 with SMTP id v1-20020a056402184100b00525469afc38mr763072edy.31.1694561542991; Tue, 12 Sep 2023 16:32:22 -0700 (PDT) Received: from localhost ([212.203.98.42]) by smtp.gmail.com with ESMTPSA id u14-20020aa7d0ce000000b0052c11951f4asm6470206edo.82.2023.09.12.16.32.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 16:32:22 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet , Puranjay Mohan Subject: [PATCH bpf-next v3 09/17] mm: kasan: Declare kasan_unpoison_task_stack_below in kasan.h Date: Wed, 13 Sep 2023 01:32:06 +0200 Message-ID: <20230912233214.1518551-10-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230912233214.1518551-1-memxor@gmail.com> References: <20230912233214.1518551-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2072; i=memxor@gmail.com; h=from:subject; bh=1JLywslARl/eCQeLo0pwx6E32+AEHxgjPliJB0G1CfA=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBlAPSt8vlk29yiHpA4r+PG7av/fkR4SzU6cnCaj 1nwE4VzaR6JAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZQD0rQAKCRBM4MiGSL8R yi0rD/0Yi9KxY096Ymw+E/lHpO9JFxOcZh3SbVGRVUFU08+vUspEigPV5Yh0Vo3tVBSRxD8wcvS 5M12qjX9Fc0NlyePCzf822AUPXnkF1ZD3+5APSlo137jOjATg6eD28HIAUcS07LC2AUsgsL/EGk jo0Hv+PruQ9cWzVFdQChcMqTFSvtmmf3xeYtejc/aSyxbHPji12Pfw4TDOAcOpaN+R+LTM4c/MQ zK0IFsY1XjxOeSKqA/WBFaS86jBuQ6uTQsHQHrWmz6UK1ZY64onRxCqhmmzOYRrx7rp4/Hzg+Ee r2aecJ7salFv81wnWg3/jzUywFpxv/ZhBhsoerzDSxPwm+0Yz6nHKmdb2FsVI7dPIMmd12i8H/n K5Ynlacj9zoJTo5AXdYSzHTZCLjOw6m9+x//cqFVgF1fZooAIF7C5obZvZJPlm9L713oG09bZeF EUrEoGbvKulh6AUuzYJJr+O1FYrBkfddUc2DLstuPOUVzhau2ex/wRFORGTR2oDYbU6OEjkIxSY kf8Jj97SrnVLxR/Y3DVvpe2CSP8+/jHTYUQq3PtKSEUS2YQSS5rjdsoJbSB3tJB0iyovVHwvjZ5 CF3tiyIG/U+2Rt4fdKcaji6YKjrH8pn3tgsnzIoYRBkxdvSbI8x4cNBxIh5AAqWWnaMcJ13L0Vb Ake5738DWja6X0A== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net We require access to this kasan helper in BPF code in the next patch where we have to unpoison the task stack when we unwind and reset the stack frame from bpf_throw, and it never really unpoisons the poisoned stack slots on entry when compiler instrumentation is generated by CONFIG_KASAN_STACK and inline instrumentation is supported. Also, remove the declaration from mm/kasan/kasan.h as we put it in the header file kasan.h. Cc: Andrey Ryabinin Cc: Alexander Potapenko Cc: Andrey Konovalov Cc: Dmitry Vyukov Cc: Vincenzo Frascino Suggested-by: Andrey Konovalov Signed-off-by: Kumar Kartikeya Dwivedi Reviewed-by: Andrey Konovalov --- include/linux/kasan.h | 2 ++ mm/kasan/kasan.h | 1 - 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/include/linux/kasan.h b/include/linux/kasan.h index 819b6bc8ac08..7a463f814db2 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -283,8 +283,10 @@ static inline bool kasan_check_byte(const void *address) #if defined(CONFIG_KASAN) && defined(CONFIG_KASAN_STACK) void kasan_unpoison_task_stack(struct task_struct *task); +asmlinkage void kasan_unpoison_task_stack_below(const void *watermark); #else static inline void kasan_unpoison_task_stack(struct task_struct *task) {} +static inline void kasan_unpoison_task_stack_below(const void *watermark) {} #endif #ifdef CONFIG_KASAN_GENERIC diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index 2e973b36fe07..5eefe202bb8f 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -558,7 +558,6 @@ void kasan_restore_multi_shot(bool enabled); * code. Declared here to avoid warnings about missing declarations. */ -asmlinkage void kasan_unpoison_task_stack_below(const void *watermark); void __asan_register_globals(void *globals, ssize_t size); void __asan_unregister_globals(void *globals, ssize_t size); void __asan_handle_no_return(void); From patchwork Tue Sep 12 23:32:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13382315 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 490A51429F for ; Tue, 12 Sep 2023 23:32:26 +0000 (UTC) Received: from mail-ed1-x544.google.com (mail-ed1-x544.google.com [IPv6:2a00:1450:4864:20::544]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 995D310FE for ; Tue, 12 Sep 2023 16:32:25 -0700 (PDT) Received: by mail-ed1-x544.google.com with SMTP id 4fb4d7f45d1cf-52f33659d09so4426760a12.1 for ; Tue, 12 Sep 2023 16:32:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1694561544; x=1695166344; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SOKs/u99SYf/gok0ubo7t7akZgu34vbpaZW0BR/hrbc=; b=k/mBncWHGbUtSjQJpt6vAynYbgfc0uATcE/Ydu/s/2hsWJoeqQYh9Yuha8beRk+KZy 51mWAtxgVs3JNN0ujKOoLX+jvxyz4k63z2m1h4Y6zQF90ofgWM14+YTlDG2guk94wISh HwVlqIhTjZgpaAyRoQv6GvBoFM4HVae8RVUZRWVtwpsBnj/3cCfF2nxzw975i8a3llf4 wJpdPnPbzcRH2wcRWq3zebZzF7N8yTLxfW2uKuut+IbfB6azWPJX+T3OcfMwFsDLNTVz oO6MBgvKjTf0INWv7QzAIUE/Wq0xUtMVXsuEM9eei5USApTA3LrsQQO/nWbDPL3fA3Q7 4okw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694561544; x=1695166344; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SOKs/u99SYf/gok0ubo7t7akZgu34vbpaZW0BR/hrbc=; b=CKxDo9hS9PKPyfzdqUGy0x3L4vhaFHkydebqRsi3gNuv66ZVSe77plIRDqgJaTyAmo LOWGUkStnX7KJi1ADHH63+YxY4nZSU2NV/36t/di+HXipfz60HkTSxANdEBGaiRCHj/S stjQWUT0h4NKUk1q+tgrwkriMAdrK41Sx5eMkr9jn0YzQHDXgeRBSej+uFG9WlOiJdVZ F2TMUvSTZEE/VUKZwnZq3ehY9bva7DIUxHaZYwNDtHmb7kK/Z0TMy88c8TKTTgx9RuV2 av/dc1GgiiwvYOH+whtxShVe9lILwMptk4+mUCkdoSpMx9TaPgXju9tkVVoEymhCve3c b/uA== X-Gm-Message-State: AOJu0YxQzLD87zEcURXBJJfnDDppw6Mz6MN9WFZ2aqRkxtkUYk1jvOFt QUP/vGkk021i8SNA9blqXvlbFJJuw2vF4Q== X-Google-Smtp-Source: AGHT+IFTZcKkft5IpTMVrOLD/YFt6SphpDuF6b+NplYeVeX3iNqU7pRvWUGW3PcrR0iszxAt5sRT4Q== X-Received: by 2002:a17:906:3297:b0:9a5:ceab:f496 with SMTP id 23-20020a170906329700b009a5ceabf496mr564053ejw.58.1694561543914; Tue, 12 Sep 2023 16:32:23 -0700 (PDT) Received: from localhost ([212.203.98.42]) by smtp.gmail.com with ESMTPSA id g21-20020a170906349500b00997e00e78e6sm7510766ejb.112.2023.09.12.16.32.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 16:32:23 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet , Puranjay Mohan Subject: [PATCH bpf-next v3 10/17] bpf: Prevent KASAN false positive with bpf_throw Date: Wed, 13 Sep 2023 01:32:07 +0200 Message-ID: <20230912233214.1518551-11-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230912233214.1518551-1-memxor@gmail.com> References: <20230912233214.1518551-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5518; i=memxor@gmail.com; h=from:subject; bh=bSTo5yopY85iWI1YQjw3VCUL+qF9FG7m7CyqMtVWf9A=; b=kA0DAAgBTODIhki/EcoByyZiAGUA9K2h1AjmTj9hZ5BtbUxeM9HGT0oVh+xlV2cjxSynmN6Ys IkCMwQAAQgAHRYhBEu+Kn4G7PnVgjxhEUzgyIZIvxHKBQJlAPStAAoJEEzgyIZIvxHKHbkQAIMw w0M3YXZDj0C7re3u7VJG6Zp/wUTDKS40jqziRNEz+VPHTrTIkTBFRIe37XdinVpNguqIfGRQuGy Dh/tCvs0gLDUAJvERqBBb+xtxw69uAAIZuhAIMCqqWliZbMdi6CKW+gSZhshGFjHVzJTgG2xBf2 8sYVoN91v4HT2h22glxylsh5w66v8R+QUNoOt5amClKI5yNKgNifmqkbOBS40vKhvbHDszpXUaW 1t1De/xEzfhGRxlzSOggaF9l4uTK4QOSlxkFncV8haFcgjHCVQoYHwZeS/XoGVFkiG70heYxvH0 LssxGbzfhqfDkb4YnJ3FrAbJhY+/MZzsqYSRGndhuTD5K18XbF4M3PT1tE0RQAR4DFT6g790r3+ LQ5WfO8v7CmrgR6kmXQalgjncOtEcpPl+e5MFohqvemkxICVHaOHZSsjSR1xMUkbw4opiQ8IQ9y b3XXIFEEqZk7dBks+bMV6JsD2XP2tyGPby8iDSKGPqGoVP7sf0N1F0kqSGL8Wb6UkVhwT5Cc5dO yEk2OkQoz1VBU5/LrQGAHBBChTf7IVbF2GmJ3Wcy6ahIzK48T89eaQOQn/wlhMYHll5P+H60ZA4 7Gt5/0YD534G36PhIBLqkSW3TXU9eN5XG5mjT8Fl7vj27QLkAlpTBZCFfJXXZ6qEERHHOES6Qdu Iu/oh X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net The KASAN stack instrumentation when CONFIG_KASAN_STACK is true poisons the stack of a function when it is entered and unpoisons it when leaving. However, in the case of bpf_throw, we will never return as we switch our stack frame to the BPF exception callback. Later, this discrepancy will lead to confusing KASAN splats when kernel resumes execution on return from the BPF program. Fix this by unpoisoning everything below the stack pointer of the BPF program, which should cover the range that would not be unpoisoned. An example splat is below: BUG: KASAN: stack-out-of-bounds in stack_trace_consume_entry+0x14e/0x170 Write of size 8 at addr ffffc900013af958 by task test_progs/227 CPU: 0 PID: 227 Comm: test_progs Not tainted 6.5.0-rc2-g43f1c6c9052a-dirty #26 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-2.fc39 04/01/2014 Call Trace: dump_stack_lvl+0x4a/0x80 print_report+0xcf/0x670 ? arch_stack_walk+0x79/0x100 kasan_report+0xda/0x110 ? stack_trace_consume_entry+0x14e/0x170 ? stack_trace_consume_entry+0x14e/0x170 ? __pfx_stack_trace_consume_entry+0x10/0x10 stack_trace_consume_entry+0x14e/0x170 ? __sys_bpf+0xf2e/0x41b0 arch_stack_walk+0x8b/0x100 ? __sys_bpf+0xf2e/0x41b0 ? bpf_prog_test_run_skb+0x341/0x1c70 ? bpf_prog_test_run_skb+0x341/0x1c70 stack_trace_save+0x9b/0xd0 ? __pfx_stack_trace_save+0x10/0x10 ? __kasan_slab_free+0x109/0x180 ? bpf_prog_test_run_skb+0x341/0x1c70 ? __sys_bpf+0xf2e/0x41b0 ? __x64_sys_bpf+0x78/0xc0 ? do_syscall_64+0x3c/0x90 ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 kasan_save_stack+0x33/0x60 ? kasan_save_stack+0x33/0x60 ? kasan_set_track+0x25/0x30 ? kasan_save_free_info+0x2b/0x50 ? __kasan_slab_free+0x109/0x180 ? kmem_cache_free+0x191/0x460 ? bpf_prog_test_run_skb+0x341/0x1c70 kasan_set_track+0x25/0x30 kasan_save_free_info+0x2b/0x50 __kasan_slab_free+0x109/0x180 kmem_cache_free+0x191/0x460 bpf_prog_test_run_skb+0x341/0x1c70 ? __pfx_bpf_prog_test_run_skb+0x10/0x10 ? __fget_light+0x51/0x220 __sys_bpf+0xf2e/0x41b0 ? __might_fault+0xa2/0x170 ? __pfx___sys_bpf+0x10/0x10 ? lock_release+0x1de/0x620 ? __might_fault+0xcd/0x170 ? __pfx_lock_release+0x10/0x10 ? __pfx_blkcg_maybe_throttle_current+0x10/0x10 __x64_sys_bpf+0x78/0xc0 ? syscall_enter_from_user_mode+0x20/0x50 do_syscall_64+0x3c/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 RIP: 0033:0x7f0fbb38880d Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d f3 45 12 00 f7 d8 64 89 01 48 RSP: 002b:00007ffe13907de8 EFLAGS: 00000206 ORIG_RAX: 0000000000000141 RAX: ffffffffffffffda RBX: 00007ffe13908708 RCX: 00007f0fbb38880d RDX: 0000000000000050 RSI: 00007ffe13907e20 RDI: 000000000000000a RBP: 00007ffe13907e00 R08: 0000000000000000 R09: 00007ffe13907e20 R10: 0000000000000064 R11: 0000000000000206 R12: 0000000000000003 R13: 0000000000000000 R14: 00007f0fbb532000 R15: 0000000000cfbd90 The buggy address belongs to stack of task test_progs/227 KASAN internal error: frame info validation failed; invalid marker: 0 The buggy address belongs to the virtual mapping at [ffffc900013a8000, ffffc900013b1000) created by: kernel_clone+0xcd/0x600 The buggy address belongs to the physical page: page:00000000b70f4332 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x11418f flags: 0x2fffe0000000000(node=0|zone=2|lastcpupid=0x7fff) page_type: 0xffffffff() raw: 02fffe0000000000 0000000000000000 dead000000000122 0000000000000000 raw: 0000000000000000 0000000000000000 00000001ffffffff 0000000000000000 page dumped because: kasan: bad access detected Memory state around the buggy address: ffffc900013af800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ffffc900013af880: 00 00 00 f1 f1 f1 f1 00 00 00 f3 f3 f3 f3 f3 00 >ffffc900013af900: 00 00 00 00 00 00 00 00 00 00 00 f1 00 00 00 00 ^ ffffc900013af980: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ffffc900013afa00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ================================================================== Disabling lock debugging due to kernel taint Cc: Andrey Ryabinin Cc: Alexander Potapenko Cc: Andrey Konovalov Cc: Dmitry Vyukov Cc: Vincenzo Frascino Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Andrey Konovalov --- kernel/bpf/helpers.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 78e8f4de6750..2c8e1ee97b71 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -22,6 +22,7 @@ #include #include #include +#include #include "../../lib/kstrtox.h" @@ -2483,6 +2484,11 @@ __bpf_kfunc void bpf_throw(u64 cookie) WARN_ON_ONCE(!ctx.aux->exception_boundary); WARN_ON_ONCE(!ctx.bp); WARN_ON_ONCE(!ctx.cnt); + /* Prevent KASAN false positives for CONFIG_KASAN_STACK by unpoisoning + * deeper stack depths than ctx.sp as we do not return from bpf_throw, + * which skips compiler generated instrumentation to do the same. + */ + kasan_unpoison_task_stack_below((void *)ctx.sp); ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp); } From patchwork Tue Sep 12 23:32:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13382316 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 12DDC1429F for ; Tue, 12 Sep 2023 23:32:27 +0000 (UTC) Received: from mail-wm1-x342.google.com (mail-wm1-x342.google.com [IPv6:2a00:1450:4864:20::342]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7189510FF for ; Tue, 12 Sep 2023 16:32:26 -0700 (PDT) Received: by mail-wm1-x342.google.com with SMTP id 5b1f17b1804b1-401d24f1f27so70514155e9.1 for ; Tue, 12 Sep 2023 16:32:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1694561545; x=1695166345; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=biVZEDEAYEKp2V+herIPqP7yZxpTsOL7p/iaRINkOLw=; b=nO6O/RfFLchSbmuq6CWirCyvS80yDDb7max4suhBLCBYRGrKOdKlrUJ7bNgO8p2n0B RqBWum/ZDbtRAHJ8iGXwNNufm7W4YR6ijRkepdxcY4dp8XBckB8PRpY/S21BDl96uqqa 7he34mPNrbQO6OGBZp2XveJexL0wakHEdghlaBwdOPOMvfRuAMzv9Jlvl96PIg8T5e/E r07Tj48bM3J6qKQpMCLnOoxdLTgRw9q88yV6iVthTgAn5itG9IZXCCwXbRaZwwpe4Gby 6eCn6VEHY1S80SqM6USayGJNGAXzN3IjJTUO4OpU2IttJ33Uv8GR/swbXEArQPbN7iny SboQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694561545; x=1695166345; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=biVZEDEAYEKp2V+herIPqP7yZxpTsOL7p/iaRINkOLw=; b=IbJCCnhUQrJgcF1LFC5PK0vn52u3QG6eBtMI50HCjQxzPXcl2EqzInA2bnc3Pv373c L5craTOr/H76AyuAf/AekhI18m2UlP74xLHRxKNE7Mg8TkkqmvQloHGP2q1tfDUFS1IE hC91mRSFK7XBADSbj8gxw4a18ePSPmUUurJXBCX11WGUaWIocbbag3DVN3HfFS9AsVxC 2aC2PpOP/cE3+iIHICpw1MWmxYp3pM4km9hEONBDshFG+UP81rI0pL2q5/OF8j2o4kqG TS/kxSLy5nNxagU4O3eGa+3VJE3hcaQ0MU4Zr3QzfTgWi92F84Q7nibo/qzK5iI30qaA 45BQ== X-Gm-Message-State: AOJu0YyPxWuF4pg0qRpbYmMMpmR2FnrUkkkdRINP2908Z7OB9SqKQ/rB BqpTPXaoipddd/cqSYszqFl+07YQWdixhg== X-Google-Smtp-Source: AGHT+IF/10U9we/4qPZgBYOYdt/ovJKo4QwujYWea7Vpt3e2S0TsrozE7pK8bqssM+1yK9P7BDxLBw== X-Received: by 2002:a05:600c:108b:b0:3fe:2079:196c with SMTP id e11-20020a05600c108b00b003fe2079196cmr723023wmd.16.1694561544704; Tue, 12 Sep 2023 16:32:24 -0700 (PDT) Received: from localhost ([212.203.98.42]) by smtp.gmail.com with ESMTPSA id dx22-20020a170906a85600b0099d959f9536sm7547968ejb.12.2023.09.12.16.32.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 16:32:24 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet , Puranjay Mohan Subject: [PATCH bpf-next v3 11/17] bpf: Detect IP == ksym.end as part of BPF program Date: Wed, 13 Sep 2023 01:32:08 +0200 Message-ID: <20230912233214.1518551-12-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230912233214.1518551-1-memxor@gmail.com> References: <20230912233214.1518551-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2617; i=memxor@gmail.com; h=from:subject; bh=pGHbRWDKfQzSJuQ7RuVgf7MLQ6H8s585iql6KmWXG7k=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBlAPSt+kbHObhSg37ORGR43DfIeHYlhc407aWY3 Tt++AHRRMKJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZQD0rQAKCRBM4MiGSL8R ynKDEACnhlZp7MyNLouJrtxXegUULiCsAU0ZoQjgtnRGchFApD7pPel1sVhpr0bwD6qyNakDXPm oHwJfG6iCaUUoRea9jHbosSZt5VN0+WYpmsqYDr4XZAYCGlsL8OId+2tWMe15A9UBnN6oPh/7iL oK1vgxmDlN67l8bBiNoNOgu8nsOQwFZsc+kjqueRotwA8rfxrxatFSmVIazUdr+u7dQ6/FXKVyK rG6Q93QNf3ovhzlGuV++jV1V+ULny9/rE8RiOMd54MaI5jRvZm1sPtD5/cnXdpQY+cGd+E+yiL0 t5C2pMDZjqVPHLJCZpHVuR7SOI1XD415wQXliXq7tiBOVNx7f+Jep9Rq+4byip2EiZa/LgtpMu9 IOujkOLl19z7rubTE1EsnVFzGCMrryfOeDK9a5sagr4M+0NQgXUSPepKSK+qYs49kMjLvSpRcuM LcJLWW6+s+qDLUK1Qst7kT0nPsMx1vyrDNyukAPSOJ+UCTtgQMjoiGIyltxdtycrOeQEmPAJ0x/ nEQLmDLmsWBvPVm6xfHhYxj5zNQ/fGK5bRW16EA6tP8tq4Yjl5y4j6d/ZmDji6tVMZ7jFrJAaVZ ePxfltM6LzXiqbMuTgVDaY13pxyTiiMl+qFtwFqU1X2CWv2bAviN9ml1JQfCJyTQ/TENXqIHtFQ raJ5B2G1zjyoL/Q== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net Now that bpf_throw kfunc is the first such call instruction that has noreturn semantics within the verifier, this also kicks in dead code elimination in unprecedented ways. For one, any instruction following a bpf_throw call will never be marked as seen. Moreover, if a callchain ends up throwing, any instructions after the call instruction to the eventually throwing subprog in callers will also never be marked as seen. The tempting way to fix this would be to emit extra 'int3' instructions which bump the jited_len of a program, and ensure that during runtime when a program throws, we can discover its boundaries even if the call instruction to bpf_throw (or to subprogs that always throw) is emitted as the final instruction in the program. An example of such a program would be this: do_something(): ... r0 = 0 exit foo(): r1 = 0 call bpf_throw r0 = 0 exit bar(cond): if r1 != 0 goto pc+2 call do_something exit call foo r0 = 0 // Never seen by verifier exit // main(ctx): r1 = ... call bar r0 = 0 exit Here, if we do end up throwing, the stacktrace would be the following: bpf_throw foo bar main In bar, the final instruction emitted will be the call to foo, as such, the return address will be the subsequent instruction (which the JIT emits as int3 on x86). This will end up lying outside the jited_len of the program, thus, when unwinding, we will fail to discover the return address as belonging to any program and end up in a panic due to the unreliable stack unwinding of BPF programs that we never expect. To remedy this case, make bpf_prog_ksym_find treat IP == ksym.end as part of the BPF program, so that is_bpf_text_address returns true when such a case occurs, and we are able to unwind reliably when the final instruction ends up being a call instruction. Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/core.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 7849b9cca749..8f921b6d6981 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -623,7 +623,11 @@ static __always_inline int bpf_tree_comp(void *key, struct latch_tree_node *n) if (val < ksym->start) return -1; - if (val >= ksym->end) + /* Ensure that we detect return addresses as part of the program, when + * the final instruction is a call for a program part of the stack + * trace. Therefore, do val > ksym->end instead of val >= ksym->end. + */ + if (val > ksym->end) return 1; return 0; From patchwork Tue Sep 12 23:32:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13382317 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F0E561429F for ; Tue, 12 Sep 2023 23:32:27 +0000 (UTC) Received: from mail-ej1-x644.google.com (mail-ej1-x644.google.com [IPv6:2a00:1450:4864:20::644]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5C38210FE for ; Tue, 12 Sep 2023 16:32:27 -0700 (PDT) Received: by mail-ej1-x644.google.com with SMTP id a640c23a62f3a-99bf3f59905so770187166b.3 for ; Tue, 12 Sep 2023 16:32:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1694561545; x=1695166345; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=W4nEz1Iccr+xZqvDM5bnGug5s7P4+oRT7ierkjOsGaE=; b=Zd909t06qpmEmHXw+DWc1Y/XGH9qWbd0bcwyfuuNf4G6KZPyhl+UfQHHgrQrhp4+1n 2JMP27mswQWqyy6MzmVEPvSMhHIYo9q9t/W6xJFSK8VmrX3iRt1pGIsj1BdGOlDv+NBM Zmscf3OoW22f66u9zdduZyS945AUAK+RGIvbbVJCJK3STdSMTWYc5HiXKgJ0ndiGgv9i o8swP/0uyE8B1UyHr6B7yUAov7hGe4n22F4C6SKYLvi0Kg8IFRF7BHe8iV+ipXpIOU8/ 5LWnvKewUIbCDxyhr28Kyol6J9BY6eth0zh1QUbxCF+YJTJhyW8D+q5ZpKRCI0zNIf47 UXPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694561545; x=1695166345; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=W4nEz1Iccr+xZqvDM5bnGug5s7P4+oRT7ierkjOsGaE=; b=TssYe5EaBAd5pp6vRD3COsYioqxqllosizXxsXsg82DYIBsbL+JXag9RbAbY7yhNvd ubeV9vbnl1VLVnMxFrDYJq8H7BrmlmGFWf+ExfyXiRenQXjwNP8WDaNic0y6DJ3EpA8U jaKt0KNgEDRtJRlm4GMdZGEEgfW3DDKc7/tzRBDoE8so4WVrPPJDh+nnTdEYV+o/EvXW 0ZNr0YH8BdNtVPaJbOCTLpQ2gm36mLl5hPumxSwKuEsYfCv4kjkmXkgxxdunkCt9QpQX WEerVtRU807wW+udNPnbTF9qaqGFoHG5lnl6gZtRadgcZb2OvD99E6rFXYUmriT/MRi0 q38Q== X-Gm-Message-State: AOJu0Yzxldku7j0qABMGNEN3aR7YjuOvx37YGtjJMYstbUn8RhM2Tmzn VEv01lKFecwC5BYbnakSuz85D53n6c7Fgg== X-Google-Smtp-Source: AGHT+IEr8d3pYighYqPSWw4xRQfn0fNAl7/N89SiBWDt1TFgsOm8aItNZpw6jtag4NbEDPU78mnqOg== X-Received: by 2002:a17:907:2c6a:b0:9a1:edfd:73bb with SMTP id ib10-20020a1709072c6a00b009a1edfd73bbmr601569ejc.47.1694561545491; Tue, 12 Sep 2023 16:32:25 -0700 (PDT) Received: from localhost ([212.203.98.42]) by smtp.gmail.com with ESMTPSA id se22-20020a170906ce5600b009920e9a3a73sm7489440ejb.115.2023.09.12.16.32.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 16:32:25 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet , Puranjay Mohan Subject: [PATCH bpf-next v3 12/17] bpf: Disallow fentry/fexit/freplace for exception callbacks Date: Wed, 13 Sep 2023 01:32:09 +0200 Message-ID: <20230912233214.1518551-13-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230912233214.1518551-1-memxor@gmail.com> References: <20230912233214.1518551-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3994; i=memxor@gmail.com; h=from:subject; bh=ZYZfPK9xehq0OlRbe34+tgm5ELBdf5+L7TGiYkXpUx8=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBlAPStPNtkQvOi1eXnTzCTxzh4HbyCSiqKhPicI 8AZlMhkLpGJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZQD0rQAKCRBM4MiGSL8R ytoBEACbKGtjeyjXZ9sBe7Z2Y3b8mnALTo14ca8u7Pt+C/NGBzgdYAZRyrm9vT+VM8OCpLNM0Of JjxbY5PvqY4a5hbs5yt95HZjAn6v33afWRCIZ4mUhew5wXCUjrK6mNq2ubrq5apiUhDo5LP6y+e H1c+8hcpUqimaC4GyoC0VCewDoJhaEv8Y70q9yE5r5r7MEuZyTzbFHymF+PPnxF2ic2YYOliDSM QteuvDOFnIYKg2JbylMYZ7l4BrkQGDygBckOvDQhad2wYullNweq06gUGk+2jQcDU4MjxgAmG8R CQ5u85N3iPTNctYT8KClmNddrlLQQVpoeuKo+rGMZlWHJD9ABdouAk+UI1+JcWJlFaA5o1JoLXa 83xkCVD1Nm/Js4LxHA6qI38HoVJ0q7xGH6sBu+qTkZZj9++AD2Keex7II63J/EWnpc4KP4eyA1g oYEaMjtJGorb5qTUB2sdAF91kFqeqyEUwcBjx2PplmUB0b2Opxn58/B9R21+bMHpeDhhRrkH6sq CLYOl/9X/3SdNf81j5aYrh4RPgvBkaKudhs1V4UQrx6Dxy96SqqlZsTkhR2NK+HXbOpuX87C3Tf N4RDPOgPSPBgFDtfjo645JW14XbkZILMcMYydMU+zt84NPaY6XBCR7Tv1NREMcwIgQO/7qoZn+A HFqWyCrWYlxu2wQ== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net During testing, it was discovered that extensions to exception callbacks had no checks, upon running a testcase, the kernel ended up running off the end of a program having final call as bpf_throw, and hitting int3 instructions. The reason is that while the default exception callback would have reset the stack frame to return back to the main program's caller, the replacing extension program will simply return back to bpf_throw, which will instead return back to the program and the program will continue execution, now in an undefined state where anything could happen. The way to support extensions to an exception callback would be to mark the BPF_PROG_TYPE_EXT main subprog as an exception_cb, and prevent it from calling bpf_throw. This would make the JIT produce a prologue that restores saved registers and reset the stack frame. But let's not do that until there is a concrete use case for this, and simply disallow this for now. Similar issues will exist for fentry and fexit cases, where trampoline saves data on the stack when invoking exception callback, which however will then end up resetting the stack frame, and on return, the fexit program will never will invoked as the return address points to the main program's caller in the kernel. Instead of additional complexity and back and forth between the two stacks to enable such a use case, simply forbid it. One key point here to note is that currently X86_TAIL_CALL_OFFSET didn't require any modifications, even though we emit instructions before the corresponding endbr64 instruction. This is because we ensure that a main subprog never serves as an exception callback, and therefore the exception callback (which will be a global subprog) can never serve as the tail call target, eliminating any discrepancies. However, once we support a BPF_PROG_TYPE_EXT to also act as an exception callback, it will end up requiring change to the tail call offset to account for the extra instructions. For simplicitly, tail calls could be disabled for such targets. Noting the above, it appears better to wait for a concrete use case before choosing to permit extension programs to replace exception callbacks. As a precaution, we disable fentry and fexit for exception callbacks as well. Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/helpers.c | 1 + kernel/bpf/verifier.c | 11 +++++++++++ 2 files changed, 12 insertions(+) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 2c8e1ee97b71..7ff2a42f1996 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -2490,6 +2490,7 @@ __bpf_kfunc void bpf_throw(u64 cookie) */ kasan_unpoison_task_stack_below((void *)ctx.sp); ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp); + WARN(1, "A call to BPF exception callback should never return\n"); } __diag_pop(); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 0ba32b626320..21e37e46d792 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -19750,6 +19750,12 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, bpf_log(log, "Subprog %s doesn't exist\n", tname); return -EINVAL; } + if (aux->func && aux->func[subprog]->aux->exception_cb) { + bpf_log(log, + "%s programs cannot attach to exception callback\n", + prog_extension ? "Extension" : "FENTRY/FEXIT"); + return -EINVAL; + } conservative = aux->func_info_aux[subprog].unreliable; if (prog_extension) { if (conservative) { @@ -19762,6 +19768,11 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, "Extension programs should be JITed\n"); return -EINVAL; } + if (aux->func && aux->func[subprog]->aux->exception_cb) { + bpf_log(log, + "Extension programs cannot replace exception callback\n"); + return -EINVAL; + } } if (!tgt_prog->jited) { bpf_log(log, "Can attach to only JITed progs\n"); From patchwork Tue Sep 12 23:32:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13382318 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C53C91429F for ; Tue, 12 Sep 2023 23:32:28 +0000 (UTC) Received: from mail-ej1-x641.google.com (mail-ej1-x641.google.com [IPv6:2a00:1450:4864:20::641]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1D93910FE for ; Tue, 12 Sep 2023 16:32:28 -0700 (PDT) Received: by mail-ej1-x641.google.com with SMTP id a640c23a62f3a-99bcc0adab4so783637166b.2 for ; Tue, 12 Sep 2023 16:32:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1694561546; x=1695166346; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yyChwgbbJqWqWiTpBQ2MkMj0sBFmU5FxpvJsH4ueDzs=; b=A1cgsnSlVlLPdI7yfXXh9AJSKrI6wRrMH1pUnUGl6i+OmGAtB/rr+86z705AIRqQWE IDymSGH+ssV3w8p52Pq+WtvRwlhgpmi0xxXMUJ1zMdBhAnzsWMbibJZqhOxv3VP+lgZD 0AuTmhD5RbjbuTMEJ9gs/jJ9NS6JziAFu1Q++V6kiZu6Rc9QlnfWTcohd2Kio48Vqnte q17nuT9r/yXavTywUQoI8rImoPT3l2QypsXrkNsgkvC4ukUlh2jL3czLAzzSzg+9Z6cP R0NnKNv+J80t0T3pIJFzyK9CMHR3piSXynUW7pjbxgrKgY7RlV1rTP88tMQgVC4qRcJY KIvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694561546; x=1695166346; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yyChwgbbJqWqWiTpBQ2MkMj0sBFmU5FxpvJsH4ueDzs=; b=vZTN7tXWuoPmU4RZ8ft5fRRx06+3f8CAGbF8WsLqVI0ijQ3VNmnNm3e4OZDapk4Zwv nUrvNsnO5O5LrY6SblEVDsESm9v9/6rLmbXR4k02nrH1V5zusqAjDpzc8Hw/67Ow+zC8 xBFu1W9vZPSebQuY1uciS1IeZu6texEke6TVp9xwFlcWrI1wPpgcxrFZ5u+okEA2So9O Ad3qx2QwuJ4twmlJ3vTySDvrUGl+A1qaHci/7JGYYV8WrKD+H0P2REHgcHRWTJU3hjCS Iyxi98ODac1UNG0etZrd13ZxVDnbldsFYhI0N2z4tbNApjRnJsRJUhW45DFSy4jScV+c 1pTg== X-Gm-Message-State: AOJu0YwDLg0oYGAe65ONKG6IUA3Y1L/Ox/qvS5tJ1xbgxwyKIcR1NG6O pNcX9ubKk2AN4iAeozlKdWETCvZyyaDm1g== X-Google-Smtp-Source: AGHT+IFKOQwe7FmXhBVo8H5o6KulCXPmmyZ+5ROq0O+/Zz+g65oeiwl13r0JH1Nzk6BC1Y+aJ0Etpg== X-Received: by 2002:a17:906:3290:b0:9a6:3d19:df7 with SMTP id 16-20020a170906329000b009a63d190df7mr487812ejw.17.1694561546367; Tue, 12 Sep 2023 16:32:26 -0700 (PDT) Received: from localhost ([212.203.98.42]) by smtp.gmail.com with ESMTPSA id y5-20020a1709064b0500b009a0955a7ad0sm7397065eju.128.2023.09.12.16.32.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 16:32:26 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Dave Marchevsky , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet , Puranjay Mohan Subject: [PATCH bpf-next v3 13/17] bpf: Fix kfunc callback register type handling Date: Wed, 13 Sep 2023 01:32:10 +0200 Message-ID: <20230912233214.1518551-14-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230912233214.1518551-1-memxor@gmail.com> References: <20230912233214.1518551-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1184; i=memxor@gmail.com; h=from:subject; bh=3b0jLk4e2ZsOFf+IyG/nlx/5RtVzoTpm+030EYo5lSI=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBlAPSt/NTVp2ktFkA3+BKrZefflXbPn2kZ3dz8q hwNrfrWHeOJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZQD0rQAKCRBM4MiGSL8R yklkD/9BRz3S2yjj4jo73e4p4/w31r2QkvU/QKGvvnf8U8mGn/iH8stiCORbjL5jlaAlUjm9qV+ gnmQRE1TkvPI5BNRWI9omHz2LKDUJfOwUfsDnxRwMAtFh85E4NB+2RTLjzgcmwX3ndRn5yVBcvH 51NdUcoTC8pgK5SLUxwEbV2QBCelA4G+lrEn+N/R1fd0X0rkmJbCjgiMGJhMJuretEgUbWOs2RK l73Zt48zEQyTx0T9SWruHyF/s9YYR4b75OieO5dsll54zI6xVbFaTzMt/kDW/wlFU8eck//ZUXe 39WNUlCdSM3fVXE4FPqoqj+ievfOGwRu6RiwUWsWa/p+SWrv+sa1fFOP+pTg5dEpjlgbPy+EmA4 Bd1XNyuLGIG/AVGcQqV4BLKw4Y63/xLkJDJVFakCkFNE5L3aJoUGOVvevhotmuDnToQvms2owuH anPBsL0E4+fp/NqpmmVQQG1VA1r5vAdKNyvp1iidz62laZuv/4YbDGdXSpLcouwmahnUD+SJC76 E+OsbiMtcHDGbvJo9Ngn8XNUe4bYb/4uWxKPEkSNWaYVU7B5cZHPuBP7X5/cLGWwxhDeq9Mz5Df NmfO05lcsKvI8xj+zUuDW+O6uQuxHc/C4W/nDwGuS/CDUGBywnOAuDtdaVkTvUJVOnEWraJXCxy WfFt+5pvrpUx/OA== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net The kfunc code to handle KF_ARG_PTR_TO_CALLBACK does not check the reg type before using reg->subprogno. This can accidently permit invalid pointers from being passed into callback helpers (e.g. silently from different paths). Likewise, reg->subprogno from the per-register type union may not be meaningful either. We need to reject any other type except PTR_TO_FUNC. Acked-by: Dave Marchevsky Fixes: 5d92ddc3de1b ("bpf: Add callback validation to kfunc verifier logic") Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 21e37e46d792..dff8c43dea0c 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -11407,6 +11407,10 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ break; } case KF_ARG_PTR_TO_CALLBACK: + if (reg->type != PTR_TO_FUNC) { + verbose(env, "arg%d expected pointer to func\n", i); + return -EINVAL; + } meta->subprogno = reg->subprogno; break; case KF_ARG_PTR_TO_REFCOUNTED_KPTR: From patchwork Tue Sep 12 23:32:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13382319 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A779E11CAD for ; Tue, 12 Sep 2023 23:32:29 +0000 (UTC) Received: from mail-ej1-x642.google.com (mail-ej1-x642.google.com [IPv6:2a00:1450:4864:20::642]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0CBC510FE for ; Tue, 12 Sep 2023 16:32:29 -0700 (PDT) Received: by mail-ej1-x642.google.com with SMTP id a640c23a62f3a-99bcf2de59cso786870266b.0 for ; Tue, 12 Sep 2023 16:32:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1694561547; x=1695166347; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=v8RcyRlAcTVPnJQprPVDfbaGCP2e90Do5WdsC5vuOCQ=; b=q4t06F97YddbwMyjpC4kPdLtW2ssw6xdPQ4ywFQxqpYocgrQ8yiOKxlbKQ0grZQccW UbZSZ0SdR1ca662OkPuFdGaflzm9O9oH+J7/gHqDTT0/wNjmiRWY1Iw0cFjLNqOruC9h 9gBu92CQyoWH9bDea5qizTgouJIWezxHNMCZ2IK5k6eP/DOZt8rkegAt7xABpK0hCUK2 A2LwmkGgyonLZ3vzw+qhgjqw3qD0EYCp/0rY4nbfe+xIA2JQZq6hYv0EPeOGM+Ef03S/ LQjwEBGFHjmRZfES2jP2++KCk1f27XQSvcYCfU2nyEaXkW4+yRYJyOiU/PoxCIvpecHA XwhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694561547; x=1695166347; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=v8RcyRlAcTVPnJQprPVDfbaGCP2e90Do5WdsC5vuOCQ=; b=VOHyxSzpR3YqMk7kCFF+wTU41hKWxk7LNn2Vdr9SO/AWfc7Z5WJ+FmyoGIPQGkQOrM +PYyKlmKLgotNOlqe++gAHe3vNRKVVjgZCyX1lFAmrKkcAEuJkenyU8VFFLeQ1Qxj+rR dM7tzCUDupyCXvQtVF9ss0lOmGxW+joj1bKQbbvEPQvPPVmNkzoWjPbsJwbAc3E1x7zU 6q4SJltli3JFkFVYxNEgIDrhH3uMmOw2aQTXtgzBiOLNLAhJohN+nhXnaUANFcvFF2m+ S0yahGOSEObFo5LBQRX875R+0IRLebvsX8zejw7LYROJoTYHs1hVmMANKVqCG/VcODRR RXNA== X-Gm-Message-State: AOJu0YyVciRD3Lc3cmvrPDrRe6s3bJZajLxr+0EWkiqRt5AuqeL0j0Sf FPq2MGxHZqPFrBK0B2Uf2giQd1qnECpeKw== X-Google-Smtp-Source: AGHT+IEUaaIHmtXQHJmVxTEP9Hbe6bYToDQnqgepC2vthuESsBCsu0Tm7afIcUubNJ7E8af1qqfcCQ== X-Received: by 2002:a17:906:20d6:b0:99b:4ed4:5527 with SMTP id c22-20020a17090620d600b0099b4ed45527mr574932ejc.25.1694561547127; Tue, 12 Sep 2023 16:32:27 -0700 (PDT) Received: from localhost ([212.203.98.42]) by smtp.gmail.com with ESMTPSA id a26-20020a170906369a00b0099cc1ffd8f5sm7444912ejc.53.2023.09.12.16.32.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 16:32:26 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet , Puranjay Mohan Subject: [PATCH bpf-next v3 14/17] libbpf: Refactor bpf_object__reloc_code Date: Wed, 13 Sep 2023 01:32:11 +0200 Message-ID: <20230912233214.1518551-15-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230912233214.1518551-1-memxor@gmail.com> References: <20230912233214.1518551-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3020; i=memxor@gmail.com; h=from:subject; bh=FSoeykkEjhk1rfwOQVrvkuG8ZCFZK7m17K028PQKB7o=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBlAPSt//1GlxwfkJsBA2MkN7m6I42SSs1stWqRE 9s2abm4EnOJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZQD0rQAKCRBM4MiGSL8R ymuCEACltpqPCjzhqBMn9EPDpegA5v9c0hdmAE6k62agzEXHSc1/Er2bkuw44jA8QRmAKDb0NRv dGZdxDy+esejEXmmyRub64/Dfwu63gqT93XysGYUceHxarlETL7q5goTJYtZUna6h/8WLjUKPuV 8nCjA60mN2f5YfNb4KGznG6h62sMAJXaIcUW4QsskQiUPb3xbu5kwIENhHyLSXjx1BZwPeFBvYR w2N6LFtR60SnJkVouucU370sAn2SuorZkH4608obv8cB8C22Vcc4D1KRC0TjXEuKZF8+Kllx3gx g8PT1pnRjOJeNbBe/YUwRy4nR1ZS7yC0egqXscSvG2hlL5p6HfTfagnfFpKurlckNMxgx7UhsjA HRmu8S5gdgfzantzRVMXbJUMpDwTt2TcOPNYckEtGnnuVtE8vgQsv3jt5O9EMiLPp2icxwR2AFz k3vI0LQj7eigRURgBCxlP0v1Cymafmhm1ekdoVLWoy5AFKDTw2YraeGI/ZwueQcJRLl5tgBJUSc sdvokI4tZxm6CP2XhthUGtuW9rEy7Qya883D9XzzayyXAp5qo8LNbhvyFGSRxpitgN57Xql49hv fRY9xwyHQLikO37pZkni3kM+5dtTqpuPva0T/0XuGP2vWrVCf7maL3H3YPrEPBjDZ5WDpw0+Mvj UKgVb05oBolh/KA== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net Refactor bpf_object__append_subprog_code out of bpf_object__reloc_code to be able to reuse it to append subprog related code for the exception callback to the main program. Signed-off-by: Kumar Kartikeya Dwivedi --- tools/lib/bpf/libbpf.c | 52 +++++++++++++++++++++++++++--------------- 1 file changed, 33 insertions(+), 19 deletions(-) diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 96ff1aa4bf6a..afc07a8f7dc7 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -6234,6 +6234,38 @@ static int append_subprog_relos(struct bpf_program *main_prog, struct bpf_progra return 0; } +static int +bpf_object__append_subprog_code(struct bpf_object *obj, struct bpf_program *main_prog, + struct bpf_program *subprog) +{ + struct bpf_insn *insns; + size_t new_cnt; + int err; + + subprog->sub_insn_off = main_prog->insns_cnt; + + new_cnt = main_prog->insns_cnt + subprog->insns_cnt; + insns = libbpf_reallocarray(main_prog->insns, new_cnt, sizeof(*insns)); + if (!insns) { + pr_warn("prog '%s': failed to realloc prog code\n", main_prog->name); + return -ENOMEM; + } + main_prog->insns = insns; + main_prog->insns_cnt = new_cnt; + + memcpy(main_prog->insns + subprog->sub_insn_off, subprog->insns, + subprog->insns_cnt * sizeof(*insns)); + + pr_debug("prog '%s': added %zu insns from sub-prog '%s'\n", + main_prog->name, subprog->insns_cnt, subprog->name); + + /* The subprog insns are now appended. Append its relos too. */ + err = append_subprog_relos(main_prog, subprog); + if (err) + return err; + return 0; +} + static int bpf_object__reloc_code(struct bpf_object *obj, struct bpf_program *main_prog, struct bpf_program *prog) @@ -6316,25 +6348,7 @@ bpf_object__reloc_code(struct bpf_object *obj, struct bpf_program *main_prog, * and relocate. */ if (subprog->sub_insn_off == 0) { - subprog->sub_insn_off = main_prog->insns_cnt; - - new_cnt = main_prog->insns_cnt + subprog->insns_cnt; - insns = libbpf_reallocarray(main_prog->insns, new_cnt, sizeof(*insns)); - if (!insns) { - pr_warn("prog '%s': failed to realloc prog code\n", main_prog->name); - return -ENOMEM; - } - main_prog->insns = insns; - main_prog->insns_cnt = new_cnt; - - memcpy(main_prog->insns + subprog->sub_insn_off, subprog->insns, - subprog->insns_cnt * sizeof(*insns)); - - pr_debug("prog '%s': added %zu insns from sub-prog '%s'\n", - main_prog->name, subprog->insns_cnt, subprog->name); - - /* The subprog insns are now appended. Append its relos too. */ - err = append_subprog_relos(main_prog, subprog); + err = bpf_object__append_subprog_code(obj, main_prog, subprog); if (err) return err; err = bpf_object__reloc_code(obj, main_prog, subprog); From patchwork Tue Sep 12 23:32:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13382320 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BFAC311CAD for ; Tue, 12 Sep 2023 23:32:30 +0000 (UTC) Received: from mail-lj1-x243.google.com (mail-lj1-x243.google.com [IPv6:2a00:1450:4864:20::243]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1FF7E10FE for ; Tue, 12 Sep 2023 16:32:30 -0700 (PDT) Received: by mail-lj1-x243.google.com with SMTP id 38308e7fff4ca-2b962535808so106247011fa.0 for ; Tue, 12 Sep 2023 16:32:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1694561548; x=1695166348; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=b7pmoitH+AiMJiFKg+FFb7MBDaxehp8zsVFc9eCNZDA=; b=C2EH5NFazHo6EfeH4g1nta0ofcOIFuasJ3+Z6tmMEVnQABci4Mt0lNKdNOOCgP6T9g fqfU5s7nJ1QfKYEoioUGG1xC4pJ8GOUukwKprN1S+DEfZm2VkkaTJcD6jwUc0GFntvvu YJvKHbXZZiSrlm6GdSygivVcTOH1jCxp4QBOkPpCdiXn1/t96q9CIeuu81oeFTSApfDD 6Z2nr9VP5cWBbRiC7IKSPlOwy3ulNixad2kAkYHcWJDroVJDI2cPK4hIYKM1x/ANrzEp HneD1gd2nANegNWLtF4jMpIx/5tGxu7OMafMcMBMccOwkPa8xlrj5kYXv+dq88ErSljd 8wOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694561548; x=1695166348; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=b7pmoitH+AiMJiFKg+FFb7MBDaxehp8zsVFc9eCNZDA=; b=eXu+bVwnMHR78q4hIPfWJfbi9vDIwcizP+zFflKDPABDS9JQkA4UA+0iDjmGHnGGY+ DU//C+Cl3a2NoJfVdVpTvpuWy13iWAvAjKVjaCFy1rB+I3H4b0orm3ZHeYuWqif1HsAM L2MbdfWHyhVR6fG0JPn2tcI0ZL/tGf5z6gIALhx52P1obrVNfv5CD+SPdAS4gxt1eNtL UupuBZseKHmZwvHrGl7QTS5tnvKGMCF9bePT6iSfwHZWXXCjARmWkcCEuJL2EXB9aoaa Enxk7xc/K9LPEDfvgvfhCP0e8Gv91WRAfbvqJJyutsF74qRLHJ1swRka5+O/f2StU2hx sSzw== X-Gm-Message-State: AOJu0YwhkuPh/QdbA/b+1RboEy+g4GwWG92ZRp9iqk59+GiED3DzETEI VSljaLMMuGgGQjFWUKBiyU1x6+DO2V/lWQ== X-Google-Smtp-Source: AGHT+IFThpaJAygVmnfCDVgda1dXZidiOaH65Th8g12rqEYx1oUdFluGvTZLZWQP6XzhTK8+bYOJwQ== X-Received: by 2002:a2e:96cf:0:b0:2bc:d38e:65ab with SMTP id d15-20020a2e96cf000000b002bcd38e65abmr1012816ljj.37.1694561547933; Tue, 12 Sep 2023 16:32:27 -0700 (PDT) Received: from localhost ([212.203.98.42]) by smtp.gmail.com with ESMTPSA id lj16-20020a170906f9d000b00992d0de8762sm7432326ejb.216.2023.09.12.16.32.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 16:32:27 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet , Puranjay Mohan Subject: [PATCH bpf-next v3 15/17] libbpf: Add support for custom exception callbacks Date: Wed, 13 Sep 2023 01:32:12 +0200 Message-ID: <20230912233214.1518551-16-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230912233214.1518551-1-memxor@gmail.com> References: <20230912233214.1518551-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6677; i=memxor@gmail.com; h=from:subject; bh=r36Y41HKrfmjdJkM3HT2I8pc/IZkGUAxlE7zci8mNdQ=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBlAPSu0C6fe+4yTY8XuEaBJ5uEBlL58uJdCvbmD +T1r8WYnv2JAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZQD0rgAKCRBM4MiGSL8R yjtzD/0dUtfvOd5m45kXvglJ2KQWeQbZBFgYDEYldNPOCG/YLYwvc/6r1bw5Qxe+S28crS7j7Ja lmBqcS3s91s0bk7eka5uTs7aRGtG95XHADAuMm1UjD/Z8ZeAAtDErMBMjCUYpDOUF6RymAFPNan Zmxsv6yiggqP0TjPxom6mpLrzNTRD25+CKQC6SvakbZ2RMAjprZLQI5WRzOHgHAY9FuFHwy2ODS v9RgEYhgJTUF+ngoHqgsdRrQ6vRlMXf3NljhYyFWZOeBpeKY9Nj2xRWwdy6bF+fFM3v6jfXx25t KJR8//s09/OeM1aJcKgpbXZWOxsx917mDY5vy43Glh9379AcK1T+czJIcbOCbQbxm5XLo1KyTvZ LAWhZ9VYPfH5TkYytjD7jwB7Wsy5z+Y+8guJjBFe1n9gu33w+cq3ddOV7Jw5+KH19cq7jf52+X2 zPPsJDVZpFOzpA7D1XaLfjf139KwTDuopsMQgFhuxhYGsyUvvevPVnm93irX6xxqjAz/txZ61qn 4ig+V5jhEOOtIhKcA7ZPB+ide5YRnwmn9Z3s57bnWwR2kyBIJPUN0kgCIZF0dZ9liQqU2Vb6dH0 bl1s4AGo/f2hXjxJv41JqYQ4QA5WNwIUt1MEIgh6gFZ+7jqJaiGUm6S+nYbOqBoxC6jtPtyatXC RP/9BLoGnp4mEFA== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net Add support to libbpf to append exception callbacks when loading a program. The exception callback is found by discovering the declaration tag 'exception_callback:' and finding the callback in the value of the tag. The process is done in two steps. First, for each main program, the bpf_object__sanitize_and_load_btf function finds and marks its corresponding exception callback as defined by the declaration tag on it. Second, bpf_object__reloc_code is modified to append the indicated exception callback at the end of the instruction iteration (since exception callback will never be appended in that loop, as it is not directly referenced). Signed-off-by: Kumar Kartikeya Dwivedi --- tools/lib/bpf/libbpf.c | 114 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 109 insertions(+), 5 deletions(-) diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index afc07a8f7dc7..3a6108e3238b 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -436,9 +436,11 @@ struct bpf_program { int fd; bool autoload; bool autoattach; + bool sym_global; bool mark_btf_static; enum bpf_prog_type type; enum bpf_attach_type expected_attach_type; + int exception_cb_idx; int prog_ifindex; __u32 attach_btf_obj_fd; @@ -765,6 +767,7 @@ bpf_object__init_prog(struct bpf_object *obj, struct bpf_program *prog, prog->type = BPF_PROG_TYPE_UNSPEC; prog->fd = -1; + prog->exception_cb_idx = -1; /* libbpf's convention for SEC("?abc...") is that it's just like * SEC("abc...") but the corresponding bpf_program starts out with @@ -871,14 +874,16 @@ bpf_object__add_programs(struct bpf_object *obj, Elf_Data *sec_data, if (err) return err; + if (ELF64_ST_BIND(sym->st_info) != STB_LOCAL) + prog->sym_global = true; + /* if function is a global/weak symbol, but has restricted * (STV_HIDDEN or STV_INTERNAL) visibility, mark its BTF FUNC * as static to enable more permissive BPF verification mode * with more outside context available to BPF verifier */ - if (ELF64_ST_BIND(sym->st_info) != STB_LOCAL - && (ELF64_ST_VISIBILITY(sym->st_other) == STV_HIDDEN - || ELF64_ST_VISIBILITY(sym->st_other) == STV_INTERNAL)) + if (prog->sym_global && (ELF64_ST_VISIBILITY(sym->st_other) == STV_HIDDEN + || ELF64_ST_VISIBILITY(sym->st_other) == STV_INTERNAL)) prog->mark_btf_static = true; nr_progs++; @@ -3142,6 +3147,86 @@ static int bpf_object__sanitize_and_load_btf(struct bpf_object *obj) } } + if (!kernel_supports(obj, FEAT_BTF_DECL_TAG)) + goto skip_exception_cb; + for (i = 0; i < obj->nr_programs; i++) { + struct bpf_program *prog = &obj->programs[i]; + int j, k, n; + + if (prog_is_subprog(obj, prog)) + continue; + n = btf__type_cnt(obj->btf); + for (j = 1; j < n; j++) { + const char *str = "exception_callback:", *name; + size_t len = strlen(str); + struct btf_type *t; + + t = btf_type_by_id(obj->btf, j); + if (!btf_is_decl_tag(t) || btf_decl_tag(t)->component_idx != -1) + continue; + + name = btf__str_by_offset(obj->btf, t->name_off); + if (strncmp(name, str, len)) + continue; + + t = btf_type_by_id(obj->btf, t->type); + if (!btf_is_func(t) || btf_func_linkage(t) != BTF_FUNC_GLOBAL) { + pr_warn("prog '%s': exception_callback: decl tag not applied to the main program\n", + prog->name); + return -EINVAL; + } + if (strcmp(prog->name, btf__str_by_offset(obj->btf, t->name_off))) + continue; + /* Multiple callbacks are specified for the same prog, + * the verifier will eventually return an error for this + * case, hence simply skip appending a subprog. + */ + if (prog->exception_cb_idx >= 0) { + prog->exception_cb_idx = -1; + break; + } + + name += len; + if (str_is_empty(name)) { + pr_warn("prog '%s': exception_callback: decl tag contains empty value\n", + prog->name); + return -EINVAL; + } + + for (k = 0; k < obj->nr_programs; k++) { + struct bpf_program *subprog = &obj->programs[k]; + + if (!prog_is_subprog(obj, subprog)) + continue; + if (strcmp(name, subprog->name)) + continue; + /* Enforce non-hidden, as from verifier point of + * view it expects global functions, whereas the + * mark_btf_static fixes up linkage as static. + */ + if (!subprog->sym_global || subprog->mark_btf_static) { + pr_warn("prog '%s': exception callback %s must be a global non-hidden function\n", + prog->name, subprog->name); + return -EINVAL; + } + /* Let's see if we already saw a static exception callback with the same name */ + if (prog->exception_cb_idx >= 0) { + pr_warn("prog '%s': multiple subprogs with same name as exception callback '%s'\n", + prog->name, subprog->name); + return -EINVAL; + } + prog->exception_cb_idx = k; + break; + } + + if (prog->exception_cb_idx >= 0) + continue; + pr_warn("prog '%s': cannot find exception callback '%s'\n", prog->name, name); + return -ENOENT; + } + } +skip_exception_cb: + sanitize = btf_needs_sanitization(obj); if (sanitize) { const void *raw_data; @@ -6270,10 +6355,10 @@ static int bpf_object__reloc_code(struct bpf_object *obj, struct bpf_program *main_prog, struct bpf_program *prog) { - size_t sub_insn_idx, insn_idx, new_cnt; + size_t sub_insn_idx, insn_idx; struct bpf_program *subprog; - struct bpf_insn *insns, *insn; struct reloc_desc *relo; + struct bpf_insn *insn; int err; err = reloc_prog_func_and_line_info(obj, main_prog, prog); @@ -6582,6 +6667,25 @@ bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_path) prog->name, err); return err; } + + /* Now, also append exception callback if it has not been done already. */ + if (prog->exception_cb_idx >= 0) { + struct bpf_program *subprog = &obj->programs[prog->exception_cb_idx]; + + /* Calling exception callback directly is disallowed, which the + * verifier will reject later. In case it was processed already, + * we can skip this step, otherwise for all other valid cases we + * have to append exception callback now. + */ + if (subprog->sub_insn_off == 0) { + err = bpf_object__append_subprog_code(obj, prog, subprog); + if (err) + return err; + err = bpf_object__reloc_code(obj, prog, subprog); + if (err) + return err; + } + } } /* Process data relos for main programs */ for (i = 0; i < obj->nr_programs; i++) { From patchwork Tue Sep 12 23:32:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13382321 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D1A2F11CAD for ; Tue, 12 Sep 2023 23:32:31 +0000 (UTC) Received: from mail-ej1-x643.google.com (mail-ej1-x643.google.com [IPv6:2a00:1450:4864:20::643]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F7AC10FE for ; Tue, 12 Sep 2023 16:32:31 -0700 (PDT) Received: by mail-ej1-x643.google.com with SMTP id a640c23a62f3a-9ad8add163cso137077266b.2 for ; Tue, 12 Sep 2023 16:32:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1694561549; x=1695166349; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DOHf7UvNL/MfIQqx3fKPtHAMQ0tFRlyKasyS8VWjH/k=; b=Pv+PMv+0jbjieCMS5tji618ss2NcxZh2yeCSyhPy2vj8qrIRgPGyhmjU5+QvWPlikW YZ5ZcEjZ3lZExqF7bRi5UcDchwz2AlKk/s0EBB85ZsjcZCdCRtoqoFAFqzzgeZSeWZTJ j96dHyOYgAQ4guB/UIzF9GDBDVo5/L6GqVHyFSSIqBieGhMYp6PSfXB7l2ktGl1XH+E2 qtmDje5gfXw1DOIc2F7gJTsB/yCVO+cfDfEHI0qECQDr/wrIqvtV5njkV1q1UQiPFG0u c9trJGhMO1dfY7jD/un85WZsJ/WbcGySJh3BthHmq9MVb/3JuHQhPJy/uuMIT2sQN6Kb rGIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694561549; x=1695166349; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DOHf7UvNL/MfIQqx3fKPtHAMQ0tFRlyKasyS8VWjH/k=; b=XKbBqJkZub7cWP1gA2cOFnshsOp1WnO5e8EuIL10+HmJd2WaEZAh0vdH+89uvVmSxo IiFEifoOSQNAqbwupOZV19kJkzaixyo09XbQfC70PXM2QKU4sn/c3SlY4XDdrq4wH/UG d+SPU47cmm+4JW+s5wNImWPj8+ZiRmi81HolutSveifmT1E3XDjx8UlD0wfr3RuBtl+p P94TKRQ3K+vOpoIj+UiIkL/UbE81iLXJs2Ru03toHwPYrfhH4i9cQm2hqzpL0EZPmBsj 5l9fb+w4CETwsywr7XAJtGGSF7izmfSKbZ7j8RM5GzXp5M+ndTfnYW9wqv5pqXBJz+pv fMpQ== X-Gm-Message-State: AOJu0YzRsRm5Ux2hSuVf2XYioovzVf8fdhT1KvRS23U0E3LOl7s8VucH SENEa8jBfJdHaFOjLEwM/2b9wsgBXLlirw== X-Google-Smtp-Source: AGHT+IFcbvjsGa9p5uJR5y/awxg7WJUq4ImmxHRbHaAOEl5FEpUkT++TpP4xgLelkQk7QTzCLlbTsw== X-Received: by 2002:a17:907:a053:b0:9a1:b144:30f0 with SMTP id gz19-20020a170907a05300b009a1b14430f0mr515211ejc.53.1694561549105; Tue, 12 Sep 2023 16:32:29 -0700 (PDT) Received: from localhost ([212.203.98.42]) by smtp.gmail.com with ESMTPSA id f20-20020a170906825400b0099bc2d1429csm7460524ejx.72.2023.09.12.16.32.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 16:32:28 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet , Puranjay Mohan Subject: [PATCH bpf-next v3 16/17] selftests/bpf: Add BPF assertion macros Date: Wed, 13 Sep 2023 01:32:13 +0200 Message-ID: <20230912233214.1518551-17-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230912233214.1518551-1-memxor@gmail.com> References: <20230912233214.1518551-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=9737; i=memxor@gmail.com; h=from:subject; bh=S7OPOB8h/TK+lGTRLmoipCDNro3S4BIBa8ptAz+GI+M=; b=kA0DAAgBTODIhki/EcoByyZiAGUA9K6g8tnxWLLEZdXXyhx4y82HncUEG3mRcdkwNht42FRDP YkCMwQAAQgAHRYhBEu+Kn4G7PnVgjxhEUzgyIZIvxHKBQJlAPSuAAoJEEzgyIZIvxHKEQ0P/jpS 61ILnbkoYWfsrb9gQmLoK/+TuKbjSEJ6tFo6MvPSZAVly/FuhDvOxv7zd4i95pwPZSlU9XH+xPl b+ZCFeB4G859PFHYN9tH/xXoR1/NMl1FRnrcXmeRz+oYYS/5EVAlsUCf/oIhZdxXe4YRI29H8l/ 12sQ7AdV5HRnS14KeFBw+TQ9rCOsXQizljsWW+ilDHEcsbuVdagedPI3lFFJ8ZjGgaearfWlfdc pQRlTeRReDTS79m0v+HGrOL0ttGxuwQcNyN4IkqasiMTojQVEUuhMOSv3a4zx4QlY7G7EkP4wDj iswaXPXF9Yx1tQux1bcFLQjWjPpBfVWTIs8Ke1yIF2VjFJZ3PwwQIihi9TRhqvQxn9R/Iaa6aUV oS5weCMYNYB7cOh0x9PJRuzhikZ22w0kbv07ioHsFjtfUW1z9VWIk3bYqHdNVWDwlQ//tO+5tSv XtK9qrVLC+enWqJLRZlKCe6TB2i3FFD7C1xlz30HA3KPc0XB/r/CnrLYPr7XRloti4CcxKCVmok xBVOrMShvH9JwzwYBQnmUr5WBqKegCh6kiv1hBwvW6YyfF7PZN8IEMRHeJ/2Y56sj/XPkGADqck /gdzNgo46u2jlc2K4nT2mnuYOpJ85tL898S2q+MfrHvV4n0ngjn+acKb5ID5+Rud5tdgv1L8Xzb TPSaD X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net Add macros implementing an 'assert' statement primitive using macros, built on top of the BPF exceptions support introduced in previous patches. The bpf_assert_*_with variants allow supplying a value which can the be inspected within the exception handler to signify the assert statement that led to the program being terminated abruptly, or be returned by the default exception handler. Note that only 64-bit scalar values are supported with these assertion macros, as during testing I found other cases quite unreliable in presence of compiler shifts/manipulations extracting the value of the right width from registers scrubbing the verifier's bounds information and knowledge about the value in the register. Thus, it is easier to reliably support this feature with only the full register width, and support both signed and unsigned variants. The bpf_assert_range is interesting in particular, which clamps the value in the [begin, end] (both inclusive) range within verifier state, and emits a check for the same at runtime. Signed-off-by: Kumar Kartikeya Dwivedi --- .../testing/selftests/bpf/bpf_experimental.h | 243 ++++++++++++++++++ 1 file changed, 243 insertions(+) diff --git a/tools/testing/selftests/bpf/bpf_experimental.h b/tools/testing/selftests/bpf/bpf_experimental.h index 9a87170524ce..9aa29564bd74 100644 --- a/tools/testing/selftests/bpf/bpf_experimental.h +++ b/tools/testing/selftests/bpf/bpf_experimental.h @@ -207,4 +207,247 @@ extern void bpf_throw(u64 cookie) __ksym; */ #define __exception_cb(name) __attribute__((btf_decl_tag("exception_callback:" #name))) +#define __bpf_assert_signed(x) _Generic((x), \ + unsigned long: 0, \ + unsigned long long: 0, \ + signed long: 1, \ + signed long long: 1 \ +) + +#define __bpf_assert_check(LHS, op, RHS) \ + _Static_assert(sizeof(&(LHS)), "1st argument must be an lvalue expression"); \ + _Static_assert(sizeof(LHS) == 8, "Only 8-byte integers are supported\n"); \ + _Static_assert(__builtin_constant_p(__bpf_assert_signed(LHS)), "internal static assert"); \ + _Static_assert(__builtin_constant_p((RHS)), "2nd argument must be a constant expression") + +#define __bpf_assert(LHS, op, cons, RHS, VAL) \ + ({ \ + (void)bpf_throw; \ + asm volatile ("if %[lhs] " op " %[rhs] goto +2; r1 = %[value]; call bpf_throw" \ + : : [lhs] "r"(LHS), [rhs] cons(RHS), [value] "ri"(VAL) : ); \ + }) + +#define __bpf_assert_op_sign(LHS, op, cons, RHS, VAL, supp_sign) \ + ({ \ + __bpf_assert_check(LHS, op, RHS); \ + if (__bpf_assert_signed(LHS) && !(supp_sign)) \ + __bpf_assert(LHS, "s" #op, cons, RHS, VAL); \ + else \ + __bpf_assert(LHS, #op, cons, RHS, VAL); \ + }) + +#define __bpf_assert_op(LHS, op, RHS, VAL, supp_sign) \ + ({ \ + if (sizeof(typeof(RHS)) == 8) { \ + const typeof(RHS) rhs_var = (RHS); \ + __bpf_assert_op_sign(LHS, op, "r", rhs_var, VAL, supp_sign); \ + } else { \ + __bpf_assert_op_sign(LHS, op, "i", RHS, VAL, supp_sign); \ + } \ + }) + +/* Description + * Assert that a conditional expression is true. + * Returns + * Void. + * Throws + * An exception with the value zero when the assertion fails. + */ +#define bpf_assert(cond) if (!(cond)) bpf_throw(0); + +/* Description + * Assert that a conditional expression is true. + * Returns + * Void. + * Throws + * An exception with the specified value when the assertion fails. + */ +#define bpf_assert_with(cond, value) if (!(cond)) bpf_throw(value); + +/* Description + * Assert that LHS is equal to RHS. This statement updates the known value + * of LHS during verification. Note that RHS must be a constant value, and + * must fit within the data type of LHS. + * Returns + * Void. + * Throws + * An exception with the value zero when the assertion fails. + */ +#define bpf_assert_eq(LHS, RHS) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, ==, RHS, 0, true); \ + }) + +/* Description + * Assert that LHS is equal to RHS. This statement updates the known value + * of LHS during verification. Note that RHS must be a constant value, and + * must fit within the data type of LHS. + * Returns + * Void. + * Throws + * An exception with the specified value when the assertion fails. + */ +#define bpf_assert_eq_with(LHS, RHS, value) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, ==, RHS, value, true); \ + }) + +/* Description + * Assert that LHS is less than RHS. This statement updates the known + * bounds of LHS during verification. Note that RHS must be a constant + * value, and must fit within the data type of LHS. + * Returns + * Void. + * Throws + * An exception with the value zero when the assertion fails. + */ +#define bpf_assert_lt(LHS, RHS) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, <, RHS, 0, false); \ + }) + +/* Description + * Assert that LHS is less than RHS. This statement updates the known + * bounds of LHS during verification. Note that RHS must be a constant + * value, and must fit within the data type of LHS. + * Returns + * Void. + * Throws + * An exception with the specified value when the assertion fails. + */ +#define bpf_assert_lt_with(LHS, RHS, value) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, <, RHS, value, false); \ + }) + +/* Description + * Assert that LHS is greater than RHS. This statement updates the known + * bounds of LHS during verification. Note that RHS must be a constant + * value, and must fit within the data type of LHS. + * Returns + * Void. + * Throws + * An exception with the value zero when the assertion fails. + */ +#define bpf_assert_gt(LHS, RHS) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, >, RHS, 0, false); \ + }) + +/* Description + * Assert that LHS is greater than RHS. This statement updates the known + * bounds of LHS during verification. Note that RHS must be a constant + * value, and must fit within the data type of LHS. + * Returns + * Void. + * Throws + * An exception with the specified value when the assertion fails. + */ +#define bpf_assert_gt_with(LHS, RHS, value) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, >, RHS, value, false); \ + }) + +/* Description + * Assert that LHS is less than or equal to RHS. This statement updates the + * known bounds of LHS during verification. Note that RHS must be a + * constant value, and must fit within the data type of LHS. + * Returns + * Void. + * Throws + * An exception with the value zero when the assertion fails. + */ +#define bpf_assert_le(LHS, RHS) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, <=, RHS, 0, false); \ + }) + +/* Description + * Assert that LHS is less than or equal to RHS. This statement updates the + * known bounds of LHS during verification. Note that RHS must be a + * constant value, and must fit within the data type of LHS. + * Returns + * Void. + * Throws + * An exception with the specified value when the assertion fails. + */ +#define bpf_assert_le_with(LHS, RHS, value) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, <=, RHS, value, false); \ + }) + +/* Description + * Assert that LHS is greater than or equal to RHS. This statement updates + * the known bounds of LHS during verification. Note that RHS must be a + * constant value, and must fit within the data type of LHS. + * Returns + * Void. + * Throws + * An exception with the value zero when the assertion fails. + */ +#define bpf_assert_ge(LHS, RHS) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, >=, RHS, 0, false); \ + }) + +/* Description + * Assert that LHS is greater than or equal to RHS. This statement updates + * the known bounds of LHS during verification. Note that RHS must be a + * constant value, and must fit within the data type of LHS. + * Returns + * Void. + * Throws + * An exception with the specified value when the assertion fails. + */ +#define bpf_assert_ge_with(LHS, RHS, value) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, >=, RHS, value, false); \ + }) + +/* Description + * Assert that LHS is in the range [BEG, END] (inclusive of both). This + * statement updates the known bounds of LHS during verification. Note + * that both BEG and END must be constant values, and must fit within the + * data type of LHS. + * Returns + * Void. + * Throws + * An exception with the value zero when the assertion fails. + */ +#define bpf_assert_range(LHS, BEG, END) \ + ({ \ + _Static_assert(BEG <= END, "BEG must be <= END"); \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, >=, BEG, 0, false); \ + __bpf_assert_op(LHS, <=, END, 0, false); \ + }) + +/* Description + * Assert that LHS is in the range [BEG, END] (inclusive of both). This + * statement updates the known bounds of LHS during verification. Note + * that both BEG and END must be constant values, and must fit within the + * data type of LHS. + * Returns + * Void. + * Throws + * An exception with the specified value when the assertion fails. + */ +#define bpf_assert_range_with(LHS, BEG, END, value) \ + ({ \ + _Static_assert(BEG <= END, "BEG must be <= END"); \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, >=, BEG, value, false); \ + __bpf_assert_op(LHS, <=, END, value, false); \ + }) + #endif From patchwork Tue Sep 12 23:32:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13382322 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 521ED11CAD for ; Tue, 12 Sep 2023 23:32:33 +0000 (UTC) Received: from mail-ed1-x542.google.com (mail-ed1-x542.google.com [IPv6:2a00:1450:4864:20::542]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5CBE810FE for ; Tue, 12 Sep 2023 16:32:32 -0700 (PDT) Received: by mail-ed1-x542.google.com with SMTP id 4fb4d7f45d1cf-52bcb8b199aso7990839a12.3 for ; Tue, 12 Sep 2023 16:32:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1694561550; x=1695166350; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kgIJrPG5nzIqraflOkVUvndB5enStjY5YsMrnJ6TL2k=; b=JH7u94oQ/YnL2UOfHGa3Gcsh28lQE+uqstWa3PaIWp3Uo7Alk0DiL1WcQ2S1fLqHLP /G2UQreTQTGFeXDqYaW7mArXug9psloUA6V0c03WK0SzRmowrNwK9eeJMjkY/qRgBWQz ehEWFHbhXFcAgi9oliZ7Wf0alumxaLFEpWQllWL0gMaCG5gbzLe1gATF4l/am5PI26UI WEweY8ODd3g8zbwsmz5lKD7Sp7rI7e3MMfGUf7htQKBg/vwE0zD0rooW5SJmI4Rw8ynq PLNsks40LDX8CKw5VNcethIXPkdSlUCzTSxUYZmzg40cpjwrlNU/HQbrfOG4RDM4r70v /L2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694561550; x=1695166350; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kgIJrPG5nzIqraflOkVUvndB5enStjY5YsMrnJ6TL2k=; b=Bv0TAuyCQRKcAPr8HecTwl4N2cJ6V+YG8dN/bc4Lw4rvf1Lf781hn6Vhxh3AUJZIJ7 DZtmCLmLimOeOs3LIXI1q6ufMaNyXfzDW9Fr5Hi3xt1kQW5DwBX7EtIEs92cmdaGT5li 5vd+QGZaPCq7tNtKhpcuZPxuU8Zrgz2iMu+CmWjQZy0VebygF9Ghh7j8ZcqGhA9JaERu BwpRv6MiIN3PtPWYH+op8mEjQxYOkRGfQ9iDWBzw76ShphKD+6/UEr/cHupq/hQqEmX4 RsAOkfMapqHNaa9Jier8N9V5hNI34ROzqf7sZyabjwtpE02PMmlJ+wwPKsK+6WeQmNuq /w9Q== X-Gm-Message-State: AOJu0YwaFf/lCWZlHRqtqQiX5s6NHyqVFLzC2zZYaHocZuIznZ5U3DIC KLAtogEuDEt51Nd1ecsLN/evwnqI4FAzug== X-Google-Smtp-Source: AGHT+IGAQLR80i50C6P5++nqobyFIRxBqFUZDmmmbDljkk0nFqK2MFfRbbj3APGkBEZmWksxnJGErw== X-Received: by 2002:aa7:c48b:0:b0:527:3a95:5bea with SMTP id m11-20020aa7c48b000000b005273a955beamr805627edq.32.1694561550255; Tue, 12 Sep 2023 16:32:30 -0700 (PDT) Received: from localhost ([212.203.98.42]) by smtp.gmail.com with ESMTPSA id ek10-20020a056402370a00b0052f3471ccf6sm4721104edb.6.2023.09.12.16.32.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Sep 2023 16:32:29 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet , Puranjay Mohan Subject: [PATCH bpf-next v3 17/17] selftests/bpf: Add tests for BPF exceptions Date: Wed, 13 Sep 2023 01:32:14 +0200 Message-ID: <20230912233214.1518551-18-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230912233214.1518551-1-memxor@gmail.com> References: <20230912233214.1518551-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=41621; i=memxor@gmail.com; h=from:subject; bh=i2yOIfPLus0oQXaEJMh50F+au0OMrqkcflrccOvChrg=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBlAPSuAmIlIf9VliajKSYPAqacJXcbJbEUyn0TE n985NXUk/qJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZQD0rgAKCRBM4MiGSL8R yt5qD/9swsFiEyKB4iCAiJDycSDhdxcvRDwogOqEFDY2loE9ohehME3w9m2NtqjHlKnfl6qCC7g qziHZbwkWi5I7RMFb6AGAIolc8WpCIiGfCXvLt5pNkv2ki3c9VykCnPCknYcwI9+M7y7XY/DPeb 10GSz7TLq1BPlPFY5ztIbEso0VHhvmF2X341hKxoFYAsA8pFRkC6iIqtZHAWOOAToN4zouuXv7A s93ExzCWM8cuwEqiALsYC+aMv3RoM+aIOehKq4eUjPtw0rznOXM13PNSSLqQUuP9BpmiYjS5R2/ 0ut/qnqIDm8HVjrROhejcG7U+DsFJotr4cHDC3KMulJpgWJG/Gx/FnPEXC8r6WsDfhe1vsYTJUs xMHPCH6vm0bN8i9T87EUkJdZQ75yTOtZol9ndGWiqba7RnigItSI2NOPUZQOo7zRuwx3EHOGy+E f1t+BdlaPg2AArp+ajSn9QTyZOOoYvTQ/tBFnAYmVGtvjNJKpUVgMfJLjwMUn738FPDI8Pqf3oN z9zU86kw3lBCr7i08f1qNduXMUmquWd3FHj8VmeL1u/WKI7KfSc6MVUon1fObexU5kP+jSO6SK7 cnBpiTPWQ8IeKGsJjQw7JhkktN9WCw5RhojdT55ynZ7x/d7cNfEVLqZLViSQgcznxPyDNVfiBvA ajRfCqPyvlxJLWA== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net Add selftests to cover success and failure cases of API usage, runtime behavior and invariants that need to be maintained for implementation correctness. Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/DENYLIST.aarch64 | 1 + tools/testing/selftests/bpf/DENYLIST.s390x | 1 + .../selftests/bpf/prog_tests/exceptions.c | 408 ++++++++++++++++++ .../testing/selftests/bpf/progs/exceptions.c | 368 ++++++++++++++++ .../selftests/bpf/progs/exceptions_assert.c | 135 ++++++ .../selftests/bpf/progs/exceptions_ext.c | 72 ++++ .../selftests/bpf/progs/exceptions_fail.c | 347 +++++++++++++++ 7 files changed, 1332 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/exceptions.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_assert.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_ext.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_fail.c diff --git a/tools/testing/selftests/bpf/DENYLIST.aarch64 b/tools/testing/selftests/bpf/DENYLIST.aarch64 index 7f768d335698..f5065576cae9 100644 --- a/tools/testing/selftests/bpf/DENYLIST.aarch64 +++ b/tools/testing/selftests/bpf/DENYLIST.aarch64 @@ -1,5 +1,6 @@ bpf_cookie/multi_kprobe_attach_api # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3 bpf_cookie/multi_kprobe_link_api # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3 +exceptions # JIT does not support calling kfunc bpf_throw: -524 fexit_sleep # The test never returns. The remaining tests cannot start. kprobe_multi_bench_attach # bpf_program__attach_kprobe_multi_opts unexpected error: -95 kprobe_multi_test/attach_api_addrs # bpf_program__attach_kprobe_multi_opts unexpected error: -95 diff --git a/tools/testing/selftests/bpf/DENYLIST.s390x b/tools/testing/selftests/bpf/DENYLIST.s390x index 5061d9e24c16..ce6f291665cf 100644 --- a/tools/testing/selftests/bpf/DENYLIST.s390x +++ b/tools/testing/selftests/bpf/DENYLIST.s390x @@ -6,6 +6,7 @@ bpf_loop # attaches to __x64_sys_nanosleep cgrp_local_storage # prog_attach unexpected error: -524 (trampoline) dynptr/test_dynptr_skb_data dynptr/test_skb_readonly +exceptions # JIT does not support calling kfunc bpf_throw (exceptions) fexit_sleep # fexit_skel_load fexit skeleton failed (trampoline) get_stack_raw_tp # user_stack corrupted user stack (no backchain userspace) iters/testmod_seq* # s390x doesn't support kfuncs in modules yet diff --git a/tools/testing/selftests/bpf/prog_tests/exceptions.c b/tools/testing/selftests/bpf/prog_tests/exceptions.c new file mode 100644 index 000000000000..5663e427dc00 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/exceptions.c @@ -0,0 +1,408 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include + +#include "exceptions.skel.h" +#include "exceptions_ext.skel.h" +#include "exceptions_fail.skel.h" +#include "exceptions_assert.skel.h" + +static char log_buf[1024 * 1024]; + +static void test_exceptions_failure(void) +{ + RUN_TESTS(exceptions_fail); +} + +static void test_exceptions_success(void) +{ + LIBBPF_OPTS(bpf_test_run_opts, ropts, + .data_in = &pkt_v4, + .data_size_in = sizeof(pkt_v4), + .repeat = 1, + ); + struct exceptions_ext *eskel = NULL; + struct exceptions *skel; + int ret; + + skel = exceptions__open(); + if (!ASSERT_OK_PTR(skel, "exceptions__open")) + return; + + ret = exceptions__load(skel); + if (!ASSERT_OK(ret, "exceptions__load")) + goto done; + + if (!ASSERT_OK(bpf_map_update_elem(bpf_map__fd(skel->maps.jmp_table), &(int){0}, + &(int){bpf_program__fd(skel->progs.exception_tail_call_target)}, BPF_ANY), + "bpf_map_update_elem jmp_table")) + goto done; + +#define RUN_SUCCESS(_prog, return_val) \ + if (!test__start_subtest(#_prog)) goto _prog##_##return_val; \ + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs._prog), &ropts); \ + ASSERT_OK(ret, #_prog " prog run ret"); \ + ASSERT_EQ(ropts.retval, return_val, #_prog " prog run retval"); \ + _prog##_##return_val: + + RUN_SUCCESS(exception_throw_always_1, 64); + RUN_SUCCESS(exception_throw_always_2, 32); + RUN_SUCCESS(exception_throw_unwind_1, 16); + RUN_SUCCESS(exception_throw_unwind_2, 32); + RUN_SUCCESS(exception_throw_default, 0); + RUN_SUCCESS(exception_throw_default_value, 5); + RUN_SUCCESS(exception_tail_call, 24); + RUN_SUCCESS(exception_ext, 0); + RUN_SUCCESS(exception_ext_mod_cb_runtime, 35); + RUN_SUCCESS(exception_throw_subprog, 1); + RUN_SUCCESS(exception_assert_nz_gfunc, 1); + RUN_SUCCESS(exception_assert_zero_gfunc, 1); + RUN_SUCCESS(exception_assert_neg_gfunc, 1); + RUN_SUCCESS(exception_assert_pos_gfunc, 1); + RUN_SUCCESS(exception_assert_negeq_gfunc, 1); + RUN_SUCCESS(exception_assert_poseq_gfunc, 1); + RUN_SUCCESS(exception_assert_nz_gfunc_with, 1); + RUN_SUCCESS(exception_assert_zero_gfunc_with, 1); + RUN_SUCCESS(exception_assert_neg_gfunc_with, 1); + RUN_SUCCESS(exception_assert_pos_gfunc_with, 1); + RUN_SUCCESS(exception_assert_negeq_gfunc_with, 1); + RUN_SUCCESS(exception_assert_poseq_gfunc_with, 1); + RUN_SUCCESS(exception_bad_assert_nz_gfunc, 0); + RUN_SUCCESS(exception_bad_assert_zero_gfunc, 0); + RUN_SUCCESS(exception_bad_assert_neg_gfunc, 0); + RUN_SUCCESS(exception_bad_assert_pos_gfunc, 0); + RUN_SUCCESS(exception_bad_assert_negeq_gfunc, 0); + RUN_SUCCESS(exception_bad_assert_poseq_gfunc, 0); + RUN_SUCCESS(exception_bad_assert_nz_gfunc_with, 100); + RUN_SUCCESS(exception_bad_assert_zero_gfunc_with, 105); + RUN_SUCCESS(exception_bad_assert_neg_gfunc_with, 200); + RUN_SUCCESS(exception_bad_assert_pos_gfunc_with, 0); + RUN_SUCCESS(exception_bad_assert_negeq_gfunc_with, 101); + RUN_SUCCESS(exception_bad_assert_poseq_gfunc_with, 99); + RUN_SUCCESS(exception_assert_range, 1); + RUN_SUCCESS(exception_assert_range_with, 1); + RUN_SUCCESS(exception_bad_assert_range, 0); + RUN_SUCCESS(exception_bad_assert_range_with, 10); + +#define RUN_EXT(load_ret, attach_err, expr, msg, after_link) \ + { \ + LIBBPF_OPTS(bpf_object_open_opts, o, .kernel_log_buf = log_buf, \ + .kernel_log_size = sizeof(log_buf), \ + .kernel_log_level = 2); \ + exceptions_ext__destroy(eskel); \ + eskel = exceptions_ext__open_opts(&o); \ + struct bpf_program *prog = NULL; \ + struct bpf_link *link = NULL; \ + if (!ASSERT_OK_PTR(eskel, "exceptions_ext__open")) \ + goto done; \ + (expr); \ + ASSERT_OK_PTR(bpf_program__name(prog), bpf_program__name(prog)); \ + if (!ASSERT_EQ(exceptions_ext__load(eskel), load_ret, \ + "exceptions_ext__load")) { \ + printf("%s\n", log_buf); \ + goto done; \ + } \ + if (load_ret != 0) { \ + printf("%s\n", log_buf); \ + if (!ASSERT_OK_PTR(strstr(log_buf, msg), "strstr")) \ + goto done; \ + } \ + if (!load_ret && attach_err) { \ + if (!ASSERT_ERR_PTR(link = bpf_program__attach(prog), "attach err")) \ + goto done; \ + } else if (!load_ret) { \ + if (!ASSERT_OK_PTR(link = bpf_program__attach(prog), "attach ok")) \ + goto done; \ + (void)(after_link); \ + bpf_link__destroy(link); \ + } \ + } + + if (test__start_subtest("non-throwing fentry -> exception_cb")) + RUN_EXT(-EINVAL, true, ({ + prog = eskel->progs.pfentry; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_ext_mod_cb_runtime), + "exception_cb_mod"), "set_attach_target")) + goto done; + }), "FENTRY/FEXIT programs cannot attach to exception callback", 0); + + if (test__start_subtest("throwing fentry -> exception_cb")) + RUN_EXT(-EINVAL, true, ({ + prog = eskel->progs.throwing_fentry; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_ext_mod_cb_runtime), + "exception_cb_mod"), "set_attach_target")) + goto done; + }), "FENTRY/FEXIT programs cannot attach to exception callback", 0); + + if (test__start_subtest("non-throwing fexit -> exception_cb")) + RUN_EXT(-EINVAL, true, ({ + prog = eskel->progs.pfexit; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_ext_mod_cb_runtime), + "exception_cb_mod"), "set_attach_target")) + goto done; + }), "FENTRY/FEXIT programs cannot attach to exception callback", 0); + + if (test__start_subtest("throwing fexit -> exception_cb")) + RUN_EXT(-EINVAL, true, ({ + prog = eskel->progs.throwing_fexit; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_ext_mod_cb_runtime), + "exception_cb_mod"), "set_attach_target")) + goto done; + }), "FENTRY/FEXIT programs cannot attach to exception callback", 0); + + if (test__start_subtest("throwing extension (with custom cb) -> exception_cb")) + RUN_EXT(-EINVAL, true, ({ + prog = eskel->progs.throwing_exception_cb_extension; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_ext_mod_cb_runtime), + "exception_cb_mod"), "set_attach_target")) + goto done; + }), "Extension programs cannot attach to exception callback", 0); + + if (test__start_subtest("throwing extension -> global func in exception_cb")) + RUN_EXT(0, false, ({ + prog = eskel->progs.throwing_exception_cb_extension; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_ext_mod_cb_runtime), + "exception_cb_mod_global"), "set_attach_target")) + goto done; + }), "", ({ RUN_SUCCESS(exception_ext_mod_cb_runtime, 131); })); + + if (test__start_subtest("throwing extension (with custom cb) -> global func in exception_cb")) + RUN_EXT(0, false, ({ + prog = eskel->progs.throwing_extension; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_ext), + "exception_ext_global"), "set_attach_target")) + goto done; + }), "", ({ RUN_SUCCESS(exception_ext, 128); })); + + if (test__start_subtest("non-throwing fentry -> non-throwing subprog")) + /* non-throwing fentry -> non-throwing subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.pfentry; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "subprog"), "set_attach_target")) + goto done; + }), "", 0); + + if (test__start_subtest("throwing fentry -> non-throwing subprog")) + /* throwing fentry -> non-throwing subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.throwing_fentry; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "subprog"), "set_attach_target")) + goto done; + }), "", 0); + + if (test__start_subtest("non-throwing fentry -> throwing subprog")) + /* non-throwing fentry -> throwing subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.pfentry; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "throwing_subprog"), "set_attach_target")) + goto done; + }), "", 0); + + if (test__start_subtest("throwing fentry -> throwing subprog")) + /* throwing fentry -> throwing subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.throwing_fentry; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "throwing_subprog"), "set_attach_target")) + goto done; + }), "", 0); + + if (test__start_subtest("non-throwing fexit -> non-throwing subprog")) + /* non-throwing fexit -> non-throwing subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.pfexit; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "subprog"), "set_attach_target")) + goto done; + }), "", 0); + + if (test__start_subtest("throwing fexit -> non-throwing subprog")) + /* throwing fexit -> non-throwing subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.throwing_fexit; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "subprog"), "set_attach_target")) + goto done; + }), "", 0); + + if (test__start_subtest("non-throwing fexit -> throwing subprog")) + /* non-throwing fexit -> throwing subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.pfexit; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "throwing_subprog"), "set_attach_target")) + goto done; + }), "", 0); + + if (test__start_subtest("throwing fexit -> throwing subprog")) + /* throwing fexit -> throwing subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.throwing_fexit; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "throwing_subprog"), "set_attach_target")) + goto done; + }), "", 0); + + /* fmod_ret not allowed for subprog - Check so we remember to handle its + * throwing specification compatibility with target when supported. + */ + if (test__start_subtest("non-throwing fmod_ret -> non-throwing subprog")) + RUN_EXT(-EINVAL, true, ({ + prog = eskel->progs.pfmod_ret; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "subprog"), "set_attach_target")) + goto done; + }), "can't modify return codes of BPF program", 0); + + /* fmod_ret not allowed for subprog - Check so we remember to handle its + * throwing specification compatibility with target when supported. + */ + if (test__start_subtest("non-throwing fmod_ret -> non-throwing global subprog")) + RUN_EXT(-EINVAL, true, ({ + prog = eskel->progs.pfmod_ret; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "global_subprog"), "set_attach_target")) + goto done; + }), "can't modify return codes of BPF program", 0); + + if (test__start_subtest("non-throwing extension -> non-throwing subprog")) + /* non-throwing extension -> non-throwing subprog : BAD (!global) */ + RUN_EXT(-EINVAL, true, ({ + prog = eskel->progs.extension; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "subprog"), "set_attach_target")) + goto done; + }), "subprog() is not a global function", 0); + + if (test__start_subtest("non-throwing extension -> throwing subprog")) + /* non-throwing extension -> throwing subprog : BAD (!global) */ + RUN_EXT(-EINVAL, true, ({ + prog = eskel->progs.extension; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "throwing_subprog"), "set_attach_target")) + goto done; + }), "throwing_subprog() is not a global function", 0); + + if (test__start_subtest("non-throwing extension -> non-throwing subprog")) + /* non-throwing extension -> non-throwing global subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.extension; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "global_subprog"), "set_attach_target")) + goto done; + }), "", 0); + + if (test__start_subtest("non-throwing extension -> throwing global subprog")) + /* non-throwing extension -> throwing global subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.extension; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "throwing_global_subprog"), "set_attach_target")) + goto done; + }), "", 0); + + if (test__start_subtest("throwing extension -> throwing global subprog")) + /* throwing extension -> throwing global subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.throwing_extension; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "throwing_global_subprog"), "set_attach_target")) + goto done; + }), "", 0); + + if (test__start_subtest("throwing extension -> non-throwing global subprog")) + /* throwing extension -> non-throwing global subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.throwing_extension; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "global_subprog"), "set_attach_target")) + goto done; + }), "", 0); + + if (test__start_subtest("non-throwing extension -> main subprog")) + /* non-throwing extension -> main subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.extension; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "exception_throw_subprog"), "set_attach_target")) + goto done; + }), "", 0); + + if (test__start_subtest("throwing extension -> main subprog")) + /* throwing extension -> main subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.throwing_extension; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "exception_throw_subprog"), "set_attach_target")) + goto done; + }), "", 0); + +done: + exceptions_ext__destroy(eskel); + exceptions__destroy(skel); +} + +static void test_exceptions_assertions(void) +{ + RUN_TESTS(exceptions_assert); +} + +void test_exceptions(void) +{ + test_exceptions_success(); + test_exceptions_failure(); + test_exceptions_assertions(); +} diff --git a/tools/testing/selftests/bpf/progs/exceptions.c b/tools/testing/selftests/bpf/progs/exceptions.c new file mode 100644 index 000000000000..2811ee842b01 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/exceptions.c @@ -0,0 +1,368 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include "bpf_misc.h" +#include "bpf_experimental.h" + +#ifndef ETH_P_IP +#define ETH_P_IP 0x0800 +#endif + +struct { + __uint(type, BPF_MAP_TYPE_PROG_ARRAY); + __uint(max_entries, 4); + __uint(key_size, sizeof(__u32)); + __uint(value_size, sizeof(__u32)); +} jmp_table SEC(".maps"); + +static __noinline int static_func(u64 i) +{ + bpf_throw(32); + return i; +} + +__noinline int global2static_simple(u64 i) +{ + static_func(i + 2); + return i - 1; +} + +__noinline int global2static(u64 i) +{ + if (i == ETH_P_IP) + bpf_throw(16); + return static_func(i); +} + +static __noinline int static2global(u64 i) +{ + return global2static(i) + i; +} + +SEC("tc") +int exception_throw_always_1(struct __sk_buff *ctx) +{ + bpf_throw(64); + return 0; +} + +/* In this case, the global func will never be seen executing after call to + * static subprog, hence verifier will DCE the remaining instructions. Ensure we + * are resilient to that. + */ +SEC("tc") +int exception_throw_always_2(struct __sk_buff *ctx) +{ + return global2static_simple(ctx->protocol); +} + +SEC("tc") +int exception_throw_unwind_1(struct __sk_buff *ctx) +{ + return static2global(bpf_ntohs(ctx->protocol)); +} + +SEC("tc") +int exception_throw_unwind_2(struct __sk_buff *ctx) +{ + return static2global(bpf_ntohs(ctx->protocol) - 1); +} + +SEC("tc") +int exception_throw_default(struct __sk_buff *ctx) +{ + bpf_throw(0); + return 1; +} + +SEC("tc") +int exception_throw_default_value(struct __sk_buff *ctx) +{ + bpf_throw(5); + return 1; +} + +SEC("tc") +int exception_tail_call_target(struct __sk_buff *ctx) +{ + bpf_throw(16); + return 0; +} + +static __noinline +int exception_tail_call_subprog(struct __sk_buff *ctx) +{ + volatile int ret = 10; + + bpf_tail_call_static(ctx, &jmp_table, 0); + return ret; +} + +SEC("tc") +int exception_tail_call(struct __sk_buff *ctx) { + volatile int ret = 0; + + ret = exception_tail_call_subprog(ctx); + return ret + 8; +} + +__noinline int exception_ext_global(struct __sk_buff *ctx) +{ + volatile int ret = 0; + + return ret; +} + +static __noinline int exception_ext_static(struct __sk_buff *ctx) +{ + return exception_ext_global(ctx); +} + +SEC("tc") +int exception_ext(struct __sk_buff *ctx) +{ + return exception_ext_static(ctx); +} + +__noinline int exception_cb_mod_global(u64 cookie) +{ + volatile int ret = 0; + + return ret; +} + +/* Example of how the exception callback supplied during verification can still + * introduce extensions by calling to dummy global functions, and alter runtime + * behavior. + * + * Right now we don't allow freplace attachment to exception callback itself, + * but if the need arises this restriction is technically feasible to relax in + * the future. + */ +__noinline int exception_cb_mod(u64 cookie) +{ + return exception_cb_mod_global(cookie) + cookie + 10; +} + +SEC("tc") +__exception_cb(exception_cb_mod) +int exception_ext_mod_cb_runtime(struct __sk_buff *ctx) +{ + bpf_throw(25); + return 0; +} + +__noinline static int subprog(struct __sk_buff *ctx) +{ + return bpf_ktime_get_ns(); +} + +__noinline static int throwing_subprog(struct __sk_buff *ctx) +{ + if (ctx->tstamp) + bpf_throw(0); + return bpf_ktime_get_ns(); +} + +__noinline int global_subprog(struct __sk_buff *ctx) +{ + return bpf_ktime_get_ns(); +} + +__noinline int throwing_global_subprog(struct __sk_buff *ctx) +{ + if (ctx->tstamp) + bpf_throw(0); + return bpf_ktime_get_ns(); +} + +SEC("tc") +int exception_throw_subprog(struct __sk_buff *ctx) +{ + switch (ctx->protocol) { + case 1: + return subprog(ctx); + case 2: + return global_subprog(ctx); + case 3: + return throwing_subprog(ctx); + case 4: + return throwing_global_subprog(ctx); + default: + break; + } + bpf_throw(1); + return 0; +} + +__noinline int assert_nz_gfunc(u64 c) +{ + volatile u64 cookie = c; + + bpf_assert(cookie != 0); + return 0; +} + +__noinline int assert_zero_gfunc(u64 c) +{ + volatile u64 cookie = c; + + bpf_assert_eq(cookie, 0); + return 0; +} + +__noinline int assert_neg_gfunc(s64 c) +{ + volatile s64 cookie = c; + + bpf_assert_lt(cookie, 0); + return 0; +} + +__noinline int assert_pos_gfunc(s64 c) +{ + volatile s64 cookie = c; + + bpf_assert_gt(cookie, 0); + return 0; +} + +__noinline int assert_negeq_gfunc(s64 c) +{ + volatile s64 cookie = c; + + bpf_assert_le(cookie, -1); + return 0; +} + +__noinline int assert_poseq_gfunc(s64 c) +{ + volatile s64 cookie = c; + + bpf_assert_ge(cookie, 1); + return 0; +} + +__noinline int assert_nz_gfunc_with(u64 c) +{ + volatile u64 cookie = c; + + bpf_assert_with(cookie != 0, cookie + 100); + return 0; +} + +__noinline int assert_zero_gfunc_with(u64 c) +{ + volatile u64 cookie = c; + + bpf_assert_eq_with(cookie, 0, cookie + 100); + return 0; +} + +__noinline int assert_neg_gfunc_with(s64 c) +{ + volatile s64 cookie = c; + + bpf_assert_lt_with(cookie, 0, cookie + 100); + return 0; +} + +__noinline int assert_pos_gfunc_with(s64 c) +{ + volatile s64 cookie = c; + + bpf_assert_gt_with(cookie, 0, cookie + 100); + return 0; +} + +__noinline int assert_negeq_gfunc_with(s64 c) +{ + volatile s64 cookie = c; + + bpf_assert_le_with(cookie, -1, cookie + 100); + return 0; +} + +__noinline int assert_poseq_gfunc_with(s64 c) +{ + volatile s64 cookie = c; + + bpf_assert_ge_with(cookie, 1, cookie + 100); + return 0; +} + +#define check_assert(name, cookie, tag) \ +SEC("tc") \ +int exception##tag##name(struct __sk_buff *ctx) \ +{ \ + return name(cookie) + 1; \ +} + +check_assert(assert_nz_gfunc, 5, _); +check_assert(assert_zero_gfunc, 0, _); +check_assert(assert_neg_gfunc, -100, _); +check_assert(assert_pos_gfunc, 100, _); +check_assert(assert_negeq_gfunc, -1, _); +check_assert(assert_poseq_gfunc, 1, _); + +check_assert(assert_nz_gfunc_with, 5, _); +check_assert(assert_zero_gfunc_with, 0, _); +check_assert(assert_neg_gfunc_with, -100, _); +check_assert(assert_pos_gfunc_with, 100, _); +check_assert(assert_negeq_gfunc_with, -1, _); +check_assert(assert_poseq_gfunc_with, 1, _); + +check_assert(assert_nz_gfunc, 0, _bad_); +check_assert(assert_zero_gfunc, 5, _bad_); +check_assert(assert_neg_gfunc, 100, _bad_); +check_assert(assert_pos_gfunc, -100, _bad_); +check_assert(assert_negeq_gfunc, 1, _bad_); +check_assert(assert_poseq_gfunc, -1, _bad_); + +check_assert(assert_nz_gfunc_with, 0, _bad_); +check_assert(assert_zero_gfunc_with, 5, _bad_); +check_assert(assert_neg_gfunc_with, 100, _bad_); +check_assert(assert_pos_gfunc_with, -100, _bad_); +check_assert(assert_negeq_gfunc_with, 1, _bad_); +check_assert(assert_poseq_gfunc_with, -1, _bad_); + +SEC("tc") +int exception_assert_range(struct __sk_buff *ctx) +{ + u64 time = bpf_ktime_get_ns(); + + bpf_assert_range(time, 0, ~0ULL); + return 1; +} + +SEC("tc") +int exception_assert_range_with(struct __sk_buff *ctx) +{ + u64 time = bpf_ktime_get_ns(); + + bpf_assert_range_with(time, 0, ~0ULL, 10); + return 1; +} + +SEC("tc") +int exception_bad_assert_range(struct __sk_buff *ctx) +{ + u64 time = bpf_ktime_get_ns(); + + bpf_assert_range(time, -100, 100); + return 1; +} + +SEC("tc") +int exception_bad_assert_range_with(struct __sk_buff *ctx) +{ + u64 time = bpf_ktime_get_ns(); + + bpf_assert_range_with(time, -1000, 1000, 10); + return 1; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/exceptions_assert.c b/tools/testing/selftests/bpf/progs/exceptions_assert.c new file mode 100644 index 000000000000..fa35832e6748 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/exceptions_assert.c @@ -0,0 +1,135 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include "bpf_misc.h" +#include "bpf_experimental.h" + +#define check_assert(type, op, name, value) \ + SEC("?tc") \ + __log_level(2) __failure \ + int check_assert_##op##_##name(void *ctx) \ + { \ + type num = bpf_ktime_get_ns(); \ + bpf_assert_##op(num, value); \ + return *(u64 *)num; \ + } + +__msg(": R0_w=-2147483648 R10=fp0") +check_assert(s64, eq, int_min, INT_MIN); +__msg(": R0_w=2147483647 R10=fp0") +check_assert(s64, eq, int_max, INT_MAX); +__msg(": R0_w=0 R10=fp0") +check_assert(s64, eq, zero, 0); +__msg(": R0_w=-9223372036854775808 R1_w=-9223372036854775808 R10=fp0") +check_assert(s64, eq, llong_min, LLONG_MIN); +__msg(": R0_w=9223372036854775807 R1_w=9223372036854775807 R10=fp0") +check_assert(s64, eq, llong_max, LLONG_MAX); + +__msg(": R0_w=scalar(smax=2147483646) R10=fp0") +check_assert(s64, lt, pos, INT_MAX); +__msg(": R0_w=scalar(umin=9223372036854775808,var_off=(0x8000000000000000; 0x7fffffffffffffff))") +check_assert(s64, lt, zero, 0); +__msg(": R0_w=scalar(umin=9223372036854775808,umax=18446744071562067967,var_off=(0x8000000000000000; 0x7fffffffffffffff))") +check_assert(s64, lt, neg, INT_MIN); + +__msg(": R0_w=scalar(smax=2147483647) R10=fp0") +check_assert(s64, le, pos, INT_MAX); +__msg(": R0_w=scalar(smax=0) R10=fp0") +check_assert(s64, le, zero, 0); +__msg(": R0_w=scalar(umin=9223372036854775808,umax=18446744071562067968,var_off=(0x8000000000000000; 0x7fffffffffffffff))") +check_assert(s64, le, neg, INT_MIN); + +__msg(": R0_w=scalar(umin=2147483648,umax=9223372036854775807,var_off=(0x0; 0x7fffffffffffffff))") +check_assert(s64, gt, pos, INT_MAX); +__msg(": R0_w=scalar(umin=1,umax=9223372036854775807,var_off=(0x0; 0x7fffffffffffffff))") +check_assert(s64, gt, zero, 0); +__msg(": R0_w=scalar(smin=-2147483647) R10=fp0") +check_assert(s64, gt, neg, INT_MIN); + +__msg(": R0_w=scalar(umin=2147483647,umax=9223372036854775807,var_off=(0x0; 0x7fffffffffffffff))") +check_assert(s64, ge, pos, INT_MAX); +__msg(": R0_w=scalar(umax=9223372036854775807,var_off=(0x0; 0x7fffffffffffffff)) R10=fp0") +check_assert(s64, ge, zero, 0); +__msg(": R0_w=scalar(smin=-2147483648) R10=fp0") +check_assert(s64, ge, neg, INT_MIN); + +SEC("?tc") +__log_level(2) __failure +__msg(": R0=0 R1=ctx(off=0,imm=0) R2=scalar(smin=-2147483646,smax=2147483645) R10=fp0") +int check_assert_range_s64(struct __sk_buff *ctx) +{ + struct bpf_sock *sk = ctx->sk; + s64 num; + + _Static_assert(_Generic((sk->rx_queue_mapping), s32: 1, default: 0), "type match"); + if (!sk) + return 0; + num = sk->rx_queue_mapping; + bpf_assert_range(num, INT_MIN + 2, INT_MAX - 2); + return *((u8 *)ctx + num); +} + +SEC("?tc") +__log_level(2) __failure +__msg(": R1=ctx(off=0,imm=0) R2=scalar(umin=4096,umax=8192,var_off=(0x0; 0x3fff))") +int check_assert_range_u64(struct __sk_buff *ctx) +{ + u64 num = ctx->len; + + bpf_assert_range(num, 4096, 8192); + return *((u8 *)ctx + num); +} + +SEC("?tc") +__log_level(2) __failure +__msg(": R0=0 R1=ctx(off=0,imm=0) R2=4096 R10=fp0") +int check_assert_single_range_s64(struct __sk_buff *ctx) +{ + struct bpf_sock *sk = ctx->sk; + s64 num; + + _Static_assert(_Generic((sk->rx_queue_mapping), s32: 1, default: 0), "type match"); + if (!sk) + return 0; + num = sk->rx_queue_mapping; + + bpf_assert_range(num, 4096, 4096); + return *((u8 *)ctx + num); +} + +SEC("?tc") +__log_level(2) __failure +__msg(": R1=ctx(off=0,imm=0) R2=4096 R10=fp0") +int check_assert_single_range_u64(struct __sk_buff *ctx) +{ + u64 num = ctx->len; + + bpf_assert_range(num, 4096, 4096); + return *((u8 *)ctx + num); +} + +SEC("?tc") +__log_level(2) __failure +__msg(": R1=pkt(off=64,r=64,imm=0) R2=pkt_end(off=0,imm=0) R6=pkt(off=0,r=64,imm=0) R10=fp0") +int check_assert_generic(struct __sk_buff *ctx) +{ + u8 *data_end = (void *)(long)ctx->data_end; + u8 *data = (void *)(long)ctx->data; + + bpf_assert(data + 64 <= data_end); + return data[128]; +} + +SEC("?fentry/bpf_check") +__failure __msg("At program exit the register R0 has value (0x40; 0x0)") +int check_assert_with_return(void *ctx) +{ + bpf_assert_with(!ctx, 64); + return 0; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/exceptions_ext.c b/tools/testing/selftests/bpf/progs/exceptions_ext.c new file mode 100644 index 000000000000..743c05185d9b --- /dev/null +++ b/tools/testing/selftests/bpf/progs/exceptions_ext.c @@ -0,0 +1,72 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include "bpf_experimental.h" + +SEC("?fentry") +int pfentry(void *ctx) +{ + return 0; +} + +SEC("?fentry") +int throwing_fentry(void *ctx) +{ + bpf_throw(0); + return 0; +} + +__noinline int exception_cb(u64 cookie) +{ + return cookie + 64; +} + +SEC("?freplace") +int extension(struct __sk_buff *ctx) +{ + return 0; +} + +SEC("?freplace") +__exception_cb(exception_cb) +int throwing_exception_cb_extension(u64 cookie) +{ + bpf_throw(32); + return 0; +} + +SEC("?freplace") +__exception_cb(exception_cb) +int throwing_extension(struct __sk_buff *ctx) +{ + bpf_throw(64); + return 0; +} + +SEC("?fexit") +int pfexit(void *ctx) +{ + return 0; +} + +SEC("?fexit") +int throwing_fexit(void *ctx) +{ + bpf_throw(0); + return 0; +} + +SEC("?fmod_ret") +int pfmod_ret(void *ctx) +{ + return 0; +} + +SEC("?fmod_ret") +int throwing_fmod_ret(void *ctx) +{ + bpf_throw(0); + return 0; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c new file mode 100644 index 000000000000..4c39e920dac2 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c @@ -0,0 +1,347 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include + +#include "bpf_misc.h" +#include "bpf_experimental.h" + +extern void bpf_rcu_read_lock(void) __ksym; + +#define private(name) SEC(".bss." #name) __hidden __attribute__((aligned(8))) + +struct foo { + struct bpf_rb_node node; +}; + +struct hmap_elem { + struct bpf_timer timer; +}; + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(max_entries, 64); + __type(key, int); + __type(value, struct hmap_elem); +} hmap SEC(".maps"); + +private(A) struct bpf_spin_lock lock; +private(A) struct bpf_rb_root rbtree __contains(foo, node); + +__noinline void *exception_cb_bad_ret_type(u64 cookie) +{ + return NULL; +} + +__noinline int exception_cb_bad_arg_0(void) +{ + return 0; +} + +__noinline int exception_cb_bad_arg_2(int a, int b) +{ + return 0; +} + +__noinline int exception_cb_ok_arg_small(int a) +{ + return 0; +} + +SEC("?tc") +__exception_cb(exception_cb_bad_ret_type) +__failure __msg("Global function exception_cb_bad_ret_type() doesn't return scalar.") +int reject_exception_cb_type_1(struct __sk_buff *ctx) +{ + bpf_throw(0); + return 0; +} + +SEC("?tc") +__exception_cb(exception_cb_bad_arg_0) +__failure __msg("exception cb only supports single integer argument") +int reject_exception_cb_type_2(struct __sk_buff *ctx) +{ + bpf_throw(0); + return 0; +} + +SEC("?tc") +__exception_cb(exception_cb_bad_arg_2) +__failure __msg("exception cb only supports single integer argument") +int reject_exception_cb_type_3(struct __sk_buff *ctx) +{ + bpf_throw(0); + return 0; +} + +SEC("?tc") +__exception_cb(exception_cb_ok_arg_small) +__success +int reject_exception_cb_type_4(struct __sk_buff *ctx) +{ + bpf_throw(0); + return 0; +} + +__noinline +static int timer_cb(void *map, int *key, struct bpf_timer *timer) +{ + bpf_throw(0); + return 0; +} + +SEC("?tc") +__failure __msg("cannot be called from callback subprog") +int reject_async_callback_throw(struct __sk_buff *ctx) +{ + struct hmap_elem *elem; + + elem = bpf_map_lookup_elem(&hmap, &(int){0}); + if (!elem) + return 0; + return bpf_timer_set_callback(&elem->timer, timer_cb); +} + +__noinline static int subprog_lock(struct __sk_buff *ctx) +{ + volatile int ret = 0; + + bpf_spin_lock(&lock); + if (ctx->len) + bpf_throw(0); + return ret; +} + +SEC("?tc") +__failure __msg("function calls are not allowed while holding a lock") +int reject_with_lock(void *ctx) +{ + bpf_spin_lock(&lock); + bpf_throw(0); + return 0; +} + +SEC("?tc") +__failure __msg("function calls are not allowed while holding a lock") +int reject_subprog_with_lock(void *ctx) +{ + return subprog_lock(ctx); +} + +SEC("?tc") +__failure __msg("bpf_rcu_read_unlock is missing") +int reject_with_rcu_read_lock(void *ctx) +{ + bpf_rcu_read_lock(); + bpf_throw(0); + return 0; +} + +__noinline static int throwing_subprog(struct __sk_buff *ctx) +{ + if (ctx->len) + bpf_throw(0); + return 0; +} + +SEC("?tc") +__failure __msg("bpf_rcu_read_unlock is missing") +int reject_subprog_with_rcu_read_lock(void *ctx) +{ + bpf_rcu_read_lock(); + return throwing_subprog(ctx); +} + +static bool rbless(struct bpf_rb_node *n1, const struct bpf_rb_node *n2) +{ + bpf_throw(0); + return true; +} + +SEC("?tc") +__failure __msg("function calls are not allowed while holding a lock") +int reject_with_rbtree_add_throw(void *ctx) +{ + struct foo *f; + + f = bpf_obj_new(typeof(*f)); + if (!f) + return 0; + bpf_spin_lock(&lock); + bpf_rbtree_add(&rbtree, &f->node, rbless); + return 0; +} + +SEC("?tc") +__failure __msg("Unreleased reference") +int reject_with_reference(void *ctx) +{ + struct foo *f; + + f = bpf_obj_new(typeof(*f)); + if (!f) + return 0; + bpf_throw(0); + return 0; +} + +__noinline static int subprog_ref(struct __sk_buff *ctx) +{ + struct foo *f; + + f = bpf_obj_new(typeof(*f)); + if (!f) + return 0; + bpf_throw(0); + return 0; +} + +__noinline static int subprog_cb_ref(u32 i, void *ctx) +{ + bpf_throw(0); + return 0; +} + +SEC("?tc") +__failure __msg("Unreleased reference") +int reject_with_cb_reference(void *ctx) +{ + struct foo *f; + + f = bpf_obj_new(typeof(*f)); + if (!f) + return 0; + bpf_loop(5, subprog_cb_ref, NULL, 0); + return 0; +} + +SEC("?tc") +__failure __msg("cannot be called from callback") +int reject_with_cb(void *ctx) +{ + bpf_loop(5, subprog_cb_ref, NULL, 0); + return 0; +} + +SEC("?tc") +__failure __msg("Unreleased reference") +int reject_with_subprog_reference(void *ctx) +{ + return subprog_ref(ctx) + 1; +} + +__noinline int throwing_exception_cb(u64 c) +{ + bpf_throw(0); + return c; +} + +__noinline int exception_cb1(u64 c) +{ + return c; +} + +__noinline int exception_cb2(u64 c) +{ + return c; +} + +static __noinline int static_func(struct __sk_buff *ctx) +{ + return exception_cb1(ctx->tstamp); +} + +__noinline int global_func(struct __sk_buff *ctx) +{ + return exception_cb1(ctx->tstamp); +} + +SEC("?tc") +__exception_cb(throwing_exception_cb) +__failure __msg("cannot be called from callback subprog") +int reject_throwing_exception_cb(struct __sk_buff *ctx) +{ + return 0; +} + +SEC("?tc") +__exception_cb(exception_cb1) +__failure __msg("cannot call exception cb directly") +int reject_exception_cb_call_global_func(struct __sk_buff *ctx) +{ + return global_func(ctx); +} + +SEC("?tc") +__exception_cb(exception_cb1) +__failure __msg("cannot call exception cb directly") +int reject_exception_cb_call_static_func(struct __sk_buff *ctx) +{ + return static_func(ctx); +} + +SEC("?tc") +__exception_cb(exception_cb1) +__exception_cb(exception_cb2) +__failure __msg("multiple exception callback tags for main subprog") +int reject_multiple_exception_cb(struct __sk_buff *ctx) +{ + bpf_throw(0); + return 16; +} + +__noinline int exception_cb_bad_ret(u64 c) +{ + return c; +} + +SEC("?fentry/bpf_check") +__exception_cb(exception_cb_bad_ret) +__failure __msg("At program exit the register R0 has unknown scalar value should") +int reject_set_exception_cb_bad_ret1(void *ctx) +{ + return 0; +} + +SEC("?fentry/bpf_check") +__failure __msg("At program exit the register R0 has value (0x40; 0x0) should") +int reject_set_exception_cb_bad_ret2(void *ctx) +{ + bpf_throw(64); + return 0; +} + +__noinline static int loop_cb1(u32 index, int *ctx) +{ + bpf_throw(0); + return 0; +} + +__noinline static int loop_cb2(u32 index, int *ctx) +{ + bpf_throw(0); + return 0; +} + +SEC("?tc") +__failure __msg("cannot be called from callback") +int reject_exception_throw_cb(struct __sk_buff *ctx) +{ + bpf_loop(5, loop_cb1, NULL, 0); + return 0; +} + +SEC("?tc") +__failure __msg("cannot be called from callback") +int reject_exception_throw_cb_diff(struct __sk_buff *ctx) +{ + if (ctx->protocol) + bpf_loop(5, loop_cb1, NULL, 0); + else + bpf_loop(5, loop_cb2, NULL, 0); + return 0; +} + +char _license[] SEC("license") = "GPL";