From patchwork Tue Apr 16 14:08:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Tissoires X-Patchwork-Id: 13631921 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A9FAF12E1F5; Tue, 16 Apr 2024 14:10:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276602; cv=none; b=uo/EtkEphWfPXXioKFs30ARadwEDMyd1Tpjth/rOKKVQJt1bope+0PYwfJyF5HHfPNYtUtRG5SIwf6jt+0NDmlLbgQ0JZJCDaymQ+IdpWV2Ig5nHboRNBlgkqiDpAoYgGJEVkKSooW3vjzFTyjJXicNKWmlGBCrhxnQgrfFWVoI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276602; c=relaxed/simple; bh=EqDSHYXSuauocbkK3UeYk6FohwrTmrLQYrjZAzvk/UM=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=fTYPhXEWu6k1IPGktpLY/9vpTr2Fz0og3qP1GCw7F/mRMexZXuj/XLsxHytAkb/DAJSzxD/c7dHhONKlRnvnXK3kaWYfGwW60x7M6V9C8/nQshuIJg0drs77pWDS3+/mm5g9rMvmoCfFxuRoLEQfSlz0G9M8K1eprd0dMu7mOfs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=cJLw0Xh4; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="cJLw0Xh4" Received: by smtp.kernel.org (Postfix) with ESMTPSA id F1592C2BD11; Tue, 16 Apr 2024 14:09:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1713276602; bh=EqDSHYXSuauocbkK3UeYk6FohwrTmrLQYrjZAzvk/UM=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=cJLw0Xh4d+TehtvzP4HNnxbb/4CdD37XZe8ozJR/0qGGqMte9uve80gAMVYlTPRzX iGLWgPuzsRxXS44mPfSrEWPq0HJsJ3BW0cnqRATVdTHpVbBmvRohgv4YvANHfkO0mB IRgRQVDJE+D+nyebt7rFOkpIADqlcPuEUfFYsSb8ALpBftSstZBl6nkTxTho8knn3d nixz7LjsJPEhLnuUP7yW0OKlv5nLMF/iYFXwrSiIICJGHL7qeEQveM0CGG66yx+1t8 3SIofX/0SNovldZwMgFK+tRIOCwOsiqyNGf2lesWU//HQSovhP0vx3whewFng772dN 0VjKBDDTBTUcQ== From: Benjamin Tissoires Date: Tue, 16 Apr 2024 16:08:14 +0200 Subject: [PATCH bpf-next 01/18] bpf: trampoline: export __bpf_prog_enter/exit_recur Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240416-bpf_wq-v1-1-c9e66092f842@kernel.org> References: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> In-Reply-To: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Mykola Lysenko , Shuah Khan Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Benjamin Tissoires X-Mailer: b4 0.12.4 X-Developer-Signature: v=1; a=ed25519-sha256; t=1713276593; l=2335; i=bentiss@kernel.org; s=20230215; h=from:subject:message-id; bh=EqDSHYXSuauocbkK3UeYk6FohwrTmrLQYrjZAzvk/UM=; b=JSkEUA6eeIT82u3qdJEkQBERbwjvE93oqsMRp+A190eCAgA9aO5pHSa6wNnvw+jnDmSX/l8/h 8Wvp+UtRcltAheyvx43kZfEqkjY127VHrKLRgxj5BvogIF7mSo9sL0Z X-Developer-Key: i=bentiss@kernel.org; a=ed25519; pk=7D1DyAVh6ajCkuUTudt/chMuXWIJHlv2qCsRkIizvFw= X-Patchwork-Delegate: bpf@iogearbox.net When dealing with workqueues, we need to also use __bpf_prog_enter/exit_recur(). bpf_trampoline_enter/exit() is not suitable because we are dealing with async callbacks that are set sleepable or not by the verifier, depending on how they are configured, so we need to rewrite our own test "if (sleepable)..." Signed-off-by: Benjamin Tissoires --- include/linux/bpf.h | 4 ++++ kernel/bpf/trampoline.c | 6 +++--- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 5034c1b4ded7..551445c47779 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1124,6 +1124,10 @@ u64 notrace __bpf_prog_enter_sleepable_recur(struct bpf_prog *prog, struct bpf_tramp_run_ctx *run_ctx); void notrace __bpf_prog_exit_sleepable_recur(struct bpf_prog *prog, u64 start, struct bpf_tramp_run_ctx *run_ctx); +u64 notrace __bpf_prog_enter_recur(struct bpf_prog *prog, + struct bpf_tramp_run_ctx *run_ctx); +void notrace __bpf_prog_exit_recur(struct bpf_prog *prog, u64 start, + struct bpf_tramp_run_ctx *run_ctx); void notrace __bpf_tramp_enter(struct bpf_tramp_image *tr); void notrace __bpf_tramp_exit(struct bpf_tramp_image *tr); typedef u64 (*bpf_trampoline_enter_t)(struct bpf_prog *prog, diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 26ae703d3c3b..c6a3e0280993 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -858,7 +858,7 @@ static __always_inline u64 notrace bpf_prog_start_time(void) * [2..MAX_U64] - execute bpf prog and record execution time. * This is start time. */ -static u64 notrace __bpf_prog_enter_recur(struct bpf_prog *prog, struct bpf_tramp_run_ctx *run_ctx) +u64 notrace __bpf_prog_enter_recur(struct bpf_prog *prog, struct bpf_tramp_run_ctx *run_ctx) __acquires(RCU) { rcu_read_lock(); @@ -896,8 +896,8 @@ static void notrace update_prog_stats(struct bpf_prog *prog, } } -static void notrace __bpf_prog_exit_recur(struct bpf_prog *prog, u64 start, - struct bpf_tramp_run_ctx *run_ctx) +void notrace __bpf_prog_exit_recur(struct bpf_prog *prog, u64 start, + struct bpf_tramp_run_ctx *run_ctx) __releases(RCU) { bpf_reset_run_ctx(run_ctx->saved_run_ctx); From patchwork Tue Apr 16 14:08:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Tissoires X-Patchwork-Id: 13631922 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8CA0C12D219; Tue, 16 Apr 2024 14:10:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276606; cv=none; b=TgPHr2L+0GxXfeQMQrhNWG0l8rV5ci8v4GbCiLUKtiz/Dp34h4DtjoltWKEBxzdBOHZe3X76EzmU1j5f7BedvlCDR2CueVOd2ilmiyEPJndfFPdC8QYrOUhafU3ISJBfQlQiQT1vu3XmOAWKYgXd248pcxhatpHKVT1m29AtrgY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276606; c=relaxed/simple; bh=uFTntohS4VBdYzfV/zA08ZLEeTEQTFJy/4/AJB/v++0=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=ogpRaxVstIE9zu1ye/5Qph98TN6Tkj3rpWzZwYqXegb64eWYPWTRcwdcZfrAT/PF8OIyKZa7qRBACjheLbrsUB1cxivcfjQapOncpD2VAErnkK499vXAI1v96jqjMgQx4JwVOL0+JOwvJcbMtF6BWYp9t0pjujGkQlh8oSrMicc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=WChLyz8f; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="WChLyz8f" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C3007C32783; Tue, 16 Apr 2024 14:10:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1713276606; bh=uFTntohS4VBdYzfV/zA08ZLEeTEQTFJy/4/AJB/v++0=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=WChLyz8fwMOmSBvTP45O0FdRwX97IcF5ktjaZRhpp9f4kxJKBHFC630ADfnJJRDgK vvuzGQD68Y9TN0ax6c/yOHJ6D9u/u6Z43SNMjKhftdhTAQwcwoCw0+zINo6SoTKHnP M1TFPBbg84YYeaRrDdcabWUOInC1E7xELr9f+Q07AqNSJxJw8oQM2D0MjmusUp8WU/ 2sf0cQ2BJJU4lOxgHUeKqVneEQS/lB4OuMwstlBaA+WUCtHLdDhoUlPAbCsFxpCs8E SL3NRdiWrWwXzjNLlJkhdazOVtS8AKPsrxBh0Wh9ELiEs1SWFBf+MnCe1LVo7iuNWg gj6jEREFoKxsQ== From: Benjamin Tissoires Date: Tue, 16 Apr 2024 16:08:15 +0200 Subject: [PATCH bpf-next 02/18] bpf: make timer data struct more generic Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240416-bpf_wq-v1-2-c9e66092f842@kernel.org> References: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> In-Reply-To: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Mykola Lysenko , Shuah Khan Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Benjamin Tissoires X-Mailer: b4 0.12.4 X-Developer-Signature: v=1; a=ed25519-sha256; t=1713276593; l=8333; i=bentiss@kernel.org; s=20230215; h=from:subject:message-id; bh=uFTntohS4VBdYzfV/zA08ZLEeTEQTFJy/4/AJB/v++0=; b=9eJSinD5HHZ6VA/RA952BSNpSY+rMA1+aWKjaCgnMWtX/JOaj+hoICrkdzx+Oa0k1+ZA1EH/B MndSYbIacZ6BMK3Ul/CkSqiITXN/iEGpfTYtD1KfwqIMQaPIJh4pSfM X-Developer-Key: i=bentiss@kernel.org; a=ed25519; pk=7D1DyAVh6ajCkuUTudt/chMuXWIJHlv2qCsRkIizvFw= X-Patchwork-Delegate: bpf@iogearbox.net To be able to add workqueues and reuse most of the timer code, we need to make bpf_hrtimer more generic. There is no code change except that the new struct gets a new u64 flags attribute. We are still below 2 cache lines, so this shouldn't impact the current running codes. The ordering is also changed. Everything related to async callback is now on top of bpf_hrtimer. Signed-off-by: Benjamin Tissoires --- kernel/bpf/helpers.c | 71 ++++++++++++++++++++++++++++------------------------ 1 file changed, 38 insertions(+), 33 deletions(-) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 8cde717137bd..5a069c70b5e6 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -1079,11 +1079,20 @@ const struct bpf_func_proto bpf_snprintf_proto = { .arg5_type = ARG_CONST_SIZE_OR_ZERO, }; +struct bpf_async_cb { + struct bpf_map *map; + struct bpf_prog *prog; + void __rcu *callback_fn; + void *value; + struct rcu_head rcu; + u64 flags; +}; + /* BPF map elements can contain 'struct bpf_timer'. * Such map owns all of its BPF timers. * 'struct bpf_timer' is allocated as part of map element allocation * and it's zero initialized. - * That space is used to keep 'struct bpf_timer_kern'. + * That space is used to keep 'struct bpf_async_kern'. * bpf_timer_init() allocates 'struct bpf_hrtimer', inits hrtimer, and * remembers 'struct bpf_map *' pointer it's part of. * bpf_timer_set_callback() increments prog refcnt and assign bpf callback_fn. @@ -1096,16 +1105,12 @@ const struct bpf_func_proto bpf_snprintf_proto = { * freeing the timers when inner map is replaced or deleted by user space. */ struct bpf_hrtimer { + struct bpf_async_cb cb; struct hrtimer timer; - struct bpf_map *map; - struct bpf_prog *prog; - void __rcu *callback_fn; - void *value; - struct rcu_head rcu; }; /* the actual struct hidden inside uapi struct bpf_timer */ -struct bpf_timer_kern { +struct bpf_async_kern { struct bpf_hrtimer *timer; /* bpf_spin_lock is used here instead of spinlock_t to make * sure that it always fits into space reserved by struct bpf_timer @@ -1119,14 +1124,14 @@ static DEFINE_PER_CPU(struct bpf_hrtimer *, hrtimer_running); static enum hrtimer_restart bpf_timer_cb(struct hrtimer *hrtimer) { struct bpf_hrtimer *t = container_of(hrtimer, struct bpf_hrtimer, timer); - struct bpf_map *map = t->map; - void *value = t->value; + struct bpf_map *map = t->cb.map; + void *value = t->cb.value; bpf_callback_t callback_fn; void *key; u32 idx; BTF_TYPE_EMIT(struct bpf_timer); - callback_fn = rcu_dereference_check(t->callback_fn, rcu_read_lock_bh_held()); + callback_fn = rcu_dereference_check(t->cb.callback_fn, rcu_read_lock_bh_held()); if (!callback_fn) goto out; @@ -1155,7 +1160,7 @@ static enum hrtimer_restart bpf_timer_cb(struct hrtimer *hrtimer) return HRTIMER_NORESTART; } -BPF_CALL_3(bpf_timer_init, struct bpf_timer_kern *, timer, struct bpf_map *, map, +BPF_CALL_3(bpf_timer_init, struct bpf_async_kern *, timer, struct bpf_map *, map, u64, flags) { clockid_t clockid = flags & (MAX_CLOCKS - 1); @@ -1163,8 +1168,8 @@ BPF_CALL_3(bpf_timer_init, struct bpf_timer_kern *, timer, struct bpf_map *, map int ret = 0; BUILD_BUG_ON(MAX_CLOCKS != 16); - BUILD_BUG_ON(sizeof(struct bpf_timer_kern) > sizeof(struct bpf_timer)); - BUILD_BUG_ON(__alignof__(struct bpf_timer_kern) != __alignof__(struct bpf_timer)); + BUILD_BUG_ON(sizeof(struct bpf_async_kern) > sizeof(struct bpf_timer)); + BUILD_BUG_ON(__alignof__(struct bpf_async_kern) != __alignof__(struct bpf_timer)); if (in_nmi()) return -EOPNOTSUPP; @@ -1187,10 +1192,10 @@ BPF_CALL_3(bpf_timer_init, struct bpf_timer_kern *, timer, struct bpf_map *, map ret = -ENOMEM; goto out; } - t->value = (void *)timer - map->record->timer_off; - t->map = map; - t->prog = NULL; - rcu_assign_pointer(t->callback_fn, NULL); + t->cb.value = (void *)timer - map->record->timer_off; + t->cb.map = map; + t->cb.prog = NULL; + rcu_assign_pointer(t->cb.callback_fn, NULL); hrtimer_init(&t->timer, clockid, HRTIMER_MODE_REL_SOFT); t->timer.function = bpf_timer_cb; WRITE_ONCE(timer->timer, t); @@ -1222,7 +1227,7 @@ static const struct bpf_func_proto bpf_timer_init_proto = { .arg3_type = ARG_ANYTHING, }; -BPF_CALL_3(bpf_timer_set_callback, struct bpf_timer_kern *, timer, void *, callback_fn, +BPF_CALL_3(bpf_timer_set_callback, struct bpf_async_kern *, timer, void *, callback_fn, struct bpf_prog_aux *, aux) { struct bpf_prog *prev, *prog = aux->prog; @@ -1237,7 +1242,7 @@ BPF_CALL_3(bpf_timer_set_callback, struct bpf_timer_kern *, timer, void *, callb ret = -EINVAL; goto out; } - if (!atomic64_read(&t->map->usercnt)) { + if (!atomic64_read(&t->cb.map->usercnt)) { /* maps with timers must be either held by user space * or pinned in bpffs. Otherwise timer might still be * running even when bpf prog is detached and user space @@ -1246,7 +1251,7 @@ BPF_CALL_3(bpf_timer_set_callback, struct bpf_timer_kern *, timer, void *, callb ret = -EPERM; goto out; } - prev = t->prog; + prev = t->cb.prog; if (prev != prog) { /* Bump prog refcnt once. Every bpf_timer_set_callback() * can pick different callback_fn-s within the same prog. @@ -1259,9 +1264,9 @@ BPF_CALL_3(bpf_timer_set_callback, struct bpf_timer_kern *, timer, void *, callb if (prev) /* Drop prev prog refcnt when swapping with new prog */ bpf_prog_put(prev); - t->prog = prog; + t->cb.prog = prog; } - rcu_assign_pointer(t->callback_fn, callback_fn); + rcu_assign_pointer(t->cb.callback_fn, callback_fn); out: __bpf_spin_unlock_irqrestore(&timer->lock); return ret; @@ -1275,7 +1280,7 @@ static const struct bpf_func_proto bpf_timer_set_callback_proto = { .arg2_type = ARG_PTR_TO_FUNC, }; -BPF_CALL_3(bpf_timer_start, struct bpf_timer_kern *, timer, u64, nsecs, u64, flags) +BPF_CALL_3(bpf_timer_start, struct bpf_async_kern *, timer, u64, nsecs, u64, flags) { struct bpf_hrtimer *t; int ret = 0; @@ -1287,7 +1292,7 @@ BPF_CALL_3(bpf_timer_start, struct bpf_timer_kern *, timer, u64, nsecs, u64, fla return -EINVAL; __bpf_spin_lock_irqsave(&timer->lock); t = timer->timer; - if (!t || !t->prog) { + if (!t || !t->cb.prog) { ret = -EINVAL; goto out; } @@ -1315,18 +1320,18 @@ static const struct bpf_func_proto bpf_timer_start_proto = { .arg3_type = ARG_ANYTHING, }; -static void drop_prog_refcnt(struct bpf_hrtimer *t) +static void drop_prog_refcnt(struct bpf_async_cb *async) { - struct bpf_prog *prog = t->prog; + struct bpf_prog *prog = async->prog; if (prog) { bpf_prog_put(prog); - t->prog = NULL; - rcu_assign_pointer(t->callback_fn, NULL); + async->prog = NULL; + rcu_assign_pointer(async->callback_fn, NULL); } } -BPF_CALL_1(bpf_timer_cancel, struct bpf_timer_kern *, timer) +BPF_CALL_1(bpf_timer_cancel, struct bpf_async_kern *, timer) { struct bpf_hrtimer *t; int ret = 0; @@ -1348,7 +1353,7 @@ BPF_CALL_1(bpf_timer_cancel, struct bpf_timer_kern *, timer) ret = -EDEADLK; goto out; } - drop_prog_refcnt(t); + drop_prog_refcnt(&t->cb); out: __bpf_spin_unlock_irqrestore(&timer->lock); /* Cancel the timer and wait for associated callback to finish @@ -1371,7 +1376,7 @@ static const struct bpf_func_proto bpf_timer_cancel_proto = { */ void bpf_timer_cancel_and_free(void *val) { - struct bpf_timer_kern *timer = val; + struct bpf_async_kern *timer = val; struct bpf_hrtimer *t; /* Performance optimization: read timer->timer without lock first. */ @@ -1383,7 +1388,7 @@ void bpf_timer_cancel_and_free(void *val) t = timer->timer; if (!t) goto out; - drop_prog_refcnt(t); + drop_prog_refcnt(&t->cb); /* The subsequent bpf_timer_start/cancel() helpers won't be able to use * this timer, since it won't be initialized. */ @@ -1410,7 +1415,7 @@ void bpf_timer_cancel_and_free(void *val) */ if (this_cpu_read(hrtimer_running) != t) hrtimer_cancel(&t->timer); - kfree_rcu(t, rcu); + kfree_rcu(t, cb.rcu); } BPF_CALL_2(bpf_kptr_xchg, void *, map_value, void *, ptr) From patchwork Tue Apr 16 14:08:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Tissoires X-Patchwork-Id: 13631923 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 704EE12D742; Tue, 16 Apr 2024 14:10:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276610; cv=none; b=aa6CdX9WnL29uglxDN7ZYsZstUlz9K4f3ZLHgNi3IboEKKiALmr9kMDhR3K8iqVCjiXPq+Mr62aD/Nfeb1PIXyU2ic02yjkrbV0lYpVZQw/JWJM/gGWUGq+MeabFh8gS+7Gvudc66eA5XrTBy2woI13OAE0MqNxgQ3U+qbLFkjo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276610; c=relaxed/simple; bh=E1Km4eXPM4X87/2UXWzVHAny461nVWmd/jXuniVLFes=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=ozMLl2oWkhF+rz7OdXAyNeovVy95pHuSDyuqt0CNfgB/M/C87Qaj7/+cvSKrARvkoAvwtzyJrAomuDl3exIZrg8lG5DTLMq/4iVKnfqFuCCjPAQ00KLetUE8Mh9Oe6nipJ6NO8/KnLbNOCckVitrEz6kl5j1BMIWjZSRDdSIizw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=g9PPKbCg; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="g9PPKbCg" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 93B44C113CE; Tue, 16 Apr 2024 14:10:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1713276610; bh=E1Km4eXPM4X87/2UXWzVHAny461nVWmd/jXuniVLFes=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=g9PPKbCgaT/J1Qx0IpnIc+FrubtqBsApFjE0PJR0472RdG80lMDriWYJ8VycaS7ET HR99CykjlsdBTs3u4mVI87cZni/8uKix99qLMrlTrFbQoOPph0Qc3FeknEfTMg2fr6 BIaiEEupgSNgPDY16MZoHMh4QM8KFX8LbYT9OQjwqprYNC4YFQkfsTSIkG9BpwgBpL vtoVypPDgr4jtuIRShPpqXt2XbbudDiZhFRvcpyBaImgnhl5dusCoxLExU5etF6Yz+ NjEpggIwoGpT08oa2U134cNS3pKyhiZ7qvpzZ50uNFAKpQrKh0pGoqqc75eQ2LkBv1 OqRIDepox+RAA== From: Benjamin Tissoires Date: Tue, 16 Apr 2024 16:08:16 +0200 Subject: [PATCH bpf-next 03/18] bpf: replace bpf_timer_init with a generic helper Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240416-bpf_wq-v1-3-c9e66092f842@kernel.org> References: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> In-Reply-To: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Mykola Lysenko , Shuah Khan Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Benjamin Tissoires X-Mailer: b4 0.12.4 X-Developer-Signature: v=1; a=ed25519-sha256; t=1713276593; l=5022; i=bentiss@kernel.org; s=20230215; h=from:subject:message-id; bh=E1Km4eXPM4X87/2UXWzVHAny461nVWmd/jXuniVLFes=; b=UHSpHZY5RN69pGZGhj1TdxTDTXpxz3m4Er7COLWN3notsINUIJyZmxH9Xy65AFhafRqXis0z5 L9AqzHNuQm6AQkfpsVXJYcsw7mws69MoYbINEwnieC+sC6c/Lc02ukb X-Developer-Key: i=bentiss@kernel.org; a=ed25519; pk=7D1DyAVh6ajCkuUTudt/chMuXWIJHlv2qCsRkIizvFw= X-Patchwork-Delegate: bpf@iogearbox.net No code change except for the new flags argument being stored in the local data struct. Signed-off-by: Benjamin Tissoires --- kernel/bpf/helpers.c | 91 ++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 63 insertions(+), 28 deletions(-) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 5a069c70b5e6..1e6d1011303b 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -1111,7 +1111,10 @@ struct bpf_hrtimer { /* the actual struct hidden inside uapi struct bpf_timer */ struct bpf_async_kern { - struct bpf_hrtimer *timer; + union { + struct bpf_async_cb *cb; + struct bpf_hrtimer *timer; + }; /* bpf_spin_lock is used here instead of spinlock_t to make * sure that it always fits into space reserved by struct bpf_timer * regardless of LOCKDEP and spinlock debug flags. @@ -1119,6 +1122,10 @@ struct bpf_async_kern { struct bpf_spin_lock lock; } __attribute__((aligned(8))); +enum bpf_async_type { + BPF_ASYNC_TYPE_TIMER = 0, +}; + static DEFINE_PER_CPU(struct bpf_hrtimer *, hrtimer_running); static enum hrtimer_restart bpf_timer_cb(struct hrtimer *hrtimer) @@ -1160,46 +1167,55 @@ static enum hrtimer_restart bpf_timer_cb(struct hrtimer *hrtimer) return HRTIMER_NORESTART; } -BPF_CALL_3(bpf_timer_init, struct bpf_async_kern *, timer, struct bpf_map *, map, - u64, flags) +static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u64 flags, + enum bpf_async_type type) { - clockid_t clockid = flags & (MAX_CLOCKS - 1); + struct bpf_async_cb *cb; struct bpf_hrtimer *t; + clockid_t clockid; + size_t size; int ret = 0; - BUILD_BUG_ON(MAX_CLOCKS != 16); - BUILD_BUG_ON(sizeof(struct bpf_async_kern) > sizeof(struct bpf_timer)); - BUILD_BUG_ON(__alignof__(struct bpf_async_kern) != __alignof__(struct bpf_timer)); - if (in_nmi()) return -EOPNOTSUPP; - if (flags >= MAX_CLOCKS || - /* similar to timerfd except _ALARM variants are not supported */ - (clockid != CLOCK_MONOTONIC && - clockid != CLOCK_REALTIME && - clockid != CLOCK_BOOTTIME)) + switch (type) { + case BPF_ASYNC_TYPE_TIMER: + size = sizeof(struct bpf_hrtimer); + break; + default: return -EINVAL; - __bpf_spin_lock_irqsave(&timer->lock); - t = timer->timer; + } + + __bpf_spin_lock_irqsave(&async->lock); + t = async->timer; if (t) { ret = -EBUSY; goto out; } + /* allocate hrtimer via map_kmalloc to use memcg accounting */ - t = bpf_map_kmalloc_node(map, sizeof(*t), GFP_ATOMIC, map->numa_node); - if (!t) { + cb = bpf_map_kmalloc_node(map, size, GFP_ATOMIC, map->numa_node); + if (!cb) { ret = -ENOMEM; goto out; } - t->cb.value = (void *)timer - map->record->timer_off; - t->cb.map = map; - t->cb.prog = NULL; - rcu_assign_pointer(t->cb.callback_fn, NULL); - hrtimer_init(&t->timer, clockid, HRTIMER_MODE_REL_SOFT); - t->timer.function = bpf_timer_cb; - WRITE_ONCE(timer->timer, t); - /* Guarantee the order between timer->timer and map->usercnt. So + + if (type == BPF_ASYNC_TYPE_TIMER) { + clockid = flags & (MAX_CLOCKS - 1); + t = (struct bpf_hrtimer *)cb; + + hrtimer_init(&t->timer, clockid, HRTIMER_MODE_REL_SOFT); + t->timer.function = bpf_timer_cb; + cb->value = (void *)async - map->record->timer_off; + } + cb->map = map; + cb->prog = NULL; + cb->flags = flags; + rcu_assign_pointer(cb->callback_fn, NULL); + + WRITE_ONCE(async->cb, cb); + /* Guarantee the order between async->cb and map->usercnt. So * when there are concurrent uref release and bpf timer init, either * bpf_timer_cancel_and_free() called by uref release reads a no-NULL * timer or atomic64_read() below returns a zero usercnt. @@ -1209,15 +1225,34 @@ BPF_CALL_3(bpf_timer_init, struct bpf_async_kern *, timer, struct bpf_map *, map /* maps with timers must be either held by user space * or pinned in bpffs. */ - WRITE_ONCE(timer->timer, NULL); - kfree(t); + WRITE_ONCE(async->cb, NULL); + kfree(cb); ret = -EPERM; } out: - __bpf_spin_unlock_irqrestore(&timer->lock); + __bpf_spin_unlock_irqrestore(&async->lock); return ret; } +BPF_CALL_3(bpf_timer_init, struct bpf_async_kern *, timer, struct bpf_map *, map, + u64, flags) +{ + clock_t clockid = flags & (MAX_CLOCKS - 1); + + BUILD_BUG_ON(MAX_CLOCKS != 16); + BUILD_BUG_ON(sizeof(struct bpf_async_kern) > sizeof(struct bpf_timer)); + BUILD_BUG_ON(__alignof__(struct bpf_async_kern) != __alignof__(struct bpf_timer)); + + if (flags >= MAX_CLOCKS || + /* similar to timerfd except _ALARM variants are not supported */ + (clockid != CLOCK_MONOTONIC && + clockid != CLOCK_REALTIME && + clockid != CLOCK_BOOTTIME)) + return -EINVAL; + + return __bpf_async_init(timer, map, flags, BPF_ASYNC_TYPE_TIMER); +} + static const struct bpf_func_proto bpf_timer_init_proto = { .func = bpf_timer_init, .gpl_only = true, From patchwork Tue Apr 16 14:08:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Tissoires X-Patchwork-Id: 13631924 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E388512FF7D; Tue, 16 Apr 2024 14:10:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276614; cv=none; b=G+ZKSc++xJ6jcy2VVCCWjDtiPYAOXbl/WaRX1mERkNGrq7p8O+8PKp3S7tU8U9SqVx32qd4xKA8RFf9MYMcJz78RB9Yf2Uu6nYDxkvE9l9Tv6Vd6/Inp7dAMJ3fRZKS+24Gy2v6N5BjgId/roL40JAMhsK34dasPS9FjwA3+NM0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276614; c=relaxed/simple; bh=QNpduiTDbFgV1XQkpwYAC0fELGqIw36IV9Gub3d6HyY=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=gRNum4EksYuGo/ZzDod6Fu3cIqVRQkjLCV/qhGwUpI7iS0678TXBzRESoqSxUIun3fPLocMUazsmEF6b7+XXprhcjRPJFcRopE3saoV6kUbsB/CV/ab7wE8jiiyb3zP+FLjsC/1wipi0LrT8bjx0cwQ8tftVKXApluSvg5XPAq4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Onh8glfz; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Onh8glfz" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 650F7C32783; Tue, 16 Apr 2024 14:10:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1713276613; bh=QNpduiTDbFgV1XQkpwYAC0fELGqIw36IV9Gub3d6HyY=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=Onh8glfzi7mAXEuSU8U45xw04zymVR2vePXbabEqvoJ2Z8JiKqfKw9dYsrDnyClRv t+Y8+u2ufqiTQ6Uggui1+sPZ0afLfHXMlmoq/s5lnySguH2wtVUBvLMbJLWAa7V2+F jxpmdTHt4qKLvFo1w+9j2ac2igGlIHgEwhNiJI6MhD54Y98nG9En/RwZTxxueoNpUE vc3oDfF/IWxVdfszU6az5HrQYnVtNKFKd6d366sS4tLlUXDrVs5vS3LXGWJvplWR01 ZTDjn0pBs/7kJ77zLD9fWfUmdS93FhIcv2S+F9afOWQU76dmg3oySgfyjwhDM35yfz L7fhnytDMzpLw== From: Benjamin Tissoires Date: Tue, 16 Apr 2024 16:08:17 +0200 Subject: [PATCH bpf-next 04/18] bpf: replace bpf_timer_set_callback with a generic helper Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240416-bpf_wq-v1-4-c9e66092f842@kernel.org> References: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> In-Reply-To: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Mykola Lysenko , Shuah Khan Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Benjamin Tissoires X-Mailer: b4 0.12.4 X-Developer-Signature: v=1; a=ed25519-sha256; t=1713276593; l=2768; i=bentiss@kernel.org; s=20230215; h=from:subject:message-id; bh=QNpduiTDbFgV1XQkpwYAC0fELGqIw36IV9Gub3d6HyY=; b=iYlPWYCC8itsdSn7+e+hNyNHdvtqZi9LbEt0Odj6Kcc5YbbbL2C8pKv56W4L8WbFaWVenOfSm Pt9oHTcJQ5GBMs1ZCVVdMy+pagTlkWpL6cLkkze3wC/4YMISsxw22HC X-Developer-Key: i=bentiss@kernel.org; a=ed25519; pk=7D1DyAVh6ajCkuUTudt/chMuXWIJHlv2qCsRkIizvFw= X-Patchwork-Delegate: bpf@iogearbox.net In the same way we have a generic __bpf_async_init(), we also need to share code between timer and workqueue for the set_callback call. We just add an unused flags parameter, as it will be used for workqueues. Signed-off-by: Benjamin Tissoires --- kernel/bpf/helpers.c | 29 ++++++++++++++++++----------- 1 file changed, 18 insertions(+), 11 deletions(-) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 1e6d1011303b..d0a645b09d3d 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -1262,22 +1262,23 @@ static const struct bpf_func_proto bpf_timer_init_proto = { .arg3_type = ARG_ANYTHING, }; -BPF_CALL_3(bpf_timer_set_callback, struct bpf_async_kern *, timer, void *, callback_fn, - struct bpf_prog_aux *, aux) +static int __bpf_async_set_callback(struct bpf_async_kern *async, void *callback_fn, + struct bpf_prog_aux *aux, unsigned int flags, + enum bpf_async_type type) { struct bpf_prog *prev, *prog = aux->prog; - struct bpf_hrtimer *t; + struct bpf_async_cb *cb; int ret = 0; if (in_nmi()) return -EOPNOTSUPP; - __bpf_spin_lock_irqsave(&timer->lock); - t = timer->timer; - if (!t) { + __bpf_spin_lock_irqsave(&async->lock); + cb = async->cb; + if (!cb) { ret = -EINVAL; goto out; } - if (!atomic64_read(&t->cb.map->usercnt)) { + if (!atomic64_read(&cb->map->usercnt)) { /* maps with timers must be either held by user space * or pinned in bpffs. Otherwise timer might still be * running even when bpf prog is detached and user space @@ -1286,7 +1287,7 @@ BPF_CALL_3(bpf_timer_set_callback, struct bpf_async_kern *, timer, void *, callb ret = -EPERM; goto out; } - prev = t->cb.prog; + prev = cb->prog; if (prev != prog) { /* Bump prog refcnt once. Every bpf_timer_set_callback() * can pick different callback_fn-s within the same prog. @@ -1299,14 +1300,20 @@ BPF_CALL_3(bpf_timer_set_callback, struct bpf_async_kern *, timer, void *, callb if (prev) /* Drop prev prog refcnt when swapping with new prog */ bpf_prog_put(prev); - t->cb.prog = prog; + cb->prog = prog; } - rcu_assign_pointer(t->cb.callback_fn, callback_fn); + rcu_assign_pointer(cb->callback_fn, callback_fn); out: - __bpf_spin_unlock_irqrestore(&timer->lock); + __bpf_spin_unlock_irqrestore(&async->lock); return ret; } +BPF_CALL_3(bpf_timer_set_callback, struct bpf_async_kern *, timer, void *, callback_fn, + struct bpf_prog_aux *, aux) +{ + return __bpf_async_set_callback(timer, callback_fn, aux, 0, BPF_ASYNC_TYPE_TIMER); +} + static const struct bpf_func_proto bpf_timer_set_callback_proto = { .func = bpf_timer_set_callback, .gpl_only = true, From patchwork Tue Apr 16 14:08:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Tissoires X-Patchwork-Id: 13631925 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E769A130ADF; Tue, 16 Apr 2024 14:10:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276618; cv=none; b=sq6RGAEzYYldv77UUOUb2Ek92Hv2OeeVUaNCfBQSgQvLqPw/VBzu8eqrwHvtEEPIHaPQTb+G68yeTdo6mxgj+n7NHz+P375vMIBYCxRYWSEdXgQ+sv3DxoismD3ck7CTfnVeNdzE73SeBKvjoZYnb0J9BPTooCqjbbfRqSRFjwE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276618; c=relaxed/simple; bh=aV4WdVYxSNjWkAmCyi+VcGi1EENyIRC1qxYyFlJXwE0=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=DAZEnXDun6DMd2J2jHWEES0w/6IVc7MVQ+AT/qMWvZPLt+y2j3qkdKK4F8qpmmka4EW/9RkiUOEo3tqOPUNsTV426UamywlpYdPghUCAtzZGyKtaKRZdaQve+wTLH49uKkWFgxNlXzBj1LKtELIfYcqNt+9lue+f5DrtdG6pUuk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=vMAyYWpk; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="vMAyYWpk" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 31F9EC2BD11; Tue, 16 Apr 2024 14:10:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1713276617; bh=aV4WdVYxSNjWkAmCyi+VcGi1EENyIRC1qxYyFlJXwE0=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=vMAyYWpk8ShSXREPy2AdBgXJ/CTwY2yCNy+uVpmQ4BHym3d2CMcxGthU7S82YEflO KWrDVYtS//sPTp73D1QC8H2s3r1lCyvkWwW/kRaDqT4K+sKzEObZjWUZqHksxD1WCN Ry964+rM2FASYSB5L584tbZ8XTnRvekJwG1TeHwxDFKUwhlSJJ+uloiyZjL+HtRhqm ib9DTPd5jPqt9O7/Jtq9W9x7NSrvzpkpqAuA9iCrV3oBhQQu/6EtQAB8kk5r+Ka+mS sJD/hNMef7dxJNEwErlDgJZdnEuTyjfkhTy3aSnhCsY4NkaLonJlWP5nWJ9vXpgq60 YJw3dOJRus6eQ== From: Benjamin Tissoires Date: Tue, 16 Apr 2024 16:08:18 +0200 Subject: [PATCH bpf-next 05/18] bpf: replace bpf_timer_cancel_and_free with a generic helper Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240416-bpf_wq-v1-5-c9e66092f842@kernel.org> References: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> In-Reply-To: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Mykola Lysenko , Shuah Khan Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Benjamin Tissoires X-Mailer: b4 0.12.4 X-Developer-Signature: v=1; a=ed25519-sha256; t=1713276593; l=3167; i=bentiss@kernel.org; s=20230215; h=from:subject:message-id; bh=aV4WdVYxSNjWkAmCyi+VcGi1EENyIRC1qxYyFlJXwE0=; b=n1oxaBLfR9BhsnTXXtu7/YF0pj8V4GcNcAG5GfaEJB1uHwzkH+PZfg4tHgNARDKNW0v5JtxNr e0wxxCiNIbICB7YhMaLwPEEFeGhKwo24/Y5vUI6TFCdznPRUQDLdPqs X-Developer-Key: i=bentiss@kernel.org; a=ed25519; pk=7D1DyAVh6ajCkuUTudt/chMuXWIJHlv2qCsRkIizvFw= X-Patchwork-Delegate: bpf@iogearbox.net Same reason than most bpf_timer* functions, we need almost the same for workqueues. So extract the generic part out of it so bpf_wq_cancel_and_free can reuse it. Signed-off-by: Benjamin Tissoires --- kernel/bpf/helpers.c | 42 +++++++++++++++++++++++++----------------- 1 file changed, 25 insertions(+), 17 deletions(-) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index d0a645b09d3d..78847f444f79 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -1413,36 +1413,44 @@ static const struct bpf_func_proto bpf_timer_cancel_proto = { .arg1_type = ARG_PTR_TO_TIMER, }; -/* This function is called by map_delete/update_elem for individual element and - * by ops->map_release_uref when the user space reference to a map reaches zero. - */ -void bpf_timer_cancel_and_free(void *val) +static struct bpf_async_cb *__bpf_async_cancel_and_free(struct bpf_async_kern *async) { - struct bpf_async_kern *timer = val; - struct bpf_hrtimer *t; + struct bpf_async_cb *cb; - /* Performance optimization: read timer->timer without lock first. */ - if (!READ_ONCE(timer->timer)) - return; + /* Performance optimization: read async->cb without lock first. */ + if (!READ_ONCE(async->cb)) + return NULL; - __bpf_spin_lock_irqsave(&timer->lock); + __bpf_spin_lock_irqsave(&async->lock); /* re-read it under lock */ - t = timer->timer; - if (!t) + cb = async->cb; + if (!cb) goto out; - drop_prog_refcnt(&t->cb); + drop_prog_refcnt(cb); /* The subsequent bpf_timer_start/cancel() helpers won't be able to use * this timer, since it won't be initialized. */ - WRITE_ONCE(timer->timer, NULL); + WRITE_ONCE(async->cb, NULL); out: - __bpf_spin_unlock_irqrestore(&timer->lock); + __bpf_spin_unlock_irqrestore(&async->lock); + return cb; +} + +/* This function is called by map_delete/update_elem for individual element and + * by ops->map_release_uref when the user space reference to a map reaches zero. + */ +void bpf_timer_cancel_and_free(void *val) +{ + struct bpf_hrtimer *t; + + t = (struct bpf_hrtimer *)__bpf_async_cancel_and_free(val); + if (!t) return; /* Cancel the timer and wait for callback to complete if it was running. * If hrtimer_cancel() can be safely called it's safe to call kfree(t) * right after for both preallocated and non-preallocated maps. - * The timer->timer = NULL was already done and no code path can + * The async->cb = NULL was already done and no code path can * see address 't' anymore. * * Check that bpf_map_delete/update_elem() wasn't called from timer @@ -1451,7 +1459,7 @@ void bpf_timer_cancel_and_free(void *val) * return -1). Though callback_fn is still running on this cpu it's * safe to do kfree(t) because bpf_timer_cb() read everything it needed * from 't'. The bpf subprog callback_fn won't be able to access 't', - * since timer->timer = NULL was already done. The timer will be + * since async->cb = NULL was already done. The timer will be * effectively cancelled because bpf_timer_cb() will return * HRTIMER_NORESTART. */ From patchwork Tue Apr 16 14:08:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Tissoires X-Patchwork-Id: 13631926 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 72DA6130ADF; Tue, 16 Apr 2024 14:10:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276621; cv=none; b=bLeJFVp+yZUxYqV4/3Cmb1/uC4/TobcQrBqv6mUVWljt06PFpBVyxmA+yFq3OHPXPWzPebCaBbA2hAIR5/j+Imj08esFhBi3elz/noOJgPt+DGMzbQu7fERH6pkt0fkIdEKLidotTjOQF9vwjA6oMnk1MfBJAihBJksokFYdzA8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276621; c=relaxed/simple; bh=3JuPv2OIH1rRFqsbCBxaC8MeE5RF3ZyHZxYsRjhBBG4=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=qMgTf40gjfokgTncPgt9V+wEJj+UbSbqPh/5ejq3uU9BTRpnlX983xUQxeXAszDp0tnQhNH3tyvNvMfOgw9IhcikzkcgoIQiyVN2a5Jp3OgmVzqv2coWYPq7lBh9s9SnNRba8ZwHVVQOJfFKJYMzFu1Kv16pmusOxb5Y3xuEDFM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=EqH+wJyh; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="EqH+wJyh" Received: by smtp.kernel.org (Postfix) with ESMTPSA id F32E8C2BD10; Tue, 16 Apr 2024 14:10:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1713276621; bh=3JuPv2OIH1rRFqsbCBxaC8MeE5RF3ZyHZxYsRjhBBG4=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=EqH+wJyhnMAlNWI4bs8b+0rRYTvyihbc+a/ong8zfmY6J4lEbMJOvtyct8hjjwm7B Ly3AILlMDs84wQPGnfQur9LcsXieTPLD7mzA1OUO24iY1oEvbSJyVstC83XRq9Xtt/ jLCU4AOhwCNb76JgO2l01KhPz9GCTAWpk4NYC8TJ3IIkOUFRV8hqjYLI1PtZ282ZEY 8jJbbWnVkOJo6x54IXB1tvzcVESrb7NvtMzQ1M/FVXHygbcVhARkCw6EuisCwHdTo1 6IlI+voqNoZezqD+LQ72GTmtxp7y3e1lgta0Da/fW+3vRnebsWMueewCFdP13kHSRX +/dAMfckmUAMg== From: Benjamin Tissoires Date: Tue, 16 Apr 2024 16:08:19 +0200 Subject: [PATCH bpf-next 06/18] bpf: add support for bpf_wq user type Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240416-bpf_wq-v1-6-c9e66092f842@kernel.org> References: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> In-Reply-To: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Mykola Lysenko , Shuah Khan Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Benjamin Tissoires X-Mailer: b4 0.12.4 X-Developer-Signature: v=1; a=ed25519-sha256; t=1713276593; l=7229; i=bentiss@kernel.org; s=20230215; h=from:subject:message-id; bh=3JuPv2OIH1rRFqsbCBxaC8MeE5RF3ZyHZxYsRjhBBG4=; b=Jczd5KIOTyu9KQsIpAXvyMm6LS0MddQCmwnYXweJmU+g10zjRy6E2ICKBet6N8A+e/coUZ4Nd V0akg4CR6S9AMe6TN7aoos7jP/qlooXtpIYzBsTmKsc0cZTUontMQnZ X-Developer-Key: i=bentiss@kernel.org; a=ed25519; pk=7D1DyAVh6ajCkuUTudt/chMuXWIJHlv2qCsRkIizvFw= X-Patchwork-Delegate: bpf@iogearbox.net Mostly a copy/paste from the bpf_timer API, without the initialization and free, as they will be done in a separate patch. Signed-off-by: Benjamin Tissoires --- include/linux/bpf.h | 11 ++++++++++- include/uapi/linux/bpf.h | 4 ++++ kernel/bpf/btf.c | 17 +++++++++++++++++ kernel/bpf/syscall.c | 6 +++++- kernel/bpf/verifier.c | 9 +++++++++ 5 files changed, 45 insertions(+), 2 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 551445c47779..45cb13dfd15e 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -185,7 +185,7 @@ struct bpf_map_ops { enum { /* Support at most 10 fields in a BTF type */ - BTF_FIELDS_MAX = 10, + BTF_FIELDS_MAX = 11, }; enum btf_field_type { @@ -202,6 +202,7 @@ enum btf_field_type { BPF_GRAPH_NODE = BPF_RB_NODE | BPF_LIST_NODE, BPF_GRAPH_ROOT = BPF_RB_ROOT | BPF_LIST_HEAD, BPF_REFCOUNT = (1 << 9), + BPF_WORKQUEUE = (1 << 10), }; typedef void (*btf_dtor_kfunc_t)(void *); @@ -238,6 +239,7 @@ struct btf_record { u32 field_mask; int spin_lock_off; int timer_off; + int wq_off; int refcount_off; struct btf_field fields[]; }; @@ -312,6 +314,8 @@ static inline const char *btf_field_type_name(enum btf_field_type type) return "bpf_spin_lock"; case BPF_TIMER: return "bpf_timer"; + case BPF_WORKQUEUE: + return "bpf_wq"; case BPF_KPTR_UNREF: case BPF_KPTR_REF: return "kptr"; @@ -340,6 +344,8 @@ static inline u32 btf_field_type_size(enum btf_field_type type) return sizeof(struct bpf_spin_lock); case BPF_TIMER: return sizeof(struct bpf_timer); + case BPF_WORKQUEUE: + return sizeof(struct bpf_wq); case BPF_KPTR_UNREF: case BPF_KPTR_REF: case BPF_KPTR_PERCPU: @@ -367,6 +373,8 @@ static inline u32 btf_field_type_align(enum btf_field_type type) return __alignof__(struct bpf_spin_lock); case BPF_TIMER: return __alignof__(struct bpf_timer); + case BPF_WORKQUEUE: + return __alignof__(struct bpf_wq); case BPF_KPTR_UNREF: case BPF_KPTR_REF: case BPF_KPTR_PERCPU: @@ -406,6 +414,7 @@ static inline void bpf_obj_init_field(const struct btf_field *field, void *addr) /* RB_ROOT_CACHED 0-inits, no need to do anything after memset */ case BPF_SPIN_LOCK: case BPF_TIMER: + case BPF_WORKQUEUE: case BPF_KPTR_UNREF: case BPF_KPTR_REF: case BPF_KPTR_PERCPU: diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index cee0a7915c08..e4ae83550fb3 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -7306,6 +7306,10 @@ struct bpf_timer { __u64 __opaque[2]; } __attribute__((aligned(8))); +struct bpf_wq { + __u64 __opaque[2]; +} __attribute__((aligned(8))); + struct bpf_dynptr { __u64 __opaque[2]; } __attribute__((aligned(8))); diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 90c4a32d89ff..a9b5b28a2630 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -3464,6 +3464,15 @@ static int btf_get_field_type(const char *name, u32 field_mask, u32 *seen_mask, goto end; } } + if (field_mask & BPF_WORKQUEUE) { + if (!strcmp(name, "bpf_wq")) { + if (*seen_mask & BPF_WORKQUEUE) + return -E2BIG; + *seen_mask |= BPF_WORKQUEUE; + type = BPF_WORKQUEUE; + goto end; + } + } field_mask_test_name(BPF_LIST_HEAD, "bpf_list_head"); field_mask_test_name(BPF_LIST_NODE, "bpf_list_node"); field_mask_test_name(BPF_RB_ROOT, "bpf_rb_root"); @@ -3515,6 +3524,7 @@ static int btf_find_struct_field(const struct btf *btf, switch (field_type) { case BPF_SPIN_LOCK: case BPF_TIMER: + case BPF_WORKQUEUE: case BPF_LIST_NODE: case BPF_RB_NODE: case BPF_REFCOUNT: @@ -3582,6 +3592,7 @@ static int btf_find_datasec_var(const struct btf *btf, const struct btf_type *t, switch (field_type) { case BPF_SPIN_LOCK: case BPF_TIMER: + case BPF_WORKQUEUE: case BPF_LIST_NODE: case BPF_RB_NODE: case BPF_REFCOUNT: @@ -3816,6 +3827,7 @@ struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type rec->spin_lock_off = -EINVAL; rec->timer_off = -EINVAL; + rec->wq_off = -EINVAL; rec->refcount_off = -EINVAL; for (i = 0; i < cnt; i++) { field_type_size = btf_field_type_size(info_arr[i].type); @@ -3846,6 +3858,11 @@ struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type /* Cache offset for faster lookup at runtime */ rec->timer_off = rec->fields[i].offset; break; + case BPF_WORKQUEUE: + WARN_ON_ONCE(rec->wq_off >= 0); + /* Cache offset for faster lookup at runtime */ + rec->wq_off = rec->fields[i].offset; + break; case BPF_REFCOUNT: WARN_ON_ONCE(rec->refcount_off >= 0); /* Cache offset for faster lookup at runtime */ diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 7d392ec83655..0848e4141b00 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -559,6 +559,7 @@ void btf_record_free(struct btf_record *rec) case BPF_SPIN_LOCK: case BPF_TIMER: case BPF_REFCOUNT: + case BPF_WORKQUEUE: /* Nothing to release */ break; default: @@ -608,6 +609,7 @@ struct btf_record *btf_record_dup(const struct btf_record *rec) case BPF_SPIN_LOCK: case BPF_TIMER: case BPF_REFCOUNT: + case BPF_WORKQUEUE: /* Nothing to acquire */ break; default: @@ -679,6 +681,8 @@ void bpf_obj_free_fields(const struct btf_record *rec, void *obj) case BPF_TIMER: bpf_timer_cancel_and_free(field_ptr); break; + case BPF_WORKQUEUE: + break; case BPF_KPTR_UNREF: WRITE_ONCE(*(u64 *)field_ptr, 0); break; @@ -1085,7 +1089,7 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token, map->record = btf_parse_fields(btf, value_type, BPF_SPIN_LOCK | BPF_TIMER | BPF_KPTR | BPF_LIST_HEAD | - BPF_RB_ROOT | BPF_REFCOUNT, + BPF_RB_ROOT | BPF_REFCOUNT | BPF_WORKQUEUE, map->value_size); if (!IS_ERR_OR_NULL(map->record)) { int i; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 2aad6d90550f..deaf2e1ab690 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -1838,6 +1838,8 @@ static void mark_ptr_not_null_reg(struct bpf_reg_state *reg) */ if (btf_record_has_field(map->inner_map_meta->record, BPF_TIMER)) reg->map_uid = reg->id; + if (btf_record_has_field(map->inner_map_meta->record, BPF_WORKQUEUE)) + reg->map_uid = reg->id; } else if (map->map_type == BPF_MAP_TYPE_XSKMAP) { reg->type = PTR_TO_XDP_SOCK; } else if (map->map_type == BPF_MAP_TYPE_SOCKMAP || @@ -18155,6 +18157,13 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, } } + if (btf_record_has_field(map->record, BPF_WORKQUEUE)) { + if (is_tracing_prog_type(prog_type)) { + verbose(env, "tracing progs cannot use bpf_wq yet\n"); + return -EINVAL; + } + } + if ((bpf_prog_is_offloaded(prog->aux) || bpf_map_is_offloaded(map)) && !bpf_offload_prog_map_match(prog, map)) { verbose(env, "offload device mismatch between prog and map\n"); From patchwork Tue Apr 16 14:08:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Tissoires X-Patchwork-Id: 13631927 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 89BC6131742; Tue, 16 Apr 2024 14:10:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276625; cv=none; b=FzJHoyoZeemb2mGMuL1EYOYmkthOV4D3l0NtSAwt8uCiZVMNlwhqavlY2y/Z/uzYQcaYbl7q0X+mOO7+jSkaWffuLnOWGGjbMTSJ/9hO96a0ZuK9smh0zu5kcr0AQzeWPF5lm9ZJWfuQa8UuGSXo3YJlNS9NXwluuvNnmC08kaY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276625; c=relaxed/simple; bh=9oDLRiiM+JPmmWah55f4UeKYjqp4suxwjRoWuUuu+3E=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=IaQQ9bWVhmPEscgz80WtYOnvryruSgghFHafNqdIG2X31oRtn27MZN/icngyIBB/qrY2Utg5J7ZLKBiyWzL4HMRDw7OdV3QIuV7ZhWhnCeFP2+vB9bbN6epxmqOaO6/x/cBisq93BcaG3M515LWZqfcjyO2KaboRo0DoSAzkftU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=l+hhsgUM; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="l+hhsgUM" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C1376C2BD10; Tue, 16 Apr 2024 14:10:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1713276625; bh=9oDLRiiM+JPmmWah55f4UeKYjqp4suxwjRoWuUuu+3E=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=l+hhsgUMj1ryVbNOkDI9Pv6zSEgiPwQASJ18l76IcVMDxByhOYQMRDfMUYGFU2p7T jEPJd/aR1IL0sMvbkMXZeVIzJRtfGCLZnx67oHpk5QKe6wCJALyoXJ0Iaou1THlpi1 V1cqzBnB6oR/Ij795wMyuaxHgl/IB8wtT+QXWhIva29SCkorMhM2PtRIy4r18nvvxm BqwPnToAQ6kz758fSg75VUDvpRtZRX/0o5MNRylH2FGoiSsYHDMQPA9FWr1+nSWVl9 ovepCyvsw/2ySzY8vP8l0NlCOheCcnFlr+K1V90Dp283/h0+AZJuRiEryoc7/HKW5U rES21gFLM6AUg== From: Benjamin Tissoires Date: Tue, 16 Apr 2024 16:08:20 +0200 Subject: [PATCH bpf-next 07/18] tools: sync include/uapi/linux/bpf.h Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240416-bpf_wq-v1-7-c9e66092f842@kernel.org> References: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> In-Reply-To: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Mykola Lysenko , Shuah Khan Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Benjamin Tissoires X-Mailer: b4 0.12.4 X-Developer-Signature: v=1; a=ed25519-sha256; t=1713276593; l=665; i=bentiss@kernel.org; s=20230215; h=from:subject:message-id; bh=9oDLRiiM+JPmmWah55f4UeKYjqp4suxwjRoWuUuu+3E=; b=2+hnYzLBcNJUrnVQt3aPnXWg1McU5HMAixFaWKJpbu8wORHXzuChF3kZemEDt3+qUbkBiocNb 8CSpi5DsBUCBb/ODxcMKaOcv4odmCHmd292kKhNED6D6KzNu/9JDbN4 X-Developer-Key: i=bentiss@kernel.org; a=ed25519; pk=7D1DyAVh6ajCkuUTudt/chMuXWIJHlv2qCsRkIizvFw= X-Patchwork-Delegate: bpf@iogearbox.net cp include/uapi/linux/bpf.h tools/include/uapi/linux/bpf.h Signed-off-by: Benjamin Tissoires --- tools/include/uapi/linux/bpf.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index cee0a7915c08..e4ae83550fb3 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -7306,6 +7306,10 @@ struct bpf_timer { __u64 __opaque[2]; } __attribute__((aligned(8))); +struct bpf_wq { + __u64 __opaque[2]; +} __attribute__((aligned(8))); + struct bpf_dynptr { __u64 __opaque[2]; } __attribute__((aligned(8))); From patchwork Tue Apr 16 14:08:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Tissoires X-Patchwork-Id: 13631928 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 168DE131BA7; Tue, 16 Apr 2024 14:10:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276629; cv=none; b=jH+RCiCks36VIeJWM1IhUrKNjrd1eN0Syc2GxPOi/tCzSsx+d9aLM9BKyFdy8UgVNdvFGy4F+wAmO//1JoOHx3VcgykV1sM3kZeHsarssL8MmeytkQzYGGEVLGDgDfh+WEgqCnnZ7DpGrknmBZRvRL4UqqAOcWhJ0ggqJV8CMy8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276629; c=relaxed/simple; bh=WEX5TCOm7sV2DRInTHa7TcVznbYdnMLXwRscaz2H/Oo=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Txagp6GduHtyoYQygbHaLjyYVRWcqO+SeLXgYDFjSNzLsTL51HyOh1EY4Uv9QIO8SIsPO+yowWLN4/HZfUqNyV4shaughamRgIBvtAbRMHhOWtFKMRMJLoM3SotJ8uqm/o1Ky9NPiGTchj45Q7yn83e9nLKoa1TQB2VxHGgLy8E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Hk+ckEvn; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Hk+ckEvn" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8F467C4AF07; Tue, 16 Apr 2024 14:10:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1713276628; bh=WEX5TCOm7sV2DRInTHa7TcVznbYdnMLXwRscaz2H/Oo=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=Hk+ckEvnbXVI+RMSK/PNKe8c/3TZsVsr2KUXQXJinefFa7YdMTfITsTj7QdheBHak e1fO6mybFETfXeYJvVcmI0PkK0i+cLCbgJUZlbVr8gKiJQ9sI7kBccP1Rx6URLi3t3 tiSJwHgygjawdX1wWgkXcaT4RGonTipX3JFLDHep8HvbKF2xEfqg10OQuWUgAxv9hc PWHvi2lY3ER56B1YQddu4pahxAk98EaNeiuIjwi6G5/Sce7+DvapCuKhUWX5P1sRCe VAByECc1lAcl9SZ+JL0biBaCtLXjC2c0OYr4zOJmAVl5bxZOHutFapXKsnX+ly/y65 6v4thz6NlXBAQ== From: Benjamin Tissoires Date: Tue, 16 Apr 2024 16:08:21 +0200 Subject: [PATCH bpf-next 08/18] bpf: add support for KF_ARG_PTR_TO_WORKQUEUE Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240416-bpf_wq-v1-8-c9e66092f842@kernel.org> References: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> In-Reply-To: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Mykola Lysenko , Shuah Khan Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Benjamin Tissoires X-Mailer: b4 0.12.4 X-Developer-Signature: v=1; a=ed25519-sha256; t=1713276593; l=6022; i=bentiss@kernel.org; s=20230215; h=from:subject:message-id; bh=WEX5TCOm7sV2DRInTHa7TcVznbYdnMLXwRscaz2H/Oo=; b=OrTLcEBuZl/n0Yju6qg5Yz+voSFvYO4cMqAy+HLbo7KZyroiTxi24d86kcsD6Qw6llNHH4g1+ f9BtKCyyzRkCl9C1U0pjE82yE69UPP+pAkc1h0sJnw3/Vy0nbedNV3x X-Developer-Key: i=bentiss@kernel.org; a=ed25519; pk=7D1DyAVh6ajCkuUTudt/chMuXWIJHlv2qCsRkIizvFw= X-Patchwork-Delegate: bpf@iogearbox.net Introduce support for KF_ARG_PTR_TO_WORKQUEUE. The kfuncs will use bpf_wq as argument and that will be recognized as workqueue argument by verifier. bpf_wq_kern casting can happen inside kfunc, but using bpf_wq in argument makes life easier for users who work with non-kern type in BPF progs. Duplicate process_timer_func into process_wq_func. meta argument is only needed to ensure bpf_wq_init's workqueue and map arguments are coming from the same map (map_uid logic is necessary for correct inner-map handling), so also amend check_kfunc_args() to match what helpers functions check is doing. Signed-off-by: Benjamin Tissoires --- kernel/bpf/verifier.c | 86 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index deaf2e1ab690..112faf2cd7e9 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -332,6 +332,10 @@ struct bpf_kfunc_call_arg_meta { u8 spi; u8 frameno; } iter; + struct { + struct bpf_map *ptr; + int uid; + } map; u64 mem_size; }; @@ -7598,6 +7602,43 @@ static int process_timer_func(struct bpf_verifier_env *env, int regno, return 0; } +static int process_wq_func(struct bpf_verifier_env *env, int regno, + struct bpf_kfunc_call_arg_meta *meta) +{ + struct bpf_reg_state *regs = cur_regs(env), *reg = ®s[regno]; + bool is_const = tnum_is_const(reg->var_off); + struct bpf_map *map = reg->map_ptr; + u64 val = reg->var_off.value; + + if (!is_const) { + verbose(env, + "R%d doesn't have constant offset. bpf_wq has to be at the constant offset\n", + regno); + return -EINVAL; + } + if (!map->btf) { + verbose(env, "map '%s' has to have BTF in order to use bpf_wq\n", + map->name); + return -EINVAL; + } + if (!btf_record_has_field(map->record, BPF_WORKQUEUE)) { + verbose(env, "map '%s' has no valid bpf_wq\n", map->name); + return -EINVAL; + } + if (map->record->wq_off != val + reg->off) { + verbose(env, "off %lld doesn't point to 'struct bpf_wq' that is at %d\n", + val + reg->off, map->record->wq_off); + return -EINVAL; + } + if (meta->map.ptr) { + verbose(env, "verifier bug. Two map pointers in a workqueue helper\n"); + return -EFAULT; + } + meta->map.uid = reg->map_uid; + meta->map.ptr = map; + return 0; +} + static int process_kptr_func(struct bpf_verifier_env *env, int regno, struct bpf_call_arg_meta *meta) { @@ -10843,6 +10884,7 @@ enum { KF_ARG_LIST_NODE_ID, KF_ARG_RB_ROOT_ID, KF_ARG_RB_NODE_ID, + KF_ARG_WORKQUEUE_ID, }; BTF_ID_LIST(kf_arg_btf_ids) @@ -10851,6 +10893,7 @@ BTF_ID(struct, bpf_list_head) BTF_ID(struct, bpf_list_node) BTF_ID(struct, bpf_rb_root) BTF_ID(struct, bpf_rb_node) +BTF_ID(struct, bpf_wq) static bool __is_kfunc_ptr_arg_type(const struct btf *btf, const struct btf_param *arg, int type) @@ -10894,6 +10937,11 @@ static bool is_kfunc_arg_rbtree_node(const struct btf *btf, const struct btf_par return __is_kfunc_ptr_arg_type(btf, arg, KF_ARG_RB_NODE_ID); } +static bool is_kfunc_arg_wq(const struct btf *btf, const struct btf_param *arg) +{ + return __is_kfunc_ptr_arg_type(btf, arg, KF_ARG_WORKQUEUE_ID); +} + static bool is_kfunc_arg_callback(struct bpf_verifier_env *env, const struct btf *btf, const struct btf_param *arg) { @@ -10963,6 +11011,7 @@ enum kfunc_ptr_arg_type { KF_ARG_PTR_TO_NULL, KF_ARG_PTR_TO_CONST_STR, KF_ARG_PTR_TO_MAP, + KF_ARG_PTR_TO_WORKQUEUE, }; enum special_kfunc_type { @@ -11119,6 +11168,9 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, if (is_kfunc_arg_map(meta->btf, &args[argno])) return KF_ARG_PTR_TO_MAP; + if (is_kfunc_arg_wq(meta->btf, &args[argno])) + return KF_ARG_PTR_TO_WORKQUEUE; + if ((base_type(reg->type) == PTR_TO_BTF_ID || reg2btf_ids[base_type(reg->type)])) { if (!btf_type_is_struct(ref_t)) { verbose(env, "kernel function %s args#%d pointer type %s %s is not supported\n", @@ -11720,6 +11772,30 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ case KF_ARG_PTR_TO_NULL: continue; case KF_ARG_PTR_TO_MAP: + if (meta->map.ptr) { + /* Use map_uid (which is unique id of inner map) to reject: + * inner_map1 = bpf_map_lookup_elem(outer_map, key1) + * inner_map2 = bpf_map_lookup_elem(outer_map, key2) + * if (inner_map1 && inner_map2) { + * timer = bpf_map_lookup_elem(inner_map1); + * if (timer) + * // mismatch would have been allowed + * bpf_timer_init(timer, inner_map2); + * } + * + * Comparing map_ptr is enough to distinguish normal and outer maps. + */ + if (meta->map.ptr != reg->map_ptr || + meta->map.uid != reg->map_uid) { + verbose(env, + "workqueue pointer in R1 map_uid=%d doesn't match map pointer in R2 map_uid=%d\n", + meta->map.uid, reg->map_uid); + return -EINVAL; + } + } + meta->map.ptr = reg->map_ptr; + meta->map.uid = reg->map_uid; + fallthrough; case KF_ARG_PTR_TO_ALLOC_BTF_ID: case KF_ARG_PTR_TO_BTF_ID: if (!is_kfunc_trusted_args(meta) && !is_kfunc_rcu(meta)) @@ -11752,6 +11828,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ case KF_ARG_PTR_TO_CALLBACK: case KF_ARG_PTR_TO_REFCOUNTED_KPTR: case KF_ARG_PTR_TO_CONST_STR: + case KF_ARG_PTR_TO_WORKQUEUE: /* Trusted by default */ break; default: @@ -12038,6 +12115,15 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ if (ret) return ret; break; + case KF_ARG_PTR_TO_WORKQUEUE: + if (reg->type != PTR_TO_MAP_VALUE) { + verbose(env, "arg#%d doesn't point to a map value\n", i); + return -EINVAL; + } + ret = process_wq_func(env, regno, meta); + if (ret < 0) + return ret; + break; } } From patchwork Tue Apr 16 14:08:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Tissoires X-Patchwork-Id: 13631929 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3F8D512EBC8; Tue, 16 Apr 2024 14:10:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276633; cv=none; b=WLr6uoC07096E1wb1A0eMOT5MuJJ4ZtoNiW9xyJBdwDib7MapthCVnx/TyH61K6aeNFp2lSb8vfvSubx4UDZLaXe69SapiaIJr99FSPkQhUKQBj+nArorQPBVNIGyozbElltFjKel/hjQkgs4nqrBoQxXFjgLyGhS4abxamnGjs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276633; c=relaxed/simple; bh=AYN3qRJfIogqn6JsNPQ3ZFAMEc9fbXRyMEOQJ+XouF0=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=VsHkHg03LN6q/TRwGXIOSk2db3GKeQwQIOLFxp1Z8j6FiXwEMca7moqJM3HGbu6RMdYh3CjdpCKMU9Y7nTrGaFO3iO1djfgi27iH+VwMeAUwQcDptvjfpmjyEASDyKziNTMG0rb0J+SxPuLSLOw1X0YS/9+6zCZfmHt1p/XRL68= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=KqPbWwCh; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="KqPbWwCh" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5E64DC2BD11; Tue, 16 Apr 2024 14:10:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1713276632; bh=AYN3qRJfIogqn6JsNPQ3ZFAMEc9fbXRyMEOQJ+XouF0=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=KqPbWwChUYh9CuIYGsM45OeR8gHirosw5lCQc7jeN7eYYKz2QT0VudbDgUPHi8/pq fJX44A/Ma6eJY7UxIEuithRmFXEaO74J++Xn54uoEVCZ4HL/fF29IDCZzfS1VWKuc6 SQ/y4Br5rQ1aFgaF192YaV03v1aqbPC/FOOE73+SpyzWE4FiskDF6jVEAeIoWF2LBa YSq1Ylq14oXywwMvr6y+ANOUiGl7dVWdWixF0fgA8ux/Q/ikVYKM1VbpM78PtZPCzJ PG++OIzBKWckMXzjAfSRyKd8otF3LdF16gbVdv+fX9pPM9XDTLmxhren8Ap8DKuT6k xFfKqjmSxfYOw== From: Benjamin Tissoires Date: Tue, 16 Apr 2024 16:08:22 +0200 Subject: [PATCH bpf-next 09/18] bpf: allow struct bpf_wq to be embedded in arraymaps and hashmaps Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240416-bpf_wq-v1-9-c9e66092f842@kernel.org> References: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> In-Reply-To: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Mykola Lysenko , Shuah Khan Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Benjamin Tissoires X-Mailer: b4 0.12.4 X-Developer-Signature: v=1; a=ed25519-sha256; t=1713276593; l=8819; i=bentiss@kernel.org; s=20230215; h=from:subject:message-id; bh=AYN3qRJfIogqn6JsNPQ3ZFAMEc9fbXRyMEOQJ+XouF0=; b=FTuU3Yl856EB61VgH3hmngk/txgG5WDx4xowYWO5W3+EDbdlnTdvw6g48o66pf87jLOWdZy/8 5lp1H4e7t53Czbnph8p3gOuxlmrXnihkYa/onTt8yD0EBC3i7Bidmhj X-Developer-Key: i=bentiss@kernel.org; a=ed25519; pk=7D1DyAVh6ajCkuUTudt/chMuXWIJHlv2qCsRkIizvFw= X-Patchwork-Delegate: bpf@iogearbox.net Currently bpf_wq_cancel_and_free() is just a placeholder as there is no memory allocation for bpf_wq just yet. Again, duplication of the bpf_timer approach Signed-off-by: Benjamin Tissoires --- include/linux/bpf.h | 2 ++ kernel/bpf/arraymap.c | 18 ++++++++++------- kernel/bpf/hashtab.c | 55 ++++++++++++++++++++++++++++++++++++++++----------- kernel/bpf/helpers.c | 8 ++++++++ kernel/bpf/syscall.c | 10 ++++++++++ 5 files changed, 74 insertions(+), 19 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 45cb13dfd15e..9ea50d9c7a0c 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -534,6 +534,7 @@ static inline void zero_map_value(struct bpf_map *map, void *dst) void copy_map_value_locked(struct bpf_map *map, void *dst, void *src, bool lock_src); void bpf_timer_cancel_and_free(void *timer); +void bpf_wq_cancel_and_free(void *timer); void bpf_list_head_free(const struct btf_field *field, void *list_head, struct bpf_spin_lock *spin_lock); void bpf_rb_root_free(const struct btf_field *field, void *rb_root, @@ -2208,6 +2209,7 @@ void bpf_map_free_record(struct bpf_map *map); struct btf_record *btf_record_dup(const struct btf_record *rec); bool btf_record_equal(const struct btf_record *rec_a, const struct btf_record *rec_b); void bpf_obj_free_timer(const struct btf_record *rec, void *obj); +void bpf_obj_free_workqueue(const struct btf_record *rec, void *obj); void bpf_obj_free_fields(const struct btf_record *rec, void *obj); void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu); diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index 8c1e6d7654bb..580d07b15471 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -428,17 +428,21 @@ static void *array_map_vmalloc_addr(struct bpf_array *array) return (void *)round_down((unsigned long)array, PAGE_SIZE); } -static void array_map_free_timers(struct bpf_map *map) +static void array_map_free_timers_wq(struct bpf_map *map) { struct bpf_array *array = container_of(map, struct bpf_array, map); int i; - /* We don't reset or free fields other than timer on uref dropping to zero. */ - if (!btf_record_has_field(map->record, BPF_TIMER)) - return; + /* We don't reset or free fields other than timer and workqueue + * on uref dropping to zero. + */ + if (btf_record_has_field(map->record, BPF_TIMER)) + for (i = 0; i < array->map.max_entries; i++) + bpf_obj_free_timer(map->record, array_map_elem_ptr(array, i)); - for (i = 0; i < array->map.max_entries; i++) - bpf_obj_free_timer(map->record, array_map_elem_ptr(array, i)); + if (btf_record_has_field(map->record, BPF_WORKQUEUE)) + for (i = 0; i < array->map.max_entries; i++) + bpf_obj_free_workqueue(map->record, array_map_elem_ptr(array, i)); } /* Called when map->refcnt goes to zero, either from workqueue or from syscall */ @@ -782,7 +786,7 @@ const struct bpf_map_ops array_map_ops = { .map_alloc = array_map_alloc, .map_free = array_map_free, .map_get_next_key = array_map_get_next_key, - .map_release_uref = array_map_free_timers, + .map_release_uref = array_map_free_timers_wq, .map_lookup_elem = array_map_lookup_elem, .map_update_elem = array_map_update_elem, .map_delete_elem = array_map_delete_elem, diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 83a9a74260e9..4f8590067c6a 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -240,6 +240,26 @@ static void htab_free_prealloced_timers(struct bpf_htab *htab) } } +static void htab_free_prealloced_wq(struct bpf_htab *htab) +{ + u32 num_entries = htab->map.max_entries; + int i; + + if (!btf_record_has_field(htab->map.record, BPF_WORKQUEUE)) + return; + if (htab_has_extra_elems(htab)) + num_entries += num_possible_cpus(); + + for (i = 0; i < num_entries; i++) { + struct htab_elem *elem; + + elem = get_htab_elem(htab, i); + bpf_obj_free_workqueue(htab->map.record, + elem->key + round_up(htab->map.key_size, 8)); + cond_resched(); + } +} + static void htab_free_prealloced_fields(struct bpf_htab *htab) { u32 num_entries = htab->map.max_entries; @@ -1495,7 +1515,7 @@ static void delete_all_elements(struct bpf_htab *htab) migrate_enable(); } -static void htab_free_malloced_timers(struct bpf_htab *htab) +static void htab_free_malloced_timers_or_wq(struct bpf_htab *htab, bool is_timer) { int i; @@ -1507,24 +1527,35 @@ static void htab_free_malloced_timers(struct bpf_htab *htab) hlist_nulls_for_each_entry(l, n, head, hash_node) { /* We only free timer on uref dropping to zero */ - bpf_obj_free_timer(htab->map.record, l->key + round_up(htab->map.key_size, 8)); + if (is_timer) + bpf_obj_free_timer(htab->map.record, + l->key + round_up(htab->map.key_size, 8)); + else + bpf_obj_free_workqueue(htab->map.record, + l->key + round_up(htab->map.key_size, 8)); } cond_resched_rcu(); } rcu_read_unlock(); } -static void htab_map_free_timers(struct bpf_map *map) +static void htab_map_free_timers_and_wq(struct bpf_map *map) { struct bpf_htab *htab = container_of(map, struct bpf_htab, map); - /* We only free timer on uref dropping to zero */ - if (!btf_record_has_field(htab->map.record, BPF_TIMER)) - return; - if (!htab_is_prealloc(htab)) - htab_free_malloced_timers(htab); - else - htab_free_prealloced_timers(htab); + /* We only free timer and workqueue on uref dropping to zero */ + if (btf_record_has_field(htab->map.record, BPF_TIMER)) { + if (!htab_is_prealloc(htab)) + htab_free_malloced_timers_or_wq(htab, true); + else + htab_free_prealloced_timers(htab); + } + if (btf_record_has_field(htab->map.record, BPF_WORKQUEUE)) { + if (!htab_is_prealloc(htab)) + htab_free_malloced_timers_or_wq(htab, false); + else + htab_free_prealloced_wq(htab); + } } /* Called when map->refcnt goes to zero, either from workqueue or from syscall */ @@ -2260,7 +2291,7 @@ const struct bpf_map_ops htab_map_ops = { .map_alloc = htab_map_alloc, .map_free = htab_map_free, .map_get_next_key = htab_map_get_next_key, - .map_release_uref = htab_map_free_timers, + .map_release_uref = htab_map_free_timers_and_wq, .map_lookup_elem = htab_map_lookup_elem, .map_lookup_and_delete_elem = htab_map_lookup_and_delete_elem, .map_update_elem = htab_map_update_elem, @@ -2281,7 +2312,7 @@ const struct bpf_map_ops htab_lru_map_ops = { .map_alloc = htab_map_alloc, .map_free = htab_map_free, .map_get_next_key = htab_map_get_next_key, - .map_release_uref = htab_map_free_timers, + .map_release_uref = htab_map_free_timers_and_wq, .map_lookup_elem = htab_lru_map_lookup_elem, .map_lookup_and_delete_elem = htab_lru_map_lookup_and_delete_elem, .map_lookup_elem_sys_only = htab_lru_map_lookup_elem_sys, diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 78847f444f79..9fd12d480b8b 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -1468,6 +1468,14 @@ void bpf_timer_cancel_and_free(void *val) kfree_rcu(t, cb.rcu); } +/* This function is called by map_delete/update_elem for individual element and + * by ops->map_release_uref when the user space reference to a map reaches zero. + */ +void bpf_wq_cancel_and_free(void *val) +{ + BTF_TYPE_EMIT(struct bpf_wq); +} + BPF_CALL_2(bpf_kptr_xchg, void *, map_value, void *, ptr) { unsigned long *kptr = map_value; diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 0848e4141b00..c793be6b34b2 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -661,6 +661,13 @@ void bpf_obj_free_timer(const struct btf_record *rec, void *obj) bpf_timer_cancel_and_free(obj + rec->timer_off); } +void bpf_obj_free_workqueue(const struct btf_record *rec, void *obj) +{ + if (WARN_ON_ONCE(!btf_record_has_field(rec, BPF_WORKQUEUE))) + return; + bpf_wq_cancel_and_free(obj + rec->wq_off); +} + void bpf_obj_free_fields(const struct btf_record *rec, void *obj) { const struct btf_field *fields; @@ -682,6 +689,7 @@ void bpf_obj_free_fields(const struct btf_record *rec, void *obj) bpf_timer_cancel_and_free(field_ptr); break; case BPF_WORKQUEUE: + bpf_wq_cancel_and_free(field_ptr); break; case BPF_KPTR_UNREF: WRITE_ONCE(*(u64 *)field_ptr, 0); @@ -1119,6 +1127,8 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token, } break; case BPF_TIMER: + fallthrough; + case BPF_WORKQUEUE: if (map->map_type != BPF_MAP_TYPE_HASH && map->map_type != BPF_MAP_TYPE_LRU_HASH && map->map_type != BPF_MAP_TYPE_ARRAY) { From patchwork Tue Apr 16 14:08:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Tissoires X-Patchwork-Id: 13631930 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A37A91327F1; Tue, 16 Apr 2024 14:10:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276636; cv=none; b=UdxBFjSKynhLJSVz2nJV91/K/SAgfOjbohcWdORxxuYNowMGfiqXqEesVFCEX/DutOhc83IA1w74s1DS2I35JVCurknc4fZFIvZm3VNvf4jm7ZEKECbxbSraa8z5471FM8jlNGFRh03yh48FfRTem1rcn+ditGJeQeV6n1OIWeY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276636; c=relaxed/simple; bh=roKe6QLjZdl9WIQ9S2oJ9wC/tRJNZ2Mefv37bh3CfbI=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=cllqX+LU8L4BjFRpHx8GUD4s1ddlCTz//Kbn1JpV2gyOscnj7k0rrIU06QLtRxliovR4y1fz6vM7Tb+osNtHNC7H+6+ZyWPUQhbtXlFZ+wxGf3CZqkNFSAR5RFsYPWO4zrb660gIw9PHCinUkqajgKYhuz5sgpMMCWUy6uxix9w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ongPUkZu; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ongPUkZu" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2E210C4AF15; Tue, 16 Apr 2024 14:10:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1713276636; bh=roKe6QLjZdl9WIQ9S2oJ9wC/tRJNZ2Mefv37bh3CfbI=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=ongPUkZusfQcK3BlftlK9CngsAvAFUxzWB+ZYyfVBSY1jezJyeSd2PRGtEdgOeJN5 q4FGuRh5m2YNQ3WO78doX1nejPeIq2mK3qhie/5IndJ4H3BooHYCzhsH95fZAEtmoE tzkJexlHdWvOWcARws2+Masnyc5bxqUdrt1VcBBRukDC7QzkclyzpF8rcZ96RCXE4H RavDL1AZZxVnRo03iUjc8J2SpJo0tO98U3xL1i78vF3GweJMtwlMsQ77Aii59Ltobe yQ3FKvUgNJakfZan/oWk9JI5oFaWlkQ5eaStICPrbbW1nTgN801xLYrUWC6S/foq81 paUuy7XFDjK/Q== From: Benjamin Tissoires Date: Tue, 16 Apr 2024 16:08:23 +0200 Subject: [PATCH bpf-next 10/18] selftests/bpf: add bpf_wq tests Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240416-bpf_wq-v1-10-c9e66092f842@kernel.org> References: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> In-Reply-To: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Mykola Lysenko , Shuah Khan Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Benjamin Tissoires X-Mailer: b4 0.12.4 X-Developer-Signature: v=1; a=ed25519-sha256; t=1713276593; l=3772; i=bentiss@kernel.org; s=20230215; h=from:subject:message-id; bh=roKe6QLjZdl9WIQ9S2oJ9wC/tRJNZ2Mefv37bh3CfbI=; b=Ey1VwPVfH8tQjTD8L4stnUcwSbR0Jl/+gsak1M1w03LAPb3x4CD1hQYqFwl9tXXmGowCGLZ0U J0SyZi6LmBQD3xbXAR8qrfNuSGr/FoZdGLXYekDcrRwbjKDZKFilOrm X-Developer-Key: i=bentiss@kernel.org; a=ed25519; pk=7D1DyAVh6ajCkuUTudt/chMuXWIJHlv2qCsRkIizvFw= X-Patchwork-Delegate: bpf@iogearbox.net We simply try in all supported map types if we can store/load a bpf_wq. Signed-off-by: Benjamin Tissoires --- tools/testing/selftests/bpf/prog_tests/wq.c | 11 +++ tools/testing/selftests/bpf/progs/wq.c | 135 ++++++++++++++++++++++++++++ 2 files changed, 146 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/wq.c b/tools/testing/selftests/bpf/prog_tests/wq.c new file mode 100644 index 000000000000..9a07b8bc2c52 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/wq.c @@ -0,0 +1,11 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Benjamin Tissoires */ +#include +#include "wq.skel.h" + +void serial_test_wq(void) +{ + LIBBPF_OPTS(bpf_test_run_opts, topts); + + RUN_TESTS(wq); +} diff --git a/tools/testing/selftests/bpf/progs/wq.c b/tools/testing/selftests/bpf/progs/wq.c new file mode 100644 index 000000000000..8c668ad04059 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/wq.c @@ -0,0 +1,135 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Benjamin Tissoires + */ + +#include "bpf_experimental.h" +#include +#include "bpf_misc.h" +#include "../bpf_testmod/bpf_testmod_kfunc.h" + +char _license[] SEC("license") = "GPL"; + +struct hmap_elem { + int counter; + struct bpf_timer timer; /* unused */ + struct bpf_spin_lock lock; /* unused */ + struct bpf_wq work; +}; + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(max_entries, 1000); + __type(key, int); + __type(value, struct hmap_elem); +} hmap SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(map_flags, BPF_F_NO_PREALLOC); + __uint(max_entries, 1000); + __type(key, int); + __type(value, struct hmap_elem); +} hmap_malloc SEC(".maps"); + +struct elem { + struct bpf_wq w; +}; + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 2); + __type(key, int); + __type(value, struct elem); +} array SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_LRU_HASH); + __uint(max_entries, 4); + __type(key, int); + __type(value, struct elem); +} lru SEC(".maps"); + +static int test_elem_callback(void *map, int *key) +{ + struct elem init = {}, *val; + + if (map == &lru && + bpf_map_update_elem(map, key, &init, 0)) + return -1; + + val = bpf_map_lookup_elem(map, key); + if (!val) + return -2; + + return 0; +} + +static int test_hmap_elem_callback(void *map, int *key) +{ + struct hmap_elem init = {}, *val; + + if (bpf_map_update_elem(map, key, &init, 0)) + return -1; + + val = bpf_map_lookup_elem(map, key); + if (!val) + return -2; + + return 0; +} + +SEC("tc") +/* test that workqueues can be used from an array + */ +__retval(0) +long test_call_array_sleepable(void *ctx) +{ + int key = 0; + + return test_elem_callback(&array, &key); +} + +SEC("syscall") +/* Same test than above but from a sleepable context. + */ +__retval(0) +long test_syscall_array_sleepable(void *ctx) +{ + int key = 1; + + return test_elem_callback(&array, &key); +} + +SEC("tc") +/* test that workqueues can be used from a hashmap + */ +__retval(0) +long test_call_hash_sleepable(void *ctx) +{ + int key = 2; + + return test_hmap_elem_callback(&hmap, &key); +} + +SEC("tc") +/* test that workqueues can be used from a hashmap + * with NO_PREALLOC. + */ +__retval(0) +long test_call_hash_malloc_sleepable(void *ctx) +{ + int key = 3; + + return test_hmap_elem_callback(&hmap_malloc, &key); +} + +SEC("tc") +/* test that workqueues can be used from a LRU map + */ +__retval(0) +long test_call_lru_sleepable(void *ctx) +{ + int key = 4; + + return test_elem_callback(&lru, &key); +} From patchwork Tue Apr 16 14:08:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Tissoires X-Patchwork-Id: 13631931 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C87D11327F1; Tue, 16 Apr 2024 14:10:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276640; cv=none; b=NKw36BiUw2w0t2l3o92fpd7Ed1eu2XqDfpXEO2tOwFZ/zvipNvFA3fQpzzjKMAVzl/yaDN693m6HdfLNLW/2x5cnqhT00Lmtwuiw3tc6bYc3kv6OdUJksBwC9x+VrgHUVN8ZHy78nOyJmmsAMeQWhiwFQRZxpYhESWQ3k6SYgjM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276640; c=relaxed/simple; bh=IGvHj5WwwNKH6tN7p6OYCT0ik3l5G0n3uBg9+Iju0zQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=LgUNdLH4x3eCP8lNvkQBZLReHgtc1nBwm0pw1vungXSaj7LZXglx6kIy4MbNgpJBhCz0/gS+Lhw0tzZ31uHA5FqHxGZ4BxMALtSuCdh6OpkBltlLaelE4g9Guvk3gtljt4oYsynua0yIc0j3qsc43sIshaIXykEl4/1QkB7OxYg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Nk7ZaIvW; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Nk7ZaIvW" Received: by smtp.kernel.org (Postfix) with ESMTPSA id F0108C4AF14; Tue, 16 Apr 2024 14:10:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1713276640; bh=IGvHj5WwwNKH6tN7p6OYCT0ik3l5G0n3uBg9+Iju0zQ=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=Nk7ZaIvW3wKaesXCqX9NFQT36ES08dWXW9f3bax5XAa4Zu0LnydQURwt0zIUt/go8 9cjAwMwdEKvyP13GCO1CtobVPaFbI0z9nozZmAy6LXoWx6nVOlDaDsarPCWTTFeTyu zdFRA5whL3FNa6zN+qgH6PF3/uGJosd2OSjVVIk/2MuYwsARqYW+nvrCBPXJ6Rnldx pY4/U494N13oKmpK1QH/Lw3Kgsxd9GfOFsAXZFBLxbjEIRejAjPhR6In48+ufvljfw aJSTabbHXvCGdX4CuwNtlF1U9z5Uw+4xshvZoxAc5E7tqCvHo6//qFKzntOy97ADRT MRhu7anzxnw1g== From: Benjamin Tissoires Date: Tue, 16 Apr 2024 16:08:24 +0200 Subject: [PATCH bpf-next 11/18] bpf: wq: add bpf_wq_init Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240416-bpf_wq-v1-11-c9e66092f842@kernel.org> References: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> In-Reply-To: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Mykola Lysenko , Shuah Khan Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Benjamin Tissoires X-Mailer: b4 0.12.4 X-Developer-Signature: v=1; a=ed25519-sha256; t=1713276593; l=8318; i=bentiss@kernel.org; s=20230215; h=from:subject:message-id; bh=IGvHj5WwwNKH6tN7p6OYCT0ik3l5G0n3uBg9+Iju0zQ=; b=vrmWzXl/BHcupYg1U+1zybKB+kDMuktsFidLvF3MzIxWddXmYuUFxSQ9aPtAvoPyuCJW7/zXt LN2+dqmQOp1B377pw5vT5RfftJG9VpO+pTXOCmocBJGu9G1oQKITpnq X-Developer-Key: i=bentiss@kernel.org; a=ed25519; pk=7D1DyAVh6ajCkuUTudt/chMuXWIJHlv2qCsRkIizvFw= X-Patchwork-Delegate: bpf@iogearbox.net We need to teach the verifier about the second argument which is declared as void * but which is of type KF_ARG_PTR_TO_MAP. We could have dropped this extra case if we declared the second argument as struct bpf_map *, but that means users will have to do extra casting to have their program compile. We also need to duplicate the timer code for the checking if the map argument is matching the provided workqueue. Signed-off-by: Benjamin Tissoires --- Note that the timer code when matching for the map is checking for constant map pointers. I wonder if this needs to be enforced too (being constant?) --- include/uapi/linux/bpf.h | 9 ++++ kernel/bpf/helpers.c | 114 ++++++++++++++++++++++++++++++++++++++++++++++- kernel/bpf/verifier.c | 6 +++ 3 files changed, 127 insertions(+), 2 deletions(-) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index e4ae83550fb3..519f6019d158 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -7502,4 +7502,13 @@ struct bpf_iter_num { __u64 __opaque[1]; } __attribute__((aligned(8))); +/* + * Flags to control bpf_wq_init() and bpf_wq_set_callback() behaviour. + * - BPF_F_WQ_SLEEPABLE: the callback needs to run in + * a sleepable context + */ +enum { + BPF_F_WQ_SLEEPABLE = (1ULL << 0), +}; + #endif /* _UAPI__LINUX_BPF_H__ */ diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 9fd12d480b8b..9ac1b8bb3a01 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -1109,11 +1109,18 @@ struct bpf_hrtimer { struct hrtimer timer; }; -/* the actual struct hidden inside uapi struct bpf_timer */ +struct bpf_work { + struct bpf_async_cb cb; + struct work_struct work; + struct work_struct delete_work; +}; + +/* the actual struct hidden inside uapi struct bpf_timer and bpf_wq */ struct bpf_async_kern { union { struct bpf_async_cb *cb; struct bpf_hrtimer *timer; + struct bpf_work *work; }; /* bpf_spin_lock is used here instead of spinlock_t to make * sure that it always fits into space reserved by struct bpf_timer @@ -1124,6 +1131,7 @@ struct bpf_async_kern { enum bpf_async_type { BPF_ASYNC_TYPE_TIMER = 0, + BPF_ASYNC_TYPE_WQ, }; static DEFINE_PER_CPU(struct bpf_hrtimer *, hrtimer_running); @@ -1167,11 +1175,75 @@ static enum hrtimer_restart bpf_timer_cb(struct hrtimer *hrtimer) return HRTIMER_NORESTART; } +static void bpf_wq_work(struct work_struct *work) +{ + struct bpf_work *w = container_of(work, struct bpf_work, work); + struct bpf_tramp_run_ctx __maybe_unused run_ctx; + struct bpf_prog *prog = w->cb.prog; + unsigned int flags = w->cb.flags; + struct bpf_map *map = w->cb.map; + bpf_callback_t callback_fn; + void *value = w->cb.value; + void *key; + u32 idx; + + BTF_TYPE_EMIT(struct bpf_wq); + + callback_fn = READ_ONCE(w->cb.callback_fn); + if (!callback_fn || !prog) + return; + + if (map->map_type == BPF_MAP_TYPE_ARRAY) { + struct bpf_array *array = container_of(map, struct bpf_array, map); + + /* compute the key */ + idx = ((char *)value - array->value) / array->elem_size; + key = &idx; + } else { /* hash or lru */ + key = value - round_up(map->key_size, 8); + } + + run_ctx.bpf_cookie = 0; + + if (flags & BPF_F_WQ_SLEEPABLE) { + if (!__bpf_prog_enter_sleepable_recur(prog, &run_ctx)) { + /* recursion detected */ + __bpf_prog_exit_sleepable_recur(prog, 0, &run_ctx); + return; + } + } else { + if (!__bpf_prog_enter_recur(prog, &run_ctx)) { + /* recursion detected */ + __bpf_prog_exit_recur(prog, 0, &run_ctx); + return; + } + } + + callback_fn((u64)(long)map, (u64)(long)key, (u64)(long)value, 0, 0); + /* The verifier checked that return value is zero. */ + + if (flags & BPF_F_WQ_SLEEPABLE) + __bpf_prog_exit_sleepable_recur(prog, 0 /* bpf_prog_run does runtime stats */, + &run_ctx); + else + __bpf_prog_exit_recur(prog, 0, &run_ctx); +} + +static void bpf_wq_delete_work(struct work_struct *work) +{ + struct bpf_work *w = container_of(work, struct bpf_work, delete_work); + + cancel_work_sync(&w->work); + + kfree_rcu(w, cb.rcu); +} + static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u64 flags, enum bpf_async_type type) { struct bpf_async_cb *cb; struct bpf_hrtimer *t; + struct bpf_work *w; clockid_t clockid; size_t size; int ret = 0; @@ -1183,6 +1255,9 @@ static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u case BPF_ASYNC_TYPE_TIMER: size = sizeof(struct bpf_hrtimer); break; + case BPF_ASYNC_TYPE_WQ: + size = sizeof(struct bpf_work); + break; default: return -EINVAL; } @@ -1201,13 +1276,22 @@ static int __bpf_async_init(struct bpf_async_kern *async, struct bpf_map *map, u goto out; } - if (type == BPF_ASYNC_TYPE_TIMER) { + switch (type) { + case BPF_ASYNC_TYPE_TIMER: clockid = flags & (MAX_CLOCKS - 1); t = (struct bpf_hrtimer *)cb; hrtimer_init(&t->timer, clockid, HRTIMER_MODE_REL_SOFT); t->timer.function = bpf_timer_cb; cb->value = (void *)async - map->record->timer_off; + break; + case BPF_ASYNC_TYPE_WQ: + w = (struct bpf_work *)cb; + + INIT_WORK(&w->work, bpf_wq_work); + INIT_WORK(&w->delete_work, bpf_wq_delete_work); + cb->value = (void *)async - map->record->wq_off; + break; } cb->map = map; cb->prog = NULL; @@ -1473,7 +1557,19 @@ void bpf_timer_cancel_and_free(void *val) */ void bpf_wq_cancel_and_free(void *val) { + struct bpf_work *work; + BTF_TYPE_EMIT(struct bpf_wq); + + work = (struct bpf_work *)__bpf_async_cancel_and_free(val); + if (!work) + return; + /* Trigger cancel of the sleepable work, but *do not* wait for + * it to finish if it was running as we might not be in a + * sleepable context. + * kfree will be called once the work has finished. + */ + schedule_work(&work->delete_work); } BPF_CALL_2(bpf_kptr_xchg, void *, map_value, void *, ptr) @@ -2612,6 +2708,19 @@ __bpf_kfunc void bpf_throw(u64 cookie) WARN(1, "A call to BPF exception callback should never return\n"); } +__bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *map, unsigned int flags) +{ + struct bpf_async_kern *async = (struct bpf_async_kern *)wq; + + BUILD_BUG_ON(sizeof(struct bpf_async_kern) > sizeof(struct bpf_wq)); + BUILD_BUG_ON(__alignof__(struct bpf_async_kern) != __alignof__(struct bpf_wq)); + + if (flags & ~BPF_F_WQ_SLEEPABLE) + return -EINVAL; + + return __bpf_async_init(async, map, flags, BPF_ASYNC_TYPE_WQ); +} + __bpf_kfunc_end_defs(); BTF_KFUNCS_START(generic_btf_ids) @@ -2689,6 +2798,7 @@ BTF_ID_FLAGS(func, bpf_dynptr_is_rdonly) BTF_ID_FLAGS(func, bpf_dynptr_size) BTF_ID_FLAGS(func, bpf_dynptr_clone) BTF_ID_FLAGS(func, bpf_modify_return_test_tp) +BTF_ID_FLAGS(func, bpf_wq_init) BTF_KFUNCS_END(common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 112faf2cd7e9..5e8c1e65fe8c 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -11038,6 +11038,7 @@ enum special_kfunc_type { KF_bpf_percpu_obj_drop_impl, KF_bpf_throw, KF_bpf_iter_css_task_new, + KF_bpf_wq_init, }; BTF_SET_START(special_kfunc_set) @@ -11064,6 +11065,7 @@ BTF_ID(func, bpf_throw) #ifdef CONFIG_CGROUPS BTF_ID(func, bpf_iter_css_task_new) #endif +BTF_ID(func, bpf_wq_init) BTF_SET_END(special_kfunc_set) BTF_ID_LIST(special_kfunc_list) @@ -11094,6 +11096,7 @@ BTF_ID(func, bpf_iter_css_task_new) #else BTF_ID_UNUSED #endif +BTF_ID(func, bpf_wq_init) static bool is_kfunc_ret_null(struct bpf_kfunc_call_arg_meta *meta) { @@ -11171,6 +11174,9 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, if (is_kfunc_arg_wq(meta->btf, &args[argno])) return KF_ARG_PTR_TO_WORKQUEUE; + if (meta->func_id == special_kfunc_list[KF_bpf_wq_init] && argno == 1) + return KF_ARG_PTR_TO_MAP; + if ((base_type(reg->type) == PTR_TO_BTF_ID || reg2btf_ids[base_type(reg->type)])) { if (!btf_type_is_struct(ref_t)) { verbose(env, "kernel function %s args#%d pointer type %s %s is not supported\n", From patchwork Tue Apr 16 14:08:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Tissoires X-Patchwork-Id: 13631932 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 95878133293; Tue, 16 Apr 2024 14:10:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276644; cv=none; b=qUcH2c7wYmKoxuiSvHow68QZ3jqtLaG4v2tMmYMw/lAFloSkGSa59v30qLg+nRUSPrswrizIEEYGcPmkpqJT1Yhf+LerOYH5WsuBAClKbuXFPweE1Jiht+jnn1/hr5qjn2YxcgDEzlnE557bZZx/QV6+8fjngoEBc352T3M1F6M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276644; c=relaxed/simple; bh=Wbo+jTnF7UNG9GPzXTp/Kv/A9HSwxFmESxWeYRoaofM=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=hXJTF0YmmcyimB4HK4FlOPptVBE4wfpht+OREQH+0r5So0mmUH9m5JYFNM9Z0Dk3Qdn+bU3IbVAiWWa5jU9SEi/bolhCaps+hSM/KkQYNWNwUvsiMUzHQwXLpaZDHdciuzr/hm1SM4cY29xB0/o6cqUMK9rnDg26guzuk6/yLhg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=C07FtjL1; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="C07FtjL1" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BE3FDC113CE; Tue, 16 Apr 2024 14:10:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1713276644; bh=Wbo+jTnF7UNG9GPzXTp/Kv/A9HSwxFmESxWeYRoaofM=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=C07FtjL1fSZVEO4J7iwoDlEzaAIek9xPBjKZziJf3HOIa6TPM49qRlfrMThuLd4zW jXHXWRKT2hBvt2K0KBtPGW6dUY9MIwZLlfzRQaBI2AW6ggYBT3HVGiEcmL45Jy2OOV sXi/tHbRpE7QYFU5e0OfyQr8M4ajcg7R+FDR7u5/Zjzgi5yJ4Vd6NQc1nf3y7N/JJf WeD4hOiv2EYNEVf5kz1fYqg40QvsY/maNb+6a5V0AGdIIFxK1hXh8GJn86YEDxiHJ3 qImAg/4SLHI3a2dIpoDa5125BsEmEqMDBjaTqwSpUiQ1x449Kb40PITm4xzduU/EkA dFzRgfs45tmnQ== From: Benjamin Tissoires Date: Tue, 16 Apr 2024 16:08:25 +0200 Subject: [PATCH bpf-next 12/18] tools: sync include/uapi/linux/bpf.h Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240416-bpf_wq-v1-12-c9e66092f842@kernel.org> References: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> In-Reply-To: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Mykola Lysenko , Shuah Khan Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Benjamin Tissoires X-Mailer: b4 0.12.4 X-Developer-Signature: v=1; a=ed25519-sha256; t=1713276593; l=785; i=bentiss@kernel.org; s=20230215; h=from:subject:message-id; bh=Wbo+jTnF7UNG9GPzXTp/Kv/A9HSwxFmESxWeYRoaofM=; b=gaHcM6/jsovDNkSeKfzaVGacq0f8WUTta54Xs16cVrsjx6nIVtAtpaJnFrLxWEDDhJi2rpgAQ TYiDQSQO70FC0Cs8psJSomQWojnRHwNK5f3epGrf8X8CHZgfZyiq7gL X-Developer-Key: i=bentiss@kernel.org; a=ed25519; pk=7D1DyAVh6ajCkuUTudt/chMuXWIJHlv2qCsRkIizvFw= X-Patchwork-Delegate: bpf@iogearbox.net cp include/uapi/linux/bpf.h tools/include/uapi/linux/bpf.h Signed-off-by: Benjamin Tissoires --- tools/include/uapi/linux/bpf.h | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index e4ae83550fb3..519f6019d158 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -7502,4 +7502,13 @@ struct bpf_iter_num { __u64 __opaque[1]; } __attribute__((aligned(8))); +/* + * Flags to control bpf_wq_init() and bpf_wq_set_callback() behaviour. + * - BPF_F_WQ_SLEEPABLE: the callback needs to run in + * a sleepable context + */ +enum { + BPF_F_WQ_SLEEPABLE = (1ULL << 0), +}; + #endif /* _UAPI__LINUX_BPF_H__ */ From patchwork Tue Apr 16 14:08:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Tissoires X-Patchwork-Id: 13631933 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6665A12EBFA; Tue, 16 Apr 2024 14:10:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276648; cv=none; b=bhZ5v9pY4XMH+VuiyPEKjg9TrUeZ20oi19oE/Pmqt4E2uy3Nlhu/UhnWzpB8Rh6G4rPrzLcrT3Harojq+B4LokCBqjfa2uQp/ba5LjuEfKRI2VqgpXBY0HHhNL6k5r1HW9JwUzMO9htM6T61b+2bUipVahW/RKL3H+JmpYjtPgA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276648; c=relaxed/simple; bh=JjBJJq1sNWGaFqkELXJ+0AXMMVvsTz+E0m4FBFgIIH0=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=ApLq7WXllE2kP2cECPpx6ZF8NFLo2eUGQK502jhSzp5QcnI5PLgmgqgTYi4uYYkGFpZdRYD6BGo9naRSenpWf2R8MSNewoSOqfMxG5px3TPO+JU+o46LvDU40StGEndcJYFMF0eDpYw8FhuXHlwszM2To/FuyDfdajI3Lk0FK7I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=uumN6pLi; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="uumN6pLi" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8C769C2BD11; Tue, 16 Apr 2024 14:10:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1713276647; bh=JjBJJq1sNWGaFqkELXJ+0AXMMVvsTz+E0m4FBFgIIH0=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=uumN6pLifqygT+JXv9qiCgenWMa00T38oCf1DsF5Ubqy4vPAACJGo6NaanYnnjDa0 4rpYYg0HKZu4hQJzi0z682qay2duOtDADQWFHY1wgrmKHWqabMR9nffi1blrVMZ4Y0 g0YcDeBEYwZmewYHNPY0TOwEkkPAUfEBj7IHdNGI3GtoJWFKTis3UL5q1GMIcudz7W ukQL+UlFGDCngSmZuMoIM3gazKqQJc2NdCo9lX06d7Bi4rfnAV3LhfcOKDyQmdTquI 1/aBF/cSbiCuMxVjn5rr4rSo9Bxp7D3ITej75KdEgq3IW1kkd0TVhZ3OMDH8pu4jO8 RiSGupd6+MbLg== From: Benjamin Tissoires Date: Tue, 16 Apr 2024 16:08:26 +0200 Subject: [PATCH bpf-next 13/18] selftests/bpf: wq: add bpf_wq_init() checks Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240416-bpf_wq-v1-13-c9e66092f842@kernel.org> References: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> In-Reply-To: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Mykola Lysenko , Shuah Khan Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Benjamin Tissoires X-Mailer: b4 0.12.4 X-Developer-Signature: v=1; a=ed25519-sha256; t=1713276593; l=4860; i=bentiss@kernel.org; s=20230215; h=from:subject:message-id; bh=JjBJJq1sNWGaFqkELXJ+0AXMMVvsTz+E0m4FBFgIIH0=; b=WYoaxHeD3s7z9EobT9ZTV15qk6OxIM/4ku1EQbAUq4le7O/c5IAocLyIPEDwNkdukraKdOAIN 5ln/q4dtvSRDwZhrDdnyxuxy9HpsoRZjxWkQKffom9ULmBDOOBSoyZb X-Developer-Key: i=bentiss@kernel.org; a=ed25519; pk=7D1DyAVh6ajCkuUTudt/chMuXWIJHlv2qCsRkIizvFw= X-Patchwork-Delegate: bpf@iogearbox.net Allows to test if allocation/free works Signed-off-by: Benjamin Tissoires --- tools/testing/selftests/bpf/bpf_experimental.h | 1 + tools/testing/selftests/bpf/prog_tests/wq.c | 8 +++ tools/testing/selftests/bpf/progs/wq.c | 10 +++ tools/testing/selftests/bpf/progs/wq_failures.c | 82 +++++++++++++++++++++++++ 4 files changed, 101 insertions(+) diff --git a/tools/testing/selftests/bpf/bpf_experimental.h b/tools/testing/selftests/bpf/bpf_experimental.h index 3329ea080865..6af6f2a43e6d 100644 --- a/tools/testing/selftests/bpf/bpf_experimental.h +++ b/tools/testing/selftests/bpf/bpf_experimental.h @@ -470,4 +470,5 @@ extern int bpf_iter_css_new(struct bpf_iter_css *it, extern struct cgroup_subsys_state *bpf_iter_css_next(struct bpf_iter_css *it) __weak __ksym; extern void bpf_iter_css_destroy(struct bpf_iter_css *it) __weak __ksym; +extern int bpf_wq_init(struct bpf_wq *wq, void *map, unsigned int flags) __weak __ksym; #endif diff --git a/tools/testing/selftests/bpf/prog_tests/wq.c b/tools/testing/selftests/bpf/prog_tests/wq.c index 9a07b8bc2c52..26ab69796103 100644 --- a/tools/testing/selftests/bpf/prog_tests/wq.c +++ b/tools/testing/selftests/bpf/prog_tests/wq.c @@ -2,6 +2,7 @@ /* Copyright (c) 2024 Benjamin Tissoires */ #include #include "wq.skel.h" +#include "wq_failures.skel.h" void serial_test_wq(void) { @@ -9,3 +10,10 @@ void serial_test_wq(void) RUN_TESTS(wq); } + +void serial_test_failures_wq(void) +{ + LIBBPF_OPTS(bpf_test_run_opts, topts); + + RUN_TESTS(wq_failures); +} diff --git a/tools/testing/selftests/bpf/progs/wq.c b/tools/testing/selftests/bpf/progs/wq.c index 8c668ad04059..f92466eb8fb1 100644 --- a/tools/testing/selftests/bpf/progs/wq.c +++ b/tools/testing/selftests/bpf/progs/wq.c @@ -52,6 +52,7 @@ struct { static int test_elem_callback(void *map, int *key) { struct elem init = {}, *val; + struct bpf_wq *wq; if (map == &lru && bpf_map_update_elem(map, key, &init, 0)) @@ -61,12 +62,17 @@ static int test_elem_callback(void *map, int *key) if (!val) return -2; + wq = &val->w; + if (bpf_wq_init(wq, map, 0) != 0) + return -3; + return 0; } static int test_hmap_elem_callback(void *map, int *key) { struct hmap_elem init = {}, *val; + struct bpf_wq *wq; if (bpf_map_update_elem(map, key, &init, 0)) return -1; @@ -75,6 +81,10 @@ static int test_hmap_elem_callback(void *map, int *key) if (!val) return -2; + wq = &val->work; + if (bpf_wq_init(wq, map, 0) != 0) + return -3; + return 0; } diff --git a/tools/testing/selftests/bpf/progs/wq_failures.c b/tools/testing/selftests/bpf/progs/wq_failures.c new file mode 100644 index 000000000000..a08b6c93a195 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/wq_failures.c @@ -0,0 +1,82 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Benjamin Tissoires + */ + +#include "bpf_experimental.h" +#include +#include "bpf_misc.h" +#include "../bpf_testmod/bpf_testmod_kfunc.h" + +char _license[] SEC("license") = "GPL"; + +struct elem { + struct bpf_wq w; +}; + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 2); + __type(key, int); + __type(value, struct elem); +} array SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_LRU_HASH); + __uint(max_entries, 4); + __type(key, int); + __type(value, struct elem); +} lru SEC(".maps"); + +SEC("tc") +/* test that bpf_wq_init takes a map as a second argument + */ +__log_level(2) +__flag(BPF_F_TEST_STATE_FREQ) +__failure +/* check that code path marks r1 as precise */ +__msg("mark_precise: frame0: regs=r1 stack= before") +__msg(": (85) call bpf_wq_init#") /* anchor message */ +__msg("workqueue pointer in R1 map_uid=0 doesn't match map pointer in R2 map_uid=0") +long test_wq_init_nomap(void *ctx) +{ + struct bpf_wq *wq; + struct elem *val; + int key = 0; + + val = bpf_map_lookup_elem(&array, &key); + if (!val) + return -1; + + wq = &val->w; + if (bpf_wq_init(wq, &key, 0) != 0) + return -3; + + return 0; +} + +SEC("tc") +/* test that the workqueue is part of the map in bpf_wq_init + */ +__log_level(2) +__flag(BPF_F_TEST_STATE_FREQ) +__failure +/* check that code path marks r1 as precise */ +__msg("mark_precise: frame0: regs=r1 stack= before") +__msg(": (85) call bpf_wq_init#") /* anchor message */ +__msg("workqueue pointer in R1 map_uid=0 doesn't match map pointer in R2 map_uid=0") +long test_wq_init_wrong_map(void *ctx) +{ + struct bpf_wq *wq; + struct elem *val; + int key = 0; + + val = bpf_map_lookup_elem(&array, &key); + if (!val) + return -1; + + wq = &val->w; + if (bpf_wq_init(wq, &lru, 0) != 0) + return -3; + + return 0; +} From patchwork Tue Apr 16 14:08:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Tissoires X-Patchwork-Id: 13631934 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6CCE1339A5; Tue, 16 Apr 2024 14:10:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276651; cv=none; b=HH3hnifyg4FysqcPZor2JDP+56c+ZHhXbo1qJfA8S19kmr00DSA9OyiamF0DL6KN2nlQnLtcVVTDdLvw4WYOi7zfIjPULWJbbntf1stYjwua7pnwHkQKEXRV6QDeJoy1J7jgzsTdFsz5DKas93WZGgtdQOzJ3efviPtQeGum4eA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276651; c=relaxed/simple; bh=PseaAsRe0UQgLtMjT0SGl7UnaF8PrYknBxggC5vUVrU=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=CoEuf++WZJl0RYHxx4hfdjVdC4auFXmG0A4G+tzVLeM/uBvc0M5Oima8G78b+Qyc56ht8zdCvW8kha9E24bFekeogAlsetAVCjl7pmuiRx0URKxJixQCVD92BSxvyBdcG2N803whx4FWAYJJQA1sizmbOO+KpUtLkfEHnY2uguE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=jNT5AKZX; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="jNT5AKZX" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5ADDAC113CE; Tue, 16 Apr 2024 14:10:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1713276651; bh=PseaAsRe0UQgLtMjT0SGl7UnaF8PrYknBxggC5vUVrU=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=jNT5AKZXe9raa0hcklIyPMICum30AslIk4X57nihBFUwLghvPU7vA4DkQrP/whpwV N5tbq8p4D+em6zZ6mObXV1Zha+oWEnRZCrphNXZJMHRL7qkxn12WhI4o1NU0Ejagbx bquZbvg26nqsf7X+tbQAqbX9olZBjxQHdX/ElF4HParMFiNhlOjVW9UpK48gYyd8D2 nOkzwrIc/mwmalCu9gIFPSZSIvpw8lvBgWfprZoG3Pr5Usni7RNPXgjieeVo9cCc6+ 57qgYUZTapOuByoNnDzQBv+mOxgXaE3UFkTHDsWMLSEUaSK91+lLMAUxGTh/Xe1ZR6 1QL3P2vceTzaQ== From: Benjamin Tissoires Date: Tue, 16 Apr 2024 16:08:27 +0200 Subject: [PATCH bpf-next 14/18] bpf/verifier: add is_sleepable argument to push_callback_call Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240416-bpf_wq-v1-14-c9e66092f842@kernel.org> References: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> In-Reply-To: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Mykola Lysenko , Shuah Khan Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Benjamin Tissoires X-Mailer: b4 0.12.4 X-Developer-Signature: v=1; a=ed25519-sha256; t=1713276593; l=5920; i=bentiss@kernel.org; s=20230215; h=from:subject:message-id; bh=PseaAsRe0UQgLtMjT0SGl7UnaF8PrYknBxggC5vUVrU=; b=SoWxMt1mk464DpWnaOlnqSg4DfOq8FtKLR75n5jkaMxB72wkkzSi+uM6qqeLgiXPxyr6yfu2h jkVAzdCBgZTBoY+bCNT3sfrXCgzY6RAqyPGDiDTNozgRwOHtTj6yCzw X-Developer-Key: i=bentiss@kernel.org; a=ed25519; pk=7D1DyAVh6ajCkuUTudt/chMuXWIJHlv2qCsRkIizvFw= X-Patchwork-Delegate: bpf@iogearbox.net To support sleepable async callbacks, we need to tell push_callback_call() whether the cb is sleepable or not. Doing so while checking for the kfunc arguments (when we call push_callback_call()) is simpler than adding a check for a function while inside push_callback_call(). When a callback is tagged as sleepable, the verifier now knows that it is the case and can allow a sleepable callback to happen. Signed-off-by: Benjamin Tissoires --- include/linux/bpf_verifier.h | 1 + kernel/bpf/verifier.c | 26 ++++++++++++++++---------- 2 files changed, 17 insertions(+), 10 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 36d19cd32eb5..9db35530c878 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -426,6 +426,7 @@ struct bpf_verifier_state { * while they are still in use. */ bool used_as_loop_entry; + bool in_sleepable; /* first and last insn idx of this verifier state */ u32 first_insn_idx; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 5e8c1e65fe8c..6a45d88244c6 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -1429,6 +1429,7 @@ static int copy_verifier_state(struct bpf_verifier_state *dst_state, } dst_state->speculative = src->speculative; dst_state->active_rcu_lock = src->active_rcu_lock; + dst_state->in_sleepable = src->in_sleepable; dst_state->curframe = src->curframe; dst_state->active_lock.ptr = src->active_lock.ptr; dst_state->active_lock.id = src->active_lock.id; @@ -2404,7 +2405,7 @@ static void init_func_state(struct bpf_verifier_env *env, /* Similar to push_stack(), but for async callbacks */ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env, int insn_idx, int prev_insn_idx, - int subprog) + int subprog, bool is_sleepable) { struct bpf_verifier_stack_elem *elem; struct bpf_func_state *frame; @@ -2431,6 +2432,7 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env, * Initialize it similar to do_check_common(). */ elem->st.branches = 1; + elem->st.in_sleepable = is_sleepable; frame = kzalloc(sizeof(*frame), GFP_KERNEL); if (!frame) goto err; @@ -5278,7 +5280,8 @@ static int map_kptr_match_type(struct bpf_verifier_env *env, static bool in_sleepable(struct bpf_verifier_env *env) { - return env->prog->sleepable; + return env->prog->sleepable || + (env->cur_state && env->cur_state->in_sleepable); } /* The non-sleepable programs and sleepable programs with explicit bpf_rcu_read_lock() @@ -9515,7 +9518,7 @@ static int btf_check_subprog_call(struct bpf_verifier_env *env, int subprog, } static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *insn, - int insn_idx, int subprog, + int insn_idx, int subprog, bool is_sleepable, set_callee_state_fn set_callee_state_cb) { struct bpf_verifier_state *state = env->cur_state, *callback_state; @@ -9550,7 +9553,7 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins /* there is no real recursion here. timer callbacks are async */ env->subprog_info[subprog].is_async_cb = true; async_cb = push_async_cb(env, env->subprog_info[subprog].start, - insn_idx, subprog); + insn_idx, subprog, is_sleepable); if (!async_cb) return -EFAULT; callee = async_cb->frame[0]; @@ -10389,15 +10392,15 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn } break; case BPF_FUNC_for_each_map_elem: - err = push_callback_call(env, insn, insn_idx, meta.subprogno, + err = push_callback_call(env, insn, insn_idx, meta.subprogno, false, set_map_elem_callback_state); break; case BPF_FUNC_timer_set_callback: - err = push_callback_call(env, insn, insn_idx, meta.subprogno, + err = push_callback_call(env, insn, insn_idx, meta.subprogno, false, set_timer_callback_state); break; case BPF_FUNC_find_vma: - err = push_callback_call(env, insn, insn_idx, meta.subprogno, + err = push_callback_call(env, insn, insn_idx, meta.subprogno, false, set_find_vma_callback_state); break; case BPF_FUNC_snprintf: @@ -10412,7 +10415,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn if (err) return err; if (cur_func(env)->callback_depth < regs[BPF_REG_1].umax_value) { - err = push_callback_call(env, insn, insn_idx, meta.subprogno, + err = push_callback_call(env, insn, insn_idx, meta.subprogno, false, set_loop_callback_state); } else { cur_func(env)->callback_depth = 0; @@ -10515,7 +10518,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn break; } case BPF_FUNC_user_ringbuf_drain: - err = push_callback_call(env, insn, insn_idx, meta.subprogno, + err = push_callback_call(env, insn, insn_idx, meta.subprogno, false, set_user_ringbuf_callback_state); break; } @@ -12232,7 +12235,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, return err; if (meta.func_id == special_kfunc_list[KF_bpf_rbtree_add_impl]) { - err = push_callback_call(env, insn, insn_idx, meta.subprogno, + err = push_callback_call(env, insn, insn_idx, meta.subprogno, false, set_rbtree_add_callback_state); if (err) { verbose(env, "kfunc %s#%d failed callback verification\n", @@ -17004,6 +17007,9 @@ static bool states_equal(struct bpf_verifier_env *env, if (old->active_rcu_lock != cur->active_rcu_lock) return false; + if (old->in_sleepable != cur->in_sleepable) + return false; + /* for states to be equal callsites have to be the same * and all frame states need to be equivalent */ From patchwork Tue Apr 16 14:08:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Tissoires X-Patchwork-Id: 13631935 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A0DCB1353F9; Tue, 16 Apr 2024 14:10:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276655; cv=none; b=L1m/bW+fzSvMfOhBkmIQx//BkCaea80LPWZBmv5Lz5s6SDYkqOiW+36M/oE6aOK/xfuwGwKKnwf39Vx0c29gZpB7Uk05Sf649ISAmARYHwK99umcBwq2wTNMlOFtJT9mDeGyknZWdAKo7nhLbbhxnydC8g2dUwY2ogpx1XyN9eI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276655; c=relaxed/simple; bh=PPXK46VhuHV4LHvO9/F2U+mh0gqt3BI4LddQFjAfCcI=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=OGPpgsDibmWyKBnu6Cf90dMsqcG8ebdoUhyctaHUfm8eyZ33vOeScYGmb/NlHpjEPwohZE3boVdacoGtcCedhJCpmqREkbeEbnJGJ9fUu4EQ4xksdo6C1BMsV3WRMcSkrGQuHvU04ig9oXoh1FJ2sain8Yfs2ZGy/y04PJ3J5HA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=PYfPuUOi; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="PYfPuUOi" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2B0F2C4AF4D; Tue, 16 Apr 2024 14:10:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1713276655; bh=PPXK46VhuHV4LHvO9/F2U+mh0gqt3BI4LddQFjAfCcI=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=PYfPuUOiPYH+zBB1jAahr4hRLjQ2z792X+k9Z1/8kfEwqoooWDCW/2Gs0EVJQH2K+ O5bFY9lBhiSQKwbdCkBOh7SePT4sKACP/gJKifr6/YlU3rGe8V00LFnw+s79Uanyxg +5QpAHkHzs5kVpLKyI5E2tMDwjpbeT+qssPW4KH6d7L23888I9jZR7OHZSqqUaXHNZ cUlPmqroYZJwdR54vSQzgaXUyF4er4/rdbuZzop+m+h/TAQCpyJsGdOxEb5j8gbJFa OdEm8kR9v5pWG2IGlMi97QDPjw/NAEpVV0BtMyZvlxOntRCLSq4uzixikaHy8tV5VC OAW3YC5bnsdnA== From: Benjamin Tissoires Date: Tue, 16 Apr 2024 16:08:28 +0200 Subject: [PATCH bpf-next 15/18] bpf: wq: add bpf_wq_set_callback_impl Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240416-bpf_wq-v1-15-c9e66092f842@kernel.org> References: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> In-Reply-To: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Mykola Lysenko , Shuah Khan Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Benjamin Tissoires X-Mailer: b4 0.12.4 X-Developer-Signature: v=1; a=ed25519-sha256; t=1713276593; l=7703; i=bentiss@kernel.org; s=20230215; h=from:subject:message-id; bh=PPXK46VhuHV4LHvO9/F2U+mh0gqt3BI4LddQFjAfCcI=; b=RSyFLk/zlbBQyIZ8BLnNNTu7HjvoidQ1Ic4eaTp/8ry4ZigR7hwKkRmYcGeNKAO4wZSx0mO4m yvpMFsvDgWmCDxCQ7HjFjP6ksfusw6wwj219GTvDpgQl9wCLJBGqWx4 X-Developer-Key: i=bentiss@kernel.org; a=ed25519; pk=7D1DyAVh6ajCkuUTudt/chMuXWIJHlv2qCsRkIizvFw= X-Patchwork-Delegate: bpf@iogearbox.net In the verifier, we need to teach it that the mentioned callback is is an async call. We look at the provided flags and set the callback to be sleepable if BPF_F_WQ_SLEEPABLE is set. Signed-off-by: Benjamin Tissoires --- kernel/bpf/helpers.c | 22 +++++++++++++++++ kernel/bpf/verifier.c | 68 ++++++++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 87 insertions(+), 3 deletions(-) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 9ac1b8bb3a01..e5c8adc44619 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -1362,6 +1362,13 @@ static int __bpf_async_set_callback(struct bpf_async_kern *async, void *callback ret = -EINVAL; goto out; } + if (type == BPF_ASYNC_TYPE_WQ) { + /* check that flags_k is consistent with work->flags */ + if ((flags ^ cb->flags) & BPF_F_WQ_SLEEPABLE) { + ret = -EINVAL; + goto out; + } + } if (!atomic64_read(&cb->map->usercnt)) { /* maps with timers must be either held by user space * or pinned in bpffs. Otherwise timer might still be @@ -2721,6 +2728,20 @@ __bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *map, unsigned int flags) return __bpf_async_init(async, map, flags, BPF_ASYNC_TYPE_WQ); } +__bpf_kfunc int bpf_wq_set_callback_impl(struct bpf_wq *wq, + int (callback_fn)(void *map, int *key, struct bpf_wq *wq), + unsigned int flags__k, + void *aux__ign) +{ + struct bpf_prog_aux *aux = (struct bpf_prog_aux *)aux__ign; + struct bpf_async_kern *async = (struct bpf_async_kern *)wq; + + if (flags__k & ~BPF_F_WQ_SLEEPABLE) + return -EINVAL; + + return __bpf_async_set_callback(async, callback_fn, aux, flags__k, BPF_ASYNC_TYPE_WQ); +} + __bpf_kfunc_end_defs(); BTF_KFUNCS_START(generic_btf_ids) @@ -2799,6 +2820,7 @@ BTF_ID_FLAGS(func, bpf_dynptr_size) BTF_ID_FLAGS(func, bpf_dynptr_clone) BTF_ID_FLAGS(func, bpf_modify_return_test_tp) BTF_ID_FLAGS(func, bpf_wq_init) +BTF_ID_FLAGS(func, bpf_wq_set_callback_impl) BTF_KFUNCS_END(common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 6a45d88244c6..947cd544ffab 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -501,8 +501,12 @@ static bool is_dynptr_ref_function(enum bpf_func_id func_id) } static bool is_sync_callback_calling_kfunc(u32 btf_id); +static bool is_async_callback_calling_kfunc(u32 btf_id); +static bool is_callback_calling_kfunc(u32 btf_id); static bool is_bpf_throw_kfunc(struct bpf_insn *insn); +static bool is_bpf_wq_set_callback_impl_kfunc(u32 btf_id); + static bool is_sync_callback_calling_function(enum bpf_func_id func_id) { return func_id == BPF_FUNC_for_each_map_elem || @@ -530,7 +534,8 @@ static bool is_sync_callback_calling_insn(struct bpf_insn *insn) static bool is_async_callback_calling_insn(struct bpf_insn *insn) { - return bpf_helper_call(insn) && is_async_callback_calling_function(insn->imm); + return (bpf_helper_call(insn) && is_async_callback_calling_function(insn->imm)) || + (bpf_pseudo_kfunc_call(insn) && is_async_callback_calling_kfunc(insn->imm)); } static bool is_may_goto_insn(struct bpf_insn *insn) @@ -9536,7 +9541,7 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins */ env->subprog_info[subprog].is_cb = true; if (bpf_pseudo_kfunc_call(insn) && - !is_sync_callback_calling_kfunc(insn->imm)) { + !is_callback_calling_kfunc(insn->imm)) { verbose(env, "verifier bug: kfunc %s#%d not marked as callback-calling\n", func_id_name(insn->imm), insn->imm); return -EFAULT; @@ -9550,7 +9555,7 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins if (is_async_callback_calling_insn(insn)) { struct bpf_verifier_state *async_cb; - /* there is no real recursion here. timer callbacks are async */ + /* there is no real recursion here. timer and workqueue callbacks are async */ env->subprog_info[subprog].is_async_cb = true; async_cb = push_async_cb(env, env->subprog_info[subprog].start, insn_idx, subprog, is_sleepable); @@ -11042,6 +11047,7 @@ enum special_kfunc_type { KF_bpf_throw, KF_bpf_iter_css_task_new, KF_bpf_wq_init, + KF_bpf_wq_set_callback_impl, }; BTF_SET_START(special_kfunc_set) @@ -11069,6 +11075,7 @@ BTF_ID(func, bpf_throw) BTF_ID(func, bpf_iter_css_task_new) #endif BTF_ID(func, bpf_wq_init) +BTF_ID(func, bpf_wq_set_callback_impl) BTF_SET_END(special_kfunc_set) BTF_ID_LIST(special_kfunc_list) @@ -11100,6 +11107,7 @@ BTF_ID(func, bpf_iter_css_task_new) BTF_ID_UNUSED #endif BTF_ID(func, bpf_wq_init) +BTF_ID(func, bpf_wq_set_callback_impl) static bool is_kfunc_ret_null(struct bpf_kfunc_call_arg_meta *meta) { @@ -11431,12 +11439,28 @@ static bool is_sync_callback_calling_kfunc(u32 btf_id) return btf_id == special_kfunc_list[KF_bpf_rbtree_add_impl]; } +static bool is_async_callback_calling_kfunc(u32 btf_id) +{ + return btf_id == special_kfunc_list[KF_bpf_wq_set_callback_impl]; +} + static bool is_bpf_throw_kfunc(struct bpf_insn *insn) { return bpf_pseudo_kfunc_call(insn) && insn->off == 0 && insn->imm == special_kfunc_list[KF_bpf_throw]; } +static bool is_bpf_wq_set_callback_impl_kfunc(u32 btf_id) +{ + return btf_id == special_kfunc_list[KF_bpf_wq_set_callback_impl]; +} + +static bool is_callback_calling_kfunc(u32 btf_id) +{ + return is_sync_callback_calling_kfunc(btf_id) || + is_async_callback_calling_kfunc(btf_id); +} + static bool is_rbtree_lock_required_kfunc(u32 btf_id) { return is_bpf_rbtree_api_kfunc(btf_id); @@ -12244,6 +12268,22 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, } } + if (is_bpf_wq_set_callback_impl_kfunc(meta.func_id)) { + if (!meta.arg_constant.found) { + verbose(env, "kfunc %s#%d failed flags verification\n", + func_name, meta.func_id); + return -EINVAL; + } + err = push_callback_call(env, insn, insn_idx, meta.subprogno, + !!(meta.arg_constant.value & BPF_F_WQ_SLEEPABLE), + set_timer_callback_state); + if (err) { + verbose(env, "kfunc %s#%d failed callback verification\n", + func_name, meta.func_id); + return err; + } + } + rcu_lock = is_kfunc_bpf_rcu_read_lock(&meta); rcu_unlock = is_kfunc_bpf_rcu_read_unlock(&meta); @@ -19681,6 +19721,28 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, desc->func_id == special_kfunc_list[KF_bpf_rdonly_cast]) { insn_buf[0] = BPF_MOV64_REG(BPF_REG_0, BPF_REG_1); *cnt = 1; + } else if (is_bpf_wq_set_callback_impl_kfunc(desc->func_id)) { + /* The verifier will process callback_fn as many times as necessary + * with different maps and the register states prepared by + * set_timer_callback_state will be accurate. + * + * The following use case is valid: + * map1 is shared by prog1, prog2, prog3. + * prog1 calls bpf_timer_init for some map1 elements + * prog2 calls bpf_timer_set_callback for some map1 elements. + * Those that were not bpf_timer_init-ed will return -EINVAL. + * prog3 calls bpf_timer_start for some map1 elements. + * Those that were not both bpf_timer_init-ed and + * bpf_timer_set_callback-ed will return -EINVAL. + */ + struct bpf_insn ld_addrs[2] = { + BPF_LD_IMM64(BPF_REG_4, (long)env->prog->aux), + }; + + insn_buf[0] = ld_addrs[0]; + insn_buf[1] = ld_addrs[1]; + insn_buf[2] = *insn; + *cnt = 3; } return 0; } From patchwork Tue Apr 16 14:08:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Tissoires X-Patchwork-Id: 13631936 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E302136679; Tue, 16 Apr 2024 14:10:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276659; cv=none; b=O0cZHtp4a5KEQrAD7M87eyLrlPHsETqSr3Z7eZfXaXKPOGl17eM48ICr0Tb/lymRv5LKhDvVBD+3HejClYc9lK5QnJMPYSfu441otzgTNFezoZ0kIpjSxCy8J860rV2HVxgJJPXBm3FWKtsm4oToyFQ6p7+XV6PgeESiIgu9XIg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276659; c=relaxed/simple; bh=lm2FY30vt1roT/QaHJVcBSUumMbZDrssuz719s9NomA=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=FNlBFVIeJwpKs8TALBDRaE1k8hqTq1fCldb3kBogBl1gvBRuWieLKERGYobfcJejOoeIXWw03ax94Ny+BHbxllKVepCpV70rtkW793l66ocLze/EZE7oE3ECW9ti4EAOl80y2XFySIpP5/hWtXAakbCDK/zxTONzVhroxJ7Bj2A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=YhqJ333w; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="YhqJ333w" Received: by smtp.kernel.org (Postfix) with ESMTPSA id EE162C4AF12; Tue, 16 Apr 2024 14:10:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1713276659; bh=lm2FY30vt1roT/QaHJVcBSUumMbZDrssuz719s9NomA=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=YhqJ333wGn6nuZFHU/tBSelQsDeTtLfq3RQC1eyOeo3BMOR0QGAC3Yx3RuTfUF7MG EMeIO5M1+NVfUmSsgJxkntwgv4HyYXRZDCNipM484DuYubMTjuHr0qtG6HBJt+3o4T zKDo+7WeoZe0pb386rRvmfotzB1GskiiFHWYitVxn/Tv3ShmXRtJxHCxpdwMuNOrUg raGglP+EF0kolAO/lFpfpjE3hZCYh4gIrZkYxWXWfj6a0zo0m+DcGoWEqLGiK22Z7+ dj0hYaaV4idWzlWUXqutU9GWyQutbJNW8t9KT9pzkG/nfOPP37JoxTlhGshiSw6zJo gmpJbtApgWkhw== From: Benjamin Tissoires Date: Tue, 16 Apr 2024 16:08:29 +0200 Subject: [PATCH bpf-next 16/18] selftests/bpf: add checks for bpf_wq_set_callback() Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240416-bpf_wq-v1-16-c9e66092f842@kernel.org> References: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> In-Reply-To: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Mykola Lysenko , Shuah Khan Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Benjamin Tissoires X-Mailer: b4 0.12.4 X-Developer-Signature: v=1; a=ed25519-sha256; t=1713276593; l=9344; i=bentiss@kernel.org; s=20230215; h=from:subject:message-id; bh=lm2FY30vt1roT/QaHJVcBSUumMbZDrssuz719s9NomA=; b=/XoRuGyVUIfvPUGrvgIEt3q9x4Yth7kpYX8XtHKjbLPbv3uYuPrLvbiMLfBQYTsnyO56dmjDo eqdo3l7tOW8AFjqXyx7B7SPcz5H9M97IEkbrmWcDXXay0SQWqo3kyU4 X-Developer-Key: i=bentiss@kernel.org; a=ed25519; pk=7D1DyAVh6ajCkuUTudt/chMuXWIJHlv2qCsRkIizvFw= X-Patchwork-Delegate: bpf@iogearbox.net We assign the callback and set everything up. The actual tests of these callbacks will be done when bpf_wq_start() is available. Signed-off-by: Benjamin Tissoires --- tools/testing/selftests/bpf/bpf_experimental.h | 5 + .../selftests/bpf/bpf_testmod/bpf_testmod.c | 5 + .../selftests/bpf/bpf_testmod/bpf_testmod_kfunc.h | 1 + tools/testing/selftests/bpf/progs/wq.c | 47 +++++++-- tools/testing/selftests/bpf/progs/wq_failures.c | 115 +++++++++++++++++++++ 5 files changed, 165 insertions(+), 8 deletions(-) diff --git a/tools/testing/selftests/bpf/bpf_experimental.h b/tools/testing/selftests/bpf/bpf_experimental.h index 6af6f2a43e6d..272604f9c4a5 100644 --- a/tools/testing/selftests/bpf/bpf_experimental.h +++ b/tools/testing/selftests/bpf/bpf_experimental.h @@ -471,4 +471,9 @@ extern struct cgroup_subsys_state *bpf_iter_css_next(struct bpf_iter_css *it) __ extern void bpf_iter_css_destroy(struct bpf_iter_css *it) __weak __ksym; extern int bpf_wq_init(struct bpf_wq *wq, void *map, unsigned int flags) __weak __ksym; +extern int bpf_wq_set_callback_impl(struct bpf_wq *wq, + int (callback_fn)(void *map, int *key, struct bpf_wq *wq), + unsigned int flags__k, void *aux__ign) __ksym; +#define bpf_wq_set_callback(timer, cb, flags) \ + bpf_wq_set_callback_impl(timer, cb, flags, NULL) #endif diff --git a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c index 39ad96a18123..eb2b78552ca2 100644 --- a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +++ b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c @@ -494,6 +494,10 @@ __bpf_kfunc static u32 bpf_kfunc_call_test_static_unused_arg(u32 arg, u32 unused return arg; } +__bpf_kfunc void bpf_kfunc_call_test_sleepable(void) +{ +} + BTF_KFUNCS_START(bpf_testmod_check_kfunc_ids) BTF_ID_FLAGS(func, bpf_testmod_test_mod_kfunc) BTF_ID_FLAGS(func, bpf_kfunc_call_test1) @@ -520,6 +524,7 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test_ref, KF_TRUSTED_ARGS | KF_RCU) BTF_ID_FLAGS(func, bpf_kfunc_call_test_destructive, KF_DESTRUCTIVE) BTF_ID_FLAGS(func, bpf_kfunc_call_test_static_unused_arg) BTF_ID_FLAGS(func, bpf_kfunc_call_test_offset) +BTF_ID_FLAGS(func, bpf_kfunc_call_test_sleepable, KF_SLEEPABLE) BTF_KFUNCS_END(bpf_testmod_check_kfunc_ids) static int bpf_testmod_ops_init(struct btf *btf) diff --git a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod_kfunc.h index 7c664dd61059..ce5cd763561c 100644 --- a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod_kfunc.h +++ b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod_kfunc.h @@ -96,6 +96,7 @@ void bpf_kfunc_call_test_pass2(struct prog_test_pass2 *p) __ksym; void bpf_kfunc_call_test_mem_len_fail2(__u64 *mem, int len) __ksym; void bpf_kfunc_call_test_destructive(void) __ksym; +void bpf_kfunc_call_test_sleepable(void) __ksym; void bpf_kfunc_call_test_offset(struct prog_test_ref_kfunc *p); struct prog_test_member *bpf_kfunc_call_memb_acquire(void); diff --git a/tools/testing/selftests/bpf/progs/wq.c b/tools/testing/selftests/bpf/progs/wq.c index f92466eb8fb1..c0a094c84834 100644 --- a/tools/testing/selftests/bpf/progs/wq.c +++ b/tools/testing/selftests/bpf/progs/wq.c @@ -49,7 +49,9 @@ struct { __type(value, struct elem); } lru SEC(".maps"); -static int test_elem_callback(void *map, int *key) +static int test_elem_callback(void *map, int *key, + int (callback_fn)(void *map, int *key, struct bpf_wq *wq), + u64 callback_flags) { struct elem init = {}, *val; struct bpf_wq *wq; @@ -63,13 +65,18 @@ static int test_elem_callback(void *map, int *key) return -2; wq = &val->w; - if (bpf_wq_init(wq, map, 0) != 0) + if (bpf_wq_init(wq, map, callback_flags) != 0) return -3; + if (bpf_wq_set_callback(wq, callback_fn, callback_flags)) + return -4; + return 0; } -static int test_hmap_elem_callback(void *map, int *key) +static int test_hmap_elem_callback(void *map, int *key, + int (callback_fn)(void *map, int *key, struct bpf_wq *wq), + u64 callback_flags) { struct hmap_elem init = {}, *val; struct bpf_wq *wq; @@ -85,6 +92,28 @@ static int test_hmap_elem_callback(void *map, int *key) if (bpf_wq_init(wq, map, 0) != 0) return -3; + if (bpf_wq_set_callback(wq, callback_fn, callback_flags)) + return -4; + + return 0; +} + +__u32 ok; +__u32 ok_sleepable; + +/* callback for non sleepable workqueue */ +static int wq_callback(void *map, int *key, struct bpf_wq *work) +{ + bpf_kfunc_common_test(); + ok |= (1 << *key); + return 0; +} + +/* callback for sleepable workqueue */ +static int wq_cb_sleepable(void *map, int *key, struct bpf_wq *work) +{ + bpf_kfunc_call_test_sleepable(); + ok_sleepable |= (1 << *key); return 0; } @@ -96,7 +125,8 @@ long test_call_array_sleepable(void *ctx) { int key = 0; - return test_elem_callback(&array, &key); + return test_elem_callback(&array, &key, wq_cb_sleepable, + BPF_F_WQ_SLEEPABLE); } SEC("syscall") @@ -107,7 +137,8 @@ long test_syscall_array_sleepable(void *ctx) { int key = 1; - return test_elem_callback(&array, &key); + return test_elem_callback(&array, &key, wq_cb_sleepable, + BPF_F_WQ_SLEEPABLE); } SEC("tc") @@ -118,7 +149,7 @@ long test_call_hash_sleepable(void *ctx) { int key = 2; - return test_hmap_elem_callback(&hmap, &key); + return test_hmap_elem_callback(&hmap, &key, wq_callback, 0); } SEC("tc") @@ -130,7 +161,7 @@ long test_call_hash_malloc_sleepable(void *ctx) { int key = 3; - return test_hmap_elem_callback(&hmap_malloc, &key); + return test_hmap_elem_callback(&hmap_malloc, &key, wq_callback, 0); } SEC("tc") @@ -141,5 +172,5 @@ long test_call_lru_sleepable(void *ctx) { int key = 4; - return test_elem_callback(&lru, &key); + return test_elem_callback(&lru, &key, wq_callback, 0); } diff --git a/tools/testing/selftests/bpf/progs/wq_failures.c b/tools/testing/selftests/bpf/progs/wq_failures.c index a08b6c93a195..efde14516358 100644 --- a/tools/testing/selftests/bpf/progs/wq_failures.c +++ b/tools/testing/selftests/bpf/progs/wq_failures.c @@ -27,6 +27,20 @@ struct { __type(value, struct elem); } lru SEC(".maps"); +/* callback for non sleepable workqueue */ +static int wq_callback(void *map, int *key, struct bpf_wq *work) +{ + bpf_kfunc_common_test(); + return 0; +} + +/* callback for sleepable workqueue */ +static int wq_cb_sleepable(void *map, int *key, struct bpf_wq *work) +{ + bpf_kfunc_call_test_sleepable(); + return 0; +} + SEC("tc") /* test that bpf_wq_init takes a map as a second argument */ @@ -80,3 +94,104 @@ long test_wq_init_wrong_map(void *ctx) return 0; } + +SEC("?tc") +__log_level(2) +__failure +/* check that bpf_wq_set_callback() can not be called with a + * sleepable callback without BPF_F_WQ_SLEEPABLE + */ +__msg("mark_precise: frame0: regs=r1 stack= before") +__msg(": (85) call bpf_kfunc_call_test_sleepable#") /* anchor message */ +__msg("program must be sleepable to call sleepable kfunc bpf_kfunc_call_test_sleepable") +long test_call_sleepable_callback_missing_flag(void *ctx) +{ + int key = 0; + struct bpf_wq *wq; + + wq = bpf_map_lookup_elem(&array, &key); + if (wq) { + bpf_wq_init(wq, &array, 0); + bpf_wq_set_callback(wq, wq_cb_sleepable, 0); + } + + return 0; +} + +SEC("tc") +/* check that calling a non sleepable callback with + * bpf_wq_set_callback() while providing BPF_F_WQ_SLEEPABLE + * fails. + */ +__retval(3) +long test_call_non_sleepable_callback_wrong_flag(void *ctx) +{ + int key = 1; + struct bpf_wq *wq; + + wq = bpf_map_lookup_elem(&array, &key); + if (!wq) + return 1; + + if (bpf_wq_init(wq, &array, 0)) + return 2; + + if (bpf_wq_set_callback(wq, wq_callback, BPF_F_WQ_SLEEPABLE)) + return 3; + + return -22; +} + +SEC("?tc") +__log_level(2) +__failure +/* check that the first argument of bpf_wq_set_callback() + * is a correct bpf_wq pointer. + */ +__msg("mark_precise: frame0: regs=r1 stack= before") +__msg(": (85) call bpf_wq_set_callback_impl#") /* anchor message */ +__msg("arg#0 doesn't point to a map value") +long test_wrong_wq_pointer(void *ctx) +{ + int key = 0; + struct bpf_wq *wq; + + wq = bpf_map_lookup_elem(&array, &key); + if (!wq) + return 1; + + if (bpf_wq_init(wq, &array, 0)) + return 2; + + if (bpf_wq_set_callback((void *)&wq, wq_callback, 0)) + return 3; + + return -22; +} + +SEC("?tc") +__log_level(2) +__failure +/* check that the first argument of bpf_wq_set_callback() + * is a correct bpf_wq pointer. + */ +__msg("mark_precise: frame0: regs=r1 stack= before") +__msg(": (85) call bpf_wq_set_callback_impl#") /* anchor message */ +__msg("off 1 doesn't point to 'struct bpf_wq' that is at 0") +long test_wrong_wq_pointer_offset(void *ctx) +{ + int key = 0; + struct bpf_wq *wq; + + wq = bpf_map_lookup_elem(&array, &key); + if (!wq) + return 1; + + if (bpf_wq_init(wq, &array, 0)) + return 2; + + if (bpf_wq_set_callback((void *)wq + 1, wq_cb_sleepable, 0)) + return 3; + + return -22; +} From patchwork Tue Apr 16 14:08:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Tissoires X-Patchwork-Id: 13631945 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7BA161369BB; Tue, 16 Apr 2024 14:11:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276663; cv=none; b=NmwDtYN4e3ZeiiP3p2DnM7jxJIB7lGxl2fuicrzbkBBEAGviIYkXwjQ6V6YmlWFMsDMrdvWhLARGkx+D20/hdbND0VmbN9/5GsIrlJvAfpfRJqkXBU/epNEWCsObNE5Im/E1PY0M+MqgWhYx7d/eLXwdtWnoYb6npIAGfIL4xvg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276663; c=relaxed/simple; bh=b3PkzLOIXQdFGv5PNduwNWgXmPt5byfF8ywwFY3Z51s=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=VM3Zy8YuHAo53wJf+mvIG/W6VjxZ7JQuBnyXeFJhVPBh/ojbgORcPE9zq84TfZrJebMlD7sO7YIgSOTn6OjDuemRHc5g2yRriw4aRYikbcHVd0uKctVIog0Jyd7NNUeV9HsNYD10kRmfBpUkGsjlNA93T3CNV11OqFxLwcXMT9A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ouzk7Scm; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ouzk7Scm" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BC20DC4AF52; Tue, 16 Apr 2024 14:10:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1713276663; bh=b3PkzLOIXQdFGv5PNduwNWgXmPt5byfF8ywwFY3Z51s=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=ouzk7ScmTGcsbcCyy5C6A0mRikAlG1gXv/S+Xsq0mHh0MtYKHOu4ZemxpiLiFCvIy kI4xi1nHY6wZoG0AHLT0vbgc1VGrmIRxu1powyaANcHKxaCKCXB5HONvS6P9rth+y9 4sz27fKJwtWe1x/fVHCU8W3lshmntQJVV65kRLoP1aMpYBpgAibjZJD+L+lcXagHUY vKn+WRpssdQR8FGecUZd/UkhkTKEz3Pn2KzPZs4gLf0xPtpHWIsBRJc0fX0QKx12U6 Hh2r/S91jSy1RDh8Z4Q8VRwYQqRqlC4gdjKzTXGSCXAN0H6yRy9Dom4ZFYSJT7J/ug ecaKwVoERRjmA== From: Benjamin Tissoires Date: Tue, 16 Apr 2024 16:08:30 +0200 Subject: [PATCH bpf-next 17/18] bpf: add bpf_wq_start Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240416-bpf_wq-v1-17-c9e66092f842@kernel.org> References: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> In-Reply-To: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Mykola Lysenko , Shuah Khan Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Benjamin Tissoires X-Mailer: b4 0.12.4 X-Developer-Signature: v=1; a=ed25519-sha256; t=1713276593; l=1510; i=bentiss@kernel.org; s=20230215; h=from:subject:message-id; bh=b3PkzLOIXQdFGv5PNduwNWgXmPt5byfF8ywwFY3Z51s=; b=qGHq6hqI33hY8ac+spbS1hKIZ4U4mwd4LG/eiNsLMlW8qLnD9HGxeaAp/QtVSNb7XJzNNUNyU X7tyfmJ0yrVCsk/g8vMgw6V2UgC8BHDlm3XwQ/ZG/GszfLKv17fsMl5 X-Developer-Key: i=bentiss@kernel.org; a=ed25519; pk=7D1DyAVh6ajCkuUTudt/chMuXWIJHlv2qCsRkIizvFw= X-Patchwork-Delegate: bpf@iogearbox.net again, copy/paste from bpf_timer_start(). Signed-off-by: Benjamin Tissoires --- kernel/bpf/helpers.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index e5c8adc44619..ed5309a37eda 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -2728,6 +2728,29 @@ __bpf_kfunc int bpf_wq_init(struct bpf_wq *wq, void *map, unsigned int flags) return __bpf_async_init(async, map, flags, BPF_ASYNC_TYPE_WQ); } +__bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags) +{ + struct bpf_async_kern *async = (struct bpf_async_kern *)wq; + struct bpf_work *w; + int ret = 0; + + if (in_nmi()) + return -EOPNOTSUPP; + if (flags) + return -EINVAL; + __bpf_spin_lock_irqsave(&async->lock); + w = async->work; + if (!w || !w->cb.prog) { + ret = -EINVAL; + goto out; + } + + schedule_work(&w->work); +out: + __bpf_spin_unlock_irqrestore(&async->lock); + return ret; +} + __bpf_kfunc int bpf_wq_set_callback_impl(struct bpf_wq *wq, int (callback_fn)(void *map, int *key, struct bpf_wq *wq), unsigned int flags__k, @@ -2821,6 +2844,7 @@ BTF_ID_FLAGS(func, bpf_dynptr_clone) BTF_ID_FLAGS(func, bpf_modify_return_test_tp) BTF_ID_FLAGS(func, bpf_wq_init) BTF_ID_FLAGS(func, bpf_wq_set_callback_impl) +BTF_ID_FLAGS(func, bpf_wq_start) BTF_KFUNCS_END(common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = { From patchwork Tue Apr 16 14:08:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Tissoires X-Patchwork-Id: 13631946 X-Patchwork-Delegate: bpf@iogearbox.net Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0D69712F5B7; Tue, 16 Apr 2024 14:11:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276667; cv=none; b=b6CtMCfSoqxVc9hVkjkuYTP8tz7LwigzrI6UzN4Sh4CH2RBGXqpF7FRlnWzf1EJOWqBCYi0jlO60hWRxDuNnOY8m9YjyjaHefu5k257g35K+1vn6ryadOQgwVx33IonZ52PvEXrG0KTfi8MpLnXycS4dLFOu3hwZXH7IecVNqgc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276667; c=relaxed/simple; bh=gnbd2RQ+3wctyheUyU6clAJleG99mpq90RdQg9GTbNQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=NhcpVKHa3p76w8dc8cWL6v7UD1utHYb59uYVa0fzmTzlrVIOWaeWlX02y3YPvn5mdjP0K6RPqB8xx/L89du39EyLW8E/6G2eHVjzu4yJO2FUg2QCMqXG7r/bF6En18c6yts/aLD/YqBl5ksor9uSYl8ChaCgxkJdEseJxr77+Cg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=BEnSi6sE; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="BEnSi6sE" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8BF52C4AF08; Tue, 16 Apr 2024 14:11:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1713276666; bh=gnbd2RQ+3wctyheUyU6clAJleG99mpq90RdQg9GTbNQ=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=BEnSi6sEfEy4ETnuX2tyAlbFU3a2WhICTdSAAhJc6cCjoZRYho6M2iM4SyMD0JWx5 QDIj9dD9GSmaPzfHCOcY64foFkxrK3fgSys79boI6ku3nDRxelSuBKDG5j7/Y2Tpdw Drg2xd/5jxoeOm/xQ40GkZQwZdd+0ExD0a4bJCIRkDCT0+Gt5DC9I7wtGTlBWeONfz epxg69KPorBQJBigXSF7VwNJdMFomIQpedr/fCtoIBPfLJ6JK3JOB/SaQIqOfViqRP cb7vjD9GI1pHp3NvkqECm3QQwY0b+kQAOC7mVFECeg9uazU268pcP9xa6/HI990T+S aYPM0Ae6Ip5EQ== From: Benjamin Tissoires Date: Tue, 16 Apr 2024 16:08:31 +0200 Subject: [PATCH bpf-next 18/18] selftests/bpf: wq: add bpf_wq_start() checks Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240416-bpf_wq-v1-18-c9e66092f842@kernel.org> References: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> In-Reply-To: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Mykola Lysenko , Shuah Khan Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Benjamin Tissoires X-Mailer: b4 0.12.4 X-Developer-Signature: v=1; a=ed25519-sha256; t=1713276593; l=3923; i=bentiss@kernel.org; s=20230215; h=from:subject:message-id; bh=gnbd2RQ+3wctyheUyU6clAJleG99mpq90RdQg9GTbNQ=; b=mzIPuuOm/lFkrF62NVNgi+3On2eJg7VFrdRf4n38fn++QOsGIWqJb9YU8tvgORWM3sVh/ni3y YhwFhuWpvFsCMCKYndxDhjdyGcYoMm0m90DEhYDSoSeKzREEXqbbpxO X-Developer-Key: i=bentiss@kernel.org; a=ed25519; pk=7D1DyAVh6ajCkuUTudt/chMuXWIJHlv2qCsRkIizvFw= X-Patchwork-Delegate: bpf@iogearbox.net Allows to test if allocation/free works Signed-off-by: Benjamin Tissoires --- tools/testing/selftests/bpf/bpf_experimental.h | 1 + tools/testing/selftests/bpf/prog_tests/wq.c | 22 ++++++++++++++++++++++ tools/testing/selftests/bpf/progs/wq.c | 22 +++++++++++++++++++--- 3 files changed, 42 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/bpf/bpf_experimental.h b/tools/testing/selftests/bpf/bpf_experimental.h index 272604f9c4a5..19dffa32fc08 100644 --- a/tools/testing/selftests/bpf/bpf_experimental.h +++ b/tools/testing/selftests/bpf/bpf_experimental.h @@ -471,6 +471,7 @@ extern struct cgroup_subsys_state *bpf_iter_css_next(struct bpf_iter_css *it) __ extern void bpf_iter_css_destroy(struct bpf_iter_css *it) __weak __ksym; extern int bpf_wq_init(struct bpf_wq *wq, void *map, unsigned int flags) __weak __ksym; +extern int bpf_wq_start(struct bpf_wq *wq, unsigned int flags) __weak __ksym; extern int bpf_wq_set_callback_impl(struct bpf_wq *wq, int (callback_fn)(void *map, int *key, struct bpf_wq *wq), unsigned int flags__k, void *aux__ign) __ksym; diff --git a/tools/testing/selftests/bpf/prog_tests/wq.c b/tools/testing/selftests/bpf/prog_tests/wq.c index 26ab69796103..8a4a91d944cc 100644 --- a/tools/testing/selftests/bpf/prog_tests/wq.c +++ b/tools/testing/selftests/bpf/prog_tests/wq.c @@ -6,9 +6,31 @@ void serial_test_wq(void) { + struct wq *wq_skel = NULL; + int err, prog_fd; + LIBBPF_OPTS(bpf_test_run_opts, topts); RUN_TESTS(wq); + + /* re-run the success test to check if the timer was actually executed */ + + wq_skel = wq__open_and_load(); + if (!ASSERT_OK_PTR(wq_skel, "wq_skel_load")) + return; + + err = wq__attach(wq_skel); + if (!ASSERT_OK(err, "wq_attach")) + return; + + prog_fd = bpf_program__fd(wq_skel->progs.test_syscall_array_sleepable); + err = bpf_prog_test_run_opts(prog_fd, &topts); + ASSERT_OK(err, "test_run"); + ASSERT_EQ(topts.retval, 0, "test_run"); + + usleep(50); /* 10 usecs should be enough, but give it extra */ + + ASSERT_EQ(wq_skel->bss->ok_sleepable, (1 << 1), "ok_sleepable"); } void serial_test_failures_wq(void) diff --git a/tools/testing/selftests/bpf/progs/wq.c b/tools/testing/selftests/bpf/progs/wq.c index c0a094c84834..0cdb9d273e60 100644 --- a/tools/testing/selftests/bpf/progs/wq.c +++ b/tools/testing/selftests/bpf/progs/wq.c @@ -49,6 +49,11 @@ struct { __type(value, struct elem); } lru SEC(".maps"); +#define CLOCK_MONOTONIC 1 + +__u32 ok; +__u32 ok_sleepable; + static int test_elem_callback(void *map, int *key, int (callback_fn)(void *map, int *key, struct bpf_wq *wq), u64 callback_flags) @@ -56,6 +61,10 @@ static int test_elem_callback(void *map, int *key, struct elem init = {}, *val; struct bpf_wq *wq; + if ((ok & (1 << *key) || + (ok_sleepable & (1 << *key)))) + return -22; + if (map == &lru && bpf_map_update_elem(map, key, &init, 0)) return -1; @@ -71,6 +80,9 @@ static int test_elem_callback(void *map, int *key, if (bpf_wq_set_callback(wq, callback_fn, callback_flags)) return -4; + if (bpf_wq_start(wq, 0)) + return -5; + return 0; } @@ -81,6 +93,10 @@ static int test_hmap_elem_callback(void *map, int *key, struct hmap_elem init = {}, *val; struct bpf_wq *wq; + if ((ok & (1 << *key) || + (ok_sleepable & (1 << *key)))) + return -22; + if (bpf_map_update_elem(map, key, &init, 0)) return -1; @@ -95,12 +111,12 @@ static int test_hmap_elem_callback(void *map, int *key, if (bpf_wq_set_callback(wq, callback_fn, callback_flags)) return -4; + if (bpf_wq_start(wq, 0)) + return -5; + return 0; } -__u32 ok; -__u32 ok_sleepable; - /* callback for non sleepable workqueue */ static int wq_callback(void *map, int *key, struct bpf_wq *work) {