From patchwork Mon Oct 16 12:57:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 13423194 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C71D6262AD; Mon, 16 Oct 2023 12:57:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="2ELmBn8Q"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="i6M5IYdJ" Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 52640A2; Mon, 16 Oct 2023 05:57:41 -0700 (PDT) Date: Mon, 16 Oct 2023 14:57:38 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1697461059; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=J4Fq3rGIS8nov1DFK/Ma53UlC+qh+yTUV7FkCGLePEw=; b=2ELmBn8QdHGkXS8KFExnyMLH04TqKwm5q5kggSZWjGJJQXDyPQbo/1qjR8ztL6Qspzzztp 2muhurZ8JeFTzRTj+b52fXECty7hv7dA/yfbbR+VwFe0j/UZC6oH8AkfUxdAcKxO8ILNmG XbRCbEUUQKC+cAa9LWlBIs7Ayhz4EWu0uwcq+wc06LU5Ju5s3dHUq8vPTGksrVPM7+fURk +KgTQ6owLEBGAZRsqD/LokQEST8f+G1HEjT9xZpF0aoS6NDK6MLu0lkrzzuqsQSK9jEfXZ KTqq/SgcNK4eSY64/M0tPA6v5IZy1nm7tK56fTGnfBihADwuAMZ2hdfVXBTUAQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1697461059; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=J4Fq3rGIS8nov1DFK/Ma53UlC+qh+yTUV7FkCGLePEw=; b=i6M5IYdJsbxkm1tan/1ajVWkqnTzZ/6Bn3+DiViKMtJtMtmwRn8CowMUI3FL3f3w2K66gF fPSZRClHgGbdAoBg== From: Sebastian Andrzej Siewior To: Daniel Borkmann Cc: John Fastabend , Jakub Kicinski , netdev@vger.kernel.org, bpf@vger.kernel.org, "David S. Miller" , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Alexei Starovoitov , Andrii Nakryiko , Eric Dumazet , Hao Luo , Jesper Dangaard Brouer , Jiri Olsa , Jonathan Lemon , KP Singh , Maciej Fijalkowski , Magnus Karlsson , Martin KaFai Lau , Paolo Abeni , Song Liu , Stanislav Fomichev , Thomas Gleixner , Yonghong Song , Toke =?utf-8?q?H=C3=B8iland-J?= =?utf-8?q?=C3=B8rgensen?= Subject: [PATCH bpf-next -v5] net: Add a warning if NAPI cb missed xdp_do_flush(). Message-ID: <20231016125738.Yt79p1uF@linutronix.de> References: <20230929165825.RvwBYGP1@linutronix.de> <20231004070926.5b4ba04c@kernel.org> <20231006154933.mQgxQHHt@linutronix.de> <20231006123139.5203444e@kernel.org> <20231007154351.UvncuBMF@linutronix.de> <20231010065745.lJLYdf_X@linutronix.de> <652627b386bbe_2d55e208d6@john.notmuch> <5efb2093-537e-0f7d-beef-d32c02ec4a3d@iogearbox.net> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <5efb2093-537e-0f7d-beef-d32c02ec4a3d@iogearbox.net> X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net A few drivers were missing a xdp_do_flush() invocation after XDP_REDIRECT. Add three helper functions each for one of the per-CPU lists. Return true if the per-CPU list is non-empty and flush the list. Add xdp_do_check_flushed() which invokes each helper functions and creates a warning if one of the functions had a non-empty list. Hide everything behind CONFIG_DEBUG_NET. Suggested-by: Jesper Dangaard Brouer Reviewed-by: Toke Høiland-Jørgensen Acked-by: Jakub Kicinski Acked-by: John Fastabend Signed-off-by: Sebastian Andrzej Siewior --- v4…v5: - rebase on top of bpf-next - Collected Reviewed/Acked from the list. v3…v4: - s/creats/creates as per Simon Horman. v2…v3: - Collected Reviewed/Acked from the list. - Added an include dev.h to filter.c, the robot pointed out a missing prototype. v1…v2: - Moved xdp_do_check_flushed() to net/core/dev.h. - Stripped __ from function names. - Removed empty lines within an ifdef block. - xdp_do_check_flushed() is now behind CONFIG_DEBUG_NET && CONFIG_BPF_SYSCALL. dev_check_flush and cpu_map_check_flush are now only behind CONFIG_DEBUG_NET. They have no empty inline function for the !CONFIG_DEBUG_NET case since they are only called in CONFIG_DEBUG_NET case. include/linux/bpf.h | 3 +++ include/net/xdp_sock.h | 9 +++++++++ kernel/bpf/cpumap.c | 10 ++++++++++ kernel/bpf/devmap.c | 10 ++++++++++ net/core/dev.c | 2 ++ net/core/dev.h | 6 ++++++ net/core/filter.c | 16 ++++++++++++++++ net/xdp/xsk.c | 10 ++++++++++ 8 files changed, 66 insertions(+) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index f0891ba24cb1c..92b76360ff58a 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -2478,6 +2478,9 @@ void bpf_dynptr_init(struct bpf_dynptr_kern *ptr, void *data, enum bpf_dynptr_type type, u32 offset, u32 size); void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr); void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr); + +bool dev_check_flush(void); +bool cpu_map_check_flush(void); #else /* !CONFIG_BPF_SYSCALL */ static inline struct bpf_prog *bpf_prog_get(u32 ufd) { diff --git a/include/net/xdp_sock.h b/include/net/xdp_sock.h index 69b472604b86f..7dd0df2f6f8e6 100644 --- a/include/net/xdp_sock.h +++ b/include/net/xdp_sock.h @@ -109,4 +109,13 @@ static inline void __xsk_map_flush(void) #endif /* CONFIG_XDP_SOCKETS */ +#if defined(CONFIG_XDP_SOCKETS) && defined(CONFIG_DEBUG_NET) +bool xsk_map_check_flush(void); +#else +static inline bool xsk_map_check_flush(void) +{ + return false; +} +#endif + #endif /* _LINUX_XDP_SOCK_H */ diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c index e42a1bdb7f536..8a0bb80fe48a3 100644 --- a/kernel/bpf/cpumap.c +++ b/kernel/bpf/cpumap.c @@ -764,6 +764,16 @@ void __cpu_map_flush(void) } } +#ifdef CONFIG_DEBUG_NET +bool cpu_map_check_flush(void) +{ + if (list_empty(this_cpu_ptr(&cpu_map_flush_list))) + return false; + __cpu_map_flush(); + return true; +} +#endif + static int __init cpu_map_init(void) { int cpu; diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 4d42f6ed6c11a..a936c704d4e77 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -418,6 +418,16 @@ void __dev_flush(void) } } +#ifdef CONFIG_DEBUG_NET +bool dev_check_flush(void) +{ + if (list_empty(this_cpu_ptr(&dev_flush_list))) + return false; + __dev_flush(); + return true; +} +#endif + /* Elements are kept alive by RCU; either by rcu_read_lock() (from syscall) or * by local_bh_disable() (from XDP calls inside NAPI). The * rcu_read_lock_bh_held() below makes lockdep accept both. diff --git a/net/core/dev.c b/net/core/dev.c index 606a366cc2095..9273b12ecf6fa 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -6526,6 +6526,8 @@ static int __napi_poll(struct napi_struct *n, bool *repoll) if (test_bit(NAPI_STATE_SCHED, &n->state)) { work = n->poll(n, weight); trace_napi_poll(n, work, weight); + + xdp_do_check_flushed(n); } if (unlikely(work > weight)) diff --git a/net/core/dev.h b/net/core/dev.h index e075e198092cc..f66125857af77 100644 --- a/net/core/dev.h +++ b/net/core/dev.h @@ -136,4 +136,10 @@ static inline void netif_set_gro_ipv4_max_size(struct net_device *dev, } int rps_cpumask_housekeeping(struct cpumask *mask); + +#if defined(CONFIG_DEBUG_NET) && defined(CONFIG_BPF_SYSCALL) +void xdp_do_check_flushed(struct napi_struct *napi); +#else +static inline void xdp_do_check_flushed(struct napi_struct *napi) { } +#endif #endif diff --git a/net/core/filter.c b/net/core/filter.c index cc2e4babc85fb..21d75108c2e94 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -83,6 +83,8 @@ #include #include +#include "dev.h" + static const struct bpf_func_proto * bpf_sk_base_func_proto(enum bpf_func_id func_id); @@ -4208,6 +4210,20 @@ void xdp_do_flush(void) } EXPORT_SYMBOL_GPL(xdp_do_flush); +#if defined(CONFIG_DEBUG_NET) && defined(CONFIG_BPF_SYSCALL) +void xdp_do_check_flushed(struct napi_struct *napi) +{ + bool ret; + + ret = dev_check_flush(); + ret |= cpu_map_check_flush(); + ret |= xsk_map_check_flush(); + + WARN_ONCE(ret, "Missing xdp_do_flush() invocation after NAPI by %ps\n", + napi->poll); +} +#endif + void bpf_clear_redirect_map(struct bpf_map *map) { struct bpf_redirect_info *ri; diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c index 7482d0aca5046..e23689b82914f 100644 --- a/net/xdp/xsk.c +++ b/net/xdp/xsk.c @@ -391,6 +391,16 @@ void __xsk_map_flush(void) } } +#ifdef CONFIG_DEBUG_NET +bool xsk_map_check_flush(void) +{ + if (list_empty(this_cpu_ptr(&xskmap_flush_list))) + return false; + __xsk_map_flush(); + return true; +} +#endif + void xsk_tx_completed(struct xsk_buff_pool *pool, u32 nb_entries) { xskq_prod_submit_n(pool->cq, nb_entries);