From patchwork Mon Jun 12 17:23:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13277001 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E99121F19A for ; Mon, 12 Jun 2023 17:23:12 +0000 (UTC) Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 37ACE1A5 for ; Mon, 12 Jun 2023 10:23:11 -0700 (PDT) Received: by mail-pg1-x549.google.com with SMTP id 41be03b00d2f7-5488dccfbdfso3346249a12.2 for ; Mon, 12 Jun 2023 10:23:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686590590; x=1689182590; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=qsajQXCojYmprUydGUF65Mhm0IhEwrrdRl69Y6JxBAY=; b=FxlIAzQJZCeMZDT0zEWVeGeQ8Zt37o/EPBh2ybLUqF2+nAbENnHB/cKJ6ihVmKQdni K7CAJa/3DZHF1E43Bkr1JLK9OPxG07vRFvbBoRCQmIROX2lOArttc4pcEHqig24Xqus6 g0aLWuvEJtrqS6CWoxwRnDLHLOKDhX0jIfgVueZmsVG16G1c1VpMAyu/K71KMgYnka7y mXgUpu5c4aG5Jh6LCwTLq1aPsR6hY2MR8yrzeE4tmUuvaSFFXqcs8R0ZLTkXWHoZk/kr mDt8uMuQFJnMelhCrpXvGIW5/u9QnpSeTh/QnnqjQWF+SoeGN849z/pK7SySfy1ULyKv ywBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686590590; x=1689182590; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qsajQXCojYmprUydGUF65Mhm0IhEwrrdRl69Y6JxBAY=; b=gaIv4BuasDkLTbopRujOiPNBSM8NUkLyvjqqYrfhDt1Lid5oUFq0i/HAebEHSc0Sjw Oc53Tj3aXbz2C2Th8go8fF63hkMbyJ4SxtiexCxL9Wv2KWJoOWGmiBm0IxlGwCjtKayq kbS/2yGMYDliCopNZsDBvMrbIea4IxYNsIfWPLqAolQfavDM0lQKETqx7X1KWrUSbqRQ t2T+fh7PUZXR3/1m2JFIYHeTzS3FcmsgFyM97xX06ieo7TKzbYBEr9npBd33cl0Oifuq viPQ/5hWRDwovcM9Vy9S7w00+bWKVn3PEWp0e6I1CHrRUelLjWwcxV3eXZXL8EziN68V IU7w== X-Gm-Message-State: AC+VfDyc6Z1dT1dOAwf/n2+dwQoc1g1+U6eHTMDOnA4acZXmLKL6Hu4O kH3CbuFgyfk1mgXsAcq2ScG6iW9rXoq0CIWX3dV8ywVe08skaHqH6GMZgbiEpZjNFwz0HQ1kUSN HyrecDM4Khwbukldpdi0XCLou2Hs7S8ru/NnxD97kWb+noWD+JA== X-Google-Smtp-Source: ACHHUZ6pUpd6PQxwI8A6BG0/L7h7zgVB4IwuR63bmoW8iPCtAJGU6HJzEZKxQ9RRI22pv3ELLAin6sQ= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a17:903:44e:b0:1b3:ddff:2b4e with SMTP id iw14-20020a170903044e00b001b3ddff2b4emr329221plb.10.1686590590652; Mon, 12 Jun 2023 10:23:10 -0700 (PDT) Date: Mon, 12 Jun 2023 10:23:01 -0700 In-Reply-To: <20230612172307.3923165-1-sdf@google.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20230612172307.3923165-1-sdf@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230612172307.3923165-2-sdf@google.com> Subject: [RFC bpf-next 1/7] bpf: rename some xdp-metadata functions into dev-bound From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, netdev@vger.kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC No functional changes. To make existing dev-bound infrastructure more generic and be less tightly bound to the xdp layer, rename some functions and move kfunc-related things into kernel/bpf/offload.c Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- include/net/offload.h | 28 ++++++++++++++++++++++++++++ include/net/xdp.h | 18 +----------------- kernel/bpf/offload.c | 26 ++++++++++++++++++++++++-- kernel/bpf/verifier.c | 4 ++-- net/core/xdp.c | 20 ++------------------ 5 files changed, 57 insertions(+), 39 deletions(-) create mode 100644 include/net/offload.h diff --git a/include/net/offload.h b/include/net/offload.h new file mode 100644 index 000000000000..264a35881473 --- /dev/null +++ b/include/net/offload.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __LINUX_NET_OFFLOAD_H__ +#define __LINUX_NET_OFFLOAD_H__ + +#include + +#define XDP_METADATA_KFUNC_xxx \ + NETDEV_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_TIMESTAMP, \ + bpf_xdp_metadata_rx_timestamp) \ + NETDEV_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_HASH, \ + bpf_xdp_metadata_rx_hash) + +enum { +#define NETDEV_METADATA_KFUNC(name, _) name, +XDP_METADATA_KFUNC_xxx +#undef NETDEV_METADATA_KFUNC +MAX_NETDEV_METADATA_KFUNC, +}; + +#ifdef CONFIG_NET +u32 bpf_dev_bound_kfunc_id(int id); +bool bpf_is_dev_bound_kfunc(u32 btf_id); +#else +static inline u32 bpf_dev_bound_kfunc_id(int id) { return 0; } +static inline bool bpf_is_dev_bound_kfunc(u32 btf_id) { return false; } +#endif + +#endif /* __LINUX_NET_OFFLOAD_H__ */ diff --git a/include/net/xdp.h b/include/net/xdp.h index d1c5381fc95f..de4c3b70abde 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -9,6 +9,7 @@ #include /* skb_shared_info */ #include #include +#include /** * DOC: XDP RX-queue information @@ -384,19 +385,6 @@ void xdp_attachment_setup(struct xdp_attachment_info *info, #define DEV_MAP_BULK_SIZE XDP_BULK_QUEUE_SIZE -#define XDP_METADATA_KFUNC_xxx \ - XDP_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_TIMESTAMP, \ - bpf_xdp_metadata_rx_timestamp) \ - XDP_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_HASH, \ - bpf_xdp_metadata_rx_hash) \ - -enum { -#define XDP_METADATA_KFUNC(name, _) name, -XDP_METADATA_KFUNC_xxx -#undef XDP_METADATA_KFUNC -MAX_XDP_METADATA_KFUNC, -}; - enum xdp_rss_hash_type { /* First part: Individual bits for L3/L4 types */ XDP_RSS_L3_IPV4 = BIT(0), @@ -444,14 +432,10 @@ enum xdp_rss_hash_type { }; #ifdef CONFIG_NET -u32 bpf_xdp_metadata_kfunc_id(int id); -bool bpf_dev_bound_kfunc_id(u32 btf_id); void xdp_set_features_flag(struct net_device *dev, xdp_features_t val); void xdp_features_set_redirect_target(struct net_device *dev, bool support_sg); void xdp_features_clear_redirect_target(struct net_device *dev); #else -static inline u32 bpf_xdp_metadata_kfunc_id(int id) { return 0; } -static inline bool bpf_dev_bound_kfunc_id(u32 btf_id) { return false; } static inline void xdp_set_features_flag(struct net_device *dev, xdp_features_t val) diff --git a/kernel/bpf/offload.c b/kernel/bpf/offload.c index 8a26cd8814c1..235d81f7e0ed 100644 --- a/kernel/bpf/offload.c +++ b/kernel/bpf/offload.c @@ -844,9 +844,9 @@ void *bpf_dev_bound_resolve_kfunc(struct bpf_prog *prog, u32 func_id) if (!ops) goto out; - if (func_id == bpf_xdp_metadata_kfunc_id(XDP_METADATA_KFUNC_RX_TIMESTAMP)) + if (func_id == bpf_dev_bound_kfunc_id(XDP_METADATA_KFUNC_RX_TIMESTAMP)) p = ops->xmo_rx_timestamp; - else if (func_id == bpf_xdp_metadata_kfunc_id(XDP_METADATA_KFUNC_RX_HASH)) + else if (func_id == bpf_dev_bound_kfunc_id(XDP_METADATA_KFUNC_RX_HASH)) p = ops->xmo_rx_hash; out: up_read(&bpf_devs_lock); @@ -854,6 +854,28 @@ void *bpf_dev_bound_resolve_kfunc(struct bpf_prog *prog, u32 func_id) return p; } +BTF_SET_START(dev_bound_kfunc_ids) +#define NETDEV_METADATA_KFUNC(name, str) BTF_ID(func, str) +XDP_METADATA_KFUNC_xxx +#undef NETDEV_METADATA_KFUNC +BTF_SET_END(dev_bound_kfunc_ids) + +BTF_ID_LIST(dev_bound_kfunc_ids_unsorted) +#define NETDEV_METADATA_KFUNC(name, str) BTF_ID(func, str) +XDP_METADATA_KFUNC_xxx +#undef NETDEV_METADATA_KFUNC + +u32 bpf_dev_bound_kfunc_id(int id) +{ + /* dev_bound_kfunc_ids is sorted and can't be used */ + return dev_bound_kfunc_ids_unsorted[id]; +} + +bool bpf_is_dev_bound_kfunc(u32 btf_id) +{ + return btf_id_set_contains(&dev_bound_kfunc_ids, btf_id); +} + static int __init bpf_offload_init(void) { return rhashtable_init(&offdevs, &offdevs_params); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 1e38584d497c..4db48b5af47e 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -2721,7 +2721,7 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset) } } - if (bpf_dev_bound_kfunc_id(func_id)) { + if (bpf_is_dev_bound_kfunc(func_id)) { err = bpf_dev_bound_kfunc_check(&env->log, prog_aux); if (err) return err; @@ -17757,7 +17757,7 @@ static void specialize_kfunc(struct bpf_verifier_env *env, void *xdp_kfunc; bool is_rdonly; - if (bpf_dev_bound_kfunc_id(func_id)) { + if (bpf_is_dev_bound_kfunc(func_id)) { xdp_kfunc = bpf_dev_bound_resolve_kfunc(prog, func_id); if (xdp_kfunc) { *addr = (unsigned long)xdp_kfunc; diff --git a/net/core/xdp.c b/net/core/xdp.c index 41e5ca8643ec..819767697370 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -741,9 +741,9 @@ __bpf_kfunc int bpf_xdp_metadata_rx_hash(const struct xdp_md *ctx, u32 *hash, __diag_pop(); BTF_SET8_START(xdp_metadata_kfunc_ids) -#define XDP_METADATA_KFUNC(_, name) BTF_ID_FLAGS(func, name, 0) +#define NETDEV_METADATA_KFUNC(_, name) BTF_ID_FLAGS(func, name, 0) XDP_METADATA_KFUNC_xxx -#undef XDP_METADATA_KFUNC +#undef NETDEV_METADATA_KFUNC BTF_SET8_END(xdp_metadata_kfunc_ids) static const struct btf_kfunc_id_set xdp_metadata_kfunc_set = { @@ -751,22 +751,6 @@ static const struct btf_kfunc_id_set xdp_metadata_kfunc_set = { .set = &xdp_metadata_kfunc_ids, }; -BTF_ID_LIST(xdp_metadata_kfunc_ids_unsorted) -#define XDP_METADATA_KFUNC(name, str) BTF_ID(func, str) -XDP_METADATA_KFUNC_xxx -#undef XDP_METADATA_KFUNC - -u32 bpf_xdp_metadata_kfunc_id(int id) -{ - /* xdp_metadata_kfunc_ids is sorted and can't be used */ - return xdp_metadata_kfunc_ids_unsorted[id]; -} - -bool bpf_dev_bound_kfunc_id(u32 btf_id) -{ - return btf_id_set8_contains(&xdp_metadata_kfunc_ids, btf_id); -} - static int __init xdp_metadata_init(void) { return register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &xdp_metadata_kfunc_set); From patchwork Mon Jun 12 17:23:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13277002 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 386D21F931 for ; Mon, 12 Jun 2023 17:23:14 +0000 (UTC) Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A97A610DC for ; Mon, 12 Jun 2023 10:23:12 -0700 (PDT) Received: by mail-pl1-x64a.google.com with SMTP id d9443c01a7336-1b3b3c69969so18458255ad.0 for ; Mon, 12 Jun 2023 10:23:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686590592; x=1689182592; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=urWJm7reT2IarzSkE9tEiPm+OWYgjit92cM3b8fi47A=; b=StfmV5glCZgD5D5jo9VaRcx/yBvLgF4KheIO0x6k12Ul0fg3XF86kgkGHqp0qefrkK LZDa5zR6RY+o8+tbdNA3oO8QmrS6/sVgdsLgC3zcX2MoTTqD2KHfIjfH/kqmYMJE0kiz q2R45wMVbOK2H0wDfapQPyy+NPIHKJrDAtxPfb9u1q+Q/pg5nuiOzrtBr/n90ZUIPeeP mD2Sh9PfOXxsZBWB/OW2eawiDApto5dHYvSdnKEi8Hwbgrd+V+Tap+o5eXnIgSEgyEgg lULHAL5mip0nX0lW3T66zLJ1fCY2xk70M3V74jJ6GczBSB7s9BdP24ar1q6B1bH2TKHd /bhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686590592; x=1689182592; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=urWJm7reT2IarzSkE9tEiPm+OWYgjit92cM3b8fi47A=; b=AwLwqbiZ8jY3c2EBdi4K6ld44YPbFCkC/3NlVDNOFPCiDPDc4l64on7/GyO8LHJw2j W1+bRiOs4jaZ/7zgDrKMd2VApHi2t9MP5JctO7Z1spiNufcM0Hj8S0KZhZoV92+vpLn+ Mp38/04P6ZfD13dNtiIDD7mz2VzdIVyHRMd3M1kp956tXzu7ItREdgXbCJzTk95+XpAC 5wb1n25XbQVxAxgqKucD+BvByd5bz4NIUx4sIXVI6pAAgeMB7cHGNG28B9gx3avAlBjX RE3KTlAzP6hCIK7LL1/C4ST/N+wmtF/yotgcBGqDqiT/+lhP4M1odGsJpLggw3Rhmb6U xmOA== X-Gm-Message-State: AC+VfDyTMg8cOFY6dzEwmxXLFvN0Gg9OLiWF8vGoy0bf3dX0rJLavc/x xl/lyXkbh+ZAKgXES8Yff0mN1u6F/KGp/l+LcdTMKw0S7ZLnxguamj7kmp4Kw26eXGraCY3Q0QP ctVvGW94Mp8XIyc5ALHshP+sgaB8G7VgQNDANuhJo6B4jbbjQhw== X-Google-Smtp-Source: ACHHUZ5Z1OS46iO1QCtf08JRymmg3unv8Ntbh6VT8DSLR3zYyv7jOGK1qE8xfw6aCKVFuNCFoPxm9XY= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a17:902:728f:b0:1b3:c0aa:2453 with SMTP id d15-20020a170902728f00b001b3c0aa2453mr943690pll.6.1686590592079; Mon, 12 Jun 2023 10:23:12 -0700 (PDT) Date: Mon, 12 Jun 2023 10:23:02 -0700 In-Reply-To: <20230612172307.3923165-1-sdf@google.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20230612172307.3923165-1-sdf@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230612172307.3923165-3-sdf@google.com> Subject: [RFC bpf-next 2/7] bpf: resolve single typedef when walking structs From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, netdev@vger.kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC It is impossible to use skb_frag_t in the tracing program. So let's resolve a single typedef when walking the struct. Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- kernel/bpf/btf.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index bd2cac057928..9bdaa1225e8a 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -6140,6 +6140,8 @@ static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf, *flag = 0; again: tname = __btf_name_by_offset(btf, t->name_off); + if (btf_type_is_typedef(t)) + t = btf_type_by_id(btf, t->type); if (!btf_type_is_struct(t)) { bpf_log(log, "Type '%s' is not a struct\n", tname); return -EINVAL; From patchwork Mon Jun 12 17:23:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13277003 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D8F2A21CD7 for ; Mon, 12 Jun 2023 17:23:16 +0000 (UTC) Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4170C10B for ; Mon, 12 Jun 2023 10:23:14 -0700 (PDT) Received: by mail-pg1-x549.google.com with SMTP id 41be03b00d2f7-5488dccfbdfso3346298a12.2 for ; Mon, 12 Jun 2023 10:23:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686590593; x=1689182593; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=FP8q+5eUSu4wPpJtNuCq9IaYUWNjGUb+IDjkhrKxnb4=; b=HMBStsomonxgE+Takijk8DdwhNkorBPxQrJsyO28txFdFFl+ezHVRkSrnprgMs331L PLKmPqg7sX3wXQ0EzcwrteWtrgqItBv1Tthf01cK7qhDbGTyH8Cz0Vf9zH4Yf5u3rYQ7 sh0bumzHOTfdE1BAMSMJPx9fZtstlaKsFoMgRON8Ew8+5gx6n1bYLoKNsL/71glKeriQ dRidk/x3tC18Y+nGJa1ZinHJ4io4FnLUW7Z0CtQbsZXCv2EvP+vx8Th+Ltg3YG4oyU78 QREfWfEBU/VhI5yfRYsu7b/oGGEAPBL3A7KZG2Y8feYPH+3wU+D6jlw5wt7Bm4NgC2Uv XpFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686590593; x=1689182593; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FP8q+5eUSu4wPpJtNuCq9IaYUWNjGUb+IDjkhrKxnb4=; b=TNGkSV9paNLQAvyWUMI+W/gaB4MbOP/rI7hB1eQiHjnMN7m7Jg9scTsyGRgf6Q/u6E 1v6gPtY4WzXt7sDVgiGMlSWDkNmYlS2Xt3P7OYC26N8kxmvlSKOYZcMbfuVC5T/XnGJB FvW8FmY41eL3bdYm31ZSdq3AnARYyqQRDTgg5WT9xCj9hYOl3REk6RPPwYsod1vvROHa XI3EQdsTjuGl47PgEXl3PuHZ8FFYXSX0GC0xs9hwKwsru+oWOhD9zjSvz7pDavGUDQkj duUX8TT4/ueU+BJk/AVLaY0u83upidm/RGb4uxzeO7RGMmKkJe4mwuVIYgi9svfsnExM ZbQQ== X-Gm-Message-State: AC+VfDw3/IS4WiBlH4RpLSKYMHPShbATEjes7NRWFIMItqMMd0ZvJNVD R1n9VBnjLNn1xkPSN0/nWFqwbY24OVxfrBMDue+SHQg0QkYh5OWQ0bRJTMqQ3MMs/b/+TfyiIMD W8DXQ2j+j0UXN/svHynHR0VysH9JN0Nh0wWxJ9ja4sW4UyK98oQ== X-Google-Smtp-Source: ACHHUZ46VZr3yPfTmWOnupyzuND0woV05E5Xrcx0bKmzQ12URiZIUXnrXE+LyUc+3yeErt6sV3E6Nzk= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a17:90a:d356:b0:256:335f:d59a with SMTP id i22-20020a17090ad35600b00256335fd59amr1633332pjx.3.1686590593535; Mon, 12 Jun 2023 10:23:13 -0700 (PDT) Date: Mon, 12 Jun 2023 10:23:03 -0700 In-Reply-To: <20230612172307.3923165-1-sdf@google.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20230612172307.3923165-1-sdf@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230612172307.3923165-4-sdf@google.com> Subject: [RFC bpf-next 3/7] bpf: implement devtx hook points From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, netdev@vger.kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC devtx is a lightweight set of hooks before and after packet transmission. The hook is supposed to work for both skb and xdp paths by exposing a light-weight packet wrapper via devtx_frame (header portion + frags). devtx is implemented as a tracing program which has access to the XDP-metadata-like kfuncs. The initial set of kfuncs is implemented in the next patch, but the idea is similar to XDP metadata: the kfuncs have netdev-specific implementation, but common interface. Upon loading, the kfuncs are resolved to direct calls against per-netdev implementation. This can be achieved by marking devtx-tracing programs as dev-bound (largely reusing xdp-dev-bound program infrastructure). Attachment and detachment is implemented via syscall BPF program by calling bpf_devtx_sb_attach (attach to tx-submission) or bpf_devtx_cp_attach (attach to tx completion). Right now, the attachment does not return a link and doesn't support multiple programs. I plan to switch to Daniel's bpf_mprog infra once it's available. Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- MAINTAINERS | 2 + include/linux/netdevice.h | 2 + include/net/devtx.h | 76 ++++++++++++++ kernel/bpf/offload.c | 6 ++ net/core/Makefile | 1 + net/core/dev.c | 2 + net/core/devtx.c | 208 ++++++++++++++++++++++++++++++++++++++ 7 files changed, 297 insertions(+) create mode 100644 include/net/devtx.h create mode 100644 net/core/devtx.c diff --git a/MAINTAINERS b/MAINTAINERS index c904dba1733b..516529b42e66 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -22976,11 +22976,13 @@ L: bpf@vger.kernel.org S: Supported F: drivers/net/ethernet/*/*/*/*/*xdp* F: drivers/net/ethernet/*/*/*xdp* +F: include/net/devtx.h F: include/net/xdp.h F: include/net/xdp_priv.h F: include/trace/events/xdp.h F: kernel/bpf/cpumap.c F: kernel/bpf/devmap.c +F: net/core/devtx.c F: net/core/xdp.c F: samples/bpf/xdp* F: tools/testing/selftests/bpf/*/*xdp* diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 08fbd4622ccf..e08e3fd39dfc 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -2238,6 +2238,8 @@ struct net_device { unsigned int real_num_rx_queues; struct bpf_prog __rcu *xdp_prog; + struct bpf_prog __rcu *devtx_sb; + struct bpf_prog __rcu *devtx_cp; unsigned long gro_flush_timeout; int napi_defer_hard_irqs; #define GRO_LEGACY_MAX_SIZE 65536u diff --git a/include/net/devtx.h b/include/net/devtx.h new file mode 100644 index 000000000000..7eab66d0ce80 --- /dev/null +++ b/include/net/devtx.h @@ -0,0 +1,76 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __LINUX_NET_DEVTX_H__ +#define __LINUX_NET_DEVTX_H__ + +#include +#include +#include +#include + +struct devtx_frame { + void *data; + u16 len; + struct skb_shared_info *sinfo; /* for frags */ +}; + +#ifdef CONFIG_NET +void devtx_submit(struct net_device *netdev, struct devtx_frame *ctx); +void devtx_complete(struct net_device *netdev, struct devtx_frame *ctx); +bool is_devtx_kfunc(u32 kfunc_id); +void devtx_shutdown(struct net_device *netdev); + +static inline void devtx_frame_from_skb(struct devtx_frame *ctx, struct sk_buff *skb) +{ + ctx->data = skb->data; + ctx->len = skb_headlen(skb); + ctx->sinfo = skb_shinfo(skb); +} + +static inline void devtx_frame_from_xdp(struct devtx_frame *ctx, struct xdp_frame *xdpf) +{ + ctx->data = xdpf->data; + ctx->len = xdpf->len; + ctx->sinfo = xdp_frame_has_frags(xdpf) ? xdp_get_shared_info_from_frame(xdpf) : NULL; +} + +DECLARE_STATIC_KEY_FALSE(devtx_enabled); + +static inline bool devtx_submit_enabled(struct net_device *netdev) +{ + return static_branch_unlikely(&devtx_enabled) && + rcu_access_pointer(netdev->devtx_sb); +} + +static inline bool devtx_complete_enabled(struct net_device *netdev) +{ + return static_branch_unlikely(&devtx_enabled) && + rcu_access_pointer(netdev->devtx_cp); +} +#else +static inline void devtx_submit(struct net_device *netdev, struct devtx_frame *ctx) +{ +} + +static inline void devtx_complete(struct net_device *netdev, struct devtx_frame *ctx) +{ +} + +static inline bool is_devtx_kfunc(u32 kfunc_id) +{ + return false; +} + +static inline void devtx_shutdown(struct net_device *netdev) +{ +} + +static inline void devtx_frame_from_skb(struct devtx_frame *ctx, struct sk_buff *skb) +{ +} + +static inline void devtx_frame_from_xdp(struct devtx_frame *ctx, struct xdp_frame *xdpf) +{ +} +#endif + +#endif /* __LINUX_NET_DEVTX_H__ */ diff --git a/kernel/bpf/offload.c b/kernel/bpf/offload.c index 235d81f7e0ed..9cfe96422c80 100644 --- a/kernel/bpf/offload.c +++ b/kernel/bpf/offload.c @@ -25,6 +25,7 @@ #include #include #include +#include /* Protects offdevs, members of bpf_offload_netdev and offload members * of all progs. @@ -228,6 +229,7 @@ int bpf_prog_dev_bound_init(struct bpf_prog *prog, union bpf_attr *attr) int err; if (attr->prog_type != BPF_PROG_TYPE_SCHED_CLS && + attr->prog_type != BPF_PROG_TYPE_TRACING && attr->prog_type != BPF_PROG_TYPE_XDP) return -EINVAL; @@ -238,6 +240,10 @@ int bpf_prog_dev_bound_init(struct bpf_prog *prog, union bpf_attr *attr) attr->prog_flags & BPF_F_XDP_DEV_BOUND_ONLY) return -EINVAL; + if (attr->prog_type == BPF_PROG_TYPE_TRACING && + !is_devtx_kfunc(prog->aux->attach_btf_id)) + return -EINVAL; + netdev = dev_get_by_index(current->nsproxy->net_ns, attr->prog_ifindex); if (!netdev) return -EINVAL; diff --git a/net/core/Makefile b/net/core/Makefile index 8f367813bc68..c1db05ccfac7 100644 --- a/net/core/Makefile +++ b/net/core/Makefile @@ -39,4 +39,5 @@ obj-$(CONFIG_FAILOVER) += failover.o obj-$(CONFIG_NET_SOCK_MSG) += skmsg.o obj-$(CONFIG_BPF_SYSCALL) += sock_map.o obj-$(CONFIG_BPF_SYSCALL) += bpf_sk_storage.o +obj-$(CONFIG_BPF_SYSCALL) += devtx.o obj-$(CONFIG_OF) += of_net.o diff --git a/net/core/dev.c b/net/core/dev.c index 3393c2f3dbe8..ef0e65e68024 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -150,6 +150,7 @@ #include #include #include +#include #include "dev.h" #include "net-sysfs.h" @@ -10875,6 +10876,7 @@ void unregister_netdevice_many_notify(struct list_head *head, dev_shutdown(dev); dev_xdp_uninstall(dev); + devtx_shutdown(dev); bpf_dev_bound_netdev_unregister(dev); netdev_offload_xstats_disable_all(dev); diff --git a/net/core/devtx.c b/net/core/devtx.c new file mode 100644 index 000000000000..b7cbc26d1c01 --- /dev/null +++ b/net/core/devtx.c @@ -0,0 +1,208 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include +#include + +DEFINE_STATIC_KEY_FALSE(devtx_enabled); +EXPORT_SYMBOL_GPL(devtx_enabled); + +static void devtx_run(struct net_device *netdev, struct devtx_frame *ctx, struct bpf_prog **pprog) +{ + struct bpf_prog *prog; + void *real_ctx[1] = {ctx}; + + prog = rcu_dereference(*pprog); + if (likely(prog)) + bpf_prog_run(prog, real_ctx); +} + +void devtx_submit(struct net_device *netdev, struct devtx_frame *ctx) +{ + rcu_read_lock(); + devtx_run(netdev, ctx, &netdev->devtx_sb); + rcu_read_unlock(); +} +EXPORT_SYMBOL_GPL(devtx_submit); + +void devtx_complete(struct net_device *netdev, struct devtx_frame *ctx) +{ + rcu_read_lock(); + devtx_run(netdev, ctx, &netdev->devtx_cp); + rcu_read_unlock(); +} +EXPORT_SYMBOL_GPL(devtx_complete); + +/** + * devtx_sb - Called for every egress netdev packet + * + * Note: this function is never actually called by the kernel and declared + * only to allow loading an attaching appropriate tracepoints. + */ +__weak noinline void devtx_sb(struct devtx_frame *ctx) +{ +} + +/** + * devtx_cp - Called upon egress netdev packet completion + * + * Note: this function is never actually called by the kernel and declared + * only to allow loading an attaching appropriate tracepoints. + */ +__weak noinline void devtx_cp(struct devtx_frame *ctx) +{ +} + +BTF_SET8_START(bpf_devtx_hook_ids) +BTF_ID_FLAGS(func, devtx_sb) +BTF_ID_FLAGS(func, devtx_cp) +BTF_SET8_END(bpf_devtx_hook_ids) + +static const struct btf_kfunc_id_set bpf_devtx_hook_set = { + .owner = THIS_MODULE, + .set = &bpf_devtx_hook_ids, +}; + +static DEFINE_MUTEX(devtx_attach_lock); + +static int __bpf_devtx_detach(struct net_device *netdev, struct bpf_prog **pprog) +{ + if (!*pprog) + return -EINVAL; + bpf_prog_put(*pprog); + *pprog = NULL; + + static_branch_dec(&devtx_enabled); + return 0; +} + +static int __bpf_devtx_attach(struct net_device *netdev, int prog_fd, + const char *attach_func_name, struct bpf_prog **pprog) +{ + struct bpf_prog *prog; + int ret = 0; + + if (prog_fd < 0) + return __bpf_devtx_detach(netdev, pprog); + + if (*pprog) + return -EBUSY; + + prog = bpf_prog_get(prog_fd); + if (IS_ERR(prog)) + return PTR_ERR(prog); + + if (prog->type != BPF_PROG_TYPE_TRACING || + prog->expected_attach_type != BPF_TRACE_FENTRY || + !bpf_prog_is_dev_bound(prog->aux) || + !bpf_offload_dev_match(prog, netdev) || + strcmp(prog->aux->attach_func_name, attach_func_name)) { + bpf_prog_put(prog); + return -EINVAL; + } + + *pprog = prog; + static_branch_inc(&devtx_enabled); + + return ret; +} + +__diag_push(); +__diag_ignore_all("-Wmissing-prototypes", + "Global functions as their definitions will be in vmlinux BTF"); + +/** + * bpf_devtx_sb_attach - Attach devtx 'packet submit' program + * @ifindex: netdev interface index. + * @prog_fd: BPF program file descriptor. + * + * Return: + * * Returns 0 on success or ``-errno`` on error. + */ +__bpf_kfunc int bpf_devtx_sb_attach(int ifindex, int prog_fd) +{ + struct net_device *netdev; + int ret; + + netdev = dev_get_by_index(current->nsproxy->net_ns, ifindex); + if (!netdev) + return -EINVAL; + + mutex_lock(&devtx_attach_lock); + ret = __bpf_devtx_attach(netdev, prog_fd, "devtx_sb", &netdev->devtx_sb); + mutex_unlock(&devtx_attach_lock); + + dev_put(netdev); + + return ret; +} + +/** + * bpf_devtx_cp_attach - Attach devtx 'packet complete' program + * @ifindex: netdev interface index. + * @prog_fd: BPF program file descriptor. + * + * Return: + * * Returns 0 on success or ``-errno`` on error. + */ +__bpf_kfunc int bpf_devtx_cp_attach(int ifindex, int prog_fd) +{ + struct net_device *netdev; + int ret; + + netdev = dev_get_by_index(current->nsproxy->net_ns, ifindex); + if (!netdev) + return -EINVAL; + + mutex_lock(&devtx_attach_lock); + ret = __bpf_devtx_attach(netdev, prog_fd, "devtx_cp", &netdev->devtx_cp); + mutex_unlock(&devtx_attach_lock); + + dev_put(netdev); + + return ret; +} + +__diag_pop(); + +bool is_devtx_kfunc(u32 kfunc_id) +{ + return !!btf_id_set8_contains(&bpf_devtx_hook_ids, kfunc_id); +} + +void devtx_shutdown(struct net_device *netdev) +{ + mutex_lock(&devtx_attach_lock); + __bpf_devtx_detach(netdev, &netdev->devtx_sb); + __bpf_devtx_detach(netdev, &netdev->devtx_cp); + mutex_unlock(&devtx_attach_lock); +} + +BTF_SET8_START(bpf_devtx_syscall_kfunc_ids) +BTF_ID_FLAGS(func, bpf_devtx_sb_attach) +BTF_ID_FLAGS(func, bpf_devtx_cp_attach) +BTF_SET8_END(bpf_devtx_syscall_kfunc_ids) + +static const struct btf_kfunc_id_set bpf_devtx_syscall_kfunc_set = { + .owner = THIS_MODULE, + .set = &bpf_devtx_syscall_kfunc_ids, +}; + +static int __init devtx_init(void) +{ + int ret; + + ret = register_btf_fmodret_id_set(&bpf_devtx_hook_set); + if (ret) { + pr_warn("failed to register devtx hooks: %d", ret); + return ret; + } + + ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &bpf_devtx_syscall_kfunc_set); + if (ret) { + pr_warn("failed to register syscall kfuncs: %d", ret); + return ret; + } + + return 0; +} +late_initcall(devtx_init); From patchwork Mon Jun 12 17:23:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13277004 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 82C7421CD7 for ; Mon, 12 Jun 2023 17:23:17 +0000 (UTC) Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2C1DA188 for ; Mon, 12 Jun 2023 10:23:16 -0700 (PDT) Received: by mail-pf1-x449.google.com with SMTP id d2e1a72fcca58-653c16b3093so2590210b3a.3 for ; Mon, 12 Jun 2023 10:23:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686590595; x=1689182595; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PIJi+YaipsBqXx9Jn519PmEfcm6UsBy0hXGpFTQSxa0=; b=Q9gQPtZnpP4Lg8G/Hfh00v+xgId8RFOBZEXFS9J93Xo2PWq2Zamz/qyXckGdQp0T91 gzisalXdK2EqtU865ZpLTAPZnQeAdrnzNSjqPSMwTX2gWd1CISUffMMCksHe5D1A9YU6 B0Dt7aIIBU7eFDqcXxNk6OpbeCAyu0VohmXMKmqrsJvJggYTykORnZzX11+bEhnazrhZ w5baAROukc7pEV63XRoruQrsk9+e+KLmTZK0uednJonBHoTahirMrRY3fwd3qDIEsJt2 MdBesAIm20GZgrCgHtzC7MQLw8ymZ0eYL6Cx50Xt9ufJRnNfQEHoWYLjV97RF1cz52gh U/jw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686590595; x=1689182595; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PIJi+YaipsBqXx9Jn519PmEfcm6UsBy0hXGpFTQSxa0=; b=kOx+shwK5SDnW084/ebiGHWpJCGTxOGgNC6NlLvayy0gGhypP2eeg0vNmDAgBxcNy9 CIqEZ8z1SyKSddCGtuCiEMen+RVnMSy4SkxugE0eSALirCt85drX3Nz2KQuF8bmh/5q2 CXmW1AKWRrkI+ZQwsZjukiD1McbCmHb1a/S5lSNwQQ9DbI+fwJn2bQqTVLl4nGMcQ/gf v/5E39aY4F/AZMjO5cvtt/wyQnRruFoGWxHxSuV0nTIdqaRWHCmfZkgjvk20hM+ZTVNu s+FfTWJTsGUWPE/N8V4uCnkAdDDWTW+LuEFlJ3ki9BPSx5x83bAspGMBWRL0CIsbyAn5 17+w== X-Gm-Message-State: AC+VfDw0Gk8ZCafWgM0ATDu0xqpwpEmxYePWAnUC1X5y1D1lTLPCWH22 vVU+O8EE425TDwmnPba7StBrFCJJECSQSMrtrOgLWXuRftvSc+9YuU516ww1rxilHhJIiWRbSQp eLDWZmmF3mEAnZBA0lOPuPOAiHxc4a02SxzEETTMn8LFUoE9+ew== X-Google-Smtp-Source: ACHHUZ5OvXDyWAez+dzRc+UAzo44mIDfjoWrK4ycPztDNKvr+BddKBvufx4U9eLykvv4HGyjeajcvQU= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a05:6a00:18a1:b0:654:8eb9:4607 with SMTP id x33-20020a056a0018a100b006548eb94607mr3157117pfh.4.1686590595480; Mon, 12 Jun 2023 10:23:15 -0700 (PDT) Date: Mon, 12 Jun 2023 10:23:04 -0700 In-Reply-To: <20230612172307.3923165-1-sdf@google.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20230612172307.3923165-1-sdf@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230612172307.3923165-5-sdf@google.com> Subject: [RFC bpf-next 4/7] bpf: implement devtx timestamp kfunc From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, netdev@vger.kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Two kfuncs, one per hook point: 1. at submit time - bpf_devtx_sb_request_timestamp - to request HW to put TX timestamp into TX completion descriptors 2. at completion time - bpf_devtx_cp_timestamp - to read out TX timestamp Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- include/linux/netdevice.h | 4 +++ include/net/offload.h | 10 +++++++ kernel/bpf/offload.c | 8 ++++++ net/core/devtx.c | 58 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 80 insertions(+) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index e08e3fd39dfc..6e42e62fd1bc 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -1651,10 +1651,14 @@ struct net_device_ops { bool cycles); }; +struct devtx_frame; + struct xdp_metadata_ops { int (*xmo_rx_timestamp)(const struct xdp_md *ctx, u64 *timestamp); int (*xmo_rx_hash)(const struct xdp_md *ctx, u32 *hash, enum xdp_rss_hash_type *rss_type); + int (*xmo_sb_request_timestamp)(const struct devtx_frame *ctx); + int (*xmo_cp_timestamp)(const struct devtx_frame *ctx, u64 *timestamp); }; /** diff --git a/include/net/offload.h b/include/net/offload.h index 264a35881473..36899b64f4c8 100644 --- a/include/net/offload.h +++ b/include/net/offload.h @@ -10,9 +10,19 @@ NETDEV_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_HASH, \ bpf_xdp_metadata_rx_hash) +#define DEVTX_SB_KFUNC_xxx \ + NETDEV_METADATA_KFUNC(DEVTX_SB_KFUNC_REQUEST_TIMESTAMP, \ + bpf_devtx_sb_request_timestamp) + +#define DEVTX_CP_KFUNC_xxx \ + NETDEV_METADATA_KFUNC(DEVTX_CP_KFUNC_TIMESTAMP, \ + bpf_devtx_cp_timestamp) + enum { #define NETDEV_METADATA_KFUNC(name, _) name, XDP_METADATA_KFUNC_xxx +DEVTX_SB_KFUNC_xxx +DEVTX_CP_KFUNC_xxx #undef NETDEV_METADATA_KFUNC MAX_NETDEV_METADATA_KFUNC, }; diff --git a/kernel/bpf/offload.c b/kernel/bpf/offload.c index 9cfe96422c80..91dc7b7e3684 100644 --- a/kernel/bpf/offload.c +++ b/kernel/bpf/offload.c @@ -854,6 +854,10 @@ void *bpf_dev_bound_resolve_kfunc(struct bpf_prog *prog, u32 func_id) p = ops->xmo_rx_timestamp; else if (func_id == bpf_dev_bound_kfunc_id(XDP_METADATA_KFUNC_RX_HASH)) p = ops->xmo_rx_hash; + else if (func_id == bpf_dev_bound_kfunc_id(DEVTX_SB_KFUNC_REQUEST_TIMESTAMP)) + p = ops->xmo_sb_request_timestamp; + else if (func_id == bpf_dev_bound_kfunc_id(DEVTX_CP_KFUNC_TIMESTAMP)) + p = ops->xmo_cp_timestamp; out: up_read(&bpf_devs_lock); @@ -863,12 +867,16 @@ void *bpf_dev_bound_resolve_kfunc(struct bpf_prog *prog, u32 func_id) BTF_SET_START(dev_bound_kfunc_ids) #define NETDEV_METADATA_KFUNC(name, str) BTF_ID(func, str) XDP_METADATA_KFUNC_xxx +DEVTX_SB_KFUNC_xxx +DEVTX_CP_KFUNC_xxx #undef NETDEV_METADATA_KFUNC BTF_SET_END(dev_bound_kfunc_ids) BTF_ID_LIST(dev_bound_kfunc_ids_unsorted) #define NETDEV_METADATA_KFUNC(name, str) BTF_ID(func, str) XDP_METADATA_KFUNC_xxx +DEVTX_SB_KFUNC_xxx +DEVTX_CP_KFUNC_xxx #undef NETDEV_METADATA_KFUNC u32 bpf_dev_bound_kfunc_id(int id) diff --git a/net/core/devtx.c b/net/core/devtx.c index b7cbc26d1c01..2b37c31f0912 100644 --- a/net/core/devtx.c +++ b/net/core/devtx.c @@ -162,6 +162,30 @@ __bpf_kfunc int bpf_devtx_cp_attach(int ifindex, int prog_fd) return ret; } +/** + * bpf_devtx_sb_request_timestamp - Request TX timestamp on the packet. + * Callable only from the devtx-submit hook. + * @ctx: devtx context pointer. + * + * Returns 0 on success or ``-errno`` on error. + */ +__bpf_kfunc int bpf_devtx_sb_request_timestamp(const struct devtx_frame *ctx) +{ + return -EOPNOTSUPP; +} + +/** + * bpf_devtx_cp_timestamp - Read TX timestamp of the packet. Callable + * only from the devtx-complete hook. + * @ctx: devtx context pointer. + * + * Returns 0 on success or ``-errno`` on error. + */ +__bpf_kfunc int bpf_devtx_cp_timestamp(const struct devtx_frame *ctx, __u64 *timestamp) +{ + return -EOPNOTSUPP; +} + __diag_pop(); bool is_devtx_kfunc(u32 kfunc_id) @@ -187,6 +211,28 @@ static const struct btf_kfunc_id_set bpf_devtx_syscall_kfunc_set = { .set = &bpf_devtx_syscall_kfunc_ids, }; +BTF_SET8_START(devtx_sb_kfunc_ids) +#define NETDEV_METADATA_KFUNC(_, name) BTF_ID_FLAGS(func, name, 0) +DEVTX_SB_KFUNC_xxx +#undef NETDEV_METADATA_KFUNC +BTF_SET8_END(devtx_sb_kfunc_ids) + +static const struct btf_kfunc_id_set devtx_sb_kfunc_set = { + .owner = THIS_MODULE, + .set = &devtx_sb_kfunc_ids, +}; + +BTF_SET8_START(devtx_cp_kfunc_ids) +#define NETDEV_METADATA_KFUNC(_, name) BTF_ID_FLAGS(func, name, 0) +DEVTX_CP_KFUNC_xxx +#undef NETDEV_METADATA_KFUNC +BTF_SET8_END(devtx_cp_kfunc_ids) + +static const struct btf_kfunc_id_set devtx_cp_kfunc_set = { + .owner = THIS_MODULE, + .set = &devtx_cp_kfunc_ids, +}; + static int __init devtx_init(void) { int ret; @@ -197,6 +243,18 @@ static int __init devtx_init(void) return ret; } + ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &devtx_sb_kfunc_set); + if (ret) { + pr_warn("failed to register devtx_sb kfuncs: %d", ret); + return ret; + } + + ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &devtx_cp_kfunc_set); + if (ret) { + pr_warn("failed to register devtx_cp completion kfuncs: %d", ret); + return ret; + } + ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &bpf_devtx_syscall_kfunc_set); if (ret) { pr_warn("failed to register syscall kfuncs: %d", ret); From patchwork Mon Jun 12 17:23:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13277005 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C697E22D5C for ; Mon, 12 Jun 2023 17:23:19 +0000 (UTC) Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C2762DB for ; Mon, 12 Jun 2023 10:23:17 -0700 (PDT) Received: by mail-pf1-x449.google.com with SMTP id d2e1a72fcca58-64f74f4578aso4531989b3a.3 for ; Mon, 12 Jun 2023 10:23:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686590597; x=1689182597; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=oDuv+g/0XRvJo0yG4qkvO5UlPZUnGZfMWyMTGOYfexg=; b=2K3cU329VMyFIb3jm2NC1xMwCoV2p3eCj99k6BZoAsFQrOfokA5YcXp2v7ocWEYoEl lUQF2jKvE6TPLRun9eXJDCg0ehSxuyMu3DGCRDNtgURj3w2Ep6OU31amOgGSwYhrOX52 FeiX1Bf42TrsOSeKlyTE8XTB+QWabKcMcAMQIVLp4oZA9lGnT5edv84OqLxV2jTJGqYW wexUVL7n/xzWWTzRkxVvUNCDmBEhn60i2ysfjgSvLh1JlEfoTuKAVu4N5mKEmGLCXgvd W8x6kvqZLNLVdQvvac92GZTt5nxmpBycZZL6YmhvoK+ifF2oLu3pVhBfpicxuEvb/4/0 LLDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686590597; x=1689182597; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=oDuv+g/0XRvJo0yG4qkvO5UlPZUnGZfMWyMTGOYfexg=; b=fIrOTnojab6pD2fDAGGUGNbqV1eDLrxuRTDgsNa1x848NlIp5k6KW7iqp3aPcObAhz bQCs10xvwFvPBeWxl+FgMtKWiAHhqqjedDBym2VQxSAm0bK6f/RXYdqRkvY3UtRh2dIF dCd8j52Vo6Nr3R5z4Dp8ixM95rOZkZkIjKjvatGf7NenvRpsL1L1MI8NaymbC1Dc8M9Z 3RwrU9fOiCtQxkiF2rVfMeCiklzIiFaqUB93CqKbXXEp/aJ2b51LRXs3F6IlBsv8m/Gl e1Fvelp657SMAXlUYlkryLfFeVNNvxhNwu2T3nfs16+cWndL/OZnxs2ylXRymHQ0uRT2 rgAQ== X-Gm-Message-State: AC+VfDyWv5aziNEMoGPdjVEPcs0qT6E+iXsmTn2s04AsYyYqCtlv9o25 ZDJW89d+QJ2b5Nxpcc0AytwAPRnQVt8jgsShSPy/0KXiURuDZbsOh0e694W+cBYT3kmXG3G34tV MZIf/v5vmgO9mR/M8AX+plO22/zcKxlFODjex4Cj7f6zv1QtAKQ== X-Google-Smtp-Source: ACHHUZ6oj9zIPCtyBLcuQ6c6H2lhHZgxEY+zCP6QyTdZNpxSiXXcWElW3XWanZwS51/DQOOrLzvVvqY= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a05:6a00:2353:b0:653:9883:40ec with SMTP id j19-20020a056a00235300b00653988340ecmr3037128pfj.5.1686590597062; Mon, 12 Jun 2023 10:23:17 -0700 (PDT) Date: Mon, 12 Jun 2023 10:23:05 -0700 In-Reply-To: <20230612172307.3923165-1-sdf@google.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20230612172307.3923165-1-sdf@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230612172307.3923165-6-sdf@google.com> Subject: [RFC bpf-next 5/7] net: veth: implement devtx timestamp kfuncs From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, netdev@vger.kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Have a software-base example for kfuncs to showcase how it can be used in the real devices and to have something to test against in the selftests. Both path (skb & xdp) are covered. Only the skb path is really tested though. Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- drivers/net/veth.c | 94 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 90 insertions(+), 4 deletions(-) diff --git a/drivers/net/veth.c b/drivers/net/veth.c index 614f3e3efab0..eb78d51d8352 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -27,6 +27,7 @@ #include #include #include +#include #define DRV_NAME "veth" #define DRV_VERSION "1.0" @@ -123,6 +124,13 @@ struct veth_xdp_buff { struct sk_buff *skb; }; +struct veth_devtx_frame { + struct devtx_frame frame; + bool request_timestamp; + ktime_t xdp_tx_timestamp; + struct sk_buff *skb; +}; + static int veth_get_link_ksettings(struct net_device *dev, struct ethtool_link_ksettings *cmd) { @@ -314,9 +322,29 @@ static int veth_xdp_rx(struct veth_rq *rq, struct sk_buff *skb) } static int veth_forward_skb(struct net_device *dev, struct sk_buff *skb, - struct veth_rq *rq, bool xdp) + struct veth_rq *rq, bool xdp, bool request_timestamp) { - return __dev_forward_skb(dev, skb) ?: xdp ? + struct net_device *src_dev = skb->dev; + int ret; + + ret = __dev_forward_skb(dev, skb); + if (ret) + return ret; + + if (devtx_complete_enabled(src_dev)) { + struct veth_devtx_frame ctx; + + if (unlikely(request_timestamp)) + __net_timestamp(skb); + + devtx_frame_from_skb(&ctx.frame, skb); + ctx.frame.data -= ETH_HLEN; /* undo eth_type_trans pull */ + ctx.frame.len += ETH_HLEN; + ctx.skb = skb; + devtx_complete(src_dev, &ctx.frame); + } + + return xdp ? veth_xdp_rx(rq, skb) : __netif_rx(skb); } @@ -343,6 +371,7 @@ static bool veth_skb_is_eligible_for_gro(const struct net_device *dev, static netdev_tx_t veth_xmit(struct sk_buff *skb, struct net_device *dev) { struct veth_priv *rcv_priv, *priv = netdev_priv(dev); + bool request_timestamp = false; struct veth_rq *rq = NULL; struct net_device *rcv; int length = skb->len; @@ -356,6 +385,15 @@ static netdev_tx_t veth_xmit(struct sk_buff *skb, struct net_device *dev) goto drop; } + if (devtx_submit_enabled(dev)) { + struct veth_devtx_frame ctx; + + devtx_frame_from_skb(&ctx.frame, skb); + ctx.request_timestamp = false; + devtx_submit(dev, &ctx.frame); + request_timestamp = ctx.request_timestamp; + } + rcv_priv = netdev_priv(rcv); rxq = skb_get_queue_mapping(skb); if (rxq < rcv->real_num_rx_queues) { @@ -370,7 +408,7 @@ static netdev_tx_t veth_xmit(struct sk_buff *skb, struct net_device *dev) } skb_tx_timestamp(skb); - if (likely(veth_forward_skb(rcv, skb, rq, use_napi) == NET_RX_SUCCESS)) { + if (likely(veth_forward_skb(rcv, skb, rq, use_napi, request_timestamp) == NET_RX_SUCCESS)) { if (!use_napi) dev_lstats_add(dev, length); } else { @@ -483,6 +521,7 @@ static int veth_xdp_xmit(struct net_device *dev, int n, { struct veth_priv *rcv_priv, *priv = netdev_priv(dev); int i, ret = -ENXIO, nxmit = 0; + ktime_t tx_timestamp = 0; struct net_device *rcv; unsigned int max_len; struct veth_rq *rq; @@ -511,9 +550,32 @@ static int veth_xdp_xmit(struct net_device *dev, int n, void *ptr = veth_xdp_to_ptr(frame); if (unlikely(xdp_get_frame_len(frame) > max_len || - __ptr_ring_produce(&rq->xdp_ring, ptr))) + __ptr_ring_full(&rq->xdp_ring))) + break; + + if (devtx_submit_enabled(dev)) { + struct veth_devtx_frame ctx; + + devtx_frame_from_xdp(&ctx.frame, frame); + ctx.request_timestamp = false; + devtx_submit(dev, &ctx.frame); + + if (unlikely(ctx.request_timestamp)) + tx_timestamp = ktime_get_real(); + } + + if (unlikely(__ptr_ring_produce(&rq->xdp_ring, ptr))) break; nxmit++; + + if (devtx_complete_enabled(dev)) { + struct veth_devtx_frame ctx; + + devtx_frame_from_xdp(&ctx.frame, frame); + ctx.xdp_tx_timestamp = tx_timestamp; + ctx.skb = NULL; + devtx_complete(dev, &ctx.frame); + } } spin_unlock(&rq->xdp_ring.producer_lock); @@ -1732,6 +1794,28 @@ static int veth_xdp_rx_hash(const struct xdp_md *ctx, u32 *hash, return 0; } +static int veth_devtx_sb_request_timestamp(const struct devtx_frame *_ctx) +{ + struct veth_devtx_frame *ctx = (struct veth_devtx_frame *)_ctx; + + ctx->request_timestamp = true; + + return 0; +} + +static int veth_devtx_cp_timestamp(const struct devtx_frame *_ctx, u64 *timestamp) +{ + struct veth_devtx_frame *ctx = (struct veth_devtx_frame *)_ctx; + + if (ctx->skb) { + *timestamp = ctx->skb->tstamp; + return 0; + } + + *timestamp = ctx->xdp_tx_timestamp; + return 0; +} + static const struct net_device_ops veth_netdev_ops = { .ndo_init = veth_dev_init, .ndo_open = veth_open, @@ -1756,6 +1840,8 @@ static const struct net_device_ops veth_netdev_ops = { static const struct xdp_metadata_ops veth_xdp_metadata_ops = { .xmo_rx_timestamp = veth_xdp_rx_timestamp, .xmo_rx_hash = veth_xdp_rx_hash, + .xmo_sb_request_timestamp = veth_devtx_sb_request_timestamp, + .xmo_cp_timestamp = veth_devtx_cp_timestamp, }; #define VETH_FEATURES (NETIF_F_SG | NETIF_F_FRAGLIST | NETIF_F_HW_CSUM | \ From patchwork Mon Jun 12 17:23:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13277006 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3F5AE22D5A for ; Mon, 12 Jun 2023 17:23:21 +0000 (UTC) Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 831D210D8 for ; Mon, 12 Jun 2023 10:23:19 -0700 (PDT) Received: by mail-pf1-x44a.google.com with SMTP id d2e1a72fcca58-6540e7d7db6so2830926b3a.1 for ; Mon, 12 Jun 2023 10:23:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686590599; x=1689182599; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=EzTWWwhyhMVAvehkvr05ND69njUUlkEoFhtpXawONGA=; b=N+Q3gS9PnTA6vkhXJ5pIx7qDXSbWCamAVwfIzEIOGBYXW1OK6c9nSmT4KBM4dJHiic 0a3eE9bvll7CjD3pyuYdvBEvTA+auNUoK9FUbpnmSdoSbsrGaNBFrKyFDrmavtM3qxPd 9IScRF31hW1M1NAmh8Vi188zYZiJuTa+nPcAPvEBqfpqelqsi+zEE2RhqsZFzWbuDu/J 7bJ1kXRi9cv48RDQDMxINDf7SoNfZ9cHIsdRTA1Pg9X9NPMR+ZELYmcS4eYMiGRkHuT8 E9NvlJT8muNHg/e2iSvVOFsObtbDXZu7aHCR4K6NY8laEg/bpol2VtYutKf48K5kAmxp hE3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686590599; x=1689182599; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=EzTWWwhyhMVAvehkvr05ND69njUUlkEoFhtpXawONGA=; b=HLHZzRZ6YFSlBA/lLiswe+EdYQaQuyiyJTqEbsU8LmdYvu+f+R1zSXDx6Sfnw91xtk DhDpLumLYViz0Kscvu2tuLpLS94eLCCchDY/zFlpkbte8ES2FIWyFYzd7XU+vc31Mcqj Kx9yrV/PPSDW26kYHu+XXwD3w9HTy/DhNmnyAPXbmJYcybAwePUlor03HlU8cu6DJlnu zjrmDSz7dxH8JvN031se7STjqxbUqmzkBw7otuIUybxdHlP37mTs8wW4VjP+dVLP6cKM qH1gmqu9r6BUmiQ5gcCFJCu9yAMm2KDA+nNBEHq3nwIxep3UvqkbjACzOIxdB8OpRz+C oLYw== X-Gm-Message-State: AC+VfDyYN0fHBQu8KUoAjPH2JGGa2F/+wYk1uDJbQMZCwnyMvXgEdXOh ce4CcGITqvmfPf/j+WO3Uh2o4gsj4XBKPafaZlOcBBrUid1bSXBsKzGiQs1y8OBr7xDB/xsww+8 Xwrj4idvnOly11t+BXrvRHA21VR7fSW7CGBMZa4xzkIpgAinCuA== X-Google-Smtp-Source: ACHHUZ68oYd1EKaptDeVMbxut9jn71wqbcYY0bIC7n4FliseC87SSjP9dZbRuhTlPGmzMYcex1tpjng= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a05:6a00:2e97:b0:665:b0bb:3f17 with SMTP id fd23-20020a056a002e9700b00665b0bb3f17mr422001pfb.1.1686590599002; Mon, 12 Jun 2023 10:23:19 -0700 (PDT) Date: Mon, 12 Jun 2023 10:23:06 -0700 In-Reply-To: <20230612172307.3923165-1-sdf@google.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20230612172307.3923165-1-sdf@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230612172307.3923165-7-sdf@google.com> Subject: [RFC bpf-next 6/7] selftests/bpf: extend xdp_metadata with devtx kfuncs From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, netdev@vger.kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Attach kfuncs that request and report TX timestamp via ringbuf. Confirm on the userspace side that the program has triggered and the timestamp is non-zero. Also make sure devtx_frame has a sensible pointers and data. Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- .../selftests/bpf/prog_tests/xdp_metadata.c | 82 +++++++++++++- .../selftests/bpf/progs/xdp_metadata.c | 101 ++++++++++++++++++ tools/testing/selftests/bpf/xdp_metadata.h | 13 +++ 3 files changed, 194 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/xdp_metadata.c b/tools/testing/selftests/bpf/prog_tests/xdp_metadata.c index 626c461fa34d..ebaa50293f85 100644 --- a/tools/testing/selftests/bpf/prog_tests/xdp_metadata.c +++ b/tools/testing/selftests/bpf/prog_tests/xdp_metadata.c @@ -42,6 +42,9 @@ struct xsk { struct xsk_ring_prod tx; struct xsk_ring_cons rx; struct xsk_socket *socket; + int tx_completions; + u32 last_tx_timestamp_retval; + u64 last_tx_timestamp; }; static int open_xsk(int ifindex, struct xsk *xsk) @@ -192,7 +195,8 @@ static int generate_packet(struct xsk *xsk, __u16 dst_port) return 0; } -static void complete_tx(struct xsk *xsk) +static void complete_tx(struct xsk *xsk, struct xdp_metadata *bpf_obj, + struct ring_buffer *ringbuf) { __u32 idx; __u64 addr; @@ -202,6 +206,13 @@ static void complete_tx(struct xsk *xsk) printf("%p: complete tx idx=%u addr=%llx\n", xsk, idx, addr); xsk_ring_cons__release(&xsk->comp, 1); + + ring_buffer__poll(ringbuf, 1000); + + ASSERT_EQ(bpf_obj->bss->pkts_fail_tx, 0, "pkts_fail_tx"); + ASSERT_GE(xsk->tx_completions, 1, "tx_completions"); + ASSERT_EQ(xsk->last_tx_timestamp_retval, 0, "last_tx_timestamp_retval"); + ASSERT_GE(xsk->last_tx_timestamp, 0, "last_tx_timestamp"); } } @@ -276,8 +287,24 @@ static int verify_xsk_metadata(struct xsk *xsk) return 0; } +static int process_sample(void *ctx, void *data, size_t len) +{ + struct devtx_sample *sample = data; + struct xsk *xsk = ctx; + + printf("%p: got tx timestamp sample %u %llu\n", + xsk, sample->timestamp_retval, sample->timestamp); + + xsk->tx_completions++; + xsk->last_tx_timestamp_retval = sample->timestamp_retval; + xsk->last_tx_timestamp = sample->timestamp; + + return 0; +} + void test_xdp_metadata(void) { + struct ring_buffer *tx_compl_ringbuf = NULL; struct xdp_metadata2 *bpf_obj2 = NULL; struct xdp_metadata *bpf_obj = NULL; struct bpf_program *new_prog, *prog; @@ -290,6 +317,7 @@ void test_xdp_metadata(void) int retries = 10; int rx_ifindex; int tx_ifindex; + int syscall_fd; int sock_fd; int ret; @@ -323,6 +351,16 @@ void test_xdp_metadata(void) if (!ASSERT_OK_PTR(bpf_obj, "open skeleton")) goto out; + prog = bpf_object__find_program_by_name(bpf_obj->obj, "devtx_sb"); + bpf_program__set_ifindex(prog, tx_ifindex); + bpf_program__set_flags(prog, BPF_F_XDP_DEV_BOUND_ONLY); + bpf_program__set_autoattach(prog, false); + + prog = bpf_object__find_program_by_name(bpf_obj->obj, "devtx_cp"); + bpf_program__set_ifindex(prog, tx_ifindex); + bpf_program__set_flags(prog, BPF_F_XDP_DEV_BOUND_ONLY); + bpf_program__set_autoattach(prog, false); + prog = bpf_object__find_program_by_name(bpf_obj->obj, "rx"); bpf_program__set_ifindex(prog, rx_ifindex); bpf_program__set_flags(prog, BPF_F_XDP_DEV_BOUND_ONLY); @@ -330,6 +368,15 @@ void test_xdp_metadata(void) if (!ASSERT_OK(xdp_metadata__load(bpf_obj), "load skeleton")) goto out; + ret = xdp_metadata__attach(bpf_obj); + if (!ASSERT_OK(ret, "xdp_metadata__attach")) + goto out; + + tx_compl_ringbuf = ring_buffer__new(bpf_map__fd(bpf_obj->maps.tx_compl_buf), + process_sample, &tx_xsk, NULL); + if (!ASSERT_OK_PTR(tx_compl_ringbuf, "ring_buffer__new")) + goto out; + /* Make sure we can't add dev-bound programs to prog maps. */ prog_arr = bpf_object__find_map_by_name(bpf_obj->obj, "prog_arr"); if (!ASSERT_OK_PTR(prog_arr, "no prog_arr map")) @@ -341,6 +388,26 @@ void test_xdp_metadata(void) "update prog_arr")) goto out; + /* Attach egress BPF programs to interface. */ + struct devtx_attach_args args = { + .ifindex = tx_ifindex, + .devtx_sb_prog_fd = bpf_program__fd(bpf_obj->progs.devtx_sb), + .devtx_cp_prog_fd = bpf_program__fd(bpf_obj->progs.devtx_cp), + .devtx_sb_retval = -1, + .devtx_cp_retval = -1, + }; + DECLARE_LIBBPF_OPTS(bpf_test_run_opts, tattr, + .ctx_in = &args, + .ctx_size_in = sizeof(args), + ); + + syscall_fd = bpf_program__fd(bpf_obj->progs.attach_prog); + ret = bpf_prog_test_run_opts(syscall_fd, &tattr); + if (!ASSERT_GE(ret, 0, "bpf_prog_test_run_opts(attach_prog)")) + goto out; + ASSERT_GE(args.devtx_sb_retval, 0, "bpf_prog_test_run_opts(attach_prog) devtx_sb_retval"); + ASSERT_GE(args.devtx_cp_retval, 0, "bpf_prog_test_run_opts(attach_prog) devtx_cp_retval"); + /* Attach BPF program to RX interface. */ ret = bpf_xdp_attach(rx_ifindex, @@ -364,7 +431,16 @@ void test_xdp_metadata(void) "verify_xsk_metadata")) goto out; - complete_tx(&tx_xsk); + /* Verify AF_XDP TX packet has completion event with a timestamp. */ + complete_tx(&tx_xsk, bpf_obj, tx_compl_ringbuf); + + /* Detach egress program. */ + syscall_fd = bpf_program__fd(bpf_obj->progs.detach_prog); + ret = bpf_prog_test_run_opts(syscall_fd, &tattr); + if (!ASSERT_GE(ret, 0, "bpf_prog_test_run_opts(detach_prog)")) + goto out; + ASSERT_GE(args.devtx_sb_retval, 0, "bpf_prog_test_run_opts(detach_prog) devtx_sb_retval"); + ASSERT_GE(args.devtx_cp_retval, 0, "bpf_prog_test_run_opts(detach_prog) devtx_cp_retval"); /* Make sure freplace correctly picks up original bound device * and doesn't crash. @@ -402,5 +478,7 @@ void test_xdp_metadata(void) xdp_metadata__destroy(bpf_obj); if (tok) close_netns(tok); + if (tx_compl_ringbuf) + ring_buffer__free(tx_compl_ringbuf); SYS_NOFAIL("ip netns del xdp_metadata"); } diff --git a/tools/testing/selftests/bpf/progs/xdp_metadata.c b/tools/testing/selftests/bpf/progs/xdp_metadata.c index d151d406a123..5b815bd03fe7 100644 --- a/tools/testing/selftests/bpf/progs/xdp_metadata.c +++ b/tools/testing/selftests/bpf/progs/xdp_metadata.c @@ -4,6 +4,11 @@ #include "xdp_metadata.h" #include #include +#include + +#ifndef ETH_P_IP +#define ETH_P_IP 0x0800 +#endif struct { __uint(type, BPF_MAP_TYPE_XSKMAP); @@ -19,10 +24,22 @@ struct { __type(value, __u32); } prog_arr SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_RINGBUF); + __uint(max_entries, 10); +} tx_compl_buf SEC(".maps"); + +__u64 pkts_fail_tx = 0; + extern int bpf_xdp_metadata_rx_timestamp(const struct xdp_md *ctx, __u64 *timestamp) __ksym; extern int bpf_xdp_metadata_rx_hash(const struct xdp_md *ctx, __u32 *hash, enum xdp_rss_hash_type *rss_type) __ksym; +extern int bpf_devtx_sb_request_timestamp(const struct devtx_frame *ctx) __ksym; +extern int bpf_devtx_cp_timestamp(const struct devtx_frame *ctx, __u64 *timestamp) __ksym; + +extern int bpf_devtx_sb_attach(int ifindex, int prog_fd) __ksym; +extern int bpf_devtx_cp_attach(int ifindex, int prog_fd) __ksym; SEC("xdp") int rx(struct xdp_md *ctx) @@ -61,4 +78,88 @@ int rx(struct xdp_md *ctx) return bpf_redirect_map(&xsk, ctx->rx_queue_index, XDP_PASS); } +static inline int verify_frame(const struct devtx_frame *frame) +{ + struct ethhdr eth = {}; + + /* all the pointers are set up correctly */ + if (!frame->data) + return -1; + if (!frame->sinfo) + return -1; + + /* can get to the frags */ + if (frame->sinfo->nr_frags != 0) + return -1; + if (frame->sinfo->frags[0].bv_page != 0) + return -1; + if (frame->sinfo->frags[0].bv_len != 0) + return -1; + if (frame->sinfo->frags[0].bv_offset != 0) + return -1; + + /* the data has something that looks like ethernet */ + if (frame->len != 46) + return -1; + bpf_probe_read_kernel(ð, sizeof(eth), frame->data); + + if (eth.h_proto != bpf_htons(ETH_P_IP)) + return -1; + + return 0; +} + +SEC("fentry/devtx_sb") +int BPF_PROG(devtx_sb, const struct devtx_frame *frame) +{ + int ret; + + ret = verify_frame(frame); + if (ret < 0) + __sync_add_and_fetch(&pkts_fail_tx, 1); + + ret = bpf_devtx_sb_request_timestamp(frame); + if (ret < 0) + __sync_add_and_fetch(&pkts_fail_tx, 1); + + return 0; +} + +SEC("fentry/devtx_cp") +int BPF_PROG(devtx_cp, const struct devtx_frame *frame) +{ + struct devtx_sample *sample; + int ret; + + ret = verify_frame(frame); + if (ret < 0) + __sync_add_and_fetch(&pkts_fail_tx, 1); + + sample = bpf_ringbuf_reserve(&tx_compl_buf, sizeof(*sample), 0); + if (!sample) + return 0; + + sample->timestamp_retval = bpf_devtx_cp_timestamp(frame, &sample->timestamp); + + bpf_ringbuf_submit(sample, 0); + + return 0; +} + +SEC("syscall") +int attach_prog(struct devtx_attach_args *ctx) +{ + ctx->devtx_sb_retval = bpf_devtx_sb_attach(ctx->ifindex, ctx->devtx_sb_prog_fd); + ctx->devtx_cp_retval = bpf_devtx_cp_attach(ctx->ifindex, ctx->devtx_cp_prog_fd); + return 0; +} + +SEC("syscall") +int detach_prog(struct devtx_attach_args *ctx) +{ + ctx->devtx_sb_retval = bpf_devtx_sb_attach(ctx->ifindex, -1); + ctx->devtx_cp_retval = bpf_devtx_cp_attach(ctx->ifindex, -1); + return 0; +} + char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/xdp_metadata.h b/tools/testing/selftests/bpf/xdp_metadata.h index 938a729bd307..add900c9035c 100644 --- a/tools/testing/selftests/bpf/xdp_metadata.h +++ b/tools/testing/selftests/bpf/xdp_metadata.h @@ -18,3 +18,16 @@ struct xdp_meta { __s32 rx_hash_err; }; }; + +struct devtx_sample { + int timestamp_retval; + __u64 timestamp; +}; + +struct devtx_attach_args { + int ifindex; + int devtx_sb_prog_fd; + int devtx_cp_prog_fd; + int devtx_sb_retval; + int devtx_cp_retval; +}; From patchwork Mon Jun 12 17:23:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13277007 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D5D623413 for ; Mon, 12 Jun 2023 17:23:23 +0000 (UTC) Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 511A410B for ; Mon, 12 Jun 2023 10:23:21 -0700 (PDT) Received: by mail-pl1-x649.google.com with SMTP id d9443c01a7336-1b3c9fee2edso7572015ad.0 for ; Mon, 12 Jun 2023 10:23:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686590601; x=1689182601; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=QinrNp91VGgxDgv0vB00Zn5PtTuK217dsIffIBOl1/c=; b=tpsyRc4fZCPwXpocku5hFsl607m29uAQqHOzrqTQpofoq4OFbXUK7wpwapr8jYm1+f YWB0lEgfThR08hYmVpL6bZWrKED0Ej09CV05tLxbm5xO5JFuDFlqVq5rO+MimNvBcS2p lUX6tAfxpWwQrYeWyEUuF9+8pInMsU+sTYSkUWnu1rV7uThxlxSBlMF9zmmsyfn2Xwkl U0nM5CRlXO1qod166MqqU9YIRbLZ8l0DfZgTPNIi4o1M0nP7OoTDC1HXeERAn/Q+npkx pvWv9Ilg7rVPy4fwPuhyRXH1y//T+JAng3kvL+9Iel0vYbwwoHbqqojA5/Q2QBXuItgF bbGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686590601; x=1689182601; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QinrNp91VGgxDgv0vB00Zn5PtTuK217dsIffIBOl1/c=; b=A1Zi6FvEUbrSd5dBTAZH5TmiNWaCtLEE3mM4EVECgStDHwReiwEEb7d9RT1mcwZhCE E9w/9xkciVWZRihhwM+HdvAcVTExJEogxt5iCuWX96QmLbawMuYwedg06GJyIKaoqybi 4OJvs/aHL5vv1IknMnCXmmDlkjDlT24AsNMQr2+Jv8fdP82piKUgUMB1/H5ueUA8mQdK O5ynUy1ID7E/Xm/toX6Lp4EmHQmnjZIMStMx8NeuYq5dNtXi70K4iezkkSxs8c8Db2NA Uk2QvLELCyzd7W/OfwntMsiL9gQfjDV1QDSm0un0PumXi/0Nd8kRh7c2mBUiSkgJvOdF WgEQ== X-Gm-Message-State: AC+VfDw/AGiT3dTxVi6FVlupG0Kv/+RszdezJi+lnWBkuehs4V6u6yIK M4Cjukmf/tWgdrpf34WCUddzZQ2lSr7il7lQP3918hAEHhf7FOD9IwHJeDS+ME486Gls4UnCTu+ SKhbCBoZ0LeDSEXsGN1lGLtfmBnoI91WhljB+7yG5pwCenbonuQ== X-Google-Smtp-Source: ACHHUZ5aFSuCUC2xqESgbeK7GtiyMCRrjswwYNuL4ck19JhSF0kT5V2oYWDEHlWPGpmW2ImWcu17V1g= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a17:902:b18a:b0:1ae:50cc:457 with SMTP id s10-20020a170902b18a00b001ae50cc0457mr1314399plr.10.1686590600591; Mon, 12 Jun 2023 10:23:20 -0700 (PDT) Date: Mon, 12 Jun 2023 10:23:07 -0700 In-Reply-To: <20230612172307.3923165-1-sdf@google.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20230612172307.3923165-1-sdf@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230612172307.3923165-8-sdf@google.com> Subject: [RFC bpf-next 7/7] selftests/bpf: extend xdp_hw_metadata with devtx kfuncs From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, netdev@vger.kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC When we get packets on port 9091, we swap src/dst and send it out. At this point, we also request the timestamp and plumb it back to the userspace. The userspace simply prints the timestamp. Haven't really tested, still working on mlx5 patches... Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- .../selftests/bpf/progs/xdp_hw_metadata.c | 59 +++++++ tools/testing/selftests/bpf/xdp_hw_metadata.c | 160 +++++++++++++++++- 2 files changed, 214 insertions(+), 5 deletions(-) diff --git a/tools/testing/selftests/bpf/progs/xdp_hw_metadata.c b/tools/testing/selftests/bpf/progs/xdp_hw_metadata.c index b2dfd7066c6e..e27823b755ef 100644 --- a/tools/testing/selftests/bpf/progs/xdp_hw_metadata.c +++ b/tools/testing/selftests/bpf/progs/xdp_hw_metadata.c @@ -4,6 +4,7 @@ #include "xdp_metadata.h" #include #include +#include struct { __uint(type, BPF_MAP_TYPE_XSKMAP); @@ -12,14 +13,26 @@ struct { __type(value, __u32); } xsk SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_RINGBUF); + __uint(max_entries, 10); +} tx_compl_buf SEC(".maps"); + __u64 pkts_skip = 0; __u64 pkts_fail = 0; __u64 pkts_redir = 0; +__u64 pkts_fail_tx = 0; +__u64 pkts_ringbuf_full = 0; extern int bpf_xdp_metadata_rx_timestamp(const struct xdp_md *ctx, __u64 *timestamp) __ksym; extern int bpf_xdp_metadata_rx_hash(const struct xdp_md *ctx, __u32 *hash, enum xdp_rss_hash_type *rss_type) __ksym; +extern int bpf_devtx_sb_request_timestamp(const struct devtx_frame *ctx) __ksym; +extern int bpf_devtx_cp_timestamp(const struct devtx_frame *ctx, __u64 *timestamp) __ksym; + +extern int bpf_devtx_sb_attach(int ifindex, int prog_fd) __ksym; +extern int bpf_devtx_cp_attach(int ifindex, int prog_fd) __ksym; SEC("xdp") int rx(struct xdp_md *ctx) @@ -90,4 +103,50 @@ int rx(struct xdp_md *ctx) return bpf_redirect_map(&xsk, ctx->rx_queue_index, XDP_PASS); } +SEC("fentry/devtx_sb") +int BPF_PROG(devtx_sb, const struct devtx_frame *frame) +{ + int ret; + + ret = bpf_devtx_sb_request_timestamp(frame); + if (ret < 0) + __sync_add_and_fetch(&pkts_fail_tx, 1); + + return 0; +} + +SEC("fentry/devtx_cp") +int BPF_PROG(devtx_cp, const struct devtx_frame *frame) +{ + struct devtx_sample *sample; + + sample = bpf_ringbuf_reserve(&tx_compl_buf, sizeof(*sample), 0); + if (!sample) { + __sync_add_and_fetch(&pkts_ringbuf_full, 1); + return 0; + } + + sample->timestamp_retval = bpf_devtx_cp_timestamp(frame, &sample->timestamp); + + bpf_ringbuf_submit(sample, 0); + + return 0; +} + +SEC("syscall") +int attach_prog(struct devtx_attach_args *ctx) +{ + ctx->devtx_sb_retval = bpf_devtx_sb_attach(ctx->ifindex, ctx->devtx_sb_prog_fd); + ctx->devtx_cp_retval = bpf_devtx_cp_attach(ctx->ifindex, ctx->devtx_cp_prog_fd); + return 0; +} + +SEC("syscall") +int detach_prog(struct devtx_attach_args *ctx) +{ + ctx->devtx_sb_retval = bpf_devtx_sb_attach(ctx->ifindex, -1); + ctx->devtx_cp_retval = bpf_devtx_cp_attach(ctx->ifindex, -1); + return 0; +} + char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/xdp_hw_metadata.c b/tools/testing/selftests/bpf/xdp_hw_metadata.c index 613321eb84c1..6cc364c2af8a 100644 --- a/tools/testing/selftests/bpf/xdp_hw_metadata.c +++ b/tools/testing/selftests/bpf/xdp_hw_metadata.c @@ -10,7 +10,8 @@ * - rx_hash * * TX: - * - TBD + * - UDP 9091 packets trigger TX reply + * - TX HW timestamp is requested and reported back upon completion */ #include @@ -228,7 +229,83 @@ static void verify_skb_metadata(int fd) printf("skb hwtstamp is not found!\n"); } -static int verify_metadata(struct xsk *rx_xsk, int rxq, int server_fd, clockid_t clock_id) +static void complete_tx(struct xsk *xsk, struct ring_buffer *ringbuf) +{ + __u32 idx; + __u64 addr; + + ring_buffer__poll(ringbuf, 1000); + + if (xsk_ring_cons__peek(&xsk->comp, 1, &idx)) { + addr = *xsk_ring_cons__comp_addr(&xsk->comp, idx); + + printf("%p: complete tx idx=%u addr=%llx\n", xsk, idx, addr); + xsk_ring_cons__release(&xsk->comp, 1); + } +} + +#define swap(a, b, len) do { \ + for (int i = 0; i < len; i++) { \ + __u8 tmp = ((__u8 *)a)[i]; \ + ((__u8 *)a)[i] = ((__u8 *)b)[i]; \ + ((__u8 *)b)[i] = tmp; \ + } \ +} while (0) + +static void ping_pong(struct xsk *xsk, void *rx_packet) +{ + struct ipv6hdr *ip6h = NULL; + struct iphdr *iph = NULL; + struct xdp_desc *tx_desc; + struct udphdr *udph; + struct ethhdr *eth; + void *data; + __u32 idx; + int ret; + int len; + + ret = xsk_ring_prod__reserve(&xsk->tx, 1, &idx); + if (ret != 1) { + printf("%p: failed to reserve tx slot\n", xsk); + return; + } + + tx_desc = xsk_ring_prod__tx_desc(&xsk->tx, idx); + tx_desc->addr = idx % (UMEM_NUM / 2) * UMEM_FRAME_SIZE; + data = xsk_umem__get_data(xsk->umem_area, tx_desc->addr); + + eth = data; + + if (eth->h_proto == htons(ETH_P_IP)) { + iph = (void *)(eth + 1); + udph = (void *)(iph + 1); + } else if (eth->h_proto == htons(ETH_P_IPV6)) { + ip6h = (void *)(eth + 1); + udph = (void *)(ip6h + 1); + } else { + xsk_ring_prod__cancel(&xsk->tx, 1); + return; + } + + len = ETH_HLEN; + if (ip6h) + len += ntohs(ip6h->payload_len); + if (iph) + len += ntohs(iph->tot_len); + + memcpy(data, rx_packet, len); + swap(eth->h_dest, eth->h_source, ETH_ALEN); + if (iph) + swap(&iph->saddr, &iph->daddr, 4); + else + swap(&ip6h->saddr, &ip6h->daddr, 16); + swap(&udph->source, &udph->dest, 2); + + xsk_ring_prod__submit(&xsk->tx, 1); +} + +static int verify_metadata(struct xsk *rx_xsk, int rxq, int server_fd, clockid_t clock_id, + struct ring_buffer *ringbuf) { const struct xdp_desc *rx_desc; struct pollfd fds[rxq + 1]; @@ -280,6 +357,11 @@ static int verify_metadata(struct xsk *rx_xsk, int rxq, int server_fd, clockid_t xsk, idx, rx_desc->addr, addr, comp_addr); verify_xdp_metadata(xsk_umem__get_data(xsk->umem_area, addr), clock_id); + + /* mirror packet back */ + ping_pong(xsk, xsk_umem__get_data(xsk->umem_area, addr)); + complete_tx(xsk, ringbuf); + xsk_ring_cons__release(&xsk->rx, 1); refill_rx(xsk, comp_addr); } @@ -370,6 +452,7 @@ static void hwtstamp_enable(const char *ifname) static void cleanup(void) { LIBBPF_OPTS(bpf_xdp_attach_opts, opts); + int syscall_fd; int ret; int i; @@ -379,8 +462,26 @@ static void cleanup(void) printf("detaching bpf program....\n"); ret = bpf_xdp_detach(ifindex, XDP_FLAGS, &opts); if (ret) - printf("failed to detach XDP program: %d\n", ret); + printf("failed to detach RX XDP program: %d\n", ret); } + + struct devtx_attach_args args = { + .ifindex = ifindex, + .devtx_sb_prog_fd = bpf_program__fd(bpf_obj->progs.devtx_sb), + .devtx_cp_prog_fd = bpf_program__fd(bpf_obj->progs.devtx_cp), + .devtx_sb_retval = -1, + .devtx_cp_retval = -1, + }; + DECLARE_LIBBPF_OPTS(bpf_test_run_opts, tattr, + .ctx_in = &args, + .ctx_size_in = sizeof(args), + ); + + syscall_fd = bpf_program__fd(bpf_obj->progs.detach_prog); + ret = bpf_prog_test_run_opts(syscall_fd, &tattr); + if (ret < 0 || args.devtx_sb_retval < 0 || args.devtx_cp_retval < 0) + printf("failed to detach TX XDP programs: %d %d %d\n", + ret, args.devtx_sb_retval, args.devtx_cp_retval); } for (i = 0; i < rxq; i++) @@ -404,10 +505,22 @@ static void timestamping_enable(int fd, int val) error(1, errno, "setsockopt(SO_TIMESTAMPING)"); } +static int process_sample(void *ctx, void *data, size_t len) +{ + struct devtx_sample *sample = data; + + printf("got tx timestamp sample %u %llu\n", + sample->timestamp_retval, sample->timestamp); + + return 0; +} + int main(int argc, char *argv[]) { + struct ring_buffer *tx_compl_ringbuf = NULL; clockid_t clock_id = CLOCK_TAI; int server_fd = -1; + int syscall_fd; int ret; int i; @@ -448,11 +561,26 @@ int main(int argc, char *argv[]) bpf_program__set_ifindex(prog, ifindex); bpf_program__set_flags(prog, BPF_F_XDP_DEV_BOUND_ONLY); + prog = bpf_object__find_program_by_name(bpf_obj->obj, "devtx_sb"); + bpf_program__set_ifindex(prog, ifindex); + bpf_program__set_flags(prog, BPF_F_XDP_DEV_BOUND_ONLY); + bpf_program__set_autoattach(prog, false); + + prog = bpf_object__find_program_by_name(bpf_obj->obj, "devtx_cp"); + bpf_program__set_ifindex(prog, ifindex); + bpf_program__set_flags(prog, BPF_F_XDP_DEV_BOUND_ONLY); + bpf_program__set_autoattach(prog, false); + printf("load bpf program...\n"); ret = xdp_hw_metadata__load(bpf_obj); if (ret) error(1, -ret, "xdp_hw_metadata__load"); + tx_compl_ringbuf = ring_buffer__new(bpf_map__fd(bpf_obj->maps.tx_compl_buf), + process_sample, NULL, NULL); + if (libbpf_get_error(tx_compl_ringbuf)) + error(1, -libbpf_get_error(tx_compl_ringbuf), "ring_buffer__new"); + printf("prepare skb endpoint...\n"); server_fd = start_server(AF_INET6, SOCK_DGRAM, NULL, 9092, 1000); if (server_fd < 0) @@ -472,15 +600,37 @@ int main(int argc, char *argv[]) error(1, -ret, "bpf_map_update_elem"); } - printf("attach bpf program...\n"); + printf("attach rx bpf program...\n"); ret = bpf_xdp_attach(ifindex, bpf_program__fd(bpf_obj->progs.rx), XDP_FLAGS, NULL); if (ret) error(1, -ret, "bpf_xdp_attach"); + printf("attach tx bpf programs...\n"); + struct devtx_attach_args args = { + .ifindex = ifindex, + .devtx_sb_prog_fd = bpf_program__fd(bpf_obj->progs.devtx_sb), + .devtx_cp_prog_fd = bpf_program__fd(bpf_obj->progs.devtx_cp), + .devtx_sb_retval = -1, + .devtx_cp_retval = -1, + }; + DECLARE_LIBBPF_OPTS(bpf_test_run_opts, tattr, + .ctx_in = &args, + .ctx_size_in = sizeof(args), + ); + + syscall_fd = bpf_program__fd(bpf_obj->progs.attach_prog); + ret = bpf_prog_test_run_opts(syscall_fd, &tattr); + if (ret) + error(1, -ret, "bpf_prog_test_run_opts"); + if (args.devtx_sb_retval < 0) + error(1, args.devtx_sb_retval, "devtx_sb_retval"); + if (args.devtx_cp_retval < 0) + error(1, args.devtx_cp_retval, "devtx_cp_retval"); + signal(SIGINT, handle_signal); - ret = verify_metadata(rx_xsk, rxq, server_fd, clock_id); + ret = verify_metadata(rx_xsk, rxq, server_fd, clock_id, tx_compl_ringbuf); close(server_fd); cleanup(); if (ret)