From patchwork Wed Jun 21 17:02:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13287724 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6914419BA1 for ; Wed, 21 Jun 2023 17:02:50 +0000 (UTC) Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E9D3C129 for ; Wed, 21 Jun 2023 10:02:48 -0700 (PDT) Received: by mail-pg1-x549.google.com with SMTP id 41be03b00d2f7-55384b34f86so2397825a12.2 for ; Wed, 21 Jun 2023 10:02:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687366968; x=1689958968; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=qsajQXCojYmprUydGUF65Mhm0IhEwrrdRl69Y6JxBAY=; b=CtVGRjt4JYkoFb07tvlGiD4Aw3ZVNE8Zkp87FkyBwE5h1f9+bd4JdtDA40x4DKGIIF WjuT++7Ib4HGEcgCQGk8IkKwIxSnOEEe7pE+z6DHo33zqL0H81T8fwke4ck8Fkp1DeyT vM1USJCfNJ22r17+4fUYKOiEvGeCgs43NBw9Fvr+OFQqkNiAkWfHfrRSUiQs69+9IvWk k+konaDkgIi4M+JdSByHuog8mYg/A/mE6i8CrC6n5iPuBsnejqRxxtkp04RP4/xkJ4YG mA4bIdU3+yms5ggUotONE3srvVoJ2q2sqNiAXolJLb/k9zIDJJMU9Ch9aWW4dtnvaLXH hr/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687366968; x=1689958968; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qsajQXCojYmprUydGUF65Mhm0IhEwrrdRl69Y6JxBAY=; b=VWlnqU1uN42EiJkxoRyMtwnJYK/tM3+E3jw5f1e7tk19r2n+DKZnK7fcVbqkVnQ7Oa h449PnXIQiwzC1zG7xFkY/hq2QqwGsrEcLowwdfXuw1Hk6OL1jXyRRV1NEqSOsjunpMt 30epB+5eyLSNrGGN+JwZlK6ZMv7C/5uZME1OuqE87o28oiU+IHbnskI/qAlgAXil/HuO rkupHmY2jrHAdXnQ7BSB+eZtehN5ubbxpKB78RqhZp1TMn+zwiqcBZ4xP4LjxSmBfQIt W1uzbb4lVcmcZ0vMBLNxnFNyyrQw2kV/Wl+NjQ4mRCZFjceBooTC+/sFWAhQtf7J9Hhd eoeA== X-Gm-Message-State: AC+VfDwsDR7cdZR30GbAzmUKP1LUy1blyKV1diqTVCVNsy+mGNPKAXOn PIr4BJqTQfsrqPdbvEIxqiivohNiI005N2QT6QOB+pY1FnxuWhwOuEGlk59AXqQMswDx4mRfCrp BoB+8hHMbDmgo3s3U5ssXoU/xbWPF6dE/I/omitKu3p79NFhNoQ== X-Google-Smtp-Source: ACHHUZ6QwAt+kQccaj4t/qokyIjDKomaB+RFYszQgMILg9wzsAmiSp25+3pL/7MCrudtxyH38SGZKvc= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a63:6286:0:b0:553:a9aa:965d with SMTP id w128-20020a636286000000b00553a9aa965dmr1509689pgb.2.1687366968316; Wed, 21 Jun 2023 10:02:48 -0700 (PDT) Date: Wed, 21 Jun 2023 10:02:34 -0700 In-Reply-To: <20230621170244.1283336-1-sdf@google.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20230621170244.1283336-1-sdf@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621170244.1283336-2-sdf@google.com> Subject: [RFC bpf-next v2 01/11] bpf: Rename some xdp-metadata functions into dev-bound From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, netdev@vger.kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC No functional changes. To make existing dev-bound infrastructure more generic and be less tightly bound to the xdp layer, rename some functions and move kfunc-related things into kernel/bpf/offload.c Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- include/net/offload.h | 28 ++++++++++++++++++++++++++++ include/net/xdp.h | 18 +----------------- kernel/bpf/offload.c | 26 ++++++++++++++++++++++++-- kernel/bpf/verifier.c | 4 ++-- net/core/xdp.c | 20 ++------------------ 5 files changed, 57 insertions(+), 39 deletions(-) create mode 100644 include/net/offload.h diff --git a/include/net/offload.h b/include/net/offload.h new file mode 100644 index 000000000000..264a35881473 --- /dev/null +++ b/include/net/offload.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __LINUX_NET_OFFLOAD_H__ +#define __LINUX_NET_OFFLOAD_H__ + +#include + +#define XDP_METADATA_KFUNC_xxx \ + NETDEV_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_TIMESTAMP, \ + bpf_xdp_metadata_rx_timestamp) \ + NETDEV_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_HASH, \ + bpf_xdp_metadata_rx_hash) + +enum { +#define NETDEV_METADATA_KFUNC(name, _) name, +XDP_METADATA_KFUNC_xxx +#undef NETDEV_METADATA_KFUNC +MAX_NETDEV_METADATA_KFUNC, +}; + +#ifdef CONFIG_NET +u32 bpf_dev_bound_kfunc_id(int id); +bool bpf_is_dev_bound_kfunc(u32 btf_id); +#else +static inline u32 bpf_dev_bound_kfunc_id(int id) { return 0; } +static inline bool bpf_is_dev_bound_kfunc(u32 btf_id) { return false; } +#endif + +#endif /* __LINUX_NET_OFFLOAD_H__ */ diff --git a/include/net/xdp.h b/include/net/xdp.h index d1c5381fc95f..de4c3b70abde 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -9,6 +9,7 @@ #include /* skb_shared_info */ #include #include +#include /** * DOC: XDP RX-queue information @@ -384,19 +385,6 @@ void xdp_attachment_setup(struct xdp_attachment_info *info, #define DEV_MAP_BULK_SIZE XDP_BULK_QUEUE_SIZE -#define XDP_METADATA_KFUNC_xxx \ - XDP_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_TIMESTAMP, \ - bpf_xdp_metadata_rx_timestamp) \ - XDP_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_HASH, \ - bpf_xdp_metadata_rx_hash) \ - -enum { -#define XDP_METADATA_KFUNC(name, _) name, -XDP_METADATA_KFUNC_xxx -#undef XDP_METADATA_KFUNC -MAX_XDP_METADATA_KFUNC, -}; - enum xdp_rss_hash_type { /* First part: Individual bits for L3/L4 types */ XDP_RSS_L3_IPV4 = BIT(0), @@ -444,14 +432,10 @@ enum xdp_rss_hash_type { }; #ifdef CONFIG_NET -u32 bpf_xdp_metadata_kfunc_id(int id); -bool bpf_dev_bound_kfunc_id(u32 btf_id); void xdp_set_features_flag(struct net_device *dev, xdp_features_t val); void xdp_features_set_redirect_target(struct net_device *dev, bool support_sg); void xdp_features_clear_redirect_target(struct net_device *dev); #else -static inline u32 bpf_xdp_metadata_kfunc_id(int id) { return 0; } -static inline bool bpf_dev_bound_kfunc_id(u32 btf_id) { return false; } static inline void xdp_set_features_flag(struct net_device *dev, xdp_features_t val) diff --git a/kernel/bpf/offload.c b/kernel/bpf/offload.c index 8a26cd8814c1..235d81f7e0ed 100644 --- a/kernel/bpf/offload.c +++ b/kernel/bpf/offload.c @@ -844,9 +844,9 @@ void *bpf_dev_bound_resolve_kfunc(struct bpf_prog *prog, u32 func_id) if (!ops) goto out; - if (func_id == bpf_xdp_metadata_kfunc_id(XDP_METADATA_KFUNC_RX_TIMESTAMP)) + if (func_id == bpf_dev_bound_kfunc_id(XDP_METADATA_KFUNC_RX_TIMESTAMP)) p = ops->xmo_rx_timestamp; - else if (func_id == bpf_xdp_metadata_kfunc_id(XDP_METADATA_KFUNC_RX_HASH)) + else if (func_id == bpf_dev_bound_kfunc_id(XDP_METADATA_KFUNC_RX_HASH)) p = ops->xmo_rx_hash; out: up_read(&bpf_devs_lock); @@ -854,6 +854,28 @@ void *bpf_dev_bound_resolve_kfunc(struct bpf_prog *prog, u32 func_id) return p; } +BTF_SET_START(dev_bound_kfunc_ids) +#define NETDEV_METADATA_KFUNC(name, str) BTF_ID(func, str) +XDP_METADATA_KFUNC_xxx +#undef NETDEV_METADATA_KFUNC +BTF_SET_END(dev_bound_kfunc_ids) + +BTF_ID_LIST(dev_bound_kfunc_ids_unsorted) +#define NETDEV_METADATA_KFUNC(name, str) BTF_ID(func, str) +XDP_METADATA_KFUNC_xxx +#undef NETDEV_METADATA_KFUNC + +u32 bpf_dev_bound_kfunc_id(int id) +{ + /* dev_bound_kfunc_ids is sorted and can't be used */ + return dev_bound_kfunc_ids_unsorted[id]; +} + +bool bpf_is_dev_bound_kfunc(u32 btf_id) +{ + return btf_id_set_contains(&dev_bound_kfunc_ids, btf_id); +} + static int __init bpf_offload_init(void) { return rhashtable_init(&offdevs, &offdevs_params); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 1e38584d497c..4db48b5af47e 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -2721,7 +2721,7 @@ static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset) } } - if (bpf_dev_bound_kfunc_id(func_id)) { + if (bpf_is_dev_bound_kfunc(func_id)) { err = bpf_dev_bound_kfunc_check(&env->log, prog_aux); if (err) return err; @@ -17757,7 +17757,7 @@ static void specialize_kfunc(struct bpf_verifier_env *env, void *xdp_kfunc; bool is_rdonly; - if (bpf_dev_bound_kfunc_id(func_id)) { + if (bpf_is_dev_bound_kfunc(func_id)) { xdp_kfunc = bpf_dev_bound_resolve_kfunc(prog, func_id); if (xdp_kfunc) { *addr = (unsigned long)xdp_kfunc; diff --git a/net/core/xdp.c b/net/core/xdp.c index 41e5ca8643ec..819767697370 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -741,9 +741,9 @@ __bpf_kfunc int bpf_xdp_metadata_rx_hash(const struct xdp_md *ctx, u32 *hash, __diag_pop(); BTF_SET8_START(xdp_metadata_kfunc_ids) -#define XDP_METADATA_KFUNC(_, name) BTF_ID_FLAGS(func, name, 0) +#define NETDEV_METADATA_KFUNC(_, name) BTF_ID_FLAGS(func, name, 0) XDP_METADATA_KFUNC_xxx -#undef XDP_METADATA_KFUNC +#undef NETDEV_METADATA_KFUNC BTF_SET8_END(xdp_metadata_kfunc_ids) static const struct btf_kfunc_id_set xdp_metadata_kfunc_set = { @@ -751,22 +751,6 @@ static const struct btf_kfunc_id_set xdp_metadata_kfunc_set = { .set = &xdp_metadata_kfunc_ids, }; -BTF_ID_LIST(xdp_metadata_kfunc_ids_unsorted) -#define XDP_METADATA_KFUNC(name, str) BTF_ID(func, str) -XDP_METADATA_KFUNC_xxx -#undef XDP_METADATA_KFUNC - -u32 bpf_xdp_metadata_kfunc_id(int id) -{ - /* xdp_metadata_kfunc_ids is sorted and can't be used */ - return xdp_metadata_kfunc_ids_unsorted[id]; -} - -bool bpf_dev_bound_kfunc_id(u32 btf_id) -{ - return btf_id_set8_contains(&xdp_metadata_kfunc_ids, btf_id); -} - static int __init xdp_metadata_init(void) { return register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &xdp_metadata_kfunc_set); From patchwork Wed Jun 21 17:02:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13287725 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 75DC819BCE for ; Wed, 21 Jun 2023 17:02:51 +0000 (UTC) Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8EB75120 for ; Wed, 21 Jun 2023 10:02:50 -0700 (PDT) Received: by mail-pg1-x54a.google.com with SMTP id 41be03b00d2f7-53f84f75bf4so2886828a12.3 for ; Wed, 21 Jun 2023 10:02:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687366970; x=1689958970; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=urWJm7reT2IarzSkE9tEiPm+OWYgjit92cM3b8fi47A=; b=CDe40wq+ByPLHcciah53IyY8ka+k/FUoh9rlvUpk0Rz5YkBccmLt3wUrapCI6HMtqD nZxp8Bqv60jel7eAZmLxSoMXdNENuypCbVFCeH8QquCA1Ip1q1nRmSimkOtsY9iWS1yl IYqe6Xx0LLp/gMBVqTyGXFOekUpDa8s4tCkYPZShFrbgwyr7qco/N+m9MyDrQmalEppi I3skQXpqbpInihHpmi1NwgCLEyg+/FMITmbNezFTMvVErPd0x7UWmDcJq50R35AxPuIq KT9dfW1d4gxU856yNbpSfj4HKOjxb27nbSJz4AMDDWJY8oPUAQqwY4ZLjgK2FnsEddvG j/Ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687366970; x=1689958970; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=urWJm7reT2IarzSkE9tEiPm+OWYgjit92cM3b8fi47A=; b=ZpKjZ0XBQdDsyl/fjR4fFI4SUUu9J1UW1zsxXFYVvbb7ciesxcySOxlzurBY4cT2uz Bv2bI/ez9dtSd2ZkyPpUxmiG45rwXDE2/wUjMQ97r+5x/NlU1g6w4Lr01vp3+pUn6jaS rkkynPz84o3n/I0ATMV/WDhXg4Fv7xAqb32zpMYNCDd6e4CfejUFVJZmwpD+d/biwiRn 5NwkVwx3bRXsu85QWV7W2RdtvXKdpx5MIQ+U/peOVKEOGCkJ1IxuoR3e38uLBysKt6W0 ushMK+JEDxXfAwL6dLo/d0Lpz1weI25/xX5PdmT9LYd/+10qfuLxS9UOeqxRGaZFYwd+ Ngog== X-Gm-Message-State: AC+VfDwnQn6Ose+9x0mq2vvayGud9Rs/F4oIEmvxm+UQTibwsGUIrYI+ ndm9eOvnQeCPNI1RP/9r1jhaZc5OSv8NTrE0Z98eTUL6cdYLtRzDPJZHF34sWlWdkwQgjiwPo8i C5Gs3wihxSWL3QlPKFDkPlE0iB8hVhZCcf4gLgRCcU2g9wKM7Ww== X-Google-Smtp-Source: ACHHUZ7T4z5BMYP8aOK9EXSfV5iL1qhQkAqfBTuu6hJZfby78AK9fENfUPBKIY8cnY//hlpEXkd6pfE= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a17:90a:5896:b0:261:2e5:b5af with SMTP id j22-20020a17090a589600b0026102e5b5afmr266907pji.1.1687366969970; Wed, 21 Jun 2023 10:02:49 -0700 (PDT) Date: Wed, 21 Jun 2023 10:02:35 -0700 In-Reply-To: <20230621170244.1283336-1-sdf@google.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20230621170244.1283336-1-sdf@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621170244.1283336-3-sdf@google.com> Subject: [RFC bpf-next v2 02/11] bpf: Resolve single typedef when walking structs From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, netdev@vger.kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC It is impossible to use skb_frag_t in the tracing program. So let's resolve a single typedef when walking the struct. Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- kernel/bpf/btf.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index bd2cac057928..9bdaa1225e8a 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -6140,6 +6140,8 @@ static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf, *flag = 0; again: tname = __btf_name_by_offset(btf, t->name_off); + if (btf_type_is_typedef(t)) + t = btf_type_by_id(btf, t->type); if (!btf_type_is_struct(t)) { bpf_log(log, "Type '%s' is not a struct\n", tname); return -EINVAL; From patchwork Wed Jun 21 17:02:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13287726 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6CA819E4B for ; Wed, 21 Jun 2023 17:02:53 +0000 (UTC) Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9491510D for ; Wed, 21 Jun 2023 10:02:52 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-bfe702f99b8so1416581276.1 for ; Wed, 21 Jun 2023 10:02:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687366972; x=1689958972; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=CmMMVFKL+uO4vUmhJXphZq0hHrHUgs2Y3e/LjEOYX1E=; b=GTXdXkLOwfBndj/ey9c5yVQ3uqcPYCXQjlas28xoEM4O/ozfC3qYpeukFNNyMm+kDN TY1WznK1/pr3ZiK4FyUswGrHOA3zcJGvqs1SjmhxLJ/WZIIycxvqMob29mUGZ1tHCf6h Fy9Uqgmj4WZuTp1QJATRm1g5ZcOIHkn49McBamdhAz9amRRT+pyF/fVRFnp63TMNqoNk kSPkq+2mNyqnKyF3OTflrGOkQNBhlyjQdt97gBZD8jgyrq19xQhncfRZ4yWHI4b/D1Pq pQM1RJ/RUnkzl3Khxo0sVZf6u2et17QrKTAPHQYdB1EyO+x3B953aRyCM5u/pjChGNbj JsSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687366972; x=1689958972; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=CmMMVFKL+uO4vUmhJXphZq0hHrHUgs2Y3e/LjEOYX1E=; b=EsTS8JZu9I9QOaA/V2AN+gKgVB1t4MDup33gbc1SjMNyhfSlkcUGHJOlOsFuT75Gke wU18JDQYijyTzrxBzwXyS/oylvchcFgcQ7aqz/zssYL7K8laJ+kFcR3/erHaqTK/bGD3 oAfOPi6GfhUFR4pkQYUpT/Wl/+nwjKHUEFCz+Oz8z6A67Pu43bKs0wI5ILI6ywlyWHsh lxkCMzAeYR6q/STypjpmIRHANuY8wwHhKU3VQFwZZicfwuq5hF5C0uw1M4DjRYodtj0e mKrdqJEWFEqAqppZ5gdQLuV2u0bcxUwA9d1GAd33+3dBBtxRRwtMJV9rIquW9VNLW8QE sxIQ== X-Gm-Message-State: AC+VfDzj8A5Ia9UqylG9QsHZOmIFo2CgcQTij+ybuNl0zU6vIZoOhN4C gUmgeEpfoJ430b09gd6+gRPp0+pAXOFuRjq+NENu1+Yy8OwAjOghLtRt74RNpmFO0bmToH75l5P yZ/7ACp9bxUbg2WmcY/sW5j2ykQkYQXrDobpcdRPLHfwJNOunWg== X-Google-Smtp-Source: ACHHUZ4iiXFc+nsiyhcFiaYXPY5UyL0vT7S0GUIb9ABTvrq36uh9VhEiCvjuAJNTkG5wd3HDfKFZgR0= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a25:addb:0:b0:be3:b9a6:a6b2 with SMTP id d27-20020a25addb000000b00be3b9a6a6b2mr2143660ybe.9.1687366971803; Wed, 21 Jun 2023 10:02:51 -0700 (PDT) Date: Wed, 21 Jun 2023 10:02:36 -0700 In-Reply-To: <20230621170244.1283336-1-sdf@google.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20230621170244.1283336-1-sdf@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621170244.1283336-4-sdf@google.com> Subject: [RFC bpf-next v2 03/11] xsk: Support XDP_TX_METADATA_LEN From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC For zerocopy mode, tx_desc->addr can point to the arbitrary offset and carry some TX metadata in the headroom. For copy mode, there is no way currently to populate skb metadata. Introduce new XDP_TX_METADATA_LEN that indicates how many bytes to treat as metadata. Metadata bytes come prior to tx_desc address (same as in RX case). Signed-off-by: Stanislav Fomichev --- include/net/xdp_sock.h | 1 + include/net/xsk_buff_pool.h | 1 + include/uapi/linux/if_xdp.h | 1 + net/xdp/xsk.c | 31 ++++++++++++++++++++++++++++++- net/xdp/xsk_buff_pool.c | 1 + net/xdp/xsk_queue.h | 7 ++++--- 6 files changed, 38 insertions(+), 4 deletions(-) diff --git a/include/net/xdp_sock.h b/include/net/xdp_sock.h index e96a1151ec75..30018b3b862d 100644 --- a/include/net/xdp_sock.h +++ b/include/net/xdp_sock.h @@ -51,6 +51,7 @@ struct xdp_sock { struct list_head flush_node; struct xsk_buff_pool *pool; u16 queue_id; + u8 tx_metadata_len; bool zc; enum { XSK_READY = 0, diff --git a/include/net/xsk_buff_pool.h b/include/net/xsk_buff_pool.h index a8d7b8a3688a..751fea51a6af 100644 --- a/include/net/xsk_buff_pool.h +++ b/include/net/xsk_buff_pool.h @@ -75,6 +75,7 @@ struct xsk_buff_pool { u32 chunk_size; u32 chunk_shift; u32 frame_len; + u8 tx_metadata_len; /* inherited from xsk_sock */ u8 cached_need_wakeup; bool uses_need_wakeup; bool dma_need_sync; diff --git a/include/uapi/linux/if_xdp.h b/include/uapi/linux/if_xdp.h index a78a8096f4ce..2374eafff7db 100644 --- a/include/uapi/linux/if_xdp.h +++ b/include/uapi/linux/if_xdp.h @@ -63,6 +63,7 @@ struct xdp_mmap_offsets { #define XDP_UMEM_COMPLETION_RING 6 #define XDP_STATISTICS 7 #define XDP_OPTIONS 8 +#define XDP_TX_METADATA_LEN 9 struct xdp_umem_reg { __u64 addr; /* Start of packet data area */ diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c index cc1e7f15fa73..c9b2daba7b6d 100644 --- a/net/xdp/xsk.c +++ b/net/xdp/xsk.c @@ -485,6 +485,7 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs, int err; hr = max(NET_SKB_PAD, L1_CACHE_ALIGN(dev->needed_headroom)); + hr = max(hr, L1_CACHE_ALIGN((u32)xs->tx_metadata_len)); tr = dev->needed_tailroom; len = desc->len; @@ -493,14 +494,21 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs, return ERR_PTR(err); skb_reserve(skb, hr); - skb_put(skb, len); + skb_put(skb, len + xs->tx_metadata_len); buffer = xsk_buff_raw_get_data(xs->pool, desc->addr); + buffer -= xs->tx_metadata_len; + err = skb_store_bits(skb, 0, buffer, len); if (unlikely(err)) { kfree_skb(skb); return ERR_PTR(err); } + + if (xs->tx_metadata_len) { + skb_metadata_set(skb, xs->tx_metadata_len); + __skb_pull(skb, xs->tx_metadata_len); + } } skb->dev = dev; @@ -1137,6 +1145,27 @@ static int xsk_setsockopt(struct socket *sock, int level, int optname, mutex_unlock(&xs->mutex); return err; } + case XDP_TX_METADATA_LEN: + { + int val; + + if (optlen < sizeof(val)) + return -EINVAL; + if (copy_from_sockptr(&val, optval, sizeof(val))) + return -EFAULT; + + if (val >= 256) + return -EINVAL; + + mutex_lock(&xs->mutex); + if (xs->state != XSK_READY) { + mutex_unlock(&xs->mutex); + return -EBUSY; + } + xs->tx_metadata_len = val; + mutex_unlock(&xs->mutex); + return err; + } default: break; } diff --git a/net/xdp/xsk_buff_pool.c b/net/xdp/xsk_buff_pool.c index 26f6d304451e..66ff9c345a67 100644 --- a/net/xdp/xsk_buff_pool.c +++ b/net/xdp/xsk_buff_pool.c @@ -85,6 +85,7 @@ struct xsk_buff_pool *xp_create_and_assign_umem(struct xdp_sock *xs, XDP_PACKET_HEADROOM; pool->umem = umem; pool->addrs = umem->addrs; + pool->tx_metadata_len = xs->tx_metadata_len; INIT_LIST_HEAD(&pool->free_list); INIT_LIST_HEAD(&pool->xsk_tx_list); spin_lock_init(&pool->xsk_tx_list_lock); diff --git a/net/xdp/xsk_queue.h b/net/xdp/xsk_queue.h index 6d40a77fccbe..c8d287c18d64 100644 --- a/net/xdp/xsk_queue.h +++ b/net/xdp/xsk_queue.h @@ -133,12 +133,13 @@ static inline bool xskq_cons_read_addr_unchecked(struct xsk_queue *q, u64 *addr) static inline bool xp_aligned_validate_desc(struct xsk_buff_pool *pool, struct xdp_desc *desc) { - u64 offset = desc->addr & (pool->chunk_size - 1); + u64 addr = desc->addr - pool->tx_metadata_len; + u64 offset = addr & (pool->chunk_size - 1); if (offset + desc->len > pool->chunk_size) return false; - if (desc->addr >= pool->addrs_cnt) + if (addr >= pool->addrs_cnt) return false; if (desc->options) @@ -149,7 +150,7 @@ static inline bool xp_aligned_validate_desc(struct xsk_buff_pool *pool, static inline bool xp_unaligned_validate_desc(struct xsk_buff_pool *pool, struct xdp_desc *desc) { - u64 addr = xp_unaligned_add_offset_to_addr(desc->addr); + u64 addr = xp_unaligned_add_offset_to_addr(desc->addr) - pool->tx_metadata_len; if (desc->len > pool->chunk_size) return false; From patchwork Wed Jun 21 17:02:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13287727 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 792FF19E5F for ; Wed, 21 Jun 2023 17:02:57 +0000 (UTC) Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CB003122 for ; Wed, 21 Jun 2023 10:02:54 -0700 (PDT) Received: by mail-pf1-x44a.google.com with SMTP id d2e1a72fcca58-666e3dad70aso4452081b3a.0 for ; Wed, 21 Jun 2023 10:02:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687366974; x=1689958974; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kb0kUssv+VWESwpyJ8lyZP4iXH/Vyyf0wzg9dJXmkwg=; b=skFLAHaedAMGEp/yTMVy1WbRiNPdv1HPQOq0/d5yOnGWq6mW7/MQYrRI/yyYJDUhqV r2fFbYAdOnjg6V8Mr/wpJZMEa3mh0LJc+A7imaDn7kPeqATJm2MYyv/BaycLeAnvEIrR YRZnXxNrjBSfYuH5ByEPOiv8U1FSq5U3HYGivJtYbys7sTW15HH2tYsKEsCrw3kDUW0q nkch99E3HqQdPhAzOkLRjvMj3u88OK/JRgCs7ZDDT5u2QfS+dZnMQIWhRP/zzBzHxT6p 5kjbHM55oQu5eXGZpvTREJWYBs0Odgf1msSbuguMXgAQBL4h/YDe2NfZnRfrZxvE5h2F f0Eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687366974; x=1689958974; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kb0kUssv+VWESwpyJ8lyZP4iXH/Vyyf0wzg9dJXmkwg=; b=PGh/lgxGtoYdH5rVvxLe6md00wSW3jSi1mZ1v9fvzuePAYB4P+k1kf3ZuTemApHes4 eH87aEZC9LBjW8bGZ9s7JXgg/ujAz59kvELSD+gxEr9JWjClFnZxhJRh2Fu1RMqn3KKc ygyfpagGs/7hyKYroUhPSpghwRfEEgK8rakk8u6pRkSww4O6RTFP36Ba21556oDkbW86 ieJZ4EIPUCFj8XVtNbQaOXACdGHl0JEe8cFTtdbMOdCtqYAJH4Jv0cm1YkEce+Wes0Jw 36Kp4+J/+gKTGGfuAGSdyvPR2VbFdZYZFT3lJmRv41zOxY+GJzsdVun4cGST44E+TZLi C48Q== X-Gm-Message-State: AC+VfDzHwywOh4E+1216ddhXBImKtOqmh7ga+mlKvejh2oBP7hnjo2bC L86iMKdOxP8DOdcHWR6iTB2I5SIUr7rYn76aoxs+nOHdYRSAo3Ihl92TYdnhzCCgG9ElHlUH7ko czGTym4a87UkSTe/EbkRGkGbez+2zu9LyVtr9hEK6ou1akNq5fg== X-Google-Smtp-Source: ACHHUZ4x4MJ1Z8wsvz6uZJLGxCK1rGj58yr2mKPdAMupvayjOLNnP7W82YSYrvPghax0fnrlfOA+nJY= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a05:6a00:b4e:b0:668:63ee:7706 with SMTP id p14-20020a056a000b4e00b0066863ee7706mr3579259pfo.3.1687366973471; Wed, 21 Jun 2023 10:02:53 -0700 (PDT) Date: Wed, 21 Jun 2023 10:02:37 -0700 In-Reply-To: <20230621170244.1283336-1-sdf@google.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20230621170244.1283336-1-sdf@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621170244.1283336-5-sdf@google.com> Subject: [RFC bpf-next v2 04/11] bpf: Implement devtx hook points From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, netdev@vger.kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC devtx is a lightweight set of hooks before and after packet transmission. The hook is supposed to work for both skb and xdp paths by exposing a light-weight packet wrapper via devtx_frame (header portion + frags). devtx is implemented as a tracing program which has access to the XDP-metadata-like kfuncs. The initial set of kfuncs is implemented in the next patch, but the idea is similar to XDP metadata: the kfuncs have netdev-specific implementation, but common interface. Upon loading, the kfuncs are resolved to direct calls against per-netdev implementation. This can be achieved by marking devtx-tracing programs as dev-bound (largely reusing xdp-dev-bound program infrastructure). Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- MAINTAINERS | 2 ++ include/net/devtx.h | 71 +++++++++++++++++++++++++++++++++++++++++ kernel/bpf/offload.c | 15 +++++++++ net/core/Makefile | 1 + net/core/dev.c | 1 + net/core/devtx.c | 76 ++++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 166 insertions(+) create mode 100644 include/net/devtx.h create mode 100644 net/core/devtx.c diff --git a/MAINTAINERS b/MAINTAINERS index c904dba1733b..516529b42e66 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -22976,11 +22976,13 @@ L: bpf@vger.kernel.org S: Supported F: drivers/net/ethernet/*/*/*/*/*xdp* F: drivers/net/ethernet/*/*/*xdp* +F: include/net/devtx.h F: include/net/xdp.h F: include/net/xdp_priv.h F: include/trace/events/xdp.h F: kernel/bpf/cpumap.c F: kernel/bpf/devmap.c +F: net/core/devtx.c F: net/core/xdp.c F: samples/bpf/xdp* F: tools/testing/selftests/bpf/*/*xdp* diff --git a/include/net/devtx.h b/include/net/devtx.h new file mode 100644 index 000000000000..d1c75fd9b377 --- /dev/null +++ b/include/net/devtx.h @@ -0,0 +1,71 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __LINUX_NET_DEVTX_H__ +#define __LINUX_NET_DEVTX_H__ + +#include +#include +#include +#include +#include + +struct devtx_frame { + void *data; + u16 len; + u8 meta_len; + struct skb_shared_info *sinfo; /* for frags */ + struct net_device *netdev; +}; + +#ifdef CONFIG_NET +void devtx_hooks_enable(void); +void devtx_hooks_disable(void); +bool devtx_hooks_match(u32 attach_btf_id, const struct xdp_metadata_ops *xmo); +int devtx_hooks_register(struct btf_id_set8 *set, const struct xdp_metadata_ops *xmo); +void devtx_hooks_unregister(struct btf_id_set8 *set); + +static inline void devtx_frame_from_skb(struct devtx_frame *ctx, struct sk_buff *skb, + struct net_device *netdev) +{ + ctx->data = skb->data; + ctx->len = skb_headlen(skb); + ctx->meta_len = skb_metadata_len(skb); + ctx->sinfo = skb_shinfo(skb); + ctx->netdev = netdev; +} + +static inline void devtx_frame_from_xdp(struct devtx_frame *ctx, struct xdp_frame *xdpf, + struct net_device *netdev) +{ + ctx->data = xdpf->data; + ctx->len = xdpf->len; + ctx->meta_len = xdpf->metasize & 0xff; + ctx->sinfo = xdp_frame_has_frags(xdpf) ? xdp_get_shared_info_from_frame(xdpf) : NULL; + ctx->netdev = netdev; +} + +DECLARE_STATIC_KEY_FALSE(devtx_enabled_key); + +static inline bool devtx_enabled(void) +{ + return static_branch_unlikely(&devtx_enabled_key); +} +#else +static inline void devtx_hooks_enable(void) {} +static inline void devtx_hooks_disable(void) {} +static inline bool devtx_hooks_match(u32 attach_btf_id, const struct xdp_metadata_ops *xmo) {} +static inline int devtx_hooks_register(struct btf_id_set8 *set, + const struct xdp_metadata_ops *xmo) {} +static inline void devtx_hooks_unregister(struct btf_id_set8 *set) {} + +static inline void devtx_frame_from_skb(struct devtx_frame *ctx, struct sk_buff *skb, + struct net_device *netdev) {} +static inline void devtx_frame_from_xdp(struct devtx_frame *ctx, struct xdp_frame *xdpf, + struct net_device *netdev) {} + +static inline bool devtx_enabled(void) +{ + return false; +} +#endif + +#endif /* __LINUX_NET_DEVTX_H__ */ diff --git a/kernel/bpf/offload.c b/kernel/bpf/offload.c index 235d81f7e0ed..f01a1aa0f627 100644 --- a/kernel/bpf/offload.c +++ b/kernel/bpf/offload.c @@ -25,6 +25,7 @@ #include #include #include +#include /* Protects offdevs, members of bpf_offload_netdev and offload members * of all progs. @@ -228,6 +229,7 @@ int bpf_prog_dev_bound_init(struct bpf_prog *prog, union bpf_attr *attr) int err; if (attr->prog_type != BPF_PROG_TYPE_SCHED_CLS && + attr->prog_type != BPF_PROG_TYPE_TRACING && attr->prog_type != BPF_PROG_TYPE_XDP) return -EINVAL; @@ -242,6 +244,15 @@ int bpf_prog_dev_bound_init(struct bpf_prog *prog, union bpf_attr *attr) if (!netdev) return -EINVAL; + /* Make sure device-bound tracing programs are being attached + * to the appropriate netdev. + */ + if (attr->prog_type == BPF_PROG_TYPE_TRACING && + !devtx_hooks_match(prog->aux->attach_btf_id, netdev->xdp_metadata_ops)) { + err = -EINVAL; + goto out; + } + err = bpf_dev_offload_check(netdev); if (err) goto out; @@ -252,6 +263,9 @@ int bpf_prog_dev_bound_init(struct bpf_prog *prog, union bpf_attr *attr) err = __bpf_prog_dev_bound_init(prog, netdev); up_write(&bpf_devs_lock); + if (!err) + devtx_hooks_enable(); + out: dev_put(netdev); return err; @@ -384,6 +398,7 @@ void bpf_prog_dev_bound_destroy(struct bpf_prog *prog) ondev = bpf_offload_find_netdev(netdev); if (!ondev->offdev && list_empty(&ondev->progs)) __bpf_offload_dev_netdev_unregister(NULL, netdev); + devtx_hooks_disable(); } up_write(&bpf_devs_lock); rtnl_unlock(); diff --git a/net/core/Makefile b/net/core/Makefile index 8f367813bc68..c1db05ccfac7 100644 --- a/net/core/Makefile +++ b/net/core/Makefile @@ -39,4 +39,5 @@ obj-$(CONFIG_FAILOVER) += failover.o obj-$(CONFIG_NET_SOCK_MSG) += skmsg.o obj-$(CONFIG_BPF_SYSCALL) += sock_map.o obj-$(CONFIG_BPF_SYSCALL) += bpf_sk_storage.o +obj-$(CONFIG_BPF_SYSCALL) += devtx.o obj-$(CONFIG_OF) += of_net.o diff --git a/net/core/dev.c b/net/core/dev.c index 3393c2f3dbe8..e2f4618ee1c5 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -150,6 +150,7 @@ #include #include #include +#include #include "dev.h" #include "net-sysfs.h" diff --git a/net/core/devtx.c b/net/core/devtx.c new file mode 100644 index 000000000000..bad694439ae3 --- /dev/null +++ b/net/core/devtx.c @@ -0,0 +1,76 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include +#include + +DEFINE_STATIC_KEY_FALSE(devtx_enabled_key); +EXPORT_SYMBOL_GPL(devtx_enabled_key); + +struct devtx_hook_entry { + struct list_head devtx_hooks; + struct btf_id_set8 *set; + const struct xdp_metadata_ops *xmo; +}; + +static LIST_HEAD(devtx_hooks); +static DEFINE_MUTEX(devtx_hooks_lock); + +void devtx_hooks_enable(void) +{ + static_branch_inc(&devtx_enabled_key); +} + +void devtx_hooks_disable(void) +{ + static_branch_dec(&devtx_enabled_key); +} + +bool devtx_hooks_match(u32 attach_btf_id, const struct xdp_metadata_ops *xmo) +{ + struct devtx_hook_entry *entry, *tmp; + bool match = false; + + mutex_lock(&devtx_hooks_lock); + list_for_each_entry_safe(entry, tmp, &devtx_hooks, devtx_hooks) { + if (btf_id_set8_contains(entry->set, attach_btf_id)) { + match = entry->xmo == xmo; + break; + } + } + mutex_unlock(&devtx_hooks_lock); + + return match; +} + +int devtx_hooks_register(struct btf_id_set8 *set, const struct xdp_metadata_ops *xmo) +{ + struct devtx_hook_entry *entry; + + entry = kzalloc(sizeof(*entry), GFP_KERNEL); + if (!entry) + return -ENOMEM; + + entry->set = set; + entry->xmo = xmo; + + mutex_lock(&devtx_hooks_lock); + list_add(&entry->devtx_hooks, &devtx_hooks); + mutex_unlock(&devtx_hooks_lock); + + return 0; +} + +void devtx_hooks_unregister(struct btf_id_set8 *set) +{ + struct devtx_hook_entry *entry, *tmp; + + mutex_lock(&devtx_hooks_lock); + list_for_each_entry_safe(entry, tmp, &devtx_hooks, devtx_hooks) { + if (entry->set == set) { + list_del(&entry->devtx_hooks); + kfree(entry); + break; + } + } + mutex_unlock(&devtx_hooks_lock); +} From patchwork Wed Jun 21 17:02:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13287728 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A181D19E6B for ; Wed, 21 Jun 2023 17:02:57 +0000 (UTC) Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 539B110D for ; Wed, 21 Jun 2023 10:02:56 -0700 (PDT) Received: by mail-pf1-x44a.google.com with SMTP id d2e1a72fcca58-65026629c1eso2880660b3a.0 for ; Wed, 21 Jun 2023 10:02:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687366976; x=1689958976; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=feixGtICBDKSGOY8r1mfjPfsz/6VV1h1omDCXVYez4I=; b=ny/L+aW+10oezDk6bM0bGp76eorWQRHnGyYJZANTy21DBw1rgYeQQ+1m4bkt990Ikg REx5PzHEFFOaMZRiRnVD3nGjvh0jNBm/DJ5wTPqcgxNBDA8fbKd0r08wxchNjWXstgVV 4JwGIaUtnDvrhH2yzLCSA/00nRa/Rfx51XtK/y9Xi0O0ZO9hzUwSsFfV+bTvOsGE5h9W E/fYMlIg9SCa33bvv2RViEoBC4FbHHTHcVu+OSBV88zbOR+d1RHKgZLomMR7N++8E8s4 QiavWPt1pwCx+PzTfcL81y+xDdXadA9EpBnhE2+Veab5V0SGo5I2ZETY2MdthpUJFnau oPSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687366976; x=1689958976; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=feixGtICBDKSGOY8r1mfjPfsz/6VV1h1omDCXVYez4I=; b=Xavb0GMgtZoel1AtfUAM0GKxuT7R2jHVgUGu/z7DiHfTCsaSJSOYNWrZ32VUWB7yAw xdVRAEdEp2oaq3uXxQ/8ATa+AY26/ABiMwkEKSkVmewnqaJ/1PHABnGnPG9W3dyTaOII dOq0oms5WZg78Z3FeXwc6gs+S40RQY2WsU7DLJrb+eAB7D9IaBJEk74ITDfkc4k8oW72 2NW+E39Uq0tkc19G+JSOtpQu1Yu3w4P2KK77Ksz4Abs5Xq0p19sX3LI7oxMv9yKA7+up bEhsv23GxOmOUdy1Rj541+TwcUpITcSIILKDISDS9QdTVVtf7N/6LdHETGQeUb+D81AN 8ThA== X-Gm-Message-State: AC+VfDyg2EztVownivgIavh3ySg8Wy4w2lOkuP6hIvHYChjxldzV/47X 0Yp4ggTSDB+82mNB/IkF38VZfXTrctCzQ7WvKvEquYVtDz0CA31OeruwgtcH0Y6IfGav2SrfhfJ KO4LDycMFDxc/rDnUbC18AE3bviE/mZf7sA+Zt0FP5G7fH8U4mg== X-Google-Smtp-Source: ACHHUZ6ISDHeZVa+4R7W5zMDkFKy97bcfl3K0mzhnyE5pRgRBS/MYd4Bk0izNG84FFWKoP1YUaMWXWQ= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a05:6a00:9a3:b0:668:7ad6:81f2 with SMTP id u35-20020a056a0009a300b006687ad681f2mr2572942pfg.4.1687366975783; Wed, 21 Jun 2023 10:02:55 -0700 (PDT) Date: Wed, 21 Jun 2023 10:02:38 -0700 In-Reply-To: <20230621170244.1283336-1-sdf@google.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20230621170244.1283336-1-sdf@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621170244.1283336-6-sdf@google.com> Subject: [RFC bpf-next v2 05/11] bpf: Implement devtx timestamp kfunc From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, netdev@vger.kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Two kfuncs, one per hook point: 1. at submit time - bpf_devtx_sb_request_timestamp - to request HW to put TX timestamp into TX completion descriptors 2. at completion time - bpf_devtx_cp_timestamp - to read out TX timestamp Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- include/linux/netdevice.h | 4 +++ include/net/offload.h | 10 ++++++ kernel/bpf/offload.c | 8 +++++ net/core/devtx.c | 73 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 95 insertions(+) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 08fbd4622ccf..2fdb0731eb67 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -1651,10 +1651,14 @@ struct net_device_ops { bool cycles); }; +struct devtx_frame; + struct xdp_metadata_ops { int (*xmo_rx_timestamp)(const struct xdp_md *ctx, u64 *timestamp); int (*xmo_rx_hash)(const struct xdp_md *ctx, u32 *hash, enum xdp_rss_hash_type *rss_type); + int (*xmo_sb_request_timestamp)(const struct devtx_frame *ctx); + int (*xmo_cp_timestamp)(const struct devtx_frame *ctx, u64 *timestamp); }; /** diff --git a/include/net/offload.h b/include/net/offload.h index 264a35881473..36899b64f4c8 100644 --- a/include/net/offload.h +++ b/include/net/offload.h @@ -10,9 +10,19 @@ NETDEV_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_HASH, \ bpf_xdp_metadata_rx_hash) +#define DEVTX_SB_KFUNC_xxx \ + NETDEV_METADATA_KFUNC(DEVTX_SB_KFUNC_REQUEST_TIMESTAMP, \ + bpf_devtx_sb_request_timestamp) + +#define DEVTX_CP_KFUNC_xxx \ + NETDEV_METADATA_KFUNC(DEVTX_CP_KFUNC_TIMESTAMP, \ + bpf_devtx_cp_timestamp) + enum { #define NETDEV_METADATA_KFUNC(name, _) name, XDP_METADATA_KFUNC_xxx +DEVTX_SB_KFUNC_xxx +DEVTX_CP_KFUNC_xxx #undef NETDEV_METADATA_KFUNC MAX_NETDEV_METADATA_KFUNC, }; diff --git a/kernel/bpf/offload.c b/kernel/bpf/offload.c index f01a1aa0f627..45a243af49be 100644 --- a/kernel/bpf/offload.c +++ b/kernel/bpf/offload.c @@ -863,6 +863,10 @@ void *bpf_dev_bound_resolve_kfunc(struct bpf_prog *prog, u32 func_id) p = ops->xmo_rx_timestamp; else if (func_id == bpf_dev_bound_kfunc_id(XDP_METADATA_KFUNC_RX_HASH)) p = ops->xmo_rx_hash; + else if (func_id == bpf_dev_bound_kfunc_id(DEVTX_SB_KFUNC_REQUEST_TIMESTAMP)) + p = ops->xmo_sb_request_timestamp; + else if (func_id == bpf_dev_bound_kfunc_id(DEVTX_CP_KFUNC_TIMESTAMP)) + p = ops->xmo_cp_timestamp; out: up_read(&bpf_devs_lock); @@ -872,12 +876,16 @@ void *bpf_dev_bound_resolve_kfunc(struct bpf_prog *prog, u32 func_id) BTF_SET_START(dev_bound_kfunc_ids) #define NETDEV_METADATA_KFUNC(name, str) BTF_ID(func, str) XDP_METADATA_KFUNC_xxx +DEVTX_SB_KFUNC_xxx +DEVTX_CP_KFUNC_xxx #undef NETDEV_METADATA_KFUNC BTF_SET_END(dev_bound_kfunc_ids) BTF_ID_LIST(dev_bound_kfunc_ids_unsorted) #define NETDEV_METADATA_KFUNC(name, str) BTF_ID(func, str) XDP_METADATA_KFUNC_xxx +DEVTX_SB_KFUNC_xxx +DEVTX_CP_KFUNC_xxx #undef NETDEV_METADATA_KFUNC u32 bpf_dev_bound_kfunc_id(int id) diff --git a/net/core/devtx.c b/net/core/devtx.c index bad694439ae3..4267a8fe6711 100644 --- a/net/core/devtx.c +++ b/net/core/devtx.c @@ -74,3 +74,76 @@ void devtx_hooks_unregister(struct btf_id_set8 *set) } mutex_unlock(&devtx_hooks_lock); } + +__diag_push(); +__diag_ignore_all("-Wmissing-prototypes", + "Global functions as their definitions will be in vmlinux BTF"); + +/** + * bpf_devtx_sb_request_timestamp - Request TX timestamp on the packet. + * Callable only from the devtx-submit hook. + * @ctx: devtx context pointer. + * + * Returns 0 on success or ``-errno`` on error. + */ +__bpf_kfunc int bpf_devtx_sb_request_timestamp(const struct devtx_frame *ctx) +{ + return -EOPNOTSUPP; +} + +/** + * bpf_devtx_cp_timestamp - Read TX timestamp of the packet. Callable + * only from the devtx-complete hook. + * @ctx: devtx context pointer. + * @timestamp: Return value pointer. + * + * Returns 0 on success or ``-errno`` on error. + */ +__bpf_kfunc int bpf_devtx_cp_timestamp(const struct devtx_frame *ctx, __u64 *timestamp) +{ + return -EOPNOTSUPP; +} + +__diag_pop(); + +BTF_SET8_START(devtx_sb_kfunc_ids) +#define NETDEV_METADATA_KFUNC(_, name) BTF_ID_FLAGS(func, name, 0) +DEVTX_SB_KFUNC_xxx +#undef NETDEV_METADATA_KFUNC +BTF_SET8_END(devtx_sb_kfunc_ids) + +static const struct btf_kfunc_id_set devtx_sb_kfunc_set = { + .owner = THIS_MODULE, + .set = &devtx_sb_kfunc_ids, +}; + +BTF_SET8_START(devtx_cp_kfunc_ids) +#define NETDEV_METADATA_KFUNC(_, name) BTF_ID_FLAGS(func, name, 0) +DEVTX_CP_KFUNC_xxx +#undef NETDEV_METADATA_KFUNC +BTF_SET8_END(devtx_cp_kfunc_ids) + +static const struct btf_kfunc_id_set devtx_cp_kfunc_set = { + .owner = THIS_MODULE, + .set = &devtx_cp_kfunc_ids, +}; + +static int __init devtx_init(void) +{ + int ret; + + ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &devtx_sb_kfunc_set); + if (ret) { + pr_warn("failed to register devtx_sb kfuncs: %d", ret); + return ret; + } + + ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &devtx_cp_kfunc_set); + if (ret) { + pr_warn("failed to register devtx_cp completion kfuncs: %d", ret); + return ret; + } + + return 0; +} +late_initcall(devtx_init); From patchwork Wed Jun 21 17:02:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13287729 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4FBF119E5E for ; Wed, 21 Jun 2023 17:02:59 +0000 (UTC) Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 209B910D for ; Wed, 21 Jun 2023 10:02:58 -0700 (PDT) Received: by mail-pf1-x44a.google.com with SMTP id d2e1a72fcca58-66872bfa48aso3313419b3a.0 for ; Wed, 21 Jun 2023 10:02:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687366977; x=1689958977; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Oh4MSgHmxCb0wLEkBD2zbZNTxXKYUJSkffgOtaf+rTs=; b=RAIsuwPnJxewMm+JMGNDPEzSp7rm1vTOL3l+y+Ek4+VM6VDxu4eE1PyQ0u9FKximZ3 koBXX5rYHhqULcsdiNZaqy29sLGW+n6d6GgP1i0tZYpp7HOtPBdAmlbuthG/TRdGhmoL vbgefmU1Lz6QrDmBv7dWneXWd6bz3BOSqLV1R9B3FzBOmD2NKWyGzCfpAProCLSkX/Vj Tc5auvESSEkxGAt+uSXPAHRCJr96XhsTHif/1PjX+0pLi9g96dZC7LmNL/YKK266Y89+ LHAxIBIKQ4cL25fOyzXQgq+vI3LaRrbpnSJlw2JnIPEQDsMuYt9L0PC/oK6NuwSVoE+v 1vMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687366977; x=1689958977; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Oh4MSgHmxCb0wLEkBD2zbZNTxXKYUJSkffgOtaf+rTs=; b=ccEnBIgwcCAIazt1w+P7cEkXg+MRLCcCXnyXFGIKTlDP9M/P0pLbAkxesXE16dnnVs NOQULVXEsu6K1/dIJCdoRlmtBR1ehxMk3OWH+NcctvQllP8zu7+2eE6j2/CVynopVRyv onAj/u+dwOYpRbjYuL4RZCIrrfzp0xBvTr10CGRhQj2GoYUHNe4ekj2aGgit3YoR+cfF lFb5PEJObpQhZh8LC5fw3trFIdpqgSWUNY9H2ir7J+b68TE7zuiNhqIUy51A8ca3Fkls zxTWgMkR+fBRk2+rd86MIIGqAC5OtklkHYPzqAT3K4FcAhcXwSi07Urwt7hnAaDZ4/9E FLLw== X-Gm-Message-State: AC+VfDzxk9kg2kIayX+ajppP7z5UcwEPRhK/hijUD69Vy/GAIp6aG75M aawoylP13QeA8zUSbNuyULPK/0BHqSbZFfx35sYHXx5QS6/cnan5f0Qo/tAzhyHx8IKbvICOKBI ZLAo2oZ4caMWJyam/2mj96Lev24OLhpnLgPs9rIj+3q8ar83pOg== X-Google-Smtp-Source: ACHHUZ7noYnqFnY0BjqO4dFxzymGP5h5Lq4p21OkqJJlR/O84AnCDKsNb4KlYY4MWyOCBqsVKwEVPSI= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a05:6a00:3912:b0:667:26bb:db7 with SMTP id fh18-20020a056a00391200b0066726bb0db7mr3521641pfb.1.1687366977569; Wed, 21 Jun 2023 10:02:57 -0700 (PDT) Date: Wed, 21 Jun 2023 10:02:39 -0700 In-Reply-To: <20230621170244.1283336-1-sdf@google.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20230621170244.1283336-1-sdf@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621170244.1283336-7-sdf@google.com> Subject: [RFC bpf-next v2 06/11] net: veth: Implement devtx timestamp kfuncs From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, netdev@vger.kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Have a software-based example for kfuncs to showcase how it can be used in the real devices and to have something to test against in the selftests. Both path (skb & xdp) are covered. Only the skb path is really tested though. Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- drivers/net/veth.c | 116 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 112 insertions(+), 4 deletions(-) diff --git a/drivers/net/veth.c b/drivers/net/veth.c index 614f3e3efab0..632f0f3771e4 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -27,6 +27,7 @@ #include #include #include +#include #define DRV_NAME "veth" #define DRV_VERSION "1.0" @@ -123,6 +124,13 @@ struct veth_xdp_buff { struct sk_buff *skb; }; +struct veth_devtx_frame { + struct devtx_frame frame; + bool request_timestamp; + ktime_t xdp_tx_timestamp; + struct sk_buff *skb; +}; + static int veth_get_link_ksettings(struct net_device *dev, struct ethtool_link_ksettings *cmd) { @@ -313,10 +321,43 @@ static int veth_xdp_rx(struct veth_rq *rq, struct sk_buff *skb) return NET_RX_SUCCESS; } +__weak noinline void veth_devtx_submit(struct devtx_frame *ctx) +{ +} + +__weak noinline void veth_devtx_complete(struct devtx_frame *ctx) +{ +} + +BTF_SET8_START(veth_devtx_hook_ids) +BTF_ID_FLAGS(func, veth_devtx_submit) +BTF_ID_FLAGS(func, veth_devtx_complete) +BTF_SET8_END(veth_devtx_hook_ids) + static int veth_forward_skb(struct net_device *dev, struct sk_buff *skb, - struct veth_rq *rq, bool xdp) + struct veth_rq *rq, bool xdp, bool request_timestamp) { - return __dev_forward_skb(dev, skb) ?: xdp ? + struct net_device *orig_dev = skb->dev; + int ret; + + ret = __dev_forward_skb(dev, skb); + if (ret) + return ret; + + if (devtx_enabled()) { + struct veth_devtx_frame ctx; + + if (unlikely(request_timestamp)) + __net_timestamp(skb); + + devtx_frame_from_skb(&ctx.frame, skb, orig_dev); + ctx.frame.data -= ETH_HLEN; /* undo eth_type_trans pull */ + ctx.frame.len += ETH_HLEN; + ctx.skb = skb; + veth_devtx_complete(&ctx.frame); + } + + return xdp ? veth_xdp_rx(rq, skb) : __netif_rx(skb); } @@ -343,6 +384,7 @@ static bool veth_skb_is_eligible_for_gro(const struct net_device *dev, static netdev_tx_t veth_xmit(struct sk_buff *skb, struct net_device *dev) { struct veth_priv *rcv_priv, *priv = netdev_priv(dev); + bool request_timestamp = false; struct veth_rq *rq = NULL; struct net_device *rcv; int length = skb->len; @@ -356,6 +398,15 @@ static netdev_tx_t veth_xmit(struct sk_buff *skb, struct net_device *dev) goto drop; } + if (devtx_enabled()) { + struct veth_devtx_frame ctx; + + devtx_frame_from_skb(&ctx.frame, skb, dev); + ctx.request_timestamp = false; + veth_devtx_submit(&ctx.frame); + request_timestamp = ctx.request_timestamp; + } + rcv_priv = netdev_priv(rcv); rxq = skb_get_queue_mapping(skb); if (rxq < rcv->real_num_rx_queues) { @@ -370,7 +421,7 @@ static netdev_tx_t veth_xmit(struct sk_buff *skb, struct net_device *dev) } skb_tx_timestamp(skb); - if (likely(veth_forward_skb(rcv, skb, rq, use_napi) == NET_RX_SUCCESS)) { + if (likely(veth_forward_skb(rcv, skb, rq, use_napi, request_timestamp) == NET_RX_SUCCESS)) { if (!use_napi) dev_lstats_add(dev, length); } else { @@ -483,6 +534,7 @@ static int veth_xdp_xmit(struct net_device *dev, int n, { struct veth_priv *rcv_priv, *priv = netdev_priv(dev); int i, ret = -ENXIO, nxmit = 0; + ktime_t tx_timestamp = 0; struct net_device *rcv; unsigned int max_len; struct veth_rq *rq; @@ -511,9 +563,32 @@ static int veth_xdp_xmit(struct net_device *dev, int n, void *ptr = veth_xdp_to_ptr(frame); if (unlikely(xdp_get_frame_len(frame) > max_len || - __ptr_ring_produce(&rq->xdp_ring, ptr))) + __ptr_ring_full(&rq->xdp_ring))) + break; + + if (devtx_enabled()) { + struct veth_devtx_frame ctx; + + devtx_frame_from_xdp(&ctx.frame, frame, dev); + ctx.request_timestamp = false; + veth_devtx_submit(&ctx.frame); + + if (unlikely(ctx.request_timestamp)) + tx_timestamp = ktime_get_real(); + } + + if (unlikely(__ptr_ring_produce(&rq->xdp_ring, ptr))) break; nxmit++; + + if (devtx_enabled()) { + struct veth_devtx_frame ctx; + + devtx_frame_from_xdp(&ctx.frame, frame, dev); + ctx.xdp_tx_timestamp = tx_timestamp; + ctx.skb = NULL; + veth_devtx_complete(&ctx.frame); + } } spin_unlock(&rq->xdp_ring.producer_lock); @@ -1732,6 +1807,28 @@ static int veth_xdp_rx_hash(const struct xdp_md *ctx, u32 *hash, return 0; } +static int veth_devtx_sb_request_timestamp(const struct devtx_frame *_ctx) +{ + struct veth_devtx_frame *ctx = (struct veth_devtx_frame *)_ctx; + + ctx->request_timestamp = true; + + return 0; +} + +static int veth_devtx_cp_timestamp(const struct devtx_frame *_ctx, u64 *timestamp) +{ + struct veth_devtx_frame *ctx = (struct veth_devtx_frame *)_ctx; + + if (ctx->skb) { + *timestamp = ctx->skb->tstamp; + return 0; + } + + *timestamp = ctx->xdp_tx_timestamp; + return 0; +} + static const struct net_device_ops veth_netdev_ops = { .ndo_init = veth_dev_init, .ndo_open = veth_open, @@ -1756,6 +1853,8 @@ static const struct net_device_ops veth_netdev_ops = { static const struct xdp_metadata_ops veth_xdp_metadata_ops = { .xmo_rx_timestamp = veth_xdp_rx_timestamp, .xmo_rx_hash = veth_xdp_rx_hash, + .xmo_sb_request_timestamp = veth_devtx_sb_request_timestamp, + .xmo_cp_timestamp = veth_devtx_cp_timestamp, }; #define VETH_FEATURES (NETIF_F_SG | NETIF_F_FRAGLIST | NETIF_F_HW_CSUM | \ @@ -2041,11 +2140,20 @@ static struct rtnl_link_ops veth_link_ops = { static __init int veth_init(void) { + int ret; + + ret = devtx_hooks_register(&veth_devtx_hook_ids, &veth_xdp_metadata_ops); + if (ret) { + pr_warn("failed to register devtx hooks: %d", ret); + return ret; + } + return rtnl_link_register(&veth_link_ops); } static __exit void veth_exit(void) { + devtx_hooks_unregister(&veth_devtx_hook_ids); rtnl_link_unregister(&veth_link_ops); } From patchwork Wed Jun 21 17:02:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13287730 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F21021B8E2 for ; Wed, 21 Jun 2023 17:03:00 +0000 (UTC) Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0328F10C for ; Wed, 21 Jun 2023 10:03:00 -0700 (PDT) Received: by mail-pg1-x549.google.com with SMTP id 41be03b00d2f7-5538f216c7aso2000790a12.0 for ; Wed, 21 Jun 2023 10:02:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687366979; x=1689958979; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=G8oJ5/OEX1pF9nKL0tJ4RgAA4zY2+vJRV/bxjv3VyJo=; b=yqIO9pNDHL5lnesoiE6dji/e40TET35LO0r4r68gm3RLNDFg78HfKUqrh77HexkKJX l1Aq2FlXV7MY2fYEPYkIJDk/Emuki2dRSX+ZtM+u/qQnddM867uFqJLy0UYK739Gz0HC pn1gffq7MNCUBNZ6QbcTw6fSkxq+j4JJ2UHqq/2etTFTMQXTvfd3bPDr67YOTw9yrcJl tMtCwgFr9RL05VnFxSkWeIBCauyIxJxlSmy8oJ8dpwnzXgbx6IOUWPlANHj9Q2F5PEdD +UkQvymnRdQM/4NbA3mBEYT72ipEfM88lU4npVevujhyHGd0ID7+YADCK0lAH9gdTgG4 ysZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687366979; x=1689958979; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=G8oJ5/OEX1pF9nKL0tJ4RgAA4zY2+vJRV/bxjv3VyJo=; b=XULHcAsKpCN85vhf8OSy65Wz+OcUVr0mCtNvKiOjRszOGk08bVKUKHDilePtxbOvBI jFZ3K6YoJfg2+4vTsj9Nv8SaF7iUMNpXI/egs/gWpoMVMJ7cva+0hBxLp/4KIaJyjADz J0jVhuYKM8s8teuGcDwukbvOW6JBF0Xr9FEIn+4NnZx/uDON4ZdzKcf63IGidwz7rUmp lAu10M+BCjzzoqQDXQZajbw6hc4Rbsqfi1WdDLBycDBmB/+zXeOyQkd4L/CXr5WNQK+0 JGFBoBh/gRzlBsoRtEik8aH85wR/WBC+Dbpw3APxcnUOyyu9119MTKiUOsDRb3/aaIBJ my7Q== X-Gm-Message-State: AC+VfDy2+Kf3Vk+fJll5P22UFU9hQgSpcp7HZD4B6I7iZoG6sbHnKpk3 oVSf/8AZ19QufnjDvzrhzzpilOFNpEjnMMWPiRbxHPTFDvplPGsq9q73HVwZGoD0SyEKHQq7n0A xUpNurcCIgDQti6vkXb/arIDL2UNaGekfuHza5n140MBYnjCcgg== X-Google-Smtp-Source: ACHHUZ7MXaeIj+L3sJhtCwKD2+lI8agv7n+0CugRgLu0uoJG7O3LAjcGs2Hk/XL2sCA1Yg6JLGvA/yQ= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a63:4382:0:b0:551:eb6:1ea6 with SMTP id q124-20020a634382000000b005510eb61ea6mr1810212pga.10.1687366979318; Wed, 21 Jun 2023 10:02:59 -0700 (PDT) Date: Wed, 21 Jun 2023 10:02:40 -0700 In-Reply-To: <20230621170244.1283336-1-sdf@google.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20230621170244.1283336-1-sdf@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621170244.1283336-8-sdf@google.com> Subject: [RFC bpf-next v2 07/11] selftests/xsk: Support XDP_TX_METADATA_LEN From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Add new config field and call setsockopt. Signed-off-by: Stanislav Fomichev --- tools/testing/selftests/bpf/xsk.c | 17 +++++++++++++++++ tools/testing/selftests/bpf/xsk.h | 1 + 2 files changed, 18 insertions(+) diff --git a/tools/testing/selftests/bpf/xsk.c b/tools/testing/selftests/bpf/xsk.c index 687d83e707f8..c659713e2d43 100644 --- a/tools/testing/selftests/bpf/xsk.c +++ b/tools/testing/selftests/bpf/xsk.c @@ -47,6 +47,10 @@ #define PF_XDP AF_XDP #endif +#ifndef XDP_TX_METADATA_LEN +#define XDP_TX_METADATA_LEN 9 +#endif + #define pr_warn(fmt, ...) fprintf(stderr, fmt, ##__VA_ARGS__) #define XSKMAP_SIZE 1 @@ -124,12 +128,14 @@ static int xsk_set_xdp_socket_config(struct xsk_socket_config *cfg, cfg->rx_size = XSK_RING_CONS__DEFAULT_NUM_DESCS; cfg->tx_size = XSK_RING_PROD__DEFAULT_NUM_DESCS; cfg->bind_flags = 0; + cfg->tx_metadata_len = 0; return 0; } cfg->rx_size = usr_cfg->rx_size; cfg->tx_size = usr_cfg->tx_size; cfg->bind_flags = usr_cfg->bind_flags; + cfg->tx_metadata_len = usr_cfg->tx_metadata_len; return 0; } @@ -479,6 +485,17 @@ int xsk_socket__create_shared(struct xsk_socket **xsk_ptr, umem->tx_ring_setup_done = true; } + if (xsk->config.tx_metadata_len) { + int optval = xsk->config.tx_metadata_len; + + err = setsockopt(xsk->fd, SOL_XDP, XDP_TX_METADATA_LEN, + &optval, sizeof(optval)); + if (err) { + err = -errno; + goto out_put_ctx; + } + } + err = xsk_get_mmap_offsets(xsk->fd, &off); if (err) { err = -errno; diff --git a/tools/testing/selftests/bpf/xsk.h b/tools/testing/selftests/bpf/xsk.h index 8da8d557768b..57e0af403aa8 100644 --- a/tools/testing/selftests/bpf/xsk.h +++ b/tools/testing/selftests/bpf/xsk.h @@ -212,6 +212,7 @@ struct xsk_socket_config { __u32 rx_size; __u32 tx_size; __u16 bind_flags; + __u8 tx_metadata_len; }; /* Set config to NULL to get the default configuration. */ From patchwork Wed Jun 21 17:02:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13287731 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E901F19932 for ; Wed, 21 Jun 2023 17:03:02 +0000 (UTC) Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B9FA10C for ; Wed, 21 Jun 2023 10:03:01 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id 98e67ed59e1d1-25e7fe2fb9bso2875933a91.3 for ; Wed, 21 Jun 2023 10:03:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687366981; x=1689958981; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GqzQApHhMin8VkBURTnRCTSovI6vRxx1tWmRLUB4Gqs=; b=4zVrGLl70cCQsEam/VuH1reik/sXInjP2h75Cd1XqJpcbkxBgocdqnz4m3dhZng1zW 2wySJLRXIG0f8vhxlS4JP6U803n+QjX3A2vo+n+j5qGh7d6/VEN74JcJ5Ux2RK6Yk0DR kTje1E9HcEJuU6nO/V3hXUXG8YbmZCMXZklwkWZ07tFb9DcIf1UcmvFFCCkYE3afS4dz dnN49yh9mpXhIl+rO4vWLjpI1Pt+sawYat8G0ybFNkHjGHcb+ddXTgtb8GyDwwbWAP0n Iz3t1nVfg+iBPz0mg7ep77YWAJN7JNulqfBrz6x17TdqtP+OlejzNUoBPo4UDy2gXiDk H12w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687366981; x=1689958981; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GqzQApHhMin8VkBURTnRCTSovI6vRxx1tWmRLUB4Gqs=; b=EnnxqBxO+TQ2qgYiKXsYZZQ23b41+yhDSXcUZyC4Z7vfcJLnD21AHpZLfyn/I2SLZz P/SohwBM4rxafJFCPkL7gXeykxJ2R2tRDgdTNbuw1dVpfrNKnYodb4ayk529NasK5ZlH rDbHwQEeOjM0HwjhKkbUdDs0wKU45U2uFcMb481sISXa9TavfAmudJgrVq0Avn/oihrZ Q2PBw2PFzZcJNkxY/kWv8vjcy0/6lKzo2SKgTVfCxb21VDpcoERru3j/de0WDl+tCP4b eAbbejwxss4Q4/JIgnVTmM2XxZNqhCyulL7tX8zaAdu196eeeVV4atVRRx5Qm+KJzb9U WcAg== X-Gm-Message-State: AC+VfDzbkRZ9YxtonKRjVNl75ROjxMO5fUBOA6OHQ/n8W4g48CSBfyS7 0cP/xtSI9wOH421N0QlkVrMFq8DfF9n8k2LRccLvGv2vazNRNVXUgDy21EE4S5eZAfsRxfCWgHb Y/NJQfPJ4Q9dOszKQWjVUvH9NTRBqCYAP0xqhjOoVCwFj86NmQg== X-Google-Smtp-Source: ACHHUZ7weWorroVTGt8F/JdLZRT+7IQ5TKHAqhzsttedPPzlsv8xagQtAuCbgamA1zqw3sRdjYZPsmc= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a17:90a:c28a:b0:255:f691:9289 with SMTP id f10-20020a17090ac28a00b00255f6919289mr2234070pjt.9.1687366980806; Wed, 21 Jun 2023 10:03:00 -0700 (PDT) Date: Wed, 21 Jun 2023 10:02:41 -0700 In-Reply-To: <20230621170244.1283336-1-sdf@google.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20230621170244.1283336-1-sdf@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621170244.1283336-9-sdf@google.com> Subject: [RFC bpf-next v2 08/11] selftests/bpf: Add helper to query current netns cookie From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Will be used by the subsequent selftests. Signed-off-by: Stanislav Fomichev --- tools/testing/selftests/bpf/network_helpers.c | 21 +++++++++++++++++++ tools/testing/selftests/bpf/network_helpers.h | 1 + 2 files changed, 22 insertions(+) diff --git a/tools/testing/selftests/bpf/network_helpers.c b/tools/testing/selftests/bpf/network_helpers.c index a105c0cd008a..34102fce5a88 100644 --- a/tools/testing/selftests/bpf/network_helpers.c +++ b/tools/testing/selftests/bpf/network_helpers.c @@ -450,3 +450,24 @@ int get_socket_local_port(int sock_fd) return -1; } + +#ifndef SO_NETNS_COOKIE +#define SO_NETNS_COOKIE 71 +#endif + +__u64 get_net_cookie(void) +{ + socklen_t optlen; + __u64 optval = 0; + int fd; + + fd = socket(AF_LOCAL, SOCK_DGRAM, 0); + if (fd >= 0) { + optlen = sizeof(optval); + getsockopt(fd, SOL_SOCKET, SO_NETNS_COOKIE, &optval, &optlen); + close(fd); + } + + return optval; +} + diff --git a/tools/testing/selftests/bpf/network_helpers.h b/tools/testing/selftests/bpf/network_helpers.h index 694185644da6..380047161aac 100644 --- a/tools/testing/selftests/bpf/network_helpers.h +++ b/tools/testing/selftests/bpf/network_helpers.h @@ -57,6 +57,7 @@ int make_sockaddr(int family, const char *addr_str, __u16 port, struct sockaddr_storage *addr, socklen_t *len); char *ping_command(int family); int get_socket_local_port(int sock_fd); +__u64 get_net_cookie(void); struct nstoken; /** From patchwork Wed Jun 21 17:02:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13287732 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 104321B90D for ; Wed, 21 Jun 2023 17:03:05 +0000 (UTC) Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F1015122 for ; Wed, 21 Jun 2023 10:03:02 -0700 (PDT) Received: by mail-pg1-x549.google.com with SMTP id 41be03b00d2f7-553d27fe4baso2160780a12.2 for ; Wed, 21 Jun 2023 10:03:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687366982; x=1689958982; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=m14zc5Ne3cCtu1j4SDPO41X9Yg8gCGGAV8qkGbFFLV4=; b=YhSyFM3AdxpwH9+6/7xUERw/pa56cCy4Y842C2uUQDnzE/jXH9HWWRW1WUXo9DXux/ P4qFXTJ8iP+GYBPlxSnweF6jhwV3iaa1aWPl/+FjDpfcc8qrYH8LjjeGK4PlxLBckvzl wTsgasMe1aTVQRxTh1g5DxgfFWagxHf25yP3TKu0pn7QQICGfjfnct06s3Uga8UaORru 9L3RG53EAjJdrJubKpCPfbM7JPui/WK20qxJ+RV2nEDMrhBhr1ZJrp8oImH64rh2d+8U jr9UN845P7Ycamae6T6ZnK6N32zO7DUaYk6datQWNAxjPYNawBNtvZbLs+fLrCb+bXTe 3tzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687366982; x=1689958982; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=m14zc5Ne3cCtu1j4SDPO41X9Yg8gCGGAV8qkGbFFLV4=; b=Pw4HvEOplH3RuUfH9ttq4ohcrTY2/opjmOAgh6rXlSs6s388gS6o9w66LoUHeM2OTs cp4Kg0pavMn0tFOTTSjuWLCAb9eFoI292r1CauSeSfE0EVsIp4gcWGfOkjgFOL5FtLgk YQ0aePXCiFYE2FtABdNAxtwtD/BdR0xXqrEszHyvJM5KhttJJ6YB0QRFEXgKYjtfZDJW DDDo8o1pX5fU6AXk6CBIMOrCjgHefpMbUZPKXgg2K6884uE/f1s7eikDLdC5z+ircU5x FzisRxrcwXdiSMRLTCLGRSqEIqvcaKhS1Qi2N94LWMno99EwN3Hy6HnTHdeP3nCJbNYR C1pg== X-Gm-Message-State: AC+VfDz/8FcAKRy9X4DXIbj7VMef0p2P7PXRHTlQ4tFhWbSDlJhyTKvK /R+1jVni2zlcrAiYh1btvmAFINrMwsztCnyZJgF9t3QIGAo4Rk3vWPQ4vLR3ebIPd6edUwvgb2J ZNcZkecAokfZoLS3QcgPlSnWpz5NhQ+HY6J/DTHKn6VkQZNYQzQ== X-Google-Smtp-Source: ACHHUZ48yecyMc2NsRJfrsCYDLlxKGn6RUEUPaYAntNMBmsDofvGlaR9Ec/L2T1AKc1Q+t+Qufj8On4= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a63:1f05:0:b0:557:3747:87b8 with SMTP id f5-20020a631f05000000b00557374787b8mr189968pgf.0.1687366982475; Wed, 21 Jun 2023 10:03:02 -0700 (PDT) Date: Wed, 21 Jun 2023 10:02:42 -0700 In-Reply-To: <20230621170244.1283336-1-sdf@google.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20230621170244.1283336-1-sdf@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621170244.1283336-10-sdf@google.com> Subject: [RFC bpf-next v2 09/11] selftests/bpf: Extend xdp_metadata with devtx kfuncs From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, netdev@vger.kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Attach kfuncs that request and report TX timestamp via ringbuf. Confirm on the userspace side that the program has triggered and the timestamp is non-zero. Also make sure devtx_frame has a sensible pointers and data. Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- .../selftests/bpf/prog_tests/xdp_metadata.c | 62 ++++++++- .../selftests/bpf/progs/xdp_metadata.c | 118 ++++++++++++++++++ tools/testing/selftests/bpf/xdp_metadata.h | 14 +++ 3 files changed, 191 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/xdp_metadata.c b/tools/testing/selftests/bpf/prog_tests/xdp_metadata.c index 626c461fa34d..ca4f3106ce6d 100644 --- a/tools/testing/selftests/bpf/prog_tests/xdp_metadata.c +++ b/tools/testing/selftests/bpf/prog_tests/xdp_metadata.c @@ -42,6 +42,9 @@ struct xsk { struct xsk_ring_prod tx; struct xsk_ring_cons rx; struct xsk_socket *socket; + int tx_completions; + u32 last_tx_timestamp_retval; + u64 last_tx_timestamp; }; static int open_xsk(int ifindex, struct xsk *xsk) @@ -51,6 +54,7 @@ static int open_xsk(int ifindex, struct xsk *xsk) .rx_size = XSK_RING_PROD__DEFAULT_NUM_DESCS, .tx_size = XSK_RING_PROD__DEFAULT_NUM_DESCS, .bind_flags = XDP_COPY, + .tx_metadata_len = TX_META_LEN, }; const struct xsk_umem_config umem_config = { .fill_size = XSK_RING_PROD__DEFAULT_NUM_DESCS, @@ -138,6 +142,7 @@ static void ip_csum(struct iphdr *iph) static int generate_packet(struct xsk *xsk, __u16 dst_port) { + struct xdp_tx_meta *meta; struct xdp_desc *tx_desc; struct udphdr *udph; struct ethhdr *eth; @@ -151,10 +156,13 @@ static int generate_packet(struct xsk *xsk, __u16 dst_port) return -1; tx_desc = xsk_ring_prod__tx_desc(&xsk->tx, idx); - tx_desc->addr = idx % (UMEM_NUM / 2) * UMEM_FRAME_SIZE; + tx_desc->addr = idx % (UMEM_NUM / 2) * UMEM_FRAME_SIZE + TX_META_LEN; printf("%p: tx_desc[%u]->addr=%llx\n", xsk, idx, tx_desc->addr); data = xsk_umem__get_data(xsk->umem_area, tx_desc->addr); + meta = data - TX_META_LEN; + meta->request_timestamp = 1; + eth = data; iph = (void *)(eth + 1); udph = (void *)(iph + 1); @@ -192,7 +200,8 @@ static int generate_packet(struct xsk *xsk, __u16 dst_port) return 0; } -static void complete_tx(struct xsk *xsk) +static void complete_tx(struct xsk *xsk, struct xdp_metadata *bpf_obj, + struct ring_buffer *ringbuf) { __u32 idx; __u64 addr; @@ -202,6 +211,13 @@ static void complete_tx(struct xsk *xsk) printf("%p: complete tx idx=%u addr=%llx\n", xsk, idx, addr); xsk_ring_cons__release(&xsk->comp, 1); + + ring_buffer__poll(ringbuf, 1000); + + ASSERT_EQ(bpf_obj->bss->pkts_fail_tx, 0, "pkts_fail_tx"); + ASSERT_GE(xsk->tx_completions, 1, "tx_completions"); + ASSERT_EQ(xsk->last_tx_timestamp_retval, 0, "last_tx_timestamp_retval"); + ASSERT_GE(xsk->last_tx_timestamp, 0, "last_tx_timestamp"); } } @@ -276,8 +292,24 @@ static int verify_xsk_metadata(struct xsk *xsk) return 0; } +static int process_sample(void *ctx, void *data, size_t len) +{ + struct devtx_sample *sample = data; + struct xsk *xsk = ctx; + + printf("%p: got tx timestamp sample %u %llu\n", + xsk, sample->timestamp_retval, sample->timestamp); + + xsk->tx_completions++; + xsk->last_tx_timestamp_retval = sample->timestamp_retval; + xsk->last_tx_timestamp = sample->timestamp; + + return 0; +} + void test_xdp_metadata(void) { + struct ring_buffer *tx_compl_ringbuf = NULL; struct xdp_metadata2 *bpf_obj2 = NULL; struct xdp_metadata *bpf_obj = NULL; struct bpf_program *new_prog, *prog; @@ -290,6 +322,7 @@ void test_xdp_metadata(void) int retries = 10; int rx_ifindex; int tx_ifindex; + int syscall_fd; int sock_fd; int ret; @@ -323,6 +356,14 @@ void test_xdp_metadata(void) if (!ASSERT_OK_PTR(bpf_obj, "open skeleton")) goto out; + prog = bpf_object__find_program_by_name(bpf_obj->obj, "tx_submit"); + bpf_program__set_ifindex(prog, tx_ifindex); + bpf_program__set_flags(prog, BPF_F_XDP_DEV_BOUND_ONLY); + + prog = bpf_object__find_program_by_name(bpf_obj->obj, "tx_complete"); + bpf_program__set_ifindex(prog, tx_ifindex); + bpf_program__set_flags(prog, BPF_F_XDP_DEV_BOUND_ONLY); + prog = bpf_object__find_program_by_name(bpf_obj->obj, "rx"); bpf_program__set_ifindex(prog, rx_ifindex); bpf_program__set_flags(prog, BPF_F_XDP_DEV_BOUND_ONLY); @@ -330,6 +371,18 @@ void test_xdp_metadata(void) if (!ASSERT_OK(xdp_metadata__load(bpf_obj), "load skeleton")) goto out; + bpf_obj->data->ifindex = tx_ifindex; + bpf_obj->data->net_cookie = get_net_cookie(); + + ret = xdp_metadata__attach(bpf_obj); + if (!ASSERT_OK(ret, "xdp_metadata__attach")) + goto out; + + tx_compl_ringbuf = ring_buffer__new(bpf_map__fd(bpf_obj->maps.tx_compl_buf), + process_sample, &tx_xsk, NULL); + if (!ASSERT_OK_PTR(tx_compl_ringbuf, "ring_buffer__new")) + goto out; + /* Make sure we can't add dev-bound programs to prog maps. */ prog_arr = bpf_object__find_map_by_name(bpf_obj->obj, "prog_arr"); if (!ASSERT_OK_PTR(prog_arr, "no prog_arr map")) @@ -364,7 +417,8 @@ void test_xdp_metadata(void) "verify_xsk_metadata")) goto out; - complete_tx(&tx_xsk); + /* Verify AF_XDP TX packet has completion event with a timestamp. */ + complete_tx(&tx_xsk, bpf_obj, tx_compl_ringbuf); /* Make sure freplace correctly picks up original bound device * and doesn't crash. @@ -402,5 +456,7 @@ void test_xdp_metadata(void) xdp_metadata__destroy(bpf_obj); if (tok) close_netns(tok); + if (tx_compl_ringbuf) + ring_buffer__free(tx_compl_ringbuf); SYS_NOFAIL("ip netns del xdp_metadata"); } diff --git a/tools/testing/selftests/bpf/progs/xdp_metadata.c b/tools/testing/selftests/bpf/progs/xdp_metadata.c index d151d406a123..fc025183d45a 100644 --- a/tools/testing/selftests/bpf/progs/xdp_metadata.c +++ b/tools/testing/selftests/bpf/progs/xdp_metadata.c @@ -4,6 +4,11 @@ #include "xdp_metadata.h" #include #include +#include + +#ifndef ETH_P_IP +#define ETH_P_IP 0x0800 +#endif struct { __uint(type, BPF_MAP_TYPE_XSKMAP); @@ -19,10 +24,25 @@ struct { __type(value, __u32); } prog_arr SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_RINGBUF); + __uint(max_entries, 10); +} tx_compl_buf SEC(".maps"); + +__u64 pkts_fail_tx = 0; + +int ifindex = -1; +__u64 net_cookie = -1; + extern int bpf_xdp_metadata_rx_timestamp(const struct xdp_md *ctx, __u64 *timestamp) __ksym; extern int bpf_xdp_metadata_rx_hash(const struct xdp_md *ctx, __u32 *hash, enum xdp_rss_hash_type *rss_type) __ksym; +extern int bpf_devtx_sb_request_timestamp(const struct devtx_frame *ctx) __ksym; +extern int bpf_devtx_cp_timestamp(const struct devtx_frame *ctx, __u64 *timestamp) __ksym; + +extern int bpf_devtx_sb_attach(int ifindex, int prog_fd) __ksym; +extern int bpf_devtx_cp_attach(int ifindex, int prog_fd) __ksym; SEC("xdp") int rx(struct xdp_md *ctx) @@ -61,4 +81,102 @@ int rx(struct xdp_md *ctx) return bpf_redirect_map(&xsk, ctx->rx_queue_index, XDP_PASS); } +static inline int verify_frame(const struct devtx_frame *frame) +{ + struct ethhdr eth = {}; + + /* all the pointers are set up correctly */ + if (!frame->data) + return -1; + if (!frame->sinfo) + return -1; + + /* can get to the frags */ + if (frame->sinfo->nr_frags != 0) + return -1; + if (frame->sinfo->frags[0].bv_page != 0) + return -1; + if (frame->sinfo->frags[0].bv_len != 0) + return -1; + if (frame->sinfo->frags[0].bv_offset != 0) + return -1; + + /* the data has something that looks like ethernet */ + if (frame->len != 46) + return -1; + bpf_probe_read_kernel(ð, sizeof(eth), frame->data); + + if (eth.h_proto != bpf_htons(ETH_P_IP)) + return -1; + + return 0; +} + +SEC("fentry/veth_devtx_submit") +int BPF_PROG(tx_submit, const struct devtx_frame *frame) +{ + struct xdp_tx_meta meta = {}; + int ret; + + if (frame->netdev->ifindex != ifindex) + return 0; + if (frame->netdev->nd_net.net->net_cookie != net_cookie) + return 0; + if (frame->meta_len != TX_META_LEN) + return 0; + + bpf_probe_read_kernel(&meta, sizeof(meta), frame->data - TX_META_LEN); + if (!meta.request_timestamp) + return 0; + + ret = verify_frame(frame); + if (ret < 0) { + __sync_add_and_fetch(&pkts_fail_tx, 1); + return 0; + } + + ret = bpf_devtx_sb_request_timestamp(frame); + if (ret < 0) { + __sync_add_and_fetch(&pkts_fail_tx, 1); + return 0; + } + + return 0; +} + +SEC("fentry/veth_devtx_complete") +int BPF_PROG(tx_complete, const struct devtx_frame *frame) +{ + struct xdp_tx_meta meta = {}; + struct devtx_sample *sample; + int ret; + + if (frame->netdev->ifindex != ifindex) + return 0; + if (frame->netdev->nd_net.net->net_cookie != net_cookie) + return 0; + if (frame->meta_len != TX_META_LEN) + return 0; + + bpf_probe_read_kernel(&meta, sizeof(meta), frame->data - TX_META_LEN); + if (!meta.request_timestamp) + return 0; + + ret = verify_frame(frame); + if (ret < 0) { + __sync_add_and_fetch(&pkts_fail_tx, 1); + return 0; + } + + sample = bpf_ringbuf_reserve(&tx_compl_buf, sizeof(*sample), 0); + if (!sample) + return 0; + + sample->timestamp_retval = bpf_devtx_cp_timestamp(frame, &sample->timestamp); + + bpf_ringbuf_submit(sample, 0); + + return 0; +} + char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/xdp_metadata.h b/tools/testing/selftests/bpf/xdp_metadata.h index 938a729bd307..e410f2b95e64 100644 --- a/tools/testing/selftests/bpf/xdp_metadata.h +++ b/tools/testing/selftests/bpf/xdp_metadata.h @@ -18,3 +18,17 @@ struct xdp_meta { __s32 rx_hash_err; }; }; + +struct devtx_sample { + int timestamp_retval; + __u64 timestamp; +}; + +#define TX_META_LEN 8 + +struct xdp_tx_meta { + __u8 request_timestamp; + __u8 padding0; + __u16 padding1; + __u32 padding2; +}; From patchwork Wed Jun 21 17:02:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13287733 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B907419935 for ; Wed, 21 Jun 2023 17:03:06 +0000 (UTC) Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B8CF2120 for ; Wed, 21 Jun 2023 10:03:04 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-570553a18deso85885117b3.2 for ; Wed, 21 Jun 2023 10:03:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687366984; x=1689958984; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8agh8JqEdvNU3UHvadoBRqAaekfLA9306UePh3v55SI=; b=byWwJVeTVryhCx3j+k5EiX1zDwGl6dx1QXr7yMStP8TaW0PHyQT/VSdXE5XoITg7K9 KnPaTDbtnnb/9QMPZnsYkmctzfV9J15pGTL91txZulHWHL8hcpTMlKL1J1zEacAi//tm JfQ3sPuJ8Yo8HarFSKo2bmW4Ux4zdQt/whNZlxpMSCSk5MeFVTMmMjSZNZsaZo/etj/a +lnEQxnLLzSuNsjAIod9GBoL9/8L7FVTvktLSkk1KXm7hR0kpmvASrH/+CEgAII8Bn4f cUHsEETT6ubH/MBTBFrGuJM3M2DCOBVzK9ZHM1wkst1i9OQLeVZnVZHjjHEpCXPiVcLS Hm7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687366984; x=1689958984; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8agh8JqEdvNU3UHvadoBRqAaekfLA9306UePh3v55SI=; b=GKPaqhtXzuI0W+ZFrscZbs3P/TkQyRX6UstSbqdSplL05Ddv3mQ6cQjowcG+kOOwKa c5gb2s55aBzGA+4IrBfJA0u5MMLJLLd2515hHspPijOZwMIl8qLaWL2h9rbYGgNcvCVQ WDM5P3sTTmqdlzsk2BgWAziJyPb7Vpjl6XfkbfDoE8mki1q1iJDT3aBEl8tHOLqTTRr5 xSORB08LDojYlK3X3FULICp4U2J/62ULYrp2vpg0/0DDh+oBRvBojMrkAI6FKZOJnudL gzruEmGJl4BGlU/VguBG1w8cAflHespYUKZPlHo6EVWnRKzHVHBnouEMBkklUG3JQdPs Kc+w== X-Gm-Message-State: AC+VfDyah6k4dy7EJeoZkxlPGJTKKCEkDfOh+FxBxkwj3umfrIU2zd0v WrcyZxJe3Dr/cCiDN0Vqz02ryvGAD8TBb7Hm7QWI1/IboP+runWYbrJ0x+osGjDJjIhZp+Y5pXi 3n1uLXrKJt/s/kdp/StVemztpAK5prnSDOS+cdO3jj4Hya7LMAw== X-Google-Smtp-Source: ACHHUZ6271UlFgq363ZV7qkRhWhv7ST2ptldt76s4nWKMP7suM4OwuBEqIYIU3dPNlajJ+koaK5rEWw= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a81:ae1f:0:b0:56f:f77c:3c7d with SMTP id m31-20020a81ae1f000000b0056ff77c3c7dmr6681623ywh.3.1687366983966; Wed, 21 Jun 2023 10:03:03 -0700 (PDT) Date: Wed, 21 Jun 2023 10:02:43 -0700 In-Reply-To: <20230621170244.1283336-1-sdf@google.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20230621170244.1283336-1-sdf@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621170244.1283336-11-sdf@google.com> Subject: [RFC bpf-next v2 10/11] selftests/bpf: Extend xdp_hw_metadata with devtx kfuncs From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, netdev@vger.kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC When we get packets on port 9091, we swap src/dst and send it out. At this point, we also request the timestamp and plumb it back to the userspace. The userspace simply prints the timestamp. Haven't really tested, still working on mlx5 patches... Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- .../selftests/bpf/progs/xdp_hw_metadata.c | 107 ++++++++++ tools/testing/selftests/bpf/xdp_hw_metadata.c | 198 ++++++++++++++++-- 2 files changed, 285 insertions(+), 20 deletions(-) diff --git a/tools/testing/selftests/bpf/progs/xdp_hw_metadata.c b/tools/testing/selftests/bpf/progs/xdp_hw_metadata.c index b2dfd7066c6e..84f10d6b11f1 100644 --- a/tools/testing/selftests/bpf/progs/xdp_hw_metadata.c +++ b/tools/testing/selftests/bpf/progs/xdp_hw_metadata.c @@ -4,6 +4,7 @@ #include "xdp_metadata.h" #include #include +#include struct { __uint(type, BPF_MAP_TYPE_XSKMAP); @@ -12,14 +13,30 @@ struct { __type(value, __u32); } xsk SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_RINGBUF); + __uint(max_entries, 10); +} tx_compl_buf SEC(".maps"); + __u64 pkts_skip = 0; +__u64 pkts_tx_skip = 0; __u64 pkts_fail = 0; __u64 pkts_redir = 0; +__u64 pkts_fail_tx = 0; +__u64 pkts_ringbuf_full = 0; + +int ifindex = -1; +__u64 net_cookie = -1; extern int bpf_xdp_metadata_rx_timestamp(const struct xdp_md *ctx, __u64 *timestamp) __ksym; extern int bpf_xdp_metadata_rx_hash(const struct xdp_md *ctx, __u32 *hash, enum xdp_rss_hash_type *rss_type) __ksym; +extern int bpf_devtx_sb_request_timestamp(const struct devtx_frame *ctx) __ksym; +extern int bpf_devtx_cp_timestamp(const struct devtx_frame *ctx, __u64 *timestamp) __ksym; + +extern int bpf_devtx_sb_attach(int ifindex, int prog_fd) __ksym; +extern int bpf_devtx_cp_attach(int ifindex, int prog_fd) __ksym; SEC("xdp") int rx(struct xdp_md *ctx) @@ -90,4 +107,94 @@ int rx(struct xdp_md *ctx) return bpf_redirect_map(&xsk, ctx->rx_queue_index, XDP_PASS); } +/* This is not strictly required; only to showcase how to access the payload. */ +static __always_inline bool tx_filter(const struct devtx_frame *frame) +{ + int port_offset = sizeof(struct ethhdr) + offsetof(struct udphdr, source); + struct ethhdr eth = {}; + struct udphdr udp = {}; + + bpf_probe_read_kernel(ð.h_proto, sizeof(eth.h_proto), + frame->data + offsetof(struct ethhdr, h_proto)); + + if (eth.h_proto == bpf_htons(ETH_P_IP)) { + port_offset += sizeof(struct iphdr); + } else if (eth.h_proto == bpf_htons(ETH_P_IPV6)) { + port_offset += sizeof(struct ipv6hdr); + } else { + __sync_add_and_fetch(&pkts_tx_skip, 1); + return false; + } + + bpf_probe_read_kernel(&udp.source, sizeof(udp.source), frame->data + port_offset); + + /* Replies to UDP:9091 */ + if (udp.source != bpf_htons(9091)) { + __sync_add_and_fetch(&pkts_tx_skip, 1); + return false; + } + + return true; +} + +SEC("fentry") +int BPF_PROG(tx_submit, const struct devtx_frame *frame) +{ + struct xdp_tx_meta meta = {}; + int ret; + + if (frame->netdev->ifindex != ifindex) + return 0; + if (frame->netdev->nd_net.net->net_cookie != net_cookie) + return 0; + if (frame->meta_len != TX_META_LEN) + return 0; + + bpf_probe_read_kernel(&meta, sizeof(meta), frame->data - TX_META_LEN); + if (!meta.request_timestamp) + return 0; + + if (!tx_filter(frame)) + return 0; + + ret = bpf_devtx_sb_request_timestamp(frame); + if (ret < 0) + __sync_add_and_fetch(&pkts_fail_tx, 1); + + return 0; +} + +SEC("fentry") +int BPF_PROG(tx_complete, const struct devtx_frame *frame) +{ + struct xdp_tx_meta meta = {}; + struct devtx_sample *sample; + + if (frame->netdev->ifindex != ifindex) + return 0; + if (frame->netdev->nd_net.net->net_cookie != net_cookie) + return 0; + if (frame->meta_len != TX_META_LEN) + return 0; + + bpf_probe_read_kernel(&meta, sizeof(meta), frame->data - TX_META_LEN); + if (!meta.request_timestamp) + return 0; + + if (!tx_filter(frame)) + return 0; + + sample = bpf_ringbuf_reserve(&tx_compl_buf, sizeof(*sample), 0); + if (!sample) { + __sync_add_and_fetch(&pkts_ringbuf_full, 1); + return 0; + } + + sample->timestamp_retval = bpf_devtx_cp_timestamp(frame, &sample->timestamp); + + bpf_ringbuf_submit(sample, 0); + + return 0; +} + char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/xdp_hw_metadata.c b/tools/testing/selftests/bpf/xdp_hw_metadata.c index 613321eb84c1..0bbe8377a34b 100644 --- a/tools/testing/selftests/bpf/xdp_hw_metadata.c +++ b/tools/testing/selftests/bpf/xdp_hw_metadata.c @@ -10,7 +10,8 @@ * - rx_hash * * TX: - * - TBD + * - UDP 9091 packets trigger TX reply + * - TX HW timestamp is requested and reported back upon completion */ #include @@ -28,6 +29,8 @@ #include #include #include +#include +#include #include "xdp_metadata.h" @@ -54,13 +57,14 @@ int rxq; void test__fail(void) { /* for network_helpers.c */ } -static int open_xsk(int ifindex, struct xsk *xsk, __u32 queue_id) +static int open_xsk(int ifindex, struct xsk *xsk, __u32 queue_id, int flags) { int mmap_flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE; const struct xsk_socket_config socket_config = { .rx_size = XSK_RING_PROD__DEFAULT_NUM_DESCS, .tx_size = XSK_RING_PROD__DEFAULT_NUM_DESCS, - .bind_flags = XDP_COPY, + .bind_flags = flags, + .tx_metadata_len = TX_META_LEN, }; const struct xsk_umem_config umem_config = { .fill_size = XSK_RING_PROD__DEFAULT_NUM_DESCS, @@ -228,7 +232,87 @@ static void verify_skb_metadata(int fd) printf("skb hwtstamp is not found!\n"); } -static int verify_metadata(struct xsk *rx_xsk, int rxq, int server_fd, clockid_t clock_id) +static void complete_tx(struct xsk *xsk, struct ring_buffer *ringbuf) +{ + __u32 idx; + __u64 addr; + + ring_buffer__poll(ringbuf, 1000); + + if (xsk_ring_cons__peek(&xsk->comp, 1, &idx)) { + addr = *xsk_ring_cons__comp_addr(&xsk->comp, idx); + + printf("%p: complete tx idx=%u addr=%llx\n", xsk, idx, addr); + xsk_ring_cons__release(&xsk->comp, 1); + } +} + +#define swap(a, b, len) do { \ + for (int i = 0; i < len; i++) { \ + __u8 tmp = ((__u8 *)a)[i]; \ + ((__u8 *)a)[i] = ((__u8 *)b)[i]; \ + ((__u8 *)b)[i] = tmp; \ + } \ +} while (0) + +static void ping_pong(struct xsk *xsk, void *rx_packet) +{ + struct ipv6hdr *ip6h = NULL; + struct iphdr *iph = NULL; + struct xdp_tx_meta *meta; + struct xdp_desc *tx_desc; + struct udphdr *udph; + struct ethhdr *eth; + void *data; + __u32 idx; + int ret; + int len; + + ret = xsk_ring_prod__reserve(&xsk->tx, 1, &idx); + if (ret != 1) { + printf("%p: failed to reserve tx slot\n", xsk); + return; + } + + tx_desc = xsk_ring_prod__tx_desc(&xsk->tx, idx); + tx_desc->addr = idx % (UMEM_NUM / 2) * UMEM_FRAME_SIZE + TX_META_LEN; + data = xsk_umem__get_data(xsk->umem_area, tx_desc->addr); + + meta = data - TX_META_LEN; + meta->request_timestamp = 1; + + eth = data; + + if (eth->h_proto == htons(ETH_P_IP)) { + iph = (void *)(eth + 1); + udph = (void *)(iph + 1); + } else if (eth->h_proto == htons(ETH_P_IPV6)) { + ip6h = (void *)(eth + 1); + udph = (void *)(ip6h + 1); + } else { + xsk_ring_prod__cancel(&xsk->tx, 1); + return; + } + + len = ETH_HLEN; + if (ip6h) + len += sizeof(*ip6h) + ntohs(ip6h->payload_len); + if (iph) + len += ntohs(iph->tot_len); + + memcpy(data, rx_packet, len); + swap(eth->h_dest, eth->h_source, ETH_ALEN); + if (iph) + swap(&iph->saddr, &iph->daddr, 4); + else + swap(&ip6h->saddr, &ip6h->daddr, 16); + swap(&udph->source, &udph->dest, 2); + + xsk_ring_prod__submit(&xsk->tx, 1); +} + +static int verify_metadata(struct xsk *rx_xsk, int rxq, int server_fd, clockid_t clock_id, + struct ring_buffer *ringbuf) { const struct xdp_desc *rx_desc; struct pollfd fds[rxq + 1]; @@ -251,8 +335,9 @@ static int verify_metadata(struct xsk *rx_xsk, int rxq, int server_fd, clockid_t while (true) { errno = 0; ret = poll(fds, rxq + 1, 1000); - printf("poll: %d (%d) skip=%llu fail=%llu redir=%llu\n", + printf("poll: %d (%d) skip=%llu/%llu fail=%llu redir=%llu\n", ret, errno, bpf_obj->bss->pkts_skip, + bpf_obj->bss->pkts_tx_skip, bpf_obj->bss->pkts_fail, bpf_obj->bss->pkts_redir); if (ret < 0) break; @@ -280,6 +365,11 @@ static int verify_metadata(struct xsk *rx_xsk, int rxq, int server_fd, clockid_t xsk, idx, rx_desc->addr, addr, comp_addr); verify_xdp_metadata(xsk_umem__get_data(xsk->umem_area, addr), clock_id); + + /* mirror packet back */ + ping_pong(xsk, xsk_umem__get_data(xsk->umem_area, addr)); + complete_tx(xsk, ringbuf); + xsk_ring_cons__release(&xsk->rx, 1); refill_rx(xsk, comp_addr); } @@ -373,16 +463,6 @@ static void cleanup(void) int ret; int i; - if (bpf_obj) { - opts.old_prog_fd = bpf_program__fd(bpf_obj->progs.rx); - if (opts.old_prog_fd >= 0) { - printf("detaching bpf program....\n"); - ret = bpf_xdp_detach(ifindex, XDP_FLAGS, &opts); - if (ret) - printf("failed to detach XDP program: %d\n", ret); - } - } - for (i = 0; i < rxq; i++) close_xsk(&rx_xsk[i]); @@ -404,21 +484,69 @@ static void timestamping_enable(int fd, int val) error(1, errno, "setsockopt(SO_TIMESTAMPING)"); } +static int process_sample(void *ctx, void *data, size_t len) +{ + struct devtx_sample *sample = data; + + printf("got tx timestamp sample %u %llu\n", + sample->timestamp_retval, sample->timestamp); + + return 0; +} + +static void usage(const char *prog) +{ + fprintf(stderr, + "usage: %s [OPTS] \n" + "OPTS:\n" + " -s symbol name for tx_submit\n" + " -c symbol name for tx_complete\n" + " -C run in copy mode\n", + prog); +} + int main(int argc, char *argv[]) { + struct ring_buffer *tx_compl_ringbuf = NULL; clockid_t clock_id = CLOCK_TAI; + char *tx_complete = NULL; + char *tx_submit = NULL; + int bind_flags = 0; int server_fd = -1; + int opt; int ret; int i; struct bpf_program *prog; - if (argc != 2) { + while ((opt = getopt(argc, argv, "s:c:C")) != -1) { + switch (opt) { + case 's': + tx_submit = optarg; + break; + case 'c': + tx_complete = optarg; + break; + case 'C': + bind_flags |= XDP_COPY; + break; + default: + usage(basename(argv[0])); + return 1; + } + } + + if (argc < 2) { fprintf(stderr, "pass device name\n"); return -1; } - ifname = argv[1]; + if (optind >= argc) { + usage(basename(argv[0])); + return 1; + } + + ifname = argv[optind]; ifindex = if_nametoindex(ifname); rxq = rxq_num(ifname); @@ -432,7 +560,7 @@ int main(int argc, char *argv[]) for (i = 0; i < rxq; i++) { printf("open_xsk(%s, %p, %d)\n", ifname, &rx_xsk[i], i); - ret = open_xsk(ifindex, &rx_xsk[i], i); + ret = open_xsk(ifindex, &rx_xsk[i], i, bind_flags); if (ret) error(1, -ret, "open_xsk"); @@ -444,15 +572,45 @@ int main(int argc, char *argv[]) if (libbpf_get_error(bpf_obj)) error(1, libbpf_get_error(bpf_obj), "xdp_hw_metadata__open"); + bpf_obj->data->ifindex = ifindex; + bpf_obj->data->net_cookie = get_net_cookie(); + prog = bpf_object__find_program_by_name(bpf_obj->obj, "rx"); bpf_program__set_ifindex(prog, ifindex); bpf_program__set_flags(prog, BPF_F_XDP_DEV_BOUND_ONLY); + prog = bpf_object__find_program_by_name(bpf_obj->obj, "tx_submit"); + bpf_program__set_ifindex(prog, ifindex); + bpf_program__set_flags(prog, BPF_F_XDP_DEV_BOUND_ONLY); + if (tx_submit) { + printf("attaching devtx submit program to %s\n", tx_submit); + bpf_program__set_attach_target(prog, 0, tx_submit); + } else { + printf("skipping devtx submit program\n"); + bpf_program__set_autoattach(prog, false); + } + + prog = bpf_object__find_program_by_name(bpf_obj->obj, "tx_complete"); + bpf_program__set_ifindex(prog, ifindex); + bpf_program__set_flags(prog, BPF_F_XDP_DEV_BOUND_ONLY); + if (tx_complete) { + printf("attaching devtx complete program to %s\n", tx_complete); + bpf_program__set_attach_target(prog, 0, tx_complete); + } else { + printf("skipping devtx complete program\n"); + bpf_program__set_autoattach(prog, false); + } + printf("load bpf program...\n"); ret = xdp_hw_metadata__load(bpf_obj); if (ret) error(1, -ret, "xdp_hw_metadata__load"); + tx_compl_ringbuf = ring_buffer__new(bpf_map__fd(bpf_obj->maps.tx_compl_buf), + process_sample, NULL, NULL); + if (libbpf_get_error(tx_compl_ringbuf)) + error(1, -libbpf_get_error(tx_compl_ringbuf), "ring_buffer__new"); + printf("prepare skb endpoint...\n"); server_fd = start_server(AF_INET6, SOCK_DGRAM, NULL, 9092, 1000); if (server_fd < 0) @@ -472,7 +630,7 @@ int main(int argc, char *argv[]) error(1, -ret, "bpf_map_update_elem"); } - printf("attach bpf program...\n"); + printf("attach rx bpf program...\n"); ret = bpf_xdp_attach(ifindex, bpf_program__fd(bpf_obj->progs.rx), XDP_FLAGS, NULL); @@ -480,7 +638,7 @@ int main(int argc, char *argv[]) error(1, -ret, "bpf_xdp_attach"); signal(SIGINT, handle_signal); - ret = verify_metadata(rx_xsk, rxq, server_fd, clock_id); + ret = verify_metadata(rx_xsk, rxq, server_fd, clock_id, tx_compl_ringbuf); close(server_fd); cleanup(); if (ret) From patchwork Wed Jun 21 17:02:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13287734 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0539B19937 for ; Wed, 21 Jun 2023 17:03:08 +0000 (UTC) Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 34E46129 for ; Wed, 21 Jun 2023 10:03:06 -0700 (PDT) Received: by mail-pf1-x44a.google.com with SMTP id d2e1a72fcca58-666e3b5d305so2753109b3a.2 for ; Wed, 21 Jun 2023 10:03:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687366985; x=1689958985; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OwbBOZ5mIbfZQrRhwVaMrHD5n/o6Wow9MHXRzVGufzQ=; b=YJ+bi+DGhEfp/XX6mXzR6WDx/uLoEQofae8or87da/BvxPxN7nm6EXG2w1Jn2R0nUr gfq/A/qu2FjUstsORErpILsVE1TrX7N6kzpYOdLNHZoaFTGNDBBMSzRb9VB8I5ob0lBO hR/Dwvm3mVMYbs7xA11K+t8I1+pGQB4zQYrMCgvLczlX7zDI94K/zBYyGbAHPeKtHBsb lzxu/Lo6Y14VmH2oInFYoMiFeb1AAZ17JKiZXS26EHLzUBin2DdDedqygOMV+6F6xHKL 89NTfmspiD/mPeiHxlyTVQc1sMGPAhJoY8I8yFiV++27QoLEatI8xA+XtcYSbBNPFnLK knSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687366985; x=1689958985; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OwbBOZ5mIbfZQrRhwVaMrHD5n/o6Wow9MHXRzVGufzQ=; b=WGhVrZkTLbzQvYva0gzhE1OKGP/yVxr7I7umqJHD5dmoS54UYJiuMYYSjUaYNR8+2X RcnhHRhvay71rUXdaG/ohG8TKPw6my16Pr9xNyDE1rEkT9loZJ7GdbhmV0LLW1OcIbee KfTkkVZ9QzdKfcck/xanhcLV9zjUvdYSo7mFYr0I+eRVtX5bTx/CufkZP3h58wSDqgOy 7xHrblWAJYO36RJa799hEzj6CZMuHrUTeHLTmOK4oYieFY3aEt/onhllWgoyrdCE6rMK t1A/BRkNlYHGKkyVRdiHxLtedDFr6N26CAzp7x9hJ62E+PhxE9zEwkpQ0Zg2c1wjIdAz Soqg== X-Gm-Message-State: AC+VfDzKn1mZCpeT7CBVHgdFqxMtJcjLqranF9s7S2lcNHRhGvaUuHdD 7MdH2oLqAYinifsc5Qbdx2Xd8fGJYBeNTbOl06p1PALArpSDM5dRtXADVP062IB9KYDqo4v4pVw xxq+MCX7aA8dbDEV7Os7mLAysqvz1uVShC7WtYrllls/7DQzQfA== X-Google-Smtp-Source: ACHHUZ4vseU2C7wszAZFipl1H5C7vWcKJw1hh0yHCLV1qgakjL/WIEjP5KQqVZeAvBAfusdrcnRxIJk= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a05:6a00:2d82:b0:668:7260:bbbb with SMTP id fb2-20020a056a002d8200b006687260bbbbmr2817243pfb.0.1687366985572; Wed, 21 Jun 2023 10:03:05 -0700 (PDT) Date: Wed, 21 Jun 2023 10:02:44 -0700 In-Reply-To: <20230621170244.1283336-1-sdf@google.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20230621170244.1283336-1-sdf@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621170244.1283336-12-sdf@google.com> Subject: [RFC bpf-next v2 11/11] net/mlx5e: Support TX timestamp metadata From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, netdev@vger.kernel.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC WIP, not tested, only to show the overall idea. Non-AF_XDP paths are marked with 'false' for now. Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- .../net/ethernet/mellanox/mlx5/core/en/txrx.h | 11 +++ .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 96 ++++++++++++++++++- .../net/ethernet/mellanox/mlx5/core/en/xdp.h | 9 +- .../ethernet/mellanox/mlx5/core/en/xsk/tx.c | 3 + .../net/ethernet/mellanox/mlx5/core/en_tx.c | 16 ++++ .../net/ethernet/mellanox/mlx5/core/main.c | 26 ++++- 6 files changed, 156 insertions(+), 5 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h index 879d698b6119..e4509464e0b1 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/txrx.h @@ -6,6 +6,7 @@ #include "en.h" #include +#include #define MLX5E_TX_WQE_EMPTY_DS_COUNT (sizeof(struct mlx5e_tx_wqe) / MLX5_SEND_WQE_DS) @@ -506,4 +507,14 @@ static inline struct mlx5e_mpw_info *mlx5e_get_mpw_info(struct mlx5e_rq *rq, int return (struct mlx5e_mpw_info *)((char *)rq->mpwqe.info + array_size(i, isz)); } + +struct mlx5e_devtx_frame { + struct devtx_frame frame; + struct mlx5_cqe64 *cqe; /* tx completion */ + struct mlx5e_tx_wqe *wqe; /* tx */ +}; + +void mlx5e_devtx_submit(struct devtx_frame *ctx); +void mlx5e_devtx_complete(struct devtx_frame *ctx); + #endif diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c index f0e6095809fa..0cb0f0799cbc 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c @@ -255,9 +255,30 @@ static int mlx5e_xdp_rx_hash(const struct xdp_md *ctx, u32 *hash, return 0; } +static int mlx5e_devtx_sb_request_timestamp(const struct devtx_frame *ctx) +{ + /* Nothing to do here, CQE always has a timestamp. */ + return 0; +} + +static int mlx5e_devtx_cp_timestamp(const struct devtx_frame *_ctx, u64 *timestamp) +{ + const struct mlx5e_devtx_frame *ctx = (void *)_ctx; + u64 ts; + + if (unlikely(!ctx->cqe)) + return -ENODATA; + + ts = get_cqe_ts(ctx->cqe); + *timestamp = mlx5_real_time_cyc2time(NULL, ts); + return 0; +} + const struct xdp_metadata_ops mlx5e_xdp_metadata_ops = { .xmo_rx_timestamp = mlx5e_xdp_rx_timestamp, .xmo_rx_hash = mlx5e_xdp_rx_hash, + .xmo_sb_request_timestamp = mlx5e_devtx_sb_request_timestamp, + .xmo_cp_timestamp = mlx5e_devtx_cp_timestamp, }; /* returns true if packet was consumed by xdp */ @@ -453,6 +474,23 @@ mlx5e_xmit_xdp_frame_mpwqe(struct mlx5e_xdpsq *sq, struct mlx5e_xmit_data *xdptx mlx5e_xdp_mpwqe_add_dseg(sq, p, stats); + if (devtx_enabled()) { + struct mlx5e_xmit_data_frags *xdptxdf = + container_of(xdptxd, struct mlx5e_xmit_data_frags, xd); + + struct mlx5e_devtx_frame ctx = { + .frame = { + .data = p->data, + .len = p->len, + .meta_len = sq->xsk_pool->tx_metadata_len, + .sinfo = xdptxd->has_frags ? xdptxdf->sinfo : NULL, + .netdev = sq->cq.netdev, + }, + .wqe = sq->mpwqe.wqe, + }; + mlx5e_devtx_submit(&ctx.frame); + } + if (unlikely(mlx5e_xdp_mpwqe_is_full(session, sq->max_sq_mpw_wqebbs))) mlx5e_xdp_mpwqe_complete(sq); @@ -560,6 +598,20 @@ mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq, struct mlx5e_xmit_data *xdptxd, dseg++; } + if (devtx_enabled()) { + struct mlx5e_devtx_frame ctx = { + .frame = { + .data = xdptxd->data, + .len = xdptxd->len, + .meta_len = sq->xsk_pool->tx_metadata_len, + .sinfo = xdptxd->has_frags ? xdptxdf->sinfo : NULL, + .netdev = sq->cq.netdev, + }, + .wqe = wqe, + }; + mlx5e_devtx_submit(&ctx.frame); + } + cseg->opmod_idx_opcode = cpu_to_be32((sq->pc << 8) | MLX5_OPCODE_SEND); if (test_bit(MLX5E_SQ_STATE_XDP_MULTIBUF, &sq->state)) { @@ -607,7 +659,8 @@ mlx5e_xmit_xdp_frame(struct mlx5e_xdpsq *sq, struct mlx5e_xmit_data *xdptxd, static void mlx5e_free_xdpsq_desc(struct mlx5e_xdpsq *sq, struct mlx5e_xdp_wqe_info *wi, u32 *xsk_frames, - struct xdp_frame_bulk *bq) + struct xdp_frame_bulk *bq, + struct mlx5_cqe64 *cqe) { struct mlx5e_xdp_info_fifo *xdpi_fifo = &sq->db.xdpi_fifo; u16 i; @@ -626,6 +679,14 @@ static void mlx5e_free_xdpsq_desc(struct mlx5e_xdpsq *sq, xdpi = mlx5e_xdpi_fifo_pop(xdpi_fifo); dma_addr = xdpi.frame.dma_addr; + if (false && devtx_enabled()) { + struct mlx5e_devtx_frame ctx; + + devtx_frame_from_xdp(&ctx.frame, xdpf, sq->cq.netdev); + ctx.cqe = cqe; + mlx5e_devtx_complete(&ctx.frame); + } + dma_unmap_single(sq->pdev, dma_addr, xdpf->len, DMA_TO_DEVICE); if (xdp_frame_has_frags(xdpf)) { @@ -659,6 +720,20 @@ static void mlx5e_free_xdpsq_desc(struct mlx5e_xdpsq *sq, xdpi = mlx5e_xdpi_fifo_pop(xdpi_fifo); page = xdpi.page.page; + if (false && devtx_enabled()) { + struct mlx5e_devtx_frame ctx = { + .frame = { + .data = page, + .len = PAGE_SIZE, + .meta_len = sq->xsk_pool->tx_metadata_len, + .netdev = sq->cq.netdev, + }, + .cqe = cqe, + }; + + mlx5e_devtx_complete(&ctx.frame); + } + /* No need to check ((page->pp_magic & ~0x3UL) == PP_SIGNATURE) * as we know this is a page_pool page. */ @@ -670,6 +745,21 @@ static void mlx5e_free_xdpsq_desc(struct mlx5e_xdpsq *sq, } case MLX5E_XDP_XMIT_MODE_XSK: /* AF_XDP send */ + + if (devtx_enabled()) { + struct mlx5e_devtx_frame ctx = { + .frame = { + .data = xdpi.frame.xsk_head, + .len = xdpi.page.xsk_head_len, + .meta_len = sq->xsk_pool->tx_metadata_len, + .netdev = sq->cq.netdev, + }, + .cqe = cqe, + }; + + mlx5e_devtx_complete(&ctx.frame); + } + (*xsk_frames)++; break; default: @@ -720,7 +810,7 @@ bool mlx5e_poll_xdpsq_cq(struct mlx5e_cq *cq) sqcc += wi->num_wqebbs; - mlx5e_free_xdpsq_desc(sq, wi, &xsk_frames, &bq); + mlx5e_free_xdpsq_desc(sq, wi, &xsk_frames, &bq, cqe); } while (!last_wqe); if (unlikely(get_cqe_opcode(cqe) != MLX5_CQE_REQ)) { @@ -767,7 +857,7 @@ void mlx5e_free_xdpsq_descs(struct mlx5e_xdpsq *sq) sq->cc += wi->num_wqebbs; - mlx5e_free_xdpsq_desc(sq, wi, &xsk_frames, &bq); + mlx5e_free_xdpsq_desc(sq, wi, &xsk_frames, &bq, NULL); } xdp_flush_frame_bulk(&bq); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h index 9e8e6184f9e4..860638e1209b 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h @@ -50,6 +50,11 @@ struct mlx5e_xdp_buff { struct mlx5e_rq *rq; }; +struct mlx5e_xdp_md { + struct xdp_md md; + struct mlx5_cqe64 *cqe; +}; + /* XDP packets can be transmitted in different ways. On completion, we need to * distinguish between them to clean up things in a proper way. */ @@ -82,18 +87,20 @@ enum mlx5e_xdp_xmit_mode { * num, page_1, page_2, ... , page_num. * * MLX5E_XDP_XMIT_MODE_XSK: - * none. + * frame.xsk_head + page.xsk_head_len for header portion only. */ union mlx5e_xdp_info { enum mlx5e_xdp_xmit_mode mode; union { struct xdp_frame *xdpf; dma_addr_t dma_addr; + void *xsk_head; } frame; union { struct mlx5e_rq *rq; u8 num; struct page *page; + u32 xsk_head_len; } page; }; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c index 597f319d4770..1b97d6f6a9ba 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/tx.c @@ -96,6 +96,9 @@ bool mlx5e_xsk_tx(struct mlx5e_xdpsq *sq, unsigned int budget) xsk_buff_raw_dma_sync_for_device(pool, xdptxd.dma_addr, xdptxd.len); + xdpi.frame.xsk_head = xdptxd.data; + xdpi.page.xsk_head_len = xdptxd.len; + ret = INDIRECT_CALL_2(sq->xmit_xdp_frame, mlx5e_xmit_xdp_frame_mpwqe, mlx5e_xmit_xdp_frame, sq, &xdptxd, check_result); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c index c7eb6b238c2b..f8d3e210408a 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c @@ -758,6 +758,14 @@ static void mlx5e_tx_wi_consume_fifo_skbs(struct mlx5e_txqsq *sq, struct mlx5e_t for (i = 0; i < wi->num_fifo_pkts; i++) { struct sk_buff *skb = mlx5e_skb_fifo_pop(&sq->db.skb_fifo); + if (false && devtx_enabled()) { + struct mlx5e_devtx_frame ctx = {}; + + devtx_frame_from_skb(&ctx.frame, skb, sq->cq.netdev); + ctx.cqe = cqe; + mlx5e_devtx_complete(&ctx.frame); + } + mlx5e_consume_skb(sq, skb, cqe, napi_budget); } } @@ -826,6 +834,14 @@ bool mlx5e_poll_tx_cq(struct mlx5e_cq *cq, int napi_budget) sqcc += wi->num_wqebbs; if (likely(wi->skb)) { + if (false && devtx_enabled()) { + struct mlx5e_devtx_frame ctx = {}; + + devtx_frame_from_skb(&ctx.frame, wi->skb, cq->netdev); + ctx.cqe = cqe; + mlx5e_devtx_complete(&ctx.frame); + } + mlx5e_tx_wi_dma_unmap(sq, wi, &dma_fifo_cc); mlx5e_consume_skb(sq, wi->skb, cqe, napi_budget); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c index a7eb65cd0bdd..7160389a5bc6 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c @@ -48,6 +48,7 @@ #include #include #include +#include #include "mlx5_core.h" #include "thermal.h" #include "lib/eq.h" @@ -73,6 +74,7 @@ #include "sf/dev/dev.h" #include "sf/sf.h" #include "mlx5_irq.h" +#include "en/xdp.h" MODULE_AUTHOR("Eli Cohen "); MODULE_DESCRIPTION("Mellanox 5th generation network adapters (ConnectX series) core driver"); @@ -2132,6 +2134,19 @@ static void mlx5_core_verify_params(void) } } +__weak noinline void mlx5e_devtx_submit(struct devtx_frame *ctx) +{ +} + +__weak noinline void mlx5e_devtx_complete(struct devtx_frame *ctx) +{ +} + +BTF_SET8_START(mlx5e_devtx_hook_ids) +BTF_ID_FLAGS(func, mlx5e_devtx_submit) +BTF_ID_FLAGS(func, mlx5e_devtx_complete) +BTF_SET8_END(mlx5e_devtx_hook_ids) + static int __init mlx5_init(void) { int err; @@ -2144,9 +2159,15 @@ static int __init mlx5_init(void) mlx5_core_verify_params(); mlx5_register_debugfs(); + err = devtx_hooks_register(&mlx5e_devtx_hook_ids, &mlx5e_xdp_metadata_ops); + if (err) { + pr_warn("failed to register devtx hooks: %d", err); + goto err_debug; + } + err = mlx5e_init(); if (err) - goto err_debug; + goto err_devtx; err = mlx5_sf_driver_register(); if (err) @@ -2162,6 +2183,8 @@ static int __init mlx5_init(void) mlx5_sf_driver_unregister(); err_sf: mlx5e_cleanup(); +err_devtx: + devtx_hooks_unregister(&mlx5e_devtx_hook_ids); err_debug: mlx5_unregister_debugfs(); return err; @@ -2169,6 +2192,7 @@ static int __init mlx5_init(void) static void __exit mlx5_cleanup(void) { + devtx_hooks_unregister(&mlx5e_devtx_hook_ids); pci_unregister_driver(&mlx5_core_driver); mlx5_sf_driver_unregister(); mlx5e_cleanup();