From patchwork Mon Nov 21 18:25:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13051533 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88B7DC43217 for ; Mon, 21 Nov 2022 18:26:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229662AbiKUS0B (ORCPT ); Mon, 21 Nov 2022 13:26:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38520 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229723AbiKUSZ5 (ORCPT ); Mon, 21 Nov 2022 13:25:57 -0500 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8ABA1CFE98 for ; Mon, 21 Nov 2022 10:25:56 -0800 (PST) Received: by mail-pl1-x649.google.com with SMTP id c7-20020a170903234700b0018729febd96so9831665plh.19 for ; Mon, 21 Nov 2022 10:25:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=HOzG/5dEM9DkbawI2NlOgkggkIO8cF5opRTpWfNApEg=; b=CSFHoG6gVZh4SBZCczWClLOJPtmruLeP1CVP4tX94SUl4kZuItSp3nkuM9V+pa4dUC Gim9gX92dMmq4n0R3k7w2ucPQQTGI9MUeok2ruJWffbwNRWCAfnXw+3hlYkyt1GmCDj1 DiDsj46t4OZ1/sEuKpRr+j/nt27flowaV+qpaLje5Q2coskhopWg40G2QmnNWED3gWaU 89LQ5MSr0YqpSrvlDXQsQJXhBZA0IvR01S7XWpQEgAPZBtcqy+KfIFBnI3oquvknnxzO 8F9StkDvJo9iX40ZMZT3K74lAMyJUgngpvaen+Svsqd/pLkdfwWnK+lokaY/uI11XWjf wMvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=HOzG/5dEM9DkbawI2NlOgkggkIO8cF5opRTpWfNApEg=; b=lYVGz+eWt5wR8NXWT1wZcpKfjnyX4e0h0314nRy1Wh+A0mabHNOPdiYssOqYArSgyB sd77JcyfQGf3671tCZi+RIRw/hvYoI2XVzrf+M11duDFbLV7flzGZkxOTwAB0D26u6GA IBzO7FvsErUFJAD88wux49N/T13CModBQ7i9Ym5C3IZdHTBpwthLGFfRd8zOfTQikGgM FWbyHOZpQzsvcNWkHTWmusr1yQtvpduAeqwJHDlc+UzBHR3tMOW38uh4VRJhjHHm2o0q 6PYahwpZRQMCpfhTJ+eyLwz4QEqrQN3W7EcNBeA+MK2qCtCY6Bwir8+xkkIk1fg9woy4 jdVQ== X-Gm-Message-State: ANoB5pn2ktO1HdGWlrQ4+K3rFDL4YSowG18P8Ziczr6AD3vIg76SgKdV rWrWyJwb4eCbukJo4ImLW9EC4uA= X-Google-Smtp-Source: AA0mqf4MB+oLIG61MneSZIxhA4uojAGlFL2tQV0bErJVWZ3UtpqMf1lMDnr+HVx2A3R6bARGixRNhjA= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a17:90a:df8c:b0:20a:fee1:8f69 with SMTP id p12-20020a17090adf8c00b0020afee18f69mr1872407pjv.0.1669055155656; Mon, 21 Nov 2022 10:25:55 -0800 (PST) Date: Mon, 21 Nov 2022 10:25:45 -0800 In-Reply-To: <20221121182552.2152891-1-sdf@google.com> Mime-Version: 1.0 References: <20221121182552.2152891-1-sdf@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog Message-ID: <20221121182552.2152891-2-sdf@google.com> Subject: [PATCH bpf-next v2 1/8] bpf: Document XDP RX metadata From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, David Ahern , Jakub Kicinski , Willem de Bruijn , Jesper Dangaard Brouer , Anatoly Burakov , Alexander Lobakin , Magnus Karlsson , Maryam Tahhan , xdp-hints@xdp-project.net, netdev@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Document all current use-cases and assumptions. Cc: John Fastabend Cc: David Ahern Cc: Martin KaFai Lau Cc: Jakub Kicinski Cc: Willem de Bruijn Cc: Jesper Dangaard Brouer Cc: Anatoly Burakov Cc: Alexander Lobakin Cc: Magnus Karlsson Cc: Maryam Tahhan Cc: xdp-hints@xdp-project.net Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- Documentation/bpf/xdp-rx-metadata.rst | 90 +++++++++++++++++++++++++++ 1 file changed, 90 insertions(+) create mode 100644 Documentation/bpf/xdp-rx-metadata.rst diff --git a/Documentation/bpf/xdp-rx-metadata.rst b/Documentation/bpf/xdp-rx-metadata.rst new file mode 100644 index 000000000000..498eae718275 --- /dev/null +++ b/Documentation/bpf/xdp-rx-metadata.rst @@ -0,0 +1,90 @@ +=============== +XDP RX Metadata +=============== + +XDP programs support creating and passing custom metadata via +``bpf_xdp_adjust_meta``. This metadata can be consumed by the following +entities: + +1. ``AF_XDP`` consumer. +2. Kernel core stack via ``XDP_PASS``. +3. Another device via ``bpf_redirect_map``. +4. Other BPF programs via ``bpf_tail_call``. + +General Design +============== + +XDP has access to a set of kfuncs to manipulate the metadata. Every +device driver implements these kfuncs. The set of kfuncs is +declared in ``include/net/xdp.h`` via ``XDP_METADATA_KFUNC_xxx``. + +Currently, the following kfuncs are supported. In the future, as more +metadata is supported, this set will grow: + +- ``bpf_xdp_metadata_rx_timestamp_supported`` returns true/false to + indicate whether the device supports RX timestamps +- ``bpf_xdp_metadata_rx_timestamp`` returns packet RX timestamp +- ``bpf_xdp_metadata_rx_hash_supported`` returns true/false to + indicate whether the device supports RX hash +- ``bpf_xdp_metadata_rx_hash`` returns packet RX hash + +Within the XDP frame, the metadata layout is as follows:: + + +----------+-----------------+------+ + | headroom | custom metadata | data | + +----------+-----------------+------+ + ^ ^ + | | + xdp_buff->data_meta xdp_buff->data + +AF_XDP +====== + +``AF_XDP`` use-case implies that there is a contract between the BPF program +that redirects XDP frames into the ``XSK`` and the final consumer. +Thus the BPF program manually allocates a fixed number of +bytes out of metadata via ``bpf_xdp_adjust_meta`` and calls a subset +of kfuncs to populate it. User-space ``XSK`` consumer, looks +at ``xsk_umem__get_data() - METADATA_SIZE`` to locate its metadata. + +Here is the ``AF_XDP`` consumer layout (note missing ``data_meta`` pointer):: + + +----------+-----------------+------+ + | headroom | custom metadata | data | + +----------+-----------------+------+ + ^ + | + rx_desc->address + +XDP_PASS +======== + +This is the path where the packets processed by the XDP program are passed +into the kernel. The kernel creates ``skb`` out of the ``xdp_buff`` contents. +Currently, every driver has a custom kernel code to parse the descriptors and +populate ``skb`` metadata when doing this ``xdp_buff->skb`` conversion. +In the future, we'd like to support a case where XDP program can override +some of that metadata. + +The plan of record is to make this path similar to ``bpf_redirect_map`` +so the program can control which metadata is passed to the skb layer. + +bpf_redirect_map +================ + +``bpf_redirect_map`` can redirect the frame to a different device. +In this case we don't know ahead of time whether that final consumer +will further redirect to an ``XSK`` or pass it to the kernel via ``XDP_PASS``. +Additionally, the final consumer doesn't have access to the original +hardware descriptor and can't access any of the original metadata. + +For this use-case, only custom metadata is currently supported. If +the frame is eventually passed to the kernel, the skb created from such +a frame won't have any skb metadata. The ``XSK`` consumer will only +have access to the custom metadata. + +bpf_tail_call +============= + +No special handling here. Tail-called program operates on the same context +as the original one. From patchwork Mon Nov 21 18:25:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13051534 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1D50C43219 for ; Mon, 21 Nov 2022 18:26:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229843AbiKUS0C (ORCPT ); Mon, 21 Nov 2022 13:26:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38556 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229750AbiKUS0A (ORCPT ); Mon, 21 Nov 2022 13:26:00 -0500 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5202ED02D2 for ; Mon, 21 Nov 2022 10:25:58 -0800 (PST) Received: by mail-pg1-x549.google.com with SMTP id 138-20020a630290000000b004708e8a8dcfso7276014pgc.11 for ; Mon, 21 Nov 2022 10:25:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=qJP1roNVztoT+rR4fEg1P2UNNoIGJSxA7B7wMS9AQHs=; b=UPbmH7yiBsnEWi8x2FbxujNPgqFUq9chmh903AQcFucMIR57BjpWjCFFoXqHL7UOW5 aWNXOotLtQlVprn1yi3Cfi73tSDCR88yHLQWUjX3ox4zrDPktAKcS53oQ5Oc11fqxR4i TfxC0kswCvBUUrlzmY6ujKRhWMsjuy/uR1hCQybJoH6HVxip7CLUNSclUxA/y5a2S1KO mqmjvTAkDW7lZsmvscZHVk3nWMXg2tPyeL8ucGsIYL9+DS2iqTOSDxdrk3ab2rvLX0tm LjmYBIlijurwnImZ5hkqF/1cyMbU4bl5pYSmMEKDTps60izF5EmPaw2jbqTTlmRsU8+u haNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qJP1roNVztoT+rR4fEg1P2UNNoIGJSxA7B7wMS9AQHs=; b=29yBtugU+a7K4CinCFqfME2EVssVhN1p6LQcmibgHTNh4GeeqRLR+xW9rQz/ujpOaT Y+hVl9Ff4eDE8HAFEOVc7eRIx8sepX6lzsQff1eIUFFTrn/4HdccfErzTtV/AsaMyF90 hx00H+tfR6N8HihnySbACmhnOu5eLiAKaH3HsM2aRByCVvGSM6i5CWBZ9ww/qaaaCdp3 lk2ger+CrsCRuoWhV85Z6u91hga1cCshBk8xGdJYRAwPv1aDcMtayPlnOMZMJ8+FokQC t+r/NSiRy62TtRL2wUzfMGNYHFpMd6e1+JVoQACcgrGdtNmDx1ra2OurToPNA1jXvvRQ RHXw== X-Gm-Message-State: ANoB5pnb4QZHfg0ULgs37KFlMjN0beuIr4NdQPfdC/8BF8IabvK5kYkM x0wxLe9cI6x1byo9K5VU2+Fnhdk= X-Google-Smtp-Source: AA0mqf7cidmciSuQZ15kDJg8qGG5UECXVkrpy9KZ9xu/+TfbrRyFrAqmlAYCsyaBELB32euuqnM14Q0= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a17:90b:1217:b0:213:36b7:1b77 with SMTP id gl23-20020a17090b121700b0021336b71b77mr27499248pjb.94.1669055157545; Mon, 21 Nov 2022 10:25:57 -0800 (PST) Date: Mon, 21 Nov 2022 10:25:46 -0800 In-Reply-To: <20221121182552.2152891-1-sdf@google.com> Mime-Version: 1.0 References: <20221121182552.2152891-1-sdf@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog Message-ID: <20221121182552.2152891-3-sdf@google.com> Subject: [PATCH bpf-next v2 2/8] bpf: XDP metadata RX kfuncs From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, David Ahern , Jakub Kicinski , Willem de Bruijn , Jesper Dangaard Brouer , Anatoly Burakov , Alexander Lobakin , Magnus Karlsson , Maryam Tahhan , xdp-hints@xdp-project.net, netdev@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net There is an ndo handler per kfunc, the verifier replaces a call to the generic kfunc with a call to the per-device one. For XDP, we define a new kfunc set (xdp_metadata_kfunc_ids) which implements all possible metatada kfuncs. Not all devices have to implement them. If kfunc is not supported by the target device, the default implementation is called instead. Upon loading, if BPF_F_XDP_HAS_METADATA is passed via prog_flags, we treat prog_index as target device for kfunc resolution. Cc: John Fastabend Cc: David Ahern Cc: Martin KaFai Lau Cc: Jakub Kicinski Cc: Willem de Bruijn Cc: Jesper Dangaard Brouer Cc: Anatoly Burakov Cc: Alexander Lobakin Cc: Magnus Karlsson Cc: Maryam Tahhan Cc: xdp-hints@xdp-project.net Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- include/linux/bpf.h | 1 + include/linux/netdevice.h | 5 ++++ include/net/xdp.h | 20 +++++++++++++ include/uapi/linux/bpf.h | 5 ++++ kernel/bpf/core.c | 1 + kernel/bpf/syscall.c | 17 ++++++++++- kernel/bpf/verifier.c | 33 +++++++++++++++++++++ net/core/dev.c | 5 ++++ net/core/xdp.c | 52 ++++++++++++++++++++++++++++++++++ tools/include/uapi/linux/bpf.h | 5 ++++ 10 files changed, 143 insertions(+), 1 deletion(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index c9eafa67f2a2..01d62355d068 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1302,6 +1302,7 @@ struct bpf_prog_aux { struct work_struct work; struct rcu_head rcu; }; + struct net_device *xdp_netdev; /* xdp metadata kfuncs */ }; struct bpf_prog { diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index ddc59ef98500..2878e4869dc8 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -74,6 +74,7 @@ struct udp_tunnel_nic_info; struct udp_tunnel_nic; struct bpf_prog; struct xdp_buff; +struct xdp_md; void synchronize_net(void); void netdev_set_default_ethtool_ops(struct net_device *dev, @@ -1604,6 +1605,10 @@ struct net_device_ops { ktime_t (*ndo_get_tstamp)(struct net_device *dev, const struct skb_shared_hwtstamps *hwtstamps, bool cycles); + bool (*ndo_xdp_rx_timestamp_supported)(const struct xdp_md *ctx); + u64 (*ndo_xdp_rx_timestamp)(const struct xdp_md *ctx); + bool (*ndo_xdp_rx_hash_supported)(const struct xdp_md *ctx); + u32 (*ndo_xdp_rx_hash)(const struct xdp_md *ctx); }; /** diff --git a/include/net/xdp.h b/include/net/xdp.h index 55dbc68bfffc..348aefd467ed 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -7,6 +7,7 @@ #define __LINUX_NET_XDP_H__ #include /* skb_shared_info */ +#include /* btf_id_set8 */ /** * DOC: XDP RX-queue information @@ -409,4 +410,23 @@ void xdp_attachment_setup(struct xdp_attachment_info *info, #define DEV_MAP_BULK_SIZE XDP_BULK_QUEUE_SIZE +#define XDP_METADATA_KFUNC_xxx \ + XDP_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_TIMESTAMP_SUPPORTED, \ + bpf_xdp_metadata_rx_timestamp_supported) \ + XDP_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_TIMESTAMP, \ + bpf_xdp_metadata_rx_timestamp) \ + XDP_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_HASH_SUPPORTED, \ + bpf_xdp_metadata_rx_hash_supported) \ + XDP_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_HASH, \ + bpf_xdp_metadata_rx_hash) \ + +enum { +#define XDP_METADATA_KFUNC(name, str) name, +XDP_METADATA_KFUNC_xxx +#undef XDP_METADATA_KFUNC +MAX_XDP_METADATA_KFUNC, +}; + +u32 xdp_metadata_kfunc_id(int id); + #endif /* __LINUX_NET_XDP_H__ */ diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index ab86145df760..55eda6f0d39b 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1156,6 +1156,11 @@ enum bpf_link_type { */ #define BPF_F_XDP_HAS_FRAGS (1U << 5) +/* If BPF_F_XDP_HAS_METADATA is used in BPF_PROG_LOAD command, the loaded + * program becomes device-bound but can access it's XDP metadata. + */ +#define BPF_F_XDP_HAS_METADATA (1U << 6) + /* link_create.kprobe_multi.flags used in LINK_CREATE command for * BPF_TRACE_KPROBE_MULTI attach type to create return probe. */ diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 2e57fc839a5c..32cb07a8939c 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -2576,6 +2576,7 @@ static void bpf_prog_free_deferred(struct work_struct *work) } else { bpf_jit_free(aux->prog); } + dev_put(aux->xdp_netdev); } void bpf_prog_free(struct bpf_prog *fp) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 35972afb6850..ece7f9234b2d 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2491,7 +2491,8 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr) BPF_F_TEST_STATE_FREQ | BPF_F_SLEEPABLE | BPF_F_TEST_RND_HI32 | - BPF_F_XDP_HAS_FRAGS)) + BPF_F_XDP_HAS_FRAGS | + BPF_F_XDP_HAS_METADATA)) return -EINVAL; if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) && @@ -2579,6 +2580,20 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr) prog->aux->sleepable = attr->prog_flags & BPF_F_SLEEPABLE; prog->aux->xdp_has_frags = attr->prog_flags & BPF_F_XDP_HAS_FRAGS; + if (attr->prog_flags & BPF_F_XDP_HAS_METADATA) { + /* Reuse prog_ifindex to bind to the device + * for XDP metadata kfuncs. + */ + prog->aux->offload_requested = false; + + prog->aux->xdp_netdev = dev_get_by_index(current->nsproxy->net_ns, + attr->prog_ifindex); + if (!prog->aux->xdp_netdev) { + err = -EINVAL; + goto free_prog; + } + } + err = security_bpf_prog_alloc(prog->aux); if (err) goto free_prog; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 9528a066cfa5..315876fa9d30 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -15171,6 +15171,25 @@ static int fixup_call_args(struct bpf_verifier_env *env) return err; } +static int fixup_xdp_kfunc_call(struct bpf_verifier_env *env, u32 func_id) +{ + struct bpf_prog_aux *aux = env->prog->aux; + void *resolved = NULL; + + if (func_id == xdp_metadata_kfunc_id(XDP_METADATA_KFUNC_RX_TIMESTAMP_SUPPORTED)) + resolved = aux->xdp_netdev->netdev_ops->ndo_xdp_rx_timestamp_supported; + else if (func_id == xdp_metadata_kfunc_id(XDP_METADATA_KFUNC_RX_TIMESTAMP)) + resolved = aux->xdp_netdev->netdev_ops->ndo_xdp_rx_timestamp; + else if (func_id == xdp_metadata_kfunc_id(XDP_METADATA_KFUNC_RX_HASH_SUPPORTED)) + resolved = aux->xdp_netdev->netdev_ops->ndo_xdp_rx_hash_supported; + else if (func_id == xdp_metadata_kfunc_id(XDP_METADATA_KFUNC_RX_HASH)) + resolved = aux->xdp_netdev->netdev_ops->ndo_xdp_rx_hash; + + if (resolved) + return BPF_CALL_IMM(resolved); + return 0; +} + static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, struct bpf_insn *insn_buf, int insn_idx, int *cnt) { @@ -15181,6 +15200,15 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, return -EINVAL; } + if (resolve_prog_type(env->prog) == BPF_PROG_TYPE_XDP) { + int imm = fixup_xdp_kfunc_call(env, insn->imm); + + if (imm) { + insn->imm = imm; + return 0; + } + } + /* insn->imm has the btf func_id. Replace it with * an address (relative to __bpf_base_call). */ @@ -15359,6 +15387,11 @@ static int do_misc_fixups(struct bpf_verifier_env *env) if (insn->src_reg == BPF_PSEUDO_CALL) continue; if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) { + if (bpf_prog_is_dev_bound(env->prog->aux)) { + verbose(env, "no metadata kfuncs offload\n"); + return -EINVAL; + } + ret = fixup_kfunc_call(env, insn, insn_buf, i + delta, &cnt); if (ret) return ret; diff --git a/net/core/dev.c b/net/core/dev.c index 117e830cabb0..b4021b7575a2 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -9248,6 +9248,11 @@ static int dev_xdp_attach(struct net_device *dev, struct netlink_ext_ack *extack NL_SET_ERR_MSG(extack, "BPF_XDP_CPUMAP programs can not be attached to a device"); return -EINVAL; } + if (new_prog->aux->xdp_netdev && + new_prog->aux->xdp_netdev->netdev_ops != dev->netdev_ops) { + NL_SET_ERR_MSG(extack, "Cannot attach to a different target device"); + return -EINVAL; + } } /* don't call drivers if the effective program didn't change */ diff --git a/net/core/xdp.c b/net/core/xdp.c index 844c9d99dc0e..e43f7d4ef4cf 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -4,6 +4,7 @@ * Copyright (c) 2017 Jesper Dangaard Brouer, Red Hat Inc. */ #include +#include #include #include #include @@ -709,3 +710,54 @@ struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) return nxdpf; } + +noinline bool bpf_xdp_metadata_rx_timestamp_supported(const struct xdp_md *ctx) +{ + return false; +} + +noinline u64 bpf_xdp_metadata_rx_timestamp(const struct xdp_md *ctx) +{ + return 0; +} + +noinline bool bpf_xdp_metadata_rx_hash_supported(const struct xdp_md *ctx) +{ + return false; +} + +noinline u32 bpf_xdp_metadata_rx_hash(const struct xdp_md *ctx) +{ + return 0; +} + +#ifdef CONFIG_DEBUG_INFO_BTF +BTF_SET8_START(xdp_metadata_kfunc_ids) +#define XDP_METADATA_KFUNC(name, str) BTF_ID_FLAGS(func, str, 0) +XDP_METADATA_KFUNC_xxx +#undef XDP_METADATA_KFUNC +BTF_SET8_END(xdp_metadata_kfunc_ids) + +static const struct btf_kfunc_id_set xdp_metadata_kfunc_set = { + .owner = THIS_MODULE, + .set = &xdp_metadata_kfunc_ids, +}; + +u32 xdp_metadata_kfunc_id(int id) +{ + return xdp_metadata_kfunc_ids.pairs[id].id; +} +EXPORT_SYMBOL_GPL(xdp_metadata_kfunc_id); + +static int __init xdp_metadata_init(void) +{ + return register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &xdp_metadata_kfunc_set); +} +late_initcall(xdp_metadata_init); +#else /* CONFIG_DEBUG_INFO_BTF */ +u32 xdp_metadata_kfunc_id(int id) +{ + return -1; +} +EXPORT_SYMBOL_GPL(xdp_metadata_kfunc_id); +#endif /* CONFIG_DEBUG_INFO_BTF */ diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 6580448e9f77..6b01ac70f564 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -1156,6 +1156,11 @@ enum bpf_link_type { */ #define BPF_F_XDP_HAS_FRAGS (1U << 5) +/* If BPF_F_XDP_HAS_METADATA is used in BPF_PROG_LOAD command, the loaded + * program becomes device-bound but can access it's XDP metadata. + */ +#define BPF_F_XDP_HAS_METADATA (1U << 6) + /* link_create.kprobe_multi.flags used in LINK_CREATE command for * BPF_TRACE_KPROBE_MULTI attach type to create return probe. */ From patchwork Mon Nov 21 18:25:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13051535 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 541C0C4332F for ; Mon, 21 Nov 2022 18:26:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229796AbiKUS0F (ORCPT ); Mon, 21 Nov 2022 13:26:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38584 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229840AbiKUS0C (ORCPT ); Mon, 21 Nov 2022 13:26:02 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C3D81D02E4 for ; Mon, 21 Nov 2022 10:25:59 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id w2-20020a17090a8a0200b002119ea856edso12403139pjn.5 for ; Mon, 21 Nov 2022 10:25:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=XzY0whdCH0LNaf1li3i+ucIqbikEA8y1fjnf4/7T9NE=; b=rDpJ3D836QYRH/Ck97Z1YUaHHYzXKXRaF5X2X0Pe2AQnGqr6ruMTwoHtVocxLshVbi kAaMjPiFAQQPk4Sh++R3hrBZpKQ7UjNuXJoF6QBs8mmdM5l/pOQdXTV0dM9fXAOZQWRb nrOgrNV7SHBCd4M6FIl2BMgUXXgfA7tM3prW3EUIIrJzOt8ykp1JmWo2LQqpI2xfb3D1 D4r0n8cTPPsA2BEslZHPuAdZP6Sv1bdT4g7RoKBde09CJzqznbiCMNhUi4SWlawc+puJ 6nXMhhwP2T0G2RqkRNb5coU/vl+dRWiTRP8eOQgyRHJ2Gt1Zb+0Pree2n6hYatL71Q86 N3iQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XzY0whdCH0LNaf1li3i+ucIqbikEA8y1fjnf4/7T9NE=; b=j61k+WFREG0hsIGLWh2oTmphY+r69iqaGZ84WAnJYxjvisMZGWv//nkTIQ2KuKDgKl v8IGf2V00Q2z4pgenUUgzCU2AKbTUSAK4faRo6ymof0wNZhGIbV7fMaQtGbD2hy85ahn KMcHFdzb63Dfsq1XniDb7ceFht5Sbn+tLPFtrS2kECoh1XwjOKvRfAygeNET4UZkPy2+ moYzTgLY39aH66R42SrL8U6KufGc0bZ5tfe1K/8OMA+jUHWOh4G/aYgHFfqEfaXzwxnc 8Sqt+y4xkRlygs3Wv2r6vOTTNsW+t0VnLsrdcKZZ5Nh8YH3RDsRrXRWds+427HQuaqP2 nzdw== X-Gm-Message-State: ANoB5pl+hokhFLsbhTKEFqZl2+FN5HEt9xnCNI4ztKRgGW6EONNpkZPR +izEW9MY0di0pZPIEqS4NgCR5cE= X-Google-Smtp-Source: AA0mqf6SvhcoRVrAaB/VE4e71ePstJVPsnbmxnNgsRhjAJec1qI7jLdZvhFdh/RtenPWa88bAfelRtQ= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a17:903:c5:b0:187:4467:7aba with SMTP id x5-20020a17090300c500b0018744677abamr12868148plc.61.1669055159307; Mon, 21 Nov 2022 10:25:59 -0800 (PST) Date: Mon, 21 Nov 2022 10:25:47 -0800 In-Reply-To: <20221121182552.2152891-1-sdf@google.com> Mime-Version: 1.0 References: <20221121182552.2152891-1-sdf@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog Message-ID: <20221121182552.2152891-4-sdf@google.com> Subject: [PATCH bpf-next v2 3/8] veth: Introduce veth_xdp_buff wrapper for xdp_buff From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, Jakub Kicinski , Willem de Bruijn , Jesper Dangaard Brouer , Anatoly Burakov , Alexander Lobakin , Magnus Karlsson , Maryam Tahhan , xdp-hints@xdp-project.net, netdev@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net No functional changes. Boilerplate to allow stuffing more data after xdp_buff. Cc: Martin KaFai Lau Cc: Jakub Kicinski Cc: Willem de Bruijn Cc: Jesper Dangaard Brouer Cc: Anatoly Burakov Cc: Alexander Lobakin Cc: Magnus Karlsson Cc: Maryam Tahhan Cc: xdp-hints@xdp-project.net Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- drivers/net/veth.c | 56 +++++++++++++++++++++++++--------------------- 1 file changed, 31 insertions(+), 25 deletions(-) diff --git a/drivers/net/veth.c b/drivers/net/veth.c index 2a4592780141..bbabc592d431 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -116,6 +116,10 @@ static struct { { "peer_ifindex" }, }; +struct veth_xdp_buff { + struct xdp_buff xdp; +}; + static int veth_get_link_ksettings(struct net_device *dev, struct ethtool_link_ksettings *cmd) { @@ -592,23 +596,24 @@ static struct xdp_frame *veth_xdp_rcv_one(struct veth_rq *rq, rcu_read_lock(); xdp_prog = rcu_dereference(rq->xdp_prog); if (likely(xdp_prog)) { - struct xdp_buff xdp; + struct veth_xdp_buff vxbuf; + struct xdp_buff *xdp = &vxbuf.xdp; u32 act; - xdp_convert_frame_to_buff(frame, &xdp); - xdp.rxq = &rq->xdp_rxq; + xdp_convert_frame_to_buff(frame, xdp); + xdp->rxq = &rq->xdp_rxq; - act = bpf_prog_run_xdp(xdp_prog, &xdp); + act = bpf_prog_run_xdp(xdp_prog, xdp); switch (act) { case XDP_PASS: - if (xdp_update_frame_from_buff(&xdp, frame)) + if (xdp_update_frame_from_buff(xdp, frame)) goto err_xdp; break; case XDP_TX: orig_frame = *frame; - xdp.rxq->mem = frame->mem; - if (unlikely(veth_xdp_tx(rq, &xdp, bq) < 0)) { + xdp->rxq->mem = frame->mem; + if (unlikely(veth_xdp_tx(rq, xdp, bq) < 0)) { trace_xdp_exception(rq->dev, xdp_prog, act); frame = &orig_frame; stats->rx_drops++; @@ -619,8 +624,8 @@ static struct xdp_frame *veth_xdp_rcv_one(struct veth_rq *rq, goto xdp_xmit; case XDP_REDIRECT: orig_frame = *frame; - xdp.rxq->mem = frame->mem; - if (xdp_do_redirect(rq->dev, &xdp, xdp_prog)) { + xdp->rxq->mem = frame->mem; + if (xdp_do_redirect(rq->dev, xdp, xdp_prog)) { frame = &orig_frame; stats->rx_drops++; goto err_xdp; @@ -801,7 +806,8 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq, { void *orig_data, *orig_data_end; struct bpf_prog *xdp_prog; - struct xdp_buff xdp; + struct veth_xdp_buff vxbuf; + struct xdp_buff *xdp = &vxbuf.xdp; u32 act, metalen; int off; @@ -815,22 +821,22 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq, } __skb_push(skb, skb->data - skb_mac_header(skb)); - if (veth_convert_skb_to_xdp_buff(rq, &xdp, &skb)) + if (veth_convert_skb_to_xdp_buff(rq, xdp, &skb)) goto drop; - orig_data = xdp.data; - orig_data_end = xdp.data_end; + orig_data = xdp->data; + orig_data_end = xdp->data_end; - act = bpf_prog_run_xdp(xdp_prog, &xdp); + act = bpf_prog_run_xdp(xdp_prog, xdp); switch (act) { case XDP_PASS: break; case XDP_TX: - veth_xdp_get(&xdp); + veth_xdp_get(xdp); consume_skb(skb); - xdp.rxq->mem = rq->xdp_mem; - if (unlikely(veth_xdp_tx(rq, &xdp, bq) < 0)) { + xdp->rxq->mem = rq->xdp_mem; + if (unlikely(veth_xdp_tx(rq, xdp, bq) < 0)) { trace_xdp_exception(rq->dev, xdp_prog, act); stats->rx_drops++; goto err_xdp; @@ -839,10 +845,10 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq, rcu_read_unlock(); goto xdp_xmit; case XDP_REDIRECT: - veth_xdp_get(&xdp); + veth_xdp_get(xdp); consume_skb(skb); - xdp.rxq->mem = rq->xdp_mem; - if (xdp_do_redirect(rq->dev, &xdp, xdp_prog)) { + xdp->rxq->mem = rq->xdp_mem; + if (xdp_do_redirect(rq->dev, xdp, xdp_prog)) { stats->rx_drops++; goto err_xdp; } @@ -862,7 +868,7 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq, rcu_read_unlock(); /* check if bpf_xdp_adjust_head was used */ - off = orig_data - xdp.data; + off = orig_data - xdp->data; if (off > 0) __skb_push(skb, off); else if (off < 0) @@ -871,21 +877,21 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq, skb_reset_mac_header(skb); /* check if bpf_xdp_adjust_tail was used */ - off = xdp.data_end - orig_data_end; + off = xdp->data_end - orig_data_end; if (off != 0) __skb_put(skb, off); /* positive on grow, negative on shrink */ /* XDP frag metadata (e.g. nr_frags) are updated in eBPF helpers * (e.g. bpf_xdp_adjust_tail), we need to update data_len here. */ - if (xdp_buff_has_frags(&xdp)) + if (xdp_buff_has_frags(xdp)) skb->data_len = skb_shinfo(skb)->xdp_frags_size; else skb->data_len = 0; skb->protocol = eth_type_trans(skb, rq->dev); - metalen = xdp.data - xdp.data_meta; + metalen = xdp->data - xdp->data_meta; if (metalen) skb_metadata_set(skb, metalen); out: @@ -898,7 +904,7 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq, return NULL; err_xdp: rcu_read_unlock(); - xdp_return_buff(&xdp); + xdp_return_buff(xdp); xdp_xmit: return NULL; } From patchwork Mon Nov 21 18:25:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13051536 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43ADBC43219 for ; Mon, 21 Nov 2022 18:26:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229888AbiKUS0G (ORCPT ); Mon, 21 Nov 2022 13:26:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38598 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229885AbiKUS0C (ORCPT ); Mon, 21 Nov 2022 13:26:02 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B3EB4CFEB8 for ; Mon, 21 Nov 2022 10:26:01 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-368994f4bc0so120311247b3.14 for ; Mon, 21 Nov 2022 10:26:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wPmPkc5zN4giDUb/1je/PGUtX+no8JSv+Kak++hjwtw=; b=nGfSDe2DYoxrfRJM92Demm7pOqfd7RPX22MBlXes4IY7HfiP5o0gVQa4NY18BS7O3V ijUK8jdXFdYCL01B3H5kIIo9vRVZ8bvATKG4HhZAEp28q0uOFHHbgjfPTY1agIOjyQJy CiUwKVBs92/QDd48/cZYuDFWrUCVsBkCfYXQLatyNh56Bps4QCp+83aXOaEq4CLrf+1q FpFx+jw3f5Hxrgn/yLI9tKsF1+Dw4lWFkEcBvCaJKW50nrqHN2PuBKfIopRRoKLak24J qCC6r2WODR39cXj0X8pj0gkBrSR+A/KeplR8T90XKvkqVxycIHUna7tm/9z9gQhxaRVN 2Kaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wPmPkc5zN4giDUb/1je/PGUtX+no8JSv+Kak++hjwtw=; b=rb/lIUfmNlelAvLH1GE1xaTTxiEtgxsWryuq0PMe74d/5Gx0v4Vu7pyOJs1rrXU6nY 9ukC4jUJOISSwqJ4vIkGux4kqpzFx+kiSLYsS4Rhzx0jtjTx8g4mY8LM0wXD6BeOh3pH fBWLF1ILJqFfb/zpSXen8G8wYEhEI3jkOerRbAJ9WbrsMIBe/M2HYtE2QTAVG6mzkOQW A/IOf0qsMTSrnH30Oj3BvxV5OxBgUjrtuawfEJBPtZsu9qNcNK0RN9bYiZ2Z0fhcdf+u MpqJZ016g2EHQGlrDoQK+lnphMF+00TDEqJJW/slYMf48UI4GzV4xaxib2QEbsBZk2ld qFbg== X-Gm-Message-State: ANoB5pmR/1EPvCoBID2jTlXMxZbuCOOgUEM70FTNqRBSaAzvd5LOQL85 2vTGY/JkDWjyOdteAl2FwpRgqB4= X-Google-Smtp-Source: AA0mqf7Zvrdzts6EXSee4UxjaQCvfRToP6ufXZ+7H0SjWmKi3dLmD4QUHbSFIPNlx2zeDuusPo3f0KU= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a25:2355:0:b0:6dd:c522:e3ba with SMTP id j82-20020a252355000000b006ddc522e3bamr18834148ybj.8.1669055160955; Mon, 21 Nov 2022 10:26:00 -0800 (PST) Date: Mon, 21 Nov 2022 10:25:48 -0800 In-Reply-To: <20221121182552.2152891-1-sdf@google.com> Mime-Version: 1.0 References: <20221121182552.2152891-1-sdf@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog Message-ID: <20221121182552.2152891-5-sdf@google.com> Subject: [PATCH bpf-next v2 4/8] veth: Support RX XDP metadata From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, David Ahern , Jakub Kicinski , Willem de Bruijn , Jesper Dangaard Brouer , Anatoly Burakov , Alexander Lobakin , Magnus Karlsson , Maryam Tahhan , xdp-hints@xdp-project.net, netdev@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net The goal is to enable end-to-end testing of the metadata for AF_XDP. Cc: John Fastabend Cc: David Ahern Cc: Martin KaFai Lau Cc: Jakub Kicinski Cc: Willem de Bruijn Cc: Jesper Dangaard Brouer Cc: Anatoly Burakov Cc: Alexander Lobakin Cc: Magnus Karlsson Cc: Maryam Tahhan Cc: xdp-hints@xdp-project.net Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- drivers/net/veth.c | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/drivers/net/veth.c b/drivers/net/veth.c index bbabc592d431..fdbca2aee33a 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -118,6 +118,7 @@ static struct { struct veth_xdp_buff { struct xdp_buff xdp; + struct sk_buff *skb; }; static int veth_get_link_ksettings(struct net_device *dev, @@ -602,6 +603,7 @@ static struct xdp_frame *veth_xdp_rcv_one(struct veth_rq *rq, xdp_convert_frame_to_buff(frame, xdp); xdp->rxq = &rq->xdp_rxq; + vxbuf.skb = NULL; act = bpf_prog_run_xdp(xdp_prog, xdp); @@ -823,6 +825,7 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq, __skb_push(skb, skb->data - skb_mac_header(skb)); if (veth_convert_skb_to_xdp_buff(rq, xdp, &skb)) goto drop; + vxbuf.skb = skb; orig_data = xdp->data; orig_data_end = xdp->data_end; @@ -1665,6 +1668,30 @@ static int veth_xdp(struct net_device *dev, struct netdev_bpf *xdp) } } +static bool veth_xdp_rx_timestamp_supported(const struct xdp_md *ctx) +{ + return true; +} + +static u64 veth_xdp_rx_timestamp(const struct xdp_md *ctx) +{ + return ktime_get_mono_fast_ns(); +} + +static bool veth_xdp_rx_hash_supported(const struct xdp_md *ctx) +{ + return true; +} + +static u32 veth_xdp_rx_hash(const struct xdp_md *ctx) +{ + struct veth_xdp_buff *_ctx = (void *)ctx; + + if (_ctx->skb) + return skb_get_hash(_ctx->skb); + return 0; +} + static const struct net_device_ops veth_netdev_ops = { .ndo_init = veth_dev_init, .ndo_open = veth_open, @@ -1684,6 +1711,11 @@ static const struct net_device_ops veth_netdev_ops = { .ndo_bpf = veth_xdp, .ndo_xdp_xmit = veth_ndo_xdp_xmit, .ndo_get_peer_dev = veth_peer_dev, + + .ndo_xdp_rx_timestamp_supported = veth_xdp_rx_timestamp_supported, + .ndo_xdp_rx_timestamp = veth_xdp_rx_timestamp, + .ndo_xdp_rx_hash_supported = veth_xdp_rx_hash_supported, + .ndo_xdp_rx_hash = veth_xdp_rx_hash, }; #define VETH_FEATURES (NETIF_F_SG | NETIF_F_FRAGLIST | NETIF_F_HW_CSUM | \ From patchwork Mon Nov 21 18:25:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13051537 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1E39C433FE for ; Mon, 21 Nov 2022 18:26:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229491AbiKUS0V (ORCPT ); Mon, 21 Nov 2022 13:26:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39096 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229635AbiKUS0T (ORCPT ); Mon, 21 Nov 2022 13:26:19 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AE0BDD02D0 for ; Mon, 21 Nov 2022 10:26:03 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id n141-20020a254093000000b006eacc13c38eso4889073yba.0 for ; Mon, 21 Nov 2022 10:26:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=QVTaBzzEQccyM/mxw7R704gPAXo7pWy8WQYRN5qCLic=; b=TND/CoeSpA6qNW3gjo6O23S3r0lP2dCLvpK1SJLit+5/HaX9Mk256xfqxICKO92ayL BC4fdJ1TPqH77VxyiFhG5tHkL2Hc2dR0y3pTgs5NgXhgRKX8hBcylnW0GRluq6bOOv97 LnjbgNsxkiT7tA0I4nsL8Wbz5/rhsNZmAjA800opnf+R2KwbpRSEBpgDIYcScuexjB+3 wax80Ysfi4qQSciJ3VF+lzFGiW7Xys4O1mVfAdiBbNSQmCqytoAtzV1fV3I3+IhGDxek 7xsU8WeM1jEUOprrmAyPF/YCEiVT7vI6vAGPVHaFexODBjDi2HVkP/Y0ho6lvm/UTnnQ 6ToQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QVTaBzzEQccyM/mxw7R704gPAXo7pWy8WQYRN5qCLic=; b=ynXq6wKnfqfOdWac696taJoDYZMKcnelM8Lapx1Gw2GQd3geKf0i+gBPseuakD+b6A 2yTo8KWTMVz4VJmPT74ViLFWNdoZsSKNEC2HSfMfC/EGTLQ2tI4CPwR31aN2pOrMeg6O BeMWC4S8AHlVgBRMUwO4cNMoCFxI1qzadE/e4D8SY5hlmPZKG00tk+tx7+FmnZe2Nd06 fVA3BtyYsROrS+G3cjzHbWt1iRPLKBfZGfC0xknmMM+8/3pZ8VurNvP8wv6b4JxMOBv7 LPfqYd+fjWjr+wqzigf/xGsdqmLm3AhqInXbtU5p74xO1clB2PJGbhLdbtF34c0qcg0u GxHg== X-Gm-Message-State: ANoB5plidZVcPgERuPSjgdaJdf7Hd9iDIq6HAX/z8a1VPgsiZcmiD6wp Ed1ABT/O7mhNXDrOYzwwy0UIPig= X-Google-Smtp-Source: AA0mqf5qdaySPE42aYWv8akmR6ZmGFEsDq7WquZm4KNJrSRu6dEO0BY81bJ33K0seX3rZGViVLFwmE4= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a81:b1c5:0:b0:381:548f:4dcc with SMTP id p188-20020a81b1c5000000b00381548f4dccmr1ywh.156.1669055162483; Mon, 21 Nov 2022 10:26:02 -0800 (PST) Date: Mon, 21 Nov 2022 10:25:49 -0800 In-Reply-To: <20221121182552.2152891-1-sdf@google.com> Mime-Version: 1.0 References: <20221121182552.2152891-1-sdf@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog Message-ID: <20221121182552.2152891-6-sdf@google.com> Subject: [PATCH bpf-next v2 5/8] selftests/bpf: Verify xdp_metadata xdp->af_xdp path From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, David Ahern , Jakub Kicinski , Willem de Bruijn , Jesper Dangaard Brouer , Anatoly Burakov , Alexander Lobakin , Magnus Karlsson , Maryam Tahhan , xdp-hints@xdp-project.net, netdev@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net - create new netns - create veth pair (veTX+veRX) - setup AF_XDP socket for both interfaces - attach bpf to veRX - send packet via veTX - verify the packet has expected metadata at veRX Cc: John Fastabend Cc: David Ahern Cc: Martin KaFai Lau Cc: Jakub Kicinski Cc: Willem de Bruijn Cc: Jesper Dangaard Brouer Cc: Anatoly Burakov Cc: Alexander Lobakin Cc: Magnus Karlsson Cc: Maryam Tahhan Cc: xdp-hints@xdp-project.net Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- tools/testing/selftests/bpf/Makefile | 2 +- .../selftests/bpf/prog_tests/xdp_metadata.c | 365 ++++++++++++++++++ .../selftests/bpf/progs/xdp_metadata.c | 57 +++ tools/testing/selftests/bpf/xdp_metadata.h | 7 + 4 files changed, 430 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/xdp_metadata.c create mode 100644 tools/testing/selftests/bpf/progs/xdp_metadata.c create mode 100644 tools/testing/selftests/bpf/xdp_metadata.h diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 6a0f043dc410..4eed22fa3681 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -527,7 +527,7 @@ TRUNNER_BPF_PROGS_DIR := progs TRUNNER_EXTRA_SOURCES := test_progs.c cgroup_helpers.c trace_helpers.c \ network_helpers.c testing_helpers.c \ btf_helpers.c flow_dissector_load.h \ - cap_helpers.c + cap_helpers.c xsk.c TRUNNER_EXTRA_FILES := $(OUTPUT)/urandom_read $(OUTPUT)/bpf_testmod.ko \ $(OUTPUT)/liburandom_read.so \ $(OUTPUT)/xdp_synproxy \ diff --git a/tools/testing/selftests/bpf/prog_tests/xdp_metadata.c b/tools/testing/selftests/bpf/prog_tests/xdp_metadata.c new file mode 100644 index 000000000000..01035ff7d783 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/xdp_metadata.c @@ -0,0 +1,365 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include "xdp_metadata.skel.h" +#include "xdp_metadata.h" +#include "xsk.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +#define TX_NAME "veTX" +#define RX_NAME "veRX" + +#define UDP_PAYLOAD_BYTES 4 + +#define AF_XDP_SOURCE_PORT 1234 +#define AF_XDP_CONSUMER_PORT 8080 + +#define UMEM_NUM 16 +#define UMEM_FRAME_SIZE XSK_UMEM__DEFAULT_FRAME_SIZE +#define UMEM_SIZE (UMEM_FRAME_SIZE * UMEM_NUM) +#define XDP_FLAGS XDP_FLAGS_DRV_MODE +#define QUEUE_ID 0 + +#define TX_ADDR "10.0.0.1" +#define RX_ADDR "10.0.0.2" +#define PREFIX_LEN "8" +#define FAMILY AF_INET + +#define SYS(cmd) ({ \ + if (!ASSERT_OK(system(cmd), (cmd))) \ + goto out; \ +}) + +struct xsk { + void *umem_area; + struct xsk_umem *umem; + struct xsk_ring_prod fill; + struct xsk_ring_cons comp; + struct xsk_ring_prod tx; + struct xsk_ring_cons rx; + struct xsk_socket *socket; +}; + +static int open_xsk(const char *ifname, struct xsk *xsk) +{ + int mmap_flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE; + const struct xsk_socket_config socket_config = { + .rx_size = XSK_RING_PROD__DEFAULT_NUM_DESCS, + .tx_size = XSK_RING_PROD__DEFAULT_NUM_DESCS, + .libbpf_flags = XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD, + .xdp_flags = XDP_FLAGS, + .bind_flags = XDP_COPY, + }; + const struct xsk_umem_config umem_config = { + .fill_size = XSK_RING_PROD__DEFAULT_NUM_DESCS, + .comp_size = XSK_RING_CONS__DEFAULT_NUM_DESCS, + .frame_size = XSK_UMEM__DEFAULT_FRAME_SIZE, + .flags = XDP_UMEM_UNALIGNED_CHUNK_FLAG, + }; + __u32 idx; + u64 addr; + int ret; + int i; + + xsk->umem_area = mmap(NULL, UMEM_SIZE, PROT_READ | PROT_WRITE, mmap_flags, -1, 0); + if (!ASSERT_NEQ(xsk->umem_area, MAP_FAILED, "mmap")) + return -1; + + ret = xsk_umem__create(&xsk->umem, + xsk->umem_area, UMEM_SIZE, + &xsk->fill, + &xsk->comp, + &umem_config); + if (!ASSERT_OK(ret, "xsk_umem__create")) + return ret; + + ret = xsk_socket__create(&xsk->socket, ifname, QUEUE_ID, + xsk->umem, + &xsk->rx, + &xsk->tx, + &socket_config); + if (!ASSERT_OK(ret, "xsk_socket__create")) + return ret; + + /* First half of umem is for TX. This way address matches 1-to-1 + * to the completion queue index. + */ + + for (i = 0; i < UMEM_NUM / 2; i++) { + addr = i * UMEM_FRAME_SIZE; + printf("%p: tx_desc[%d] -> %lx\n", xsk, i, addr); + } + + /* Second half of umem is for RX. */ + + ret = xsk_ring_prod__reserve(&xsk->fill, UMEM_NUM / 2, &idx); + if (!ASSERT_EQ(UMEM_NUM / 2, ret, "xsk_ring_prod__reserve")) + return ret; + if (!ASSERT_EQ(idx, 0, "fill idx != 0")) + return -1; + + for (i = 0; i < UMEM_NUM / 2; i++) { + addr = (UMEM_NUM / 2 + i) * UMEM_FRAME_SIZE; + printf("%p: rx_desc[%d] -> %lx\n", xsk, i, addr); + *xsk_ring_prod__fill_addr(&xsk->fill, i) = addr; + } + xsk_ring_prod__submit(&xsk->fill, ret); + + return 0; +} + +static void close_xsk(struct xsk *xsk) +{ + if (xsk->umem) + xsk_umem__delete(xsk->umem); + if (xsk->socket) + xsk_socket__delete(xsk->socket); + munmap(xsk->umem, UMEM_SIZE); +} + +static void ip_csum(struct iphdr *iph) +{ + __u32 sum = 0; + __u16 *p; + int i; + + iph->check = 0; + p = (void *)iph; + for (i = 0; i < sizeof(*iph) / sizeof(*p); i++) + sum += p[i]; + + while (sum >> 16) + sum = (sum & 0xffff) + (sum >> 16); + + iph->check = ~sum; +} + +static int generate_packet(struct xsk *xsk, __u16 dst_port) +{ + struct xdp_desc *tx_desc; + struct udphdr *udph; + struct ethhdr *eth; + struct iphdr *iph; + void *data; + __u32 idx; + int ret; + + ret = xsk_ring_prod__reserve(&xsk->tx, 1, &idx); + if (!ASSERT_EQ(ret, 1, "xsk_ring_prod__reserve")) + return -1; + + tx_desc = xsk_ring_prod__tx_desc(&xsk->tx, idx); + tx_desc->addr = idx % (UMEM_NUM / 2) * UMEM_FRAME_SIZE; + printf("%p: tx_desc[%u]->addr=%llx\n", xsk, idx, tx_desc->addr); + data = xsk_umem__get_data(xsk->umem_area, tx_desc->addr); + + eth = data; + iph = (void *)(eth + 1); + udph = (void *)(iph + 1); + + memcpy(eth->h_dest, "\x00\x00\x00\x00\x00\x02", ETH_ALEN); + memcpy(eth->h_source, "\x00\x00\x00\x00\x00\x01", ETH_ALEN); + eth->h_proto = htons(ETH_P_IP); + + iph->version = 0x4; + iph->ihl = 0x5; + iph->tos = 0x9; + iph->tot_len = htons(sizeof(*iph) + sizeof(*udph) + UDP_PAYLOAD_BYTES); + iph->id = 0; + iph->frag_off = 0; + iph->ttl = 0; + iph->protocol = IPPROTO_UDP; + ASSERT_EQ(inet_pton(FAMILY, TX_ADDR, &iph->saddr), 1, "inet_pton(TX_ADDR)"); + ASSERT_EQ(inet_pton(FAMILY, RX_ADDR, &iph->daddr), 1, "inet_pton(RX_ADDR)"); + ip_csum(iph); + + udph->source = htons(AF_XDP_SOURCE_PORT); + udph->dest = htons(dst_port); + udph->len = htons(sizeof(*udph) + UDP_PAYLOAD_BYTES); + udph->check = 0; + + memset(udph + 1, 0xAA, UDP_PAYLOAD_BYTES); + + tx_desc->len = sizeof(*eth) + sizeof(*iph) + sizeof(*udph) + UDP_PAYLOAD_BYTES; + xsk_ring_prod__submit(&xsk->tx, 1); + + ret = sendto(xsk_socket__fd(xsk->socket), NULL, 0, MSG_DONTWAIT, NULL, 0); + if (!ASSERT_GE(ret, 0, "sendto")) + return ret; + + return 0; +} + +static void complete_tx(struct xsk *xsk) +{ + __u32 idx; + __u64 addr; + + if (ASSERT_EQ(xsk_ring_cons__peek(&xsk->comp, 1, &idx), 1, "xsk_ring_cons__peek")) { + addr = *xsk_ring_cons__comp_addr(&xsk->comp, idx); + + printf("%p: refill idx=%u addr=%llx\n", xsk, idx, addr); + *xsk_ring_prod__fill_addr(&xsk->fill, idx) = addr; + xsk_ring_prod__submit(&xsk->fill, 1); + } +} + +static void refill_rx(struct xsk *xsk, __u64 addr) +{ + __u32 idx; + + if (ASSERT_EQ(xsk_ring_prod__reserve(&xsk->fill, 1, &idx), 1, "xsk_ring_prod__reserve")) { + printf("%p: complete idx=%u addr=%llx\n", xsk, idx, addr); + *xsk_ring_prod__fill_addr(&xsk->fill, idx) = addr; + xsk_ring_prod__submit(&xsk->fill, 1); + } +} + +static int verify_xsk_metadata(struct xsk *xsk) +{ + const struct xdp_desc *rx_desc; + struct pollfd fds = {}; + struct xdp_meta *meta; + struct ethhdr *eth; + struct iphdr *iph; + __u64 comp_addr; + void *data; + __u64 addr; + __u32 idx; + int ret; + + ret = recvfrom(xsk_socket__fd(xsk->socket), NULL, 0, MSG_DONTWAIT, NULL, NULL); + if (!ASSERT_EQ(ret, 0, "recvfrom")) + return -1; + + fds.fd = xsk_socket__fd(xsk->socket); + fds.events = POLLIN; + + ret = poll(&fds, 1, 1000); + if (!ASSERT_GT(ret, 0, "poll")) + return -1; + + ret = xsk_ring_cons__peek(&xsk->rx, 1, &idx); + if (!ASSERT_EQ(ret, 1, "xsk_ring_cons__peek")) + return -2; + + rx_desc = xsk_ring_cons__rx_desc(&xsk->rx, idx); + comp_addr = xsk_umem__extract_addr(rx_desc->addr); + addr = xsk_umem__add_offset_to_addr(rx_desc->addr); + printf("%p: rx_desc[%u]->addr=%llx addr=%llx comp_addr=%llx\n", + xsk, idx, rx_desc->addr, addr, comp_addr); + data = xsk_umem__get_data(xsk->umem_area, addr); + + /* Make sure we got the packet offset correctly. */ + + eth = data; + ASSERT_EQ(eth->h_proto, htons(ETH_P_IP), "eth->h_proto"); + iph = (void *)(eth + 1); + ASSERT_EQ((int)iph->version, 4, "iph->version"); + + /* custom metadata */ + + meta = data - sizeof(struct xdp_meta); + + if (!ASSERT_NEQ(meta->rx_timestamp, 0, "rx_timestamp")) + return -1; + + if (!ASSERT_NEQ(meta->rx_hash, 0, "rx_hash")) + return -1; + + xsk_ring_cons__release(&xsk->rx, 1); + refill_rx(xsk, comp_addr); + + return 0; +} + +void test_xdp_metadata(void) +{ + struct xdp_metadata *bpf_obj = NULL; + struct nstoken *tok = NULL; + __u32 queue_id = QUEUE_ID; + struct bpf_program *prog; + struct xsk tx_xsk = {}; + struct xsk rx_xsk = {}; + int rx_ifindex; + int sock_fd; + int ret; + + /* Setup new networking namespace, with a veth pair. */ + + SYS("ip netns add xdp_metadata"); + tok = open_netns("xdp_metadata"); + SYS("ip link add numtxqueues 1 numrxqueues 1 " TX_NAME + " type veth peer " RX_NAME " numtxqueues 1 numrxqueues 1"); + SYS("ip link set dev " TX_NAME " address 00:00:00:00:00:01"); + SYS("ip link set dev " RX_NAME " address 00:00:00:00:00:02"); + SYS("ip link set dev " TX_NAME " up"); + SYS("ip link set dev " RX_NAME " up"); + SYS("ip addr add " TX_ADDR "/" PREFIX_LEN " dev " TX_NAME); + SYS("ip addr add " RX_ADDR "/" PREFIX_LEN " dev " RX_NAME); + + rx_ifindex = if_nametoindex(RX_NAME); + + /* Setup separate AF_XDP for TX and RX interfaces. */ + + ret = open_xsk(TX_NAME, &tx_xsk); + if (!ASSERT_OK(ret, "open_xsk(TX_NAME)")) + goto out; + + ret = open_xsk(RX_NAME, &rx_xsk); + if (!ASSERT_OK(ret, "open_xsk(RX_NAME)")) + goto out; + + /* Attach BPF program to RX interface. */ + + bpf_obj = xdp_metadata__open(); + if (!ASSERT_OK_PTR(bpf_obj, "open skeleton")) + goto out; + + prog = bpf_object__find_program_by_name(bpf_obj->obj, "rx"); + bpf_program__set_ifindex(prog, rx_ifindex); + bpf_program__set_flags(prog, BPF_F_XDP_HAS_METADATA); + + if (!ASSERT_OK(xdp_metadata__load(bpf_obj), "load skeleton")) + goto out; + + ret = bpf_xdp_attach(rx_ifindex, + bpf_program__fd(bpf_obj->progs.rx), + XDP_FLAGS, NULL); + if (!ASSERT_GE(ret, 0, "bpf_xdp_attach")) + goto out; + + sock_fd = xsk_socket__fd(rx_xsk.socket); + ret = bpf_map_update_elem(bpf_map__fd(bpf_obj->maps.xsk), &queue_id, &sock_fd, 0); + if (!ASSERT_GE(ret, 0, "bpf_map_update_elem")) + goto out; + + /* Send packet destined to RX AF_XDP socket. */ + if (!ASSERT_GE(generate_packet(&tx_xsk, AF_XDP_CONSUMER_PORT), 0, + "generate AF_XDP_CONSUMER_PORT")) + goto out; + + /* Verify AF_XDP RX packet has proper metadata. */ + if (!ASSERT_GE(verify_xsk_metadata(&rx_xsk), 0, + "verify_xsk_metadata")) + goto out; + + complete_tx(&tx_xsk); + +out: + close_xsk(&rx_xsk); + close_xsk(&tx_xsk); + if (bpf_obj) + xdp_metadata__destroy(bpf_obj); + system("ip netns del xdp_metadata"); + if (tok) + close_netns(tok); +} diff --git a/tools/testing/selftests/bpf/progs/xdp_metadata.c b/tools/testing/selftests/bpf/progs/xdp_metadata.c new file mode 100644 index 000000000000..1b19a8d86efe --- /dev/null +++ b/tools/testing/selftests/bpf/progs/xdp_metadata.c @@ -0,0 +1,57 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include + +#ifndef ETH_P_IP +#define ETH_P_IP 0x0800 +#endif + +#include "xdp_metadata.h" +#include +#include + +struct { + __uint(type, BPF_MAP_TYPE_XSKMAP); + __uint(max_entries, 4); + __type(key, __u32); + __type(value, __u32); +} xsk SEC(".maps"); + +extern bool bpf_xdp_metadata_rx_timestamp_supported(const struct xdp_md *ctx) __ksym; +extern __u64 bpf_xdp_metadata_rx_timestamp(const struct xdp_md *ctx) __ksym; +extern bool bpf_xdp_metadata_rx_hash_supported(const struct xdp_md *ctx) __ksym; +extern __u32 bpf_xdp_metadata_rx_hash(const struct xdp_md *ctx) __ksym; + +SEC("xdp") +int rx(struct xdp_md *ctx) +{ + void *data, *data_meta; + struct xdp_meta *meta; + int ret; + + /* Reserve enough for all custom metadata. */ + + ret = bpf_xdp_adjust_meta(ctx, -(int)sizeof(struct xdp_meta)); + if (ret != 0) + return XDP_DROP; + + data = (void *)(long)ctx->data; + data_meta = (void *)(long)ctx->data_meta; + + if (data_meta + sizeof(struct xdp_meta) > data) + return XDP_DROP; + + meta = data_meta; + + /* Export metadata. */ + + if (bpf_xdp_metadata_rx_timestamp_supported(ctx)) + meta->rx_timestamp = bpf_xdp_metadata_rx_timestamp(ctx); + + if (bpf_xdp_metadata_rx_hash_supported(ctx)) + meta->rx_hash = bpf_xdp_metadata_rx_hash(ctx); + + return bpf_redirect_map(&xsk, ctx->rx_queue_index, XDP_PASS); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/xdp_metadata.h b/tools/testing/selftests/bpf/xdp_metadata.h new file mode 100644 index 000000000000..c4892d122b7f --- /dev/null +++ b/tools/testing/selftests/bpf/xdp_metadata.h @@ -0,0 +1,7 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#pragma once + +struct xdp_meta { + __u64 rx_timestamp; + __u32 rx_hash; +}; From patchwork Mon Nov 21 18:25:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13051538 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8878C43219 for ; Mon, 21 Nov 2022 18:26:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230062AbiKUS0W (ORCPT ); Mon, 21 Nov 2022 13:26:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39100 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229622AbiKUS0U (ORCPT ); Mon, 21 Nov 2022 13:26:20 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4C43DD06CA for ; Mon, 21 Nov 2022 10:26:05 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id b1-20020a17090a10c100b0020da29fa5e5so6325071pje.2 for ; Mon, 21 Nov 2022 10:26:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mZqP3yNBrpAwk4TJsSnLiwPq87Wubt3mJUNd7RioCb8=; b=n5EeXpIol277FfH8pVgfNkp0LXATkS74tOOiofe8A4P24VMW2KShluIJzFmIxha/IL xqncVwIdIQIiRDzOvq88kcaINR78SbKb5sTbNuKg2OOjquU0O+Ml9tmuaPXC6oWuRmNe q9J4g1w5mKmX5iZa/VdZdy+y/4X/ViKVDVyaWRxTF4ZRwIE5XJZVbIhOx+Q+utCp0plH nM9eYGUgJsbKUqqZPucB8Q9vToTJb581zJu/PbSHJP3tx8u/7HkqwrL3fiEarhdZzTZg mPOFcHrmD4Nx+0Aqv9U9DBGNgP9ETuJq7uLW1AOsGzM5ckXe9G70nRGmiBEl70XGC3cQ TwGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mZqP3yNBrpAwk4TJsSnLiwPq87Wubt3mJUNd7RioCb8=; b=LDG0N6PExdTCYrE4z9Ij/UZlPTcD1RbNVXbkxIwWYi2MwnHgp9wOzhUwRyu+OFvUP6 jZSMS4tUh4dnf0/dE0Sl0x0bjAS2uBwSlk4sZr8JpqaZ21hSkItkwgYU8SxshoBBLzDo 4cRwJlhfylozGzYYfV0/+SPQpmA5kDo9xNBk5nGLo/xe6fEetjeFwFFmr4d03xMFG+E0 Ne1biRCiNcpeTomWKyLEIZUJNc0DZI6IFlU5AXAl18+3KNN8OyoxMNq2mXL6jXQfmczr 2rd5L43e7pmq4mlPz2fGijbX7NpBsZ2k3TCzlmgMw3Ptg5mORcTeLXND2cAkVOXtVNrq Vpyg== X-Gm-Message-State: ANoB5pnueE6oQfCjsvJAvrX4GR8OU0bKLsMDgVDErzPWcu7VtQlp1A7z QonEWmo/kplWoo+7TVZRDuZGNxg= X-Google-Smtp-Source: AA0mqf4IdK+Ed1+mNhKMzH5S4cC4/t5Tc7FBAGO03ycP1MeB/RalnXA5jDvqks2Bs1nqsSDs+wEWRv0= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:aa7:880c:0:b0:573:3397:f804 with SMTP id c12-20020aa7880c000000b005733397f804mr4367415pfo.86.1669055164507; Mon, 21 Nov 2022 10:26:04 -0800 (PST) Date: Mon, 21 Nov 2022 10:25:50 -0800 In-Reply-To: <20221121182552.2152891-1-sdf@google.com> Mime-Version: 1.0 References: <20221121182552.2152891-1-sdf@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog Message-ID: <20221121182552.2152891-7-sdf@google.com> Subject: [PATCH bpf-next v2 6/8] mlx4: Introduce mlx4_xdp_buff wrapper for xdp_buff From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, Tariq Toukan , David Ahern , Jakub Kicinski , Willem de Bruijn , Jesper Dangaard Brouer , Anatoly Burakov , Alexander Lobakin , Magnus Karlsson , Maryam Tahhan , xdp-hints@xdp-project.net, netdev@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net No functional changes. Boilerplate to allow stuffing more data after xdp_buff. Cc: Tariq Toukan Cc: John Fastabend Cc: David Ahern Cc: Martin KaFai Lau Cc: Jakub Kicinski Cc: Willem de Bruijn Cc: Jesper Dangaard Brouer Cc: Anatoly Burakov Cc: Alexander Lobakin Cc: Magnus Karlsson Cc: Maryam Tahhan Cc: xdp-hints@xdp-project.net Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- drivers/net/ethernet/mellanox/mlx4/en_rx.c | 26 +++++++++++++--------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx4/en_rx.c b/drivers/net/ethernet/mellanox/mlx4/en_rx.c index 8f762fc170b3..467356633172 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_rx.c @@ -661,17 +661,21 @@ static int check_csum(struct mlx4_cqe *cqe, struct sk_buff *skb, void *va, #define MLX4_CQE_STATUS_IP_ANY (MLX4_CQE_STATUS_IPV4) #endif +struct mlx4_xdp_buff { + struct xdp_buff xdp; +}; + int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int budget) { struct mlx4_en_priv *priv = netdev_priv(dev); int factor = priv->cqe_factor; struct mlx4_en_rx_ring *ring; + struct mlx4_xdp_buff mxbuf; struct bpf_prog *xdp_prog; int cq_ring = cq->ring; bool doorbell_pending; bool xdp_redir_flush; struct mlx4_cqe *cqe; - struct xdp_buff xdp; int polled = 0; int index; @@ -681,7 +685,7 @@ int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int bud ring = priv->rx_ring[cq_ring]; xdp_prog = rcu_dereference_bh(ring->xdp_prog); - xdp_init_buff(&xdp, priv->frag_info[0].frag_stride, &ring->xdp_rxq); + xdp_init_buff(&mxbuf.xdp, priv->frag_info[0].frag_stride, &ring->xdp_rxq); doorbell_pending = false; xdp_redir_flush = false; @@ -776,24 +780,24 @@ int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int bud priv->frag_info[0].frag_size, DMA_FROM_DEVICE); - xdp_prepare_buff(&xdp, va - frags[0].page_offset, + xdp_prepare_buff(&mxbuf.xdp, va - frags[0].page_offset, frags[0].page_offset, length, false); - orig_data = xdp.data; + orig_data = mxbuf.xdp.data; - act = bpf_prog_run_xdp(xdp_prog, &xdp); + act = bpf_prog_run_xdp(xdp_prog, &mxbuf.xdp); - length = xdp.data_end - xdp.data; - if (xdp.data != orig_data) { - frags[0].page_offset = xdp.data - - xdp.data_hard_start; - va = xdp.data; + length = mxbuf.xdp.data_end - mxbuf.xdp.data; + if (mxbuf.xdp.data != orig_data) { + frags[0].page_offset = mxbuf.xdp.data - + mxbuf.xdp.data_hard_start; + va = mxbuf.xdp.data; } switch (act) { case XDP_PASS: break; case XDP_REDIRECT: - if (likely(!xdp_do_redirect(dev, &xdp, xdp_prog))) { + if (likely(!xdp_do_redirect(dev, &mxbuf.xdp, xdp_prog))) { ring->xdp_redirect++; xdp_redir_flush = true; frags[0].page = NULL; From patchwork Mon Nov 21 18:25:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13051539 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40B9DC433FE for ; Mon, 21 Nov 2022 18:26:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229777AbiKUS0Y (ORCPT ); Mon, 21 Nov 2022 13:26:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39150 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229966AbiKUS0V (ORCPT ); Mon, 21 Nov 2022 13:26:21 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BD4F9D06FB for ; Mon, 21 Nov 2022 10:26:06 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id x8-20020a17090a6b4800b00218ae9b2a47so1839395pjl.6 for ; Mon, 21 Nov 2022 10:26:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=pgA8r6q+vHqw+YrYOl3aP+dNQ95F67luBXwFSR2T9OY=; b=hbR+Vdv0kdy8wD7qfjR+qjrEvtkWb5nrEFaFlg28TgCv4WVOMdFGkmCXsEn/M+HnOz ncaJfwM2yKCRhfmoQZ3IeagvNaNrnjSoB8O4PxgEvS4DkTBVPaYJoMnQspR9PPga21zm 1Wi/+eN6rz26LJkaMY63IIWuNJBXRb4AU6kKZwMqUDr5A8htVNKHRuAoBtCSXbhZUwIY oDyc5KyTIWFCoPRiLEbvz4MRH/up9wZ8fI6lvIGokdb8elM/fPnQLEZcLFagO628craY BGfqcIGoeAuEL5+CUJgHXbtNLGCJPPOrKFvcirLDNX78dWvV69yLu4wXotUmJiAVXLQM IlrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pgA8r6q+vHqw+YrYOl3aP+dNQ95F67luBXwFSR2T9OY=; b=p2fxG22wOXMRK8hu1Hq4Onkwp2Uumd3O1JTPOGXRaXeadMl+1nImyRBAGGv4wO3f1C j7KMGE2vhGbUR9HV78/I0NuM0LNGSSEm3iBktix7BCGUhbIOUrwvLVm0D7mqYK/kUyKk CVpoLInsYrtCuUrdZ6oaKEk9jGQu5j5ZiNljA8hSw5uTy3MnD+6wg4U+fKS0wB604JO+ 1XBw8tp572ISvGhlSJssk5ngIpnDACPdEC8UlIG58Bcu+SrA2H11ApiS4XZ2JMsNS1ng LWW9y6GQqGvh1WaySfulB6dJv6VboGm2IeKPHFfV4RWdDJf33t0WjBWhd9XPwdjvE+jV x2ug== X-Gm-Message-State: ANoB5pkzs7DwQo8W+nismryovbHzl3TaqoFZXwjlDSsjbDi1FISPKb0I 3oLpiSR8DBQxA6V6iFsY8BEqsBk= X-Google-Smtp-Source: AA0mqf7n2thXFA68LqvTVLbekOytw+rcf5+iC2F8qRSb4JAMHqx5uoXQ/b2Fe+aTVqOn2NwBeTq8XCE= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:aa7:8709:0:b0:572:2189:84ef with SMTP id b9-20020aa78709000000b00572218984efmr923589pfo.28.1669055166038; Mon, 21 Nov 2022 10:26:06 -0800 (PST) Date: Mon, 21 Nov 2022 10:25:51 -0800 In-Reply-To: <20221121182552.2152891-1-sdf@google.com> Mime-Version: 1.0 References: <20221121182552.2152891-1-sdf@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog Message-ID: <20221121182552.2152891-8-sdf@google.com> Subject: [PATCH bpf-next v2 7/8] mxl4: Support RX XDP metadata From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, Tariq Toukan , David Ahern , Jakub Kicinski , Willem de Bruijn , Jesper Dangaard Brouer , Anatoly Burakov , Alexander Lobakin , Magnus Karlsson , Maryam Tahhan , xdp-hints@xdp-project.net, netdev@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net RX timestamp and hash for now. Tested using the prog from the next patch. Also enabling xdp metadata support; don't see why it's disabled, there is enough headroom.. Cc: Tariq Toukan Cc: John Fastabend Cc: David Ahern Cc: Martin KaFai Lau Cc: Jakub Kicinski Cc: Willem de Bruijn Cc: Jesper Dangaard Brouer Cc: Anatoly Burakov Cc: Alexander Lobakin Cc: Magnus Karlsson Cc: Maryam Tahhan Cc: xdp-hints@xdp-project.net Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- .../net/ethernet/mellanox/mlx4/en_netdev.c | 10 ++++ drivers/net/ethernet/mellanox/mlx4/en_rx.c | 48 ++++++++++++++++++- include/linux/mlx4/device.h | 7 +++ 3 files changed, 64 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c index 8800d3f1f55c..1cb63746a851 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c @@ -2855,6 +2855,11 @@ static const struct net_device_ops mlx4_netdev_ops = { .ndo_features_check = mlx4_en_features_check, .ndo_set_tx_maxrate = mlx4_en_set_tx_maxrate, .ndo_bpf = mlx4_xdp, + + .ndo_xdp_rx_timestamp_supported = mlx4_xdp_rx_timestamp_supported, + .ndo_xdp_rx_timestamp = mlx4_xdp_rx_timestamp, + .ndo_xdp_rx_hash_supported = mlx4_xdp_rx_hash_supported, + .ndo_xdp_rx_hash = mlx4_xdp_rx_hash, }; static const struct net_device_ops mlx4_netdev_ops_master = { @@ -2887,6 +2892,11 @@ static const struct net_device_ops mlx4_netdev_ops_master = { .ndo_features_check = mlx4_en_features_check, .ndo_set_tx_maxrate = mlx4_en_set_tx_maxrate, .ndo_bpf = mlx4_xdp, + + .ndo_xdp_rx_timestamp_supported = mlx4_xdp_rx_timestamp_supported, + .ndo_xdp_rx_timestamp = mlx4_xdp_rx_timestamp, + .ndo_xdp_rx_hash_supported = mlx4_xdp_rx_hash_supported, + .ndo_xdp_rx_hash = mlx4_xdp_rx_hash, }; struct mlx4_en_bond { diff --git a/drivers/net/ethernet/mellanox/mlx4/en_rx.c b/drivers/net/ethernet/mellanox/mlx4/en_rx.c index 467356633172..fd14d59f6cbf 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_rx.c @@ -663,8 +663,50 @@ static int check_csum(struct mlx4_cqe *cqe, struct sk_buff *skb, void *va, struct mlx4_xdp_buff { struct xdp_buff xdp; + struct mlx4_cqe *cqe; + struct mlx4_en_dev *mdev; + struct mlx4_en_rx_ring *ring; + struct net_device *dev; }; +bool mlx4_xdp_rx_timestamp_supported(const struct xdp_md *ctx) +{ + struct mlx4_xdp_buff *_ctx = (void *)ctx; + + return _ctx->ring->hwtstamp_rx_filter == HWTSTAMP_FILTER_ALL; +} + +u64 mlx4_xdp_rx_timestamp(const struct xdp_md *ctx) +{ + struct mlx4_xdp_buff *_ctx = (void *)ctx; + unsigned int seq; + u64 timestamp; + u64 nsec; + + timestamp = mlx4_en_get_cqe_ts(_ctx->cqe); + + do { + seq = read_seqbegin(&_ctx->mdev->clock_lock); + nsec = timecounter_cyc2time(&_ctx->mdev->clock, timestamp); + } while (read_seqretry(&_ctx->mdev->clock_lock, seq)); + + return ns_to_ktime(nsec); +} + +bool mlx4_xdp_rx_hash_supported(const struct xdp_md *ctx) +{ + struct mlx4_xdp_buff *_ctx = (void *)ctx; + + return _ctx->dev->features & NETIF_F_RXHASH; +} + +u32 mlx4_xdp_rx_hash(const struct xdp_md *ctx) +{ + struct mlx4_xdp_buff *_ctx = (void *)ctx; + + return be32_to_cpu(_ctx->cqe->immed_rss_invalid); +} + int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int budget) { struct mlx4_en_priv *priv = netdev_priv(dev); @@ -781,8 +823,12 @@ int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int bud DMA_FROM_DEVICE); xdp_prepare_buff(&mxbuf.xdp, va - frags[0].page_offset, - frags[0].page_offset, length, false); + frags[0].page_offset, length, true); orig_data = mxbuf.xdp.data; + mxbuf.cqe = cqe; + mxbuf.mdev = priv->mdev; + mxbuf.ring = ring; + mxbuf.dev = dev; act = bpf_prog_run_xdp(xdp_prog, &mxbuf.xdp); diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h index 6646634a0b9d..d5904da1d490 100644 --- a/include/linux/mlx4/device.h +++ b/include/linux/mlx4/device.h @@ -1585,4 +1585,11 @@ static inline int mlx4_get_num_reserved_uar(struct mlx4_dev *dev) /* The first 128 UARs are used for EQ doorbells */ return (128 >> (PAGE_SHIFT - dev->uar_page_shift)); } + +struct xdp_md; +bool mlx4_xdp_rx_timestamp_supported(const struct xdp_md *ctx); +u64 mlx4_xdp_rx_timestamp(const struct xdp_md *ctx); +bool mlx4_xdp_rx_hash_supported(const struct xdp_md *ctx); +u32 mlx4_xdp_rx_hash(const struct xdp_md *ctx); + #endif /* MLX4_DEVICE_H */ From patchwork Mon Nov 21 18:25:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13051540 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2174C43217 for ; Mon, 21 Nov 2022 18:26:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230089AbiKUS0j (ORCPT ); Mon, 21 Nov 2022 13:26:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39168 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230041AbiKUS0W (ORCPT ); Mon, 21 Nov 2022 13:26:22 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6E601D08BC for ; Mon, 21 Nov 2022 10:26:08 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id n4-20020a17090a2fc400b002132adb9485so6340724pjm.0 for ; Mon, 21 Nov 2022 10:26:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wvh3gZHIniCaSTYTx2lCpveng1kO6OVPl3qsCJW9rDQ=; b=cj8NdaeD7H8Ybvf7D8UkJVCzPeR2LQKVyee4PhdOjB5o+iruo1F9w2o9ajRuuqNoq+ aV7NzXq5Z9AvydR0uVESDomhwK0jigeaXJg4/50J/NP6aAd84sGGs1jmNO6XF3wlyhBh Ydg3tywpTBHCoTH9ULZ22Ebyb83aebh099dW6mBZG4LhSVWijYMgcJQdoFC00nVHpS7o PzyeGM5AIcoOvYi3JGrDs3jcR/6UZMRCeSUm2NftXBtOZNk0t1A3P+mmZJtPTRPuXG84 WUpr9p1ShFo8BEfEuepcHcxm+ZZ2JvprNiIACzbBeDuLLPLztxmtTMoDs9HdS3PoZA4y yyNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wvh3gZHIniCaSTYTx2lCpveng1kO6OVPl3qsCJW9rDQ=; b=IwY7T/NOg3sN3hW+bVy1RCbz65hTIT3ChP0ExmLhUHGrLwvKRJiP+yyjT0nhVieI8r q6NTttTmYZKGZuLrMxxAQiK+cmJK6HRlqO0szn4jDIzG4dEzYNrWUYx2Lz7kj1eFo3Ir rj1/BIoq/EX/R/AMz4gOegbor0RjSb8Be8OU4gBhDTHdOgaz96kBIjNQEWc2ndVM9d4M /rxYZHUi9CgGbcVmruzOVyD92plXm1BbmNwOBPKjr0AtoIKYhrppy0GltgBo1+VV89RA lfIp1m8CeJbMARoLHQxkiHzpbXjiFf5AETIti1Ur6Sx7ayJf/QYs4gFRf2Bh6rJxyIqP Tmow== X-Gm-Message-State: ANoB5pleoIygR+RyuTqznn6wfJLoF2aaQ0qyqf3rZq/GfH/l/UI0wBnM PfizWqLqWfVk9ln6HCHyslUi73A= X-Google-Smtp-Source: AA0mqf6a6hvdEPyTPlz8uNMj0u/5iJ/2dFFVYUowcTKkYzF9zPIaHMl3KHozX+IranVPWNztyC+yBBc= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a62:cd8f:0:b0:563:321a:9155 with SMTP id o137-20020a62cd8f000000b00563321a9155mr21721103pfg.46.1669055167929; Mon, 21 Nov 2022 10:26:07 -0800 (PST) Date: Mon, 21 Nov 2022 10:25:52 -0800 In-Reply-To: <20221121182552.2152891-1-sdf@google.com> Mime-Version: 1.0 References: <20221121182552.2152891-1-sdf@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog Message-ID: <20221121182552.2152891-9-sdf@google.com> Subject: [PATCH bpf-next v2 8/8] selftests/bpf: Simple program to dump XDP RX metadata From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, David Ahern , Jakub Kicinski , Willem de Bruijn , Jesper Dangaard Brouer , Anatoly Burakov , Alexander Lobakin , Magnus Karlsson , Maryam Tahhan , xdp-hints@xdp-project.net, netdev@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net To be used for verification of driver implementations. Note that the skb path is gone from the series, but I'm still keeping the implementation for any possible future work. $ xdp_hw_metadata On the other machine: $ echo -n xdp | nc -u -q1 9091 # for AF_XDP $ echo -n skb | nc -u -q1 9092 # for skb Sample output: # xdp xsk_ring_cons__peek: 1 0x19f9090: rx_desc[0]->addr=100000000008000 addr=8100 comp_addr=8000 rx_timestamp_supported: 1 rx_timestamp: 1667850075063948829 0x19f9090: complete idx=8 addr=8000 # skb found skb hwtstamp = 1668314052.854274681 Decoding: # xdp rx_timestamp=1667850075.063948829 $ date -d @1667850075 Mon Nov 7 11:41:15 AM PST 2022 $ date Mon Nov 7 11:42:05 AM PST 2022 # skb $ date -d @1668314052 Sat Nov 12 08:34:12 PM PST 2022 $ date Sat Nov 12 08:37:06 PM PST 2022 Cc: John Fastabend Cc: David Ahern Cc: Martin KaFai Lau Cc: Jakub Kicinski Cc: Willem de Bruijn Cc: Jesper Dangaard Brouer Cc: Anatoly Burakov Cc: Alexander Lobakin Cc: Magnus Karlsson Cc: Maryam Tahhan Cc: xdp-hints@xdp-project.net Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- tools/testing/selftests/bpf/.gitignore | 1 + tools/testing/selftests/bpf/Makefile | 6 +- .../selftests/bpf/progs/xdp_hw_metadata.c | 93 ++++ tools/testing/selftests/bpf/xdp_hw_metadata.c | 405 ++++++++++++++++++ 4 files changed, 504 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/progs/xdp_hw_metadata.c create mode 100644 tools/testing/selftests/bpf/xdp_hw_metadata.c diff --git a/tools/testing/selftests/bpf/.gitignore b/tools/testing/selftests/bpf/.gitignore index 07d2d0a8c5cb..01e3baeefd4f 100644 --- a/tools/testing/selftests/bpf/.gitignore +++ b/tools/testing/selftests/bpf/.gitignore @@ -46,3 +46,4 @@ test_cpp xskxceiver xdp_redirect_multi xdp_synproxy +xdp_hw_metadata diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 4eed22fa3681..189b39b0e5d0 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -83,7 +83,7 @@ TEST_PROGS_EXTENDED := with_addr.sh \ TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \ flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \ test_lirc_mode2_user xdping test_cpp runqslower bench bpf_testmod.ko \ - xskxceiver xdp_redirect_multi xdp_synproxy veristat + xskxceiver xdp_redirect_multi xdp_synproxy veristat xdp_hw_metadata TEST_CUSTOM_PROGS = $(OUTPUT)/urandom_read $(OUTPUT)/sign-file TEST_GEN_FILES += liburandom_read.so @@ -241,6 +241,9 @@ $(OUTPUT)/test_maps: $(TESTING_HELPERS) $(OUTPUT)/test_verifier: $(TESTING_HELPERS) $(CAP_HELPERS) $(OUTPUT)/xsk.o: $(BPFOBJ) $(OUTPUT)/xskxceiver: $(OUTPUT)/xsk.o +$(OUTPUT)/xdp_hw_metadata: $(OUTPUT)/xsk.o $(OUTPUT)/xdp_hw_metadata.skel.h +$(OUTPUT)/xdp_hw_metadata: $(OUTPUT)/network_helpers.o +$(OUTPUT)/xdp_hw_metadata: LDFLAGS += -static BPFTOOL ?= $(DEFAULT_BPFTOOL) $(DEFAULT_BPFTOOL): $(wildcard $(BPFTOOLDIR)/*.[ch] $(BPFTOOLDIR)/Makefile) \ @@ -383,6 +386,7 @@ linked_maps.skel.h-deps := linked_maps1.bpf.o linked_maps2.bpf.o test_subskeleton.skel.h-deps := test_subskeleton_lib2.bpf.o test_subskeleton_lib.bpf.o test_subskeleton.bpf.o test_subskeleton_lib.skel.h-deps := test_subskeleton_lib2.bpf.o test_subskeleton_lib.bpf.o test_usdt.skel.h-deps := test_usdt.bpf.o test_usdt_multispec.bpf.o +xdp_hw_metadata.skel.h-deps := xdp_hw_metadata.bpf.o LINKED_BPF_SRCS := $(patsubst %.bpf.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(skel)-deps))) diff --git a/tools/testing/selftests/bpf/progs/xdp_hw_metadata.c b/tools/testing/selftests/bpf/progs/xdp_hw_metadata.c new file mode 100644 index 000000000000..0ae409094883 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/xdp_hw_metadata.c @@ -0,0 +1,93 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include "xdp_metadata.h" + +struct { + __uint(type, BPF_MAP_TYPE_XSKMAP); + __uint(max_entries, 256); + __type(key, __u32); + __type(value, __u32); +} xsk SEC(".maps"); + +extern bool bpf_xdp_metadata_rx_timestamp_supported(const struct xdp_md *ctx) __ksym; +extern __u64 bpf_xdp_metadata_rx_timestamp(const struct xdp_md *ctx) __ksym; +extern bool bpf_xdp_metadata_rx_hash_supported(const struct xdp_md *ctx) __ksym; +extern __u32 bpf_xdp_metadata_rx_hash(const struct xdp_md *ctx) __ksym; + +SEC("xdp") +int rx(struct xdp_md *ctx) +{ + void *data, *data_meta, *data_end; + struct ipv6hdr *ip6h = NULL; + struct ethhdr *eth = NULL; + struct udphdr *udp = NULL; + struct iphdr *iph = NULL; + struct xdp_meta *meta; + int ret; + + data = (void *)(long)ctx->data; + data_end = (void *)(long)ctx->data_end; + eth = data; + if (eth + 1 < data_end) { + if (eth->h_proto == bpf_htons(ETH_P_IP)) { + iph = (void *)(eth + 1); + if (iph + 1 < data_end && iph->protocol == IPPROTO_UDP) + udp = (void *)(iph + 1); + } + if (eth->h_proto == bpf_htons(ETH_P_IPV6)) { + ip6h = (void *)(eth + 1); + if (ip6h + 1 < data_end && ip6h->nexthdr == IPPROTO_UDP) + udp = (void *)(ip6h + 1); + } + if (udp && udp + 1 > data_end) + udp = NULL; + } + + if (!udp) + return XDP_PASS; + + if (udp->dest != bpf_htons(9091)) + return XDP_PASS; + + bpf_printk("forwarding UDP:9091 to AF_XDP"); + + ret = bpf_xdp_adjust_meta(ctx, -(int)sizeof(struct xdp_meta)); + if (ret != 0) { + bpf_printk("bpf_xdp_adjust_meta returned %d", ret); + return XDP_PASS; + } + + data = (void *)(long)ctx->data; + data_meta = (void *)(long)ctx->data_meta; + meta = data_meta; + + if (meta + 1 > data) { + bpf_printk("bpf_xdp_adjust_meta doesn't appear to work"); + return XDP_PASS; + } + + if (bpf_xdp_metadata_rx_timestamp_supported(ctx)) { + meta->rx_timestamp = bpf_xdp_metadata_rx_timestamp(ctx); + bpf_printk("populated rx_timestamp with %u", meta->rx_timestamp); + } + + if (bpf_xdp_metadata_rx_hash_supported(ctx)) { + meta->rx_hash = bpf_xdp_metadata_rx_hash(ctx); + bpf_printk("populated rx_hash with %u", meta->rx_hash); + } + + return bpf_redirect_map(&xsk, ctx->rx_queue_index, XDP_PASS); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/xdp_hw_metadata.c b/tools/testing/selftests/bpf/xdp_hw_metadata.c new file mode 100644 index 000000000000..7823a35a1ef7 --- /dev/null +++ b/tools/testing/selftests/bpf/xdp_hw_metadata.c @@ -0,0 +1,405 @@ +// SPDX-License-Identifier: GPL-2.0 + +/* Reference program for verifying XDP metadata on real HW. Functional test + * only, doesn't test the performance. + * + * RX: + * - UDP 9091 packets are diverted into AF_XDP + * - Metadata verified: + * - rx_timestamp + * - rx_hash + * + * TX: + * - TBD + */ + +#include +#include +#include "xdp_hw_metadata.skel.h" +#include "xsk.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "xdp_metadata.h" + +#define UMEM_NUM 16 +#define UMEM_FRAME_SIZE XSK_UMEM__DEFAULT_FRAME_SIZE +#define UMEM_SIZE (UMEM_FRAME_SIZE * UMEM_NUM) +#define XDP_FLAGS (XDP_FLAGS_DRV_MODE | XDP_FLAGS_REPLACE) + +struct xsk { + void *umem_area; + struct xsk_umem *umem; + struct xsk_ring_prod fill; + struct xsk_ring_cons comp; + struct xsk_ring_prod tx; + struct xsk_ring_cons rx; + struct xsk_socket *socket; +}; + +struct xdp_hw_metadata *bpf_obj; +struct xsk *rx_xsk; +const char *ifname; +int ifindex; +int rxq; + +void test__fail(void) { /* for network_helpers.c */ } + +static int open_xsk(const char *ifname, struct xsk *xsk, __u32 queue_id) +{ + int mmap_flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE; + const struct xsk_socket_config socket_config = { + .rx_size = XSK_RING_PROD__DEFAULT_NUM_DESCS, + .tx_size = XSK_RING_PROD__DEFAULT_NUM_DESCS, + .libbpf_flags = XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD, + .xdp_flags = XDP_FLAGS, + .bind_flags = XDP_COPY, + }; + const struct xsk_umem_config umem_config = { + .fill_size = XSK_RING_PROD__DEFAULT_NUM_DESCS, + .comp_size = XSK_RING_CONS__DEFAULT_NUM_DESCS, + .frame_size = XSK_UMEM__DEFAULT_FRAME_SIZE, + .flags = XDP_UMEM_UNALIGNED_CHUNK_FLAG, + }; + __u32 idx; + u64 addr; + int ret; + int i; + + xsk->umem_area = mmap(NULL, UMEM_SIZE, PROT_READ | PROT_WRITE, mmap_flags, -1, 0); + if (xsk->umem_area == MAP_FAILED) + return -ENOMEM; + + ret = xsk_umem__create(&xsk->umem, + xsk->umem_area, UMEM_SIZE, + &xsk->fill, + &xsk->comp, + &umem_config); + if (ret) + return ret; + + ret = xsk_socket__create(&xsk->socket, ifname, queue_id, + xsk->umem, + &xsk->rx, + &xsk->tx, + &socket_config); + if (ret) + return ret; + + /* First half of umem is for TX. This way address matches 1-to-1 + * to the completion queue index. + */ + + for (i = 0; i < UMEM_NUM / 2; i++) { + addr = i * UMEM_FRAME_SIZE; + printf("%p: tx_desc[%d] -> %lx\n", xsk, i, addr); + } + + /* Second half of umem is for RX. */ + + ret = xsk_ring_prod__reserve(&xsk->fill, UMEM_NUM / 2, &idx); + for (i = 0; i < UMEM_NUM / 2; i++) { + addr = (UMEM_NUM / 2 + i) * UMEM_FRAME_SIZE; + printf("%p: rx_desc[%d] -> %lx\n", xsk, i, addr); + *xsk_ring_prod__fill_addr(&xsk->fill, i) = addr; + } + xsk_ring_prod__submit(&xsk->fill, ret); + + return 0; +} + +static void close_xsk(struct xsk *xsk) +{ + if (xsk->umem) + xsk_umem__delete(xsk->umem); + if (xsk->socket) + xsk_socket__delete(xsk->socket); + munmap(xsk->umem, UMEM_SIZE); +} + +static void refill_rx(struct xsk *xsk, __u64 addr) +{ + __u32 idx; + + if (xsk_ring_prod__reserve(&xsk->fill, 1, &idx) == 1) { + printf("%p: complete idx=%u addr=%llx\n", xsk, idx, addr); + *xsk_ring_prod__fill_addr(&xsk->fill, idx) = addr; + xsk_ring_prod__submit(&xsk->fill, 1); + } +} + +static void verify_xdp_metadata(void *data) +{ + struct xdp_meta *meta; + + meta = data - sizeof(*meta); + + printf("rx_timestamp: %llu\n", meta->rx_timestamp); + printf("rx_hash: %u\n", meta->rx_hash); +} + +static void verify_skb_metadata(int fd) +{ + char cmsg_buf[1024]; + char packet_buf[128]; + + struct scm_timestamping *ts; + struct iovec packet_iov; + struct cmsghdr *cmsg; + struct msghdr hdr; + + memset(&hdr, 0, sizeof(hdr)); + hdr.msg_iov = &packet_iov; + hdr.msg_iovlen = 1; + packet_iov.iov_base = packet_buf; + packet_iov.iov_len = sizeof(packet_buf); + + hdr.msg_control = cmsg_buf; + hdr.msg_controllen = sizeof(cmsg_buf); + + if (recvmsg(fd, &hdr, 0) < 0) + error(-1, errno, "recvmsg"); + + for (cmsg = CMSG_FIRSTHDR(&hdr); cmsg != NULL; + cmsg = CMSG_NXTHDR(&hdr, cmsg)) { + + if (cmsg->cmsg_level != SOL_SOCKET) + continue; + + switch (cmsg->cmsg_type) { + case SCM_TIMESTAMPING: + ts = (struct scm_timestamping *)CMSG_DATA(cmsg); + if (ts->ts[2].tv_sec || ts->ts[2].tv_nsec) { + printf("found skb hwtstamp = %lu.%lu\n", + ts->ts[2].tv_sec, ts->ts[2].tv_nsec); + return; + } + break; + default: + break; + } + } + + printf("skb hwtstamp is not found!\n"); +} + +static int verify_metadata(struct xsk *rx_xsk, int rxq, int server_fd) +{ + const struct xdp_desc *rx_desc; + struct pollfd fds[rxq + 1]; + __u64 comp_addr; + __u64 addr; + __u32 idx; + int ret; + int i; + + for (i = 0; i < rxq; i++) { + fds[i].fd = xsk_socket__fd(rx_xsk[i].socket); + fds[i].events = POLLIN; + fds[i].revents = 0; + } + + fds[rxq].fd = server_fd; + fds[rxq].events = POLLIN; + fds[rxq].revents = 0; + + while (true) { + errno = 0; + ret = poll(fds, rxq + 1, 1000); + printf("poll: %d (%d)\n", ret, errno); + if (ret < 0) + break; + if (ret == 0) + continue; + + if (fds[rxq].revents) + verify_skb_metadata(server_fd); + + for (i = 0; i < rxq; i++) { + if (fds[i].revents == 0) + continue; + + struct xsk *xsk = &rx_xsk[i]; + + ret = xsk_ring_cons__peek(&xsk->rx, 1, &idx); + printf("xsk_ring_cons__peek: %d\n", ret); + if (ret != 1) + continue; + + rx_desc = xsk_ring_cons__rx_desc(&xsk->rx, idx); + comp_addr = xsk_umem__extract_addr(rx_desc->addr); + addr = xsk_umem__add_offset_to_addr(rx_desc->addr); + printf("%p: rx_desc[%u]->addr=%llx addr=%llx comp_addr=%llx\n", + xsk, idx, rx_desc->addr, addr, comp_addr); + verify_xdp_metadata(xsk_umem__get_data(xsk->umem_area, addr)); + xsk_ring_cons__release(&xsk->rx, 1); + refill_rx(xsk, comp_addr); + } + } + + return 0; +} + +struct ethtool_channels { + __u32 cmd; + __u32 max_rx; + __u32 max_tx; + __u32 max_other; + __u32 max_combined; + __u32 rx_count; + __u32 tx_count; + __u32 other_count; + __u32 combined_count; +}; + +#define ETHTOOL_GCHANNELS 0x0000003c /* Get no of channels */ + +static int rxq_num(const char *ifname) +{ + struct ethtool_channels ch = { + .cmd = ETHTOOL_GCHANNELS, + }; + + struct ifreq ifr = { + .ifr_data = (void *)&ch, + }; + strcpy(ifr.ifr_name, ifname); + int fd, ret; + + fd = socket(AF_UNIX, SOCK_DGRAM, 0); + if (fd < 0) + error(-1, errno, "socket"); + + ret = ioctl(fd, SIOCETHTOOL, &ifr); + if (ret < 0) + error(-1, errno, "socket"); + + close(fd); + + return ch.rx_count; +} + +static void cleanup(void) +{ + LIBBPF_OPTS(bpf_xdp_attach_opts, opts); + int ret; + int i; + + if (bpf_obj) { + opts.old_prog_fd = bpf_program__fd(bpf_obj->progs.rx); + if (opts.old_prog_fd >= 0) { + printf("detaching bpf program....\n"); + ret = bpf_xdp_detach(ifindex, XDP_FLAGS, &opts); + if (ret) + printf("failed to detach XDP program: %d\n", ret); + } + } + + for (i = 0; i < rxq; i++) + close_xsk(&rx_xsk[i]); + + if (bpf_obj) + xdp_hw_metadata__destroy(bpf_obj); +} + +static void handle_signal(int sig) +{ + /* interrupting poll() is all we need */ +} + +static void timestamping_enable(int fd, int val) +{ + int ret; + + ret = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val)); + if (ret < 0) + error(-1, errno, "setsockopt(SO_TIMESTAMPING)"); +} + +int main(int argc, char *argv[]) +{ + int server_fd = -1; + int ret; + int i; + + struct bpf_program *prog; + + if (argc != 2) { + fprintf(stderr, "pass device name\n"); + return -1; + } + + ifname = argv[1]; + ifindex = if_nametoindex(ifname); + rxq = rxq_num(ifname); + + printf("rxq: %d\n", rxq); + + rx_xsk = malloc(sizeof(struct xsk) * rxq); + if (!rx_xsk) + error(-1, ENOMEM, "malloc"); + + for (i = 0; i < rxq; i++) { + printf("open_xsk(%s, %p, %d)\n", ifname, &rx_xsk[i], i); + ret = open_xsk(ifname, &rx_xsk[i], i); + if (ret) + error(-1, -ret, "open_xsk"); + + printf("xsk_socket__fd() -> %d\n", xsk_socket__fd(rx_xsk[i].socket)); + } + + printf("open bpf program...\n"); + bpf_obj = xdp_hw_metadata__open(); + if (libbpf_get_error(bpf_obj)) + error(-1, libbpf_get_error(bpf_obj), "xdp_hw_metadata__open"); + + prog = bpf_object__find_program_by_name(bpf_obj->obj, "rx"); + bpf_program__set_ifindex(prog, ifindex); + bpf_program__set_flags(prog, BPF_F_XDP_HAS_METADATA); + + printf("load bpf program...\n"); + ret = xdp_hw_metadata__load(bpf_obj); + if (ret) + error(-1, -ret, "xdp_hw_metadata__load"); + + printf("prepare skb endpoint...\n"); + server_fd = start_server(AF_INET6, SOCK_DGRAM, NULL, 9092, 1000); + if (server_fd < 0) + error(-1, errno, "start_server"); + timestamping_enable(server_fd, + SOF_TIMESTAMPING_SOFTWARE | + SOF_TIMESTAMPING_RAW_HARDWARE); + + printf("prepare xsk map...\n"); + for (i = 0; i < rxq; i++) { + int sock_fd = xsk_socket__fd(rx_xsk[i].socket); + __u32 queue_id = i; + + printf("map[%d] = %d\n", queue_id, sock_fd); + ret = bpf_map_update_elem(bpf_map__fd(bpf_obj->maps.xsk), &queue_id, &sock_fd, 0); + if (ret) + error(-1, -ret, "bpf_map_update_elem"); + } + + printf("attach bpf program...\n"); + ret = bpf_xdp_attach(ifindex, + bpf_program__fd(bpf_obj->progs.rx), + XDP_FLAGS, NULL); + if (ret) + error(-1, -ret, "bpf_xdp_attach"); + + signal(SIGINT, handle_signal); + ret = verify_metadata(rx_xsk, rxq, server_fd); + close(server_fd); + cleanup(); + if (ret) + error(-1, -ret, "verify_metadata"); +}