From patchwork Thu Oct 27 20:00:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13022673 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 015A1FA3743 for ; Thu, 27 Oct 2022 20:00:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236881AbiJ0UAz (ORCPT ); Thu, 27 Oct 2022 16:00:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34028 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236890AbiJ0UAc (ORCPT ); Thu, 27 Oct 2022 16:00:32 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D54E2DEE for ; Thu, 27 Oct 2022 13:00:24 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id a2-20020a5b0002000000b006b48689da76so2468511ybp.16 for ; Thu, 27 Oct 2022 13:00:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=f+NDMLwTx3z6guRuLy0i3T+BrbbI2yk7jdZCWNoru5w=; b=kckCvJK1ZXuP0y/wT3kKWVuzeKfZ34hPXaWMB10PG64ne2ldY/dTc6cNbHAd43x+wd 6a40em8FT9rfAGXWJfM+VXtAV3aE4mFVZGXRqRpyeXeUyRCbGStDNHOggmgnhav6qrkJ Z+N/BWOt2XGAezr6SlCfRXo3SqvOPQQC8w842asEuTNz0iX7rZDEVRRPmMKMsG+aQHsg vMPRNDdJFWAYFpshd9g3KujtU62mgC92oOXlQkKi1vpgXAE6wLOx7prr4apDn2PrV17x JhRaiaGMRJ8jYDK+BA7nEuIUqt3PXcMUg4Jgs+cSSHY4F7syvSKGBzm6sGJlJWPPh3x1 INCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=f+NDMLwTx3z6guRuLy0i3T+BrbbI2yk7jdZCWNoru5w=; b=i6HQ6N59DbwwMR0yn5gxhjjDsntWLI/LbWunfOEECKe2sBR4EvNM6hZwL/csDVNK37 gK53wakOeBn2il5Ra5BAam2B+PDkg3WCgu5fMgSCKu1SAd/V7QihXy6UDfJkeZObnWXk 7hAHAbRzMiFdtEyoC7Oify0JxwbxRSIczGXWA/3QzbnRSxvkb5lftS/gwq4zHO6LgxtD 57TOxB+mrlLuU6Ksxt5lq6Nf9Gi9wX2PUmNpRxvjats455mNtmFM0F2EpX3IOEO/wd6c ul3GctuxlywcC1F9pRfnx2O24VFWh1/a/gqH8sicz0hBawREmbswiapEKer78JSSefXw tDvw== X-Gm-Message-State: ACrzQf3UJExGD/8k4R0UiC4fPJh4vZyOSU841TzcUX7XjZnqsfuMPN2K lIiDkP5XKXe032tffq/q5WFXV/4= X-Google-Smtp-Source: AMsMyM7shuumXhH3cueW3Q+n6aS7ZRgu8sfCPjdoy4zGJbuHTtQspR15iQdTXt9QZK+YQwVYN4EKwS4= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a05:6902:124e:b0:668:222c:e8da with SMTP id t14-20020a056902124e00b00668222ce8damr44730950ybu.383.1666900823694; Thu, 27 Oct 2022 13:00:23 -0700 (PDT) Date: Thu, 27 Oct 2022 13:00:15 -0700 In-Reply-To: <20221027200019.4106375-1-sdf@google.com> Mime-Version: 1.0 References: <20221027200019.4106375-1-sdf@google.com> X-Mailer: git-send-email 2.38.1.273.g43a17bfeac-goog Message-ID: <20221027200019.4106375-2-sdf@google.com> Subject: [RFC bpf-next 1/5] bpf: Support inlined/unrolled kfuncs for xdp metadata From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, Jakub Kicinski , Willem de Bruijn , Jesper Dangaard Brouer , Anatoly Burakov , Alexander Lobakin , Magnus Karlsson , Maryam Tahhan , xdp-hints@xdp-project.net, netdev@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Kfuncs have to be defined with KF_UNROLL for an attempted unroll. For now, only XDP programs can have their kfuncs unrolled, but we can extend this later on if more programs would like to use it. For XDP, we define a new kfunc set (xdp_metadata_kfunc_ids) which implements all possible metatada kfuncs. Not all devices have to implement them. If unrolling is not supported by the target device, the default implementation is called instead. We might also unroll this default implementation for performance sake in the future, but that's something I'm putting off for now. Upon loading, if BPF_F_XDP_HAS_METADATA is passed via prog_flags, we treat prog_index as target device for kfunc unrolling. net_device_ops gains new ndo_unroll_kfunc which does the actual dirty work per device. The kfunc unrolling itself largely follows the existing map_gen_lookup unrolling example, so there is nothing new here. Cc: Martin KaFai Lau Cc: Jakub Kicinski Cc: Willem de Bruijn Cc: Jesper Dangaard Brouer Cc: Anatoly Burakov Cc: Alexander Lobakin Cc: Magnus Karlsson Cc: Maryam Tahhan Cc: xdp-hints@xdp-project.net Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- include/linux/bpf.h | 1 + include/linux/btf.h | 1 + include/linux/btf_ids.h | 4 +++ include/linux/netdevice.h | 3 ++ include/net/xdp.h | 22 +++++++++++++ include/uapi/linux/bpf.h | 5 +++ kernel/bpf/syscall.c | 28 +++++++++++++++- kernel/bpf/verifier.c | 60 ++++++++++++++++++++++++++++++++++ net/core/dev.c | 7 ++++ net/core/xdp.c | 28 ++++++++++++++++ tools/include/uapi/linux/bpf.h | 5 +++ 11 files changed, 163 insertions(+), 1 deletion(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 9fd68b0b3e9c..54983347e20e 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1176,6 +1176,7 @@ struct bpf_prog_aux { struct work_struct work; struct rcu_head rcu; }; + const struct net_device_ops *xdp_kfunc_ndo; }; struct bpf_prog { diff --git a/include/linux/btf.h b/include/linux/btf.h index f9aababc5d78..23ad5f8313e4 100644 --- a/include/linux/btf.h +++ b/include/linux/btf.h @@ -51,6 +51,7 @@ #define KF_TRUSTED_ARGS (1 << 4) /* kfunc only takes trusted pointer arguments */ #define KF_SLEEPABLE (1 << 5) /* kfunc may sleep */ #define KF_DESTRUCTIVE (1 << 6) /* kfunc performs destructive actions */ +#define KF_UNROLL (1 << 7) /* kfunc unrolling can be attempted */ /* * Return the name of the passed struct, if exists, or halt the build if for diff --git a/include/linux/btf_ids.h b/include/linux/btf_ids.h index c9744efd202f..eb448e9c79bb 100644 --- a/include/linux/btf_ids.h +++ b/include/linux/btf_ids.h @@ -195,6 +195,10 @@ asm( \ __BTF_ID_LIST(name, local) \ __BTF_SET8_START(name, local) +#define BTF_SET8_START_GLOBAL(name) \ +__BTF_ID_LIST(name, global) \ +__BTF_SET8_START(name, global) + #define BTF_SET8_END(name) \ asm( \ ".pushsection " BTF_IDS_SECTION ",\"a\"; \n" \ diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index a36edb0ec199..90052271a502 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -73,6 +73,7 @@ struct udp_tunnel_info; struct udp_tunnel_nic_info; struct udp_tunnel_nic; struct bpf_prog; +struct bpf_insn; struct xdp_buff; void synchronize_net(void); @@ -1609,6 +1610,8 @@ struct net_device_ops { ktime_t (*ndo_get_tstamp)(struct net_device *dev, const struct skb_shared_hwtstamps *hwtstamps, bool cycles); + int (*ndo_unroll_kfunc)(struct bpf_prog *prog, + struct bpf_insn *insn); }; /** diff --git a/include/net/xdp.h b/include/net/xdp.h index 55dbc68bfffc..b465001936ac 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -7,6 +7,7 @@ #define __LINUX_NET_XDP_H__ #include /* skb_shared_info */ +#include /* btf_id_set8 */ /** * DOC: XDP RX-queue information @@ -83,6 +84,7 @@ struct xdp_buff { struct xdp_txq_info *txq; u32 frame_sz; /* frame size to deduce data_hard_end/reserved tailroom*/ u32 flags; /* supported values defined in xdp_buff_flags */ + void *priv; }; static __always_inline bool xdp_buff_has_frags(struct xdp_buff *xdp) @@ -409,4 +411,24 @@ void xdp_attachment_setup(struct xdp_attachment_info *info, #define DEV_MAP_BULK_SIZE XDP_BULK_QUEUE_SIZE +#define XDP_METADATA_KFUNC_xxx \ + XDP_METADATA_KFUNC(XDP_METADATA_KFUNC_HAVE_RX_TIMESTAMP, \ + bpf_xdp_metadata_have_rx_timestamp) \ + XDP_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_TIMESTAMP, \ + bpf_xdp_metadata_rx_timestamp) \ + +extern struct btf_id_set8 xdp_metadata_kfunc_ids; + +enum { +#define XDP_METADATA_KFUNC(name, str) name, +XDP_METADATA_KFUNC_xxx +#undef XDP_METADATA_KFUNC +MAX_XDP_METADATA_KFUNC, +}; + +static inline u32 xdp_metadata_kfunc_id(int id) +{ + return xdp_metadata_kfunc_ids.pairs[id].id; +} + #endif /* __LINUX_NET_XDP_H__ */ diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 94659f6b3395..6938fc4f1ec5 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1156,6 +1156,11 @@ enum bpf_link_type { */ #define BPF_F_XDP_HAS_FRAGS (1U << 5) +/* If BPF_F_XDP_HAS_METADATA is used in BPF_PROG_LOAD command, the loaded + * program becomes device-bound but can access it's XDP metadata. + */ +#define BPF_F_XDP_HAS_METADATA (1U << 6) + /* link_create.kprobe_multi.flags used in LINK_CREATE command for * BPF_TRACE_KPROBE_MULTI attach type to create return probe. */ diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 11df90962101..5376604961bc 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2461,6 +2461,20 @@ static bool is_perfmon_prog_type(enum bpf_prog_type prog_type) /* last field in 'union bpf_attr' used by this command */ #define BPF_PROG_LOAD_LAST_FIELD core_relo_rec_size +static int xdp_resolve_netdev(struct bpf_prog *prog, int ifindex) +{ + struct net *net = current->nsproxy->net_ns; + struct net_device *dev; + + for_each_netdev(net, dev) { + if (dev->ifindex == ifindex) { + prog->aux->xdp_kfunc_ndo = dev->netdev_ops; + return 0; + } + } + return -EINVAL; +} + static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr) { enum bpf_prog_type type = attr->prog_type; @@ -2478,7 +2492,8 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr) BPF_F_TEST_STATE_FREQ | BPF_F_SLEEPABLE | BPF_F_TEST_RND_HI32 | - BPF_F_XDP_HAS_FRAGS)) + BPF_F_XDP_HAS_FRAGS | + BPF_F_XDP_HAS_METADATA)) return -EINVAL; if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) && @@ -2566,6 +2581,17 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr) prog->aux->sleepable = attr->prog_flags & BPF_F_SLEEPABLE; prog->aux->xdp_has_frags = attr->prog_flags & BPF_F_XDP_HAS_FRAGS; + if (attr->prog_flags & BPF_F_XDP_HAS_METADATA) { + /* Reuse prog_ifindex to carry request to unroll + * metadata kfuncs. + */ + prog->aux->offload_requested = false; + + err = xdp_resolve_netdev(prog, attr->prog_ifindex); + if (err < 0) + goto free_prog; + } + err = security_bpf_prog_alloc(prog->aux); if (err) goto free_prog; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 8f849a763b79..3e734e15ffa3 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -13864,6 +13864,37 @@ static int fixup_call_args(struct bpf_verifier_env *env) return err; } +static int unroll_kfunc_call(struct bpf_verifier_env *env, + struct bpf_insn *insn) +{ + enum bpf_prog_type prog_type; + struct bpf_prog_aux *aux; + struct btf *desc_btf; + u32 *kfunc_flags; + u32 func_id; + + desc_btf = find_kfunc_desc_btf(env, insn->off); + if (IS_ERR(desc_btf)) + return PTR_ERR(desc_btf); + + prog_type = resolve_prog_type(env->prog); + func_id = insn->imm; + + kfunc_flags = btf_kfunc_id_set_contains(desc_btf, prog_type, func_id); + if (!kfunc_flags) + return 0; + if (!(*kfunc_flags & KF_UNROLL)) + return 0; + if (prog_type != BPF_PROG_TYPE_XDP) + return 0; + + aux = env->prog->aux; + if (!aux->xdp_kfunc_ndo) + return 0; + + return aux->xdp_kfunc_ndo->ndo_unroll_kfunc(env->prog, insn); +} + static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn) { @@ -14027,6 +14058,35 @@ static int do_misc_fixups(struct bpf_verifier_env *env) if (insn->src_reg == BPF_PSEUDO_CALL) continue; if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) { + if (bpf_prog_is_dev_bound(env->prog->aux)) { + verbose(env, "no metadata kfuncs offload\n"); + return -EINVAL; + } + + insn_buf[0] = *insn; + + cnt = unroll_kfunc_call(env, insn_buf); + if (cnt < 0) { + verbose(env, "failed to unroll kfunc with func_id=%d\n", insn->imm); + return cnt; + } + if (cnt > 0) { + if (cnt >= ARRAY_SIZE(insn_buf)) { + verbose(env, "bpf verifier is misconfigured (kfunc unroll)\n"); + return -EINVAL; + } + + new_prog = bpf_patch_insn_data(env, i + delta, + insn_buf, cnt); + if (!new_prog) + return -ENOMEM; + + delta += cnt - 1; + env->prog = prog = new_prog; + insn = new_prog->insnsi + i + delta; + continue; + } + ret = fixup_kfunc_call(env, insn); if (ret) return ret; diff --git a/net/core/dev.c b/net/core/dev.c index fa53830d0683..8d3e34d073a8 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -9254,6 +9254,13 @@ static int dev_xdp_attach(struct net_device *dev, struct netlink_ext_ack *extack return -EOPNOTSUPP; } + if (new_prog && + new_prog->aux->xdp_kfunc_ndo && + new_prog->aux->xdp_kfunc_ndo != dev->netdev_ops) { + NL_SET_ERR_MSG(extack, "Target device was specified at load time; can only attach to the same device type"); + return -EINVAL; + } + err = dev_xdp_install(dev, mode, bpf_op, extack, flags, new_prog); if (err) return err; diff --git a/net/core/xdp.c b/net/core/xdp.c index 844c9d99dc0e..ea4ff00cb22b 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -4,6 +4,7 @@ * Copyright (c) 2017 Jesper Dangaard Brouer, Red Hat Inc. */ #include +#include #include #include #include @@ -709,3 +710,30 @@ struct xdp_frame *xdpf_clone(struct xdp_frame *xdpf) return nxdpf; } + +noinline int bpf_xdp_metadata_have_rx_timestamp(struct xdp_md *ctx) +{ + return false; +} + +noinline __u32 bpf_xdp_metadata_rx_timestamp(struct xdp_md *ctx) +{ + return 0; +} + +BTF_SET8_START_GLOBAL(xdp_metadata_kfunc_ids) +#define XDP_METADATA_KFUNC(name, str) BTF_ID_FLAGS(func, str, KF_UNROLL) +XDP_METADATA_KFUNC_xxx +#undef XDP_METADATA_KFUNC +BTF_SET8_END(xdp_metadata_kfunc_ids) + +static const struct btf_kfunc_id_set xdp_metadata_kfunc_set = { + .owner = THIS_MODULE, + .set = &xdp_metadata_kfunc_ids, +}; + +static int __init xdp_metadata_init(void) +{ + return register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &xdp_metadata_kfunc_set); +} +late_initcall(xdp_metadata_init); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 94659f6b3395..6938fc4f1ec5 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -1156,6 +1156,11 @@ enum bpf_link_type { */ #define BPF_F_XDP_HAS_FRAGS (1U << 5) +/* If BPF_F_XDP_HAS_METADATA is used in BPF_PROG_LOAD command, the loaded + * program becomes device-bound but can access it's XDP metadata. + */ +#define BPF_F_XDP_HAS_METADATA (1U << 6) + /* link_create.kprobe_multi.flags used in LINK_CREATE command for * BPF_TRACE_KPROBE_MULTI attach type to create return probe. */ From patchwork Thu Oct 27 20:00:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13022674 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7830ECAAA1 for ; Thu, 27 Oct 2022 20:00:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236883AbiJ0UA4 (ORCPT ); Thu, 27 Oct 2022 16:00:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35306 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236889AbiJ0UAc (ORCPT ); Thu, 27 Oct 2022 16:00:32 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 04CCE11829 for ; Thu, 27 Oct 2022 13:00:26 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id o15-20020a17090aac0f00b00212e93524c0so4258911pjq.2 for ; Thu, 27 Oct 2022 13:00:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=x1Wy4MtVrL47Mg6ubq0m6la2HOdUZD2ps1PI5efyQcs=; b=SyIs+Emv1OBwjcyUosDYnXifGanJYSjH84/nXaLyaA5afcxeH3mO8R0OkOrVKyLtud colO6y3DBMP++hfxqwQiWOEw8/pwZvEj1kYeJFT2Xw3X9ZltE5YGffc67Ko8KpATeT2K a+WBWCh9VZZTbOLPAqeamWIpZDzp5a1i4xXDZIewe0q9rMxAULjB1A8SBhCDQ8wx1V/l 8Qa5HthLZz94xj48WkINPKiY4QnYILSE+SEtwsN5CzXuBS+F8stvHwCmbd7bgAzV1yWU qjPXtDvukNrC+qqR1vfQLEdObH869HnuF6Qpen3iytklLrT7RmDZRKxScbVwvWXoy5Z7 iK2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=x1Wy4MtVrL47Mg6ubq0m6la2HOdUZD2ps1PI5efyQcs=; b=HVgKxVre8X8JXm+pQH/AQSy38ZF1pbKvDhoGt/y5y+kEegM5km1YfhW4mctnl1QEm4 sOrEMFEGVLWW1qDK/HrQgsYH0ZSMjlsfwOfGOzbc96Mc26qXuw3XRmvHSwErSxPFcsCW jub/eCe02acwscAn7g1pMk8KgLgbPUb13kTef83e4sJ9YUXkW+RhFpK5Vvi60ZnhbmWk WrhP3mvHMBlsjoKIwJkyKDX92InVg4DEpRyKDGU8AoCu3dZ/MSSY8N/RlLpC9dNaCFoL YcRzj2MZTY4o1D2yhmZhxg7CuraygmdlnvR+70oY4a/wwDyJZAB82WHIypoM08+GUV15 k+yQ== X-Gm-Message-State: ACrzQf0r5N3uhu+hi2wAbMXnpuIB1ikvCxIOHhdWx8S0b+M4veFKAv+C m/Ybirohnu1Afr9jIgZUcyv1tSo= X-Google-Smtp-Source: AMsMyM74Nmf3sqNhRUcydUqTY8hM2m32VR/qgcCbmIBnAg2SN9JSYoBjNFvsPVPTnWinR5B/f4Wg1/4= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a17:903:2447:b0:186:e200:6109 with SMTP id l7-20020a170903244700b00186e2006109mr8840520pls.104.1666900825517; Thu, 27 Oct 2022 13:00:25 -0700 (PDT) Date: Thu, 27 Oct 2022 13:00:16 -0700 In-Reply-To: <20221027200019.4106375-1-sdf@google.com> Mime-Version: 1.0 References: <20221027200019.4106375-1-sdf@google.com> X-Mailer: git-send-email 2.38.1.273.g43a17bfeac-goog Message-ID: <20221027200019.4106375-3-sdf@google.com> Subject: [RFC bpf-next 2/5] veth: Support rx timestamp metadata for xdp From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, Jakub Kicinski , Willem de Bruijn , Jesper Dangaard Brouer , Anatoly Burakov , Alexander Lobakin , Magnus Karlsson , Maryam Tahhan , xdp-hints@xdp-project.net, netdev@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC xskxceiver conveniently setups up veth pairs so it seems logical to use veth as an example for some of the metadata handling. We timestamp skb right when we "receive" it, store its pointer in xdp_buff->priv and generate BPF bytecode to reach it from the BPF program. This largely follows the idea of "store some queue context in the xdp_buff/xdp_frame so the metadata can be reached out from the BPF program". Cc: Martin KaFai Lau Cc: Jakub Kicinski Cc: Willem de Bruijn Cc: Jesper Dangaard Brouer Cc: Anatoly Burakov Cc: Alexander Lobakin Cc: Magnus Karlsson Cc: Maryam Tahhan Cc: xdp-hints@xdp-project.net Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- drivers/net/veth.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/drivers/net/veth.c b/drivers/net/veth.c index 09682ea3354e..35396dd73de0 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -597,6 +597,7 @@ static struct xdp_frame *veth_xdp_rcv_one(struct veth_rq *rq, xdp_convert_frame_to_buff(frame, &xdp); xdp.rxq = &rq->xdp_rxq; + xdp.priv = NULL; act = bpf_prog_run_xdp(xdp_prog, &xdp); @@ -820,6 +821,7 @@ static struct sk_buff *veth_xdp_rcv_skb(struct veth_rq *rq, orig_data = xdp.data; orig_data_end = xdp.data_end; + xdp.priv = skb; act = bpf_prog_run_xdp(xdp_prog, &xdp); @@ -936,6 +938,7 @@ static int veth_xdp_rcv(struct veth_rq *rq, int budget, struct sk_buff *skb = ptr; stats->xdp_bytes += skb->len; + __net_timestamp(skb); skb = veth_xdp_rcv_skb(rq, skb, bq, stats); if (skb) { if (skb_shared(skb) || skb_unclone(skb, GFP_ATOMIC)) @@ -1595,6 +1598,33 @@ static int veth_xdp(struct net_device *dev, struct netdev_bpf *xdp) } } +static int veth_unroll_kfunc(struct bpf_prog *prog, struct bpf_insn *insn) +{ + u32 func_id = insn->imm; + + if (func_id == xdp_metadata_kfunc_id(XDP_METADATA_KFUNC_HAVE_RX_TIMESTAMP)) { + /* return true; */ + insn[0] = BPF_MOV64_IMM(BPF_REG_0, 1); + return 1; + } else if (func_id == xdp_metadata_kfunc_id(XDP_METADATA_KFUNC_RX_TIMESTAMP)) { + /* r1 = ((struct xdp_buff *)r1)->priv; [skb] */ + insn[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, + offsetof(struct xdp_buff, priv)); + /* if (r1 == NULL) { */ + insn[1] = BPF_JMP_IMM(BPF_JEQ, BPF_REG_1, 0, 1); + /* return 0; */ + insn[2] = BPF_MOV64_IMM(BPF_REG_0, 0); + /* } else { */ + /* return ((struct sk_buff *)r1)->tstamp; */ + insn[3] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, + offsetof(struct sk_buff, tstamp)); + /* } */ + return 4; + } + + return 0; +} + static const struct net_device_ops veth_netdev_ops = { .ndo_init = veth_dev_init, .ndo_open = veth_open, @@ -1614,6 +1644,7 @@ static const struct net_device_ops veth_netdev_ops = { .ndo_bpf = veth_xdp, .ndo_xdp_xmit = veth_ndo_xdp_xmit, .ndo_get_peer_dev = veth_peer_dev, + .ndo_unroll_kfunc = veth_unroll_kfunc, }; #define VETH_FEATURES (NETIF_F_SG | NETIF_F_FRAGLIST | NETIF_F_HW_CSUM | \ From patchwork Thu Oct 27 20:00:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13022675 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3746BFA3740 for ; Thu, 27 Oct 2022 20:01:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236887AbiJ0UA6 (ORCPT ); Thu, 27 Oct 2022 16:00:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34032 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236830AbiJ0UAc (ORCPT ); Thu, 27 Oct 2022 16:00:32 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 154CE6141 for ; Thu, 27 Oct 2022 13:00:28 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-3697bd55974so23420867b3.15 for ; Thu, 27 Oct 2022 13:00:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=RP55M8cA5M2RXo8rOon2JD5KhcZkcbkmc4ixPrmMem4=; b=hjkhJabFkn0xjp3ooIiT6qUbyMYEDUSI2QuChlBbDXQQ5HNfNN8+I+bFe8j2cBJgoZ oRPArOxcW/M864tAkzcPY671i8RDYxN32TcbY0cIcLB2IwIWfm6AVePXM2FVjeCo0b2R Bx4Eqgyb9+H6iWLZjDlytBat4d6JouHpsFYU3tmCifz24e6LWempBi7DjILFIksvBAt1 omEbPEMyso4TjhRN6ebWSGvztHkBG22SVWdgcyWjYlPZhUKir1TIeop1Iduzc2g3Lz+g s+cQkgUSf4zaL0tQ6quMUjRUaiqlq0sr3IkqoAEnSu6XW6aIbWEUJ1Ys/3cTE8RapL1T L9QA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=RP55M8cA5M2RXo8rOon2JD5KhcZkcbkmc4ixPrmMem4=; b=tk2MZpiH8mAthxPGKY58hPKP4EHhw7ak+7PxwpzV76PMOQjY+4DYfeW7SsLqBM9j40 vqZxKW5y8o83sRQIr701vAptO0s9H6tqHlcbrujHSt8tHC7ZJfaRUzCs+3l0eVagtDAB enX3WOiO7E9UtQL2vkL3X6MKeNxar+q7PWZjS132sUt7w/AYLrU+uSfVvkhb1j0Hpxi5 PUxahFEfQRCFVxyoUWF9fR+Z0v0UagQbWidGFDqo7O23Sst1jdNRWu4wlcQf8xcSwFhv 7wu9ltxSpFskDQ4vzrJ+LxFS/ojyo+HgA0cKE88Df9w36Ye43wJ7Wy8PE8QbppdiRRsu 1ccg== X-Gm-Message-State: ACrzQf33BurxDPzhUgApwS64oRU/iKyuKWwpuPMV7ApNhSCJ64idHrsx 8oLAbmu0H8J3Rb7EXSI6sjKTy4M= X-Google-Smtp-Source: AMsMyM77QeyllYbofvOa5Xukf+6LO5LkGydr8MQ+Mrn0YajmX3NrsZDFSyNNTfTCbD+ZFdb3Hxi/bY0= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a05:6902:10a:b0:6af:b884:840e with SMTP id o10-20020a056902010a00b006afb884840emr46744379ybh.330.1666900827359; Thu, 27 Oct 2022 13:00:27 -0700 (PDT) Date: Thu, 27 Oct 2022 13:00:17 -0700 In-Reply-To: <20221027200019.4106375-1-sdf@google.com> Mime-Version: 1.0 References: <20221027200019.4106375-1-sdf@google.com> X-Mailer: git-send-email 2.38.1.273.g43a17bfeac-goog Message-ID: <20221027200019.4106375-4-sdf@google.com> Subject: [RFC bpf-next 3/5] libbpf: Pass prog_ifindex via bpf_object_open_opts From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, Jakub Kicinski , Willem de Bruijn , Jesper Dangaard Brouer , Anatoly Burakov , Alexander Lobakin , Magnus Karlsson , Maryam Tahhan , xdp-hints@xdp-project.net, netdev@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Allow passing prog_ifindex to BPF_PROG_LOAD. This patch is not XDP metadata specific but it's here because we (ab)use prog_ifindex as "target metadata" device during loading. We can figure out proper UAPI story if we decide to go forward with the kfunc approach. Cc: Martin KaFai Lau Cc: Jakub Kicinski Cc: Willem de Bruijn Cc: Jesper Dangaard Brouer Cc: Anatoly Burakov Cc: Alexander Lobakin Cc: Magnus Karlsson Cc: Maryam Tahhan Cc: xdp-hints@xdp-project.net Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- tools/lib/bpf/libbpf.c | 1 + tools/lib/bpf/libbpf.h | 6 +++++- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 5d7819edf074..61bc37006fe4 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -7190,6 +7190,7 @@ static int bpf_object_init_progs(struct bpf_object *obj, const struct bpf_object prog->type = prog->sec_def->prog_type; prog->expected_attach_type = prog->sec_def->expected_attach_type; + prog->prog_ifindex = opts->prog_ifindex; /* sec_def can have custom callback which should be called * after bpf_program is initialized to adjust its properties diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h index eee883f007f9..4a40b7623099 100644 --- a/tools/lib/bpf/libbpf.h +++ b/tools/lib/bpf/libbpf.h @@ -170,9 +170,13 @@ struct bpf_object_open_opts { */ __u32 kernel_log_level; + /* Optional ifindex of netdev for offload purposes. + */ + int prog_ifindex; + size_t :0; }; -#define bpf_object_open_opts__last_field kernel_log_level +#define bpf_object_open_opts__last_field prog_ifindex LIBBPF_API struct bpf_object *bpf_object__open(const char *path); From patchwork Thu Oct 27 20:00:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13022676 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D81EFA3740 for ; Thu, 27 Oct 2022 20:01:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236925AbiJ0UBS (ORCPT ); Thu, 27 Oct 2022 16:01:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33902 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236874AbiJ0UAe (ORCPT ); Thu, 27 Oct 2022 16:00:34 -0400 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AAF04495CC for ; Thu, 27 Oct 2022 13:00:29 -0700 (PDT) Received: by mail-pl1-x649.google.com with SMTP id n1-20020a170902f60100b00179c0a5c51fso1705762plg.7 for ; Thu, 27 Oct 2022 13:00:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1k5yzkMHPKVpJxfSBsEcHHj8plrhgO9vuyoJKASsEmU=; b=n/UREpPnnfxUxStjCXxqowwL2ThEEnX9FQtVh5YzRyLm3FRk8nYtEeSPHheYShmRFE dQp+z/ltMDVcH40cWXqO3ducFy/S7FQXAzoPHRPsCL89E/HlgdKue7moQYaPuacXzsNQ egrDtG9z3RAfm624yVeFqOAVBb5r73TaRylF8e6OGqkqw5gEhG4l3plf2py/ZEet2I3O XFrwzOlGrQPJr/z3P9IO7bCQSMQNgwGlLIgw6dcxC3CyGF+C7D1NtEgc+vg6r+jqtEQe JLkSyfiJx+sIL1bAkiyESVld9LUyeR0ESNa4iSOT42G0oFTQcGaQMhFklsehNpGmQG3w b4GQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1k5yzkMHPKVpJxfSBsEcHHj8plrhgO9vuyoJKASsEmU=; b=gjYjxh1UAodKK9KwskkO1Kkf1bkcAjjlQ/Av74TkHb/fY2vgZI8OZHnolPk/W0q9++ xNP97qr8QwwugIickyzFprk5IpW9JYNaDmtaQJ4CzWgnsVS/3SPWGam35hNIeyUbtjpj 2zValH+nnZT+4kmX46Z1bjV+0TOTCM8T2dyG6qpKCn0K112ZR+a3jAKky8Y5VV3Gwl1k J2piH0DMbQg99mmf+q02/fuInTRvyJKHZRlG36Z/TszQ8ItRHIV7b2PuZY73SJmUMVOU O3M5TuOhKm+0yCZZab64FWwErkPGW3P8TPhcLIMOWFqQrDp5z9ZrhoYjXfqnCHcheljJ rQCg== X-Gm-Message-State: ACrzQf0rx2AAl9CzQgkd3go2pT4rrohlTTA6rvs4fEzatiwM2pHFzZBg re6V1acATHis7vzz83UEgVMX7Cs= X-Google-Smtp-Source: AMsMyM42i2qmj5/tum0aiLkRrGO//i6iA+au9HoLSXhG8Jardj8mUd8pmeDSUGDth8T8hAOsIOqdmsQ= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a05:6a00:80e:b0:563:4ad2:9d39 with SMTP id m14-20020a056a00080e00b005634ad29d39mr51074982pfk.66.1666900828817; Thu, 27 Oct 2022 13:00:28 -0700 (PDT) Date: Thu, 27 Oct 2022 13:00:18 -0700 In-Reply-To: <20221027200019.4106375-1-sdf@google.com> Mime-Version: 1.0 References: <20221027200019.4106375-1-sdf@google.com> X-Mailer: git-send-email 2.38.1.273.g43a17bfeac-goog Message-ID: <20221027200019.4106375-5-sdf@google.com> Subject: [RFC bpf-next 4/5] selftests/bpf: Convert xskxceiver to use custom program From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, Jakub Kicinski , Willem de Bruijn , Jesper Dangaard Brouer , Anatoly Burakov , Alexander Lobakin , Magnus Karlsson , Maryam Tahhan , xdp-hints@xdp-project.net, netdev@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC No functional changes (in theory): convert libxsk-generated program bytecode to the C code to better illustrate kfunc metadata (see next patch in the series). There is also a bunch of unrelated changes, ignore them for the sake of demo: - stats.rx_dopped == 2048 vs 2047 ? - buggy ksft_print_msg calls - test is limited only to TEST_MODE_DRV+TEST_TYPE_RUN_TO_COMPLETION_SINGLE_PKT Cc: Martin KaFai Lau Cc: Jakub Kicinski Cc: Willem de Bruijn Cc: Jesper Dangaard Brouer Cc: Anatoly Burakov Cc: Alexander Lobakin Cc: Magnus Karlsson Cc: Maryam Tahhan Cc: xdp-hints@xdp-project.net Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- tools/testing/selftests/bpf/Makefile | 1 + .../testing/selftests/bpf/progs/xskxceiver.c | 21 ++++ tools/testing/selftests/bpf/xskxceiver.c | 98 +++++++++++++++---- tools/testing/selftests/bpf/xskxceiver.h | 5 +- 4 files changed, 105 insertions(+), 20 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/xskxceiver.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 79edef1dbda4..3cab2e1b0e74 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -378,6 +378,7 @@ linked_maps.skel.h-deps := linked_maps1.bpf.o linked_maps2.bpf.o test_subskeleton.skel.h-deps := test_subskeleton_lib2.bpf.o test_subskeleton_lib.bpf.o test_subskeleton.bpf.o test_subskeleton_lib.skel.h-deps := test_subskeleton_lib2.bpf.o test_subskeleton_lib.bpf.o test_usdt.skel.h-deps := test_usdt.bpf.o test_usdt_multispec.bpf.o +xskxceiver-deps := xskxceiver.bpf.o LINKED_BPF_SRCS := $(patsubst %.bpf.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(skel)-deps))) diff --git a/tools/testing/selftests/bpf/progs/xskxceiver.c b/tools/testing/selftests/bpf/progs/xskxceiver.c new file mode 100644 index 000000000000..b135daddad3a --- /dev/null +++ b/tools/testing/selftests/bpf/progs/xskxceiver.c @@ -0,0 +1,21 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include + +#include +#include + +struct { + __uint(type, BPF_MAP_TYPE_XSKMAP); + __uint(max_entries, 4); + __type(key, __u32); + __type(value, __u32); +} xsk SEC(".maps"); + +SEC("xdp") +int rx(struct xdp_md *ctx) +{ + return bpf_redirect_map(&xsk, ctx->rx_queue_index, XDP_PASS); +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/xskxceiver.c b/tools/testing/selftests/bpf/xskxceiver.c index 681a5db80dae..066bd691db13 100644 --- a/tools/testing/selftests/bpf/xskxceiver.c +++ b/tools/testing/selftests/bpf/xskxceiver.c @@ -399,6 +399,58 @@ static void usage(const char *prog) ksft_print_msg(str, prog); } +static void bpf_update_xsk_map(struct ifobject *ifobject, __u32 queue_id) +{ + int map_fd; + int sock_fd; + int ret; + + map_fd = bpf_map__fd(ifobject->bpf_obj->maps.xsk); + sock_fd = xsk_socket__fd(ifobject->xsk->xsk); + + (void)bpf_map_delete_elem(map_fd, &queue_id); + ret = bpf_map_update_elem(map_fd, &queue_id, &sock_fd, 0); + if (ret) + exit_with_error(-ret); +} + +static int bpf_attach(struct ifobject *ifobject) +{ + __u32 prog_id = 0; + + bpf_xdp_query_id(ifobject->ifindex, ifobject->xdp_flags, &prog_id); + if (prog_id) + return 0; + + int ret = bpf_xdp_attach(ifobject->ifindex, + bpf_program__fd(ifobject->bpf_obj->progs.rx), + ifobject->xdp_flags, NULL); + if (ret < 0) { + if (errno != EEXIST && errno != EBUSY) { + exit_with_error(errno); + } + } + + bpf_update_xsk_map(ifobject, 0); + + return 0; +} + +static void bpf_detach(struct ifobject *ifobject) +{ + int my_ns = open("/proc/self/ns/net", O_RDONLY); + + /* Make sure we're in the right namespace when detaching. + * Relevant only for TEST_TYPE_BIDI. + */ + if (ifobject->ns_fd > 0) + setns(ifobject->ns_fd, 0); + + bpf_xdp_detach(ifobject->ifindex, ifobject->xdp_flags, NULL); + + setns(my_ns, 0); +} + static int switch_namespace(const char *nsname) { char fqns[26] = "/var/run/netns/"; @@ -1141,9 +1193,10 @@ static int validate_rx_dropped(struct ifobject *ifobject) if (err) return TEST_FAILURE; - if (stats.rx_dropped == ifobject->pkt_stream->nb_pkts / 2) + if (stats.rx_dropped == ifobject->pkt_stream->nb_pkts / 2 - 1) return TEST_PASS; + printf("%lld != %d\n", stats.rx_dropped, ifobject->pkt_stream->nb_pkts / 2 - 1); return TEST_FAILURE; } @@ -1239,7 +1292,6 @@ static void thread_common_ops_tx(struct test_spec *test, struct ifobject *ifobje { xsk_configure_socket(test, ifobject, test->ifobj_rx->umem, true); ifobject->xsk = &ifobject->xsk_arr[0]; - ifobject->xsk_map_fd = test->ifobj_rx->xsk_map_fd; memcpy(ifobject->umem, test->ifobj_rx->umem, sizeof(struct xsk_umem_info)); } @@ -1284,6 +1336,14 @@ static void thread_common_ops(struct test_spec *test, struct ifobject *ifobject) ifobject->ns_fd = switch_namespace(ifobject->nsname); + ifindex = if_nametoindex(ifobject->ifname); + if (!ifindex) + exit_with_error(errno); + + ifobject->bpf_obj = xskxceiver__open_and_load(); + if (libbpf_get_error(ifobject->bpf_obj)) + exit_with_error(libbpf_get_error(ifobject->bpf_obj)); + if (ifobject->umem->unaligned_mode) mmap_flags |= MAP_HUGETLB; @@ -1307,11 +1367,8 @@ static void thread_common_ops(struct test_spec *test, struct ifobject *ifobject) if (!ifobject->rx_on) return; - ifindex = if_nametoindex(ifobject->ifname); - if (!ifindex) - exit_with_error(errno); - - ret = xsk_setup_xdp_prog_xsk(ifobject->xsk->xsk, &ifobject->xsk_map_fd); + ifobject->ifindex = ifindex; + ret = bpf_attach(ifobject); if (ret) exit_with_error(-ret); @@ -1321,19 +1378,17 @@ static void thread_common_ops(struct test_spec *test, struct ifobject *ifobject) if (ifobject->xdp_flags & XDP_FLAGS_SKB_MODE) { if (opts.attach_mode != XDP_ATTACHED_SKB) { - ksft_print_msg("ERROR: [%s] XDP prog not in SKB mode\n"); + ksft_print_msg("ERROR: XDP prog not in SKB mode\n"); exit_with_error(-EINVAL); } } else if (ifobject->xdp_flags & XDP_FLAGS_DRV_MODE) { if (opts.attach_mode != XDP_ATTACHED_DRV) { - ksft_print_msg("ERROR: [%s] XDP prog not in DRV mode\n"); + ksft_print_msg("ERROR: XDP prog not in DRV mode\n"); exit_with_error(-EINVAL); } } - ret = xsk_socket__update_xskmap(ifobject->xsk->xsk, ifobject->xsk_map_fd); - if (ret) - exit_with_error(-ret); + bpf_update_xsk_map(ifobject, 0); } static void *worker_testapp_validate_tx(void *arg) @@ -1372,8 +1427,7 @@ static void *worker_testapp_validate_rx(void *arg) if (test->current_step == 1) { thread_common_ops(test, ifobject); } else { - bpf_map_delete_elem(ifobject->xsk_map_fd, &id); - xsk_socket__update_xskmap(ifobject->xsk->xsk, ifobject->xsk_map_fd); + bpf_update_xsk_map(ifobject, id); } fds.fd = xsk_socket__fd(ifobject->xsk->xsk); @@ -1481,6 +1535,8 @@ static int testapp_validate_traffic(struct test_spec *test) if (test->total_steps == test->current_step || test->fail) { xsk_socket__delete(ifobj_tx->xsk->xsk); xsk_socket__delete(ifobj_rx->xsk->xsk); + bpf_detach(ifobj_tx); + bpf_detach(ifobj_rx); testapp_clean_xsk_umem(ifobj_rx); if (!ifobj_tx->shared_umem) testapp_clean_xsk_umem(ifobj_tx); @@ -1531,16 +1587,12 @@ static void testapp_bidi(struct test_spec *test) static void swap_xsk_resources(struct ifobject *ifobj_tx, struct ifobject *ifobj_rx) { - int ret; - xsk_socket__delete(ifobj_tx->xsk->xsk); xsk_socket__delete(ifobj_rx->xsk->xsk); ifobj_tx->xsk = &ifobj_tx->xsk_arr[1]; ifobj_rx->xsk = &ifobj_rx->xsk_arr[1]; - ret = xsk_socket__update_xskmap(ifobj_rx->xsk->xsk, ifobj_rx->xsk_map_fd); - if (ret) - exit_with_error(-ret); + bpf_update_xsk_map(ifobj_tx, 0); } static void testapp_bpf_res(struct test_spec *test) @@ -1635,6 +1687,8 @@ static bool testapp_unaligned(struct test_spec *test) { if (!hugepages_present(test->ifobj_tx)) { ksft_test_result_skip("No 2M huge pages present.\n"); + bpf_detach(test->ifobj_tx); + bpf_detach(test->ifobj_rx); return false; } @@ -1947,10 +2001,16 @@ int main(int argc, char **argv) for (i = 0; i < modes; i++) for (j = 0; j < TEST_TYPE_MAX; j++) { + if (j != TEST_TYPE_RUN_TO_COMPLETION_SINGLE_PKT) continue; // XXX + if (i != TEST_MODE_DRV) continue; // XXX + test_spec_init(&test, ifobj_tx, ifobj_rx, i); run_pkt_test(&test, i, j); usleep(USLEEP_MAX); + xskxceiver__destroy(ifobj_tx->bpf_obj); + xskxceiver__destroy(ifobj_rx->bpf_obj); + if (test.fail) failed_tests++; } diff --git a/tools/testing/selftests/bpf/xskxceiver.h b/tools/testing/selftests/bpf/xskxceiver.h index edb76d2def9f..c27dcbdb030f 100644 --- a/tools/testing/selftests/bpf/xskxceiver.h +++ b/tools/testing/selftests/bpf/xskxceiver.h @@ -5,6 +5,8 @@ #ifndef XSKXCEIVER_H_ #define XSKXCEIVER_H_ +#include "xskxceiver.skel.h" + #ifndef SOL_XDP #define SOL_XDP 283 #endif @@ -134,6 +136,7 @@ typedef void *(*thread_func_t)(void *arg); struct ifobject { char ifname[MAX_INTERFACE_NAME_CHARS]; char nsname[MAX_INTERFACES_NAMESPACE_CHARS]; + struct xskxceiver *bpf_obj; struct xsk_socket_info *xsk; struct xsk_socket_info *xsk_arr; struct xsk_umem_info *umem; @@ -141,7 +144,7 @@ struct ifobject { validation_func_t validation_func; struct pkt_stream *pkt_stream; int ns_fd; - int xsk_map_fd; + int ifindex; u32 dst_ip; u32 src_ip; u32 xdp_flags; From patchwork Thu Oct 27 20:00:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stanislav Fomichev X-Patchwork-Id: 13022677 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FF16FA3743 for ; Thu, 27 Oct 2022 20:01:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236808AbiJ0UBU (ORCPT ); Thu, 27 Oct 2022 16:01:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35350 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236832AbiJ0UAf (ORCPT ); Thu, 27 Oct 2022 16:00:35 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CD9ED4BD39 for ; Thu, 27 Oct 2022 13:00:31 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id pa16-20020a17090b265000b0020a71040b4cso1333447pjb.6 for ; Thu, 27 Oct 2022 13:00:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=RjiCuwRFNlnGD8cnL4JM+H69C0y0ToQ/JKpjmpN3Aww=; b=jGx0jWs2Us6GuLLv8OMPFZo7n/PLLGK+RS0ZV9NHlxhQBnPqEaZIA9yny72qQpnfs7 JCsUSvAGkqy6csSiTBDbUEOoxXR6o5VHlrxAwYufPXhV5/wiFx2K0Xc3zDkLYbn2XGmk E6CZsWMAcjDmZip6+OU8Dsz+akRInE1cyGzwzX8ng4TdrkTymJ1ezTDxnsRd7mLTwbFY 3xX/HPdGQvnVStHGRwIF8rU3BBdt5JdkKdhDaru8R5HCtHepzlHAFXo3gKhQM5PV04ak Khj/Wx1Isy9+6KRqDxKWFJz4A73xF2TXYQwVjq5zluD7i68b/6LTGTxZ+BPqDlRFFICH Z80w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=RjiCuwRFNlnGD8cnL4JM+H69C0y0ToQ/JKpjmpN3Aww=; b=cJd05fE/ffCut9StobaVBiUhEnUIXae9XNj4phSLqMr7wdSfYUxeXP3hEYLsvOrCN8 INvdiNFJfY8oKujOlB81JuJtfBV5pmcKtbhS1JP/O/Y3amnDEmZNyIlItN9QkY2g0niR oWRAIgBKlhPoWzL0GotXdAzbCFTRLe8VYEbnuY4OKPIlgwnaHOdL65t1kSRU0ZqbEzAP J6GipSsZ8us70sz5AMHORiHTom7KktCbagpvjkVnG4oncdAV3oxGwKOL92mvvFCTLid6 fjD2UbeSQ84k/RQ9QLkF65/Zzd3jIUqZVYailPK+ARcrs9u4DnQPkn0w/mR1M5HqO75C bVFA== X-Gm-Message-State: ACrzQf0u+jd8EFuIVkAvVWaJ6yxq+5wYQxEuTp9oZ9imdVuwHCkDfinu 4DXjaQoH/O77KDEzFMQ5MgTU9Nc= X-Google-Smtp-Source: AMsMyM6KG2YeGD9M6kECN/tEJXXjlRxeLzwKqwkdyna5gTM1eaGl/2VGJS2wxfP9b/OFY45/cystJms= X-Received: from sdf.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5935]) (user=sdf job=sendgmr) by 2002:a05:6a00:1687:b0:565:a932:f05a with SMTP id k7-20020a056a00168700b00565a932f05amr50614703pfc.21.1666900830534; Thu, 27 Oct 2022 13:00:30 -0700 (PDT) Date: Thu, 27 Oct 2022 13:00:19 -0700 In-Reply-To: <20221027200019.4106375-1-sdf@google.com> Mime-Version: 1.0 References: <20221027200019.4106375-1-sdf@google.com> X-Mailer: git-send-email 2.38.1.273.g43a17bfeac-goog Message-ID: <20221027200019.4106375-6-sdf@google.com> Subject: [RFC bpf-next 5/5] selftests/bpf: Test rx_timestamp metadata in xskxceiver From: Stanislav Fomichev To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, Jakub Kicinski , Willem de Bruijn , Jesper Dangaard Brouer , Anatoly Burakov , Alexander Lobakin , Magnus Karlsson , Maryam Tahhan , xdp-hints@xdp-project.net, netdev@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Example on how the metadata is prepared from the BPF context and consumed by AF_XDP: - bpf_xdp_metadata_have_rx_timestamp to test whether it's supported; if not, I'm assuming verifier will remove this "if (0)" branch - bpf_xdp_metadata_rx_timestamp returns a _copy_ of metadata; the program has to bpf_xdp_adjust_meta+memcpy it; maybe returning a pointer is better? - af_xdp consumer grabs it from data- and makes sure timestamp is not empty - when loading the program, we pass BPF_F_XDP_HAS_METADATA+prog_ifindex Cc: Martin KaFai Lau Cc: Jakub Kicinski Cc: Willem de Bruijn Cc: Jesper Dangaard Brouer Cc: Anatoly Burakov Cc: Alexander Lobakin Cc: Magnus Karlsson Cc: Maryam Tahhan Cc: xdp-hints@xdp-project.net Cc: netdev@vger.kernel.org Signed-off-by: Stanislav Fomichev --- .../testing/selftests/bpf/progs/xskxceiver.c | 22 ++++++++++++++++++ tools/testing/selftests/bpf/xskxceiver.c | 23 ++++++++++++++++++- 2 files changed, 44 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/progs/xskxceiver.c b/tools/testing/selftests/bpf/progs/xskxceiver.c index b135daddad3a..83c879aa3581 100644 --- a/tools/testing/selftests/bpf/progs/xskxceiver.c +++ b/tools/testing/selftests/bpf/progs/xskxceiver.c @@ -12,9 +12,31 @@ struct { __type(value, __u32); } xsk SEC(".maps"); +extern int bpf_xdp_metadata_have_rx_timestamp(struct xdp_md *ctx) __ksym; +extern __u32 bpf_xdp_metadata_rx_timestamp(struct xdp_md *ctx) __ksym; + SEC("xdp") int rx(struct xdp_md *ctx) { + void *data, *data_meta; + __u32 rx_timestamp; + int ret; + + if (bpf_xdp_metadata_have_rx_timestamp(ctx)) { + ret = bpf_xdp_adjust_meta(ctx, -(int)sizeof(__u32)); + if (ret != 0) + return XDP_DROP; + + data = (void *)(long)ctx->data; + data_meta = (void *)(long)ctx->data_meta; + + if (data_meta + sizeof(__u32) > data) + return XDP_DROP; + + rx_timestamp = bpf_xdp_metadata_rx_timestamp(ctx); + __builtin_memcpy(data_meta, &rx_timestamp, sizeof(__u32)); + } + return bpf_redirect_map(&xsk, ctx->rx_queue_index, XDP_PASS); } diff --git a/tools/testing/selftests/bpf/xskxceiver.c b/tools/testing/selftests/bpf/xskxceiver.c index 066bd691db13..ce82c89a432e 100644 --- a/tools/testing/selftests/bpf/xskxceiver.c +++ b/tools/testing/selftests/bpf/xskxceiver.c @@ -871,7 +871,9 @@ static bool is_offset_correct(struct xsk_umem_info *umem, struct pkt_stream *pkt static bool is_pkt_valid(struct pkt *pkt, void *buffer, u64 addr, u32 len) { void *data = xsk_umem__get_data(buffer, addr); + void *data_meta = data - sizeof(__u32); struct iphdr *iphdr = (struct iphdr *)(data + sizeof(struct ethhdr)); + __u32 rx_timestamp = 0; if (!pkt) { ksft_print_msg("[%s] too many packets received\n", __func__); @@ -907,6 +909,13 @@ static bool is_pkt_valid(struct pkt *pkt, void *buffer, u64 addr, u32 len) return false; } + memcpy(&rx_timestamp, data_meta, sizeof(rx_timestamp)); + if (rx_timestamp == 0) { + ksft_print_msg("Invalid metadata received: "); + ksft_print_msg("got %08x, expected != 0\n", rx_timestamp); + return false; + } + return true; } @@ -1331,6 +1340,7 @@ static void thread_common_ops(struct test_spec *test, struct ifobject *ifobject) u64 umem_sz = ifobject->umem->num_frames * ifobject->umem->frame_size; int mmap_flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE; LIBBPF_OPTS(bpf_xdp_query_opts, opts); + LIBBPF_OPTS(bpf_object_open_opts, open_opts); int ret, ifindex; void *bufs; @@ -1340,10 +1350,21 @@ static void thread_common_ops(struct test_spec *test, struct ifobject *ifobject) if (!ifindex) exit_with_error(errno); - ifobject->bpf_obj = xskxceiver__open_and_load(); + open_opts.prog_ifindex = ifindex; + + ifobject->bpf_obj = xskxceiver__open_opts(&open_opts); if (libbpf_get_error(ifobject->bpf_obj)) exit_with_error(libbpf_get_error(ifobject->bpf_obj)); + ret = bpf_program__set_flags(bpf_object__find_program_by_name(ifobject->bpf_obj->obj, "rx"), + BPF_F_XDP_HAS_METADATA); + if (ret < 0) + exit_with_error(ret); + + ret = xskxceiver__load(ifobject->bpf_obj); + if (ret < 0) + exit_with_error(ret); + if (ifobject->umem->unaligned_mode) mmap_flags |= MAP_HUGETLB;