From patchwork Tue Jun 22 20:28:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 12338559 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C65D8C2B9F4 for ; Tue, 22 Jun 2021 20:30:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id ACFA961370 for ; Tue, 22 Jun 2021 20:30:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229995AbhFVUdO (ORCPT ); Tue, 22 Jun 2021 16:33:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44056 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229667AbhFVUdN (ORCPT ); Tue, 22 Jun 2021 16:33:13 -0400 Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 016A6C061574; Tue, 22 Jun 2021 13:30:57 -0700 (PDT) Received: by mail-pf1-x441.google.com with SMTP id w71so492676pfd.4; Tue, 22 Jun 2021 13:30:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LtOupg8zZLOGLS8qU72uV1cLOLhIh5uV+DMN2/xgYL4=; b=hHsRQnvVHV4DSWMUybuxzvxWkZ5vOYMrjgN0nSDMwgbWBGEED9lcwdT15Pe3KWwhkP 2cP6Dv1N5eMhZA1S3LWDjW7DJTrYgwrolkBuQF5sq1LsoKbediYcxopObA3CNFY8G5zH uX+SueKdK6ZGPYUcO69Kx13uk88vhpD4vHefbTpU5fM8/LN4G9b/n/L4dRwKCHwyFtD7 VA61U2vBz5mXlErKHbQIPgu6ogOREf0EB46SRp8HSDI4ZOKtE8Mcq4ytZPOtnQgr+axq 93GY7vFHFq6cBfGQp2Rj4i/i5r5hejqmrymHlmZQvwviVvXBWlRGxDEq2YDDKvpE6a0o bBaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LtOupg8zZLOGLS8qU72uV1cLOLhIh5uV+DMN2/xgYL4=; b=uFDsZ3Ird3FSBXbYKwBHDcitQrfWKaVUvkMsy5UjcSNg1NKBWrLUk06gJsEa2dlY9E 35f/r/FPSsrOP86XbxrNPyigbWO4/UWhO/GQfStxow3vFzfGSJQ+ZYrgXPouVVkcHI/l fdCfiK21sDRt86KXrkSnVKFzem4sMk5uIIeurhVQKFjUha3E0OG5g4g85dheUnn/d5i7 3MWPcJrszc1kz9107Kv1PCaGrpVFirlPNdnMTifhpMA1qXnQNTbavN8Hl9iXQId1PfIe iW4Y8qxChXwIqGua00osiujJwQF0Mv9xIPEzykuCVqxamlPfEfXxNYB76yAYObs++Utm hVqA== X-Gm-Message-State: AOAM533l6wkgmK2o/g3f5/gdjLA1M43Jlz4cDnAU8XnsGbSBSt8SoTNS M+7Ze85hieKZAbNA9IuV74QYr2Oln5U= X-Google-Smtp-Source: ABdhPJzKEIFLukD/fT2Ehw00xlGN31jrXNzpFj/NtkK2uVEFEjHOgxacx6xpeNkRIaHWukGRdWi5PA== X-Received: by 2002:a63:ff09:: with SMTP id k9mr423727pgi.113.1624393856386; Tue, 22 Jun 2021 13:30:56 -0700 (PDT) Received: from localhost ([2402:3a80:11bb:33b3:7f0c:3646:8bde:417e]) by smtp.gmail.com with ESMTPSA id z204sm189082pfc.84.2021.06.22.13.30.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Jun 2021 13:30:56 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: netdev@vger.kernel.org Cc: Kumar Kartikeya Dwivedi , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rge?= =?utf-8?q?nsen?= , Jesper Dangaard Brouer , "David S. Miller" , Jakub Kicinski , John Fastabend , Martin KaFai Lau , bpf@vger.kernel.org Subject: [PATCH net-next v3 1/5] net: core: split out code to run generic XDP prog Date: Wed, 23 Jun 2021 01:58:31 +0530 Message-Id: <20210622202835.1151230-2-memxor@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210622202835.1151230-1-memxor@gmail.com> References: <20210622202835.1151230-1-memxor@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org This helper can later be utilized in code that runs cpumap and devmap programs in generic redirect mode and adjust skb based on changes made to xdp_buff. When returning XDP_REDIRECT/XDP_TX, it invokes __skb_push, so whenever a generic redirect path invokes devmap/cpumap prog if set, it must __skb_pull again as we expect mac header to be pulled. It also drops the skb_reset_mac_len call after do_xdp_generic, as the mac_header and network_header are advanced by the same offset, so the difference (mac_len) remains constant. Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/netdevice.h | 2 + net/core/dev.c | 84 ++++++++++++++++++++++++--------------- 2 files changed, 55 insertions(+), 31 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index be1dcceda5e4..90472ea70db2 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -3984,6 +3984,8 @@ static inline void dev_consume_skb_any(struct sk_buff *skb) __dev_kfree_skb_any(skb, SKB_REASON_CONSUMED); } +u32 bpf_prog_run_generic_xdp(struct sk_buff *skb, struct xdp_buff *xdp, + struct bpf_prog *xdp_prog); void generic_xdp_tx(struct sk_buff *skb, struct bpf_prog *xdp_prog); int do_xdp_generic(struct bpf_prog *xdp_prog, struct sk_buff *skb); int netif_rx(struct sk_buff *skb); diff --git a/net/core/dev.c b/net/core/dev.c index 50531a2d0b20..c34ff1dbf6e6 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4717,45 +4717,18 @@ static struct netdev_rx_queue *netif_get_rxqueue(struct sk_buff *skb) return rxqueue; } -static u32 netif_receive_generic_xdp(struct sk_buff *skb, - struct xdp_buff *xdp, - struct bpf_prog *xdp_prog) +u32 bpf_prog_run_generic_xdp(struct sk_buff *skb, struct xdp_buff *xdp, + struct bpf_prog *xdp_prog) { void *orig_data, *orig_data_end, *hard_start; struct netdev_rx_queue *rxqueue; - u32 metalen, act = XDP_DROP; bool orig_bcast, orig_host; u32 mac_len, frame_sz; __be16 orig_eth_type; struct ethhdr *eth; + u32 metalen, act; int off; - /* Reinjected packets coming from act_mirred or similar should - * not get XDP generic processing. - */ - if (skb_is_redirected(skb)) - return XDP_PASS; - - /* XDP packets must be linear and must have sufficient headroom - * of XDP_PACKET_HEADROOM bytes. This is the guarantee that also - * native XDP provides, thus we need to do it here as well. - */ - if (skb_cloned(skb) || skb_is_nonlinear(skb) || - skb_headroom(skb) < XDP_PACKET_HEADROOM) { - int hroom = XDP_PACKET_HEADROOM - skb_headroom(skb); - int troom = skb->tail + skb->data_len - skb->end; - - /* In case we have to go down the path and also linearize, - * then lets do the pskb_expand_head() work just once here. - */ - if (pskb_expand_head(skb, - hroom > 0 ? ALIGN(hroom, NET_SKB_PAD) : 0, - troom > 0 ? troom + 128 : 0, GFP_ATOMIC)) - goto do_drop; - if (skb_linearize(skb)) - goto do_drop; - } - /* The XDP program wants to see the packet starting at the MAC * header. */ @@ -4810,6 +4783,13 @@ static u32 netif_receive_generic_xdp(struct sk_buff *skb, skb->protocol = eth_type_trans(skb, skb->dev); } + /* Redirect/Tx gives L2 packet, code that will reuse skb must __skb_pull + * before calling us again on redirect path. We do not call do_redirect + * as we leave that up to the caller. + * + * Caller is responsible for managing lifetime of skb (i.e. calling + * kfree_skb in response to actions it cannot handle/XDP_DROP). + */ switch (act) { case XDP_REDIRECT: case XDP_TX: @@ -4820,6 +4800,49 @@ static u32 netif_receive_generic_xdp(struct sk_buff *skb, if (metalen) skb_metadata_set(skb, metalen); break; + } + + return act; +} + +static u32 netif_receive_generic_xdp(struct sk_buff *skb, + struct xdp_buff *xdp, + struct bpf_prog *xdp_prog) +{ + u32 act = XDP_DROP; + + /* Reinjected packets coming from act_mirred or similar should + * not get XDP generic processing. + */ + if (skb_is_redirected(skb)) + return XDP_PASS; + + /* XDP packets must be linear and must have sufficient headroom + * of XDP_PACKET_HEADROOM bytes. This is the guarantee that also + * native XDP provides, thus we need to do it here as well. + */ + if (skb_cloned(skb) || skb_is_nonlinear(skb) || + skb_headroom(skb) < XDP_PACKET_HEADROOM) { + int hroom = XDP_PACKET_HEADROOM - skb_headroom(skb); + int troom = skb->tail + skb->data_len - skb->end; + + /* In case we have to go down the path and also linearize, + * then lets do the pskb_expand_head() work just once here. + */ + if (pskb_expand_head(skb, + hroom > 0 ? ALIGN(hroom, NET_SKB_PAD) : 0, + troom > 0 ? troom + 128 : 0, GFP_ATOMIC)) + goto do_drop; + if (skb_linearize(skb)) + goto do_drop; + } + + act = bpf_prog_run_generic_xdp(skb, xdp, xdp_prog); + switch (act) { + case XDP_REDIRECT: + case XDP_TX: + case XDP_PASS: + break; default: bpf_warn_invalid_xdp_action(act); fallthrough; @@ -5285,7 +5308,6 @@ static int __netif_receive_skb_core(struct sk_buff **pskb, bool pfmemalloc, ret = NET_RX_DROP; goto out; } - skb_reset_mac_len(skb); } if (eth_type_vlan(skb->protocol)) { From patchwork Tue Jun 22 20:28:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 12338561 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D168BC2B9F4 for ; Tue, 22 Jun 2021 20:31:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B58C760240 for ; Tue, 22 Jun 2021 20:31:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230018AbhFVUdS (ORCPT ); Tue, 22 Jun 2021 16:33:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44078 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230040AbhFVUdR (ORCPT ); Tue, 22 Jun 2021 16:33:17 -0400 Received: from mail-pf1-x444.google.com (mail-pf1-x444.google.com [IPv6:2607:f8b0:4864:20::444]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8C4EAC061574; Tue, 22 Jun 2021 13:31:00 -0700 (PDT) Received: by mail-pf1-x444.google.com with SMTP id u18so462348pfk.11; Tue, 22 Jun 2021 13:31:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jivxYGcJYbE7eSkXipGR1sWBMuZY0V3FzEQSe7d96iA=; b=AhxZBywa1Rb8izZOPZJBN6vS7pvo0FqkGRFYp2BU+TV8HwB7NB2hvHsu5WyA2Pz9tP s2Ugk0d2ooebU0aMpYbDUbrV/V9INpAY5qtQgQ0wzvNwwNpiRepkx/8Lqpq8dgt3Yoch 999FNohKjOTyVyap/83nrCLhqtDuoewbTOl9UzLz8YhaJ+KgKYAbBp1djp7kazjgBJGm 6xb/zl2ZaFLyq7+IKmIrZ3ou6zXW8zfiYp0O6J0FDAyN2F2qKfNlwySA0z9PurnzaHoD OcUEFUiK6dQ1cH77EzLMZaRESzJyMt36XZ0WqSANmzty8xZiooHI2F7a9Fefy4jxQHUQ D6bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jivxYGcJYbE7eSkXipGR1sWBMuZY0V3FzEQSe7d96iA=; b=rRPKk+ACSqmGb4+A7YHTj9CDr9akheDHtoqogrJLP2v88fi4ibJBb/ZsCORv9PLwMv kYt4TPxDhZCUFpJqBVes4OMvtga3+BUGfP7J8MjKabhJ4jFuQG2xe2Q0LJNuRtXy9sDz iaeHejhBo9mD1z0HmOCaeERUdrsju84GT7H6ZFZCd5KlJ3D6zIXzp+Lto2VFThd6m+8Q bMO0WAifDaRfiqEQuzrWqDUXJbB3rH/mGsH5tfTdQmLVWoAff7vunXEq/om6mWM7rF0j EGOnv+fZW87+Lyqw8G622de9EK3eWrgjSXhGGBNfVTmBN4w7LdMZ/tZ4lrw9FXFBfLEC xE1A== X-Gm-Message-State: AOAM533cBeU3aEYpILfcCjxmDwbj+QKFnwkaevo+Jmz7C95LdteuFUs1 TItcC3O5ijkDq8SRGkzJEw6wEKql6hM= X-Google-Smtp-Source: ABdhPJwmFFhzJrl5upL+F5lt+MzsNok+YMDyQai0YHQPJj9eTlqcmmZaXaY4wQluwcGezhf/ObeXjw== X-Received: by 2002:a65:6243:: with SMTP id q3mr414908pgv.297.1624393859999; Tue, 22 Jun 2021 13:30:59 -0700 (PDT) Received: from localhost ([2402:3a80:11bb:33b3:7f0c:3646:8bde:417e]) by smtp.gmail.com with ESMTPSA id e13sm188164pfd.8.2021.06.22.13.30.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Jun 2021 13:30:59 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: netdev@vger.kernel.org Cc: Kumar Kartikeya Dwivedi , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rge?= =?utf-8?q?nsen?= , Jesper Dangaard Brouer , "David S. Miller" , Jakub Kicinski , John Fastabend , Martin KaFai Lau , bpf@vger.kernel.org Subject: [PATCH net-next v3 2/5] bitops: add non-atomic bitops for pointers Date: Wed, 23 Jun 2021 01:58:32 +0530 Message-Id: <20210622202835.1151230-3-memxor@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210622202835.1151230-1-memxor@gmail.com> References: <20210622202835.1151230-1-memxor@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org cpumap needs to set, clear, and test the lowest bit in skb pointer in various places. To make these checks less noisy, add pointer friendly bitop macros that also do some typechecking to sanitize the argument. These wrap the non-atomic bitops __set_bit, __clear_bit, and test_bit but for pointer arguments. Pointer's address has to be passed in and it is treated as an unsigned long *, since width and representation of pointer and unsigned long match on targets Linux supports. They are prefixed with double underscore to indicate lack of atomicity. Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bitops.h | 19 +++++++++++++++++++ include/linux/typecheck.h | 10 ++++++++++ 2 files changed, 29 insertions(+) diff --git a/include/linux/bitops.h b/include/linux/bitops.h index 26bf15e6cd35..a9e336b9fa4d 100644 --- a/include/linux/bitops.h +++ b/include/linux/bitops.h @@ -4,6 +4,7 @@ #include #include +#include #include @@ -253,6 +254,24 @@ static __always_inline void __assign_bit(long nr, volatile unsigned long *addr, __clear_bit(nr, addr); } +#define __ptr_set_bit(nr, addr) \ + ({ \ + typecheck_pointer(*(addr)); \ + __set_bit(nr, (unsigned long *)(addr)); \ + }) + +#define __ptr_clear_bit(nr, addr) \ + ({ \ + typecheck_pointer(*(addr)); \ + __clear_bit(nr, (unsigned long *)(addr)); \ + }) + +#define __ptr_test_bit(nr, addr) \ + ({ \ + typecheck_pointer(*(addr)); \ + test_bit(nr, (unsigned long *)(addr)); \ + }) + #ifdef __KERNEL__ #ifndef set_mask_bits diff --git a/include/linux/typecheck.h b/include/linux/typecheck.h index 20d310331eb5..33c78f27147a 100644 --- a/include/linux/typecheck.h +++ b/include/linux/typecheck.h @@ -22,4 +22,14 @@ (void)__tmp; \ }) +/* + * Check at compile that something is a pointer type. + * Always evaluates to 1 so you may use it easily in comparisons. + */ +#define typecheck_pointer(x) \ +({ typeof(x) __dummy; \ + (void)sizeof(*__dummy); \ + 1; \ +}) + #endif /* TYPECHECK_H_INCLUDED */ From patchwork Tue Jun 22 20:28:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 12338563 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CE86C2B9F4 for ; Tue, 22 Jun 2021 20:31:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 01DDD61370 for ; Tue, 22 Jun 2021 20:31:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230054AbhFVUdW (ORCPT ); Tue, 22 Jun 2021 16:33:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44100 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229667AbhFVUdU (ORCPT ); Tue, 22 Jun 2021 16:33:20 -0400 Received: from mail-pg1-x541.google.com (mail-pg1-x541.google.com [IPv6:2607:f8b0:4864:20::541]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 51194C061574; Tue, 22 Jun 2021 13:31:04 -0700 (PDT) Received: by mail-pg1-x541.google.com with SMTP id y14so6615002pgs.12; Tue, 22 Jun 2021 13:31:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=CAGsheAv5JPBQrwcoYISVybTGgYXUV7KF4ogKgpayZE=; b=bYnUF7HM+lJb2Hht4nuqRoCLsxGmyZnOY6jB4GgnzP+6GIYrLyHO0umfVjBODGuK9v W1fcaVTJdIme1gm6JJaIjp5GrkIR2z8UJ8C1VVEa/lZyOUa5oUc/V0Atwm5q0nTWsaeP DrXiQi4zbz8H1SHRING3HLsppHGbqDpjQ35Y4rMzPkHq9lKwWOztqe7ch8E1fQTlBeRb agUzMvLlDGD+3uZjZ9ChaCmUNijfLOvSC+EJm94i4f5HO5AEVFm9CDL8ooOquB0OaMiB cOVyQKUWEG7SyIkV8iMdoUbhORHgnaAFg97yJe9ceNGbzecIlAKKEOUr0Hzgl0hWTGSx WWlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CAGsheAv5JPBQrwcoYISVybTGgYXUV7KF4ogKgpayZE=; b=gPa8TAyZRNdbrPcHkM4UoH4P4s1sLMJrVKnbnevrZQxl7VBOW1jNfjTSuI760LWNZH H7r3pq5iTiYNgGkrSYNqWK5kLFOhI2xG8K6SsUUQATqZGoTzzqbGUGO36VsA62xuMveT oFzp0RgDuecGJQkN0+9v/8o7ih3mIDlVHEQBsWJPNrP4YoIBzJWfIRE0Uz9yyxc0CXnU Mk3GdMda/TSkZXLNwWJ+7/NDTOkUmz4X4zV6/FKels+z0jsuGk0O3/bdj5BFyBMOEEtC 1iI2SLjiZHPkuLxvP679lRMjic3zrjhKOwy1wPI1/aHnjoGM6cgaNEiynwdohwan8drL ACwA== X-Gm-Message-State: AOAM533gRgNlFgn1sVMDclUrWYjiQnEQbbmkrqino+nJPCGLTrcOxXwy QPUWfse0xLslvpbeDDJFrf/bCaU+Mww= X-Google-Smtp-Source: ABdhPJwQPNNIgOcoHN96K/QVkN4jnVydQV3lPPTmM5IKr6ZoR3WbNLLi6Lpqu864Y4HlU+tY3VsWtQ== X-Received: by 2002:a63:4555:: with SMTP id u21mr410915pgk.333.1624393863670; Tue, 22 Jun 2021 13:31:03 -0700 (PDT) Received: from localhost ([2402:3a80:11bb:33b3:7f0c:3646:8bde:417e]) by smtp.gmail.com with ESMTPSA id x20sm186194pfh.112.2021.06.22.13.31.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Jun 2021 13:31:03 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: netdev@vger.kernel.org Cc: Kumar Kartikeya Dwivedi , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rge?= =?utf-8?q?nsen?= , Jesper Dangaard Brouer , "David S. Miller" , Jakub Kicinski , John Fastabend , Martin KaFai Lau , bpf@vger.kernel.org Subject: [PATCH net-next v3 3/5] bpf: cpumap: implement generic cpumap Date: Wed, 23 Jun 2021 01:58:33 +0530 Message-Id: <20210622202835.1151230-4-memxor@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210622202835.1151230-1-memxor@gmail.com> References: <20210622202835.1151230-1-memxor@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org This change implements CPUMAP redirect support for generic XDP programs. The idea is to reuse the cpu map entry's queue that is used to push native xdp frames for redirecting skb to a different CPU. This will match native XDP behavior (in that RPS is invoked again for packet reinjected into networking stack). To be able to determine whether the incoming skb is from the driver or cpumap, we reuse skb->redirected bit that skips generic XDP processing when it is set. To always make use of this, CONFIG_NET_REDIRECT guard on it has been lifted and it is always available. From the redirect side, we add the skb to ptr_ring with its lowest bit set to 1. This should be safe as skb is not 1-byte aligned. This allows kthread to discern between xdp_frames and sk_buff. On consumption of the ptr_ring item, the lowest bit is unset. In the end, the skb is simply added to the list that kthread is anyway going to maintain for xdp_frames converted to skb, and then received again by using netif_receive_skb_list. Bulking optimization for generic cpumap is left as an exercise for a future patch for now. Since cpumap entry progs are now supported, also remove check in generic_xdp_install for the cpumap. Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf.h | 9 +++- include/linux/skbuff.h | 10 +--- kernel/bpf/cpumap.c | 115 +++++++++++++++++++++++++++++++++++------ net/core/dev.c | 3 +- net/core/filter.c | 6 ++- 5 files changed, 115 insertions(+), 28 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index f309fc1509f2..095aaa104c56 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1513,7 +1513,8 @@ bool dev_map_can_have_prog(struct bpf_map *map); void __cpu_map_flush(void); int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_buff *xdp, struct net_device *dev_rx); -bool cpu_map_prog_allowed(struct bpf_map *map); +int cpu_map_generic_redirect(struct bpf_cpu_map_entry *rcpu, + struct sk_buff *skb); /* Return map's numa specified by userspace */ static inline int bpf_map_attr_numa_node(const union bpf_attr *attr) @@ -1710,6 +1711,12 @@ static inline int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, return 0; } +static inline int cpu_map_generic_redirect(struct bpf_cpu_map_entry *rcpu, + struct sk_buff *skb) +{ + return -EOPNOTSUPP; +} + static inline bool cpu_map_prog_allowed(struct bpf_map *map) { return false; diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index b2db9cd9a73f..f19190820e63 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -863,8 +863,8 @@ struct sk_buff { __u8 tc_skip_classify:1; __u8 tc_at_ingress:1; #endif -#ifdef CONFIG_NET_REDIRECT __u8 redirected:1; +#ifdef CONFIG_NET_REDIRECT __u8 from_ingress:1; #endif #ifdef CONFIG_TLS_DEVICE @@ -4664,17 +4664,13 @@ static inline __wsum lco_csum(struct sk_buff *skb) static inline bool skb_is_redirected(const struct sk_buff *skb) { -#ifdef CONFIG_NET_REDIRECT return skb->redirected; -#else - return false; -#endif } static inline void skb_set_redirected(struct sk_buff *skb, bool from_ingress) { -#ifdef CONFIG_NET_REDIRECT skb->redirected = 1; +#ifdef CONFIG_NET_REDIRECT skb->from_ingress = from_ingress; if (skb->from_ingress) skb->tstamp = 0; @@ -4683,9 +4679,7 @@ static inline void skb_set_redirected(struct sk_buff *skb, bool from_ingress) static inline void skb_reset_redirect(struct sk_buff *skb) { -#ifdef CONFIG_NET_REDIRECT skb->redirected = 0; -#endif } static inline bool skb_csum_is_sctp(struct sk_buff *skb) diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c index a1a0c4e791c6..274353e2cd70 100644 --- a/kernel/bpf/cpumap.c +++ b/kernel/bpf/cpumap.c @@ -16,6 +16,7 @@ * netstack, and assigning dedicated CPUs for this stage. This * basically allows for 10G wirespeed pre-filtering via bpf. */ +#include #include #include #include @@ -168,6 +169,49 @@ static void put_cpu_map_entry(struct bpf_cpu_map_entry *rcpu) } } +static void cpu_map_bpf_prog_run_skb(struct bpf_cpu_map_entry *rcpu, + struct list_head *listp, + struct xdp_cpumap_stats *stats) +{ + struct sk_buff *skb, *tmp; + struct xdp_buff xdp; + u32 act; + int err; + + if (!rcpu->prog) + return; + + list_for_each_entry_safe(skb, tmp, listp, list) { + act = bpf_prog_run_generic_xdp(skb, &xdp, rcpu->prog); + switch (act) { + case XDP_PASS: + break; + case XDP_REDIRECT: + skb_list_del_init(skb); + err = xdp_do_generic_redirect(skb->dev, skb, &xdp, + rcpu->prog); + if (unlikely(err)) { + kfree_skb(skb); + stats->drop++; + } else { + stats->redirect++; + } + return; + default: + bpf_warn_invalid_xdp_action(act); + fallthrough; + case XDP_ABORTED: + trace_xdp_exception(skb->dev, rcpu->prog, act); + fallthrough; + case XDP_DROP: + skb_list_del_init(skb); + kfree_skb(skb); + stats->drop++; + return; + } + } +} + static int cpu_map_bpf_prog_run_xdp(struct bpf_cpu_map_entry *rcpu, void **frames, int n, struct xdp_cpumap_stats *stats) @@ -179,8 +223,6 @@ static int cpu_map_bpf_prog_run_xdp(struct bpf_cpu_map_entry *rcpu, if (!rcpu->prog) return n; - rcu_read_lock_bh(); - xdp_set_return_frame_no_direct(); xdp.rxq = &rxq; @@ -227,17 +269,34 @@ static int cpu_map_bpf_prog_run_xdp(struct bpf_cpu_map_entry *rcpu, } } + xdp_clear_return_frame_no_direct(); + + return nframes; +} + +#define CPUMAP_BATCH 8 + +static int cpu_map_bpf_prog_run(struct bpf_cpu_map_entry *rcpu, void **frames, + int xdp_n, struct xdp_cpumap_stats *stats, + struct list_head *list) +{ + int nframes; + + rcu_read_lock_bh(); + + nframes = cpu_map_bpf_prog_run_xdp(rcpu, frames, xdp_n, stats); + if (stats->redirect) - xdp_do_flush_map(); + xdp_do_flush(); - xdp_clear_return_frame_no_direct(); + if (unlikely(!list_empty(list))) + cpu_map_bpf_prog_run_skb(rcpu, list, stats); - rcu_read_unlock_bh(); /* resched point, may call do_softirq() */ + rcu_read_unlock_bh(); return nframes; } -#define CPUMAP_BATCH 8 static int cpu_map_kthread_run(void *data) { @@ -254,9 +313,9 @@ static int cpu_map_kthread_run(void *data) struct xdp_cpumap_stats stats = {}; /* zero stats */ unsigned int kmem_alloc_drops = 0, sched = 0; gfp_t gfp = __GFP_ZERO | GFP_ATOMIC; + int i, n, m, nframes, xdp_n; void *frames[CPUMAP_BATCH]; void *skbs[CPUMAP_BATCH]; - int i, n, m, nframes; LIST_HEAD(list); /* Release CPU reschedule checks */ @@ -280,9 +339,20 @@ static int cpu_map_kthread_run(void *data) */ n = __ptr_ring_consume_batched(rcpu->queue, frames, CPUMAP_BATCH); - for (i = 0; i < n; i++) { + for (i = 0, xdp_n = 0; i < n; i++) { void *f = frames[i]; - struct page *page = virt_to_page(f); + struct page *page; + + if (unlikely(__ptr_test_bit(0, &f))) { + struct sk_buff *skb = f; + + __ptr_clear_bit(0, &skb); + list_add_tail(&skb->list, &list); + continue; + } + + frames[xdp_n++] = f; + page = virt_to_page(f); /* Bring struct page memory area to curr CPU. Read by * build_skb_around via page_is_pfmemalloc(), and when @@ -292,7 +362,7 @@ static int cpu_map_kthread_run(void *data) } /* Support running another XDP prog on this CPU */ - nframes = cpu_map_bpf_prog_run_xdp(rcpu, frames, n, &stats); + nframes = cpu_map_bpf_prog_run(rcpu, frames, xdp_n, &stats, &list); if (nframes) { m = kmem_cache_alloc_bulk(skbuff_head_cache, gfp, nframes, skbs); if (unlikely(m == 0)) { @@ -330,12 +400,6 @@ static int cpu_map_kthread_run(void *data) return 0; } -bool cpu_map_prog_allowed(struct bpf_map *map) -{ - return map->map_type == BPF_MAP_TYPE_CPUMAP && - map->value_size != offsetofend(struct bpf_cpumap_val, qsize); -} - static int __cpu_map_load_bpf_program(struct bpf_cpu_map_entry *rcpu, int fd) { struct bpf_prog *prog; @@ -696,6 +760,25 @@ int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_buff *xdp, return 0; } +int cpu_map_generic_redirect(struct bpf_cpu_map_entry *rcpu, + struct sk_buff *skb) +{ + int ret; + + __skb_pull(skb, skb->mac_len); + skb_set_redirected(skb, false); + __ptr_set_bit(0, &skb); + + ret = ptr_ring_produce(rcpu->queue, skb); + if (ret < 0) + goto trace; + + wake_up_process(rcpu->kthread); +trace: + trace_xdp_cpumap_enqueue(rcpu->map_id, !ret, !!ret, rcpu->cpu); + return ret; +} + void __cpu_map_flush(void) { struct list_head *flush_list = this_cpu_ptr(&cpu_map_flush_list); diff --git a/net/core/dev.c b/net/core/dev.c index c34ff1dbf6e6..a00421e9ee16 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -5642,8 +5642,7 @@ static int generic_xdp_install(struct net_device *dev, struct netdev_bpf *xdp) * have a bpf_prog installed on an entry */ for (i = 0; i < new->aux->used_map_cnt; i++) { - if (dev_map_can_have_prog(new->aux->used_maps[i]) || - cpu_map_prog_allowed(new->aux->used_maps[i])) { + if (dev_map_can_have_prog(new->aux->used_maps[i])) { mutex_unlock(&new->aux->used_maps_mutex); return -EINVAL; } diff --git a/net/core/filter.c b/net/core/filter.c index 0b13d8157a8f..4a21fde3028f 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -4038,8 +4038,12 @@ static int xdp_do_generic_redirect_map(struct net_device *dev, goto err; consume_skb(skb); break; + case BPF_MAP_TYPE_CPUMAP: + err = cpu_map_generic_redirect(fwd, skb); + if (unlikely(err)) + goto err; + break; default: - /* TODO: Handle BPF_MAP_TYPE_CPUMAP */ err = -EBADRQC; goto err; } From patchwork Tue Jun 22 20:28:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 12338565 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8EA4EC2B9F4 for ; Tue, 22 Jun 2021 20:31:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7984861370 for ; Tue, 22 Jun 2021 20:31:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230083AbhFVUdZ (ORCPT ); Tue, 22 Jun 2021 16:33:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44122 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229667AbhFVUdZ (ORCPT ); Tue, 22 Jun 2021 16:33:25 -0400 Received: from mail-pl1-x642.google.com (mail-pl1-x642.google.com [IPv6:2607:f8b0:4864:20::642]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9A16FC061574; Tue, 22 Jun 2021 13:31:08 -0700 (PDT) Received: by mail-pl1-x642.google.com with SMTP id x22so9505190pll.11; Tue, 22 Jun 2021 13:31:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2M+/jke7oI/v1bHKeQ1t1eI5gEmwcggeXPdID1191qU=; b=N0nK/svAuC7AIHjqpiLyovimqrdAVO3vPDc3px0VJ/KbEgf6a9upKvzxQC12D/3DdP 0dekTPFS3gNUVEE6z6hNhKL73CqGMiOOHu7VIp/bE8QYc8wEvxr4HAlJiVIuo+UMlVD8 LlM4PuRnaMc7/9/XrA8nA2mg/rjiU5cHVfbseACTO5gUENC+gSJt17uCv41upqY99n+S F0MfsOcFCXngjDQWNPHoSUaCQnxZtjnHboStCUbr+voODSRD2qtsnkhwwu3LZRbDTa/R c2Q9KA9tOAG6MdS2jmpgwSBEhW3XUKUjr9Ui/r7Vf0l64Ec3a+ABPAnJXpPteByMQffk Flhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2M+/jke7oI/v1bHKeQ1t1eI5gEmwcggeXPdID1191qU=; b=qEep6XXdZ/PcY7TwFzdzxCIKFoYkhnB5bj+b52mp66dZ8h50Vlfao4xtjiMXUIUr9c vlM0YZOIL/UnNtRvUyFROZxZStFcBVtVJbNvE2Bq/hpPFzqJhOkY3gC/1FYF7vmGyy6I Gp7ceX3qw+wuenuNsqAuzDd9DjWn5k0Jd6D80TI9RlmrvUXUlkbUsqmdo0zXm4dsd6K/ EowjyeKCgN3FvhU2bpGyuKEyjdPryq7wqI7NTp2mCwr2oUnxDtim4aDS/WwA2yPZauuV DawwC/vulmvuFakg6lnR+L3d7AUxBOrsrFSfjLHUZ9Awt4G40IUAUR8hRmAlf1U110uC ++6w== X-Gm-Message-State: AOAM5334ylikhovvHmnLHgBOAm43gt7GL4LpfJikN6E37HShqMq1tZ+Y m28ccBOavEJyVQFhjSZBUUYiq62+34U= X-Google-Smtp-Source: ABdhPJw5NmcV1zqNDRZjhyYk34cp+B95MmGclSBdWwyz42jrDkfa/lARWAiiNsD1I/dsU6u0R6Na7Q== X-Received: by 2002:a17:90a:aa93:: with SMTP id l19mr5745716pjq.142.1624393867382; Tue, 22 Jun 2021 13:31:07 -0700 (PDT) Received: from localhost ([2402:3a80:11bb:33b3:7f0c:3646:8bde:417e]) by smtp.gmail.com with ESMTPSA id o8sm192470pfp.48.2021.06.22.13.31.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Jun 2021 13:31:07 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: netdev@vger.kernel.org Cc: Kumar Kartikeya Dwivedi , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rge?= =?utf-8?q?nsen?= , Jesper Dangaard Brouer , "David S. Miller" , Jakub Kicinski , John Fastabend , Martin KaFai Lau , bpf@vger.kernel.org Subject: [PATCH net-next v3 4/5] bpf: devmap: implement devmap prog execution for generic XDP Date: Wed, 23 Jun 2021 01:58:34 +0530 Message-Id: <20210622202835.1151230-5-memxor@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210622202835.1151230-1-memxor@gmail.com> References: <20210622202835.1151230-1-memxor@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org This lifts the restriction on running devmap BPF progs in generic redirect mode. To match native XDP behavior, it is invoked right before generic_xdp_tx is called, and only supports XDP_PASS/XDP_ABORTED/ XDP_DROP actions. We also return 0 even if devmap program drops the packet, as semantically redirect has already succeeded and the devmap prog is the last point before TX of the packet to device where it can deliver a verdict on the packet. This also means it must take care of freeing the skb, as xdp_do_generic_redirect callers only do that in case an error is returned. Since devmap entry prog is supported, remove the check in generic_xdp_install entirely. Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf.h | 1 - kernel/bpf/devmap.c | 49 ++++++++++++++++++++++++++++++++++++--------- net/core/dev.c | 18 ----------------- 3 files changed, 39 insertions(+), 29 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 095aaa104c56..4afbff308ca3 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1508,7 +1508,6 @@ int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb, struct bpf_prog *xdp_prog, struct bpf_map *map, bool exclude_ingress); -bool dev_map_can_have_prog(struct bpf_map *map); void __cpu_map_flush(void); int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_buff *xdp, diff --git a/kernel/bpf/devmap.c b/kernel/bpf/devmap.c index 2a75e6c2d27d..49f03e8e5561 100644 --- a/kernel/bpf/devmap.c +++ b/kernel/bpf/devmap.c @@ -318,16 +318,6 @@ static int dev_map_hash_get_next_key(struct bpf_map *map, void *key, return -ENOENT; } -bool dev_map_can_have_prog(struct bpf_map *map) -{ - if ((map->map_type == BPF_MAP_TYPE_DEVMAP || - map->map_type == BPF_MAP_TYPE_DEVMAP_HASH) && - map->value_size != offsetofend(struct bpf_devmap_val, ifindex)) - return true; - - return false; -} - static int dev_map_bpf_prog_run(struct bpf_prog *xdp_prog, struct xdp_frame **frames, int n, struct net_device *dev) @@ -499,6 +489,37 @@ static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, return 0; } +static u32 dev_map_bpf_prog_run_skb(struct sk_buff *skb, struct bpf_dtab_netdev *dst) +{ + struct xdp_txq_info txq = { .dev = dst->dev }; + struct xdp_buff xdp; + u32 act; + + if (!dst->xdp_prog) + return XDP_PASS; + + __skb_pull(skb, skb->mac_len); + xdp.txq = &txq; + + act = bpf_prog_run_generic_xdp(skb, &xdp, dst->xdp_prog); + switch (act) { + case XDP_PASS: + __skb_push(skb, skb->mac_len); + break; + default: + bpf_warn_invalid_xdp_action(act); + fallthrough; + case XDP_ABORTED: + trace_xdp_exception(dst->dev, dst->xdp_prog, act); + fallthrough; + case XDP_DROP: + kfree_skb(skb); + break; + } + + return act; +} + int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, struct net_device *dev_rx) { @@ -614,6 +635,14 @@ int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, err = xdp_ok_fwd_dev(dst->dev, skb->len); if (unlikely(err)) return err; + + /* Redirect has already succeeded semantically at this point, so we just + * return 0 even if packet is dropped. Helper below takes care of + * freeing skb. + */ + if (dev_map_bpf_prog_run_skb(skb, dst) != XDP_PASS) + return 0; + skb->dev = dst->dev; generic_xdp_tx(skb, xdp_prog); diff --git a/net/core/dev.c b/net/core/dev.c index a00421e9ee16..9d9c78496459 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -5633,24 +5633,6 @@ static int generic_xdp_install(struct net_device *dev, struct netdev_bpf *xdp) struct bpf_prog *new = xdp->prog; int ret = 0; - if (new) { - u32 i; - - mutex_lock(&new->aux->used_maps_mutex); - - /* generic XDP does not work with DEVMAPs that can - * have a bpf_prog installed on an entry - */ - for (i = 0; i < new->aux->used_map_cnt; i++) { - if (dev_map_can_have_prog(new->aux->used_maps[i])) { - mutex_unlock(&new->aux->used_maps_mutex); - return -EINVAL; - } - } - - mutex_unlock(&new->aux->used_maps_mutex); - } - switch (xdp->command) { case XDP_SETUP_PROG: rcu_assign_pointer(dev->xdp_prog, new); From patchwork Tue Jun 22 20:28:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 12338567 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0A82C2B9F4 for ; Tue, 22 Jun 2021 20:31:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9BC6861370 for ; Tue, 22 Jun 2021 20:31:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230109AbhFVUd3 (ORCPT ); Tue, 22 Jun 2021 16:33:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44142 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230114AbhFVUd2 (ORCPT ); Tue, 22 Jun 2021 16:33:28 -0400 Received: from mail-pl1-x643.google.com (mail-pl1-x643.google.com [IPv6:2607:f8b0:4864:20::643]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC816C061574; Tue, 22 Jun 2021 13:31:11 -0700 (PDT) Received: by mail-pl1-x643.google.com with SMTP id f10so8916020plg.0; Tue, 22 Jun 2021 13:31:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Ho8mpG9beAl+0ggB96aMls1JfFIVbnP4cgBTDr5kAic=; b=moz/96mMW6dUOyAGHojY7v7orYfOZIHjrqog+XI4uttLQIgS4Jlwh3LKNtpVWyxVDz b0A3Vmluc4NpSzDKqbuSk0SUxBg0LYhlitHBnSlKNDbPoNJ5aAxHVrHzgXAZq/7RhK+m s2sN6eOL7N4JNCBIuB6ppMG01Vr2RF+uTwCCNV97LkDnVFUmOds/n/opeTwK3lO/3Ugw R7+3UizZx2G0V1JKGcMhxTMmC2eYXJtiS8D7ymknixw0x0gvh/eCyQZJJaNIgk9bNODa S9qbvsf9FkcN4uajXFtIAVdBHeLn1YdDX76Na8Vz3nV1QfivWg93IEIlfWevvH/Mx/KC IPhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Ho8mpG9beAl+0ggB96aMls1JfFIVbnP4cgBTDr5kAic=; b=JRMprbOUvNOLDQEZszi3TUz9rOIPrfi3vdsS+C/X6ZLTHBTJO5MpqLI+C3eFQ5AuFg 1MCUffmLHVa/Ojxx7SdzHG7Z8obKsPnFRUbTfhYzPRLdnuR3NxiR+yxfPd5qL4uuRaqm 7xzauMiCMVh2Vsphc4XDv4u/jHHOI3zj+cyyW5GjxZpMa6gaNKusuPcSMwgjS2TnIZms kPkQxbDg5nvda3OlR5K7ETeHfru2s+L6zi+Sl5uMYbIlwEt5msJeavDKSuCas78XhT2x 6LhNbvwvcThTHjGGuLIBElBOwOgWCI2UcTW/gCG499KVQ7qdaKAIFWecnCDndzYH9t4u AEFw== X-Gm-Message-State: AOAM532sjfWbRPmKre1Jz9wqBWhU849oT4WMgjAxwZSr6WgIm9qs4rO7 w+oYoKey8WhxJPcEstmtZ+2H5d+DCzg= X-Google-Smtp-Source: ABdhPJxXNUDzs2e7ypHVjRfEmRuzDEU+LSLZ8ZT5crga9Wl6qI5WmW+7PcktfsCfM9MRvfer2hvGOQ== X-Received: by 2002:a17:902:720c:b029:11e:787d:407e with SMTP id ba12-20020a170902720cb029011e787d407emr24689318plb.31.1624393871194; Tue, 22 Jun 2021 13:31:11 -0700 (PDT) Received: from localhost ([2402:3a80:11bb:33b3:7f0c:3646:8bde:417e]) by smtp.gmail.com with ESMTPSA id o1sm3118465pjf.56.2021.06.22.13.31.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Jun 2021 13:31:10 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: netdev@vger.kernel.org Cc: Kumar Kartikeya Dwivedi , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , =?utf-8?q?Toke_H=C3=B8iland-J=C3=B8rge?= =?utf-8?q?nsen?= , Jesper Dangaard Brouer , "David S. Miller" , Jakub Kicinski , John Fastabend , Martin KaFai Lau , bpf@vger.kernel.org Subject: [PATCH net-next v3 5/5] bpf: update XDP selftests to not fail with generic XDP Date: Wed, 23 Jun 2021 01:58:35 +0530 Message-Id: <20210622202835.1151230-6-memxor@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210622202835.1151230-1-memxor@gmail.com> References: <20210622202835.1151230-1-memxor@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Generic XDP devmaps and cpumaps now allow setting value_size to 8 bytes (so that prog_fd can be specified) and XDP progs using them succeed in SKB mode now. Adjust the checks. Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/prog_tests/xdp_cpumap_attach.c | 4 ++-- tools/testing/selftests/bpf/prog_tests/xdp_devmap_attach.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/xdp_cpumap_attach.c b/tools/testing/selftests/bpf/prog_tests/xdp_cpumap_attach.c index 0176573fe4e7..42e46d2ae349 100644 --- a/tools/testing/selftests/bpf/prog_tests/xdp_cpumap_attach.c +++ b/tools/testing/selftests/bpf/prog_tests/xdp_cpumap_attach.c @@ -29,8 +29,8 @@ void test_xdp_with_cpumap_helpers(void) */ prog_fd = bpf_program__fd(skel->progs.xdp_redir_prog); err = bpf_set_link_xdp_fd(IFINDEX_LO, prog_fd, XDP_FLAGS_SKB_MODE); - CHECK(err == 0, "Generic attach of program with 8-byte CPUMAP", - "should have failed\n"); + CHECK(err, "Generic attach of program with 8-byte CPUMAP", + "shouldn't have failed\n"); prog_fd = bpf_program__fd(skel->progs.xdp_dummy_cm); map_fd = bpf_map__fd(skel->maps.cpu_map); diff --git a/tools/testing/selftests/bpf/prog_tests/xdp_devmap_attach.c b/tools/testing/selftests/bpf/prog_tests/xdp_devmap_attach.c index 88ef3ec8ac4c..861db508ace2 100644 --- a/tools/testing/selftests/bpf/prog_tests/xdp_devmap_attach.c +++ b/tools/testing/selftests/bpf/prog_tests/xdp_devmap_attach.c @@ -31,8 +31,8 @@ void test_xdp_with_devmap_helpers(void) */ dm_fd = bpf_program__fd(skel->progs.xdp_redir_prog); err = bpf_set_link_xdp_fd(IFINDEX_LO, dm_fd, XDP_FLAGS_SKB_MODE); - CHECK(err == 0, "Generic attach of program with 8-byte devmap", - "should have failed\n"); + CHECK(err, "Generic attach of program with 8-byte devmap", + "shouldn't have failed\n"); dm_fd = bpf_program__fd(skel->progs.xdp_dummy_dm); map_fd = bpf_map__fd(skel->maps.dm_ports);