From patchwork Sat Mar 19 14:04:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taehee Yoo X-Patchwork-Id: 12786198 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44FC7C433F5 for ; Sat, 19 Mar 2022 14:05:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243099AbiCSOGZ (ORCPT ); Sat, 19 Mar 2022 10:06:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41966 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229470AbiCSOGY (ORCPT ); Sat, 19 Mar 2022 10:06:24 -0400 Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 21B641BA6BB for ; Sat, 19 Mar 2022 07:05:02 -0700 (PDT) Received: by mail-pj1-x1029.google.com with SMTP id m22so9617325pja.0 for ; Sat, 19 Mar 2022 07:05:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=cmk8PqJ86ZeCJKZoJy0SWvFhCsGbpWpxdQNL1d6J61E=; b=WjVvDnuZ8xdidYzyylnPRciQzrOWXoAIs2sc0DWYo5OtL2iCdHAEw384mVBdH2roxF rgyHlA0PU6OM31TFyBwabHIjrWWYRS3Tp4cNSsJKwhrOA9LXfrN/6f51xAqEurUVm/1n y7FU/sKLwwIAKtNJlvM0pt40q/p7wNEBrKa9WcSXx4aSp4qQQchxzpejjYcIR10mlIjq mWGQYs3PSi4Deg86LbqwY3JOs9Z+PJ1/5NK5tNiZFC4xSrx8FwRt3xbD59yoFmLFNtJg wWZzulF8/EN27E8qHg4GzQ+mw5qQayULWV7AhY5ZzVC2eSAAdFPzbZgTCgT66mgGFFgc ohDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=cmk8PqJ86ZeCJKZoJy0SWvFhCsGbpWpxdQNL1d6J61E=; b=y3UqwVBX/tv1eBERxZpUQb72T8iNObaRezmlBs7AlDC7OWVtv60DlOGgG6Es25+vno iG1OYMYbWtVUmn9gHR/k7U3sux19crQokglxp33D6ct4z/CxZtMs+lgm4K6OTuk+fe1z wbYIM5jxmDq9skIfLmQnFD83La90x2I+Y5YraY5VddSoaZhQ1kqeZVlxOOOOvXs3dRo7 1KLDvDLNsXlh1efIwfSYk9nlwBthjVcgrBSkLDGzyAJ5Azy51NAbo+0CgZDV7ZrwVlGR iAnVg8GHkhNR3WPPjjVkxPaR8S7et2coKJ6BLHEM6ba4wpoCk3KV7zIAvBxNarWIDUHq 8Wsg== X-Gm-Message-State: AOAM531QOp8s5w26qoRWlOS6GvtyL1s3BybgBgIc9ZbVtPRlCk4p0gZS YGmtQajZPhfOAkNygPrKjM9qynTDOYrTJg== X-Google-Smtp-Source: ABdhPJzwsQGHLmYS6V9t+9srltw2fxDWikqVj+VftkjT8r60UlMs62Q2Mh8BF1fQax2R1ycmXyGqLg== X-Received: by 2002:a17:90b:905:b0:1c6:9747:1939 with SMTP id bo5-20020a17090b090500b001c697471939mr12923473pjb.37.1647698701526; Sat, 19 Mar 2022 07:05:01 -0700 (PDT) Received: from localhost.localdomain ([182.213.254.91]) by smtp.gmail.com with ESMTPSA id e11-20020a63e00b000000b0037341d979b8sm10168438pgh.94.2022.03.19.07.04.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Mar 2022 07:05:00 -0700 (PDT) From: Taehee Yoo To: davem@davemloft.net, kuba@kernel.org, pabeni@redhat.com, irusskikh@marvell.com, epomozov@marvell.com, netdev@vger.kernel.org Cc: ap420073@gmail.com Subject: [PATCH net-next v2 1/3] net: atlantic: Implement xdp control plane Date: Sat, 19 Mar 2022 14:04:41 +0000 Message-Id: <20220319140443.6645-2-ap420073@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220319140443.6645-1-ap420073@gmail.com> References: <20220319140443.6645-1-ap420073@gmail.com> Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org aq_xdp() is a xdp setup callback function for Atlantic driver. When XDP is attached or detached, the device will be restarted because it uses different maximum frame sizes. If XDP is disabled, 2K RX frame size is used and alloc_skb() is used too. So, It can reuse the next half of that page(flip like the ixgbe). Or, if a reference count is 1(no other user) that page can be fully reused. ethtool -S | grep Page Queue[0] PageFlips: 0 <-- use half of page Queue[0] PageReuses: 7 <-- reuse full page But if XDP is enabled, it uses 3K frame size. So, the Flip strategy can not be applied. And reusing a full page is possible only when XDP_DROP, XDP_ABORT. Also, It supports xdp fragment feature. mtu can be 16K if xdp prog supports xdp fragment. if not, mtu can not exceed 3K - ETH_HLEN - ETH_FCS. And a static key is added and It will be used to call the xdp_clean handler in ->poll(). data plane implementation will be contained the followed patch. Signed-off-by: Taehee Yoo --- v2: - No changed .../net/ethernet/aquantia/atlantic/aq_cfg.h | 1 + .../net/ethernet/aquantia/atlantic/aq_main.c | 58 +++++++++++++++++++ .../net/ethernet/aquantia/atlantic/aq_main.h | 2 + .../net/ethernet/aquantia/atlantic/aq_nic.h | 3 + .../net/ethernet/aquantia/atlantic/aq_ring.c | 57 +++++++++++------- .../net/ethernet/aquantia/atlantic/aq_ring.h | 9 +++ .../net/ethernet/aquantia/atlantic/aq_vec.c | 23 ++++++-- .../net/ethernet/aquantia/atlantic/aq_vec.h | 6 ++ .../aquantia/atlantic/hw_atl/hw_atl_a0.c | 6 +- .../aquantia/atlantic/hw_atl/hw_atl_b0.c | 10 ++-- 10 files changed, 141 insertions(+), 34 deletions(-) diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_cfg.h b/drivers/net/ethernet/aquantia/atlantic/aq_cfg.h index 52b9833fda99..22575adedf66 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_cfg.h +++ b/drivers/net/ethernet/aquantia/atlantic/aq_cfg.h @@ -32,6 +32,7 @@ #define AQ_CFG_TX_FRAME_MAX (16U * 1024U) #define AQ_CFG_RX_FRAME_MAX (2U * 1024U) +#define AQ_CFG_XDP_FRAME_MAX (3U * 1024U) #define AQ_CFG_TX_CLEAN_BUDGET 256U diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_main.c b/drivers/net/ethernet/aquantia/atlantic/aq_main.c index e65ce7199dac..07b4f4ee715f 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_main.c +++ b/drivers/net/ethernet/aquantia/atlantic/aq_main.c @@ -14,17 +14,22 @@ #include "aq_ptp.h" #include "aq_filters.h" #include "aq_hw_utils.h" +#include "aq_vec.h" #include #include #include #include #include +#include MODULE_LICENSE("GPL v2"); MODULE_AUTHOR(AQ_CFG_DRV_AUTHOR); MODULE_DESCRIPTION(AQ_CFG_DRV_DESC); +DEFINE_STATIC_KEY_FALSE(aq_xdp_locking_key); +EXPORT_SYMBOL(aq_xdp_locking_key); + static const char aq_ndev_driver_name[] = AQ_CFG_DRV_NAME; static const struct net_device_ops aq_ndev_ops; @@ -126,9 +131,19 @@ static netdev_tx_t aq_ndev_start_xmit(struct sk_buff *skb, struct net_device *nd static int aq_ndev_change_mtu(struct net_device *ndev, int new_mtu) { + int new_frame_size = new_mtu + ETH_HLEN + ETH_FCS_LEN; struct aq_nic_s *aq_nic = netdev_priv(ndev); + struct bpf_prog *prog; int err; + prog = READ_ONCE(aq_nic->xdp_prog); + if (prog && !prog->aux->xdp_has_frags && + new_frame_size > AQ_CFG_XDP_FRAME_MAX) { + netdev_info(ndev, "Illegal MTU %d for XDP prog without frags\n", + ndev->mtu); + return -EOPNOTSUPP; + } + err = aq_nic_set_mtu(aq_nic, new_mtu + ETH_HLEN); if (err < 0) @@ -410,6 +425,48 @@ static int aq_ndo_setup_tc(struct net_device *dev, enum tc_setup_type type, mqprio->qopt.prio_tc_map); } +static int aq_xdp_setup(struct net_device *ndev, struct bpf_prog *prog, + struct netlink_ext_ack *extack) +{ + bool need_update, running = netif_running(ndev); + struct aq_nic_s *aq_nic = netdev_priv(ndev); + struct bpf_prog *old_prog; + + if (prog && !prog->aux->xdp_has_frags && + ndev->mtu > AQ_CFG_XDP_FRAME_MAX) { + NL_SET_ERR_MSG_MOD(extack, "prog does not support XDP frags"); + return -EOPNOTSUPP; + } + + need_update = !!aq_nic->xdp_prog != !!prog; + if (running && need_update) + aq_ndev_close(ndev); + + old_prog = xchg(&aq_nic->xdp_prog, prog); + if (old_prog) + bpf_prog_put(old_prog); + + if (!old_prog && prog) + static_branch_inc(&aq_xdp_locking_key); + else if (old_prog && !prog) + static_branch_dec(&aq_xdp_locking_key); + + if (running && need_update) + return aq_ndev_open(ndev); + + return 0; +} + +static int aq_xdp(struct net_device *dev, struct netdev_bpf *xdp) +{ + switch (xdp->command) { + case XDP_SETUP_PROG: + return aq_xdp_setup(dev, xdp->prog, xdp->extack); + default: + return -EINVAL; + } +} + static const struct net_device_ops aq_ndev_ops = { .ndo_open = aq_ndev_open, .ndo_stop = aq_ndev_close, @@ -422,6 +479,7 @@ static const struct net_device_ops aq_ndev_ops = { .ndo_vlan_rx_add_vid = aq_ndo_vlan_rx_add_vid, .ndo_vlan_rx_kill_vid = aq_ndo_vlan_rx_kill_vid, .ndo_setup_tc = aq_ndo_setup_tc, + .ndo_bpf = aq_xdp, }; static int __init aq_ndev_init_module(void) diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_main.h b/drivers/net/ethernet/aquantia/atlantic/aq_main.h index a5a624b9ce73..99870865f66d 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_main.h +++ b/drivers/net/ethernet/aquantia/atlantic/aq_main.h @@ -12,6 +12,8 @@ #include "aq_common.h" #include "aq_nic.h" +DECLARE_STATIC_KEY_FALSE(aq_xdp_locking_key); + void aq_ndev_schedule_work(struct work_struct *work); struct net_device *aq_ndev_alloc(void); diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_nic.h b/drivers/net/ethernet/aquantia/atlantic/aq_nic.h index 1a7148041e3d..47123baabd5e 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_nic.h +++ b/drivers/net/ethernet/aquantia/atlantic/aq_nic.h @@ -11,6 +11,8 @@ #define AQ_NIC_H #include +#include +#include #include "aq_common.h" #include "aq_rss.h" @@ -128,6 +130,7 @@ struct aq_nic_s { struct aq_vec_s *aq_vec[AQ_CFG_VECS_MAX]; struct aq_ring_s *aq_ring_tx[AQ_HW_QUEUES_MAX]; struct aq_hw_s *aq_hw; + struct bpf_prog *xdp_prog; struct net_device *ndev; unsigned int aq_vecs; unsigned int packet_filter; diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_ring.c b/drivers/net/ethernet/aquantia/atlantic/aq_ring.c index 77e76c9efd32..b261283641a7 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_ring.c +++ b/drivers/net/ethernet/aquantia/atlantic/aq_ring.c @@ -7,12 +7,16 @@ /* File aq_ring.c: Definition of functions for Rx/Tx rings. */ -#include "aq_ring.h" #include "aq_nic.h" #include "aq_hw.h" #include "aq_hw_utils.h" #include "aq_ptp.h" +#include "aq_vec.h" +#include "aq_main.h" +#include +#include +#include #include #include @@ -27,9 +31,10 @@ static inline void aq_free_rxpage(struct aq_rxpage *rxpage, struct device *dev) rxpage->page = NULL; } -static int aq_get_rxpage(struct aq_rxpage *rxpage, unsigned int order, - struct device *dev) +static int aq_get_rxpage(struct aq_rxpage *rxpage, struct aq_ring_s *rx_ring) { + struct device *dev = aq_nic_get_dev(rx_ring->aq_nic); + unsigned int order = rx_ring->page_order; struct page *page; int ret = -ENOMEM; dma_addr_t daddr; @@ -47,7 +52,7 @@ static int aq_get_rxpage(struct aq_rxpage *rxpage, unsigned int order, rxpage->page = page; rxpage->daddr = daddr; rxpage->order = order; - rxpage->pg_off = 0; + rxpage->pg_off = rx_ring->page_offset; return 0; @@ -58,21 +63,24 @@ static int aq_get_rxpage(struct aq_rxpage *rxpage, unsigned int order, return ret; } -static int aq_get_rxpages(struct aq_ring_s *self, struct aq_ring_buff_s *rxbuf, - int order) +static int aq_get_rxpages(struct aq_ring_s *self, struct aq_ring_buff_s *rxbuf) { + unsigned int order = self->page_order; + u16 page_offset = self->page_offset; + u16 frame_max = self->frame_max; int ret; if (rxbuf->rxdata.page) { /* One means ring is the only user and can reuse */ if (page_ref_count(rxbuf->rxdata.page) > 1) { /* Try reuse buffer */ - rxbuf->rxdata.pg_off += AQ_CFG_RX_FRAME_MAX; - if (rxbuf->rxdata.pg_off + AQ_CFG_RX_FRAME_MAX <= + rxbuf->rxdata.pg_off += frame_max + page_offset; + if (rxbuf->rxdata.pg_off + frame_max <= (PAGE_SIZE << order)) { u64_stats_update_begin(&self->stats.rx.syncp); self->stats.rx.pg_flips++; u64_stats_update_end(&self->stats.rx.syncp); + } else { /* Buffer exhausted. We have other users and * should release this page and realloc @@ -84,7 +92,7 @@ static int aq_get_rxpages(struct aq_ring_s *self, struct aq_ring_buff_s *rxbuf, u64_stats_update_end(&self->stats.rx.syncp); } } else { - rxbuf->rxdata.pg_off = 0; + rxbuf->rxdata.pg_off = page_offset; u64_stats_update_begin(&self->stats.rx.syncp); self->stats.rx.pg_reuses++; u64_stats_update_end(&self->stats.rx.syncp); @@ -92,8 +100,7 @@ static int aq_get_rxpages(struct aq_ring_s *self, struct aq_ring_buff_s *rxbuf, } if (!rxbuf->rxdata.page) { - ret = aq_get_rxpage(&rxbuf->rxdata, order, - aq_nic_get_dev(self->aq_nic)); + ret = aq_get_rxpage(&rxbuf->rxdata, self); if (ret) { u64_stats_update_begin(&self->stats.rx.syncp); self->stats.rx.alloc_fails++; @@ -117,6 +124,7 @@ static struct aq_ring_s *aq_ring_alloc(struct aq_ring_s *self, err = -ENOMEM; goto err_exit; } + self->dx_ring = dma_alloc_coherent(aq_nic_get_dev(aq_nic), self->size * self->dx_size, &self->dx_ring_pa, GFP_KERNEL); @@ -172,11 +180,21 @@ struct aq_ring_s *aq_ring_rx_alloc(struct aq_ring_s *self, self->idx = idx; self->size = aq_nic_cfg->rxds; self->dx_size = aq_nic_cfg->aq_hw_caps->rxd_size; - self->page_order = fls(AQ_CFG_RX_FRAME_MAX / PAGE_SIZE + - (AQ_CFG_RX_FRAME_MAX % PAGE_SIZE ? 1 : 0)) - 1; + self->xdp_prog = aq_nic->xdp_prog; - if (aq_nic_cfg->rxpageorder > self->page_order) - self->page_order = aq_nic_cfg->rxpageorder; + /* Only order-0 is allowed if XDP is enabled */ + if (READ_ONCE(self->xdp_prog)) { + self->page_offset = AQ_XDP_HEADROOM; + self->page_order = 0; + self->frame_max = AQ_CFG_XDP_FRAME_MAX; + } else { + self->page_offset = 0; + self->frame_max = AQ_CFG_RX_FRAME_MAX; + self->page_order = fls(self->frame_max / PAGE_SIZE + + (self->frame_max % PAGE_SIZE ? 1 : 0)) - 1; + if (aq_nic_cfg->rxpageorder > self->page_order) + self->page_order = aq_nic_cfg->rxpageorder; + } self = aq_ring_alloc(self, aq_nic); if (!self) { @@ -449,7 +467,7 @@ int aq_ring_rx_clean(struct aq_ring_s *self, skb_add_rx_frag(skb, 0, buff->rxdata.page, buff->rxdata.pg_off + hdr_len, buff->len - hdr_len, - AQ_CFG_RX_FRAME_MAX); + self->frame_max); page_ref_inc(buff->rxdata.page); } @@ -469,7 +487,7 @@ int aq_ring_rx_clean(struct aq_ring_s *self, buff_->rxdata.page, buff_->rxdata.pg_off, buff_->len, - AQ_CFG_RX_FRAME_MAX); + self->frame_max); page_ref_inc(buff_->rxdata.page); buff_->is_cleaned = 1; @@ -529,7 +547,6 @@ void aq_ring_hwts_rx_clean(struct aq_ring_s *self, struct aq_nic_s *aq_nic) int aq_ring_rx_fill(struct aq_ring_s *self) { - unsigned int page_order = self->page_order; struct aq_ring_buff_s *buff = NULL; int err = 0; int i = 0; @@ -543,9 +560,9 @@ int aq_ring_rx_fill(struct aq_ring_s *self) buff = &self->buff_ring[self->sw_tail]; buff->flags = 0U; - buff->len = AQ_CFG_RX_FRAME_MAX; + buff->len = self->frame_max; - err = aq_get_rxpages(self, buff, page_order); + err = aq_get_rxpages(self, buff); if (err) goto err_exit; diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_ring.h b/drivers/net/ethernet/aquantia/atlantic/aq_ring.h index 93659e58f1ce..168002657629 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_ring.h +++ b/drivers/net/ethernet/aquantia/atlantic/aq_ring.h @@ -11,6 +11,9 @@ #define AQ_RING_H #include "aq_common.h" +#include "aq_vec.h" + +#define AQ_XDP_HEADROOM ALIGN(max(NET_SKB_PAD, XDP_PACKET_HEADROOM), 8) struct page; struct aq_nic_cfg_s; @@ -51,6 +54,7 @@ struct __packed aq_ring_buff_s { struct { dma_addr_t pa_eop; struct sk_buff *skb; + struct xdp_frame *xdpf; }; /* TxC */ struct { @@ -133,9 +137,13 @@ struct aq_ring_s { unsigned int dx_size; /* TX or RX descriptor size, */ /* stored here for fater math */ unsigned int page_order; + u16 page_offset; + u16 frame_max; union aq_ring_stats_s stats; dma_addr_t dx_ring_pa; + struct bpf_prog *xdp_prog; enum atl_ring_type ring_type; + struct xdp_rxq_info xdp_rxq; }; struct aq_ring_param_s { @@ -175,6 +183,7 @@ struct aq_ring_s *aq_ring_rx_alloc(struct aq_ring_s *self, struct aq_nic_s *aq_nic, unsigned int idx, struct aq_nic_cfg_s *aq_nic_cfg); + int aq_ring_init(struct aq_ring_s *self, const enum atl_ring_type ring_type); void aq_ring_rx_deinit(struct aq_ring_s *self); void aq_ring_free(struct aq_ring_s *self); diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_vec.c b/drivers/net/ethernet/aquantia/atlantic/aq_vec.c index f4774cf051c9..9657adcdd3c4 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_vec.c +++ b/drivers/net/ethernet/aquantia/atlantic/aq_vec.c @@ -10,11 +10,6 @@ */ #include "aq_vec.h" -#include "aq_nic.h" -#include "aq_ring.h" -#include "aq_hw.h" - -#include struct aq_vec_s { const struct aq_hw_ops *aq_hw_ops; @@ -153,9 +148,23 @@ int aq_vec_ring_alloc(struct aq_vec_s *self, struct aq_nic_s *aq_nic, aq_nic_set_tx_ring(aq_nic, idx_ring, ring); + if (xdp_rxq_info_reg(&self->ring[i][AQ_VEC_RX_ID].xdp_rxq, + aq_nic->ndev, idx, + self->napi.napi_id) < 0) { + err = -ENOMEM; + goto err_exit; + } + if (xdp_rxq_info_reg_mem_model(&self->ring[i][AQ_VEC_RX_ID].xdp_rxq, + MEM_TYPE_PAGE_ORDER0, NULL) < 0) { + xdp_rxq_info_unreg(&self->ring[i][AQ_VEC_RX_ID].xdp_rxq); + err = -ENOMEM; + goto err_exit; + } + ring = aq_ring_rx_alloc(&self->ring[i][AQ_VEC_RX_ID], aq_nic, idx_ring, aq_nic_cfg); if (!ring) { + xdp_rxq_info_unreg(&self->ring[i][AQ_VEC_RX_ID].xdp_rxq); err = -ENOMEM; goto err_exit; } @@ -300,8 +309,10 @@ void aq_vec_ring_free(struct aq_vec_s *self) for (i = 0U, ring = self->ring[0]; self->tx_rings > i; ++i, ring = self->ring[i]) { aq_ring_free(&ring[AQ_VEC_TX_ID]); - if (i < self->rx_rings) + if (i < self->rx_rings) { + xdp_rxq_info_unreg(&ring[AQ_VEC_RX_ID].xdp_rxq); aq_ring_free(&ring[AQ_VEC_RX_ID]); + } } self->tx_rings = 0; diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_vec.h b/drivers/net/ethernet/aquantia/atlantic/aq_vec.h index 567f3d4b79a2..78fac609b71d 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_vec.h +++ b/drivers/net/ethernet/aquantia/atlantic/aq_vec.h @@ -13,7 +13,13 @@ #define AQ_VEC_H #include "aq_common.h" +#include "aq_nic.h" +#include "aq_ring.h" +#include "aq_hw.h" + #include +#include +#include struct aq_hw_s; struct aq_hw_ops; diff --git a/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_a0.c b/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_a0.c index 4625ccb79499..9dfd68f0fda9 100644 --- a/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_a0.c +++ b/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_a0.c @@ -531,7 +531,7 @@ static int hw_atl_a0_hw_ring_rx_init(struct aq_hw_s *self, hw_atl_rdm_rx_desc_len_set(self, aq_ring->size / 8U, aq_ring->idx); hw_atl_rdm_rx_desc_data_buff_size_set(self, - AQ_CFG_RX_FRAME_MAX / 1024U, + aq_ring->frame_max / 1024U, aq_ring->idx); hw_atl_rdm_rx_desc_head_buff_size_set(self, 0U, aq_ring->idx); @@ -706,9 +706,9 @@ static int hw_atl_a0_hw_ring_rx_receive(struct aq_hw_s *self, if (HW_ATL_A0_RXD_WB_STAT2_EOP & rxd_wb->status) { buff->len = rxd_wb->pkt_len % - AQ_CFG_RX_FRAME_MAX; + ring->frame_max; buff->len = buff->len ? - buff->len : AQ_CFG_RX_FRAME_MAX; + buff->len : ring->frame_max; buff->next = 0U; buff->is_eop = 1U; } else { diff --git a/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_b0.c b/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_b0.c index d875ce3ec759..1cf177561aad 100644 --- a/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_b0.c +++ b/drivers/net/ethernet/aquantia/atlantic/hw_atl/hw_atl_b0.c @@ -766,7 +766,7 @@ int hw_atl_b0_hw_ring_rx_init(struct aq_hw_s *self, struct aq_ring_s *aq_ring, hw_atl_rdm_rx_desc_len_set(self, aq_ring->size / 8U, aq_ring->idx); hw_atl_rdm_rx_desc_data_buff_size_set(self, - AQ_CFG_RX_FRAME_MAX / 1024U, + aq_ring->frame_max / 1024U, aq_ring->idx); hw_atl_rdm_rx_desc_head_buff_size_set(self, 0U, aq_ring->idx); @@ -969,15 +969,15 @@ int hw_atl_b0_hw_ring_rx_receive(struct aq_hw_s *self, struct aq_ring_s *ring) rxd_wb->status); if (HW_ATL_B0_RXD_WB_STAT2_EOP & rxd_wb->status) { buff->len = rxd_wb->pkt_len % - AQ_CFG_RX_FRAME_MAX; + ring->frame_max; buff->len = buff->len ? - buff->len : AQ_CFG_RX_FRAME_MAX; + buff->len : ring->frame_max; buff->next = 0U; buff->is_eop = 1U; } else { buff->len = - rxd_wb->pkt_len > AQ_CFG_RX_FRAME_MAX ? - AQ_CFG_RX_FRAME_MAX : rxd_wb->pkt_len; + rxd_wb->pkt_len > ring->frame_max ? + ring->frame_max : rxd_wb->pkt_len; if (buff->is_lro) { /* LRO */ From patchwork Sat Mar 19 14:04:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taehee Yoo X-Patchwork-Id: 12786200 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A59DC433FE for ; Sat, 19 Mar 2022 14:05:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243105AbiCSOGb (ORCPT ); Sat, 19 Mar 2022 10:06:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42176 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229470AbiCSOG1 (ORCPT ); Sat, 19 Mar 2022 10:06:27 -0400 Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 235351BE0D9 for ; Sat, 19 Mar 2022 07:05:05 -0700 (PDT) Received: by mail-pl1-x62f.google.com with SMTP id n2so9245380plf.4 for ; Sat, 19 Mar 2022 07:05:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=8vd6rjFo5MDV4GkLyXevsU20OroNc9yzyYA/hrhY5wA=; b=nAdP+XpNZ4htWFV2F+vJ2d/CIx6QtWoGJqyICersoDnrQccHF7oFWID3R5kNAOl9fK ghAs2xfzLTWg55rPUMhdSNf5oLiHLMnsPbwkCPsozZ6+ojyZcKdnLm45DAIecE9z3u3A ADXYkeTO95jJg8Ba4czhDDt3895k6DNQc9Ks/e3Hu+tl0LgBRIh4Xv6tZ6Eu/KbufBsi 1UQKZMyMpTvDnKsWXh9Z4j7zAehRb92o4L/r9nUhehaCcs4P5yYRUJ275iSOLxUUXvVp bORmWOJ1f4T0EqADuMXJhOEuBXqi94IZztDpWhEZw/NJymNqoW5ShNcSUm0wx9ekR12a pmyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=8vd6rjFo5MDV4GkLyXevsU20OroNc9yzyYA/hrhY5wA=; b=r+ueXE88YJCDc5WGc3DHnmxdkI/3iZcMZsJYdGVt8dplmSYrYRbZ3yoQCcZY/3CzD0 waYXXHs1sjanY3pM5ZarJccyBq7vFxy6dcBczf6RtrrZBScck56wZOAvO8ss5USfH05G Os5X15lDl0w/1euCjIbzzWak2PFMjihPWD7CfPT04wqk1EiZxk0r9r3jbB1CzTjnioED EJ+yqEDyXurqadPkytdAiPtp3c7OvsUM0zu4QSlPSn8bZY871u5wMaLBfy0IsjxnYc4T QSLppXUsyCiaSb6AXrRxTqacB8WIrkc5UmaOu+KU4SwnXdUnTogVQHgzip+/rX6joxsA bCgQ== X-Gm-Message-State: AOAM531tTRloPF8CEJcFssROTFMpdDZFehtmUkwZWmucZIyMLsF6kya7 xyhTvYIEFDArW2i6JhXW/z8= X-Google-Smtp-Source: ABdhPJxTa6mDyvkz5v1X25j8lM54QbVEVHIORHIET2rtjYbVeyty1tSBmoTmWNU8gVxjdINedwwuuA== X-Received: by 2002:a17:90b:38c9:b0:1bf:8668:9399 with SMTP id nn9-20020a17090b38c900b001bf86689399mr26770350pjb.87.1647698704436; Sat, 19 Mar 2022 07:05:04 -0700 (PDT) Received: from localhost.localdomain ([182.213.254.91]) by smtp.gmail.com with ESMTPSA id e11-20020a63e00b000000b0037341d979b8sm10168438pgh.94.2022.03.19.07.05.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Mar 2022 07:05:03 -0700 (PDT) From: Taehee Yoo To: davem@davemloft.net, kuba@kernel.org, pabeni@redhat.com, irusskikh@marvell.com, epomozov@marvell.com, netdev@vger.kernel.org Cc: ap420073@gmail.com Subject: [PATCH net-next v2 2/3] net: atlantic: Implement xdp data plane Date: Sat, 19 Mar 2022 14:04:42 +0000 Message-Id: <20220319140443.6645-3-ap420073@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220319140443.6645-1-ap420073@gmail.com> References: <20220319140443.6645-1-ap420073@gmail.com> Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org It supports XDP_PASS, XDP_DROP and multi buffer. From now on aq_nic_map_skb() supports xdp_frame to send packet. So, TX path of both skb and xdp_frame can use aq_nic_map_skb(). aq_nic_xmit() is used to send packet with skb and internally it calls aq_nic_map_skb(). aq_nic_xmit_xdpf() is used to send packet with xdp_frame and internally it calls aq_nic_map_skb(). AQC chip supports 32 multi-queues and 8 vectors(irq). there are two option 1. under 8 cores and 4 tx queues per core. 2. under 4 cores and 8 tx queues per core. Like ixgbe, these tx queues can be used only for XDP_TX, XDP_REDIRECT queue. If so, no tx_lock is needed. But this patchset doesn't use this strategy because getting hardware tx queue index cost is too high. So, tx_lock is used in the aq_nic_xmit_xdpf(). single-core, single queue, 40% cpu utilization. 30.75% bpf_prog_xxx_xdp_prog_tx [k] bpf_prog_xxx_xdp_prog_tx 10.35% [kernel] [k] aq_hw_read_reg <---------- here 4.38% [kernel] [k] get_page_from_freelist single-core, 8 queues, 100% cpu utilization, half PPS. 45.56% [kernel] [k] aq_hw_read_reg <---------- here 17.58% bpf_prog_xxx_xdp_prog_tx [k] bpf_prog_xxx_xdp_prog_tx 4.72% [kernel] [k] hw_atl_b0_hw_ring_rx_receive The new function __aq_ring_xdp_clean() is a xdp rx handler and this is called only when XDP is attached. Signed-off-by: Taehee Yoo --- V2: - Do not use inline in C file .../ethernet/aquantia/atlantic/aq_ethtool.c | 8 + .../net/ethernet/aquantia/atlantic/aq_nic.c | 126 +++++-- .../net/ethernet/aquantia/atlantic/aq_nic.h | 4 +- .../net/ethernet/aquantia/atlantic/aq_ptp.c | 2 +- .../net/ethernet/aquantia/atlantic/aq_ring.c | 329 +++++++++++++++++- .../net/ethernet/aquantia/atlantic/aq_ring.h | 6 + 6 files changed, 428 insertions(+), 47 deletions(-) diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_ethtool.c b/drivers/net/ethernet/aquantia/atlantic/aq_ethtool.c index a418238f6309..b33979bdfde0 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_ethtool.c +++ b/drivers/net/ethernet/aquantia/atlantic/aq_ethtool.c @@ -97,6 +97,14 @@ static const char * const aq_ethtool_queue_rx_stat_names[] = { "%sQueue[%d] AllocFails", "%sQueue[%d] SkbAllocFails", "%sQueue[%d] Polls", + "%sQueue[%d] PageFlips", + "%sQueue[%d] PageReuses", + "%sQueue[%d] XdpAbort", + "%sQueue[%d] XdpDrop", + "%sQueue[%d] XdpPass", + "%sQueue[%d] XdpTx", + "%sQueue[%d] XdpInvalid", + "%sQueue[%d] XdpRedirect", }; static const char * const aq_ethtool_queue_tx_stat_names[] = { diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_nic.c b/drivers/net/ethernet/aquantia/atlantic/aq_nic.c index 33f1a1377588..c79967370af3 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_nic.c +++ b/drivers/net/ethernet/aquantia/atlantic/aq_nic.c @@ -570,29 +570,50 @@ int aq_nic_start(struct aq_nic_s *self) } unsigned int aq_nic_map_skb(struct aq_nic_s *self, struct sk_buff *skb, - struct aq_ring_s *ring) + struct xdp_frame *xdpf, struct aq_ring_s *ring) { - unsigned int nr_frags = skb_shinfo(skb)->nr_frags; struct aq_nic_cfg_s *cfg = aq_nic_get_cfg(self); struct device *dev = aq_nic_get_dev(self); struct aq_ring_buff_s *first = NULL; - u8 ipver = ip_hdr(skb)->version; struct aq_ring_buff_s *dx_buff; + struct skb_shared_info *sinfo; bool need_context_tag = false; unsigned int frag_count = 0U; + unsigned int nr_frags = 0; unsigned int ret = 0U; unsigned int dx; + void *data_ptr; u8 l4proto = 0; + u16 total_len; + u8 ipver; + + dx = ring->sw_tail; + dx_buff = &ring->buff_ring[dx]; + dx_buff->flags = 0U; + + if (xdpf) { + sinfo = xdp_get_shared_info_from_frame(xdpf); + total_len = xdpf->len; + dx_buff->len = total_len; + data_ptr = xdpf->data; + if (xdp_frame_has_frags(xdpf)) { + nr_frags = sinfo->nr_frags; + total_len += sinfo->xdp_frags_size; + } + goto start_xdp; + } else { + sinfo = skb_shinfo(skb); + ipver = ip_hdr(skb)->version; + nr_frags = sinfo->nr_frags; + total_len = skb->len; + data_ptr = skb->data; + } if (ipver == 4) l4proto = ip_hdr(skb)->protocol; else if (ipver == 6) l4proto = ipv6_hdr(skb)->nexthdr; - dx = ring->sw_tail; - dx_buff = &ring->buff_ring[dx]; - dx_buff->flags = 0U; - if (unlikely(skb_is_gso(skb))) { dx_buff->mss = skb_shinfo(skb)->gso_size; if (l4proto == IPPROTO_TCP) { @@ -630,9 +651,16 @@ unsigned int aq_nic_map_skb(struct aq_nic_s *self, struct sk_buff *skb, ++ret; } + if (skb->ip_summed == CHECKSUM_PARTIAL) { + dx_buff->is_ip_cso = (htons(ETH_P_IP) == skb->protocol); + dx_buff->is_tcp_cso = (l4proto == IPPROTO_TCP); + dx_buff->is_udp_cso = (l4proto == IPPROTO_UDP); + } + dx_buff->len = skb_headlen(skb); +start_xdp: dx_buff->pa = dma_map_single(dev, - skb->data, + data_ptr, dx_buff->len, DMA_TO_DEVICE); @@ -642,25 +670,17 @@ unsigned int aq_nic_map_skb(struct aq_nic_s *self, struct sk_buff *skb, } first = dx_buff; - dx_buff->len_pkt = skb->len; + dx_buff->len_pkt = total_len; dx_buff->is_sop = 1U; dx_buff->is_mapped = 1U; ++ret; - if (skb->ip_summed == CHECKSUM_PARTIAL) { - dx_buff->is_ip_cso = (htons(ETH_P_IP) == skb->protocol); - dx_buff->is_tcp_cso = (l4proto == IPPROTO_TCP); - dx_buff->is_udp_cso = (l4proto == IPPROTO_UDP); - } - for (; nr_frags--; ++frag_count) { - unsigned int frag_len = 0U; + skb_frag_t *frag = &sinfo->frags[frag_count]; + unsigned int frag_len = skb_frag_size(frag); unsigned int buff_offset = 0U; unsigned int buff_size = 0U; dma_addr_t frag_pa; - skb_frag_t *frag = &skb_shinfo(skb)->frags[frag_count]; - - frag_len = skb_frag_size(frag); while (frag_len) { if (frag_len > AQ_CFG_TX_FRAME_MAX) @@ -697,6 +717,7 @@ unsigned int aq_nic_map_skb(struct aq_nic_s *self, struct sk_buff *skb, first->eop_index = dx; dx_buff->is_eop = 1U; dx_buff->skb = skb; + dx_buff->xdpf = xdpf; goto exit; mapping_error: @@ -725,6 +746,54 @@ unsigned int aq_nic_map_skb(struct aq_nic_s *self, struct sk_buff *skb, return ret; } +static int __aq_nic_xmit(struct aq_nic_s *aq_nic, struct aq_ring_s *tx_ring, + struct sk_buff *skb, struct xdp_frame *xdpf, + unsigned int frags) +{ + u16 queue_index = AQ_NIC_RING2QMAP(aq_nic, tx_ring->idx); + int err = NETDEV_TX_BUSY; + + aq_ring_update_queue_state(tx_ring); + + /* Above status update may stop the queue. Check this. */ + if (__netif_subqueue_stopped(aq_nic_get_ndev(aq_nic), queue_index)) + return err; + + frags = aq_nic_map_skb(aq_nic, skb, xdpf, tx_ring); + if (likely(frags)) + err = aq_nic->aq_hw_ops->hw_ring_tx_xmit(aq_nic->aq_hw, tx_ring, + frags); + + return err; +} + +int aq_nic_xmit_xdpf(struct aq_nic_s *aq_nic, struct aq_ring_s *tx_ring, + struct xdp_frame *xdpf) +{ + struct net_device *ndev = aq_nic_get_ndev(aq_nic); + struct skb_shared_info *sinfo; + int cpu = smp_processor_id(); + int err = NETDEV_TX_BUSY; + struct netdev_queue *nq; + unsigned int frags = 1; + + if (xdp_frame_has_frags(xdpf)) { + sinfo = xdp_get_shared_info_from_frame(xdpf); + frags += sinfo->nr_frags; + } + + if (frags > AQ_CFG_SKB_FRAGS_MAX) + return err; + + nq = netdev_get_tx_queue(ndev, tx_ring->idx); + __netif_tx_lock(nq, cpu); + err = __aq_nic_xmit(aq_nic, tx_ring, NULL, xdpf, frags); + txq_trans_cond_update(nq); + __netif_tx_unlock(nq); + + return err; +} + int aq_nic_xmit(struct aq_nic_s *self, struct sk_buff *skb) { struct aq_nic_cfg_s *cfg = aq_nic_get_cfg(self); @@ -743,29 +812,12 @@ int aq_nic_xmit(struct aq_nic_s *self, struct sk_buff *skb) goto err_exit; } - aq_ring_update_queue_state(ring); - if (cfg->priv_flags & BIT(AQ_HW_LOOPBACK_DMA_NET)) { err = NETDEV_TX_BUSY; goto err_exit; } - /* Above status update may stop the queue. Check this. */ - if (__netif_subqueue_stopped(self->ndev, - AQ_NIC_RING2QMAP(self, ring->idx))) { - err = NETDEV_TX_BUSY; - goto err_exit; - } - - frags = aq_nic_map_skb(self, skb, ring); - - if (likely(frags)) { - err = self->aq_hw_ops->hw_ring_tx_xmit(self->aq_hw, - ring, frags); - } else { - err = NETDEV_TX_BUSY; - } - + err = __aq_nic_xmit(self, ring, skb, NULL, frags); err_exit: return err; } diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_nic.h b/drivers/net/ethernet/aquantia/atlantic/aq_nic.h index 47123baabd5e..a4c314f31c3a 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_nic.h +++ b/drivers/net/ethernet/aquantia/atlantic/aq_nic.h @@ -179,7 +179,9 @@ int aq_nic_ndev_register(struct aq_nic_s *self); void aq_nic_ndev_free(struct aq_nic_s *self); int aq_nic_start(struct aq_nic_s *self); unsigned int aq_nic_map_skb(struct aq_nic_s *self, struct sk_buff *skb, - struct aq_ring_s *ring); + struct xdp_frame *xdpf, struct aq_ring_s *ring); +int aq_nic_xmit_xdpf(struct aq_nic_s *aq_nic, struct aq_ring_s *tx_ring, + struct xdp_frame *xdpf); int aq_nic_xmit(struct aq_nic_s *self, struct sk_buff *skb); int aq_nic_get_regs(struct aq_nic_s *self, struct ethtool_regs *regs, void *p); int aq_nic_get_regs_count(struct aq_nic_s *self); diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_ptp.c b/drivers/net/ethernet/aquantia/atlantic/aq_ptp.c index 06de19f63287..24e50c89f7cf 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_ptp.c +++ b/drivers/net/ethernet/aquantia/atlantic/aq_ptp.c @@ -778,7 +778,7 @@ int aq_ptp_xmit(struct aq_nic_s *aq_nic, struct sk_buff *skb) skb_tx_timestamp(skb); spin_lock_irqsave(&aq_nic->aq_ptp->ptp_ring_lock, irq_flags); - frags = aq_nic_map_skb(aq_nic, skb, ring); + frags = aq_nic_map_skb(aq_nic, skb, NULL, ring); if (likely(frags)) { err = aq_nic->aq_hw_ops->hw_ring_tx_xmit(aq_nic->aq_hw, diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_ring.c b/drivers/net/ethernet/aquantia/atlantic/aq_ring.c index b261283641a7..85c3b7a9f387 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_ring.c +++ b/drivers/net/ethernet/aquantia/atlantic/aq_ring.c @@ -31,6 +31,39 @@ static inline void aq_free_rxpage(struct aq_rxpage *rxpage, struct device *dev) rxpage->page = NULL; } +static void aq_unset_rxpage_xdp(struct aq_ring_s *rx_ring, + struct aq_ring_buff_s *buff, + struct device *dev, + struct xdp_buff *xdp) +{ + struct aq_ring_buff_s *nbuff = buff; + struct skb_shared_info *sinfo; + unsigned int next = 0; + int i; + + if (xdp_buff_has_frags(xdp)) { + sinfo = xdp_get_shared_info_from_buff(xdp); + + for (i = 0; i < sinfo->nr_frags; i++) { + skb_frag_t *frag = &sinfo->frags[i]; + struct page *page; + + page = skb_frag_page(frag); + + dma_unmap_page(dev, page->dma_addr, skb_frag_size(frag), + DMA_FROM_DEVICE); + } + + do { + next = nbuff->next; + nbuff = &rx_ring->buff_ring[next]; + nbuff->rxdata.page = NULL; + } while (!nbuff->is_eop); + } + dma_unmap_page(dev, buff->rxdata.daddr, PAGE_SIZE, DMA_FROM_DEVICE); + buff->rxdata.page = NULL; +} + static int aq_get_rxpage(struct aq_rxpage *rxpage, struct aq_ring_s *rx_ring) { struct device *dev = aq_nic_get_dev(rx_ring->aq_nic); @@ -293,6 +326,7 @@ void aq_ring_queue_stop(struct aq_ring_s *ring) bool aq_ring_tx_clean(struct aq_ring_s *self) { struct device *dev = aq_nic_get_dev(self->aq_nic); + struct skb_shared_info *sinfo; unsigned int budget; for (budget = AQ_CFG_TX_CLEAN_BUDGET; @@ -316,15 +350,33 @@ bool aq_ring_tx_clean(struct aq_ring_s *self) } } - if (unlikely(buff->is_eop && buff->skb)) { + if (likely(!buff->is_eop)) + goto out; + + if (buff->skb) { u64_stats_update_begin(&self->stats.tx.syncp); ++self->stats.tx.packets; self->stats.tx.bytes += buff->skb->len; u64_stats_update_end(&self->stats.tx.syncp); - dev_kfree_skb_any(buff->skb); - buff->skb = NULL; + } else if (buff->xdpf) { + u64_stats_update_begin(&self->stats.tx.syncp); + ++self->stats.tx.packets; + self->stats.tx.bytes += buff->xdpf->len; + u64_stats_update_end(&self->stats.tx.syncp); + if (xdp_frame_has_frags(buff->xdpf)) { + sinfo = xdp_get_shared_info_from_frame(buff->xdpf); + u64_stats_update_begin(&self->stats.rx.syncp); + self->stats.tx.bytes += sinfo->xdp_frags_size; + u64_stats_update_end(&self->stats.rx.syncp); + } + + xdp_return_frame_rx_napi(buff->xdpf); } + +out: + buff->skb = NULL; + buff->xdpf = NULL; buff->pa = 0U; buff->eop_index = 0xffffU; self->sw_head = aq_ring_next_dx(self, self->sw_head); @@ -357,11 +409,130 @@ static void aq_rx_checksum(struct aq_ring_s *self, __skb_incr_checksum_unnecessary(skb); } -#define AQ_SKB_ALIGN SKB_DATA_ALIGN(sizeof(struct skb_shared_info)) -int aq_ring_rx_clean(struct aq_ring_s *self, - struct napi_struct *napi, - int *work_done, - int budget) +static struct sk_buff *aq_xdp_run_prog(struct aq_nic_s *aq_nic, + struct xdp_buff *xdp, + struct aq_ring_s *rx_ring, + struct aq_ring_buff_s *buff) +{ + struct skb_shared_info *sinfo; + int result = NETDEV_TX_BUSY; + struct aq_ring_s *tx_ring; + struct xdp_frame *xdpf; + struct bpf_prog *prog; + struct sk_buff *skb; + u32 act; + + u64_stats_update_begin(&rx_ring->stats.rx.syncp); + ++rx_ring->stats.rx.packets; + rx_ring->stats.rx.bytes += buff->len; + if (xdp_buff_has_frags(xdp)) { + sinfo = xdp_get_shared_info_from_buff(xdp); + rx_ring->stats.rx.bytes += sinfo->xdp_frags_size; + } + u64_stats_update_end(&rx_ring->stats.rx.syncp); + + prog = READ_ONCE(rx_ring->xdp_prog); + if (!prog) + goto pass; + + prefetchw(xdp->data_hard_start); /* xdp_frame write */ + + act = bpf_prog_run_xdp(prog, xdp); + switch (act) { + case XDP_PASS: +pass: + xdpf = xdp_convert_buff_to_frame(xdp); + if (unlikely(!xdpf)) + goto out_aborted; + skb = xdp_build_skb_from_frame(xdpf, aq_nic->ndev); + if (!skb) + goto out_aborted; + u64_stats_update_begin(&rx_ring->stats.rx.syncp); + ++rx_ring->stats.rx.xdp_pass; + u64_stats_update_end(&rx_ring->stats.rx.syncp); + aq_unset_rxpage_xdp(rx_ring, buff, &aq_nic->ndev->dev, xdp); + return skb; + case XDP_TX: + xdpf = xdp_convert_buff_to_frame(xdp); + if (unlikely(!xdpf)) + goto out_aborted; + tx_ring = aq_nic->aq_ring_tx[rx_ring->idx]; + result = aq_nic_xmit_xdpf(aq_nic, tx_ring, xdpf); + if (result == NETDEV_TX_BUSY) + goto out_aborted; + u64_stats_update_begin(&rx_ring->stats.rx.syncp); + ++rx_ring->stats.rx.xdp_tx; + u64_stats_update_end(&rx_ring->stats.rx.syncp); + aq_unset_rxpage_xdp(rx_ring, buff, &aq_nic->ndev->dev, xdp); + break; + case XDP_REDIRECT: + if (xdp_do_redirect(aq_nic->ndev, xdp, prog) < 0) + goto out_aborted; + xdp_do_flush(); + u64_stats_update_begin(&rx_ring->stats.rx.syncp); + ++rx_ring->stats.rx.xdp_redirect; + u64_stats_update_end(&rx_ring->stats.rx.syncp); + aq_unset_rxpage_xdp(rx_ring, buff, &aq_nic->ndev->dev, xdp); + break; + default: + fallthrough; + case XDP_ABORTED: +out_aborted: + u64_stats_update_begin(&rx_ring->stats.rx.syncp); + ++rx_ring->stats.rx.xdp_aborted; + u64_stats_update_end(&rx_ring->stats.rx.syncp); + trace_xdp_exception(aq_nic->ndev, prog, act); + bpf_warn_invalid_xdp_action(aq_nic->ndev, prog, act); + break; + case XDP_DROP: + u64_stats_update_begin(&rx_ring->stats.rx.syncp); + ++rx_ring->stats.rx.xdp_drop; + u64_stats_update_end(&rx_ring->stats.rx.syncp); + break; + } + + return ERR_PTR(-result); +} + +static void aq_add_rx_fragment(struct device *dev, + struct aq_ring_s *ring, + struct aq_ring_buff_s *buff, + struct xdp_buff *xdp) +{ + struct skb_shared_info *sinfo = xdp_get_shared_info_from_buff(xdp); + struct aq_ring_buff_s *buff_ = buff; + + memset(sinfo, 0, sizeof(*sinfo)); + do { + skb_frag_t *frag = &sinfo->frags[sinfo->nr_frags++]; + + buff_ = &ring->buff_ring[buff_->next]; + dma_sync_single_range_for_cpu(dev, + buff_->rxdata.daddr, + buff_->rxdata.pg_off, + buff_->len, + DMA_FROM_DEVICE); + skb_frag_off_set(frag, buff_->rxdata.pg_off); + skb_frag_size_set(frag, buff_->len); + sinfo->xdp_frags_size += buff_->len; + __skb_frag_set_page(frag, buff_->rxdata.page); + + buff_->is_cleaned = 1; + + buff->is_ip_cso &= buff_->is_ip_cso; + buff->is_udp_cso &= buff_->is_udp_cso; + buff->is_tcp_cso &= buff_->is_tcp_cso; + buff->is_cso_err |= buff_->is_cso_err; + + if (page_is_pfmemalloc(buff_->rxdata.page)) + xdp_buff_set_frag_pfmemalloc(xdp); + } while (!buff_->is_eop); + + xdp_buff_set_frags_flag(xdp); +} + +static int __aq_ring_rx_clean(struct aq_ring_s *self, struct napi_struct *napi, + int *work_done, int budget) { struct net_device *ndev = aq_nic_get_ndev(self->aq_nic); bool is_rsc_completed = true; @@ -528,6 +699,140 @@ int aq_ring_rx_clean(struct aq_ring_s *self, return err; } +static int __aq_ring_xdp_clean(struct aq_ring_s *rx_ring, + struct napi_struct *napi, int *work_done, + int budget) +{ + struct aq_nic_s *aq_nic = rx_ring->aq_nic; + bool is_rsc_completed = true; + struct device *dev; + int err = 0; + + dev = aq_nic_get_dev(aq_nic); + + for (; (rx_ring->sw_head != rx_ring->hw_head) && budget; + rx_ring->sw_head = aq_ring_next_dx(rx_ring, rx_ring->sw_head), + --budget, ++(*work_done)) { + struct aq_ring_buff_s *buff = &rx_ring->buff_ring[rx_ring->sw_head]; + bool is_ptp_ring = aq_ptp_ring(rx_ring->aq_nic, rx_ring); + struct aq_ring_buff_s *buff_ = NULL; + struct sk_buff *skb = NULL; + unsigned int next_ = 0U; + struct xdp_buff xdp; + void *hard_start; + + if (buff->is_cleaned) + continue; + + if (!buff->is_eop) { + buff_ = buff; + do { + if (buff_->next >= rx_ring->size) { + err = -EIO; + goto err_exit; + } + next_ = buff_->next; + buff_ = &rx_ring->buff_ring[next_]; + is_rsc_completed = + aq_ring_dx_in_range(rx_ring->sw_head, + next_, + rx_ring->hw_head); + + if (unlikely(!is_rsc_completed)) + break; + + buff->is_error |= buff_->is_error; + buff->is_cso_err |= buff_->is_cso_err; + } while (!buff_->is_eop); + + if (!is_rsc_completed) { + err = 0; + goto err_exit; + } + if (buff->is_error || + (buff->is_lro && buff->is_cso_err)) { + buff_ = buff; + do { + if (buff_->next >= rx_ring->size) { + err = -EIO; + goto err_exit; + } + next_ = buff_->next; + buff_ = &rx_ring->buff_ring[next_]; + + buff_->is_cleaned = true; + } while (!buff_->is_eop); + + u64_stats_update_begin(&rx_ring->stats.rx.syncp); + ++rx_ring->stats.rx.errors; + u64_stats_update_end(&rx_ring->stats.rx.syncp); + continue; + } + } + + if (buff->is_error) { + u64_stats_update_begin(&rx_ring->stats.rx.syncp); + ++rx_ring->stats.rx.errors; + u64_stats_update_end(&rx_ring->stats.rx.syncp); + continue; + } + + dma_sync_single_range_for_cpu(dev, + buff->rxdata.daddr, + buff->rxdata.pg_off, + buff->len, DMA_FROM_DEVICE); + hard_start = page_address(buff->rxdata.page); + + if (is_ptp_ring) + buff->len -= + aq_ptp_extract_ts(rx_ring->aq_nic, skb, + aq_buf_vaddr(&buff->rxdata), + buff->len); + + /* page_order is always 0 if xdp is enabled */ + xdp_init_buff(&xdp, PAGE_SIZE, &rx_ring->xdp_rxq); + xdp_prepare_buff(&xdp, hard_start, rx_ring->page_offset, + buff->len, true); + if (!buff->is_eop) + aq_add_rx_fragment(dev, rx_ring, buff, &xdp); + + skb = aq_xdp_run_prog(aq_nic, &xdp, rx_ring, buff); + if (IS_ERR(skb) || !skb) + continue; + + if (buff->is_vlan) + __vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q), + buff->vlan_rx_tag); + + aq_rx_checksum(rx_ring, buff, skb); + + skb_set_hash(skb, buff->rss_hash, + buff->is_hash_l4 ? PKT_HASH_TYPE_L4 : + PKT_HASH_TYPE_NONE); + /* Send all PTP traffic to 0 queue */ + skb_record_rx_queue(skb, + is_ptp_ring ? 0 + : AQ_NIC_RING2QMAP(rx_ring->aq_nic, + rx_ring->idx)); + + napi_gro_receive(napi, skb); + } + +err_exit: + return err; +} + +int aq_ring_rx_clean(struct aq_ring_s *self, + struct napi_struct *napi, + int *work_done, + int budget) +{ + if (static_branch_unlikely(&aq_xdp_locking_key)) + return __aq_ring_xdp_clean(self, napi, work_done, budget); + else + return __aq_ring_rx_clean(self, napi, work_done, budget); +} + void aq_ring_hwts_rx_clean(struct aq_ring_s *self, struct aq_nic_s *aq_nic) { #if IS_REACHABLE(CONFIG_PTP_1588_CLOCK) @@ -617,6 +922,14 @@ unsigned int aq_ring_fill_stats_data(struct aq_ring_s *self, u64 *data) data[++count] = self->stats.rx.alloc_fails; data[++count] = self->stats.rx.skb_alloc_fails; data[++count] = self->stats.rx.polls; + data[++count] = self->stats.rx.pg_flips; + data[++count] = self->stats.rx.pg_reuses; + data[++count] = self->stats.rx.xdp_aborted; + data[++count] = self->stats.rx.xdp_drop; + data[++count] = self->stats.rx.xdp_pass; + data[++count] = self->stats.rx.xdp_tx; + data[++count] = self->stats.rx.xdp_invalid; + data[++count] = self->stats.rx.xdp_redirect; } while (u64_stats_fetch_retry_irq(&self->stats.rx.syncp, start)); } else { /* This data should mimic aq_ethtool_queue_tx_stat_names structure */ diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_ring.h b/drivers/net/ethernet/aquantia/atlantic/aq_ring.h index 168002657629..6a86d9cfac35 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_ring.h +++ b/drivers/net/ethernet/aquantia/atlantic/aq_ring.h @@ -105,6 +105,12 @@ struct aq_ring_stats_rx_s { u64 pg_losts; u64 pg_flips; u64 pg_reuses; + u64 xdp_aborted; + u64 xdp_drop; + u64 xdp_pass; + u64 xdp_tx; + u64 xdp_invalid; + u64 xdp_redirect; }; struct aq_ring_stats_tx_s { From patchwork Sat Mar 19 14:04:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taehee Yoo X-Patchwork-Id: 12786199 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39B98C433EF for ; Sat, 19 Mar 2022 14:05:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243100AbiCSOGc (ORCPT ); Sat, 19 Mar 2022 10:06:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42334 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243102AbiCSOG3 (ORCPT ); Sat, 19 Mar 2022 10:06:29 -0400 Received: from mail-pl1-x62e.google.com (mail-pl1-x62e.google.com [IPv6:2607:f8b0:4864:20::62e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 03AB21C3481 for ; Sat, 19 Mar 2022 07:05:07 -0700 (PDT) Received: by mail-pl1-x62e.google.com with SMTP id n15so9263224plh.2 for ; Sat, 19 Mar 2022 07:05:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=QqG6CZzxDth1w59RNVMmueDFSF6gkFVjXyxjL8FH2eI=; b=HWAChnIu0ZWewiPb3M/EdnSnZTr1cmVFJaCPIVrsqDpad+a7OAtUekAr38XtjP8eg/ Xfi3PRbV7DKr77DmOvfhHlMT5IiQH49LLVla8t6hTXJ3hTBBTTxG0PuuPs0c6b4YDxyw WuXZL8wRFXo5qI6m+BvUan5wT+3GF3TukGU/2spYarRn+rInCza/8LjiWg5dUUcrGeHH hlXZhxRn2HThPoubAfdhCKiFUxzpNOdkVz4G8pjdli5/S86gmyvwXBUr9CCyw5IkUiDm JK9JwC21Hp9NYuF2ALfV7K8E3dzbD4E6x5y4KqZmBGaRDR18Jw/zwAP61P1yghRLrkHb tFgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=QqG6CZzxDth1w59RNVMmueDFSF6gkFVjXyxjL8FH2eI=; b=BX7/9268sJ1R+O3aubsRDi5WPlak/gQ1872YNYsRpOa1nfRtQ0AGMgnXZSm94NZ4eP Apx57vPKYhZS5lTRBTQ094EHiULy+af3SPuqmDWu8AfuAY52EmS/6tuZJIQ2qgUTZjj8 SHiQ8XtzEaRiRGnE47xighjl02ztad3oaAG6ejNaty1OgORMyp6nSwf60RwX0xhQT3OM BDvyOJjjcaJbOTdV03iye6alEz+IC7jQTU/1UWlqWzyFELC/kVxiDfqB26mexsy9yLlH 5xhWmvpn+Bf5978AICMfmxH/8fmwfdfpqwoPeNKoQmmnkcxd+0Zmp9IDSPe1PJwyLDF3 4m9Q== X-Gm-Message-State: AOAM530TBsE0IQLtNiQCP+TgkU5G1hMtG9sjvyarf3H5wU867VETbqog iTMSYNw2SaJ7nvKaF7C1EvM= X-Google-Smtp-Source: ABdhPJzQQ5rmmGKu4E5ufhvT4AvlQPQMB8m6VvWIbabLBmv8ru3wjLSxusuISxFxdS7E5o7T3qJqyA== X-Received: by 2002:a17:90b:1b0f:b0:1c6:ed78:67ad with SMTP id nu15-20020a17090b1b0f00b001c6ed7867admr1092916pjb.41.1647698707399; Sat, 19 Mar 2022 07:05:07 -0700 (PDT) Received: from localhost.localdomain ([182.213.254.91]) by smtp.gmail.com with ESMTPSA id e11-20020a63e00b000000b0037341d979b8sm10168438pgh.94.2022.03.19.07.05.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Mar 2022 07:05:06 -0700 (PDT) From: Taehee Yoo To: davem@davemloft.net, kuba@kernel.org, pabeni@redhat.com, irusskikh@marvell.com, epomozov@marvell.com, netdev@vger.kernel.org Cc: ap420073@gmail.com Subject: [PATCH net-next v2 3/3] net: atlantic: Implement .ndo_xdp_xmit handler Date: Sat, 19 Mar 2022 14:04:43 +0000 Message-Id: <20220319140443.6645-4-ap420073@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220319140443.6645-1-ap420073@gmail.com> References: <20220319140443.6645-1-ap420073@gmail.com> Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org aq_xdp_xmit() is the callback function of .ndo_xdp_xmit. It internally calls aq_nic_xmit_xdpf() to send packet. Signed-off-by: Taehee Yoo --- V2: - No changed .../net/ethernet/aquantia/atlantic/aq_main.c | 1 + .../net/ethernet/aquantia/atlantic/aq_ring.c | 23 +++++++++++++++++++ .../net/ethernet/aquantia/atlantic/aq_ring.h | 2 ++ 3 files changed, 26 insertions(+) diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_main.c b/drivers/net/ethernet/aquantia/atlantic/aq_main.c index 07b4f4ee715f..6c396927438d 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_main.c +++ b/drivers/net/ethernet/aquantia/atlantic/aq_main.c @@ -480,6 +480,7 @@ static const struct net_device_ops aq_ndev_ops = { .ndo_vlan_rx_kill_vid = aq_ndo_vlan_rx_kill_vid, .ndo_setup_tc = aq_ndo_setup_tc, .ndo_bpf = aq_xdp, + .ndo_xdp_xmit = aq_xdp_xmit, }; static int __init aq_ndev_init_module(void) diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_ring.c b/drivers/net/ethernet/aquantia/atlantic/aq_ring.c index 85c3b7a9f387..5bcb999600ac 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_ring.c +++ b/drivers/net/ethernet/aquantia/atlantic/aq_ring.c @@ -409,6 +409,29 @@ static void aq_rx_checksum(struct aq_ring_s *self, __skb_incr_checksum_unnecessary(skb); } +int aq_xdp_xmit(struct net_device *dev, int num_frames, + struct xdp_frame **frames, u32 flags) +{ + struct aq_nic_s *aq_nic = netdev_priv(dev); + unsigned int vec, i, drop = 0; + int cpu = smp_processor_id(); + struct aq_nic_cfg_s *aq_cfg; + struct aq_ring_s *ring; + + aq_cfg = aq_nic_get_cfg(aq_nic); + vec = cpu % aq_cfg->vecs; + ring = aq_nic->aq_ring_tx[AQ_NIC_CFG_TCVEC2RING(aq_cfg, 0, vec)]; + + for (i = 0; i < num_frames; i++) { + struct xdp_frame *xdpf = frames[i]; + + if (aq_nic_xmit_xdpf(aq_nic, ring, xdpf) == NETDEV_TX_BUSY) + drop++; + } + + return num_frames - drop; +} + static struct sk_buff *aq_xdp_run_prog(struct aq_nic_s *aq_nic, struct xdp_buff *xdp, struct aq_ring_s *rx_ring, diff --git a/drivers/net/ethernet/aquantia/atlantic/aq_ring.h b/drivers/net/ethernet/aquantia/atlantic/aq_ring.h index 6a86d9cfac35..31e19dff6c1e 100644 --- a/drivers/net/ethernet/aquantia/atlantic/aq_ring.h +++ b/drivers/net/ethernet/aquantia/atlantic/aq_ring.h @@ -197,6 +197,8 @@ void aq_ring_update_queue_state(struct aq_ring_s *ring); void aq_ring_queue_wake(struct aq_ring_s *ring); void aq_ring_queue_stop(struct aq_ring_s *ring); bool aq_ring_tx_clean(struct aq_ring_s *self); +int aq_xdp_xmit(struct net_device *dev, int num_frames, + struct xdp_frame **frames, u32 flags); int aq_ring_rx_clean(struct aq_ring_s *self, struct napi_struct *napi, int *work_done,