From patchwork Fri Jul 22 07:19:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 12926078 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 29DF3C43334 for ; Fri, 22 Jul 2022 07:20:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=aO6QjN+ndztRcwhQDrZL8Z787GRhvB6e9GtAEigGe0Y=; b=0IOeaNlyMnzk45sLubK+Xm69D1 YLEh6ouSIBfXlVvgAzSCePa/NcOqXg0QpsBYEcD+7ZTwthyBQ63V43x+MBZ3K3j7c5UbNx8/BiuJM E5BNNRCTY7V1lYbZ5rWwE+nXX9moHRneWS9eqHqmlclA7kwckJtnLGEkEJsgZ9fMxUKl/H9SgsvDe 8WrgneJmGfxkGkPqWC3MwxrNw0qA9w9FFDVkxn1SGGj+NVWP7Xt4U4/UA0nWAtNOtueEecdWPlnd+ gG8IvPb+LdLQbOXNFu+R2+BS4Xa5dE5lcha+Bsgr0+SaSHXpYHV5Q6FHdh7UGqXmrPvY9PAT5xIaU KZVlmdEA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oEmxo-000b53-U9; Fri, 22 Jul 2022 07:20:44 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oEmxl-000b2b-Fw for linux-mediatek@lists.infradead.org; Fri, 22 Jul 2022 07:20:43 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 1369D621A9; Fri, 22 Jul 2022 07:20:41 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 205F2C341C6; Fri, 22 Jul 2022 07:20:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1658474440; bh=35yzR2p2Q3Ck4oS7liLj3pEuLgLDqJpiPpQIVlQLrLY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=m5qIZJBZwItkf18rGncBI7yklmubDBolbXHBHbThrWNAUccKTbcSsdbILgUXaYfSs te70xD698tOlwtfzQx0nXvT36ebFWpPnQYuCcB/qnezLaQJaXkYCGlSttPJc1mXUiW lfPGHWhpRn8pkD7SqG8yg/EbGp3Du0aZMrD53GZP2juL1PrbWpOReWK2mic56XvNMt hTmCRR5lSljwGjsRrM9tt2ype2wVly5zyw+j0JPR/8uz6HCqt+CuOEfd9IT+MDResI px0nGgfJ7CTl79ykzTCk8D38Kfwb3WWbltYag1t8UcU0v+uzAt4ERegvILZo+muBCx wIn3pC5MX69TQ== From: Lorenzo Bianconi To: netdev@vger.kernel.org Cc: nbd@nbd.name, john@phrozen.org, sean.wang@mediatek.com, Mark-MC.Lee@mediatek.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, matthias.bgg@gmail.com, linux-mediatek@lists.infradead.org, ilias.apalodimas@linaro.org, lorenzo.bianconi@redhat.com, jbrouer@redhat.com Subject: [PATCH v4 net-next 1/5] net: ethernet: mtk_eth_soc: rely on page_pool for single page buffers Date: Fri, 22 Jul 2022 09:19:36 +0200 Message-Id: <77d76ef9208769f4b7d550378bb841f446083334.1658474059.git.lorenzo@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220722_002041_627184_47AB6F7F X-CRM114-Status: GOOD ( 23.03 ) X-BeenThere: linux-mediatek@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Linux-mediatek" Errors-To: linux-mediatek-bounces+linux-mediatek=archiver.kernel.org@lists.infradead.org Rely on page_pool allocator for single page buffers in order to keep them dma mapped and add skb recycling support. Signed-off-by: Lorenzo Bianconi --- drivers/net/ethernet/mediatek/Kconfig | 1 + drivers/net/ethernet/mediatek/mtk_eth_soc.c | 185 +++++++++++++++----- drivers/net/ethernet/mediatek/mtk_eth_soc.h | 10 ++ 3 files changed, 156 insertions(+), 40 deletions(-) diff --git a/drivers/net/ethernet/mediatek/Kconfig b/drivers/net/ethernet/mediatek/Kconfig index da4ec235d146..d2422c7b31b0 100644 --- a/drivers/net/ethernet/mediatek/Kconfig +++ b/drivers/net/ethernet/mediatek/Kconfig @@ -17,6 +17,7 @@ config NET_MEDIATEK_SOC select PINCTRL select PHYLINK select DIMLIB + select PAGE_POOL help This driver supports the gigabit ethernet MACs in the MediaTek SoC family. diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.c b/drivers/net/ethernet/mediatek/mtk_eth_soc.c index 6beb3d4873a3..9a92d602ebd5 100644 --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c @@ -1432,6 +1432,68 @@ static void mtk_update_rx_cpu_idx(struct mtk_eth *eth) } } +static struct page_pool *mtk_create_page_pool(struct mtk_eth *eth, + struct xdp_rxq_info *xdp_q, + int id, int size) +{ + struct page_pool_params pp_params = { + .order = 0, + .flags = PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV, + .pool_size = size, + .nid = NUMA_NO_NODE, + .dev = eth->dma_dev, + .dma_dir = DMA_FROM_DEVICE, + .offset = MTK_PP_HEADROOM, + .max_len = MTK_PP_MAX_BUF_SIZE, + }; + struct page_pool *pp; + int err; + + pp = page_pool_create(&pp_params); + if (IS_ERR(pp)) + return pp; + + err = __xdp_rxq_info_reg(xdp_q, ð->dummy_dev, eth->rx_napi.napi_id, + id, PAGE_SIZE); + if (err < 0) + goto err_free_pp; + + err = xdp_rxq_info_reg_mem_model(xdp_q, MEM_TYPE_PAGE_POOL, pp); + if (err) + goto err_unregister_rxq; + + return pp; + +err_unregister_rxq: + xdp_rxq_info_unreg(xdp_q); +err_free_pp: + page_pool_destroy(pp); + + return ERR_PTR(err); +} + +static void *mtk_page_pool_get_buff(struct page_pool *pp, dma_addr_t *dma_addr, + gfp_t gfp_mask) +{ + struct page *page; + + page = page_pool_alloc_pages(pp, gfp_mask | __GFP_NOWARN); + if (!page) + return NULL; + + *dma_addr = page_pool_get_dma_addr(page) + MTK_PP_HEADROOM; + return page_address(page); +} + +static void mtk_rx_put_buff(struct mtk_rx_ring *ring, void *data, bool napi) +{ + if (ring->page_pool) + page_pool_put_full_page(ring->page_pool, + virt_to_head_page(data), napi); + else + skb_free_frag(data); +} + static int mtk_poll_rx(struct napi_struct *napi, int budget, struct mtk_eth *eth) { @@ -1445,9 +1507,9 @@ static int mtk_poll_rx(struct napi_struct *napi, int budget, while (done < budget) { unsigned int pktlen, *rxdcsum; + u32 hash, reason, reserve_len; struct net_device *netdev; dma_addr_t dma_addr; - u32 hash, reason; int mac = 0; ring = mtk_get_rx_ring(eth); @@ -1478,36 +1540,54 @@ static int mtk_poll_rx(struct napi_struct *napi, int budget, goto release_desc; /* alloc new buffer */ - if (ring->frag_size <= PAGE_SIZE) - new_data = napi_alloc_frag(ring->frag_size); - else - new_data = mtk_max_lro_buf_alloc(GFP_ATOMIC); - if (unlikely(!new_data)) { - netdev->stats.rx_dropped++; - goto release_desc; - } - dma_addr = dma_map_single(eth->dma_dev, - new_data + NET_SKB_PAD + - eth->ip_align, - ring->buf_size, - DMA_FROM_DEVICE); - if (unlikely(dma_mapping_error(eth->dma_dev, dma_addr))) { - skb_free_frag(new_data); - netdev->stats.rx_dropped++; - goto release_desc; - } + if (ring->page_pool) { + new_data = mtk_page_pool_get_buff(ring->page_pool, + &dma_addr, + GFP_ATOMIC); + if (unlikely(!new_data)) { + netdev->stats.rx_dropped++; + goto release_desc; + } + } else { + if (ring->frag_size <= PAGE_SIZE) + new_data = napi_alloc_frag(ring->frag_size); + else + new_data = mtk_max_lro_buf_alloc(GFP_ATOMIC); + + if (unlikely(!new_data)) { + netdev->stats.rx_dropped++; + goto release_desc; + } - dma_unmap_single(eth->dma_dev, trxd.rxd1, - ring->buf_size, DMA_FROM_DEVICE); + dma_addr = dma_map_single(eth->dma_dev, + new_data + NET_SKB_PAD + eth->ip_align, + ring->buf_size, DMA_FROM_DEVICE); + if (unlikely(dma_mapping_error(eth->dma_dev, + dma_addr))) { + skb_free_frag(new_data); + netdev->stats.rx_dropped++; + goto release_desc; + } + + dma_unmap_single(eth->dma_dev, trxd.rxd1, + ring->buf_size, DMA_FROM_DEVICE); + } /* receive data */ skb = build_skb(data, ring->frag_size); if (unlikely(!skb)) { - skb_free_frag(data); + mtk_rx_put_buff(ring, data, true); netdev->stats.rx_dropped++; goto skip_rx; } - skb_reserve(skb, NET_SKB_PAD + NET_IP_ALIGN); + + if (ring->page_pool) { + reserve_len = MTK_PP_HEADROOM; + skb_mark_for_recycle(skb); + } else { + reserve_len = NET_SKB_PAD + NET_IP_ALIGN; + } + skb_reserve(skb, reserve_len); pktlen = RX_DMA_GET_PLEN0(trxd.rxd2); skb->dev = netdev; @@ -1561,7 +1641,6 @@ static int mtk_poll_rx(struct napi_struct *napi, int budget, skip_rx: ring->data[idx] = new_data; rxd->rxd1 = (unsigned int)dma_addr; - release_desc: if (MTK_HAS_CAPS(eth->soc->caps, MTK_SOC_MT7628)) rxd->rxd2 = RX_DMA_LSO; @@ -1569,7 +1648,6 @@ static int mtk_poll_rx(struct napi_struct *napi, int budget, rxd->rxd2 = RX_DMA_PREP_PLEN0(ring->buf_size); ring->calc_idx = idx; - done++; } @@ -1933,13 +2011,15 @@ static int mtk_rx_alloc(struct mtk_eth *eth, int ring_no, int rx_flag) if (!ring->data) return -ENOMEM; - for (i = 0; i < rx_dma_size; i++) { - if (ring->frag_size <= PAGE_SIZE) - ring->data[i] = netdev_alloc_frag(ring->frag_size); - else - ring->data[i] = mtk_max_lro_buf_alloc(GFP_KERNEL); - if (!ring->data[i]) - return -ENOMEM; + if (!eth->hwlro) { + struct page_pool *pp; + + pp = mtk_create_page_pool(eth, &ring->xdp_q, ring_no, + rx_dma_size); + if (IS_ERR(pp)) + return PTR_ERR(pp); + + ring->page_pool = pp; } ring->dma = dma_alloc_coherent(eth->dma_dev, @@ -1950,16 +2030,33 @@ static int mtk_rx_alloc(struct mtk_eth *eth, int ring_no, int rx_flag) for (i = 0; i < rx_dma_size; i++) { struct mtk_rx_dma_v2 *rxd; - - dma_addr_t dma_addr = dma_map_single(eth->dma_dev, - ring->data[i] + NET_SKB_PAD + eth->ip_align, - ring->buf_size, - DMA_FROM_DEVICE); - if (unlikely(dma_mapping_error(eth->dma_dev, dma_addr))) - return -ENOMEM; + dma_addr_t dma_addr; + void *data; rxd = ring->dma + i * eth->soc->txrx.rxd_size; + if (ring->page_pool) { + data = mtk_page_pool_get_buff(ring->page_pool, + &dma_addr, GFP_KERNEL); + if (!data) + return -ENOMEM; + } else { + if (ring->frag_size <= PAGE_SIZE) + data = netdev_alloc_frag(ring->frag_size); + else + data = mtk_max_lro_buf_alloc(GFP_KERNEL); + + if (!data) + return -ENOMEM; + + dma_addr = dma_map_single(eth->dma_dev, + data + NET_SKB_PAD + eth->ip_align, + ring->buf_size, DMA_FROM_DEVICE); + if (unlikely(dma_mapping_error(eth->dma_dev, + dma_addr))) + return -ENOMEM; + } rxd->rxd1 = (unsigned int)dma_addr; + ring->data[i] = data; if (MTK_HAS_CAPS(eth->soc->caps, MTK_SOC_MT7628)) rxd->rxd2 = RX_DMA_LSO; @@ -1975,6 +2072,7 @@ static int mtk_rx_alloc(struct mtk_eth *eth, int ring_no, int rx_flag) rxd->rxd8 = 0; } } + ring->dma_size = rx_dma_size; ring->calc_idx_update = false; ring->calc_idx = rx_dma_size - 1; @@ -2026,7 +2124,7 @@ static void mtk_rx_clean(struct mtk_eth *eth, struct mtk_rx_ring *ring) dma_unmap_single(eth->dma_dev, rxd->rxd1, ring->buf_size, DMA_FROM_DEVICE); - skb_free_frag(ring->data[i]); + mtk_rx_put_buff(ring, ring->data[i], false); } kfree(ring->data); ring->data = NULL; @@ -2038,6 +2136,13 @@ static void mtk_rx_clean(struct mtk_eth *eth, struct mtk_rx_ring *ring) ring->dma, ring->phys); ring->dma = NULL; } + + if (ring->page_pool) { + if (xdp_rxq_info_is_reg(&ring->xdp_q)) + xdp_rxq_info_unreg(&ring->xdp_q); + page_pool_destroy(ring->page_pool); + ring->page_pool = NULL; + } } static int mtk_hwlro_rx_init(struct mtk_eth *eth) diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.h b/drivers/net/ethernet/mediatek/mtk_eth_soc.h index 0a632896451a..26c019319055 100644 --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.h +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.h @@ -18,6 +18,8 @@ #include #include #include +#include +#include #include "mtk_ppe.h" #define MTK_QDMA_PAGE_SIZE 2048 @@ -49,6 +51,11 @@ #define MTK_HW_FEATURES_MT7628 (NETIF_F_SG | NETIF_F_RXCSUM) #define NEXT_DESP_IDX(X, Y) (((X) + 1) & ((Y) - 1)) +#define MTK_PP_HEADROOM XDP_PACKET_HEADROOM +#define MTK_PP_PAD (MTK_PP_HEADROOM + \ + SKB_DATA_ALIGN(sizeof(struct skb_shared_info))) +#define MTK_PP_MAX_BUF_SIZE (PAGE_SIZE - MTK_PP_PAD) + #define MTK_QRX_OFFSET 0x10 #define MTK_MAX_RX_RING_NUM 4 @@ -745,6 +752,9 @@ struct mtk_rx_ring { bool calc_idx_update; u16 calc_idx; u32 crx_idx_reg; + /* page_pool */ + struct page_pool *page_pool; + struct xdp_rxq_info xdp_q; }; enum mkt_eth_capabilities { From patchwork Fri Jul 22 07:19:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 12926079 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 442F3C433EF for ; Fri, 22 Jul 2022 07:21:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=XB81VnaoXikp+oivrMS6UcyCLGWclDmKXLHIzIH744E=; b=HNXYZTb+3SiFVX/mnC2XjQ3f2f +v6JK5uOU0VlEkSpZu8BxSGdaq9IMD6I35Ogw4+ZHce0Tki+yYQnGy/nWPlI5bzAvhxzCXKq/oGgh T0Y0NXsCMk89s/QcKM6rFv6J0lEFV6Ycic/G3+aAjMaIzHvNFYiUoiA4mfdVvZEJs68Bd++nns1h3 Zv3nvPDJAYBpVfp43b/3z0c/9sTCMOCS4l6lxlGhxClUPqRiO2S9VHko4mwdSR0hI8533N7QnUG1R fVzzJ6qwwWRXPRwEKHu7eg2WwL4546ycIuNSIasoh1iTXNG9eXBSG24+F2flYwNYePsDTstiymNSz /nxNsQHg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oEmyK-000bIM-Po; Fri, 22 Jul 2022 07:21:16 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oEmyI-000bHH-Ap for linux-mediatek@lists.infradead.org; Fri, 22 Jul 2022 07:21:16 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9BB46621C3; Fri, 22 Jul 2022 07:21:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7AB92C341C6; Fri, 22 Jul 2022 07:21:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1658474473; bh=kpLqIwyhx5Mkngpodf9zhchYBbDCqagoWs8dhA4A2UU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VwtxPD0WjLQozT2GUl6DvrxjjBvAsphO7xo1IKdYqf9+7MIO8RkaIXk54TBTHot8X tEDO4wAFm5A4aqIUSw2NWDAeaTjDmy3GWeNxjjpDir+VFyeCA4dH5HGGU+JpsXoqUo ByqHTzvh3gaKNESTcxCohM7AiTT6lgCm7ag9EXlH6Dj+SKQemmFmhnjL0NZkzsHf5J tSnjkP7dueTtfjDY3ZIcDRU9Vl6MamwQdvOTP4E+j6r3wRECq9Nk+k/V8ChYRaR32m kJOMe9E5qKy4Tm17Hrg6etvOQHf68gHRNcqd/ekCDCao4kFfm0zQPVrNXInGeiXcWl ZdFNDD95/HBVw== From: Lorenzo Bianconi To: netdev@vger.kernel.org Cc: nbd@nbd.name, john@phrozen.org, sean.wang@mediatek.com, Mark-MC.Lee@mediatek.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, matthias.bgg@gmail.com, linux-mediatek@lists.infradead.org, ilias.apalodimas@linaro.org, lorenzo.bianconi@redhat.com, jbrouer@redhat.com Subject: [PATCH v4 net-next 2/5] net: ethernet: mtk_eth_soc: add basic XDP support Date: Fri, 22 Jul 2022 09:19:37 +0200 Message-Id: <280b4e0dd175840dcfba683f01c461685e4692f3.1658474059.git.lorenzo@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220722_002114_477312_50D36AFB X-CRM114-Status: GOOD ( 19.47 ) X-BeenThere: linux-mediatek@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Linux-mediatek" Errors-To: linux-mediatek-bounces+linux-mediatek=archiver.kernel.org@lists.infradead.org Introduce basic XDP support to mtk_eth_soc driver. Supported XDP verdicts: - XDP_PASS - XDP_DROP - XDP_REDIRECT Signed-off-by: Lorenzo Bianconi --- drivers/net/ethernet/mediatek/mtk_eth_soc.c | 162 +++++++++++++++++--- drivers/net/ethernet/mediatek/mtk_eth_soc.h | 2 + 2 files changed, 145 insertions(+), 19 deletions(-) diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.c b/drivers/net/ethernet/mediatek/mtk_eth_soc.c index 9a92d602ebd5..9a40876e5417 100644 --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c @@ -1432,6 +1432,11 @@ static void mtk_update_rx_cpu_idx(struct mtk_eth *eth) } } +static bool mtk_page_pool_enabled(struct mtk_eth *eth) +{ + return !eth->hwlro; +} + static struct page_pool *mtk_create_page_pool(struct mtk_eth *eth, struct xdp_rxq_info *xdp_q, int id, int size) @@ -1494,11 +1499,52 @@ static void mtk_rx_put_buff(struct mtk_rx_ring *ring, void *data, bool napi) skb_free_frag(data); } +static u32 mtk_xdp_run(struct mtk_eth *eth, struct mtk_rx_ring *ring, + struct xdp_buff *xdp, struct net_device *dev) +{ + struct bpf_prog *prog; + u32 act = XDP_PASS; + + rcu_read_lock(); + + prog = rcu_dereference(eth->prog); + if (!prog) + goto out; + + act = bpf_prog_run_xdp(prog, xdp); + switch (act) { + case XDP_PASS: + goto out; + case XDP_REDIRECT: + if (unlikely(xdp_do_redirect(dev, xdp, prog))) { + act = XDP_DROP; + break; + } + goto out; + default: + bpf_warn_invalid_xdp_action(dev, prog, act); + fallthrough; + case XDP_ABORTED: + trace_xdp_exception(dev, prog, act); + fallthrough; + case XDP_DROP: + break; + } + + page_pool_put_full_page(ring->page_pool, + virt_to_head_page(xdp->data), true); +out: + rcu_read_unlock(); + + return act; +} + static int mtk_poll_rx(struct napi_struct *napi, int budget, struct mtk_eth *eth) { struct dim_sample dim_sample = {}; struct mtk_rx_ring *ring; + bool xdp_flush = false; int idx; struct sk_buff *skb; u8 *data, *new_data; @@ -1507,9 +1553,9 @@ static int mtk_poll_rx(struct napi_struct *napi, int budget, while (done < budget) { unsigned int pktlen, *rxdcsum; - u32 hash, reason, reserve_len; struct net_device *netdev; dma_addr_t dma_addr; + u32 hash, reason; int mac = 0; ring = mtk_get_rx_ring(eth); @@ -1539,8 +1585,14 @@ static int mtk_poll_rx(struct napi_struct *napi, int budget, if (unlikely(test_bit(MTK_RESETTING, ð->state))) goto release_desc; + pktlen = RX_DMA_GET_PLEN0(trxd.rxd2); + /* alloc new buffer */ if (ring->page_pool) { + struct page *page = virt_to_head_page(data); + struct xdp_buff xdp; + u32 ret; + new_data = mtk_page_pool_get_buff(ring->page_pool, &dma_addr, GFP_ATOMIC); @@ -1548,6 +1600,34 @@ static int mtk_poll_rx(struct napi_struct *napi, int budget, netdev->stats.rx_dropped++; goto release_desc; } + + dma_sync_single_for_cpu(eth->dma_dev, + page_pool_get_dma_addr(page) + MTK_PP_HEADROOM, + pktlen, page_pool_get_dma_dir(ring->page_pool)); + + xdp_init_buff(&xdp, PAGE_SIZE, &ring->xdp_q); + xdp_prepare_buff(&xdp, data, MTK_PP_HEADROOM, pktlen, + false); + xdp_buff_clear_frags_flag(&xdp); + + ret = mtk_xdp_run(eth, ring, &xdp, netdev); + if (ret == XDP_REDIRECT) + xdp_flush = true; + + if (ret != XDP_PASS) + goto skip_rx; + + skb = build_skb(data, PAGE_SIZE); + if (unlikely(!skb)) { + page_pool_put_full_page(ring->page_pool, + page, true); + netdev->stats.rx_dropped++; + goto skip_rx; + } + + skb_reserve(skb, xdp.data - xdp.data_hard_start); + skb_put(skb, xdp.data_end - xdp.data); + skb_mark_for_recycle(skb); } else { if (ring->frag_size <= PAGE_SIZE) new_data = napi_alloc_frag(ring->frag_size); @@ -1571,27 +1651,20 @@ static int mtk_poll_rx(struct napi_struct *napi, int budget, dma_unmap_single(eth->dma_dev, trxd.rxd1, ring->buf_size, DMA_FROM_DEVICE); - } - /* receive data */ - skb = build_skb(data, ring->frag_size); - if (unlikely(!skb)) { - mtk_rx_put_buff(ring, data, true); - netdev->stats.rx_dropped++; - goto skip_rx; - } + skb = build_skb(data, ring->frag_size); + if (unlikely(!skb)) { + netdev->stats.rx_dropped++; + skb_free_frag(data); + goto skip_rx; + } - if (ring->page_pool) { - reserve_len = MTK_PP_HEADROOM; - skb_mark_for_recycle(skb); - } else { - reserve_len = NET_SKB_PAD + NET_IP_ALIGN; + skb_reserve(skb, NET_SKB_PAD + NET_IP_ALIGN); + skb_put(skb, pktlen); } - skb_reserve(skb, reserve_len); - pktlen = RX_DMA_GET_PLEN0(trxd.rxd2); skb->dev = netdev; - skb_put(skb, pktlen); + bytes += skb->len; if (MTK_HAS_CAPS(eth->soc->caps, MTK_NETSYS_V2)) rxdcsum = &trxd.rxd3; @@ -1603,7 +1676,6 @@ static int mtk_poll_rx(struct napi_struct *napi, int budget, else skb_checksum_none_assert(skb); skb->protocol = eth_type_trans(skb, netdev); - bytes += pktlen; hash = trxd.rxd4 & MTK_RXD4_FOE_ENTRY; if (hash != MTK_RXD4_FOE_ENTRY) { @@ -1666,6 +1738,9 @@ static int mtk_poll_rx(struct napi_struct *napi, int budget, &dim_sample); net_dim(ð->rx_dim, dim_sample); + if (xdp_flush) + xdp_do_flush_map(); + return done; } @@ -2011,7 +2086,7 @@ static int mtk_rx_alloc(struct mtk_eth *eth, int ring_no, int rx_flag) if (!ring->data) return -ENOMEM; - if (!eth->hwlro) { + if (mtk_page_pool_enabled(eth)) { struct page_pool *pp; pp = mtk_create_page_pool(eth, &ring->xdp_q, ring_no, @@ -2750,6 +2825,48 @@ static int mtk_stop(struct net_device *dev) return 0; } +static int mtk_xdp_setup(struct net_device *dev, struct bpf_prog *prog, + struct netlink_ext_ack *extack) +{ + struct mtk_mac *mac = netdev_priv(dev); + struct mtk_eth *eth = mac->hw; + struct bpf_prog *old_prog; + bool need_update; + + if (eth->hwlro) { + NL_SET_ERR_MSG_MOD(extack, "XDP not supported with HWLRO"); + return -EOPNOTSUPP; + } + + if (dev->mtu > MTK_PP_MAX_BUF_SIZE) { + NL_SET_ERR_MSG_MOD(extack, "MTU too large for XDP"); + return -EOPNOTSUPP; + } + + need_update = !!eth->prog != !!prog; + if (netif_running(dev) && need_update) + mtk_stop(dev); + + old_prog = rcu_replace_pointer(eth->prog, prog, lockdep_rtnl_is_held()); + if (old_prog) + bpf_prog_put(old_prog); + + if (netif_running(dev) && need_update) + return mtk_open(dev); + + return 0; +} + +static int mtk_xdp(struct net_device *dev, struct netdev_bpf *xdp) +{ + switch (xdp->command) { + case XDP_SETUP_PROG: + return mtk_xdp_setup(dev, xdp->prog, xdp->extack); + default: + return -EINVAL; + } +} + static void ethsys_reset(struct mtk_eth *eth, u32 reset_bits) { regmap_update_bits(eth->ethsys, ETHSYS_RSTCTRL, @@ -3045,6 +3162,12 @@ static int mtk_change_mtu(struct net_device *dev, int new_mtu) struct mtk_eth *eth = mac->hw; u32 mcr_cur, mcr_new; + if (rcu_access_pointer(eth->prog) && + length > MTK_PP_MAX_BUF_SIZE) { + netdev_err(dev, "Invalid MTU for XDP mode\n"); + return -EINVAL; + } + if (!MTK_HAS_CAPS(eth->soc->caps, MTK_SOC_MT7628)) { mcr_cur = mtk_r32(mac->hw, MTK_MAC_MCR(mac->id)); mcr_new = mcr_cur & ~MAC_MCR_MAX_RX_MASK; @@ -3372,6 +3495,7 @@ static const struct net_device_ops mtk_netdev_ops = { .ndo_poll_controller = mtk_poll_controller, #endif .ndo_setup_tc = mtk_eth_setup_tc, + .ndo_bpf = mtk_xdp, }; static int mtk_add_mac(struct mtk_eth *eth, struct device_node *np) diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.h b/drivers/net/ethernet/mediatek/mtk_eth_soc.h index 26c019319055..cfb7aeda3f49 100644 --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.h +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.h @@ -1088,6 +1088,8 @@ struct mtk_eth { struct mtk_ppe *ppe; struct rhashtable flow_table; + + struct bpf_prog __rcu *prog; }; /* struct mtk_mac - the structure that holds the info about the MACs of the From patchwork Fri Jul 22 07:19:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 12926080 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 26133C43334 for ; Fri, 22 Jul 2022 07:22:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Tfuh7WGJZuExOKqsNNWvz8zpqvnlZLqe4Bms5iwGffM=; b=t8oDBngi6nohRBh9Ef/gQg2BUt Th48Rv5yVHFDbqEy7L+B0rWPzU9ntt8ZDb0qq1bVsBim+on/4xbgpJr4R2lr45XKbTXGeF8VKV6tO nT3t8MRBj71APW8sp3UBdLlgA1H6hkx3b1+IHdU7pBXdmbQF68ihvAQbYf5R+28NhGvWfkFIRBgCH AgMzcihBAKMrdm3LPsF4Wu0oWQ3OifpKTct0ueSB1cZmf4B7MI8Bn6WNu850vFhZxSRc06XjBtuca qC6UzEZ2LarO+6HicJjQh9OxZNKQLc07icJ56t5hQP8JU3xkVCJ+YLQIZLX/r6pZ1FRZcxlreO0vw SmWlDCOg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oEmyv-000bZl-Mo; Fri, 22 Jul 2022 07:21:53 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oEmyt-000bYW-6w for linux-mediatek@lists.infradead.org; Fri, 22 Jul 2022 07:21:52 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id BE294621C4; Fri, 22 Jul 2022 07:21:50 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C903DC341C6; Fri, 22 Jul 2022 07:21:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1658474510; bh=gs3Y7OdFr1SCPPKNy/wWytrZ55M3gULs+vke01PyqXw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bbgE+UFPqqmimPs729cTMlBgfJT3YG7OolXl8m9iPmCYPWkYp/u8rY4bFsyAO/hap 7jgkvRVwYatw7fdYDAYGG/VIs7k91f9enJEyFNrqyxCHKZlHbRAvUi7gudAU0jCEtY lLu8UQjUx6TCqHuX267meYdmjGdwo5hLwnlBylse1N6I4r3gFdbKF/45M63rIpv2p9 5EwkApNczQvvi9jzpKq7+ZVmBT/6IzUaZYjy164P6A87EsHm0rIXSlA8osFAbohDvU +MBu5bMBtZ/XEmzePEzNN7WFnPWVxDNvdbf2hd89wBwXzfnimuaze7kRONrLt1OV7u TTRPvrQPsoCqg== From: Lorenzo Bianconi To: netdev@vger.kernel.org Cc: nbd@nbd.name, john@phrozen.org, sean.wang@mediatek.com, Mark-MC.Lee@mediatek.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, matthias.bgg@gmail.com, linux-mediatek@lists.infradead.org, ilias.apalodimas@linaro.org, lorenzo.bianconi@redhat.com, jbrouer@redhat.com Subject: [PATCH v4 net-next 3/5] net: ethernet: mtk_eth_soc: introduce xdp ethtool counters Date: Fri, 22 Jul 2022 09:19:38 +0200 Message-Id: <44260c788c793aa6ca63d87eac1b7d0938eb949a.1658474059.git.lorenzo@kernel.org> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220722_002151_351224_F6F35FA1 X-CRM114-Status: GOOD ( 13.88 ) X-BeenThere: linux-mediatek@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Linux-mediatek" Errors-To: linux-mediatek-bounces+linux-mediatek=archiver.kernel.org@lists.infradead.org Report xdp stats through ethtool Signed-off-by: Lorenzo Bianconi --- drivers/net/ethernet/mediatek/mtk_eth_soc.c | 26 +++++++++++++++++++-- drivers/net/ethernet/mediatek/mtk_eth_soc.h | 12 ++++++++++ 2 files changed, 36 insertions(+), 2 deletions(-) diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.c b/drivers/net/ethernet/mediatek/mtk_eth_soc.c index 9a40876e5417..cd96eaf72b6b 100644 --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c @@ -34,6 +34,10 @@ MODULE_PARM_DESC(msg_level, "Message level (-1=defaults,0=none,...,16=all)"); #define MTK_ETHTOOL_STAT(x) { #x, \ offsetof(struct mtk_hw_stats, x) / sizeof(u64) } +#define MTK_ETHTOOL_XDP_STAT(x) { #x, \ + offsetof(struct mtk_hw_stats, xdp_stats.x) / \ + sizeof(u64) } + static const struct mtk_reg_map mtk_reg_map = { .tx_irq_mask = 0x1a1c, .tx_irq_status = 0x1a18, @@ -141,6 +145,13 @@ static const struct mtk_ethtool_stats { MTK_ETHTOOL_STAT(rx_long_errors), MTK_ETHTOOL_STAT(rx_checksum_errors), MTK_ETHTOOL_STAT(rx_flow_control_packets), + MTK_ETHTOOL_XDP_STAT(rx_xdp_redirect), + MTK_ETHTOOL_XDP_STAT(rx_xdp_pass), + MTK_ETHTOOL_XDP_STAT(rx_xdp_drop), + MTK_ETHTOOL_XDP_STAT(rx_xdp_tx), + MTK_ETHTOOL_XDP_STAT(rx_xdp_tx_errors), + MTK_ETHTOOL_XDP_STAT(tx_xdp_xmit), + MTK_ETHTOOL_XDP_STAT(tx_xdp_xmit_errors), }; static const char * const mtk_clks_source_name[] = { @@ -1502,6 +1513,9 @@ static void mtk_rx_put_buff(struct mtk_rx_ring *ring, void *data, bool napi) static u32 mtk_xdp_run(struct mtk_eth *eth, struct mtk_rx_ring *ring, struct xdp_buff *xdp, struct net_device *dev) { + struct mtk_mac *mac = netdev_priv(dev); + struct mtk_hw_stats *hw_stats = mac->hw_stats; + u64 *count = &hw_stats->xdp_stats.rx_xdp_drop; struct bpf_prog *prog; u32 act = XDP_PASS; @@ -1514,13 +1528,16 @@ static u32 mtk_xdp_run(struct mtk_eth *eth, struct mtk_rx_ring *ring, act = bpf_prog_run_xdp(prog, xdp); switch (act) { case XDP_PASS: - goto out; + count = &hw_stats->xdp_stats.rx_xdp_pass; + goto update_stats; case XDP_REDIRECT: if (unlikely(xdp_do_redirect(dev, xdp, prog))) { act = XDP_DROP; break; } - goto out; + + count = &hw_stats->xdp_stats.rx_xdp_redirect; + goto update_stats; default: bpf_warn_invalid_xdp_action(dev, prog, act); fallthrough; @@ -1533,6 +1550,11 @@ static u32 mtk_xdp_run(struct mtk_eth *eth, struct mtk_rx_ring *ring, page_pool_put_full_page(ring->page_pool, virt_to_head_page(xdp->data), true); + +update_stats: + u64_stats_update_begin(&hw_stats->syncp); + *count = *count + 1; + u64_stats_update_end(&hw_stats->syncp); out: rcu_read_unlock(); diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.h b/drivers/net/ethernet/mediatek/mtk_eth_soc.h index cfb7aeda3f49..2775da1a6ec3 100644 --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.h +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.h @@ -570,6 +570,16 @@ struct mtk_tx_dma_v2 { struct mtk_eth; struct mtk_mac; +struct mtk_xdp_stats { + u64 rx_xdp_redirect; + u64 rx_xdp_pass; + u64 rx_xdp_drop; + u64 rx_xdp_tx; + u64 rx_xdp_tx_errors; + u64 tx_xdp_xmit; + u64 tx_xdp_xmit_errors; +}; + /* struct mtk_hw_stats - the structure that holds the traffic statistics. * @stats_lock: make sure that stats operations are atomic * @reg_offset: the status register offset of the SoC @@ -593,6 +603,8 @@ struct mtk_hw_stats { u64 rx_checksum_errors; u64 rx_flow_control_packets; + struct mtk_xdp_stats xdp_stats; + spinlock_t stats_lock; u32 reg_offset; struct u64_stats_sync syncp; From patchwork Fri Jul 22 07:19:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 12926081 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 546F7C43334 for ; Fri, 22 Jul 2022 07:22:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=RxZNUArlsTH4UDHywLjwCtzPUXUVx/21gl1KljGqM48=; b=M7HE6xLaf38jF4NTOxogtzUyzr gkArcJb/WtyjqmozMUE9K88YgAsN5NX/ggDQDciAf7vThLvDiFl72dN9gBCIUpB7us8gj1pIsFmxk RQ6qi0oUJrDnw4kPHJVj9P5LE7746c7+O7lrSutaGzkcpDUgkAy23S5tUJNiWdQqvZLOJdvXlSkfa QYFuBZdnZ+8yWtCprnE6XDEkoaY35r9DZ92xJcwnlbXAjDQBE61+PBwvcmR1fwUIgVi+va7TP017S Zl+VGZR3SqDfVu3oX2TL8g7WwkFxq3vkhFwuzGpPRamiM2Gli47aR/54Z98Zmtk3eU6BaEEJDakw2 iYQ8izqw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oEmzD-000bjO-EZ; Fri, 22 Jul 2022 07:22:11 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oEmzA-000bgL-03 for linux-mediatek@lists.infradead.org; Fri, 22 Jul 2022 07:22:10 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 8050CB827A0; Fri, 22 Jul 2022 07:22:06 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CE2E6C341C6; Fri, 22 Jul 2022 07:22:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1658474525; bh=mLF633ysa+rlMIaW85kQ89y9ybghbbfygiqrq9kYvLg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IrmsLuXQflDws66rey6+jCLV0XwZXN+zPmNva2+D3hI5HDldvQFiKxsc/M43IVh67 zozj0iV0VnW+EE7lJ4QKvXSVvfrdsF/JPw/yrFfIwnVgXt8nRO58DDQMhDHHT2261N Dk4BSCY5bbwyvJPAMf+/+zecUNla58dnt/UqBAiiGfTqICZroQ0xKJ528YftCNjzIa SbT79cTA239clTDLlhpiEHP9+gbLBG1sWzilme8GD94CpQrz+MM6yMjJEXFg+h50kC ZRPswqwN5e6DPOscLJnvX1JIhbN830BTghc3AKPbX4XPNoaP15qC97Z2w40eoffDhs SzVgqOX13BAxA== From: Lorenzo Bianconi To: netdev@vger.kernel.org Cc: nbd@nbd.name, john@phrozen.org, sean.wang@mediatek.com, Mark-MC.Lee@mediatek.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, matthias.bgg@gmail.com, linux-mediatek@lists.infradead.org, ilias.apalodimas@linaro.org, lorenzo.bianconi@redhat.com, jbrouer@redhat.com Subject: [PATCH v4 net-next 4/5] net: ethernet: mtk_eth_soc: add xmit XDP support Date: Fri, 22 Jul 2022 09:19:39 +0200 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220722_002208_342162_53CCB2B1 X-CRM114-Status: GOOD ( 24.03 ) X-BeenThere: linux-mediatek@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Linux-mediatek" Errors-To: linux-mediatek-bounces+linux-mediatek=archiver.kernel.org@lists.infradead.org Introduce XDP support for XDP_TX verdict and ndo_xdp_xmit function pointer. Signed-off-by: Lorenzo Bianconi --- drivers/net/ethernet/mediatek/mtk_eth_soc.c | 192 +++++++++++++++++--- drivers/net/ethernet/mediatek/mtk_eth_soc.h | 10 +- 2 files changed, 180 insertions(+), 22 deletions(-) diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.c b/drivers/net/ethernet/mediatek/mtk_eth_soc.c index cd96eaf72b6b..cfbdcf68f9b9 100644 --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c @@ -1031,15 +1031,26 @@ static void mtk_tx_unmap(struct mtk_eth *eth, struct mtk_tx_buf *tx_buf, } } - tx_buf->flags = 0; - if (tx_buf->skb && - (tx_buf->skb != (struct sk_buff *)MTK_DMA_DUMMY_DESC)) { - if (napi) - napi_consume_skb(tx_buf->skb, napi); + if (tx_buf->type == MTK_TYPE_SKB) { + if (tx_buf->data && + tx_buf->data != (void *)MTK_DMA_DUMMY_DESC) { + struct sk_buff *skb = tx_buf->data; + + if (napi) + napi_consume_skb(skb, napi); + else + dev_kfree_skb_any(skb); + } + } else if (tx_buf->data) { + struct xdp_frame *xdpf = tx_buf->data; + + if (napi && tx_buf->type == MTK_TYPE_XDP_TX) + xdp_return_frame_rx_napi(xdpf); else - dev_kfree_skb_any(tx_buf->skb); + xdp_return_frame(xdpf); } - tx_buf->skb = NULL; + tx_buf->flags = 0; + tx_buf->data = NULL; } static void setup_tx_buf(struct mtk_eth *eth, struct mtk_tx_buf *tx_buf, @@ -1056,7 +1067,7 @@ static void setup_tx_buf(struct mtk_eth *eth, struct mtk_tx_buf *tx_buf, dma_unmap_addr_set(tx_buf, dma_addr1, mapped_addr); dma_unmap_len_set(tx_buf, dma_len1, size); } else { - tx_buf->skb = (struct sk_buff *)MTK_DMA_DUMMY_DESC; + tx_buf->data = (void *)MTK_DMA_DUMMY_DESC; txd->txd1 = mapped_addr; txd->txd2 = TX_DMA_PLEN0(size); dma_unmap_addr_set(tx_buf, dma_addr0, mapped_addr); @@ -1232,7 +1243,7 @@ static int mtk_tx_map(struct sk_buff *skb, struct net_device *dev, soc->txrx.txd_size); if (new_desc) memset(tx_buf, 0, sizeof(*tx_buf)); - tx_buf->skb = (struct sk_buff *)MTK_DMA_DUMMY_DESC; + tx_buf->data = (void *)MTK_DMA_DUMMY_DESC; tx_buf->flags |= MTK_TX_FLAGS_PAGE0; tx_buf->flags |= (!mac->id) ? MTK_TX_FLAGS_FPORT0 : MTK_TX_FLAGS_FPORT1; @@ -1246,7 +1257,8 @@ static int mtk_tx_map(struct sk_buff *skb, struct net_device *dev, } /* store skb to cleanup */ - itx_buf->skb = skb; + itx_buf->type = MTK_TYPE_SKB; + itx_buf->data = skb; if (!MTK_HAS_CAPS(soc->caps, MTK_QDMA)) { if (k & 0x1) @@ -1458,13 +1470,14 @@ static struct page_pool *mtk_create_page_pool(struct mtk_eth *eth, .pool_size = size, .nid = NUMA_NO_NODE, .dev = eth->dma_dev, - .dma_dir = DMA_FROM_DEVICE, .offset = MTK_PP_HEADROOM, .max_len = MTK_PP_MAX_BUF_SIZE, }; struct page_pool *pp; int err; + pp_params.dma_dir = rcu_access_pointer(eth->prog) ? DMA_BIDIRECTIONAL + : DMA_FROM_DEVICE; pp = page_pool_create(&pp_params); if (IS_ERR(pp)) return pp; @@ -1510,6 +1523,122 @@ static void mtk_rx_put_buff(struct mtk_rx_ring *ring, void *data, bool napi) skb_free_frag(data); } +static int mtk_xdp_submit_frame(struct mtk_eth *eth, struct xdp_frame *xdpf, + struct net_device *dev, bool dma_map) +{ + const struct mtk_soc_data *soc = eth->soc; + struct mtk_tx_ring *ring = ð->tx_ring; + struct mtk_tx_dma_desc_info txd_info = { + .size = xdpf->len, + .first = true, + .last = true, + }; + struct mtk_mac *mac = netdev_priv(dev); + struct mtk_tx_dma *txd, *txd_pdma; + int err = 0, index = 0, n_desc = 1; + struct mtk_tx_buf *tx_buf; + + if (unlikely(test_bit(MTK_RESETTING, ð->state))) + return -EBUSY; + + if (unlikely(atomic_read(&ring->free_count) <= 1)) + return -EBUSY; + + spin_lock(ð->page_lock); + + txd = ring->next_free; + if (txd == ring->last_free) { + err = -ENOMEM; + goto out; + } + + tx_buf = mtk_desc_to_tx_buf(ring, txd, soc->txrx.txd_size); + memset(tx_buf, 0, sizeof(*tx_buf)); + + if (dma_map) { /* ndo_xdp_xmit */ + txd_info.addr = dma_map_single(eth->dma_dev, xdpf->data, + txd_info.size, DMA_TO_DEVICE); + if (unlikely(dma_mapping_error(eth->dma_dev, txd_info.addr))) { + err = -ENOMEM; + goto out; + } + tx_buf->flags |= MTK_TX_FLAGS_SINGLE0; + } else { + struct page *page = virt_to_head_page(xdpf->data); + + txd_info.addr = page_pool_get_dma_addr(page) + + sizeof(*xdpf) + xdpf->headroom; + dma_sync_single_for_device(eth->dma_dev, txd_info.addr, + txd_info.size, + DMA_BIDIRECTIONAL); + } + mtk_tx_set_dma_desc(dev, txd, &txd_info); + + tx_buf->flags |= !mac->id ? MTK_TX_FLAGS_FPORT0 : MTK_TX_FLAGS_FPORT1; + + txd_pdma = qdma_to_pdma(ring, txd); + setup_tx_buf(eth, tx_buf, txd_pdma, txd_info.addr, txd_info.size, + index++); + + /* store xdpf for cleanup */ + tx_buf->type = dma_map ? MTK_TYPE_XDP_NDO : MTK_TYPE_XDP_TX; + tx_buf->data = xdpf; + + if (!MTK_HAS_CAPS(soc->caps, MTK_QDMA)) { + if (index & 1) + txd_pdma->txd2 |= TX_DMA_LS0; + else + txd_pdma->txd2 |= TX_DMA_LS1; + } + + ring->next_free = mtk_qdma_phys_to_virt(ring, txd->txd2); + atomic_sub(n_desc, &ring->free_count); + + /* make sure that all changes to the dma ring are flushed before we + * continue + */ + wmb(); + + if (MTK_HAS_CAPS(soc->caps, MTK_QDMA)) { + mtk_w32(eth, txd->txd2, soc->reg_map->qdma.ctx_ptr); + } else { + int idx; + + idx = txd_to_idx(ring, txd, soc->txrx.txd_size); + mtk_w32(eth, NEXT_DESP_IDX(idx, ring->dma_size), + MT7628_TX_CTX_IDX0); + } +out: + spin_unlock(ð->page_lock); + + return err; +} + +static int mtk_xdp_xmit(struct net_device *dev, int num_frame, + struct xdp_frame **frames, u32 flags) +{ + struct mtk_mac *mac = netdev_priv(dev); + struct mtk_hw_stats *hw_stats = mac->hw_stats; + struct mtk_eth *eth = mac->hw; + int i, nxmit = 0; + + if (unlikely(flags & ~XDP_XMIT_FLAGS_MASK)) + return -EINVAL; + + for (i = 0; i < num_frame; i++) { + if (mtk_xdp_submit_frame(eth, frames[i], dev, true)) + break; + nxmit++; + } + + u64_stats_update_begin(&hw_stats->syncp); + hw_stats->xdp_stats.tx_xdp_xmit += nxmit; + hw_stats->xdp_stats.tx_xdp_xmit_errors += num_frame - nxmit; + u64_stats_update_end(&hw_stats->syncp); + + return nxmit; +} + static u32 mtk_xdp_run(struct mtk_eth *eth, struct mtk_rx_ring *ring, struct xdp_buff *xdp, struct net_device *dev) { @@ -1538,6 +1667,18 @@ static u32 mtk_xdp_run(struct mtk_eth *eth, struct mtk_rx_ring *ring, count = &hw_stats->xdp_stats.rx_xdp_redirect; goto update_stats; + case XDP_TX: { + struct xdp_frame *xdpf = xdp_convert_buff_to_frame(xdp); + + if (mtk_xdp_submit_frame(eth, xdpf, dev, false)) { + count = &hw_stats->xdp_stats.rx_xdp_tx_errors; + act = XDP_DROP; + break; + } + + count = &hw_stats->xdp_stats.rx_xdp_tx; + goto update_stats; + } default: bpf_warn_invalid_xdp_action(dev, prog, act); fallthrough; @@ -1771,9 +1912,8 @@ static int mtk_poll_tx_qdma(struct mtk_eth *eth, int budget, { const struct mtk_reg_map *reg_map = eth->soc->reg_map; struct mtk_tx_ring *ring = ð->tx_ring; - struct mtk_tx_dma *desc; - struct sk_buff *skb; struct mtk_tx_buf *tx_buf; + struct mtk_tx_dma *desc; u32 cpu, dma; cpu = ring->last_free_ptr; @@ -1794,15 +1934,21 @@ static int mtk_poll_tx_qdma(struct mtk_eth *eth, int budget, if (tx_buf->flags & MTK_TX_FLAGS_FPORT1) mac = 1; - skb = tx_buf->skb; - if (!skb) + if (!tx_buf->data) break; - if (skb != (struct sk_buff *)MTK_DMA_DUMMY_DESC) { + if (tx_buf->type == MTK_TYPE_SKB && + tx_buf->data != (void *)MTK_DMA_DUMMY_DESC) { + struct sk_buff *skb = tx_buf->data; + bytes[mac] += skb->len; done[mac]++; budget--; + } else if (tx_buf->type == MTK_TYPE_XDP_TX || + tx_buf->type == MTK_TYPE_XDP_NDO) { + budget--; } + mtk_tx_unmap(eth, tx_buf, true); ring->last_free = desc; @@ -1821,9 +1967,8 @@ static int mtk_poll_tx_pdma(struct mtk_eth *eth, int budget, unsigned int *done, unsigned int *bytes) { struct mtk_tx_ring *ring = ð->tx_ring; - struct mtk_tx_dma *desc; - struct sk_buff *skb; struct mtk_tx_buf *tx_buf; + struct mtk_tx_dma *desc; u32 cpu, dma; cpu = ring->cpu_idx; @@ -1831,14 +1976,18 @@ static int mtk_poll_tx_pdma(struct mtk_eth *eth, int budget, while ((cpu != dma) && budget) { tx_buf = &ring->buf[cpu]; - skb = tx_buf->skb; - if (!skb) + if (!tx_buf->data) break; - if (skb != (struct sk_buff *)MTK_DMA_DUMMY_DESC) { + if (tx_buf->type == MTK_TYPE_SKB && + tx_buf->data != (void *)MTK_DMA_DUMMY_DESC) { + struct sk_buff *skb = tx_buf->data; bytes[0] += skb->len; done[0]++; budget--; + } else if (tx_buf->type == MTK_TYPE_XDP_TX || + tx_buf->type == MTK_TYPE_XDP_NDO) { + budget--; } mtk_tx_unmap(eth, tx_buf, true); @@ -3518,6 +3667,7 @@ static const struct net_device_ops mtk_netdev_ops = { #endif .ndo_setup_tc = mtk_eth_setup_tc, .ndo_bpf = mtk_xdp, + .ndo_xdp_xmit = mtk_xdp_xmit, }; static int mtk_add_mac(struct mtk_eth *eth, struct device_node *np) diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.h b/drivers/net/ethernet/mediatek/mtk_eth_soc.h index 2775da1a6ec3..7405c97cda66 100644 --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.h +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.h @@ -696,6 +696,12 @@ enum mtk_dev_state { MTK_RESETTING }; +enum mtk_tx_buf_type { + MTK_TYPE_SKB, + MTK_TYPE_XDP_TX, + MTK_TYPE_XDP_NDO, +}; + /* struct mtk_tx_buf - This struct holds the pointers to the memory pointed at * by the TX descriptor s * @skb: The SKB pointer of the packet being sent @@ -705,7 +711,9 @@ enum mtk_dev_state { * @dma_len1: The length of the second segment */ struct mtk_tx_buf { - struct sk_buff *skb; + enum mtk_tx_buf_type type; + void *data; + u32 flags; DEFINE_DMA_UNMAP_ADDR(dma_addr0); DEFINE_DMA_UNMAP_LEN(dma_len0); From patchwork Fri Jul 22 07:19:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 12926082 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B98F7C43334 for ; Fri, 22 Jul 2022 07:22:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=sPw0AEXGUBQeLjXB7uKRQl5QYLH2OGDue6n5OGWYXr0=; b=LRz6VTUBa9Vc1BqjCj3UeHoPWP FSkJ8ADQD0Q5x/3kuwcIG8q0kmln2i30NqVs8z6I0+6096mMa9eKn+n1+3Pkzh2nfXG2ynvB231Sn y7TZilA1ivkVZ9IQxRKZ2klq8uFqhWwITX5nynoPN5Y6lpzSVsmvSK+3B0LuHdCpBij54PuECEopv f44Nq8vjq1ptQAv4EAh0WURS6re1E8Y6taYx4oPx02KJ8gxidcgwi3GbyLYBwTaybfMwfrvk3IG0h gp/50oNX3czWllvQbOSWdeDCUNMAbHQ+BUaNJ7e6TiTQvAhxDeUdPTXxPwhMiazX4W42wVEaLisK/ hnUi8P4g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oEmzo-000bya-C7; Fri, 22 Jul 2022 07:22:48 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oEmzl-000bwh-4e for linux-mediatek@lists.infradead.org; Fri, 22 Jul 2022 07:22:46 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id CBC0AB827A0; Fri, 22 Jul 2022 07:22:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2F0B9C341C6; Fri, 22 Jul 2022 07:22:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1658474562; bh=CnYkj1Ii7nMMi7fZxhtZxbEj7WgtArU1b5/mp8OWqGE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=a70lAY6XQLqHg2aG7Fkzt8v1FTX9zLHzNgjDlriEXee+rQLAA31RXZTZ/sTvHNFTn SSj2GGuMrIEcR9XAS6mYLc4pAWlBSGAA3b3UBKHdxVUmNO6u+nuZ+YAIoUG9TnRRjH FcsBn6qh4AF5srqSqsatxxl8CIuu9hjxqsFrJTc/1mwQuYvGh/5uXJwk0+CtFCuY9b +YxCj26M6Hb3o5vvYfz1OgvcwaOIjIjoEDtz7cSXSmDcttEJXJ2MktyUSgsbr3cQVq a92rRLQSUFBMIYopacnJyC55aX8kguYDojSkwnQiYrMqWzi1XzoXPGrQOei8rk1yUU gi5bPFSrQTDGw== From: Lorenzo Bianconi To: netdev@vger.kernel.org Cc: nbd@nbd.name, john@phrozen.org, sean.wang@mediatek.com, Mark-MC.Lee@mediatek.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, matthias.bgg@gmail.com, linux-mediatek@lists.infradead.org, ilias.apalodimas@linaro.org, lorenzo.bianconi@redhat.com, jbrouer@redhat.com Subject: [PATCH v4 net-next 5/5] net: ethernet: mtk_eth_soc: add support for page_pool_get_stats Date: Fri, 22 Jul 2022 09:19:40 +0200 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220722_002245_493629_C84913E4 X-CRM114-Status: GOOD ( 15.44 ) X-BeenThere: linux-mediatek@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Linux-mediatek" Errors-To: linux-mediatek-bounces+linux-mediatek=archiver.kernel.org@lists.infradead.org Introduce support for the page_pool stats API into mtk_eth_soc driver. Report page_pool stats through ethtool. Signed-off-by: Lorenzo Bianconi --- drivers/net/ethernet/mediatek/Kconfig | 1 + drivers/net/ethernet/mediatek/mtk_eth_soc.c | 37 +++++++++++++++++++-- 2 files changed, 35 insertions(+), 3 deletions(-) diff --git a/drivers/net/ethernet/mediatek/Kconfig b/drivers/net/ethernet/mediatek/Kconfig index d2422c7b31b0..97374fb3ee79 100644 --- a/drivers/net/ethernet/mediatek/Kconfig +++ b/drivers/net/ethernet/mediatek/Kconfig @@ -18,6 +18,7 @@ config NET_MEDIATEK_SOC select PHYLINK select DIMLIB select PAGE_POOL + select PAGE_POOL_STATS help This driver supports the gigabit ethernet MACs in the MediaTek SoC family. diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.c b/drivers/net/ethernet/mediatek/mtk_eth_soc.c index cfbdcf68f9b9..c370d6589596 100644 --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c @@ -3528,11 +3528,18 @@ static void mtk_get_strings(struct net_device *dev, u32 stringset, u8 *data) int i; switch (stringset) { - case ETH_SS_STATS: + case ETH_SS_STATS: { + struct mtk_mac *mac = netdev_priv(dev); + for (i = 0; i < ARRAY_SIZE(mtk_ethtool_stats); i++) { memcpy(data, mtk_ethtool_stats[i].str, ETH_GSTRING_LEN); data += ETH_GSTRING_LEN; } + if (mtk_page_pool_enabled(mac->hw)) + page_pool_ethtool_stats_get_strings(data); + break; + } + default: break; } } @@ -3540,13 +3547,35 @@ static void mtk_get_strings(struct net_device *dev, u32 stringset, u8 *data) static int mtk_get_sset_count(struct net_device *dev, int sset) { switch (sset) { - case ETH_SS_STATS: - return ARRAY_SIZE(mtk_ethtool_stats); + case ETH_SS_STATS: { + int count = ARRAY_SIZE(mtk_ethtool_stats); + struct mtk_mac *mac = netdev_priv(dev); + + if (mtk_page_pool_enabled(mac->hw)) + count += page_pool_ethtool_stats_get_count(); + return count; + } default: return -EOPNOTSUPP; } } +static void mtk_ethtool_pp_stats(struct mtk_eth *eth, u64 *data) +{ + struct page_pool_stats stats = {}; + int i; + + for (i = 0; i < ARRAY_SIZE(eth->rx_ring); i++) { + struct mtk_rx_ring *ring = ð->rx_ring[i]; + + if (!ring->page_pool) + continue; + + page_pool_get_stats(ring->page_pool, &stats); + } + page_pool_ethtool_stats_get(data, &stats); +} + static void mtk_get_ethtool_stats(struct net_device *dev, struct ethtool_stats *stats, u64 *data) { @@ -3574,6 +3603,8 @@ static void mtk_get_ethtool_stats(struct net_device *dev, for (i = 0; i < ARRAY_SIZE(mtk_ethtool_stats); i++) *data_dst++ = *(data_src + mtk_ethtool_stats[i].offset); + if (mtk_page_pool_enabled(mac->hw)) + mtk_ethtool_pp_stats(mac->hw, data_dst); } while (u64_stats_fetch_retry_irq(&hwstats->syncp, start)); }