From patchwork Fri Nov 20 17:05:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 11921409 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63007C56201 for ; Fri, 20 Nov 2020 17:06:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F2FB32078B for ; Fri, 20 Nov 2020 17:06:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="nF6ybdm1" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730279AbgKTRGS (ORCPT ); Fri, 20 Nov 2020 12:06:18 -0500 Received: from mail.kernel.org ([198.145.29.99]:56882 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728223AbgKTRGS (ORCPT ); Fri, 20 Nov 2020 12:06:18 -0500 Received: from lore-desk.redhat.com (unknown [151.66.8.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id B03B52225B; Fri, 20 Nov 2020 17:06:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1605891976; bh=nSj88FBcp1WSuiTw1D70+ian/NVTZr4S4PTnuLkYEYE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nF6ybdm1Gp9634FrmXd6f0izqEIhvW75hD2d6FRc8UP8EHg+ds5NDFFe86xLQ2UY5 EKOI48EN8srvGv/3cFePbH7ieV0ml+AhIcsBmkPW8ImQNy5qGipysgVGOzlFUGEEd0 oYz+Qzfap38qjKbeNWKWxxyCGCj5sBE99XiIl5UM= From: Lorenzo Bianconi To: netdev@vger.kernel.org Cc: lorenzo.bianconi@redhat.com, davem@davemloft.net, kuba@kernel.org, brouer@redhat.com, echaudro@redhat.com, john.fastabend@gmail.com Subject: [PATCH net-next 1/3] net: mvneta: avoid unnecessary xdp_buff initialization Date: Fri, 20 Nov 2020 18:05:42 +0100 Message-Id: <611eb5973eb88b0a7a342f44c818b3bcb6e94c8c.1605889259.git.lorenzo@kernel.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Explicitly initialize mandatory fields defining xdp_buff struct in mvneta_rx_swbm routine Signed-off-by: Lorenzo Bianconi --- drivers/net/ethernet/marvell/mvneta.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c index 5d559117f78c..60a285f3a55f 100644 --- a/drivers/net/ethernet/marvell/mvneta.c +++ b/drivers/net/ethernet/marvell/mvneta.c @@ -2347,13 +2347,14 @@ static int mvneta_rx_swbm(struct napi_struct *napi, { int rx_proc = 0, rx_todo, refill, size = 0; struct net_device *dev = pp->dev; - struct xdp_buff xdp_buf = { - .frame_sz = PAGE_SIZE, - .rxq = &rxq->xdp_rxq, - }; struct mvneta_stats ps = {}; struct bpf_prog *xdp_prog; u32 desc_status, frame_sz; + struct xdp_buff xdp_buf; + + xdp_buf.data_hard_start = NULL; + xdp_buf.frame_sz = PAGE_SIZE; + xdp_buf.rxq = &rxq->xdp_rxq; /* Get number of received packets */ rx_todo = mvneta_rxq_busy_desc_num_get(pp, rxq); From patchwork Fri Nov 20 17:05:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 11921407 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE32BC6379D for ; Fri, 20 Nov 2020 17:06:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6702B2245B for ; Fri, 20 Nov 2020 17:06:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="wIFl/NcI" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730285AbgKTRGU (ORCPT ); Fri, 20 Nov 2020 12:06:20 -0500 Received: from mail.kernel.org ([198.145.29.99]:56892 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730280AbgKTRGU (ORCPT ); Fri, 20 Nov 2020 12:06:20 -0500 Received: from lore-desk.redhat.com (unknown [151.66.8.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 61AA72240B; Fri, 20 Nov 2020 17:06:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1605891979; bh=9XCqAiXAevArfrajvpC72jyXrSmmo6xHkuituXKHelc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=wIFl/NcIKkxBLif+U40Ls0eh94ZYzHDQLmaY7WYPqSlMwO0zUeYhiAR9K4VkZirqc nriQqVKf4liyX0JqWIJ7Gj0a47t2k+djAnWD70P7KJmU4aCxDS09XVSq6rc4KdYd9Y NA1JKcFOWCosAQy7KEMoJ7XA2P3d9LdcTK4XbRQo= From: Lorenzo Bianconi To: netdev@vger.kernel.org Cc: lorenzo.bianconi@redhat.com, davem@davemloft.net, kuba@kernel.org, brouer@redhat.com, echaudro@redhat.com, john.fastabend@gmail.com Subject: [PATCH net-next 2/3] net: mvneta: move skb_shared_info in mvneta_xdp_put_buff caller Date: Fri, 20 Nov 2020 18:05:43 +0100 Message-Id: X-Mailer: git-send-email 2.26.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Pass skb_shared_info pointer from mvneta_xdp_put_buff caller. This is a preliminary patch to reduce accesses to skb_shared_info area and reduce cache misses. Remove napi parameter in mvneta_xdp_put_buff signature since it is always run in NAPI context Signed-off-by: Lorenzo Bianconi --- drivers/net/ethernet/marvell/mvneta.c | 32 ++++++++++++++++++--------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c index 60a285f3a55f..17c446b1cb94 100644 --- a/drivers/net/ethernet/marvell/mvneta.c +++ b/drivers/net/ethernet/marvell/mvneta.c @@ -2033,16 +2033,16 @@ int mvneta_rx_refill_queue(struct mvneta_port *pp, struct mvneta_rx_queue *rxq) static void mvneta_xdp_put_buff(struct mvneta_port *pp, struct mvneta_rx_queue *rxq, - struct xdp_buff *xdp, int sync_len, bool napi) + struct xdp_buff *xdp, struct skb_shared_info *sinfo, + int sync_len) { - struct skb_shared_info *sinfo = xdp_get_shared_info_from_buff(xdp); int i; for (i = 0; i < sinfo->nr_frags; i++) page_pool_put_full_page(rxq->page_pool, - skb_frag_page(&sinfo->frags[i]), napi); + skb_frag_page(&sinfo->frags[i]), true); page_pool_put_page(rxq->page_pool, virt_to_head_page(xdp->data), - sync_len, napi); + sync_len, true); } static int @@ -2179,6 +2179,7 @@ mvneta_run_xdp(struct mvneta_port *pp, struct mvneta_rx_queue *rxq, struct bpf_prog *prog, struct xdp_buff *xdp, u32 frame_sz, struct mvneta_stats *stats) { + struct skb_shared_info *sinfo = xdp_get_shared_info_from_buff(xdp); unsigned int len, data_len, sync; u32 ret, act; @@ -2199,7 +2200,7 @@ mvneta_run_xdp(struct mvneta_port *pp, struct mvneta_rx_queue *rxq, err = xdp_do_redirect(pp->dev, xdp, prog); if (unlikely(err)) { - mvneta_xdp_put_buff(pp, rxq, xdp, sync, true); + mvneta_xdp_put_buff(pp, rxq, xdp, sinfo, sync); ret = MVNETA_XDP_DROPPED; } else { ret = MVNETA_XDP_REDIR; @@ -2210,7 +2211,7 @@ mvneta_run_xdp(struct mvneta_port *pp, struct mvneta_rx_queue *rxq, case XDP_TX: ret = mvneta_xdp_xmit_back(pp, xdp); if (ret != MVNETA_XDP_TX) - mvneta_xdp_put_buff(pp, rxq, xdp, sync, true); + mvneta_xdp_put_buff(pp, rxq, xdp, sinfo, sync); break; default: bpf_warn_invalid_xdp_action(act); @@ -2219,7 +2220,7 @@ mvneta_run_xdp(struct mvneta_port *pp, struct mvneta_rx_queue *rxq, trace_xdp_exception(pp->dev, prog, act); fallthrough; case XDP_DROP: - mvneta_xdp_put_buff(pp, rxq, xdp, sync, true); + mvneta_xdp_put_buff(pp, rxq, xdp, sinfo, sync); ret = MVNETA_XDP_DROPPED; stats->xdp_drop++; break; @@ -2406,7 +2407,10 @@ static int mvneta_rx_swbm(struct napi_struct *napi, continue; if (size) { - mvneta_xdp_put_buff(pp, rxq, &xdp_buf, -1, true); + struct skb_shared_info *sinfo; + + sinfo = xdp_get_shared_info_from_buff(&xdp_buf); + mvneta_xdp_put_buff(pp, rxq, &xdp_buf, sinfo, -1); goto next; } @@ -2417,8 +2421,10 @@ static int mvneta_rx_swbm(struct napi_struct *napi, skb = mvneta_swbm_build_skb(pp, rxq, &xdp_buf, desc_status); if (IS_ERR(skb)) { struct mvneta_pcpu_stats *stats = this_cpu_ptr(pp->stats); + struct skb_shared_info *sinfo; - mvneta_xdp_put_buff(pp, rxq, &xdp_buf, -1, true); + sinfo = xdp_get_shared_info_from_buff(&xdp_buf); + mvneta_xdp_put_buff(pp, rxq, &xdp_buf, sinfo, -1); u64_stats_update_begin(&stats->syncp); stats->es.skb_alloc_error++; @@ -2438,8 +2444,12 @@ static int mvneta_rx_swbm(struct napi_struct *napi, } rcu_read_unlock(); - if (xdp_buf.data_hard_start) - mvneta_xdp_put_buff(pp, rxq, &xdp_buf, -1, true); + if (xdp_buf.data_hard_start) { + struct skb_shared_info *sinfo; + + sinfo = xdp_get_shared_info_from_buff(&xdp_buf); + mvneta_xdp_put_buff(pp, rxq, &xdp_buf, sinfo, -1); + } if (ps.xdp_redirect) xdp_do_flush_map(); From patchwork Fri Nov 20 17:05:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lorenzo Bianconi X-Patchwork-Id: 11921411 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F854C2D0E4 for ; Fri, 20 Nov 2020 17:06:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C16982078B for ; Fri, 20 Nov 2020 17:06:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="cRwGfodn" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730289AbgKTRGX (ORCPT ); Fri, 20 Nov 2020 12:06:23 -0500 Received: from mail.kernel.org ([198.145.29.99]:56912 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727560AbgKTRGW (ORCPT ); Fri, 20 Nov 2020 12:06:22 -0500 Received: from lore-desk.redhat.com (unknown [151.66.8.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 7E57122252; Fri, 20 Nov 2020 17:06:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1605891982; bh=/INFXjbcblAoycPvhGQ002NZFMiDjRHP7HSbLO1dTUk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cRwGfodnCliDE1wAZYBpRMExh67aM22BdkSF4wwdej4445EApO4QPYV4qtKOj2T8X MVqWCmDSfdClGdtBM34V7X5meWklcFrv5nIWjiO1zUNqybV+pgD/jLn8cTjT4tW5vZ +9B2qD4VkIOeZzw2AfIlZksIY/81e76mcTPIDSKc= From: Lorenzo Bianconi To: netdev@vger.kernel.org Cc: lorenzo.bianconi@redhat.com, davem@davemloft.net, kuba@kernel.org, brouer@redhat.com, echaudro@redhat.com, john.fastabend@gmail.com Subject: [PATCH net-next 3/3] net: mvneta: alloc skb_shared_info on the mvneta_rx_swbm stack Date: Fri, 20 Nov 2020 18:05:44 +0100 Message-Id: <4dfbab639c10d60c17d6aaca96addccbb2aeb737.1605889259.git.lorenzo@kernel.org> X-Mailer: git-send-email 2.26.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Build skb_shared_info on mvneta_rx_swbm stack and sync it to xdp_buff skb_shared_info area only on the last fragment. Leftover cache miss in mvneta_swbm_rx_frame will be addressed introducing mb bit in xdp_buff/xdp_frame struct Signed-off-by: Lorenzo Bianconi --- drivers/net/ethernet/marvell/mvneta.c | 42 +++++++++++++++------------ 1 file changed, 23 insertions(+), 19 deletions(-) diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c index 17c446b1cb94..b40804e421a7 100644 --- a/drivers/net/ethernet/marvell/mvneta.c +++ b/drivers/net/ethernet/marvell/mvneta.c @@ -2278,9 +2278,9 @@ mvneta_swbm_add_rx_fragment(struct mvneta_port *pp, struct mvneta_rx_desc *rx_desc, struct mvneta_rx_queue *rxq, struct xdp_buff *xdp, int *size, + struct skb_shared_info *xdp_sinfo, struct page *page) { - struct skb_shared_info *sinfo = xdp_get_shared_info_from_buff(xdp); struct net_device *dev = pp->dev; enum dma_data_direction dma_dir; int data_len, len; @@ -2298,13 +2298,22 @@ mvneta_swbm_add_rx_fragment(struct mvneta_port *pp, len, dma_dir); rx_desc->buf_phys_addr = 0; - if (data_len > 0 && sinfo->nr_frags < MAX_SKB_FRAGS) { - skb_frag_t *frag = &sinfo->frags[sinfo->nr_frags]; + if (data_len > 0 && xdp_sinfo->nr_frags < MAX_SKB_FRAGS) { + skb_frag_t *frag = &xdp_sinfo->frags[xdp_sinfo->nr_frags++]; skb_frag_off_set(frag, pp->rx_offset_correction); skb_frag_size_set(frag, data_len); __skb_frag_set_page(frag, page); - sinfo->nr_frags++; + + /* last fragment */ + if (len == *size) { + struct skb_shared_info *sinfo; + + sinfo = xdp_get_shared_info_from_buff(xdp); + sinfo->nr_frags = xdp_sinfo->nr_frags; + memcpy(sinfo->frags, xdp_sinfo->frags, + sinfo->nr_frags * sizeof(skb_frag_t)); + } } else { page_pool_put_full_page(rxq->page_pool, page, true); } @@ -2348,6 +2357,7 @@ static int mvneta_rx_swbm(struct napi_struct *napi, { int rx_proc = 0, rx_todo, refill, size = 0; struct net_device *dev = pp->dev; + struct skb_shared_info sinfo; struct mvneta_stats ps = {}; struct bpf_prog *xdp_prog; u32 desc_status, frame_sz; @@ -2357,6 +2367,8 @@ static int mvneta_rx_swbm(struct napi_struct *napi, xdp_buf.frame_sz = PAGE_SIZE; xdp_buf.rxq = &rxq->xdp_rxq; + sinfo.nr_frags = 0; + /* Get number of received packets */ rx_todo = mvneta_rxq_busy_desc_num_get(pp, rxq); @@ -2395,11 +2407,11 @@ static int mvneta_rx_swbm(struct napi_struct *napi, rx_desc->buf_phys_addr = 0; page_pool_put_full_page(rxq->page_pool, page, true); - continue; + goto next; } mvneta_swbm_add_rx_fragment(pp, rx_desc, rxq, &xdp_buf, - &size, page); + &size, &sinfo, page); } /* Middle or Last descriptor */ if (!(rx_status & MVNETA_RXD_LAST_DESC)) @@ -2407,10 +2419,7 @@ static int mvneta_rx_swbm(struct napi_struct *napi, continue; if (size) { - struct skb_shared_info *sinfo; - - sinfo = xdp_get_shared_info_from_buff(&xdp_buf); - mvneta_xdp_put_buff(pp, rxq, &xdp_buf, sinfo, -1); + mvneta_xdp_put_buff(pp, rxq, &xdp_buf, &sinfo, -1); goto next; } @@ -2421,10 +2430,8 @@ static int mvneta_rx_swbm(struct napi_struct *napi, skb = mvneta_swbm_build_skb(pp, rxq, &xdp_buf, desc_status); if (IS_ERR(skb)) { struct mvneta_pcpu_stats *stats = this_cpu_ptr(pp->stats); - struct skb_shared_info *sinfo; - sinfo = xdp_get_shared_info_from_buff(&xdp_buf); - mvneta_xdp_put_buff(pp, rxq, &xdp_buf, sinfo, -1); + mvneta_xdp_put_buff(pp, rxq, &xdp_buf, &sinfo, -1); u64_stats_update_begin(&stats->syncp); stats->es.skb_alloc_error++; @@ -2441,15 +2448,12 @@ static int mvneta_rx_swbm(struct napi_struct *napi, napi_gro_receive(napi, skb); next: xdp_buf.data_hard_start = NULL; + sinfo.nr_frags = 0; } rcu_read_unlock(); - if (xdp_buf.data_hard_start) { - struct skb_shared_info *sinfo; - - sinfo = xdp_get_shared_info_from_buff(&xdp_buf); - mvneta_xdp_put_buff(pp, rxq, &xdp_buf, sinfo, -1); - } + if (xdp_buf.data_hard_start) + mvneta_xdp_put_buff(pp, rxq, &xdp_buf, &sinfo, -1); if (ps.xdp_redirect) xdp_do_flush_map();