From patchwork Wed Nov 25 22:51:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kiyanovski, Arthur" X-Patchwork-Id: 11932343 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BCB7C56201 for ; Wed, 25 Nov 2020 22:53:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 11D3220872 for ; Wed, 25 Nov 2020 22:53:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="EjISJeCo" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728855AbgKYWxB (ORCPT ); Wed, 25 Nov 2020 17:53:01 -0500 Received: from smtp-fw-6001.amazon.com ([52.95.48.154]:22348 "EHLO smtp-fw-6001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728808AbgKYWxA (ORCPT ); Wed, 25 Nov 2020 17:53:00 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1606344780; x=1637880780; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version; bh=TgRZ0yK3r5FaTO9KUQ7uC6vb8LyENFQKc6LY7cuT2dY=; b=EjISJeCoidfk/OTNSXmmlsLYT7hmILLQRjkQ0ydF+G+87tcQKcXwYLrY YFi4++a4FUnl+kTnoiSr/qwQUR77uOnCVu912OcVt9WCpTc46prBg20/8 WeqzW45aZ1HNQI+em7GnfDsVfp67fPFBVt6m9V6uMAW5ctJ0COVMJbmYj c=; X-IronPort-AV: E=Sophos;i="5.78,370,1599523200"; d="scan'208";a="68816342" Received: from iad12-co-svc-p1-lb1-vlan2.amazon.com (HELO email-inbound-relay-2c-cc689b93.us-west-2.amazon.com) ([10.43.8.2]) by smtp-border-fw-out-6001.iad6.amazon.com with ESMTP; 25 Nov 2020 22:52:53 +0000 Received: from EX13MTAUWA001.ant.amazon.com (pdx1-ws-svc-p6-lb9-vlan3.pdx.amazon.com [10.236.137.198]) by email-inbound-relay-2c-cc689b93.us-west-2.amazon.com (Postfix) with ESMTPS id F0EF1120EA5; Wed, 25 Nov 2020 22:52:51 +0000 (UTC) Received: from EX13D10UWA004.ant.amazon.com (10.43.160.64) by EX13MTAUWA001.ant.amazon.com (10.43.160.118) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 25 Nov 2020 22:52:40 +0000 Received: from EX13MTAUWA001.ant.amazon.com (10.43.160.58) by EX13D10UWA004.ant.amazon.com (10.43.160.64) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 25 Nov 2020 22:52:40 +0000 Received: from HFA15-G63729NC.amazon.com (10.1.213.33) by mail-relay.amazon.com (10.43.160.118) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Wed, 25 Nov 2020 22:52:35 +0000 From: To: , CC: Arthur Kiyanovski , , , , , , , , , , , , , , Subject: [PATCH V1 net-next 9/9] net: ena: introduce ndo_xdp_xmit() function for XDP_REDIRECT Date: Thu, 26 Nov 2020 00:51:48 +0200 Message-ID: <1606344708-11100-10-git-send-email-akiyano@amazon.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1606344708-11100-1-git-send-email-akiyano@amazon.com> References: <1606344708-11100-1-git-send-email-akiyano@amazon.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Arthur Kiyanovski This patch implements the ndo_xdp_xmit() net_device function which is called when a packet is redirected to this driver using an XDP_REDIRECT directive. The function receives an array of xdp frames that it needs to xmit. The TX queues that are used to xmit these frames are the XDP queues used by the XDP_TX flow. Therefore a lock is added to synchronize both flows (XDP_TX and XDP_REDIRECT). Signed-off-by: Shay Agroskin Signed-off-by: Arthur Kiyanovski --- drivers/net/ethernet/amazon/ena/ena_netdev.c | 84 +++++++++++++++++--- drivers/net/ethernet/amazon/ena/ena_netdev.h | 1 + 2 files changed, 72 insertions(+), 13 deletions(-) diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c index f110f9dbf955..231aed8871a1 100644 --- a/drivers/net/ethernet/amazon/ena/ena_netdev.c +++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c @@ -282,20 +282,18 @@ static int ena_xdp_tx_map_frame(struct ena_ring *xdp_ring, return -EINVAL; } -static int ena_xdp_xmit_frame(struct net_device *dev, +static int ena_xdp_xmit_frame(struct ena_ring *xdp_ring, + struct net_device *dev, struct xdp_frame *xdpf, - int qid) + int flags) { - struct ena_adapter *adapter = netdev_priv(dev); struct ena_com_tx_ctx ena_tx_ctx = {}; struct ena_tx_buffer *tx_info; - struct ena_ring *xdp_ring; u16 next_to_use, req_id; - int rc; void *push_hdr; u32 push_len; + int rc; - xdp_ring = &adapter->tx_ring[qid]; next_to_use = xdp_ring->next_to_use; req_id = xdp_ring->free_ids[next_to_use]; tx_info = &xdp_ring->tx_buffer_info[req_id]; @@ -322,26 +320,76 @@ static int ena_xdp_xmit_frame(struct net_device *dev, /* trigger the dma engine. ena_com_write_sq_doorbell() * has a mb */ - ena_com_write_sq_doorbell(xdp_ring->ena_com_io_sq); - ena_increase_stat_atomic(&xdp_ring->tx_stats.doorbells, 1, - &xdp_ring->syncp); + if (flags & XDP_XMIT_FLUSH) { + ena_com_write_sq_doorbell(xdp_ring->ena_com_io_sq); + ena_increase_stat_atomic(&xdp_ring->tx_stats.doorbells, 1, + &xdp_ring->syncp); + } - return NETDEV_TX_OK; + return rc; error_unmap_dma: ena_unmap_tx_buff(xdp_ring, tx_info); tx_info->xdpf = NULL; error_drop_packet: xdp_return_frame(xdpf); - return NETDEV_TX_OK; + return rc; +} + +static int ena_xdp_xmit(struct net_device *dev, int n, + struct xdp_frame **frames, u32 flags) +{ + struct ena_adapter *adapter = netdev_priv(dev); + int qid, i, err, drops = 0; + struct ena_ring *xdp_ring; + + if (unlikely(flags & ~XDP_XMIT_FLAGS_MASK)) + return -EINVAL; + + if (!test_bit(ENA_FLAG_DEV_UP, &adapter->flags)) + return -ENETDOWN; + + /* We assume that all rings have the same XDP program */ + if (!READ_ONCE(adapter->rx_ring->xdp_bpf_prog)) + return -ENXIO; + + qid = smp_processor_id() % adapter->xdp_num_queues; + qid += adapter->xdp_first_ring; + xdp_ring = &adapter->tx_ring[qid]; + + /* Other CPU ids might try to send thorugh this queue */ + spin_lock(&xdp_ring->xdp_tx_lock); + + for (i = 0; i < n; i++) { + err = ena_xdp_xmit_frame(xdp_ring, dev, frames[i], 0); + /* The descriptor is freed by ena_xdp_xmit_frame in case + * of an error. + */ + if (err) + drops++; + } + + /* Ring doorbell to make device aware of the packets */ + if (flags & XDP_XMIT_FLUSH) { + ena_com_write_sq_doorbell(xdp_ring->ena_com_io_sq); + ena_increase_stat_atomic(&xdp_ring->tx_stats.doorbells, 1, + &xdp_ring->syncp); + } + + spin_unlock(&xdp_ring->xdp_tx_lock); + + /* Return number of packets sent */ + return n - drops; } static int ena_xdp_execute(struct ena_ring *rx_ring, struct xdp_buff *xdp) { struct bpf_prog *xdp_prog; + struct ena_ring *xdp_ring; u32 verdict = XDP_PASS; struct xdp_frame *xdpf; u64 *xdp_stat; + int qid; rcu_read_lock(); xdp_prog = READ_ONCE(rx_ring->xdp_bpf_prog); @@ -354,8 +402,16 @@ static int ena_xdp_execute(struct ena_ring *rx_ring, struct xdp_buff *xdp) switch (verdict) { case XDP_TX: xdpf = xdp_convert_buff_to_frame(xdp); - ena_xdp_xmit_frame(rx_ring->netdev, xdpf, - rx_ring->qid + rx_ring->adapter->num_io_queues); + /* Find xmit queue */ + qid = rx_ring->qid + rx_ring->adapter->num_io_queues; + xdp_ring = &rx_ring->adapter->tx_ring[qid]; + + /* The XDP queues are shared between XDP_TX and XDP_REDIRECT */ + spin_lock(&xdp_ring->xdp_tx_lock); + + ena_xdp_xmit_frame(xdp_ring, rx_ring->netdev, xdpf, XDP_XMIT_FLUSH); + + spin_unlock(&xdp_ring->xdp_tx_lock); xdp_stat = &rx_ring->rx_stats.xdp_tx; break; case XDP_REDIRECT: @@ -646,6 +702,7 @@ static void ena_init_io_rings(struct ena_adapter *adapter, txr->smoothed_interval = ena_com_get_nonadaptive_moderation_interval_tx(ena_dev); txr->disable_meta_caching = adapter->disable_meta_caching; + spin_lock_init(&txr->xdp_tx_lock); /* Don't init RX queues for xdp queues */ if (!ENA_IS_XDP_INDEX(adapter, i)) { @@ -3239,6 +3296,7 @@ static const struct net_device_ops ena_netdev_ops = { .ndo_set_mac_address = NULL, .ndo_validate_addr = eth_validate_addr, .ndo_bpf = ena_xdp, + .ndo_xdp_xmit = ena_xdp_xmit, }; static int ena_device_validate_params(struct ena_adapter *adapter, diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.h b/drivers/net/ethernet/amazon/ena/ena_netdev.h index fed79c50a870..74af15d62ee1 100644 --- a/drivers/net/ethernet/amazon/ena/ena_netdev.h +++ b/drivers/net/ethernet/amazon/ena/ena_netdev.h @@ -258,6 +258,7 @@ struct ena_ring { struct ena_com_io_sq *ena_com_io_sq; struct bpf_prog *xdp_bpf_prog; struct xdp_rxq_info xdp_rxq; + spinlock_t xdp_tx_lock; /* synchronize XDP TX/Redirect traffic */ u16 next_to_use; u16 next_to_clean;