From patchwork Tue Jul 4 09:59:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sriram Yagnaraman X-Patchwork-Id: 13300928 X-Patchwork-Delegate: kuba@kernel.org Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 279AA539F; Tue, 4 Jul 2023 09:59:27 +0000 (UTC) Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05on2127.outbound.protection.outlook.com [40.107.21.127]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 100C7D8; Tue, 4 Jul 2023 02:59:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mSbxb2T4XVo9Ybo/X7/bQpNhrWVzHszQy0Rlqi3QqVX6DznzrpMK4XSx6xK7Ox5bOkaxtpIBOR2YPFrE9WbdXuLwu2vtccMI/OpPDDqgFmm3FjVGdPPfOASPJG3krwbYRB1fCI5rBn+vzRVGyLXVd1m83vqCm17SaDpsytLJyKq2y4OWqSR7++Ct9g51FfcA/huiKYZKMplv4UQHaAHmZJcJRMKBICYygAWqzqd4ek/yL1q8fk2MZunVFIYc63RkD7bDHnbnrG39JTATxuxG4DYwc8Qf6UiO1BOqmDuZY5Qa/Na/0auK3g4fYnhR8AvdFk25rIPXA7O6A+xMOG4d7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=gJNzk6ortZTQ0/sbBAse5RTvyGBIs2ID7DgTjqYF3oE=; b=fUah104m7pESXscbYNGZwJyTlZPRnroGFX/OL4JAoOeh+AMaAM2XJZgYHjYefY48Z15vjd5ReWF9epvPeLaxB203jav+f1gq0hq2/mdwcfjubguyCg7oTKdMvjr6i9sNJv1O7mDq2Fl5ooth/s/MTB9oYABcChdTVk5BJJnEHro3tdeV9wQcO3fcZo672gjTD0hLFzB0PIuoOmNp7DKKPQ2FwGb44zDUNfIgQn4PQqnB7HiitiydK/FzFvylZxF1yRnZdnQxdXroKUQun+gFVsB7z9HiDpJO4RE0fCzieNl0WsHra0IkPlPV7TUNnqCwqA0LaNrWajlixYh/SJPKFA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=est.tech; dmarc=pass action=none header.from=est.tech; dkim=pass header.d=est.tech; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=estab.onmicrosoft.com; s=selector2-estab-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gJNzk6ortZTQ0/sbBAse5RTvyGBIs2ID7DgTjqYF3oE=; b=C68JV4pFRnqiZBZdo827rKGjSd0aIAXHwDUOdVQObcffvn1NYLY1CyvJGKSeyx+CJRPHxqPzCcGjTFyFoDe5sg1pbzeLoTkhFbg+EkajxpuHdT6gNimF2nvMBQn4U012g2g2bGs2BEMYxxYmI5QNScQZjM/2z1YyB/OPCoOFfgw= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=est.tech; Received: from DBBP189MB1433.EURP189.PROD.OUTLOOK.COM (2603:10a6:10:1e7::15) by AM9P189MB1571.EURP189.PROD.OUTLOOK.COM (2603:10a6:20b:306::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6544.24; Tue, 4 Jul 2023 09:59:22 +0000 Received: from DBBP189MB1433.EURP189.PROD.OUTLOOK.COM ([fe80::57d:1bb3:9180:73b6]) by DBBP189MB1433.EURP189.PROD.OUTLOOK.COM ([fe80::57d:1bb3:9180:73b6%4]) with mapi id 15.20.6544.024; Tue, 4 Jul 2023 09:59:22 +0000 From: Sriram Yagnaraman To: Cc: intel-wired-lan@lists.osuosl.org, bpf@vger.kernel.org, netdev@vger.kernel.org, Jesse Brandeburg , Tony Nguyen , "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Sriram Yagnaraman Subject: [PATCH 1/4] igb: prepare for AF_XDP zero-copy support Date: Tue, 4 Jul 2023 11:59:12 +0200 Message-Id: <20230704095915.9750-2-sriram.yagnaraman@est.tech> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230704095915.9750-1-sriram.yagnaraman@est.tech> References: <20230704095915.9750-1-sriram.yagnaraman@est.tech> X-ClientProxiedBy: GV2PEPF000000FF.SWEP280.PROD.OUTLOOK.COM (2603:10a6:144:1:0:1:0:9) To DBBP189MB1433.EURP189.PROD.OUTLOOK.COM (2603:10a6:10:1e7::15) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DBBP189MB1433:EE_|AM9P189MB1571:EE_ X-MS-Office365-Filtering-Correlation-Id: 2522826c-91cd-41c3-9734-08db7c75577c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: YNQdgalNqepkwmyBlHKs1yj0645vDyDOUbKXOyJLJ7LK0gOfdOxSyPO3/D6dOuMpESqkzzvXgzuN3A/uwyIplN81LL4GRF4y5HxXJ9izQ/I7RvYVNdjqHeMWYV8tixWjs99vz4lOhZVIeVyJ2Po2WQVKbRDIMmvaam77E0E6/Lbb3E6xEmQCNPFIHfrfQ5ZY245TTaKJYvJCLumxe7uEpOuYQbQ+0FiARGf6SnDVy9xXi0lb6boUppfflumwHbj8ssGsvkYRRfsmLD8gNgomkKQKkjznK6pL0HmGet/RKcLTV6T2kGDozpJ3paTv1Uixl9EtFX8bHBAhJx1QZBl5F6GouXO/Kw9Ao1jFo9q+w/eampRovSpTlLZp5RjMHDTafafICPSZeVLJTvLSDgo61nJ14ETUi2qgAjKTHWjrTLmxcB+7BGZkPDk80vhQGcEuRb7+tA5WZ/k7eXKy62Bop8F3i+Wx+1+xKQl6QzpoHBQS8G4lLUkVdICFqketuxeFGKWvHT3XmhJaD0hpt2BNjnK7bbX4j2ODPC1vvUze9JzBXBreIr8dmJyvCDpABtdvBVkRuRXUTJI4cfG3EUk9ne3bnJx8CSYt5qlWWb+dR+o00U19HsMeSKIaS3i2yTZ18SP868y6nrWUdH3i9KzIGueKTYdAns+2rrlzOMsXTp4= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DBBP189MB1433.EURP189.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230028)(346002)(396003)(136003)(376002)(39840400004)(451199021)(109986019)(2906002)(41300700001)(7416002)(5660300002)(44832011)(8676002)(8936002)(36756003)(86362001)(6512007)(6666004)(6486002)(478600001)(83380400001)(26005)(186003)(2616005)(4326008)(1076003)(66556008)(54906003)(66476007)(70586007)(6506007)(66946007)(316002)(38100700002)(266003)(37730700002)(309714004);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 42xiECoKgyETZqGK7HgFKdNuYIv/oqVgp95yngccjlN4P++NG1rCNlEvOqhPG2sLF5Y9Nve+VbUk83kwD7xi0im6oFPJNRRSpzEXONK4DrTQ8ykUZ3sHZ1O53QARgh+bWiqC9FoHjoYFPWtc/lGKIiyZISkvH6zCrv2SyZUUdp9H5keGKFMdQ+Z4jKZbv3ObKAALkEceBFVeD/rPjrVS5vM6DYKFP2Zjjl0NAER9HQdpUDpV1Tph2mEBZiIQ+zJo9t79irZkw5TwIFysD1H+5wmtopczEL8z0/6YT6uFjNnFoztobIXHPl3I5Owcll0SQSjIhSUI31E7nX3Bp6nCO79I7YZwY+XNrCPuzIjTC7fQvRf5OBNL2MjwTyDXxhNwUJmlZIvwAq8maa+y6ooBL5vXi2X39v1udSjYkOf+hzkDRTelBi/DEScHfSiCYnIC2dpmFJUTtx9rTMsIk+/bU5hZuQDpH2BsIbym1Hgwq+LGPh2G8gSqecotd/CxG22hyN5kPup8Znny0G5zSzWfa6hvxqmz6HkN89LgYVl7Ew4A/prZB/BvyZI+0djbY5y2s0yqrSSsFoOPVGdkgEDtaraOyjiuN9mlPjm1RbPUiuLO5/Nlf/vPS8f4P2McAlRci7B0IQ2d+2k5MB3zLtlGj5e2BJg2RTUax1euUHeIGXiB6Cvd9IJOjEJKHNNqg4jIxqd0UcHBF7I3O/laMBZrNI/OlLiPhQ0juE8YpshmUVMQDU3yA0gBpwDcN1EulXb7dpTGolS4SGvplyNsnLKv3dtPm7SBbUC7LX0edrarXW8RR2dQ89iej/H2pZpxZvTn/qvL9VzNybwcTrdnOumgyahh2IYrRjSWd8VaYrRGwGQKUvE05kgEOq5PmqkUBZxjXSeKaARdo6ZbsPUb/oIz9/hgPnRo3vXy3rliOPH8eQd/SAnfkzjZVuYNFdqhbrPSirPjlmHbC8vEInEF7l0Nap/c/qwGKlc/liUtxgc2EzlwE5TrZOntkvz8AMo0iBp5EZQ7OMo2LXUEb7xRLOoCQD/FuRQhayaokF2tnV/q5kgb7d6t5/dYc98LUsoTC3fcv6Pht6qyr0fRMsa08bJoSXNoMNcLStCns1KVB7i1nQEMUOvafyU4G/5iPuxdU6MgGPMXt/qAfCwPfLk/NxvOMqre7USyM6ZgrPPBn3OTQw1V9x/Ephk/RRKmjaZl9L1vatK1aV5hjiKJR0R5AGQq5cZzkPAt2JdY4YNd30eFlA1GVEPmm3zZ0s98mtKMjCiM++kAsmC3bxM+Y0SlPklqF83aL29Ai1BFw/2PnmBv7ATqPh76zJM545q7qlyE25UiCJFHPCtgVVu1bPYZueXgtePzEAZtx+FICyDpjWCF/zoKNjDHDYj+GIa3jkbLmJEKAHeFas2RsBj7346bV7xb7yHjaXZcTPhXt7F6vggIDb61mznKjveYzcYzgDggo77hSDBipuRQumw2mgmRIfUzU8WItc3bfQMb/Bx9jyn3p8n4aS/zkhQWvmQdkMnC+wTUh4z9MYTjOhoUXxi5i5IkUdbYXIMRES6nULI4Oxs+oWJdw5hRBZRBbNoQLSRWs6Mf2IfIlig2R/zZ5Tw+yAgwdQ== X-OriginatorOrg: est.tech X-MS-Exchange-CrossTenant-Network-Message-Id: 2522826c-91cd-41c3-9734-08db7c75577c X-MS-Exchange-CrossTenant-AuthSource: DBBP189MB1433.EURP189.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jul 2023 09:59:22.5044 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: d2585e63-66b9-44b6-a76e-4f4b217d97fd X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: xYbtWTK+qvpB3RYq1Jy2yQ/d65a22XqrNNgCWBbuwnI43gyK9pCObyJoLdWaP68E13TzvDL2XudcRNzIyyG/crt2aKpwmxDCuwkGV75Ru8g= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM9P189MB1571 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: kuba@kernel.org Remove static qualifiers on the following functions to be able to call from XSK specific file that is added in the later patches - igb_xdp_tx_queue_mapping - igb_xdp_ring_update_tail - igb_run_xdp - igb_process_skb_fields Introduce igb_xdp_is_enabled() to check if an XDP program is assigned to the device. Signed-off-by: Sriram Yagnaraman --- drivers/net/ethernet/intel/igb/igb.h | 13 +++++++++++++ drivers/net/ethernet/intel/igb/igb_main.c | 23 ++++++++++++----------- 2 files changed, 25 insertions(+), 11 deletions(-) diff --git a/drivers/net/ethernet/intel/igb/igb.h b/drivers/net/ethernet/intel/igb/igb.h index 015b78144114..94440af6cf4b 100644 --- a/drivers/net/ethernet/intel/igb/igb.h +++ b/drivers/net/ethernet/intel/igb/igb.h @@ -718,6 +718,8 @@ extern char igb_driver_name[]; int igb_xmit_xdp_ring(struct igb_adapter *adapter, struct igb_ring *ring, struct xdp_frame *xdpf); +struct igb_ring *igb_xdp_tx_queue_mapping(struct igb_adapter *adapter); +void igb_xdp_ring_update_tail(struct igb_ring *ring); int igb_open(struct net_device *netdev); int igb_close(struct net_device *netdev); int igb_up(struct igb_adapter *); @@ -737,6 +739,12 @@ void igb_setup_tctl(struct igb_adapter *); void igb_setup_rctl(struct igb_adapter *); void igb_setup_srrctl(struct igb_adapter *, struct igb_ring *); netdev_tx_t igb_xmit_frame_ring(struct sk_buff *, struct igb_ring *); +struct sk_buff *igb_run_xdp(struct igb_adapter *adapter, + struct igb_ring *rx_ring, + struct xdp_buff *xdp); +void igb_process_skb_fields(struct igb_ring *rx_ring, + union e1000_adv_rx_desc *rx_desc, + struct sk_buff *skb); void igb_alloc_rx_buffers(struct igb_ring *, u16); void igb_update_stats(struct igb_adapter *); bool igb_has_link(struct igb_adapter *adapter); @@ -797,6 +805,11 @@ static inline struct netdev_queue *txring_txq(const struct igb_ring *tx_ring) return netdev_get_tx_queue(tx_ring->netdev, tx_ring->queue_index); } +static inline bool igb_xdp_is_enabled(struct igb_adapter *adapter) +{ + return !!adapter->xdp_prog; +} + int igb_add_filter(struct igb_adapter *adapter, struct igb_nfc_filter *input); int igb_erase_filter(struct igb_adapter *adapter, diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c index 9a2561409b06..dadc3d423cfd 100644 --- a/drivers/net/ethernet/intel/igb/igb_main.c +++ b/drivers/net/ethernet/intel/igb/igb_main.c @@ -2939,7 +2939,8 @@ static int igb_xdp(struct net_device *dev, struct netdev_bpf *xdp) } } -static void igb_xdp_ring_update_tail(struct igb_ring *ring) +/* This function assumes __netif_tx_lock is held by the caller. */ +void igb_xdp_ring_update_tail(struct igb_ring *ring) { /* Force memory writes to complete before letting h/w know there * are new descriptors to fetch. @@ -2948,7 +2949,7 @@ static void igb_xdp_ring_update_tail(struct igb_ring *ring) writel(ring->next_to_use, ring->tail); } -static struct igb_ring *igb_xdp_tx_queue_mapping(struct igb_adapter *adapter) +struct igb_ring *igb_xdp_tx_queue_mapping(struct igb_adapter *adapter) { unsigned int r_idx = smp_processor_id(); @@ -3025,11 +3026,11 @@ static int igb_xdp_xmit(struct net_device *dev, int n, nxmit++; } - __netif_tx_unlock(nq); - if (unlikely(flags & XDP_XMIT_FLUSH)) igb_xdp_ring_update_tail(tx_ring); + __netif_tx_unlock(nq); + return nxmit; } @@ -6631,7 +6632,7 @@ static int igb_change_mtu(struct net_device *netdev, int new_mtu) struct igb_adapter *adapter = netdev_priv(netdev); int max_frame = new_mtu + IGB_ETH_PKT_HDR_PAD; - if (adapter->xdp_prog) { + if (igb_xdp_is_enabled(adapter)) { int i; for (i = 0; i < adapter->num_rx_queues; i++) { @@ -8600,9 +8601,9 @@ static struct sk_buff *igb_build_skb(struct igb_ring *rx_ring, return skb; } -static struct sk_buff *igb_run_xdp(struct igb_adapter *adapter, - struct igb_ring *rx_ring, - struct xdp_buff *xdp) +struct sk_buff *igb_run_xdp(struct igb_adapter *adapter, + struct igb_ring *rx_ring, + struct xdp_buff *xdp) { int err, result = IGB_XDP_PASS; struct bpf_prog *xdp_prog; @@ -8798,9 +8799,9 @@ static bool igb_cleanup_headers(struct igb_ring *rx_ring, * order to populate the hash, checksum, VLAN, timestamp, protocol, and * other fields within the skb. **/ -static void igb_process_skb_fields(struct igb_ring *rx_ring, - union e1000_adv_rx_desc *rx_desc, - struct sk_buff *skb) +void igb_process_skb_fields(struct igb_ring *rx_ring, + union e1000_adv_rx_desc *rx_desc, + struct sk_buff *skb) { struct net_device *dev = rx_ring->netdev; From patchwork Tue Jul 4 09:59:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sriram Yagnaraman X-Patchwork-Id: 13300937 X-Patchwork-Delegate: kuba@kernel.org Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 639D1566C; Tue, 4 Jul 2023 10:04:58 +0000 (UTC) Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-db3eur04on2109.outbound.protection.outlook.com [40.107.6.109]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A8A0DE52; Tue, 4 Jul 2023 03:04:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=b0/jljdPyQZK4acWFKB2eKv7dhHT8A7wHDydEd5zRKLovVZHgQ3wU/tlI1GVGAxKOKGF2Ljji1MgwACj5enMKxNGOpeHyGYiIpjo9Nqdfr88QY1D+7gr6r33B5LQxZEjCssp9UGhFf7rFd+g2ic2vsGZd9MCJDkf1EKB3SHWpdoocTWWxKu9JNVbfHlezrASZ+zT1uIpXaiKNVAT1qTHm8iqwmvx70+qeIEXs/EDdBwBu2yOs+GRRPu5sfvTRc/pwZahYxvfIj1188rCrfN/8ofp6dL2euJZMgL2J1fwF8j0D5Zh2hrmlFyryQYHHyspIGpDCUj5EvHhsVrsHeU8cw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=hyccUYh2pEpQWEmch0Qj4pn+fn6yEzbnJjPn0aC669Q=; b=JcCJL3FUIdsByRdBTuxPdivdIPM6uDjHN8gJCgR/rdqcm7YNCzTxvwYrE9yI24wSgTts3FC0919/nrBpBc5bQ6O7z2GT927iT38MncYOk2b3ynZMq8owfJbocN0ARo3wgoZ7LaRlOxATXugNjvCyyrJsQr68Elyn3jpDXHY5WnxlV7zyge6TEVWaonX5ybXRhTFfyRoGlta15xiLGoTDGLoS3kREMjauxhdGiWdYvobF3ByMeko3SaqTjOCACDB9PG5HNWCpdr5F24sd/gNcm2acUJyKrue+8QyWTwN6S6XoE7o4LbDFjqXVoFt4BEA5rZYgB1puqQty3SHNlLCjUw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=est.tech; dmarc=pass action=none header.from=est.tech; dkim=pass header.d=est.tech; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=estab.onmicrosoft.com; s=selector2-estab-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hyccUYh2pEpQWEmch0Qj4pn+fn6yEzbnJjPn0aC669Q=; b=bjKH72fMcCbUjTPixy8obj7lt+k9xaEDSHrJfasWdYPFwYvC9oV7kzfjdf9l5CHRF/1nlMRufnmk8BZlQYL1xdO+Crc0dyyV4b6jAdH+w/jaQEAQ8lHFke4O9SesS1geH90aVLfwfsc5UqX+7y4wsMswPDkMzKu3lRLsvGRUu8o= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=est.tech; Received: from DBBP189MB1433.EURP189.PROD.OUTLOOK.COM (2603:10a6:10:1e7::15) by DB3P189MB2624.EURP189.PROD.OUTLOOK.COM (2603:10a6:10:438::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6544.24; Tue, 4 Jul 2023 10:04:15 +0000 Received: from DBBP189MB1433.EURP189.PROD.OUTLOOK.COM ([fe80::57d:1bb3:9180:73b6]) by DBBP189MB1433.EURP189.PROD.OUTLOOK.COM ([fe80::57d:1bb3:9180:73b6%4]) with mapi id 15.20.6544.024; Tue, 4 Jul 2023 10:04:14 +0000 From: Sriram Yagnaraman To: Cc: intel-wired-lan@lists.osuosl.org, bpf@vger.kernel.org, netdev@vger.kernel.org, Jesse Brandeburg , Tony Nguyen , "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Sriram Yagnaraman Subject: [PATCH 2/4] igb: Introduce txrx ring enable/disable functions Date: Tue, 4 Jul 2023 11:59:13 +0200 Message-Id: <20230704095915.9750-3-sriram.yagnaraman@est.tech> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230704095915.9750-1-sriram.yagnaraman@est.tech> References: <20230704095915.9750-1-sriram.yagnaraman@est.tech> X-ClientProxiedBy: GV3PEPF000000D5.SWEP280.PROD.OUTLOOK.COM (2603:10a6:144:1:0:2:0:1e) To DBBP189MB1433.EURP189.PROD.OUTLOOK.COM (2603:10a6:10:1e7::15) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DBBP189MB1433:EE_|DB3P189MB2624:EE_ X-MS-Office365-Filtering-Correlation-Id: b768c1ce-32a8-4e80-b3a3-08db7c7605ac X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: UplPeRPknJnru7CfhLc+VJXr2gtqdnANp1XwaaBfI7prWplUuO9YHwwrs27r2rXaKyjPy1Tx62ZvRMh5M6uaMJHi8MWm3wgYb3DCUiyuHf0baX7VAZEl85DqBaDrIuM+/DUo5SqpURXRx8jPXz/uSl6BFkoIwfFDwErQUMfqzAZ9VC8+O2sM6FtV0vxLmTIL5bc6bumPMkFWA4v7EvGi6seN3EErZiFvivSqzKsknWT++se5w3aynm8MJZoxEXdgTbpYjmFExaVVL6rWiJmFvSIiB5V81CIA8btIMK1Aq62lQU1Cp9QZxYIscER6+l5ro4knpjWZbBg3hsomc7P+hVWwJGn0obsJTc9SsYl8fYbIFYTTW35eVRxhBOKOdov2TjuQo3/m2LoIKQRnuV6OgBbAFX2qeCgHZ2R+9RqAnZjHbfd/FHgY6JNcvixHN2piTz1pUNrUgSnp7qMEuSlwpJ8qbeOAXwuxcIpUnlzSNHPKGhX3h6d16Sk85Va41m7HsN3d2mtvk6HINYG7lkGzrR0cEkU/n8WZJ7t2etV/AUHyr4QcY3PBE7kagFDsn1MwJ8qr4iRkqqS3JAJznSFlaJef7PG09pFuI1iDhOxswKsKYo6++998pLe+43A9a2+TZ8MkXW1W0QmyHzqgmXW7wA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DBBP189MB1433.EURP189.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230028)(396003)(39830400003)(346002)(376002)(136003)(109986019)(451199021)(54906003)(6512007)(26005)(44832011)(186003)(86362001)(8676002)(7416002)(2906002)(8936002)(36756003)(6486002)(5660300002)(41300700001)(478600001)(38100700002)(1076003)(6506007)(66556008)(4326008)(66946007)(66476007)(70586007)(2616005)(83380400001)(316002)(266003)(37730700002);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: O5JoL16tYzx7zLNIk9ZuB3fJvr3j3rIT35lYmEfw30eajSztFvRL5MnuHywSl5ghnexPbw37RGkwuu1rRbZxvNAuYaRp1mVQ4sH0al6JM81OuWGW5ywFIcVHAqvsb7kDEPGibdIQGWyw3dNYGCollHm1aVAvWfM3spaspek02s584VB22fpGQ65smNENY4oBN0/akXFILFqkJWVAPoDuKG43S/G9Zm+DxFaGxjUGT66NnlFFNUe52QrgZaOZ/5NK2yDMp0faX2JsU+6yTHW6R8X+yp6+KBmR+7gNUMZMSZOJziAZX3WI/91PM/votCe4opBGNXX0RrzosyMWNXBx2/2QhVdqevkVnHAYGPml7jU/HyWYKYnBXWp/k7fyX9+UJyxRd8NExeeuCsP8Iplw4aaFHsvib5rWcyswGvFVzJAXQk2mhxgKfw0G/39oXpkR8QQYg+67aOhauRAesQGXfrc77zxpd+5Cn4eZrMZew/kIUB7/ydJPt0sdQXdEetbk5hquVGJCdf58aISwbKHQiMuUIo9XGP9L6orKr6Ab+lHxotnwmCAeVzVsn/BY+4v/PUJhiaYEq8QTsSrZbGYxY8uMRIqSn712859/2VJYbu2yKtHq7LhtQ4ZNr2FZnoLF+cppmr+YTOo90VjM5HjDlOfJNWXJsNoQ8wbtw308n+BOGc7NZ4KsamOneZPDrEYtVPrRTN846cWpQIb19PdwpXGtbV1JK6UbY4jPM5kEmdIjJogss8OIXVGJRAQHu9VBxyyhsY14U/mHr3aqGJm7ng0vd7UeGJOwtNOBHSBeBFAVSfbZa5NeGmpndyBLefpj01WxBCdzuZkgxzSJ52DXCcRzb7jNLHiPFy5uUXLy4yjQ1eMbYLpOKtW7ltNIHMxw/TosIdpMMdxvu6pXJA6Ilqk4ndhglUfuHZXl3sMkwfqwrbcFxQG/80MoyGJrxGONAPIdEpt3VHARcuGhrT5XRNvvgAWd2VB9T7sgwLbF3viDPKbES4D1vToXN4jQJ1qmXQFqlONtwymWMOt0FCkSQdPvrW3Z7sXFLUGG9oAawnMeCXpQBNeqtHIHAxOfkhSg7EKYlGudx+bFfds+6QWj4lkNmTYzfnz9xPoOBT648rTKq4fCB1QJ+6Drx/YrXh1X47vY0EluTkrDjadSB5E5/nL4Mx0S+KQoXgZ7F1vGHAHD4qOx8BccXZJGXLud4DGVq5Pa+kYtlFKmylMAKATms/J0hqj7o6zG5fhPOLcPafKrqIdj9xiskIh6MuO4QINhTMzGqBGLi1ElmNX+8wSA6ZlXleD6pPWeNvLuzY0jSEL/vJK8UWaiVLPiDGRBJBTA7jvOpJSmWWputZw8Duum9jL4oAuwqSBPCJLxQZc2QeFvjqf7duAY5huYELQnZJli6YTBtBDtx1NILwJeg0Z23pGYWCE025prMqLjx8s1vNPiA+rK9GyIre4GQqf1ZYSY5+Usy0K+tOEagXlE0u3CVWePhYfPcKIXaGJtLmtm8cxYRVsPsd+bi3BMUV4w4w4bz7gL7s+cQ035b/Y1E/JgeAvq9nsqHN3cRaaLTzFjfutJaNn/ZSqgD9x4t0I/acD32qxiYGAdSvLnhFBT5jwfuA== X-OriginatorOrg: est.tech X-MS-Exchange-CrossTenant-Network-Message-Id: b768c1ce-32a8-4e80-b3a3-08db7c7605ac X-MS-Exchange-CrossTenant-AuthSource: DBBP189MB1433.EURP189.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jul 2023 10:04:14.7479 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: d2585e63-66b9-44b6-a76e-4f4b217d97fd X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: hmUtjT7qPc8KEY3NYztXwNHmqrRANM94z/yyd4aIN43isbjwoCaZBQVChxHjtgdkCN9DU7HJNVYTYc1WGd3b0LsnJdVIqBX6fWFUR5NN4Qs= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3P189MB2624 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: kuba@kernel.org Add enable/disable functions for TX and RX rings, will be used in later patches when AF_XDP zero-copy support is added. Signed-off-by: Sriram Yagnaraman --- drivers/net/ethernet/intel/igb/igb.h | 5 ++- drivers/net/ethernet/intel/igb/igb_main.c | 41 +++++++++++++++++++++++ 2 files changed, 45 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/intel/igb/igb.h b/drivers/net/ethernet/intel/igb/igb.h index 94440af6cf4b..5fa011c6ef2f 100644 --- a/drivers/net/ethernet/intel/igb/igb.h +++ b/drivers/net/ethernet/intel/igb/igb.h @@ -384,7 +384,8 @@ enum e1000_ring_flags_t { IGB_RING_FLAG_RX_SCTP_CSUM, IGB_RING_FLAG_RX_LB_VLAN_BSWAP, IGB_RING_FLAG_TX_CTX_IDX, - IGB_RING_FLAG_TX_DETECT_HANG + IGB_RING_FLAG_TX_DETECT_HANG, + IGB_RING_FLAG_TX_DISABLED }; #define ring_uses_large_buffer(ring) \ @@ -735,6 +736,8 @@ void igb_free_tx_resources(struct igb_ring *); void igb_free_rx_resources(struct igb_ring *); void igb_configure_tx_ring(struct igb_adapter *, struct igb_ring *); void igb_configure_rx_ring(struct igb_adapter *, struct igb_ring *); +void igb_txrx_ring_disable(struct igb_adapter *adapter, u16 qid); +void igb_txrx_ring_enable(struct igb_adapter *adapter, u16 qid); void igb_setup_tctl(struct igb_adapter *); void igb_setup_rctl(struct igb_adapter *); void igb_setup_srrctl(struct igb_adapter *, struct igb_ring *); diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c index dadc3d423cfd..391c0eb136d9 100644 --- a/drivers/net/ethernet/intel/igb/igb_main.c +++ b/drivers/net/ethernet/intel/igb/igb_main.c @@ -4856,6 +4856,47 @@ static void igb_configure_rx(struct igb_adapter *adapter) } } +void igb_txrx_ring_disable(struct igb_adapter *adapter, u16 qid) +{ + struct e1000_hw *hw = &adapter->hw; + struct igb_ring *tx_ring = adapter->tx_ring[qid]; + struct igb_ring *rx_ring = adapter->rx_ring[qid]; + + set_bit(IGB_RING_FLAG_TX_DISABLED, &tx_ring->flags); + + wr32(E1000_TXDCTL(tx_ring->reg_idx), 0); + wr32(E1000_RXDCTL(rx_ring->reg_idx), 0); + + /* Rx/Tx share the same napi context. */ + napi_disable(&rx_ring->q_vector->napi); + + igb_clean_tx_ring(tx_ring); + igb_clean_rx_ring(rx_ring); + + memset(&rx_ring->rx_stats, 0, sizeof(rx_ring->rx_stats)); + memset(&tx_ring->tx_stats, 0, sizeof(tx_ring->tx_stats)); +} + +void igb_txrx_ring_enable(struct igb_adapter *adapter, u16 qid) +{ + struct igb_ring *tx_ring = adapter->tx_ring[qid]; + struct igb_ring *rx_ring = adapter->rx_ring[qid]; + + /* Rx/Tx share the same napi context. */ + napi_enable(&rx_ring->q_vector->napi); + + igb_configure_tx_ring(adapter, tx_ring); + igb_configure_rx_ring(adapter, rx_ring); + + /* call igb_desc_unused which always leaves + * at least 1 descriptor unused to make sure + * next_to_use != next_to_clean + */ + igb_alloc_rx_buffers(rx_ring, igb_desc_unused(rx_ring)); + + clear_bit(IGB_RING_FLAG_TX_DISABLED, &tx_ring->flags); +} + /** * igb_free_tx_resources - Free Tx Resources per Queue * @tx_ring: Tx descriptor ring for a specific queue From patchwork Tue Jul 4 09:59:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sriram Yagnaraman X-Patchwork-Id: 13300930 X-Patchwork-Delegate: kuba@kernel.org Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 65E085675; Tue, 4 Jul 2023 09:59:31 +0000 (UTC) Received: from EUR03-DBA-obe.outbound.protection.outlook.com (mail-dbaeur03on2127.outbound.protection.outlook.com [40.107.104.127]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BCBCBD8; Tue, 4 Jul 2023 02:59:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=auZM0887El8mWDXSgdZXcRPaYX92w0GplhlJd4UOmZDZed/9JCJrQNtQV5BHQGoUjhwvHdVYiqKeF8m14HBY3KayQc197mPjFCc0erXbd5WsqpxYWQMBW3C/NdLzTeN7w30GECQ+R1N/a0DxJTriuZ1AqSUe1RK9HXblL5Vk2Z2fDxvmB9gD4XJdtsA2prwKDTalkGwrAMwzfs5mCYiPDkF340uok07KmxezSttyAiPUMXySmdViNQ1php/U+QHG3QcTAugJtPqsNMUk3Lik1jXlUCNoBIsKRlMUY/RwZJRJXL9Jwmk5QUpmJ18TdnSNuxy6jE46G1ABZ1tFmHDCmg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=qw1EF8drZGNNP3tA0KIsCrzE+OpJ//eA1Ms/6WzqSR4=; b=AG8G0486BRqEtWohdXlvqqvy6R4TtsK6872fn5XTqc8HQCqa2DDhI+WdO62BQj24ZOLG+ajI/nEZb6nBBterGhFuaypxorJz3bLgNl/r8XqAd4tVdhxBDo5Il0iEgNv3d3ODLUATMDjZ/vMJedb34vAWcMkQ2ItbP5tUS2nR4e34HfdBpjwF5yGbmb8DLnVD9D+fE63VJxs/oPgpuG2aexcCU7aTOPF8XwuBd02eiYva8oX6btIf92DLLrX0zbz6VOobC6nzNZJ8+vltnbW/fgqSsBqfncln7RaXIoxdWl/1lPfCniBCJgEJxZMqjQyBCPpVJ8y+APwKhziNVKbx5w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=est.tech; dmarc=pass action=none header.from=est.tech; dkim=pass header.d=est.tech; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=estab.onmicrosoft.com; s=selector2-estab-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qw1EF8drZGNNP3tA0KIsCrzE+OpJ//eA1Ms/6WzqSR4=; b=UIokEjrbUPGqwVTCUaUiMWr1gnFxOH4i0YJhhehppSh6/iVuiL9Mw3PVSrqcKesoYarPdGxAgvqjTuY+sACDDch7UOfMA4FUBsFGHv3c0nsP+B4gY6gPTcse2OPZgiZzDeY7AwP3ouEGYRPeFy99kcWufXYaNr8sRIE9VA2ELqg= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=est.tech; Received: from DBBP189MB1433.EURP189.PROD.OUTLOOK.COM (2603:10a6:10:1e7::15) by VI1P18901MB0783.EURP189.PROD.OUTLOOK.COM (2603:10a6:800:123::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6544.24; Tue, 4 Jul 2023 09:59:24 +0000 Received: from DBBP189MB1433.EURP189.PROD.OUTLOOK.COM ([fe80::57d:1bb3:9180:73b6]) by DBBP189MB1433.EURP189.PROD.OUTLOOK.COM ([fe80::57d:1bb3:9180:73b6%4]) with mapi id 15.20.6544.024; Tue, 4 Jul 2023 09:59:23 +0000 From: Sriram Yagnaraman To: Cc: intel-wired-lan@lists.osuosl.org, bpf@vger.kernel.org, netdev@vger.kernel.org, Jesse Brandeburg , Tony Nguyen , "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Sriram Yagnaraman Subject: [PATCH 3/4] igb: add AF_XDP zero-copy Rx support Date: Tue, 4 Jul 2023 11:59:14 +0200 Message-Id: <20230704095915.9750-4-sriram.yagnaraman@est.tech> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230704095915.9750-1-sriram.yagnaraman@est.tech> References: <20230704095915.9750-1-sriram.yagnaraman@est.tech> X-ClientProxiedBy: GV2PEPF000000FF.SWEP280.PROD.OUTLOOK.COM (2603:10a6:144:1:0:1:0:9) To DBBP189MB1433.EURP189.PROD.OUTLOOK.COM (2603:10a6:10:1e7::15) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DBBP189MB1433:EE_|VI1P18901MB0783:EE_ X-MS-Office365-Filtering-Correlation-Id: a5eb64e5-aaf3-4fbc-9734-08db7c75577c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: nCsPodp5VJG7IasUbvOh5LE/Rasrvioblpo0lmKXRW5ngS64xBnZMJOt6urMWsUS6bMUYUa7KPZY6lWahuiu2T28DJDaX+fXj+S3R0rZGbGRFdWmoZrWjtV9rtWDsG1ztuzZzo8XfIyCntDC8Jx32/BpAZ5CfiCkXidHl0Zf8txaXkdR0mclfzmo8HWg6qcXHPyDIUN7PqYzNyHENYZdShquC8pJdRneaOZtXDR6wn/8AYqCgGlkbr7nAqVSQlOi9W+P/m2fjp19toN+OzT264ta9gtGG7Jy1RkGKeJMoeWdU/JMtXkNTDj5/ozpAMn3CHIAnWF32O2l2pyjym5JcAwWJOgcoCZH091Ce8OdFJgv0VQFjn+mxD2WajL2P9q17eYPnlLyB91Gl5NKrnVDpCfk+/lyMO60bmltPMYXMmDT7uWZXR6m5H4NawH41l/BzdJkLumjs78dv+VEc6cRjZTwVvtXbvuKPZ9OKWFR19//e2Z3rPZIUVqzVfIIXJsYtQw67ax8av4VfeUb79B9YGJK0EYBzOu9PS8M4kQk/UK5+F0dJOoHULypay75aR8waEvP3E2U5Pw0uDtHAdJJLQtxJEpfjAg+ZRzyjOtFu7L/4hCZHmQGbT1ctL+fEub/WKE9S62zSZJPY/H51BeC4g== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DBBP189MB1433.EURP189.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230028)(396003)(39830400003)(376002)(346002)(136003)(451199021)(109986019)(316002)(41300700001)(6512007)(6486002)(26005)(83380400001)(6506007)(1076003)(2616005)(186003)(478600001)(66899021)(6666004)(54906003)(38100700002)(44832011)(66556008)(66476007)(4326008)(66946007)(70586007)(5660300002)(7416002)(86362001)(30864003)(36756003)(8936002)(2906002)(8676002)(266003)(37730700002);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: +KBm1UWvEbSTbgcgE9zGGSvMh3MhmbLROocnw97GCRtD9m4vUug3bf8s98FHQxDPGvEtv7+/VQ2MAHi7cEuFXLHnzer++1LCP8BJ1RTzP8bgS08lU46syFWI4cJcULL64EvOLBj+PPXXPOWuoVBvfQyzikaLKKTDwIOh1u+umNXDSI0A8l0tw1ZOQIeOdMQBJwTfRxGSgN2kShghGsQAiomsNTRAMpWUjDPWddz0akHBsL6VSR5Qluaql9xNmIFd2TDsFO36ef1E6r6rrnMjIQycRDnqyof94u/ZZO6HDSPaj7xfeGqszuxiaUYbfBCJkrPP8Bty0K1fP9jJM61qurw1xUqa9ejk1lloD67beOtU2fLALrnYm2OEvRGP38K1X4tLAYYu0oQ5/nEIFeFqKgrUSWr9jUgb8KZcsQS23eBIqtDT40k0+qUH+WBKsyRqcqarjMbpWCyblpHvgfhI1WVuAvM425FxeZnA+TxqwIKmibpUcujUUn++qIwjH7+e/pf1Xx9a4nbKnu0xpdgHbwfepc84PuO7UmooxxSiYYek5sGsHOj7jIpIiHQIR33MsfJj3eRS+9kysCS0qJwb8vhLIijXUEJuMd3zAgAT72hdlX0CAVnpe43g3Z3UYx4Tmo5fP0fYLvjHbwJJCqoHxDXIoLq8u2IB9JB85azaG6hZVLfzB1mr2xR9fx/YWE9QmX1nxDnfbYOLxgr9Y/uDlSsupDW87WpbpEidd3YllyCFjuJNn7Eb3GSFNGhXdFmPt8Gf3ER3r3kTopfn+PcZKyV1r6EZevHB8uUPr79tUUcqJ60xhtBu5rT895YXBflQdd+5pKeyjVmvpEaeFXqIhGwagHdwNdyXt3b/BD4BO8ELVl/DuIYnknmijqbELoLgnKbWe0TY+MWYI9xYXgFTaqFwzGpbtQkW30mraHSQqReCwwOHoWYMTfMHEVwHtZs5uVM0qJ8Ds3BgqGLlLFfseG6yqEB0Pp+zU26ZJ7TCTGnV0lrs/G8PtsTpGYEP7AhZXrOkBbHwCK1m2/46jgHlnDwBzbsH57+19TE5mwagD1C9vuRk1CvrX9v0+/wEyaT5LBS+FNp9W+wLSyWwwFaZeTf83YxXumbwVhaaYxfGZ6JFmufPDa0Y37y4ypMnSOPLpMGqO7ZmyGpPVN/iVsbZ1y8NdA1Y14ZJvVVHSEazdyeyfxw4zlDB7XOc4MLSl7An2ic6+On+/S7qNbR9kHap7bNxdYIFUNbwTYefUvw9f/j4qZL49/Ba3KxTMi2NQRG8q9+YXXcf4VTDpB+K4ptD1lhX1ZJk/Lm795aigxi6JgZm7rAA+QrW4yBxHC6BlXJHWqMFOqER3KHN3NF9xBm08dm7TckGmj7GGKwjmFyU2aA7os/1/9mwhfdc3HPMA9UHcbvDmKiVxTtdMHboChWpfRr7PFvkwmZbVV96lPi+l+5q5UGI8Pk/+Pptp1fw8czGtrMe7J8m/+o9zvczLveJJgY+WfY3PWIA4yklXk6e10mk5Uu5/2FB3GOdKr3fzHKhIahb1I/ICoWgVp5QIRXKWMWLlRX42lvRDrRrX1bPbdsIKH5TQB2J/BgBJjkJ5ETaEnLc1gcBDrIfbFrcf7FQwg== X-OriginatorOrg: est.tech X-MS-Exchange-CrossTenant-Network-Message-Id: a5eb64e5-aaf3-4fbc-9734-08db7c75577c X-MS-Exchange-CrossTenant-AuthSource: DBBP189MB1433.EURP189.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jul 2023 09:59:22.5405 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: d2585e63-66b9-44b6-a76e-4f4b217d97fd X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: qF12Gg3QEicec8tsGpQjNP+qhyHgbyTABUryP3riBJEKyowyWqOfBjDOugKU4gm/YX18ebhlGp17j8y78DcHDqA2cQaCSg1eBBo/HGycWpU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1P18901MB0783 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: kuba@kernel.org Add support for AF_XDP zero-copy receive path. Add IGB_RING_FLAG_AF_XDP_ZC ring flag to indicate that a ring has AF_XDP zero-copy support. This flag is used in igb_configure_rx_ring to register XSK_BUFF_POOL (if zero-copy is enabled) memory model or the default PAGE_SHARED model otherwise. When AF_XDP zero-copy is enabled, the rx buffers are allocated from the xsk buff pool using igb_alloc_rx_buffers_zc. Signed-off-by: Sriram Yagnaraman --- drivers/net/ethernet/intel/igb/Makefile | 2 +- drivers/net/ethernet/intel/igb/igb.h | 34 +- drivers/net/ethernet/intel/igb/igb_main.c | 72 ++++- drivers/net/ethernet/intel/igb/igb_xsk.c | 373 ++++++++++++++++++++++ 4 files changed, 463 insertions(+), 18 deletions(-) create mode 100644 drivers/net/ethernet/intel/igb/igb_xsk.c diff --git a/drivers/net/ethernet/intel/igb/Makefile b/drivers/net/ethernet/intel/igb/Makefile index 394c1e0656b9..86d25dba507d 100644 --- a/drivers/net/ethernet/intel/igb/Makefile +++ b/drivers/net/ethernet/intel/igb/Makefile @@ -8,4 +8,4 @@ obj-$(CONFIG_IGB) += igb.o igb-objs := igb_main.o igb_ethtool.o e1000_82575.o \ e1000_mac.o e1000_nvm.o e1000_phy.o e1000_mbx.o \ - e1000_i210.o igb_ptp.o igb_hwmon.o + e1000_i210.o igb_ptp.o igb_hwmon.o igb_xsk.o diff --git a/drivers/net/ethernet/intel/igb/igb.h b/drivers/net/ethernet/intel/igb/igb.h index 5fa011c6ef2f..4f474d7338b5 100644 --- a/drivers/net/ethernet/intel/igb/igb.h +++ b/drivers/net/ethernet/intel/igb/igb.h @@ -20,6 +20,7 @@ #include #include +#include struct igb_adapter; @@ -86,6 +87,7 @@ struct igb_adapter; #define IGB_XDP_CONSUMED BIT(0) #define IGB_XDP_TX BIT(1) #define IGB_XDP_REDIR BIT(2) +#define IGB_XDP_EXIT BIT(3) struct vf_data_storage { unsigned char vf_mac_addresses[ETH_ALEN]; @@ -278,14 +280,23 @@ struct igb_tx_buffer { }; struct igb_rx_buffer { - dma_addr_t dma; - struct page *page; + union { + struct { + struct sk_buff *skb; + dma_addr_t dma; + struct page *page; #if (BITS_PER_LONG > 32) || (PAGE_SIZE >= 65536) - __u32 page_offset; + __u32 page_offset; #else - __u16 page_offset; + __u16 page_offset; #endif - __u16 pagecnt_bias; + __u16 pagecnt_bias; + }; + struct { + struct xdp_buff *xdp; + }; + }; + }; struct igb_tx_queue_stats { @@ -357,6 +368,7 @@ struct igb_ring { }; }; struct xdp_rxq_info xdp_rxq; + struct xsk_buff_pool *xsk_pool; } ____cacheline_internodealigned_in_smp; struct igb_q_vector { @@ -385,7 +397,8 @@ enum e1000_ring_flags_t { IGB_RING_FLAG_RX_LB_VLAN_BSWAP, IGB_RING_FLAG_TX_CTX_IDX, IGB_RING_FLAG_TX_DETECT_HANG, - IGB_RING_FLAG_TX_DISABLED + IGB_RING_FLAG_TX_DISABLED, + IGB_RING_FLAG_AF_XDP_ZC }; #define ring_uses_large_buffer(ring) \ @@ -823,4 +836,13 @@ int igb_add_mac_steering_filter(struct igb_adapter *adapter, int igb_del_mac_steering_filter(struct igb_adapter *adapter, const u8 *addr, u8 queue, u8 flags); +struct xsk_buff_pool *igb_xsk_pool(struct igb_adapter *adapter, + struct igb_ring *ring); +int igb_xsk_pool_setup(struct igb_adapter *adapter, + struct xsk_buff_pool *pool, + u16 qid); +bool igb_alloc_rx_buffers_zc(struct igb_ring *rx_ring, u16 count); +int igb_clean_rx_irq_zc(struct igb_q_vector *q_vector, const int budget); +int igb_xsk_wakeup(struct net_device *dev, u32 qid, u32 flags); + #endif /* _IGB_H_ */ diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c index 391c0eb136d9..f4dbb75d6eac 100644 --- a/drivers/net/ethernet/intel/igb/igb_main.c +++ b/drivers/net/ethernet/intel/igb/igb_main.c @@ -2013,7 +2013,9 @@ static void igb_configure(struct igb_adapter *adapter) */ for (i = 0; i < adapter->num_rx_queues; i++) { struct igb_ring *ring = adapter->rx_ring[i]; - igb_alloc_rx_buffers(ring, igb_desc_unused(ring)); + ring->xsk_pool ? + igb_alloc_rx_buffers_zc(ring, igb_desc_unused(ring)) : + igb_alloc_rx_buffers(ring, igb_desc_unused(ring)); } } @@ -2931,9 +2933,14 @@ static int igb_xdp_setup(struct net_device *dev, struct netdev_bpf *bpf) static int igb_xdp(struct net_device *dev, struct netdev_bpf *xdp) { + struct igb_adapter *adapter = netdev_priv(dev); + switch (xdp->command) { case XDP_SETUP_PROG: return igb_xdp_setup(dev, xdp); + case XDP_SETUP_XSK_POOL: + return igb_xsk_pool_setup(adapter, xdp->xsk.pool, + xdp->xsk.queue_id); default: return -EINVAL; } @@ -3010,6 +3017,9 @@ static int igb_xdp_xmit(struct net_device *dev, int n, if (unlikely(!tx_ring)) return -ENXIO; + if (unlikely(test_bit(IGB_RING_FLAG_TX_DISABLED, &tx_ring->flags))) + return -ENXIO; + nq = txring_txq(tx_ring); __netif_tx_lock(nq, cpu); @@ -3060,6 +3070,7 @@ static const struct net_device_ops igb_netdev_ops = { .ndo_setup_tc = igb_setup_tc, .ndo_bpf = igb_xdp, .ndo_xdp_xmit = igb_xdp_xmit, + .ndo_xsk_wakeup = igb_xsk_wakeup, }; /** @@ -3356,7 +3367,8 @@ static int igb_probe(struct pci_dev *pdev, const struct pci_device_id *ent) netdev->priv_flags |= IFF_SUPP_NOFCS; netdev->priv_flags |= IFF_UNICAST_FLT; - netdev->xdp_features = NETDEV_XDP_ACT_BASIC | NETDEV_XDP_ACT_REDIRECT; + netdev->xdp_features = NETDEV_XDP_ACT_BASIC | NETDEV_XDP_ACT_REDIRECT | + NETDEV_XDP_ACT_XSK_ZEROCOPY; /* MTU range: 68 - 9216 */ netdev->min_mtu = ETH_MIN_MTU; @@ -4734,12 +4746,17 @@ void igb_setup_srrctl(struct igb_adapter *adapter, struct igb_ring *ring) struct e1000_hw *hw = &adapter->hw; int reg_idx = ring->reg_idx; u32 srrctl = 0; + u32 buf_size; - srrctl = IGB_RX_HDR_LEN << E1000_SRRCTL_BSIZEHDRSIZE_SHIFT; - if (ring_uses_large_buffer(ring)) - srrctl |= IGB_RXBUFFER_3072 >> E1000_SRRCTL_BSIZEPKT_SHIFT; + if (ring->xsk_pool) + buf_size = xsk_pool_get_rx_frame_size(ring->xsk_pool); + else if (ring_uses_large_buffer(ring)) + buf_size = IGB_RXBUFFER_3072; else - srrctl |= IGB_RXBUFFER_2048 >> E1000_SRRCTL_BSIZEPKT_SHIFT; + buf_size = IGB_RXBUFFER_2048; + + srrctl = IGB_RX_HDR_LEN << E1000_SRRCTL_BSIZEHDRSIZE_SHIFT; + srrctl |= buf_size >> E1000_SRRCTL_BSIZEPKT_SHIFT; srrctl |= E1000_SRRCTL_DESCTYPE_ADV_ONEBUF; if (hw->mac.type >= e1000_82580) srrctl |= E1000_SRRCTL_TIMESTAMP; @@ -4771,8 +4788,16 @@ void igb_configure_rx_ring(struct igb_adapter *adapter, u32 rxdctl = 0; xdp_rxq_info_unreg_mem_model(&ring->xdp_rxq); - WARN_ON(xdp_rxq_info_reg_mem_model(&ring->xdp_rxq, - MEM_TYPE_PAGE_SHARED, NULL)); + ring->xsk_pool = igb_xsk_pool(adapter, ring); + if (ring->xsk_pool) { + WARN_ON(xdp_rxq_info_reg_mem_model(&ring->xdp_rxq, + MEM_TYPE_XSK_BUFF_POOL, + NULL)); + xsk_pool_set_rxq_info(ring->xsk_pool, &ring->xdp_rxq); + } else { + WARN_ON(xdp_rxq_info_reg_mem_model(&ring->xdp_rxq, + MEM_TYPE_PAGE_SHARED, NULL)); + } /* disable the queue */ wr32(E1000_RXDCTL(reg_idx), 0); @@ -4892,7 +4917,9 @@ void igb_txrx_ring_enable(struct igb_adapter *adapter, u16 qid) * at least 1 descriptor unused to make sure * next_to_use != next_to_clean */ - igb_alloc_rx_buffers(rx_ring, igb_desc_unused(rx_ring)); + rx_ring->xsk_pool ? + igb_alloc_rx_buffers_zc(rx_ring, igb_desc_unused(rx_ring)) : + igb_alloc_rx_buffers(rx_ring, igb_desc_unused(rx_ring)); clear_bit(IGB_RING_FLAG_TX_DISABLED, &tx_ring->flags); } @@ -5069,6 +5096,13 @@ static void igb_clean_rx_ring(struct igb_ring *rx_ring) while (i != rx_ring->next_to_alloc) { struct igb_rx_buffer *buffer_info = &rx_ring->rx_buffer_info[i]; + if (rx_ring->xsk_pool) { + if (buffer_info->xdp) + xsk_buff_free(buffer_info->xdp); + buffer_info->xdp = NULL; + goto skip_for_xsk; + } + /* Invalidate cache lines that may have been written to by * device so that we avoid corrupting memory. */ @@ -5087,6 +5121,7 @@ static void igb_clean_rx_ring(struct igb_ring *rx_ring) __page_frag_cache_drain(buffer_info->page, buffer_info->pagecnt_bias); +skip_for_xsk: i++; if (i == rx_ring->count) i = 0; @@ -6518,6 +6553,9 @@ netdev_tx_t igb_xmit_frame_ring(struct sk_buff *skb, return NETDEV_TX_BUSY; } + if (unlikely(test_bit(IGB_RING_FLAG_TX_DISABLED, &tx_ring->flags))) + return NETDEV_TX_BUSY; + /* record the location of the first descriptor for this packet */ first = &tx_ring->tx_buffer_info[tx_ring->next_to_use]; first->type = IGB_TYPE_SKB; @@ -8260,7 +8298,9 @@ static int igb_poll(struct napi_struct *napi, int budget) clean_complete = igb_clean_tx_irq(q_vector, budget); if (q_vector->rx.ring) { - int cleaned = igb_clean_rx_irq(q_vector, budget); + int cleaned = q_vector->rx.ring->xsk_pool ? + igb_clean_rx_irq_zc(q_vector, budget) : + igb_clean_rx_irq(q_vector, budget); work_done += cleaned; if (cleaned >= budget) @@ -8668,8 +8708,13 @@ struct sk_buff *igb_run_xdp(struct igb_adapter *adapter, break; case XDP_REDIRECT: err = xdp_do_redirect(adapter->netdev, xdp, xdp_prog); - if (err) + if (err) { + if (xsk_uses_need_wakeup(rx_ring->xsk_pool) && err == -ENOBUFS) + result = IGB_XDP_EXIT; + else + result = IGB_XDP_CONSUMED; goto out_failure; + } result = IGB_XDP_REDIR; break; default: @@ -8929,6 +8974,8 @@ static int igb_clean_rx_irq(struct igb_q_vector *q_vector, const int budget) struct igb_adapter *adapter = q_vector->adapter; struct igb_ring *rx_ring = q_vector->rx.ring; struct sk_buff *skb = rx_ring->skb; + int cpu = smp_processor_id(); + struct netdev_queue *nq; unsigned int total_bytes = 0, total_packets = 0; u16 cleaned_count = igb_desc_unused(rx_ring); unsigned int xdp_xmit = 0; @@ -9059,7 +9106,10 @@ static int igb_clean_rx_irq(struct igb_q_vector *q_vector, const int budget) if (xdp_xmit & IGB_XDP_TX) { struct igb_ring *tx_ring = igb_xdp_tx_queue_mapping(adapter); + nq = txring_txq(tx_ring); + __netif_tx_lock(nq, cpu); igb_xdp_ring_update_tail(tx_ring); + __netif_tx_unlock(nq); } u64_stats_update_begin(&rx_ring->rx_syncp); diff --git a/drivers/net/ethernet/intel/igb/igb_xsk.c b/drivers/net/ethernet/intel/igb/igb_xsk.c new file mode 100644 index 000000000000..eae616e7608c --- /dev/null +++ b/drivers/net/ethernet/intel/igb/igb_xsk.c @@ -0,0 +1,373 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2018 Intel Corporation. */ + +#include +#include +#include + +#include "e1000_hw.h" +#include "igb.h" + +struct xsk_buff_pool *igb_xsk_pool(struct igb_adapter *adapter, + struct igb_ring *ring) +{ + int qid = ring->queue_index; + + if (!igb_xdp_is_enabled(adapter) || + !test_bit(IGB_RING_FLAG_AF_XDP_ZC, &ring->flags)) + return NULL; + + return xsk_get_pool_from_qid(adapter->netdev, qid); +} + +static int igb_xsk_pool_enable(struct igb_adapter *adapter, + struct xsk_buff_pool *pool, + u16 qid) +{ + struct net_device *netdev = adapter->netdev; + struct igb_ring *rx_ring; + bool if_running; + int err; + + if (qid >= adapter->num_rx_queues) + return -EINVAL; + + if (qid >= netdev->real_num_rx_queues) + return -EINVAL; + + err = xsk_pool_dma_map(pool, &adapter->pdev->dev, IGB_RX_DMA_ATTR); + if (err) + return err; + + rx_ring = adapter->rx_ring[qid]; + if_running = netif_running(adapter->netdev) && igb_xdp_is_enabled(adapter); + if (if_running) + igb_txrx_ring_disable(adapter, qid); + + set_bit(IGB_RING_FLAG_AF_XDP_ZC, &rx_ring->flags); + + if (if_running) { + igb_txrx_ring_enable(adapter, qid); + + /* Kick start the NAPI context so that receiving will start */ + err = igb_xsk_wakeup(adapter->netdev, qid, XDP_WAKEUP_RX); + if (err) { + clear_bit(IGB_RING_FLAG_AF_XDP_ZC, &rx_ring->flags); + xsk_pool_dma_unmap(pool, IGB_RX_DMA_ATTR); + return err; + } + } + + return 0; +} + +static int igb_xsk_pool_disable(struct igb_adapter *adapter, u16 qid) +{ + struct igb_ring *rx_ring; + struct xsk_buff_pool *pool; + bool if_running; + + pool = xsk_get_pool_from_qid(adapter->netdev, qid); + if (!pool) + return -EINVAL; + + rx_ring = adapter->rx_ring[qid]; + if_running = netif_running(adapter->netdev) && igb_xdp_is_enabled(adapter); + if (if_running) + igb_txrx_ring_disable(adapter, qid); + + xsk_pool_dma_unmap(pool, IGB_RX_DMA_ATTR); + clear_bit(IGB_RING_FLAG_AF_XDP_ZC, &rx_ring->flags); + + if (if_running) + igb_txrx_ring_enable(adapter, qid); + + return 0; +} + +int igb_xsk_pool_setup(struct igb_adapter *adapter, + struct xsk_buff_pool *pool, + u16 qid) +{ + return pool ? igb_xsk_pool_enable(adapter, pool, qid) : + igb_xsk_pool_disable(adapter, qid); +} + +bool igb_alloc_rx_buffers_zc(struct igb_ring *rx_ring, u16 count) +{ + union e1000_adv_rx_desc *rx_desc; + struct igb_rx_buffer *buffer_info; + u16 i = rx_ring->next_to_use; + dma_addr_t dma; + bool ok = true; + + /* nothing to do */ + if (!count) + return true; + + rx_desc = IGB_RX_DESC(rx_ring, i); + buffer_info = &rx_ring->rx_buffer_info[i]; + i -= rx_ring->count; + + do { + buffer_info->xdp = xsk_buff_alloc(rx_ring->xsk_pool); + if (!buffer_info->xdp) { + ok = false; + break; + } + + dma = xsk_buff_xdp_get_dma(buffer_info->xdp); + + /* Refresh the desc even if buffer_addrs didn't change + * because each write-back erases this info. + */ + rx_desc->read.pkt_addr = cpu_to_le64(dma); + + rx_desc++; + buffer_info++; + i++; + if (unlikely(!i)) { + rx_desc = IGB_RX_DESC(rx_ring, 0); + buffer_info = rx_ring->rx_buffer_info; + i -= rx_ring->count; + } + + /* clear the length for the next_to_use descriptor */ + rx_desc->wb.upper.length = 0; + + count--; + } while (count); + + i += rx_ring->count; + + if (rx_ring->next_to_use != i) { + rx_ring->next_to_use = i; + + /* Force memory writes to complete before letting h/w + * know there are new descriptors to fetch. (Only + * applicable for weak-ordered memory model archs, + * such as IA-64). + */ + wmb(); + writel(i, rx_ring->tail); + } + + return ok; +} + +static struct sk_buff *igb_construct_skb_zc(struct igb_ring *rx_ring, + struct xdp_buff *xdp, + ktime_t timestamp) +{ + unsigned int totalsize = xdp->data_end - xdp->data_meta; + unsigned int metasize = xdp->data - xdp->data_meta; + struct sk_buff *skb; + + net_prefetch(xdp->data_meta); + + /* allocate a skb to store the frags */ + skb = __napi_alloc_skb(&rx_ring->q_vector->napi, totalsize, + GFP_ATOMIC | __GFP_NOWARN); + if (unlikely(!skb)) + return NULL; + + if (timestamp) + skb_hwtstamps(skb)->hwtstamp = timestamp; + + memcpy(__skb_put(skb, totalsize), xdp->data_meta, + ALIGN(totalsize, sizeof(long))); + + if (metasize) { + skb_metadata_set(skb, metasize); + __skb_pull(skb, metasize); + } + + return skb; +} + +static void igb_update_ntc(struct igb_ring *rx_ring) +{ + u32 ntc = rx_ring->next_to_clean + 1; + + /* fetch, update, and store next to clean */ + ntc = (ntc < rx_ring->count) ? ntc : 0; + rx_ring->next_to_clean = ntc; + + prefetch(IGB_RX_DESC(rx_ring, ntc)); +} + +int igb_clean_rx_irq_zc(struct igb_q_vector *q_vector, const int budget) +{ + struct igb_adapter *adapter = q_vector->adapter; + struct igb_ring *rx_ring = q_vector->rx.ring; + struct sk_buff *skb; + int cpu = smp_processor_id(); + struct netdev_queue *nq; + unsigned int total_bytes = 0, total_packets = 0; + u16 cleaned_count = igb_desc_unused(rx_ring); + unsigned int xdp_xmit = 0; + bool failure = false; + + while (likely(total_packets < budget)) { + union e1000_adv_rx_desc *rx_desc; + struct igb_rx_buffer *rx_buffer; + ktime_t timestamp = 0; + unsigned int size; + + /* return some buffers to hardware, one at a time is too slow */ + if (cleaned_count >= IGB_RX_BUFFER_WRITE) { + igb_alloc_rx_buffers_zc(rx_ring, cleaned_count); + cleaned_count = 0; + } + + rx_desc = IGB_RX_DESC(rx_ring, rx_ring->next_to_clean); + size = le16_to_cpu(rx_desc->wb.upper.length); + if (!size) + break; + + /* This memory barrier is needed to keep us from reading + * any other fields out of the rx_desc until we know the + * descriptor has been written back + */ + dma_rmb(); + + rx_buffer = &rx_ring->rx_buffer_info[rx_ring->next_to_clean]; + rx_buffer->xdp->data_end = rx_buffer->xdp->data + size; + xsk_buff_dma_sync_for_cpu(rx_buffer->xdp, rx_ring->xsk_pool); + + /* pull rx packet timestamp if available and valid */ + if (igb_test_staterr(rx_desc, E1000_RXDADV_STAT_TSIP)) { + int ts_hdr_len; + + ts_hdr_len = igb_ptp_rx_pktstamp(rx_ring->q_vector, + rx_buffer->xdp->data, + ×tamp); + + rx_buffer->xdp->data += ts_hdr_len; + rx_buffer->xdp->data_meta += ts_hdr_len; + size -= ts_hdr_len; + } + + skb = igb_run_xdp(adapter, rx_ring, rx_buffer->xdp); + + if (IS_ERR(skb)) { + unsigned int xdp_res = -PTR_ERR(skb); + + if (likely(xdp_res & (IGB_XDP_TX | IGB_XDP_REDIR))) { + xdp_xmit |= xdp_res; + } else if (xdp_res == IGB_XDP_EXIT) { + failure = true; + break; + } else if (xdp_res == IGB_XDP_CONSUMED) { + xsk_buff_free(rx_buffer->xdp); + } + + total_packets++; + total_bytes += size; + + rx_buffer->xdp = NULL; + cleaned_count++; + igb_update_ntc(rx_ring); + continue; + } + + skb = igb_construct_skb_zc(rx_ring, rx_buffer->xdp, timestamp); + + /* exit if we failed to retrieve a buffer */ + if (!skb) { + rx_ring->rx_stats.alloc_failed++; + break; + } + + xsk_buff_free(rx_buffer->xdp); + rx_buffer->xdp = NULL; + cleaned_count++; + igb_update_ntc(rx_ring); + + if (eth_skb_pad(skb)) + continue; + + /* probably a little skewed due to removing CRC */ + total_bytes += skb->len; + + /* populate checksum, timestamp, VLAN, and protocol */ + igb_process_skb_fields(rx_ring, rx_desc, skb); + + napi_gro_receive(&q_vector->napi, skb); + + /* reset skb pointer */ + skb = NULL; + + /* update budget accounting */ + total_packets++; + } + + if (xdp_xmit & IGB_XDP_REDIR) + xdp_do_flush(); + + if (xdp_xmit & IGB_XDP_TX) { + struct igb_ring *tx_ring = igb_xdp_tx_queue_mapping(adapter); + + nq = txring_txq(tx_ring); + __netif_tx_lock(nq, cpu); + igb_xdp_ring_update_tail(tx_ring); + __netif_tx_unlock(nq); + } + + u64_stats_update_begin(&rx_ring->rx_syncp); + rx_ring->rx_stats.packets += total_packets; + rx_ring->rx_stats.bytes += total_bytes; + u64_stats_update_end(&rx_ring->rx_syncp); + q_vector->rx.total_packets += total_packets; + q_vector->rx.total_bytes += total_bytes; + + if (cleaned_count) + igb_alloc_rx_buffers_zc(rx_ring, cleaned_count); + + if (xsk_uses_need_wakeup(rx_ring->xsk_pool)) { + if (failure || rx_ring->next_to_clean == rx_ring->next_to_use) + xsk_set_rx_need_wakeup(rx_ring->xsk_pool); + else + xsk_clear_rx_need_wakeup(rx_ring->xsk_pool); + + return (int)total_packets; + } + return failure ? budget : (int)total_packets; +} + +int igb_xsk_wakeup(struct net_device *dev, u32 qid, u32 flags) +{ + struct igb_adapter *adapter = netdev_priv(dev); + struct igb_ring *ring; + struct e1000_hw *hw = &adapter->hw; + u32 eics = 0; + + if (test_bit(__IGB_DOWN, &adapter->state)) + return -ENETDOWN; + + if (!igb_xdp_is_enabled(adapter)) + return -EINVAL; + + if (qid >= adapter->num_tx_queues) + return -EINVAL; + + ring = adapter->tx_ring[qid]; + + if (test_bit(IGB_RING_FLAG_TX_DISABLED, &ring->flags)) + return -ENETDOWN; + + if (!ring->xsk_pool) + return -EINVAL; + + if (!napi_if_scheduled_mark_missed(&ring->q_vector->napi)) { + /* Cause software interrupt to ensure Rx ring is cleaned */ + if (adapter->flags & IGB_FLAG_HAS_MSIX) { + eics |= ring->q_vector->eims_value; + wr32(E1000_EICS, eics); + } else { + wr32(E1000_ICS, E1000_ICS_RXDMT0); + } + } + + return 0; +} From patchwork Tue Jul 4 09:59:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sriram Yagnaraman X-Patchwork-Id: 13300929 X-Patchwork-Delegate: kuba@kernel.org Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3FD8B539F; Tue, 4 Jul 2023 09:59:29 +0000 (UTC) Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05on2127.outbound.protection.outlook.com [40.107.21.127]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 97BCEE3; Tue, 4 Jul 2023 02:59:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bdZ8zNqVbdVPmt3nmC7wb6G8rt32EUEruWsaYoKFZ+f7WZ0CUORAIovKfpb+pQowlB+ZZgy8MY1BOkkW+QNZIf+34wdv2HNciGqTaOsGSvS+1PbjJqXsiOJAHK47ZDCfUtkhUf2JVSXt8Kd3V//A5WpzIGaA3UVom2cCc8Df/HKwcuqg1vErMtqDd55so0BqJF9LYN0qKTBymHUy0+1UkrwosvWWQ35voao2q7o/2MwqFZ0sH6ET8kAIIvby+Iyjg7L93si2lciKtLlOXU71X9u+sLfii6XNH1Oc/GKeDRatkH+SMoidfNTduGCXs9JymlUVAknXG98jI3SRcXnGmg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=cVLebntA80EuG73ibgsfp75GvO+YyAx6vaDBJzrEMGA=; b=McUgYUIa7mNrn6Uop+CuOoCEw3kFssd/M8RIKCly+FGTCYn2UwYNGgAXRG02o8dkTEY/KBqqyvP+KLYk151uaCs8EEO2qZUWonNCGsiHRWUenMAOdXfwHJdEZGNigy03zF0Es49i+b746UQRpzoERVj07l6AkUkHgyjVHU3CaN6l8ydcl6LwZKKfcIuN4yTh+/xWSvqo3JigXgwQtgN2wsWZaISQlFKQ3MzULa/5UR+V5ejeqe84PXWz1EEb1Zeb99kJMqs/sRe6QICZYSPeVTSKKzWf68H+Dhh4xx8KlOQsPhqOzY3OMh0ZmN4k3ff64BVn2KAzzAezBJlErbB6IA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=est.tech; dmarc=pass action=none header.from=est.tech; dkim=pass header.d=est.tech; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=estab.onmicrosoft.com; s=selector2-estab-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cVLebntA80EuG73ibgsfp75GvO+YyAx6vaDBJzrEMGA=; b=hqqvrIlR7uyTF0kIiOncQyVzhE6NoeImv20WRC/Z/NdZ6n+rGamZMCT8DwPFtpWhUw9pDMPC4RfrDwq7NxT7FtXBdF4QcavDfDsFYFxaRLw4nws6vydSF7wwTg3567259faz7GhqdbD/jVn4JXzH8ryEMjR/R9AnFIn0gvbwCCU= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=est.tech; Received: from DBBP189MB1433.EURP189.PROD.OUTLOOK.COM (2603:10a6:10:1e7::15) by AM9P189MB1571.EURP189.PROD.OUTLOOK.COM (2603:10a6:20b:306::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6544.24; Tue, 4 Jul 2023 09:59:23 +0000 Received: from DBBP189MB1433.EURP189.PROD.OUTLOOK.COM ([fe80::57d:1bb3:9180:73b6]) by DBBP189MB1433.EURP189.PROD.OUTLOOK.COM ([fe80::57d:1bb3:9180:73b6%4]) with mapi id 15.20.6544.024; Tue, 4 Jul 2023 09:59:23 +0000 From: Sriram Yagnaraman To: Cc: intel-wired-lan@lists.osuosl.org, bpf@vger.kernel.org, netdev@vger.kernel.org, Jesse Brandeburg , Tony Nguyen , "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Sriram Yagnaraman Subject: [PATCH 4/4] igb: add AF_XDP zero-copy Tx support Date: Tue, 4 Jul 2023 11:59:15 +0200 Message-Id: <20230704095915.9750-5-sriram.yagnaraman@est.tech> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230704095915.9750-1-sriram.yagnaraman@est.tech> References: <20230704095915.9750-1-sriram.yagnaraman@est.tech> X-ClientProxiedBy: GV2PEPF00000105.SWEP280.PROD.OUTLOOK.COM (2603:10a6:144:1:0:1:0:1c) To DBBP189MB1433.EURP189.PROD.OUTLOOK.COM (2603:10a6:10:1e7::15) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DBBP189MB1433:EE_|AM9P189MB1571:EE_ X-MS-Office365-Filtering-Correlation-Id: 3c385f38-bc44-4d77-9734-08db7c75577c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 6XreAdHUQepLfue36w89f/TQfgYJWi2pcdZbQjlfUj9DQB/czh/Xiorq+FaV04ZmZwopk7/h9IPYGMkyiZOa3ufKA3BnKU6CHG0zW+pwi5MY20t+8/2Try2NUZMk9FIM6jtgF4VgAC/VxkWSuUhH2OOJBCzRKyYfiOLOyvjMfOZ8qS9dylTfyarTcw50Rel+Y4133WanxoOcEWlQ3KdgrVswaBBxkYT1lW8drQpr8LQT1aZlKEyzjJquZ19GUa7Ff5q93VYQI6rj3lNcSLWU+ajcN+HJpQwrp1Blu6ZeNC8wTlNoqL70QHbYnuRRtrGXwiULhGpPR/hyCTyAfgJJczRBeP1BpzsAEGdckgYEqRhkindSbUoqbNQZOJ4BRjH4b2krI1MFH9pu6SSyJh6Ie/xWZXheOH5TktDchkJ8Z4E+T98IBl4iRkPNpALI0DAURzAGz2RYNHiiM7mHOFzsRffN5CZJ38PQf9CKBbUf3bAAI0R3inYN5XNCeK2+wjWClZff9NwidnKo4wxyCRYTINQOwGmgT+mhJFIwWlAL3wSqeYMxccfZdMqFSRMIqG4pbX9bTWOfClXPYNkc7WnLKyjig+njB98go9/E7zUCwKbJ0GBntINUVjSuH1Bl/VoOtRj8CzusjddgU0hAgiDNeeIruosHTMK+W6Kp0UJUtYA= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DBBP189MB1433.EURP189.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230028)(346002)(396003)(136003)(376002)(39840400004)(451199021)(109986019)(2906002)(41300700001)(7416002)(5660300002)(44832011)(8676002)(8936002)(36756003)(86362001)(6512007)(6666004)(6486002)(478600001)(83380400001)(26005)(186003)(2616005)(4326008)(1076003)(66556008)(54906003)(66476007)(70586007)(6506007)(66946007)(316002)(38100700002)(266003)(37730700002)(309714004);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: naV15ASSGc6eAJDwRP1ugPU0gouUMwPouQGcmijJWSAuN1z0+FT+gG5HK1hlJ6LBpTQDMweRwjLymSpD/k9/PRR7ui5jPQ57895bZPbFCkhRIfg/ACsleJ8s490JrFik6zRSaSq8HFVm4yvzr2HHHfOT1+sQN/3ayLEMIAGxF9EcCiqgTftTlF5QqOVYr2k9K+dfY/pTfQ1irAJkPGRpHR3rffGHtE8sYO9sptlpSg00yvmbYZXCGEwUEDnL18IpiNY0YdWVrvGpOZfh5O0efkfDcRSHt1Qp1dwZgNa/EgpJ5b833/IFtKOYV96qTa3YEzhlbqnUmbMvXeQMWjtPVK3mhfIfs2M39+V0Y+4TaJ08t+gbqBnWYDI/TqJqkiyWGmj2FtdRQrgOFVpNkdahECqlEhtE8FgCoFfV+H1VQVSdppWP/PDAQfDQtbUR6ZDLCjaBBB2iUctRR49drDpMzdIBZyfC3Fyx4L+CowWYHeWLjOz1az+44gK6RzqfQUHRp/WsHqak/emiRlAaguMGNlB7ls/3dt/WrXaxuJskCY8OxNa6my3WR5SiiGplTItZ3xGyzsXwT13rr2WxWtTmHMmyQmz2OOklZvghBI8Zgm+OC0B1Sx9z8e88kiiixPClOC5QoYniZLQavkf6LxyT16V0z67avM13uMQwezoyd8Pr6S87DY3Fq/7TjTnGJNnndw1trzTVnV7hOxoxYe98ZPs4AkC200icglJAAkcoJsjNOCeIpIvzOrEuBg0UoaQq4EkNalPiKDl0DwCzqdfa/YdU2xHcx+aAF87Sjacz9ihRvcVAlcwFwVBgfearuTO7D5tcRGmcm4XpYU52LgWqNik4Tgwr+2UIGceLaxYMWQP/KNnvo+8LOIoGSBydE6O4tqSg3jz6zbnNmS3ErANInQ7877rFbeOGpBAxnTngx8PeUDBEcwudFLmeyUCShC/gNeT9XoobwNwbQoEHN6raOXiGDRS7G9gYT+WFDBAlgti6G22EB167j79jv1d+kjlTSnudf5RUCIpusVia5iM7bdiJsGRpXSfOL22aQVUHpNlBunLuhPgzDttGOxfUbqQjteQ3A8qEpmTvS4YTQb8LXNYaU0UFJ0ZqdMzUnXyHKg+5Lkil+Sll6glvuOptXzDNwgy3Q29MqOEj9MVHk/aELdPwnc7sX+yKHSE4nANJNHh8bKdG7lC+M5fVlYhr3kiE3ZSOqyxyoVkoiTxDN8xyecpnpk79OU1BO8JTpYcI6/+pbjxyAJUCMDonnFpQEiTSnrdHH3TyZW+JjxXBAgPTtEh2iJXCc9COWvw4ZYWXSO0L70NfDlyBLVEmVtRYbmQ7nm2wKMukC85XXCXtLVRIo0+dgYrtCZUMngGkyr2c8X22GUScdhxQfT3mpfyFf8jBItxpQeI11H2bMEtyh+e+PGNVw+O+Tue31RpM3tFmh+A3LXVx/9yTxVaFxjugls+r87m85NsLLstOKpUj2vT1dA6SgwjpWv3QKdz/IWzN1F5peHnOkcF6cFjsN1zbu970v4Oqavu/5Lb8e3M16MLkWhVAsfYab1LKnTupUxpUDAtVS0q94WPqGGcCX9ssiafZgMdwUdMuJ+uxkeELfLHYCw== X-OriginatorOrg: est.tech X-MS-Exchange-CrossTenant-Network-Message-Id: 3c385f38-bc44-4d77-9734-08db7c75577c X-MS-Exchange-CrossTenant-AuthSource: DBBP189MB1433.EURP189.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jul 2023 09:59:22.5316 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: d2585e63-66b9-44b6-a76e-4f4b217d97fd X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: zV2eGMnNxdNkQbhB9Pq5JGPwbVsQNZzdsEP0/0dxJ6xyL8JJpvAT8O8Lz7KcQ3fBY8BThMfTVSweyKzRk7VpsdVS7FmVv0zmv/YLz2dOPeI= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM9P189MB1571 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: kuba@kernel.org Add support for AF_XDP zero-copy transmit path. A new TX buffer type IGB_TYPE_XSK is introduced to indicate that the Tx frame was allocated from the xsk buff pool, so igb_clean_tx_ring and igb_clean_tx_irq can clean the buffers correctly based on type. igb_xmit_zc performs the actual packet transmit when AF_XDP zero-copy is enabled. We share the TX ring between slow path, XDP and AF_XDP zero-copy, so we use the netdev queue lock to ensure mutual exclusion. Signed-off-by: Sriram Yagnaraman --- drivers/net/ethernet/intel/igb/igb.h | 2 + drivers/net/ethernet/intel/igb/igb_main.c | 44 +++++++++++++-- drivers/net/ethernet/intel/igb/igb_xsk.c | 67 ++++++++++++++++++++++- 3 files changed, 105 insertions(+), 8 deletions(-) diff --git a/drivers/net/ethernet/intel/igb/igb.h b/drivers/net/ethernet/intel/igb/igb.h index 4f474d7338b5..564706ab0646 100644 --- a/drivers/net/ethernet/intel/igb/igb.h +++ b/drivers/net/ethernet/intel/igb/igb.h @@ -257,6 +257,7 @@ enum igb_tx_flags { enum igb_tx_buf_type { IGB_TYPE_SKB = 0, IGB_TYPE_XDP, + IGB_TYPE_XSK }; /* wrapper around a pointer to a socket buffer, @@ -843,6 +844,7 @@ int igb_xsk_pool_setup(struct igb_adapter *adapter, u16 qid); bool igb_alloc_rx_buffers_zc(struct igb_ring *rx_ring, u16 count); int igb_clean_rx_irq_zc(struct igb_q_vector *q_vector, const int budget); +bool igb_xmit_zc(struct igb_ring *tx_ring, unsigned int budget); int igb_xsk_wakeup(struct net_device *dev, u32 qid, u32 flags); #endif /* _IGB_H_ */ diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c index f4dbb75d6eac..c6bb5b1944c8 100644 --- a/drivers/net/ethernet/intel/igb/igb_main.c +++ b/drivers/net/ethernet/intel/igb/igb_main.c @@ -4390,6 +4390,8 @@ void igb_configure_tx_ring(struct igb_adapter *adapter, u64 tdba = ring->dma; int reg_idx = ring->reg_idx; + ring->xsk_pool = igb_xsk_pool(adapter, ring); + wr32(E1000_TDLEN(reg_idx), ring->count * sizeof(union e1000_adv_tx_desc)); wr32(E1000_TDBAL(reg_idx), @@ -4970,15 +4972,20 @@ static void igb_clean_tx_ring(struct igb_ring *tx_ring) { u16 i = tx_ring->next_to_clean; struct igb_tx_buffer *tx_buffer = &tx_ring->tx_buffer_info[i]; + u32 xsk_frames = 0; while (i != tx_ring->next_to_use) { union e1000_adv_tx_desc *eop_desc, *tx_desc; /* Free all the Tx ring sk_buffs or xdp frames */ - if (tx_buffer->type == IGB_TYPE_SKB) + if (tx_buffer->type == IGB_TYPE_SKB) { dev_kfree_skb_any(tx_buffer->skb); - else + } else if (tx_buffer->type == IGB_TYPE_XDP) { xdp_return_frame(tx_buffer->xdpf); + } else if (tx_buffer->type == IGB_TYPE_XSK) { + xsk_frames++; + goto skip_for_xsk; + } /* unmap skb header data */ dma_unmap_single(tx_ring->dev, @@ -5009,6 +5016,7 @@ static void igb_clean_tx_ring(struct igb_ring *tx_ring) DMA_TO_DEVICE); } +skip_for_xsk: tx_buffer->next_to_watch = NULL; /* move us one more past the eop_desc for start of next pkt */ @@ -5023,6 +5031,9 @@ static void igb_clean_tx_ring(struct igb_ring *tx_ring) /* reset BQL for queue */ netdev_tx_reset_queue(txring_txq(tx_ring)); + if (tx_ring->xsk_pool && xsk_frames) + xsk_tx_completed(tx_ring->xsk_pool, xsk_frames); + /* reset next_to_use and next_to_clean */ tx_ring->next_to_use = 0; tx_ring->next_to_clean = 0; @@ -8330,12 +8341,16 @@ static int igb_poll(struct napi_struct *napi, int budget) static bool igb_clean_tx_irq(struct igb_q_vector *q_vector, int napi_budget) { struct igb_adapter *adapter = q_vector->adapter; + int cpu = smp_processor_id(); struct igb_ring *tx_ring = q_vector->tx.ring; struct igb_tx_buffer *tx_buffer; union e1000_adv_tx_desc *tx_desc; + struct netdev_queue *nq; unsigned int total_bytes = 0, total_packets = 0; unsigned int budget = q_vector->tx.work_limit; unsigned int i = tx_ring->next_to_clean; + u32 xsk_frames = 0; + bool xsk_xmit_done = true; if (test_bit(__IGB_DOWN, &adapter->state)) return true; @@ -8366,10 +8381,14 @@ static bool igb_clean_tx_irq(struct igb_q_vector *q_vector, int napi_budget) total_packets += tx_buffer->gso_segs; /* free the skb */ - if (tx_buffer->type == IGB_TYPE_SKB) + if (tx_buffer->type == IGB_TYPE_SKB) { napi_consume_skb(tx_buffer->skb, napi_budget); - else + } else if (tx_buffer->type == IGB_TYPE_XDP) { xdp_return_frame(tx_buffer->xdpf); + } else if (tx_buffer->type == IGB_TYPE_XSK) { + xsk_frames++; + goto skip_for_xsk; + } /* unmap skb header data */ dma_unmap_single(tx_ring->dev, @@ -8401,6 +8420,7 @@ static bool igb_clean_tx_irq(struct igb_q_vector *q_vector, int napi_budget) } } +skip_for_xsk: /* move us one more past the eop_desc for start of next pkt */ tx_buffer++; tx_desc++; @@ -8429,6 +8449,20 @@ static bool igb_clean_tx_irq(struct igb_q_vector *q_vector, int napi_budget) q_vector->tx.total_bytes += total_bytes; q_vector->tx.total_packets += total_packets; + if (tx_ring->xsk_pool) { + if (xsk_frames) + xsk_tx_completed(tx_ring->xsk_pool, xsk_frames); + if (xsk_uses_need_wakeup(tx_ring->xsk_pool)) + xsk_set_tx_need_wakeup(tx_ring->xsk_pool); + + nq = txring_txq(tx_ring); + __netif_tx_lock(nq, cpu); + /* Avoid transmit queue timeout since we share it with the slow path */ + txq_trans_cond_update(nq); + xsk_xmit_done = igb_xmit_zc(tx_ring, q_vector->tx.work_limit); + __netif_tx_unlock(nq); + } + if (test_bit(IGB_RING_FLAG_TX_DETECT_HANG, &tx_ring->flags)) { struct e1000_hw *hw = &adapter->hw; @@ -8491,7 +8525,7 @@ static bool igb_clean_tx_irq(struct igb_q_vector *q_vector, int napi_budget) } } - return !!budget; + return !!budget && xsk_xmit_done; } /** diff --git a/drivers/net/ethernet/intel/igb/igb_xsk.c b/drivers/net/ethernet/intel/igb/igb_xsk.c index eae616e7608c..8b60285ec242 100644 --- a/drivers/net/ethernet/intel/igb/igb_xsk.c +++ b/drivers/net/ethernet/intel/igb/igb_xsk.c @@ -25,25 +25,28 @@ static int igb_xsk_pool_enable(struct igb_adapter *adapter, u16 qid) { struct net_device *netdev = adapter->netdev; - struct igb_ring *rx_ring; + struct igb_ring *tx_ring, *rx_ring; bool if_running; int err; if (qid >= adapter->num_rx_queues) return -EINVAL; - if (qid >= netdev->real_num_rx_queues) + if (qid >= netdev->real_num_rx_queues || + qid >= netdev->real_num_tx_queues) return -EINVAL; err = xsk_pool_dma_map(pool, &adapter->pdev->dev, IGB_RX_DMA_ATTR); if (err) return err; + tx_ring = adapter->tx_ring[qid]; rx_ring = adapter->rx_ring[qid]; if_running = netif_running(adapter->netdev) && igb_xdp_is_enabled(adapter); if (if_running) igb_txrx_ring_disable(adapter, qid); + set_bit(IGB_RING_FLAG_AF_XDP_ZC, &tx_ring->flags); set_bit(IGB_RING_FLAG_AF_XDP_ZC, &rx_ring->flags); if (if_running) { @@ -52,6 +55,7 @@ static int igb_xsk_pool_enable(struct igb_adapter *adapter, /* Kick start the NAPI context so that receiving will start */ err = igb_xsk_wakeup(adapter->netdev, qid, XDP_WAKEUP_RX); if (err) { + clear_bit(IGB_RING_FLAG_AF_XDP_ZC, &tx_ring->flags); clear_bit(IGB_RING_FLAG_AF_XDP_ZC, &rx_ring->flags); xsk_pool_dma_unmap(pool, IGB_RX_DMA_ATTR); return err; @@ -63,7 +67,7 @@ static int igb_xsk_pool_enable(struct igb_adapter *adapter, static int igb_xsk_pool_disable(struct igb_adapter *adapter, u16 qid) { - struct igb_ring *rx_ring; + struct igb_ring *tx_ring, *rx_ring; struct xsk_buff_pool *pool; bool if_running; @@ -71,12 +75,14 @@ static int igb_xsk_pool_disable(struct igb_adapter *adapter, u16 qid) if (!pool) return -EINVAL; + tx_ring = adapter->tx_ring[qid]; rx_ring = adapter->rx_ring[qid]; if_running = netif_running(adapter->netdev) && igb_xdp_is_enabled(adapter); if (if_running) igb_txrx_ring_disable(adapter, qid); xsk_pool_dma_unmap(pool, IGB_RX_DMA_ATTR); + clear_bit(IGB_RING_FLAG_AF_XDP_ZC, &tx_ring->flags); clear_bit(IGB_RING_FLAG_AF_XDP_ZC, &rx_ring->flags); if (if_running) @@ -335,6 +341,61 @@ int igb_clean_rx_irq_zc(struct igb_q_vector *q_vector, const int budget) return failure ? budget : (int)total_packets; } +bool igb_xmit_zc(struct igb_ring *tx_ring, unsigned int budget) +{ + struct xsk_buff_pool *pool = tx_ring->xsk_pool; + union e1000_adv_tx_desc *tx_desc = NULL; + struct igb_tx_buffer *tx_bi; + bool work_done = true; + struct xdp_desc desc; + dma_addr_t dma; + u32 cmd_type; + + while (budget-- > 0) { + if (unlikely(!igb_desc_unused(tx_ring))) { + work_done = false; + break; + } + + if (!netif_carrier_ok(tx_ring->netdev)) + break; + + if (!xsk_tx_peek_desc(pool, &desc)) + break; + + dma = xsk_buff_raw_get_dma(pool, desc.addr); + xsk_buff_raw_dma_sync_for_device(pool, dma, desc.len); + + tx_bi = &tx_ring->tx_buffer_info[tx_ring->next_to_use]; + tx_bi->bytecount = desc.len; + tx_bi->type = IGB_TYPE_XSK; + tx_bi->xdpf = NULL; + tx_bi->gso_segs = 1; + + tx_desc = IGB_TX_DESC(tx_ring, tx_ring->next_to_use); + tx_desc->read.buffer_addr = cpu_to_le64(dma); + + /* put descriptor type bits */ + cmd_type = E1000_ADVTXD_DTYP_DATA | E1000_ADVTXD_DCMD_DEXT | + E1000_ADVTXD_DCMD_IFCS; + + cmd_type |= desc.len | IGB_TXD_DCMD; + tx_desc->read.cmd_type_len = cpu_to_le32(cmd_type); + tx_desc->read.olinfo_status = 0; + + tx_ring->next_to_use++; + if (tx_ring->next_to_use == tx_ring->count) + tx_ring->next_to_use = 0; + } + + if (tx_desc) { + igb_xdp_ring_update_tail(tx_ring); + xsk_tx_release(pool); + } + + return !!budget && work_done; +} + int igb_xsk_wakeup(struct net_device *dev, u32 qid, u32 flags) { struct igb_adapter *adapter = netdev_priv(dev);