From patchwork Mon May 9 19:46:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Hancock X-Patchwork-Id: 12844022 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6AE44C433EF for ; Mon, 9 May 2022 19:47:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240669AbiEITvY (ORCPT ); Mon, 9 May 2022 15:51:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44422 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240639AbiEITvV (ORCPT ); Mon, 9 May 2022 15:51:21 -0400 Received: from mx0d-0054df01.pphosted.com (mx0d-0054df01.pphosted.com [67.231.150.19]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1D11B248FF for ; Mon, 9 May 2022 12:47:26 -0700 (PDT) Received: from pps.filterd (m0209000.ppops.net [127.0.0.1]) by mx0c-0054df01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 249JYU8V007872; Mon, 9 May 2022 15:47:06 -0400 Received: from can01-yt3-obe.outbound.protection.outlook.com (mail-yt3can01lp2169.outbound.protection.outlook.com [104.47.75.169]) by mx0c-0054df01.pphosted.com (PPS) with ESMTPS id 3fwnp40wq0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 09 May 2022 15:47:06 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=G1Tjk75hFYUrS7/c2iu02hV1yGybhMy3mUO/1FejcHssxEJfLSG69+Ceh2EE/l5oGNzLV9jI85jT1eAPd14dYun0SEHy6Uv4gc5v4okWiBdT3zHifKUzhqoETPsmt49yi8fXIaq4GXn2tXu5/BjJS+pRjoVMfBKfvwwHMv31KyitXEkpocDe8lxbpXLkhuXHsyOU2+VsiiTjtBWnJE8hu6LKqUeEcHwY2vTMa1/xeD0LuSn5oYCV4CIVIanHNGdRazhcmmjxBp3pREfVVi+JpQKpl5M/aP7Vw6L+B8PRZjkXmuh2WKvvH/Uqm3wYqFTyzrTtFE11tCvm1h8QlVpz6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=trSxq4evSUgRS9zcY07+Vli3az9XHfM28VbK2LfYQYg=; b=gqmRbIp23g5/qlrZ1RGwtL/52mkDMsn25sTKVnLEXJ7ubeyrbpa4xw8mYeNMUMzp1kQFFtDYWlsvm5BI1bVYwKrG/tRy63q3OOafXkksNntUXpsuflo9QePJlYPIapMMHB9omacG+67ajETnP8kDFgyMVygSxx1bEIO3evt0ucIcDFOxF9W2tDHc8QPHjOzVwQFBLPVZwvW2nvFUR0juLOQmtjGXhbuwxuFTn468h7eooVfbyGPE1YFSD8BqeNRQkSF007gW3nn5VjnEmFpUKpuXuQAQkDmJXAxWP5MvFC2JcC69Xxz7dwzqwWns9WxLT3qiUeG0Lqdy1PEA6zwVDg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=calian.com; dmarc=pass action=none header.from=calian.com; dkim=pass header.d=calian.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=calian.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=trSxq4evSUgRS9zcY07+Vli3az9XHfM28VbK2LfYQYg=; b=0ePg4KjAvdhiWSPhLq8Zx/LV/Zpwb7sfnKTd5+1XTLQKw8P7F5gH/oPo7PoCtSxso5iRQeWqUtf3oLenlzwbgstifVGkJddAGRZ7oVU0CaBb2/ZPH37H5sQPSz0MLn7Zp0DpxuD9Ex1xjJHcShjgocPt8y7zS/0g5g2mihjtVtU= Received: from YT3PR01MB6274.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:66::9) by YT2PR01MB5904.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:58::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.20; Mon, 9 May 2022 19:47:05 +0000 Received: from YT3PR01MB6274.CANPRD01.PROD.OUTLOOK.COM ([fe80::4dea:7f7d:cc9a:1c83]) by YT3PR01MB6274.CANPRD01.PROD.OUTLOOK.COM ([fe80::4dea:7f7d:cc9a:1c83%2]) with mapi id 15.20.5227.023; Mon, 9 May 2022 19:47:05 +0000 From: Robert Hancock To: netdev@vger.kernel.org Cc: nicolas.ferre@microchip.com, claudiu.beznea@microchip.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, linux@armlinux.org.uk, tomas.melin@vaisala.com, harinik@xilinx.com, Robert Hancock Subject: [PATCH net-next v2 1/2] net: macb: simplify/cleanup NAPI reschedule checking Date: Mon, 9 May 2022 13:46:34 -0600 Message-Id: <20220509194635.3094080-2-robert.hancock@calian.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220509194635.3094080-1-robert.hancock@calian.com> References: <20220509194635.3094080-1-robert.hancock@calian.com> X-ClientProxiedBy: YT3PR01CA0002.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:86::10) To YT3PR01MB6274.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:66::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3b929053-1a35-4935-ea27-08da31f4b1be X-MS-TrafficTypeDiagnostic: YT2PR01MB5904:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: AUzcxEtW0l8L3QjJHwn9O1VmyTA15Z4Zk2DvlDRobR+DrQvDz5qlwmCbH3d5Sx2Nt7Fd+5W5aC4feGHAyLmFmh+6WcgdeTlZ38r19sLRPSBICAvGVRBEHTIjZDcOifQTkZ2Qcp9THt6s/65JICYYl0WfyEQMikcNNzE5n3Y4nAJql4kAboz/5IfN92VGfnBT3hytFEneteX47rEYnJyB0mlXWvvwvqlDY04BPVPIzjzkNBM2ogRF9WL24pquyu/85LGeA1lwJLat2NyWFzy8m5S7CG7cr9Ljdm1R6f4kEp4JSo1ltp0+03cN0hGNcwnjP/PdeFSV8X+WtIIyFo78ZA4QtpUfk/V9W9YovykVJTpWWNocCmXbu11sRVXG9EQhgIaSmJWHLKrfQsZJt4CC+/Lr9mnJkaMPEJSjkb6X8Jp3qdgd5DKE4QjNAZmnAg1pL6B+3Uwqa2IQd8jKCGlef1xA3Hwoj0B+yCjwXmaQlFZ5dD04YcT+Tb0dWVPKf7Wo/2J0enYFbIdga/2sc9U/aNhJrP4ml/EVbnBGcu3ql7qPOh/PbfKD+5iWlu6PdM6AVLYIA9i53XknQ6Hjfq7DOliLe9ejcH9BsapEGFWU6pI93alMUE/lEhO704A9OkkHjyG3GCd2+GhUa/gGiwtx3B7IqYiJo8EDzcu3cD1yyaWQqc1GodguR6mx4PHDucOwo+2WnTqsuapCXdZ2Hn2gOA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YT3PR01MB6274.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(508600001)(52116002)(38350700002)(6506007)(6916009)(86362001)(316002)(6486002)(2906002)(6666004)(38100700002)(186003)(66946007)(36756003)(26005)(83380400001)(2616005)(5660300002)(6512007)(107886003)(1076003)(7416002)(8676002)(4326008)(44832011)(66556008)(8936002)(66476007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: QaidtaH0E/mtqQcYd0U0Y+Nvt+ew8k5vvsWCRNCUhlCHAzKGTqrf1LXBTZdXTSxnzB/EWSVd090FfQbfKrcpnYnNt1fNWEceyBf5C8xs5bPMTyK/yHKN/Eo8SK3Tgc81MTnHOa8uxqecLgGCETqCuB4POCqYnwhozjDz1VudE5Xp2RC7veWvEg69rTGY3gJmBsMXNU9KeAnfSnKZXjPQZjowmfHbskrjF1jBHKU+WrQUpTfXYzsd6LmBpps1NHoqBZNCpfJkroe0XfrMxO/c9uYXjpOvrV0mD4rRUcvS6RdL80rl+OiwphwgNd2T5nOu0OMvQRcNPISAel2Vw+7c0RDsQlaOBmd1QOCPKeFxLrnFwk3hQRVptjw6jYewKbAMZ1bpSTWQYWnSB2SSzhM8ubvCe7zojmJACqTj+l2COXeufrvbW8qEYxJFvGmAsrULXHHsQoSLAMyA8CYaAxSFDbmWB0KRlPtccsogC58I1OGaR/o+pRpk8dB+r4vdZnKx5CzLppd5I80PfKVm8hm8GDhshQ7/zjghv5/vwzm9yvTtKgxRWt3tl5q7BypQrbyZd1+r3p0My0r8KxTzzWvUOss29a/TYAARlvWhcXYvfTwQnWAXGZPMpwhuqC2Du85rn2fdhKrFaOJiSM6xZKQYdcdnGxNmQu9xieWuMe/d28j6wWPMstdMBm38+1kjezEdpdsqT9ous/SWhuWDSz5s9HI6EmIn8gIW0RJ29RPEP5EUHX1jdpR/TJ7Tk7h1Vd3cqiYWFNIka/KaLIqYo7XYJcvDmtAmYQgaWQno64SblwtHEAa0iY31Vg63UKIOZMIz8srd8aBW66rrG4tJzUsROyxolWaQBsfiDKJRw9IqpLDYrFivAkaNp6v4D/2+xlomkxWyXfSYn1pDMrZLv0zLarRAisyR7jh18L1SAC8VEtYz14v3e0mHk9CnrjL8pvmmrodVDuIqLTZBS6Fs4S8l7JWJLUSuLiyJ3HVikiJacy0A4eNChe+0YaNqt+IXnQiVo1b6Ar7eIls25D7nMJcMVjksMPFRS5OQNYsS9ZOvctglXtQwhEiAlzqQQYEjYprtSalVb9Y3diFHScqUX8rM62F+Hk6jLyKpdPxZLGVoJJeaTL4oD83LCXwsEinqDrLZr5jFcH5ACxOYSuVChd2OzCGjmTuzCNpBfxL8sivLKvmQnxpXXISiJcOxmbSFusX5G2b7jaVcIefhLJA69yAsMRrB+iTmnkDT/fFjSaOD7Mi8uljMXPGMGP1H9/Ey4dRwNDwtf4w85aCI73V7nQUtNSVOOoi0Q4FfLTOrSJ0W4nfyZNgtTKiN0TuHkVrcXAlgHpNnxi7kmOc8bj8uqaOkj6LClUYXxlgKaFYWsNC5xL84WarGwnb2MnpkqXA2EzdDyKS9058CogqzZoLJnQ5eTCnUV1mtbc2mIx4AbRywKFuvxqdk/qncISqfLKf+vn9mD+PZ27olhqA+q0tgIeSGaO9uuuFigU4ktYHNflb4uzsGm2OeZ+Gstlx7S7LmaDr2Ajan5WMTuUQ1sBvPJhBD/P0Y22gdbFQ2PFnidIQq3SBsB6yfkyj+h78hfZew2lrC5DBQZAy8tmIEFWKywto7vvdBt9axAkyw/fykP/j6NbfcWtY8iypCOaTCvjBwqyxTEmd3wYMJQkaA9imSp4b7tgB5cChM4jGpodtvQGFGaJtDTB2aNFFiRkQwqWBPpkC9mLqMz0AB74x7WNeWTKDAQEvbch1nRAS5FgvxiAhcPaU= X-OriginatorOrg: calian.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3b929053-1a35-4935-ea27-08da31f4b1be X-MS-Exchange-CrossTenant-AuthSource: YT3PR01MB6274.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 May 2022 19:47:05.1300 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 23b57807-562f-49ad-92c4-3bb0f07a1fdf X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: /sj17t9+gNmX3iMqZP0xBTK3XSZkWyn/JNkQ276PwBntrb7mRCstSlkpAZ9PHXwbUE0Rap6QajBO9H7zroQyNDf54TDl7QI76MgD9nEa/1o= X-MS-Exchange-Transport-CrossTenantHeadersStamped: YT2PR01MB5904 X-Proofpoint-ORIG-GUID: ZYj9Zg2c4pcx63ocCA8bgOrmRm7eYqMe X-Proofpoint-GUID: ZYj9Zg2c4pcx63ocCA8bgOrmRm7eYqMe X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.858,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-05-09_05,2022-05-09_02,2022-02-23_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 clxscore=1015 adultscore=0 bulkscore=0 impostorscore=0 lowpriorityscore=0 phishscore=0 malwarescore=0 priorityscore=1501 mlxscore=0 mlxlogscore=768 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205090102 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Previously the macb_poll method was checking the RSR register after completing its RX receive work to see if additional packets had been received since IRQs were disabled, since this controller does not maintain the pending IRQ status across IRQ disable. It also had to double-check the register after re-enabling IRQs to detect if packets were received after the first check but before IRQs were enabled. Using the RSR register for this purpose is problematic since it reflects the global device state rather than the per-queue state, so if packets are being received on multiple queues it may end up retriggering receive on a queue where the packets did not actually arrive and not on the one where they did arrive. This will also cause problems with an upcoming change to use NAPI for the TX path where use of multiple queues is more likely. Add a macb_rx_pending function to check the RX ring to see if more packets have arrived in the queue, and use that to check if NAPI should be rescheduled rather than the RSR register. By doing this, we can just ignore the global RSR register entirely, and thus save some extra device register accesses at the same time. This also makes the previous first check for pending packets rather redundant, since it would be checking the RX ring state which was just checked in the receive work function. Therefore we can get rid of it and just check after enabling interrupts whether packets are already pending. Signed-off-by: Robert Hancock --- drivers/net/ethernet/cadence/macb_main.c | 65 +++++++++++------------- 1 file changed, 31 insertions(+), 34 deletions(-) diff --git a/drivers/net/ethernet/cadence/macb_main.c b/drivers/net/ethernet/cadence/macb_main.c index 6434e74c04f1..160dc5ad84ae 100644 --- a/drivers/net/ethernet/cadence/macb_main.c +++ b/drivers/net/ethernet/cadence/macb_main.c @@ -1554,54 +1554,51 @@ static int macb_rx(struct macb_queue *queue, struct napi_struct *napi, return received; } +static bool macb_rx_pending(struct macb_queue *queue) +{ + struct macb *bp = queue->bp; + unsigned int entry; + struct macb_dma_desc *desc; + + entry = macb_rx_ring_wrap(bp, queue->rx_tail); + desc = macb_rx_desc(queue, entry); + + /* Make hw descriptor updates visible to CPU */ + rmb(); + + return (desc->addr & MACB_BIT(RX_USED)) != 0; +} + static int macb_poll(struct napi_struct *napi, int budget) { struct macb_queue *queue = container_of(napi, struct macb_queue, napi); struct macb *bp = queue->bp; int work_done; - u32 status; - status = macb_readl(bp, RSR); - macb_writel(bp, RSR, status); + work_done = bp->macbgem_ops.mog_rx(queue, napi, budget); - netdev_vdbg(bp->dev, "poll: status = %08lx, budget = %d\n", - (unsigned long)status, budget); + netdev_vdbg(bp->dev, "poll: queue = %u, work_done = %d, budget = %d\n", + (unsigned int)(queue - bp->queues), work_done, budget); - work_done = bp->macbgem_ops.mog_rx(queue, napi, budget); - if (work_done < budget) { - napi_complete_done(napi, work_done); + if (work_done < budget && napi_complete_done(napi, work_done)) { + queue_writel(queue, IER, bp->rx_intr_mask); - /* RSR bits only seem to propagate to raise interrupts when - * interrupts are enabled at the time, so if bits are already - * set due to packets received while interrupts were disabled, + /* Packet completions only seem to propagate to raise + * interrupts when interrupts are enabled at the time, so if + * packets were received while interrupts were disabled, * they will not cause another interrupt to be generated when * interrupts are re-enabled. - * Check for this case here. This has been seen to happen - * around 30% of the time under heavy network load. + * Check for this case here to avoid losing a wakeup. This can + * potentially race with the interrupt handler doing the same + * actions if an interrupt is raised just after enabling them, + * but this should be harmless. */ - status = macb_readl(bp, RSR); - if (status) { + if (macb_rx_pending(queue)) { + queue_writel(queue, IDR, bp->rx_intr_mask); if (bp->caps & MACB_CAPS_ISR_CLEAR_ON_WRITE) queue_writel(queue, ISR, MACB_BIT(RCOMP)); - napi_reschedule(napi); - } else { - queue_writel(queue, IER, bp->rx_intr_mask); - - /* In rare cases, packets could have been received in - * the window between the check above and re-enabling - * interrupts. Therefore, a double-check is required - * to avoid losing a wakeup. This can potentially race - * with the interrupt handler doing the same actions - * if an interrupt is raised just after enabling them, - * but this should be harmless. - */ - status = macb_readl(bp, RSR); - if (unlikely(status)) { - queue_writel(queue, IDR, bp->rx_intr_mask); - if (bp->caps & MACB_CAPS_ISR_CLEAR_ON_WRITE) - queue_writel(queue, ISR, MACB_BIT(RCOMP)); - napi_schedule(napi); - } + netdev_vdbg(bp->dev, "poll: packets pending, reschedule\n"); + napi_schedule(napi); } } From patchwork Mon May 9 19:46:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Hancock X-Patchwork-Id: 12844023 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1AD9C433F5 for ; Mon, 9 May 2022 19:47:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240671AbiEITv0 (ORCPT ); Mon, 9 May 2022 15:51:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44506 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240666AbiEITvX (ORCPT ); Mon, 9 May 2022 15:51:23 -0400 Received: from mx0d-0054df01.pphosted.com (mx0d-0054df01.pphosted.com [67.231.150.19]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B9A21248DF for ; Mon, 9 May 2022 12:47:27 -0700 (PDT) Received: from pps.filterd (m0209000.ppops.net [127.0.0.1]) by mx0c-0054df01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 249JYHIN026003; Mon, 9 May 2022 15:47:07 -0400 Received: from can01-yt3-obe.outbound.protection.outlook.com (mail-yt3can01lp2170.outbound.protection.outlook.com [104.47.75.170]) by mx0c-0054df01.pphosted.com (PPS) with ESMTPS id 3fwnp40wq2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 09 May 2022 15:47:07 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mouQopV8dIxlQ1gaE8h5nnPa5w145Qn8UE7xfe4x+6G8REO5XbVRyqYsLA3tS9Aa+YrByUSU/YX9WaVrB9YsPoNNzzR3OJvWqMzIUkBo16kzMyjGF0lFO4HR50D//4BbuWiRtDRQK/6zvFIHp9gktdzBzLvpdPy6PgzRJtF/Gqj/7n4N6dwYNDWOBhc1umINC8yd4OvIVed686XSNgN2QPCrSdyj8wJioE4vuXhVaK4nG4osUVLd3/wqo6rgUww9aOLvxlewujdqLJEd4L3yEkav+A7jOd59hSl2HNTPFoLWkCVFTIgXgqAj2yA3n45acd7IQX4K6B3lkCptkXFBAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZGCBPWD87fqtUJ0w3QdnNp2RL6pV57WHp/w1Vfve2BI=; b=LOJWtiLbCfVtIK1LBqPcm1cXPZ4lKRu/4ktRe5oVzhfs8m7X0M6JoYnOm1FhLGGDPTOTI1xl2bUYKmnIC+rke0ccrt9H6tb8tq9Ls881VaZI7Za/yRraO2qYK8bna6SC1spzuzD1V9CuYDcJriWK43xHsgYz6CCFlnZRA4/iNXUH6P08Z1CZe2n0dEXnGJks3gSFQHcPiH+bF5KxJpk1FsTY7AhrpHFJDiTO9iTYmYV57RNmNPxbRzNeMtrxsUc4YfMiTh+97IJc2lAF47G3Aj4UNBy+3vjSJ41Paxe1Mwst9wHx7pEXAoTHtPs664l2+msSbxgN9v5/aGBqKsUB6Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=calian.com; dmarc=pass action=none header.from=calian.com; dkim=pass header.d=calian.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=calian.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZGCBPWD87fqtUJ0w3QdnNp2RL6pV57WHp/w1Vfve2BI=; b=2AO3x67Be6kEAhxS+B+ipHrHV9gZKZRQ93D71B7n+n5LhwinYMLWKdcWZTUn2PWHR5YgnRnHKrAUD7CJnxEiG6XCx0Rh8fxXWqiZW2tLns7KNwR9TcDeMEtc5xCYjAtGWPjjNNk8zKgZrD0P4MZoW5S4iq6T3nxr/hspVZ0ivFM= Received: from YT3PR01MB6274.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:66::9) by YT2PR01MB5904.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:58::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.20; Mon, 9 May 2022 19:47:06 +0000 Received: from YT3PR01MB6274.CANPRD01.PROD.OUTLOOK.COM ([fe80::4dea:7f7d:cc9a:1c83]) by YT3PR01MB6274.CANPRD01.PROD.OUTLOOK.COM ([fe80::4dea:7f7d:cc9a:1c83%2]) with mapi id 15.20.5227.023; Mon, 9 May 2022 19:47:06 +0000 From: Robert Hancock To: netdev@vger.kernel.org Cc: nicolas.ferre@microchip.com, claudiu.beznea@microchip.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, linux@armlinux.org.uk, tomas.melin@vaisala.com, harinik@xilinx.com, Robert Hancock Subject: [PATCH net-next v2 2/2] net: macb: use NAPI for TX completion path Date: Mon, 9 May 2022 13:46:35 -0600 Message-Id: <20220509194635.3094080-3-robert.hancock@calian.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220509194635.3094080-1-robert.hancock@calian.com> References: <20220509194635.3094080-1-robert.hancock@calian.com> X-ClientProxiedBy: YT3PR01CA0002.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:86::10) To YT3PR01MB6274.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:66::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9d82ee3d-1a6d-46b4-1264-08da31f4b23a X-MS-TrafficTypeDiagnostic: YT2PR01MB5904:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: UL24ZIn7KnzcBT18ZZuqPMyfMJcD9TIWwbxBxG95IgOg++EWFYYio6UmBC0VUqEVaGVqV8qGswZ0V5lETbxCye2Ru87efIJZNRtIBvGkRFw3kI0ybrQZiUT2wowUYTvaQWTSUnO7pfhEZ19sIjo+nVpVw22EAC0tgLCHxluIgub7G9lAsgG45I5xc+dtl+eJzLXr6nkCy1jVNWSgyl8XpV0k1lCQtSIo/rkd/GIEtzobhAofpgus4A1JTuLjesWmycgQRRdbFYY3j+hDJlJHJVlU5uDHgGetXVRbJdeo2T7WPY2YjN1m7h3zx65L2RWFhfrdYU8Llbv7bygzQu9k+EgX9e6FxeyNrJp4ucSKyHlplLU5HOS8v1mZjT9ae+7j0uJt6UvfcpsV/qJ+KIoneHgmr4qThlaszDeCF7zh8SgnNQ5tvb9By12N5Ht3En3lioM7FayrmbjH0zOOdFsULAzaE/hIlh0XKSm6JJLpKakJlrG7ZHVY+8UIQ4e2xq/vlC4g/PJ3OERuJonv7jzAKcukv1qk5JF2CdFNJN+is/tXOF0iJwh5k7+6160CsH3AjydbGZ7stN4DbrR8dDNNXxDgDsmWINaPNdDLsV69Moa9nn8ot5vzpufiPbYWtKenrplj9Tx1+oX4FBivLTbySWAfcJZj6c05Gm71HXrav+WWIUTzmqSlBBW8rNuCTz6/kZqnPBZEKTYtU1h1ZrX43w== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YT3PR01MB6274.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(508600001)(52116002)(38350700002)(6506007)(6916009)(86362001)(316002)(6486002)(2906002)(6666004)(38100700002)(186003)(66946007)(36756003)(26005)(83380400001)(2616005)(5660300002)(6512007)(107886003)(1076003)(7416002)(30864003)(8676002)(4326008)(44832011)(66556008)(8936002)(66476007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: HjUe29yOtVhipz+8Hl/RBM02ifDKzYAq9QR67+3+1DF7IE+Uiz2O8rkjx8YORT3/LZcgJtERHu5vBaBCdAfUkXGamAlnZddce7HAp3EXGz2j2jOTAVTsCBRCqRQcJ4G2c/UmR3Elvp1joTyUEFxBbGMswCO/HCF6zcq5i7YWRATgNfq9UnCbzYOmIPzJqTA1YCo771iKGTirN28lRlrCGmwr8xHFdaeRVeKFTeXs15y177v6p6EvMBG4U/FO814H2aN6+4rzmuu7LMBw3kD1SPWcUoM2wxhQdEeEazJi5T+h93rY4BjfRFD6QXrkAfNjwZaECftMdlPWbSGwOwGWT9DkLeWuUIq74HEUeGoz9YN07ZqvYQ4nzQGQ67CqwLj9UlZfnyJCv8ujI/nTtut/bu+tOPRS9OoAaxPPRfaOpaC8sxgVK8f7caEB8s1NgxuO02vUodGC+SWJs9t0n0crqrX3pMpmwZu1hx1wyMknvrkHKTQ8JsFAkslZFn6IeHsZQWl2DG0Z+5imyZlOERYDGsZqPQl1xwwik/2XjF0tr1gLnfbh88EJyc557gfl3DhlVBJggZLW4UNn+Sc5L1W/wzVONsQRerY3MJG4xYe4h/QOluQrADET5wZ92Pic1B2tMHXMm4JAhVd+bJSY70ftF9U64/gW8+g0+51krErD+9rCnq5F3w5uBUGyKK3FKM+BzK1r7CPV8Kc1kHQcwHU1dTly/e9JUnstUsOtBkrfv6lQKwOUJIjAxJ8tN/1R5xaPrZDHPwQkHuCaGYrDIxGD5LFdNFqKER3Hugd+BvDTcvo+WycetkyQtWmwfAhaLZ8OTrc+HYsGIm5OB/tUtiD1Q96b1b81Ay4uuF1yDhlPH9CkL5qTEvlpvSDFe7Vt+w1a3QLyVrjUKqByG/93w7JYgLBbfVTOKO2xJCBoeQl6bZZSU7QQCQ3DSWsf1XaNK79aN7pIAJdVGjr1ctNxzYg28RlyytsFA/KDnJzRRvpbTRyMMp/4pEt64zGFJ56EdbLWhE4Qj7wTZZQp5YOFoL3YjpYjTi0K+FaaKpkQNh2RThh5Gnc6/Idgn06TmbWiX/VkzB1jXdbJmDk6YeepGOqZcBgL49QWBDOT+ea1OjYBsHP8APqMzZsJoemPhetdjrblgjZ1Ac5Cz/H/pwMfV5L3Q9tO58RXFkdMeiwX5Hl33HGh+SilZDfXE4H+oGUs37Vh8YdOSLlUf4eKMN23TVR6Lru1zQjqV45NDFJQKqM/9NiWZMfzg0bhVhxMJkXnii4hbXXPlDOqzmRMiE/Xt/6r2JIA3djqH2kTFW1/SwUT9XuqGj92VQn+0Lv1e4Xn2R2Q5z+MJj1Qj2hFpQhHvxOATSxOT2eeL1rFQIQtk3OZexSDCXGz9YihPdywG+s+C7B3wIhWTpe+jwipbwKNXZWsfNXXUHnbn5Hhb6cGnwEYnTsTA3lg4Nze2E0QbIhTctrETjzksetH/JnM1jQKlF/Ei3PTNeFtA+EFErbCHFuzbeI/A/CfFvk68/sQNn88EpZ1rGVMoywTHyXJfgCusHWLi8dB9+ozPPh96HaPedaOSVjDH0bu8t6Jh5Syx0w7JTGdaUxnc6XbFY4yNEAYgkYsdUdzOPV0GSZAF7pZ03ddj1zjIfGRF9T13r7nw4od/+uCTAFX0fPegu+UUt2DzHw5qETTfbLliNuEJarCO9rUkAZdqeSQfJ9c7FbGhnBuYnFD1j54WFDszQ7IGJF51ZrSV6rAquCF2WkRRahWhsfPXaM= X-OriginatorOrg: calian.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9d82ee3d-1a6d-46b4-1264-08da31f4b23a X-MS-Exchange-CrossTenant-AuthSource: YT3PR01MB6274.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 May 2022 19:47:05.9683 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 23b57807-562f-49ad-92c4-3bb0f07a1fdf X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 6AoWVPs3E0+lhHjmmZIU0isjBLJHcq3/ZuV3+5UVVEP1myuuvpaQ24Z8rLsqvdRimN+ot+jl+6jfG0ueuC4TW8eMWTsqwlL3MJDNQL0v4oo= X-MS-Exchange-Transport-CrossTenantHeadersStamped: YT2PR01MB5904 X-Proofpoint-ORIG-GUID: 3qa5mQB3PZGuEkRd2nSST8UMzPcJU1I5 X-Proofpoint-GUID: 3qa5mQB3PZGuEkRd2nSST8UMzPcJU1I5 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.858,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-05-09_05,2022-05-09_02,2022-02-23_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 clxscore=1015 adultscore=0 bulkscore=0 impostorscore=0 lowpriorityscore=0 phishscore=0 malwarescore=0 priorityscore=1501 mlxscore=0 mlxlogscore=930 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205090102 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org This driver was using the TX IRQ handler to perform all TX completion tasks. Under heavy TX network load, this can cause significant irqs-off latencies (found to be in the hundreds of microseconds using ftrace). This can cause other issues, such as overrunning serial UART FIFOs when using high baud rates with limited UART FIFO sizes. Switch to using a NAPI poll handler to perform the TX completion work to get this out of hard IRQ context and avoid the IRQ latency impact. A separate NAPI instance is used for TX and RX to avoid checking the other ring's state unnecessarily when doing the poll, and so that the NAPI budget handling can work for both TX and RX packets. A new per-queue tx_ptr_lock spinlock has been added to avoid using the main device lock (with IRQs needing to be disabled) across the entire TX mapping operation, and also to protect the TX queue pointers from concurrent access between the TX start and TX poll operations. The TX Used Bit Read interrupt (TXUBR) handling also needs to be moved into the TX NAPI poll handler to maintain the proper order of operations. A flag is used to notify the poll handler that a UBR condition needs to be handled. The macb_tx_restart handler has had some locking added for global register access, since this could now potentially happen concurrently on different queues. Signed-off-by: Robert Hancock --- drivers/net/ethernet/cadence/macb.h | 6 +- drivers/net/ethernet/cadence/macb_main.c | 228 ++++++++++++++++------- 2 files changed, 161 insertions(+), 73 deletions(-) diff --git a/drivers/net/ethernet/cadence/macb.h b/drivers/net/ethernet/cadence/macb.h index f0a7d8396a4a..7ca077b65eaa 100644 --- a/drivers/net/ethernet/cadence/macb.h +++ b/drivers/net/ethernet/cadence/macb.h @@ -1204,11 +1204,15 @@ struct macb_queue { unsigned int RBQP; unsigned int RBQPH; + /* Lock to protect tx_head and tx_tail */ + spinlock_t tx_ptr_lock; unsigned int tx_head, tx_tail; struct macb_dma_desc *tx_ring; struct macb_tx_skb *tx_skb; dma_addr_t tx_ring_dma; struct work_struct tx_error_task; + bool txubr_pending; + struct napi_struct napi_tx; dma_addr_t rx_ring_dma; dma_addr_t rx_buffers_dma; @@ -1217,7 +1221,7 @@ struct macb_queue { struct macb_dma_desc *rx_ring; struct sk_buff **rx_skbuff; void *rx_buffers; - struct napi_struct napi; + struct napi_struct napi_rx; struct queue_stats stats; #ifdef CONFIG_MACB_USE_HWSTAMP diff --git a/drivers/net/ethernet/cadence/macb_main.c b/drivers/net/ethernet/cadence/macb_main.c index 160dc5ad84ae..e993616308f8 100644 --- a/drivers/net/ethernet/cadence/macb_main.c +++ b/drivers/net/ethernet/cadence/macb_main.c @@ -959,7 +959,7 @@ static int macb_halt_tx(struct macb *bp) return -ETIMEDOUT; } -static void macb_tx_unmap(struct macb *bp, struct macb_tx_skb *tx_skb) +static void macb_tx_unmap(struct macb *bp, struct macb_tx_skb *tx_skb, int budget) { if (tx_skb->mapping) { if (tx_skb->mapped_as_page) @@ -972,7 +972,7 @@ static void macb_tx_unmap(struct macb *bp, struct macb_tx_skb *tx_skb) } if (tx_skb->skb) { - dev_kfree_skb_any(tx_skb->skb); + napi_consume_skb(tx_skb->skb, budget); tx_skb->skb = NULL; } } @@ -1025,12 +1025,13 @@ static void macb_tx_error_task(struct work_struct *work) (unsigned int)(queue - bp->queues), queue->tx_tail, queue->tx_head); - /* Prevent the queue IRQ handlers from running: each of them may call - * macb_tx_interrupt(), which in turn may call netif_wake_subqueue(). + /* Prevent the queue NAPI TX poll from running, as it calls + * macb_tx_complete(), which in turn may call netif_wake_subqueue(). * As explained below, we have to halt the transmission before updating * TBQP registers so we call netif_tx_stop_all_queues() to notify the * network engine about the macb/gem being halted. */ + napi_disable(&queue->napi_tx); spin_lock_irqsave(&bp->lock, flags); /* Make sure nobody is trying to queue up new packets */ @@ -1058,7 +1059,7 @@ static void macb_tx_error_task(struct work_struct *work) if (ctrl & MACB_BIT(TX_USED)) { /* skb is set for the last buffer of the frame */ while (!skb) { - macb_tx_unmap(bp, tx_skb); + macb_tx_unmap(bp, tx_skb, 0); tail++; tx_skb = macb_tx_skb(queue, tail); skb = tx_skb->skb; @@ -1088,7 +1089,7 @@ static void macb_tx_error_task(struct work_struct *work) desc->ctrl = ctrl | MACB_BIT(TX_USED); } - macb_tx_unmap(bp, tx_skb); + macb_tx_unmap(bp, tx_skb, 0); } /* Set end of TX queue */ @@ -1118,27 +1119,20 @@ static void macb_tx_error_task(struct work_struct *work) macb_writel(bp, NCR, macb_readl(bp, NCR) | MACB_BIT(TSTART)); spin_unlock_irqrestore(&bp->lock, flags); + napi_enable(&queue->napi_tx); } -static void macb_tx_interrupt(struct macb_queue *queue) +static int macb_tx_complete(struct macb_queue *queue, int budget) { - unsigned int tail; - unsigned int head; - u32 status; struct macb *bp = queue->bp; u16 queue_index = queue - bp->queues; + unsigned int tail; + unsigned int head; + int packets = 0; - status = macb_readl(bp, TSR); - macb_writel(bp, TSR, status); - - if (bp->caps & MACB_CAPS_ISR_CLEAR_ON_WRITE) - queue_writel(queue, ISR, MACB_BIT(TCOMP)); - - netdev_vdbg(bp->dev, "macb_tx_interrupt status = 0x%03lx\n", - (unsigned long)status); - + spin_lock(&queue->tx_ptr_lock); head = queue->tx_head; - for (tail = queue->tx_tail; tail != head; tail++) { + for (tail = queue->tx_tail; tail != head && packets < budget; tail++) { struct macb_tx_skb *tx_skb; struct sk_buff *skb; struct macb_dma_desc *desc; @@ -1179,10 +1173,11 @@ static void macb_tx_interrupt(struct macb_queue *queue) queue->stats.tx_packets++; bp->dev->stats.tx_bytes += skb->len; queue->stats.tx_bytes += skb->len; + packets++; } /* Now we can safely release resources */ - macb_tx_unmap(bp, tx_skb); + macb_tx_unmap(bp, tx_skb, budget); /* skb is set only for the last buffer of the frame. * WARNING: at this point skb has been freed by @@ -1198,6 +1193,9 @@ static void macb_tx_interrupt(struct macb_queue *queue) CIRC_CNT(queue->tx_head, queue->tx_tail, bp->tx_ring_size) <= MACB_TX_WAKEUP_THRESH(bp)) netif_wake_subqueue(bp->dev, queue_index); + spin_unlock(&queue->tx_ptr_lock); + + return packets; } static void gem_rx_refill(struct macb_queue *queue) @@ -1569,15 +1567,15 @@ static bool macb_rx_pending(struct macb_queue *queue) return (desc->addr & MACB_BIT(RX_USED)) != 0; } -static int macb_poll(struct napi_struct *napi, int budget) +static int macb_rx_poll(struct napi_struct *napi, int budget) { - struct macb_queue *queue = container_of(napi, struct macb_queue, napi); + struct macb_queue *queue = container_of(napi, struct macb_queue, napi_rx); struct macb *bp = queue->bp; int work_done; work_done = bp->macbgem_ops.mog_rx(queue, napi, budget); - netdev_vdbg(bp->dev, "poll: queue = %u, work_done = %d, budget = %d\n", + netdev_vdbg(bp->dev, "RX poll: queue = %u, work_done = %d, budget = %d\n", (unsigned int)(queue - bp->queues), work_done, budget); if (work_done < budget && napi_complete_done(napi, work_done)) { @@ -1607,6 +1605,90 @@ static int macb_poll(struct napi_struct *napi, int budget) return work_done; } +static void macb_tx_restart(struct macb_queue *queue) +{ + struct macb *bp = queue->bp; + unsigned int head_idx, tbqp; + + spin_lock(&queue->tx_ptr_lock); + + if (queue->tx_head == queue->tx_tail) + goto out_tx_ptr_unlock; + + tbqp = queue_readl(queue, TBQP) / macb_dma_desc_get_size(bp); + tbqp = macb_adj_dma_desc_idx(bp, macb_tx_ring_wrap(bp, tbqp)); + head_idx = macb_adj_dma_desc_idx(bp, macb_tx_ring_wrap(bp, queue->tx_head)); + + if (tbqp == head_idx) + goto out_tx_ptr_unlock; + + spin_lock_irq(&bp->lock); + macb_writel(bp, NCR, macb_readl(bp, NCR) | MACB_BIT(TSTART)); + spin_unlock_irq(&bp->lock); + +out_tx_ptr_unlock: + spin_unlock(&queue->tx_ptr_lock); +} + +static bool macb_tx_complete_pending(struct macb_queue *queue) +{ + bool retval = false; + + spin_lock(&queue->tx_ptr_lock); + if (queue->tx_head != queue->tx_tail) { + /* Make hw descriptor updates visible to CPU */ + rmb(); + + if (macb_tx_desc(queue, queue->tx_tail)->ctrl & MACB_BIT(TX_USED)) + retval = true; + } + spin_unlock(&queue->tx_ptr_lock); + return retval; +} + +static int macb_tx_poll(struct napi_struct *napi, int budget) +{ + struct macb_queue *queue = container_of(napi, struct macb_queue, napi_tx); + struct macb *bp = queue->bp; + int work_done; + + work_done = macb_tx_complete(queue, budget); + + rmb(); // ensure txubr_pending is up to date + if (queue->txubr_pending) { + queue->txubr_pending = false; + netdev_vdbg(bp->dev, "poll: tx restart\n"); + macb_tx_restart(queue); + } + + netdev_vdbg(bp->dev, "TX poll: queue = %u, work_done = %d, budget = %d\n", + (unsigned int)(queue - bp->queues), work_done, budget); + + if (work_done < budget && napi_complete_done(napi, work_done)) { + queue_writel(queue, IER, MACB_BIT(TCOMP)); + + /* Packet completions only seem to propagate to raise + * interrupts when interrupts are enabled at the time, so if + * packets were sent while interrupts were disabled, + * they will not cause another interrupt to be generated when + * interrupts are re-enabled. + * Check for this case here to avoid losing a wakeup. This can + * potentially race with the interrupt handler doing the same + * actions if an interrupt is raised just after enabling them, + * but this should be harmless. + */ + if (macb_tx_complete_pending(queue)) { + queue_writel(queue, IDR, MACB_BIT(TCOMP)); + if (bp->caps & MACB_CAPS_ISR_CLEAR_ON_WRITE) + queue_writel(queue, ISR, MACB_BIT(TCOMP)); + netdev_vdbg(bp->dev, "TX poll: packets pending, reschedule\n"); + napi_schedule(napi); + } + } + + return work_done; +} + static void macb_hresp_error_task(struct tasklet_struct *t) { struct macb *bp = from_tasklet(bp, t, hresp_err_tasklet); @@ -1646,29 +1728,6 @@ static void macb_hresp_error_task(struct tasklet_struct *t) netif_tx_start_all_queues(dev); } -static void macb_tx_restart(struct macb_queue *queue) -{ - unsigned int head = queue->tx_head; - unsigned int tail = queue->tx_tail; - struct macb *bp = queue->bp; - unsigned int head_idx, tbqp; - - if (bp->caps & MACB_CAPS_ISR_CLEAR_ON_WRITE) - queue_writel(queue, ISR, MACB_BIT(TXUBR)); - - if (head == tail) - return; - - tbqp = queue_readl(queue, TBQP) / macb_dma_desc_get_size(bp); - tbqp = macb_adj_dma_desc_idx(bp, macb_tx_ring_wrap(bp, tbqp)); - head_idx = macb_adj_dma_desc_idx(bp, macb_tx_ring_wrap(bp, head)); - - if (tbqp == head_idx) - return; - - macb_writel(bp, NCR, macb_readl(bp, NCR) | MACB_BIT(TSTART)); -} - static irqreturn_t macb_wol_interrupt(int irq, void *dev_id) { struct macb_queue *queue = dev_id; @@ -1765,9 +1824,27 @@ static irqreturn_t macb_interrupt(int irq, void *dev_id) if (bp->caps & MACB_CAPS_ISR_CLEAR_ON_WRITE) queue_writel(queue, ISR, MACB_BIT(RCOMP)); - if (napi_schedule_prep(&queue->napi)) { + if (napi_schedule_prep(&queue->napi_rx)) { netdev_vdbg(bp->dev, "scheduling RX softirq\n"); - __napi_schedule(&queue->napi); + __napi_schedule(&queue->napi_rx); + } + } + + if (status & (MACB_BIT(TCOMP) | + MACB_BIT(TXUBR))) { + queue_writel(queue, IDR, MACB_BIT(TCOMP)); + if (bp->caps & MACB_CAPS_ISR_CLEAR_ON_WRITE) + queue_writel(queue, ISR, MACB_BIT(TCOMP) | + MACB_BIT(TXUBR)); + + if (status & MACB_BIT(TXUBR)) { + queue->txubr_pending = true; + wmb(); // ensure softirq can see update + } + + if (napi_schedule_prep(&queue->napi_tx)) { + netdev_vdbg(bp->dev, "scheduling TX softirq\n"); + __napi_schedule(&queue->napi_tx); } } @@ -1781,12 +1858,6 @@ static irqreturn_t macb_interrupt(int irq, void *dev_id) break; } - if (status & MACB_BIT(TCOMP)) - macb_tx_interrupt(queue); - - if (status & MACB_BIT(TXUBR)) - macb_tx_restart(queue); - /* Link change detection isn't possible with RMII, so we'll * add that if/when we get our hands on a full-blown MII PHY. */ @@ -2019,7 +2090,7 @@ static unsigned int macb_tx_map(struct macb *bp, for (i = queue->tx_head; i != tx_head; i++) { tx_skb = macb_tx_skb(queue, i); - macb_tx_unmap(bp, tx_skb); + macb_tx_unmap(bp, tx_skb, 0); } return 0; @@ -2141,7 +2212,6 @@ static netdev_tx_t macb_start_xmit(struct sk_buff *skb, struct net_device *dev) u16 queue_index = skb_get_queue_mapping(skb); struct macb *bp = netdev_priv(dev); struct macb_queue *queue = &bp->queues[queue_index]; - unsigned long flags; unsigned int desc_cnt, nr_frags, frag_size, f; unsigned int hdrlen; bool is_lso; @@ -2198,16 +2268,16 @@ static netdev_tx_t macb_start_xmit(struct sk_buff *skb, struct net_device *dev) desc_cnt += DIV_ROUND_UP(frag_size, bp->max_tx_length); } - spin_lock_irqsave(&bp->lock, flags); + spin_lock_bh(&queue->tx_ptr_lock); /* This is a hard error, log it. */ if (CIRC_SPACE(queue->tx_head, queue->tx_tail, bp->tx_ring_size) < desc_cnt) { netif_stop_subqueue(dev, queue_index); - spin_unlock_irqrestore(&bp->lock, flags); netdev_dbg(bp->dev, "tx_head = %u, tx_tail = %u\n", queue->tx_head, queue->tx_tail); - return NETDEV_TX_BUSY; + ret = NETDEV_TX_BUSY; + goto unlock; } /* Map socket buffer for DMA transfer */ @@ -2220,13 +2290,15 @@ static netdev_tx_t macb_start_xmit(struct sk_buff *skb, struct net_device *dev) wmb(); skb_tx_timestamp(skb); + spin_lock_irq(&bp->lock); macb_writel(bp, NCR, macb_readl(bp, NCR) | MACB_BIT(TSTART)); + spin_unlock_irq(&bp->lock); if (CIRC_SPACE(queue->tx_head, queue->tx_tail, bp->tx_ring_size) < 1) netif_stop_subqueue(dev, queue_index); unlock: - spin_unlock_irqrestore(&bp->lock, flags); + spin_unlock_bh(&queue->tx_ptr_lock); return ret; } @@ -2760,8 +2832,10 @@ static int macb_open(struct net_device *dev) goto pm_exit; } - for (q = 0, queue = bp->queues; q < bp->num_queues; ++q, ++queue) - napi_enable(&queue->napi); + for (q = 0, queue = bp->queues; q < bp->num_queues; ++q, ++queue) { + napi_enable(&queue->napi_rx); + napi_enable(&queue->napi_tx); + } macb_init_hw(bp); @@ -2785,8 +2859,10 @@ static int macb_open(struct net_device *dev) reset_hw: macb_reset_hw(bp); - for (q = 0, queue = bp->queues; q < bp->num_queues; ++q, ++queue) - napi_disable(&queue->napi); + for (q = 0, queue = bp->queues; q < bp->num_queues; ++q, ++queue) { + napi_disable(&queue->napi_rx); + napi_disable(&queue->napi_tx); + } macb_free_consistent(bp); pm_exit: pm_runtime_put_sync(&bp->pdev->dev); @@ -2802,8 +2878,10 @@ static int macb_close(struct net_device *dev) netif_tx_stop_all_queues(dev); - for (q = 0, queue = bp->queues; q < bp->num_queues; ++q, ++queue) - napi_disable(&queue->napi); + for (q = 0, queue = bp->queues; q < bp->num_queues; ++q, ++queue) { + napi_disable(&queue->napi_rx); + napi_disable(&queue->napi_tx); + } phylink_stop(bp->phylink); phylink_disconnect_phy(bp->phylink); @@ -3865,7 +3943,9 @@ static int macb_init(struct platform_device *pdev) queue = &bp->queues[q]; queue->bp = bp; - netif_napi_add(dev, &queue->napi, macb_poll, NAPI_POLL_WEIGHT); + spin_lock_init(&queue->tx_ptr_lock); + netif_napi_add(dev, &queue->napi_rx, macb_rx_poll, NAPI_POLL_WEIGHT); + netif_napi_add(dev, &queue->napi_tx, macb_tx_poll, NAPI_POLL_WEIGHT); if (hw_q) { queue->ISR = GEM_ISR(hw_q - 1); queue->IER = GEM_IER(hw_q - 1); @@ -4972,8 +5052,10 @@ static int __maybe_unused macb_suspend(struct device *dev) netif_device_detach(netdev); for (q = 0, queue = bp->queues; q < bp->num_queues; - ++q, ++queue) - napi_disable(&queue->napi); + ++q, ++queue) { + napi_disable(&queue->napi_rx); + napi_disable(&queue->napi_tx); + } if (!(bp->wol & MACB_WOL_ENABLED)) { rtnl_lock(); @@ -5051,8 +5133,10 @@ static int __maybe_unused macb_resume(struct device *dev) } for (q = 0, queue = bp->queues; q < bp->num_queues; - ++q, ++queue) - napi_enable(&queue->napi); + ++q, ++queue) { + napi_enable(&queue->napi_rx); + napi_enable(&queue->napi_tx); + } if (netdev->hw_features & NETIF_F_NTUPLE) gem_writel_n(bp, ETHT, SCRT2_ETHT, bp->pm_data.scrt2);