| Message ID | 20220111211358.2699350-7-robert.hancock@calian.com (mailing list archive) |
|---|---|
| State | Superseded |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | Xilinx axienet fixes | expand |
On Tue, 11 Jan 2022 15:13:57 -0600 Robert Hancock wrote: > We should be avoiding returning NETDEV_TX_BUSY from ndo_start_xmit in > normal cases. Move the main check for a full TX ring to the end of the > function so that we stop the queue after the last available space is used > up, and only wake up the queue if enough space is available for a full > maximally fragmented packet. Print a warning if there is insufficient > space at the start of start_xmit, since this should no longer happen. > > Fixes: 8a3b7a252dca9 ("drivers/net/ethernet/xilinx: added Xilinx AXI Ethernet driver") > Signed-off-by: Robert Hancock <robert.hancock@calian.com> Feels a little more like an optimization than strictly a fix. Can we apply this and the following patch to net-next in two week's time? It's not too much of a stretch to take it in now if it's a bit convenience but I don't think the Fixes tags should stay. > - netif_wake_queue(ndev); > + netdev_warn(ndev, "TX ring unexpectedly full\n"); Probably wise to make this netdev_warn_once() or at least rate limit it. > + return NETDEV_TX_BUSY; > } > > if (skb->ip_summed == CHECKSUM_PARTIAL) {
On Tue, 2022-01-11 at 19:49 -0800, Jakub Kicinski wrote: > On Tue, 11 Jan 2022 15:13:57 -0600 Robert Hancock wrote: > > We should be avoiding returning NETDEV_TX_BUSY from ndo_start_xmit in > > normal cases. Move the main check for a full TX ring to the end of the > > function so that we stop the queue after the last available space is used > > up, and only wake up the queue if enough space is available for a full > > maximally fragmented packet. Print a warning if there is insufficient > > space at the start of start_xmit, since this should no longer happen. > > > > Fixes: 8a3b7a252dca9 ("drivers/net/ethernet/xilinx: added Xilinx AXI > > Ethernet driver") > > Signed-off-by: Robert Hancock <robert.hancock@calian.com> > > Feels a little more like an optimization than strictly a fix. > Can we apply this and the following patch to net-next in two > week's time? It's not too much of a stretch to take it in now > if it's a bit convenience but I don't think the Fixes tags should > stay. Well it's a fix in the sense that it complies with what Documentation/networking/driver.rst says drivers should do - I'm not too familiar with the consequences of not doing that are, I guess mostly performance from having to requeue the packet? From that standpoint, I guess the concern with breaking those two patches out is that the previous patches can introduce a bit of a performance hit (by actually caring about the state of the TX ring instead of trampling over it in some cases) and so without the last two you might end up with some performance regression. So I'd probably prefer to keep them together with the rest of the patch set. > > > - netif_wake_queue(ndev); > > + netdev_warn(ndev, "TX ring unexpectedly full\n"); > > Probably wise to make this netdev_warn_once() or at least rate limit it. Might want it more than once (so you can tell if it is a one-off or happening more often), but I can put in a rate limit.. > > > + return NETDEV_TX_BUSY; > > } > > > > if (skb->ip_summed == CHECKSUM_PARTIAL) {
On Wed, 12 Jan 2022 16:45:18 +0000 Robert Hancock wrote: > On Tue, 2022-01-11 at 19:49 -0800, Jakub Kicinski wrote: > > On Tue, 11 Jan 2022 15:13:57 -0600 Robert Hancock wrote: > > > We should be avoiding returning NETDEV_TX_BUSY from ndo_start_xmit in > > > normal cases. Move the main check for a full TX ring to the end of the > > > function so that we stop the queue after the last available space is used > > > up, and only wake up the queue if enough space is available for a full > > > maximally fragmented packet. Print a warning if there is insufficient > > > space at the start of start_xmit, since this should no longer happen. > > > > > > Fixes: 8a3b7a252dca9 ("drivers/net/ethernet/xilinx: added Xilinx AXI > > > Ethernet driver") > > > Signed-off-by: Robert Hancock <robert.hancock@calian.com> > > > > Feels a little more like an optimization than strictly a fix. > > Can we apply this and the following patch to net-next in two > > week's time? It's not too much of a stretch to take it in now > > if it's a bit convenience but I don't think the Fixes tags should > > stay. > > Well it's a fix in the sense that it complies with what > Documentation/networking/driver.rst says drivers should do - I'm not too > familiar with the consequences of not doing that are, I guess mostly > performance from having to requeue the packet? Yes, it's just the re-queuing overhead AFAIU. > From that standpoint, I guess the concern with breaking those two patches out > is that the previous patches can introduce a bit of a performance hit (by > actually caring about the state of the TX ring instead of trampling over it in > some cases) and so without the last two you might end up with some performance > regression. So I'd probably prefer to keep them together with the rest of the > patch set. Alright, if you have any numbers on this it'd be great to include them in the commit message.
On Wed, 2022-01-12 at 09:01 -0800, Jakub Kicinski wrote: > On Wed, 12 Jan 2022 16:45:18 +0000 Robert Hancock wrote: > > On Tue, 2022-01-11 at 19:49 -0800, Jakub Kicinski wrote: > > > On Tue, 11 Jan 2022 15:13:57 -0600 Robert Hancock wrote: > > > > We should be avoiding returning NETDEV_TX_BUSY from ndo_start_xmit in > > > > normal cases. Move the main check for a full TX ring to the end of the > > > > function so that we stop the queue after the last available space is > > > > used > > > > up, and only wake up the queue if enough space is available for a full > > > > maximally fragmented packet. Print a warning if there is insufficient > > > > space at the start of start_xmit, since this should no longer happen. > > > > > > > > Fixes: 8a3b7a252dca9 ("drivers/net/ethernet/xilinx: added Xilinx AXI > > > > Ethernet driver") > > > > Signed-off-by: Robert Hancock <robert.hancock@calian.com> > > > > > > Feels a little more like an optimization than strictly a fix. > > > Can we apply this and the following patch to net-next in two > > > week's time? It's not too much of a stretch to take it in now > > > if it's a bit convenience but I don't think the Fixes tags should > > > stay. > > > > Well it's a fix in the sense that it complies with what > > Documentation/networking/driver.rst says drivers should do - I'm not too > > familiar with the consequences of not doing that are, I guess mostly > > performance from having to requeue the packet? > > Yes, it's just the re-queuing overhead AFAIU. > > > From that standpoint, I guess the concern with breaking those two patches > > out > > is that the previous patches can introduce a bit of a performance hit (by > > actually caring about the state of the TX ring instead of trampling over it > > in > > some cases) and so without the last two you might end up with some > > performance > > regression. So I'd probably prefer to keep them together with the rest of > > the > > patch set. > > Alright, if you have any numbers on this it'd be great to include them > in the commit message. I don't have any numbers from that individual change unfortunately, just from both of the two together (the second change's commit message mentions the TX rate went from 600 Mbps up to near 1 Gbps on a 1 Gbps link). But I'll add some some more rationale to the commit message on this one.
diff --git a/drivers/net/ethernet/xilinx/xilinx_axienet_main.c b/drivers/net/ethernet/xilinx/xilinx_axienet_main.c index c5d214abd4d5..2191f813ed78 100644 --- a/drivers/net/ethernet/xilinx/xilinx_axienet_main.c +++ b/drivers/net/ethernet/xilinx/xilinx_axienet_main.c @@ -660,6 +660,32 @@ static int axienet_free_tx_chain(struct net_device *ndev, u32 first_bd, return i; } +/** + * axienet_check_tx_bd_space - Checks if a BD/group of BDs are currently busy + * @lp: Pointer to the axienet_local structure + * @num_frag: The number of BDs to check for + * + * Return: 0, on success + * NETDEV_TX_BUSY, if any of the descriptors are not free + * + * This function is invoked before BDs are allocated and transmission starts. + * This function returns 0 if a BD or group of BDs can be allocated for + * transmission. If the BD or any of the BDs are not free the function + * returns a busy status. This is invoked from axienet_start_xmit. + */ +static inline int axienet_check_tx_bd_space(struct axienet_local *lp, + int num_frag) +{ + struct axidma_bd *cur_p; + + /* Ensure we see all descriptor updates from device or TX IRQ path */ + rmb(); + cur_p = &lp->tx_bd_v[(lp->tx_bd_tail + num_frag) % lp->tx_bd_num]; + if (cur_p->cntrl) + return NETDEV_TX_BUSY; + return 0; +} + /** * axienet_start_xmit_done - Invoked once a transmit is completed by the * Axi DMA Tx channel. @@ -689,33 +715,8 @@ static void axienet_start_xmit_done(struct net_device *ndev) /* Matches barrier in axienet_start_xmit */ smp_mb(); - netif_wake_queue(ndev); -} - -/** - * axienet_check_tx_bd_space - Checks if a BD/group of BDs are currently busy - * @lp: Pointer to the axienet_local structure - * @num_frag: The number of BDs to check for - * - * Return: 0, on success - * NETDEV_TX_BUSY, if any of the descriptors are not free - * - * This function is invoked before BDs are allocated and transmission starts. - * This function returns 0 if a BD or group of BDs can be allocated for - * transmission. If the BD or any of the BDs are not free the function - * returns a busy status. This is invoked from axienet_start_xmit. - */ -static inline int axienet_check_tx_bd_space(struct axienet_local *lp, - int num_frag) -{ - struct axidma_bd *cur_p; - - /* Ensure we see all descriptor updates from device or TX IRQ path */ - rmb(); - cur_p = &lp->tx_bd_v[(lp->tx_bd_tail + num_frag) % lp->tx_bd_num]; - if (cur_p->cntrl) - return NETDEV_TX_BUSY; - return 0; + if (!axienet_check_tx_bd_space(lp, MAX_SKB_FRAGS + 1)) + netif_wake_queue(ndev); } /** @@ -748,19 +749,13 @@ axienet_start_xmit(struct sk_buff *skb, struct net_device *ndev) cur_p = &lp->tx_bd_v[lp->tx_bd_tail]; if (axienet_check_tx_bd_space(lp, num_frag + 1)) { - if (netif_queue_stopped(ndev)) - return NETDEV_TX_BUSY; - + /* Should not happen as last start_xmit call should have + * checked for sufficient space and queue should only be + * woken when sufficient space is available. + */ netif_stop_queue(ndev); - - /* Matches barrier in axienet_start_xmit_done */ - smp_mb(); - - /* Space might have just been freed - check again */ - if (axienet_check_tx_bd_space(lp, num_frag + 1)) - return NETDEV_TX_BUSY; - - netif_wake_queue(ndev); + netdev_warn(ndev, "TX ring unexpectedly full\n"); + return NETDEV_TX_BUSY; } if (skb->ip_summed == CHECKSUM_PARTIAL) { @@ -821,6 +816,18 @@ axienet_start_xmit(struct sk_buff *skb, struct net_device *ndev) if (++lp->tx_bd_tail >= lp->tx_bd_num) lp->tx_bd_tail = 0; + /* Stop queue if next transmit may not have space */ + if (axienet_check_tx_bd_space(lp, MAX_SKB_FRAGS + 1)) { + netif_stop_queue(ndev); + + /* Matches barrier in axienet_start_xmit_done */ + smp_mb(); + + /* Space might have just been freed - check again */ + if (!axienet_check_tx_bd_space(lp, MAX_SKB_FRAGS + 1)) + netif_wake_queue(ndev); + } + return NETDEV_TX_OK; }
We should be avoiding returning NETDEV_TX_BUSY from ndo_start_xmit in normal cases. Move the main check for a full TX ring to the end of the function so that we stop the queue after the last available space is used up, and only wake up the queue if enough space is available for a full maximally fragmented packet. Print a warning if there is insufficient space at the start of start_xmit, since this should no longer happen. Fixes: 8a3b7a252dca9 ("drivers/net/ethernet/xilinx: added Xilinx AXI Ethernet driver") Signed-off-by: Robert Hancock <robert.hancock@calian.com> --- .../net/ethernet/xilinx/xilinx_axienet_main.c | 85 ++++++++++--------- 1 file changed, 46 insertions(+), 39 deletions(-)