diff mbox series

[v2] ntb_netdev: Use dev_kfree_skb_any() in interrupt context

Message ID 20221209000659.8318-1-epilmore@gigaio.com (mailing list archive)
State Accepted
Commit 5f7d78b2b12a9d561f48fa00bab29b40f4616dad
Headers show
Series [v2] ntb_netdev: Use dev_kfree_skb_any() in interrupt context | expand

Commit Message

Eric Pilmore Dec. 9, 2022, 12:06 a.m. UTC 
From: Eric Pilmore <epilmore@gigaio.com>

TX/RX callback handlers (ntb_netdev_tx_handler(),
ntb_netdev_rx_handler()) can be called in interrupt
context via the DMA framework when the respective
DMA operations have completed. As such, any calls
by these routines to free skb's, should use the
interrupt context safe dev_kfree_skb_any() function.

Previously, these callback handlers would call the
interrupt unsafe version of dev_kfree_skb(). This has
not presented an issue on Intel IOAT DMA engines as
that driver utilizes tasklets rather than a hard
interrupt handler, like the AMD PTDMA DMA driver.
On AMD systems, a kernel WARNING message is
encountered, which is being issued from
skb_release_head_state() due to in_hardirq()
being true.

Besides the user visible WARNING from the kernel,
the other symptom of this bug was that TCP/IP performance
across the ntb_netdev interface was very poor, i.e.
approximately an order of magnitude below what was
expected. With the repair to use dev_kfree_skb_any(),
kernel WARNINGs from skb_release_head_state() ceased
and TCP/IP performance, as measured by iperf, was on
par with expected results, approximately 20 Gb/s on
AMD Milan based server. Note that this performance
is comparable with Intel based servers.

Fixes: 765ccc7bc3d91 ("ntb_netdev: correct skb leak")
Fixes: 548c237c0a997 ("net: Add support for NTB virtual ethernet device")
Signed-off-by: Eric Pilmore <epilmore@gigaio.com>
---
 drivers/net/ntb_netdev.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Comments

Dave Jiang Dec. 9, 2022, 12:20 a.m. UTC | #1 
On 12/8/2022 5:06 PM, epilmore@gigaio.com wrote:
> From: Eric Pilmore <epilmore@gigaio.com>
> 
> TX/RX callback handlers (ntb_netdev_tx_handler(),
> ntb_netdev_rx_handler()) can be called in interrupt
> context via the DMA framework when the respective
> DMA operations have completed. As such, any calls
> by these routines to free skb's, should use the
> interrupt context safe dev_kfree_skb_any() function.
> 
> Previously, these callback handlers would call the
> interrupt unsafe version of dev_kfree_skb(). This has
> not presented an issue on Intel IOAT DMA engines as
> that driver utilizes tasklets rather than a hard
> interrupt handler, like the AMD PTDMA DMA driver.
> On AMD systems, a kernel WARNING message is
> encountered, which is being issued from
> skb_release_head_state() due to in_hardirq()
> being true.
> 
> Besides the user visible WARNING from the kernel,
> the other symptom of this bug was that TCP/IP performance
> across the ntb_netdev interface was very poor, i.e.
> approximately an order of magnitude below what was
> expected. With the repair to use dev_kfree_skb_any(),
> kernel WARNINGs from skb_release_head_state() ceased
> and TCP/IP performance, as measured by iperf, was on
> par with expected results, approximately 20 Gb/s on
> AMD Milan based server. Note that this performance
> is comparable with Intel based servers.
> 
> Fixes: 765ccc7bc3d91 ("ntb_netdev: correct skb leak")
> Fixes: 548c237c0a997 ("net: Add support for NTB virtual ethernet device")
> Signed-off-by: Eric Pilmore <epilmore@gigaio.com>

Reviewed-by: Dave Jiang <dave.jiang@intel.com>

As an FYI for future. Typically you would add the patch revision change 
log under the "---" line just as an FYI for reviewers on what you've 
changed and who suggested the change.

> ---

i.e.

v2:
- Use dev_kfree_skb_any() instead of dev_kfree_skb_irq(). (DaveJ)


>   drivers/net/ntb_netdev.c | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/net/ntb_netdev.c b/drivers/net/ntb_netdev.c
> index 80bdc07f2cd3..59250b7accfb 100644
> --- a/drivers/net/ntb_netdev.c
> +++ b/drivers/net/ntb_netdev.c
> @@ -137,7 +137,7 @@ static void ntb_netdev_rx_handler(struct ntb_transport_qp *qp, void *qp_data,
>   enqueue_again:
>   	rc = ntb_transport_rx_enqueue(qp, skb, skb->data, ndev->mtu + ETH_HLEN);
>   	if (rc) {
> -		dev_kfree_skb(skb);
> +		dev_kfree_skb_any(skb);
>   		ndev->stats.rx_errors++;
>   		ndev->stats.rx_fifo_errors++;
>   	}
> @@ -192,7 +192,7 @@ static void ntb_netdev_tx_handler(struct ntb_transport_qp *qp, void *qp_data,
>   		ndev->stats.tx_aborted_errors++;
>   	}
>   
> -	dev_kfree_skb(skb);
> +	dev_kfree_skb_any(skb);
>   
>   	if (ntb_transport_tx_free_entry(dev->qp) >= tx_start) {
>   		/* Make sure anybody stopping the queue after this sees the new
patchwork-bot+netdevbpf@kernel.org Dec. 12, 2022, 9 p.m. UTC | #2 
Hello:

This patch was applied to netdev/net.git (master)
by Jakub Kicinski <kuba@kernel.org>:

On Thu,  8 Dec 2022 16:06:59 -0800 you wrote:
> From: Eric Pilmore <epilmore@gigaio.com>
> 
> TX/RX callback handlers (ntb_netdev_tx_handler(),
> ntb_netdev_rx_handler()) can be called in interrupt
> context via the DMA framework when the respective
> DMA operations have completed. As such, any calls
> by these routines to free skb's, should use the
> interrupt context safe dev_kfree_skb_any() function.
> 
> [...]

Here is the summary with links:
  - [v2] ntb_netdev: Use dev_kfree_skb_any() in interrupt context
    https://git.kernel.org/netdev/net/c/5f7d78b2b12a

You are awesome, thank you!
diff mbox series

Patch

diff --git a/drivers/net/ntb_netdev.c b/drivers/net/ntb_netdev.c
index 80bdc07f2cd3..59250b7accfb 100644
--- a/drivers/net/ntb_netdev.c
+++ b/drivers/net/ntb_netdev.c
@@ -137,7 +137,7 @@  static void ntb_netdev_rx_handler(struct ntb_transport_qp *qp, void *qp_data,
 enqueue_again:
 	rc = ntb_transport_rx_enqueue(qp, skb, skb->data, ndev->mtu + ETH_HLEN);
 	if (rc) {
-		dev_kfree_skb(skb);
+		dev_kfree_skb_any(skb);
 		ndev->stats.rx_errors++;
 		ndev->stats.rx_fifo_errors++;
 	}
@@ -192,7 +192,7 @@  static void ntb_netdev_tx_handler(struct ntb_transport_qp *qp, void *qp_data,
 		ndev->stats.tx_aborted_errors++;
 	}
 
-	dev_kfree_skb(skb);
+	dev_kfree_skb_any(skb);
 
 	if (ntb_transport_tx_free_entry(dev->qp) >= tx_start) {
 		/* Make sure anybody stopping the queue after this sees the new