| Message ID | 92e01f05c4f506a4f0a9b39c10175dcc01994910.1729866134.git.petrm@nvidia.com (mailing list archive) |
|---|---|
| State | New |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | mlxsw: Fixes | expand |
From: Petr Machata <petrm@nvidia.com> Date: Fri, 25 Oct 2024 16:26:27 +0200 > From: Amit Cohen <amcohen@nvidia.com> > > Non-coherent architectures, like ARM, may require invalidating caches > before the device can use the DMA mapped memory, which means that before > posting pages to device, drivers should sync the memory for device. > > Sync for device can be configured as page pool responsibility. Set the > relevant flag and define max_len for sync. > > Cc: Jiri Pirko <jiri@resnulli.us> > Fixes: b5b60bb491b2 ("mlxsw: pci: Use page pool for Rx buffers allocation") > Signed-off-by: Amit Cohen <amcohen@nvidia.com> > Reviewed-by: Ido Schimmel <idosch@nvidia.com> > Signed-off-by: Petr Machata <petrm@nvidia.com> > --- > drivers/net/ethernet/mellanox/mlxsw/pci.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/drivers/net/ethernet/mellanox/mlxsw/pci.c b/drivers/net/ethernet/mellanox/mlxsw/pci.c > index 2320a5f323b4..d6f37456fb31 100644 > --- a/drivers/net/ethernet/mellanox/mlxsw/pci.c > +++ b/drivers/net/ethernet/mellanox/mlxsw/pci.c > @@ -996,12 +996,13 @@ static int mlxsw_pci_cq_page_pool_init(struct mlxsw_pci_queue *q, > if (cq_type != MLXSW_PCI_CQ_RDQ) > return 0; > > - pp_params.flags = PP_FLAG_DMA_MAP; > + pp_params.flags = PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV; > pp_params.pool_size = MLXSW_PCI_WQE_COUNT * mlxsw_pci->num_sg_entries; > pp_params.nid = dev_to_node(&mlxsw_pci->pdev->dev); > pp_params.dev = &mlxsw_pci->pdev->dev; > pp_params.napi = &q->u.cq.napi; > pp_params.dma_dir = DMA_FROM_DEVICE; > + pp_params.max_len = PAGE_SIZE; max_len is the maximum HW-writable area of a buffer. Headroom and tailroom must be excluded. In your case pp_params.max_len = PAGE_SIZE - MLXSW_PCI_RX_BUF_SW_OVERHEAD; > > page_pool = page_pool_create(&pp_params); > if (IS_ERR(page_pool)) Thanks, Olek
> -----Original Message----- > From: Alexander Lobakin <aleksander.lobakin@intel.com> > Sent: Friday, 25 October 2024 18:03 > To: Petr Machata <petrm@nvidia.com>; Amit Cohen <amcohen@nvidia.com> > Cc: netdev@vger.kernel.org; Andrew Lunn <andrew+netdev@lunn.ch>; David S. Miller <davem@davemloft.net>; Eric Dumazet > <edumazet@google.com>; Jakub Kicinski <kuba@kernel.org>; Paolo Abeni <pabeni@redhat.com>; Simon Horman <horms@kernel.org>; > Danielle Ratson <danieller@nvidia.com>; Ido Schimmel <idosch@nvidia.com>; mlxsw <mlxsw@nvidia.com>; Jiri Pirko <jiri@resnulli.us> > Subject: Re: [PATCH net 3/5] mlxsw: pci: Sync Rx buffers for device > > From: Petr Machata <petrm@nvidia.com> > Date: Fri, 25 Oct 2024 16:26:27 +0200 > > > From: Amit Cohen <amcohen@nvidia.com> > > > > Non-coherent architectures, like ARM, may require invalidating caches > > before the device can use the DMA mapped memory, which means that before > > posting pages to device, drivers should sync the memory for device. > > > > Sync for device can be configured as page pool responsibility. Set the > > relevant flag and define max_len for sync. > > > > Cc: Jiri Pirko <jiri@resnulli.us> > > Fixes: b5b60bb491b2 ("mlxsw: pci: Use page pool for Rx buffers allocation") > > Signed-off-by: Amit Cohen <amcohen@nvidia.com> > > Reviewed-by: Ido Schimmel <idosch@nvidia.com> > > Signed-off-by: Petr Machata <petrm@nvidia.com> > > --- > > drivers/net/ethernet/mellanox/mlxsw/pci.c | 3 ++- > > 1 file changed, 2 insertions(+), 1 deletion(-) > > > > diff --git a/drivers/net/ethernet/mellanox/mlxsw/pci.c b/drivers/net/ethernet/mellanox/mlxsw/pci.c > > index 2320a5f323b4..d6f37456fb31 100644 > > --- a/drivers/net/ethernet/mellanox/mlxsw/pci.c > > +++ b/drivers/net/ethernet/mellanox/mlxsw/pci.c > > @@ -996,12 +996,13 @@ static int mlxsw_pci_cq_page_pool_init(struct mlxsw_pci_queue *q, > > if (cq_type != MLXSW_PCI_CQ_RDQ) > > return 0; > > > > - pp_params.flags = PP_FLAG_DMA_MAP; > > + pp_params.flags = PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV; > > pp_params.pool_size = MLXSW_PCI_WQE_COUNT * mlxsw_pci->num_sg_entries; > > pp_params.nid = dev_to_node(&mlxsw_pci->pdev->dev); > > pp_params.dev = &mlxsw_pci->pdev->dev; > > pp_params.napi = &q->u.cq.napi; > > pp_params.dma_dir = DMA_FROM_DEVICE; > > + pp_params.max_len = PAGE_SIZE; > > max_len is the maximum HW-writable area of a buffer. Headroom and > tailroom must be excluded. In your case > > pp_params.max_len = PAGE_SIZE - MLXSW_PCI_RX_BUF_SW_OVERHEAD; > mlxsw driver uses fragmented buffers and the page pool is used to allocate the buffers for all scatter/gather entries. For each packet, the HW-writable area of a buffer of the *first* entry is 'PAGE_SIZE - MLXSW_PCI_RX_BUF_SW_OVERHEAD', but for other entries we map PAGE_SIZE to HW. That's why we set page pool to sync PAGE_SIZE and use offset=0. > > > > page_pool = page_pool_create(&pp_params); > > if (IS_ERR(page_pool)) > > Thanks, > Olek
diff --git a/drivers/net/ethernet/mellanox/mlxsw/pci.c b/drivers/net/ethernet/mellanox/mlxsw/pci.c index 2320a5f323b4..d6f37456fb31 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/pci.c +++ b/drivers/net/ethernet/mellanox/mlxsw/pci.c @@ -996,12 +996,13 @@ static int mlxsw_pci_cq_page_pool_init(struct mlxsw_pci_queue *q, if (cq_type != MLXSW_PCI_CQ_RDQ) return 0; - pp_params.flags = PP_FLAG_DMA_MAP; + pp_params.flags = PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV; pp_params.pool_size = MLXSW_PCI_WQE_COUNT * mlxsw_pci->num_sg_entries; pp_params.nid = dev_to_node(&mlxsw_pci->pdev->dev); pp_params.dev = &mlxsw_pci->pdev->dev; pp_params.napi = &q->u.cq.napi; pp_params.dma_dir = DMA_FROM_DEVICE; + pp_params.max_len = PAGE_SIZE; page_pool = page_pool_create(&pp_params); if (IS_ERR(page_pool))