diff mbox series

[net,3/5] mlxsw: pci: Sync Rx buffers for device

Message ID 92e01f05c4f506a4f0a9b39c10175dcc01994910.1729866134.git.petrm@nvidia.com (mailing list archive)
State Accepted
Commit d0fbdc3ae9ecc614ddffde55dccbcacef353da0b
Delegated to: Netdev Maintainers
Headers show
Series mlxsw: Fixes | expand

Checks

Context Check Description
netdev/series_format success Posting correctly formatted
netdev/tree_selection success Clearly marked for net
netdev/ynl success Generated files up to date; no warnings/errors; no diff in generated;
netdev/fixes_present success Fixes tag present in non-next series
netdev/header_inline success No static functions without inline keyword in header files
netdev/build_32bit success Errors and warnings before: 5 this patch: 5
netdev/build_tools success No tools touched, skip
netdev/cc_maintainers success CCed 9 of 9 maintainers
netdev/build_clang success Errors and warnings before: 3 this patch: 3
netdev/verify_signedoff success Signed-off-by tag matches author and committer
netdev/deprecated_api success None detected
netdev/check_selftest success No net selftest shell script
netdev/verify_fixes success Fixes tag looks correct
netdev/build_allmodconfig_warn success Errors and warnings before: 4 this patch: 4
netdev/checkpatch success total: 0 errors, 0 warnings, 0 checks, 14 lines checked
netdev/build_clang_rust success No Rust files in patch. Skipping build
netdev/kdoc success Errors and warnings before: 0 this patch: 0
netdev/source_inline success Was 0 now: 0
netdev/contest success net-next-2024-10-26--21-00 (tests: 777)

Commit Message

Petr Machata Oct. 25, 2024, 2:26 p.m. UTC 
From: Amit Cohen <amcohen@nvidia.com>

Non-coherent architectures, like ARM, may require invalidating caches
before the device can use the DMA mapped memory, which means that before
posting pages to device, drivers should sync the memory for device.

Sync for device can be configured as page pool responsibility. Set the
relevant flag and define max_len for sync.

Cc: Jiri Pirko <jiri@resnulli.us>
Fixes: b5b60bb491b2 ("mlxsw: pci: Use page pool for Rx buffers allocation")
Signed-off-by: Amit Cohen <amcohen@nvidia.com>
Reviewed-by: Ido Schimmel <idosch@nvidia.com>
Signed-off-by: Petr Machata <petrm@nvidia.com>
---
 drivers/net/ethernet/mellanox/mlxsw/pci.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Comments

Alexander Lobakin Oct. 25, 2024, 3:02 p.m. UTC | #1 
From: Petr Machata <petrm@nvidia.com>
Date: Fri, 25 Oct 2024 16:26:27 +0200

> From: Amit Cohen <amcohen@nvidia.com>
> 
> Non-coherent architectures, like ARM, may require invalidating caches
> before the device can use the DMA mapped memory, which means that before
> posting pages to device, drivers should sync the memory for device.
> 
> Sync for device can be configured as page pool responsibility. Set the
> relevant flag and define max_len for sync.
> 
> Cc: Jiri Pirko <jiri@resnulli.us>
> Fixes: b5b60bb491b2 ("mlxsw: pci: Use page pool for Rx buffers allocation")
> Signed-off-by: Amit Cohen <amcohen@nvidia.com>
> Reviewed-by: Ido Schimmel <idosch@nvidia.com>
> Signed-off-by: Petr Machata <petrm@nvidia.com>
> ---
>  drivers/net/ethernet/mellanox/mlxsw/pci.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/net/ethernet/mellanox/mlxsw/pci.c b/drivers/net/ethernet/mellanox/mlxsw/pci.c
> index 2320a5f323b4..d6f37456fb31 100644
> --- a/drivers/net/ethernet/mellanox/mlxsw/pci.c
> +++ b/drivers/net/ethernet/mellanox/mlxsw/pci.c
> @@ -996,12 +996,13 @@ static int mlxsw_pci_cq_page_pool_init(struct mlxsw_pci_queue *q,
>  	if (cq_type != MLXSW_PCI_CQ_RDQ)
>  		return 0;
>  
> -	pp_params.flags = PP_FLAG_DMA_MAP;
> +	pp_params.flags = PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV;
>  	pp_params.pool_size = MLXSW_PCI_WQE_COUNT * mlxsw_pci->num_sg_entries;
>  	pp_params.nid = dev_to_node(&mlxsw_pci->pdev->dev);
>  	pp_params.dev = &mlxsw_pci->pdev->dev;
>  	pp_params.napi = &q->u.cq.napi;
>  	pp_params.dma_dir = DMA_FROM_DEVICE;
> +	pp_params.max_len = PAGE_SIZE;

max_len is the maximum HW-writable area of a buffer. Headroom and
tailroom must be excluded. In your case

	pp_params.max_len = PAGE_SIZE - MLXSW_PCI_RX_BUF_SW_OVERHEAD;

>  
>  	page_pool = page_pool_create(&pp_params);
>  	if (IS_ERR(page_pool))

Thanks,
Olek
Amit Cohen Oct. 27, 2024, 6:51 a.m. UTC | #2 
> -----Original Message-----
> From: Alexander Lobakin <aleksander.lobakin@intel.com>
> Sent: Friday, 25 October 2024 18:03
> To: Petr Machata <petrm@nvidia.com>; Amit Cohen <amcohen@nvidia.com>
> Cc: netdev@vger.kernel.org; Andrew Lunn <andrew+netdev@lunn.ch>; David S. Miller <davem@davemloft.net>; Eric Dumazet
> <edumazet@google.com>; Jakub Kicinski <kuba@kernel.org>; Paolo Abeni <pabeni@redhat.com>; Simon Horman <horms@kernel.org>;
> Danielle Ratson <danieller@nvidia.com>; Ido Schimmel <idosch@nvidia.com>; mlxsw <mlxsw@nvidia.com>; Jiri Pirko <jiri@resnulli.us>
> Subject: Re: [PATCH net 3/5] mlxsw: pci: Sync Rx buffers for device
> 
> From: Petr Machata <petrm@nvidia.com>
> Date: Fri, 25 Oct 2024 16:26:27 +0200
> 
> > From: Amit Cohen <amcohen@nvidia.com>
> >
> > Non-coherent architectures, like ARM, may require invalidating caches
> > before the device can use the DMA mapped memory, which means that before
> > posting pages to device, drivers should sync the memory for device.
> >
> > Sync for device can be configured as page pool responsibility. Set the
> > relevant flag and define max_len for sync.
> >
> > Cc: Jiri Pirko <jiri@resnulli.us>
> > Fixes: b5b60bb491b2 ("mlxsw: pci: Use page pool for Rx buffers allocation")
> > Signed-off-by: Amit Cohen <amcohen@nvidia.com>
> > Reviewed-by: Ido Schimmel <idosch@nvidia.com>
> > Signed-off-by: Petr Machata <petrm@nvidia.com>
> > ---
> >  drivers/net/ethernet/mellanox/mlxsw/pci.c | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/net/ethernet/mellanox/mlxsw/pci.c b/drivers/net/ethernet/mellanox/mlxsw/pci.c
> > index 2320a5f323b4..d6f37456fb31 100644
> > --- a/drivers/net/ethernet/mellanox/mlxsw/pci.c
> > +++ b/drivers/net/ethernet/mellanox/mlxsw/pci.c
> > @@ -996,12 +996,13 @@ static int mlxsw_pci_cq_page_pool_init(struct mlxsw_pci_queue *q,
> >  	if (cq_type != MLXSW_PCI_CQ_RDQ)
> >  		return 0;
> >
> > -	pp_params.flags = PP_FLAG_DMA_MAP;
> > +	pp_params.flags = PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV;
> >  	pp_params.pool_size = MLXSW_PCI_WQE_COUNT * mlxsw_pci->num_sg_entries;
> >  	pp_params.nid = dev_to_node(&mlxsw_pci->pdev->dev);
> >  	pp_params.dev = &mlxsw_pci->pdev->dev;
> >  	pp_params.napi = &q->u.cq.napi;
> >  	pp_params.dma_dir = DMA_FROM_DEVICE;
> > +	pp_params.max_len = PAGE_SIZE;
> 
> max_len is the maximum HW-writable area of a buffer. Headroom and
> tailroom must be excluded. In your case
> 
> 	pp_params.max_len = PAGE_SIZE - MLXSW_PCI_RX_BUF_SW_OVERHEAD;
> 

mlxsw driver uses fragmented buffers and the page pool is used to allocate the buffers for all scatter/gather entries.
For each packet, the HW-writable area of a buffer of the *first* entry is 'PAGE_SIZE - MLXSW_PCI_RX_BUF_SW_OVERHEAD', but for other entries we map PAGE_SIZE to HW.
That's why we set page pool to sync PAGE_SIZE and use offset=0.

> >
> >  	page_pool = page_pool_create(&pp_params);
> >  	if (IS_ERR(page_pool))
> 
> Thanks,
> Olek
Alexander Lobakin Oct. 29, 2024, 3:12 p.m. UTC | #3 
From: Amit Cohen <amcohen@nvidia.com>
Date: Sun, 27 Oct 2024 06:51:00 +0000

> 
> 
>> -----Original Message-----
>> From: Alexander Lobakin <aleksander.lobakin@intel.com>
>> Sent: Friday, 25 October 2024 18:03
>> To: Petr Machata <petrm@nvidia.com>; Amit Cohen <amcohen@nvidia.com>
>> Cc: netdev@vger.kernel.org; Andrew Lunn <andrew+netdev@lunn.ch>; David S. Miller <davem@davemloft.net>; Eric Dumazet
>> <edumazet@google.com>; Jakub Kicinski <kuba@kernel.org>; Paolo Abeni <pabeni@redhat.com>; Simon Horman <horms@kernel.org>;
>> Danielle Ratson <danieller@nvidia.com>; Ido Schimmel <idosch@nvidia.com>; mlxsw <mlxsw@nvidia.com>; Jiri Pirko <jiri@resnulli.us>
>> Subject: Re: [PATCH net 3/5] mlxsw: pci: Sync Rx buffers for device
>>
>> From: Petr Machata <petrm@nvidia.com>
>> Date: Fri, 25 Oct 2024 16:26:27 +0200
>>
>>> From: Amit Cohen <amcohen@nvidia.com>
>>>
>>> Non-coherent architectures, like ARM, may require invalidating caches
>>> before the device can use the DMA mapped memory, which means that before
>>> posting pages to device, drivers should sync the memory for device.
>>>
>>> Sync for device can be configured as page pool responsibility. Set the
>>> relevant flag and define max_len for sync.
>>>
>>> Cc: Jiri Pirko <jiri@resnulli.us>
>>> Fixes: b5b60bb491b2 ("mlxsw: pci: Use page pool for Rx buffers allocation")
>>> Signed-off-by: Amit Cohen <amcohen@nvidia.com>
>>> Reviewed-by: Ido Schimmel <idosch@nvidia.com>
>>> Signed-off-by: Petr Machata <petrm@nvidia.com>
>>> ---
>>>  drivers/net/ethernet/mellanox/mlxsw/pci.c | 3 ++-
>>>  1 file changed, 2 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/net/ethernet/mellanox/mlxsw/pci.c b/drivers/net/ethernet/mellanox/mlxsw/pci.c
>>> index 2320a5f323b4..d6f37456fb31 100644
>>> --- a/drivers/net/ethernet/mellanox/mlxsw/pci.c
>>> +++ b/drivers/net/ethernet/mellanox/mlxsw/pci.c
>>> @@ -996,12 +996,13 @@ static int mlxsw_pci_cq_page_pool_init(struct mlxsw_pci_queue *q,
>>>  	if (cq_type != MLXSW_PCI_CQ_RDQ)
>>>  		return 0;
>>>
>>> -	pp_params.flags = PP_FLAG_DMA_MAP;
>>> +	pp_params.flags = PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV;
>>>  	pp_params.pool_size = MLXSW_PCI_WQE_COUNT * mlxsw_pci->num_sg_entries;
>>>  	pp_params.nid = dev_to_node(&mlxsw_pci->pdev->dev);
>>>  	pp_params.dev = &mlxsw_pci->pdev->dev;
>>>  	pp_params.napi = &q->u.cq.napi;
>>>  	pp_params.dma_dir = DMA_FROM_DEVICE;
>>> +	pp_params.max_len = PAGE_SIZE;
>>
>> max_len is the maximum HW-writable area of a buffer. Headroom and
>> tailroom must be excluded. In your case
>>
>> 	pp_params.max_len = PAGE_SIZE - MLXSW_PCI_RX_BUF_SW_OVERHEAD;
>>
> 
> mlxsw driver uses fragmented buffers and the page pool is used to allocate the buffers for all scatter/gather entries.
> For each packet, the HW-writable area of a buffer of the *first* entry is 'PAGE_SIZE - MLXSW_PCI_RX_BUF_SW_OVERHEAD', but for other entries we map PAGE_SIZE to HW.
> That's why we set page pool to sync PAGE_SIZE and use offset=0.

Ooops, I didn't notice this particular configuration has offset == 0, sorry.

Thanks,
Olek
diff mbox series

Patch

diff --git a/drivers/net/ethernet/mellanox/mlxsw/pci.c b/drivers/net/ethernet/mellanox/mlxsw/pci.c
index 2320a5f323b4..d6f37456fb31 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/pci.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/pci.c
@@ -996,12 +996,13 @@  static int mlxsw_pci_cq_page_pool_init(struct mlxsw_pci_queue *q,
 	if (cq_type != MLXSW_PCI_CQ_RDQ)
 		return 0;
 
-	pp_params.flags = PP_FLAG_DMA_MAP;
+	pp_params.flags = PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV;
 	pp_params.pool_size = MLXSW_PCI_WQE_COUNT * mlxsw_pci->num_sg_entries;
 	pp_params.nid = dev_to_node(&mlxsw_pci->pdev->dev);
 	pp_params.dev = &mlxsw_pci->pdev->dev;
 	pp_params.napi = &q->u.cq.napi;
 	pp_params.dma_dir = DMA_FROM_DEVICE;
+	pp_params.max_len = PAGE_SIZE;
 
 	page_pool = page_pool_create(&pp_params);
 	if (IS_ERR(page_pool))