diff mbox series

[04/15] net: hbl_cn: QP state machine

Message ID 20240613082208.1439968-5-oshpigelman@habana.ai (mailing list archive)
State Changes Requested
Delegated to: Netdev Maintainers
Headers show
Series Introduce HabanaLabs network drivers | expand

Checks

Context Check Description
netdev/series_format success Posting correctly formatted
netdev/tree_selection success Guessed tree name to be net-next, async
netdev/ynl success Generated files up to date; no warnings/errors; no diff in generated;
netdev/fixes_present success Fixes tag not required for -next series
netdev/header_inline success No static functions without inline keyword in header files
netdev/build_32bit success Errors and warnings before: 8 this patch: 8
netdev/build_tools success No tools touched, skip
netdev/cc_maintainers warning 9 maintainers not CCed: pabeni@redhat.com edumazet@google.com intel-wired-lan@lists.osuosl.org dmeriin@habana.ai kuba@kernel.org sozeri@habana.ai anthony.l.nguyen@intel.com jesse.brandeburg@intel.com bjauhari@habana.ai
netdev/build_clang success Errors and warnings before: 854 this patch: 854
netdev/verify_signedoff success Signed-off-by tag matches author and committer
netdev/deprecated_api success None detected
netdev/check_selftest success No net selftest shell script
netdev/verify_fixes success No Fixes tag
netdev/build_allmodconfig_warn success Errors and warnings before: 849 this patch: 849
netdev/checkpatch warning WARNING: line length of 81 exceeds 80 columns WARNING: line length of 82 exceeds 80 columns WARNING: line length of 83 exceeds 80 columns WARNING: line length of 87 exceeds 80 columns WARNING: line length of 89 exceeds 80 columns WARNING: line length of 91 exceeds 80 columns WARNING: line length of 92 exceeds 80 columns WARNING: line length of 93 exceeds 80 columns WARNING: line length of 94 exceeds 80 columns WARNING: line length of 95 exceeds 80 columns WARNING: line length of 97 exceeds 80 columns WARNING: line length of 98 exceeds 80 columns WARNING: line length of 99 exceeds 80 columns
netdev/build_clang_rust success No Rust files in patch. Skipping build
netdev/kdoc success Errors and warnings before: 0 this patch: 0
netdev/source_inline fail Was 0 now: 2

Commit Message

Omer Shpigelman June 13, 2024, 8:21 a.m. UTC
Add a common QP state machine which handles the moving for a QP from one
state to another including performing necessary checks, draining
in-flight transactions, invalidating caches and error reporting.

Signed-off-by: Omer Shpigelman <oshpigelman@habana.ai>
Co-developed-by: Abhilash K V <kvabhilash@habana.ai>
Signed-off-by: Abhilash K V <kvabhilash@habana.ai>
Co-developed-by: Andrey Agranovich <aagranovich@habana.ai>
Signed-off-by: Andrey Agranovich <aagranovich@habana.ai>
Co-developed-by: Bharat Jauhari <bjauhari@habana.ai>
Signed-off-by: Bharat Jauhari <bjauhari@habana.ai>
Co-developed-by: David Meriin <dmeriin@habana.ai>
Signed-off-by: David Meriin <dmeriin@habana.ai>
Co-developed-by: Sagiv Ozeri <sozeri@habana.ai>
Signed-off-by: Sagiv Ozeri <sozeri@habana.ai>
Co-developed-by: Zvika Yehudai <zyehudai@habana.ai>
Signed-off-by: Zvika Yehudai <zyehudai@habana.ai>
---
 .../ethernet/intel/hbl_cn/common/hbl_cn_qp.c  | 480 +++++++++++++++++-
 1 file changed, 479 insertions(+), 1 deletion(-)

Comments

Leon Romanovsky June 17, 2024, 1:18 p.m. UTC | #1
On Thu, Jun 13, 2024 at 11:21:57AM +0300, Omer Shpigelman wrote:
> Add a common QP state machine which handles the moving for a QP from one
> state to another including performing necessary checks, draining
> in-flight transactions, invalidating caches and error reporting.
> 
> Signed-off-by: Omer Shpigelman <oshpigelman@habana.ai>
> Co-developed-by: Abhilash K V <kvabhilash@habana.ai>
> Signed-off-by: Abhilash K V <kvabhilash@habana.ai>
> Co-developed-by: Andrey Agranovich <aagranovich@habana.ai>
> Signed-off-by: Andrey Agranovich <aagranovich@habana.ai>
> Co-developed-by: Bharat Jauhari <bjauhari@habana.ai>
> Signed-off-by: Bharat Jauhari <bjauhari@habana.ai>
> Co-developed-by: David Meriin <dmeriin@habana.ai>
> Signed-off-by: David Meriin <dmeriin@habana.ai>
> Co-developed-by: Sagiv Ozeri <sozeri@habana.ai>
> Signed-off-by: Sagiv Ozeri <sozeri@habana.ai>
> Co-developed-by: Zvika Yehudai <zyehudai@habana.ai>
> Signed-off-by: Zvika Yehudai <zyehudai@habana.ai>
> ---
>  .../ethernet/intel/hbl_cn/common/hbl_cn_qp.c  | 480 +++++++++++++++++-
>  1 file changed, 479 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c b/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c
> index 9ddc23bf8194..26ebdf448193 100644
> --- a/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c
> +++ b/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c
> @@ -6,8 +6,486 @@

<...>

> +/* The following table represents the (valid) operations that can be performed on
> + * a QP in order to move it from one state to another
> + * For example: a QP in RTR state can be moved to RTS state using the CN_QP_OP_RTR_2RTS
> + * operation.
> + */
> +static const enum hbl_cn_qp_state_op qp_valid_state_op[CN_QP_NUM_STATE][CN_QP_NUM_STATE] = {
> +	[CN_QP_STATE_RESET] = {
> +		[CN_QP_STATE_RESET]	= CN_QP_OP_2RESET,
> +		[CN_QP_STATE_INIT]	= CN_QP_OP_RST_2INIT,
> +		[CN_QP_STATE_SQD]	= CN_QP_OP_NOP,
> +		[CN_QP_STATE_QPD]	= CN_QP_OP_NOP,
> +	},
> +	[CN_QP_STATE_INIT] = {
> +		[CN_QP_STATE_RESET]	= CN_QP_OP_2RESET,
> +		[CN_QP_STATE_ERR]	= CN_QP_OP_2ERR,
> +		[CN_QP_STATE_INIT]	= CN_QP_OP_NOP,
> +		[CN_QP_STATE_RTR]	= CN_QP_OP_INIT_2RTR,
> +		[CN_QP_STATE_SQD]	= CN_QP_OP_NOP,
> +		[CN_QP_STATE_QPD]	= CN_QP_OP_NOP,
> +	},
> +	[CN_QP_STATE_RTR] = {
> +		[CN_QP_STATE_RESET]	= CN_QP_OP_2RESET,
> +		[CN_QP_STATE_ERR]	= CN_QP_OP_2ERR,
> +		[CN_QP_STATE_RTR]	= CN_QP_OP_RTR_2RTR,
> +		[CN_QP_STATE_RTS]	= CN_QP_OP_RTR_2RTS,
> +		[CN_QP_STATE_SQD]	= CN_QP_OP_NOP,
> +		[CN_QP_STATE_QPD]	= CN_QP_OP_RTR_2QPD,
> +	},
> +	[CN_QP_STATE_RTS] = {
> +		[CN_QP_STATE_RESET]	= CN_QP_OP_2RESET,
> +		[CN_QP_STATE_ERR]	= CN_QP_OP_2ERR,
> +		[CN_QP_STATE_RTS]	= CN_QP_OP_RTS_2RTS,
> +		[CN_QP_STATE_SQD]	= CN_QP_OP_RTS_2SQD,
> +		[CN_QP_STATE_QPD]	= CN_QP_OP_RTS_2QPD,
> +		[CN_QP_STATE_SQERR]	= CN_QP_OP_RTS_2SQERR,
> +	},
> +	[CN_QP_STATE_SQD] = {
> +		[CN_QP_STATE_RESET]	= CN_QP_OP_2RESET,
> +		[CN_QP_STATE_ERR]	= CN_QP_OP_2ERR,
> +		[CN_QP_STATE_SQD]	= CN_QP_OP_SQD_2SQD,
> +		[CN_QP_STATE_RTS]	= CN_QP_OP_SQD_2RTS,
> +		[CN_QP_STATE_QPD]	= CN_QP_OP_SQD_2QPD,
> +		[CN_QP_STATE_SQERR]	= CN_QP_OP_SQD_2SQ_ERR,
> +	},
> +	[CN_QP_STATE_QPD] = {
> +		[CN_QP_STATE_RESET]	= CN_QP_OP_2RESET,
> +		[CN_QP_STATE_ERR]	= CN_QP_OP_2ERR,
> +		[CN_QP_STATE_SQD]	= CN_QP_OP_NOP,
> +		[CN_QP_STATE_QPD]	= CN_QP_OP_NOP,
> +		[CN_QP_STATE_RTR]	= CN_QP_OP_QPD_2RTR,
> +	},
> +	[CN_QP_STATE_SQERR] = {
> +		[CN_QP_STATE_RESET]	= CN_QP_OP_2RESET,
> +		[CN_QP_STATE_ERR]	= CN_QP_OP_2ERR,
> +		[CN_QP_STATE_SQD]	= CN_QP_OP_SQ_ERR_2SQD,
> +		[CN_QP_STATE_SQERR]	= CN_QP_OP_NOP,
> +	},
> +	[CN_QP_STATE_ERR] = {
> +		[CN_QP_STATE_RESET]	= CN_QP_OP_2RESET,
> +		[CN_QP_STATE_ERR]	= CN_QP_OP_2ERR,
> +	}
> +};

I don't understand why IBTA QP state machine is declared in ETH driver
and not in IB driver.

> +

<...>

> +		/* Release lock while we wait before retry.
> +		 * Note, we can assert that we are already locked.
> +		 */
> +		port_funcs->cfg_unlock(cn_port);
> +
> +		msleep(20);
> +
> +		port_funcs->cfg_lock(cn_port);

lock/unlock through ops pointer doesn't look like a good idea.

Thanks
Omer Shpigelman June 18, 2024, 5:50 a.m. UTC | #2
On 6/17/24 16:18, Leon Romanovsky wrote:
> [Some people who received this message don't often get email from leon@kernel.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> 
> On Thu, Jun 13, 2024 at 11:21:57AM +0300, Omer Shpigelman wrote:
>> Add a common QP state machine which handles the moving for a QP from one
>> state to another including performing necessary checks, draining
>> in-flight transactions, invalidating caches and error reporting.
>>
>> Signed-off-by: Omer Shpigelman <oshpigelman@habana.ai>
>> Co-developed-by: Abhilash K V <kvabhilash@habana.ai>
>> Signed-off-by: Abhilash K V <kvabhilash@habana.ai>
>> Co-developed-by: Andrey Agranovich <aagranovich@habana.ai>
>> Signed-off-by: Andrey Agranovich <aagranovich@habana.ai>
>> Co-developed-by: Bharat Jauhari <bjauhari@habana.ai>
>> Signed-off-by: Bharat Jauhari <bjauhari@habana.ai>
>> Co-developed-by: David Meriin <dmeriin@habana.ai>
>> Signed-off-by: David Meriin <dmeriin@habana.ai>
>> Co-developed-by: Sagiv Ozeri <sozeri@habana.ai>
>> Signed-off-by: Sagiv Ozeri <sozeri@habana.ai>
>> Co-developed-by: Zvika Yehudai <zyehudai@habana.ai>
>> Signed-off-by: Zvika Yehudai <zyehudai@habana.ai>
>> ---
>>  .../ethernet/intel/hbl_cn/common/hbl_cn_qp.c  | 480 +++++++++++++++++-
>>  1 file changed, 479 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c b/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c
>> index 9ddc23bf8194..26ebdf448193 100644
>> --- a/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c
>> +++ b/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c
>> @@ -6,8 +6,486 @@
> 
> <...>
> 
>> +/* The following table represents the (valid) operations that can be performed on
>> + * a QP in order to move it from one state to another
>> + * For example: a QP in RTR state can be moved to RTS state using the CN_QP_OP_RTR_2RTS
>> + * operation.
>> + */
>> +static const enum hbl_cn_qp_state_op qp_valid_state_op[CN_QP_NUM_STATE][CN_QP_NUM_STATE] = {
>> +     [CN_QP_STATE_RESET] = {
>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>> +             [CN_QP_STATE_INIT]      = CN_QP_OP_RST_2INIT,
>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_NOP,
>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_NOP,
>> +     },
>> +     [CN_QP_STATE_INIT] = {
>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
>> +             [CN_QP_STATE_INIT]      = CN_QP_OP_NOP,
>> +             [CN_QP_STATE_RTR]       = CN_QP_OP_INIT_2RTR,
>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_NOP,
>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_NOP,
>> +     },
>> +     [CN_QP_STATE_RTR] = {
>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
>> +             [CN_QP_STATE_RTR]       = CN_QP_OP_RTR_2RTR,
>> +             [CN_QP_STATE_RTS]       = CN_QP_OP_RTR_2RTS,
>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_NOP,
>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_RTR_2QPD,
>> +     },
>> +     [CN_QP_STATE_RTS] = {
>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
>> +             [CN_QP_STATE_RTS]       = CN_QP_OP_RTS_2RTS,
>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_RTS_2SQD,
>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_RTS_2QPD,
>> +             [CN_QP_STATE_SQERR]     = CN_QP_OP_RTS_2SQERR,
>> +     },
>> +     [CN_QP_STATE_SQD] = {
>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_SQD_2SQD,
>> +             [CN_QP_STATE_RTS]       = CN_QP_OP_SQD_2RTS,
>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_SQD_2QPD,
>> +             [CN_QP_STATE_SQERR]     = CN_QP_OP_SQD_2SQ_ERR,
>> +     },
>> +     [CN_QP_STATE_QPD] = {
>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_NOP,
>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_NOP,
>> +             [CN_QP_STATE_RTR]       = CN_QP_OP_QPD_2RTR,
>> +     },
>> +     [CN_QP_STATE_SQERR] = {
>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_SQ_ERR_2SQD,
>> +             [CN_QP_STATE_SQERR]     = CN_QP_OP_NOP,
>> +     },
>> +     [CN_QP_STATE_ERR] = {
>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
>> +     }
>> +};
> 
> I don't understand why IBTA QP state machine is declared in ETH driver
> and not in IB driver.
> 

Implementing the actual transitions between the states requires full
knowledge of the HW e.g. when to flush, cache invalidation, timeouts.
Our IB driver is agnostic to the ASIC type by design. Note that more ASIC
generations are planned to be added and the IB driver should not be aware
of these additional HWs.
Hence we implemeted the QP state machine in the CN driver which is aware
of the actual HW.

>> +
> 
> <...>
> 
>> +             /* Release lock while we wait before retry.
>> +              * Note, we can assert that we are already locked.
>> +              */
>> +             port_funcs->cfg_unlock(cn_port);
>> +
>> +             msleep(20);
>> +
>> +             port_funcs->cfg_lock(cn_port);
> 
> lock/unlock through ops pointer doesn't look like a good idea.
> 

More ASIC generations will be added once we merge the current Gaudi2 code.
On other ASICs the locking granularity is different because some of the HW
resources are shared between different logical ports.
Hence it is was logical for us to implement it with a function pointer so
each ASIC specific code can implemnet the locking properly.

> Thanks
Leon Romanovsky June 18, 2024, 7:08 a.m. UTC | #3
On Tue, Jun 18, 2024 at 05:50:15AM +0000, Omer Shpigelman wrote:
> On 6/17/24 16:18, Leon Romanovsky wrote:
> > [Some people who received this message don't often get email from leon@kernel.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> > 
> > On Thu, Jun 13, 2024 at 11:21:57AM +0300, Omer Shpigelman wrote:
> >> Add a common QP state machine which handles the moving for a QP from one
> >> state to another including performing necessary checks, draining
> >> in-flight transactions, invalidating caches and error reporting.
> >>
> >> Signed-off-by: Omer Shpigelman <oshpigelman@habana.ai>
> >> Co-developed-by: Abhilash K V <kvabhilash@habana.ai>
> >> Signed-off-by: Abhilash K V <kvabhilash@habana.ai>
> >> Co-developed-by: Andrey Agranovich <aagranovich@habana.ai>
> >> Signed-off-by: Andrey Agranovich <aagranovich@habana.ai>
> >> Co-developed-by: Bharat Jauhari <bjauhari@habana.ai>
> >> Signed-off-by: Bharat Jauhari <bjauhari@habana.ai>
> >> Co-developed-by: David Meriin <dmeriin@habana.ai>
> >> Signed-off-by: David Meriin <dmeriin@habana.ai>
> >> Co-developed-by: Sagiv Ozeri <sozeri@habana.ai>
> >> Signed-off-by: Sagiv Ozeri <sozeri@habana.ai>
> >> Co-developed-by: Zvika Yehudai <zyehudai@habana.ai>
> >> Signed-off-by: Zvika Yehudai <zyehudai@habana.ai>
> >> ---
> >>  .../ethernet/intel/hbl_cn/common/hbl_cn_qp.c  | 480 +++++++++++++++++-
> >>  1 file changed, 479 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c b/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c
> >> index 9ddc23bf8194..26ebdf448193 100644
> >> --- a/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c
> >> +++ b/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c
> >> @@ -6,8 +6,486 @@
> > 
> > <...>
> > 
> >> +/* The following table represents the (valid) operations that can be performed on
> >> + * a QP in order to move it from one state to another
> >> + * For example: a QP in RTR state can be moved to RTS state using the CN_QP_OP_RTR_2RTS
> >> + * operation.
> >> + */
> >> +static const enum hbl_cn_qp_state_op qp_valid_state_op[CN_QP_NUM_STATE][CN_QP_NUM_STATE] = {
> >> +     [CN_QP_STATE_RESET] = {
> >> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
> >> +             [CN_QP_STATE_INIT]      = CN_QP_OP_RST_2INIT,
> >> +             [CN_QP_STATE_SQD]       = CN_QP_OP_NOP,
> >> +             [CN_QP_STATE_QPD]       = CN_QP_OP_NOP,
> >> +     },
> >> +     [CN_QP_STATE_INIT] = {
> >> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
> >> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
> >> +             [CN_QP_STATE_INIT]      = CN_QP_OP_NOP,
> >> +             [CN_QP_STATE_RTR]       = CN_QP_OP_INIT_2RTR,
> >> +             [CN_QP_STATE_SQD]       = CN_QP_OP_NOP,
> >> +             [CN_QP_STATE_QPD]       = CN_QP_OP_NOP,
> >> +     },
> >> +     [CN_QP_STATE_RTR] = {
> >> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
> >> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
> >> +             [CN_QP_STATE_RTR]       = CN_QP_OP_RTR_2RTR,
> >> +             [CN_QP_STATE_RTS]       = CN_QP_OP_RTR_2RTS,
> >> +             [CN_QP_STATE_SQD]       = CN_QP_OP_NOP,
> >> +             [CN_QP_STATE_QPD]       = CN_QP_OP_RTR_2QPD,
> >> +     },
> >> +     [CN_QP_STATE_RTS] = {
> >> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
> >> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
> >> +             [CN_QP_STATE_RTS]       = CN_QP_OP_RTS_2RTS,
> >> +             [CN_QP_STATE_SQD]       = CN_QP_OP_RTS_2SQD,
> >> +             [CN_QP_STATE_QPD]       = CN_QP_OP_RTS_2QPD,
> >> +             [CN_QP_STATE_SQERR]     = CN_QP_OP_RTS_2SQERR,
> >> +     },
> >> +     [CN_QP_STATE_SQD] = {
> >> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
> >> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
> >> +             [CN_QP_STATE_SQD]       = CN_QP_OP_SQD_2SQD,
> >> +             [CN_QP_STATE_RTS]       = CN_QP_OP_SQD_2RTS,
> >> +             [CN_QP_STATE_QPD]       = CN_QP_OP_SQD_2QPD,
> >> +             [CN_QP_STATE_SQERR]     = CN_QP_OP_SQD_2SQ_ERR,
> >> +     },
> >> +     [CN_QP_STATE_QPD] = {
> >> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
> >> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
> >> +             [CN_QP_STATE_SQD]       = CN_QP_OP_NOP,
> >> +             [CN_QP_STATE_QPD]       = CN_QP_OP_NOP,
> >> +             [CN_QP_STATE_RTR]       = CN_QP_OP_QPD_2RTR,
> >> +     },
> >> +     [CN_QP_STATE_SQERR] = {
> >> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
> >> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
> >> +             [CN_QP_STATE_SQD]       = CN_QP_OP_SQ_ERR_2SQD,
> >> +             [CN_QP_STATE_SQERR]     = CN_QP_OP_NOP,
> >> +     },
> >> +     [CN_QP_STATE_ERR] = {
> >> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
> >> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
> >> +     }
> >> +};
> > 
> > I don't understand why IBTA QP state machine is declared in ETH driver
> > and not in IB driver.
> > 
> 
> Implementing the actual transitions between the states requires full
> knowledge of the HW e.g. when to flush, cache invalidation, timeouts.
> Our IB driver is agnostic to the ASIC type by design. Note that more ASIC
> generations are planned to be added and the IB driver should not be aware
> of these additional HWs.
> Hence we implemeted the QP state machine in the CN driver which is aware
> of the actual HW.

Somehow ALL other IB drivers are able to implement this logic in the IB,
while supporting multiple ASICs. I don't see a reason why you can't do
the same.

> 
> >> +
> > 
> > <...>
> > 
> >> +             /* Release lock while we wait before retry.
> >> +              * Note, we can assert that we are already locked.
> >> +              */
> >> +             port_funcs->cfg_unlock(cn_port);
> >> +
> >> +             msleep(20);
> >> +
> >> +             port_funcs->cfg_lock(cn_port);
> > 
> > lock/unlock through ops pointer doesn't look like a good idea.
> > 
> 
> More ASIC generations will be added once we merge the current Gaudi2 code.
> On other ASICs the locking granularity is different because some of the HW
> resources are shared between different logical ports.
> Hence it is was logical for us to implement it with a function pointer so
> each ASIC specific code can implemnet the locking properly.

We are reviewing this code which is for the current ASIC, not for the
unknown future ASICs. Please don't over engineer the first submission.
You will always be able to improve/change the code once you decide to
upstream your future ASICs.

Thanks
Omer Shpigelman June 18, 2024, 7:58 a.m. UTC | #4
On 6/18/24 10:08, Leon Romanovsky wrote:
> On Tue, Jun 18, 2024 at 05:50:15AM +0000, Omer Shpigelman wrote:
>> On 6/17/24 16:18, Leon Romanovsky wrote:
>>> [Some people who received this message don't often get email from leon@kernel.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>>>
>>> On Thu, Jun 13, 2024 at 11:21:57AM +0300, Omer Shpigelman wrote:
>>>> Add a common QP state machine which handles the moving for a QP from one
>>>> state to another including performing necessary checks, draining
>>>> in-flight transactions, invalidating caches and error reporting.
>>>>
>>>> Signed-off-by: Omer Shpigelman <oshpigelman@habana.ai>
>>>> Co-developed-by: Abhilash K V <kvabhilash@habana.ai>
>>>> Signed-off-by: Abhilash K V <kvabhilash@habana.ai>
>>>> Co-developed-by: Andrey Agranovich <aagranovich@habana.ai>
>>>> Signed-off-by: Andrey Agranovich <aagranovich@habana.ai>
>>>> Co-developed-by: Bharat Jauhari <bjauhari@habana.ai>
>>>> Signed-off-by: Bharat Jauhari <bjauhari@habana.ai>
>>>> Co-developed-by: David Meriin <dmeriin@habana.ai>
>>>> Signed-off-by: David Meriin <dmeriin@habana.ai>
>>>> Co-developed-by: Sagiv Ozeri <sozeri@habana.ai>
>>>> Signed-off-by: Sagiv Ozeri <sozeri@habana.ai>
>>>> Co-developed-by: Zvika Yehudai <zyehudai@habana.ai>
>>>> Signed-off-by: Zvika Yehudai <zyehudai@habana.ai>
>>>> ---
>>>>  .../ethernet/intel/hbl_cn/common/hbl_cn_qp.c  | 480 +++++++++++++++++-
>>>>  1 file changed, 479 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c b/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c
>>>> index 9ddc23bf8194..26ebdf448193 100644
>>>> --- a/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c
>>>> +++ b/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c
>>>> @@ -6,8 +6,486 @@
>>>
>>> <...>
>>>
>>>> +/* The following table represents the (valid) operations that can be performed on
>>>> + * a QP in order to move it from one state to another
>>>> + * For example: a QP in RTR state can be moved to RTS state using the CN_QP_OP_RTR_2RTS
>>>> + * operation.
>>>> + */
>>>> +static const enum hbl_cn_qp_state_op qp_valid_state_op[CN_QP_NUM_STATE][CN_QP_NUM_STATE] = {
>>>> +     [CN_QP_STATE_RESET] = {
>>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>>>> +             [CN_QP_STATE_INIT]      = CN_QP_OP_RST_2INIT,
>>>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_NOP,
>>>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_NOP,
>>>> +     },
>>>> +     [CN_QP_STATE_INIT] = {
>>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>>>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
>>>> +             [CN_QP_STATE_INIT]      = CN_QP_OP_NOP,
>>>> +             [CN_QP_STATE_RTR]       = CN_QP_OP_INIT_2RTR,
>>>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_NOP,
>>>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_NOP,
>>>> +     },
>>>> +     [CN_QP_STATE_RTR] = {
>>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>>>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
>>>> +             [CN_QP_STATE_RTR]       = CN_QP_OP_RTR_2RTR,
>>>> +             [CN_QP_STATE_RTS]       = CN_QP_OP_RTR_2RTS,
>>>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_NOP,
>>>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_RTR_2QPD,
>>>> +     },
>>>> +     [CN_QP_STATE_RTS] = {
>>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>>>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
>>>> +             [CN_QP_STATE_RTS]       = CN_QP_OP_RTS_2RTS,
>>>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_RTS_2SQD,
>>>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_RTS_2QPD,
>>>> +             [CN_QP_STATE_SQERR]     = CN_QP_OP_RTS_2SQERR,
>>>> +     },
>>>> +     [CN_QP_STATE_SQD] = {
>>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>>>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
>>>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_SQD_2SQD,
>>>> +             [CN_QP_STATE_RTS]       = CN_QP_OP_SQD_2RTS,
>>>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_SQD_2QPD,
>>>> +             [CN_QP_STATE_SQERR]     = CN_QP_OP_SQD_2SQ_ERR,
>>>> +     },
>>>> +     [CN_QP_STATE_QPD] = {
>>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>>>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
>>>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_NOP,
>>>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_NOP,
>>>> +             [CN_QP_STATE_RTR]       = CN_QP_OP_QPD_2RTR,
>>>> +     },
>>>> +     [CN_QP_STATE_SQERR] = {
>>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>>>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
>>>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_SQ_ERR_2SQD,
>>>> +             [CN_QP_STATE_SQERR]     = CN_QP_OP_NOP,
>>>> +     },
>>>> +     [CN_QP_STATE_ERR] = {
>>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>>>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
>>>> +     }
>>>> +};
>>>
>>> I don't understand why IBTA QP state machine is declared in ETH driver
>>> and not in IB driver.
>>>
>>
>> Implementing the actual transitions between the states requires full
>> knowledge of the HW e.g. when to flush, cache invalidation, timeouts.
>> Our IB driver is agnostic to the ASIC type by design. Note that more ASIC
>> generations are planned to be added and the IB driver should not be aware
>> of these additional HWs.
>> Hence we implemeted the QP state machine in the CN driver which is aware
>> of the actual HW.
> 
> Somehow ALL other IB drivers are able to implement this logic in the IB,
> while supporting multiple ASICs. I don't see a reason why you can't do
> the same.
> 

If we are referring to this actual table, then I can move it to the IB
driver and the CN driver will fetch the needed opcode via a function
pointer.
Is that ok?

>>
>>>> +
>>>
>>> <...>
>>>
>>>> +             /* Release lock while we wait before retry.
>>>> +              * Note, we can assert that we are already locked.
>>>> +              */
>>>> +             port_funcs->cfg_unlock(cn_port);
>>>> +
>>>> +             msleep(20);
>>>> +
>>>> +             port_funcs->cfg_lock(cn_port);
>>>
>>> lock/unlock through ops pointer doesn't look like a good idea.
>>>
>>
>> More ASIC generations will be added once we merge the current Gaudi2 code.
>> On other ASICs the locking granularity is different because some of the HW
>> resources are shared between different logical ports.
>> Hence it is was logical for us to implement it with a function pointer so
>> each ASIC specific code can implemnet the locking properly.
> 
> We are reviewing this code which is for the current ASIC, not for the
> unknown future ASICs. Please don't over engineer the first submission.
> You will always be able to improve/change the code once you decide to
> upstream your future ASICs.
> 

I see. I'll refactor the code then.

> Thanks
Leon Romanovsky June 18, 2024, 9 a.m. UTC | #5
On Tue, Jun 18, 2024 at 07:58:55AM +0000, Omer Shpigelman wrote:
> On 6/18/24 10:08, Leon Romanovsky wrote:
> > On Tue, Jun 18, 2024 at 05:50:15AM +0000, Omer Shpigelman wrote:
> >> On 6/17/24 16:18, Leon Romanovsky wrote:
> >>> [Some people who received this message don't often get email from leon@kernel.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> >>>
> >>> On Thu, Jun 13, 2024 at 11:21:57AM +0300, Omer Shpigelman wrote:
> >>>> Add a common QP state machine which handles the moving for a QP from one
> >>>> state to another including performing necessary checks, draining
> >>>> in-flight transactions, invalidating caches and error reporting.
> >>>>
> >>>> Signed-off-by: Omer Shpigelman <oshpigelman@habana.ai>
> >>>> Co-developed-by: Abhilash K V <kvabhilash@habana.ai>
> >>>> Signed-off-by: Abhilash K V <kvabhilash@habana.ai>
> >>>> Co-developed-by: Andrey Agranovich <aagranovich@habana.ai>
> >>>> Signed-off-by: Andrey Agranovich <aagranovich@habana.ai>
> >>>> Co-developed-by: Bharat Jauhari <bjauhari@habana.ai>
> >>>> Signed-off-by: Bharat Jauhari <bjauhari@habana.ai>
> >>>> Co-developed-by: David Meriin <dmeriin@habana.ai>
> >>>> Signed-off-by: David Meriin <dmeriin@habana.ai>
> >>>> Co-developed-by: Sagiv Ozeri <sozeri@habana.ai>
> >>>> Signed-off-by: Sagiv Ozeri <sozeri@habana.ai>
> >>>> Co-developed-by: Zvika Yehudai <zyehudai@habana.ai>
> >>>> Signed-off-by: Zvika Yehudai <zyehudai@habana.ai>
> >>>> ---
> >>>>  .../ethernet/intel/hbl_cn/common/hbl_cn_qp.c  | 480 +++++++++++++++++-
> >>>>  1 file changed, 479 insertions(+), 1 deletion(-)
> >>>>
> >>>> diff --git a/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c b/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c
> >>>> index 9ddc23bf8194..26ebdf448193 100644
> >>>> --- a/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c
> >>>> +++ b/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c
> >>>> @@ -6,8 +6,486 @@
> >>>
> >>> <...>
> >>>
> >>>> +/* The following table represents the (valid) operations that can be performed on
> >>>> + * a QP in order to move it from one state to another
> >>>> + * For example: a QP in RTR state can be moved to RTS state using the CN_QP_OP_RTR_2RTS
> >>>> + * operation.
> >>>> + */
> >>>> +static const enum hbl_cn_qp_state_op qp_valid_state_op[CN_QP_NUM_STATE][CN_QP_NUM_STATE] = {
> >>>> +     [CN_QP_STATE_RESET] = {
> >>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
> >>>> +             [CN_QP_STATE_INIT]      = CN_QP_OP_RST_2INIT,
> >>>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_NOP,
> >>>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_NOP,
> >>>> +     },
> >>>> +     [CN_QP_STATE_INIT] = {
> >>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
> >>>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
> >>>> +             [CN_QP_STATE_INIT]      = CN_QP_OP_NOP,
> >>>> +             [CN_QP_STATE_RTR]       = CN_QP_OP_INIT_2RTR,
> >>>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_NOP,
> >>>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_NOP,
> >>>> +     },
> >>>> +     [CN_QP_STATE_RTR] = {
> >>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
> >>>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
> >>>> +             [CN_QP_STATE_RTR]       = CN_QP_OP_RTR_2RTR,
> >>>> +             [CN_QP_STATE_RTS]       = CN_QP_OP_RTR_2RTS,
> >>>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_NOP,
> >>>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_RTR_2QPD,
> >>>> +     },
> >>>> +     [CN_QP_STATE_RTS] = {
> >>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
> >>>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
> >>>> +             [CN_QP_STATE_RTS]       = CN_QP_OP_RTS_2RTS,
> >>>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_RTS_2SQD,
> >>>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_RTS_2QPD,
> >>>> +             [CN_QP_STATE_SQERR]     = CN_QP_OP_RTS_2SQERR,
> >>>> +     },
> >>>> +     [CN_QP_STATE_SQD] = {
> >>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
> >>>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
> >>>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_SQD_2SQD,
> >>>> +             [CN_QP_STATE_RTS]       = CN_QP_OP_SQD_2RTS,
> >>>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_SQD_2QPD,
> >>>> +             [CN_QP_STATE_SQERR]     = CN_QP_OP_SQD_2SQ_ERR,
> >>>> +     },
> >>>> +     [CN_QP_STATE_QPD] = {
> >>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
> >>>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
> >>>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_NOP,
> >>>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_NOP,
> >>>> +             [CN_QP_STATE_RTR]       = CN_QP_OP_QPD_2RTR,
> >>>> +     },
> >>>> +     [CN_QP_STATE_SQERR] = {
> >>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
> >>>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
> >>>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_SQ_ERR_2SQD,
> >>>> +             [CN_QP_STATE_SQERR]     = CN_QP_OP_NOP,
> >>>> +     },
> >>>> +     [CN_QP_STATE_ERR] = {
> >>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
> >>>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
> >>>> +     }
> >>>> +};
> >>>
> >>> I don't understand why IBTA QP state machine is declared in ETH driver
> >>> and not in IB driver.
> >>>
> >>
> >> Implementing the actual transitions between the states requires full
> >> knowledge of the HW e.g. when to flush, cache invalidation, timeouts.
> >> Our IB driver is agnostic to the ASIC type by design. Note that more ASIC
> >> generations are planned to be added and the IB driver should not be aware
> >> of these additional HWs.
> >> Hence we implemeted the QP state machine in the CN driver which is aware
> >> of the actual HW.
> > 
> > Somehow ALL other IB drivers are able to implement this logic in the IB,
> > while supporting multiple ASICs. I don't see a reason why you can't do
> > the same.
> > 
> 
> If we are referring to this actual table, then I can move it to the IB
> driver and the CN driver will fetch the needed opcode via a function
> pointer.
> Is that ok?

This table spotted my attention, but right separation shouldn't be limited
to only this table. The outcome of this conversation should be:
"IB specific logic should be in IB driver, and CN driver should be able to
handle only low-level operations".

Thanks
Omer Shpigelman June 24, 2024, 7:24 a.m. UTC | #6
On 6/18/24 12:00, Leon Romanovsky wrote:
> On Tue, Jun 18, 2024 at 07:58:55AM +0000, Omer Shpigelman wrote:
>> On 6/18/24 10:08, Leon Romanovsky wrote:
>>> On Tue, Jun 18, 2024 at 05:50:15AM +0000, Omer Shpigelman wrote:
>>>> On 6/17/24 16:18, Leon Romanovsky wrote:
>>>>> [Some people who received this message don't often get email from leon@kernel.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>>>>>
>>>>> On Thu, Jun 13, 2024 at 11:21:57AM +0300, Omer Shpigelman wrote:
>>>>>> Add a common QP state machine which handles the moving for a QP from one
>>>>>> state to another including performing necessary checks, draining
>>>>>> in-flight transactions, invalidating caches and error reporting.
>>>>>>
>>>>>> Signed-off-by: Omer Shpigelman <oshpigelman@habana.ai>
>>>>>> Co-developed-by: Abhilash K V <kvabhilash@habana.ai>
>>>>>> Signed-off-by: Abhilash K V <kvabhilash@habana.ai>
>>>>>> Co-developed-by: Andrey Agranovich <aagranovich@habana.ai>
>>>>>> Signed-off-by: Andrey Agranovich <aagranovich@habana.ai>
>>>>>> Co-developed-by: Bharat Jauhari <bjauhari@habana.ai>
>>>>>> Signed-off-by: Bharat Jauhari <bjauhari@habana.ai>
>>>>>> Co-developed-by: David Meriin <dmeriin@habana.ai>
>>>>>> Signed-off-by: David Meriin <dmeriin@habana.ai>
>>>>>> Co-developed-by: Sagiv Ozeri <sozeri@habana.ai>
>>>>>> Signed-off-by: Sagiv Ozeri <sozeri@habana.ai>
>>>>>> Co-developed-by: Zvika Yehudai <zyehudai@habana.ai>
>>>>>> Signed-off-by: Zvika Yehudai <zyehudai@habana.ai>
>>>>>> ---
>>>>>>  .../ethernet/intel/hbl_cn/common/hbl_cn_qp.c  | 480 +++++++++++++++++-
>>>>>>  1 file changed, 479 insertions(+), 1 deletion(-)
>>>>>>
>>>>>> diff --git a/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c b/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c
>>>>>> index 9ddc23bf8194..26ebdf448193 100644
>>>>>> --- a/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c
>>>>>> +++ b/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c
>>>>>> @@ -6,8 +6,486 @@
>>>>>
>>>>> <...>
>>>>>
>>>>>> +/* The following table represents the (valid) operations that can be performed on
>>>>>> + * a QP in order to move it from one state to another
>>>>>> + * For example: a QP in RTR state can be moved to RTS state using the CN_QP_OP_RTR_2RTS
>>>>>> + * operation.
>>>>>> + */
>>>>>> +static const enum hbl_cn_qp_state_op qp_valid_state_op[CN_QP_NUM_STATE][CN_QP_NUM_STATE] = {
>>>>>> +     [CN_QP_STATE_RESET] = {
>>>>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>>>>>> +             [CN_QP_STATE_INIT]      = CN_QP_OP_RST_2INIT,
>>>>>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_NOP,
>>>>>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_NOP,
>>>>>> +     },
>>>>>> +     [CN_QP_STATE_INIT] = {
>>>>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>>>>>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
>>>>>> +             [CN_QP_STATE_INIT]      = CN_QP_OP_NOP,
>>>>>> +             [CN_QP_STATE_RTR]       = CN_QP_OP_INIT_2RTR,
>>>>>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_NOP,
>>>>>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_NOP,
>>>>>> +     },
>>>>>> +     [CN_QP_STATE_RTR] = {
>>>>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>>>>>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
>>>>>> +             [CN_QP_STATE_RTR]       = CN_QP_OP_RTR_2RTR,
>>>>>> +             [CN_QP_STATE_RTS]       = CN_QP_OP_RTR_2RTS,
>>>>>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_NOP,
>>>>>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_RTR_2QPD,
>>>>>> +     },
>>>>>> +     [CN_QP_STATE_RTS] = {
>>>>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>>>>>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
>>>>>> +             [CN_QP_STATE_RTS]       = CN_QP_OP_RTS_2RTS,
>>>>>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_RTS_2SQD,
>>>>>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_RTS_2QPD,
>>>>>> +             [CN_QP_STATE_SQERR]     = CN_QP_OP_RTS_2SQERR,
>>>>>> +     },
>>>>>> +     [CN_QP_STATE_SQD] = {
>>>>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>>>>>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
>>>>>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_SQD_2SQD,
>>>>>> +             [CN_QP_STATE_RTS]       = CN_QP_OP_SQD_2RTS,
>>>>>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_SQD_2QPD,
>>>>>> +             [CN_QP_STATE_SQERR]     = CN_QP_OP_SQD_2SQ_ERR,
>>>>>> +     },
>>>>>> +     [CN_QP_STATE_QPD] = {
>>>>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>>>>>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
>>>>>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_NOP,
>>>>>> +             [CN_QP_STATE_QPD]       = CN_QP_OP_NOP,
>>>>>> +             [CN_QP_STATE_RTR]       = CN_QP_OP_QPD_2RTR,
>>>>>> +     },
>>>>>> +     [CN_QP_STATE_SQERR] = {
>>>>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>>>>>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
>>>>>> +             [CN_QP_STATE_SQD]       = CN_QP_OP_SQ_ERR_2SQD,
>>>>>> +             [CN_QP_STATE_SQERR]     = CN_QP_OP_NOP,
>>>>>> +     },
>>>>>> +     [CN_QP_STATE_ERR] = {
>>>>>> +             [CN_QP_STATE_RESET]     = CN_QP_OP_2RESET,
>>>>>> +             [CN_QP_STATE_ERR]       = CN_QP_OP_2ERR,
>>>>>> +     }
>>>>>> +};
>>>>>
>>>>> I don't understand why IBTA QP state machine is declared in ETH driver
>>>>> and not in IB driver.
>>>>>
>>>>
>>>> Implementing the actual transitions between the states requires full
>>>> knowledge of the HW e.g. when to flush, cache invalidation, timeouts.
>>>> Our IB driver is agnostic to the ASIC type by design. Note that more ASIC
>>>> generations are planned to be added and the IB driver should not be aware
>>>> of these additional HWs.
>>>> Hence we implemeted the QP state machine in the CN driver which is aware
>>>> of the actual HW.
>>>
>>> Somehow ALL other IB drivers are able to implement this logic in the IB,
>>> while supporting multiple ASICs. I don't see a reason why you can't do
>>> the same.
>>>
>>
>> If we are referring to this actual table, then I can move it to the IB
>> driver and the CN driver will fetch the needed opcode via a function
>> pointer.
>> Is that ok?
> 
> This table spotted my attention, but right separation shouldn't be limited
> to only this table. The outcome of this conversation should be:
> "IB specific logic should be in IB driver, and CN driver should be able to
> handle only low-level operations".
> 
> Thanks

Ok, I'll check how we can move the IB specific logic to the IB driver.
diff mbox series

Patch

diff --git a/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c b/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c
index 9ddc23bf8194..26ebdf448193 100644
--- a/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c
+++ b/drivers/net/ethernet/intel/hbl_cn/common/hbl_cn_qp.c
@@ -6,8 +6,486 @@ 
 
 #include "hbl_cn.h"
 
+#define OP_RETRY_COUNT		4
+#define OPC_SETTLE_RETRY_COUNT	20
+
+/* The following table represents the (valid) operations that can be performed on
+ * a QP in order to move it from one state to another
+ * For example: a QP in RTR state can be moved to RTS state using the CN_QP_OP_RTR_2RTS
+ * operation.
+ */
+static const enum hbl_cn_qp_state_op qp_valid_state_op[CN_QP_NUM_STATE][CN_QP_NUM_STATE] = {
+	[CN_QP_STATE_RESET] = {
+		[CN_QP_STATE_RESET]	= CN_QP_OP_2RESET,
+		[CN_QP_STATE_INIT]	= CN_QP_OP_RST_2INIT,
+		[CN_QP_STATE_SQD]	= CN_QP_OP_NOP,
+		[CN_QP_STATE_QPD]	= CN_QP_OP_NOP,
+	},
+	[CN_QP_STATE_INIT] = {
+		[CN_QP_STATE_RESET]	= CN_QP_OP_2RESET,
+		[CN_QP_STATE_ERR]	= CN_QP_OP_2ERR,
+		[CN_QP_STATE_INIT]	= CN_QP_OP_NOP,
+		[CN_QP_STATE_RTR]	= CN_QP_OP_INIT_2RTR,
+		[CN_QP_STATE_SQD]	= CN_QP_OP_NOP,
+		[CN_QP_STATE_QPD]	= CN_QP_OP_NOP,
+	},
+	[CN_QP_STATE_RTR] = {
+		[CN_QP_STATE_RESET]	= CN_QP_OP_2RESET,
+		[CN_QP_STATE_ERR]	= CN_QP_OP_2ERR,
+		[CN_QP_STATE_RTR]	= CN_QP_OP_RTR_2RTR,
+		[CN_QP_STATE_RTS]	= CN_QP_OP_RTR_2RTS,
+		[CN_QP_STATE_SQD]	= CN_QP_OP_NOP,
+		[CN_QP_STATE_QPD]	= CN_QP_OP_RTR_2QPD,
+	},
+	[CN_QP_STATE_RTS] = {
+		[CN_QP_STATE_RESET]	= CN_QP_OP_2RESET,
+		[CN_QP_STATE_ERR]	= CN_QP_OP_2ERR,
+		[CN_QP_STATE_RTS]	= CN_QP_OP_RTS_2RTS,
+		[CN_QP_STATE_SQD]	= CN_QP_OP_RTS_2SQD,
+		[CN_QP_STATE_QPD]	= CN_QP_OP_RTS_2QPD,
+		[CN_QP_STATE_SQERR]	= CN_QP_OP_RTS_2SQERR,
+	},
+	[CN_QP_STATE_SQD] = {
+		[CN_QP_STATE_RESET]	= CN_QP_OP_2RESET,
+		[CN_QP_STATE_ERR]	= CN_QP_OP_2ERR,
+		[CN_QP_STATE_SQD]	= CN_QP_OP_SQD_2SQD,
+		[CN_QP_STATE_RTS]	= CN_QP_OP_SQD_2RTS,
+		[CN_QP_STATE_QPD]	= CN_QP_OP_SQD_2QPD,
+		[CN_QP_STATE_SQERR]	= CN_QP_OP_SQD_2SQ_ERR,
+	},
+	[CN_QP_STATE_QPD] = {
+		[CN_QP_STATE_RESET]	= CN_QP_OP_2RESET,
+		[CN_QP_STATE_ERR]	= CN_QP_OP_2ERR,
+		[CN_QP_STATE_SQD]	= CN_QP_OP_NOP,
+		[CN_QP_STATE_QPD]	= CN_QP_OP_NOP,
+		[CN_QP_STATE_RTR]	= CN_QP_OP_QPD_2RTR,
+	},
+	[CN_QP_STATE_SQERR] = {
+		[CN_QP_STATE_RESET]	= CN_QP_OP_2RESET,
+		[CN_QP_STATE_ERR]	= CN_QP_OP_2ERR,
+		[CN_QP_STATE_SQD]	= CN_QP_OP_SQ_ERR_2SQD,
+		[CN_QP_STATE_SQERR]	= CN_QP_OP_NOP,
+	},
+	[CN_QP_STATE_ERR] = {
+		[CN_QP_STATE_RESET]	= CN_QP_OP_2RESET,
+		[CN_QP_STATE_ERR]	= CN_QP_OP_2ERR,
+	}
+};
+
+static char *cn_qp_state_2name(enum hbl_cn_qp_state state)
+{
+	static char *arr[CN_QP_NUM_STATE] = {
+						"Reset",
+						"Init",
+						"RTR",
+						"RTS",
+						"SQD",
+						"QPD",
+						"SQERR",
+						"ERR",
+	};
+
+	return arr[state];
+}
+
+static inline int wait_for_qpc_idle(struct hbl_cn_port *cn_port, struct hbl_cn_qp *qp, bool is_req)
+{
+	struct hbl_cn_device *hdev = cn_port->hdev;
+	struct hbl_cn_asic_port_funcs *port_funcs;
+	struct hbl_cn_qpc_attr qpc_attr;
+	int i, rc;
+
+	if (!hdev->qp_wait_for_idle)
+		return 0;
+
+	port_funcs = hdev->asic_funcs->port_funcs;
+
+	for (i = 0; i < OPC_SETTLE_RETRY_COUNT; i++) {
+		rc = port_funcs->qpc_query(cn_port, qp->qp_id, is_req, &qpc_attr);
+
+		if (rc && (rc != -EBUSY && rc != -ETIMEDOUT))
+			return rc;
+
+		if (!(rc || qpc_attr.in_work))
+			return 0;
+
+		/* Release lock while we wait before retry.
+		 * Note, we can assert that we are already locked.
+		 */
+		port_funcs->cfg_unlock(cn_port);
+
+		msleep(20);
+
+		port_funcs->cfg_lock(cn_port);
+	}
+
+	rc = port_funcs->qpc_query(cn_port, qp->qp_id, is_req, &qpc_attr);
+
+	if (rc && (rc != -EBUSY && rc != -ETIMEDOUT))
+		return rc;
+
+	if (rc || qpc_attr.in_work)
+		return -ETIMEDOUT;
+
+	return 0;
+}
+
+static int cn_qp_op_reset(struct hbl_cn_port *cn_port, struct hbl_cn_qp *qp)
+{
+	struct hbl_cn_device *hdev = cn_port->hdev;
+	struct hbl_cn_asic_funcs *asic_funcs;
+	int rc, rc1;
+
+	asic_funcs = hdev->asic_funcs;
+
+	/* clear the QPCs */
+	rc = asic_funcs->port_funcs->qpc_clear(cn_port, qp, false);
+	if (rc && hbl_cn_comp_device_operational(hdev))
+		/* Device might not respond during reset if the reset was due to error */
+		dev_err(hdev->dev, "Port %d QP %d: Failed to clear responder QPC\n",
+			qp->port, qp->qp_id);
+	else
+		qp->is_res = false;
+
+	rc1 = asic_funcs->port_funcs->qpc_clear(cn_port, qp, true);
+	if (rc1) {
+		rc = rc1;
+		if (hbl_cn_comp_device_operational(hdev))
+			/* Device might not respond during reset if the reset was due to error */
+			dev_err(hdev->dev, "Port %d QP %d: Failed to clear requestor QPC\n",
+				qp->port, qp->qp_id);
+	} else {
+		qp->is_req = false;
+	}
+
+	/* wait for REQ idle, RES idle is already done in cn_qp_op_2qpd */
+	rc = wait_for_qpc_idle(cn_port, qp, true);
+	if (rc) {
+		dev_err(hdev->dev, "Port %d QP %d, Requestor QPC is not idle (rc %d)\n",
+			cn_port->port, qp->qp_id, rc);
+		return rc;
+	}
+
+	qp->curr_state = CN_QP_STATE_RESET;
+
+	return rc;
+}
+
+static int cn_qp_op_reset_2init(struct hbl_cn_port *cn_port, struct hbl_cn_qp *qp)
+{
+	if (ZERO_OR_NULL_PTR(qp))
+		return -EINVAL;
+
+	qp->curr_state = CN_QP_STATE_INIT;
+
+	return 0;
+}
+
+static int cn_qp_op_2rts(struct hbl_cn_port *cn_port, struct hbl_cn_qp *qp,
+			 struct hbl_cni_req_conn_ctx_in *in)
+{
+	struct hbl_cn_device *hdev = cn_port->hdev;
+	struct hbl_cn_asic_funcs *asic_funcs;
+	int rc;
+
+	asic_funcs = hdev->asic_funcs;
+
+	rc = asic_funcs->set_req_qp_ctx(hdev, in, qp);
+	if (rc)
+		return rc;
+
+	qp->curr_state = CN_QP_STATE_RTS;
+	qp->is_req = true;
+
+	return 0;
+}
+
+static int cn_qp_op_2rtr(struct hbl_cn_port *cn_port, struct hbl_cn_qp *qp,
+			 struct hbl_cni_res_conn_ctx_in *in)
+{
+	struct hbl_cn_device *hdev = cn_port->hdev;
+	struct hbl_cn_asic_funcs *asic_funcs;
+	int rc;
+
+	asic_funcs = hdev->asic_funcs;
+
+	rc = asic_funcs->set_res_qp_ctx(hdev, in, qp);
+	if (rc)
+		return rc;
+
+	qp->curr_state = CN_QP_STATE_RTR;
+	qp->is_res = true;
+
+	return 0;
+}
+
+static inline int cn_qp_invalidate_qpc(struct hbl_cn_port *cn_port, struct hbl_cn_qp *qp,
+				       bool is_req)
+{
+	struct hbl_cn_device *hdev = cn_port->hdev;
+	struct hbl_cn_asic_funcs *asic_funcs;
+	int i, rc;
+
+	asic_funcs = hdev->asic_funcs;
+
+	for (i = 0; i < OP_RETRY_COUNT; i++) {
+		rc = asic_funcs->port_funcs->qpc_invalidate(cn_port, qp, is_req);
+		if (!rc)
+			break;
+
+		usleep_range(100, 200);
+	}
+
+	return rc;
+}
+
+static int cn_qp_invalidate(struct hbl_cn_port *cn_port, struct hbl_cn_qp *qp, bool is_req,
+			    bool wait_for_idle)
+{
+	struct hbl_cn_device *hdev = cn_port->hdev;
+	int rc;
+
+	rc = cn_qp_invalidate_qpc(cn_port, qp, is_req);
+	if (rc) {
+		if (hbl_cn_comp_device_operational(hdev))
+			dev_err(hdev->dev, "Port %d QP %d, failed to invalidate %s QPC (rc %d)\n",
+				cn_port->port, qp->qp_id, is_req ? "Requester" : "Responder", rc);
+		return rc;
+	}
+
+	if (!wait_for_idle || is_req)
+		return 0;
+
+	/* check for QP idle in case of responder only */
+	rc = wait_for_qpc_idle(cn_port, qp, false);
+	if (rc) {
+		dev_err(hdev->dev, "Port %d QP %d, Responder QPC is not idle (rc %d)\n",
+			cn_port->port, qp->qp_id, rc);
+		return rc;
+	}
+
+	return rc;
+}
+
+/* Drain the Requester */
+static int cn_qp_op_rts_2sqd(struct hbl_cn_port *cn_port, struct hbl_cn_qp *qp, void *attr)
+{
+	struct hbl_cn_device *hdev = cn_port->hdev;
+	struct hbl_cn_qpc_drain_attr *drain = attr;
+	int rc = 0;
+
+	switch (qp->curr_state) {
+	case CN_QP_STATE_RTS:
+		cn_qp_invalidate(cn_port, qp, true, drain->wait_for_idle);
+		if (drain->wait_for_idle)
+			ssleep(hdev->qp_drain_time);
+
+		break;
+	default:
+		rc = -EOPNOTSUPP;
+		break;
+	}
+
+	if (!rc)
+		qp->curr_state = CN_QP_STATE_SQD;
+
+	return rc;
+}
+
+/* Re-drain the Requester. This function is called without holding the cfg lock so it must not
+ * access the HW or do anything other than just sleeping.
+ */
+static int cn_qp_op_sqd_2sqd(struct hbl_cn_port *cn_port, struct hbl_cn_qp *qp, void *attr)
+{
+	struct hbl_cn_device *hdev = cn_port->hdev;
+	struct hbl_cn_qpc_drain_attr *drain = attr;
+
+	/* no need to invalidate the QP as it was already invalidated just extend the wait time */
+	if (drain->wait_for_idle)
+		ssleep(hdev->qp_drain_time);
+
+	return 0;
+}
+
+/* Drain the QP (Requester and Responder) */
+static int cn_qp_op_2qpd(struct hbl_cn_port *cn_port, struct hbl_cn_qp *qp, void *attr)
+{
+	struct hbl_cn_qpc_drain_attr *drain = attr;
+	int rc = 0;
+
+	switch (qp->curr_state) {
+	case CN_QP_STATE_RTR:
+		/* In RTR only the Resp is working */
+		cn_qp_invalidate(cn_port, qp, false, drain->wait_for_idle);
+		break;
+	case CN_QP_STATE_RTS:
+		/* In RTS both the Resp and Req are working */
+		cn_qp_op_rts_2sqd(cn_port, qp, attr);
+		cn_qp_invalidate(cn_port, qp, false, drain->wait_for_idle);
+		break;
+	case CN_QP_STATE_SQD:
+		/* In SQD only the Resp is working */
+		cn_qp_invalidate(cn_port, qp, false, drain->wait_for_idle);
+		break;
+	case CN_QP_STATE_QPD:
+		break;
+	default:
+		rc = -EOPNOTSUPP;
+		break;
+	}
+
+	if (!rc)
+		qp->curr_state = CN_QP_STATE_QPD;
+
+	return rc;
+}
+
+static int cn_qp_op_2reset(struct hbl_cn_port *cn_port, struct hbl_cn_qp *qp,
+			   const struct hbl_cn_qpc_reset_attr *attr)
+{
+	struct hbl_cn_device *hdev = cn_port->hdev;
+	struct hbl_cn_asic_funcs *asic_funcs;
+	struct hbl_cn_qpc_drain_attr drain;
+
+	asic_funcs = hdev->asic_funcs;
+
+	/* brute-force reset when reset mode is Hard */
+	if (attr->reset_mode == CN_QP_RESET_MODE_HARD && qp->curr_state != CN_QP_STATE_RESET) {
+		/* invalidate */
+		asic_funcs->port_funcs->qpc_invalidate(cn_port, qp, true);
+		asic_funcs->port_funcs->qpc_invalidate(cn_port, qp, false);
+
+		/* wait for HW digest the invalidation */
+		usleep_range(100, 150);
+
+		cn_qp_op_reset(cn_port, qp);
+		return 0;
+	}
+
+	if (attr->reset_mode == CN_QP_RESET_MODE_GRACEFUL)
+		drain.wait_for_idle = true;
+	else
+		drain.wait_for_idle = false;
+
+	switch (qp->curr_state) {
+	case CN_QP_STATE_RESET:
+		break;
+	case CN_QP_STATE_INIT:
+		cn_qp_op_reset(cn_port, qp);
+		break;
+	case CN_QP_STATE_RTR:
+	case CN_QP_STATE_RTS:
+	case CN_QP_STATE_SQD:
+		cn_qp_op_2qpd(cn_port, qp, &drain);
+		cn_qp_op_reset(cn_port, qp);
+		break;
+	case CN_QP_STATE_QPD:
+		cn_qp_op_reset(cn_port, qp);
+		break;
+	case CN_QP_STATE_SQERR:
+	case CN_QP_STATE_ERR:
+		cn_qp_op_reset(cn_port, qp);
+		break;
+	default:
+		dev_err(hdev->dev, "Port %d QP %d: Unknown state %d, moving to RESET state\n",
+			qp->port, qp->qp_id, qp->curr_state);
+		cn_qp_op_reset(cn_port, qp);
+		break;
+	}
+
+	return 0;
+}
+
+/* QP state handling routines  */
 int hbl_cn_qp_modify(struct hbl_cn_port *cn_port, struct hbl_cn_qp *qp,
 		     enum hbl_cn_qp_state new_state, void *params)
 {
-	return 0;
+	enum hbl_cn_qp_state prev_state = qp->curr_state;
+	struct hbl_cn_device *hdev = cn_port->hdev;
+	struct hbl_cn_asic_port_funcs *port_funcs;
+	enum hbl_cn_qp_state_op op;
+	int rc;
+
+	port_funcs = hdev->asic_funcs->port_funcs;
+
+	/* only SQD->SQD transition can be executed without holding the configuration lock */
+	if (prev_state != CN_QP_STATE_SQD || new_state != CN_QP_STATE_SQD) {
+		if (!port_funcs->cfg_is_locked(cn_port)) {
+			dev_err(hdev->dev,
+				"Configuration lock must be held while moving Port %u QP %u from state %s to %s\n",
+				qp->port, qp->qp_id, cn_qp_state_2name(prev_state),
+				cn_qp_state_2name(new_state));
+			return -EACCES;
+		}
+	}
+
+	if (qp->curr_state >= CN_QP_NUM_STATE || new_state >= CN_QP_NUM_STATE ||
+	    qp_valid_state_op[qp->curr_state][new_state] == CN_QP_OP_INVAL) {
+		dev_err(hdev->dev,
+			"Invalid QP state transition, Port %u QP %u from state %s to %s\n",
+			qp->port, qp->qp_id, cn_qp_state_2name(prev_state),
+			cn_qp_state_2name(new_state));
+		return -EINVAL;
+	}
+
+	if (new_state >= CN_QP_NUM_STATE) {
+		dev_err(hdev->dev, "Invalid QP state %d\n", new_state);
+		return -EINVAL;
+	}
+
+	/* get the operation needed for this state transition */
+	op = qp_valid_state_op[qp->curr_state][new_state];
+
+	switch (op) {
+	case CN_QP_OP_2RESET:
+		rc = cn_qp_op_2reset(cn_port, qp, params);
+		break;
+	case CN_QP_OP_RST_2INIT:
+		rc = cn_qp_op_reset_2init(cn_port, qp);
+		break;
+	case CN_QP_OP_INIT_2RTR:
+		rc = cn_qp_op_2rtr(cn_port, qp, params);
+		break;
+	case CN_QP_OP_RTR_2RTR:
+		rc = cn_qp_op_2rtr(cn_port, qp, params);
+		break;
+	case CN_QP_OP_RTR_2QPD:
+		rc = cn_qp_op_2qpd(cn_port, qp, params);
+		break;
+	case CN_QP_OP_RTR_2RTS:
+		rc = cn_qp_op_2rts(cn_port, qp, params);
+		break;
+	case CN_QP_OP_RTS_2RTS:
+		rc = cn_qp_op_2rts(cn_port, qp, params);
+		break;
+	case CN_QP_OP_RTS_2SQD:
+		rc = cn_qp_op_rts_2sqd(cn_port, qp, params);
+		break;
+	case CN_QP_OP_SQD_2SQD:
+		rc = cn_qp_op_sqd_2sqd(cn_port, qp, params);
+		break;
+	case CN_QP_OP_RTS_2QPD:
+		rc = cn_qp_op_2qpd(cn_port, qp, params);
+		break;
+	case CN_QP_OP_SQD_2QPD:
+		rc = cn_qp_op_2qpd(cn_port, qp, params);
+		break;
+	case CN_QP_OP_INVAL:
+		rc = -EINVAL;
+		break;
+	case CN_QP_OP_NOP:
+		rc = 0;
+		break;
+	default:
+		rc = -EOPNOTSUPP;
+		break;
+	}
+
+	if (rc)
+		dev_err(hdev->dev,
+			"Errors detected while moving Port %u QP %u from state %s to %s, (rc %d)\n",
+			qp->port, qp->qp_id, cn_qp_state_2name(prev_state),
+			cn_qp_state_2name(new_state), rc);
+
+	return rc;
 }