cxl/port: Fix decoder initialization when nr_targets > interleave_ways

Commit Message

Dan Williams Dec. 22, 2023, 6:12 a.m. UTC

From: Huang Ying <[e-mail]>

The decoder_populate_targets() helper walks all of the targets in a port
and makes sure they can be looked up in @target_map. Where @target_map
is a lookup table from target position to target id (corresponding to a
cxl_dport instance). However @target_map is only responsible for
conveying the active dport instances as conveyed by interleave_ways.

When nr_targets > interleave_ways it results in
decoder_populate_targets() walking off the end of the valid entries in
@target_map. Given target_map is initialized to 0 it results in the
dport lookup failing if position 0 is not mapped to a dport with an id
of 0:

  cxl_port port3: Failed to populate active decoder targets
  cxl_port port3: Failed to add decoder
  cxl_port port3: Failed to add decoder3.0
  cxl_bus_probe: cxl_port port3: probe: -6

This bug also highlights that when the decoder's ->targets[] array is
written in cxl_port_setup_targets() it is missing a hold of the
targets_lock to synchronize against sysfs readers of the target list. A
fix for that is saved for a later patch.

Fixes: a5c258021689 ("cxl/bus: Populate the target list at decoder create")
Cc: <[e-mail]>
Signed-off-by: "Huang, Ying" <[e-mail]>
[djbw: rewrite the changelog, find the Fixes: tag]
Co-developed-by: Dan Williams <[e-mail]>
Signed-off-by: Dan Williams <[e-mail]>
---
 drivers/cxl/core/port.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Comments

Dave Jiang Dec. 22, 2023, 3:57 p.m. UTC | #1

On 12/21/23 23:12, Dan Williams wrote:
> From: Huang Ying <[e-mail]>
> 
> The decoder_populate_targets() helper walks all of the targets in a port
> and makes sure they can be looked up in @target_map. Where @target_map
> is a lookup table from target position to target id (corresponding to a
> cxl_dport instance). However @target_map is only responsible for
> conveying the active dport instances as conveyed by interleave_ways.
> 
> When nr_targets > interleave_ways it results in
> decoder_populate_targets() walking off the end of the valid entries in
> @target_map. Given target_map is initialized to 0 it results in the
> dport lookup failing if position 0 is not mapped to a dport with an id
> of 0:
> 
>   cxl_port port3: Failed to populate active decoder targets
>   cxl_port port3: Failed to add decoder
>   cxl_port port3: Failed to add decoder3.0
>   cxl_bus_probe: cxl_port port3: probe: -6
> 
> This bug also highlights that when the decoder's ->targets[] array is
> written in cxl_port_setup_targets() it is missing a hold of the
> targets_lock to synchronize against sysfs readers of the target list. A
> fix for that is saved for a later patch.
> 
> Fixes: a5c258021689 ("cxl/bus: Populate the target list at decoder create")
> Cc: <[e-mail]>
> Signed-off-by: "Huang, Ying" <[e-mail]>
> [djbw: rewrite the changelog, find the Fixes: tag]
> Co-developed-by: Dan Williams <[e-mail]>
> Signed-off-by: Dan Williams <[e-mail]>

Reviewed-by: Dave Jiang <[e-mail]>
> ---
>  drivers/cxl/core/port.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
> index b7c93bb18f6e..57495cdc181f 100644
> --- a/drivers/cxl/core/port.c
> +++ b/drivers/cxl/core/port.c
> @@ -1644,7 +1644,7 @@ static int decoder_populate_targets(struct cxl_switch_decoder *cxlsd,
>  		return -EINVAL;
>  
>  	write_seqlock(&cxlsd->target_lock);
> -	for (i = 0; i < cxlsd->nr_targets; i++) {
> +	for (i = 0; i < cxlsd->cxld.interleave_ways; i++) {
>  		struct cxl_dport *dport = find_dport(port, target_map[i]);
>  
>  		if (!dport) {
> 
>

Alison Schofield Dec. 22, 2023, 8:12 p.m. UTC | #2

On Thu, Dec 21, 2023 at 10:12:12PM -0800, Dan Williams wrote:
> From: Huang Ying <[e-mail]>
> 
> The decoder_populate_targets() helper walks all of the targets in a port
> and makes sure they can be looked up in @target_map. Where @target_map
> is a lookup table from target position to target id (corresponding to a
> cxl_dport instance). However @target_map is only responsible for
> conveying the active dport instances as conveyed by interleave_ways.
> 
> When nr_targets > interleave_ways it results in
> decoder_populate_targets() walking off the end of the valid entries in
> @target_map. Given target_map is initialized to 0 it results in the
> dport lookup failing if position 0 is not mapped to a dport with an id
> of 0:
> 
>   cxl_port port3: Failed to populate active decoder targets
>   cxl_port port3: Failed to add decoder
>   cxl_port port3: Failed to add decoder3.0
>   cxl_bus_probe: cxl_port port3: probe: -6
> 
> This bug also highlights that when the decoder's ->targets[] array is
> written in cxl_port_setup_targets() it is missing a hold of the
> targets_lock to synchronize against sysfs readers of the target list. A
> fix for that is saved for a later patch.
> 
> Fixes: a5c258021689 ("cxl/bus: Populate the target list at decoder create")
> Cc: <[e-mail]>
> Signed-off-by: "Huang, Ying" <[e-mail]>
> [djbw: rewrite the changelog, find the Fixes: tag]
> Co-developed-by: Dan Williams <[e-mail]>
> Signed-off-by: Dan Williams <[e-mail]>
> ---
>  drivers/cxl/core/port.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
> index b7c93bb18f6e..57495cdc181f 100644
> --- a/drivers/cxl/core/port.c
> +++ b/drivers/cxl/core/port.c
> @@ -1644,7 +1644,7 @@ static int decoder_populate_targets(struct cxl_switch_decoder *cxlsd,
>  		return -EINVAL;
>  
>  	write_seqlock(&cxlsd->target_lock);
> -	for (i = 0; i < cxlsd->nr_targets; i++) {
> +	for (i = 0; i < cxlsd->cxld.interleave_ways; i++) {
>  		struct cxl_dport *dport = find_dport(port, target_map[i]);
>  

Does this loop need to protect against interleave_ways > nr_targets?
ie protect from walking off the target_map[nr_targets].

There is a check for that in cxl_port_setup_targets() 
>>   if (iw > 8 || iw > cxlsd->nr_targets) {
>> 		dev_dbg(&cxlr->dev,
>> 			"%s:%s:%s: ways: %d overflows targets: %d\n",

Wondering if a check at the time of walking is clearer, esp since
nr_targets is explicitly defined w the target list in cxlsd.

>> struct cxl_switch_decoder {
>>         struct cxl_decoder cxld;
>>         int nr_targets;
>>         struct cxl_dport *target[];
>> };







>  		if (!dport) {
>

Dan Williams Dec. 22, 2023, 9:10 p.m. UTC | #3

Alison Schofield wrote:
> On Thu, Dec 21, 2023 at 10:12:12PM -0800, Dan Williams wrote:
> > From: Huang Ying <[e-mail]>
> > 
> > The decoder_populate_targets() helper walks all of the targets in a port
> > and makes sure they can be looked up in @target_map. Where @target_map
> > is a lookup table from target position to target id (corresponding to a
> > cxl_dport instance). However @target_map is only responsible for
> > conveying the active dport instances as conveyed by interleave_ways.
> > 
> > When nr_targets > interleave_ways it results in
> > decoder_populate_targets() walking off the end of the valid entries in
> > @target_map. Given target_map is initialized to 0 it results in the
> > dport lookup failing if position 0 is not mapped to a dport with an id
> > of 0:
> > 
> >   cxl_port port3: Failed to populate active decoder targets
> >   cxl_port port3: Failed to add decoder
> >   cxl_port port3: Failed to add decoder3.0
> >   cxl_bus_probe: cxl_port port3: probe: -6
> > 
> > This bug also highlights that when the decoder's ->targets[] array is
> > written in cxl_port_setup_targets() it is missing a hold of the
> > targets_lock to synchronize against sysfs readers of the target list. A
> > fix for that is saved for a later patch.
> > 
> > Fixes: a5c258021689 ("cxl/bus: Populate the target list at decoder create")
> > Cc: <[e-mail]>
> > Signed-off-by: "Huang, Ying" <[e-mail]>
> > [djbw: rewrite the changelog, find the Fixes: tag]
> > Co-developed-by: Dan Williams <[e-mail]>
> > Signed-off-by: Dan Williams <[e-mail]>
> > ---
> >  drivers/cxl/core/port.c |    2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
> > index b7c93bb18f6e..57495cdc181f 100644
> > --- a/drivers/cxl/core/port.c
> > +++ b/drivers/cxl/core/port.c
> > @@ -1644,7 +1644,7 @@ static int decoder_populate_targets(struct cxl_switch_decoder *cxlsd,
> >  		return -EINVAL;
> >  
> >  	write_seqlock(&cxlsd->target_lock);
> > -	for (i = 0; i < cxlsd->nr_targets; i++) {
> > +	for (i = 0; i < cxlsd->cxld.interleave_ways; i++) {
> >  		struct cxl_dport *dport = find_dport(port, target_map[i]);
> >  
> 
> Does this loop need to protect against interleave_ways > nr_targets?
> ie protect from walking off the target_map[nr_targets].

It's a good review question, but I think target_map[] is safe from those
shenanigans. For the CFMWS case interleave_ways == nr_targets, see the
@nr_tagets argument to cxl_root_decoder_alloc(). For the mid-level
switch decoder case it is protected by the fact that the decoder's
interleave_ways setting is sanity checked by the eiw_to_ways() call in
init_hdm_decoder(). So there's never any danger of walking off the end
of the target_map[] because that is allocated to support the
spec-defined hardware-max of CXL_DECODER_MAX_INTERLEAVE.

> There is a check for that in cxl_port_setup_targets() 
> >>   if (iw > 8 || iw > cxlsd->nr_targets) {
> >> 		dev_dbg(&cxlr->dev,
> >> 			"%s:%s:%s: ways: %d overflows targets: %d\n",

That check is for programming mid-level decoders where we find out at
run time that the interleave_ways of the region can not be satisfied by
one of the decoders in the chain, so that one is not about walking past
the end of a target list, that one is about detecting impossible region
configurations.

Alison Schofield Dec. 22, 2023, 10:01 p.m. UTC | #4

On Fri, Dec 22, 2023 at 01:10:52PM -0800, Dan Williams wrote:
> Alison Schofield wrote:
> > On Thu, Dec 21, 2023 at 10:12:12PM -0800, Dan Williams wrote:
> > > From: Huang Ying <[e-mail]>
> > > 
> > > The decoder_populate_targets() helper walks all of the targets in a port
> > > and makes sure they can be looked up in @target_map. Where @target_map
> > > is a lookup table from target position to target id (corresponding to a
> > > cxl_dport instance). However @target_map is only responsible for
> > > conveying the active dport instances as conveyed by interleave_ways.
> > > 
> > > When nr_targets > interleave_ways it results in
> > > decoder_populate_targets() walking off the end of the valid entries in
> > > @target_map. Given target_map is initialized to 0 it results in the
> > > dport lookup failing if position 0 is not mapped to a dport with an id
> > > of 0:
> > > 
> > >   cxl_port port3: Failed to populate active decoder targets
> > >   cxl_port port3: Failed to add decoder
> > >   cxl_port port3: Failed to add decoder3.0
> > >   cxl_bus_probe: cxl_port port3: probe: -6
> > > 
> > > This bug also highlights that when the decoder's ->targets[] array is
> > > written in cxl_port_setup_targets() it is missing a hold of the
> > > targets_lock to synchronize against sysfs readers of the target list. A
> > > fix for that is saved for a later patch.
> > > 
> > > Fixes: a5c258021689 ("cxl/bus: Populate the target list at decoder create")
> > > Cc: <[e-mail]>
> > > Signed-off-by: "Huang, Ying" <[e-mail]>
> > > [djbw: rewrite the changelog, find the Fixes: tag]
> > > Co-developed-by: Dan Williams <[e-mail]>
> > > Signed-off-by: Dan Williams <[e-mail]>
> > > ---

Thanks for answering my questions -

Reviewed-by: Alison Schofield <[e-mail]>



> > >  drivers/cxl/core/port.c |    2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
> > > index b7c93bb18f6e..57495cdc181f 100644
> > > --- a/drivers/cxl/core/port.c
> > > +++ b/drivers/cxl/core/port.c
> > > @@ -1644,7 +1644,7 @@ static int decoder_populate_targets(struct cxl_switch_decoder *cxlsd,
> > >  		return -EINVAL;
> > >  
> > >  	write_seqlock(&cxlsd->target_lock);
> > > -	for (i = 0; i < cxlsd->nr_targets; i++) {
> > > +	for (i = 0; i < cxlsd->cxld.interleave_ways; i++) {
> > >  		struct cxl_dport *dport = find_dport(port, target_map[i]);
> > >  
> > 
> > Does this loop need to protect against interleave_ways > nr_targets?
> > ie protect from walking off the target_map[nr_targets].
> 
> It's a good review question, but I think target_map[] is safe from those
> shenanigans. For the CFMWS case interleave_ways == nr_targets, see the
> @nr_tagets argument to cxl_root_decoder_alloc(). For the mid-level
> switch decoder case it is protected by the fact that the decoder's
> interleave_ways setting is sanity checked by the eiw_to_ways() call in
> init_hdm_decoder(). So there's never any danger of walking off the end
> of the target_map[] because that is allocated to support the
> spec-defined hardware-max of CXL_DECODER_MAX_INTERLEAVE.
> 
> > There is a check for that in cxl_port_setup_targets() 
> > >>   if (iw > 8 || iw > cxlsd->nr_targets) {
> > >> 		dev_dbg(&cxlr->dev,
> > >> 			"%s:%s:%s: ways: %d overflows targets: %d\n",
> 
> That check is for programming mid-level decoders where we find out at
> run time that the interleave_ways of the region can not be satisfied by
> one of the decoders in the chain, so that one is not about walking past
> the end of a target list, that one is about detecting impossible region
> configurations.

Patch

diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
index b7c93bb18f6e..57495cdc181f 100644
--- a/drivers/cxl/core/port.c
+++ b/drivers/cxl/core/port.c
@@ -1644,7 +1644,7 @@  static int decoder_populate_targets(struct cxl_switch_decoder *cxlsd,
 		return -EINVAL;
 
 	write_seqlock(&cxlsd->target_lock);
-	for (i = 0; i < cxlsd->nr_targets; i++) {
+	for (i = 0; i < cxlsd->cxld.interleave_ways; i++) {
 		struct cxl_dport *dport = find_dport(port, target_map[i]);
 
 		if (!dport) {