| Message ID | 20230329170619.183064-2-saikrishnag@marvell.com (mailing list archive) |
|---|---|
| State | Superseded |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | octeontx2: Miscellaneous fixes | expand |
On Wed, Mar 29, 2023 at 10:36:13PM +0530, Sai Krishna wrote: > From: Geetha sowjanya <gakula@marvell.com> > > APR table contains the lmtst base address of PF/VFs. > These entries are updated by the PF/VF during the > device probe. Due to race condition while updating the > entries are getting corrupted. Hence secure the APR > table update with the lock. However, I don't see rsrc_lock in probe path. otx2_probe() -> cn10k_lmtst_init() -> lmt_base/lmstst is updated with and without mbox.lock. Where did you take rsrc_lock in probe flow? Thanks > > Fixes: 893ae97214c3 ("octeontx2-af: cn10k: Support configurable LMTST regions") > Signed-off-by: Geetha sowjanya <gakula@marvell.com> > Signed-off-by: Sunil Kovvuri Goutham <sgoutham@marvell.com> > Signed-off-by: Sai Krishna <saikrishnag@marvell.com> > --- > drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c | 8 +++++--- > 1 file changed, 5 insertions(+), 3 deletions(-) > > diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c > index 4ad9ff025c96..8530250f6fba 100644 > --- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c > +++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c > @@ -142,16 +142,17 @@ int rvu_mbox_handler_lmtst_tbl_setup(struct rvu *rvu, > * region, if so, convert that IOVA to physical address and > * populate LMT table with that address > */ > + mutex_lock(&rvu->rsrc_lock); > if (req->use_local_lmt_region) { > err = rvu_get_lmtaddr(rvu, req->hdr.pcifunc, > req->lmt_iova, &lmt_addr); > if (err < 0) > - return err; > + goto error; > > /* Update the lmt addr for this PFFUNC in the LMT table */ > err = rvu_update_lmtaddr(rvu, req->hdr.pcifunc, lmt_addr); > if (err) > - return err; > + goto error; > } > > /* Reconfiguring lmtst map table in lmt region shared mode i.e. make > @@ -181,7 +182,7 @@ int rvu_mbox_handler_lmtst_tbl_setup(struct rvu *rvu, > */ > err = rvu_update_lmtaddr(rvu, req->hdr.pcifunc, val); > if (err) > - return err; > + goto error; > } > > /* This mailbox can also be used to update word1 of APR_LMT_MAP_ENTRY_S > @@ -230,6 +231,7 @@ int rvu_mbox_handler_lmtst_tbl_setup(struct rvu *rvu, > } > > error: > + mutex_unlock(&rvu->rsrc_lock); > return err; > } > > -- > 2.25.1 >
>-----Original Message----- >From: Leon Romanovsky <leon@kernel.org> >Sent: Thursday, March 30, 2023 11:26 AM >To: Sai Krishna Gajula <saikrishnag@marvell.com> >Cc: davem@davemloft.net; edumazet@google.com; kuba@kernel.org; pabeni@redhat.com; netdev@vger.kernel.org; linux-kernel@vger.kernel.org; Sunil Kovvuri Goutham <sgoutham@marvell.com>; >richardcochran@gmail.com; Geethasowjanya Akula <gakula@marvell.com> >Subject: [EXT] Re: [net PATCH 1/7] octeontx2-af: Secure APR table update with the lock >External Email >---------------------------------------------------------------------- >On Wed, Mar 29, 2023 at 10:36:13PM +0530, Sai Krishna wrote: >> From: Geetha sowjanya <gakula@marvell.com> >> >> APR table contains the lmtst base address of PF/VFs. >> These entries are updated by the PF/VF during the device probe. Due to >> race condition while updating the entries are getting corrupted. Hence >> secure the APR table update with the lock. >However, I don't see rsrc_lock in probe path. >otx2_probe() >-> cn10k_lmtst_init() > -> lmt_base/lmstst is updated with and without mbox.lock. >Where did you take rsrc_lock in probe flow? rsrc_lock is initialized in AF driver. PF/VF driver in cn10k_lmtst_init() send a mbox request to AF to update the lmtst table. mbox handler in AF takes rsrc_lock to update lmtst table. Thanks, Geetha. >Thanks >> >> Fixes: 893ae97214c3 ("octeontx2-af: cn10k: Support configurable LMTST >> regions") >> Signed-off-by: Geetha sowjanya <gakula@marvell.com> >> Signed-off-by: Sunil Kovvuri Goutham <sgoutham@marvell.com> >> Signed-off-by: Sai Krishna <saikrishnag@marvell.com> >> --- >> drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c | 8 +++++--- >> 1 file changed, 5 insertions(+), 3 deletions(-) >> >> diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c >> b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c >> index 4ad9ff025c96..8530250f6fba 100644 >> --- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c >> +++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c >> @@ -142,16 +142,17 @@ int rvu_mbox_handler_lmtst_tbl_setup(struct rvu *rvu, >> * region, if so, convert that IOVA to physical address and >> * populate LMT table with that address >> */ >> + mutex_lock(&rvu->rsrc_lock); >> if (req->use_local_lmt_region) { >> err = rvu_get_lmtaddr(rvu, req->hdr.pcifunc, >> req->lmt_iova, &lmt_addr); >> if (err < 0) >> - return err; >> + goto error; >> >> /* Update the lmt addr for this PFFUNC in the LMT table */ >> err = rvu_update_lmtaddr(rvu, req->hdr.pcifunc, lmt_addr); >> if (err) >> - return err; >> + goto error; >> } >> >> /* Reconfiguring lmtst map table in lmt region shared mode i.e. make >> @@ -181,7 +182,7 @@ int rvu_mbox_handler_lmtst_tbl_setup(struct rvu *rvu, >> */ >> err = rvu_update_lmtaddr(rvu, req->hdr.pcifunc, val); >> if (err) >> - return err; >> + goto error; >> } >> >> /* This mailbox can also be used to update word1 of >> APR_LMT_MAP_ENTRY_S @@ -230,6 +231,7 @@ int rvu_mbox_handler_lmtst_tbl_setup(struct rvu *rvu, >> } >> >> error: >> + mutex_unlock(&rvu->rsrc_lock); >> return err; >> } >> >> -- >> 2.25.1 >>
On Thu, Mar 30, 2023 at 06:56:54AM +0000, Geethasowjanya Akula wrote: > > >-----Original Message----- > >From: Leon Romanovsky <leon@kernel.org> > >Sent: Thursday, March 30, 2023 11:26 AM > >To: Sai Krishna Gajula <saikrishnag@marvell.com> > >Cc: davem@davemloft.net; edumazet@google.com; kuba@kernel.org; pabeni@redhat.com; netdev@vger.kernel.org; linux-kernel@vger.kernel.org; Sunil Kovvuri Goutham <sgoutham@marvell.com>; >richardcochran@gmail.com; Geethasowjanya Akula <gakula@marvell.com> > >Subject: [EXT] Re: [net PATCH 1/7] octeontx2-af: Secure APR table update with the lock > > >External Email > > >---------------------------------------------------------------------- > >On Wed, Mar 29, 2023 at 10:36:13PM +0530, Sai Krishna wrote: > >> From: Geetha sowjanya <gakula@marvell.com> > >> > >> APR table contains the lmtst base address of PF/VFs. > >> These entries are updated by the PF/VF during the device probe. Due to > >> race condition while updating the entries are getting corrupted. Hence > >> secure the APR table update with the lock. > > >However, I don't see rsrc_lock in probe path. > >otx2_probe() > >-> cn10k_lmtst_init() > > -> lmt_base/lmstst is updated with and without mbox.lock. > > >Where did you take rsrc_lock in probe flow? > > rsrc_lock is initialized in AF driver. PF/VF driver in cn10k_lmtst_init() send a mbox request to AF to update the lmtst table. > mbox handler in AF takes rsrc_lock to update lmtst table. Can you please present the stack trace of such flow? What are the actual variables/struct rsrc_lock is protecting? Thanks > > Thanks, > Geetha. > > >Thanks > > >> > >> Fixes: 893ae97214c3 ("octeontx2-af: cn10k: Support configurable LMTST > >> regions") > >> Signed-off-by: Geetha sowjanya <gakula@marvell.com> > >> Signed-off-by: Sunil Kovvuri Goutham <sgoutham@marvell.com> > >> Signed-off-by: Sai Krishna <saikrishnag@marvell.com> > >> --- > >> drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c | 8 +++++--- > >> 1 file changed, 5 insertions(+), 3 deletions(-) > >> > >> diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c > >> b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c > >> index 4ad9ff025c96..8530250f6fba 100644 > >> --- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c > >> +++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c > >> @@ -142,16 +142,17 @@ int rvu_mbox_handler_lmtst_tbl_setup(struct rvu *rvu, > >> * region, if so, convert that IOVA to physical address and > >> * populate LMT table with that address > >> */ > >> + mutex_lock(&rvu->rsrc_lock); > >> if (req->use_local_lmt_region) { > >> err = rvu_get_lmtaddr(rvu, req->hdr.pcifunc, > >> req->lmt_iova, &lmt_addr); > >> if (err < 0) > >> - return err; > >> + goto error; > >> > >> /* Update the lmt addr for this PFFUNC in the LMT table */ > >> err = rvu_update_lmtaddr(rvu, req->hdr.pcifunc, lmt_addr); > >> if (err) > >> - return err; > >> + goto error; > >> } > >> > >> /* Reconfiguring lmtst map table in lmt region shared mode i.e. make > >> @@ -181,7 +182,7 @@ int rvu_mbox_handler_lmtst_tbl_setup(struct rvu *rvu, > >> */ > >> err = rvu_update_lmtaddr(rvu, req->hdr.pcifunc, val); > >> if (err) > >> - return err; > >> + goto error; > >> } > >> > >> /* This mailbox can also be used to update word1 of > >> APR_LMT_MAP_ENTRY_S @@ -230,6 +231,7 @@ int rvu_mbox_handler_lmtst_tbl_setup(struct rvu *rvu, > >> } > >> > >> error: > >> + mutex_unlock(&rvu->rsrc_lock); > >> return err; > >> } > >> > >> -- > >> 2.25.1 > >>
>-----Original Message----- >From: Leon Romanovsky <leon@kernel.org> >Sent: Thursday, March 30, 2023 12:46 PM >To: Geethasowjanya Akula <gakula@marvell.com> >Cc: Sai Krishna Gajula <saikrishnag@marvell.com>; davem@davemloft.net; edumazet@google.com; kuba@kernel.org; pabeni@redhat.com; netdev@vger.kernel.org; linux-kernel@vger.kernel.org; Sunil >Kovvuri Goutham <sgoutham@marvell.com>; richardcochran@gmail.com >Subject: Re: [EXT] Re: [net PATCH 1/7] octeontx2-af: Secure APR table update with the lock > >On Thu, Mar 30, 2023 at 06:56:54AM +0000, Geethasowjanya Akula wrote: >> >> >-----Original Message----- >> >From: Leon Romanovsky <leon@kernel.org> >> >Sent: Thursday, March 30, 2023 11:26 AM >> >To: Sai Krishna Gajula <saikrishnag@marvell.com> >> >Cc: davem@davemloft.net; edumazet@google.com; kuba@kernel.org; >> >pabeni@redhat.com; netdev@vger.kernel.org; >> >linux-kernel@vger.kernel.org; Sunil Kovvuri Goutham >> ><sgoutham@marvell.com>; >richardcochran@gmail.com; Geethasowjanya >> >Akula <gakula@marvell.com> >> >Subject: [EXT] Re: [net PATCH 1/7] octeontx2-af: Secure APR table >> >update with the lock >> >> >External Email >> >> >--------------------------------------------------------------------- >> >- On Wed, Mar 29, 2023 at 10:36:13PM +0530, Sai Krishna wrote: >> >> From: Geetha sowjanya <gakula@marvell.com> >> >> >> >> APR table contains the lmtst base address of PF/VFs. >> >> These entries are updated by the PF/VF during the device probe. Due >> >> to race condition while updating the entries are getting corrupted. >> >> Hence secure the APR table update with the lock. >> >> >However, I don't see rsrc_lock in probe path. >> >otx2_probe() >> >-> cn10k_lmtst_init() >> > -> lmt_base/lmstst is updated with and without mbox.lock. >> >> >Where did you take rsrc_lock in probe flow? >> >> rsrc_lock is initialized in AF driver. PF/VF driver in cn10k_lmtst_init() send a mbox request to AF to update the lmtst table. >> mbox handler in AF takes rsrc_lock to update lmtst table. >Can you please present the stack trace of such flow? What are the actual variables/struct rsrc_lock is protecting? The lock tries to protect the request and response register at line#73 and line#83 in below function, from getting overwritten when Multiple PFs invokes rvu_get_lmtaddr() simultaneously. For example, if PF1 submit the request at line#73 and got permitted before it reads the response at line#80. PF2 got scheduled submit the request then the response of PF1 is overwritten by the PF2 response. When PF1 gets reschedule, it reads wrong data. #static int rvu_get_lmtaddr(struct rvu *rvu, u16 pcifunc, 59 u64 iova, u64 *lmt_addr) 60 { 61 [...] 68 69 rvu_write64(rvu, BLKADDR_RVUM, RVU_AF_SMMU_ADDR_REQ, iova); 70 pf = rvu_get_pf(pcifunc) & 0x1F; 71 val = BIT_ULL(63) | BIT_ULL(14) | BIT_ULL(13) | pf << 8 | 72 ((pcifunc & RVU_PFVF_FUNC_MASK) & 0xFF); 73 rvu_write64(rvu, BLKADDR_RVUM, RVU_AF_SMMU_TXN_REQ, val); 74 75 err = rvu_poll_reg(rvu, BLKADDR_RVUM, RVU_AF_SMMU_ADDR_RSP_STS, BIT_ULL(0), false); 76 if (err) { 77 dev_err(rvu->dev, "%s LMTLINE iova transulation failed\n", __func__); 78 return err; 79 } 80 val = rvu_read64(rvu, BLKADDR_RVUM, RVU_AF_SMMU_ADDR_RSP_STS); 81 if (val & ~0x1ULL) { 82 dev_err(rvu->dev, "%s LMTLINE iova transulation failed err:%llx\n", __func__, val); 83 return -EIO; 84 } 85 Thanks. >>Thanks >> >> Thanks, >> Geetha. >> >> >Thanks >> >> >> >> >> Fixes: 893ae97214c3 ("octeontx2-af: cn10k: Support configurable >> >> LMTST >> >> regions") >> >> Signed-off-by: Geetha sowjanya <gakula@marvell.com> >> >> Signed-off-by: Sunil Kovvuri Goutham <sgoutham@marvell.com> >> >> Signed-off-by: Sai Krishna <saikrishnag@marvell.com> >> >> --- >> >> drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c | 8 +++++--- >> >> 1 file changed, 5 insertions(+), 3 deletions(-) >> >> >> >> diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c >> >> b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c >> >> index 4ad9ff025c96..8530250f6fba 100644 >> >> --- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c >> >> +++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c >> >> @@ -142,16 +142,17 @@ int rvu_mbox_handler_lmtst_tbl_setup(struct rvu *rvu, >> >> * region, if so, convert that IOVA to physical address and >> >> * populate LMT table with that address >> >> */ >> >> + mutex_lock(&rvu->rsrc_lock); >> >> if (req->use_local_lmt_region) { >> >> err = rvu_get_lmtaddr(rvu, req->hdr.pcifunc, >> >> req->lmt_iova, &lmt_addr); >> >> if (err < 0) >> >> - return err; >> >> + goto error; >> >> >> >> /* Update the lmt addr for this PFFUNC in the LMT table */ >> >> err = rvu_update_lmtaddr(rvu, req->hdr.pcifunc, lmt_addr); >> >> if (err) >> >> - return err; >> >> + goto error; >> >> } >> >> >> >> /* Reconfiguring lmtst map table in lmt region shared mode i.e. >> >> make @@ -181,7 +182,7 @@ int rvu_mbox_handler_lmtst_tbl_setup(struct rvu *rvu, >> >> */ >> >> err = rvu_update_lmtaddr(rvu, req->hdr.pcifunc, val); >> >> if (err) >> >> - return err; >> >> + goto error; >> >> } >> >> >> >> /* This mailbox can also be used to update word1 of >> >> APR_LMT_MAP_ENTRY_S @@ -230,6 +231,7 @@ int rvu_mbox_handler_lmtst_tbl_setup(struct rvu *rvu, >> >> } >> >> >> >> error: >> >> + mutex_unlock(&rvu->rsrc_lock); >> >> return err; >> >> } >> >> >> >> -- >> >> 2.25.1 >> >>
On Fri, Mar 31, 2023 at 08:56:59AM +0000, Geethasowjanya Akula wrote: > > > >-----Original Message----- > >From: Leon Romanovsky <leon@kernel.org> > >Sent: Thursday, March 30, 2023 12:46 PM > >To: Geethasowjanya Akula <gakula@marvell.com> > >Cc: Sai Krishna Gajula <saikrishnag@marvell.com>; davem@davemloft.net; edumazet@google.com; kuba@kernel.org; pabeni@redhat.com; netdev@vger.kernel.org; linux-kernel@vger.kernel.org; Sunil >Kovvuri Goutham <sgoutham@marvell.com>; richardcochran@gmail.com > >Subject: Re: [EXT] Re: [net PATCH 1/7] octeontx2-af: Secure APR table update with the lock > > > >On Thu, Mar 30, 2023 at 06:56:54AM +0000, Geethasowjanya Akula wrote: > >> > >> >-----Original Message----- > >> >From: Leon Romanovsky <leon@kernel.org> > >> >Sent: Thursday, March 30, 2023 11:26 AM > >> >To: Sai Krishna Gajula <saikrishnag@marvell.com> > >> >Cc: davem@davemloft.net; edumazet@google.com; kuba@kernel.org; > >> >pabeni@redhat.com; netdev@vger.kernel.org; > >> >linux-kernel@vger.kernel.org; Sunil Kovvuri Goutham > >> ><sgoutham@marvell.com>; >richardcochran@gmail.com; Geethasowjanya > >> >Akula <gakula@marvell.com> > >> >Subject: [EXT] Re: [net PATCH 1/7] octeontx2-af: Secure APR table > >> >update with the lock > >> > >> >External Email > >> > >> >--------------------------------------------------------------------- > >> >- On Wed, Mar 29, 2023 at 10:36:13PM +0530, Sai Krishna wrote: > >> >> From: Geetha sowjanya <gakula@marvell.com> > >> >> > >> >> APR table contains the lmtst base address of PF/VFs. > >> >> These entries are updated by the PF/VF during the device probe. Due > >> >> to race condition while updating the entries are getting corrupted. > >> >> Hence secure the APR table update with the lock. > >> > >> >However, I don't see rsrc_lock in probe path. > >> >otx2_probe() > >> >-> cn10k_lmtst_init() > >> > -> lmt_base/lmstst is updated with and without mbox.lock. > >> > >> >Where did you take rsrc_lock in probe flow? > >> > >> rsrc_lock is initialized in AF driver. PF/VF driver in cn10k_lmtst_init() send a mbox request to AF to update the lmtst table. > >> mbox handler in AF takes rsrc_lock to update lmtst table. > > >Can you please present the stack trace of such flow? What are the actual variables/struct rsrc_lock is protecting? > > The lock tries to protect the request and response register at line#73 and line#83 in below function, from getting overwritten when > Multiple PFs invokes rvu_get_lmtaddr() simultaneously. > For example, if PF1 submit the request at line#73 and got permitted before it reads the response at line#80. > PF2 got scheduled submit the request then the response of PF1 is overwritten by the PF2 response. > When PF1 gets reschedule, it reads wrong data. I see, thanks
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c index 4ad9ff025c96..8530250f6fba 100644 --- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c +++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c @@ -142,16 +142,17 @@ int rvu_mbox_handler_lmtst_tbl_setup(struct rvu *rvu, * region, if so, convert that IOVA to physical address and * populate LMT table with that address */ + mutex_lock(&rvu->rsrc_lock); if (req->use_local_lmt_region) { err = rvu_get_lmtaddr(rvu, req->hdr.pcifunc, req->lmt_iova, &lmt_addr); if (err < 0) - return err; + goto error; /* Update the lmt addr for this PFFUNC in the LMT table */ err = rvu_update_lmtaddr(rvu, req->hdr.pcifunc, lmt_addr); if (err) - return err; + goto error; } /* Reconfiguring lmtst map table in lmt region shared mode i.e. make @@ -181,7 +182,7 @@ int rvu_mbox_handler_lmtst_tbl_setup(struct rvu *rvu, */ err = rvu_update_lmtaddr(rvu, req->hdr.pcifunc, val); if (err) - return err; + goto error; } /* This mailbox can also be used to update word1 of APR_LMT_MAP_ENTRY_S @@ -230,6 +231,7 @@ int rvu_mbox_handler_lmtst_tbl_setup(struct rvu *rvu, } error: + mutex_unlock(&rvu->rsrc_lock); return err; }