| Message ID | 20220926110938.2800005-1-jiri@resnulli.us (mailing list archive) |
|---|---|
| Headers | show |
| Series | devlink: fix order of port and netdev register in drivers | expand |
Hello: This series was applied to netdev/net-next.git (master) by Jakub Kicinski <kuba@kernel.org>: On Mon, 26 Sep 2022 13:09:35 +0200 you wrote: > From: Jiri Pirko <jiri@nvidia.com> > > Some of the drivers use wrong order in registering devlink port and > netdev, registering netdev first. That was not intended as the devlink > port is some sort of parent for the netdev. Fix the ordering. > > Note that the follow-up patchset is going to make this ordering > mandatory. > > [...] Here is the summary with links: - [net-next,1/3] funeth: unregister devlink port after netdevice unregister https://git.kernel.org/netdev/net-next/c/dfe609491476 - [net-next,2/3] ice: reorder PF/representor devlink port register/unregister flows https://git.kernel.org/netdev/net-next/c/a286ba738714 - [net-next,3/3] ionic: change order of devlink port register and netdev register https://git.kernel.org/netdev/net-next/c/1fd7c08286ce You are awesome, thank you!
Hi Jiri, I think we have another issue with devlink_unregister and related devlink_port_unregister. It is likely not an issue with current drivers because the devlink ports are managed by netdev register/unregister code, and with your patch that will be fine. But by definition, devlink does exist for those things not matching smoothly to netdevs, so it is expected devlink ports not related to existing netdevs at all. That is the case in a patch I'm working on for sfc ef100, where devlink ports are created at PF initialization, so related netdevs will not be there at that point, and they can not exist when the devlink ports are removed when the driver is removed. So the question in this case is, should the devlink ports unregister before or after their devlink unregisters? Since the ports are in a list owned by the devlink struct, I think it seems logical to unregister the ports first, and that is what I did. It works but there exists a potential concurrency issue with devlink user space operations. The devlink code takes care of race conditions involving the devlink struct with rcu plus get/put operations, but that is not the case for devlink ports. Interestingly, unregistering the devlink first, and doing so with the ports without touching/releasing the devlink struct would solve the problem, but not sure this is the right approach here. It does not seem clean, and it would require documenting the right unwinding order and to add a check for DEVLINK_REGISTERED in devlink_port_unregister. I think the right solution would be to add protection to devlink ports and likely other devlink objects with similar concurrency issues. Let me know what you think about it. On 9/26/22 13:09, Jiri Pirko wrote: > CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email. > > > From: Jiri Pirko <jiri@nvidia.com> > > Some of the drivers use wrong order in registering devlink port and > netdev, registering netdev first. That was not intended as the devlink > port is some sort of parent for the netdev. Fix the ordering. > > Note that the follow-up patchset is going to make this ordering > mandatory. > > Jiri Pirko (3): > funeth: unregister devlink port after netdevice unregister > ice: reorder PF/representor devlink port register/unregister flows > ionic: change order of devlink port register and netdev register > > .../net/ethernet/fungible/funeth/funeth_main.c | 2 +- > drivers/net/ethernet/intel/ice/ice_lib.c | 6 +++--- > drivers/net/ethernet/intel/ice/ice_main.c | 12 ++++++------ > drivers/net/ethernet/intel/ice/ice_repr.c | 2 +- > .../net/ethernet/pensando/ionic/ionic_bus_pci.c | 16 ++++++++-------- > 5 files changed, 19 insertions(+), 19 deletions(-) > > -- > 2.37.1 >
Tue, Oct 04, 2022 at 05:31:10PM CEST, alejandro.lucero-palau@amd.com wrote: >Hi Jiri, I don't understand why you send this as a reply to this patchset. I don't see the relation to it. > >I think we have another issue with devlink_unregister and related >devlink_port_unregister. It is likely not an issue with current drivers >because the devlink ports are managed by netdev register/unregister >code, and with your patch that will be fine. > >But by definition, devlink does exist for those things not matching >smoothly to netdevs, so it is expected devlink ports not related to >existing netdevs at all. That is the case in a patch I'm working on for >sfc ef100, where devlink ports are created at PF initialization, so >related netdevs will not be there at that point, and they can not exist >when the devlink ports are removed when the driver is removed. > >So the question in this case is, should the devlink ports unregister >before or after their devlink unregisters? Before. If devlink instance should be unregistered only after all other related instances are gone. Also, the devlink ports come and go during the devlink lifetime. When you add a VF, split a port for example. There are many other cases. > >Since the ports are in a list owned by the devlink struct, I think it >seems logical to unregister the ports first, and that is what I did. It >works but there exists a potential concurrency issue with devlink user What concurrency issue are you talking about? >space operations. The devlink code takes care of race conditions involving the >devlink struct with rcu plus get/put operations, but that is not the >case for devlink ports. > >Interestingly, unregistering the devlink first, and doing so with the >ports without touching/releasing the devlink struct would solve the >problem, but not sure this is the right approach here. It does not seem It is not. As I wrote above, the devlink ports come and go. >clean, and it would require documenting the right unwinding order and >to add a check for DEVLINK_REGISTERED in devlink_port_unregister. > >I think the right solution would be to add protection to devlink ports >and likely other devlink objects with similar concurrency issues. > > >Let me know what you think about it. > > > >On 9/26/22 13:09, Jiri Pirko wrote: >> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email. >> >> >> From: Jiri Pirko <jiri@nvidia.com> >> >> Some of the drivers use wrong order in registering devlink port and >> netdev, registering netdev first. That was not intended as the devlink >> port is some sort of parent for the netdev. Fix the ordering. >> >> Note that the follow-up patchset is going to make this ordering >> mandatory. >> >> Jiri Pirko (3): >> funeth: unregister devlink port after netdevice unregister >> ice: reorder PF/representor devlink port register/unregister flows >> ionic: change order of devlink port register and netdev register >> >> .../net/ethernet/fungible/funeth/funeth_main.c | 2 +- >> drivers/net/ethernet/intel/ice/ice_lib.c | 6 +++--- >> drivers/net/ethernet/intel/ice/ice_main.c | 12 ++++++------ >> drivers/net/ethernet/intel/ice/ice_repr.c | 2 +- >> .../net/ethernet/pensando/ionic/ionic_bus_pci.c | 16 ++++++++-------- >> 5 files changed, 19 insertions(+), 19 deletions(-) >> >> -- >> 2.37.1 >> >
On 10/5/22 09:49, Jiri Pirko wrote: > Tue, Oct 04, 2022 at 05:31:10PM CEST, alejandro.lucero-palau@amd.com wrote: >> Hi Jiri, > I don't understand why you send this as a reply to this patchset. I > don't see the relation to it. I thought there was a relationship with ordering being the issue. Apologies if this is not the right way for rising my concern. > >> I think we have another issue with devlink_unregister and related >> devlink_port_unregister. It is likely not an issue with current drivers >> because the devlink ports are managed by netdev register/unregister >> code, and with your patch that will be fine. >> >> But by definition, devlink does exist for those things not matching >> smoothly to netdevs, so it is expected devlink ports not related to >> existing netdevs at all. That is the case in a patch I'm working on for >> sfc ef100, where devlink ports are created at PF initialization, so >> related netdevs will not be there at that point, and they can not exist >> when the devlink ports are removed when the driver is removed. >> >> So the question in this case is, should the devlink ports unregister >> before or after their devlink unregisters? > Before. If devlink instance should be unregistered only after all other > related instances are gone. > > Also, the devlink ports come and go during the devlink lifetime. When > you add a VF, split a port for example. There are many other cases. > > >> Since the ports are in a list owned by the devlink struct, I think it >> seems logical to unregister the ports first, and that is what I did. It >> works but there exists a potential concurrency issue with devlink user > What concurrency issue are you talking about? > 1) devlink port function set ... 2) predoit inside devlink obtains devlink then the reference to devlink port. Code does a put on devlink but not on the devlink port. 3) driver is removed. devlink port is removed. devlink is not because the put. 4) devlink port reference is wrong. >> space operations. The devlink code takes care of race conditions involving the >> devlink struct with rcu plus get/put operations, but that is not the >> case for devlink ports. >> >> Interestingly, unregistering the devlink first, and doing so with the >> ports without touching/releasing the devlink struct would solve the >> problem, but not sure this is the right approach here. It does not seem > It is not. As I wrote above, the devlink ports come and go. > > >> clean, and it would require documenting the right unwinding order and >> to add a check for DEVLINK_REGISTERED in devlink_port_unregister. >> >> I think the right solution would be to add protection to devlink ports >> and likely other devlink objects with similar concurrency issues. >> >> >> Let me know what you think about it. >> >> >> >> On 9/26/22 13:09, Jiri Pirko wrote: >>> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email. >>> >>> >>> From: Jiri Pirko <jiri@nvidia.com> >>> >>> Some of the drivers use wrong order in registering devlink port and >>> netdev, registering netdev first. That was not intended as the devlink >>> port is some sort of parent for the netdev. Fix the ordering. >>> >>> Note that the follow-up patchset is going to make this ordering >>> mandatory. >>> >>> Jiri Pirko (3): >>> funeth: unregister devlink port after netdevice unregister >>> ice: reorder PF/representor devlink port register/unregister flows >>> ionic: change order of devlink port register and netdev register >>> >>> .../net/ethernet/fungible/funeth/funeth_main.c | 2 +- >>> drivers/net/ethernet/intel/ice/ice_lib.c | 6 +++--- >>> drivers/net/ethernet/intel/ice/ice_main.c | 12 ++++++------ >>> drivers/net/ethernet/intel/ice/ice_repr.c | 2 +- >>> .../net/ethernet/pensando/ionic/ionic_bus_pci.c | 16 ++++++++-------- >>> 5 files changed, 19 insertions(+), 19 deletions(-) >>> >>> -- >>> 2.37.1 >>>
Wed, Oct 05, 2022 at 10:18:29AM CEST, alejandro.lucero-palau@amd.com wrote: > >On 10/5/22 09:49, Jiri Pirko wrote: >> Tue, Oct 04, 2022 at 05:31:10PM CEST, alejandro.lucero-palau@amd.com wrote: >>> Hi Jiri, >> I don't understand why you send this as a reply to this patchset. I >> don't see the relation to it. > >I thought there was a relationship with ordering being the issue. > >Apologies if this is not the right way for rising my concern. > > >> >>> I think we have another issue with devlink_unregister and related >>> devlink_port_unregister. It is likely not an issue with current drivers >>> because the devlink ports are managed by netdev register/unregister >>> code, and with your patch that will be fine. >>> >>> But by definition, devlink does exist for those things not matching >>> smoothly to netdevs, so it is expected devlink ports not related to >>> existing netdevs at all. That is the case in a patch I'm working on for >>> sfc ef100, where devlink ports are created at PF initialization, so >>> related netdevs will not be there at that point, and they can not exist >>> when the devlink ports are removed when the driver is removed. >>> >>> So the question in this case is, should the devlink ports unregister >>> before or after their devlink unregisters? >> Before. If devlink instance should be unregistered only after all other >> related instances are gone. >> >> Also, the devlink ports come and go during the devlink lifetime. When >> you add a VF, split a port for example. There are many other cases. >> >> >>> Since the ports are in a list owned by the devlink struct, I think it >>> seems logical to unregister the ports first, and that is what I did. It >>> works but there exists a potential concurrency issue with devlink user >> What concurrency issue are you talking about? >> >1) devlink port function set ... > >2) predoit inside devlink obtains devlink then the reference to devlink >port. Code does a put on devlink but not on the devlink port. devl_lock is taken here. > >3) driver is removed. devlink port is removed. devlink is not because devl_lock taken before port is removed and will block there. I don't see any problem. Did you actually encoutered any problem? >the put. > >4) devlink port reference is wrong. > > >>> space operations. The devlink code takes care of race conditions involving the >>> devlink struct with rcu plus get/put operations, but that is not the >>> case for devlink ports. >>> >>> Interestingly, unregistering the devlink first, and doing so with the >>> ports without touching/releasing the devlink struct would solve the >>> problem, but not sure this is the right approach here. It does not seem >> It is not. As I wrote above, the devlink ports come and go. >> >> >>> clean, and it would require documenting the right unwinding order and >>> to add a check for DEVLINK_REGISTERED in devlink_port_unregister. >>> >>> I think the right solution would be to add protection to devlink ports >>> and likely other devlink objects with similar concurrency issues. >>> >>> >>> Let me know what you think about it. >>> >>> >>> >>> On 9/26/22 13:09, Jiri Pirko wrote: >>>> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email. >>>> >>>> >>>> From: Jiri Pirko <jiri@nvidia.com> >>>> >>>> Some of the drivers use wrong order in registering devlink port and >>>> netdev, registering netdev first. That was not intended as the devlink >>>> port is some sort of parent for the netdev. Fix the ordering. >>>> >>>> Note that the follow-up patchset is going to make this ordering >>>> mandatory. >>>> >>>> Jiri Pirko (3): >>>> funeth: unregister devlink port after netdevice unregister >>>> ice: reorder PF/representor devlink port register/unregister flows >>>> ionic: change order of devlink port register and netdev register >>>> >>>> .../net/ethernet/fungible/funeth/funeth_main.c | 2 +- >>>> drivers/net/ethernet/intel/ice/ice_lib.c | 6 +++--- >>>> drivers/net/ethernet/intel/ice/ice_main.c | 12 ++++++------ >>>> drivers/net/ethernet/intel/ice/ice_repr.c | 2 +- >>>> .../net/ethernet/pensando/ionic/ionic_bus_pci.c | 16 ++++++++-------- >>>> 5 files changed, 19 insertions(+), 19 deletions(-) >>>> >>>> -- >>>> 2.37.1 >>>> >
On 10/6/22 14:44, Jiri Pirko wrote: > Wed, Oct 05, 2022 at 10:18:29AM CEST, alejandro.lucero-palau@amd.com wrote: >> On 10/5/22 09:49, Jiri Pirko wrote: >>> Tue, Oct 04, 2022 at 05:31:10PM CEST, alejandro.lucero-palau@amd.com wrote: >>>> Hi Jiri, >>> I don't understand why you send this as a reply to this patchset. I >>> don't see the relation to it. >> I thought there was a relationship with ordering being the issue. >> >> Apologies if this is not the right way for rising my concern. >> >> >>>> I think we have another issue with devlink_unregister and related >>>> devlink_port_unregister. It is likely not an issue with current drivers >>>> because the devlink ports are managed by netdev register/unregister >>>> code, and with your patch that will be fine. >>>> >>>> But by definition, devlink does exist for those things not matching >>>> smoothly to netdevs, so it is expected devlink ports not related to >>>> existing netdevs at all. That is the case in a patch I'm working on for >>>> sfc ef100, where devlink ports are created at PF initialization, so >>>> related netdevs will not be there at that point, and they can not exist >>>> when the devlink ports are removed when the driver is removed. >>>> >>>> So the question in this case is, should the devlink ports unregister >>>> before or after their devlink unregisters? >>> Before. If devlink instance should be unregistered only after all other >>> related instances are gone. >>> >>> Also, the devlink ports come and go during the devlink lifetime. When >>> you add a VF, split a port for example. There are many other cases. >>> >>> >>>> Since the ports are in a list owned by the devlink struct, I think it >>>> seems logical to unregister the ports first, and that is what I did. It >>>> works but there exists a potential concurrency issue with devlink user >>> What concurrency issue are you talking about? >>> >> 1) devlink port function set ... >> >> 2) predoit inside devlink obtains devlink then the reference to devlink >> port. Code does a put on devlink but not on the devlink port. > devl_lock is taken here. This is embarrassing. Somehow I misread the code assuming the protection was only based on the get operation, that the devlink lock was released there and not in the post_doit. That goto unlock confused me, I guess, along with a bias looking for ordering issues. Apologies. Happy to see all is fine. Thank you. > >> 3) driver is removed. devlink port is removed. devlink is not because > devl_lock taken before port is removed and will block there. > > I don't see any problem. Did you actually encoutered any problem? > > >> the put. >> >> 4) devlink port reference is wrong. >> >> >>>> space operations. The devlink code takes care of race conditions involving the >>>> devlink struct with rcu plus get/put operations, but that is not the >>>> case for devlink ports. >>>> >>>> Interestingly, unregistering the devlink first, and doing so with the >>>> ports without touching/releasing the devlink struct would solve the >>>> problem, but not sure this is the right approach here. It does not seem >>> It is not. As I wrote above, the devlink ports come and go. >>> >>> >>>> clean, and it would require documenting the right unwinding order and >>>> to add a check for DEVLINK_REGISTERED in devlink_port_unregister. >>>> >>>> I think the right solution would be to add protection to devlink ports >>>> and likely other devlink objects with similar concurrency issues. >>>> >>>> >>>> Let me know what you think about it. >>>> >>>> >>>> >>>> On 9/26/22 13:09, Jiri Pirko wrote: >>>>> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email. >>>>> >>>>> >>>>> From: Jiri Pirko <jiri@nvidia.com> >>>>> >>>>> Some of the drivers use wrong order in registering devlink port and >>>>> netdev, registering netdev first. That was not intended as the devlink >>>>> port is some sort of parent for the netdev. Fix the ordering. >>>>> >>>>> Note that the follow-up patchset is going to make this ordering >>>>> mandatory. >>>>> >>>>> Jiri Pirko (3): >>>>> funeth: unregister devlink port after netdevice unregister >>>>> ice: reorder PF/representor devlink port register/unregister flows >>>>> ionic: change order of devlink port register and netdev register >>>>> >>>>> .../net/ethernet/fungible/funeth/funeth_main.c | 2 +- >>>>> drivers/net/ethernet/intel/ice/ice_lib.c | 6 +++--- >>>>> drivers/net/ethernet/intel/ice/ice_main.c | 12 ++++++------ >>>>> drivers/net/ethernet/intel/ice/ice_repr.c | 2 +- >>>>> .../net/ethernet/pensando/ionic/ionic_bus_pci.c | 16 ++++++++-------- >>>>> 5 files changed, 19 insertions(+), 19 deletions(-) >>>>> >>>>> -- >>>>> 2.37.1 >>>>>
Thu, Oct 06, 2022 at 03:45:48PM CEST, alejandro.lucero-palau@amd.com wrote: > >On 10/6/22 14:44, Jiri Pirko wrote: >> Wed, Oct 05, 2022 at 10:18:29AM CEST, alejandro.lucero-palau@amd.com wrote: >>> On 10/5/22 09:49, Jiri Pirko wrote: >>>> Tue, Oct 04, 2022 at 05:31:10PM CEST, alejandro.lucero-palau@amd.com wrote: >>>>> Hi Jiri, >>>> I don't understand why you send this as a reply to this patchset. I >>>> don't see the relation to it. >>> I thought there was a relationship with ordering being the issue. >>> >>> Apologies if this is not the right way for rising my concern. >>> >>> >>>>> I think we have another issue with devlink_unregister and related >>>>> devlink_port_unregister. It is likely not an issue with current drivers >>>>> because the devlink ports are managed by netdev register/unregister >>>>> code, and with your patch that will be fine. >>>>> >>>>> But by definition, devlink does exist for those things not matching >>>>> smoothly to netdevs, so it is expected devlink ports not related to >>>>> existing netdevs at all. That is the case in a patch I'm working on for >>>>> sfc ef100, where devlink ports are created at PF initialization, so >>>>> related netdevs will not be there at that point, and they can not exist >>>>> when the devlink ports are removed when the driver is removed. >>>>> >>>>> So the question in this case is, should the devlink ports unregister >>>>> before or after their devlink unregisters? >>>> Before. If devlink instance should be unregistered only after all other >>>> related instances are gone. >>>> >>>> Also, the devlink ports come and go during the devlink lifetime. When >>>> you add a VF, split a port for example. There are many other cases. >>>> >>>> >>>>> Since the ports are in a list owned by the devlink struct, I think it >>>>> seems logical to unregister the ports first, and that is what I did. It >>>>> works but there exists a potential concurrency issue with devlink user >>>> What concurrency issue are you talking about? >>>> >>> 1) devlink port function set ... >>> >>> 2) predoit inside devlink obtains devlink then the reference to devlink >>> port. Code does a put on devlink but not on the devlink port. >> devl_lock is taken here. > >This is embarrassing. > >Somehow I misread the code assuming the protection was only based on the >get operation, that the devlink lock was released there and not in the >post_doit. > >That goto unlock confused me, I guess, along with a bias looking for >ordering issues. > >Apologies. Np :) Happy to help. > >Happy to see all is fine. > >Thank you. > >> >>> 3) driver is removed. devlink port is removed. devlink is not because >> devl_lock taken before port is removed and will block there. >> >> I don't see any problem. Did you actually encoutered any problem? >> >> >>> the put. >>> >>> 4) devlink port reference is wrong. >>> >>> >>>>> space operations. The devlink code takes care of race conditions involving the >>>>> devlink struct with rcu plus get/put operations, but that is not the >>>>> case for devlink ports. >>>>> >>>>> Interestingly, unregistering the devlink first, and doing so with the >>>>> ports without touching/releasing the devlink struct would solve the >>>>> problem, but not sure this is the right approach here. It does not seem >>>> It is not. As I wrote above, the devlink ports come and go. >>>> >>>> >>>>> clean, and it would require documenting the right unwinding order and >>>>> to add a check for DEVLINK_REGISTERED in devlink_port_unregister. >>>>> >>>>> I think the right solution would be to add protection to devlink ports >>>>> and likely other devlink objects with similar concurrency issues. >>>>> >>>>> >>>>> Let me know what you think about it. >>>>> >>>>> >>>>> >>>>> On 9/26/22 13:09, Jiri Pirko wrote: >>>>>> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email. >>>>>> >>>>>> >>>>>> From: Jiri Pirko <jiri@nvidia.com> >>>>>> >>>>>> Some of the drivers use wrong order in registering devlink port and >>>>>> netdev, registering netdev first. That was not intended as the devlink >>>>>> port is some sort of parent for the netdev. Fix the ordering. >>>>>> >>>>>> Note that the follow-up patchset is going to make this ordering >>>>>> mandatory. >>>>>> >>>>>> Jiri Pirko (3): >>>>>> funeth: unregister devlink port after netdevice unregister >>>>>> ice: reorder PF/representor devlink port register/unregister flows >>>>>> ionic: change order of devlink port register and netdev register >>>>>> >>>>>> .../net/ethernet/fungible/funeth/funeth_main.c | 2 +- >>>>>> drivers/net/ethernet/intel/ice/ice_lib.c | 6 +++--- >>>>>> drivers/net/ethernet/intel/ice/ice_main.c | 12 ++++++------ >>>>>> drivers/net/ethernet/intel/ice/ice_repr.c | 2 +- >>>>>> .../net/ethernet/pensando/ionic/ionic_bus_pci.c | 16 ++++++++-------- >>>>>> 5 files changed, 19 insertions(+), 19 deletions(-) >>>>>> >>>>>> -- >>>>>> 2.37.1 >>>>>> >
From: Jiri Pirko <jiri@nvidia.com> Some of the drivers use wrong order in registering devlink port and netdev, registering netdev first. That was not intended as the devlink port is some sort of parent for the netdev. Fix the ordering. Note that the follow-up patchset is going to make this ordering mandatory. Jiri Pirko (3): funeth: unregister devlink port after netdevice unregister ice: reorder PF/representor devlink port register/unregister flows ionic: change order of devlink port register and netdev register .../net/ethernet/fungible/funeth/funeth_main.c | 2 +- drivers/net/ethernet/intel/ice/ice_lib.c | 6 +++--- drivers/net/ethernet/intel/ice/ice_main.c | 12 ++++++------ drivers/net/ethernet/intel/ice/ice_repr.c | 2 +- .../net/ethernet/pensando/ionic/ionic_bus_pci.c | 16 ++++++++-------- 5 files changed, 19 insertions(+), 19 deletions(-)