| Message ID | 1546335025-31360-2-git-send-email-galpress@amazon.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | IB device in-kernel API support indication | expand |
On Tue, Jan 01, 2019 at 11:30:24AM +0200, Gal Pressman wrote: > Drivers that do not provide kernel verbs support should communicate that > to the ULPs before they try to use the device and fail. > This patch allows drivers to indicate whether the device provides > support for kernel API usage with the 'kverbs_provider' flag and makes > ib clients not add unsupported devices. General comment, if we decide to proceed this path, the implementation shouldn't touch ULPs and drivers and needs to be implemented in IB/core only. Thanks > > No functional changes were introduced in this patch, all devices are > marked as kernel providers. Downstream patch will adjust relevant > drivers.
On 01-Jan-19 21:17, Leon Romanovsky wrote: > On Tue, Jan 01, 2019 at 11:30:24AM +0200, Gal Pressman wrote: >> Drivers that do not provide kernel verbs support should communicate that >> to the ULPs before they try to use the device and fail. >> This patch allows drivers to indicate whether the device provides >> support for kernel API usage with the 'kverbs_provider' flag and makes >> ib clients not add unsupported devices. > > General comment, if we decide to proceed this path, the implementation > shouldn't touch ULPs and drivers and needs to be implemented in IB/core > only. > > Thanks Hi Leon, How would that work? Each ULP should decide whether it wants to add the device or not, shouldn't that be coded in the ULP? > >> >> No functional changes were introduced in this patch, all devices are >> marked as kernel providers. Downstream patch will adjust relevant >> drivers.
On Wed, Jan 02, 2019 at 10:40:50AM +0200, Gal Pressman wrote: > On 01-Jan-19 21:17, Leon Romanovsky wrote: > > On Tue, Jan 01, 2019 at 11:30:24AM +0200, Gal Pressman wrote: > >> Drivers that do not provide kernel verbs support should communicate that > >> to the ULPs before they try to use the device and fail. > >> This patch allows drivers to indicate whether the device provides > >> support for kernel API usage with the 'kverbs_provider' flag and makes > >> ib clients not add unsupported devices. > > > > General comment, if we decide to proceed this path, the implementation > > shouldn't touch ULPs and drivers and needs to be implemented in IB/core > > only. > > > > Thanks > > Hi Leon, > How would that work? > Each ULP should decide whether it wants to add the device or not, shouldn't that > be coded in the ULP? Something like that, diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c index ac011836bb54..b2d307d3c193 100644 --- a/drivers/infiniband/core/verbs.c +++ b/drivers/infiniband/core/verbs.c @@ -512,6 +512,13 @@ static struct ib_ah *_rdma_create_ah(struct ib_pd *pd, return ah; } +static bool is_kverbs_supported(struct ib_device *dev) +{ + if (unlikely(dev->driver_id == RDMA_DRIVER_USNIC)) + return false; + return true; +} + /** * rdma_create_ah - Creates an address handle for the * given address vector. @@ -530,6 +537,9 @@ struct ib_ah *rdma_create_ah(struct ib_pd *pd, struct rdma_ah_attr *ah_attr, struct ib_ah *ah; int ret; + if (!is_kverbs_supported(pd->device)) + return ERR_PTR(-ENOSYS); + ret = rdma_fill_sgid_attr(pd->device, ah_attr, &old_sgid_attr); if (ret) return ERR_PTR(ret); > > > > >> > >> No functional changes were introduced in this patch, all devices are > >> marked as kernel providers. Downstream patch will adjust relevant > >> drivers. >
On 02-Jan-19 13:22, Leon Romanovsky wrote: > On Wed, Jan 02, 2019 at 10:40:50AM +0200, Gal Pressman wrote: >> On 01-Jan-19 21:17, Leon Romanovsky wrote: >>> On Tue, Jan 01, 2019 at 11:30:24AM +0200, Gal Pressman wrote: >>>> Drivers that do not provide kernel verbs support should communicate that >>>> to the ULPs before they try to use the device and fail. >>>> This patch allows drivers to indicate whether the device provides >>>> support for kernel API usage with the 'kverbs_provider' flag and makes >>>> ib clients not add unsupported devices. >>> >>> General comment, if we decide to proceed this path, the implementation >>> shouldn't touch ULPs and drivers and needs to be implemented in IB/core >>> only. >>> >>> Thanks >> >> Hi Leon, >> How would that work? >> Each ULP should decide whether it wants to add the device or not, shouldn't that >> be coded in the ULP? > > Something like that, > > diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c > index ac011836bb54..b2d307d3c193 100644 > --- a/drivers/infiniband/core/verbs.c > +++ b/drivers/infiniband/core/verbs.c > @@ -512,6 +512,13 @@ static struct ib_ah *_rdma_create_ah(struct ib_pd *pd, > return ah; > } > > +static bool is_kverbs_supported(struct ib_device *dev) > +{ > + if (unlikely(dev->driver_id == RDMA_DRIVER_USNIC)) > + return false; > + return true; > +} > + > /** > * rdma_create_ah - Creates an address handle for the > * given address vector. > @@ -530,6 +537,9 @@ struct ib_ah *rdma_create_ah(struct ib_pd *pd, struct rdma_ah_attr *ah_attr, > struct ib_ah *ah; > int ret; > > + if (!is_kverbs_supported(pd->device)) > + return ERR_PTR(-ENOSYS); > + > ret = rdma_fill_sgid_attr(pd->device, ah_attr, &old_sgid_attr); > if (ret) > return ERR_PTR(ret); > Thanks, This approach falls back to the "try and fail" way, why should the client even try to use the device if it's not supported in the first place? In the ULP's perspective create AH fails, does it really matter if the core returns the error or the driver? >> >>> >>>> >>>> No functional changes were introduced in this patch, all devices are >>>> marked as kernel providers. Downstream patch will adjust relevant >>>> drivers. >>
On Wed, Jan 02, 2019 at 01:51:29PM +0200, Gal Pressman wrote: > On 02-Jan-19 13:22, Leon Romanovsky wrote: > > On Wed, Jan 02, 2019 at 10:40:50AM +0200, Gal Pressman wrote: > >> On 01-Jan-19 21:17, Leon Romanovsky wrote: > >>> On Tue, Jan 01, 2019 at 11:30:24AM +0200, Gal Pressman wrote: > >>>> Drivers that do not provide kernel verbs support should communicate that > >>>> to the ULPs before they try to use the device and fail. > >>>> This patch allows drivers to indicate whether the device provides > >>>> support for kernel API usage with the 'kverbs_provider' flag and makes > >>>> ib clients not add unsupported devices. > >>> > >>> General comment, if we decide to proceed this path, the implementation > >>> shouldn't touch ULPs and drivers and needs to be implemented in IB/core > >>> only. > >>> > >>> Thanks > >> > >> Hi Leon, > >> How would that work? > >> Each ULP should decide whether it wants to add the device or not, shouldn't that > >> be coded in the ULP? > > > > Something like that, > > > > diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c > > index ac011836bb54..b2d307d3c193 100644 > > --- a/drivers/infiniband/core/verbs.c > > +++ b/drivers/infiniband/core/verbs.c > > @@ -512,6 +512,13 @@ static struct ib_ah *_rdma_create_ah(struct ib_pd *pd, > > return ah; > > } > > > > +static bool is_kverbs_supported(struct ib_device *dev) > > +{ > > + if (unlikely(dev->driver_id == RDMA_DRIVER_USNIC)) > > + return false; > > + return true; > > +} > > + > > /** > > * rdma_create_ah - Creates an address handle for the > > * given address vector. > > @@ -530,6 +537,9 @@ struct ib_ah *rdma_create_ah(struct ib_pd *pd, struct rdma_ah_attr *ah_attr, > > struct ib_ah *ah; > > int ret; > > > > + if (!is_kverbs_supported(pd->device)) > > + return ERR_PTR(-ENOSYS); > > + > > ret = rdma_fill_sgid_attr(pd->device, ah_attr, &old_sgid_attr); > > if (ret) > > return ERR_PTR(ret); > > > > Thanks, > This approach falls back to the "try and fail" way, why should the client even > try to use the device if it's not supported in the first place? > In the ULP's perspective create AH fails, does it really matter if the core > returns the error or the driver? It was an example, put is_kverbs_supported() into ib_register_client() and it will do the trick. Thanks > > >> > >>> > >>>> > >>>> No functional changes were introduced in this patch, all devices are > >>>> marked as kernel providers. Downstream patch will adjust relevant > >>>> drivers. > >>
On 02-Jan-19 14:39, Leon Romanovsky wrote: > On Wed, Jan 02, 2019 at 01:51:29PM +0200, Gal Pressman wrote: >> On 02-Jan-19 13:22, Leon Romanovsky wrote: >>> On Wed, Jan 02, 2019 at 10:40:50AM +0200, Gal Pressman wrote: >>>> On 01-Jan-19 21:17, Leon Romanovsky wrote: >>>>> On Tue, Jan 01, 2019 at 11:30:24AM +0200, Gal Pressman wrote: >>>>>> Drivers that do not provide kernel verbs support should communicate that >>>>>> to the ULPs before they try to use the device and fail. >>>>>> This patch allows drivers to indicate whether the device provides >>>>>> support for kernel API usage with the 'kverbs_provider' flag and makes >>>>>> ib clients not add unsupported devices. >>>>> >>>>> General comment, if we decide to proceed this path, the implementation >>>>> shouldn't touch ULPs and drivers and needs to be implemented in IB/core >>>>> only. >>>>> >>>>> Thanks >>>> >>>> Hi Leon, >>>> How would that work? >>>> Each ULP should decide whether it wants to add the device or not, shouldn't that >>>> be coded in the ULP? >>> >>> Something like that, >>> >>> diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c >>> index ac011836bb54..b2d307d3c193 100644 >>> --- a/drivers/infiniband/core/verbs.c >>> +++ b/drivers/infiniband/core/verbs.c >>> @@ -512,6 +512,13 @@ static struct ib_ah *_rdma_create_ah(struct ib_pd *pd, >>> return ah; >>> } >>> >>> +static bool is_kverbs_supported(struct ib_device *dev) >>> +{ >>> + if (unlikely(dev->driver_id == RDMA_DRIVER_USNIC)) >>> + return false; >>> + return true; >>> +} >>> + >>> /** >>> * rdma_create_ah - Creates an address handle for the >>> * given address vector. >>> @@ -530,6 +537,9 @@ struct ib_ah *rdma_create_ah(struct ib_pd *pd, struct rdma_ah_attr *ah_attr, >>> struct ib_ah *ah; >>> int ret; >>> >>> + if (!is_kverbs_supported(pd->device)) >>> + return ERR_PTR(-ENOSYS); >>> + >>> ret = rdma_fill_sgid_attr(pd->device, ah_attr, &old_sgid_attr); >>> if (ret) >>> return ERR_PTR(ret); >>> >> >> Thanks, >> This approach falls back to the "try and fail" way, why should the client even >> try to use the device if it's not supported in the first place? >> In the ULP's perspective create AH fails, does it really matter if the core >> returns the error or the driver? > > It was an example, put is_kverbs_supported() into ib_register_client() > and it will do the trick. > > Thanks I can do that, it will require an additional check of "if (client != uverbs)", that should be registered regardless of whether kverbs are supported or not. This feels off to me, but it's possible.
On Wed, Jan 02, 2019 at 03:43:13PM +0200, Gal Pressman wrote: > On 02-Jan-19 14:39, Leon Romanovsky wrote: > > On Wed, Jan 02, 2019 at 01:51:29PM +0200, Gal Pressman wrote: > >> On 02-Jan-19 13:22, Leon Romanovsky wrote: > >>> On Wed, Jan 02, 2019 at 10:40:50AM +0200, Gal Pressman wrote: > >>>> On 01-Jan-19 21:17, Leon Romanovsky wrote: > >>>>> On Tue, Jan 01, 2019 at 11:30:24AM +0200, Gal Pressman wrote: > >>>>>> Drivers that do not provide kernel verbs support should communicate that > >>>>>> to the ULPs before they try to use the device and fail. > >>>>>> This patch allows drivers to indicate whether the device provides > >>>>>> support for kernel API usage with the 'kverbs_provider' flag and makes > >>>>>> ib clients not add unsupported devices. > >>>>> > >>>>> General comment, if we decide to proceed this path, the implementation > >>>>> shouldn't touch ULPs and drivers and needs to be implemented in IB/core > >>>>> only. > >>>>> > >>>>> Thanks > >>>> > >>>> Hi Leon, > >>>> How would that work? > >>>> Each ULP should decide whether it wants to add the device or not, shouldn't that > >>>> be coded in the ULP? > >>> > >>> Something like that, > >>> > >>> diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c > >>> index ac011836bb54..b2d307d3c193 100644 > >>> --- a/drivers/infiniband/core/verbs.c > >>> +++ b/drivers/infiniband/core/verbs.c > >>> @@ -512,6 +512,13 @@ static struct ib_ah *_rdma_create_ah(struct ib_pd *pd, > >>> return ah; > >>> } > >>> > >>> +static bool is_kverbs_supported(struct ib_device *dev) > >>> +{ > >>> + if (unlikely(dev->driver_id == RDMA_DRIVER_USNIC)) > >>> + return false; > >>> + return true; > >>> +} > >>> + > >>> /** > >>> * rdma_create_ah - Creates an address handle for the > >>> * given address vector. > >>> @@ -530,6 +537,9 @@ struct ib_ah *rdma_create_ah(struct ib_pd *pd, struct rdma_ah_attr *ah_attr, > >>> struct ib_ah *ah; > >>> int ret; > >>> > >>> + if (!is_kverbs_supported(pd->device)) > >>> + return ERR_PTR(-ENOSYS); > >>> + > >>> ret = rdma_fill_sgid_attr(pd->device, ah_attr, &old_sgid_attr); > >>> if (ret) > >>> return ERR_PTR(ret); > >>> > >> > >> Thanks, > >> This approach falls back to the "try and fail" way, why should the client even > >> try to use the device if it's not supported in the first place? > >> In the ULP's perspective create AH fails, does it really matter if the core > >> returns the error or the driver? > > > > It was an example, put is_kverbs_supported() into ib_register_client() > > and it will do the trick. > > > > Thanks > > I can do that, it will require an additional check of > "if (client != uverbs)", that should be registered regardless of whether kverbs > are supported or not. > This feels off to me, but it's possible. introduce ib_register_client_nocheck() and use it for uverbs. Anyway, it is better to close main thing raised in cover letter, before rushing and doing extra version. Thanks
> -----Original Message----- > From: linux-rdma-owner@vger.kernel.org <linux-rdma- > owner@vger.kernel.org> On Behalf Of Leon Romanovsky > Sent: Wednesday, January 2, 2019 7:24 PM > To: Gal Pressman <galpress@amazon.com> > Cc: Jason Gunthorpe <jgg@ziepe.ca>; Doug Ledford <dledford@redhat.com>; > linux-rdma@vger.kernel.org; Alexander Matushevsky > <matua@amazon.com>; Yossi Leybovich <sleybo@amazon.com>; Dave > Goodell <goodell@amazon.com>; Brian Barrett <bbarrett@amazon.com>; > Leah Shalev <shalevl@amazon.com>; Sean Hefty <sean.hefty@intel.com> > Subject: Re: [PATCH RFC 1/2] RDMA: Add indication for in kernel API support > to IB device > > On Wed, Jan 02, 2019 at 03:43:13PM +0200, Gal Pressman wrote: > > On 02-Jan-19 14:39, Leon Romanovsky wrote: > > > On Wed, Jan 02, 2019 at 01:51:29PM +0200, Gal Pressman wrote: > > >> On 02-Jan-19 13:22, Leon Romanovsky wrote: > > >>> On Wed, Jan 02, 2019 at 10:40:50AM +0200, Gal Pressman wrote: > > >>>> On 01-Jan-19 21:17, Leon Romanovsky wrote: > > >>>>> On Tue, Jan 01, 2019 at 11:30:24AM +0200, Gal Pressman wrote: > > >>>>>> Drivers that do not provide kernel verbs support should > > >>>>>> communicate that to the ULPs before they try to use the device > and fail. > > >>>>>> This patch allows drivers to indicate whether the device > > >>>>>> provides support for kernel API usage with the > > >>>>>> 'kverbs_provider' flag and makes ib clients not add unsupported > devices. > > >>>>> > > >>>>> General comment, if we decide to proceed this path, the > > >>>>> implementation shouldn't touch ULPs and drivers and needs to be > > >>>>> implemented in IB/core only. > > >>>>> > > >>>>> Thanks > > >>>> > > >>>> Hi Leon, > > >>>> How would that work? > > >>>> Each ULP should decide whether it wants to add the device or not, > > >>>> shouldn't that be coded in the ULP? > > >>> > > >>> Something like that, > > >>> > > >>> diff --git a/drivers/infiniband/core/verbs.c > > >>> b/drivers/infiniband/core/verbs.c index ac011836bb54..b2d307d3c193 > > >>> 100644 > > >>> --- a/drivers/infiniband/core/verbs.c > > >>> +++ b/drivers/infiniband/core/verbs.c > > >>> @@ -512,6 +512,13 @@ static struct ib_ah *_rdma_create_ah(struct > ib_pd *pd, > > >>> return ah; > > >>> } > > >>> > > >>> +static bool is_kverbs_supported(struct ib_device *dev) { > > >>> + if (unlikely(dev->driver_id == RDMA_DRIVER_USNIC)) > > >>> + return false; > > >>> + return true; > > >>> +} > > >>> + > > >>> /** > > >>> * rdma_create_ah - Creates an address handle for the > > >>> * given address vector. > > >>> @@ -530,6 +537,9 @@ struct ib_ah *rdma_create_ah(struct ib_pd > *pd, struct rdma_ah_attr *ah_attr, > > >>> struct ib_ah *ah; > > >>> int ret; > > >>> > > >>> + if (!is_kverbs_supported(pd->device)) > > >>> + return ERR_PTR(-ENOSYS); > > >>> + > > >>> ret = rdma_fill_sgid_attr(pd->device, ah_attr, &old_sgid_attr); > > >>> if (ret) > > >>> return ERR_PTR(ret); > > >>> > > >> > > >> Thanks, > > >> This approach falls back to the "try and fail" way, why should the > > >> client even try to use the device if it's not supported in the first place? > > >> In the ULP's perspective create AH fails, does it really matter if > > >> the core returns the error or the driver? > > > > > > It was an example, put is_kverbs_supported() into > > > ib_register_client() and it will do the trick. > > > > > > Thanks > > > > I can do that, it will require an additional check of "if (client != > > uverbs)", that should be registered regardless of whether kverbs are > > supported or not. > > This feels off to me, but it's possible. > > introduce ib_register_client_nocheck() and use it for uverbs. > > Anyway, it is better to close main thing raised in cover letter, before rushing > and doing extra version. Not to miss out the need of an additional link layer type.
On Wed, Jan 02, 2019 at 10:40:50AM +0200, Gal Pressman wrote: > On 01-Jan-19 21:17, Leon Romanovsky wrote: > > On Tue, Jan 01, 2019 at 11:30:24AM +0200, Gal Pressman wrote: > >> Drivers that do not provide kernel verbs support should communicate that > >> to the ULPs before they try to use the device and fail. > >> This patch allows drivers to indicate whether the device provides > >> support for kernel API usage with the 'kverbs_provider' flag and makes > >> ib clients not add unsupported devices. > > > > General comment, if we decide to proceed this path, the implementation > > shouldn't touch ULPs and drivers and needs to be implemented in IB/core > > only. > > > > Thanks > > Hi Leon, > How would that work? The core code should not call registered client callbacks for non verbs device at all. Jason
diff --git a/drivers/infiniband/core/cm.c b/drivers/infiniband/core/cm.c index 37980c7564c0..e704100b78ea 100644 --- a/drivers/infiniband/core/cm.c +++ b/drivers/infiniband/core/cm.c @@ -4360,6 +4360,9 @@ static void cm_add_one(struct ib_device *ib_device) int count = 0; u8 i; + if (!ib_device->kverbs_provider) + return; + cm_dev = kzalloc(struct_size(cm_dev, port, ib_device->phys_port_cnt), GFP_KERNEL); if (!cm_dev) diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c index 63a7cc00bae0..535347c4bbcc 100644 --- a/drivers/infiniband/core/cma.c +++ b/drivers/infiniband/core/cma.c @@ -4481,6 +4481,9 @@ static void cma_add_one(struct ib_device *device) unsigned int i; unsigned long supported_gids = 0; + if (!device->kverbs_provider) + return; + cma_dev = kmalloc(sizeof *cma_dev, GFP_KERNEL); if (!cma_dev) return; diff --git a/drivers/infiniband/core/mad.c b/drivers/infiniband/core/mad.c index 7870823bac47..58517f37f9cf 100644 --- a/drivers/infiniband/core/mad.c +++ b/drivers/infiniband/core/mad.c @@ -3289,6 +3289,9 @@ static void ib_mad_init_device(struct ib_device *device) { int start, i; + if (!device->kverbs_provider) + return; + start = rdma_start_port(device); for (i = start; i <= rdma_end_port(device); i++) { diff --git a/drivers/infiniband/core/multicast.c b/drivers/infiniband/core/multicast.c index d50ff70bb24b..863c2d72b1f0 100644 --- a/drivers/infiniband/core/multicast.c +++ b/drivers/infiniband/core/multicast.c @@ -823,6 +823,9 @@ static void mcast_add_one(struct ib_device *device) int i; int count = 0; + if (!device->kverbs_provider) + return; + dev = kmalloc(struct_size(dev, port, device->phys_port_cnt), GFP_KERNEL); if (!dev) diff --git a/drivers/infiniband/core/sa_query.c b/drivers/infiniband/core/sa_query.c index 97e6d7b69abf..fe563bedbb7e 100644 --- a/drivers/infiniband/core/sa_query.c +++ b/drivers/infiniband/core/sa_query.c @@ -2339,6 +2339,9 @@ static void ib_sa_add_one(struct ib_device *device) int s, e, i; int count = 0; + if (!device->kverbs_provider) + return; + s = rdma_start_port(device); e = rdma_end_port(device); diff --git a/drivers/infiniband/core/ucm.c b/drivers/infiniband/core/ucm.c index 7541fbaf58a3..592b1281fefe 100644 --- a/drivers/infiniband/core/ucm.c +++ b/drivers/infiniband/core/ucm.c @@ -1242,7 +1242,8 @@ static void ib_ucm_add_one(struct ib_device *device) dev_t base; struct ib_ucm_device *ucm_dev; - if (!device->ops.alloc_ucontext || !rdma_cap_ib_cm(device, 1)) + if (!device->ops.alloc_ucontext || !rdma_cap_ib_cm(device, 1) || + !device->kverbs_provider) return; ucm_dev = kzalloc(sizeof *ucm_dev, GFP_KERNEL); diff --git a/drivers/infiniband/core/user_mad.c b/drivers/infiniband/core/user_mad.c index de8d31ab8945..79f91e9bd192 100644 --- a/drivers/infiniband/core/user_mad.c +++ b/drivers/infiniband/core/user_mad.c @@ -1295,6 +1295,9 @@ static void ib_umad_add_one(struct ib_device *device) int s, e, i; int count = 0; + if (!device->kverbs_provider) + return; + s = rdma_start_port(device); e = rdma_end_port(device); diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index e7a997f2a537..0348f879c3b6 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -661,6 +661,7 @@ static int bnxt_re_register_ib(struct bnxt_re_dev *rdev) rdma_set_device_sysfs_group(ibdev, &bnxt_re_dev_attr_group); ibdev->driver_id = RDMA_DRIVER_BNXT_RE; + ibdev->kverbs_provider = 1; ib_set_device_ops(ibdev, &bnxt_re_dev_ops); return ib_register_device(ibdev, "bnxt_re%d", NULL); } diff --git a/drivers/infiniband/hw/cxgb3/iwch_provider.c b/drivers/infiniband/hw/cxgb3/iwch_provider.c index b34b1a1bd94b..6f3ff0355e82 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_provider.c +++ b/drivers/infiniband/hw/cxgb3/iwch_provider.c @@ -1407,6 +1407,7 @@ int iwch_register_device(struct iwch_dev *dev) sizeof(dev->ibdev.iwcm->ifname)); dev->ibdev.driver_id = RDMA_DRIVER_CXGB3; + dev->ibdev.kverbs_provider = 1; rdma_set_device_sysfs_group(&dev->ibdev, &iwch_attr_group); ib_set_device_ops(&dev->ibdev, &iwch_dev_ops); ret = ib_register_device(&dev->ibdev, "cxgb3_%d", NULL); diff --git a/drivers/infiniband/hw/cxgb4/provider.c b/drivers/infiniband/hw/cxgb4/provider.c index 586b0c37481f..f6868aa66b45 100644 --- a/drivers/infiniband/hw/cxgb4/provider.c +++ b/drivers/infiniband/hw/cxgb4/provider.c @@ -633,6 +633,7 @@ void c4iw_register_device(struct work_struct *work) rdma_set_device_sysfs_group(&dev->ibdev, &c4iw_attr_group); dev->ibdev.driver_id = RDMA_DRIVER_CXGB4; + dev->ibdev.kverbs_provider = 1; ib_set_device_ops(&dev->ibdev, &c4iw_dev_ops); ret = ib_register_device(&dev->ibdev, "cxgb4_%d", NULL); if (ret) diff --git a/drivers/infiniband/hw/hns/hns_roce_main.c b/drivers/infiniband/hw/hns/hns_roce_main.c index c79054ba9495..4052656086e2 100644 --- a/drivers/infiniband/hw/hns/hns_roce_main.c +++ b/drivers/infiniband/hw/hns/hns_roce_main.c @@ -562,6 +562,7 @@ static int hns_roce_register_device(struct hns_roce_dev *hr_dev) } ib_dev->driver_id = RDMA_DRIVER_HNS; + ib_dev->kverbs_provider = 1; ib_set_device_ops(ib_dev, hr_dev->hw->hns_roce_dev_ops); ib_set_device_ops(ib_dev, &hns_roce_dev_ops); ret = ib_register_device(ib_dev, "hns_%d", NULL); diff --git a/drivers/infiniband/hw/i40iw/i40iw_verbs.c b/drivers/infiniband/hw/i40iw/i40iw_verbs.c index 0b675b0742c2..da6b2137ec49 100644 --- a/drivers/infiniband/hw/i40iw/i40iw_verbs.c +++ b/drivers/infiniband/hw/i40iw/i40iw_verbs.c @@ -2868,6 +2868,7 @@ int i40iw_register_rdma_device(struct i40iw_device *iwdev) iwibdev = iwdev->iwibdev; rdma_set_device_sysfs_group(&iwibdev->ibdev, &i40iw_attr_group); iwibdev->ibdev.driver_id = RDMA_DRIVER_I40IW; + iwibdev->ibdev.kverbs_provider = 1; ret = ib_register_device(&iwibdev->ibdev, "i40iw%d", NULL); if (ret) goto error; diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c index 1f15ec3e2b83..ac371e112b22 100644 --- a/drivers/infiniband/hw/mlx4/main.c +++ b/drivers/infiniband/hw/mlx4/main.c @@ -2856,6 +2856,7 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) rdma_set_device_sysfs_group(&ibdev->ib_dev, &mlx4_attr_group); ibdev->ib_dev.driver_id = RDMA_DRIVER_MLX4; + ibdev->ib_dev.kverbs_provider = 1; if (ib_register_device(&ibdev->ib_dev, "mlx4_%d", NULL)) goto err_diag_counters; diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c index a57183413e8b..18c4f6ee36a2 100644 --- a/drivers/infiniband/hw/mlx5/main.c +++ b/drivers/infiniband/hw/mlx5/main.c @@ -5992,6 +5992,7 @@ int mlx5_ib_stage_caps_init(struct mlx5_ib_dev *dev) MLX5_ACCEL_IPSEC_CAP_DEVICE) ib_set_device_ops(&dev->ib_dev, &mlx5_ib_dev_flow_ipsec_ops); dev->ib_dev.driver_id = RDMA_DRIVER_MLX5; + dev->ib_dev.kverbs_provider = 1; ib_set_device_ops(&dev->ib_dev, &mlx5_ib_dev_ops); if (IS_ENABLED(CONFIG_INFINIBAND_USER_ACCESS)) diff --git a/drivers/infiniband/hw/mthca/mthca_provider.c b/drivers/infiniband/hw/mthca/mthca_provider.c index 82cb6b71ac7c..d4e9f50f5d8f 100644 --- a/drivers/infiniband/hw/mthca/mthca_provider.c +++ b/drivers/infiniband/hw/mthca/mthca_provider.c @@ -1338,6 +1338,7 @@ int mthca_register_device(struct mthca_dev *dev) rdma_set_device_sysfs_group(&dev->ib_dev, &mthca_attr_group); dev->ib_dev.driver_id = RDMA_DRIVER_MTHCA; + dev->ib_dev.kverbs_provider = 1; ret = ib_register_device(&dev->ib_dev, "mthca%d", NULL); if (ret) return ret; diff --git a/drivers/infiniband/hw/nes/nes_verbs.c b/drivers/infiniband/hw/nes/nes_verbs.c index 4e7f08ee1907..8cc01bce530f 100644 --- a/drivers/infiniband/hw/nes/nes_verbs.c +++ b/drivers/infiniband/hw/nes/nes_verbs.c @@ -3801,6 +3801,7 @@ int nes_register_ofa_device(struct nes_ib_device *nesibdev) rdma_set_device_sysfs_group(&nesvnic->nesibdev->ibdev, &nes_attr_group); nesvnic->nesibdev->ibdev.driver_id = RDMA_DRIVER_NES; + nesvnic->nesibdev->ibdev.kverbs_provider = 1; ret = ib_register_device(&nesvnic->nesibdev->ibdev, "nes%d", NULL); if (ret) { return ret; diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_main.c b/drivers/infiniband/hw/ocrdma/ocrdma_main.c index 1f393842453a..0df688e9fd47 100644 --- a/drivers/infiniband/hw/ocrdma/ocrdma_main.c +++ b/drivers/infiniband/hw/ocrdma/ocrdma_main.c @@ -243,6 +243,7 @@ static int ocrdma_register_device(struct ocrdma_dev *dev) } rdma_set_device_sysfs_group(&dev->ibdev, &ocrdma_attr_group); dev->ibdev.driver_id = RDMA_DRIVER_OCRDMA; + dev->ibdev.kverbs_provider = 1; return ib_register_device(&dev->ibdev, "ocrdma%d", NULL); } diff --git a/drivers/infiniband/hw/qedr/main.c b/drivers/infiniband/hw/qedr/main.c index 75940e2a8791..24b526142923 100644 --- a/drivers/infiniband/hw/qedr/main.c +++ b/drivers/infiniband/hw/qedr/main.c @@ -290,6 +290,7 @@ static int qedr_register_device(struct qedr_dev *dev) ib_set_device_ops(&dev->ibdev, &qedr_dev_ops); dev->ibdev.driver_id = RDMA_DRIVER_QEDR; + dev->ibdev.kverbs_provider = 1; return ib_register_device(&dev->ibdev, "qedr%d", NULL); } diff --git a/drivers/infiniband/hw/usnic/usnic_ib_main.c b/drivers/infiniband/hw/usnic/usnic_ib_main.c index b2323a52a0dd..01ce45d72e68 100644 --- a/drivers/infiniband/hw/usnic/usnic_ib_main.c +++ b/drivers/infiniband/hw/usnic/usnic_ib_main.c @@ -420,6 +420,7 @@ static void *usnic_ib_device_add(struct pci_dev *dev) ib_set_device_ops(&us_ibdev->ib_dev, &usnic_dev_ops); us_ibdev->ib_dev.driver_id = RDMA_DRIVER_USNIC; + us_ibdev->ib_dev.kverbs_provider = 1; rdma_set_device_sysfs_group(&us_ibdev->ib_dev, &usnic_attr_group); if (ib_register_device(&us_ibdev->ib_dev, "usnic_%d", NULL)) diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c index eaa109dbc96a..6a4314e59d2c 100644 --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c @@ -275,6 +275,7 @@ static int pvrdma_register_device(struct pvrdma_dev *dev) goto err_qp_free; } dev->ib_dev.driver_id = RDMA_DRIVER_VMW_PVRDMA; + dev->ib_dev.kverbs_provider = 1; spin_lock_init(&dev->srq_tbl_lock); rdma_set_device_sysfs_group(&dev->ib_dev, &pvrdma_attr_group); diff --git a/drivers/infiniband/sw/rdmavt/vt.c b/drivers/infiniband/sw/rdmavt/vt.c index aef3aa3fe667..d32c5057f998 100644 --- a/drivers/infiniband/sw/rdmavt/vt.c +++ b/drivers/infiniband/sw/rdmavt/vt.c @@ -643,6 +643,7 @@ int rvt_register_device(struct rvt_dev_info *rdi, u32 driver_id) rdi->ibdev.num_comp_vectors = 1; rdi->ibdev.driver_id = driver_id; + rdi->ibdev.kverbs_provider = 1; /* We are now good to announce we exist */ ret = ib_register_device(&rdi->ibdev, dev_name(&rdi->ibdev.dev), rdi->driver_f.port_callback); diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index b20e6e0415f5..73d705cb5d1c 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -1258,6 +1258,7 @@ int rxe_register_device(struct rxe_dev *rxe) rdma_set_device_sysfs_group(dev, &rxe_attr_group); dev->driver_id = RDMA_DRIVER_RXE; + dev->kverbs_provider = 1; err = ib_register_device(dev, "rxe%d", NULL); if (err) { pr_warn("%s failed with error %d\n", __func__, err); diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c index 5224c42f9d08..835b265c0345 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_main.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c @@ -2498,6 +2498,9 @@ static void ipoib_add_one(struct ib_device *device) int p; int count = 0; + if (!device->kverbs_provider) + return; + dev_list = kmalloc(sizeof(*dev_list), GFP_KERNEL); if (!dev_list) return; diff --git a/drivers/infiniband/ulp/opa_vnic/opa_vnic_vema.c b/drivers/infiniband/ulp/opa_vnic/opa_vnic_vema.c index 560e4f2d466e..76176e50b78f 100644 --- a/drivers/infiniband/ulp/opa_vnic/opa_vnic_vema.c +++ b/drivers/infiniband/ulp/opa_vnic/opa_vnic_vema.c @@ -1010,7 +1010,7 @@ static void opa_vnic_vema_add_one(struct ib_device *device) struct opa_vnic_ctrl_port *cport; int rc, size = sizeof(*cport); - if (!rdma_cap_opa_vnic(device)) + if (!rdma_cap_opa_vnic(device) || !device->kverbs_provider) return; size += device->phys_port_cnt * sizeof(struct opa_vnic_vema_port); diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c index cccd00ed9baa..3982ca173605 100644 --- a/drivers/infiniband/ulp/srp/ib_srp.c +++ b/drivers/infiniband/ulp/srp/ib_srp.c @@ -4135,6 +4135,9 @@ static void srp_add_one(struct ib_device *device) u64 max_pages_per_mr; unsigned int flags = 0; + if (!device->kverbs_provider) + return; + srp_dev = kzalloc(sizeof(*srp_dev), GFP_KERNEL); if (!srp_dev) return; diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c index e75c908e4bc2..bfb251a180d6 100644 --- a/drivers/infiniband/ulp/srpt/ib_srpt.c +++ b/drivers/infiniband/ulp/srpt/ib_srpt.c @@ -3096,6 +3096,9 @@ static void srpt_add_one(struct ib_device *device) pr_debug("device = %p\n", device); + if (!device->kverbs_provider) + return; + sdev = kzalloc(struct_size(sdev, port, device->phys_port_cnt), GFP_KERNEL); if (!sdev) diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index a3ceed3a040a..9bc0140940df 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -2561,6 +2561,7 @@ struct ib_device { __be64 node_guid; u32 local_dma_lkey; u16 is_switch:1; + u8 kverbs_provider:1; u8 node_type; u8 phys_port_cnt; struct ib_device_attr attrs; diff --git a/net/rds/ib.c b/net/rds/ib.c index 9d7b7586f240..b8f08cd95f56 100644 --- a/net/rds/ib.c +++ b/net/rds/ib.c @@ -130,8 +130,8 @@ static void rds_ib_add_one(struct ib_device *device) struct rds_ib_device *rds_ibdev; bool has_fr, has_fmr; - /* Only handle IB (no iWARP) devices */ - if (device->node_type != RDMA_NODE_IB_CA) + /* Only handle IB (no iWARP) and kernel verbs providers devices */ + if (device->node_type != RDMA_NODE_IB_CA || !device->kverbs_provider) return; rds_ibdev = kzalloc_node(sizeof(struct rds_ib_device), GFP_KERNEL, diff --git a/net/smc/smc_ib.c b/net/smc/smc_ib.c index e519ef29c0ff..923048dc4b37 100644 --- a/net/smc/smc_ib.c +++ b/net/smc/smc_ib.c @@ -515,7 +515,7 @@ static void smc_ib_add_dev(struct ib_device *ibdev) u8 port_cnt; int i; - if (ibdev->node_type != RDMA_NODE_IB_CA) + if (ibdev->node_type != RDMA_NODE_IB_CA || !ibdev->kverbs_provider) return; smcibdev = kzalloc(sizeof(*smcibdev), GFP_KERNEL);
Drivers that do not provide kernel verbs support should communicate that to the ULPs before they try to use the device and fail. This patch allows drivers to indicate whether the device provides support for kernel API usage with the 'kverbs_provider' flag and makes ib clients not add unsupported devices. No functional changes were introduced in this patch, all devices are marked as kernel providers. Downstream patch will adjust relevant drivers. Signed-off-by: Gal Pressman <galpress@amazon.com> --- drivers/infiniband/core/cm.c | 3 +++ drivers/infiniband/core/cma.c | 3 +++ drivers/infiniband/core/mad.c | 3 +++ drivers/infiniband/core/multicast.c | 3 +++ drivers/infiniband/core/sa_query.c | 3 +++ drivers/infiniband/core/ucm.c | 3 ++- drivers/infiniband/core/user_mad.c | 3 +++ drivers/infiniband/hw/bnxt_re/main.c | 1 + drivers/infiniband/hw/cxgb3/iwch_provider.c | 1 + drivers/infiniband/hw/cxgb4/provider.c | 1 + drivers/infiniband/hw/hns/hns_roce_main.c | 1 + drivers/infiniband/hw/i40iw/i40iw_verbs.c | 1 + drivers/infiniband/hw/mlx4/main.c | 1 + drivers/infiniband/hw/mlx5/main.c | 1 + drivers/infiniband/hw/mthca/mthca_provider.c | 1 + drivers/infiniband/hw/nes/nes_verbs.c | 1 + drivers/infiniband/hw/ocrdma/ocrdma_main.c | 1 + drivers/infiniband/hw/qedr/main.c | 1 + drivers/infiniband/hw/usnic/usnic_ib_main.c | 1 + drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c | 1 + drivers/infiniband/sw/rdmavt/vt.c | 1 + drivers/infiniband/sw/rxe/rxe_verbs.c | 1 + drivers/infiniband/ulp/ipoib/ipoib_main.c | 3 +++ drivers/infiniband/ulp/opa_vnic/opa_vnic_vema.c | 2 +- drivers/infiniband/ulp/srp/ib_srp.c | 3 +++ drivers/infiniband/ulp/srpt/ib_srpt.c | 3 +++ include/rdma/ib_verbs.h | 1 + net/rds/ib.c | 4 ++-- net/smc/smc_ib.c | 2 +- 29 files changed, 49 insertions(+), 5 deletions(-)