| Message ID | 20190930231707.48259-10-bvanassche@acm.org (mailing list archive) |
|---|---|
| State | Accepted |
| Delegated to: | Jason Gunthorpe |
| Headers | show |
| Series | RDMA patches for kernel v5.5 | expand |
On Mon, Sep 30, 2019 at 04:17:01PM -0700, Bart Van Assche wrote: > Management datagrams (MADs) are not supported by SR-IOV VFs nor by > iWARP Really? This seems surprising to me.. Jason
On 10/2/19 7:14 AM, Jason Gunthorpe wrote: > On Mon, Sep 30, 2019 at 04:17:01PM -0700, Bart Van Assche wrote: >> Management datagrams (MADs) are not supported by SR-IOV VFs nor by >> iWARP > > Really? This seems surprising to me.. Hi Jason, Last time I checked the Mellanox drivers allow MADs to be sent over a SR-IOV VF but do not allow MADs to be received through such a VF. I haven't been able to find any reference to management datagrams in the iWARP RFC. Maybe that means that I overlooked something? Thanks, Bart.
On Wed, Oct 02, 2019 at 08:21:45AM -0700, Bart Van Assche wrote: > On 10/2/19 7:14 AM, Jason Gunthorpe wrote: > > On Mon, Sep 30, 2019 at 04:17:01PM -0700, Bart Van Assche wrote: > >> Management datagrams (MADs) are not supported by SR-IOV VFs nor by > >> iWARP > > > > Really? This seems surprising to me.. > > Hi Jason, > > Last time I checked the Mellanox drivers allow MADs to be sent over a > SR-IOV VF but do not allow MADs to be received through such a VF. I think that is only true of mlx4, mlx5 allows receive, AFAIK I don't know if registering a mad agent fails though. Jack? > I haven't been able to find any reference to management datagrams in the > iWARP RFC. Maybe that means that I overlooked something? Iwarp makes sense Jason
On Wed, Oct 02, 2019 at 01:51:00PM -0300, Jason Gunthorpe wrote: > On Wed, Oct 02, 2019 at 08:21:45AM -0700, Bart Van Assche wrote: > > On 10/2/19 7:14 AM, Jason Gunthorpe wrote: > > > On Mon, Sep 30, 2019 at 04:17:01PM -0700, Bart Van Assche wrote: > > >> Management datagrams (MADs) are not supported by SR-IOV VFs nor by > > >> iWARP > > > > > > Really? This seems surprising to me.. > > > > Hi Jason, > > > > Last time I checked the Mellanox drivers allow MADs to be sent over a > > SR-IOV VF but do not allow MADs to be received through such a VF. > > I think that is only true of mlx4, mlx5 allows receive, AFAIK > > I don't know if registering a mad agent fails though. Jack? According to internal mlx5 specification, MAD is fully operational for every Virtual HCA used to connect such virtual devices to IBTA virtualization spec. "Each PCI function (PF or VF) represents a vHCA. Each vHCA virtual port is mapped to an InfiniBand vport. The mapping is arbitrary and determined by the device, as the InfiniBand management is agnostic to it (the InfiniBand specification has no notion of hosts or PCI functions)." Most probably the observed by Bart behaviour is related to the fact that vport0 has special meaning to allow legacy SMs to connect. Thanks > > > I haven't been able to find any reference to management datagrams in the > > iWARP RFC. Maybe that means that I overlooked something? > > Iwarp makes sense > > Jason
On 10/2/19 10:24 AM, Leon Romanovsky wrote: > On Wed, Oct 02, 2019 at 01:51:00PM -0300, Jason Gunthorpe wrote: >> On Wed, Oct 02, 2019 at 08:21:45AM -0700, Bart Van Assche wrote: >>> On 10/2/19 7:14 AM, Jason Gunthorpe wrote: >>>> On Mon, Sep 30, 2019 at 04:17:01PM -0700, Bart Van Assche wrote: >>>>> Management datagrams (MADs) are not supported by SR-IOV VFs nor by >>>>> iWARP >>>> >>>> Really? This seems surprising to me.. >>> >>> Hi Jason, >>> >>> Last time I checked the Mellanox drivers allow MADs to be sent over a >>> SR-IOV VF but do not allow MADs to be received through such a VF. >> >> I think that is only true of mlx4, mlx5 allows receive, AFAIK >> >> I don't know if registering a mad agent fails though. Jack? > > According to internal mlx5 specification, MAD is fully operational for > every Virtual HCA used to connect such virtual devices to IBTA > virtualization spec. > > "Each PCI function (PF or VF) represents a vHCA. Each vHCA virtual port > is mapped to an InfiniBand vport. The mapping is arbitrary and determined > by the device, as the InfiniBand management is agnostic to it (the > InfiniBand specification has no notion of hosts or PCI functions)." > > Most probably the observed by Bart behaviour is related to the fact that > vport0 has special meaning to allow legacy SMs to connect. Hi Jason and Leon, Is it essential that we figure out which HCAs support MADs for VFs or is it perhaps sufficient that I change the description of this patch such that it mentions that device management and MAD support is not guaranteed to be available? Thanks, Bart.
On Wed, Oct 02, 2019 at 10:43:59AM -0700, Bart Van Assche wrote: > On 10/2/19 10:24 AM, Leon Romanovsky wrote: > > On Wed, Oct 02, 2019 at 01:51:00PM -0300, Jason Gunthorpe wrote: > > > On Wed, Oct 02, 2019 at 08:21:45AM -0700, Bart Van Assche wrote: > > > > On 10/2/19 7:14 AM, Jason Gunthorpe wrote: > > > > > On Mon, Sep 30, 2019 at 04:17:01PM -0700, Bart Van Assche wrote: > > > > > > Management datagrams (MADs) are not supported by SR-IOV VFs nor by > > > > > > iWARP > > > > > > > > > > Really? This seems surprising to me.. > > > > > > > > Hi Jason, > > > > > > > > Last time I checked the Mellanox drivers allow MADs to be sent over a > > > > SR-IOV VF but do not allow MADs to be received through such a VF. > > > > > > I think that is only true of mlx4, mlx5 allows receive, AFAIK > > > > > > I don't know if registering a mad agent fails though. Jack? > > > > According to internal mlx5 specification, MAD is fully operational for > > every Virtual HCA used to connect such virtual devices to IBTA > > virtualization spec. > > > > "Each PCI function (PF or VF) represents a vHCA. Each vHCA virtual port > > is mapped to an InfiniBand vport. The mapping is arbitrary and determined > > by the device, as the InfiniBand management is agnostic to it (the > > InfiniBand specification has no notion of hosts or PCI functions)." > > > > Most probably the observed by Bart behaviour is related to the fact that > > vport0 has special meaning to allow legacy SMs to connect. > > Hi Jason and Leon, > > Is it essential that we figure out which HCAs support MADs for VFs or is it > perhaps sufficient that I change the description of this patch such that it > mentions that device management and MAD support is not guaranteed to be > available? I have no idea, sorry. > > Thanks, > > Bart. >
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c index e25c70a56be6..4f99a5e040c3 100644 --- a/drivers/infiniband/ulp/srpt/ib_srpt.c +++ b/drivers/infiniband/ulp/srpt/ib_srpt.c @@ -556,24 +556,16 @@ static int srpt_refresh_port(struct srpt_port *sport) struct ib_port_attr port_attr; int ret; - memset(&port_modify, 0, sizeof(port_modify)); - port_modify.set_port_cap_mask = IB_PORT_DEVICE_MGMT_SUP; - port_modify.clr_port_cap_mask = 0; - - ret = ib_modify_port(sport->sdev->device, sport->port, 0, &port_modify); - if (ret) - goto err_mod_port; - ret = ib_query_port(sport->sdev->device, sport->port, &port_attr); if (ret) - goto err_query_port; + return ret; sport->sm_lid = port_attr.sm_lid; sport->lid = port_attr.lid; ret = rdma_query_gid(sport->sdev->device, sport->port, 0, &sport->gid); if (ret) - goto err_query_port; + return ret; sport->port_guid_wwn.priv = sport; srpt_format_guid(sport->port_guid, sizeof(sport->port_guid), @@ -584,6 +576,20 @@ static int srpt_refresh_port(struct srpt_port *sport) be64_to_cpu(sport->gid.global.subnet_prefix), be64_to_cpu(sport->gid.global.interface_id)); + if (rdma_protocol_iwarp(sport->sdev->device, sport->port)) + return 0; + + memset(&port_modify, 0, sizeof(port_modify)); + port_modify.set_port_cap_mask = IB_PORT_DEVICE_MGMT_SUP; + port_modify.clr_port_cap_mask = 0; + + ret = ib_modify_port(sport->sdev->device, sport->port, 0, &port_modify); + if (ret) { + pr_warn("%s-%d: enabling device management failed (%d). Note: this is expected if SR-IOV is enabled.\n", + dev_name(&sport->sdev->device->dev), sport->port, ret); + return 0; + } + if (!sport->mad_agent) { memset(®_req, 0, sizeof(reg_req)); reg_req.mgmt_class = IB_MGMT_CLASS_DEVICE_MGMT; @@ -599,23 +605,14 @@ static int srpt_refresh_port(struct srpt_port *sport) srpt_mad_recv_handler, sport, 0); if (IS_ERR(sport->mad_agent)) { - ret = PTR_ERR(sport->mad_agent); + pr_err("%s-%d: MAD agent registration failed (%ld). Note: this is expected if SR-IOV is enabled.\n", + dev_name(&sport->sdev->device->dev), sport->port, + PTR_ERR(sport->mad_agent)); sport->mad_agent = NULL; - goto err_query_port; } } return 0; - -err_query_port: - - port_modify.set_port_cap_mask = 0; - port_modify.clr_port_cap_mask = IB_PORT_DEVICE_MGMT_SUP; - ib_modify_port(sport->sdev->device, sport->port, 0, &port_modify); - -err_mod_port: - - return ret; } /**
Management datagrams (MADs) are not supported by SR-IOV VFs nor by iWARP ports. Support SR-IOV VFs and iWARP ports by only logging an error message if MAD handler registration fails. Cc: Honggang LI <honli@redhat.com> Cc: Laurence Oberman <loberman@redhat.com> Signed-off-by: Bart Van Assche <bvanassche@acm.org> --- drivers/infiniband/ulp/srpt/ib_srpt.c | 41 +++++++++++++-------------- 1 file changed, 19 insertions(+), 22 deletions(-)