| Message ID | 1451395447-5198-2-git-send-email-matanb@mellanox.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
On 12/29/2015 3:24 PM, Matan Barak wrote: > @@ -905,6 +906,8 @@ int mlx4_QUERY_DEV_CAP(struct mlx4_dev *dev, struct mlx4_dev_cap *dev_cap) > dev_cap->flags2 |= MLX4_DEV_CAP_FLAG2_EQE_STRIDE; > MLX4_GET(dev_cap->bmme_flags, outbox, > QUERY_DEV_CAP_BMME_FLAGS_OFFSET); > + if (dev_cap->bmme_flags & MLX4_FLAG_ROCE_V1_V2) > + dev_cap->flags2 |= MLX4_DEV_CAP_FLAG2_ROCE_V1_V2; Did you make sure that the query dev cap wrapper unsets this bit when proxing VF queries? > if (dev_cap->bmme_flags & MLX4_FLAG_PORT_REMAP) > dev_cap->flags2 |= MLX4_DEV_CAP_FLAG2_PORT_REMAP; > MLX4_GET(field, outbox, QUERY_DEV_CAP_CONFIG_DEV_OFFSET); -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 12/29/2015 5:19 PM, Or Gerlitz wrote: > On 12/29/2015 3:24 PM, Matan Barak wrote: >> @@ -905,6 +906,8 @@ int mlx4_QUERY_DEV_CAP(struct mlx4_dev *dev, >> struct mlx4_dev_cap *dev_cap) >> dev_cap->flags2 |= MLX4_DEV_CAP_FLAG2_EQE_STRIDE; >> MLX4_GET(dev_cap->bmme_flags, outbox, >> QUERY_DEV_CAP_BMME_FLAGS_OFFSET); >> + if (dev_cap->bmme_flags & MLX4_FLAG_ROCE_V1_V2) >> + dev_cap->flags2 |= MLX4_DEV_CAP_FLAG2_ROCE_V1_V2; > > Did you make sure that the query dev cap wrapper unsets this bit when > proxing VF queries? In mlx4_dev_cap: if (mlx4_is_mfunc(dev)) { dev->caps.flags &= ~MLX4_DEV_CAP_FLAG_SENSE_SUPPORT; dev_cap->flags2 &= ~MLX4_DEV_CAP_FLAG2_ROCE_V1_V2; mlx4_dbg(dev, "RoCE V2 is not supported when SR-IOV is enabled\n"); } mlx4_slave_cap calls mlx4_dev_cap and uses the dev_caps it queried, so we should be safe here. > >> if (dev_cap->bmme_flags & MLX4_FLAG_PORT_REMAP) >> dev_cap->flags2 |= MLX4_DEV_CAP_FLAG2_PORT_REMAP; >> MLX4_GET(field, outbox, QUERY_DEV_CAP_CONFIG_DEV_OFFSET); > > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 12/30/2015 10:27 AM, Matan Barak wrote: > > > On 12/29/2015 5:19 PM, Or Gerlitz wrote: >> On 12/29/2015 3:24 PM, Matan Barak wrote: >>> @@ -905,6 +906,8 @@ int mlx4_QUERY_DEV_CAP(struct mlx4_dev *dev, >>> struct mlx4_dev_cap *dev_cap) >>> dev_cap->flags2 |= MLX4_DEV_CAP_FLAG2_EQE_STRIDE; >>> MLX4_GET(dev_cap->bmme_flags, outbox, >>> QUERY_DEV_CAP_BMME_FLAGS_OFFSET); >>> + if (dev_cap->bmme_flags & MLX4_FLAG_ROCE_V1_V2) >>> + dev_cap->flags2 |= MLX4_DEV_CAP_FLAG2_ROCE_V1_V2; >> >> Did you make sure that the query dev cap wrapper unsets this bit when >> proxing VF queries? > > In mlx4_dev_cap: > if (mlx4_is_mfunc(dev)) { > dev->caps.flags &= ~MLX4_DEV_CAP_FLAG_SENSE_SUPPORT; > dev_cap->flags2 &= ~MLX4_DEV_CAP_FLAG2_ROCE_V1_V2; > mlx4_dbg(dev, "RoCE V2 is not supported when SR-IOV is enabled\n"); > } > > mlx4_slave_cap calls mlx4_dev_cap and uses the dev_caps it queried, so > we should be safe here. mlx4_slave_cap is part of the Linux VF driver flow, right? So... NO, this is the Linux implementation. You should make things robust against any guest driver. The only way to do that is patch the command wrapper used by the PF to filter out unwanted cap bits, see other filtering we do in mlx4_QUERY_DEV_CAP_wrapper Or. > >> >>> if (dev_cap->bmme_flags & MLX4_FLAG_PORT_REMAP) >>> dev_cap->flags2 |= MLX4_DEV_CAP_FLAG2_PORT_REMAP; >>> MLX4_GET(field, outbox, QUERY_DEV_CAP_CONFIG_DEV_OFFSET); >> >> -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 12/30/2015 10:44 AM, Or Gerlitz wrote: > On 12/30/2015 10:27 AM, Matan Barak wrote: >> >> >> On 12/29/2015 5:19 PM, Or Gerlitz wrote: >>> On 12/29/2015 3:24 PM, Matan Barak wrote: >>>> @@ -905,6 +906,8 @@ int mlx4_QUERY_DEV_CAP(struct mlx4_dev *dev, >>>> struct mlx4_dev_cap *dev_cap) >>>> dev_cap->flags2 |= MLX4_DEV_CAP_FLAG2_EQE_STRIDE; >>>> MLX4_GET(dev_cap->bmme_flags, outbox, >>>> QUERY_DEV_CAP_BMME_FLAGS_OFFSET); >>>> + if (dev_cap->bmme_flags & MLX4_FLAG_ROCE_V1_V2) >>>> + dev_cap->flags2 |= MLX4_DEV_CAP_FLAG2_ROCE_V1_V2; >>> >>> Did you make sure that the query dev cap wrapper unsets this bit when >>> proxing VF queries? >> >> In mlx4_dev_cap: >> if (mlx4_is_mfunc(dev)) { >> dev->caps.flags &= ~MLX4_DEV_CAP_FLAG_SENSE_SUPPORT; >> dev_cap->flags2 &= ~MLX4_DEV_CAP_FLAG2_ROCE_V1_V2; >> mlx4_dbg(dev, "RoCE V2 is not supported when SR-IOV is enabled\n"); >> } >> >> mlx4_slave_cap calls mlx4_dev_cap and uses the dev_caps it queried, so >> we should be safe here. > > mlx4_slave_cap is part of the Linux VF driver flow, right? > > So... NO, this is the Linux implementation. > > You should make things robust against any guest driver. > > The only way to do that is patch the command wrapper used by the PF > to filter out unwanted cap bits, see other filtering we do in > mlx4_QUERY_DEV_CAP_wrapper > I agree, thanks > Or. Matan > > >> >>> >>>> if (dev_cap->bmme_flags & MLX4_FLAG_PORT_REMAP) >>>> dev_cap->flags2 |= MLX4_DEV_CAP_FLAG2_PORT_REMAP; >>>> MLX4_GET(field, outbox, QUERY_DEV_CAP_CONFIG_DEV_OFFSET); >>> >>> > -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/drivers/net/ethernet/mellanox/mlx4/fw.c b/drivers/net/ethernet/mellanox/mlx4/fw.c index 90db94e..bdd6822 100644 --- a/drivers/net/ethernet/mellanox/mlx4/fw.c +++ b/drivers/net/ethernet/mellanox/mlx4/fw.c @@ -157,6 +157,7 @@ static void dump_dev_cap_flags2(struct mlx4_dev *dev, u64 flags) [29] = "802.1ad offload support", [31] = "Modifying loopback source checks using UPDATE_QP support", [32] = "Loopback source checks support", + [33] = "RoCEv2 support" }; int i; @@ -905,6 +906,8 @@ int mlx4_QUERY_DEV_CAP(struct mlx4_dev *dev, struct mlx4_dev_cap *dev_cap) dev_cap->flags2 |= MLX4_DEV_CAP_FLAG2_EQE_STRIDE; MLX4_GET(dev_cap->bmme_flags, outbox, QUERY_DEV_CAP_BMME_FLAGS_OFFSET); + if (dev_cap->bmme_flags & MLX4_FLAG_ROCE_V1_V2) + dev_cap->flags2 |= MLX4_DEV_CAP_FLAG2_ROCE_V1_V2; if (dev_cap->bmme_flags & MLX4_FLAG_PORT_REMAP) dev_cap->flags2 |= MLX4_DEV_CAP_FLAG2_PORT_REMAP; MLX4_GET(field, outbox, QUERY_DEV_CAP_CONFIG_DEV_OFFSET); diff --git a/drivers/net/ethernet/mellanox/mlx4/main.c b/drivers/net/ethernet/mellanox/mlx4/main.c index 31c491e..fb4968f 100644 --- a/drivers/net/ethernet/mellanox/mlx4/main.c +++ b/drivers/net/ethernet/mellanox/mlx4/main.c @@ -424,8 +424,12 @@ static int mlx4_dev_cap(struct mlx4_dev *dev, struct mlx4_dev_cap *dev_cap) if (mlx4_priv(dev)->pci_dev_data & MLX4_PCI_DEV_FORCE_SENSE_PORT) dev->caps.flags |= MLX4_DEV_CAP_FLAG_SENSE_SUPPORT; /* Don't do sense port on multifunction devices (for now at least) */ - if (mlx4_is_mfunc(dev)) + /* Don't do enable RoCE V2 on multifunction devices */ + if (mlx4_is_mfunc(dev)) { dev->caps.flags &= ~MLX4_DEV_CAP_FLAG_SENSE_SUPPORT; + dev_cap->flags2 &= ~MLX4_DEV_CAP_FLAG2_ROCE_V1_V2; + mlx4_dbg(dev, "RoCE V2 is not supported when SR-IOV is enabled\n"); + } if (mlx4_low_memory_profile()) { dev->caps.log_num_macs = MLX4_MIN_LOG_NUM_MAC; diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h index d3133be..dbf39ab 100644 --- a/include/linux/mlx4/device.h +++ b/include/linux/mlx4/device.h @@ -216,6 +216,7 @@ enum { MLX4_DEV_CAP_FLAG2_SKIP_OUTER_VLAN = 1LL << 30, MLX4_DEV_CAP_FLAG2_UPDATE_QP_SRC_CHECK_LB = 1ULL << 31, MLX4_DEV_CAP_FLAG2_LB_SRC_CHK = 1ULL << 32, + MLX4_DEV_CAP_FLAG2_ROCE_V1_V2 = 1LL << 33, }; enum { @@ -267,6 +268,7 @@ enum { MLX4_BMME_FLAG_TYPE_2_WIN = 1 << 9, MLX4_BMME_FLAG_RESERVED_LKEY = 1 << 10, MLX4_BMME_FLAG_FAST_REG_WR = 1 << 11, + MLX4_BMME_FLAG_ROCE_V1_V2 = 1 << 19, MLX4_BMME_FLAG_PORT_REMAP = 1 << 24, MLX4_BMME_FLAG_VSD_INIT2RTR = 1 << 28, }; @@ -275,6 +277,10 @@ enum { MLX4_FLAG_PORT_REMAP = MLX4_BMME_FLAG_PORT_REMAP }; +enum { + MLX4_FLAG_ROCE_V1_V2 = MLX4_BMME_FLAG_ROCE_V1_V2 +}; + enum mlx4_event { MLX4_EVENT_TYPE_COMP = 0x00, MLX4_EVENT_TYPE_PATH_MIG = 0x01, @@ -984,9 +990,10 @@ struct mlx4_mad_ifc { if (((dev)->caps.port_mask[port] != MLX4_PORT_TYPE_IB)) #define mlx4_foreach_ib_transport_port(port, dev) \ - for ((port) = 1; (port) <= (dev)->caps.num_ports; (port)++) \ + for ((port) = 1; (port) <= (dev)->caps.num_ports; (port)++) \ if (((dev)->caps.port_mask[port] == MLX4_PORT_TYPE_IB) || \ - ((dev)->caps.flags & MLX4_DEV_CAP_FLAG_IBOE)) + ((dev)->caps.flags & MLX4_DEV_CAP_FLAG_IBOE) || \ + ((dev)->caps.flags2 & MLX4_DEV_CAP_FLAG2_ROCE_V1_V2)) #define MLX4_INVALID_SLAVE_ID 0xFF #define MLX4_SINK_COUNTER_INDEX(dev) (dev->caps.max_counters - 1)