| Message ID | 20220909093822.33868-5-chengyou@linux.alibaba.com (mailing list archive) |
|---|---|
| State | Accepted |
| Headers | show |
| Series | RDMA/erdma: cleanups and updates 9-9-2022 | expand |
On Fri, Sep 09, 2022 at 05:38:22PM +0800, Cheng Xu wrote: > Hardware now support jumbo frame for RDMA. So we introduce a new CMDQ > message to support mtu change notification. > > Signed-off-by: Cheng Xu <chengyou@linux.alibaba.com> > --- > drivers/infiniband/hw/erdma/erdma.h | 1 + > drivers/infiniband/hw/erdma/erdma_hw.h | 6 ++++++ > drivers/infiniband/hw/erdma/erdma_main.c | 8 +++++++- > drivers/infiniband/hw/erdma/erdma_verbs.c | 11 +++++++++++ > drivers/infiniband/hw/erdma/erdma_verbs.h | 1 + > 5 files changed, 26 insertions(+), 1 deletion(-) <...> > --- a/drivers/infiniband/hw/erdma/erdma_main.c > +++ b/drivers/infiniband/hw/erdma/erdma_main.c > @@ -34,10 +34,15 @@ static int erdma_netdev_event(struct notifier_block *nb, unsigned long event, > dev->state = IB_PORT_DOWN; > erdma_port_event(dev, IB_EVENT_PORT_ERR); > break; > + case NETDEV_CHANGEMTU: > + if (dev->mtu != netdev->mtu) { > + erdma_set_mtu(dev, netdev->mtu); > + dev->mtu = netdev->mtu; > + } > + break; > case NETDEV_REGISTER: > case NETDEV_UNREGISTER: > case NETDEV_CHANGEADDR: > - case NETDEV_CHANGEMTU: > case NETDEV_GOING_DOWN: > case NETDEV_CHANGE: > default: > @@ -95,6 +100,7 @@ static int erdma_device_register(struct erdma_dev *dev) > if (ret) > return ret; > > + dev->mtu = dev->netdev->mtu; > addrconf_addr_eui48((u8 *)&ibdev->node_guid, dev->netdev->dev_addr); > > ret = ib_register_device(ibdev, "erdma_%d", &dev->pdev->dev); > diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.c b/drivers/infiniband/hw/erdma/erdma_verbs.c > index aea93d08af95..62be98e2b941 100644 > --- a/drivers/infiniband/hw/erdma/erdma_verbs.c > +++ b/drivers/infiniband/hw/erdma/erdma_verbs.c > @@ -1436,6 +1436,17 @@ int erdma_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, > return ret; > } > > +void erdma_set_mtu(struct erdma_dev *dev, u32 mtu) > +{ > + struct erdma_cmdq_config_mtu_req req; > + > + erdma_cmdq_build_reqhdr(&req.hdr, CMDQ_SUBMOD_COMMON, > + CMDQ_OPCODE_CONF_MTU); > + req.mtu = mtu; > + > + erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL); > +} I don't see any backward compatibility here. How can you make sure that new code that supports MTU change works correctly on old FW/device? Thanks
On 9/20/22 7:32 PM, Leon Romanovsky wrote: > On Fri, Sep 09, 2022 at 05:38:22PM +0800, Cheng Xu wrote: >> Hardware now support jumbo frame for RDMA. So we introduce a new CMDQ >> message to support mtu change notification. >> <...> > > I don't see any backward compatibility here. How can you make sure that > new code that supports MTU change works correctly on old FW/device? > In this case, driver needn't to consider backward compatibility. ERDMA hardware is programmable part of our iaas infrastructure, and can be hot-update without BMs/VMs awareness.Before I submitted this patch, all the FWs has been updated, and support this feature, no old FWs exist. Thanks, Cheng Xu
On Fri, 9 Sep 2022 17:38:22 +0800, Cheng Xu wrote: > Hardware now support jumbo frame for RDMA. So we introduce a new CMDQ > message to support mtu change notification. > > Applied, thanks! [4/4] RDMA/erdma: Support dynamic mtu https://git.kernel.org/rdma/rdma/c/9bdb9350f3808b Best regards,
On Wed, Sep 21, 2022 at 10:28:57AM +0800, Cheng Xu wrote: > > > On 9/20/22 7:32 PM, Leon Romanovsky wrote: > > On Fri, Sep 09, 2022 at 05:38:22PM +0800, Cheng Xu wrote: > >> Hardware now support jumbo frame for RDMA. So we introduce a new CMDQ > >> message to support mtu change notification. > >> > <...> > > > > I don't see any backward compatibility here. How can you make sure that > > new code that supports MTU change works correctly on old FW/device? > > > > In this case, driver needn't to consider backward compatibility. > > ERDMA hardware is programmable part of our iaas infrastructure, and can be > hot-update without BMs/VMs awareness.Before I submitted this patch, all the > FWs has been updated, and support this feature, no old FWs exist. I'm not big fan of such answers, but ok, applied. > > Thanks, > Cheng Xu > >
diff --git a/drivers/infiniband/hw/erdma/erdma.h b/drivers/infiniband/hw/erdma/erdma.h index cc5e4eb3a21e..730783fbc894 100644 --- a/drivers/infiniband/hw/erdma/erdma.h +++ b/drivers/infiniband/hw/erdma/erdma.h @@ -197,6 +197,7 @@ struct erdma_dev { struct erdma_devattr attrs; /* physical port state (only one port per device) */ enum ib_port_state state; + u32 mtu; /* cmdq and aeq use the same msix vector */ struct erdma_irq comm_irq; diff --git a/drivers/infiniband/hw/erdma/erdma_hw.h b/drivers/infiniband/hw/erdma/erdma_hw.h index 3004cf3ac481..e788887732e1 100644 --- a/drivers/infiniband/hw/erdma/erdma_hw.h +++ b/drivers/infiniband/hw/erdma/erdma_hw.h @@ -153,6 +153,7 @@ enum CMDQ_COMMON_OPCODE { CMDQ_OPCODE_CREATE_EQ = 0, CMDQ_OPCODE_DESTROY_EQ = 1, CMDQ_OPCODE_QUERY_FW_INFO = 2, + CMDQ_OPCODE_CONF_MTU = 3, }; /* cmdq-SQE HDR */ @@ -190,6 +191,11 @@ struct erdma_cmdq_destroy_eq_req { u8 qtype; }; +struct erdma_cmdq_config_mtu_req { + u64 hdr; + u32 mtu; +}; + /* create_cq cfg0 */ #define ERDMA_CMD_CREATE_CQ_DEPTH_MASK GENMASK(31, 24) #define ERDMA_CMD_CREATE_CQ_PAGESIZE_MASK GENMASK(23, 20) diff --git a/drivers/infiniband/hw/erdma/erdma_main.c b/drivers/infiniband/hw/erdma/erdma_main.c index 6d3e02ba9e77..49778bb294ae 100644 --- a/drivers/infiniband/hw/erdma/erdma_main.c +++ b/drivers/infiniband/hw/erdma/erdma_main.c @@ -34,10 +34,15 @@ static int erdma_netdev_event(struct notifier_block *nb, unsigned long event, dev->state = IB_PORT_DOWN; erdma_port_event(dev, IB_EVENT_PORT_ERR); break; + case NETDEV_CHANGEMTU: + if (dev->mtu != netdev->mtu) { + erdma_set_mtu(dev, netdev->mtu); + dev->mtu = netdev->mtu; + } + break; case NETDEV_REGISTER: case NETDEV_UNREGISTER: case NETDEV_CHANGEADDR: - case NETDEV_CHANGEMTU: case NETDEV_GOING_DOWN: case NETDEV_CHANGE: default: @@ -95,6 +100,7 @@ static int erdma_device_register(struct erdma_dev *dev) if (ret) return ret; + dev->mtu = dev->netdev->mtu; addrconf_addr_eui48((u8 *)&ibdev->node_guid, dev->netdev->dev_addr); ret = ib_register_device(ibdev, "erdma_%d", &dev->pdev->dev); diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.c b/drivers/infiniband/hw/erdma/erdma_verbs.c index aea93d08af95..62be98e2b941 100644 --- a/drivers/infiniband/hw/erdma/erdma_verbs.c +++ b/drivers/infiniband/hw/erdma/erdma_verbs.c @@ -1436,6 +1436,17 @@ int erdma_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, return ret; } +void erdma_set_mtu(struct erdma_dev *dev, u32 mtu) +{ + struct erdma_cmdq_config_mtu_req req; + + erdma_cmdq_build_reqhdr(&req.hdr, CMDQ_SUBMOD_COMMON, + CMDQ_OPCODE_CONF_MTU); + req.mtu = mtu; + + erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL); +} + void erdma_port_event(struct erdma_dev *dev, enum ib_event_type reason) { struct ib_event event; diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.h b/drivers/infiniband/hw/erdma/erdma_verbs.h index fe93e1ac9674..ab6380635e9e 100644 --- a/drivers/infiniband/hw/erdma/erdma_verbs.h +++ b/drivers/infiniband/hw/erdma/erdma_verbs.h @@ -330,5 +330,6 @@ struct ib_mr *erdma_ib_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type mr_type, int erdma_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sg, int sg_nents, unsigned int *sg_offset); void erdma_port_event(struct erdma_dev *dev, enum ib_event_type reason); +void erdma_set_mtu(struct erdma_dev *dev, u32 mtu); #endif
Hardware now support jumbo frame for RDMA. So we introduce a new CMDQ message to support mtu change notification. Signed-off-by: Cheng Xu <chengyou@linux.alibaba.com> --- drivers/infiniband/hw/erdma/erdma.h | 1 + drivers/infiniband/hw/erdma/erdma_hw.h | 6 ++++++ drivers/infiniband/hw/erdma/erdma_main.c | 8 +++++++- drivers/infiniband/hw/erdma/erdma_verbs.c | 11 +++++++++++ drivers/infiniband/hw/erdma/erdma_verbs.h | 1 + 5 files changed, 26 insertions(+), 1 deletion(-)