| Message ID | 20211218204438.1345160-1-yanjun.zhu@linux.dev (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | [PATCHv2,1/1] RDMA/irdma: Make the source udp port vary | expand |
On 12/19/2021 4:44 AM, yanjun.zhu@linux.dev wrote: > External email: Use caution opening links or attachments > > > From: Zhu Yanjun <yanjun.zhu@linux.dev> > > Based on the link https://www.spinics.net/lists/linux-rdma/msg73735.html, > get the source udp port number for a QP based on the grh.flow_label or > lqpn/rqpn. This provides a better spread of traffic across NIC RX queues. > The method in the commit 2b880b2e5e03 ("RDMA/mlx5: Define RoCEv2 udp > source port when set path") is a standard way. So it is also adopted in > this commit. > > Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev> > --- > V1->V2: Adopt a standard method to get udp source port. > --- > drivers/infiniband/hw/irdma/verbs.c | 13 +++++++++++++ > 1 file changed, 13 insertions(+) > > diff --git a/drivers/infiniband/hw/irdma/verbs.c b/drivers/infiniband/hw/irdma/verbs.c > index 8cd5f9261692..9fe73d1db0d9 100644 > --- a/drivers/infiniband/hw/irdma/verbs.c > +++ b/drivers/infiniband/hw/irdma/verbs.c > @@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device *ibdev, u32 port, u16 index, > return 0; > } > > + > +static u16 irdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn) > +{ > + if (!fl) > + fl = rdma_calc_flow_label(lqpn, rqpn); > + > + return rdma_flow_label_to_udp_sport(fl); > +} > + > /** > * irdma_modify_qp_roce - modify qp request > * @ibqp: qp's pointer for modify > @@ -1159,6 +1168,10 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr, > > ctx_info->roce_info->pd_id = iwpd->sc_pd.pd_id; > > + udp_info->src_port = irdma_get_udp_sport(udp_info->flow_label, > + ibqp->qp_num, > + roce_info->dest_qp); > + FYI that in mlx5 driver the udp_sport is only set when address vector and dest qpn (IB_QP_AV and IB_QP_DEST_QPN) is provided. > if (attr_mask & IB_QP_AV) { > struct irdma_av *av = &iwqp->roce_ah.av; > const struct ib_gid_attr *sgid_attr; > -- > 2.27.0 >
在 2021/12/19 15:11, Mark Zhang 写道: > On 12/19/2021 4:44 AM, yanjun.zhu@linux.dev wrote: >> External email: Use caution opening links or attachments >> >> >> From: Zhu Yanjun <yanjun.zhu@linux.dev> >> >> Based on the link >> https://www.spinics.net/lists/linux-rdma/msg73735.html, >> get the source udp port number for a QP based on the grh.flow_label or >> lqpn/rqpn. This provides a better spread of traffic across NIC RX >> queues. >> The method in the commit 2b880b2e5e03 ("RDMA/mlx5: Define RoCEv2 udp >> source port when set path") is a standard way. So it is also adopted in >> this commit. >> >> Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev> >> --- >> V1->V2: Adopt a standard method to get udp source port. >> --- >> drivers/infiniband/hw/irdma/verbs.c | 13 +++++++++++++ >> 1 file changed, 13 insertions(+) >> >> diff --git a/drivers/infiniband/hw/irdma/verbs.c >> b/drivers/infiniband/hw/irdma/verbs.c >> index 8cd5f9261692..9fe73d1db0d9 100644 >> --- a/drivers/infiniband/hw/irdma/verbs.c >> +++ b/drivers/infiniband/hw/irdma/verbs.c >> @@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device >> *ibdev, u32 port, u16 index, >> return 0; >> } >> >> + >> +static u16 irdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn) >> +{ >> + if (!fl) >> + fl = rdma_calc_flow_label(lqpn, rqpn); >> + >> + return rdma_flow_label_to_udp_sport(fl); >> +} >> + >> /** >> * irdma_modify_qp_roce - modify qp request >> * @ibqp: qp's pointer for modify >> @@ -1159,6 +1168,10 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, >> struct ib_qp_attr *attr, >> >> ctx_info->roce_info->pd_id = iwpd->sc_pd.pd_id; >> >> + udp_info->src_port = irdma_get_udp_sport(udp_info->flow_label, >> + ibqp->qp_num, >> + roce_info->dest_qp); >> + > > FYI that in mlx5 driver the udp_sport is only set when address vector > and dest qpn (IB_QP_AV and IB_QP_DEST_QPN) is provided. Sure. From my tests, when this function irdma_get_udp_sport is called, dest qpn is ready. Zhu Yanjun > >> if (attr_mask & IB_QP_AV) { >> struct irdma_av *av = &iwqp->roce_ah.av; >> const struct ib_gid_attr *sgid_attr; >> -- >> 2.27.0 >> >
> Subject: Re: [PATCHv2 1/1] RDMA/irdma: Make the source udp port vary > > > 在 2021/12/19 15:11, Mark Zhang 写道: > > On 12/19/2021 4:44 AM, yanjun.zhu@linux.dev wrote: > >> External email: Use caution opening links or attachments > >> > >> > >> From: Zhu Yanjun <yanjun.zhu@linux.dev> > >> > >> Based on the link > >> https://www.spinics.net/lists/linux-rdma/msg73735.html, > >> get the source udp port number for a QP based on the grh.flow_label > >> or lqpn/rqpn. This provides a better spread of traffic across NIC RX > >> queues. > >> The method in the commit 2b880b2e5e03 ("RDMA/mlx5: Define RoCEv2 udp > >> source port when set path") is a standard way. So it is also adopted > >> in this commit. > >> > >> Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev> > >> --- > >> V1->V2: Adopt a standard method to get udp source port. > >> --- > >> drivers/infiniband/hw/irdma/verbs.c | 13 +++++++++++++ > >> 1 file changed, 13 insertions(+) > >> > >> diff --git a/drivers/infiniband/hw/irdma/verbs.c > >> b/drivers/infiniband/hw/irdma/verbs.c > >> index 8cd5f9261692..9fe73d1db0d9 100644 > >> --- a/drivers/infiniband/hw/irdma/verbs.c > >> +++ b/drivers/infiniband/hw/irdma/verbs.c > >> @@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device > >> *ibdev, u32 port, u16 index, > >> return 0; > >> } > >> > >> + > >> +static u16 irdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn) { > >> + if (!fl) > >> + fl = rdma_calc_flow_label(lqpn, rqpn); > >> + > >> + return rdma_flow_label_to_udp_sport(fl); } > >> + > >> /** > >> * irdma_modify_qp_roce - modify qp request > >> * @ibqp: qp's pointer for modify > >> @@ -1159,6 +1168,10 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, > >> struct ib_qp_attr *attr, > >> > >> ctx_info->roce_info->pd_id = iwpd->sc_pd.pd_id; > >> > >> + udp_info->src_port = > >> +irdma_get_udp_sport(udp_info->flow_label, > >> + ibqp->qp_num, > >> + roce_info->dest_qp); > >> + > > > > FYI that in mlx5 driver the udp_sport is only set when address vector > > and dest qpn (IB_QP_AV and IB_QP_DEST_QPN) is provided. > > Sure. From my tests, when this function irdma_get_udp_sport is called, dest qpn > is ready. > I think this needs to move to when we have IB_QP_AV in the mask and IB_AH_GRH in ah_flags. So, in this block, https://elixir.bootlin.com/linux/v5.16-rc6/source/drivers/infiniband/hw/irdma/verbs.c#L1171 Shiraz
在 2021/12/21 1:54, Saleem, Shiraz 写道: >> Subject: Re: [PATCHv2 1/1] RDMA/irdma: Make the source udp port vary >> >> >> 在 2021/12/19 15:11, Mark Zhang 写道: >>> On 12/19/2021 4:44 AM, yanjun.zhu@linux.dev wrote: >>>> External email: Use caution opening links or attachments >>>> >>>> >>>> From: Zhu Yanjun <yanjun.zhu@linux.dev> >>>> >>>> Based on the link >>>> https://www.spinics.net/lists/linux-rdma/msg73735.html, >>>> get the source udp port number for a QP based on the grh.flow_label >>>> or lqpn/rqpn. This provides a better spread of traffic across NIC RX >>>> queues. >>>> The method in the commit 2b880b2e5e03 ("RDMA/mlx5: Define RoCEv2 udp >>>> source port when set path") is a standard way. So it is also adopted >>>> in this commit. >>>> >>>> Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev> >>>> --- >>>> V1->V2: Adopt a standard method to get udp source port. >>>> --- >>>> drivers/infiniband/hw/irdma/verbs.c | 13 +++++++++++++ >>>> 1 file changed, 13 insertions(+) >>>> >>>> diff --git a/drivers/infiniband/hw/irdma/verbs.c >>>> b/drivers/infiniband/hw/irdma/verbs.c >>>> index 8cd5f9261692..9fe73d1db0d9 100644 >>>> --- a/drivers/infiniband/hw/irdma/verbs.c >>>> +++ b/drivers/infiniband/hw/irdma/verbs.c >>>> @@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device >>>> *ibdev, u32 port, u16 index, >>>> return 0; >>>> } >>>> >>>> + >>>> +static u16 irdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn) { >>>> + if (!fl) >>>> + fl = rdma_calc_flow_label(lqpn, rqpn); >>>> + >>>> + return rdma_flow_label_to_udp_sport(fl); } >>>> + >>>> /** >>>> * irdma_modify_qp_roce - modify qp request >>>> * @ibqp: qp's pointer for modify >>>> @@ -1159,6 +1168,10 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, >>>> struct ib_qp_attr *attr, >>>> >>>> ctx_info->roce_info->pd_id = iwpd->sc_pd.pd_id; >>>> >>>> + udp_info->src_port = >>>> +irdma_get_udp_sport(udp_info->flow_label, >>>> + ibqp->qp_num, >>>> + roce_info->dest_qp); >>>> + >>> >>> FYI that in mlx5 driver the udp_sport is only set when address vector >>> and dest qpn (IB_QP_AV and IB_QP_DEST_QPN) is provided. >> >> Sure. From my tests, when this function irdma_get_udp_sport is called, dest qpn >> is ready. >> > > I think this needs to move to when we have IB_QP_AV in the mask and IB_AH_GRH in ah_flags. > > So, in this block, > https://elixir.bootlin.com/linux/v5.16-rc6/source/drivers/infiniband/hw/irdma/verbs.c#L1171 Thanks. The latest patch has been sent following your suggestions. Zhu Yanjun > > Shiraz
diff --git a/drivers/infiniband/hw/irdma/verbs.c b/drivers/infiniband/hw/irdma/verbs.c index 8cd5f9261692..9fe73d1db0d9 100644 --- a/drivers/infiniband/hw/irdma/verbs.c +++ b/drivers/infiniband/hw/irdma/verbs.c @@ -1094,6 +1094,15 @@ static int irdma_query_pkey(struct ib_device *ibdev, u32 port, u16 index, return 0; } + +static u16 irdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn) +{ + if (!fl) + fl = rdma_calc_flow_label(lqpn, rqpn); + + return rdma_flow_label_to_udp_sport(fl); +} + /** * irdma_modify_qp_roce - modify qp request * @ibqp: qp's pointer for modify @@ -1159,6 +1168,10 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr, ctx_info->roce_info->pd_id = iwpd->sc_pd.pd_id; + udp_info->src_port = irdma_get_udp_sport(udp_info->flow_label, + ibqp->qp_num, + roce_info->dest_qp); + if (attr_mask & IB_QP_AV) { struct irdma_av *av = &iwqp->roce_ah.av; const struct ib_gid_attr *sgid_attr;