[00/10] lustre: lnet: fixes for non-x86 systems
mbox series

Message ID 1539543332-28679-1-git-send-email-jsimmons@infradead.org
Headers show
Series
  • lustre: lnet: fixes for non-x86 systems
Related show

Message

James Simmons Oct. 14, 2018, 6:55 p.m. UTC
These are the required patches to make LNet work with non-x86
platforms like ARM or Power8. The tunables map_on_demand and
concurrent_sends assumed pages were 4K in size which is not
correct. Massively reworked to basically defunct those tunables.
Also the size of the LNet packet was always 256 pages but when
the page size is 64K like some ARM or Power8 systems the maximum
LNet message sent was 16MB not 1MB which is what is expected.
Fixing up the RDMA handling in the ko2iblnd driver also resolved
some performance issues.

Alexey Lyashkov (1):
  lustre: lnd: use less CQ entries for each connection

Amir Shehata (6):
  lustre: lnd: set device capabilities
  lustre: o2iblnd: use IB_MR_TYPE_SG_GAPS
  lustre: lnd: rework map_on_demand behavior
  lustre: lnd: calculate qp max_send_wrs properly
  lustre: lnd: remove concurrent_sends tunable
  lustre: lnd: correct WR fast reg accounting

Dmitry Eremin (1):
  lustre: o2iblnd: limit cap.max_send_wr for MLX5

James Simmons (1):
  lustre: lnet: make LNET_MAX_IOV dependent on page size

John L. Hammond (1):
  lustre: o2ib: use splice in kiblnd_peer_connect_failed()

 .../staging/lustre/include/linux/lnet/lib-types.h  |  10 +-
 .../lustre/include/uapi/linux/lnet/lnet-types.h    |   3 -
 drivers/staging/lustre/lnet/Kconfig                |  10 --
 .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c    | 138 +++++++++++++++------
 .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h    |  68 +++-------
 .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c | 133 ++++++++++++++------
 .../lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c  |  85 ++++++-------
 drivers/staging/lustre/lustre/include/lustre_net.h |   1 +
 8 files changed, 256 insertions(+), 192 deletions(-)

Comments

NeilBrown Oct. 18, 2018, 4:48 a.m. UTC | #1
On Sun, Oct 14 2018, James Simmons wrote:

> These are the required patches to make LNet work with non-x86
> platforms like ARM or Power8. The tunables map_on_demand and
> concurrent_sends assumed pages were 4K in size which is not
> correct. Massively reworked to basically defunct those tunables.
> Also the size of the LNet packet was always 256 pages but when
> the page size is 64K like some ARM or Power8 systems the maximum
> LNet message sent was 16MB not 1MB which is what is expected.
> Fixing up the RDMA handling in the ko2iblnd driver also resolved
> some performance issues.

Thanks.
I've applied this series, but I've still suspicious of that little
section in patch 3.
I hope that can be clarified, or fixed, before I feel the need to  move
the series from 'lustre-testing' to 'lustre'.

Thanks,
NeilBrown


>
> Alexey Lyashkov (1):
>   lustre: lnd: use less CQ entries for each connection
>
> Amir Shehata (6):
>   lustre: lnd: set device capabilities
>   lustre: o2iblnd: use IB_MR_TYPE_SG_GAPS
>   lustre: lnd: rework map_on_demand behavior
>   lustre: lnd: calculate qp max_send_wrs properly
>   lustre: lnd: remove concurrent_sends tunable
>   lustre: lnd: correct WR fast reg accounting
>
> Dmitry Eremin (1):
>   lustre: o2iblnd: limit cap.max_send_wr for MLX5
>
> James Simmons (1):
>   lustre: lnet: make LNET_MAX_IOV dependent on page size
>
> John L. Hammond (1):
>   lustre: o2ib: use splice in kiblnd_peer_connect_failed()
>
>  .../staging/lustre/include/linux/lnet/lib-types.h  |  10 +-
>  .../lustre/include/uapi/linux/lnet/lnet-types.h    |   3 -
>  drivers/staging/lustre/lnet/Kconfig                |  10 --
>  .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c    | 138 +++++++++++++++------
>  .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h    |  68 +++-------
>  .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c | 133 ++++++++++++++------
>  .../lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c  |  85 ++++++-------
>  drivers/staging/lustre/lustre/include/lustre_net.h |   1 +
>  8 files changed, 256 insertions(+), 192 deletions(-)
>
> -- 
> 1.8.3.1
James Simmons Oct. 20, 2018, 7 p.m. UTC | #2
> On Sun, Oct 14 2018, James Simmons wrote:
> 
> > These are the required patches to make LNet work with non-x86
> > platforms like ARM or Power8. The tunables map_on_demand and
> > concurrent_sends assumed pages were 4K in size which is not
> > correct. Massively reworked to basically defunct those tunables.
> > Also the size of the LNet packet was always 256 pages but when
> > the page size is 64K like some ARM or Power8 systems the maximum
> > LNet message sent was 16MB not 1MB which is what is expected.
> > Fixing up the RDMA handling in the ko2iblnd driver also resolved
> > some performance issues.
> 
> Thanks.
> I've applied this series, but I've still suspicious of that little
> section in patch 3.
> I hope that can be clarified, or fixed, before I feel the need to  move
> the series from 'lustre-testing' to 'lustre'.

Thank you. Yes as I pointed out that code in patch 3 is wrong. I pushed
a fix to the OpenSFS branch already. You can drop that change in patch 3.


> > Alexey Lyashkov (1):
> >   lustre: lnd: use less CQ entries for each connection
> >
> > Amir Shehata (6):
> >   lustre: lnd: set device capabilities
> >   lustre: o2iblnd: use IB_MR_TYPE_SG_GAPS
> >   lustre: lnd: rework map_on_demand behavior
> >   lustre: lnd: calculate qp max_send_wrs properly
> >   lustre: lnd: remove concurrent_sends tunable
> >   lustre: lnd: correct WR fast reg accounting
> >
> > Dmitry Eremin (1):
> >   lustre: o2iblnd: limit cap.max_send_wr for MLX5
> >
> > James Simmons (1):
> >   lustre: lnet: make LNET_MAX_IOV dependent on page size
> >
> > John L. Hammond (1):
> >   lustre: o2ib: use splice in kiblnd_peer_connect_failed()
> >
> >  .../staging/lustre/include/linux/lnet/lib-types.h  |  10 +-
> >  .../lustre/include/uapi/linux/lnet/lnet-types.h    |   3 -
> >  drivers/staging/lustre/lnet/Kconfig                |  10 --
> >  .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.c    | 138 +++++++++++++++------
> >  .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd.h    |  68 +++-------
> >  .../staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c | 133 ++++++++++++++------
> >  .../lustre/lnet/klnds/o2iblnd/o2iblnd_modparams.c  |  85 ++++++-------
> >  drivers/staging/lustre/lustre/include/lustre_net.h |   1 +
> >  8 files changed, 256 insertions(+), 192 deletions(-)
> >
> > -- 
> > 1.8.3.1
>