mbox series

[00/32] Convert the Infiniband subsystem to XArray

Message ID 20190221002107.22625-1-willy@infradead.org (mailing list archive)
Headers show
Series Convert the Infiniband subsystem to XArray | expand

Message

Matthew Wilcox (Oracle) Feb. 21, 2019, 12:20 a.m. UTC
I intend to remove the IDR and the radix tree interfaces from Linux.
Converting each user from either the IDR or radix tree interface varies
from the trivial 1:1 replacement to a complete rewrite of the locking.
Despite the best efforts of our automated testers (who have caught many
of my mistakes), I cannot claim that my conversions of code are free
from bugs.

Please check these patches over carefully and test them; there may be
off-by-one errors, locking mistakes, or various other failures on my part.

 drivers/infiniband/core/cm.c                    |   40 +---
 drivers/infiniband/core/cma.c                   |   44 +---
 drivers/infiniband/core/mad.c                   |   39 +---
 drivers/infiniband/core/rdma_core.c             |   78 +-------
 drivers/infiniband/core/rdma_core.h             |    8 
 drivers/infiniband/core/sa_query.c              |   43 +---
 drivers/infiniband/core/ucm.c                   |   34 +--
 drivers/infiniband/core/ucma.c                  |   82 +++------
 drivers/infiniband/core/uverbs.h                |    4 
 drivers/infiniband/core/uverbs_ioctl.c          |   67 +------
 drivers/infiniband/core/uverbs_uapi.c           |  191 +++++++++------------
 drivers/infiniband/hw/bnxt_re/ib_verbs.c        |    2 
 drivers/infiniband/hw/cxgb3/iwch.c              |   53 ++---
 drivers/infiniband/hw/cxgb3/iwch.h              |   38 ----
 drivers/infiniband/hw/cxgb3/iwch_ev.c           |   18 -
 drivers/infiniband/hw/cxgb3/iwch_mem.c          |    2 
 drivers/infiniband/hw/cxgb3/iwch_provider.c     |   16 -
 drivers/infiniband/hw/cxgb4/cm.c                |   67 ++++---
 drivers/infiniband/hw/cxgb4/cq.c                |    6 
 drivers/infiniband/hw/cxgb4/device.c            |  217 ++++++++++--------------
 drivers/infiniband/hw/cxgb4/ev.c                |   18 -
 drivers/infiniband/hw/cxgb4/iw_cxgb4.h          |   77 --------
 drivers/infiniband/hw/cxgb4/mem.c               |   16 -
 drivers/infiniband/hw/cxgb4/qp.c                |   33 +--
 drivers/infiniband/hw/hfi1/chip.c               |   16 -
 drivers/infiniband/hw/hfi1/debugfs.c            |    8 
 drivers/infiniband/hw/hfi1/driver.c             |   10 -
 drivers/infiniband/hw/hfi1/hfi.h                |    8 
 drivers/infiniband/hw/hfi1/init.c               |   51 -----
 drivers/infiniband/hw/hfi1/verbs.c              |    8 
 drivers/infiniband/hw/hfi1/vnic_main.c          |   15 -
 drivers/infiniband/hw/hns/hns_roce_cq.c         |   33 +--
 drivers/infiniband/hw/hns/hns_roce_device.h     |    9 
 drivers/infiniband/hw/hns/hns_roce_qp.c         |   51 +----
 drivers/infiniband/hw/mlx4/cm.c                 |   36 +--
 drivers/infiniband/hw/mlx4/cq.c                 |    2 
 drivers/infiniband/hw/mlx4/mlx4_ib.h            |    5 
 drivers/infiniband/hw/mlx5/cq.c                 |    4 
 drivers/infiniband/hw/mlx5/mr.c                 |   10 -
 drivers/infiniband/hw/mlx5/srq.h                |    5 
 drivers/infiniband/hw/mlx5/srq_cmd.c            |   27 --
 drivers/infiniband/hw/ocrdma/ocrdma_main.c      |   16 -
 drivers/infiniband/hw/qedr/main.c               |   10 -
 drivers/infiniband/hw/qedr/qedr.h               |   11 -
 drivers/infiniband/hw/qedr/qedr_iw_cm.c         |   10 -
 drivers/infiniband/hw/qedr/verbs.c              |   36 ---
 drivers/infiniband/hw/qib/qib.h                 |    4 
 drivers/infiniband/hw/qib/qib_driver.c          |   20 --
 drivers/infiniband/hw/qib/qib_fs.c              |   12 -
 drivers/infiniband/hw/qib/qib_iba7322.c         |    4 
 drivers/infiniband/hw/qib/qib_init.c            |   55 +-----
 drivers/infiniband/ulp/opa_vnic/opa_vnic_vema.c |   60 ++----
 52 files changed, 633 insertions(+), 1096 deletions(-)

Substantive interface changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 - The IDR and radix tree required callers to handle their own locking.
   The XArray embeds a spinlock which is taken for modifications to
   the data structure; plain lookups occur under the RCU read lock or
   under the spinlock.
 - You can take the spinlock yourself (xa_lock() and friends) to protect
   related data.
 - idr_alloc() returned -ENOSPC, radix_tree_insert() returned -EEXIST.
   xa_insert() and xa_alloc() return -EBUSY.
 - The search keys which the radix tree calls "tags", the XArray calls
   "marks".
 - There is no preloading in the XArray API.  If your locking is
   exceptionally complicated, you may need to use xa_reserve(), but
   there are only 6 callers of xa_reserve(), so it's quite uncommon.
 - The radix tree provided GFP flags as part of the tree definition;
   the XArray (like the IDR) passes GFP flags at the point of allocation.
 - radix_tree_insert() of a NULL pointer was not well-specified.  The
   XArray treats it as reserving the entry (it reads back as NULL but
   a subsequent xa_insert() to that slot will fail).
 - xa_alloc_cyclic() returns 1 if the allocation wraps, unlike
   idr_alloc_cyclic() which provides no indication.
 - There is no equivalent to idr_for_each(); the xa_for_each() iterator
   is similar to idr_for_each_entry().
 - idr_replace() has no exact equivalent.  Some users relied on its exact
   semantics of only storing if the entry was non-NULL, but all users of
   idr_replace() were able to use xa_store().
 - The family of radix tree gang lookup functions have been replaced with
   xa_extract().

Matthew Wilcox (32):
  mlx4: Convert srq_table->tree to XArray
  mlx5: Convert mlx5_srq_table to XArray
  mlx5: Convert mkey_table to XArray
  RDMA/hns: Convert cq_table to XArray
  RDMA/hns: Convert qp_table_tree to XArray
  infiniband/core: Convert uverbs to XArray
  IB/mad: Convert ib_mad_clients to XArray
  RDMA/cm: Convert local_id_table to XArray
  mlx4: Convert pv_id_table to XArray
  uverbs: Convert idr to XArray
  ib core: Convert query_idr to XArray
  cxgb3: Convert cqidr to XArray
  cxgb3: Convert qpidr to XArray
  cxgb3: Convert mmidr to XArray
  cxgb4: Convert cqidr to XArray
  cxgb4: Convert qpidr to XArray
  cxgb4: Convert mmidr to XArray
  cxgb4: Convert hwtid_idr to XArray
  cxgb4: Convert atid_idr to XArray
  cxgb4: Convert stid_idr to XArray
  hfi1: Convert hfi1_unit_table to XArray
  hfi1: Convert vesw_idr to XArray
  qedr: Convert qpidr to XArray
  qedr: Convert srqidr to XArray
  qib: Convert qib_unit_table to XArray
  opa_vnic: Convert vport_idr to XArray
  ocrdma: Convert ocrdma_dev_id to IDA
  ucm: Convert ctx_id_table to XArray
  ucma: Convert multicast_idr to XArray
  ucma: Convert ctx_idr to XArray
  cma: Convert portspace IDRs to XArray
  ib/bnxt: Remove mention of idr_alloc from comment

Comments

Jason Gunthorpe Feb. 21, 2019, 5:43 p.m. UTC | #1
On Wed, Feb 20, 2019 at 04:20:35PM -0800, Matthew Wilcox wrote:
> I intend to remove the IDR and the radix tree interfaces from Linux.
> Converting each user from either the IDR or radix tree interface varies
> from the trivial 1:1 replacement to a complete rewrite of the locking.
> Despite the best efforts of our automated testers (who have caught many
> of my mistakes), I cannot claim that my conversions of code are free
> from bugs.

Wow, big.. I didn't notice all these in your git tree before. This is
all based on your xarray tree, so I can't merge anything until -rc1
comes out again.

But people should be looking at this and we can chip away at it in 3
weeks.

Thanks,
Jason
Jason Gunthorpe Feb. 21, 2019, 7:27 p.m. UTC | #2
On Wed, Feb 20, 2019 at 04:20:35PM -0800, Matthew Wilcox wrote:
> I intend to remove the IDR and the radix tree interfaces from Linux.
> Converting each user from either the IDR or radix tree interface varies
> from the trivial 1:1 replacement to a complete rewrite of the locking.
> Despite the best efforts of our automated testers (who have caught many
> of my mistakes), I cannot claim that my conversions of code are free
> from bugs.
> 
> Please check these patches over carefully and test them; there may be
> off-by-one errors, locking mistakes, or various other failures on my part.

RDMA Driver Maintainers:

I read through these and the driver changes look reasonable. Please
review and ack. If I hear no objection I will likely apply Matt's
patches for drivers around rc3/4

Thanks
Jason


>  drivers/infiniband/core/cm.c                    |   40 +---
>  drivers/infiniband/core/cma.c                   |   44 +---
>  drivers/infiniband/core/mad.c                   |   39 +---
>  drivers/infiniband/core/rdma_core.c             |   78 +-------
>  drivers/infiniband/core/rdma_core.h             |    8 
>  drivers/infiniband/core/sa_query.c              |   43 +---
>  drivers/infiniband/core/ucm.c                   |   34 +--
>  drivers/infiniband/core/ucma.c                  |   82 +++------
>  drivers/infiniband/core/uverbs.h                |    4 
>  drivers/infiniband/core/uverbs_ioctl.c          |   67 +------
>  drivers/infiniband/core/uverbs_uapi.c           |  191 +++++++++------------
>  drivers/infiniband/hw/bnxt_re/ib_verbs.c        |    2 
>  drivers/infiniband/hw/cxgb3/iwch.c              |   53 ++---
>  drivers/infiniband/hw/cxgb3/iwch.h              |   38 ----
>  drivers/infiniband/hw/cxgb3/iwch_ev.c           |   18 -
>  drivers/infiniband/hw/cxgb3/iwch_mem.c          |    2 
>  drivers/infiniband/hw/cxgb3/iwch_provider.c     |   16 -
>  drivers/infiniband/hw/cxgb4/cm.c                |   67 ++++---
>  drivers/infiniband/hw/cxgb4/cq.c                |    6 
>  drivers/infiniband/hw/cxgb4/device.c            |  217 ++++++++++--------------
>  drivers/infiniband/hw/cxgb4/ev.c                |   18 -
>  drivers/infiniband/hw/cxgb4/iw_cxgb4.h          |   77 --------
>  drivers/infiniband/hw/cxgb4/mem.c               |   16 -
>  drivers/infiniband/hw/cxgb4/qp.c                |   33 +--
>  drivers/infiniband/hw/hfi1/chip.c               |   16 -
>  drivers/infiniband/hw/hfi1/debugfs.c            |    8 
>  drivers/infiniband/hw/hfi1/driver.c             |   10 -
>  drivers/infiniband/hw/hfi1/hfi.h                |    8 
>  drivers/infiniband/hw/hfi1/init.c               |   51 -----
>  drivers/infiniband/hw/hfi1/verbs.c              |    8 
>  drivers/infiniband/hw/hfi1/vnic_main.c          |   15 -
>  drivers/infiniband/hw/hns/hns_roce_cq.c         |   33 +--
>  drivers/infiniband/hw/hns/hns_roce_device.h     |    9 
>  drivers/infiniband/hw/hns/hns_roce_qp.c         |   51 +----
>  drivers/infiniband/hw/mlx4/cm.c                 |   36 +--
>  drivers/infiniband/hw/mlx4/cq.c                 |    2 
>  drivers/infiniband/hw/mlx4/mlx4_ib.h            |    5 
>  drivers/infiniband/hw/mlx5/cq.c                 |    4 
>  drivers/infiniband/hw/mlx5/mr.c                 |   10 -
>  drivers/infiniband/hw/mlx5/srq.h                |    5 
>  drivers/infiniband/hw/mlx5/srq_cmd.c            |   27 --
>  drivers/infiniband/hw/ocrdma/ocrdma_main.c      |   16 -
>  drivers/infiniband/hw/qedr/main.c               |   10 -
>  drivers/infiniband/hw/qedr/qedr.h               |   11 -
>  drivers/infiniband/hw/qedr/qedr_iw_cm.c         |   10 -
>  drivers/infiniband/hw/qedr/verbs.c              |   36 ---
>  drivers/infiniband/hw/qib/qib.h                 |    4 
>  drivers/infiniband/hw/qib/qib_driver.c          |   20 --
>  drivers/infiniband/hw/qib/qib_fs.c              |   12 -
>  drivers/infiniband/hw/qib/qib_iba7322.c         |    4 
>  drivers/infiniband/hw/qib/qib_init.c            |   55 +-----
>  drivers/infiniband/ulp/opa_vnic/opa_vnic_vema.c |   60 ++----
>  52 files changed, 633 insertions(+), 1096 deletions(-)
Jason Gunthorpe March 25, 2019, 11:58 p.m. UTC | #3
On Wed, Feb 20, 2019 at 04:20:35PM -0800, Matthew Wilcox wrote:
> 
> Matthew Wilcox (32):
>   cxgb3: Convert cqidr to XArray
>   cxgb3: Convert qpidr to XArray
>   cxgb3: Convert mmidr to XArray
>   cxgb4: Convert cqidr to XArray
>   cxgb4: Convert qpidr to XArray
>   cxgb4: Convert mmidr to XArray
>   cxgb4: Convert hwtid_idr to XArray
>   cxgb4: Convert atid_idr to XArray
>   cxgb4: Convert stid_idr to XArray

I applied these patches, with a slight change to the goto names and
WARN_ON changes instead of idr_destroy to for-next

The others are still in patchworks

Thanks,
Jason
Jason Gunthorpe March 26, 2019, 3:04 p.m. UTC | #4
On Wed, Feb 20, 2019 at 04:20:35PM -0800, Matthew Wilcox wrote:

> Matthew Wilcox (32):
>   mlx4: Convert pv_id_table to XArray
>   mlx5: Convert mlx5_srq_table to XArray
>   IB/mad: Convert ib_mad_clients to XArray
>   RDMA/cm: Convert local_id_table to XArray
>   ib core: Convert query_idr to XArray
>   ucm: Convert ctx_id_table to XArray
>   cma: Convert portspace IDRs to XArray

I applied these patches to for-next with the idr_destroy -> WARN_ON
conversion

The rest of the patches are still in patchworks

Thanks,
Jason
Jason Gunthorpe March 29, 2019, 6:02 p.m. UTC | #5
On Wed, Feb 20, 2019 at 04:20:35PM -0800, Matthew Wilcox wrote:
> Matthew Wilcox (32):
>   RDMA/hns: Convert cq_table to XArray
>   RDMA/hns: Convert qp_table_tree to XArray
>   hfi1: Convert vesw_idr to XArray
>   qedr: Convert qpidr to XArray
>   qedr: Convert srqidr to XArray

Applied to for-next, thanks. Looks to me like the hns patches fix some
small races as well.

There are 10 more left on patchworks which need some combination of
more detailed review, respin or discussion closure.

Thanks,
Jason