mbox series

[for-next,v5,00/12] rdma_rxe: API extensions

Message ID 20200918211517.5295-1-rpearson@hpe.com (mailing list archive)
Headers show
Series rdma_rxe: API extensions | expand

Message

Bob Pearson Sept. 18, 2020, 9:15 p.m. UTC
This patch series is a collection of API extensions for the rdma_rxe driver.
With this patch set installed there are no errors in pyverbs run-tests and
31 tests are skipped down from 56. The remaining skipped test cases include
	- XRC tests
	- ODP tests
	- Parent device tests
	- Import tests
	- Device memory
	- MLX5 specific tests
	- EFA tests

It continues from the previous (v4) set which implemented memory windows and
has had a number of individual patches picked up in for-next.

This set (v5) includes:
	Ported to current head of tree
	Memory windows patches not yet picked up
	kernel support for the extended user space APIs:
	  - ibv_query_device_ex
	  - ibv_create_cq_ex
	  - ibv_create_qp_ex
	Fixes for multicast which is not currently working

This patch set depends on a matching rdma-core user space library patch set.

In order to run correctly it is necessary to configure by hand the EUI64 link
local IPV6 address on systems which use a random link local address (like
Ubuntu).

Bob Pearson (12):
  rdma_rxe: Separate MEM into MR and MW objects.
  rdma_rxe: Enable MW objects
  rdma_rxe: Let pools support both keys and indices
  rdma_rxe: Add alloc_mw and dealloc_mw verbs
  rdma_rxe: Add bind_mw and invalidate_mw verbs
  rdma_rxe: Add memory access through MWs
  rdma_rxe: Add support for ibv_query_device_ex
  rdma_rxe: Add support for extended CQ operations
  rdma_rxe: Add support for extended QP operations
  rdma_rxe: Fix pool related bugs for multicast
  rdma_rxe: Fix multicast group allocation bug
  rdma_rxe: Fix bugs in the multicast receive path

 drivers/infiniband/sw/rxe/Makefile     |   1 +
 drivers/infiniband/sw/rxe/rxe.c        | 100 ++++--
 drivers/infiniband/sw/rxe/rxe_comp.c   |  12 +-
 drivers/infiniband/sw/rxe/rxe_cq.c     |  12 +-
 drivers/infiniband/sw/rxe/rxe_loc.h    |  43 ++-
 drivers/infiniband/sw/rxe/rxe_mcast.c  | 110 ++++---
 drivers/infiniband/sw/rxe/rxe_mr.c     | 350 +++++++++++----------
 drivers/infiniband/sw/rxe/rxe_mw.c     | 416 +++++++++++++++++++++++++
 drivers/infiniband/sw/rxe/rxe_opcode.c |  11 +-
 drivers/infiniband/sw/rxe/rxe_opcode.h |   1 -
 drivers/infiniband/sw/rxe/rxe_param.h  |  10 +-
 drivers/infiniband/sw/rxe/rxe_pool.c   | 330 +++++++++++++-------
 drivers/infiniband/sw/rxe/rxe_pool.h   | 109 +++++--
 drivers/infiniband/sw/rxe/rxe_recv.c   |  64 ++--
 drivers/infiniband/sw/rxe/rxe_req.c    | 113 ++++---
 drivers/infiniband/sw/rxe/rxe_resp.c   | 125 +++++---
 drivers/infiniband/sw/rxe/rxe_verbs.c  | 101 +++---
 drivers/infiniband/sw/rxe/rxe_verbs.h  |  64 ++--
 include/uapi/rdma/rdma_user_rxe.h      |  68 +++-
 19 files changed, 1448 insertions(+), 592 deletions(-)
 create mode 100644 drivers/infiniband/sw/rxe/rxe_mw.c

Comments

Jason Gunthorpe Sept. 18, 2020, 11:51 p.m. UTC | #1
On Fri, Sep 18, 2020 at 04:15:05PM -0500, Bob Pearson wrote:
> This patch series is a collection of API extensions for the rdma_rxe driver.
> With this patch set installed there are no errors in pyverbs run-tests and
> 31 tests are skipped down from 56. The remaining skipped test cases include
> 	- XRC tests
> 	- ODP tests
> 	- Parent device tests
> 	- Import tests
> 	- Device memory
> 	- MLX5 specific tests
> 	- EFA tests

It seems like a big improvement! Thanks!

Zhu, can you look through this too?

Jason

> It continues from the previous (v4) set which implemented memory windows and
> has had a number of individual patches picked up in for-next.
> 
> This set (v5) includes:
> 	Ported to current head of tree
> 	Memory windows patches not yet picked up
> 	kernel support for the extended user space APIs:
> 	  - ibv_query_device_ex
> 	  - ibv_create_cq_ex
> 	  - ibv_create_qp_ex
> 	Fixes for multicast which is not currently working

I would like to progress the simple independent obviously OK bits,
could you split them out?

Jason
Zhu Yanjun Sept. 19, 2020, 8:44 a.m. UTC | #2
On 9/19/2020 7:51 AM, Jason Gunthorpe wrote:
> On Fri, Sep 18, 2020 at 04:15:05PM -0500, Bob Pearson wrote:
>> This patch series is a collection of API extensions for the rdma_rxe driver.
>> With this patch set installed there are no errors in pyverbs run-tests and
>> 31 tests are skipped down from 56. The remaining skipped test cases include
>> 	- XRC tests
>> 	- ODP tests
>> 	- Parent device tests
>> 	- Import tests
>> 	- Device memory
>> 	- MLX5 specific tests
>> 	- EFA tests
> It seems like a big improvement! Thanks!
>
> Zhu, can you look through this too?

OK. It seems that a problem occurred in this patch set.

Hi, Bob

Please fix this problem. Thanks a lot.

Zhu Yanjun

>
> Jason
>
>> It continues from the previous (v4) set which implemented memory windows and
>> has had a number of individual patches picked up in for-next.
>>
>> This set (v5) includes:
>> 	Ported to current head of tree
>> 	Memory windows patches not yet picked up
>> 	kernel support for the extended user space APIs:
>> 	  - ibv_query_device_ex
>> 	  - ibv_create_cq_ex
>> 	  - ibv_create_qp_ex
>> 	Fixes for multicast which is not currently working
> I would like to progress the simple independent obviously OK bits,
> could you split them out?
>
> Jason
Zhu Yanjun Sept. 19, 2020, 8:46 a.m. UTC | #3
On 9/19/2020 4:44 PM, Zhu Yanjun wrote:
>
> On 9/19/2020 7:51 AM, Jason Gunthorpe wrote:
>> On Fri, Sep 18, 2020 at 04:15:05PM -0500, Bob Pearson wrote:
>>> This patch series is a collection of API extensions for the rdma_rxe 
>>> driver.
>>> With this patch set installed there are no errors in pyverbs 
>>> run-tests and
>>> 31 tests are skipped down from 56. The remaining skipped test cases 
>>> include
>>>     - XRC tests
>>>     - ODP tests
>>>     - Parent device tests
>>>     - Import tests
>>>     - Device memory
>>>     - MLX5 specific tests
>>>     - EFA tests
>> It seems like a big improvement! Thanks!
>>
>> Zhu, can you look through this too?
>
> OK. It seems that a problem occurred in this patch set.
>
> Hi, Bob
>
> Please fix this problem. Thanks a lot.

The problem mail is in the attachment. Please check and fix it.

Zhu Yanjun

>
> Zhu Yanjun
>
>>
>> Jason
>>
>>> It continues from the previous (v4) set which implemented memory 
>>> windows and
>>> has had a number of individual patches picked up in for-next.
>>>
>>> This set (v5) includes:
>>>     Ported to current head of tree
>>>     Memory windows patches not yet picked up
>>>     kernel support for the extended user space APIs:
>>>       - ibv_query_device_ex
>>>       - ibv_create_cq_ex
>>>       - ibv_create_qp_ex
>>>     Fixes for multicast which is not currently working
>> I would like to progress the simple independent obviously OK bits,
>> could you split them out?
>>
>> Jason
Hi Bob,

Thank you for the patch! Perhaps something to improve:

[auto build test WARNING on rdma/for-next]
[also build test WARNING on v5.9-rc5 next-20200918]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Bob-Pearson/rdma_rxe-API-extensions/20200919-051909
base:   https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git for-next
config: riscv-randconfig-r031-20200917 (attached as .config)
compiler: clang version 12.0.0 (https://github.com/llvm/llvm-project ed79827aea444e6995fb3d36abc2bfd36331773c)
reproduce (this is a W=1 build):
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # install riscv cross compiling tool for clang build
        # apt-get install binutils-riscv64-linux-gnu
        # save the attached .config to linux build tree
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=riscv 

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>

All warnings (new ones prefixed by >>):

   In file included from arch/riscv/include/asm/io.h:148:
   include/asm-generic/io.h:556:9: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           return inb(addr);
                  ^~~~~~~~~
   arch/riscv/include/asm/io.h:54:76: note: expanded from macro 'inb'
   #define inb(c)          ({ u8  __v; __io_pbr(); __v = readb_cpu((void*)(PCI_IOBASE + (c))); __io_par(__v); __v; })
                                                                           ~~~~~~~~~~ ^
   arch/riscv/include/asm/mmio.h:87:48: note: expanded from macro 'readb_cpu'
   #define readb_cpu(c)            ({ u8  __r = __raw_readb(c); __r; })
                                                            ^
   In file included from drivers/infiniband/sw/rxe/rxe_mr.c:7:
   In file included from drivers/infiniband/sw/rxe/rxe.h:16:
   In file included from include/linux/skbuff.h:31:
   In file included from include/linux/dma-mapping.h:11:
   In file included from include/linux/scatterlist.h:9:
   In file included from arch/riscv/include/asm/io.h:148:
   include/asm-generic/io.h:564:9: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           return inw(addr);
                  ^~~~~~~~~
   arch/riscv/include/asm/io.h:55:76: note: expanded from macro 'inw'
   #define inw(c)          ({ u16 __v; __io_pbr(); __v = readw_cpu((void*)(PCI_IOBASE + (c))); __io_par(__v); __v; })
                                                                           ~~~~~~~~~~ ^
   arch/riscv/include/asm/mmio.h:88:76: note: expanded from macro 'readw_cpu'
   #define readw_cpu(c)            ({ u16 __r = le16_to_cpu((__force __le16)__raw_readw(c)); __r; })
                                                                                        ^
   include/uapi/linux/byteorder/little_endian.h:36:51: note: expanded from macro '__le16_to_cpu'
   #define __le16_to_cpu(x) ((__force __u16)(__le16)(x))
                                                     ^
   In file included from drivers/infiniband/sw/rxe/rxe_mr.c:7:
   In file included from drivers/infiniband/sw/rxe/rxe.h:16:
   In file included from include/linux/skbuff.h:31:
   In file included from include/linux/dma-mapping.h:11:
   In file included from include/linux/scatterlist.h:9:
   In file included from arch/riscv/include/asm/io.h:148:
   include/asm-generic/io.h:572:9: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           return inl(addr);
                  ^~~~~~~~~
   arch/riscv/include/asm/io.h:56:76: note: expanded from macro 'inl'
   #define inl(c)          ({ u32 __v; __io_pbr(); __v = readl_cpu((void*)(PCI_IOBASE + (c))); __io_par(__v); __v; })
                                                                           ~~~~~~~~~~ ^
   arch/riscv/include/asm/mmio.h:89:76: note: expanded from macro 'readl_cpu'
   #define readl_cpu(c)            ({ u32 __r = le32_to_cpu((__force __le32)__raw_readl(c)); __r; })
                                                                                        ^
   include/uapi/linux/byteorder/little_endian.h:34:51: note: expanded from macro '__le32_to_cpu'
   #define __le32_to_cpu(x) ((__force __u32)(__le32)(x))
                                                     ^
   In file included from drivers/infiniband/sw/rxe/rxe_mr.c:7:
   In file included from drivers/infiniband/sw/rxe/rxe.h:16:
   In file included from include/linux/skbuff.h:31:
   In file included from include/linux/dma-mapping.h:11:
   In file included from include/linux/scatterlist.h:9:
   In file included from arch/riscv/include/asm/io.h:148:
   include/asm-generic/io.h:580:2: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           outb(value, addr);
           ^~~~~~~~~~~~~~~~~
   arch/riscv/include/asm/io.h:58:68: note: expanded from macro 'outb'
   #define outb(v,c)       ({ __io_pbw(); writeb_cpu((v),(void*)(PCI_IOBASE + (c))); __io_paw(); })
                                                                 ~~~~~~~~~~ ^
   arch/riscv/include/asm/mmio.h:91:52: note: expanded from macro 'writeb_cpu'
   #define writeb_cpu(v, c)        ((void)__raw_writeb((v), (c)))
                                                             ^
   In file included from drivers/infiniband/sw/rxe/rxe_mr.c:7:
   In file included from drivers/infiniband/sw/rxe/rxe.h:16:
   In file included from include/linux/skbuff.h:31:
   In file included from include/linux/dma-mapping.h:11:
   In file included from include/linux/scatterlist.h:9:
   In file included from arch/riscv/include/asm/io.h:148:
   include/asm-generic/io.h:588:2: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           outw(value, addr);
           ^~~~~~~~~~~~~~~~~
   arch/riscv/include/asm/io.h:59:68: note: expanded from macro 'outw'
   #define outw(v,c)       ({ __io_pbw(); writew_cpu((v),(void*)(PCI_IOBASE + (c))); __io_paw(); })
                                                                 ~~~~~~~~~~ ^
   arch/riscv/include/asm/mmio.h:92:76: note: expanded from macro 'writew_cpu'
   #define writew_cpu(v, c)        ((void)__raw_writew((__force u16)cpu_to_le16(v), (c)))
                                                                                     ^
   In file included from drivers/infiniband/sw/rxe/rxe_mr.c:7:
   In file included from drivers/infiniband/sw/rxe/rxe.h:16:
   In file included from include/linux/skbuff.h:31:
   In file included from include/linux/dma-mapping.h:11:
   In file included from include/linux/scatterlist.h:9:
   In file included from arch/riscv/include/asm/io.h:148:
   include/asm-generic/io.h:596:2: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           outl(value, addr);
           ^~~~~~~~~~~~~~~~~
   arch/riscv/include/asm/io.h:60:68: note: expanded from macro 'outl'
   #define outl(v,c)       ({ __io_pbw(); writel_cpu((v),(void*)(PCI_IOBASE + (c))); __io_paw(); })
                                                                 ~~~~~~~~~~ ^
   arch/riscv/include/asm/mmio.h:93:76: note: expanded from macro 'writel_cpu'
   #define writel_cpu(v, c)        ((void)__raw_writel((__force u32)cpu_to_le32(v), (c)))
                                                                                     ^
   In file included from drivers/infiniband/sw/rxe/rxe_mr.c:7:
   In file included from drivers/infiniband/sw/rxe/rxe.h:16:
   In file included from include/linux/skbuff.h:31:
   In file included from include/linux/dma-mapping.h:11:
   In file included from include/linux/scatterlist.h:9:
   In file included from arch/riscv/include/asm/io.h:148:
   include/asm-generic/io.h:1017:55: warning: performing pointer arithmetic on a null pointer has undefined behavior [-Wnull-pointer-arithmetic]
           return (port > MMIO_UPPER_LIMIT) ? NULL : PCI_IOBASE + port;
                                                     ~~~~~~~~~~ ^
>> drivers/infiniband/sw/rxe/rxe_mr.c:531:5: warning: no previous prototype for function 'rxe_invalidate_mr' [-Wmissing-prototypes]
   int rxe_invalidate_mr(struct rxe_qp *qp, struct rxe_mr *mr)
       ^
   drivers/infiniband/sw/rxe/rxe_mr.c:531:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
   int rxe_invalidate_mr(struct rxe_qp *qp, struct rxe_mr *mr)
   ^
   static 
   8 warnings generated.

# https://github.com/0day-ci/linux/commit/ea6524464bcccdce769f9a046224290a51dee110
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Bob-Pearson/rdma_rxe-API-extensions/20200919-051909
git checkout ea6524464bcccdce769f9a046224290a51dee110
vim +/rxe_invalidate_mr +531 drivers/infiniband/sw/rxe/rxe_mr.c

   530	
 > 531	int rxe_invalidate_mr(struct rxe_qp *qp, struct rxe_mr *mr)
   532	{
   533		mr->state = RXE_MEM_STATE_FREE;
   534		return 0;
   535	}
   536	

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
Bob Pearson Sept. 20, 2020, 9:13 p.m. UTC | #4
On 9/19/20 3:46 AM, Zhu Yanjun wrote:
> On 9/19/2020 4:44 PM, Zhu Yanjun wrote:
>>
>> On 9/19/2020 7:51 AM, Jason Gunthorpe wrote:
>>> On Fri, Sep 18, 2020 at 04:15:05PM -0500, Bob Pearson wrote:
>>>> This patch series is a collection of API extensions for the rdma_rxe driver.
>>>> With this patch set installed there are no errors in pyverbs run-tests and
>>>> 31 tests are skipped down from 56. The remaining skipped test cases include
>>>>     - XRC tests
>>>>     - ODP tests
>>>>     - Parent device tests
>>>>     - Import tests
>>>>     - Device memory
>>>>     - MLX5 specific tests
>>>>     - EFA tests
>>> It seems like a big improvement! Thanks!
>>>
>>> Zhu, can you look through this too?
>>
>> OK. It seems that a problem occurred in this patch set.
>>
>> Hi, Bob
>>
>> Please fix this problem. Thanks a lot.
> 
> The problem mail is in the attachment. Please check and fix it.
> 
> Zhu Yanjun
> 
>>
>> Zhu Yanjun
>>
>>>
>>> Jason
>>>
>>>> It continues from the previous (v4) set which implemented memory windows and
>>>> has had a number of individual patches picked up in for-next.
>>>>
>>>> This set (v5) includes:
>>>>     Ported to current head of tree
>>>>     Memory windows patches not yet picked up
>>>>     kernel support for the extended user space APIs:
>>>>       - ibv_query_device_ex
>>>>       - ibv_create_cq_ex
>>>>       - ibv_create_qp_ex
>>>>     Fixes for multicast which is not currently working
>>> I would like to progress the simple independent obviously OK bits,
>>> could you split them out?
>>>
>>> Jason
> 
> 
Zhu,

The missing prototype was a good catch. (That is resolved now.) But I can't figure out the null pointer arithmetic warnings. They are all in obscure distant header files. Do you have any insight about those?

Bob
Jason Gunthorpe Sept. 21, 2020, 2:10 p.m. UTC | #5
On Sun, Sep 20, 2020 at 04:13:29PM -0500, Bob Pearson wrote:

> The missing prototype was a good catch. (That is resolved now.) But
> I can't figure out the null pointer arithmetic warnings. They are
> all in obscure distant header files. Do you have any insight about
> those?

Just ignore thouse

Jason