| Message ID | 20200923102702.590008-1-yishaih@nvidia.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | verbs: Query GID table API | expand |
On 9/23/2020 1:26 PM, Yishai Hadas wrote: > When an application is not using RDMA CM and if it is using multiple RDMA > devices with one or more RoCE ports, finding the right GID table entry is a > long process. > > For example, with two RoCE dual-port devices in a system, when IP failover is > used between two RoCE ports, searching a suitable GID entry for a given source > IP, matching netdevice of given RoCEv1/v2 type requires iterating over all 4 > ports * 256 entry GID table. > > Even though the best first match GID table for given criteria is used, when the > matching entry is on the 4th port, it requires reading 3 ports * 256 entries * > 3 files (GID, netdev, type) = 2304 files. The GID table needs to be referred > on every QP creation during IP failover on other netdevice of an RDMA device. > > We introduce this series of patches, which introduces an API to query the > complete GID tables of an RDMA device, that returns all valid GID table > entries. > > This is done through single ioctl, eliminating 2304 read, 2304 open and 2304 > close system calls to just a total of 2 calls (one for each device). > > While at it, we also introduce an API to query an individual GID entry over > ioctl interface, which provides all GID attributes information. > > The APIs are based on the below RFC [1], the matching kernel part was sent to > rdma-next. > > PR was sent as well [2]. > > [1] https://www.spinics.net/lists/linux-rdma/msg91825.html > [2] https://github.com/linux-rdma/rdma-core/pull/828 > > Changelog: > v1: > * Move some GIDs stuff to cmd_device.c with better function's names. > * Cleanup in few places. > > Avihai Horon (7): > verbs: Change the name of enum ibv_gid_type > verbs: Introduce a new query GID entry API > verbs: Implement ibv_query_gid and ibv_query_gid_type over ioctl > verbs: Optimize ibv_query_gid and ibv_query_gid_type > verbs: Introduce a new query GID table API > pyverbs: Add query_gid_table and query_gid_ex methods > tests: Add tests for ibv_query_gid_table and ibv_query_gid_ex > > Yishai Hadas (1): > Update kernel headers > > debian/libibverbs1.symbols | 3 + > kernel-headers/rdma/ib_user_ioctl_cmds.h | 16 ++ > kernel-headers/rdma/ib_user_ioctl_verbs.h | 14 ++ > kernel-headers/rdma/ib_user_verbs.h | 11 ++ > kernel-headers/rdma/rdma_user_rxe.h | 6 +- > libibverbs/CMakeLists.txt | 2 +- > libibverbs/cmd_device.c | 314 ++++++++++++++++++++++++++++++ > libibverbs/driver.h | 19 +- > libibverbs/examples/devinfo.c | 14 +- > libibverbs/libibverbs.map.in | 6 + > libibverbs/man/CMakeLists.txt | 2 + > libibverbs/man/ibv_query_gid_ex.3.md | 93 +++++++++ > libibverbs/man/ibv_query_gid_table.3.md | 73 +++++++ > libibverbs/verbs.c | 105 ++++------ > libibverbs/verbs.h | 45 +++++ > providers/mlx5/verbs.c | 2 +- > pyverbs/device.pxd | 3 + > pyverbs/device.pyx | 108 +++++++++- > pyverbs/libibverbs.pxd | 15 +- > pyverbs/libibverbs_enums.pxd | 11 +- > tests/base.py | 3 +- > tests/test_device.py | 32 +++ > 22 files changed, 804 insertions(+), 93 deletions(-) > create mode 100644 libibverbs/man/ibv_query_gid_ex.3.md > create mode 100644 libibverbs/man/ibv_query_gid_table.3.md > The PR was merged. Thanks, Yishai