| Message ID | 1557236319-9986-1-git-send-email-maxg@mellanox.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Introduce new API for T10-PI offload | expand |
On Tue, May 07, 2019 at 04:38:14PM +0300, Max Gurtovoy wrote: > Israel Rukshin (12): > RDMA/core: Introduce IB_MR_TYPE_INTEGRITY and ib_alloc_mr_integrity > API > IB/iser: Refactor iscsi_iser_check_protection function > IB/iser: Use IB_WR_REG_MR_INTEGRITY for PI handover > IB/iser: Unwind WR union at iser_tx_desc > IB/iser: Remove unused sig_attrs argument > IB/isert: Remove unused sig_attrs argument > RDMA/core: Add an integrity MR pool support > RDMA/rw: Fix doc typo > RDMA/rw: Print the correct number of sig MRs > RDMA/rw: Use IB_WR_REG_MR_INTEGRITY for PI handover > RDMA/core: Remove unused IB_WR_REG_SIG_MR code > RDMA/mlx5: Improve PI handover performance > > Max Gurtovoy (13): > RDMA/core: Introduce new header file for signature operations > RDMA/core: Save the MR type in the ib_mr structure > RDMA/core: Introduce ib_map_mr_sg_pi to map data/protection sgl's > RDMA/core: Add signature attrs element for ib_mr structure > RDMA/mlx5: Implement mlx5_ib_map_mr_sg_pi and > mlx5_ib_alloc_mr_integrity > RDMA/mlx5: Add attr for max number page list length for PI operation > RDMA/mlx5: Pass UMR segment flags instead of boolean > RDMA/mlx5: Update set_sig_data_segment attribute for new signature API > RDMA/mlx5: Introduce and implement new IB_WR_REG_MR_INTEGRITY work > request > RDMA/mlx5: Move signature_en attribute from mlx5_qp to ib_qp > RDMA/core: Validate signature handover device cap > RDMA/rw: Add info regarding SG count failure > RDMA/mlx5: Use PA mapping for PI handover Max this is really too many patches now, can you please split this up. Can several patches be applied right now as bug fixes like: RDMA/rw: Fix doc typo RDMA/rw: Print the correct number of sig MRs RDMA/core: Remove unused IB_WR_REG_SIG_MR code RDMA/rw: Add info regarding SG count failure ?? Thanks, Jason
On 5/7/2019 4:42 PM, Jason Gunthorpe wrote: > On Tue, May 07, 2019 at 04:38:14PM +0300, Max Gurtovoy wrote: >> Israel Rukshin (12): >> RDMA/core: Introduce IB_MR_TYPE_INTEGRITY and ib_alloc_mr_integrity >> API >> IB/iser: Refactor iscsi_iser_check_protection function >> IB/iser: Use IB_WR_REG_MR_INTEGRITY for PI handover >> IB/iser: Unwind WR union at iser_tx_desc >> IB/iser: Remove unused sig_attrs argument >> IB/isert: Remove unused sig_attrs argument >> RDMA/core: Add an integrity MR pool support >> RDMA/rw: Fix doc typo >> RDMA/rw: Print the correct number of sig MRs >> RDMA/rw: Use IB_WR_REG_MR_INTEGRITY for PI handover >> RDMA/core: Remove unused IB_WR_REG_SIG_MR code >> RDMA/mlx5: Improve PI handover performance >> >> Max Gurtovoy (13): >> RDMA/core: Introduce new header file for signature operations >> RDMA/core: Save the MR type in the ib_mr structure >> RDMA/core: Introduce ib_map_mr_sg_pi to map data/protection sgl's >> RDMA/core: Add signature attrs element for ib_mr structure >> RDMA/mlx5: Implement mlx5_ib_map_mr_sg_pi and >> mlx5_ib_alloc_mr_integrity >> RDMA/mlx5: Add attr for max number page list length for PI operation >> RDMA/mlx5: Pass UMR segment flags instead of boolean >> RDMA/mlx5: Update set_sig_data_segment attribute for new signature API >> RDMA/mlx5: Introduce and implement new IB_WR_REG_MR_INTEGRITY work >> request >> RDMA/mlx5: Move signature_en attribute from mlx5_qp to ib_qp >> RDMA/core: Validate signature handover device cap >> RDMA/rw: Add info regarding SG count failure >> RDMA/mlx5: Use PA mapping for PI handover > Max this is really too many patches now, can you please split this > up. > > Can several patches be applied right now as bug fixes like: > > RDMA/rw: Fix doc typo > RDMA/rw: Print the correct number of sig MRs > RDMA/core: Remove unused IB_WR_REG_SIG_MR code > RDMA/rw: Add info regarding SG count failure > > ?? Yes we can. Except of "RDMA/core: Remove unused IB_WR_REG_SIG_MR code". Patches that also can be merged now are: "IB/iser: Remove unused sig_attrs argument" "IB/isert: Remove unused sig_attrs argument" what is the merge plan ? are we going to squeeze this to 5.2 or 5.3 ? which branch should we sent the 5 patches from above ? > > Thanks, > Jason
On Tue, May 07, 2019 at 04:54:56PM +0300, Max Gurtovoy wrote: > > On 5/7/2019 4:42 PM, Jason Gunthorpe wrote: > > On Tue, May 07, 2019 at 04:38:14PM +0300, Max Gurtovoy wrote: > > > Israel Rukshin (12): > > > RDMA/core: Introduce IB_MR_TYPE_INTEGRITY and ib_alloc_mr_integrity > > > API > > > IB/iser: Refactor iscsi_iser_check_protection function > > > IB/iser: Use IB_WR_REG_MR_INTEGRITY for PI handover > > > IB/iser: Unwind WR union at iser_tx_desc > > > IB/iser: Remove unused sig_attrs argument > > > IB/isert: Remove unused sig_attrs argument > > > RDMA/core: Add an integrity MR pool support > > > RDMA/rw: Fix doc typo > > > RDMA/rw: Print the correct number of sig MRs > > > RDMA/rw: Use IB_WR_REG_MR_INTEGRITY for PI handover > > > RDMA/core: Remove unused IB_WR_REG_SIG_MR code > > > RDMA/mlx5: Improve PI handover performance > > > > > > Max Gurtovoy (13): > > > RDMA/core: Introduce new header file for signature operations > > > RDMA/core: Save the MR type in the ib_mr structure > > > RDMA/core: Introduce ib_map_mr_sg_pi to map data/protection sgl's > > > RDMA/core: Add signature attrs element for ib_mr structure > > > RDMA/mlx5: Implement mlx5_ib_map_mr_sg_pi and > > > mlx5_ib_alloc_mr_integrity > > > RDMA/mlx5: Add attr for max number page list length for PI operation > > > RDMA/mlx5: Pass UMR segment flags instead of boolean > > > RDMA/mlx5: Update set_sig_data_segment attribute for new signature API > > > RDMA/mlx5: Introduce and implement new IB_WR_REG_MR_INTEGRITY work > > > request > > > RDMA/mlx5: Move signature_en attribute from mlx5_qp to ib_qp > > > RDMA/core: Validate signature handover device cap > > > RDMA/rw: Add info regarding SG count failure > > > RDMA/mlx5: Use PA mapping for PI handover > > Max this is really too many patches now, can you please split this > > up. > > > > Can several patches be applied right now as bug fixes like: > > > > RDMA/rw: Fix doc typo > > RDMA/rw: Print the correct number of sig MRs > > RDMA/core: Remove unused IB_WR_REG_SIG_MR code > > RDMA/rw: Add info regarding SG count failure > > > > ?? > > Yes we can. Except of "RDMA/core: Remove unused IB_WR_REG_SIG_MR code". > > Patches that also can be merged now are: > > "IB/iser: Remove unused sig_attrs argument" > > "IB/isert: Remove unused sig_attrs argument" > > what is the merge plan ? > > are we going to squeeze this to 5.2 or 5.3 ? The 5.2 merge window is now open so it will not make 5.2 > which branch should we sent the 5 patches from above ? It is probably best to repost this thing split up against 5.2-rc1 in two weeks, I'll drop it off patchworks until then. Jason
On 5/7/2019 5:08 PM, Jason Gunthorpe wrote: > On Tue, May 07, 2019 at 04:54:56PM +0300, Max Gurtovoy wrote: >> On 5/7/2019 4:42 PM, Jason Gunthorpe wrote: >>> On Tue, May 07, 2019 at 04:38:14PM +0300, Max Gurtovoy wrote: >>>> Israel Rukshin (12): >>>> RDMA/core: Introduce IB_MR_TYPE_INTEGRITY and ib_alloc_mr_integrity >>>> API >>>> IB/iser: Refactor iscsi_iser_check_protection function >>>> IB/iser: Use IB_WR_REG_MR_INTEGRITY for PI handover >>>> IB/iser: Unwind WR union at iser_tx_desc >>>> IB/iser: Remove unused sig_attrs argument >>>> IB/isert: Remove unused sig_attrs argument >>>> RDMA/core: Add an integrity MR pool support >>>> RDMA/rw: Fix doc typo >>>> RDMA/rw: Print the correct number of sig MRs >>>> RDMA/rw: Use IB_WR_REG_MR_INTEGRITY for PI handover >>>> RDMA/core: Remove unused IB_WR_REG_SIG_MR code >>>> RDMA/mlx5: Improve PI handover performance >>>> >>>> Max Gurtovoy (13): >>>> RDMA/core: Introduce new header file for signature operations >>>> RDMA/core: Save the MR type in the ib_mr structure >>>> RDMA/core: Introduce ib_map_mr_sg_pi to map data/protection sgl's >>>> RDMA/core: Add signature attrs element for ib_mr structure >>>> RDMA/mlx5: Implement mlx5_ib_map_mr_sg_pi and >>>> mlx5_ib_alloc_mr_integrity >>>> RDMA/mlx5: Add attr for max number page list length for PI operation >>>> RDMA/mlx5: Pass UMR segment flags instead of boolean >>>> RDMA/mlx5: Update set_sig_data_segment attribute for new signature API >>>> RDMA/mlx5: Introduce and implement new IB_WR_REG_MR_INTEGRITY work >>>> request >>>> RDMA/mlx5: Move signature_en attribute from mlx5_qp to ib_qp >>>> RDMA/core: Validate signature handover device cap >>>> RDMA/rw: Add info regarding SG count failure >>>> RDMA/mlx5: Use PA mapping for PI handover >>> Max this is really too many patches now, can you please split this >>> up. >>> >>> Can several patches be applied right now as bug fixes like: >>> >>> RDMA/rw: Fix doc typo >>> RDMA/rw: Print the correct number of sig MRs >>> RDMA/core: Remove unused IB_WR_REG_SIG_MR code >>> RDMA/rw: Add info regarding SG count failure >>> >>> ?? >> Yes we can. Except of "RDMA/core: Remove unused IB_WR_REG_SIG_MR code". >> >> Patches that also can be merged now are: >> >> "IB/iser: Remove unused sig_attrs argument" >> >> "IB/isert: Remove unused sig_attrs argument" >> >> what is the merge plan ? >> >> are we going to squeeze this to 5.2 or 5.3 ? > The 5.2 merge window is now open so it will not make 5.2 Can we merge it to your for-5.3 branch after getting green light on this series ? > >> which branch should we sent the 5 patches from above ? > It is probably best to repost this thing split up against 5.2-rc1 in > two weeks, I'll drop it off patchworks until then. Sure, but please approve it to avoid another review cycle. > Jason
On Tue, May 07, 2019 at 05:17:46PM +0300, Max Gurtovoy wrote: > > On 5/7/2019 5:08 PM, Jason Gunthorpe wrote: > > On Tue, May 07, 2019 at 04:54:56PM +0300, Max Gurtovoy wrote: > > > On 5/7/2019 4:42 PM, Jason Gunthorpe wrote: > > > > On Tue, May 07, 2019 at 04:38:14PM +0300, Max Gurtovoy wrote: > > > > > Israel Rukshin (12): > > > > > RDMA/core: Introduce IB_MR_TYPE_INTEGRITY and ib_alloc_mr_integrity > > > > > API > > > > > IB/iser: Refactor iscsi_iser_check_protection function > > > > > IB/iser: Use IB_WR_REG_MR_INTEGRITY for PI handover > > > > > IB/iser: Unwind WR union at iser_tx_desc > > > > > IB/iser: Remove unused sig_attrs argument > > > > > IB/isert: Remove unused sig_attrs argument > > > > > RDMA/core: Add an integrity MR pool support > > > > > RDMA/rw: Fix doc typo > > > > > RDMA/rw: Print the correct number of sig MRs > > > > > RDMA/rw: Use IB_WR_REG_MR_INTEGRITY for PI handover > > > > > RDMA/core: Remove unused IB_WR_REG_SIG_MR code > > > > > RDMA/mlx5: Improve PI handover performance > > > > > > > > > > Max Gurtovoy (13): > > > > > RDMA/core: Introduce new header file for signature operations > > > > > RDMA/core: Save the MR type in the ib_mr structure > > > > > RDMA/core: Introduce ib_map_mr_sg_pi to map data/protection sgl's > > > > > RDMA/core: Add signature attrs element for ib_mr structure > > > > > RDMA/mlx5: Implement mlx5_ib_map_mr_sg_pi and > > > > > mlx5_ib_alloc_mr_integrity > > > > > RDMA/mlx5: Add attr for max number page list length for PI operation > > > > > RDMA/mlx5: Pass UMR segment flags instead of boolean > > > > > RDMA/mlx5: Update set_sig_data_segment attribute for new signature API > > > > > RDMA/mlx5: Introduce and implement new IB_WR_REG_MR_INTEGRITY work > > > > > request > > > > > RDMA/mlx5: Move signature_en attribute from mlx5_qp to ib_qp > > > > > RDMA/core: Validate signature handover device cap > > > > > RDMA/rw: Add info regarding SG count failure > > > > > RDMA/mlx5: Use PA mapping for PI handover > > > > Max this is really too many patches now, can you please split this > > > > up. > > > > > > > > Can several patches be applied right now as bug fixes like: > > > > > > > > RDMA/rw: Fix doc typo > > > > RDMA/rw: Print the correct number of sig MRs > > > > RDMA/core: Remove unused IB_WR_REG_SIG_MR code > > > > RDMA/rw: Add info regarding SG count failure > > > > > > > > ?? > > > Yes we can. Except of "RDMA/core: Remove unused IB_WR_REG_SIG_MR code". > > > > > > Patches that also can be merged now are: > > > > > > "IB/iser: Remove unused sig_attrs argument" > > > > > > "IB/isert: Remove unused sig_attrs argument" > > > > > > what is the merge plan ? > > > > > > are we going to squeeze this to 5.2 or 5.3 ? > > The 5.2 merge window is now open so it will not make 5.2 > > Can we merge it to your for-5.3 branch after getting green light on this > series ? That branch will start only when rc1 comes out. > > > which branch should we sent the 5 patches from above ? > > It is probably best to repost this thing split up against 5.2-rc1 in > > two weeks, I'll drop it off patchworks until then. > > Sure, but please approve it to avoid another review cycle. Well you almost certainly will need to repost it after rebasing it.. So I need new patches - both others can Ack your approach.. Jason
Hello Sagi, Christoph, Jason, Doug, Leon and Co This patchset adds a new verbs API for T10-PI offload and implementation for iSER initiator and iSER target (NVMe-oF/RDMA host side was completed and will be sent on a different patchset). This set starts with a few preparation commits to the RDMA/core layer. It continues with introducing a new MR type IB_MR_TYPE_INTEGRITY. Using this MR all the needed mappings will be done in the low level drivers and not be visible to the ULP. Later patches implement the needed functionality in the mlx5 layer. As suggested by Sagi, in the new API, the mlx5 driver will allocate a single internal memory region for the UMR operation to register both PI and data SG lists and it will look like: data start meta start | | ------------------------- |d1|d2|d3|d4|m1|m2|m3|m4| ------------------------- The sig_mr stride block would be using the same lkey but different offsets in it (offset 0 and offset d1+d2+d3+d4). The verbs layer will use a special mr type that will describe everything and will replace the old API, that enforce using 3 different memory regions (data_mr, protection_mr, sig_mr) and their local invalidations. This will ease the code in the ULP and will improve the abstraction of the HW (see iSER code changes). The patchset contains also iSER initator patches that using this new API. For iSER, the code was tested vs. LIO iSER target using Mellanox's ConnectX-4/ConnectX-5. This series applies cleanly on top of kernel 5.1.0 tag. We should aim to push this code during the next merge window. Linux fork with this series can be found at: https://github.com/Mellanox/NVMEoF-P2P (t10_new_api_iser-5_1 branch). Next steps are: - merge NVMe-oF/RDMA host side after merging this patchset - Implement metadata support for NVMe-oF/RDMA target side with new API --------- Changes since v3: - Add new mr types IB_MR_TYPE_USER and IB_MR_TYPE_DMA at patch 02/25 - Fix kernel-doc syntax at include/rdma/signature.h - Remove struct ib_scaterlist - Rebase the code over kernel 5.1.0 - Added Reviewed-by signatures - Use new API in iSER LIO target and remove the old API - If possibe, avoid doing a UMR operation to register data and protection buffers at patch 25/25 --------- Changes since v2: - Rename IB_MR_TYPE_PI to IB_MR_TYPE_INTEGRITY (Sagi) - Rename IB_WR_REG_PI_MR to IB_WR_REG_MR_INTEGRITY (Sagi) - Refactor iser_login_rsp (Christoph) - Unwind WR union at iser_tx_desc (patch 16/18 - Christoph) - Rebase the code over kernel 5.0 plus 2 iser fixes - Added Reviewed-by signatures --------- Changes since v1: - Add a missing comma at patch 01/17 - Fix coding style at patches 03/17, 05/17 and 09/17 - Fix srp_map_finish_fr function at patch 04/17 - Rebase the code over 5.0-rc5 --------- Israel Rukshin (12): RDMA/core: Introduce IB_MR_TYPE_INTEGRITY and ib_alloc_mr_integrity API IB/iser: Refactor iscsi_iser_check_protection function IB/iser: Use IB_WR_REG_MR_INTEGRITY for PI handover IB/iser: Unwind WR union at iser_tx_desc IB/iser: Remove unused sig_attrs argument IB/isert: Remove unused sig_attrs argument RDMA/core: Add an integrity MR pool support RDMA/rw: Fix doc typo RDMA/rw: Print the correct number of sig MRs RDMA/rw: Use IB_WR_REG_MR_INTEGRITY for PI handover RDMA/core: Remove unused IB_WR_REG_SIG_MR code RDMA/mlx5: Improve PI handover performance Max Gurtovoy (13): RDMA/core: Introduce new header file for signature operations RDMA/core: Save the MR type in the ib_mr structure RDMA/core: Introduce ib_map_mr_sg_pi to map data/protection sgl's RDMA/core: Add signature attrs element for ib_mr structure RDMA/mlx5: Implement mlx5_ib_map_mr_sg_pi and mlx5_ib_alloc_mr_integrity RDMA/mlx5: Add attr for max number page list length for PI operation RDMA/mlx5: Pass UMR segment flags instead of boolean RDMA/mlx5: Update set_sig_data_segment attribute for new signature API RDMA/mlx5: Introduce and implement new IB_WR_REG_MR_INTEGRITY work request RDMA/mlx5: Move signature_en attribute from mlx5_qp to ib_qp RDMA/core: Validate signature handover device cap RDMA/rw: Add info regarding SG count failure RDMA/mlx5: Use PA mapping for PI handover drivers/infiniband/core/device.c | 2 + drivers/infiniband/core/mr_pool.c | 8 +- drivers/infiniband/core/rw.c | 200 +++++++------- drivers/infiniband/core/uverbs_cmd.c | 2 + drivers/infiniband/core/uverbs_std_types_mr.c | 1 + drivers/infiniband/core/verbs.c | 105 +++++++- drivers/infiniband/hw/mlx5/main.c | 4 + drivers/infiniband/hw/mlx5/mlx5_ib.h | 20 +- drivers/infiniband/hw/mlx5/mr.c | 367 +++++++++++++++++++++++++- drivers/infiniband/hw/mlx5/qp.c | 217 ++++++++++----- drivers/infiniband/hw/vmw_pvrdma/pvrdma.h | 2 +- drivers/infiniband/ulp/iser/iscsi_iser.c | 12 +- drivers/infiniband/ulp/iser/iscsi_iser.h | 64 +---- drivers/infiniband/ulp/iser/iser_initiator.c | 12 +- drivers/infiniband/ulp/iser/iser_memory.c | 123 ++++----- drivers/infiniband/ulp/iser/iser_verbs.c | 103 +++----- drivers/infiniband/ulp/isert/ib_isert.c | 15 +- drivers/nvme/host/rdma.c | 2 +- include/linux/mlx5/qp.h | 3 +- include/rdma/ib_verbs.h | 165 +++--------- include/rdma/mr_pool.h | 2 +- include/rdma/rw.h | 9 - include/rdma/signature.h | 120 +++++++++ 23 files changed, 1001 insertions(+), 557 deletions(-) create mode 100644 include/rdma/signature.h