| Message ID | CAFgAxU_p4OR7qUe+HjK_rYP+LxUUOxYs+Qp6g4_e64WvOXMWDg@mail.gmail.com (mailing list archive) |
|---|---|
| State | RFC |
| Headers | show |
On Wed, Jan 10, 2018 at 9:28 PM, Alex Rosenbaum <rosenbaumalex@gmail.com> wrote: > RDMA applications require an indication they are operating in an > environment which is based on GID addressing, such as SR-IOV IB > Virtualization. In this mode, all sent traffic must include a GRH. Hi Alex, Isn't this the case for RoCE since it's day one? how does it work there? maybe a hop limit > 1 is used and the kernel rdma stack tweaks it over RoCE ports? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Jan 10, 2018 at 11:59:18PM +0200, Or Gerlitz wrote: > On Wed, Jan 10, 2018 at 9:28 PM, Alex Rosenbaum <rosenbaumalex@gmail.com> wrote: > > RDMA applications require an indication they are operating in an > > environment which is based on GID addressing, such as SR-IOV IB > > Virtualization. In this mode, all sent traffic must include a GRH. > > Hi Alex, > > Isn't this the case for RoCE since it's day one? how does it work there? maybe > a hop limit > 1 is used and the kernel rdma stack tweaks it over RoCE ports? roce requires GRH because it is roce. Everything should just use that as the criteria. The hop limit trick is only for IB and it is only for path records from the SA. It allows the SA to say 'a GRH is required for this path' even if the path has local GIDs. The SA *should* be returning hop limit > 1 for any GID that has grh_required=1 in this patch, otherwise CM won't work right if the node is a target. Presumably the attr is required for non SA based schemes? Jason -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/libibverbs/man/ibv_create_ah.3 b/libibverbs/man/ibv_create_ah.3 index 0ca2198..b645ea3 100644 --- a/libibverbs/man/ibv_create_ah.3 +++ b/libibverbs/man/ibv_create_ah.3 @@ -53,6 +53,10 @@ destroys the AH .SH "RETURN VALUE" .B ibv_create_ah() returns a pointer to the created AH, or NULL if the request fails. +.SH "NOTES" +If port cap flag IBV_PORT_GRH_REQUIRED is set then +.B ibv_create_ah() +must be created with definition of 'struct ibv_ah_attr { .is_global = 1; .grh = {...}; }'. .PP .B ibv_destroy_ah() returns 0 on success, or the value of errno on failure (which indicates the failure reason). diff --git a/libibverbs/verbs.h b/libibverbs/verbs.h index 997597a..12e80b3 100644 --- a/libibverbs/verbs.h +++ b/libibverbs/verbs.h @@ -338,7 +338,8 @@ enum ibv_port_cap_flags { IBV_PORT_BOOT_MGMT_SUP = 1 << 23, IBV_PORT_LINK_LATENCY_SUP = 1 << 24, IBV_PORT_CLIENT_REG_SUP = 1 << 25, - IBV_PORT_IP_BASED_GIDS = 1 << 26 + IBV_PORT_IP_BASED_GIDS = 1 << 26, + IBV_PORT_GRH_REQUIRED = 1 << 27 }; struct ibv_port_attr {
RDMA applications require an indication they are operating in an environment which is based on GID addressing, such as SR-IOV IB Virtualization. In this mode, all sent traffic must include a GRH. This RFC exposes a new verbs port capability flag: GRH_REQUIRED. When GRH_REQUIRED port capability flag is set, the applications must create all AH with GRH configured. Meaning calling ibv_create_ah() with 'struct ibv_ah_attr { is_global = 1 }' Signed-off-by: Alex Rosenbaum <alexr@mellanox.com> --- libibverbs/man/ibv_create_ah.3 | 4 ++++ libibverbs/verbs.h | 3 ++- 2 files changed, 6 insertions(+), 1 deletion(-)