From patchwork Thu Jul 22 21:22:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12394819 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A55FC432BE for ; Thu, 22 Jul 2021 21:23:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 41BD760C41 for ; Thu, 22 Jul 2021 21:23:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231295AbhGVUnF (ORCPT ); Thu, 22 Jul 2021 16:43:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57264 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231403AbhGVUnE (ORCPT ); Thu, 22 Jul 2021 16:43:04 -0400 Received: from mail-oi1-x235.google.com (mail-oi1-x235.google.com [IPv6:2607:f8b0:4864:20::235]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8D70DC061757 for ; Thu, 22 Jul 2021 14:23:39 -0700 (PDT) Received: by mail-oi1-x235.google.com with SMTP id u11so8153130oiv.1 for ; Thu, 22 Jul 2021 14:23:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=v3nSwBWyPfbr1tgRxF/E5AHscIsy+xBM+wai75Yl1DM=; b=tQ8X2A56YdJGFwEBfpbCICRcmGuxJGSV5K1GLsk5V+ijmDP84KrIzhXu0nCyalzslc ZTPkR6vLHcEaLA80N6gfH3Fx73wLyC6WC30FEt/ks/2Yt7seWB+7VMqgw8PElnZl8JnK 0NVBU2jVS8MgvLt5C7P7ty83VgWY21ZenIbVmVEprcn8kbF8kKhkwutPukqdLfuXNqyJ rAKkx7mma9AJh9wBzlMMtoxJvqxqWxfJNGts4nqCUhBPMcJniw3ac0jeIgUHiCl5zOgV j8DkBKLx4JQeaFzj9ubJwm6m4nK5tDSB9lYbWLz5RUWXpkRPBpZU/PEvc1f1y7+34Jn7 9bqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=v3nSwBWyPfbr1tgRxF/E5AHscIsy+xBM+wai75Yl1DM=; b=s755oXOwi2JlHLPRFTJLrzGd+1i/V/fkKYWnhCHpCYyMFb2K/gt0P2pmeh7WqcX7o4 Aukozm16N6q5ec5pOHgG6ZlqGA7u1YeCva1osdKLT4O7ThK2BBHe/Uv6QhQfpUDamWgw ZqDzO5e+toFfWyVfAoi7TGP6OwlFQfwRPdp3YX4vu1LcmPXayMe6XmjamA3LMZ6inI69 Y2u1OrlVza3QXz7FzGM/q/cULJx4/FcZwoZXgKMj4gdQkTr6v68enpkrDeEVMMjvLaBW nWyLWOmAnAVZweoifB1RKiTs0JnS9JOyVE47qAmo7LJXp2RibvkDUYxyy0IfED4RdJqI 2XHA== X-Gm-Message-State: AOAM530jlxaYaJosAPGwuiQpvcS5fyuIOoNZmotVdZzSWUNqyZGGktr/ TUJsrd5xUFMBAlzJsMKTi/E= X-Google-Smtp-Source: ABdhPJzUIGS6wvMmKTdBe7fk/YLlV1dp7m5Cy0m5ar1s4nCmFchQ0s93rsDkCflkBvfihHjtJiwm9g== X-Received: by 2002:a05:6808:aaf:: with SMTP id r15mr6982288oij.80.1626989018431; Thu, 22 Jul 2021 14:23:38 -0700 (PDT) Received: from localhost (2603-8081-140c-1a00-d50d-298a-08dc-a5ed.res6.spectrum.com. [2603:8081:140c:1a00:d50d:298a:8dc:a5ed]) by smtp.gmail.com with ESMTPSA id s6sm4367678otd.6.2021.07.22.14.23.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jul 2021 14:23:38 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH v3 1/6] RDMA/rxe: Move AV from rxe_send_wqe to rxe_send_wr Date: Thu, 22 Jul 2021 16:22:40 -0500 Message-Id: <20210722212244.412157-2-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210722212244.412157-1-rpearsonhpe@gmail.com> References: <20210722212244.412157-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Move the struct rxe_av av from rxe_send_wqe to rxe_send_wr placing it in wr.ud at the same offset as it was previously. This has the effect of increasing the size of struct rxe_send_wr and keeping the size of struct rxe_send_wqe the same. This better reflects the use of this field which is only used for UD sends. This change has no effect on ABI compatibility so the modified rxe driver will operate with older versions of rdma-core. Signed-off-by: Bob Pearson --- v3: Per Jason's suggestion split up this patch into two pieces one of which just moves AV and combine adding the ah_num changes to the third patch. drivers/infiniband/sw/rxe/rxe_av.c | 2 +- drivers/infiniband/sw/rxe/rxe_verbs.c | 3 ++- include/uapi/rdma/rdma_user_rxe.h | 4 +++- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_av.c b/drivers/infiniband/sw/rxe/rxe_av.c index da2e867a1ed9..85580ea5eed0 100644 --- a/drivers/infiniband/sw/rxe/rxe_av.c +++ b/drivers/infiniband/sw/rxe/rxe_av.c @@ -107,5 +107,5 @@ struct rxe_av *rxe_get_av(struct rxe_pkt_info *pkt) if (qp_type(pkt->qp) == IB_QPT_RC || qp_type(pkt->qp) == IB_QPT_UC) return &pkt->qp->pri_av; - return (pkt->wqe) ? &pkt->wqe->av : NULL; + return (pkt->wqe) ? &pkt->wqe->wr.wr.ud.av : NULL; } diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index f7b1a1f64c13..725015a2e84c 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -605,7 +605,8 @@ static void init_send_wqe(struct rxe_qp *qp, const struct ib_send_wr *ibwr, if (qp_type(qp) == IB_QPT_UD || qp_type(qp) == IB_QPT_SMI || qp_type(qp) == IB_QPT_GSI) - memcpy(&wqe->av, &to_rah(ud_wr(ibwr)->ah)->av, sizeof(wqe->av)); + memcpy(&wqe->wr.wr.ud.av, &to_rah(ud_wr(ibwr)->ah)->av, + sizeof(struct rxe_av)); if (unlikely(ibwr->send_flags & IB_SEND_INLINE)) copy_inline_data_to_wqe(wqe, ibwr); diff --git a/include/uapi/rdma/rdma_user_rxe.h b/include/uapi/rdma/rdma_user_rxe.h index e283c2220aba..2f1ebbe96434 100644 --- a/include/uapi/rdma/rdma_user_rxe.h +++ b/include/uapi/rdma/rdma_user_rxe.h @@ -98,6 +98,9 @@ struct rxe_send_wr { __u32 remote_qpn; __u32 remote_qkey; __u16 pkey_index; + __u16 reserved; + __u32 pad[5]; + struct rxe_av av; } ud; struct { __aligned_u64 addr; @@ -148,7 +151,6 @@ struct rxe_dma_info { struct rxe_send_wqe { struct rxe_send_wr wr; - struct rxe_av av; __u32 status; __u32 state; __aligned_u64 iova; From patchwork Thu Jul 22 21:22:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12394821 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B492C4320E for ; Thu, 22 Jul 2021 21:23:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0E47560EB5 for ; Thu, 22 Jul 2021 21:23:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231495AbhGVUnH (ORCPT ); Thu, 22 Jul 2021 16:43:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57270 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231489AbhGVUnG (ORCPT ); Thu, 22 Jul 2021 16:43:06 -0400 Received: from mail-oi1-x22b.google.com (mail-oi1-x22b.google.com [IPv6:2607:f8b0:4864:20::22b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AAE13C06175F for ; Thu, 22 Jul 2021 14:23:39 -0700 (PDT) Received: by mail-oi1-x22b.google.com with SMTP id u15so8139678oiw.3 for ; Thu, 22 Jul 2021 14:23:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=f0Y/ocDdRQ4XTDM8mFWl3U3xVeDb3pxbEATe5M1rfu8=; b=b529q1f30oQMbBHGwnh7geMM9z5tDzEEBrh0lSkOAl+AzCFLbrvNrduIdTCMm1U5Ve h3YfyBjpOa+D+hiLzB1EkNQY5ba3KcAIfElwSNu/4NXsTTmIjszq7F2Bt9FRRgFbepCf VxliCrBt1sG5VeeV7pOh9UmhD3I6PTFaG9EkJXpPdCVWonAREaG7BtBWfJWc2qf/2Vu8 g9333MJZs2VRuTnEZJYfAMDmNk6MLNdopLGmJoW4iauV9WXQsDd1xHnlHn98Q58Gmwyc +rx3sMpycuP2LZm5TlwinEbtJmIE5UzuVoiu3MPY9cC0KGgMz6leLwT88vLL2GLKmngt PPCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=f0Y/ocDdRQ4XTDM8mFWl3U3xVeDb3pxbEATe5M1rfu8=; b=H1g0NCCkQUgeJs25rlVSiRkMq2L25bblVJ3EhbnpgmWqt/wyFkD2/duJuxkccbqt/3 TRdEem/ss3M0pgPirdVL4xdCc0GGmwzlBFis2eWa89WZ9xDxHcudrIT/1eoXnF3yr134 dVNxBo/ZRhHMbtsc0E/vuTxmxIlXQFAGS2BYr0ihRCyBKaRG6vVoQAO/XOE9n7b9jfdY IVtjtnwNAEqLZN7f/zGphWhHASxW/xbiBYj6cJeh7WpMX+PEdSVbJvHLnSqcW2tWWnSc fwUFB0Zt+zBSFTOKgf6C7wFqdLjRCoHdFzBoIprv+SxiuFhrhQGMbp4i/OgdosILso1g FlVg== X-Gm-Message-State: AOAM531apclRy/efBWFt0S3U37H8pDquqfkrmwEIuJqyLIlhQYxOnwZ0 JnHkO+ebq94bpWoRsposCLo= X-Google-Smtp-Source: ABdhPJwU+r5d48mdjXECIpXYoZfb4xHgcEaMqaR7age4NZ0Jc8KInzp/C5nxCosN4hJzC/0KU44a1w== X-Received: by 2002:a05:6808:2015:: with SMTP id q21mr1285464oiw.52.1626989019145; Thu, 22 Jul 2021 14:23:39 -0700 (PDT) Received: from localhost (2603-8081-140c-1a00-d50d-298a-08dc-a5ed.res6.spectrum.com. [2603:8081:140c:1a00:d50d:298a:8dc:a5ed]) by smtp.gmail.com with ESMTPSA id s19sm1105787ooj.7.2021.07.22.14.23.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jul 2021 14:23:38 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH v3 2/6] RDMA/rxe: Change AH objects to indexed Date: Thu, 22 Jul 2021 16:22:41 -0500 Message-Id: <20210722212244.412157-3-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210722212244.412157-1-rpearsonhpe@gmail.com> References: <20210722212244.412157-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Make changes to rxe_param.h and rxe_pool.c to allow indexing of AH objects. Valid indices are non-zero so older providers can be detected. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_param.h | 4 +++- drivers/infiniband/sw/rxe/rxe_pool.c | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_param.h b/drivers/infiniband/sw/rxe/rxe_param.h index 742e6ec93686..ec5c6331bee8 100644 --- a/drivers/infiniband/sw/rxe/rxe_param.h +++ b/drivers/infiniband/sw/rxe/rxe_param.h @@ -67,7 +67,9 @@ enum rxe_device_param { RXE_MAX_MCAST_GRP = 8192, RXE_MAX_MCAST_QP_ATTACH = 56, RXE_MAX_TOT_MCAST_QP_ATTACH = 0x70000, - RXE_MAX_AH = 100, + RXE_MAX_AH = 16383, + RXE_MIN_AH_INDEX = 1, + RXE_MAX_AH_INDEX = 16383, RXE_MAX_SRQ_WR = 0x4000, RXE_MIN_SRQ_WR = 1, RXE_MAX_SRQ_SGE = 27, diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c index 0b8e7c6255a2..342f090152d1 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.c +++ b/drivers/infiniband/sw/rxe/rxe_pool.c @@ -26,7 +26,9 @@ struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { .name = "rxe-ah", .size = sizeof(struct rxe_ah), .elem_offset = offsetof(struct rxe_ah, pelem), - .flags = RXE_POOL_NO_ALLOC, + .flags = RXE_POOL_INDEX | RXE_POOL_NO_ALLOC, + .min_index = RXE_MIN_AH_INDEX, + .max_index = RXE_MAX_AH_INDEX, }, [RXE_TYPE_SRQ] = { .name = "rxe-srq", From patchwork Thu Jul 22 21:22:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12394823 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67933C4320A for ; Thu, 22 Jul 2021 21:23:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3F1E260C41 for ; Thu, 22 Jul 2021 21:23:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231512AbhGVUnH (ORCPT ); Thu, 22 Jul 2021 16:43:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57272 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231403AbhGVUnG (ORCPT ); Thu, 22 Jul 2021 16:43:06 -0400 Received: from mail-oi1-x236.google.com (mail-oi1-x236.google.com [IPv6:2607:f8b0:4864:20::236]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 867DDC061575 for ; Thu, 22 Jul 2021 14:23:40 -0700 (PDT) Received: by mail-oi1-x236.google.com with SMTP id r80so8070319oie.13 for ; Thu, 22 Jul 2021 14:23:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=b51+QyWnP4GsydKYsDZr65dmDBEIovJplf60P7dpXy4=; b=qD8H0IwAJHLDtQqCUgwKyuwd8u//qldnlG7rN8CbBuSpVTWHGyzK/4Nx2Q1I4V0thq VLEpEZfqOVrBjwJofmDtB4TJRYvEXKvYr7B6v2ihlxvXVQtxtgm8hHrwyCsB5AslnGEN P7flzjmu4OLEr7Yxq0ViR9mBSzDcyj+HvIExsVMWtis6gRf13ay0DZuM0Ww2JF2Dee9B Dn9y/AK2dz/esmexq6XnexJkQsCu/K/PwASJx6OvvqYEfWQx5in8mEnkHhnvsrTxrpNU b2CSrNKqAnnL/dMnMc2w8hypoXXyQ6DgRUtY9wM96vt6v0JRCBvWw0BZh8WFaPQUC9Xi pFmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=b51+QyWnP4GsydKYsDZr65dmDBEIovJplf60P7dpXy4=; b=YpecEACyiL8DFpRhkVeBtCHEx7e1lTtgQDJ5GdnwxWGjB07kKrfReUPjEzYhqzJ1rO 826Of/IMCATJTnKoFYWZ37mQRxHcf9btHUXupCcYNBBVI+XzlugqMBnvNMvm7qSTgpwp xF6brAPRmNiJUF5fyxYMXhGCDqc96idp2HTEidi9nVd1XL2n/E0lkQJzeZkDjSPbO6EL uxZ4YPTGJwNE4qS1SCUxWNDTXgLX910R4Yx/zC/FbUKm1QaHl3VIYbeQ8uzpq7qCNpp2 8gwS/Jku0PXAmvyOShVGxp5rT6Hzd49i/np2qfeynDXM/0XYHqakj92eC/qDjUHJCV5N dwuQ== X-Gm-Message-State: AOAM530pHiIbQIyDxisuDaqexYhveBzS1ujSqF6TyYeiB7inkmBfFE7d BanlTmk9vDUsdg/eHK1N8r0= X-Google-Smtp-Source: ABdhPJx90deijZSCvGTzKI1BtoJa3zUEFWzZ0Moxpl25/udEX1S8EZpU6oza0ctqLC3nWxCIf1T9KA== X-Received: by 2002:aca:5882:: with SMTP id m124mr1284372oib.153.1626989019894; Thu, 22 Jul 2021 14:23:39 -0700 (PDT) Received: from localhost (2603-8081-140c-1a00-d50d-298a-08dc-a5ed.res6.spectrum.com. [2603:8081:140c:1a00:d50d:298a:8dc:a5ed]) by smtp.gmail.com with ESMTPSA id a44sm5047073ooj.12.2021.07.22.14.23.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jul 2021 14:23:39 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH v3 3/6] RDMA/rxe: Create AH index and return to user space Date: Thu, 22 Jul 2021 16:22:42 -0500 Message-Id: <20210722212244.412157-4-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210722212244.412157-1-rpearsonhpe@gmail.com> References: <20210722212244.412157-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Make changes to rdma_user_rxe.h to allow indexing AH objects, passing the index in UD send WRs to the driver and returning the index to the rxe provider. Modify rxe_create_ah() to add an index to AH when created and if called from a new user provider return it to user space. If called from an old provider mark the AH as not having a useful index. Modify rxe_destroy_ah to drop the index before deleting the object. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_verbs.c | 31 ++++++++++++++++++++++++++- drivers/infiniband/sw/rxe/rxe_verbs.h | 2 ++ include/uapi/rdma/rdma_user_rxe.h | 8 ++++++- 3 files changed, 39 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index 725015a2e84c..7181e21f0c55 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -161,9 +161,19 @@ static int rxe_create_ah(struct ib_ah *ibah, struct ib_udata *udata) { - int err; struct rxe_dev *rxe = to_rdev(ibah->device); struct rxe_ah *ah = to_rah(ibah); + struct rxe_create_ah_resp __user *uresp = NULL; + int err; + + if (udata) { + /* test if new user provider */ + if (udata->outlen >= sizeof(*uresp)) + uresp = udata->outbuf; + ah->is_user = true; + } else { + ah->is_user = false; + } err = rxe_av_chk_attr(rxe, init_attr->ah_attr); if (err) @@ -173,6 +183,24 @@ static int rxe_create_ah(struct ib_ah *ibah, if (err) return err; + /* create index > 0 */ + rxe_add_index(ah); + ah->ah_num = ah->pelem.index; + + if (uresp) { + /* only if new user provider */ + err = copy_to_user(&uresp->ah_num, &ah->ah_num, + sizeof(uresp->ah_num)); + if (err) { + rxe_drop_index(ah); + rxe_drop_ref(ah); + return -EFAULT; + } + } else if (ah->is_user) { + /* only if old user provider */ + ah->ah_num = 0; + } + rxe_init_av(init_attr->ah_attr, &ah->av); return 0; } @@ -205,6 +233,7 @@ static int rxe_destroy_ah(struct ib_ah *ibah, u32 flags) { struct rxe_ah *ah = to_rah(ibah); + rxe_drop_index(ah); rxe_drop_ref(ah); return 0; } diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index 959a3260fcab..500c47d84500 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -48,6 +48,8 @@ struct rxe_ah { struct rxe_pool_entry pelem; struct rxe_pd *pd; struct rxe_av av; + bool is_user; + int ah_num; }; struct rxe_cqe { diff --git a/include/uapi/rdma/rdma_user_rxe.h b/include/uapi/rdma/rdma_user_rxe.h index 2f1ebbe96434..dc9f7a5e203a 100644 --- a/include/uapi/rdma/rdma_user_rxe.h +++ b/include/uapi/rdma/rdma_user_rxe.h @@ -99,7 +99,8 @@ struct rxe_send_wr { __u32 remote_qkey; __u16 pkey_index; __u16 reserved; - __u32 pad[5]; + __u32 ah_num; + __u32 pad[4]; struct rxe_av av; } ud; struct { @@ -170,6 +171,11 @@ struct rxe_recv_wqe { struct rxe_dma_info dma; }; +struct rxe_create_ah_resp { + __u32 ah_num; + __u32 reserved; +}; + struct rxe_create_cq_resp { struct mminfo mi; }; From patchwork Thu Jul 22 21:22:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12394825 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F822C4338F for ; Thu, 22 Jul 2021 21:23:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F087960C41 for ; Thu, 22 Jul 2021 21:23:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231336AbhGVUnI (ORCPT ); Thu, 22 Jul 2021 16:43:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57264 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231513AbhGVUnH (ORCPT ); Thu, 22 Jul 2021 16:43:07 -0400 Received: from mail-ot1-x331.google.com (mail-ot1-x331.google.com [IPv6:2607:f8b0:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 53D09C061757 for ; Thu, 22 Jul 2021 14:23:41 -0700 (PDT) Received: by mail-ot1-x331.google.com with SMTP id z6-20020a9d24860000b02904d14e47202cso150999ota.4 for ; Thu, 22 Jul 2021 14:23:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+HQ31i/pqFNTDrQ+NUGSeG0oHZz1HpEMJc3Rnl4Gn9E=; b=px0l3XWMBT7+nbJLJYtlz7W/LGhoJcILrlTeC5xJOkGnGaDqGDcogRrtOtW3rwrYhN v4LZl1/8sTdKxPpNWF/MbvHeTICyHc7J+86WiljJkECeQDU8hVrR6Tkm9RAbC0SVqI92 BzNQgPZY0jpi/PYlUuv4p8DigCbdVNP8wNARk/zvXuCvX/3IlahGpZRvW0c0IiBjDihq AbInLE1TdnpCYa0jTbbxocF7ePw2vhQUgCO9Mfyd5l9JvNbK1xwuvwxWz1Su7kebBqbc JUs/0FA9TKkG9q4h9UtH8qwildW2Wk3ZmVVRcv614xXzvlNT2/tCuP5Ij8+i5qwqqTwT 0MAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+HQ31i/pqFNTDrQ+NUGSeG0oHZz1HpEMJc3Rnl4Gn9E=; b=C+oHAPNTXWTAw5rLBYU6xcOU/iPja3uDYkIp6+MlUmJ8SEGCjh6dhuahn8XVtMEkJT 42PBjl53ID8WVsjxOzS/ISQJqyU66W+u+Sj4+LJLfsIYqX9Alno2cReiQ8IZJg4eIezW sTBDp4O3L9vkLWoWfE4h4tX66fGFDq4RQpRZlrGY8+KeenkiVSngX3msTyZkhp8YOmJj whQfXw19Q8Q75+QVI7rApm4Xy9Z9R6ALLKISIlwnKSWaayB3erkbadRHMvnGYTij/BkY o+WaIrXAQDRiGEzqKJRKq/9Me3FnK1FT60usdJM73rd1caYKYVFXE05eDcI21ZUwIo6z zJ5w== X-Gm-Message-State: AOAM530bshHQNItO0ils8l8NQonNlMbmz0o2cgUo6kK3Yk+KnFVb0AiX sXhd/YoMiXRbRgXjtvtG0nQ= X-Google-Smtp-Source: ABdhPJyDShMBK/4WFKwWZ2gE3ncHOa361QfAdKluzdt67D9k29FnkoD2b0b1qeCItgJkNyiZ/DuqkQ== X-Received: by 2002:a05:6830:1e34:: with SMTP id t20mr1132403otr.166.1626989020802; Thu, 22 Jul 2021 14:23:40 -0700 (PDT) Received: from localhost (2603-8081-140c-1a00-d50d-298a-08dc-a5ed.res6.spectrum.com. [2603:8081:140c:1a00:d50d:298a:8dc:a5ed]) by smtp.gmail.com with ESMTPSA id p186sm2699937oif.49.2021.07.22.14.23.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jul 2021 14:23:40 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH v3 4/6] RDMA/rxe: Replace ah->pd by ah->ibah.pd Date: Thu, 22 Jul 2021 16:22:43 -0500 Message-Id: <20210722212244.412157-5-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210722212244.412157-1-rpearsonhpe@gmail.com> References: <20210722212244.412157-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The pd field in struct rxe_ah is redundant with the pd field in the rdma-core's ib_ah. Eliminate the pd field in rxe_ah and add an inline to extract the pd from the ibah field. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_verbs.h | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index 500c47d84500..6e48dcd21c5f 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -46,7 +46,6 @@ struct rxe_pd { struct rxe_ah { struct ib_ah ibah; struct rxe_pool_entry pelem; - struct rxe_pd *pd; struct rxe_av av; bool is_user; int ah_num; @@ -486,6 +485,11 @@ static inline u32 mr_rkey(struct rxe_mr *mr) return mr->ibmr.rkey; } +static inline struct rxe_pd *rxe_ah_pd(struct rxe_ah *ah) +{ + return to_rpd(ah->ibah.pd); +} + static inline struct rxe_pd *rxe_mw_pd(struct rxe_mw *mw) { return to_rpd(mw->ibmw.pd); From patchwork Thu Jul 22 21:22:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12394827 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9557AC43214 for ; Thu, 22 Jul 2021 21:23:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 780A760EB5 for ; Thu, 22 Jul 2021 21:23:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231403AbhGVUnH (ORCPT ); Thu, 22 Jul 2021 16:43:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57282 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231489AbhGVUnH (ORCPT ); Thu, 22 Jul 2021 16:43:07 -0400 Received: from mail-oi1-x22c.google.com (mail-oi1-x22c.google.com [IPv6:2607:f8b0:4864:20::22c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0AD71C061575 for ; Thu, 22 Jul 2021 14:23:42 -0700 (PDT) Received: by mail-oi1-x22c.google.com with SMTP id r80so8070374oie.13 for ; Thu, 22 Jul 2021 14:23:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kFKyck3dS3F6W5tdQpY8hIQHb+fJvz4kHfwtrIX/xPI=; b=A4MhcTXmM98rt6wNVlGIGKd1kC5D82dP6FV7a6YGu8+/xDkhFwhnpP+y9kYMR79K0E sufr2cn5XZKH2NTJuAA3fgk/bGssGqQ4gvZAycBSnVHveTidvZPfaRAdtQBSL9+SNnx9 SjFxi8/m8/quk5KjPtwbH9A1NtoJFZqJKIy2fMYkGU/KLaXpw1TUL12xTOIQnPAM1Cyt YO6ZhrIq/mXnCQ64zSZDEOeXrDtXU5SKncgk/+JK6qIDQXzFChhEJkkGmDD8qA38rLvz lk8mdkepUjQ5IkN8hYvwLt8+Xr5GmZgKQsz513fBw7ge5n6y4GHYPQAT1Dd11yPKqdc6 RFWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kFKyck3dS3F6W5tdQpY8hIQHb+fJvz4kHfwtrIX/xPI=; b=G+5pWk15WeOMZs55F/U3+AckrkDMW4J/XhllbDM6vdzF2xBGTCtEKUwr1zQYGi2UhN afQ49Sn9dlfOAV3XEJ5j3bj5oehQKxgq5RygAQjqtHPJCfJV9OHK+bZbasblicpQsgFY /o1dhoVWhYxUzV7ZNb+UlvfhS80EPlyQnBAWzlTWYa5yofiYs6TGVKQ+nXNs/A8rM7SA YCbwsQNxTdePfbCQKqwvb7fNygsB47kHwKwURaW4cy49v2xvxKkuprOYtQDV02tisFlq Ho7LsWrMPH+4TQpNkD+mqgiXwKl1aFqQr5E730vrEOBJnKzCeQVI8xf6YEnBhJvPXFtF 8YsA== X-Gm-Message-State: AOAM533+wdAHSJYDhGAfaaOsLaiaaMT4TshgTL+fDmmwwpjDD8EFKNgV LDEL6A/9sNrVoIFkyjNBcnQ= X-Google-Smtp-Source: ABdhPJyIzjb4xo+Qin0cS3QT6zBSXX98xTkPit+qFvI+fFLx0rMHaOr4shRcLxx2r3JGkJPiQ7V+Hw== X-Received: by 2002:aca:c204:: with SMTP id s4mr2183843oif.49.1626989021517; Thu, 22 Jul 2021 14:23:41 -0700 (PDT) Received: from localhost (2603-8081-140c-1a00-d50d-298a-08dc-a5ed.res6.spectrum.com. [2603:8081:140c:1a00:d50d:298a:8dc:a5ed]) by smtp.gmail.com with ESMTPSA id c8sm2452152oto.17.2021.07.22.14.23.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jul 2021 14:23:41 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH v3 5/6] RDMA/rxe: Lookup kernel AH from ah index in UD WQEs Date: Thu, 22 Jul 2021 16:22:44 -0500 Message-Id: <20210722212244.412157-6-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210722212244.412157-1-rpearsonhpe@gmail.com> References: <20210722212244.412157-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Add code to rxe_get_av in rxe_av.c to use the AH index in UD send WQEs to lookup the kernel AH. For old user providers continue to use the AV passed in WQEs. Move setting pkt->rxe to before the call to rxe_get_av() to get access to the AH pool. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_av.c | 20 +++++++++++++++++++- drivers/infiniband/sw/rxe/rxe_req.c | 8 +++++--- 2 files changed, 24 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_av.c b/drivers/infiniband/sw/rxe/rxe_av.c index 85580ea5eed0..38c7b6fb39d7 100644 --- a/drivers/infiniband/sw/rxe/rxe_av.c +++ b/drivers/infiniband/sw/rxe/rxe_av.c @@ -101,11 +101,29 @@ void rxe_av_fill_ip_info(struct rxe_av *av, struct rdma_ah_attr *attr) struct rxe_av *rxe_get_av(struct rxe_pkt_info *pkt) { + struct rxe_ah *ah; + u32 ah_num; + if (!pkt || !pkt->qp) return NULL; if (qp_type(pkt->qp) == IB_QPT_RC || qp_type(pkt->qp) == IB_QPT_UC) return &pkt->qp->pri_av; - return (pkt->wqe) ? &pkt->wqe->wr.wr.ud.av : NULL; + if (!pkt->wqe) + return NULL; + + ah_num = pkt->wqe->wr.wr.ud.ah_num; + if (ah_num) { + /* only new user provider or kernel client */ + ah = rxe_pool_get_index(&pkt->rxe->ah_pool, ah_num); + if (!ah || ah->ah_num != ah_num || rxe_ah_pd(ah) != pkt->qp->pd) { + pr_warn("Unable to find AH matching ah_num\n"); + return NULL; + } + return &ah->av; + } + + /* only old user provider for UD sends*/ + return &pkt->wqe->wr.wr.ud.av; } diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index 3894197a82f6..86979a9e7afc 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -390,9 +390,8 @@ static struct sk_buff *init_req_packet(struct rxe_qp *qp, /* length from start of bth to end of icrc */ paylen = rxe_opcode[opcode].length + payload + pad + RXE_ICRC_SIZE; - /* pkt->hdr, rxe, port_num and mask are initialized in ifc - * layer - */ + /* pkt->hdr, port_num and mask are initialized in ifc layer */ + pkt->rxe = rxe; pkt->opcode = opcode; pkt->qp = qp; pkt->psn = qp->req.psn; @@ -402,6 +401,9 @@ static struct sk_buff *init_req_packet(struct rxe_qp *qp, /* init skb */ av = rxe_get_av(pkt); + if (!av) + return NULL; + skb = rxe_init_packet(rxe, av, paylen, pkt); if (unlikely(!skb)) return NULL; From patchwork Thu Jul 22 21:22:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12394829 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 602ECC4338F for ; Thu, 22 Jul 2021 21:23:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3DE5460EB6 for ; Thu, 22 Jul 2021 21:23:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231630AbhGVUnK (ORCPT ); Thu, 22 Jul 2021 16:43:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57284 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231513AbhGVUnJ (ORCPT ); Thu, 22 Jul 2021 16:43:09 -0400 Received: from mail-ot1-x32c.google.com (mail-ot1-x32c.google.com [IPv6:2607:f8b0:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B404FC061575 for ; Thu, 22 Jul 2021 14:23:42 -0700 (PDT) Received: by mail-ot1-x32c.google.com with SMTP id o2-20020a9d22020000b0290462f0ab0800so82415ota.11 for ; Thu, 22 Jul 2021 14:23:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=v3RIBh5TUYedqT7jW+xGyfalIzfFfS9vUC2W8TmzJQw=; b=aI+CtR+ov6qj5xriqNNVfvXWjYI7mDot6UpjpliI6+jp1a/NOiOTBDdqJEACPiHhrH ghhT/piuF4p8R5Vj4bxBXyA41N+4vMfaNyiOR01sPjcqCePu+aaRBJ3pZyAfz+8Rb5kX KYBwV0SIwr8lqiR2mLSuxpSbxathv9w96Fq4bTHS0y+Au8qry7drqe8SzyF8k/rGuI1w siyS6eGfZMCAbLA0Hwqg+s5iIG29FrVBGAHQX0dutr9Cgz55KHiz6uzPDdFhVi5wv7Vd 3I+mk4Y7ej0lg8P2bt5CTdB7LLEFuXyNvje2KE1cITPAhfA2/huIAA/RlIIafAqsxPx6 jcEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=v3RIBh5TUYedqT7jW+xGyfalIzfFfS9vUC2W8TmzJQw=; b=t3QeMffvIWO3tNGwM18F6lrMBZjnNbLgouwwLCB291UAGtxa3gU56zz0v3JKFegDmx lkel51hAqUnqYvDuZwaYSz8cechnqF8r4sClCcZBYf59X2D20UDHjqfHt5nFii8IDzMp gCFZf7c83iCchol8axYz6FB/Z002yJQ8zjWmR/3NEvpS3Gnh7bLuTjnnFsorHyrIGAe1 Za8TCIRrG20B6PxjBnk7ff2xPAR5zlMNgrSum0EEJtXuEVcLfyezU6uZw8TGB/fouTvz sCWUG5fgw77COKnb+isnqSq1gTI9ueKiXNqlQ645QARCgaiPZXDWEUYgYeBwsEDQW7+K kLfw== X-Gm-Message-State: AOAM532Dotdi9yGG5a1iNSjFrK78t/bJghSoHvXcc0k/bX+KAlA/KpfM 1dEioDPZodsrflC6wYM/vzgSEAmE6nE= X-Google-Smtp-Source: ABdhPJzze8OHpL3eLWgvjac20e0UYwTJGxfL6Q3t/yVGW0SizWhGULWu5tN/1nqfM6wrbIXDsMQrUQ== X-Received: by 2002:a05:6830:140d:: with SMTP id v13mr1135969otp.296.1626989022194; Thu, 22 Jul 2021 14:23:42 -0700 (PDT) Received: from localhost (2603-8081-140c-1a00-d50d-298a-08dc-a5ed.res6.spectrum.com. [2603:8081:140c:1a00:d50d:298a:8dc:a5ed]) by smtp.gmail.com with ESMTPSA id 68sm5276220otd.74.2021.07.22.14.23.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Jul 2021 14:23:41 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH v3 6/6] RDMA/rxe: Convert kernel UD post send to use ah_num Date: Thu, 22 Jul 2021 16:22:45 -0500 Message-Id: <20210722212244.412157-7-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210722212244.412157-1-rpearsonhpe@gmail.com> References: <20210722212244.412157-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Modify ib_post_send for kernel UD sends to put the AH index into the WQE instead of the address vector. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_verbs.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index 7181e21f0c55..ee3f70c8809a 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -558,8 +558,11 @@ static void init_send_wr(struct rxe_qp *qp, struct rxe_send_wr *wr, if (qp_type(qp) == IB_QPT_UD || qp_type(qp) == IB_QPT_SMI || qp_type(qp) == IB_QPT_GSI) { + struct ib_ah *ibah = ud_wr(ibwr)->ah; + wr->wr.ud.remote_qpn = ud_wr(ibwr)->remote_qpn; wr->wr.ud.remote_qkey = ud_wr(ibwr)->remote_qkey; + wr->wr.ud.ah_num = to_rah(ibah)->ah_num; if (qp_type(qp) == IB_QPT_GSI) wr->wr.ud.pkey_index = ud_wr(ibwr)->pkey_index; if (wr->opcode == IB_WR_SEND_WITH_IMM) @@ -631,12 +634,6 @@ static void init_send_wqe(struct rxe_qp *qp, const struct ib_send_wr *ibwr, return; } - if (qp_type(qp) == IB_QPT_UD || - qp_type(qp) == IB_QPT_SMI || - qp_type(qp) == IB_QPT_GSI) - memcpy(&wqe->wr.wr.ud.av, &to_rah(ud_wr(ibwr)->ah)->av, - sizeof(struct rxe_av)); - if (unlikely(ibwr->send_flags & IB_SEND_INLINE)) copy_inline_data_to_wqe(wqe, ibwr); else