From patchwork Thu Oct 7 20:40:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12543263 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6869DC433EF for ; Thu, 7 Oct 2021 20:41:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4C9F160EDF for ; Thu, 7 Oct 2021 20:41:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232797AbhJGUn1 (ORCPT ); Thu, 7 Oct 2021 16:43:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59742 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230467AbhJGUn1 (ORCPT ); Thu, 7 Oct 2021 16:43:27 -0400 Received: from mail-ot1-x32b.google.com (mail-ot1-x32b.google.com [IPv6:2607:f8b0:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 02422C061570 for ; Thu, 7 Oct 2021 13:41:33 -0700 (PDT) Received: by mail-ot1-x32b.google.com with SMTP id g15-20020a9d128f000000b0054e3d55dd81so3934499otg.12 for ; Thu, 07 Oct 2021 13:41:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pcxAdiImyK4B9yAh9TbVrtYx8EXTONj/TcOyvXcVc/w=; b=gqdo2tw4NkUK4iQB9vA4o+8ctjPEBAqV0lHbTl2E/qgq76uGjHpTY2PUDQNIG7wNXu rAK5W+Lqz6cAtQEweVAHKfKDJ8kgoP4pyNYdKwiPRXQbQp/Qd94SKtiSul2nF5mfalg6 Tl/YwQ9hHemhztGx/UVxwAYeFOLrMBPMefsvIsp1+dydb0/hBXMAYelZ3kDHeKzPtnCk /anwsho1cXaAezDfMDEqqWl14FSB82rWF+Vr88zvJoMI9uUEQ0FIzi8LwfRsZA4RK9Bh y+hb6xIsokO6gTkwDsiQchmEUKY8XnDCodR7IXZcOjxpGG5opOBaDdrcwRKZeqixoa7C PMeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pcxAdiImyK4B9yAh9TbVrtYx8EXTONj/TcOyvXcVc/w=; b=fi2yMmsa33+PjBhL1UmrYpjEvBb0rZ6h1OsT6HRmZtXR0hwVfMqkZVD9dlrZ2nilcA NXvyQoggO3SfyR8cyuTbzoCBGOHi5wtg+Rm9JY/SN2Emnl8Qy4qkiit4W4t/7pxy7tWk GS2+L1HPdSstpuDE6P3stbehKkT3Ny7T+TcoEgQxZh+tvShcpwCweuzJWB8wla1Ornkk Zzt4DWIn8kpf1vrCfDR2GSH3WmLSA3tI7BC6ubw5a2biegCxYTrt4+WTR7dPubA2ZZLC RWshVKeVjTV/Y6Rv/G8ZfHjCOrBIBiRG5mSAFxYc2nF1NiUAN+JJ9EJpJIwOl9v7dc7L NLFA== X-Gm-Message-State: AOAM531AeUQsJRMlbQIso5C2a/TsRZp0ALIBK3eCkkUCDuNbLQhmdzir emvZboP1H/JgihnX5fFYROVGb9kwLJ8= X-Google-Smtp-Source: ABdhPJyTtcQwzv/OmcZFm2nvyXqfxRT9CowdP4GCH1+hmRpAuvzskKFYCHCt5Cn93uvpDGFWLLGdOQ== X-Received: by 2002:a9d:490:: with SMTP id 16mr5502612otm.184.1633639292326; Thu, 07 Oct 2021 13:41:32 -0700 (PDT) Received: from ubunto-21.tx.rr.com (2603-8081-140c-1a00-1259-16f0-10f5-1724.res6.spectrum.com. [2603:8081:140c:1a00:1259:16f0:10f5:1724]) by smtp.gmail.com with ESMTPSA id f10sm71607ooh.42.2021.10.07.13.41.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 13:41:31 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v6 1/6] RDMA/rxe: Move AV from rxe_send_wqe to rxe_send_wr Date: Thu, 7 Oct 2021 15:40:47 -0500 Message-Id: <20211007204051.10086-2-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211007204051.10086-1-rpearsonhpe@gmail.com> References: <20211007204051.10086-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Move the struct rxe_av av from struct rxe_send_wqe to struct rxe_send_wr placing it in wr.ud at the same offset as it was previously. This has the effect of increasing the size of struct rxe_send_wr while keeping the size of struct rxe_send_wqe the same. This better reflects the use of this field which is only used for UD sends. This change has no effect on ABI compatibility so the modified rxe driver will operate with older versions of rdma-core. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_av.c | 2 +- drivers/infiniband/sw/rxe/rxe_verbs.c | 3 ++- include/uapi/rdma/rdma_user_rxe.h | 4 +++- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_av.c b/drivers/infiniband/sw/rxe/rxe_av.c index da2e867a1ed9..85580ea5eed0 100644 --- a/drivers/infiniband/sw/rxe/rxe_av.c +++ b/drivers/infiniband/sw/rxe/rxe_av.c @@ -107,5 +107,5 @@ struct rxe_av *rxe_get_av(struct rxe_pkt_info *pkt) if (qp_type(pkt->qp) == IB_QPT_RC || qp_type(pkt->qp) == IB_QPT_UC) return &pkt->qp->pri_av; - return (pkt->wqe) ? &pkt->wqe->av : NULL; + return (pkt->wqe) ? &pkt->wqe->wr.wr.ud.av : NULL; } diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index 9d0bb9aa7514..c09e1c25ce66 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -584,7 +584,8 @@ static void init_send_wqe(struct rxe_qp *qp, const struct ib_send_wr *ibwr, if (qp_type(qp) == IB_QPT_UD || qp_type(qp) == IB_QPT_SMI || qp_type(qp) == IB_QPT_GSI) - memcpy(&wqe->av, &to_rah(ud_wr(ibwr)->ah)->av, sizeof(wqe->av)); + memcpy(&wqe->wr.wr.ud.av, &to_rah(ud_wr(ibwr)->ah)->av, + sizeof(struct rxe_av)); if (unlikely(ibwr->send_flags & IB_SEND_INLINE)) copy_inline_data_to_wqe(wqe, ibwr); diff --git a/include/uapi/rdma/rdma_user_rxe.h b/include/uapi/rdma/rdma_user_rxe.h index e283c2220aba..2f1ebbe96434 100644 --- a/include/uapi/rdma/rdma_user_rxe.h +++ b/include/uapi/rdma/rdma_user_rxe.h @@ -98,6 +98,9 @@ struct rxe_send_wr { __u32 remote_qpn; __u32 remote_qkey; __u16 pkey_index; + __u16 reserved; + __u32 pad[5]; + struct rxe_av av; } ud; struct { __aligned_u64 addr; @@ -148,7 +151,6 @@ struct rxe_dma_info { struct rxe_send_wqe { struct rxe_send_wr wr; - struct rxe_av av; __u32 status; __u32 state; __aligned_u64 iova; From patchwork Thu Oct 7 20:40:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12543265 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC379C433FE for ; Thu, 7 Oct 2021 20:41:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A04FA60EDF for ; Thu, 7 Oct 2021 20:41:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232852AbhJGUn1 (ORCPT ); Thu, 7 Oct 2021 16:43:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59748 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230467AbhJGUn1 (ORCPT ); Thu, 7 Oct 2021 16:43:27 -0400 Received: from mail-ot1-x332.google.com (mail-ot1-x332.google.com [IPv6:2607:f8b0:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 77DC8C061570 for ; Thu, 7 Oct 2021 13:41:33 -0700 (PDT) Received: by mail-ot1-x332.google.com with SMTP id r43-20020a05683044ab00b0054716b40005so9046138otv.4 for ; Thu, 07 Oct 2021 13:41:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VsIAYwzznZD6Db2echMgTKqJJdNRBioS0R3C8i4z0Z4=; b=p8gJvYiwqfpvt1LUHe9wPYO6YVCxlaVTy2v/WHeWuWXutwaGunphksDIS3fxJWn15d kKr2WYeTTxwplt6SgngFtS3x86ecM3T+HoehjoD3+s9+b6Qt25Ac+qscRQDYfglQQCzu I+IYWW9h1MputM2ekdSr4Pyz9HEUlhqR3zV3f5tKtxU+nmTVvjutglGerZ1AY07kQ0iP Fkoaq2VM2+d/8sS+lBje0JXDexO1a8cIcu2Ye8bR8aGdMJ7okozSfKAUBKDsNJPf0Fc7 UjLRc+7641LpU0yGymbVCju57rjJKJfN7nc7obSmEdqpQQ8jO5P7SWi+l/KGOmVwEBc1 AOgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VsIAYwzznZD6Db2echMgTKqJJdNRBioS0R3C8i4z0Z4=; b=omO3Cbn35WqmcOflarNvE92aAmajxXlb6qlxa+HbzCYMXwaZdB1ni/k46fFKp5XK/T zi206XGeiseIuEpkgyB6P609X8Bim+H1FayzIQKqiWvhyew8IQyRD2qGz0S3nDskavQp wCbOSsu0Z8yg2TnLrrbrGN3kyu7zCGAfWEGxGzuPd6IzYDa9iZ8RNZqDhEvsno69lwE4 B3rtAUALpdWWqNM+2H3qSvN3DhTqJFPtsDatqNpEzsFDHwQledrVnzhHeEl18aOLJW9y 2xVSXqQKwYAVd4nCeCd5KDK0uBaMMsWUBw4defD0ky1qnZ84vxybeHKOpihr5YRXGtEU LiPw== X-Gm-Message-State: AOAM532mNx76RRWakgbHFuLu22JelyRI7Xq6C+zVMvn3xwBnGHlmIlPk hTt8FYKYZxYEgF8dETuhDM2IVxI/Pkk= X-Google-Smtp-Source: ABdhPJy7NQMU4BGk8yWbqJMy/peyVDtmKOinmhxwFqzgrR/QkFrQpyb0jy7jn+ftKkpbTrDa3OA/Hg== X-Received: by 2002:a9d:3e15:: with SMTP id a21mr5601375otd.60.1633639292931; Thu, 07 Oct 2021 13:41:32 -0700 (PDT) Received: from ubunto-21.tx.rr.com (2603-8081-140c-1a00-1259-16f0-10f5-1724.res6.spectrum.com. [2603:8081:140c:1a00:1259:16f0:10f5:1724]) by smtp.gmail.com with ESMTPSA id f10sm71607ooh.42.2021.10.07.13.41.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 13:41:32 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v6 2/6] RDMA/rxe: Change AH objects to indexed Date: Thu, 7 Oct 2021 15:40:48 -0500 Message-Id: <20211007204051.10086-3-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211007204051.10086-1-rpearsonhpe@gmail.com> References: <20211007204051.10086-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Make changes to rxe_param.h and rxe_pool.c to allow indexing of AH objects. Valid indices are non-zero so older providers can be detected. Signed-off-by: Bob Pearson --- v6: Reduced the max number of AH to 32K to limit the size of the index bit map. --- drivers/infiniband/sw/rxe/rxe_param.h | 4 +++- drivers/infiniband/sw/rxe/rxe_pool.c | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_param.h b/drivers/infiniband/sw/rxe/rxe_param.h index aeffaef8f381..918270e34a35 100644 --- a/drivers/infiniband/sw/rxe/rxe_param.h +++ b/drivers/infiniband/sw/rxe/rxe_param.h @@ -69,7 +69,9 @@ enum rxe_device_param { RXE_MAX_MCAST_GRP = 8192, RXE_MAX_MCAST_QP_ATTACH = 56, RXE_MAX_TOT_MCAST_QP_ATTACH = 0x70000, - RXE_MAX_AH = DEFAULT_MAX_VALUE, + RXE_MAX_AH = (1<<15) - 1, /* 32Ki - 1 */ + RXE_MIN_AH_INDEX = 1, + RXE_MAX_AH_INDEX = RXE_MAX_AH, RXE_MAX_SRQ_WR = DEFAULT_MAX_VALUE, RXE_MIN_SRQ_WR = 1, RXE_MAX_SRQ_SGE = 27, diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c index ffa8420b4765..7b4cb46edfd9 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.c +++ b/drivers/infiniband/sw/rxe/rxe_pool.c @@ -26,7 +26,9 @@ struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { .name = "rxe-ah", .size = sizeof(struct rxe_ah), .elem_offset = offsetof(struct rxe_ah, pelem), - .flags = RXE_POOL_NO_ALLOC, + .flags = RXE_POOL_INDEX | RXE_POOL_NO_ALLOC, + .min_index = RXE_MIN_AH_INDEX, + .max_index = RXE_MAX_AH_INDEX, }, [RXE_TYPE_SRQ] = { .name = "rxe-srq", From patchwork Thu Oct 7 20:40:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12543267 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3E99C4332F for ; Thu, 7 Oct 2021 20:41:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 99BDB60F90 for ; Thu, 7 Oct 2021 20:41:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233168AbhJGUn2 (ORCPT ); Thu, 7 Oct 2021 16:43:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59750 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230467AbhJGUn2 (ORCPT ); Thu, 7 Oct 2021 16:43:28 -0400 Received: from mail-ot1-x330.google.com (mail-ot1-x330.google.com [IPv6:2607:f8b0:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 01192C061570 for ; Thu, 7 Oct 2021 13:41:34 -0700 (PDT) Received: by mail-ot1-x330.google.com with SMTP id u20-20020a9d7214000000b0054e170300adso8986251otj.13 for ; Thu, 07 Oct 2021 13:41:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lBxkpKC/Ie6Rn+yTC69M869crKSuud+tFzqyCacmKP8=; b=nMXTotcVtAjvqZnAIleAXAT6LviaZpPAli99KVVbRhZ/NnZ2WdH3Qy8Al6CxsUEgcH H1doknBMHlpLVQGCZIAGNG+JQT3qhHCsf1qLBjZ4TK2f9s6jLzWjh8Uk7ZrEYCOZPqZt 1+o8nhG4a0B0pVfUnfaFVuTUzyy6APHsT7g7Bz2aYhQ3hEW0lPfXVSV8Dy7weOlbxdUU sFJWuMfqBNgAqKpd+Opixqzgjcf9n+dsjiatj5A3uO/aSIPQc12nSPoxPrtIdsElXs2d W9vMabSpP6Ew3uMS/CMllfcTTdqJITC27rtWBXOZii8GJEWdWvex1LmLq4hwNiddXQrJ tJ1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lBxkpKC/Ie6Rn+yTC69M869crKSuud+tFzqyCacmKP8=; b=xEaNiZil1ZLrSyBVr7D0GTqrkTi8dj4GQJ2r6K8sgJja279yU95NTi19lASHhcdM9y yNKPMjxEgCxaK+xNxTTIPA0JFkG26ntme1rhSCe3RwXwESJ4ouOxs5cNzuJkJuyR/o8b qM6irlv7ACAYDVqRnBlyNPjwkaHRwsgu1THiwXV/3GBJkJNdRm+UWfvN1d9pnPOKA6kj m9jTj/9Jdk8AHOHAjjI6n/h/Tl2JtShUzGoQ1pGHFSk4dsPiH5Vg1koBhnY+jMNKR8yN tRsrfba1pcbBDhczZaV1oRQAPIrw5uncsuwMJGGbC7KoMg18pS9afpw0F9KqeR1EdT26 B46w== X-Gm-Message-State: AOAM531my8KYqVitBzchnL1w6oPALHDRv2Bxu2WnGotxh/OahWILiFGK 6pMlR4MKwu4SqC4PLKEBeqE= X-Google-Smtp-Source: ABdhPJzzNU/mlVCdLI4w/GOSwRx6HNbx7qhMMS1+28WT3MDeYuC8PVqYsEavMf3RaVY4rpFxZoKjTw== X-Received: by 2002:a9d:6092:: with SMTP id m18mr5461337otj.215.1633639293439; Thu, 07 Oct 2021 13:41:33 -0700 (PDT) Received: from ubunto-21.tx.rr.com (2603-8081-140c-1a00-1259-16f0-10f5-1724.res6.spectrum.com. [2603:8081:140c:1a00:1259:16f0:10f5:1724]) by smtp.gmail.com with ESMTPSA id f10sm71607ooh.42.2021.10.07.13.41.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 13:41:33 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v6 3/6] RDMA/rxe: Create AH index and return to user space Date: Thu, 7 Oct 2021 15:40:49 -0500 Message-Id: <20211007204051.10086-4-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211007204051.10086-1-rpearsonhpe@gmail.com> References: <20211007204051.10086-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Make changes to rdma_user_rxe.h to allow indexing AH objects, passing the index in UD send WRs to the driver and returning the index to the rxe provider. Modify rxe_create_ah() to add an index to AH when created and if called from a new user provider return it to user space. If called from an old provider mark the AH as not having a useful index. Modify rxe_destroy_ah to drop the index before deleting the object. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_verbs.c | 31 ++++++++++++++++++++++++++- drivers/infiniband/sw/rxe/rxe_verbs.h | 2 ++ include/uapi/rdma/rdma_user_rxe.h | 8 ++++++- 3 files changed, 39 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index c09e1c25ce66..8854ace63acd 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -158,9 +158,19 @@ static int rxe_create_ah(struct ib_ah *ibah, struct ib_udata *udata) { - int err; struct rxe_dev *rxe = to_rdev(ibah->device); struct rxe_ah *ah = to_rah(ibah); + struct rxe_create_ah_resp __user *uresp = NULL; + int err; + + if (udata) { + /* test if new user provider */ + if (udata->outlen >= sizeof(*uresp)) + uresp = udata->outbuf; + ah->is_user = true; + } else { + ah->is_user = false; + } err = rxe_av_chk_attr(rxe, init_attr->ah_attr); if (err) @@ -170,6 +180,24 @@ static int rxe_create_ah(struct ib_ah *ibah, if (err) return err; + /* create index > 0 */ + rxe_add_index(ah); + ah->ah_num = ah->pelem.index; + + if (uresp) { + /* only if new user provider */ + err = copy_to_user(&uresp->ah_num, &ah->ah_num, + sizeof(uresp->ah_num)); + if (err) { + rxe_drop_index(ah); + rxe_drop_ref(ah); + return -EFAULT; + } + } else if (ah->is_user) { + /* only if old user provider */ + ah->ah_num = 0; + } + rxe_init_av(init_attr->ah_attr, &ah->av); return 0; } @@ -202,6 +230,7 @@ static int rxe_destroy_ah(struct ib_ah *ibah, u32 flags) { struct rxe_ah *ah = to_rah(ibah); + rxe_drop_index(ah); rxe_drop_ref(ah); return 0; } diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index c807639435eb..9cd203f1fa22 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -48,6 +48,8 @@ struct rxe_ah { struct rxe_pool_entry pelem; struct rxe_pd *pd; struct rxe_av av; + bool is_user; + int ah_num; }; struct rxe_cqe { diff --git a/include/uapi/rdma/rdma_user_rxe.h b/include/uapi/rdma/rdma_user_rxe.h index 2f1ebbe96434..dc9f7a5e203a 100644 --- a/include/uapi/rdma/rdma_user_rxe.h +++ b/include/uapi/rdma/rdma_user_rxe.h @@ -99,7 +99,8 @@ struct rxe_send_wr { __u32 remote_qkey; __u16 pkey_index; __u16 reserved; - __u32 pad[5]; + __u32 ah_num; + __u32 pad[4]; struct rxe_av av; } ud; struct { @@ -170,6 +171,11 @@ struct rxe_recv_wqe { struct rxe_dma_info dma; }; +struct rxe_create_ah_resp { + __u32 ah_num; + __u32 reserved; +}; + struct rxe_create_cq_resp { struct mminfo mi; }; From patchwork Thu Oct 7 20:40:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12543269 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46A07C433EF for ; Thu, 7 Oct 2021 20:41:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 29FB460F90 for ; Thu, 7 Oct 2021 20:41:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233135AbhJGUn3 (ORCPT ); Thu, 7 Oct 2021 16:43:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59758 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233176AbhJGUn2 (ORCPT ); Thu, 7 Oct 2021 16:43:28 -0400 Received: from mail-ot1-x331.google.com (mail-ot1-x331.google.com [IPv6:2607:f8b0:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 94A08C061760 for ; Thu, 7 Oct 2021 13:41:34 -0700 (PDT) Received: by mail-ot1-x331.google.com with SMTP id j11-20020a9d190b000000b00546fac94456so9038136ota.6 for ; Thu, 07 Oct 2021 13:41:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qApN8UnhyEUIjIpM/GmsOjj23Il/VJ1qkJLR9SYk3QA=; b=fNn2ZisNjPf5g7UiC4S6LFZhDEpLzy7PsZuXxVZu4jC5o1QpKZ3gG5ITuD0mbUR+/C pjkkPZh9rI0b4Xy1vcklP7xPwIfK6E/iQYc2fzONQB8NMPOM2pI8f9hFytYrEbQAQKeg CHbrFDsuMOWe3lOcD9cN48PIGo1Ie8oYR1PC6sPhIJJAqEL97zZLcLmZGHCdIN1ExnLL xCLZFMf91pyjeR6MJQkCwEbsz55HoklpI+Y3RoZe0LAFtGkxoGqTe0FK9+IjHDDRH8lo J7IaT3Izdg5+cYCBtjpjrHwaDOaWTrcCQRdMbqAUk9uPJpVU01cxModdxBStXTrIAyCj ZCxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qApN8UnhyEUIjIpM/GmsOjj23Il/VJ1qkJLR9SYk3QA=; b=IpX+qhfn6jZnhvMyxPaJN64axO62wmtJ8BOUOJDfJlybOLC14foKG2BIjzcTCHmfPc YnG7c58OQxy9peTx/if5CaTJ+intFgOwOLFlOJ9Bl1a6LKB0pq1AceiqAltjwHk1WSDG MC2HHM1VB1qnzyEKPfbblqysyikGt8QDaYHuyH4+GdlWnrwsHvDNmED1nt2E93Xf7uIH NPiDjs08SQPA9UmcQpHU10CgWynfkKco8df1OOEEDqaVAgpUJWcozsjrYNLIMYS4kI/m bqWG+2cESzcE68vEtq6lFS9AWa2Qn0wbA7ZL9o2PWQg5tYgdz7OTXkcmosTWu2QSMm98 vaWw== X-Gm-Message-State: AOAM532IRRvYEWGK0tYnL0PtJjt78KlwD2452CuWDQqRrINrWGPvBnUB TQ09hFSqy/MISpjONeXvlKOp/Jt7lSI= X-Google-Smtp-Source: ABdhPJxWaMkPHVg8VwGAP44oCsEdE9PnjCjecWkjskQ5s/DFE9sjnUsvZlj6FvtuAso3VRvYSY6ieg== X-Received: by 2002:a9d:1b4d:: with SMTP id l71mr5439794otl.188.1633639293997; Thu, 07 Oct 2021 13:41:33 -0700 (PDT) Received: from ubunto-21.tx.rr.com (2603-8081-140c-1a00-1259-16f0-10f5-1724.res6.spectrum.com. [2603:8081:140c:1a00:1259:16f0:10f5:1724]) by smtp.gmail.com with ESMTPSA id f10sm71607ooh.42.2021.10.07.13.41.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 13:41:33 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v6 4/6] RDMA/rxe: Replace ah->pd by ah->ibah.pd Date: Thu, 7 Oct 2021 15:40:50 -0500 Message-Id: <20211007204051.10086-5-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211007204051.10086-1-rpearsonhpe@gmail.com> References: <20211007204051.10086-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The pd field in struct rxe_ah is redundant with the pd field in the rdma-core's ib_ah. Eliminate the pd field in rxe_ah and add an inline to extract the pd from the ibah field. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_verbs.h | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index 9cd203f1fa22..881a5159a7d0 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -46,7 +46,6 @@ struct rxe_pd { struct rxe_ah { struct ib_ah ibah; struct rxe_pool_entry pelem; - struct rxe_pd *pd; struct rxe_av av; bool is_user; int ah_num; @@ -471,6 +470,11 @@ static inline struct rxe_mw *to_rmw(struct ib_mw *mw) return mw ? container_of(mw, struct rxe_mw, ibmw) : NULL; } +static inline struct rxe_pd *rxe_ah_pd(struct rxe_ah *ah) +{ + return to_rpd(ah->ibah.pd); +} + static inline struct rxe_pd *mr_pd(struct rxe_mr *mr) { return to_rpd(mr->ibmr.pd); From patchwork Thu Oct 7 20:40:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12543271 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94A2BC433FE for ; Thu, 7 Oct 2021 20:41:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 82ACE60F90 for ; Thu, 7 Oct 2021 20:41:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233176AbhJGUn3 (ORCPT ); Thu, 7 Oct 2021 16:43:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59764 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230467AbhJGUn3 (ORCPT ); Thu, 7 Oct 2021 16:43:29 -0400 Received: from mail-ot1-x334.google.com (mail-ot1-x334.google.com [IPv6:2607:f8b0:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1163FC061570 for ; Thu, 7 Oct 2021 13:41:35 -0700 (PDT) Received: by mail-ot1-x334.google.com with SMTP id g62-20020a9d2dc4000000b0054752cfbc59so9145381otb.1 for ; Thu, 07 Oct 2021 13:41:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=R28ci1vfu6y2BAkpjnPtPdKQKjUHlIJGx5SO1oi5Sfc=; b=M68TJDklFdmZlj222a99vVIRadmoUMvId+iOk+pBun4sJ3aiLfVvYSQ+pzO3yl6xq/ MQcZVx1E5DBrXsWldOmfMOKPLcUpZw3SM+jftKJs+o9sd0789BZNGLgxk/VgF1MYKsZh nl4ZUlKSb0pRm4J8KAWVuFpoJpcJTPlpcstFfPzCE9R8JkI+IyFDmivmDU2/4/RiesDx iL8Tt2mihlrNpY8yKUV31JAl7T/7vjczxEEHMa1glsk+WfdyfQH4rJFMy4Jq5NR3VwXX xkjIHcQe4nkJiSfga+8vp3SKakaeTKJjsiZ+gz8oRiYsKjUjh6aGD7WyNcSXnBIrlBG2 5ekA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=R28ci1vfu6y2BAkpjnPtPdKQKjUHlIJGx5SO1oi5Sfc=; b=TZ7qOqjkuKOjecQ7NJRgtEWMCS0lRJU66NBREIu54hzCQcdZCCPwCZq22pm4xZrcuf oImuOEoto7boJlnbGs+mHTqSSbN7vuvf7iZRDbjbqvNIKA2TgaRlar4GCpwxohyqo80E G1BeBz8/jKbY9D4/t87ecLFTD8kWZte8ak+zJtpQhYB8MqmPBXvkuwoLrNk2mX9/sEpA /1fSXF4RisE1pkOg4PdKTQCwkKgiX1jJiXb//esbPNJa6XquC4nsH/KN+1rcyUE3hvV1 JQPcyCd73X58GBtPcy0meJOq3aqGvBn+5tKq/H2LgTImj8sCtj8G2XFCuLglkFnOr5He RQ3w== X-Gm-Message-State: AOAM533myS40jD4fO5uuZmUzW50KETcsWBjhrQcwM44cMibMqOoQxXvz dcaDlpmDhGeaHlUkmGlIc94lF/cG3Cs= X-Google-Smtp-Source: ABdhPJzLZwC75WgkRpB4GJ5Dhp0ZEVPoZFxRCCHnPdHon4NSOggtjp5TULUrGfTEOag/fbYeXQyKYQ== X-Received: by 2002:a05:6830:1f54:: with SMTP id u20mr5298815oth.207.1633639294461; Thu, 07 Oct 2021 13:41:34 -0700 (PDT) Received: from ubunto-21.tx.rr.com (2603-8081-140c-1a00-1259-16f0-10f5-1724.res6.spectrum.com. [2603:8081:140c:1a00:1259:16f0:10f5:1724]) by smtp.gmail.com with ESMTPSA id f10sm71607ooh.42.2021.10.07.13.41.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 13:41:34 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v6 5/6] RDMA/rxe: Lookup kernel AH from ah index in UD WQEs Date: Thu, 7 Oct 2021 15:40:51 -0500 Message-Id: <20211007204051.10086-6-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211007204051.10086-1-rpearsonhpe@gmail.com> References: <20211007204051.10086-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Add code to rxe_get_av in rxe_av.c to use the AH index in UD send WQEs to lookup the kernel AH. For old user providers continue to use the AV passed in WQEs. Move setting pkt->rxe to before the call to rxe_get_av() to get access to the AH pool. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_av.c | 20 +++++++++++++++++++- drivers/infiniband/sw/rxe/rxe_req.c | 8 +++++--- 2 files changed, 24 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_av.c b/drivers/infiniband/sw/rxe/rxe_av.c index 85580ea5eed0..38c7b6fb39d7 100644 --- a/drivers/infiniband/sw/rxe/rxe_av.c +++ b/drivers/infiniband/sw/rxe/rxe_av.c @@ -101,11 +101,29 @@ void rxe_av_fill_ip_info(struct rxe_av *av, struct rdma_ah_attr *attr) struct rxe_av *rxe_get_av(struct rxe_pkt_info *pkt) { + struct rxe_ah *ah; + u32 ah_num; + if (!pkt || !pkt->qp) return NULL; if (qp_type(pkt->qp) == IB_QPT_RC || qp_type(pkt->qp) == IB_QPT_UC) return &pkt->qp->pri_av; - return (pkt->wqe) ? &pkt->wqe->wr.wr.ud.av : NULL; + if (!pkt->wqe) + return NULL; + + ah_num = pkt->wqe->wr.wr.ud.ah_num; + if (ah_num) { + /* only new user provider or kernel client */ + ah = rxe_pool_get_index(&pkt->rxe->ah_pool, ah_num); + if (!ah || ah->ah_num != ah_num || rxe_ah_pd(ah) != pkt->qp->pd) { + pr_warn("Unable to find AH matching ah_num\n"); + return NULL; + } + return &ah->av; + } + + /* only old user provider for UD sends*/ + return &pkt->wqe->wr.wr.ud.av; } diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index fe275fcaffbd..0c9d2af15f3d 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -379,9 +379,8 @@ static struct sk_buff *init_req_packet(struct rxe_qp *qp, /* length from start of bth to end of icrc */ paylen = rxe_opcode[opcode].length + payload + pad + RXE_ICRC_SIZE; - /* pkt->hdr, rxe, port_num and mask are initialized in ifc - * layer - */ + /* pkt->hdr, port_num and mask are initialized in ifc layer */ + pkt->rxe = rxe; pkt->opcode = opcode; pkt->qp = qp; pkt->psn = qp->req.psn; @@ -391,6 +390,9 @@ static struct sk_buff *init_req_packet(struct rxe_qp *qp, /* init skb */ av = rxe_get_av(pkt); + if (!av) + return NULL; + skb = rxe_init_packet(rxe, av, paylen, pkt); if (unlikely(!skb)) return NULL; From patchwork Thu Oct 7 20:40:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12543273 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86480C4332F for ; Thu, 7 Oct 2021 20:41:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6A41B60FC1 for ; Thu, 7 Oct 2021 20:41:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238605AbhJGUna (ORCPT ); Thu, 7 Oct 2021 16:43:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59770 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230467AbhJGUn3 (ORCPT ); Thu, 7 Oct 2021 16:43:29 -0400 Received: from mail-ot1-x32f.google.com (mail-ot1-x32f.google.com [IPv6:2607:f8b0:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8FC42C061760 for ; Thu, 7 Oct 2021 13:41:35 -0700 (PDT) Received: by mail-ot1-x32f.google.com with SMTP id 77-20020a9d0ed3000000b00546e10e6699so9124851otj.2 for ; Thu, 07 Oct 2021 13:41:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Fo+XA5ZhNhrA0pZdOhXylGStZB1nDPPpygrSUz+ZKz8=; b=iDrdrnhdQM9PLEKBJQUGjc67pUr5MnTaC1lR87D8wAlk9KhXe7ZfHb60HpF7kkAi4H uKYLGUcDbTI7xr47MKWOjtBGFHG5xTEOhdXv+Rnf5bkI73Fl53HHqFWXO/tkjskqKPqt bilfTwBIjeEsamhBU3l27ooTH2znaq4VFOaTZaJnv6iRyt8dWUA7qBzp1IlU0sojnVYp ElWlYFvhO1SE6+5oPU/AWoF9jfkmeIA82zHA6bDsZM5p6+U/i36/8knxxUkxrX6X6AAy jrExvYoiqIIIPcuT6fglQRVtOWLEGdpK1+owYiY7LQB7K7Al8NVRzE6Tz2A90pQNeuG8 RLJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Fo+XA5ZhNhrA0pZdOhXylGStZB1nDPPpygrSUz+ZKz8=; b=EfOMCqflXgGe2GnzFIttEUQCCV886S4vhEuJztqcQ0HLdypkYqRRInNgDmvEV5M0Sn IRfWKCSLd3+kVUGOGdcRh12APtVej+CmJPpicTeTYcpo19jq+noexEplN5IkSt+/+xww zSIgCW8OmI+7bl/zqMy2nhbh042fVQlA8j+nR1JWO17s9bPB5gevphzyi/frtRkboR5R jIXpViKLx2QxK351UAbJXQqA15/kP60y3hE5BZdfe8okWUvARfx278A67p/nLbSU6OmZ l5/hRAn/RIiD221FFmnD+uQ+ZG5XobDprCvR2jmuVlUvHTVDQcJAPX7ABBY85YdRWoY+ o5Xw== X-Gm-Message-State: AOAM530j8nYmpH9hN5VbBy1xSIQ4BfBCuS6YtJPlfbDK3TRbkX7ddobP +xDssAAdrJhsqDUG7xdUuweTTrUnOro= X-Google-Smtp-Source: ABdhPJz3XVQDEmiP4GWWyJWvRGAyBxQOylnXGbZuGJYsmFC0PK34FfX2b55LCRQVTlCOS3HFWBZN+g== X-Received: by 2002:a05:6830:438a:: with SMTP id s10mr5504644otv.173.1633639295033; Thu, 07 Oct 2021 13:41:35 -0700 (PDT) Received: from ubunto-21.tx.rr.com (2603-8081-140c-1a00-1259-16f0-10f5-1724.res6.spectrum.com. [2603:8081:140c:1a00:1259:16f0:10f5:1724]) by smtp.gmail.com with ESMTPSA id f10sm71607ooh.42.2021.10.07.13.41.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 13:41:34 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next v6 6/6] RDMA/rxe: Convert kernel UD post send to use ah_num Date: Thu, 7 Oct 2021 15:40:52 -0500 Message-Id: <20211007204051.10086-7-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211007204051.10086-1-rpearsonhpe@gmail.com> References: <20211007204051.10086-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Modify ib_post_send for kernel UD sends to put the AH index into the WQE instead of the address vector. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_verbs.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index 8854ace63acd..b808777e2221 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -537,8 +537,11 @@ static void init_send_wr(struct rxe_qp *qp, struct rxe_send_wr *wr, if (qp_type(qp) == IB_QPT_UD || qp_type(qp) == IB_QPT_SMI || qp_type(qp) == IB_QPT_GSI) { + struct ib_ah *ibah = ud_wr(ibwr)->ah; + wr->wr.ud.remote_qpn = ud_wr(ibwr)->remote_qpn; wr->wr.ud.remote_qkey = ud_wr(ibwr)->remote_qkey; + wr->wr.ud.ah_num = to_rah(ibah)->ah_num; if (qp_type(qp) == IB_QPT_GSI) wr->wr.ud.pkey_index = ud_wr(ibwr)->pkey_index; if (wr->opcode == IB_WR_SEND_WITH_IMM) @@ -610,12 +613,6 @@ static void init_send_wqe(struct rxe_qp *qp, const struct ib_send_wr *ibwr, return; } - if (qp_type(qp) == IB_QPT_UD || - qp_type(qp) == IB_QPT_SMI || - qp_type(qp) == IB_QPT_GSI) - memcpy(&wqe->wr.wr.ud.av, &to_rah(ud_wr(ibwr)->ah)->av, - sizeof(struct rxe_av)); - if (unlikely(ibwr->send_flags & IB_SEND_INLINE)) copy_inline_data_to_wqe(wqe, ibwr); else