From patchwork Mon Jun 28 22:00:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12348585 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4CF7C11F67 for ; Mon, 28 Jun 2021 22:02:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C320D61CF9 for ; Mon, 28 Jun 2021 22:02:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238204AbhF1WE5 (ORCPT ); Mon, 28 Jun 2021 18:04:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52834 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238206AbhF1WEB (ORCPT ); Mon, 28 Jun 2021 18:04:01 -0400 Received: from mail-oo1-xc33.google.com (mail-oo1-xc33.google.com [IPv6:2607:f8b0:4864:20::c33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 474BDC0617AE for ; Mon, 28 Jun 2021 15:01:33 -0700 (PDT) Received: by mail-oo1-xc33.google.com with SMTP id v11-20020a4a8c4b0000b029024be8350c45so5134418ooj.12 for ; Mon, 28 Jun 2021 15:01:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kneSU+FCYQQzfxYvj6GYx0NWFJrVtEMFTDXc/C6eRjk=; b=A/jafmVrT4/14UuJOxu3gEPLRQGoCqaptL+SJOSsvUzYKsottbvjL1PhPsk3xtOfI2 2TB9SUtD0fVqL9wGqvs3y2U2fL5rAe3T9GTEoEjsow2r2grqwV0ydWkyiIdmXB4v3w+3 7Gc1TctYAWS1H0ahQjU03cFNoxOlkAXzka+XqPSiEnOUU8G0kXgKmaqsoXrYPwQ87AGg sqnoK0tC5n/23kAmIfmxpaxcUlKXJveTtl5LWQPwv5HFKvLxj9Im5sQEWpaSEstxFQFu FFSqgi/gHbfJTLafMUjBtpPsaq1CvCeRThlnNmoSW/IKF8akUNDpX2Z/FM72nDCswtxt CqAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kneSU+FCYQQzfxYvj6GYx0NWFJrVtEMFTDXc/C6eRjk=; b=PnIXEliPS4sj1qPUaefQXIC74drX+wu86zd/buu/iQWtexDhNN4Y1JZQRqujalMalR AtMNfH6CVkoXIa3r4fJi2lYQzOlgB/3R14kCJIcYEf0OGtcoCUKajI81RgJ4Bu3n5bh2 4b+p4aKPUSRncwzh94lNuvBwR/Q0KS8rULP91Fhbx0Jv6io1kScfDo2G82mNJxctdZuO Xsd0FkHr0mBqlvFw6H/b0TQmAUx9XGnUFid4SCCnCMxVdvSMRhf8ocCPiR7tmpaT5xYS MS+L4vPw/Ho565tZD/fy60qqIkCK8EtnvgoPV4+wv+23/BFZiB7TvAI+vc+4TMo20GJb 7iRw== X-Gm-Message-State: AOAM5306eRU2r45u0avnTfYXufbN0blhqZ7v0H/DCCMopPRQDwudPFRA cUxNNBzOp6e3HLWesy+5ZoM= X-Google-Smtp-Source: ABdhPJxTq0v/E9Ru3J1g0oi4Co7LBzsXkb/gNShqpMtkt4hcUan4oaTDQGGvPb75RjRL8BdCqtCVMA== X-Received: by 2002:a05:6820:168:: with SMTP id k8mr1261735ood.76.1624917693039; Mon, 28 Jun 2021 15:01:33 -0700 (PDT) Received: from localhost (2603-8081-140c-1a00-aaa9-75eb-6e0f-9f85.res6.spectrum.com. [2603:8081:140c:1a00:aaa9:75eb:6e0f:9f85]) by smtp.gmail.com with ESMTPSA id a76sm1390985oii.26.2021.06.28.15.01.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 15:01:32 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH 1/5] RDMA/rxe: Change user/kernel API to allow indexing AH Date: Mon, 28 Jun 2021 17:00:40 -0500 Message-Id: <20210628220043.9851-2-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210628220043.9851-1-rpearsonhpe@gmail.com> References: <20210628220043.9851-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Make changes to rdma_user_rxe.h to allow indexing AH objects, passing the index in UD send WRs to the driver and returning the index to the rxe provider. This change will allow removing handling of the AV in the user space provider. This change is backwards compatible with the current API so new or old providers and drivers can work together. Signed-off-by: Bob Pearson --- include/uapi/rdma/rdma_user_rxe.h | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/include/uapi/rdma/rdma_user_rxe.h b/include/uapi/rdma/rdma_user_rxe.h index e283c2220aba..e544832ed073 100644 --- a/include/uapi/rdma/rdma_user_rxe.h +++ b/include/uapi/rdma/rdma_user_rxe.h @@ -98,6 +98,8 @@ struct rxe_send_wr { __u32 remote_qpn; __u32 remote_qkey; __u16 pkey_index; + __u16 reserved; + __u32 ah_num; } ud; struct { __aligned_u64 addr; @@ -148,7 +150,12 @@ struct rxe_dma_info { struct rxe_send_wqe { struct rxe_send_wr wr; - struct rxe_av av; + union { + struct rxe_av av; + struct { + __u32 reserved[0]; + } ex; + }; __u32 status; __u32 state; __aligned_u64 iova; @@ -168,6 +175,11 @@ struct rxe_recv_wqe { struct rxe_dma_info dma; }; +struct rxe_create_ah_resp { + __u32 ah_num; + __u32 reserved; +}; + struct rxe_create_cq_resp { struct mminfo mi; }; From patchwork Mon Jun 28 22:00:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12348587 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5854C11F65 for ; Mon, 28 Jun 2021 22:02:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A06FD61CF9 for ; Mon, 28 Jun 2021 22:02:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238197AbhF1WE4 (ORCPT ); Mon, 28 Jun 2021 18:04:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238204AbhF1WEB (ORCPT ); Mon, 28 Jun 2021 18:04:01 -0400 Received: from mail-oi1-x232.google.com (mail-oi1-x232.google.com [IPv6:2607:f8b0:4864:20::232]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 72B93C0617AF for ; Mon, 28 Jun 2021 15:01:34 -0700 (PDT) Received: by mail-oi1-x232.google.com with SMTP id w127so23778040oig.12 for ; Mon, 28 Jun 2021 15:01:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=f0Y/ocDdRQ4XTDM8mFWl3U3xVeDb3pxbEATe5M1rfu8=; b=K5kqQylNOIeWx+uBOymFKtv0aqGgao/LM8PjFMSqwZvi9jqCawYO1O0Frl6fF4n08K xurKKauRWRA5t1Shg6GamR60t5TQF980yxg/nJT/rPFmjR/TbfOs3iQT7iCswJFC1wdd l2d3GH6GeLhqdOITi1uAXaacWNh8Bf6SW1+8ewRKCPaeFJLO0rdLCSzNqd56inm7oIBU FwiP3Ata8qSI7oYPxf6jrDEt9A74X0qHxOaCoDzwtjyMB5sN9bTycwPWYqFiVvRY3Fvo Eacb1G4TLejvq/Qy9wm1MObE+2M0MoMTjCwQP70WOmp3I+XQE/YhuEmLCE8kgiY9tjos BCJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=f0Y/ocDdRQ4XTDM8mFWl3U3xVeDb3pxbEATe5M1rfu8=; b=NehYSpls5ybKWUa9YX+2/tCn7tN7B0Fq2JuejrdLaCivoyLeh9MA1+UYdA6jp8LFcu R1EpJ9y4JmGfD+dzcVEsINlEQ0cfahliU7ZfW4M50nMPSoKdeMqdXBahLWu+HZFCYAjv jzFFB8qKu7ek5mfozIBb8VtQLLnsN421WKH23aXCLtS0C6n1bBehqXHPTKYL1Sf2lRZc ZnHgS2EDGqNN5Gy9YeE28fPwLB5uQo3qLyfjKdyQ3b1Ih1tRvtrb3JBJ/Bd1FJ9VXjFN DFLAdv4d57IKwx40QcxuNN0qlGkgLuiNqSrrb1qVuYp6gKqOUhGqNx7EuknN5ymKpVG7 KlJA== X-Gm-Message-State: AOAM532k3p3m8U8NIxF9bhW9P96H4WtSns1bZRf8qZtxrk6Q9CNAZaVX k/Y4mg5W/kmPQ0Q2DGDffng= X-Google-Smtp-Source: ABdhPJxgNaDGDvZOhDP4AjMFRPm1r7SGueaJ+ElE9dBH3MWk+bENsDth0CLIbteLNxwJcE/CorKXQg== X-Received: by 2002:aca:b609:: with SMTP id g9mr22211216oif.141.1624917693909; Mon, 28 Jun 2021 15:01:33 -0700 (PDT) Received: from localhost (2603-8081-140c-1a00-aaa9-75eb-6e0f-9f85.res6.spectrum.com. [2603:8081:140c:1a00:aaa9:75eb:6e0f:9f85]) by smtp.gmail.com with ESMTPSA id t12sm1137942otp.34.2021.06.28.15.01.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 15:01:33 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH 2/5] RDMA/rxe: Change AH objects to indexed Date: Mon, 28 Jun 2021 17:00:41 -0500 Message-Id: <20210628220043.9851-3-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210628220043.9851-1-rpearsonhpe@gmail.com> References: <20210628220043.9851-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Make changes to rxe_param.h and rxe_pool.c to allow indexing of AH objects. Valid indices are non-zero so older providers can be detected. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_param.h | 4 +++- drivers/infiniband/sw/rxe/rxe_pool.c | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_param.h b/drivers/infiniband/sw/rxe/rxe_param.h index 742e6ec93686..ec5c6331bee8 100644 --- a/drivers/infiniband/sw/rxe/rxe_param.h +++ b/drivers/infiniband/sw/rxe/rxe_param.h @@ -67,7 +67,9 @@ enum rxe_device_param { RXE_MAX_MCAST_GRP = 8192, RXE_MAX_MCAST_QP_ATTACH = 56, RXE_MAX_TOT_MCAST_QP_ATTACH = 0x70000, - RXE_MAX_AH = 100, + RXE_MAX_AH = 16383, + RXE_MIN_AH_INDEX = 1, + RXE_MAX_AH_INDEX = 16383, RXE_MAX_SRQ_WR = 0x4000, RXE_MIN_SRQ_WR = 1, RXE_MAX_SRQ_SGE = 27, diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c index 0b8e7c6255a2..342f090152d1 100644 --- a/drivers/infiniband/sw/rxe/rxe_pool.c +++ b/drivers/infiniband/sw/rxe/rxe_pool.c @@ -26,7 +26,9 @@ struct rxe_type_info rxe_type_info[RXE_NUM_TYPES] = { .name = "rxe-ah", .size = sizeof(struct rxe_ah), .elem_offset = offsetof(struct rxe_ah, pelem), - .flags = RXE_POOL_NO_ALLOC, + .flags = RXE_POOL_INDEX | RXE_POOL_NO_ALLOC, + .min_index = RXE_MIN_AH_INDEX, + .max_index = RXE_MAX_AH_INDEX, }, [RXE_TYPE_SRQ] = { .name = "rxe-srq", From patchwork Mon Jun 28 22:00:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12348591 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B0E3C11F64 for ; Mon, 28 Jun 2021 22:02:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E00F861CEF for ; Mon, 28 Jun 2021 22:02:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238210AbhF1WFA (ORCPT ); Mon, 28 Jun 2021 18:05:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52844 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237380AbhF1WED (ORCPT ); Mon, 28 Jun 2021 18:04:03 -0400 Received: from mail-ot1-x332.google.com (mail-ot1-x332.google.com [IPv6:2607:f8b0:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1F254C0613A2 for ; Mon, 28 Jun 2021 15:01:35 -0700 (PDT) Received: by mail-ot1-x332.google.com with SMTP id d21-20020a9d72d50000b02904604cda7e66so18709800otk.7 for ; Mon, 28 Jun 2021 15:01:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Y5l6kftMSkFZYVSaVgxLdOb1qSScAKIujctFXK0ELPE=; b=rPPFYq85wf+9n2csiILmhSPINCYMYlszzmu3/7WGCUamFp78TJaLbEIVCLyrWRcmge hJajmO+TDIpiJGpuC9sXDuMRPEOxTiNZwG8KBSxfP5TI4u4BfaL+/sdTKWvSVBdwHa2o XxWwth/47dLM56WIVwXZUWG3bigWtBRjYYwNGqY+roA335hU6M48LAc2/SHLt8jBk0B3 qo3+Or/nQBNrhKW3mX7NhWm15hhSo98F2e3Tf1jWTVz7MKfmIU06i3zKRbDR78MbnzAU U8AuSlyebpI9Z2bHg1MvcW9Uautvn37pbsGkp7ZPVjd0RrNLmR1Z8sGT8R52YQqnTNvm jObA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Y5l6kftMSkFZYVSaVgxLdOb1qSScAKIujctFXK0ELPE=; b=NrYTDZ3nSd+TDFhWQ+L0FXT42Ghg2tlOxobX81p13mP199px9K9aJc1wEoTIxTWHuq tzRUJpwrbBRHOy/sznuxbgd2uoeW/5PEZhgplMavldxi8wKD8H4esL8YzBf2gOkGavVm iwrPgPLQ/m3R8XqQMG/buE2oH1C9DPvx6pZH86frD/8smH4ACQyjbxWu8IL/joqRyqAX cfhyHDQr7YYF9HtdM8HHNX2I35PhrA5ZMTgUAFFlwmcyz9ntdPVubO40d0kgXfbxJELw nMoIrjQdBPZf6T/Oo/r06exkRi/Y/KEqOlFCSxVfKfR70aLUfZopKyoxW7w+tLFnHqom OrQw== X-Gm-Message-State: AOAM533sTfWiNyfJJ+0bXwBxeSZWnU2Qrzx4Y7f/rGP/63riVNSeqWGc JMmgwDM0DCF9dXB489xzWMM= X-Google-Smtp-Source: ABdhPJw6JufNRoAm52MPIZ3D26NuoUkqocw4hyWGHzSY3LkIT7SgvIruB2LtNvO5/iRtWsXNt2h2dg== X-Received: by 2002:a05:6830:40b4:: with SMTP id x52mr1453761ott.117.1624917694565; Mon, 28 Jun 2021 15:01:34 -0700 (PDT) Received: from localhost (2603-8081-140c-1a00-aaa9-75eb-6e0f-9f85.res6.spectrum.com. [2603:8081:140c:1a00:aaa9:75eb:6e0f:9f85]) by smtp.gmail.com with ESMTPSA id 59sm3715499oto.3.2021.06.28.15.01.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 15:01:34 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH 3/5] RDMA/rxe: Create AH index and return to user space Date: Mon, 28 Jun 2021 17:00:42 -0500 Message-Id: <20210628220043.9851-4-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210628220043.9851-1-rpearsonhpe@gmail.com> References: <20210628220043.9851-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Modify rxe_create_ah() to add an index to AH when created and if called from a new user provider return it to user space. If called from an old provider mark the AH as not having a useful index. Modify rxe_destroy_ah to drop the index before deleting the object. Also use the PD in the core ib_ah struct instead of a duplicate copy in rxe_ah. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_verbs.c | 31 ++++++++++++++++++++++++++- drivers/infiniband/sw/rxe/rxe_verbs.h | 8 ++++++- 2 files changed, 37 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index c223959ac174..11d166bc11e3 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -161,9 +161,19 @@ static int rxe_create_ah(struct ib_ah *ibah, struct ib_udata *udata) { - int err; struct rxe_dev *rxe = to_rdev(ibah->device); struct rxe_ah *ah = to_rah(ibah); + struct rxe_create_ah_resp __user *uresp = NULL; + int err; + + if (udata) { + /* test if new user provider */ + if (udata->outlen >= sizeof(*uresp)) + uresp = udata->outbuf; + ah->is_user = true; + } else { + ah->is_user = false; + } err = rxe_av_chk_attr(rxe, init_attr->ah_attr); if (err) @@ -173,6 +183,24 @@ static int rxe_create_ah(struct ib_ah *ibah, if (err) return err; + /* create index > 0 */ + rxe_add_index(ah); + ah->ah_num = ah->pelem.index; + + if (uresp) { + /* only if new user provider */ + err = copy_to_user(&uresp->ah_num, &ah->ah_num, + sizeof(uresp->ah_num)); + if (err) { + rxe_drop_index(ah); + rxe_drop_ref(ah); + return -EFAULT; + } + } else if (ah->is_user) { + /* only if old user provider */ + ah->ah_num = 0; + } + rxe_init_av(init_attr->ah_attr, &ah->av); return 0; } @@ -205,6 +233,7 @@ static int rxe_destroy_ah(struct ib_ah *ibah, u32 flags) { struct rxe_ah *ah = to_rah(ibah); + rxe_drop_index(ah); rxe_drop_ref(ah); return 0; } diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index 959a3260fcab..fa234d7ad382 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -46,8 +46,9 @@ struct rxe_pd { struct rxe_ah { struct ib_ah ibah; struct rxe_pool_entry pelem; - struct rxe_pd *pd; struct rxe_av av; + bool is_user; + int ah_num; }; struct rxe_cqe { @@ -469,6 +470,11 @@ static inline struct rxe_mw *to_rmw(struct ib_mw *mw) return mw ? container_of(mw, struct rxe_mw, ibmw) : NULL; } +static inline struct rxe_pd *ah_pd(struct rxe_ah *ah) +{ + return to_rpd(ah->ibah.pd); +} + static inline struct rxe_pd *mr_pd(struct rxe_mr *mr) { return to_rpd(mr->ibmr.pd); From patchwork Mon Jun 28 22:00:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12348589 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D0CFC11F66 for ; Mon, 28 Jun 2021 22:02:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6624461CF9 for ; Mon, 28 Jun 2021 22:02:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238206AbhF1WE6 (ORCPT ); Mon, 28 Jun 2021 18:04:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52846 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238210AbhF1WEC (ORCPT ); Mon, 28 Jun 2021 18:04:02 -0400 Received: from mail-ot1-x335.google.com (mail-ot1-x335.google.com [IPv6:2607:f8b0:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D1C35C06124C for ; Mon, 28 Jun 2021 15:01:35 -0700 (PDT) Received: by mail-ot1-x335.google.com with SMTP id o17-20020a9d76510000b02903eabfc221a9so20529559otl.0 for ; Mon, 28 Jun 2021 15:01:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MEyGuPKeUddZhaw30N5dtanpUK1tdzdpPP/MHcDXAoY=; b=M0QUPu1T9jsX055wQiDPK0hl9uBoRF1DG51wDbQz+PnaS08xEx4OtmrFElEXnptiKY UqkGCEu9fz4BVmZzeJ/ssP38ibDRPclzkxzfjNkFYOf7x7A8jNw0H0EdaOeemZa4r427 QVY893bJksLKdVf3cCW8iYpEPT6geSpfil1981f9UA1ZdkN9wcjzXjKXwVdIfEcKUQyR jSnDUzSx8xBGk7x32ibZSF4qHf9UoMNulPblRDzxwPrJLuWrBE2crEqXxh5HfJDmEs5X ltbhQydRYJBrUJJpJpgi+fgE8iHVNm6Hi0TC7gHIWDThdD4W9rnjWCbixu/iEW6J4jeq MRng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MEyGuPKeUddZhaw30N5dtanpUK1tdzdpPP/MHcDXAoY=; b=es/VsZdDsjr50PxAOdoN14/eM88kXiJnLNZ5HjWwW/FWpXr6l3963Q7EBwhX+47ccb AoxK7t0Vn1N5CeLHntbWzNHz6HzlRcav78vv+jhQjGRMK1rFC7IZNLfPuGHj0dC0DBgN OCKSaMWIkrw1cZprSaO32lAHBk5gMJ/GHNkBjgOSMf/XGuTcH6tG7ggLwXWrhZv+QejF RT7saa9DAN8zvrWvMVqrPbx2EBDNm1yAvhSy8l9CpH4edpEzmxv3rfigwJJlLYJakxV0 CHPLGcWo2x/q2kvxKg+NHFE7sZ2TxNq7ff1zuKL1yYS57nooxCH48csjMfMCepPt/xxv UrgA== X-Gm-Message-State: AOAM530ibJe/YgE5p+3n+YRGQKz79V5laMFibtKFu0xijEjvJu6DIg1z 9pWeJQPU0d5skeSF25g20UE= X-Google-Smtp-Source: ABdhPJzlIlDcVKWeeZy4IEV1hXey+w+RB89Rx87jNDLazPzyjOIHoxIgfpBpmFTMS1Z3N1xJykru1Q== X-Received: by 2002:a9d:609e:: with SMTP id m30mr1544712otj.168.1624917695300; Mon, 28 Jun 2021 15:01:35 -0700 (PDT) Received: from localhost (2603-8081-140c-1a00-aaa9-75eb-6e0f-9f85.res6.spectrum.com. [2603:8081:140c:1a00:aaa9:75eb:6e0f:9f85]) by smtp.gmail.com with ESMTPSA id f138sm3462639oig.21.2021.06.28.15.01.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 15:01:35 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH 4/5] RDMA/rxe: Lookup kernel AH from ah index in UD WQEs Date: Mon, 28 Jun 2021 17:00:43 -0500 Message-Id: <20210628220043.9851-5-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210628220043.9851-1-rpearsonhpe@gmail.com> References: <20210628220043.9851-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Add code to rxe_get_av in rxe_av.c to use the AH index in UD send WQEs to lookup the kernel AH. For old user providers continue to use the AV passed in WQEs. Move setting pkt->rxe to before the call to rxe_get_av() to get access to the AH pool. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_av.c | 20 +++++++++++++++++++- drivers/infiniband/sw/rxe/rxe_req.c | 8 +++++--- 2 files changed, 24 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_av.c b/drivers/infiniband/sw/rxe/rxe_av.c index da2e867a1ed9..4176e00fe9df 100644 --- a/drivers/infiniband/sw/rxe/rxe_av.c +++ b/drivers/infiniband/sw/rxe/rxe_av.c @@ -101,11 +101,29 @@ void rxe_av_fill_ip_info(struct rxe_av *av, struct rdma_ah_attr *attr) struct rxe_av *rxe_get_av(struct rxe_pkt_info *pkt) { + struct rxe_ah *ah; + u32 ah_num; + if (!pkt || !pkt->qp) return NULL; if (qp_type(pkt->qp) == IB_QPT_RC || qp_type(pkt->qp) == IB_QPT_UC) return &pkt->qp->pri_av; - return (pkt->wqe) ? &pkt->wqe->av : NULL; + if (!pkt->wqe) + return NULL; + + ah_num = pkt->wqe->wr.wr.ud.ah_num; + if (ah_num) { + /* only new user provider or kernel client */ + ah = rxe_pool_get_index(&pkt->rxe->ah_pool, ah_num); + if (!ah || ah->ah_num != ah_num || ah_pd(ah) != pkt->qp->pd) { + pr_warn("Unable to find AH matching ah_num\n"); + return NULL; + } + return &ah->av; + } + + /* only old user provider for UD sends*/ + return &pkt->wqe->av; } diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index c57699cc6578..01485de2cc7d 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -390,9 +390,8 @@ static struct sk_buff *init_req_packet(struct rxe_qp *qp, /* length from start of bth to end of icrc */ paylen = rxe_opcode[opcode].length + payload + pad + RXE_ICRC_SIZE; - /* pkt->hdr, rxe, port_num and mask are initialized in ifc - * layer - */ + /* pkt->hdr, port_num and mask are initialized in ifc layer */ + pkt->rxe = rxe; pkt->opcode = opcode; pkt->qp = qp; pkt->psn = qp->req.psn; @@ -402,6 +401,9 @@ static struct sk_buff *init_req_packet(struct rxe_qp *qp, /* init skb */ av = rxe_get_av(pkt); + if (!av) + return NULL; + skb = rxe_init_packet(rxe, av, paylen, pkt); if (unlikely(!skb)) return NULL; From patchwork Mon Jun 28 22:00:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 12348593 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC65AC11F64 for ; Mon, 28 Jun 2021 22:02:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D4EA361CEF for ; Mon, 28 Jun 2021 22:02:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237380AbhF1WFB (ORCPT ); Mon, 28 Jun 2021 18:05:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52742 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238233AbhF1WEE (ORCPT ); Mon, 28 Jun 2021 18:04:04 -0400 Received: from mail-ot1-x32e.google.com (mail-ot1-x32e.google.com [IPv6:2607:f8b0:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7E154C06121D for ; Mon, 28 Jun 2021 15:01:36 -0700 (PDT) Received: by mail-ot1-x32e.google.com with SMTP id 59-20020a9d0ac10000b0290462f0ab0800so12569216otq.11 for ; Mon, 28 Jun 2021 15:01:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=EZrKiku+/v6vMSCb2MWkZlqaTxKjyI39A6bB7JbysTo=; b=DyMXh1Vg17ikh5JLJJpWyzyVfTkEuu8qUpRVxYdFAA9/KNtEhes2LZtlDLspSQMGk7 chCvVKYVVuq524dtE78QyT0/m2S1FIM8S6QFpXHHaiTWEpJ8W5U6YrAYCACGDqoprU38 rb1TvmQ3/tieIbsYXkEQSjIzujaIUXrtCGpyk7DuaiNSDmzEmORiVq9mu10SeaF6d6Si CCSD1+3vyLyL0K97OdBaWcez8iKHO+NAucNNSCFAFNZrHBaUhoaFusooi6nIdYI3Sa2C tHQ2rzQjeoDpZ/FbIWHkVsfS+WsscxO1s2AatD2Moy6/GG8kSFB5sQi4xYI3lhbru5/F Y0Qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=EZrKiku+/v6vMSCb2MWkZlqaTxKjyI39A6bB7JbysTo=; b=QJg7OVEXlv6RGEZySwS186+jjBXdQn6audJGOBIjOwVPVs48F/1RZ49ctKQmDKbbij 67MEE8zJwT52Kd/dydBk1SHCAroIsSWy6fgFKRXFExZs+3gk2AG78Y7v2Nb1pGwFO8rS 0XOu83jPbUf6oQO1m0W2S+bW++5uqKdeop5fORxUWWAC9eZ0bHHLQZF62rgte7LR4MWf 07u0c6qThA2t+3DgSRPEo+vIhXCxBGjrcg/DJxeYxAcglYv5o795VLSOyrDkCpSBeL3K O5EsVfsBIZoS4UZnYk0NjrubAEMzS3aPOX33/euStRvfh8ZuZU7qdqpk//psud+u88X5 WfTQ== X-Gm-Message-State: AOAM530FuwabKNcojHDkSaWXi5zPuG9V+T5muYoRchgy6eePPMQQ2JHh Cmt7HzpsmjguHXBWiDR0zYo= X-Google-Smtp-Source: ABdhPJxjRpdPg5crlrHMVUFDCVEE4NvzS8ToBnZBwBaCQl4K6JFDdI38TWF0/xXdP/fVbmwY1GCHVQ== X-Received: by 2002:a9d:8d0:: with SMTP id 74mr1534681otf.89.1624917695946; Mon, 28 Jun 2021 15:01:35 -0700 (PDT) Received: from localhost (2603-8081-140c-1a00-aaa9-75eb-6e0f-9f85.res6.spectrum.com. [2603:8081:140c:1a00:aaa9:75eb:6e0f:9f85]) by smtp.gmail.com with ESMTPSA id 2sm3623910ota.58.2021.06.28.15.01.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 15:01:35 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH 5/5] RDMA/rxe: Convert kernel UD post send to use ah_num Date: Mon, 28 Jun 2021 17:00:44 -0500 Message-Id: <20210628220043.9851-6-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210628220043.9851-1-rpearsonhpe@gmail.com> References: <20210628220043.9851-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Modify ib_post_send for kernel UD sends to put the AH index into the WQE instead of the address vector. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_verbs.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index 11d166bc11e3..820f67c846e9 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -558,8 +558,11 @@ static void init_send_wr(struct rxe_qp *qp, struct rxe_send_wr *wr, if (qp_type(qp) == IB_QPT_UD || qp_type(qp) == IB_QPT_SMI || qp_type(qp) == IB_QPT_GSI) { + struct ib_ah *ibah = ud_wr(ibwr)->ah; + wr->wr.ud.remote_qpn = ud_wr(ibwr)->remote_qpn; wr->wr.ud.remote_qkey = ud_wr(ibwr)->remote_qkey; + wr->wr.ud.ah_num = to_rah(ibah)->ah_num; if (qp_type(qp) == IB_QPT_GSI) wr->wr.ud.pkey_index = ud_wr(ibwr)->pkey_index; if (wr->opcode == IB_WR_SEND_WITH_IMM) @@ -631,11 +634,6 @@ static void init_send_wqe(struct rxe_qp *qp, const struct ib_send_wr *ibwr, return; } - if (qp_type(qp) == IB_QPT_UD || - qp_type(qp) == IB_QPT_SMI || - qp_type(qp) == IB_QPT_GSI) - memcpy(&wqe->av, &to_rah(ud_wr(ibwr)->ah)->av, sizeof(wqe->av)); - if (unlikely(ibwr->send_flags & IB_SEND_INLINE)) copy_inline_data_to_wqe(wqe, ibwr); else