From patchwork Tue May 30 22:13:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13261174 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 403E4C77B7A for ; Tue, 30 May 2023 22:13:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233666AbjE3WNx (ORCPT ); Tue, 30 May 2023 18:13:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41978 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233665AbjE3WNw (ORCPT ); Tue, 30 May 2023 18:13:52 -0400 Received: from mail-ot1-x333.google.com (mail-ot1-x333.google.com [IPv6:2607:f8b0:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0AF0DEC for ; Tue, 30 May 2023 15:13:51 -0700 (PDT) Received: by mail-ot1-x333.google.com with SMTP id 46e09a7af769-6af7d6f6f41so2235073a34.1 for ; Tue, 30 May 2023 15:13:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685484830; x=1688076830; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=L1LuLVWDztIGCc51u3Rxwm9gCFuUsa6Ji6Oea2b3QkI=; b=GBeP8zDBfTOglnTK2vDP8XtAXxu6ZHjxRYQInQ8KiNQ7zJ2RHyQbfWh0mcerBS0F4x xOqdn7v60ajIwEBef/KpnMowdP/ADEtV35lM2ZFm+j1hYG6h/hC/YxcPPAAD09YeBjod l5bS/GDwL6FjV8iWqLfMjawZXc6zJCwzubgY+GY9bJ9NURYvTmwGDaPq1pF8q/OdcvNb z1ko7lIVHlHXaDs1B0fics53UHLe2McElobtCAZYrewuzp7e4HVYRVrgeA4lkYN8fRNH PKL9dwICj4wAQL0wVA2xHbFgGBhB5xSp61Niu63fNAVjzuKurNtvN70p1OxlLr19+0t0 kI1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685484830; x=1688076830; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=L1LuLVWDztIGCc51u3Rxwm9gCFuUsa6Ji6Oea2b3QkI=; b=Ev//xTaJmdY3USOPEVr2tpJ0BG62tG+3hKSs66/K1mOyxjf3Zjkdtk79cczrNacyUb o2U0xn2E0LgLcp4UeH2925JsNKrRtbuMwGBhhOtWS8qGr8BQJUpn/WS/5FIGHRdyANSn +VTFNyeaYy0nGD7AHbIsPzUTj1KOWFZDbYtqv/ZpXlHnlSO4TSxknm5p/RQkTPv59gVt kiLFpoBJ94ERofcCHvezj2rPIfpgRgAtZgaiKVIFkZ5pRG1eEfZzyGnZDxVTQiTW1HGA SZ1bhLtKWj2cTrHNU80az3qH3ECslfMcG4kje61Iy3XnXDKuK+AjF3IVsK1/N88ZAs8z i7NA== X-Gm-Message-State: AC+VfDw498bYiD3MNRixFx4z1qbeFQ52fDW4J/esbW9Aqwli8FBh3eah 2wqxhZa28OkbPmCccJI/1Wr+tFuWMF4= X-Google-Smtp-Source: ACHHUZ7uckYt8HlOaIZ8VEGWi4mnYOQzvmHZPl2sFmcRrqAUO6C4IQ+tVIjO8/P3IZp+AEtyMcMuXA== X-Received: by 2002:a05:6808:306:b0:398:4601:4d06 with SMTP id i6-20020a056808030600b0039846014d06mr1708225oie.59.1685484830293; Tue, 30 May 2023 15:13:50 -0700 (PDT) Received: from rpearson-X570-AORUS-PRO-WIFI.tx.rr.com (2603-8081-140c-1a00-61e7-5a75-8a81-5bfc.res6.spectrum.com. [2603:8081:140c:1a00:61e7:5a75:8a81:5bfc]) by smtp.gmail.com with ESMTPSA id r77-20020a4a3750000000b00541854ce607sm6156772oor.28.2023.05.30.15.13.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 May 2023 15:13:49 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, edwards@nvidia.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next 1/6] RDMA/rxe: Rename IB_ACCESS_REMOTE Date: Tue, 30 May 2023 17:13:30 -0500 Message-Id: <20230530221334.89432-2-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230530221334.89432-1-rpearsonhpe@gmail.com> References: <20230530221334.89432-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Rename IB_ACCESS_REMOTE to RXE_ACCESS_REMOTE and move to rxe_verbs.h as an enum instead of a #define. Shouldn't use IB_xxx for rxe symbols. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_mr.c | 10 +++------- drivers/infiniband/sw/rxe/rxe_verbs.h | 6 ++++++ 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c index 0e538fafcc20..b3bc4ac5fedd 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -45,14 +45,10 @@ int mr_check_range(struct rxe_mr *mr, u64 iova, size_t length) } } -#define IB_ACCESS_REMOTE (IB_ACCESS_REMOTE_READ \ - | IB_ACCESS_REMOTE_WRITE \ - | IB_ACCESS_REMOTE_ATOMIC) - static void rxe_mr_init(int access, struct rxe_mr *mr) { u32 lkey = mr->elem.index << 8 | rxe_get_next_key(-1); - u32 rkey = (access & IB_ACCESS_REMOTE) ? lkey : 0; + u32 rkey = (access & RXE_ACCESS_REMOTE) ? lkey : 0; /* set ibmr->l/rkey and also copy into private l/rkey * for user MRs these will always be the same @@ -195,7 +191,7 @@ int rxe_mr_init_fast(int max_pages, struct rxe_mr *mr) int err; /* always allow remote access for FMRs */ - rxe_mr_init(IB_ACCESS_REMOTE, mr); + rxe_mr_init(RXE_ACCESS_REMOTE, mr); err = rxe_mr_alloc(mr, max_pages); if (err) @@ -715,7 +711,7 @@ int rxe_reg_fast_mr(struct rxe_qp *qp, struct rxe_send_wqe *wqe) mr->access = access; mr->lkey = key; - mr->rkey = (access & IB_ACCESS_REMOTE) ? key : 0; + mr->rkey = (access & RXE_ACCESS_REMOTE) ? key : 0; mr->ibmr.iova = wqe->wr.wr.reg.mr->iova; mr->state = RXE_MR_STATE_VALID; diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index 26a20f088692..0a2b7343e38f 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -253,6 +253,12 @@ struct rxe_qp { struct execute_work cleanup_work; }; +enum rxe_access { + RXE_ACCESS_REMOTE = (IB_ACCESS_REMOTE_READ + | IB_ACCESS_REMOTE_WRITE + | IB_ACCESS_REMOTE_ATOMIC), +}; + enum rxe_mr_state { RXE_MR_STATE_INVALID, RXE_MR_STATE_FREE, From patchwork Tue May 30 22:13:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13261175 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2FEC3C7EE23 for ; Tue, 30 May 2023 22:13:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233670AbjE3WNz (ORCPT ); Tue, 30 May 2023 18:13:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41982 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233631AbjE3WNx (ORCPT ); Tue, 30 May 2023 18:13:53 -0400 Received: from mail-oo1-xc35.google.com (mail-oo1-xc35.google.com [IPv6:2607:f8b0:4864:20::c35]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B647B93 for ; Tue, 30 May 2023 15:13:51 -0700 (PDT) Received: by mail-oo1-xc35.google.com with SMTP id 006d021491bc7-55564892accso2567848eaf.2 for ; Tue, 30 May 2023 15:13:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685484831; x=1688076831; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Dz2nwIGxOzz+SVosFIWkmYlpgXcgM7vnvgGkGH9MErI=; b=V4EuvVkkgfyM2sF3eDJgYYq6LsJUl8qpG0edGEKSgn9jkUIYF/UkBi3iZWtg7V3Urb rgSugAKOhnWUCwb/ij5iaz5vBnrJjkiV6HjcpfTcpoMnvTJRPh5ZFaEp8NHszERAlTam A6dRPGcD3K3BQRDk95XRXrQWG/QGWDVJY+dxYs3Q54D5LhlQJ0XiyzX/aQ09asBsVRkt h8Dk+72T4ogHbn+7ZVx7OLSLnRynrD0HufDeOpzshq46/euyUw+gr/7i9gc7cFZx6mfs zOPUoh+Cy0vGu5BfumM/r31Amc1sxIldsWPPIaGqNguQ3yLDmeIOGu9FDK4euKSds1lh p5Pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685484831; x=1688076831; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Dz2nwIGxOzz+SVosFIWkmYlpgXcgM7vnvgGkGH9MErI=; b=lQPPIJN/2UsuXR/60taLUjddWYbzSGgsP68wprVNVx4BtmnlETW/rIHpTvobTBnq7h 7ly/mL9hc8B90yCvlExMLlngck7RB4RXZCnIO+CkpBirILk/K0U6cIfZTJ34i57NhWdV FYGlRdGWxax6HICLq79Lb8W992rC6/boZ2GN6tj9NCBlssMHBLI/eKCHV0Ls6XCOnHte jdZF4WB1BGy69iBAczZY64QnuOnSw12yZYse8m1+QCDmNSjFyEkO3jMtJ5P1+Ov3QrMw Ttz9/ShQ8L9IrpPfokKHHOnrEgNraBNGLpOLCaP+azE0WrXCAe61QGeQNmkwaw914WyU QnPw== X-Gm-Message-State: AC+VfDy3buhIft+5KFn8VS6kJHyU3X+oWw4q/BrdC1/W56a2FX1W//1s RQgyZLLby3WLCxjv40HlrL2sVW1FewI= X-Google-Smtp-Source: ACHHUZ5qYhs6ypjxi5XP7ZHBPhfqespySQ2AhqOMoVg80LBI1Yny6c8qECOrJ/lBpDaIo2Hbj6IElw== X-Received: by 2002:a05:6808:181a:b0:398:45bf:6e65 with SMTP id bh26-20020a056808181a00b0039845bf6e65mr2118261oib.9.1685484831033; Tue, 30 May 2023 15:13:51 -0700 (PDT) Received: from rpearson-X570-AORUS-PRO-WIFI.tx.rr.com (2603-8081-140c-1a00-61e7-5a75-8a81-5bfc.res6.spectrum.com. [2603:8081:140c:1a00:61e7:5a75:8a81:5bfc]) by smtp.gmail.com with ESMTPSA id r77-20020a4a3750000000b00541854ce607sm6156772oor.28.2023.05.30.15.13.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 May 2023 15:13:50 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, edwards@nvidia.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next 2/6] rdma/rxe: Optimize send path in rxe_resp.c Date: Tue, 30 May 2023 17:13:31 -0500 Message-Id: <20230530221334.89432-3-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230530221334.89432-1-rpearsonhpe@gmail.com> References: <20230530221334.89432-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Bypass calling check_rkey() in rxe_resp.c for non-rdma messages. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_opcode.h | 3 +++ drivers/infiniband/sw/rxe/rxe_resp.c | 12 ++++++++++-- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_opcode.h b/drivers/infiniband/sw/rxe/rxe_opcode.h index cea4e0a63919..5686b691d6b8 100644 --- a/drivers/infiniband/sw/rxe/rxe_opcode.h +++ b/drivers/infiniband/sw/rxe/rxe_opcode.h @@ -91,6 +91,9 @@ enum rxe_hdr_mask { RXE_READ_OR_ATOMIC_MASK = (RXE_READ_MASK | RXE_ATOMIC_MASK), RXE_WRITE_OR_SEND_MASK = (RXE_WRITE_MASK | RXE_SEND_MASK), RXE_READ_OR_WRITE_MASK = (RXE_READ_MASK | RXE_WRITE_MASK), + RXE_RDMA_OP_MASK = (RXE_READ_MASK | RXE_WRITE_MASK | + RXE_ATOMIC_WRITE_MASK | RXE_FLUSH_MASK | + RXE_ATOMIC_MASK), }; #define OPCODE_NONE (-1) diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index b92c41cdb620..07299205242e 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -387,7 +387,10 @@ static enum resp_states rxe_resp_check_length(struct rxe_qp *qp, } } - return RESPST_CHK_RKEY; + if (pkt->mask & RXE_RDMA_OP_MASK) + return RESPST_CHK_RKEY; + else + return RESPST_EXECUTE; } /* if the reth length field is zero we can assume nothing @@ -434,6 +437,10 @@ static enum resp_states check_rkey(struct rxe_qp *qp, enum resp_states state; int access = 0; + /* parse RETH or ATMETH header for first/only packets + * for va, length, rkey, etc. or use current value for + * middle/last packets. + */ if (pkt->mask & (RXE_READ_OR_WRITE_MASK | RXE_ATOMIC_WRITE_MASK)) { if (pkt->mask & RXE_RETH_MASK) qp_resp_from_reth(qp, pkt); @@ -454,7 +461,8 @@ static enum resp_states check_rkey(struct rxe_qp *qp, qp_resp_from_atmeth(qp, pkt); access = IB_ACCESS_REMOTE_ATOMIC; } else { - return RESPST_EXECUTE; + /* shouldn't happen */ + WARN_ON(1); } /* A zero-byte read or write op is not required to From patchwork Tue May 30 22:13:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13261176 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3948C77B7A for ; Tue, 30 May 2023 22:13:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233217AbjE3WNz (ORCPT ); Tue, 30 May 2023 18:13:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41992 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233668AbjE3WNx (ORCPT ); Tue, 30 May 2023 18:13:53 -0400 Received: from mail-oo1-xc31.google.com (mail-oo1-xc31.google.com [IPv6:2607:f8b0:4864:20::c31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AE07BE5 for ; Tue, 30 May 2023 15:13:52 -0700 (PDT) Received: by mail-oo1-xc31.google.com with SMTP id 006d021491bc7-557ca32515eso2478251eaf.3 for ; Tue, 30 May 2023 15:13:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685484832; x=1688076832; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CM7bB/GOM7i6f1ZOjOjhI1OP8ejWKH7xFDVU2suPgaU=; b=I3A9ZTjrHeEesmIhz3NSLZL3tEW7fAbcMvO7wXMZyE4mgfcH3wrPixT0cAOnBQxFNQ LVRRusL42VP0hl4ltWBL3CTr1U0zI4HbPdwyVKB+SsMwkU9BnW6jgOaopHp6K1/QSub9 ywqjcepp/AxE0HHhSjJdpRKDEt97offaD4hJ6MHHp6aNAQ8QG8zwRP0exUJbh3JWUvfY GyiKcxudYfxwqcSGN6TMFVM9InBjUq1mzyEdQRMnfNSon/lcVqueSg3bm47im/urbbie 2tuan8dMP1VmNyoIw4UysWype/o4qFeDlxVGu1qJ4qKPMXrcWeuHy/a9vHty4BsgFNwC ID8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685484832; x=1688076832; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CM7bB/GOM7i6f1ZOjOjhI1OP8ejWKH7xFDVU2suPgaU=; b=lzjEYpDzuZRQQaYl75bddJJoKbLvRLUY24+PTcHXUfUXmLpOeP+WWzZlaZdwRBaO7H kSkwnboKa+Imk/uYpj1hABnks6LxcfktKUGZ6lXOFmHVrO6DZGFyEEG402z58qBeaEqm DkXjf+hH6Q4jVNQhhh9cANm/8HF5zazEM8XwknOesWG+nDvFgj4UlLXJmW32slrBf8I0 BMMc51yzg8d2z01tXgEr7azNYrS3FFBF1ays+fjAKH5C0/UMlv1515KT9kFNNhjVH+/m XKu5BfZjtBF3+acqXDXNqb/7xdwAjL5h+YWADKkV+IeAFgYiBRTZU5S/R3K94ndnnlkk C1uA== X-Gm-Message-State: AC+VfDz1YKphH3/ADAd/JR/sHejMYz10J+rRZQrc//bzt3dOAuZ9FAh5 8B88dGispCGEUDiKev7NT3ary75lI38= X-Google-Smtp-Source: ACHHUZ6+7AFWZQ5I7nHqry+WDLi5B0Y4MS/FfklFelpD91kxtx4Vfpok2R8lXiB6Txbj3I6lcwQjPQ== X-Received: by 2002:a4a:8c05:0:b0:555:50bb:2e2c with SMTP id u5-20020a4a8c05000000b0055550bb2e2cmr1352166ooj.4.1685484831864; Tue, 30 May 2023 15:13:51 -0700 (PDT) Received: from rpearson-X570-AORUS-PRO-WIFI.tx.rr.com (2603-8081-140c-1a00-61e7-5a75-8a81-5bfc.res6.spectrum.com. [2603:8081:140c:1a00:61e7:5a75:8a81:5bfc]) by smtp.gmail.com with ESMTPSA id r77-20020a4a3750000000b00541854ce607sm6156772oor.28.2023.05.30.15.13.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 May 2023 15:13:51 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, edwards@nvidia.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next 3/6] RDMA/rxe: Fix access checks in rxe_check_bind_mw Date: Tue, 30 May 2023 17:13:32 -0500 Message-Id: <20230530221334.89432-4-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230530221334.89432-1-rpearsonhpe@gmail.com> References: <20230530221334.89432-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The subroutine rxe_check_bind_mw() in rxe_mw.c performs checks on the mw access flags before they are set so they always succeed. This patch instead checks the access flags passed in the send wqe. Fixes: 32a577b4c3a9 ("RDMA/rxe: Add support for bind MW work requests") Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_mw.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_mw.c b/drivers/infiniband/sw/rxe/rxe_mw.c index afa5ce1a7116..a7ec57ab8fad 100644 --- a/drivers/infiniband/sw/rxe/rxe_mw.c +++ b/drivers/infiniband/sw/rxe/rxe_mw.c @@ -48,7 +48,7 @@ int rxe_dealloc_mw(struct ib_mw *ibmw) } static int rxe_check_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe, - struct rxe_mw *mw, struct rxe_mr *mr) + struct rxe_mw *mw, struct rxe_mr *mr, int access) { if (mw->ibmw.type == IB_MW_TYPE_1) { if (unlikely(mw->state != RXE_MW_STATE_VALID)) { @@ -58,7 +58,7 @@ static int rxe_check_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe, } /* o10-36.2.2 */ - if (unlikely((mw->access & IB_ZERO_BASED))) { + if (unlikely((access & IB_ZERO_BASED))) { rxe_dbg_mw(mw, "attempt to bind a zero based type 1 MW\n"); return -EINVAL; } @@ -104,7 +104,7 @@ static int rxe_check_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe, } /* C10-74 */ - if (unlikely((mw->access & + if (unlikely((access & (IB_ACCESS_REMOTE_WRITE | IB_ACCESS_REMOTE_ATOMIC)) && !(mr->access & IB_ACCESS_LOCAL_WRITE))) { rxe_dbg_mw(mw, @@ -113,7 +113,7 @@ static int rxe_check_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe, } /* C10-75 */ - if (mw->access & IB_ZERO_BASED) { + if (access & IB_ZERO_BASED) { if (unlikely(wqe->wr.wr.mw.length > mr->ibmr.length)) { rxe_dbg_mw(mw, "attempt to bind a ZB MW outside of the MR\n"); @@ -133,12 +133,12 @@ static int rxe_check_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe, } static void rxe_do_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe, - struct rxe_mw *mw, struct rxe_mr *mr) + struct rxe_mw *mw, struct rxe_mr *mr, int access) { u32 key = wqe->wr.wr.mw.rkey & 0xff; mw->rkey = (mw->rkey & ~0xff) | key; - mw->access = wqe->wr.wr.mw.access; + mw->access = access; mw->state = RXE_MW_STATE_VALID; mw->addr = wqe->wr.wr.mw.addr; mw->length = wqe->wr.wr.mw.length; @@ -169,6 +169,7 @@ int rxe_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe) struct rxe_dev *rxe = to_rdev(qp->ibqp.device); u32 mw_rkey = wqe->wr.wr.mw.mw_rkey; u32 mr_lkey = wqe->wr.wr.mw.mr_lkey; + int access = wqe->wr.wr.mw.access; mw = rxe_pool_get_index(&rxe->mw_pool, mw_rkey >> 8); if (unlikely(!mw)) { @@ -198,11 +199,11 @@ int rxe_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe) spin_lock_bh(&mw->lock); - ret = rxe_check_bind_mw(qp, wqe, mw, mr); + ret = rxe_check_bind_mw(qp, wqe, mw, mr, access); if (ret) goto err_unlock; - rxe_do_bind_mw(qp, wqe, mw, mr); + rxe_do_bind_mw(qp, wqe, mw, mr, access); err_unlock: spin_unlock_bh(&mw->lock); err_drop_mr: From patchwork Tue May 30 22:13:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13261177 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F886C7EE24 for ; Tue, 30 May 2023 22:13:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231346AbjE3WN4 (ORCPT ); Tue, 30 May 2023 18:13:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41996 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233669AbjE3WNy (ORCPT ); Tue, 30 May 2023 18:13:54 -0400 Received: from mail-ot1-x32c.google.com (mail-ot1-x32c.google.com [IPv6:2607:f8b0:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3AF18EC for ; Tue, 30 May 2023 15:13:53 -0700 (PDT) Received: by mail-ot1-x32c.google.com with SMTP id 46e09a7af769-6af93a6166fso3890635a34.3 for ; Tue, 30 May 2023 15:13:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685484832; x=1688076832; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jd+papdTmBQcLij6aQA1Ym+/2ejsB/pepvaXOrJzmTI=; b=FZFKfq67+UGMA15Dd6nlbxYsN1dRORDV1AtYitUbO7+P2ODLM32na/h5VCwhwJ3QQo 3WwWMan2kSnHw+VCkL9o+nDwhLQfGM5Ryf4DHA54yIGFBwSxXCbfoSqRt19bNSDlmJ/p GsGJRU4hINj8HNdyRr4pn8kX+FMAf1mE2Ap2av+OWi7IMfoAzFDGB6AtGhHpxO2iIf3o nzsuyqO+ifZg2LdW6IdvWR9iHDUMZV89OrfAJvjHRhi2+Z46/dErJI7C74c0/JHTX2FS 2cG8EbXbzPaDP4WE0TJPT+xMfPUK3PMq+sXzt/sF7UNOvITeJm1+8zTaUv7X7dTuWeV9 9JzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685484832; x=1688076832; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jd+papdTmBQcLij6aQA1Ym+/2ejsB/pepvaXOrJzmTI=; b=F9jxeraa3cVLUMY8Xj+B/7vgb+GkEB28MDdW9EKXloOKZnXFAyO4t48XykldnJzsva SsWfqlR6mJkMJmgRGW1aekgJJ2zkC+H4+LE3wCgwxvR+0j+cs81UJXZ3qEKC/mar/yaV 3VHwyxh2cnbOsi/Sakxdyy1d4lMlT2jJ8arlWmZhGIpMUv96pmoZxm75s2CKdmrbvnEe vqy30rwa83XyJAvz1xR/K7+IBe4hzCawjOya03SspLWYzpzzCaiEEWjRMTO2KiDmEHGV sqnWcc2jOfXDpRyraDZp7dzcgc1Er5XnZB3gze2KBN06dfJxmkn7R5EjhfyLmQhWseHY IjCg== X-Gm-Message-State: AC+VfDxwpzeGjST825tOBdGLl1ur7SgOcuuRRUvLfBTdtyAoKshpUC9m j41gL4Ybs/Q/tmJopz9bx0wczAGbPv8= X-Google-Smtp-Source: ACHHUZ7OEYE0MN0AJ49LwmQpl6hbrdWdHpoq8M/Y/A7Uqgj6oDxtKMfGGXqQtebUvEBdc2pcLUZJYw== X-Received: by 2002:a05:6808:291a:b0:394:39dc:a4b0 with SMTP id ev26-20020a056808291a00b0039439dca4b0mr1567324oib.52.1685484832542; Tue, 30 May 2023 15:13:52 -0700 (PDT) Received: from rpearson-X570-AORUS-PRO-WIFI.tx.rr.com (2603-8081-140c-1a00-61e7-5a75-8a81-5bfc.res6.spectrum.com. [2603:8081:140c:1a00:61e7:5a75:8a81:5bfc]) by smtp.gmail.com with ESMTPSA id r77-20020a4a3750000000b00541854ce607sm6156772oor.28.2023.05.30.15.13.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 May 2023 15:13:52 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, edwards@nvidia.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next 4/6] RDMA/rxe: Introduce rxe access supported flags Date: Tue, 30 May 2023 17:13:33 -0500 Message-Id: <20230530221334.89432-5-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230530221334.89432-1-rpearsonhpe@gmail.com> References: <20230530221334.89432-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Introduce supported bit masks for setting the access attributes of MWs, MRs, and QPs. Check these when attributes are set. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_mw.c | 5 +++++ drivers/infiniband/sw/rxe/rxe_qp.c | 7 +++++++ drivers/infiniband/sw/rxe/rxe_verbs.c | 6 ++++++ drivers/infiniband/sw/rxe/rxe_verbs.h | 15 ++++++++++++--- 4 files changed, 30 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_mw.c b/drivers/infiniband/sw/rxe/rxe_mw.c index a7ec57ab8fad..d8a43d87de93 100644 --- a/drivers/infiniband/sw/rxe/rxe_mw.c +++ b/drivers/infiniband/sw/rxe/rxe_mw.c @@ -197,6 +197,11 @@ int rxe_bind_mw(struct rxe_qp *qp, struct rxe_send_wqe *wqe) mr = NULL; } + if (access & ~RXE_ACCESS_SUPPORTED_MW) { + rxe_err_mw(mw, "access %#x not supported", access); + return -EOPNOTSUPP; + } + spin_lock_bh(&mw->lock); ret = rxe_check_bind_mw(qp, wqe, mw, mr, access); diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c index c5451a4488ca..95d4a6760c33 100644 --- a/drivers/infiniband/sw/rxe/rxe_qp.c +++ b/drivers/infiniband/sw/rxe/rxe_qp.c @@ -392,6 +392,13 @@ int rxe_qp_chk_attr(struct rxe_dev *rxe, struct rxe_qp *qp, if (mask & IB_QP_CAP && rxe_qp_chk_cap(rxe, &attr->cap, !!qp->srq)) goto err1; + if (mask & IB_QP_ACCESS_FLAGS) { + if (!(qp_type(qp) == IB_QPT_RC || qp_type(qp) == IB_QPT_UC)) + goto err1; + if (attr->qp_access_flags & ~RXE_ACCESS_SUPPORTED_QP) + goto err1; + } + if (mask & IB_QP_AV && rxe_av_chk_attr(qp, &attr->ah_attr)) goto err1; diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index dea605b7f683..bb2b9d40e242 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -1260,6 +1260,12 @@ static struct ib_mr *rxe_reg_user_mr(struct ib_pd *ibpd, u64 start, struct rxe_mr *mr; int err, cleanup_err; + if (access & ~RXE_ACCESS_SUPPORTED_MR) { + rxe_err_pd(pd, "access = %#x not supported (%#x)", access, + RXE_ACCESS_SUPPORTED_MR); + return ERR_PTR(-EOPNOTSUPP); + } + mr = kzalloc(sizeof(*mr), GFP_KERNEL); if (!mr) return ERR_PTR(-ENOMEM); diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index 0a2b7343e38f..2f2dc67f03dd 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -253,10 +253,19 @@ struct rxe_qp { struct execute_work cleanup_work; }; -enum rxe_access { - RXE_ACCESS_REMOTE = (IB_ACCESS_REMOTE_READ +enum { + RXE_ACCESS_REMOTE = IB_ACCESS_REMOTE_READ | IB_ACCESS_REMOTE_WRITE - | IB_ACCESS_REMOTE_ATOMIC), + | IB_ACCESS_REMOTE_ATOMIC, + RXE_ACCESS_SUPPORTED_MR = RXE_ACCESS_REMOTE + | IB_ACCESS_LOCAL_WRITE + | IB_ACCESS_MW_BIND + | IB_ACCESS_ON_DEMAND + | IB_ACCESS_FLUSH_GLOBAL + | IB_ACCESS_FLUSH_PERSISTENT, + RXE_ACCESS_SUPPORTED_QP = RXE_ACCESS_SUPPORTED_MR, + RXE_ACCESS_SUPPORTED_MW = RXE_ACCESS_SUPPORTED_MR + | IB_ZERO_BASED, }; enum rxe_mr_state { From patchwork Tue May 30 22:13:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13261178 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBCF5C7EE23 for ; Tue, 30 May 2023 22:13:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233437AbjE3WN6 (ORCPT ); Tue, 30 May 2023 18:13:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41998 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233665AbjE3WNy (ORCPT ); Tue, 30 May 2023 18:13:54 -0400 Received: from mail-oo1-xc35.google.com (mail-oo1-xc35.google.com [IPv6:2607:f8b0:4864:20::c35]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8EBBA93 for ; Tue, 30 May 2023 15:13:53 -0700 (PDT) Received: by mail-oo1-xc35.google.com with SMTP id 006d021491bc7-55564892accso2567876eaf.2 for ; Tue, 30 May 2023 15:13:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685484833; x=1688076833; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RROA+UCxw4Q5NEPetgrpyXLK0oTWWTCIDqOiKpL19pk=; b=HQyuAJW1gMpW3B+RiopZOee08bApkhpL/uN+m/XOGljziTNLIEixIyIlCDM2Cyuymn pzrdeGUyZfqJtOLUb4BI6dolbkPj3Vdb7RwhnF2406lsfP7OqLmmos11DQ+fgH8/s3sO QWUOxe5f3ksh+OE2ZwCF0QXPMILxrdElLFFu/EusY+lafKnSJbezR/iiTIRPqo8ZRN54 WPudIINuTVoXR9xbYwZxtRFBNL7oJj9sZ5FC2Lmx7N48MZT7FUF/j69hShQ1EPapIwvk 6tcS5goL4aWPPjon1TuI5WyfpMPSXOXuF2l66WXbU6UyV2GxyCbRtigB6TRuHH2Ir1fy 8ePg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685484833; x=1688076833; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RROA+UCxw4Q5NEPetgrpyXLK0oTWWTCIDqOiKpL19pk=; b=jF+wA0F0XA4lDx1moPRoEwP5RNc35Poh0fnBMd0zGaGGKifo/lrdGhOsrKgC+Xzvyv r59qJMAJgsml4mbJJ/Jq9Qhiyw0jP0F8Ow1JQyV6PzF2EsBBPEnVGnjU/NCH2NLev+Bl kVPcqzoyJUb+mNg0o8jha8T7/+gSkZoIhWxyExs/NqKVCCZCVGjq2TJeTePlWiGNpADD Ako8EvngZhCseqjEFcFufFNYvPAhR8Fzx4mo/IATEsOx8DAoh6xNhSAqB1kcNFjTIhlH hu41Wu/pR4iPgAEhaT+SWyZSt0fccHXYYxBFdklGJ3rhh8mnoxUv7KOQmaPwMt4Ulz2x Gccg== X-Gm-Message-State: AC+VfDzhjjZtiAx6tFsM3yGoG1ofbzdMIRIioSWpeGBu3wiLCHkIPHz/ P7wUbd1ypO2lFYrvnqUcmHM= X-Google-Smtp-Source: ACHHUZ6DP6FwAMu8C8P30BRR/tjwTheNDO07eJ1ARS8CXVQGIExmQDunwWBv31qoEpW22sV3dEcd2A== X-Received: by 2002:a05:6808:218d:b0:398:1c55:82bd with SMTP id be13-20020a056808218d00b003981c5582bdmr2955341oib.45.1685484833235; Tue, 30 May 2023 15:13:53 -0700 (PDT) Received: from rpearson-X570-AORUS-PRO-WIFI.tx.rr.com (2603-8081-140c-1a00-61e7-5a75-8a81-5bfc.res6.spectrum.com. [2603:8081:140c:1a00:61e7:5a75:8a81:5bfc]) by smtp.gmail.com with ESMTPSA id r77-20020a4a3750000000b00541854ce607sm6156772oor.28.2023.05.30.15.13.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 May 2023 15:13:52 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, edwards@nvidia.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next 5/6] RDMA/rxe: Let rkey == lkey for local access Date: Tue, 30 May 2023 17:13:34 -0500 Message-Id: <20230530221334.89432-6-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230530221334.89432-1-rpearsonhpe@gmail.com> References: <20230530221334.89432-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org In order to conform to other drivers stop using rkey == 0 as an indication that there are no remote access flags set. Set rkey == lkey by default for all MRs. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_mr.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c index b3bc4ac5fedd..f54042e9aeb2 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -47,16 +47,15 @@ int mr_check_range(struct rxe_mr *mr, u64 iova, size_t length) static void rxe_mr_init(int access, struct rxe_mr *mr) { - u32 lkey = mr->elem.index << 8 | rxe_get_next_key(-1); - u32 rkey = (access & RXE_ACCESS_REMOTE) ? lkey : 0; + u32 key = mr->elem.index << 8 | rxe_get_next_key(-1); /* set ibmr->l/rkey and also copy into private l/rkey * for user MRs these will always be the same * for cases where caller 'owns' the key portion * they may be different until REG_MR WQE is executed. */ - mr->lkey = mr->ibmr.lkey = lkey; - mr->rkey = mr->ibmr.rkey = rkey; + mr->lkey = mr->ibmr.lkey = key; + mr->rkey = mr->ibmr.rkey = key; mr->access = access; mr->ibmr.page_size = PAGE_SIZE; @@ -640,6 +639,7 @@ int rxe_invalidate_mr(struct rxe_qp *qp, u32 key) { struct rxe_dev *rxe = to_rdev(qp->ibqp.device); struct rxe_mr *mr; + int remote; int ret; mr = rxe_pool_get_index(&rxe->mr_pool, key >> 8); @@ -649,9 +649,10 @@ int rxe_invalidate_mr(struct rxe_qp *qp, u32 key) goto err; } - if (mr->rkey ? (key != mr->rkey) : (key != mr->lkey)) { + remote = mr->access & RXE_ACCESS_REMOTE; + if (remote ? (key != mr->rkey) : (key != mr->lkey)) { rxe_dbg_mr(mr, "wr key (%#x) doesn't match mr key (%#x)\n", - key, (mr->rkey ? mr->rkey : mr->lkey)); + key, (remote ? mr->rkey : mr->lkey)); ret = -EINVAL; goto err_drop_ref; } @@ -711,7 +712,7 @@ int rxe_reg_fast_mr(struct rxe_qp *qp, struct rxe_send_wqe *wqe) mr->access = access; mr->lkey = key; - mr->rkey = (access & RXE_ACCESS_REMOTE) ? key : 0; + mr->rkey = key; mr->ibmr.iova = wqe->wr.wr.reg.mr->iova; mr->state = RXE_MR_STATE_VALID; From patchwork Tue May 30 22:13:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Pearson X-Patchwork-Id: 13261179 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3181C77B7A for ; Tue, 30 May 2023 22:14:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233615AbjE3WN7 (ORCPT ); Tue, 30 May 2023 18:13:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42086 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233420AbjE3WN6 (ORCPT ); Tue, 30 May 2023 18:13:58 -0400 Received: from mail-oi1-x235.google.com (mail-oi1-x235.google.com [IPv6:2607:f8b0:4864:20::235]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 54C70E5 for ; Tue, 30 May 2023 15:13:56 -0700 (PDT) Received: by mail-oi1-x235.google.com with SMTP id 5614622812f47-3909756b8b1so1774466b6e.1 for ; Tue, 30 May 2023 15:13:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685484835; x=1688076835; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5joMuKjZqEF29Zy4iKhDoohhfPjvd0ykaakmkeg//1c=; b=p/853iM4wi+DCVC7rT3hPjcGI8zJo5u3M+qMCVGDDc0+iYl9mLFVpbB3vZEUxZ9HMX nPvOrw1a/k+HDc/6y8yLCL8Q7+KISkWRj7GZqJ9AIjLj3h2c3alsfKouJ/CR7e9IQvLD w2kY6+rqj6MRIkvtuKxm6THIiNDa0v/tTHRx8ByLLUbEmiK1IaR0IaTAX4dYtLLlhUue KNrNnTBQFQdUnXNrm43dDK6To261XlCqpugBRaPf1ElHgn6liW/NYmrZg3/GichHABUj QLlzhKdjrafeVdRCfhLDHjziPutdpEdHjhKPAvS7lC/3Qza7XQvO5+D+FfeES50V70ig Ju6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685484835; x=1688076835; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5joMuKjZqEF29Zy4iKhDoohhfPjvd0ykaakmkeg//1c=; b=Ct6sAWDBIxxbVdANOxJKq9+ETinTE0fQy0pUq3t9Untz5ryFNVKBjZZOUx5xZ1Durp 7468ZbBeEGnJN4BF7jnNU/LTz4AVfz2zDAxWoRnNr83kM+vMCC1KDhIzjfiVkDTT/tV3 ea3wdqG4A+/q33wy65ZubUlfpeQ4VpxBznb+BaVfsuuysS7qA9MZqlVxzipR8ht2MbjL hmWpmm4Bl7O7ltoXdWW41qOGDo89TK04pMVY+ifYXonMPvvnllVCJk1WNiEICVKkmxQL QiEdYzWXFHxnOQmkwFXEAgmvn/qCXwC9mHQvT5wF/sT5SS7v+i8ICHHC4sDSOI0gLG5l NC1A== X-Gm-Message-State: AC+VfDwYWCMrbziqrQyxs92xh9eBL3HX109PWOHjNGN7Q0UpP4Br4Cus JB964eovJZscWadhK5CuFQPh9pKAVzI= X-Google-Smtp-Source: ACHHUZ7jE4OfritkEFyZYGmJte5MfLHe4iGy6iYsTEMeSYtD5CaiPwL3RPIl99jLCUWGVyUoZSYzHg== X-Received: by 2002:a05:6808:2816:b0:39a:bf0:4fd2 with SMTP id et22-20020a056808281600b0039a0bf04fd2mr2003155oib.11.1685484834186; Tue, 30 May 2023 15:13:54 -0700 (PDT) Received: from rpearson-X570-AORUS-PRO-WIFI.tx.rr.com (2603-8081-140c-1a00-61e7-5a75-8a81-5bfc.res6.spectrum.com. [2603:8081:140c:1a00:61e7:5a75:8a81:5bfc]) by smtp.gmail.com with ESMTPSA id r77-20020a4a3750000000b00541854ce607sm6156772oor.28.2023.05.30.15.13.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 May 2023 15:13:53 -0700 (PDT) From: Bob Pearson To: jgg@nvidia.com, zyjzyj2000@gmail.com, edwards@nvidia.com, linux-rdma@vger.kernel.org Cc: Bob Pearson Subject: [PATCH for-next 6/6] RDMA/rxe: Implement rereg_user_mr Date: Tue, 30 May 2023 17:13:35 -0500 Message-Id: <20230530221334.89432-7-rpearsonhpe@gmail.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230530221334.89432-1-rpearsonhpe@gmail.com> References: <20230530221334.89432-1-rpearsonhpe@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Implement the two easy cases of ib_rereg_user_mr. Signed-off-by: Bob Pearson --- drivers/infiniband/sw/rxe/rxe_verbs.c | 35 +++++++++++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_verbs.h | 5 ++++ 2 files changed, 40 insertions(+) diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index bb2b9d40e242..f6396333bcef 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -1299,6 +1299,40 @@ static struct ib_mr *rxe_reg_user_mr(struct ib_pd *ibpd, u64 start, return ERR_PTR(err); } +static struct ib_mr *rxe_rereg_user_mr(struct ib_mr *ibmr, int flags, + u64 start, u64 length, u64 iova, + int access, struct ib_pd *ibpd, + struct ib_udata *udata) +{ + struct rxe_mr *mr = to_rmr(ibmr); + struct rxe_pd *old_pd = to_rpd(ibmr->pd); + struct rxe_pd *pd = to_rpd(ibpd); + + /* for now only support the two easy cases: + * rereg_pd and rereg_access + */ + if (flags & ~RXE_MR_REREG_SUPPORTED) { + rxe_err_mr(mr, "flags = %#x not supported", flags); + return ERR_PTR(-EOPNOTSUPP); + } + + if (flags & IB_MR_REREG_PD) { + rxe_put(old_pd); + rxe_get(pd); + mr->ibmr.pd = ibpd; + } + + if (flags & IB_MR_REREG_ACCESS) { + if (access & ~RXE_ACCESS_SUPPORTED_MR) { + rxe_err_mr(mr, "access = %#x not supported", access); + return ERR_PTR(-EOPNOTSUPP); + } + mr->access = access; + } + + return NULL; +} + static struct ib_mr *rxe_alloc_mr(struct ib_pd *ibpd, enum ib_mr_type mr_type, u32 max_num_sg) { @@ -1451,6 +1485,7 @@ static const struct ib_device_ops rxe_dev_ops = { .query_srq = rxe_query_srq, .reg_user_mr = rxe_reg_user_mr, .req_notify_cq = rxe_req_notify_cq, + .rereg_user_mr = rxe_rereg_user_mr, .resize_cq = rxe_resize_cq, INIT_RDMA_OBJ_SIZE(ib_ah, rxe_ah, ibah), diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index 2f2dc67f03dd..cb18b83b73c1 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -284,6 +284,11 @@ enum rxe_mr_lookup_type { RXE_LOOKUP_REMOTE, }; +enum rxe_rereg { + RXE_MR_REREG_SUPPORTED = IB_MR_REREG_PD + | IB_MR_REREG_ACCESS, +}; + static inline int rkey_is_mw(u32 rkey) { u32 index = rkey >> 8;