From patchwork Fri Apr 26 13:12:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Konstantin Taranov X-Patchwork-Id: 13644785 Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 4DEF8146D55; Fri, 26 Apr 2024 13:12:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=13.77.154.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714137168; cv=none; b=Ek23phF8LG7rqYUekb/XmxXlqOUCsGWyJNfnwd325ny7u5jWAsELB/6iI+p5M+3JEUBj/UvF7d1enz0/8jaTpsWn6cQeUouliCTWP2kgFn5CRce+2Bp02+AHenh1d4uvW+xi7pq8sAHi10XUQhSgpGkyWpNn+HCMD+4X7ze/npk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714137168; c=relaxed/simple; bh=Usu77U3GYcspHtKEiK5azthR80eo1pinrp+TtsB6MkU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=ul6oX/JnwLwGS3N9gxenTh+NwgkdbhA72ux2/qE3xCQgRnYMdsrA4u/oHpN9tKwDST1w2goEARBFoaCZyCcY3WhsS/6u7KDzE/op78CM7dH2XspflyWiYqxnKxivv7S/qUSFhJvEoCKN8934ISVCukllG9/NZmnj2WE5ln9VbkM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com; spf=pass smtp.mailfrom=linux.microsoft.com; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b=S47mHtzp; arc=none smtp.client-ip=13.77.154.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b="S47mHtzp" Received: from linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net (linux.microsoft.com [13.77.154.182]) by linux.microsoft.com (Postfix) with ESMTPSA id E3788210EF25; Fri, 26 Apr 2024 06:12:45 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com E3788210EF25 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1714137165; bh=ltCMK8jIH6z2pJvdSWPrhUfu9Ob4wk1gjWILq73eM6I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=S47mHtzpRhOyeXa5ecUVA7kMQWNoJJFbLbJwvrRke/JmsYyrKynNbvKEUjHhw3Cg+ /ON2Ni6W8aClC00SX2HxNjkOWH1mT/Iv8OpvUsAgjkokVRpeF5DUo2vB3+kRThKMYd INAA3ITsv/3IjiJJZJfir1OKm1o0Jrxcs9vYbiaU= From: Konstantin Taranov To: kotaranov@microsoft.com, sharmaajay@microsoft.com, longli@microsoft.com, jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH rdma-next v2 1/5] RDMA/mana_ib: create EQs for RNIC CQs Date: Fri, 26 Apr 2024 06:12:36 -0700 Message-Id: <1714137160-5222-2-git-send-email-kotaranov@linux.microsoft.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1714137160-5222-1-git-send-email-kotaranov@linux.microsoft.com> References: <1714137160-5222-1-git-send-email-kotaranov@linux.microsoft.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Konstantin Taranov Create EQs within mana_ib device. Such EQs are required for creation of RNIC CQs. Signed-off-by: Konstantin Taranov Reviewed-by: Long Li --- drivers/infiniband/hw/mana/main.c | 34 ++++++++++++++++++++++++++-- drivers/infiniband/hw/mana/mana_ib.h | 1 + 2 files changed, 33 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/hw/mana/main.c b/drivers/infiniband/hw/mana/main.c index f540147..546d059 100644 --- a/drivers/infiniband/hw/mana/main.c +++ b/drivers/infiniband/hw/mana/main.c @@ -658,7 +658,7 @@ int mana_ib_create_eqs(struct mana_ib_dev *mdev) { struct gdma_context *gc = mdev_to_gc(mdev); struct gdma_queue_spec spec = {}; - int err; + int err, i; spec.type = GDMA_EQ; spec.monitor_avl_buf = false; @@ -672,12 +672,42 @@ int mana_ib_create_eqs(struct mana_ib_dev *mdev) if (err) return err; + mdev->eqs = kcalloc(mdev->ib_dev.num_comp_vectors, sizeof(struct gdma_queue *), + GFP_KERNEL); + if (!mdev->eqs) { + err = -ENOMEM; + goto destroy_fatal_eq; + } + + for (i = 0; i < mdev->ib_dev.num_comp_vectors; i++) { + spec.eq.msix_index = (i + 1) % gc->num_msix_usable; + err = mana_gd_create_mana_eq(mdev->gdma_dev, &spec, &mdev->eqs[i]); + if (err) + goto destroy_eqs; + } + return 0; + +destroy_eqs: + while (i-- > 0) + mana_gd_destroy_queue(gc, mdev->eqs[i]); + kfree(mdev->eqs); +destroy_fatal_eq: + mana_gd_destroy_queue(gc, mdev->fatal_err_eq); + return err; } void mana_ib_destroy_eqs(struct mana_ib_dev *mdev) { - mana_gd_destroy_queue(mdev_to_gc(mdev), mdev->fatal_err_eq); + struct gdma_context *gc = mdev_to_gc(mdev); + int i; + + mana_gd_destroy_queue(gc, mdev->fatal_err_eq); + + for (i = 0; i < mdev->ib_dev.num_comp_vectors; i++) + mana_gd_destroy_queue(gc, mdev->eqs[i]); + + kfree(mdev->eqs); } int mana_ib_gd_create_rnic_adapter(struct mana_ib_dev *mdev) diff --git a/drivers/infiniband/hw/mana/mana_ib.h b/drivers/infiniband/hw/mana/mana_ib.h index 4c1240d..bfcf6df 100644 --- a/drivers/infiniband/hw/mana/mana_ib.h +++ b/drivers/infiniband/hw/mana/mana_ib.h @@ -56,6 +56,7 @@ struct mana_ib_dev { struct gdma_dev *gdma_dev; mana_handle_t adapter_handle; struct gdma_queue *fatal_err_eq; + struct gdma_queue **eqs; struct mana_ib_adapter_caps adapter_caps; }; From patchwork Fri Apr 26 13:12:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Konstantin Taranov X-Patchwork-Id: 13644783 Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 4DEB5146D52; Fri, 26 Apr 2024 13:12:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=13.77.154.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714137167; cv=none; b=JIxL8+iiNvK7xBhqB5sQQI47FSb2VKjcNiST57CYspCzjHlvpAQPip5ON3wuQl47BHPFkXvoQE8eHANrnmEa93y+cOecve2ua2gRIi1g4DTmQq9NqTvCd1aengJ3rPBxTOdK2Hr76nB81RbFoqfDFMF/Lh4vFUlAOonWDRqXNns= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714137167; c=relaxed/simple; bh=IgGML0fgJ8FIxgu9wUJmI92FBHaBdGYmeXjsJuCWcd0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=axmpQR7xCnGT/ukViB7fQ5PDffczoHJYPmdBKSxHjqaehSTS/AtIZ1xNvqmjZi2VDlA7J1NBMafZWex7PwESNgXSjutjcrVCBf+zRnsTbPSgsviYslzZ5dSDmQXibTocV3lEhm1cJT5Cpa7SIhayj+c4N0teW/rw07atltnihFo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com; spf=pass smtp.mailfrom=linux.microsoft.com; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b=e4Gzd42/; arc=none smtp.client-ip=13.77.154.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b="e4Gzd42/" Received: from linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net (linux.microsoft.com [13.77.154.182]) by linux.microsoft.com (Postfix) with ESMTPSA id 0827B210EF27; Fri, 26 Apr 2024 06:12:46 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 0827B210EF27 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1714137166; bh=gE7Ou1irGU+YLTnkid1jAQnRHHkTvBHMHZzISwJetCA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=e4Gzd42/wsr4nsQ4VCDE/BJDTrKmPveakT64D/+F8p1PV5cKyhfwOnrxw9SakOsif mzubS1w6bn5GIawW5+8mewLASjRBf6vbaoZBL6m6b/aZVxZ6aslJy+1i9HuHRViNCa tVziAoUMkysO3ukzVfJHsLwQF2V5syCqXeoTJqW4= From: Konstantin Taranov To: kotaranov@microsoft.com, sharmaajay@microsoft.com, longli@microsoft.com, jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH rdma-next v2 2/5] RDMA/mana_ib: create and destroy RNIC cqs Date: Fri, 26 Apr 2024 06:12:37 -0700 Message-Id: <1714137160-5222-3-git-send-email-kotaranov@linux.microsoft.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1714137160-5222-1-git-send-email-kotaranov@linux.microsoft.com> References: <1714137160-5222-1-git-send-email-kotaranov@linux.microsoft.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Konstantin Taranov Implement RNIC requests for creation and destruction of RNIC CQs. Signed-off-by: Konstantin Taranov Reviewed-by: Long Li --- drivers/infiniband/hw/mana/main.c | 54 ++++++++++++++++++++++++++++ drivers/infiniband/hw/mana/mana_ib.h | 32 +++++++++++++++++ 2 files changed, 86 insertions(+) diff --git a/drivers/infiniband/hw/mana/main.c b/drivers/infiniband/hw/mana/main.c index 546d059..2a41135 100644 --- a/drivers/infiniband/hw/mana/main.c +++ b/drivers/infiniband/hw/mana/main.c @@ -834,3 +834,57 @@ int mana_ib_gd_config_mac(struct mana_ib_dev *mdev, enum mana_ib_addr_op op, u8 return 0; } + +int mana_ib_gd_create_cq(struct mana_ib_dev *mdev, struct mana_ib_cq *cq, u32 doorbell) +{ + struct gdma_context *gc = mdev_to_gc(mdev); + struct mana_rnic_create_cq_resp resp = {}; + struct mana_rnic_create_cq_req req = {}; + int err; + + mana_gd_init_req_hdr(&req.hdr, MANA_IB_CREATE_CQ, sizeof(req), sizeof(resp)); + req.hdr.dev_id = gc->mana_ib.dev_id; + req.adapter = mdev->adapter_handle; + req.gdma_region = cq->queue.gdma_region; + req.eq_id = mdev->eqs[cq->comp_vector]->id; + req.doorbell_page = doorbell; + + err = mana_gd_send_request(gc, sizeof(req), &req, sizeof(resp), &resp); + + if (err) { + ibdev_err(&mdev->ib_dev, "Failed to create cq err %d", err); + return err; + } + + cq->queue.id = resp.cq_id; + cq->cq_handle = resp.cq_handle; + /* The GDMA region is now owned by the CQ handle */ + cq->queue.gdma_region = GDMA_INVALID_DMA_REGION; + + return 0; +} + +int mana_ib_gd_destroy_cq(struct mana_ib_dev *mdev, struct mana_ib_cq *cq) +{ + struct gdma_context *gc = mdev_to_gc(mdev); + struct mana_rnic_destroy_cq_resp resp = {}; + struct mana_rnic_destroy_cq_req req = {}; + int err; + + if (cq->cq_handle == INVALID_MANA_HANDLE) + return 0; + + mana_gd_init_req_hdr(&req.hdr, MANA_IB_DESTROY_CQ, sizeof(req), sizeof(resp)); + req.hdr.dev_id = gc->mana_ib.dev_id; + req.adapter = mdev->adapter_handle; + req.cq_handle = cq->cq_handle; + + err = mana_gd_send_request(gc, sizeof(req), &req, sizeof(resp), &resp); + + if (err) { + ibdev_err(&mdev->ib_dev, "Failed to destroy cq err %d", err); + return err; + } + + return 0; +} diff --git a/drivers/infiniband/hw/mana/mana_ib.h b/drivers/infiniband/hw/mana/mana_ib.h index bfcf6df..9162f29 100644 --- a/drivers/infiniband/hw/mana/mana_ib.h +++ b/drivers/infiniband/hw/mana/mana_ib.h @@ -92,6 +92,7 @@ struct mana_ib_cq { struct mana_ib_queue queue; int cqe; u32 comp_vector; + mana_handle_t cq_handle; }; struct mana_ib_qp { @@ -119,6 +120,8 @@ enum mana_ib_command_code { MANA_IB_DESTROY_ADAPTER = 0x30003, MANA_IB_CONFIG_IP_ADDR = 0x30004, MANA_IB_CONFIG_MAC_ADDR = 0x30005, + MANA_IB_CREATE_CQ = 0x30008, + MANA_IB_DESTROY_CQ = 0x30009, }; struct mana_ib_query_adapter_caps_req { @@ -202,6 +205,31 @@ struct mana_rnic_config_mac_addr_resp { struct gdma_resp_hdr hdr; }; /* HW Data */ +struct mana_rnic_create_cq_req { + struct gdma_req_hdr hdr; + mana_handle_t adapter; + u64 gdma_region; + u32 eq_id; + u32 doorbell_page; +}; /* HW Data */ + +struct mana_rnic_create_cq_resp { + struct gdma_resp_hdr hdr; + mana_handle_t cq_handle; + u32 cq_id; + u32 reserved; +}; /* HW Data */ + +struct mana_rnic_destroy_cq_req { + struct gdma_req_hdr hdr; + mana_handle_t adapter; + mana_handle_t cq_handle; +}; /* HW Data */ + +struct mana_rnic_destroy_cq_resp { + struct gdma_resp_hdr hdr; +}; /* HW Data */ + static inline struct gdma_context *mdev_to_gc(struct mana_ib_dev *mdev) { return mdev->gdma_dev->gdma_context; @@ -321,4 +349,8 @@ int mana_ib_gd_add_gid(const struct ib_gid_attr *attr, void **context); int mana_ib_gd_del_gid(const struct ib_gid_attr *attr, void **context); int mana_ib_gd_config_mac(struct mana_ib_dev *mdev, enum mana_ib_addr_op op, u8 *mac); + +int mana_ib_gd_create_cq(struct mana_ib_dev *mdev, struct mana_ib_cq *cq, u32 doorbell); + +int mana_ib_gd_destroy_cq(struct mana_ib_dev *mdev, struct mana_ib_cq *cq); #endif From patchwork Fri Apr 26 13:12:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Konstantin Taranov X-Patchwork-Id: 13644784 Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 79BA0146D62; Fri, 26 Apr 2024 13:12:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=13.77.154.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714137167; cv=none; b=FlXvm3qUtLgCfMpmLSVlnwu21FWa0lfuKpT5UB9nTRnrTNLVCOVDOOWF2TLLCnTH9ZVAhzcAss8stGQmoun1c/tP+Hyx1ekdgkHJnteO9TVK3brJhrRhQ0Vml3J7+i2tLNWFT18bqON+T0r1jEWX4c+AJyqzGachYwJvquAYBHU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714137167; c=relaxed/simple; bh=AbVyBbv8hoN9cGSNf2K0NleJvEmeRiwc6JqObMe0nuY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=D2e7Ahhf3A7Q1KXZLAJBQaTiGNGVETCEcPZBCgoKT+OmszMa5eNOw3LvOcsyMxSLZU06wAv8u7mcus42uZ+lIVD3/k68U+Oy1Yqfmn8xdV5n4mCs7Sht40TD5qb/tUWtHO63HTx/x/g+ijXQQxDbAcc/yDRMUw4LCjQjLJQlhx4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com; spf=pass smtp.mailfrom=linux.microsoft.com; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b=adTg/DOW; arc=none smtp.client-ip=13.77.154.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b="adTg/DOW" Received: from linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net (linux.microsoft.com [13.77.154.182]) by linux.microsoft.com (Postfix) with ESMTPSA id 20B99210EF28; Fri, 26 Apr 2024 06:12:46 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 20B99210EF28 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1714137166; bh=pQic5QgUwNhMTp2g55PyxrZma9wDByHV+2LR4eUCugQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=adTg/DOWCwOJ/6dPvpZyHbkboHgq6qZJP0/earzmMOsiQjhrHyGgyIfZU2Nq8w2C/ eOp8+5uBcQGgUAOop0X3X+CYfBwUoFBguLq/lbsrO6DWdzV4HGJyG59XyD6WrwImz2 1TbeHqEQTuQ7SnGijOUJbocP2bmlYcYL5mAoVXt4= From: Konstantin Taranov To: kotaranov@microsoft.com, sharmaajay@microsoft.com, longli@microsoft.com, jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH rdma-next v2 3/5] RDMA/mana_ib: introduce a helper to remove cq callbacks Date: Fri, 26 Apr 2024 06:12:38 -0700 Message-Id: <1714137160-5222-4-git-send-email-kotaranov@linux.microsoft.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1714137160-5222-1-git-send-email-kotaranov@linux.microsoft.com> References: <1714137160-5222-1-git-send-email-kotaranov@linux.microsoft.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Konstantin Taranov Intoduce the mana_ib_remove_cq_cb helper to remove cq callbacks. The helper removes code duplicates. Signed-off-by: Konstantin Taranov Reviewed-by: Long Li --- drivers/infiniband/hw/mana/cq.c | 19 ++++++++++++------- drivers/infiniband/hw/mana/mana_ib.h | 1 + drivers/infiniband/hw/mana/qp.c | 26 ++++---------------------- 3 files changed, 17 insertions(+), 29 deletions(-) diff --git a/drivers/infiniband/hw/mana/cq.c b/drivers/infiniband/hw/mana/cq.c index dc931b9..298e8f1 100644 --- a/drivers/infiniband/hw/mana/cq.c +++ b/drivers/infiniband/hw/mana/cq.c @@ -48,16 +48,10 @@ int mana_ib_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata) struct mana_ib_cq *cq = container_of(ibcq, struct mana_ib_cq, ibcq); struct ib_device *ibdev = ibcq->device; struct mana_ib_dev *mdev; - struct gdma_context *gc; mdev = container_of(ibdev, struct mana_ib_dev, ib_dev); - gc = mdev_to_gc(mdev); - - if (cq->queue.id != INVALID_QUEUE_ID) { - kfree(gc->cq_table[cq->queue.id]); - gc->cq_table[cq->queue.id] = NULL; - } + mana_ib_remove_cq_cb(mdev, cq); mana_ib_destroy_queue(mdev, &cq->queue); return 0; @@ -89,3 +83,14 @@ int mana_ib_install_cq_cb(struct mana_ib_dev *mdev, struct mana_ib_cq *cq) gc->cq_table[cq->queue.id] = gdma_cq; return 0; } + +void mana_ib_remove_cq_cb(struct mana_ib_dev *mdev, struct mana_ib_cq *cq) +{ + struct gdma_context *gc = mdev_to_gc(mdev); + + if (cq->queue.id >= gc->max_num_cqs || cq->queue.id == INVALID_QUEUE_ID) + return; + + kfree(gc->cq_table[cq->queue.id]); + gc->cq_table[cq->queue.id] = NULL; +} diff --git a/drivers/infiniband/hw/mana/mana_ib.h b/drivers/infiniband/hw/mana/mana_ib.h index 9162f29..68c3b4f 100644 --- a/drivers/infiniband/hw/mana/mana_ib.h +++ b/drivers/infiniband/hw/mana/mana_ib.h @@ -255,6 +255,7 @@ static inline void copy_in_reverse(u8 *dst, const u8 *src, u32 size) } int mana_ib_install_cq_cb(struct mana_ib_dev *mdev, struct mana_ib_cq *cq); +void mana_ib_remove_cq_cb(struct mana_ib_dev *mdev, struct mana_ib_cq *cq); int mana_ib_create_zero_offset_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem, mana_handle_t *gdma_region); diff --git a/drivers/infiniband/hw/mana/qp.c b/drivers/infiniband/hw/mana/qp.c index 280e85a..ba13c5a 100644 --- a/drivers/infiniband/hw/mana/qp.c +++ b/drivers/infiniband/hw/mana/qp.c @@ -95,11 +95,9 @@ static int mana_ib_create_qp_rss(struct ib_qp *ibqp, struct ib_pd *pd, struct mana_ib_qp *qp = container_of(ibqp, struct mana_ib_qp, ibqp); struct mana_ib_dev *mdev = container_of(pd->device, struct mana_ib_dev, ib_dev); - struct gdma_context *gc = mdev_to_gc(mdev); struct ib_rwq_ind_table *ind_tbl = attr->rwq_ind_tbl; struct mana_ib_create_qp_rss_resp resp = {}; struct mana_ib_create_qp_rss ucmd = {}; - struct gdma_queue **gdma_cq_allocated; mana_handle_t *mana_ind_table; struct mana_port_context *mpc; unsigned int ind_tbl_size; @@ -173,13 +171,6 @@ static int mana_ib_create_qp_rss(struct ib_qp *ibqp, struct ib_pd *pd, goto fail; } - gdma_cq_allocated = kcalloc(ind_tbl_size, sizeof(*gdma_cq_allocated), - GFP_KERNEL); - if (!gdma_cq_allocated) { - ret = -ENOMEM; - goto fail; - } - qp->port = port; for (i = 0; i < ind_tbl_size; i++) { @@ -229,8 +220,6 @@ static int mana_ib_create_qp_rss(struct ib_qp *ibqp, struct ib_pd *pd, ret = mana_ib_install_cq_cb(mdev, cq); if (ret) goto fail; - - gdma_cq_allocated[i] = gc->cq_table[cq->queue.id]; } resp.num_entries = i; @@ -250,7 +239,6 @@ static int mana_ib_create_qp_rss(struct ib_qp *ibqp, struct ib_pd *pd, goto fail; } - kfree(gdma_cq_allocated); kfree(mana_ind_table); return 0; @@ -262,13 +250,10 @@ fail: wq = container_of(ibwq, struct mana_ib_wq, ibwq); cq = container_of(ibcq, struct mana_ib_cq, ibcq); - gc->cq_table[cq->queue.id] = NULL; - kfree(gdma_cq_allocated[i]); - + mana_ib_remove_cq_cb(mdev, cq); mana_destroy_wq_obj(mpc, GDMA_RQ, wq->rx_object); } - kfree(gdma_cq_allocated); kfree(mana_ind_table); return ret; @@ -287,10 +272,8 @@ static int mana_ib_create_qp_raw(struct ib_qp *ibqp, struct ib_pd *ibpd, struct mana_ib_ucontext *mana_ucontext = rdma_udata_to_drv_context(udata, struct mana_ib_ucontext, ibucontext); - struct gdma_context *gc = mdev_to_gc(mdev); struct mana_ib_create_qp_resp resp = {}; struct mana_ib_create_qp ucmd = {}; - struct gdma_queue *gdma_cq = NULL; struct mana_obj_spec wq_spec = {}; struct mana_obj_spec cq_spec = {}; struct mana_port_context *mpc; @@ -395,14 +378,13 @@ static int mana_ib_create_qp_raw(struct ib_qp *ibqp, struct ib_pd *ibpd, ibdev_dbg(&mdev->ib_dev, "Failed copy udata for create qp-raw, %d\n", err); - goto err_release_gdma_cq; + goto err_remove_cq_cb; } return 0; -err_release_gdma_cq: - kfree(gdma_cq); - gc->cq_table[send_cq->queue.id] = NULL; +err_remove_cq_cb: + mana_ib_remove_cq_cb(mdev, send_cq); err_destroy_wq_obj: mana_destroy_wq_obj(mpc, GDMA_SQ, qp->qp_handle); From patchwork Fri Apr 26 13:12:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Konstantin Taranov X-Patchwork-Id: 13644786 Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 8190F146D69; Fri, 26 Apr 2024 13:12:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=13.77.154.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714137169; cv=none; b=dc8L3JGM5HEXRyFHG3C6MSbSzXK7eF4JaH9NKKnEyckgxHLKz6v3vMIerUqetETBJr9N3X5CFQLdLtfqB6zu3SncUvx+Lx6AByCwLq/PYiLmVdOgNNBjIFcWDkVsmg6IQOZGQqe+DkYJMGI7zv+eMtBSIKMY5XJRH5/0+KZDHv0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714137169; c=relaxed/simple; bh=71RwQwi9TFTUPNDyPLmoKi09b5OBg6kwngITmrLKZ2Y=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=aOmLgGoV2vn4ui6UJox/ws8MkgJsJrvXTbmvCDIlHk+b0tAiqITFf7CvAixDNpvRzbyYklLECjXajlAR66n3PrLuZPnfyEvkow+brdeAuu+p3Z/sC63z9raV8G7DsJS84PQR4qVgkkeJB4h5gKOkznEOe/4exYMTX4u0RiHZfgI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com; spf=pass smtp.mailfrom=linux.microsoft.com; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b=mUjKyMn7; arc=none smtp.client-ip=13.77.154.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b="mUjKyMn7" Received: from linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net (linux.microsoft.com [13.77.154.182]) by linux.microsoft.com (Postfix) with ESMTPSA id 39D7C210EF29; Fri, 26 Apr 2024 06:12:46 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 39D7C210EF29 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1714137166; bh=HfSrTbBlObJZ5QgMLDom1PtUczMN227beKLXYqpagmc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mUjKyMn7qOr7/Q+0ayG2nOy7nleg3db2XX5+gBFcQ7+i0kyuntO8WfMloWqgy0vtN Ad/7NFleTrYKOvmaKR/OLa3pI1P+gyXL4KsBpfpwgTR16MdV3VIr55WQ+miwQOhtQI sv698RBGFXQ18nymwLF+//vM7ol27Rs8cqLeqTjk= From: Konstantin Taranov To: kotaranov@microsoft.com, sharmaajay@microsoft.com, longli@microsoft.com, jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH rdma-next v2 4/5] RDMA/mana_ib: boundary check before installing cq callbacks Date: Fri, 26 Apr 2024 06:12:39 -0700 Message-Id: <1714137160-5222-5-git-send-email-kotaranov@linux.microsoft.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1714137160-5222-1-git-send-email-kotaranov@linux.microsoft.com> References: <1714137160-5222-1-git-send-email-kotaranov@linux.microsoft.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Konstantin Taranov Add a boundary check inside mana_ib_install_cq_cb to prevent index overflow. Fixes: 2a31c5a7e0d8 ("RDMA/mana_ib: Introduce mana_ib_install_cq_cb helper function") Signed-off-by: Konstantin Taranov Reviewed-by: Long Li --- drivers/infiniband/hw/mana/cq.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/infiniband/hw/mana/cq.c b/drivers/infiniband/hw/mana/cq.c index 298e8f1..688ffe6 100644 --- a/drivers/infiniband/hw/mana/cq.c +++ b/drivers/infiniband/hw/mana/cq.c @@ -70,6 +70,8 @@ int mana_ib_install_cq_cb(struct mana_ib_dev *mdev, struct mana_ib_cq *cq) struct gdma_context *gc = mdev_to_gc(mdev); struct gdma_queue *gdma_cq; + if (cq->queue.id >= gc->max_num_cqs) + return -EINVAL; /* Create CQ table entry */ WARN_ON(gc->cq_table[cq->queue.id]); gdma_cq = kzalloc(sizeof(*gdma_cq), GFP_KERNEL); From patchwork Fri Apr 26 13:12:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Konstantin Taranov X-Patchwork-Id: 13644787 Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D66271474D8; Fri, 26 Apr 2024 13:12:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=13.77.154.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714137169; cv=none; b=Jhix8OTJMd8/roBskFNN7tPA/Ki2MwLV2wNA/+XTtRyfOWBNmwPdj6UhU0++jNRGL2Br+bkaa2NipB5dv6Y2q1ZyPGoOCeC1tPIVqtG+wnWEP5w8Ubf3NXY3uB9oXM2kcvN2kHJiG52bJJW8KaKTsZwuh7bBVVoXlFn1CvS1Fuc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714137169; c=relaxed/simple; bh=rr6aW1+kcjEmEqWlckgXGdFCPP642C4Jtl5MwHN1w0Q=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=FAGIOfF4H3LpNdjLCT1gEAaoDcQ9CaZOBs3/35HOnkqiW1970XPYmFOcizye+C0ip3wJbObV6UR1ztMUuRNAaKB0kv0oBSPGW8VuKIYwzCMG8+k99VOfS6TLSZsgmZlpiDl8QzhFoIqwKGn+56PVZvNPuoBAVT3lDQh+5bcj4aU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com; spf=pass smtp.mailfrom=linux.microsoft.com; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b=fli+4dtA; arc=none smtp.client-ip=13.77.154.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b="fli+4dtA" Received: from linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net (linux.microsoft.com [13.77.154.182]) by linux.microsoft.com (Postfix) with ESMTPSA id 52C76210EF2A; Fri, 26 Apr 2024 06:12:46 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 52C76210EF2A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1714137166; bh=Mq+rN7E58bisCALoStLLEzSdNeZidytgD0uOuQCvnM8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fli+4dtAQ6pk/mK1bVD3m43hgWdV9DhUCClxNGUdoxHKu6oGYFawF8nbLhxLyLMFW GH1WY8Sh0vl8keZc/eKCpNsfrvXgHeh8wLSaX2TSNZxU3hb1+SWjh/Dp5vTY7zjHFl HruaoEH57IV91QwAwSF5ohjfY52Vs0w25tB0m0cQ= From: Konstantin Taranov To: kotaranov@microsoft.com, sharmaajay@microsoft.com, longli@microsoft.com, jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH rdma-next v2 5/5] RDMA/mana_ib: implement uapi for creation of rnic cq Date: Fri, 26 Apr 2024 06:12:40 -0700 Message-Id: <1714137160-5222-6-git-send-email-kotaranov@linux.microsoft.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1714137160-5222-1-git-send-email-kotaranov@linux.microsoft.com> References: <1714137160-5222-1-git-send-email-kotaranov@linux.microsoft.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Konstantin Taranov Enable users to create RNIC CQs using a corresponding flag. With the previous request size, an ethernet CQ is created. As a response, return ID of the created CQ. Signed-off-by: Konstantin Taranov Reviewed-by: Long Li --- drivers/infiniband/hw/mana/cq.c | 55 ++++++++++++++++++++++++++++++--- include/uapi/rdma/mana-abi.h | 12 +++++++ 2 files changed, 63 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/hw/mana/cq.c b/drivers/infiniband/hw/mana/cq.c index 688ffe6..c6a3fd5 100644 --- a/drivers/infiniband/hw/mana/cq.c +++ b/drivers/infiniband/hw/mana/cq.c @@ -9,17 +9,22 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, struct ib_udata *udata) { struct mana_ib_cq *cq = container_of(ibcq, struct mana_ib_cq, ibcq); + struct mana_ib_create_cq_resp resp = {}; + struct mana_ib_ucontext *mana_ucontext; struct ib_device *ibdev = ibcq->device; struct mana_ib_create_cq ucmd = {}; struct mana_ib_dev *mdev; + bool is_rnic_cq; + u32 doorbell; int err; mdev = container_of(ibdev, struct mana_ib_dev, ib_dev); - if (udata->inlen < sizeof(ucmd)) - return -EINVAL; - cq->comp_vector = attr->comp_vector % ibdev->num_comp_vectors; + cq->cq_handle = INVALID_MANA_HANDLE; + + if (udata->inlen < offsetof(struct mana_ib_create_cq, flags)) + return -EINVAL; err = ib_copy_from_udata(&ucmd, udata, min(sizeof(ucmd), udata->inlen)); if (err) { @@ -28,7 +33,9 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, return err; } - if (attr->cqe > mdev->adapter_caps.max_qp_wr) { + is_rnic_cq = !!(ucmd.flags & MANA_IB_CREATE_RNIC_CQ); + + if (!is_rnic_cq && attr->cqe > mdev->adapter_caps.max_qp_wr) { ibdev_dbg(ibdev, "CQE %d exceeding limit\n", attr->cqe); return -EINVAL; } @@ -40,7 +47,41 @@ int mana_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, return err; } + mana_ucontext = rdma_udata_to_drv_context(udata, struct mana_ib_ucontext, + ibucontext); + doorbell = mana_ucontext->doorbell; + + if (is_rnic_cq) { + err = mana_ib_gd_create_cq(mdev, cq, doorbell); + if (err) { + ibdev_dbg(ibdev, "Failed to create RNIC cq, %d\n", err); + goto err_destroy_queue; + } + + err = mana_ib_install_cq_cb(mdev, cq); + if (err) { + ibdev_dbg(ibdev, "Failed to install cq callback, %d\n", err); + goto err_destroy_rnic_cq; + } + } + + resp.cqid = cq->queue.id; + err = ib_copy_to_udata(udata, &resp, min(sizeof(resp), udata->outlen)); + if (err) { + ibdev_dbg(&mdev->ib_dev, "Failed to copy to udata, %d\n", err); + goto err_remove_cq_cb; + } + return 0; + +err_remove_cq_cb: + mana_ib_remove_cq_cb(mdev, cq); +err_destroy_rnic_cq: + mana_ib_gd_destroy_cq(mdev, cq); +err_destroy_queue: + mana_ib_destroy_queue(mdev, &cq->queue); + + return err; } int mana_ib_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata) @@ -52,6 +93,12 @@ int mana_ib_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata) mdev = container_of(ibdev, struct mana_ib_dev, ib_dev); mana_ib_remove_cq_cb(mdev, cq); + + /* Ignore return code as there is not much we can do about it. + * The error message is printed inside. + */ + mana_ib_gd_destroy_cq(mdev, cq); + mana_ib_destroy_queue(mdev, &cq->queue); return 0; diff --git a/include/uapi/rdma/mana-abi.h b/include/uapi/rdma/mana-abi.h index 5fcb31b..2c41cc3 100644 --- a/include/uapi/rdma/mana-abi.h +++ b/include/uapi/rdma/mana-abi.h @@ -16,8 +16,20 @@ #define MANA_IB_UVERBS_ABI_VERSION 1 +enum mana_ib_create_cq_flags { + MANA_IB_CREATE_RNIC_CQ = 1 << 0, +}; + struct mana_ib_create_cq { __aligned_u64 buf_addr; + __u16 flags; + __u16 reserved0; + __u32 reserved1; +}; + +struct mana_ib_create_cq_resp { + __u32 cqid; + __u32 reserved; }; struct mana_ib_create_qp {