From patchwork Wed Jan 31 10:55:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Konstantin Taranov X-Patchwork-Id: 13539193 Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 063D669E07; Wed, 31 Jan 2024 10:55:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=13.77.154.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706698559; cv=none; b=GxojwrV6PbypJhb3zmdRK4UuI1H9YqYrNe0eSx/Wx0s8JhnJ9pnRlPP2CqEI8Jg1q4f8vh3sh6IgZoRbXgZJN9Wq3tEDT75e3ORJ/par+Hq+Xjg63ouExhay1QTaQ9s2WLQVwx+GcsulkPLTarSoJx9Syuxb6bUejK8cASqpOxc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706698559; c=relaxed/simple; bh=4ZmPsc+rDgX7rX3N3RoK8DZaj+dzW3V5EQi2cFZw9c4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=bVrNBZfi1uQCvvq3ggLbGPkwx1IRUfzzmqvR/N9RV+7yp9stIyUdEFlQVD/jnvXi7nfyAgD+aZsGmBPPyCwl/IBvZroTYihxIMJbjS6vm956y3DO0d0rA7ZXBlRUkealJIwwWFCtvotfYhkP+vrSGlpeD1Eb/jPOmUWL1dlpvXQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com; spf=pass smtp.mailfrom=linux.microsoft.com; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b=gNepmmLp; arc=none smtp.client-ip=13.77.154.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b="gNepmmLp" Received: from linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net (linux.microsoft.com [13.77.154.182]) by linux.microsoft.com (Postfix) with ESMTPSA id AAE0720B2001; Wed, 31 Jan 2024 02:55:57 -0800 (PST) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com AAE0720B2001 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1706698557; bh=saDdL5sYNRP0RdrEfccKfGdzxJcVkUQ2VWnSyr2q/F4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gNepmmLp+bdtSeHkAgPlZ4++Kj2eo3GdZt371DiIVOMy8sQ1+PqN8T2ayv+z3lwCZ wJnazxG3Z5n5Ciy28iJO2vQV406v9lcRpGeKmWBtopB1rYZ0P5mxtNZzYz58Dj/SA2 uNVMXBcu1KV4fa2RVkJ5852mtYi9wuwMcJbWwTWA= From: Konstantin Taranov To: kotaranov@microsoft.com, sharmaajay@microsoft.com, longli@microsoft.com, jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH rdma-next v1 1/5] RDMA/mana_ib: Add EQ creation for rnic adapter Date: Wed, 31 Jan 2024 02:55:48 -0800 Message-Id: <1706698552-25383-2-git-send-email-kotaranov@linux.microsoft.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1706698552-25383-1-git-send-email-kotaranov@linux.microsoft.com> References: <1706698552-25383-1-git-send-email-kotaranov@linux.microsoft.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: This patch introduces functions for RNIC creation and creates one EQ for RNIC creation. Signed-off-by: Konstantin Taranov --- drivers/infiniband/hw/mana/device.c | 9 ++++-- drivers/infiniband/hw/mana/main.c | 53 ++++++++++++++++++++++++++++++++++++ drivers/infiniband/hw/mana/mana_ib.h | 5 ++++ 3 files changed, 64 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/hw/mana/device.c b/drivers/infiniband/hw/mana/device.c index 6fa902e..d8e8b10 100644 --- a/drivers/infiniband/hw/mana/device.c +++ b/drivers/infiniband/hw/mana/device.c @@ -92,15 +92,19 @@ static int mana_ib_probe(struct auxiliary_device *adev, goto deregister_device; } + mana_ib_gd_create_rnic_adapter(dev); + ret = ib_register_device(&dev->ib_dev, "mana_%d", mdev->gdma_context->dev); if (ret) - goto deregister_device; + goto destroy_rnic_adapter; dev_set_drvdata(&adev->dev, dev); return 0; +destroy_rnic_adapter: + mana_ib_gd_destroy_rnic_adapter(dev); deregister_device: mana_gd_deregister_device(dev->gdma_dev); free_ib_device: @@ -113,9 +117,8 @@ static void mana_ib_remove(struct auxiliary_device *adev) struct mana_ib_dev *dev = dev_get_drvdata(&adev->dev); ib_unregister_device(&dev->ib_dev); - + mana_ib_gd_destroy_rnic_adapter(dev); mana_gd_deregister_device(dev->gdma_dev); - ib_dealloc_device(&dev->ib_dev); } diff --git a/drivers/infiniband/hw/mana/main.c b/drivers/infiniband/hw/mana/main.c index 29dd243..c64d569 100644 --- a/drivers/infiniband/hw/mana/main.c +++ b/drivers/infiniband/hw/mana/main.c @@ -548,3 +548,56 @@ int mana_ib_gd_query_adapter_caps(struct mana_ib_dev *dev) return 0; } + +static int mana_ib_create_eqs(struct mana_ib_dev *mdev) +{ + struct gdma_context *gc = mdev_to_gc(mdev); + struct gdma_queue_spec spec = {}; + int err; + + spec.type = GDMA_EQ; + spec.monitor_avl_buf = false; + spec.queue_size = EQ_SIZE; + spec.eq.callback = NULL; + spec.eq.context = mdev; + spec.eq.log2_throttle_limit = LOG2_EQ_THROTTLE; + spec.eq.msix_index = 0; + + err = mana_gd_create_mana_eq(&gc->mana_ib, &spec, &mdev->fatal_err_eq); + if (err) + return err; + + return 0; +} + +static void mana_ib_destroy_eqs(struct mana_ib_dev *mdev) +{ + if (!mdev->fatal_err_eq) + return; + + mana_gd_destroy_queue(mdev_to_gc(mdev), mdev->fatal_err_eq); + mdev->fatal_err_eq = NULL; +} + +void mana_ib_gd_create_rnic_adapter(struct mana_ib_dev *mdev) +{ + int err; + + err = mana_ib_create_eqs(mdev); + if (err) { + ibdev_err(&mdev->ib_dev, "Failed to create EQs for RNIC err %d", err); + goto cleanup; + } + + return; + +cleanup: + ibdev_warn(&mdev->ib_dev, + "RNIC is not available. Only RAW QPs are supported"); + mana_ib_destroy_eqs(mdev); +} + +void mana_ib_gd_destroy_rnic_adapter(struct mana_ib_dev *mdev) +{ + mana_ib_destroy_eqs(mdev); +} diff --git a/drivers/infiniband/hw/mana/mana_ib.h b/drivers/infiniband/hw/mana/mana_ib.h index 6a03ae6..a4b94ee 100644 --- a/drivers/infiniband/hw/mana/mana_ib.h +++ b/drivers/infiniband/hw/mana/mana_ib.h @@ -48,6 +48,7 @@ struct mana_ib_adapter_caps { struct mana_ib_dev { struct ib_device ib_dev; struct gdma_dev *gdma_dev; + struct gdma_queue *fatal_err_eq; struct mana_ib_adapter_caps adapter_caps; }; @@ -228,4 +229,8 @@ int mana_ib_query_gid(struct ib_device *ibdev, u32 port, int index, void mana_ib_disassociate_ucontext(struct ib_ucontext *ibcontext); int mana_ib_gd_query_adapter_caps(struct mana_ib_dev *mdev); + +void mana_ib_gd_create_rnic_adapter(struct mana_ib_dev *mdev); + +void mana_ib_gd_destroy_rnic_adapter(struct mana_ib_dev *mdev); #endif From patchwork Wed Jan 31 10:55:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Konstantin Taranov X-Patchwork-Id: 13539195 Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 397536A030; Wed, 31 Jan 2024 10:55:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=13.77.154.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706698559; cv=none; b=rFpROYn1MjJezdru3mpaR95J/kb7Ip+y7uzeso18wBcUQTNCOq4n7ESQAvcq39xm4DrrS6Lkmb6UUZ9uK6f/91sFzC7kXysyH8YZoKBMa547cnml+xz2x/TrrpDDZspIWnJ+7dM6UdWxPnlbphHg/yylWHwszSBq4HThlwuHnBE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706698559; c=relaxed/simple; bh=GI6QY98SEMz0zqOuQidUbYonmgSgitJXQUyspqe4rNo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=fpYF+IoV17o2REa2Tn1PCvaJUZqmrS8QTvwh2/N9p8QvRlZyKNYW6xi3EvvFoqiixxImEm/d1qPi440k3k+arU+1pVXKLjGA0D16uj5XGpbskUivQ23ZhrJNzomYV8OYAo7KZ8GjWxjqi0YGN9VVrlIfL3ev2WvQu3M5ebdTSA0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com; spf=pass smtp.mailfrom=linux.microsoft.com; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b=S3JrxK1p; arc=none smtp.client-ip=13.77.154.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b="S3JrxK1p" Received: from linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net (linux.microsoft.com [13.77.154.182]) by linux.microsoft.com (Postfix) with ESMTPSA id C2D7F20B2002; Wed, 31 Jan 2024 02:55:57 -0800 (PST) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com C2D7F20B2002 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1706698557; bh=FQMJvC08Xx+rruS+xJDdv53mAl3GJPxfohVLYNulxEA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=S3JrxK1pO/X+3876uWpPrkKDWepe9Vt4MtQXVlmCX64d3WHz/zd7d+5oyQgIYhwdO umuzfIPe5kdz3Z+Pt0J0sWk06449UeLvCdOdjXosVxc53ZsLQ1ojYj283Y0kgxEJJ1 zCjsENNIHlMzHWM35uivZp6tyUSr22W+J5DixdAE= From: Konstantin Taranov To: kotaranov@microsoft.com, sharmaajay@microsoft.com, longli@microsoft.com, jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH rdma-next v1 2/5] RDMA/mana_ib: Create and destroy rnic adapter Date: Wed, 31 Jan 2024 02:55:49 -0800 Message-Id: <1706698552-25383-3-git-send-email-kotaranov@linux.microsoft.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1706698552-25383-1-git-send-email-kotaranov@linux.microsoft.com> References: <1706698552-25383-1-git-send-email-kotaranov@linux.microsoft.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: This patch adds RNIC creation and destruction. If creation of RNIC fails, we support only RAW QPs as they are served by ethernet driver. Signed-off-by: Konstantin Taranov --- drivers/infiniband/hw/mana/main.c | 31 +++++++++++++++++++++++++++++++ drivers/infiniband/hw/mana/mana_ib.h | 29 +++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+) diff --git a/drivers/infiniband/hw/mana/main.c b/drivers/infiniband/hw/mana/main.c index c64d569..33cd69e 100644 --- a/drivers/infiniband/hw/mana/main.c +++ b/drivers/infiniband/hw/mana/main.c @@ -581,14 +581,31 @@ static void mana_ib_destroy_eqs(struct mana_ib_dev *mdev) void mana_ib_gd_create_rnic_adapter(struct mana_ib_dev *mdev) { + struct mana_rnic_create_adapter_resp resp = {}; + struct mana_rnic_create_adapter_req req = {}; + struct gdma_context *gc = mdev_to_gc(mdev); int err; + mdev->adapter_handle = INVALID_MANA_HANDLE; + err = mana_ib_create_eqs(mdev); if (err) { ibdev_err(&mdev->ib_dev, "Failed to create EQs for RNIC err %d", err); goto cleanup; } + mana_gd_init_req_hdr(&req.hdr, MANA_IB_CREATE_ADAPTER, sizeof(req), sizeof(resp)); + req.hdr.req.msg_version = GDMA_MESSAGE_V2; + req.hdr.dev_id = gc->mana_ib.dev_id; + req.notify_eq_id = mdev->fatal_err_eq->id; + + err = mana_gd_send_request(gc, sizeof(req), &req, sizeof(resp), &resp); + if (err) { + ibdev_err(&mdev->ib_dev, "Failed to create RNIC adapter err %d", err); + goto cleanup; + } + mdev->adapter_handle = resp.adapter; + return; cleanup: @@ -599,5 +616,19 @@ void mana_ib_gd_create_rnic_adapter(struct mana_ib_dev *mdev) void mana_ib_gd_destroy_rnic_adapter(struct mana_ib_dev *mdev) { + struct mana_rnic_destroy_adapter_resp resp = {}; + struct mana_rnic_destroy_adapter_req req = {}; + struct gdma_context *gc; + + if (!rnic_is_enabled(mdev)) + return; + + gc = mdev_to_gc(mdev); + mana_gd_init_req_hdr(&req.hdr, MANA_IB_DESTROY_ADAPTER, sizeof(req), sizeof(resp)); + req.hdr.dev_id = gc->mana_ib.dev_id; + req.adapter = mdev->adapter_handle; + + mana_gd_send_request(gc, sizeof(req), &req, sizeof(resp), &resp); + mdev->adapter_handle = INVALID_MANA_HANDLE; mana_ib_destroy_eqs(mdev); } diff --git a/drivers/infiniband/hw/mana/mana_ib.h b/drivers/infiniband/hw/mana/mana_ib.h index a4b94ee..96454cf 100644 --- a/drivers/infiniband/hw/mana/mana_ib.h +++ b/drivers/infiniband/hw/mana/mana_ib.h @@ -48,6 +48,7 @@ struct mana_ib_adapter_caps { struct mana_ib_dev { struct ib_device ib_dev; struct gdma_dev *gdma_dev; + mana_handle_t adapter_handle; struct gdma_queue *fatal_err_eq; struct mana_ib_adapter_caps adapter_caps; }; @@ -115,6 +116,8 @@ struct mana_ib_rwq_ind_table { enum mana_ib_command_code { MANA_IB_GET_ADAPTER_CAP = 0x30001, + MANA_IB_CREATE_ADAPTER = 0x30002, + MANA_IB_DESTROY_ADAPTER = 0x30003, }; struct mana_ib_query_adapter_caps_req { @@ -143,6 +146,32 @@ struct mana_ib_query_adapter_caps_resp { u32 max_inline_data_size; }; /* HW Data */ +struct mana_rnic_create_adapter_req { + struct gdma_req_hdr hdr; + u32 notify_eq_id; + u32 reserved; + u64 feature_flags; +}; /*HW Data */ + +struct mana_rnic_create_adapter_resp { + struct gdma_resp_hdr hdr; + mana_handle_t adapter; +}; /* HW Data */ + +struct mana_rnic_destroy_adapter_req { + struct gdma_req_hdr hdr; + mana_handle_t adapter; +}; /*HW Data */ + +struct mana_rnic_destroy_adapter_resp { + struct gdma_resp_hdr hdr; +}; /* HW Data */ + +static inline bool rnic_is_enabled(struct mana_ib_dev *mdev) +{ + return mdev->adapter_handle != INVALID_MANA_HANDLE; +} + static inline struct gdma_context *mdev_to_gc(struct mana_ib_dev *mdev) { return mdev->gdma_dev->gdma_context; From patchwork Wed Jan 31 10:55:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Konstantin Taranov X-Patchwork-Id: 13539194 Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 417F86A353; Wed, 31 Jan 2024 10:55:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=13.77.154.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706698559; cv=none; b=bweNXF22iXTxdROeX9blPwcPVgoiA6lc8l8kiBNclhrlYTbQqYYpWaXeF2bavneiLHIaeR8JHxSRKarHABEYBlbpH4J2DpkFqBFxG+Hd0jI9BxsRobrYoVqe2bVoZuVfqVeyc2zbBP9qtORlozNqg2w/OepLN/XuyTTYXMG8vFs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706698559; c=relaxed/simple; bh=lwkOSAnGEnMRxgE2eYpnXuty91szYnqPYwpQeHF0pik=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=uQJlTUtoLWCf0BKHjR5cdNs7+s64dqAscWIP7paIraP22Fru7WUdWHrO9+yTvNfOfm017LDyiYtIXzX3XvkNgxtRq5KgrClCBgQck7I7j/mYvM/3PGTE0/ZGKkM2njuO1IhX2rPTk6vqB2x6LDcq9z54451lFaTfYxSVcA29Cmw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com; spf=pass smtp.mailfrom=linux.microsoft.com; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b=GB+vxnbL; arc=none smtp.client-ip=13.77.154.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b="GB+vxnbL" Received: from linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net (linux.microsoft.com [13.77.154.182]) by linux.microsoft.com (Postfix) with ESMTPSA id DB96C20B2003; Wed, 31 Jan 2024 02:55:57 -0800 (PST) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com DB96C20B2003 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1706698557; bh=5yXFaEp9vKBshRa+NgGc9Zh0uI+t9u7kQk3cCqXTndQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GB+vxnbLTjT5Cnt7unO+5vs7vkPwsXIQa+KctyE3Mac1SWpuauMpA/7NTrY3DtkI1 sUNbJdGmFonKb7p7PPqw+sfZxDcRQ6RPXJb3hzFqDbDhm8Kutug1KLwv4S48AZS0BU 1rCrPaeAerPUAqgKgZvX1bzGT5bltj7dNZ5gMp04= From: Konstantin Taranov To: kotaranov@microsoft.com, sharmaajay@microsoft.com, longli@microsoft.com, jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH rdma-next v1 3/5] RDMA/mana_ib: Implement port parameters Date: Wed, 31 Jan 2024 02:55:50 -0800 Message-Id: <1706698552-25383-4-git-send-email-kotaranov@linux.microsoft.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1706698552-25383-1-git-send-email-kotaranov@linux.microsoft.com> References: <1706698552-25383-1-git-send-email-kotaranov@linux.microsoft.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Implement port parameters for RNIC. Only port 1 is RoCEv2 capable. Signed-off-by: Konstantin Taranov --- drivers/infiniband/hw/mana/device.c | 2 ++ drivers/infiniband/hw/mana/main.c | 37 +++++++++++++++++++++++++++++++++++- drivers/infiniband/hw/mana/mana_ib.h | 4 ++++ 3 files changed, 42 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/hw/mana/device.c b/drivers/infiniband/hw/mana/device.c index d8e8b10..11b0410 100644 --- a/drivers/infiniband/hw/mana/device.c +++ b/drivers/infiniband/hw/mana/device.c @@ -29,12 +29,14 @@ .destroy_rwq_ind_table = mana_ib_destroy_rwq_ind_table, .destroy_wq = mana_ib_destroy_wq, .disassociate_ucontext = mana_ib_disassociate_ucontext, + .get_link_layer = mana_ib_get_link_layer, .get_port_immutable = mana_ib_get_port_immutable, .mmap = mana_ib_mmap, .modify_qp = mana_ib_modify_qp, .modify_wq = mana_ib_modify_wq, .query_device = mana_ib_query_device, .query_gid = mana_ib_query_gid, + .query_pkey = mana_ib_query_pkey, .query_port = mana_ib_query_port, .reg_user_mr = mana_ib_reg_user_mr, diff --git a/drivers/infiniband/hw/mana/main.c b/drivers/infiniband/hw/mana/main.c index 33cd69e..3e05a62 100644 --- a/drivers/infiniband/hw/mana/main.c +++ b/drivers/infiniband/hw/mana/main.c @@ -492,7 +492,42 @@ int mana_ib_query_device(struct ib_device *ibdev, struct ib_device_attr *props, int mana_ib_query_port(struct ib_device *ibdev, u32 port, struct ib_port_attr *props) { - /* This version doesn't return port properties */ + struct net_device *ndev = mana_ib_get_netdev(ibdev, port); + + if (!ndev) + return -EINVAL; + + memset(props, 0, sizeof(*props)); + props->max_mtu = IB_MTU_4096; + props->active_mtu = ib_mtu_int_to_enum(ndev->mtu); + + if (netif_carrier_ok(ndev) && netif_running(ndev)) { + props->state = IB_PORT_ACTIVE; + props->phys_state = IB_PORT_PHYS_STATE_LINK_UP; + } else { + props->state = IB_PORT_DOWN; + props->phys_state = IB_PORT_PHYS_STATE_DISABLED; + } + + props->active_width = IB_WIDTH_4X; + props->active_speed = IB_SPEED_EDR; + props->pkey_tbl_len = 1; + if (port == 1) + props->gid_tbl_len = 16; + + return 0; +} + +enum rdma_link_layer mana_ib_get_link_layer(struct ib_device *device, u32 port_num) +{ + return IB_LINK_LAYER_ETHERNET; +} + +int mana_ib_query_pkey(struct ib_device *ibdev, u32 port, u16 index, u16 *pkey) +{ + if (index != 0) + return -EINVAL; + *pkey = IB_DEFAULT_PKEY_FULL; return 0; } diff --git a/drivers/infiniband/hw/mana/mana_ib.h b/drivers/infiniband/hw/mana/mana_ib.h index 96454cf..196f3c8 100644 --- a/drivers/infiniband/hw/mana/mana_ib.h +++ b/drivers/infiniband/hw/mana/mana_ib.h @@ -262,4 +262,8 @@ int mana_ib_query_gid(struct ib_device *ibdev, u32 port, int index, void mana_ib_gd_create_rnic_adapter(struct mana_ib_dev *mdev); void mana_ib_gd_destroy_rnic_adapter(struct mana_ib_dev *mdev); + +int mana_ib_query_pkey(struct ib_device *ibdev, u32 port, u16 index, u16 *pkey); + +enum rdma_link_layer mana_ib_get_link_layer(struct ib_device *device, u32 port_num); #endif From patchwork Wed Jan 31 10:55:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Konstantin Taranov X-Patchwork-Id: 13539196 Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 418506D1D4; Wed, 31 Jan 2024 10:55:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=13.77.154.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706698561; cv=none; b=s+zox8VCtKW2//vuYXAnlpmOFNol80pQqZ0TSs/i0YugamRxNp3kpDtxFP+UPh0keYPK7X2JeFPmDLrIrNl8rsNHztNTxMGorbLsBw4knRBg8JafKQaEoXUnSYI4rLNvim/L/r10uEIBowlZ73e7ixLu7YTfEozPkK5jYn0N+ZQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706698561; c=relaxed/simple; bh=ykFVDqFPB4lY6V66losZpGAUIa9JbgeucCmBSnLGxnU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=SgRGU8J/2k2pEe71ZXyf9sZLmwJEiPrD00GpytQ2zb8X1q2mLOkCHD/kyJTRPurVy6DyfZT2ki6lYVRZpoP4StSht/j57nnUswd2PqMg5+lBaeZ88bgyaErJnoUTUgtItuZ9w5iWapJgw5VhW7Xge8A4LO/TfawpHHFhwFAwnvY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com; spf=pass smtp.mailfrom=linux.microsoft.com; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b=iII5BczB; arc=none smtp.client-ip=13.77.154.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b="iII5BczB" Received: from linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net (linux.microsoft.com [13.77.154.182]) by linux.microsoft.com (Postfix) with ESMTPSA id F3EF920B2004; Wed, 31 Jan 2024 02:55:57 -0800 (PST) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com F3EF920B2004 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1706698558; bh=snXzVvd1Es1P5EFCe7tEDBGYBTXVzvrZs/jkgBz6PiU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iII5BczBqDghQYdz16lH6P0WkkFIkrwADtobdOVYg48E8A/y6nDssi7bcU7GilcPt JxTK6JBmLHB3cG7sRC1ZYLfEW7JOvSESXczdslow/HlU+mI7SQSgHbeqgwMDPdzcF9 wvrF9w8cUCFoffgzm1B/GFZSkxabAneZfzWj8XnQ= From: Konstantin Taranov To: kotaranov@microsoft.com, sharmaajay@microsoft.com, longli@microsoft.com, jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH rdma-next v1 4/5] RDMA/mana_ib: Enable RoCE on port 1 Date: Wed, 31 Jan 2024 02:55:51 -0800 Message-Id: <1706698552-25383-5-git-send-email-kotaranov@linux.microsoft.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1706698552-25383-1-git-send-email-kotaranov@linux.microsoft.com> References: <1706698552-25383-1-git-send-email-kotaranov@linux.microsoft.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Set netdev and RoCEv2 flag to be used in GID population. mana_ib is auxiliary device, thus we need GIDs of the master netdev. Signed-off-by: Konstantin Taranov --- drivers/infiniband/hw/mana/device.c | 14 ++++++++++++++ drivers/infiniband/hw/mana/main.c | 16 ++++++++++++---- 2 files changed, 26 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/hw/mana/device.c b/drivers/infiniband/hw/mana/device.c index 11b0410..b9ff3fd 100644 --- a/drivers/infiniband/hw/mana/device.c +++ b/drivers/infiniband/hw/mana/device.c @@ -53,6 +53,7 @@ static int mana_ib_probe(struct auxiliary_device *adev, { struct mana_adev *madev = container_of(adev, struct mana_adev, adev); struct gdma_dev *mdev = madev->mdev; + struct net_device *upper_ndev; struct mana_context *mc; struct mana_ib_dev *dev; int ret; @@ -79,6 +80,19 @@ static int mana_ib_probe(struct auxiliary_device *adev, dev->ib_dev.num_comp_vectors = 1; dev->ib_dev.dev.parent = mdev->gdma_context->dev; + rcu_read_lock(); /* required to get upper dev */ + upper_ndev = netdev_master_upper_dev_get_rcu(mc->ports[0]); + rcu_read_unlock(); + if (!upper_ndev) { + ibdev_err(&dev->ib_dev, "Failed to get master netdev"); + goto free_ib_device; + } + ret = ib_device_set_netdev(&dev->ib_dev, upper_ndev, 1); + if (ret) { + ibdev_err(&dev->ib_dev, "Failed to set ib netdev, ret %d", ret); + goto free_ib_device; + } + ret = mana_gd_register_device(&mdev->gdma_context->mana_ib); if (ret) { ibdev_err(&dev->ib_dev, "Failed to register device, ret %d", diff --git a/drivers/infiniband/hw/mana/main.c b/drivers/infiniband/hw/mana/main.c index 3e05a62..645abf3 100644 --- a/drivers/infiniband/hw/mana/main.c +++ b/drivers/infiniband/hw/mana/main.c @@ -462,11 +462,19 @@ int mana_ib_mmap(struct ib_ucontext *ibcontext, struct vm_area_struct *vma) int mana_ib_get_port_immutable(struct ib_device *ibdev, u32 port_num, struct ib_port_immutable *immutable) { - /* - * This version only support RAW_PACKET - * other values need to be filled for other types - */ + struct mana_ib_dev *mdev = container_of(ibdev, struct mana_ib_dev, ib_dev); + struct ib_port_attr attr; + int err; + + err = ib_query_port(ibdev, port_num, &attr); + if (err) + return err; + + immutable->pkey_tbl_len = attr.pkey_tbl_len; + immutable->gid_tbl_len = attr.gid_tbl_len; immutable->core_cap_flags = RDMA_CORE_PORT_RAW_PACKET; + if (port_num == 1 && rnic_is_enabled(mdev)) + immutable->core_cap_flags |= RDMA_CORE_PORT_IBA_ROCE_UDP_ENCAP; return 0; } From patchwork Wed Jan 31 10:55:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Konstantin Taranov X-Patchwork-Id: 13539197 Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by smtp.subspace.kernel.org (Postfix) with ESMTP id AADA26EB6E; Wed, 31 Jan 2024 10:55:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=13.77.154.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706698562; cv=none; b=rqiMAmIWyl1rbYzFsiC5qw69D0kNVdLFegsSsr6hKblcGL3Ome+Slfoe25qR9QKQpkedXFsjUto45t195K8C/A3uXqJPxB5MKP7Of3lMbKd2LfeTGojgP9HKijSPh1cmUTodm9zDP/b+8HSI7XObST4HdiwEi9vs9r8b3OExf3g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706698562; c=relaxed/simple; bh=LHBIj/8uoDBRrS1dlZPeCz2qf0GZ0mc5vqMb8Dcyy6U=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=sSfJqTRaFhIlREW0cMpPSoS3WWTve1HRpfs/X/LuJhQgILvlJcGvq5E+W20HYvaLNJ4+iO2SktQrlmAnxWQozpv+tEGgHCqG9pmncRgiYPB9j12SsC+fa6uDcN8jAnrqt1fmXUQcVFFQbkAwHplkZwhvQtoW6wsNLOlVgRhKM9s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com; spf=pass smtp.mailfrom=linux.microsoft.com; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b=L7xBFSUO; arc=none smtp.client-ip=13.77.154.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.microsoft.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.microsoft.com header.i=@linux.microsoft.com header.b="L7xBFSUO" Received: from linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net (linux.microsoft.com [13.77.154.182]) by linux.microsoft.com (Postfix) with ESMTPSA id 1859020B2005; Wed, 31 Jan 2024 02:55:58 -0800 (PST) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 1859020B2005 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1706698558; bh=MsUo2v+//cFlf6nJMuZ1DMVlU3JLOjmKGXGXkSEimVc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=L7xBFSUON0y+jHp0RHicnx4GOcP3xCMFb6wyib2qcVkrCV4YXENV/EBgdnPq/AW1T t6fjax+zreYGHyrT/FTObwgcTI2XXzPsHWPiSMa5jFGT72oV3Ng9dnZiazoqVyWkX1 Rd8KPyE++IipAvl46yMwOVL88ODVAnYR0pVsveXY= From: Konstantin Taranov To: kotaranov@microsoft.com, sharmaajay@microsoft.com, longli@microsoft.com, jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH rdma-next v1 5/5] RDMA/mana_ib: Adding and deleting GIDs Date: Wed, 31 Jan 2024 02:55:52 -0800 Message-Id: <1706698552-25383-6-git-send-email-kotaranov@linux.microsoft.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1706698552-25383-1-git-send-email-kotaranov@linux.microsoft.com> References: <1706698552-25383-1-git-send-email-kotaranov@linux.microsoft.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Implement add_gid and del_gid for RNIC. We support ipv4 and ipv6 addresses. Signed-off-by: Konstantin Taranov --- drivers/infiniband/hw/mana/device.c | 2 ++ drivers/infiniband/hw/mana/main.c | 66 ++++++++++++++++++++++++++++++++++++ drivers/infiniband/hw/mana/mana_ib.h | 37 ++++++++++++++++++++ 3 files changed, 105 insertions(+) diff --git a/drivers/infiniband/hw/mana/device.c b/drivers/infiniband/hw/mana/device.c index b9ff3fd..9655307 100644 --- a/drivers/infiniband/hw/mana/device.c +++ b/drivers/infiniband/hw/mana/device.c @@ -15,6 +15,7 @@ .driver_id = RDMA_DRIVER_MANA, .uverbs_abi_ver = MANA_IB_UVERBS_ABI_VERSION, + .add_gid = mana_ib_gd_add_gid, .alloc_pd = mana_ib_alloc_pd, .alloc_ucontext = mana_ib_alloc_ucontext, .create_cq = mana_ib_create_cq, @@ -23,6 +24,7 @@ .create_wq = mana_ib_create_wq, .dealloc_pd = mana_ib_dealloc_pd, .dealloc_ucontext = mana_ib_dealloc_ucontext, + .del_gid = mana_ib_gd_del_gid, .dereg_mr = mana_ib_dereg_mr, .destroy_cq = mana_ib_destroy_cq, .destroy_qp = mana_ib_destroy_qp, diff --git a/drivers/infiniband/hw/mana/main.c b/drivers/infiniband/hw/mana/main.c index 645abf3..282c024 100644 --- a/drivers/infiniband/hw/mana/main.c +++ b/drivers/infiniband/hw/mana/main.c @@ -675,3 +675,69 @@ void mana_ib_gd_destroy_rnic_adapter(struct mana_ib_dev *mdev) mdev->adapter_handle = INVALID_MANA_HANDLE; mana_ib_destroy_eqs(mdev); } + +int mana_ib_gd_add_gid(const struct ib_gid_attr *attr, void **context) +{ + struct mana_ib_dev *mdev = container_of(attr->device, struct mana_ib_dev, ib_dev); + enum rdma_network_type ntype = rdma_gid_attr_network_type(attr); + struct mana_rnic_config_addr_resp resp = {}; + struct gdma_context *gc = mdev_to_gc(mdev); + struct mana_rnic_config_addr_req req = {}; + int err; + + if (!rnic_is_enabled(mdev)) + return -EINVAL; + + if (ntype != RDMA_NETWORK_IPV4 && ntype != RDMA_NETWORK_IPV6) { + ibdev_dbg(&mdev->ib_dev, "Unsupported rdma network type %d", ntype); + return -EINVAL; + } + + mana_gd_init_req_hdr(&req.hdr, MANA_IB_CONFIG_IP_ADDR, sizeof(req), sizeof(resp)); + req.hdr.dev_id = gc->mana_ib.dev_id; + req.adapter = mdev->adapter_handle; + req.op = ADDR_OP_ADD; + req.sgid_type = (ntype == RDMA_NETWORK_IPV6) ? SGID_TYPE_IPV6 : SGID_TYPE_IPV4; + copy_in_reverse(req.ip_addr, attr->gid.raw, sizeof(union ib_gid)); + + err = mana_gd_send_request(gc, sizeof(req), &req, sizeof(resp), &resp); + if (err) { + ibdev_err(&mdev->ib_dev, "Failed to config IP addr err %d\n", err); + return err; + } + + return 0; +} + +int mana_ib_gd_del_gid(const struct ib_gid_attr *attr, void **context) +{ + struct mana_ib_dev *mdev = container_of(attr->device, struct mana_ib_dev, ib_dev); + enum rdma_network_type ntype = rdma_gid_attr_network_type(attr); + struct mana_rnic_config_addr_resp resp = {}; + struct gdma_context *gc = mdev_to_gc(mdev); + struct mana_rnic_config_addr_req req = {}; + int err; + + if (!rnic_is_enabled(mdev)) + return -EINVAL; + + if (ntype != RDMA_NETWORK_IPV4 && ntype != RDMA_NETWORK_IPV6) { + ibdev_dbg(&mdev->ib_dev, "Unsupported rdma network type %d", ntype); + return -EINVAL; + } + + mana_gd_init_req_hdr(&req.hdr, MANA_IB_CONFIG_IP_ADDR, sizeof(req), sizeof(resp)); + req.hdr.dev_id = gc->mana_ib.dev_id; + req.adapter = mdev->adapter_handle; + req.op = ADDR_OP_REMOVE; + req.sgid_type = (ntype == RDMA_NETWORK_IPV6) ? SGID_TYPE_IPV6 : SGID_TYPE_IPV4; + copy_in_reverse(req.ip_addr, attr->gid.raw, sizeof(union ib_gid)); + + err = mana_gd_send_request(gc, sizeof(req), &req, sizeof(resp), &resp); + if (err) { + ibdev_err(&mdev->ib_dev, "Failed to config IP addr err %d\n", err); + return err; + } + + return 0; +} diff --git a/drivers/infiniband/hw/mana/mana_ib.h b/drivers/infiniband/hw/mana/mana_ib.h index 196f3c8..2a3e3b0 100644 --- a/drivers/infiniband/hw/mana/mana_ib.h +++ b/drivers/infiniband/hw/mana/mana_ib.h @@ -118,6 +118,7 @@ enum mana_ib_command_code { MANA_IB_GET_ADAPTER_CAP = 0x30001, MANA_IB_CREATE_ADAPTER = 0x30002, MANA_IB_DESTROY_ADAPTER = 0x30003, + MANA_IB_CONFIG_IP_ADDR = 0x30004, }; struct mana_ib_query_adapter_caps_req { @@ -167,6 +168,30 @@ struct mana_rnic_destroy_adapter_resp { struct gdma_resp_hdr hdr; }; /* HW Data */ +enum mana_ib_addr_op { + ADDR_OP_ADD = 1, + ADDR_OP_REMOVE, +}; + +enum sgid_entry_type { + SGID_TYPE_INVALID = 0, + SGID_TYPE_IPV4 = 1, + SGID_TYPE_IPV6 = 2, + SGID_TYPE_HYBRID = 3 +}; + +struct mana_rnic_config_addr_req { + struct gdma_req_hdr hdr; + mana_handle_t adapter; + enum mana_ib_addr_op op; + enum sgid_entry_type sgid_type; + u8 ip_addr[16]; +}; /* HW Data */ + +struct mana_rnic_config_addr_resp { + struct gdma_resp_hdr hdr; +}; /* HW Data */ + static inline bool rnic_is_enabled(struct mana_ib_dev *mdev) { return mdev->adapter_handle != INVALID_MANA_HANDLE; @@ -188,6 +213,14 @@ static inline struct net_device *mana_ib_get_netdev(struct ib_device *ibdev, u32 return mc->ports[port - 1]; } +static inline void copy_in_reverse(u8 *dst, const u8 *src, u32 size) +{ + u32 i; + + for (i = 0; i < size; i++) + dst[size - 1 - i] = src[i]; +} + int mana_ib_install_cq_cb(struct mana_ib_dev *mdev, struct mana_ib_cq *cq); int mana_ib_gd_create_dma_region(struct mana_ib_dev *dev, struct ib_umem *umem, @@ -266,4 +299,8 @@ int mana_ib_query_gid(struct ib_device *ibdev, u32 port, int index, int mana_ib_query_pkey(struct ib_device *ibdev, u32 port, u16 index, u16 *pkey); enum rdma_link_layer mana_ib_get_link_layer(struct ib_device *device, u32 port_num); + +int mana_ib_gd_add_gid(const struct ib_gid_attr *attr, void **context); + +int mana_ib_gd_del_gid(const struct ib_gid_attr *attr, void **context); #endif