From patchwork Mon May 15 09:06:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13241031 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 748C7C7EE22 for ; Mon, 15 May 2023 09:23:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237651AbjEOJXQ (ORCPT ); Mon, 15 May 2023 05:23:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57242 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239936AbjEOJWU (ORCPT ); Mon, 15 May 2023 05:22:20 -0400 Received: from mail-pf1-x42a.google.com (mail-pf1-x42a.google.com [IPv6:2607:f8b0:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA405E5D for ; Mon, 15 May 2023 02:18:44 -0700 (PDT) Received: by mail-pf1-x42a.google.com with SMTP id d2e1a72fcca58-643465067d1so9177582b3a.0 for ; Mon, 15 May 2023 02:18:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1684142324; x=1686734324; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=zYbXOJKl1Z7gXY+QlsDzkZRz5qWf9Pz0U+G5Dc4See8=; b=fFjnibgMjiUE/AO4vJ9FMtmnLMmJ3K1ATG8gxItRo+6P65dJd06+g/pJNJSMOtGcPo CgzWGsrbgKjbVBcjOQ5b42GFUkWzZZa/h9jInpl6E8F1Wta7u9zp9poDeCSxU23iEvv9 OTZVjTDk8tYQrcil9eD+IpLHduG/EpZOk/VZ4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684142324; x=1686734324; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zYbXOJKl1Z7gXY+QlsDzkZRz5qWf9Pz0U+G5Dc4See8=; b=DsIFhSYFb3lWVVS47R/oW7XjMGs4Xle3PPsW4PkthjxparV4imkFaDVV+TnBylcKNU Q2y/RbvMPOARPbso7J2Q6AC+0e1pn08SJVglY9hu6Ci12vJpah2wVtAzfJsRdLXQQsz1 QulfxrI3tnSyug9szFplJoi1QE9okTNJr59+NBz2LlAnXXHRGVPdEj8VyvvH3f0lxb98 P+DNIwTpKYCggkW1H8FhPJswh60xDdFssHm1Sd3Vi9sXubbfUzt7HxTmtzr/+423aacx 0spgWqibsOWdBESjcvEWVRy2rZp0xJao1cA4EVz9MCVjvao8Tl3YEPwPBfDQupR0GuMN heSw== X-Gm-Message-State: AC+VfDyYoLPPGL2IXOkh+c24YWdjSNRVsxyNW8eS2xn39UcrqogmLUnd Fy5zdKeUBhpyzMvEzBAiwoLvfg== X-Google-Smtp-Source: ACHHUZ5cKUlMEBcbuqi2gO62VeqhROOh992tsUzJmiKGSFdGiT0xM7v3Gk/Rs+YPdIEXQqmATklehQ== X-Received: by 2002:a05:6a20:3d06:b0:100:900f:74a5 with SMTP id y6-20020a056a203d0600b00100900f74a5mr32132332pzi.56.1684142324109; Mon, 15 May 2023 02:18:44 -0700 (PDT) Received: from dhcp-10-192-206-197.iig.avagotech.net.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id q7-20020a654947000000b00520f316ebe3sm10402169pgs.62.2023.05.15.02.18.41 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 15 May 2023 02:18:43 -0700 (PDT) From: Selvin Xavier To: jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Selvin Xavier Subject: [PATCH v3 for-next 6/6] RDMA/bnxt_re: Enable low latency push Date: Mon, 15 May 2023 02:06:50 -0700 Message-Id: <1684141610-17588-7-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1684141610-17588-1-git-send-email-selvin.xavier@broadcom.com> References: <1684141610-17588-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Introduce driver specific uapi functionalites. Added a alloc_page functionality for user library to allocate specific pages. Currently added support for allocating write combine pages for push functinality. This interface shall be extended for other page allocations. Allocate a WC page using the uapi hook for enabling the low latency push in Gen P5 adapters for small packets. This is supported only for the user space QPs. Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 3 + drivers/infiniband/hw/bnxt_re/ib_verbs.c | 149 ++++++++++++++++++++++++++++++ drivers/infiniband/hw/bnxt_re/ib_verbs.h | 4 + drivers/infiniband/hw/bnxt_re/main.c | 17 +++- drivers/infiniband/hw/bnxt_re/qplib_res.c | 3 + drivers/infiniband/hw/bnxt_re/qplib_res.h | 3 +- include/uapi/rdma/bnxt_re-abi.h | 32 +++++++ 7 files changed, 209 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h index 5a2baf4..d436327 100644 --- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h +++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h @@ -39,6 +39,7 @@ #ifndef __BNXT_RE_H__ #define __BNXT_RE_H__ +#include #include "hw_counters.h" #define ROCE_DRV_MODULE_NAME "bnxt_re" @@ -187,4 +188,6 @@ static inline struct device *rdev_to_dev(struct bnxt_re_dev *rdev) return &rdev->ibdev.dev; return NULL; } + +extern const struct uapi_definition bnxt_re_uapi_defs[]; #endif diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index bc85433..60169fc 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -61,6 +61,15 @@ #include "bnxt_re.h" #include "ib_verbs.h" + +#include +#include + +#include + +#define UVERBS_MODULE_NAME bnxt_re +#include + #include static int __from_ib_access_flags(int iflags) @@ -546,6 +555,7 @@ bnxt_re_mmap_entry_insert(struct bnxt_re_ucontext *uctx, u64 mem_offset, entry->mem_offset = mem_offset; entry->mmap_flag = mmap_flag; + entry->uctx = uctx; ret = rdma_user_mmap_entry_insert(&uctx->ib_uctx, &entry->rdma_entry, PAGE_SIZE); @@ -599,6 +609,7 @@ int bnxt_re_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata) } if (udata) { + struct bnxt_qplib_chip_ctx *cctx = rdev->chip_ctx; struct bnxt_re_pd_resp resp = {}; if (!ucntx->dpi.dbr) { @@ -611,6 +622,9 @@ int bnxt_re_alloc_pd(struct ib_pd *ibpd, struct ib_udata *udata) rc = -ENOMEM; goto dbfail; } + /* Set the comp_mask to indicate users to enable WC DPI */ + if (cctx->modes.db_push) + resp.comp_mask = BNXT_RE_COMP_MASK_PD_WC_DPI_SUP; } resp.pdid = pd->qplib_pd.id; @@ -4036,6 +4050,9 @@ int bnxt_re_alloc_ucontext(struct ib_ucontext *ctx, struct ib_udata *udata) resp.comp_mask |= BNXT_RE_UCNTX_CMASK_HAVE_MODE; resp.mode = rdev->chip_ctx->modes.wqe_mode; + if (rdev->chip_ctx->modes.db_push) + resp.comp_mask |= BNXT_RE_UCNTX_CMASK_WC_DPI_ENABLED; + entry = bnxt_re_mmap_entry_insert(uctx, 0, BNXT_RE_MMAP_SH_PAGE, NULL); if (!entry) { rc = -ENOMEM; @@ -4099,6 +4116,12 @@ int bnxt_re_mmap(struct ib_ucontext *ib_uctx, struct vm_area_struct *vma) rdma_entry); switch (bnxt_entry->mmap_flag) { + case BNXT_RE_MMAP_WC_DB: + pfn = bnxt_entry->mem_offset >> PAGE_SHIFT; + ret = rdma_user_mmap_io(ib_uctx, vma, pfn, PAGE_SIZE, + pgprot_writecombine(vma->vm_page_prot), + rdma_entry); + break; case BNXT_RE_MMAP_UC_DB: pfn = bnxt_entry->mem_offset >> PAGE_SHIFT; ret = rdma_user_mmap_io(ib_uctx, vma, pfn, PAGE_SIZE, @@ -4126,3 +4149,129 @@ void bnxt_re_mmap_free(struct rdma_user_mmap_entry *rdma_entry) kfree(bnxt_entry); } + +static int UVERBS_HANDLER(BNXT_RE_METHOD_ALLOC_PAGE)(struct uverbs_attr_bundle *attrs) +{ + struct ib_uobject *uobj = uverbs_attr_get_uobject(attrs, BNXT_RE_ALLOC_PAGE_HANDLE); + enum bnxt_re_alloc_page_type alloc_type; + struct bnxt_re_user_mmap_entry *entry; + enum bnxt_re_mmap_flag mmap_flag; + struct bnxt_qplib_chip_ctx *cctx; + struct bnxt_re_ucontext *uctx; + struct bnxt_re_dev *rdev; + u64 mmap_offset; + u32 length; + u32 dpi; + u64 dbr; + int err; + + uctx = container_of(ib_uverbs_get_ucontext(attrs), struct bnxt_re_ucontext, ib_uctx); + if (IS_ERR(uctx)) + return PTR_ERR(uctx); + + err = uverbs_get_const(&alloc_type, attrs, BNXT_RE_ALLOC_PAGE_TYPE); + if (err) + return err; + + rdev = uctx->rdev; + cctx = rdev->chip_ctx; + + switch (alloc_type) { + case BNXT_RE_ALLOC_WC_PAGE: + if (cctx->modes.db_push) { + if (bnxt_qplib_alloc_dpi(&rdev->qplib_res, &uctx->wcdpi, + uctx, BNXT_QPLIB_DPI_TYPE_WC)) + return -ENOMEM; + } + length = PAGE_SIZE; + dpi = uctx->wcdpi.dpi; + dbr = (u64)uctx->wcdpi.umdbr; + mmap_flag = BNXT_RE_MMAP_WC_DB; + + break; + + default: + return -EOPNOTSUPP; + } + + entry = bnxt_re_mmap_entry_insert(uctx, dbr, mmap_flag, &mmap_offset); + if (IS_ERR(entry)) + return PTR_ERR(entry); + + uobj->object = entry; + uverbs_finalize_uobj_create(attrs, BNXT_RE_ALLOC_PAGE_HANDLE); + err = uverbs_copy_to(attrs, BNXT_RE_ALLOC_PAGE_MMAP_OFFSET, + &mmap_offset, sizeof(mmap_offset)); + if (err) + return err; + + err = uverbs_copy_to(attrs, BNXT_RE_ALLOC_PAGE_MMAP_LENGTH, + &length, sizeof(length)); + if (err) + return err; + + err = uverbs_copy_to(attrs, BNXT_RE_ALLOC_PAGE_DPI, + &dpi, sizeof(length)); + if (err) + return err; + + return 0; +} + +static int alloc_page_obj_cleanup(struct ib_uobject *uobject, + enum rdma_remove_reason why, + struct uverbs_attr_bundle *attrs) +{ + struct bnxt_re_user_mmap_entry *entry = uobject->object; + struct bnxt_re_ucontext *uctx = entry->uctx; + + switch (entry->mmap_flag) { + case BNXT_RE_MMAP_WC_DB: + if (uctx && uctx->wcdpi.dbr) { + struct bnxt_re_dev *rdev = uctx->rdev; + + bnxt_qplib_dealloc_dpi(&rdev->qplib_res, &uctx->wcdpi); + uctx->wcdpi.dbr = NULL; + } + break; + default: + goto exit; + } + rdma_user_mmap_entry_remove(&entry->rdma_entry); +exit: + return 0; +} + +DECLARE_UVERBS_NAMED_METHOD(BNXT_RE_METHOD_ALLOC_PAGE, + UVERBS_ATTR_IDR(BNXT_RE_ALLOC_PAGE_HANDLE, + BNXT_RE_OBJECT_ALLOC_PAGE, + UVERBS_ACCESS_NEW, + UA_MANDATORY), + UVERBS_ATTR_CONST_IN(BNXT_RE_ALLOC_PAGE_TYPE, + enum bnxt_re_alloc_page_type, + UA_MANDATORY), + UVERBS_ATTR_PTR_OUT(BNXT_RE_ALLOC_PAGE_MMAP_OFFSET, + UVERBS_ATTR_TYPE(u64), + UA_MANDATORY), + UVERBS_ATTR_PTR_OUT(BNXT_RE_ALLOC_PAGE_MMAP_LENGTH, + UVERBS_ATTR_TYPE(u32), + UA_MANDATORY), + UVERBS_ATTR_PTR_OUT(BNXT_RE_ALLOC_PAGE_DPI, + UVERBS_ATTR_TYPE(u32), + UA_MANDATORY)); + +DECLARE_UVERBS_NAMED_METHOD_DESTROY(BNXT_RE_METHOD_DESTROY_PAGE, + UVERBS_ATTR_IDR(BNXT_RE_DESTROY_PAGE_HANDLE, + BNXT_RE_OBJECT_ALLOC_PAGE, + UVERBS_ACCESS_DESTROY, + UA_MANDATORY)); + +DECLARE_UVERBS_NAMED_OBJECT(BNXT_RE_OBJECT_ALLOC_PAGE, + UVERBS_TYPE_ALLOC_IDR(alloc_page_obj_cleanup), + &UVERBS_METHOD(BNXT_RE_METHOD_ALLOC_PAGE), + &UVERBS_METHOD(BNXT_RE_METHOD_DESTROY_PAGE)); + +const struct uapi_definition bnxt_re_uapi_defs[] = { + UAPI_DEF_CHAIN_OBJ_TREE_NAMED(BNXT_RE_OBJECT_ALLOC_PAGE), + {} +}; diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.h b/drivers/infiniband/hw/bnxt_re/ib_verbs.h index dcd31ae..32d9e9d 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.h +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.h @@ -61,6 +61,7 @@ struct bnxt_re_pd { struct bnxt_qplib_pd qplib_pd; struct bnxt_re_fence_data fence; struct rdma_user_mmap_entry *pd_db_mmap; + struct rdma_user_mmap_entry *pd_wcdb_mmap; }; struct bnxt_re_ah { @@ -135,6 +136,7 @@ struct bnxt_re_ucontext { struct ib_ucontext ib_uctx; struct bnxt_re_dev *rdev; struct bnxt_qplib_dpi dpi; + struct bnxt_qplib_dpi wcdpi; void *shpg; spinlock_t sh_lock; /* protect shpg */ struct rdma_user_mmap_entry *shpage_mmap; @@ -143,10 +145,12 @@ struct bnxt_re_ucontext { enum bnxt_re_mmap_flag { BNXT_RE_MMAP_SH_PAGE, BNXT_RE_MMAP_UC_DB, + BNXT_RE_MMAP_WC_DB, }; struct bnxt_re_user_mmap_entry { struct rdma_user_mmap_entry rdma_entry; + struct bnxt_re_ucontext *uctx; u64 mem_offset; u8 mmap_flag; }; diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 69d4d8a..3744cda 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -66,6 +66,7 @@ #include #include "bnxt.h" #include "hw_counters.h" +#include "ib_verbs.h" static char version[] = BNXT_RE_DESC "\n"; @@ -117,6 +118,10 @@ static void bnxt_re_set_db_offset(struct bnxt_re_dev *rdev) * in such cases and DB-push will be disabled. */ barlen = pci_resource_len(res->pdev, RCFW_DBR_PCI_BAR_REGION); + if (cctx->modes.db_push && l2db_len && en_dev->l2_db_size != barlen) { + res->dpi_tbl.wcreg.offset = en_dev->l2_db_size; + dev_info(rdev_to_dev(rdev), "Low latency framework is enabled\n"); + } } static void bnxt_re_set_drv_mode(struct bnxt_re_dev *rdev, u8 mode) @@ -411,6 +416,7 @@ int bnxt_re_hwrm_qcaps(struct bnxt_re_dev *rdev) struct hwrm_func_qcaps_input req = {0}; struct bnxt_qplib_chip_ctx *cctx; struct bnxt_fw_msg fw_msg; + int rc; cctx = rdev->chip_ctx; memset(&fw_msg, 0, sizeof(fw_msg)); @@ -419,7 +425,13 @@ int bnxt_re_hwrm_qcaps(struct bnxt_re_dev *rdev) req.fid = cpu_to_le16(0xffff); bnxt_re_fill_fw_msg(&fw_msg, (void *)&req, sizeof(req), (void *)&resp, sizeof(resp), DFLT_HWRM_CMD_TIMEOUT); - return bnxt_send_msg(en_dev, &fw_msg); + + rc = bnxt_send_msg(en_dev, &fw_msg); + if (rc) + return rc; + cctx->modes.db_push = le32_to_cpu(resp.flags) & FUNC_QCAPS_RESP_FLAGS_WCB_PUSH_MODE; + + return 0; } static int bnxt_re_net_ring_free(struct bnxt_re_dev *rdev, @@ -673,6 +685,9 @@ static int bnxt_re_register_ib(struct bnxt_re_dev *rdev) ibdev->dev.parent = &rdev->en_dev->pdev->dev; ibdev->local_dma_lkey = BNXT_QPLIB_RSVD_LKEY; + if (IS_ENABLED(CONFIG_INFINIBAND_USER_ACCESS)) + ibdev->driver_def = bnxt_re_uapi_defs; + ib_set_device_ops(ibdev, &bnxt_re_dev_ops); ret = ib_device_set_netdev(&rdev->ibdev, rdev->netdev, 1); if (ret) diff --git a/drivers/infiniband/hw/bnxt_re/qplib_res.c b/drivers/infiniband/hw/bnxt_re/qplib_res.c index e1cbe59..174db83 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_res.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_res.c @@ -740,6 +740,9 @@ int bnxt_qplib_alloc_dpi(struct bnxt_qplib_res *res, dpi->dbr = dpit->priv_db; dpi->dpi = dpi->bit; break; + case BNXT_QPLIB_DPI_TYPE_WC: + dpi->dbr = ioremap_wc(umaddr, PAGE_SIZE); + break; default: dpi->dbr = ioremap(umaddr, PAGE_SIZE); break; diff --git a/drivers/infiniband/hw/bnxt_re/qplib_res.h b/drivers/infiniband/hw/bnxt_re/qplib_res.h index 95b1d6c..dc39f67 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_res.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_res.h @@ -47,7 +47,7 @@ extern const struct bnxt_qplib_gid bnxt_qplib_gid_zero; struct bnxt_qplib_drv_modes { u8 wqe_mode; - /* Other modes to follow here */ + bool db_push; }; struct bnxt_qplib_chip_ctx { @@ -193,6 +193,7 @@ struct bnxt_qplib_sgid_tbl { enum { BNXT_QPLIB_DPI_TYPE_KERNEL = 0, BNXT_QPLIB_DPI_TYPE_UC = 1, + BNXT_QPLIB_DPI_TYPE_WC = 2 }; struct bnxt_qplib_dpi { diff --git a/include/uapi/rdma/bnxt_re-abi.h b/include/uapi/rdma/bnxt_re-abi.h index c4e9077..3efc068 100644 --- a/include/uapi/rdma/bnxt_re-abi.h +++ b/include/uapi/rdma/bnxt_re-abi.h @@ -41,6 +41,7 @@ #define __BNXT_RE_UVERBS_ABI_H__ #include +#include #define BNXT_RE_ABI_VERSION 1 @@ -51,6 +52,7 @@ enum { BNXT_RE_UCNTX_CMASK_HAVE_CCTX = 0x1ULL, BNXT_RE_UCNTX_CMASK_HAVE_MODE = 0x02ULL, + BNXT_RE_UCNTX_CMASK_WC_DPI_ENABLED = 0x04ULL, }; enum bnxt_re_wqe_mode { @@ -78,10 +80,15 @@ struct bnxt_re_uctx_resp { * not 8 byted aligned. To avoid undesired padding in various cases we have to * set this struct to packed. */ +enum { + BNXT_RE_COMP_MASK_PD_WC_DPI_SUP = 0x01, +}; + struct bnxt_re_pd_resp { __u32 pdid; __u32 dpi; __u64 dbr; + __u64 comp_mask; } __attribute__((packed, aligned(4))); struct bnxt_re_cq_req { @@ -127,4 +134,29 @@ enum bnxt_re_shpg_offt { BNXT_RE_END_RESV_OFFT = 0xFF0 }; +enum bnxt_re_objects { + BNXT_RE_OBJECT_ALLOC_PAGE = (1U << UVERBS_ID_NS_SHIFT), +}; + +enum bnxt_re_alloc_page_type { + BNXT_RE_ALLOC_WC_PAGE = 0, +}; + +enum bnxt_re_var_alloc_page_attrs { + BNXT_RE_ALLOC_PAGE_HANDLE = (1U << UVERBS_ID_NS_SHIFT), + BNXT_RE_ALLOC_PAGE_TYPE, + BNXT_RE_ALLOC_PAGE_DPI, + BNXT_RE_ALLOC_PAGE_MMAP_OFFSET, + BNXT_RE_ALLOC_PAGE_MMAP_LENGTH, +}; + +enum bnxt_re_alloc_page_attrs { + BNXT_RE_DESTROY_PAGE_HANDLE = (1U << UVERBS_ID_NS_SHIFT), +}; + +enum bnxt_re_alloc_page_methods { + BNXT_RE_METHOD_ALLOC_PAGE = (1U << UVERBS_ID_NS_SHIFT), + BNXT_RE_METHOD_DESTROY_PAGE, +}; + #endif /* __BNXT_RE_UVERBS_ABI_H__*/