From patchwork Thu Aug 29 15:34:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13783385 Received: from mail-ot1-f43.google.com (mail-ot1-f43.google.com [209.85.210.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 327241B1D7F for ; Thu, 29 Aug 2024 15:55:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724946907; cv=none; b=UVksboxckY1tDh/vCv92y9XAl8MM/88EDk3c12OQ3ax/HxSftED1Pupm8qRnDtNm1n8sIjBorA9qW9NtY1cu8mtERLw7RPxGuhR7dm19S9xXqwBIDzgtTUL3OhTuJqEoytFc5t9ir45eb5kXJTJQPIP3ev/An0oRP6n2VuppIJ0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724946907; c=relaxed/simple; bh=yQ81tQkbalyIglTbg8R6nyKp/acgwIJeHlAqVCfiubw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=SFP0kdr0soIdbEZunSGwqT0s5e4RH57tftXD1pt8BUTj7uz2ZZ1SJj+oqMXhgl9h67AjftjTU2UgS89sKOsxRBMssM+IIdMJ01tDjKQ1Zr3LQzSE6MAD74fi2vcf01XDQH2byG5Ld94UjczGPg0Grakr/CS5Kbfr+hwTvc+0RVQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=RzwlzFhy; arc=none smtp.client-ip=209.85.210.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="RzwlzFhy" Received: by mail-ot1-f43.google.com with SMTP id 46e09a7af769-70cb1b959a6so493782a34.3 for ; Thu, 29 Aug 2024 08:55:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1724946905; x=1725551705; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=10OHoNG68SXU9rV3TemHjbm/ccgfAAiHoAdx+wtfzAI=; b=RzwlzFhymWb6C0jvzA2bxwCapDIkOTk7Ka4zaEdPDbv4EaCDmKpGst7wLnjCLmh86z wAwIUjqPAAPHFfJFLuBXZ5KrBcAMccFqvJi6HsOjVCM+VHz2hHbgs1S/awhqQFxmmXRm JJ6B4Qe476EpNScD5+kVtB++a8G52h0M6PtCA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724946905; x=1725551705; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=10OHoNG68SXU9rV3TemHjbm/ccgfAAiHoAdx+wtfzAI=; b=ev+khW0CjGlTQ9MZAjSvGsMDjI+UJcnQNmtCzwTX4p0+fFD+PWSyqNH31kE++haFuG N6PIXKpaaPIVldaIJH+7zhCqAbctjnZoLXMi6rN2EUGP6oEYJaYLiGNOipGdN6F5AV7O NslkmXLI92pwy8FiqqhAajAH+k/J7rqiNZehLeBUiNF7neltPBZrfibHxltvJFK5yeJV QsNw7eMWVn3zgRd7WNXZo2zZIu134sWZ0Sbird3H3RQrfcWidQtJxNMJ/zRlAyqNBQi5 +sPfucfcQ7AytV/+vXHpqmcoZrfnu+XQnOneg9Q6D3yHT/3x4QPY1wZxknFMmZCjiB/n 6tZA== X-Gm-Message-State: AOJu0Yy2fIOduemt/LII/cY8r33QdmHvaUPdVILgFNIadqbWhL+/QBn6 4CwSCSEA5VpRFej8L5dI+rv2vRg8SoQ97AdHxCAl8O0KMMuW2dip4D3+Xpgkzw== X-Google-Smtp-Source: AGHT+IF8b/XFLPqgiPd5r5OG5+4LBLbCbkb6+EzQMWAAaH6OPgThPrOjYIt+lJJ39dD7t28o/fq0TA== X-Received: by 2002:a05:6358:910d:b0:1af:1b40:5357 with SMTP id e5c5f4694b2df-1b603cb5ab7mr451894855d.24.1724946905189; Thu, 29 Aug 2024 08:55:05 -0700 (PDT) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7d22e9be627sm1396735a12.57.2024.08.29.08.55.02 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 29 Aug 2024 08:55:04 -0700 (PDT) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, chandramohan.akula@broadcom.com, Hongguang Gao , Selvin Xavier Subject: [PATCH for-next v2 1/3] RDMA/bnxt_re: Get the toggle bits from SRQ events Date: Thu, 29 Aug 2024 08:34:03 -0700 Message-Id: <1724945645-14989-2-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1724945645-14989-1-git-send-email-selvin.xavier@broadcom.com> References: <1724945645-14989-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Hongguang Gao SRQ arming requires the toggle bits received from hardware. Get the toggle bits from SRQ notification for the gen p7 adapters. This value will be zero for the older adapters. Signed-off-by: Hongguang Gao Signed-off-by: Chandramohan Akula Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/ib_verbs.h | 1 + drivers/infiniband/hw/bnxt_re/qplib_fp.c | 11 +++++++++++ drivers/infiniband/hw/bnxt_re/qplib_fp.h | 1 + 3 files changed, 13 insertions(+) diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.h b/drivers/infiniband/hw/bnxt_re/ib_verbs.h index 3ddeda3..4e113b9 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.h +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.h @@ -77,6 +77,7 @@ struct bnxt_re_srq { struct bnxt_qplib_srq qplib_srq; struct ib_umem *umem; spinlock_t lock; /* protect srq */ + void *uctx_srq_page; }; struct bnxt_re_qp { diff --git a/drivers/infiniband/hw/bnxt_re/qplib_fp.c b/drivers/infiniband/hw/bnxt_re/qplib_fp.c index 49e4a4a..19bb453 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_fp.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_fp.c @@ -54,6 +54,10 @@ #include "qplib_rcfw.h" #include "qplib_sp.h" #include "qplib_fp.h" +#include +#include "bnxt_ulp.h" +#include "bnxt_re.h" +#include "ib_verbs.h" static void __clean_cq(struct bnxt_qplib_cq *cq, u64 qp); @@ -347,6 +351,7 @@ static void bnxt_qplib_service_nq(struct tasklet_struct *t) case NQ_BASE_TYPE_SRQ_EVENT: { struct bnxt_qplib_srq *srq; + struct bnxt_re_srq *srq_p; struct nq_srq_event *nqsrqe = (struct nq_srq_event *)nqe; @@ -354,6 +359,12 @@ static void bnxt_qplib_service_nq(struct tasklet_struct *t) q_handle |= (u64)le32_to_cpu(nqsrqe->srq_handle_high) << 32; srq = (struct bnxt_qplib_srq *)q_handle; + srq->toggle = (le16_to_cpu(nqe->info10_type) & NQ_CN_TOGGLE_MASK) + >> NQ_CN_TOGGLE_SFT; + srq->dbinfo.toggle = srq->toggle; + srq_p = container_of(srq, struct bnxt_re_srq, qplib_srq); + if (srq_p->uctx_srq_page) + *((u32 *)srq_p->uctx_srq_page) = srq->toggle; bnxt_qplib_armen_db(&srq->dbinfo, DBC_DBC_TYPE_SRQ_ARMENA); if (nq->srqn_handler(nq, diff --git a/drivers/infiniband/hw/bnxt_re/qplib_fp.h b/drivers/infiniband/hw/bnxt_re/qplib_fp.h index 56538b9..e714caa 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_fp.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_fp.h @@ -105,6 +105,7 @@ struct bnxt_qplib_srq { struct bnxt_qplib_sg_info sg_info; u16 eventq_hw_ring_id; spinlock_t lock; /* protect SRQE link list */ + u8 toggle; }; struct bnxt_qplib_sge { From patchwork Thu Aug 29 15:34:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13783386 Received: from mail-oo1-f54.google.com (mail-oo1-f54.google.com [209.85.161.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 289361B1D7F for ; Thu, 29 Aug 2024 15:55:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724946911; cv=none; b=QNEO4QH6ppfYu85qqb5Jb7NgwltpBBw3bxG+UEjb4gqf0Ci6nnPt/Kep8hYUr93diQMC4i1hZZx4bZCMcgb+L6a+TpOgervYeN0Cek611fzSBTKKt6sI22AKGvTUWHa9eYHhlsbfrPCw45JV1XOAeo9XIY9FLoZEveu5/XXeQJM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724946911; c=relaxed/simple; bh=L221gx8TMVtgUhpzyJXv+niB3Ixc62z7WLWBuVAmB1c=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=p/E+Nkgrkkq8wpYREo/1iW1KaLXnB19vWAaqVN1k0vlEqjP7ahDFw1ZIVXsw+i7nBhM5ISnnezoG2211H+sfVQ0BmmRURXbMlosjKflnkaCYvVdK1mVq9XybsL8MXqCoC+T7NrzQMtSw9vuc3jwek0tk5ogguybrWZldqvByjsg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=BoG46FeW; arc=none smtp.client-ip=209.85.161.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="BoG46FeW" Received: by mail-oo1-f54.google.com with SMTP id 006d021491bc7-5dcd8403656so549324eaf.1 for ; Thu, 29 Aug 2024 08:55:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1724946908; x=1725551708; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=66WM3UKiAkh51wtV1SpgB1n6RvJTxOU2qjO0gblS3oo=; b=BoG46FeWKecoqGSIAWUOFpTu+OpNMzH8QCZ3fJr0lWy51hHeF5uPctg8wvnobNpoEf oj/VWngq2DN2TNBnl6u+YzLwrQ0vaFM8gJzF5dRmjMa1yt+1UL1ovvOYJmXRa3sUCjem 44kfb3kIduTg3ipvMUeWsLOC/AXe4uJmTNynI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724946908; x=1725551708; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=66WM3UKiAkh51wtV1SpgB1n6RvJTxOU2qjO0gblS3oo=; b=K0L/QxRK68pCrrRIfXhgA0UDSSedvZHxL47hQ7BDgwkhTsgMj1K7xA0hDBDzx5/ID4 Xa1GTPQgPzequvh5hOpN6zQ3P5l/YqNpcCxVrMCab7xbCRohFWFbCgyhCnkf83OksACl /bIEUsPGX/4whQ9umfJG2FSgCrXK3DLiAmAQvula/63KhGZBzJaNmFoBf9qfTm6C625k 7tuhkOn22m19ezd2phj/cQhKXY8966mscnIgdxExkB4iswGhL/W1aPuSJcGSnHg/sIAG gRxZm4V0Y8BnPisosL1IX1x9nNMKd0f2G8/VTFiPCNtp4OKQxtj7Fp80wlp2cjGhe4KU 5/jQ== X-Gm-Message-State: AOJu0YyHQs+/0I3s5Rc2h3L8mLvmt1exaib8VlGCSkxGqwCmvXYLxV4b QtDmX3J7CgJtoabkYruHxU6I6+reWUdM8uWS5LjwgFfDh6C37UtzYwTx1Ty2LA== X-Google-Smtp-Source: AGHT+IHwtgASzziiFNYhFFV5N8cyl8+K+/7meuN3ZCgSR2fN+h184jzWDUn5QgNPe1tyavuyP8vpsw== X-Received: by 2002:a05:6358:61c2:b0:1b1:a810:85a5 with SMTP id e5c5f4694b2df-1b603c29ee4mr440469255d.12.1724946907997; Thu, 29 Aug 2024 08:55:07 -0700 (PDT) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7d22e9be627sm1396735a12.57.2024.08.29.08.55.05 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 29 Aug 2024 08:55:07 -0700 (PDT) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, chandramohan.akula@broadcom.com, Selvin Xavier Subject: [PATCH for-next v2 2/3] RDMA/bnxt_re: Refactor the BNXT_RE_METHOD_GET_TOGGLE_MEM method Date: Thu, 29 Aug 2024 08:34:04 -0700 Message-Id: <1724945645-14989-3-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1724945645-14989-1-git-send-email-selvin.xavier@broadcom.com> References: <1724945645-14989-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Chandramohan Akula Refactor the code in this function to have common code. This is used in subsequent patches. Signed-off-by: Chandramohan Akula Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index 43a68e7..1e76093 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -4494,12 +4494,12 @@ static int UVERBS_HANDLER(BNXT_RE_METHOD_GET_TOGGLE_MEM)(struct uverbs_attr_bund struct bnxt_re_ucontext *uctx; struct ib_ucontext *ib_uctx; struct bnxt_re_dev *rdev; + u32 length = PAGE_SIZE; struct bnxt_re_cq *cq; u64 mem_offset; + u32 offset = 0; u64 addr = 0; - u32 length; - u32 offset; - u32 cq_id; + u32 res_id; int err; ib_uctx = ib_uverbs_get_ucontext(attrs); @@ -4512,21 +4512,17 @@ static int UVERBS_HANDLER(BNXT_RE_METHOD_GET_TOGGLE_MEM)(struct uverbs_attr_bund uctx = container_of(ib_uctx, struct bnxt_re_ucontext, ib_uctx); rdev = uctx->rdev; + err = uverbs_copy_from(&res_id, attrs, BNXT_RE_TOGGLE_MEM_RES_ID); + if (err) + return err; switch (res_type) { case BNXT_RE_CQ_TOGGLE_MEM: - err = uverbs_copy_from(&cq_id, attrs, BNXT_RE_TOGGLE_MEM_RES_ID); - if (err) - return err; - - cq = bnxt_re_search_for_cq(rdev, cq_id); + cq = bnxt_re_search_for_cq(rdev, res_id); if (!cq) return -EINVAL; - length = PAGE_SIZE; addr = (u64)cq->uctx_cq_page; - mmap_flag = BNXT_RE_MMAP_TOGGLE_PAGE; - offset = 0; break; case BNXT_RE_SRQ_TOGGLE_MEM: break; From patchwork Thu Aug 29 15:34:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13783387 Received: from mail-ot1-f50.google.com (mail-ot1-f50.google.com [209.85.210.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 055DC1B3B1C for ; Thu, 29 Aug 2024 15:55:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724946913; cv=none; b=FAdtlVUEEioKG5/bUR+BnR7qvBkhdX0lFxX9WEVJguQ/RNT8f/7BIjMpWNPrsMzQPzMvBgOjQuYQWWPQScffkYuP3hlsZxaxQ3H6XWElH5GwTeSC19EfZlVhnBpqQVrcFndOVZZJ3XRa4E/1O0gwql4edvdcTISTJVJZ4o5AgLQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724946913; c=relaxed/simple; bh=yFKXbRr2knPG3/1UJO81I+KF4oUXgVVNPqcysbccjnc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=NDHvdPdnfcXE0JKEnLPK0ZvWs+jIe0Ckal4xzIA0wBjpIrmZZ5q+fbBXOkRzKafoGl9WaIdroQjJyaxLf5oLaiywhU9JBkp/M3/icTBBTg1eYQo56u9dCePDDTmHzqowTyB1JXNVgWCYqpQJNGnc7vu0PDzh8UzXNLK1Tnq0gmE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=G4R8w3F5; arc=none smtp.client-ip=209.85.210.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="G4R8w3F5" Received: by mail-ot1-f50.google.com with SMTP id 46e09a7af769-7093abb12edso594863a34.3 for ; Thu, 29 Aug 2024 08:55:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1724946911; x=1725551711; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=rWq8xb9Dyv6rXSK7AG+b5Q7iGXq1GJ1Fw+EtwirSeCY=; b=G4R8w3F5XuOL0XgUwJwMDiaflsdHglBtP4uq3zisJe2s7i4LNvYQ8kAXC/QlIrlKT8 rjnIcpbP9rLlkepmYekaDl5u2zDFlHud2IOBUcyy2QR1awdL7OxfGg6Pty/dxhWw23aQ B81Iy5Xs4ADdej7U6Wj6SUrsPt3MllznqRWos= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724946911; x=1725551711; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rWq8xb9Dyv6rXSK7AG+b5Q7iGXq1GJ1Fw+EtwirSeCY=; b=aMqL0NniquxEEuV6qbU7mnew9mlurt+Lo6K/z8H64pxy4RMXEtYZrbKk0hQG98m+4u Gvsg7edkqB6ItDF1G/Z0pFOHo7jFmDTG6R19CIU5MM6IDklxMl8bs6gSkaxjPQiyy17b wPKRIMOSRDV/g6dPahdYt3BP2zJIKZZlFARsnXfqshy2KE4NlUvHK/ActdEEUUlfxObx 4KLByokAmddAuunn639KU1tSkG0IyG0Kwg2QSD0U08Sg29kf7Kf55Eva8g+VVrxeyG3p 4parpd8glBBQcF+mb/TxFGk5mJVw9NPJwE9HPPUM6+rNPkRAv3t8CUk+ocdQuJEDquMy cBLA== X-Gm-Message-State: AOJu0YwbiqmMSsahGLsJgTNrOp065REaiOMwZvLrGpMCuJ6RtFDEoPTO kgK0fFtdrcPSehaPKnOxUgrFB/Ikncr+Xw3qF6AQ7xaSSii270PMGWv0ePEXmg== X-Google-Smtp-Source: AGHT+IE0SHf/++FNldEXVc4ETQ0EfP8oyLnf7Vsn3A5sDoL8ni0DAnKlO3Ug69+o5cDzbcIM6axLVA== X-Received: by 2002:a05:6358:3392:b0:1b3:9506:7531 with SMTP id e5c5f4694b2df-1b603bebbbemr405260255d.5.1724946910865; Thu, 29 Aug 2024 08:55:10 -0700 (PDT) Received: from sxavier-dev.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7d22e9be627sm1396735a12.57.2024.08.29.08.55.08 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 29 Aug 2024 08:55:10 -0700 (PDT) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, chandramohan.akula@broadcom.com, Selvin Xavier Subject: [PATCH for-next v2 3/3] RDMA/bnxt_re: Share a page to expose per SRQ info with userspace Date: Thu, 29 Aug 2024 08:34:05 -0700 Message-Id: <1724945645-14989-4-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1724945645-14989-1-git-send-email-selvin.xavier@broadcom.com> References: <1724945645-14989-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: From: Chandramohan Akula Gen P7 adapters needs to share a toggle bits information received in kernel driver with the user space. User space needs this info to arm the SRQ. User space application can get this page using the UAPI routines. Library will mmap this page and get the toggle bits to be used in the next ARM Doorbell. Uses a hash list to map the SRQ structure from the SRQ ID. SRQ structure is retrieved from the hash list while the library calls the UAPI routine to get the toggle page mapping. Currently the full page is mapped per SRQ. This can be optimized to enable multiple SRQs from the same application share the same page and different offsets in the page Signed-off-by: Chandramohan Akula Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 2 ++ drivers/infiniband/hw/bnxt_re/ib_verbs.c | 34 +++++++++++++++++++++++++++++++- drivers/infiniband/hw/bnxt_re/ib_verbs.h | 1 + drivers/infiniband/hw/bnxt_re/main.c | 6 +++++- include/uapi/rdma/bnxt_re-abi.h | 6 ++++++ 5 files changed, 47 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h index 0912d2f..2be9a62 100644 --- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h +++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h @@ -141,6 +141,7 @@ struct bnxt_re_pacing { #define BNXT_RE_GRC_FIFO_REG_BASE 0x2000 #define MAX_CQ_HASH_BITS (16) +#define MAX_SRQ_HASH_BITS (16) struct bnxt_re_dev { struct ib_device ibdev; struct list_head list; @@ -196,6 +197,7 @@ struct bnxt_re_dev { struct work_struct dbq_fifo_check_work; struct delayed_work dbq_pacing_work; DECLARE_HASHTABLE(cq_hash, MAX_CQ_HASH_BITS); + DECLARE_HASHTABLE(srq_hash, MAX_SRQ_HASH_BITS); }; #define to_bnxt_re_dev(ptr, member) \ diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index 1e76093..0219c8a 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -1685,6 +1685,10 @@ int bnxt_re_destroy_srq(struct ib_srq *ib_srq, struct ib_udata *udata) if (qplib_srq->cq) nq = qplib_srq->cq->nq; + if (rdev->chip_ctx->modes.toggle_bits & BNXT_QPLIB_SRQ_TOGGLE_BIT) { + free_page((unsigned long)srq->uctx_srq_page); + hash_del(&srq->hash_entry); + } bnxt_qplib_destroy_srq(&rdev->qplib_res, qplib_srq); ib_umem_release(srq->umem); atomic_dec(&rdev->stats.res.srq_count); @@ -1789,9 +1793,18 @@ int bnxt_re_create_srq(struct ib_srq *ib_srq, } if (udata) { - struct bnxt_re_srq_resp resp; + struct bnxt_re_srq_resp resp = {}; resp.srqid = srq->qplib_srq.id; + if (rdev->chip_ctx->modes.toggle_bits & BNXT_QPLIB_SRQ_TOGGLE_BIT) { + hash_add(rdev->srq_hash, &srq->hash_entry, srq->qplib_srq.id); + srq->uctx_srq_page = (void *)get_zeroed_page(GFP_KERNEL); + if (!srq->uctx_srq_page) { + rc = -ENOMEM; + goto fail; + } + resp.comp_mask |= BNXT_RE_SRQ_TOGGLE_PAGE_SUPPORT; + } rc = ib_copy_to_udata(udata, &resp, sizeof(resp)); if (rc) { ibdev_err(&rdev->ibdev, "SRQ copy to udata failed!"); @@ -4266,6 +4279,19 @@ static struct bnxt_re_cq *bnxt_re_search_for_cq(struct bnxt_re_dev *rdev, u32 cq return cq; } +static struct bnxt_re_srq *bnxt_re_search_for_srq(struct bnxt_re_dev *rdev, u32 srq_id) +{ + struct bnxt_re_srq *srq = NULL, *tmp_srq; + + hash_for_each_possible(rdev->srq_hash, tmp_srq, hash_entry, srq_id) { + if (tmp_srq->qplib_srq.id == srq_id) { + srq = tmp_srq; + break; + } + } + return srq; +} + /* Helper function to mmap the virtual memory from user app */ int bnxt_re_mmap(struct ib_ucontext *ib_uctx, struct vm_area_struct *vma) { @@ -4494,6 +4520,7 @@ static int UVERBS_HANDLER(BNXT_RE_METHOD_GET_TOGGLE_MEM)(struct uverbs_attr_bund struct bnxt_re_ucontext *uctx; struct ib_ucontext *ib_uctx; struct bnxt_re_dev *rdev; + struct bnxt_re_srq *srq; u32 length = PAGE_SIZE; struct bnxt_re_cq *cq; u64 mem_offset; @@ -4525,6 +4552,11 @@ static int UVERBS_HANDLER(BNXT_RE_METHOD_GET_TOGGLE_MEM)(struct uverbs_attr_bund addr = (u64)cq->uctx_cq_page; break; case BNXT_RE_SRQ_TOGGLE_MEM: + srq = bnxt_re_search_for_srq(rdev, res_id); + if (!srq) + return -EINVAL; + + addr = (u64)srq->uctx_srq_page; break; default: diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.h b/drivers/infiniband/hw/bnxt_re/ib_verbs.h index 4e113b9..9c74dfe 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.h +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.h @@ -78,6 +78,7 @@ struct bnxt_re_srq { struct ib_umem *umem; spinlock_t lock; /* protect srq */ void *uctx_srq_page; + struct hlist_node hash_entry; }; struct bnxt_re_qp { diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 9714b9a..1211fe5 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -139,8 +139,10 @@ static void bnxt_re_set_drv_mode(struct bnxt_re_dev *rdev, u8 mode) if (bnxt_re_hwrm_qcaps(rdev)) dev_err(rdev_to_dev(rdev), "Failed to query hwrm qcaps\n"); - if (bnxt_qplib_is_chip_gen_p7(rdev->chip_ctx)) + if (bnxt_qplib_is_chip_gen_p7(rdev->chip_ctx)) { cctx->modes.toggle_bits |= BNXT_QPLIB_CQ_TOGGLE_BIT; + cctx->modes.toggle_bits |= BNXT_QPLIB_SRQ_TOGGLE_BIT; + } } static void bnxt_re_destroy_chip_ctx(struct bnxt_re_dev *rdev) @@ -1771,6 +1773,8 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 wqe_mode) bnxt_re_vf_res_config(rdev); } hash_init(rdev->cq_hash); + if (rdev->chip_ctx->modes.toggle_bits & BNXT_QPLIB_SRQ_TOGGLE_BIT) + hash_init(rdev->srq_hash); return 0; free_sctx: diff --git a/include/uapi/rdma/bnxt_re-abi.h b/include/uapi/rdma/bnxt_re-abi.h index e61104f..46ad66a 100644 --- a/include/uapi/rdma/bnxt_re-abi.h +++ b/include/uapi/rdma/bnxt_re-abi.h @@ -134,8 +134,14 @@ struct bnxt_re_srq_req { __aligned_u64 srq_handle; }; +enum bnxt_re_srq_mask { + BNXT_RE_SRQ_TOGGLE_PAGE_SUPPORT = 0x1, +}; + struct bnxt_re_srq_resp { __u32 srqid; + __u32 rsvd; /* padding */ + __aligned_u64 comp_mask; }; enum bnxt_re_shpg_offt {