From patchwork Fri Feb 16 05:20:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 10224067 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 5AC9B603EE for ; Fri, 16 Feb 2018 05:21:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4CFB0290BB for ; Fri, 16 Feb 2018 05:21:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 41B5029507; Fri, 16 Feb 2018 05:21:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B0EC2290BB for ; Fri, 16 Feb 2018 05:21:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752669AbeBPFVK (ORCPT ); Fri, 16 Feb 2018 00:21:10 -0500 Received: from mail-qt0-f195.google.com ([209.85.216.195]:46161 "EHLO mail-qt0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752642AbeBPFVJ (ORCPT ); Fri, 16 Feb 2018 00:21:09 -0500 Received: by mail-qt0-f195.google.com with SMTP id u6so2485624qtg.13 for ; Thu, 15 Feb 2018 21:21:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=q3pziRv13/FqLI4u6n5gEHh7n9eu4tkHVT9QWgXIdXY=; b=h1pEhN9YRtTRbW3hRNTaQ6ZJxULFkGqK3dE5L9v648xCxY5hdQeSrWFM1pDux4V53R RwPkrCwKA/q0ChlewRkHLgPMM5AklQ7Zg+G8Bo4ACjhTSOHK1OPiVu1jECa2MEe9IYF8 Q2yOL8Acoij/fv5aFpO1l7iqlXXtSJDjppYFc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=q3pziRv13/FqLI4u6n5gEHh7n9eu4tkHVT9QWgXIdXY=; b=QT70wqjCI5ryG0UuUzQPiGwabfGemYmj1wwe4sih9Ct3omelOs6cTbLJVEsZdHh0Hg 0d7hAUb73FsooW1e0xZ57RTN4/1A6y1ktLPD61AEvbwVkD8iQVtdGfAJh4OgBZ1eSSm2 qb8pTXVTOglcUWDBJ9yvo8tCG8yJe4RxzXUkgfMxgjDUlmLbqkcM8VCuksMK8qa0V86E Z81ORORtZUeHVrZJI8+j6FCOK1DbIvWCsF6WLltLIMjtF9fT4M4KcQk2hU6tlWc3+JTb kDM7vJDndCFur7hpjNrKkRvBXZASqNovkYWZJ5eQaklYblSJ+Qpd2Vb7xfBPHHKjk1+U KtCw== X-Gm-Message-State: APf1xPDxGAdcciKnzXN5Yoyx7xZ8ZUHqWpGnBtTOXh9XDuSMzEXXQYE7 XCkQjVxMfsajpt0NRYqqlp1gIQ== X-Google-Smtp-Source: AH8x225b+QCxaKvXJ9ce9l5uyI3cMIEviOtm0YWbvHZAgNQACVNiZ1ZZu7YUDgJaqKHG7mIQZvg+Bw== X-Received: by 10.237.42.132 with SMTP id t4mr8203655qtd.297.1518758468851; Thu, 15 Feb 2018 21:21:08 -0800 (PST) Received: from dhcp-10-192-206-197.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id u4sm11257732qtg.28.2018.02.15.21.21.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 15 Feb 2018 21:21:08 -0800 (PST) From: Selvin Xavier To: dledford@redhat.com, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, Selvin Xavier Subject: [PATCH for-rc 2/6] RDMA/bnxt_re: Maintain GID index mapping between stack and hardware Date: Thu, 15 Feb 2018 21:20:09 -0800 Message-Id: <1518758413-20850-3-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1518758413-20850-1-git-send-email-selvin.xavier@broadcom.com> References: <1518758413-20850-1-git-send-email-selvin.xavier@broadcom.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Every GID has a duplicate entry in the stack for RoCE v2. HW table maintain only single entries. Currently the hw index is calculated by dividing the stack index by two. This is prone to failure after a series of add/delete gid operation in random order. Maintain a mapping between stack GID index and hardware index to avoid failure. Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/bnxt_re.h | 3 +++ drivers/infiniband/hw/bnxt_re/ib_verbs.c | 21 ++++++++++----------- drivers/infiniband/hw/bnxt_re/main.c | 13 +++++++++++++ 3 files changed, 26 insertions(+), 11 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/bnxt_re.h b/drivers/infiniband/hw/bnxt_re/bnxt_re.h index ca32057..5692335 100644 --- a/drivers/infiniband/hw/bnxt_re/bnxt_re.h +++ b/drivers/infiniband/hw/bnxt_re/bnxt_re.h @@ -109,6 +109,7 @@ struct bnxt_re_sqp_entries { #define BNXT_RE_MAX_MSIX 9 #define BNXT_RE_AEQ_IDX 0 #define BNXT_RE_NQ_IDX 1 +#define BNXT_RE_MAX_SGID_ENTRIES 256 struct bnxt_re_dev { struct ib_device ibdev; @@ -170,6 +171,8 @@ struct bnxt_re_dev { u32 is_virtfn; u32 num_vfs; struct bnxt_qplib_roce_stats stats; + /* Array to handle gid mapping */ + char *gid_map; }; #define to_bnxt_re_dev(ptr, member) \ diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index ce2a6d0..7943707 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -348,6 +348,7 @@ int bnxt_re_del_gid(struct ib_device *ibdev, u8 port_num, return -EFAULT; } ctx->refcnt--; + rdev->gid_map[index] = -1; if (!ctx->refcnt) { rc = bnxt_qplib_del_sgid(sgid_tbl, gid_to_del, true); if (rc) { @@ -386,6 +387,8 @@ int bnxt_re_add_gid(struct ib_device *ibdev, u8 port_num, ctx_tbl = sgid_tbl->ctx; ctx_tbl[tbl_idx]->refcnt++; *context = ctx_tbl[tbl_idx]; + /* tbl_idx is the HW table index and index is the stack index */ + rdev->gid_map[index] = tbl_idx; return 0; } @@ -401,6 +404,8 @@ int bnxt_re_add_gid(struct ib_device *ibdev, u8 port_num, ctx->idx = tbl_idx; ctx->refcnt = 1; ctx_tbl[tbl_idx] = ctx; + /* tbl_idx is the HW table index and index is the stack index */ + rdev->gid_map[index] = tbl_idx; *context = ctx; return rc; @@ -691,12 +696,8 @@ struct ib_ah *bnxt_re_create_ah(struct ib_pd *ib_pd, /* Supply the configuration for the HW */ memcpy(ah->qplib_ah.dgid.data, grh->dgid.raw, sizeof(union ib_gid)); - /* - * If RoCE V2 is enabled, stack will have two entries for - * each GID entry. Avoiding this duplicte entry in HW. Dividing - * the GID index by 2 for RoCE V2 - */ - ah->qplib_ah.sgid_index = grh->sgid_index / 2; + /* Retrieve the HW index from the driver SGID map */ + ah->qplib_ah.sgid_index = rdev->gid_map[ah_attr->grh.sgid_index]; ah->qplib_ah.host_sgid_index = grh->sgid_index; ah->qplib_ah.traffic_class = grh->traffic_class; ah->qplib_ah.flow_label = grh->flow_label; @@ -1641,11 +1642,9 @@ int bnxt_re_modify_qp(struct ib_qp *ib_qp, struct ib_qp_attr *qp_attr, memcpy(qp->qplib_qp.ah.dgid.data, grh->dgid.raw, sizeof(qp->qplib_qp.ah.dgid.data)); qp->qplib_qp.ah.flow_label = grh->flow_label; - /* If RoCE V2 is enabled, stack will have two entries for - * each GID entry. Avoiding this duplicte entry in HW. Dividing - * the GID index by 2 for RoCE V2 - */ - qp->qplib_qp.ah.sgid_index = grh->sgid_index / 2; + /* Retrieve the HW index from the driver SGID map */ + qp->qplib_qp.ah.sgid_index = + rdev->gid_map[qp_attr->ah_attr.grh.sgid_index]; qp->qplib_qp.ah.host_sgid_index = grh->sgid_index; qp->qplib_qp.ah.hop_limit = grh->hop_limit; qp->qplib_qp.ah.traffic_class = grh->traffic_class; diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 508d00a..3d5bbf9 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -658,6 +658,7 @@ static void bnxt_re_dev_remove(struct bnxt_re_dev *rdev) synchronize_rcu(); flush_workqueue(bnxt_re_wq); + kfree(rdev->gid_map); ib_dealloc_device(&rdev->ibdev); /* rdev is gone */ } @@ -666,6 +667,7 @@ static struct bnxt_re_dev *bnxt_re_dev_add(struct net_device *netdev, struct bnxt_en_dev *en_dev) { struct bnxt_re_dev *rdev; + u32 count; /* Allocate bnxt_re_dev instance here */ rdev = (struct bnxt_re_dev *)ib_alloc_device(sizeof(*rdev)); @@ -689,6 +691,17 @@ static struct bnxt_re_dev *bnxt_re_dev_add(struct net_device *netdev, rdev->cosq[0] = 0xFFFF; rdev->cosq[1] = 0xFFFF; + rdev->gid_map = kzalloc(sizeof(*rdev->gid_map) * + BNXT_RE_MAX_SGID_ENTRIES, + GFP_KERNEL); + if (!rdev->gid_map) { + ib_dealloc_device(&rdev->ibdev); + return NULL; + } + + for (count = 0; count < BNXT_RE_MAX_SGID_ENTRIES; count++) + rdev->gid_map[count] = -1; + mutex_lock(&bnxt_re_dev_lock); list_add_tail_rcu(&rdev->list, &bnxt_re_dev_list); mutex_unlock(&bnxt_re_dev_lock);