From patchwork Thu Mar 29 16:09:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Wise X-Patchwork-Id: 10316125 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id E230260383 for ; Thu, 29 Mar 2018 20:30:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D30E12A4F2 for ; Thu, 29 Mar 2018 20:30:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C7F542A4F7; Thu, 29 Mar 2018 20:30:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=2.0 tests=BAYES_00, DATE_IN_PAST_03_06, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 89CF12A4F2 for ; Thu, 29 Mar 2018 20:30:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751434AbeC2UaP (ORCPT ); Thu, 29 Mar 2018 16:30:15 -0400 Received: from opengridcomputing.com ([70.118.0.34]:49967 "EHLO smtp.opengridcomputing.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751070AbeC2UaP (ORCPT ); Thu, 29 Mar 2018 16:30:15 -0400 Received: by smtp.opengridcomputing.com (Postfix, from userid 503) id 82CB92BBEC; Thu, 29 Mar 2018 15:30:14 -0500 (CDT) Message-Id: <5344566bfd9c21d865bf7b04dc30c5a37c7c824a.1522341686.git.swise@opengridcomputing.com> In-Reply-To: References: From: Steve Wise Date: Thu, 29 Mar 2018 09:09:59 -0700 Subject: [PATCH v2 rdma-next 5/5] iw_cxgb4: dump detailed provider-specific QP information To: jgg@mellanox.com, dledford@redhat.com Cc: leon@kernel.org, linux-rdma@vger.kernel.org Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Provide a cxgb4-specific function to fill in qp state details. This allows dumping important c4iw_qp state useful for debugging. Included in the dump are the t4_sq, t4_rq structs, plus a dump of the t4_swsqe and t4swrqe descriptors for the first and last pending entries. Signed-off-by: Steve Wise --- drivers/infiniband/hw/cxgb4/Makefile | 3 +- drivers/infiniband/hw/cxgb4/iw_cxgb4.h | 6 + drivers/infiniband/hw/cxgb4/provider.c | 9 + drivers/infiniband/hw/cxgb4/restrack.c | 336 +++++++++++++++++++++++++++++++++ 4 files changed, 353 insertions(+), 1 deletion(-) create mode 100644 drivers/infiniband/hw/cxgb4/restrack.c diff --git a/drivers/infiniband/hw/cxgb4/Makefile b/drivers/infiniband/hw/cxgb4/Makefile index fa40b68..9edd920 100644 --- a/drivers/infiniband/hw/cxgb4/Makefile +++ b/drivers/infiniband/hw/cxgb4/Makefile @@ -3,4 +3,5 @@ ccflags-y += -Idrivers/net/ethernet/chelsio/libcxgb obj-$(CONFIG_INFINIBAND_CXGB4) += iw_cxgb4.o -iw_cxgb4-y := device.o cm.o provider.o mem.o cq.o qp.o resource.o ev.o id_table.o +iw_cxgb4-y := device.o cm.o provider.o mem.o cq.o qp.o resource.o ev.o id_table.o \ + restrack.o diff --git a/drivers/infiniband/hw/cxgb4/iw_cxgb4.h b/drivers/infiniband/hw/cxgb4/iw_cxgb4.h index cc92900..bbdc8dd 100644 --- a/drivers/infiniband/hw/cxgb4/iw_cxgb4.h +++ b/drivers/infiniband/hw/cxgb4/iw_cxgb4.h @@ -55,6 +55,7 @@ #include #include #include +#include #include "cxgb4.h" #include "cxgb4_uld.h" @@ -1078,4 +1079,9 @@ void __iomem *c4iw_bar2_addrs(struct c4iw_rdev *rdev, unsigned int qid, void c4iw_invalidate_mr(struct c4iw_dev *rhp, u32 rkey); struct c4iw_wr_wait *c4iw_alloc_wr_wait(gfp_t gfp); +typedef int c4iw_restrack_func(struct sk_buff *msg, + struct netlink_callback *cb, + struct rdma_restrack_entry *res); +extern c4iw_restrack_func *c4iw_restrack_funcs[RDMA_RESTRACK_MAX]; + #endif diff --git a/drivers/infiniband/hw/cxgb4/provider.c b/drivers/infiniband/hw/cxgb4/provider.c index 0b9cc73..1dda32b 100644 --- a/drivers/infiniband/hw/cxgb4/provider.c +++ b/drivers/infiniband/hw/cxgb4/provider.c @@ -551,6 +551,14 @@ static struct net_device *get_netdev(struct ib_device *dev, u8 port) return ndev; } +static int fill_res_entry(struct sk_buff *msg, struct netlink_callback *cb, + struct rdma_restrack_entry *res) +{ + return (res->type < ARRAY_SIZE(c4iw_restrack_funcs) && + c4iw_restrack_funcs[res->type]) ? + c4iw_restrack_funcs[res->type](msg, cb, res) : 0; +} + void c4iw_register_device(struct work_struct *work) { int ret; @@ -645,6 +653,7 @@ void c4iw_register_device(struct work_struct *work) dev->ibdev.iwcm->add_ref = c4iw_qp_add_ref; dev->ibdev.iwcm->rem_ref = c4iw_qp_rem_ref; dev->ibdev.iwcm->get_qp = c4iw_get_qp; + dev->ibdev.res.fill_res_entry = fill_res_entry; memcpy(dev->ibdev.iwcm->ifname, dev->rdev.lldi.ports[0]->name, sizeof(dev->ibdev.iwcm->ifname)); diff --git a/drivers/infiniband/hw/cxgb4/restrack.c b/drivers/infiniband/hw/cxgb4/restrack.c new file mode 100644 index 0000000..00d2759 --- /dev/null +++ b/drivers/infiniband/hw/cxgb4/restrack.c @@ -0,0 +1,336 @@ +/* + * Copyright (c) 2018 Chelsio, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include "iw_cxgb4.h" +#include +#include + +static int fill_sq(struct sk_buff *msg, struct t4_wq *wq) +{ + struct nlattr *entry_attr; + + /* WQ+SQ */ + entry_attr = nla_nest_start(msg, RDMA_NLDEV_ATTR_PROVIDER_ENTRY); + if (!entry_attr) + goto err; + + if (rdma_nl_put_provider_u32(msg, "sqid", RDMA_NLDEV_PRINT_TYPE_UNSPEC, + wq->sq.qid)) + goto err_cancel_entry; + if (rdma_nl_put_provider_u32(msg, "flushed", + RDMA_NLDEV_PRINT_TYPE_UNSPEC, wq->flushed)) + goto err_cancel_entry; + if (rdma_nl_put_provider_u32(msg, "memsize", + RDMA_NLDEV_PRINT_TYPE_UNSPEC, + wq->sq.memsize)) + goto err_cancel_entry; + if (rdma_nl_put_provider_u32(msg, "cidx", RDMA_NLDEV_PRINT_TYPE_UNSPEC, + wq->sq.cidx)) + goto err_cancel_entry; + if (rdma_nl_put_provider_u32(msg, "pidx", RDMA_NLDEV_PRINT_TYPE_UNSPEC, + wq->sq.pidx)) + goto err_cancel_entry; + if (rdma_nl_put_provider_u32(msg, "wq_pidx", + RDMA_NLDEV_PRINT_TYPE_UNSPEC, + wq->sq.wq_pidx)) + goto err_cancel_entry; + if (rdma_nl_put_provider_u32(msg, "flush_cidx", + RDMA_NLDEV_PRINT_TYPE_UNSPEC, + wq->sq.flush_cidx)) + goto err_cancel_entry; + if (rdma_nl_put_provider_u32(msg, "in_use", + RDMA_NLDEV_PRINT_TYPE_UNSPEC, + wq->sq.in_use)) + goto err_cancel_entry; + if (rdma_nl_put_provider_u32(msg, "size", RDMA_NLDEV_PRINT_TYPE_UNSPEC, + wq->sq.size)) + goto err_cancel_entry; + if (rdma_nl_put_provider_u32(msg, "flags", RDMA_NLDEV_PRINT_TYPE_HEX, + wq->sq.flags)) + goto err_cancel_entry; + + nla_nest_end(msg, entry_attr); + return 0; + +err_cancel_entry: + nla_nest_cancel(msg, entry_attr); +err: + return -EMSGSIZE; +} + +static int fill_rq(struct sk_buff *msg, struct t4_wq *wq) +{ + struct nlattr *entry_attr; + + /* RQ */ + entry_attr = nla_nest_start(msg, RDMA_NLDEV_ATTR_PROVIDER_ENTRY); + if (!entry_attr) + goto err; + + if (rdma_nl_put_provider_u32(msg, "rqid", RDMA_NLDEV_PRINT_TYPE_UNSPEC, + wq->rq.qid)) + goto err_cancel_entry; + if (rdma_nl_put_provider_u32(msg, "memsize", + RDMA_NLDEV_PRINT_TYPE_UNSPEC, + wq->rq.memsize)) + goto err_cancel_entry; + if (rdma_nl_put_provider_u32(msg, "cidx", RDMA_NLDEV_PRINT_TYPE_UNSPEC, + wq->rq.cidx)) + goto err_cancel_entry; + if (rdma_nl_put_provider_u32(msg, "pidx", RDMA_NLDEV_PRINT_TYPE_UNSPEC, + wq->rq.pidx)) + goto err_cancel_entry; + if (rdma_nl_put_provider_u32(msg, "wq_pidx", + RDMA_NLDEV_PRINT_TYPE_UNSPEC, + wq->rq.wq_pidx)) + goto err_cancel_entry; + if (rdma_nl_put_provider_u32(msg, "msn", RDMA_NLDEV_PRINT_TYPE_UNSPEC, + wq->rq.msn)) + goto err_cancel_entry; + if (rdma_nl_put_provider_u32(msg, "rqt_hwaddr", + RDMA_NLDEV_PRINT_TYPE_HEX, + wq->rq.rqt_hwaddr)) + goto err_cancel_entry; + if (rdma_nl_put_provider_u32(msg, "rqt_size", + RDMA_NLDEV_PRINT_TYPE_UNSPEC, + wq->rq.rqt_size)) + goto err_cancel_entry; + if (rdma_nl_put_provider_u32(msg, "in_use", + RDMA_NLDEV_PRINT_TYPE_UNSPEC, + wq->rq.in_use)) + goto err_cancel_entry; + if (rdma_nl_put_provider_u32(msg, "size", RDMA_NLDEV_PRINT_TYPE_UNSPEC, + wq->rq.size)) + goto err_cancel_entry; + + nla_nest_end(msg, entry_attr); + return 0; + +err_cancel_entry: + nla_nest_cancel(msg, entry_attr); +err: + return -EMSGSIZE; +} + +static int fill_swsqe(struct sk_buff *msg, struct t4_sq *sq, u16 idx, + struct t4_swsqe *sqe) +{ + struct nlattr *entry_attr; + + entry_attr = nla_nest_start(msg, RDMA_NLDEV_ATTR_PROVIDER_ENTRY); + if (!entry_attr) + goto err; + + if (rdma_nl_put_provider_u32(msg, " idx", + RDMA_NLDEV_PRINT_TYPE_UNSPEC, idx)) + goto err_cancel_entry; + if (rdma_nl_put_provider_u32(msg, "opcode", + RDMA_NLDEV_PRINT_TYPE_UNSPEC, sqe->opcode)) + goto err_cancel_entry; + if (rdma_nl_put_provider_u64(msg, "wr_id", + RDMA_NLDEV_PRINT_TYPE_HEX, sqe->wr_id)) + goto err_cancel_entry; + if (rdma_nl_put_provider_u32(msg, "complete", + RDMA_NLDEV_PRINT_TYPE_UNSPEC, + sqe->complete)) + goto err_cancel_entry; + if (sqe->complete) { + if (rdma_nl_put_provider_u32(msg, "cqe_status", + RDMA_NLDEV_PRINT_TYPE_UNSPEC, + CQE_STATUS(&sqe->cqe))) + goto err_cancel_entry; + } + if (rdma_nl_put_provider_u32(msg, "signaled", + RDMA_NLDEV_PRINT_TYPE_UNSPEC, + sqe->signaled)) + goto err_cancel_entry; + if (rdma_nl_put_provider_u32(msg, "flushed", + RDMA_NLDEV_PRINT_TYPE_UNSPEC, + sqe->flushed)) + goto err_cancel_entry; + + nla_nest_end(msg, entry_attr); + return 0; + +err_cancel_entry: + nla_nest_cancel(msg, entry_attr); +err: + return -EMSGSIZE; +} + +/* + * Dump the first and last pending sqes. + */ +static int fill_swsqes(struct sk_buff *msg, struct t4_sq *sq, + u16 first_idx, struct t4_swsqe *first_sqe, + u16 last_idx, struct t4_swsqe *last_sqe) +{ + if (!first_sqe) + goto out; + if (fill_swsqe(msg, sq, first_idx, first_sqe)) + goto err; + if (!last_sqe) + goto out; + if (fill_swsqe(msg, sq, last_idx, last_sqe)) + goto err; +out: + return 0; +err: + return -EMSGSIZE; +} + +static int fill_swrqe(struct sk_buff *msg, struct t4_rq *rq, u16 idx, + struct t4_swrqe *rqe) +{ + struct nlattr *entry_attr; + + entry_attr = nla_nest_start(msg, RDMA_NLDEV_ATTR_PROVIDER_ENTRY); + if (!entry_attr) + goto err; + + if (rdma_nl_put_provider_u32(msg, " idx", + RDMA_NLDEV_PRINT_TYPE_UNSPEC, idx)) + goto err_cancel_entry; + if (rdma_nl_put_provider_u64(msg, "wr_id", + RDMA_NLDEV_PRINT_TYPE_HEX, rqe->wr_id)) + goto err_cancel_entry; + + nla_nest_end(msg, entry_attr); + return 0; + +err_cancel_entry: + nla_nest_cancel(msg, entry_attr); +err: + return -EMSGSIZE; +} + +/* + * Dump the first and last pending rqes. + */ +static int fill_swrqes(struct sk_buff *msg, struct t4_rq *rq, + u16 first_idx, struct t4_swrqe *first_rqe, + u16 last_idx, struct t4_swrqe *last_rqe) +{ + if (!first_rqe) + goto out; + if (fill_swrqe(msg, rq, first_idx, first_rqe)) + goto err; + if (!last_rqe) + goto out; + if (fill_swrqe(msg, rq, last_idx, last_rqe)) + goto err; +out: + return 0; +err: + return -EMSGSIZE; +} + +static int fill_res_qp_entry(struct sk_buff *msg, + struct netlink_callback *cb, + struct rdma_restrack_entry *res) +{ + struct ib_qp *ibqp = container_of(res, struct ib_qp, res); + struct t4_swsqe *fsp = NULL, *lsp = NULL; + struct t4_swrqe *frp = NULL, *lrp = NULL; + struct c4iw_qp *qhp = to_c4iw_qp(ibqp); + struct t4_swsqe first_sqe, last_sqe; + struct t4_swrqe first_rqe, last_rqe; + u16 first_sq_idx, last_sq_idx; + u16 first_rq_idx, last_rq_idx; + struct nlattr *table_attr; + struct t4_wq wq; + + /* User qp state is not available, so don't dump user qps */ + if (qhp->ucontext) + return 0; + + table_attr = nla_nest_start(msg, RDMA_NLDEV_ATTR_PROVIDER); + if (!table_attr) + goto err; + + /* Get a consistent snapshot */ + spin_lock_irq(&qhp->lock); + wq = qhp->wq; + + /* If there are any pending sqes, copy the first and last */ + if (wq.sq.cidx != wq.sq.pidx) { + first_sq_idx = wq.sq.cidx; + first_sqe = qhp->wq.sq.sw_sq[first_sq_idx]; + fsp = &first_sqe; + last_sq_idx = wq.sq.pidx; + if (last_sq_idx-- == 0) + last_sq_idx = wq.sq.size - 1; + if (last_sq_idx != first_sq_idx) { + last_sqe = qhp->wq.sq.sw_sq[last_sq_idx]; + lsp = &last_sqe; + } + } + + /* If there are any pending rqes, copy the first and last */ + if (wq.rq.cidx != wq.rq.pidx) { + first_rq_idx = wq.rq.cidx; + first_rqe = qhp->wq.rq.sw_rq[first_rq_idx]; + frp = &first_rqe; + last_rq_idx = wq.rq.pidx; + if (last_rq_idx-- == 0) + last_rq_idx = wq.rq.size - 1; + if (last_rq_idx != first_rq_idx) { + last_rqe = qhp->wq.rq.sw_rq[last_rq_idx]; + lrp = &last_rqe; + } + } + spin_unlock_irq(&qhp->lock); + + if (fill_sq(msg, &wq)) + goto err_cancel_table; + + if (fill_swsqes(msg, &wq.sq, first_sq_idx, fsp, last_sq_idx, lsp)) + goto err_cancel_table; + + if (fill_rq(msg, &wq)) + goto err_cancel_table; + + if (fill_swrqes(msg, &wq.rq, first_rq_idx, frp, last_rq_idx, lrp)) + goto err_cancel_table; + + nla_nest_end(msg, table_attr); + return 0; + +err_cancel_table: + nla_nest_cancel(msg, table_attr); +err: + return -EMSGSIZE; +} + +c4iw_restrack_func *c4iw_restrack_funcs[RDMA_RESTRACK_MAX] = { + [RDMA_RESTRACK_QP] = fill_res_qp_entry, +};