From patchwork Mon Dec 28 21:17:58 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dennis Dalessandro X-Patchwork-Id: 7927391 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 24E39BEEE5 for ; Mon, 28 Dec 2015 21:18:03 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 2E6E4201CE for ; Mon, 28 Dec 2015 21:18:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1577F201DD for ; Mon, 28 Dec 2015 21:18:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752643AbbL1VSA (ORCPT ); Mon, 28 Dec 2015 16:18:00 -0500 Received: from mga09.intel.com ([134.134.136.24]:14464 "EHLO mga09.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752744AbbL1VR7 (ORCPT ); Mon, 28 Dec 2015 16:17:59 -0500 Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga102.jf.intel.com with ESMTP; 28 Dec 2015 13:17:58 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,492,1444719600"; d="scan'208";a="20375717" Received: from scymds02.sc.intel.com ([10.82.195.37]) by fmsmga004.fm.intel.com with ESMTP; 28 Dec 2015 13:17:59 -0800 Received: from scvm10.sc.intel.com (scvm10.sc.intel.com [10.82.195.27]) by scymds02.sc.intel.com with ESMTP id tBSLHw4T019337; Mon, 28 Dec 2015 13:17:58 -0800 Received: from scvm10.sc.intel.com (localhost [127.0.0.1]) by scvm10.sc.intel.com with ESMTP id tBSLHwLw022937; Mon, 28 Dec 2015 13:17:58 -0800 Subject: [PATCH v2 31/36] IB/rdmavt: Add an ibport data structure to rdmavt To: dledford@redhat.com From: Dennis Dalessandro Cc: linux-rdma@vger.kernel.org, Harish Chegondi , Ira Weiny Date: Mon, 28 Dec 2015 13:17:58 -0800 Message-ID: <20151228211753.29850.63471.stgit@scvm10.sc.intel.com> In-Reply-To: <20151228205612.29850.7328.stgit@scvm10.sc.intel.com> References: <20151228205612.29850.7328.stgit@scvm10.sc.intel.com> User-Agent: StGit/0.16 MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Converge the ibport data structures of qib and hfi1 into a common ib port structure. Also provides a place to keep track of these ports in case rdmavt needs it. Along with this goes an attach and detach function for drivers to use to notify rdmavt of the ports. Reviewed-by: Ira Weiny Reviewed-by: Harish Chegondi Signed-off-by: Dennis Dalessandro --- drivers/infiniband/sw/rdmavt/vt.c | 24 +++++++++++++ include/rdma/rdma_vt.h | 66 ++++++++++++++++++++++++++++++++++++- 2 files changed, 89 insertions(+), 1 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/infiniband/sw/rdmavt/vt.c b/drivers/infiniband/sw/rdmavt/vt.c index 7dab0ca..44de280 100644 --- a/drivers/infiniband/sw/rdmavt/vt.c +++ b/drivers/infiniband/sw/rdmavt/vt.c @@ -300,6 +300,19 @@ int rvt_register_device(struct rvt_dev_info *rdi) spin_lock_init(&rdi->n_pds_lock); rdi->n_pds_allocated = 0; + if (rdi->dparms.nports) { + rdi->ports = kcalloc(rdi->dparms.nports, + sizeof(struct rvt_ibport **), + GFP_KERNEL); + if (!rdi->ports) { + rvt_pr_err(rdi, "Could not allocate port mem.\n"); + ret = -ENOMEM; + goto bail_mr; + } + } else { + rvt_pr_warn(rdi, "Driver says it has no ports.\n"); + } + /* We are now good to announce we exist */ ret = ib_register_device(&rdi->ibdev, rdi->driver_f.port_callback); if (ret) { @@ -327,3 +340,14 @@ void rvt_unregister_device(struct rvt_dev_info *rdi) rvt_mr_exit(rdi); } EXPORT_SYMBOL(rvt_unregister_device); + +/* + * Keep track of a list of ports. No need to have a detach port. + * They persist until the driver goes away. + */ +void rvt_attach_port(struct rvt_dev_info *rdi, struct rvt_ibport *port, + int portnum) +{ + rdi->ports[portnum] = port; +} +EXPORT_SYMBOL(rvt_attach_port); diff --git a/include/rdma/rdma_vt.h b/include/rdma/rdma_vt.h index fcf3ec0..a3d6a5b 100644 --- a/include/rdma/rdma_vt.h +++ b/include/rdma/rdma_vt.h @@ -53,6 +53,8 @@ * rdmavt layer. */ +#include +#include #include "ib_verbs.h" #define RVT_MULTICAST_LID_BASE 0xC000 @@ -359,6 +361,65 @@ struct rvt_srq { /* End QP section */ +struct rvt_ibport { + struct rvt_qp __rcu *qp[2]; + struct ib_mad_agent *send_agent; /* agent for SMI (traps) */ + struct rb_root mcast_tree; + spinlock_t lock; /* protect changes in this struct */ + + /* non-zero when timer is set */ + unsigned long mkey_lease_timeout; + unsigned long trap_timeout; + __be64 gid_prefix; /* in network order */ + __be64 mkey; + u64 tid; + u32 port_cap_flags; + u32 pma_sample_start; + u32 pma_sample_interval; + __be16 pma_counter_select[5]; + u16 pma_tag; + u16 mkey_lease_period; + u16 sm_lid; + u8 sm_sl; + u8 mkeyprot; + u8 subnet_timeout; + u8 vl_high_limit; + + /* + * Driver is expected to keep these up to date. These + * counters are informational only and not required to be + * completely accurate. + */ + u64 n_rc_resends; + u64 n_seq_naks; + u64 n_rdma_seq; + u64 n_rnr_naks; + u64 n_other_naks; + u64 n_loop_pkts; + u64 n_pkt_drops; + u64 n_vl15_dropped; + u64 n_rc_timeouts; + u64 n_dmawait; + u64 n_unaligned; + u64 n_rc_dupreq; + u64 n_rc_seqnak; + u16 pkey_violations; + u16 qkey_violations; + u16 mkey_violations; + + /* Hot-path per CPU counters to avoid cacheline trading to update */ + u64 z_rc_acks; + u64 z_rc_qacks; + u64 z_rc_delayed_comp; + u64 __percpu *rc_acks; + u64 __percpu *rc_qacks; + u64 __percpu *rc_delayed_comp; + + void *priv; /* driver private data */ + + /* TODO: Move sm_ah and smi_ah into here as well*/ +}; + /* * Things that are driver specific, module parameters in hfi1 and qib */ @@ -403,6 +464,7 @@ struct rvt_driver_params { * For instance special module parameters. Goes here. */ unsigned int lkey_table_size; + int nports; }; /* @@ -465,6 +527,7 @@ struct rvt_dev_info { spinlock_t n_ahs_lock; /* Protect ah allocated count */ int flags; + struct rvt_ibport **ports; }; static inline struct rvt_pd *ibpd_to_rvtpd(struct ib_pd *ibpd) @@ -501,9 +564,10 @@ static inline struct rvt_srq *ibsrq_to_rvtsrq(struct ib_srq *ibsrq) int rvt_register_device(struct rvt_dev_info *rvd); void rvt_unregister_device(struct rvt_dev_info *rvd); int rvt_check_ah(struct ib_device *ibdev, struct ib_ah_attr *ah_attr); +void rvt_attach_port(struct rvt_dev_info *rdi, struct rvt_ibport *port, + int portnum); int rvt_rkey_ok(struct rvt_qp *qp, struct rvt_sge *sge, u32 len, u64 vaddr, u32 rkey, int acc); int rvt_lkey_ok(struct rvt_lkey_table *rkt, struct rvt_pd *pd, struct rvt_sge *isge, struct ib_sge *sge, int acc); - #endif /* DEF_RDMA_VT_H */