From patchwork Thu Aug 11 00:42:07 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Arlin Davis X-Patchwork-Id: 1055422 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by demeter1.kernel.org (8.14.4/8.14.4) with ESMTP id p7B0fe5n016848 for ; Thu, 11 Aug 2011 00:42:10 GMT Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755490Ab1HKAmJ (ORCPT ); Wed, 10 Aug 2011 20:42:09 -0400 Received: from mga14.intel.com ([143.182.124.37]:14831 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752290Ab1HKAmI convert rfc822-to-8bit (ORCPT ); Wed, 10 Aug 2011 20:42:08 -0400 Received: from azsmga001.ch.intel.com ([10.2.17.19]) by azsmga102.ch.intel.com with ESMTP; 10 Aug 2011 17:42:08 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.67,353,1309762800"; d="scan'208";a="37259029" Received: from azsmsx601.amr.corp.intel.com ([10.2.121.193]) by azsmga001.ch.intel.com with ESMTP; 10 Aug 2011 17:42:08 -0700 Received: from fmsmsx153.amr.corp.intel.com (10.19.8.175) by azsmsx601.amr.corp.intel.com (10.2.121.193) with Microsoft SMTP Server (TLS) id 8.2.255.0; Wed, 10 Aug 2011 17:42:08 -0700 Received: from fmsmsx151.amr.corp.intel.com ([169.254.6.155]) by FMSMSX153.amr.corp.intel.com ([169.254.3.82]) with mapi id 14.01.0323.003; Wed, 10 Aug 2011 17:42:07 -0700 From: "Davis, Arlin R" To: linux-rdma , "ofw@lists.openfabrics.org" , "ewg@lists.openfabrics.org" Subject: [PATCH 6/10] DAPL v2.0: cma: add support for IB collective drivers in rdma_cm provider Thread-Topic: [PATCH 6/10] DAPL v2.0: cma: add support for IB collective drivers in rdma_cm provider Thread-Index: AcxXv38aXyL06wGFTC+zlkySvvWFBA== Date: Thu, 11 Aug 2011 00:42:07 +0000 Message-ID: <54347E5A035A054EAE9D05927FB467F916E42B54@FMSMSX151.amr.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.22.254.138] MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Greylist: IP, sender and recipient auto-whitelisted, not delayed by milter-greylist-4.2.6 (demeter1.kernel.org [140.211.167.41]); Thu, 11 Aug 2011 00:42:10 +0000 (UTC) Add collective member address and threading information on a per transport basis. Call create/free service with HCA open/close. Signed-off-by: Arlin Davis --- dapl/openib_cma/dapl_ib_util.h | 17 +++++++++++++++++ dapl/openib_cma/device.c | 13 +++++++++++++ 2 files changed, 30 insertions(+), 0 deletions(-) diff --git a/dapl/openib_cma/dapl_ib_util.h b/dapl/openib_cma/dapl_ib_util.h index 471bd7f..454f7e1 100755 --- a/dapl/openib_cma/dapl_ib_util.h +++ b/dapl/openib_cma/dapl_ib_util.h @@ -124,6 +124,23 @@ typedef struct _ib_hca_transport uint8_t sl; uint16_t pkey; int pkey_idx; +#ifdef DAT_IB_COLLECTIVES + /* Collective member device and address information */ + ib_thread_state_t coll_thread_state; + DAPL_OS_THREAD coll_thread; + DAPL_OS_LOCK coll_lock; + DAPL_OS_WAIT_OBJECT coll_event; + struct dapl_llist_entry *grp_list; + user_progress_func_t *user_func; + int l_sock; + struct sockaddr_in m_addr; + void *m_ctx; + void *m_info; + void *f_info; + int m_size; + int f_size; + int t_id; +#endif } ib_hca_transport_t; /* prototypes */ diff --git a/dapl/openib_cma/device.c b/dapl/openib_cma/device.c index 454c394..d1a3ab6 100644 --- a/dapl/openib_cma/device.c +++ b/dapl/openib_cma/device.c @@ -47,6 +47,10 @@ static const char rcsid[] = "$Id: $"; #include +#ifdef DAT_IB_COLLECTIVES +#include +#endif + struct rdma_event_channel *g_cm_events = NULL; ib_thread_state_t g_ib_thread_state = 0; DAPL_OS_THREAD g_ib_thread; @@ -376,6 +380,11 @@ DAT_RETURN dapls_ib_open_hca(IN IB_HCA_NAME hca_name, IN DAPL_HCA * hca_ptr) &hca_ptr->hca_address)->sin_addr.s_addr >> 24 & 0xff, hca_ptr->ib_trans.max_inline_send); +#ifdef DAT_IB_COLLECTIVES + if (dapli_create_collective_service(hca_ptr)) + return DAT_INTERNAL_ERROR; +#endif + return DAT_SUCCESS; } @@ -400,6 +409,10 @@ DAT_RETURN dapls_ib_close_hca(IN DAPL_HCA * hca_ptr) dapl_dbg_log(DAPL_DBG_TYPE_UTIL, " close_hca: %p->%p\n", hca_ptr, hca_ptr->ib_hca_handle); +#ifdef DAT_IB_COLLECTIVES + dapli_free_collective_service(hca_ptr); +#endif + dapl_os_lock(&g_hca_lock); if (g_ib_thread_state != IB_THREAD_RUN) { dapl_os_unlock(&g_hca_lock);