From patchwork Thu Oct 3 23:03:06 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Arlin Davis X-Patchwork-Id: 2986531 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 9A7069F288 for ; Thu, 3 Oct 2013 23:03:13 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 0F8FE20426 for ; Thu, 3 Oct 2013 23:03:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 593022035C for ; Thu, 3 Oct 2013 23:03:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753544Ab3JCXDL (ORCPT ); Thu, 3 Oct 2013 19:03:11 -0400 Received: from mga02.intel.com ([134.134.136.20]:15128 "EHLO mga02.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751374Ab3JCXDK (ORCPT ); Thu, 3 Oct 2013 19:03:10 -0400 Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga101.jf.intel.com with ESMTP; 03 Oct 2013 16:03:08 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.90,1028,1371106800"; d="scan'208";a="413746227" Received: from orsmsx105.amr.corp.intel.com ([10.22.225.132]) by orsmga002.jf.intel.com with ESMTP; 03 Oct 2013 16:03:07 -0700 Received: from orsmsx111.amr.corp.intel.com (10.22.240.12) by ORSMSX105.amr.corp.intel.com (10.22.225.132) with Microsoft SMTP Server (TLS) id 14.3.123.3; Thu, 3 Oct 2013 16:03:07 -0700 Received: from orsmsx101.amr.corp.intel.com ([169.254.8.125]) by ORSMSX111.amr.corp.intel.com ([169.254.11.8]) with mapi id 14.03.0123.003; Thu, 3 Oct 2013 16:03:07 -0700 From: "Davis, Arlin R" To: "linux-rdma@vger.kernel.org" , "ofw@lists.openfabrics.org" Subject: [PATCH] DAPL v2.0: ucm, scm: UD mode triggers list_head assert with large scale alltoall test Thread-Topic: [PATCH] DAPL v2.0: ucm, scm: UD mode triggers list_head assert with large scale alltoall test Thread-Index: Ac7AjIX3Vt2MIG+6RFqDMUO2FDKXaw== Date: Thu, 3 Oct 2013 23:03:06 +0000 Message-ID: <54347E5A035A054EAE9D05927FB467F972DB44CB@ORSMSX101.amr.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.22.254.140] MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-7.6 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP 1024+ ranks, IMB alltoall may hit assert when running Intel MPI in UD mode. CR cleanup was implemented with EP to CR references still linked. During cr_accept, the CR remote_ia_address is linked to EP object by mistake with UD mode. UD mode my have multiple CRs per EP so no direct mappings to CR memory can exist. Only with RC mode which always has one EP to CR mapping. In scm, ucm: for CM object free with CR references the search and unlinking from SP must be under SP lock to serialize. Also, change cleanup thread wakeup logic to only trigger the thread if reference count indicates the need for more processing. Signed-off-by: Arlin Davis --- dapl/common/dapl_cr_accept.c | 10 ++++++---- dapl/openib_scm/cm.c | 14 +++++++------- dapl/openib_ucm/cm.c | 26 +++++++++++++------------- dapl/openib_ucm/dapl_ib_util.h | 1 + 4 files changed, 27 insertions(+), 24 deletions(-) -- 1.7.3 diff --git a/dapl/common/dapl_cr_accept.c b/dapl/common/dapl_cr_accept.c index 5df9458..4e48fea 100644 --- a/dapl/common/dapl_cr_accept.c +++ b/dapl/common/dapl_cr_accept.c @@ -180,11 +180,13 @@ dapl_cr_accept(IN DAT_CR_HANDLE cr_handle, entry_ep_state = ep_ptr->param.ep_state; entry_ep_handle = cr_ptr->param.local_ep_handle; ep_ptr->param.ep_state = DAT_EP_STATE_COMPLETION_PENDING; - ep_ptr->cr_ptr = cr_ptr; - ep_ptr->param.remote_ia_address_ptr = - cr_ptr->param.remote_ia_address_ptr; - cr_ptr->param.local_ep_handle = ep_handle; + /* UD supports multiple CR's per EP, provider will manage CR's */ + if (ep_ptr->param.ep_attr.service_type == DAT_SERVICE_TYPE_RC) { + ep_ptr->cr_ptr = cr_ptr; + ep_ptr->param.remote_ia_address_ptr = cr_ptr->param.remote_ia_address_ptr; + } + cr_ptr->param.local_ep_handle = ep_handle; dapl_os_unlock(&ep_ptr->header.lock); dat_status = dapls_ib_accept_connection(cr_handle, diff --git a/dapl/openib_scm/cm.c b/dapl/openib_scm/cm.c index d4964bd..f7838f2 100644 --- a/dapl/openib_scm/cm.c +++ b/dapl/openib_scm/cm.c @@ -439,25 +439,25 @@ void dapls_cm_free(dp_ib_cm_handle_t cm_ptr) cm_ptr, dapl_cm_state_str(cm_ptr->state), cm_ptr->ep, sp_ptr, cm_ptr->ref_count); + dapl_os_lock(&cm_ptr->lock); if (sp_ptr && cm_ptr->state == DCM_CONNECTED && cm_ptr->msg.daddr.ib.qp_type == IBV_QPT_UD) { - DAPL_CR *cr_ptr = dapl_sp_search_cr(sp_ptr, cm_ptr); + DAPL_CR *cr_ptr; + + dapl_os_lock(&sp_ptr->header.lock); + cr_ptr = dapl_sp_search_cr(sp_ptr, cm_ptr); if (cr_ptr != NULL) { - dapl_os_lock(&sp_ptr->header.lock); dapl_sp_remove_cr(sp_ptr, cr_ptr); - dapl_os_unlock(&sp_ptr->header.lock); dapls_cr_free(cr_ptr); } + dapl_os_unlock(&sp_ptr->header.lock); } /* free from internal workq, wait until EP is last ref */ - dapl_os_lock(&cm_ptr->lock); cm_ptr->state = DCM_FREE; - dapl_os_unlock(&cm_ptr->lock); - dapli_cm_thread_signal(cm_ptr); - dapl_os_lock(&cm_ptr->lock); if (cm_ptr->ref_count != 1) { + dapli_cm_thread_signal(cm_ptr); dapl_os_unlock(&cm_ptr->lock); dapl_os_wait_object_wait(&cm_ptr->event, DAT_TIMEOUT_INFINITE); dapl_os_lock(&cm_ptr->lock); diff --git a/dapl/openib_ucm/cm.c b/dapl/openib_ucm/cm.c index 05cff10..d6f923e 100644 --- a/dapl/openib_ucm/cm.c +++ b/dapl/openib_ucm/cm.c @@ -779,19 +779,19 @@ void dapli_cm_free(dp_ib_cm_handle_t cm) cm, dapl_cm_state_str(cm->state), cm->ep, sp_ptr, sp_ptr ? sp_ptr->cr_list_count:0, cm->ref_count); + dapl_os_lock(&cm->lock); if (sp_ptr && cm->state == DCM_CONNECTED && cm->msg.daddr.ib.qp_type == IBV_QPT_UD) { - DAPL_CR *cr_ptr = dapl_sp_search_cr(sp_ptr, cm); - dapl_log(DAPL_DBG_TYPE_CM, " dapli_cm_free: UD CR %p\n", cr_ptr); - if (cr_ptr != NULL) { - dapl_os_lock(&sp_ptr->header.lock); - dapl_sp_remove_cr(sp_ptr, cr_ptr); - dapl_os_unlock(&sp_ptr->header.lock); - dapls_cr_free(cr_ptr); + dapl_os_lock(&sp_ptr->header.lock); + cm->cr = dapl_sp_search_cr(sp_ptr, cm); + dapl_log(DAPL_DBG_TYPE_CM, " dapli_cm_free: UD CR %p\n", cm->cr); + + if (cm->cr != NULL) { + dapl_sp_remove_cr(sp_ptr, cm->cr); + /* free CR at EP destroy */ } + dapl_os_unlock(&sp_ptr->header.lock); } - - dapl_os_lock(&cm->lock); cm->state = DCM_FREE; dapls_thread_signal(&cm->hca->ib_trans.signal); dapl_os_unlock(&cm->lock); @@ -809,12 +809,12 @@ void dapls_cm_free(dp_ib_cm_handle_t cm) dapl_os_lock(&cm->lock); if (cm->state != DCM_FREE) cm->state = DCM_FREE; - - dapl_os_unlock(&cm->lock); - dapls_thread_signal(&cm->hca->ib_trans.signal); - dapl_os_lock(&cm->lock); + if (cm->cr) + dapls_cr_free(cm->cr); + if (cm->ref_count != 1) { + dapls_thread_signal(&cm->hca->ib_trans.signal); dapl_os_unlock(&cm->lock); dapl_os_wait_object_wait(&cm->f_event, DAT_TIMEOUT_INFINITE); dapl_os_lock(&cm->lock); diff --git a/dapl/openib_ucm/dapl_ib_util.h b/dapl/openib_ucm/dapl_ib_util.h index b5bd082..469560e 100644 --- a/dapl/openib_ucm/dapl_ib_util.h +++ b/dapl/openib_ucm/dapl_ib_util.h @@ -48,6 +48,7 @@ struct ib_cm_handle struct dapl_hca *hca; struct dapl_sp *sp; struct dapl_ep *ep; + struct dapl_cr *cr; struct ibv_ah *ah; uint16_t p_size; /* accept p_data, for retries */ uint8_t p_data[DCM_MAX_PDATA_SIZE];