From patchwork Thu Feb 27 21:15:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 11410483 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C04EA17E0 for ; Thu, 27 Feb 2020 21:39:32 +0000 (UTC) Received: from pdx1-mailman02.dreamhost.com (pdx1-mailman02.dreamhost.com [64.90.62.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A907424690 for ; Thu, 27 Feb 2020 21:39:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A907424690 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lustre-devel-bounces@lists.lustre.org Received: from pdx1-mailman02.dreamhost.com (localhost [IPv6:::1]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id 103E3348953; Thu, 27 Feb 2020 13:32:13 -0800 (PST) X-Original-To: lustre-devel@lists.lustre.org Delivered-To: lustre-devel-lustre.org@pdx1-mailman02.dreamhost.com Received: from smtp3.ccs.ornl.gov (smtp3.ccs.ornl.gov [160.91.203.39]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id 8D2E5220000 for ; Thu, 27 Feb 2020 13:20:48 -0800 (PST) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp3.ccs.ornl.gov (Postfix) with ESMTP id EBCF5917C; Thu, 27 Feb 2020 16:18:18 -0500 (EST) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id EAAF146C; Thu, 27 Feb 2020 16:18:18 -0500 (EST) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Thu, 27 Feb 2020 16:15:51 -0500 Message-Id: <1582838290-17243-484-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1582838290-17243-1-git-send-email-jsimmons@infradead.org> References: <1582838290-17243-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 483/622] lustre: ptlrpc: Hold imp lock for idle reconnect X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" From: Patrick Farrell Idle reconnect sets import state to IMP_NEW, then releases the import lock before calling ptlrpc_connect_import. This creates a gap where an import in IMP_NEW state is exposed, which can cause new requests to fail with EIO. Hold the lock across the call so as not to expose imports in this state. WC-bug-id: https://jira.whamcloud.com/browse/LU-12559 Lustre-commit: e9472c54ac82 ("LU-12559 ptlrpc: Hold imp lock for idle reconnect") Signed-off-by: Patrick Farrell Reviewed-on: https://review.whamcloud.com/35530 Reviewed-by: Alex Zhuravlev Reviewed-by: Wang Shilong Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- fs/lustre/include/lustre_net.h | 1 + fs/lustre/ptlrpc/client.c | 13 ++++++------- fs/lustre/ptlrpc/import.c | 19 +++++++++++++++---- 3 files changed, 22 insertions(+), 11 deletions(-) diff --git a/fs/lustre/include/lustre_net.h b/fs/lustre/include/lustre_net.h index aaf5cb8..8dad08e 100644 --- a/fs/lustre/include/lustre_net.h +++ b/fs/lustre/include/lustre_net.h @@ -2015,6 +2015,7 @@ struct ptlrpc_service *ptlrpc_register_service(struct ptlrpc_service_conf *conf, * @{ */ int ptlrpc_connect_import(struct obd_import *imp); +int ptlrpc_connect_import_locked(struct obd_import *imp); int ptlrpc_init_import(struct obd_import *imp); int ptlrpc_disconnect_import(struct obd_import *imp, int noclose); int ptlrpc_disconnect_and_idle_import(struct obd_import *imp); diff --git a/fs/lustre/ptlrpc/client.c b/fs/lustre/ptlrpc/client.c index 478ba85..c359ac0 100644 --- a/fs/lustre/ptlrpc/client.c +++ b/fs/lustre/ptlrpc/client.c @@ -870,7 +870,6 @@ struct ptlrpc_request *__ptlrpc_request_alloc(struct obd_import *imp, const struct req_format *format) { struct ptlrpc_request *request; - int connect = 0; request = __ptlrpc_request_alloc(imp, pool); if (!request) @@ -890,17 +889,17 @@ struct ptlrpc_request *__ptlrpc_request_alloc(struct obd_import *imp, if (imp->imp_state == LUSTRE_IMP_IDLE) { imp->imp_generation++; imp->imp_initiated_at = imp->imp_generation; - imp->imp_state = LUSTRE_IMP_NEW; - connect = 1; - } - spin_unlock(&imp->imp_lock); - if (connect) { - rc = ptlrpc_connect_import(imp); + imp->imp_state = LUSTRE_IMP_NEW; + + /* connect_import_locked releases imp_lock */ + rc = ptlrpc_connect_import_locked(imp); if (rc < 0) { ptlrpc_request_free(request); return NULL; } ptlrpc_pinger_add_import(imp); + } else { + spin_unlock(&imp->imp_lock); } } diff --git a/fs/lustre/ptlrpc/import.c b/fs/lustre/ptlrpc/import.c index ff1b810..c4a732d 100644 --- a/fs/lustre/ptlrpc/import.c +++ b/fs/lustre/ptlrpc/import.c @@ -611,13 +611,22 @@ static int ptlrpc_first_transno(struct obd_import *imp, u64 *transno) return 0; } +int ptlrpc_connect_import(struct obd_import *imp) +{ + spin_lock(&imp->imp_lock); + return ptlrpc_connect_import_locked(imp); +} + /** * Attempt to (re)connect import @imp. This includes all preparations, * initializing CONNECT RPC request and passing it to ptlrpcd for * actual sending. + * + * Assumes imp->imp_lock is held, and releases it. + * * Returns 0 on success or error code. */ -int ptlrpc_connect_import(struct obd_import *imp) +int ptlrpc_connect_import_locked(struct obd_import *imp) { struct obd_device *obd = imp->imp_obd; int initial_connect = 0; @@ -634,7 +643,8 @@ int ptlrpc_connect_import(struct obd_import *imp) struct ptlrpc_connect_async_args *aa; int rc; - spin_lock(&imp->imp_lock); + assert_spin_locked(&imp->imp_lock); + if (imp->imp_state == LUSTRE_IMP_CLOSED) { spin_unlock(&imp->imp_lock); CERROR("can't connect to a closed import\n"); @@ -1701,12 +1711,13 @@ static int ptlrpc_disconnect_idle_interpret(const struct lu_env *env, connect = 1; } } - spin_unlock(&imp->imp_lock); if (connect) { - rc = ptlrpc_connect_import(imp); + rc = ptlrpc_connect_import_locked(imp); if (rc >= 0) ptlrpc_pinger_add_import(imp); + } else { + spin_unlock(&imp->imp_lock); } return 0;