From patchwork Thu May 3 18:57:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Layton X-Patchwork-Id: 10378981 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 8680560327 for ; Thu, 3 May 2018 18:58:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 738B92924F for ; Thu, 3 May 2018 18:58:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6735A29254; Thu, 3 May 2018 18:58:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E55172924F for ; Thu, 3 May 2018 18:58:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751527AbeECS6d (ORCPT ); Thu, 3 May 2018 14:58:33 -0400 Received: from mail.kernel.org ([198.145.29.99]:44774 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751369AbeECS6K (ORCPT ); Thu, 3 May 2018 14:58:10 -0400 Received: from tleilax.poochiereds.net (cpe-71-70-156-158.nc.res.rr.com [71.70.156.158]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 14CDC217DF; Thu, 3 May 2018 18:58:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1525373890; bh=fR/HOS7Sh7tt6gxUTTY+utdqbQt4nyk5hri72hc2jYU=; h=From:To:Subject:Date:In-Reply-To:References:From; b=lo/jVsTf3YUWxPwmqqfZF6EfDzQB7s1uMJmsrTDuUHmuSF0oV55XCegPgYZrOnvgf AgsFq+5lJ7WnfbQV1I53Vd5vMJ7fWD6Jin9hz8e4gn5GoYnVmc2nXGWfWEksILgIN0 rtUlcxnXsebttsWhZpjO4OZXzH0C5RGXupByyQLA= From: Jeff Layton To: devel@lists.nfs-ganesha.org, ceph-devel@vger.kernel.org Subject: [nfs-ganesha RFC PATCH v2 09/13] main: add way to stall server until grace is being enforced Date: Thu, 3 May 2018 14:57:59 -0400 Message-Id: <20180503185803.25417-10-jlayton@kernel.org> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180503185803.25417-1-jlayton@kernel.org> References: <20180503185803.25417-1-jlayton@kernel.org> Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Jeff Layton To prevent conflicting state being handed out before you do reclaim, we need to wait until everyone is in enforcing mode before we ask the MDS to tear down any old sessions that may be present. The simplest way to do this is to delay startup of the server until everyone else is enforcing. Note that a singleton server should never end up waiting here. Change-Id: I0af053a3e2c91ee72ee6a0a0a9474d411c553e97 Signed-off-by: Jeff Layton --- src/MainNFSD/nfs_init.c | 3 --- src/MainNFSD/nfs_lib.c | 6 ++++++ src/MainNFSD/nfs_main.c | 6 ++++++ src/SAL/nfs4_recovery.c | 32 ++++++++++++++++++++++++++++++++ src/include/sal_functions.h | 2 ++ 5 files changed, 46 insertions(+), 3 deletions(-) diff --git a/src/MainNFSD/nfs_init.c b/src/MainNFSD/nfs_init.c index d4190ddc5911..2aebf41ba72c 100644 --- a/src/MainNFSD/nfs_init.c +++ b/src/MainNFSD/nfs_init.c @@ -794,9 +794,6 @@ static void nfs_Init(const nfs_start_info_t *p_start_info) /* Save Ganesha thread credentials with Frank's routine for later use */ fsal_save_ganesha_credentials(); - /* Start grace period */ - nfs_start_grace(NULL); - /* RPC Initialisation - exits on failure */ nfs_Init_svc(); LogInfo(COMPONENT_INIT, "RPC resources successfully initialized"); diff --git a/src/MainNFSD/nfs_lib.c b/src/MainNFSD/nfs_lib.c index d0e8cfe25038..646db20fa720 100644 --- a/src/MainNFSD/nfs_lib.c +++ b/src/MainNFSD/nfs_lib.c @@ -239,6 +239,12 @@ int nfs_libmain(const char *ganesha_conf, */ nfs4_recovery_init(); + /* Start grace period */ + nfs_start_grace(NULL); + + /* Wait for enforcement to begin */ + nfs_wait_for_grace_enforcement(); + /* Load export entries from parsed file * returns the number of export entries. */ diff --git a/src/MainNFSD/nfs_main.c b/src/MainNFSD/nfs_main.c index 652fb0da1ec2..075f1ee4ea91 100644 --- a/src/MainNFSD/nfs_main.c +++ b/src/MainNFSD/nfs_main.c @@ -484,6 +484,12 @@ int main(int argc, char *argv[]) */ nfs4_recovery_init(); + /* Start grace period */ + nfs_start_grace(NULL); + + /* Wait for enforcement to begin */ + nfs_wait_for_grace_enforcement(); + /* Load export entries from parsed file * returns the number of export entries. */ diff --git a/src/SAL/nfs4_recovery.c b/src/SAL/nfs4_recovery.c index 400d8b528d26..a0bd986ef247 100644 --- a/src/SAL/nfs4_recovery.c +++ b/src/SAL/nfs4_recovery.c @@ -280,6 +280,38 @@ void nfs_try_lift_grace(void) } } +static pthread_cond_t enforcing_cond = PTHREAD_COND_INITIALIZER; +static pthread_mutex_t enforcing_mutex = PTHREAD_MUTEX_INITIALIZER; + +/* Poll every 5s, just in case we miss the wakeup for some reason */ +void nfs_wait_for_grace_enforcement(void) +{ + nfs_grace_start_t gsp = { .event = EVENT_JUST_GRACE }; + + pthread_mutex_lock(&enforcing_mutex); + nfs_try_lift_grace(); + while (nfs_in_grace() && !nfs_grace_enforcing()) { + struct timespec timeo = { .tv_sec = time(NULL) + 5, + .tv_nsec = 0 }; + + pthread_cond_timedwait(&enforcing_cond, &enforcing_mutex, + &timeo); + + pthread_mutex_unlock(&enforcing_mutex); + nfs_start_grace(&gsp); + nfs_try_lift_grace(); + pthread_mutex_lock(&enforcing_mutex); + } + pthread_mutex_unlock(&enforcing_mutex); +} + +void nfs_notify_grace_waiters(void) +{ + pthread_mutex_lock(&enforcing_mutex); + pthread_cond_broadcast(&enforcing_cond); + pthread_mutex_unlock(&enforcing_mutex); +} + /** * @brief Create an entry in the recovery directory * diff --git a/src/include/sal_functions.h b/src/include/sal_functions.h index 96a4eb40a50b..708290495731 100644 --- a/src/include/sal_functions.h +++ b/src/include/sal_functions.h @@ -978,6 +978,8 @@ bool nfs_in_grace(void); bool simple_try_lift_grace(void); void nfs_maybe_start_grace(void); void nfs_try_lift_grace(void); +void nfs_wait_for_grace_enforcement(void); +void nfs_notify_grace_waiters(void); void nfs4_add_clid(nfs_client_id_t *); void nfs4_rm_clid(nfs_client_id_t *); void nfs4_recovery_reclaim_complete(nfs_client_id_t *clientid);