From patchwork Thu May 3 18:58:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Layton X-Patchwork-Id: 10378987 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 3B2DD60159 for ; Thu, 3 May 2018 18:58:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 289692924F for ; Thu, 3 May 2018 18:58:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1D77F29254; Thu, 3 May 2018 18:58:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 977FE2924F for ; Thu, 3 May 2018 18:58:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751503AbeECS6a (ORCPT ); Thu, 3 May 2018 14:58:30 -0400 Received: from mail.kernel.org ([198.145.29.99]:44774 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751339AbeECS6N (ORCPT ); Thu, 3 May 2018 14:58:13 -0400 Received: from tleilax.poochiereds.net (cpe-71-70-156-158.nc.res.rr.com [71.70.156.158]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 7D3DA2183D; Thu, 3 May 2018 18:58:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1525373892; bh=kJ/UXu8XlIgZk2F2sjxGxF3SDGaVF42ejcF9JRUfhqI=; h=From:To:Subject:Date:In-Reply-To:References:From; b=qhTjqDu7+LjVNcqIomUj6y3aa/RvNBS/aHsbgCAXlsyqc7jcWmhgYaoHfBmTHyC8w RqMO8XaLo1swoUIDLUJ35Lm2Z3T32owQZLNh7ebvBipV1He9ikymc5BuVyGJqnLBbe FL4TiCrkVGYn5Nzkt20xwKof4O2JnCjbo7R/Gln0= From: Jeff Layton To: devel@lists.nfs-ganesha.org, ceph-devel@vger.kernel.org Subject: [nfs-ganesha RFC PATCH v2 13/13] FSAL_CEPH: kill off old session before the mount Date: Thu, 3 May 2018 14:58:03 -0400 Message-Id: <20180503185803.25417-14-jlayton@kernel.org> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180503185803.25417-1-jlayton@kernel.org> References: <20180503185803.25417-1-jlayton@kernel.org> Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Jeff Layton Add a test for the new cephfs reclaim infrastructure. We don't actually try to reclaim at this point, only ask the MDS to destroy the old session. For now, we just use the hostname as the uuid here. The libcephfs interface is still under debate, so this may look quite different in its final form. Change-Id: I70e93cbff2e2ae80989aaa9ee9c96d3031f5e919 Signed-off-by: Jeff Layton --- src/CMakeLists.txt | 1 + src/FSAL/FSAL_CEPH/main.c | 39 ++++++++++++++++++++++++++++++ src/cmake/modules/FindCEPHFS.cmake | 8 ++++++ src/include/config-h.in.cmake | 1 + 4 files changed, 49 insertions(+) diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index fa1f6bc2e54d..845ac46d1f14 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -1101,6 +1101,7 @@ message(STATUS "USE_FSAL_CEPH_LL_LOOKUP_ROOT = ${USE_FSAL_CEPH_LL_LOOKUP_ROOT}") message(STATUS "USE_FSAL_CEPH_STATX = ${USE_FSAL_CEPH_STATX}") message(STATUS "USE_FSAL_CEPH_LL_DELEGATION = ${USE_FSAL_CEPH_LL_DELEGATION}") message(STATUS "USE_FSAL_CEPH_LL_SYNC_INODE = ${USE_FSAL_CEPH_LL_SYNC_INODE}") +message(STATUS "USE_FSAL_CEPH_RECLAIM_RESET = ${USE_FSAL_CEPH_RECLAIM_RESET}") message(STATUS "USE_FSAL_RGW = ${USE_FSAL_RGW}") message(STATUS "USE_FSAL_XFS = ${USE_FSAL_XFS}") message(STATUS "USE_FSAL_PANFS = ${USE_FSAL_PANFS}") diff --git a/src/FSAL/FSAL_CEPH/main.c b/src/FSAL/FSAL_CEPH/main.c index e78a4162c754..8d7039722fae 100644 --- a/src/FSAL/FSAL_CEPH/main.c +++ b/src/FSAL/FSAL_CEPH/main.c @@ -248,6 +248,10 @@ static fsal_status_t create_export(struct fsal_module *module_in, int rc; /* Return code from Ceph calls */ int ceph_status; +#ifdef USE_FSAL_CEPH_RECLAIM_RESET + long maxlen = sysconf(_SC_HOST_NAME_MAX); + char *nodeid; +#endif fsal_export_init(&export->export); export_ops_init(&export->export.exp_ops); @@ -311,6 +315,41 @@ static fsal_status_t create_export(struct fsal_module *module_in, goto error; } + ceph_status = ceph_init(export->cmount); + if (ceph_status != 0) { + status.major = ERR_FSAL_SERVERFAULT; + LogCrit(COMPONENT_FSAL, + "Unable to init Ceph handle for %s.", + op_ctx->ctx_export->fullpath); + goto error; + } + +#ifdef USE_FSAL_CEPH_RECLAIM_RESET + /* + * Set long timeout for the session to ensure that MDS doesn't lose + * state before server can come back and do recovery. + */ + ceph_set_timeout(export->cmount, 300); + + nodeid = gsh_malloc(maxlen); + rc = gethostname(nodeid, maxlen); + if (rc) { + status.major = ERR_FSAL_SERVERFAULT; + LogEvent(COMPONENT_FSAL, "gethostname failed: %d", errno); + free(nodeid); + goto error; + } + + ceph_status = ceph_start_reclaim(export->cmount, nodeid, + CEPH_RECLAIM_RESET); + if (ceph_status) + LogEvent(COMPONENT_FSAL, "start_reclaim failed: %d", + ceph_status); + ceph_finish_reclaim(export->cmount); + ceph_set_uuid(export->cmount, nodeid); + free(nodeid); +#endif /* USE_FSAL_CEPH_RECLAIM_RESET */ + ceph_status = ceph_mount(export->cmount, op_ctx->ctx_export->fullpath); if (ceph_status != 0) { status.major = ERR_FSAL_SERVERFAULT; diff --git a/src/cmake/modules/FindCEPHFS.cmake b/src/cmake/modules/FindCEPHFS.cmake index 4705d5d3135c..2f08f1f2dc5f 100644 --- a/src/cmake/modules/FindCEPHFS.cmake +++ b/src/cmake/modules/FindCEPHFS.cmake @@ -94,6 +94,14 @@ else (NOT CEPH_FS) set(USE_FSAL_CEPH_LL_SYNC_INODE ON) endif(NOT CEPH_FS_SYNC_INODE) + check_library_exists(cephfs ceph_start_reclaim ${CEPHFS_LIBRARY_DIR} CEPH_FS_RECLAIM_RESET) + if(NOT CEPH_FS_RECLAIM_RESET) + message("Cannot find ceph_start_reclaim. FSAL_CEPH will not kill off old sessions.") + set(USE_FSAL_CEPH_RECLAIM_RESET OFF) + else(NOT CEPH_FS_RECLAIM_RESET) + set(USE_FSAL_CEPH_RECLAIM_RESET ON) + endif(NOT CEPH_FS_RECLAIM_RESET) + set(CMAKE_REQUIRED_INCLUDES ${CEPHFS_INCLUDE_DIR}) check_symbol_exists(CEPH_STATX_INO "cephfs/libcephfs.h" CEPH_FS_CEPH_STATX) if(NOT CEPH_FS_CEPH_STATX) diff --git a/src/include/config-h.in.cmake b/src/include/config-h.in.cmake index 5f5685307c19..a7c8a80cd80a 100644 --- a/src/include/config-h.in.cmake +++ b/src/include/config-h.in.cmake @@ -63,6 +63,7 @@ #cmakedefine USE_FSAL_CEPH_STATX 1 #cmakedefine USE_FSAL_CEPH_LL_DELEGATION 1 #cmakedefine USE_FSAL_CEPH_LL_SYNC_INODE 1 +#cmakedefine USE_FSAL_CEPH_RECLAIM_RESET 1 #cmakedefine USE_FSAL_RGW_MOUNT2 1 #cmakedefine ENABLE_LOCKTRACE 1 #cmakedefine SANITIZE_ADDRESS 1