From patchwork Sat Dec 21 12:54:48 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hal Rosenstock X-Patchwork-Id: 3393551 X-Patchwork-Delegate: hal@mellanox.com Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 5365BC0D4A for ; Sat, 21 Dec 2013 12:55:00 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id F207F206F6 for ; Sat, 21 Dec 2013 12:54:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5C047206F0 for ; Sat, 21 Dec 2013 12:54:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752571Ab3LUMyy (ORCPT ); Sat, 21 Dec 2013 07:54:54 -0500 Received: from mail-ee0-f53.google.com ([74.125.83.53]:33494 "EHLO mail-ee0-f53.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751766Ab3LUMyx (ORCPT ); Sat, 21 Dec 2013 07:54:53 -0500 Received: by mail-ee0-f53.google.com with SMTP id b57so1527967eek.40 for ; Sat, 21 Dec 2013 04:54:52 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:content-type:content-transfer-encoding; bh=UjE+9Lg6o9sRzirkn2/TRFanXr9tVXborZzSJDirabU=; b=XlKIOO3hBJeT+HAdvgtwN48rQXNeZJl6EVcf71QFo/30Rr0Gw/CW+IiEBbbPyfV7qj UBzNmpr96BYeE1rkJdsEYRGfz0uVKz04Vfv8zLnZ38b0YKAof4MweDJySVwr9jCw2FFv 3Tivza+FGLcZrT56pvCTe5qzPPgG+zuYwVI0hV5jqpf/vb7j842TnJyD64bBwTxfF2aW 9nym5GwENOZJrmjwVRlL91Qg9SzGxTBGRq43j1Wj8DFX6qOlWRBVa3MhxVZcf4Ehp2Ib ExTJKjK75AzliUtYwc0f8XLhoSXvWMur3cJ8aNtq3KzRow43U8v5rZYqIWDOllQc4zVF XRiw== X-Gm-Message-State: ALoCoQlF0ywc42pTX4WKWoGGCj2wrF43ZWYgUqStFn6pf59tWM+NifTJvs+PutgcgEfBqJG5kUuo X-Received: by 10.14.172.130 with SMTP id t2mr2804205eel.68.1387630491355; Sat, 21 Dec 2013 04:54:51 -0800 (PST) Received: from [192.168.1.102] (c-98-229-118-119.hsd1.ma.comcast.net. [98.229.118.119]) by mx.google.com with ESMTPSA id a45sm27402502eem.6.2013.12.21.04.54.49 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 21 Dec 2013 04:54:51 -0800 (PST) Message-ID: <52B58F98.7040804@dev.mellanox.co.il> Date: Sat, 21 Dec 2013 07:54:48 -0500 From: Hal Rosenstock User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:9.0) Gecko/20111222 Thunderbird/9.0.1 MIME-Version: 1.0 To: "linux-rdma (linux-rdma@vger.kernel.org)" CC: Bart Van Assche , Vladimir Koushnir , Jim Mott Subject: [PATCH v2 opensm] Add support for synchronizing in memory files with storage Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-7.4 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP OpenSM produces certain in memory files critical to high availability operation. These include guid2lid, guid2mkey, and neighbors as well as the SA registration database (opensm-sa.dump). These in memory files can be synchronized with storage immediately rather than waiting for pdflush daemon to do this. This is done based on new fsync_high_avail_files option which defaults to true. In some embedded systems with flash based storage, this option might be set to false as tradeoff not to unduly affect OpenSM operation due to possible long delay. Pointed-out-by: Bart Van Assche in thread on "[PATCH opensm] Implement atomic update operation for sa_db_file" http://marc.info/?l=linux-rdma&m=138436562629008&w=2 Signed-off-by: Hal Rosenstock --- Change since v1: Added missing fflush call prior to fsync Pointed-out-by: Bart Van Assche -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/include/opensm/osm_db.h b/include/opensm/osm_db.h index 05332c0..e8860f3 100644 --- a/include/opensm/osm_db.h +++ b/include/opensm/osm_db.h @@ -288,7 +288,8 @@ int osm_db_clear(IN osm_db_domain_t * p_domain); * * SYNOPSIS */ -int osm_db_store(IN osm_db_domain_t * p_domain); +int osm_db_store(IN osm_db_domain_t * p_domain, + IN boolean_t fsync_high_avail_files); /* * PARAMETERS * @@ -296,6 +297,10 @@ int osm_db_store(IN osm_db_domain_t * p_domain); * [in] Pointer to the database domain object to restore from * persistent db * +* fsync_high_avail_files +* [in] Boolean that indicates whether or not to synchronize +* in-memory high availability files with storage +* * RETURN VALUES * 0 if successful 1 otherwize * diff --git a/include/opensm/osm_subnet.h b/include/opensm/osm_subnet.h index 19f2079..606c63e 100644 --- a/include/opensm/osm_subnet.h +++ b/include/opensm/osm_subnet.h @@ -328,6 +328,7 @@ typedef struct osm_subn_opt { boolean_t babbling_port_policy; boolean_t drop_event_subscriptions; boolean_t use_optimized_slvl; + boolean_t fsync_high_avail_files; osm_qos_options_t qos_options; osm_qos_options_t qos_ca_options; osm_qos_options_t qos_sw0_options; @@ -604,6 +605,10 @@ typedef struct osm_subn_opt { * Use optimized SLtoVLMappingTable programming if * device indicates it supports this. * +* fsync_high_avail_files +* Synchronize high availability in memory files +* with storage. +* * perfmgr * Enable or disable the performance manager * diff --git a/opensm/osm_db_files.c b/opensm/osm_db_files.c index cab7170..38ee9a9 100644 --- a/opensm/osm_db_files.c +++ b/opensm/osm_db_files.c @@ -48,6 +48,7 @@ #include #include #include +#include #include #define FILE_ID OSM_FILE_DB_FILES_C #include @@ -461,12 +462,13 @@ static int dump_tbl_entry(st_data_t key, st_data_t val, st_data_t arg) return ST_CONTINUE; } -int osm_db_store(IN osm_db_domain_t * p_domain) +int osm_db_store(IN osm_db_domain_t * p_domain, + IN boolean_t fsync_high_avail_files) { osm_log_t *p_log = p_domain->p_db->p_log; osm_db_domain_imp_t *p_domain_imp; FILE *p_file = NULL; - int status = 0; + int fd, status = 0; char *p_tmp_file_name = NULL; OSM_LOG_ENTER(p_log); @@ -494,6 +496,26 @@ int osm_db_store(IN osm_db_domain_t * p_domain) } st_foreach(p_domain_imp->p_hash, dump_tbl_entry, (st_data_t) p_file); + + if (fsync_high_avail_files) { + if (fflush(p_file) == 0) { + fd = fileno(p_file); + if (fd != -1) { + if (fsync(fd) == -1) + OSM_LOG(p_log, OSM_LOG_ERROR, + "ERR 6110: fsync() failed (%s) for %s\n", + strerror(errno), + p_domain_imp->file_name); + } else + OSM_LOG(p_log, OSM_LOG_ERROR, "ERR 6111: " + "fileno() failed for %s\n", + p_domain_imp->file_name); + } else + OSM_LOG(p_log, OSM_LOG_ERROR, "ERR 6112: " + "fflush() failed (%s) for %s\n", + strerror(errno), p_domain_imp->file_name); + } + fclose(p_file); status = rename(p_tmp_file_name, p_domain_imp->file_name); @@ -734,7 +756,7 @@ int main(int argc, char **argv) printf("key = %s val = %s\n", p_key, p_val); } } - if (osm_db_store(p_dbd)) + if (osm_db_store(p_dbd, FALSE)) printf("failed to store\n"); osm_db_destroy(&db); diff --git a/opensm/osm_lid_mgr.c b/opensm/osm_lid_mgr.c index f8a3739..3ba1a79 100644 --- a/opensm/osm_lid_mgr.c +++ b/opensm/osm_lid_mgr.c @@ -1233,7 +1233,7 @@ int osm_lid_mgr_process_subnet(IN osm_lid_mgr_t * p_mgr) } /* all ports */ /* store the guid to lid table in persistent db */ - osm_db_store(p_mgr->p_g2l); + osm_db_store(p_mgr->p_g2l, p_mgr->p_subn->opt.fsync_high_avail_files); CL_PLOCK_RELEASE(p_mgr->p_lock); diff --git a/opensm/osm_sa.c b/opensm/osm_sa.c index 4b15d39..ea4a4d0 100644 --- a/opensm/osm_sa.c +++ b/opensm/osm_sa.c @@ -49,6 +49,7 @@ #include #include #include +#include #include #include #include @@ -508,7 +509,7 @@ opensm_dump_to_file(osm_opensm_t * p_osm, const char *file_name, char path[1024]; char path_tmp[1032]; FILE *file; - int status = 0; + int fd, status = 0; snprintf(path, sizeof(path), "%s/%s", p_osm->subn.opt.dump_files_dir, file_name); @@ -527,6 +528,23 @@ opensm_dump_to_file(osm_opensm_t * p_osm, const char *file_name, dump_func(p_osm, file); + if (p_osm->subn.opt.fsync_high_avail_files) { + if (fflush(file) == 0) { + fd = fileno(file); + if (fd != -1) { + if (fsync(fd) == -1) + OSM_LOG(&p_osm->log, OSM_LOG_ERROR, + "ERR 4C08: fsync() failed (%s) for %s\n", + strerror(errno), path_tmp); + } else + OSM_LOG(&p_osm->log, OSM_LOG_ERROR, "ERR 4C09: " + "fileno() failed for %s\n", path_tmp); + } else + OSM_LOG(&p_osm->log, OSM_LOG_ERROR, "ERR 4C0A: " + "fflush() failed (%s) for %s\n", + strerror(errno), path_tmp); + } + fclose(file); status = rename(path_tmp, path); diff --git a/opensm/osm_state_mgr.c b/opensm/osm_state_mgr.c index 10ae17a..85b233d 100644 --- a/opensm/osm_state_mgr.c +++ b/opensm/osm_state_mgr.c @@ -1493,8 +1493,9 @@ repeat_discovery: osm_sm_signal(sm, OSM_SIGNAL_SWEEP); /* Write a new copy of our persistent guid2mkey database */ - osm_db_store(sm->p_subn->p_g2m); - osm_db_store(sm->p_subn->p_neighbor); + osm_db_store(sm->p_subn->p_g2m, sm->p_subn->opt.fsync_high_avail_files); + osm_db_store(sm->p_subn->p_neighbor, + sm->p_subn->opt.fsync_high_avail_files); } static void do_process_mgrp_queue(osm_sm_t * sm) diff --git a/opensm/osm_subnet.c b/opensm/osm_subnet.c index f7b6942..b69b54e 100644 --- a/opensm/osm_subnet.c +++ b/opensm/osm_subnet.c @@ -778,6 +778,7 @@ static const opt_rec_t opt_tbl[] = { { "babbling_port_policy", OPT_OFFSET(babbling_port_policy), opts_parse_boolean, NULL, 1 }, { "drop_event_subscriptions", OPT_OFFSET(drop_event_subscriptions), opts_parse_boolean, NULL, 1 }, { "use_optimized_slvl", OPT_OFFSET(use_optimized_slvl), opts_parse_boolean, NULL, 1 }, + { "fsync_high_avail_files", OPT_OFFSET(fsync_high_avail_files), opts_parse_boolean, NULL, 1 }, #ifdef ENABLE_OSM_PERF_MGR { "perfmgr", OPT_OFFSET(perfmgr), opts_parse_boolean, NULL, 0 }, { "perfmgr_redir", OPT_OFFSET(perfmgr_redir), opts_parse_boolean, NULL, 0 }, @@ -1483,6 +1484,7 @@ void osm_subn_set_default_opt(IN osm_subn_opt_t * p_opt) p_opt->babbling_port_policy = FALSE; p_opt->drop_event_subscriptions = FALSE; p_opt->use_optimized_slvl = FALSE; + p_opt->fsync_high_avail_files = TRUE; #ifdef ENABLE_OSM_PERF_MGR p_opt->perfmgr = FALSE; p_opt->perfmgr_redir = TRUE; @@ -2538,12 +2540,15 @@ int osm_subn_output_conf(FILE *out, IN osm_subn_opt_t * p_opts) "# Drop event subscriptions (InformInfo) if the port goes away\n" "drop_event_subscriptions %s\n\n" "# Use Optimized SLtoVLMapping programming if supported by device\n" - "use_optimized_slvl %s\n\n", + "use_optimized_slvl %s\n\n" + "# Sync in memory files used for high availability with storage\n" + "fsync_high_avail_files %s\n\n", p_opts->daemon ? "TRUE" : "FALSE", p_opts->sm_inactive ? "TRUE" : "FALSE", p_opts->babbling_port_policy ? "TRUE" : "FALSE", p_opts->drop_event_subscriptions ? "TRUE" : "FALSE", - p_opts->use_optimized_slvl ? "TRUE" : "FALSE"); + p_opts->use_optimized_slvl ? "TRUE" : "FALSE", + p_opts->fsync_high_avail_files ? "TRUE" : "FALSE"); #ifdef ENABLE_OSM_PERF_MGR fprintf(out,