From patchwork Mon Nov 6 01:15:25 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: piaojun X-Patchwork-Id: 10042461 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 8B65E604D3 for ; Mon, 6 Nov 2017 01:20:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8D9AD2929C for ; Mon, 6 Nov 2017 01:20:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 816FC292A0; Mon, 6 Nov 2017 01:20:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id F17612929C for ; Mon, 6 Nov 2017 01:20:32 +0000 (UTC) Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id vA61KEp9016649 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 6 Nov 2017 01:20:14 GMT Received: from oss.oracle.com (oss-old-reserved.oracle.com [137.254.22.2]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id vA61KAkU002924 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 6 Nov 2017 01:20:10 GMT Received: from localhost ([127.0.0.1] helo=lb-oss.oracle.com) by oss.oracle.com with esmtp (Exim 4.63) (envelope-from ) id 1eBW5C-00068g-7a; Sun, 05 Nov 2017 17:20:10 -0800 Received: from aserv0021.oracle.com ([141.146.126.233]) by oss.oracle.com with esmtp (Exim 4.63) (envelope-from ) id 1eBW4i-00062a-Kd for ocfs2-devel@oss.oracle.com; Sun, 05 Nov 2017 17:19:42 -0800 Received: from userp2040.oracle.com (userp2040.oracle.com [156.151.31.90]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id vA61JeG2032408 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO) for ; Mon, 6 Nov 2017 01:19:40 GMT Received: from pps.filterd (userp2040.oracle.com [127.0.0.1]) by userp2040.oracle.com (8.16.0.21/8.16.0.21) with SMTP id vA61HZWG041443 for ; Mon, 6 Nov 2017 01:19:40 GMT Authentication-Results: oracle.com; spf=pass smtp.mailfrom=piaojun@huawei.com Received: from szxga04-in.huawei.com (szxga04-in.huawei.com [45.249.212.190]) by userp2040.oracle.com with ESMTP id 2e1gycph6b-1 (version=TLSv1 cipher=RC4-SHA bits=128 verify=NOT) for ; Mon, 06 Nov 2017 01:19:39 +0000 Received: from 172.30.72.58 (EHLO DGGEMS403-HUB.china.huawei.com) ([172.30.72.58]) by dggrg04-dlp.huawei.com (MOS 4.4.6-GA FastPath queued) with ESMTP id DKH99829; Mon, 06 Nov 2017 09:15:36 +0800 (CST) Received: from [10.177.253.249] (10.177.253.249) by smtp.huawei.com (10.3.19.203) with Microsoft SMTP Server id 14.3.361.1; Mon, 6 Nov 2017 09:15:28 +0800 To: "akpm@linux-foundation.org" , Mark Fasheh , Joel Becker , "srinivas.eeda@oracle.com" From: piaojun Message-ID: <59FFB7AD.90108@huawei.com> Date: Mon, 6 Nov 2017 09:15:25 +0800 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 X-Originating-IP: [10.177.253.249] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A090203.59FFB7B9.00E4, ss=1, re=0.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 0ec418e183901cbdb4517e1614e895fe X-CLX-Shades: MLX X-CLX-Response: 1TFkXGx0YEQpMehcZGhEKWU0XZ2ZyEQpZSRcacRoQGncGGx8YcRkYEBp3Bhg aBhoRClleF2hueREKSUYXRVhLSUZPdVpYRU5fSV5DRUQZdU9LEQpDThcYf29Gb3J4GUR+S0Jybk Z7T3hOUk5rTXpbTmheHWd1ZREKWFwXHwQaBBsbEwcbSBpOGE5LTwUbGgQbGhoEHhIEHxAbHhofG hEKXlkXeGR8G20RCk1cFxgaHxEKTFoXaGlCTXsRCk1OF2gRCkNaFx4fBBgeEwQYGxgEGxMaEQpC XhcbEQpEXhccEQpESRcfEQpCRhdsQlhpfEJGcHkaHxEKQlwXGhEKQkUXZlxse3BkYnoSfEMRCkJ OF2xCSFlTGk1leHgdEQpCTBdvSxkSEkRZeVsbXxEKQmwXYwVCUmZAYl5ae1IRCkJAF2hIcHxJe3 59HmtBEQpCWBdifW95AU8YGXBwexEKTV4XGxEKWlgXGxEKcGgXZWJrWV5rHE8FSEAQGRoRCnBoF 2xQWm4SAX8cfxNjEBkaEQpwaBdueHx6QQVEZBh5GBAZGhEKcGgXbktaS11kSWdZHFMQGRoRCnBo F29mZ2lbG21aGRpFEBkaEQpwbBdhSXlDenNJeGdsYhAZGhEKbX4XGxEKWE0XSxEg X-PDR: PASS X-ServerName: szxga04-in.huawei.com X-Proofpoint-SPF-Result: pass X-Proofpoint-SPF-Record: v=spf1 ip4:45.249.212.32 ip4:45.249.212.35 ip4:119.145.14.93 ip4:58.251.152.93 ip4:194.213.3.17 ip4:206.16.17.72 ip4:45.249.212.255 ip4:45.249.212.187/29 ip4:45.249.212.191 ~all X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=8706 signatures=668613 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=30 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=172 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000 definitions=main-1711060016 Cc: "ocfs2-devel@oss.oracle.com" Subject: [Ocfs2-devel] [PATCH v3] ocfs2/dlm: wait for dlm recovery done when migrating all lockres X-BeenThere: ocfs2-devel@oss.oracle.com X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: ocfs2-devel-bounces@oss.oracle.com Errors-To: ocfs2-devel-bounces@oss.oracle.com X-Source-IP: aserv0021.oracle.com [141.146.126.233] X-Virus-Scanned: ClamAV using ClamSMTP wait for dlm recovery done when migrating all lockres in case of new lockres to be left after leaving dlm domain. NodeA NodeB NodeC umount and migrate all lockres node down do recovery for NodeB and collect a new lockres form other live nodes leave domain but the new lockres remains mount and join domain request for the owner of the new lockres, but all the other nodes said 'NO', so NodeC decide to the owner, and send do assert msg to other nodes. other nodes receive the msg and found two masters exist. at last cause BUG in dlm_assert_master_handler() -->BUG(); Fixes: bc9838c4d44a ("dlm: allow dlm do recovery during shutdown") Signed-off-by: Jun Piao Reviewed-by: Alex Chen Reviewed-by: Yiwen Jiang --- fs/ocfs2/dlm/dlmcommon.h | 1 + fs/ocfs2/dlm/dlmdomain.c | 14 ++++++++++++++ fs/ocfs2/dlm/dlmrecovery.c | 13 ++++++++++--- 3 files changed, 25 insertions(+), 3 deletions(-) diff --git a/fs/ocfs2/dlm/dlmcommon.h b/fs/ocfs2/dlm/dlmcommon.h index e9f3705..999ab7d 100644 --- a/fs/ocfs2/dlm/dlmcommon.h +++ b/fs/ocfs2/dlm/dlmcommon.h @@ -140,6 +140,7 @@ struct dlm_ctxt u8 node_num; u32 key; u8 joining_node; + u8 migrate_done; /* set to 1 means node has migrated all lockres */ wait_queue_head_t dlm_join_events; unsigned long live_nodes_map[BITS_TO_LONGS(O2NM_MAX_NODES)]; unsigned long domain_map[BITS_TO_LONGS(O2NM_MAX_NODES)]; diff --git a/fs/ocfs2/dlm/dlmdomain.c b/fs/ocfs2/dlm/dlmdomain.c index e1fea14..98a8f56 100644 --- a/fs/ocfs2/dlm/dlmdomain.c +++ b/fs/ocfs2/dlm/dlmdomain.c @@ -461,6 +461,18 @@ static int dlm_migrate_all_locks(struct dlm_ctxt *dlm) cond_resched_lock(&dlm->spinlock); num += n; } + + if (!num) { + if (dlm->reco.state & DLM_RECO_STATE_ACTIVE) { + mlog(0, "%s: perhaps there are more lock resources need to " + "be migrated after dlm recovery\n", dlm->name); + ret = -EAGAIN; + } else { + mlog(0, "%s: we won't do dlm recovery after migrating all lockres", + dlm->name); + dlm->migrate_done = 1; + } + } spin_unlock(&dlm->spinlock); wake_up(&dlm->dlm_thread_wq); @@ -2052,6 +2064,8 @@ static struct dlm_ctxt *dlm_alloc_ctxt(const char *domain, dlm->joining_node = DLM_LOCK_RES_OWNER_UNKNOWN; init_waitqueue_head(&dlm->dlm_join_events); + dlm->migrate_done = 0; + dlm->reco.new_master = O2NM_INVALID_NODE_NUM; dlm->reco.dead_node = O2NM_INVALID_NODE_NUM; diff --git a/fs/ocfs2/dlm/dlmrecovery.c b/fs/ocfs2/dlm/dlmrecovery.c index 74407c6..c4cf682 100644 --- a/fs/ocfs2/dlm/dlmrecovery.c +++ b/fs/ocfs2/dlm/dlmrecovery.c @@ -423,12 +423,11 @@ void dlm_wait_for_recovery(struct dlm_ctxt *dlm) static void dlm_begin_recovery(struct dlm_ctxt *dlm) { - spin_lock(&dlm->spinlock); + assert_spin_locked(&dlm->spinlock); BUG_ON(dlm->reco.state & DLM_RECO_STATE_ACTIVE); printk(KERN_NOTICE "o2dlm: Begin recovery on domain %s for node %u\n", dlm->name, dlm->reco.dead_node); dlm->reco.state |= DLM_RECO_STATE_ACTIVE; - spin_unlock(&dlm->spinlock); } static void dlm_end_recovery(struct dlm_ctxt *dlm) @@ -456,6 +455,13 @@ static int dlm_do_recovery(struct dlm_ctxt *dlm) spin_lock(&dlm->spinlock); + if (dlm->migrate_done) { + mlog(0, "%s: no need do recovery after migrating all lockres\n", + dlm->name); + spin_unlock(&dlm->spinlock); + return 0; + } + /* check to see if the new master has died */ if (dlm->reco.new_master != O2NM_INVALID_NODE_NUM && test_bit(dlm->reco.new_master, dlm->recovery_map)) { @@ -490,12 +496,13 @@ static int dlm_do_recovery(struct dlm_ctxt *dlm) mlog(0, "%s(%d):recovery thread found node %u in the recovery map!\n", dlm->name, task_pid_nr(dlm->dlm_reco_thread_task), dlm->reco.dead_node); - spin_unlock(&dlm->spinlock); /* take write barrier */ /* (stops the list reshuffling thread, proxy ast handling) */ dlm_begin_recovery(dlm); + spin_unlock(&dlm->spinlock); + if (dlm->reco.new_master == dlm->node_num) goto master_here;