From patchwork Fri May 10 07:41:02 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Guozhonghua X-Patchwork-Id: 2548551 Return-Path: X-Original-To: patchwork-ocfs2-devel@patchwork.kernel.org Delivered-To: patchwork-process-083081@patchwork2.kernel.org Received: from aserp1040.oracle.com (aserp1040.oracle.com [141.146.126.69]) by patchwork2.kernel.org (Postfix) with ESMTP id 1C480DF2A2 for ; Fri, 10 May 2013 07:42:06 +0000 (UTC) Received: from acsinet22.oracle.com (acsinet22.oracle.com [141.146.126.238]) by aserp1040.oracle.com (Sentrion-MTA-4.3.1/Sentrion-MTA-4.3.1) with ESMTP id r4A7fRc0028781 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 10 May 2013 07:41:28 GMT Received: from oss.oracle.com (oss-external.oracle.com [137.254.96.51]) by acsinet22.oracle.com (8.14.4+Sun/8.14.4) with ESMTP id r4A7fPcc019777 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 10 May 2013 07:41:25 GMT Received: from localhost ([127.0.0.1] helo=oss.oracle.com) by oss.oracle.com with esmtp (Exim 4.63) (envelope-from ) id 1UahxF-0007mJ-1Y; Fri, 10 May 2013 00:41:25 -0700 Received: from acsinet22.oracle.com ([141.146.126.238]) by oss.oracle.com with esmtp (Exim 4.63) (envelope-from ) id 1Uahwy-0007lk-VU; Fri, 10 May 2013 00:41:08 -0700 Received: from aserp1020.oracle.com (aserp1020.oracle.com [141.146.126.67]) by acsinet22.oracle.com (8.14.4+Sun/8.14.4) with ESMTP id r4A7f81l019313 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 10 May 2013 07:41:08 GMT Received: from h3cedge04-ex.h3c.com (h3c.com [60.191.123.50] (may be forged)) by aserp1020.oracle.com (Sentrion-MTA-4.3.1/Sentrion-MTA-4.3.1) with ESMTP id r4A7f6Z0027466; Fri, 10 May 2013 07:41:06 GMT Received: from H3CHUB02-EX.srv.huawei-3com.com (10.63.16.140) by h3cedge04-ex.h3c.com (172.25.12.84) with Microsoft SMTP Server (TLS) id 14.1.218.12; Fri, 10 May 2013 15:41:04 +0800 Received: from H3CHUB04-EX.srv.huawei-3com.com (10.63.20.170) by H3CHUB02-EX.srv.huawei-3com.com (10.63.16.140) with Microsoft SMTP Server (TLS) id 14.1.218.12; Fri, 10 May 2013 15:41:04 +0800 Received: from H3CMLB02-EX.srv.huawei-3com.com ([fe80::399d:90f6:c5bb:9cc0]) by H3CHUB04-EX.srv.huawei-3com.com ([fe80::7967:781a:df8b:b87f%11]) with mapi id 14.01.0218.012; Fri, 10 May 2013 15:41:03 +0800 From: Guozhonghua To: Sunil Mushran Thread-Topic: [Ocfs2-devel] Patch request reviews, for node reconnecting with other nodes whose node number is little than local, thanks a lot. Thread-Index: Ac5LlctjEUD/IL+DSnGzWx7AK0VTXAA/gwgAAAC8gIAALmxmYA== Date: Fri, 10 May 2013 07:41:02 +0000 Message-ID: <71604351584F6A4EBAE558C676F37CA417BD79D5@H3CMLB02-EX.srv.huawei-3com.com> References: <71604351584F6A4EBAE558C676F37CA417BD78AA@H3CMLB02-EX.srv.huawei-3com.com> In-Reply-To: Accept-Language: zh-CN, en-US Content-Language: zh-CN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.153.28.108] MIME-Version: 1.0 X-Flow-Control-Info: class=Pass-to-MM reputation=ipRisk-All ip=60.191.123.50 ct-class=T1 ct-vol1=0 ct-vol2=4 ct-vol3=3 ct-risk=42 ct-spam1=0 ct-spam2=0 ct-bulk=100 rcpts=2 size=26162 X-Sendmail-CM-Score: 0.00% X-Sendmail-CM-Analysis: v=2.1 cv=XeuwkuJ5 c=1 sm=1 tr=0 a=hiO3Mi/nOff1JfvnnVpamA==:117 a=hiO3Mi/nOff1JfvnnVpamA==:17 a=O2wkX6l01XoA:10 a=ynFjR5-sq1AA:10 a=Pkar6KgMEUUA:10 a=xqWC_Br6kY4A:10 a=UtV5FjuMAAAA:8 a=slbrsl1HAAAA:8 a=XW4Y3ujQbNsA:10 a=pGLkceISAAAA:8 a=y PCof4ZbAAAA:8 a=2eWhy7yqWAE_NZqmhesA:9 a=5rVHaI3lgR4dja2X:21 a=3TeBUTDK_M7LqMFV:21 a=QEXdDO2ut3YA:10 a=MSl-tDqOz04A:10 a=UrVzQrUuqIYA:10 a=dFmKiUmSU1-1IkgxJGQA:9 a=CQbclHmeMyJSp4CR:21 a=PABhoW6RmtwL_j5p:21 a=TwojkqaIKODa57zw:21 a=UiCQ7L4-1S4A:10 a=_W_S_ 7VecoQA:10 a=frz4AuCg-hUA:10 a=tXsnliwV7b4A:10 X-Sendmail-CT-Classification: not spam X-Sendmail-CT-RefID: str=0001.0A090202.518CA494.001F:SCFSTAT4330676, ss=1, re=-4.101, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0 Cc: Changlimin , "ocfs2-users@oss.oracle.com" , "ocfs2-devel@oss.oracle.com" Subject: [Ocfs2-devel] =?utf-8?b?562U5aSNOiAgUGF0Y2ggcmVxdWVzdCByZXZpZXdz?= =?utf-8?q?=2C_for_node_reconnecting_with_other_nodes_whose_node_number_is?= =?utf-8?q?_little_than_local=2C_thanks_a_lot=2E?= X-BeenThere: ocfs2-devel@oss.oracle.com X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: ocfs2-devel-bounces@oss.oracle.com Errors-To: ocfs2-devel-bounces@oss.oracle.com X-Source-IP: acsinet22.oracle.com [141.146.126.238] Yeah, It is better not disconnect the TCP source channel than reconnecting. During the time between disconnection and reconnection, the dlm may be blocked, and the nodes may hangs with each other, waiting for dlms. Is it correct as the change below? Disabling the shutdown of the TCP. static void o2net_idle_timer(unsigned long data) { ……………………………………. printk(KERN_NOTICE "o2net: Connection to " SC_NODEF_FMT " has been " "idle for %lu.%lu secs, shutting it down.\n", SC_NODEF_ARGS(sc), msecs / 1000, msecs % 1000); /* o2net_sc_queue_work(sc, &sc->sc_shutdown_work); */ } Thanks a lot. ???: Sunil Mushran [mailto:sunil.mushran@gmail.com] ????: 2013?5?10? 1:23 ???: guozhonghua 02084 ??: ocfs2-devel@oss.oracle.com; changlimin 00148 ??: Re: [Ocfs2-devel] Patch request reviews, for node reconnecting with other nodes whose node number is little than local, thanks a lot. Resending as my reply bounced. On Thu, May 9, 2013 at 10:01 AM, Sunil Mushran > wrote: A better fix is to _not_ disconnect on o2net timeout once a connection has been cleanly established. Only disconnect on o2hb timeout. The reconnects are a problem as we could lose packets and not be aware of it leading to o2dlm hangs. IOW, this patch looks to be papering over one specific problem and does not fix the underlying issue. On Tue, May 7, 2013 at 7:43 PM, Guozhonghua > wrote: Hi, everyone, I had have a test with eight nodes and find one issue. The Linux kernel version is 3.2.40. As I migrate processes from one node to another, those processes is open the files on the OCFS2 storage. Sometime one node shutdown TCP connection with that node whose node number is larger because long time without any message from it. As the TCP connection shutdown, the node whose number larger did not restart connection to the node, whose number is little and shutdown the TCP connection. So I review the code of the cluster and find it may be a bug. I changed it and have a test. Is there anybody having time to view and make sure that those changes is correct? Thanks a lot. The diff file is as below, of the file is /cluster/tcp.c: root@gzh-dev:/home/dev/test_replace/ocfs2_ko# diff -pu ocfs2-ko-3.2-compare/cluster/tcp.c ocfs2-ko-3.2/cluster/tcp.c spin_lock(&nn->nn_lock); if (!nn->nn_sc_valid) { + /** trigger reconnect with other nodes whose node number is little than local + * while they are still able to access the storage + */ + atomic_set(&nn->nn_timeout, 1); + printk(KERN_NOTICE "o2net: No connection established with " "node %u after %u.%u seconds, giving up.\n", o2net_num_from_nn(nn), ------------------------------------------------------------------------------------------------------------------------------------- ???????????????????????????????????????? ???????????????????????????????????????? ???????????????????????????????????????? ??? This e-mail and its attachments contain confidential information from H3C, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! --- ocfs2-ko-3.2-compare/cluster/tcp.c 2012-10-29 19:33:19.534200000 +0800 +++ ocfs2-ko-3.2/cluster/tcp.c 2013-05-08 09:33:16.386277310 +0800 @@ -1699,6 +1698,10 @@ static void o2net_start_connect(struct w if (ret == -EINPROGRESS) ret = 0; + /** Reset the timeout with 0 to avoid connection again */ + if (ret == 0) { + atomic_set(&nn->nn_timeout, 0); + } out: if (ret) { printk(KERN_NOTICE "o2net: Connect attempt to " SC_NODEF_FMT @@ -1725,6 +1728,11 @@ static void o2net_connect_expired(struct