From patchwork Tue Jan 19 16:46:53 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhen Ren X-Patchwork-Id: 8064471 Return-Path: X-Original-To: patchwork-ocfs2-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 51167BEEE5 for ; Tue, 19 Jan 2016 18:07:51 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 5677220501 for ; Tue, 19 Jan 2016 18:07:50 +0000 (UTC) X-Greylist: delayed 4703 seconds by postgrey-1.34 at mail.kernel.org; Tue, 19 Jan 2016 18:07:49 UTC Received: from aserp1050.oracle.com (aserp1050.oracle.com [141.146.126.70]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 21767204E3 for ; Tue, 19 Jan 2016 18:07:49 +0000 (UTC) Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) by aserp1050.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0JGnPeC018399 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Tue, 19 Jan 2016 16:49:25 GMT Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0JGlh7d018768 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Tue, 19 Jan 2016 16:47:44 GMT Received: from oss.oracle.com (oss-old-reserved.oracle.com [137.254.22.2]) by userv0022.oracle.com (8.14.4/8.13.8) with ESMTP id u0JGlbqY000380 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 19 Jan 2016 16:47:38 GMT Received: from localhost ([127.0.0.1] helo=lb-oss.oracle.com) by oss.oracle.com with esmtp (Exim 4.63) (envelope-from ) id 1aLZRR-0008A4-96; Tue, 19 Jan 2016 08:47:37 -0800 Received: from userv0022.oracle.com ([156.151.31.74]) by oss.oracle.com with esmtp (Exim 4.63) (envelope-from ) id 1aLZR2-00089F-Cb for ocfs2-devel@oss.oracle.com; Tue, 19 Jan 2016 08:47:12 -0800 Received: from aserp1020.oracle.com (aserp1020.oracle.com [141.146.126.67]) by userv0022.oracle.com (8.14.4/8.13.8) with ESMTP id u0JGlAHO031075 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL) for ; Tue, 19 Jan 2016 16:47:10 GMT Received: from userp2030.oracle.com (userp2030.oracle.com [156.151.31.89]) by aserp1020.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id u0JGl9AC022279 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 19 Jan 2016 16:47:10 GMT Received: from pps.filterd (userp2030.oracle.com [127.0.0.1]) by userp2030.oracle.com (8.15.0.59/8.15.0.59) with SMTP id u0JGjJdd002675 for ; Tue, 19 Jan 2016 16:47:09 GMT Received: from prv3-mh.provo.novell.com (prv3-mh.provo.novell.com [137.65.250.26]) by userp2030.oracle.com with ESMTP id 20ff6tjrap-1 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 19 Jan 2016 16:47:09 +0000 Received: from laptop.ha (prv-ext-foundry1int.gns.novell.com [137.65.251.240]) by prv3-mh.provo.novell.com with ESMTP (TLS encrypted); Tue, 19 Jan 2016 09:47:04 -0700 From: Eric Ren To: akpm@linux-foundation.org, mfasheh@suse.com Date: Wed, 20 Jan 2016 00:46:53 +0800 Message-Id: <1453222013-9425-1-git-send-email-zren@suse.com> X-Mailer: git-send-email 2.6.2 X-Proofpoint-SPF-Result: permerror X-Proofpoint-SPF-Record: v=spf1 ip4:137.65.0.0/16 ip4:151.155.28.0/17 ip4:149.44.0.0/16 ip4:147.2.0.0/16 ip4:164.99.0.0/16 ip4:130.57.0.0/16 ip4:192.31.114.0/24 ip4:195.135.221.0/24 ip4:195.135.220.0/24 ip4:69.7.179.0/24 include:_spf2.suse.com -all X-ServerName: prv3-mh.provo.novell.com X-Proofpoint-Virus-Version: vendor=nai engine=5800 definitions=8049 signatures=670681 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=2 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1507310008 definitions=main-1601190286 Cc: ocfs2-devel@oss.oracle.com Subject: [Ocfs2-devel] [PATCH] ocfs2: dlmglue: fix false deadlock caused by clearing UPCONVERT_FINISHING too early X-BeenThere: ocfs2-devel@oss.oracle.com X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ocfs2-devel-bounces@oss.oracle.com Errors-To: ocfs2-devel-bounces@oss.oracle.com X-Source-IP: userp1040.oracle.com [156.151.31.81] X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This problem was introduced by commit a19128260107f951d1b4c421cf98b92f8092b069. OCFS2_LOCK_UPCONVERT_FINISHING is set just before clearing OCFS2_LOCK_BUSY. This will prevent dc thread from downconverting immediately, and let mask-waiters in ->l_mask_waiters list whose requesting level is compatible with ->l_level to take the lock. But if we have two waiters in mw list, the first is to get EX lock, and the second is to to get PR lock. The first may fail to get lock and then clear UPCONVERT_FINISHING. It's too early to clear the flag because this second will be also queued again even if ->l_level is PR. As a result, nobody would kick up dc thread, leaving dlmglue a deadlock until another lockres relative thread wake it up. More specifically, for example: On node1, there is thread W1 keeping writing; on node2, there are thread R1 and R2 keeping reading; sure this 3 threads make IO on the same shared file. At a time, node2 is receiving ast(0=>3), followed immediately by a bast requesting EX lock on behave of node1. Then this may happen: node2: node1: l_level==3; R1(3); R2(3) l_level==3 R1(unlock); R1(3=>5, update atime) W1(3=>5) BAST R2(unlock); AST(3=>0) R2(0=>3) BAST AST(0=>3) set OCFS2_LOCK_UPCONVERT_FINISHING clear OCFS2_LOCK_BUSY W1(3=>5) BAST dc thread requeue=yes R1(clear OCFS2_LOCK_UPCONVERT_FINISHING,wait) R2(wait) ... dlmglue deadlock util dc thread woken up by others This fix is to clear OCFS2_LOCK_UPCONVERT_FINISHING util OCFS2_LOCK_BUSY has been cleared and every waiters has been looped. Signed-off-by: Eric Ren --- fs/ocfs2/dlmglue.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fs/ocfs2/dlmglue.c b/fs/ocfs2/dlmglue.c index f92612e..72f8b6c 100644 --- a/fs/ocfs2/dlmglue.c +++ b/fs/ocfs2/dlmglue.c @@ -824,6 +824,8 @@ static void lockres_clear_flags(struct ocfs2_lock_res *lockres, unsigned long clear) { lockres_set_flags(lockres, lockres->l_flags & ~clear); + if(clear & OCFS2_LOCK_BUSY) + lockres->l_flags &= ~OCFS2_LOCK_UPCONVERT_FINISHING; } static inline void ocfs2_generic_handle_downconvert_action(struct ocfs2_lock_res *lockres) @@ -1522,8 +1524,6 @@ update_holders: ret = 0; unlock: - lockres_clear_flags(lockres, OCFS2_LOCK_UPCONVERT_FINISHING); - spin_unlock_irqrestore(&lockres->l_lock, flags); out: /*