From patchwork Tue Apr 2 06:18:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: sunnyZhang X-Patchwork-Id: 10881053 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4D6EF13B5 for ; Tue, 2 Apr 2019 06:20:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 220292868C for ; Tue, 2 Apr 2019 06:20:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 15985286A1; Tue, 2 Apr 2019 06:20:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 6D2442868C for ; Tue, 2 Apr 2019 06:20:28 +0000 (UTC) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x326J2Av141202; Tue, 2 Apr 2019 06:19:49 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : date : message-id : subject : list-id : list-unsubscribe : list-archive : list-post : list-help : list-subscribe : mime-version : content-type : content-transfer-encoding : sender; s=corp-2018-07-02; bh=I25sI/T9PvJJa3LPpPke8Y8lC76LXxcLD9LqQFevE8o=; b=OmAkbzSWGzu01afSZvY3J5JIooP0I1+rsMBMcls/7cpyfyUvOmHxpemlyHvKM5BbnNux EGfz7PHyseckyNIWLlBVt/pguuBwP8WU2Tc/QGeuM5mhWH5EGfVwOE5W+c7lqk3ItPmr eUmint0fs0kREv51qghTp/Gz3fbLnfBdAM9+jX2HV9HwgrMn718SCCugIeqJQSOSQWzF 8y5R8iS+2oIlZgm5oIS5xtny/cbdblJDZ/7RLwvhDGFcUSIjPq6Q8fpwlxF44t87JveG HeSUoA3LLAf/7CkoXy9s4r7NglYc6MNMvRX+NErQxzFe1oKa1X6TniwLVy2WZyMrVLcz mw== Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp2120.oracle.com with ESMTP id 2rj13q2wu6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 02 Apr 2019 06:19:49 +0000 Received: from oss.oracle.com (oss-old-reserved.oracle.com [137.254.22.2]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x326Jg19026296 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 2 Apr 2019 06:19:43 GMT Received: from localhost ([127.0.0.1] helo=lb-oss.oracle.com) by oss.oracle.com with esmtp (Exim 4.63) (envelope-from ) id 1hBClq-0001iU-QB; Mon, 01 Apr 2019 23:19:42 -0700 Received: from aserv0022.oracle.com ([141.146.126.234]) by oss.oracle.com with esmtp (Exim 4.63) (envelope-from ) id 1hBClB-0001h4-Bw for ocfs2-devel@oss.oracle.com; Mon, 01 Apr 2019 23:19:01 -0700 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x326J0EJ010494 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Apr 2019 06:19:00 GMT Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x326Ixhv024314; Tue, 2 Apr 2019 06:19:00 GMT Received: from ol6u9ext3.cn.oracle.com (/10.182.70.234) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 01 Apr 2019 23:18:59 -0700 From: Shuning Zhang To: ocfs2-devel@oss.oracle.com Date: Tue, 2 Apr 2019 14:18:39 +0800 Message-Id: <1554185919-3010-1-git-send-email-sunny.s.zhang@oracle.com> X-Mailer: git-send-email 1.7.1 Subject: [Ocfs2-devel] [PATCH v2] ocfs2: fix ocfs2 read inode data panic in ocfs2_iget X-BeenThere: ocfs2-devel@oss.oracle.com X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ocfs2-devel-bounces@oss.oracle.com Errors-To: ocfs2-devel-bounces@oss.oracle.com X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9214 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904020046 X-Virus-Scanned: ClamAV using ClamSMTP In some cases, the function ocfs2_iget read the data of inode, which has been deleted for some reason. That will make the system panic. So We should judge whether this inode has been deleted, and tell the caller that the inode is a bad inode. For example, the ocfs2 is used as the backed of nfs, and the client is nfsv3. This issue can be reproduced by the following steps. on the nfs server side, ..../patha/pathb Step 1: The process A was scheduled before calling the function fh_verify. Step 2: The process B is removing the 'pathb', and just completed the call to function dput. Then the dentry of 'pathb' has been deleted from the dcache, and all ancestors have been deleted also. The relationship of dentry and inode was deleted through the function hlist_del_init. The following is the call stack. dentry_iput->hlist_del_init(&dentry->d_u.d_alias) At this time, the inode is still in the dcache. Step 3: The process A call the function ocfs2_get_dentry, which get the inode from dcache. Then the refcount of inode is 1. The following is the call stack. nfsd3_proc_getacl->fh_verify->exportfs_decode_fh->fh_to_dentry(ocfs2_get_dentry) Step 4: Dirty pages are flushed by bdi threads. So the inode of 'patha' is evicted, and this directory was deleted. But the inode of 'pathb' can't be evicted, because the refcount of the inode was 1. Step 5: The process A keep running, and call the function reconnect_path(in exportfs_decode_fh), which call function ocfs2_get_parent of ocfs2. Get the block number of parent directory(patha) by the name of ... Then read the data from disk by the block number. But this inode has been deleted, so the system panic. Process A Process B 1. in nfsd3_proc_getacl | 2. | dput 3. fh_to_dentry(ocfs2_get_dentry) | 4. bdi flush dirty cache | 5. ocfs2_iget | [283465.542049] OCFS2: ERROR (device sdp): ocfs2_validate_inode_block: Invalid dinode #580640: OCFS2_VALID_FL not set [283465.545490] Kernel panic - not syncing: OCFS2: (device sdp): panic forced after error [283465.546889] CPU: 5 PID: 12416 Comm: nfsd Tainted: G W 4.1.12-124.18.6.el6uek.bug28762940v3.x86_64 #2 [283465.548382] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015 [283465.549657] 0000000000000000 ffff8800a56fb7b8 ffffffff816e839c ffffffffa0514758 [283465.550392] 000000000008dc20 ffff8800a56fb838 ffffffff816e62d3 0000000000000008 [283465.551056] ffff880000000010 ffff8800a56fb848 ffff8800a56fb7e8 ffff88005df9f000 [283465.551710] Call Trace: [283465.552516] [] dump_stack+0x63/0x81 [283465.553291] [] panic+0xcb/0x21b [283465.554037] [] ocfs2_handle_error+0xf0/0xf0 [ocfs2] [283465.554882] [] __ocfs2_error+0x67/0x70 [ocfs2] [283465.555768] [] ocfs2_validate_inode_block+0x229/0x230 [ocfs2] [283465.556683] [] ocfs2_read_blocks+0x46c/0x7b0 [ocfs2] [283465.557408] [] ? ocfs2_inode_cache_io_unlock+0x20/0x20 [ocfs2] [283465.557973] [] ocfs2_read_inode_block_full+0x3b/0x60 [ocfs2] [283465.558525] [] ocfs2_iget+0x4aa/0x880 [ocfs2] [283465.559082] [] ocfs2_get_parent+0x9e/0x220 [ocfs2] [283465.559622] [] reconnect_path+0xb5/0x300 [283465.560156] [] exportfs_decode_fh+0xf6/0x2b0 [283465.560708] [] ? nfsd_proc_getattr+0xa0/0xa0 [nfsd] [283465.561262] [] ? prepare_creds+0x26/0x110 [283465.561932] [] fh_verify+0x350/0x660 [nfsd] [283465.562862] [] ? nfsd_cache_lookup+0x44/0x630 [nfsd] [283465.563697] [] nfsd3_proc_getattr+0x69/0xf0 [nfsd] [283465.564510] [] nfsd_dispatch+0xe0/0x290 [nfsd] [283465.565358] [] ? svc_tcp_adjust_wspace+0x12/0x30 [sunrpc] [283465.566272] [] svc_process_common+0x412/0x6a0 [sunrpc] [283465.567155] [] svc_process+0x123/0x210 [sunrpc] [283465.568020] [] nfsd+0xff/0x170 [nfsd] [283465.568962] [] ? nfsd_destroy+0x80/0x80 [nfsd] [283465.570112] [] kthread+0xcb/0xf0 [283465.571099] [] ? kthread_create_on_node+0x180/0x180 [283465.572114] [] ret_from_fork+0x58/0x90 [283465.573156] [] ? kthread_create_on_node+0x180/0x180 Changes in v2: - add description of reporducing this issue - use ocfs2_test_inode_bit instead of i_dtime Signed-off-by: Shuning Zhang Reviewed-by: Joseph Qi --- fs/ocfs2/export.c | 30 +++++++++++++++++++++++++++++- 1 files changed, 29 insertions(+), 1 deletions(-) diff --git a/fs/ocfs2/export.c b/fs/ocfs2/export.c index 827fc98..da17aa6 100644 --- a/fs/ocfs2/export.c +++ b/fs/ocfs2/export.c @@ -148,16 +148,24 @@ static struct dentry *ocfs2_get_parent(struct dentry *child) u64 blkno; struct dentry *parent; struct inode *dir = d_inode(child); + int set; trace_ocfs2_get_parent(child, child->d_name.len, child->d_name.name, (unsigned long long)OCFS2_I(dir)->ip_blkno); + status = ocfs2_nfs_sync_lock(OCFS2_SB(dir->i_sb), 1); + if (status < 0) { + mlog(ML_ERROR, "getting nfs sync lock(EX) failed %d\n", status); + parent = ERR_PTR(status); + goto bail; + } + status = ocfs2_inode_lock(dir, NULL, 0); if (status < 0) { if (status != -ENOENT) mlog_errno(status); parent = ERR_PTR(status); - goto bail; + goto unlock_nfs_sync; } status = ocfs2_lookup_ino_from_name(dir, "..", 2, &blkno); @@ -166,11 +174,31 @@ static struct dentry *ocfs2_get_parent(struct dentry *child) goto bail_unlock; } + status = ocfs2_test_inode_bit(OCFS2_SB(dir->i_sb), blkno, &set); + if (status < 0) { + if (status == -EINVAL) { + status = -ESTALE; + } else + mlog(ML_ERROR, "test inode bit failed %d\n", status); + parent = ERR_PTR(status); + goto bail_unlock; + } + + trace_ocfs2_get_dentry_test_bit(status, set); + if (!set) { + status = -ESTALE; + parent = ERR_PTR(status); + goto bail_unlock; + } + parent = d_obtain_alias(ocfs2_iget(OCFS2_SB(dir->i_sb), blkno, 0, 0)); bail_unlock: ocfs2_inode_unlock(dir, 0); +unlock_nfs_sync: + ocfs2_nfs_sync_unlock(OCFS2_SB(dir->i_sb), 1); + bail: trace_ocfs2_get_parent_end(parent);