From patchwork Fri Feb 24 14:30:37 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Foster X-Patchwork-Id: 9590475 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 46526601AE for ; Fri, 24 Feb 2017 14:30:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4198028701 for ; Fri, 24 Feb 2017 14:30:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 35082287A4; Fri, 24 Feb 2017 14:30:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 547B22876C for ; Fri, 24 Feb 2017 14:30:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751390AbdBXOaj (ORCPT ); Fri, 24 Feb 2017 09:30:39 -0500 Received: from mx1.redhat.com ([209.132.183.28]:45844 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751208AbdBXOaj (ORCPT ); Fri, 24 Feb 2017 09:30:39 -0500 Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 6E6213B755; Fri, 24 Feb 2017 14:30:39 +0000 (UTC) Received: from bfoster.bfoster (dhcp-41-20.bos.redhat.com [10.18.41.20]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v1OEUccM019390; Fri, 24 Feb 2017 09:30:39 -0500 Received: by bfoster.bfoster (Postfix, from userid 1000) id DDDAC1202CD; Fri, 24 Feb 2017 09:30:37 -0500 (EST) Date: Fri, 24 Feb 2017 09:30:37 -0500 From: Brian Foster To: Mathias Troiden Cc: linux-xfs@vger.kernel.org Subject: Re: xfs_repair fails to recognize corruption reported by kernel - possible bug? Message-ID: <20170224143037.GD59560@bfoster.bfoster> References: <20170224123017.GA59560@bfoster.bfoster> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20170224123017.GA59560@bfoster.bfoster> User-Agent: Mutt/1.7.1 (2016-10-04) X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Fri, 24 Feb 2017 14:30:39 +0000 (UTC) Sender: linux-xfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Fri, Feb 24, 2017 at 07:30:18AM -0500, Brian Foster wrote: > On Thu, Feb 23, 2017 at 11:14:47PM +0300, Mathias Troiden wrote: > > Original topic: https://bbs.archlinux.org/viewtopic.php?pid=1692896 > > > > Hi list, > > > > My system fails to start login manager with following messages in journal: > > > > >kernel: ffff88040e8bc030: 58 67 db ca 2a 3a dd b8 00 00 00 00 00 00 00 00 Xg..*:.......... > > >kernel: XFS (sda1): Internal error xfs_iread at line 514 of file fs/xfs/libxfs/xfs_inode_buf.c. Caller xfs_iget+0x2b1/0x940 [xfs] > > >kernel: XFS (sda1): Corruption detected. Unmount and run xfs_repair > > >kernel: XFS (sda1): xfs_iread: validation failed for inode 34110192 failed > > >kernel: ffff88040e8bc000: 49 4e a1 ff 03 01 00 00 00 00 00 00 00 00 00 00 IN.............. > > >kernel: ffff88040e8bc010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................ > > >kernel: ffff88040e8bc020: 58 aa 04 b8 2e e3 65 3a 57 41 fe 12 00 00 00 00 X.....e:WA...... > > >kernel: ffff88040e8bc030: 58 67 db ca 2a 3a dd b8 00 00 00 00 00 00 00 00 Xg..*:.......... > > >kernel: XFS (sda1): Internal error xfs_iread at line 514 of file fs/xfs/libxfs/xfs_inode_buf.c. Caller xfs_iget+0x2b1/0x940 [xfs] > > >kernel: XFS (sda1): Corruption detected. Unmount and run xfs_repair > > > > > > and subsequent core dump of the login manager. > > > > What kernel and xfsprogs versions? Also, please provide 'xfs_info ' > output for the fs. > > From the output above, it looks like you could have a zero-sized > symlink, which triggers xfs_dinode_verify() failure. It's quite possible > I'm misreading the raw inode buffer output above too, however.. Did you > have any interesting "events" before this problem started to occur? For > example, a crash or hard reset, etc.? > > Could you run 'find -inum 34110192 -print' on the fs and report > the associated filename? You could try 'stat ' as well but I'm > guessing that's just going to report an error. > > Note that another way to get us details of the fs is to send an > xfs_metadump image. An md image skips all file data in the fs and > obfuscates metadata (such as filenames) such that no sensitive > information is shared. It simply provides a skeleton metadata image for > us to debug. To create an obfuscated metadump, run 'xfs_metadump -g > ,' compress the resulting image file and send it along > (feel free to send directly) or upload it somewhere. > After looking at a metadump, this is indeed a zero-sized symlink. The immediate fix here is probably to allow xfs_repair to detect this situation and recover, which most likely means clearing out the inode. Unfortunately, it's not clear how we got into this situation in the first place. I'm still curious if you've had any crash or reset events that might have required log recovery recently..? Regardless, you'll probably have to try something like the appended xfsprogs patch, which clears out the offending inode and means you'll have to recreate it manually to recover system functionality (Mathias has pointed out offline that the offending link is a standard /usr/lib/lib*.so symlink with a known target, so fortunately recovery should be simple). Brian --- 8< --- --- To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/repair/dinode.c b/repair/dinode.c index 8d01409..d664f87 100644 --- a/repair/dinode.c +++ b/repair/dinode.c @@ -1385,6 +1385,11 @@ process_symlink( return(1); } + if (be64_to_cpu(dino->di_size) == 0) { + do_warn(_("zero size symlink in inode %" PRIu64 "\n"), lino); + return 1; + } + /* * have to check symlink component by component. * get symlink contents into data area