From patchwork Tue Jan 5 15:47:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Russell King (Oracle)" X-Patchwork-Id: 11999623 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1A0DC433E0 for ; Tue, 5 Jan 2021 15:49:50 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8D8442229C for ; Tue, 5 Jan 2021 15:49:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8D8442229C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=armlinux.org.uk Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:Message-ID:Subject:To:From:Date: Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender :Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=SlNh99xiovubs/HKI5EH9RP3RkN5ExnyFaVsphAY1qw=; b=bB52kSK1QityfQx/HRQQGUkZEz TyE2466vXf5E9o5t3neilwO1tU5YZGFgHasbNoVDFjK7q93CXugJegnBtEfDECp10hJxxTPmzxJ5b U10/wkElRqjNzcsq96oFjiIOEnz6XxufdzPTvA0LA7W2fSZY04OAkGG1yAYO4QpcvLlaYOtN5+mGA DtE13tsjBHVZcZ2WBmk7O/uSPMlgV8Bycu8ALfMYe23pnTaP67CM01r7JWBCWz60jDriXboDGEKJP 8ULYGX0V6dwofAbXH/d6gog+ysnuD19tmLIJ3gN26QQmDrJKSrgbPd6V7qtE5+lzIU6ebu7K0AiYY pjyCqVMw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kwoYa-0004Yc-BZ; Tue, 05 Jan 2021 15:47:36 +0000 Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kwoYX-0004Xm-Rj for linux-arm-kernel@lists.infradead.org; Tue, 05 Jan 2021 15:47:34 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Sender:Content-Type:MIME-Version: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=RFRsY6eYacOkuoBfCDxrzCNNPunDtJE052SSVHEGFv4=; b=rkuqa+ijBqIxqYg8vNP41CFJD rq3l3JT7T/mFkKlwCPk0y/DBeG9oXDhaCFgutY4E0kWdpRQsYJaU2BBQlZRUL2QeFATWKzQJGvlbS a6AyPxolGcIW3+RS4LtMfeAPgaQbXO8dGsubS+2yiLiDin2I4TLt/nZgOnslX5dZm9m4pnnyYUUbe jGPA/Z8MwwrnJabg0c55UYKHx583LU7BfPQgtDWkV8nQSxkycqzDWNl21fw3THGeMlY6cGz9SnuEW OUUxgnr81WXLA4dbveniAZvscYx7QvrN/BS2PQguefOe8k8yDjdpArt//otkL+SBXtfjmiQffx1fC MMUo27clg==; Received: from shell.armlinux.org.uk ([fd8f:7570:feb6:1:5054:ff:fe00:4ec]:45104) by pandora.armlinux.org.uk with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kwoYT-0000tT-Ag; Tue, 05 Jan 2021 15:47:29 +0000 Received: from linux by shell.armlinux.org.uk with local (Exim 4.92) (envelope-from ) id 1kwoYQ-0007qC-97; Tue, 05 Jan 2021 15:47:26 +0000 Date: Tue, 5 Jan 2021 15:47:26 +0000 From: Russell King - ARM Linux admin To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: Aarch64 EXT4FS inode checksum failures - seems to be weak memory ordering issues Message-ID: <20210105154726.GD1551@shell.armlinux.org.uk> MIME-Version: 1.0 Content-Disposition: inline User-Agent: Mutt/1.10.1 (2018-07-13) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210105_104733_925870_AECA3CD9 X-CRM114-Status: GOOD ( 30.18 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Andreas Dilger , linux-ext4@vger.kernel.org, Theodore Ts'o , Will Deacon Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi, This is an update on where I am with this long standing issue at the current time. Since 5.4, I have been struggling with several of my ARM64 systems, of different SoC vendors and differing filesystem media, were sporadically reporting inode checksum failures on their root filesystems. The time taken to report this has been anything between a few hours and three months of uptime, making the problem unrealistic to bisect. The issue was first seen on my SolidRun Clearfog CX LX2160A based system, but was also subsequently noticed on my Armada 8040 based systems running kernels 5.4 and later. Kernel 5.2 has proven stable with 566 days of uptime with no issue. It has taken a long time to get debugging in place to see what is going on - and this is currently detailed on the front page of www.armlinux.org.uk right now, which has formed a blog of this problem - since almost no one has taken any interest in it. However, over the last couple of days, a way to reproduce it has been found, at least for the LX2160A based system. Power down, leave the machine powered off for some time. Power up, log in and run: while :; do sleep 5; find /var /usr /bin /sbin -type f -print0 | \ xargs -0 md5sum >/dev/null; done Within a few minutes it seems to have spat out an inode checksum failure if the problem exists. However, testing for the problem _not_ existing is quite difficult - just because it doesn't appear in the first few minutes does not mean it has been solved - see above where it can take three months. However, evidence is currently pointing towards commit 22ec71615d82 ("arm64: io: Relax implicit barriers in default I/O accessors") having revealed this problem. Will is very certain that this change is correct, and we feel that it may have exposed some other issue in the Aarch64 code. Further attempts seem to suggest that the problem is specifically the barrier in __iormb(). Leaving __iowmb() untouched, and changing the barrier in __iormb() from dma_rmb() to rmb() _appears_ to result in the problem disappearing. "Appears" is stressed because further testing is needed - and that is probably going to take many months before we know for certain. However, this suggests that there is a memory ordering bug with aarch64 somewhere. Will can follow up with his own thoughts to this email. We don't know if it is: - the kernel. - the Cortex A72. - the Cache coherent interconnect. I don't think it's the CCI, as I believe the Armada 8040 uses Marvell's own IP for that based around Aurora 2 (the functional spec doesn't make it clear.) Remember, I'm seeing this problem on both Armada 8040 and LX2160A. We don't know of any known errata for the A72 in this area. So, we're down to something in the kernel. It is possible that it could be compiler related, but I don't see that; if the "dmb oshld" were strong enough, then it should mean that the subsequent reads to checksum the inode data after the inode data has been DMA'd into memory should be reading the correct values from memory already - but they aren't. And if changing "dmb oshld" to "dsb ld" means that the code can then read the right values, that to me points fairly definitively to a hardware problem. Now, ext4fs is pretty good at checksumming the metadata in the filesystem - each inode is individually checksummed with CRC32C and two 16-bit halves are stored in each inode. Directories are also checksummed. ext4fs validates the inode checksum on every ext4_iget() call. Do other filesystems do similar? Anyway, here is the patch I'm currently running, which _seems_ so far to be the minimal fix for my problems. Will thinks that this is hiding the real problem by adding barriers, but I don't see there's much choice but to apply this - I don't see what other debugging could be done without the use of expensive hardware simulation, or detailed hardware level tracing - the kind of which a silicon vendor or ARM Ltd would have. I'm at the end of what I can do with this; I'm going to keep this patch in my kernel, since it fixes it for me. Will would like a reliable reproducer - yes, that would be ideal, but I'm afraid that's a mamoth task in itself. It's taken a year to find this method of reproducing it. There's also the matter that in one case I've seen, the ext4 checksum has been wrong. The subsequent hexdump has been correct, and the post- hexdump checksum recalculation has remained incorrect - and the same value as the first incorrect checksum. However, the inode with the _same_ checksum has subsequently validated correctly by the kernel and by e2fsck. I can not explain this. diff --git a/arch/arm64/include/asm/io.h b/arch/arm64/include/asm/io.h index ff50dd731852..be63c47aecc4 100644 --- a/arch/arm64/include/asm/io.h +++ b/arch/arm64/include/asm/io.h @@ -95,7 +95,7 @@ static inline u64 __raw_readq(const volatile void __iomem *addr) ({ \ unsigned long tmp; \ \ - dma_rmb(); \ + rmb(); \ \ /* \ * Create a dummy control dependency from the IO read to any \