From patchwork Mon Jul 18 06:02:15 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Chinner X-Patchwork-Id: 9234087 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 45D436075D for ; Mon, 18 Jul 2016 06:02:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 37E742094D for ; Mon, 18 Jul 2016 06:02:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2C46E228C8; Mon, 18 Jul 2016 06:02:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from oss.sgi.com (oss.sgi.com [192.48.182.195]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id A772C2094D for ; Mon, 18 Jul 2016 06:02:33 +0000 (UTC) Received: from oss.sgi.com (localhost [IPv6:::1]) by oss.sgi.com (Postfix) with ESMTP id 29FAA7CA1; Mon, 18 Jul 2016 01:02:32 -0500 (CDT) X-Original-To: xfs@oss.sgi.com Delivered-To: xfs@oss.sgi.com Received: from relay.sgi.com (relay2.corp.sgi.com [137.38.102.29]) by oss.sgi.com (Postfix) with ESMTP id 078EE7CA0 for ; Mon, 18 Jul 2016 01:02:30 -0500 (CDT) Received: from cuda.sgi.com (cuda1.sgi.com [192.48.157.11]) by relay2.corp.sgi.com (Postfix) with ESMTP id D8A30304032 for ; Sun, 17 Jul 2016 23:02:20 -0700 (PDT) X-ASG-Debug-ID: 1468821737-04bdf0280c35d10001-NocioJ Received: from ipmail07.adl2.internode.on.net (ipmail07.adl2.internode.on.net [150.101.137.131]) by cuda.sgi.com with ESMTP id 3B7aUhoVvLaNNqGD for ; Sun, 17 Jul 2016 23:02:17 -0700 (PDT) X-Barracuda-Envelope-From: david@fromorbit.com X-Barracuda-Effective-Source-IP: ipmail07.adl2.internode.on.net[150.101.137.131] X-Barracuda-Apparent-Source-IP: 150.101.137.131 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2ASDwA+cIxXEHvHLHlcgz+BUoZvnRIBAQEBAQEGjFWKEYYUAgIBAQKBK00BAQEBAQEHAQEBAQEBAj5AhFwBAQUnExwjEAgDFQMJJQ8FJQMHGhOIL78YAQEBAQEFAQEBAQEBIR6FRIUVhAgUAYNPgi8FhgpMkk6FKoZXglSBdYgIhUSGX4k/gl8cgV4qMoV2AQ4XgR4BAQE Received: from ppp121-44-199-123.lns20.syd7.internode.on.net (HELO dastard) ([121.44.199.123]) by ipmail07.adl2.internode.on.net with ESMTP; 18 Jul 2016 15:32:16 +0930 Received: from dave by dastard with local (Exim 4.80) (envelope-from ) id 1bP1d9-0004F4-I0; Mon, 18 Jul 2016 16:02:15 +1000 Date: Mon, 18 Jul 2016 16:02:15 +1000 From: Dave Chinner To: Calvin Owens Subject: Re: [BUG] Slab corruption during XFS writeback under memory pressure Message-ID: <20160718060215.GB16044@dastard> X-ASG-Orig-Subj: Re: [BUG] Slab corruption during XFS writeback under memory pressure References: <28f77d74-5ab4-d913-2921-df90da53f393@fb.com> <20160717000003.GW1922@dastard> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20160717000003.GW1922@dastard> User-Agent: Mutt/1.5.21 (2010-09-15) X-Barracuda-Connect: ipmail07.adl2.internode.on.net[150.101.137.131] X-Barracuda-Start-Time: 1468821737 X-Barracuda-URL: https://192.48.157.11:443/cgi-mod/mark.cgi X-Barracuda-Scan-Msg-Size: 5623 X-Virus-Scanned: by bsmtpd at sgi.com X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using per-user scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=2.7 tests=BSF_SC0_MISMATCH_TO X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.31338 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- 0.00 BSF_SC0_MISMATCH_TO Envelope rcpt doesn't match header Cc: linux-block@vger.kernel.org, kernel-team@fb.com, linux-kernel@vger.kernel.org, linux-scsi@vger.kernel.org, xfs@oss.sgi.com X-BeenThere: xfs@oss.sgi.com X-Mailman-Version: 2.1.14 Precedence: list List-Id: XFS Filesystem from SGI List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: xfs-bounces@oss.sgi.com Sender: xfs-bounces@oss.sgi.com X-Virus-Scanned: ClamAV using ClamSMTP On Sun, Jul 17, 2016 at 10:00:03AM +1000, Dave Chinner wrote: > On Fri, Jul 15, 2016 at 05:18:02PM -0700, Calvin Owens wrote: > > Hello all, > > > > I've found a nasty source of slab corruption. Based on seeing similar symptoms > > on boxes at Facebook, I suspect it's been around since at least 3.10. > > > > It only reproduces under memory pressure so far as I can tell: the issue seems > > to be that XFS reclaims pages from buffers that are still in use by > > scsi/block. I'm not sure which side the bug lies on, but I've only observed it > > with XFS. [....] > But this indicates that the page is under writeback at this point, > so that tends to indicate that the above freeing was incorrect. > > Hmmm - it's clear we've got direct reclaim involved here, and the > suspicion of a dirty page that has had it's bufferheads cleared. > Are there any other warnings in the log from XFS prior to kasan > throwing the error? Can you try the patch below? -Dave. Tested-by: Calvin Owens diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c index 80714eb..0cfb944 100644 --- a/fs/xfs/xfs_aops.c +++ b/fs/xfs/xfs_aops.c @@ -87,6 +87,12 @@ xfs_find_bdev_for_inode( * We're now finished for good with this page. Update the page state via the * associated buffer_heads, paying attention to the start and end offsets that * we need to process on the page. + * + * Landmine Warning: bh->b_end_io() will call end_page_writeback() on the last + * buffer in the IO. Once it does this, it is unsafe to access the bufferhead or + * the page at all, as we may be racing with memory reclaim and it can free both + * the bufferhead chain and the page as it will see the page as clean and + * unused. */ static void xfs_finish_page_writeback( @@ -95,8 +101,9 @@ xfs_finish_page_writeback( int error) { unsigned int end = bvec->bv_offset + bvec->bv_len - 1; - struct buffer_head *head, *bh; + struct buffer_head *head, *bh, *next; unsigned int off = 0; + unsigned int bsize; ASSERT(bvec->bv_offset < PAGE_SIZE); ASSERT((bvec->bv_offset & ((1 << inode->i_blkbits) - 1)) == 0); @@ -105,15 +112,17 @@ xfs_finish_page_writeback( bh = head = page_buffers(bvec->bv_page); + bsize = bh->b_size; do { + next = bh->b_this_page; if (off < bvec->bv_offset) goto next_bh; if (off > end) break; bh->b_end_io(bh, !error); next_bh: - off += bh->b_size; - } while ((bh = bh->b_this_page) != head); + off += bsize; + } while ((bh = next) != head); } /*