From patchwork Wed Nov 6 04:43:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 11229279 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C66D813BD for ; Wed, 6 Nov 2019 04:44:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8F81D2075C for ; Wed, 6 Nov 2019 04:44:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="l7g+ffSh" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729632AbfKFEoO (ORCPT ); Tue, 5 Nov 2019 23:44:14 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:55684 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729303AbfKFEoN (ORCPT ); Tue, 5 Nov 2019 23:44:13 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id xA64i8HA078441; Wed, 6 Nov 2019 04:44:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=subject : from : to : cc : date : message-id : in-reply-to : references : mime-version : content-type : content-transfer-encoding; s=corp-2019-08-05; bh=9OdNinNJ7DNtRfWC2HFWqKGc9Pft7kDNZx+7+SHneSw=; b=l7g+ffSh6JyV7hj3kQmGzw8SmGJfKD5XWKLpidZYsFKPIUtC6JZ+Hgz7Zn9TNQGJZDdd lC0I6vr2AjTGPrjKYUmtR9ITmHhm59uLjXnDIp0SepwMvWnDANE0gthG2VEMVVKkisN8 hMKRTQEaC0tFStd1cxrXP5mwpZ8TrzhMZYKyYI8eL4pol/RK6OefKvAx4k/+QESIeNJp +3XUfrxqf4kCeHVbpsWBzh8SczrOhMdrRFXi4q88FzYwTPIHGv/ZoZG7cQ+6v9mVr5lk 0otZnLK2zulWNFrBX9FT/ZeVnUiTfLiM0sGr/fbM48h48CRFRJFhST2Ge/fW2RacHVCR ng== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by userp2120.oracle.com with ESMTP id 2w12eraxt8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 06 Nov 2019 04:44:08 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.27/8.16.0.27) with SMTP id xA64i45G033825; Wed, 6 Nov 2019 04:44:07 GMT Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by aserp3030.oracle.com with ESMTP id 2w35pq7bsx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 06 Nov 2019 04:44:05 +0000 Received: from abhmp0004.oracle.com (abhmp0004.oracle.com [141.146.116.10]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id xA64h0ls031424; Wed, 6 Nov 2019 04:43:00 GMT Received: from localhost (/67.169.218.210) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 05 Nov 2019 20:43:00 -0800 Subject: [PATCH 1/2] xfs: add missing early termination checks to record scrubbing functions From: "Darrick J. Wong" To: darrick.wong@oracle.com Cc: linux-xfs@vger.kernel.org, hch@lst.de Date: Tue, 05 Nov 2019 20:43:00 -0800 Message-ID: <157301538007.678524.17905821115324746213.stgit@magnolia> In-Reply-To: <157301537390.678524.16085197974806955970.stgit@magnolia> References: <157301537390.678524.16085197974806955970.stgit@magnolia> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9432 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=1 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1908290000 definitions=main-1911060049 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9432 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1908290000 definitions=main-1911060049 Sender: linux-xfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org From: Darrick J. Wong Scrubbing directories, quotas, and fs counters all involve iterating some collection of metadata items. The per-item scrub functions for these three are missing some of the components they need to be able to check for a fatal signal and terminate early. Per-item scrub functions need to call xchk_should_terminate to look for fatal signals, and they need to check the scrub context's corruption flag because there's no point in continuing a scan once we've decided the data structure is bad. Add both of these where missing. Signed-off-by: Darrick J. Wong Reviewed-by: Christoph Hellwig --- fs/xfs/scrub/dir.c | 3 +++ fs/xfs/scrub/fscounters.c | 8 ++++++-- fs/xfs/scrub/quota.c | 7 +++++++ 3 files changed, 16 insertions(+), 2 deletions(-) diff --git a/fs/xfs/scrub/dir.c b/fs/xfs/scrub/dir.c index 1e2e11721eb9..dca5f159f82a 100644 --- a/fs/xfs/scrub/dir.c +++ b/fs/xfs/scrub/dir.c @@ -113,6 +113,9 @@ xchk_dir_actor( offset = xfs_dir2_db_to_da(mp->m_dir_geo, xfs_dir2_dataptr_to_db(mp->m_dir_geo, pos)); + if (xchk_should_terminate(sdc->sc, &error)) + return error; + /* Does this inode number make sense? */ if (!xfs_verify_dir_ino(mp, ino)) { xchk_fblock_set_corrupt(sdc->sc, XFS_DATA_FORK, offset); diff --git a/fs/xfs/scrub/fscounters.c b/fs/xfs/scrub/fscounters.c index 98f82d7c8b40..7251c66a82c9 100644 --- a/fs/xfs/scrub/fscounters.c +++ b/fs/xfs/scrub/fscounters.c @@ -104,7 +104,7 @@ xchk_fscount_warmup( pag = NULL; error = 0; - if (fatal_signal_pending(current)) + if (xchk_should_terminate(sc, &error)) break; } @@ -163,6 +163,7 @@ xchk_fscount_aggregate_agcounts( uint64_t delayed; xfs_agnumber_t agno; int tries = 8; + int error = 0; retry: fsc->icount = 0; @@ -196,10 +197,13 @@ xchk_fscount_aggregate_agcounts( xfs_perag_put(pag); - if (fatal_signal_pending(current)) + if (xchk_should_terminate(sc, &error)) break; } + if (error) + return error; + /* * The global incore space reservation is taken from the incore * counters, so leave that out of the computation. diff --git a/fs/xfs/scrub/quota.c b/fs/xfs/scrub/quota.c index 0a33b4421c32..905a34558361 100644 --- a/fs/xfs/scrub/quota.c +++ b/fs/xfs/scrub/quota.c @@ -93,6 +93,10 @@ xchk_quota_item( unsigned long long rcount; xfs_ino_t fs_icount; xfs_dqid_t id = be32_to_cpu(d->d_id); + int error = 0; + + if (xchk_should_terminate(sc, &error)) + return error; /* * Except for the root dquot, the actual dquot we got must either have @@ -178,6 +182,9 @@ xchk_quota_item( if (id != 0 && rhard != 0 && rcount > rhard) xchk_fblock_set_warning(sc, XFS_DATA_FORK, offset); + if (sc->sm->sm_flags & XFS_SCRUB_OFLAG_CORRUPT) + return -EFSCORRUPTED; + return 0; } From patchwork Wed Nov 6 04:43:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 11229283 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 64BF81747 for ; Wed, 6 Nov 2019 04:44:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 389482173B for ; Wed, 6 Nov 2019 04:44:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="hqXeteqn" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730944AbfKFEoa (ORCPT ); Tue, 5 Nov 2019 23:44:30 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:55982 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729303AbfKFEo3 (ORCPT ); Tue, 5 Nov 2019 23:44:29 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id xA64i6ru078411; Wed, 6 Nov 2019 04:44:07 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=subject : from : to : cc : date : message-id : in-reply-to : references : mime-version : content-type : content-transfer-encoding; s=corp-2019-08-05; bh=snGtq/nWttwN90OKhn98bwpRunQm/1+ub5EJOLJ66LQ=; b=hqXeteqnWSzzx2R3CiT5bBOY6U57ZWALBVVpNJ10HvXy+7WJ6ZbhwTFGjmdbg/V6JS6l f+bOXimsPd2HmqqQPVWJaMhFhidi0jzLrOOH/2ko4u7taEmxAnFya3STGgv1yYahY0sM UviyToWBhCNQ6pdHI48n8du/dJJjaP2zXkkTAhHphfG81nFQaYmFFN+N+PJJRUIP2/MW JrBxdSDCXY3cpoM1RQSaDrl7fayCFqsAiz6mZMGq0EZB8JGq/C5ATMB7KGiXqNb1TWzm Njs0YyNNAvJ81F8sf369K8c/qldTTWG/n9VoGKh99rT401lmTdcKPMyoZlpDImdCrjQB Sw== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by userp2120.oracle.com with ESMTP id 2w12eraxt4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 06 Nov 2019 04:44:07 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.27/8.16.0.27) with SMTP id xA64i2Er175785; Wed, 6 Nov 2019 04:44:06 GMT Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userp3020.oracle.com with ESMTP id 2w3162ns6k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 06 Nov 2019 04:44:06 +0000 Received: from abhmp0011.oracle.com (abhmp0011.oracle.com [141.146.116.17]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id xA64h6Em031459; Wed, 6 Nov 2019 04:43:06 GMT Received: from localhost (/67.169.218.210) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 05 Nov 2019 20:43:06 -0800 Subject: [PATCH 2/2] xfs: periodically yield scrub threads to the scheduler From: "Darrick J. Wong" To: darrick.wong@oracle.com Cc: Christoph Hellwig , linux-xfs@vger.kernel.org, hch@lst.de Date: Tue, 05 Nov 2019 20:43:06 -0800 Message-ID: <157301538629.678524.5328247190031479757.stgit@magnolia> In-Reply-To: <157301537390.678524.16085197974806955970.stgit@magnolia> References: <157301537390.678524.16085197974806955970.stgit@magnolia> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9432 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=3 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1908290000 definitions=main-1911060049 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9432 signatures=668685 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=3 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1908290000 definitions=main-1911060049 Sender: linux-xfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org From: Darrick J. Wong Christoph Hellwig complained about the following soft lockup warning when running scrub after generic/175 when preemption is disabled and slub debugging is enabled: watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [xfs_scrub:161] Modules linked in: irq event stamp: 41692326 hardirqs last enabled at (41692325): [] _raw_0 hardirqs last disabled at (41692326): [] trace0 softirqs last enabled at (41684994): [] __do_e softirqs last disabled at (41684987): [] irq_e0 CPU: 3 PID: 16189 Comm: xfs_scrub Not tainted 5.4.0-rc3+ #30 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.124 RIP: 0010:_raw_spin_unlock_irqrestore+0x39/0x40 Code: 89 f3 be 01 00 00 00 e8 d5 3a e5 fe 48 89 ef e8 ed 87 e5 f2 RSP: 0018:ffffc9000233f970 EFLAGS: 00000286 ORIG_RAX: ffffffffff3 RAX: ffff88813b398040 RBX: 0000000000000286 RCX: 0000000000000006 RDX: 0000000000000006 RSI: ffff88813b3988c0 RDI: ffff88813b398040 RBP: ffff888137958640 R08: 0000000000000001 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: ffffea00042b0c00 R13: 0000000000000001 R14: ffff88810ac32308 R15: ffff8881376fc040 FS: 00007f6113dea700(0000) GS:ffff88813bb80000(0000) knlGS:00000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f6113de8ff8 CR3: 000000012f290000 CR4: 00000000000006e0 Call Trace: free_debug_processing+0x1dd/0x240 __slab_free+0x231/0x410 kmem_cache_free+0x30e/0x360 xchk_ag_btcur_free+0x76/0xb0 xchk_ag_free+0x10/0x80 xchk_bmap_iextent_xref.isra.14+0xd9/0x120 xchk_bmap_iextent+0x187/0x210 xchk_bmap+0x2e0/0x3b0 xfs_scrub_metadata+0x2e7/0x500 xfs_ioc_scrub_metadata+0x4a/0xa0 xfs_file_ioctl+0x58a/0xcd0 do_vfs_ioctl+0xa0/0x6f0 ksys_ioctl+0x5b/0x90 __x64_sys_ioctl+0x11/0x20 do_syscall_64+0x4b/0x1a0 entry_SYSCALL_64_after_hwframe+0x49/0xbe If preemption is disabled, all metadata buffers needed to perform the scrub are already in memory, and there are a lot of records to check, it's possible that the scrub thread will run for an extended period of time without sleeping for IO or any other reason. Then the watchdog timer or the RCU stall timeout can trigger, producing the backtrace above. To fix this problem, we detect when preemption is disabled and explicitly schedule() the scrub thread every few seconds. Reported-by: Christoph Hellwig Signed-off-by: Darrick J. Wong --- fs/xfs/scrub/common.h | 14 +++++++++++++- fs/xfs/scrub/scrub.h | 9 +++++++++ 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/fs/xfs/scrub/common.h b/fs/xfs/scrub/common.h index 003a772cd26c..597b4d45e990 100644 --- a/fs/xfs/scrub/common.h +++ b/fs/xfs/scrub/common.h @@ -14,8 +14,20 @@ static inline bool xchk_should_terminate( struct xfs_scrub *sc, - int *error) + int *error) { +#if !IS_ENABLED(CONFIG_PREEMPT) + /* + * If preemption is disabled, we need to yield to the scheduler every + * few seconds so that we don't run afoul of the soft lockup watchdog + * or RCU stall detector. + */ + if (sc->next_yield != 0 && time_after(jiffies, sc->next_yield)) + return false; + schedule(); + sc->next_yield = jiffies + msecs_to_jiffies(5000); +#endif + if (fatal_signal_pending(current)) { if (*error == 0) *error = -EAGAIN; diff --git a/fs/xfs/scrub/scrub.h b/fs/xfs/scrub/scrub.h index ad1ceb44a628..ada8e4976024 100644 --- a/fs/xfs/scrub/scrub.h +++ b/fs/xfs/scrub/scrub.h @@ -75,6 +75,15 @@ struct xfs_scrub { /* State tracking for single-AG operations. */ struct xchk_ag sa; + +#if !IS_ENABLED(CONFIG_PREEMPT) + /* + * This is the time (in jiffies) when this scrub thread needs to + * yield the processor back to the scheduler so that we don't run + * afoul of either the soft lockup watchdog or RCU stall detector. + */ + unsigned long next_yield; +#endif }; /* XCHK state flags grow up from zero, XREP state flags grown down from 2^31 */