From patchwork Tue Oct 15 17:21:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 11191309 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0EFB814ED for ; Tue, 15 Oct 2019 17:21:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id E33DB20872 for ; Tue, 15 Oct 2019 17:21:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="kKyhYNQp" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726674AbfJORVd (ORCPT ); Tue, 15 Oct 2019 13:21:33 -0400 Received: from aserp2120.oracle.com ([141.146.126.78]:51246 "EHLO aserp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726525AbfJORVd (ORCPT ); Tue, 15 Oct 2019 13:21:33 -0400 Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x9FHIiup051008; Tue, 15 Oct 2019 17:21:30 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=date : from : to : cc : subject : message-id : references : mime-version : content-type : in-reply-to; s=corp-2019-08-05; bh=OOCEQHgOj8lJMlQIqETEMupbshHfwsicNoRtTRTIAzQ=; b=kKyhYNQp9Kbd6AEt0l3Eqx/MrG3yVK1ohc8s2yB5qNuxzxEcweHu2jl7AGGPCgmXbqns KOV1RkSpM3fDXoInSVfGD2kKXa0B4/2kON2lw1BViJ2uVBrMZAzg37ETAEy0K3Seyv37 paVrRa+RzDOxbUyLsUqdBUlV+7qb07efHDsFhU+uBs8NqS7tJIku+y52jHqeBxFUPoFJ /sqswaAFE2UOT2Ce9U+NYeyimPVhDRcb+UcTsIWlMxodvoWCYyc0PGSaL04rchUe0ocB oWjSNAylIcJp05JIrY2x1ws6WvFbJS4PXWZup+CutaXutxnjHoeXNA5bWyuK8ZY8blNC 3A== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by aserp2120.oracle.com with ESMTP id 2vk6sqhkej-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 15 Oct 2019 17:21:30 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x9FHJ9vp030387; Tue, 15 Oct 2019 17:21:29 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userp3020.oracle.com with ESMTP id 2vn718pxgr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 15 Oct 2019 17:21:29 +0000 Received: from abhmp0005.oracle.com (abhmp0005.oracle.com [141.146.116.11]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x9FHLSlt011902; Tue, 15 Oct 2019 17:21:28 GMT Received: from localhost (/67.169.218.210) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 15 Oct 2019 17:21:28 +0000 Date: Tue, 15 Oct 2019 10:21:27 -0700 From: "Darrick J. Wong" To: sandeen@sandeen.net Cc: linux-xfs@vger.kernel.org Subject: [PATCH v2 01/11] xfs_scrub: fix handling of read-verify pool runtime errors Message-ID: <20191015172127.GI13108@magnolia> References: <156944728875.298887.8311229116097714980.stgit@magnolia> <156944729476.298887.15638727982082805193.stgit@magnolia> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <156944729476.298887.15638727982082805193.stgit@magnolia> User-Agent: Mutt/1.9.4 (2018-02-28) X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9411 signatures=668684 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=3 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1908290000 definitions=main-1910150148 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9411 signatures=668684 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=3 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1908290000 definitions=main-1910150148 Sender: linux-xfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org From: Darrick J. Wong Fix some bogosity with how we handle runtime errors in the read verify pool functions. First of all, memory allocation failures shouldn't be recorded as disk IO errors, they should just complain and abort the phase. Second, we need to collect any other runtime errors in the IO thread and abort the phase instead of silently ignoring them. Signed-off-by: Darrick J. Wong --- v2: fix errors_seen bogosity --- scrub/read_verify.c | 27 +++++++++++++++++++++++---- 1 file changed, 23 insertions(+), 4 deletions(-) diff --git a/scrub/read_verify.c b/scrub/read_verify.c index b890c92f..d020ce2f 100644 --- a/scrub/read_verify.c +++ b/scrub/read_verify.c @@ -53,6 +53,12 @@ struct read_verify_pool { struct disk *disk; /* which disk? */ read_verify_ioerr_fn_t ioerr_fn; /* io error callback */ size_t miniosz; /* minimum io size, bytes */ + + /* + * Store a runtime error code here so that we can stop the pool and + * return it to the caller. + */ + int runtime_error; }; /* @@ -149,6 +155,7 @@ read_verify( unsigned long long verified = 0; ssize_t sz; ssize_t len; + int ret; rvp = (struct read_verify_pool *)wq->wq_ctx; while (rv->io_length > 0) { @@ -173,7 +180,12 @@ read_verify( } free(rv); - ptcounter_add(rvp->verified_bytes, verified); + ret = ptcounter_add(rvp->verified_bytes, verified); + if (ret) { + str_liberror(rvp->ctx, ret, + _("updating bytes verified counter")); + rvp->runtime_error = ret; + } } /* Queue a read verify request. */ @@ -188,18 +200,25 @@ read_verify_queue( dbg_printf("verify fd %d start %"PRIu64" len %"PRIu64"\n", rvp->disk->d_fd, rv->io_start, rv->io_length); + /* Worker thread saw a runtime error, don't queue more. */ + if (rvp->runtime_error) + return false; + + /* Otherwise clone the request and queue the copy. */ tmp = malloc(sizeof(struct read_verify)); if (!tmp) { - rvp->ioerr_fn(rvp->ctx, rvp->disk, rv->io_start, - rv->io_length, errno, rv->io_end_arg); - return true; + rvp->runtime_error = errno; + str_errno(rvp->ctx, _("allocating read-verify request")); + return false; } + memcpy(tmp, rv, sizeof(*tmp)); ret = workqueue_add(&rvp->wq, read_verify, 0, tmp); if (ret) { str_liberror(rvp->ctx, ret, _("queueing read-verify work")); free(tmp); + rvp->runtime_error = ret; return false; } rv->io_length = 0; From patchwork Tue Oct 15 17:21:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 11191311 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CCF711390 for ; Tue, 15 Oct 2019 17:21:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id AC70B20650 for ; Tue, 15 Oct 2019 17:21:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="ipIpAYmV" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726525AbfJORV7 (ORCPT ); Tue, 15 Oct 2019 13:21:59 -0400 Received: from aserp2120.oracle.com ([141.146.126.78]:52020 "EHLO aserp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726125AbfJORV6 (ORCPT ); Tue, 15 Oct 2019 13:21:58 -0400 Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x9FHJ0aI051109; Tue, 15 Oct 2019 17:21:57 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=date : from : to : cc : subject : message-id : mime-version : content-type; s=corp-2019-08-05; bh=hWm/rwZRIE/e564edzCu8V2GBvr/9e3s0yKEeSCZMPQ=; b=ipIpAYmVYf4xcr14D4etduY+IYcgJnFCMl+AqwfVsarO60UguIIpKUIbPKD/kT20wyLt pUYut8qHdRRSvnodfuxhktgvfLH+5MB0RFWNwlmWhnsbfGONf97NxAwVvu3ceszFCX4y ztoKyqZVWAfnf6vwhlKEp44hzq0gLOcDrPwHrgZGSvQ/M2K0czJ/uk/fsdo7W2O2RbZV P9GQnlXX/InWuUMraQHkeVVAIGDbB9XKPpOrfxSS2PFJUP2pEGgyhe9QZnmgeWk4+qfU OEd6gZrL8wq6/aqBWI6Y9IKg+vVK14IRw89m2IIUh8eqMftfUqePByqQCUCq6j+JBmDT 2Q== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by aserp2120.oracle.com with ESMTP id 2vk6sqhkjd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 15 Oct 2019 17:21:56 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x9FHJ67F001354; Tue, 15 Oct 2019 17:21:56 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userp3030.oracle.com with ESMTP id 2vn8enjpju-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 15 Oct 2019 17:21:56 +0000 Received: from abhmp0020.oracle.com (abhmp0020.oracle.com [141.146.116.26]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x9FHLtWa019817; Tue, 15 Oct 2019 17:21:55 GMT Received: from localhost (/67.169.218.210) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 15 Oct 2019 17:21:54 +0000 Date: Tue, 15 Oct 2019 10:21:53 -0700 From: "Darrick J. Wong" To: sandeen@sandeen.net Cc: linux-xfs@vger.kernel.org Subject: [PATCH v2 02/11] xfs_scrub: abort all read verification work immediately on error Message-ID: <20191015172153.GJ13108@magnolia> MIME-Version: 1.0 Content-Disposition: inline User-Agent: Mutt/1.9.4 (2018-02-28) X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9411 signatures=668684 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=3 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1908290000 definitions=main-1910150148 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9411 signatures=668684 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=3 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1908290000 definitions=main-1910150148 Sender: linux-xfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org From: Darrick J. Wong Add a new abort function to the read verify pool code so that the caller can immediately abort all pending verification work if things start going wrong. There's no point in waiting for queued work to run if we've already decided to bail. Signed-off-by: Darrick J. Wong --- v2: fix errors_seen bogosity --- scrub/phase6.c | 6 +++--- scrub/read_verify.c | 10 ++++++++++ scrub/read_verify.h | 1 + 3 files changed, 14 insertions(+), 3 deletions(-) diff --git a/scrub/phase6.c b/scrub/phase6.c index d9285fee..4c81ee7b 100644 --- a/scrub/phase6.c +++ b/scrub/phase6.c @@ -514,16 +514,16 @@ _("Could not create data device media verifier.")); out_rtpool: if (vs.rvp_realtime) { - read_verify_pool_flush(vs.rvp_realtime); + read_verify_pool_abort(vs.rvp_realtime); read_verify_pool_destroy(vs.rvp_realtime); } out_logpool: if (vs.rvp_log) { - read_verify_pool_flush(vs.rvp_log); + read_verify_pool_abort(vs.rvp_log); read_verify_pool_destroy(vs.rvp_log); } out_datapool: - read_verify_pool_flush(vs.rvp_data); + read_verify_pool_abort(vs.rvp_data); read_verify_pool_destroy(vs.rvp_data); out_rbad: bitmap_free(&vs.r_bad); diff --git a/scrub/read_verify.c b/scrub/read_verify.c index d020ce2f..835b9660 100644 --- a/scrub/read_verify.c +++ b/scrub/read_verify.c @@ -121,6 +121,16 @@ read_verify_pool_init( return NULL; } +/* Abort all verification work. */ +void +read_verify_pool_abort( + struct read_verify_pool *rvp) +{ + if (!rvp->runtime_error) + rvp->runtime_error = ECANCELED; + workqueue_terminate(&rvp->wq); +} + /* Finish up any read verification work. */ void read_verify_pool_flush( diff --git a/scrub/read_verify.h b/scrub/read_verify.h index 5fabe5e0..f0ed8902 100644 --- a/scrub/read_verify.h +++ b/scrub/read_verify.h @@ -19,6 +19,7 @@ struct read_verify_pool *read_verify_pool_init(struct scrub_ctx *ctx, struct disk *disk, size_t miniosz, read_verify_ioerr_fn_t ioerr_fn, unsigned int submitter_threads); +void read_verify_pool_abort(struct read_verify_pool *rvp); void read_verify_pool_flush(struct read_verify_pool *rvp); void read_verify_pool_destroy(struct read_verify_pool *rvp); From patchwork Tue Oct 15 17:22:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 11191313 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ADA151390 for ; Tue, 15 Oct 2019 17:22:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7C7AA20650 for ; Tue, 15 Oct 2019 17:22:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="aRl0CGP5" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726125AbfJORWg (ORCPT ); Tue, 15 Oct 2019 13:22:36 -0400 Received: from userp2130.oracle.com ([156.151.31.86]:38712 "EHLO userp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731903AbfJORWf (ORCPT ); Tue, 15 Oct 2019 13:22:35 -0400 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x9FHInrI073903; Tue, 15 Oct 2019 17:22:33 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=date : from : to : cc : subject : message-id : mime-version : content-type; s=corp-2019-08-05; bh=sjhApLfoXKQ1yUiLwr3AZSNZRBo3chmFpnkVvtF74fY=; b=aRl0CGP5akRo3UmSs84F0Q6NHVNQ6hMxYYOTgA1Z15PcQ56yAflRaxBcPox9+BU9/nyH 9HIAXsi7krKMHZ3a87yLVCWXT5L/YzbgCC96ozUIxzFE3R2muBTOvDIYeXDT+lsR5XPT URUdchkHUIMfh34Ag+IAVoH9bp6uPY97P+XWCNFjUFLrxVA7dc5nAoo0r4ucAPAybQKn OEbLzY4+Opv9g/UYMIoC+JRbSdyd7byrZZowR6ckxgXtLyllWN8jqEw3B0MjH/5FC6nx RTlbUcVAuovPG2kprxSuXG5azqWnmg3KXyf9FZUe9AEkF0yC5j59JaUNlJOcI/CIIHz2 iA== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by userp2130.oracle.com with ESMTP id 2vk68uhmsk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 15 Oct 2019 17:22:33 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x9FHILVq172038; Tue, 15 Oct 2019 17:22:32 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserp3020.oracle.com with ESMTP id 2vnb0fhvw5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 15 Oct 2019 17:22:32 +0000 Received: from abhmp0012.oracle.com (abhmp0012.oracle.com [141.146.116.18]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x9FHMUvc020112; Tue, 15 Oct 2019 17:22:30 GMT Received: from localhost (/67.169.218.210) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 15 Oct 2019 17:22:30 +0000 Date: Tue, 15 Oct 2019 10:22:29 -0700 From: "Darrick J. Wong" To: sandeen@sandeen.net Cc: linux-xfs@vger.kernel.org Subject: [PATCH v2 03/11] xfs_scrub: fix read-verify pool error communication problems Message-ID: <20191015172229.GK13108@magnolia> MIME-Version: 1.0 Content-Disposition: inline User-Agent: Mutt/1.9.4 (2018-02-28) X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9411 signatures=668684 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=3 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1908290000 definitions=main-1910150148 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9411 signatures=668684 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=3 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1908290000 definitions=main-1910150148 Sender: linux-xfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org From: Darrick J. Wong Fix all the places in the read-verify pool functions either we fail to check for runtime errors or fail to communicate them properly to callers. Then fix all the callers to report the error messages instead of hiding them. Signed-off-by: Darrick J. Wong --- v2: fix errors_seen bogosity --- scrub/phase6.c | 89 ++++++++++++++++++++++++++++++++++++--------------- scrub/read_verify.c | 79 ++++++++++++++++++++------------------------- scrub/read_verify.h | 16 +++++---- 3 files changed, 105 insertions(+), 79 deletions(-) diff --git a/scrub/phase6.c b/scrub/phase6.c index 4c81ee7b..f6274a49 100644 --- a/scrub/phase6.c +++ b/scrub/phase6.c @@ -387,6 +387,7 @@ xfs_check_rmap( { struct media_verify_state *vs = arg; struct read_verify_pool *rvp; + int ret; rvp = xfs_dev_to_pool(ctx, vs, map->fmr_device); @@ -415,28 +416,48 @@ xfs_check_rmap( /* XXX: Filter out directory data blocks. */ /* Schedule the read verify command for (eventual) running. */ - read_verify_schedule_io(rvp, map->fmr_physical, map->fmr_length, vs); + ret = read_verify_schedule_io(rvp, map->fmr_physical, map->fmr_length, + vs); + if (ret) { + str_liberror(ctx, ret, descr); + return false; + } out: /* Is this the last extent? Fire off the read. */ - if (map->fmr_flags & FMR_OF_LAST) - read_verify_force_io(rvp); + if (map->fmr_flags & FMR_OF_LAST) { + ret = read_verify_force_io(rvp); + if (ret) { + str_liberror(ctx, ret, descr); + return false; + } + } return true; } /* Wait for read/verify actions to finish, then return # bytes checked. */ -static uint64_t +static int clean_pool( - struct read_verify_pool *rvp) + struct read_verify_pool *rvp, + unsigned long long *bytes_checked) { - uint64_t ret; + uint64_t pool_checked; + int ret; if (!rvp) return 0; - read_verify_pool_flush(rvp); - ret = read_verify_bytes(rvp); + ret = read_verify_pool_flush(rvp); + if (ret) + goto out_destroy; + + ret = read_verify_bytes(rvp, &pool_checked); + if (ret) + goto out_destroy; + + *bytes_checked += pool_checked; +out_destroy: read_verify_pool_destroy(rvp); return ret; } @@ -469,43 +490,57 @@ xfs_scan_blocks( goto out_dbad; } - vs.rvp_data = read_verify_pool_init(ctx, ctx->datadev, + ret = read_verify_pool_alloc(ctx, ctx->datadev, ctx->mnt.fsgeom.blocksize, xfs_check_rmap_ioerr, - scrub_nproc(ctx)); - if (!vs.rvp_data) { - str_info(ctx, ctx->mntpoint, -_("Could not create data device media verifier.")); + scrub_nproc(ctx), &vs.rvp_data); + if (ret) { + str_liberror(ctx, ret, _("creating datadev media verifier")); goto out_rbad; } if (ctx->logdev) { - vs.rvp_log = read_verify_pool_init(ctx, ctx->logdev, + ret = read_verify_pool_alloc(ctx, ctx->logdev, ctx->mnt.fsgeom.blocksize, xfs_check_rmap_ioerr, - scrub_nproc(ctx)); - if (!vs.rvp_log) { - str_info(ctx, ctx->mntpoint, - _("Could not create log device media verifier.")); + scrub_nproc(ctx), &vs.rvp_log); + if (ret) { + str_liberror(ctx, ret, + _("creating logdev media verifier")); goto out_datapool; } } if (ctx->rtdev) { - vs.rvp_realtime = read_verify_pool_init(ctx, ctx->rtdev, + ret = read_verify_pool_alloc(ctx, ctx->rtdev, ctx->mnt.fsgeom.blocksize, xfs_check_rmap_ioerr, - scrub_nproc(ctx)); - if (!vs.rvp_realtime) { - str_info(ctx, ctx->mntpoint, - _("Could not create realtime device media verifier.")); + scrub_nproc(ctx), &vs.rvp_realtime); + if (ret) { + str_liberror(ctx, ret, + _("creating rtdev media verifier")); goto out_logpool; } } moveon = xfs_scan_all_spacemaps(ctx, xfs_check_rmap, &vs); if (!moveon) goto out_rtpool; - ctx->bytes_checked += clean_pool(vs.rvp_data); - ctx->bytes_checked += clean_pool(vs.rvp_log); - ctx->bytes_checked += clean_pool(vs.rvp_realtime); + + ret = clean_pool(vs.rvp_data, &ctx->bytes_checked); + if (ret) { + str_liberror(ctx, ret, _("flushing datadev verify pool")); + moveon = false; + } + + ret = clean_pool(vs.rvp_log, &ctx->bytes_checked); + if (ret) { + str_liberror(ctx, ret, _("flushing logdev verify pool")); + moveon = false; + } + + ret = clean_pool(vs.rvp_realtime, &ctx->bytes_checked); + if (ret) { + str_liberror(ctx, ret, _("flushing rtdev verify pool")); + moveon = false; + } /* Scan the whole dir tree to see what matches the bad extents. */ - if (!bitmap_empty(vs.d_bad) || !bitmap_empty(vs.r_bad)) + if (moveon && (!bitmap_empty(vs.d_bad) || !bitmap_empty(vs.r_bad))) moveon = xfs_report_verify_errors(ctx, &vs); bitmap_free(&vs.r_bad); diff --git a/scrub/read_verify.c b/scrub/read_verify.c index 835b9660..c19b2289 100644 --- a/scrub/read_verify.c +++ b/scrub/read_verify.c @@ -70,36 +70,36 @@ struct read_verify_pool { * @submitter_threads is the number of threads that may be sending verify * requests at any given time. */ -struct read_verify_pool * -read_verify_pool_init( +int +read_verify_pool_alloc( struct scrub_ctx *ctx, struct disk *disk, size_t miniosz, read_verify_ioerr_fn_t ioerr_fn, - unsigned int submitter_threads) + unsigned int submitter_threads, + struct read_verify_pool **prvp) { struct read_verify_pool *rvp; - bool ret; - int error; + int ret; rvp = calloc(1, sizeof(struct read_verify_pool)); if (!rvp) - return NULL; + return errno; - error = posix_memalign((void **)&rvp->readbuf, page_size, + ret = posix_memalign((void **)&rvp->readbuf, page_size, RVP_IO_MAX_SIZE); - if (error || !rvp->readbuf) + if (ret) goto out_free; - error = ptcounter_alloc(nproc, &rvp->verified_bytes); - if (error) + ret = ptcounter_alloc(nproc, &rvp->verified_bytes); + if (ret) goto out_buf; rvp->miniosz = miniosz; rvp->ctx = ctx; rvp->disk = disk; rvp->ioerr_fn = ioerr_fn; - error = ptvar_alloc(submitter_threads, sizeof(struct read_verify), + ret = ptvar_alloc(submitter_threads, sizeof(struct read_verify), &rvp->rvstate); - if (error) + if (ret) goto out_counter; /* Run in the main thread if we only want one thread. */ if (nproc == 1) @@ -108,7 +108,8 @@ read_verify_pool_init( disk_heads(disk)); if (ret) goto out_rvstate; - return rvp; + *prvp = rvp; + return 0; out_rvstate: ptvar_free(rvp->rvstate); @@ -118,7 +119,7 @@ read_verify_pool_init( free(rvp->readbuf); out_free: free(rvp); - return NULL; + return ret; } /* Abort all verification work. */ @@ -132,11 +133,11 @@ read_verify_pool_abort( } /* Finish up any read verification work. */ -void +int read_verify_pool_flush( struct read_verify_pool *rvp) { - workqueue_terminate(&rvp->wq); + return workqueue_terminate(&rvp->wq); } /* Finish up any read verification work and tear it down. */ @@ -191,15 +192,12 @@ read_verify( free(rv); ret = ptcounter_add(rvp->verified_bytes, verified); - if (ret) { - str_liberror(rvp->ctx, ret, - _("updating bytes verified counter")); + if (ret) rvp->runtime_error = ret; - } } /* Queue a read verify request. */ -static bool +static int read_verify_queue( struct read_verify_pool *rvp, struct read_verify *rv) @@ -212,34 +210,33 @@ read_verify_queue( /* Worker thread saw a runtime error, don't queue more. */ if (rvp->runtime_error) - return false; + return rvp->runtime_error; /* Otherwise clone the request and queue the copy. */ tmp = malloc(sizeof(struct read_verify)); if (!tmp) { rvp->runtime_error = errno; - str_errno(rvp->ctx, _("allocating read-verify request")); - return false; + return errno; } memcpy(tmp, rv, sizeof(*tmp)); ret = workqueue_add(&rvp->wq, read_verify, 0, tmp); if (ret) { - str_liberror(rvp->ctx, ret, _("queueing read-verify work")); free(tmp); rvp->runtime_error = ret; - return false; + return ret; } + rv->io_length = 0; - return true; + return 0; } /* * Issue an IO request. We'll batch subsequent requests if they're * within 64k of each other */ -bool +int read_verify_schedule_io( struct read_verify_pool *rvp, uint64_t start, @@ -254,7 +251,7 @@ read_verify_schedule_io( assert(rvp->readbuf); rv = ptvar_get(rvp->rvstate, &ret); if (ret) - return false; + return ret; req_end = start + length; rv_end = rv->io_start + rv->io_length; @@ -281,38 +278,32 @@ read_verify_schedule_io( rv->io_end_arg = end_arg; } - return true; + return 0; } /* Force any stashed IOs into the verifier. */ -bool +int read_verify_force_io( struct read_verify_pool *rvp) { struct read_verify *rv; - bool moveon; int ret; assert(rvp->readbuf); rv = ptvar_get(rvp->rvstate, &ret); if (ret) - return false; + return ret; if (rv->io_length == 0) - return true; + return 0; - moveon = read_verify_queue(rvp, rv); - if (moveon) - rv->io_length = 0; - return moveon; + return read_verify_queue(rvp, rv); } /* How many bytes has this process verified? */ -uint64_t +int read_verify_bytes( - struct read_verify_pool *rvp) + struct read_verify_pool *rvp, + uint64_t *bytes_checked) { - uint64_t ret; - - ptcounter_value(rvp->verified_bytes, &ret); - return ret; + return ptcounter_value(rvp->verified_bytes, bytes_checked); } diff --git a/scrub/read_verify.h b/scrub/read_verify.h index f0ed8902..650c46d4 100644 --- a/scrub/read_verify.h +++ b/scrub/read_verify.h @@ -15,17 +15,17 @@ typedef void (*read_verify_ioerr_fn_t)(struct scrub_ctx *ctx, struct disk *disk, uint64_t start, uint64_t length, int error, void *arg); -struct read_verify_pool *read_verify_pool_init(struct scrub_ctx *ctx, - struct disk *disk, size_t miniosz, - read_verify_ioerr_fn_t ioerr_fn, - unsigned int submitter_threads); +int read_verify_pool_alloc(struct scrub_ctx *ctx, struct disk *disk, + size_t miniosz, read_verify_ioerr_fn_t ioerr_fn, + unsigned int submitter_threads, + struct read_verify_pool **prvp); void read_verify_pool_abort(struct read_verify_pool *rvp); -void read_verify_pool_flush(struct read_verify_pool *rvp); +int read_verify_pool_flush(struct read_verify_pool *rvp); void read_verify_pool_destroy(struct read_verify_pool *rvp); -bool read_verify_schedule_io(struct read_verify_pool *rvp, uint64_t start, +int read_verify_schedule_io(struct read_verify_pool *rvp, uint64_t start, uint64_t length, void *end_arg); -bool read_verify_force_io(struct read_verify_pool *rvp); -uint64_t read_verify_bytes(struct read_verify_pool *rvp); +int read_verify_force_io(struct read_verify_pool *rvp); +int read_verify_bytes(struct read_verify_pool *rvp, uint64_t *bytes); #endif /* XFS_SCRUB_READ_VERIFY_H_ */ From patchwork Tue Oct 15 17:23:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 11191315 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6250B1390 for ; Tue, 15 Oct 2019 17:23:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 378F5214AE for ; Tue, 15 Oct 2019 17:23:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="J0rnXwzo" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388425AbfJORXk (ORCPT ); Tue, 15 Oct 2019 13:23:40 -0400 Received: from userp2120.oracle.com ([156.151.31.85]:44812 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727387AbfJORXk (ORCPT ); Tue, 15 Oct 2019 13:23:40 -0400 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x9FHInus059692; Tue, 15 Oct 2019 17:23:38 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=date : from : to : cc : subject : message-id : mime-version : content-type; s=corp-2019-08-05; bh=zdUJIw2yrRA17eHu1vY34DvjBL6df6zATXz0rzYu1Hk=; b=J0rnXwzoQ6EIhfBRcrOTQcPht/oWjL4a2ecuUUvZNOfB3Giiewmm2hfJ7yS6MlaAyBVL bFMa38sj4Kf/V/XbQiYxYXtyiNbDSBQRaRRx66eWYw8TYfs5dFQU3+/NwS7+udLfxvRs 1nnltt/CE9JpLtUE9tf9ON6GgAbp9GzEyMvDjn3t2ywrBiD5U66eOiHXvyJgKbgg4Mhi aw+vk1mp2bzjL9a32zcPeLdNxlBjTbspXZvUKE1DU1iBK3tmiWm1IY1HiNSuPSx92eVH pACyQdPA2yBfeSVkdgHnzrt9KosOrQdvp+p2Toy1W3nEbjyAsdT0U3jn/uhjdNDBOX5M Zw== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by userp2120.oracle.com with ESMTP id 2vk7fr9f07-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 15 Oct 2019 17:23:38 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x9FHJBN1030467; Tue, 15 Oct 2019 17:23:37 GMT Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userp3020.oracle.com with ESMTP id 2vn718q36n-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 15 Oct 2019 17:23:37 +0000 Received: from abhmp0022.oracle.com (abhmp0022.oracle.com [141.146.116.28]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x9FHNarj013627; Tue, 15 Oct 2019 17:23:36 GMT Received: from localhost (/67.169.218.210) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 15 Oct 2019 17:23:36 +0000 Date: Tue, 15 Oct 2019 10:23:33 -0700 From: "Darrick J. Wong" To: sandeen@sandeen.net Cc: linux-xfs@vger.kernel.org Subject: [PATCH v2 10/11] xfs_scrub: fix read verify disk error handling strategy Message-ID: <20191015172333.GL13108@magnolia> MIME-Version: 1.0 Content-Disposition: inline User-Agent: Mutt/1.9.4 (2018-02-28) X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9411 signatures=668684 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=3 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1908290000 definitions=main-1910150148 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9411 signatures=668684 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=3 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1908290000 definitions=main-1910150148 Sender: linux-xfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org From: Darrick J. Wong The error handling strategy for media errors is totally bogus. First of all, short reads are entirely unhandled -- when we encounter a short read, we know the disk was able to feed us the beginning of what we asked for, so we need to single-step through the remainder to try to capture the exact error that we hit. Second, an actual IO error causes the entire region to be marked bad even though it could be just a few MB of a multi-gigabyte extent that's bad. Therefore, single-step each block in the IO request until we stop getting IO errors to find out if all the blocks are bad or if it's just that extent. Third, fix the fact that the loop updates its own counter variables with the length fed to read(), which doesn't necessarily have anything to do with the amount of data that the read actually produced. Signed-off-by: Darrick J. Wong Reviewed-by: Eric Sandeen --- v2: fix errors_seen bogosity --- scrub/read_verify.c | 86 ++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 74 insertions(+), 12 deletions(-) diff --git a/scrub/read_verify.c b/scrub/read_verify.c index 231df802..8aec25de 100644 --- a/scrub/read_verify.c +++ b/scrub/read_verify.c @@ -173,30 +173,92 @@ read_verify( struct read_verify *rv = arg; struct read_verify_pool *rvp; unsigned long long verified = 0; + ssize_t io_max_size; ssize_t sz; ssize_t len; + int io_error; int ret; rvp = (struct read_verify_pool *)wq->wq_ctx; + if (rvp->runtime_error) + return; + + io_max_size = RVP_IO_MAX_SIZE; + while (rv->io_length > 0) { - len = min(rv->io_length, RVP_IO_MAX_SIZE); + io_error = 0; + len = min(rv->io_length, io_max_size); dbg_printf("diskverify %d %"PRIu64" %zu\n", rvp->disk->d_fd, rv->io_start, len); sz = disk_read_verify(rvp->disk, rvp->readbuf, rv->io_start, len); - if (sz < 0) { - dbg_printf("IOERR %d %"PRIu64" %zu\n", - rvp->disk->d_fd, rv->io_start, len); - /* IO error, so try the next logical block. */ - len = rvp->miniosz; - rvp->ioerr_fn(rvp->ctx, rvp->disk, rv->io_start, len, - errno, rv->io_end_arg); + if (sz == len && io_max_size < rvp->miniosz) { + /* + * If the verify request was 100% successful and less + * than a single block in length, we were trying to + * read to the end of a block after a short read. That + * suggests there's something funny with this device, + * so single-step our way through the rest of the @rv + * range. + */ + io_max_size = rvp->miniosz; + } else if (sz < 0) { + io_error = errno; + + /* Runtime error, bail out... */ + if (io_error != EIO && io_error != EILSEQ) { + rvp->runtime_error = io_error; + return; + } + + /* + * A direct read encountered an error while performing + * a multi-block read. Reduce the transfer size to a + * single block so that we can identify the exact range + * of bad blocks and good blocks. We single-step all + * the way to the end of the @rv range, (re)starting + * with the block that just failed. + */ + if (io_max_size > rvp->miniosz) { + io_max_size = rvp->miniosz; + continue; + } + + /* + * A direct read hit an error while we were stepping + * through single blocks. Mark everything bad from + * io_start to the next miniosz block. + */ + sz = rvp->miniosz - (rv->io_start % rvp->miniosz); + dbg_printf("IOERR %d @ %"PRIu64" %zu err %d\n", + rvp->disk->d_fd, rv->io_start, sz, + io_error); + rvp->ioerr_fn(rvp->ctx, rvp->disk, rv->io_start, sz, + io_error, rv->io_end_arg); + } else if (sz < len) { + /* + * A short direct read suggests that we might have hit + * an IO error midway through the read but still had to + * return the number of bytes that were actually read. + * + * We need to force an EIO, so try reading the rest of + * the block (if it was a partial block read) or the + * next full block. + */ + io_max_size = rvp->miniosz - (sz % rvp->miniosz); + dbg_printf("SHORT %d READ @ %"PRIu64" %zu try for %zd\n", + rvp->disk->d_fd, rv->io_start, sz, + io_max_size); + } else { + /* We should never get back more bytes than we asked. */ + assert(sz == len); } - progress_add(len); - verified += len; - rv->io_start += len; - rv->io_length -= len; + progress_add(sz); + if (io_error == 0) + verified += sz; + rv->io_start += sz; + rv->io_length -= sz; } free(rv);