From patchwork Wed Feb 13 09:50:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Liu X-Patchwork-Id: 10809473 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E5C7913B5 for ; Wed, 13 Feb 2019 09:53:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D0C5B2CB22 for ; Wed, 13 Feb 2019 09:53:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CEE7E2CB4A; Wed, 13 Feb 2019 09:53:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E93772CB22 for ; Wed, 13 Feb 2019 09:53:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387503AbfBMJx2 (ORCPT ); Wed, 13 Feb 2019 04:53:28 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:53044 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729649AbfBMJx2 (ORCPT ); Wed, 13 Feb 2019 04:53:28 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1D9mh9L163832; Wed, 13 Feb 2019 09:53:18 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=MihbRRexlvZwk3kT5HIEpmz3TXdAhtLG86Pn0Hh8QqU=; b=5JUeBMc7iXJarnpFhmKAgCkgTekBV6cXtWyyg4DrSNtBdRI746Eu9ytEaTmxA9KLahei iRhdSB35+5Lb7o3+ouRMaflBus05G/BjA1R8Y9eX0h636AJL4l4xYj+ej3zrTvmqQoiu AzQNzFCw730lYWtKS/AAf1DkDMw2kyIRhcQxAZ+FUBonVafsSO8KOhoch6U8mdXdyOEJ VLgvCK46Tpjwt/aAqTsFOBvqRI4/nqf7etGfHXo/afHn3q3m5UTixFmCCWOZh8Ca7509 XTe6LEo5jonx/qJk5zRTB8Ai6YERSqM+Y1ibfuApPoYbMz0ujEtyfZaxYxw+vEHUe5pq EQ== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2120.oracle.com with ESMTP id 2qhree11rb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Feb 2019 09:53:17 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x1D9rHMr029237 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Feb 2019 09:53:17 GMT Received: from abhmp0006.oracle.com (abhmp0006.oracle.com [141.146.116.12]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x1D9rGCu026494; Wed, 13 Feb 2019 09:53:17 GMT Received: from localhost.localdomain (/116.239.187.160) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 09:53:16 +0000 From: Bob Liu To: linux-block@vger.kernel.org Cc: linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, martin.petersen@oracle.com, shirley.ma@oracle.com, allison.henderson@oracle.com, david@fromorbit.com, darrick.wong@oracle.com, hch@infradead.org, adilger@dilger.ca, Bob Liu Subject: [RFC PATCH v2 1/9] block: add nr_mirrors to request_queue Date: Wed, 13 Feb 2019 17:50:36 +0800 Message-Id: <20190213095044.29628-2-bob.liu@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190213095044.29628-1-bob.liu@oracle.com> References: <20190213095044.29628-1-bob.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9165 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130072 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When fs data/metadata checksum mismatch, lower block devices may have other correct copies. e.g if we did raid1 for protecting fs metadata. Then fs could try other copies of metadata instead of panic, but fs need be awared how many mirrors the block devices have. This patch add @nr_mirrors to struct request_queue which is similar as blk_queue_nonrot(), filesystem can grab device request queue and check the number of mirrors of this block device. @nr_mirrors is 1 by default which means only one copy, drivers e.g raid1 are responsible for setting the right value. The maximum value is BITS_PER_LONG which is 32 or 64. That should be big enough else retry lantency may be too high. Also added helper functions for get/set the number of mirrors for a specific device request queue. Todo: * Export nr_mirrors through /sysfs. Signed-off-by: Bob Liu --- block/blk-core.c | 3 +++ block/blk-settings.c | 24 ++++++++++++++++++++++++ include/linux/blkdev.h | 3 +++ include/linux/types.h | 3 +++ 4 files changed, 33 insertions(+) diff --git a/block/blk-core.c b/block/blk-core.c index 6b78ec56a4f2..b838c6dc5357 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -537,6 +537,9 @@ struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id) if (blkcg_init_queue(q)) goto fail_ref; + /* Set queue default mirrors to 1 explicitly. */ + blk_queue_set_mirrors(q, 1); + return q; fail_ref: diff --git a/block/blk-settings.c b/block/blk-settings.c index 3e7038e475ee..38e4d7e675e6 100644 --- a/block/blk-settings.c +++ b/block/blk-settings.c @@ -844,6 +844,30 @@ void blk_queue_write_cache(struct request_queue *q, bool wc, bool fua) } EXPORT_SYMBOL_GPL(blk_queue_write_cache); +/* + * Get the number of read redundant mirrors. + */ +unsigned short blk_queue_get_mirrors(struct request_queue *q) +{ + return q->nr_mirrors; +} +EXPORT_SYMBOL(blk_queue_get_mirrors); + +/* + * Set the number of read redundant mirrors. + */ +bool blk_queue_set_mirrors(struct request_queue *q, unsigned short mirrors) +{ + if(q->nr_mirrors >= BLKDEV_MAX_MIRRORS) { + printk("blk_queue_set_mirrors: %d exceed max mirrors(%d)\n", + mirrors, BLKDEV_MAX_MIRRORS); + return false; + } + q->nr_mirrors = mirrors; + return true; +} +EXPORT_SYMBOL(blk_queue_set_mirrors); + static int __init blk_settings_init(void) { blk_max_low_pfn = max_low_pfn - 1; diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 338604dff7d0..0191dc4d3f2d 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -570,6 +570,7 @@ struct request_queue { #define BLK_MAX_WRITE_HINTS 5 u64 write_hints[BLK_MAX_WRITE_HINTS]; + unsigned long nr_mirrors; /* Default value is 1 */ }; #define QUEUE_FLAG_STOPPED 1 /* queue is stopped */ @@ -1071,6 +1072,8 @@ extern void blk_queue_update_dma_alignment(struct request_queue *, int); extern void blk_queue_rq_timeout(struct request_queue *, unsigned int); extern void blk_queue_flush_queueable(struct request_queue *q, bool queueable); extern void blk_queue_write_cache(struct request_queue *q, bool enabled, bool fua); +extern unsigned short blk_queue_get_mirrors(struct request_queue *q); +extern bool blk_queue_set_mirrors(struct request_queue *q, unsigned short mirrors); /* * Number of physical segments as sent to the device. diff --git a/include/linux/types.h b/include/linux/types.h index c2615d6a019e..a29135772f3a 100644 --- a/include/linux/types.h +++ b/include/linux/types.h @@ -7,6 +7,9 @@ #ifndef __ASSEMBLY__ +/* max mirrors of blkdev */ +#define BLKDEV_MAX_MIRRORS BITS_PER_LONG + #define DECLARE_BITMAP(name,bits) \ unsigned long name[BITS_TO_LONGS(bits)] From patchwork Wed Feb 13 09:50:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Liu X-Patchwork-Id: 10809475 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5B07A1575 for ; Wed, 13 Feb 2019 09:53:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 491522CB5D for ; Wed, 13 Feb 2019 09:53:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3DBB92CC20; Wed, 13 Feb 2019 09:53:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AEBA02CC2E for ; Wed, 13 Feb 2019 09:53:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387539AbfBMJx3 (ORCPT ); Wed, 13 Feb 2019 04:53:29 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:53074 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732639AbfBMJx2 (ORCPT ); Wed, 13 Feb 2019 04:53:28 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1D9mmfO163884; Wed, 13 Feb 2019 09:53:22 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=2wZDrRNrAY5VIL2j189QrnXnQMZPgZexfpu7AfD7Jcw=; b=D6iQnOMZNOFS/c+y1HeMx1uNa9jPD/OyIL4PivtVrhfQQt2vBkaw0YKVSfRD4PyA7BuK L7XVWNpWzFCg+xLC8AHtbq2ftI+JnoYsE2Mm/AHLpcLlDKFPZRP6xqZXCdAIMnHsEH/4 Mg2Kf3cm5Sx+lmyA09YRc7KkdAi9Wbr3lGy8Q6f3t66ycEtruAW1QBvs5PDKvreQj+Sa 3oxoUg15riR13PRUS+V6Sbltt35wYJ5OIHUqgSxjd3bxnrEXlqRjUlY4djczIvKqbH/R 8U9DMX8vR8Misw39v/iogCp5mOujbIcavcQpou9sfhhjLQnZcgFim2atnsK3cerYi0X5 cw== Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp2120.oracle.com with ESMTP id 2qhree11rv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Feb 2019 09:53:22 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x1D9rLbh019626 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Feb 2019 09:53:21 GMT Received: from abhmp0006.oracle.com (abhmp0006.oracle.com [141.146.116.12]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x1D9rLtL000518; Wed, 13 Feb 2019 09:53:21 GMT Received: from localhost.localdomain (/116.239.187.160) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 09:53:20 +0000 From: Bob Liu To: linux-block@vger.kernel.org Cc: linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, martin.petersen@oracle.com, shirley.ma@oracle.com, allison.henderson@oracle.com, david@fromorbit.com, darrick.wong@oracle.com, hch@infradead.org, adilger@dilger.ca, Bob Liu Subject: [RFC PATCH v2 2/9] block: add rd_hint to bio and request Date: Wed, 13 Feb 2019 17:50:37 +0800 Message-Id: <20190213095044.29628-3-bob.liu@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190213095044.29628-1-bob.liu@oracle.com> References: <20190213095044.29628-1-bob.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9165 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130072 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP rd_hint is a bitmap for stacked layer support(see patch 4/9), set a bit to 1 means already read from the corresponding mirror device. rd_hint will be set properly recording read i/o went to which real device during end_bio(). If the upper layer want to retry other mirrors, just preserve the returned bi_rd_hint and resubmit bio. The upper layer e.g fs can set bitmap_zero(rd_hint) if don't care about alt mirror device retry feature which is also the default setting. Signed-off-by: Bob Liu --- Documentation/block/biodoc.txt | 3 +++ block/bio.c | 1 + block/blk-core.c | 1 + block/blk-merge.c | 6 ++++++ block/bounce.c | 1 + drivers/md/raid1.c | 1 + include/linux/blk_types.h | 1 + include/linux/blkdev.h | 1 + 8 files changed, 15 insertions(+) diff --git a/Documentation/block/biodoc.txt b/Documentation/block/biodoc.txt index ac18b488cb5e..c6b5dfc9314b 100644 --- a/Documentation/block/biodoc.txt +++ b/Documentation/block/biodoc.txt @@ -430,6 +430,7 @@ struct bio { struct bio *bi_next; /* request queue link */ struct block_device *bi_bdev; /* target device */ unsigned long bi_flags; /* status, command, etc */ + DECLARE_BITMAP(bi_rd_hint, BLKDEV_MAX_MIRRORS); /* bio read hint */ unsigned long bi_opf; /* low bits: r/w, high: priority */ unsigned int bi_vcnt; /* how may bio_vec's */ @@ -464,6 +465,8 @@ With this multipage bio design: (e.g a 1MB bio_vec needs to be handled in max 128kB chunks for IDE) [TBD: Should preferably also have a bi_voffset and bi_vlen to avoid modifying bi_offset an len fields] +- bi_rd_hint is an in/out bitmap parameter, set a bit to 1 means already read + from the corresponding mirror device. (*) unrelated merges -- a request ends up containing two or more bios that didn't originate from the same place. diff --git a/block/bio.c b/block/bio.c index 4db1008309ed..0e97d75edbd4 100644 --- a/block/bio.c +++ b/block/bio.c @@ -606,6 +606,7 @@ void __bio_clone_fast(struct bio *bio, struct bio *bio_src) bio->bi_opf = bio_src->bi_opf; bio->bi_ioprio = bio_src->bi_ioprio; bio->bi_write_hint = bio_src->bi_write_hint; + bitmap_copy(bio->bi_rd_hint, bio_src->bi_rd_hint, BLKDEV_MAX_MIRRORS); bio->bi_iter = bio_src->bi_iter; bio->bi_io_vec = bio_src->bi_io_vec; diff --git a/block/blk-core.c b/block/blk-core.c index b838c6dc5357..c93162b7140c 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -742,6 +742,7 @@ void blk_init_request_from_bio(struct request *req, struct bio *bio) req->__sector = bio->bi_iter.bi_sector; req->ioprio = bio_prio(bio); req->write_hint = bio->bi_write_hint; + bitmap_copy(req->rd_hint, bio->bi_rd_hint, BLKDEV_MAX_MIRRORS); blk_rq_bio_prep(req->q, req, bio); } EXPORT_SYMBOL_GPL(blk_init_request_from_bio); diff --git a/block/blk-merge.c b/block/blk-merge.c index 71e9ac03f621..58982a80eca8 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -745,6 +745,9 @@ static struct request *attempt_merge(struct request_queue *q, if (req->write_hint != next->write_hint) return NULL; + if (!bitmap_equal(req->rd_hint, next->rd_hint, BLKDEV_MAX_MIRRORS)) + return NULL; + if (req->ioprio != next->ioprio) return NULL; @@ -877,6 +880,9 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio) if (rq->write_hint != bio->bi_write_hint) return false; + if (!bitmap_equal(rq->rd_hint, bio->bi_rd_hint, BLKDEV_MAX_MIRRORS)) + return false; + if (rq->ioprio != bio_prio(bio)) return false; diff --git a/block/bounce.c b/block/bounce.c index ffb9e9ecfa7e..fba66e06b735 100644 --- a/block/bounce.c +++ b/block/bounce.c @@ -250,6 +250,7 @@ static struct bio *bounce_clone_bio(struct bio *bio_src, gfp_t gfp_mask, bio->bi_opf = bio_src->bi_opf; bio->bi_ioprio = bio_src->bi_ioprio; bio->bi_write_hint = bio_src->bi_write_hint; + bitmap_copy(bio->bi_rd_hint, bio_src->bi_rd_hint, BLKDEV_MAX_MIRRORS); bio->bi_iter.bi_sector = bio_src->bi_iter.bi_sector; bio->bi_iter.bi_size = bio_src->bi_iter.bi_size; diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index 1d54109071cc..1e5a51f22332 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -1103,6 +1103,7 @@ static void alloc_behind_master_bio(struct r1bio *r1_bio, } behind_bio->bi_write_hint = bio->bi_write_hint; + bitmap_copy(behind_bio->bi_rd_hint, bio->bi_rd_hint, BLKDEV_MAX_MIRRORS); while (i < vcnt && size) { struct page *page; diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index d66bf5f32610..49bdd96e2623 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -151,6 +151,7 @@ struct bio { unsigned short bi_flags; /* status, etc and bvec pool number */ unsigned short bi_ioprio; unsigned short bi_write_hint; + DECLARE_BITMAP(bi_rd_hint, BLKDEV_MAX_MIRRORS); blk_status_t bi_status; u8 bi_partno; diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 0191dc4d3f2d..0a1e93b282c4 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -214,6 +214,7 @@ struct request { #endif unsigned short write_hint; + DECLARE_BITMAP(rd_hint, BLKDEV_MAX_MIRRORS); unsigned short ioprio; void *special; /* opaque pointer available for LLD use */ From patchwork Wed Feb 13 09:50:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Liu X-Patchwork-Id: 10809483 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7452C1575 for ; Wed, 13 Feb 2019 09:53:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 64F362CC25 for ; Wed, 13 Feb 2019 09:53:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5976A2CC29; Wed, 13 Feb 2019 09:53:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 194A02CBD8 for ; Wed, 13 Feb 2019 09:53:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387651AbfBMJxe (ORCPT ); Wed, 13 Feb 2019 04:53:34 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:53130 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728665AbfBMJxd (ORCPT ); Wed, 13 Feb 2019 04:53:33 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1D9mp4X163899; Wed, 13 Feb 2019 09:53:28 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=8RLfCTrDcrzJnlGUfPMBd1ABpR9q+jsNRstFzMVJu+o=; b=w81dAFmofZMmWa21i3wr9TmwSX5qfFtjkthb0W67XpL7Xq1/JBecgIgOlRDNXUCsFFnP hs7CU/nrWETS5Sgwz2qeIQhK6KJ2jMyt12vuhZzxAQtQ3jhFYWUGpFSxk3XEt2nPcFVx f9Y3T5sSobraXLu5rJInoEnyacREzfjMZJt3hD2GbytxIvxDypJNnlzmfEDs+2r7uOUp 0Rec377BjerVyUkGBs0rZxbNg+9OSavPo6nV1q/uq5VJC1dThrNfVtIYZwbS5+BHYey1 CvsfbZC8FuGcF/qIRJljPJfdcZtZ5n7439i+EuU5w9GO6AgcgkMs6DZq84Y9i883PxE1 7Q== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2120.oracle.com with ESMTP id 2qhree11s2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Feb 2019 09:53:27 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x1D9rQ1K029677 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Feb 2019 09:53:26 GMT Received: from abhmp0006.oracle.com (abhmp0006.oracle.com [141.146.116.12]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x1D9rQ9O000533; Wed, 13 Feb 2019 09:53:26 GMT Received: from localhost.localdomain (/116.239.187.160) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 09:53:25 +0000 From: Bob Liu To: linux-block@vger.kernel.org Cc: linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, martin.petersen@oracle.com, shirley.ma@oracle.com, allison.henderson@oracle.com, david@fromorbit.com, darrick.wong@oracle.com, hch@infradead.org, adilger@dilger.ca, Bob Liu Subject: [RFC PATCH v2 3/9] md:raid1: set mirrors correctly Date: Wed, 13 Feb 2019 17:50:38 +0800 Message-Id: <20190213095044.29628-4-bob.liu@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190213095044.29628-1-bob.liu@oracle.com> References: <20190213095044.29628-1-bob.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9165 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=912 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130072 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In stack layer case, the mirror of upper layer device is the sum of mirrors of all lower layer devices. Signed-off-by: Bob Liu --- drivers/md/raid1.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index 1e5a51f22332..0de28714e9b5 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -3050,6 +3050,7 @@ static int raid1_run(struct mddev *mddev) struct md_rdev *rdev; int ret; bool discard_supported = false; + unsigned long mirrors = 0; if (mddev->level != 1) { pr_warn("md/raid1:%s: raid level not set to mirroring (%d)\n", @@ -3084,11 +3085,15 @@ static int raid1_run(struct mddev *mddev) rdev_for_each(rdev, mddev) { if (!mddev->gendisk) continue; + mirrors += blk_queue_get_mirrors(bdev_get_queue(rdev->bdev)); disk_stack_limits(mddev->gendisk, rdev->bdev, rdev->data_offset << 9); if (blk_queue_discard(bdev_get_queue(rdev->bdev))) discard_supported = true; } + if (mddev->queue) + if (!blk_queue_set_mirrors(mddev->queue, mirrors)) + return -EINVAL; mddev->degraded = 0; for (i=0; i < conf->raid_disks; i++) From patchwork Wed Feb 13 09:50:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Liu X-Patchwork-Id: 10809495 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2B6AC13B5 for ; Wed, 13 Feb 2019 09:53:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 19D412CB22 for ; Wed, 13 Feb 2019 09:53:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0DC102CBDD; Wed, 13 Feb 2019 09:53:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 55DB72CB22 for ; Wed, 13 Feb 2019 09:53:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387762AbfBMJxk (ORCPT ); Wed, 13 Feb 2019 04:53:40 -0500 Received: from aserp2130.oracle.com ([141.146.126.79]:58376 "EHLO aserp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2387608AbfBMJxj (ORCPT ); Wed, 13 Feb 2019 04:53:39 -0500 Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1D9mvIb165103; Wed, 13 Feb 2019 09:53:31 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=jfJEGyUia68X02HRL/FFy0BAdRTlo865x0zKmnFbipc=; b=cPEOtW0IK+izQ10dNCphR8MF8OqxZOQVKzmYRnwI2InBY5hgzuFbZUnvxPGrYYie+2KG UuoRXAsndS3StgRGkxEEEZkGEI+TDK32mbJ/A9HWcEcS2WWIfMH1k/hht3jDBeaQVa3e HsrJXVeTMC8Z9m5xm6RLAf0/SSRdOFrHPiEwmKuzS4FMW9A8kwKGwkNPW5ARQ9L36zx/ 4G68qU2XJiHaoA4D4zw5n1UEHHRwanm2SNC7kTd+aqlJ+fLKddjGXaiDv9T8DCJo0apO V0wpgOWWdPdVn3mBNGAKdLE/qc+ScRtBsaQs7nPosMzNCQPkClLooZ/A5lws5X1D6fji 3g== Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by aserp2130.oracle.com with ESMTP id 2qhre5gyut-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Feb 2019 09:53:31 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x1D9rTJ6010256 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Feb 2019 09:53:30 GMT Received: from abhmp0006.oracle.com (abhmp0006.oracle.com [141.146.116.12]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x1D9rTXx005217; Wed, 13 Feb 2019 09:53:29 GMT Received: from localhost.localdomain (/116.239.187.160) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 09:53:28 +0000 From: Bob Liu To: linux-block@vger.kernel.org Cc: linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, martin.petersen@oracle.com, shirley.ma@oracle.com, allison.henderson@oracle.com, david@fromorbit.com, darrick.wong@oracle.com, hch@infradead.org, adilger@dilger.ca, Bob Liu Subject: [RFC PATCH v2 4/9] md:raid1: rd_hint support and consider stacked layer case Date: Wed, 13 Feb 2019 17:50:39 +0800 Message-Id: <20190213095044.29628-5-bob.liu@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190213095044.29628-1-bob.liu@oracle.com> References: <20190213095044.29628-1-bob.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9165 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130072 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP rd_hint is a bit map for stacked md layer supporting. When submit bio to a lower md layer, the bio->bi_rd_hint should be split according mirror number of each device of lower layer. And merge bio->bi_rd_hint in the end path vise versa. For a two layer stacked md case like: /dev/md0 / | \ /dev/md1-a /dev/md1-b /dev/md1-c / \ / | \ / | \ /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh - 1) First the top layer sumbit bio with bi_rd_hint = [00 000 000], then the value of bi_rd_hint changed as below when bio goes to lower layer. [00 000 000] / | \ [00] [000] [000] / \ / | \ / | \ [0] [0] [0] [0] [0] [0] [0] [0] - 2) i/o may goes to /dev/sda at first: [1] [0] [0] [0] [0] [0] [0] [0] \ / \ | / \ | / [10] [000] [000] \ | / [10 000 000] The top layer will get bio->bi_rd_hint = [10 000 000] - 3) Fs check the data is corrupt, resumbit bio with bi_rd_hint = [10 000 000] [10 000 000] / | \ [10] [000] [000] / \ / | \ / | \ [1] [0] [0] [0] [0] [0] [0] [0] - 4) i/o can go to any dev except /dev/sda(already tried), assum goes to /dev/sdg this time. [1] [0] [0] [0] [0] [0] [1] [0] \ / \ | / \ | / [10] [000] [010] \ | / [10 000 010] The top layer will get bio->bi_rd_hint = [10 000 010], which means we already tried /dev/sda and /dev/sdg. - 5) If the data is corrupt again, resumbit bio with bi_rd_hint = [10 000 010]. Loop until all mirrors are tried.. Signed-off-by: Bob Liu --- drivers/md/raid1.c | 117 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 116 insertions(+), 1 deletion(-) diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index 0de28714e9b5..75fde3a3fd3d 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -325,6 +325,41 @@ static int find_bio_disk(struct r1bio *r1_bio, struct bio *bio) return mirror; } +/* merge children's rd hint to master bio */ +static void raid1_merge_rd_hint(struct bio *bio) +{ + struct r1bio *r1_bio = bio->bi_private; + struct r1conf *conf = r1_bio->mddev->private; + struct md_rdev *tmp_rdev = NULL; + int i = conf->raid_disks - 1; + int cnt = 0; + int read_disk = r1_bio->read_disk; + DECLARE_BITMAP(tmp_bitmap, BLKDEV_MAX_MIRRORS); + + if (!r1_bio->master_bio) + return; + + /* ignore replace case now */ + if (read_disk > conf->raid_disks - 1) + read_disk = r1_bio->read_disk - conf->raid_disks; + + for (; i >= 0; i--) { + tmp_rdev = conf->mirrors[i].rdev; + if (i == read_disk) + break; + cnt += blk_queue_get_mirrors(bdev_get_queue(tmp_rdev->bdev)); + } + + /* init map properly from most lower layer */ + if (blk_queue_get_mirrors(bdev_get_queue(tmp_rdev->bdev)) == 1) + bitmap_set(bio->bi_rd_hint, 0, 1); + + bitmap_shift_left(tmp_bitmap, bio->bi_rd_hint, cnt, BLKDEV_MAX_MIRRORS); + bitmap_or(r1_bio->master_bio->bi_rd_hint, + r1_bio->master_bio->bi_rd_hint, tmp_bitmap, + BLKDEV_MAX_MIRRORS); +} + static void raid1_end_read_request(struct bio *bio) { int uptodate = !bio->bi_status; @@ -332,6 +367,7 @@ static void raid1_end_read_request(struct bio *bio) struct r1conf *conf = r1_bio->mddev->private; struct md_rdev *rdev = conf->mirrors[r1_bio->read_disk].rdev; + raid1_merge_rd_hint(bio); /* * this branch is our 'one mirror IO has finished' event handler: */ @@ -539,6 +575,37 @@ static sector_t align_to_barrier_unit_end(sector_t start_sector, return len; } +static long choose_disk_from_rd_hint(struct r1conf *conf, struct r1bio *r1_bio) +{ + struct md_rdev *tmp_rdev; + unsigned long bit, cnt; + struct bio *bio = r1_bio->master_bio; + int mirror = conf->raid_disks - 1; + + cnt = blk_queue_get_mirrors(r1_bio->mddev->queue); + /* Find a never-readed device */ + bit = bitmap_find_next_zero_area(bio->bi_rd_hint, cnt, 0, 1, 0); + if (bit >= cnt) + /* Already tried all mirrors */ + return -1; + + /* Decide this device belongs to which mirror for stacked-layer raid + * devices. */ + cnt = 0; + for ( ; mirror >= 0; mirror--) { + tmp_rdev = conf->mirrors[mirror].rdev; + cnt += blk_queue_get_mirrors(bdev_get_queue(tmp_rdev->bdev)); + /* bit start from 0, while mirrors start from 1. So should compare + * with (bit + 1) */ + if (cnt >= (bit + 1)) { + return mirror; + } + } + + /* Should not arrive here. */ + return -1; +} + /* * This routine returns the disk from which the requested read should * be done. There is a per-array 'next expected sequential IO' sector @@ -566,6 +633,7 @@ static int read_balance(struct r1conf *conf, struct r1bio *r1_bio, int *max_sect struct md_rdev *rdev; int choose_first; int choose_next_idle; + int max_disks; rcu_read_lock(); /* @@ -593,7 +661,18 @@ static int read_balance(struct r1conf *conf, struct r1bio *r1_bio, int *max_sect else choose_first = 0; - for (disk = 0 ; disk < conf->raid_disks * 2 ; disk++) { + if (!bitmap_empty(r1_bio->master_bio->bi_rd_hint, BLKDEV_MAX_MIRRORS)) { + disk = choose_disk_from_rd_hint(conf, r1_bio); + if (disk < 0) + return -1; + + /* Use the specific disk */ + max_disks = disk + 1; + } else { + disk = 0; + max_disks = conf->raid_disks * 2; + } + for (; disk < max_disks; disk++) { sector_t dist; sector_t first_bad; int bad_sectors; @@ -1186,6 +1265,34 @@ alloc_r1bio(struct mddev *mddev, struct bio *bio) return r1_bio; } +static void raid1_split_rd_hint(struct bio *bio) +{ + struct r1bio *r1_bio = bio->bi_private; + struct r1conf *conf = r1_bio->mddev->private; + unsigned int cnt = 0; + DECLARE_BITMAP(tmp_bitmap, BLKDEV_MAX_MIRRORS); + + int i = conf->raid_disks - 1; + struct md_rdev *tmp_rdev = NULL; + + for (; i >= 0; i--) { + tmp_rdev = conf->mirrors[i].rdev; + if (i == r1_bio->read_disk) + break; + cnt += blk_queue_get_mirrors(bdev_get_queue(tmp_rdev->bdev)); + } + + bitmap_zero(tmp_bitmap, BLKDEV_MAX_MIRRORS); + bitmap_shift_right(bio->bi_rd_hint, r1_bio->master_bio->bi_rd_hint, cnt, + BLKDEV_MAX_MIRRORS); + + cnt = blk_queue_get_mirrors(bdev_get_queue(tmp_rdev->bdev)); + bitmap_set(tmp_bitmap, 0, cnt); + + bitmap_and(bio->bi_rd_hint, bio->bi_rd_hint, tmp_bitmap, + BLKDEV_MAX_MIRRORS); +} + static void raid1_read_request(struct mddev *mddev, struct bio *bio, int max_read_sectors, struct r1bio *r1_bio) { @@ -1199,6 +1306,7 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio, int rdisk; bool print_msg = !!r1_bio; char b[BDEVNAME_SIZE]; + bool auto_select_mirror; /* * If r1_bio is set, we are blocking the raid1d thread @@ -1230,6 +1338,8 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio, else init_r1bio(r1_bio, mddev, bio); r1_bio->sectors = max_read_sectors; + auto_select_mirror = bitmap_empty(r1_bio->master_bio->bi_rd_hint, BLKDEV_MAX_MIRRORS); + /* * make_request() can abort the operation when read-ahead is being @@ -1238,6 +1348,9 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio, rdisk = read_balance(conf, r1_bio, &max_sectors); if (rdisk < 0) { + if (auto_select_mirror) + bitmap_set(r1_bio->master_bio->bi_rd_hint, 0, BLKDEV_MAX_MIRRORS); + /* couldn't find anywhere to read from */ if (print_msg) { pr_crit_ratelimited("md/raid1:%s: %s: unrecoverable I/O read error for block %llu\n", @@ -1292,6 +1405,8 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio, test_bit(R1BIO_FailFast, &r1_bio->state)) read_bio->bi_opf |= MD_FAILFAST; read_bio->bi_private = r1_bio; + /* rd_hint of read_bio is a subset of master_bio. */ + raid1_split_rd_hint(read_bio); if (mddev->gendisk) trace_block_bio_remap(read_bio->bi_disk->queue, read_bio, From patchwork Wed Feb 13 09:50:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Liu X-Patchwork-Id: 10809493 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 69DC813B5 for ; Wed, 13 Feb 2019 09:53:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 592B02CB22 for ; Wed, 13 Feb 2019 09:53:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4D6A62CBD8; Wed, 13 Feb 2019 09:53:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0664C2CB22 for ; Wed, 13 Feb 2019 09:53:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387608AbfBMJxl (ORCPT ); Wed, 13 Feb 2019 04:53:41 -0500 Received: from userp2130.oracle.com ([156.151.31.86]:48876 "EHLO userp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2387742AbfBMJxl (ORCPT ); Wed, 13 Feb 2019 04:53:41 -0500 Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1D9mlgD158547; Wed, 13 Feb 2019 09:53:35 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=hMbuxyPzORcwzH0eUzMPHjc5zsz92jBZFARf1sdQ3s0=; b=lJAC9QXeZbFwhPIEtMceRxnqxxlGuIgM4idbBLeJ7iC8EyuoJBblqOkQD7TmUKkqR7qD 9CX9vJeVxCwo+FEhkH5OD/o8Rd2ErxsZ/K5iuWcvsFJ7/Mxg/mwujuiL1arZsJJ6r6GS ODGk8Shh8RL079/r/yiTWDeMt10D7Igv6h9O7wYaDYgPMbPBOWBM0ECqJz+VguwkrS2b Ll3HKa03R/w5YRSItpKxJR0/XxCC8/71VTrFxSsxaQxIAiMe3zYNvM0hA4ssDwL+nP6s 2N7bVlqHBDNI+wW/aeqTcsmfDBQrchoocwQTFDxJw+QzykEWswFYDxaXM1hizsIyfG7V 3A== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2130.oracle.com with ESMTP id 2qhrekh0vv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Feb 2019 09:53:35 +0000 Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x1D9rYYi029957 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Feb 2019 09:53:34 GMT Received: from abhmp0006.oracle.com (abhmp0006.oracle.com [141.146.116.12]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id x1D9rY1b026559; Wed, 13 Feb 2019 09:53:34 GMT Received: from localhost.localdomain (/116.239.187.160) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 09:53:33 +0000 From: Bob Liu To: linux-block@vger.kernel.org Cc: linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, martin.petersen@oracle.com, shirley.ma@oracle.com, allison.henderson@oracle.com, david@fromorbit.com, darrick.wong@oracle.com, hch@infradead.org, adilger@dilger.ca Subject: [RFC PATCH v2 5/9] Add b_alt_retry to xfs_buf Date: Wed, 13 Feb 2019 17:50:40 +0800 Message-Id: <20190213095044.29628-6-bob.liu@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190213095044.29628-1-bob.liu@oracle.com> References: <20190213095044.29628-1-bob.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9165 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130072 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Allison Henderson This patch adds b_alt_retry boolean to xfs_buf. We will use this to enable alternate device retry when the bio completes for single buffer bios. At this time, we do not yet support alternate device retry for multi buffer bio Signed-off-by: Allison Henderson --- fs/xfs/xfs_buf.c | 8 ++++++++ fs/xfs/xfs_buf.h | 1 + 2 files changed, 9 insertions(+) diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c index 4f5f2ff3f70f..e2683c8e868c 100644 --- a/fs/xfs/xfs_buf.c +++ b/fs/xfs/xfs_buf.c @@ -1409,6 +1409,14 @@ xfs_buf_ioapply_map( flush_kernel_vmap_range(bp->b_addr, xfs_buf_vmap_len(bp)); } + + /* + * At the moment, we only support alternate + * device retry on single bio buffers + */ + if (size == 0) + bp->b_alt_retry = true; + submit_bio(bio); if (size) goto next_chunk; diff --git a/fs/xfs/xfs_buf.h b/fs/xfs/xfs_buf.h index b9f5511ea998..989b97a17486 100644 --- a/fs/xfs/xfs_buf.h +++ b/fs/xfs/xfs_buf.h @@ -198,6 +198,7 @@ typedef struct xfs_buf { int b_last_error; const struct xfs_buf_ops *b_ops; + bool b_alt_retry; /* toggle alt device retry */ } xfs_buf_t; /* Finding and Reading Buffers */ From patchwork Wed Feb 13 09:50:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Liu X-Patchwork-Id: 10809503 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EF6CA746 for ; Wed, 13 Feb 2019 09:53:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DEED82CB22 for ; Wed, 13 Feb 2019 09:53:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D2ED42CBD7; Wed, 13 Feb 2019 09:53:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7DFBC2CB22 for ; Wed, 13 Feb 2019 09:53:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387599AbfBMJxq (ORCPT ); Wed, 13 Feb 2019 04:53:46 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:53302 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2387821AbfBMJxp (ORCPT ); Wed, 13 Feb 2019 04:53:45 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1D9mi8e163835; Wed, 13 Feb 2019 09:53:39 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=djq8BbFMatF5YBAugCJjJvi0D9KcFtTqCJonR8JZfsA=; b=DBEDCLXMe4fSmDntnCJNF8b5JT2JIGK9LXq9nKSxRZmSbgm/AMhuceh8GFffJ4U+oNAn wxgkmAMmw1xhl+CaegZH7v9LFAtDWg5P5Yx7188Z62hvrSmY9Ko1CWjL/bI7V9qRtQ34 QKDQo0h1jNVR137jFtHaa6PLyKnsb0k6saFxgYdtbrl4Izt5eWZcuUForZ49juqrSlmv LFNSkFSdqk44B1ZcvdyiaC1JjoeqTOVRCskGV4EMcskj9sDWPmOpvpUV826T49WKm6fD 6vDvTQvF2qRPqpEifX2yeS+V932yyqfFLpRued61f4UYYHfyvdx+c5bB2UunKv+/5ZHL Xw== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2120.oracle.com with ESMTP id 2qhree11sq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Feb 2019 09:53:39 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x1D9rbnT021083 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Feb 2019 09:53:38 GMT Received: from abhmp0006.oracle.com (abhmp0006.oracle.com [141.146.116.12]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x1D9rbGh000718; Wed, 13 Feb 2019 09:53:37 GMT Received: from localhost.localdomain (/116.239.187.160) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 09:53:36 +0000 From: Bob Liu To: linux-block@vger.kernel.org Cc: linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, martin.petersen@oracle.com, shirley.ma@oracle.com, allison.henderson@oracle.com, david@fromorbit.com, darrick.wong@oracle.com, hch@infradead.org, adilger@dilger.ca Subject: [RFC PATCH v2 6/9] xfs: Add b_rd_hint to xfs_buf Date: Wed, 13 Feb 2019 17:50:41 +0800 Message-Id: <20190213095044.29628-7-bob.liu@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190213095044.29628-1-bob.liu@oracle.com> References: <20190213095044.29628-1-bob.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9165 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=988 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130072 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Allison Henderson This patch adds a new field b_rd_hint to xfs_buf. We will need this to properly initialize the new bio->bi_rw_hint when submitting the read request. Signed-off-by: Allison Henderson --- fs/xfs/xfs_buf.c | 6 +++++- fs/xfs/xfs_buf.h | 12 ++++++++++++ 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c index e2683c8e868c..6098195ecaf4 100644 --- a/fs/xfs/xfs_buf.c +++ b/fs/xfs/xfs_buf.c @@ -1338,8 +1338,11 @@ xfs_buf_bio_end_io( if (!bp->b_error && xfs_buf_is_vmapped(bp) && (bp->b_flags & XBF_READ)) invalidate_kernel_vmap_range(bp->b_addr, xfs_buf_vmap_len(bp)); - if (atomic_dec_and_test(&bp->b_io_remaining) == 1) + if (atomic_dec_and_test(&bp->b_io_remaining) == 1) { + if (bp->b_flags & XBF_RW_HINT) + bitmap_copy(bp->b_rd_hint, bio->bi_rd_hint, BLKDEV_MAX_MIRRORS); xfs_buf_ioend_async(bp); + } bio_put(bio); } @@ -1385,6 +1388,7 @@ xfs_buf_ioapply_map( bio->bi_iter.bi_sector = sector; bio->bi_end_io = xfs_buf_bio_end_io; bio->bi_private = bp; + bitmap_copy(bio->bi_rd_hint, bp->b_rd_hint, BLKDEV_MAX_MIRRORS); bio_set_op_attrs(bio, op, op_flags); for (; size && nr_pages; nr_pages--, page_index++) { diff --git a/fs/xfs/xfs_buf.h b/fs/xfs/xfs_buf.h index 989b97a17486..af9bdff29e66 100644 --- a/fs/xfs/xfs_buf.h +++ b/fs/xfs/xfs_buf.h @@ -40,6 +40,7 @@ typedef enum { #define XBF_SYNCIO (1 << 10)/* treat this buffer as synchronous I/O */ #define XBF_FUA (1 << 11)/* force cache write through mode */ #define XBF_FLUSH (1 << 12)/* flush the disk cache before a write */ +#define XBF_RW_HINT (1 << 13)/* Read/write hint used for alt dev retry */ /* flags used only as arguments to access routines */ #define XBF_TRYLOCK (1 << 16)/* lock requested, but do not wait */ @@ -65,6 +66,7 @@ typedef unsigned int xfs_buf_flags_t; { XBF_SYNCIO, "SYNCIO" }, \ { XBF_FUA, "FUA" }, \ { XBF_FLUSH, "FLUSH" }, \ + { XBF_RW_HINT, "RW_HINT" }, \ { XBF_TRYLOCK, "TRYLOCK" }, /* should never be set */\ { XBF_UNMAPPED, "UNMAPPED" }, /* ditto */\ { _XBF_PAGES, "PAGES" }, \ @@ -197,6 +199,16 @@ typedef struct xfs_buf { unsigned long b_first_retry_time; /* in jiffies */ int b_last_error; + /* + * Bitmask used by block device for alternate device retry + * + * To retry a read with the next device, resubmit the bio with + * the bi_rd_hint returned from the last read. + * Otherwise use bitmap_zero() if we don't care about alt mirror + * device retry. + */ + DECLARE_BITMAP(b_rd_hint, BLKDEV_MAX_MIRRORS); + const struct xfs_buf_ops *b_ops; bool b_alt_retry; /* toggle alt device retry */ } xfs_buf_t; From patchwork Wed Feb 13 09:50:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Liu X-Patchwork-Id: 10809505 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C6ADA746 for ; Wed, 13 Feb 2019 09:53:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B40552CB22 for ; Wed, 13 Feb 2019 09:53:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A80822CBD7; Wed, 13 Feb 2019 09:53:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5488E2CB22 for ; Wed, 13 Feb 2019 09:53:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2387816AbfBMJxt (ORCPT ); Wed, 13 Feb 2019 04:53:49 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:53348 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2387886AbfBMJxs (ORCPT ); Wed, 13 Feb 2019 04:53:48 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1D9mgcv163804; Wed, 13 Feb 2019 09:53:42 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=Ad+zZ6/ZPUMB/Kv9Vg5krh7HY7WfIeBOyxxzOXDmo7w=; b=ctttriTrfRyyTI0foAO+2oPHy9NIG15gjAcHNkL9jzu/f49ylX24i2f7alNXBNBUz8bm yg4eNM8+Gl6DJq1VuaZhIJNkctIp5g7WbS9N4F7/mR7eS79R4oy6nExMJC2Brdj5SmA3 H4GahO6Kt8lcGWiq7ug5VOF7h4G7fxscQyL+G7iLWrmDg6XXWybirHN7GbtGp/Reh7TB 2AZuYOlq03DJIz4BJEzGs/yoGS0E7VJiALdnHYczh+wtZ2AfU8gTBUBErKFh8jxoLaJp uq64g6tGz/3sWhwZ3nP5x9lv5L6XSfyu1PpvrXtbJpC9SrTtUvKIYuprserIXqPUQZ/n 1A== Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp2120.oracle.com with ESMTP id 2qhree11sx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Feb 2019 09:53:42 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x1D9re4n021320 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Feb 2019 09:53:41 GMT Received: from abhmp0006.oracle.com (abhmp0006.oracle.com [141.146.116.12]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x1D9rea9000731; Wed, 13 Feb 2019 09:53:40 GMT Received: from localhost.localdomain (/116.239.187.160) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 09:53:40 +0000 From: Bob Liu To: linux-block@vger.kernel.org Cc: linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, martin.petersen@oracle.com, shirley.ma@oracle.com, allison.henderson@oracle.com, david@fromorbit.com, darrick.wong@oracle.com, hch@infradead.org, adilger@dilger.ca Subject: [RFC PATCH v2 7/9] xfs: Add device retry Date: Wed, 13 Feb 2019 17:50:42 +0800 Message-Id: <20190213095044.29628-8-bob.liu@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190213095044.29628-1-bob.liu@oracle.com> References: <20190213095044.29628-1-bob.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9165 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130072 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Allison Henderson Check to see if the _xfs_buf_read fails. If so loop over the available mirrors and retry the read Signed-off-by: Allison Henderson --- fs/xfs/xfs_buf.c | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c index 6098195ecaf4..2c250221cb78 100644 --- a/fs/xfs/xfs_buf.c +++ b/fs/xfs/xfs_buf.c @@ -21,6 +21,7 @@ #include #include #include +#include #include "xfs_format.h" #include "xfs_log_format.h" @@ -824,6 +825,9 @@ xfs_buf_read_map( const struct xfs_buf_ops *ops) { struct xfs_buf *bp; + struct request_queue *q; + unsigned long i; + int retries = 0; flags |= XBF_READ; @@ -836,7 +840,27 @@ xfs_buf_read_map( if (!(bp->b_flags & XBF_DONE)) { XFS_STATS_INC(target->bt_mount, xb_get_read); bp->b_ops = ops; - _xfs_buf_read(bp, flags); + q = bdev_get_queue(bp->b_target->bt_bdev); + + if (bp->b_alt_retry) + retries = blk_queue_get_mirrors(q); + + for (i = 0; i <= retries; i++) { + bp->b_error = 0; + _xfs_buf_read(bp, flags); + + switch (bp->b_error) { + case -EIO: + case -EFSCORRUPTED: + case -EFSBADCRC: + /* loop again */ + continue; + default: + goto retry_done; + } + + } +retry_done: return bp; } From patchwork Wed Feb 13 09:50:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Liu X-Patchwork-Id: 10809517 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F3D0113B5 for ; Wed, 13 Feb 2019 09:53:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E1D2A28761 for ; Wed, 13 Feb 2019 09:53:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D53A12A5AD; Wed, 13 Feb 2019 09:53:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8512528761 for ; Wed, 13 Feb 2019 09:53:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2391386AbfBMJx4 (ORCPT ); Wed, 13 Feb 2019 04:53:56 -0500 Received: from aserp2130.oracle.com ([141.146.126.79]:58650 "EHLO aserp2130.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2387886AbfBMJx4 (ORCPT ); Wed, 13 Feb 2019 04:53:56 -0500 Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1D9mg99164813; Wed, 13 Feb 2019 09:53:50 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=4aVe1qcdo+EJ2dG63MtwaP+x1ZR6HAKRsDn8i3GaN9E=; b=ecxi0IdOKpnOGDmk6nXAIr/+fAC4uQ6O0kTcUsALwIGnNVV4If/S1lPzdxvnE7IZ9LId EwvODUlyD4/QoIKeWnsfcIvQYl0mipVKMAIsRB+WwF8k+BlctE0NXp6plBNtnBoR8dfE pk7uVosEL8u5F26wDp5cm0+6ukPvfMqgmcM0MivRdrGzD/UswVFTqtbGSIW5vZwIKBK1 XCZgSwjI7OZbTxUvUcljstjL262r+6YxqD+LfVN7JzryJQ6bKfOF4y8zFC8SQrYxpKcI Xrwutb2PENMRel0xWaMY1FRN4pXobUbOywNkWCe8ylqf6aJcorttPj0YbM+8xo8c7yoC jQ== Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by aserp2130.oracle.com with ESMTP id 2qhre5gyvw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Feb 2019 09:53:49 +0000 Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id x1D9rhR7010664 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Feb 2019 09:53:43 GMT Received: from abhmp0006.oracle.com (abhmp0006.oracle.com [141.146.116.12]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x1D9rhgM005276; Wed, 13 Feb 2019 09:53:43 GMT Received: from localhost.localdomain (/116.239.187.160) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 09:53:43 +0000 From: Bob Liu To: linux-block@vger.kernel.org Cc: linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, martin.petersen@oracle.com, shirley.ma@oracle.com, allison.henderson@oracle.com, david@fromorbit.com, darrick.wong@oracle.com, hch@infradead.org, adilger@dilger.ca Subject: [RFC PATCH v2 8/9] xfs: Rewrite retried read Date: Wed, 13 Feb 2019 17:50:43 +0800 Message-Id: <20190213095044.29628-9-bob.liu@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190213095044.29628-1-bob.liu@oracle.com> References: <20190213095044.29628-1-bob.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9165 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=883 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130072 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Allison Henderson If we had to try more than one mirror to get a successful read, then write that buffer back to correct the bad mirror Signed-off-by: Allison Henderson --- fs/xfs/xfs_buf.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c index 2c250221cb78..e54dbc776d15 100644 --- a/fs/xfs/xfs_buf.c +++ b/fs/xfs/xfs_buf.c @@ -861,6 +861,14 @@ xfs_buf_read_map( } retry_done: + + /* + * if we had to try more than one mirror to sucessfully read + * the buffer, write the buffer back + */ + if (!bp->b_error && i > 0) + xfs_bwrite(bp); + return bp; } From patchwork Wed Feb 13 09:50:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bob Liu X-Patchwork-Id: 10809511 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 225E213B5 for ; Wed, 13 Feb 2019 09:53:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0CEE828761 for ; Wed, 13 Feb 2019 09:53:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 00E7A2A2C7; Wed, 13 Feb 2019 09:53:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DD63628761 for ; Wed, 13 Feb 2019 09:53:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2391384AbfBMJxz (ORCPT ); Wed, 13 Feb 2019 04:53:55 -0500 Received: from userp2120.oracle.com ([156.151.31.85]:53472 "EHLO userp2120.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732639AbfBMJxy (ORCPT ); Wed, 13 Feb 2019 04:53:54 -0500 Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.27/8.16.0.27) with SMTP id x1D9mgQd163807; Wed, 13 Feb 2019 09:53:48 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2018-07-02; bh=btF1v4tjdgTzSKMBIbwAhAtSt7P45B069orpzvat5Jc=; b=qWLuD1DZD6/wUpoWBSTdHIxSeQKRBla1wZg8UPPgt5bqzTn5W9KPRlKvO8bAJn8EHY4/ PvvoQHDCUAwmLe/41X+ECE0oYqhMbyfh6nzT0AzNUBDMFxtk4e7zJPJZ2kvs/B+HoqK+ kCwnoGxysZLmsveQgorOIinbY5Epx0RBj4h3ziSm+l/j7bzJ9KkZZIM0GTlmXm8vO45P 3k6KfGRPm4to53iAH+bg02GgPuO5XaGVlI3ngciTQrfsu004DGT76UWwlRgjO7ZO+i1M dITJbA/FsRA9kNAdWHm6Tg5ttAhPla0QaIeJSK6uXyTYiMsJQVg9aTHMwgNLYa6dxgms fQ== Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp2120.oracle.com with ESMTP id 2qhree11te-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Feb 2019 09:53:48 +0000 Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id x1D9rlxn030542 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Feb 2019 09:53:47 GMT Received: from abhmp0006.oracle.com (abhmp0006.oracle.com [141.146.116.12]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id x1D9rkId000779; Wed, 13 Feb 2019 09:53:46 GMT Received: from localhost.localdomain (/116.239.187.160) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 13 Feb 2019 09:53:46 +0000 From: Bob Liu To: linux-block@vger.kernel.org Cc: linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, martin.petersen@oracle.com, shirley.ma@oracle.com, allison.henderson@oracle.com, david@fromorbit.com, darrick.wong@oracle.com, hch@infradead.org, adilger@dilger.ca Subject: [RFC PATCH v2 9/9] xfs: Add tracepoints and logging to alternate device retry Date: Wed, 13 Feb 2019 17:50:44 +0800 Message-Id: <20190213095044.29628-10-bob.liu@oracle.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190213095044.29628-1-bob.liu@oracle.com> References: <20190213095044.29628-1-bob.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=9165 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 priorityscore=1501 malwarescore=0 suspectscore=3 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1902130072 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Allison Henderson This patch adds new log entries and trace points to the alternate device retry error path Signed-off-by: Allison Henderson --- fs/xfs/xfs_buf.c | 10 ++++++++++ fs/xfs/xfs_buf.h | 1 + fs/xfs/xfs_trace.h | 6 +++++- 3 files changed, 16 insertions(+), 1 deletion(-) diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c index e54dbc776d15..1a0427137883 100644 --- a/fs/xfs/xfs_buf.c +++ b/fs/xfs/xfs_buf.c @@ -847,6 +847,11 @@ xfs_buf_read_map( for (i = 0; i <= retries; i++) { bp->b_error = 0; + + if (i > 0) + xfs_alert(bp->b_target->bt_mount, + "Retrying read from disk %lu",i); + _xfs_buf_read(bp, flags); switch (bp->b_error) { @@ -854,6 +859,11 @@ xfs_buf_read_map( case -EFSCORRUPTED: case -EFSBADCRC: /* loop again */ + trace_xfs_buf_ioretry(bp, _RET_IP_); + xfs_alert(bp->b_target->bt_mount, + "Read error:%d from disk number %lu", + bp->b_error, *bp->b_rd_hint); + continue; default: goto retry_done; diff --git a/fs/xfs/xfs_buf.h b/fs/xfs/xfs_buf.h index af9bdff29e66..69605a50c15e 100644 --- a/fs/xfs/xfs_buf.h +++ b/fs/xfs/xfs_buf.h @@ -306,6 +306,7 @@ extern void __xfs_buf_ioerror(struct xfs_buf *bp, int error, xfs_failaddr_t failaddr); #define xfs_buf_ioerror(bp, err) __xfs_buf_ioerror((bp), (err), __this_address) extern void xfs_buf_ioerror_alert(struct xfs_buf *, const char *func); +extern void xfs_buf_ioretry_alert(struct xfs_buf *, const char *func); extern int __xfs_buf_submit(struct xfs_buf *bp, bool); static inline int xfs_buf_submit(struct xfs_buf *bp) diff --git a/fs/xfs/xfs_trace.h b/fs/xfs/xfs_trace.h index 6fcc893dfc91..4b948cf2dd65 100644 --- a/fs/xfs/xfs_trace.h +++ b/fs/xfs/xfs_trace.h @@ -276,6 +276,7 @@ DECLARE_EVENT_CLASS(xfs_buf_class, __field(int, pincount) __field(unsigned, lockval) __field(unsigned, flags) + __field(unsigned short, rd_hint) __field(unsigned long, caller_ip) ), TP_fast_assign( @@ -289,10 +290,11 @@ DECLARE_EVENT_CLASS(xfs_buf_class, __entry->pincount = atomic_read(&bp->b_pin_count); __entry->lockval = bp->b_sema.count; __entry->flags = bp->b_flags; + __entry->rd_hint = bp->b_rd_hint; __entry->caller_ip = caller_ip; ), TP_printk("dev %d:%d bno 0x%llx nblks 0x%x hold %d pincount %d " - "lock %d flags %s caller %pS", + "lock %d flags %s rd_hint %hu caller %pS", MAJOR(__entry->dev), MINOR(__entry->dev), (unsigned long long)__entry->bno, __entry->nblks, @@ -300,6 +302,7 @@ DECLARE_EVENT_CLASS(xfs_buf_class, __entry->pincount, __entry->lockval, __print_flags(__entry->flags, "|", XFS_BUF_FLAGS), + __entry->rd_hint, (void *)__entry->caller_ip) ) @@ -312,6 +315,7 @@ DEFINE_BUF_EVENT(xfs_buf_free); DEFINE_BUF_EVENT(xfs_buf_hold); DEFINE_BUF_EVENT(xfs_buf_rele); DEFINE_BUF_EVENT(xfs_buf_iodone); +DEFINE_BUF_EVENT(xfs_buf_ioretry); DEFINE_BUF_EVENT(xfs_buf_submit); DEFINE_BUF_EVENT(xfs_buf_lock); DEFINE_BUF_EVENT(xfs_buf_lock_done);