From patchwork Fri May 22 12:18:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Coly Li X-Patchwork-Id: 11565493 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C8999138A for ; Fri, 22 May 2020 12:18:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B8DF3207F7 for ; Fri, 22 May 2020 12:18:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729016AbgEVMSw (ORCPT ); Fri, 22 May 2020 08:18:52 -0400 Received: from mx2.suse.de ([195.135.220.15]:52390 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728772AbgEVMSv (ORCPT ); Fri, 22 May 2020 08:18:51 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 2F168AF01; Fri, 22 May 2020 12:18:52 +0000 (UTC) From: Coly Li To: linux-bcache@vger.kernel.org Cc: linux-block@vger.kernel.org, Coly Li , Damien Le Moal , Hannes Reinecke , Johannes Thumshirn Subject: [RFC PATCH v4 1/3] bcache: export bcache zone information for zoned backing device Date: Fri, 22 May 2020 20:18:35 +0800 Message-Id: <20200522121837.109651-2-colyli@suse.de> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200522121837.109651-1-colyli@suse.de> References: <20200522121837.109651-1-colyli@suse.de> MIME-Version: 1.0 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org When using a zoned device e.g. SMR hard drive as the backing device, if bcache can export the zoned device information then it is possible to help the upper layer code to accelerate hot READ I/O requests. This patch adds the report_zones method for the bcache device which has zoned device as backing device. Now such bcache device can be treated as a zoned device, by configured to writethrough, writearound mode or none mode, zonefs can be formated on top of such bcache device. Here is a simple performance data for read requests via zonefs on top of bcache. The cache device of bcache is a 4TB NVMe SSD, the backing device is a 14TB host managed SMR hard drive. The formatted zonefs has 52155 zone files, 523 of them are for convential zones (1 zone is reserved for bcache super block and not reported), 51632 of them are for sequential zones. First run to read first 4KB from all the zone files with 50 processes, it takes 5 minutes 55 seconds. Second run takes 12 seconds only because all the read requests hit on cache device. 29 times faster is as expected for an ideal case when all READ I/Os hit on NVMe cache device. Besides providing report_zones method of the bcache gendisk structure, this patch also initializes the following zoned device attribution for the bcache device, - zones number: the total zones number minus reserved zone(s) for bcache super block. - zone size: same size as reported from the underlying zoned device - zone mode: same mode as reported from the underlying zoned device Currently blk_revalidate_disk_zones() does not accept non-mq drivers, so all the above attribution are initialized mannally in bcache code. This patch just provides the report_zones method only. Handling all zone management commands will be addressed in following patches. Signed-off-by: Coly Li Cc: Damien Le Moal Cc: Hannes Reinecke Cc: Johannes Thumshirn --- Changelog: v4: the version without any generic block layer change. v3: the version depends on other generic block layer changes. v2: an improved version for comments. v1: initial version. drivers/md/bcache/bcache.h | 10 ++++ drivers/md/bcache/request.c | 69 ++++++++++++++++++++++++++ drivers/md/bcache/super.c | 96 ++++++++++++++++++++++++++++++++++++- 3 files changed, 174 insertions(+), 1 deletion(-) diff --git a/drivers/md/bcache/bcache.h b/drivers/md/bcache/bcache.h index 74a9849ea164..0d298b48707f 100644 --- a/drivers/md/bcache/bcache.h +++ b/drivers/md/bcache/bcache.h @@ -221,6 +221,7 @@ BITMASK(GC_MOVE, struct bucket, gc_mark, 15, 1); struct search; struct btree; struct keybuf; +struct bch_report_zones_args; struct keybuf_key { struct rb_node node; @@ -277,6 +278,8 @@ struct bcache_device { struct bio *bio, unsigned int sectors); int (*ioctl)(struct bcache_device *d, fmode_t mode, unsigned int cmd, unsigned long arg); + int (*report_zones)(struct bch_report_zones_args *args, + unsigned int nr_zones); }; struct io { @@ -748,6 +751,13 @@ struct bbio { struct bio bio; }; +struct bch_report_zones_args { + struct bcache_device *bcache_device; + sector_t sector; + void *orig_data; + report_zones_cb orig_cb; +}; + #define BTREE_PRIO USHRT_MAX #define INITIAL_PRIO 32768U diff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c index 71a90fbec314..34f63da2338d 100644 --- a/drivers/md/bcache/request.c +++ b/drivers/md/bcache/request.c @@ -1233,6 +1233,30 @@ static int cached_dev_ioctl(struct bcache_device *d, fmode_t mode, if (dc->io_disable) return -EIO; + /* + * All zoned device ioctl commands are handled in + * other code paths, + * - BLKREPORTZONE: by report_zones method of bcache_ops. + * - BLKRESETZONE/BLKOPENZONE/BLKCLOSEZONE/BLKFINISHZONE: all handled + * by bio code path. + * - BLKGETZONESZ/BLKGETNRZONES:directly handled inside generic block + * ioctl handler blkdev_common_ioctl(). + */ + switch (cmd) { + case BLKREPORTZONE: + case BLKRESETZONE: + case BLKGETZONESZ: + case BLKGETNRZONES: + case BLKOPENZONE: + case BLKCLOSEZONE: + case BLKFINISHZONE: + pr_warn("Zoned device ioctl cmd should not be here.\n"); + return -EOPNOTSUPP; + default: + /* Other commands */ + break; + } + return __blkdev_driver_ioctl(dc->bdev, mode, cmd, arg); } @@ -1261,6 +1285,50 @@ static int cached_dev_congested(void *data, int bits) return ret; } +/* + * The callback routine to parse a specific zone from all reporting + * zones. args->orig_cb() is the upper layer report zones callback, + * which should be called after the LBA conversion. + * Notice: all zones after zone 0 will be reported, including the + * offlined zones, how to handle the different types of zones are + * fully decided by upper layer who calss for reporting zones of + * the bcache device. + */ +static int cached_dev_report_zones_cb(struct blk_zone *zone, + unsigned int idx, + void *data) +{ + struct bch_report_zones_args *args = data; + struct bcache_device *d = args->bcache_device; + struct cached_dev *dc = container_of(d, struct cached_dev, disk); + unsigned long data_offset = dc->sb.data_offset; + + /* Zone 0 should not be reported */ + BUG_ON(zone->start < data_offset); + + /* convert back to LBA of the bcache device*/ + zone->start -= data_offset; + zone->wp -= data_offset; + + return args->orig_cb(zone, idx, args->orig_data); +} + +static int cached_dev_report_zones(struct bch_report_zones_args *args, + unsigned int nr_zones) +{ + struct bcache_device *d = args->bcache_device; + struct cached_dev *dc = container_of(d, struct cached_dev, disk); + /* skip zone 0 which is fully occupied by bcache super block */ + sector_t sector = args->sector + dc->sb.data_offset; + + /* sector is real LBA of backing device */ + return blkdev_report_zones(dc->bdev, + sector, + nr_zones, + cached_dev_report_zones_cb, + args); +} + void bch_cached_dev_request_init(struct cached_dev *dc) { struct gendisk *g = dc->disk.disk; @@ -1268,6 +1336,7 @@ void bch_cached_dev_request_init(struct cached_dev *dc) g->queue->backing_dev_info->congested_fn = cached_dev_congested; dc->disk.cache_miss = cached_dev_cache_miss; dc->disk.ioctl = cached_dev_ioctl; + dc->disk.report_zones = cached_dev_report_zones; } /* Flash backed devices */ diff --git a/drivers/md/bcache/super.c b/drivers/md/bcache/super.c index d98354fa28e3..d5da7ad5157d 100644 --- a/drivers/md/bcache/super.c +++ b/drivers/md/bcache/super.c @@ -679,10 +679,36 @@ static int ioctl_dev(struct block_device *b, fmode_t mode, return d->ioctl(d, mode, cmd, arg); } +static int report_zones_dev(struct gendisk *disk, + sector_t sector, + unsigned int nr_zones, + report_zones_cb cb, + void *data) +{ + struct bcache_device *d = disk->private_data; + struct bch_report_zones_args args = { + .bcache_device = d, + .sector = sector, + .orig_data = data, + .orig_cb = cb, + }; + + /* + * only bcache device with backing device has + * report_zones method, flash device does not. + */ + if (d->report_zones) + return d->report_zones(&args, nr_zones); + + /* flash dev does not have report_zones method */ + return -EOPNOTSUPP; +} + static const struct block_device_operations bcache_ops = { .open = open_dev, .release = release_dev, .ioctl = ioctl_dev, + .report_zones = report_zones_dev, .owner = THIS_MODULE, }; @@ -817,6 +843,7 @@ static void bcache_device_free(struct bcache_device *d) static int bcache_device_init(struct bcache_device *d, unsigned int block_size, sector_t sectors, make_request_fn make_request_fn) + { struct request_queue *q; const size_t max_stripes = min_t(size_t, INT_MAX, @@ -1307,6 +1334,67 @@ static void cached_dev_flush(struct closure *cl) continue_at(cl, cached_dev_free, system_wq); } +static inline int cached_dev_data_offset_check(struct cached_dev *dc) +{ + if (!bdev_is_zoned(dc->bdev)) + return 0; + + /* + * If the backing hard drive is zoned device, sb.data_offset + * should be aligned to zone size, which is automatically + * handled by 'bcache' util of bcache-tools. If the data_offset + * is not aligned to zone size, it means the bcache-tools is + * outdated. + */ + if (dc->sb.data_offset & (bdev_zone_sectors(dc->bdev) - 1)) { + pr_err("data_offset %llu is not aligned to zone size %llu, please update bcache-tools and re-make the zoned backing device.\n", + dc->sb.data_offset, bdev_zone_sectors(dc->bdev)); + return -EINVAL; + } + + return 0; +} + +/* + * Initialize zone information for the bcache device, this function + * assumes the bcache device has a cached device (dc != NULL), and + * the cached device is zoned device (bdev_is_zoned(dc->bdev) == true). + * + * The following zone information of the bcache device will be set, + * - zoned mode, same as the mode of zoned backing device + * - zone size in sectors, same as the zoned backing device + * - zones number, it is zones number of zoned backing device minus the + * reserved zones for bcache super blocks. + */ +static int bch_cached_dev_zone_init(struct cached_dev *dc) +{ + struct request_queue *d_q, *b_q; + enum blk_zoned_model mode; + + if (!bdev_is_zoned(dc->bdev)) + return 0; + + /* queue of bcache device */ + d_q = dc->disk.disk->queue; + /* queue of backing device */ + b_q = bdev_get_queue(dc->bdev); + + mode = blk_queue_zoned_model(b_q); + if (mode != BLK_ZONED_NONE) { + d_q->limits.zoned = mode; + blk_queue_chunk_sectors(d_q, b_q->limits.chunk_sectors); + /* + * (dc->sb.data_offset / q->limits.chunk_sectors) is the + * zones number reserved for bcache super block. By default + * it is set to 1 by bcache-tools. + */ + d_q->nr_zones = b_q->nr_zones - + (dc->sb.data_offset / d_q->limits.chunk_sectors); + } + + return 0; +} + static int cached_dev_init(struct cached_dev *dc, unsigned int block_size) { int ret; @@ -1333,6 +1421,10 @@ static int cached_dev_init(struct cached_dev *dc, unsigned int block_size) dc->disk.stripe_size = q->limits.io_opt >> 9; + ret = cached_dev_data_offset_check(dc); + if (ret) + return ret; + if (dc->disk.stripe_size) dc->partial_stripes_expensive = q->limits.raid_partial_stripes_expensive; @@ -1355,7 +1447,9 @@ static int cached_dev_init(struct cached_dev *dc, unsigned int block_size) bch_cached_dev_request_init(dc); bch_cached_dev_writeback_init(dc); - return 0; + ret = bch_cached_dev_zone_init(dc); + + return ret; } /* Cached device - bcache superblock */ From patchwork Fri May 22 12:18:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Coly Li X-Patchwork-Id: 11565495 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 15D1B90 for ; Fri, 22 May 2020 12:18:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0A65320756 for ; Fri, 22 May 2020 12:18:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729686AbgEVMS4 (ORCPT ); Fri, 22 May 2020 08:18:56 -0400 Received: from mx2.suse.de ([195.135.220.15]:52468 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728772AbgEVMS4 (ORCPT ); Fri, 22 May 2020 08:18:56 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 4E77EAEF8; Fri, 22 May 2020 12:18:55 +0000 (UTC) From: Coly Li To: linux-bcache@vger.kernel.org Cc: linux-block@vger.kernel.org, Coly Li , Damien Le Moal , Hannes Reinecke , Johannes Thumshirn Subject: [RFC PATCH v4 2/3] bcache: handle zone management bios for bcache device Date: Fri, 22 May 2020 20:18:36 +0800 Message-Id: <20200522121837.109651-3-colyli@suse.de> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200522121837.109651-1-colyli@suse.de> References: <20200522121837.109651-1-colyli@suse.de> MIME-Version: 1.0 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Fortunately the zoned device management interface is designed to use bio operations, the bcache code just needs to do a little change to handle the zone management bios. Bcache driver needs to handle 5 zone management bios for now, all of them are handled by cached_dev_nodata() since their bi_size values are 0. - REQ_OP_ZONE_OPEN, REQ_OP_ZONE_CLOSE, REQ_OP_ZONE_FINISH: The LBA conversion is already done in cached_dev_make_request(), just simply send such zone management bios to backing device is enough. - REQ_OP_ZONE_RESET, REQ_OP_ZONE_RESET_ALL: If thre is cached data (clean or dirty) on cache device covered by the LBA range of resetting zone(s), such cached data must be invalidate from the cache device before forwarding the zone reset bios to the backing device. Otherwise data inconsistency or further corruption may happen from the view of bcache device. The difference of REQ_OP_ZONE_RESET_ALL and REQ_OP_ZONE_RESET is when the zone management bio is to reset all zones, send all zones number reported by the bcache device (s->d->disk->queue->nr_zones) into bch_data_invalidate_zones() as parameter 'size_t nr_zones'. If only reset a single zone, just set 1 as 'size_t nr_zones'. By supporting zone managememnt bios with this patch, now a bcache device can be formatted by zonefs, and the zones can be reset by truncate -s 0 on the zone files under seq/ directory. Supporting REQ_OP_ZONE_RESET_ALL makes the whole disk zones reset much faster. In my testing on a 14TB zoned SMR hard drive, 1 by 1 resetting 51632 seq zones by sending REQ_OP_ZONE_RESET bios takes 4m34s, by sending a single REQ_OP_ZONE_RESET_ALL bio takes 12s, which is 22x times faster. REQ_OP_ZONE_RESET_ALL is supported by bcache only when the backing device supports it. So the bcache queue flag is set QUEUE_FLAG_ZONE_RESETALL on only when queue of backing device has such flag, which can be checked by calling blk_queue_zone_resetall() on backing device's request queue. Signed-off-by: Coly Li Cc: Damien Le Moal Cc: Hannes Reinecke Cc: Johannes Thumshirn --- Changelog: v2: an improved version without any generic block layer change. v1: initial version depends on other generic block layer changes. drivers/md/bcache/request.c | 99 ++++++++++++++++++++++++++++++++++++- drivers/md/bcache/super.c | 2 + 2 files changed, 100 insertions(+), 1 deletion(-) diff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c index 34f63da2338d..700b8ab5dec9 100644 --- a/drivers/md/bcache/request.c +++ b/drivers/md/bcache/request.c @@ -1052,18 +1052,115 @@ static void cached_dev_write(struct cached_dev *dc, struct search *s) continue_at(cl, cached_dev_write_complete, NULL); } +/* + * Invalidate the LBA range on cache device which is covered by the + * the resetting zones. + */ +static int bch_data_invalidate_zones(struct closure *cl, + size_t zone_sector, + size_t nr_zones) +{ + struct search *s = container_of(cl, struct search, cl); + struct data_insert_op *iop = &s->iop; + int max_keys, free_keys; + size_t start = zone_sector; + int ret; + + max_keys = (block_bytes(iop->c) - sizeof(struct jset)) / + sizeof(uint64_t); + bch_keylist_init(&iop->insert_keys); + ret = bch_keylist_realloc(&iop->insert_keys, max_keys, iop->c); + if (ret < 0) + return -ENOMEM; + + while (nr_zones-- > 0) { + atomic_t *journal_ref = NULL; + size_t size = s->d->disk->queue->limits.chunk_sectors; +more_keys: + bch_keylist_reset(&iop->insert_keys); + free_keys = max_keys; + + while (size > 0 && free_keys >= 2) { + size_t sectors = min_t(size_t, size, + 1U << (KEY_SIZE_BITS - 1)); + + bch_keylist_add(&iop->insert_keys, + &KEY(iop->inode, start, sectors)); + start += sectors; + size -= sectors; + free_keys -= 2; + } + + /* Insert invalidate keys into btree */ + journal_ref = bch_journal(iop->c, &iop->insert_keys, NULL); + if (!journal_ref) { + ret = -EIO; + break; + } + + ret = bch_btree_insert(iop->c, + &iop->insert_keys, journal_ref, NULL); + atomic_dec_bug(journal_ref); + if (ret < 0) + break; + + if (size) + goto more_keys; + } + + bch_keylist_free(&iop->insert_keys); + + return ret; +} + static void cached_dev_nodata(struct closure *cl) { struct search *s = container_of(cl, struct search, cl); struct bio *bio = &s->bio.bio; + int nr_zones = 1; if (s->iop.flush_journal) bch_journal_meta(s->iop.c, cl); - /* If it's a flush, we send the flush to the backing device too */ + if (bio_op(bio) == REQ_OP_ZONE_RESET_ALL || + bio_op(bio) == REQ_OP_ZONE_RESET) { + int err = 0; + /* + * If this is REQ_OP_ZONE_RESET_ALL bio, cached data + * covered by all zones should be invalidate from the + * cache device. + */ + if (bio_op(bio) == REQ_OP_ZONE_RESET_ALL) + nr_zones = s->d->disk->queue->nr_zones; + + err = bch_data_invalidate_zones( + cl, bio->bi_iter.bi_sector, nr_zones); + + if (err < 0) { + s->iop.status = errno_to_blk_status(err); + /* debug, should be removed before post patch */ + bio->bi_status = BLK_STS_TIMEOUT; + /* set by bio_cnt_set() in do_bio_hook() */ + bio_put(bio); + /* + * Invalidate cached data fails, don't send + * the zone reset bio to backing device and + * return failure. Otherwise potential data + * corruption on bcache device may happen. + */ + goto continue_bio_complete; + } + + } + + /* + * For flush or zone management bios, of cause + * they should be sent to backing device too. + */ bio->bi_end_io = backing_request_endio; closure_bio_submit(s->iop.c, bio, cl); +continue_bio_complete: continue_at(cl, cached_dev_bio_complete, NULL); } diff --git a/drivers/md/bcache/super.c b/drivers/md/bcache/super.c index d5da7ad5157d..70c31950ec1b 100644 --- a/drivers/md/bcache/super.c +++ b/drivers/md/bcache/super.c @@ -1390,6 +1390,8 @@ static int bch_cached_dev_zone_init(struct cached_dev *dc) */ d_q->nr_zones = b_q->nr_zones - (dc->sb.data_offset / d_q->limits.chunk_sectors); + if (blk_queue_zone_resetall(b_q)) + blk_queue_flag_set(QUEUE_FLAG_ZONE_RESETALL, d_q); } return 0; From patchwork Fri May 22 12:18:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Coly Li X-Patchwork-Id: 11565497 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 92FCC138A for ; Fri, 22 May 2020 12:18:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 86F27207CB for ; Fri, 22 May 2020 12:18:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729377AbgEVMS5 (ORCPT ); Fri, 22 May 2020 08:18:57 -0400 Received: from mx2.suse.de ([195.135.220.15]:52518 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728772AbgEVMS5 (ORCPT ); Fri, 22 May 2020 08:18:57 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 637A5AF0F; Fri, 22 May 2020 12:18:58 +0000 (UTC) From: Coly Li To: linux-bcache@vger.kernel.org Cc: linux-block@vger.kernel.org, Coly Li , Damien Le Moal , Hannes Reinecke , Johannes Thumshirn Subject: [RFC PATCH v4 3/3] bcache: reject writeback cache mode for zoned backing device Date: Fri, 22 May 2020 20:18:37 +0800 Message-Id: <20200522121837.109651-4-colyli@suse.de> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200522121837.109651-1-colyli@suse.de> References: <20200522121837.109651-1-colyli@suse.de> MIME-Version: 1.0 Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Currently we don't support writeback mode for zoned device as backing device. So reject it by sysfs interface. This rejection will be removed after the writeback cache mode support for zoned device gets done. Signed-off-by: Coly Li Cc: Damien Le Moal Cc: Hannes Reinecke Cc: Johannes Thumshirn Reviewed-by: Damien Le Moal --- drivers/md/bcache/sysfs.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/drivers/md/bcache/sysfs.c b/drivers/md/bcache/sysfs.c index 323276994aab..41bdbc42a17d 100644 --- a/drivers/md/bcache/sysfs.c +++ b/drivers/md/bcache/sysfs.c @@ -359,6 +359,11 @@ STORE(__cached_dev) if (v < 0) return v; + if ((unsigned int) v == CACHE_MODE_WRITEBACK) { + pr_err("writeback mode is not supported for zoned backing device.\n"); + return -ENOTSUPP; + } + if ((unsigned int) v != BDEV_CACHE_MODE(&dc->sb)) { SET_BDEV_CACHE_MODE(&dc->sb, v); bch_write_bdev_super(dc, NULL);