From patchwork Wed Dec 13 10:42:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongyu Jin X-Patchwork-Id: 13490648 X-Patchwork-Delegate: snitzer@redhat.com Received: from mail-io1-f49.google.com (mail-io1-f49.google.com [209.85.166.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D140922EF1 for ; Wed, 13 Dec 2023 10:42:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VlVo5BE8" Received: by mail-io1-f49.google.com with SMTP id ca18e2360f4ac-7b70de199f6so195957539f.2 for ; Wed, 13 Dec 2023 02:42:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702464163; x=1703068963; darn=lists.linux.dev; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kk71sp729csxdUViLno20l6lcdfkMyL/Wdv2yhvpzTA=; b=VlVo5BE8PdxBh1RcQzy57soOOm6+yn/Hd1XYeMOS4GBQKPbKafvpX5tpZW3K1clun4 KRtRADjH15sxX4h8Ps5P6sLQveWAXzZS1ZY7Z4+CVYI3JA+pmkxjZxWNxfPUySIw4+qt uZSFfABE1MKileHYXP2XHFOpq8LgBkWhFSHOheu2tnBg51Q66qtnYAaz4Im5ZXvIcrDF r19U3/LiJoESitunGauL0TSMBiCYY9j32Tqf9MhHM1gpJ8IoSU+6slaxRMTN24bfMOL8 3DhN45ZP5CcRGCUVrXUuSv7cCLEv/5ec1cub+0fp1DqjCQ5MhmF4rcsM+cKjgBk29WYI BXOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702464163; x=1703068963; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kk71sp729csxdUViLno20l6lcdfkMyL/Wdv2yhvpzTA=; b=eP2zAOCQP54couP6LHUMlA6mwB3xEFAQTfqcEO38v96SSm7QShXJ2mLJTpIiZN2DMY 06taa061FtTVzTGh5ibgFLEcJpldBjaqwNF5CWbxTQ2pM1gcWKl+3mye1utxWA1RoiQ/ ge64/TkTQsTNtyRZSGYQeL6ySM0wt5xv5IHktqlX/OUcIcj9tq9r1mNXOI0B3i/zRkDH unRgu4cYImMipQMXJU3yvRxAZ0yhc4AMZ2Kq2X0/cMM4eGg84PbJIyWIVqCh0VUM8i4t DFjRL1eo4eGD7mHanC68iI6K2kYa7/IL/jQWuDdPRbAn4kPZ9aeA4r/IAteN0K1yX82u vdEg== X-Gm-Message-State: AOJu0YxIPgkXuUcyD6SIXKjFYGzQejkgCxAUZQ8ytVCumhZrqaduSQrl ebLoL7QO4arfYxOoq4EuBno= X-Google-Smtp-Source: AGHT+IF8KhM4LKu59z4wp0wI/kfaBUgeyltzCePLtvl91gtiRSE3aEV5k+2UHPjWxXWBRbof2CjeiA== X-Received: by 2002:a5e:d70b:0:b0:7b7:4cd:6b16 with SMTP id v11-20020a5ed70b000000b007b704cd6b16mr7899907iom.31.1702464162859; Wed, 13 Dec 2023 02:42:42 -0800 (PST) Received: from ubuntu.. ([117.18.48.102]) by smtp.gmail.com with ESMTPSA id c11-20020a6b4e0b000000b0078647b08ab0sm3058186iob.6.2023.12.13.02.42.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Dec 2023 02:42:42 -0800 (PST) From: Hongyu Jin To: agk@redhat.com, snitzer@kernel.org, mpatocka@redhat.com, axboe@kernel.dk, ebiggers@kernel.org Cc: zhiguo.niu@unisoc.com, ke.wang@unisoc.com, yibin.ding@unisoc.com, hongyu.jin@unisoc.com, linux-kernel@vger.kernel.org, dm-devel@lists.linux.dev, linux-block@vger.kernel.org Subject: [PATCH v5 1/5] block: Fix bio IO priority setting Date: Wed, 13 Dec 2023 18:42:12 +0800 Message-Id: <20231213104216.27845-2-hongyu.jin.cn@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231213104216.27845-1-hongyu.jin.cn@gmail.com> References: <20231213104216.27845-1-hongyu.jin.cn@gmail.com> Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Hongyu Jin Move bio_set_ioprio() into submit_bio(): 1. Only call bio_set_ioprio() once to set the priority of original bio, the bio that cloned and splited from original bio will auto inherit the priority of original bio in clone process. 2. The IO priority can be passed to module that implement struct gendisk::fops::submit_bio, help resolve some of the IO priority loss issues. This patch depends on commit 82b74cac2849 ("blk-ioprio: Convert from rqos policy to direct call") Fixes: a78418e6a04c ("block: Always initialize bio IO priority on submit") Co-developed-by: Yibin Ding Signed-off-by: Yibin Ding Signed-off-by: Hongyu Jin --- block/blk-core.c | 10 ++++++++++ block/blk-mq.c | 11 ----------- 2 files changed, 10 insertions(+), 11 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index fdf25b8d6e78..68158c327aea 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -49,6 +49,7 @@ #include "blk-pm.h" #include "blk-cgroup.h" #include "blk-throttle.h" +#include "blk-ioprio.h" struct dentry *blk_debugfs_root; @@ -809,6 +810,14 @@ void submit_bio_noacct(struct bio *bio) } EXPORT_SYMBOL(submit_bio_noacct); +static void bio_set_ioprio(struct bio *bio) +{ + /* Nobody set ioprio so far? Initialize it based on task's nice value */ + if (IOPRIO_PRIO_CLASS(bio->bi_ioprio) == IOPRIO_CLASS_NONE) + bio->bi_ioprio = get_current_ioprio(); + blkcg_set_ioprio(bio); +} + /** * submit_bio - submit a bio to the block device layer for I/O * @bio: The &struct bio which describes the I/O @@ -831,6 +840,7 @@ void submit_bio(struct bio *bio) count_vm_events(PGPGOUT, bio_sectors(bio)); } + bio_set_ioprio(bio); submit_bio_noacct(bio); } EXPORT_SYMBOL(submit_bio); diff --git a/block/blk-mq.c b/block/blk-mq.c index e2d11183f62e..a6e2609df9c9 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -40,7 +40,6 @@ #include "blk-stat.h" #include "blk-mq-sched.h" #include "blk-rq-qos.h" -#include "blk-ioprio.h" static DEFINE_PER_CPU(struct llist_head, blk_cpu_done); static DEFINE_PER_CPU(call_single_data_t, blk_cpu_csd); @@ -2922,14 +2921,6 @@ static inline struct request *blk_mq_get_cached_request(struct request_queue *q, return rq; } -static void bio_set_ioprio(struct bio *bio) -{ - /* Nobody set ioprio so far? Initialize it based on task's nice value */ - if (IOPRIO_PRIO_CLASS(bio->bi_ioprio) == IOPRIO_CLASS_NONE) - bio->bi_ioprio = get_current_ioprio(); - blkcg_set_ioprio(bio); -} - /** * blk_mq_submit_bio - Create and send a request to block device. * @bio: Bio pointer. @@ -2963,8 +2954,6 @@ void blk_mq_submit_bio(struct bio *bio) if (!bio_integrity_prep(bio)) return; - bio_set_ioprio(bio); - rq = blk_mq_get_cached_request(q, plug, &bio, nr_segs); if (!rq) { if (!bio) From patchwork Wed Dec 13 10:42:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongyu Jin X-Patchwork-Id: 13490649 X-Patchwork-Delegate: snitzer@redhat.com Received: from mail-io1-f42.google.com (mail-io1-f42.google.com [209.85.166.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E6E0622EE5 for ; Wed, 13 Dec 2023 10:42:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Omz9EIp1" Received: by mail-io1-f42.google.com with SMTP id ca18e2360f4ac-7b6fc484b77so156196039f.0 for ; Wed, 13 Dec 2023 02:42:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702464167; x=1703068967; darn=lists.linux.dev; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RjojUSEOZbZ35+vIKwudAm9m3vU4GuyELsvW3NVGAV4=; b=Omz9EIp1ig56NI2WB4NbvIQieJEcTaC5RdHI36x7NJrV4UHx27HuknhZDb05Qn6Vvl r1/Qg4z+9ppciGAiMJBG8eCP2lyoVqVn6NAZntToXaDIQoQMoeI0EV2RayKmhMtIL2IV dHV4LGNtGYrBQ2xD8kzRNvw/iGi8MfoIxkOKGwdBORF9vgi5UC7iLdOi2cnrIDgwAsqY 321ItY5Q3a2c8reHVdo/ySmM9oHM8IWodf4xxiu4NoXTrfxUbJDiW8DW7PwzaALSgAMa JlJ9euSVkU15aPJtPoMsyaVlH4Kle1tyFezSmfSMXRYFSzSuxbETrzbN0UBm5NzPdC73 fYJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702464167; x=1703068967; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RjojUSEOZbZ35+vIKwudAm9m3vU4GuyELsvW3NVGAV4=; b=AudwdAuWSeNEJYJ/pkrqEG8ZnHUDnfqQylHYCdEuVPcN5ti+XelB8oa3eocGuxH5fO H5QQx25/xY/yTfOerahmSSgwYr3yg7VzjNPpb8fX7kWuJ1ZAhM+IoI/JceShpABZPRzE wAD/9zSOlF3yHqXI15caat5dqgry6X0KY1pG8aDIuIBF6KfK1coS47bzD/Vmv3UR8qt9 G/CclTKXs3JB2RUjwBtfqw2VI7JJ2SunrU0/6Eor6SQSkThMrD8uOSA6qVPRcYG3UTBQ P+xHgEBN5gLmlaGnICPSj3RsPEGMY47wKOVJjRBhRfP0jRut9q//Vik0zYVjm9rVVSLB nW/w== X-Gm-Message-State: AOJu0Yzxfm085cK2/Hh66l1gGudUFv8Xi7wqUjWaWfhzKZv3GStddqXy 1f+W6SmSsMRYhGPi3NeJwyE= X-Google-Smtp-Source: AGHT+IEJm2CVbLv3Mruv6RaIegpjcCltNGOoiuUWrt0Eqi4WYZ2y+H2F2QG/sf8SdHUO/4g9fSLyVw== X-Received: by 2002:a6b:5b17:0:b0:7b4:376f:3991 with SMTP id v23-20020a6b5b17000000b007b4376f3991mr5155964ioh.6.1702464166897; Wed, 13 Dec 2023 02:42:46 -0800 (PST) Received: from ubuntu.. ([117.18.48.102]) by smtp.gmail.com with ESMTPSA id c11-20020a6b4e0b000000b0078647b08ab0sm3058186iob.6.2023.12.13.02.42.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Dec 2023 02:42:46 -0800 (PST) From: Hongyu Jin To: agk@redhat.com, snitzer@kernel.org, mpatocka@redhat.com, axboe@kernel.dk, ebiggers@kernel.org Cc: zhiguo.niu@unisoc.com, ke.wang@unisoc.com, yibin.ding@unisoc.com, hongyu.jin@unisoc.com, linux-kernel@vger.kernel.org, dm-devel@lists.linux.dev, linux-block@vger.kernel.org Subject: [PATCH v5 2/5] dm: Support I/O priority for dm_io() Date: Wed, 13 Dec 2023 18:42:13 +0800 Message-Id: <20231213104216.27845-3-hongyu.jin.cn@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231213104216.27845-1-hongyu.jin.cn@gmail.com> References: <20231213104216.27845-1-hongyu.jin.cn@gmail.com> Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Hongyu Jin Add I/O priority parameter for dm_io(). Co-developed-by: Yibin Ding Signed-off-by: Yibin Ding Signed-off-by: Hongyu Jin --- drivers/md/dm-bufio.c | 6 +++--- drivers/md/dm-integrity.c | 10 +++++----- drivers/md/dm-io.c | 23 +++++++++++++---------- drivers/md/dm-kcopyd.c | 4 ++-- drivers/md/dm-log.c | 4 ++-- drivers/md/dm-raid1.c | 6 +++--- drivers/md/dm-snap-persistent.c | 4 ++-- drivers/md/dm-writecache.c | 8 ++++---- include/linux/dm-io.h | 3 ++- 9 files changed, 36 insertions(+), 32 deletions(-) diff --git a/drivers/md/dm-bufio.c b/drivers/md/dm-bufio.c index 62eb27639c9b..91b05bf4b920 100644 --- a/drivers/md/dm-bufio.c +++ b/drivers/md/dm-bufio.c @@ -1279,7 +1279,7 @@ static void use_dmio(struct dm_buffer *b, enum req_op op, sector_t sector, io_req.mem.ptr.vma = (char *)b->data + offset; } - r = dm_io(&io_req, 1, ®ion, NULL); + r = dm_io(&io_req, 1, ®ion, NULL, IOPRIO_DEFAULT); if (unlikely(r)) b->end_io(b, errno_to_blk_status(r)); } @@ -2130,7 +2130,7 @@ int dm_bufio_issue_flush(struct dm_bufio_client *c) if (WARN_ON_ONCE(dm_bufio_in_request())) return -EINVAL; - return dm_io(&io_req, 1, &io_reg, NULL); + return dm_io(&io_req, 1, &io_reg, NULL, IOPRIO_DEFAULT); } EXPORT_SYMBOL_GPL(dm_bufio_issue_flush); @@ -2154,7 +2154,7 @@ int dm_bufio_issue_discard(struct dm_bufio_client *c, sector_t block, sector_t c if (WARN_ON_ONCE(dm_bufio_in_request())) return -EINVAL; /* discards are optional */ - return dm_io(&io_req, 1, &io_reg, NULL); + return dm_io(&io_req, 1, &io_reg, NULL, IOPRIO_DEFAULT); } EXPORT_SYMBOL_GPL(dm_bufio_issue_discard); diff --git a/drivers/md/dm-integrity.c b/drivers/md/dm-integrity.c index e85c688fd91e..9ffd093ad6cc 100644 --- a/drivers/md/dm-integrity.c +++ b/drivers/md/dm-integrity.c @@ -553,7 +553,7 @@ static int sync_rw_sb(struct dm_integrity_c *ic, blk_opf_t opf) } } - r = dm_io(&io_req, 1, &io_loc, NULL); + r = dm_io(&io_req, 1, &io_loc, NULL, IOPRIO_DEFAULT); if (unlikely(r)) return r; @@ -1071,7 +1071,7 @@ static void rw_journal_sectors(struct dm_integrity_c *ic, blk_opf_t opf, io_loc.sector = ic->start + SB_SECTORS + sector; io_loc.count = n_sectors; - r = dm_io(&io_req, 1, &io_loc, NULL); + r = dm_io(&io_req, 1, &io_loc, NULL, IOPRIO_DEFAULT); if (unlikely(r)) { dm_integrity_io_error(ic, (opf & REQ_OP_MASK) == REQ_OP_READ ? "reading journal" : "writing journal", r); @@ -1188,7 +1188,7 @@ static void copy_from_journal(struct dm_integrity_c *ic, unsigned int section, u io_loc.sector = target; io_loc.count = n_sectors; - r = dm_io(&io_req, 1, &io_loc, NULL); + r = dm_io(&io_req, 1, &io_loc, NULL, IOPRIO_DEFAULT); if (unlikely(r)) { WARN_ONCE(1, "asynchronous dm_io failed: %d", r); fn(-1UL, data); @@ -1517,7 +1517,7 @@ static void dm_integrity_flush_buffers(struct dm_integrity_c *ic, bool flush_dat fr.io_reg.count = 0, fr.ic = ic; init_completion(&fr.comp); - r = dm_io(&fr.io_req, 1, &fr.io_reg, NULL); + r = dm_io(&fr.io_req, 1, &fr.io_reg, NULL, IOPRIO_DEFAULT); BUG_ON(r); } @@ -2739,7 +2739,7 @@ static void integrity_recalc(struct work_struct *w) io_loc.sector = get_data_sector(ic, area, offset); io_loc.count = n_sectors; - r = dm_io(&io_req, 1, &io_loc, NULL); + r = dm_io(&io_req, 1, &io_loc, NULL, IOPRIO_DEFAULT); if (unlikely(r)) { dm_integrity_io_error(ic, "reading data", r); goto err; diff --git a/drivers/md/dm-io.c b/drivers/md/dm-io.c index f053ce245814..7409490259d1 100644 --- a/drivers/md/dm-io.c +++ b/drivers/md/dm-io.c @@ -305,7 +305,7 @@ static void km_dp_init(struct dpages *dp, void *data) */ static void do_region(const blk_opf_t opf, unsigned int region, struct dm_io_region *where, struct dpages *dp, - struct io *io) + struct io *io, unsigned short ioprio) { struct bio *bio; struct page *page; @@ -354,6 +354,7 @@ static void do_region(const blk_opf_t opf, unsigned int region, &io->client->bios); bio->bi_iter.bi_sector = where->sector + (where->count - remaining); bio->bi_end_io = endio; + bio->bi_ioprio = ioprio; store_io_and_region_in_bio(bio, io, region); if (op == REQ_OP_DISCARD || op == REQ_OP_WRITE_ZEROES) { @@ -383,7 +384,7 @@ static void do_region(const blk_opf_t opf, unsigned int region, static void dispatch_io(blk_opf_t opf, unsigned int num_regions, struct dm_io_region *where, struct dpages *dp, - struct io *io, int sync) + struct io *io, int sync, unsigned short ioprio) { int i; struct dpages old_pages = *dp; @@ -400,7 +401,7 @@ static void dispatch_io(blk_opf_t opf, unsigned int num_regions, for (i = 0; i < num_regions; i++) { *dp = old_pages; if (where[i].count || (opf & REQ_PREFLUSH)) - do_region(opf, i, where + i, dp, io); + do_region(opf, i, where + i, dp, io, ioprio); } /* @@ -425,7 +426,7 @@ static void sync_io_complete(unsigned long error, void *context) static int sync_io(struct dm_io_client *client, unsigned int num_regions, struct dm_io_region *where, blk_opf_t opf, struct dpages *dp, - unsigned long *error_bits) + unsigned long *error_bits, unsigned short ioprio) { struct io *io; struct sync_io sio; @@ -447,7 +448,7 @@ static int sync_io(struct dm_io_client *client, unsigned int num_regions, io->vma_invalidate_address = dp->vma_invalidate_address; io->vma_invalidate_size = dp->vma_invalidate_size; - dispatch_io(opf, num_regions, where, dp, io, 1); + dispatch_io(opf, num_regions, where, dp, io, 1, ioprio); wait_for_completion_io(&sio.wait); @@ -459,7 +460,8 @@ static int sync_io(struct dm_io_client *client, unsigned int num_regions, static int async_io(struct dm_io_client *client, unsigned int num_regions, struct dm_io_region *where, blk_opf_t opf, - struct dpages *dp, io_notify_fn fn, void *context) + struct dpages *dp, io_notify_fn fn, void *context, + unsigned short ioprio) { struct io *io; @@ -479,7 +481,7 @@ static int async_io(struct dm_io_client *client, unsigned int num_regions, io->vma_invalidate_address = dp->vma_invalidate_address; io->vma_invalidate_size = dp->vma_invalidate_size; - dispatch_io(opf, num_regions, where, dp, io, 0); + dispatch_io(opf, num_regions, where, dp, io, 0, ioprio); return 0; } @@ -521,7 +523,8 @@ static int dp_init(struct dm_io_request *io_req, struct dpages *dp, } int dm_io(struct dm_io_request *io_req, unsigned int num_regions, - struct dm_io_region *where, unsigned long *sync_error_bits) + struct dm_io_region *where, unsigned long *sync_error_bits, + unsigned short ioprio) { int r; struct dpages dp; @@ -532,11 +535,11 @@ int dm_io(struct dm_io_request *io_req, unsigned int num_regions, if (!io_req->notify.fn) return sync_io(io_req->client, num_regions, where, - io_req->bi_opf, &dp, sync_error_bits); + io_req->bi_opf, &dp, sync_error_bits, ioprio); return async_io(io_req->client, num_regions, where, io_req->bi_opf, &dp, io_req->notify.fn, - io_req->notify.context); + io_req->notify.context, ioprio); } EXPORT_SYMBOL(dm_io); diff --git a/drivers/md/dm-kcopyd.c b/drivers/md/dm-kcopyd.c index d01807c50f20..79c65c9ad5fa 100644 --- a/drivers/md/dm-kcopyd.c +++ b/drivers/md/dm-kcopyd.c @@ -578,9 +578,9 @@ static int run_io_job(struct kcopyd_job *job) io_job_start(job->kc->throttle); if (job->op == REQ_OP_READ) - r = dm_io(&io_req, 1, &job->source, NULL); + r = dm_io(&io_req, 1, &job->source, NULL, IOPRIO_DEFAULT); else - r = dm_io(&io_req, job->num_dests, job->dests, NULL); + r = dm_io(&io_req, job->num_dests, job->dests, NULL, IOPRIO_DEFAULT); return r; } diff --git a/drivers/md/dm-log.c b/drivers/md/dm-log.c index f9f84236dfcd..f7f9c2100937 100644 --- a/drivers/md/dm-log.c +++ b/drivers/md/dm-log.c @@ -300,7 +300,7 @@ static int rw_header(struct log_c *lc, enum req_op op) { lc->io_req.bi_opf = op; - return dm_io(&lc->io_req, 1, &lc->header_location, NULL); + return dm_io(&lc->io_req, 1, &lc->header_location, NULL, IOPRIO_DEFAULT); } static int flush_header(struct log_c *lc) @@ -313,7 +313,7 @@ static int flush_header(struct log_c *lc) lc->io_req.bi_opf = REQ_OP_WRITE | REQ_PREFLUSH; - return dm_io(&lc->io_req, 1, &null_location, NULL); + return dm_io(&lc->io_req, 1, &null_location, NULL, IOPRIO_DEFAULT); } static int read_header(struct log_c *log) diff --git a/drivers/md/dm-raid1.c b/drivers/md/dm-raid1.c index ddcb2bc4a617..9511dae5b556 100644 --- a/drivers/md/dm-raid1.c +++ b/drivers/md/dm-raid1.c @@ -278,7 +278,7 @@ static int mirror_flush(struct dm_target *ti) } error_bits = -1; - dm_io(&io_req, ms->nr_mirrors, io, &error_bits); + dm_io(&io_req, ms->nr_mirrors, io, &error_bits, IOPRIO_DEFAULT); if (unlikely(error_bits != 0)) { for (i = 0; i < ms->nr_mirrors; i++) if (test_bit(i, &error_bits)) @@ -554,7 +554,7 @@ static void read_async_bio(struct mirror *m, struct bio *bio) map_region(&io, m, bio); bio_set_m(bio, m); - BUG_ON(dm_io(&io_req, 1, &io, NULL)); + BUG_ON(dm_io(&io_req, 1, &io, NULL, IOPRIO_DEFAULT)); } static inline int region_in_sync(struct mirror_set *ms, region_t region, @@ -681,7 +681,7 @@ static void do_write(struct mirror_set *ms, struct bio *bio) */ bio_set_m(bio, get_default_mirror(ms)); - BUG_ON(dm_io(&io_req, ms->nr_mirrors, io, NULL)); + BUG_ON(dm_io(&io_req, ms->nr_mirrors, io, NULL, IOPRIO_DEFAULT)); } static void do_writes(struct mirror_set *ms, struct bio_list *writes) diff --git a/drivers/md/dm-snap-persistent.c b/drivers/md/dm-snap-persistent.c index 15649921f2a9..568d10842b1f 100644 --- a/drivers/md/dm-snap-persistent.c +++ b/drivers/md/dm-snap-persistent.c @@ -223,7 +223,7 @@ static void do_metadata(struct work_struct *work) { struct mdata_req *req = container_of(work, struct mdata_req, work); - req->result = dm_io(req->io_req, 1, req->where, NULL); + req->result = dm_io(req->io_req, 1, req->where, NULL, IOPRIO_DEFAULT); } /* @@ -247,7 +247,7 @@ static int chunk_io(struct pstore *ps, void *area, chunk_t chunk, blk_opf_t opf, struct mdata_req req; if (!metadata) - return dm_io(&io_req, 1, &where, NULL); + return dm_io(&io_req, 1, &where, NULL, IOPRIO_DEFAULT); req.where = &where; req.io_req = &io_req; diff --git a/drivers/md/dm-writecache.c b/drivers/md/dm-writecache.c index 074cb785eafc..6a4279bfb1e7 100644 --- a/drivers/md/dm-writecache.c +++ b/drivers/md/dm-writecache.c @@ -531,7 +531,7 @@ static void ssd_commit_flushed(struct dm_writecache *wc, bool wait_for_ios) req.notify.context = &endio; /* writing via async dm-io (implied by notify.fn above) won't return an error */ - (void) dm_io(&req, 1, ®ion, NULL); + (void) dm_io(&req, 1, ®ion, NULL, IOPRIO_DEFAULT); i = j; } @@ -568,7 +568,7 @@ static void ssd_commit_superblock(struct dm_writecache *wc) req.notify.fn = NULL; req.notify.context = NULL; - r = dm_io(&req, 1, ®ion, NULL); + r = dm_io(&req, 1, ®ion, NULL, IOPRIO_DEFAULT); if (unlikely(r)) writecache_error(wc, r, "error writing superblock"); } @@ -596,7 +596,7 @@ static void writecache_disk_flush(struct dm_writecache *wc, struct dm_dev *dev) req.client = wc->dm_io; req.notify.fn = NULL; - r = dm_io(&req, 1, ®ion, NULL); + r = dm_io(&req, 1, ®ion, NULL, IOPRIO_DEFAULT); if (unlikely(r)) writecache_error(wc, r, "error flushing metadata: %d", r); } @@ -990,7 +990,7 @@ static int writecache_read_metadata(struct dm_writecache *wc, sector_t n_sectors req.client = wc->dm_io; req.notify.fn = NULL; - return dm_io(&req, 1, ®ion, NULL); + return dm_io(&req, 1, ®ion, NULL, IOPRIO_DEFAULT); } static void writecache_resume(struct dm_target *ti) diff --git a/include/linux/dm-io.h b/include/linux/dm-io.h index 7595142f3fc5..7b2968612b7e 100644 --- a/include/linux/dm-io.h +++ b/include/linux/dm-io.h @@ -80,7 +80,8 @@ void dm_io_client_destroy(struct dm_io_client *client); * error occurred doing io to the corresponding region. */ int dm_io(struct dm_io_request *io_req, unsigned int num_regions, - struct dm_io_region *region, unsigned int long *sync_error_bits); + struct dm_io_region *region, unsigned int long *sync_error_bits, + unsigned short ioprio); #endif /* __KERNEL__ */ #endif /* _LINUX_DM_IO_H */ From patchwork Wed Dec 13 10:42:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongyu Jin X-Patchwork-Id: 13490650 X-Patchwork-Delegate: snitzer@redhat.com Received: from mail-io1-f53.google.com (mail-io1-f53.google.com [209.85.166.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 76D6522EE6 for ; Wed, 13 Dec 2023 10:42:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="RlOrQ+zU" Received: by mail-io1-f53.google.com with SMTP id ca18e2360f4ac-7b75452c3f4so101330039f.1 for ; Wed, 13 Dec 2023 02:42:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702464171; x=1703068971; darn=lists.linux.dev; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wCH7qHK5H8vXi2OAOx3SCYKWSwB7ChpMXiuHfRRRRKQ=; b=RlOrQ+zUHVR48QPxG8nMB1NW771ZYCgg7xufzQ2/ofW5/8pEJLIM8yqEUMrJ3Yt6Sw tyvoxc0ADotiGeAmIrKMtHV/I03otbAYqIrScZ5sMhiMiPpegw2nBj4inVphZV5PMX1W Vmp8MosX6x2c586TH9c1Zye+j+Pz4AxB2GVNpmpLWRBJyKG3EedyU95u8PLChL5jr6vK JyTUvi3LMdaBFEmB23anOxGhVy08vx6AQcRjj2UzykUC1qhaNu81w1Bc/ol9/iRKKdS1 91ggP/mCScUoWe0yafYLuLv1hfi+lGEJ54X+Q3hiK/zj5XEOn1mjteWNBYXukrDE9lEi P+Mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702464171; x=1703068971; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wCH7qHK5H8vXi2OAOx3SCYKWSwB7ChpMXiuHfRRRRKQ=; b=SEzyHwJcnIy06IgT8JmFK2h+pVoBE3qJWpCHW9V6AMHR1JCO2Mjess+7lAi/V86+en /kf/j6ab1dRxZ6oNygwnt2NK2tcePK8qhgE3h0hBEtHz4ZztlLavUqr35A9h1oZAfXJ2 dOQSIMlZ/B+Z0EVmcGonU6TPYbDxQb7EQGkFzzA23cC0HGnmxeCCvQl6uaks0JLlluRU uQ7v8idDp/D/rBx74tlEsqB7zsGKb/aqeO/98Kwu9xYnibaMq6xglQR0dTdtrrQ8WFb8 Pho0sv7+jKlzv0aCJEwm9SfRF46X4LPTAXRRIXuPV+EpkJJ65A9tWu+0GgjAtVICZNvg 7Wag== X-Gm-Message-State: AOJu0YwHr6lR18OjXN0NzPtnIpXGWFcEObykQIvlQO9QFyshw6vWFutg 2odapYTtBAyJCnTN4IbzWNY= X-Google-Smtp-Source: AGHT+IGUkDYY/69pPfSdo256SxRGG9oyvhuwKzyARdXNbQ9J9lAf+CZWaD1DPntTJa2IuS7L3v9tEQ== X-Received: by 2002:a5e:d70b:0:b0:7b7:4cd:6b16 with SMTP id v11-20020a5ed70b000000b007b704cd6b16mr7900060iom.31.1702464171468; Wed, 13 Dec 2023 02:42:51 -0800 (PST) Received: from ubuntu.. ([117.18.48.102]) by smtp.gmail.com with ESMTPSA id c11-20020a6b4e0b000000b0078647b08ab0sm3058186iob.6.2023.12.13.02.42.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Dec 2023 02:42:51 -0800 (PST) From: Hongyu Jin To: agk@redhat.com, snitzer@kernel.org, mpatocka@redhat.com, axboe@kernel.dk, ebiggers@kernel.org Cc: zhiguo.niu@unisoc.com, ke.wang@unisoc.com, yibin.ding@unisoc.com, hongyu.jin@unisoc.com, linux-kernel@vger.kernel.org, dm-devel@lists.linux.dev, linux-block@vger.kernel.org Subject: [PATCH v5 3/5] dm-bufio: Support I/O priority Date: Wed, 13 Dec 2023 18:42:14 +0800 Message-Id: <20231213104216.27845-4-hongyu.jin.cn@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231213104216.27845-1-hongyu.jin.cn@gmail.com> References: <20231213104216.27845-1-hongyu.jin.cn@gmail.com> Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Hongyu Jin Add I/O priority parameter for dm_bufio_read() and dm_bufio_prefetch(). Co-developed-by: Yibin Ding Signed-off-by: Yibin Ding Signed-off-by: Hongyu Jin --- drivers/md/dm-bufio.c | 35 ++++++++++--------- drivers/md/dm-ebs-target.c | 8 ++--- drivers/md/dm-integrity.c | 2 +- drivers/md/dm-snap-persistent.c | 4 +-- drivers/md/dm-verity-fec.c | 4 +-- drivers/md/dm-verity-target.c | 4 +-- drivers/md/persistent-data/dm-block-manager.c | 6 ++-- include/linux/dm-bufio.h | 5 ++- 8 files changed, 34 insertions(+), 34 deletions(-) diff --git a/drivers/md/dm-bufio.c b/drivers/md/dm-bufio.c index 91b05bf4b920..f3b051d3517e 100644 --- a/drivers/md/dm-bufio.c +++ b/drivers/md/dm-bufio.c @@ -1256,7 +1256,7 @@ static void dmio_complete(unsigned long error, void *context) } static void use_dmio(struct dm_buffer *b, enum req_op op, sector_t sector, - unsigned int n_sectors, unsigned int offset) + unsigned int n_sectors, unsigned int offset, unsigned short ioprio) { int r; struct dm_io_request io_req = { @@ -1279,7 +1279,7 @@ static void use_dmio(struct dm_buffer *b, enum req_op op, sector_t sector, io_req.mem.ptr.vma = (char *)b->data + offset; } - r = dm_io(&io_req, 1, ®ion, NULL, IOPRIO_DEFAULT); + r = dm_io(&io_req, 1, ®ion, NULL, ioprio); if (unlikely(r)) b->end_io(b, errno_to_blk_status(r)); } @@ -1295,7 +1295,7 @@ static void bio_complete(struct bio *bio) } static void use_bio(struct dm_buffer *b, enum req_op op, sector_t sector, - unsigned int n_sectors, unsigned int offset) + unsigned int n_sectors, unsigned int offset, unsigned short ioprio) { struct bio *bio; char *ptr; @@ -1303,13 +1303,14 @@ static void use_bio(struct dm_buffer *b, enum req_op op, sector_t sector, bio = bio_kmalloc(1, GFP_NOWAIT | __GFP_NORETRY | __GFP_NOWARN); if (!bio) { - use_dmio(b, op, sector, n_sectors, offset); + use_dmio(b, op, sector, n_sectors, offset, ioprio); return; } bio_init(bio, b->c->bdev, bio->bi_inline_vecs, 1, op); bio->bi_iter.bi_sector = sector; bio->bi_end_io = bio_complete; bio->bi_private = b; + bio->bi_ioprio = ioprio; ptr = (char *)b->data + offset; len = n_sectors << SECTOR_SHIFT; @@ -1332,7 +1333,7 @@ static inline sector_t block_to_sector(struct dm_bufio_client *c, sector_t block return sector; } -static void submit_io(struct dm_buffer *b, enum req_op op, +static void submit_io(struct dm_buffer *b, enum req_op op, unsigned short ioprio, void (*end_io)(struct dm_buffer *, blk_status_t)) { unsigned int n_sectors; @@ -1362,9 +1363,9 @@ static void submit_io(struct dm_buffer *b, enum req_op op, } if (b->data_mode != DATA_MODE_VMALLOC) - use_bio(b, op, sector, n_sectors, offset); + use_bio(b, op, sector, n_sectors, offset, ioprio); else - use_dmio(b, op, sector, n_sectors, offset); + use_dmio(b, op, sector, n_sectors, offset, ioprio); } /* @@ -1420,7 +1421,7 @@ static void __write_dirty_buffer(struct dm_buffer *b, b->write_end = b->dirty_end; if (!write_list) - submit_io(b, REQ_OP_WRITE, write_endio); + submit_io(b, REQ_OP_WRITE, IOPRIO_DEFAULT, write_endio); else list_add_tail(&b->write_list, write_list); } @@ -1434,7 +1435,7 @@ static void __flush_write_list(struct list_head *write_list) struct dm_buffer *b = list_entry(write_list->next, struct dm_buffer, write_list); list_del(&b->write_list); - submit_io(b, REQ_OP_WRITE, write_endio); + submit_io(b, REQ_OP_WRITE, IOPRIO_DEFAULT, write_endio); cond_resched(); } blk_finish_plug(&plug); @@ -1816,7 +1817,7 @@ static void read_endio(struct dm_buffer *b, blk_status_t status) * and uses dm_bufio_mark_buffer_dirty to write new data back). */ static void *new_read(struct dm_bufio_client *c, sector_t block, - enum new_flag nf, struct dm_buffer **bp) + enum new_flag nf, struct dm_buffer **bp, unsigned short ioprio) { int need_submit = 0; struct dm_buffer *b; @@ -1869,7 +1870,7 @@ static void *new_read(struct dm_bufio_client *c, sector_t block, return NULL; if (need_submit) - submit_io(b, REQ_OP_READ, read_endio); + submit_io(b, REQ_OP_READ, ioprio, read_endio); wait_on_bit_io(&b->state, B_READING, TASK_UNINTERRUPTIBLE); @@ -1889,17 +1890,17 @@ static void *new_read(struct dm_bufio_client *c, sector_t block, void *dm_bufio_get(struct dm_bufio_client *c, sector_t block, struct dm_buffer **bp) { - return new_read(c, block, NF_GET, bp); + return new_read(c, block, NF_GET, bp, IOPRIO_DEFAULT); } EXPORT_SYMBOL_GPL(dm_bufio_get); void *dm_bufio_read(struct dm_bufio_client *c, sector_t block, - struct dm_buffer **bp) + struct dm_buffer **bp, unsigned short ioprio) { if (WARN_ON_ONCE(dm_bufio_in_request())) return ERR_PTR(-EINVAL); - return new_read(c, block, NF_READ, bp); + return new_read(c, block, NF_READ, bp, ioprio); } EXPORT_SYMBOL_GPL(dm_bufio_read); @@ -1909,12 +1910,12 @@ void *dm_bufio_new(struct dm_bufio_client *c, sector_t block, if (WARN_ON_ONCE(dm_bufio_in_request())) return ERR_PTR(-EINVAL); - return new_read(c, block, NF_FRESH, bp); + return new_read(c, block, NF_FRESH, bp, IOPRIO_DEFAULT); } EXPORT_SYMBOL_GPL(dm_bufio_new); void dm_bufio_prefetch(struct dm_bufio_client *c, - sector_t block, unsigned int n_blocks) + sector_t block, unsigned int n_blocks, unsigned short ioprio) { struct blk_plug plug; @@ -1950,7 +1951,7 @@ void dm_bufio_prefetch(struct dm_bufio_client *c, dm_bufio_unlock(c); if (need_submit) - submit_io(b, REQ_OP_READ, read_endio); + submit_io(b, REQ_OP_READ, ioprio, read_endio); dm_bufio_release(b); cond_resched(); diff --git a/drivers/md/dm-ebs-target.c b/drivers/md/dm-ebs-target.c index 435b45201f4d..8198c8a7b416 100644 --- a/drivers/md/dm-ebs-target.c +++ b/drivers/md/dm-ebs-target.c @@ -84,7 +84,7 @@ static int __ebs_rw_bvec(struct ebs_c *ec, enum req_op op, struct bio_vec *bv, /* Avoid reading for writes in case bio vector's page overwrites block completely. */ if (op == REQ_OP_READ || buf_off || bv_len < dm_bufio_get_block_size(ec->bufio)) - ba = dm_bufio_read(ec->bufio, block, &b); + ba = dm_bufio_read(ec->bufio, block, &b, IOPRIO_DEFAULT); else ba = dm_bufio_new(ec->bufio, block, &b); @@ -194,13 +194,13 @@ static void __ebs_process_bios(struct work_struct *ws) bio_list_for_each(bio, &bios) { block1 = __sector_to_block(ec, bio->bi_iter.bi_sector); if (bio_op(bio) == REQ_OP_READ) - dm_bufio_prefetch(ec->bufio, block1, __nr_blocks(ec, bio)); + dm_bufio_prefetch(ec->bufio, block1, __nr_blocks(ec, bio), IOPRIO_DEFAULT); else if (bio_op(bio) == REQ_OP_WRITE && !(bio->bi_opf & REQ_PREFLUSH)) { block2 = __sector_to_block(ec, bio_end_sector(bio)); if (__block_mod(bio->bi_iter.bi_sector, ec->u_bs)) - dm_bufio_prefetch(ec->bufio, block1, 1); + dm_bufio_prefetch(ec->bufio, block1, 1, IOPRIO_DEFAULT); if (__block_mod(bio_end_sector(bio), ec->u_bs) && block2 != block1) - dm_bufio_prefetch(ec->bufio, block2, 1); + dm_bufio_prefetch(ec->bufio, block2, 1, IOPRIO_DEFAULT); } } diff --git a/drivers/md/dm-integrity.c b/drivers/md/dm-integrity.c index 9ffd093ad6cc..1e40e712bcd7 100644 --- a/drivers/md/dm-integrity.c +++ b/drivers/md/dm-integrity.c @@ -1418,7 +1418,7 @@ static int dm_integrity_rw_tag(struct dm_integrity_c *ic, unsigned char *tag, se if (unlikely(r)) return r; - data = dm_bufio_read(ic->bufio, *metadata_block, &b); + data = dm_bufio_read(ic->bufio, *metadata_block, &b, IOPRIO_DEFAULT); if (IS_ERR(data)) return PTR_ERR(data); diff --git a/drivers/md/dm-snap-persistent.c b/drivers/md/dm-snap-persistent.c index 568d10842b1f..a2072b95e28c 100644 --- a/drivers/md/dm-snap-persistent.c +++ b/drivers/md/dm-snap-persistent.c @@ -524,7 +524,7 @@ static int read_exceptions(struct pstore *ps, if (unlikely(pf_chunk >= dm_bufio_get_device_size(client))) break; - dm_bufio_prefetch(client, pf_chunk, 1); + dm_bufio_prefetch(client, pf_chunk, 1, IOPRIO_DEFAULT); prefetch_area++; if (unlikely(!prefetch_area)) break; @@ -533,7 +533,7 @@ static int read_exceptions(struct pstore *ps, chunk = area_location(ps, ps->current_area); - area = dm_bufio_read(client, chunk, &bp); + area = dm_bufio_read(client, chunk, &bp, IOPRIO_DEFAULT); if (IS_ERR(area)) { r = PTR_ERR(area); goto ret_destroy_bufio; diff --git a/drivers/md/dm-verity-fec.c b/drivers/md/dm-verity-fec.c index 3ef9f018da60..715173cbf0ee 100644 --- a/drivers/md/dm-verity-fec.c +++ b/drivers/md/dm-verity-fec.c @@ -68,7 +68,7 @@ static u8 *fec_read_parity(struct dm_verity *v, u64 rsb, int index, block = div64_u64_rem(position, v->fec->io_size, &rem); *offset = (unsigned int)rem; - res = dm_bufio_read(v->fec->bufio, block, buf); + res = dm_bufio_read(v->fec->bufio, block, buf, IOPRIO_DEFAULT); if (IS_ERR(res)) { DMERR("%s: FEC %llu: parity read failed (block %llu): %ld", v->data_dev->name, (unsigned long long)rsb, @@ -247,7 +247,7 @@ static int fec_read_bufs(struct dm_verity *v, struct dm_verity_io *io, bufio = v->bufio; } - bbuf = dm_bufio_read(bufio, block, &buf); + bbuf = dm_bufio_read(bufio, block, &buf, IOPRIO_DEFAULT); if (IS_ERR(bbuf)) { DMWARN_LIMIT("%s: FEC %llu: read failed (%llu): %ld", v->data_dev->name, diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c index 26adcfea0302..0038e168f3d7 100644 --- a/drivers/md/dm-verity-target.c +++ b/drivers/md/dm-verity-target.c @@ -307,7 +307,7 @@ static int verity_verify_level(struct dm_verity *v, struct dm_verity_io *io, return -EAGAIN; } } else - data = dm_bufio_read(v->bufio, hash_block, &buf); + data = dm_bufio_read(v->bufio, hash_block, &buf, IOPRIO_DEFAULT); if (IS_ERR(data)) return PTR_ERR(data); @@ -717,7 +717,7 @@ static void verity_prefetch_io(struct work_struct *work) } no_prefetch_cluster: dm_bufio_prefetch(v->bufio, hash_block_start, - hash_block_end - hash_block_start + 1); + hash_block_end - hash_block_start + 1, IOPRIO_DEFAULT); } kfree(pw); diff --git a/drivers/md/persistent-data/dm-block-manager.c b/drivers/md/persistent-data/dm-block-manager.c index 0e010e1204aa..86a4f73d2f3d 100644 --- a/drivers/md/persistent-data/dm-block-manager.c +++ b/drivers/md/persistent-data/dm-block-manager.c @@ -474,7 +474,7 @@ int dm_bm_read_lock(struct dm_block_manager *bm, dm_block_t b, void *p; int r; - p = dm_bufio_read(bm->bufio, b, (struct dm_buffer **) result); + p = dm_bufio_read(bm->bufio, b, (struct dm_buffer **) result, IOPRIO_DEFAULT); if (IS_ERR(p)) return PTR_ERR(p); @@ -510,7 +510,7 @@ int dm_bm_write_lock(struct dm_block_manager *bm, if (dm_bm_is_read_only(bm)) return -EPERM; - p = dm_bufio_read(bm->bufio, b, (struct dm_buffer **) result); + p = dm_bufio_read(bm->bufio, b, (struct dm_buffer **) result, IOPRIO_DEFAULT); if (IS_ERR(p)) return PTR_ERR(p); @@ -624,7 +624,7 @@ EXPORT_SYMBOL_GPL(dm_bm_flush); void dm_bm_prefetch(struct dm_block_manager *bm, dm_block_t b) { - dm_bufio_prefetch(bm->bufio, b, 1); + dm_bufio_prefetch(bm->bufio, b, 1, IOPRIO_DEFAULT); } bool dm_bm_is_read_only(struct dm_block_manager *bm) diff --git a/include/linux/dm-bufio.h b/include/linux/dm-bufio.h index 75e7d8cbb532..6cdd9cb66dd5 100644 --- a/include/linux/dm-bufio.h +++ b/include/linux/dm-bufio.h @@ -62,7 +62,7 @@ void dm_bufio_set_sector_offset(struct dm_bufio_client *c, sector_t start); * it dirty. */ void *dm_bufio_read(struct dm_bufio_client *c, sector_t block, - struct dm_buffer **bp); + struct dm_buffer **bp, unsigned short ioprio); /* * Like dm_bufio_read, but return buffer from cache, don't read @@ -84,8 +84,7 @@ void *dm_bufio_new(struct dm_bufio_client *c, sector_t block, * I/O to finish. */ void dm_bufio_prefetch(struct dm_bufio_client *c, - sector_t block, unsigned int n_blocks); - + sector_t block, unsigned int n_blocks, unsigned short ioprio); /* * Release a reference obtained with dm_bufio_{read,get,new}. The data * pointer and dm_buffer pointer is no longer valid after this call. From patchwork Wed Dec 13 10:42:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongyu Jin X-Patchwork-Id: 13490651 X-Patchwork-Delegate: snitzer@redhat.com Received: from mail-io1-f51.google.com (mail-io1-f51.google.com [209.85.166.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4691622EE2 for ; Wed, 13 Dec 2023 10:42:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="g0SqtNn8" Received: by mail-io1-f51.google.com with SMTP id ca18e2360f4ac-7b751f23ff1so123236139f.2 for ; Wed, 13 Dec 2023 02:42:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702464175; x=1703068975; darn=lists.linux.dev; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vN+RlAGZeGH2zMVmYb6ghj5GDWJcjTyD9/i+J4cSzo8=; b=g0SqtNn8IjkgPs/sjG13ZuLuGCiZWT5u9FKn63TfX8MVhQqndURe6xKJK6Z1okHQKs JcYX1H22IT5sbEoyBJNmMhxFvnO8rMcb8iEB4yzFb9LPYNs6WI5X9owz8UY1AsbPRXg9 jqEs/BHpjDflPvQpGMx8DhWLm7dXDtN2h4chxZRGq4fANK4rZVoiMQAf9edzF+2fwtYa mo2N8ECvjQ4c7wCHeE210e+nvxpJ5FMa5y+Fm5YZ5CwE6nBcTLalAsDVuxMqN1VvckW0 qsEn6tXsHowCOraAvAhVC0V9gCVyo95R3NkEaXKV24iu/Ds2GfFznsZ/47fueBlLbXli EkAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702464175; x=1703068975; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vN+RlAGZeGH2zMVmYb6ghj5GDWJcjTyD9/i+J4cSzo8=; b=gixto2e010d5aCQUP14QXono5V44kT1JOWlOQvpOJvHlmIVEoy70hpkaGQRsdqtqia VzzLfCmYdfMLyYpR1wT6j8hAjk/vmzN+jNJBOiIu+cy4IhBCwX8zFPTYI6cRL11H/HZX ALzpsEbnaV0v1Snxv02fipf9Z5wNPSy+d52i8vmd+FKAP+kQvVFTwBWm9soh3ohGCPdO 4AGPuZ3Kfil+DZJYL7+zQoslF4BXGo++rNiQSIzVc7XYLiYNuJvIArOA1921V6y/yv87 atWwhsWn8pmrItY6GoyadmdcbYmcVaEy94IbyPb5MaiZkjZ3Zf7cKFW3QbDIJrhkJp5C Gp8A== X-Gm-Message-State: AOJu0YyzcDU37xA8urdez2u7jIsDRCTZ1EyRZyDbXb+eclGwtwKs7NMb tmOiDRzjBVhcF/6FaU1rNtlZtHr9pkQ= X-Google-Smtp-Source: AGHT+IGoch6RTIOwXmPLNxfnlyc0R6ReURZ94TXC+ilir7bZMnEgbfvKhLu5OZ1NPGQWdG/WIsk0qw== X-Received: by 2002:a6b:dc05:0:b0:7b7:c41:4bf3 with SMTP id s5-20020a6bdc05000000b007b70c414bf3mr10133796ioc.14.1702464175434; Wed, 13 Dec 2023 02:42:55 -0800 (PST) Received: from ubuntu.. ([117.18.48.102]) by smtp.gmail.com with ESMTPSA id c11-20020a6b4e0b000000b0078647b08ab0sm3058186iob.6.2023.12.13.02.42.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Dec 2023 02:42:55 -0800 (PST) From: Hongyu Jin To: agk@redhat.com, snitzer@kernel.org, mpatocka@redhat.com, axboe@kernel.dk, ebiggers@kernel.org Cc: zhiguo.niu@unisoc.com, ke.wang@unisoc.com, yibin.ding@unisoc.com, hongyu.jin@unisoc.com, linux-kernel@vger.kernel.org, dm-devel@lists.linux.dev, linux-block@vger.kernel.org Subject: [PATCH v5 4/5] dm verity: Fix I/O priority lost when read FEC and hash Date: Wed, 13 Dec 2023 18:42:15 +0800 Message-Id: <20231213104216.27845-5-hongyu.jin.cn@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231213104216.27845-1-hongyu.jin.cn@gmail.com> References: <20231213104216.27845-1-hongyu.jin.cn@gmail.com> Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Hongyu Jin To fix this problem, when read FEC and hash from disk, I/O priority are inconsistent with data block and blocked by other I/O with low I/O priority. Make I/O for FEC and hash has same I/O priority with original data I/O. Co-developed-by: Yibin Ding Signed-off-by: Yibin Ding Signed-off-by: Hongyu Jin --- drivers/md/dm-verity-fec.c | 3 ++- drivers/md/dm-verity-target.c | 8 ++++++-- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/drivers/md/dm-verity-fec.c b/drivers/md/dm-verity-fec.c index 715173cbf0ee..6a5a679e7e8a 100644 --- a/drivers/md/dm-verity-fec.c +++ b/drivers/md/dm-verity-fec.c @@ -209,6 +209,7 @@ static int fec_read_bufs(struct dm_verity *v, struct dm_verity_io *io, u8 *bbuf, *rs_block; u8 want_digest[HASH_MAX_DIGESTSIZE]; unsigned int n, k; + struct bio *bio = dm_bio_from_per_bio_data(io, v->ti->per_io_data_size); if (neras) *neras = 0; @@ -247,7 +248,7 @@ static int fec_read_bufs(struct dm_verity *v, struct dm_verity_io *io, bufio = v->bufio; } - bbuf = dm_bufio_read(bufio, block, &buf, IOPRIO_DEFAULT); + bbuf = dm_bufio_read(bufio, block, &buf, bio_prio(bio)); if (IS_ERR(bbuf)) { DMWARN_LIMIT("%s: FEC %llu: read failed (%llu): %ld", v->data_dev->name, diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c index 0038e168f3d7..8c911b6722ce 100644 --- a/drivers/md/dm-verity-target.c +++ b/drivers/md/dm-verity-target.c @@ -51,6 +51,7 @@ static DEFINE_STATIC_KEY_FALSE(use_tasklet_enabled); struct dm_verity_prefetch_work { struct work_struct work; struct dm_verity *v; + unsigned short ioprio; sector_t block; unsigned int n_blocks; }; @@ -293,6 +294,7 @@ static int verity_verify_level(struct dm_verity *v, struct dm_verity_io *io, int r; sector_t hash_block; unsigned int offset; + struct bio *bio = dm_bio_from_per_bio_data(io, v->ti->per_io_data_size); verity_hash_at_level(v, block, level, &hash_block, &offset); @@ -307,7 +309,7 @@ static int verity_verify_level(struct dm_verity *v, struct dm_verity_io *io, return -EAGAIN; } } else - data = dm_bufio_read(v->bufio, hash_block, &buf, IOPRIO_DEFAULT); + data = dm_bufio_read(v->bufio, hash_block, &buf, bio_prio(bio)); if (IS_ERR(data)) return PTR_ERR(data); @@ -717,7 +719,7 @@ static void verity_prefetch_io(struct work_struct *work) } no_prefetch_cluster: dm_bufio_prefetch(v->bufio, hash_block_start, - hash_block_end - hash_block_start + 1, IOPRIO_DEFAULT); + hash_block_end - hash_block_start + 1, pw->ioprio); } kfree(pw); @@ -728,6 +730,7 @@ static void verity_submit_prefetch(struct dm_verity *v, struct dm_verity_io *io) sector_t block = io->block; unsigned int n_blocks = io->n_blocks; struct dm_verity_prefetch_work *pw; + struct bio *bio = dm_bio_from_per_bio_data(io, v->ti->per_io_data_size); if (v->validated_blocks) { while (n_blocks && test_bit(block, v->validated_blocks)) { @@ -751,6 +754,7 @@ static void verity_submit_prefetch(struct dm_verity *v, struct dm_verity_io *io) pw->v = v; pw->block = block; pw->n_blocks = n_blocks; + pw->ioprio = bio_prio(bio); queue_work(v->verify_wq, &pw->work); } From patchwork Wed Dec 13 10:42:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hongyu Jin X-Patchwork-Id: 13490652 X-Patchwork-Delegate: snitzer@redhat.com Received: from mail-io1-f44.google.com (mail-io1-f44.google.com [209.85.166.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2889122EF6 for ; Wed, 13 Dec 2023 10:43:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dg75IXOa" Received: by mail-io1-f44.google.com with SMTP id ca18e2360f4ac-7b435966249so297655039f.0 for ; Wed, 13 Dec 2023 02:42:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702464179; x=1703068979; darn=lists.linux.dev; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fRWJvmM8+hpVsGJoyvu2YuvVcQVJUfQcuTmMveSE8/s=; b=dg75IXOaiVwGYhhvdOsHYmS1cZqIE+a4zoR2f4lIgpVVe/E8gxpU/EyFJQGwhLXPJ3 dGvg2nRfyWzpxX4QSwS5uicDMBv6rAxDlPXCJKLYMK8lue9SlSuozVQCnY7wLyJ81Qma iEqXRbluEJo07jz29zoQGQdW3M6Mjk1SSpzifDvpCtUxIhfOPieUXIX1N/wvi6wwxioB gTRw3UipyFohX2rcZndSOYomqH0fybhxnJ/7fhrYepJf98+gFSnl0/4kFIVlMUh3aWZa t1zANIxxR8v6rym09RkThOIy8wPHouy873xbMsGtXa0GE0RhC7VJdPlL9vn331TJv4Z+ 3XIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702464179; x=1703068979; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fRWJvmM8+hpVsGJoyvu2YuvVcQVJUfQcuTmMveSE8/s=; b=kGwyGhZpyEp/oFJUjYMwQMFa7WdZV/ZoCudVNMKOlRgkp2e5O+Ofq+DKttIVkt1iCI 11A+UZGhLuwm72kJ+VfgPL1C16myILKS6bTIWDE6sRkBJKp6DkFk4Luy305tJzR1BJBm Soi/W03176K01hpN+Hl7g7eymGGhYug/aHgkxhqTsCzGiSndKYkBb+EkoJHE87LyTnIg ilguot1RPijYs0TE1t0iA7Poj6Uj1TVC1GZ9gsyhcLlR50xhF8Tb/xQnXwIbIBywiEVi FxxV5Fjx56qmI8JTN7JkrtQyzN20PULzqkokCaH3SbEagI4v5e0p+ZMXuJf6AH9SFz6M Yqyw== X-Gm-Message-State: AOJu0Yz9ACzojlUN0poWLCCOu+3OsvW9YrdtFTLw2UdevDZnIMyTV+N4 kHgFmsexOmwbdxQiY17LH/o= X-Google-Smtp-Source: AGHT+IFqnQH5xHRTW7lkDH3nUXotSoQk9CUeHU1CHahsXGxY0QsPY5pXBPOQ53OTwye2jv3CgtPipw== X-Received: by 2002:a5e:c742:0:b0:7b7:1717:693f with SMTP id g2-20020a5ec742000000b007b71717693fmr8044235iop.10.1702464179355; Wed, 13 Dec 2023 02:42:59 -0800 (PST) Received: from ubuntu.. ([117.18.48.102]) by smtp.gmail.com with ESMTPSA id c11-20020a6b4e0b000000b0078647b08ab0sm3058186iob.6.2023.12.13.02.42.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Dec 2023 02:42:59 -0800 (PST) From: Hongyu Jin To: agk@redhat.com, snitzer@kernel.org, mpatocka@redhat.com, axboe@kernel.dk, ebiggers@kernel.org Cc: zhiguo.niu@unisoc.com, ke.wang@unisoc.com, yibin.ding@unisoc.com, hongyu.jin@unisoc.com, linux-kernel@vger.kernel.org, dm-devel@lists.linux.dev, linux-block@vger.kernel.org Subject: [PATCH v5 5/5] dm-crypt: Fix lost ioprio when queuing write bios Date: Wed, 13 Dec 2023 18:42:16 +0800 Message-Id: <20231213104216.27845-6-hongyu.jin.cn@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231213104216.27845-1-hongyu.jin.cn@gmail.com> References: <20231213104216.27845-1-hongyu.jin.cn@gmail.com> Precedence: bulk X-Mailing-List: dm-devel@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Hongyu Jin The original submitting bio->bi_ioprio setting can be retained by struct dm_crypt_io::base_bio, we set the original bio's ioprio to the cloned bio for write. Link: https://lore.kernel.org/dm-devel/alpine.LRH.2.11.1612141049250.13402@mail.ewheeler.net Signed-off-by: Hongyu Jin --- drivers/md/dm-crypt.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c index 6de107aff331..7149da6555b8 100644 --- a/drivers/md/dm-crypt.c +++ b/drivers/md/dm-crypt.c @@ -1683,6 +1683,7 @@ static struct bio *crypt_alloc_buffer(struct dm_crypt_io *io, unsigned int size) GFP_NOIO, &cc->bs); clone->bi_private = io; clone->bi_end_io = crypt_endio; + clone->bi_ioprio = io->base_bio->bi_ioprio; remaining_size = size;