| Message ID | 20240415194921.6404-1-bvanassche@acm.org (mailing list archive) |
|---|---|
| State | Rejected, archived |
| Delegated to: | Mike Snitzer |
| Headers | show |
| Series | dm: Change the default value of rq_affinity from 0 into 1 | expand |
On Mon, 15 Apr 2024, Bart Van Assche wrote: > The following behavior is inconsistent: > * For request-based dm queues the default value of rq_affinity is 1. > * For bio-based dm queues the default value of rq_affinity is 0. > > The default value for request-based dm queues is 1 because of the following > code in blk_mq_init_allocated_queue(): > > q->queue_flags |= QUEUE_FLAG_MQ_DEFAULT; > > >From <linux/blkdev.h>: > > #define QUEUE_FLAG_MQ_DEFAULT ((1UL << QUEUE_FLAG_IO_STAT) | \ > (1UL << QUEUE_FLAG_SAME_COMP) | \ > (1UL << QUEUE_FLAG_NOWAIT)) > > The default value of rq_affinity for bio-based dm queues is 0 because the > dm alloc_dev() function does not set any of the QUEUE_FLAG_SAME_* flags. I > think the different default values are the result of an oversight when > blk-mq support was added in the device mapper code. Hence this patch that > changes the default value of rq_affinity from 0 to 1 for bio-based dm > queues. > > This patch reduces the boot time from 12.23 to 12.20 seconds on my test Are you sure that this is not jitter? I am wondering how should QUEUE_FLAG_SAME_COMP work for bio-based devices. I grepped the kernel for QUEUE_FLAG_SAME_COMP and it is tested in block/blk-mq.c in blk_mq_complete_need_ipi (this code path is taken only for request-based devices) and in block/blk-sysfs.c in queue_rq_affinity_show (this just displays the value in sysfs). There are no other places where QUEUE_FLAG_SAME_COMP is tested, so I don't see what effect is it supposed to have. Mikulas > setup, a Pixel 2023 development board. The storage controller on that test > setup supports a single completion interrupt and hence benefits from > redirecting I/O completions to a CPU core that is closer to the submitter. > > Cc: Mikulas Patocka <mpatocka@redhat.com> > Cc: Eric Biggers <ebiggers@kernel.org> > Cc: Jaegeuk Kim <jaegeuk@kernel.org> > Cc: Daniel Lee <chullee@google.com> > Cc: stable@vger.kernel.org > Fixes: bfebd1cdb497 ("dm: add full blk-mq support to request-based DM") > Signed-off-by: Bart Van Assche <bvanassche@acm.org> > --- > drivers/md/dm.c | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/drivers/md/dm.c b/drivers/md/dm.c > index 56aa2a8b9d71..9af216c11cf7 100644 > --- a/drivers/md/dm.c > +++ b/drivers/md/dm.c > @@ -2106,6 +2106,7 @@ static struct mapped_device *alloc_dev(int minor) > if (IS_ERR(md->disk)) > goto bad; > md->queue = md->disk->queue; > + blk_queue_flag_set(QUEUE_FLAG_SAME_COMP, md->queue); > > init_waitqueue_head(&md->wait); > INIT_WORK(&md->work, dm_wq_work); >
On 4/15/24 12:56, Mikulas Patocka wrote: > I am wondering how should QUEUE_FLAG_SAME_COMP work for bio-based > devices. > > I grepped the kernel for QUEUE_FLAG_SAME_COMP and it is tested in > block/blk-mq.c in blk_mq_complete_need_ipi (this code path is taken only > for request-based devices) and in block/blk-sysfs.c in > queue_rq_affinity_show (this just displays the value in sysfs). There are > no other places where QUEUE_FLAG_SAME_COMP is tested, so I don't see what > effect is it supposed to have. I think the answer depends on whether or not the underlying device defines the .submit_bio() callback. From block/blk-core.c: static void __submit_bio(struct bio *bio) { if (unlikely(!blk_crypto_bio_prep(&bio))) return; if (!bio->bi_bdev->bd_has_submit_bio) { blk_mq_submit_bio(bio); } else if (likely(bio_queue_enter(bio) == 0)) { struct gendisk *disk = bio->bi_bdev->bd_disk; disk->fops->submit_bio(bio); blk_queue_exit(disk->queue); } } In other words, if the .submit_bio() callback is defined, that function is called. If it is not defined, blk_mq_submit_bio() converts the bio into a request. QUEUE_FLAG_SAME_COMP affects the request completion path. On my test setup there are multiple dm instances defined on top of SCSI devices. The SCSI core does not implement the .submit_bio() callback. Thanks, Bart.
On Mon, 15 Apr 2024, Bart Van Assche wrote: > On 4/15/24 12:56, Mikulas Patocka wrote: > > I am wondering how should QUEUE_FLAG_SAME_COMP work for bio-based > > devices. > > > > I grepped the kernel for QUEUE_FLAG_SAME_COMP and it is tested in > > block/blk-mq.c in blk_mq_complete_need_ipi (this code path is taken only > > for request-based devices) and in block/blk-sysfs.c in > > queue_rq_affinity_show (this just displays the value in sysfs). There are > > no other places where QUEUE_FLAG_SAME_COMP is tested, so I don't see what > > effect is it supposed to have. > > I think the answer depends on whether or not the underlying device > defines the .submit_bio() callback. From block/blk-core.c: > > static void __submit_bio(struct bio *bio) > { > if (unlikely(!blk_crypto_bio_prep(&bio))) > return; > > if (!bio->bi_bdev->bd_has_submit_bio) { > blk_mq_submit_bio(bio); > } else if (likely(bio_queue_enter(bio) == 0)) { > struct gendisk *disk = bio->bi_bdev->bd_disk; > > disk->fops->submit_bio(bio); > blk_queue_exit(disk->queue); > } > } > > In other words, if the .submit_bio() callback is defined, that function > is called. If it is not defined, blk_mq_submit_bio() converts the bio > into a request. QUEUE_FLAG_SAME_COMP affects the request completion > path. On my test setup there are multiple dm instances defined on top of > SCSI devices. The SCSI core does not implement the .submit_bio() > callback. > > Thanks, > > Bart. Yes, setting the flag QUEUE_FLAG_SAME_COMP for bio-based drivers (those that define .submit_bio) has no effect. So, I think that you patch doesn't have any effect too. Mikulas
diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 56aa2a8b9d71..9af216c11cf7 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -2106,6 +2106,7 @@ static struct mapped_device *alloc_dev(int minor) if (IS_ERR(md->disk)) goto bad; md->queue = md->disk->queue; + blk_queue_flag_set(QUEUE_FLAG_SAME_COMP, md->queue); init_waitqueue_head(&md->wait); INIT_WORK(&md->work, dm_wq_work);
The following behavior is inconsistent: * For request-based dm queues the default value of rq_affinity is 1. * For bio-based dm queues the default value of rq_affinity is 0. The default value for request-based dm queues is 1 because of the following code in blk_mq_init_allocated_queue(): q->queue_flags |= QUEUE_FLAG_MQ_DEFAULT; From <linux/blkdev.h>: #define QUEUE_FLAG_MQ_DEFAULT ((1UL << QUEUE_FLAG_IO_STAT) | \ (1UL << QUEUE_FLAG_SAME_COMP) | \ (1UL << QUEUE_FLAG_NOWAIT)) The default value of rq_affinity for bio-based dm queues is 0 because the dm alloc_dev() function does not set any of the QUEUE_FLAG_SAME_* flags. I think the different default values are the result of an oversight when blk-mq support was added in the device mapper code. Hence this patch that changes the default value of rq_affinity from 0 to 1 for bio-based dm queues. This patch reduces the boot time from 12.23 to 12.20 seconds on my test setup, a Pixel 2023 development board. The storage controller on that test setup supports a single completion interrupt and hence benefits from redirecting I/O completions to a CPU core that is closer to the submitter. Cc: Mikulas Patocka <mpatocka@redhat.com> Cc: Eric Biggers <ebiggers@kernel.org> Cc: Jaegeuk Kim <jaegeuk@kernel.org> Cc: Daniel Lee <chullee@google.com> Cc: stable@vger.kernel.org Fixes: bfebd1cdb497 ("dm: add full blk-mq support to request-based DM") Signed-off-by: Bart Van Assche <bvanassche@acm.org> --- drivers/md/dm.c | 1 + 1 file changed, 1 insertion(+)