From patchwork Tue Mar 2 19:05:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mikulas Patocka X-Patchwork-Id: 12112391 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3139FC4332B for ; Tue, 2 Mar 2021 23:53:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E339264F40 for ; Tue, 2 Mar 2021 23:52:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231575AbhCBXrh (ORCPT ); Tue, 2 Mar 2021 18:47:37 -0500 Received: from smtp02.tmcz.cz ([93.153.104.113]:47295 "EHLO smtp02.tmcz.cz" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1581890AbhCBT2p (ORCPT ); Tue, 2 Mar 2021 14:28:45 -0500 Received: from leontynka.twibright.com (109-183-129-149.customers.tmcz.cz [109.183.129.149]) by smtp02.tmcz.cz (Postfix) with ESMTPS id 01629405E0; Tue, 2 Mar 2021 20:05:54 +0100 (CET) Received: from debian-a64.vm ([192.168.208.2]) by leontynka.twibright.com with smtp (Exim 4.92) (envelope-from ) id 1lHALA-0003k7-OQ; Tue, 02 Mar 2021 20:05:53 +0100 Received: by debian-a64.vm (sSMTP sendmail emulation); Tue, 02 Mar 2021 20:05:51 +0100 Message-Id: <20210302190551.473015400@debian-a64.vm> User-Agent: quilt/0.65 Date: Tue, 02 Mar 2021 20:05:14 +0100 From: Mikulas Patocka To: JeffleXu , Mike Snitzer , Heinz Mauelshagen , axboe@kernel.dk, caspar@linux.alibaba.com, io-uring@vger.kernel.org, linux-block@vger.kernel.org, joseph.qi@linux.alibaba.com, dm-devel@redhat.com, hch@lst.de Cc: Mikulas Patocka Subject: [PATCH 1/4] block: introduce a function submit_bio_noacct_mq_direct MIME-Version: 1.0 Content-Disposition: inline; filename=block-submit-bio-mq-direct.patch Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Introduce a function that submits bio to a request-based device driver. The function doesn't offload requests to current->bio_list. It is expected to be called from device mapper, where current->bio_list is already set up. Signed-off-by: Mikulas Patocka --- block/blk-core.c | 16 ++++++++++++++++ include/linux/blkdev.h | 1 + 2 files changed, 17 insertions(+) Index: linux-2.6/block/blk-core.c =================================================================== --- linux-2.6.orig/block/blk-core.c 2021-03-01 19:47:27.000000000 +0100 +++ linux-2.6/block/blk-core.c 2021-03-02 10:43:28.000000000 +0100 @@ -992,6 +992,22 @@ static blk_qc_t __submit_bio_noacct(stru return ret; } +blk_qc_t submit_bio_noacct_mq_direct(struct bio *bio) +{ + struct gendisk *disk = bio->bi_bdev->bd_disk; + + if (unlikely(bio_queue_enter(bio) != 0)) + return BLK_QC_T_NONE; + + if (!blk_crypto_bio_prep(&bio)) { + blk_queue_exit(disk->queue); + return BLK_QC_T_NONE; + } + + return blk_mq_submit_bio(bio); +} +EXPORT_SYMBOL(submit_bio_noacct_mq_direct); + static blk_qc_t __submit_bio_noacct_mq(struct bio *bio) { struct bio_list bio_list[2] = { }; Index: linux-2.6/include/linux/blkdev.h =================================================================== --- linux-2.6.orig/include/linux/blkdev.h 2021-03-01 19:47:29.000000000 +0100 +++ linux-2.6/include/linux/blkdev.h 2021-03-02 10:44:04.000000000 +0100 @@ -912,6 +912,7 @@ static inline void rq_flush_dcache_pages extern int blk_register_queue(struct gendisk *disk); extern void blk_unregister_queue(struct gendisk *disk); +blk_qc_t submit_bio_noacct_mq_direct(struct bio *bio); blk_qc_t submit_bio_noacct(struct bio *bio); extern void blk_rq_init(struct request_queue *q, struct request *rq); extern void blk_put_request(struct request *); From patchwork Tue Mar 2 19:05:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mikulas Patocka X-Patchwork-Id: 12112383 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93FB3C433E6 for ; Tue, 2 Mar 2021 23:52:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 629C864F41 for ; Tue, 2 Mar 2021 23:52:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231710AbhCBXrC (ORCPT ); Tue, 2 Mar 2021 18:47:02 -0500 Received: from smtp01.tmcz.cz ([93.153.104.112]:32050 "EHLO smtp01.tmcz.cz" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1581885AbhCBT2l (ORCPT ); Tue, 2 Mar 2021 14:28:41 -0500 Received: from leontynka.twibright.com (109-183-129-149.customers.tmcz.cz [109.183.129.149]) by smtp01.tmcz.cz (Postfix) with ESMTPS id 38EDF4039C; Tue, 2 Mar 2021 20:05:55 +0100 (CET) Received: from debian-a64.vm ([192.168.208.2]) by leontynka.twibright.com with smtp (Exim 4.92) (envelope-from ) id 1lHALB-0003kD-WF; Tue, 02 Mar 2021 20:05:55 +0100 Received: by debian-a64.vm (sSMTP sendmail emulation); Tue, 02 Mar 2021 20:05:52 +0100 Message-Id: <20210302190552.715551440@debian-a64.vm> User-Agent: quilt/0.65 Date: Tue, 02 Mar 2021 20:05:15 +0100 From: Mikulas Patocka To: JeffleXu , Mike Snitzer , Heinz Mauelshagen , axboe@kernel.dk, caspar@linux.alibaba.com, io-uring@vger.kernel.org, linux-block@vger.kernel.org, joseph.qi@linux.alibaba.com, dm-devel@redhat.com, hch@lst.de Cc: Mikulas Patocka Subject: [PATCH 2/4] block: dont clear REQ_HIPRI for bio-based drivers MIME-Version: 1.0 Content-Disposition: inline; filename=block-allow-bio-hipri.patch Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Don't clear REQ_HIPRI for bio-based drivers. Device mapper will need to see this flag in order to support polling. Signed-off-by: Mikulas Patocka --- block/blk-core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) Index: linux-2.6/block/blk-core.c =================================================================== --- linux-2.6.orig/block/blk-core.c 2021-03-02 10:43:28.000000000 +0100 +++ linux-2.6/block/blk-core.c 2021-03-02 10:53:50.000000000 +0100 @@ -836,7 +836,7 @@ static noinline_for_stack bool submit_bi } } - if (!test_bit(QUEUE_FLAG_POLL, &q->queue_flags)) + if (!test_bit(QUEUE_FLAG_POLL, &q->queue_flags) && !bdev->bd_disk->fops->submit_bio) bio->bi_opf &= ~REQ_HIPRI; switch (bio_op(bio)) { From patchwork Tue Mar 2 19:05:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mikulas Patocka X-Patchwork-Id: 12112389 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DC14C43381 for ; Tue, 2 Mar 2021 23:53:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BA92964F3A for ; Tue, 2 Mar 2021 23:52:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231384AbhCBXr3 (ORCPT ); Tue, 2 Mar 2021 18:47:29 -0500 Received: from smtp01.tmcz.cz ([93.153.104.112]:33440 "EHLO smtp01.tmcz.cz" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1581887AbhCBT2o (ORCPT ); Tue, 2 Mar 2021 14:28:44 -0500 Received: from leontynka.twibright.com (109-183-129-149.customers.tmcz.cz [109.183.129.149]) by smtp01.tmcz.cz (Postfix) with ESMTPS id 81892405A8; Tue, 2 Mar 2021 20:05:56 +0100 (CET) Received: from debian-a64.vm ([192.168.208.2]) by leontynka.twibright.com with smtp (Exim 4.92) (envelope-from ) id 1lHALD-0003kH-7p; Tue, 02 Mar 2021 20:05:56 +0100 Received: by debian-a64.vm (sSMTP sendmail emulation); Tue, 02 Mar 2021 20:05:54 +0100 Message-Id: <20210302190553.961608080@debian-a64.vm> User-Agent: quilt/0.65 Date: Tue, 02 Mar 2021 20:05:16 +0100 From: Mikulas Patocka To: JeffleXu , Mike Snitzer , Heinz Mauelshagen , axboe@kernel.dk, caspar@linux.alibaba.com, io-uring@vger.kernel.org, linux-block@vger.kernel.org, joseph.qi@linux.alibaba.com, dm-devel@redhat.com, hch@lst.de Cc: Mikulas Patocka Subject: [PATCH 3/4] dm: use submit_bio_noacct_mq_direct MIME-Version: 1.0 Content-Disposition: inline; filename=dm-submit-bio-mq-direct.patch Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Use submit_bio_noacct_mq_direct and pass the returned cookie through the device mapper stack. The cookie and queue is stored in the structure clone_info. Signed-off-by: Mikulas Patocka --- drivers/md/dm.c | 42 +++++++++++++++++++++++------------------- 1 file changed, 23 insertions(+), 19 deletions(-) Index: linux-2.6/drivers/md/dm.c =================================================================== --- linux-2.6.orig/drivers/md/dm.c 2021-03-02 19:24:13.000000000 +0100 +++ linux-2.6/drivers/md/dm.c 2021-03-02 19:25:54.000000000 +0100 @@ -72,6 +72,8 @@ struct clone_info { struct dm_io *io; sector_t sector; unsigned sector_count; + blk_qc_t poll_cookie; + struct request_queue *poll_queue; }; /* @@ -1294,14 +1296,13 @@ static noinline void __set_swap_bios_lim mutex_unlock(&md->swap_bios_lock); } -static blk_qc_t __map_bio(struct dm_target_io *tio) +static void __map_bio(struct clone_info *ci, struct dm_target_io *tio) { int r; sector_t sector; struct bio *clone = &tio->clone; struct dm_io *io = tio->io; struct dm_target *ti = tio->ti; - blk_qc_t ret = BLK_QC_T_NONE; clone->bi_end_io = clone_endio; @@ -1328,7 +1329,14 @@ static blk_qc_t __map_bio(struct dm_targ case DM_MAPIO_REMAPPED: /* the bio has been remapped so dispatch it */ trace_block_bio_remap(clone, bio_dev(io->orig_bio), sector); - ret = submit_bio_noacct(clone); + if (clone->bi_opf & REQ_HIPRI && + test_bit(QUEUE_FLAG_POLL, &clone->bi_bdev->bd_disk->queue->queue_flags)) { + ci->poll_queue = clone->bi_bdev->bd_disk->queue; + ci->poll_cookie = submit_bio_noacct_mq_direct(clone); + } else { + ci->poll_cookie = BLK_QC_T_NONE; + submit_bio_noacct(clone); + } break; case DM_MAPIO_KILL: if (unlikely(swap_bios_limit(ti, clone))) { @@ -1350,8 +1358,6 @@ static blk_qc_t __map_bio(struct dm_targ DMWARN("unimplemented target map return value: %d", r); BUG(); } - - return ret; } static void bio_setup_sector(struct bio *bio, sector_t sector, unsigned len) @@ -1438,8 +1444,8 @@ static void alloc_multiple_bios(struct b } } -static blk_qc_t __clone_and_map_simple_bio(struct clone_info *ci, - struct dm_target_io *tio, unsigned *len) +static void __clone_and_map_simple_bio(struct clone_info *ci, + struct dm_target_io *tio, unsigned *len) { struct bio *clone = &tio->clone; @@ -1449,7 +1455,7 @@ static blk_qc_t __clone_and_map_simple_b if (len) bio_setup_sector(clone, ci->sector, *len); - return __map_bio(tio); + __map_bio(ci, tio); } static void __send_duplicate_bios(struct clone_info *ci, struct dm_target *ti, @@ -1463,7 +1469,7 @@ static void __send_duplicate_bios(struct while ((bio = bio_list_pop(&blist))) { tio = container_of(bio, struct dm_target_io, clone); - (void) __clone_and_map_simple_bio(ci, tio, len); + __clone_and_map_simple_bio(ci, tio, len); } } @@ -1507,7 +1513,7 @@ static int __clone_and_map_data_bio(stru free_tio(tio); return r; } - (void) __map_bio(tio); + __map_bio(ci, tio); return 0; } @@ -1614,6 +1620,7 @@ static void init_clone_info(struct clone ci->map = map; ci->io = alloc_io(md, bio); ci->sector = bio->bi_iter.bi_sector; + ci->poll_cookie = BLK_QC_T_NONE; } #define __dm_part_stat_sub(part, field, subnd) \ @@ -1622,11 +1629,10 @@ static void init_clone_info(struct clone /* * Entry point to split a bio into clones and submit them to the targets. */ -static blk_qc_t __split_and_process_bio(struct mapped_device *md, - struct dm_table *map, struct bio *bio) +static void __split_and_process_bio(struct mapped_device *md, + struct dm_table *map, struct bio *bio) { struct clone_info ci; - blk_qc_t ret = BLK_QC_T_NONE; int error = 0; init_clone_info(&ci, md, map, bio); @@ -1643,7 +1649,7 @@ static blk_qc_t __split_and_process_bio( ci.sector_count = bio_sectors(bio); while (ci.sector_count && !error) { error = __split_and_process_non_flush(&ci); - if (ci.sector_count && !error) { + if (ci.sector_count && !error && ci.poll_cookie == BLK_QC_T_NONE) { /* * Remainder must be passed to submit_bio_noacct() * so that it gets handled *after* bios already submitted @@ -1670,7 +1676,7 @@ static blk_qc_t __split_and_process_bio( bio_chain(b, bio); trace_block_split(b, bio->bi_iter.bi_sector); - ret = submit_bio_noacct(bio); + submit_bio_noacct(bio); break; } } @@ -1678,13 +1684,11 @@ static blk_qc_t __split_and_process_bio( /* drop the extra reference count */ dec_pending(ci.io, errno_to_blk_status(error)); - return ret; } static blk_qc_t dm_submit_bio(struct bio *bio) { struct mapped_device *md = bio->bi_bdev->bd_disk->private_data; - blk_qc_t ret = BLK_QC_T_NONE; int srcu_idx; struct dm_table *map; @@ -1714,10 +1718,10 @@ static blk_qc_t dm_submit_bio(struct bio if (is_abnormal_io(bio)) blk_queue_split(&bio); - ret = __split_and_process_bio(md, map, bio); + __split_and_process_bio(md, map, bio); out: dm_put_live_table(md, srcu_idx); - return ret; + return BLK_QC_T_NONE; } /*----------------------------------------------------------------- From patchwork Tue Mar 2 19:05:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mikulas Patocka X-Patchwork-Id: 12112387 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BED87C433E9 for ; Tue, 2 Mar 2021 23:52:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9C0AC64F55 for ; Tue, 2 Mar 2021 23:52:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231829AbhCBXrN (ORCPT ); Tue, 2 Mar 2021 18:47:13 -0500 Received: from smtp01.tmcz.cz ([93.153.104.112]:33301 "EHLO smtp01.tmcz.cz" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1581886AbhCBT2m (ORCPT ); Tue, 2 Mar 2021 14:28:42 -0500 Received: from leontynka.twibright.com (109-183-129-149.customers.tmcz.cz [109.183.129.149]) by smtp01.tmcz.cz (Postfix) with ESMTPS id ABE784059E; Tue, 2 Mar 2021 20:05:57 +0100 (CET) Received: from debian-a64.vm ([192.168.208.2]) by leontynka.twibright.com with smtp (Exim 4.92) (envelope-from ) id 1lHALE-0003kL-Fd; Tue, 02 Mar 2021 20:05:57 +0100 Received: by debian-a64.vm (sSMTP sendmail emulation); Tue, 02 Mar 2021 20:05:55 +0100 Message-Id: <20210302190555.201228400@debian-a64.vm> User-Agent: quilt/0.65 Date: Tue, 02 Mar 2021 20:05:17 +0100 From: Mikulas Patocka To: JeffleXu , Mike Snitzer , Heinz Mauelshagen , axboe@kernel.dk, caspar@linux.alibaba.com, io-uring@vger.kernel.org, linux-block@vger.kernel.org, joseph.qi@linux.alibaba.com, dm-devel@redhat.com, hch@lst.de Cc: Mikulas Patocka Subject: [PATCH 4/4] dm: support I/O polling MIME-Version: 1.0 Content-Disposition: inline; filename=dm-poll.patch Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Support I/O polling if submit_bio_noacct_mq_direct returned non-empty cookie. Signed-off-by: Mikulas Patocka --- drivers/md/dm.c | 5 +++++ 1 file changed, 5 insertions(+) Index: linux-2.6/drivers/md/dm.c =================================================================== --- linux-2.6.orig/drivers/md/dm.c 2021-03-02 19:26:34.000000000 +0100 +++ linux-2.6/drivers/md/dm.c 2021-03-02 19:26:34.000000000 +0100 @@ -1682,6 +1682,11 @@ static void __split_and_process_bio(stru } } + if (ci.poll_cookie != BLK_QC_T_NONE) { + while (atomic_read(&ci.io->io_count) > 1 && + blk_poll(ci.poll_queue, ci.poll_cookie, true)) ; + } + /* drop the extra reference count */ dec_pending(ci.io, errno_to_blk_status(error)); }