From patchwork Wed Jun 16 13:05:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12325147 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36C57C49361 for ; Wed, 16 Jun 2021 13:07:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1B2AE6135C for ; Wed, 16 Jun 2021 13:07:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233225AbhFPNJG (ORCPT ); Wed, 16 Jun 2021 09:09:06 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:28679 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233161AbhFPNJE (ORCPT ); Wed, 16 Jun 2021 09:09:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1623848817; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=f0TWFgscOaXm0gyNlFEAqt3NFjx/gooC5Rhr0DBU4js=; b=Bkukzh+4/mNoEuFywQuXUCXf/U316FY4qQI5YkvUWSAavLwFIguPfP3QnrfBwEDm3Nww3Z /2o2zkp147uO1hXsam73a9d6F0WGaaY4ON5XEZBwgiaE79qkj8xfvysI/2+EbuupnxZXHu rr4aLUlIlQ4sf6AWQrhD1a+3NVtWgik= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-25-oTzAfjQKOsSH-x1DgYsCCA-1; Wed, 16 Jun 2021 09:06:56 -0400 X-MC-Unique: oTzAfjQKOsSH-x1DgYsCCA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 22E86100B3AB; Wed, 16 Jun 2021 13:06:55 +0000 (UTC) Received: from localhost (ovpn-12-48.pek2.redhat.com [10.72.12.48]) by smtp.corp.redhat.com (Postfix) with ESMTP id E5C835C26D; Wed, 16 Jun 2021 13:06:10 +0000 (UTC) From: Ming Lei To: Jens Axboe , Mike Snitzer Cc: linux-block@vger.kernel.org, Jeffle Xu , dm-devel@redhat.com, Hannes Reinecke , Christoph Hellwig , Ming Lei , Chaitanya Kulkarni Subject: [RFC PATCH 1/4] block: add helper of blk_queue_poll Date: Wed, 16 Jun 2021 21:05:30 +0800 Message-Id: <20210616130533.754248-2-ming.lei@redhat.com> In-Reply-To: <20210616130533.754248-1-ming.lei@redhat.com> References: <20210616130533.754248-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org There has been 3 users, and will be more, so add one such helper. Reviewed-by: Chaitanya Kulkarni Reviewed-by: Jeffle Xu Reviewed-by: Hannes Reinecke Signed-off-by: Ming Lei --- block/blk-core.c | 5 ++--- block/blk-sysfs.c | 4 ++-- drivers/nvme/host/core.c | 2 +- include/linux/blkdev.h | 1 + 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 531176578221..1e24c71c6738 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -835,7 +835,7 @@ static noinline_for_stack bool submit_bio_checks(struct bio *bio) } } - if (!test_bit(QUEUE_FLAG_POLL, &q->queue_flags)) + if (!blk_queue_poll(q)) bio->bi_opf &= ~REQ_POLLED; switch (bio_op(bio)) { @@ -1117,8 +1117,7 @@ int bio_poll(struct bio *bio, unsigned int flags) blk_qc_t cookie = READ_ONCE(bio->bi_cookie); int ret; - if (cookie == BLK_QC_T_NONE || - !test_bit(QUEUE_FLAG_POLL, &q->queue_flags)) + if (cookie == BLK_QC_T_NONE || !blk_queue_poll(q)) return 0; if (current->plug) diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c index f78e73ca6091..93dcf2dfaafd 100644 --- a/block/blk-sysfs.c +++ b/block/blk-sysfs.c @@ -422,13 +422,13 @@ static ssize_t queue_poll_delay_store(struct request_queue *q, const char *page, static ssize_t queue_poll_show(struct request_queue *q, char *page) { - return queue_var_show(test_bit(QUEUE_FLAG_POLL, &q->queue_flags), page); + return queue_var_show(blk_queue_poll(q), page); } static ssize_t queue_poll_store(struct request_queue *q, const char *page, size_t count) { - if (!test_bit(QUEUE_FLAG_POLL, &q->queue_flags)) + if (!blk_queue_poll(q)) return -EINVAL; pr_info_ratelimited("writes to the poll attribute are ignored.\n"); pr_info_ratelimited("please use driver specific parameters instead.\n"); diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index fe0b8da3de7f..e31c7704ef4d 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -1025,7 +1025,7 @@ static void nvme_execute_rq_polled(struct request_queue *q, { DECLARE_COMPLETION_ONSTACK(wait); - WARN_ON_ONCE(!test_bit(QUEUE_FLAG_POLL, &q->queue_flags)); + WARN_ON_ONCE(!blk_queue_poll(q)); rq->cmd_flags |= REQ_POLLED; rq->end_io_data = &wait; diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 561b04117bd4..fc0ba0b80776 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -677,6 +677,7 @@ bool blk_queue_flag_test_and_set(unsigned int flag, struct request_queue *q); #define blk_queue_fua(q) test_bit(QUEUE_FLAG_FUA, &(q)->queue_flags) #define blk_queue_registered(q) test_bit(QUEUE_FLAG_REGISTERED, &(q)->queue_flags) #define blk_queue_nowait(q) test_bit(QUEUE_FLAG_NOWAIT, &(q)->queue_flags) +#define blk_queue_poll(q) test_bit(QUEUE_FLAG_POLL, &(q)->queue_flags) extern void blk_set_pm_only(struct request_queue *q); extern void blk_clear_pm_only(struct request_queue *q); From patchwork Wed Jun 16 13:05:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12325149 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26A90C48BE6 for ; Wed, 16 Jun 2021 13:07:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 10ADE6024A for ; Wed, 16 Jun 2021 13:07:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233206AbhFPNJJ (ORCPT ); Wed, 16 Jun 2021 09:09:09 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:35414 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233228AbhFPNJH (ORCPT ); Wed, 16 Jun 2021 09:09:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1623848820; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LxN8jIHyhdhyntZFjGmGbaw2e17fq9m2ICTyCfnEP5g=; b=e96S/exCucnlR028EjCVes3q4j5aEQjCWcLQlw0YooY1c0b+5KvIqfoIEtxvoV+ps/36mR 7w5yGhfxbS714qiYltlYleCqikhv1iiYnhW51K20sWG/8zuQbYrqFotJtuCDb1OdqcPt04 s1k+J0eUHLxF0hLX/XWcrZXLFcA7y4I= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-86-PxR36WMWO06s-oeKDJg6Tw-1; Wed, 16 Jun 2021 09:06:59 -0400 X-MC-Unique: PxR36WMWO06s-oeKDJg6Tw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 418251926DAB; Wed, 16 Jun 2021 13:06:58 +0000 (UTC) Received: from localhost (ovpn-12-48.pek2.redhat.com [10.72.12.48]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7159160C13; Wed, 16 Jun 2021 13:06:57 +0000 (UTC) From: Ming Lei To: Jens Axboe , Mike Snitzer Cc: linux-block@vger.kernel.org, Jeffle Xu , dm-devel@redhat.com, Hannes Reinecke , Christoph Hellwig , Ming Lei Subject: [RFC PATCH 2/4] block: add field of .bi_bio_drv_data to bio Date: Wed, 16 Jun 2021 21:05:31 +0800 Message-Id: <20210616130533.754248-3-ming.lei@redhat.com> In-Reply-To: <20210616130533.754248-1-ming.lei@redhat.com> References: <20210616130533.754248-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org After bio is submitted, bio->bi_next is used for IO merge for request based queue only. Reuse the filed for bio based driver for storing driver data, and name it as .bi_bio_drv_data. Signed-off-by: Ming Lei --- include/linux/blk_types.h | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 6e6c2af48d74..acb45213338c 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -220,7 +220,16 @@ typedef unsigned int blk_qc_t; * stacking drivers) */ struct bio { - struct bio *bi_next; /* request queue link */ + union { + /* request queue link */ + struct bio *bi_next; + + /* + * once bio is submitted to bio based queue, driver can use + * this field to store its own data + */ + void *bi_bio_drv_data; + }; struct block_device *bi_bdev; unsigned int bi_opf; /* bottom bits req flags, * top bits REQ_OP. Use From patchwork Wed Jun 16 13:05:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12325151 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 683E1C48BE6 for ; Wed, 16 Jun 2021 13:07:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 53A9B61359 for ; Wed, 16 Jun 2021 13:07:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233240AbhFPNJO (ORCPT ); Wed, 16 Jun 2021 09:09:14 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:33622 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233218AbhFPNJL (ORCPT ); Wed, 16 Jun 2021 09:09:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1623848824; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YIkgsb2GruWXtknprN6c4/O3dDtMstZMPs68M+U4+/o=; b=MsKD02dZSVfy2HaxY74qQSm7DYE2njyQKnzJ0gZ97Dfkye5XfTufBWaWBvn9ZQjzbB7bGF sCmfWQYIJ71nBFdeOfSDrOpbwB4yObPjHADHooiccmv7dKovAy14ggyhO8hC61QSvy5hQR 2WhyMOl0SHYUmond0mtqdCSst9sYivE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-253-SslDtkamOtGAtFJjL0R2lw-1; Wed, 16 Jun 2021 09:07:02 -0400 X-MC-Unique: SslDtkamOtGAtFJjL0R2lw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id F1EDCA40DB; Wed, 16 Jun 2021 13:07:00 +0000 (UTC) Received: from localhost (ovpn-12-48.pek2.redhat.com [10.72.12.48]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4723860C13; Wed, 16 Jun 2021 13:06:59 +0000 (UTC) From: Ming Lei To: Jens Axboe , Mike Snitzer Cc: linux-block@vger.kernel.org, Jeffle Xu , dm-devel@redhat.com, Hannes Reinecke , Christoph Hellwig , Ming Lei Subject: [RFC PATCH 3/4] block: add ->poll_bio to block_device_operations Date: Wed, 16 Jun 2021 21:05:32 +0800 Message-Id: <20210616130533.754248-4-ming.lei@redhat.com> In-Reply-To: <20210616130533.754248-1-ming.lei@redhat.com> References: <20210616130533.754248-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Prepare for supporting IO polling for bio based driver. Add ->poll_bio callback so that bio driver can provide their own logic for polling bio. Signed-off-by: Ming Lei --- block/blk-core.c | 18 +++++++++++++----- block/genhd.c | 3 +++ include/linux/blkdev.h | 2 ++ 3 files changed, 18 insertions(+), 5 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 1e24c71c6738..a1552ec8d608 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1113,11 +1113,13 @@ EXPORT_SYMBOL(submit_bio); */ int bio_poll(struct bio *bio, unsigned int flags) { - struct request_queue *q = bio->bi_bdev->bd_disk->queue; + struct gendisk *disk = bio->bi_bdev->bd_disk; + struct request_queue *q = disk->queue; blk_qc_t cookie = READ_ONCE(bio->bi_cookie); int ret; - if (cookie == BLK_QC_T_NONE || !blk_queue_poll(q)) + if ((queue_is_mq(q) && cookie == BLK_QC_T_NONE) || + !blk_queue_poll(q)) return 0; if (current->plug) @@ -1125,10 +1127,16 @@ int bio_poll(struct bio *bio, unsigned int flags) if (blk_queue_enter(q, BLK_MQ_REQ_NOWAIT)) return 0; - if (WARN_ON_ONCE(!queue_is_mq(q))) - ret = 0; /* not yet implemented, should not happen */ - else + if (!queue_is_mq(q)) { + if (disk->fops->poll_bio) { + ret = disk->fops->poll_bio(bio, flags); + } else { + WARN_ON_ONCE(1); + ret = 0; + } + } else { ret = blk_mq_poll(q, cookie, flags); + } blk_queue_exit(q); return ret; } diff --git a/block/genhd.c b/block/genhd.c index 5f5628216295..042dfa5b3f79 100644 --- a/block/genhd.c +++ b/block/genhd.c @@ -471,6 +471,9 @@ static void __device_add_disk(struct device *parent, struct gendisk *disk, { int ret; + /* ->poll_bio is only for bio based driver */ + WARN_ON_ONCE(queue_is_mq(disk->queue) && disk->fops->poll_bio); + /* * The disk queue should now be all set with enough information about * the device for the elevator code to pick an adequate default diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index fc0ba0b80776..6da6fb120148 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1858,6 +1858,8 @@ static inline void blk_ksm_unregister(struct request_queue *q) { } struct block_device_operations { void (*submit_bio)(struct bio *bio); + /* ->poll_bio is for bio driver only */ + int (*poll_bio)(struct bio *bio, unsigned int flags); int (*open) (struct block_device *, fmode_t); void (*release) (struct gendisk *, fmode_t); int (*rw_page)(struct block_device *, sector_t, struct page *, unsigned int); From patchwork Wed Jun 16 13:05:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12325153 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADABEC48BE5 for ; Wed, 16 Jun 2021 13:07:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 99E3F61356 for ; Wed, 16 Jun 2021 13:07:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233263AbhFPNJW (ORCPT ); Wed, 16 Jun 2021 09:09:22 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:29459 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233216AbhFPNJO (ORCPT ); Wed, 16 Jun 2021 09:09:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1623848828; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BEhUb1+MSBqGbDrvS0aXPNhm/VOfq1IJXBnWhNckb9Y=; b=Svrybgb/+q5lqqUrl2O6LdZEDmZjB09UzU3C0IJSmJaAHvFGPKccWE+BRF5jzJFz8J/jWv y4KDqX/+l0Djh2MpgJ7BTiBAZSYRoAQjmlb8qC45hMq9m+go9V3dU1r/MLaSqcAzRnFyu9 U8IPmdVC+1WYyMBsIxJkvQsZj65Z/pU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-237-ZwfcfrUQM3iVN1CF2GXPCw-1; Wed, 16 Jun 2021 09:07:06 -0400 X-MC-Unique: ZwfcfrUQM3iVN1CF2GXPCw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id EE0A9100CAAF; Wed, 16 Jun 2021 13:07:03 +0000 (UTC) Received: from localhost (ovpn-12-48.pek2.redhat.com [10.72.12.48]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3E15D60C13; Wed, 16 Jun 2021 13:07:02 +0000 (UTC) From: Ming Lei To: Jens Axboe , Mike Snitzer Cc: linux-block@vger.kernel.org, Jeffle Xu , dm-devel@redhat.com, Hannes Reinecke , Christoph Hellwig , Ming Lei Subject: [RFC PATCH 4/4] dm: support bio polling Date: Wed, 16 Jun 2021 21:05:33 +0800 Message-Id: <20210616130533.754248-5-ming.lei@redhat.com> In-Reply-To: <20210616130533.754248-1-ming.lei@redhat.com> References: <20210616130533.754248-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Support bio(REQ_POLLED) polling in the following approach: 1) setup one list in instance of 'struct dm_io', adds every 'struct dm_target_io' instance cloned for current dm bio into this list; store the list in 1) into bio->bi_bio_drv_data 2) hold one refcnt on io->io_count after submitting this dm bio with REQ_POLLED 4) implement .poll_bio() callback, and iterate over the list in 1) and polled on each ->clone of 'dm_target_io' instance; call dec_pending() if all target ios are done in .poll_bio(). 4) enable QUEUE_FLAG_POLL if all underlying queues enable QUEUE_FLAG_POLL, which is based on Jeffle's previous patch. Signed-off-by: Ming Lei --- drivers/md/dm-table.c | 24 ++++++++++++++++++ drivers/md/dm.c | 59 ++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 79 insertions(+), 4 deletions(-) diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c index ee47a332b462..b14b379442d2 100644 --- a/drivers/md/dm-table.c +++ b/drivers/md/dm-table.c @@ -1491,6 +1491,12 @@ struct dm_target *dm_table_find_target(struct dm_table *t, sector_t sector) return &t->targets[(KEYS_PER_NODE * n) + k]; } +static int device_not_poll_capable(struct dm_target *ti, struct dm_dev *dev, + sector_t start, sector_t len, void *data) +{ + return !blk_queue_poll(bdev_get_queue(dev->bdev)); +} + /* * type->iterate_devices() should be called when the sanity check needs to * iterate and check all underlying data devices. iterate_devices() will @@ -1541,6 +1547,11 @@ static int count_device(struct dm_target *ti, struct dm_dev *dev, return 0; } +static int dm_table_supports_poll(struct dm_table *t) +{ + return !dm_table_any_dev_attr(t, device_not_poll_capable, NULL); +} + /* * Check whether a table has no data devices attached using each * target's iterate_devices method. @@ -2078,6 +2089,19 @@ void dm_table_set_restrictions(struct dm_table *t, struct request_queue *q, dm_update_keyslot_manager(q, t); blk_queue_update_readahead(q); + + /* + * Check for request-based device is remained to + * dm_mq_init_request_queue()->blk_mq_init_allocated_queue(). + * For bio-based device, only set QUEUE_FLAG_POLL when all underlying + * devices supporting polling. + */ + if (__table_type_bio_based(t->type)) { + if (dm_table_supports_poll(t)) + blk_queue_flag_set(QUEUE_FLAG_POLL, q); + else + blk_queue_flag_clear(QUEUE_FLAG_POLL, q); + } } unsigned int dm_table_get_num_targets(struct dm_table *t) diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 363f12a285ce..0a0e4a38f435 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -84,6 +84,7 @@ struct dm_target_io { struct dm_target *ti; unsigned target_bio_nr; unsigned *len_ptr; + struct list_head list; bool inside_dm_io; struct bio clone; }; @@ -99,6 +100,7 @@ struct dm_io { blk_status_t status; atomic_t io_count; struct bio *orig_bio; + struct list_head poll_head; unsigned long start_time; spinlock_t endio_lock; struct dm_stats_aux stats_aux; @@ -655,6 +657,11 @@ static struct dm_io *alloc_io(struct mapped_device *md, struct bio *bio) io->md = md; spin_lock_init(&io->endio_lock); + if (bio->bi_opf & REQ_POLLED) { + bio->bi_bio_drv_data = io; + INIT_LIST_HEAD(&io->poll_head); + } + start_io_acct(io); return io; @@ -692,6 +699,8 @@ static struct dm_target_io *alloc_tio(struct clone_info *ci, struct dm_target *t static void free_tio(struct dm_target_io *tio) { + list_del_init(&tio->list); + if (tio->inside_dm_io) return; bio_put(&tio->clone); @@ -936,10 +945,15 @@ static void dec_pending(struct dm_io *io, blk_status_t error) io_error = io->status; bio = io->orig_bio; end_io_acct(io); + free_io(md, io); - if (io_error == BLK_STS_DM_REQUEUE) + if (io_error == BLK_STS_DM_REQUEUE) { + /* not poll any more in case of requeue */ + if (bio->bi_opf & REQ_POLLED) + bio->bi_opf &= ~REQ_POLLED; return; + } if ((bio->bi_opf & REQ_PREFLUSH) && bio->bi_iter.bi_size) { /* @@ -1043,7 +1057,9 @@ static void clone_endio(struct bio *bio) up(&md->swap_bios_semaphore); } - free_tio(tio); + /* Any cloned bio submitted as POLLED, free them all after dm_io is done */ + if (list_empty(&tio->list)) + free_tio(tio); dec_pending(io, error); } @@ -1300,6 +1316,11 @@ static void __map_bio(struct dm_target_io *tio) struct dm_io *io = tio->io; struct dm_target *ti = tio->ti; + if (clone->bi_opf & REQ_POLLED) + list_add_tail(&tio->list, &io->poll_head); + else + INIT_LIST_HEAD(&tio->list); + clone->bi_end_io = clone_endio; /* @@ -1666,8 +1687,9 @@ static void __split_and_process_bio(struct mapped_device *md, } } - /* drop the extra reference count */ - dec_pending(ci.io, errno_to_blk_status(error)); + /* drop the extra reference count for non-POLLED bio */ + if (!(bio->bi_opf & REQ_POLLED)) + dec_pending(ci.io, errno_to_blk_status(error)); } static void dm_submit_bio(struct bio *bio) @@ -1707,6 +1729,34 @@ static void dm_submit_bio(struct bio *bio) dm_put_live_table(md, srcu_idx); } +static int dm_poll_bio(struct bio *bio, unsigned int flags) +{ + struct dm_io *io = bio->bi_bio_drv_data; + struct dm_target_io *tio; + + if (!(bio->bi_opf & REQ_POLLED) || !io) + return 0; + + list_for_each_entry(tio, &io->poll_head, list) + bio_poll(&tio->clone, flags); + + /* bio_poll holds the last reference */ + if (atomic_read(&io->io_count) == 1) { + /* free all target IOs submitted as POLLED */ + while (!list_empty(&io->poll_head)) { + struct dm_target_io *tio = + list_entry(io->poll_head.next, + struct dm_target_io, list); + free_tio(tio); + } + bio->bi_bio_drv_data = NULL; + dec_pending(io, 0); + return 1; + } + + return 0; +} + /*----------------------------------------------------------------- * An IDR is used to keep track of allocated minor numbers. *---------------------------------------------------------------*/ @@ -3121,6 +3171,7 @@ static const struct pr_ops dm_pr_ops = { static const struct block_device_operations dm_blk_dops = { .submit_bio = dm_submit_bio, + .poll_bio = dm_poll_bio, .open = dm_blk_open, .release = dm_blk_close, .ioctl = dm_blk_ioctl,