From patchwork Sat Feb 20 11:06:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12098113 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C286BC433E0 for ; Mon, 22 Feb 2021 08:52:42 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2C59264DD3 for ; Mon, 22 Feb 2021 08:52:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2C59264DD3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-421-FRAxTZy_OwG4jFRTFYoMKw-1; Mon, 22 Feb 2021 03:52:39 -0500 X-MC-Unique: FRAxTZy_OwG4jFRTFYoMKw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 4FA3F107ACF5; Mon, 22 Feb 2021 08:52:34 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2C6B260C64; Mon, 22 Feb 2021 08:52:34 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id EFB1018095CC; Mon, 22 Feb 2021 08:52:33 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 11KB6pxL010436 for ; Sat, 20 Feb 2021 06:06:52 -0500 Received: by smtp.corp.redhat.com (Postfix) id 066DC13E781; Sat, 20 Feb 2021 11:06:51 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast04.extmail.prod.ext.rdu2.redhat.com [10.11.55.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 00D64121125 for ; Sat, 20 Feb 2021 11:06:48 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [205.139.110.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6F2BF1022F09 for ; Sat, 20 Feb 2021 11:06:48 +0000 (UTC) Received: from out30-131.freemail.mail.aliyun.com (out30-131.freemail.mail.aliyun.com [115.124.30.131]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-176-82L8BP0PNkOeJn4PnNL9Rg-1; Sat, 20 Feb 2021 06:06:45 -0500 X-MC-Unique: 82L8BP0PNkOeJn4PnNL9Rg-1 X-Alimail-AntiSpam: AC=PASS; BC=-1|-1; BR=01201311R521e4; CH=green; DM=||false|; DS=||; FP=0|-1|-1|-1|0|-1|-1|-1; HT=e01e01424; MF=jefflexu@linux.alibaba.com; NM=1; PH=DS; RN=9; SR=0; TI=SMTPD_---0UP0sPTZ_1613819198 Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0UP0sPTZ_1613819198) by smtp.aliyun-inc.com(127.0.0.1); Sat, 20 Feb 2021 19:06:38 +0800 From: Jeffle Xu To: snitzer@redhat.com, axboe@kernel.dk Date: Sat, 20 Feb 2021 19:06:26 +0800 Message-Id: <20210220110637.50305-2-jefflexu@linux.alibaba.com> In-Reply-To: <20210220110637.50305-1-jefflexu@linux.alibaba.com> References: <20210220110637.50305-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-loop: dm-devel@redhat.com X-Mailman-Approved-At: Mon, 22 Feb 2021 03:49:50 -0500 Cc: caspar@linux.alibaba.com, io-uring@vger.kernel.org, linux-block@vger.kernel.org, joseph.qi@linux.alibaba.com, dm-devel@redhat.com, ming.lei@redhat.com, hch@lst.de Subject: [dm-devel] [PATCH v4 01/12] block: move definition of blk_qc_t to types.h X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com So that kiocb.ki_cookie can be defined as blk_qc_t, which will enforce the encapsulation. Signed-off-by: Jeffle Xu Reviewed-by: Christoph Hellwig Reviewed-by: Mike Snitzer --- include/linux/blk_types.h | 2 +- include/linux/fs.h | 2 +- include/linux/types.h | 3 +++ 3 files changed, 5 insertions(+), 2 deletions(-) diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 866f74261b3b..2e05244fc16d 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -532,7 +532,7 @@ static inline int op_stat_group(unsigned int op) return op_is_write(op); } -typedef unsigned int blk_qc_t; +/* Macros for blk_qc_t */ #define BLK_QC_T_NONE -1U #define BLK_QC_T_SHIFT 16 #define BLK_QC_T_INTERNAL (1U << 31) diff --git a/include/linux/fs.h b/include/linux/fs.h index fd47deea7c17..04b687150736 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -330,7 +330,7 @@ struct kiocb { u16 ki_hint; u16 ki_ioprio; /* See linux/ioprio.h */ union { - unsigned int ki_cookie; /* for ->iopoll */ + blk_qc_t ki_cookie; /* for ->iopoll */ struct wait_page_queue *ki_waitq; /* for async buffered IO */ }; diff --git a/include/linux/types.h b/include/linux/types.h index a147977602b5..da5ca7e1bea9 100644 --- a/include/linux/types.h +++ b/include/linux/types.h @@ -125,6 +125,9 @@ typedef s64 int64_t; typedef u64 sector_t; typedef u64 blkcnt_t; +/* cookie used for IO polling */ +typedef unsigned int blk_qc_t; + /* * The type of an index into the pagecache. */ From patchwork Sat Feb 20 11:06:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12098101 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D6ADC433E0 for ; Mon, 22 Feb 2021 08:51:19 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1C77164E61 for ; Mon, 22 Feb 2021 08:51:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1C77164E61 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-332-D2W7v8M2MxerJFOsS0TOVw-1; Mon, 22 Feb 2021 03:51:14 -0500 X-MC-Unique: D2W7v8M2MxerJFOsS0TOVw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2B97D18449E0; Mon, 22 Feb 2021 08:51:10 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C7A7960C0F; Mon, 22 Feb 2021 08:51:08 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 3A89618095CC; Mon, 22 Feb 2021 08:51:02 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 11KB6pJ5010438 for ; Sat, 20 Feb 2021 06:06:51 -0500 Received: by smtp.corp.redhat.com (Postfix) id 896B1200BD73; Sat, 20 Feb 2021 11:06:51 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast01.extmail.prod.ext.rdu2.redhat.com [10.11.55.17]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 82D7F2126C9E for ; Sat, 20 Feb 2021 11:06:50 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E6EFE85A5A6 for ; Sat, 20 Feb 2021 11:06:49 +0000 (UTC) Received: from out30-57.freemail.mail.aliyun.com (out30-57.freemail.mail.aliyun.com [115.124.30.57]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-271-seylu2FVNdin9focofr30g-1; Sat, 20 Feb 2021 06:06:45 -0500 X-MC-Unique: seylu2FVNdin9focofr30g-1 X-Alimail-AntiSpam: AC=PASS; BC=-1|-1; BR=01201311R161e4; CH=green; DM=||false|; DS=||; FP=0|-1|-1|-1|0|-1|-1|-1; HT=e01e04423; MF=jefflexu@linux.alibaba.com; NM=1; PH=DS; RN=9; SR=0; TI=SMTPD_---0UP1O7oY_1613819199 Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0UP1O7oY_1613819199) by smtp.aliyun-inc.com(127.0.0.1); Sat, 20 Feb 2021 19:06:40 +0800 From: Jeffle Xu To: snitzer@redhat.com, axboe@kernel.dk Date: Sat, 20 Feb 2021 19:06:27 +0800 Message-Id: <20210220110637.50305-3-jefflexu@linux.alibaba.com> In-Reply-To: <20210220110637.50305-1-jefflexu@linux.alibaba.com> References: <20210220110637.50305-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-loop: dm-devel@redhat.com X-Mailman-Approved-At: Mon, 22 Feb 2021 03:49:50 -0500 Cc: caspar@linux.alibaba.com, io-uring@vger.kernel.org, linux-block@vger.kernel.org, joseph.qi@linux.alibaba.com, dm-devel@redhat.com, ming.lei@redhat.com, hch@lst.de Subject: [dm-devel] [PATCH v4 02/12] block: add queue_to_disk() to get gendisk from request_queue X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Sometimes we need to get the corresponding gendisk from request_queue. It is preferred that block drivers store private data in gendisk->private_data rather than request_queue->queuedata, e.g. see: commit c4a59c4e5db3 ("dm: stop using ->queuedata"). So if only request_queue is given, we need to get its corresponding gendisk to get the private data stored in that gendisk. Signed-off-by: Jeffle Xu Reviewed-by: Mike Snitzer --- include/linux/blkdev.h | 2 ++ include/trace/events/kyber.h | 6 +++--- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index f94ee3089e01..2a802e011a95 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -687,6 +687,8 @@ static inline bool blk_account_rq(struct request *rq) dma_map_page_attrs(dev, (bv)->bv_page, (bv)->bv_offset, (bv)->bv_len, \ (dir), (attrs)) +#define queue_to_disk(q) (dev_to_disk(kobj_to_dev((q)->kobj.parent))) + static inline bool queue_is_mq(struct request_queue *q) { return q->mq_ops; diff --git a/include/trace/events/kyber.h b/include/trace/events/kyber.h index c0e7d24ca256..f9802562edf6 100644 --- a/include/trace/events/kyber.h +++ b/include/trace/events/kyber.h @@ -30,7 +30,7 @@ TRACE_EVENT(kyber_latency, ), TP_fast_assign( - __entry->dev = disk_devt(dev_to_disk(kobj_to_dev(q->kobj.parent))); + __entry->dev = disk_devt(queue_to_disk(q)); strlcpy(__entry->domain, domain, sizeof(__entry->domain)); strlcpy(__entry->type, type, sizeof(__entry->type)); __entry->percentile = percentile; @@ -59,7 +59,7 @@ TRACE_EVENT(kyber_adjust, ), TP_fast_assign( - __entry->dev = disk_devt(dev_to_disk(kobj_to_dev(q->kobj.parent))); + __entry->dev = disk_devt(queue_to_disk(q)); strlcpy(__entry->domain, domain, sizeof(__entry->domain)); __entry->depth = depth; ), @@ -81,7 +81,7 @@ TRACE_EVENT(kyber_throttled, ), TP_fast_assign( - __entry->dev = disk_devt(dev_to_disk(kobj_to_dev(q->kobj.parent))); + __entry->dev = disk_devt(queue_to_disk(q)); strlcpy(__entry->domain, domain, sizeof(__entry->domain)); ), From patchwork Sat Feb 20 11:06:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12098105 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3899C433DB for ; Mon, 22 Feb 2021 08:51:42 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3069E64E61 for ; Mon, 22 Feb 2021 08:51:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3069E64E61 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-270-Nw1H7iqvOJCwz-8eC5pwYA-1; Mon, 22 Feb 2021 03:51:39 -0500 X-MC-Unique: Nw1H7iqvOJCwz-8eC5pwYA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 91907107ACE4; Mon, 22 Feb 2021 08:51:34 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 04B5A5D9DB; Mon, 22 Feb 2021 08:51:34 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id AB38018095CD; Mon, 22 Feb 2021 08:51:32 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 11KB6ppT010437 for ; Sat, 20 Feb 2021 06:06:51 -0500 Received: by smtp.corp.redhat.com (Postfix) id 485BB2026D13; Sat, 20 Feb 2021 11:06:51 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast04.extmail.prod.ext.rdu2.redhat.com [10.11.55.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 42D172026D46 for ; Sat, 20 Feb 2021 11:06:49 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-1.mimecast.com [207.211.31.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EFDDD1022F0A for ; Sat, 20 Feb 2021 11:06:48 +0000 (UTC) Received: from out30-133.freemail.mail.aliyun.com (out30-133.freemail.mail.aliyun.com [115.124.30.133]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-477-BFDqRq-cP0mka_R3ABZOxA-1; Sat, 20 Feb 2021 06:06:45 -0500 X-MC-Unique: BFDqRq-cP0mka_R3ABZOxA-1 X-Alimail-AntiSpam: AC=PASS; BC=-1|-1; BR=01201311R831e4; CH=green; DM=||false|; DS=||; FP=0|-1|-1|-1|0|-1|-1|-1; HT=e01e01424; MF=jefflexu@linux.alibaba.com; NM=1; PH=DS; RN=9; SR=0; TI=SMTPD_---0UP1XFxd_1613819200 Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0UP1XFxd_1613819200) by smtp.aliyun-inc.com(127.0.0.1); Sat, 20 Feb 2021 19:06:41 +0800 From: Jeffle Xu To: snitzer@redhat.com, axboe@kernel.dk Date: Sat, 20 Feb 2021 19:06:28 +0800 Message-Id: <20210220110637.50305-4-jefflexu@linux.alibaba.com> In-Reply-To: <20210220110637.50305-1-jefflexu@linux.alibaba.com> References: <20210220110637.50305-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Mimecast-Bulk-Signature: yes X-Mimecast-Spam-Signature: bulk X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-loop: dm-devel@redhat.com X-Mailman-Approved-At: Mon, 22 Feb 2021 03:49:50 -0500 Cc: caspar@linux.alibaba.com, io-uring@vger.kernel.org, linux-block@vger.kernel.org, joseph.qi@linux.alibaba.com, dm-devel@redhat.com, ming.lei@redhat.com, hch@lst.de Subject: [dm-devel] [PATCH v4 03/12] block: add poll method to support bio-based IO polling X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com ->poll_fn was introduced in commit ea435e1b9392 ("block: add a poll_fn callback to struct request_queue") to support bio-based queues such as nvme multipath, but was later removed in commit 529262d56dbe ("block: remove ->poll_fn"). Given commit c62b37d96b6e ("block: move ->make_request_fn to struct block_device_operations") restore the possibility of bio-based IO polling support by adding an ->poll method to gendisk->fops. Make blk_mq_poll() specific to blk-mq, while blk_bio_poll() is originally a copy from blk_mq_poll(), and is specific to bio-based polling. Currently hybrid polling is not supported by bio-based polling. Signed-off-by: Jeffle Xu --- block/blk-core.c | 58 ++++++++++++++++++++++++++++++++++++++++++ block/blk-mq.c | 22 +--------------- include/linux/blk-mq.h | 1 + include/linux/blkdev.h | 1 + 4 files changed, 61 insertions(+), 21 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 7663a9b94b80..37aa513da5f2 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -1132,6 +1132,64 @@ blk_qc_t submit_bio(struct bio *bio) } EXPORT_SYMBOL(submit_bio); + +static int blk_bio_poll(struct request_queue *q, blk_qc_t cookie, bool spin) +{ + long state; + struct gendisk *disk = queue_to_disk(q); + + state = current->state; + do { + int ret; + + ret = disk->fops->poll(q, cookie); + if (ret > 0) { + __set_current_state(TASK_RUNNING); + return ret; + } + + if (signal_pending_state(state, current)) + __set_current_state(TASK_RUNNING); + + if (current->state == TASK_RUNNING) + return 1; + if (ret < 0 || !spin) + break; + cpu_relax(); + } while (!need_resched()); + + __set_current_state(TASK_RUNNING); + return 0; +} + +/** + * blk_poll - poll for IO completions + * @q: the queue + * @cookie: cookie passed back at IO submission time + * @spin: whether to spin for completions + * + * Description: + * Poll for completions on the passed in queue. Returns number of + * completed entries found. If @spin is true, then blk_poll will continue + * looping until at least one completion is found, unless the task is + * otherwise marked running (or we need to reschedule). + */ +int blk_poll(struct request_queue *q, blk_qc_t cookie, bool spin) +{ + if (!blk_qc_t_valid(cookie) || + !test_bit(QUEUE_FLAG_POLL, &q->queue_flags)) + return 0; + + if (current->plug) + blk_flush_plug_list(current->plug, false); + + if (queue_is_mq(q)) + return blk_mq_poll(q, cookie, spin); + else + return blk_bio_poll(q, cookie, spin); +} +EXPORT_SYMBOL_GPL(blk_poll); + /** * blk_cloned_rq_check_limits - Helper function to check a cloned request * for the new queue limits diff --git a/block/blk-mq.c b/block/blk-mq.c index f285a9123a8b..c18bc23834a0 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -3826,30 +3826,11 @@ static bool blk_mq_poll_hybrid(struct request_queue *q, return blk_mq_poll_hybrid_sleep(q, rq); } -/** - * blk_poll - poll for IO completions - * @q: the queue - * @cookie: cookie passed back at IO submission time - * @spin: whether to spin for completions - * - * Description: - * Poll for completions on the passed in queue. Returns number of - * completed entries found. If @spin is true, then blk_poll will continue - * looping until at least one completion is found, unless the task is - * otherwise marked running (or we need to reschedule). - */ -int blk_poll(struct request_queue *q, blk_qc_t cookie, bool spin) +int blk_mq_poll(struct request_queue *q, blk_qc_t cookie, bool spin) { struct blk_mq_hw_ctx *hctx; long state; - if (!blk_qc_t_valid(cookie) || - !test_bit(QUEUE_FLAG_POLL, &q->queue_flags)) - return 0; - - if (current->plug) - blk_flush_plug_list(current->plug, false); - hctx = q->queue_hw_ctx[blk_qc_t_to_queue_num(cookie)]; /* @@ -3891,7 +3872,6 @@ int blk_poll(struct request_queue *q, blk_qc_t cookie, bool spin) __set_current_state(TASK_RUNNING); return 0; } -EXPORT_SYMBOL_GPL(blk_poll); unsigned int blk_mq_rq_cpu(struct request *rq) { diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index d705b174d346..adc789d5888e 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -607,6 +607,7 @@ static inline void blk_rq_bio_prep(struct request *rq, struct bio *bio, } blk_qc_t blk_mq_submit_bio(struct bio *bio); +int blk_mq_poll(struct request_queue *q, blk_qc_t cookie, bool spin); void blk_mq_hctx_set_fq_lock_class(struct blk_mq_hw_ctx *hctx, struct lock_class_key *key); diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 2a802e011a95..ebe16f55cba4 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1853,6 +1853,7 @@ static inline void blk_ksm_unregister(struct request_queue *q) { } struct block_device_operations { blk_qc_t (*submit_bio) (struct bio *bio); + int (*poll)(struct request_queue *q, blk_qc_t cookie); int (*open) (struct block_device *, fmode_t); void (*release) (struct gendisk *, fmode_t); int (*rw_page)(struct block_device *, sector_t, struct page *, unsigned int); From patchwork Sat Feb 20 11:06:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12098107 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C54C0C433E0 for ; Mon, 22 Feb 2021 08:51:47 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 209F064E77 for ; Mon, 22 Feb 2021 08:51:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 209F064E77 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-563-dbJOTWXIOWuRy1EMYIQP3g-1; Mon, 22 Feb 2021 03:51:43 -0500 X-MC-Unique: dbJOTWXIOWuRy1EMYIQP3g-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 53C1518449FC; Mon, 22 Feb 2021 08:51:38 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2C4F63CC5; Mon, 22 Feb 2021 08:51:38 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id F21F718095CE; Mon, 22 Feb 2021 08:51:37 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 11KB6pxT010440 for ; Sat, 20 Feb 2021 06:06:51 -0500 Received: by smtp.corp.redhat.com (Postfix) id 8FB8D121156; Sat, 20 Feb 2021 11:06:51 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast04.extmail.prod.ext.rdu2.redhat.com [10.11.55.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8A75913E790 for ; Sat, 20 Feb 2021 11:06:51 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-2.mimecast.com [205.139.110.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 721041022F09 for ; Sat, 20 Feb 2021 11:06:51 +0000 (UTC) Received: from out30-57.freemail.mail.aliyun.com (out30-57.freemail.mail.aliyun.com [115.124.30.57]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-404-a7jetn46OQmywrhwhyzV4Q-1; Sat, 20 Feb 2021 06:06:47 -0500 X-MC-Unique: a7jetn46OQmywrhwhyzV4Q-1 X-Alimail-AntiSpam: AC=PASS; BC=-1|-1; BR=01201311R171e4; CH=green; DM=||false|; DS=||; FP=0|-1|-1|-1|0|-1|-1|-1; HT=e01e04420; MF=jefflexu@linux.alibaba.com; NM=1; PH=DS; RN=9; SR=0; TI=SMTPD_---0UP1XFxp_1613819201 Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0UP1XFxp_1613819201) by smtp.aliyun-inc.com(127.0.0.1); Sat, 20 Feb 2021 19:06:42 +0800 From: Jeffle Xu To: snitzer@redhat.com, axboe@kernel.dk Date: Sat, 20 Feb 2021 19:06:29 +0800 Message-Id: <20210220110637.50305-5-jefflexu@linux.alibaba.com> In-Reply-To: <20210220110637.50305-1-jefflexu@linux.alibaba.com> References: <20210220110637.50305-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Mimecast-Bulk-Signature: yes X-Mimecast-Spam-Signature: bulk X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-loop: dm-devel@redhat.com X-Mailman-Approved-At: Mon, 22 Feb 2021 03:49:50 -0500 Cc: caspar@linux.alibaba.com, io-uring@vger.kernel.org, linux-block@vger.kernel.org, joseph.qi@linux.alibaba.com, dm-devel@redhat.com, ming.lei@redhat.com, hch@lst.de Subject: [dm-devel] [PATCH v4 04/12] block: add poll_capable method to support bio-based IO polling X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com This method can be used to check if bio-based device supports IO polling or not. For mq devices, checking for hw queue in polling mode is adequate, while the sanity check shall be implementation specific for bio-based devices. For example, dm device needs to check if all underlying devices are capable of IO polling. Though bio-based device may have done the sanity check during the device initialization phase, cacheing the result of this sanity check (such as by cacheing in the queue_flags) may not work. Because for dm devices, users could change the state of the underlying devices through '/sys/block//io_poll', bypassing the dm device above. In this case, the cached result of the very beginning sanity check could be out-of-date. Thus the sanity check needs to be done every time 'io_poll' is to be modified. Signed-off-by: Jeffle Xu --- block/blk-sysfs.c | 14 +++++++++++--- include/linux/blkdev.h | 1 + 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c index b513f1683af0..f11fedefc37d 100644 --- a/block/blk-sysfs.c +++ b/block/blk-sysfs.c @@ -420,9 +420,17 @@ static ssize_t queue_poll_store(struct request_queue *q, const char *page, unsigned long poll_on; ssize_t ret; - if (!q->tag_set || q->tag_set->nr_maps <= HCTX_TYPE_POLL || - !q->tag_set->map[HCTX_TYPE_POLL].nr_queues) - return -EINVAL; + if (queue_is_mq(q)) { + if (!q->tag_set || q->tag_set->nr_maps <= HCTX_TYPE_POLL || + !q->tag_set->map[HCTX_TYPE_POLL].nr_queues) + return -EINVAL; + } else { + struct gendisk *disk = queue_to_disk(q); + + if (!disk->fops->poll_capable || + !disk->fops->poll_capable(disk)) + return -EINVAL; + } ret = queue_var_store(&poll_on, page, count); if (ret < 0) diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index ebe16f55cba4..8a84088642ce 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1854,6 +1854,7 @@ static inline void blk_ksm_unregister(struct request_queue *q) { } struct block_device_operations { blk_qc_t (*submit_bio) (struct bio *bio); int (*poll)(struct request_queue *q, blk_qc_t cookie); + bool (*poll_capable)(struct gendisk *disk); int (*open) (struct block_device *, fmode_t); void (*release) (struct gendisk *, fmode_t); int (*rw_page)(struct block_device *, sector_t, struct page *, unsigned int); From patchwork Sat Feb 20 11:06:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12098111 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52974C433E0 for ; Mon, 22 Feb 2021 08:51:58 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A877564E61 for ; Mon, 22 Feb 2021 08:51:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A877564E61 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-603-UnaaDFj-NnG_5albzawEjg-1; Mon, 22 Feb 2021 03:51:45 -0500 X-MC-Unique: UnaaDFj-NnG_5albzawEjg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 74A831020C22; Mon, 22 Feb 2021 08:51:40 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 5380769322; Mon, 22 Feb 2021 08:51:40 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 23F1E4EBC6; Mon, 22 Feb 2021 08:51:40 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 11KB6s47010467 for ; Sat, 20 Feb 2021 06:06:54 -0500 Received: by smtp.corp.redhat.com (Postfix) id EFFAD21121A3; Sat, 20 Feb 2021 11:06:53 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast04.extmail.prod.ext.rdu2.redhat.com [10.11.55.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id EAF35200ACC4 for ; Sat, 20 Feb 2021 11:06:53 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D2B861022F09 for ; Sat, 20 Feb 2021 11:06:53 +0000 (UTC) Received: from out30-44.freemail.mail.aliyun.com (out30-44.freemail.mail.aliyun.com [115.124.30.44]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-89-7Q0E90YIPKKex4mdxjo5QA-1; Sat, 20 Feb 2021 06:06:48 -0500 X-MC-Unique: 7Q0E90YIPKKex4mdxjo5QA-1 X-Alimail-AntiSpam: AC=PASS; BC=-1|-1; BR=01201311R251e4; CH=green; DM=||false|; DS=||; FP=0|-1|-1|-1|0|-1|-1|-1; HT=e01e04394; MF=jefflexu@linux.alibaba.com; NM=1; PH=DS; RN=9; SR=0; TI=SMTPD_---0UP1N.EP_1613819202 Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0UP1N.EP_1613819202) by smtp.aliyun-inc.com(127.0.0.1); Sat, 20 Feb 2021 19:06:43 +0800 From: Jeffle Xu To: snitzer@redhat.com, axboe@kernel.dk Date: Sat, 20 Feb 2021 19:06:30 +0800 Message-Id: <20210220110637.50305-6-jefflexu@linux.alibaba.com> In-Reply-To: <20210220110637.50305-1-jefflexu@linux.alibaba.com> References: <20210220110637.50305-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-loop: dm-devel@redhat.com X-Mailman-Approved-At: Mon, 22 Feb 2021 03:49:50 -0500 Cc: caspar@linux.alibaba.com, io-uring@vger.kernel.org, linux-block@vger.kernel.org, joseph.qi@linux.alibaba.com, dm-devel@redhat.com, ming.lei@redhat.com, hch@lst.de Subject: [dm-devel] [PATCH v4 05/12] blk-mq: extract one helper function polling hw queue X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Extract the logic of polling one hw queue and related statistics handling out as the helper function. Signed-off-by: Jeffle Xu --- block/blk-mq.c | 5 +---- include/linux/blk-mq.h | 13 +++++++++++++ 2 files changed, 14 insertions(+), 4 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index c18bc23834a0..b4de2b37b826 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -3850,11 +3850,8 @@ int blk_mq_poll(struct request_queue *q, blk_qc_t cookie, bool spin) do { int ret; - hctx->poll_invoked++; - - ret = q->mq_ops->poll(hctx); + ret = blk_mq_poll_hctx(q, hctx); if (ret > 0) { - hctx->poll_success++; __set_current_state(TASK_RUNNING); return ret; } diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index adc789d5888e..cf1910c6d5ae 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -606,6 +606,19 @@ static inline void blk_rq_bio_prep(struct request *rq, struct bio *bio, rq->rq_disk = bio->bi_disk; } +static inline int blk_mq_poll_hctx(struct request_queue *q, + struct blk_mq_hw_ctx *hctx) +{ + int ret; + + hctx->poll_invoked++; + ret = q->mq_ops->poll(hctx); + if (ret > 0) + hctx->poll_success++; + + return ret; +} + blk_qc_t blk_mq_submit_bio(struct bio *bio); int blk_mq_poll(struct request_queue *q, blk_qc_t cookie, bool spin); void blk_mq_hctx_set_fq_lock_class(struct blk_mq_hw_ctx *hctx, From patchwork Sat Feb 20 11:06:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12098117 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3AB8FC433E9 for ; Mon, 22 Feb 2021 08:52:47 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B46A764EBE for ; Mon, 22 Feb 2021 08:52:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B46A764EBE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-242-beI228mfO2C-35KaOuW6fg-1; Mon, 22 Feb 2021 03:52:43 -0500 X-MC-Unique: beI228mfO2C-35KaOuW6fg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2D93787498C; Mon, 22 Feb 2021 08:52:39 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 0F0E95D9D0; Mon, 22 Feb 2021 08:52:39 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id D84E957DFE; Mon, 22 Feb 2021 08:52:38 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 11KB6rFh010462 for ; Sat, 20 Feb 2021 06:06:53 -0500 Received: by smtp.corp.redhat.com (Postfix) id 36F33200ACC4; Sat, 20 Feb 2021 11:06:53 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast05.extmail.prod.ext.rdu2.redhat.com [10.11.55.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 31680200BFD3 for ; Sat, 20 Feb 2021 11:06:53 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1D40A8919BA for ; Sat, 20 Feb 2021 11:06:53 +0000 (UTC) Received: from out30-133.freemail.mail.aliyun.com (out30-133.freemail.mail.aliyun.com [115.124.30.133]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-7-gsUVCL7UNUO4GGAsotE2bg-1; Sat, 20 Feb 2021 06:06:48 -0500 X-MC-Unique: gsUVCL7UNUO4GGAsotE2bg-1 X-Alimail-AntiSpam: AC=PASS; BC=-1|-1; BR=01201311R161e4; CH=green; DM=||false|; DS=||; FP=0|-1|-1|-1|0|-1|-1|-1; HT=e01e04395; MF=jefflexu@linux.alibaba.com; NM=1; PH=DS; RN=9; SR=0; TI=SMTPD_---0UP1BfWE_1613819203 Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0UP1BfWE_1613819203) by smtp.aliyun-inc.com(127.0.0.1); Sat, 20 Feb 2021 19:06:44 +0800 From: Jeffle Xu To: snitzer@redhat.com, axboe@kernel.dk Date: Sat, 20 Feb 2021 19:06:31 +0800 Message-Id: <20210220110637.50305-7-jefflexu@linux.alibaba.com> In-Reply-To: <20210220110637.50305-1-jefflexu@linux.alibaba.com> References: <20210220110637.50305-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-loop: dm-devel@redhat.com X-Mailman-Approved-At: Mon, 22 Feb 2021 03:49:50 -0500 Cc: caspar@linux.alibaba.com, io-uring@vger.kernel.org, linux-block@vger.kernel.org, joseph.qi@linux.alibaba.com, dm-devel@redhat.com, ming.lei@redhat.com, hch@lst.de Subject: [dm-devel] [PATCH v4 06/12] blk-mq: add iterator for polling hw queues X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Add one helper function for iterating all hardware queues in polling mode. Signed-off-by: Jeffle Xu --- include/linux/blk-mq.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index cf1910c6d5ae..0c511555ec16 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -574,6 +574,13 @@ static inline void *blk_mq_rq_to_pdu(struct request *rq) for ((i) = 0; (i) < (q)->nr_hw_queues && \ ({ hctx = (q)->queue_hw_ctx[i]; 1; }); (i)++) +#define queue_for_each_poll_hw_ctx(q, hctx, i) \ + for ((i) = 0; ((q)->tag_set->nr_maps > HCTX_TYPE_POLL) && \ + (i) < (q)->tag_set->map[HCTX_TYPE_POLL].nr_queues && \ + ({ int __idx = (q)->tag_set->map[HCTX_TYPE_POLL].queue_offset + (i); \ + hctx = (q)->queue_hw_ctx[__idx]; 1; }); \ + (i)++) + #define hctx_for_each_ctx(hctx, ctx, i) \ for ((i) = 0; (i) < (hctx)->nr_ctx && \ ({ ctx = (hctx)->ctxs[(i)]; 1; }); (i)++) From patchwork Sat Feb 20 11:06:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12098121 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 311F7C433E0 for ; Mon, 22 Feb 2021 08:53:41 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8D7B864EBE for ; Mon, 22 Feb 2021 08:53:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8D7B864EBE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-69-0eZ_NJcHOnux58qlheO3dQ-1; Mon, 22 Feb 2021 03:53:36 -0500 X-MC-Unique: 0eZ_NJcHOnux58qlheO3dQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 19672801975; Mon, 22 Feb 2021 08:53:32 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id EEBB160C04; Mon, 22 Feb 2021 08:53:31 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id BE4E058074; Mon, 22 Feb 2021 08:53:31 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 11KB6tUG010474 for ; Sat, 20 Feb 2021 06:06:56 -0500 Received: by smtp.corp.redhat.com (Postfix) id CEDB92026D13; Sat, 20 Feb 2021 11:06:55 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast02.extmail.prod.ext.rdu2.redhat.com [10.11.55.18]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C9C722026D46 for ; Sat, 20 Feb 2021 11:06:55 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [205.139.110.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B5922800B29 for ; Sat, 20 Feb 2021 11:06:55 +0000 (UTC) Received: from out30-45.freemail.mail.aliyun.com (out30-45.freemail.mail.aliyun.com [115.124.30.45]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-544-US2aGmSJMQaN4gecGg6Mvg-1; Sat, 20 Feb 2021 06:06:51 -0500 X-MC-Unique: US2aGmSJMQaN4gecGg6Mvg-1 X-Alimail-AntiSpam: AC=PASS; BC=-1|-1; BR=01201311R201e4; CH=green; DM=||false|; DS=||; FP=0|-1|-1|-1|0|-1|-1|-1; HT=e01e01424; MF=jefflexu@linux.alibaba.com; NM=1; PH=DS; RN=9; SR=0; TI=SMTPD_---0UP1AWdu_1613819204 Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0UP1AWdu_1613819204) by smtp.aliyun-inc.com(127.0.0.1); Sat, 20 Feb 2021 19:06:45 +0800 From: Jeffle Xu To: snitzer@redhat.com, axboe@kernel.dk Date: Sat, 20 Feb 2021 19:06:32 +0800 Message-Id: <20210220110637.50305-8-jefflexu@linux.alibaba.com> In-Reply-To: <20210220110637.50305-1-jefflexu@linux.alibaba.com> References: <20210220110637.50305-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-loop: dm-devel@redhat.com X-Mailman-Approved-At: Mon, 22 Feb 2021 03:49:50 -0500 Cc: caspar@linux.alibaba.com, io-uring@vger.kernel.org, linux-block@vger.kernel.org, joseph.qi@linux.alibaba.com, dm-devel@redhat.com, ming.lei@redhat.com, hch@lst.de Subject: [dm-devel] [PATCH v4 07/12] blk-mq: add one helper function getting hw queue X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Add one helper function getting hw queue mapping to specific CPU, and of specific type. Signed-off-by: Jeffle Xu --- block/blk-mq.c | 10 ++++++++++ include/linux/blk-mq.h | 2 ++ 2 files changed, 12 insertions(+) diff --git a/block/blk-mq.c b/block/blk-mq.c index b4de2b37b826..734aa1338b69 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -3876,6 +3876,16 @@ unsigned int blk_mq_rq_cpu(struct request *rq) } EXPORT_SYMBOL(blk_mq_rq_cpu); + +struct blk_mq_hw_ctx *blk_mq_get_hctx(struct request_queue *q, int cpu, + unsigned int flags) +{ + struct blk_mq_ctx *ctx = __blk_mq_get_ctx(q, cpu); + + return blk_mq_map_queue(q, flags, ctx); +} +EXPORT_SYMBOL(blk_mq_get_hctx); + static int __init blk_mq_init(void) { int i; diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 0c511555ec16..9b6a5c1fcb2d 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -630,5 +630,7 @@ blk_qc_t blk_mq_submit_bio(struct bio *bio); int blk_mq_poll(struct request_queue *q, blk_qc_t cookie, bool spin); void blk_mq_hctx_set_fq_lock_class(struct blk_mq_hw_ctx *hctx, struct lock_class_key *key); +struct blk_mq_hw_ctx *blk_mq_get_hctx(struct request_queue *q, int cpu, + unsigned int flags); #endif From patchwork Sat Feb 20 11:06:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12098119 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34F78C433E0 for ; Mon, 22 Feb 2021 08:52:50 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7BC4E64EBE for ; Mon, 22 Feb 2021 08:52:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7BC4E64EBE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-122-4NV8wtOLMoWG1tgrNl4_7Q-1; Mon, 22 Feb 2021 03:52:45 -0500 X-MC-Unique: 4NV8wtOLMoWG1tgrNl4_7Q-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 542296EE2C; Mon, 22 Feb 2021 08:52:41 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2DC371A8A8; Mon, 22 Feb 2021 08:52:41 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id EE82557DFE; Mon, 22 Feb 2021 08:52:40 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 11KB70AV010511 for ; Sat, 20 Feb 2021 06:07:00 -0500 Received: by smtp.corp.redhat.com (Postfix) id EA2DB10F2702; Sat, 20 Feb 2021 11:06:59 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast01.extmail.prod.ext.rdu2.redhat.com [10.11.55.17]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E5D9810F26EA for ; Sat, 20 Feb 2021 11:06:57 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [205.139.110.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4D1B385A59D for ; Sat, 20 Feb 2021 11:06:57 +0000 (UTC) Received: from out30-130.freemail.mail.aliyun.com (out30-130.freemail.mail.aliyun.com [115.124.30.130]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-282-hQCHp_LPMXuZM0ejJyB_Rw-1; Sat, 20 Feb 2021 06:06:52 -0500 X-MC-Unique: hQCHp_LPMXuZM0ejJyB_Rw-1 X-Alimail-AntiSpam: AC=PASS; BC=-1|-1; BR=01201311R121e4; CH=green; DM=||false|; DS=||; FP=0|-1|-1|-1|0|-1|-1|-1; HT=e01e04395; MF=jefflexu@linux.alibaba.com; NM=1; PH=DS; RN=9; SR=0; TI=SMTPD_---0UP0sPUY_1613819205 Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0UP0sPUY_1613819205) by smtp.aliyun-inc.com(127.0.0.1); Sat, 20 Feb 2021 19:06:46 +0800 From: Jeffle Xu To: snitzer@redhat.com, axboe@kernel.dk Date: Sat, 20 Feb 2021 19:06:33 +0800 Message-Id: <20210220110637.50305-9-jefflexu@linux.alibaba.com> In-Reply-To: <20210220110637.50305-1-jefflexu@linux.alibaba.com> References: <20210220110637.50305-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-loop: dm-devel@redhat.com X-Mailman-Approved-At: Mon, 22 Feb 2021 03:49:50 -0500 Cc: caspar@linux.alibaba.com, io-uring@vger.kernel.org, linux-block@vger.kernel.org, joseph.qi@linux.alibaba.com, dm-devel@redhat.com, ming.lei@redhat.com, hch@lst.de Subject: [dm-devel] [PATCH v4 08/12] dm: always return BLK_QC_T_NONE for bio-based device X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Currently the returned cookie of bio-based device is not used at all. Cookie of bio-based device will be refactored in the following patch. Signed-off-by: Jeffle Xu Reviewed-by: Mike Snitzer --- drivers/md/dm.c | 26 ++++++++++---------------- 1 file changed, 10 insertions(+), 16 deletions(-) diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 8a3d73efb9dd..c2945c90745e 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -1252,14 +1252,13 @@ void dm_accept_partial_bio(struct bio *bio, unsigned n_sectors) } EXPORT_SYMBOL_GPL(dm_accept_partial_bio); -static blk_qc_t __map_bio(struct dm_target_io *tio) +static void __map_bio(struct dm_target_io *tio) { int r; sector_t sector; struct bio *clone = &tio->clone; struct dm_io *io = tio->io; struct dm_target *ti = tio->ti; - blk_qc_t ret = BLK_QC_T_NONE; clone->bi_end_io = clone_endio; @@ -1278,7 +1277,7 @@ static blk_qc_t __map_bio(struct dm_target_io *tio) case DM_MAPIO_REMAPPED: /* the bio has been remapped so dispatch it */ trace_block_bio_remap(clone, bio_dev(io->orig_bio), sector); - ret = submit_bio_noacct(clone); + submit_bio_noacct(clone); break; case DM_MAPIO_KILL: free_tio(tio); @@ -1292,8 +1291,6 @@ static blk_qc_t __map_bio(struct dm_target_io *tio) DMWARN("unimplemented target map return value: %d", r); BUG(); } - - return ret; } static void bio_setup_sector(struct bio *bio, sector_t sector, unsigned len) @@ -1380,7 +1377,7 @@ static void alloc_multiple_bios(struct bio_list *blist, struct clone_info *ci, } } -static blk_qc_t __clone_and_map_simple_bio(struct clone_info *ci, +static void __clone_and_map_simple_bio(struct clone_info *ci, struct dm_target_io *tio, unsigned *len) { struct bio *clone = &tio->clone; @@ -1391,7 +1388,7 @@ static blk_qc_t __clone_and_map_simple_bio(struct clone_info *ci, if (len) bio_setup_sector(clone, ci->sector, *len); - return __map_bio(tio); + __map_bio(tio); } static void __send_duplicate_bios(struct clone_info *ci, struct dm_target *ti, @@ -1405,7 +1402,7 @@ static void __send_duplicate_bios(struct clone_info *ci, struct dm_target *ti, while ((bio = bio_list_pop(&blist))) { tio = container_of(bio, struct dm_target_io, clone); - (void) __clone_and_map_simple_bio(ci, tio, len); + __clone_and_map_simple_bio(ci, tio, len); } } @@ -1450,7 +1447,7 @@ static int __clone_and_map_data_bio(struct clone_info *ci, struct dm_target *ti, free_tio(tio); return r; } - (void) __map_bio(tio); + __map_bio(tio); return 0; } @@ -1565,11 +1562,10 @@ static void init_clone_info(struct clone_info *ci, struct mapped_device *md, /* * Entry point to split a bio into clones and submit them to the targets. */ -static blk_qc_t __split_and_process_bio(struct mapped_device *md, +static void __split_and_process_bio(struct mapped_device *md, struct dm_table *map, struct bio *bio) { struct clone_info ci; - blk_qc_t ret = BLK_QC_T_NONE; int error = 0; init_clone_info(&ci, md, map, bio); @@ -1613,7 +1609,7 @@ static blk_qc_t __split_and_process_bio(struct mapped_device *md, bio_chain(b, bio); trace_block_split(b, bio->bi_iter.bi_sector); - ret = submit_bio_noacct(bio); + submit_bio_noacct(bio); break; } } @@ -1621,13 +1617,11 @@ static blk_qc_t __split_and_process_bio(struct mapped_device *md, /* drop the extra reference count */ dec_pending(ci.io, errno_to_blk_status(error)); - return ret; } static blk_qc_t dm_submit_bio(struct bio *bio) { struct mapped_device *md = bio->bi_disk->private_data; - blk_qc_t ret = BLK_QC_T_NONE; int srcu_idx; struct dm_table *map; @@ -1657,10 +1651,10 @@ static blk_qc_t dm_submit_bio(struct bio *bio) if (is_abnormal_io(bio)) blk_queue_split(&bio); - ret = __split_and_process_bio(md, map, bio); + __split_and_process_bio(md, map, bio); out: dm_put_live_table(md, srcu_idx); - return ret; + return BLK_QC_T_NONE; } /*----------------------------------------------------------------- From patchwork Sat Feb 20 11:06:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12098123 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 321B7C433E0 for ; Mon, 22 Feb 2021 08:53:43 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 83FE264E61 for ; Mon, 22 Feb 2021 08:53:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 83FE264E61 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-177-ZqdA7YF4Oy2vdEfeWDrc5Q-1; Mon, 22 Feb 2021 03:53:39 -0500 X-MC-Unique: ZqdA7YF4Oy2vdEfeWDrc5Q-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 62EE41936B6C; Mon, 22 Feb 2021 08:53:34 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 4377C60C04; Mon, 22 Feb 2021 08:53:34 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 10AF458076; Mon, 22 Feb 2021 08:53:34 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 11KB6xfm010506 for ; Sat, 20 Feb 2021 06:06:59 -0500 Received: by smtp.corp.redhat.com (Postfix) id 7E50813E781; Sat, 20 Feb 2021 11:06:59 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast03.extmail.prod.ext.rdu2.redhat.com [10.11.55.19]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 790FA121125 for ; Sat, 20 Feb 2021 11:06:59 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 65FA99291AB for ; Sat, 20 Feb 2021 11:06:59 +0000 (UTC) Received: from out30-130.freemail.mail.aliyun.com (out30-130.freemail.mail.aliyun.com [115.124.30.130]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-3-TKDMzGshOkC6haNplnU6Vw-1; Sat, 20 Feb 2021 06:06:53 -0500 X-MC-Unique: TKDMzGshOkC6haNplnU6Vw-1 X-Alimail-AntiSpam: AC=PASS; BC=-1|-1; BR=01201311R181e4; CH=green; DM=||false|; DS=||; FP=0|-1|-1|-1|0|-1|-1|-1; HT=alimailimapcm10staff010182156082; MF=jefflexu@linux.alibaba.com; NM=1; PH=DS; RN=9; SR=0; TI=SMTPD_---0UP1O7pW_1613819207 Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0UP1O7pW_1613819207) by smtp.aliyun-inc.com(127.0.0.1); Sat, 20 Feb 2021 19:06:47 +0800 From: Jeffle Xu To: snitzer@redhat.com, axboe@kernel.dk Date: Sat, 20 Feb 2021 19:06:34 +0800 Message-Id: <20210220110637.50305-10-jefflexu@linux.alibaba.com> In-Reply-To: <20210220110637.50305-1-jefflexu@linux.alibaba.com> References: <20210220110637.50305-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-loop: dm-devel@redhat.com X-Mailman-Approved-At: Mon, 22 Feb 2021 03:49:50 -0500 Cc: caspar@linux.alibaba.com, io-uring@vger.kernel.org, linux-block@vger.kernel.org, joseph.qi@linux.alibaba.com, dm-devel@redhat.com, ming.lei@redhat.com, hch@lst.de Subject: [dm-devel] [PATCH v4 09/12] nvme/pci: don't wait for locked polling queue X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com There's no sense waiting for the hw queue when it currently has been locked by another polling instance. The polling instance currently occupying the hw queue will help reap the completion events. It shall be safe to surrender the hw queue, as long as we could reapply for polling later. For Synchronous polling, blk_poll() will reapply for polling, since @spin is always True in this case. While For asynchronous polling, i.e. io_uring itself will reapply for polling when the previous polling returns 0. Besides, it shall do no harm to the polling performance of mq devices. Signed-off-by: Jeffle Xu --- drivers/nvme/host/pci.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 81e6389b2042..db164fcd04e2 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -1106,7 +1106,9 @@ static int nvme_poll(struct blk_mq_hw_ctx *hctx) if (!nvme_cqe_pending(nvmeq)) return 0; - spin_lock(&nvmeq->cq_poll_lock); + if (!spin_trylock(&nvmeq->cq_poll_lock)) + return 0; + found = nvme_process_cq(nvmeq); spin_unlock(&nvmeq->cq_poll_lock); From patchwork Sat Feb 20 11:06:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12098109 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CE38C433E0 for ; Mon, 22 Feb 2021 08:51:52 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BA93D64E77 for ; Mon, 22 Feb 2021 08:51:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BA93D64E77 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-61-kJh8nS5bNi6b51v9XwASwA-1; Mon, 22 Feb 2021 03:51:47 -0500 X-MC-Unique: kJh8nS5bNi6b51v9XwASwA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 917D480196E; Mon, 22 Feb 2021 08:51:42 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 736AF1B059; Mon, 22 Feb 2021 08:51:42 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 3D03249; Mon, 22 Feb 2021 08:51:42 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 11KB6uQM010482 for ; Sat, 20 Feb 2021 06:06:56 -0500 Received: by smtp.corp.redhat.com (Postfix) id 8D986200ACC4; Sat, 20 Feb 2021 11:06:56 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast05.extmail.prod.ext.rdu2.redhat.com [10.11.55.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8822221121A3 for ; Sat, 20 Feb 2021 11:06:56 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [205.139.110.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6F331800962 for ; Sat, 20 Feb 2021 11:06:56 +0000 (UTC) Received: from out30-45.freemail.mail.aliyun.com (out30-45.freemail.mail.aliyun.com [115.124.30.45]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-554-zqFzIMfYP-6WyyNhznEiFA-1; Sat, 20 Feb 2021 06:06:53 -0500 X-MC-Unique: zqFzIMfYP-6WyyNhznEiFA-1 X-Alimail-AntiSpam: AC=PASS; BC=-1|-1; BR=01201311R121e4; CH=green; DM=||false|; DS=||; FP=0|-1|-1|-1|0|-1|-1|-1; HT=e01e04420; MF=jefflexu@linux.alibaba.com; NM=1; PH=DS; RN=9; SR=0; TI=SMTPD_---0UP0sPUk_1613819208 Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0UP0sPUk_1613819208) by smtp.aliyun-inc.com(127.0.0.1); Sat, 20 Feb 2021 19:06:48 +0800 From: Jeffle Xu To: snitzer@redhat.com, axboe@kernel.dk Date: Sat, 20 Feb 2021 19:06:35 +0800 Message-Id: <20210220110637.50305-11-jefflexu@linux.alibaba.com> In-Reply-To: <20210220110637.50305-1-jefflexu@linux.alibaba.com> References: <20210220110637.50305-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-loop: dm-devel@redhat.com X-Mailman-Approved-At: Mon, 22 Feb 2021 03:49:50 -0500 Cc: caspar@linux.alibaba.com, io-uring@vger.kernel.org, linux-block@vger.kernel.org, joseph.qi@linux.alibaba.com, dm-devel@redhat.com, ming.lei@redhat.com, hch@lst.de Subject: [dm-devel] [PATCH v4 10/12] block: fastpath for bio-based polling X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Offer one fastpath for bio-based polling when bio submitted to dm device is not split. In this case, there will be only one bio submitted to only one polling hw queue of one underlying mq device, and thus we don't need to track all split bios or iterate through all polling hw queues. The pointer to the polling hw queue the bio submitted to is returned here as the returned cookie. In this case, the polling routine will call mq_ops->poll() directly with the hw queue converted from the input cookie. If the original bio submitted to dm device is split to multiple bios and thus submitted to multiple polling hw queues, the polling routine will fall back to iterating all hw queues (in polling mode) of all underlying mq devices. Signed-off-by: Jeffle Xu --- block/blk-core.c | 38 ++++++++++++++++++++++++++++++++++++-- include/linux/blk_types.h | 8 ++++++++ include/linux/types.h | 2 +- 3 files changed, 45 insertions(+), 3 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 37aa513da5f2..82349e3c2a24 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -956,11 +956,19 @@ static blk_qc_t __submit_bio(struct bio *bio) * bio_list_on_stack[0] contains bios submitted by the current ->submit_bio. * bio_list_on_stack[1] contains bios that were submitted before the current * ->submit_bio_bio, but that haven't been processed yet. + * + * Return: + * - BLK_QC_T_NONE, no need for IO polling. + * - BLK_QC_T_BIO_POLL_ALL, @bio gets split and enqueued into multi hw queues. + * - Otherwise, @bio is not split, returning the pointer to the corresponding + * hw queue that the bio enqueued into as the returned cookie. */ static blk_qc_t __submit_bio_noacct(struct bio *bio) { struct bio_list bio_list_on_stack[2]; blk_qc_t ret = BLK_QC_T_NONE; + struct request_queue *top_q = bio->bi_disk->queue; + bool poll_on = test_bit(QUEUE_FLAG_POLL, &top_q->queue_flags); BUG_ON(bio->bi_next); @@ -968,6 +976,7 @@ static blk_qc_t __submit_bio_noacct(struct bio *bio) current->bio_list = bio_list_on_stack; do { + blk_qc_t cookie; struct request_queue *q = bio->bi_disk->queue; struct bio_list lower, same; @@ -980,7 +989,23 @@ static blk_qc_t __submit_bio_noacct(struct bio *bio) bio_list_on_stack[1] = bio_list_on_stack[0]; bio_list_init(&bio_list_on_stack[0]); - ret = __submit_bio(bio); + cookie = __submit_bio(bio); + + if (poll_on && blk_qc_t_valid(cookie)) { + unsigned int queue_num = blk_qc_t_to_queue_num(cookie); + struct blk_mq_hw_ctx *hctx = q->queue_hw_ctx[queue_num]; + + cookie = (blk_qc_t)hctx; + + if (!blk_qc_t_valid(ret)) { + /* set initial value */ + ret = cookie; + } else if (ret != cookie) { + /* bio gets split and enqueued to multi hctxs */ + ret = BLK_QC_T_BIO_POLL_ALL; + poll_on = false; + } + } /* * Sort new bios into those for a lower level and those for the @@ -1003,6 +1028,7 @@ static blk_qc_t __submit_bio_noacct(struct bio *bio) } while ((bio = bio_list_pop(&bio_list_on_stack[0]))); current->bio_list = NULL; + return ret; } @@ -1142,7 +1168,15 @@ static int blk_bio_poll(struct request_queue *q, blk_qc_t cookie, bool spin) do { int ret; - ret = disk->fops->poll(q, cookie); + if (unlikely(blk_qc_t_is_poll_multi(cookie))) + ret = disk->fops->poll(q, cookie); + else { + struct blk_mq_hw_ctx *hctx = (struct blk_mq_hw_ctx *)cookie; + struct request_queue *target_q = hctx->queue; + + ret = blk_mq_poll_hctx(target_q, hctx); + } + if (ret > 0) { __set_current_state(TASK_RUNNING); return ret; diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 2e05244fc16d..6f27446ebada 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -557,6 +557,14 @@ static inline bool blk_qc_t_is_internal(blk_qc_t cookie) return (cookie & BLK_QC_T_INTERNAL) != 0; } +/* Macros for blk_qc_t used for bio-based polling */ +#define BLK_QC_T_BIO_POLL_ALL 1U + +static inline bool blk_qc_t_is_poll_multi(blk_qc_t cookie) +{ + return cookie & BLK_QC_T_BIO_POLL_ALL; +} + struct blk_rq_stat { u64 mean; u64 min; diff --git a/include/linux/types.h b/include/linux/types.h index da5ca7e1bea9..f6301014a459 100644 --- a/include/linux/types.h +++ b/include/linux/types.h @@ -126,7 +126,7 @@ typedef u64 sector_t; typedef u64 blkcnt_t; /* cookie used for IO polling */ -typedef unsigned int blk_qc_t; +typedef uintptr_t blk_qc_t; /* * The type of an index into the pagecache. From patchwork Sat Feb 20 11:06:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12098127 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9EF8DC433DB for ; Mon, 22 Feb 2021 08:53:46 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 06A6664E61 for ; Mon, 22 Feb 2021 08:53:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 06A6664E61 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-389-_qL58AF9P2ulTI2tUlR3Kg-1; Mon, 22 Feb 2021 03:53:43 -0500 X-MC-Unique: _qL58AF9P2ulTI2tUlR3Kg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 88F6185EE69; Mon, 22 Feb 2021 08:53:38 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 6BD5A70481; Mon, 22 Feb 2021 08:53:38 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 3D54058075; Mon, 22 Feb 2021 08:53:38 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 11KB710B010524 for ; Sat, 20 Feb 2021 06:07:01 -0500 Received: by smtp.corp.redhat.com (Postfix) id B95EA200BD73; Sat, 20 Feb 2021 11:07:01 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast05.extmail.prod.ext.rdu2.redhat.com [10.11.55.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B415F200ACC4 for ; Sat, 20 Feb 2021 11:07:01 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-2.mimecast.com [207.211.31.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9B276800962 for ; Sat, 20 Feb 2021 11:07:01 +0000 (UTC) Received: from out30-43.freemail.mail.aliyun.com (out30-43.freemail.mail.aliyun.com [115.124.30.43]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-203-GVpTsIscNPOjfVLgKBFk1w-1; Sat, 20 Feb 2021 06:06:56 -0500 X-MC-Unique: GVpTsIscNPOjfVLgKBFk1w-1 X-Alimail-AntiSpam: AC=PASS; BC=-1|-1; BR=01201311R961e4; CH=green; DM=||false|; DS=||; FP=0|-1|-1|-1|0|-1|-1|-1; HT=e01e04394; MF=jefflexu@linux.alibaba.com; NM=1; PH=DS; RN=9; SR=0; TI=SMTPD_---0UP1O7pq_1613819209 Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0UP1O7pq_1613819209) by smtp.aliyun-inc.com(127.0.0.1); Sat, 20 Feb 2021 19:06:49 +0800 From: Jeffle Xu To: snitzer@redhat.com, axboe@kernel.dk Date: Sat, 20 Feb 2021 19:06:36 +0800 Message-Id: <20210220110637.50305-12-jefflexu@linux.alibaba.com> In-Reply-To: <20210220110637.50305-1-jefflexu@linux.alibaba.com> References: <20210220110637.50305-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.78 on 10.11.54.6 X-loop: dm-devel@redhat.com X-Mailman-Approved-At: Mon, 22 Feb 2021 03:49:50 -0500 Cc: caspar@linux.alibaba.com, io-uring@vger.kernel.org, linux-block@vger.kernel.org, joseph.qi@linux.alibaba.com, dm-devel@redhat.com, ming.lei@redhat.com, hch@lst.de Subject: [dm-devel] [PATCH v4 11/12] block: sub-fastpath for bio-based polling X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Offer one sub-fastpath for bio-based polling when bio submitted to dm device gets split and enqueued into multiple hw queues, while the IO submission process has not been migrated to another CPU. In this case, the IO submission routine will return the CPU number on which the IO submission happened as the returned cookie, while the polling routine will only iterate and poll on hw queues that this CPU number maps, instead of iterating *all* hw queues. This optimization can dramatically reduce cache ping-pong and thus improve the polling performance, when multiple hw queues in polling mode per device could be reserved when there are multiple polling processes. It will fall back to iterating all hw queues in polling mode, once the process has ever been migrated to another CPU during the IO submission phase. Signed-off-by: Jeffle Xu --- block/blk-core.c | 24 ++++++++++++++-- include/linux/blk_types.h | 60 +++++++++++++++++++++++++++++++++++++-- 2 files changed, 79 insertions(+), 5 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 82349e3c2a24..e13219908ef8 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -959,7 +959,10 @@ static blk_qc_t __submit_bio(struct bio *bio) * * Return: * - BLK_QC_T_NONE, no need for IO polling. - * - BLK_QC_T_BIO_POLL_ALL, @bio gets split and enqueued into multi hw queues. + * - When @bio gets split and enqueued into multi hw queues, return the + * corresponding CPU number with BLK_QC_T_BIO_POLL_CPU flag set when the + * current process has not been migrated to another CPU, + * BLK_QC_T_BIO_POLL_ALL otherwise. * - Otherwise, @bio is not split, returning the pointer to the corresponding * hw queue that the bio enqueued into as the returned cookie. */ @@ -968,13 +971,17 @@ static blk_qc_t __submit_bio_noacct(struct bio *bio) struct bio_list bio_list_on_stack[2]; blk_qc_t ret = BLK_QC_T_NONE; struct request_queue *top_q = bio->bi_disk->queue; - bool poll_on = test_bit(QUEUE_FLAG_POLL, &top_q->queue_flags); + bool orig_poll_on, poll_on; + u64 old_nr_migrations; BUG_ON(bio->bi_next); bio_list_init(&bio_list_on_stack[0]); current->bio_list = bio_list_on_stack; + orig_poll_on = poll_on = test_bit(QUEUE_FLAG_POLL, &top_q->queue_flags); + old_nr_migrations = READ_ONCE(current->se.nr_migrations); + do { blk_qc_t cookie; struct request_queue *q = bio->bi_disk->queue; @@ -1002,7 +1009,7 @@ static blk_qc_t __submit_bio_noacct(struct bio *bio) ret = cookie; } else if (ret != cookie) { /* bio gets split and enqueued to multi hctxs */ - ret = BLK_QC_T_BIO_POLL_ALL; + ret = blk_qc_t_get(); poll_on = false; } } @@ -1029,6 +1036,17 @@ static blk_qc_t __submit_bio_noacct(struct bio *bio) current->bio_list = NULL; + /* + * For cases when bio gets split and enqueued into multi hctxs, return + * the corresponding CPU number when current process has not been + * migrated to another CPU. Return BLK_QC_T_BIO_POLL_ALL otherwise, + * falling back to iterating and polling on all hw queues, since split + * bios are submitted to different CPUs in this case. + */ + if (orig_poll_on != poll_on && + old_nr_migrations != READ_ONCE(current->se.nr_migrations)) + ret = BLK_QC_T_BIO_POLL_ALL; + return ret; } diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 6f27446ebada..f8d34f02378d 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -557,12 +557,68 @@ static inline bool blk_qc_t_is_internal(blk_qc_t cookie) return (cookie & BLK_QC_T_INTERNAL) != 0; } -/* Macros for blk_qc_t used for bio-based polling */ +/* + * Macros for blk_qc_t used for bio-based polling. + * + * 1. When @bio is not split, the returned cookie is actually the pointer to the + * corresponding hw queue that the bio is enqueued into, thus the low two + * bits could be reused as other flags. + * +-----------------------------------------+ + * | struct blk_mq_hw_ctx* |0|0| + * +-----------------------------------------+ + * + * 2. When @bio gets split and enqueued into multi hw queues, and current + * process has not been migrated to another CPU, the returned cookie actually + * stores the corresponding CPU number on which the IO submission happened, + * Also with BLK_QC_T_BIO_POLL_CPU flag set. + * In this case, bio-based polling could only iterate and poll on hw queues + * that this CPU number maps. + * +-----------------------------------------+ + * | CPU | |1| | + * +-----------------------------------------+ + * ^ + * BLK_QC_T_BIO_POLL_CPU + * + * 3. When @bio gets split and enqueued into multi hw queues, and current + * process has ever been migrated to another CPU, the returned cookie is just + * BLK_QC_T_BIO_POLL_ALL flag. + * In this case, bio-based polling should iterate and poll on all hw queues + * in polling mode. + * +-----------------------------------------+ + * | | |1| + * +-----------------------------------------+ + * ^ + * BLK_QC_T_BIO_POLL_ALL + * + * 4. Otherwise, return BLK_QC_T_NONE as the cookie. + * No need for IO polling in this case. + * +-----------------------------------------+ + * | BLK_QC_T_NONE | + * +-----------------------------------------+ + */ + +/* The low two bits of cookie for bio-based polling are reused for flags. */ #define BLK_QC_T_BIO_POLL_ALL 1U +#define BLK_QC_T_BIO_POLL_CPU 2U static inline bool blk_qc_t_is_poll_multi(blk_qc_t cookie) { - return cookie & BLK_QC_T_BIO_POLL_ALL; + return cookie & (BLK_QC_T_BIO_POLL_ALL | BLK_QC_T_BIO_POLL_CPU); +} + +static inline bool blk_qc_t_is_poll_cpu(blk_qc_t cookie) +{ + return cookie & BLK_QC_T_BIO_POLL_CPU; +} + +static inline blk_qc_t blk_qc_t_get(void) +{ + return (raw_smp_processor_id() << BLK_QC_T_SHIFT) | BLK_QC_T_BIO_POLL_CPU; +} + +static inline int blk_qc_t_to_cpu(blk_qc_t cookie) +{ + return cookie >> BLK_QC_T_SHIFT; } struct blk_rq_stat { From patchwork Sat Feb 20 11:06:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12098125 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81942C433E0 for ; Mon, 22 Feb 2021 08:53:45 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D8C0664ED7 for ; Mon, 22 Feb 2021 08:53:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D8C0664ED7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-343-BjcCS9KhOPKRRos-ZS1h8A-1; Mon, 22 Feb 2021 03:53:41 -0500 X-MC-Unique: BjcCS9KhOPKRRos-ZS1h8A-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 82C801936B6F; Mon, 22 Feb 2021 08:53:36 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 63B0C77710; Mon, 22 Feb 2021 08:53:36 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 074B518095CD; Mon, 22 Feb 2021 08:53:36 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 11KB70qc010517 for ; Sat, 20 Feb 2021 06:07:00 -0500 Received: by smtp.corp.redhat.com (Postfix) id BA4112026D13; Sat, 20 Feb 2021 11:07:00 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast04.extmail.prod.ext.rdu2.redhat.com [10.11.55.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B4D9C2026D46 for ; Sat, 20 Feb 2021 11:07:00 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-2.mimecast.com [207.211.31.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9C4FD1022F09 for ; Sat, 20 Feb 2021 11:07:00 +0000 (UTC) Received: from out30-56.freemail.mail.aliyun.com (out30-56.freemail.mail.aliyun.com [115.124.30.56]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-59-6NKDVdDSPXC-5YVBGyVZcg-1; Sat, 20 Feb 2021 06:06:54 -0500 X-MC-Unique: 6NKDVdDSPXC-5YVBGyVZcg-1 X-Alimail-AntiSpam: AC=PASS; BC=-1|-1; BR=01201311R171e4; CH=green; DM=||false|; DS=||; FP=0|-1|-1|-1|0|-1|-1|-1; HT=e01e04423; MF=jefflexu@linux.alibaba.com; NM=1; PH=DS; RN=9; SR=0; TI=SMTPD_---0UP1XFyk_1613819210 Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0UP1XFyk_1613819210) by smtp.aliyun-inc.com(127.0.0.1); Sat, 20 Feb 2021 19:06:50 +0800 From: Jeffle Xu To: snitzer@redhat.com, axboe@kernel.dk Date: Sat, 20 Feb 2021 19:06:37 +0800 Message-Id: <20210220110637.50305-13-jefflexu@linux.alibaba.com> In-Reply-To: <20210220110637.50305-1-jefflexu@linux.alibaba.com> References: <20210220110637.50305-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-loop: dm-devel@redhat.com X-Mailman-Approved-At: Mon, 22 Feb 2021 03:49:50 -0500 Cc: caspar@linux.alibaba.com, io-uring@vger.kernel.org, linux-block@vger.kernel.org, joseph.qi@linux.alibaba.com, dm-devel@redhat.com, ming.lei@redhat.com, hch@lst.de Subject: [dm-devel] [PATCH v4 12/12] dm: support IO polling for bio-based dm device X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com IO polling is enabled when all underlying target devices are capable of IO polling. The sanity check supports the stacked device model, in which one dm device may be build upon another dm device. In this case, the mapped device will check if the underlying dm target device supports IO polling. Signed-off-by: Jeffle Xu --- drivers/md/dm-table.c | 26 ++++++++++++ drivers/md/dm.c | 78 +++++++++++++++++++++++++++++++++++ include/linux/device-mapper.h | 1 + 3 files changed, 105 insertions(+) diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c index 2bc256d61550..0210ec7a3dee 100644 --- a/drivers/md/dm-table.c +++ b/drivers/md/dm-table.c @@ -1768,6 +1768,19 @@ static int device_requires_stable_pages(struct dm_target *ti, return blk_queue_stable_writes(q); } +static int device_not_poll_capable(struct dm_target *ti, struct dm_dev *dev, + sector_t start, sector_t len, void *data) +{ + struct request_queue *q = bdev_get_queue(dev->bdev); + + return !test_bit(QUEUE_FLAG_POLL, &q->queue_flags); +} + +int dm_table_supports_poll(struct dm_table *t) +{ + return !dm_table_any_dev_attr(t, device_not_poll_capable, NULL); +} + void dm_table_set_restrictions(struct dm_table *t, struct request_queue *q, struct queue_limits *limits) { @@ -1864,6 +1877,19 @@ void dm_table_set_restrictions(struct dm_table *t, struct request_queue *q, #endif blk_queue_update_readahead(q); + + /* + * Check for request-based device is remained to + * dm_mq_init_request_queue()->blk_mq_init_allocated_queue(). + * For bio-based device, only set QUEUE_FLAG_POLL when all underlying + * devices supporting polling. + */ + if (__table_type_bio_based(t->type)) { + if (dm_table_supports_poll(t)) + blk_queue_flag_set(QUEUE_FLAG_POLL, q); + else + blk_queue_flag_clear(QUEUE_FLAG_POLL, q); + } } unsigned int dm_table_get_num_targets(struct dm_table *t) diff --git a/drivers/md/dm.c b/drivers/md/dm.c index c2945c90745e..5f358efda50f 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -1657,6 +1657,82 @@ static blk_qc_t dm_submit_bio(struct bio *bio) return BLK_QC_T_NONE; } +struct dm_poll_data { + blk_qc_t cookie; + int count; +}; + +static void dm_poll_one_md(struct mapped_device *md, struct dm_poll_data *pdata); + +static int dm_poll_one_dev(struct dm_target *ti, struct dm_dev *dev, + sector_t start, sector_t len, void *data) +{ + struct dm_poll_data *pdata = data; + struct request_queue *q = bdev_get_queue(dev->bdev); + struct blk_mq_hw_ctx *hctx; + int i, cpu; + + if (queue_is_mq(q)) { + if (!percpu_ref_tryget(&q->q_usage_counter)) + return 0; + + if (blk_qc_t_is_poll_cpu(pdata->cookie)) { + cpu = blk_qc_t_to_cpu(pdata->cookie); + hctx = blk_mq_get_hctx(q, cpu, REQ_HIPRI); + pdata->count += blk_mq_poll_hctx(q, hctx); + } else { + queue_for_each_poll_hw_ctx(q, hctx, i) + pdata->count += blk_mq_poll_hctx(q, hctx); + } + + percpu_ref_put(&q->q_usage_counter); + } else + dm_poll_one_md(dev->bdev->bd_disk->private_data, pdata); + + return 0; +} + +static void dm_poll_one_md(struct mapped_device *md, struct dm_poll_data *pdata) +{ + int i, srcu_idx; + struct dm_table *t; + struct dm_target *ti; + + t = dm_get_live_table(md, &srcu_idx); + + for (i = 0; i < dm_table_get_num_targets(t); i++) { + ti = dm_table_get_target(t, i); + ti->type->iterate_devices(ti, dm_poll_one_dev, pdata); + } + + dm_put_live_table(md, srcu_idx); +} + +static int dm_bio_poll(struct request_queue *q, blk_qc_t cookie) +{ + struct gendisk *disk = queue_to_disk(q); + struct mapped_device *md = disk->private_data; + struct dm_poll_data pdata = { + .cookie = cookie, + }; + + dm_poll_one_md(md, &pdata); + return pdata.count; +} + +static bool dm_bio_poll_capable(struct gendisk *disk) +{ + int ret, srcu_idx; + struct mapped_device *md = disk->private_data; + struct dm_table *t; + + t = dm_get_live_table(md, &srcu_idx); + ret = dm_table_supports_poll(t); + dm_put_live_table(md, srcu_idx); + + return ret; +} + /*----------------------------------------------------------------- * An IDR is used to keep track of allocated minor numbers. *---------------------------------------------------------------*/ @@ -3049,6 +3125,8 @@ static const struct pr_ops dm_pr_ops = { }; static const struct block_device_operations dm_blk_dops = { + .poll = dm_bio_poll, + .poll_capable = dm_bio_poll_capable, .submit_bio = dm_submit_bio, .open = dm_blk_open, .release = dm_blk_close, diff --git a/include/linux/device-mapper.h b/include/linux/device-mapper.h index 61a66fb8ebb3..6a9de3fd0087 100644 --- a/include/linux/device-mapper.h +++ b/include/linux/device-mapper.h @@ -515,6 +515,7 @@ unsigned int dm_table_get_num_targets(struct dm_table *t); fmode_t dm_table_get_mode(struct dm_table *t); struct mapped_device *dm_table_get_md(struct dm_table *t); const char *dm_table_device_name(struct dm_table *t); +int dm_table_supports_poll(struct dm_table *t); /* * Trigger an event.