From patchwork Thu Mar 18 16:48:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12148601 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD47EC433E6 for ; Thu, 18 Mar 2021 16:49:48 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2F73E64F11 for ; Thu, 18 Mar 2021 16:49:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2F73E64F11 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1616086187; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=NJs47o1ScHzrX8vyc9ZWp+48M88+C0yeac430fE3Gvo=; b=U2KPpoWy+9+GjPAEKX03488z60bsW2gkk+bqG4foUaapygTWsrOsx+V7gPgj9ARs+iIQ55 fWfmreGvx+cO+6PAongOxh+uEKNmWsVV2uwGqeteNYZdfYHLlitJAnwXacMnqKT8j4h/iz yxbB99zBN6DYO1QhSY1tZ2MY1EATirM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-318-zP9SYJgINP2D7wfR8WwChQ-1; Thu, 18 Mar 2021 12:49:44 -0400 X-MC-Unique: zP9SYJgINP2D7wfR8WwChQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8062263CD6; Thu, 18 Mar 2021 16:49:19 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8543560916; Thu, 18 Mar 2021 16:49:18 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 8BADB1809C84; Thu, 18 Mar 2021 16:49:16 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 12IGnD7h030443 for ; Thu, 18 Mar 2021 12:49:13 -0400 Received: by smtp.corp.redhat.com (Postfix) id BEA0F1A266; Thu, 18 Mar 2021 16:49:13 +0000 (UTC) Received: from localhost (ovpn-12-24.pek2.redhat.com [10.72.12.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id E55C519D9B; Thu, 18 Mar 2021 16:49:12 +0000 (UTC) From: Ming Lei To: Jens Axboe Date: Fri, 19 Mar 2021 00:48:15 +0800 Message-Id: <20210318164827.1481133-2-ming.lei@redhat.com> In-Reply-To: <20210318164827.1481133-1-ming.lei@redhat.com> References: <20210318164827.1481133-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-loop: dm-devel@redhat.com Cc: Mike Snitzer , Ming Lei , linux-block@vger.kernel.org, dm-devel@redhat.com, Jeffle Xu , Christoph Hellwig Subject: [dm-devel] [RFC PATCH V2 01/13] block: add helper of blk_queue_poll X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com There has been 3 users, and will be more, so add one such helper. Signed-off-by: Ming Lei --- block/blk-core.c | 2 +- block/blk-mq.c | 3 +-- drivers/nvme/host/core.c | 2 +- include/linux/blkdev.h | 1 + 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index fc60ff208497..a31371d55b9d 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -836,7 +836,7 @@ static noinline_for_stack bool submit_bio_checks(struct bio *bio) } } - if (!test_bit(QUEUE_FLAG_POLL, &q->queue_flags)) + if (!blk_queue_poll(q)) bio->bi_opf &= ~REQ_HIPRI; switch (bio_op(bio)) { diff --git a/block/blk-mq.c b/block/blk-mq.c index d4d7c1caa439..63c81df3b8b5 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -3869,8 +3869,7 @@ int blk_poll(struct request_queue *q, blk_qc_t cookie, bool spin) struct blk_mq_hw_ctx *hctx; long state; - if (!blk_qc_t_valid(cookie) || - !test_bit(QUEUE_FLAG_POLL, &q->queue_flags)) + if (!blk_qc_t_valid(cookie) || !blk_queue_poll(q)) return 0; if (current->plug) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index a5653892d773..1bf94f0d2e8d 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -956,7 +956,7 @@ static void nvme_execute_rq_polled(struct request_queue *q, { DECLARE_COMPLETION_ONSTACK(wait); - WARN_ON_ONCE(!test_bit(QUEUE_FLAG_POLL, &q->queue_flags)); + WARN_ON_ONCE(!blk_queue_poll(q)); rq->cmd_flags |= REQ_HIPRI; rq->end_io_data = &wait; diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index bc6bc8383b43..89a01850cf12 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -665,6 +665,7 @@ bool blk_queue_flag_test_and_set(unsigned int flag, struct request_queue *q); #define blk_queue_fua(q) test_bit(QUEUE_FLAG_FUA, &(q)->queue_flags) #define blk_queue_registered(q) test_bit(QUEUE_FLAG_REGISTERED, &(q)->queue_flags) #define blk_queue_nowait(q) test_bit(QUEUE_FLAG_NOWAIT, &(q)->queue_flags) +#define blk_queue_poll(q) test_bit(QUEUE_FLAG_POLL, &(q)->queue_flags) extern void blk_set_pm_only(struct request_queue *q); extern void blk_clear_pm_only(struct request_queue *q); From patchwork Thu Mar 18 16:48:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12148599 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BDB92C433DB for ; Thu, 18 Mar 2021 16:49:46 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 651B964DF2 for ; Thu, 18 Mar 2021 16:49:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 651B964DF2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1616086185; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=Jc0Ha6BCHq4t794OXrtztv+2uK8DmvnOhRgAxZDvJ+I=; b=YoBoBHE5CXj8ZZbqkHE2rSDOBPqdsjPcIXntkbPS+Oy1v4H6i8VRs3m6nonfLciPds4imO MnDdzwe2cerqU3Dk9DSfTC/w/MuXcVZd3oc6mwcyjQzQAQJwwXdNvzkQMeNIuy3Ytg3V9O sAu6Hi+sT3RwbOkr+biwfCicnWnpzRE= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-481-v4psDr8HP_-bPt8gVBCaJg-1; Thu, 18 Mar 2021 12:49:43 -0400 X-MC-Unique: v4psDr8HP_-bPt8gVBCaJg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 76FB0101F7DE; Thu, 18 Mar 2021 16:49:19 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E02DC5D9C6; Thu, 18 Mar 2021 16:49:18 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id DB44A4BB7C; Thu, 18 Mar 2021 16:49:17 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 12IGnG1v030462 for ; Thu, 18 Mar 2021 12:49:17 -0400 Received: by smtp.corp.redhat.com (Postfix) id EDB8C5D9CA; Thu, 18 Mar 2021 16:49:16 +0000 (UTC) Received: from localhost (ovpn-12-24.pek2.redhat.com [10.72.12.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id EBD865D9C6; Thu, 18 Mar 2021 16:49:15 +0000 (UTC) From: Ming Lei To: Jens Axboe Date: Fri, 19 Mar 2021 00:48:16 +0800 Message-Id: <20210318164827.1481133-3-ming.lei@redhat.com> In-Reply-To: <20210318164827.1481133-1-ming.lei@redhat.com> References: <20210318164827.1481133-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-loop: dm-devel@redhat.com Cc: Mike Snitzer , Ming Lei , linux-block@vger.kernel.org, dm-devel@redhat.com, Jeffle Xu , Christoph Hellwig Subject: [dm-devel] [RFC PATCH V2 02/13] block: add one helper to free io_context X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Prepare for putting bio poll queue into io_context, so add one helper for free io_context. Signed-off-by: Ming Lei --- block/blk-ioc.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/block/blk-ioc.c b/block/blk-ioc.c index 57299f860d41..b0cde18c4b8c 100644 --- a/block/blk-ioc.c +++ b/block/blk-ioc.c @@ -17,6 +17,11 @@ */ static struct kmem_cache *iocontext_cachep; +static inline void free_io_context(struct io_context *ioc) +{ + kmem_cache_free(iocontext_cachep, ioc); +} + /** * get_io_context - increment reference count to io_context * @ioc: io_context to get @@ -129,7 +134,7 @@ static void ioc_release_fn(struct work_struct *work) spin_unlock_irq(&ioc->lock); - kmem_cache_free(iocontext_cachep, ioc); + free_io_context(ioc); } /** @@ -164,7 +169,7 @@ void put_io_context(struct io_context *ioc) } if (free_ioc) - kmem_cache_free(iocontext_cachep, ioc); + free_io_context(ioc); } /** @@ -278,7 +283,7 @@ int create_task_io_context(struct task_struct *task, gfp_t gfp_flags, int node) (task == current || !(task->flags & PF_EXITING))) task->io_context = ioc; else - kmem_cache_free(iocontext_cachep, ioc); + free_io_context(ioc); ret = task->io_context ? 0 : -EBUSY; From patchwork Thu Mar 18 16:48:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12148615 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E2DCC433DB for ; Thu, 18 Mar 2021 16:51:31 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D8DF964ED2 for ; Thu, 18 Mar 2021 16:51:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D8DF964ED2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1616086289; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=NUppJdWriToEulnS9oayHw9dM3k8sGGW6Owz6UkVr1M=; b=g8+htTjjzx2rQPZILE9C7iLpiGi2pzo3q5dmP0iFqWqT1kPhxdNiB/uW5K9iZbMOf7NrUq Gv7RJqEEJrU+Z/Q6MFQqHNQx1AGusSGEsvUCikTp3aEOmniOTzmkoUY6SMX4EgxgczEzfi YMu6QTb12jqa2BKmLKLE/pU0pRI8O48= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-61-dGKoSFFSML6ij8IzhI9Puw-1; Thu, 18 Mar 2021 12:49:51 -0400 X-MC-Unique: dGKoSFFSML6ij8IzhI9Puw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 0EFC1100A8F9; Thu, 18 Mar 2021 16:49:33 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C8BD81992D; Thu, 18 Mar 2021 16:49:32 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 8F90E1809C83; Thu, 18 Mar 2021 16:49:32 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 12IGnKBI030483 for ; Thu, 18 Mar 2021 12:49:20 -0400 Received: by smtp.corp.redhat.com (Postfix) id B47DA6064B; Thu, 18 Mar 2021 16:49:20 +0000 (UTC) Received: from localhost (ovpn-12-24.pek2.redhat.com [10.72.12.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id D0C662BFE1; Thu, 18 Mar 2021 16:49:19 +0000 (UTC) From: Ming Lei To: Jens Axboe Date: Fri, 19 Mar 2021 00:48:17 +0800 Message-Id: <20210318164827.1481133-4-ming.lei@redhat.com> In-Reply-To: <20210318164827.1481133-1-ming.lei@redhat.com> References: <20210318164827.1481133-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-loop: dm-devel@redhat.com Cc: Mike Snitzer , Ming Lei , linux-block@vger.kernel.org, dm-devel@redhat.com, Jeffle Xu , Christoph Hellwig Subject: [dm-devel] [RFC PATCH V2 03/13] block: add helper of blk_create_io_context X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Add one helper for creating io context and prepare for supporting efficient bio based io poll. Meantime move the code of creating io_context before checking bio's REQ_HIPRI flag because the following patch may change to clear REQ_HIPRI if io_context can't be created. Signed-off-by: Ming Lei --- block/blk-core.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index a31371d55b9d..d58f8a0c80de 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -792,6 +792,18 @@ static inline blk_status_t blk_check_zone_append(struct request_queue *q, return BLK_STS_OK; } +static inline void blk_create_io_context(struct request_queue *q) +{ + /* + * Various block parts want %current->io_context, so allocate it up + * front rather than dealing with lots of pain to allocate it only + * where needed. This may fail and the block layer knows how to live + * with it. + */ + if (unlikely(!current->io_context)) + create_task_io_context(current, GFP_ATOMIC, q->node); +} + static noinline_for_stack bool submit_bio_checks(struct bio *bio) { struct block_device *bdev = bio->bi_bdev; @@ -836,6 +848,8 @@ static noinline_for_stack bool submit_bio_checks(struct bio *bio) } } + blk_create_io_context(q); + if (!blk_queue_poll(q)) bio->bi_opf &= ~REQ_HIPRI; @@ -876,15 +890,6 @@ static noinline_for_stack bool submit_bio_checks(struct bio *bio) break; } - /* - * Various block parts want %current->io_context, so allocate it up - * front rather than dealing with lots of pain to allocate it only - * where needed. This may fail and the block layer knows how to live - * with it. - */ - if (unlikely(!current->io_context)) - create_task_io_context(current, GFP_ATOMIC, q->node); - if (blk_throtl_bio(bio)) { blkcg_bio_issue_init(bio); return false; From patchwork Thu Mar 18 16:48:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12148603 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B7D1C433DB for ; Thu, 18 Mar 2021 16:49:56 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CEE2464F11 for ; Thu, 18 Mar 2021 16:49:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CEE2464F11 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1616086195; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=TDLkaZ0wiBW6ZscogH0Kguo+5c3fc/3Ta2MeZq0icBc=; b=Efj5JXnlKekccoeBW420LTQudjZVW5nMrGCKhDzIQRGntYNSI8jYmGtnSvXjiUBREdBZAc aNGq7GwgkPFgXwWbMYv5kZvskuAzTAbh5yajYjzCqUAz60ycgnCI7p1yh/NhFsdr/U5Ffc /buOAKzYVtm8RgmmWpkbNrY49gkq8ds= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-66-kvUz43v3MBeo4cbwiYSkDQ-1; Thu, 18 Mar 2021 12:49:52 -0400 X-MC-Unique: kvUz43v3MBeo4cbwiYSkDQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 606EA180FCDC; Thu, 18 Mar 2021 16:49:35 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 3B9B810027A5; Thu, 18 Mar 2021 16:49:35 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 06D0E4BB7C; Thu, 18 Mar 2021 16:49:35 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 12IGnOlj030501 for ; Thu, 18 Mar 2021 12:49:24 -0400 Received: by smtp.corp.redhat.com (Postfix) id 251A95D9D0; Thu, 18 Mar 2021 16:49:24 +0000 (UTC) Received: from localhost (ovpn-12-24.pek2.redhat.com [10.72.12.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5D3BB5D9C6; Thu, 18 Mar 2021 16:49:23 +0000 (UTC) From: Ming Lei To: Jens Axboe Date: Fri, 19 Mar 2021 00:48:18 +0800 Message-Id: <20210318164827.1481133-5-ming.lei@redhat.com> In-Reply-To: <20210318164827.1481133-1-ming.lei@redhat.com> References: <20210318164827.1481133-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-loop: dm-devel@redhat.com Cc: Mike Snitzer , Ming Lei , linux-block@vger.kernel.org, dm-devel@redhat.com, Jeffle Xu , Christoph Hellwig Subject: [dm-devel] [RFC PATCH V2 04/13] block: create io poll context for submission and poll task X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Create per-task io poll context for both IO submission and poll task if the queue is bio based and supports polling. This io polling context includes two queues: submission queue(sq) for storing HIPRI bio submission result(cookie) and the bio, written by submission task and read by poll task; polling queue(pq) for holding data moved from sq, only used in poll context for running bio polling. Following patches will support bio poll. Signed-off-by: Ming Lei --- block/blk-core.c | 71 ++++++++++++++++++++++++++++++++------- block/blk-ioc.c | 1 + block/blk-mq.c | 14 ++++++++ block/blk.h | 46 +++++++++++++++++++++++++ include/linux/iocontext.h | 2 ++ 5 files changed, 122 insertions(+), 12 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index d58f8a0c80de..0b00c21cbefb 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -792,16 +792,59 @@ static inline blk_status_t blk_check_zone_append(struct request_queue *q, return BLK_STS_OK; } -static inline void blk_create_io_context(struct request_queue *q) +static inline struct blk_bio_poll_ctx *blk_get_bio_poll_ctx(void) { - /* - * Various block parts want %current->io_context, so allocate it up - * front rather than dealing with lots of pain to allocate it only - * where needed. This may fail and the block layer knows how to live - * with it. - */ - if (unlikely(!current->io_context)) - create_task_io_context(current, GFP_ATOMIC, q->node); + struct io_context *ioc = current->io_context; + + return ioc ? ioc->data : NULL; +} + +static inline unsigned int bio_grp_list_size(unsigned int nr_grps) +{ + return sizeof(struct bio_grp_list) + nr_grps * + sizeof(struct bio_grp_list_data); +} + +static void bio_poll_ctx_init(struct blk_bio_poll_ctx *pc) +{ + pc->sq = (void *)pc + sizeof(*pc); + pc->sq->max_nr_grps = BLK_BIO_POLL_SQ_SZ; + + pc->pq = (void *)pc->sq + bio_grp_list_size(BLK_BIO_POLL_SQ_SZ); + pc->pq->max_nr_grps = BLK_BIO_POLL_PQ_SZ; + + spin_lock_init(&pc->sq_lock); + mutex_init(&pc->pq_lock); +} + +void bio_poll_ctx_alloc(struct io_context *ioc) +{ + struct blk_bio_poll_ctx *pc; + unsigned int size = sizeof(*pc) + + bio_grp_list_size(BLK_BIO_POLL_SQ_SZ) + + bio_grp_list_size(BLK_BIO_POLL_PQ_SZ); + + pc = kzalloc(GFP_ATOMIC, size); + if (pc) { + bio_poll_ctx_init(pc); + if (cmpxchg(&ioc->data, NULL, (void *)pc)) + kfree(pc); + } +} + +static inline bool blk_queue_support_bio_poll(struct request_queue *q) +{ + return !queue_is_mq(q) && blk_queue_poll(q); +} + +static inline void blk_bio_poll_preprocess(struct request_queue *q, + struct bio *bio) +{ + if (!(bio->bi_opf & REQ_HIPRI)) + return; + + if (!blk_queue_poll(q) || (!queue_is_mq(q) && !blk_get_bio_poll_ctx())) + bio->bi_opf &= ~REQ_HIPRI; } static noinline_for_stack bool submit_bio_checks(struct bio *bio) @@ -848,10 +891,14 @@ static noinline_for_stack bool submit_bio_checks(struct bio *bio) } } - blk_create_io_context(q); + /* + * Created per-task io poll queue if we supports bio polling + * and it is one HIPRI bio. + */ + blk_create_io_context(q, blk_queue_support_bio_poll(q) && + (bio->bi_opf & REQ_HIPRI)); - if (!blk_queue_poll(q)) - bio->bi_opf &= ~REQ_HIPRI; + blk_bio_poll_preprocess(q, bio); switch (bio_op(bio)) { case REQ_OP_DISCARD: diff --git a/block/blk-ioc.c b/block/blk-ioc.c index b0cde18c4b8c..5574c398eff6 100644 --- a/block/blk-ioc.c +++ b/block/blk-ioc.c @@ -19,6 +19,7 @@ static struct kmem_cache *iocontext_cachep; static inline void free_io_context(struct io_context *ioc) { + kfree(ioc->data); kmem_cache_free(iocontext_cachep, ioc); } diff --git a/block/blk-mq.c b/block/blk-mq.c index 63c81df3b8b5..c832faa52ca0 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -3852,6 +3852,17 @@ static bool blk_mq_poll_hybrid(struct request_queue *q, return blk_mq_poll_hybrid_sleep(q, rq); } +static int blk_bio_poll(struct request_queue *q, blk_qc_t cookie, bool spin) +{ + /* + * Create poll queue for storing poll bio and its cookie from + * submission queue + */ + blk_create_io_context(q, true); + + return 0; +} + /** * blk_poll - poll for IO completions * @q: the queue @@ -3875,6 +3886,9 @@ int blk_poll(struct request_queue *q, blk_qc_t cookie, bool spin) if (current->plug) blk_flush_plug_list(current->plug, false); + if (!queue_is_mq(q)) + return blk_bio_poll(q, cookie, spin); + hctx = q->queue_hw_ctx[blk_qc_t_to_queue_num(cookie)]; /* diff --git a/block/blk.h b/block/blk.h index 3b53e44b967e..ae58a706327e 100644 --- a/block/blk.h +++ b/block/blk.h @@ -357,4 +357,50 @@ int bio_add_hw_page(struct request_queue *q, struct bio *bio, struct page *page, unsigned int len, unsigned int offset, unsigned int max_sectors, bool *same_page); +/* grouping bios belonging to same group into one list */ +struct bio_grp_list_data { + /* group data */ + void *grp_data; + + /* all bios in this list share same 'grp_data' */ + struct bio_list list; +}; + +struct bio_grp_list { + unsigned int max_nr_grps, nr_grps; + struct bio_grp_list_data head[0]; +}; + +struct blk_bio_poll_ctx { + spinlock_t sq_lock; + struct bio_grp_list *sq; + + struct mutex pq_lock; + struct bio_grp_list *pq; +}; + +#define BLK_BIO_POLL_SQ_SZ 32U +#define BLK_BIO_POLL_PQ_SZ (BLK_BIO_POLL_SQ_SZ * 2) + +void bio_poll_ctx_alloc(struct io_context *ioc); + +static inline void blk_create_io_context(struct request_queue *q, + bool need_poll_ctx) +{ + struct io_context *ioc; + + /* + * Various block parts want %current->io_context, so allocate it up + * front rather than dealing with lots of pain to allocate it only + * where needed. This may fail and the block layer knows how to live + * with it. + */ + if (unlikely(!current->io_context)) + create_task_io_context(current, GFP_ATOMIC, q->node); + + ioc = current->io_context; + if (need_poll_ctx && unlikely(ioc && !ioc->data)) + bio_poll_ctx_alloc(ioc); +} + #endif /* BLK_INTERNAL_H */ diff --git a/include/linux/iocontext.h b/include/linux/iocontext.h index 0a9dc40b7be8..f9a467571356 100644 --- a/include/linux/iocontext.h +++ b/include/linux/iocontext.h @@ -110,6 +110,8 @@ struct io_context { struct io_cq __rcu *icq_hint; struct hlist_head icq_list; + void *data; + struct work_struct release_work; }; From patchwork Thu Mar 18 16:48:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12148607 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21F27C433E6 for ; Thu, 18 Mar 2021 16:49:59 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B086E64F11 for ; Thu, 18 Mar 2021 16:49:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B086E64F11 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1616086197; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=kZiphOORTiJm6vhvubRN6CaPyvhBsOlM2TbtfQIvco0=; b=g8e1ePeWYTFSmFY2UK+wWgzDNc8qAZAufrqrxY3sdoSOkKcWzePEr1Qm/rxergXmdLCbcZ L92ZyA0wFg+de9o+91Z/sXqMx7noJNmAXjdpuTA3N4dEmMmQiHCKXRjADGk36w2ZkzYB4a AcjwNCjkHsfTJdkh2mNZJ0ueGUUD0+s= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-335-nQxnH_eCPMiAynpt5I2daw-1; Thu, 18 Mar 2021 12:49:55 -0400 X-MC-Unique: nQxnH_eCPMiAynpt5I2daw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C121D100945E; Thu, 18 Mar 2021 16:49:37 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A316F60CE6; Thu, 18 Mar 2021 16:49:37 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 6ED401809C83; Thu, 18 Mar 2021 16:49:37 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 12IGnSsi030520 for ; Thu, 18 Mar 2021 12:49:28 -0400 Received: by smtp.corp.redhat.com (Postfix) id 80AF310074FC; Thu, 18 Mar 2021 16:49:28 +0000 (UTC) Received: from localhost (ovpn-12-24.pek2.redhat.com [10.72.12.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8FC0A10023BE; Thu, 18 Mar 2021 16:49:27 +0000 (UTC) From: Ming Lei To: Jens Axboe Date: Fri, 19 Mar 2021 00:48:19 +0800 Message-Id: <20210318164827.1481133-6-ming.lei@redhat.com> In-Reply-To: <20210318164827.1481133-1-ming.lei@redhat.com> References: <20210318164827.1481133-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-loop: dm-devel@redhat.com Cc: Mike Snitzer , Ming Lei , linux-block@vger.kernel.org, dm-devel@redhat.com, Jeffle Xu , Christoph Hellwig Subject: [dm-devel] [RFC PATCH V2 05/13] block: add req flag of REQ_TAG X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Add one req flag REQ_TAG which will be used in the following patch for supporting bio based IO polling. Exactly this flag can help us to do: 1) request flag is cloned in bio_fast_clone(), so if we mark one FS bio as REQ_TAG, all bios cloned from this FS bio will be marked as REQ_TAG. 2)create per-task io polling context if the bio based queue supports polling and the submitted bio is HIPRI. This per-task io polling context will be created during submit_bio() before marking this HIPRI bio as REQ_TAG. Then we can avoid to create such io polling context if one cloned bio with REQ_TAG is submitted from another kernel context. 3) for supporting bio based io polling, we need to poll IOs from all underlying queues of bio device/driver, this way help us to recognize which IOs need to polled in bio based style, which will be implemented in next patch. Signed-off-by: Ming Lei --- block/blk-core.c | 29 +++++++++++++++++++++++++++-- include/linux/blk_types.h | 4 ++++ 2 files changed, 31 insertions(+), 2 deletions(-) diff --git a/block/blk-core.c b/block/blk-core.c index 0b00c21cbefb..efc7a61a84b4 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -840,11 +840,30 @@ static inline bool blk_queue_support_bio_poll(struct request_queue *q) static inline void blk_bio_poll_preprocess(struct request_queue *q, struct bio *bio) { + bool mq; + if (!(bio->bi_opf & REQ_HIPRI)) return; - if (!blk_queue_poll(q) || (!queue_is_mq(q) && !blk_get_bio_poll_ctx())) + /* + * Can't support bio based IO poll without per-task poll queue + * + * Now we have created per-task io poll context, and mark this + * bio as REQ_TAG, so: 1) if any cloned bio from this bio is + * submitted from another kernel context, we won't create bio + * poll context for it, so that bio will be completed by IRQ; + * 2) If such bio is submitted from current context, we will + * complete it via blk_poll(); 3) If driver knows that one + * underlying bio allocated from driver is for FS bio, meantime + * it is submitted in current context, driver can mark such bio + * as REQ_TAG manually, so the bio can be completed via blk_poll + * too. + */ + mq = queue_is_mq(q); + if (!blk_queue_poll(q) || (!mq && !blk_get_bio_poll_ctx())) bio->bi_opf &= ~REQ_HIPRI; + else if (!mq) + bio->bi_opf |= REQ_TAG; } static noinline_for_stack bool submit_bio_checks(struct bio *bio) @@ -893,9 +912,15 @@ static noinline_for_stack bool submit_bio_checks(struct bio *bio) /* * Created per-task io poll queue if we supports bio polling - * and it is one HIPRI bio. + * and it is one HIPRI bio, and this HIPRI bio has to be from + * FS. If REQ_TAG isn't set for HIPRI bio, we think it originated + * from FS. + * + * Driver may allocated bio by itself and REQ_TAG is set, but they + * won't be marked as HIPRI. */ blk_create_io_context(q, blk_queue_support_bio_poll(q) && + !(bio->bi_opf & REQ_TAG) && (bio->bi_opf & REQ_HIPRI)); blk_bio_poll_preprocess(q, bio); diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index db026b6ec15a..a1bcade4bcc3 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -394,6 +394,9 @@ enum req_flag_bits { __REQ_HIPRI, + /* for marking IOs originated from same FS bio in same context */ + __REQ_TAG, + /* for driver use */ __REQ_DRV, __REQ_SWAP, /* swapping request. */ @@ -418,6 +421,7 @@ enum req_flag_bits { #define REQ_NOUNMAP (1ULL << __REQ_NOUNMAP) #define REQ_HIPRI (1ULL << __REQ_HIPRI) +#define REQ_TAG (1ULL << __REQ_TAG) #define REQ_DRV (1ULL << __REQ_DRV) #define REQ_SWAP (1ULL << __REQ_SWAP) From patchwork Thu Mar 18 16:48:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12148669 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E204EC433E9 for ; Thu, 18 Mar 2021 16:54:06 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 867C764E07 for ; Thu, 18 Mar 2021 16:54:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 867C764E07 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1616086445; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=AAivrgbs8+NOQ08tQkaS6T1vORdaFOPPBYrGurAfoEo=; b=EzgPcwErD66cQaB7bOb5hMCAF/bxaOu/UuEWv8FA+jvyUJ4xmMHjp+yfpcjw/zqu5Wely5 5uaMr5figVUjPn0D9NDQ3gwkTPuVtXIjLEBSSszPuWwVF2txwHEXsH1Lc2IemhYawW3Dif 00gBw90kfKji4+FhqY2ZcGBlN43TrO0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-121-cAC2-BIrO7CXkzwa8za7qA-1; Thu, 18 Mar 2021 12:53:53 -0400 X-MC-Unique: cAC2-BIrO7CXkzwa8za7qA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2B79B180FCAA; Thu, 18 Mar 2021 16:53:48 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 0D85060938; Thu, 18 Mar 2021 16:53:48 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id CAAB21809C84; Thu, 18 Mar 2021 16:53:47 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 12IGnVD7030534 for ; Thu, 18 Mar 2021 12:49:31 -0400 Received: by smtp.corp.redhat.com (Postfix) id 7DCEC5D9CA; Thu, 18 Mar 2021 16:49:31 +0000 (UTC) Received: from localhost (ovpn-12-24.pek2.redhat.com [10.72.12.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 754FC5D9C6; Thu, 18 Mar 2021 16:49:30 +0000 (UTC) From: Ming Lei To: Jens Axboe Date: Fri, 19 Mar 2021 00:48:20 +0800 Message-Id: <20210318164827.1481133-7-ming.lei@redhat.com> In-Reply-To: <20210318164827.1481133-1-ming.lei@redhat.com> References: <20210318164827.1481133-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-loop: dm-devel@redhat.com Cc: Mike Snitzer , Ming Lei , linux-block@vger.kernel.org, dm-devel@redhat.com, Jeffle Xu , Christoph Hellwig Subject: [dm-devel] [RFC PATCH V2 06/13] block: add new field into 'struct bvec_iter' X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com There is a hole at the end of 'struct bvec_iter', so put a new field here and we can save cookie returned from submit_bio() here for supporting bio based polling. This way can avoid to extend bio unnecessarily. Signed-off-by: Ming Lei --- include/linux/bvec.h | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/include/linux/bvec.h b/include/linux/bvec.h index ff832e698efb..61c0f55f7165 100644 --- a/include/linux/bvec.h +++ b/include/linux/bvec.h @@ -43,6 +43,15 @@ struct bvec_iter { unsigned int bi_bvec_done; /* number of bytes completed in current bvec */ + + /* + * There is a hole at the end of bvec_iter, define one filed to + * hold something which isn't relate with 'bvec_iter', so that we can + * avoid to extend bio. So far this new field is used for bio based + * pooling, we will store returning value of underlying queue's + * submit_bio() here. + */ + unsigned int bi_private_data; }; struct bvec_iter_all { From patchwork Thu Mar 18 16:48:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12148663 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0BDBC433E0 for ; Thu, 18 Mar 2021 16:54:00 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 892A564E07 for ; Thu, 18 Mar 2021 16:54:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 892A564E07 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1616086439; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=2xpu5nczlM4aCvnuQvkZY9HDS4GgOXa4DqYwaWSzr38=; b=a+Z+osioIPDIDzXOdmF3fTGfYKzn51MoznpzQepOfKz3XX/y2rH9/6YFAv+Bg9COjdWO0Z tRkRWMFsCox0454CXYt0tXQpr1kQpSdlBkdNqZew+TDcJZEXvMne9n0/ZlT0CqodmdgjWI +n3E5kMAxHG/lkB7CXHv+sdKlRuur7U= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-554-AKtVhi5GNQCrqFOEAmBpqA-1; Thu, 18 Mar 2021 12:53:57 -0400 X-MC-Unique: AKtVhi5GNQCrqFOEAmBpqA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 094F08189CA; Thu, 18 Mar 2021 16:53:51 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id DA78810027A5; Thu, 18 Mar 2021 16:53:50 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 998544BB7C; Thu, 18 Mar 2021 16:53:50 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 12IGnZlT030552 for ; Thu, 18 Mar 2021 12:49:35 -0400 Received: by smtp.corp.redhat.com (Postfix) id 1AF195D9D0; Thu, 18 Mar 2021 16:49:35 +0000 (UTC) Received: from localhost (ovpn-12-24.pek2.redhat.com [10.72.12.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 371DE5D9C6; Thu, 18 Mar 2021 16:49:33 +0000 (UTC) From: Ming Lei To: Jens Axboe Date: Fri, 19 Mar 2021 00:48:21 +0800 Message-Id: <20210318164827.1481133-8-ming.lei@redhat.com> In-Reply-To: <20210318164827.1481133-1-ming.lei@redhat.com> References: <20210318164827.1481133-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-loop: dm-devel@redhat.com Cc: Mike Snitzer , Ming Lei , linux-block@vger.kernel.org, dm-devel@redhat.com, Jeffle Xu , Christoph Hellwig Subject: [dm-devel] [RFC PATCH V2 07/13] block/mq: extract one helper function polling hw queue X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com From: Jeffle Xu Extract the logic of polling one hw queue and related statistics handling out as the helper function. Signed-off-by: Jeffle Xu Signed-off-by: Ming Lei --- block/blk-mq.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index c832faa52ca0..03f59915fe2c 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -3852,6 +3852,19 @@ static bool blk_mq_poll_hybrid(struct request_queue *q, return blk_mq_poll_hybrid_sleep(q, rq); } +static inline int blk_mq_poll_hctx(struct request_queue *q, + struct blk_mq_hw_ctx *hctx) +{ + int ret; + + hctx->poll_invoked++; + ret = q->mq_ops->poll(hctx); + if (ret > 0) + hctx->poll_success++; + + return ret; +} + static int blk_bio_poll(struct request_queue *q, blk_qc_t cookie, bool spin) { /* @@ -3908,11 +3921,8 @@ int blk_poll(struct request_queue *q, blk_qc_t cookie, bool spin) do { int ret; - hctx->poll_invoked++; - - ret = q->mq_ops->poll(hctx); + ret = blk_mq_poll_hctx(q, hctx); if (ret > 0) { - hctx->poll_success++; __set_current_state(TASK_RUNNING); return ret; } From patchwork Thu Mar 18 16:48:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12148605 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83E56C433E0 for ; Thu, 18 Mar 2021 16:49:57 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 33F5764ED2 for ; Thu, 18 Mar 2021 16:49:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 33F5764ED2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1616086196; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=zMxll5BfnZjaq8QVu/mkaXq0RAiSbwtwMX8CrzlHFg8=; b=eUnv0pOX3Dv9QIU9lDp9DW7PVr0Olwua4wlqM7eXuKNqJLTYWqCFiVeiwpp0zM1JL93+nt IlqZK08UidN0KKtITLzhI4vQs7dtz0Ujh9lmZwPWUx4HUsKp5I06VO/pLPgJAWQDkYZAy1 mtNcQ0JmkM8SxIrseV+OoPGblmoM398= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-554-VY_RSsVrNqS3fIm9xFWCdw-1; Thu, 18 Mar 2021 12:49:54 -0400 X-MC-Unique: VY_RSsVrNqS3fIm9xFWCdw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5A8DD107B05B; Thu, 18 Mar 2021 16:49:40 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 3AB856064B; Thu, 18 Mar 2021 16:49:40 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 0623C4BB7C; Thu, 18 Mar 2021 16:49:40 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 12IGncFW030569 for ; Thu, 18 Mar 2021 12:49:38 -0400 Received: by smtp.corp.redhat.com (Postfix) id 2399F50415; Thu, 18 Mar 2021 16:49:38 +0000 (UTC) Received: from localhost (ovpn-12-24.pek2.redhat.com [10.72.12.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 632A937DD; Thu, 18 Mar 2021 16:49:36 +0000 (UTC) From: Ming Lei To: Jens Axboe Date: Fri, 19 Mar 2021 00:48:22 +0800 Message-Id: <20210318164827.1481133-9-ming.lei@redhat.com> In-Reply-To: <20210318164827.1481133-1-ming.lei@redhat.com> References: <20210318164827.1481133-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-loop: dm-devel@redhat.com Cc: Mike Snitzer , Ming Lei , linux-block@vger.kernel.org, dm-devel@redhat.com, Jeffle Xu , Christoph Hellwig Subject: [dm-devel] [RFC PATCH V2 08/13] block: prepare for supporting bio_list via other link X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com So far bio list helpers always use .bi_next to traverse the list, we will support to link bios by other bio field. Prepare for such support by adding a macro so that users can define another helpers for linking bios by other bio field. Signed-off-by: Ming Lei --- include/linux/bio.h | 132 +++++++++++++++++++++++--------------------- 1 file changed, 68 insertions(+), 64 deletions(-) diff --git a/include/linux/bio.h b/include/linux/bio.h index d0246c92a6e8..619edd26a6c0 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -608,75 +608,11 @@ static inline unsigned bio_list_size(const struct bio_list *bl) return sz; } -static inline void bio_list_add(struct bio_list *bl, struct bio *bio) -{ - bio->bi_next = NULL; - - if (bl->tail) - bl->tail->bi_next = bio; - else - bl->head = bio; - - bl->tail = bio; -} - -static inline void bio_list_add_head(struct bio_list *bl, struct bio *bio) -{ - bio->bi_next = bl->head; - - bl->head = bio; - - if (!bl->tail) - bl->tail = bio; -} - -static inline void bio_list_merge(struct bio_list *bl, struct bio_list *bl2) -{ - if (!bl2->head) - return; - - if (bl->tail) - bl->tail->bi_next = bl2->head; - else - bl->head = bl2->head; - - bl->tail = bl2->tail; -} - -static inline void bio_list_merge_head(struct bio_list *bl, - struct bio_list *bl2) -{ - if (!bl2->head) - return; - - if (bl->head) - bl2->tail->bi_next = bl->head; - else - bl->tail = bl2->tail; - - bl->head = bl2->head; -} - static inline struct bio *bio_list_peek(struct bio_list *bl) { return bl->head; } -static inline struct bio *bio_list_pop(struct bio_list *bl) -{ - struct bio *bio = bl->head; - - if (bio) { - bl->head = bl->head->bi_next; - if (!bl->head) - bl->tail = NULL; - - bio->bi_next = NULL; - } - - return bio; -} - static inline struct bio *bio_list_get(struct bio_list *bl) { struct bio *bio = bl->head; @@ -686,6 +622,74 @@ static inline struct bio *bio_list_get(struct bio_list *bl) return bio; } +#define BIO_LIST_HELPERS(_pre, link) \ + \ +static inline void _pre##_add(struct bio_list *bl, struct bio *bio) \ +{ \ + bio->bi_##link = NULL; \ + \ + if (bl->tail) \ + bl->tail->bi_##link = bio; \ + else \ + bl->head = bio; \ + \ + bl->tail = bio; \ +} \ + \ +static inline void _pre##_add_head(struct bio_list *bl, struct bio *bio) \ +{ \ + bio->bi_##link = bl->head; \ + \ + bl->head = bio; \ + \ + if (!bl->tail) \ + bl->tail = bio; \ +} \ + \ +static inline void _pre##_merge(struct bio_list *bl, struct bio_list *bl2) \ +{ \ + if (!bl2->head) \ + return; \ + \ + if (bl->tail) \ + bl->tail->bi_##link = bl2->head; \ + else \ + bl->head = bl2->head; \ + \ + bl->tail = bl2->tail; \ +} \ + \ +static inline void _pre##_merge_head(struct bio_list *bl, \ + struct bio_list *bl2) \ +{ \ + if (!bl2->head) \ + return; \ + \ + if (bl->head) \ + bl2->tail->bi_##link = bl->head; \ + else \ + bl->tail = bl2->tail; \ + \ + bl->head = bl2->head; \ +} \ + \ +static inline struct bio *_pre##_pop(struct bio_list *bl) \ +{ \ + struct bio *bio = bl->head; \ + \ + if (bio) { \ + bl->head = bl->head->bi_##link; \ + if (!bl->head) \ + bl->tail = NULL; \ + \ + bio->bi_##link = NULL; \ + } \ + \ + return bio; \ +} \ + +BIO_LIST_HELPERS(bio_list, next); + /* * Increment chain count for the bio. Make sure the CHAIN flag update * is visible before the raised count. From patchwork Thu Mar 18 16:48:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12148665 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7483C433DB for ; Thu, 18 Mar 2021 16:54:01 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7A65E64E07 for ; Thu, 18 Mar 2021 16:54:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7A65E64E07 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1616086440; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=m0hUSbDds7H0nRHvInZdePwFa4Z6bwp+Lk8tdkQKQ9A=; b=QdUWQWVXyAP/Ox8ww+CnNGjGaSF2b2N7qkMU318Uranp5eFUE0Hx7nTbdynL+/LW/vnDdE k+sIRHw1/KJPHx7905cKFLZsopdRuiuxUmv95Miz011V6H6auD19DmzsNRbjDjanrcTrP1 JYbvLPpW2ZW9Tfi51ZSe9PymdPQ16CA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-556-FyqxjarHP5mI6BgD6z5egw-1; Thu, 18 Mar 2021 12:53:57 -0400 X-MC-Unique: FyqxjarHP5mI6BgD6z5egw-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7AC341009471; Thu, 18 Mar 2021 16:53:53 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 4C59C19D9B; Thu, 18 Mar 2021 16:53:53 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 0A8784A7C9; Thu, 18 Mar 2021 16:53:53 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 12IGngKS030591 for ; Thu, 18 Mar 2021 12:49:42 -0400 Received: by smtp.corp.redhat.com (Postfix) id 0D24010074FC; Thu, 18 Mar 2021 16:49:42 +0000 (UTC) Received: from localhost (ovpn-12-24.pek2.redhat.com [10.72.12.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id C6F3110027A5; Thu, 18 Mar 2021 16:49:40 +0000 (UTC) From: Ming Lei To: Jens Axboe Date: Fri, 19 Mar 2021 00:48:23 +0800 Message-Id: <20210318164827.1481133-10-ming.lei@redhat.com> In-Reply-To: <20210318164827.1481133-1-ming.lei@redhat.com> References: <20210318164827.1481133-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-loop: dm-devel@redhat.com Cc: Mike Snitzer , Ming Lei , linux-block@vger.kernel.org, dm-devel@redhat.com, Jeffle Xu , Christoph Hellwig Subject: [dm-devel] [RFC PATCH V2 09/13] block: use per-task poll context to implement bio based io poll X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Currently bio based IO poll needs to poll all hw queue blindly, this way is very inefficient, and the big reason is that we can't pass bio submission result to io poll task. In IO submission context, track associated underlying bios by per-task submission queue and save 'cookie' poll data in bio->bi_iter.bi_private_data, and return current->pid to caller of submit_bio() for any bio based driver's IO, which is submitted from FS. In IO poll context, the passed cookie tells us the PID of submission context, and we can find the bio from that submission context. Moving bio from submission queue to poll queue of the poll context, and keep polling until these bios are ended. Remove bio from poll queue if the bio is ended. Add BIO_DONE and BIO_END_BY_POLL for such purpose. In previous version, kfifo is used to implement submission queue, and Jeffle Xu found that kfifo can't scale well in case of high queue depth. So far bio's size is close to 2 cacheline size, and it may not be accepted to add new field into bio for solving the scalability issue by tracking bios via linked list, switch to bio group list for tracking bio, the idea is to reuse .bi_end_io for linking bios into a linked list for all sharing same .bi_end_io(call it bio group), which is recovered before really end bio, since BIO_END_BY_POLL is added for enhancing this point. Usually .bi_end_bio is same for all bios in same layer, so it is enough to provide very limited groups, such as 32 for fixing the scalability issue. Usually submission shares context with io poll. The per-task poll context is just like stack variable, and it is cheap to move data between the two per-task queues. Signed-off-by: Ming Lei --- block/bio.c | 5 ++ block/blk-core.c | 149 +++++++++++++++++++++++++++++++- block/blk-mq.c | 173 +++++++++++++++++++++++++++++++++++++- block/blk.h | 9 ++ include/linux/blk_types.h | 16 +++- 5 files changed, 348 insertions(+), 4 deletions(-) diff --git a/block/bio.c b/block/bio.c index 26b7f721cda8..04c043dc60fc 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1402,6 +1402,11 @@ static inline bool bio_remaining_done(struct bio *bio) **/ void bio_endio(struct bio *bio) { + /* BIO_END_BY_POLL has to be set before calling submit_bio */ + if (bio_flagged(bio, BIO_END_BY_POLL)) { + bio_set_flag(bio, BIO_DONE); + return; + } again: if (!bio_remaining_done(bio)) return; diff --git a/block/blk-core.c b/block/blk-core.c index efc7a61a84b4..778d25a7e76c 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -805,6 +805,77 @@ static inline unsigned int bio_grp_list_size(unsigned int nr_grps) sizeof(struct bio_grp_list_data); } +static inline void *bio_grp_data(struct bio *bio) +{ + return bio->bi_poll; +} + +/* add bio into bio group list, return true if it is added */ +static bool bio_grp_list_add(struct bio_grp_list *list, struct bio *bio) +{ + int i; + struct bio_grp_list_data *grp; + + for (i = 0; i < list->nr_grps; i++) { + grp = &list->head[i]; + if (grp->grp_data == bio_grp_data(bio)) { + __bio_grp_list_add(&grp->list, bio); + return true; + } + } + + if (i == list->max_nr_grps) + return false; + + /* create a new group */ + grp = &list->head[i]; + bio_list_init(&grp->list); + grp->grp_data = bio_grp_data(bio); + __bio_grp_list_add(&grp->list, bio); + list->nr_grps++; + + return true; +} + +static int bio_grp_list_find_grp(struct bio_grp_list *list, void *grp_data) +{ + int i; + struct bio_grp_list_data *grp; + + for (i = 0; i < list->max_nr_grps; i++) { + grp = &list->head[i]; + if (grp->grp_data == grp_data) + return i; + } + for (i = 0; i < list->max_nr_grps; i++) { + grp = &list->head[i]; + if (bio_grp_list_grp_empty(grp)) + return i; + } + return -1; +} + +/* Move as many as possible groups from 'src' to 'dst' */ +void bio_grp_list_move(struct bio_grp_list *dst, struct bio_grp_list *src) +{ + int i, j, cnt = 0; + struct bio_grp_list_data *grp; + + for (i = src->nr_grps - 1; i >= 0; i--) { + grp = &src->head[i]; + j = bio_grp_list_find_grp(dst, grp->grp_data); + if (j < 0) + break; + if (bio_grp_list_grp_empty(&dst->head[j])) + dst->head[j].grp_data = grp->grp_data; + __bio_grp_list_merge(&dst->head[j].list, &grp->list); + bio_list_init(&grp->list); + cnt++; + } + + src->nr_grps -= cnt; +} + static void bio_poll_ctx_init(struct blk_bio_poll_ctx *pc) { pc->sq = (void *)pc + sizeof(*pc); @@ -866,6 +937,46 @@ static inline void blk_bio_poll_preprocess(struct request_queue *q, bio->bi_opf |= REQ_TAG; } +static bool blk_bio_poll_prep_submit(struct io_context *ioc, struct bio *bio) +{ + struct blk_bio_poll_ctx *pc = ioc->data; + unsigned int queued; + + /* + * We rely on immutable .bi_end_io between blk-mq bio submission + * and completion. However, bio crypt may update .bi_end_io during + * submitting, so simply not support bio based polling for this + * setting. + */ + if (likely(!bio_has_crypt_ctx(bio))) { + /* track this bio via bio group list */ + spin_lock(&pc->sq_lock); + queued = bio_grp_list_add(pc->sq, bio); + spin_unlock(&pc->sq_lock); + } else { + queued = false; + } + + /* + * Now the bio is added per-task fifo, mark it as END_BY_POLL, + * and the bio is always completed from the pair poll context. + * + * One invariant is that if bio isn't completed, blk_poll() will + * be called by passing cookie returned from submitting this bio. + */ + if (!queued) + bio->bi_opf &= ~(REQ_HIPRI | REQ_TAG); + else + bio_set_flag(bio, BIO_END_BY_POLL); + + return queued; +} + +static void blk_bio_poll_post_submit(struct bio *bio, blk_qc_t cookie) +{ + bio->bi_iter.bi_private_data = cookie; +} + static noinline_for_stack bool submit_bio_checks(struct bio *bio) { struct block_device *bdev = bio->bi_bdev; @@ -1020,7 +1131,7 @@ static blk_qc_t __submit_bio(struct bio *bio) * bio_list_on_stack[1] contains bios that were submitted before the current * ->submit_bio_bio, but that haven't been processed yet. */ -static blk_qc_t __submit_bio_noacct(struct bio *bio) +static blk_qc_t __submit_bio_noacct_int(struct bio *bio, struct io_context *ioc) { struct bio_list bio_list_on_stack[2]; blk_qc_t ret = BLK_QC_T_NONE; @@ -1043,7 +1154,16 @@ static blk_qc_t __submit_bio_noacct(struct bio *bio) bio_list_on_stack[1] = bio_list_on_stack[0]; bio_list_init(&bio_list_on_stack[0]); - ret = __submit_bio(bio); + if (ioc && queue_is_mq(q) && + (bio->bi_opf & (REQ_HIPRI | REQ_TAG))) { + bool queued = blk_bio_poll_prep_submit(ioc, bio); + + ret = __submit_bio(bio); + if (queued) + blk_bio_poll_post_submit(bio, ret); + } else { + ret = __submit_bio(bio); + } /* * Sort new bios into those for a lower level and those for the @@ -1069,6 +1189,31 @@ static blk_qc_t __submit_bio_noacct(struct bio *bio) return ret; } +static inline blk_qc_t __submit_bio_noacct_poll(struct bio *bio, + struct io_context *ioc) +{ + struct blk_bio_poll_ctx *pc = ioc->data; + + __submit_bio_noacct_int(bio, ioc); + + /* bio submissions queued to per-task poll context */ + if (READ_ONCE(pc->sq->nr_grps)) + return current->pid; + + /* swapper's pid is 0, but it can't submit poll IO for us */ + return 0; +} + +static inline blk_qc_t __submit_bio_noacct(struct bio *bio) +{ + struct io_context *ioc = current->io_context; + + if (ioc && ioc->data && (bio->bi_opf & REQ_HIPRI)) + return __submit_bio_noacct_poll(bio, ioc); + + return __submit_bio_noacct_int(bio, NULL); +} + static blk_qc_t __submit_bio_noacct_mq(struct bio *bio) { struct bio_list bio_list[2] = { }; diff --git a/block/blk-mq.c b/block/blk-mq.c index 03f59915fe2c..f26950a51f4a 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -3865,14 +3865,185 @@ static inline int blk_mq_poll_hctx(struct request_queue *q, return ret; } +static blk_qc_t bio_get_poll_cookie(struct bio *bio) +{ + return bio->bi_iter.bi_private_data; +} + +static int blk_mq_poll_io(struct bio *bio) +{ + struct request_queue *q = bio->bi_bdev->bd_disk->queue; + blk_qc_t cookie = bio_get_poll_cookie(bio); + int ret = 0; + + if (!bio_flagged(bio, BIO_DONE) && blk_qc_t_valid(cookie)) { + struct blk_mq_hw_ctx *hctx = + q->queue_hw_ctx[blk_qc_t_to_queue_num(cookie)]; + + ret += blk_mq_poll_hctx(q, hctx); + } + return ret; +} + +static int blk_bio_poll_and_end_io(struct request_queue *q, + struct blk_bio_poll_ctx *poll_ctx) +{ + int ret = 0; + int i; + + /* + * Poll hw queue first. + * + * TODO: limit max poll times and make sure to not poll same + * hw queue one more time. + */ + for (i = 0; i < poll_ctx->pq->max_nr_grps; i++) { + struct bio_grp_list_data *grp = &poll_ctx->pq->head[i]; + struct bio *bio; + + if (bio_grp_list_grp_empty(grp)) + continue; + + for (bio = grp->list.head; bio; bio = bio->bi_poll) + ret += blk_mq_poll_io(bio); + } + + /* reap bios */ + for (i = 0; i < poll_ctx->pq->max_nr_grps; i++) { + struct bio_grp_list_data *grp = &poll_ctx->pq->head[i]; + struct bio *bio; + struct bio_list bl; + + if (bio_grp_list_grp_empty(grp)) + continue; + + bio_list_init(&bl); + + while ((bio = __bio_grp_list_pop(&grp->list))) { + if (bio_flagged(bio, BIO_DONE)) { + + /* now recover original data */ + bio->bi_poll = grp->grp_data; + + /* clear BIO_END_BY_POLL and end me really */ + bio_clear_flag(bio, BIO_END_BY_POLL); + bio_endio(bio); + } else { + __bio_grp_list_add(&bl, bio); + } + } + __bio_grp_list_merge(&grp->list, &bl); + } + return ret; +} + +static int __blk_bio_poll_io(struct request_queue *q, + struct blk_bio_poll_ctx *submit_ctx, + struct blk_bio_poll_ctx *poll_ctx) +{ + /* + * Move IO submission result from submission queue in submission + * context to poll queue of poll context. + */ + spin_lock(&submit_ctx->sq_lock); + bio_grp_list_move(poll_ctx->pq, submit_ctx->sq); + spin_unlock(&submit_ctx->sq_lock); + + return blk_bio_poll_and_end_io(q, poll_ctx); +} + +static int blk_bio_poll_io(struct request_queue *q, + struct io_context *submit_ioc, + struct io_context *poll_ioc) +{ + struct blk_bio_poll_ctx *submit_ctx = submit_ioc->data; + struct blk_bio_poll_ctx *poll_ctx = poll_ioc->data; + int ret; + + if (unlikely(atomic_read(&poll_ioc->nr_tasks) > 1)) { + mutex_lock(&poll_ctx->pq_lock); + ret = __blk_bio_poll_io(q, submit_ctx, poll_ctx); + mutex_unlock(&poll_ctx->pq_lock); + } else { + ret = __blk_bio_poll_io(q, submit_ctx, poll_ctx); + } + return ret; +} + +static bool blk_bio_ioc_valid(struct task_struct *t) +{ + if (!t) + return false; + + if (!t->io_context) + return false; + + if (!t->io_context->data) + return false; + + return true; +} + +static int __blk_bio_poll(struct request_queue *q, blk_qc_t cookie) +{ + struct io_context *poll_ioc = current->io_context; + pid_t pid; + struct task_struct *submit_task; + int ret; + + pid = (pid_t)cookie; + + /* io poll often share io submission context */ + if (likely(current->pid == pid && blk_bio_ioc_valid(current))) + return blk_bio_poll_io(q, poll_ioc, poll_ioc); + + submit_task = find_get_task_by_vpid(pid); + if (likely(blk_bio_ioc_valid(submit_task))) + ret = blk_bio_poll_io(q, submit_task->io_context, + poll_ioc); + else + ret = 0; + + put_task_struct(submit_task); + + return ret; +} + static int blk_bio_poll(struct request_queue *q, blk_qc_t cookie, bool spin) { + long state; + + /* no need to poll */ + if (cookie == 0) + return 0; + /* * Create poll queue for storing poll bio and its cookie from * submission queue */ blk_create_io_context(q, true); + state = current->state; + do { + int ret; + + ret = __blk_bio_poll(q, cookie); + if (ret > 0) { + __set_current_state(TASK_RUNNING); + return ret; + } + + if (signal_pending_state(state, current)) + __set_current_state(TASK_RUNNING); + + if (current->state == TASK_RUNNING) + return 1; + if (ret < 0 || !spin) + break; + cpu_relax(); + } while (!need_resched()); + + __set_current_state(TASK_RUNNING); return 0; } @@ -3893,7 +4064,7 @@ int blk_poll(struct request_queue *q, blk_qc_t cookie, bool spin) struct blk_mq_hw_ctx *hctx; long state; - if (!blk_qc_t_valid(cookie) || !blk_queue_poll(q)) + if (!blk_queue_poll(q) || (queue_is_mq(q) && !blk_qc_t_valid(cookie))) return 0; if (current->plug) diff --git a/block/blk.h b/block/blk.h index ae58a706327e..05b9f5eafdd1 100644 --- a/block/blk.h +++ b/block/blk.h @@ -403,4 +403,13 @@ static inline void blk_create_io_context(struct request_queue *q, bio_poll_ctx_alloc(ioc); } +BIO_LIST_HELPERS(__bio_grp_list, poll); + +static inline bool bio_grp_list_grp_empty(struct bio_grp_list_data *grp) +{ + return bio_list_empty(&grp->list); +} + +void bio_grp_list_move(struct bio_grp_list *dst, struct bio_grp_list *src); + #endif /* BLK_INTERNAL_H */ diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index a1bcade4bcc3..2d47679bac71 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -235,7 +235,18 @@ struct bio { struct bvec_iter bi_iter; - bio_end_io_t *bi_end_io; + union { + bio_end_io_t *bi_end_io; + /* + * bio based io poll need to track bio via bio group list + * which groups bio by same .bi_end_io, and original + * .bi_end_io is save into the group head. Will recover + * .bi_end_io before end this bio really. BIO_END_BY_POLL + * will make sure that this bio won't be really ended + * before recovering .bi_end_io. + */ + struct bio *bi_poll; + }; void *bi_private; #ifdef CONFIG_BLK_CGROUP @@ -304,6 +315,9 @@ enum { BIO_CGROUP_ACCT, /* has been accounted to a cgroup */ BIO_TRACKED, /* set if bio goes through the rq_qos path */ BIO_REMAPPED, + BIO_END_BY_POLL, /* end by blk_bio_poll() explicitly */ + /* set when bio can be ended, used for bio with BIO_END_BY_POLL */ + BIO_DONE, BIO_FLAG_LAST }; From patchwork Thu Mar 18 16:48:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12148609 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA54CC433DB for ; Thu, 18 Mar 2021 16:50:00 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 969D264F1D for ; Thu, 18 Mar 2021 16:50:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 969D264F1D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1616086199; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=j8fMGCtiOW1I69yVN/xM1CBaS5ZRku5D6qBVML7cQho=; b=AzZqqLVvUOzWo46TkUEMN80Wwmnf6k/sKo8yv7IQOIkFsy+tdV5Yu1aF0EnK/s+68l7L1C FI1c1QrTeJhGUWn+40ENEHYCqNDcVi+E9L56v5LpYGYWQAuW0bG4pBDdZx+D1oAVZG9aUm F9WMYXCyNjkvWgqETvEo3oWe0KYDyxg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-604-oVDfxhkrM8KlcWvW5jsDWA-1; Thu, 18 Mar 2021 12:49:57 -0400 X-MC-Unique: oVDfxhkrM8KlcWvW5jsDWA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D444C107ACCD; Thu, 18 Mar 2021 16:49:52 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A94A937DD; Thu, 18 Mar 2021 16:49:52 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 5BA8E4BB7C; Thu, 18 Mar 2021 16:49:52 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 12IGnpEg030629 for ; Thu, 18 Mar 2021 12:49:51 -0400 Received: by smtp.corp.redhat.com (Postfix) id 262095D9D0; Thu, 18 Mar 2021 16:49:51 +0000 (UTC) Received: from localhost (ovpn-12-24.pek2.redhat.com [10.72.12.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2C11C5D9C6; Thu, 18 Mar 2021 16:49:43 +0000 (UTC) From: Ming Lei To: Jens Axboe Date: Fri, 19 Mar 2021 00:48:24 +0800 Message-Id: <20210318164827.1481133-11-ming.lei@redhat.com> In-Reply-To: <20210318164827.1481133-1-ming.lei@redhat.com> References: <20210318164827.1481133-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-loop: dm-devel@redhat.com Cc: linux-block@vger.kernel.org, Jeffle Xu , dm-devel@redhat.com, Christoph Hellwig , Mike Snitzer Subject: [dm-devel] [RFC PATCH V2 10/13] block: add queue_to_disk() to get gendisk from request_queue X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com From: Jeffle Xu Sometimes we need to get the corresponding gendisk from request_queue. It is preferred that block drivers store private data in gendisk->private_data rather than request_queue->queuedata, e.g. see: commit c4a59c4e5db3 ("dm: stop using ->queuedata"). So if only request_queue is given, we need to get its corresponding gendisk to get the private data stored in that gendisk. Signed-off-by: Jeffle Xu Reviewed-by: Mike Snitzer --- include/linux/blkdev.h | 2 ++ include/trace/events/kyber.h | 6 +++--- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 89a01850cf12..bfab74b45f15 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -686,6 +686,8 @@ static inline bool blk_account_rq(struct request *rq) dma_map_page_attrs(dev, (bv)->bv_page, (bv)->bv_offset, (bv)->bv_len, \ (dir), (attrs)) +#define queue_to_disk(q) (dev_to_disk(kobj_to_dev((q)->kobj.parent))) + static inline bool queue_is_mq(struct request_queue *q) { return q->mq_ops; diff --git a/include/trace/events/kyber.h b/include/trace/events/kyber.h index c0e7d24ca256..f9802562edf6 100644 --- a/include/trace/events/kyber.h +++ b/include/trace/events/kyber.h @@ -30,7 +30,7 @@ TRACE_EVENT(kyber_latency, ), TP_fast_assign( - __entry->dev = disk_devt(dev_to_disk(kobj_to_dev(q->kobj.parent))); + __entry->dev = disk_devt(queue_to_disk(q)); strlcpy(__entry->domain, domain, sizeof(__entry->domain)); strlcpy(__entry->type, type, sizeof(__entry->type)); __entry->percentile = percentile; @@ -59,7 +59,7 @@ TRACE_EVENT(kyber_adjust, ), TP_fast_assign( - __entry->dev = disk_devt(dev_to_disk(kobj_to_dev(q->kobj.parent))); + __entry->dev = disk_devt(queue_to_disk(q)); strlcpy(__entry->domain, domain, sizeof(__entry->domain)); __entry->depth = depth; ), @@ -81,7 +81,7 @@ TRACE_EVENT(kyber_throttled, ), TP_fast_assign( - __entry->dev = disk_devt(dev_to_disk(kobj_to_dev(q->kobj.parent))); + __entry->dev = disk_devt(queue_to_disk(q)); strlcpy(__entry->domain, domain, sizeof(__entry->domain)); ), From patchwork Thu Mar 18 16:48:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12148611 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7953C433E0 for ; Thu, 18 Mar 2021 16:50:06 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7B2D764F11 for ; Thu, 18 Mar 2021 16:50:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7B2D764F11 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1616086205; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=j/15/r0K9wiCAgIZrfL/7IyYJDlC5jCHr7qjYC2y/PY=; b=F3EjJglOITjvoNypweL6ZV9v6M9iePS+9wwwMYVMGrXpo0wNdFMfwxpc4Cj8JEhDX2x87t nZsUyWaBxcc9YM/m3jV+hcf9Ptt8wXY4OSCsvqV8Toe/RH2uVKQxGggf2wyO+fc1we5fgN ww4fLJWpRHxQsSllYziSj/hr6aWjb60= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-587-fyPLjxUUP8ubug_XXloBCA-1; Thu, 18 Mar 2021 12:50:02 -0400 X-MC-Unique: fyPLjxUUP8ubug_XXloBCA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C529018C8C00; Thu, 18 Mar 2021 16:49:56 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A083150415; Thu, 18 Mar 2021 16:49:56 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 6743A4A7C8; Thu, 18 Mar 2021 16:49:56 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 12IGnscc030658 for ; Thu, 18 Mar 2021 12:49:54 -0400 Received: by smtp.corp.redhat.com (Postfix) id 3FE8310023BE; Thu, 18 Mar 2021 16:49:54 +0000 (UTC) Received: from localhost (ovpn-12-24.pek2.redhat.com [10.72.12.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8FEF410027A5; Thu, 18 Mar 2021 16:49:53 +0000 (UTC) From: Ming Lei To: Jens Axboe Date: Fri, 19 Mar 2021 00:48:25 +0800 Message-Id: <20210318164827.1481133-12-ming.lei@redhat.com> In-Reply-To: <20210318164827.1481133-1-ming.lei@redhat.com> References: <20210318164827.1481133-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-loop: dm-devel@redhat.com Cc: linux-block@vger.kernel.org, Jeffle Xu , dm-devel@redhat.com, Christoph Hellwig , Mike Snitzer Subject: [dm-devel] [RFC PATCH V2 11/13] block: add poll_capable method to support bio-based IO polling X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com From: Jeffle Xu This method can be used to check if bio-based device supports IO polling or not. For mq devices, checking for hw queue in polling mode is adequate, while the sanity check shall be implementation specific for bio-based devices. For example, dm device needs to check if all underlying devices are capable of IO polling. Though bio-based device may have done the sanity check during the device initialization phase, cacheing the result of this sanity check (such as by cacheing in the queue_flags) may not work. Because for dm devices, users could change the state of the underlying devices through '/sys/block//io_poll', bypassing the dm device above. In this case, the cached result of the very beginning sanity check could be out-of-date. Thus the sanity check needs to be done every time 'io_poll' is to be modified. Signed-off-by: Jeffle Xu --- block/blk-sysfs.c | 14 +++++++++++--- include/linux/blkdev.h | 1 + 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c index 0f4f0c8a7825..367c1d9a55c6 100644 --- a/block/blk-sysfs.c +++ b/block/blk-sysfs.c @@ -426,9 +426,17 @@ static ssize_t queue_poll_store(struct request_queue *q, const char *page, unsigned long poll_on; ssize_t ret; - if (!q->tag_set || q->tag_set->nr_maps <= HCTX_TYPE_POLL || - !q->tag_set->map[HCTX_TYPE_POLL].nr_queues) - return -EINVAL; + if (queue_is_mq(q)) { + if (!q->tag_set || q->tag_set->nr_maps <= HCTX_TYPE_POLL || + !q->tag_set->map[HCTX_TYPE_POLL].nr_queues) + return -EINVAL; + } else { + struct gendisk *disk = queue_to_disk(q); + + if (!disk->fops->poll_capable || + !disk->fops->poll_capable(disk)) + return -EINVAL; + } ret = queue_var_store(&poll_on, page, count); if (ret < 0) diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index bfab74b45f15..a46f975f2a2f 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1881,6 +1881,7 @@ struct block_device_operations { int (*report_zones)(struct gendisk *, sector_t sector, unsigned int nr_zones, report_zones_cb cb, void *data); char *(*devnode)(struct gendisk *disk, umode_t *mode); + bool (*poll_capable)(struct gendisk *disk); struct module *owner; const struct pr_ops *pr_ops; }; From patchwork Thu Mar 18 16:48:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12148667 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57322C433E6 for ; Thu, 18 Mar 2021 16:54:05 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EE87964E07 for ; Thu, 18 Mar 2021 16:54:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EE87964E07 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1616086444; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=JTcdCk2Jork8fcv/ey+SihCHDkPyR28rjIrK8eSyg/o=; b=XaLOsDQOqelAJGJx90KTk/mdguLuSIPMSDdr7COpsFiPe82ADKQlVwrPIZN3mPo54ChdKR Y/pARde6uL3NYyzQ/xZmp8t2Co2BjJWZFfXv+Si1d1rcfVjgQxsV/WJ3EdQyTcVUYF0XjX jcLxISBmFMHb9JXJbRMWvaJSvM39F3c= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-59-iA_GRqMyPSW8-45_7OTBoA-1; Thu, 18 Mar 2021 12:54:01 -0400 X-MC-Unique: iA_GRqMyPSW8-45_7OTBoA-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id CD58A180FCC2; Thu, 18 Mar 2021 16:53:55 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id AD6885D72E; Thu, 18 Mar 2021 16:53:55 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 73CF04A7CA; Thu, 18 Mar 2021 16:53:55 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 12IGnwR0030698 for ; Thu, 18 Mar 2021 12:49:58 -0400 Received: by smtp.corp.redhat.com (Postfix) id 1B01210023BE; Thu, 18 Mar 2021 16:49:58 +0000 (UTC) Received: from localhost (ovpn-12-24.pek2.redhat.com [10.72.12.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1C9AB10027A5; Thu, 18 Mar 2021 16:49:56 +0000 (UTC) From: Ming Lei To: Jens Axboe Date: Fri, 19 Mar 2021 00:48:26 +0800 Message-Id: <20210318164827.1481133-13-ming.lei@redhat.com> In-Reply-To: <20210318164827.1481133-1-ming.lei@redhat.com> References: <20210318164827.1481133-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-loop: dm-devel@redhat.com Cc: Mike Snitzer , Ming Lei , linux-block@vger.kernel.org, dm-devel@redhat.com, Jeffle Xu , Christoph Hellwig Subject: [dm-devel] [RFC PATCH V2 12/13] dm: support IO polling for bio-based dm device X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com From: Jeffle Xu IO polling is enabled when all underlying target devices are capable of IO polling. The sanity check supports the stacked device model, in which one dm device may be build upon another dm device. In this case, the mapped device will check if the underlying dm target device supports IO polling. Signed-off-by: Jeffle Xu Signed-off-by: Ming Lei --- drivers/md/dm-table.c | 24 ++++++++++++++++++++++++ drivers/md/dm.c | 14 ++++++++++++++ include/linux/device-mapper.h | 1 + 3 files changed, 39 insertions(+) diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c index 95391f78b8d5..a8f3575fb118 100644 --- a/drivers/md/dm-table.c +++ b/drivers/md/dm-table.c @@ -1509,6 +1509,12 @@ struct dm_target *dm_table_find_target(struct dm_table *t, sector_t sector) return &t->targets[(KEYS_PER_NODE * n) + k]; } +static int device_not_poll_capable(struct dm_target *ti, struct dm_dev *dev, + sector_t start, sector_t len, void *data) +{ + return !blk_queue_poll(bdev_get_queue(dev->bdev)); +} + /* * type->iterate_devices() should be called when the sanity check needs to * iterate and check all underlying data devices. iterate_devices() will @@ -1559,6 +1565,11 @@ static int count_device(struct dm_target *ti, struct dm_dev *dev, return 0; } +int dm_table_supports_poll(struct dm_table *t) +{ + return !dm_table_any_dev_attr(t, device_not_poll_capable, NULL); +} + /* * Check whether a table has no data devices attached using each * target's iterate_devices method. @@ -2079,6 +2090,19 @@ void dm_table_set_restrictions(struct dm_table *t, struct request_queue *q, dm_update_keyslot_manager(q, t); blk_queue_update_readahead(q); + + /* + * Check for request-based device is remained to + * dm_mq_init_request_queue()->blk_mq_init_allocated_queue(). + * For bio-based device, only set QUEUE_FLAG_POLL when all underlying + * devices supporting polling. + */ + if (__table_type_bio_based(t->type)) { + if (dm_table_supports_poll(t)) + blk_queue_flag_set(QUEUE_FLAG_POLL, q); + else + blk_queue_flag_clear(QUEUE_FLAG_POLL, q); + } } unsigned int dm_table_get_num_targets(struct dm_table *t) diff --git a/drivers/md/dm.c b/drivers/md/dm.c index 50b693d776d6..fe6893b078dc 100644 --- a/drivers/md/dm.c +++ b/drivers/md/dm.c @@ -1720,6 +1720,19 @@ static blk_qc_t dm_submit_bio(struct bio *bio) return ret; } +static bool dm_bio_poll_capable(struct gendisk *disk) +{ + int ret, srcu_idx; + struct mapped_device *md = disk->private_data; + struct dm_table *t; + + t = dm_get_live_table(md, &srcu_idx); + ret = dm_table_supports_poll(t); + dm_put_live_table(md, srcu_idx); + + return ret; +} + /*----------------------------------------------------------------- * An IDR is used to keep track of allocated minor numbers. *---------------------------------------------------------------*/ @@ -3132,6 +3145,7 @@ static const struct pr_ops dm_pr_ops = { }; static const struct block_device_operations dm_blk_dops = { + .poll_capable = dm_bio_poll_capable, .submit_bio = dm_submit_bio, .open = dm_blk_open, .release = dm_blk_close, diff --git a/include/linux/device-mapper.h b/include/linux/device-mapper.h index 7f4ac87c0b32..31bfd6f70013 100644 --- a/include/linux/device-mapper.h +++ b/include/linux/device-mapper.h @@ -538,6 +538,7 @@ unsigned int dm_table_get_num_targets(struct dm_table *t); fmode_t dm_table_get_mode(struct dm_table *t); struct mapped_device *dm_table_get_md(struct dm_table *t); const char *dm_table_device_name(struct dm_table *t); +int dm_table_supports_poll(struct dm_table *t); /* * Trigger an event. From patchwork Thu Mar 18 16:48:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 12148613 X-Patchwork-Delegate: snitzer@redhat.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAC6BC433DB for ; Thu, 18 Mar 2021 16:50:12 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 64ACC64F1C for ; Thu, 18 Mar 2021 16:50:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 64ACC64F1C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=tempfail smtp.mailfrom=dm-devel-bounces@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1616086211; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=sp1Du772nyRSvA5JpD/a4dWd9SeUu3+IsQGqKmOBjmI=; b=XcU8hK7asy10wnSuMg76AjHAj9FUK73DoEOiFfGHjrYmxAzdEQaKOHmCxqKRvb/sn9Lcr6 BzdsSjs+oTY6Tt8wuG6f5NYIXAKad2MEMFeAp6/OPTGzAjYqdjDMmqHbYMCwMnRsK8cmMi G7ITYkcIk+Vpzo3uCrdG/A+/7cP4/qs= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-566-1nSLcF66NVmj4OL52G_wPA-1; Thu, 18 Mar 2021 12:50:09 -0400 X-MC-Unique: 1nSLcF66NVmj4OL52G_wPA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 6C8C98042C1; Thu, 18 Mar 2021 16:50:03 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 3EC4410023BE; Thu, 18 Mar 2021 16:50:03 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 09E7918155DD; Thu, 18 Mar 2021 16:50:03 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 12IGo1CK030728 for ; Thu, 18 Mar 2021 12:50:01 -0400 Received: by smtp.corp.redhat.com (Postfix) id 77EF0610F3; Thu, 18 Mar 2021 16:50:01 +0000 (UTC) Received: from localhost (ovpn-12-24.pek2.redhat.com [10.72.12.24]) by smtp.corp.redhat.com (Postfix) with ESMTP id B9E61610F1; Thu, 18 Mar 2021 16:50:00 +0000 (UTC) From: Ming Lei To: Jens Axboe Date: Fri, 19 Mar 2021 00:48:27 +0800 Message-Id: <20210318164827.1481133-14-ming.lei@redhat.com> In-Reply-To: <20210318164827.1481133-1-ming.lei@redhat.com> References: <20210318164827.1481133-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-loop: dm-devel@redhat.com Cc: Mike Snitzer , Ming Lei , linux-block@vger.kernel.org, dm-devel@redhat.com, Jeffle Xu , Christoph Hellwig Subject: [dm-devel] [RFC PATCH V2 13/13] blk-mq: limit hw queues to be polled in each blk_poll() X-BeenThere: dm-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: device-mapper development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dm-devel-bounces@redhat.com Errors-To: dm-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=dm-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Limit at most 8 queues are polled in each blk_pull(), avoid to add extra latency when queue depth is high. Signed-off-by: Ming Lei --- block/blk-mq.c | 66 +++++++++++++++++++++++++++++++++++--------------- 1 file changed, 46 insertions(+), 20 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index f26950a51f4a..9c94b7f0bf4b 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -3870,33 +3870,31 @@ static blk_qc_t bio_get_poll_cookie(struct bio *bio) return bio->bi_iter.bi_private_data; } -static int blk_mq_poll_io(struct bio *bio) +#define POLL_HCTX_MAX_CNT 8 + +static bool blk_add_unique_hctx(struct blk_mq_hw_ctx **data, int *cnt, + struct blk_mq_hw_ctx *hctx) { - struct request_queue *q = bio->bi_bdev->bd_disk->queue; - blk_qc_t cookie = bio_get_poll_cookie(bio); - int ret = 0; + int i; - if (!bio_flagged(bio, BIO_DONE) && blk_qc_t_valid(cookie)) { - struct blk_mq_hw_ctx *hctx = - q->queue_hw_ctx[blk_qc_t_to_queue_num(cookie)]; + for (i = 0; i < *cnt; i++) { + if (data[i] == hctx) + goto exit; + } - ret += blk_mq_poll_hctx(q, hctx); + if (i < POLL_HCTX_MAX_CNT) { + data[i] = hctx; + (*cnt)++; } - return ret; + exit: + return *cnt == POLL_HCTX_MAX_CNT; } -static int blk_bio_poll_and_end_io(struct request_queue *q, - struct blk_bio_poll_ctx *poll_ctx) +static void blk_build_poll_queues(struct blk_bio_poll_ctx *poll_ctx, + struct blk_mq_hw_ctx **data, int *cnt) { - int ret = 0; int i; - /* - * Poll hw queue first. - * - * TODO: limit max poll times and make sure to not poll same - * hw queue one more time. - */ for (i = 0; i < poll_ctx->pq->max_nr_grps; i++) { struct bio_grp_list_data *grp = &poll_ctx->pq->head[i]; struct bio *bio; @@ -3904,9 +3902,37 @@ static int blk_bio_poll_and_end_io(struct request_queue *q, if (bio_grp_list_grp_empty(grp)) continue; - for (bio = grp->list.head; bio; bio = bio->bi_poll) - ret += blk_mq_poll_io(bio); + for (bio = grp->list.head; bio; bio = bio->bi_poll) { + blk_qc_t cookie; + struct blk_mq_hw_ctx *hctx; + struct request_queue *q; + + if (bio_flagged(bio, BIO_DONE)) + continue; + cookie = bio_get_poll_cookie(bio); + if (!blk_qc_t_valid(cookie)) + continue; + + q = bio->bi_bdev->bd_disk->queue; + hctx = q->queue_hw_ctx[blk_qc_t_to_queue_num(cookie)]; + if (blk_add_unique_hctx(data, cnt, hctx)) + return; + } } +} + +static int blk_bio_poll_and_end_io(struct request_queue *q, + struct blk_bio_poll_ctx *poll_ctx) +{ + int ret = 0; + int i; + struct blk_mq_hw_ctx *hctx[POLL_HCTX_MAX_CNT]; + int cnt = 0; + + blk_build_poll_queues(poll_ctx, hctx, &cnt); + + for (i = 0; i < cnt; i++) + ret += blk_mq_poll_hctx(hctx[i]->queue, hctx[i]); /* reap bios */ for (i = 0; i < poll_ctx->pq->max_nr_grps; i++) {