From patchwork Wed Jul 19 18:22:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13319315 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76E3BC0015E for ; Wed, 19 Jul 2023 18:23:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229655AbjGSSXR (ORCPT ); Wed, 19 Jul 2023 14:23:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50508 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229601AbjGSSXQ (ORCPT ); Wed, 19 Jul 2023 14:23:16 -0400 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A933F1BF6 for ; Wed, 19 Jul 2023 11:23:15 -0700 (PDT) Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-1b89600a37fso45327775ad.2 for ; Wed, 19 Jul 2023 11:23:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689790995; x=1692382995; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=824u76hYSw99543u2htvcLKDRWPlhBTji6Kvikd0vyI=; b=hFd7yRvjZvZhbaxb/tW20T9WXtU1p+Zm9TFSUh0+vi92AzPD0aqrK5mM0K9aB+Rmkj Npg4eIVSckCmTvtDWdG2/P0Vs18iArK34v4Lygb7+CZw/rEZaKFAD70Tc5dv7mWwwUG7 9bO1HvE1g9cvKcnbeiJKFzWfmG/LezcjbDws/G0LltBYKHVSXTeIC/T0X5gX71RTolB1 +GAcdrFSu7XrY8XtHv7CY71ld5M5Cyb99MbjVrd/yqmnpbvGiSO/fdNDJkaYZgGGXbw4 8ccYX7SVEXwtfmpvnG5Axg0uQPAykPQXgNs/YD6zbTw7+jJ5rTP1Ut6Xdnr8x1DUvmhm 3M+A== X-Gm-Message-State: ABy/qLb+/CvwVy3HYmCCBMs8xRZ8wPM5fy4jEyvPTboAfxEu5/pIXPmg k7tUI+k/LKLKMKyC9q3EzZ3s6dsJl+4= X-Google-Smtp-Source: APBJJlEP0QrWbl4FbFXdzE15XdGk5NtwNIi94GeCSxCOLXjauLCLFCL8/33nbj+nCKkoJsELGgaFTw== X-Received: by 2002:a17:903:1d0:b0:1b6:bced:1dd6 with SMTP id e16-20020a17090301d000b001b6bced1dd6mr2168929plh.35.1689790994975; Wed, 19 Jul 2023 11:23:14 -0700 (PDT) Received: from bvanassche-linux.mtv.corp.google.com ([2620:15c:211:201:a2ab:183f:c76c:d30d]) by smtp.gmail.com with ESMTPSA id l16-20020a170903245000b001bb33ee4057sm4316061pls.43.2023.07.19.11.23.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jul 2023 11:23:14 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , "Martin K . Petersen" , Bart Van Assche , "James E.J. Bottomley" Subject: [PATCH v2 1/3] scsi: Inline scsi_kick_queue() Date: Wed, 19 Jul 2023 11:22:40 -0700 Message-ID: <20230719182243.2810134-2-bvanassche@acm.org> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog In-Reply-To: <20230719182243.2810134-1-bvanassche@acm.org> References: <20230719182243.2810134-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org scsi_kick_queue() is too short to keep it as a separate function. Hence inline it. This patch prepares for modifying the second argument passed to blk_mq_run_hw_queues(). Cc: Martin K. Petersen Signed-off-by: Bart Van Assche Reviewed-by: Christoph Hellwig --- drivers/scsi/scsi_lib.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c index ad9afae49544..414d29eef968 100644 --- a/drivers/scsi/scsi_lib.c +++ b/drivers/scsi/scsi_lib.c @@ -300,11 +300,6 @@ void scsi_device_unbusy(struct scsi_device *sdev, struct scsi_cmnd *cmd) cmd->budget_token = -1; } -static void scsi_kick_queue(struct request_queue *q) -{ - blk_mq_run_hw_queues(q, false); -} - /* * Kick the queue of SCSI device @sdev if @sdev != current_sdev. Called with * interrupts disabled. @@ -340,7 +335,7 @@ static void scsi_single_lun_run(struct scsi_device *current_sdev) * but in most cases, we will be first. Ideally, each LU on the * target would get some limited time or requests on the target. */ - scsi_kick_queue(current_sdev->request_queue); + blk_mq_run_hw_queues(current_sdev->request_queue, false); spin_lock_irqsave(shost->host_lock, flags); if (!starget->starget_sdev_user) @@ -427,7 +422,7 @@ static void scsi_starved_list_run(struct Scsi_Host *shost) continue; spin_unlock_irqrestore(shost->host_lock, flags); - scsi_kick_queue(slq); + blk_mq_run_hw_queues(slq, false); blk_put_queue(slq); spin_lock_irqsave(shost->host_lock, flags); From patchwork Wed Jul 19 18:22:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13319316 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7E03C001E0 for ; Wed, 19 Jul 2023 18:23:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229601AbjGSSXS (ORCPT ); Wed, 19 Jul 2023 14:23:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50520 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229525AbjGSSXR (ORCPT ); Wed, 19 Jul 2023 14:23:17 -0400 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A4011B6 for ; Wed, 19 Jul 2023 11:23:16 -0700 (PDT) Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-1bb1baf55f5so44507715ad.0 for ; Wed, 19 Jul 2023 11:23:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689790996; x=1692382996; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QC6pt/SvGlEEBPsJW6GWFPvxw3EFvrXnyNrPZICgHZg=; b=XkqoYNn/fofwCnmoIo3CKbn04qc3tXgmFi9MAN6Ohjul6MP1d8IVUUEyDtyjeSsWUA GbxYPthK81qCiKNul7zkPOlCu1vNJ6sfNswww2+GpJzCaTSqNzXo+vtILudZrUc/y/uj gNLte4Plm+Ib0BwcW3dnd5rHsNhDxyrQZeabvSswcvhGHhN4UxxQHc5HSY2Cvg8OHBgS zeyeC3U57+Hmc9u382baK6T8Hsancrl+lK8df2aNT6osSr24SmHgVeKHRi+40xw0hBFH S8hy85IKWWP0M+Zs9gxaPRS29vNQV/E1r0lNQykdEj1TX1WnBggSmRC/bAVIHbJ+hwMJ kMEg== X-Gm-Message-State: ABy/qLbmQAtW12magd7OmXXsN33HJ+v0LF+JqF2q8X/LRzOSAOUm0TpV o8I02NkcPShG1y+lGjX1ysk= X-Google-Smtp-Source: APBJJlGrFo5pEATgnKVlxpj4un8c+3FKfrTdqmc7skgXyJRhauljCvmYYuzKu20YLwCWWyPr/Lef9w== X-Received: by 2002:a17:902:6a8c:b0:1b8:90bd:d157 with SMTP id n12-20020a1709026a8c00b001b890bdd157mr19345582plk.26.1689790996075; Wed, 19 Jul 2023 11:23:16 -0700 (PDT) Received: from bvanassche-linux.mtv.corp.google.com ([2620:15c:211:201:a2ab:183f:c76c:d30d]) by smtp.gmail.com with ESMTPSA id l16-20020a170903245000b001bb33ee4057sm4316061pls.43.2023.07.19.11.23.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jul 2023 11:23:15 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , "Martin K . Petersen" , Bart Van Assche , "James E.J. Bottomley" Subject: [PATCH v2 2/3] scsi: Remove a blk_mq_run_hw_queues() call Date: Wed, 19 Jul 2023 11:22:41 -0700 Message-ID: <20230719182243.2810134-3-bvanassche@acm.org> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog In-Reply-To: <20230719182243.2810134-1-bvanassche@acm.org> References: <20230719182243.2810134-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org blk_mq_kick_requeue_list() calls blk_mq_run_hw_queues() asynchronously. Leave out the direct blk_mq_run_hw_queues() call. This patch causes scsi_run_queue() to call blk_mq_run_hw_queues() asynchronously instead of synchronously. Since scsi_run_queue() is not called from the hot I/O submission path, this patch does not affect the hot path. Cc: Martin K. Petersen Signed-off-by: Bart Van Assche --- drivers/scsi/scsi_lib.c | 1 - 1 file changed, 1 deletion(-) diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c index 414d29eef968..7043ca0f4da9 100644 --- a/drivers/scsi/scsi_lib.c +++ b/drivers/scsi/scsi_lib.c @@ -448,7 +448,6 @@ static void scsi_run_queue(struct request_queue *q) scsi_starved_list_run(sdev->host); blk_mq_kick_requeue_list(q); - blk_mq_run_hw_queues(q, false); } void scsi_requeue_run_queue(struct work_struct *work) From patchwork Wed Jul 19 18:22:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 13319317 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6DE57C0015E for ; Wed, 19 Jul 2023 18:23:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230086AbjGSSXU (ORCPT ); Wed, 19 Jul 2023 14:23:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50550 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229864AbjGSSXT (ORCPT ); Wed, 19 Jul 2023 14:23:19 -0400 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31FFE1FD9 for ; Wed, 19 Jul 2023 11:23:18 -0700 (PDT) Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-1b89e10d356so46235515ad.3 for ; Wed, 19 Jul 2023 11:23:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689790997; x=1692382997; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ieODJHKrtRMWFM3HYy2VJVeJK1X6Tdhz5Qq8QKeRaN0=; b=Za+5W60vceIdfu01OFA8tlgdjqGGIlxmdrx8MtA2b5QRBb7TOYIwHLrFhRbzNW4pcm zErCQ+9465P2hrxwnbOs3B9bUi0r3pJEu4zLaZx7ris8ydagLUNzGkY6ISludop7WoTp fBYgkhC0zq7fSj8noNFG6+WGOokkT+fFaGEYgBHsTHMmE+KUTAIIgLwWrU7v0EJ2Ahyp +zVst2NTxrrb0BQrU1aZ6Dqk7TBTqMdPmdkHx0D+HfiMA8+xt7oEbRzqD/b+11FaLjDm yoX6wnW71Gyo9BPhA4ZG0tBmJ/poMgVdY1QBxfhu536WYBarkH8+7lcRxPLhPRwT5/8R 4bdA== X-Gm-Message-State: ABy/qLZ5bYlLvuqprfop1X4Lk34t4M2ZIvcm+mid3yFwtG+8BrjtOWwt Nd+mpGCTAkVYoJMAiYXAR4o= X-Google-Smtp-Source: APBJJlFRQGO/WyivZOhv1uh7JpVAJNgR1y/Iiqcm+HSV+E1qZwx9xTJzvrE/JCsBRxjiTGV8iFVdqw== X-Received: by 2002:a17:902:c3d5:b0:1b3:b3c5:1d1f with SMTP id j21-20020a170902c3d500b001b3b3c51d1fmr5638008plj.8.1689790997353; Wed, 19 Jul 2023 11:23:17 -0700 (PDT) Received: from bvanassche-linux.mtv.corp.google.com ([2620:15c:211:201:a2ab:183f:c76c:d30d]) by smtp.gmail.com with ESMTPSA id l16-20020a170903245000b001bb33ee4057sm4316061pls.43.2023.07.19.11.23.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Jul 2023 11:23:16 -0700 (PDT) From: Bart Van Assche To: Jens Axboe Cc: linux-block@vger.kernel.org, Christoph Hellwig , "Martin K . Petersen" , Bart Van Assche , Ming Lei , "James E.J. Bottomley" Subject: [PATCH v2 3/3] block: Improve performance for BLK_MQ_F_BLOCKING drivers Date: Wed, 19 Jul 2023 11:22:42 -0700 Message-ID: <20230719182243.2810134-4-bvanassche@acm.org> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog In-Reply-To: <20230719182243.2810134-1-bvanassche@acm.org> References: <20230719182243.2810134-1-bvanassche@acm.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org blk_mq_run_queue() runs the queue asynchronously if BLK_MQ_F_BLOCKING has been set. This is suboptimal since running the queue asynchronously is slower than running the queue synchronously. This patch modifies blk_mq_run_queue() as follows if BLK_MQ_F_BLOCKING has been set: - Run the queue synchronously if it is allowed to sleep. - Run the queue asynchronously if it is not allowed to sleep. Additionally, blk_mq_run_hw_queue(hctx, false) calls are modified into blk_mq_run_hw_queue(hctx, hctx->flags & BLK_MQ_F_BLOCKING) if the caller may be invoked from atomic context. The following caller chains have been reviewed: blk_mq_run_hw_queue(hctx, false) blk_mq_get_tag() /* may sleep, hence the functions it calls may also sleep */ blk_execute_rq_nowait() nvme_*() /* the NVMe driver does not set BLK_MQ_F_BLOCKING */ scsi_eh_lock_door() /* may sleep */ sg_common_write() /* implements an ioctl and hence may sleep */ st_scsi_execute() /* may sleep */ pscsi_execute_cmd() /* may sleep */ ufshpb_execute_umap_req() /* may sleep */ ufshbp_execute_map_req() /* may sleep */ blk_execute_rq() /* may sleep */ blk_mq_run_hw_queues(q, async=false) blk_freeze_queue_start() /* may sleep */ blk_mq_requeue_work() /* may sleep */ scsi_kick_queue() scsi_requeue_run_queue() /* may sleep */ scsi_run_host_queues() scsi_ioctl_reset() /* may sleep */ blk_mq_insert_requests(hctx, ctx, list, run_queue_async=false) blk_mq_dispatch_plug_list(plug, from_sched=false) blk_mq_flush_plug_list(plug, from_schedule=false) __blk_flush_plug(plug, from_schedule=false) blk_add_rq_to_plug() blk_execute_rq_nowait() /* see above */ blk_mq_submit_bio() /* may sleep if REQ_NOWAIT has not been set */ blk_mq_plug_issue_direct() blk_mq_flush_plug_list() /* see above */ blk_mq_dispatch_plug_list(plug, from_sched=false) blk_mq_flush_plug_list() /* see above */ blk_mq_try_issue_directly() blk_mq_submit_bio() /* may sleep if REQ_NOWAIT has not been set */ blk_mq_try_issue_list_directly(hctx, list) blk_mq_insert_requests() /* see above */ Cc: Christoph Hellwig Cc: Ming Lei Signed-off-by: Bart Van Assche --- block/blk-mq.c | 17 +++++++++++------ drivers/scsi/scsi_lib.c | 3 ++- 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 5504719b970d..d5ab0bd8b472 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1289,7 +1289,8 @@ static void blk_add_rq_to_plug(struct blk_plug *plug, struct request *rq) * * Description: * Insert a fully prepared request at the back of the I/O scheduler queue - * for execution. Don't wait for completion. + * for execution. Don't wait for completion. May sleep if BLK_MQ_F_BLOCKING + * has been set. * * Note: * This function will invoke @done directly if the queue is dead. @@ -2213,6 +2214,8 @@ void blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async) */ WARN_ON_ONCE(!async && in_interrupt()); + might_sleep_if(!async && hctx->flags & BLK_MQ_F_BLOCKING); + /* * When queue is quiesced, we may be switching io scheduler, or * updating nr_hw_queues, or other things, and we can't run queue @@ -2228,8 +2231,7 @@ void blk_mq_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async) if (!need_run) return; - if (async || (hctx->flags & BLK_MQ_F_BLOCKING) || - !cpumask_test_cpu(raw_smp_processor_id(), hctx->cpumask)) { + if (async || !cpumask_test_cpu(raw_smp_processor_id(), hctx->cpumask)) { blk_mq_delay_run_hw_queue(hctx, 0); return; } @@ -2364,7 +2366,7 @@ void blk_mq_start_hw_queue(struct blk_mq_hw_ctx *hctx) { clear_bit(BLK_MQ_S_STOPPED, &hctx->state); - blk_mq_run_hw_queue(hctx, false); + blk_mq_run_hw_queue(hctx, hctx->flags & BLK_MQ_F_BLOCKING); } EXPORT_SYMBOL(blk_mq_start_hw_queue); @@ -2394,7 +2396,8 @@ void blk_mq_start_stopped_hw_queues(struct request_queue *q, bool async) unsigned long i; queue_for_each_hw_ctx(q, hctx, i) - blk_mq_start_stopped_hw_queue(hctx, async); + blk_mq_start_stopped_hw_queue(hctx, async || + (hctx->flags & BLK_MQ_F_BLOCKING)); } EXPORT_SYMBOL(blk_mq_start_stopped_hw_queues); @@ -2452,6 +2455,8 @@ static void blk_mq_insert_requests(struct blk_mq_hw_ctx *hctx, list_for_each_entry(rq, list, queuelist) { BUG_ON(rq->mq_ctx != ctx); trace_block_rq_insert(rq); + if (rq->cmd_flags & REQ_NOWAIT) + run_queue_async = true; } spin_lock(&ctx->lock); @@ -2612,7 +2617,7 @@ static void blk_mq_try_issue_directly(struct blk_mq_hw_ctx *hctx, if ((rq->rq_flags & RQF_USE_SCHED) || !blk_mq_get_budget_and_tag(rq)) { blk_mq_insert_request(rq, 0); - blk_mq_run_hw_queue(hctx, false); + blk_mq_run_hw_queue(hctx, rq->cmd_flags & REQ_NOWAIT); return; } diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c index 7043ca0f4da9..84fb0feb047f 100644 --- a/drivers/scsi/scsi_lib.c +++ b/drivers/scsi/scsi_lib.c @@ -335,7 +335,8 @@ static void scsi_single_lun_run(struct scsi_device *current_sdev) * but in most cases, we will be first. Ideally, each LU on the * target would get some limited time or requests on the target. */ - blk_mq_run_hw_queues(current_sdev->request_queue, false); + blk_mq_run_hw_queues(current_sdev->request_queue, + shost->queuecommand_may_block); spin_lock_irqsave(shost->host_lock, flags); if (!starget->starget_sdev_user)