From patchwork Tue Aug 8 10:42:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13346220 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C8A6C001DB for ; Tue, 8 Aug 2023 16:00:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231532AbjHHQAU (ORCPT ); Tue, 8 Aug 2023 12:00:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35850 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230457AbjHHP6j (ORCPT ); Tue, 8 Aug 2023 11:58:39 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F35455FE1 for ; Tue, 8 Aug 2023 08:44:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691509416; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QvKrmms9fUHhxbhcGxDz7QDKl8B/hBW7QeIiIj8phvY=; b=Urk+r0LSvD+EBiH4zGbdOFsyq3bqjROlouhAoJs1itDG9Q4mpEFROIEgbTXPJfVbERK5Q5 fkYgyGnjj2Hv1kEPClKLpaM41+NIv1q/dt7VjsTsYhQUL+kKyllAKZ53ouur8Oq3qPVvtP ZNZLWUj9LaWI/DSbh2tewAfblSTI4sk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-586-xZnrD3mGMTGBaneeBOzJFQ-1; Tue, 08 Aug 2023 06:42:52 -0400 X-MC-Unique: xZnrD3mGMTGBaneeBOzJFQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 955AC800270; Tue, 8 Aug 2023 10:42:51 +0000 (UTC) Received: from localhost (unknown [10.72.120.3]) by smtp.corp.redhat.com (Postfix) with ESMTP id CAD5A2026D4B; Tue, 8 Aug 2023 10:42:50 +0000 (UTC) From: Ming Lei To: Jens Axboe , Christoph Hellwig , linux-nvme@lists.infradead.org, "Martin K . Petersen" , linux-scsi@vger.kernel.org Cc: linux-block@vger.kernel.org, Wen Xiong , Keith Busch , Ming Lei Subject: [PATCH V3 01/14] blk-mq: add blk_mq_max_nr_hw_queues() Date: Tue, 8 Aug 2023 18:42:26 +0800 Message-Id: <20230808104239.146085-2-ming.lei@redhat.com> In-Reply-To: <20230808104239.146085-1-ming.lei@redhat.com> References: <20230808104239.146085-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org blk_mq_alloc_tag_set() may override set->nr_hw_queues as 1 in case of kdump kernel. This way causes trouble for driver, because blk-mq and driver see different queue mapping. Especially the only online CPU may not be 1 for kdump kernel, in which 'maxcpus=1' is passed from kernel command line, then driver may map hctx0 into one inactive real hw queue which cpu affinity is 0(offline). The issue exists on all drivers which use managed irq and support multiple hw queue. Prepare for fixing this kind of issue by applying the added helper, so driver can take blk-mq max nr_hw_queues knowledge into account when calculating io queues. Signed-off-by: Ming Lei --- block/blk-mq.c | 16 ++++++++++++++++ include/linux/blk-mq.h | 1 + 2 files changed, 17 insertions(+) diff --git a/block/blk-mq.c b/block/blk-mq.c index b04ff6f56926..617d6f849a7b 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -140,6 +140,22 @@ void blk_mq_freeze_queue_wait(struct request_queue *q) } EXPORT_SYMBOL_GPL(blk_mq_freeze_queue_wait); +/* + * Return the max supported nr_hw_queues for each hw queue type + * + * blk_mq_alloc_tag_set() may change nr_hw_queues for kdump kernel, so + * driver has to take blk-mq max supported nr_hw_queues into account + * when figuring out nr_hw_queues from hardware info, for avoiding + * inconsistency between driver and blk-mq. + */ +unsigned int blk_mq_max_nr_hw_queues(void) +{ + if (is_kdump_kernel()) + return 1; + return nr_cpu_ids; +} +EXPORT_SYMBOL_GPL(blk_mq_max_nr_hw_queues); + int blk_mq_freeze_queue_wait_timeout(struct request_queue *q, unsigned long timeout) { diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 495ca198775f..4c0cfd1f9e52 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -711,6 +711,7 @@ int blk_mq_alloc_sq_tag_set(struct blk_mq_tag_set *set, const struct blk_mq_ops *ops, unsigned int queue_depth, unsigned int set_flags); void blk_mq_free_tag_set(struct blk_mq_tag_set *set); +unsigned int blk_mq_max_nr_hw_queues(void); void blk_mq_free_request(struct request *rq); int blk_rq_poll(struct request *rq, struct io_comp_batch *iob, From patchwork Tue Aug 8 10:42:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13346250 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8FECC04A6A for ; Tue, 8 Aug 2023 16:11:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231959AbjHHQLk (ORCPT ); Tue, 8 Aug 2023 12:11:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46220 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231975AbjHHQJh (ORCPT ); Tue, 8 Aug 2023 12:09:37 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 78F717A80 for ; Tue, 8 Aug 2023 08:46:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691509566; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=k/XiItvuETzuFaV0q2/0Ml9SkHL0l2Lbxykjfon98Ww=; b=gzWBponId6Rk3ISzI2vUdqURoGvJDI7RBQf/FK+rvwyj9HojA96sH8kSFesaWxyPoz7sFs zUj9NytMkBJhEHqbi3/nIUPGZJq9646WDcXtaPMumsN9WLzakryZG4IaSd+C0hbpFwK++r nIG3PXEj1yznjMkudL9DVCrYkaI3JVY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-484-dHf5P5j1NCCY04pi7Cukww-1; Tue, 08 Aug 2023 06:42:55 -0400 X-MC-Unique: dHf5P5j1NCCY04pi7Cukww-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3A89B85CCE0; Tue, 8 Aug 2023 10:42:55 +0000 (UTC) Received: from localhost (unknown [10.72.120.3]) by smtp.corp.redhat.com (Postfix) with ESMTP id 65E124E3D1; Tue, 8 Aug 2023 10:42:53 +0000 (UTC) From: Ming Lei To: Jens Axboe , Christoph Hellwig , linux-nvme@lists.infradead.org, "Martin K . Petersen" , linux-scsi@vger.kernel.org Cc: linux-block@vger.kernel.org, Wen Xiong , Keith Busch , Ming Lei Subject: [PATCH V3 02/14] nvme-pci: use blk_mq_max_nr_hw_queues() to calculate io queues Date: Tue, 8 Aug 2023 18:42:27 +0800 Message-Id: <20230808104239.146085-3-ming.lei@redhat.com> In-Reply-To: <20230808104239.146085-1-ming.lei@redhat.com> References: <20230808104239.146085-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Take blk-mq's knowledge into account for calculating io queues. Fix wrong queue mapping in case of kdump kernel. On arm and ppc64, 'maxcpus=1' is passed to kdump command line, see `Documentation/admin-guide/kdump/kdump.rst`, so num_possible_cpus() still returns all CPUs because 'maxcpus=1' just bring up one single cpu core during booting. blk-mq sees single queue in kdump kernel, and in driver's viewpoint there are still multiple queues, this inconsistency causes driver to apply wrong queue mapping for handling IO, and IO timeout is triggered. Meantime, single queue makes much less resource utilization, and reduce risk of kernel failure. Reported-by: Wen Xiong Signed-off-by: Ming Lei --- drivers/nvme/host/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index baf69af7ea78..a1227ae7eb39 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -2251,7 +2251,7 @@ static unsigned int nvme_max_io_queues(struct nvme_dev *dev) */ if (dev->ctrl.quirks & NVME_QUIRK_SHARED_TAGS) return 1; - return num_possible_cpus() + dev->nr_write_queues + dev->nr_poll_queues; + return blk_mq_max_nr_hw_queues() + dev->nr_write_queues + dev->nr_poll_queues; } static int nvme_setup_io_queues(struct nvme_dev *dev) From patchwork Tue Aug 8 10:42:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13346221 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2ACC1C001B0 for ; Tue, 8 Aug 2023 16:00:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231335AbjHHQA0 (ORCPT ); Tue, 8 Aug 2023 12:00:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47414 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231358AbjHHP6q (ORCPT ); Tue, 8 Aug 2023 11:58:46 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3349F5FE5 for ; Tue, 8 Aug 2023 08:44:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691509416; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9SdkdgSqxhMyRTJ5vgJIEJL/WyHf61DHGM5o0eLsF7g=; b=XZK+GbTeDJMKd/V3zgkrmM+Nmoc04YNYpnu0YV1L6G9tFwLwnmK4a+cK9KxigI9nQbhlaf gF53rHHt/aj0gwB4wiouBQK4q1werjgQ/qxh0ESWEoBdwoqALmOouLPaRqCgJUgENnvHl+ h/MKIm4WAlRCzK38ShLtDdy+/IwYNJo= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-36-HN97ZHsCP7utxVyyzgqj-Q-1; Tue, 08 Aug 2023 06:42:59 -0400 X-MC-Unique: HN97ZHsCP7utxVyyzgqj-Q-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7EAA6185A78B; Tue, 8 Aug 2023 10:42:58 +0000 (UTC) Received: from localhost (unknown [10.72.120.3]) by smtp.corp.redhat.com (Postfix) with ESMTP id A81182166B27; Tue, 8 Aug 2023 10:42:57 +0000 (UTC) From: Ming Lei To: Jens Axboe , Christoph Hellwig , linux-nvme@lists.infradead.org, "Martin K . Petersen" , linux-scsi@vger.kernel.org Cc: linux-block@vger.kernel.org, Wen Xiong , Keith Busch , Ming Lei Subject: [PATCH V3 03/14] ublk: limit max allowed nr_hw_queues Date: Tue, 8 Aug 2023 18:42:28 +0800 Message-Id: <20230808104239.146085-4-ming.lei@redhat.com> In-Reply-To: <20230808104239.146085-1-ming.lei@redhat.com> References: <20230808104239.146085-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Take blk-mq's knowledge into account for calculating io queues. Fix wrong queue mapping in case of kdump kernel. On arm and ppc64, 'maxcpus=1' is passed to kdump command line, see `Documentation/admin-guide/kdump/kdump.rst`, so num_possible_cpus() still returns all CPUs because 'maxcpus=1' just bring up one single cpu core during booting. blk-mq sees single queue in kdump kernel, and in driver's viewpoint there are still multiple queues, this inconsistency causes driver to apply wrong queue mapping for handling IO, and IO timeout is triggered. Meantime, single queue makes much less resource utilization, and reduce risk of kernel failure. Signed-off-by: Ming Lei --- drivers/block/ublk_drv.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c index 21d2e71c5514..6cf4981951e2 100644 --- a/drivers/block/ublk_drv.c +++ b/drivers/block/ublk_drv.c @@ -2045,7 +2045,7 @@ static int ublk_ctrl_add_dev(struct io_uring_cmd *cmd) ub->dev_info.flags &= ~UBLK_F_SUPPORT_ZERO_COPY; ub->dev_info.nr_hw_queues = min_t(unsigned int, - ub->dev_info.nr_hw_queues, nr_cpu_ids); + ub->dev_info.nr_hw_queues, blk_mq_max_nr_hw_queues()); ublk_align_max_io_size(ub); ret = ublk_init_queues(ub); From patchwork Tue Aug 8 10:42:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13346334 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4823FC001B0 for ; Tue, 8 Aug 2023 16:24:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229791AbjHHQYL (ORCPT ); Tue, 8 Aug 2023 12:24:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38878 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229903AbjHHQWu (ORCPT ); Tue, 8 Aug 2023 12:22:50 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E79D09EE6 for ; Tue, 8 Aug 2023 08:49:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691509756; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9r/GcSpKKC2dt0ISzH0rvZ1VrInscXGexGqDsV7Zgfk=; b=ZyTdwg/4IiOUYZxFYz/47uzuCMYqdoq3Or7stz2alILwts9mojUHe7z6BuasrPX++KI2Bp y2BSLnoB5vGi0n7/9teV3h05+wmAoai0eKBuj4iu+gFk2X6csG7MyVvNEQ3F+/Htpk1qus ml914Ckps2bNkZ6RlwyrCIY0mDMSqdY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-610-MBMMJmWsMYifeEoAwq7xkg-1; Tue, 08 Aug 2023 06:48:09 -0400 X-MC-Unique: MBMMJmWsMYifeEoAwq7xkg-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 354F4185A7A5; Tue, 8 Aug 2023 10:43:02 +0000 (UTC) Received: from localhost (unknown [10.72.120.3]) by smtp.corp.redhat.com (Postfix) with ESMTP id 62944401061; Tue, 8 Aug 2023 10:43:01 +0000 (UTC) From: Ming Lei To: Jens Axboe , Christoph Hellwig , linux-nvme@lists.infradead.org, "Martin K . Petersen" , linux-scsi@vger.kernel.org Cc: linux-block@vger.kernel.org, Wen Xiong , Keith Busch , Ming Lei , "Michael S . Tsirkin" , Jason Wang , virtualization@lists.linux-foundation.org Subject: [PATCH V3 04/14] virtio-blk: limit max allowed submit queues Date: Tue, 8 Aug 2023 18:42:29 +0800 Message-Id: <20230808104239.146085-5-ming.lei@redhat.com> In-Reply-To: <20230808104239.146085-1-ming.lei@redhat.com> References: <20230808104239.146085-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Take blk-mq's knowledge into account for calculating io queues. Fix wrong queue mapping in case of kdump kernel. On arm and ppc64, 'maxcpus=1' is passed to kdump command line, see `Documentation/admin-guide/kdump/kdump.rst`, so num_possible_cpus() still returns all CPUs because 'maxcpus=1' just bring up one single cpu core during booting. blk-mq sees single queue in kdump kernel, and in driver's viewpoint there are still multiple queues, this inconsistency causes driver to apply wrong queue mapping for handling IO, and IO timeout is triggered. Meantime, single queue makes much less resource utilization, and reduce risk of kernel failure. Cc: Michael S. Tsirkin Cc: Jason Wang Cc: virtualization@lists.linux-foundation.org Signed-off-by: Ming Lei Acked-by: Michael S. Tsirkin --- drivers/block/virtio_blk.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c index 1fe011676d07..4ba79fe2a1b4 100644 --- a/drivers/block/virtio_blk.c +++ b/drivers/block/virtio_blk.c @@ -1047,7 +1047,8 @@ static int init_vq(struct virtio_blk *vblk) num_poll_vqs = min_t(unsigned int, poll_queues, num_vqs - 1); - vblk->io_queues[HCTX_TYPE_DEFAULT] = num_vqs - num_poll_vqs; + vblk->io_queues[HCTX_TYPE_DEFAULT] = min_t(unsigned, + num_vqs - num_poll_vqs, blk_mq_max_nr_hw_queues()); vblk->io_queues[HCTX_TYPE_READ] = 0; vblk->io_queues[HCTX_TYPE_POLL] = num_poll_vqs; From patchwork Tue Aug 8 10:42:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13346364 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59007C0729B for ; Tue, 8 Aug 2023 16:33:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231402AbjHHQdV (ORCPT ); Tue, 8 Aug 2023 12:33:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46080 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233065AbjHHQcJ (ORCPT ); Tue, 8 Aug 2023 12:32:09 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7B1DB1396D for ; Tue, 8 Aug 2023 08:51:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691509863; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XUAbwUholcdP/E78pjIWjWfXWvuCT6ODN3CqmEOBnlY=; b=ASoKs8F9Xmvt5CjphZbOCQkRtOQETA90iyAIU59MIo0Uzm1yRmTKEunqXbJwy8TPfq/TkY gz5IES1R7vHWa2Cl7fHJGweo59LWyPc5b4FJ7aTOhq4t2ALEf1TmxUafL/2RnoJU5rhOgJ Voga8VkqyNG9TJAwdcMqmI+5aftBqJ8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-281-mqAwsN78NDGj8C053Wpfuw-1; Tue, 08 Aug 2023 06:43:06 -0400 X-MC-Unique: mqAwsN78NDGj8C053Wpfuw-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 952398015AA; Tue, 8 Aug 2023 10:43:05 +0000 (UTC) Received: from localhost (unknown [10.72.120.3]) by smtp.corp.redhat.com (Postfix) with ESMTP id 93E12492C13; Tue, 8 Aug 2023 10:43:04 +0000 (UTC) From: Ming Lei To: Jens Axboe , Christoph Hellwig , linux-nvme@lists.infradead.org, "Martin K . Petersen" , linux-scsi@vger.kernel.org Cc: linux-block@vger.kernel.org, Wen Xiong , Keith Busch , Ming Lei Subject: [PATCH V3 05/14] scsi: core: add helper of scsi_max_nr_hw_queues() Date: Tue, 8 Aug 2023 18:42:30 +0800 Message-Id: <20230808104239.146085-6-ming.lei@redhat.com> In-Reply-To: <20230808104239.146085-1-ming.lei@redhat.com> References: <20230808104239.146085-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org blk_mq_alloc_tag_set() may change nr_hw_queues for kdump kernel, so driver has to take blk-mq max supported nr_hw_queues into account when figuring out nr_hw_queues from hardware info, for avoiding inconsistency between scsi driver and blk-mq. Add helper of scsi_max_nr_hw_queues() for avoiding nr_hw_queues inconsistency between scsi driver and blk-mq. Signed-off-by: Ming Lei --- include/scsi/scsi_host.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/include/scsi/scsi_host.h b/include/scsi/scsi_host.h index 70b7475dcf56..2273f855940f 100644 --- a/include/scsi/scsi_host.h +++ b/include/scsi/scsi_host.h @@ -803,6 +803,11 @@ extern int scsi_host_unblock(struct Scsi_Host *shost, int new_state); void scsi_host_busy_iter(struct Scsi_Host *, bool (*fn)(struct scsi_cmnd *, void *), void *priv); +static inline unsigned int scsi_max_nr_hw_queues(void) +{ + return blk_mq_max_nr_hw_queues(); +} + struct class_container; /* From patchwork Tue Aug 8 10:42:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13346272 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0C75C001DE for ; Tue, 8 Aug 2023 16:13:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231960AbjHHQNV (ORCPT ); Tue, 8 Aug 2023 12:13:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51568 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232244AbjHHQMR (ORCPT ); Tue, 8 Aug 2023 12:12:17 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31F0D3AA1 for ; Tue, 8 Aug 2023 08:46:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691509590; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xmDg+NZ+45CiVZsrcRhVlp7iBoS4n+VtLARyxp0/jeQ=; b=VTBHpU+PoA/HSfc8+jnyKhjcp3I/AhRZom4LKdLZ+PUGqdeSTciHuzc7i9gXpGhND+j2Lj rXP8RV7FE0cz92JT+3l4oI07GvvBzB8J0VWB+cH4H82a5Bdc1RrKztA1+UP4ElsXVnV6YX Jg8EgI4OGkn7WFuN7vW+wLDC5oE1iE0= Received: from mimecast-mx02.redhat.com (66.187.233.73 [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-517-p5xl1a_TOn6Td8XWkcJAXw-1; Tue, 08 Aug 2023 06:43:09 -0400 X-MC-Unique: p5xl1a_TOn6Td8XWkcJAXw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EDE4C280FEC7; Tue, 8 Aug 2023 10:43:08 +0000 (UTC) Received: from localhost (unknown [10.72.120.3]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3417A2166B25; Tue, 8 Aug 2023 10:43:07 +0000 (UTC) From: Ming Lei To: Jens Axboe , Christoph Hellwig , linux-nvme@lists.infradead.org, "Martin K . Petersen" , linux-scsi@vger.kernel.org Cc: linux-block@vger.kernel.org, Wen Xiong , Keith Busch , Ming Lei , Justin Tee , Justin Tee , James Smart Subject: [PATCH V3 06/14] scsi: lpfc: use blk_mq_max_nr_hw_queues() to calculate io vectors Date: Tue, 8 Aug 2023 18:42:31 +0800 Message-Id: <20230808104239.146085-7-ming.lei@redhat.com> In-Reply-To: <20230808104239.146085-1-ming.lei@redhat.com> References: <20230808104239.146085-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Take blk-mq's knowledge into account for calculating io queues. Fix wrong queue mapping in case of kdump kernel. On arm and ppc64, 'maxcpus=1' is passed to kdump kernel command line, see `Documentation/admin-guide/kdump/kdump.rst`, so num_possible_cpus() still returns all CPUs because 'maxcpus=1' just bring up one single cpu core during booting. blk-mq sees single queue in kdump kernel, and in driver's viewpoint there are still multiple queues, this inconsistency causes driver to apply wrong queue mapping for handling IO, and IO timeout is triggered. Meantime, single queue makes much less resource utilization, and reduce risk of kernel failure. Reviewed-by: Justin Tee Cc: Justin Tee Cc: James Smart Signed-off-by: Ming Lei --- drivers/scsi/lpfc/lpfc_init.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/scsi/lpfc/lpfc_init.c b/drivers/scsi/lpfc/lpfc_init.c index 3221a934066b..961ff392f1c5 100644 --- a/drivers/scsi/lpfc/lpfc_init.c +++ b/drivers/scsi/lpfc/lpfc_init.c @@ -13025,6 +13025,8 @@ lpfc_sli4_enable_msix(struct lpfc_hba *phba) flags |= PCI_IRQ_AFFINITY; } + vectors = min_t(unsigned int, vectors, scsi_max_nr_hw_queues()); + rc = pci_alloc_irq_vectors(phba->pcidev, 1, vectors, flags); if (rc < 0) { lpfc_printf_log(phba, KERN_INFO, LOG_INIT, From patchwork Tue Aug 8 10:42:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13346274 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91784C05052 for ; Tue, 8 Aug 2023 16:13:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231972AbjHHQNW (ORCPT ); Tue, 8 Aug 2023 12:13:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40088 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232216AbjHHQMM (ORCPT ); Tue, 8 Aug 2023 12:12:12 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31F677A99 for ; Tue, 8 Aug 2023 08:46:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691509594; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2DxJS8RxHnnXfV+djc15xnTM9KjPIA9NavQgadLTDR8=; b=ThuTsKVBA37ROSvCE5/ZehqklxA3aQMm/nFjqxSwgDqDa0iiR7C5tun/rgBI4mk2FPgUn9 d9s4p8IniVrMYgCfYElXgkOIAWeYcFcocYYd9CarTWeTfotrdX5kUYwvg6NZdmCF0j+RmT hFDDE81xaVLGKXILck+j56bRNcpWaK0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-599-PuO5GixiPT2JItgpoi6aPg-1; Tue, 08 Aug 2023 06:43:12 -0400 X-MC-Unique: PuO5GixiPT2JItgpoi6aPg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2682885CCE2; Tue, 8 Aug 2023 10:43:12 +0000 (UTC) Received: from localhost (unknown [10.72.120.3]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4F450140E963; Tue, 8 Aug 2023 10:43:10 +0000 (UTC) From: Ming Lei To: Jens Axboe , Christoph Hellwig , linux-nvme@lists.infradead.org, "Martin K . Petersen" , linux-scsi@vger.kernel.org Cc: linux-block@vger.kernel.org, Wen Xiong , Keith Busch , Ming Lei , Sreekanth Reddy , Sathya Prakash Veerichetty , Kashyap Desai , Sumit Saxena Subject: [PATCH V3 07/14] scsi: mpi3mr: take blk_mq_max_nr_hw_queues() into account for calculating io vectors Date: Tue, 8 Aug 2023 18:42:32 +0800 Message-Id: <20230808104239.146085-8-ming.lei@redhat.com> In-Reply-To: <20230808104239.146085-1-ming.lei@redhat.com> References: <20230808104239.146085-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Take blk-mq's knowledge into account for calculating io queues. Fix wrong queue mapping in case of kdump kernel. On arm and ppc64, 'maxcpus=1' is passed to kdump kernel command line, see `Documentation/admin-guide/kdump/kdump.rst`, so num_possible_cpus() still returns all CPUs because 'maxcpus=1' just bring up one single cpu core during booting. blk-mq sees single queue in kdump kernel, and in driver's viewpoint there are still multiple queues, this inconsistency causes driver to apply wrong queue mapping for handling IO, and IO timeout is triggered. Meantime, single queue makes much less resource utilization, and reduce risk of kernel failure. Cc: Sreekanth Reddy Cc: Sathya Prakash Veerichetty Cc: Kashyap Desai Cc: Sumit Saxena Signed-off-by: Ming Lei --- drivers/scsi/mpi3mr/mpi3mr_fw.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/scsi/mpi3mr/mpi3mr_fw.c b/drivers/scsi/mpi3mr/mpi3mr_fw.c index 5fa07d6ee5b8..8fe14e728af6 100644 --- a/drivers/scsi/mpi3mr/mpi3mr_fw.c +++ b/drivers/scsi/mpi3mr/mpi3mr_fw.c @@ -815,6 +815,9 @@ static int mpi3mr_setup_isr(struct mpi3mr_ioc *mrioc, u8 setup_one) desc.post_vectors = mrioc->requested_poll_qcount; min_vec = desc.pre_vectors + desc.post_vectors; + if (max_vectors - min_vec > scsi_max_nr_hw_queues()) + max_vectors = min_vec + scsi_max_nr_hw_queues(); + irq_flags |= PCI_IRQ_AFFINITY | PCI_IRQ_ALL_TYPES; retval = pci_alloc_irq_vectors_affinity(mrioc->pdev, From patchwork Tue Aug 8 10:42:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13346273 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE7CCC001DB for ; Tue, 8 Aug 2023 16:13:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231622AbjHHQNV (ORCPT ); Tue, 8 Aug 2023 12:13:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46102 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231986AbjHHQL3 (ORCPT ); Tue, 8 Aug 2023 12:11:29 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D056F7D80 for ; Tue, 8 Aug 2023 08:46:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691509580; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ketWVMOak+Jss00Ni7DMSRSVvfoPSxigoCA6y+QxH1E=; b=jKh6uXq7BF4zTTwBOAcr5IHI2+YE7Rufx/tWRpQ5kHAzS5C7tSxGcaqkRHY+1n4Z6u35hY 6hUFg4ol2paYTnHEthhZLYCp9df+MgU6Drsj7iOFkgDvT1ulWM/hlngoz08pdNiZkY0Usa 9rZQEkJUZhBcUVgmIu6a7wt4mxk/7xk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-443-MxRzmOzlP2azCBUyTJYr2w-1; Tue, 08 Aug 2023 06:43:16 -0400 X-MC-Unique: MxRzmOzlP2azCBUyTJYr2w-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E0DF880557C; Tue, 8 Aug 2023 10:43:15 +0000 (UTC) Received: from localhost (unknown [10.72.120.3]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0D5A51121314; Tue, 8 Aug 2023 10:43:14 +0000 (UTC) From: Ming Lei To: Jens Axboe , Christoph Hellwig , linux-nvme@lists.infradead.org, "Martin K . Petersen" , linux-scsi@vger.kernel.org Cc: linux-block@vger.kernel.org, Wen Xiong , Keith Busch , Ming Lei , Sreekanth Reddy , Sathya Prakash Veerichetty , Kashyap Desai , Sumit Saxena Subject: [PATCH V3 08/14] scsi: megaraid: take blk_mq_max_nr_hw_queues() into account for calculating io vectors Date: Tue, 8 Aug 2023 18:42:33 +0800 Message-Id: <20230808104239.146085-9-ming.lei@redhat.com> In-Reply-To: <20230808104239.146085-1-ming.lei@redhat.com> References: <20230808104239.146085-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Take blk-mq's knowledge into account for calculating io queues. Fix wrong queue mapping in case of kdump kernel. On arm and ppc64, 'maxcpus=1' is passed to kdump kernel command line, see `Documentation/admin-guide/kdump/kdump.rst`, so num_possible_cpus() still returns all CPUs because 'maxcpus=1' just bring up one single cpu core during booting. blk-mq sees single queue in kdump kernel, and in driver's viewpoint there are still multiple queues, this inconsistency causes driver to apply wrong queue mapping for handling IO, and IO timeout is triggered. Meantime, single queue makes much less resource utilization, and reduce risk of kernel failure. Cc: Sreekanth Reddy Cc: Sathya Prakash Veerichetty Cc: Kashyap Desai Cc: Sumit Saxena Signed-off-by: Ming Lei --- drivers/scsi/megaraid/megaraid_sas_base.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/drivers/scsi/megaraid/megaraid_sas_base.c b/drivers/scsi/megaraid/megaraid_sas_base.c index 050eed8e2684..587ccbc1ec92 100644 --- a/drivers/scsi/megaraid/megaraid_sas_base.c +++ b/drivers/scsi/megaraid/megaraid_sas_base.c @@ -5921,6 +5921,10 @@ __megasas_alloc_irq_vectors(struct megasas_instance *instance) int i, irq_flags; struct irq_affinity desc = { .pre_vectors = instance->low_latency_index_start }; struct irq_affinity *descp = &desc; + unsigned max_vecs = instance->msix_vectors - instance->iopoll_q_count; + + if (max_vecs > scsi_max_nr_hw_queues()) + max_vecs = scsi_max_nr_hw_queues(); irq_flags = PCI_IRQ_MSIX; @@ -5934,7 +5938,7 @@ __megasas_alloc_irq_vectors(struct megasas_instance *instance) */ i = pci_alloc_irq_vectors_affinity(instance->pdev, instance->low_latency_index_start, - instance->msix_vectors - instance->iopoll_q_count, irq_flags, descp); + max_vecs, irq_flags, descp); return i; } From patchwork Tue Aug 8 10:42:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13346271 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 240EDC001B0 for ; Tue, 8 Aug 2023 16:13:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232108AbjHHQMt (ORCPT ); Tue, 8 Aug 2023 12:12:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43820 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231371AbjHHQLY (ORCPT ); Tue, 8 Aug 2023 12:11:24 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E443B3A81 for ; Tue, 8 Aug 2023 08:46:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691509578; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jmP+yqN1CMl4seuzyOdC4zCzUZWNFx4ZDmn6s0/7WWk=; b=Jsf9oT2dLPdUiTsJQNZ0BUo+q/dWB2CrdZzPdXmTf8qzfGf6+/qYQjqOSgjIwVl3TMZvKJ oEyzr4/FcaIe7XbFOSjSuQFbiUNNQ8NTzAcg6+CdvKgZ50JmhofbfBH3YSIX3qKyCq4hD4 QC3GzSJAVxJJMo02+sUEX/Qlv/4YwAI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-642-wIauOLLpNZKM2kg1dDb2WQ-1; Tue, 08 Aug 2023 06:43:19 -0400 X-MC-Unique: wIauOLLpNZKM2kg1dDb2WQ-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 201908DC666; Tue, 8 Aug 2023 10:43:19 +0000 (UTC) Received: from localhost (unknown [10.72.120.3]) by smtp.corp.redhat.com (Postfix) with ESMTP id 54067492C13; Tue, 8 Aug 2023 10:43:17 +0000 (UTC) From: Ming Lei To: Jens Axboe , Christoph Hellwig , linux-nvme@lists.infradead.org, "Martin K . Petersen" , linux-scsi@vger.kernel.org Cc: linux-block@vger.kernel.org, Wen Xiong , Keith Busch , Ming Lei , Sathya Prakash , Sreekanth Reddy , Suganath Prabu Subramani Subject: [PATCH V3 09/14] scsi: mpt3sas: take blk_mq_max_nr_hw_queues() into account for calculating io vectors Date: Tue, 8 Aug 2023 18:42:34 +0800 Message-Id: <20230808104239.146085-10-ming.lei@redhat.com> In-Reply-To: <20230808104239.146085-1-ming.lei@redhat.com> References: <20230808104239.146085-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Take blk-mq's knowledge into account for calculating io queues. Fix wrong queue mapping in case of kdump kernel. On arm and ppc64, 'maxcpus=1' is passed to kdump kernel command line, see `Documentation/admin-guide/kdump/kdump.rst`, so num_possible_cpus() still returns all CPUs because 'maxcpus=1' just bring up one single cpu core during booting. blk-mq sees single queue in kdump kernel, and in driver's viewpoint there are still multiple queues, this inconsistency causes driver to apply wrong queue mapping for handling IO, and IO timeout is triggered. Meantime, single queue makes much less resource utilization, and reduce risk of kernel failure. Cc: Sathya Prakash Cc: Sreekanth Reddy Cc: Suganath Prabu Subramani Signed-off-by: Ming Lei --- drivers/scsi/mpt3sas/mpt3sas_base.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c index 53f5492579cb..d238e0275edd 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.c +++ b/drivers/scsi/mpt3sas/mpt3sas_base.c @@ -3332,8 +3332,8 @@ _base_alloc_irq_vectors(struct MPT3SAS_ADAPTER *ioc) * Don't allocate msix vectors for poll_queues. * msix_vectors is always within a range of FW supported reply queue. */ - int nr_msix_vectors = ioc->iopoll_q_start_index; - + int nr_msix_vectors = min_t(unsigned int, ioc->iopoll_q_start_index, + scsi_max_nr_hw_queues()); if (ioc->smp_affinity_enable) irq_flags |= PCI_IRQ_AFFINITY | PCI_IRQ_ALL_TYPES; From patchwork Tue Aug 8 10:42:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13346249 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1E60C41513 for ; Tue, 8 Aug 2023 16:11:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232070AbjHHQLj (ORCPT ); Tue, 8 Aug 2023 12:11:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32886 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231959AbjHHQJh (ORCPT ); Tue, 8 Aug 2023 12:09:37 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A374D7A84 for ; Tue, 8 Aug 2023 08:46:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691509567; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=G5vWqgVWZ30SumBVkEoD+1IKC/3TIbPs2fN+ccFAG78=; b=Qc3mQm44NdPKkw+rmI4arLzhuWmPRklf1CDFz/dVftSHcNKiojpkCrPsOwypamLFIRju1E Aw3+P9x8ouI2gSaslQsPCHiIkvuxJTnX5C+IKLTbTgFdwIRBRjDyOIusd3ldXmfNlYdLwJ ZRQ0M9U+W6EofBY8bywfp/IW0LFgdSI= Received: from mimecast-mx02.redhat.com (66.187.233.73 [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-221-vLpqu9fwMnezhqdyB-j8NQ-1; Tue, 08 Aug 2023 06:43:23 -0400 X-MC-Unique: vLpqu9fwMnezhqdyB-j8NQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F2C6F3815EF3; Tue, 8 Aug 2023 10:43:22 +0000 (UTC) Received: from localhost (unknown [10.72.120.3]) by smtp.corp.redhat.com (Postfix) with ESMTP id 13BA11121314; Tue, 8 Aug 2023 10:43:21 +0000 (UTC) From: Ming Lei To: Jens Axboe , Christoph Hellwig , linux-nvme@lists.infradead.org, "Martin K . Petersen" , linux-scsi@vger.kernel.org Cc: linux-block@vger.kernel.org, Wen Xiong , Keith Busch , Ming Lei , Jack Wang Subject: [PATCH V3 10/14] scsi: pm8001: take blk_mq_max_nr_hw_queues() into account for calculating io vectors Date: Tue, 8 Aug 2023 18:42:35 +0800 Message-Id: <20230808104239.146085-11-ming.lei@redhat.com> In-Reply-To: <20230808104239.146085-1-ming.lei@redhat.com> References: <20230808104239.146085-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Take blk-mq's knowledge into account for calculating io queues. Fix wrong queue mapping in case of kdump kernel. On arm and ppc64, 'maxcpus=1' is passed to kdump kernel command line, see `Documentation/admin-guide/kdump/kdump.rst`, so num_possible_cpus() still returns all CPUs because 'maxcpus=1' just bring up one single cpu core during booting. blk-mq sees single queue in kdump kernel, and in driver's viewpoint there are still multiple queues, this inconsistency causes driver to apply wrong queue mapping for handling IO, and IO timeout is triggered. Meantime, single queue makes much less resource utilization, and reduce risk of kernel failure. Acked-by: Jack Wang Signed-off-by: Ming Lei --- drivers/scsi/pm8001/pm8001_init.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/scsi/pm8001/pm8001_init.c b/drivers/scsi/pm8001/pm8001_init.c index 4995e1ef4e0e..220ae504e5a5 100644 --- a/drivers/scsi/pm8001/pm8001_init.c +++ b/drivers/scsi/pm8001/pm8001_init.c @@ -965,6 +965,8 @@ static u32 pm8001_setup_msix(struct pm8001_hba_info *pm8001_ha) rc = pci_alloc_irq_vectors(pm8001_ha->pdev, 1, 1, PCI_IRQ_MSIX); } else { + unsigned int max_vecs = min_t(unsigned int, PM8001_MAX_MSIX_VEC, + scsi_max_nr_hw_queues() + 1); /* * Queue index #0 is used always for housekeeping, so don't * include in the affinity spreading. @@ -973,7 +975,7 @@ static u32 pm8001_setup_msix(struct pm8001_hba_info *pm8001_ha) .pre_vectors = 1, }; rc = pci_alloc_irq_vectors_affinity( - pm8001_ha->pdev, 2, PM8001_MAX_MSIX_VEC, + pm8001_ha->pdev, 2, max_vecs, PCI_IRQ_MSIX | PCI_IRQ_AFFINITY, &desc); } From patchwork Tue Aug 8 10:42:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13346233 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D07D9C001DE for ; Tue, 8 Aug 2023 16:05:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231635AbjHHQFI (ORCPT ); Tue, 8 Aug 2023 12:05:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34312 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231637AbjHHQDe (ORCPT ); Tue, 8 Aug 2023 12:03:34 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A87961B6 for ; Tue, 8 Aug 2023 08:44:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691509476; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=O+NLl0HWC1rjA2zoBp3evkIVtVVCQOGJ99N11skgClg=; b=f9NSJN+Ec8PS8BeK6vBpcQbhgOVLjVz1E0/H+DECSo6lUaVWwU7GIsmgEZf4UJON0nkPkP kbGsIAcZ4xPoV6XjRoEcv0QLlRSuROzfkgVcoKk1V2yf+6dz05hB7qx7HBVQQ+orVvsyVc JFWShIM2K83sPOBh0Dqi3KDuxGHujWY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-287-MhQueYY_PJC-llP_5SeNaw-1; Tue, 08 Aug 2023 06:43:27 -0400 X-MC-Unique: MhQueYY_PJC-llP_5SeNaw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7404B801FA0; Tue, 8 Aug 2023 10:43:26 +0000 (UTC) Received: from localhost (unknown [10.72.120.3]) by smtp.corp.redhat.com (Postfix) with ESMTP id 69ED140C2077; Tue, 8 Aug 2023 10:43:25 +0000 (UTC) From: Ming Lei To: Jens Axboe , Christoph Hellwig , linux-nvme@lists.infradead.org, "Martin K . Petersen" , linux-scsi@vger.kernel.org Cc: linux-block@vger.kernel.org, Wen Xiong , Keith Busch , Ming Lei , John Garry , Xiang Chen Subject: [PATCH V3 11/14] scsi: hisi: take blk_mq_max_nr_hw_queues() into account for calculating io vectors Date: Tue, 8 Aug 2023 18:42:36 +0800 Message-Id: <20230808104239.146085-12-ming.lei@redhat.com> In-Reply-To: <20230808104239.146085-1-ming.lei@redhat.com> References: <20230808104239.146085-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Take blk-mq's knowledge into account for calculating io queues. Fix wrong queue mapping in case of kdump kernel. On arm and ppc64, 'maxcpus=1' is passed to kdump kernel command line, see `Documentation/admin-guide/kdump/kdump.rst`, so num_possible_cpus() still returns all CPUs because 'maxcpus=1' just bring up one single cpu core during booting. blk-mq sees single queue in kdump kernel, and in driver's viewpoint there are still multiple queues, this inconsistency causes driver to apply wrong queue mapping for handling IO, and IO timeout is triggered. Meantime, single queue makes much less resource utilization, and reduce risk of kernel failure. Cc: John Garry Cc: Xiang Chen Signed-off-by: Ming Lei --- drivers/scsi/hisi_sas/hisi_sas_v3_hw.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c b/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c index 20e1607c6282..60d2301e7f9d 100644 --- a/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c +++ b/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c @@ -2550,6 +2550,9 @@ static int interrupt_preinit_v3_hw(struct hisi_hba *hisi_hba) hisi_hba->cq_nvecs = vectors - BASE_VECTORS_V3_HW - hisi_hba->iopoll_q_cnt; + if (hisi_hba->cq_nvecs > scsi_max_nr_hw_queues()) + hisi_hba->cq_nvecs = scsi_max_nr_hw_queues(); + shost->nr_hw_queues = hisi_hba->cq_nvecs + hisi_hba->iopoll_q_cnt; return devm_add_action(&pdev->dev, hisi_sas_v3_free_vectors, pdev); From patchwork Tue Aug 8 10:42:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13346251 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1945C001DB for ; Tue, 8 Aug 2023 16:11:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232082AbjHHQLq (ORCPT ); Tue, 8 Aug 2023 12:11:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32870 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229823AbjHHQJp (ORCPT ); Tue, 8 Aug 2023 12:09:45 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7D9CA7A83 for ; Tue, 8 Aug 2023 08:46:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691509567; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cH8+J+v5eCSEAo5zzPYOsTrKPWy+zXGNhCQOIixYmz0=; b=hEWW1uGTe+46JA2K7EtdqN8NNzv49+gZ8PjIEjr3Ukg8FDmNUNn+NLWUK/ucVmvwWNVVPD pzSedgcNk/jOHF+lP2UqsezxvEz2s1gOXDE+J1uf+7XlxHAf4YzbAZwvXCJfmyCGzK8JDY l2IRjU6HBtldgGIvlVlfBwuDX2Ps8UI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-694-ORIgDiMuM2yLXyVmCPXrgg-1; Tue, 08 Aug 2023 06:43:30 -0400 X-MC-Unique: ORIgDiMuM2yLXyVmCPXrgg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 363D685CCE1; Tue, 8 Aug 2023 10:43:30 +0000 (UTC) Received: from localhost (unknown [10.72.120.3]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5FD0C140E962; Tue, 8 Aug 2023 10:43:29 +0000 (UTC) From: Ming Lei To: Jens Axboe , Christoph Hellwig , linux-nvme@lists.infradead.org, "Martin K . Petersen" , linux-scsi@vger.kernel.org Cc: linux-block@vger.kernel.org, Wen Xiong , Keith Busch , Ming Lei Subject: [PATCH V3 12/14] scsi: ufs: limit max allowed nr_hw_queues Date: Tue, 8 Aug 2023 18:42:37 +0800 Message-Id: <20230808104239.146085-13-ming.lei@redhat.com> In-Reply-To: <20230808104239.146085-1-ming.lei@redhat.com> References: <20230808104239.146085-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Take blk-mq's knowledge into account for calculating io queues. Fix wrong queue mapping in case of kdump kernel. On arm and ppc64, 'maxcpus=1' is passed to kdump command line, see `Documentation/admin-guide/kdump/kdump.rst`, so num_possible_cpus() still returns all CPUs because 'maxcpus=1' just bring up one single cpu core during booting. blk-mq sees single queue in kdump kernel, and in driver's viewpoint there are still multiple queues, this inconsistency causes driver to apply wrong queue mapping for handling IO, and IO timeout is triggered. Meantime, single queue makes much less resource utilization, and reduce risk of kernel failure. Signed-off-by: Ming Lei --- drivers/ufs/core/ufs-mcq.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/ufs/core/ufs-mcq.c b/drivers/ufs/core/ufs-mcq.c index 6fb0e007af63..7d808836b018 100644 --- a/drivers/ufs/core/ufs-mcq.c +++ b/drivers/ufs/core/ufs-mcq.c @@ -186,7 +186,7 @@ static int ufshcd_mcq_config_nr_queues(struct ufs_hba *hba) if (!hba->nr_queues[HCTX_TYPE_DEFAULT]) hba->nr_queues[HCTX_TYPE_DEFAULT] = min3(rem, rw_queues, - num_possible_cpus()); + scsi_max_nr_hw_queues()); for (i = 0; i < HCTX_MAX_TYPES; i++) host->nr_hw_queues += hba->nr_queues[i]; From patchwork Tue Aug 8 10:42:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13346365 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5911CC001DE for ; Tue, 8 Aug 2023 16:33:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232953AbjHHQdh (ORCPT ); Tue, 8 Aug 2023 12:33:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58070 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232988AbjHHQcp (ORCPT ); Tue, 8 Aug 2023 12:32:45 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C0E1C31007 for ; Tue, 8 Aug 2023 08:51:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691509863; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2TEtH5yVIZkcHhXlx7zXOq0kKCJ4wWHrHIE0jkzlzuw=; b=SG38S9sH9NxrPtGfu1M2GpEk9LKC2SwJZR6LY5Wqw9y1aigjNCgmfs/F2S+Ya+9Z7S8KyO E/aWP3Gi6+DdDTELP0bOW24CXJblox5I/RnV13rJS8ze5aXdmehBCeG3/faqbtOqker2/I 41O+2obluAlhGX2q8dNSwRXakfpFoCk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-90-leUcPBTANJW2QpPc0r0CxA-1; Tue, 08 Aug 2023 06:43:33 -0400 X-MC-Unique: leUcPBTANJW2QpPc0r0CxA-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 604A785CCE1; Tue, 8 Aug 2023 10:43:33 +0000 (UTC) Received: from localhost (unknown [10.72.120.3]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9C31B140E962; Tue, 8 Aug 2023 10:43:32 +0000 (UTC) From: Ming Lei To: Jens Axboe , Christoph Hellwig , linux-nvme@lists.infradead.org, "Martin K . Petersen" , linux-scsi@vger.kernel.org Cc: linux-block@vger.kernel.org, Wen Xiong , Keith Busch , Ming Lei Subject: [PATCH V3 13/14] scsi: storvsc: limit max allowed nr_hw_queues Date: Tue, 8 Aug 2023 18:42:38 +0800 Message-Id: <20230808104239.146085-14-ming.lei@redhat.com> In-Reply-To: <20230808104239.146085-1-ming.lei@redhat.com> References: <20230808104239.146085-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Take blk-mq's knowledge into account for calculating io queues. Fix wrong queue mapping in case of kdump kernel. On arm and ppc64, 'maxcpus=1' is passed to kdump command line, see `Documentation/admin-guide/kdump/kdump.rst`, so num_possible_cpus() still returns all CPUs because 'maxcpus=1' just bring up one single cpu core during booting. blk-mq sees single queue in kdump kernel, and in driver's viewpoint there are still multiple queues, this inconsistency causes driver to apply wrong queue mapping for handling IO, and IO timeout is triggered. Meantime, single queue makes much less resource utilization, and reduce risk of kernel failure. Signed-off-by: Ming Lei --- drivers/scsi/storvsc_drv.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/scsi/storvsc_drv.c b/drivers/scsi/storvsc_drv.c index f2823218670a..50df040b100d 100644 --- a/drivers/scsi/storvsc_drv.c +++ b/drivers/scsi/storvsc_drv.c @@ -2040,6 +2040,9 @@ static int storvsc_probe(struct hv_device *device, host->nr_hw_queues = storvsc_max_hw_queues; else host->nr_hw_queues = num_present_cpus; + if (host->nr_hw_queues > scsi_max_nr_hw_queues()) + host->nr_hw_queues = scsi_max_nr_hw_queues(); + } /* From patchwork Tue Aug 8 10:42:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13346248 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C341C001DB for ; Tue, 8 Aug 2023 16:09:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232018AbjHHQJz (ORCPT ); Tue, 8 Aug 2023 12:09:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57000 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231757AbjHHQIH (ORCPT ); Tue, 8 Aug 2023 12:08:07 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 08CFD768D for ; Tue, 8 Aug 2023 08:46:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1691509537; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3+EnSRWzHYRgAwaPfCvnjx1xuC05RKXl9LNASqVbR8A=; b=GUk9nE2rk0+gzS9M01TAarR+gCYwrich9dRn+rGMDfI+OBqt6ou5n8oETVWB03Tl/or+uD UbU+hk9hnZnc3e3JPxdiZM1VLujQes2zCwLwQCehtryR8hhaKVY6tmBt33fSqrlxYH6xll UdfvZaM9Xl0qpyfE5zEVJ9oQmWNF1U4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-219-u2FxbA53OHqCOhATeSOWcQ-1; Tue, 08 Aug 2023 06:43:37 -0400 X-MC-Unique: u2FxbA53OHqCOhATeSOWcQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DB4BC8007CE; Tue, 8 Aug 2023 10:43:36 +0000 (UTC) Received: from localhost (unknown [10.72.120.3]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1547A140E962; Tue, 8 Aug 2023 10:43:35 +0000 (UTC) From: Ming Lei To: Jens Axboe , Christoph Hellwig , linux-nvme@lists.infradead.org, "Martin K . Petersen" , linux-scsi@vger.kernel.org Cc: linux-block@vger.kernel.org, Wen Xiong , Keith Busch , Ming Lei Subject: [PATCH V3 14/14] blk-mq: add helpers for treating kdump kernel Date: Tue, 8 Aug 2023 18:42:39 +0800 Message-Id: <20230808104239.146085-15-ming.lei@redhat.com> In-Reply-To: <20230808104239.146085-1-ming.lei@redhat.com> References: <20230808104239.146085-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Clean up code a bit by adding helpers for treating kdump kernel specially. Suggested-by: Christoph Hellwig Signed-off-by: Ming Lei --- block/blk-mq.c | 39 +++++++++++++++++++++++++++------------ 1 file changed, 27 insertions(+), 12 deletions(-) diff --git a/block/blk-mq.c b/block/blk-mq.c index 617d6f849a7b..afa51df2f0d3 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -147,6 +147,8 @@ EXPORT_SYMBOL_GPL(blk_mq_freeze_queue_wait); * driver has to take blk-mq max supported nr_hw_queues into account * when figuring out nr_hw_queues from hardware info, for avoiding * inconsistency between driver and blk-mq. + * + * Limit to single queue in case of kdump kernel */ unsigned int blk_mq_max_nr_hw_queues(void) { @@ -4370,7 +4372,7 @@ static void blk_mq_update_queue_map(struct blk_mq_tag_set *set) if (set->nr_maps == 1) set->map[HCTX_TYPE_DEFAULT].nr_queues = set->nr_hw_queues; - if (set->ops->map_queues && !is_kdump_kernel()) { + if (set->ops->map_queues) { int i; /* @@ -4420,6 +4422,22 @@ static int blk_mq_realloc_tag_set_tags(struct blk_mq_tag_set *set, return 0; } +/* Limit to single map in case of kdump kernel */ +static unsigned int blk_mq_max_nr_maps(void) +{ + if (is_kdump_kernel()) + return 1; + return HCTX_MAX_TYPES; +} + +/* Limit to 64 in case of kdump kernel */ +static unsigned int blk_mq_max_depth(void) +{ + if (is_kdump_kernel()) + return 64; + return BLK_MQ_MAX_DEPTH; +} + /* * Alloc a tag set to be associated with one or more request queues. * May fail with EINVAL for various error conditions. May adjust the @@ -4456,16 +4474,13 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set) else if (set->nr_maps > HCTX_MAX_TYPES) return -EINVAL; - /* - * If a crashdump is active, then we are potentially in a very - * memory constrained environment. Limit us to 1 queue and - * 64 tags to prevent using too much memory. - */ - if (is_kdump_kernel()) { - set->nr_hw_queues = 1; - set->nr_maps = 1; - set->queue_depth = min(64U, set->queue_depth); - } + if (set->nr_hw_queues > blk_mq_max_nr_hw_queues()) + set->nr_hw_queues = blk_mq_max_nr_hw_queues(); + if (set->nr_maps > blk_mq_max_nr_maps()) + set->nr_maps = blk_mq_max_nr_maps(); + if (set->queue_depth > blk_mq_max_depth()) + set->queue_depth = blk_mq_max_depth(); + /* * There is no use for more h/w queues than cpus if we just have * a single map @@ -4495,7 +4510,7 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set) GFP_KERNEL, set->numa_node); if (!set->map[i].mq_map) goto out_free_mq_map; - set->map[i].nr_queues = is_kdump_kernel() ? 1 : set->nr_hw_queues; + set->map[i].nr_queues = set->nr_hw_queues; } blk_mq_update_queue_map(set);