From patchwork Wed Jul 12 12:54:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13310333 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03FABEB64DA for ; Wed, 12 Jul 2023 12:55:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232421AbjGLMzz (ORCPT ); Wed, 12 Jul 2023 08:55:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34540 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232202AbjGLMzy (ORCPT ); Wed, 12 Jul 2023 08:55:54 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AD3A61986 for ; Wed, 12 Jul 2023 05:55:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689166509; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=W8fid7Se3iKV/DWozR9uU+XgmFzc7Pg3ogm0vbx3SeA=; b=HDiwPvrRi/wbOow3isedpOaiw1Hu9AL4SnrkXHvLx5M8FxUgOIdogLIi9sQbU16lBo32IV QW6B4zKDLe/uFtoWVRmkYrVWQRayi6u0/kJQCO6ZN5Ntqr8qOXK/91py7xO/XMNHUcL67W F/x3Zbk+hemO/9RrTc6tbiWGsH3ZTCg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-14-xxHbe_10P9uVjg_Q3woLZA-1; Wed, 12 Jul 2023 08:55:07 -0400 X-MC-Unique: xxHbe_10P9uVjg_Q3woLZA-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 24A3F8FBA26; Wed, 12 Jul 2023 12:55:07 +0000 (UTC) Received: from localhost (ovpn-8-25.pek2.redhat.com [10.72.8.25]) by smtp.corp.redhat.com (Postfix) with ESMTP id E80FEF66C6; Wed, 12 Jul 2023 12:55:05 +0000 (UTC) From: Ming Lei To: Jens Axboe , Christoph Hellwig , linux-nvme@lists.infradead.org, "Martin K . Petersen" , linux-scsi@vger.kernel.org Cc: linux-block@vger.kernel.org, Wen Xiong , Keith Busch , Ming Lei Subject: [PATCH 1/8] blk-mq: add blk_mq_max_nr_hw_queues() Date: Wed, 12 Jul 2023 20:54:48 +0800 Message-Id: <20230712125455.1986455-2-ming.lei@redhat.com> In-Reply-To: <20230712125455.1986455-1-ming.lei@redhat.com> References: <20230712125455.1986455-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org blk_mq_alloc_tag_set() may override set->nr_hw_queues as 1 in case of kdump kernel. This way causes trouble for driver, because blk-mq and driver see different queue mapping. Especially the only online CPU may not be 1 for kdump kernel, in which 'maxcpus=1' is passed from kernel command line, then driver may map hctx0 into one inactive real hw queue which cpu affinity is 0(offline). The issue exists on all drivers which use managed irq and support multiple hw queue. Prepare for fixing this kind of issue by applying the added helper, so driver can take blk-mq max nr_hw_queues knowledge into account when calculating io queues. Signed-off-by: Ming Lei --- block/blk-mq.c | 9 +++++++++ include/linux/blk-mq.h | 1 + 2 files changed, 10 insertions(+) diff --git a/block/blk-mq.c b/block/blk-mq.c index 5504719b970d..b764da69a416 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -140,6 +140,15 @@ void blk_mq_freeze_queue_wait(struct request_queue *q) } EXPORT_SYMBOL_GPL(blk_mq_freeze_queue_wait); +/* Max nr_hw_queues for each hw queue type */ +unsigned int blk_mq_max_nr_hw_queues(void) +{ + if (is_kdump_kernel()) + return 1; + return nr_cpu_ids; +} +EXPORT_SYMBOL_GPL(blk_mq_max_nr_hw_queues); + int blk_mq_freeze_queue_wait_timeout(struct request_queue *q, unsigned long timeout) { diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 2b7fb8e87793..2407978fbc30 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -713,6 +713,7 @@ int blk_mq_alloc_sq_tag_set(struct blk_mq_tag_set *set, const struct blk_mq_ops *ops, unsigned int queue_depth, unsigned int set_flags); void blk_mq_free_tag_set(struct blk_mq_tag_set *set); +unsigned int blk_mq_max_nr_hw_queues(void); void blk_mq_free_request(struct request *rq); int blk_rq_poll(struct request *rq, struct io_comp_batch *iob, From patchwork Wed Jul 12 12:54:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13310334 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8AD37EB64D9 for ; Wed, 12 Jul 2023 12:56:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232506AbjGLM4I (ORCPT ); Wed, 12 Jul 2023 08:56:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34576 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232463AbjGLM4E (ORCPT ); Wed, 12 Jul 2023 08:56:04 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D04411982 for ; Wed, 12 Jul 2023 05:55:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689166516; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=b8qT0SCfP03LYvevv18i58qSjcd5shmc7x0amzf2CBc=; b=E/kqzs50NpUgo+iiC+8RZ3tDy5Z+zcYtzkF9kGAkAnRsQ/5k5JyBt0OqP0NJ079lWXVGUq e2FJ+bhmZ1EHDPPj5j7W5tdLissc1RuMEXmEUgTTVJoGDZcuAceF/fla27X9Ko505R9b0A uhqHY/+fKlLSFhXKsHt7PKXHNQAxQKY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-575-5karMAE_NmenoVqLsch49A-1; Wed, 12 Jul 2023 08:55:11 -0400 X-MC-Unique: 5karMAE_NmenoVqLsch49A-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 28C4B104D513; Wed, 12 Jul 2023 12:55:11 +0000 (UTC) Received: from localhost (ovpn-8-25.pek2.redhat.com [10.72.8.25]) by smtp.corp.redhat.com (Postfix) with ESMTP id 56A5E4067A00; Wed, 12 Jul 2023 12:15:06 +0000 (UTC) From: Ming Lei To: Jens Axboe , Christoph Hellwig , linux-nvme@lists.infradead.org, "Martin K . Petersen" , linux-scsi@vger.kernel.org Cc: linux-block@vger.kernel.org, Wen Xiong , Keith Busch , Ming Lei Subject: [PATCH 2/8] nvme-pci: use blk_mq_max_nr_hw_queues() to calculate io queues Date: Wed, 12 Jul 2023 20:54:49 +0800 Message-Id: <20230712125455.1986455-3-ming.lei@redhat.com> In-Reply-To: <20230712125455.1986455-1-ming.lei@redhat.com> References: <20230712125455.1986455-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Take blk-mq's knowledge into account for calculating io queues. Fix wrong queue mapping in case of kdump kernel. On arm and ppc64, 'maxcpus=1' is passed to kdump command line, see `Documentation/admin-guide/kdump/kdump.rst`, so num_possible_cpus() still returns all CPUs because 'maxcpus=1' just bring up one single cpu core during booting. blk-mq sees single queue in kdump kernel, and in driver's viewpoint there are still multiple queues, this inconsistency causes driver to apply wrong queue mapping for handling IO, and IO timeout is triggered. Meantime, single queue makes much less resource utilization, and reduce risk of kernel failure. Reported-by: Wen Xiong Signed-off-by: Ming Lei --- drivers/nvme/host/pci.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 72725729cb6c..cb13ba203956 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -2247,7 +2247,7 @@ static unsigned int nvme_max_io_queues(struct nvme_dev *dev) */ if (dev->ctrl.quirks & NVME_QUIRK_SHARED_TAGS) return 1; - return num_possible_cpus() + dev->nr_write_queues + dev->nr_poll_queues; + return blk_mq_max_nr_hw_queues() + dev->nr_write_queues + dev->nr_poll_queues; } static int nvme_setup_io_queues(struct nvme_dev *dev) From patchwork Wed Jul 12 12:54:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13310335 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A657DEB64D9 for ; Wed, 12 Jul 2023 12:56:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232202AbjGLM4L (ORCPT ); Wed, 12 Jul 2023 08:56:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34560 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232306AbjGLM4I (ORCPT ); Wed, 12 Jul 2023 08:56:08 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A9FD1981 for ; Wed, 12 Jul 2023 05:55:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689166518; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YJ4imw3iHzN6mbjqWrKiopdQuaapVjZy4MvSNpK4PGw=; b=Zrtac84Bfb/mVVEC1dilDpK4vvypzzqrA3skzGIUIWeekc+o2n8ju5JjzYDVPE7aoNy/HC wOTZtg2QD4Ex7YEumCXnmCOLOuS6SjCaidqizZOOLFKxi0NZTRpa576jSLye/yE18a03Ys fqsYufUh1Gu1F9KG3EjEeGZMHYS4FwY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-139-V3DQFlR7NK6gTouM1E1UAg-1; Wed, 12 Jul 2023 08:55:15 -0400 X-MC-Unique: V3DQFlR7NK6gTouM1E1UAg-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id DB5AC10504AA; Wed, 12 Jul 2023 12:55:14 +0000 (UTC) Received: from localhost (ovpn-8-25.pek2.redhat.com [10.72.8.25]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0338A1454142; Wed, 12 Jul 2023 12:55:13 +0000 (UTC) From: Ming Lei To: Jens Axboe , Christoph Hellwig , linux-nvme@lists.infradead.org, "Martin K . Petersen" , linux-scsi@vger.kernel.org Cc: linux-block@vger.kernel.org, Wen Xiong , Keith Busch , Ming Lei , James Smart Subject: [PATCH 3/8] scsi: lpfc: use blk_mq_max_nr_hw_queues() to calculate io vectors Date: Wed, 12 Jul 2023 20:54:50 +0800 Message-Id: <20230712125455.1986455-4-ming.lei@redhat.com> In-Reply-To: <20230712125455.1986455-1-ming.lei@redhat.com> References: <20230712125455.1986455-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Take blk-mq's knowledge into account for calculating io queues. Fix wrong queue mapping in case of kdump kernel. On arm and ppc64, 'maxcpus=1' is passed to kdump kernel command line, see `Documentation/admin-guide/kdump/kdump.rst`, so num_possible_cpus() still returns all CPUs because 'maxcpus=1' just bring up one single cpu core during booting. blk-mq sees single queue in kdump kernel, and in driver's viewpoint there are still multiple queues, this inconsistency causes driver to apply wrong queue mapping for handling IO, and IO timeout is triggered. Meantime, single queue makes much less resource utilization, and reduce risk of kernel failure. Cc: James Smart Signed-off-by: Ming Lei --- drivers/scsi/lpfc/lpfc_init.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/scsi/lpfc/lpfc_init.c b/drivers/scsi/lpfc/lpfc_init.c index 3221a934066b..88314c3f1dc1 100644 --- a/drivers/scsi/lpfc/lpfc_init.c +++ b/drivers/scsi/lpfc/lpfc_init.c @@ -13012,6 +13012,7 @@ lpfc_sli4_enable_msix(struct lpfc_hba *phba) if (phba->irq_chann_mode != NORMAL_MODE) aff_mask = &phba->sli4_hba.irq_aff_mask; + vectors = min_t(unsigned int, vectors, blk_mq_max_nr_hw_queues()); if (aff_mask) { cpu_cnt = cpumask_weight(aff_mask); vectors = min(phba->cfg_irq_chann, cpu_cnt); From patchwork Wed Jul 12 12:54:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13310336 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A538EB64DD for ; Wed, 12 Jul 2023 12:56:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231219AbjGLM4M (ORCPT ); Wed, 12 Jul 2023 08:56:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34618 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232448AbjGLM4K (ORCPT ); Wed, 12 Jul 2023 08:56:10 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8EC23198B for ; Wed, 12 Jul 2023 05:55:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689166523; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qRXG76wkhKDOmusO4D8ben9Sp69zHeCOETl4h69/8KI=; b=KLiI0+61R/6CuVOV7OQIfS4bbmUyvSEyn4s8QGleBPTEWA9ecocSCq8tgXkKMaHq22dzs1 NOKqXVeNiJ3pHODcKMmzHsi6S1FX8rMsg0v8nSt2ukz3o0v8wXHBsCRc8AEUWjHcupGu03 jl0WcWYJcVKSEJ0aI2KUiLW/usNpqVY= Received: from mimecast-mx02.redhat.com (66.187.233.73 [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-658-foZJhVnfNHeMWGzZzB-qfg-1; Wed, 12 Jul 2023 08:55:19 -0400 X-MC-Unique: foZJhVnfNHeMWGzZzB-qfg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 00813299E741; Wed, 12 Jul 2023 12:55:19 +0000 (UTC) Received: from localhost (ovpn-8-25.pek2.redhat.com [10.72.8.25]) by smtp.corp.redhat.com (Postfix) with ESMTP id D6381F66C6; Wed, 12 Jul 2023 12:55:17 +0000 (UTC) From: Ming Lei To: Jens Axboe , Christoph Hellwig , linux-nvme@lists.infradead.org, "Martin K . Petersen" , linux-scsi@vger.kernel.org Cc: linux-block@vger.kernel.org, Wen Xiong , Keith Busch , Ming Lei , Xiang Chen Subject: [PATCH 4/8] scsi: hisi: take blk_mq_max_nr_hw_queues() into account for calculating io vectors Date: Wed, 12 Jul 2023 20:54:51 +0800 Message-Id: <20230712125455.1986455-5-ming.lei@redhat.com> In-Reply-To: <20230712125455.1986455-1-ming.lei@redhat.com> References: <20230712125455.1986455-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Take blk-mq's knowledge into account for calculating io queues. Fix wrong queue mapping in case of kdump kernel. On arm and ppc64, 'maxcpus=1' is passed to kdump kernel command line, see `Documentation/admin-guide/kdump/kdump.rst`, so num_possible_cpus() still returns all CPUs because 'maxcpus=1' just bring up one single cpu core during booting. blk-mq sees single queue in kdump kernel, and in driver's viewpoint there are still multiple queues, this inconsistency causes driver to apply wrong queue mapping for handling IO, and IO timeout is triggered. Meantime, single queue makes much less resource utilization, and reduce risk of kernel failure. Cc: Xiang Chen Signed-off-by: Ming Lei --- drivers/scsi/hisi_sas/hisi_sas_v3_hw.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c b/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c index 20e1607c6282..bbec11220286 100644 --- a/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c +++ b/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c @@ -2550,6 +2550,9 @@ static int interrupt_preinit_v3_hw(struct hisi_hba *hisi_hba) hisi_hba->cq_nvecs = vectors - BASE_VECTORS_V3_HW - hisi_hba->iopoll_q_cnt; + if (hisi_hba->cq_nvecs > blk_mq_max_nr_hw_queues()) + hisi_hba->cq_nvecs = blk_mq_max_nr_hw_queues(); + shost->nr_hw_queues = hisi_hba->cq_nvecs + hisi_hba->iopoll_q_cnt; return devm_add_action(&pdev->dev, hisi_sas_v3_free_vectors, pdev); From patchwork Wed Jul 12 12:54:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13310347 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 302D7EB64D9 for ; Wed, 12 Jul 2023 12:56:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232392AbjGLM4T (ORCPT ); Wed, 12 Jul 2023 08:56:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34634 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232541AbjGLM4Q (ORCPT ); Wed, 12 Jul 2023 08:56:16 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 63DFC198A for ; Wed, 12 Jul 2023 05:55:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689166526; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eJWmie5dktu1pTYEtIQqGDADYs5UCcoprVcDDbH555o=; b=Ec/WDj9qQlmZjuiydk7J47T0Hhxyakz5hU/1R970qtJf6wviPEJleFnKZ0kVvikilYfMR+ S3KolKBbtgrag/OHMKEW2fvKHVb7i43/31XNJy3gvFYFDqLUzsOOgcdaw5tBnOyLPuV3h6 8YSEhVY/+srfwBt37iybEnC/sRh75us= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-647-kSxoauWNPl63rVhMXXUeoA-1; Wed, 12 Jul 2023 08:55:23 -0400 X-MC-Unique: kSxoauWNPl63rVhMXXUeoA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C2F2888D0F8; Wed, 12 Jul 2023 12:55:22 +0000 (UTC) Received: from localhost (ovpn-8-25.pek2.redhat.com [10.72.8.25]) by smtp.corp.redhat.com (Postfix) with ESMTP id D6F8B200A7CA; Wed, 12 Jul 2023 12:55:21 +0000 (UTC) From: Ming Lei To: Jens Axboe , Christoph Hellwig , linux-nvme@lists.infradead.org, "Martin K . Petersen" , linux-scsi@vger.kernel.org Cc: linux-block@vger.kernel.org, Wen Xiong , Keith Busch , Ming Lei , Sreekanth Reddy , Sathya Prakash Veerichetty , Kashyap Desai , Sumit Saxena Subject: [PATCH 5/8] scsi: mpi3mr: take blk_mq_max_nr_hw_queues() into account for calculating io vectors Date: Wed, 12 Jul 2023 20:54:52 +0800 Message-Id: <20230712125455.1986455-6-ming.lei@redhat.com> In-Reply-To: <20230712125455.1986455-1-ming.lei@redhat.com> References: <20230712125455.1986455-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Take blk-mq's knowledge into account for calculating io queues. Fix wrong queue mapping in case of kdump kernel. On arm and ppc64, 'maxcpus=1' is passed to kdump kernel command line, see `Documentation/admin-guide/kdump/kdump.rst`, so num_possible_cpus() still returns all CPUs because 'maxcpus=1' just bring up one single cpu core during booting. blk-mq sees single queue in kdump kernel, and in driver's viewpoint there are still multiple queues, this inconsistency causes driver to apply wrong queue mapping for handling IO, and IO timeout is triggered. Meantime, single queue makes much less resource utilization, and reduce risk of kernel failure. Cc: Sreekanth Reddy Cc: Sathya Prakash Veerichetty Cc: Kashyap Desai Cc: Sumit Saxena Signed-off-by: Ming Lei --- drivers/scsi/mpi3mr/mpi3mr_fw.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/scsi/mpi3mr/mpi3mr_fw.c b/drivers/scsi/mpi3mr/mpi3mr_fw.c index 5fa07d6ee5b8..79ffd161575c 100644 --- a/drivers/scsi/mpi3mr/mpi3mr_fw.c +++ b/drivers/scsi/mpi3mr/mpi3mr_fw.c @@ -815,6 +815,9 @@ static int mpi3mr_setup_isr(struct mpi3mr_ioc *mrioc, u8 setup_one) desc.post_vectors = mrioc->requested_poll_qcount; min_vec = desc.pre_vectors + desc.post_vectors; + if (max_vectors - min_vec > blk_mq_max_nr_hw_queues()) + max_vectors = min_vec + blk_mq_max_nr_hw_queues(); + irq_flags |= PCI_IRQ_AFFINITY | PCI_IRQ_ALL_TYPES; retval = pci_alloc_irq_vectors_affinity(mrioc->pdev, From patchwork Wed Jul 12 12:54:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13310348 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A04EEB64DD for ; Wed, 12 Jul 2023 12:56:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232608AbjGLM4X (ORCPT ); Wed, 12 Jul 2023 08:56:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34662 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232333AbjGLM4U (ORCPT ); Wed, 12 Jul 2023 08:56:20 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DCE821BC6 for ; Wed, 12 Jul 2023 05:55:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689166531; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gV9SQll/aiZU7jZHjbK/6CqsGn0qDXytoHDv6FeUPyo=; b=Jv1x8vGC84/rIMffpUkklQGYcLtg/3onybr80fyRj6wNiX6vsXJJkUiSZRpXzuznVX1TUM +trJY0BQ8efYaqqDYVCow5V7hQ76rhh1O0fx2vX0pfTxCQL4eRr567O0ahIF9CN4t3PeUv o+7u2o33af8SJ3p3c2A4/BSJPlCt/Fs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-361-hvkH-1cePRinCSykJgDspQ-1; Wed, 12 Jul 2023 08:55:27 -0400 X-MC-Unique: hvkH-1cePRinCSykJgDspQ-1 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CC4BE10504AE; Wed, 12 Jul 2023 12:55:26 +0000 (UTC) Received: from localhost (ovpn-8-25.pek2.redhat.com [10.72.8.25]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0BB08C1ED96; Wed, 12 Jul 2023 12:55:25 +0000 (UTC) From: Ming Lei To: Jens Axboe , Christoph Hellwig , linux-nvme@lists.infradead.org, "Martin K . Petersen" , linux-scsi@vger.kernel.org Cc: linux-block@vger.kernel.org, Wen Xiong , Keith Busch , Ming Lei , Sreekanth Reddy , Sathya Prakash Veerichetty , Kashyap Desai , Sumit Saxena Subject: [PATCH 6/8] scsi: megaraid: take blk_mq_max_nr_hw_queues() into account for calculating io vectors Date: Wed, 12 Jul 2023 20:54:53 +0800 Message-Id: <20230712125455.1986455-7-ming.lei@redhat.com> In-Reply-To: <20230712125455.1986455-1-ming.lei@redhat.com> References: <20230712125455.1986455-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.8 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Take blk-mq's knowledge into account for calculating io queues. Fix wrong queue mapping in case of kdump kernel. On arm and ppc64, 'maxcpus=1' is passed to kdump kernel command line, see `Documentation/admin-guide/kdump/kdump.rst`, so num_possible_cpus() still returns all CPUs because 'maxcpus=1' just bring up one single cpu core during booting. blk-mq sees single queue in kdump kernel, and in driver's viewpoint there are still multiple queues, this inconsistency causes driver to apply wrong queue mapping for handling IO, and IO timeout is triggered. Meantime, single queue makes much less resource utilization, and reduce risk of kernel failure. Cc: Sreekanth Reddy Cc: Sathya Prakash Veerichetty Cc: Kashyap Desai Cc: Sumit Saxena Signed-off-by: Ming Lei --- drivers/scsi/megaraid/megaraid_sas_base.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/drivers/scsi/megaraid/megaraid_sas_base.c b/drivers/scsi/megaraid/megaraid_sas_base.c index 050eed8e2684..214713803c2e 100644 --- a/drivers/scsi/megaraid/megaraid_sas_base.c +++ b/drivers/scsi/megaraid/megaraid_sas_base.c @@ -5921,6 +5921,10 @@ __megasas_alloc_irq_vectors(struct megasas_instance *instance) int i, irq_flags; struct irq_affinity desc = { .pre_vectors = instance->low_latency_index_start }; struct irq_affinity *descp = &desc; + unsigned max_vecs = instance->msix_vectors - instance->iopoll_q_count; + + if (max_vecs > blk_mq_max_nr_hw_queues()) + max_vecs = blk_mq_max_nr_hw_queues(); irq_flags = PCI_IRQ_MSIX; @@ -5934,7 +5938,7 @@ __megasas_alloc_irq_vectors(struct megasas_instance *instance) */ i = pci_alloc_irq_vectors_affinity(instance->pdev, instance->low_latency_index_start, - instance->msix_vectors - instance->iopoll_q_count, irq_flags, descp); + max_vecs, irq_flags, descp); return i; } From patchwork Wed Jul 12 12:54:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13310350 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48A82EB64DD for ; Wed, 12 Jul 2023 12:56:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231522AbjGLM4a (ORCPT ); Wed, 12 Jul 2023 08:56:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34716 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232515AbjGLM42 (ORCPT ); Wed, 12 Jul 2023 08:56:28 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 766311BC2 for ; Wed, 12 Jul 2023 05:55:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689166537; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Nk/Wqaa+Kq1K/rJO90p9bF5IpxDn2p06PBFgsL5mmuI=; b=JjTMe04OpwbKbtG09Lwx9P272mWIoJL3qnPDR+AdGu9zOefQOFd1FJwzuASoruhaww1bXw 5P+mTrmxN73cbTuNNMTBxG49JcruCAFyLGkw741GG6uH3rgOPTNOVaZkEDypQG7/jry1D4 GrfjmqQhjvOoGotpiLi51owCQFYCFeI= Received: from mimecast-mx02.redhat.com (66.187.233.73 [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-58-FEyENvIFP9Gm5X0qWJ9PYQ-1; Wed, 12 Jul 2023 08:55:31 -0400 X-MC-Unique: FEyENvIFP9Gm5X0qWJ9PYQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8A4943C0FC94; Wed, 12 Jul 2023 12:55:30 +0000 (UTC) Received: from localhost (ovpn-8-25.pek2.redhat.com [10.72.8.25]) by smtp.corp.redhat.com (Postfix) with ESMTP id BC6362017EB9; Wed, 12 Jul 2023 12:55:29 +0000 (UTC) From: Ming Lei To: Jens Axboe , Christoph Hellwig , linux-nvme@lists.infradead.org, "Martin K . Petersen" , linux-scsi@vger.kernel.org Cc: linux-block@vger.kernel.org, Wen Xiong , Keith Busch , Ming Lei , Sathya Prakash , Sreekanth Reddy , Suganath Prabu Subramani Subject: [PATCH 7/8] scsi: mpt3sas: take blk_mq_max_nr_hw_queues() into account for calculating io vectors Date: Wed, 12 Jul 2023 20:54:54 +0800 Message-Id: <20230712125455.1986455-8-ming.lei@redhat.com> In-Reply-To: <20230712125455.1986455-1-ming.lei@redhat.com> References: <20230712125455.1986455-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Take blk-mq's knowledge into account for calculating io queues. Fix wrong queue mapping in case of kdump kernel. On arm and ppc64, 'maxcpus=1' is passed to kdump kernel command line, see `Documentation/admin-guide/kdump/kdump.rst`, so num_possible_cpus() still returns all CPUs because 'maxcpus=1' just bring up one single cpu core during booting. blk-mq sees single queue in kdump kernel, and in driver's viewpoint there are still multiple queues, this inconsistency causes driver to apply wrong queue mapping for handling IO, and IO timeout is triggered. Meantime, single queue makes much less resource utilization, and reduce risk of kernel failure. Cc: Sathya Prakash Cc: Sreekanth Reddy Cc: Suganath Prabu Subramani Signed-off-by: Ming Lei --- drivers/scsi/mpt3sas/mpt3sas_base.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c index 53f5492579cb..18a73d3ff00b 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.c +++ b/drivers/scsi/mpt3sas/mpt3sas_base.c @@ -3332,8 +3332,8 @@ _base_alloc_irq_vectors(struct MPT3SAS_ADAPTER *ioc) * Don't allocate msix vectors for poll_queues. * msix_vectors is always within a range of FW supported reply queue. */ - int nr_msix_vectors = ioc->iopoll_q_start_index; - + int nr_msix_vectors = min_t(unsigned int, ioc->iopoll_q_start_index, + blk_mq_max_nr_hw_queues()); if (ioc->smp_affinity_enable) irq_flags |= PCI_IRQ_AFFINITY | PCI_IRQ_ALL_TYPES; From patchwork Wed Jul 12 12:54:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13310349 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC921EB64DA for ; Wed, 12 Jul 2023 12:56:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232463AbjGLM43 (ORCPT ); Wed, 12 Jul 2023 08:56:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34714 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232688AbjGLM41 (ORCPT ); Wed, 12 Jul 2023 08:56:27 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F1EC61995 for ; Wed, 12 Jul 2023 05:55:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1689166536; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PQF1B10WKGfg+P76Yk7ZVr9AG6GdrqWyQBiZqmHzAGM=; b=YNQ2AIVaSmmj1MwJYB4+8bRl3gxZinZ8O5Z5tA2AYppqQi40Uep5/XJ/47jAj/OhhkdKXE C+N1nNF8PNzXWX6npd3rLSOdbniOQhpeiRc2gbTpQWmucmeidtAymSG3XFUcmO3gjKymxq TkY0Q8baa98p35dFelU14PEYPj60VvI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-519-yetSOoJlMSGzHUFd_mTmtg-1; Wed, 12 Jul 2023 08:55:34 -0400 X-MC-Unique: yetSOoJlMSGzHUFd_mTmtg-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 437AD104458C; Wed, 12 Jul 2023 12:55:34 +0000 (UTC) Received: from localhost (ovpn-8-25.pek2.redhat.com [10.72.8.25]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6214A200B402; Wed, 12 Jul 2023 12:55:33 +0000 (UTC) From: Ming Lei To: Jens Axboe , Christoph Hellwig , linux-nvme@lists.infradead.org, "Martin K . Petersen" , linux-scsi@vger.kernel.org Cc: linux-block@vger.kernel.org, Wen Xiong , Keith Busch , Ming Lei , Jack Wang Subject: [PATCH 8/8] scsi: pm8001: take blk_mq_max_nr_hw_queues() into account for calculating io vectors Date: Wed, 12 Jul 2023 20:54:55 +0800 Message-Id: <20230712125455.1986455-9-ming.lei@redhat.com> In-Reply-To: <20230712125455.1986455-1-ming.lei@redhat.com> References: <20230712125455.1986455-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org Take blk-mq's knowledge into account for calculating io queues. Fix wrong queue mapping in case of kdump kernel. On arm and ppc64, 'maxcpus=1' is passed to kdump kernel command line, see `Documentation/admin-guide/kdump/kdump.rst`, so num_possible_cpus() still returns all CPUs because 'maxcpus=1' just bring up one single cpu core during booting. blk-mq sees single queue in kdump kernel, and in driver's viewpoint there are still multiple queues, this inconsistency causes driver to apply wrong queue mapping for handling IO, and IO timeout is triggered. Meantime, single queue makes much less resource utilization, and reduce risk of kernel failure. Cc: Jack Wang Signed-off-by: Ming Lei Acked-by: Jack Wang --- drivers/scsi/pm8001/pm8001_init.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/scsi/pm8001/pm8001_init.c b/drivers/scsi/pm8001/pm8001_init.c index 2e886c1d867d..e2416f556560 100644 --- a/drivers/scsi/pm8001/pm8001_init.c +++ b/drivers/scsi/pm8001/pm8001_init.c @@ -965,6 +965,8 @@ static u32 pm8001_setup_msix(struct pm8001_hba_info *pm8001_ha) rc = pci_alloc_irq_vectors(pm8001_ha->pdev, 1, 1, PCI_IRQ_MSIX); } else { + unsigned int max_vecs = min_t(unsigned int, PM8001_MAX_MSIX_VEC, + blk_mq_max_nr_hw_queues() + 1); /* * Queue index #0 is used always for housekeeping, so don't * include in the affinity spreading. @@ -973,7 +975,7 @@ static u32 pm8001_setup_msix(struct pm8001_hba_info *pm8001_ha) .pre_vectors = 1, }; rc = pci_alloc_irq_vectors_affinity( - pm8001_ha->pdev, 2, PM8001_MAX_MSIX_VEC, + pm8001_ha->pdev, 2, max_vecs, PCI_IRQ_MSIX | PCI_IRQ_AFFINITY, &desc); }