From patchwork Fri Mar 9 03:32:16 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 10269609 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id ADF42605CE for ; Fri, 9 Mar 2018 03:33:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A0F2629A92 for ; Fri, 9 Mar 2018 03:33:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 957E429AC4; Fri, 9 Mar 2018 03:33:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 23CC529AE0 for ; Fri, 9 Mar 2018 03:33:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751001AbeCIDdL (ORCPT ); Thu, 8 Mar 2018 22:33:11 -0500 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:35668 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1750932AbeCIDdK (ORCPT ); Thu, 8 Mar 2018 22:33:10 -0500 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 78E2C814F0CF; Fri, 9 Mar 2018 03:33:09 +0000 (UTC) Received: from localhost (ovpn-12-83.pek2.redhat.com [10.72.12.83]) by smtp.corp.redhat.com (Postfix) with ESMTP id F350511701C3; Fri, 9 Mar 2018 03:32:57 +0000 (UTC) From: Ming Lei To: James Bottomley , Jens Axboe , "Martin K . Petersen" Cc: Christoph Hellwig , linux-scsi@vger.kernel.org, linux-block@vger.kernel.org, Meelis Roos , Don Brace , Kashyap Desai , Laurence Oberman , Mike Snitzer , Ming Lei , Hannes Reinecke , James Bottomley , Artem Bityutskiy Subject: [PATCH V4 2/4] scsi: megaraid_sas: fix selection of reply queue Date: Fri, 9 Mar 2018 11:32:16 +0800 Message-Id: <20180309033218.23042-3-ming.lei@redhat.com> In-Reply-To: <20180309033218.23042-1-ming.lei@redhat.com> References: <20180309033218.23042-1-ming.lei@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Fri, 09 Mar 2018 03:33:09 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Fri, 09 Mar 2018 03:33:09 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'ming.lei@redhat.com' RCPT:'' Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From 84676c1f21 (genirq/affinity: assign vectors to all possible CPUs), one msix vector can be created without any online CPU mapped, then command may be queued, and won't be notified after its completion. This patch setups mapping between cpu and reply queue according to irq affinity info retrived by pci_irq_get_affinity(), and uses this info to choose reply queue for queuing one command. Then the chosen reply queue has to be active, and fixes IO hang caused by using inactive reply queue which doesn't have any online CPU mapped. Cc: Hannes Reinecke Cc: "Martin K. Petersen" , Cc: James Bottomley , Cc: Christoph Hellwig , Cc: Don Brace Cc: Kashyap Desai Cc: Laurence Oberman Cc: Mike Snitzer Cc: Meelis Roos Cc: Artem Bityutskiy Fixes: 84676c1f21e8 ("genirq/affinity: assign vectors to all possible CPUs") Signed-off-by: Ming Lei --- drivers/scsi/megaraid/megaraid_sas.h | 2 +- drivers/scsi/megaraid/megaraid_sas_base.c | 34 ++++++++++++++++++++++++++++- drivers/scsi/megaraid/megaraid_sas_fusion.c | 12 ++++------ 3 files changed, 38 insertions(+), 10 deletions(-) diff --git a/drivers/scsi/megaraid/megaraid_sas.h b/drivers/scsi/megaraid/megaraid_sas.h index ba6503f37756..a644d2be55b6 100644 --- a/drivers/scsi/megaraid/megaraid_sas.h +++ b/drivers/scsi/megaraid/megaraid_sas.h @@ -2127,7 +2127,7 @@ enum MR_PD_TYPE { #define MR_NVME_PAGE_SIZE_MASK 0x000000FF struct megasas_instance { - + unsigned int *reply_map; __le32 *producer; dma_addr_t producer_h; __le32 *consumer; diff --git a/drivers/scsi/megaraid/megaraid_sas_base.c b/drivers/scsi/megaraid/megaraid_sas_base.c index a71ee67df084..065956cb2aeb 100644 --- a/drivers/scsi/megaraid/megaraid_sas_base.c +++ b/drivers/scsi/megaraid/megaraid_sas_base.c @@ -5165,6 +5165,26 @@ megasas_setup_jbod_map(struct megasas_instance *instance) instance->use_seqnum_jbod_fp = false; } +static void megasas_setup_reply_map(struct megasas_instance *instance) +{ + const struct cpumask *mask; + unsigned int queue, cpu; + + for (queue = 0; queue < instance->msix_vectors; queue++) { + mask = pci_irq_get_affinity(instance->pdev, queue); + if (!mask) + goto fallback; + + for_each_cpu(cpu, mask) + instance->reply_map[cpu] = queue; + } + return; + +fallback: + for_each_possible_cpu(cpu) + instance->reply_map[cpu] = 0; +} + /** * megasas_init_fw - Initializes the FW * @instance: Adapter soft state @@ -5343,6 +5363,8 @@ static int megasas_init_fw(struct megasas_instance *instance) goto fail_setup_irqs; } + megasas_setup_reply_map(instance); + dev_info(&instance->pdev->dev, "firmware supports msix\t: (%d)", fw_msix_count); dev_info(&instance->pdev->dev, @@ -6448,6 +6470,11 @@ static int megasas_probe_one(struct pci_dev *pdev, memset(instance, 0, sizeof(*instance)); atomic_set(&instance->fw_reset_no_pci_access, 0); + instance->reply_map = kzalloc(sizeof(unsigned int) * nr_cpu_ids, + GFP_KERNEL); + if (!instance->reply_map) + goto fail_alloc_reply_map; + /* * Initialize PCI related and misc parameters */ @@ -6539,8 +6566,9 @@ static int megasas_probe_one(struct pci_dev *pdev, if (instance->msix_vectors) pci_free_irq_vectors(instance->pdev); fail_init_mfi: + kfree(instance->reply_map); +fail_alloc_reply_map: scsi_host_put(host); - fail_alloc_instance: pci_disable_device(pdev); @@ -6746,6 +6774,8 @@ megasas_resume(struct pci_dev *pdev) if (rval < 0) goto fail_reenable_msix; + megasas_setup_reply_map(instance); + if (instance->adapter_type != MFI_SERIES) { megasas_reset_reply_desc(instance); if (megasas_ioc_init_fusion(instance)) { @@ -6960,6 +6990,8 @@ static void megasas_detach_one(struct pci_dev *pdev) megasas_free_ctrl_mem(instance); + kfree(instance->reply_map); + scsi_host_put(host); pci_disable_device(pdev); diff --git a/drivers/scsi/megaraid/megaraid_sas_fusion.c b/drivers/scsi/megaraid/megaraid_sas_fusion.c index 073ced07e662..2994176a0121 100644 --- a/drivers/scsi/megaraid/megaraid_sas_fusion.c +++ b/drivers/scsi/megaraid/megaraid_sas_fusion.c @@ -2655,11 +2655,8 @@ megasas_build_ldio_fusion(struct megasas_instance *instance, fp_possible = (io_info.fpOkForIo > 0) ? true : false; } - /* Use raw_smp_processor_id() for now until cmd->request->cpu is CPU - id by default, not CPU group id, otherwise all MSI-X queues won't - be utilized */ - cmd->request_desc->SCSIIO.MSIxIndex = instance->msix_vectors ? - raw_smp_processor_id() % instance->msix_vectors : 0; + cmd->request_desc->SCSIIO.MSIxIndex = + instance->reply_map[raw_smp_processor_id()]; praid_context = &io_request->RaidContext; @@ -2985,10 +2982,9 @@ megasas_build_syspd_fusion(struct megasas_instance *instance, } cmd->request_desc->SCSIIO.DevHandle = io_request->DevHandle; - cmd->request_desc->SCSIIO.MSIxIndex = - instance->msix_vectors ? - (raw_smp_processor_id() % instance->msix_vectors) : 0; + cmd->request_desc->SCSIIO.MSIxIndex = + instance->reply_map[raw_smp_processor_id()]; if (!fp_possible) { /* system pd firmware path */