From patchwork Wed Dec 26 23:33:24 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Smart X-Patchwork-Id: 10743361 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 40AE06C2 for ; Wed, 26 Dec 2018 23:34:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 320CA2849B for ; Wed, 26 Dec 2018 23:34:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2672F28748; Wed, 26 Dec 2018 23:34:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 620D62849B for ; Wed, 26 Dec 2018 23:34:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727453AbeLZXeK (ORCPT ); Wed, 26 Dec 2018 18:34:10 -0500 Received: from mail-yw1-f66.google.com ([209.85.161.66]:44484 "EHLO mail-yw1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727368AbeLZXeJ (ORCPT ); Wed, 26 Dec 2018 18:34:09 -0500 Received: by mail-yw1-f66.google.com with SMTP id b63so5475675ywc.11 for ; Wed, 26 Dec 2018 15:34:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=yt1MMHFAjszWWN0oXQHsgI0dmHNb4YvQ2wDlJM9mWNw=; b=dOmUhdkkK2AeV2Ato1DEz6NKDOGf3Rtwr3W/KJ1WY2ZPurXfqZhEdbUl/owz5T6/UY facsPrD6Xuga+qODflXc5Gf/+eyGGNCEDKT3Onr08X4nIzw34dumKCZFDc+/wbca1IN6 PMD1A96GKHCvB3LOWTNS1ILkL+LwteUy7VeKlelf6cLPLrsrjmhtV/DhPKzkphc1ppDo oGBzZuAWoQJTEabBeH2YJH4CEZkNcG/rZAz7RdGlmHoJDEDvesZujpSkEtQRueGE652p zfm3r4Kj5ASryPN2fWdycG8lEgaziEERMDtZR8RAR6V1c48Po7LH3PvMAKCTN3Bd3kNZ VaZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=yt1MMHFAjszWWN0oXQHsgI0dmHNb4YvQ2wDlJM9mWNw=; b=rQhVEKHK5zT0G8vJxEuQaDGe8SbkV28pccC2e4uijHtseVNMmHl6KQlCNgjpwyFzWv 9G7K9KVZzYahttcpsLgiT4O/mvWWZNt6NoFGdgaCdheIMEJFe+x5o/wbiinFXp3i/oBZ An013qIk0c4GR6JitQ1SSnoBJj/uWdFvlyrhVH9wGuvrJGTHKICUMtFsi/0HprZD130X 1oY9JG2zpyq3JMncWMM4Ju36QZfsMVfdIUrxHaGuNfi7zGOZMGVw4e3xs6j5UwMEc1dX lD8lZgP0OnQJjv6hDgWsa6CrBVHKyiNP+uHyPaIbqwgKon1fPtPfCnne2PHxbw8WM/r4 q7qA== X-Gm-Message-State: AA+aEWb1Duo24XTANLIGj/j7Wh6Zh2Fxtm4Ni7v/Ws830iPpj+wVECyB wsUZgmESKoZarsMQO8HW6rM4V27w X-Google-Smtp-Source: AFSGD/X+4bS/EmzqjxDGjhHVbMqtJziAT1/oTqh92miFXjjekfe39D5M0GPUADqZwyZRk8oIAVJ4Aw== X-Received: by 2002:a81:d87:: with SMTP id 129mr21683650ywn.248.1545867248071; Wed, 26 Dec 2018 15:34:08 -0800 (PST) Received: from os42.localdomain ([192.19.228.250]) by smtp.gmail.com with ESMTPSA id h145sm13616483ywc.72.2018.12.26.15.34.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 26 Dec 2018 15:34:07 -0800 (PST) From: James Smart To: linux-scsi@vger.kernel.org Cc: James Smart , Dick Kennedy Subject: [PATCH 15/25] lpfc: Fix setting affinity hints to correlate with hardware queues Date: Wed, 26 Dec 2018 15:33:24 -0800 Message-Id: <20181226233334.27518-16-jsmart2021@gmail.com> X-Mailer: git-send-email 2.13.7 In-Reply-To: <20181226233334.27518-1-jsmart2021@gmail.com> References: <20181226233334.27518-1-jsmart2021@gmail.com> Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The desired affinity for the hardware queue behavior is for hdwq 0 to be affinitized with cpu 0, hdwq 1 to cpu 1, and so on. The implementation so far does not do this if the number of cpus is greating than the number of hardware queues (e.g. hardware queue allocation was administratively reduced or hardware queue resources could not scale to the cpu count). Correct the queue affinitization logic, when queue count is less than cpu count. Signed-off-by: Dick Kennedy Signed-off-by: James Smart Reviewed-by: Hannes Reinecke --- drivers/scsi/lpfc/lpfc_attr.c | 38 +++++++++++++++++++++------- drivers/scsi/lpfc/lpfc_init.c | 58 +++++++++++++++++++------------------------ drivers/scsi/lpfc/lpfc_sli4.h | 2 +- 3 files changed, 56 insertions(+), 42 deletions(-) diff --git a/drivers/scsi/lpfc/lpfc_attr.c b/drivers/scsi/lpfc/lpfc_attr.c index b00373215aa0..27fafad63c6f 100644 --- a/drivers/scsi/lpfc/lpfc_attr.c +++ b/drivers/scsi/lpfc/lpfc_attr.c @@ -5071,21 +5071,41 @@ lpfc_fcp_cpu_map_show(struct device *dev, struct device_attribute *attr, while (phba->sli4_hba.curr_disp_cpu < phba->sli4_hba.num_present_cpu) { cpup = &phba->sli4_hba.cpu_map[phba->sli4_hba.curr_disp_cpu]; - /* margin should fit in this and the truncated message */ - if (cpup->irq == LPFC_VECTOR_MAP_EMPTY) - len += snprintf(buf + len, PAGE_SIZE-len, - "CPU %02d io_chan %02d " + if (cpup->irq == LPFC_VECTOR_MAP_EMPTY) { + if (cpup->hdwq == LPFC_VECTOR_MAP_EMPTY) + len += snprintf( + buf + len, PAGE_SIZE - len, + "CPU %02d hdwq None " "physid %d coreid %d\n", phba->sli4_hba.curr_disp_cpu, - cpup->channel_id, cpup->phys_id, + cpup->phys_id, cpup->core_id); - else - len += snprintf(buf + len, PAGE_SIZE-len, - "CPU %02d io_chan %02d " + else + len += snprintf( + buf + len, PAGE_SIZE - len, + "CPU %02d hdwq %04d " + "physid %d coreid %d\n", + phba->sli4_hba.curr_disp_cpu, + cpup->hdwq, cpup->phys_id, + cpup->core_id); + } else { + if (cpup->hdwq == LPFC_VECTOR_MAP_EMPTY) + len += snprintf( + buf + len, PAGE_SIZE - len, + "CPU %02d hdwq None " + "physid %d coreid %d IRQ %d\n", + phba->sli4_hba.curr_disp_cpu, + cpup->phys_id, + cpup->core_id, cpup->irq); + else + len += snprintf( + buf + len, PAGE_SIZE - len, + "CPU %02d hdwq %04d " "physid %d coreid %d IRQ %d\n", phba->sli4_hba.curr_disp_cpu, - cpup->channel_id, cpup->phys_id, + cpup->hdwq, cpup->phys_id, cpup->core_id, cpup->irq); + } phba->sli4_hba.curr_disp_cpu++; diff --git a/drivers/scsi/lpfc/lpfc_init.c b/drivers/scsi/lpfc/lpfc_init.c index fc1d3ac7ac46..1ab23394fdc3 100644 --- a/drivers/scsi/lpfc/lpfc_init.c +++ b/drivers/scsi/lpfc/lpfc_init.c @@ -71,7 +71,6 @@ unsigned long _dump_buf_dif_order; spinlock_t _dump_buf_lock; /* Used when mapping IRQ vectors in a driver centric manner */ -uint16_t *lpfc_used_cpu; uint32_t lpfc_present_cpu; static void lpfc_get_hba_model_desc(struct lpfc_hba *, uint8_t *, uint8_t *); @@ -6835,20 +6834,6 @@ lpfc_sli4_driver_resource_setup(struct lpfc_hba *phba) rc = -ENOMEM; goto out_free_hba_eq_hdl; } - if (lpfc_used_cpu == NULL) { - lpfc_used_cpu = kcalloc(lpfc_present_cpu, sizeof(uint16_t), - GFP_KERNEL); - if (!lpfc_used_cpu) { - lpfc_printf_log(phba, KERN_ERR, LOG_INIT, - "3335 Failed allocate memory for msi-x " - "interrupt vector mapping\n"); - kfree(phba->sli4_hba.cpu_map); - rc = -ENOMEM; - goto out_free_hba_eq_hdl; - } - for (i = 0; i < lpfc_present_cpu; i++) - lpfc_used_cpu[i] = LPFC_VECTOR_MAP_EMPTY; - } /* * Enable sr-iov virtual functions if supported and configured @@ -10457,7 +10442,6 @@ lpfc_sli_disable_intr(struct lpfc_hba *phba) /** * lpfc_cpu_affinity_check - Check vector CPU affinity mappings * @phba: pointer to lpfc hba data structure. - * @vectors: number of msix vectors allocated. * * The routine will figure out the CPU affinity assignment for every * MSI-X vector allocated for the HBA. The hba_eq_hdl will be updated @@ -10468,12 +10452,10 @@ lpfc_sli_disable_intr(struct lpfc_hba *phba) * and the phba->sli4_hba.cpu_map array will reflect this. */ static void -lpfc_cpu_affinity_check(struct lpfc_hba *phba, int vectors) +lpfc_cpu_affinity_check(struct lpfc_hba *phba) { struct lpfc_vector_map_info *cpup; - int index = 0; - int vec = 0; - int cpu; + int cpu, idx; #ifdef CONFIG_X86 struct cpuinfo_x86 *cpuinfo; #endif @@ -10495,16 +10477,26 @@ lpfc_cpu_affinity_check(struct lpfc_hba *phba, int vectors) cpup->phys_id = 0; cpup->core_id = 0; #endif - cpup->channel_id = index; /* For now round robin */ - cpup->irq = pci_irq_vector(phba->pcidev, vec); - vec++; - if (vec >= vectors) - vec = 0; - index++; - if (index >= phba->cfg_hdw_queue) - index = 0; + lpfc_printf_log(phba, KERN_INFO, LOG_INIT, + "3328 CPU physid %d coreid %d\n", + cpup->phys_id, cpup->core_id); cpup++; } + + for (idx = 0; idx < phba->cfg_hdw_queue; idx++) { + cpup = &phba->sli4_hba.cpu_map[idx]; + cpup->irq = pci_irq_vector(phba->pcidev, idx); + + /* For now assume vector N maps to CPU N */ + irq_set_affinity_hint(cpup->irq, get_cpu_mask(idx)); + cpup->hdwq = idx; + + lpfc_printf_log(phba, KERN_INFO, LOG_INIT, + "3336 Set Affinity: CPU %d " + "hdwq %d irq %d\n", + cpu, cpup->hdwq, cpup->irq); + } + return; } @@ -10570,7 +10562,7 @@ lpfc_sli4_enable_msix(struct lpfc_hba *phba) if (phba->cfg_nvmet_mrq > vectors) phba->cfg_nvmet_mrq = vectors; } - lpfc_cpu_affinity_check(phba, vectors); + lpfc_cpu_affinity_check(phba); return rc; @@ -10718,9 +10710,13 @@ lpfc_sli4_disable_intr(struct lpfc_hba *phba) int index; /* Free up MSI-X multi-message vectors */ - for (index = 0; index < phba->cfg_hdw_queue; index++) + for (index = 0; index < phba->cfg_hdw_queue; index++) { + irq_set_affinity_hint( + pci_irq_vector(phba->pcidev, index), + NULL); free_irq(pci_irq_vector(phba->pcidev, index), &phba->sli4_hba.hba_eq_hdl[index]); + } } else { free_irq(phba->pcidev->irq, phba); } @@ -12987,7 +12983,6 @@ lpfc_init(void) lpfc_nvmet_cmd_template(); /* Initialize in case vector mapping is needed */ - lpfc_used_cpu = NULL; lpfc_present_cpu = num_present_cpus(); error = pci_register_driver(&lpfc_driver); @@ -13026,7 +13021,6 @@ lpfc_exit(void) (1L << _dump_buf_dif_order), _dump_buf_dif); free_pages((unsigned long)_dump_buf_dif, _dump_buf_dif_order); } - kfree(lpfc_used_cpu); idr_destroy(&lpfc_hba_index); } diff --git a/drivers/scsi/lpfc/lpfc_sli4.h b/drivers/scsi/lpfc/lpfc_sli4.h index 6893e8547431..f5e58cd4c6ac 100644 --- a/drivers/scsi/lpfc/lpfc_sli4.h +++ b/drivers/scsi/lpfc/lpfc_sli4.h @@ -529,7 +529,7 @@ struct lpfc_vector_map_info { uint16_t phys_id; uint16_t core_id; uint16_t irq; - uint16_t channel_id; + uint16_t hdwq; }; #define LPFC_VECTOR_MAP_EMPTY 0xffff