From patchwork Thu Dec 17 19:57:05 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Himanshu Madhani X-Patchwork-Id: 7877421 Return-Path: X-Original-To: patchwork-linux-scsi@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id E85B5BEEE5 for ; Thu, 17 Dec 2015 20:38:58 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id CB0DE2042C for ; Thu, 17 Dec 2015 20:38:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id BE7E42041C for ; Thu, 17 Dec 2015 20:38:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932922AbbLQUiv (ORCPT ); Thu, 17 Dec 2015 15:38:51 -0500 Received: from mx0a-0016ce01.pphosted.com ([67.231.148.157]:5135 "EHLO mx0a-0016ce01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755287AbbLQUh7 (ORCPT ); Thu, 17 Dec 2015 15:37:59 -0500 Received: from pps.filterd (m0045602.ppops.net [127.0.0.1]) by mx0a-0016ce01.pphosted.com (8.15.0.59/8.15.0.59) with SMTP id tBHKWlFM007560; Thu, 17 Dec 2015 12:37:58 -0800 Received: from avcashub1.qlogic.com (avcashub2.qlogic.com [198.70.193.116]) by mx0a-0016ce01.pphosted.com with ESMTP id 1ytshtmat6-6 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Thu, 17 Dec 2015 12:37:58 -0800 Received: from dut6217.mv.qlogic.com (172.29.56.217) by qlc.com (10.1.4.191) with Microsoft SMTP Server id 14.3.235.1; Thu, 17 Dec 2015 12:37:57 -0800 Received: by dut6217.mv.qlogic.com (Postfix, from userid 0) id 1EC9C522137; Thu, 17 Dec 2015 14:57:12 -0500 (EST) From: Himanshu Madhani To: , CC: , , Subject: [PATCH v2 10/16] qla2xxx: Add irq affinity notification Date: Thu, 17 Dec 2015 14:57:05 -0500 Message-ID: <1450382231-4259-11-git-send-email-himanshu.madhani@qlogic.com> X-Mailer: git-send-email 1.7.7 In-Reply-To: <1450382231-4259-1-git-send-email-himanshu.madhani@qlogic.com> References: <1450382231-4259-1-git-send-email-himanshu.madhani@qlogic.com> MIME-Version: 1.0 disclaimer: bypass X-Proofpoint-Virus-Version: vendor=nai engine=5700 definitions=8018 signatures=670672 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 clxscore=1015 suspectscore=2 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1507310007 definitions=main-1512170334 Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Quinn Tran Register to receive notification of when irq setting change occured. Signed-off-by: Quinn Tran Signed-off-by: Himanshu Madhani --- drivers/scsi/qla2xxx/qla_def.h | 6 +++ drivers/scsi/qla2xxx/qla_isr.c | 76 ++++++++++++++++++++++++++++++++++++- drivers/scsi/qla2xxx/qla_target.c | 12 ++++++ 3 files changed, 93 insertions(+), 1 deletions(-) diff --git a/drivers/scsi/qla2xxx/qla_def.h b/drivers/scsi/qla2xxx/qla_def.h index 216a944..c4fc32e 100644 --- a/drivers/scsi/qla2xxx/qla_def.h +++ b/drivers/scsi/qla2xxx/qla_def.h @@ -2714,11 +2714,16 @@ struct isp_operations { struct scsi_qla_host; + +#define QLA83XX_RSPQ_MSIX_ENTRY_NUMBER 1 /* refer to qla83xx_msix_entries */ + struct qla_msix_entry { int have_irq; uint32_t vector; uint16_t entry; struct rsp_que *rsp; + struct irq_affinity_notify irq_notify; + int cpuid; }; #define WATCH_INTERVAL 1 /* number of seconds */ @@ -2930,6 +2935,7 @@ struct qlt_hw_data { spinlock_t q_full_lock; uint32_t leak_exchg_thresh_hold; spinlock_t sess_lock; + int rspq_vector_cpuid; }; #define MAX_QFULL_CMDS_ALLOC 8192 diff --git a/drivers/scsi/qla2xxx/qla_isr.c b/drivers/scsi/qla2xxx/qla_isr.c index ea7e8e8..cf0fe8e 100644 --- a/drivers/scsi/qla2xxx/qla_isr.c +++ b/drivers/scsi/qla2xxx/qla_isr.c @@ -18,6 +18,10 @@ static void qla2x00_status_entry(scsi_qla_host_t *, struct rsp_que *, void *); static void qla2x00_status_cont_entry(struct rsp_que *, sts_cont_entry_t *); static void qla2x00_error_entry(scsi_qla_host_t *, struct rsp_que *, sts_entry_t *); +static void qla_irq_affinity_notify(struct irq_affinity_notify *, + const cpumask_t *); +static void qla_irq_affinity_release(struct kref *); + /** * qla2100_intr_handler() - Process interrupts for the ISP2100 and ISP2200. @@ -2548,6 +2552,14 @@ void qla24xx_process_response_queue(struct scsi_qla_host *vha, if (!vha->flags.online) return; + if (rsp->msix->cpuid != smp_processor_id()) { + /* if kernel does not notify qla of IRQ's CPU change, + * then set it here. + */ + rsp->msix->cpuid = smp_processor_id(); + ha->tgt.rspq_vector_cpuid = rsp->msix->cpuid; + } + while (rsp->ring_ptr->signature != RESPONSE_PROCESSED) { pkt = (struct sts_entry_24xx *)rsp->ring_ptr; @@ -2979,8 +2991,11 @@ qla24xx_disable_msix(struct qla_hw_data *ha) for (i = 0; i < ha->msix_count; i++) { qentry = &ha->msix_entries[i]; - if (qentry->have_irq) + if (qentry->have_irq) { + /* un-register irq cpu affinity notification */ + irq_set_affinity_notifier(qentry->vector, NULL); free_irq(qentry->vector, qentry->rsp); + } } pci_disable_msix(ha->pdev); kfree(ha->msix_entries); @@ -3043,6 +3058,9 @@ qla24xx_enable_msix(struct qla_hw_data *ha, struct rsp_que *rsp) qentry->entry = entries[i].entry; qentry->have_irq = 0; qentry->rsp = NULL; + qentry->irq_notify.notify = qla_irq_affinity_notify; + qentry->irq_notify.release = qla_irq_affinity_release; + qentry->cpuid = -1; } /* Enable MSI-X vectors for the base queue */ @@ -3061,6 +3079,18 @@ qla24xx_enable_msix(struct qla_hw_data *ha, struct rsp_que *rsp) qentry->have_irq = 1; qentry->rsp = rsp; rsp->msix = qentry; + + /* Register for CPU affinity notification. */ + irq_set_affinity_notifier(qentry->vector, &qentry->irq_notify); + + /* Schedule work (ie. trigger a notification) to read cpu + * mask for this specific irq. + * kref_get is required because + * irq_affinity_notify() will do + * kref_put(). + */ + kref_get(&qentry->irq_notify.kref); + schedule_work(&qentry->irq_notify.work); } /* @@ -3240,3 +3270,47 @@ int qla25xx_request_irq(struct rsp_que *rsp) msix->rsp = rsp; return ret; } + + +/* irq_set_affinity/irqbalance will trigger notification of cpu mask update */ +static void qla_irq_affinity_notify(struct irq_affinity_notify *notify, + const cpumask_t *mask) +{ + struct qla_msix_entry *e = + container_of(notify, struct qla_msix_entry, irq_notify); + struct qla_hw_data *ha; + struct scsi_qla_host *base_vha; + + /* user is recommended to set mask to just 1 cpu */ + e->cpuid = cpumask_first(mask); + + ha = e->rsp->hw; + base_vha = pci_get_drvdata(ha->pdev); + + ql_dbg(ql_dbg_init, base_vha, 0xffff, + "%s: host %ld : vector %d cpu %d \n", __func__, + base_vha->host_no, e->vector, e->cpuid); + + if (e->have_irq) { + if ((IS_QLA83XX(ha) || IS_QLA27XX(ha)) && + (e->entry == QLA83XX_RSPQ_MSIX_ENTRY_NUMBER)) { + ha->tgt.rspq_vector_cpuid = e->cpuid; + ql_dbg(ql_dbg_init, base_vha, 0xffff, + "%s: host%ld: rspq vector %d cpu %d runtime change\n", + __func__, base_vha->host_no, e->vector, e->cpuid); + } + } +} + +void qla_irq_affinity_release(struct kref *ref) +{ + struct irq_affinity_notify *notify = + container_of(ref, struct irq_affinity_notify, kref); + struct qla_msix_entry *e = + container_of(notify, struct qla_msix_entry, irq_notify); + struct scsi_qla_host *base_vha = pci_get_drvdata(e->rsp->hw->pdev); + + ql_dbg(ql_dbg_init, base_vha, 0xffff, + "%s: host%ld: vector %d cpu %d \n", __func__, + base_vha->host_no, e->vector, e->cpuid); +} diff --git a/drivers/scsi/qla2xxx/qla_target.c b/drivers/scsi/qla2xxx/qla_target.c index 6136987..9a4aed0 100644 --- a/drivers/scsi/qla2xxx/qla_target.c +++ b/drivers/scsi/qla2xxx/qla_target.c @@ -6232,6 +6232,7 @@ qlt_enable_vha(struct scsi_qla_host *vha) struct qla_tgt *tgt = vha->vha_tgt.qla_tgt; unsigned long flags; scsi_qla_host_t *base_vha = pci_get_drvdata(ha->pdev); + int rspq_ent = QLA83XX_RSPQ_MSIX_ENTRY_NUMBER; if (!tgt) { ql_dbg(ql_dbg_tgt, vha, 0xe069, @@ -6250,6 +6251,17 @@ qlt_enable_vha(struct scsi_qla_host *vha) qla24xx_disable_vp(vha); qla24xx_enable_vp(vha); } else { + if (ha->msix_entries) { + ql_dbg(ql_dbg_tgt, vha, 0xffff, + "%s: host%ld : vector %d cpu %d\n", + __func__, vha->host_no, + ha->msix_entries[rspq_ent].vector, + ha->msix_entries[rspq_ent].cpuid); + + ha->tgt.rspq_vector_cpuid = + ha->msix_entries[rspq_ent].cpuid; + } + set_bit(ISP_ABORT_NEEDED, &base_vha->dpc_flags); qla2xxx_wake_dpc(base_vha); qla2x00_wait_for_hba_online(base_vha);