From patchwork Tue Jun 30 06:54:47 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sreekanth Reddy X-Patchwork-Id: 6694281 Return-Path: X-Original-To: patchwork-linux-scsi@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 97008C05AC for ; Tue, 30 Jun 2015 07:02:41 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 5B183205CA for ; Tue, 30 Jun 2015 07:02:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id EADD820279 for ; Tue, 30 Jun 2015 07:02:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753445AbbF3HCf (ORCPT ); Tue, 30 Jun 2015 03:02:35 -0400 Received: from mail-pd0-f175.google.com ([209.85.192.175]:34237 "EHLO mail-pd0-f175.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752621AbbF3Gz3 (ORCPT ); Tue, 30 Jun 2015 02:55:29 -0400 Received: by pdbep18 with SMTP id ep18so777249pdb.1 for ; Mon, 29 Jun 2015 23:55:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=avagotech.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=L2P+Egp3mReHwVS0Q6p9Dm4a4g86uIf2uoeuxjPRQO4=; b=jngG0SukLsnn+0miSIgS+TUR2Qf9509C3dKWxQt60GvAssOpBLHwnHGlQ5Tta8uyz5 z5DNMS2/NpbgWyjjxO2VVEPPWDkN3tbh11ibbhGPx2L9xV8oys333SEKf0COdPx6lOAa BCJN7U71ZO6AM/JGW0/7pptnzGxQBT583wn4M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=L2P+Egp3mReHwVS0Q6p9Dm4a4g86uIf2uoeuxjPRQO4=; b=jGTcf1kJobyAQOIHG71xugRrQI9XByJo64LQKIUQkNQ4Kzi3S2Fh7wbWfGV6CGjzlB GU+nEOIZ70BcErdcREL0stzEnRivJf0FNPDNOXM4gKDVf32Uq0+UE8IKhMwW8YehCo++ xbLODRjDqpJ1j9GM6PeejgnULVNuN6N4Gtcr7vUrpWjQOHBF2AIqFU6h9ibrBGjSsMUX XQX2diDKuGiSaR5kmgxOffC11q/SiaJ3G4smwcM+x9TByxCoSL6JjU62US7Kn2pJGg1+ gFpU/xSoCEoVLUGZ+Amf+2SgEOlHHYumSJIH78e50/ZGf/79L5olJ5kMlH5bQ2wbeEvT slTQ== X-Gm-Message-State: ALoCoQmyH3vtY3KPYdcJHthB+JHQK2Y6NATytW/FIRJuMqId2w9OqOarLPCKNZJMCA4yhEcOQ4DS X-Received: by 10.70.49.229 with SMTP id x5mr39774513pdn.81.1435647327392; Mon, 29 Jun 2015 23:55:27 -0700 (PDT) Received: from host1.lsi.com ([192.19.239.250]) by mx.google.com with ESMTPSA id 1sm935543pds.79.2015.06.29.23.55.23 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 29 Jun 2015 23:55:26 -0700 (PDT) From: Sreekanth Reddy X-Google-Original-From: Sreekanth Reddy To: jejb@kernel.org, hch@infradead.org Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, JBottomley@Parallels.com, Sathya.Prakash@avagotech.com, linux-kernel@vger.kernel.org, jthumshirn@suse.de, Sreekanth Reddy Subject: [PATCH v2 01/20] [SCSI] mpt3sas: Added Combined Reply Queue feature to extend up-to 96 MSIX vector support Date: Tue, 30 Jun 2015 12:24:47 +0530 Message-Id: <1435647306-36956-2-git-send-email-Sreekanth.Reddy@avagotech.com> X-Mailer: git-send-email 2.0.2 In-Reply-To: <1435647306-36956-1-git-send-email-Sreekanth.Reddy@avagotech.com> References: <1435647306-36956-1-git-send-email-Sreekanth.Reddy@avagotech.com> Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Spam-Status: No, score=-7.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,T_DKIM_INVALID,UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In this patch, increased the number of MSIX vector support for SAS3 C0 HBAs to up-to 96. Following are changes that are done in this patch 1. This feature is enabled only for SAS3 C0 and higher revision cards and also only when reply post free queue count is greater than 8. 2. To support this feature 12 SupplementalReplyPostHostIndex system interfaces are used. MSI-X index numbered from 0 to 7 use the first SupplementalReplyPostHostIndex system interface to update its corresponding ReplyPostHostIndex values, MSI-X index numbered from 8 to 15 will use the second SupplementalReplyPostHostIndex system interface and so on. These 12 SuppementalReplyPostHostIndex system interfaces address are saved in the array replyPostRegisterIndex[]. 3. As each SupplementalReplyPostHostIndex register supports 8 MSI-X vectors. So MSIxIndex field in these register must contain a value between 0 and 7. 4. After processing the reply descriptors from a reply post free queues then update the new reply post host index value in ReplyPostHostIndex field and (msix_index mod 8) value in MSIxIndex field of SupplementalReplyPostHostIndex register. The Address of this SupplementalReplyPostHostIndex register is retrived from (msix_index/8)th entry of replyPostRegisterIndex[] array. Changes in v1: Updated the description of module parameter max_msix_vectors Changes in v2: * Defined some Combined Reply Queue's feature magic values, * Removed unnecessory brackets, * Added comments where ever required. Signed-off-by: Sreekanth Reddy Reviewed-by: Martin K. Petersen Reviewed-by: Johannes Thumshirn --- drivers/scsi/mpt3sas/mpt3sas_base.c | 93 +++++++++++++++++++++++++++++++++---- drivers/scsi/mpt3sas/mpt3sas_base.h | 14 +++++- 2 files changed, 98 insertions(+), 9 deletions(-) diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c index 14a781b..63a8588 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.c +++ b/drivers/scsi/mpt3sas/mpt3sas_base.c @@ -83,10 +83,10 @@ static int msix_disable = -1; module_param(msix_disable, int, 0); MODULE_PARM_DESC(msix_disable, " disable msix routed interrupts (default=0)"); -static int max_msix_vectors = 8; +static int max_msix_vectors = -1; module_param(max_msix_vectors, int, 0); MODULE_PARM_DESC(max_msix_vectors, - " max msix vectors - (default=8)"); + " max msix vectors"); static int mpt3sas_fwfault_debug; MODULE_PARM_DESC(mpt3sas_fwfault_debug, @@ -1009,8 +1009,30 @@ _base_interrupt(int irq, void *bus_id) } wmb(); - writel(reply_q->reply_post_host_index | (msix_index << - MPI2_RPHI_MSIX_INDEX_SHIFT), &ioc->chip->ReplyPostHostIndex); + + /* Update Reply Post Host Index. + * For those HBA's which support combined reply queue feature + * 1. Get the correct Supplemental Reply Post Host Index Register. + * i.e. (msix_index / 8)th entry from Supplemental Reply Post Host + * Index Register address bank i.e replyPostRegisterIndex[], + * 2. Then update this register with new reply host index value + * in ReplyPostIndex field and the MSIxIndex field with + * msix_index value reduced to a value between 0 and 7, + * using a modulo 8 operation. Since each Supplemental Reply Post + * Host Index Register supports 8 MSI-X vectors. + * + * For other HBA's just update the Reply Post Host Index register with + * new reply host index value in ReplyPostIndex Field and msix_index + * value in MSIxIndex field. + */ + if (ioc->msix96_vector) + writel(reply_q->reply_post_host_index | ((msix_index & 7) << + MPI2_RPHI_MSIX_INDEX_SHIFT), + ioc->replyPostRegisterIndex[msix_index/8]); + else + writel(reply_q->reply_post_host_index | (msix_index << + MPI2_RPHI_MSIX_INDEX_SHIFT), + &ioc->chip->ReplyPostHostIndex); atomic_dec(&reply_q->busy); return IRQ_HANDLED; } @@ -1560,8 +1582,6 @@ _base_check_enable_msix(struct MPT3SAS_ADAPTER *ioc) pci_read_config_word(ioc->pdev, base + 2, &message_control); ioc->msix_vector_count = (message_control & 0x3FF) + 1; - if (ioc->msix_vector_count > 8) - ioc->msix_vector_count = 8; dinitprintk(ioc, pr_info(MPT3SAS_FMT "msix is supported, vector_count(%d)\n", ioc->name, ioc->msix_vector_count)); @@ -1880,6 +1900,36 @@ mpt3sas_base_map_resources(struct MPT3SAS_ADAPTER *ioc) if (r) goto out_fail; + /* Use the Combined reply queue feature only for SAS3 C0 & higher + * revision HBAs and also only when reply queue count is greater than 8 + */ + if (ioc->msix96_vector && ioc->reply_queue_count > 8) { + /* Determine the Supplemental Reply Post Host Index Registers + * Addresse. Supplemental Reply Post Host Index Registers + * starts at offset MPI25_SUP_REPLY_POST_HOST_INDEX_OFFSET and + * each register is at offset bytes of + * MPT3_SUP_REPLY_POST_HOST_INDEX_REG_OFFSET from previous one. + */ + ioc->replyPostRegisterIndex = kcalloc( + MPT3_SUP_REPLY_POST_HOST_INDEX_REG_COUNT, + sizeof(resource_size_t *), GFP_KERNEL); + if (!ioc->replyPostRegisterIndex) { + dfailprintk(ioc, printk(MPT3SAS_FMT + "allocation for reply Post Register Index failed!!!\n", + ioc->name)); + r = -ENOMEM; + goto out_fail; + } + + for (i = 0; i < MPT3_SUP_REPLY_POST_HOST_INDEX_REG_COUNT; i++) { + ioc->replyPostRegisterIndex[i] = (resource_size_t *) + ((u8 *)&ioc->chip->Doorbell + + MPI25_SUP_REPLY_POST_HOST_INDEX_OFFSET + + (i * MPT3_SUP_REPLY_POST_HOST_INDEX_REG_OFFSET)); + } + } else + ioc->msix96_vector = 0; + list_for_each_entry(reply_q, &ioc->reply_queue_list, list) pr_info(MPT3SAS_FMT "%s: IRQ %d\n", reply_q->name, ((ioc->msix_enable) ? "PCI-MSI-X enabled" : @@ -1901,6 +1951,8 @@ mpt3sas_base_map_resources(struct MPT3SAS_ADAPTER *ioc) pci_release_selected_regions(ioc->pdev, ioc->bars); pci_disable_pcie_error_reporting(pdev); pci_disable_device(pdev); + if (ioc->msix96_vector) + kfree(ioc->replyPostRegisterIndex); return r; } @@ -4522,8 +4574,15 @@ _base_make_ioc_operational(struct MPT3SAS_ADAPTER *ioc, int sleep_flag) /* initialize reply post host index */ list_for_each_entry(reply_q, &ioc->reply_queue_list, list) { - writel(reply_q->msix_index << MPI2_RPHI_MSIX_INDEX_SHIFT, - &ioc->chip->ReplyPostHostIndex); + if (ioc->msix96_vector) + writel((reply_q->msix_index & 7)<< + MPI2_RPHI_MSIX_INDEX_SHIFT, + ioc->replyPostRegisterIndex[reply_q->msix_index/8]); + else + writel(reply_q->msix_index << + MPI2_RPHI_MSIX_INDEX_SHIFT, + &ioc->chip->ReplyPostHostIndex); + if (!_base_is_controller_msix_enabled(ioc)) goto skip_init_reply_post_host_index; } @@ -4577,6 +4636,9 @@ mpt3sas_base_free_resources(struct MPT3SAS_ADAPTER *ioc) _base_free_irq(ioc); _base_disable_msix(ioc); + if (ioc->msix96_vector) + kfree(ioc->replyPostRegisterIndex); + if (ioc->chip_phys && ioc->chip) iounmap(ioc->chip); ioc->chip_phys = 0; @@ -4600,6 +4662,7 @@ mpt3sas_base_attach(struct MPT3SAS_ADAPTER *ioc) { int r, i; int cpu_id, last_cpu_id = 0; + u8 revision; dinitprintk(ioc, pr_info(MPT3SAS_FMT "%s\n", ioc->name, __func__)); @@ -4619,6 +4682,20 @@ mpt3sas_base_attach(struct MPT3SAS_ADAPTER *ioc) goto out_free_resources; } + /* Check whether the controller revision is C0 or above. + * only C0 and above revision controllers support 96 MSI-X vectors. + */ + revision = ioc->pdev->revision; + + if ((ioc->pdev->device == MPI25_MFGPAGE_DEVID_SAS3004 || + ioc->pdev->device == MPI25_MFGPAGE_DEVID_SAS3008 || + ioc->pdev->device == MPI25_MFGPAGE_DEVID_SAS3108_1 || + ioc->pdev->device == MPI25_MFGPAGE_DEVID_SAS3108_2 || + ioc->pdev->device == MPI25_MFGPAGE_DEVID_SAS3108_5 || + ioc->pdev->device == MPI25_MFGPAGE_DEVID_SAS3108_6) && + (revision >= 0x02)) + ioc->msix96_vector = 1; + ioc->rdpq_array_enable_assigned = 0; ioc->dma_mask = 0; r = mpt3sas_base_map_resources(ioc); diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.h b/drivers/scsi/mpt3sas/mpt3sas_base.h index afa8816..a7386ee 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.h +++ b/drivers/scsi/mpt3sas/mpt3sas_base.h @@ -158,6 +158,13 @@ #define MPT3_DIAG_BUFFER_IS_RELEASED (0x02) #define MPT3_DIAG_BUFFER_IS_DIAG_RESET (0x04) +/* + * Combined Reply Queue constants, + * There are twelve Supplemental Reply Post Host Index Registers + * and each register is at offset 0x10 bytes from the previous one. + */ +#define MPT3_SUP_REPLY_POST_HOST_INDEX_REG_COUNT 12 +#define MPT3_SUP_REPLY_POST_HOST_INDEX_REG_OFFSET (0x10) /* OEM Identifiers */ #define MFG10_OEM_ID_INVALID (0x00000000) @@ -728,7 +735,8 @@ typedef void (*MPT3SAS_FLUSH_RUNNING_CMDS)(struct MPT3SAS_ADAPTER *ioc); * is assigned only ones * @reply_queue_count: number of reply queue's * @reply_queue_list: link list contaning the reply queue info - * @reply_post_host_index: head index in the pool where FW completes IO + * @msix96_vector: 96 MSI-X vector support + * @replyPostRegisterIndex: index of next position in Reply Desc Post Queue * @delayed_tr_list: target reset link list * @delayed_tr_volume_list: volume target reset link list * @@temp_sensors_count: flag to carry the number of temperature sensors @@ -937,6 +945,10 @@ struct MPT3SAS_ADAPTER { u8 reply_queue_count; struct list_head reply_queue_list; + u8 msix96_vector; + /* reply post register index */ + resource_size_t **replyPostRegisterIndex; + struct list_head delayed_tr_list; struct list_head delayed_tr_volume_list; u8 temp_sensors_count;