From patchwork Fri Apr 21 23:04:59 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Smart X-Patchwork-Id: 9693907 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 6AA3F6038E for ; Fri, 21 Apr 2017 23:06:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5C3582867F for ; Fri, 21 Apr 2017 23:06:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 511ED28688; Fri, 21 Apr 2017 23:06:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C85A028684 for ; Fri, 21 Apr 2017 23:06:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1426518AbdDUXG0 (ORCPT ); Fri, 21 Apr 2017 19:06:26 -0400 Received: from mail-oi0-f65.google.com ([209.85.218.65]:36858 "EHLO mail-oi0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1426479AbdDUXFc (ORCPT ); Fri, 21 Apr 2017 19:05:32 -0400 Received: by mail-oi0-f65.google.com with SMTP id a3so18441439oii.3 for ; Fri, 21 Apr 2017 16:05:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=XGpkxxp1uim15otahfpplvRDK+ZipDNj5b/w9L/9Wqg=; b=Q4ZUOLVHst8+8iSJyE2yaWM01lI5NlGUQnIj5PibZDNn2FPgMwrx5jW1okaTvjtHGg V994usY22ADWPzriw2NMBLbaBdbQJz0yKA0GF7a41WuVDIw5vCbpN/6wZfGMkSD/MqBD oQZonP0jzMR8TAMagrpDv2AdO7BCyMTU3R6LGwBAJl4ehy0gpb8MomDQzduPQkumSl8n uVR9Pwi2ITcNY1Z599t+I/90+49u6fDPEuFITGcKzSLpmVzlwg8AIjT+77WE+//++GM4 +4OTmjZAX2emahok8iSBOo0Yxewu0ZMbxb7M7FghoKwj6dPE39CYLo82QF94B0tfGrzz FEUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=XGpkxxp1uim15otahfpplvRDK+ZipDNj5b/w9L/9Wqg=; b=WSRtYWCEyc5UnV2NFUPM4VyXlITQAUwE5tJy1f3B6lN69Ses6fivxSMA6IJjZeSAgu 8gR7TLYkkPQiP7EUglGylyhlCOD6wewa3KRdGdsdJFAZbHMMfVU8bqQrNEfbFXM2USlh UMnv0wO6M0TVzyw9UnxlRnuo0vA+ztvRGFhtAhtUT1kmv4VDzBfQ/X8vAigv7wzPXyxz PD/a6nNy7jX5Hi9ydyn8zcYup/8ibU3bnGh52fbxJAwjaetQMSg4Qlattz63vUI+bW9+ rIzbo0z8AKAuzB59Iajw5chZKNMA3ALURBN5gZcEbBZMBpsVKbTdJbXOeomz4y/9qsJ3 YHBQ== X-Gm-Message-State: AN3rC/6D6igvl1QDkriFEe0RlREZiiIuUZ1amNa0ACuNotQ9ynfhz5sy bCXsSyzv6njglg== X-Received: by 10.157.66.42 with SMTP id q42mr10482629ote.2.1492815931026; Fri, 21 Apr 2017 16:05:31 -0700 (PDT) Received: from flash.localdomain (ip68-5-29-33.oc.oc.cox.net. [68.5.29.33]) by smtp.gmail.com with ESMTPSA id c17sm4794676oih.27.2017.04.21.16.05.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Apr 2017 16:05:30 -0700 (PDT) From: jsmart2021@gmail.com To: linux-scsi@vger.kernel.org, linux-nvme@lists.infradead.org, sagi@grimberg.me, martin.petersen@oracle.com Cc: James Smart , Dick Kennedy , James Smart Subject: [PATCH v3 15/22] Fix driver load issues when MRQ=8 Date: Fri, 21 Apr 2017 16:04:59 -0700 Message-Id: <20170421230506.925-16-jsmart2021@gmail.com> X-Mailer: git-send-email 2.9.3 In-Reply-To: <20170421230506.925-1-jsmart2021@gmail.com> References: <20170421230506.925-1-jsmart2021@gmail.com> Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: James Smart The symptom is that the driver will fail to login to the fabric. The reason is because it is out of iocb resources. There is a one to one relationship between MRQs (receive buffers for NVMET-FC) and iocbs and the default number of IOCBs was not accounting for the number of MRQs that were being created. This fix aligns the number of MRQ resources with the total resources so that it can handle fabric events when needed. Also the initialization of ctxlock to be on FCP commands, NOT LS commands. And modified log messages so that the log output can be correlated with the analyzer trace. Signed-off-by: Dick Kennedy Signed-off-by: James Smart Reviewed-by: Johannes Thumshirn --- drivers/scsi/lpfc/lpfc_init.c | 12 ++++++++---- drivers/scsi/lpfc/lpfc_nvme.c | 23 ++++++++++++----------- drivers/scsi/lpfc/lpfc_nvmet.c | 6 +++--- 3 files changed, 23 insertions(+), 18 deletions(-) diff --git a/drivers/scsi/lpfc/lpfc_init.c b/drivers/scsi/lpfc/lpfc_init.c index cca7f81..3bd43f4 100644 --- a/drivers/scsi/lpfc/lpfc_init.c +++ b/drivers/scsi/lpfc/lpfc_init.c @@ -11061,7 +11061,7 @@ lpfc_pci_probe_one_s4(struct pci_dev *pdev, const struct pci_device_id *pid) struct lpfc_hba *phba; struct lpfc_vport *vport = NULL; struct Scsi_Host *shost = NULL; - int error; + int error, cnt; uint32_t cfg_mode, intr_mode; /* Allocate memory for HBA structure */ @@ -11095,11 +11095,15 @@ lpfc_pci_probe_one_s4(struct pci_dev *pdev, const struct pci_device_id *pid) goto out_unset_pci_mem_s4; } + cnt = phba->cfg_iocb_cnt * 1024; + if (phba->nvmet_support) + cnt += phba->cfg_nvmet_mrq_post * phba->cfg_nvmet_mrq; + /* Initialize and populate the iocb list per host */ lpfc_printf_log(phba, KERN_INFO, LOG_INIT, - "2821 initialize iocb list %d.\n", - phba->cfg_iocb_cnt*1024); - error = lpfc_init_iocb_list(phba, phba->cfg_iocb_cnt*1024); + "2821 initialize iocb list %d total %d\n", + phba->cfg_iocb_cnt, cnt); + error = lpfc_init_iocb_list(phba, cnt); if (error) { lpfc_printf_log(phba, KERN_ERR, LOG_INIT, diff --git a/drivers/scsi/lpfc/lpfc_nvme.c b/drivers/scsi/lpfc/lpfc_nvme.c index 264da8f..cb1e82b 100644 --- a/drivers/scsi/lpfc/lpfc_nvme.c +++ b/drivers/scsi/lpfc/lpfc_nvme.c @@ -1495,6 +1495,7 @@ lpfc_nvme_fcp_abort(struct nvme_fc_local_port *pnvme_lport, "io buffer. Skipping abort req.\n"); return; } + nvmereq_wqe = &lpfc_nbuf->cur_iocbq; /* * The lpfc_nbuf and the mapped nvme_fcreq in the driver's @@ -1508,20 +1509,19 @@ lpfc_nvme_fcp_abort(struct nvme_fc_local_port *pnvme_lport, lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME, "6143 NVME req mismatch: " "lpfc_nbuf %p nvmeCmd %p, " - "pnvme_fcreq %p. Skipping Abort\n", + "pnvme_fcreq %p. Skipping Abort xri x%x\n", lpfc_nbuf, lpfc_nbuf->nvmeCmd, - pnvme_fcreq); + pnvme_fcreq, nvmereq_wqe->sli4_xritag); return; } /* Don't abort IOs no longer on the pending queue. */ - nvmereq_wqe = &lpfc_nbuf->cur_iocbq; if (!(nvmereq_wqe->iocb_flag & LPFC_IO_ON_TXCMPLQ)) { spin_unlock_irqrestore(&phba->hbalock, flags); lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME, "6142 NVME IO req %p not queued - skipping " - "abort req\n", - pnvme_fcreq); + "abort req xri x%x\n", + pnvme_fcreq, nvmereq_wqe->sli4_xritag); return; } @@ -1535,8 +1535,9 @@ lpfc_nvme_fcp_abort(struct nvme_fc_local_port *pnvme_lport, lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME, "6144 Outstanding NVME I/O Abort Request " "still pending on nvme_fcreq %p, " - "lpfc_ncmd %p\n", - pnvme_fcreq, lpfc_nbuf); + "lpfc_ncmd %p xri x%x\n", + pnvme_fcreq, lpfc_nbuf, + nvmereq_wqe->sli4_xritag); return; } @@ -1545,8 +1546,8 @@ lpfc_nvme_fcp_abort(struct nvme_fc_local_port *pnvme_lport, spin_unlock_irqrestore(&phba->hbalock, flags); lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME, "6136 No available abort wqes. Skipping " - "Abts req for nvme_fcreq %p.\n", - pnvme_fcreq); + "Abts req for nvme_fcreq %p xri x%x\n", + pnvme_fcreq, nvmereq_wqe->sli4_xritag); return; } @@ -1604,7 +1605,7 @@ lpfc_nvme_fcp_abort(struct nvme_fc_local_port *pnvme_lport, } lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME, - "6138 Transport Abort NVME Request Issued for\n" + "6138 Transport Abort NVME Request Issued for " "ox_id x%x on reqtag x%x\n", nvmereq_wqe->sli4_xritag, abts_buf->iotag); @@ -2491,7 +2492,7 @@ lpfc_nvme_unregister_port(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp) input_err: #endif lpfc_printf_vlog(vport, KERN_ERR, LOG_NVME_DISC, - "6168: State error: lport %p, rport%p FCID x%06x\n", + "6168 State error: lport %p, rport%p FCID x%06x\n", vport->localport, ndlp->rport, ndlp->nlp_DID); } diff --git a/drivers/scsi/lpfc/lpfc_nvmet.c b/drivers/scsi/lpfc/lpfc_nvmet.c index 5a994a8..8348bb5 100644 --- a/drivers/scsi/lpfc/lpfc_nvmet.c +++ b/drivers/scsi/lpfc/lpfc_nvmet.c @@ -872,7 +872,6 @@ lpfc_nvmet_unsol_ls_buffer(struct lpfc_hba *phba, struct lpfc_sli_ring *pring, ctxp->wqeq = NULL; ctxp->state = LPFC_NVMET_STE_RCV; ctxp->rqb_buffer = (void *)nvmebuf; - spin_lock_init(&ctxp->ctxlock); lpfc_nvmeio_data(phba, "NVMET LS RCV: xri x%x sz %d from %06x\n", oxid, size, sid); @@ -981,6 +980,7 @@ lpfc_nvmet_unsol_fcp_buffer(struct lpfc_hba *phba, ctxp->rqb_buffer = nvmebuf; ctxp->entry_cnt = 1; ctxp->flag = 0; + spin_lock_init(&ctxp->ctxlock); #ifdef CONFIG_SCSI_LPFC_DEBUG_FS if (phba->ktime_on) { @@ -1003,8 +1003,8 @@ lpfc_nvmet_unsol_fcp_buffer(struct lpfc_hba *phba, } #endif - lpfc_nvmeio_data(phba, "NVMET FCP RCV: xri x%x sz %d from %06x\n", - oxid, size, sid); + lpfc_nvmeio_data(phba, "NVMET FCP RCV: xri x%x sz %d CPU %02x\n", + oxid, size, smp_processor_id()); atomic_inc(&tgtp->rcv_fcp_cmd_in); /*