From patchwork Fri Jul 14 13:22:31 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suganath Prabu S X-Patchwork-Id: 9841045 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 8C4D160381 for ; Fri, 14 Jul 2017 13:24:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7DF0728770 for ; Fri, 14 Jul 2017 13:24:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7270D28788; Fri, 14 Jul 2017 13:24:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8D78A28770 for ; Fri, 14 Jul 2017 13:24:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754507AbdGNNYL (ORCPT ); Fri, 14 Jul 2017 09:24:11 -0400 Received: from mail-qt0-f176.google.com ([209.85.216.176]:34763 "EHLO mail-qt0-f176.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754381AbdGNNYG (ORCPT ); Fri, 14 Jul 2017 09:24:06 -0400 Received: by mail-qt0-f176.google.com with SMTP id 32so61880980qtv.1 for ; Fri, 14 Jul 2017 06:24:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=r3h8qaEV0Yx2B3cmZdPfp1SJFgvRKHQULBGwMpnccVo=; b=Ed1EX2cmURnsEzDVxfLEhoRDdWsBYFxw6AzdkOjzv1Er48aX5dqFiksXAqJGvZTeWA DEsCk5RCFfEnJyBmxVlaDHoVZ9/lYMfZRPuKbceQKUP6O756P6O2WKuKMaCLrE9xEO6H DeVspHKUt3sszeYFDbqctnIce3L7slEydXfAA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=r3h8qaEV0Yx2B3cmZdPfp1SJFgvRKHQULBGwMpnccVo=; b=HDr2tIddGEh8ufIpBHa0SKfSupMC9XENFwG61FpsxBGu7j2z4EnEpjwVyD8cqEbzB1 O8dqsDYVmrhqqLBrmbCt+mIZLuVYo8sipR7bEtZ8saubtAOSd2+1buIK6G/HrCzRa2R4 Bir6gUYj/lWWPOGhpjbJvNJGtZPS6Rbi/Rf/HmBo+VCnV/ltbi+IG8IizSXOMEm0EbXI 3lxcdjpAnB8NeZLXuNXG5uR0Qj7y8hOl8zDkHY2d0M2HNnXS+9FuWOhCp8AnCvhSHOxw GhY0nGf9qMeiMm9VN0EzoxBrDfz9eLnMrp0/n/+6eQW06e28dtbyd8J0FBqAILGlChm2 RGmA== X-Gm-Message-State: AIVw111EBoaNdvNO3Jo79opjzf6ilrmhWVk+JVuh/+jjKesWfWGVbIsP 5fcrfpGMKg2tWaccsdQ= X-Received: by 10.200.46.100 with SMTP id s33mr12742810qta.48.1500038640243; Fri, 14 Jul 2017 06:24:00 -0700 (PDT) Received: from host1.dhcp.avagotech.net ([192.19.239.250]) by smtp.gmail.com with ESMTPSA id a123sm6439964qkc.0.2017.07.14.06.23.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 14 Jul 2017 06:23:59 -0700 (PDT) From: Suganath Prabu S To: jejb@kernel.org, hch@infradead.org Cc: martin.petersen@oracle.com, linux-scsi@vger.kernel.org, Sathya.Prakash@broadcom.com, kashyap.desai@broadcom.com, linux-kernel@vger.kernel.org, suganath-prabu.subramani@broadcom.com, chaitra.basappa@broadcom.com, sreekanth.reddy@broadcom.com, linux-nvme@lists.infradead.org Subject: [PATCH v2 04/13] mpt3sas: Added support for nvme encapsulated request message. Date: Fri, 14 Jul 2017 18:52:31 +0530 Message-Id: <1500038560-11231-5-git-send-email-suganath-prabu.subramani@broadcom.com> X-Mailer: git-send-email 2.0.2 In-Reply-To: <1500038560-11231-1-git-send-email-suganath-prabu.subramani@broadcom.com> References: <1500038560-11231-1-git-send-email-suganath-prabu.subramani@broadcom.com> Sender: linux-scsi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-scsi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP * Mpt3sas driver uses the NVMe Encapsulated Request message to send an NVMe command to an NVMe device attached to the IOC. * Normal I/O commands like reads and writes are passed to the controller as SCSI commands and the controller has the ability to translate the commands to NVMe equivalent. * This encapsulated NVMe command is used by applications to send direct NVMe commands to NVMe drives or for handling unmap where the translation at controller/firmware level is having performance issues. Signed-off-by: Chaitra P B Signed-off-by: Suganath Prabu S Reviewed-by: Hannes Reinecke --- drivers/scsi/mpt3sas/mpt3sas_base.c | 56 ++++++++++++++++++++++++++++- drivers/scsi/mpt3sas/mpt3sas_base.h | 1 + drivers/scsi/mpt3sas/mpt3sas_ctl.c | 69 ++++++++++++++++++++++++++++++++--- 3 files changed, 119 insertions(+), 7 deletions(-) diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c index a90c386..c3a6c7a 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.c +++ b/drivers/scsi/mpt3sas/mpt3sas_base.c @@ -557,6 +557,11 @@ _base_sas_ioc_info(struct MPT3SAS_ADAPTER *ioc, MPI2DefaultReply_t *mpi_reply, frame_sz = sizeof(Mpi2SmpPassthroughRequest_t) + ioc->sge_size; func_str = "smp_passthru"; break; + case MPI2_FUNCTION_NVME_ENCAPSULATED: + frame_sz = sizeof(Mpi26NVMeEncapsulatedRequest_t) + + ioc->sge_size; + func_str = "nvme_encapsulated"; + break; default: frame_sz = 32; func_str = "unknown"; @@ -985,7 +990,9 @@ _base_interrupt(int irq, void *bus_id) if (request_desript_type == MPI25_RPY_DESCRIPT_FLAGS_FAST_PATH_SCSI_IO_SUCCESS || request_desript_type == - MPI2_RPY_DESCRIPT_FLAGS_SCSI_IO_SUCCESS) { + MPI2_RPY_DESCRIPT_FLAGS_SCSI_IO_SUCCESS || + request_desript_type == + MPI26_RPY_DESCRIPT_FLAGS_PCIE_ENCAPSULATED_SUCCESS) { cb_idx = _base_get_cb_idx(ioc, smid); if ((likely(cb_idx < MPT_MAX_CALLBACKS)) && (likely(mpt_callbacks[cb_idx] != NULL))) { @@ -3080,6 +3087,30 @@ _base_put_smid_hi_priority(struct MPT3SAS_ADAPTER *ioc, u16 smid, } /** + * _base_put_smid_nvme_encap - send NVMe encapsulated request to + * firmware + * @ioc: per adapter object + * @smid: system request message index + * + * Return nothing. + */ +static void +_base_put_smid_nvme_encap(struct MPT3SAS_ADAPTER *ioc, u16 smid) +{ + Mpi2RequestDescriptorUnion_t descriptor; + u64 *request = (u64 *)&descriptor; + + descriptor.Default.RequestFlags = + MPI26_REQ_DESCRIPT_FLAGS_PCIE_ENCAPSULATED; + descriptor.Default.MSIxIndex = _base_get_msix_index(ioc); + descriptor.Default.SMID = cpu_to_le16(smid); + descriptor.Default.LMID = 0; + descriptor.Default.DescriptorTypeDependent = 0; + _base_writeq(*request, &ioc->chip->RequestDescriptorPostLow, + &ioc->scsi_lookup_lock); +} + +/** * _base_put_smid_default - Default, primarily used for config pages * @ioc: per adapter object * @smid: system request message index @@ -3170,6 +3201,27 @@ _base_put_smid_hi_priority_atomic(struct MPT3SAS_ADAPTER *ioc, u16 smid, } /** + * _base_put_smid_nvme_encap_atomic - send NVMe encapsulated request to + * firmware using Atomic Request Descriptor + * @ioc: per adapter object + * @smid: system request message index + * + * Return nothing. + */ +static void +_base_put_smid_nvme_encap_atomic(struct MPT3SAS_ADAPTER *ioc, u16 smid) +{ + Mpi26AtomicRequestDescriptor_t descriptor; + u32 *request = (u32 *)&descriptor; + + descriptor.RequestFlags = MPI26_REQ_DESCRIPT_FLAGS_PCIE_ENCAPSULATED; + descriptor.MSIxIndex = _base_get_msix_index(ioc); + descriptor.SMID = cpu_to_le16(smid); + + writel(cpu_to_le32(*request), &ioc->chip->AtomicRequestDescriptorPost); +} + +/** * _base_put_smid_default - Default, primarily used for config pages * use Atomic Request Descriptor * @ioc: per adapter object @@ -6002,11 +6054,13 @@ mpt3sas_base_attach(struct MPT3SAS_ADAPTER *ioc) ioc->put_smid_scsi_io = &_base_put_smid_scsi_io_atomic; ioc->put_smid_fast_path = &_base_put_smid_fast_path_atomic; ioc->put_smid_hi_priority = &_base_put_smid_hi_priority_atomic; + ioc->put_smid_nvme_encap = &_base_put_smid_nvme_encap_atomic; } else { ioc->put_smid_default = &_base_put_smid_default; ioc->put_smid_scsi_io = &_base_put_smid_scsi_io; ioc->put_smid_fast_path = &_base_put_smid_fast_path; ioc->put_smid_hi_priority = &_base_put_smid_hi_priority; + ioc->put_smid_nvme_encap = &_base_put_smid_nvme_encap; } diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.h b/drivers/scsi/mpt3sas/mpt3sas_base.h index fed3d97..bd32b0c 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.h +++ b/drivers/scsi/mpt3sas/mpt3sas_base.h @@ -1351,6 +1351,7 @@ struct MPT3SAS_ADAPTER { PUT_SMID_IO_FP_HIP put_smid_fast_path; PUT_SMID_IO_FP_HIP put_smid_hi_priority; PUT_SMID_DEFAULT put_smid_default; + PUT_SMID_DEFAULT put_smid_nvme_encap; }; diff --git a/drivers/scsi/mpt3sas/mpt3sas_ctl.c b/drivers/scsi/mpt3sas/mpt3sas_ctl.c index 0c18831..6362d60 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_ctl.c +++ b/drivers/scsi/mpt3sas/mpt3sas_ctl.c @@ -272,6 +272,7 @@ mpt3sas_ctl_done(struct MPT3SAS_ADAPTER *ioc, u16 smid, u8 msix_index, { MPI2DefaultReply_t *mpi_reply; Mpi2SCSIIOReply_t *scsiio_reply; + Mpi26NVMeEncapsulatedErrorReply_t *nvme_error_reply; const void *sense_data; u32 sz; @@ -298,6 +299,18 @@ mpt3sas_ctl_done(struct MPT3SAS_ADAPTER *ioc, u16 smid, u8 msix_index, memcpy(ioc->ctl_cmds.sense, sense_data, sz); } } + /* + * Get Error Response data for NVMe device. The ctl_cmds.sense + * buffer is used to store the Error Response data. + */ + if (mpi_reply->Function == MPI2_FUNCTION_NVME_ENCAPSULATED) { + nvme_error_reply = + (Mpi26NVMeEncapsulatedErrorReply_t *)mpi_reply; + sz = min_t(u32, NVME_ERROR_RESPONSE_SIZE, + le32_to_cpu(nvme_error_reply->ErrorResponseCount)); + sense_data = mpt3sas_base_get_sense_buffer(ioc, smid); + memcpy(ioc->ctl_cmds.sense, sense_data, sz); + } } _ctl_display_some_debug(ioc, smid, "ctl_done", mpi_reply); @@ -641,11 +654,12 @@ _ctl_do_mpt_command(struct MPT3SAS_ADAPTER *ioc, struct mpt3_ioctl_command karg, { MPI2RequestHeader_t *mpi_request = NULL, *request; MPI2DefaultReply_t *mpi_reply; + Mpi26NVMeEncapsulatedRequest_t *nvme_encap_request = NULL; u32 ioc_state; u16 smid; unsigned long timeout; u8 issue_reset; - u32 sz; + u32 sz, sz_arg; void *psge; void *data_out = NULL; dma_addr_t data_out_dma = 0; @@ -742,7 +756,8 @@ _ctl_do_mpt_command(struct MPT3SAS_ADAPTER *ioc, struct mpt3_ioctl_command karg, if (mpi_request->Function == MPI2_FUNCTION_SCSI_IO_REQUEST || mpi_request->Function == MPI2_FUNCTION_RAID_SCSI_IO_PASSTHROUGH || mpi_request->Function == MPI2_FUNCTION_SCSI_TASK_MGMT || - mpi_request->Function == MPI2_FUNCTION_SATA_PASSTHROUGH) { + mpi_request->Function == MPI2_FUNCTION_SATA_PASSTHROUGH || + mpi_request->Function == MPI2_FUNCTION_NVME_ENCAPSULATED) { device_handle = le16_to_cpu(mpi_request->FunctionDependent1); if (!device_handle || (device_handle > @@ -793,6 +808,38 @@ _ctl_do_mpt_command(struct MPT3SAS_ADAPTER *ioc, struct mpt3_ioctl_command karg, init_completion(&ioc->ctl_cmds.done); switch (mpi_request->Function) { + case MPI2_FUNCTION_NVME_ENCAPSULATED: + { + nvme_encap_request = (Mpi26NVMeEncapsulatedRequest_t *)request; + /* + * Get the Physical Address of the sense buffer. + * Use Error Response buffer address field to hold the sense + * buffer address. + * Clear the internal sense buffer, which will potentially hold + * the Completion Queue Entry on return, or 0 if no Entry. + * Build the PRPs and set direction bits. + * Send the request. + */ + nvme_encap_request->ErrorResponseBaseAddress = ioc->sense_dma & + 0xFFFFFFFF00000000; + nvme_encap_request->ErrorResponseBaseAddress |= + (U64)mpt3sas_base_get_sense_buffer_dma(ioc, smid); + nvme_encap_request->ErrorResponseAllocationLength = + NVME_ERROR_RESPONSE_SIZE; + memset(ioc->ctl_cmds.sense, 0, NVME_ERROR_RESPONSE_SIZE); + ioc->build_nvme_prp(ioc, smid, nvme_encap_request, + data_out_dma, data_out_sz, data_in_dma, data_in_sz); + if (test_bit(device_handle, ioc->device_remove_in_progress)) { + dtmprintk(ioc, pr_info(MPT3SAS_FMT "handle(0x%04x) :" + "ioctl failed due to device removal in progress\n", + ioc->name, device_handle)); + mpt3sas_base_free_smid(ioc, smid); + ret = -EINVAL; + goto out; + } + ioc->put_smid_nvme_encap(ioc, smid); + break; + } case MPI2_FUNCTION_SCSI_IO_REQUEST: case MPI2_FUNCTION_RAID_SCSI_IO_PASSTHROUGH: { @@ -1008,15 +1055,25 @@ _ctl_do_mpt_command(struct MPT3SAS_ADAPTER *ioc, struct mpt3_ioctl_command karg, } } - /* copy out sense to user */ + /* copy out sense/NVMe Error Response to user */ if (karg.max_sense_bytes && (mpi_request->Function == MPI2_FUNCTION_SCSI_IO_REQUEST || mpi_request->Function == - MPI2_FUNCTION_RAID_SCSI_IO_PASSTHROUGH)) { - sz = min_t(u32, karg.max_sense_bytes, SCSI_SENSE_BUFFERSIZE); + MPI2_FUNCTION_RAID_SCSI_IO_PASSTHROUGH || mpi_request->Function == + MPI2_FUNCTION_NVME_ENCAPSULATED)) { + if (karg.sense_data_ptr == NULL) { + pr_info(MPT3SAS_FMT "Response buffer provided" + " by application is NULL; Response data will" + " not be returned.\n", ioc->name); + goto out; + } + sz_arg = (mpi_request->Function == + MPI2_FUNCTION_NVME_ENCAPSULATED) ? NVME_ERROR_RESPONSE_SIZE : + SCSI_SENSE_BUFFERSIZE; + sz = min_t(u32, karg.max_sense_bytes, sz_arg); if (copy_to_user(karg.sense_data_ptr, ioc->ctl_cmds.sense, sz)) { pr_err("failure at %s:%d/%s()!\n", __FILE__, - __LINE__, __func__); + __LINE__, __func__); ret = -ENODATA; goto out; }