From patchwork Sat Sep 30 09:28:42 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Abhishek Shah X-Patchwork-Id: 9979333 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id E58DB60375 for ; Sat, 30 Sep 2017 09:29:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D637929754 for ; Sat, 30 Sep 2017 09:29:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CAA782975D; Sat, 30 Sep 2017 09:29:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id CEA7F29754 for ; Sat, 30 Sep 2017 09:29:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=xTzDCISl0e3EyzQD+X5554ckYM0/WuV1jJHLtR3LHns=; b=U+3 mHLFJh4RlSX8m5WIBdmarZK1m1ytXdd5nXBq1MSjtIpUTqr/YFMezoNqKEJgeG627Nvl7li/dIy9E twknpwZaKGK7QDmaMNCJkYCUjlbWl7R/ByxweCYO2bkiA8B3PH8+ew77TD4TZMufkgxcPoTEhErQ7 VGZDa06tExv5OfmzEjdOrxWBaTids9T82GlYjYSVtUN/jAtqeuegfaVe1iBL7SzP+DhooDRswXysx qL4JKN73nQIP2IIcIGeOVfuNSplRZ8g+AcFbpHRXrfRbZTbVatXgcCSTKr+pr1Lwe7U2htOEAm1lS mVLdjTFpt6MU3H0POYM0H8B7HAIeRlw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1dyE5W-0001HX-Vd; Sat, 30 Sep 2017 09:29:34 +0000 Received: from mail-wr0-x231.google.com ([2a00:1450:400c:c0c::231]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1dyE5N-00017q-1O for linux-arm-kernel@lists.infradead.org; Sat, 30 Sep 2017 09:29:27 +0000 Received: by mail-wr0-x231.google.com with SMTP id y95so1058093wrb.4 for ; Sat, 30 Sep 2017 02:29:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id; bh=2ST9+Pq1+tDpCOgviUf/jWVXef1k6wTNkXIRoP7DSb4=; b=NJdGNARg1JtgU3ffzeS3D4+TFA3g/GGMuPZ33tNAx6bL/nZdtdYk5S/KqYrzfOL5Bh Fqgc7fYaDeV0rt6eAq/ZM8icF4ylICxSk22KYFEirh51Y/vCfeF4hAzHZekTywYP6Lj4 63znz7hKF3Yr2cF2PkpHBCJbFn+DMnrY7pc5Q= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=2ST9+Pq1+tDpCOgviUf/jWVXef1k6wTNkXIRoP7DSb4=; b=X5FnLtNV0g5B+yojToNvyZkCVzzhOhonw7GOvScva1cOgMqBhrvWlWOmHS8+plIz6e vz9JrUZk53hEZ4iGjd08u3NtJhNY0IRdSN4VWQVOCejyc0rSrwnww/2ay+VbvzTzi2Yo usHJbZnxL1iavzI+LckivUZw1cd33ZN7CFNimxN170+ld3qJt/ipoemPlRC4koozlT07 EqWzElpUvn+PgFn/QY9cs4IzLzzVvi2iP4rR3AB2SOSfNBWLAvjc2uXKlxHgclkkWKBb Op7MdUypO7/+6grUciXZoPQstUWGuRia20s7urUi54X2e0mXxJhcQmB0OPUvSmQXslqd Z3Jg== X-Gm-Message-State: AHPjjUhaS9FSM4IK30bIGIpoPbCZG1ELZykk6tU5ddaZy8mqp1LrHsmk ODRhUze4TbbbJ66UvaG4Z1651Q== X-Google-Smtp-Source: AOwi7QC1k/JfoBvZ1gqTNmqI+vDSJX/8ymDcOcCGwjHuL7/6wg5H23VkTYYDjl4SfQVoRoLiiqP5XQ== X-Received: by 10.223.134.83 with SMTP id 19mr9069902wrw.223.1506763739976; Sat, 30 Sep 2017 02:28:59 -0700 (PDT) Received: from ashah1-OptiPlex-7010.dhcp.avagotech.net ([192.19.237.250]) by smtp.gmail.com with ESMTPSA id b190sm5286354wma.41.2017.09.30.02.28.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 30 Sep 2017 02:28:59 -0700 (PDT) From: Abhishek Shah To: Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg Subject: [PATCH v2] nvme-pci: Use PCI bus address for data/queues in CMB Date: Sat, 30 Sep 2017 14:58:42 +0530 Message-Id: <1506763722-10687-1-git-send-email-abhishek.shah@broadcom.com> X-Mailer: git-send-email 2.7.4 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20170930_022925_248674_49A6CBBE X-CRM114-Status: GOOD ( 13.75 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, bcm-kernel-feedback-list@broadcom.com, stable@vger.kernel.org, Abhishek Shah , linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Currently, NVMe PCI host driver is programming CMB dma address as I/O SQs addresses. This results in failures on systems where 1:1 outbound mapping is not used (example Broadcom iProc SOCs) because CMB BAR will be progammed with PCI bus address but NVMe PCI EP will try to access CMB using dma address. To have CMB working on systems without 1:1 outbound mapping, we program PCI bus address for I/O SQs instead of dma address. This approach will work on systems with/without 1:1 outbound mapping. The patch is tested on Broadcom Stingray platform(arm64), which does not have 1:1 outbound mapping, as well as on x86 platform, which has 1:1 outbound mapping. Fixes: 8ffaadf7 ("NVMe: Use CMB for the IO SQes if available") Cc: stable@vger.kernel.org Signed-off-by: Abhishek Shah Reviewed-by: Anup Patel Reviewed-by: Ray Jui Reviewed-by: Scott Branden --- drivers/nvme/host/pci.c | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 4a21213..1387050 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -94,6 +94,7 @@ struct nvme_dev { bool subsystem; void __iomem *cmb; dma_addr_t cmb_dma_addr; + pci_bus_addr_t cmb_bus_addr; u64 cmb_size; u32 cmbsz; u32 cmbloc; @@ -1220,7 +1221,7 @@ static int nvme_alloc_sq_cmds(struct nvme_dev *dev, struct nvme_queue *nvmeq, if (qid && dev->cmb && use_cmb_sqes && NVME_CMB_SQS(dev->cmbsz)) { unsigned offset = (qid - 1) * roundup(SQ_SIZE(depth), dev->ctrl.page_size); - nvmeq->sq_dma_addr = dev->cmb_dma_addr + offset; + nvmeq->sq_dma_addr = dev->cmb_bus_addr + offset; nvmeq->sq_cmds_io = dev->cmb + offset; } else { nvmeq->sq_cmds = dma_alloc_coherent(dev->dev, SQ_SIZE(depth), @@ -1514,6 +1515,25 @@ static ssize_t nvme_cmb_show(struct device *dev, } static DEVICE_ATTR(cmb, S_IRUGO, nvme_cmb_show, NULL); +static int nvme_find_cmb_bus_addr(struct pci_dev *pdev, + dma_addr_t dma_addr, + u64 size, + pci_bus_addr_t *bus_addr) +{ + struct resource *res; + struct pci_bus_region region; + struct resource tres = DEFINE_RES_MEM(dma_addr, size); + + res = pci_find_resource(pdev, &tres); + if (!res) + return -EIO; + + pcibios_resource_to_bus(pdev->bus, ®ion, res); + *bus_addr = region.start + (dma_addr - res->start); + + return 0; +} + static void __iomem *nvme_map_cmb(struct nvme_dev *dev) { u64 szu, size, offset; @@ -1547,6 +1567,9 @@ static void __iomem *nvme_map_cmb(struct nvme_dev *dev) size = bar_size - offset; dma_addr = pci_resource_start(pdev, NVME_CMB_BIR(dev->cmbloc)) + offset; + if (nvme_find_cmb_bus_addr(pdev, dma_addr, size, &dev->cmb_bus_addr)) + return NULL; + cmb = ioremap_wc(dma_addr, size); if (!cmb) return NULL;