From patchwork Fri Oct 6 13:32:02 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 9989491 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 45C8B6020F for ; Fri, 6 Oct 2017 13:47:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 23B97283AE for ; Fri, 6 Oct 2017 13:47:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 18463284B5; Fri, 6 Oct 2017 13:47:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 84B56283AE for ; Fri, 6 Oct 2017 13:47:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=XfA5XJBjXAaxYTsxL0zkY8zokkREgXeOl5JRnARnfRM=; b=eWK7IfaCPjPkDZij9IrKCHo5CB blRnR1Tmlqt/b9iljbwqSDZ4VRM2j0VIUapBq2qMMfMuLp2S5NSTQIMYGsty/0TmqrPVuwiqPVF/0 B11OXrUOox6pJwehwTk5jF101e/A83bfWxgW2lgR/KKXg4X0pmkrp8jkXm0t9F33ftjgXL/6cOqcI kR3BYiOyrHZd14lLV0WgbJp8bzA/WnDJR248Tb5/VMrNyXsQrUBfCRo9+qF3shmDsjC85f5FaXJKg glBQh+kTNSKWOkjRPsVK1rtbUgvBcE4s8xrkQQarlGJuc+2bgnWqQUd9/dpUN0lhEaLhVYR1EywYG Ob/wMwsw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1e0Syf-0000aH-8b; Fri, 06 Oct 2017 13:47:45 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1e0SwD-0005nM-DV for linux-arm-kernel@bombadil.infradead.org; Fri, 06 Oct 2017 13:45:13 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=FGL/ORsB6FRRjf44zFBlEQ77c9hJrS/pAw3a4fnf0uM=; b=ZE+jfwkDGBnmqbL1FFN7xGUeo 87DuaTpzWAe1m07UExgxUdh9puPcP80BDPVX/dHQ3XIMPrpZIeIGIWOoTjooyvNYTfsWF7wdlfi0I VPuuAZ1DLFKMt+4lZZOOPhoEk94tqb5y5v7ua1DXUIhkyH+3jRLc6iszl6CAkIGvAJzNwHvcbPAqA veolqw0WstlS6jN9cI7U7swELAt9Gl+kNopiZKm4BIb9Cs2emz8apKDkogV80Zoib7LVKG1Kp/PDy 1xy70zWsyG3UyFUAHS8lfU/wWQ7CRZBK9M+ud7axHAysheL+bScf/ExGJrQQrE5DF7h0TE5DCn3/v FXVb6B0bQ==; Received: from foss.arm.com ([217.140.101.70]) by casper.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1e0Sia-0007I9-Fk for linux-arm-kernel@lists.infradead.org; Fri, 06 Oct 2017 13:31:11 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 199F01435; Fri, 6 Oct 2017 06:30:51 -0700 (PDT) Received: from e106794-lin.cambridge.arm.com (e106794-lin.cambridge.arm.com [10.1.211.72]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 4041D3F578; Fri, 6 Oct 2017 06:30:46 -0700 (PDT) From: Jean-Philippe Brucker To: linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org, linux-acpi@vger.kernel.org, devicetree@vger.kernel.org, iommu@lists.linux-foundation.org Subject: [RFCv2 PATCH 35/36] iommu/arm-smmu-v3: Add support for PRI Date: Fri, 6 Oct 2017 14:32:02 +0100 Message-Id: <20171006133203.22803-36-jean-philippe.brucker@arm.com> X-Mailer: git-send-email 2.13.3 In-Reply-To: <20171006133203.22803-1-jean-philippe.brucker@arm.com> References: <20171006133203.22803-1-jean-philippe.brucker@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20171006_143108_873881_8C927192 X-CRM114-Status: GOOD ( 19.57 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, xieyisheng1@huawei.com, gabriele.paoloni@huawei.com, catalin.marinas@arm.com, will.deacon@arm.com, okaya@codeaurora.org, yi.l.liu@intel.com, lorenzo.pieralisi@arm.com, ashok.raj@intel.com, tn@semihalf.com, joro@8bytes.org, rfranz@cavium.com, lenb@kernel.org, jacob.jun.pan@linux.intel.com, alex.williamson@redhat.com, robh+dt@kernel.org, thunder.leizhen@huawei.com, bhelgaas@google.com, dwmw2@infradead.org, liubo95@huawei.com, rjw@rjwysocki.net, robdclark@gmail.com, hanjun.guo@linaro.org, sudeep.holla@arm.com, robin.murphy@arm.com, nwatters@codeaurora.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP For PCI devices that support it, enable the PRI capability and handle PRI Page Requests with the generic fault handler. Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/arm-smmu-v3.c | 176 ++++++++++++++++++++++++++++++-------------- 1 file changed, 122 insertions(+), 54 deletions(-) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index f008b4617cd4..852714f35010 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -272,6 +272,7 @@ #define STRTAB_STE_1_S1COR_SHIFT 4 #define STRTAB_STE_1_S1CSH_SHIFT 6 +#define STRTAB_STE_1_PPAR (1UL << 18) #define STRTAB_STE_1_S1STALLD (1UL << 27) #define STRTAB_STE_1_EATS_ABT 0UL @@ -426,9 +427,9 @@ #define CMDQ_PRI_1_GRPID_SHIFT 0 #define CMDQ_PRI_1_GRPID_MASK 0x1ffUL #define CMDQ_PRI_1_RESP_SHIFT 12 -#define CMDQ_PRI_1_RESP_DENY (0UL << CMDQ_PRI_1_RESP_SHIFT) -#define CMDQ_PRI_1_RESP_FAIL (1UL << CMDQ_PRI_1_RESP_SHIFT) -#define CMDQ_PRI_1_RESP_SUCC (2UL << CMDQ_PRI_1_RESP_SHIFT) +#define CMDQ_PRI_1_RESP_FAILURE (0UL << CMDQ_PRI_1_RESP_SHIFT) +#define CMDQ_PRI_1_RESP_INVALID (1UL << CMDQ_PRI_1_RESP_SHIFT) +#define CMDQ_PRI_1_RESP_SUCCESS (2UL << CMDQ_PRI_1_RESP_SHIFT) #define CMDQ_RESUME_0_SID_SHIFT 32 #define CMDQ_RESUME_0_SID_MASK 0xffffffffUL @@ -504,6 +505,7 @@ /* Flags for iommu_data in iommu_fault */ #define ARM_SMMU_FAULT_STALL (1 << 0) +#define ARM_SMMU_FAULT_RESP_PASID (1 << 1); /* Until ACPICA headers cover IORT rev. C */ #ifndef ACPI_IORT_SMMU_HISILICON_HI161X @@ -524,12 +526,6 @@ module_param_named(disable_ats_check, disable_ats_check, bool, S_IRUGO); MODULE_PARM_DESC(disable_ats_check, "By default, the SMMU checks whether each incoming transaction marked as translated is allowed by the stream configuration. This option disables the check."); -enum pri_resp { - PRI_RESP_DENY, - PRI_RESP_FAIL, - PRI_RESP_SUCC, -}; - enum arm_smmu_msi_index { EVTQ_MSI_INDEX, GERROR_MSI_INDEX, @@ -613,7 +609,7 @@ struct arm_smmu_cmdq_ent { u32 sid; u32 ssid; u16 grpid; - enum pri_resp resp; + enum iommu_fault_status resp; } pri; #define CMDQ_OP_RESUME 0x44 @@ -720,6 +716,7 @@ struct arm_smmu_strtab_ent { struct arm_smmu_s2_cfg *s2_cfg; bool can_stall; + bool prg_resp_needs_ssid; }; struct arm_smmu_strtab_cfg { @@ -1078,14 +1075,14 @@ static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent) cmd[0] |= (u64)ent->pri.sid << CMDQ_PRI_0_SID_SHIFT; cmd[1] |= ent->pri.grpid << CMDQ_PRI_1_GRPID_SHIFT; switch (ent->pri.resp) { - case PRI_RESP_DENY: - cmd[1] |= CMDQ_PRI_1_RESP_DENY; + case IOMMU_FAULT_STATUS_FAILURE: + cmd[1] |= CMDQ_PRI_1_RESP_FAILURE; break; - case PRI_RESP_FAIL: - cmd[1] |= CMDQ_PRI_1_RESP_FAIL; + case IOMMU_FAULT_STATUS_INVALID: + cmd[1] |= CMDQ_PRI_1_RESP_INVALID; break; - case PRI_RESP_SUCC: - cmd[1] |= CMDQ_PRI_1_RESP_SUCC; + case IOMMU_FAULT_STATUS_HANDLED: + cmd[1] |= CMDQ_PRI_1_RESP_SUCCESS; break; default: return -EINVAL; @@ -1214,8 +1211,13 @@ static int arm_smmu_fault_response(struct iommu_domain *domain, cmd.resume.stag = fault->id; cmd.resume.resp = resp; } else { - /* TODO: put PRI response here */ - return -EINVAL; + cmd.opcode = CMDQ_OP_PRI_RESP; + cmd.substream_valid = fault->iommu_data & + ARM_SMMU_FAULT_RESP_PASID; + cmd.pri.sid = sid; + cmd.pri.ssid = fault->pasid; + cmd.pri.grpid = fault->id; + cmd.pri.resp = resp; } arm_smmu_cmdq_issue_cmd(smmu_domain->smmu, &cmd); @@ -1631,6 +1633,9 @@ static void arm_smmu_write_strtab_ent(struct arm_smmu_device *smmu, u32 sid, STRTAB_STE_1_STRW_EL2 : STRTAB_STE_1_STRW_NSEL1) << STRTAB_STE_1_STRW_SHIFT); + if (ste->prg_resp_needs_ssid) + dst[1] |= STRTAB_STE_1_PPAR; + if (smmu->features & ARM_SMMU_FEAT_STALLS && !(smmu->features & ARM_SMMU_FEAT_STALL_FORCE) && !ste->can_stall) @@ -1856,40 +1861,42 @@ static irqreturn_t arm_smmu_evtq_thread(int irq, void *dev) static void arm_smmu_handle_ppr(struct arm_smmu_device *smmu, u64 *evt) { - u32 sid, ssid; - u16 grpid; - bool ssv, last; - - sid = evt[0] >> PRIQ_0_SID_SHIFT & PRIQ_0_SID_MASK; - ssv = evt[0] & PRIQ_0_SSID_V; - ssid = ssv ? evt[0] >> PRIQ_0_SSID_SHIFT & PRIQ_0_SSID_MASK : 0; - last = evt[0] & PRIQ_0_PRG_LAST; - grpid = evt[1] >> PRIQ_1_PRG_IDX_SHIFT & PRIQ_1_PRG_IDX_MASK; - - dev_info(smmu->dev, "unexpected PRI request received:\n"); - dev_info(smmu->dev, - "\tsid 0x%08x.0x%05x: [%u%s] %sprivileged %s%s%s access at iova 0x%016llx\n", - sid, ssid, grpid, last ? "L" : "", - evt[0] & PRIQ_0_PERM_PRIV ? "" : "un", - evt[0] & PRIQ_0_PERM_READ ? "R" : "", - evt[0] & PRIQ_0_PERM_WRITE ? "W" : "", - evt[0] & PRIQ_0_PERM_EXEC ? "X" : "", - evt[1] & PRIQ_1_ADDR_MASK << PRIQ_1_ADDR_SHIFT); + u32 sid = evt[0] >> PRIQ_0_SID_SHIFT & PRIQ_0_SID_MASK; - if (last) { - struct arm_smmu_cmdq_ent cmd = { - .opcode = CMDQ_OP_PRI_RESP, - .substream_valid = ssv, - .pri = { - .sid = sid, - .ssid = ssid, - .grpid = grpid, - .resp = PRI_RESP_DENY, - }, - }; + struct arm_smmu_master_data *master; + struct iommu_domain *domain; + struct iommu_fault fault = { + .pasid = evt[0] >> PRIQ_0_SSID_SHIFT & PRIQ_0_SSID_MASK, + .id = evt[1] >> PRIQ_1_PRG_IDX_SHIFT & PRIQ_1_PRG_IDX_MASK, + .address = evt[1] & PRIQ_1_ADDR_MASK << PRIQ_1_ADDR_SHIFT, + .flags = IOMMU_FAULT_GROUP | IOMMU_FAULT_RECOVERABLE, + }; - arm_smmu_cmdq_issue_cmd(smmu, &cmd); - } + if (evt[0] & PRIQ_0_SSID_V) + fault.flags |= IOMMU_FAULT_PASID; + if (evt[0] & PRIQ_0_PRG_LAST) + fault.flags |= IOMMU_FAULT_LAST; + if (evt[0] & PRIQ_0_PERM_READ) + fault.flags |= IOMMU_FAULT_READ; + if (evt[0] & PRIQ_0_PERM_WRITE) + fault.flags |= IOMMU_FAULT_WRITE; + if (evt[0] & PRIQ_0_PERM_EXEC) + fault.flags |= IOMMU_FAULT_EXEC; + if (evt[0] & PRIQ_0_PERM_PRIV) + fault.flags |= IOMMU_FAULT_PRIV; + + master = arm_smmu_find_master(smmu, sid); + if (WARN_ON(!master)) + return; + + if (fault.flags & IOMMU_FAULT_PASID && master->ste.prg_resp_needs_ssid) + fault.iommu_data |= ARM_SMMU_FAULT_RESP_PASID; + + domain = iommu_get_domain_for_dev(master->dev); + if (WARN_ON(!domain)) + return; + + handle_iommu_fault(domain, master->dev, &fault); } static irqreturn_t arm_smmu_priq_thread(int irq, void *dev) @@ -1967,7 +1974,8 @@ static int arm_smmu_flush_queues(struct notifier_block *nb, if (master) { if (master->ste.can_stall) arm_smmu_flush_queue(smmu, &smmu->evtq.q, "evtq"); - /* TODO: add support for PRI */ + else if (master->can_fault) + arm_smmu_flush_queue(smmu, &smmu->priq.q, "priq"); return 0; } @@ -2933,6 +2941,46 @@ static int arm_smmu_enable_ats(struct arm_smmu_master_data *master) return 0; } +static int arm_smmu_enable_pri(struct arm_smmu_master_data *master) +{ + int ret, pos; + struct pci_dev *pdev; + /* + * TODO: find a good inflight PPR number. We should divide the PRI queue + * by the number of PRI-capable devices, but it's impossible to know + * about current and future (hotplugged) devices. So we're at risk of + * dropping PPRs (and leaking pending requests in the FQ). + */ + size_t max_inflight_pprs = 16; + struct arm_smmu_device *smmu = master->smmu; + + if (!(smmu->features & ARM_SMMU_FEAT_PRI) || !dev_is_pci(master->dev)) + return -ENOSYS; + + pdev = to_pci_dev(master->dev); + + pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_PRI); + if (!pos) + return -ENOSYS; + + ret = pci_reset_pri(pdev); + if (ret) + return ret; + + ret = pci_enable_pri(pdev, max_inflight_pprs); + if (ret) { + dev_err(master->dev, "cannot enable PRI: %d\n", ret); + return ret; + } + + master->can_fault = true; + master->ste.prg_resp_needs_ssid = pci_prg_resp_requires_prefix(pdev); + + dev_dbg(master->dev, "enabled PRI"); + + return 0; +} + static void arm_smmu_disable_ats(struct arm_smmu_master_data *master) { struct pci_dev *pdev; @@ -2948,6 +2996,22 @@ static void arm_smmu_disable_ats(struct arm_smmu_master_data *master) pci_disable_ats(pdev); } +static void arm_smmu_disable_pri(struct arm_smmu_master_data *master) +{ + struct pci_dev *pdev; + + if (!dev_is_pci(master->dev)) + return; + + pdev = to_pci_dev(master->dev); + + if (!pdev->pri_enabled) + return; + + pci_disable_pri(pdev); + master->can_fault = false; +} + static int arm_smmu_insert_master(struct arm_smmu_device *smmu, struct arm_smmu_master_data *master) { @@ -3070,12 +3134,13 @@ static int arm_smmu_add_device(struct device *dev) master->ste.can_stall = true; } - arm_smmu_enable_ats(master); + if (!arm_smmu_enable_ats(master)) + arm_smmu_enable_pri(master); group = iommu_group_get_for_dev(dev); if (IS_ERR(group)) { ret = PTR_ERR(group); - goto err_disable_ats; + goto err_disable_pri; } iommu_group_put(group); @@ -3084,7 +3149,8 @@ static int arm_smmu_add_device(struct device *dev) return 0; -err_disable_ats: +err_disable_pri: + arm_smmu_disable_pri(master); arm_smmu_disable_ats(master); return ret; @@ -3104,6 +3170,8 @@ static void arm_smmu_remove_device(struct device *dev) if (master && master->ste.assigned) arm_smmu_detach_dev(dev); arm_smmu_remove_master(smmu, master); + + arm_smmu_disable_pri(master); arm_smmu_disable_ats(master); iommu_group_remove_device(dev);