From patchwork Mon Feb 27 19:54:38 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jean-Philippe Brucker X-Patchwork-Id: 9594005 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 939F0604AB for ; Mon, 27 Feb 2017 20:20:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 863B52818B for ; Mon, 27 Feb 2017 20:20:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7A702281DB; Mon, 27 Feb 2017 20:20:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id E455A2818B for ; Mon, 27 Feb 2017 20:20:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=aro8KRTdAL81P72oNzuewYjJ/7x9Fxf5mRRsSKOOhEE=; b=Rsadh0jVN7i+Tgbcx8sNiZoi+I Z+Y/9yXwkpDbzytsortZi2MZfTOZaASK7GZZkeSH3De6r6xmCCANqTAz8Ikb9FjfSFCFiOphR+pgx b3lf58jfPjCq9iyLrUmzSJmT8HGGLJ/mLezyDKC3iXH0BSN/CI6hbgtxyi35yMihtIBTpgL0QdyfV 59god5MY35m0EIa210djHDh72YxJzRKueOUZ7s2MN+2TvxOV2vcld/RkZCbhNNBwMU1H8UpRPocut hwNnf9gByApMkaxgoMNmwXkncDd6om5BFUW1bcTUtO/R1lSx+bkamHfsmA1BgNKBM/5VckT7a5Qbi h4W15SxQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1ciRmE-000077-38; Mon, 27 Feb 2017 20:20:10 +0000 Received: from casper.infradead.org ([2001:770:15f::2]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1ciRSR-0000Jq-TT for linux-arm-kernel@bombadil.infradead.org; Mon, 27 Feb 2017 19:59:44 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=kYmG1Wy2dKxySG5/XoIOOjjNtNr4ugF38vGBTpvRahQ=; b=s4l58gambNAdyLjEEciCNxcTZ FGjpWRA7ETbVlXRTDEq+1K8yPCMBjfDW8jhcC3TqpTBzQL4QOzf2iInouZyeUrSXV/+yLCMahMI9H NMZvDTmF91TfGQ+NYvZIj4aMOTMKVQfN3nn91bEajYrirTnZ6qglQxgCkGQ0/26RKR0q5RSNYhvD8 eCDNwdyMFso+HSLd35tVsIXiR1Uw+x9jx7aK/1PYsKfJydCKIX7ParMNZoBluZbSaWCrcrQW3LIsK 2VizD/gbXD/zaQ1u1KaD8oZKbE/9T79QD8UuS1OC5Z7SeYMmf03RkytZ/9PAtkwrZrqnjBhdvVNHz 2JYMzCrBg==; Received: from foss.arm.com ([217.140.101.70]) by casper.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1ciRSM-0003Hh-BW for linux-arm-kernel@lists.infradead.org; Mon, 27 Feb 2017 19:59:42 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 50A971991; Mon, 27 Feb 2017 11:59:22 -0800 (PST) Received: from e106794-lin.cambridge.arm.com (e106794-lin.cambridge.arm.com [10.1.210.60]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id A56EB3F3E1; Mon, 27 Feb 2017 11:59:19 -0800 (PST) From: Jean-Philippe Brucker To: Subject: [RFC PATCH 27/30] iommu/arm-smmu-v3: Handle PRI queue overflow Date: Mon, 27 Feb 2017 19:54:38 +0000 Message-Id: <20170227195441.5170-28-jean-philippe.brucker@arm.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170227195441.5170-1-jean-philippe.brucker@arm.com> References: <20170227195441.5170-1-jean-philippe.brucker@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20170227_195938_697133_8C706093 X-CRM114-Status: GOOD ( 17.14 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lorenzo Pieralisi , Shanker Donthineni , kvm@vger.kernel.org, Catalin Marinas , Joerg Roedel , Sinan Kaya , Will Deacon , iommu@lists.linux-foundation.org, Harv Abdulhamid , Alex Williamson , linux-pci@vger.kernel.org, Bjorn Helgaas , Robin Murphy , David Woodhouse , linux-arm-kernel@lists.infradead.org, Nate Watterson MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP When the PRI queue is full, it enters overflow condition, which is sticky and exited by the PRI thread once it has had time to free up some slots. During that time, no new entry is added to the queue. The SMMU automatically replies to PRI Page Requests (PPR) that have "last=1" with "success", to let the device retry later. PPRs that have "last=0" and PASID Stop Markers are silently ignored. Two related issues need to be fixed: * Any PPR in the PRI queue prior to the overflow condition might be in a Page Request Group (PRG) that has its last entry auto-responded while in overflow. Until we fix up the overflow, ignore any non-last PPR received by the PRI thread. * In addition, any PRG of PPRs already committed to the fault queue is now potentially invalid, since their last PPR might have been lost. Wait until the overflow condition is fixed, and destroy *all* remaining PRG structures :( We do that by appending a PRG sweeper work to the fault queue, that will do some inefficient sweeping and lock up the fault queue for a while. Awful, but necessary. Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/arm-smmu-v3.c | 37 ++++++++++++++++++++++++++++++++++--- 1 file changed, 34 insertions(+), 3 deletions(-) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index b5d45c1e14d1..1a5e72752e6d 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -723,6 +723,7 @@ struct arm_smmu_device { struct list_head tasks; struct workqueue_struct *fault_queue; + struct work_struct flush_prgs; struct list_head domains; struct mutex domains_mutex; @@ -1798,7 +1799,8 @@ static irqreturn_t arm_smmu_evtq_thread(int irq, void *dev) static void arm_smmu_handle_fault(struct work_struct *work); -static void arm_smmu_handle_ppr(struct arm_smmu_device *smmu, u64 *evt) +static void arm_smmu_handle_ppr(struct arm_smmu_device *smmu, u64 *evt, + bool overflowing) { struct arm_smmu_fault *fault; struct arm_smmu_fault params = { @@ -1817,6 +1819,9 @@ static void arm_smmu_handle_ppr(struct arm_smmu_device *smmu, u64 *evt) .priv = evt[0] & PRIQ_0_PERM_PRIV, }; + if (overflowing && !params.last) + return; + fault = kmem_cache_alloc(arm_smmu_fault_cache, GFP_KERNEL); if (!fault) { /* Out of memory, tell the device to retry later */ @@ -1834,6 +1839,7 @@ static irqreturn_t arm_smmu_priq_thread(int irq, void *dev) struct arm_smmu_device *smmu = dev; struct arm_smmu_queue *q = &smmu->priq.q; size_t queue_size = 1 << q->max_n_shift; + bool overflowing = false; u64 evt[PRIQ_ENT_DWORDS]; size_t i = 0; @@ -1842,7 +1848,7 @@ static irqreturn_t arm_smmu_priq_thread(int irq, void *dev) do { while (!queue_remove_raw(q, evt)) { spin_unlock(&smmu->priq.wq.lock); - arm_smmu_handle_ppr(smmu, evt); + arm_smmu_handle_ppr(smmu, evt, overflowing); spin_lock(&smmu->priq.wq.lock); if (++i == queue_size) { smmu->priq.batch++; @@ -1851,8 +1857,10 @@ static irqreturn_t arm_smmu_priq_thread(int irq, void *dev) } } - if (queue_sync_prod(q) == -EOVERFLOW) + if (queue_sync_prod(q) == -EOVERFLOW) { dev_err(smmu->dev, "PRIQ overflow detected -- requests lost\n"); + overflowing = true; + } } while (!queue_empty(q)); /* Sync our overflow flag, as we believe we're up to speed */ @@ -1863,6 +1871,9 @@ static irqreturn_t arm_smmu_priq_thread(int irq, void *dev) spin_unlock(&smmu->priq.wq.lock); + if (overflowing) + queue_work(smmu->fault_queue, &smmu->flush_prgs); + return IRQ_HANDLED; } @@ -2820,6 +2831,24 @@ static void arm_smmu_handle_fault(struct work_struct *work) kfree(fault); } +static void arm_smmu_flush_prgs(struct work_struct *work) +{ + struct arm_smmu_device *smmu; + struct arm_smmu_task *smmu_task; + struct arm_smmu_pri_group *prg, *next_prg; + + smmu = container_of(work, struct arm_smmu_device, flush_prgs); + + spin_lock(&smmu->contexts_lock); + list_for_each_entry(smmu_task, &smmu->tasks, smmu_head) { + list_for_each_entry_safe(prg, next_prg, &smmu_task->prgs, list) { + list_del(&prg->list); + kfree(prg); + } + } + spin_unlock(&smmu->contexts_lock); +} + static void arm_smmu_sweep_contexts(struct work_struct *work) { u64 batch; @@ -4269,6 +4298,8 @@ static int arm_smmu_init_structures(struct arm_smmu_device *smmu) smmu->fault_queue = alloc_ordered_workqueue("smmu_fault_queue", 0); if (!smmu->fault_queue) return -ENOMEM; + + INIT_WORK(&smmu->flush_prgs, arm_smmu_flush_prgs); } return arm_smmu_init_strtab(smmu);