From patchwork Fri Oct 13 14:40:13 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joerg Roedel X-Patchwork-Id: 10004923 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id B8E3060360 for ; Fri, 13 Oct 2017 14:41:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AC2E12902A for ; Fri, 13 Oct 2017 14:41:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A0F38290A6; Fri, 13 Oct 2017 14:41:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0E0C829094 for ; Fri, 13 Oct 2017 14:41:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758467AbdJMOlZ (ORCPT ); Fri, 13 Oct 2017 10:41:25 -0400 Received: from 8bytes.org ([81.169.241.247]:58126 "EHLO theia.8bytes.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757931AbdJMOkS (ORCPT ); Fri, 13 Oct 2017 10:40:18 -0400 Received: by theia.8bytes.org (Postfix, from userid 1000) id DD945438; Fri, 13 Oct 2017 16:40:16 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=8bytes.org; s=mail-1; t=1507905616; bh=tcbIi7VBXRjSdhn7XVRu3EVFz/UuB8DRGonmmtWAhRs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pdYaY352SvRAl6V6+exL92EPDdXzBQsO99TSKYPFYCbk96YHCVxWJ8XR7lyEvWAf+ StYCRFSjC3uKQZWaXnLDXUgDmy+8ipwuffwNepZlPuznyRytSm/1iRCDSplRcUVY6y hAVuT5pA8AM8EpN2fU6ZyS9Q9iAcZFryNxaDLnZe6Zf8nKt+hXbYXRe1Cpac9ASNVN yFq2KthbKuEbDDx6SmIvw4qCQ5ju28A149bszn51aQmZb/PjPBLZn/QrClnKi9evKw 8A+x5jkuF3zsGB/VAe0Fiw7Tgk6Hfm7E60y1k+hvhqZ5qdL6XRuviwPhQYegsagcFG fYZGWs1Zo5tQg== From: Joerg Roedel To: iommu@lists.linux-foundation.org, Alex Williamson Cc: Suravee Suthikulpanit , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Joerg Roedel Subject: [PATCH 4/4] vfio/type1: Gather TLB-syncs and pages to unpin Date: Fri, 13 Oct 2017 16:40:13 +0200 Message-Id: <1507905613-30695-5-git-send-email-joro@8bytes.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1507905613-30695-1-git-send-email-joro@8bytes.org> References: <1507905613-30695-1-git-send-email-joro@8bytes.org> Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Joerg Roedel After every unmap VFIO unpins the pages that where mapped by the IOMMU. This requires an IOTLB flush after every unmap and puts a high load on the IOMMU hardware and the device TLBs. Gather up to 32 ranges to flush and unpin and do the IOTLB flush once for all these ranges. This significantly reduces the number of IOTLB flushes in the unmapping path. Signed-off-by: Joerg Roedel --- drivers/vfio/vfio_iommu_type1.c | 106 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 101 insertions(+), 5 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 2b1e81f..86fc1da 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -107,6 +107,92 @@ struct vfio_pfn { static int put_pfn(unsigned long pfn, int prot); +static long vfio_unpin_pages_remote(struct vfio_dma *dma, dma_addr_t iova, + unsigned long pfn, long npage, + bool do_accounting); + +#define GATHER_ENTRIES 32 + +/* + * Gather TLB flushes before unpinning pages + */ +struct vfio_gather_entry { + dma_addr_t iova; + phys_addr_t phys; + size_t size; +}; + +struct vfio_gather { + unsigned fill; + struct vfio_gather_entry entries[GATHER_ENTRIES]; +}; + +/* + * The vfio_gather* functions below keep track of flushing the IOMMU TLB + * and unpinning the pages. It is safe to call them gather == NULL, in + * which case they will fall-back to flushing the TLB and unpinning the + * pages at every call. + */ +static long vfio_gather_flush(struct iommu_domain *domain, + struct vfio_dma *dma, + struct vfio_gather *gather) +{ + long unlocked = 0; + unsigned i; + + if (!gather) + goto out; + + /* First flush unmapped TLB entries */ + iommu_tlb_sync(domain); + + for (i = 0; i < gather->fill; i++) { + dma_addr_t iova = gather->entries[i].iova; + phys_addr_t phys = gather->entries[i].phys; + size_t size = gather->entries[i].size; + + unlocked += vfio_unpin_pages_remote(dma, iova, + phys >> PAGE_SHIFT, + size >> PAGE_SHIFT, + false); + } + + gather->fill = 0; + +out: + return unlocked; +} + +static long vfio_gather_add(struct iommu_domain *domain, + struct vfio_dma *dma, + struct vfio_gather *gather, + dma_addr_t iova, phys_addr_t phys, size_t size) +{ + long unlocked = 0; + + if (gather) { + unsigned index; + + if (gather->fill == GATHER_ENTRIES) + unlocked = vfio_gather_flush(domain, dma, gather); + + index = gather->fill++; + + gather->entries[index].iova = iova; + gather->entries[index].phys = phys; + gather->entries[index].size = size; + } else { + iommu_tlb_sync(domain); + + unlocked = vfio_unpin_pages_remote(dma, iova, + phys >> PAGE_SHIFT, + size >> PAGE_SHIFT, + false); + } + + return unlocked; +} + /* * This code handles mapping and unmapping of user data buffers * into DMA'ble space using the IOMMU @@ -653,6 +739,7 @@ static long vfio_unmap_unpin(struct vfio_iommu *iommu, struct vfio_dma *dma, { dma_addr_t iova = dma->iova, end = dma->iova + dma->size; struct vfio_domain *domain, *d; + struct vfio_gather *gather; long unlocked = 0; if (!dma->size) @@ -662,6 +749,12 @@ static long vfio_unmap_unpin(struct vfio_iommu *iommu, struct vfio_dma *dma, return 0; /* + * No need to check return value - It is safe to continue with a + * NULL pointer. + */ + gather = kzalloc(sizeof(*gather), GFP_KERNEL); + + /* * We use the IOMMU to track the physical addresses, otherwise we'd * need a much more complicated tracking system. Unfortunately that * means we need to use one of the iommu domains to figure out the @@ -706,17 +799,20 @@ static long vfio_unmap_unpin(struct vfio_iommu *iommu, struct vfio_dma *dma, break; iommu_tlb_range_add(domain->domain, iova, unmapped); - iommu_tlb_sync(domain->domain); - unlocked += vfio_unpin_pages_remote(dma, iova, - phys >> PAGE_SHIFT, - unmapped >> PAGE_SHIFT, - false); + unlocked += vfio_gather_add(domain->domain, dma, gather, + iova, phys, unmapped); + iova += unmapped; cond_resched(); } + unlocked += vfio_gather_flush(domain->domain, dma, gather); + + kfree(gather); + gather = NULL; + dma->iommu_mapped = false; if (do_accounting) { vfio_lock_acct(dma->task, -unlocked, NULL);