From patchwork Fri Apr 9 03:44:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shenming Lu X-Patchwork-Id: 12192963 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC777C43460 for ; Fri, 9 Apr 2021 03:47:57 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5949461042 for ; Fri, 9 Apr 2021 03:47:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5949461042 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:CC:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=j0Zyd3t+XgXZbZF6bherLg2q5qk1P1vWsx3wxSh51ao=; b=VQumaaNPtnpABORDRW9en2cmz 4xZ7A/LXzAzRXgzRJHL9XcJbPtAucyh6HxF72FcJx556iGA2Oy5kh4+11TH8T3HPpza7CvsVtfqld 1zT3ELLcIOSgGjextrU4JbJVdnxCODHo9/pdWeotn/nZBju01aJ7Fr0QLOc3RH27T5wjpOEH+19FM BFFL40z/CTyTNIMJx4CJCoIIrKnMSTFhLGkQ44qMvIapeAuvDD4yJxe7dkoO8J1TQrUzsC6PkeZTf iBOJZRyNnM+nuT0ULIQvR9ES9FmSdbeK7XeY+xi1EB3Q6sosQq05nFRFscVc5OlEdmtamcDFOKdOy XdiO3yjrQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lUi5c-009u6y-U1; Fri, 09 Apr 2021 03:45:49 +0000 Received: from szxga04-in.huawei.com ([45.249.212.190]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lUi4f-009tvv-MX for linux-arm-kernel@lists.infradead.org; Fri, 09 Apr 2021 03:45:00 +0000 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.60]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4FGkTL39vRznYwQ; Fri, 9 Apr 2021 11:41:58 +0800 (CST) Received: from DESKTOP-7FEPK9S.china.huawei.com (10.174.184.135) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.498.0; Fri, 9 Apr 2021 11:44:38 +0800 From: Shenming Lu To: Alex Williamson , Cornelia Huck , Will Deacon , Robin Murphy , Joerg Roedel , Jean-Philippe Brucker , Eric Auger , , , , , CC: Kevin Tian , Lu Baolu , , Christoph Hellwig , Jonathan Cameron , Barry Song , , , Subject: [RFC PATCH v3 4/8] vfio/type1: Pre-map more pages than requested in the IOPF handling Date: Fri, 9 Apr 2021 11:44:16 +0800 Message-ID: <20210409034420.1799-5-lushenming@huawei.com> X-Mailer: git-send-email 2.27.0.windows.1 In-Reply-To: <20210409034420.1799-1-lushenming@huawei.com> References: <20210409034420.1799-1-lushenming@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.174.184.135] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210409_044454_583203_F5CEBA10 X-CRM114-Status: GOOD ( 17.42 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To optimize for fewer page fault handlings, we can pre-map more pages than requested at once. Note that IOPF_PREMAP_LEN is just an arbitrary value for now, which we could try further tuning. Signed-off-by: Shenming Lu --- drivers/vfio/vfio_iommu_type1.c | 131 ++++++++++++++++++++++++++++++-- 1 file changed, 123 insertions(+), 8 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 1cb9d1f2717b..01e296c6dc9e 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -3217,6 +3217,91 @@ static int vfio_iommu_type1_dirty_pages(struct vfio_iommu *iommu, return -EINVAL; } +/* + * To optimize for fewer page fault handlings, try to + * pre-map more pages than requested. + */ +#define IOPF_PREMAP_LEN 512 + +/* + * Return 0 on success or a negative error code, the + * number of pages contiguously pinned is in @pinned. + */ +static int pin_pages_iopf(struct vfio_dma *dma, unsigned long vaddr, + unsigned long npages, unsigned long *pfn_base, + unsigned long *pinned, struct vfio_batch *batch) +{ + struct mm_struct *mm; + unsigned long pfn; + int ret = 0; + *pinned = 0; + + mm = get_task_mm(dma->task); + if (!mm) + return -ENODEV; + + if (batch->size) { + *pfn_base = page_to_pfn(batch->pages[batch->offset]); + pfn = *pfn_base; + } else { + *pfn_base = 0; + } + + while (npages) { + if (!batch->size) { + unsigned long req_pages = min_t(unsigned long, npages, + batch->capacity); + + ret = vaddr_get_pfns(mm, vaddr, req_pages, dma->prot, + &pfn, batch->pages); + if (ret < 0) + goto out; + + batch->size = ret; + batch->offset = 0; + ret = 0; + + if (!*pfn_base) + *pfn_base = pfn; + } + + while (true) { + if (pfn != *pfn_base + *pinned) + goto out; + + (*pinned)++; + npages--; + vaddr += PAGE_SIZE; + batch->offset++; + batch->size--; + + if (!batch->size) + break; + + pfn = page_to_pfn(batch->pages[batch->offset]); + } + + if (unlikely(disable_hugepages)) + break; + } + +out: + if (batch->size == 1 && !batch->offset) { + put_pfn(pfn, dma->prot); + batch->size = 0; + } + + mmput(mm); + return ret; +} + +static void unpin_pages_iopf(struct vfio_dma *dma, + unsigned long pfn, unsigned long npages) +{ + while (npages--) + put_pfn(pfn++, dma->prot); +} + /* VFIO I/O Page Fault handler */ static int vfio_iommu_type1_dma_map_iopf(struct iommu_fault *fault, void *data) { @@ -3225,9 +3310,11 @@ static int vfio_iommu_type1_dma_map_iopf(struct iommu_fault *fault, void *data) struct vfio_iopf_group *iopf_group; struct vfio_iommu *iommu; struct vfio_dma *dma; + struct vfio_batch batch; dma_addr_t iova = ALIGN_DOWN(fault->prm.addr, PAGE_SIZE); int access_flags = 0; - unsigned long bit_offset, vaddr, pfn; + size_t premap_len, map_len, mapped_len = 0; + unsigned long bit_offset, vaddr, pfn, i, npages; int ret; enum iommu_page_response_code status = IOMMU_PAGE_RESP_INVALID; struct iommu_page_response resp = {0}; @@ -3263,19 +3350,47 @@ static int vfio_iommu_type1_dma_map_iopf(struct iommu_fault *fault, void *data) if (IOPF_MAPPED_BITMAP_GET(dma, bit_offset)) goto out_success; + premap_len = IOPF_PREMAP_LEN << PAGE_SHIFT; + npages = dma->size >> PAGE_SHIFT; + map_len = PAGE_SIZE; + for (i = bit_offset + 1; i < npages; i++) { + if (map_len >= premap_len || IOPF_MAPPED_BITMAP_GET(dma, i)) + break; + map_len += PAGE_SIZE; + } vaddr = iova - dma->iova + dma->vaddr; + vfio_batch_init(&batch); - if (vfio_pin_page_external(dma, vaddr, &pfn, false)) - goto out_invalid; + while (map_len) { + ret = pin_pages_iopf(dma, vaddr + mapped_len, + map_len >> PAGE_SHIFT, &pfn, + &npages, &batch); + if (!npages) + break; - if (vfio_iommu_map(iommu, iova, pfn, 1, dma->prot)) { - put_pfn(pfn, dma->prot); - goto out_invalid; + if (vfio_iommu_map(iommu, iova + mapped_len, pfn, + npages, dma->prot)) { + unpin_pages_iopf(dma, pfn, npages); + vfio_batch_unpin(&batch, dma); + break; + } + + bitmap_set(dma->iopf_mapped_bitmap, + bit_offset + (mapped_len >> PAGE_SHIFT), npages); + + unpin_pages_iopf(dma, pfn, npages); + + map_len -= npages << PAGE_SHIFT; + mapped_len += npages << PAGE_SHIFT; + + if (ret) + break; } - bitmap_set(dma->iopf_mapped_bitmap, bit_offset, 1); + vfio_batch_fini(&batch); - put_pfn(pfn, dma->prot); + if (!mapped_len) + goto out_invalid; out_success: status = IOMMU_PAGE_RESP_SUCCESS;