From patchwork Fri Apr 9 03:44:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shenming Lu X-Patchwork-Id: 12192931 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE901C43462 for ; Fri, 9 Apr 2021 03:44:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A996E61042 for ; Fri, 9 Apr 2021 03:44:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233253AbhDIDo5 (ORCPT ); Thu, 8 Apr 2021 23:44:57 -0400 Received: from szxga04-in.huawei.com ([45.249.212.190]:15637 "EHLO szxga04-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232662AbhDIDoz (ORCPT ); Thu, 8 Apr 2021 23:44:55 -0400 Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.59]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4FGkTF20x7znZ7c; Fri, 9 Apr 2021 11:41:53 +0800 (CST) Received: from DESKTOP-7FEPK9S.china.huawei.com (10.174.184.135) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.498.0; Fri, 9 Apr 2021 11:44:30 +0800 From: Shenming Lu To: Alex Williamson , Cornelia Huck , Will Deacon , Robin Murphy , Joerg Roedel , Jean-Philippe Brucker , Eric Auger , , , , , CC: Kevin Tian , Lu Baolu , , Christoph Hellwig , Jonathan Cameron , Barry Song , , , Subject: [RFC PATCH v3 0/8] Add IOPF support for VFIO passthrough Date: Fri, 9 Apr 2021 11:44:12 +0800 Message-ID: <20210409034420.1799-1-lushenming@huawei.com> X-Mailer: git-send-email 2.27.0.windows.1 MIME-Version: 1.0 X-Originating-IP: [10.174.184.135] X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Hi, Requesting for your comments and suggestions. :-) The static pinning and mapping problem in VFIO and possible solutions have been discussed a lot [1, 2]. One of the solutions is to add I/O Page Fault support for VFIO devices. Different from those relatively complicated software approaches such as presenting a vIOMMU that provides the DMA buffer information (might include para-virtualized optimizations), IOPF mainly depends on the hardware faulting capability, such as the PCIe PRI extension or Arm SMMU stall model. What's more, the IOPF support in the IOMMU driver has already been implemented in SVA [3]. So we add IOPF support for VFIO passthrough based on the IOPF part of SVA in this series. We have measured its performance with UADK [4] (passthrough an accelerator to a VM(1U16G)) on Hisilicon Kunpeng920 board (and compared with host SVA): Run hisi_sec_test... - with varying sending times and message lengths - with/without IOPF enabled (speed slowdown) when msg_len = 1MB (and PREMAP_LEN (in Patch 4) = 1): slowdown (num of faults) times VFIO IOPF host SVA 1 63.4% (518) 82.8% (512) 100 22.9% (1058) 47.9% (1024) 1000 2.6% (1071) 8.5% (1024) when msg_len = 10MB (and PREMAP_LEN = 512): slowdown (num of faults) times VFIO IOPF 1 32.6% (13) 100 3.5% (26) 1000 1.6% (26) History: v2 -> v3 - Nit fixes. - No reason to disable reporting the unrecoverable faults. (baolu) - Maintain a global IOPF enabled group list. - Split the pre-mapping optimization to be a separate patch. - Add selective faulting support (use vfio_pin_pages to indicate the non-faultable scope and add a new struct vfio_range to record it, untested). (Kevin) v1 -> v2 - Numerous improvements following the suggestions. Thanks a lot to all of you. Note that PRI is not supported at the moment since there is no hardware. Links: [1] Lesokhin I, et al. Page Fault Support for Network Controllers. In ASPLOS, 2016. [2] Tian K, et al. coIOMMU: A Virtual IOMMU with Cooperative DMA Buffer Tracking for Efficient Memory Management in Direct I/O. In USENIX ATC, 2020. [3] https://patchwork.kernel.org/project/linux-arm-kernel/cover/20210401154718.307519-1-jean-philippe@linaro.org/ [4] https://github.com/Linaro/uadk Thanks, Shenming Shenming Lu (8): iommu: Evolve the device fault reporting framework vfio/type1: Add a page fault handler vfio/type1: Add an MMU notifier to avoid pinning vfio/type1: Pre-map more pages than requested in the IOPF handling vfio/type1: VFIO_IOMMU_ENABLE_IOPF vfio/type1: No need to statically pin and map if IOPF enabled vfio/type1: Add selective DMA faulting support vfio: Add nested IOPF support .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 3 +- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 18 +- drivers/iommu/iommu.c | 56 +- drivers/vfio/vfio.c | 85 +- drivers/vfio/vfio_iommu_type1.c | 1000 ++++++++++++++++- include/linux/iommu.h | 19 +- include/linux/vfio.h | 13 + include/uapi/linux/iommu.h | 4 + include/uapi/linux/vfio.h | 6 + 9 files changed, 1181 insertions(+), 23 deletions(-)