From patchwork Thu May 18 05:34:35 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "lan,Tianyu" X-Patchwork-Id: 9733873 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id E47C3602C8 for ; Thu, 18 May 2017 11:42:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D362320952 for ; Thu, 18 May 2017 11:42:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C81232621D; Thu, 18 May 2017 11:42:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00, DATE_IN_PAST_06_12, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 5F9DF20952 for ; Thu, 18 May 2017 11:42:41 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dBJn5-0004Fp-7i; Thu, 18 May 2017 11:40:23 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dBJn4-0004F0-3t for xen-devel@lists.xen.org; Thu, 18 May 2017 11:40:22 +0000 Received: from [85.158.143.35] by server-9.bemta-6.messagelabs.com id F8/78-03557-5288D195; Thu, 18 May 2017 11:40:21 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrPLMWRWlGSWpSXmKPExsVywNxEW1elQzb SYNZ5E4slHxezODB6HN39mymAMYo1My8pvyKBNaP1yRv2ghbdim0nvzI2MB6V7WLk5BASqJT4 fG45M4gtIcArcWTZDFYI219i19UFQHEuoJoORolv7w6ygSTYBNQlTiyeyAhiiwhIS1z7fJkRp IhZYDOjxKxvK9hBEsIC9hLTbi0Fm8QioCrRvfgOWJxXwEVi9uN2FogNChJTHr4H28wp4Cqxbe stFoiLXCQaF/ayTmDkXcDIsIpRozi1qCy1SNfISC+pKDM9oyQ3MTNH19DATC83tbg4MT01JzG pWC85P3cTIzAgGIBgB+Oa+YGHGCU5mJREeQ8XyEYK8SXlp1RmJBZnxBeV5qQWH2KU4eBQkuDd 0AaUEyxKTU+tSMvMAYYmTFqCg0dJhHcNSJq3uCAxtzgzHSJ1ilFRSpx3I0hCACSRUZoH1waLh 0uMslLCvIxAhwjxFKQW5WaWoMq/YhTnYFQS5u0CmcKTmVcCN/0V0GImoMXND6RBFpckIqSkGh g7zb6yijVt/3CtW9XfzE9nRUlYW9x+vbMXp11nEdGrVdxvq+63cdMFyY1/BTv0Pc1kSprlZrU GKW/4v3GOtvss0aj270qzb1yeoC4hsndih17AswNZHt8vJ+w+5BZSF2v7QuPD44AoXmHdhqs5 07IL9H6sbPpm//GRSfcMvSlTP2gmLdsRyKPEUpyRaKjFXFScCAB6YteOggIAAA== X-Env-Sender: tianyu.lan@intel.com X-Msg-Ref: server-8.tower-21.messagelabs.com!1495107618!69216238!1 X-Originating-IP: [192.55.52.43] X-SpamReason: No, hits=0.8 required=7.0 tests=DATE_IN_PAST_06_12 X-StarScan-Received: X-StarScan-Version: 9.4.12; banners=-,-,- X-VirusChecked: Checked Received: (qmail 29166 invoked from network); 18 May 2017 11:40:20 -0000 Received: from mga05.intel.com (HELO mga05.intel.com) (192.55.52.43) by server-8.tower-21.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 18 May 2017 11:40:20 -0000 Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga105.fm.intel.com with ESMTP; 18 May 2017 04:40:18 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.38,358,1491289200"; d="scan'208";a="858507292" Received: from sky-ws.sh.intel.com (HELO localhost) ([10.239.48.124]) by FMSMGA003.fm.intel.com with ESMTP; 18 May 2017 04:40:16 -0700 From: Lan Tianyu To: xen-devel@lists.xen.org Date: Thu, 18 May 2017 01:34:35 -0400 Message-Id: <1495085696-10819-6-git-send-email-tianyu.lan@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1495085696-10819-1-git-send-email-tianyu.lan@intel.com> References: <1495085696-10819-1-git-send-email-tianyu.lan@intel.com> Cc: Lan Tianyu , kevin.tian@intel.com, wei.liu2@citrix.com, andrew.cooper3@citrix.com, ian.jackson@eu.citrix.com, jbeulich@suse.com, chao.gao@intel.com Subject: [Xen-devel] [RFC PATCH V2 5/26] Xen/doc: Add Xen virtual IOMMU doc X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP This patch is to add Xen virtual IOMMU doc to introduce motivation, framework, vIOMMU hypercall and xl configuration. Signed-off-by: Lan Tianyu --- docs/misc/viommu.txt | 129 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 129 insertions(+) create mode 100644 docs/misc/viommu.txt diff --git a/docs/misc/viommu.txt b/docs/misc/viommu.txt new file mode 100644 index 0000000..76d4cee --- /dev/null +++ b/docs/misc/viommu.txt @@ -0,0 +1,129 @@ +Xen virtual IOMMU + +Motivation +========== +*) Enable more than 255 vcpu support +HPC cloud service requires VM provides high performance parallel +computing and we hope to create a huge VM with >255 vcpu on one machine +to meet such requirement. Pin each vcpu to separate pcpus. + +To support >255 vcpus, X2APIC mode in guest is necessary because legacy +APIC(XAPIC) just supports 8-bit APIC ID and it only can support 255 +vcpus at most. X2APIC mode supports 32-bit APIC ID and it requires +interrupt mapping function of vIOMMU. + +The reason for this is that there is no modification to existing PCI MSI +and IOAPIC with the introduction of X2APIC. PCI MSI/IOAPIC can only send +interrupt message containing 8-bit APIC ID, which cannot address >255 +cpus. Interrupt remapping supports 32-bit APIC ID and so it's necessary +to enable >255 cpus with x2apic mode. + + +vIOMMU Architecture +=================== +vIOMMU device model is inside Xen hypervisor for following factors + 1) Avoid round trips between Qemu and Xen hypervisor + 2) Ease of integration with the rest of hypervisor + 3) HVMlite/PVH doesn't use Qemu + +* Interrupt remapping overview. +Interrupts from virtual devices and physical devices are delivered +to vLAPIC from vIOAPIC and vMSI. vIOMMU needs to remap interrupt during +this procedure. + ++---------------------------------------------------+ +|Qemu |VM | +| | +----------------+ | +| | | Device driver | | +| | +--------+-------+ | +| | ^ | +| +----------------+ | +--------+-------+ | +| | Virtual device | | | IRQ subsystem | | +| +-------+--------+ | +--------+-------+ | +| | | ^ | +| | | | | ++---------------------------+-----------------------+ +|hyperviosr | | VIRQ | +| | +---------+--------+ | +| | | vLAPIC | | +| |VIRQ +---------+--------+ | +| | ^ | +| | | | +| | +---------+--------+ | +| | | vIOMMU | | +| | +---------+--------+ | +| | ^ | +| | | | +| | +---------+--------+ | +| | | vIOAPIC/vMSI | | +| | +----+----+--------+ | +| | ^ ^ | +| +-----------------+ | | +| | | ++---------------------------------------------------+ +HW |IRQ + +-------------------+ + | PCI Device | + +-------------------+ + + +vIOMMU hypercall +================ +Introduce new domctl hypercall "xen_domctl_viommu_op" to create/destroy +vIOMMU and query vIOMMU capabilities that device model can support. + +* vIOMMU hypercall parameter structure +struct xen_domctl_viommu_op { + uint32_t cmd; +#define XEN_DOMCTL_create_viommu 0 +#define XEN_DOMCTL_destroy_viommu 1 +#define XEN_DOMCTL_query_viommu_caps 2 + union { + struct { + /* IN - vIOMMU type */ + uint64_t viommu_type; + /* IN - MMIO base address of vIOMMU. */ + uint64_t base_address; + /* IN - Length of MMIO region */ + uint64_t length; + /* IN - Capabilities with which we want to create */ + uint64_t capabilities; + /* OUT - vIOMMU identity */ + uint32_t viommu_id; + } create_viommu; + + struct { + /* IN - vIOMMU identity */ + uint32_t viommu_id; + } destroy_viommu; + + struct { + /* IN - vIOMMU type */ + uint64_t viommu_type; + /* OUT - vIOMMU Capabilities */ + uint64_t caps; + } query_caps; + } u; +}; + +- XEN_DOMCTL_query_viommu_caps + Query capabilities of vIOMMU device model. vIOMMU_type specifies +which vendor vIOMMU device model(E,G Intel VTD) is targeted and hypervisor +returns capability bits(E,G interrupt remapping bit). + +- XEN_DOMCTL_create_viommu + Create vIOMMU device with vIOMMU_type, capabilities, MMIO +base address and length. Hypervisor returns viommu_id. Capabilities should +be in range of value returned by query_viommu_caps hypercall. + +- XEN_DOMCTL_destroy_viommu + Destroy vIOMMU in Xen hypervisor with viommu_id as parameters. + +xl vIOMMU configuration +======================= +viommu="type=vtd,intremap=1,x2apic=1" + +"type" - Specify vIOMMU device model type. Currently only supports Intel vtd +device model. +"intremap" - Enable vIOMMU interrupt remapping function. +"x2apic" - Support x2apic mode with interrupt remapping function.