From patchwork Fri Sep 22 03:01:42 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "lan,Tianyu" X-Patchwork-Id: 9965589 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id B1DAC600C5 for ; Fri, 22 Sep 2017 09:12:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ACB8B28501 for ; Fri, 22 Sep 2017 09:12:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A1A2F2885F; Fri, 22 Sep 2017 09:12:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00, DATE_IN_PAST_06_12, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 2F4B528501 for ; Fri, 22 Sep 2017 09:12:18 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dvJxn-0003Ad-AY; Fri, 22 Sep 2017 09:09:35 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dvJxl-0003AR-Nm for xen-devel@lists.xen.org; Fri, 22 Sep 2017 09:09:33 +0000 Received: from [85.158.139.211] by server-17.bemta-5.messagelabs.com id C0/DE-25201-C43D4C95; Fri, 22 Sep 2017 09:09:32 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmplkeJIrShJLcpLzFFi42Jpa+uQ0PW5fCT S4PwsPoslHxezODB6HN39mymAMYo1My8pvyKBNePZG/WC6XoVm/9fYm1g/C7XxcjFISQwnVFi 4tfFLF2MnBwSArwSR5bNYIWwAyRWb33NBmILCXQwSpzY4A1iswmoS5xYPJERxBYRkJa49vkyI 8ggZoEVTBIrHy8BSwgLWEtsbJ7LBGKzCKhKbOs7zNzFyMHBK+Ai8emRLcR8BYkpD98zg9icAq 4Sv/q2M0HscpFoXbqfaQIj7wJGhlWMGsWpRWWpRbpGZnpJRZnpGSW5iZk5uoYGpnq5qcXFiem pOYlJxXrJ+bmbGIHBUM/AwLiD8fZkv0OMkhxMSqK8788fiRTiS8pPqcxILM6ILyrNSS0+xCjD waEkwSt6CSgnWJSanlqRlpkDDEuYtAQHj5II76uLQGne4oLE3OLMdIjUKUZLjo6bd/8wcWwCk xu+P/jDJMSSl5+XKiXOqwcyTwCkIaM0D24cLHYuMcpKCfMyMjAwCPEUpBblZpagyr9iFOdgVB LmNQCZwpOZVwK39RXQQUxAB5WvBjuoJBEhJdXA6H711T0d39ab/VNfC1UeaJgTfz3FTC75ve4 pEYa1ga+lG3o2n3nGK37/m/40/yuBaxJCPYUri5Ir7Xdc6qp9U+HE5TVX6+7xf3vdZyfG+i62 fjqVk4v/elCEXdxRu8mPBYRVMpZP8S3P6zmq+OmGp3SER7/T+4r18XH+FawPF30smMXT+EZFi aU4I9FQi7moOBEA/JuMRZgCAAA= X-Env-Sender: tianyu.lan@intel.com X-Msg-Ref: server-2.tower-206.messagelabs.com!1506071370!92157135!1 X-Originating-IP: [134.134.136.24] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTM0LjEzNC4xMzYuMjQgPT4gMzkwOTcx\n X-StarScan-Received: X-StarScan-Version: 9.4.45; banners=-,-,- X-VirusChecked: Checked Received: (qmail 6773 invoked from network); 22 Sep 2017 09:09:31 -0000 Received: from mga09.intel.com (HELO mga09.intel.com) (134.134.136.24) by server-2.tower-206.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 22 Sep 2017 09:09:31 -0000 Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Sep 2017 02:09:29 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos; i="5.42,427,1500966000"; d="scan'208"; a="1222293249" Received: from sky-ws.sh.intel.com (HELO localhost) ([10.239.48.141]) by fmsmga002.fm.intel.com with ESMTP; 22 Sep 2017 02:09:26 -0700 From: Lan Tianyu To: xen-devel@lists.xen.org Date: Thu, 21 Sep 2017 23:01:42 -0400 Message-Id: <1506049330-11196-2-git-send-email-tianyu.lan@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1506049330-11196-1-git-send-email-tianyu.lan@intel.com> References: <1506049330-11196-1-git-send-email-tianyu.lan@intel.com> Cc: Lan Tianyu , kevin.tian@intel.com, sstabellini@kernel.org, wei.liu2@citrix.com, George.Dunlap@eu.citrix.com, andrew.cooper3@citrix.com, ian.jackson@eu.citrix.com, tim@xen.org, jbeulich@suse.com, roger.pau@citrix.com, chao.gao@intel.com Subject: [Xen-devel] [PATCH V3 1/29] Xen/doc: Add Xen virtual IOMMU doc X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP This patch is to add Xen virtual IOMMU doc to introduce motivation, framework, vIOMMU hypercall and xl configuration. Signed-off-by: Lan Tianyu --- docs/misc/viommu.txt | 136 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 136 insertions(+) create mode 100644 docs/misc/viommu.txt diff --git a/docs/misc/viommu.txt b/docs/misc/viommu.txt new file mode 100644 index 0000000..348e8c4 --- /dev/null +++ b/docs/misc/viommu.txt @@ -0,0 +1,136 @@ +Xen virtual IOMMU + +Motivation +========== +Enable more than 128 vcpu support + +The current requirements of HPC cloud service requires VM with a high +number of CPUs in order to achieve high performance in parallel +computing. + +To support >128 vcpus, X2APIC mode in guest is necessary because legacy +APIC(XAPIC) just supports 8-bit APIC ID. The APIC ID used by Xen is +CPU ID * 2 (ie: CPU 127 has APIC ID 254, which is the last one available +in xAPIC mode) and so it only can support 128 vcpus at most. x2APIC mode +supports 32-bit APIC ID and it requires the interrupt remapping functionality +of a vIOMMU if the guest wishes to route interrupts to all available vCPUs + +The reason for this is that there is no modification for existing PCI MSI +and IOAPIC when introduce X2APIC. PCI MSI/IOAPIC can only send interrupt +message containing 8-bit APIC ID, which cannot address cpus with >254 +APIC ID. Interrupt remapping supports 32-bit APIC ID and so it's necessary +for >128 vcpus support. + + +vIOMMU Architecture +=================== +vIOMMU device model is inside Xen hypervisor for following factors + 1) Avoid round trips between Qemu and Xen hypervisor + 2) Ease of integration with the rest of hypervisor + 3) HVMlite/PVH doesn't use Qemu + +* Interrupt remapping overview. +Interrupts from virtual devices and physical devices are delivered +to vLAPIC from vIOAPIC and vMSI. vIOMMU needs to remap interrupt during +this procedure. + ++---------------------------------------------------+ +|Qemu |VM | +| | +----------------+ | +| | | Device driver | | +| | +--------+-------+ | +| | ^ | +| +----------------+ | +--------+-------+ | +| | Virtual device | | | IRQ subsystem | | +| +-------+--------+ | +--------+-------+ | +| | | ^ | +| | | | | ++---------------------------+-----------------------+ +|hypervisor | | VIRQ | +| | +---------+--------+ | +| | | vLAPIC | | +| |VIRQ +---------+--------+ | +| | ^ | +| | | | +| | +---------+--------+ | +| | | vIOMMU | | +| | +---------+--------+ | +| | ^ | +| | | | +| | +---------+--------+ | +| | | vIOAPIC/vMSI | | +| | +----+----+--------+ | +| | ^ ^ | +| +-----------------+ | | +| | | ++---------------------------------------------------+ +HW |IRQ + +-------------------+ + | PCI Device | + +-------------------+ + + +vIOMMU hypercall +================ +Introduce a new domctl hypercall "xen_domctl_viommu_op" to create/destroy +vIOMMUs. + +* vIOMMU hypercall parameter structure + +/* vIOMMU type - specify vendor vIOMMU device model */ +#define VIOMMU_TYPE_INTEL_VTD 0 + +/* vIOMMU capabilities */ +#define VIOMMU_CAP_IRQ_REMAPPING (1u << 0) + +struct xen_domctl_viommu_op { + uint32_t cmd; +#define XEN_DOMCTL_create_viommu 0 +#define XEN_DOMCTL_destroy_viommu 1 + union { + struct { + /* IN - vIOMMU type */ + uint64_t viommu_type; + /* IN - MMIO base address of vIOMMU. */ + uint64_t base_address; + /* IN - Capabilities with which we want to create */ + uint64_t capabilities; + /* OUT - vIOMMU identity */ + uint32_t viommu_id; + } create_viommu; + + struct { + /* IN - vIOMMU identity */ + uint32_t viommu_id; + } destroy_viommu; + } u; +}; + +- XEN_DOMCTL_create_viommu + Create vIOMMU device with vIOMMU_type, capabilities and MMIO base +address. Hypervisor allocates viommu_id for new vIOMMU instance and return +back. The vIOMMU device model in hypervisor should check whether it can +support the input capabilities and return error if not. + +- XEN_DOMCTL_destroy_viommu + Destroy vIOMMU in Xen hypervisor with viommu_id as parameter. + +These vIOMMU domctl and vIOMMU option in configure file consider multi-vIOMMU +support for single VM.(e.g, parameters of create/destroy vIOMMU includes +vIOMMU id). But function implementation only supports one vIOMMU per VM so far. + +Xen hypervisor vIOMMU command +============================= +Introduce vIOMMU command "viommu=1" to enable vIOMMU function in hypervisor. +It's default disabled. + +xl x86 vIOMMU configuration" +============================ +viommu = [ + 'type=intel_vtd,intremap=1', + ... +] + +"type" - Specify vIOMMU device model type. Currently only supports Intel vtd +device model. +"intremap" - Enable vIOMMU interrupt remapping function.