From patchwork Fri Nov 17 06:22:08 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chao Gao X-Patchwork-Id: 10062337 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 45C866023A for ; Fri, 17 Nov 2017 06:27:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 37BC12A539 for ; Fri, 17 Nov 2017 06:27:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2C2462A997; Fri, 17 Nov 2017 06:27:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id B76822A539 for ; Fri, 17 Nov 2017 06:27:28 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1eFa4o-0000OM-Ox; Fri, 17 Nov 2017 06:24:34 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1eFa4n-0000O1-J8 for xen-devel@lists.xen.org; Fri, 17 Nov 2017 06:24:33 +0000 Received: from [85.158.143.35] by server-1.bemta-6.messagelabs.com id A2/05-04165-0A08E0A5; Fri, 17 Nov 2017 06:24:32 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFupmkeJIrShJLcpLzFFi42I5YG4Sq7uggS/ K4MBKPoslHxezODB6HN39mymAMYo1My8pvyKBNaNh91X2gpdaFVPn6jQwrpXpYuTiEBKYzigx 4+AW9i5GTg4JAV6JI8tmsELYARLfGvoZQWwhgSqJJ+2fmUBsNgFliYtfe9lAbBEBaYlrny8zg gxiFpjKIvFm9SdmkISwgI3EtuYusKEsAqoSC6e1g9m8Ak4Se4/8ZIFYoCAx5eF7sHpOAWeJ/W 8OMkEsc5I4uG4a0wRG3gWMDKsY1YtTi8pSi3Qt9ZKKMtMzSnITM3N0DQ3M9HJTi4sT01NzEpO K9ZLzczcxAoOBAQh2MN7dFHCIUZKDSUmUl8ecL0qILyk/pTIjsTgjvqg0J7X4EKMMB4eSBK9X PVBOsCg1PbUiLTMHGJYwaQkOHiUR3gKQNG9xQWJucWY6ROoUoz3Hhu8P/jBxPJv5uoGZY9rV1 iZmjnnHvzUxC7Hk5eelSonzFoO0CYC0ZZTmwQ2FxdElRlkpYV5GoDOFeApSi3IzS1DlXzGKcz AqCfNWgkzhycwrgdv9CugsJqCzbG5wg5xVkoiQkmpgFIoSmX//8sWDz2pTMx+x5wvEVrkmvfU qPJv+Lf/FtmUnGnepPcvxP3JYbKG/fU3vwpKKFBPD0wyyITv0Z6+QfLKQZf4Nz9jLK3r2qByY e+7g8Z3nvigtYXUyviwutyxJLtEr1nNWstuC2i22+as1WXaLXpHJOLekX0rZoPLCrYsrjhpZ2 a00VGIpzkg01GIuKk4EALXZH3OeAgAA X-Env-Sender: chao.gao@intel.com X-Msg-Ref: server-12.tower-21.messagelabs.com!1510899867!82378134!2 X-Originating-IP: [192.55.52.93] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTkyLjU1LjUyLjkzID0+IDMyNDY2NQ==\n X-StarScan-Received: X-StarScan-Version: 9.4.45; banners=-,-,- X-VirusChecked: Checked Received: (qmail 54263 invoked from network); 17 Nov 2017 06:24:31 -0000 Received: from mga11.intel.com (HELO mga11.intel.com) (192.55.52.93) by server-12.tower-21.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 17 Nov 2017 06:24:31 -0000 Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Nov 2017 22:24:31 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos; i="5.44,407,1505804400"; d="scan'208"; a="1245165011" Received: from skl-4s-chao.sh.intel.com ([10.239.48.9]) by fmsmga002.fm.intel.com with ESMTP; 16 Nov 2017 22:24:28 -0800 From: Chao Gao To: xen-devel@lists.xen.org Date: Fri, 17 Nov 2017 14:22:08 +0800 Message-Id: <1510899755-40237-2-git-send-email-chao.gao@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1510899755-40237-1-git-send-email-chao.gao@intel.com> References: <1510899755-40237-1-git-send-email-chao.gao@intel.com> Cc: Lan Tianyu , Kevin Tian , Stefano Stabellini , Wei Liu , Konrad Rzeszutek Wilk , George Dunlap , Ian Jackson , Tim Deegan , Jan Beulich , Andrew Cooper , Chao Gao , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= Subject: [Xen-devel] [PATCH v4 01/28] Xen/doc: Add Xen virtual IOMMU doc X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Lan Tianyu This patch is to add Xen virtual IOMMU doc to introduce motivation, framework, vIOMMU hypercall and xl configuration. Signed-off-by: Lan Tianyu Signed-off-by: Chao Gao --- docs/misc/viommu.txt | 120 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 120 insertions(+) create mode 100644 docs/misc/viommu.txt diff --git a/docs/misc/viommu.txt b/docs/misc/viommu.txt new file mode 100644 index 0000000..472d2b5 --- /dev/null +++ b/docs/misc/viommu.txt @@ -0,0 +1,120 @@ +Xen virtual IOMMU + +Motivation +========== +Enable more than 128 vcpu support + +The current requirements of HPC cloud service requires VM with a high +number of CPUs in order to achieve high performance in parallel +computing. + +To support >128 vcpus, X2APIC mode in guest is necessary because legacy +APIC(XAPIC) just supports 8-bit APIC ID. The APIC ID used by Xen is +CPU ID * 2 (ie: CPU 127 has APIC ID 254, which is the last one available +in xAPIC mode) and so it only can support 128 vcpus at most. x2APIC mode +supports 32-bit APIC ID and it requires the interrupt remapping functionality +of a vIOMMU if the guest wishes to route interrupts to all available vCPUs + +PCI MSI/IOAPIC can only send interrupt message containing 8-bit APIC ID, +which cannot address cpus with >254 APIC ID. Interrupt remapping supports +32-bit APIC ID and so it's necessary for >128 vcpus support. + +vIOMMU Architecture +=================== +vIOMMU device model is inside Xen hypervisor for following factors + 1) Avoid round trips between Qemu and Xen hypervisor + 2) Ease of integration with the rest of hypervisor + 3) PVH doesn't use Qemu + +* Interrupt remapping overview. +Interrupts from virtual devices and physical devices are delivered +to vLAPIC from vIOAPIC and vMSI. vIOMMU needs to remap interrupt during +this procedure. + ++---------------------------------------------------+ +|Qemu |VM | +| | +----------------+ | +| | | Device driver | | +| | +--------+-------+ | +| | ^ | +| +----------------+ | +--------+-------+ | +| | Virtual device | | | IRQ subsystem | | +| +-------+--------+ | +--------+-------+ | +| | | ^ | +| | | | | ++---------------------------+-----------------------+ +|hypervisor | | VIRQ | +| | +---------+--------+ | +| | | vLAPIC | | +| |VIRQ +---------+--------+ | +| | ^ | +| | | | +| | +---------+--------+ | +| | | vIOMMU | | +| | +---------+--------+ | +| | ^ | +| | | | +| | +---------+--------+ | +| | | vIOAPIC/vMSI | | +| | +----+----+--------+ | +| | ^ ^ | +| +-----------------+ | | +| | | ++---------------------------------------------------+ +HW |IRQ + +-------------------+ + | PCI Device | + +-------------------+ + + +vIOMMU hypercall +================ +Introduce a new domctl hypercall "xen_domctl_viommu_op" to create +vIOMMUs instance in hypervisor. vIOMMU instance will be destroyed +during destroying domain. + +* vIOMMU hypercall parameter structure + +/* vIOMMU type - specify vendor vIOMMU device model */ +#define VIOMMU_TYPE_INTEL_VTD 0 + +/* vIOMMU capabilities */ +#define VIOMMU_CAP_IRQ_REMAPPING (1u << 0) + +struct xen_domctl_viommu_op { + uint32_t cmd; +#define XEN_DOMCTL_viommu_create 0 + union { + struct { + /* IN - vIOMMU type */ + uint8_t type; + /* IN - MMIO base address of vIOMMU. */ + uint64_t base_address; + /* IN - Capabilities with which we want to create */ + uint64_t capabilities; + /* OUT - vIOMMU identity */ + uint32_t id; + } create; + } u; +}; + +- XEN_DOMCTL_create_viommu + Create vIOMMU device with type, capabilities and MMIO base address. +Hypervisor allocates viommu_id for new vIOMMU instance and return back. +The vIOMMU device model in hypervisor should check whether it can +support the input capabilities and return error if not. + +vIOMMU domctl and vIOMMU option in configure file consider multi-vIOMMU +support for single VM.(e.g, parameters of create vIOMMU includes vIOMMU id). +But function implementation only supports one vIOMMU per VM so far. + +xl x86 vIOMMU configuration" +============================ +viommu = [ + 'type=intel_vtd,intremap=1', + ... +] + +"type" - Specify vIOMMU device model type. Currently only supports Intel vtd +device model. +"intremap" - Enable vIOMMU interrupt remapping function.