From patchwork Tue Jan 10 22:57:36 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Venu Busireddy X-Patchwork-Id: 9509267 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 998B560710 for ; Wed, 11 Jan 2017 04:55:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 89FFB28571 for ; Wed, 11 Jan 2017 04:55:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7EA8528596; Wed, 11 Jan 2017 04:55:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id C484428571 for ; Wed, 11 Jan 2017 04:55:47 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cRAuL-0008F9-3O; Wed, 11 Jan 2017 04:53:09 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cR5Nk-0007ks-W1 for xen-devel@lists.xen.org; Tue, 10 Jan 2017 22:59:09 +0000 Received: from [85.158.139.211] by server-9.bemta-5.messagelabs.com id 37/BC-06369-C3765785; Tue, 10 Jan 2017 22:59:08 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFupjkeJIrShJLcpLzFFi42KZM10+UNc6vTT CYPckEYslHxezODB6HN39mymAMYo1My8pvyKBNWNSx0G2guPBFV9WH2drYFxo28XIxSEkMJlJ 4v6jW8xdjJxAzm9GiTvzUiASGxkl9lzYwwzhdDJKnHl6kAmkik3AQOLo4R5WEFtEwFdi76WzL CBFzAIzGSW2tF9i62Lk4BAWCJBovxkAUsMioCrx9/ArsHpeAQ+JDUdWsYPYEgJyEjfPdYJt5h TwlPi3cSI7xBUeEs1PWpggaowl2t9eZJvAyLeAkWEVo0ZxalFZapGukaleUlFmekZJbmJmjq6 hgalebmpxcWJ6ak5iUrFecn7uJkZgqNQzMDDuYNzV7neIUZKDSUmU18q2NEKILyk/pTIjsTgj vqg0J7X4EKMMB4eSBG9bKlBOsCg1PbUiLTMHGLQwaQkOHiUR3v8pQGne4oLE3OLMdIjUKUZjj mnPFj9l4lh3aslTJiGWvPy8VClx3i6QSQIgpRmleXCDYNF0iVFWSpiXkYGBQYinILUoN7MEVf 4VozgHo5IwbzvIFJ7MvBK4fa+ATmECOiXSrhjklJJEhJRUA6PzI4F2I31GmWDZxBl5p7WYapZ k6cu5LYzvUJ3ydvGqydZzXm58za12Ma+Et2Gtw8SMtD8/Zu6XYTyhzHym7cPPW3qf79eICMn1 iYfyOORvqbjn6xSYYDvx8bnpR2z/ufKf61lW32wc9vj7rp+qokt2TtpXblUvWs4/mT2lquRH3 L25MzXbTyixFGckGmoxFxUnAgDezfgYoQIAAA== X-Env-Sender: venu.busireddy@oracle.com X-Msg-Ref: server-3.tower-206.messagelabs.com!1484089145!75668573!1 X-Originating-IP: [156.151.31.81] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTU2LjE1MS4zMS44MSA9PiAyODgzMzk=\n X-StarScan-Received: X-StarScan-Version: 9.1.1; banners=-,-,- X-VirusChecked: Checked Received: (qmail 5442 invoked from network); 10 Jan 2017 22:59:06 -0000 Received: from userp1040.oracle.com (HELO userp1040.oracle.com) (156.151.31.81) by server-3.tower-206.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 10 Jan 2017 22:59:06 -0000 Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id v0AMx2lm013123 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 10 Jan 2017 22:59:03 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id v0AMx244007773 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 10 Jan 2017 22:59:02 GMT Received: from abhmp0012.oracle.com (abhmp0012.oracle.com [141.146.116.18]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id v0AMx2GI027774; Tue, 10 Jan 2017 22:59:02 GMT Received: from lenovo.localdomain (/24.55.20.143) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 10 Jan 2017 14:59:01 -0800 From: Venu Busireddy To: venu.busireddy@oracle.com, Konrad Rzeszutek Wilk Date: Tue, 10 Jan 2017 16:57:36 -0600 Message-Id: <1484089056-8762-4-git-send-email-venu.busireddy@oracle.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1484089056-8762-1-git-send-email-venu.busireddy@oracle.com> References: <1484089056-8762-1-git-send-email-venu.busireddy@oracle.com> X-Source-IP: aserv0022.oracle.com [141.146.126.234] X-Mailman-Approved-At: Wed, 11 Jan 2017 04:53:07 +0000 Cc: Elena Ufimtseva , Kevin Tian , Feng Wu , xen-devel@lists.xen.org Subject: [Xen-devel] [PATCH v13 3/3] iommu: add rmrr Xen command line option for extra rmrrs X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Elena Ufimtseva On some platforms firmware fails to specify RMRR regions in ACPI tables and thus those regions will not be mapped in dom0 or guests and may cause IO Page Faults and prevent dom0 from booting if "iommu=dom0-strict" option is specified on the Xen command line. New Xen command line option rmrr allows to specify such devices and memory regions. These regions are added to the list of RMRR defined in ACPI if the device is present in system. As a result, additional RMRRs will be mapped 1:1 in dom0 with correct permissions. The above mentioned problems were discovered during the PVH work with ThinkCentre M and Dell 5600T. No official documentation was found so far in regards to what devices and why cause this. Experiments show that ThinkCentre M USB devices with enabled debug port generate DMA read transactions to the regions of memory marked reserved in host e820 map. For Dell 5600T the device and faulting addresses are not found yet. For detailed history of the discussion please check following threads: http://lists.Xen.org/archives/html/xen-devel/2015-02/msg01724.html http://lists.Xen.org/archives/html/xen-devel/2015-01/msg02513.html Format for rmrr Xen command line option: rmrr=start<-end>=[s1]bdf1[,[s1]bdf2[,...]];start<-end>=[s2]bdf1[,[s2]bdf2[,...]] For example, for Lenovo ThinkCentre M, use: rmrr=0xd5d45=0:0:1d.0;0xd5d46=0:0:1a.0 If grub2 used and multiple ranges are specified, ';' should be quoted/escaped, refer to grub2 manual for more information. Signed-off-by: Elena Ufimtseva Signed-off-by: Venu Busireddy --- docs/misc/xen-command-line.markdown | 13 +++ xen/drivers/passthrough/vtd/dmar.c | 201 +++++++++++++++++++++++++++++++++++- 2 files changed, 213 insertions(+), 1 deletion(-) Changes in v13: - Implement feedback from Kevin Tian. https://lists.xenproject.org/archives/html/xen-devel/2015-10/msg03169.html https://lists.xenproject.org/archives/html/xen-devel/2015-10/msg03170.html https://lists.xenproject.org/archives/html/xen-devel/2015-10/msg03171.html - Limit all source lines and comments to 80 characters per line. - Implement coding style suggestions from Konrad Wilk. diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown index 0138978..a11fdf9 100644 --- a/docs/misc/xen-command-line.markdown +++ b/docs/misc/xen-command-line.markdown @@ -1377,6 +1377,19 @@ Specify the host reboot method. 'efi' instructs Xen to reboot using the EFI reboot call (in EFI mode by default it will use that method first). +### rmrr +> '= start<-end>=[s1]bdf1[,[s1]bdf2[,...]];start<-end>=[s2]bdf1[,[s2]bdf2[,...]] + +Define RMRR units that are missing from ACPI table along with device they +belong to and use them for 1:1 mapping. End addresses can be omitted and one +page will be mapped. The ranges are inclusive when start and end are specified. +If segment of the first device is not specified, segment zero will be used. +If other segments are not specified, first device segment will be used. +If a segment is specified for other than the first device and it does not match +the one specified for the first one, an error will be reported. +Note: grub2 requires to escape or use quotations if special characters are used, +namely ';', refer to the grub2 documentation if multiple ranges are specified. + ### ro-hpet > `= ` diff --git a/xen/drivers/passthrough/vtd/dmar.c b/xen/drivers/passthrough/vtd/dmar.c index 84bf63d..23ba7a6 100644 --- a/xen/drivers/passthrough/vtd/dmar.c +++ b/xen/drivers/passthrough/vtd/dmar.c @@ -859,6 +859,132 @@ out: return ret; } +#define MAX_EXTRA_RMRR_PAGES 16 +#define MAX_EXTRA_RMRR 10 + +/* RMRR units derived from command line rmrr option. */ +#define MAX_EXTRA_RMRR_DEV 20 +struct user_rmrr { + struct list_head list; + unsigned long base_pfn, end_pfn; + unsigned int dev_count; + u32 sbdf[MAX_EXTRA_RMRR_DEV]; +}; + +static __initdata unsigned int nr_rmrr; +static struct __initdata user_rmrr user_rmrrs[MAX_EXTRA_RMRR]; + +/* Macro for RMRR inclusive range formatting. */ +#define ERMRRU_FMT "[%lx-%lx]" +#define ERMRRU_ARG(eru) eru.base_pfn, eru.end_pfn + +static int __init add_user_rmrr(void) +{ + struct acpi_rmrr_unit *rmrr, *rmrru; + unsigned int idx, seg, i; + unsigned long base, end; + bool overlap; + + for ( i = 0; i < nr_rmrr; i++ ) + { + base = user_rmrrs[i].base_pfn; + end = user_rmrrs[i].end_pfn; + + if ( base > end ) + { + printk(XENLOG_ERR VTDPREFIX + "Invalid RMRR Range "ERMRRU_FMT"\n", + ERMRRU_ARG(user_rmrrs[i])); + continue; + } + + if ( (end - base) >= MAX_EXTRA_RMRR_PAGES ) + { + printk(XENLOG_ERR VTDPREFIX + "RMRR range "ERMRRU_FMT" exceeds "\ + __stringify(MAX_EXTRA_RMRR_PAGES)" pages\n", + ERMRRU_ARG(user_rmrrs[i])); + continue; + } + + overlap = false; + list_for_each_entry(rmrru, &acpi_rmrr_units, list) + { + if ( pfn_to_paddr(base) < rmrru->end_address && + rmrru->base_address < pfn_to_paddr(end + 1) ) + { + printk(XENLOG_ERR VTDPREFIX + "Overlapping RMRRs: "ERMRRU_FMT" and [%lx-%lx]\n", + ERMRRU_ARG(user_rmrrs[i]), + paddr_to_pfn(rmrru->base_address), + paddr_to_pfn(rmrru->end_address)); + overlap = true; + break; + } + } + /* Don't add overlapping RMRR. */ + if ( overlap ) + continue; + + do + { + if ( !mfn_valid(base) ) + { + printk(XENLOG_ERR VTDPREFIX + "Invalid pfn in RMRR range "ERMRRU_FMT"\n", + ERMRRU_ARG(user_rmrrs[i])); + break; + } + } while ( base++ < end ); + + /* Invalid pfn in range as the loop ended before end_pfn was reached. */ + if ( base <= end ) + continue; + + rmrr = xzalloc(struct acpi_rmrr_unit); + if ( !rmrr ) + return -ENOMEM; + + rmrr->scope.devices = xmalloc_array(u16, user_rmrrs[i].dev_count); + if ( !rmrr->scope.devices ) + { + xfree(rmrr); + return -ENOMEM; + } + + seg = 0; + for ( idx = 0; idx < user_rmrrs[i].dev_count; idx++ ) + { + rmrr->scope.devices[idx] = user_rmrrs[i].sbdf[idx]; + seg |= PCI_SEG(user_rmrrs[i].sbdf[idx]); + } + if ( seg != PCI_SEG(user_rmrrs[i].sbdf[0]) ) + { + printk(XENLOG_ERR VTDPREFIX + "Segments are not equal for RMRR range "ERMRRU_FMT"\n", + ERMRRU_ARG(user_rmrrs[i])); + scope_devices_free(&rmrr->scope); + xfree(rmrr); + continue; + } + + rmrr->segment = seg; + rmrr->base_address = pfn_to_paddr(user_rmrrs[i].base_pfn); + rmrr->end_address = pfn_to_paddr(user_rmrrs[i].end_pfn + 1); + rmrr->scope.devices_cnt = user_rmrrs[i].dev_count; + + if ( register_one_rmrr(rmrr) ) + { + printk(XENLOG_ERR VTDPREFIX + "Could not register RMMR range "ERMRRU_FMT"\n", + ERMRRU_ARG(user_rmrrs[i])); + scope_devices_free(&rmrr->scope); + xfree(rmrr); + } + } + return 0; +} + #include /* ACPI tables may not be DMA protected by tboot, so use DMAR copy */ /* SINIT saved in SinitMleData in TXT heap (which is DMA protected) */ @@ -868,6 +994,7 @@ int __init acpi_dmar_init(void) { acpi_physical_address dmar_addr; acpi_native_uint dmar_len; + int ret; if ( ACPI_SUCCESS(acpi_get_table_phys(ACPI_SIG_DMAR, 0, &dmar_addr, &dmar_len)) ) @@ -878,7 +1005,12 @@ int __init acpi_dmar_init(void) dmar_table = __va(dmar_addr); } - return parse_dmar_table(acpi_parse_dmar); + ret = parse_dmar_table(acpi_parse_dmar); + + if ( !ret ) + return add_user_rmrr(); + + return ret; } void acpi_dmar_reinstate(void) @@ -937,3 +1069,70 @@ int intel_iommu_get_reserved_device_memory(iommu_grdm_t *func, void *ctxt) return 0; } + +/* + * Parse rmrr Xen command line options and add parsed devices and regions into + * acpi_rmrr_unit list to mapped as RMRRs parsed from ACPI. + * Format: + * rmrr=start<-end>=[s1]bdf1[,[s1]bdf2[,...]];start<-end>=[s2]bdf1[,[s2]bdf2[,...]] + * If the segment of the first device is not specified, + * segment zero will be used. + * If other segments are not specified, first device segment will be used. + * If a segment is specified for other than the first device, and it does not + * match the one specified for the first one, an error will be reported. + */ +static void __init parse_rmrr_param(const char *str) +{ + const char *s = str, *cur, *stmp; + unsigned int seg, bus, dev, func, dev_count; + unsigned long start, end; + + do { + start = simple_strtoul(cur = s, &s, 0); + if ( cur == s ) + break; + + if ( *s == '-' ) + { + end = simple_strtoul(cur = s + 1, &s, 0); + if ( cur == s ) + break; + } + else + end = start; + + user_rmrrs[nr_rmrr].base_pfn = start; + user_rmrrs[nr_rmrr].end_pfn = end; + + if ( *s != '=' ) + continue; + + do { + bool def_seg = false; + + stmp = parse_pci_seg(s + 1, &seg, &bus, &dev, &func, &def_seg); + if ( !stmp ) + break; + + /* + * Not specified. + * Segment will be replaced with one from first device. + */ + if ( user_rmrrs[nr_rmrr].dev_count && def_seg ) + seg = PCI_SEG(user_rmrrs[nr_rmrr].sbdf[0]); + + /* Keep sbdf's even if they differ and later report an error. */ + dev_count = user_rmrrs[nr_rmrr].dev_count; + user_rmrrs[nr_rmrr].sbdf[dev_count] = PCI_SBDF(seg, bus, dev, func); + + user_rmrrs[nr_rmrr].dev_count++; + s = stmp; + } while ( *s == ',' && + user_rmrrs[nr_rmrr].dev_count < MAX_EXTRA_RMRR_DEV ); + + if ( user_rmrrs[nr_rmrr].dev_count ) + nr_rmrr++; + + } while ( *s++ == ';' && nr_rmrr < MAX_EXTRA_RMRR ); +} +custom_param("rmrr", parse_rmrr_param);