From patchwork Tue Jan 10 15:02:56 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Razvan Cojocaru X-Patchwork-Id: 9507833 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 0840C606E1 for ; Tue, 10 Jan 2017 15:03:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EEBF92859B for ; Tue, 10 Jan 2017 15:03:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E36382859F; Tue, 10 Jan 2017 15:03:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 506272859B for ; Tue, 10 Jan 2017 15:03:53 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cQxvj-0007ES-Cd; Tue, 10 Jan 2017 15:01:43 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cQxvi-0007EM-LF for xen-devel@lists.xenproject.org; Tue, 10 Jan 2017 15:01:42 +0000 Received: from [85.158.139.211] by server-17.bemta-5.messagelabs.com id AC/B8-12366-557F4785; Tue, 10 Jan 2017 15:01:41 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrHIsWRWlGSWpSXmKPExsUSfTxjoW7o95I Ig0/nJS2+b5nM5MDocfjDFZYAxijWzLyk/IoE1owFE34yFszXrPj6YiFzA+MfuS5GTg4hAQ+J ae83sHcxcgHZaxklur92sUE4dxgl3j5ewQJTdejTTKiqRYwSMzsPMoMkhAUSJI7P/Q5WJCIQJ fFu9UUWiKIHjBLfmg+wgSSYBTQl2t6/ZgWx2QQMJVZvbAGKc3DwCjhJvP5qDhJmEVCV+PxvAl i5qEC4RMeua+wgNq+AoMTJmU9YQMo5Bewk1vayQExUl/gz7xIzhC0vsf3tHGaQEgmBHIl/R9I gTCmJ/61KIMdICKxnkdh3sgusVUJARuLRxJtsExhFZyFZMAvJ1FlIpi5gZF7FqFGcWlSWWqRr aK6XVJSZnlGSm5iZo2toYKqXm1pcnJiempOYVKyXnJ+7iREYFwxAsIPx4mnPQ4ySHExKorzLP pdECPEl5adUZiQWZ8QXleakFh9ilOHgUJLgffUVKCdYlJqeWpGWmQOMUJi0BAePkgjvKpA0b3 FBYm5xZjpE6hSjLse0Z4ufMgmx5OXnpUqJ834AKRIAKcoozYMbAUsWlxhlpYR5GYGOEuIpSC3 KzSxBlX/FKM7BqCTMq/kNaApPZl4J3KZXQEcwAR0RaVcMckRJIkJKqoHxVO61unenuwoW3VNZ Xy9UYn7zasGR8zs1vDjd7PMuaO/T7HZ/qLTlq+efOf25flov7FP7FwouiryavLUljDs94+7k3 0UCQReLP2is8Io/fSa8vfxR9BbDLadOW/LM2sSy2nnl5N9nL2drZi5ate75VcMfZctEe0o2nC r8yLfoHu/jh8tqOxZzKLEUZyQaajEXFScCANzfTo0RAwAA X-Env-Sender: rcojocaru@bitdefender.com X-Msg-Ref: server-6.tower-206.messagelabs.com!1484060499!79009251!1 X-Originating-IP: [91.199.104.161] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 9.1.1; banners=-,-,- X-VirusChecked: Checked Received: (qmail 53756 invoked from network); 10 Jan 2017 15:01:40 -0000 Received: from mx01.bbu.dsd.mx.bitdefender.com (HELO mx01.bbu.dsd.mx.bitdefender.com) (91.199.104.161) by server-6.tower-206.messagelabs.com with DHE-RSA-AES128-GCM-SHA256 encrypted SMTP; 10 Jan 2017 15:01:40 -0000 Received: (qmail 26310 invoked from network); 10 Jan 2017 17:01:39 +0200 Received: from unknown (HELO mx-sr.buh.bitdefender.com) (10.17.80.103) by mx01.bbu.dsd.mx.bitdefender.com with AES256-GCM-SHA384 encrypted SMTP; 10 Jan 2017 17:01:39 +0200 Received: from smtp01.buh.bitdefender.com (smtp.bitdefender.biz [10.17.80.75]) by mx-sr.buh.bitdefender.com (Postfix) with ESMTP id E62567FC40 for ; Tue, 10 Jan 2017 17:01:38 +0200 (EET) Received: (qmail 12943 invoked from network); 10 Jan 2017 17:01:38 +0200 Received: from rcojocaru.dsd.ro (HELO ?10.10.14.59?) (rcojocaru@bitdefender.com@10.10.14.59) by smtp01.buh.bitdefender.com with SMTP; 10 Jan 2017 17:01:38 +0200 To: Andrew Cooper , xen-devel References: <790e8423-00b5-79ac-f0e4-f0ec92d79c32@bitdefender.com> <58fb1e1a-0fdd-f31a-77e4-e136fa0931de@citrix.com> <661a3921-e18f-8236-dde3-93d9327a657a@bitdefender.com> From: Razvan Cojocaru Message-ID: <25413f94-8183-e497-5d56-20ef794f5a41@bitdefender.com> Date: Tue, 10 Jan 2017 17:02:56 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.6.0 MIME-Version: 1.0 In-Reply-To: X-BitDefender-Scanner: Clean, Agent: BitDefender qmail 3.1.6 on smtp01.buh.bitdefender.com, sigver: 7.69043 X-BitDefender-Spam: No (0) X-BitDefender-SpamStamp: Build: [Engines: 2.15.8.1074, Dats: 438278, Stamp: 3], Multi: [Enabled, t: (0.000012, 0.023606)], BW: [Enabled, t: (0.000030,0.000001)], RBL DNSBL: [Disabled], APM: [Enabled, Score: 500, t: (0.007559), Flags: 85D2ED72; NN_LEGIT_VALID_REPLY; NN_LEGIT_SUMM_400_WORDS; NN_NO_LINK_NMD; NN_LEGIT_BITDEFENDER; NN_LEGIT_S_SQARE_BRACKETS; NN_LEGIT_MAILING_LIST_TO], SGN: [Enabled, t: (0.008989,0.000324)], URL: [Enabled, t: (0.000005)], RTDA: [Enabled, t: (0.035584), Hit: No, Details: v2.4.3; Id: 5eu5o9.1b61q342j.hbsv], total: 0(775) X-BitDefender-CF-Stamp: none Cc: Tamas K Lengyel Subject: Re: [Xen-devel] Ubuntu 16.04.1 LTS kernel 4.4.0-57 over-allocation and xen-access fail X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP On 01/10/2017 04:13 PM, Andrew Cooper wrote: > On 10/01/17 09:06, Razvan Cojocaru wrote: >> On 01/09/2017 02:54 PM, Andrew Cooper wrote: >>> On 09/01/17 11:36, Razvan Cojocaru wrote: >>>> Hello, >>>> >>>> We've come across a weird phenomenon: an Ubuntu 16.04.1 LTS HVM guest >>>> running kernel 4.4.0 installed via XenCenter in XenServer Dundee seems >>>> to eat up all the RAM it can: >>>> >>>> (XEN) [ 394.379760] d1v1 Over-allocation for domain 1: 524545 > 524544 >>>> >>>> This leads to a problem with xen-access, specifically libxc which does >>>> this in xc_vm_event_enable() (this is Xen 4.6): >>>> >>>> ring_page = xc_map_foreign_batch(xch, domain_id, PROT_READ | PROT_WRITE, >>>> &mmap_pfn, 1); >>>> >>>> if ( mmap_pfn & XEN_DOMCTL_PFINFO_XTAB ) >>>> { >>>> /* Map failed, populate ring page */ >>>> rc1 = xc_domain_populate_physmap_exact(xch, domain_id, 1, 0, 0, >>>> &ring_pfn); >>>> if ( rc1 != 0 ) >>>> { >>>> PERROR("Failed to populate ring pfn\n"); >>>> goto out; >>>> } >>>> >>>> The first time everything works fine, xen-access can map the ring page. >>>> But most of the time the second time fails in the >>>> xc_domain_populate_physmap_exact() call, and again this is dumped in the >>>> Xen log (once for each failed attempt): >>>> >>>> (XEN) [ 395.952188] d0v3 Over-allocation for domain 1: 524545 > 524544 >>> Thinking further about this, what happens if you avoid removing the page >>> on exit? >>> >>> The first populate succeeds, and if you leave the page populated, the >>> second time you come around the loop, it should not be of type XTAB, and >>> the map should succeed. >> Sorry for the late reply, had to put out another fire yesterday. >> >> I've taken your recommendation to roughly mean this: >> >> diff --git a/xen/common/vm_event.c b/xen/common/vm_event.c >> index ba9690a..805564b 100644 >> --- a/xen/common/vm_event.c >> +++ b/xen/common/vm_event.c >> @@ -100,8 +100,11 @@ static int vm_event_enable( >> return 0; >> >> err: >> + /* >> destroy_ring_for_helper(&ved->ring_page, >> ved->ring_pg_struct); >> + */ >> + ved->ring_page = NULL; >> vm_event_ring_unlock(ved); >> >> return rc; >> @@ -229,9 +232,12 @@ static int vm_event_disable(struct domain *d, >> struct vm_event_domain *ved) >> } >> } >> >> + /* >> destroy_ring_for_helper(&ved->ring_page, >> ved->ring_pg_struct); >> + */ >> >> + ved->ring_page = NULL; >> vm_event_cleanup_domain(d); >> >> vm_event_ring_unlock(ved); >> >> but this unfortunately still fails to map the page the second time. Do >> you mean to simply no longer munmap() the ring page from libxc / the >> client application? > > Neither. > > First of all, I notice that this is probably buggy: > > ring_pfn = pfn; > mmap_pfn = pfn; > rc1 = xc_get_pfn_type_batch(xch, domain_id, 1, &mmap_pfn); > if ( rc1 || mmap_pfn & XEN_DOMCTL_PFINFO_XTAB ) > { > /* Page not in the physmap, try to populate it */ > rc1 = xc_domain_populate_physmap_exact(xch, domain_id, 1, 0, 0, > &ring_pfn); > if ( rc1 != 0 ) > { > PERROR("Failed to populate ring pfn\n"); > goto out; > } > } > > A failure of xc_get_pfn_type_batch() is not a suggestion that population > might work. > > > What I meant was taking out this call: > > /* Remove the ring_pfn from the guest's physmap */ > rc1 = xc_domain_decrease_reservation_exact(xch, domain_id, 1, 0, > &ring_pfn); > if ( rc1 != 0 ) > PERROR("Failed to remove ring page from guest physmap"); > > To leave the frame in the guest physmap. The issue is fundamentally > that after this frame has been taken out, something kicks the VM to > realise it has an extra frame of balloonable space, which it clearly > compensates for. > > You can work around the added attack surface by marking it RO in EPT; > neither Xen's nor dom0's mappings are translated via EPT, so they can > still make updates, but the guest won't be able to write to it. > > I should say that this is all a gross hack, and is in desperate need of > a proper API to make rings entirely outside of the gfn space, but this > hack should work for now. Thanks! So far, it seems to work like a charm like this: out: saved_errno = errno; Should I send this as a patch for mainline as well? Thanks, Razvan diff --git a/tools/libxc/xc_vm_event.c b/tools/libxc/xc_vm_event.c index 2fef96a..5dd00a6 100644 --- a/tools/libxc/xc_vm_event.c +++ b/tools/libxc/xc_vm_event.c @@ -130,9 +130,17 @@ void *xc_vm_event_enable(xc_interface *xch, domid_t domain_id, int param, } /* Remove the ring_pfn from the guest's physmap */ + /* rc1 = xc_domain_decrease_reservation_exact(xch, domain_id, 1, 0, &ring_pfn); if ( rc1 != 0 ) PERROR("Failed to remove ring page from guest physmap"); + */ + + if ( xc_set_mem_access(xch, domain_id, XENMEM_access_r, mmap_pfn, 1) ) + { + PERROR("Could not set ring page read-only\n"); + goto out; + }