From patchwork Tue Aug 29 09:58:24 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Olaf Hering X-Patchwork-Id: 9927025 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 6BAA960380 for ; Tue, 29 Aug 2017 10:01:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 57EF2288B1 for ; Tue, 29 Aug 2017 10:01:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4CB5F288B7; Tue, 29 Aug 2017 10:01:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 3374C288B1 for ; Tue, 29 Aug 2017 10:01:07 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dmdI8-0004kf-Mb; Tue, 29 Aug 2017 09:58:40 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dmdI6-0004kM-PT for xen-devel@lists.xen.org; Tue, 29 Aug 2017 09:58:39 +0000 Received: from [85.158.139.211] by server-5.bemta-5.messagelabs.com id 9E/80-02228-ECA35A95; Tue, 29 Aug 2017 09:58:38 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrMIsWRWlGSWpSXmKPExsUSuHLSTd1TVks jDbZNYbdY8nExiwOjx9Hdv5kCGKNYM/OS8isSWDPefFnAWLDGoGLG9x/sDYzfVLsYuThYBOYz S7y4foOpi5GDQ0IgVGL5FVsIU0Tiyf80kBIhgctMEleuTWDtYuQEqleV6F7+A8xmE1CS2HvwO CNIkYjABEaJo093giWEBUIkJj9fwwxi8wroShydsx/MFhKIkWje/oQVIi4ocXLmExYQm1mgWG Lnt3/sIIuZBaQllv/jAAlzCphINO6awwhiiwrISty/soFxAiP/LCTds5B0z0LohgirS/yZd4k ZQ1hbYtnC18wQtq3EunXvWRYwsq9i1ChOLSpLLdI1NNdLKspMzyjJTczM0TU0MNXLTS0uTkxP zUlMKtZLzs/dxAgMcAYg2MF48bTnIUZJDiYlUV5J1aWRQnxJ+SmVGYnFGfFFpTmpxYcYZTg4l CR4+YERIyRYlJqeWpGWmQOMNZi0BAePkgivpCVQmre4IDG3ODMdInWKUZdjw+r1X5iEWPLy81 KlxHnPghQJgBRllObBjYDF/SVGWSlhXkago4R4ClKLcjNLUOVfMYpzMCoJ8x4GmcKTmVcCt+k V0BFMQEcoCi4EOaIkESEl1cAY8y936ayV1+X/VDPxa9g+EXnmdGzm7ZLupm1sggKW1XtuXBCd ev/dovi2l0x/e7imvow4NMVe5Ivwx8KAf1/qtx71Sc0VT2PuEMs7Ide32SRy/uKPh9lTv0tN/ 1iUfFOtJkpjxkv3lP8qIee0rVL0y/bdu6rRvOtEa3jusx2hDx6s5fm7WOSrEktxRqKhFnNRcS IAcS+J3PYCAAA= X-Env-Sender: olaf@aepfle.de X-Msg-Ref: server-12.tower-206.messagelabs.com!1504000713!73040478!1 X-Originating-IP: [81.169.146.217] X-SpamReason: No, hits=0.5 required=7.0 tests=BODY_RANDOM_LONG X-StarScan-Received: X-StarScan-Version: 9.4.45; banners=-,-,- X-VirusChecked: Checked Received: (qmail 29295 invoked from network); 29 Aug 2017 09:58:34 -0000 Received: from mo4-p00-ob.smtp.rzone.de (HELO mo4-p00-ob.smtp.rzone.de) (81.169.146.217) by server-12.tower-206.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 29 Aug 2017 09:58:34 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1504000713; s=domk; d=aepfle.de; h=In-Reply-To:Content-Disposition:Content-Type:MIME-Version: References:Subject:To:From:Date; bh=rkmYVdPaUK/w7CtpgnEBjOReVZcGbnLzk09XY1izqdI=; b=GB34dn8un0a/eiIFRfnEHUZVxMr03OVyRNVubjq+LnAmu8Zhnsa1OBRYHU+iBnpShM OJcirTCLa0h8gl2vX7ppuXLwdZMtIxR277UatPpcnI8RIkxl4d2osh0Qyi9NDBPWctdX GRBSlusdZB95RcQdYgazF3tOnRPJwKvVPdXtk= X-RZG-AUTH: :P2EQZWCpfu+qG7CngxMFH1J+yackYocTD1iAi8x+OWi5z/J1IL7CYRxLEwpkY2XtLzyRUcY= X-RZG-CLASS-ID: mo00 Received: from aepfle.de (nat.nue.novell.com [IPv6:2620:113:80c0:5::2222]) by smtp.strato.de (RZmta 41.4 AUTH) with ESMTPSA id Y015f9t7T9wRzjj (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (curve secp521r1 with 521 ECDH bits, eq. 15360 bits RSA)) (Client did not present a certificate); Tue, 29 Aug 2017 11:58:27 +0200 (CEST) Date: Tue, 29 Aug 2017 11:58:24 +0200 From: Olaf Hering To: xen-devel@lists.xen.org, Andrew Cooper , Ian Jackson , Wei Liu Message-ID: <20170829095823.GA9803@aepfle.de> References: <20170826103332.24570-1-olaf@aepfle.de> <20170826103332.24570-4-olaf@aepfle.de> MIME-Version: 1.0 In-Reply-To: <20170826103332.24570-4-olaf@aepfle.de> User-Agent: Mutt/1.8.3 (7138) Subject: Re: [Xen-devel] [PATCH v6 3/3] tools/libxc: use superpages during restore of HVM guest X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP On Sat, Aug 26, Olaf Hering wrote: > +static int x86_hvm_populate_pfns(struct xc_sr_context *ctx, unsigned count, > + /* > + * Scan the entire superpage because several batches will fit into > + * a superpage, and it is unknown which pfn triggered the allocation. > + */ > + order = SUPERPAGE_1GB_SHIFT; > + pfn = min_pfn = (min_pfn >> order) << order; Scanning an entire superpage again and again looked expensive, but with the debug change below it turned out that the loop which peeks at each single bit in populated_pfns is likely not a bootleneck. Migrating a domU with a simple workload that touches pages to mark them dirty will set the min_pfn/max_pfn to a large range anyway after the first iteration. This large range may also happen with an idle domU. A small domU takes 78 seconds to migrate, and just the freeing part takes 1.4 seconds. Similar for a large domain, the loop takes 1% of the time. 78 seconds, 1.4 seconds, 2119 calls (8GB, 12*512M memdirty) 695 seconds, 7.6 seconds, 18076 calls (72GB, 12*5G memdirty) Olaf track time spent if decrease_reservation is needed diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h index 0fa0fbea4d..5ec8b6fee6 100644 --- a/tools/libxc/xc_sr_common.h +++ b/tools/libxc/xc_sr_common.h @@ -353,6 +353,9 @@ struct xc_sr_context struct xc_sr_bitmap attempted_1g; struct xc_sr_bitmap attempted_2m; struct xc_sr_bitmap allocated_pfns; + + unsigned long tv_nsec; + unsigned long iterations; } restore; }; } x86_hvm; diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c index 8cd9289d1a..f6aad329e2 100644 --- a/tools/libxc/xc_sr_restore.c +++ b/tools/libxc/xc_sr_restore.c @@ -769,6 +769,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom, { ctx.restore.ops = restore_ops_x86_hvm; if ( restore(&ctx) ) + ; return -1; } else diff --git a/tools/libxc/xc_sr_restore_x86_hvm.c b/tools/libxc/xc_sr_restore_x86_hvm.c index 2b0eca0c7c..11758b3f7d 100644 --- a/tools/libxc/xc_sr_restore_x86_hvm.c +++ b/tools/libxc/xc_sr_restore_x86_hvm.c @@ -1,5 +1,6 @@ #include #include +#include #include "xc_sr_common_x86.h" @@ -248,6 +249,12 @@ static int x86_hvm_stream_complete(struct xc_sr_context *ctx) static int x86_hvm_cleanup(struct xc_sr_context *ctx) { + xc_interface *xch = ctx->xch; + errno = 0; + PERROR("tv_nsec %lu.%lu iterations %lu", + ctx->x86_hvm.restore.tv_nsec / 1000000000UL, + ctx->x86_hvm.restore.tv_nsec % 1000000000UL, + ctx->x86_hvm.restore.iterations); free(ctx->x86_hvm.restore.context); xc_sr_bitmap_free(&ctx->x86_hvm.restore.attempted_1g); xc_sr_bitmap_free(&ctx->x86_hvm.restore.attempted_2m); @@ -440,6 +447,28 @@ static int x86_hvm_allocate_pfn(struct xc_sr_context *ctx, xen_pfn_t pfn) return rc; } +static void diff_timespec(struct xc_sr_context *ctx, const struct timespec *old, const struct timespec *new, struct timespec *diff) +{ + xc_interface *xch = ctx->xch; + if (new->tv_sec == old->tv_sec && new->tv_nsec == old->tv_nsec) + PERROR("%s: time did not move: %ld/%ld == %ld/%ld", __func__, old->tv_sec, old->tv_nsec, new->tv_sec, new->tv_nsec); + if ( (new->tv_sec < old->tv_sec) || (new->tv_sec == old->tv_sec && new->tv_nsec < old->tv_nsec) ) + { + PERROR("%s: time went backwards: %ld/%ld -> %ld/%ld", __func__, old->tv_sec, old->tv_nsec, new->tv_sec, new->tv_nsec); + diff->tv_sec = diff->tv_nsec = 0; + return; + } + if ((new->tv_nsec - old->tv_nsec) < 0) { + diff->tv_sec = new->tv_sec - old->tv_sec - 1; + diff->tv_nsec = new->tv_nsec - old->tv_nsec + 1000000000UL; + } else { + diff->tv_sec = new->tv_sec - old->tv_sec; + diff->tv_nsec = new->tv_nsec - old->tv_nsec; + } + if (diff->tv_sec < 0) + PERROR("%s: time diff broken. old: %ld/%ld new: %ld/%ld diff: %ld/%ld ", __func__, old->tv_sec, old->tv_nsec, new->tv_sec, new->tv_nsec, diff->tv_sec, diff->tv_nsec); +} + static int x86_hvm_populate_pfns(struct xc_sr_context *ctx, unsigned count, const xen_pfn_t *original_pfns, const uint32_t *types) @@ -448,6 +477,7 @@ static int x86_hvm_populate_pfns(struct xc_sr_context *ctx, unsigned count, xen_pfn_t pfn, min_pfn = original_pfns[0], max_pfn = original_pfns[0]; unsigned i, freed = 0, order; int rc = -1; + struct timespec a, b, d; for ( i = 0; i < count; ++i ) { @@ -474,6 +504,8 @@ static int x86_hvm_populate_pfns(struct xc_sr_context *ctx, unsigned count, } } + if (clock_gettime(CLOCK_MONOTONIC, &a)) + PERROR("clock_gettime start"); /* * Scan the entire superpage because several batches will fit into * a superpage, and it is unknown which pfn triggered the allocation. @@ -504,10 +536,17 @@ static int x86_hvm_populate_pfns(struct xc_sr_context *ctx, unsigned count, } pfn++; } - if ( freed ) + if ( 0 && freed ) DPRINTF("freed %u between %" PRI_xen_pfn " %" PRI_xen_pfn "\n", freed, min_pfn, max_pfn); + if (clock_gettime(CLOCK_MONOTONIC, &b)) + PERROR("clock_gettime end"); + + diff_timespec(ctx, &a, &b, &d); + ctx->x86_hvm.restore.tv_nsec += d.tv_nsec + (1000000000UL * d.tv_sec); + ctx->x86_hvm.restore.iterations++; + rc = 0; err: