From patchwork Tue Apr 26 19:26:47 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Cao, Lei" X-Patchwork-Id: 8943401 Return-Path: X-Original-To: patchwork-kvm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 1E2DD9F441 for ; Tue, 26 Apr 2016 19:26:59 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id D10FF2015A for ; Tue, 26 Apr 2016 19:26:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A1FA3200DE for ; Tue, 26 Apr 2016 19:26:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752866AbcDZT0x (ORCPT ); Tue, 26 Apr 2016 15:26:53 -0400 Received: from mail-by2on0110.outbound.protection.outlook.com ([207.46.100.110]:36672 "EHLO na01-by2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752255AbcDZT0w convert rfc822-to-8bit (ORCPT ); Tue, 26 Apr 2016 15:26:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=StratusTechnologies.onmicrosoft.com; s=selector1-stratus-com; h=From:To:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=1Jmt6Cmz8Cz1byZTqd2QK1MNK3OjEzyRBtULcRGnC3o=; b=orl94uddCJJ23hcsH1aEM7+hTHsa+flj9QAZgZUFGNbiCZQSiRE0Zp7lTrNisNbQAlFJrLvpWmAAb5n3fKudaVVGowNSjM/tQcdveLq1TUokeacXlrBe0XioPC+qrkoKZUL0SsVk/R1W0T+8KFrIiayVy9NpVkkZq2Vjy0zmsNg= Received: from BL2PR08MB481.namprd08.prod.outlook.com (10.141.93.14) by BL2PR08MB481.namprd08.prod.outlook.com (10.141.93.14) with Microsoft SMTP Server (TLS) id 15.1.485.3; Tue, 26 Apr 2016 19:26:47 +0000 Received: from BL2PR08MB481.namprd08.prod.outlook.com ([10.141.93.14]) by BL2PR08MB481.namprd08.prod.outlook.com ([10.141.93.14]) with mapi id 15.01.0485.005; Tue, 26 Apr 2016 19:26:47 +0000 From: "Cao, Lei" To: Paolo Bonzini , =?iso-8859-2?Q?Radim_Kr=E8m=E1=F8?= , "kvm@vger.kernel.org" Subject: [PATCH 6/6] KVM: Dirty memory tracking for performant checkpointing and improved live migration Thread-Topic: [PATCH 6/6] KVM: Dirty memory tracking for performant checkpointing and improved live migration Thread-Index: AQHRn/GS2jPcATI470iMv4IfI7dtfA== Date: Tue, 26 Apr 2016 19:26:47 +0000 Message-ID: References: <201604261856.u3QIuJMs025122@dev1.sn.stratus.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: redhat.com; dkim=none (message not signed) header.d=none; redhat.com; dmarc=none action=none header.from=stratus.com; x-originating-ip: [198.97.42.5] x-ms-office365-filtering-correlation-id: ec8525fe-8529-4f4c-c73d-08d36e08b590 x-microsoft-exchange-diagnostics: 1; BL2PR08MB481; 5:FJ7x5oI8qWniOnUZ/utENAFHuFjNa2j5mlNMX3RDcWqWSVJJmQ8zW6Gk0E4JShx66+0+86OJgjR1gbyQBW5OT/XWEZgYEvOSUzAFkEFPwfDDD86MXkQ+nggCL8DeQNr5H+xAXo9owpm4ZWNyR0g7Ig==; 24:CrG+UaWdYe1agSpxhZJKZ4Mf8t4333cyBAnox3bcy9dNfJuWx8R859Frtn25VjeCvpPT+9mrS4ZpI22PtT8NxzsmH0ZG9rRl424zzsPggdw=; 7:C9gJdoZL12XAdlypVlmknhdsnrO/jF5YCZtzYZbnKzBaBs+0lNLiy3pVU4cw6v7gv4Bk+e2GhzM9yh1BPcYLswJ8BQpLD7ZlNOC7ijZ9Ti0r53hT207EAPAtgM4YtgZvVNsdueBdjTT6Qj68Q634zCJH3nABnTScoOx99tSMfdMSsprRH/5y5WhD4+Zr4/zC; 20:JG17h9tprTRcMPr2ZZ87pHfCl3NIBR5s5HRdlpN4HeWuS2ReAaNAjDiiSYZvH8oMsjL3w3zg7IKmMVtUfovFk9EgpristzIEwkwvQM52OlT3nbak16QM0XT+NC5uXfo1VcESpK8GDHzwrIQ6Q7YG0pkcWt/GYWToS449NJkKF1Y= x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BL2PR08MB481; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(9101521072)(601004)(2401047)(8121501046)(5005006)(3002001)(10201501046); SRVR:BL2PR08MB481; BCL:0; PCL:0; RULEID:; SRVR:BL2PR08MB481; x-forefront-prvs: 0924C6A0D5 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(122556002)(77096005)(2900100001)(5008740100001)(76176999)(54356999)(5003600100002)(50986999)(33656002)(11100500001)(19580395003)(87936001)(6116002)(102836003)(106116001)(3660700001)(1220700001)(586003)(9686002)(2906002)(1096002)(3280700002)(99286002)(86362001)(74316001)(66066001)(2501003)(10400500002)(76576001)(5001770100001)(107886002)(5004730100002)(81166005)(229853001)(189998001)(5002640100001)(92566002)(14143004); DIR:OUT; SFP:1102; SCL:1; SRVR:BL2PR08MB481; H:BL2PR08MB481.namprd08.prod.outlook.com; FPR:; SPF:None; MLV:sfv; LANG:en; spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: stratus.com X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Apr 2016 19:26:47.1829 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: de36b473-b8ad-46ff-837f-9da16b8d1b77 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL2PR08MB481 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Spam-Status: No, score=-7.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,T_DKIM_INVALID,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Updates to KVM API documentation --- Documentation/virtual/kvm/api.txt | 170 ++++++++++++++++++++++++++++ 1 file changed, 170 insertions(+) diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt index 4d0542c..3f5367a 100644 --- a/Documentation/virtual/kvm/api.txt +++ b/Documentation/virtual/kvm/api.txt @@ -3120,6 +3120,176 @@ struct kvm_reinject_control { pit_reinject = 0 (!reinject mode) is recommended, unless running an old operating system that uses the PIT for timing (e.g. Linux 2.4.x). +4.99 KVM_INIT_MT + +Capability: basic +Architectures: x86 +Type: vm ioctl +Parameters: struct mt_setup (in) +Returns: 0 on success, -1 on error + +/* for KVM_INIT_MT */ +struct mt_setup { +#define KVM_MT_VERSION 1 +struct mt_setup { + __u32 version; + + /* which operation to perform? */ +#define KVM_MT_OP_INIT 1 +#define KVM_MT_OP_CLEANUP 2 + __u32 op; + + /* + * Features. + * 1. Avoid logging duplicate entries + */ +#define KVM_MT_OPTION_NO_DUPS (1 << 2) + + __u32 flags; + + /* max number of dirty pages per checkpoint cycle */ + __u32 max_dirty; +}; + +This instructs the memory tracking (MT) subsystem to initialize or +cleanup memory tracking data structures. Userspace specifies the +memory tracking version to make sure it and KVM are on the same +page. For initialization, userspace specifies the maxinum number +of dirty pages that is allowed per checkpoint cycle. It can tell +KVM to avoid logging duplicate pages via 'flags', and KVM would +create bitmap to track dirty pages. + +Called once during initialization. + +4.100 KVM_ENABLE_MT + +Capability: basic +Architectures: x86 +Type: vm ioctl +Parameters: struct mt_enable (in) +Returns: 0 on success, -1 on error + +/* for KVM_ENABLE_MT */ +struct mt_enable { + __u32 flags; /* 1 -> on, 0 -> off */ +}; + +This instructs the MT subsystem to start/stop logging dirty +VM pages. On hosts that support fault based memory tracking, KVM +write-protects all VM pages to start dirty logging. On hosts that +support PML, KVM clears the dirty bits for all VM pages to start +dirty logging, and sets the dirty bits to stop dirty logging. + +Called once when entering/exiting live migration/checkpoint mode. + +4.101 KVM_PREPARE_MT_CP + +Capability: basic +Architectures: x86 +Type: vm ioctl +Parameters: struct mt_prepare_cp (in) +Returns: 0 on success, -1 on error + +/* for KVM_PREPARE_MT_CP */ +struct mt_prepare_cp { + __s64 cpid; +}; + +This instructs the MT subsystem that a new checkpoint cycle is +about to start and provides the cycle ID. The MT subsystem resets +all the relevant variables, assuming all dirty pages have been +fetched. + +Called once per checkpoint cycle. + +4.102 KVM_MT_SUBLIST_FETCH + +Capability: basic +Architectures: x86 +Type: vm ioctl +Parameters: struct mt_sublist_fetch_info (in/out) +Returns: 0 on success, -1 on error + +/* for KVM_MT_SUBLIST_FETCH */ +struct mt_gfn_list { + __s32 count; + __u32 max_dirty; + __u64 *gfnlist; +}; + +struct mt_sublist_fetch_info { + struct mt_gfn_list gfn_info; + + /* + * flags bit defs: + */ + + /* caller sleeps until dirty count is reached */ +#define MT_FETCH_WAIT (1 << 0) + /* dirty tracking is re-armed for each page in returned list */ +#define MT_FETCH_REARM (1 << 1) + + __u32 flags; +}; + +This fetches a subset of the current dirty pages. Userspace thread +specifies the maximum number of dirty pages it wants to fetch via +(struct mt_gfn_list).count. It also tells the MT subsystem if it is +going to wait until the specified maxinum number is reached. Userspace +thread can instruct the MT subsystem to re-arm the dirty trap for +each page that is fetched. The dirty pages are returned to userspace +in (struct mt_gfn_list).gfnlist, and (struct mt_gfn_list).count +indicates the number of dirty pages that are returned. + +Called multiple times by multiple threads per checkpoint cycle. + +4.103 KVM_REARM_DIRTY_PAGES + +Capability: basic +Architectures: x86 +Type: vm ioctl +Parameters: +Returns: 0 on success, -1 on error + +This instructs the MT subsystem to rearm the dirty traps for all +the pages that were dirtied during the last checkpoint cycle. + +Called once per checkpoint cycle. The call is not necessary if dirty +traps are rearmed when dirty pages are being fetched. + +4.104 KVM_MT_VM_QUIESCED + +Capability: basic +Architectures: x86 +Type: vm ioctl +Parameters: +Returns: 0 on success, -1 on error + +This instructs the MT subsystem that the VM has been quiesced and no +more pages will be dirtied this checkpoint cycle. The MT subsystem +will wake up userspace threads that are waiting for new dirty pages +to fetch, if any. + +Called once per checkpoint cycle. + +4.105 KVM_MT_DIRTY_TRIGGER + +Capability: basic +Architectures: x86 +Type: vm ioctl +Parameters: struct mt_dirty_trigger (in) +Returns: 0 on success, -1 on error + +/* for KVM_MT_DIRTY_TRIGGER */ +struct mt_dirty_trigger { + /* force vcpus to exit when trigger is reached */ + __u32 dirty_trigger; +}; + +This sets the VM exit trigger point based on dirty page count. + +Called once when entering live migration/checkpoint mode. + 5. The kvm_run structure ------------------------