From patchwork Tue Apr 26 19:21:56 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Cao, Lei" X-Patchwork-Id: 8944031 Return-Path: X-Original-To: patchwork-kvm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 44D8FBF29F for ; Tue, 26 Apr 2016 19:50:11 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 2E1DC2015A for ; Tue, 26 Apr 2016 19:50:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 30E5C201C7 for ; Tue, 26 Apr 2016 19:50:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752515AbcDZTuE (ORCPT ); Tue, 26 Apr 2016 15:50:04 -0400 Received: from mail-by2on0131.outbound.protection.outlook.com ([207.46.100.131]:34485 "EHLO na01-by2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752210AbcDZTgu convert rfc822-to-8bit (ORCPT ); Tue, 26 Apr 2016 15:36:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=StratusTechnologies.onmicrosoft.com; s=selector1-stratus-com; h=From:To:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=znees0mK8pcfa460DptNgBzdZ4pnbhsT38KT2iQX3uY=; b=ZJIc1DSrmRmYbEtp9N0T2vqIeQ/J7RsRRqi0atEAagjaqPvs0CjOx6B3e5cAqQ8YUbI1d26VzRt5qYoyoonu3D9qm0kFHcYmW0FIXhOcOb64QNz1+sv/+hAUWvkj01+8bwCyzjjIzlTTZcdO3Hj6cHjppygtJxRJUuSudIMxMFw= Received: from BL2PR08MB481.namprd08.prod.outlook.com (10.141.93.14) by BL2PR08MB484.namprd08.prod.outlook.com (10.141.93.26) with Microsoft SMTP Server (TLS) id 15.1.466.19; Tue, 26 Apr 2016 19:21:56 +0000 Received: from BL2PR08MB481.namprd08.prod.outlook.com ([10.141.93.14]) by BL2PR08MB481.namprd08.prod.outlook.com ([10.141.93.14]) with mapi id 15.01.0485.005; Tue, 26 Apr 2016 19:21:56 +0000 From: "Cao, Lei" To: Paolo Bonzini , =?iso-8859-2?Q?Radim_Kr=E8m=E1=F8?= , "kvm@vger.kernel.org" Subject: [PATCH 1/6] KVM: Dirty memory tracking for performant checkpointing and improved live migration Thread-Topic: [PATCH 1/6] KVM: Dirty memory tracking for performant checkpointing and improved live migration Thread-Index: AQHRn/DlTvPr0ewTC0i7HqsDzRMccQ== Date: Tue, 26 Apr 2016 19:21:56 +0000 Message-ID: References: <201604261855.u3QItTqS023891@dev1.sn.stratus.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: redhat.com; dkim=none (message not signed) header.d=none; redhat.com; dmarc=none action=none header.from=stratus.com; x-originating-ip: [198.97.42.5] x-ms-office365-filtering-correlation-id: ce654607-4db8-4e52-2fef-08d36e080869 x-microsoft-exchange-diagnostics: 1; BL2PR08MB484; 5:9CKyrqK57Vgh0jhmgJCK3wsy1nllodiPVORfXB2wDyD4Wve+3ZfWX9eXtFDCzkDDgwz9TZJTLhuPqucKAFehDTz1VbeTA2TugZ5F0zfVt0jsKJ3ihXNHvevB0yBoKvQcEQixRSWfP/nWHBVX91qZ8OeyMeL12SXF/tP36rMLqCOEWfdPYPzdv2ieSy19Booj; 24:J9trJaVAxxqLrV9KYd0806bqMPfSEQSFGeqoinHZv/fJuEqE/Z8/yeCDa85HUIkBIbp2sOCW3TV9LXBD52TXLvf8LqXAhLGNhrw/HdBtHwU=; 7:b3KLBa7gJXrvposMFsCMxnsw781qwcQYXUWVTOHw7CIUZNRliiAhR0MmJJUq7TCLe61OafjE7V+qO+ONexSUTkIexsl3sBlvXELgbTNUJKD5W4Gq/xet3APkpncFLofeu5c8RcdDqTeKo52KQfdxLvPNjUrQbad6F11PJke2K7xh7y997QJzcEgOg1IrpYQ/9RD+jK8qVcAYUu2NOaFV5b+8YySOXWwK+gS5mfihgdE=; 20:z10gdsoZR70ZskXgJVTv3NbPU/dvKcdtIs8bj7Sr0ASp4kQ+tAD5T6eB3E/f2R7lR4rzxSqbZVrSqiI1+jfp+hbZO9IDkkNGeY0lzai9IaraGwYko3HURXJ2IBZSBIxYuIr8e9N26DyslUR3gB5cGuSuCNcROfGMMaZ01Snza5s= x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BL2PR08MB484; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(9101521072)(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001); SRVR:BL2PR08MB484; BCL:0; PCL:0; RULEID:; SRVR:BL2PR08MB484; x-forefront-prvs: 0924C6A0D5 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(5003600100002)(54356999)(76176999)(9686002)(50986999)(86362001)(5008740100001)(122556002)(77096005)(74316001)(5002640100001)(3280700002)(76576001)(3660700001)(5004730100002)(10400500002)(19580405001)(81166005)(19580395003)(2906002)(1096002)(87936001)(586003)(2501003)(229853001)(1220700001)(2900100001)(11100500001)(92566002)(107886002)(99286002)(189998001)(106116001)(66066001)(102836003)(6116002)(5001770100001)(33656002)(14143004); DIR:OUT; SFP:1102; SCL:1; SRVR:BL2PR08MB484; H:BL2PR08MB481.namprd08.prod.outlook.com; FPR:; SPF:None; MLV:sfv; LANG:en; spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: stratus.com X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Apr 2016 19:21:56.6620 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: de36b473-b8ad-46ff-837f-9da16b8d1b77 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL2PR08MB484 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Spam-Status: No, score=-7.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,T_DKIM_INVALID,UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Introduce new memory tracking ioctls to support performant checkpoint/restore implementations. See patch 6 for details on the new ioctls. Signed-off-by: Lei Cao --- include/uapi/linux/kvm.h | 94 +++++++++++++++++++++++++++++++++++++ virt/kvm/kvm_main.c | 96 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 190 insertions(+) diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index a7f1f80..2bce4db 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -1312,4 +1312,98 @@ struct kvm_assigned_msix_entry { __u16 padding[3]; }; +#define KVM_MT_VERSION 1 +struct mt_setup { + __u32 version; + + /* which operation to perform? */ +#define KVM_MT_OP_INIT 1 +#define KVM_MT_OP_CLEANUP 2 + __u32 op; + + /* + * flags bit defs: + */ + + /* + * Features. + * 1. Avoid logging duplicate entries + */ +#define KVM_MT_OPTION_NO_DUPS (1 << 2) + + __u32 flags; + + /* max number of dirty pages per checkpoint cycle */ + __u32 max_dirty; +}; + +struct mt_enable { + __u32 flags; /* 1 -> on, 0 -> off */ +}; + +#define MT_OFFSET_MASK (0x0000ffffffffffffUL) + +#define MT_MAKE_SLOT_OFFSET(slot, offset) \ + do { \ + __u64 slot_off = offset & MT_OFFSET_MASK; \ + slot_off |= ((__u64)slot << 48); \ + slot_off; \ + } while (0) + +#define MT_OFFSET_FROM_SLOT_OFFSET(slot_off) \ + (slot_off & MT_OFFSET_MASK) + +#define MT_SLOT_FROM_SLOT_OFFSET(slot_off) \ + (slot_off >> 48) + +struct mt_gfn_list { + __s32 count; + __u32 max_dirty; + __u64 *gfnlist; +}; + +struct mt_prepare_cp { + __s64 cpid; +}; + +struct mt_sublist_fetch_info { + struct mt_gfn_list gfn_info; + + /* + * flags bit defs: + */ + + /* caller sleeps until dirty count is reached */ +#define MT_FETCH_WAIT (1 << 0) + /* dirty tracking is re-armed for each page in returned list */ +#define MT_FETCH_REARM (1 << 1) + + __u32 flags; +}; + +struct mt_dirty_trigger { + /* force vcpus to exit when trigger is reached */ + __u32 dirty_trigger; +}; + +/* Initialize/Cleanup MT data structures, allocate/free list buffers, etc. */ +#define KVM_INIT_MT _IOW(KVMIO, 0xf0, struct mt_setup) +/* Active/Deactivate Memory Tracking */ +#define KVM_ENABLE_MT _IOW(KVMIO, 0xf1, struct mt_enable) +/* notify MT subsystem that VM is about to be unpaused */ +#define KVM_PREPARE_MT_CP _IOW(KVMIO, 0xf2, struct mt_prepare_cp) +/* Rearm dirty traps for specified pages */ +#define KVM_REARM_DIRTY_PAGES _IO(KVMIO, 0xf3) +/* notify MT subsystem no more pages will be dirtied this cycle */ +#define KVM_MT_VM_QUIESCED _IO(KVMIO, 0xf4) +/* + * Return specified number of dirty pages. May return fewer than requested. + * Optionally, caller can request to sleep until desired number is reached. + * The KVM_MT_VM_QUIESCED call above will wake this sleeper even if the + * number of dirty pages is not yet the requested amount. + */ +#define KVM_MT_SUBLIST_FETCH _IOWR(KVMIO, 0xf5, struct mt_sublist_fetch_info) +/* Set VM exit trigger point based on dirty page count */ +#define KVM_MT_DIRTY_TRIGGER _IOW(KVMIO, 0xf6, struct mt_dirty_trigger) + #endif /* __LINUX_KVM_H */ diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 4fd482f..8a582e5 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -2752,6 +2752,43 @@ static long kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg) return kvm_vm_ioctl_check_extension(kvm, arg); } +static int kvm_vm_ioctl_mt_init(struct kvm *kvm, struct mt_setup *mts) +{ + return -EINVAL; +} + +static int kvm_vm_ioctl_mt_enable(struct kvm *kvm, struct mt_enable *mte) +{ + return -EINVAL; +} + +static int kvm_vm_ioctl_mt_prepare_cp(struct kvm *kvm, + struct mt_prepare_cp *mtpcp) +{ + return -EINVAL; +} + +static int kvm_vm_ioctl_mt_rearm_gfns(struct kvm *kvm) +{ + return -EINVAL; +} + +static int kvm_vm_ioctl_mt_quiesced(struct kvm *kvm) +{ + return -EINVAL; +} + +static int kvm_vm_ioctl_mt_sublist_fetch(struct kvm *kvm, + struct mt_sublist_fetch_info *mtsfi) +{ + return -EINVAL; +} + +static int kvm_vm_ioctl_mt_dirty_trigger(struct kvm *kvm, int dirty_trigger) +{ + return -EINVAL; +} + static long kvm_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) { @@ -2909,6 +2946,65 @@ out_free_irq_routing: case KVM_CHECK_EXTENSION: r = kvm_vm_ioctl_check_extension_generic(kvm, arg); break; + case KVM_INIT_MT: { + struct mt_setup mts; + + r = -EFAULT; + if (copy_from_user(&mts, (void __user *)arg, sizeof(mts))) + goto out; + r = kvm_vm_ioctl_mt_init(kvm, &mts); + break; + } + case KVM_ENABLE_MT: { + struct mt_enable mte; + + r = -EFAULT; + if (copy_from_user(&mte, (void __user *)arg, sizeof(mte))) + goto out; + r = kvm_vm_ioctl_mt_enable(kvm, &mte); + break; + } + case KVM_PREPARE_MT_CP: { + struct mt_prepare_cp mtpcp; + + r = -EFAULT; + if (copy_from_user(&mtpcp, (void __user *)arg, sizeof(mtpcp))) + goto out; + r = kvm_vm_ioctl_mt_prepare_cp(kvm, &mtpcp); + break; + } + case KVM_REARM_DIRTY_PAGES: { + r = kvm_vm_ioctl_mt_rearm_gfns(kvm); + break; + } + case KVM_MT_VM_QUIESCED: { + r = kvm_vm_ioctl_mt_quiesced(kvm); + break; + } + case KVM_MT_SUBLIST_FETCH: { + struct mt_sublist_fetch_info mtsfi; + + r = -EFAULT; + if (copy_from_user(&mtsfi, (void __user *)arg, sizeof(mtsfi))) + goto out; + r = kvm_vm_ioctl_mt_sublist_fetch(kvm, &mtsfi); + if (r) + goto out; + r = -EFAULT; + if (copy_to_user((void __user *)arg, &mtsfi, sizeof(mtsfi))) + goto out; + r = 0; + break; + } + case KVM_MT_DIRTY_TRIGGER: { + struct mt_dirty_trigger mtdt; + + r = -EFAULT; + if (copy_from_user(&mtdt, (void __user *)arg, sizeof(mtdt))) + goto out; + r = kvm_vm_ioctl_mt_dirty_trigger(kvm, mtdt.dirty_trigger); + break; + } default: r = kvm_arch_vm_ioctl(filp, ioctl, arg); }