From patchwork Thu Jun 3 16:43:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 12297653 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 856A7C47096 for ; Thu, 3 Jun 2021 16:45:39 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 59A306024A for ; Thu, 3 Jun 2021 16:45:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 59A306024A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:Cc :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=rtsP3yPXulKrv6PaEPgAFkcOl1KorkcdbRqr8Tug2V4=; b=j89CZqNSzIdYgW 9qc2ptLlpCQNcksWqJ2U6R3KLwJVVftXRYCeQkgJRPacc2RMUIeNDvRrHc7X9FbIkkW+z5weRzYlv c8dJAgK5bdKm8ic+8HWhh3DLFfznss60qawKmruaymSn0pwT7kJluPUh/Pfrw44OL5OlI2hGjzDBe sXF0Bk02TVs0Qfp5+12UjUv6wPdwy7t2m/mDgfPUssSNjQ8WCujW50kKwWhHYii+XTCGTjM+KtIbR fssJzRI/AKVGzSg1CKBbGgTvFn9+9d32d8LKxnP3cUfyDUI0ShuaPpn/EzUNW5pAckc19/tX1Jq5D LaTQEZMbOMEEb8LYor9A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1loqRr-009aeH-AX; Thu, 03 Jun 2021 16:43:59 +0000 Received: from mail-pg1-x536.google.com ([2607:f8b0:4864:20::536]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1loqRm-009acv-S8 for linux-arm-kernel@lists.infradead.org; Thu, 03 Jun 2021 16:43:56 +0000 Received: by mail-pg1-x536.google.com with SMTP id 27so5602514pgy.3 for ; Thu, 03 Jun 2021 09:43:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=9Q2DATu13P1pzwQHWEVjGMuJXKz5nnidG53u8t/T69Y=; b=oZWknSmeVZZfZwya3xUfjHMkrSowYXFuz2O2dLlUa9GiJ/jfDEeFrwHMXwjBA4wEI+ KgJVTITaDxvaEi2nzAHfwWTEAk9ONAyP6YJm2+/EYC4K3inpb83m3E64GzZtZ9U1kDBl lZo5D5cvDzPWuhMzO9hZfc3PzZmWok9dwf5rg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=9Q2DATu13P1pzwQHWEVjGMuJXKz5nnidG53u8t/T69Y=; b=Iy8de9mTazYE46kln9w4X9SUb51SiCyY2E3LOb9Xe59XP75NV6Ho4Dyr30/zc0jPAR dS9jQIvSZVN94VeI+q56WB9DkXyiGfc3HPp2wyRhnaIHx1xJa0vdQHXPocByv7ZeNz7u tBk9DfHaZYBttwmulQz/CD9dM6rvep4+xMtXKFV+vDDY/+w9Un9J5JtYpCjvX7EqtB0s XB6JeDVB/qf1XdxHUcKzQukIpzafDGZhyWwv05V8AYzXwKAcg0QOH8m8Y/Fx7Ifz5fIE WGs3TAhbYj8gq6vBB4zEy6bXRG5+2rNJ0rK5YUaViDa8xFPn5upcD7YKQ5FK/FYtDjec WLOw== X-Gm-Message-State: AOAM531woyGs5CYesNk458ld8EDISgJnC5d+Sps8S5PqW+MB/SdQfifv ZfsnvLdSMMXlhoQqVA+ezydZ5Q== X-Google-Smtp-Source: ABdhPJx7I5PtkMSKl9b8zDas4WwIKw7SMCpW5CkmtGzDxyA7T/ENxk1ZY0pC15QcFzNPM4JDV5yvgQ== X-Received: by 2002:a63:dc4e:: with SMTP id f14mr368523pgj.378.1622738632157; Thu, 03 Jun 2021 09:43:52 -0700 (PDT) Received: from senozhatsky.flets-east.jp ([2409:10:2e40:5100:1585:76c1:e367:901a]) by smtp.gmail.com with ESMTPSA id 76sm2749589pfy.82.2021.06.03.09.43.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Jun 2021 09:43:51 -0700 (PDT) From: Sergey Senozhatsky To: Paolo Bonzini , Sean Christopherson , Vitaly Kuznetsov , Jim Mattson Cc: Marc Zyngier , Huacai Chen , Paul Mackerras , Christian Borntraeger , Suleiman Souhlal , x86@kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, kvm@vger.kernel.org, kvm-ppc@vger.kernel.org, linux-s390@vger.kernel.org, Sergey Senozhatsky Subject: [RFC][PATCH] kvm: add suspend pm-notifier Date: Fri, 4 Jun 2021 01:43:15 +0900 Message-Id: <20210603164315.682994-1-senozhatsky@chromium.org> X-Mailer: git-send-email 2.32.0.rc0.204.g9fa02ecfa5-goog MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210603_094354_970955_E8C2B101 X-CRM114-Status: GOOD ( 16.56 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add KVM suspend/hibernate PM-notifier which lets architectures to implement arch-specific VM suspend code. For instance, on x86 this sets PVCLOCK_GUEST_STOPPED on all the VCPUs. Our case is that user puts the host system into sleep multiple times a day (e.g. closes the laptop's lid) so we need a reliable way to suspend VMs properly. Signed-off-by: Sergey Senozhatsky --- arch/arm64/kvm/arm.c | 4 ++++ arch/mips/kvm/mips.c | 4 ++++ arch/powerpc/kvm/powerpc.c | 4 ++++ arch/s390/kvm/kvm-s390.c | 4 ++++ arch/x86/kvm/x86.c | 21 ++++++++++++++++++++ include/linux/kvm_host.h | 8 ++++++++ virt/kvm/kvm_main.c | 40 ++++++++++++++++++++++++++++++++++++++ 7 files changed, 85 insertions(+) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 1126eae27400..547dbe44d039 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -1311,6 +1311,10 @@ static int kvm_vm_ioctl_set_device_addr(struct kvm *kvm, } } +void kvm_arch_pm_notifier(struct kvm *kvm) +{ +} + long kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) { diff --git a/arch/mips/kvm/mips.c b/arch/mips/kvm/mips.c index 4d4af97dcc88..d4408acd2be6 100644 --- a/arch/mips/kvm/mips.c +++ b/arch/mips/kvm/mips.c @@ -980,6 +980,10 @@ void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm, kvm_flush_remote_tlbs(kvm); } +void kvm_arch_pm_notifier(struct kvm *kvm) +{ +} + long kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) { long r; diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c index a2a68a958fa0..96e8a7b6fcf0 100644 --- a/arch/powerpc/kvm/powerpc.c +++ b/arch/powerpc/kvm/powerpc.c @@ -2334,6 +2334,10 @@ static int kvmppc_get_cpu_char(struct kvm_ppc_cpu_char *cp) } #endif +void kvm_arch_pm_notifier(struct kvm *kvm) +{ +} + long kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) { diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c index 1296fc10f80c..c5f86fc1e497 100644 --- a/arch/s390/kvm/kvm-s390.c +++ b/arch/s390/kvm/kvm-s390.c @@ -2367,6 +2367,10 @@ static int kvm_s390_handle_pv(struct kvm *kvm, struct kvm_pv_cmd *cmd) return r; } +void kvm_arch_pm_notifier(struct kvm *kvm) +{ +} + long kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) { diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index bbc4e04e67ad..3f3d6497593f 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -5613,6 +5613,27 @@ static int kvm_vm_ioctl_set_msr_filter(struct kvm *kvm, void __user *argp) return 0; } +void kvm_arch_pm_notifier(struct kvm *kvm) +{ +#ifdef CONFIG_PM + int c; + + mutex_lock(&kvm->lock); + for (c = 0; c < kvm->created_vcpus; c++) { + struct kvm_vcpu *vcpu = kvm->vcpus[c]; + int r; + + if (!vcpu) + continue; + r = kvm_set_guest_paused(vcpu); + if (!r) + continue; + pr_err("Failed to suspend VCPU-%d: %d\n", vcpu->vcpu_id, r); + } + mutex_unlock(&kvm->lock); +#endif +} + long kvm_arch_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) { diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 2f34487e21f2..86695320a6b7 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -37,6 +37,8 @@ #include #include +#include + #ifndef KVM_MAX_VCPU_ID #define KVM_MAX_VCPU_ID KVM_MAX_VCPUS #endif @@ -579,6 +581,10 @@ struct kvm { pid_t userspace_pid; unsigned int max_halt_poll_ns; u32 dirty_ring_size; + +#ifdef CONFIG_PM + struct notifier_block pm_notifier; +#endif }; #define kvm_err(fmt, ...) \ @@ -992,6 +998,8 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu); void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu); void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu); +void kvm_arch_pm_notifier(struct kvm *kvm); + #ifdef __KVM_HAVE_ARCH_VCPU_DEBUGFS void kvm_arch_create_vcpu_debugfs(struct kvm_vcpu *vcpu, struct dentry *debugfs_dentry); #endif diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 6b4feb92dc79..86925ab7d162 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -51,6 +51,7 @@ #include #include #include +#include #include #include @@ -779,6 +780,43 @@ static int kvm_init_mmu_notifier(struct kvm *kvm) #endif /* CONFIG_MMU_NOTIFIER && KVM_ARCH_WANT_MMU_NOTIFIER */ +#ifdef CONFIG_PM +static int kvm_pm_notifier_call(struct notifier_block *bl, + unsigned long state, + void *unused) +{ + struct kvm *kvm = container_of(bl, struct kvm, pm_notifier); + + switch (state) { + case PM_HIBERNATION_PREPARE: + case PM_SUSPEND_PREPARE: + kvm_arch_pm_notifier(kvm); + break; + } + return NOTIFY_DONE; +} + +static void kvm_init_pm_notifier(struct kvm *kvm) +{ + kvm->pm_notifier.notifier_call = kvm_pm_notifier_call; + kvm->pm_notifier.priority = INT_MAX; + register_pm_notifier(&kvm->pm_notifier); +} + +static void kvm_destroy_pm_notifier(struct kvm *kvm) +{ + unregister_pm_notifier(&kvm->pm_notifier); +} +#else +static void kvm_init_pm_notifier(struct kvm *kvm) +{ +} + +static void kvm_destroy_pm_notifier(struct kvm *kvm) +{ +} +#endif /* CONFIG_PM */ + static struct kvm_memslots *kvm_alloc_memslots(void) { int i; @@ -962,6 +1000,7 @@ static struct kvm *kvm_create_vm(unsigned long type) mutex_unlock(&kvm_lock); preempt_notifier_inc(); + kvm_init_pm_notifier(kvm); return kvm; @@ -1009,6 +1048,7 @@ static void kvm_destroy_vm(struct kvm *kvm) int i; struct mm_struct *mm = kvm->mm; + kvm_destroy_pm_notifier(kvm); kvm_uevent_notify_change(KVM_EVENT_DESTROY_VM, kvm); kvm_destroy_vm_debugfs(kvm); kvm_arch_sync_events(kvm);