From patchwork Mon Mar 15 14:37:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vitaly Kuznetsov X-Patchwork-Id: 12139669 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85C41C41621 for ; Mon, 15 Mar 2021 14:39:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 710D264F1A for ; Mon, 15 Mar 2021 14:39:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233343AbhCOOih (ORCPT ); Mon, 15 Mar 2021 10:38:37 -0400 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:57079 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239526AbhCOOhS (ORCPT ); Mon, 15 Mar 2021 10:37:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1615819037; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=O5Co9d4ozrTngCdGCM1S57GEotFMyYjlIiNeBZ4sqxk=; b=h/y4RDRqbhVbyTehf8d12v62U+fddJGdoRUSxKRFm7qWekwNUpIdyPDTXy26tWVGLgWw3B Z3Rs63n9jLqJF9osvhWFH9pLjrUYgm5hQERoi4/9wxmCYvL0rFQtRQlCbmmckLogNSR1Oc 78NAZZHqFtcuZfmwgXhzpZ4tJivnPxA= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-578-c6DaTcNyO3aTxQeaW05zmw-1; Mon, 15 Mar 2021 10:37:13 -0400 X-MC-Unique: c6DaTcNyO3aTxQeaW05zmw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5199C18460E1; Mon, 15 Mar 2021 14:37:12 +0000 (UTC) Received: from vitty.brq.redhat.com (unknown [10.40.195.229]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3FDAB5C3E6; Mon, 15 Mar 2021 14:37:10 +0000 (UTC) From: Vitaly Kuznetsov To: kvm@vger.kernel.org, Paolo Bonzini Cc: Sean Christopherson , Wanpeng Li , Jim Mattson , Marcelo Tosatti Subject: [PATCH 1/4] KVM: x86: hyper-v: Limit guest to writing zero to HV_X64_MSR_TSC_EMULATION_STATUS Date: Mon, 15 Mar 2021 15:37:03 +0100 Message-Id: <20210315143706.859293-2-vkuznets@redhat.com> In-Reply-To: <20210315143706.859293-1-vkuznets@redhat.com> References: <20210315143706.859293-1-vkuznets@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org HV_X64_MSR_TSC_EMULATION_STATUS indicates whether TSC accesses are emulated after migration (to accommodate for a different host TSC frequency when TSC scaling is not supported; we don't implement this in KVM). Guest can use the same MSR to stop TSC access emulation by writing zero. Writing anything else is forbidden. Signed-off-by: Vitaly Kuznetsov --- arch/x86/kvm/hyperv.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c index 58fa8c029867..eefb85b86fe8 100644 --- a/arch/x86/kvm/hyperv.c +++ b/arch/x86/kvm/hyperv.c @@ -1229,6 +1229,9 @@ static int kvm_hv_set_msr_pw(struct kvm_vcpu *vcpu, u32 msr, u64 data, hv->hv_tsc_emulation_control = data; break; case HV_X64_MSR_TSC_EMULATION_STATUS: + if (data && !host) + return 1; + hv->hv_tsc_emulation_status = data; break; case HV_X64_MSR_TIME_REF_COUNT: From patchwork Mon Mar 15 14:37:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vitaly Kuznetsov X-Patchwork-Id: 12139675 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 064BAC43381 for ; Mon, 15 Mar 2021 14:39:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B674B65002 for ; Mon, 15 Mar 2021 14:39:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232000AbhCOOjM (ORCPT ); Mon, 15 Mar 2021 10:39:12 -0400 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:35494 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240198AbhCOOhV (ORCPT ); Mon, 15 Mar 2021 10:37:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1615819041; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=f4OpybUJz5iJaVvVR75EyFkJKEepA9rPE6zzgFFhznU=; b=GNLfjXPyyOlQg2fK+tHhJUN3bKP0zxcRkshmgXZyewR7XzrdMB56h/D9PHlXnA01VfSZa/ HnOqfIzF2MsAw1/B6Y9wFPEh5T3sX9fiI/h5HytD5N73Kw28JORJ3Qodge8mqG4pQ2ooBv NMIufZq9S4yVfhd3+nJXltMp7NgyuWQ= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-474-ZOjlbchJPKSb5ZP1a_7fpg-1; Mon, 15 Mar 2021 10:37:16 -0400 X-MC-Unique: ZOjlbchJPKSb5ZP1a_7fpg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 99F8218460E6; Mon, 15 Mar 2021 14:37:14 +0000 (UTC) Received: from vitty.brq.redhat.com (unknown [10.40.195.229]) by smtp.corp.redhat.com (Postfix) with ESMTP id B008E5C3E6; Mon, 15 Mar 2021 14:37:12 +0000 (UTC) From: Vitaly Kuznetsov To: kvm@vger.kernel.org, Paolo Bonzini Cc: Sean Christopherson , Wanpeng Li , Jim Mattson , Marcelo Tosatti Subject: [PATCH 2/4] KVM: x86: hyper-v: Prevent using not-yet-updated TSC page by secondary CPUs Date: Mon, 15 Mar 2021 15:37:04 +0100 Message-Id: <20210315143706.859293-3-vkuznets@redhat.com> In-Reply-To: <20210315143706.859293-1-vkuznets@redhat.com> References: <20210315143706.859293-1-vkuznets@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org When KVM_REQ_MASTERCLOCK_UPDATE request is issued (e.g. after migration) we need to make sure no vCPU sees stale values in PV clock structures and thus all vCPUs are kicked with KVM_REQ_CLOCK_UPDATE. Hyper-V TSC page clocksource is global and kvm_guest_time_update() only updates in on vCPU0 but this is not entirely correct: nothing blocks some other vCPU from entering the guest before we finish the update on CPU0 and it can read stale values from the page. Call kvm_hv_setup_tsc_page() on all vCPUs. Normally, KVM_REQ_CLOCK_UPDATE should be very rare so we may not care much about being wasteful. Signed-off-by: Vitaly Kuznetsov --- arch/x86/kvm/x86.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 47e021bdcc94..882c509bfc86 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -2748,8 +2748,9 @@ static int kvm_guest_time_update(struct kvm_vcpu *v) offsetof(struct compat_vcpu_info, time)); if (vcpu->xen.vcpu_time_info_set) kvm_setup_pvclock_page(v, &vcpu->xen.vcpu_time_info_cache, 0); - if (v == kvm_get_vcpu(v->kvm, 0)) - kvm_hv_setup_tsc_page(v->kvm, &vcpu->hv_clock); + + kvm_hv_setup_tsc_page(v->kvm, &vcpu->hv_clock); + return 0; } From patchwork Mon Mar 15 14:37:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vitaly Kuznetsov X-Patchwork-Id: 12139671 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ABFC4C2BA1A for ; Mon, 15 Mar 2021 14:39:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 84BC764E74 for ; Mon, 15 Mar 2021 14:39:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233374AbhCOOij (ORCPT ); Mon, 15 Mar 2021 10:38:39 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:39527 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239821AbhCOOhU (ORCPT ); Mon, 15 Mar 2021 10:37:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1615819040; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=twIrqfzsNtHvqJ7vSwQnUKzlQW5N7NMT68476wDhutM=; b=c1IvyNWorNEUEQhFDYjChroHv63Ekrd31vDJiP6/IaNATGPS0ttq8h42Nk/0DNg7qyjHZr 7qoLf1PcPviF0m5zdNJkGjwn9gKLSdg2Nk4pHIm1EnafEJLINAG4/1zqPEFD7tDUSvuoHl YbMnfV95nGor/UQ0WRXTTT6TeYeicpM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-427-n5mCGrdjOb-lBdbH3LQt-w-1; Mon, 15 Mar 2021 10:37:18 -0400 X-MC-Unique: n5mCGrdjOb-lBdbH3LQt-w-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E6E21108BD08; Mon, 15 Mar 2021 14:37:16 +0000 (UTC) Received: from vitty.brq.redhat.com (unknown [10.40.195.229]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0D3E15C3E6; Mon, 15 Mar 2021 14:37:14 +0000 (UTC) From: Vitaly Kuznetsov To: kvm@vger.kernel.org, Paolo Bonzini Cc: Sean Christopherson , Wanpeng Li , Jim Mattson , Marcelo Tosatti Subject: [PATCH 3/4] KVM: x86: hyper-v: Track Hyper-V TSC page status Date: Mon, 15 Mar 2021 15:37:05 +0100 Message-Id: <20210315143706.859293-4-vkuznets@redhat.com> In-Reply-To: <20210315143706.859293-1-vkuznets@redhat.com> References: <20210315143706.859293-1-vkuznets@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Create an infrastructure for tracking Hyper-V TSC page status, i.e. if it was updated from guest/host side or if we've failed to set it up (because e.g. guest wrote some garbage to HV_X64_MSR_REFERENCE_TSC) and there's no need to retry. Also, in a hypothetical situation when we are in 'always catchup' mode for TSC we can now avoid contending 'hv->hv_lock' on every guest enter by setting the state to HV_TSC_PAGE_BROKEN after compute_tsc_page_parameters() returns false. Signed-off-by: Vitaly Kuznetsov --- arch/x86/include/asm/kvm_host.h | 9 +++++++++ arch/x86/kvm/hyperv.c | 23 +++++++++++++++++------ 2 files changed, 26 insertions(+), 6 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 9bc091ecaaeb..87448e9e6b28 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -884,12 +884,21 @@ struct kvm_hv_syndbg { u64 options; }; +enum hv_tsc_page_status { + HV_TSC_PAGE_UNSET = 0, + HV_TSC_PAGE_GUEST_CHANGED, + HV_TSC_PAGE_HOST_CHANGED, + HV_TSC_PAGE_SET, + HV_TSC_PAGE_BROKEN, +}; + /* Hyper-V emulation context */ struct kvm_hv { struct mutex hv_lock; u64 hv_guest_os_id; u64 hv_hypercall; u64 hv_tsc_page; + enum hv_tsc_page_status hv_tsc_page_status; /* Hyper-v based guest crash (NT kernel bugcheck) parameters */ u64 hv_crash_param[HV_X64_MSR_CRASH_PARAMS]; diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c index eefb85b86fe8..2a8d078b16cb 100644 --- a/arch/x86/kvm/hyperv.c +++ b/arch/x86/kvm/hyperv.c @@ -1087,7 +1087,7 @@ void kvm_hv_setup_tsc_page(struct kvm *kvm, BUILD_BUG_ON(sizeof(tsc_seq) != sizeof(hv->tsc_ref.tsc_sequence)); BUILD_BUG_ON(offsetof(struct ms_hyperv_tsc_page, tsc_sequence) != 0); - if (!(hv->hv_tsc_page & HV_X64_MSR_TSC_REFERENCE_ENABLE)) + if (hv->hv_tsc_page_status == HV_TSC_PAGE_BROKEN) return; mutex_lock(&hv->hv_lock); @@ -1101,7 +1101,7 @@ void kvm_hv_setup_tsc_page(struct kvm *kvm, */ if (unlikely(kvm_read_guest(kvm, gfn_to_gpa(gfn), &tsc_seq, sizeof(tsc_seq)))) - goto out_unlock; + goto out_err; /* * While we're computing and writing the parameters, force the @@ -1110,15 +1110,15 @@ void kvm_hv_setup_tsc_page(struct kvm *kvm, hv->tsc_ref.tsc_sequence = 0; if (kvm_write_guest(kvm, gfn_to_gpa(gfn), &hv->tsc_ref, sizeof(hv->tsc_ref.tsc_sequence))) - goto out_unlock; + goto out_err; if (!compute_tsc_page_parameters(hv_clock, &hv->tsc_ref)) - goto out_unlock; + goto out_err; /* Ensure sequence is zero before writing the rest of the struct. */ smp_wmb(); if (kvm_write_guest(kvm, gfn_to_gpa(gfn), &hv->tsc_ref, sizeof(hv->tsc_ref))) - goto out_unlock; + goto out_err; /* * Now switch to the TSC page mechanism by writing the sequence. @@ -1133,6 +1133,12 @@ void kvm_hv_setup_tsc_page(struct kvm *kvm, hv->tsc_ref.tsc_sequence = tsc_seq; kvm_write_guest(kvm, gfn_to_gpa(gfn), &hv->tsc_ref, sizeof(hv->tsc_ref.tsc_sequence)); + + hv->hv_tsc_page_status = HV_TSC_PAGE_SET; + goto out_unlock; + +out_err: + hv->hv_tsc_page_status = HV_TSC_PAGE_BROKEN; out_unlock: mutex_unlock(&hv->hv_lock); } @@ -1193,8 +1199,13 @@ static int kvm_hv_set_msr_pw(struct kvm_vcpu *vcpu, u32 msr, u64 data, } case HV_X64_MSR_REFERENCE_TSC: hv->hv_tsc_page = data; - if (hv->hv_tsc_page & HV_X64_MSR_TSC_REFERENCE_ENABLE) + if (hv->hv_tsc_page & HV_X64_MSR_TSC_REFERENCE_ENABLE) { + if (!host) + hv->hv_tsc_page_status = HV_TSC_PAGE_GUEST_CHANGED; + else + hv->hv_tsc_page_status = HV_TSC_PAGE_HOST_CHANGED; kvm_make_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu); + } break; case HV_X64_MSR_CRASH_P0 ... HV_X64_MSR_CRASH_P4: return kvm_hv_msr_set_crash_data(kvm, From patchwork Mon Mar 15 14:37:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vitaly Kuznetsov X-Patchwork-Id: 12139673 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE9ABC41519 for ; Mon, 15 Mar 2021 14:39:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9AD4464F17 for ; Mon, 15 Mar 2021 14:39:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233506AbhCOOik (ORCPT ); Mon, 15 Mar 2021 10:38:40 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:27708 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240219AbhCOOhX (ORCPT ); Mon, 15 Mar 2021 10:37:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1615819042; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yF/DRECI57j8pYNPafUBHiPKLqMI/OZXnNRjB907b1Q=; b=cbhfZf4VJiBP6xqGaE2RQj6Fyg9g73ik4lR/nxJMOqHg2he8KVQEJ9MGmv8/gDzjviSwIm hkP6XazuiUn4/RL+l1MCIwFiwju/r81YzIbyM7woCsW8fxqLeaF7EjLSPuuAR9XrfQy8Ae iePVLih0NurRShBgChkg5gxiOQ05Cp4= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-157-m1gV5_plM6-b04MT6-c8vg-1; Mon, 15 Mar 2021 10:37:20 -0400 X-MC-Unique: m1gV5_plM6-b04MT6-c8vg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 48551107ACCD; Mon, 15 Mar 2021 14:37:19 +0000 (UTC) Received: from vitty.brq.redhat.com (unknown [10.40.195.229]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4FFFB5C8B3; Mon, 15 Mar 2021 14:37:17 +0000 (UTC) From: Vitaly Kuznetsov To: kvm@vger.kernel.org, Paolo Bonzini Cc: Sean Christopherson , Wanpeng Li , Jim Mattson , Marcelo Tosatti Subject: [PATCH 4/4] KVM: x86: hyper-v: Don't touch TSC page values when guest opted for re-enlightenment Date: Mon, 15 Mar 2021 15:37:06 +0100 Message-Id: <20210315143706.859293-5-vkuznets@redhat.com> In-Reply-To: <20210315143706.859293-1-vkuznets@redhat.com> References: <20210315143706.859293-1-vkuznets@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org When guest opts for re-enlightenment notifications upon migration, it is in its right to assume that TSC page values never change (as they're only supposed to change upon migration and the host has to keep things as they are before it receives confirmation from the guest). This is mostly true until the guest is migrated somewhere. KVM userspace (e.g. QEMU) will trigger masterclock update by writing to HV_X64_MSR_REFERENCE_TSC, by calling KVM_SET_CLOCK,... and as TSC value and kvmclock reading drift apart (even slightly), the update causes TSC page values to change. The issue at hand is that when Hyper-V is migrated, it uses stale (cached) TSC page values to compute the difference between its own clocksource (provided by KVM) and its guests' TSC pages to program synthetic timers and in some cases, when TSC page is updated, this puts all stimer expirations in the past. This, in its turn, causes an interrupt storm and L2 guests not making much forward progress. Note, KVM doesn't fully implement re-enlightenment notification. Basically, the support for reenlightenment MSRs is just a stub and userspace is only expected to expose the feature when TSC scaling on the expected destination hosts is available. With TSC scaling, no real re-enlightenment is needed as TSC frequency doesn't change. With TSC scaling becoming ubiquitous, it likely makes little sense to fully implement re-enlightenment in KVM. Prevent TSC page from being updated after migration. In case it's not the guest who's initiating the change and when TSC page is already enabled, just keep it as it is: TSC value is supposed to be preserved across migration and TSC frequency can't change with re-enlightenment enabled. The guest is doomed anyway if any of this is not true. Signed-off-by: Vitaly Kuznetsov --- arch/x86/kvm/hyperv.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c index 2a8d078b16cb..5889de580e37 100644 --- a/arch/x86/kvm/hyperv.c +++ b/arch/x86/kvm/hyperv.c @@ -1103,6 +1103,24 @@ void kvm_hv_setup_tsc_page(struct kvm *kvm, &tsc_seq, sizeof(tsc_seq)))) goto out_err; + /* + * Don't touch TSC page values if the guest has opted for TSC emulation + * after migration. KVM doesn't fully support reenlightenment + * notifications and TSC access emulation and Hyper-V is known to expect + * the values in TSC page to stay constant before TSC access emulation + * is disabled from guest side (HV_X64_MSR_TSC_EMULATION_STATUS). + * Userspace is expected to preserve TSC frequency and guest visible TSC + * value across migration (and prevent it when TSC scaling is + * unsupported). + */ + if ((hv->hv_tsc_page_status != HV_TSC_PAGE_GUEST_CHANGED) && + hv->hv_tsc_emulation_control && tsc_seq) { + if (kvm_read_guest(kvm, gfn_to_gpa(gfn), &hv->tsc_ref, sizeof(hv->tsc_ref))) + goto out_err; + + goto out_unlock; + } + /* * While we're computing and writing the parameters, force the * guest to use the time reference count MSR.