From patchwork Thu Apr 15 22:20:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12206183 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90DE1C433B4 for ; Thu, 15 Apr 2021 22:21:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6B625610FC for ; Thu, 15 Apr 2021 22:21:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236712AbhDOWVl (ORCPT ); Thu, 15 Apr 2021 18:21:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57134 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236304AbhDOWVk (ORCPT ); Thu, 15 Apr 2021 18:21:40 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B5902C061574 for ; Thu, 15 Apr 2021 15:21:16 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id w9so3970423ybw.7 for ; Thu, 15 Apr 2021 15:21:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=2rHkFHcKdsKtPlphUZSPjmFsPrJ5FQMml9mM0e4W7Zc=; b=Sdv8k1IdZLOeBzPcIHjmgwJvr30lVQhykx0hvcsxfUl49I6CdmrwLkVbh3hV+Ki+lU 9FgNVGTL6AhKmBPLLJsOtFvFnVAmd9es66AmtmX93zsZySZKpzLCQcstWKqR18p9xu1V A4SAYuvwFCZj+IFvJs55MwqCJ3rSkkS8OincKBNAMhgM9zYj7ywd9/eNQQk8yr2ATmah fXun1sJdo3rL8im+W9tt+twTRmtH/ASLsnxpA84IpCEoOfjOfsRXmg8gMsxVBKugvghq /EQcpFNjuzdNG9gjvw1gOUYvKGfYWCL1jr88Kg0XSNlF0/0+spGD+TtUWO2AwUlYXQst i2TQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=2rHkFHcKdsKtPlphUZSPjmFsPrJ5FQMml9mM0e4W7Zc=; b=Lkb+qO2qVdTFDDx3+avuF5WON5aLBBZiyEtFI/0x7oFd4yRaJsknl7iQ4HvRyfcckJ wcaWTurBBaV9KazHusxkZNHlggplDMq2SE3HbuRVyLnhAeDHNn1xGTwXv73NWWdv4x7Y UwRdugQL3bQBk0u3/F9gx9zP5Ky1uvXDsAsg1r5+zF2cpQ05TuMHODA85JksD1D/VZDu tlVBXCRfAQ1v5XriNBTOYUO++IgRC7qSWYL1USxFVlais5iQdjRBtjQZvToFYOgbcinh ssdZDV3bPNgoREP32KnrNlA7VNEMNbcBIJl03t8mboRyT72U1Qgg8qw+rmwYTe+b4V2t u+yA== X-Gm-Message-State: AOAM533HRGgaGSaCXfNOWq36JqHYDM/PczJvT7G7rxDBj2ZmM7hRglG0 l0Szg8Hf5gz9hyDf9GDw0yO38ymxegw= X-Google-Smtp-Source: ABdhPJxt/mukU9KkDXUa7r3vWCK3xUAeHdvtmdgfysot73ik6ImSoiiXCJDED9AESLevjYuAltjuZtEJn5U= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:6c93:ada0:6bbf:e7db]) (user=seanjc job=sendgmr) by 2002:a25:e6c7:: with SMTP id d190mr6937056ybh.394.1618525275978; Thu, 15 Apr 2021 15:21:15 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 15 Apr 2021 15:20:58 -0700 In-Reply-To: <20210415222106.1643837-1-seanjc@google.com> Message-Id: <20210415222106.1643837-2-seanjc@google.com> Mime-Version: 1.0 References: <20210415222106.1643837-1-seanjc@google.com> X-Mailer: git-send-email 2.31.1.368.gbe11c130af-goog Subject: [PATCH v3 1/9] context_tracking: Move guest exit context tracking to separate helpers From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Thomas Gleixner , Michael Tokarev , Christian Borntraeger Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Wanpeng Li Provide separate context tracking helpers for guest exit, the standalone helpers will be called separately by KVM x86 in later patches to fix tick-based accounting. Suggested-by: Thomas Gleixner Cc: Thomas Gleixner Cc: Sean Christopherson Cc: Michael Tokarev Cc: Christian Borntraeger Signed-off-by: Wanpeng Li Co-developed-by: Sean Christopherson Signed-off-by: Sean Christopherson Reviewed-by: Christian Borntraeger --- include/linux/context_tracking.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/include/linux/context_tracking.h b/include/linux/context_tracking.h index bceb06498521..200d30cb3a82 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -131,10 +131,15 @@ static __always_inline void guest_enter_irqoff(void) } } -static __always_inline void guest_exit_irqoff(void) +static __always_inline void context_tracking_guest_exit_irqoff(void) { if (context_tracking_enabled()) __context_tracking_exit(CONTEXT_GUEST); +} + +static __always_inline void guest_exit_irqoff(void) +{ + context_tracking_guest_exit_irqoff(); instrumentation_begin(); if (vtime_accounting_enabled_this_cpu()) @@ -159,6 +164,8 @@ static __always_inline void guest_enter_irqoff(void) instrumentation_end(); } +static __always_inline void context_tracking_guest_exit_irqoff(void) { } + static __always_inline void guest_exit_irqoff(void) { instrumentation_begin(); From patchwork Thu Apr 15 22:20:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12206185 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C7F2C433ED for ; Thu, 15 Apr 2021 22:21:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0DE2C610FB for ; Thu, 15 Apr 2021 22:21:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237026AbhDOWVp (ORCPT ); Thu, 15 Apr 2021 18:21:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57146 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236808AbhDOWVo (ORCPT ); Thu, 15 Apr 2021 18:21:44 -0400 Received: from mail-qv1-xf4a.google.com (mail-qv1-xf4a.google.com [IPv6:2607:f8b0:4864:20::f4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BBBCDC061574 for ; Thu, 15 Apr 2021 15:21:18 -0700 (PDT) Received: by mail-qv1-xf4a.google.com with SMTP id z17-20020a0cf0110000b029019a84330150so2769018qvk.13 for ; Thu, 15 Apr 2021 15:21:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=CDwOh0IaNcX9UTyCuJ5raB5qoSEoFUMF7dTqu9Of2Ts=; b=lCEbzxTsjVC2q/MVHl7FRpujXgUXcFG8C46zRmpuIcruXU/6VkHbQYSNS+tYU/BApW DHoFCG1brARGrYVwHDCVJfZYQLaWOr4nEHaLOF6dpBzWtcjIxL3lSWY8+PLlRl4Fxafu 5h+9VSGPVxUZjEk+KQy1w1ihTt7dQiu5iiOg6+WH/DBehbWOJ/OJB5xmPlG56fLNSAZH IMjKH6L4HjImCvJSnXXtkkLMp1SvoD3eMpjgzAIeXYeYDIO2FVOjIbgKX0SQWguccYRJ W0lnPa9n+A+yWuhLsRrCB6q9KrAe+fTDw3TXY66O1oyudFe4loCqxDB1cL+NpzVvU9O8 Ao2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=CDwOh0IaNcX9UTyCuJ5raB5qoSEoFUMF7dTqu9Of2Ts=; b=Pivtnv84foal28cyaB3jkrM17atkPl5tbkMKecgVFJmPzC/5ljRD6FU4CPuXLTd0c9 P7XhWMeA2ZCy56anlRPcRFOo2Bj05ALCD8m0I4JirsbuqG6aGgBVTVOV3TlC3Jseb9Ll akKuicnPW2BL96VXo9VLqilRacHxMdFcyAZyIPirp6iYA4yjlz4EKtv/XMcA+xSc3Orq mq1l2A/WrEJf1e6Jf+InelikwZk4QkRNvUFFo8Jq2TnJlRxSRDF2QZeklMhSKullmv87 7JzP5UYl2Rj/PAvmqjEVuJGIty9+OZxUOC/wzjTUOWSScZ800QpcBguO/erf0yvtpOQJ rP/w== X-Gm-Message-State: AOAM532KO8gmwRvxFuN2wP4qayxMcPoT38bNc8OoBfZfKpFopIJ/c0rM wx0HHc1d0WUwgJFFgxDq6N0u+L3ZMAw= X-Google-Smtp-Source: ABdhPJzhT3ylCOgQVx7Kp/AL3/m97MEii6C75poeS9pzETAxvcMrJInW9ZuTQ08c+KJWL/31kAbTYAM0VHA= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:6c93:ada0:6bbf:e7db]) (user=seanjc job=sendgmr) by 2002:a0c:e444:: with SMTP id d4mr5446587qvm.51.1618525277964; Thu, 15 Apr 2021 15:21:17 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 15 Apr 2021 15:20:59 -0700 In-Reply-To: <20210415222106.1643837-1-seanjc@google.com> Message-Id: <20210415222106.1643837-3-seanjc@google.com> Mime-Version: 1.0 References: <20210415222106.1643837-1-seanjc@google.com> X-Mailer: git-send-email 2.31.1.368.gbe11c130af-goog Subject: [PATCH v3 2/9] context_tracking: Move guest exit vtime accounting to separate helpers From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Thomas Gleixner , Michael Tokarev , Christian Borntraeger Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Wanpeng Li Provide separate vtime accounting functions for guest exit instead of open coding the logic within the context tracking code. This will allow KVM x86 to handle vtime accounting slightly differently when using tick-based accounting. Suggested-by: Thomas Gleixner Cc: Thomas Gleixner Cc: Michael Tokarev Cc: Christian Borntraeger Signed-off-by: Wanpeng Li Co-developed-by: Sean Christopherson Signed-off-by: Sean Christopherson Reviewed-by: Christian Borntraeger --- include/linux/context_tracking.h | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/include/linux/context_tracking.h b/include/linux/context_tracking.h index 200d30cb3a82..7cf03a8e5708 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -137,15 +137,20 @@ static __always_inline void context_tracking_guest_exit_irqoff(void) __context_tracking_exit(CONTEXT_GUEST); } -static __always_inline void guest_exit_irqoff(void) +static __always_inline void vtime_account_guest_exit(void) { - context_tracking_guest_exit_irqoff(); - - instrumentation_begin(); if (vtime_accounting_enabled_this_cpu()) vtime_guest_exit(current); else current->flags &= ~PF_VCPU; +} + +static __always_inline void guest_exit_irqoff(void) +{ + context_tracking_guest_exit_irqoff(); + + instrumentation_begin(); + vtime_account_guest_exit(); instrumentation_end(); } @@ -166,12 +171,17 @@ static __always_inline void guest_enter_irqoff(void) static __always_inline void context_tracking_guest_exit_irqoff(void) { } -static __always_inline void guest_exit_irqoff(void) +static __always_inline void vtime_account_guest_exit(void) { - instrumentation_begin(); - /* Flush the guest cputime we spent on the guest */ vtime_account_kernel(current); current->flags &= ~PF_VCPU; +} + +static __always_inline void guest_exit_irqoff(void) +{ + instrumentation_begin(); + /* Flush the guest cputime we spent on the guest */ + vtime_account_guest_exit(); instrumentation_end(); } #endif /* CONFIG_VIRT_CPU_ACCOUNTING_GEN */ From patchwork Thu Apr 15 22:21:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12206187 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 384CFC433ED for ; Thu, 15 Apr 2021 22:21:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1C0BD61131 for ; Thu, 15 Apr 2021 22:21:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237068AbhDOWVt (ORCPT ); Thu, 15 Apr 2021 18:21:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57160 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236800AbhDOWVq (ORCPT ); Thu, 15 Apr 2021 18:21:46 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1C21DC061756 for ; Thu, 15 Apr 2021 15:21:21 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id k199so3963264ybf.0 for ; Thu, 15 Apr 2021 15:21:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=5k2yPQB3tTqToSff4yHkVz7D+OA/2KiSXpBAyhB9+lI=; b=P+X4dRzttCOn1AtwO3Us9bAYEEDl7hRiwPFkhyXs45yQw5GmsnydX6gAcaDzLpHeps IDvv9a7x9KP/VxIoj4nZZhaXocUOljqtWhWzfopnChTUZndQRs/5rwlaLMnZpSDHpp0H 3y3ZkHKKtOUGfLl02dp+/xvDfihLq4xeku0TYPZA6EERlxk+1jf6I75OI4vFgO0qDpyd T7syDagEV6jXbMtbCXTFBNbfkUOoNrdWvkEpVT/XyLRVX5OQJkVqiE3W/MjUzn0bUY7e XT1VcpdvDWYRc0TEDF1n0w9Q4s/vRD3hEBsWzTNRy7GuD5csqfhFnnpKoa8AsKQzfcA5 RXtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=5k2yPQB3tTqToSff4yHkVz7D+OA/2KiSXpBAyhB9+lI=; b=MdeO3u8p0xPc6r0Te2td0mlOqa2Cl9LcXoG5uenem/XyRCN/M5jO6FqzzD5ijAdoXj H74S3PFz5TaqX3I5AnhU4emEyGOX0yDqkUeUlBXIx/yYC8L4iSk86rOWqn6ldzsqmR6r yGR2JAECspu/YwYV2ArYZQthvNR8ZfKsfGwNUeDQrXbKfNeZtwZnYfAyj1zKbgOLMc/g TdLuo4wtKnb0IlSWZfD+ufYnUtavoNIfeEZUktQ33MHe88HW5/EvoEA8cp+4oTZ/bNy/ UnvfUt6CbVMBUC5q1eGB5ZsR8Ws0Wh2hRSU/QrGaEA57ZKao9ymfoVgvocgTC3AuF/Hg NuyQ== X-Gm-Message-State: AOAM532mk00ur+F4dUemnueN96Lq6WOQoe+5CA3E36wQBbp3trKrKDFW 21srHneifsVS9cADVhbrg1VbHdRpt5M= X-Google-Smtp-Source: ABdhPJz1oundGzIq0A2am+wei1yKQWeOWwOHUCr74uXl2hTR4OvZmcPrkn5oxqDnj08dAOy94TF8YMQdc5o= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:6c93:ada0:6bbf:e7db]) (user=seanjc job=sendgmr) by 2002:a25:b983:: with SMTP id r3mr8023430ybg.238.1618525280362; Thu, 15 Apr 2021 15:21:20 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 15 Apr 2021 15:21:00 -0700 In-Reply-To: <20210415222106.1643837-1-seanjc@google.com> Message-Id: <20210415222106.1643837-4-seanjc@google.com> Mime-Version: 1.0 References: <20210415222106.1643837-1-seanjc@google.com> X-Mailer: git-send-email 2.31.1.368.gbe11c130af-goog Subject: [PATCH v3 3/9] KVM: x86: Defer tick-based accounting 'til after IRQ handling From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Thomas Gleixner , Michael Tokarev , Christian Borntraeger Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Wanpeng Li When using tick-based accounting, defer the call to account guest time until after servicing any IRQ(s) that happened in the guest or immediately after VM-Exit. Tick-based accounting of vCPU time relies on PF_VCPU being set when the tick IRQ handler runs, and IRQs are blocked throughout {svm,vmx}_vcpu_enter_exit(). This fixes a bug[*] where reported guest time remains '0', even when running an infinite loop in the guest. [*] https://bugzilla.kernel.org/show_bug.cgi?id=209831 Fixes: 87fa7f3e98a131 ("x86/kvm: Move context tracking where it belongs") Cc: Thomas Gleixner Cc: Sean Christopherson Cc: Michael Tokarev Cc: stable@vger.kernel.org#v5.9-rc1+ Suggested-by: Thomas Gleixner Signed-off-by: Wanpeng Li Co-developed-by: Sean Christopherson Signed-off-by: Sean Christopherson --- arch/x86/kvm/svm/svm.c | 13 ++++++++++--- arch/x86/kvm/vmx/vmx.c | 13 ++++++++++--- arch/x86/kvm/x86.c | 8 ++++++++ 3 files changed, 28 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 48b396f33bee..bb2aa0dde7c5 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -3750,17 +3750,24 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu) * have them in state 'on' as recorded before entering guest mode. * Same as enter_from_user_mode(). * - * guest_exit_irqoff() restores host context and reinstates RCU if - * enabled and required. + * context_tracking_guest_exit_irqoff() restores host context and + * reinstates RCU if enabled and required. * * This needs to be done before the below as native_read_msr() * contains a tracepoint and x86_spec_ctrl_restore_host() calls * into world and some more. */ lockdep_hardirqs_off(CALLER_ADDR0); - guest_exit_irqoff(); + context_tracking_guest_exit_irqoff(); instrumentation_begin(); + /* + * Account guest time when precise accounting based on context tracking + * is enabled. Tick based accounting is deferred until after IRQs that + * occurred within the VM-Enter/VM-Exit "window" are handled. + */ + if (vtime_accounting_enabled_this_cpu()) + vtime_account_guest_exit(); trace_hardirqs_off_finish(); instrumentation_end(); } diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index c05e6e2854b5..5ae9dc197048 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6639,17 +6639,24 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, * have them in state 'on' as recorded before entering guest mode. * Same as enter_from_user_mode(). * - * guest_exit_irqoff() restores host context and reinstates RCU if - * enabled and required. + * context_tracking_guest_exit_irqoff() restores host context and + * reinstates RCU if enabled and required. * * This needs to be done before the below as native_read_msr() * contains a tracepoint and x86_spec_ctrl_restore_host() calls * into world and some more. */ lockdep_hardirqs_off(CALLER_ADDR0); - guest_exit_irqoff(); + context_tracking_guest_exit_irqoff(); instrumentation_begin(); + /* + * Account guest time when precise accounting based on context tracking + * is enabled. Tick based accounting is deferred until after IRQs that + * occurred within the VM-Enter/VM-Exit "window" are handled. + */ + if (vtime_accounting_enabled_this_cpu()) + vtime_account_guest_exit(); trace_hardirqs_off_finish(); instrumentation_end(); } diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 16fb39503296..e4d475df1d4a 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -9230,6 +9230,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) local_irq_disable(); kvm_after_interrupt(vcpu); + /* + * When using tick-based accounting, wait until after servicing IRQs to + * account guest time so that any ticks that occurred while running the + * guest are properly accounted to the guest. + */ + if (!vtime_accounting_enabled_this_cpu()) + vtime_account_guest_exit(); + if (lapic_in_kernel(vcpu)) { s64 delta = vcpu->arch.apic->lapic_timer.advance_expire_delta; if (delta != S64_MIN) { From patchwork Thu Apr 15 22:21:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12206189 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57F58C433B4 for ; Thu, 15 Apr 2021 22:21:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3D21E610FB for ; Thu, 15 Apr 2021 22:21:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237334AbhDOWVw (ORCPT ); Thu, 15 Apr 2021 18:21:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57174 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237001AbhDOWVq (ORCPT ); Thu, 15 Apr 2021 18:21:46 -0400 Received: from mail-qt1-x849.google.com (mail-qt1-x849.google.com [IPv6:2607:f8b0:4864:20::849]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 51FD2C061574 for ; Thu, 15 Apr 2021 15:21:23 -0700 (PDT) Received: by mail-qt1-x849.google.com with SMTP id i9-20020ac85e490000b02901b186fa5716so4931801qtx.22 for ; Thu, 15 Apr 2021 15:21:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=ejNz6/mRycQ/8a4gZzvJd7025j920aJ4l8YEJ/7oL7w=; b=ex93+VhLDEnZyrU3fa14LC/l+d4QG+7TX3kkWOj6bc2ImObzfFwIA0H3Ej6OsaJ25/ bXuSwIJLlLOWYitYmKqt6QfIfi4S55+UeHd6P/lXrdZqzvAFESJQrDcATMLmaeFyu/bJ V0IszkhtV/E8vmxCdhqRdF6h9PDRJHpPfOe+QryANb8Fwh5Gi8f8IZF+VpBy7KqNY7qV 3h/Om4MO5KMZyf5Il785WcYyhTzZo/B/Uqxzh3/fzQJYfDdAUpzb50KLrFkzjYHgFWsA rhzCNKhQfbSItqxPNuB+B4tUE0G0MBCvwwAUYHlxRPE8XatqhsPf7AjfuNmKVeH5BjxK Xyeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=ejNz6/mRycQ/8a4gZzvJd7025j920aJ4l8YEJ/7oL7w=; b=PnJq/z+wt++K5qSEFuPhBJk4OLLcLgkmYoPCWH7kAboQgc2PJazWv94mhFaTtskDby 69FGGO0h33/9IUPc4PBbUhX1yYpjeBzdKBRDJIdcuLxssGBJbT+eIuN7OZEJMfPYCvQQ BLa8jWbKCQ6+hbsqMbPOzoF9dvMExUrkKKukQesiJaX6U9xqEKTE/kKuZDKXRdzu+PAJ d6SfWrlYUtdHXigw2VYJn2mpEcIuOiRotwdeai5HPiKJZ7rdD2C9txLSAIXfhO92ZCp0 y4h1oAIPes4e60vQzsTSRoeYPD2mtSjrCKLQ7lin/nBvh9hs/m6IQ9FaVbGvZ4nRi+Us bscg== X-Gm-Message-State: AOAM530nxsNLE/9sw0dbJDlufgyH+Z6BcYrMX9ZIPtvWKiLyBFgX7eG0 HnVHNllUNNOHo6ag0hefQPRv/9jumH4= X-Google-Smtp-Source: ABdhPJzf+ldPvj6Fg4gh6aAz/cRIuHFFV6MjRkz9wisPDwSyKPmpjYPOqJcN0ic3Jp8qnjnXL7/Vcfx6ldE= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:6c93:ada0:6bbf:e7db]) (user=seanjc job=sendgmr) by 2002:ad4:4f84:: with SMTP id em4mr5435608qvb.26.1618525282494; Thu, 15 Apr 2021 15:21:22 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 15 Apr 2021 15:21:01 -0700 In-Reply-To: <20210415222106.1643837-1-seanjc@google.com> Message-Id: <20210415222106.1643837-5-seanjc@google.com> Mime-Version: 1.0 References: <20210415222106.1643837-1-seanjc@google.com> X-Mailer: git-send-email 2.31.1.368.gbe11c130af-goog Subject: [PATCH v3 4/9] sched/vtime: Move vtime accounting external declarations above inlines From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Thomas Gleixner , Michael Tokarev , Christian Borntraeger Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Move the blob of external declarations (and their stubs) above the set of inline definitions (and their stubs) for vtime accounting. This will allow a future patch to bring in more inline definitions without also having to shuffle large chunks of code. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Christian Borntraeger --- include/linux/vtime.h | 94 +++++++++++++++++++++---------------------- 1 file changed, 47 insertions(+), 47 deletions(-) diff --git a/include/linux/vtime.h b/include/linux/vtime.h index 041d6524d144..6a4317560539 100644 --- a/include/linux/vtime.h +++ b/include/linux/vtime.h @@ -10,53 +10,6 @@ struct task_struct; -/* - * vtime_accounting_enabled_this_cpu() definitions/declarations - */ -#if defined(CONFIG_VIRT_CPU_ACCOUNTING_NATIVE) - -static inline bool vtime_accounting_enabled_this_cpu(void) { return true; } -extern void vtime_task_switch(struct task_struct *prev); - -#elif defined(CONFIG_VIRT_CPU_ACCOUNTING_GEN) - -/* - * Checks if vtime is enabled on some CPU. Cputime readers want to be careful - * in that case and compute the tickless cputime. - * For now vtime state is tied to context tracking. We might want to decouple - * those later if necessary. - */ -static inline bool vtime_accounting_enabled(void) -{ - return context_tracking_enabled(); -} - -static inline bool vtime_accounting_enabled_cpu(int cpu) -{ - return context_tracking_enabled_cpu(cpu); -} - -static inline bool vtime_accounting_enabled_this_cpu(void) -{ - return context_tracking_enabled_this_cpu(); -} - -extern void vtime_task_switch_generic(struct task_struct *prev); - -static inline void vtime_task_switch(struct task_struct *prev) -{ - if (vtime_accounting_enabled_this_cpu()) - vtime_task_switch_generic(prev); -} - -#else /* !CONFIG_VIRT_CPU_ACCOUNTING */ - -static inline bool vtime_accounting_enabled_cpu(int cpu) {return false; } -static inline bool vtime_accounting_enabled_this_cpu(void) { return false; } -static inline void vtime_task_switch(struct task_struct *prev) { } - -#endif - /* * Common vtime APIs */ @@ -94,6 +47,53 @@ static inline void vtime_account_hardirq(struct task_struct *tsk) { } static inline void vtime_flush(struct task_struct *tsk) { } #endif +/* + * vtime_accounting_enabled_this_cpu() definitions/declarations + */ +#if defined(CONFIG_VIRT_CPU_ACCOUNTING_NATIVE) + +static inline bool vtime_accounting_enabled_this_cpu(void) { return true; } +extern void vtime_task_switch(struct task_struct *prev); + +#elif defined(CONFIG_VIRT_CPU_ACCOUNTING_GEN) + +/* + * Checks if vtime is enabled on some CPU. Cputime readers want to be careful + * in that case and compute the tickless cputime. + * For now vtime state is tied to context tracking. We might want to decouple + * those later if necessary. + */ +static inline bool vtime_accounting_enabled(void) +{ + return context_tracking_enabled(); +} + +static inline bool vtime_accounting_enabled_cpu(int cpu) +{ + return context_tracking_enabled_cpu(cpu); +} + +static inline bool vtime_accounting_enabled_this_cpu(void) +{ + return context_tracking_enabled_this_cpu(); +} + +extern void vtime_task_switch_generic(struct task_struct *prev); + +static inline void vtime_task_switch(struct task_struct *prev) +{ + if (vtime_accounting_enabled_this_cpu()) + vtime_task_switch_generic(prev); +} + +#else /* !CONFIG_VIRT_CPU_ACCOUNTING */ + +static inline bool vtime_accounting_enabled_cpu(int cpu) {return false; } +static inline bool vtime_accounting_enabled_this_cpu(void) { return false; } +static inline void vtime_task_switch(struct task_struct *prev) { } + +#endif + #ifdef CONFIG_IRQ_TIME_ACCOUNTING extern void irqtime_account_irq(struct task_struct *tsk, unsigned int offset); From patchwork Thu Apr 15 22:21:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12206191 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88C5CC433ED for ; Thu, 15 Apr 2021 22:21:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6B4B3610F7 for ; Thu, 15 Apr 2021 22:21:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237429AbhDOWWA (ORCPT ); Thu, 15 Apr 2021 18:22:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57204 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237344AbhDOWVx (ORCPT ); Thu, 15 Apr 2021 18:21:53 -0400 Received: from mail-qk1-x74a.google.com (mail-qk1-x74a.google.com [IPv6:2607:f8b0:4864:20::74a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A80FFC06138C for ; Thu, 15 Apr 2021 15:21:25 -0700 (PDT) Received: by mail-qk1-x74a.google.com with SMTP id c129so2502201qkd.6 for ; Thu, 15 Apr 2021 15:21:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=NoxzXKy25RgDTK+Npf4Da6Ion6Bw41f/CgEilQohoV8=; b=ll1uAXpdlaV5cnuGtI1njjjBfpvjpK1FIrl1mcXZ1H8XGuPHtg54vXM+qgJMX8P1Aj jwUrwlz8ahLHRGKNCVY7XjKoSsRrIw6QGWlawj3LwAxzApEwedpflNktBtFSkE1JSQH3 AzrtTK51T6VDg6d3DHQY8PvFUt83f+AgHKkgau8a4oceLdqJrVyCo5CyLaRhPVcr0l/2 ebF/GiwocYNSo2EIVlzcTogFFS1Dh52D2z6JE80JdHvBvO0T+ysc/jypfzdCePJ1Jc9s MWsXLmJ0oXSrZazyF2M4XEDdjxSqMlDXVjUvZ6U602pCgB5jygCa8BoI1q4FcDKYU/Bp +RoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=NoxzXKy25RgDTK+Npf4Da6Ion6Bw41f/CgEilQohoV8=; b=d9yiZrqlH3UyndPkS5lRnr6ExS3eMeFnIF5pK2XSYvRl2UeWX3A3ZrmKeKqa3cQtuZ jreTfgVFLotWQDZJ2+P75FyAxpgo1lkCYkR37sNwdcFCbBZVXexCJrprZy5CItulv9Yp Ct8uGxcsa8E1hpBDAmHe6hgI6gfPyzGk0ecVUY54ytPMxulDjSkrRzLzCyIqUeIOngtL oo9D5YlNIwNCxDxCyqK6I6a5Ls/BWV5QCPQys7bwKC4N8sUVARySRJRzlK08NbY7E28h 5MnrnXAMe5X7G2stNfLIc8xpIYtRw/tmEVgFjKhZv66Yg7wWWwEBcxQWvsSFkz0PbPSQ 8VwQ== X-Gm-Message-State: AOAM533P7Zie2DMAV5GYd0bJMvGdfdu0+CKi1myfctSyUF+/aVFSZ2nK EKD2EDeRfM0L/G9FvAbkokJetcgKMKY= X-Google-Smtp-Source: ABdhPJyvJhqvaVHDZY4RvEdltK5JFGbOu/AGcbMBQJyXWeAzdbLMT9fvmebhPqJ2/ofqK6F2n8nBxpCaMN0= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:6c93:ada0:6bbf:e7db]) (user=seanjc job=sendgmr) by 2002:a05:6214:176a:: with SMTP id et10mr5598011qvb.23.1618525284837; Thu, 15 Apr 2021 15:21:24 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 15 Apr 2021 15:21:02 -0700 In-Reply-To: <20210415222106.1643837-1-seanjc@google.com> Message-Id: <20210415222106.1643837-6-seanjc@google.com> Mime-Version: 1.0 References: <20210415222106.1643837-1-seanjc@google.com> X-Mailer: git-send-email 2.31.1.368.gbe11c130af-goog Subject: [PATCH v3 5/9] sched/vtime: Move guest enter/exit vtime accounting to vtime.h From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Thomas Gleixner , Michael Tokarev , Christian Borntraeger Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Provide separate helpers for guest enter vtime accounting (in addition to the existing guest exit helpers), and move all vtime accounting helpers to vtime.h where the existing #ifdef infrastructure can be leveraged to better delineate the different types of accounting. This will also allow future cleanups via deduplication of context tracking code. Opportunstically delete the vtime_account_kernel() stub now that all callers are wrapped with CONFIG_VIRT_CPU_ACCOUNTING_NATIVE=y. No functional change intended. Signed-off-by: Sean Christopherson --- include/linux/context_tracking.h | 17 +----------- include/linux/vtime.h | 46 +++++++++++++++++++++++++++----- 2 files changed, 41 insertions(+), 22 deletions(-) diff --git a/include/linux/context_tracking.h b/include/linux/context_tracking.h index 7cf03a8e5708..1c05035396ad 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -137,14 +137,6 @@ static __always_inline void context_tracking_guest_exit_irqoff(void) __context_tracking_exit(CONTEXT_GUEST); } -static __always_inline void vtime_account_guest_exit(void) -{ - if (vtime_accounting_enabled_this_cpu()) - vtime_guest_exit(current); - else - current->flags &= ~PF_VCPU; -} - static __always_inline void guest_exit_irqoff(void) { context_tracking_guest_exit_irqoff(); @@ -163,20 +155,13 @@ static __always_inline void guest_enter_irqoff(void) * to flush. */ instrumentation_begin(); - vtime_account_kernel(current); - current->flags |= PF_VCPU; + vtime_account_guest_enter(); rcu_virt_note_context_switch(smp_processor_id()); instrumentation_end(); } static __always_inline void context_tracking_guest_exit_irqoff(void) { } -static __always_inline void vtime_account_guest_exit(void) -{ - vtime_account_kernel(current); - current->flags &= ~PF_VCPU; -} - static __always_inline void guest_exit_irqoff(void) { instrumentation_begin(); diff --git a/include/linux/vtime.h b/include/linux/vtime.h index 6a4317560539..3684487d01e1 100644 --- a/include/linux/vtime.h +++ b/include/linux/vtime.h @@ -3,21 +3,18 @@ #define _LINUX_KERNEL_VTIME_H #include +#include + #ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE #include #endif - -struct task_struct; - /* * Common vtime APIs */ #ifdef CONFIG_VIRT_CPU_ACCOUNTING extern void vtime_account_kernel(struct task_struct *tsk); extern void vtime_account_idle(struct task_struct *tsk); -#else /* !CONFIG_VIRT_CPU_ACCOUNTING */ -static inline void vtime_account_kernel(struct task_struct *tsk) { } #endif /* !CONFIG_VIRT_CPU_ACCOUNTING */ #ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN @@ -55,6 +52,18 @@ static inline void vtime_flush(struct task_struct *tsk) { } static inline bool vtime_accounting_enabled_this_cpu(void) { return true; } extern void vtime_task_switch(struct task_struct *prev); +static __always_inline void vtime_account_guest_enter(void) +{ + vtime_account_kernel(current); + current->flags |= PF_VCPU; +} + +static __always_inline void vtime_account_guest_exit(void) +{ + vtime_account_kernel(current); + current->flags &= ~PF_VCPU; +} + #elif defined(CONFIG_VIRT_CPU_ACCOUNTING_GEN) /* @@ -86,12 +95,37 @@ static inline void vtime_task_switch(struct task_struct *prev) vtime_task_switch_generic(prev); } +static __always_inline void vtime_account_guest_enter(void) +{ + if (vtime_accounting_enabled_this_cpu()) + vtime_guest_enter(current); + else + current->flags |= PF_VCPU; +} + +static __always_inline void vtime_account_guest_exit(void) +{ + if (vtime_accounting_enabled_this_cpu()) + vtime_guest_exit(current); + else + current->flags &= ~PF_VCPU; +} + #else /* !CONFIG_VIRT_CPU_ACCOUNTING */ -static inline bool vtime_accounting_enabled_cpu(int cpu) {return false; } static inline bool vtime_accounting_enabled_this_cpu(void) { return false; } static inline void vtime_task_switch(struct task_struct *prev) { } +static __always_inline void vtime_account_guest_enter(void) +{ + current->flags |= PF_VCPU; +} + +static __always_inline void vtime_account_guest_exit(void) +{ + current->flags &= ~PF_VCPU; +} + #endif From patchwork Thu Apr 15 22:21:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12206193 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1851C43460 for ; Thu, 15 Apr 2021 22:21:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D40B261131 for ; Thu, 15 Apr 2021 22:21:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237480AbhDOWWC (ORCPT ); Thu, 15 Apr 2021 18:22:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57206 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237356AbhDOWVx (ORCPT ); Thu, 15 Apr 2021 18:21:53 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DDF71C06138E for ; Thu, 15 Apr 2021 15:21:27 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id z39so3898511ybh.23 for ; Thu, 15 Apr 2021 15:21:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=GxVp+ryze51SPBwC3MTXA+xy9qBEqXXT0EKwo/t1PP8=; b=FfHMOfr4Q3Myap3t2mwmp1oRd7b57KdkfdzZDDLFiAyG4mWLRZ4cIjmr7JpMPhg9Ad TyAQWEXgFiD3k2RYMGxRkZ3OatJQwIH9Grt6VHWA1kkb9kI/sz3dUMFv8Mv0JDgN3pUk ZHZMOXha6V3X5iMGJxDeNzgP5e4nomVqXO5PMpAQyH4uOaQTk5BgS877mSycz9vH+XcA 0vjLrFAKPizs/JHbpGX2IENWOJMGz5vAp8DD3ywh2xKBE3/VlV8KorPLiDUkQdEKqZ9s KDyltpcE81/KElbpgmWkVMT8u01t4/p1/4fbBHUFENaSsgywO4IPTGoqM2d7g4riOxFM zytQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=GxVp+ryze51SPBwC3MTXA+xy9qBEqXXT0EKwo/t1PP8=; b=UU3IC/Mc0kE8ptM6ogWzThEss7OjdER38m7oLu3N7Mt177td6F9CucnHN6Giymvvlb gp/v0ktyRfjw6qmFAeYuS8gc1XhQ/C3/J6S/l52zZRNcJ7vXgot+N1X380/UFKcuwDvi /2bHVzoVXr9IfoB9tFLLjWq9D+s1Ffvyrtnmi0d9O+IufJFr1QPunYj0ZJy0XY9oFe/6 56Ks1UAdcOyhuc3GR+l8y5jzlU1YJ29pU7NWNmGCd2ggMHKDyZ7rz9gSCasKb372MkTg e+72xZgZSSyRLWRzfOlR56ijjfvPxMyUsA01+0k+weABNE72LzH5OyDQZpxNwopUR95Z zqDw== X-Gm-Message-State: AOAM531X8eu4FNu4TGOH8JffDEvBCdVYpcPjr5r9Z6D+RvYvCf0pdQ4F 15VPmg0Cft2/qlemoAR4TjjL91++BAw= X-Google-Smtp-Source: ABdhPJy13lXczJDkYzHt9Br8KeDanZBPTQtGMChtFt9J9YcQr5yvscDjY6/LGh4Ti4U7Y9aMDQCI1NIGrnM= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:6c93:ada0:6bbf:e7db]) (user=seanjc job=sendgmr) by 2002:a25:818f:: with SMTP id p15mr7234120ybk.135.1618525287170; Thu, 15 Apr 2021 15:21:27 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 15 Apr 2021 15:21:03 -0700 In-Reply-To: <20210415222106.1643837-1-seanjc@google.com> Message-Id: <20210415222106.1643837-7-seanjc@google.com> Mime-Version: 1.0 References: <20210415222106.1643837-1-seanjc@google.com> X-Mailer: git-send-email 2.31.1.368.gbe11c130af-goog Subject: [PATCH v3 6/9] context_tracking: Consolidate guest enter/exit wrappers From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Thomas Gleixner , Michael Tokarev , Christian Borntraeger Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Consolidate the guest enter/exit wrappers, providing and tweaking stubs as needed. This will allow moving the wrappers under KVM without having to bleed #ifdefs into the soon-to-be KVM code. No functional change intended. Signed-off-by: Sean Christopherson --- include/linux/context_tracking.h | 65 ++++++++++++-------------------- 1 file changed, 24 insertions(+), 41 deletions(-) diff --git a/include/linux/context_tracking.h b/include/linux/context_tracking.h index 1c05035396ad..e172a547b2d0 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -71,6 +71,19 @@ static inline void exception_exit(enum ctx_state prev_ctx) } } +static __always_inline bool context_tracking_guest_enter_irqoff(void) +{ + if (context_tracking_enabled()) + __context_tracking_enter(CONTEXT_GUEST); + + return context_tracking_enabled_this_cpu(); +} + +static __always_inline void context_tracking_guest_exit_irqoff(void) +{ + if (context_tracking_enabled()) + __context_tracking_exit(CONTEXT_GUEST); +} /** * ct_state() - return the current context tracking state if known @@ -92,6 +105,9 @@ static inline void user_exit_irqoff(void) { } static inline enum ctx_state exception_enter(void) { return 0; } static inline void exception_exit(enum ctx_state prev_ctx) { } static inline enum ctx_state ct_state(void) { return CONTEXT_DISABLED; } +static inline bool context_tracking_guest_enter_irqoff(void) { return false; } +static inline void context_tracking_guest_exit_irqoff(void) { } + #endif /* !CONFIG_CONTEXT_TRACKING */ #define CT_WARN_ON(cond) WARN_ON(context_tracking_enabled() && (cond)) @@ -102,74 +118,41 @@ extern void context_tracking_init(void); static inline void context_tracking_init(void) { } #endif /* CONFIG_CONTEXT_TRACKING_FORCE */ - -#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN /* must be called with irqs disabled */ static __always_inline void guest_enter_irqoff(void) { + /* + * This is running in ioctl context so its safe to assume that it's the + * stime pending cputime to flush. + */ instrumentation_begin(); - if (vtime_accounting_enabled_this_cpu()) - vtime_guest_enter(current); - else - current->flags |= PF_VCPU; + vtime_account_guest_enter(); instrumentation_end(); - if (context_tracking_enabled()) - __context_tracking_enter(CONTEXT_GUEST); - - /* KVM does not hold any references to rcu protected data when it + /* + * KVM does not hold any references to rcu protected data when it * switches CPU into a guest mode. In fact switching to a guest mode * is very similar to exiting to userspace from rcu point of view. In * addition CPU may stay in a guest mode for quite a long time (up to * one time slice). Lets treat guest mode as quiescent state, just like * we do with user-mode execution. */ - if (!context_tracking_enabled_this_cpu()) { + if (!context_tracking_guest_enter_irqoff()) { instrumentation_begin(); rcu_virt_note_context_switch(smp_processor_id()); instrumentation_end(); } } -static __always_inline void context_tracking_guest_exit_irqoff(void) -{ - if (context_tracking_enabled()) - __context_tracking_exit(CONTEXT_GUEST); -} - static __always_inline void guest_exit_irqoff(void) { context_tracking_guest_exit_irqoff(); - instrumentation_begin(); - vtime_account_guest_exit(); - instrumentation_end(); -} - -#else -static __always_inline void guest_enter_irqoff(void) -{ - /* - * This is running in ioctl context so its safe - * to assume that it's the stime pending cputime - * to flush. - */ - instrumentation_begin(); - vtime_account_guest_enter(); - rcu_virt_note_context_switch(smp_processor_id()); - instrumentation_end(); -} - -static __always_inline void context_tracking_guest_exit_irqoff(void) { } - -static __always_inline void guest_exit_irqoff(void) -{ instrumentation_begin(); /* Flush the guest cputime we spent on the guest */ vtime_account_guest_exit(); instrumentation_end(); } -#endif /* CONFIG_VIRT_CPU_ACCOUNTING_GEN */ static inline void guest_exit(void) { From patchwork Thu Apr 15 22:21:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12206195 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9896C43461 for ; Thu, 15 Apr 2021 22:21:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A08DB61132 for ; Thu, 15 Apr 2021 22:21:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237521AbhDOWWD (ORCPT ); Thu, 15 Apr 2021 18:22:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57210 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237081AbhDOWVx (ORCPT ); Thu, 15 Apr 2021 18:21:53 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4FDDFC061756 for ; Thu, 15 Apr 2021 15:21:30 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id c4so3972755ybp.6 for ; Thu, 15 Apr 2021 15:21:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=+Q7N2cYAdSmHRUValmwElnJxz6XY8f0FCfRpSDgKNLE=; b=eKlmba/fklhyQO8ifozDbg41fKwUqk0wTnP7HmwHaFZzjpnne81CqI7rWXtb0p2jwi ql36e6v8p+NondHs3HGfguWrWObcA/hD6fjQwYn8ZAxfo1hZsfBf9BKnKuCHLpse16sz Q4oyko9ZUBY1CLMZscwAYlzHF/LOKGQapa1rNireRLUpK653l3cPbr8U4HP6U3ne62Mm hj82i08I88bJ7c3p2+IvFpL5fEqVNw9ya34NY2XwdBc7vz4Fiu0K0c2LXJ5dDHZCFCup 3w6g3ejgsaKCvyEuvKmJFk05p2DQBNEQkK3GPdT/+gGkrkuOxNVRksT8I8CVkp2Qc8Hb qRZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=+Q7N2cYAdSmHRUValmwElnJxz6XY8f0FCfRpSDgKNLE=; b=roPlqnJlVMQEugRHQZmF92ciJ9xOpKOfOxxI5uqqtLvQ3y3On+SCGHqZyKLhhgVa9W 2VORccOq26l8M5gvO1bX/2Ct/nU2M7HklYPm/5fKPy6HDGFxkfh3QKgJN/Is5Gv5aLzn C3pLapeOMCWu3LdlDTrZVoyyqlGx+hqe6fqzCb1er7g5awTkOiKIagSxaHFt1kBu0INN oIsyXW6iG0sRwBsQYqd81Ja2OsGs8Oxj3GkvLxjVPSmX1wGBySF3RE/zXaRVPwe/vi1t RVP3VWUxq1ku6y6fsHeRbjUPGqZu6+8r0jDpcqaicUOb05M/2us+mGy/JvNnuIDJ3Kj9 JTGA== X-Gm-Message-State: AOAM532QkSThHJaqi2tJEX//XK6RwFguiMpVwE1/ufHRHeSxa6ocRBro wRFVqMJtDK1NwSCKqn3lNNQXL571q0Y= X-Google-Smtp-Source: ABdhPJwdfxfknW8XueOKXwvg9xz3GccSdgvEBXBAfRtjv22Xy2oZZAR+QwtxE1rJgrN0vgqKUuB5WFcHiC4= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:6c93:ada0:6bbf:e7db]) (user=seanjc job=sendgmr) by 2002:a25:1905:: with SMTP id 5mr7510706ybz.302.1618525289526; Thu, 15 Apr 2021 15:21:29 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 15 Apr 2021 15:21:04 -0700 In-Reply-To: <20210415222106.1643837-1-seanjc@google.com> Message-Id: <20210415222106.1643837-8-seanjc@google.com> Mime-Version: 1.0 References: <20210415222106.1643837-1-seanjc@google.com> X-Mailer: git-send-email 2.31.1.368.gbe11c130af-goog Subject: [PATCH v3 7/9] context_tracking: KVM: Move guest enter/exit wrappers to KVM's domain From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Thomas Gleixner , Michael Tokarev , Christian Borntraeger Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Move the guest enter/exit wrappers to kvm_host.h so that KVM can manage its context tracking vs. vtime accounting without bleeding too many KVM details into the context tracking code. No functional change intended. Signed-off-by: Sean Christopherson --- include/linux/context_tracking.h | 45 -------------------------------- include/linux/kvm_host.h | 45 ++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 45 deletions(-) diff --git a/include/linux/context_tracking.h b/include/linux/context_tracking.h index e172a547b2d0..d4dc9c4d79aa 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -118,49 +118,4 @@ extern void context_tracking_init(void); static inline void context_tracking_init(void) { } #endif /* CONFIG_CONTEXT_TRACKING_FORCE */ -/* must be called with irqs disabled */ -static __always_inline void guest_enter_irqoff(void) -{ - /* - * This is running in ioctl context so its safe to assume that it's the - * stime pending cputime to flush. - */ - instrumentation_begin(); - vtime_account_guest_enter(); - instrumentation_end(); - - /* - * KVM does not hold any references to rcu protected data when it - * switches CPU into a guest mode. In fact switching to a guest mode - * is very similar to exiting to userspace from rcu point of view. In - * addition CPU may stay in a guest mode for quite a long time (up to - * one time slice). Lets treat guest mode as quiescent state, just like - * we do with user-mode execution. - */ - if (!context_tracking_guest_enter_irqoff()) { - instrumentation_begin(); - rcu_virt_note_context_switch(smp_processor_id()); - instrumentation_end(); - } -} - -static __always_inline void guest_exit_irqoff(void) -{ - context_tracking_guest_exit_irqoff(); - - instrumentation_begin(); - /* Flush the guest cputime we spent on the guest */ - vtime_account_guest_exit(); - instrumentation_end(); -} - -static inline void guest_exit(void) -{ - unsigned long flags; - - local_irq_save(flags); - guest_exit_irqoff(); - local_irq_restore(flags); -} - #endif diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 3b06d12ec37e..444d5f0225cb 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -332,6 +332,51 @@ struct kvm_vcpu { struct kvm_dirty_ring dirty_ring; }; +/* must be called with irqs disabled */ +static __always_inline void guest_enter_irqoff(void) +{ + /* + * This is running in ioctl context so its safe to assume that it's the + * stime pending cputime to flush. + */ + instrumentation_begin(); + vtime_account_guest_enter(); + instrumentation_end(); + + /* + * KVM does not hold any references to rcu protected data when it + * switches CPU into a guest mode. In fact switching to a guest mode + * is very similar to exiting to userspace from rcu point of view. In + * addition CPU may stay in a guest mode for quite a long time (up to + * one time slice). Lets treat guest mode as quiescent state, just like + * we do with user-mode execution. + */ + if (!context_tracking_guest_enter_irqoff()) { + instrumentation_begin(); + rcu_virt_note_context_switch(smp_processor_id()); + instrumentation_end(); + } +} + +static __always_inline void guest_exit_irqoff(void) +{ + context_tracking_guest_exit_irqoff(); + + instrumentation_begin(); + /* Flush the guest cputime we spent on the guest */ + vtime_account_guest_exit(); + instrumentation_end(); +} + +static inline void guest_exit(void) +{ + unsigned long flags; + + local_irq_save(flags); + guest_exit_irqoff(); + local_irq_restore(flags); +} + static inline int kvm_vcpu_exiting_guest_mode(struct kvm_vcpu *vcpu) { /* From patchwork Thu Apr 15 22:21:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12206197 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5B0BC433ED for ; Thu, 15 Apr 2021 22:21:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C1A31610FC for ; Thu, 15 Apr 2021 22:21:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237566AbhDOWWH (ORCPT ); Thu, 15 Apr 2021 18:22:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57204 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236017AbhDOWV6 (ORCPT ); Thu, 15 Apr 2021 18:21:58 -0400 Received: from mail-qt1-x84a.google.com (mail-qt1-x84a.google.com [IPv6:2607:f8b0:4864:20::84a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 85F3BC061574 for ; Thu, 15 Apr 2021 15:21:32 -0700 (PDT) Received: by mail-qt1-x84a.google.com with SMTP id h4-20020ac858440000b029019d657b9f21so4961035qth.9 for ; Thu, 15 Apr 2021 15:21:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=UAC36Umw1y4N2XTzBeWJ62cig8EoZ7urvgIOFbPyNVs=; b=hdBoKyZq93hljglLeFSOkjraFuyeHrftvH1IWOLFybaQggYb39Eo+yBxuHDX7vMCY8 28tqvfYRRtbAeKfKSSeCXch5CIe3N/nwduJM+oCpYz97tWxFJIYJYTf//FC6kmn4js2r JRnGFw72gnQCiGlQudymiLyL6pB03MEjdIRxghqW+xi2N1UcgRFR4SnNd5ot1g1cbQBB x0Kg871VTPiSbOhxn1deKmo9nvQXwCXxBT6swl0sAWqLVjQnObaggeFQanPPkOYiFDn7 9Mx0RIfNwQoA4imUxNHEqpQ8Z8raJjJprLXKEcGYRwGDZ+80dN91NCI1SvkLNLuPjxg8 VHTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=UAC36Umw1y4N2XTzBeWJ62cig8EoZ7urvgIOFbPyNVs=; b=LJimywQcPbJwtgxbQQUANSwCjeRi5Oz09rP0oCckvkd7r/3S5dP9CypD/jz3Q2I5cu O9pDM5E4u4stLsLX9AsBhFd7qEplzVNJCyhCklObI0Yw819oz4VI/HDRuox21jUPkMc+ 8GNob0Voa/yRFBUjMKYAXFGxTApZLmiEZPO6PI0IFuuyt+obdSaBEVSf9J62upm7fKuG lDZfaHPEuqth7x1g2lKhpsNB6ebMFyVC4wjxexXS5YTmHuAHS+Ok9sCs+Tr6+YefC9o4 xcwZpB/ppaPaYe7qsYV03fu8apwiRvJMW70FHIDBQm97MTuw5uvDE1YrOU7EYD1omtZ5 J/Tg== X-Gm-Message-State: AOAM533qHgwTvxTRE/zDt5mqBR348RBIKoOMkKC7voeB99IbVG3ezXuJ zUyn9HpaM3Kq4s7yyZMbAqV7mA54b/M= X-Google-Smtp-Source: ABdhPJwUZIGHXwDCmD9Ec2vGqDNqiOgbzXxuMPLd6Y1HUTcMEki9QsgPmUp+XAYk4F9XG/4mMf5xt6XbAWk= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:6c93:ada0:6bbf:e7db]) (user=seanjc job=sendgmr) by 2002:a05:6214:1c45:: with SMTP id if5mr5551635qvb.48.1618525291699; Thu, 15 Apr 2021 15:21:31 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 15 Apr 2021 15:21:05 -0700 In-Reply-To: <20210415222106.1643837-1-seanjc@google.com> Message-Id: <20210415222106.1643837-9-seanjc@google.com> Mime-Version: 1.0 References: <20210415222106.1643837-1-seanjc@google.com> X-Mailer: git-send-email 2.31.1.368.gbe11c130af-goog Subject: [PATCH v3 8/9] KVM: x86: Consolidate guest enter/exit logic to common helpers From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Thomas Gleixner , Michael Tokarev , Christian Borntraeger Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Move the enter/exit logic in {svm,vmx}_vcpu_enter_exit() to common helpers. Opportunistically update the somewhat stale comment about the updates needing to occur immediately after VM-Exit. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/kvm/svm/svm.c | 46 ++----------------------------------- arch/x86/kvm/vmx/vmx.c | 46 ++----------------------------------- arch/x86/kvm/x86.h | 52 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 56 insertions(+), 88 deletions(-) diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index bb2aa0dde7c5..0677595d07e5 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -3713,25 +3713,7 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu) { struct vcpu_svm *svm = to_svm(vcpu); - /* - * VMENTER enables interrupts (host state), but the kernel state is - * interrupts disabled when this is invoked. Also tell RCU about - * it. This is the same logic as for exit_to_user_mode(). - * - * This ensures that e.g. latency analysis on the host observes - * guest mode as interrupt enabled. - * - * guest_enter_irqoff() informs context tracking about the - * transition to guest mode and if enabled adjusts RCU state - * accordingly. - */ - instrumentation_begin(); - trace_hardirqs_on_prepare(); - lockdep_hardirqs_on_prepare(CALLER_ADDR0); - instrumentation_end(); - - guest_enter_irqoff(); - lockdep_hardirqs_on(CALLER_ADDR0); + kvm_guest_enter_irqoff(); if (sev_es_guest(vcpu->kvm)) { __svm_sev_es_vcpu_run(svm->vmcb_pa); @@ -3745,31 +3727,7 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu) vmload(__sme_page_pa(sd->save_area)); } - /* - * VMEXIT disables interrupts (host state), but tracing and lockdep - * have them in state 'on' as recorded before entering guest mode. - * Same as enter_from_user_mode(). - * - * context_tracking_guest_exit_irqoff() restores host context and - * reinstates RCU if enabled and required. - * - * This needs to be done before the below as native_read_msr() - * contains a tracepoint and x86_spec_ctrl_restore_host() calls - * into world and some more. - */ - lockdep_hardirqs_off(CALLER_ADDR0); - context_tracking_guest_exit_irqoff(); - - instrumentation_begin(); - /* - * Account guest time when precise accounting based on context tracking - * is enabled. Tick based accounting is deferred until after IRQs that - * occurred within the VM-Enter/VM-Exit "window" are handled. - */ - if (vtime_accounting_enabled_this_cpu()) - vtime_account_guest_exit(); - trace_hardirqs_off_finish(); - instrumentation_end(); + kvm_guest_exit_irqoff(); } static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 5ae9dc197048..19b0e25bf598 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6600,25 +6600,7 @@ static fastpath_t vmx_exit_handlers_fastpath(struct kvm_vcpu *vcpu) static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, struct vcpu_vmx *vmx) { - /* - * VMENTER enables interrupts (host state), but the kernel state is - * interrupts disabled when this is invoked. Also tell RCU about - * it. This is the same logic as for exit_to_user_mode(). - * - * This ensures that e.g. latency analysis on the host observes - * guest mode as interrupt enabled. - * - * guest_enter_irqoff() informs context tracking about the - * transition to guest mode and if enabled adjusts RCU state - * accordingly. - */ - instrumentation_begin(); - trace_hardirqs_on_prepare(); - lockdep_hardirqs_on_prepare(CALLER_ADDR0); - instrumentation_end(); - - guest_enter_irqoff(); - lockdep_hardirqs_on(CALLER_ADDR0); + kvm_guest_enter_irqoff(); /* L1D Flush includes CPU buffer clear to mitigate MDS */ if (static_branch_unlikely(&vmx_l1d_should_flush)) @@ -6634,31 +6616,7 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, vcpu->arch.cr2 = native_read_cr2(); - /* - * VMEXIT disables interrupts (host state), but tracing and lockdep - * have them in state 'on' as recorded before entering guest mode. - * Same as enter_from_user_mode(). - * - * context_tracking_guest_exit_irqoff() restores host context and - * reinstates RCU if enabled and required. - * - * This needs to be done before the below as native_read_msr() - * contains a tracepoint and x86_spec_ctrl_restore_host() calls - * into world and some more. - */ - lockdep_hardirqs_off(CALLER_ADDR0); - context_tracking_guest_exit_irqoff(); - - instrumentation_begin(); - /* - * Account guest time when precise accounting based on context tracking - * is enabled. Tick based accounting is deferred until after IRQs that - * occurred within the VM-Enter/VM-Exit "window" are handled. - */ - if (vtime_accounting_enabled_this_cpu()) - vtime_account_guest_exit(); - trace_hardirqs_off_finish(); - instrumentation_end(); + kvm_guest_exit_irqoff(); } static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu) diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index daccf20fbcd5..285953e81777 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -8,6 +8,58 @@ #include "kvm_cache_regs.h" #include "kvm_emulate.h" +static __always_inline void kvm_guest_enter_irqoff(void) +{ + /* + * VMENTER enables interrupts (host state), but the kernel state is + * interrupts disabled when this is invoked. Also tell RCU about + * it. This is the same logic as for exit_to_user_mode(). + * + * This ensures that e.g. latency analysis on the host observes + * guest mode as interrupt enabled. + * + * guest_enter_irqoff() informs context tracking about the + * transition to guest mode and if enabled adjusts RCU state + * accordingly. + */ + instrumentation_begin(); + trace_hardirqs_on_prepare(); + lockdep_hardirqs_on_prepare(CALLER_ADDR0); + instrumentation_end(); + + guest_enter_irqoff(); + lockdep_hardirqs_on(CALLER_ADDR0); +} + +static __always_inline void kvm_guest_exit_irqoff(void) +{ + /* + * VMEXIT disables interrupts (host state), but tracing and lockdep + * have them in state 'on' as recorded before entering guest mode. + * Same as enter_from_user_mode(). + * + * context_tracking_guest_exit_irqoff() restores host context and + * reinstates RCU if enabled and required. + * + * This needs to be done immediately after VM-Exit, before any code + * that might contain tracepoints or call out to the greater world, + * e.g. before x86_spec_ctrl_restore_host(). + */ + lockdep_hardirqs_off(CALLER_ADDR0); + context_tracking_guest_exit_irqoff(); + + instrumentation_begin(); + /* + * Account guest time when precise accounting based on context tracking + * is enabled. Tick based accounting is deferred until after IRQs that + * occurred within the VM-Enter/VM-Exit "window" are handled. + */ + if (vtime_accounting_enabled_this_cpu()) + vtime_account_guest_exit(); + trace_hardirqs_off_finish(); + instrumentation_end(); +} + #define KVM_NESTED_VMENTER_CONSISTENCY_CHECK(consistency_check) \ ({ \ bool failed = (consistency_check); \ From patchwork Thu Apr 15 22:21:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12206199 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FA0AC433ED for ; Thu, 15 Apr 2021 22:21:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3671961107 for ; Thu, 15 Apr 2021 22:21:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237336AbhDOWWO (ORCPT ); Thu, 15 Apr 2021 18:22:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57242 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237404AbhDOWWA (ORCPT ); Thu, 15 Apr 2021 18:22:00 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C0B08C061761 for ; Thu, 15 Apr 2021 15:21:34 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id n13so3907505ybp.14 for ; Thu, 15 Apr 2021 15:21:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=5bZDkTOmF30k3ylXE7mjWEUF6Se+XKGXeTtOG326tls=; b=u8/EHms+P5QHbtTJ6SsWhwk29xFysKHQdf/9xINwhk2Wxp21WIFSsVeUCy33wKtjFy lUmj0nJPEi1n3Oh0A/Bgumm6PBfeRQ6rZ6U/hC52JnbgBrzx6Jwpq61xXK+tFBYn07IA kAc09ehAkcpVLpAu8SPSKStvPEd9Ubx9twCwkgnSJZd30C8OLAntIF86x0Zdvht71NN5 s84XV5pqBHrcqFv2Av1nDsm/Dn5rp/O/+/cSldZguahj7hUC9TWg1fG2Nd5xxJ9MgVCK hmjkzhRlEIteP178WSGuZ8PcLY0tZW6hKtnGHn1ErP7Lmb892CbNGFt4X8R+qosWioiv 3SSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=5bZDkTOmF30k3ylXE7mjWEUF6Se+XKGXeTtOG326tls=; b=WMSf6dOMG04I7O9E0xCj4w1yDxlC3Cjk9KPvaIu7IqkipIr08X3/3lML0HOeo7TpKy wfr23yberxivfwGLGR73lGig4x7TdR7Ef25JCdvaAwOUYPIGd+Y7rM0Eq8aOmiWSrXl2 8NYwHizHKdaQkDBFo8yL5FGaK+UndEt7eyt5Oop8CAZ/Eh8yFpTLQOuyGwVCsw8yzF16 WZ7nv3GKdcWDeI8imnu6gszzo7hKXEKIrrG7W+dxV4mxkEn8UoG7rY87XnJV8OvGUW8l f8hYi4FXhnTZOdoyNOv+93ysy2//OlVY4dITROMp9qKUYpi8kTCuX9ShVkaiGWIVLuag retg== X-Gm-Message-State: AOAM5316lmxm4froxLyNPqcgo9aCT+2uh3+o9oZIrpJQfgtRjYqY9JSt YbSfrYgYxyAzITk+W02DpgkVki1lOXQ= X-Google-Smtp-Source: ABdhPJzkSOmVcTYZbMZ5mFcI1gbLLhi87LjETeJ5cPYgxWeheNNd0DFYGFGeWJlxcCCk0oPTkVNLc+BLJCU= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:6c93:ada0:6bbf:e7db]) (user=seanjc job=sendgmr) by 2002:a25:f80e:: with SMTP id u14mr7738720ybd.428.1618525294052; Thu, 15 Apr 2021 15:21:34 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 15 Apr 2021 15:21:06 -0700 In-Reply-To: <20210415222106.1643837-1-seanjc@google.com> Message-Id: <20210415222106.1643837-10-seanjc@google.com> Mime-Version: 1.0 References: <20210415222106.1643837-1-seanjc@google.com> X-Mailer: git-send-email 2.31.1.368.gbe11c130af-goog Subject: [PATCH v3 9/9] KVM: Move instrumentation-safe annotations for enter/exit to x86 code From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Thomas Gleixner , Michael Tokarev , Christian Borntraeger Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Drop the instrumentation_{begin,end}() annonations from the common KVM guest enter/exit helpers, and massage the x86 code as needed to preserve the necessary annotations. x86 is the only architecture whose transition flow is tagged as noinstr, and more specifically, it is the only architecture for which instrumentation_{begin,end}() can be non-empty. No other architecture supports CONFIG_STACK_VALIDATION=y, and s390 is the only other architecture that support CONFIG_DEBUG_ENTRY=y. For instrumentation annontations to be meaningful, both aformentioned configs must be enabled. Letting x86 deal with the annotations avoids unnecessary nops by squashing back-to-back instrumention-safe sequences. Signed-off-by: Sean Christopherson --- arch/x86/kvm/x86.h | 4 ++-- include/linux/kvm_host.h | 9 +-------- 2 files changed, 3 insertions(+), 10 deletions(-) diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 285953e81777..b17857ac540b 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -25,9 +25,9 @@ static __always_inline void kvm_guest_enter_irqoff(void) instrumentation_begin(); trace_hardirqs_on_prepare(); lockdep_hardirqs_on_prepare(CALLER_ADDR0); - instrumentation_end(); - guest_enter_irqoff(); + instrumentation_end(); + lockdep_hardirqs_on(CALLER_ADDR0); } diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 444d5f0225cb..e5eb64019f47 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -339,9 +339,7 @@ static __always_inline void guest_enter_irqoff(void) * This is running in ioctl context so its safe to assume that it's the * stime pending cputime to flush. */ - instrumentation_begin(); vtime_account_guest_enter(); - instrumentation_end(); /* * KVM does not hold any references to rcu protected data when it @@ -351,21 +349,16 @@ static __always_inline void guest_enter_irqoff(void) * one time slice). Lets treat guest mode as quiescent state, just like * we do with user-mode execution. */ - if (!context_tracking_guest_enter_irqoff()) { - instrumentation_begin(); + if (!context_tracking_guest_enter_irqoff()) rcu_virt_note_context_switch(smp_processor_id()); - instrumentation_end(); - } } static __always_inline void guest_exit_irqoff(void) { context_tracking_guest_exit_irqoff(); - instrumentation_begin(); /* Flush the guest cputime we spent on the guest */ vtime_account_guest_exit(); - instrumentation_end(); } static inline void guest_exit(void)