From patchwork Tue Apr 13 18:29:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12201287 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.2 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A7EBC43460 for ; Tue, 13 Apr 2021 18:29:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CFD06613BD for ; Tue, 13 Apr 2021 18:29:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347764AbhDMSaF (ORCPT ); Tue, 13 Apr 2021 14:30:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55344 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347757AbhDMSaD (ORCPT ); Tue, 13 Apr 2021 14:30:03 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 67938C06175F for ; Tue, 13 Apr 2021 11:29:43 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id v2so14919059ybc.17 for ; Tue, 13 Apr 2021 11:29:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=DBgKpsy5j/P3hgAEc9fw7Y1G0+qqP32d1bF863cdlm4=; b=TMO4lZk/DQbX58V5PUc5wkHjNyjUGqKC+eUZ1OM7WF9rm8l+FfiCAEkg3MCT9Zb6D0 LeRqBSimMGn7bZpyvLInauMaoq8e1Y+aT9xC9RAGn4INHwTzqrhLmO9ddRJ2SrUku42I gqKIBM7ycs/kOaMy9YRNA3fM/OZkGENEIl/vUManqirXHGPWtcDqCQOdO2DDxFC3wxpW 5unGle02ljfxlxLq4PmG4kaYsWUUHGf41jnlz3dNiA4mDb7M/9siWZ4vHWpNraCEMqZG G3XJT646bH/U7BhLzNKJleVm9lm54fCaw4LQmaRd7ntZd9K+k7lmd+QwSjGTCxv+P9NS BWWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=DBgKpsy5j/P3hgAEc9fw7Y1G0+qqP32d1bF863cdlm4=; b=WkS1tFmacZIhaJQY/bvoYqPciiseup/SCNgpflkHUr+30ozKZIr1v2N8I9foJQdtaz iOa+4NHoqzXvDkuw4ovfqwHyIkbuMCkOf/e3DYuVaxeI9fnWr5BQZ4JYfIIOzrDwX6TV o78RT6epdqf0wWfuzqHf3/IBZlPJq5o5FkSAMgxugzc6w82TKiWzE1qUy+1j8s6tHrNJ UHTqgJpkPD3fjNGlBVQ25Za9+PiDtp+35C/N+y69JP3iZOBuRA0OPmqyYsLFVQXsBkOL DeGnlGbA7PS/ybk3gYca6yVnS/WbGPHrsiyvshfv8U4dJQb4rqUV8ygEGYDMFOtssfXt KYJw== X-Gm-Message-State: AOAM530baLja9CS5xuuMqfBR3Q3A2Fa0gvJ7+DpsMrWQUFGab9xEUb5c LQ/ViQt/J5CIR279u5ZfJ6akBRVu8S8= X-Google-Smtp-Source: ABdhPJxyESESndmX5ChjYM7X/7wj39TKAIr9yH8AoLODme0G1B9J1nf+VUlLvnk+RaSnnO2UKE3J/uXgd9I= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:f031:9c1c:56c7:c3bf]) (user=seanjc job=sendgmr) by 2002:a25:ce81:: with SMTP id x123mr17691277ybe.283.1618338582701; Tue, 13 Apr 2021 11:29:42 -0700 (PDT) Reply-To: Sean Christopherson Date: Tue, 13 Apr 2021 11:29:27 -0700 In-Reply-To: <20210413182933.1046389-1-seanjc@google.com> Message-Id: <20210413182933.1046389-2-seanjc@google.com> Mime-Version: 1.0 References: <20210413182933.1046389-1-seanjc@google.com> X-Mailer: git-send-email 2.31.1.295.g9ea45b61b8-goog Subject: [RFC PATCH 1/7] sched/vtime: Move guest enter/exit vtime accounting to separate helpers From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Thomas Gleixner , Christian Borntraeger , Michael Tokarev Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Provide separate helpers for guest enter/exit vtime accounting instead of open coding the logic within the context tracking code. This will allow KVM x86 to handle vtime accounting slightly differently when using tick- based accounting. Opportunstically delete the vtime_account_kernel() stub now that all callers are wrapped with CONFIG_VIRT_CPU_ACCOUNTING_NATIVE=y. No functional change intended. Suggested-by: Thomas Gleixner Cc: Christian Borntraeger Signed-off-by: Sean Christopherson --- include/linux/context_tracking.h | 17 +++--------- include/linux/vtime.h | 45 +++++++++++++++++++++++++++++--- 2 files changed, 45 insertions(+), 17 deletions(-) diff --git a/include/linux/context_tracking.h b/include/linux/context_tracking.h index bceb06498521..58f9a7251d3b 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -102,16 +102,12 @@ extern void context_tracking_init(void); static inline void context_tracking_init(void) { } #endif /* CONFIG_CONTEXT_TRACKING_FORCE */ - #ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN /* must be called with irqs disabled */ static __always_inline void guest_enter_irqoff(void) { instrumentation_begin(); - if (vtime_accounting_enabled_this_cpu()) - vtime_guest_enter(current); - else - current->flags |= PF_VCPU; + vtime_account_guest_enter(); instrumentation_end(); if (context_tracking_enabled()) @@ -137,10 +133,7 @@ static __always_inline void guest_exit_irqoff(void) __context_tracking_exit(CONTEXT_GUEST); instrumentation_begin(); - if (vtime_accounting_enabled_this_cpu()) - vtime_guest_exit(current); - else - current->flags &= ~PF_VCPU; + vtime_account_guest_exit(); instrumentation_end(); } @@ -153,8 +146,7 @@ static __always_inline void guest_enter_irqoff(void) * to flush. */ instrumentation_begin(); - vtime_account_kernel(current); - current->flags |= PF_VCPU; + vtime_account_guest_enter(); rcu_virt_note_context_switch(smp_processor_id()); instrumentation_end(); } @@ -163,8 +155,7 @@ static __always_inline void guest_exit_irqoff(void) { instrumentation_begin(); /* Flush the guest cputime we spent on the guest */ - vtime_account_kernel(current); - current->flags &= ~PF_VCPU; + vtime_account_guest_exit(); instrumentation_end(); } #endif /* CONFIG_VIRT_CPU_ACCOUNTING_GEN */ diff --git a/include/linux/vtime.h b/include/linux/vtime.h index 041d6524d144..f30b472a2201 100644 --- a/include/linux/vtime.h +++ b/include/linux/vtime.h @@ -3,6 +3,8 @@ #define _LINUX_KERNEL_VTIME_H #include +#include + #ifdef CONFIG_VIRT_CPU_ACCOUNTING_NATIVE #include #endif @@ -18,6 +20,17 @@ struct task_struct; static inline bool vtime_accounting_enabled_this_cpu(void) { return true; } extern void vtime_task_switch(struct task_struct *prev); +static __always_inline void vtime_account_guest_enter(void) +{ + vtime_account_kernel(current); + current->flags |= PF_VCPU; +} + +static __always_inline void vtime_account_guest_exit(void) +{ + +} + #elif defined(CONFIG_VIRT_CPU_ACCOUNTING_GEN) /* @@ -49,12 +62,38 @@ static inline void vtime_task_switch(struct task_struct *prev) vtime_task_switch_generic(prev); } +static __always_inline void vtime_account_guest_enter(void) +{ + if (vtime_accounting_enabled_this_cpu()) + vtime_guest_enter(current); + else + current->flags |= PF_VCPU; +} + +static __always_inline void vtime_account_guest_exit(void) +{ + if (vtime_accounting_enabled_this_cpu()) + vtime_guest_exit(current); + else + current->flags &= ~PF_VCPU; +} + + #else /* !CONFIG_VIRT_CPU_ACCOUNTING */ -static inline bool vtime_accounting_enabled_cpu(int cpu) {return false; } static inline bool vtime_accounting_enabled_this_cpu(void) { return false; } static inline void vtime_task_switch(struct task_struct *prev) { } +static __always_inline void vtime_account_guest_enter(void) +{ + current->flags |= PF_VCPU; +} + +static __always_inline void vtime_account_guest_exit(void) +{ + current->flags &= ~PF_VCPU; +} + #endif /* @@ -63,9 +102,7 @@ static inline void vtime_task_switch(struct task_struct *prev) { } #ifdef CONFIG_VIRT_CPU_ACCOUNTING extern void vtime_account_kernel(struct task_struct *tsk); extern void vtime_account_idle(struct task_struct *tsk); -#else /* !CONFIG_VIRT_CPU_ACCOUNTING */ -static inline void vtime_account_kernel(struct task_struct *tsk) { } -#endif /* !CONFIG_VIRT_CPU_ACCOUNTING */ +#endif /* CONFIG_VIRT_CPU_ACCOUNTING */ #ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN extern void arch_vtime_task_switch(struct task_struct *tsk); From patchwork Tue Apr 13 18:29:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12201289 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA448C433B4 for ; Tue, 13 Apr 2021 18:29:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C0BD6613BD for ; Tue, 13 Apr 2021 18:29:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347770AbhDMSaH (ORCPT ); Tue, 13 Apr 2021 14:30:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55354 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347763AbhDMSaF (ORCPT ); Tue, 13 Apr 2021 14:30:05 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 83FFBC06175F for ; Tue, 13 Apr 2021 11:29:45 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id p68so10354843ybg.20 for ; Tue, 13 Apr 2021 11:29:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=3pnXNHyPtgGcg37a0kVWxRztf39QNHtD0Ft3zXb2qAc=; b=lhG/UfcHROVd7145tjdD3BwEBuqi+DUF9UbliOiMwye9gVed6mLHJO2wx//Xr/G++W 5wGxFA6bEExFF6iD59g/Ry0/dLm2bxLqGjt+zvz3jnczKvxYQUhqUJsaAK7ah3PtQz16 LWK+pzC8efhXYd0kQGJQzTTRNcj8AClyrDM0OGsB25R/8c3/lAAunUecBkozode5ZuT8 sN6zeNZrE0BllHAwx1VURW5YbckNh8ovpfh9VLvL9EUFuaHHw8v3zv20iHTprVg0oKGl h2jjSfSNfEnH10NoWbGbzNqXLZvm8LPMZ9UKaQSB+MTQz0qo6nH1xqgRgZHdgXQk3Lac BEcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=3pnXNHyPtgGcg37a0kVWxRztf39QNHtD0Ft3zXb2qAc=; b=ndwLyiPgFxj5reqq9aBv/dDJwgH3j7miZcNmlI85bTHkE3EfYT7v7NTQRj7rC8qj2l WYAGa5R++5/Wl3fuREqBJfsRM9SZlFXaz5usPTqtB7Rcb+F1FJC5nGDueUK+uNbQiK9s m+n9GpGKkrNDMkjMcemCdBWayddY9xviqUX4cSYWB5Ts0xvR6kzsJj54tiNf1SS3B69y XcjlWLlBAurIKsW3yMHpNpwvCeRiyeIa/nfuNE/6WNG9HJEbggydmtRlOEtRI1MJnanH YTR3zc+ESk4U2s5g/kRv0E53m0K27Zk2hs1q0LABUCEPGK9FhCfD9L3LzykEnAT7706s hjvA== X-Gm-Message-State: AOAM530HsGHpp5kURi2CG6lTCOh+Cd1wWuJxpv4IksYoqkwcwr3TN2ko D+zlLWZEkhniflokTQNj8ucN7KiwrN8= X-Google-Smtp-Source: ABdhPJx7JC3LIVuSvYqMH4v9P52mqXsTq132vjyl53+XXdyUDNA/ptFo3YlhSAEZQyBkRDGL4tjVVk7sGAE= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:f031:9c1c:56c7:c3bf]) (user=seanjc job=sendgmr) by 2002:a25:dfd0:: with SMTP id w199mr13235033ybg.92.1618338584829; Tue, 13 Apr 2021 11:29:44 -0700 (PDT) Reply-To: Sean Christopherson Date: Tue, 13 Apr 2021 11:29:28 -0700 In-Reply-To: <20210413182933.1046389-1-seanjc@google.com> Message-Id: <20210413182933.1046389-3-seanjc@google.com> Mime-Version: 1.0 References: <20210413182933.1046389-1-seanjc@google.com> X-Mailer: git-send-email 2.31.1.295.g9ea45b61b8-goog Subject: [RFC PATCH 2/7] context_tracking: Move guest enter/exit logic to standalone helpers From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Thomas Gleixner , Christian Borntraeger , Michael Tokarev Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Move guest enter/exit context tracking to standalone helpers, so that the existing wrappers can be moved under KVM. No functional change intended. Signed-off-by: Sean Christopherson --- include/linux/context_tracking.h | 43 +++++++++++++++++++------------- 1 file changed, 26 insertions(+), 17 deletions(-) diff --git a/include/linux/context_tracking.h b/include/linux/context_tracking.h index 58f9a7251d3b..89a1a5ccb2ab 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -71,6 +71,30 @@ static inline void exception_exit(enum ctx_state prev_ctx) } } +static __always_inline void context_tracking_guest_enter_irqoff(void) +{ + if (context_tracking_enabled()) + __context_tracking_enter(CONTEXT_GUEST); + + /* KVM does not hold any references to rcu protected data when it + * switches CPU into a guest mode. In fact switching to a guest mode + * is very similar to exiting to userspace from rcu point of view. In + * addition CPU may stay in a guest mode for quite a long time (up to + * one time slice). Lets treat guest mode as quiescent state, just like + * we do with user-mode execution. + */ + if (!context_tracking_enabled_this_cpu()) { + instrumentation_begin(); + rcu_virt_note_context_switch(smp_processor_id()); + instrumentation_end(); + } +} + +static __always_inline void context_tracking_guest_exit_irqoff(void) +{ + if (context_tracking_enabled()) + __context_tracking_exit(CONTEXT_GUEST); +} /** * ct_state() - return the current context tracking state if known @@ -110,27 +134,12 @@ static __always_inline void guest_enter_irqoff(void) vtime_account_guest_enter(); instrumentation_end(); - if (context_tracking_enabled()) - __context_tracking_enter(CONTEXT_GUEST); - - /* KVM does not hold any references to rcu protected data when it - * switches CPU into a guest mode. In fact switching to a guest mode - * is very similar to exiting to userspace from rcu point of view. In - * addition CPU may stay in a guest mode for quite a long time (up to - * one time slice). Lets treat guest mode as quiescent state, just like - * we do with user-mode execution. - */ - if (!context_tracking_enabled_this_cpu()) { - instrumentation_begin(); - rcu_virt_note_context_switch(smp_processor_id()); - instrumentation_end(); - } + context_tracking_guest_enter_irqoff(); } static __always_inline void guest_exit_irqoff(void) { - if (context_tracking_enabled()) - __context_tracking_exit(CONTEXT_GUEST); + context_tracking_guest_exit_irqoff(); instrumentation_begin(); vtime_account_guest_exit(); From patchwork Tue Apr 13 18:29:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12201291 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12925C43460 for ; Tue, 13 Apr 2021 18:29:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D624061221 for ; Tue, 13 Apr 2021 18:29:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347780AbhDMSaK (ORCPT ); Tue, 13 Apr 2021 14:30:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55368 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347772AbhDMSaH (ORCPT ); Tue, 13 Apr 2021 14:30:07 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C4B30C061574 for ; Tue, 13 Apr 2021 11:29:47 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id w9so17226821ybw.7 for ; Tue, 13 Apr 2021 11:29:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=lh12whvdVlWojKFGWoJVnOsvOAGi35z7h6pCvooT88w=; b=knvgUSepx0V9t4J0QTZdB70Jcq5D9d1SZitbQ2kN0EAg6qWs/WLC2lpZ42DkPGYslT MY8Gti4dV2x6eJMIdHVadk+quFUQnmAoAruxfsp6Bgde7qCfuEPoWFDqfaDZupsRBu72 lQwX1NoDBW/+mXjIlULknL1Jj5SJJnQ9dwUUgWlH2Jqyr66osR27ecHkzGlas2lCbfWA r12b8l7uRzOu1c5b+NYB0x+sp0/lKfwlK5BW3pqftCtAjrh2q3wcdZ19ByV970ILfVos 4ByrzSL+Ma1wXTwdjChN498/4O9XihMyNTc00i6Ov8Q1b3nszJEkGwXzP0uGHgm3gAgQ 34cQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=lh12whvdVlWojKFGWoJVnOsvOAGi35z7h6pCvooT88w=; b=rsdluXts/jU7iRxNjlrfsMj12D52DpSQFYk4snOiSfZJ/vXdyfO3MrYfYhopb+U2f9 rND9riwsrBuVIJSInkYE+M5nVrueApxlL3H9qveY2gu/W4ibwiMExGjSqfHl2Kj1YvB/ 4aFp3xrogh85nyLZmiTEESgUOcdtFHSz/Tt54MQQtfaYXt3zcoUvVbUFMUtSrd6KifGI AWyx0WgbkQFH516psZHnSp4bA608puFPkRayHa3NA7plzp2R/THHpAvQV6DO//R5d73l h3WWRQhVGm0ZI3G5v2OqPugjfwtfcgFYcW0suBlmvYiVD/E2sLD8/6vwJR+sWxtwnqXR qR8g== X-Gm-Message-State: AOAM533Nj2MFuNbjFGbi3A9M9THH2vcaQ8aZn7gtG/+uO1ikcf9CpnBD NIHEnbEtQ7HkmQS7MJbWdyYwktzxDgE= X-Google-Smtp-Source: ABdhPJwVIzcNRZPsp7++1aMbdNm5bALHvjb8xd9BRghvIDyt/tLXtadRk5JUroV90kUJruVeoh96AZLWekY= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:f031:9c1c:56c7:c3bf]) (user=seanjc job=sendgmr) by 2002:a25:7085:: with SMTP id l127mr3272390ybc.293.1618338587108; Tue, 13 Apr 2021 11:29:47 -0700 (PDT) Reply-To: Sean Christopherson Date: Tue, 13 Apr 2021 11:29:29 -0700 In-Reply-To: <20210413182933.1046389-1-seanjc@google.com> Message-Id: <20210413182933.1046389-4-seanjc@google.com> Mime-Version: 1.0 References: <20210413182933.1046389-1-seanjc@google.com> X-Mailer: git-send-email 2.31.1.295.g9ea45b61b8-goog Subject: [RFC PATCH 3/7] context_tracking: Consolidate guest enter/exit wrappers From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Thomas Gleixner , Christian Borntraeger , Michael Tokarev Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Consolidate the guest enter/exit wrappers by providing stubs for the context tracking helpers as necessary. This will allow moving the wrappers under KVM without having to bleed too many #ifdefs into the soon-to-be KVM code. No functional change intended. Signed-off-by: Sean Christopherson --- include/linux/context_tracking.h | 65 ++++++++++++++------------------ 1 file changed, 29 insertions(+), 36 deletions(-) diff --git a/include/linux/context_tracking.h b/include/linux/context_tracking.h index 89a1a5ccb2ab..ded56aed539a 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -76,18 +76,7 @@ static __always_inline void context_tracking_guest_enter_irqoff(void) if (context_tracking_enabled()) __context_tracking_enter(CONTEXT_GUEST); - /* KVM does not hold any references to rcu protected data when it - * switches CPU into a guest mode. In fact switching to a guest mode - * is very similar to exiting to userspace from rcu point of view. In - * addition CPU may stay in a guest mode for quite a long time (up to - * one time slice). Lets treat guest mode as quiescent state, just like - * we do with user-mode execution. - */ - if (!context_tracking_enabled_this_cpu()) { - instrumentation_begin(); - rcu_virt_note_context_switch(smp_processor_id()); - instrumentation_end(); - } + return context_tracking_enabled_this_cpu(); } static __always_inline void context_tracking_guest_exit_irqoff(void) @@ -116,6 +105,17 @@ static inline void user_exit_irqoff(void) { } static inline enum ctx_state exception_enter(void) { return 0; } static inline void exception_exit(enum ctx_state prev_ctx) { } static inline enum ctx_state ct_state(void) { return CONTEXT_DISABLED; } + +static __always_inline bool context_tracking_guest_enter_irqoff(void) +{ + return false; +} + +static __always_inline void context_tracking_guest_exit_irqoff(void) +{ + +} + #endif /* !CONFIG_CONTEXT_TRACKING */ #define CT_WARN_ON(cond) WARN_ON(context_tracking_enabled() && (cond)) @@ -126,48 +126,41 @@ extern void context_tracking_init(void); static inline void context_tracking_init(void) { } #endif /* CONFIG_CONTEXT_TRACKING_FORCE */ -#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN /* must be called with irqs disabled */ static __always_inline void guest_enter_irqoff(void) { + /* + * This is running in ioctl context so its safe to assume that it's the + * stime pending cputime to flush. + */ instrumentation_begin(); vtime_account_guest_enter(); instrumentation_end(); - context_tracking_guest_enter_irqoff(); + /* + * KVM does not hold any references to rcu protected data when it + * switches CPU into a guest mode. In fact switching to a guest mode + * is very similar to exiting to userspace from rcu point of view. In + * addition CPU may stay in a guest mode for quite a long time (up to + * one time slice). Lets treat guest mode as quiescent state, just like + * we do with user-mode execution. + */ + if (!context_tracking_guest_enter_irqoff()) { + instrumentation_begin(); + rcu_virt_note_context_switch(smp_processor_id()); + instrumentation_end(); + } } static __always_inline void guest_exit_irqoff(void) { context_tracking_guest_exit_irqoff(); - instrumentation_begin(); - vtime_account_guest_exit(); - instrumentation_end(); -} - -#else -static __always_inline void guest_enter_irqoff(void) -{ - /* - * This is running in ioctl context so its safe - * to assume that it's the stime pending cputime - * to flush. - */ - instrumentation_begin(); - vtime_account_guest_enter(); - rcu_virt_note_context_switch(smp_processor_id()); - instrumentation_end(); -} - -static __always_inline void guest_exit_irqoff(void) -{ instrumentation_begin(); /* Flush the guest cputime we spent on the guest */ vtime_account_guest_exit(); instrumentation_end(); } -#endif /* CONFIG_VIRT_CPU_ACCOUNTING_GEN */ static inline void guest_exit(void) { From patchwork Tue Apr 13 18:29:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12201293 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF45CC433B4 for ; Tue, 13 Apr 2021 18:29:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 81575613C0 for ; Tue, 13 Apr 2021 18:29:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347786AbhDMSaL (ORCPT ); Tue, 13 Apr 2021 14:30:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55380 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347761AbhDMSaK (ORCPT ); Tue, 13 Apr 2021 14:30:10 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 010F2C06175F for ; Tue, 13 Apr 2021 11:29:50 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id p68so10355021ybg.20 for ; Tue, 13 Apr 2021 11:29:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=AhK+PdPT0LRbakoGPPXo/hynrvPFe53sqM1A79hFvVA=; b=QsrzaSyiEiTWFirkcNGK9TJrojxXCLaN/e7xs774HbN2WOrLyoP6OKOrPFl7JAHwrP hHX6XDaVdzgeIh7IIKCFi05cekPZrhoMhzvt8bRYabcQIWLn14p08OXNq5kCbDLNjJ5I Al6XeuqqogCeFQS8QQgKSNhjv/2x37QTT381MvINIA2vgK8awxT0DFgu1fqZKdOh4nZ8 nU0w+f3UPdFrn/0ZiMFAfGs6a1D0/OCaxzxY5F2qyPCX7xyt3F9zM4k5zKK1RNFDXQJh s9K859eLgXhDIIr7pDD2oLvAyQTyHsbRssJRDgv4RR21Na376L2i4vksVNoM/J9c27IH f6/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=AhK+PdPT0LRbakoGPPXo/hynrvPFe53sqM1A79hFvVA=; b=XI3tdWHca0RnCYb6uvLh7wdFOq44Th5R52TG09kRctbxktomHm9SjZvLZVP2DFVivG uP0QXAOSBajO4gzFy1ecy4ZEM/ZDDqPs82+wOvyo45eleiAi0QxtooyapruMLvSzP99y yht8N9tayZA/vJdkmIwqctFf/wOqf1QelPiysOgT8oHvI3ueRhIRp5CLQgpICN43J7hR CXTtRTrQP7HwMSls1XRe/uiymxpf8Q47IjwzPxaSP7yrFtNBn8dbpNvoCQe3lff0F7rG RAJ7S01W6BEEMWqgLFXlmMR3J4LFHMfud0BOErRQvUlDrn2phkJr25kwQlZqiHGr5fik tUhA== X-Gm-Message-State: AOAM533UVdkPlSZWro6k7HIixEGYpGuegVg1AiWKTrTk+tYqjxc2RyNP MquGUeHbFaYYHsDl/NwpK60aFhhnmQs= X-Google-Smtp-Source: ABdhPJwvR1+cpyxhL5YKp5jJoNNdVmTT+kVymEazBfx+83t2uMh2pzzpWf4WT350QSUJynWww11fBJSHo7k= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:f031:9c1c:56c7:c3bf]) (user=seanjc job=sendgmr) by 2002:a25:8a83:: with SMTP id h3mr40519051ybl.354.1618338589262; Tue, 13 Apr 2021 11:29:49 -0700 (PDT) Reply-To: Sean Christopherson Date: Tue, 13 Apr 2021 11:29:30 -0700 In-Reply-To: <20210413182933.1046389-1-seanjc@google.com> Message-Id: <20210413182933.1046389-5-seanjc@google.com> Mime-Version: 1.0 References: <20210413182933.1046389-1-seanjc@google.com> X-Mailer: git-send-email 2.31.1.295.g9ea45b61b8-goog Subject: [RFC PATCH 4/7] context_tracking: KVM: Move guest enter/exit wrappers to KVM's domain From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Thomas Gleixner , Christian Borntraeger , Michael Tokarev Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Move the guest enter/exit wrappers to kvm_host.h so that KVM can manage its context tracking vs. vtime accounting without bleeding too many KVM details into the context tracking code. No functional change intended. Signed-off-by: Sean Christopherson --- include/linux/context_tracking.h | 45 -------------------------------- include/linux/kvm_host.h | 45 ++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 45 deletions(-) diff --git a/include/linux/context_tracking.h b/include/linux/context_tracking.h index ded56aed539a..2f4538380a8d 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -126,49 +126,4 @@ extern void context_tracking_init(void); static inline void context_tracking_init(void) { } #endif /* CONFIG_CONTEXT_TRACKING_FORCE */ -/* must be called with irqs disabled */ -static __always_inline void guest_enter_irqoff(void) -{ - /* - * This is running in ioctl context so its safe to assume that it's the - * stime pending cputime to flush. - */ - instrumentation_begin(); - vtime_account_guest_enter(); - instrumentation_end(); - - /* - * KVM does not hold any references to rcu protected data when it - * switches CPU into a guest mode. In fact switching to a guest mode - * is very similar to exiting to userspace from rcu point of view. In - * addition CPU may stay in a guest mode for quite a long time (up to - * one time slice). Lets treat guest mode as quiescent state, just like - * we do with user-mode execution. - */ - if (!context_tracking_guest_enter_irqoff()) { - instrumentation_begin(); - rcu_virt_note_context_switch(smp_processor_id()); - instrumentation_end(); - } -} - -static __always_inline void guest_exit_irqoff(void) -{ - context_tracking_guest_exit_irqoff(); - - instrumentation_begin(); - /* Flush the guest cputime we spent on the guest */ - vtime_account_guest_exit(); - instrumentation_end(); -} - -static inline void guest_exit(void) -{ - unsigned long flags; - - local_irq_save(flags); - guest_exit_irqoff(); - local_irq_restore(flags); -} - #endif diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 3b06d12ec37e..444d5f0225cb 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -332,6 +332,51 @@ struct kvm_vcpu { struct kvm_dirty_ring dirty_ring; }; +/* must be called with irqs disabled */ +static __always_inline void guest_enter_irqoff(void) +{ + /* + * This is running in ioctl context so its safe to assume that it's the + * stime pending cputime to flush. + */ + instrumentation_begin(); + vtime_account_guest_enter(); + instrumentation_end(); + + /* + * KVM does not hold any references to rcu protected data when it + * switches CPU into a guest mode. In fact switching to a guest mode + * is very similar to exiting to userspace from rcu point of view. In + * addition CPU may stay in a guest mode for quite a long time (up to + * one time slice). Lets treat guest mode as quiescent state, just like + * we do with user-mode execution. + */ + if (!context_tracking_guest_enter_irqoff()) { + instrumentation_begin(); + rcu_virt_note_context_switch(smp_processor_id()); + instrumentation_end(); + } +} + +static __always_inline void guest_exit_irqoff(void) +{ + context_tracking_guest_exit_irqoff(); + + instrumentation_begin(); + /* Flush the guest cputime we spent on the guest */ + vtime_account_guest_exit(); + instrumentation_end(); +} + +static inline void guest_exit(void) +{ + unsigned long flags; + + local_irq_save(flags); + guest_exit_irqoff(); + local_irq_restore(flags); +} + static inline int kvm_vcpu_exiting_guest_mode(struct kvm_vcpu *vcpu) { /* From patchwork Tue Apr 13 18:29:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12201295 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53924C433B4 for ; Tue, 13 Apr 2021 18:29:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 24674613C4 for ; Tue, 13 Apr 2021 18:29:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347803AbhDMSaO (ORCPT ); Tue, 13 Apr 2021 14:30:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55390 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347794AbhDMSaN (ORCPT ); Tue, 13 Apr 2021 14:30:13 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1241CC061574 for ; Tue, 13 Apr 2021 11:29:52 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id w9so17227000ybw.7 for ; Tue, 13 Apr 2021 11:29:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=ZH3R+r1mP9A7TFpksm2/wMBCPzsWCgXGqDl2URF8e0c=; b=owXXzC9qI/WTs8JgAJXw2iBbcM6GJRcX2Sj6STWjD9aSRhwAbeEK+txS3UWUFW9By8 dahEF9t8wUHcbNjUdFMLjjiUv0wNPCBwDdkQsDekYzS1SxwnoPqNbrJZ/g2U9p9tOX5u mi9xmwpGek12LGIbxowSGaJR2wMehTEQxaBfeGLgyZp6PltwiSLyqLnHO9UvopYJHVTy JXnt6ayOnj+2lHzYaoN20O68hCIniSd1dEgjGRCYJtfClhnarbcncH5CkoanyyxZMBYt Wdg///Eyt/h0W4i7T1Cl7baBswwDRlpP32QsswS9izaACJSP3jd7onIPTb1bcIv4tJpK i0tA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=ZH3R+r1mP9A7TFpksm2/wMBCPzsWCgXGqDl2URF8e0c=; b=uQW403CObSjcBxiR1Q0G1wTZDBwIHP+JdolUhwLvao1DDBcLlJYTGg78UAOxdB0Hcv RcwIf1KSejiYao2WvQ/oWrhIwxL1ya5CXLOjzbtf8JnFw1tx1HaZ8W4MeTiVd592omK8 Mxw0TuWIXat1Zs1BoKfFf5MuxuVOKJLMhk3ZRtt5xbaO6haGC+lzV05XtB6Mp26luZqR XN8NqyCoZlb+lbzEhrzufscTzqSiHSIlqr3b1sdNlyeg2qxQuU/kR6gTYBoGRdr93NGv Xnto9LoIMD17TIji0tmhROJfe9Xomzw1jPvRN90EF2OfIIuvgbXXyB4ZCC2xpxCbaTmp Z1gQ== X-Gm-Message-State: AOAM533Z6+tkJDRnZ3X0SjyxbYq0ihqBF/7mKh69mrUHxMw5g4T+Ndmk A+ZpNb0xQmITMtE1C4OG/wwKCJs9ybA= X-Google-Smtp-Source: ABdhPJwe5BGu6diD7tVMyJxmAxDs5AV++7eUpQW5aAxTwDILzmzjS+kIRe1a6098uYNdg4gfxqHkdgfqddk= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:f031:9c1c:56c7:c3bf]) (user=seanjc job=sendgmr) by 2002:a25:d70f:: with SMTP id o15mr35167752ybg.403.1618338591313; Tue, 13 Apr 2021 11:29:51 -0700 (PDT) Reply-To: Sean Christopherson Date: Tue, 13 Apr 2021 11:29:31 -0700 In-Reply-To: <20210413182933.1046389-1-seanjc@google.com> Message-Id: <20210413182933.1046389-6-seanjc@google.com> Mime-Version: 1.0 References: <20210413182933.1046389-1-seanjc@google.com> X-Mailer: git-send-email 2.31.1.295.g9ea45b61b8-goog Subject: [RFC PATCH 5/7] KVM: Move vtime accounting of guest exit to separate helper From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Thomas Gleixner , Christian Borntraeger , Michael Tokarev Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Provide a standalone helper for guest exit vtime accounting so that x86 can defer tick-based accounting until the appropriate time, while still updating context tracking immediately after VM-Exit. No functional change intended. Signed-off-by: Sean Christopherson --- include/linux/kvm_host.h | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 444d5f0225cb..20604bfae5a8 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -358,16 +358,21 @@ static __always_inline void guest_enter_irqoff(void) } } -static __always_inline void guest_exit_irqoff(void) +static __always_inline void kvm_vtime_account_guest_exit(void) { - context_tracking_guest_exit_irqoff(); - instrumentation_begin(); /* Flush the guest cputime we spent on the guest */ vtime_account_guest_exit(); instrumentation_end(); } +static __always_inline void guest_exit_irqoff(void) +{ + context_tracking_guest_exit_irqoff(); + + kvm_vtime_account_guest_exit(); +} + static inline void guest_exit(void) { unsigned long flags; From patchwork Tue Apr 13 18:29:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12201297 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FE90C433ED for ; Tue, 13 Apr 2021 18:30:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7080F613BD for ; Tue, 13 Apr 2021 18:30:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347820AbhDMSaT (ORCPT ); Tue, 13 Apr 2021 14:30:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55402 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347801AbhDMSaO (ORCPT ); Tue, 13 Apr 2021 14:30:14 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 729E9C061756 for ; Tue, 13 Apr 2021 11:29:54 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id h69so12519897ybg.10 for ; Tue, 13 Apr 2021 11:29:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=CEYCqY3YmprwzNzHmrb5PG3zL9vhoX2SFLxftmzmH28=; b=TBTjbKnx1EUQ2y9m+ZG68jdyp/Ip7on/lMBwHyHq64SuDLGtIFYP/c06SMP2sfo+k/ NMSncL3PrXNBg+SAsCukB+lMCcmQ4TdgvKEKkQwiMMnHc1dDeTbK7/MjTMvWYsFrZZiM PFGwZ9BYCd6mhnyRFj/9IfStt6nTc6zLfm6qMdze7iTeqxpkwyN4uAucwv83EbrXC6eb CWa3dwqlYu0YwzgARENnwdEfmH9Y4vNeE9bx38RjzRoZtlcApAb2wLwG+rd05QNXZjSj VLIczG8FhfBDPD+OaCG4819gZzccst9RC6pgdqHhMpaLxgCrpd4oUjukUx0e9gxXxt1A a65A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=CEYCqY3YmprwzNzHmrb5PG3zL9vhoX2SFLxftmzmH28=; b=kndUdvBX57fX2MS+z3t0I+980VrFMEBy8dOJisz48wlPpui0Z8J22Da82mzo1AodJc yLZlC5o/E6T6MXY24Bm9HjpeqqpHcDk3LXtD4+FortSTSkiBQQCzCOkfIHO5DT634PpM eWPK+piw4y4zPgl0/zwr2W28W9Dbu/xydCCT4JqdxUVfeKAD9XnpjWGHCuyeuIX6p4tp rXX8oHQj0ov8yMkXZpPCaLbBl3C7xHoZP11jHSELwK7+y8hsWgbkOuHXoGA4uN6KWNt+ 3rOQ6sftNyVRG+t5RJXVd1wfHZiZNwcTxc3l4NEm2LcdjGhtBm9ui+Jes0JeBMjNr3nZ MywA== X-Gm-Message-State: AOAM533A7S1RF5NofV+uh1sOnig9XdoSTdw3Dz+YrjxAy3atsGZysXKd DkgjgFZPY+2CbJVzNscwPOuQeprLqws= X-Google-Smtp-Source: ABdhPJz0378Bfftq2L3C9vl/L1mZmRh3nq5AECtQL95uGhNpglOks/bpp90SbduruRCfzKVnP+wRa9C5PtY= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:f031:9c1c:56c7:c3bf]) (user=seanjc job=sendgmr) by 2002:a5b:452:: with SMTP id s18mr46985082ybp.482.1618338593729; Tue, 13 Apr 2021 11:29:53 -0700 (PDT) Reply-To: Sean Christopherson Date: Tue, 13 Apr 2021 11:29:32 -0700 In-Reply-To: <20210413182933.1046389-1-seanjc@google.com> Message-Id: <20210413182933.1046389-7-seanjc@google.com> Mime-Version: 1.0 References: <20210413182933.1046389-1-seanjc@google.com> X-Mailer: git-send-email 2.31.1.295.g9ea45b61b8-goog Subject: [RFC PATCH 6/7] KVM: x86: Consolidate guest enter/exit logic to common helpers From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Thomas Gleixner , Christian Borntraeger , Michael Tokarev Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Move the enter/exit logic in {svm,vmx}_vcpu_enter_exit() to common helpers. In addition to deduplicating code, this will allow tweaking the vtime accounting in the VM-Exit path without splitting logic across x86, VMX, and SVM. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/kvm/svm/svm.c | 39 ++---------------------------------- arch/x86/kvm/vmx/vmx.c | 39 ++---------------------------------- arch/x86/kvm/x86.h | 45 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 49 insertions(+), 74 deletions(-) diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 48b396f33bee..0677595d07e5 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -3713,25 +3713,7 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu) { struct vcpu_svm *svm = to_svm(vcpu); - /* - * VMENTER enables interrupts (host state), but the kernel state is - * interrupts disabled when this is invoked. Also tell RCU about - * it. This is the same logic as for exit_to_user_mode(). - * - * This ensures that e.g. latency analysis on the host observes - * guest mode as interrupt enabled. - * - * guest_enter_irqoff() informs context tracking about the - * transition to guest mode and if enabled adjusts RCU state - * accordingly. - */ - instrumentation_begin(); - trace_hardirqs_on_prepare(); - lockdep_hardirqs_on_prepare(CALLER_ADDR0); - instrumentation_end(); - - guest_enter_irqoff(); - lockdep_hardirqs_on(CALLER_ADDR0); + kvm_guest_enter_irqoff(); if (sev_es_guest(vcpu->kvm)) { __svm_sev_es_vcpu_run(svm->vmcb_pa); @@ -3745,24 +3727,7 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu) vmload(__sme_page_pa(sd->save_area)); } - /* - * VMEXIT disables interrupts (host state), but tracing and lockdep - * have them in state 'on' as recorded before entering guest mode. - * Same as enter_from_user_mode(). - * - * guest_exit_irqoff() restores host context and reinstates RCU if - * enabled and required. - * - * This needs to be done before the below as native_read_msr() - * contains a tracepoint and x86_spec_ctrl_restore_host() calls - * into world and some more. - */ - lockdep_hardirqs_off(CALLER_ADDR0); - guest_exit_irqoff(); - - instrumentation_begin(); - trace_hardirqs_off_finish(); - instrumentation_end(); + kvm_guest_exit_irqoff(); } static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index c05e6e2854b5..19b0e25bf598 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6600,25 +6600,7 @@ static fastpath_t vmx_exit_handlers_fastpath(struct kvm_vcpu *vcpu) static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, struct vcpu_vmx *vmx) { - /* - * VMENTER enables interrupts (host state), but the kernel state is - * interrupts disabled when this is invoked. Also tell RCU about - * it. This is the same logic as for exit_to_user_mode(). - * - * This ensures that e.g. latency analysis on the host observes - * guest mode as interrupt enabled. - * - * guest_enter_irqoff() informs context tracking about the - * transition to guest mode and if enabled adjusts RCU state - * accordingly. - */ - instrumentation_begin(); - trace_hardirqs_on_prepare(); - lockdep_hardirqs_on_prepare(CALLER_ADDR0); - instrumentation_end(); - - guest_enter_irqoff(); - lockdep_hardirqs_on(CALLER_ADDR0); + kvm_guest_enter_irqoff(); /* L1D Flush includes CPU buffer clear to mitigate MDS */ if (static_branch_unlikely(&vmx_l1d_should_flush)) @@ -6634,24 +6616,7 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, vcpu->arch.cr2 = native_read_cr2(); - /* - * VMEXIT disables interrupts (host state), but tracing and lockdep - * have them in state 'on' as recorded before entering guest mode. - * Same as enter_from_user_mode(). - * - * guest_exit_irqoff() restores host context and reinstates RCU if - * enabled and required. - * - * This needs to be done before the below as native_read_msr() - * contains a tracepoint and x86_spec_ctrl_restore_host() calls - * into world and some more. - */ - lockdep_hardirqs_off(CALLER_ADDR0); - guest_exit_irqoff(); - - instrumentation_begin(); - trace_hardirqs_off_finish(); - instrumentation_end(); + kvm_guest_exit_irqoff(); } static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu) diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index daccf20fbcd5..74ef92f47db8 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -8,6 +8,51 @@ #include "kvm_cache_regs.h" #include "kvm_emulate.h" +static __always_inline void kvm_guest_enter_irqoff(void) +{ + /* + * VMENTER enables interrupts (host state), but the kernel state is + * interrupts disabled when this is invoked. Also tell RCU about + * it. This is the same logic as for exit_to_user_mode(). + * + * This ensures that e.g. latency analysis on the host observes + * guest mode as interrupt enabled. + * + * guest_enter_irqoff() informs context tracking about the + * transition to guest mode and if enabled adjusts RCU state + * accordingly. + */ + instrumentation_begin(); + trace_hardirqs_on_prepare(); + lockdep_hardirqs_on_prepare(CALLER_ADDR0); + instrumentation_end(); + + guest_enter_irqoff(); + lockdep_hardirqs_on(CALLER_ADDR0); +} + +static __always_inline void kvm_guest_exit_irqoff(void) +{ + /* + * VMEXIT disables interrupts (host state), but tracing and lockdep + * have them in state 'on' as recorded before entering guest mode. + * Same as enter_from_user_mode(). + * + * guest_exit_irqoff() restores host context and reinstates RCU if + * enabled and required. + * + * This needs to be done before the below as native_read_msr() + * contains a tracepoint and x86_spec_ctrl_restore_host() calls + * into world and some more. + */ + lockdep_hardirqs_off(CALLER_ADDR0); + guest_exit_irqoff(); + + instrumentation_begin(); + trace_hardirqs_off_finish(); + instrumentation_end(); +} + #define KVM_NESTED_VMENTER_CONSISTENCY_CHECK(consistency_check) \ ({ \ bool failed = (consistency_check); \ From patchwork Tue Apr 13 18:29:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 12201299 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.2 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08B41C433ED for ; Tue, 13 Apr 2021 18:30:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DCA9161221 for ; Tue, 13 Apr 2021 18:30:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347815AbhDMSaZ (ORCPT ); Tue, 13 Apr 2021 14:30:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55418 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347817AbhDMSaR (ORCPT ); Tue, 13 Apr 2021 14:30:17 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B6629C061574 for ; Tue, 13 Apr 2021 11:29:56 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id h69so12519984ybg.10 for ; Tue, 13 Apr 2021 11:29:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=reply-to:date:in-reply-to:message-id:mime-version:references :subject:from:to:cc; bh=ZLBo4g8Ay3j59zTWAeAVK1AEJcKiEUGCOtGCkug1Xm8=; b=AaD3yleh8mz8kgSJJgqPAiZzvg6sryX2ta9uwSnAsRDcOcNyBAbY6KGhgzO1rlhZBk 11s8XkfFEJZlnp8hq6zog+bkCHpC1mn1LqL072WlJoBjJps9nBO25CktotduMpIWW42P +26fZ5B1q4UHwQZ03a8ZOZHEXU/wZ7TS+4UiaRSR4AaacU0iDirwJlg2swGqzp1U5QO4 5i0+2mdrYyoIVyZJgfVttKgwtzIOH8soEck0FTGklOL7GtKvwrVKHnXlGXeh46xptfAQ 2rZgtnvJI3SifCu2tSfdEDQjj0Aqup16XyOP85/CBIB32dYa9W6QT4FyVkW9E/HDUFGG 4kaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:reply-to:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=ZLBo4g8Ay3j59zTWAeAVK1AEJcKiEUGCOtGCkug1Xm8=; b=FI4ltMZ+HnXigBMFM9yr0w06h/Uk89rhr9xAfbobFImZX/MvJt0WUpHoiDBxq1iN0W NbRPGGtA6mEDyCYXVmesyPCqeS99UvXLaVnXdm+4kki9dyAVIb/0Kisfwr2BxKoHHBhE eEuGAhNumoYWIl35nfIObHipalT5u+wsjqmWVscy2SfjXiWx/pWGOC7Xxn76hC1mk4VG D7iDvRis7pXTD+MeO3hMPfT94Wk4Zy0g23htDAxql1T95E0hh3Q9aZ28mqCIPXeVuCHw BbYIbrnxcBPFLoNQ9kZFfHrEWW41ye5hEPNAYjJSz8eTwn682KvVDOGbSxZP47VPVCMa OHYA== X-Gm-Message-State: AOAM532S0sHyUKsbIoudeQFvdP8qucKVSzSmAL5YrRdoJB33SNxS7JZ3 VxOs9urES3JSa8VD0SEU/YG6ZL5aG4w= X-Google-Smtp-Source: ABdhPJzgDbopp2o77po8l7A/S0LFZa5mDiEt8CIMZnU2TNFpHgkoh9deOcwEfKMMMuWk2+TVZO+w/WwC2bU= X-Received: from seanjc798194.pdx.corp.google.com ([2620:15c:f:10:f031:9c1c:56c7:c3bf]) (user=seanjc job=sendgmr) by 2002:a25:ba87:: with SMTP id s7mr45194375ybg.222.1618338596029; Tue, 13 Apr 2021 11:29:56 -0700 (PDT) Reply-To: Sean Christopherson Date: Tue, 13 Apr 2021 11:29:33 -0700 In-Reply-To: <20210413182933.1046389-1-seanjc@google.com> Message-Id: <20210413182933.1046389-8-seanjc@google.com> Mime-Version: 1.0 References: <20210413182933.1046389-1-seanjc@google.com> X-Mailer: git-send-email 2.31.1.295.g9ea45b61b8-goog Subject: [RFC PATCH 7/7] KVM: x86: Defer tick-based accounting 'til after IRQ handling From: Sean Christopherson To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Thomas Gleixner , Christian Borntraeger , Michael Tokarev Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org When using tick-based accounting, defer the call to account guest time until after servicing any IRQ(s) that happened in the guest (or immediately after VM-Exit). When using tick-based accounting, time is accounted to the guest when PF_VCPU is set when the tick IRQ handler runs. The current approach of unconditionally accounting time in kvm_guest_exit_irqoff() prevents IRQs that occur in the guest from ever being processed with PF_VCPU set, since PF_VCPU ends up being set only during the relatively short VM-Enter sequence, which runs entirely with IRQs disabled. Fixes: 87fa7f3e98a131 ("x86/kvm: Move context tracking where it belongs") Cc: Thomas Gleixner Cc: Michael Tokarev Signed-off-by: Sean Christopherson --- arch/x86/kvm/x86.c | 8 ++++++++ arch/x86/kvm/x86.h | 9 ++++++--- 2 files changed, 14 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 16fb39503296..096bbf50b7a9 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -9230,6 +9230,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) local_irq_disable(); kvm_after_interrupt(vcpu); + /* + * When using tick-based account, wait until after servicing IRQs to + * account guest time so that any ticks that occurred while running the + * guest are properly accounted to the guest. + */ + if (!IS_ENABLED(CONFIG_VIRT_CPU_ACCOUNTING_GEN)) + kvm_vtime_account_guest_exit(); + if (lapic_in_kernel(vcpu)) { s64 delta = vcpu->arch.apic->lapic_timer.advance_expire_delta; if (delta != S64_MIN) { diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 74ef92f47db8..039a7d585925 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -38,15 +38,18 @@ static __always_inline void kvm_guest_exit_irqoff(void) * have them in state 'on' as recorded before entering guest mode. * Same as enter_from_user_mode(). * - * guest_exit_irqoff() restores host context and reinstates RCU if - * enabled and required. + * context_tracking_guest_exit_irqoff() restores host context and + * reinstates RCU if enabled and required. * * This needs to be done before the below as native_read_msr() * contains a tracepoint and x86_spec_ctrl_restore_host() calls * into world and some more. */ lockdep_hardirqs_off(CALLER_ADDR0); - guest_exit_irqoff(); + context_tracking_guest_exit_irqoff(); + + if (IS_ENABLED(CONFIG_VIRT_CPU_ACCOUNTING_GEN)) + kvm_vtime_account_guest_exit(); instrumentation_begin(); trace_hardirqs_off_finish();