From patchwork Wed Nov 3 07:03:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Like Xu X-Patchwork-Id: 12600289 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48DFCC433F5 for ; Wed, 3 Nov 2021 07:03:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 21DB0611AE for ; Wed, 3 Nov 2021 07:03:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232048AbhKCHGC (ORCPT ); Wed, 3 Nov 2021 03:06:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40816 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232040AbhKCHGA (ORCPT ); Wed, 3 Nov 2021 03:06:00 -0400 Received: from mail-pl1-x62b.google.com (mail-pl1-x62b.google.com [IPv6:2607:f8b0:4864:20::62b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 73157C061714; Wed, 3 Nov 2021 00:03:24 -0700 (PDT) Received: by mail-pl1-x62b.google.com with SMTP id r5so1890800pls.1; Wed, 03 Nov 2021 00:03:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5/kU5qotulIdz4RQvPukUhsY06HNcD2se+CSZ3yeiVQ=; b=Q/imCNSQpHfxAwAiUrHxA8kVFvenR/FxeeDJkz3T0vP9ldB6zksXW3ojq9DSUFUXOe O3YxCExeWvIEERp2RKVih5Sm3PiTqGYQhz2oHqnylnSTCPqhzMN5xPlI1BMs/WUErQj1 FC4qXtP9nDqTpw3TFJDHH6PMTCE7AcTgaoggQh3w5A0lcAZ6G/fcTZrviQ83dylLfILm JIkLpIdb1BC21ItFbnoupv2jQeRcU5Ue7YOoxBFFCRSUO5faC+ENeaAfUgBEhnsdt+Q1 hvi/VUxXaYYKjSGn7iYQjjfsoX3JISvlJ5jiL6adJfxomrjbXXwurHd0Khh/XjgYW5wr ub5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5/kU5qotulIdz4RQvPukUhsY06HNcD2se+CSZ3yeiVQ=; b=BlRmKoP9RkDjCLRmD6jztfr1y2c3enoQZW0KXKRVI/KlfYDtFKnctlTfBF1s/Sfi3V 0rNHqkZAZ8CO42s0sKdfnsobhjE2nFo2JpntjnzmInkmR/0M85op5iXsNaXQIy7XFG+r qgqb9rdLoguJ9uMcfc+oogLARrGW9qpPUORrIghPQQEnLC1l8p0ZeFx1mJ3XX7HyhX69 IouvWKxhAH5fexvG7OThzxgSs+d3VSsKsUewuWPuUajpd2uu1fFbhO9LNmnVCJO31cDS Fr5qUHdRxfqEIYa2RHdLfXNeoAunQ+VjdJlxxtBCk/Irfv/ubJFeYwuoRlCE0pZAcSjW eFXg== X-Gm-Message-State: AOAM532xmA9GCh0RRl11xYzmK7izeRICt5F3wOZBkJkCPsg6Bmy/TSfj fPGg9HDHz0FruAg16wNToX4= X-Google-Smtp-Source: ABdhPJy21IF+pKr92lL0FteerlSRfxwxusTlcnZcLXocJiGmO3MnKjAYwmx31CM9rSzSmZ+iol3PJA== X-Received: by 2002:a17:90a:4e02:: with SMTP id n2mr8902359pjh.170.1635923004036; Wed, 03 Nov 2021 00:03:24 -0700 (PDT) Received: from localhost.localdomain ([103.7.29.32]) by smtp.gmail.com with ESMTPSA id x9sm4242564pjp.50.2021.11.03.00.03.21 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Nov 2021 00:03:23 -0700 (PDT) From: Like Xu X-Google-Original-From: Like Xu To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Thomas Gleixner , Ingo Molnar , Borislav Petkov , kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 1/3] KVM: x86: Copy kvm_pmu_ops by value to eliminate layer of indirection Date: Wed, 3 Nov 2021 15:03:08 +0800 Message-Id: <20211103070310.43380-2-likexu@tencent.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20211103070310.43380-1-likexu@tencent.com> References: <20211103070310.43380-1-likexu@tencent.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Replace the kvm_pmu_ops pointer in common x86 with an instance of the struct to save one pointer dereference when invoking functions. Copy the struct by value to set the ops during kvm_init(). Using kvm_x86_ops.hardware_enable to track whether or not the ops have been initialized, i.e. a vendor KVM module has been loaded. Signed-off-by: Like Xu --- arch/x86/kvm/pmu.c | 41 +++++++++++++++++++++------------------ arch/x86/kvm/pmu.h | 4 +++- arch/x86/kvm/vmx/nested.c | 2 +- arch/x86/kvm/x86.c | 3 +++ 4 files changed, 29 insertions(+), 21 deletions(-) diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c index 0772bad9165c..0db1887137d9 100644 --- a/arch/x86/kvm/pmu.c +++ b/arch/x86/kvm/pmu.c @@ -47,6 +47,9 @@ * * AMD: [0 .. AMD64_NUM_COUNTERS-1] <=> gp counters */ +struct kvm_pmu_ops kvm_pmu_ops __read_mostly; +EXPORT_SYMBOL_GPL(kvm_pmu_ops); + static void kvm_pmi_trigger_fn(struct irq_work *irq_work) { struct kvm_pmu *pmu = container_of(irq_work, struct kvm_pmu, irq_work); @@ -214,7 +217,7 @@ void reprogram_gp_counter(struct kvm_pmc *pmc, u64 eventsel) ARCH_PERFMON_EVENTSEL_CMASK | HSW_IN_TX | HSW_IN_TX_CHECKPOINTED))) { - config = kvm_x86_ops.pmu_ops->find_arch_event(pmc_to_pmu(pmc), + config = kvm_pmu_ops.find_arch_event(pmc_to_pmu(pmc), event_select, unit_mask); if (config != PERF_COUNT_HW_MAX) @@ -268,7 +271,7 @@ void reprogram_fixed_counter(struct kvm_pmc *pmc, u8 ctrl, int idx) pmc->current_config = (u64)ctrl; pmc_reprogram_counter(pmc, PERF_TYPE_HARDWARE, - kvm_x86_ops.pmu_ops->find_fixed_event(idx), + kvm_pmu_ops.find_fixed_event(idx), !(en_field & 0x2), /* exclude user */ !(en_field & 0x1), /* exclude kernel */ pmi, false, false); @@ -277,7 +280,7 @@ EXPORT_SYMBOL_GPL(reprogram_fixed_counter); void reprogram_counter(struct kvm_pmu *pmu, int pmc_idx) { - struct kvm_pmc *pmc = kvm_x86_ops.pmu_ops->pmc_idx_to_pmc(pmu, pmc_idx); + struct kvm_pmc *pmc = kvm_pmu_ops.pmc_idx_to_pmc(pmu, pmc_idx); if (!pmc) return; @@ -299,7 +302,7 @@ void kvm_pmu_handle_event(struct kvm_vcpu *vcpu) int bit; for_each_set_bit(bit, pmu->reprogram_pmi, X86_PMC_IDX_MAX) { - struct kvm_pmc *pmc = kvm_x86_ops.pmu_ops->pmc_idx_to_pmc(pmu, bit); + struct kvm_pmc *pmc = kvm_pmu_ops.pmc_idx_to_pmc(pmu, bit); if (unlikely(!pmc || !pmc->perf_event)) { clear_bit(bit, pmu->reprogram_pmi); @@ -321,7 +324,7 @@ void kvm_pmu_handle_event(struct kvm_vcpu *vcpu) /* check if idx is a valid index to access PMU */ int kvm_pmu_is_valid_rdpmc_ecx(struct kvm_vcpu *vcpu, unsigned int idx) { - return kvm_x86_ops.pmu_ops->is_valid_rdpmc_ecx(vcpu, idx); + return kvm_pmu_ops.is_valid_rdpmc_ecx(vcpu, idx); } bool is_vmware_backdoor_pmc(u32 pmc_idx) @@ -371,7 +374,7 @@ int kvm_pmu_rdpmc(struct kvm_vcpu *vcpu, unsigned idx, u64 *data) if (is_vmware_backdoor_pmc(idx)) return kvm_pmu_rdpmc_vmware(vcpu, idx, data); - pmc = kvm_x86_ops.pmu_ops->rdpmc_ecx_to_pmc(vcpu, idx, &mask); + pmc = kvm_pmu_ops.rdpmc_ecx_to_pmc(vcpu, idx, &mask); if (!pmc) return 1; @@ -387,22 +390,22 @@ int kvm_pmu_rdpmc(struct kvm_vcpu *vcpu, unsigned idx, u64 *data) void kvm_pmu_deliver_pmi(struct kvm_vcpu *vcpu) { if (lapic_in_kernel(vcpu)) { - if (kvm_x86_ops.pmu_ops->deliver_pmi) - kvm_x86_ops.pmu_ops->deliver_pmi(vcpu); + if (kvm_pmu_ops.deliver_pmi) + kvm_pmu_ops.deliver_pmi(vcpu); kvm_apic_local_deliver(vcpu->arch.apic, APIC_LVTPC); } } bool kvm_pmu_is_valid_msr(struct kvm_vcpu *vcpu, u32 msr) { - return kvm_x86_ops.pmu_ops->msr_idx_to_pmc(vcpu, msr) || - kvm_x86_ops.pmu_ops->is_valid_msr(vcpu, msr); + return kvm_pmu_ops.msr_idx_to_pmc(vcpu, msr) || + kvm_pmu_ops.is_valid_msr(vcpu, msr); } static void kvm_pmu_mark_pmc_in_use(struct kvm_vcpu *vcpu, u32 msr) { struct kvm_pmu *pmu = vcpu_to_pmu(vcpu); - struct kvm_pmc *pmc = kvm_x86_ops.pmu_ops->msr_idx_to_pmc(vcpu, msr); + struct kvm_pmc *pmc = kvm_pmu_ops.msr_idx_to_pmc(vcpu, msr); if (pmc) __set_bit(pmc->idx, pmu->pmc_in_use); @@ -410,13 +413,13 @@ static void kvm_pmu_mark_pmc_in_use(struct kvm_vcpu *vcpu, u32 msr) int kvm_pmu_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) { - return kvm_x86_ops.pmu_ops->get_msr(vcpu, msr_info); + return kvm_pmu_ops.get_msr(vcpu, msr_info); } int kvm_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) { kvm_pmu_mark_pmc_in_use(vcpu, msr_info->index); - return kvm_x86_ops.pmu_ops->set_msr(vcpu, msr_info); + return kvm_pmu_ops.set_msr(vcpu, msr_info); } /* refresh PMU settings. This function generally is called when underlying @@ -425,7 +428,7 @@ int kvm_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) */ void kvm_pmu_refresh(struct kvm_vcpu *vcpu) { - kvm_x86_ops.pmu_ops->refresh(vcpu); + kvm_pmu_ops.refresh(vcpu); } void kvm_pmu_reset(struct kvm_vcpu *vcpu) @@ -433,7 +436,7 @@ void kvm_pmu_reset(struct kvm_vcpu *vcpu) struct kvm_pmu *pmu = vcpu_to_pmu(vcpu); irq_work_sync(&pmu->irq_work); - kvm_x86_ops.pmu_ops->reset(vcpu); + kvm_pmu_ops.reset(vcpu); } void kvm_pmu_init(struct kvm_vcpu *vcpu) @@ -441,7 +444,7 @@ void kvm_pmu_init(struct kvm_vcpu *vcpu) struct kvm_pmu *pmu = vcpu_to_pmu(vcpu); memset(pmu, 0, sizeof(*pmu)); - kvm_x86_ops.pmu_ops->init(vcpu); + kvm_pmu_ops.init(vcpu); init_irq_work(&pmu->irq_work, kvm_pmi_trigger_fn); pmu->event_count = 0; pmu->need_cleanup = false; @@ -473,14 +476,14 @@ void kvm_pmu_cleanup(struct kvm_vcpu *vcpu) pmu->pmc_in_use, X86_PMC_IDX_MAX); for_each_set_bit(i, bitmask, X86_PMC_IDX_MAX) { - pmc = kvm_x86_ops.pmu_ops->pmc_idx_to_pmc(pmu, i); + pmc = kvm_pmu_ops.pmc_idx_to_pmc(pmu, i); if (pmc && pmc->perf_event && !pmc_speculative_in_use(pmc)) pmc_stop_counter(pmc); } - if (kvm_x86_ops.pmu_ops->cleanup) - kvm_x86_ops.pmu_ops->cleanup(vcpu); + if (kvm_pmu_ops.cleanup) + kvm_pmu_ops.cleanup(vcpu); bitmap_zero(pmu->pmc_in_use, X86_PMC_IDX_MAX); } diff --git a/arch/x86/kvm/pmu.h b/arch/x86/kvm/pmu.h index 0e4f2b1fa9fb..b2fe135d395a 100644 --- a/arch/x86/kvm/pmu.h +++ b/arch/x86/kvm/pmu.h @@ -17,6 +17,8 @@ #define MAX_FIXED_COUNTERS 3 +extern struct kvm_pmu_ops kvm_pmu_ops; + struct kvm_event_hw_type_mapping { u8 eventsel; u8 unit_mask; @@ -92,7 +94,7 @@ static inline bool pmc_is_fixed(struct kvm_pmc *pmc) static inline bool pmc_is_enabled(struct kvm_pmc *pmc) { - return kvm_x86_ops.pmu_ops->pmc_is_enabled(pmc); + return kvm_pmu_ops.pmc_is_enabled(pmc); } static inline bool kvm_valid_perf_global_ctrl(struct kvm_pmu *pmu, diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index b4ee5e9f9e20..1e793e44b5ff 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -4796,7 +4796,7 @@ void nested_vmx_pmu_entry_exit_ctls_update(struct kvm_vcpu *vcpu) return; vmx = to_vmx(vcpu); - if (kvm_x86_ops.pmu_ops->is_valid_msr(vcpu, MSR_CORE_PERF_GLOBAL_CTRL)) { + if (kvm_pmu_ops.is_valid_msr(vcpu, MSR_CORE_PERF_GLOBAL_CTRL)) { vmx->nested.msrs.entry_ctls_high |= VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL; vmx->nested.msrs.exit_ctls_high |= diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index ac83d873d65b..72d286595012 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -11317,6 +11317,9 @@ int kvm_arch_hardware_setup(void *opaque) memcpy(&kvm_x86_ops, ops->runtime_ops, sizeof(kvm_x86_ops)); kvm_ops_static_call_update(); + if (kvm_x86_ops.hardware_enable) + memcpy(&kvm_pmu_ops, kvm_x86_ops.pmu_ops, sizeof(kvm_pmu_ops)); + if (!kvm_cpu_cap_has(X86_FEATURE_XSAVES)) supported_xss = 0; From patchwork Wed Nov 3 07:03:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Like Xu X-Patchwork-Id: 12600291 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91EC4C433F5 for ; Wed, 3 Nov 2021 07:03:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 734A8611AE for ; Wed, 3 Nov 2021 07:03:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232081AbhKCHGE (ORCPT ); Wed, 3 Nov 2021 03:06:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40830 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232040AbhKCHGD (ORCPT ); Wed, 3 Nov 2021 03:06:03 -0400 Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 30EB2C061714; Wed, 3 Nov 2021 00:03:27 -0700 (PDT) Received: by mail-pl1-x630.google.com with SMTP id r5so1890884pls.1; Wed, 03 Nov 2021 00:03:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nEzrkqMWOPHSw09kJBsw5hSTh6gQrdiHdsIlswJkoXg=; b=HwfD5B5dF9WXE2dzhFVUComcRJ6EdQI4lRQDjf0oi5/F4uo7Z+WSSk5/QN3fB6+CSt ek3NJzvPcwMqEozk0Jmk013MzNYLi1Ge5wkMxtdbQWwpMxYU+jw2Iqf8ioz/UQBBaBML 0YnaKS2bQOC89by9NHAnq0QZ8Tzaj0JvxJjq7LL3vRt7N0FPRIMSHyMBelrkFlDJxILe mOic4kXwhDCzKiMpeK36c162UVBem0nBl7kpqlNFexP/lzdg9RfuwOb+zCLmHrh/xpqp UpHGS7QRt9A96uAgQi/w9wCOgCIvnC9rgKpJ3oeVtK4pHC4lxe3CkzkxYCdNwz0AnuWa VlVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nEzrkqMWOPHSw09kJBsw5hSTh6gQrdiHdsIlswJkoXg=; b=VXwSCd6Jk9LMWczjcsF7ewMafwFcdNtT2n+Oc+rJtfTnxAEkrKDElPusPt93e6OS7z Etv2E8B0Uq1ZZZUb8LORTM5J0jAPDo/xcACsJS14Z4VO3RcBiOz8OJa1SZJl5v86unt9 KaybHZgs4tuMje/6je2TJFFR/P8LBGjjZU1cx5toc+a+wft7qlvOL+FD7gAbmiqmsGc2 Y1iDNFyiCickgNQFojjN/qFVSU1MgyanrEE77+jwC7ODTdEmRiwJkF2ku1elu+W9QBq5 OEkg78lYdWbvCeC03Bt8wRS2uXI4RQJx09B1FQmL/7CQRphqEzzYAx7IjEvszY+sv4Zn 5hjA== X-Gm-Message-State: AOAM531Q/YJdNG/TLrZDS4Apb3M7KSVEtpsE54W9mSuz0Tr4gRQryQOl fkSpRZCyx/pLU6GHEVlehIM= X-Google-Smtp-Source: ABdhPJzlo8AIIklSaBIiJncQaZB0PbBnFX6Jq9hTWfYQsRx6gir4zi6kfOldb3VoWyNNERxNPVNZyQ== X-Received: by 2002:a17:90a:2c02:: with SMTP id m2mr12639696pjd.109.1635923006758; Wed, 03 Nov 2021 00:03:26 -0700 (PDT) Received: from localhost.localdomain ([103.7.29.32]) by smtp.gmail.com with ESMTPSA id x9sm4242564pjp.50.2021.11.03.00.03.24 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Nov 2021 00:03:26 -0700 (PDT) From: Like Xu X-Google-Original-From: Like Xu To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Thomas Gleixner , Ingo Molnar , Borislav Petkov , kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/3] KVM: x86: Introduce definitions to support static calls for kvm_pmu_ops Date: Wed, 3 Nov 2021 15:03:09 +0800 Message-Id: <20211103070310.43380-3-likexu@tencent.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20211103070310.43380-1-likexu@tencent.com> References: <20211103070310.43380-1-likexu@tencent.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Use static calls to improve kvm_pmu_ops performance. Introduce the definitions that will be used by a subsequent patch to actualize the savings. Add a new kvm-x86-pmu-ops.h header that can be used for the definition of static calls. This header is also intended to be used to simplify the defition of amd_pmu_ops and intel_pmu_ops. Like what we did for kvm_x86_ops, 'pmu_ops' can be covered by static calls in a simlilar manner for insignificant but not negligible performance impact, especially on older models. Signed-off-by: Like Xu --- arch/x86/include/asm/kvm-x86-pmu-ops.h | 32 ++++++++++++++++++++++++++ arch/x86/kvm/pmu.c | 7 ++++++ arch/x86/kvm/pmu.h | 15 ++++++++++++ 3 files changed, 54 insertions(+) create mode 100644 arch/x86/include/asm/kvm-x86-pmu-ops.h diff --git a/arch/x86/include/asm/kvm-x86-pmu-ops.h b/arch/x86/include/asm/kvm-x86-pmu-ops.h new file mode 100644 index 000000000000..b7713b16d21d --- /dev/null +++ b/arch/x86/include/asm/kvm-x86-pmu-ops.h @@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#if !defined(KVM_X86_PMU_OP) || !defined(KVM_X86_PMU_OP_NULL) +BUILD_BUG_ON(1) +#endif + +/* + * KVM_X86_PMU_OP() and KVM_X86_PMU_OP_NULL() are used to + * help generate "static_call()"s. They are also intended for use when defining + * the amd/intel KVM_X86_PMU_OPs. KVM_X86_PMU_OP() can be used + * for those functions that follow the [amd|intel]_func_name convention. + * KVM_X86_PMU_OP_NULL() can leave a NULL definition for the + * case where there is no definition or a function name that + * doesn't match the typical naming convention is supplied. + */ +KVM_X86_PMU_OP(find_arch_event); +KVM_X86_PMU_OP(find_fixed_event); +KVM_X86_PMU_OP(pmc_is_enabled); +KVM_X86_PMU_OP(pmc_idx_to_pmc); +KVM_X86_PMU_OP(rdpmc_ecx_to_pmc); +KVM_X86_PMU_OP(msr_idx_to_pmc); +KVM_X86_PMU_OP(is_valid_rdpmc_ecx); +KVM_X86_PMU_OP(is_valid_msr); +KVM_X86_PMU_OP(get_msr); +KVM_X86_PMU_OP(set_msr); +KVM_X86_PMU_OP(refresh); +KVM_X86_PMU_OP(init); +KVM_X86_PMU_OP(reset); +KVM_X86_PMU_OP_NULL(deliver_pmi); +KVM_X86_PMU_OP_NULL(cleanup); + +#undef KVM_X86_PMU_OP +#undef KVM_X86_PMU_OP_NULL diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c index 0db1887137d9..b6f08c719125 100644 --- a/arch/x86/kvm/pmu.c +++ b/arch/x86/kvm/pmu.c @@ -50,6 +50,13 @@ struct kvm_pmu_ops kvm_pmu_ops __read_mostly; EXPORT_SYMBOL_GPL(kvm_pmu_ops); +#define KVM_X86_PMU_OP(func) \ + DEFINE_STATIC_CALL_NULL(kvm_x86_pmu_##func, \ + *(((struct kvm_pmu_ops *)0)->func)) +#define KVM_X86_PMU_OP_NULL KVM_X86_PMU_OP +#include +EXPORT_STATIC_CALL_GPL(kvm_x86_pmu_is_valid_msr); + static void kvm_pmi_trigger_fn(struct irq_work *irq_work) { struct kvm_pmu *pmu = container_of(irq_work, struct kvm_pmu, irq_work); diff --git a/arch/x86/kvm/pmu.h b/arch/x86/kvm/pmu.h index b2fe135d395a..e5550d4acf14 100644 --- a/arch/x86/kvm/pmu.h +++ b/arch/x86/kvm/pmu.h @@ -3,6 +3,8 @@ #define __KVM_X86_PMU_H #include +#include +#include #define vcpu_to_pmu(vcpu) (&(vcpu)->arch.pmu) #define pmu_to_vcpu(pmu) (container_of((pmu), struct kvm_vcpu, arch.pmu)) @@ -45,6 +47,19 @@ struct kvm_pmu_ops { void (*cleanup)(struct kvm_vcpu *vcpu); }; +#define KVM_X86_PMU_OP(func) \ + DECLARE_STATIC_CALL(kvm_x86_pmu_##func, *(((struct kvm_pmu_ops *)0)->func)) +#define KVM_X86_PMU_OP_NULL KVM_X86_PMU_OP +#include + +static inline void kvm_pmu_ops_static_call_update(void) +{ +#define KVM_X86_PMU_OP(func) \ + static_call_update(kvm_x86_pmu_##func, kvm_pmu_ops.func) +#define KVM_X86_PMU_OP_NULL KVM_X86_PMU_OP +#include +} + static inline u64 pmc_bitmask(struct kvm_pmc *pmc) { struct kvm_pmu *pmu = pmc_to_pmu(pmc); From patchwork Wed Nov 3 07:03:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Like Xu X-Patchwork-Id: 12600293 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06954C433F5 for ; Wed, 3 Nov 2021 07:03:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E379761175 for ; Wed, 3 Nov 2021 07:03:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232110AbhKCHGJ (ORCPT ); Wed, 3 Nov 2021 03:06:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40848 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232068AbhKCHGG (ORCPT ); Wed, 3 Nov 2021 03:06:06 -0400 Received: from mail-pl1-x62a.google.com (mail-pl1-x62a.google.com [IPv6:2607:f8b0:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0781EC061714; Wed, 3 Nov 2021 00:03:30 -0700 (PDT) Received: by mail-pl1-x62a.google.com with SMTP id t21so1851693plr.6; Wed, 03 Nov 2021 00:03:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=eATASc+1pQX/h5gZEnL+VSURBOS+6YrgcKZxXUN5CpM=; b=JzDP0gC+qaI/TG3l6az6zTgfLAxFQ32ZncKXBCcmwUzgArAHwRcTHvscLzLRME/BTn pEDeoScXd90JbjfbYV7o9x/e3Sh61niF/MEFLpwRl7YudhEB98WqZxvTJZwBGDWVJLfg LoDP/l5LK4lAtuZxOZ+p1sDw/0tlvSdJfVr/W2Ww+0mIHaieHBZ2Ospw9El8qlzQIcL1 d3rTFYXzWK+xL/0yNIdZsHvU4fR3Ea5w0NU//ssExfZxIWh+yALnqKwsyjsV9UwijiD7 0a92d6hX8ENfHkTyQbJKaWgJIyyfpnyRDKfPKekjru1HBxVIke7zp74WLxvfDe5b4xJ3 9gzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=eATASc+1pQX/h5gZEnL+VSURBOS+6YrgcKZxXUN5CpM=; b=Z2Gu/AudTbqKGS3QqWdews1Ci/icGTVY50d0g6jPyU6aFI54FIC0Ge+QbwBof1mSpo IAJJL1DZg4FVek+CVZZCD+9qC4SgYXPGN/JfjidFYg2jxVYi7FDNmKKEd2kRoYUQ6LW9 3idXOUcLe6vHildlj90ZgsbJDb5jOf+SrSWpPjkr89QxNaRUCd6RAUvcEI2FmOXr87L4 BOHfOXj9PnjdpDbWiJn8MTSxOg9X2QvfOWv2iO4Wj/ygPSMG1/sq3hYHTWeyx0WnhBT8 39mkbwBdVBKOTDmmqBzYF+IeQmL2qYqIgTc2eEle9EpNKxhrE4rUpeoMkfnCzm/w4wip 2MLg== X-Gm-Message-State: AOAM530BZXClsO5M1wbETc6FOd5KUxwxYo5J4A/XLztR4C8djOFv/4Jb mKffUIt0aaDcGuwvITDIFOE= X-Google-Smtp-Source: ABdhPJxwO/Bv077H2yaqB/lSGJtWGSYzPCgTJu73IvUiziVDPXMyZeWei6g/tzWlfkVpVrZdPCLYvw== X-Received: by 2002:a17:902:9b95:b0:13f:c60c:683b with SMTP id y21-20020a1709029b9500b0013fc60c683bmr35471450plp.33.1635923009522; Wed, 03 Nov 2021 00:03:29 -0700 (PDT) Received: from localhost.localdomain ([103.7.29.32]) by smtp.gmail.com with ESMTPSA id x9sm4242564pjp.50.2021.11.03.00.03.27 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Nov 2021 00:03:29 -0700 (PDT) From: Like Xu X-Google-Original-From: Like Xu To: Paolo Bonzini Cc: Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Thomas Gleixner , Ingo Molnar , Borislav Petkov , kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/3] KVM: x86: Use static calls to reduce kvm_pmu_ops overhead Date: Wed, 3 Nov 2021 15:03:10 +0800 Message-Id: <20211103070310.43380-4-likexu@tencent.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20211103070310.43380-1-likexu@tencent.com> References: <20211103070310.43380-1-likexu@tencent.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Convert kvm_pmu_ops to use static calls. Here are the worst sched_clock() nanosecond numbers for the kvm_pmu_ops functions that is most often called (up to 7 digits of calls) when running a single perf test case in a guest on an ICX 2.70GHz host (mitigations=on): | legacy | static call ------------------------------------------------------------ .pmc_idx_to_pmc | 10946 | 10047 (8%) .pmc_is_enabled | 11291 | 11175 (1%) .msr_idx_to_pmc | 13526 | 12346 (8%) .is_valid_msr | 10895 | 10484 (3%) Signed-off-by: Like Xu --- arch/x86/kvm/pmu.c | 36 +++++++++++++++++------------------- arch/x86/kvm/pmu.h | 2 +- arch/x86/kvm/vmx/nested.c | 2 +- arch/x86/kvm/x86.c | 4 +++- 4 files changed, 22 insertions(+), 22 deletions(-) diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c index b6f08c719125..193f925e2064 100644 --- a/arch/x86/kvm/pmu.c +++ b/arch/x86/kvm/pmu.c @@ -224,7 +224,7 @@ void reprogram_gp_counter(struct kvm_pmc *pmc, u64 eventsel) ARCH_PERFMON_EVENTSEL_CMASK | HSW_IN_TX | HSW_IN_TX_CHECKPOINTED))) { - config = kvm_pmu_ops.find_arch_event(pmc_to_pmu(pmc), + config = static_call(kvm_x86_pmu_find_arch_event)(pmc_to_pmu(pmc), event_select, unit_mask); if (config != PERF_COUNT_HW_MAX) @@ -278,7 +278,7 @@ void reprogram_fixed_counter(struct kvm_pmc *pmc, u8 ctrl, int idx) pmc->current_config = (u64)ctrl; pmc_reprogram_counter(pmc, PERF_TYPE_HARDWARE, - kvm_pmu_ops.find_fixed_event(idx), + static_call(kvm_x86_pmu_find_fixed_event)(idx), !(en_field & 0x2), /* exclude user */ !(en_field & 0x1), /* exclude kernel */ pmi, false, false); @@ -287,7 +287,7 @@ EXPORT_SYMBOL_GPL(reprogram_fixed_counter); void reprogram_counter(struct kvm_pmu *pmu, int pmc_idx) { - struct kvm_pmc *pmc = kvm_pmu_ops.pmc_idx_to_pmc(pmu, pmc_idx); + struct kvm_pmc *pmc = static_call(kvm_x86_pmu_pmc_idx_to_pmc)(pmu, pmc_idx); if (!pmc) return; @@ -309,7 +309,7 @@ void kvm_pmu_handle_event(struct kvm_vcpu *vcpu) int bit; for_each_set_bit(bit, pmu->reprogram_pmi, X86_PMC_IDX_MAX) { - struct kvm_pmc *pmc = kvm_pmu_ops.pmc_idx_to_pmc(pmu, bit); + struct kvm_pmc *pmc = static_call(kvm_x86_pmu_pmc_idx_to_pmc)(pmu, bit); if (unlikely(!pmc || !pmc->perf_event)) { clear_bit(bit, pmu->reprogram_pmi); @@ -331,7 +331,7 @@ void kvm_pmu_handle_event(struct kvm_vcpu *vcpu) /* check if idx is a valid index to access PMU */ int kvm_pmu_is_valid_rdpmc_ecx(struct kvm_vcpu *vcpu, unsigned int idx) { - return kvm_pmu_ops.is_valid_rdpmc_ecx(vcpu, idx); + return static_call(kvm_x86_pmu_is_valid_rdpmc_ecx)(vcpu, idx); } bool is_vmware_backdoor_pmc(u32 pmc_idx) @@ -381,7 +381,7 @@ int kvm_pmu_rdpmc(struct kvm_vcpu *vcpu, unsigned idx, u64 *data) if (is_vmware_backdoor_pmc(idx)) return kvm_pmu_rdpmc_vmware(vcpu, idx, data); - pmc = kvm_pmu_ops.rdpmc_ecx_to_pmc(vcpu, idx, &mask); + pmc = static_call(kvm_x86_pmu_rdpmc_ecx_to_pmc)(vcpu, idx, &mask); if (!pmc) return 1; @@ -397,22 +397,21 @@ int kvm_pmu_rdpmc(struct kvm_vcpu *vcpu, unsigned idx, u64 *data) void kvm_pmu_deliver_pmi(struct kvm_vcpu *vcpu) { if (lapic_in_kernel(vcpu)) { - if (kvm_pmu_ops.deliver_pmi) - kvm_pmu_ops.deliver_pmi(vcpu); + static_call_cond(kvm_x86_pmu_deliver_pmi)(vcpu); kvm_apic_local_deliver(vcpu->arch.apic, APIC_LVTPC); } } bool kvm_pmu_is_valid_msr(struct kvm_vcpu *vcpu, u32 msr) { - return kvm_pmu_ops.msr_idx_to_pmc(vcpu, msr) || - kvm_pmu_ops.is_valid_msr(vcpu, msr); + return static_call(kvm_x86_pmu_msr_idx_to_pmc)(vcpu, msr) || + static_call(kvm_x86_pmu_is_valid_msr)(vcpu, msr); } static void kvm_pmu_mark_pmc_in_use(struct kvm_vcpu *vcpu, u32 msr) { struct kvm_pmu *pmu = vcpu_to_pmu(vcpu); - struct kvm_pmc *pmc = kvm_pmu_ops.msr_idx_to_pmc(vcpu, msr); + struct kvm_pmc *pmc = static_call(kvm_x86_pmu_msr_idx_to_pmc)(vcpu, msr); if (pmc) __set_bit(pmc->idx, pmu->pmc_in_use); @@ -420,13 +419,13 @@ static void kvm_pmu_mark_pmc_in_use(struct kvm_vcpu *vcpu, u32 msr) int kvm_pmu_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) { - return kvm_pmu_ops.get_msr(vcpu, msr_info); + return static_call(kvm_x86_pmu_get_msr)(vcpu, msr_info); } int kvm_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) { kvm_pmu_mark_pmc_in_use(vcpu, msr_info->index); - return kvm_pmu_ops.set_msr(vcpu, msr_info); + return static_call(kvm_x86_pmu_set_msr)(vcpu, msr_info); } /* refresh PMU settings. This function generally is called when underlying @@ -435,7 +434,7 @@ int kvm_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) */ void kvm_pmu_refresh(struct kvm_vcpu *vcpu) { - kvm_pmu_ops.refresh(vcpu); + static_call(kvm_x86_pmu_refresh)(vcpu); } void kvm_pmu_reset(struct kvm_vcpu *vcpu) @@ -443,7 +442,7 @@ void kvm_pmu_reset(struct kvm_vcpu *vcpu) struct kvm_pmu *pmu = vcpu_to_pmu(vcpu); irq_work_sync(&pmu->irq_work); - kvm_pmu_ops.reset(vcpu); + static_call(kvm_x86_pmu_reset)(vcpu); } void kvm_pmu_init(struct kvm_vcpu *vcpu) @@ -451,7 +450,7 @@ void kvm_pmu_init(struct kvm_vcpu *vcpu) struct kvm_pmu *pmu = vcpu_to_pmu(vcpu); memset(pmu, 0, sizeof(*pmu)); - kvm_pmu_ops.init(vcpu); + static_call(kvm_x86_pmu_init)(vcpu); init_irq_work(&pmu->irq_work, kvm_pmi_trigger_fn); pmu->event_count = 0; pmu->need_cleanup = false; @@ -483,14 +482,13 @@ void kvm_pmu_cleanup(struct kvm_vcpu *vcpu) pmu->pmc_in_use, X86_PMC_IDX_MAX); for_each_set_bit(i, bitmask, X86_PMC_IDX_MAX) { - pmc = kvm_pmu_ops.pmc_idx_to_pmc(pmu, i); + pmc = static_call(kvm_x86_pmu_pmc_idx_to_pmc)(pmu, i); if (pmc && pmc->perf_event && !pmc_speculative_in_use(pmc)) pmc_stop_counter(pmc); } - if (kvm_pmu_ops.cleanup) - kvm_pmu_ops.cleanup(vcpu); + static_call_cond(kvm_x86_pmu_cleanup)(vcpu); bitmap_zero(pmu->pmc_in_use, X86_PMC_IDX_MAX); } diff --git a/arch/x86/kvm/pmu.h b/arch/x86/kvm/pmu.h index e5550d4acf14..1818d1371ece 100644 --- a/arch/x86/kvm/pmu.h +++ b/arch/x86/kvm/pmu.h @@ -109,7 +109,7 @@ static inline bool pmc_is_fixed(struct kvm_pmc *pmc) static inline bool pmc_is_enabled(struct kvm_pmc *pmc) { - return kvm_pmu_ops.pmc_is_enabled(pmc); + return static_call(kvm_x86_pmu_pmc_is_enabled)(pmc); } static inline bool kvm_valid_perf_global_ctrl(struct kvm_pmu *pmu, diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index 1e793e44b5ff..a61661de1f39 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -4796,7 +4796,7 @@ void nested_vmx_pmu_entry_exit_ctls_update(struct kvm_vcpu *vcpu) return; vmx = to_vmx(vcpu); - if (kvm_pmu_ops.is_valid_msr(vcpu, MSR_CORE_PERF_GLOBAL_CTRL)) { + if (static_call(kvm_x86_pmu_is_valid_msr)(vcpu, MSR_CORE_PERF_GLOBAL_CTRL)) { vmx->nested.msrs.entry_ctls_high |= VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL; vmx->nested.msrs.exit_ctls_high |= diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 72d286595012..88a3ef809c98 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -11317,8 +11317,10 @@ int kvm_arch_hardware_setup(void *opaque) memcpy(&kvm_x86_ops, ops->runtime_ops, sizeof(kvm_x86_ops)); kvm_ops_static_call_update(); - if (kvm_x86_ops.hardware_enable) + if (kvm_x86_ops.hardware_enable) { memcpy(&kvm_pmu_ops, kvm_x86_ops.pmu_ops, sizeof(kvm_pmu_ops)); + kvm_pmu_ops_static_call_update(); + } if (!kvm_cpu_cap_has(X86_FEATURE_XSAVES)) supported_xss = 0;