From patchwork Wed Aug 31 08:53:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Like Xu X-Patchwork-Id: 12960563 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55BDBECAAD4 for ; Wed, 31 Aug 2022 08:53:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229999AbiHaIx4 (ORCPT ); Wed, 31 Aug 2022 04:53:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44674 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231475AbiHaIxx (ORCPT ); Wed, 31 Aug 2022 04:53:53 -0400 Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 662FDCAC40; Wed, 31 Aug 2022 01:53:49 -0700 (PDT) Received: by mail-pl1-x632.google.com with SMTP id d12so13519413plr.6; Wed, 31 Aug 2022 01:53:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=uKjXaCvraXKdFShGCWadz+/E1NVHvgjhtf2XIArgSac=; b=GWUqEesQbD5/Q9UnVR3R1W9wGubBzOeCuDhCRAJM9YFlj9i/uFMw+0GsT/Dck5kUtg w0sPTopTebcUmcOmywdE44BNiz13noJHi2DMCP0D0j2Syze16gaSzUWoyhcspVKp6Xdq S0FXMEszLrTUQlkC4jW3vcQ/AWrk7CWjk4sBOTB3EemlqqLgk6xTSsAOS6+lYgp93618 8hB/6lJ0W8sHQEsZmUT1IkqNSmQAZ24VQnVn8nTmVZfn8mNIvN4Rg28XSPyfmTYRMXCC GPKrZ7bA8RcMWZd2eVuSt/Nd7vuX6wTt1JUn0ANhrgDT9QasVD47XXB9HjwioBxwhTMA Fs8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=uKjXaCvraXKdFShGCWadz+/E1NVHvgjhtf2XIArgSac=; b=zq5ByVlh49l9gk3dr/FEMU+7afWrxgNf44jKoKYNI544bgaU/BzgAWOBDQrcQgXUCg HVpiT5X4YeqiJhGsJUAxf0HznRZLmzZptQKRuBXOsYSfwF/aynSL+OfbTS3Bp/veUC51 bO+ycUFfYYqWRyWKMdWHcQBs8LTQGlYLIomFh8G7ngRaVy3Y/DysRgaQb0eAz2vx4bP6 7Q7Hj10ewWx6BS+w44ScgblbxpEhzbApY+Ls6jEvnUDF/OlKm3mgk3Znmmbhz09XF5N/ B0/0q1zPXnUc+Ms9bnYt9v4BcF53Nq39GhLitEMbTxaze6rNXc/+DsGZ0DDQ/Gp04KGt uK4g== X-Gm-Message-State: ACgBeo2Ci1xBJJP5xBldTIc0tkoMNy/3bYJOhDeQugsO/BfMDoCGV9O3 Kp/pQHTzoLo/URnJ3klt6Mo= X-Google-Smtp-Source: AA6agR4sUkABqoKYIWRVOfJfk77ZcNzOkJqZCIoCyvSudVU2xCRVF3PVf9PKAn+v4ElShCJ93GxDNQ== X-Received: by 2002:a17:902:f641:b0:172:e2f8:7efb with SMTP id m1-20020a170902f64100b00172e2f87efbmr24210882plg.140.1661936028708; Wed, 31 Aug 2022 01:53:48 -0700 (PDT) Received: from localhost.localdomain ([103.7.29.32]) by smtp.gmail.com with ESMTPSA id 26-20020a17090a1a1a00b001fab208523esm868772pjk.3.2022.08.31.01.53.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Aug 2022 01:53:48 -0700 (PDT) From: Like Xu X-Google-Original-From: Like Xu To: Paolo Bonzini , Sean Christopherson Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 1/7] KVM: x86/pmu: Avoid setting BIT_ULL(-1) to pmu->host_cross_mapped_mask Date: Wed, 31 Aug 2022 16:53:22 +0800 Message-Id: <20220831085328.45489-2-likexu@tencent.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220831085328.45489-1-likexu@tencent.com> References: <20220831085328.45489-1-likexu@tencent.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Like Xu In the extreme case of host counters multiplexing and contention, the perf_event requested by the guest's pebs counter is not allocated to any actual physical counter, in which case hw.idx is bookkept as -1, resulting in an out-of-bounds access to host_cross_mapped_mask. Fixes: 854250329c02 ("KVM: x86/pmu: Disable guest PEBS temporarily in two rare situations") Signed-off-by: Like Xu --- arch/x86/kvm/vmx/pmu_intel.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/vmx/pmu_intel.c b/arch/x86/kvm/vmx/pmu_intel.c index c399637a3a79..d595ff33d32d 100644 --- a/arch/x86/kvm/vmx/pmu_intel.c +++ b/arch/x86/kvm/vmx/pmu_intel.c @@ -776,20 +776,20 @@ static void intel_pmu_cleanup(struct kvm_vcpu *vcpu) void intel_pmu_cross_mapped_check(struct kvm_pmu *pmu) { struct kvm_pmc *pmc = NULL; - int bit; + int bit, hw_idx; for_each_set_bit(bit, (unsigned long *)&pmu->global_ctrl, X86_PMC_IDX_MAX) { pmc = intel_pmc_idx_to_pmc(pmu, bit); if (!pmc || !pmc_speculative_in_use(pmc) || - !intel_pmc_is_enabled(pmc)) + !intel_pmc_is_enabled(pmc) || !pmc->perf_event) continue; - if (pmc->perf_event && pmc->idx != pmc->perf_event->hw.idx) { - pmu->host_cross_mapped_mask |= - BIT_ULL(pmc->perf_event->hw.idx); - } + hw_idx = pmc->perf_event->hw.idx; + /* make it a little less dependent on perf's exact behavior */ + if (hw_idx != pmc->idx && hw_idx > -1) + pmu->host_cross_mapped_mask |= BIT_ULL(hw_idx); } } From patchwork Wed Aug 31 08:53:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Like Xu X-Patchwork-Id: 12960564 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35DCAC0502C for ; Wed, 31 Aug 2022 08:53:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231600AbiHaIx5 (ORCPT ); Wed, 31 Aug 2022 04:53:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44618 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231499AbiHaIxx (ORCPT ); Wed, 31 Aug 2022 04:53:53 -0400 Received: from mail-pg1-x531.google.com (mail-pg1-x531.google.com [IPv6:2607:f8b0:4864:20::531]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 01E98C9EA1; Wed, 31 Aug 2022 01:53:51 -0700 (PDT) Received: by mail-pg1-x531.google.com with SMTP id v4so12941802pgi.10; Wed, 31 Aug 2022 01:53:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=cdByzgdS6GVsCXzYCxwV/Q4bVr0REmGp0IsegOGQh7g=; b=C/p/nHPybikK8PwRKe3rnCgXVe4NS0OrPhhTuNSZV6vETUkIbw26XyE8Jzd+jwyui3 yg5N+fFiHieDqVmtLKjW1brF+5aCi3sorhnCfq9cD/4dtpI8jczJKhsEBhbPxHCbf9Vg 97iwH+0+KFqCFBirejw070jTwtdTuyP6Fob4DJMDxBjGWVxlapTF/P11QEjzYyXIK1PD f8Ww8Nl9d59AFD8bLJWPjuaALu+BnZ2dpX8PIXnT3MDbN+yqbTzvqfZucQibScWwZiQx IWEp8UTDGcq1u/X4gm708+N+TkEtbAJ30ErWHZldRt2hYUGUlkie986bM7McR+N9Sz0i pWPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=cdByzgdS6GVsCXzYCxwV/Q4bVr0REmGp0IsegOGQh7g=; b=Rq59VWHuhqW1N3LBOfMMD3aLX5bgFLHIV1f1PCnuq6/dqpojE1G/ABmUD5STQmy8+h EZav0UgbgIsSZkLwSgL4Jejn2coqwwy92yj6FOJE9YSwQkqNWjdo9eyGW0ExrK4Pugum Q8PwsgDAEsAI2zfVJBCaDEUe54CDoxT9MxPRDVFFYskDWhDcvyOU0U6rb+CpP1WWxPTZ R6jM3wuVIrbZjz6Jr8/r9nBmU5Gs6igZKGgPay9+EF1NpmtymXJBaqTKC5zs7KpKCSx9 Mq8XkDunZQeuNnbarKAziWPgN1nS473ZlUW+4/LCFuHkkozA/mBDP8NywDjEj5DtHQBn iubA== X-Gm-Message-State: ACgBeo3m+dysm+jReubHY7ui0RpxpF9NH3F6bWQ36iVC0J/AySsvpSgB z+Eoz1KkpZM5xM0nsbMT6BuTFbRXso9OlJxu X-Google-Smtp-Source: AA6agR6PR5h52Q2dB62cd2N+8kBgBRCMeVozKK5VtWbzkZ+ZTQeNQ+FIIpAMYTPU2wCOMRDiUS70SA== X-Received: by 2002:a63:7d58:0:b0:42b:484c:979f with SMTP id m24-20020a637d58000000b0042b484c979fmr21397692pgn.7.1661936030265; Wed, 31 Aug 2022 01:53:50 -0700 (PDT) Received: from localhost.localdomain ([103.7.29.32]) by smtp.gmail.com with ESMTPSA id 26-20020a17090a1a1a00b001fab208523esm868772pjk.3.2022.08.31.01.53.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Aug 2022 01:53:50 -0700 (PDT) From: Like Xu X-Google-Original-From: Like Xu To: Paolo Bonzini , Sean Christopherson Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 2/7] KVM: x86/pmu: Don't generate PEBS records for emulated instructions Date: Wed, 31 Aug 2022 16:53:23 +0800 Message-Id: <20220831085328.45489-3-likexu@tencent.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220831085328.45489-1-likexu@tencent.com> References: <20220831085328.45489-1-likexu@tencent.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Like Xu KVM will accumulate an enabled counter for at least INSTRUCTIONS or BRANCH_INSTRUCTION hw event from any KVM emulated instructions, generating emulated overflow interrupt on counter overflow, which in theory should also happen when the PEBS counter overflows but it currently lacks this part of the underlying support (e.g. through software injection of records in the irq context or a lazy approach). In this case, KVM skips the injection of this BUFFER_OVF PMI (effectively dropping one PEBS record) and let the overflow counter move on. The loss of a single sample does not introduce a loss of accuracy, but is easily noticeable for certain specific instructions. This issue is expected to be addressed along with the issue of PEBS cross-mapped counters with a slow-path proposal. Fixes: 79f3e3b58386 ("KVM: x86/pmu: Reprogram PEBS event to emulate guest PEBS counter") Signed-off-by: Like Xu --- arch/x86/kvm/pmu.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c index 02f9e4f245bd..390d697efde1 100644 --- a/arch/x86/kvm/pmu.c +++ b/arch/x86/kvm/pmu.c @@ -106,9 +106,19 @@ static inline void __kvm_perf_overflow(struct kvm_pmc *pmc, bool in_pmi) return; if (pmc->perf_event && pmc->perf_event->attr.precise_ip) { - /* Indicate PEBS overflow PMI to guest. */ - skip_pmi = __test_and_set_bit(GLOBAL_STATUS_BUFFER_OVF_BIT, - (unsigned long *)&pmu->global_status); + if (!in_pmi) { + /* + * TODO: KVM is currently _choosing_ to not generate records + * for emulated instructions, avoiding BUFFER_OVF PMI when + * there are no records. Strictly speaking, it should be done + * as well in the right context to improve sampling accuracy. + */ + skip_pmi = true; + } else { + /* Indicate PEBS overflow PMI to guest. */ + skip_pmi = __test_and_set_bit(GLOBAL_STATUS_BUFFER_OVF_BIT, + (unsigned long *)&pmu->global_status); + } } else { __set_bit(pmc->idx, (unsigned long *)&pmu->global_status); } From patchwork Wed Aug 31 08:53:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Like Xu X-Patchwork-Id: 12960565 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1596ECAAD1 for ; Wed, 31 Aug 2022 08:54:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231497AbiHaIyA (ORCPT ); Wed, 31 Aug 2022 04:54:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44686 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231520AbiHaIxx (ORCPT ); Wed, 31 Aug 2022 04:53:53 -0400 Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 39390CAC65; Wed, 31 Aug 2022 01:53:52 -0700 (PDT) Received: by mail-pl1-x632.google.com with SMTP id d12so13519488plr.6; Wed, 31 Aug 2022 01:53:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=n4xJJQm16hxaBCpj8YKFv/yui3kLzCuA1z3ufWBUg3c=; b=pXl4nL3Ns57dEP1Rlyt00cG4GpWjsssRkrOLB2WSCJlb7ttj4sYoS0SYXrPOuwoQ1R Duqbsr2KcHjgiZbNcJJfPRNK6KNZPIoVTo4XvjTOEkmJt1iylG5dDMQhWyKR+RHhBeqC hhv9wsZ1aw4k0adebJQe0BdPMkzYPKSIUXL/yJ6bwyxHTKg+AhmG9s0P4A7zyY+gXE3d dTz3JnKgLWa0NMDKgNmU2RMiMt0jC5hKWMQS7Xk83dVGaN2oNcXxOfHK5zYrJnCBXkjp LRgsNTISaF23t54AKQqu6zjRe45OLzLELCg41Lp8Sm2+4M/7ZuwD0jzZpWl37RS6pB36 uoKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=n4xJJQm16hxaBCpj8YKFv/yui3kLzCuA1z3ufWBUg3c=; b=Tg4nVJd8r2Dv46/8JR/hrj3VwBBs36o/DqoYvsC6N2KYIOsaAD9dHLs/53IXplZE/E 9R/i03cpE6IHmI7d5F3FkA2165ca7+Crg2ZWvsfAyTGaWU6YzZ8aH8DnDG4OmWyloA3h wXpQExJsIxDfWoO5Ne3B/ecYvK+YDs32Ot+mqtR3tugu+4yBRlw9VmqXFP2cJG7jo/n+ 2IwSq68Q8+KP19NLpcpKzzldisvnGtZplE98S5ZU45kOumdJzgtdEBLCEzyR9PMIMZGL uPiH+TmJ8s6Qt0R6YTJh5jCIEyvl4i/VlR0zbvrrsi+VE6AogAQXfFyQuTRaHVswvkpM sjVw== X-Gm-Message-State: ACgBeo2xDEJIIBAZxkQEcdJnRHjyclmr5qVcsN7ggdxLmA4oS9HZ0yK0 x+7e+JV/QZu1/zuSkW0c51c= X-Google-Smtp-Source: AA6agR5NPOoN7Rb/jnDVS4jLTMlettiK675j5edx1RbIoI99uHmqrF57PqAXEKel5KNJALcyV6iU4g== X-Received: by 2002:a17:902:e80a:b0:175:41cd:2684 with SMTP id u10-20020a170902e80a00b0017541cd2684mr2518738plg.159.1661936031884; Wed, 31 Aug 2022 01:53:51 -0700 (PDT) Received: from localhost.localdomain ([103.7.29.32]) by smtp.gmail.com with ESMTPSA id 26-20020a17090a1a1a00b001fab208523esm868772pjk.3.2022.08.31.01.53.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Aug 2022 01:53:51 -0700 (PDT) From: Like Xu X-Google-Original-From: Like Xu To: Paolo Bonzini , Sean Christopherson Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 3/7] KVM: x86/pmu: Avoid using PEBS perf_events for normal counters Date: Wed, 31 Aug 2022 16:53:24 +0800 Message-Id: <20220831085328.45489-4-likexu@tencent.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220831085328.45489-1-likexu@tencent.com> References: <20220831085328.45489-1-likexu@tencent.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Like Xu The check logic in the pmc_resume_counter() to determine whether a perf_event is reusable is partial and flawed, especially when it comes to a pseudocode sequence (not correct but clearly valid) like: - enabling a counter and its PEBS bit - enable global_ctrl - run workload - disable only the PEBS bit, leaving the global_ctrl bit enabled In this corner case, a perf_event created for PEBS can be reused by a normal counter before it has been released and recreated, and when this normal counter overflows, it triggers a PEBS interrupt (precise_ip != 0). To address this issue, the reuse check has been revamped and KVM will go back to do reprogram_counter() when any bit of guest PEBS_ENABLE msr has changed, which is similar to what global_ctrl_changed() does. Fixes: 79f3e3b58386 ("KVM: x86/pmu: Reprogram PEBS event to emulate guest PEBS counter") Signed-off-by: Like Xu --- arch/x86/kvm/pmu.c | 4 ++-- arch/x86/kvm/vmx/pmu_intel.c | 14 +++++++------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c index 390d697efde1..d9b9a0f0db17 100644 --- a/arch/x86/kvm/pmu.c +++ b/arch/x86/kvm/pmu.c @@ -237,8 +237,8 @@ static bool pmc_resume_counter(struct kvm_pmc *pmc) get_sample_period(pmc, pmc->counter))) return false; - if (!test_bit(pmc->idx, (unsigned long *)&pmc_to_pmu(pmc)->pebs_enable) && - pmc->perf_event->attr.precise_ip) + if (test_bit(pmc->idx, (unsigned long *)&pmc_to_pmu(pmc)->pebs_enable) != + (!!pmc->perf_event->attr.precise_ip)) return false; /* reuse perf_event to serve as pmc_reprogram_counter() does*/ diff --git a/arch/x86/kvm/vmx/pmu_intel.c b/arch/x86/kvm/vmx/pmu_intel.c index d595ff33d32d..6242b0b81116 100644 --- a/arch/x86/kvm/vmx/pmu_intel.c +++ b/arch/x86/kvm/vmx/pmu_intel.c @@ -68,15 +68,11 @@ static struct kvm_pmc *intel_pmc_idx_to_pmc(struct kvm_pmu *pmu, int pmc_idx) } } -/* function is called when global control register has been updated. */ -static void global_ctrl_changed(struct kvm_pmu *pmu, u64 data) +static void reprogram_counters(struct kvm_pmu *pmu, u64 diff) { int bit; - u64 diff = pmu->global_ctrl ^ data; struct kvm_pmc *pmc; - pmu->global_ctrl = data; - for_each_set_bit(bit, (unsigned long *)&diff, X86_PMC_IDX_MAX) { pmc = intel_pmc_idx_to_pmc(pmu, bit); if (pmc) @@ -397,7 +393,7 @@ static int intel_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) struct kvm_pmc *pmc; u32 msr = msr_info->index; u64 data = msr_info->data; - u64 reserved_bits; + u64 reserved_bits, diff; switch (msr) { case MSR_CORE_PERF_FIXED_CTR_CTRL: @@ -418,7 +414,9 @@ static int intel_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) if (pmu->global_ctrl == data) return 0; if (kvm_valid_perf_global_ctrl(pmu, data)) { - global_ctrl_changed(pmu, data); + diff = pmu->global_ctrl ^ data; + pmu->global_ctrl = data; + reprogram_counters(pmu, diff); return 0; } break; @@ -433,7 +431,9 @@ static int intel_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) if (pmu->pebs_enable == data) return 0; if (!(data & pmu->pebs_enable_mask)) { + diff = pmu->pebs_enable ^ data; pmu->pebs_enable = data; + reprogram_counters(pmu, diff); return 0; } break; From patchwork Wed Aug 31 08:53:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Like Xu X-Patchwork-Id: 12960566 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF6F8C0502C for ; Wed, 31 Aug 2022 08:54:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231634AbiHaIyD (ORCPT ); Wed, 31 Aug 2022 04:54:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44674 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231562AbiHaIxz (ORCPT ); Wed, 31 Aug 2022 04:53:55 -0400 Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F2CC2C9259; Wed, 31 Aug 2022 01:53:53 -0700 (PDT) Received: by mail-pl1-x632.google.com with SMTP id p18so13513969plr.8; Wed, 31 Aug 2022 01:53:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=gDuWmSP1BE6Srfu1dJWvvdDAMEOps0ym11WanWVP6BA=; b=OI40+/eIKMqWFL1WqscdIgiizJoDVmAMYC+b2MRH3AVNE3p0QshhGbXMarH0q0bOox HycYXXZX+VuOD7NBdrwA01RsPaRCsV7ked8GR7TLwcF17iNwAeuHq7O/mKYJlTKQy7N2 i0V3gckJe/6FXBihyFos6RxkL4X5wKcLY7QzwHMzjDpzFCS3N8fMvTmuIj0ZF910eCOo Hwvlgdz4RFdJKqWdv9WY00OT6hbaWC1meWnu3m3xzeTQqo7eEpiYZJVyueQXoE6k1r4b h7q5AQqD5tsi6Xt9vGFvkD8Gdm1PXxpd4Cl7cKMPJ1eB9fn/qbbbP49xvnA3Neq8uemI kxUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=gDuWmSP1BE6Srfu1dJWvvdDAMEOps0ym11WanWVP6BA=; b=RrURq4TRF4V1CyraDVb2/njxGhevf6K8jkSq6lhzlbWamnrUNtw2dkWqdMXgorZ7UM ophU/WOVmBLYqdTASOP2smPkADo4d8vlqF2WjFUWwlMtrZEntVkNwaRTc70gGU9C7TCv URXa4+IQ5yRL21aw/FL8Bbhvk2MxtMqSXxixJewKK+8RjXLPqxxCeTZselPDneLJzawJ S4BK/kuGIi2HP48tIrmnrE5kzcUk+xAC9pImQyKrJ9f9XDYV5e6wspWOvAlYpfPwCLH1 o/LbZMaHdFNz5KrqTdBW0wgUfY+giXRQHXqG0BoluLZG8vquzmEHBf+7Q+eslmN7kelj VbIQ== X-Gm-Message-State: ACgBeo0rzvu1eBZEZZZfHPNMIazAgjbQeAC7cRACW0ED3x72fSGfsSzQ Oi4FX8c0iXfCxUfskH3g0CY= X-Google-Smtp-Source: AA6agR477QpgtTTJ4QYHNQZc00xn+dTZFYw1GMIHBos87wE1oL2/KNnU9ffAnxal3UzFPa3UgU0+1Q== X-Received: by 2002:a17:902:ccc9:b0:174:de2b:b19a with SMTP id z9-20020a170902ccc900b00174de2bb19amr11706067ple.100.1661936033484; Wed, 31 Aug 2022 01:53:53 -0700 (PDT) Received: from localhost.localdomain ([103.7.29.32]) by smtp.gmail.com with ESMTPSA id 26-20020a17090a1a1a00b001fab208523esm868772pjk.3.2022.08.31.01.53.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Aug 2022 01:53:53 -0700 (PDT) From: Like Xu X-Google-Original-From: Like Xu To: Paolo Bonzini , Sean Christopherson Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 4/7] KVM: x86/pmu: Defer reprogram_counter() to kvm_pmu_handle_event() Date: Wed, 31 Aug 2022 16:53:25 +0800 Message-Id: <20220831085328.45489-5-likexu@tencent.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220831085328.45489-1-likexu@tencent.com> References: <20220831085328.45489-1-likexu@tencent.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Like Xu Batch reprogramming PMU counters by setting KVM_REQ_PMU and thus deferring reprogramming kvm_pmu_handle_event() to avoid reprogramming a counter multiple times during a single VM-Exit. Deferring programming will also allow KVM to fix a bug where immediately reprogramming a counter can result in sleeping (taking a mutex) while interrupts are disabled in the VM-Exit fastpath. Introducing kvm_pmu_request_counter_reprogam() to make it obvious that KVM is _requesting_ a reprogram and not actually doing the reprogram. Opportunistically refine related comments to avoid misunderstandings. Signed-off-by: Like Xu --- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/pmu.c | 11 ++++++----- arch/x86/kvm/pmu.h | 6 +++++- arch/x86/kvm/svm/pmu.c | 2 +- arch/x86/kvm/vmx/pmu_intel.c | 6 +++--- 5 files changed, 16 insertions(+), 10 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 2c96c43c313a..4e568a7ef464 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -493,6 +493,7 @@ struct kvm_pmc { struct perf_event *perf_event; struct kvm_vcpu *vcpu; /* + * only for creating or reusing perf_event, * eventsel value for general purpose counters, * ctrl value for fixed counters. */ diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c index d9b9a0f0db17..7f391750ebd3 100644 --- a/arch/x86/kvm/pmu.c +++ b/arch/x86/kvm/pmu.c @@ -101,7 +101,6 @@ static inline void __kvm_perf_overflow(struct kvm_pmc *pmc, bool in_pmi) struct kvm_pmu *pmu = pmc_to_pmu(pmc); bool skip_pmi = false; - /* Ignore counters that have been reprogrammed already. */ if (test_and_set_bit(pmc->idx, pmu->reprogram_pmi)) return; @@ -293,7 +292,7 @@ static bool check_pmu_event_filter(struct kvm_pmc *pmc) return allow_event; } -void reprogram_counter(struct kvm_pmc *pmc) +static void reprogram_counter(struct kvm_pmc *pmc) { struct kvm_pmu *pmu = pmc_to_pmu(pmc); u64 eventsel = pmc->eventsel; @@ -335,7 +334,6 @@ void reprogram_counter(struct kvm_pmc *pmc) !(eventsel & ARCH_PERFMON_EVENTSEL_OS), eventsel & ARCH_PERFMON_EVENTSEL_INT); } -EXPORT_SYMBOL_GPL(reprogram_counter); void kvm_pmu_handle_event(struct kvm_vcpu *vcpu) { @@ -345,10 +343,11 @@ void kvm_pmu_handle_event(struct kvm_vcpu *vcpu) for_each_set_bit(bit, pmu->reprogram_pmi, X86_PMC_IDX_MAX) { struct kvm_pmc *pmc = static_call(kvm_x86_pmu_pmc_idx_to_pmc)(pmu, bit); - if (unlikely(!pmc || !pmc->perf_event)) { + if (unlikely(!pmc)) { clear_bit(bit, pmu->reprogram_pmi); continue; } + reprogram_counter(pmc); } @@ -542,7 +541,9 @@ static inline bool eventsel_match_perf_hw_id(struct kvm_pmc *pmc, static inline bool cpl_is_matched(struct kvm_pmc *pmc) { bool select_os, select_user; - u64 config = pmc->current_config; + u64 config = pmc_is_gp(pmc) ? pmc->eventsel : + (u64)fixed_ctrl_field(pmc_to_pmu(pmc)->fixed_ctr_ctrl, + pmc->idx - INTEL_PMC_IDX_FIXED); if (pmc_is_gp(pmc)) { select_os = config & ARCH_PERFMON_EVENTSEL_OS; diff --git a/arch/x86/kvm/pmu.h b/arch/x86/kvm/pmu.h index 5cc5721f260b..847e7112a5d3 100644 --- a/arch/x86/kvm/pmu.h +++ b/arch/x86/kvm/pmu.h @@ -183,7 +183,11 @@ static inline void kvm_init_pmu_capability(void) KVM_PMC_MAX_FIXED); } -void reprogram_counter(struct kvm_pmc *pmc); +static inline void kvm_pmu_request_counter_reprogam(struct kvm_pmc *pmc) +{ + __set_bit(pmc->idx, pmc_to_pmu(pmc)->reprogram_pmi); + kvm_make_request(KVM_REQ_PMU, pmc->vcpu); +} void kvm_pmu_deliver_pmi(struct kvm_vcpu *vcpu); void kvm_pmu_handle_event(struct kvm_vcpu *vcpu); diff --git a/arch/x86/kvm/svm/pmu.c b/arch/x86/kvm/svm/pmu.c index f24613a108c5..70219c19b872 100644 --- a/arch/x86/kvm/svm/pmu.c +++ b/arch/x86/kvm/svm/pmu.c @@ -238,7 +238,7 @@ static int amd_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) data &= ~pmu->reserved_bits; if (data != pmc->eventsel) { pmc->eventsel = data; - reprogram_counter(pmc); + kvm_pmu_request_counter_reprogam(pmc); } return 0; } diff --git a/arch/x86/kvm/vmx/pmu_intel.c b/arch/x86/kvm/vmx/pmu_intel.c index 6242b0b81116..863a6ff9e214 100644 --- a/arch/x86/kvm/vmx/pmu_intel.c +++ b/arch/x86/kvm/vmx/pmu_intel.c @@ -52,7 +52,7 @@ static void reprogram_fixed_counters(struct kvm_pmu *pmu, u64 data) pmc = get_fixed_pmc(pmu, MSR_CORE_PERF_FIXED_CTR0 + i); __set_bit(INTEL_PMC_IDX_FIXED + i, pmu->pmc_in_use); - reprogram_counter(pmc); + kvm_pmu_request_counter_reprogam(pmc); } } @@ -76,7 +76,7 @@ static void reprogram_counters(struct kvm_pmu *pmu, u64 diff) for_each_set_bit(bit, (unsigned long *)&diff, X86_PMC_IDX_MAX) { pmc = intel_pmc_idx_to_pmc(pmu, bit); if (pmc) - reprogram_counter(pmc); + kvm_pmu_request_counter_reprogam(pmc); } } @@ -477,7 +477,7 @@ static int intel_pmu_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) reserved_bits ^= HSW_IN_TX_CHECKPOINTED; if (!(data & reserved_bits)) { pmc->eventsel = data; - reprogram_counter(pmc); + kvm_pmu_request_counter_reprogam(pmc); return 0; } } else if (intel_pmu_handle_lbr_msrs_access(vcpu, msr_info, false)) From patchwork Wed Aug 31 08:53:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Like Xu X-Patchwork-Id: 12960567 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33611ECAAD1 for ; Wed, 31 Aug 2022 08:54:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231641AbiHaIyE (ORCPT ); Wed, 31 Aug 2022 04:54:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44672 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231459AbiHaIx6 (ORCPT ); Wed, 31 Aug 2022 04:53:58 -0400 Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 23160C9EA1; Wed, 31 Aug 2022 01:53:56 -0700 (PDT) Received: by mail-pj1-x1030.google.com with SMTP id q3so10270468pjg.3; Wed, 31 Aug 2022 01:53:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=Gcm2IRQTqs+/zd8fZUOqHHTB6+RBczdToyXqQTmBoEc=; b=T18AiWlRi04ewNMzE3Y7uPaapmJYaFVOTjtOepcQ0GfIy5ouQQSw0g+ZIPHNsKtYhY RsaWxlAp8Q5/xOXeU0ivK0iyVuANRJkWGBRbxXoSK2cfYvKxgai8+ygd2x8vGm75Dopa ZlwCRI8vJZAnQv/CHGMXRXyAWp6lscAdj96d0EYXaJR+pvf2/k4TqRJo8422ucv5aEbH 2GxByNR69DVz1HNAL09cPBsaSj/EBF8jFOtJa4ws+20gcQEKqvCymsca8UIykya78gRz eiZuK37lTxQ0VZd+DpDMwvDCmHL5t6zuLUYXH+alak34YSD8T/F5S260KyZzUOW6pKSL J37w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=Gcm2IRQTqs+/zd8fZUOqHHTB6+RBczdToyXqQTmBoEc=; b=WsKjj9ccdSqJEhXrQGQEGrXNNAj2TB251bpRaZ/1KXoDDun4kOC/JIV4TAF2U5uzoy a5DaaiDuPqdMo1hsrcceLlDu8i929QcUo31hnfQokx1IeTbsE6suHr3QQgM0qAOVI6aD EdPCgJ9T3lIT+bH8uag7E7ai0FXdWIWl04OjgS/2JnxFq3ilw/ZtvpUY4aqqdD+U6t8u ZlHCz03+3oTjkQa2xgQeDjxGJaPbYY1s5KG4aa7yvZLnHNgArR2V8IQPw7GzTxC6bM9l MHmcQi93JY6GutlbZ9Z5j27cssVu5JzF4bgwepU66RX9q8hF+ZjXEJq82FfGQROuUron feZQ== X-Gm-Message-State: ACgBeo3Oo5uJdRFQdv1MZHVQ6nyeWUWxfnckdHwSb4eExrwxY7BSoDIC UEKabedIgEu72Vkx7j2Oyxo= X-Google-Smtp-Source: AA6agR5/UBexJglfYrq5LHFgtkDhtnwSjNhhnXinG0hsWVHNVzgqRw6BCv+0Y1tb/ywYSdA1zWdQsw== X-Received: by 2002:a17:90a:868c:b0:1fd:cfe8:5511 with SMTP id p12-20020a17090a868c00b001fdcfe85511mr2151436pjn.174.1661936035476; Wed, 31 Aug 2022 01:53:55 -0700 (PDT) Received: from localhost.localdomain ([103.7.29.32]) by smtp.gmail.com with ESMTPSA id 26-20020a17090a1a1a00b001fab208523esm868772pjk.3.2022.08.31.01.53.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Aug 2022 01:53:55 -0700 (PDT) From: Like Xu X-Google-Original-From: Like Xu To: Paolo Bonzini , Sean Christopherson Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Wanpeng Li Subject: [PATCH v3 5/7] KVM: x86/pmu: Defer counter emulated overflow via pmc->prev_counter Date: Wed, 31 Aug 2022 16:53:26 +0800 Message-Id: <20220831085328.45489-6-likexu@tencent.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220831085328.45489-1-likexu@tencent.com> References: <20220831085328.45489-1-likexu@tencent.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Like Xu Defer reprogramming counters and handling overflow via KVM_REQ_PMU when incrementing counters. KVM skips emulated WRMSR in the VM-Exit fastpath, the fastpath runs with IRQs disabled, skipping instructions can increment and reprogram counters, reprogramming counters can sleep, and sleeping is disallowed while IRQs are disabled. [*] BUG: sleeping function called from invalid context at kernel/locking/mutex.c:580 [*] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 2981888, name: CPU 15/KVM [*] preempt_count: 1, expected: 0 [*] RCU nest depth: 0, expected: 0 [*] INFO: lockdep is turned off. [*] irq event stamp: 0 [*] hardirqs last enabled at (0): [<0000000000000000>] 0x0 [*] hardirqs last disabled at (0): [] copy_process+0x146a/0x62d0 [*] softirqs last enabled at (0): [] copy_process+0x14a9/0x62d0 [*] softirqs last disabled at (0): [<0000000000000000>] 0x0 [*] Preemption disabled at: [*] [] vcpu_enter_guest+0x1001/0x3dc0 [kvm] [*] CPU: 17 PID: 2981888 Comm: CPU 15/KVM Kdump: 5.19.0-rc1-g239111db364c-dirty #2 [*] Call Trace: [*] [*] dump_stack_lvl+0x6c/0x9b [*] __might_resched.cold+0x22e/0x297 [*] __mutex_lock+0xc0/0x23b0 [*] perf_event_ctx_lock_nested+0x18f/0x340 [*] perf_event_pause+0x1a/0x110 [*] reprogram_counter+0x2af/0x1490 [kvm] [*] kvm_pmu_trigger_event+0x429/0x950 [kvm] [*] kvm_skip_emulated_instruction+0x48/0x90 [kvm] [*] handle_fastpath_set_msr_irqoff+0x349/0x3b0 [kvm] [*] vmx_vcpu_run+0x268e/0x3b80 [kvm_intel] [*] vcpu_enter_guest+0x1d22/0x3dc0 [kvm] Add a field to kvm_pmc to track the previous counter value in order to defer overflow detection to kvm_pmu_handle_event() (reprogramming must be done before handling overflow). Opportunistically shrink sizeof(struct kvm_pmc) a bit. Suggested-by: Wanpeng Li Fixes: 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions") Signed-off-by: Like Xu --- arch/x86/include/asm/kvm_host.h | 5 +++-- arch/x86/kvm/pmu.c | 13 ++++++------- arch/x86/kvm/svm/pmu.c | 2 +- arch/x86/kvm/vmx/pmu_intel.c | 4 ++-- 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 4e568a7ef464..08c3f90b4ac3 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -488,7 +488,10 @@ enum pmc_type { struct kvm_pmc { enum pmc_type type; u8 idx; + bool is_paused; + bool intr; u64 counter; + u64 prev_counter; u64 eventsel; struct perf_event *perf_event; struct kvm_vcpu *vcpu; @@ -498,8 +501,6 @@ struct kvm_pmc { * ctrl value for fixed counters. */ u64 current_config; - bool is_paused; - bool intr; }; #define KVM_PMC_MAX_FIXED 3 diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c index 7f391750ebd3..3c42df3a55ff 100644 --- a/arch/x86/kvm/pmu.c +++ b/arch/x86/kvm/pmu.c @@ -349,6 +349,10 @@ void kvm_pmu_handle_event(struct kvm_vcpu *vcpu) } reprogram_counter(pmc); + + if (pmc->counter < pmc->prev_counter) + __kvm_perf_overflow(pmc, false); + pmc->prev_counter = 0; } /* @@ -521,14 +525,9 @@ void kvm_pmu_destroy(struct kvm_vcpu *vcpu) static void kvm_pmu_incr_counter(struct kvm_pmc *pmc) { - u64 prev_count; - - prev_count = pmc->counter; + pmc->prev_counter = pmc->counter; pmc->counter = (pmc->counter + 1) & pmc_bitmask(pmc); - - reprogram_counter(pmc); - if (pmc->counter < prev_count) - __kvm_perf_overflow(pmc, false); + kvm_pmu_request_counter_reprogam(pmc); } static inline bool eventsel_match_perf_hw_id(struct kvm_pmc *pmc, diff --git a/arch/x86/kvm/svm/pmu.c b/arch/x86/kvm/svm/pmu.c index 70219c19b872..0166f3bc6447 100644 --- a/arch/x86/kvm/svm/pmu.c +++ b/arch/x86/kvm/svm/pmu.c @@ -290,7 +290,7 @@ static void amd_pmu_reset(struct kvm_vcpu *vcpu) struct kvm_pmc *pmc = &pmu->gp_counters[i]; pmc_stop_counter(pmc); - pmc->counter = pmc->eventsel = 0; + pmc->counter = pmc->prev_counter = pmc->eventsel = 0; } } diff --git a/arch/x86/kvm/vmx/pmu_intel.c b/arch/x86/kvm/vmx/pmu_intel.c index 863a6ff9e214..1d3d0bd3e0e7 100644 --- a/arch/x86/kvm/vmx/pmu_intel.c +++ b/arch/x86/kvm/vmx/pmu_intel.c @@ -647,14 +647,14 @@ static void intel_pmu_reset(struct kvm_vcpu *vcpu) pmc = &pmu->gp_counters[i]; pmc_stop_counter(pmc); - pmc->counter = pmc->eventsel = 0; + pmc->counter = pmc->prev_counter = pmc->eventsel = 0; } for (i = 0; i < KVM_PMC_MAX_FIXED; i++) { pmc = &pmu->fixed_counters[i]; pmc_stop_counter(pmc); - pmc->counter = 0; + pmc->counter = pmc->prev_counter = 0; } pmu->fixed_ctr_ctrl = pmu->global_ctrl = pmu->global_status = 0; From patchwork Wed Aug 31 08:53:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Like Xu X-Patchwork-Id: 12960568 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A265EECAAD3 for ; Wed, 31 Aug 2022 08:54:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231660AbiHaIyF (ORCPT ); Wed, 31 Aug 2022 04:54:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44678 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231605AbiHaIx7 (ORCPT ); Wed, 31 Aug 2022 04:53:59 -0400 Received: from mail-pf1-x430.google.com (mail-pf1-x430.google.com [IPv6:2607:f8b0:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C286FC9259; Wed, 31 Aug 2022 01:53:57 -0700 (PDT) Received: by mail-pf1-x430.google.com with SMTP id t129so13782937pfb.6; Wed, 31 Aug 2022 01:53:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=M85TwFsU78GuR/7MUBP0eSGrfptr74I6oc9jGB2T7VM=; b=Cspwi3zn09b/MwMNZiK7l/lGBbAB3fZcqxlrwsFDv6GySpcp6MdOyp2OOXPePLXbF0 ZVhbHKtmCeHaQzmvcdRJdh1pUSO0W9FmeYZD/sB3w0SkVXeanrI6KkBQsEOAgheXn6zX obc8K3RxE0CAzrV1bZ3cH1LhrkuW2xhiN/pmuVL47DppXbaM/SOSynhHtElEDyAZhlLl TN/EAk9WtPcvpycd50kVr2NydPGqhrUjdAWOSeGCewYknzh3bYqs2eupKDBLGc9H3AqL LjTXBw8ju23ESUuxwgOxgGg3ErHsHHYGqzxLiZNbvoKRNAYoHj+DgBwBwGvRi0wdM9r9 GnTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=M85TwFsU78GuR/7MUBP0eSGrfptr74I6oc9jGB2T7VM=; b=EK/fGvvUz1NUFq7Oa1zBCYVdj9eDB8kFbiZSVm3Y1AT/0XTcRLNe1QTJQXhQc24Zcp iXxvU39i+87b+QxRimTln5aPbH3KDSsLy2hcXzZ2l86MrUTFXwthMMb91hqw1uRbvVJI RsDh5n3NRqRXS4gVKzaEwfqW4quKVyVatCQsFk2WV3rcLRAD6edRF+BN3tjqRHYSNqvN Ll+TPYW0z334h2XE74/d49Q9SHtPqoqTMjm4FwjtXl3BimuzYTdp97DTHAX4Tj/km3wv nktN5T/caKlIbLYm1Fpp4KqpwoWag4D9t5IhuUFT3UkpTZwZkcve7+eOzEx5mRcdzm+L PlHA== X-Gm-Message-State: ACgBeo2+6wsVDv+ce9R2D81dScM3qTgshbVbWRc/Ht4xSZtvRKfC0tZC TQwBzrEX9FF3SnrWjRDD/Op7eSpLBubIKBUd X-Google-Smtp-Source: AA6agR7ojae63TXUxDxAsKxmg8yiWyE1+VTYv1EZ/jpbXf4zDDovjgrt1V0AWhgI1vSaKTaUOzvtsQ== X-Received: by 2002:a63:ce17:0:b0:42a:bfb6:f218 with SMTP id y23-20020a63ce17000000b0042abfb6f218mr20727351pgf.484.1661936037089; Wed, 31 Aug 2022 01:53:57 -0700 (PDT) Received: from localhost.localdomain ([103.7.29.32]) by smtp.gmail.com with ESMTPSA id 26-20020a17090a1a1a00b001fab208523esm868772pjk.3.2022.08.31.01.53.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Aug 2022 01:53:56 -0700 (PDT) From: Like Xu X-Google-Original-From: Like Xu To: Paolo Bonzini , Sean Christopherson Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 6/7] KVM: x86/svm/pmu: Direct access pmu->gp_counter[] to implement amd_*_to_pmc() Date: Wed, 31 Aug 2022 16:53:27 +0800 Message-Id: <20220831085328.45489-7-likexu@tencent.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220831085328.45489-1-likexu@tencent.com> References: <20220831085328.45489-1-likexu@tencent.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Like Xu Access PMU counters on AMD by directly indexing the array of general purpose counters instead of translating the PMC index to an MSR index. AMD only supports gp counters, there's no need to translate a PMC index to an MSR index and back to a PMC index. Opportunistically apply array_index_nospec() to reduce the attack surface for speculative execution and remove the dead code. Signed-off-by: Like Xu --- arch/x86/kvm/svm/pmu.c | 41 +++++------------------------------------ 1 file changed, 5 insertions(+), 36 deletions(-) diff --git a/arch/x86/kvm/svm/pmu.c b/arch/x86/kvm/svm/pmu.c index 0166f3bc6447..c736757c29d2 100644 --- a/arch/x86/kvm/svm/pmu.c +++ b/arch/x86/kvm/svm/pmu.c @@ -33,23 +33,6 @@ enum index { INDEX_ERROR, }; -static unsigned int get_msr_base(struct kvm_pmu *pmu, enum pmu_type type) -{ - struct kvm_vcpu *vcpu = pmu_to_vcpu(pmu); - - if (guest_cpuid_has(vcpu, X86_FEATURE_PERFCTR_CORE)) { - if (type == PMU_TYPE_COUNTER) - return MSR_F15H_PERF_CTR; - else - return MSR_F15H_PERF_CTL; - } else { - if (type == PMU_TYPE_COUNTER) - return MSR_K7_PERFCTR0; - else - return MSR_K7_EVNTSEL0; - } -} - static enum index msr_to_index(u32 msr) { switch (msr) { @@ -141,18 +124,12 @@ static bool amd_pmc_is_enabled(struct kvm_pmc *pmc) static struct kvm_pmc *amd_pmc_idx_to_pmc(struct kvm_pmu *pmu, int pmc_idx) { - unsigned int base = get_msr_base(pmu, PMU_TYPE_COUNTER); - struct kvm_vcpu *vcpu = pmu_to_vcpu(pmu); + unsigned int num_counters = pmu->nr_arch_gp_counters; - if (guest_cpuid_has(vcpu, X86_FEATURE_PERFCTR_CORE)) { - /* - * The idx is contiguous. The MSRs are not. The counter MSRs - * are interleaved with the event select MSRs. - */ - pmc_idx *= 2; - } + if (pmc_idx >= num_counters) + return NULL; - return get_gp_pmc_amd(pmu, base + pmc_idx, PMU_TYPE_COUNTER); + return &pmu->gp_counters[array_index_nospec(pmc_idx, num_counters)]; } static bool amd_is_valid_rdpmc_ecx(struct kvm_vcpu *vcpu, unsigned int idx) @@ -168,15 +145,7 @@ static bool amd_is_valid_rdpmc_ecx(struct kvm_vcpu *vcpu, unsigned int idx) static struct kvm_pmc *amd_rdpmc_ecx_to_pmc(struct kvm_vcpu *vcpu, unsigned int idx, u64 *mask) { - struct kvm_pmu *pmu = vcpu_to_pmu(vcpu); - struct kvm_pmc *counters; - - idx &= ~(3u << 30); - if (idx >= pmu->nr_arch_gp_counters) - return NULL; - counters = pmu->gp_counters; - - return &counters[idx]; + return amd_pmc_idx_to_pmc(vcpu_to_pmu(vcpu), idx & ~(3u << 30)); } static bool amd_is_valid_msr(struct kvm_vcpu *vcpu, u32 msr) From patchwork Wed Aug 31 08:53:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Like Xu X-Patchwork-Id: 12960569 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68E35ECAAD3 for ; Wed, 31 Aug 2022 08:54:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231627AbiHaIyP (ORCPT ); Wed, 31 Aug 2022 04:54:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44878 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231612AbiHaIyB (ORCPT ); Wed, 31 Aug 2022 04:54:01 -0400 Received: from mail-pj1-x1031.google.com (mail-pj1-x1031.google.com [IPv6:2607:f8b0:4864:20::1031]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E3135CAC70; Wed, 31 Aug 2022 01:53:59 -0700 (PDT) Received: by mail-pj1-x1031.google.com with SMTP id t11-20020a17090a510b00b001fac77e9d1fso20421267pjh.5; Wed, 31 Aug 2022 01:53:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=Ra1XTrPkeUnQjVN7gF2ynGZOQwRo6+U5v64sosBSSNE=; b=TbkoULMu/PebQrH/fNWCuJwq4vyr3xHDPitg7hlRD1vc4gSv9QKl0ZxV0dLIu5wSfY Qe0LjMtqW1/fuWy/GqCmLiFU5lVfbNQulN8Cbh/qoQaKBVGrk2NHcKtA2rlscb3Xv6gJ 4DbJhemF9peqdmNc7QyHLaDzKEb/+1lhsfyZh/r1BQ1c5Uxi4PSJRFdmeow8BaLxCmy9 a/x8WJHBdyeHBMs5Ez8GT8TFP7+DpnFCpMJpqiafqz6r29tKiBEAmXqBMZl3isqxqEX4 bXMkhhS+UliHQl4joPAfy8DdqskL2gsOC/T9RIjciSypsP1FmUQnHdG4V/UFSRKUi9aM Eu9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=Ra1XTrPkeUnQjVN7gF2ynGZOQwRo6+U5v64sosBSSNE=; b=wz7UvHKwgo0+PRoerdba/NMpwID7UccR3AvINftUktb5r/2nQwYFH/UgRgHIaBn9oA f+rUWOr++HTYzJeD4qC0LINASYKzJHnO2BDCB2C7nfk7H50o0J+faaPon17QniRsEW6x XCLmx0oIxlcfcVwbH+tKnj+Szb1HdWZHIDtizJVq6+4u+VQ4AogkVtEoDp6coY6uQ+jc hDaoDnhkcGiGIWU9ujCu0hAd9lYAJ6J59D37/s90O92pHqNDS8q5aD/if47xAeZ46tC8 Dfld3mpZARKFl1muE0MNEU2VTG+thrYECSiEbQeXaorLKV7995oBiDYdCV69v477c5Ec RtLw== X-Gm-Message-State: ACgBeo03TQnkZgxaaIL/mr9HBHYiE1yOe3tL2x4MJ/QTinN060GVCu/8 70R+HTeO2MR1ZF/lZNbtva4YbOCM2kpZ8kWd X-Google-Smtp-Source: AA6agR6h1miFrjqk/OZbSh0MJNoFTggNs1WBgnb5Ln0YDv29L5LbRf8d9O1+628O8BPVterl6Gg6JQ== X-Received: by 2002:a17:90a:c782:b0:1fb:307f:7cbd with SMTP id gn2-20020a17090ac78200b001fb307f7cbdmr2202279pjb.14.1661936038736; Wed, 31 Aug 2022 01:53:58 -0700 (PDT) Received: from localhost.localdomain ([103.7.29.32]) by smtp.gmail.com with ESMTPSA id 26-20020a17090a1a1a00b001fab208523esm868772pjk.3.2022.08.31.01.53.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Aug 2022 01:53:58 -0700 (PDT) From: Like Xu X-Google-Original-From: Like Xu To: Paolo Bonzini , Sean Christopherson Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 7/7] KVM: x86/svm/pmu: Rewrite get_gp_pmc_amd() for more counters scalability Date: Wed, 31 Aug 2022 16:53:28 +0800 Message-Id: <20220831085328.45489-8-likexu@tencent.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220831085328.45489-1-likexu@tencent.com> References: <20220831085328.45489-1-likexu@tencent.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Like Xu If the number of AMD gp counters continues to grow, the code will be very clumsy and the switch-case design of inline get_gp_pmc_amd() will also bloat the kernel text size. The target code is taught to manage two groups of MSRs, each representing a different version of the AMD PMU counter MSRs. The MSR addresses of each group are contiguous, with no holes, and there is no intersection between two sets of addresses, but they are discrete in functionality by design like this: [Group A : All counter MSRs are tightly bound to all event select MSRs ] MSR_K7_EVNTSEL0 0xc0010000 MSR_K7_EVNTSELi 0xc0010000 + i ... MSR_K7_EVNTSEL3 0xc0010003 MSR_K7_PERFCTR0 0xc0010004 MSR_K7_PERFCTRi 0xc0010004 + i ... MSR_K7_PERFCTR3 0xc0010007 [Group B : The counter MSRs are interleaved with the event select MSRs ] MSR_F15H_PERF_CTL0 0xc0010200 MSR_F15H_PERF_CTR0 (0xc0010200 + 1) ... MSR_F15H_PERF_CTLi (0xc0010200 + 2 * i) MSR_F15H_PERF_CTRi (0xc0010200 + 2 * i + 1) ... MSR_F15H_PERF_CTL5 (0xc0010200 + 2 * 5) MSR_F15H_PERF_CTR5 (0xc0010200 + 2 * 5 + 1) Rewrite get_gp_pmc_amd() in this way: first determine which group of registers is accessed, then determine if it matches its requested type, applying different scaling ratios respectively, and finally get pmc_idx to pass into amd_pmc_idx_to_pmc(). Signed-off-by: Like Xu --- arch/x86/kvm/svm/pmu.c | 88 ++++++++++-------------------------------- 1 file changed, 20 insertions(+), 68 deletions(-) diff --git a/arch/x86/kvm/svm/pmu.c b/arch/x86/kvm/svm/pmu.c index c736757c29d2..2ec420b85d6a 100644 --- a/arch/x86/kvm/svm/pmu.c +++ b/arch/x86/kvm/svm/pmu.c @@ -23,90 +23,52 @@ enum pmu_type { PMU_TYPE_EVNTSEL, }; -enum index { - INDEX_ZERO = 0, - INDEX_ONE, - INDEX_TWO, - INDEX_THREE, - INDEX_FOUR, - INDEX_FIVE, - INDEX_ERROR, -}; - -static enum index msr_to_index(u32 msr) +static struct kvm_pmc *amd_pmc_idx_to_pmc(struct kvm_pmu *pmu, int pmc_idx) { - switch (msr) { - case MSR_F15H_PERF_CTL0: - case MSR_F15H_PERF_CTR0: - case MSR_K7_EVNTSEL0: - case MSR_K7_PERFCTR0: - return INDEX_ZERO; - case MSR_F15H_PERF_CTL1: - case MSR_F15H_PERF_CTR1: - case MSR_K7_EVNTSEL1: - case MSR_K7_PERFCTR1: - return INDEX_ONE; - case MSR_F15H_PERF_CTL2: - case MSR_F15H_PERF_CTR2: - case MSR_K7_EVNTSEL2: - case MSR_K7_PERFCTR2: - return INDEX_TWO; - case MSR_F15H_PERF_CTL3: - case MSR_F15H_PERF_CTR3: - case MSR_K7_EVNTSEL3: - case MSR_K7_PERFCTR3: - return INDEX_THREE; - case MSR_F15H_PERF_CTL4: - case MSR_F15H_PERF_CTR4: - return INDEX_FOUR; - case MSR_F15H_PERF_CTL5: - case MSR_F15H_PERF_CTR5: - return INDEX_FIVE; - default: - return INDEX_ERROR; - } + unsigned int num_counters = pmu->nr_arch_gp_counters; + + if (pmc_idx >= num_counters) + return NULL; + + return &pmu->gp_counters[array_index_nospec(pmc_idx, num_counters)]; } static inline struct kvm_pmc *get_gp_pmc_amd(struct kvm_pmu *pmu, u32 msr, enum pmu_type type) { struct kvm_vcpu *vcpu = pmu_to_vcpu(pmu); + unsigned int idx; if (!vcpu->kvm->arch.enable_pmu) return NULL; switch (msr) { - case MSR_F15H_PERF_CTL0: - case MSR_F15H_PERF_CTL1: - case MSR_F15H_PERF_CTL2: - case MSR_F15H_PERF_CTL3: - case MSR_F15H_PERF_CTL4: - case MSR_F15H_PERF_CTL5: + case MSR_F15H_PERF_CTL0 ... MSR_F15H_PERF_CTR5: if (!guest_cpuid_has(vcpu, X86_FEATURE_PERFCTR_CORE)) return NULL; - fallthrough; + /* + * Each PMU counter has a pair of CTL and CTR MSRs. CTLn + * MSRs (accessed via EVNTSEL) are even, CTRn MSRs are odd. + */ + idx = (unsigned int)((msr - MSR_F15H_PERF_CTL0) / 2); + if (!(msr & 0x1) != (type == PMU_TYPE_EVNTSEL)) + return NULL; + break; case MSR_K7_EVNTSEL0 ... MSR_K7_EVNTSEL3: if (type != PMU_TYPE_EVNTSEL) return NULL; + idx = msr - MSR_K7_EVNTSEL0; break; - case MSR_F15H_PERF_CTR0: - case MSR_F15H_PERF_CTR1: - case MSR_F15H_PERF_CTR2: - case MSR_F15H_PERF_CTR3: - case MSR_F15H_PERF_CTR4: - case MSR_F15H_PERF_CTR5: - if (!guest_cpuid_has(vcpu, X86_FEATURE_PERFCTR_CORE)) - return NULL; - fallthrough; case MSR_K7_PERFCTR0 ... MSR_K7_PERFCTR3: if (type != PMU_TYPE_COUNTER) return NULL; + idx = msr - MSR_K7_PERFCTR0; break; default: return NULL; } - return &pmu->gp_counters[msr_to_index(msr)]; + return amd_pmc_idx_to_pmc(pmu, idx); } static bool amd_hw_event_available(struct kvm_pmc *pmc) @@ -122,16 +84,6 @@ static bool amd_pmc_is_enabled(struct kvm_pmc *pmc) return true; } -static struct kvm_pmc *amd_pmc_idx_to_pmc(struct kvm_pmu *pmu, int pmc_idx) -{ - unsigned int num_counters = pmu->nr_arch_gp_counters; - - if (pmc_idx >= num_counters) - return NULL; - - return &pmu->gp_counters[array_index_nospec(pmc_idx, num_counters)]; -} - static bool amd_is_valid_rdpmc_ecx(struct kvm_vcpu *vcpu, unsigned int idx) { struct kvm_pmu *pmu = vcpu_to_pmu(vcpu);