From patchwork Wed Jan 19 23:07:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12717866 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35AA1C433FE for ; Wed, 19 Jan 2022 23:09:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1357685AbiASXJG (ORCPT ); Wed, 19 Jan 2022 18:09:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50262 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232883AbiASXHu (ORCPT ); Wed, 19 Jan 2022 18:07:50 -0500 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D549BC06174E for ; Wed, 19 Jan 2022 15:07:49 -0800 (PST) Received: by mail-pf1-x44a.google.com with SMTP id c23-20020aa78817000000b004be3f452a95so2423094pfo.19 for ; Wed, 19 Jan 2022 15:07:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=f0uwr62rbAHNSLgeo8auSREOnsbQjdEp+C2AD/si1YA=; b=CjI2dMMJIv+A3zoVcQPFWjDCrtC3xjc3ypT1b1yN0YnnNqq1eBKDZG2jNJDaigWYE0 /I/STWQ9Ldv4WQiQpaIy9kJSQsr1Zknmd89NWR2BPEdAk00QM7N7PTe+JVLOR3XT2Cy+ wn8c1eS9pQAigk6coGRgFDzi9/XMGg+pboe3M5OgfOntrsFaYmQSrmdnZRNO41s/yBw9 3qCfyJk+x9dSzXVf8kQoCiYmzcPjkdyIMm5X1hpZbz6bBtLFaiJa0KL6wpzsBarx4vFr 2/gwtIJvveCA4swVfLhiUEfwF1iXauYA3KNJiERFr6KkvEnx5UYDECSw5n4SXJaQaUL8 DIXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=f0uwr62rbAHNSLgeo8auSREOnsbQjdEp+C2AD/si1YA=; b=c8S4CyEIzeVWk8Qlm+3IDBeaWxoK4zp3bp0Nt3O5+jgIyVIS/kw7Y34Sf82u7tShxv Hg3Xzz0parlXU+nsuR4KyAeGNM9LxoGgGH52Q+Y+gy+CUUXT2CCMnT7XhEqH+yr6Xz58 2zrdYlO6C9tAaI5fnwf+q+nUJdWDzki7DSRflVtPQx2hHQ97+B4FvbdhvhF6H16VSKrr WCEHW1+Nqdz9R33n2sslCQoe4vO3I74aKDtBuev1EdEWM+DD4MS761RO9VOVMeB6/kNw 8GGT86uAOhQKXeeadqdfqtKJn7cmGJXgiiaBMbOW2hURygUQ7j71aYXIp1vZyrIap/6p y29g== X-Gm-Message-State: AOAM531Ak0lFLKIlmkdl1qiYzlQc6RtK/TIsBQb9fLppBnWnF1DqXwJo v7WG2xIiu6XZdUxxK7KiD/yyyxNd846hkA== X-Google-Smtp-Source: ABdhPJwb25CSinPWl5zCagVYhNF6QLlZuGdWkoCnfltU123tFBbwwNnT4H0K4OfAcVQDJdhkSYtqenbTUkhzLA== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:90a:5996:: with SMTP id l22mr7048761pji.27.1642633669310; Wed, 19 Jan 2022 15:07:49 -0800 (PST) Date: Wed, 19 Jan 2022 23:07:22 +0000 In-Reply-To: <20220119230739.2234394-1-dmatlack@google.com> Message-Id: <20220119230739.2234394-2-dmatlack@google.com> Mime-Version: 1.0 References: <20220119230739.2234394-1-dmatlack@google.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog Subject: [PATCH v2 01/18] KVM: x86/mmu: Rename rmap_write_protect() to kvm_vcpu_write_protect_gfn() From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org rmap_write_protect() is a poor name because it also write-protects SPTEs in the TDP MMU, not just SPTEs in the rmap. It is also confusing that rmap_write_protect() is not a simple wrapper around __rmap_write_protect(), since that is the common pattern for functions with double-underscore names. Rename rmap_write_protect() to kvm_vcpu_write_protect_gfn() to convey that KVM is write-protecting a specific gfn in the context of a vCPU. No functional change intended. Reviewed-by: Ben Gardon Reviewed-by: Peter Xu Reviewed-by: Sean Christopherson Signed-off-by: David Matlack --- arch/x86/kvm/mmu/mmu.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) base-commit: edb9e50dbe18394d0fc9d0494f5b6046fc912d33 diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 8ed2b42a7aa3..b541683c29c7 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1421,7 +1421,7 @@ bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm, return write_protected; } -static bool rmap_write_protect(struct kvm_vcpu *vcpu, u64 gfn) +static bool kvm_vcpu_write_protect_gfn(struct kvm_vcpu *vcpu, u64 gfn) { struct kvm_memory_slot *slot; @@ -2024,7 +2024,7 @@ static int mmu_sync_children(struct kvm_vcpu *vcpu, bool protected = false; for_each_sp(pages, sp, parents, i) - protected |= rmap_write_protect(vcpu, sp->gfn); + protected |= kvm_vcpu_write_protect_gfn(vcpu, sp->gfn); if (protected) { kvm_mmu_remote_flush_or_zap(vcpu->kvm, &invalid_list, true); @@ -2149,7 +2149,7 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu, hlist_add_head(&sp->hash_link, sp_list); if (!direct) { account_shadowed(vcpu->kvm, sp); - if (level == PG_LEVEL_4K && rmap_write_protect(vcpu, gfn)) + if (level == PG_LEVEL_4K && kvm_vcpu_write_protect_gfn(vcpu, gfn)) kvm_flush_remote_tlbs_with_address(vcpu->kvm, gfn, 1); } trace_kvm_mmu_get_page(sp, true); From patchwork Wed Jan 19 23:07:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12717867 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1E8BC433EF for ; Wed, 19 Jan 2022 23:09:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1357700AbiASXJI (ORCPT ); Wed, 19 Jan 2022 18:09:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50310 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344236AbiASXHv (ORCPT ); Wed, 19 Jan 2022 18:07:51 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3DA22C061751 for ; Wed, 19 Jan 2022 15:07:51 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id f1-20020a17090a8e8100b001b44bb75678so2700475pjo.0 for ; Wed, 19 Jan 2022 15:07:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Bvp8e17RJHmgn68Vah/vaY6v0G1B7Poi9MrN2aFSo5g=; b=muNvJHqM8X4S9ixv2lWhzHuetbLR8BcDVG1Wx6rqD1IaWS4bNtQlo9XWiiIbZWyNKm nR3jhl4zHNP+oQb/rtHZQG8Dg4/BMi+Hb0Y3A9jp43hAJfEGvsZ8HHebjCMwFpBM8JyZ RzM74mDTg94Lo2MjMACNgpoUPowFsElw/Ykl3J8N2fhISr29U9Z2y5TvZWecNIHqHXKL d9woTJuiW4voRpWWmHTECta02r7EAb/wZi2yBO2CQh8Ny7Cr1X7SNNlj22TE0m0A7zbN J/8vOJaSHtrAXeGV7pNaJvoOpN2vCPZu5x4YoteTg7agoKfHRO//Z2+bYmFRUS6AZ887 +cTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Bvp8e17RJHmgn68Vah/vaY6v0G1B7Poi9MrN2aFSo5g=; b=CtVJ/SLzwCij6BXkq8O7ukeaz5AqV7XgrSxximVAHarp6wqJb3tPVh417wWwNg+WwE QkwQx5v77qGMohrxABls90TepK4TJeNF6iB56u1OpmcUSbVXiDzgK9WO+INHzr3LfQ/z OeLjp3lNJPmtQdlQD8DmpzNpigACt16c9BWztMdGpbds0R7E01cSJi7hoOvEF0R0lvk3 9qHmvoeDauQK6gS0c4bUQTIYKn8trLIu6rwYXvfqgiUUCK7zxV68bzzZsovjNzPNpyz3 Zs/lyJ8wM+FZXV3x9spQIrf9driAj8+4jtl8H5x/+xFH2olcnhaXOc6eqXL71nJVlcgs +Xtw== X-Gm-Message-State: AOAM5322m6iA04buRiADnzANXjpI/CWwSzPdbm7I0GiTiyg5S6tAF0Ul YwQFLzP/KZoscZI9blu1XEncRhk8FgfJvw== X-Google-Smtp-Source: ABdhPJxdzOuYev/P1jZoL+qyuJRnakrO1gtLhPSB6s9Wos/fZF6h3rG8o09UcwrOoCZIUBEvEqs8uEIpb7HBMg== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a05:6a00:883:b0:4bc:39c1:9644 with SMTP id q3-20020a056a00088300b004bc39c19644mr33083544pfj.14.1642633670737; Wed, 19 Jan 2022 15:07:50 -0800 (PST) Date: Wed, 19 Jan 2022 23:07:23 +0000 In-Reply-To: <20220119230739.2234394-1-dmatlack@google.com> Message-Id: <20220119230739.2234394-3-dmatlack@google.com> Mime-Version: 1.0 References: <20220119230739.2234394-1-dmatlack@google.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog Subject: [PATCH v2 02/18] KVM: x86/mmu: Rename __rmap_write_protect() to rmap_write_protect() From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The function formerly known as rmap_write_protect() has been renamed to kvm_vcpu_write_protect_gfn(), so we can get rid of the double underscores in front of __rmap_write_protect(). No functional change intended. Reviewed-by: Ben Gardon Reviewed-by: Peter Xu Reviewed-by: Sean Christopherson Signed-off-by: David Matlack --- arch/x86/kvm/mmu/mmu.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index b541683c29c7..fb6718714caa 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1229,9 +1229,9 @@ static bool spte_write_protect(u64 *sptep, bool pt_protect) return mmu_spte_update(sptep, spte); } -static bool __rmap_write_protect(struct kvm *kvm, - struct kvm_rmap_head *rmap_head, - bool pt_protect) +static bool rmap_write_protect(struct kvm *kvm, + struct kvm_rmap_head *rmap_head, + bool pt_protect) { u64 *sptep; struct rmap_iterator iter; @@ -1311,7 +1311,7 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm, while (mask) { rmap_head = gfn_to_rmap(slot->base_gfn + gfn_offset + __ffs(mask), PG_LEVEL_4K, slot); - __rmap_write_protect(kvm, rmap_head, false); + rmap_write_protect(kvm, rmap_head, false); /* clear the first set bit */ mask &= mask - 1; @@ -1410,7 +1410,7 @@ bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm, if (kvm_memslots_have_rmaps(kvm)) { for (i = min_level; i <= KVM_MAX_HUGEPAGE_LEVEL; ++i) { rmap_head = gfn_to_rmap(gfn, i, slot); - write_protected |= __rmap_write_protect(kvm, rmap_head, true); + write_protected |= rmap_write_protect(kvm, rmap_head, true); } } @@ -5802,7 +5802,7 @@ static bool slot_rmap_write_protect(struct kvm *kvm, struct kvm_rmap_head *rmap_head, const struct kvm_memory_slot *slot) { - return __rmap_write_protect(kvm, rmap_head, false); + return rmap_write_protect(kvm, rmap_head, false); } void kvm_mmu_slot_remove_write_access(struct kvm *kvm, From patchwork Wed Jan 19 23:07:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12717868 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FE4BC4332F for ; Wed, 19 Jan 2022 23:09:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1357724AbiASXJK (ORCPT ); Wed, 19 Jan 2022 18:09:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50492 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233006AbiASXHx (ORCPT ); Wed, 19 Jan 2022 18:07:53 -0500 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D40FEC061759 for ; Wed, 19 Jan 2022 15:07:52 -0800 (PST) Received: by mail-pf1-x449.google.com with SMTP id n203-20020a628fd4000000b004be3d8ec115so2482461pfd.0 for ; Wed, 19 Jan 2022 15:07:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Q370c26UHWHufnByiPvqbiDjuZrFLKjArAf8vKj7cww=; b=j5J0G6zwiPPCOxXNTGikOijBUS3vUC0uLsRrVNMWf54iB/6xrOSAkGh0wVqLbKftUl Y5pqd27leWvM2Na5my/DLuNlDob/9LehjdApD+mQiaWzqCa/H0vWc+jPpawO4Qo3m1zs LGo8YGqZEc7mvw3Ydk0azkYJfk4GjWDXEqDjKfK4ap6NqeCXRSR86QN0TnT2Qp0Sc6j6 iDj/AzuQKeziecfO3Rxn2mbmvBImMh8FB7tjZB+6KiYVJF7BAgvVz1FULKdEgJCOoyhr ZpT70aD+RP+SwH+3tF3vBqMSATJCUJksnlHcP+uVSheUFdt/bqzsWzcQZ0rU8A3w1rut vDwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Q370c26UHWHufnByiPvqbiDjuZrFLKjArAf8vKj7cww=; b=JPTkd//kItL0exMnPRUMY3ZnOapT88io6niVFZiX2Uq8bqmdvFxpjZA2N/qVIKq+yp RKVX+KQZ8S3iRhsLVLHuuYoPI8bpBUWLS3PpI8rtWVHLPPWNv8C5huon7TgSLyWCHw8E mSHLFmPXWNxB/62qk+fEoqTUDxun54gX6z229o7NMqcosyRi3/K2X/wNjIeb9o/zdXG5 2wBO6gsMobixKTXQIesJ4EU+qzhsjiumuLDI4J+tIWTgUNsv/eiyJpkQICBzVh3Z35zr wDByAGaiHMgSC0kXOeplKEFJCCoUabchihLuVJhvemsAnONytbdrLpa03LmSI7fS640m OdXw== X-Gm-Message-State: AOAM531wlyixoIdr/5LMGoWUeB3GCTMpbjxZQlSSwP/GiaFWQ+cJjMVB UoutoU28ucqcedwCxZUjYvpUdVh7sqAWiw== X-Google-Smtp-Source: ABdhPJx853GHJeu0GL84+krVXeoSbSJq9cD3emKtyCI63mrZEG+hEoGCcbVkev/XDbExROp5SBWbbI3fLv/5Bg== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:903:120a:b0:149:8dd5:f0e1 with SMTP id l10-20020a170903120a00b001498dd5f0e1mr35008348plh.71.1642633672349; Wed, 19 Jan 2022 15:07:52 -0800 (PST) Date: Wed, 19 Jan 2022 23:07:24 +0000 In-Reply-To: <20220119230739.2234394-1-dmatlack@google.com> Message-Id: <20220119230739.2234394-4-dmatlack@google.com> Mime-Version: 1.0 References: <20220119230739.2234394-1-dmatlack@google.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog Subject: [PATCH v2 03/18] KVM: x86/mmu: Automatically update iter->old_spte if cmpxchg fails From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Consolidate a bunch of code that was manually re-reading the spte if the cmpxchg failed. There is no extra cost of doing this because we already have the spte value as a result of the cmpxchg (and in fact this eliminates re-reading the spte), and none of the call sites depend on iter->old_spte retaining the stale spte value. Reviewed-by: Ben Gardon Reviewed-by: Peter Xu Reviewed-by: Sean Christopherson Signed-off-by: David Matlack --- arch/x86/kvm/mmu/tdp_mmu.c | 52 +++++++++++++++++--------------------- 1 file changed, 23 insertions(+), 29 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index bc9e3553fba2..cb7c4e3e9001 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -492,16 +492,23 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, * and handle the associated bookkeeping. Do not mark the page dirty * in KVM's dirty bitmaps. * + * If setting the SPTE fails because it has changed, iter->old_spte will be + * refreshed to the current value of the spte. + * * @kvm: kvm instance * @iter: a tdp_iter instance currently on the SPTE that should be set * @new_spte: The value the SPTE should be set to * Returns: true if the SPTE was set, false if it was not. If false is returned, - * this function will have no side-effects. + * this function will have no side-effects other than setting + * iter->old_spte to the last known value of spte. */ static inline bool tdp_mmu_set_spte_atomic(struct kvm *kvm, struct tdp_iter *iter, u64 new_spte) { + u64 *sptep = rcu_dereference(iter->sptep); + u64 old_spte; + WARN_ON_ONCE(iter->yielded); lockdep_assert_held_read(&kvm->mmu_lock); @@ -517,9 +524,17 @@ static inline bool tdp_mmu_set_spte_atomic(struct kvm *kvm, * Note, fast_pf_fix_direct_spte() can also modify TDP MMU SPTEs and * does not hold the mmu_lock. */ - if (cmpxchg64(rcu_dereference(iter->sptep), iter->old_spte, - new_spte) != iter->old_spte) + old_spte = cmpxchg64(sptep, iter->old_spte, new_spte); + if (old_spte != iter->old_spte) { + /* + * The page table entry was modified by a different logical + * CPU. Refresh iter->old_spte with the current value so the + * caller operates on fresh data, e.g. if it retries + * tdp_mmu_set_spte_atomic(). + */ + iter->old_spte = old_spte; return false; + } __handle_changed_spte(kvm, iter->as_id, iter->gfn, iter->old_spte, new_spte, iter->level, true); @@ -751,11 +766,6 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root, tdp_mmu_set_spte(kvm, &iter, 0); flush = true; } else if (!tdp_mmu_zap_spte_atomic(kvm, &iter)) { - /* - * The iter must explicitly re-read the SPTE because - * the atomic cmpxchg failed. - */ - iter.old_spte = READ_ONCE(*rcu_dereference(iter.sptep)); goto retry; } } @@ -1193,14 +1203,9 @@ static bool wrprot_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root, new_spte = iter.old_spte & ~PT_WRITABLE_MASK; - if (!tdp_mmu_set_spte_atomic(kvm, &iter, new_spte)) { - /* - * The iter must explicitly re-read the SPTE because - * the atomic cmpxchg failed. - */ - iter.old_spte = READ_ONCE(*rcu_dereference(iter.sptep)); + if (!tdp_mmu_set_spte_atomic(kvm, &iter, new_spte)) goto retry; - } + spte_set = true; } @@ -1261,14 +1266,9 @@ static bool clear_dirty_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root, continue; } - if (!tdp_mmu_set_spte_atomic(kvm, &iter, new_spte)) { - /* - * The iter must explicitly re-read the SPTE because - * the atomic cmpxchg failed. - */ - iter.old_spte = READ_ONCE(*rcu_dereference(iter.sptep)); + if (!tdp_mmu_set_spte_atomic(kvm, &iter, new_spte)) goto retry; - } + spte_set = true; } @@ -1392,14 +1392,8 @@ static void zap_collapsible_spte_range(struct kvm *kvm, continue; /* Note, a successful atomic zap also does a remote TLB flush. */ - if (!tdp_mmu_zap_spte_atomic(kvm, &iter)) { - /* - * The iter must explicitly re-read the SPTE because - * the atomic cmpxchg failed. - */ - iter.old_spte = READ_ONCE(*rcu_dereference(iter.sptep)); + if (!tdp_mmu_zap_spte_atomic(kvm, &iter)) goto retry; - } } rcu_read_unlock(); From patchwork Wed Jan 19 23:07:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12717883 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27022C433F5 for ; Wed, 19 Jan 2022 23:09:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344613AbiASXJt (ORCPT ); Wed, 19 Jan 2022 18:09:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50494 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241922AbiASXHy (ORCPT ); Wed, 19 Jan 2022 18:07:54 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A479C06175A for ; Wed, 19 Jan 2022 15:07:54 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id mn21-20020a17090b189500b001b4fa60efcbso2441019pjb.2 for ; Wed, 19 Jan 2022 15:07:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=HuPbbBZmFDP457aUW0hKNepMxy+Xj/+aJ5rP29iDMxk=; b=j9FEtusOAOCC78Snr6/jpsZHIwo9CVKgRS6UeK8GXTITS6YkOPs0CE+0euzVtZmuBx EfJPbd6knPZ9PqlyDAbGR5WWpL7CTy65u0I0TKIDW3NDz1ASVyTEsEvktqmP7n+LPH5p MYSHq4A2ViJcwtm7QZw4pO3fqkb4HdOfbJ2Tstp8VaA/ZMziD5+htpZ1k5+1FFqkBfG/ nrhM72a4Uzs+5WLmrCVmqKvq8b77Y0G8sWaxhHqk3+oxcQe0tMlqb5q2ovZApUjUOeU0 GuoNoCau8bUY6lHEKSkUHiv4q4OfnEOUl+f2qKvSkyOD9k9RlX7enhHj19zg37mUjlYb PvLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=HuPbbBZmFDP457aUW0hKNepMxy+Xj/+aJ5rP29iDMxk=; b=GYsjMsmxKkrYMBoqn3zykMZu+V0dOnREaw1O75twsHuYiju4JDIwjdbNXpHcVfip0U xmSPACa3Mvk3piww3cHgDECypYgx3W6J4nUKLaX9KJfdD4/LU9j4vlDJrxXR8u+myLjs NM+6JSUEZp6+jD+B4jkxcvLZVNZNuvnIrKnRv0CojR/MghNhbAJhKC1aeY2LGYxDOh0D 5D3qY8+C1+DntfS70AP/UknIpw4caOi3PRWnO/GatKbccAZKe+wK7YkBYgBEyNxZQxpg iyFPK0nkZhWkrWa/HfsqK8xLvDO8P/sqOknaX4O8p1uAim473xorJmvX1P3tzy6srnUV c7cA== X-Gm-Message-State: AOAM530+imj+UNrsT6/5udLKQUJtxFfYADGS6dIBwoyRO9BdVATyT2eX PM7I9xd7b/gPMC+Cxunc8A9sDK10PXATzg== X-Google-Smtp-Source: ABdhPJzw9tNYW6JacuwK7RnjzHjtwOdnz6rNkwVjqy0W2gJQbxwVQZj5EBpEFfFNTcrYeN8tN9gE/Igzu7UW9g== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a05:6a00:ac4:b0:4bd:6555:1746 with SMTP id c4-20020a056a000ac400b004bd65551746mr33000183pfl.39.1642633673958; Wed, 19 Jan 2022 15:07:53 -0800 (PST) Date: Wed, 19 Jan 2022 23:07:25 +0000 In-Reply-To: <20220119230739.2234394-1-dmatlack@google.com> Message-Id: <20220119230739.2234394-5-dmatlack@google.com> Mime-Version: 1.0 References: <20220119230739.2234394-1-dmatlack@google.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog Subject: [PATCH v2 04/18] KVM: x86/mmu: Change tdp_mmu_{set,zap}_spte_atomic() to return 0/-EBUSY From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org tdp_mmu_set_spte_atomic() and tdp_mmu_zap_spte_atomic() return a bool with true indicating the SPTE modification was successful and false indicating failure. Change these functions to return an int instead since that is the common practice. Opportunistically fix up the kernel-doc style for the Return section above tdp_mmu_set_spte_atomic(). No functional change intended. Suggested-by: Sean Christopherson Signed-off-by: David Matlack --- arch/x86/kvm/mmu/tdp_mmu.c | 47 +++++++++++++++++++++----------------- 1 file changed, 26 insertions(+), 21 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index cb7c4e3e9001..3dc2e2a6d439 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -498,13 +498,15 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, * @kvm: kvm instance * @iter: a tdp_iter instance currently on the SPTE that should be set * @new_spte: The value the SPTE should be set to - * Returns: true if the SPTE was set, false if it was not. If false is returned, - * this function will have no side-effects other than setting - * iter->old_spte to the last known value of spte. + * Return: + * * 0 - If the SPTE was set. + * * -EBUSY - If the SPTE cannot be set. In this case this function will have + * no side-effects other than setting iter->old_spte to the last + * known value of the spte. */ -static inline bool tdp_mmu_set_spte_atomic(struct kvm *kvm, - struct tdp_iter *iter, - u64 new_spte) +static inline int tdp_mmu_set_spte_atomic(struct kvm *kvm, + struct tdp_iter *iter, + u64 new_spte) { u64 *sptep = rcu_dereference(iter->sptep); u64 old_spte; @@ -518,7 +520,7 @@ static inline bool tdp_mmu_set_spte_atomic(struct kvm *kvm, * may modify it. */ if (is_removed_spte(iter->old_spte)) - return false; + return -EBUSY; /* * Note, fast_pf_fix_direct_spte() can also modify TDP MMU SPTEs and @@ -533,27 +535,30 @@ static inline bool tdp_mmu_set_spte_atomic(struct kvm *kvm, * tdp_mmu_set_spte_atomic(). */ iter->old_spte = old_spte; - return false; + return -EBUSY; } __handle_changed_spte(kvm, iter->as_id, iter->gfn, iter->old_spte, new_spte, iter->level, true); handle_changed_spte_acc_track(iter->old_spte, new_spte, iter->level); - return true; + return 0; } -static inline bool tdp_mmu_zap_spte_atomic(struct kvm *kvm, - struct tdp_iter *iter) +static inline int tdp_mmu_zap_spte_atomic(struct kvm *kvm, + struct tdp_iter *iter) { + int ret; + /* * Freeze the SPTE by setting it to a special, * non-present value. This will stop other threads from * immediately installing a present entry in its place * before the TLBs are flushed. */ - if (!tdp_mmu_set_spte_atomic(kvm, iter, REMOVED_SPTE)) - return false; + ret = tdp_mmu_set_spte_atomic(kvm, iter, REMOVED_SPTE); + if (ret) + return ret; kvm_flush_remote_tlbs_with_address(kvm, iter->gfn, KVM_PAGES_PER_HPAGE(iter->level)); @@ -568,7 +573,7 @@ static inline bool tdp_mmu_zap_spte_atomic(struct kvm *kvm, */ WRITE_ONCE(*rcu_dereference(iter->sptep), 0); - return true; + return 0; } @@ -765,7 +770,7 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root, if (!shared) { tdp_mmu_set_spte(kvm, &iter, 0); flush = true; - } else if (!tdp_mmu_zap_spte_atomic(kvm, &iter)) { + } else if (tdp_mmu_zap_spte_atomic(kvm, &iter)) { goto retry; } } @@ -923,7 +928,7 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu, if (new_spte == iter->old_spte) ret = RET_PF_SPURIOUS; - else if (!tdp_mmu_set_spte_atomic(vcpu->kvm, iter, new_spte)) + else if (tdp_mmu_set_spte_atomic(vcpu->kvm, iter, new_spte)) return RET_PF_RETRY; /* @@ -989,7 +994,7 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) */ if (is_shadow_present_pte(iter.old_spte) && is_large_pte(iter.old_spte)) { - if (!tdp_mmu_zap_spte_atomic(vcpu->kvm, &iter)) + if (tdp_mmu_zap_spte_atomic(vcpu->kvm, &iter)) break; /* @@ -1015,7 +1020,7 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) new_spte = make_nonleaf_spte(child_pt, !shadow_accessed_mask); - if (tdp_mmu_set_spte_atomic(vcpu->kvm, &iter, new_spte)) { + if (!tdp_mmu_set_spte_atomic(vcpu->kvm, &iter, new_spte)) { tdp_mmu_link_page(vcpu->kvm, sp, fault->huge_page_disallowed && fault->req_level >= iter.level); @@ -1203,7 +1208,7 @@ static bool wrprot_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root, new_spte = iter.old_spte & ~PT_WRITABLE_MASK; - if (!tdp_mmu_set_spte_atomic(kvm, &iter, new_spte)) + if (tdp_mmu_set_spte_atomic(kvm, &iter, new_spte)) goto retry; spte_set = true; @@ -1266,7 +1271,7 @@ static bool clear_dirty_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root, continue; } - if (!tdp_mmu_set_spte_atomic(kvm, &iter, new_spte)) + if (tdp_mmu_set_spte_atomic(kvm, &iter, new_spte)) goto retry; spte_set = true; @@ -1392,7 +1397,7 @@ static void zap_collapsible_spte_range(struct kvm *kvm, continue; /* Note, a successful atomic zap also does a remote TLB flush. */ - if (!tdp_mmu_zap_spte_atomic(kvm, &iter)) + if (tdp_mmu_zap_spte_atomic(kvm, &iter)) goto retry; } From patchwork Wed Jan 19 23:07:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12717882 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E6DEC433FE for ; Wed, 19 Jan 2022 23:09:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344443AbiASXJr (ORCPT ); Wed, 19 Jan 2022 18:09:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50504 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243366AbiASXH4 (ORCPT ); Wed, 19 Jan 2022 18:07:56 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D5FEC06175B for ; Wed, 19 Jan 2022 15:07:56 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id u133-20020a63798b000000b0034c0630b044so2525398pgc.3 for ; Wed, 19 Jan 2022 15:07:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=2+TXnk2CNBszqcZB7JPnnvcTPXMeVHBO+bdIpTEIGzA=; b=dcUwx2rlgs36HWTHcqS54KNlxs3522SVb8E+ZJSZLgRQj5YUXhNEcYhQanKJO97DsQ NVYzJsyj103+jglwiGuP07jwbtdkUBNVOtmi6IlboSNW+oPrXXWiq8ebhNQ8Q9+g/mm4 JIdIjRAgazTyk7aGa2k4rXQG8H/5wfECZt0XifvSwOg+sMI3tZWOtr8Eo1uZl8znNGAK HpSgYv2XbfjXGIq1zxQ73R3ML7OVzl0ybf/Ek6cIxTBSxaEKiRu77g0A9HlpjMLqV2DR xur1m7iEpek0wrLEqlYoznvSR2SiVpC0OnMhkw99A4XSJxeMXlEUnWEgYcyDnkj4ZC4u p6Pg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=2+TXnk2CNBszqcZB7JPnnvcTPXMeVHBO+bdIpTEIGzA=; b=SRlWuXnrujLB6pqG8sRL6FmBPM2xhwJRQy9bcqn5qFMZpSd8ZwyUFJ9DcxmLy3QW7F eoVxNnZ8oOX/dSYBkwsgaa+2SbNPihjYj2mqM98HxVBozyL8FSxC+kG9S7eWb7A4P1ID MSGCAI0PO9nyFotv7906RygUENyezjCrGhl98kq82fyY9UV1KjHYxKrO+8KNTTQvITqn mCd6sXsnKimXJcw799bOH98zb84USshZCYs+aQ7mIDisaW5M8HEn9BOiKopllI4LjarM aYqvFyL3ip1BcalD6lH+hhXfYjdMhk1Gez2tOHSqLAUU8L+APDM8lUBxM+2+t3h/RENm Lj4Q== X-Gm-Message-State: AOAM532FWOhepN1TbuqokwpiZKYYIVd21Ubyr7/rO0Se+8wSRpbGUmEE ftfpjbCPstJTjOw1UVOcwn8RpKcWRbFZJQ== X-Google-Smtp-Source: ABdhPJwBE5ZSNm9oSfmnkMUO2mX+3jPIuIhZ+UdP7SoKsH4QAzfbGu02OyhLPalfoM6h9Zwo8/PP5XCvBmwiEw== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a05:6a00:21c7:b0:4bc:1d4d:dfe with SMTP id t7-20020a056a0021c700b004bc1d4d0dfemr32722511pfj.15.1642633675603; Wed, 19 Jan 2022 15:07:55 -0800 (PST) Date: Wed, 19 Jan 2022 23:07:26 +0000 In-Reply-To: <20220119230739.2234394-1-dmatlack@google.com> Message-Id: <20220119230739.2234394-6-dmatlack@google.com> Mime-Version: 1.0 References: <20220119230739.2234394-1-dmatlack@google.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog Subject: [PATCH v2 05/18] KVM: x86/mmu: Rename TDP MMU functions that handle shadow pages From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Rename 3 functions in tdp_mmu.c that handle shadow pages: alloc_tdp_mmu_page() -> tdp_mmu_alloc_sp() tdp_mmu_link_page() -> tdp_mmu_link_sp() tdp_mmu_unlink_page() -> tdp_mmu_unlink_sp() These changed make tdp_mmu a consistent prefix before the verb in the function name, and make it more clear that these functions deal with kvm_mmu_page structs rather than struct pages. One could argue that "shadow page" is the wrong term for a page table in the TDP MMU since it never actually shadows a guest page table. However, "shadow page" (or "sp" for short) has evolved to become the standard term in KVM when referring to a kvm_mmu_page struct, and its associated page table and other metadata, regardless of whether the page table shadows a guest page table. So this commit just makes the TDP MMU more consistent with the rest of KVM. No functional change intended. Signed-off-by: David Matlack --- arch/x86/kvm/mmu/tdp_mmu.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 3dc2e2a6d439..15cce503ffde 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -172,8 +172,8 @@ static union kvm_mmu_page_role page_role_for_level(struct kvm_vcpu *vcpu, return role; } -static struct kvm_mmu_page *alloc_tdp_mmu_page(struct kvm_vcpu *vcpu, gfn_t gfn, - int level) +static struct kvm_mmu_page *tdp_mmu_alloc_sp(struct kvm_vcpu *vcpu, gfn_t gfn, + int level) { struct kvm_mmu_page *sp; @@ -207,7 +207,7 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu) goto out; } - root = alloc_tdp_mmu_page(vcpu, 0, vcpu->arch.mmu->shadow_root_level); + root = tdp_mmu_alloc_sp(vcpu, 0, vcpu->arch.mmu->shadow_root_level); refcount_set(&root->tdp_mmu_root_count, 1); spin_lock(&kvm->arch.tdp_mmu_pages_lock); @@ -252,15 +252,15 @@ static void handle_changed_spte_dirty_log(struct kvm *kvm, int as_id, gfn_t gfn, } /** - * tdp_mmu_link_page - Add a new page to the list of pages used by the TDP MMU + * tdp_mmu_link_sp() - Add a new shadow page to the list of used pages * * @kvm: kvm instance * @sp: the new page * @account_nx: This page replaces a NX large page and should be marked for * eventual reclaim. */ -static void tdp_mmu_link_page(struct kvm *kvm, struct kvm_mmu_page *sp, - bool account_nx) +static void tdp_mmu_link_sp(struct kvm *kvm, struct kvm_mmu_page *sp, + bool account_nx) { spin_lock(&kvm->arch.tdp_mmu_pages_lock); list_add(&sp->link, &kvm->arch.tdp_mmu_pages); @@ -270,7 +270,7 @@ static void tdp_mmu_link_page(struct kvm *kvm, struct kvm_mmu_page *sp, } /** - * tdp_mmu_unlink_page - Remove page from the list of pages used by the TDP MMU + * tdp_mmu_unlink_sp() - Remove a shadow page from the list of used pages * * @kvm: kvm instance * @sp: the page to be removed @@ -278,8 +278,8 @@ static void tdp_mmu_link_page(struct kvm *kvm, struct kvm_mmu_page *sp, * the MMU lock and the operation must synchronize with other * threads that might be adding or removing pages. */ -static void tdp_mmu_unlink_page(struct kvm *kvm, struct kvm_mmu_page *sp, - bool shared) +static void tdp_mmu_unlink_sp(struct kvm *kvm, struct kvm_mmu_page *sp, + bool shared) { if (shared) spin_lock(&kvm->arch.tdp_mmu_pages_lock); @@ -321,7 +321,7 @@ static void handle_removed_tdp_mmu_page(struct kvm *kvm, tdp_ptep_t pt, trace_kvm_mmu_prepare_zap_page(sp); - tdp_mmu_unlink_page(kvm, sp, shared); + tdp_mmu_unlink_sp(kvm, sp, shared); for (i = 0; i < PT64_ENT_PER_PAGE; i++) { u64 *sptep = rcu_dereference(pt) + i; @@ -1014,16 +1014,16 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) if (is_removed_spte(iter.old_spte)) break; - sp = alloc_tdp_mmu_page(vcpu, iter.gfn, iter.level - 1); + sp = tdp_mmu_alloc_sp(vcpu, iter.gfn, iter.level - 1); child_pt = sp->spt; new_spte = make_nonleaf_spte(child_pt, !shadow_accessed_mask); if (!tdp_mmu_set_spte_atomic(vcpu->kvm, &iter, new_spte)) { - tdp_mmu_link_page(vcpu->kvm, sp, - fault->huge_page_disallowed && - fault->req_level >= iter.level); + tdp_mmu_link_sp(vcpu->kvm, sp, + fault->huge_page_disallowed && + fault->req_level >= iter.level); trace_kvm_mmu_get_page(sp, true); } else { From patchwork Wed Jan 19 23:07:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12717869 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99D79C433EF for ; Wed, 19 Jan 2022 23:09:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1357732AbiASXJL (ORCPT ); Wed, 19 Jan 2022 18:09:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50512 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343638AbiASXH6 (ORCPT ); Wed, 19 Jan 2022 18:07:58 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D667C06175C for ; Wed, 19 Jan 2022 15:07:58 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id q3-20020a638c43000000b0034c9c0fb2d2so2488969pgn.22 for ; Wed, 19 Jan 2022 15:07:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=YDqrNo/wldgkip0bjkyjtlYC24PWdJYaAJiVksfVa1s=; b=pI6o0jOUFXYEfIpTgP1sKyK34MbesDr4s9XQNKYpB5+6eeJuz9urLWcDbJWTbdBj3K EnYrWlVWHCJG43hl+szowXw7XKp6pBkR0QIGXGG3NAkdrxeeHj6Wp12m9/kPuz5kLmkM TuYlOS+R5Zfe0KcdiztQnlwnmZcBEm4U+u1pG4RFnTrfdOfHf+Lh9TuANno4tDR8RxBP EM0G/sRzNn9LjlWXxO1JJAvubiyWiD22PnUCc6pO/pjQ6caAODelxoTiUCScfK/yOH5i 5ZIdq7LXbv9WA9I9Vqoztfdre8mNHyx0h3KfO2GJQ8T/PlC9LsGtFKJgp25cE0RF0t7d TXtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=YDqrNo/wldgkip0bjkyjtlYC24PWdJYaAJiVksfVa1s=; b=4Y7CW7Olfej87mu2qyqchIHBmGDguaDQ7yODjWufYk8bI6UxfTcXivQXyPeLGAZXXd 04U55rVeqNX4pOwFXs4z7W5+t6plxcnka8njgZmXV4lj1C1xzztohxnWTJpLfNmzCxLE ODrFo5eHGOUYo9gOTmqn/nYArnF1taApNJERBhKXc9Yq5wjOMGD79iBzUV5rd9/f+Nq6 5Dl5UJV4058ez/1gNzzIPPWVPGfv6cX5EODLTK+JR2XxNjID9aiMqlYnIpGLpgNfL9Ny Ku6Xy0CouFCR1wBNf297MZW5b9Gud0AnJQO4m1V2E/rCiyX+REhHGB6lwwsOFTzeBt6r n3eg== X-Gm-Message-State: AOAM532qLTSmCF2quhDND8iA/yTlzZ2qaPrSmFZAh+1bZYqxIneptbXL qcOznmmriCWvwLJ0eGWdrHJx8Gx+87VCvw== X-Google-Smtp-Source: ABdhPJxC3RGwtAncFp0wYVbbZ2N9aYR6GBFCFJtxe8PRZ2utdWX60ujqYUINh4yx92vGsYBiRKEg0vafeL1S+g== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:90b:4d84:: with SMTP id oj4mr741490pjb.0.1642633677370; Wed, 19 Jan 2022 15:07:57 -0800 (PST) Date: Wed, 19 Jan 2022 23:07:27 +0000 In-Reply-To: <20220119230739.2234394-1-dmatlack@google.com> Message-Id: <20220119230739.2234394-7-dmatlack@google.com> Mime-Version: 1.0 References: <20220119230739.2234394-1-dmatlack@google.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog Subject: [PATCH v2 06/18] KVM: x86/mmu: Rename handle_removed_tdp_mmu_page() to handle_removed_pt() From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org First remove tdp_mmu_ from the name since it is redundant given that it is a static function in tdp_mmu.c. There is a pattern of using tdp_mmu_ as a prefix in the names of static TDP MMU functions, but all of the other handle_*() variants do not include such a prefix. So drop it entirely. Then change "page" to "pt" to convey that this is operating on a page table rather than an struct page. Purposely use "pt" instead of "sp" since this function takes the raw RCU-protected page table pointer as an argument rather than a pointer to the struct kvm_mmu_page. No functional change intended. Signed-off-by: David Matlack --- arch/x86/kvm/mmu/tdp_mmu.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 15cce503ffde..902dd6e49e50 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -295,7 +295,7 @@ static void tdp_mmu_unlink_sp(struct kvm *kvm, struct kvm_mmu_page *sp, } /** - * handle_removed_tdp_mmu_page - handle a pt removed from the TDP structure + * handle_removed_pt() - handle a page table removed from the TDP structure * * @kvm: kvm instance * @pt: the page removed from the paging structure @@ -311,8 +311,7 @@ static void tdp_mmu_unlink_sp(struct kvm *kvm, struct kvm_mmu_page *sp, * this thread will be responsible for ensuring the page is freed. Hence the * early rcu_dereferences in the function. */ -static void handle_removed_tdp_mmu_page(struct kvm *kvm, tdp_ptep_t pt, - bool shared) +static void handle_removed_pt(struct kvm *kvm, tdp_ptep_t pt, bool shared) { struct kvm_mmu_page *sp = sptep_to_sp(rcu_dereference(pt)); int level = sp->role.level; @@ -472,8 +471,7 @@ static void __handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, * the paging structure. */ if (was_present && !was_leaf && (pfn_changed || !is_present)) - handle_removed_tdp_mmu_page(kvm, - spte_to_child_pt(old_spte, level), shared); + handle_removed_pt(kvm, spte_to_child_pt(old_spte, level), shared); } static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, From patchwork Wed Jan 19 23:07:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12717880 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D42FDC433F5 for ; Wed, 19 Jan 2022 23:09:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344272AbiASXJn (ORCPT ); Wed, 19 Jan 2022 18:09:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50522 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240071AbiASXH7 (ORCPT ); Wed, 19 Jan 2022 18:07:59 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BC8B9C06175D for ; Wed, 19 Jan 2022 15:07:59 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id j186-20020a636ec3000000b00340c5f3a0cbso2542219pgc.0 for ; Wed, 19 Jan 2022 15:07:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=M/E6s/KdP8Nhq2HrOUVfsx3LUjXAmeogBGtH3TxkicY=; b=YqoHC8NPXtG95qTU5rAxxQgNFb8IjTkeXRsA4alfO+8aIptQbt4vJ/HtpODcbD2K2K agkI7RtojQkj2rPvIRQdrZ5I+6D10xLraJ1hmaKgy/vrLkgHuHA6Je0bWbZzYbcPI3pb SkYSl0Q18beAlW8hq5dtJHbE0itxdEKgRoEHmkQceRpasZ2pJXyDz91QPf38baIabqy4 c75NmSyNbozY4Rc1K9+2u6BW/Ax68KQLhqCDBCT0zrLziZ3arVmYQBQXoXxT8aap0VaD oC5bKDmibnqtZfYkrbyRQ6iYxQCKmOqZ2nKX0iCqCOqeXk4wf0esl0A6Kgs2AETjCq+G P2UA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=M/E6s/KdP8Nhq2HrOUVfsx3LUjXAmeogBGtH3TxkicY=; b=rIKODSUJng1Q5ck0ExSD91007bcgYbN5WMdqXTqzbxCoNT8vsSwJaSGsaUAammw9+y QoLw7w+ko6DVwT2gO/VLBOHQETZFU/jV5c/G8hPigppW0OGCkd/tjKDV4XYWAb6wmi8D WAJHfxTJaU2rUs4GybQ/dmLEz4U2MustnOfcYoqSw9jwis7TF5T1AmAzqDq5jCrOXJRl 0d3p57AaPKnFcRhpiNFaJ1VNJSiHFqeLMmSsuHzsDGOPKViWIfmwmfmLtPCP5NOSlZBq H/ok705RwyULiSVrVb4lRMiSSOewQ8e9hGj2+GMxUuQlH3UWEKKO98LIAXmHHGUpEcpX LwHA== X-Gm-Message-State: AOAM531mOQ65r01XfK/7r40vdr8p2L1XGJt5wvCV2dwg9VNuN/qQHU2x ZLJ48+GLrDSUxp4KQW4qN85IAzRNMLDSrw== X-Google-Smtp-Source: ABdhPJx7XqrEq/4TJkaQcjPDMOPEQRIHEJJFKAfaOUpV0DS4BqYcdJqAYuJKc52gCaZvyBBGoSyOIts+HR4Y1g== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:90a:7bce:: with SMTP id d14mr965518pjl.7.1642633679272; Wed, 19 Jan 2022 15:07:59 -0800 (PST) Date: Wed, 19 Jan 2022 23:07:28 +0000 In-Reply-To: <20220119230739.2234394-1-dmatlack@google.com> Message-Id: <20220119230739.2234394-8-dmatlack@google.com> Mime-Version: 1.0 References: <20220119230739.2234394-1-dmatlack@google.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog Subject: [PATCH v2 07/18] KVM: x86/mmu: Consolidate logic to atomically install a new TDP MMU page table From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Consolidate the logic to atomically replace an SPTE with an SPTE that points to a new page table into a single helper function. This will be used in a follow-up commit to split huge pages, which involves replacing each huge page SPTE with an SPTE that points to a page table. Opportunistically drop the call to trace_kvm_mmu_get_page() in kvm_tdp_mmu_map() since it is redundant with the identical tracepoint in tdp_mmu_alloc_sp(). No functional change intended. Reviewed-by: Peter Xu Signed-off-by: David Matlack --- arch/x86/kvm/mmu/tdp_mmu.c | 68 ++++++++++++++++++++------------------ 1 file changed, 36 insertions(+), 32 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 902dd6e49e50..f6144db48367 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -251,24 +251,6 @@ static void handle_changed_spte_dirty_log(struct kvm *kvm, int as_id, gfn_t gfn, } } -/** - * tdp_mmu_link_sp() - Add a new shadow page to the list of used pages - * - * @kvm: kvm instance - * @sp: the new page - * @account_nx: This page replaces a NX large page and should be marked for - * eventual reclaim. - */ -static void tdp_mmu_link_sp(struct kvm *kvm, struct kvm_mmu_page *sp, - bool account_nx) -{ - spin_lock(&kvm->arch.tdp_mmu_pages_lock); - list_add(&sp->link, &kvm->arch.tdp_mmu_pages); - if (account_nx) - account_huge_nx_page(kvm, sp); - spin_unlock(&kvm->arch.tdp_mmu_pages_lock); -} - /** * tdp_mmu_unlink_sp() - Remove a shadow page from the list of used pages * @@ -959,6 +941,38 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu, return ret; } +/* + * tdp_mmu_link_sp_atomic - Atomically replace the given spte with an spte + * pointing to the provided page table. + * + * @kvm: kvm instance + * @iter: a tdp_iter instance currently on the SPTE that should be set + * @sp: The new TDP page table to install. + * @account_nx: True if this page table is being installed to split a + * non-executable huge page. + * + * Returns: 0 if the new page table was installed. Non-0 if the page table + * could not be installed (e.g. the atomic compare-exchange failed). + */ +static int tdp_mmu_link_sp_atomic(struct kvm *kvm, struct tdp_iter *iter, + struct kvm_mmu_page *sp, bool account_nx) +{ + u64 spte = make_nonleaf_spte(sp->spt, !shadow_accessed_mask); + int ret; + + ret = tdp_mmu_set_spte_atomic(kvm, iter, spte); + if (ret) + return ret; + + spin_lock(&kvm->arch.tdp_mmu_pages_lock); + list_add(&sp->link, &kvm->arch.tdp_mmu_pages); + if (account_nx) + account_huge_nx_page(kvm, sp); + spin_unlock(&kvm->arch.tdp_mmu_pages_lock); + + return 0; +} + /* * Handle a TDP page fault (NPT/EPT violation/misconfiguration) by installing * page tables and SPTEs to translate the faulting guest physical address. @@ -968,8 +982,6 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) struct kvm_mmu *mmu = vcpu->arch.mmu; struct tdp_iter iter; struct kvm_mmu_page *sp; - u64 *child_pt; - u64 new_spte; int ret; kvm_mmu_hugepage_adjust(vcpu, fault); @@ -1004,6 +1016,9 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) } if (!is_shadow_present_pte(iter.old_spte)) { + bool account_nx = fault->huge_page_disallowed && + fault->req_level >= iter.level; + /* * If SPTE has been frozen by another thread, just * give up and retry, avoiding unnecessary page table @@ -1013,18 +1028,7 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) break; sp = tdp_mmu_alloc_sp(vcpu, iter.gfn, iter.level - 1); - child_pt = sp->spt; - - new_spte = make_nonleaf_spte(child_pt, - !shadow_accessed_mask); - - if (!tdp_mmu_set_spte_atomic(vcpu->kvm, &iter, new_spte)) { - tdp_mmu_link_sp(vcpu->kvm, sp, - fault->huge_page_disallowed && - fault->req_level >= iter.level); - - trace_kvm_mmu_get_page(sp, true); - } else { + if (tdp_mmu_link_sp_atomic(vcpu->kvm, &iter, sp, account_nx)) { tdp_mmu_free_sp(sp); break; } From patchwork Wed Jan 19 23:07:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12717881 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90B6DC433EF for ; Wed, 19 Jan 2022 23:09:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344502AbiASXJp (ORCPT ); Wed, 19 Jan 2022 18:09:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50530 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344302AbiASXIB (ORCPT ); Wed, 19 Jan 2022 18:08:01 -0500 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7A779C06175E for ; Wed, 19 Jan 2022 15:08:01 -0800 (PST) Received: by mail-pg1-x549.google.com with SMTP id t27-20020a63461b000000b00342c204e4f3so2510499pga.11 for ; Wed, 19 Jan 2022 15:08:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=bFqSO9GhIrq7/q8ry8SyB115cIj+EzEpZTQAJaeeMIs=; b=X4NmAkWWyB4nE67ptW29AuU7BCKuQoryzYksX7w0p/BzEIgdJq6u7km1g10xXyUQ0U YinvhuwI/cf1zzzbz9MdGtHg/oyCBypC1/4i3/mmQFE6TzpXDWCt64BuDgG8xs4HrEeh /CAoZWkzbagMeq05YKn8RlCGgM1onVUyf03JpCaEVdf0+MOBylGF3Vq+dAQ76pw1fF68 5QD5U/9TgEYvfQAlWAChC5PWiFWdO81IM/eGktXstn1n8jJDukhqRwkLQeVStsgF0g// xXBjiEzAEd4tFJTogF3vldznG6ppFaV5DNnDhJHijfymjL1dM/oYS1UXJ+fCfhVT6NaX omYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=bFqSO9GhIrq7/q8ry8SyB115cIj+EzEpZTQAJaeeMIs=; b=MRjdfN857DAerVyjl1YT0RCe506HMpdfYDOUHrNu48t+72UAbsZErgxgv5UuxXD+9O xWnXJ+jiUlD1EyeFokGNr97z5Qb2zLqpVPAHD4Oxe8f/nUnIVFBnrglK+CwqAwbUQy+k kC6kCaZdD7fcQ3M+2DWZuf/YRw3HIxlmkxVkWcXOsLmNdPBvo1SFlDAg1/Xlr8m+1mFn CN0KtPcfy2qewu+YKuMGoq06VYJ7lOVy7LD8wtvUVIjtJ0qerGi8gqIpxBWG/LsSBznh C8j2kpNwDaPKzWnw2p3pGPNl2CqR0l8u+m9Zj4u+1rvyuUpBDwrJCBEZKtrLub7k0iCj qlng== X-Gm-Message-State: AOAM531VnCS65PvJSQJdMh8ZI+6ZZkrslBHffn6QcsAVCXs6osRTQoen 1idqYxE+EfrMaXLpPWiM12yFSDu14sVw4A== X-Google-Smtp-Source: ABdhPJyIzi1D797C8LkwEjjKFr8fCIu/AYIXhZzIeEP59gVyjM/BZSMOzh7duMK+KtpkqIIvpF9kMgxyTBsr4w== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a05:6a00:2181:b0:4c1:3b84:b43f with SMTP id h1-20020a056a00218100b004c13b84b43fmr32747307pfi.50.1642633680986; Wed, 19 Jan 2022 15:08:00 -0800 (PST) Date: Wed, 19 Jan 2022 23:07:29 +0000 In-Reply-To: <20220119230739.2234394-1-dmatlack@google.com> Message-Id: <20220119230739.2234394-9-dmatlack@google.com> Mime-Version: 1.0 References: <20220119230739.2234394-1-dmatlack@google.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog Subject: [PATCH v2 08/18] KVM: x86/mmu: Remove unnecessary warnings from restore_acc_track_spte() From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The warnings in restore_acc_track_spte() can be removed because the only caller checks is_access_track_spte(), and is_access_track_spte() checks !spte_ad_enabled(). In other words, the warning can never be triggered. No functional change intended. Suggested-by: Sean Christopherson Signed-off-by: David Matlack --- arch/x86/kvm/mmu/mmu.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index fb6718714caa..4485554336d7 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -653,9 +653,6 @@ static u64 restore_acc_track_spte(u64 spte) u64 saved_bits = (spte >> SHADOW_ACC_TRACK_SAVED_BITS_SHIFT) & SHADOW_ACC_TRACK_SAVED_BITS_MASK; - WARN_ON_ONCE(spte_ad_enabled(spte)); - WARN_ON_ONCE(!is_access_track_spte(spte)); - new_spte &= ~shadow_acc_track_mask; new_spte &= ~(SHADOW_ACC_TRACK_SAVED_BITS_MASK << SHADOW_ACC_TRACK_SAVED_BITS_SHIFT); From patchwork Wed Jan 19 23:07:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12717879 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4051DC433F5 for ; Wed, 19 Jan 2022 23:09:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1357960AbiASXJf (ORCPT ); Wed, 19 Jan 2022 18:09:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50270 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344102AbiASXID (ORCPT ); Wed, 19 Jan 2022 18:08:03 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3CFC1C06175F for ; Wed, 19 Jan 2022 15:08:03 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id o7-20020a17090a3d4700b001b4243e9ea2so2664772pjf.6 for ; Wed, 19 Jan 2022 15:08:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=SzpUxNwCpNHnqmiexHxG+Qtmmb9j6ESj4QKd4NHNfOQ=; b=C+r/BHH/umnNW0kEXIkSCoIWX49+z4SnFuuyW6miFrnrmQ+77SO1VkA3slUrrtZql4 5GRYMnWLqFe4zeeDoO0GEbnfe5pJdzWuPwx0UtA+1nwhBdGVYol3Wz9orTOvjDAmgim9 lCya+7edLKEfXTMjeQ9XFDviHkJx1c3BiyKerS7ubQGJa54fLsTcdX5fqTUbOhvQFBCx hwWbe/sWSbrtfx+pRNIzH09KNS9PwmXHeItEuQomCCblfgop6EJI2peGttCv2AX+JOr8 vWMc/PYHMWIYoWw0uxREj0Ffv8CFpMb9TkMWhZvii2Dti0/BhvToaGUXZhIdEve0TFr/ lg6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=SzpUxNwCpNHnqmiexHxG+Qtmmb9j6ESj4QKd4NHNfOQ=; b=1GDIWC7PcbjSTtlWwgj41O5/NhKITYG+Ktk9TUjJUktyv3q0/efPtJ73X59PZFboih +ivBv1onQZrP7cc4cW84ZdzuMjtGERlJr/e/krJAu+zho8VPKA8mX+16spe5lB/ogk45 AnG2/6bueGyMvAzxJV8QjUBPyulMbQ9Dliz7FZ2Mk3Hgt6U/ir/Esu6K+IhHNmUwKCzY PQVKaysUD/NzIvXOAMM1KBXuS07QOm7oJsbPK7Ib1iL+nlU/twVUorYgGvZZIcQA5Ngi SwrAEW0huvfqrPp8rW3LN1rKTUP87T3mcmDkEwTq/Wr5gdsSPbQ3UYdUWwvb2EG7lY0O KfrQ== X-Gm-Message-State: AOAM531dGClW7hpS2TRtOXNOLXG3PXPBhyGZtIt76MBBKknkAhWsxDEJ KgswSlvEppN9qeWxbsQ1pDUCguJa9Pk5Rw== X-Google-Smtp-Source: ABdhPJyW7GagFRt2Sm7iNbU/IhIEPapew83E2GC3abrXggeOVVmpXAyk4ISaoNPPyifSi15cVe/YiWdktNRqCw== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:902:bd44:b0:14a:2c66:a06c with SMTP id b4-20020a170902bd4400b0014a2c66a06cmr35829040plx.152.1642633682725; Wed, 19 Jan 2022 15:08:02 -0800 (PST) Date: Wed, 19 Jan 2022 23:07:30 +0000 In-Reply-To: <20220119230739.2234394-1-dmatlack@google.com> Message-Id: <20220119230739.2234394-10-dmatlack@google.com> Mime-Version: 1.0 References: <20220119230739.2234394-1-dmatlack@google.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog Subject: [PATCH v2 09/18] KVM: x86/mmu: Drop new_spte local variable from restore_acc_track_spte() From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The new_spte local variable is unnecessary. Deleting it can save a line of code and simplify the remaining lines a bit. No functional change intended. Suggested-by: Sean Christopherson Signed-off-by: David Matlack --- arch/x86/kvm/mmu/mmu.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 4485554336d7..7a70c238cd26 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -649,16 +649,15 @@ static u64 mmu_spte_get_lockless(u64 *sptep) /* Restore an acc-track PTE back to a regular PTE */ static u64 restore_acc_track_spte(u64 spte) { - u64 new_spte = spte; u64 saved_bits = (spte >> SHADOW_ACC_TRACK_SAVED_BITS_SHIFT) & SHADOW_ACC_TRACK_SAVED_BITS_MASK; - new_spte &= ~shadow_acc_track_mask; - new_spte &= ~(SHADOW_ACC_TRACK_SAVED_BITS_MASK << - SHADOW_ACC_TRACK_SAVED_BITS_SHIFT); - new_spte |= saved_bits; + spte &= ~shadow_acc_track_mask; + spte &= ~(SHADOW_ACC_TRACK_SAVED_BITS_MASK << + SHADOW_ACC_TRACK_SAVED_BITS_SHIFT); + spte |= saved_bits; - return new_spte; + return spte; } /* Returns the Accessed status of the PTE and resets it at the same time. */ From patchwork Wed Jan 19 23:07:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12717878 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCB58C4332F for ; Wed, 19 Jan 2022 23:09:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1357926AbiASXJc (ORCPT ); Wed, 19 Jan 2022 18:09:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50546 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344315AbiASXIF (ORCPT ); Wed, 19 Jan 2022 18:08:05 -0500 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DEBC6C061760 for ; Wed, 19 Jan 2022 15:08:04 -0800 (PST) Received: by mail-pg1-x549.google.com with SMTP id j28-20020a637a5c000000b00344d66c3c56so2481778pgn.21 for ; Wed, 19 Jan 2022 15:08:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=8Pw7Aqpn0cUEA7Y4SFxOipMTDgK0HIMSO7Lb2esjwuk=; b=TcKM1h7/P2uugJkhId+Y2PU0zOqHJC9W+0K0iH2ie0re2dBJo3qYK0rrucaDLKpd3p y4aANyWppWLyHV41hySBCt7UHccL0fMJW4Lwm9YgmbSOXSIeDwMVAi4+nUhwnsdITkg5 /qtDowYosYPT5iaxzN4BUAu6Ti9mpw8Qq6msKV0d1XowrYYddnXk7Pl2h/i+qEnaufcM lIgpMzLcAZLW7MBVhfHYwkdTKMeDTaCDSpcnd51bX4GqToYCj4QdSfJDP3bsRHMW0Jcd VlJ7bktFQDut3d7BmpSzbSKfRwMs0sWt/rkF0mRpXsWIaFpcJbpHQxmthe4OOwB7AJgm /38A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=8Pw7Aqpn0cUEA7Y4SFxOipMTDgK0HIMSO7Lb2esjwuk=; b=ay2MN3Q/GkwaOvMUftpZ+WZryEVfDVShVZNAuCuLM1K1/mhZr/QCxZdekfqYZHEFio ceQh8vHKBO4nmIWJ7orWb+qis877eCGNL7qRfOaIp4qFWzXBpNl/qlZdPZS8uq1WPdce u+BynAEvfKjh9dqU+ToTLGatildQIALTfEVkUTmqTrDNz8Uc7ZMulqcSsBelOdfXBMfV 8q/zki7OWQ/y1kln71c8OLahN7ivjMpjK0Onk2zvXAFU7r1SzSnZCV+ZepNcd/ymTMh+ q0ZzsEHaOu5Q9Ds3cekbKih6DQb5lOvMG9y8JvLnTecxGZ/D6Ij4JDi0mH0qRmkFTJUZ AlnA== X-Gm-Message-State: AOAM533wj5Qe2b5WFolVHb7a5pju1w2Ubfk1b9UB1J6mr9EEvTPCpmlA g+wR5uD1zofzp9EqsBG+BWyVe/+HOzFuwg== X-Google-Smtp-Source: ABdhPJykd/VixXLJY76BpFN6uDLRps03jE3W6C2TpxeniSVoo1qcdSppQvoa8vCmRqS6SOJ0aVhXFk3sz/a+uA== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a65:464e:: with SMTP id k14mr17418292pgr.225.1642633684427; Wed, 19 Jan 2022 15:08:04 -0800 (PST) Date: Wed, 19 Jan 2022 23:07:31 +0000 In-Reply-To: <20220119230739.2234394-1-dmatlack@google.com> Message-Id: <20220119230739.2234394-11-dmatlack@google.com> Mime-Version: 1.0 References: <20220119230739.2234394-1-dmatlack@google.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog Subject: [PATCH v2 10/18] KVM: x86/mmu: Move restore_acc_track_spte() to spte.h From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org restore_acc_track_spte() is pure SPTE bit manipulation, making it a good fit for spte.h. And now that the WARN_ON_ONCE() calls have been removed, there isn't any good reason to not inline it. This move also prepares for a follow-up commit that will need to call restore_acc_track_spte() from spte.c No functional change intended. Reviewed-by: Ben Gardon Reviewed-by: Peter Xu Signed-off-by: David Matlack --- arch/x86/kvm/mmu/mmu.c | 14 -------------- arch/x86/kvm/mmu/spte.h | 15 +++++++++++++++ 2 files changed, 15 insertions(+), 14 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 7a70c238cd26..51aa38bdb858 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -646,20 +646,6 @@ static u64 mmu_spte_get_lockless(u64 *sptep) return __get_spte_lockless(sptep); } -/* Restore an acc-track PTE back to a regular PTE */ -static u64 restore_acc_track_spte(u64 spte) -{ - u64 saved_bits = (spte >> SHADOW_ACC_TRACK_SAVED_BITS_SHIFT) - & SHADOW_ACC_TRACK_SAVED_BITS_MASK; - - spte &= ~shadow_acc_track_mask; - spte &= ~(SHADOW_ACC_TRACK_SAVED_BITS_MASK << - SHADOW_ACC_TRACK_SAVED_BITS_SHIFT); - spte |= saved_bits; - - return spte; -} - /* Returns the Accessed status of the PTE and resets it at the same time. */ static bool mmu_spte_age(u64 *sptep) { diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index be6a007a4af3..e8ac1fab3185 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -367,6 +367,21 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, u64 make_nonleaf_spte(u64 *child_pt, bool ad_disabled); u64 make_mmio_spte(struct kvm_vcpu *vcpu, u64 gfn, unsigned int access); u64 mark_spte_for_access_track(u64 spte); + +/* Restore an acc-track PTE back to a regular PTE */ +static inline u64 restore_acc_track_spte(u64 spte) +{ + u64 saved_bits = (spte >> SHADOW_ACC_TRACK_SAVED_BITS_SHIFT) + & SHADOW_ACC_TRACK_SAVED_BITS_MASK; + + spte &= ~shadow_acc_track_mask; + spte &= ~(SHADOW_ACC_TRACK_SAVED_BITS_MASK << + SHADOW_ACC_TRACK_SAVED_BITS_SHIFT); + spte |= saved_bits; + + return spte; +} + u64 kvm_mmu_changed_pte_notifier_make_spte(u64 old_spte, kvm_pfn_t new_pfn); void kvm_mmu_reset_all_pte_masks(void); From patchwork Wed Jan 19 23:07:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12717877 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A700FC433EF for ; Wed, 19 Jan 2022 23:09:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1357857AbiASXJZ (ORCPT ); Wed, 19 Jan 2022 18:09:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50238 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344356AbiASXIH (ORCPT ); Wed, 19 Jan 2022 18:08:07 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 845ECC061574 for ; Wed, 19 Jan 2022 15:08:06 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id z3-20020a17090a468300b001b4df1f5a6eso2736445pjf.6 for ; Wed, 19 Jan 2022 15:08:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=d5uxOLMY02+aBC8yKnGiHIkhgr9+Yp4wHSTMVG68uAQ=; b=r6VnYXb5zYYD5SKWbRpgrdI/VAFBVT6tDjUhrOHxsaR3wr5zFjAOucfXkGipalUJ2D DHzoCEdEIGKHBSWvCDjTfJFIq70aDLsCMhewINvEOFXs1uswTV0ZOYXHMU82brG/GAQT UfZziZDQoR9f6m2GPdD+frHDpfjczANCAk4fgbRwaWDT5u6dGa6cKvnQ6MRK6qDQhxnH +/5jT9t7mtxR9Ww++OU+kTkCM4pKsiPD3oVcvBx0QAVwkcbHkuRBXJBzEGpy58jR8e/1 vJn2bdyBVUqBqaSPVc4AouDr3hkx399Venw/k+r8LfBnf+LVSqTK12zbbuRs7Awqsild 5Ihg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=d5uxOLMY02+aBC8yKnGiHIkhgr9+Yp4wHSTMVG68uAQ=; b=lo3mEDvLDyf8TwVTpeNy26i7OasGxLhXraLKobBRKalpcpDA6dkNKuJbZbD1YVsE20 l2r4tjeeq+c9/3h4/XZXE5LXETQJXUwWjERW/OjZCKz3BNiJ7MPxjnWP/sm33KXeuAII 5X/4V89FEuGGTrOomTms2xBqeIKwyH26uULFS+meVqZqQUI6SSj2sef0nDAlM6QUA/eK yGv3mnsyHEqx4IfM4j3FavEH71ZAQSxeoVEG2+PEVULUzYWFpRFSKrjfkhCNGQKm8hEk USbBoi0Z5sGJgLf3gPN2XhDUa63xK+YrHBiNmi3Y3W/ci0RFIIR3kTEfN2zgZ+u5Uoa2 xg7A== X-Gm-Message-State: AOAM5305KiHxoDJD48rQovQlzmVrTf/3uTnjpEFkGVeYU5rp51MxFSrD b0CSBMHkjqkTPXoJ3/MTd/QpzEvgLi6Ffg== X-Google-Smtp-Source: ABdhPJzlrgX4j2rKE0nqZkNwo6RAb5D+2ILaFqpxh3XJjxjjukyRPAp1vdRpOVsDGDcRqQqrXHmoZzSVkSqpoQ== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a05:6a00:10cd:b0:4bc:a950:41e2 with SMTP id d13-20020a056a0010cd00b004bca95041e2mr33400165pfu.30.1642633686015; Wed, 19 Jan 2022 15:08:06 -0800 (PST) Date: Wed, 19 Jan 2022 23:07:32 +0000 In-Reply-To: <20220119230739.2234394-1-dmatlack@google.com> Message-Id: <20220119230739.2234394-12-dmatlack@google.com> Mime-Version: 1.0 References: <20220119230739.2234394-1-dmatlack@google.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog Subject: [PATCH v2 11/18] KVM: x86/mmu: Refactor TDP MMU iterators to take kvm_mmu_page root From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Instead of passing a pointer to the root page table and the root level separately, pass in a pointer to the root kvm_mmu_page struct. This reduces the number of arguments by 1, cutting down on line lengths. No functional change intended. Reviewed-by: Ben Gardon Reviewed-by: Peter Xu Signed-off-by: David Matlack --- arch/x86/kvm/mmu/tdp_iter.c | 8 +++++--- arch/x86/kvm/mmu/tdp_iter.h | 10 +++++----- arch/x86/kvm/mmu/tdp_mmu.c | 14 +++++--------- 3 files changed, 15 insertions(+), 17 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_iter.c b/arch/x86/kvm/mmu/tdp_iter.c index caa96c270b95..be3f096db2eb 100644 --- a/arch/x86/kvm/mmu/tdp_iter.c +++ b/arch/x86/kvm/mmu/tdp_iter.c @@ -40,17 +40,19 @@ void tdp_iter_restart(struct tdp_iter *iter) * Sets a TDP iterator to walk a pre-order traversal of the paging structure * rooted at root_pt, starting with the walk to translate next_last_level_gfn. */ -void tdp_iter_start(struct tdp_iter *iter, u64 *root_pt, int root_level, +void tdp_iter_start(struct tdp_iter *iter, struct kvm_mmu_page *root, int min_level, gfn_t next_last_level_gfn) { + int root_level = root->role.level; + WARN_ON(root_level < 1); WARN_ON(root_level > PT64_ROOT_MAX_LEVEL); iter->next_last_level_gfn = next_last_level_gfn; iter->root_level = root_level; iter->min_level = min_level; - iter->pt_path[iter->root_level - 1] = (tdp_ptep_t)root_pt; - iter->as_id = kvm_mmu_page_as_id(sptep_to_sp(root_pt)); + iter->pt_path[iter->root_level - 1] = (tdp_ptep_t)root->spt; + iter->as_id = kvm_mmu_page_as_id(root); tdp_iter_restart(iter); } diff --git a/arch/x86/kvm/mmu/tdp_iter.h b/arch/x86/kvm/mmu/tdp_iter.h index e19cabbcb65c..216ebbe76ddd 100644 --- a/arch/x86/kvm/mmu/tdp_iter.h +++ b/arch/x86/kvm/mmu/tdp_iter.h @@ -57,17 +57,17 @@ struct tdp_iter { * Iterates over every SPTE mapping the GFN range [start, end) in a * preorder traversal. */ -#define for_each_tdp_pte_min_level(iter, root, root_level, min_level, start, end) \ - for (tdp_iter_start(&iter, root, root_level, min_level, start); \ +#define for_each_tdp_pte_min_level(iter, root, min_level, start, end) \ + for (tdp_iter_start(&iter, root, min_level, start); \ iter.valid && iter.gfn < end; \ tdp_iter_next(&iter)) -#define for_each_tdp_pte(iter, root, root_level, start, end) \ - for_each_tdp_pte_min_level(iter, root, root_level, PG_LEVEL_4K, start, end) +#define for_each_tdp_pte(iter, root, start, end) \ + for_each_tdp_pte_min_level(iter, root, PG_LEVEL_4K, start, end) tdp_ptep_t spte_to_child_pt(u64 pte, int level); -void tdp_iter_start(struct tdp_iter *iter, u64 *root_pt, int root_level, +void tdp_iter_start(struct tdp_iter *iter, struct kvm_mmu_page *root, int min_level, gfn_t next_last_level_gfn); void tdp_iter_next(struct tdp_iter *iter); void tdp_iter_restart(struct tdp_iter *iter); diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index f6144db48367..38ec5a61dbff 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -624,7 +624,7 @@ static inline void tdp_mmu_set_spte_no_dirty_log(struct kvm *kvm, } #define tdp_root_for_each_pte(_iter, _root, _start, _end) \ - for_each_tdp_pte(_iter, _root->spt, _root->role.level, _start, _end) + for_each_tdp_pte(_iter, _root, _start, _end) #define tdp_root_for_each_leaf_pte(_iter, _root, _start, _end) \ tdp_root_for_each_pte(_iter, _root, _start, _end) \ @@ -634,8 +634,7 @@ static inline void tdp_mmu_set_spte_no_dirty_log(struct kvm *kvm, else #define tdp_mmu_for_each_pte(_iter, _mmu, _start, _end) \ - for_each_tdp_pte(_iter, __va(_mmu->root_hpa), \ - _mmu->shadow_root_level, _start, _end) + for_each_tdp_pte(_iter, to_shadow_page(_mmu->root_hpa), _start, _end) /* * Yield if the MMU lock is contended or this thread needs to return control @@ -724,8 +723,7 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root, rcu_read_lock(); - for_each_tdp_pte_min_level(iter, root->spt, root->role.level, - min_level, start, end) { + for_each_tdp_pte_min_level(iter, root, min_level, start, end) { retry: if (can_yield && tdp_mmu_iter_cond_resched(kvm, &iter, flush, shared)) { @@ -1197,8 +1195,7 @@ static bool wrprot_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root, BUG_ON(min_level > KVM_MAX_HUGEPAGE_LEVEL); - for_each_tdp_pte_min_level(iter, root->spt, root->role.level, - min_level, start, end) { + for_each_tdp_pte_min_level(iter, root, min_level, start, end) { retry: if (tdp_mmu_iter_cond_resched(kvm, &iter, false, true)) continue; @@ -1437,8 +1434,7 @@ static bool write_protect_gfn(struct kvm *kvm, struct kvm_mmu_page *root, rcu_read_lock(); - for_each_tdp_pte_min_level(iter, root->spt, root->role.level, - min_level, gfn, gfn + 1) { + for_each_tdp_pte_min_level(iter, root, min_level, gfn, gfn + 1) { if (!is_shadow_present_pte(iter.old_spte) || !is_last_spte(iter.old_spte, iter.level)) continue; From patchwork Wed Jan 19 23:07:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12717875 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 921E8C4332F for ; Wed, 19 Jan 2022 23:09:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1357842AbiASXJY (ORCPT ); Wed, 19 Jan 2022 18:09:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50556 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344374AbiASXIJ (ORCPT ); Wed, 19 Jan 2022 18:08:09 -0500 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2EFCCC061762 for ; Wed, 19 Jan 2022 15:08:08 -0800 (PST) Received: by mail-pg1-x549.google.com with SMTP id k13-20020a65434d000000b00342d8eb46b4so2482825pgq.23 for ; Wed, 19 Jan 2022 15:08:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=tb/bgoSf75NpcQTVrVWSlXxFAGhBf9G4A7XAijOZNQY=; b=qpsw03RSzHxVn0qOAVYN4tG0fp0hvQBdQPptT4fmaNxgonKflMBW0kTSXTntJf9jyO Gu7ZX+xZHyFtIi5d8nO/8gKC6PHXTTqIcYxTHt8GXaMcZltEkXSZHKFy++U/8TX+yVzS o0lqFfIKA6G9gAZRcfT/AkEI0uha6stVdNVj2nxLUCf0ZIfmaEW3fpGgbk/3laqVmGAq mgwUz68YhTX0q7yJzTQhAfv2bO2nDZhOQ0GG6C9X2hB3EmvHCYnHnynVrowy4bu//22/ cC7mOrN6wi98WyOL7iX1kd/hO31UczBRksExrnf9lI5HXbCYQpEfnq5rI12RqSn61loD 8C2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=tb/bgoSf75NpcQTVrVWSlXxFAGhBf9G4A7XAijOZNQY=; b=To6G9/EoauR9bIAdquOLN1IYGdBts6+rt8KMAhry2lbCNMEiiDPcwWB9/YCv0Gl1UP Wtrro5QjTOHQbyYei4onQ0xWdv4ioPft02kWFLVSbFLgiyAwiVWNKoDjIjptUiZF38zl AlKRo8w1XyR9Zwy78hm9xLViHBrZPYzipKxS4qAJaKSmjqkbygIZDxtv8JN+k2ktHnf5 /bnIYOmcQamhIhYm6lJn+o7wnZLkypTKkb9aBWW45iuhVr3Qo+X4rVYLZXuMw9WIZN45 j5pJwRlzfijgmiml8EhoGpgCwpilkhvOUCl5FoKKfO/PAvKM0V7+vgBrugwKROXz9xsr Ghwg== X-Gm-Message-State: AOAM5325Zk0vmFsrtCE7DOSiLFkWk8pRmOjtgiE2J+WblAeuJsNSVK0L +Jhe3nNSlpPRfL+Zet+VXDGbABGhuTW7vQ== X-Google-Smtp-Source: ABdhPJzUI92PR6qe/UaqXaBTRDo8ANf9xBf4SekIY3I1czcBCAFO6PmZ8/ig4oEEhlF3ozwaAfK2pkwn+Cat8A== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a62:503:0:b0:4bc:764c:5524 with SMTP id 3-20020a620503000000b004bc764c5524mr32860483pff.25.1642633687686; Wed, 19 Jan 2022 15:08:07 -0800 (PST) Date: Wed, 19 Jan 2022 23:07:33 +0000 In-Reply-To: <20220119230739.2234394-1-dmatlack@google.com> Message-Id: <20220119230739.2234394-13-dmatlack@google.com> Mime-Version: 1.0 References: <20220119230739.2234394-1-dmatlack@google.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog Subject: [PATCH v2 12/18] KVM: x86/mmu: Remove redundant role overrides for TDP MMU shadow pages From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The vCPU's mmu_role already has the correct values for direct, has_4_byte_gpte, access, and ad_disabled. Remove the code that was redundantly overwriting these fields with the same values. No functional change intended. Suggested-by: Sean Christopherson Signed-off-by: David Matlack --- arch/x86/kvm/mmu/tdp_mmu.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 38ec5a61dbff..90b6fbec83cc 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -164,10 +164,6 @@ static union kvm_mmu_page_role page_role_for_level(struct kvm_vcpu *vcpu, role = vcpu->arch.mmu->mmu_role.base; role.level = level; - role.direct = true; - role.has_4_byte_gpte = false; - role.access = ACC_ALL; - role.ad_disabled = !shadow_accessed_mask; return role; } From patchwork Wed Jan 19 23:07:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12717874 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E307C433F5 for ; Wed, 19 Jan 2022 23:09:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1357821AbiASXJW (ORCPT ); Wed, 19 Jan 2022 18:09:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50564 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344386AbiASXIK (ORCPT ); Wed, 19 Jan 2022 18:08:10 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D05FCC061763 for ; Wed, 19 Jan 2022 15:08:09 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id x15-20020a17090a46cf00b001b35ee9643fso4813434pjg.6 for ; Wed, 19 Jan 2022 15:08:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=X1xFe3lwJlWFwZpWylh7a3VDSNBVAiA26u7lyQbChJo=; b=jvbMEkwSZeTEeOuV8CErttwAOXO/5jJ+6kmeUkwv5T6uXogwK+f8N3gzw/Y39GDvvx 25xcS4aE47u831G5Wj/4As4ThZm8Bxo621Tauxy3YIb/SyDahoyiOSLuCYeoyvEi33re D4d0gk0FWofQIf3gvYpzcfKvUTzkmPpuRHkVJdRFle2JZFrA6e1ZfTYDKZ5aXk+DSkSW 9bEX11C1mEFVxhfm3b4D9LawSWjOtuzLoIdoMpkhv+5ezNnMua8zyHITVfg+Li77yPv+ oOp9YBREo31SHznl2kb9mqapq1wrMqDPX9H29+DMt1UN5uHzrIIyNNuTOL+yYSxOgwrl iLzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=X1xFe3lwJlWFwZpWylh7a3VDSNBVAiA26u7lyQbChJo=; b=Uak00pNRf7aqOMSloeo9w0VzEXFd0QHQN/qh2cwUdMVvpgBiUZ1tj4qQCmWfj2wcXg DEKZG+4I/I/qxBd+dHI6MVAdaB/xoNTM/VgaGJanzLLTmAy2is53r7C19/nEAVtngrwz aX7l0jeWSeniVvG8L00LdoFIr1kv57SDn18USM6OAji6tLlU9SUMrwK5Gf/7daohSTqz YsW9MD7zoWInBicRKAq0COFomYqEWhKRA7x2Jvzo272JJVBW4IdQZs4dbSWTEQ4py9g2 FBTyKiEsjKTH8P/1YviR+/inHjPxFfejgg6EuVWyMpZ1bbcgTr59KiDkjWZVxrsiPou0 H9yQ== X-Gm-Message-State: AOAM531vIspx33yAtd1mzMPT6b56jICc1RTKyQnSQxjQUSLAs4HEXeh6 T3GYDSD4rYJOznqiUhGjNVXBY3ddJULYrg== X-Google-Smtp-Source: ABdhPJyX6n0W7sUsc+jwt5qL6W/yPKs9MZ4mL8ia3N/Lf9fTpfrmzHuUPi404CV+PVsVThypgnx+/6fleGODbQ== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:90a:8b95:: with SMTP id z21mr7060563pjn.29.1642633689344; Wed, 19 Jan 2022 15:08:09 -0800 (PST) Date: Wed, 19 Jan 2022 23:07:34 +0000 In-Reply-To: <20220119230739.2234394-1-dmatlack@google.com> Message-Id: <20220119230739.2234394-14-dmatlack@google.com> Mime-Version: 1.0 References: <20220119230739.2234394-1-dmatlack@google.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog Subject: [PATCH v2 13/18] KVM: x86/mmu: Derive page role for TDP MMU shadow pages from parent From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Derive the page role from the parent shadow page, since the only thing that changes is the level. This is in preparation for splitting huge pages during VM-ioctls which do not have access to the vCPU MMU context. No functional change intended. Reviewed-by: Peter Xu Signed-off-by: David Matlack --- arch/x86/kvm/mmu/tdp_mmu.c | 37 +++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 90b6fbec83cc..5c1f1777e3d3 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -157,19 +157,8 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm, if (kvm_mmu_page_as_id(_root) != _as_id) { \ } else -static union kvm_mmu_page_role page_role_for_level(struct kvm_vcpu *vcpu, - int level) -{ - union kvm_mmu_page_role role; - - role = vcpu->arch.mmu->mmu_role.base; - role.level = level; - - return role; -} - static struct kvm_mmu_page *tdp_mmu_alloc_sp(struct kvm_vcpu *vcpu, gfn_t gfn, - int level) + union kvm_mmu_page_role role) { struct kvm_mmu_page *sp; @@ -177,7 +166,7 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp(struct kvm_vcpu *vcpu, gfn_t gfn, sp->spt = kvm_mmu_memory_cache_alloc(&vcpu->arch.mmu_shadow_page_cache); set_page_private(virt_to_page(sp->spt), (unsigned long)sp); - sp->role.word = page_role_for_level(vcpu, level).word; + sp->role = role; sp->gfn = gfn; sp->tdp_mmu_page = true; @@ -186,16 +175,28 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp(struct kvm_vcpu *vcpu, gfn_t gfn, return sp; } -hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu) +static struct kvm_mmu_page *tdp_mmu_alloc_child_sp(struct kvm_vcpu *vcpu, + struct tdp_iter *iter) { + struct kvm_mmu_page *parent_sp; union kvm_mmu_page_role role; + + parent_sp = sptep_to_sp(rcu_dereference(iter->sptep)); + + role = parent_sp->role; + role.level--; + + return tdp_mmu_alloc_sp(vcpu, iter->gfn, role); +} + +hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu) +{ + union kvm_mmu_page_role role = vcpu->arch.mmu->mmu_role.base; struct kvm *kvm = vcpu->kvm; struct kvm_mmu_page *root; lockdep_assert_held_write(&kvm->mmu_lock); - role = page_role_for_level(vcpu, vcpu->arch.mmu->shadow_root_level); - /* Check for an existing root before allocating a new one. */ for_each_tdp_mmu_root(kvm, root, kvm_mmu_role_as_id(role)) { if (root->role.word == role.word && @@ -203,7 +204,7 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu) goto out; } - root = tdp_mmu_alloc_sp(vcpu, 0, vcpu->arch.mmu->shadow_root_level); + root = tdp_mmu_alloc_sp(vcpu, 0, role); refcount_set(&root->tdp_mmu_root_count, 1); spin_lock(&kvm->arch.tdp_mmu_pages_lock); @@ -1021,7 +1022,7 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) if (is_removed_spte(iter.old_spte)) break; - sp = tdp_mmu_alloc_sp(vcpu, iter.gfn, iter.level - 1); + sp = tdp_mmu_alloc_child_sp(vcpu, &iter); if (tdp_mmu_link_sp_atomic(vcpu->kvm, &iter, sp, account_nx)) { tdp_mmu_free_sp(sp); break; From patchwork Wed Jan 19 23:07:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12717873 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20230C433EF for ; Wed, 19 Jan 2022 23:09:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344623AbiASXJU (ORCPT ); Wed, 19 Jan 2022 18:09:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344392AbiASXIL (ORCPT ); Wed, 19 Jan 2022 18:08:11 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3793DC061765 for ; Wed, 19 Jan 2022 15:08:11 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id x1-20020a17090ab00100b001b380b8ed35so2733332pjq.7 for ; Wed, 19 Jan 2022 15:08:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=tWmcJzsgGR4GDuJlLqa8FcjNIzENtjsOvf5WT6cMtRs=; b=M/YtWqls5toIttXKec13oXAajJC3r+/s/9kvwFpjiwejO0bejmPMSRTi8yVoO0YtQb zzB7FN0BmHTgU6A/vfuE1740vudZVhmHY2KZN6u1a4QA4lJ4AK/2I7a8YUNjiqrbigKS K5tPJ9y4jVHCCzFG0dTdtC/Q61zQXeHvOfz2ZzvqyjW8xZY+5psdUseJV0vUsT1VMvDf xURIraF3gu2lrWLpBVEt26u79YCELHDNvqeGTJ+GEEMW2QxNsgzcBEPjrnYmXjexsuKw Yy1MfiNS8aTeKHV7jdB+7iuToWil0OYgT2mUWS/ffxMFbqFutbL58NjSTgcSMsm5/wPr 0lrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=tWmcJzsgGR4GDuJlLqa8FcjNIzENtjsOvf5WT6cMtRs=; b=badso3RQ8sm3ypZDGGI/NRkLBKZNTBJlbDqLHhkHCeOsg4SeLdAxDvxkCSCq8eKb73 YVt6f2QUgDMQBL1liYvrw8QvbJPFyGFljhrjuwClgPZDh+988DbVYkxIDLD2hgY46bEa 7ceFSaBqVGAdKIfGu1/4dfln2vnQktRclG4DZKmgmePLc5xOYv42Va46xBUdxIICCxO+ /tsA9uEZAHXtYQbxeakUhk19XgvOUJMg8CUChBMg/BVgEtYFihLzYpll7BpCrWZshxBD IXKmC2jAd8NhzYsXyOuIWjP0PmJvV8e7RRE1HB8l32WQ6/dsk9G2GPglj/XvX+qX2r29 +O0w== X-Gm-Message-State: AOAM531q+ZYqS9ozi7MNvmmcKhhoGzvbvsWidfanpCHqEHPHi2++pg+Y sEf1Ansa3Kqq2g+XwkfoP/+dVc+YYsHQsg== X-Google-Smtp-Source: ABdhPJwuryg9HLFTmojVb0lCgI5CQkHshRZipEPIKE0750i/cOILuTWKCWQd5aAt4KHZg67Aoxc2DEQ8ebinHQ== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:90a:df15:: with SMTP id gp21mr7048329pjb.108.1642633690701; Wed, 19 Jan 2022 15:08:10 -0800 (PST) Date: Wed, 19 Jan 2022 23:07:35 +0000 In-Reply-To: <20220119230739.2234394-1-dmatlack@google.com> Message-Id: <20220119230739.2234394-15-dmatlack@google.com> Mime-Version: 1.0 References: <20220119230739.2234394-1-dmatlack@google.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog Subject: [PATCH v2 14/18] KVM: x86/mmu: Separate TDP MMU shadow page allocation and initialization From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Separate the allocation of shadow pages from their initialization. This is in preparation for splitting huge pages outside of the vCPU fault context, which requires a different allocation mechanism. No functional changed intended. Reviewed-by: Peter Xu Signed-off-by: David Matlack --- arch/x86/kvm/mmu/tdp_mmu.c | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 5c1f1777e3d3..b526a1873f30 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -157,13 +157,19 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm, if (kvm_mmu_page_as_id(_root) != _as_id) { \ } else -static struct kvm_mmu_page *tdp_mmu_alloc_sp(struct kvm_vcpu *vcpu, gfn_t gfn, - union kvm_mmu_page_role role) +static struct kvm_mmu_page *tdp_mmu_alloc_sp(struct kvm_vcpu *vcpu) { struct kvm_mmu_page *sp; sp = kvm_mmu_memory_cache_alloc(&vcpu->arch.mmu_page_header_cache); sp->spt = kvm_mmu_memory_cache_alloc(&vcpu->arch.mmu_shadow_page_cache); + + return sp; +} + +static void tdp_mmu_init_sp(struct kvm_mmu_page *sp, gfn_t gfn, + union kvm_mmu_page_role role) +{ set_page_private(virt_to_page(sp->spt), (unsigned long)sp); sp->role = role; @@ -171,12 +177,10 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp(struct kvm_vcpu *vcpu, gfn_t gfn, sp->tdp_mmu_page = true; trace_kvm_mmu_get_page(sp, true); - - return sp; } -static struct kvm_mmu_page *tdp_mmu_alloc_child_sp(struct kvm_vcpu *vcpu, - struct tdp_iter *iter) +static void tdp_mmu_init_child_sp(struct kvm_mmu_page *child_sp, + struct tdp_iter *iter) { struct kvm_mmu_page *parent_sp; union kvm_mmu_page_role role; @@ -186,7 +190,7 @@ static struct kvm_mmu_page *tdp_mmu_alloc_child_sp(struct kvm_vcpu *vcpu, role = parent_sp->role; role.level--; - return tdp_mmu_alloc_sp(vcpu, iter->gfn, role); + tdp_mmu_init_sp(child_sp, iter->gfn, role); } hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu) @@ -204,7 +208,9 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu) goto out; } - root = tdp_mmu_alloc_sp(vcpu, 0, role); + root = tdp_mmu_alloc_sp(vcpu); + tdp_mmu_init_sp(root, 0, role); + refcount_set(&root->tdp_mmu_root_count, 1); spin_lock(&kvm->arch.tdp_mmu_pages_lock); @@ -1022,7 +1028,9 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) if (is_removed_spte(iter.old_spte)) break; - sp = tdp_mmu_alloc_child_sp(vcpu, &iter); + sp = tdp_mmu_alloc_sp(vcpu); + tdp_mmu_init_child_sp(sp, &iter); + if (tdp_mmu_link_sp_atomic(vcpu->kvm, &iter, sp, account_nx)) { tdp_mmu_free_sp(sp); break; From patchwork Wed Jan 19 23:07:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12717871 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1672C433FE for ; Wed, 19 Jan 2022 23:09:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234805AbiASXJQ (ORCPT ); Wed, 19 Jan 2022 18:09:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50328 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245397AbiASXIM (ORCPT ); Wed, 19 Jan 2022 18:08:12 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8A469C061768 for ; Wed, 19 Jan 2022 15:08:12 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id o72-20020a17090a0a4e00b001b4e5b5b6c0so2740764pjo.5 for ; Wed, 19 Jan 2022 15:08:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=XjPgovmtjyh6QrYmFw2GcuVMQcGqsRFhbseVOX6BIwI=; b=AKaHiirRkH09K5F7EIdIsbslnrIA8o5U8ZbhSA6jlk63LiZ0JoytCrTtPHG1RH5RKB BLSUOYhQEQEMkDVroVggSbuG9iICzP5+UGG0T1d766DzYkZ/ZMGscElWpU8bi46MSvkD Ync8/P866UuTWT/AsnLBS7mPYx5J/tBUZI7O0co3ZJHXSWZwpD7wehJ5OCcrAozq0x88 rXjeRl7gyzNRbg2RnYEiPFZ7u5NJ4vZ+sJowfk2+8fyZk3dLv4KKnnp3j4jJ3VRlo6oF wyxH7tCzUM2yHF4AqV0PwVsdwVO3KW3n41y8OGkULmmXAg+dF7qWSboeztHb6vBJlE6g Hc9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=XjPgovmtjyh6QrYmFw2GcuVMQcGqsRFhbseVOX6BIwI=; b=RvCSpWMDNF/G6oB0tH877Ikl3j4oE7r5sjRxKn73N7nsqqFh4ECukgO29nXUEs1zae MmbBpc1TwwWRXLdL1NCUpN3FucuFTqNqt/P4L4/9QB+ia39R+xgP4xXZAkbibFNtRpvF 86kCCK6AXybSPsek/U1i7fndtghOPi/Y1PDp8rgOIgqCnpLtYKcoPDeUzgklTjiAjvTe 158yMZhQxg+RhhybGF56Cozw74WZGaaRADv0TQbSgizgvPCXzmd+m8dxfho5OaBfvJPK yegWVuaMt+cJGk/TR45mAzLpknO84MdR9KfYNbmhWsIaIuxg/kFEniY5nScy+MhwyFWm Q9GQ== X-Gm-Message-State: AOAM530+sWTdKFqTjl5l5HogmYyDfTIyNQexSMxlC7zvVun6uzjbntiD PZTfQKLolqtdnPyNLG3AuLSFCO9KKG+ELg== X-Google-Smtp-Source: ABdhPJy/od0aqU+gfzAdgerrwmYIy+hOwE97AxqI/+NBlY1xsdcuGIt62nygObXISY+W04IX3e8Koruqh/6Aaw== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a05:6a00:8cb:b0:4bc:3def:b616 with SMTP id s11-20020a056a0008cb00b004bc3defb616mr33093699pfu.18.1642633692082; Wed, 19 Jan 2022 15:08:12 -0800 (PST) Date: Wed, 19 Jan 2022 23:07:36 +0000 In-Reply-To: <20220119230739.2234394-1-dmatlack@google.com> Message-Id: <20220119230739.2234394-16-dmatlack@google.com> Mime-Version: 1.0 References: <20220119230739.2234394-1-dmatlack@google.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog Subject: [PATCH v2 15/18] KVM: x86/mmu: Split huge pages mapped by the TDP MMU when dirty logging is enabled From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org When dirty logging is enabled without initially-all-set, try to split all huge pages in the memslot down to 4KB pages so that vCPUs do not have to take expensive write-protection faults to split huge pages. Eager page splitting is best-effort only. This commit only adds the support for the TDP MMU, and even there splitting may fail due to out of memory conditions. Failures to split a huge page is fine from a correctness standpoint because KVM will always follow up splitting by write-protecting any remaining huge pages. Eager page splitting moves the cost of splitting huge pages off of the vCPU threads and onto the thread enabling dirty logging on the memslot. This is useful because: 1. Splitting on the vCPU thread interrupts vCPUs execution and is disruptive to customers whereas splitting on VM ioctl threads can run in parallel with vCPU execution. 2. Splitting all huge pages at once is more efficient because it does not require performing VM-exit handling or walking the page table for every 4KiB page in the memslot, and greatly reduces the amount of contention on the mmu_lock. For example, when running dirty_log_perf_test with 96 virtual CPUs, 1GiB per vCPU, and 1GiB HugeTLB memory, the time it takes vCPUs to write to all of their memory after dirty logging is enabled decreased by 95% from 2.94s to 0.14s. Eager Page Splitting is over 100x more efficient than the current implementation of splitting on fault under the read lock. For example, taking the same workload as above, Eager Page Splitting reduced the CPU required to split all huge pages from ~270 CPU-seconds ((2.94s - 0.14s) * 96 vCPU threads) to only 1.55 CPU-seconds. Eager page splitting does increase the amount of time it takes to enable dirty logging since it has split all huge pages. For example, the time it took to enable dirty logging in the 96GiB region of the aforementioned test increased from 0.001s to 1.55s. Reviewed-by: Peter Xu Signed-off-by: David Matlack --- .../admin-guide/kernel-parameters.txt | 24 +++ arch/x86/include/asm/kvm_host.h | 3 + arch/x86/kvm/mmu/mmu.c | 24 +++ arch/x86/kvm/mmu/spte.c | 59 ++++++ arch/x86/kvm/mmu/spte.h | 1 + arch/x86/kvm/mmu/tdp_mmu.c | 173 ++++++++++++++++++ arch/x86/kvm/mmu/tdp_mmu.h | 5 + arch/x86/kvm/x86.c | 6 + 8 files changed, 295 insertions(+) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 86a2456d94ba..f5e9c4a45aef 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2330,6 +2330,30 @@ kvm.ignore_msrs=[KVM] Ignore guest accesses to unhandled MSRs. Default is 0 (don't ignore, but inject #GP) + kvm.eager_page_split= + [KVM,X86] Controls whether or not KVM will try to + proactively split all huge pages during dirty logging. + Eager page splitting reduces interruptions to vCPU + execution by eliminating the write-protection faults + and MMU lock contention that would otherwise be + required to split huge pages lazily. + + VM workloads that rarely perform writes or that write + only to a small region of VM memory may benefit from + disabling eager page splitting to allow huge pages to + still be used for reads. + + The behavior of eager page splitting depends on whether + KVM_DIRTY_LOG_INITIALLY_SET is enabled or disabled. If + disabled, all huge pages in a memslot will be eagerly + split when dirty logging is enabled on that memslot. If + enabled, huge pages will not be eagerly split. + + Eager page splitting currently only supports splitting + huge pages mapped by the TDP MMU. + + Default is Y (on). + kvm.enable_vmware_backdoor=[KVM] Support VMware backdoor PV interface. Default is false (don't support). diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 682ad02a4e58..97560980456d 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1579,6 +1579,9 @@ void kvm_mmu_reset_context(struct kvm_vcpu *vcpu); void kvm_mmu_slot_remove_write_access(struct kvm *kvm, const struct kvm_memory_slot *memslot, int start_level); +void kvm_mmu_slot_try_split_huge_pages(struct kvm *kvm, + const struct kvm_memory_slot *memslot, + int target_level); void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm, const struct kvm_memory_slot *memslot); void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm, diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 51aa38bdb858..a273536e8b25 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5834,6 +5834,30 @@ void kvm_mmu_slot_remove_write_access(struct kvm *kvm, kvm_arch_flush_remote_tlbs_memslot(kvm, memslot); } +void kvm_mmu_slot_try_split_huge_pages(struct kvm *kvm, + const struct kvm_memory_slot *memslot, + int target_level) +{ + u64 start = memslot->base_gfn; + u64 end = start + memslot->npages; + + if (is_tdp_mmu_enabled(kvm)) { + read_lock(&kvm->mmu_lock); + kvm_tdp_mmu_try_split_huge_pages(kvm, memslot, start, end, target_level); + read_unlock(&kvm->mmu_lock); + } + + /* + * No TLB flush is necessary here. KVM will flush TLBs after + * write-protecting and/or clearing dirty on the newly split SPTEs to + * ensure that guest writes are reflected in the dirty log before the + * ioctl to enable dirty logging on this memslot completes. Since the + * split SPTEs retain the write and dirty bits of the huge SPTE, it is + * safe for KVM to decide if a TLB flush is necessary based on the split + * SPTEs. + */ +} + static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm, struct kvm_rmap_head *rmap_head, const struct kvm_memory_slot *slot) diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index f8677404c93c..17f226bb8ec2 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -191,6 +191,65 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, return wrprot; } +static u64 make_spte_executable(u64 spte) +{ + bool is_access_track = is_access_track_spte(spte); + + if (is_access_track) + spte = restore_acc_track_spte(spte); + + spte &= ~shadow_nx_mask; + spte |= shadow_x_mask; + + if (is_access_track) + spte = mark_spte_for_access_track(spte); + + return spte; +} + +/* + * Construct an SPTE that maps a sub-page of the given huge page SPTE where + * `index` identifies which sub-page. + * + * This is used during huge page splitting to build the SPTEs that make up the + * new page table. + */ +u64 make_huge_page_split_spte(u64 huge_spte, int huge_level, int index) +{ + u64 child_spte; + int child_level; + + if (WARN_ON_ONCE(!is_shadow_present_pte(huge_spte))) + return 0; + + if (WARN_ON_ONCE(!is_large_pte(huge_spte))) + return 0; + + child_spte = huge_spte; + child_level = huge_level - 1; + + /* + * The child_spte already has the base address of the huge page being + * split. So we just have to OR in the offset to the page at the next + * lower level for the given index. + */ + child_spte |= (index * KVM_PAGES_PER_HPAGE(child_level)) << PAGE_SHIFT; + + if (child_level == PG_LEVEL_4K) { + child_spte &= ~PT_PAGE_SIZE_MASK; + + /* + * When splitting to a 4K page, mark the page executable as the + * NX hugepage mitigation no longer applies. + */ + if (is_nx_huge_page_enabled()) + child_spte = make_spte_executable(child_spte); + } + + return child_spte; +} + + u64 make_nonleaf_spte(u64 *child_pt, bool ad_disabled) { u64 spte = SPTE_MMU_PRESENT_MASK; diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index e8ac1fab3185..9d6bb74bbb54 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -364,6 +364,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, unsigned int pte_access, gfn_t gfn, kvm_pfn_t pfn, u64 old_spte, bool prefetch, bool can_unsync, bool host_writable, u64 *new_spte); +u64 make_huge_page_split_spte(u64 huge_spte, int huge_level, int index); u64 make_nonleaf_spte(u64 *child_pt, bool ad_disabled); u64 make_mmio_spte(struct kvm_vcpu *vcpu, u64 gfn, unsigned int access); u64 mark_spte_for_access_track(u64 spte); diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index b526a1873f30..88f723fc0d1f 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1242,6 +1242,179 @@ bool kvm_tdp_mmu_wrprot_slot(struct kvm *kvm, return spte_set; } +static struct kvm_mmu_page *__tdp_mmu_alloc_sp_for_split(gfp_t gfp) +{ + struct kvm_mmu_page *sp; + + gfp |= __GFP_ZERO; + + sp = kmem_cache_alloc(mmu_page_header_cache, gfp); + if (!sp) + return NULL; + + sp->spt = (void *)__get_free_page(gfp); + if (!sp->spt) { + kmem_cache_free(mmu_page_header_cache, sp); + return NULL; + } + + return sp; +} + +static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(struct kvm *kvm, + struct tdp_iter *iter) +{ + struct kvm_mmu_page *sp; + + lockdep_assert_held_read(&kvm->mmu_lock); + + /* + * Since we are allocating while under the MMU lock we have to be + * careful about GFP flags. Use GFP_NOWAIT to avoid blocking on direct + * reclaim and to avoid making any filesystem callbacks (which can end + * up invoking KVM MMU notifiers, resulting in a deadlock). + * + * If this allocation fails we drop the lock and retry with reclaim + * allowed. + */ + sp = __tdp_mmu_alloc_sp_for_split(GFP_NOWAIT | __GFP_ACCOUNT); + if (sp) + return sp; + + rcu_read_unlock(); + read_unlock(&kvm->mmu_lock); + + iter->yielded = true; + sp = __tdp_mmu_alloc_sp_for_split(GFP_KERNEL_ACCOUNT); + + read_lock(&kvm->mmu_lock); + rcu_read_lock(); + + return sp; +} + +static int tdp_mmu_split_huge_page_atomic(struct kvm *kvm, + struct tdp_iter *iter, + struct kvm_mmu_page *sp) +{ + const u64 huge_spte = iter->old_spte; + const int level = iter->level; + int ret, i; + + tdp_mmu_init_child_sp(sp, iter); + + /* + * No need for atomics when writing to sp->spt since the page table has + * not been linked in yet and thus is not reachable from any other CPU. + */ + for (i = 0; i < PT64_ENT_PER_PAGE; i++) + sp->spt[i] = make_huge_page_split_spte(huge_spte, level, i); + + /* + * Replace the huge spte with a pointer to the populated lower level + * page table. Since we are making this change without a TLB flush vCPUs + * will see a mix of the split mappings and the original huge mapping, + * depending on what's currently in their TLB. This is fine from a + * correctness standpoint since the translation will be the same either + * way. + */ + ret = tdp_mmu_link_sp_atomic(kvm, iter, sp, false); + if (ret) + return ret; + + /* + * tdp_mmu_link_sp_atomic() will handle subtracting the huge page we + * are overwriting from the page stats. But we have to manually update + * the page stats with the new present child pages. + */ + kvm_update_page_stats(kvm, level - 1, PT64_ENT_PER_PAGE); + + return 0; +} + +static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, + struct kvm_mmu_page *root, + gfn_t start, gfn_t end, + int target_level) +{ + struct kvm_mmu_page *sp = NULL; + struct tdp_iter iter; + int ret = 0; + + rcu_read_lock(); + + /* + * Traverse the page table splitting all huge pages above the target + * level into one lower level. For example, if we encounter a 1GB page + * we split it into 512 2MB pages. + * + * Since the TDP iterator uses a pre-order traversal, we are guaranteed + * to visit an SPTE before ever visiting its children, which means we + * will correctly recursively split huge pages that are more than one + * level above the target level (e.g. splitting a 1GB to 512 2MB pages, + * and then splitting each of those to 512 4KB pages). + */ + for_each_tdp_pte_min_level(iter, root, target_level + 1, start, end) { +retry: + if (tdp_mmu_iter_cond_resched(kvm, &iter, false, true)) + continue; + + if (!is_shadow_present_pte(iter.old_spte) || !is_large_pte(iter.old_spte)) + continue; + + if (!sp) { + sp = tdp_mmu_alloc_sp_for_split(kvm, &iter); + if (!sp) { + ret = -ENOMEM; + break; + } + + if (iter.yielded) + continue; + } + + if (tdp_mmu_split_huge_page_atomic(kvm, &iter, sp)) + goto retry; + + sp = NULL; + } + + rcu_read_unlock(); + + /* + * It's possible to exit the loop having never used the last sp if, for + * example, a vCPU doing HugePage NX splitting wins the race and + * installs its own sp in place of the last sp we tried to split. + */ + if (sp) + tdp_mmu_free_sp(sp); + + + return ret; +} + +/* + * Try to split all huge pages mapped by the TDP MMU down to the target level. + */ +void kvm_tdp_mmu_try_split_huge_pages(struct kvm *kvm, + const struct kvm_memory_slot *slot, + gfn_t start, gfn_t end, + int target_level) +{ + struct kvm_mmu_page *root; + int r = 0; + + lockdep_assert_held_read(&kvm->mmu_lock); + + for_each_tdp_mmu_root_yield_safe(kvm, root, slot->as_id, true) { + r = tdp_mmu_split_huge_pages_root(kvm, root, start, end, target_level); + if (r) { + kvm_tdp_mmu_put_root(kvm, root, true); + break; + } + } +} + /* * Clear the dirty status of all the SPTEs mapping GFNs in the memslot. If * AD bits are enabled, this will involve clearing the dirty bit on each SPTE. diff --git a/arch/x86/kvm/mmu/tdp_mmu.h b/arch/x86/kvm/mmu/tdp_mmu.h index 3899004a5d91..4a8756507829 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.h +++ b/arch/x86/kvm/mmu/tdp_mmu.h @@ -71,6 +71,11 @@ bool kvm_tdp_mmu_write_protect_gfn(struct kvm *kvm, struct kvm_memory_slot *slot, gfn_t gfn, int min_level); +void kvm_tdp_mmu_try_split_huge_pages(struct kvm *kvm, + const struct kvm_memory_slot *slot, + gfn_t start, gfn_t end, + int target_level); + static inline void kvm_tdp_mmu_walk_lockless_begin(void) { rcu_read_lock(); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 55518b7d3b96..f5aad3e8e0a0 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -192,6 +192,9 @@ bool __read_mostly enable_pmu = true; EXPORT_SYMBOL_GPL(enable_pmu); module_param(enable_pmu, bool, 0444); +static bool __read_mostly eager_page_split = true; +module_param(eager_page_split, bool, 0644); + /* * Restoring the host value for MSRs that are only consumed when running in * usermode, e.g. SYSCALL MSRs and TSC_AUX, can be deferred until the CPU @@ -11948,6 +11951,9 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm, if (kvm_dirty_log_manual_protect_and_init_set(kvm)) return; + if (READ_ONCE(eager_page_split)) + kvm_mmu_slot_try_split_huge_pages(kvm, new, PG_LEVEL_4K); + if (kvm_x86_ops.cpu_dirty_log_size) { kvm_mmu_slot_leaf_clear_dirty(kvm, new); kvm_mmu_slot_remove_write_access(kvm, new, PG_LEVEL_2M); From patchwork Wed Jan 19 23:07:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12717872 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64537C433EF for ; Wed, 19 Jan 2022 23:09:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1357785AbiASXJS (ORCPT ); Wed, 19 Jan 2022 18:09:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50582 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344247AbiASXIO (ORCPT ); Wed, 19 Jan 2022 18:08:14 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 321D2C06161C for ; Wed, 19 Jan 2022 15:08:14 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id e16-20020a17090a119000b001b28f7b2a3bso2661002pja.8 for ; Wed, 19 Jan 2022 15:08:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=NxdE4UkoNDKVd1y+9gUH/nA5kkYNo8vtuqAG3qr2pl4=; b=MQd9qwzBIUoVvDmovqiIgJ+aMFXiW3irmm/vgYXtIOZmwXMsOjEuQKFf1et1yAHOh7 9uWYslps0Xkar0SqqF85i5EmY4iZ7tDqwz6DJwhSSdhfTv9g0Muk+k/lzWJszpGAD2GZ SYw39RxQd+NSve3/phujAWQ6jfHMJiYxWhy6pwDze8ctruSSnYF8GR8ew9/fNngK5lZK 2zZx1FGvF8HmDy/xcNRgTEmO/UXbzlZftKhfLl5wIFomT+Kkou37+6EWYvUA0NozmcTJ hsFlC/F0I+jfs/MNv+3LNwwIThVT8g20CIHos2bPJmkJ9uhORilEz2z0r7CIER89kNnT XOJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=NxdE4UkoNDKVd1y+9gUH/nA5kkYNo8vtuqAG3qr2pl4=; b=pSseIfKkop0pJMKUbfYMWfx/qiBBByCpwxPJnuWbjGA1tZ1lUZYUXoFcfyotiDAMot rP08rkR3OgNvjOO8zdpWdHfjbUegbezBUB4bhfiyNmir2Q4wQGJGA0ZJM2G10/nYhRnf JF5F/6/KuykARkvsd3u6wYD7xbWDoLqJSgQtJr9ge6DlaGkoRPpNUuMDN79hOgp8f5er iDgekbbGaw1wQaZoHQCQ5cjB/PHaKGpme00VtlO/+g4w2hplVG472Iph8AZF0TVnSWU2 O0h/QKu3v9KHOibqkr9zPu8hh5nfBitPil/MUUi4LoAR7liN4MWbXHyH4sn9qIpz3UQ7 nbnw== X-Gm-Message-State: AOAM532dzBHJDhVoahVFXNyb467gn+CDm+Ge9F0ZNwmIIBN1WmQMcMis Vu/fy+NIZ5czArRuRVGrWIHRQciYZ/6TjA== X-Google-Smtp-Source: ABdhPJy6umejyHZdX24IGaSqfRX226Z6tRO+lD1J37Irg+7ZPkHKTzweeLF6Su+48Yk0mvVKFvhCcrOfrkH8fg== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:aa7:88cb:0:b0:4c4:452:2e96 with SMTP id k11-20020aa788cb000000b004c404522e96mr17751627pff.38.1642633693674; Wed, 19 Jan 2022 15:08:13 -0800 (PST) Date: Wed, 19 Jan 2022 23:07:37 +0000 In-Reply-To: <20220119230739.2234394-1-dmatlack@google.com> Message-Id: <20220119230739.2234394-17-dmatlack@google.com> Mime-Version: 1.0 References: <20220119230739.2234394-1-dmatlack@google.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog Subject: [PATCH v2 16/18] KVM: x86/mmu: Split huge pages mapped by the TDP MMU during KVM_CLEAR_DIRTY_LOG From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org When using KVM_DIRTY_LOG_INITIALLY_SET, huge pages are not write-protected when dirty logging is enabled on the memslot. Instead they are write-protected once userspace invokes KVM_CLEAR_DIRTY_LOG for the first time and only for the specific sub-region being cleared. Enhance KVM_CLEAR_DIRTY_LOG to also try to split huge pages prior to write-protecting to avoid causing write-protection faults on vCPU threads. This also allows userspace to smear the cost of huge page splitting across multiple ioctls rather than splitting the entire memslot when not using initially-all-set. Signed-off-by: David Matlack --- .../admin-guide/kernel-parameters.txt | 4 +- arch/x86/include/asm/kvm_host.h | 4 ++ arch/x86/kvm/mmu/mmu.c | 25 ++++++- arch/x86/kvm/mmu/tdp_mmu.c | 67 +++++++++++-------- arch/x86/kvm/mmu/tdp_mmu.h | 2 +- arch/x86/kvm/x86.c | 2 +- arch/x86/kvm/x86.h | 2 + 7 files changed, 73 insertions(+), 33 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index f5e9c4a45aef..1b54e410e206 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2347,7 +2347,9 @@ KVM_DIRTY_LOG_INITIALLY_SET is enabled or disabled. If disabled, all huge pages in a memslot will be eagerly split when dirty logging is enabled on that memslot. If - enabled, huge pages will not be eagerly split. + enabled, eager page splitting will be performed during + the KVM_CLEAR_DIRTY ioctl, and only for the pages being + cleared. Eager page splitting currently only supports splitting huge pages mapped by the TDP MMU. diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 97560980456d..e089f34a66eb 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1582,6 +1582,10 @@ void kvm_mmu_slot_remove_write_access(struct kvm *kvm, void kvm_mmu_slot_try_split_huge_pages(struct kvm *kvm, const struct kvm_memory_slot *memslot, int target_level); +void kvm_mmu_try_split_huge_pages(struct kvm *kvm, + const struct kvm_memory_slot *memslot, + u64 start, u64 end, + int target_level); void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm, const struct kvm_memory_slot *memslot); void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm, diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index a273536e8b25..62caf5b6d82e 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1360,6 +1360,9 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, gfn_t start = slot->base_gfn + gfn_offset + __ffs(mask); gfn_t end = slot->base_gfn + gfn_offset + __fls(mask); + if (READ_ONCE(eager_page_split)) + kvm_mmu_try_split_huge_pages(kvm, slot, start, end, PG_LEVEL_4K); + kvm_mmu_slot_gfn_write_protect(kvm, slot, start, PG_LEVEL_2M); /* Cross two large pages? */ @@ -5834,16 +5837,32 @@ void kvm_mmu_slot_remove_write_access(struct kvm *kvm, kvm_arch_flush_remote_tlbs_memslot(kvm, memslot); } +/* Must be called with the mmu_lock held in write-mode. */ +void kvm_mmu_try_split_huge_pages(struct kvm *kvm, + const struct kvm_memory_slot *memslot, + u64 start, u64 end, + int target_level) +{ + if (is_tdp_mmu_enabled(kvm)) + kvm_tdp_mmu_try_split_huge_pages(kvm, memslot, start, end, + target_level, false); + + /* + * A TLB flush is unnecessary at this point for the same resons as in + * kvm_mmu_slot_try_split_huge_pages(). + */ +} + void kvm_mmu_slot_try_split_huge_pages(struct kvm *kvm, - const struct kvm_memory_slot *memslot, - int target_level) + const struct kvm_memory_slot *memslot, + int target_level) { u64 start = memslot->base_gfn; u64 end = start + memslot->npages; if (is_tdp_mmu_enabled(kvm)) { read_lock(&kvm->mmu_lock); - kvm_tdp_mmu_try_split_huge_pages(kvm, memslot, start, end, target_level); + kvm_tdp_mmu_try_split_huge_pages(kvm, memslot, start, end, target_level, true); read_unlock(&kvm->mmu_lock); } diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 88f723fc0d1f..d5e713b849e9 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -943,27 +943,33 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu, } /* - * tdp_mmu_link_sp_atomic - Atomically replace the given spte with an spte - * pointing to the provided page table. + * tdp_mmu_link_sp - Replace the given spte with an spte pointing to the + * provided page table. * * @kvm: kvm instance * @iter: a tdp_iter instance currently on the SPTE that should be set * @sp: The new TDP page table to install. * @account_nx: True if this page table is being installed to split a * non-executable huge page. + * @shared: This operation is running under the MMU lock in read mode. * * Returns: 0 if the new page table was installed. Non-0 if the page table * could not be installed (e.g. the atomic compare-exchange failed). */ -static int tdp_mmu_link_sp_atomic(struct kvm *kvm, struct tdp_iter *iter, - struct kvm_mmu_page *sp, bool account_nx) +static int tdp_mmu_link_sp(struct kvm *kvm, struct tdp_iter *iter, + struct kvm_mmu_page *sp, bool account_nx, + bool shared) { u64 spte = make_nonleaf_spte(sp->spt, !shadow_accessed_mask); - int ret; + int ret = 0; - ret = tdp_mmu_set_spte_atomic(kvm, iter, spte); - if (ret) - return ret; + if (shared) { + ret = tdp_mmu_set_spte_atomic(kvm, iter, spte); + if (ret) + return ret; + } else { + tdp_mmu_set_spte(kvm, iter, spte); + } spin_lock(&kvm->arch.tdp_mmu_pages_lock); list_add(&sp->link, &kvm->arch.tdp_mmu_pages); @@ -1031,7 +1037,7 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) sp = tdp_mmu_alloc_sp(vcpu); tdp_mmu_init_child_sp(sp, &iter); - if (tdp_mmu_link_sp_atomic(vcpu->kvm, &iter, sp, account_nx)) { + if (tdp_mmu_link_sp(vcpu->kvm, &iter, sp, account_nx, true)) { tdp_mmu_free_sp(sp); break; } @@ -1262,12 +1268,11 @@ static struct kvm_mmu_page *__tdp_mmu_alloc_sp_for_split(gfp_t gfp) } static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(struct kvm *kvm, - struct tdp_iter *iter) + struct tdp_iter *iter, + bool shared) { struct kvm_mmu_page *sp; - lockdep_assert_held_read(&kvm->mmu_lock); - /* * Since we are allocating while under the MMU lock we have to be * careful about GFP flags. Use GFP_NOWAIT to avoid blocking on direct @@ -1282,20 +1287,27 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(struct kvm *kvm, return sp; rcu_read_unlock(); - read_unlock(&kvm->mmu_lock); + + if (shared) + read_unlock(&kvm->mmu_lock); + else + write_unlock(&kvm->mmu_lock); iter->yielded = true; sp = __tdp_mmu_alloc_sp_for_split(GFP_KERNEL_ACCOUNT); - read_lock(&kvm->mmu_lock); + if (shared) + read_lock(&kvm->mmu_lock); + else + write_lock(&kvm->mmu_lock); + rcu_read_lock(); return sp; } -static int tdp_mmu_split_huge_page_atomic(struct kvm *kvm, - struct tdp_iter *iter, - struct kvm_mmu_page *sp) +static int tdp_mmu_split_huge_page(struct kvm *kvm, struct tdp_iter *iter, + struct kvm_mmu_page *sp, bool shared) { const u64 huge_spte = iter->old_spte; const int level = iter->level; @@ -1318,7 +1330,7 @@ static int tdp_mmu_split_huge_page_atomic(struct kvm *kvm, * correctness standpoint since the translation will be the same either * way. */ - ret = tdp_mmu_link_sp_atomic(kvm, iter, sp, false); + ret = tdp_mmu_link_sp(kvm, iter, sp, false, shared); if (ret) return ret; @@ -1335,7 +1347,7 @@ static int tdp_mmu_split_huge_page_atomic(struct kvm *kvm, static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, struct kvm_mmu_page *root, gfn_t start, gfn_t end, - int target_level) + int target_level, bool shared) { struct kvm_mmu_page *sp = NULL; struct tdp_iter iter; @@ -1356,14 +1368,14 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, */ for_each_tdp_pte_min_level(iter, root, target_level + 1, start, end) { retry: - if (tdp_mmu_iter_cond_resched(kvm, &iter, false, true)) + if (tdp_mmu_iter_cond_resched(kvm, &iter, false, shared)) continue; if (!is_shadow_present_pte(iter.old_spte) || !is_large_pte(iter.old_spte)) continue; if (!sp) { - sp = tdp_mmu_alloc_sp_for_split(kvm, &iter); + sp = tdp_mmu_alloc_sp_for_split(kvm, &iter, shared); if (!sp) { ret = -ENOMEM; break; @@ -1373,7 +1385,7 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, continue; } - if (tdp_mmu_split_huge_page_atomic(kvm, &iter, sp)) + if (tdp_mmu_split_huge_page(kvm, &iter, sp, shared)) goto retry; sp = NULL; @@ -1393,23 +1405,24 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, return ret; } + /* * Try to split all huge pages mapped by the TDP MMU down to the target level. */ void kvm_tdp_mmu_try_split_huge_pages(struct kvm *kvm, const struct kvm_memory_slot *slot, gfn_t start, gfn_t end, - int target_level) + int target_level, bool shared) { struct kvm_mmu_page *root; int r = 0; - lockdep_assert_held_read(&kvm->mmu_lock); + kvm_lockdep_assert_mmu_lock_held(kvm, shared); - for_each_tdp_mmu_root_yield_safe(kvm, root, slot->as_id, true) { - r = tdp_mmu_split_huge_pages_root(kvm, root, start, end, target_level); + for_each_tdp_mmu_root_yield_safe(kvm, root, slot->as_id, shared) { + r = tdp_mmu_split_huge_pages_root(kvm, root, start, end, target_level, shared); if (r) { - kvm_tdp_mmu_put_root(kvm, root, true); + kvm_tdp_mmu_put_root(kvm, root, shared); break; } } diff --git a/arch/x86/kvm/mmu/tdp_mmu.h b/arch/x86/kvm/mmu/tdp_mmu.h index 4a8756507829..ed9f6fbf5f25 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.h +++ b/arch/x86/kvm/mmu/tdp_mmu.h @@ -74,7 +74,7 @@ bool kvm_tdp_mmu_write_protect_gfn(struct kvm *kvm, void kvm_tdp_mmu_try_split_huge_pages(struct kvm *kvm, const struct kvm_memory_slot *slot, gfn_t start, gfn_t end, - int target_level); + int target_level, bool shared); static inline void kvm_tdp_mmu_walk_lockless_begin(void) { diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index f5aad3e8e0a0..e2ee6fc92dbc 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -192,7 +192,7 @@ bool __read_mostly enable_pmu = true; EXPORT_SYMBOL_GPL(enable_pmu); module_param(enable_pmu, bool, 0444); -static bool __read_mostly eager_page_split = true; +bool __read_mostly eager_page_split = true; module_param(eager_page_split, bool, 0644); /* diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 1ebd5a7594da..d1836f69f20c 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -352,6 +352,8 @@ extern int pi_inject_timer; extern bool report_ignored_msrs; +extern bool eager_page_split; + static inline u64 nsec_to_cycles(struct kvm_vcpu *vcpu, u64 nsec) { return pvclock_scale_delta(nsec, vcpu->arch.virtual_tsc_mult, From patchwork Wed Jan 19 23:07:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12717870 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40967C433F5 for ; Wed, 19 Jan 2022 23:09:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235311AbiASXJN (ORCPT ); Wed, 19 Jan 2022 18:09:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50592 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344397AbiASXIQ (ORCPT ); Wed, 19 Jan 2022 18:08:16 -0500 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 502F6C061769 for ; Wed, 19 Jan 2022 15:08:16 -0800 (PST) Received: by mail-pg1-x549.google.com with SMTP id t1-20020a6564c1000000b002e7f31cf59fso2497663pgv.14 for ; Wed, 19 Jan 2022 15:08:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=MhfdLV2MTzZVWycYse2Rj3CRq19OgFZ+cwv24VRJY58=; b=aZfYz7IJbgQradgseDLIlHc/MdP1bw5kgVTbegaBGLOSHtCi30radQFvyRg9VqsOLw AlL5F0kN5xODLebfzI1SgLI+Y8pum+n99PQzG1o8MxUtkJSnnu5MWX6z3eUHiPqVe6Ag iZpGlUV78ee8+em2avbVTibi06XlZh9A42M8OCwyQFnmOQlEdBTlx618KJcdaKXsmrvx jxMDJrZta+leB4xrjag7WoHlwOWLvb0b2Ijz2iuUJPMFJz/n1NoHrdlPWUANsxtYbVPK 7zXh/otZ3oiY9oGlK3CG+nT32bp0cVLv0PFdoltK/9tYA6gWgIBF6gcQEU/CpxCI+ktV yo2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=MhfdLV2MTzZVWycYse2Rj3CRq19OgFZ+cwv24VRJY58=; b=7pfR5zP6/ET2Icvjy/CWoQvRn082Y9axblHuMW5a+5QPVPEBATu4qRMU6l+hCO956g GTLvlyA+j+zhzcVYu8JVAvEMpl01zOxXZg4+O72XYEDDYyo9ixXk3lg37fbxTgHwtPaT gOCj2//CJH+eaPwDPyH6bvEyEE8PBIEEFNxfM9sw20LVqPG/9JujKtdPAX5Mr20OdVYq lbagi6HHq6uVWTFIGFaTXF8ubhprofd190qgLlW7pfaeJ5ghoHI8IqMJgE8u2VQ13x35 BhqreFU2VfThMUvWFGXfuaf3SZUj18aklJodhOdFo+gp/0e1amiH/8j5z0WcJvIVbv/6 IhFw== X-Gm-Message-State: AOAM532bZWqVtNm/0XL7QAL6+lOOtl+FGZ8JrgKS7AO2vdfNljfpiwnp 5R/r7z/st9bsKBS3REY8B67rpGPDVeAqKA== X-Google-Smtp-Source: ABdhPJxbRD3heZAlfS9i1V5faPgoS+9fkxWd67rk0P0mqu91fowvZvxnW2DMrNThlZBBrifhXG0lyYXhFqM4MA== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:90b:4f44:: with SMTP id pj4mr3583pjb.1.1642633695542; Wed, 19 Jan 2022 15:08:15 -0800 (PST) Date: Wed, 19 Jan 2022 23:07:38 +0000 In-Reply-To: <20220119230739.2234394-1-dmatlack@google.com> Message-Id: <20220119230739.2234394-18-dmatlack@google.com> Mime-Version: 1.0 References: <20220119230739.2234394-1-dmatlack@google.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog Subject: [PATCH v2 17/18] KVM: x86/mmu: Add tracepoint for splitting huge pages From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add a tracepoint that records whenever KVM eagerly splits a huge page and the error status of the split to indicate if it succeeded or failed and why. Reviewed-by: Peter Xu Signed-off-by: David Matlack --- arch/x86/kvm/mmu/mmutrace.h | 23 +++++++++++++++++++++++ arch/x86/kvm/mmu/tdp_mmu.c | 10 +++++++--- 2 files changed, 30 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/mmu/mmutrace.h b/arch/x86/kvm/mmu/mmutrace.h index de5e8e4e1aa7..12247b96af01 100644 --- a/arch/x86/kvm/mmu/mmutrace.h +++ b/arch/x86/kvm/mmu/mmutrace.h @@ -416,6 +416,29 @@ TRACE_EVENT( ) ); +TRACE_EVENT( + kvm_mmu_split_huge_page, + TP_PROTO(u64 gfn, u64 spte, int level, int errno), + TP_ARGS(gfn, spte, level, errno), + + TP_STRUCT__entry( + __field(u64, gfn) + __field(u64, spte) + __field(int, level) + __field(int, errno) + ), + + TP_fast_assign( + __entry->gfn = gfn; + __entry->spte = spte; + __entry->level = level; + __entry->errno = errno; + ), + + TP_printk("gfn %llx spte %llx level %d errno %d", + __entry->gfn, __entry->spte, __entry->level, __entry->errno) +); + #endif /* _TRACE_KVMMMU_H */ #undef TRACE_INCLUDE_PATH diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index d5e713b849e9..1ed8e20270f0 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1332,7 +1332,7 @@ static int tdp_mmu_split_huge_page(struct kvm *kvm, struct tdp_iter *iter, */ ret = tdp_mmu_link_sp(kvm, iter, sp, false, shared); if (ret) - return ret; + goto out; /* * tdp_mmu_link_sp_atomic() will handle subtracting the huge page we @@ -1341,7 +1341,9 @@ static int tdp_mmu_split_huge_page(struct kvm *kvm, struct tdp_iter *iter, */ kvm_update_page_stats(kvm, level - 1, PT64_ENT_PER_PAGE); - return 0; +out: + trace_kvm_mmu_split_huge_page(iter->gfn, huge_spte, level, ret); + return ret; } static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, @@ -1378,6 +1380,9 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, sp = tdp_mmu_alloc_sp_for_split(kvm, &iter, shared); if (!sp) { ret = -ENOMEM; + trace_kvm_mmu_split_huge_page(iter.gfn, + iter.old_spte, + iter.level, ret); break; } @@ -1401,7 +1406,6 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, if (sp) tdp_mmu_free_sp(sp); - return ret; } From patchwork Wed Jan 19 23:07:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12717876 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56B1AC433F5 for ; Wed, 19 Jan 2022 23:09:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1357870AbiASXJ0 (ORCPT ); Wed, 19 Jan 2022 18:09:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50600 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344287AbiASXIR (ORCPT ); Wed, 19 Jan 2022 18:08:17 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 93E51C06176A for ; Wed, 19 Jan 2022 15:08:17 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id j28-20020a637a5c000000b00344d66c3c56so2482049pgn.21 for ; Wed, 19 Jan 2022 15:08:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=1mwS8UFzPeGHbHB3oNsb9qE1Tl9VmXoGSH+niC6z9fo=; b=SnlhlOMX8OV6lr19uJ20DDApqPmv3JqDhwEBgkvu7RY8vwYrrqnKTu5YmrShxQfei3 XiE0ju5a2v3/RkAtIUT57inttXeESzg2YPZkURkmLe7J4IrqJKCh7fehi657X2Fhyf9M 7FYWeOy2YjZrv53Q3q7b3RDn0DZCFOqinKdDjXqq+U0uXHQ1NaEeY594vvLhattfijkf IfL9aCCcRW8j5z2ayf01yZYoPAhZyrDPfs0rdEfHHrqok2V4XEYLE1mAa0jr3VJKl6ty HX7s2uQVDs3cwGtf49WYTC7yEUVkcrhrYztS0r++r3hgqmkE4pI0F1KePHH2EaJ+XdRz w9Cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=1mwS8UFzPeGHbHB3oNsb9qE1Tl9VmXoGSH+niC6z9fo=; b=pyq7zPF+fF1yd/5O5H+nFCKBa9UhzIH2PRGwez9FQVmlqz1/LehL5jolSHIvRoDju+ XoxNrHSSOWdoO5tCaLOo7BF3Xn8s/TAoZICYYkJETZfBhuB8ARDw8Dkwx7hHyd6EpV8g X3WFsAL9tybMPkXpTgBZwXgjQM08D4WkAn0IqY/Upto5zhoU2nBHNywE//xMFsHJ1Cy8 jJeGV/aAv8pdmhD72YF4UnE1ta+aPT1jxgpbySEe+GKfznyF+1iEbYqL6e/SdPtSPWiH wxXjA5mhoAxz+uzZ+WPttLEw28kOQVcKSnGiySvAk4cx7GPwgE/BAeznWspg+rka60NV YBDg== X-Gm-Message-State: AOAM533MziOoRJZCQeUR0rLro7r64A/Rpnw3zIilThyazZfzqI5/fLcg Wnv8Z18aSUreOw9iF6kOMLs7YMqGUK00HA== X-Google-Smtp-Source: ABdhPJyG3e5WSJTPDMLJOh9AIB118lYazD2h2vEPy+XXFKnesLILNWoLQYBFQ0qo4wdcWg6RT65Iy1BxLni5xQ== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:902:6a89:b0:149:732e:d335 with SMTP id n9-20020a1709026a8900b00149732ed335mr35509433plk.136.1642633697066; Wed, 19 Jan 2022 15:08:17 -0800 (PST) Date: Wed, 19 Jan 2022 23:07:39 +0000 In-Reply-To: <20220119230739.2234394-1-dmatlack@google.com> Message-Id: <20220119230739.2234394-19-dmatlack@google.com> Mime-Version: 1.0 References: <20220119230739.2234394-1-dmatlack@google.com> X-Mailer: git-send-email 2.35.0.rc0.227.g00780c9af4-goog Subject: [PATCH v2 18/18] KVM: selftests: Add an option to disable MANUAL_PROTECT_ENABLE and INITIALLY_SET From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add an option to dirty_log_perf_test.c to disable KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE and KVM_DIRTY_LOG_INITIALLY_SET so the legacy dirty logging code path can be tested. Reviewed-by: Peter Xu Signed-off-by: David Matlack --- tools/testing/selftests/kvm/dirty_log_perf_test.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/kvm/dirty_log_perf_test.c b/tools/testing/selftests/kvm/dirty_log_perf_test.c index 1954b964d1cf..101759ac93a4 100644 --- a/tools/testing/selftests/kvm/dirty_log_perf_test.c +++ b/tools/testing/selftests/kvm/dirty_log_perf_test.c @@ -298,12 +298,18 @@ static void run_test(enum vm_guest_mode mode, void *arg) static void help(char *name) { puts(""); - printf("usage: %s [-h] [-i iterations] [-p offset] " + printf("usage: %s [-h] [-i iterations] [-p offset] [-g]" "[-m mode] [-b vcpu bytes] [-v vcpus] [-o] [-s mem type]" "[-x memslots]\n", name); puts(""); printf(" -i: specify iteration counts (default: %"PRIu64")\n", TEST_HOST_LOOP_N); + printf(" -g: Do not enable KVM_CAP_MANUAL_DIRTY_LOG_PROTECT2. This\n" + " makes KVM_GET_DIRTY_LOG clear the dirty log (i.e.\n" + " KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE is not enabled)\n" + " and writes will be tracked as soon as dirty logging is\n" + " enabled on the memslot (i.e. KVM_DIRTY_LOG_INITIALLY_SET\n" + " is not enabled).\n"); printf(" -p: specify guest physical test memory offset\n" " Warning: a low offset can conflict with the loaded test code.\n"); guest_modes_help(); @@ -343,8 +349,11 @@ int main(int argc, char *argv[]) guest_modes_append_default(); - while ((opt = getopt(argc, argv, "hi:p:m:b:f:v:os:x:")) != -1) { + while ((opt = getopt(argc, argv, "ghi:p:m:b:f:v:os:x:")) != -1) { switch (opt) { + case 'g': + dirty_log_manual_caps = 0; + break; case 'i': p.iterations = atoi(optarg); break;