From patchwork Mon Dec 13 22:59:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12674797 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 390DBC433EF for ; Mon, 13 Dec 2021 22:59:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244041AbhLMW7Y (ORCPT ); Mon, 13 Dec 2021 17:59:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47482 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242437AbhLMW7X (ORCPT ); Mon, 13 Dec 2021 17:59:23 -0500 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A34CC061574 for ; Mon, 13 Dec 2021 14:59:23 -0800 (PST) Received: by mail-pf1-x449.google.com with SMTP id h17-20020aa79f51000000b0049473df362dso1596690pfr.12 for ; Mon, 13 Dec 2021 14:59:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=gtNPOCDR3DbNyqbZWIIfJnGYbH6wsvGmr+VRgwX1ZFs=; b=qYPn8QtLxWzWyCID5h01OZ8d2Y1qcV38gxv2flMXQFxiMNxN6TsNQ8OfSUMQBCyQWl RiAknD7NM0UJXk6myByqfXLAu0Rvgg+ZSCdZZ9qDolADAHtWs8pigBmiydN2gW7HSbkB Pa+7PKHKcVzkRZvSv67uR+M/nhpw3QsRGeMdpQ2WbD0TUlB6gPk8f6DUMWCejc/mAtIE K2jeqZ5DAZ5fzFV0vNglF3PEBrcMGVdjyb3utgMT5PeWpvgC5kEc5m4Y8BEsurB5hoxg SssVOsegmBN3lhSGkeoUzCnbkz6BUo3W2QAxdRvojWz4Ke8OK1BokVAvHUaKM03jR9oZ S6Rg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=gtNPOCDR3DbNyqbZWIIfJnGYbH6wsvGmr+VRgwX1ZFs=; b=bcs3XQ5p7KJFYfcJMW2SGLuz1jaBFYaiOPBPnWQaldyG0jcGJrnCHq0GpNLwo4aIPw lHnHqGb/QZNLNPN/QucHAJ/OK9p/P5XC5tI5HPI6R5IfgCN+cytSnwxqnc/URD+2TbmU eec/xY/w4/icxXfHAbBomIHBAGTUKeYRh7kqhkNd04mBD6O9H6/BNdwI5j9ZPkHwS/4m MhuqM5t3mAuXJWKfKp9DHlXojHSW2BXl/7+VbbEuL/0RXrKlcWMMUgQPDEcrgNhbZ0h6 Jxo8Iu1xQ0KScs97mfQPV6X7db+I+95Jr5QrI4qc16hSgQHPbHb+54TgUK96U9npdpJs a2tw== X-Gm-Message-State: AOAM530Cy36SpCPa+Da0S+NSOLiQOcsZuOTqVQAEiFu5pzA68zZrfzTJ S3ERwGr0tvR6teR7FSxgxyt0+So4az+93A== X-Google-Smtp-Source: ABdhPJwsaCFlbJed4eBWy4RskyP0rehNDsaD+zpJgRfqOYjOstc3oQpb1kQb4lwXtEpFUY5vPQBNNgsqAXpNXQ== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:902:b7cb:b0:141:b33a:9589 with SMTP id v11-20020a170902b7cb00b00141b33a9589mr1182476plz.9.1639436362919; Mon, 13 Dec 2021 14:59:22 -0800 (PST) Date: Mon, 13 Dec 2021 22:59:06 +0000 In-Reply-To: <20211213225918.672507-1-dmatlack@google.com> Message-Id: <20211213225918.672507-2-dmatlack@google.com> Mime-Version: 1.0 References: <20211213225918.672507-1-dmatlack@google.com> X-Mailer: git-send-email 2.34.1.173.g76aa8bc2d0-goog Subject: [PATCH v1 01/13] KVM: x86/mmu: Rename rmap_write_protect to kvm_vcpu_write_protect_gfn From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org rmap_write_protect is a poor name because we may not even touch the rmap if the TDP MMU is in use. It is also confusing that rmap_write_protect is not a simpler wrapper around __rmap_write_protect, since that is the typical flow for functions with double-underscore names. Rename it to kvm_vcpu_write_protect_gfn to convey that we are write-protecting a specific gfn in the context of a vCPU. No functional change intended. Signed-off-by: David Matlack Reviewed-by: Ben Gardon Reviewed-by: Peter Xu Reviewed-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) base-commit: 1c10f4b4877ffaed602d12ff8cbbd5009e82c970 diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 1ccee4d17481..87c3135222b3 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1421,7 +1421,7 @@ bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm, return write_protected; } -static bool rmap_write_protect(struct kvm_vcpu *vcpu, u64 gfn) +static bool kvm_vcpu_write_protect_gfn(struct kvm_vcpu *vcpu, u64 gfn) { struct kvm_memory_slot *slot; @@ -2024,7 +2024,7 @@ static int mmu_sync_children(struct kvm_vcpu *vcpu, bool protected = false; for_each_sp(pages, sp, parents, i) - protected |= rmap_write_protect(vcpu, sp->gfn); + protected |= kvm_vcpu_write_protect_gfn(vcpu, sp->gfn); if (protected) { kvm_mmu_remote_flush_or_zap(vcpu->kvm, &invalid_list, true); @@ -2149,7 +2149,7 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu, hlist_add_head(&sp->hash_link, sp_list); if (!direct) { account_shadowed(vcpu->kvm, sp); - if (level == PG_LEVEL_4K && rmap_write_protect(vcpu, gfn)) + if (level == PG_LEVEL_4K && kvm_vcpu_write_protect_gfn(vcpu, gfn)) kvm_flush_remote_tlbs_with_address(vcpu->kvm, gfn, 1); } trace_kvm_mmu_get_page(sp, true); From patchwork Mon Dec 13 22:59:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12674799 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C13A3C433F5 for ; Mon, 13 Dec 2021 22:59:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244058AbhLMW70 (ORCPT ); Mon, 13 Dec 2021 17:59:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47494 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242437AbhLMW7Z (ORCPT ); Mon, 13 Dec 2021 17:59:25 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 90777C061574 for ; Mon, 13 Dec 2021 14:59:25 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id z13-20020a63e10d000000b0033b165097ccso2825414pgh.6 for ; Mon, 13 Dec 2021 14:59:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=2TA0P4t2+QQyG2Bby8P9l3x7+4tWbJz2qK7x6SwpSAM=; b=Z4+oqeg3MDKtSXeSdu4mIvuIO/hXt4vXDxoJDwPaetMmB7RscplC93rG56++9Lmthr QMTRRrAgdFQhyrJVU3DgW0zlLqk3G4BDynIepMkm7iOQNvQVHQzwNXvyVt7g4QEOhzm8 rwCtpchrwX2PhP01ww3Km4d3Neuil0Ja03oL3ed7QMRDg+dBcYkmia1iKQbExLpUHk+i Nl1iRO9sTy4CtvuCmTBCbPPFWjCR33HUZeOfQM8fpdFZnU+387HWqUT0TT/4qDrst9x6 s/JufiSmoSFjLfA3TrGvDiCRTg3ykXmhHo9AZITWVHJmqhRYXqF4Kf7lZfaYCKgr0S3K VtmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=2TA0P4t2+QQyG2Bby8P9l3x7+4tWbJz2qK7x6SwpSAM=; b=a2rRCL9S3SR3CMVQJPo0jMRKHRpkyNrFeHaC+DpTARtb9xFX0UtKGIhHAQK4NldQzN C4uvEb7bL37w/PjDEYfj6bI6DpP5dmxRDggBmzpMP1o+CYgMq9VfM5Pr0X4rhhVWhpCb /erHBzStoF/+RYA3snFlEbDnzQvbxmQiX3hsqy/ooCJ5DrSVsphULmOu1j05CuiWU+EN K3dnaRUBFm1+P/6qFFi1d8utkR/DjVFRMLjcRJ0SPa0OZaSPrrW0OxA5o6rd5O9qBF3b V12kM/gceY9lK328LgB4KmGr69fwufUQ75nDbWmYLt/xZ1/GIIOlUScsGKsd/ZEfIKxj nEhw== X-Gm-Message-State: AOAM531oaL6YEEoYq4zC5AK5RpYI4vGWh7d/kSYvVARrJ13n9E+wq448 FXKuOvRRMS0BxDwMbVxAcgVmcg33V0iy+A== X-Google-Smtp-Source: ABdhPJyqpNq4jk0Hi8blCyO9o4k21Ls7fQyDDPkXGGsF9MaFNL8YuZXCr/FiE0HziCqj3o3bdglt/xkGPa6gCg== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:90b:a16:: with SMTP id gg22mr1061071pjb.0.1639436364524; Mon, 13 Dec 2021 14:59:24 -0800 (PST) Date: Mon, 13 Dec 2021 22:59:07 +0000 In-Reply-To: <20211213225918.672507-1-dmatlack@google.com> Message-Id: <20211213225918.672507-3-dmatlack@google.com> Mime-Version: 1.0 References: <20211213225918.672507-1-dmatlack@google.com> X-Mailer: git-send-email 2.34.1.173.g76aa8bc2d0-goog Subject: [PATCH v1 02/13] KVM: x86/mmu: Rename __rmap_write_protect to rmap_write_protect From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Now that rmap_write_protect has been renamed, there is no need for the double underscores in front of __rmap_write_protect. No functional change intended. Signed-off-by: David Matlack Reviewed-by: Ben Gardon Reviewed-by: Peter Xu Reviewed-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 87c3135222b3..8b702f2b6a70 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1229,9 +1229,9 @@ static bool spte_write_protect(u64 *sptep, bool pt_protect) return mmu_spte_update(sptep, spte); } -static bool __rmap_write_protect(struct kvm *kvm, - struct kvm_rmap_head *rmap_head, - bool pt_protect) +static bool rmap_write_protect(struct kvm *kvm, + struct kvm_rmap_head *rmap_head, + bool pt_protect) { u64 *sptep; struct rmap_iterator iter; @@ -1311,7 +1311,7 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm, while (mask) { rmap_head = gfn_to_rmap(slot->base_gfn + gfn_offset + __ffs(mask), PG_LEVEL_4K, slot); - __rmap_write_protect(kvm, rmap_head, false); + rmap_write_protect(kvm, rmap_head, false); /* clear the first set bit */ mask &= mask - 1; @@ -1410,7 +1410,7 @@ bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm, if (kvm_memslots_have_rmaps(kvm)) { for (i = min_level; i <= KVM_MAX_HUGEPAGE_LEVEL; ++i) { rmap_head = gfn_to_rmap(gfn, i, slot); - write_protected |= __rmap_write_protect(kvm, rmap_head, true); + write_protected |= rmap_write_protect(kvm, rmap_head, true); } } @@ -5787,7 +5787,7 @@ static bool slot_rmap_write_protect(struct kvm *kvm, struct kvm_rmap_head *rmap_head, const struct kvm_memory_slot *slot) { - return __rmap_write_protect(kvm, rmap_head, false); + return rmap_write_protect(kvm, rmap_head, false); } void kvm_mmu_slot_remove_write_access(struct kvm *kvm, From patchwork Mon Dec 13 22:59:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12674801 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 819E7C433EF for ; Mon, 13 Dec 2021 22:59:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244065AbhLMW72 (ORCPT ); Mon, 13 Dec 2021 17:59:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47504 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244059AbhLMW71 (ORCPT ); Mon, 13 Dec 2021 17:59:27 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 368EDC061574 for ; Mon, 13 Dec 2021 14:59:27 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id k11-20020a63d84b000000b003252e72da7eso9687204pgj.23 for ; Mon, 13 Dec 2021 14:59:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=9BHI/mobmlpPhu8DAqt3Cjhleh/JUHkhj3UVTpYC7Qo=; b=Tydtockk7dWRR2uLeiPyQoQfTCpoN26GVQ+IjYUt0TSV148yw/H08wBBoWGgAtE8M3 bkEwTT4kNDgAbwxeFOr+JQDNymcfTXeyoihlBY9l6AGlyXrGyzE3F1nOe5CrlVOz+giD rB4A1U4hqjPTnMwDdYBdLGmvpazhd1u/fj3vcMWagi6JPictAQT9y0LJNS/GcvpCJzIH F1SYj8tXLmjPY8fsea3VTIXv7BvaUjcPqEPZRDvMIkXgpPcY70x+MZAGkrPY+Nj8+QzI 9supmF0/c2P9gtrPx0X5njCVyhTjBdKGlEFcGhtQDLmZcDSVFt1Shlo7BxPjrYXpkAl8 8Gkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=9BHI/mobmlpPhu8DAqt3Cjhleh/JUHkhj3UVTpYC7Qo=; b=VWUYKKhXssbYr6ZzWxnKIz/zK7swmx3bdbdQcNuBJ3R1di6uWRiJszaR13rw03viW/ k58o3Lx38OeLYiFznudiHGhs7oDcWR/8x7GYfYzoghI6GyeOq81scL3qNt4t96gMA121 B8XrWFSBLmzIt5yXBg3RxgrhmXB7M0OBhcV7mcGm1GOmwRpaynRaJH9Mbo//LPXLdYdm oqJynsNTT/P7bjoIfga1UcVaqz1RRqr2HFsNCz83Fu4a+AnCVNS8gdHymJt3rkPCYpy5 i+4X/TIqpgVV467u4MYlY5lYfSIgyOIiFzqkP5s6Moih30MNGVGoTIR15iVIuhI3nk3M XbNg== X-Gm-Message-State: AOAM531eE6Zu2Qmt4ht7cjyrPJ05bbNADis5XxyG2Moo3V5MbniocUNq 57bSVlSov+bEuLUnZpyFR5UmN0KlXeKxRg== X-Google-Smtp-Source: ABdhPJx2fyZCvqWmZyyqRKneoB+xXbF/HSrFUSyDoLLZR1CuuDxpNCdnrIF2rWFyhkbfEOOgVz4VPjRdJk7b+Q== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:902:6b8c:b0:148:8a86:a01c with SMTP id p12-20020a1709026b8c00b001488a86a01cmr1209381plk.50.1639436366709; Mon, 13 Dec 2021 14:59:26 -0800 (PST) Date: Mon, 13 Dec 2021 22:59:08 +0000 In-Reply-To: <20211213225918.672507-1-dmatlack@google.com> Message-Id: <20211213225918.672507-4-dmatlack@google.com> Mime-Version: 1.0 References: <20211213225918.672507-1-dmatlack@google.com> X-Mailer: git-send-email 2.34.1.173.g76aa8bc2d0-goog Subject: [PATCH v1 03/13] KVM: x86/mmu: Automatically update iter->old_spte if cmpxchg fails From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Consolidate a bunch of code that was manually re-reading the spte if the cmpxchg fails. There is no extra cost of doing this because we already have the spte value as a result of the cmpxchg (and in fact this eliminates re-reading the spte), and none of the call sites depend on iter->old_spte retaining the stale spte value. Signed-off-by: David Matlack Reviewed-by: Peter Xu Reviewed-by: Ben Gardon Reviewed-by: Sean Christopherson --- arch/x86/kvm/mmu/tdp_mmu.c | 50 ++++++++++++++++---------------------- 1 file changed, 21 insertions(+), 29 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index b69e47e68307..656ebf5b20dc 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -492,16 +492,22 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, * and handle the associated bookkeeping. Do not mark the page dirty * in KVM's dirty bitmaps. * + * If setting the SPTE fails because it has changed, iter->old_spte will be + * updated with the updated value of the spte. + * * @kvm: kvm instance * @iter: a tdp_iter instance currently on the SPTE that should be set * @new_spte: The value the SPTE should be set to * Returns: true if the SPTE was set, false if it was not. If false is returned, - * this function will have no side-effects. + * this function will have no side-effects other than updating + * iter->old_spte to the latest value of spte. */ static inline bool tdp_mmu_set_spte_atomic(struct kvm *kvm, struct tdp_iter *iter, u64 new_spte) { + u64 old_spte; + lockdep_assert_held_read(&kvm->mmu_lock); /* @@ -515,9 +521,15 @@ static inline bool tdp_mmu_set_spte_atomic(struct kvm *kvm, * Note, fast_pf_fix_direct_spte() can also modify TDP MMU SPTEs and * does not hold the mmu_lock. */ - if (cmpxchg64(rcu_dereference(iter->sptep), iter->old_spte, - new_spte) != iter->old_spte) + old_spte = cmpxchg64(rcu_dereference(iter->sptep), iter->old_spte, new_spte); + if (old_spte != iter->old_spte) { + /* + * The cmpxchg failed because the spte was updated by another + * thread so record the updated spte in old_spte. + */ + iter->old_spte = old_spte; return false; + } __handle_changed_spte(kvm, iter->as_id, iter->gfn, iter->old_spte, new_spte, iter->level, true); @@ -748,11 +760,6 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root, tdp_mmu_set_spte(kvm, &iter, 0); flush = true; } else if (!tdp_mmu_zap_spte_atomic(kvm, &iter)) { - /* - * The iter must explicitly re-read the SPTE because - * the atomic cmpxchg failed. - */ - iter.old_spte = READ_ONCE(*rcu_dereference(iter.sptep)); goto retry; } } @@ -985,6 +992,7 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) * path below. */ iter.old_spte = READ_ONCE(*rcu_dereference(iter.sptep)); + } if (!is_shadow_present_pte(iter.old_spte)) { @@ -1190,14 +1198,9 @@ static bool wrprot_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root, new_spte = iter.old_spte & ~PT_WRITABLE_MASK; - if (!tdp_mmu_set_spte_atomic(kvm, &iter, new_spte)) { - /* - * The iter must explicitly re-read the SPTE because - * the atomic cmpxchg failed. - */ - iter.old_spte = READ_ONCE(*rcu_dereference(iter.sptep)); + if (!tdp_mmu_set_spte_atomic(kvm, &iter, new_spte)) goto retry; - } + spte_set = true; } @@ -1258,14 +1261,9 @@ static bool clear_dirty_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root, continue; } - if (!tdp_mmu_set_spte_atomic(kvm, &iter, new_spte)) { - /* - * The iter must explicitly re-read the SPTE because - * the atomic cmpxchg failed. - */ - iter.old_spte = READ_ONCE(*rcu_dereference(iter.sptep)); + if (!tdp_mmu_set_spte_atomic(kvm, &iter, new_spte)) goto retry; - } + spte_set = true; } @@ -1389,14 +1387,8 @@ static void zap_collapsible_spte_range(struct kvm *kvm, continue; /* Note, a successful atomic zap also does a remote TLB flush. */ - if (!tdp_mmu_zap_spte_atomic(kvm, &iter)) { - /* - * The iter must explicitly re-read the SPTE because - * the atomic cmpxchg failed. - */ - iter.old_spte = READ_ONCE(*rcu_dereference(iter.sptep)); + if (!tdp_mmu_zap_spte_atomic(kvm, &iter)) goto retry; - } } rcu_read_unlock(); From patchwork Mon Dec 13 22:59:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12674803 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CE13C433FE for ; Mon, 13 Dec 2021 22:59:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244053AbhLMW7a (ORCPT ); Mon, 13 Dec 2021 17:59:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47518 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244066AbhLMW72 (ORCPT ); Mon, 13 Dec 2021 17:59:28 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B60C1C061751 for ; Mon, 13 Dec 2021 14:59:28 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id gf15-20020a17090ac7cf00b001a9a31687d0so9486269pjb.1 for ; Mon, 13 Dec 2021 14:59:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=7FL2w8dpPfO+G2QLXIxjm0csK2DC0KxYpauIDjxm0W4=; b=JZJEl3xodRJIDgfu8OkIMkbS7DBGGMWibPsNL7OqBNacy1HHNZJkU/XLzr71r0zHxl 6ejE1eCRfIN9J+cAWg0/ms7ZJfYa0+CMF0sbvM3VmbM/bl2ZTAzA2B6GCLZnXb9YKfGQ vIRsRQ4hF5MNj7Uvc41GNTkl4amUj8iACgfohHSJFnQRX/62aAOfu8JJuNrn+FbR80jF ze5GuK2omvjVaGp5c9cdwIkdHhnm+0eY8JcVFBVMib47fOe0NsIGE0KnJ5D3i4nmKYgF dwMUaTRFZHH1WOcqRAzjtmyUmD095UIi3RgOaXDXAyWkECpDeGYF0pSIKLAaL8KGiXnG ejOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=7FL2w8dpPfO+G2QLXIxjm0csK2DC0KxYpauIDjxm0W4=; b=YG610YprjkZY9QuveD6l1BU+KO57Ld2lDs+rWWOKLJQjjED2DYSmex6XLGo4ZPfl50 crQibGoBxMThJ5En8Eh6f2uiFK74vxbGqFINktkm5RLtlCrMwhnuHhmmYaAuHsms64ah ZZiU935+elnmX2bKFSnn66SwfFxB8qGlXE9mn98BpP8pY7POwU3IWV/WjmJn2oF1ihJV LpwVDPtleb/J9GYwACau5m7pNC2e2bqM5wUal4FEr9e5ewt0srMz2EklWyyiFq8bZT/F PhKr4CJSuasRVzU3gr8oz/3jV7cIMM1BCvRNdA0tvld7ctYBYRUbMyQ0GVMqCi/WxVwi 5xoQ== X-Gm-Message-State: AOAM532MwcVsIID+5ANoIoR1vc99NIXViWC+LOT89D/hGzJcEMhS0bU2 oPuihZiltnNXE9oFditffxv2Xcm0cEuKTg== X-Google-Smtp-Source: ABdhPJzN2VsU0o8SxPQHyOSfFLQ6BjlTNn/thWcNFkZjJSo8shVVvaFc9PGJJ2RiiUpvuBzsWqQoN4TbH0jRsA== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:902:e804:b0:142:1c0b:c2a6 with SMTP id u4-20020a170902e80400b001421c0bc2a6mr1160949plg.23.1639436368190; Mon, 13 Dec 2021 14:59:28 -0800 (PST) Date: Mon, 13 Dec 2021 22:59:09 +0000 In-Reply-To: <20211213225918.672507-1-dmatlack@google.com> Message-Id: <20211213225918.672507-5-dmatlack@google.com> Mime-Version: 1.0 References: <20211213225918.672507-1-dmatlack@google.com> X-Mailer: git-send-email 2.34.1.173.g76aa8bc2d0-goog Subject: [PATCH v1 04/13] KVM: x86/mmu: Factor out logic to atomically install a new page table From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Factor out the logic to atomically replace an SPTE with an SPTE that points to a new page table. This will be used in a follow-up commit to split a large page SPTE into one level lower. Opportunistically drop the kvm_mmu_get_page tracepoint in kvm_tdp_mmu_map() since it is redundant with the identical tracepoint in alloc_tdp_mmu_page(). Signed-off-by: David Matlack Reviewed-by: Peter Xu --- arch/x86/kvm/mmu/tdp_mmu.c | 48 +++++++++++++++++++++++++++----------- 1 file changed, 34 insertions(+), 14 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 656ebf5b20dc..dbd07c10d11a 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -950,6 +950,36 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu, return ret; } +/* + * tdp_mmu_install_sp_atomic - Atomically replace the given spte with an + * spte pointing to the provided page table. + * + * @kvm: kvm instance + * @iter: a tdp_iter instance currently on the SPTE that should be set + * @sp: The new TDP page table to install. + * @account_nx: True if this page table is being installed to split a + * non-executable huge page. + * + * Returns: True if the new page table was installed. False if spte being + * replaced changed, causing the atomic compare-exchange to fail. + * If this function returns false the sp will be freed before + * returning. + */ +static bool tdp_mmu_install_sp_atomic(struct kvm *kvm, + struct tdp_iter *iter, + struct kvm_mmu_page *sp, + bool account_nx) +{ + u64 spte = make_nonleaf_spte(sp->spt, !shadow_accessed_mask); + + if (!tdp_mmu_set_spte_atomic(kvm, iter, spte)) + return false; + + tdp_mmu_link_page(kvm, sp, account_nx); + + return true; +} + /* * Handle a TDP page fault (NPT/EPT violation/misconfiguration) by installing * page tables and SPTEs to translate the faulting guest physical address. @@ -959,8 +989,6 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) struct kvm_mmu *mmu = vcpu->arch.mmu; struct tdp_iter iter; struct kvm_mmu_page *sp; - u64 *child_pt; - u64 new_spte; int ret; kvm_mmu_hugepage_adjust(vcpu, fault); @@ -996,6 +1024,9 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) } if (!is_shadow_present_pte(iter.old_spte)) { + bool account_nx = fault->huge_page_disallowed && + fault->req_level >= iter.level; + /* * If SPTE has been frozen by another thread, just * give up and retry, avoiding unnecessary page table @@ -1005,18 +1036,7 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) break; sp = alloc_tdp_mmu_page(vcpu, iter.gfn, iter.level - 1); - child_pt = sp->spt; - - new_spte = make_nonleaf_spte(child_pt, - !shadow_accessed_mask); - - if (tdp_mmu_set_spte_atomic(vcpu->kvm, &iter, new_spte)) { - tdp_mmu_link_page(vcpu->kvm, sp, - fault->huge_page_disallowed && - fault->req_level >= iter.level); - - trace_kvm_mmu_get_page(sp, true); - } else { + if (!tdp_mmu_install_sp_atomic(vcpu->kvm, &iter, sp, account_nx)) { tdp_mmu_free_sp(sp); break; } From patchwork Mon Dec 13 22:59:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12674805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79B8AC433F5 for ; Mon, 13 Dec 2021 22:59:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244062AbhLMW7b (ORCPT ); Mon, 13 Dec 2021 17:59:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47526 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242437AbhLMW7a (ORCPT ); Mon, 13 Dec 2021 17:59:30 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6F4BBC061574 for ; Mon, 13 Dec 2021 14:59:30 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id gf15-20020a17090ac7cf00b001a9a31687d0so9486300pjb.1 for ; Mon, 13 Dec 2021 14:59:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=CUA8Kpd1Qp4b7Qr2EMkSmyJ+XKRmg8wbahlTIVobdrE=; b=PbcEzTdKafMKp+eUzL1oodmKvuQHx6/c4TZfNF7+r53N2md0gHPMcpwQGuYtHgZr8g hGJe738lQgrtLog6yPCTT705/UXkkpbi+rItGaBZ8gpBO0NRF64lT6BW3lc0+frB8SLV QHNl9lSZoofOnbrYx23ziOF0Q4fN7ktsb9K7Ssk4McM+JDbsFk9Kj8mKmB1OiayU/zrS 4407ac0dAC/NCwvd3M9+ZFUrs/nwitysaupXFOikbSWq2VrwABUrvNqEa5CnfK2c75N6 X8+6pMc0t2yrB7w8PEV/AE+JLnf38/H0aDV0+mU4YE6fDfcxC7f7vuWpDgLKSa6nPjLI w5VA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=CUA8Kpd1Qp4b7Qr2EMkSmyJ+XKRmg8wbahlTIVobdrE=; b=vtsDpHhr7js3KOWPA1m7Wlh2UFzsM3HCcfX5FWOrq46aOwzzouvE+SijBistgI180E qBmvcQVpB6bQ1BXwBbHf+24tV9uscdD37t4GStul9eaLGl6hT3/XZVR0YMzC0DWdesFG Reh3/BYlA8DlJ99DAI5fXoWI0O+9g/5/yY6phEXMl/crM38S9ZT3PQ4AIbTPAGs8Wq9H E0H6RMpFfwhrqVwtiP3OQiUv6na9vsrkTiy/MptUKuRUGBR3BGGSFMTt9SFOlUBG1Glg XvCCATUCjAGAyXjYLkh6Dl366Wdz3GwO2Gi7ds/PAk9qhPHB6UKo2mnNkpMnMDAblmfh 6LAQ== X-Gm-Message-State: AOAM530v6q8+xG9en+/hiHTod9rC5WD3nGGhqHkHr9WIchjonMoUzX3j f/4UyVPph2mdZHz9qrxgwIl9jAhQ6LxS2A== X-Google-Smtp-Source: ABdhPJyF9vipN2UHMSw4CPV8el2571Um5meXegkAgICDEMzS+8ut7ADT8jfu+VcMHESL5tC+aPch9d0L8jE01A== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:90b:3889:: with SMTP id mu9mr1275973pjb.160.1639436369901; Mon, 13 Dec 2021 14:59:29 -0800 (PST) Date: Mon, 13 Dec 2021 22:59:10 +0000 In-Reply-To: <20211213225918.672507-1-dmatlack@google.com> Message-Id: <20211213225918.672507-6-dmatlack@google.com> Mime-Version: 1.0 References: <20211213225918.672507-1-dmatlack@google.com> X-Mailer: git-send-email 2.34.1.173.g76aa8bc2d0-goog Subject: [PATCH v1 05/13] KVM: x86/mmu: Move restore_acc_track_spte to spte.c From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org restore_acc_track_spte is purely an SPTE manipulation, making it a good fit for spte.c. It is also needed in spte.c in a follow-up commit so we can construct child SPTEs during large page splitting. No functional change intended. Signed-off-by: David Matlack Reviewed-by: Ben Gardon Reviewed-by: Peter Xu --- arch/x86/kvm/mmu/mmu.c | 18 ------------------ arch/x86/kvm/mmu/spte.c | 18 ++++++++++++++++++ arch/x86/kvm/mmu/spte.h | 1 + 3 files changed, 19 insertions(+), 18 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 8b702f2b6a70..3c2cb4dd1f11 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -646,24 +646,6 @@ static u64 mmu_spte_get_lockless(u64 *sptep) return __get_spte_lockless(sptep); } -/* Restore an acc-track PTE back to a regular PTE */ -static u64 restore_acc_track_spte(u64 spte) -{ - u64 new_spte = spte; - u64 saved_bits = (spte >> SHADOW_ACC_TRACK_SAVED_BITS_SHIFT) - & SHADOW_ACC_TRACK_SAVED_BITS_MASK; - - WARN_ON_ONCE(spte_ad_enabled(spte)); - WARN_ON_ONCE(!is_access_track_spte(spte)); - - new_spte &= ~shadow_acc_track_mask; - new_spte &= ~(SHADOW_ACC_TRACK_SAVED_BITS_MASK << - SHADOW_ACC_TRACK_SAVED_BITS_SHIFT); - new_spte |= saved_bits; - - return new_spte; -} - /* Returns the Accessed status of the PTE and resets it at the same time. */ static bool mmu_spte_age(u64 *sptep) { diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index 8a7b03207762..fd34ae5d6940 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -268,6 +268,24 @@ u64 mark_spte_for_access_track(u64 spte) return spte; } +/* Restore an acc-track PTE back to a regular PTE */ +u64 restore_acc_track_spte(u64 spte) +{ + u64 new_spte = spte; + u64 saved_bits = (spte >> SHADOW_ACC_TRACK_SAVED_BITS_SHIFT) + & SHADOW_ACC_TRACK_SAVED_BITS_MASK; + + WARN_ON_ONCE(spte_ad_enabled(spte)); + WARN_ON_ONCE(!is_access_track_spte(spte)); + + new_spte &= ~shadow_acc_track_mask; + new_spte &= ~(SHADOW_ACC_TRACK_SAVED_BITS_MASK << + SHADOW_ACC_TRACK_SAVED_BITS_SHIFT); + new_spte |= saved_bits; + + return new_spte; +} + void kvm_mmu_set_mmio_spte_mask(u64 mmio_value, u64 mmio_mask, u64 access_mask) { BUG_ON((u64)(unsigned)access_mask != access_mask); diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index a4af2a42695c..9b0c7b27f23f 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -337,6 +337,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, u64 make_nonleaf_spte(u64 *child_pt, bool ad_disabled); u64 make_mmio_spte(struct kvm_vcpu *vcpu, u64 gfn, unsigned int access); u64 mark_spte_for_access_track(u64 spte); +u64 restore_acc_track_spte(u64 spte); u64 kvm_mmu_changed_pte_notifier_make_spte(u64 old_spte, kvm_pfn_t new_pfn); void kvm_mmu_reset_all_pte_masks(void); From patchwork Mon Dec 13 22:59:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12674807 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C548C433F5 for ; Mon, 13 Dec 2021 22:59:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244092AbhLMW7d (ORCPT ); Mon, 13 Dec 2021 17:59:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47540 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244071AbhLMW7d (ORCPT ); Mon, 13 Dec 2021 17:59:33 -0500 Received: from mail-qk1-x74a.google.com (mail-qk1-x74a.google.com [IPv6:2607:f8b0:4864:20::74a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ACED5C061574 for ; Mon, 13 Dec 2021 14:59:32 -0800 (PST) Received: by mail-qk1-x74a.google.com with SMTP id bs14-20020a05620a470e00b0046b1e29f53cso15974422qkb.0 for ; Mon, 13 Dec 2021 14:59:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=NOgYZl20nuzj8fG4+DAG1wWXIv0xjdk8c+/nZNXlwis=; b=jFZn4tVVuETK5sVmX65cljDVjDeVNpmI/wox6rhf53J2ZOd1EagBapndwYSMWKA/0W ho0XOi6nntwM+SrHIu1Qf6d5z+WaAEd+UMn4l/WgDAbVropsGO1lTfuLscPqlMZb/QgZ Q9hWYyI4qGtV3pVwTG3rTLPKNwXb7qwHe0VTqwk1TTLrqDOjoGXViTZGH426NGClFFEb 5hBghiDOXJFnsTD7epFX4Ofw4+Fzgnod4jpiZXbVaJ7dwEDzd/DkgTspSHIbZqVATuSG khtnZsEDPbG/MRWJc5DUBNXfZuWY4EIJ2ustuGpulRmlKopTAkNS2x9Eppotta7C8ZXz eeJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=NOgYZl20nuzj8fG4+DAG1wWXIv0xjdk8c+/nZNXlwis=; b=V5Xh/fDCkEkp+joCY8stmIOQNwOO4ACbwZuZ8jsULOZWoATw0dlVP2Zr+XYfidEizG N2kD8k7NXdGmcwgbWfrkL6W5IVPckj05i+Sp1WfFpVJMMMU2vlP78O/3/4+Ch0qf95LR JxuMFrBsXKizlB8YEaXWBgZHIWQ4vC/NgW46ODJRiA4XZqttCU80ojQfmChrij0doT0b k8abB8d4pu37xHXbH2MNKmAJpE5flZPWUA08f+rSe5HVAekXdS7zQ9O06oVovsKBy31d Wp8bK1A08yNi2vmTapEZM1lna3/RzYDARIlU+S6bHafFgD0sXkol0mqZI4z8AxGFD/3v B/Vg== X-Gm-Message-State: AOAM533WLn+EFcFxgqRgmMi7VK+RWRww39XmQXx98EwE+i+SATesJoq8 QEWC4lP2eE4XNrnfCUYcpvzLRS+NAzYDbg== X-Google-Smtp-Source: ABdhPJzBPWNKAk0WbXAy88XhQI/1ggjFYVsEExKh6UMrl/2jN0yx8LHNgf9a+i+YX1wjV1Tvz/gPQfa7yYPIXg== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:ac8:7595:: with SMTP id s21mr1724407qtq.132.1639436371689; Mon, 13 Dec 2021 14:59:31 -0800 (PST) Date: Mon, 13 Dec 2021 22:59:11 +0000 In-Reply-To: <20211213225918.672507-1-dmatlack@google.com> Message-Id: <20211213225918.672507-7-dmatlack@google.com> Mime-Version: 1.0 References: <20211213225918.672507-1-dmatlack@google.com> X-Mailer: git-send-email 2.34.1.173.g76aa8bc2d0-goog Subject: [PATCH v1 06/13] KVM: x86/mmu: Refactor tdp_mmu iterators to take kvm_mmu_page root From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Instead of passing a pointer to the root page table and the root level separately, pass in a pointer to the kvm_mmu_page that backs the root. This reduces the number of arguments by 1, cutting down on line lengths. No functional change intended. Signed-off-by: David Matlack Reviewed-by: Ben Gardon Reviewed-by: Peter Xu --- arch/x86/kvm/mmu/tdp_iter.c | 5 ++++- arch/x86/kvm/mmu/tdp_iter.h | 10 +++++----- arch/x86/kvm/mmu/tdp_mmu.c | 14 +++++--------- 3 files changed, 14 insertions(+), 15 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_iter.c b/arch/x86/kvm/mmu/tdp_iter.c index b3ed302c1a35..92b3a075525a 100644 --- a/arch/x86/kvm/mmu/tdp_iter.c +++ b/arch/x86/kvm/mmu/tdp_iter.c @@ -39,9 +39,12 @@ void tdp_iter_restart(struct tdp_iter *iter) * Sets a TDP iterator to walk a pre-order traversal of the paging structure * rooted at root_pt, starting with the walk to translate next_last_level_gfn. */ -void tdp_iter_start(struct tdp_iter *iter, u64 *root_pt, int root_level, +void tdp_iter_start(struct tdp_iter *iter, struct kvm_mmu_page *root, int min_level, gfn_t next_last_level_gfn) { + u64 *root_pt = root->spt; + int root_level = root->role.level; + WARN_ON(root_level < 1); WARN_ON(root_level > PT64_ROOT_MAX_LEVEL); diff --git a/arch/x86/kvm/mmu/tdp_iter.h b/arch/x86/kvm/mmu/tdp_iter.h index b1748b988d3a..ec1f58013428 100644 --- a/arch/x86/kvm/mmu/tdp_iter.h +++ b/arch/x86/kvm/mmu/tdp_iter.h @@ -51,17 +51,17 @@ struct tdp_iter { * Iterates over every SPTE mapping the GFN range [start, end) in a * preorder traversal. */ -#define for_each_tdp_pte_min_level(iter, root, root_level, min_level, start, end) \ - for (tdp_iter_start(&iter, root, root_level, min_level, start); \ +#define for_each_tdp_pte_min_level(iter, root, min_level, start, end) \ + for (tdp_iter_start(&iter, root, min_level, start); \ iter.valid && iter.gfn < end; \ tdp_iter_next(&iter)) -#define for_each_tdp_pte(iter, root, root_level, start, end) \ - for_each_tdp_pte_min_level(iter, root, root_level, PG_LEVEL_4K, start, end) +#define for_each_tdp_pte(iter, root, start, end) \ + for_each_tdp_pte_min_level(iter, root, PG_LEVEL_4K, start, end) tdp_ptep_t spte_to_child_pt(u64 pte, int level); -void tdp_iter_start(struct tdp_iter *iter, u64 *root_pt, int root_level, +void tdp_iter_start(struct tdp_iter *iter, struct kvm_mmu_page *root, int min_level, gfn_t next_last_level_gfn); void tdp_iter_next(struct tdp_iter *iter); void tdp_iter_restart(struct tdp_iter *iter); diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index dbd07c10d11a..2fb2d7677fbf 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -632,7 +632,7 @@ static inline void tdp_mmu_set_spte_no_dirty_log(struct kvm *kvm, } #define tdp_root_for_each_pte(_iter, _root, _start, _end) \ - for_each_tdp_pte(_iter, _root->spt, _root->role.level, _start, _end) + for_each_tdp_pte(_iter, _root, _start, _end) #define tdp_root_for_each_leaf_pte(_iter, _root, _start, _end) \ tdp_root_for_each_pte(_iter, _root, _start, _end) \ @@ -642,8 +642,7 @@ static inline void tdp_mmu_set_spte_no_dirty_log(struct kvm *kvm, else #define tdp_mmu_for_each_pte(_iter, _mmu, _start, _end) \ - for_each_tdp_pte(_iter, __va(_mmu->root_hpa), \ - _mmu->shadow_root_level, _start, _end) + for_each_tdp_pte(_iter, to_shadow_page(_mmu->root_hpa), _start, _end) /* * Yield if the MMU lock is contended or this thread needs to return control @@ -733,8 +732,7 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root, rcu_read_lock(); - for_each_tdp_pte_min_level(iter, root->spt, root->role.level, - min_level, start, end) { + for_each_tdp_pte_min_level(iter, root, min_level, start, end) { retry: if (can_yield && tdp_mmu_iter_cond_resched(kvm, &iter, flush, shared)) { @@ -1205,8 +1203,7 @@ static bool wrprot_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root, BUG_ON(min_level > KVM_MAX_HUGEPAGE_LEVEL); - for_each_tdp_pte_min_level(iter, root->spt, root->role.level, - min_level, start, end) { + for_each_tdp_pte_min_level(iter, root, min_level, start, end) { retry: if (tdp_mmu_iter_cond_resched(kvm, &iter, false, true)) continue; @@ -1445,8 +1442,7 @@ static bool write_protect_gfn(struct kvm *kvm, struct kvm_mmu_page *root, rcu_read_lock(); - for_each_tdp_pte_min_level(iter, root->spt, root->role.level, - min_level, gfn, gfn + 1) { + for_each_tdp_pte_min_level(iter, root, min_level, gfn, gfn + 1) { if (!is_shadow_present_pte(iter.old_spte) || !is_last_spte(iter.old_spte, iter.level)) continue; From patchwork Mon Dec 13 22:59:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12674809 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD6E1C433EF for ; Mon, 13 Dec 2021 22:59:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244105AbhLMW7g (ORCPT ); Mon, 13 Dec 2021 17:59:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47548 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244093AbhLMW7e (ORCPT ); Mon, 13 Dec 2021 17:59:34 -0500 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CB124C06173F for ; Mon, 13 Dec 2021 14:59:33 -0800 (PST) Received: by mail-pf1-x44a.google.com with SMTP id s22-20020a056a00179600b004b31f2cdb19so2181281pfg.7 for ; Mon, 13 Dec 2021 14:59:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=qbJieaZuu87kEMt04U4TNMVcgVWYL2WjrOpD1ljrWDc=; b=lEG+wIsncHNBbz6yrYgLYNCMI/UhW6vi7GQ6gSpVK4Q2P3Lwq4xs1/bZMF7j+wkvXx hXuTA08YuuuCtw/PuJKYf30Een5c+N6++PFcuG7jVrMppaCwjb0i0Fm8Sfpfcf6lxwoL hTzSHm+xe89MpzLv/+cq0iwxqt3vbXgngFEs0CW3Q35YD0QNOZs+QD+4Gzj4umrNXmRf DucFfTSp63r5LINLWuqpdvvefCf0D3rOhC1mWI652SFivBLBtMpS/TXmu8ug7+kt1geW vsfAOdmbZLONbDU8sRjzwu4VOUKaI7kESXMz6qenPjpBpTWoXnrzcwwBoKXz5weVQNgz j/yg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=qbJieaZuu87kEMt04U4TNMVcgVWYL2WjrOpD1ljrWDc=; b=we3cN+nGZ3/zwwbY/3WffGdmDTiW7hELiZbowfumGJMJkPTgrcjvGNPYdT7Qn7necs cLU6rCivylPk7XFpVB5jADqIQJM4fEQNEv3MCAQIYEiEH2VEvya2Ril8MKApw75JcQ+V nrlUbWpzv6qnqkahOQAKcN80y3QYwmT+R54aTYh4zDSvrD7vbWcQtHALb8PdcoVKhvAW 74cnejN2rxcHN+R00c74cFtDCGa0ITdY2sTABVLGW1qtrmuETJCIVS98E7S0rwAFdR1C fJu4u8yzPvLP+CLZu0t0q9oKtQ+TTbOK6/kPO6IAPecDDJlQb+YzY67RouoE6j4gogat fKPw== X-Gm-Message-State: AOAM530z2kMrtnBml5TcNDc70PHL/Y4/+UFXdhI4XjrTGnSq3UXB2OAf DaL4xO0Efz0om5DcuN826CK37kG2DW/HbA== X-Google-Smtp-Source: ABdhPJz43pp1XwQ5G6OTwEPPV5fRJP87AnvL1NLsovbaGTAhycIMv81QOzG1WYQk7qhqi+zV6uWVXsQ6ZJiLwA== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:90b:1a92:: with SMTP id ng18mr1076877pjb.19.1639436373299; Mon, 13 Dec 2021 14:59:33 -0800 (PST) Date: Mon, 13 Dec 2021 22:59:12 +0000 In-Reply-To: <20211213225918.672507-1-dmatlack@google.com> Message-Id: <20211213225918.672507-8-dmatlack@google.com> Mime-Version: 1.0 References: <20211213225918.672507-1-dmatlack@google.com> X-Mailer: git-send-email 2.34.1.173.g76aa8bc2d0-goog Subject: [PATCH v1 07/13] KVM: x86/mmu: Derive page role from parent From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Derive the page role from the parent shadow page, since the only thing that changes is the level. This is in preparation for eagerly splitting large pages during VM-ioctls which does not have access to the vCPU MMU context. No functional change intended. Signed-off-by: David Matlack Reviewed-by: Peter Xu --- arch/x86/kvm/mmu/tdp_mmu.c | 43 ++++++++++++++++++++------------------ 1 file changed, 23 insertions(+), 20 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 2fb2d7677fbf..582d9a798899 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -157,23 +157,8 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm, if (kvm_mmu_page_as_id(_root) != _as_id) { \ } else -static union kvm_mmu_page_role page_role_for_level(struct kvm_vcpu *vcpu, - int level) -{ - union kvm_mmu_page_role role; - - role = vcpu->arch.mmu->mmu_role.base; - role.level = level; - role.direct = true; - role.has_4_byte_gpte = false; - role.access = ACC_ALL; - role.ad_disabled = !shadow_accessed_mask; - - return role; -} - static struct kvm_mmu_page *alloc_tdp_mmu_page(struct kvm_vcpu *vcpu, gfn_t gfn, - int level) + union kvm_mmu_page_role role) { struct kvm_mmu_page *sp; @@ -181,7 +166,7 @@ static struct kvm_mmu_page *alloc_tdp_mmu_page(struct kvm_vcpu *vcpu, gfn_t gfn, sp->spt = kvm_mmu_memory_cache_alloc(&vcpu->arch.mmu_shadow_page_cache); set_page_private(virt_to_page(sp->spt), (unsigned long)sp); - sp->role.word = page_role_for_level(vcpu, level).word; + sp->role = role; sp->gfn = gfn; sp->tdp_mmu_page = true; @@ -190,6 +175,19 @@ static struct kvm_mmu_page *alloc_tdp_mmu_page(struct kvm_vcpu *vcpu, gfn_t gfn, return sp; } +static struct kvm_mmu_page *alloc_child_tdp_mmu_page(struct kvm_vcpu *vcpu, struct tdp_iter *iter) +{ + struct kvm_mmu_page *parent_sp; + union kvm_mmu_page_role role; + + parent_sp = sptep_to_sp(rcu_dereference(iter->sptep)); + + role = parent_sp->role; + role.level--; + + return alloc_tdp_mmu_page(vcpu, iter->gfn, role); +} + hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu) { union kvm_mmu_page_role role; @@ -198,7 +196,12 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu) lockdep_assert_held_write(&kvm->mmu_lock); - role = page_role_for_level(vcpu, vcpu->arch.mmu->shadow_root_level); + role = vcpu->arch.mmu->mmu_role.base; + role.level = vcpu->arch.mmu->shadow_root_level; + role.direct = true; + role.has_4_byte_gpte = false; + role.access = ACC_ALL; + role.ad_disabled = !shadow_accessed_mask; /* Check for an existing root before allocating a new one. */ for_each_tdp_mmu_root(kvm, root, kvm_mmu_role_as_id(role)) { @@ -207,7 +210,7 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu) goto out; } - root = alloc_tdp_mmu_page(vcpu, 0, vcpu->arch.mmu->shadow_root_level); + root = alloc_tdp_mmu_page(vcpu, 0, role); refcount_set(&root->tdp_mmu_root_count, 1); spin_lock(&kvm->arch.tdp_mmu_pages_lock); @@ -1033,7 +1036,7 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) if (is_removed_spte(iter.old_spte)) break; - sp = alloc_tdp_mmu_page(vcpu, iter.gfn, iter.level - 1); + sp = alloc_child_tdp_mmu_page(vcpu, &iter); if (!tdp_mmu_install_sp_atomic(vcpu->kvm, &iter, sp, account_nx)) { tdp_mmu_free_sp(sp); break; From patchwork Mon Dec 13 22:59:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12674811 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64629C433FE for ; Mon, 13 Dec 2021 22:59:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244101AbhLMW7g (ORCPT ); Mon, 13 Dec 2021 17:59:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47558 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244090AbhLMW7f (ORCPT ); Mon, 13 Dec 2021 17:59:35 -0500 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 27B87C0613F8 for ; Mon, 13 Dec 2021 14:59:35 -0800 (PST) Received: by mail-pg1-x549.google.com with SMTP id d2-20020a656202000000b00325603f7d0bso9698751pgv.12 for ; Mon, 13 Dec 2021 14:59:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=vq4mbu6fz8Ow9uMLnS6d9eLKMx/rjEnAVHjxQnrA+mQ=; b=QVTLQGpenttmwWwLZqpjBqB4eIk5+PzIhwr/0o9M/0ICD/6rRAo5JU07SrrCfFeSbv tnR5WHNWfbmewJDwtXgrA7jNwOzOGYn0Ls0SsNHaZy6d0hEDohe2DHrMiOzNOr7XrJeH BUXTaXt5B5ya8PN/b5adRlX1xUv25QmWNIIBt6PSsFA5PiiIqQ6uGydRq1ogo1CicDmF 1oPkvNrGDXwiJqYwE6DcAxZMQStXncjz0mSDJwCuRnhGhtUd2VNBOjDGnBDH1NqJaAb2 lWMfwjGcXccAGoPMuyZ80ETfOGkp5i/GbIJasldz5WblzE4fyPa6BN6ffL3Isa7jVPmn GAcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=vq4mbu6fz8Ow9uMLnS6d9eLKMx/rjEnAVHjxQnrA+mQ=; b=rQTa1PrcDUM3YNdVI7RIyBcAGyF/DDAJs1ehshTk7y/jpg0N/iymreJ9REtGDzCjMp bpor2CmoTRog9sFizaTL0M0w1/slv5oKa3ptVcT0tD0eXJ1BZwo8W2AtepWaqjOc+n00 wQzqYCxZDkug5Gzuhoqle7up0qVjGVSn76Gbb6IYLzt49kUaVCmCjOMUm2XBrlDPbvGn pGImf648zKeqidc8MZCUFOd6YTEGw148n4+jq1SMqHB6rDYL0yG3cmYCXOMmm3zWZGPf VGCFKwBmiAyGsIV4WW/fBQz3GWDxwV+H142HGmXrsLGXXlwJwi1wn9qcMtIQ1ErbdVFX lM/A== X-Gm-Message-State: AOAM533fLAGrZbdKhF6tXV7ETmcoVAwJ8nbgQfoz8OgcFndSvwRrQYdU BGhiBsdCtLZpk+GwkfTobbowfc3UtE3UDA== X-Google-Smtp-Source: ABdhPJwwNB2Z/LMLffQBlfS70bMPPswHXuWQk7NnsUDg9E95Wx8FFAyJTpwkjjyh9SS87Epffnb3Q+4yW6OuxA== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:902:e8d4:b0:143:88c2:e2c9 with SMTP id v20-20020a170902e8d400b0014388c2e2c9mr1235324plg.12.1639436374670; Mon, 13 Dec 2021 14:59:34 -0800 (PST) Date: Mon, 13 Dec 2021 22:59:13 +0000 In-Reply-To: <20211213225918.672507-1-dmatlack@google.com> Message-Id: <20211213225918.672507-9-dmatlack@google.com> Mime-Version: 1.0 References: <20211213225918.672507-1-dmatlack@google.com> X-Mailer: git-send-email 2.34.1.173.g76aa8bc2d0-goog Subject: [PATCH v1 08/13] KVM: x86/mmu: Refactor TDP MMU child page initialization From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Separate the allocation of child pages from the initialization. This is in preparation for doing page splitting outside of the vCPU fault context which requires a different allocation mechanism. No functional changed intended. Signed-off-by: David Matlack Reviewed-by: Peter Xu --- arch/x86/kvm/mmu/tdp_mmu.c | 30 +++++++++++++++++++++++------- 1 file changed, 23 insertions(+), 7 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 582d9a798899..a8354d8578f1 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -157,13 +157,18 @@ static struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm, if (kvm_mmu_page_as_id(_root) != _as_id) { \ } else -static struct kvm_mmu_page *alloc_tdp_mmu_page(struct kvm_vcpu *vcpu, gfn_t gfn, - union kvm_mmu_page_role role) +static struct kvm_mmu_page *alloc_tdp_mmu_page_from_caches(struct kvm_vcpu *vcpu) { struct kvm_mmu_page *sp; sp = kvm_mmu_memory_cache_alloc(&vcpu->arch.mmu_page_header_cache); sp->spt = kvm_mmu_memory_cache_alloc(&vcpu->arch.mmu_shadow_page_cache); + + return sp; +} + +static void init_tdp_mmu_page(struct kvm_mmu_page *sp, gfn_t gfn, union kvm_mmu_page_role role) +{ set_page_private(virt_to_page(sp->spt), (unsigned long)sp); sp->role = role; @@ -171,11 +176,9 @@ static struct kvm_mmu_page *alloc_tdp_mmu_page(struct kvm_vcpu *vcpu, gfn_t gfn, sp->tdp_mmu_page = true; trace_kvm_mmu_get_page(sp, true); - - return sp; } -static struct kvm_mmu_page *alloc_child_tdp_mmu_page(struct kvm_vcpu *vcpu, struct tdp_iter *iter) +static void init_child_tdp_mmu_page(struct kvm_mmu_page *child_sp, struct tdp_iter *iter) { struct kvm_mmu_page *parent_sp; union kvm_mmu_page_role role; @@ -185,7 +188,17 @@ static struct kvm_mmu_page *alloc_child_tdp_mmu_page(struct kvm_vcpu *vcpu, stru role = parent_sp->role; role.level--; - return alloc_tdp_mmu_page(vcpu, iter->gfn, role); + init_tdp_mmu_page(child_sp, iter->gfn, role); +} + +static struct kvm_mmu_page *alloc_child_tdp_mmu_page(struct kvm_vcpu *vcpu, struct tdp_iter *iter) +{ + struct kvm_mmu_page *child_sp; + + child_sp = alloc_tdp_mmu_page_from_caches(vcpu); + init_child_tdp_mmu_page(child_sp, iter); + + return child_sp; } hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu) @@ -210,7 +223,10 @@ hpa_t kvm_tdp_mmu_get_vcpu_root_hpa(struct kvm_vcpu *vcpu) goto out; } - root = alloc_tdp_mmu_page(vcpu, 0, role); + root = alloc_tdp_mmu_page_from_caches(vcpu); + + init_tdp_mmu_page(root, 0, role); + refcount_set(&root->tdp_mmu_root_count, 1); spin_lock(&kvm->arch.tdp_mmu_pages_lock); From patchwork Mon Dec 13 22:59:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12674813 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DCCFC433F5 for ; Mon, 13 Dec 2021 22:59:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244098AbhLMW7h (ORCPT ); Mon, 13 Dec 2021 17:59:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47574 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244095AbhLMW7g (ORCPT ); Mon, 13 Dec 2021 17:59:36 -0500 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AE0F6C061574 for ; Mon, 13 Dec 2021 14:59:36 -0800 (PST) Received: by mail-pf1-x44a.google.com with SMTP id 184-20020a6217c1000000b0049f9aad0040so10863884pfx.21 for ; Mon, 13 Dec 2021 14:59:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=wktwpqxo/6qTxBsMQfGKkeq9jR6p0bwTKGhJS/M9Pow=; b=FCfu0ELRNKVG3A4yCaH8M00UGHvxJyk0t2bJpUB9rMeUvU2Bzy68poW4iTzlt6aUlr AySR7lV6OQGAzKMca/O+BtxB9/evD2NZUXTNRxGlmlo71DH8GI8ZjQSTUcS14QiJLYDe kCBLTFeV6+KNYzxS6PuNTFHQEZzUSgq/a5JqRnbB57hXaAsbs5aSPKOt/1Ay0hIiuimf vZ98sqzLa5pqQw6X/Ly/n+okDkBsQNsK2mAaCBQRLcaGkquziC229KSgiaTbaxHDBm6V rFE5WuVcx1jgcGMWLUfi7gDNS0Ewtn3cEG1VqQSWzan7CVAmVe+Ze9Z0LgsLuKPFYTC0 9Lig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=wktwpqxo/6qTxBsMQfGKkeq9jR6p0bwTKGhJS/M9Pow=; b=ZYMCSLTJ6R2PeO4cJypeyKsssuPx8X2AW8w/1Xxlyg7qhv9U5/dsnU6rIgWlhxIXwr IuGX3RlAWUK+mlF2XgqhpHVdaQSBFQgMk0RoU0NoA2B445QUqeUgySyj3zO6bYvlxQgB y1dO7HnNml2uNrOl9pGWdp2ZFqihyTdjH3PJC/rxyDDLptMo299U91ANaeE5G99N0XXf XDqR1JBiFXrJtaf91wqQt+yrSJxE268vxN/cAh1yhCSQf5Vl7aoGwTwtvd9ifLg4nnJG wUPoN6RbZf5adC9AsgMReVqZAq5ARvu4ddHLaY3KtucVbUjfzwmfwSFoCr1WtQst7sa5 xM7A== X-Gm-Message-State: AOAM533k6tEOxPvcXZxZwigW0BPm6SrjY2Iz9pEjZ7X7yzwEMqyOk5nZ i0AG40Y5DLIhmRN6d4UwjNlqPWCz4t4Y3g== X-Google-Smtp-Source: ABdhPJxyo/GUOS8gPiMJcLphvRe+qQEVZat0bdstHzETXXXjZIHgnVF+voGlbI4s+ylfNhgbzMuF9SkS524aHg== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:902:8f93:b0:142:8731:1a5d with SMTP id z19-20020a1709028f9300b0014287311a5dmr1218885plo.60.1639436376196; Mon, 13 Dec 2021 14:59:36 -0800 (PST) Date: Mon, 13 Dec 2021 22:59:14 +0000 In-Reply-To: <20211213225918.672507-1-dmatlack@google.com> Message-Id: <20211213225918.672507-10-dmatlack@google.com> Mime-Version: 1.0 References: <20211213225918.672507-1-dmatlack@google.com> X-Mailer: git-send-email 2.34.1.173.g76aa8bc2d0-goog Subject: [PATCH v1 09/13] KVM: x86/mmu: Split huge pages when dirty logging is enabled From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org When dirty logging is enabled without initially-all-set, attempt to split all huge pages in the memslot down to 4KB pages so that vCPUs do not have to take expensive write-protection faults to split huge pages. Huge page splitting is best-effort only. This commit only adds the support for the TDP MMU, and even there splitting may fail due to out of memory conditions. Failures to split a huge page is fine from a correctness standpoint because we still always follow it up by write- protecting any remaining huge pages. Signed-off-by: David Matlack Reviewed-by: Peter Xu --- arch/x86/include/asm/kvm_host.h | 3 + arch/x86/kvm/mmu/mmu.c | 14 +++ arch/x86/kvm/mmu/spte.c | 59 ++++++++++++ arch/x86/kvm/mmu/spte.h | 1 + arch/x86/kvm/mmu/tdp_mmu.c | 165 ++++++++++++++++++++++++++++++++ arch/x86/kvm/mmu/tdp_mmu.h | 5 + arch/x86/kvm/x86.c | 10 ++ 7 files changed, 257 insertions(+) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index e863d569c89a..4a507109e886 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1573,6 +1573,9 @@ void kvm_mmu_reset_context(struct kvm_vcpu *vcpu); void kvm_mmu_slot_remove_write_access(struct kvm *kvm, const struct kvm_memory_slot *memslot, int start_level); +void kvm_mmu_slot_try_split_huge_pages(struct kvm *kvm, + const struct kvm_memory_slot *memslot, + int target_level); void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm, const struct kvm_memory_slot *memslot); void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm, diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 3c2cb4dd1f11..9116c6a4ced1 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5807,6 +5807,20 @@ void kvm_mmu_slot_remove_write_access(struct kvm *kvm, kvm_arch_flush_remote_tlbs_memslot(kvm, memslot); } +void kvm_mmu_slot_try_split_huge_pages(struct kvm *kvm, + const struct kvm_memory_slot *memslot, + int target_level) +{ + u64 start = memslot->base_gfn; + u64 end = start + memslot->npages; + + if (is_tdp_mmu_enabled(kvm)) { + read_lock(&kvm->mmu_lock); + kvm_tdp_mmu_try_split_huge_pages(kvm, memslot, start, end, target_level); + read_unlock(&kvm->mmu_lock); + } +} + static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm, struct kvm_rmap_head *rmap_head, const struct kvm_memory_slot *slot) diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index fd34ae5d6940..11d0b3993ba5 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -191,6 +191,65 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, return wrprot; } +static u64 mark_spte_executable(u64 spte) +{ + bool is_access_track = is_access_track_spte(spte); + + if (is_access_track) + spte = restore_acc_track_spte(spte); + + spte &= ~shadow_nx_mask; + spte |= shadow_x_mask; + + if (is_access_track) + spte = mark_spte_for_access_track(spte); + + return spte; +} + +/* + * Construct an SPTE that maps a sub-page of the given huge page SPTE where + * `index` identifies which sub-page. + * + * This is used during huge page splitting to build the SPTEs that make up the + * new page table. + */ +u64 make_huge_page_split_spte(u64 huge_spte, int huge_level, int index, unsigned int access) +{ + u64 child_spte; + int child_level; + + if (WARN_ON(is_mmio_spte(huge_spte))) + return 0; + + if (WARN_ON(!is_shadow_present_pte(huge_spte))) + return 0; + + if (WARN_ON(!is_large_pte(huge_spte))) + return 0; + + child_spte = huge_spte; + child_level = huge_level - 1; + + /* + * The child_spte already has the base address of the huge page being + * split. So we just have to OR in the offset to the page at the next + * lower level for the given index. + */ + child_spte |= (index * KVM_PAGES_PER_HPAGE(child_level)) << PAGE_SHIFT; + + if (child_level == PG_LEVEL_4K) { + child_spte &= ~PT_PAGE_SIZE_MASK; + + /* Allow execution for 4K pages if it was disabled for NX HugePages. */ + if (is_nx_huge_page_enabled() && access & ACC_EXEC_MASK) + child_spte = mark_spte_executable(child_spte); + } + + return child_spte; +} + + u64 make_nonleaf_spte(u64 *child_pt, bool ad_disabled) { u64 spte = SPTE_MMU_PRESENT_MASK; diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index 9b0c7b27f23f..e13f335b4fef 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -334,6 +334,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, unsigned int pte_access, gfn_t gfn, kvm_pfn_t pfn, u64 old_spte, bool prefetch, bool can_unsync, bool host_writable, u64 *new_spte); +u64 make_huge_page_split_spte(u64 huge_spte, int huge_level, int index, unsigned int access); u64 make_nonleaf_spte(u64 *child_pt, bool ad_disabled); u64 make_mmio_spte(struct kvm_vcpu *vcpu, u64 gfn, unsigned int access); u64 mark_spte_for_access_track(u64 spte); diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index a8354d8578f1..be5eb74ac053 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1264,6 +1264,171 @@ bool kvm_tdp_mmu_wrprot_slot(struct kvm *kvm, return spte_set; } +static struct kvm_mmu_page *alloc_tdp_mmu_page_from_kernel(gfp_t gfp) +{ + struct kvm_mmu_page *sp; + + gfp |= __GFP_ZERO; + + sp = kmem_cache_alloc(mmu_page_header_cache, gfp); + if (!sp) + return NULL; + + sp->spt = (void *)__get_free_page(gfp); + if (!sp->spt) { + kmem_cache_free(mmu_page_header_cache, sp); + return NULL; + } + + return sp; +} + +static struct kvm_mmu_page *alloc_tdp_mmu_page_for_split(struct kvm *kvm, bool *dropped_lock) +{ + struct kvm_mmu_page *sp; + + lockdep_assert_held_read(&kvm->mmu_lock); + + *dropped_lock = false; + + /* + * Since we are allocating while under the MMU lock we have to be + * careful about GFP flags. Use GFP_NOWAIT to avoid blocking on direct + * reclaim and to avoid making any filesystem callbacks (which can end + * up invoking KVM MMU notifiers, resulting in a deadlock). + * + * If this allocation fails we drop the lock and retry with reclaim + * allowed. + */ + sp = alloc_tdp_mmu_page_from_kernel(GFP_NOWAIT | __GFP_ACCOUNT); + if (sp) + return sp; + + rcu_read_unlock(); + read_unlock(&kvm->mmu_lock); + + *dropped_lock = true; + + sp = alloc_tdp_mmu_page_from_kernel(GFP_KERNEL_ACCOUNT); + + read_lock(&kvm->mmu_lock); + rcu_read_lock(); + + return sp; +} + +static bool +tdp_mmu_split_huge_page_atomic(struct kvm *kvm, struct tdp_iter *iter, struct kvm_mmu_page *sp) +{ + const u64 huge_spte = iter->old_spte; + const int level = iter->level; + u64 child_spte; + int i; + + init_child_tdp_mmu_page(sp, iter); + + for (i = 0; i < PT64_ENT_PER_PAGE; i++) { + child_spte = make_huge_page_split_spte(huge_spte, level, i, ACC_ALL); + + /* + * No need for atomics since child_sp has not been installed + * in the table yet and thus is not reachable by any other + * thread. + */ + sp->spt[i] = child_spte; + } + + if (!tdp_mmu_install_sp_atomic(kvm, iter, sp, false)) + return false; + + /* + * tdp_mmu_install_sp_atomic will handle subtracting the split huge + * page from stats, but we have to manually update the new present child + * pages. + */ + kvm_update_page_stats(kvm, level - 1, PT64_ENT_PER_PAGE); + + return true; +} + +static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, struct kvm_mmu_page *root, + gfn_t start, gfn_t end, int target_level) +{ + struct kvm_mmu_page *sp = NULL; + struct tdp_iter iter; + bool dropped_lock; + + rcu_read_lock(); + + /* + * Traverse the page table splitting all huge pages above the target + * level into one lower level. For example, if we encounter a 1GB page + * we split it into 512 2MB pages. + * + * Since the TDP iterator uses a pre-order traversal, we are guaranteed + * to visit an SPTE before ever visiting its children, which means we + * will correctly recursively split huge pages that are more than one + * level above the target level (e.g. splitting 1GB to 2MB to 4KB). + */ + for_each_tdp_pte_min_level(iter, root, target_level + 1, start, end) { +retry: + if (tdp_mmu_iter_cond_resched(kvm, &iter, false, true)) + continue; + + if (!is_shadow_present_pte(iter.old_spte) || !is_large_pte(iter.old_spte)) + continue; + + if (!sp) { + sp = alloc_tdp_mmu_page_for_split(kvm, &dropped_lock); + if (!sp) + return -ENOMEM; + + if (dropped_lock) { + tdp_iter_restart(&iter); + continue; + } + } + + if (!tdp_mmu_split_huge_page_atomic(kvm, &iter, sp)) + goto retry; + + sp = NULL; + } + + /* + * It's possible to exit the loop having never used the last sp if, for + * example, a vCPU doing HugePage NX splitting wins the race and + * installs its own sp in place of the last sp we tried to split. + */ + if (sp) + tdp_mmu_free_sp(sp); + + rcu_read_unlock(); + + return 0; +} + +int kvm_tdp_mmu_try_split_huge_pages(struct kvm *kvm, + const struct kvm_memory_slot *slot, + gfn_t start, gfn_t end, + int target_level) +{ + struct kvm_mmu_page *root; + int r = 0; + + lockdep_assert_held_read(&kvm->mmu_lock); + + for_each_tdp_mmu_root_yield_safe(kvm, root, slot->as_id, true) { + r = tdp_mmu_split_huge_pages_root(kvm, root, start, end, target_level); + if (r) { + kvm_tdp_mmu_put_root(kvm, root, true); + break; + } + } + + return r; +} + /* * Clear the dirty status of all the SPTEs mapping GFNs in the memslot. If * AD bits are enabled, this will involve clearing the dirty bit on each SPTE. diff --git a/arch/x86/kvm/mmu/tdp_mmu.h b/arch/x86/kvm/mmu/tdp_mmu.h index 3899004a5d91..3557a7fcf927 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.h +++ b/arch/x86/kvm/mmu/tdp_mmu.h @@ -71,6 +71,11 @@ bool kvm_tdp_mmu_write_protect_gfn(struct kvm *kvm, struct kvm_memory_slot *slot, gfn_t gfn, int min_level); +int kvm_tdp_mmu_try_split_huge_pages(struct kvm *kvm, + const struct kvm_memory_slot *slot, + gfn_t start, gfn_t end, + int target_level); + static inline void kvm_tdp_mmu_walk_lockless_begin(void) { rcu_read_lock(); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 85127b3e3690..fb5592bf2eee 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -187,6 +187,9 @@ module_param(force_emulation_prefix, bool, S_IRUGO); int __read_mostly pi_inject_timer = -1; module_param(pi_inject_timer, bint, S_IRUGO | S_IWUSR); +static bool __read_mostly eagerly_split_huge_pages_for_dirty_logging = true; +module_param(eagerly_split_huge_pages_for_dirty_logging, bool, 0644); + /* * Restoring the host value for MSRs that are only consumed when running in * usermode, e.g. SYSCALL MSRs and TSC_AUX, can be deferred until the CPU @@ -11837,6 +11840,13 @@ static void kvm_mmu_slot_apply_flags(struct kvm *kvm, if (kvm_dirty_log_manual_protect_and_init_set(kvm)) return; + /* + * Attempt to split all large pages into 4K pages so that vCPUs + * do not have to take write-protection faults. + */ + if (READ_ONCE(eagerly_split_huge_pages_for_dirty_logging)) + kvm_mmu_slot_try_split_huge_pages(kvm, new, PG_LEVEL_4K); + if (kvm_x86_ops.cpu_dirty_log_size) { kvm_mmu_slot_leaf_clear_dirty(kvm, new); kvm_mmu_slot_remove_write_access(kvm, new, PG_LEVEL_2M); From patchwork Mon Dec 13 22:59:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12674815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBEAAC433EF for ; Mon, 13 Dec 2021 22:59:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244111AbhLMW7j (ORCPT ); Mon, 13 Dec 2021 17:59:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47580 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244071AbhLMW7i (ORCPT ); Mon, 13 Dec 2021 17:59:38 -0500 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 215C3C06173F for ; Mon, 13 Dec 2021 14:59:38 -0800 (PST) Received: by mail-pf1-x44a.google.com with SMTP id e68-20020a621e47000000b004b13a82634eso6731295pfe.23 for ; Mon, 13 Dec 2021 14:59:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=r9yRjg9BMyYVZKVb9TRMFJxQhrx7asVzPH3IeS46ZBM=; b=h0DWCZaUCcedLKkBjEE3uJkekKZ1VIBGo5hWgx2wS5U0SkUQAdU2XZbuVMwD60nVix +CJkVkjVVd8lOwsGR1dLt4V0ZOhgiU7hmEeiGEIJnh3xmL6qL+pUNcqiVI172+gUDZsb Qgsvtunq9mdmojh4nP2wgHqDa/hhGh4/tEKftlQTZLSmoIzKOLmQeUbbY0tRgAKsTeXs pQrx+NPrwUfx5mhUblat++H9xCCfyn2BVZNnQjl97KUcANOYGJmhLDtpRbqXiW97hXTt sQ8dkJ54otTI84i+HEw2abRmIn4Jou7b8xoOlPKOpOIrYUDMWAGPSWzekC4521uz4+kK zfxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=r9yRjg9BMyYVZKVb9TRMFJxQhrx7asVzPH3IeS46ZBM=; b=x4Q2mrNN48+2FFqi9Q9SlA0c2eytiv74GwaUaZNPIARx5tZ5fGdY9zYHX2FnAp94MR JNgdIZD8qwm4j2gdr0LkEnGROmL5qYWJCm/h7qpD3IDRPugVgtnVYKMM4cPmFtTQmZ6a npHHTsTcsPJ3MG/vM5jVn1nRjxTz7N3+9urHXz/9u5YzYMLUQ4aaRkdB8U41+2OEMXzf aE7ctT/z4QM9zuIsCeGXSrWbUC+XQpHCqbM77DyTXF1+RVaqPxjXB18dlUQybJ8GOB82 uMjVQNXHk3R/NvG0cNdreQyRt7L4gGI1IVTnoy9OE6zt3HtSUj3QvfBOVDtJfqjoJ2ji KWBQ== X-Gm-Message-State: AOAM530FCct+h3dShZaZx3KvGZP7oV6ueBQOZfiu9Y2ZfqFRYIem2E7D UcDgzMQyONRKoVIH44IdSqRlry/hr2ixwQ== X-Google-Smtp-Source: ABdhPJxJZ5C4VR0Xm7ItN0P7uAYzN3Kx6zVFXx9ZigyjYbL6g8i3FP4UIRct3a8OvcwFl2DnRVJ5K9EoW60E7A== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:903:22c6:b0:141:fac1:b722 with SMTP id y6-20020a17090322c600b00141fac1b722mr1887176plg.23.1639436377559; Mon, 13 Dec 2021 14:59:37 -0800 (PST) Date: Mon, 13 Dec 2021 22:59:15 +0000 In-Reply-To: <20211213225918.672507-1-dmatlack@google.com> Message-Id: <20211213225918.672507-11-dmatlack@google.com> Mime-Version: 1.0 References: <20211213225918.672507-1-dmatlack@google.com> X-Mailer: git-send-email 2.34.1.173.g76aa8bc2d0-goog Subject: [PATCH v1 10/13] KVM: Push MMU locking down into kvm_arch_mmu_enable_log_dirty_pt_masked From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Instead of acquiring the MMU lock in the arch-generic code, force each implementation of kvm_arch_mmu_enable_log_dirty_pt_masked to acquire the MMU lock as needed. This is in preparation for performing eager page splitting in the x86 implementation of kvm_arch_mmu_enable_log_dirty_pt_masked, which involves dropping the MMU lock in write-mode and re-acquiring it in read mode (and possibly rescheduling) during splitting. Pushing the MMU lock down into the arch code makes the x86 synchronization much easier to reason about, and does not harm readability of other architectures. This should be a safe change because: * No architecture requires a TLB flush before dropping the MMU lock. * The dirty bitmap does not need to be synchronized with changes to the page tables by the MMU lock as evidenced by the fact that x86 modifies the dirty bitmap without acquiring the MMU lock in fast_page_fault. This change does increase the number of times the MMU lock is acquired and released during KVM_CLEAR_DIRTY_LOG, but this is not a performance critical path and breaking up the lock duration may reduce contention on vCPU threads. Signed-off-by: David Matlack --- arch/arm64/kvm/mmu.c | 2 ++ arch/mips/kvm/mmu.c | 5 +++-- arch/riscv/kvm/mmu.c | 2 ++ arch/x86/kvm/mmu/mmu.c | 4 ++++ virt/kvm/dirty_ring.c | 2 -- virt/kvm/kvm_main.c | 4 ---- 6 files changed, 11 insertions(+), 8 deletions(-) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index e65acf35cee3..48085cb534d5 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -749,7 +749,9 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, struct kvm_memory_slot *slot, gfn_t gfn_offset, unsigned long mask) { + spin_lock(&kvm->mmu_lock); kvm_mmu_write_protect_pt_masked(kvm, slot, gfn_offset, mask); + spin_unlock(&kvm->mmu_lock); } static void kvm_send_hwpoison_signal(unsigned long address, short lsb) diff --git a/arch/mips/kvm/mmu.c b/arch/mips/kvm/mmu.c index 1bfd1b501d82..7e67edcd5aae 100644 --- a/arch/mips/kvm/mmu.c +++ b/arch/mips/kvm/mmu.c @@ -409,8 +409,7 @@ int kvm_mips_mkclean_gpa_pt(struct kvm *kvm, gfn_t start_gfn, gfn_t end_gfn) * @mask: The mask of dirty pages at offset 'gfn_offset' in this memory * slot to be write protected * - * Walks bits set in mask write protects the associated pte's. Caller must - * acquire @kvm->mmu_lock. + * Walks bits set in mask write protects the associated pte's. */ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, struct kvm_memory_slot *slot, @@ -420,7 +419,9 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, gfn_t start = base_gfn + __ffs(mask); gfn_t end = base_gfn + __fls(mask); + spin_lock(&kvm->mmu_lock); kvm_mips_mkclean_gpa_pt(kvm, start, end); + spin_unlock(&kvm->mmu_lock); } /* diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c index 7d884b15cf5e..d084ac939b0f 100644 --- a/arch/riscv/kvm/mmu.c +++ b/arch/riscv/kvm/mmu.c @@ -424,7 +424,9 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, phys_addr_t start = (base_gfn + __ffs(mask)) << PAGE_SHIFT; phys_addr_t end = (base_gfn + __fls(mask) + 1) << PAGE_SHIFT; + spin_lock(&kvm->mmu_lock); stage2_wp_range(kvm, start, end); + spin_unlock(&kvm->mmu_lock); } void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 9116c6a4ced1..c9e5fe290714 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1347,6 +1347,8 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, struct kvm_memory_slot *slot, gfn_t gfn_offset, unsigned long mask) { + write_lock(&kvm->mmu_lock); + /* * Huge pages are NOT write protected when we start dirty logging in * initially-all-set mode; must write protect them here so that they @@ -1374,6 +1376,8 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, kvm_mmu_clear_dirty_pt_masked(kvm, slot, gfn_offset, mask); else kvm_mmu_write_protect_pt_masked(kvm, slot, gfn_offset, mask); + + write_unlock(&kvm->mmu_lock); } int kvm_cpu_dirty_log_size(void) diff --git a/virt/kvm/dirty_ring.c b/virt/kvm/dirty_ring.c index 88f4683198ea..6b26ec60c96a 100644 --- a/virt/kvm/dirty_ring.c +++ b/virt/kvm/dirty_ring.c @@ -61,9 +61,7 @@ static void kvm_reset_dirty_gfn(struct kvm *kvm, u32 slot, u64 offset, u64 mask) if (!memslot || (offset + __fls(mask)) >= memslot->npages) return; - KVM_MMU_LOCK(kvm); kvm_arch_mmu_enable_log_dirty_pt_masked(kvm, memslot, offset, mask); - KVM_MMU_UNLOCK(kvm); } int kvm_dirty_ring_alloc(struct kvm_dirty_ring *ring, int index, u32 size) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 3595eddd476a..da4850fb2982 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -2048,7 +2048,6 @@ static int kvm_get_dirty_log_protect(struct kvm *kvm, struct kvm_dirty_log *log) dirty_bitmap_buffer = kvm_second_dirty_bitmap(memslot); memset(dirty_bitmap_buffer, 0, n); - KVM_MMU_LOCK(kvm); for (i = 0; i < n / sizeof(long); i++) { unsigned long mask; gfn_t offset; @@ -2064,7 +2063,6 @@ static int kvm_get_dirty_log_protect(struct kvm *kvm, struct kvm_dirty_log *log) kvm_arch_mmu_enable_log_dirty_pt_masked(kvm, memslot, offset, mask); } - KVM_MMU_UNLOCK(kvm); } if (flush) @@ -2159,7 +2157,6 @@ static int kvm_clear_dirty_log_protect(struct kvm *kvm, if (copy_from_user(dirty_bitmap_buffer, log->dirty_bitmap, n)) return -EFAULT; - KVM_MMU_LOCK(kvm); for (offset = log->first_page, i = offset / BITS_PER_LONG, n = DIV_ROUND_UP(log->num_pages, BITS_PER_LONG); n--; i++, offset += BITS_PER_LONG) { @@ -2182,7 +2179,6 @@ static int kvm_clear_dirty_log_protect(struct kvm *kvm, offset, mask); } } - KVM_MMU_UNLOCK(kvm); if (flush) kvm_arch_flush_remote_tlbs_memslot(kvm, memslot); From patchwork Mon Dec 13 22:59:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12674817 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7009FC433EF for ; Mon, 13 Dec 2021 22:59:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244114AbhLMW7k (ORCPT ); Mon, 13 Dec 2021 17:59:40 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47598 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244116AbhLMW7j (ORCPT ); Mon, 13 Dec 2021 17:59:39 -0500 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BFA82C061756 for ; Mon, 13 Dec 2021 14:59:39 -0800 (PST) Received: by mail-pg1-x549.google.com with SMTP id a8-20020a63cd48000000b00330605939c0so9724322pgj.5 for ; Mon, 13 Dec 2021 14:59:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=G1lTebD+2U0O4p131ZNw2bZ43siPK30QECOBusO54QM=; b=r1IVQAoNwPe/u5vtL4RoKMuhMWHrGR3OYMY5tjiyd77lkplG8sPbDT7Ar9J4zeJBLJ +0fM970c3aPeKunDiTxo6eL2Fa4O8mM5i33jjOTmgmxvhXlyLgX9BbMOU5jG6KSzlfQo 3r41zJFyiKZ5xo8vzOAVWMnaZJJEMIprubVlI/z17VNrx3bijvouALcEc6vWjvDspH4n MY1BGL6QS4vL9xf8RUzgkn2NfrFy4yTj2p/K3awA2oLT43fDOXzu9eehLWVU9XB+Zyid 3cKo6X0gw+AipwwlHwf0mK8TLuLYEvap6nht+akt4qtOakE4wisWyNKBPTByFWvfaLnS Uruw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=G1lTebD+2U0O4p131ZNw2bZ43siPK30QECOBusO54QM=; b=aGsA4wWHrEJau+jY5ZVlHgrFzfHN+pZ1LZqQZmauvlfZt4v5AFWa0sRtyNX9Wuse+6 AwHeROE0BF8OPEnxJELbrvmKXIvMIc8j0tMoGvwdthIvEx4rE2Th3bXXgmViKabCVAXW 1Bx7QfXVJIXPp4GCewlzpC88amEWWFOuoT8jdSVm1gDQKUMStVqTdaUzAJCAvyxQ7ZG2 +wKjJeD1No0c97ILGUnNItnGfHhiupQyku4FkjHH8hrXCgMtNDNF1O8kC9Oud6mk/Vrt YWucTH9RIw0NOW7iCcwpep9gSzzFhjDv4Khrm9OY0+r3BJEHc8KZ22kmwibTfdgbEg4m rsiA== X-Gm-Message-State: AOAM533sWlgrwWG70nh1QGxj/dBP135F4fiuCEFVnAxRv4cMxwvvDvP1 Qp3U0DpAu69FBj1Qno+ae/DdVDYyvCsUTA== X-Google-Smtp-Source: ABdhPJxyFYy/gzEsRWxnsbvnhYk8O728htIICky2J8vlpbm6pk0/fLOYO6yzngEf5Q7uldshTAXkoCpYNwWiUg== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a05:6a00:26e3:b0:49f:c0ca:850e with SMTP id p35-20020a056a0026e300b0049fc0ca850emr1068213pfw.4.1639436379236; Mon, 13 Dec 2021 14:59:39 -0800 (PST) Date: Mon, 13 Dec 2021 22:59:16 +0000 In-Reply-To: <20211213225918.672507-1-dmatlack@google.com> Message-Id: <20211213225918.672507-12-dmatlack@google.com> Mime-Version: 1.0 References: <20211213225918.672507-1-dmatlack@google.com> X-Mailer: git-send-email 2.34.1.173.g76aa8bc2d0-goog Subject: [PATCH v1 11/13] KVM: x86/mmu: Split huge pages during CLEAR_DIRTY_LOG From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org When using initially-all-set, huge pages are not write-protected when dirty logging is enabled on the memslot. Instead they are write-protected once userspace invokes CLEAR_DIRTY_LOG for the first time and only for the specific sub-region being cleared. Enhance CLEAR_DIRTY_LOG to also try to split huge pages prior to write-protecting to avoid causing write-protection faults on vCPU threads. This also allows userspace to smear the cost of huge page splitting across multiple ioctls rather than splitting the entire memslot when not using initially-all-set. Signed-off-by: David Matlack --- arch/x86/include/asm/kvm_host.h | 4 ++++ arch/x86/kvm/mmu/mmu.c | 36 +++++++++++++++++++++++++++------ arch/x86/kvm/x86.c | 2 +- arch/x86/kvm/x86.h | 2 ++ 4 files changed, 37 insertions(+), 7 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 4a507109e886..3e537e261562 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1576,6 +1576,10 @@ void kvm_mmu_slot_remove_write_access(struct kvm *kvm, void kvm_mmu_slot_try_split_huge_pages(struct kvm *kvm, const struct kvm_memory_slot *memslot, int target_level); +void kvm_mmu_try_split_huge_pages(struct kvm *kvm, + const struct kvm_memory_slot *memslot, + u64 start, u64 end, + int target_level); void kvm_mmu_zap_collapsible_sptes(struct kvm *kvm, const struct kvm_memory_slot *memslot); void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm, diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index c9e5fe290714..55640d73df5a 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1362,6 +1362,20 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, gfn_t start = slot->base_gfn + gfn_offset + __ffs(mask); gfn_t end = slot->base_gfn + gfn_offset + __fls(mask); + /* + * Try to proactively split any huge pages down to 4KB so that + * vCPUs don't have to take write-protection faults. + * + * Drop the MMU lock since huge page splitting uses its own + * locking scheme and does not require the write lock in all + * cases. + */ + if (READ_ONCE(eagerly_split_huge_pages_for_dirty_logging)) { + write_unlock(&kvm->mmu_lock); + kvm_mmu_try_split_huge_pages(kvm, slot, start, end, PG_LEVEL_4K); + write_lock(&kvm->mmu_lock); + } + kvm_mmu_slot_gfn_write_protect(kvm, slot, start, PG_LEVEL_2M); /* Cross two large pages? */ @@ -5811,13 +5825,11 @@ void kvm_mmu_slot_remove_write_access(struct kvm *kvm, kvm_arch_flush_remote_tlbs_memslot(kvm, memslot); } -void kvm_mmu_slot_try_split_huge_pages(struct kvm *kvm, - const struct kvm_memory_slot *memslot, - int target_level) +void kvm_mmu_try_split_huge_pages(struct kvm *kvm, + const struct kvm_memory_slot *memslot, + u64 start, u64 end, + int target_level) { - u64 start = memslot->base_gfn; - u64 end = start + memslot->npages; - if (is_tdp_mmu_enabled(kvm)) { read_lock(&kvm->mmu_lock); kvm_tdp_mmu_try_split_huge_pages(kvm, memslot, start, end, target_level); @@ -5825,6 +5837,18 @@ void kvm_mmu_slot_try_split_huge_pages(struct kvm *kvm, } } +void kvm_mmu_slot_try_split_huge_pages(struct kvm *kvm, + const struct kvm_memory_slot *memslot, + int target_level) +{ + u64 start, end; + + start = memslot->base_gfn; + end = start + memslot->npages; + + kvm_mmu_try_split_huge_pages(kvm, memslot, start, end, target_level); +} + static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm, struct kvm_rmap_head *rmap_head, const struct kvm_memory_slot *slot) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index fb5592bf2eee..e27a3d6e3978 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -187,7 +187,7 @@ module_param(force_emulation_prefix, bool, S_IRUGO); int __read_mostly pi_inject_timer = -1; module_param(pi_inject_timer, bint, S_IRUGO | S_IWUSR); -static bool __read_mostly eagerly_split_huge_pages_for_dirty_logging = true; +bool __read_mostly eagerly_split_huge_pages_for_dirty_logging = true; module_param(eagerly_split_huge_pages_for_dirty_logging, bool, 0644); /* diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 4abcd8d9836d..825e47451875 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -352,6 +352,8 @@ extern int pi_inject_timer; extern bool report_ignored_msrs; +extern bool eagerly_split_huge_pages_for_dirty_logging; + static inline u64 nsec_to_cycles(struct kvm_vcpu *vcpu, u64 nsec) { return pvclock_scale_delta(nsec, vcpu->arch.virtual_tsc_mult, From patchwork Mon Dec 13 22:59:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12674819 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F0E0C433EF for ; Mon, 13 Dec 2021 22:59:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244071AbhLMW7o (ORCPT ); Mon, 13 Dec 2021 17:59:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47610 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244112AbhLMW7l (ORCPT ); Mon, 13 Dec 2021 17:59:41 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3A049C06173F for ; Mon, 13 Dec 2021 14:59:41 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id hg9-20020a17090b300900b001a6aa0b7d8cso10790035pjb.2 for ; Mon, 13 Dec 2021 14:59:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=VdGaN5L7OJ2U0UndQcZO7SPHQBeyLglC8eNYzsCUojk=; b=WwCQ6wfjwENs0PeaEtri9xJS77rN0uifxOv5MLSbmdJv5Bdh9V8Gd8i3Sv071LoFym mVNO5j78asLUedsxb0P7D3tlYmTmstqTEd0XYh0Xqeo91EJUb3c0UiXXyn2ElZGh1Gm7 ZfVO2Vbo+ZYMQYbyXBfCTu0N0hPOH4lLVlH6OJ0A3nee2DZdY+EMw5vgYkEvBePyEImj XTN4RowSCy3C8/+Yx4tHSZWc+LwwlBhmuzr6BmQ0j9x4hH8eQRjGqBUPpLy8fEI++oQA /1kR62t7jDgCkqtAfoO1eu6ELEPW7TNxtzk0S0izLZdruR/YMJSIBd2M9nDMOr8+DpQd Cqwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=VdGaN5L7OJ2U0UndQcZO7SPHQBeyLglC8eNYzsCUojk=; b=QeLyDldtuSGuwl0J07xuKphSvrD5IFME+XSqK4DeKSPLY/LCM+SCXFgDSDGoIhcWHn 34t70259dy6I1qo9IzZFJnyEThHEcCs1fyotq3WQNF+g1XMEfnUanKQs0X3C9RyEmPQR 4ONGSkTjqxUV61DmDdiKOP0D6MAUN9MZMwmUMcSPJn42NGplL1nhkCwyXy9s2T5mcS7n Y6tEm0BGFEaR2T5Q+PrY7sn1B8GJTlN5ZARnWhA7s7aysen35cvoCx9d+HVhea3hmwSP TN7wfjoJwJbJrauZTq2Rin+xwlftbwE4zW4D2TQEPAc3AO4tDvPaICOKLj5Rng09WVlr aabw== X-Gm-Message-State: AOAM533GdUdM4Y3n5jHKVwmADSbZOPuA/UmTJ+ER2+IJymsmbNwB1s5Q K5mDw+bIC7Iiux0czFk72YtunxIXjV7pAw== X-Google-Smtp-Source: ABdhPJwMeVajrFBE90cPxlJ/0DxK5HsgOXuYwfYm0L1wrQtN9xwxAt6wYplA6UhkNc0vp3gnVuKh9mpBnQy3/g== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a63:6703:: with SMTP id b3mr1178490pgc.18.1639436380650; Mon, 13 Dec 2021 14:59:40 -0800 (PST) Date: Mon, 13 Dec 2021 22:59:17 +0000 In-Reply-To: <20211213225918.672507-1-dmatlack@google.com> Message-Id: <20211213225918.672507-13-dmatlack@google.com> Mime-Version: 1.0 References: <20211213225918.672507-1-dmatlack@google.com> X-Mailer: git-send-email 2.34.1.173.g76aa8bc2d0-goog Subject: [PATCH v1 12/13] KVM: x86/mmu: Add tracepoint for splitting huge pages From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add a tracepoint that records whenever KVM eagerly splits a huge page. Signed-off-by: David Matlack Reviewed-by: Peter Xu --- arch/x86/kvm/mmu/mmutrace.h | 20 ++++++++++++++++++++ arch/x86/kvm/mmu/tdp_mmu.c | 2 ++ 2 files changed, 22 insertions(+) diff --git a/arch/x86/kvm/mmu/mmutrace.h b/arch/x86/kvm/mmu/mmutrace.h index de5e8e4e1aa7..4feabf773387 100644 --- a/arch/x86/kvm/mmu/mmutrace.h +++ b/arch/x86/kvm/mmu/mmutrace.h @@ -416,6 +416,26 @@ TRACE_EVENT( ) ); +TRACE_EVENT( + kvm_mmu_split_huge_page, + TP_PROTO(u64 gfn, u64 spte, int level), + TP_ARGS(gfn, spte, level), + + TP_STRUCT__entry( + __field(u64, gfn) + __field(u64, spte) + __field(int, level) + ), + + TP_fast_assign( + __entry->gfn = gfn; + __entry->spte = spte; + __entry->level = level; + ), + + TP_printk("gfn %llx spte %llx level %d", __entry->gfn, __entry->spte, __entry->level) +); + #endif /* _TRACE_KVMMMU_H */ #undef TRACE_INCLUDE_PATH diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index be5eb74ac053..e6910b9b5c12 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1325,6 +1325,8 @@ tdp_mmu_split_huge_page_atomic(struct kvm *kvm, struct tdp_iter *iter, struct kv u64 child_spte; int i; + trace_kvm_mmu_split_huge_page(iter->gfn, huge_spte, level); + init_child_tdp_mmu_page(sp, iter); for (i = 0; i < PT64_ENT_PER_PAGE; i++) { From patchwork Mon Dec 13 22:59:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12674821 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96B78C433F5 for ; Mon, 13 Dec 2021 22:59:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244112AbhLMW7o (ORCPT ); Mon, 13 Dec 2021 17:59:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47622 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244109AbhLMW7n (ORCPT ); Mon, 13 Dec 2021 17:59:43 -0500 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E8BF5C061748 for ; Mon, 13 Dec 2021 14:59:42 -0800 (PST) Received: by mail-pf1-x449.google.com with SMTP id l7-20020a622507000000b00494608c84a4so10898475pfl.6 for ; Mon, 13 Dec 2021 14:59:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=qw/gci8IX1okT5YydzCOdlI7c/eHaurUYfspM0RXE+0=; b=Dp9m6CaTA99gmmT5gmsRnzU95V2hN1Hndv8sfxTz7ZB+pG/Wc4zURE24VPitdFxv2I Sj7fQ7LyiBR/WoFNHlzQasvbEl+DLcVjxVOBIodkSEP7rVYL227Z070+2ZsnoME8LSBe crJ/mUA5xD2mO6oMVCsM1jymYigsdm3mjG62Vt3/K840NMHuxckMoUPCX0A+u4A/Mzac u8o8XRxQ91D4rlf1rz0Xm/h6F4Ig9ZvR0r9MdcVcG6k053yS5JlHVrTCTFXFgHDgQMzp K+0jHC3siLzccTpGxwBs6Vg1lvAWx2OxH3Ct8E2CrrIS3ssc8Z56IK5c4e4NsCQegjYE Goaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=qw/gci8IX1okT5YydzCOdlI7c/eHaurUYfspM0RXE+0=; b=Sed53e22HfGpmrdik1cl89HbOX+q53NpkYuFcOeXqdLTiy/WYaMS+W1K3qyjtG1tZY UKhxZ3PqMkQwr0xukAWR4ONfWOB0SooLcNdVco4Ff98norxJGc95BMapZRi8xikxDfZl nzevmJFqd9eW+tvNjbKp4i+WTtubcnvU/CZqNp7xYWsaaGB4RTZbg7Z6LKR8R1F+xYKz Qmf7yK4xngIPKM/9d15c4AdC3oM/zsGKNDtDudSEcAf2zg5qLdKQb6gf0Uyh7h2L79HP UDHTfPiH1tZcOw/WZ8tEHb9lE10cEczLf3s85nYztOri/LdPUdg9rGwTGt8eh74q/5vi hewQ== X-Gm-Message-State: AOAM533udS/dcSZCUfML2CvfCXJSXHJXbQTD0CWbb4ewH6+dZlij0Gsg F6GY26J9nzJIBpY9bwXENf739nWQOz7QsQ== X-Google-Smtp-Source: ABdhPJw1PtDtdtBdiIicIEZAR/xU2ss5GOasKvclx/0EvAYEZ3m4CJGf/UUcDcviwTnTsJdcORTT776h5wEDDQ== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a63:3285:: with SMTP id y127mr1098792pgy.479.1639436382406; Mon, 13 Dec 2021 14:59:42 -0800 (PST) Date: Mon, 13 Dec 2021 22:59:18 +0000 In-Reply-To: <20211213225918.672507-1-dmatlack@google.com> Message-Id: <20211213225918.672507-14-dmatlack@google.com> Mime-Version: 1.0 References: <20211213225918.672507-1-dmatlack@google.com> X-Mailer: git-send-email 2.34.1.173.g76aa8bc2d0-goog Subject: [PATCH v1 13/13] KVM: selftests: Add an option to disable MANUAL_PROTECT_ENABLE and INITIALLY_SET From: David Matlack To: Paolo Bonzini Cc: kvm@vger.kernel.org, Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Janis Schoetterl-Glausch , Junaid Shahid , Oliver Upton , Harish Barathvajasankar , Peter Xu , Peter Shier , "Nikunj A . Dadhania" , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add an option to dirty_log_perf_test to disable MANUAL_PROTECT_ENABLE and INITIALLY_SET so the legacy dirty logging code path can be tested. Signed-off-by: David Matlack Reviewed-by: Peter Xu --- tools/testing/selftests/kvm/dirty_log_perf_test.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/kvm/dirty_log_perf_test.c b/tools/testing/selftests/kvm/dirty_log_perf_test.c index 1954b964d1cf..a0c2247855f6 100644 --- a/tools/testing/selftests/kvm/dirty_log_perf_test.c +++ b/tools/testing/selftests/kvm/dirty_log_perf_test.c @@ -298,12 +298,15 @@ static void run_test(enum vm_guest_mode mode, void *arg) static void help(char *name) { puts(""); - printf("usage: %s [-h] [-i iterations] [-p offset] " + printf("usage: %s [-h] [-i iterations] [-p offset] [-g]" "[-m mode] [-b vcpu bytes] [-v vcpus] [-o] [-s mem type]" "[-x memslots]\n", name); puts(""); printf(" -i: specify iteration counts (default: %"PRIu64")\n", TEST_HOST_LOOP_N); + printf(" -g: Use the legacy dirty logging mode where KVM_GET_DIRTY_LOG\n" + " fetches and *clears* the dirty log. By default the test will\n" + " use MANUAL_PROTECT_ENABLE and INITIALLY_SET.\n"); printf(" -p: specify guest physical test memory offset\n" " Warning: a low offset can conflict with the loaded test code.\n"); guest_modes_help(); @@ -343,8 +346,11 @@ int main(int argc, char *argv[]) guest_modes_append_default(); - while ((opt = getopt(argc, argv, "hi:p:m:b:f:v:os:x:")) != -1) { + while ((opt = getopt(argc, argv, "ghi:p:m:b:f:v:os:x:")) != -1) { switch (opt) { + case 'g': + dirty_log_manual_caps = 0; + break; case 'i': p.iterations = atoi(optarg); break;