From patchwork Tue Jan 5 23:31:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Gardon X-Patchwork-Id: 12000555 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.6 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E96C1C433E9 for ; Tue, 5 Jan 2021 23:32:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C4CC522EBE for ; Tue, 5 Jan 2021 23:32:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727043AbhAEXcW (ORCPT ); Tue, 5 Jan 2021 18:32:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57486 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726610AbhAEXcV (ORCPT ); Tue, 5 Jan 2021 18:32:21 -0500 Received: from mail-qt1-x84a.google.com (mail-qt1-x84a.google.com [IPv6:2607:f8b0:4864:20::84a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 20E30C061574 for ; Tue, 5 Jan 2021 15:31:41 -0800 (PST) Received: by mail-qt1-x84a.google.com with SMTP id w3so902533qti.17 for ; Tue, 05 Jan 2021 15:31:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:message-id:mime-version:subject:from:to:cc; bh=dDiuE/L+TEOVkofWeADZyQOWx+prBVUG+hG/zE7qjqw=; b=g0QzlZiBXFKXnhwcB2i5sdfEuv61jQ/gqu5LPmOitwt436dX8/BoHXI5hB7eRkLIfs ZZ5pXK56kfNdrrSz2v88EaseQ/vE7rvZMyii/saTD1b555ShCSSLo10RQTfzsBh9nQlg kQTUeoodNDdBM5WFHuAMBgseEWiO5eX+tGsE/z19RmgZiQ+SVfANI5sXKtVjBEeiIwi6 6qUtS25WleI4JxfxUuDEGU1ORzvHKEqDvVPMujBJjYgZI4PwhAbDIkc2/xs9i75rRkGK F8ct/L3iru64+Z1/V3DV9HOyVETB9DWAe/Pm/dmikgC6jyVhP3+SHuisGhnsxBT/cQpB nCfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:message-id:mime-version:subject:from :to:cc; bh=dDiuE/L+TEOVkofWeADZyQOWx+prBVUG+hG/zE7qjqw=; b=IfAE6CBbHd33eA5qmx5TiMHXmLhsXgXGH8wHU9PXuzBIgpjmRTAfLgxfZkkCnONpV4 V0Cgl+5NGnk1qpy6Nsr3EspZDgW7h+nnW364st8rtzQOko0F85i9WnV3PA+BjJZafmn8 a96SVGmZv1eT9xra6eyWhsndG61ggnsJLSWd5ivjMwiSLGVapLoHNm3gRAvyP3wMxenU L9j6jShksW6BCoP4Bj/PIxGOrvUhpYTAUnzW7AoH7IZMeIqXkq5FsNF8USiwcgkYu9ZG qjUL8w2g6zNPNHH4YNI9RIyTbJ8hll5NDvitf1nJnDytr5eve0cAcQXRsegPL7PYxNk7 rFPQ== X-Gm-Message-State: AOAM532PAc18mC/b5kN4HCXdZGbBRkDvL5GGZK/d/pzgJYqTjbbhEDTu 78p6kknypxVUQZcofBj54X8KNdR2uNcp X-Google-Smtp-Source: ABdhPJyuqHDKvNJCgAD1VSvNZHytfUGZ2RnhLCOtS6Vfa+72p2t5/2Gh8C8r309kAlPmQART5K58nDKsBsTv Sender: "bgardon via sendgmr" X-Received: from bgardon.sea.corp.google.com ([2620:15c:100:202:f693:9fff:fef4:a293]) (user=bgardon job=sendgmr) by 2002:a0c:f74a:: with SMTP id e10mr1887824qvo.47.1609889500231; Tue, 05 Jan 2021 15:31:40 -0800 (PST) Date: Tue, 5 Jan 2021 15:31:34 -0800 Message-Id: <20210105233136.2140335-1-bgardon@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.29.2.729.g45daf8777d-goog Subject: [PATCH 1/3] kvm: x86/mmu: Clarify TDP MMU page list invariants From: Ben Gardon To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Paolo Bonzini , Sean Christopherson , Peter Shier , "Maciej S . Szmigiero" , Leo Hou , Ben Gardon Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The tdp_mmu_roots and tdp_mmu_pages in struct kvm_arch should only contain pages with tdp_mmu_page set to true. tdp_mmu_pages should not contain any pages with a non-zero root_count and tdp_mmu_roots should only contain pages with a positive root_count, unless a thread holds the MMU lock and is in the process of modifying the list. Various functions expect these invariants to be maintained, but they are not explictily documented. Add to the comments on both fields to document the above invariants. Signed-off-by: Ben Gardon --- arch/x86/include/asm/kvm_host.h | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 39707e72b062..2389735a29f3 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1010,9 +1010,21 @@ struct kvm_arch { */ bool tdp_mmu_enabled; - /* List of struct tdp_mmu_pages being used as roots */ + /* + * List of struct tdp_mmu_pages being used as roots. + * All struct kvm_mmu_pages in the list should have + * tdp_mmu_page set. + * All struct kvm_mmu_pages in the list should have a positive + * root_count except when a thread holds the MMU lock and is removing + * an entry from the list. + */ struct list_head tdp_mmu_roots; - /* List of struct tdp_mmu_pages not being used as roots */ + + /* + * List of struct tdp_mmu_pages not being used as roots. + * All struct kvm_mmu_pages in the list should have + * tdp_mmu_page set and a root_count of 0. + */ struct list_head tdp_mmu_pages; }; From patchwork Tue Jan 5 23:31:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Gardon X-Patchwork-Id: 12000557 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.6 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D814EC433E6 for ; Tue, 5 Jan 2021 23:32:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A288D230F9 for ; Tue, 5 Jan 2021 23:32:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727387AbhAEXcX (ORCPT ); Tue, 5 Jan 2021 18:32:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57494 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726641AbhAEXcW (ORCPT ); Tue, 5 Jan 2021 18:32:22 -0500 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9F992C061796 for ; Tue, 5 Jan 2021 15:31:42 -0800 (PST) Received: by mail-pl1-x64a.google.com with SMTP id ba10so600431plb.11 for ; Tue, 05 Jan 2021 15:31:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=ZyTlZ/k15y59etqIsKWXwuwdp0Ptq0PY9G2YOfirv10=; b=SmguYd94DqsefZDIkAqdPP7uTQvjPgC9cTP0H4eemFqpdBI1P60rgFvn9I9vphADaV kSP3+kxTg/EYQN90MrAPRyoC1htJekswogFgtlJL9UuydQzQ5hWkWa+J81FEkoyTCJJR KbVatbLypdphE1JplrNP/k6jIDwex4leD5yLbfFppMEfpvjJDrhlyKgYjBkao7ng9Bxh IwQDuzxi1ljbl7zO9vGvsUyM/XGCWKAkkqqb4XWb+BmxVz1Kc4wNE+XbT49yebYTrqTm bJ+O5XvdXc9BeuMUXM1qCQT0fs8nC9J+exRXJcw6fWgGgWg7t+iwFo/kDizNH3LYD4hW ApCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ZyTlZ/k15y59etqIsKWXwuwdp0Ptq0PY9G2YOfirv10=; b=UZ2BYpTPv2wz+IzeoZFgAcPLCb0HAEAfC9wllem9OCEjUV0PRZc//5+z1Be9oVjLGO DAE0vmx/7uXtOQj72Bl/qtfOW5AEfs+w0BjSvat+v++w4ZGCk55182j/nFWf73ls8S2O vAH/WtVd9ZrNOGIMh8cgHdXEcukZZWSTFhjoTq88X7hKEFsINN7X3BYz9Z+7B9jKK2x3 wCgL1gvtzuKbDDil3lUTC/BkxYlkCeAVrphDlO1ONUUhf6r+/oB7sK3/HcTAv+cNmDEU F3Kq5kBu3esRGpikdwuTzStFXiYbMSMU4EEyQjx96/imOLyMgvPEPBY/F3bmelINSkEb JR/w== X-Gm-Message-State: AOAM531e8D0pCJe59faKzEbBStJ1/AnevRvE5VS5PwZdnI+mXw1Ncqe5 3PWE81enrABloBxeVrXuh2prjRnY5W0E X-Google-Smtp-Source: ABdhPJwZ6yAMJafxX6iur1GyjDFFCTvcv+8fp18C24zD31Xn5DRrVmLWALKKAzXWJlo8+8ZcrXKFAcB0Oc2T Sender: "bgardon via sendgmr" X-Received: from bgardon.sea.corp.google.com ([2620:15c:100:202:f693:9fff:fef4:a293]) (user=bgardon job=sendgmr) by 2002:a17:90a:930f:: with SMTP id p15mr1435933pjo.73.1609889502096; Tue, 05 Jan 2021 15:31:42 -0800 (PST) Date: Tue, 5 Jan 2021 15:31:35 -0800 In-Reply-To: <20210105233136.2140335-1-bgardon@google.com> Message-Id: <20210105233136.2140335-2-bgardon@google.com> Mime-Version: 1.0 References: <20210105233136.2140335-1-bgardon@google.com> X-Mailer: git-send-email 2.29.2.729.g45daf8777d-goog Subject: [PATCH 2/3] kvm: x86/mmu: Ensure TDP MMU roots are freed after yield From: Ben Gardon To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Paolo Bonzini , Sean Christopherson , Peter Shier , "Maciej S . Szmigiero" , Leo Hou , Ben Gardon Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Many TDP MMU functions which need to perform some action on all TDP MMU roots hold a reference on that root so that they can safely drop the MMU lock in order to yield to other threads. However, when releasing the reference on the root, there is a bug: the root will not be freed even if its reference count (root_count) is reduced to 0. Ensure that these roots are properly freed. Reported-by: Maciej S. Szmigiero Fixes: faaf05b00aec ("kvm: x86/mmu: Support zapping SPTEs in the TDP MMU") Fixes: 063afacd8730 ("kvm: x86/mmu: Support invalidate range MMU notifier for TDP MMU") Fixes: a6a0b05da9f3 ("kvm: x86/mmu: Support dirty logging for the TDP MMU") Fixes: 14881998566d ("kvm: x86/mmu: Support disabling dirty logging for the tdp MMU") Signed-off-by: Ben Gardon --- arch/x86/kvm/mmu/tdp_mmu.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 75db27fda8f3..5ec6fae36e33 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -83,6 +83,12 @@ void kvm_tdp_mmu_free_root(struct kvm *kvm, struct kvm_mmu_page *root) kmem_cache_free(mmu_page_header_cache, root); } +static void tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root) +{ + if (kvm_mmu_put_root(kvm, root)) + kvm_tdp_mmu_free_root(kvm, root); +} + static union kvm_mmu_page_role page_role_for_level(struct kvm_vcpu *vcpu, int level) { @@ -456,7 +462,7 @@ bool kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, gfn_t start, gfn_t end) flush |= zap_gfn_range(kvm, root, start, end, true); - kvm_mmu_put_root(kvm, root); + tdp_mmu_put_root(kvm, root); } return flush; @@ -648,7 +654,7 @@ static int kvm_tdp_mmu_handle_hva_range(struct kvm *kvm, unsigned long start, gfn_end, data); } - kvm_mmu_put_root(kvm, root); + tdp_mmu_put_root(kvm, root); } return ret; @@ -852,7 +858,7 @@ bool kvm_tdp_mmu_wrprot_slot(struct kvm *kvm, struct kvm_memory_slot *slot, spte_set |= wrprot_gfn_range(kvm, root, slot->base_gfn, slot->base_gfn + slot->npages, min_level); - kvm_mmu_put_root(kvm, root); + tdp_mmu_put_root(kvm, root); } return spte_set; @@ -920,7 +926,7 @@ bool kvm_tdp_mmu_clear_dirty_slot(struct kvm *kvm, struct kvm_memory_slot *slot) spte_set |= clear_dirty_gfn_range(kvm, root, slot->base_gfn, slot->base_gfn + slot->npages); - kvm_mmu_put_root(kvm, root); + tdp_mmu_put_root(kvm, root); } return spte_set; @@ -1043,7 +1049,7 @@ bool kvm_tdp_mmu_slot_set_dirty(struct kvm *kvm, struct kvm_memory_slot *slot) spte_set |= set_dirty_gfn_range(kvm, root, slot->base_gfn, slot->base_gfn + slot->npages); - kvm_mmu_put_root(kvm, root); + tdp_mmu_put_root(kvm, root); } return spte_set; } @@ -1103,7 +1109,7 @@ void kvm_tdp_mmu_zap_collapsible_sptes(struct kvm *kvm, zap_collapsible_spte_range(kvm, root, slot->base_gfn, slot->base_gfn + slot->npages); - kvm_mmu_put_root(kvm, root); + tdp_mmu_put_root(kvm, root); } } From patchwork Tue Jan 5 23:31:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Gardon X-Patchwork-Id: 12000553 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.6 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BABD1C433E0 for ; Tue, 5 Jan 2021 23:32:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8C94A22EBF for ; Tue, 5 Jan 2021 23:32:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727698AbhAEXc2 (ORCPT ); Tue, 5 Jan 2021 18:32:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57504 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727260AbhAEXcZ (ORCPT ); Tue, 5 Jan 2021 18:32:25 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC368C06179A for ; Tue, 5 Jan 2021 15:31:44 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id d187so1711978ybc.6 for ; Tue, 05 Jan 2021 15:31:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=yobAeCu9/CRaXkHztpEWt3T9SErbOcVCTVp7Taw/teg=; b=cotQWsaTZX+jEnA0InHChyabxaJcwvlomvHc+t1xyAQpFN/IzbqCnOE09XLB6oywJ2 0Zva107ckeIsSzW1JWeN/dbCGs/L8J5bJt6T5Z3UcQiohXQv1Muvxxewl6KAW0DZD/FP 5JE5cQpHxtLVZxIk2aHNfAHwa5QVgOFpYEmlSuWDMlK1bZCOP4klg9zlA3iI7mrkPzFe LqZHSQxKHG6+7vkDQ17S9tuaE5huJJkRKxvQ26VsSSEN/7IGRXWZq5wBnC/g64m3wxwa FPMQC+LIAq9IGUv2+Fs7FlpnVcbjOvujXz4zH5KSnFVz1iX/8IIxxC9J0qrDeYYPt1Ij RVdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=yobAeCu9/CRaXkHztpEWt3T9SErbOcVCTVp7Taw/teg=; b=dWzoNXzy1Q/9mlm2riEFSDiI2yVYwZ9VibBR1aluDdgdUkX4wZ8kEOFYChxiuLJ46n yqI/5f4RspCvPGNhRk6fk07V/lvpO2mk5dhRCMAX94eChIQUZfEr1Vl8uQcm42XS2963 6E7krh1dr47rEOOXh8JhkLsJDzjskc7TGbRZVFSev3JeMiD2gQ9HPX9x/+aJ+brHMQfs zGW+X7ZXlqAyrc0UW+WGid27Exk/8k+bhVnVcmXJcgkODvZp2I0uQq+OpsMKYeFlk48w Vt24lAgSdfztCn7Q/2RsGcEy6b6xLFHVtV5O3DVgnBSJsbvcWYi9G4c4RlLOJA1RdTE9 vrZA== X-Gm-Message-State: AOAM532aCFePa/51t8EkuxvHwb01Ip22Q5WC7T3jsPKFqmtDKUTKw//I 6hsFbZS4ql+zk5we2w9Emd9FHYQbaKWL X-Google-Smtp-Source: ABdhPJyaQfETm9KfrxqQjPNIrSDFs9roAaP3NZN+uJeBKTPHCNqy4vd0T4gXoZMb6aDA8WNKjxRAD/vgjMvF Sender: "bgardon via sendgmr" X-Received: from bgardon.sea.corp.google.com ([2620:15c:100:202:f693:9fff:fef4:a293]) (user=bgardon job=sendgmr) by 2002:a25:938a:: with SMTP id a10mr2386436ybm.49.1609889504006; Tue, 05 Jan 2021 15:31:44 -0800 (PST) Date: Tue, 5 Jan 2021 15:31:36 -0800 In-Reply-To: <20210105233136.2140335-1-bgardon@google.com> Message-Id: <20210105233136.2140335-3-bgardon@google.com> Mime-Version: 1.0 References: <20210105233136.2140335-1-bgardon@google.com> X-Mailer: git-send-email 2.29.2.729.g45daf8777d-goog Subject: [PATCH 3/3] kvm: x86/mmu: Get/put TDP MMU root refs in iterator From: Ben Gardon To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Paolo Bonzini , Sean Christopherson , Peter Shier , "Maciej S . Szmigiero" , Leo Hou , Ben Gardon Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org To simplify acquiring and releasing references on TDP MMU root pages, move the get/put operations into the TDP MMU root iterator macro. Not all functions which use the macro currently get and put a reference to the root, but adding this behavior is harmless. Suggested-by: Paolo Bonzini Signed-off-by: Ben Gardon --- arch/x86/kvm/mmu/tdp_mmu.c | 100 +++++++++++++++---------------------- 1 file changed, 41 insertions(+), 59 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 5ec6fae36e33..fc69216839c6 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -44,8 +44,41 @@ void kvm_mmu_uninit_tdp_mmu(struct kvm *kvm) WARN_ON(!list_empty(&kvm->arch.tdp_mmu_roots)); } -#define for_each_tdp_mmu_root(_kvm, _root) \ - list_for_each_entry(_root, &_kvm->arch.tdp_mmu_roots, link) +static void tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root) +{ + if (kvm_mmu_put_root(kvm, root)) + kvm_tdp_mmu_free_root(kvm, root); +} + +static inline bool tdp_mmu_next_root_valid(struct kvm *kvm, + struct kvm_mmu_page *root) +{ + if (list_entry_is_head(root, &kvm->arch.tdp_mmu_roots, link)) + return false; + + kvm_mmu_get_root(kvm, root); + return true; + +} + +static inline struct kvm_mmu_page *tdp_mmu_next_root(struct kvm *kvm, + struct kvm_mmu_page *root) +{ + tdp_mmu_put_root(kvm, root); + return list_next_entry(root, link); +} + +/* + * Note: this iterator gets and puts references to the roots it iterates over. + * This makes it safe to release the MMU lock and yield within the loop, but + * if exiting the loop early, the caller must drop the reference to the most + * recent root. (Unless keeping a live reference is desirable.) + */ +#define for_each_tdp_mmu_root(_kvm, _root) \ + for (_root = list_first_entry(&_kvm->arch.tdp_mmu_roots, \ + typeof(*_root), link); \ + tdp_mmu_next_root_valid(_kvm, _root); \ + _root = tdp_mmu_next_root(_kvm, _root)) bool is_tdp_mmu_root(struct kvm *kvm, hpa_t hpa) { @@ -83,12 +116,6 @@ void kvm_tdp_mmu_free_root(struct kvm *kvm, struct kvm_mmu_page *root) kmem_cache_free(mmu_page_header_cache, root); } -static void tdp_mmu_put_root(struct kvm *kvm, struct kvm_mmu_page *root) -{ - if (kvm_mmu_put_root(kvm, root)) - kvm_tdp_mmu_free_root(kvm, root); -} - static union kvm_mmu_page_role page_role_for_level(struct kvm_vcpu *vcpu, int level) { @@ -134,7 +161,11 @@ static struct kvm_mmu_page *get_tdp_mmu_vcpu_root(struct kvm_vcpu *vcpu) /* Check for an existing root before allocating a new one. */ for_each_tdp_mmu_root(kvm, root) { if (root->role.word == role.word) { - kvm_mmu_get_root(kvm, root); + /* + * The iterator already acquired a reference to this + * root, so simply return early without dropping the + * reference. + */ spin_unlock(&kvm->mmu_lock); return root; } @@ -453,18 +484,9 @@ bool kvm_tdp_mmu_zap_gfn_range(struct kvm *kvm, gfn_t start, gfn_t end) struct kvm_mmu_page *root; bool flush = false; - for_each_tdp_mmu_root(kvm, root) { - /* - * Take a reference on the root so that it cannot be freed if - * this thread releases the MMU lock and yields in this loop. - */ - kvm_mmu_get_root(kvm, root); - + for_each_tdp_mmu_root(kvm, root) flush |= zap_gfn_range(kvm, root, start, end, true); - tdp_mmu_put_root(kvm, root); - } - return flush; } @@ -626,12 +648,6 @@ static int kvm_tdp_mmu_handle_hva_range(struct kvm *kvm, unsigned long start, int as_id; for_each_tdp_mmu_root(kvm, root) { - /* - * Take a reference on the root so that it cannot be freed if - * this thread releases the MMU lock and yields in this loop. - */ - kvm_mmu_get_root(kvm, root); - as_id = kvm_mmu_page_as_id(root); slots = __kvm_memslots(kvm, as_id); kvm_for_each_memslot(memslot, slots) { @@ -653,8 +669,6 @@ static int kvm_tdp_mmu_handle_hva_range(struct kvm *kvm, unsigned long start, ret |= handler(kvm, memslot, root, gfn_start, gfn_end, data); } - - tdp_mmu_put_root(kvm, root); } return ret; @@ -849,16 +863,8 @@ bool kvm_tdp_mmu_wrprot_slot(struct kvm *kvm, struct kvm_memory_slot *slot, if (root_as_id != slot->as_id) continue; - /* - * Take a reference on the root so that it cannot be freed if - * this thread releases the MMU lock and yields in this loop. - */ - kvm_mmu_get_root(kvm, root); - spte_set |= wrprot_gfn_range(kvm, root, slot->base_gfn, slot->base_gfn + slot->npages, min_level); - - tdp_mmu_put_root(kvm, root); } return spte_set; @@ -917,16 +923,8 @@ bool kvm_tdp_mmu_clear_dirty_slot(struct kvm *kvm, struct kvm_memory_slot *slot) if (root_as_id != slot->as_id) continue; - /* - * Take a reference on the root so that it cannot be freed if - * this thread releases the MMU lock and yields in this loop. - */ - kvm_mmu_get_root(kvm, root); - spte_set |= clear_dirty_gfn_range(kvm, root, slot->base_gfn, slot->base_gfn + slot->npages); - - tdp_mmu_put_root(kvm, root); } return spte_set; @@ -1040,16 +1038,8 @@ bool kvm_tdp_mmu_slot_set_dirty(struct kvm *kvm, struct kvm_memory_slot *slot) if (root_as_id != slot->as_id) continue; - /* - * Take a reference on the root so that it cannot be freed if - * this thread releases the MMU lock and yields in this loop. - */ - kvm_mmu_get_root(kvm, root); - spte_set |= set_dirty_gfn_range(kvm, root, slot->base_gfn, slot->base_gfn + slot->npages); - - tdp_mmu_put_root(kvm, root); } return spte_set; } @@ -1100,16 +1090,8 @@ void kvm_tdp_mmu_zap_collapsible_sptes(struct kvm *kvm, if (root_as_id != slot->as_id) continue; - /* - * Take a reference on the root so that it cannot be freed if - * this thread releases the MMU lock and yields in this loop. - */ - kvm_mmu_get_root(kvm, root); - zap_collapsible_spte_range(kvm, root, slot->base_gfn, slot->base_gfn + slot->npages); - - tdp_mmu_put_root(kvm, root); } }