From patchwork Wed Jun 30 21:47:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12352833 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-23.6 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, UNWANTED_LANGUAGE_BODY,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90E1EC11F68 for ; Wed, 30 Jun 2021 21:48:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 77A7B61477 for ; Wed, 30 Jun 2021 21:48:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232327AbhF3Vuv (ORCPT ); Wed, 30 Jun 2021 17:50:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39598 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232042AbhF3Vuu (ORCPT ); Wed, 30 Jun 2021 17:50:50 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 57513C061756 for ; Wed, 30 Jun 2021 14:48:21 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id c10-20020a17090a558ab029017019f7ec8fso2068037pji.8 for ; Wed, 30 Jun 2021 14:48:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=1h8e+apG3TTK1wRFhX1YgohezWt8EbM/GUKeu+wHNLA=; b=TtIfBmZfHCJ2cr5jLzgT3clLRpf8lqr4lK1J+gSBwh8JoI65zvLR3SA46owrDYkD4z cOzWzvLa4J28heFnsTro9kHcfuwn3uD2vI85vK1rXG6bQXPwHWjirmsuPRO61MoDQcfM ZwaKFSy+44VC+oZuRhPtkbVfjSy/pgaug39/jpiS8pERPzFSw7DTRoiCG9panhRREmL+ DShfcinj3BZTWc24EZF0ORkQjJyF6PSkn4CDLlmCN1C+VLIofRtF9mA7Z5W5RG6rFEpA jtFO6k/HCqX+Jgq1hve/RYZnKGr3CN4JMyDKMNxIOYeJFEHqXc9InxXzkpGUnAH40hqQ U1nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=1h8e+apG3TTK1wRFhX1YgohezWt8EbM/GUKeu+wHNLA=; b=MgW2vVJr2Nup+Eq4zejPnI3bIq1FOu0dyAvWvtBIqjP0p3FvgP4j2v08JpRhAK8z8C 08HlD1FyfIvsyJSYLYxNkulrjoJMRi0yYMMhM8Yp1zby9ZDBhUo/sGtbUAQGyu93imV3 n+gMfvkKnAp0OAwwdAKULgMTTMrwa0Oo3zKJSTfh2kf5ISYLGw021tVZJxSfL+J9njPv 8dQ8fEiFyu+HvvR8TiKNyJhRCtthe4Q9pgXWZkGy5wWfp5hjHGje4BtsV3LF00d5W4KI CM/XpIxPhm+YEn0f0/bzmKpf3Il8lc1sUdRNAuKffVA4TkE9xDqJprRUNnrhGJZ4QhPz ktkA== X-Gm-Message-State: AOAM530JCAZ2rEQJIZ+lH3NfGlZKNBKvIpqQYyNX06AQhQgbQ5aOlqxO hYBrtAt/S1DH9C4SOXkvhvnEKgweOejK0PKLH4p1HfvD5cO+AuBILB4BqXJokxRjiHwMYaVgjDt IoBXJA1RnBBDBtO0Rpc9/NhmURYnf2/WUn5L+wuhGvUtFnEW83fakmy60vWAYEKo= X-Google-Smtp-Source: ABdhPJy/+bjigqQpu7LXftTmsvbM3zANLjDLlj3pASUu0wshlCndt+Uzut1h4+IlM2ZnzFbToNGOnteFCAUpdg== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:90b:4a0a:: with SMTP id kk10mr40665935pjb.16.1625089700683; Wed, 30 Jun 2021 14:48:20 -0700 (PDT) Date: Wed, 30 Jun 2021 21:47:57 +0000 In-Reply-To: <20210630214802.1902448-1-dmatlack@google.com> Message-Id: <20210630214802.1902448-2-dmatlack@google.com> Mime-Version: 1.0 References: <20210630214802.1902448-1-dmatlack@google.com> X-Mailer: git-send-email 2.32.0.93.g670b81a890-goog Subject: [PATCH v2 1/6] KVM: x86/mmu: Rename cr2_or_gpa to gpa in fast_page_fault From: David Matlack To: kvm@vger.kernel.org Cc: Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Paolo Bonzini , Junaid Shahid , Andrew Jones , Matthew Wilcox , Yu Zhao , David Hildenbrand , Andrew Morton , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org fast_page_fault is only called from direct_page_fault where we know the address is a gpa. Fixes: 736c291c9f36 ("KVM: x86: Use gpa_t for cr2/gpa to fix TDP support on 32-bit KVM") Reviewed-by: Ben Gardon Signed-off-by: David Matlack Reviewed-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index b888385d1933..45274436d3c0 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3098,8 +3098,7 @@ static bool is_access_allowed(u32 fault_err_code, u64 spte) /* * Returns one of RET_PF_INVALID, RET_PF_FIXED or RET_PF_SPURIOUS. */ -static int fast_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, - u32 error_code) +static int fast_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code) { struct kvm_shadow_walk_iterator iterator; struct kvm_mmu_page *sp; @@ -3115,7 +3114,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, do { u64 new_spte; - for_each_shadow_entry_lockless(vcpu, cr2_or_gpa, iterator, spte) + for_each_shadow_entry_lockless(vcpu, gpa, iterator, spte) if (!is_shadow_present_pte(spte)) break; @@ -3194,8 +3193,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, } while (true); - trace_fast_page_fault(vcpu, cr2_or_gpa, error_code, iterator.sptep, - spte, ret); + trace_fast_page_fault(vcpu, gpa, error_code, iterator.sptep, spte, ret); walk_shadow_page_lockless_end(vcpu); return ret; From patchwork Wed Jun 30 21:47:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12352835 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.4 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8F25C11F65 for ; Wed, 30 Jun 2021 21:48:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8D34A61185 for ; Wed, 30 Jun 2021 21:48:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232432AbhF3Vux (ORCPT ); Wed, 30 Jun 2021 17:50:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39606 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232397AbhF3Vuw (ORCPT ); Wed, 30 Jun 2021 17:50:52 -0400 Received: from mail-qt1-x849.google.com (mail-qt1-x849.google.com [IPv6:2607:f8b0:4864:20::849]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2A4DAC061756 for ; Wed, 30 Jun 2021 14:48:23 -0700 (PDT) Received: by mail-qt1-x849.google.com with SMTP id c17-20020ac87dd10000b0290250fd339409so2037713qte.6 for ; Wed, 30 Jun 2021 14:48:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=MWYYk9Sr8HKYlfAS2NHSlFPP/eKAy+6T+SEm5ULRBgE=; b=q+c1Q3WcUvx9b5dZrgTHagM+IM3jIXcjW40U79ExET56cSUFS2poOMLEtjQuvyUa3d +E+8lRvLOEA65qIxRru5VVAWYC7AHlPCo0ccaYGaltik+EheZUkoj4ozlE6m0DyPm+cy USx/H9SYRAF6d53fZBlbJMnZG2I7phaLYYG8SGGTRNwnmTrkqPt67kgqP4DRrklWs92A O6Gq3VcfR7NSwZeb0s72SY4G5Rco/BfjFaCAKxnKBYdSQTW2Vn7hK5ez2LB/WXVroiOx owxLPBGQzJTRl8G5KXB/H1CuGmbsRzWAkSd8E4W0ZjAz7QYdvryRME46YHUtcd+qX8tu BPdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=MWYYk9Sr8HKYlfAS2NHSlFPP/eKAy+6T+SEm5ULRBgE=; b=ufDWIf1R263wEew9c7squgbQC27ai9vwnGZpLnEfPWKt46tFmKKZ9/qdifivKQwwLU eiDhG3boiFwkc+4+2x9Xes1qnlNXwUDkqOZ9OMAnnz4zCPDZnAgEANPMyQ8WyHvEpZX2 BGbYTg1g1SVR4iP8A7pndpS/Thv84KpjST3tD9/wNFUw2ZIGlJlyPnhV8jd2Xf81KvDi DkOrpdmUhnEJv7zmNce/TGLZOcGcI+RGSF4EBULSvHlaP2whdtRgHU7Ogx81VQHzG0VL EmNSq7Te9BMQcefs/cMCTibUqUz72Pvsa+HWR277YoPiQqXayVuwxwTz1dV0CZ+YKkmL Wk0Q== X-Gm-Message-State: AOAM531LIIbjVM1F+4GS3as96lh0tUKvrdFWJYB139MMFJ1wS8duvUgz BH8nOuXK0KNq4bFr2kC1SS3iC3II2Wq/YT1SMDrSSGnt3czTBO3qNZ9rOnblNkIUoAPSNUzBDwb iyzPeSOXf02jECH8YrZHRqJspsRPg4AKrMB35arxQoSZ6B06KaInk6RGWRCumAgw= X-Google-Smtp-Source: ABdhPJySf6pu89+nwMZgxjcqOXquBVQqyRpYf27QInfPRVYjrw70yBB+TIUWLuKxKlWtGN8YDSkhZYlM74eaLw== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a0c:dd8c:: with SMTP id v12mr15116890qvk.23.1625089702229; Wed, 30 Jun 2021 14:48:22 -0700 (PDT) Date: Wed, 30 Jun 2021 21:47:58 +0000 In-Reply-To: <20210630214802.1902448-1-dmatlack@google.com> Message-Id: <20210630214802.1902448-3-dmatlack@google.com> Mime-Version: 1.0 References: <20210630214802.1902448-1-dmatlack@google.com> X-Mailer: git-send-email 2.32.0.93.g670b81a890-goog Subject: [PATCH v2 2/6] KVM: x86/mmu: Fix use of enums in trace_fast_page_fault From: David Matlack To: kvm@vger.kernel.org Cc: Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Paolo Bonzini , Junaid Shahid , Andrew Jones , Matthew Wilcox , Yu Zhao , David Hildenbrand , Andrew Morton , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Enum values have to be exported to userspace since the formatting is not done in the kernel. Without doing this perf maps RET_PF_FIXED and RET_PF_SPURIOUS to 0, which results in incorrect output: $ perf record -a -e kvmmmu:fast_page_fault --filter "ret==3" -- ./access_tracking_perf_test $ perf script | head -1 [...] new 610006048d25877 spurious 0 fixed 0 <------ should be 1 Fix this by exporting the enum values to userspace with TRACE_DEFINE_ENUM. Fixes: c4371c2a682e ("KVM: x86/mmu: Return unique RET_PF_* values if the fault was fixed") Signed-off-by: David Matlack --- arch/x86/kvm/mmu/mmutrace.h | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/x86/kvm/mmu/mmutrace.h b/arch/x86/kvm/mmu/mmutrace.h index efbad33a0645..55c7e0fcda52 100644 --- a/arch/x86/kvm/mmu/mmutrace.h +++ b/arch/x86/kvm/mmu/mmutrace.h @@ -244,6 +244,9 @@ TRACE_EVENT( __entry->access) ); +TRACE_DEFINE_ENUM(RET_PF_FIXED); +TRACE_DEFINE_ENUM(RET_PF_SPURIOUS); + TRACE_EVENT( fast_page_fault, TP_PROTO(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u32 error_code, From patchwork Wed Jun 30 21:47:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12352837 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.4 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D149AC11F68 for ; Wed, 30 Jun 2021 21:48:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B65E0611AC for ; Wed, 30 Jun 2021 21:48:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232511AbhF3Vuy (ORCPT ); Wed, 30 Jun 2021 17:50:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39612 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232397AbhF3Vux (ORCPT ); Wed, 30 Jun 2021 17:50:53 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 727C7C0617A8 for ; Wed, 30 Jun 2021 14:48:24 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id q9-20020a17090a0649b029016ffc6b9665so4644584pje.1 for ; Wed, 30 Jun 2021 14:48:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=c7z8kdWFSIxQNMBObfdt2Re6PF35LU4DZgXWH2TIoqU=; b=iKiSbqHdGezXssdhrW4FVZ14QDpjn5Yt8UO5CX57jCLusCGMLMN9K0tQVjw4YoJ6AN 1z32q0ZEFCLAWG1AQ2rWbj5mwzULKUbDtyyN1kSNWcO7Zx2hvGsWcnEbaec7tatEg+pA 71mID9BVOFsA/8ag24Qbgbpv1iF+6lvsxAY/60u8iNSZZCD2qWKtwbyrXhsF7NEICqn9 jai1cUYNwChfhHXTaY+DozTKe4M+M740W8MzLjx4GfJwCEYjq09eZjKpBrvJT8PFteZg clSzHZmD2p8ryNTLuAOmQgoavsl3Hq4zQl1gkkXQHrat5YeI9OUhj6e7TmgYciC5Riql NFVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=c7z8kdWFSIxQNMBObfdt2Re6PF35LU4DZgXWH2TIoqU=; b=HOfsGpCJExrHqGEsmh8WFsjRJr276u6ha9X0QE/lRhNHwprHGm9shCqBq3YcMiBAWy eMutKFaHxzFRyf9IZD09acOlylygkFi1Tv8X3+mFs/5XKHbfwB4eAlKrZs6ariCV1fbz 6cpU8Ccq2RYUoY4+pPOjpbTiLj4sTV68ntbKz3hEnnc2CVccTnTQUErmItQtnEuBjh0s f4pYNliTmB+wiScLSteiQK5ZifPNnNk+zWkfyaUxE94xPZpF2V1G/1V6mWZ7j6RXDDjL /Cw6li62UcUa5L6p5dAvKZ0yLJZI3gOq2YLADDFC03wVo6wCs7dSNehtBO+IjBQb8T95 HrqA== X-Gm-Message-State: AOAM531vnipJkpWL20kIqFiGznPDr+d3RApq6TFhImM0Jp1tIgdSajCF TbMAuuqzr56Y2W4nXeczwdlCPIavgGNnom8Z2/9YEWFk+SKIl5DxnfeAMBSU/JMCfQnX+Lo2b8/ 9duz2b+9WjmcwssaU62Gw1MUFZ4RXpqVePUVcWF7pOlXJ4nHrp7TuFV80wj2SKCA= X-Google-Smtp-Source: ABdhPJxgiUu0EL/PBbIT39F4ybj8K110+gsAkbpv2UB2yUYgBxb4BY3NTf5wamQkgYzedjTfQ+y/aKoz5ts7AA== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:903:18e:b029:127:a5ba:7243 with SMTP id z14-20020a170903018eb0290127a5ba7243mr29683618plg.4.1625089703843; Wed, 30 Jun 2021 14:48:23 -0700 (PDT) Date: Wed, 30 Jun 2021 21:47:59 +0000 In-Reply-To: <20210630214802.1902448-1-dmatlack@google.com> Message-Id: <20210630214802.1902448-4-dmatlack@google.com> Mime-Version: 1.0 References: <20210630214802.1902448-1-dmatlack@google.com> X-Mailer: git-send-email 2.32.0.93.g670b81a890-goog Subject: [PATCH v2 3/6] KVM: x86/mmu: Make walk_shadow_page_lockless_{begin,end} interoperate with the TDP MMU From: David Matlack To: kvm@vger.kernel.org Cc: Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Paolo Bonzini , Junaid Shahid , Andrew Jones , Matthew Wilcox , Yu Zhao , David Hildenbrand , Andrew Morton , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Acquire the RCU read lock in walk_shadow_page_lockless_begin and release it in walk_shadow_page_lockless_end when the TDP MMU is enabled. This should not introduce any functional changes but is used in the following commit to make fast_page_fault interoperate with the TDP MMU. Signed-off-by: David Matlack Reviewed-by: Ben Gardon --- arch/x86/kvm/mmu/mmu.c | 24 ++++++++++++++++++------ arch/x86/kvm/mmu/tdp_mmu.c | 20 ++++++++++++++------ arch/x86/kvm/mmu/tdp_mmu.h | 6 ++++-- 3 files changed, 36 insertions(+), 14 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 45274436d3c0..88c71a8a55f1 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -686,6 +686,11 @@ static bool mmu_spte_age(u64 *sptep) static void walk_shadow_page_lockless_begin(struct kvm_vcpu *vcpu) { + if (is_tdp_mmu(vcpu->arch.mmu)) { + kvm_tdp_mmu_walk_lockless_begin(); + return; + } + /* * Prevent page table teardown by making any free-er wait during * kvm_flush_remote_tlbs() IPI to all active vcpus. @@ -701,6 +706,11 @@ static void walk_shadow_page_lockless_begin(struct kvm_vcpu *vcpu) static void walk_shadow_page_lockless_end(struct kvm_vcpu *vcpu) { + if (is_tdp_mmu(vcpu->arch.mmu)) { + kvm_tdp_mmu_walk_lockless_end(); + return; + } + /* * Make sure the write to vcpu->mode is not reordered in front of * reads to sptes. If it does, kvm_mmu_commit_zap_page() can see us @@ -3621,6 +3631,12 @@ static int get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, int *root_level walk_shadow_page_lockless_begin(vcpu); + if (is_tdp_mmu(vcpu->arch.mmu)) { + leaf = kvm_tdp_mmu_get_walk_lockless(vcpu, addr, sptes, + root_level); + goto out; + } + for (shadow_walk_init(&iterator, vcpu, addr), *root_level = iterator.level; shadow_walk_okay(&iterator); @@ -3634,8 +3650,8 @@ static int get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, int *root_level break; } +out: walk_shadow_page_lockless_end(vcpu); - return leaf; } @@ -3647,11 +3663,7 @@ static bool get_mmio_spte(struct kvm_vcpu *vcpu, u64 addr, u64 *sptep) int root, leaf, level; bool reserved = false; - if (is_tdp_mmu(vcpu->arch.mmu)) - leaf = kvm_tdp_mmu_get_walk(vcpu, addr, sptes, &root); - else - leaf = get_walk(vcpu, addr, sptes, &root); - + leaf = get_walk(vcpu, addr, sptes, &root); if (unlikely(leaf < 0)) { *sptep = 0ull; return reserved; diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index caac4ddb46df..c6fa8d00bf9f 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1513,12 +1513,24 @@ bool kvm_tdp_mmu_write_protect_gfn(struct kvm *kvm, return spte_set; } +void kvm_tdp_mmu_walk_lockless_begin(void) +{ + rcu_read_lock(); +} + +void kvm_tdp_mmu_walk_lockless_end(void) +{ + rcu_read_unlock(); +} + /* * Return the level of the lowest level SPTE added to sptes. * That SPTE may be non-present. + * + * Must be called between kvm_tdp_mmu_walk_lockless_{begin,end}. */ -int kvm_tdp_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, - int *root_level) +int kvm_tdp_mmu_get_walk_lockless(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, + int *root_level) { struct tdp_iter iter; struct kvm_mmu *mmu = vcpu->arch.mmu; @@ -1527,14 +1539,10 @@ int kvm_tdp_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, *root_level = vcpu->arch.mmu->shadow_root_level; - rcu_read_lock(); - tdp_mmu_for_each_pte(iter, mmu, gfn, gfn + 1) { leaf = iter.level; sptes[leaf] = iter.old_spte; } - rcu_read_unlock(); - return leaf; } diff --git a/arch/x86/kvm/mmu/tdp_mmu.h b/arch/x86/kvm/mmu/tdp_mmu.h index 1cae4485b3bc..e9dde5f9c0ef 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.h +++ b/arch/x86/kvm/mmu/tdp_mmu.h @@ -77,8 +77,10 @@ bool kvm_tdp_mmu_write_protect_gfn(struct kvm *kvm, struct kvm_memory_slot *slot, gfn_t gfn, int min_level); -int kvm_tdp_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, - int *root_level); +void kvm_tdp_mmu_walk_lockless_begin(void); +void kvm_tdp_mmu_walk_lockless_end(void); +int kvm_tdp_mmu_get_walk_lockless(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, + int *root_level); #ifdef CONFIG_X86_64 bool kvm_mmu_init_tdp_mmu(struct kvm *kvm); From patchwork Wed Jun 30 21:48:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12352839 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.4 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06F8DC11F65 for ; Wed, 30 Jun 2021 21:48:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E199161474 for ; Wed, 30 Jun 2021 21:48:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232541AbhF3Vu7 (ORCPT ); Wed, 30 Jun 2021 17:50:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39624 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232394AbhF3Vuz (ORCPT ); Wed, 30 Jun 2021 17:50:55 -0400 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4691BC061756 for ; Wed, 30 Jun 2021 14:48:26 -0700 (PDT) Received: by mail-pg1-x54a.google.com with SMTP id x7-20020a63db470000b029022199758419so2571718pgi.11 for ; Wed, 30 Jun 2021 14:48:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=qhds78sLytz0NDmqseIIdT2KolYiSOyxHvyuNC8rmzc=; b=aEbI1MkbLTQQS1YNxtiBWLfIibs00RAC+i8k4bpp0AJ6FESuKxPJMchdoxI4zVOnp7 2NjErx7+rJbJZ8AYTgjv5k/hEflaKQjfh7SL1W4Iglb8XwtEv0zHD8PTizNfDeP9rqYV 3eV8tLUhWKefqI8GXGcsxzyNHMHPOlyWaSrqyHSJzpa7r61BxhqGghow9ebG1bDXkyph c/9uk9r2tvfFA0Jbhu0VeZyS5roIaPeaIvcdO/tPOFY4EUHXC4OdSJ4KuHPQ+7o1pvUF EK94I10SQBExq8y6kkN0E0Svvo+MvlLCrOCTCFwA4+scxDAHSdxeRLMKKPdWUxccyIHs 7yxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=qhds78sLytz0NDmqseIIdT2KolYiSOyxHvyuNC8rmzc=; b=LKhEx02ogpwQO6ZLmyPAirgzG/HF8c1VU8z6Ocs2ZNEY7XfidTbFFlOuG2+aGHrjpa mNpCHkJSAOY5g531DcaBqLCi0imf5JTlyFiGQy5QsAhblI+gv2R0jQtZndWXameZCMmk JzCFJg4YsMqHf1QHGu3l/qlRkko9HL8exwm6B9tWFmSyt15qr1a8Ya4E+lE1wOS799b8 2QD21QH3uAPwKTrX7Lrixii3goEP70EwmuEv/QqVCZtsC6wCi2We0dCqqQnn6H0kSg8F y+KgDJhmtBKAHMFjcrdwNx3vvqSZ/Swi3aqAyqOpsPiSYPjP4fxZwf+Jyw7k0t0sjdkM 6xXg== X-Gm-Message-State: AOAM532hSc/HS05bAUoR/kchw+Of3sWhaO0YdvxKxJQ99DGRDyx2gpSr Cs3JxZSUtoSXQWHfduSm8NOtCF0d7JD0pu40buR2O5vcBcJ70mZxYIK6tlTLZJXOC+3JrwJ0YQA DkC8t5GP3peKn3bwDfhCX3ha6M6V2MTgZ8RvQermywqVdq9u5pvYBEzQh5w0NUgs= X-Google-Smtp-Source: ABdhPJwq3GA75LkHv4v9akGqohzTIuxtEfhxL1oHRMxrF2J/TM5vNmUmHJVdzeQQtbdsJ5mbdxSuBQJl9a1xKA== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:aa7:9ed1:0:b029:2f2:fb20:bac3 with SMTP id r17-20020aa79ed10000b02902f2fb20bac3mr22739375pfq.6.1625089705578; Wed, 30 Jun 2021 14:48:25 -0700 (PDT) Date: Wed, 30 Jun 2021 21:48:00 +0000 In-Reply-To: <20210630214802.1902448-1-dmatlack@google.com> Message-Id: <20210630214802.1902448-5-dmatlack@google.com> Mime-Version: 1.0 References: <20210630214802.1902448-1-dmatlack@google.com> X-Mailer: git-send-email 2.32.0.93.g670b81a890-goog Subject: [PATCH v2 4/6] KVM: x86/mmu: fast_page_fault support for the TDP MMU From: David Matlack To: kvm@vger.kernel.org Cc: Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Paolo Bonzini , Junaid Shahid , Andrew Jones , Matthew Wilcox , Yu Zhao , David Hildenbrand , Andrew Morton , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Make fast_page_fault interoperate with the TDP MMU by leveraging walk_shadow_page_lockless_{begin,end} to acquire the RCU read lock and introducing a new helper function kvm_tdp_mmu_get_last_sptep_lockless to grab the lowest level sptep. Suggested-by: Ben Gardon Signed-off-by: David Matlack Reported-by: kernel test robot Reported-by: kernel test robot --- arch/x86/kvm/mmu/mmu.c | 55 +++++++++++++++++++++++++++----------- arch/x86/kvm/mmu/tdp_mmu.c | 36 +++++++++++++++++++++++++ arch/x86/kvm/mmu/tdp_mmu.h | 2 ++ 3 files changed, 78 insertions(+), 15 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 88c71a8a55f1..1d410278a4cc 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3105,15 +3105,45 @@ static bool is_access_allowed(u32 fault_err_code, u64 spte) return spte & PT_PRESENT_MASK; } +/* + * Returns the last level spte pointer of the shadow page walk for the given + * gpa, and sets *spte to the spte value. This spte may be non-preset. + * + * If no walk could be performed, returns NULL and *spte does not contain valid + * data. + * + * Constraints: + * - Must be called between walk_shadow_page_lockless_{begin,end}. + * - The returned sptep must not be used after walk_shadow_page_lockless_end. + */ +u64 *get_last_sptep_lockless(struct kvm_vcpu *vcpu, gpa_t gpa, u64 *spte) +{ + struct kvm_shadow_walk_iterator iterator; + u64 old_spte; + u64 *sptep = NULL; + + if (is_tdp_mmu(vcpu->arch.mmu)) + return kvm_tdp_mmu_get_last_sptep_lockless(vcpu, gpa, spte); + + for_each_shadow_entry_lockless(vcpu, gpa, iterator, old_spte) { + sptep = iterator.sptep; + *spte = old_spte; + + if (!is_shadow_present_pte(old_spte)) + break; + } + + return sptep; +} + /* * Returns one of RET_PF_INVALID, RET_PF_FIXED or RET_PF_SPURIOUS. */ static int fast_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code) { - struct kvm_shadow_walk_iterator iterator; - struct kvm_mmu_page *sp; int ret = RET_PF_INVALID; u64 spte = 0ull; + u64 *sptep = NULL; uint retry_count = 0; if (!page_fault_can_be_fast(error_code)) @@ -3122,16 +3152,14 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code) walk_shadow_page_lockless_begin(vcpu); do { + struct kvm_mmu_page *sp; u64 new_spte; - for_each_shadow_entry_lockless(vcpu, gpa, iterator, spte) - if (!is_shadow_present_pte(spte)) - break; - + sptep = get_last_sptep_lockless(vcpu, gpa, &spte); if (!is_shadow_present_pte(spte)) break; - sp = sptep_to_sp(iterator.sptep); + sp = sptep_to_sp(sptep); if (!is_last_spte(spte, sp->role.level)) break; @@ -3189,8 +3217,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code) * since the gfn is not stable for indirect shadow page. See * Documentation/virt/kvm/locking.rst to get more detail. */ - if (fast_pf_fix_direct_spte(vcpu, sp, iterator.sptep, spte, - new_spte)) { + if (fast_pf_fix_direct_spte(vcpu, sp, sptep, spte, new_spte)) { ret = RET_PF_FIXED; break; } @@ -3203,7 +3230,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code) } while (true); - trace_fast_page_fault(vcpu, gpa, error_code, iterator.sptep, spte, ret); + trace_fast_page_fault(vcpu, gpa, error_code, sptep, spte, ret); walk_shadow_page_lockless_end(vcpu); return ret; @@ -3838,11 +3865,9 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code, if (page_fault_handle_page_track(vcpu, error_code, gfn)) return RET_PF_EMULATE; - if (!is_tdp_mmu_fault) { - r = fast_page_fault(vcpu, gpa, error_code); - if (r != RET_PF_INVALID) - return r; - } + r = fast_page_fault(vcpu, gpa, error_code); + if (r != RET_PF_INVALID) + return r; r = mmu_topup_memory_caches(vcpu, false); if (r) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index c6fa8d00bf9f..2c9e0ed71fa0 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -527,6 +527,10 @@ static inline bool tdp_mmu_set_spte_atomic_no_dirty_log(struct kvm *kvm, if (is_removed_spte(iter->old_spte)) return false; + /* + * TDP MMU sptes can also be concurrently cmpxchg'd in + * fast_pf_fix_direct_spte as part of fast_page_fault. + */ if (cmpxchg64(rcu_dereference(iter->sptep), iter->old_spte, new_spte) != iter->old_spte) return false; @@ -1546,3 +1550,35 @@ int kvm_tdp_mmu_get_walk_lockless(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, return leaf; } + +/* + * Must be called between kvm_tdp_mmu_walk_shadow_page_lockless_{begin,end}. + * + * The returned sptep must not be used after + * kvm_tdp_mmu_walk_shadow_page_lockless_end. + */ +u64 *kvm_tdp_mmu_get_last_sptep_lockless(struct kvm_vcpu *vcpu, u64 addr, + u64 *spte) +{ + struct tdp_iter iter; + struct kvm_mmu *mmu = vcpu->arch.mmu; + gfn_t gfn = addr >> PAGE_SHIFT; + tdp_ptep_t sptep = NULL; + + tdp_mmu_for_each_pte(iter, mmu, gfn, gfn + 1) { + *spte = iter.old_spte; + sptep = iter.sptep; + } + + if (sptep) + /* + * Perform the rcu dereference here since we are passing the + * sptep up to the generic MMU code which does not know the + * synchronization details of the TDP MMU. This is safe as long + * as the caller obeys the contract that the sptep is not used + * after kvm_tdp_mmu_walk_shadow_page_lockless_end. + */ + return rcu_dereference(sptep); + + return NULL; +} diff --git a/arch/x86/kvm/mmu/tdp_mmu.h b/arch/x86/kvm/mmu/tdp_mmu.h index e9dde5f9c0ef..508a23bdf7da 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.h +++ b/arch/x86/kvm/mmu/tdp_mmu.h @@ -81,6 +81,8 @@ void kvm_tdp_mmu_walk_lockless_begin(void); void kvm_tdp_mmu_walk_lockless_end(void); int kvm_tdp_mmu_get_walk_lockless(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, int *root_level); +u64 *kvm_tdp_mmu_get_last_sptep_lockless(struct kvm_vcpu *vcpu, u64 addr, + u64 *spte); #ifdef CONFIG_X86_64 bool kvm_mmu_init_tdp_mmu(struct kvm *kvm); From patchwork Wed Jun 30 21:48:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12352841 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.4 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D3F6C11F65 for ; Wed, 30 Jun 2021 21:48:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 16DD0611AC for ; Wed, 30 Jun 2021 21:48:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232559AbhF3VvF (ORCPT ); Wed, 30 Jun 2021 17:51:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232689AbhF3Vu6 (ORCPT ); Wed, 30 Jun 2021 17:50:58 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 22D7EC0617A8 for ; Wed, 30 Jun 2021 14:48:28 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id c20-20020a17090ab294b029016fccb9582dso2072924pjr.7 for ; Wed, 30 Jun 2021 14:48:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=YypJ74lmVV39roAzI2i/DT39zCOCRxagBpvdMWnNzDc=; b=dXfuUM5fUb7MEtYMYaWBLLu4qCZpdBqagSYXoYay0Te9r9EjgMBTVuyxQ+Fa/RL5kL 3dOAn/0fzSIdgrUpxHiFxRB1wd+hxkiI9XGvfbYz8qVqhIQvCsSkDhNCzRobQFPwgxxl 6JFdrLlwcT5USW1BVI90NLuftpWxS3B3YgSN2RprJg73aOMWVG5Rlh9SsxxnyBjZquxz DneU9VxWZuOQJ3K2qvyx5t0tYpPhPUpFfrP03g7rpFAnqwXBhMhLSFVAAtlNpSULtbUo DL4iH1hMzetLcmNPncAyWlRvUsmsDxKy2UXDRggZCVy+Rq+NlSPkU09mvNAVD2UsU6vx DEcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=YypJ74lmVV39roAzI2i/DT39zCOCRxagBpvdMWnNzDc=; b=pNVZO4naFTFhr37/IaFO88jHdMtFdjIELK8BYwlXW6o+e8bM0ByZ3VdX+rgMDyt3av LxfQ0+9wOjXeWBW1eKET8mCBpbvksVLk1fEXwIFAmQoO++FZQBVnkh8mTIaJimf+vdle zBUD2gKwa0LSg2bFhOBnifSU26FfW4ylY2FnZsLHy50w4CFD2r/wuAcG2NQHVUaoAQLb E73OYlYAavOUQsLXKMdsnLzZrbjO19Wqe2v8jQtwGAykXj4PzrFv1Hsd4vBiN8lUgDkG aSC79knTHEvVhfoTp1qoHH4yms27WkB0QwOe7HjFaW3DhLcwLPQFJ0d7u9BF2v41j/eE BC0A== X-Gm-Message-State: AOAM532Pk1duRisg2inbhND6g3JTxk/z58r0ZD0o10QM7A2S2sjBCact zacNXWBSn4uOJC1t389wPem2Gwex+5F/SaQnz3/IgCVl4tk6T//IveGlJqxxMc3cBexrx2Cn0xi oMYsE1mohkMe+/y5wEO7100MOy4Lq64rv2/yVc7PRUFBwAmt9ZprYM8sK1J6oLqw= X-Google-Smtp-Source: ABdhPJzTAO5PjVdslNaUsLkX1FxHllgR8bDnlHNWt5o46T/rE8p/IyghmugnqkWDSrZfyjQLqWqiXIOcUS0UTQ== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:90a:1da3:: with SMTP id v32mr6518035pjv.192.1625089707340; Wed, 30 Jun 2021 14:48:27 -0700 (PDT) Date: Wed, 30 Jun 2021 21:48:01 +0000 In-Reply-To: <20210630214802.1902448-1-dmatlack@google.com> Message-Id: <20210630214802.1902448-6-dmatlack@google.com> Mime-Version: 1.0 References: <20210630214802.1902448-1-dmatlack@google.com> X-Mailer: git-send-email 2.32.0.93.g670b81a890-goog Subject: [PATCH v2 5/6] KVM: selftests: Fix missing break in dirty_log_perf_test arg parsing From: David Matlack To: kvm@vger.kernel.org Cc: Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Paolo Bonzini , Junaid Shahid , Andrew Jones , Matthew Wilcox , Yu Zhao , David Hildenbrand , Andrew Morton , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org There is a missing break statement which causes a fallthrough to the next statement where optarg will be null and a segmentation fault will be generated. Fixes: 9e965bb75aae ("KVM: selftests: Add backing src parameter to dirty_log_perf_test") Reviewed-by: Ben Gardon Signed-off-by: David Matlack --- tools/testing/selftests/kvm/dirty_log_perf_test.c | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/testing/selftests/kvm/dirty_log_perf_test.c b/tools/testing/selftests/kvm/dirty_log_perf_test.c index 04a2641261be..80cbd3a748c0 100644 --- a/tools/testing/selftests/kvm/dirty_log_perf_test.c +++ b/tools/testing/selftests/kvm/dirty_log_perf_test.c @@ -312,6 +312,7 @@ int main(int argc, char *argv[]) break; case 'o': p.partition_vcpu_memory_access = false; + break; case 's': p.backing_src = parse_backing_src_type(optarg); break; From patchwork Wed Jun 30 21:48:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12352843 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.4 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB0DBC11F68 for ; Wed, 30 Jun 2021 21:48:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C4FCE61474 for ; Wed, 30 Jun 2021 21:48:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232604AbhF3VvG (ORCPT ); Wed, 30 Jun 2021 17:51:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39644 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232748AbhF3Vu7 (ORCPT ); Wed, 30 Jun 2021 17:50:59 -0400 Received: from mail-qk1-x749.google.com (mail-qk1-x749.google.com [IPv6:2607:f8b0:4864:20::749]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ED6CAC061756 for ; Wed, 30 Jun 2021 14:48:29 -0700 (PDT) Received: by mail-qk1-x749.google.com with SMTP id n195-20020a3740cc0000b02903b2ccb7bbe6so2683529qka.20 for ; Wed, 30 Jun 2021 14:48:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=+uiwqEz6p8XQTY9ZsTXgjKWNLycVEoinELUzv5S6vDA=; b=jjg1kjgPluwrJVeE1wLEt6avxIIZZb2VkKJViCPN5AMr1Pon+SkvZno8Agy5776Rj9 Vrt2Bl8wVHODMwO6RSE7RoMyIDlzFx0d4QdsR6Y6zkbYunKWiN0uyDy5nQJnMrhiKWKC IGWAuDdPM5x/9SddyB8AgSB7JJj8vz59G+HyKgO00z/3hs5ENFU6NWQK9QeFzXRVE76T fnoQQcmtG9tdNevd22s+8WNSFTTgQY4xSM340iGiJ6L2f8IyvSeHfsOe1IiH99MZiCyb 9HJXdfjqf9fKUPtyhpTjXgiNCCztn93wu6lJHnkKlLxI7SD7GUMaNvwCy4YZunR1X/Zn QFTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=+uiwqEz6p8XQTY9ZsTXgjKWNLycVEoinELUzv5S6vDA=; b=OcIbiyVHLZv2e88YkR2nsnNnUYrjHxil37ObwWRw9jL4TsUbnNf04MnmTWGh6n5b6C rCy+Mgcw2voBLyKKTYLVc+2KUNN28mCwDyH+jtuw/bRji6pAUYric/ra7KGkaXtSW71y baxGq9byMC/6HLBWEgwXMjrif2BxQNVnPAxaUkJAO36v1y3orBXgOSBMukuhxIL3CVbd /BbSKieQ53szV6xARcGbvHINnjrGQgNnrHkhcagvfpU8StpMnUU4fKBDcGD2VKZirmxQ yKt1hEaYLVQoFoG8tja+1XtdHRPTcmykbDS/6F+qMnHBVFhFWpLD3M2l+IeT6uNasaHF HZNw== X-Gm-Message-State: AOAM5338KNHGc3TCEXd5pAai48ZTSF1MghouknTWdl9QzOQTIpoIsxHZ 6YNibYKI69Vw6PlMrK17UTJcRo5HsA9DpefFBE8Y+FSVsBvt0I6GrH067H6myx8MwCksbIV40nH NuPUGiDWKLzh1WJliGbvV6rC5aTG0Uj/rVQzw/UFAI70wTy/+ChqVgGs0a8ZdLZQ= X-Google-Smtp-Source: ABdhPJy3p+PnDzIJcDAiu8PHnHxGEp++zQQx+BrWmL9jqHqprsxl2GqJ2ZsTw7qWFZI7OQ6ybnrtHLTLgGsJ9Q== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a05:6214:8c4:: with SMTP id da4mr15901615qvb.3.1625089708935; Wed, 30 Jun 2021 14:48:28 -0700 (PDT) Date: Wed, 30 Jun 2021 21:48:02 +0000 In-Reply-To: <20210630214802.1902448-1-dmatlack@google.com> Message-Id: <20210630214802.1902448-7-dmatlack@google.com> Mime-Version: 1.0 References: <20210630214802.1902448-1-dmatlack@google.com> X-Mailer: git-send-email 2.32.0.93.g670b81a890-goog Subject: [PATCH v2 6/6] KVM: selftests: Introduce access_tracking_perf_test From: David Matlack To: kvm@vger.kernel.org Cc: Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Paolo Bonzini , Junaid Shahid , Andrew Jones , Matthew Wilcox , Yu Zhao , David Hildenbrand , Andrew Morton , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This test measures the performance effects of KVM's access tracking. Access tracking is driven by the MMU notifiers test_young, clear_young, and clear_flush_young. These notifiers do not have a direct userspace API, however the clear_young notifier can be triggered by marking a pages as idle in /sys/kernel/mm/page_idle/bitmap. This test leverages that mechanism to enable access tracking on guest memory. To measure performance this test runs a VM with a configurable number of vCPUs that each touch every page in disjoint regions of memory. Performance is measured in the time it takes all vCPUs to finish touching their predefined region. Example invocation: $ ./access_tracking_perf_test -v 8 Testing guest mode: PA-bits:ANY, VA-bits:48, 4K pages guest physical test memory offset: 0xffdfffff000 Populating memory : 1.337752570s Writing to populated memory : 0.010177640s Reading from populated memory : 0.009548239s Mark memory idle : 23.973131748s Writing to idle memory : 0.063584496s Mark memory idle : 24.924652964s Reading from idle memory : 0.062042814s Breaking down the results: * "Populating memory": The time it takes for all vCPUs to perform the first write to every page in their region. * "Writing to populated memory" / "Reading from populated memory": The time it takes for all vCPUs to write and read to every page in their region after it has been populated. This serves as a control for the later results. * "Mark memory idle": The time it takes for every vCPU to mark every page in their region as idle through page_idle. * "Writing to idle memory" / "Reading from idle memory": The time it takes for all vCPUs to write and read to every page in their region after it has been marked idle. This test should be portable across architectures but it is only enabled for x86_64 since that's all I have tested. Reviewed-by: Ben Gardon Signed-off-by: David Matlack --- tools/testing/selftests/kvm/.gitignore | 1 + tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/access_tracking_perf_test.c | 429 ++++++++++++++++++ 3 files changed, 431 insertions(+) create mode 100644 tools/testing/selftests/kvm/access_tracking_perf_test.c diff --git a/tools/testing/selftests/kvm/.gitignore b/tools/testing/selftests/kvm/.gitignore index d5bc9bf3b528..e16cbe7b9ce3 100644 --- a/tools/testing/selftests/kvm/.gitignore +++ b/tools/testing/selftests/kvm/.gitignore @@ -38,6 +38,7 @@ /x86_64/xen_vmcall_test /x86_64/xss_msr_test /x86_64/vmx_pmu_msrs_test +/access_tracking_perf_test /demand_paging_test /dirty_log_test /dirty_log_perf_test diff --git a/tools/testing/selftests/kvm/Makefile b/tools/testing/selftests/kvm/Makefile index bc65c57ae40d..e1ce404f3c67 100644 --- a/tools/testing/selftests/kvm/Makefile +++ b/tools/testing/selftests/kvm/Makefile @@ -71,6 +71,7 @@ TEST_GEN_PROGS_x86_64 += x86_64/tsc_msrs_test TEST_GEN_PROGS_x86_64 += x86_64/vmx_pmu_msrs_test TEST_GEN_PROGS_x86_64 += x86_64/xen_shinfo_test TEST_GEN_PROGS_x86_64 += x86_64/xen_vmcall_test +TEST_GEN_PROGS_x86_64 += access_tracking_perf_test TEST_GEN_PROGS_x86_64 += demand_paging_test TEST_GEN_PROGS_x86_64 += dirty_log_test TEST_GEN_PROGS_x86_64 += dirty_log_perf_test diff --git a/tools/testing/selftests/kvm/access_tracking_perf_test.c b/tools/testing/selftests/kvm/access_tracking_perf_test.c new file mode 100644 index 000000000000..e2baa187a21e --- /dev/null +++ b/tools/testing/selftests/kvm/access_tracking_perf_test.c @@ -0,0 +1,429 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * access_tracking_perf_test + * + * Copyright (C) 2021, Google, Inc. + * + * This test measures the performance effects of KVM's access tracking. + * Access tracking is driven by the MMU notifiers test_young, clear_young, and + * clear_flush_young. These notifiers do not have a direct userspace API, + * however the clear_young notifier can be triggered by marking a pages as idle + * in /sys/kernel/mm/page_idle/bitmap. This test leverages that mechanism to + * enable access tracking on guest memory. + * + * To measure performance this test runs a VM with a configurable number of + * vCPUs that each touch every page in disjoint regions of memory. Performance + * is measured in the time it takes all vCPUs to finish touching their + * predefined region. + * + * Note that a deterministic correctness test of access tracking is not possible + * by using page_idle as it exists today. This is for a few reasons: + * + * 1. page_idle only issues clear_young notifiers, which lack a TLB flush. This + * means subsequent guest accesses are not guaranteed to see page table + * updates made by KVM until some time in the future. + * + * 2. page_idle only operates on LRU pages. Newly allocated pages are not + * immediately allocated to LRU lists. Instead they are held in a "pagevec", + * which is drained to LRU lists some time in the future. There is no + * userspace API to force this drain to occur. + * + * These limitations are worked around in this test by using a large enough + * region of memory for each vCPU such that the number of translations cached in + * the TLB and the number of pages held in pagevecs are a small fraction of the + * overall workload. And if either of those conditions are not true this test + * will fail rather than silently passing. + */ +#include +#include +#include +#include +#include +#include + +#include "kvm_util.h" +#include "test_util.h" +#include "perf_test_util.h" +#include "guest_modes.h" + +/* Global variable used to synchronize all of the vCPU threads. */ +static int iteration = -1; + +/* Defines what vCPU threads should do during a given iteration. */ +static enum { + /* Run the vCPU to access all its memory. */ + ITERATION_ACCESS_MEMORY, + /* Mark the vCPU's memory idle in page_idle. */ + ITERATION_MARK_IDLE, +} iteration_work; + +/* Set to true when vCPU threads should exit. */ +static bool done; + +/* The iteration that was last completed by each vCPU. */ +static int vcpu_last_completed_iteration[KVM_MAX_VCPUS]; + +/* Whether to overlap the regions of memory vCPUs access. */ +static bool overlap_memory_access; + +struct test_params { + /* The backing source for the region of memory. */ + enum vm_mem_backing_src_type backing_src; + + /* The amount of memory to allocate for each vCPU. */ + uint64_t vcpu_memory_bytes; + + /* The number of vCPUs to create in the VM. */ + int vcpus; +}; + +static uint64_t pread_uint64(int fd, const char *filename, uint64_t index) +{ + uint64_t value; + off_t offset = index * sizeof(value); + + TEST_ASSERT(pread(fd, &value, sizeof(value), offset) == sizeof(value), + "pread from %s offset 0x%" PRIx64 " failed!", + filename, offset); + + return value; + +} + +#define PAGEMAP_PRESENT (1ULL << 63) +#define PAGEMAP_PFN_MASK ((1ULL << 55) - 1) + +static uint64_t lookup_pfn(int pagemap_fd, struct kvm_vm *vm, uint64_t gva) +{ + uint64_t hva = (uint64_t) addr_gva2hva(vm, gva); + uint64_t entry; + uint64_t pfn; + + entry = pread_uint64(pagemap_fd, "pagemap", hva / getpagesize()); + if (!(entry & PAGEMAP_PRESENT)) + return 0; + + pfn = entry & PAGEMAP_PFN_MASK; + if (!pfn) { + print_skip("Looking up PFNs requires CAP_SYS_ADMIN"); + exit(KSFT_SKIP); + } + + return pfn; +} + +static bool is_page_idle(int page_idle_fd, uint64_t pfn) +{ + uint64_t bits = pread_uint64(page_idle_fd, "page_idle", pfn / 64); + + return !!((bits >> (pfn % 64)) & 1); +} + +static void mark_page_idle(int page_idle_fd, uint64_t pfn) +{ + uint64_t bits = 1ULL << (pfn % 64); + + TEST_ASSERT(pwrite(page_idle_fd, &bits, 8, 8 * (pfn / 64)) == 8, + "Set page_idle bits for PFN 0x%" PRIx64, pfn); +} + +static void mark_vcpu_memory_idle(struct kvm_vm *vm, int vcpu_id) +{ + uint64_t base_gva = perf_test_args.vcpu_args[vcpu_id].gva; + uint64_t pages = perf_test_args.vcpu_args[vcpu_id].pages; + uint64_t page; + uint64_t still_idle = 0; + uint64_t no_pfn = 0; + int page_idle_fd; + int pagemap_fd; + + /* If vCPUs are using an overlapping region, let vCPU 0 mark it idle. */ + if (overlap_memory_access && vcpu_id) + return; + + page_idle_fd = open("/sys/kernel/mm/page_idle/bitmap", O_RDWR); + TEST_ASSERT(page_idle_fd > 0, "Failed to open page_idle."); + + pagemap_fd = open("/proc/self/pagemap", O_RDONLY); + TEST_ASSERT(pagemap_fd > 0, "Failed to open pagemap."); + + for (page = 0; page < pages; page++) { + uint64_t gva = base_gva + page * perf_test_args.guest_page_size; + uint64_t pfn = lookup_pfn(pagemap_fd, vm, gva); + + if (!pfn) { + no_pfn++; + continue; + } + + if (is_page_idle(page_idle_fd, pfn)) { + still_idle++; + continue; + } + + mark_page_idle(page_idle_fd, pfn); + } + + /* + * Assumption: Less than 1% of pages are going to be swapped out from + * under us during this test. + */ + TEST_ASSERT(no_pfn < pages / 100, + "vCPU %d: No PFN for %" PRIu64 " out of %" PRIu64 " pages.", + vcpu_id, no_pfn, pages); + + /* + * Test that at least 90% of memory has been marked idle (the rest might + * not be marked idle because the pages have not yet made it to an LRU + * list or the translations are still cached in the TLB). 90% is + * arbitrary; high enough that we ensure most memory access went through + * access tracking but low enough as to not make the test too brittle + * over time and across architectures. + */ + TEST_ASSERT(still_idle < pages / 10, + "vCPU%d: Too many pages still idle (%"PRIu64 " out of %" + PRIu64 ").\n", + vcpu_id, still_idle, pages); + + close(page_idle_fd); + close(pagemap_fd); +} + +static void assert_ucall(struct kvm_vm *vm, uint32_t vcpu_id, + uint64_t expected_ucall) +{ + struct ucall uc; + uint64_t actual_ucall = get_ucall(vm, vcpu_id, &uc); + + TEST_ASSERT(expected_ucall == actual_ucall, + "Guest exited unexpectedly (expected ucall %" PRIu64 + ", got %" PRIu64 ")", + expected_ucall, actual_ucall); +} + +static bool spin_wait_for_next_iteration(int *current_iteration) +{ + int last_iteration = *current_iteration; + + do { + if (READ_ONCE(done)) + return false; + + *current_iteration = READ_ONCE(iteration); + } while (last_iteration == *current_iteration); + + return true; +} + +static void *vcpu_thread_main(void *arg) +{ + struct perf_test_vcpu_args *vcpu_args = arg; + struct kvm_vm *vm = perf_test_args.vm; + int vcpu_id = vcpu_args->vcpu_id; + int current_iteration = -1; + + vcpu_args_set(vm, vcpu_id, 1, vcpu_id); + + while (spin_wait_for_next_iteration(¤t_iteration)) { + switch (READ_ONCE(iteration_work)) { + case ITERATION_ACCESS_MEMORY: + vcpu_run(vm, vcpu_id); + assert_ucall(vm, vcpu_id, UCALL_SYNC); + break; + case ITERATION_MARK_IDLE: + mark_vcpu_memory_idle(vm, vcpu_id); + break; + }; + + vcpu_last_completed_iteration[vcpu_id] = current_iteration; + } + + return NULL; +} + +static void spin_wait_for_vcpu(int vcpu_id, int target_iteration) +{ + while (READ_ONCE(vcpu_last_completed_iteration[vcpu_id]) != + target_iteration) { + continue; + } +} + +/* The type of memory accesses to perform in the VM. */ +enum access_type { + ACCESS_READ, + ACCESS_WRITE, +}; + +static void run_iteration(struct kvm_vm *vm, int vcpus, const char *description) +{ + struct timespec ts_start; + struct timespec ts_elapsed; + int next_iteration; + int vcpu_id; + + /* Kick off the vCPUs by incrementing iteration. */ + next_iteration = ++iteration; + + clock_gettime(CLOCK_MONOTONIC, &ts_start); + + /* Wait for all vCPUs to finish the iteration. */ + for (vcpu_id = 0; vcpu_id < vcpus; vcpu_id++) + spin_wait_for_vcpu(vcpu_id, next_iteration); + + ts_elapsed = timespec_elapsed(ts_start); + pr_info("%-30s: %ld.%09lds\n", + description, ts_elapsed.tv_sec, ts_elapsed.tv_nsec); +} + +static void access_memory(struct kvm_vm *vm, int vcpus, enum access_type access, + const char *description) +{ + perf_test_args.wr_fract = (access == ACCESS_READ) ? INT_MAX : 1; + sync_global_to_guest(vm, perf_test_args); + iteration_work = ITERATION_ACCESS_MEMORY; + run_iteration(vm, vcpus, description); +} + +static void mark_memory_idle(struct kvm_vm *vm, int vcpus) +{ + /* + * Even though this parallelizes the work across vCPUs, this is still a + * very slow operation because page_idle forces the test to mark one pfn + * at a time and the clear_young notifier serializes on the KVM MMU + * lock. + */ + pr_debug("Marking VM memory idle (slow)...\n"); + iteration_work = ITERATION_MARK_IDLE; + run_iteration(vm, vcpus, "Mark memory idle"); +} + +static pthread_t *create_vcpu_threads(int vcpus) +{ + pthread_t *vcpu_threads; + int i; + + vcpu_threads = malloc(vcpus * sizeof(vcpu_threads[0])); + TEST_ASSERT(vcpu_threads, "Failed to allocate vcpu_threads."); + + for (i = 0; i < vcpus; i++) { + vcpu_last_completed_iteration[i] = iteration; + pthread_create(&vcpu_threads[i], NULL, vcpu_thread_main, + &perf_test_args.vcpu_args[i]); + } + + return vcpu_threads; +} + +static void terminate_vcpu_threads(pthread_t *vcpu_threads, int vcpus) +{ + int i; + + /* Set done to signal the vCPU threads to exit */ + done = true; + + for (i = 0; i < vcpus; i++) + pthread_join(vcpu_threads[i], NULL); +} + +static void run_test(enum vm_guest_mode mode, void *arg) +{ + struct test_params *params = arg; + struct kvm_vm *vm; + pthread_t *vcpu_threads; + int vcpus = params->vcpus; + + vm = perf_test_create_vm(mode, vcpus, params->vcpu_memory_bytes, + params->backing_src); + + perf_test_setup_vcpus(vm, vcpus, params->vcpu_memory_bytes, + !overlap_memory_access); + + vcpu_threads = create_vcpu_threads(vcpus); + + pr_info("\n"); + access_memory(vm, vcpus, ACCESS_WRITE, "Populating memory"); + + /* As a control, read and write to the populated memory first. */ + access_memory(vm, vcpus, ACCESS_WRITE, "Writing to populated memory"); + access_memory(vm, vcpus, ACCESS_READ, "Reading from populated memory"); + + /* Repeat on memory that has been marked as idle. */ + mark_memory_idle(vm, vcpus); + access_memory(vm, vcpus, ACCESS_WRITE, "Writing to idle memory"); + mark_memory_idle(vm, vcpus); + access_memory(vm, vcpus, ACCESS_READ, "Reading from idle memory"); + + terminate_vcpu_threads(vcpu_threads, vcpus); + free(vcpu_threads); + perf_test_destroy_vm(vm); +} + +static void help(char *name) +{ + puts(""); + printf("usage: %s [-h] [-m mode] [-b vcpu_bytes] [-v vcpus] [-o] [-s mem_type]\n", + name); + puts(""); + printf(" -h: Display this help message."); + guest_modes_help(); + printf(" -b: specify the size of the memory region which should be\n" + " dirtied by each vCPU. e.g. 10M or 3G.\n" + " (default: 1G)\n"); + printf(" -v: specify the number of vCPUs to run.\n"); + printf(" -o: Overlap guest memory accesses instead of partitioning\n" + " them into a separate region of memory for each vCPU.\n"); + printf(" -s: specify the type of memory that should be used to\n" + " back the guest data region.\n\n"); + backing_src_help(); + puts(""); + exit(0); +} + +int main(int argc, char *argv[]) +{ + struct test_params params = { + .backing_src = VM_MEM_SRC_ANONYMOUS, + .vcpu_memory_bytes = DEFAULT_PER_VCPU_MEM_SIZE, + .vcpus = 1, + }; + int page_idle_fd; + int opt; + + guest_modes_append_default(); + + while ((opt = getopt(argc, argv, "hm:b:v:os:")) != -1) { + switch (opt) { + case 'm': + guest_modes_cmdline(optarg); + break; + case 'b': + params.vcpu_memory_bytes = parse_size(optarg); + break; + case 'v': + params.vcpus = atoi(optarg); + break; + case 'o': + overlap_memory_access = true; + break; + case 's': + params.backing_src = parse_backing_src_type(optarg); + break; + case 'h': + default: + help(argv[0]); + break; + } + } + + page_idle_fd = open("/sys/kernel/mm/page_idle/bitmap", O_RDWR); + if (page_idle_fd < 0) { + print_skip("CONFIG_IDLE_PAGE_TRACKING is not enabled"); + exit(KSFT_SKIP); + } + close(page_idle_fd); + + for_each_guest_mode(run_test, ¶ms); + + return 0; +}