From patchwork Fri Jul 30 22:04:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12412021 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 509B6C4338F for ; Fri, 30 Jul 2021 22:05:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3AE0460F94 for ; Fri, 30 Jul 2021 22:05:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233028AbhG3WFI (ORCPT ); Fri, 30 Jul 2021 18:05:08 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:32585 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229709AbhG3WFH (ORCPT ); Fri, 30 Jul 2021 18:05:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1627682701; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j/8ZgOgV9FABxcwPWKwd6pFZTEbb8PsS8jPvc0Wk6tg=; b=fbHcs5jva1KPzejhf47LlSW+Za7QPEydHHRqQeVcrQt301tqS6UHYDjDDctQ5yCyNIYM9k 2r6ZG8zcX1j+YqiyFn60XdeOqtvV0E8+UihbXNfMg4SAkw+ksnvAUO5YmU1kGHXgUqFr90 dKGq5E2bQOE9R0uDxL4Ka845tFdG7ZU= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-543-u1iXp_e4NwCWI_5JL64ATg-1; Fri, 30 Jul 2021 18:05:00 -0400 X-MC-Unique: u1iXp_e4NwCWI_5JL64ATg-1 Received: by mail-qt1-f198.google.com with SMTP id z16-20020ac87cb00000b029025bf786be09so974388qtv.20 for ; Fri, 30 Jul 2021 15:04:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=j/8ZgOgV9FABxcwPWKwd6pFZTEbb8PsS8jPvc0Wk6tg=; b=XcBvDpMuKjNTOKGgzbVMN07Q5nn9JW0NQm5YOk9TQ7NQaDybsBQizBFnvXX5vTXkCi XU7toaQQqYIAXGex/0wj3De96RB3aElNgzqHGF74xkB7/C0055ZtoyXtoCg+WOKNgopb I7G175kszViIssQGxq5SN/tmz0rGyy6TMD+M3fY/Eai4dIEJad3qpjslVZ3BO2g5jEv+ 2Rb8FL2XSVRE3aUnIkp4+pJRCglAI1Va6GF6hKbhCPlRMHpIUWLYwcRDM73uMC3W1XUV Zrhpp01LPEGNTUtJMIpZnqXYWMFMeNYRo2S7wq8Sj/emUU0EE8diR4tyNy4eCLTbbVDz FH5g== X-Gm-Message-State: AOAM530XHdkHNJFJI2XrIG2hanpj78dAeLKMY73QOZYJ6fNMy1IOyOyL YQMeA9M8OeiGPKYjtmN/OkTnjeiBYMLuFJrSoi048lYIpR4onA9VdnTgoSCVXlLkbWiaFZXmEL2 kSoSQXcGf+GKYbEAVZtYbgKy3zePMEDbqpUZb/Tt8rSRv5JfF85d1SWAuguYRxg== X-Received: by 2002:a05:6214:d6e:: with SMTP id 14mr5095106qvs.53.1627682698999; Fri, 30 Jul 2021 15:04:58 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxMTQNHX7PoV1dhRPzDF85NElhalv3f5AD26oUEYimWzW4mHcCPuq4K8RhKwVszC874lpIkBw== X-Received: by 2002:a05:6214:d6e:: with SMTP id 14mr5095079qvs.53.1627682698761; Fri, 30 Jul 2021 15:04:58 -0700 (PDT) Received: from t490s.. (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id l12sm1199651qtx.45.2021.07.30.15.04.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Jul 2021 15:04:58 -0700 (PDT) From: Peter Xu To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Vitaly Kuznetsov , Sean Christopherson , peterx@redhat.com, Maxim Levitsky , Paolo Bonzini Subject: [PATCH v3 1/7] KVM: Allow to have arch-specific per-vm debugfs files Date: Fri, 30 Jul 2021 18:04:49 -0400 Message-Id: <20210730220455.26054-2-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210730220455.26054-1-peterx@redhat.com> References: <20210730220455.26054-1-peterx@redhat.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Allow archs to create arch-specific nodes under kvm->debugfs_dentry directory besides the stats fields. The new interface kvm_arch_create_vm_debugfs() is defined but not yet used. It's called after kvm->debugfs_dentry is created, so it can be referenced directly in kvm_arch_create_vm_debugfs(). Arch should define their own versions when they want to create extra debugfs nodes. Signed-off-by: Peter Xu --- include/linux/kvm_host.h | 1 + virt/kvm/kvm_main.c | 20 +++++++++++++++++++- 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 9d6b4ad407b8..a3ec3271c4c8 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -1067,6 +1067,7 @@ bool kvm_arch_dy_runnable(struct kvm_vcpu *vcpu); bool kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu); int kvm_arch_post_init_vm(struct kvm *kvm); void kvm_arch_pre_destroy_vm(struct kvm *kvm); +int kvm_arch_create_vm_debugfs(struct kvm *kvm); #ifndef __KVM_HAVE_ARCH_VM_ALLOC /* diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index a96cbe24c688..327f8fae80a5 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -915,7 +915,7 @@ static int kvm_create_vm_debugfs(struct kvm *kvm, int fd) char dir_name[ITOA_MAX_LEN * 2]; struct kvm_stat_data *stat_data; const struct _kvm_stats_desc *pdesc; - int i; + int i, ret; int kvm_debugfs_num_entries = kvm_vm_stats_header.num_desc + kvm_vcpu_stats_header.num_desc; @@ -960,6 +960,13 @@ static int kvm_create_vm_debugfs(struct kvm *kvm, int fd) kvm->debugfs_dentry, stat_data, &stat_fops_per_vm); } + + ret = kvm_arch_create_vm_debugfs(kvm); + if (ret) { + kvm_destroy_vm_debugfs(kvm); + return i; + } + return 0; } @@ -980,6 +987,17 @@ void __weak kvm_arch_pre_destroy_vm(struct kvm *kvm) { } +/* + * Called after per-vm debugfs created. When called kvm->debugfs_dentry should + * be setup already, so we can create arch-specific debugfs entries under it. + * Cleanup should be automatic done in kvm_destroy_vm_debugfs() recursively, so + * a per-arch destroy interface is not needed. + */ +int __weak kvm_arch_create_vm_debugfs(struct kvm *kvm) +{ + return 0; +} + static struct kvm *kvm_create_vm(unsigned long type) { struct kvm *kvm = kvm_arch_alloc_vm(); From patchwork Fri Jul 30 22:04:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12412023 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B981C4338F for ; Fri, 30 Jul 2021 22:05:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 72EE561050 for ; Fri, 30 Jul 2021 22:05:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233112AbhG3WFL (ORCPT ); Fri, 30 Jul 2021 18:05:11 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:46348 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232672AbhG3WFI (ORCPT ); Fri, 30 Jul 2021 18:05:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1627682702; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NSBJ7FBNCe1tAX/jT7A68XhwxPR3lX/uR04jjJ3Q1pU=; b=OwyGDBTT/M7qGmLVGiI6Kkxf8SXrE+XwX21M/fklZHrOXj8UnJCRxkkNJzNF2mXTQnjTju YnD9eizbujfomM4gRGRUIceC1/QR4nEq8Uyr3iqpGP+R1JyVntUTgg35Zb9wO+EnryriDB n07is+roabaZ6xXkX0v3kQqpIRLEnCQ= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-574-iDbodSd0MpervcqRp5oMWw-1; Fri, 30 Jul 2021 18:05:01 -0400 X-MC-Unique: iDbodSd0MpervcqRp5oMWw-1 Received: by mail-qt1-f197.google.com with SMTP id l7-20020ac848c70000b0290252173fe79cso5158500qtr.2 for ; Fri, 30 Jul 2021 15:05:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NSBJ7FBNCe1tAX/jT7A68XhwxPR3lX/uR04jjJ3Q1pU=; b=PU+wazW4wUumtSR7s5eDRcIEqdfDubIs+Ds5gwYUUBJFmRB7jwNDrIH4rVYJkR6M0p WxkYMSBeikxkrfx6oOs1R9Wub02b+cuSBvP9liGCwT6pDY1Ndozb/XhYzCB/F6zIIFnl c6M9RvWkhFPQ2s6TxlKOGBzdjfXplGJTB/BH072j3jt4NhmOT0MEwewZbZxfwc9WSKNq TyNhnKr0EJFh3AqKUEsi9YMjN0D8HKUpnr+nPyc2ukhLhsptvcEyJ0TaSmjTmmsPKIQA e2CEDuSzkxwfjvuwX4Jdj63kCZk95EhhmNZaptHZIdFavftHXVmFGsgMFLPKG1+J6oTz uzwg== X-Gm-Message-State: AOAM5335P0weYM792wdCF0/KE4nRA0J8moCxLxn4LwlKOjSJcsSEZTKt UXknwBjHJY3l0IvG7EnvDgzHVkVEmT2oRs9pefk7EUsPyXZ74SHVcMBgFEw5ZIYCRR0Sg1FCCks 0OLLgdGu5qAsRVVGZTY7UiDS4F/4g1emj+cA/nssugROpau/LKGv0huxMB4l3oA== X-Received: by 2002:a05:620a:1423:: with SMTP id k3mr4382996qkj.311.1627682700541; Fri, 30 Jul 2021 15:05:00 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwsE0R7naZuNPnJJC2QXc2HOhX6dmnUYtwJ3AeyZw/7H252RCpv0ekq01gRagRLcYrLB58hlA== X-Received: by 2002:a05:620a:1423:: with SMTP id k3mr4382968qkj.311.1627682700254; Fri, 30 Jul 2021 15:05:00 -0700 (PDT) Received: from t490s.. (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id l12sm1199651qtx.45.2021.07.30.15.04.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Jul 2021 15:04:59 -0700 (PDT) From: Peter Xu To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Vitaly Kuznetsov , Sean Christopherson , peterx@redhat.com, Maxim Levitsky , Paolo Bonzini Subject: [PATCH v3 2/7] KVM: X86: Introduce pte_list_count() helper Date: Fri, 30 Jul 2021 18:04:50 -0400 Message-Id: <20210730220455.26054-3-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210730220455.26054-1-peterx@redhat.com> References: <20210730220455.26054-1-peterx@redhat.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org This helper is used to count the number of rmap entries in the rmap list pointed by the kvm_rmap_head. Signed-off-by: Peter Xu --- arch/x86/kvm/mmu/mmu.c | 21 +++++++++++++++++++++ arch/x86/kvm/mmu/mmu_internal.h | 1 + 2 files changed, 22 insertions(+) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 5429c20cf2cf..16c99f771c9e 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -998,6 +998,27 @@ static void pte_list_remove(struct kvm_rmap_head *rmap_head, u64 *sptep) __pte_list_remove(sptep, rmap_head); } +unsigned int pte_list_count(struct kvm_rmap_head *rmap_head) +{ + struct pte_list_desc *desc; + unsigned int i, count = 0; + + if (!rmap_head->val) + return 0; + else if (!(rmap_head->val & 1)) + return 1; + + desc = (struct pte_list_desc *)(rmap_head->val & ~1ul); + + while (desc) { + for (i = 0; (i < PTE_LIST_EXT) && desc->sptes[i]; i++) + count++; + desc = desc->more; + } + + return count; +} + static struct kvm_rmap_head *__gfn_to_rmap(gfn_t gfn, int level, const struct kvm_memory_slot *slot) { diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_internal.h index ca7b7595bbfc..62bb8f758b3f 100644 --- a/arch/x86/kvm/mmu/mmu_internal.h +++ b/arch/x86/kvm/mmu/mmu_internal.h @@ -131,6 +131,7 @@ bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm, int min_level); void kvm_flush_remote_tlbs_with_address(struct kvm *kvm, u64 start_gfn, u64 pages); +unsigned int pte_list_count(struct kvm_rmap_head *rmap_head); /* * Return values of handle_mmio_page_fault, mmu.page_fault, and fast_page_fault(). From patchwork Fri Jul 30 22:04:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12412025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77077C4320A for ; Fri, 30 Jul 2021 22:05:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 60BDB60F9B for ; Fri, 30 Jul 2021 22:05:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233187AbhG3WFN (ORCPT ); Fri, 30 Jul 2021 18:05:13 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:20583 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232978AbhG3WFJ (ORCPT ); Fri, 30 Jul 2021 18:05:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1627682703; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FfXjy58FasEdIFi4ZWT72LgLttnZ5Iij0BOop3iMW7A=; b=HP3UC0P5LFqQAccVzSJKwe+lggWCjHQrT4kuug+KlsoxmSF1FF6kT10hhUkESmGvtGdV3t b8qq5iE8YwjePg807/d5NcBpVmEx3CnP2GoeYDjLLUO2RTgVdJrL0NhWR5PIhKQZUTIxFi 7LURc1cI5bX+NvIMs1QIb7uhkREdw3Y= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-563-IjTtpinnMkeY83tx9rbMdQ-1; Fri, 30 Jul 2021 18:05:02 -0400 X-MC-Unique: IjTtpinnMkeY83tx9rbMdQ-1 Received: by mail-qt1-f200.google.com with SMTP id v7-20020ac874870000b029024e8ccfcd07so5125668qtq.11 for ; Fri, 30 Jul 2021 15:05:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FfXjy58FasEdIFi4ZWT72LgLttnZ5Iij0BOop3iMW7A=; b=lb6cRyV5yhfa59RQYhqd3TxIYKyvl1C+uAz79XzZfnHuowqZcok8+xCqTQSNtk/ZdE p+/zflhzhp2cmcIV8TF69PZUjzbECi/xFgIuWgX8BNMAEx6rSEILAFjHvahQVGPnWW5g Ejj83wI6AMZoMJ8K4EO5c5g/6LCLRdtlBL9xlPVlS8BRh0HMW/wsfmt/t/SufFGl3joz sg5wc17BNFtSD9D3pFSXx+OuClNR2D1uiaZaX3zrkqBuASiNMegJS6rgSLl1uxOGtOMX GGSfB9fDn2LyDzeyO4U/UaoDtCQTZOHutdgXuYrm2sqefkEHNKI+hZkBCejwehW/geKq 2BKg== X-Gm-Message-State: AOAM532Qj10Van7YuQI3XiUG6rjimtMpCLii9j+s0ubAAYKw4P9kWL9M +Fa2AjULAhLMqEp6OJpHen1CVgXykbPzmqp9fCkcLtUlOU13okibSH1NFJ6Dxq8c3vN8fWyRPJa I4rrnLeJWC+sj4c0pRMwPtwwBkLwTZPJOmVXvvDIeJCPk+xXbWvNNa4MeRLyaCA== X-Received: by 2002:ad4:59c6:: with SMTP id el6mr5069441qvb.61.1627682702055; Fri, 30 Jul 2021 15:05:02 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyH4DMk7eOeaMahyezhFuknnvxsQN34SBZyghGosNbzTWVGxLOwL+A9XhXocD3T6vS3E8BoDA== X-Received: by 2002:ad4:59c6:: with SMTP id el6mr5069414qvb.61.1627682701786; Fri, 30 Jul 2021 15:05:01 -0700 (PDT) Received: from t490s.. (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id l12sm1199651qtx.45.2021.07.30.15.05.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Jul 2021 15:05:00 -0700 (PDT) From: Peter Xu To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Vitaly Kuznetsov , Sean Christopherson , peterx@redhat.com, Maxim Levitsky , Paolo Bonzini Subject: [PATCH v3 3/7] KVM: X86: Introduce kvm_mmu_slot_lpages() helpers Date: Fri, 30 Jul 2021 18:04:51 -0400 Message-Id: <20210730220455.26054-4-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210730220455.26054-1-peterx@redhat.com> References: <20210730220455.26054-1-peterx@redhat.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Introduce kvm_mmu_slot_lpages() to calculcate lpage_info and rmap array size. The other __kvm_mmu_slot_lpages() can take an extra parameter of npages rather than fetching from the memslot pointer. Start to use the latter one in kvm_alloc_memslot_metadata(). Signed-off-by: Peter Xu --- arch/x86/kvm/x86.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 916c976e99ab..e44d8f7781b6 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -299,6 +299,20 @@ static struct kmem_cache *x86_fpu_cache; static struct kmem_cache *x86_emulator_cache; +static inline unsigned long +__kvm_mmu_slot_lpages(struct kvm_memory_slot *slot, unsigned long npages, + int level) +{ + return gfn_to_index(slot->base_gfn + npages - 1, + slot->base_gfn, level) + 1; +} + +static inline unsigned long +kvm_mmu_slot_lpages(struct kvm_memory_slot *slot, int level) +{ + return __kvm_mmu_slot_lpages(slot, slot->npages, level); +} + /* * When called, it means the previous get/set msr reached an invalid msr. * Return true if we want to ignore/silent this failed msr access. @@ -11443,8 +11457,7 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm, int lpages; int level = i + 1; - lpages = gfn_to_index(slot->base_gfn + npages - 1, - slot->base_gfn, level) + 1; + lpages = __kvm_mmu_slot_lpages(slot, npages, level); linfo = kvcalloc(lpages, sizeof(*linfo), GFP_KERNEL_ACCOUNT); if (!linfo) From patchwork Fri Jul 30 22:04:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12412027 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, UNWANTED_LANGUAGE_BODY,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F5DEC4338F for ; Fri, 30 Jul 2021 22:05:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5E22E60F01 for ; Fri, 30 Jul 2021 22:05:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233266AbhG3WFQ (ORCPT ); Fri, 30 Jul 2021 18:05:16 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:49006 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233100AbhG3WFL (ORCPT ); Fri, 30 Jul 2021 18:05:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1627682706; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/RDoZRQ4z39utw+snkzSoyuPXl0N6gUtnPUuN5iXfGY=; b=ZkYLlso5yUt6Uwo8B3lgie+RhS9Pr0ljKVnGOUGSDtTGtD+YssdslyqorGHebSPvv8Ci2+ NN5Ozo+9ixCSoJ2O5NCDo2Qlp5voxkgEcPKLVMOijBWipk/LeGCsDy9LbErJUe0FGydM0B +ivIlKZF25prFa1bjSPysTGeMHoOrNY= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-118-KnnevuvXNq2hXe9jaJDudQ-1; Fri, 30 Jul 2021 18:05:04 -0400 X-MC-Unique: KnnevuvXNq2hXe9jaJDudQ-1 Received: by mail-qk1-f199.google.com with SMTP id x2-20020a05620a0ec2b02903b8bd8b612eso6521713qkm.19 for ; Fri, 30 Jul 2021 15:05:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/RDoZRQ4z39utw+snkzSoyuPXl0N6gUtnPUuN5iXfGY=; b=twaLPCpVdKpnfocQX5F7KZVRhgO4cc5Vzd95dTWpAO3lTUwZ4/mo3yOUl+l98RX70w k6CEHIdPMIFX+sGEum+lt3o9Bl/5OYVHx5Q0L4suD6GhKg4YTtBzZDM2ORBuJt8xlr+j 2MTJnrc+Ro8ClveLg7rIiwsqOyP2wuvuQzpgdWY9BfiH/u7Yd7XFmoHKvSAYiQo3a1U8 0FDU+JHXA3RD5SaXD4rs+P9t/xcOg3JvwsE36JDNcBmei8roFWb0R8Y+tJyJ3hChBsC+ jolBPV+1D7/QRTUH2cvsfxZe+i9rmg5cGUg06RbayHb5O3M0eqW2ZiqDXShzxaqUSZha omoQ== X-Gm-Message-State: AOAM531QxBgZ15c0+dO/YS4b5q6I1JkF91csVlzuc060LzckyxdkUi2O e0AjXRdTZsHGvJcAf286+/SFV1XnlwMFYFXtkWs9yYNh/e/dH5so7kjkAfUhYLR2QhwiJ3g+jL/ MQJ3uvaZa6diX2U7gTSiLIR5K9dBlPHYqnHb0V6m6sD51edVoAbCWQ55PmY9NhA== X-Received: by 2002:a05:6214:301d:: with SMTP id ke29mr5138492qvb.30.1627682703169; Fri, 30 Jul 2021 15:05:03 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwWCmoNcaZU19195qKr1Fps5K3/JOg1+6YD1Cz0f/pAtcjGUvYAKUdj8qKiLNrXgJM3H6DyoA== X-Received: by 2002:a05:6214:301d:: with SMTP id ke29mr5138465qvb.30.1627682702921; Fri, 30 Jul 2021 15:05:02 -0700 (PDT) Received: from t490s.. (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id l12sm1199651qtx.45.2021.07.30.15.05.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Jul 2021 15:05:02 -0700 (PDT) From: Peter Xu To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Vitaly Kuznetsov , Sean Christopherson , peterx@redhat.com, Maxim Levitsky , Paolo Bonzini Subject: [PATCH v3 4/7] KVM: X86: Introduce mmu_rmaps_stat per-vm debugfs file Date: Fri, 30 Jul 2021 18:04:52 -0400 Message-Id: <20210730220455.26054-5-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210730220455.26054-1-peterx@redhat.com> References: <20210730220455.26054-1-peterx@redhat.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Use this file to dump rmap statistic information. The statistic is done by calculating the rmap count and the result is log-2-based. An example output of this looks like (idle 6GB guest, right after boot linux): Rmap_Count: 0 1 2-3 4-7 8-15 16-31 32-63 64-127 128-255 256-511 512-1023 Level=4K: 3086676 53045 12330 1272 502 121 76 2 0 0 0 Level=2M: 5947 231 0 0 0 0 0 0 0 0 0 Level=1G: 32 0 0 0 0 0 0 0 0 0 0 Signed-off-by: Peter Xu --- arch/x86/kvm/x86.c | 113 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index e44d8f7781b6..0877340dc6ff 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -30,6 +30,7 @@ #include "hyperv.h" #include "lapic.h" #include "xen.h" +#include "mmu/mmu_internal.h" #include #include @@ -59,6 +60,7 @@ #include #include #include +#include #include @@ -11193,6 +11195,117 @@ int kvm_arch_post_init_vm(struct kvm *kvm) return kvm_mmu_post_init_vm(kvm); } +/* + * This covers statistics <1024 (11=log(1024)+1), which should be enough to + * cover RMAP_RECYCLE_THRESHOLD. + */ +#define RMAP_LOG_SIZE 11 + +static const char *kvm_lpage_str[KVM_NR_PAGE_SIZES] = { "4K", "2M", "1G" }; + +static int kvm_mmu_rmaps_stat_show(struct seq_file *m, void *v) +{ + struct kvm_rmap_head *rmap; + struct kvm *kvm = m->private; + struct kvm_memory_slot *slot; + struct kvm_memslots *slots; + unsigned int lpage_size, index; + /* Still small enough to be on the stack */ + unsigned int *log[KVM_NR_PAGE_SIZES], *cur; + int i, j, k, l, ret; + + memset(log, 0, sizeof(log)); + + ret = -ENOMEM; + for (i = 0; i < KVM_NR_PAGE_SIZES; i++) { + log[i] = kzalloc(RMAP_LOG_SIZE * sizeof(unsigned int), GFP_KERNEL); + if (!log[i]) + goto out; + } + + mutex_lock(&kvm->slots_lock); + write_lock(&kvm->mmu_lock); + + for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) { + slots = __kvm_memslots(kvm, i); + for (j = 0; j < slots->used_slots; j++) { + slot = &slots->memslots[j]; + for (k = 0; k < KVM_NR_PAGE_SIZES; k++) { + rmap = slot->arch.rmap[k]; + lpage_size = kvm_mmu_slot_lpages(slot, k + 1); + cur = log[k]; + for (l = 0; l < lpage_size; l++) { + index = ffs(pte_list_count(&rmap[l])); + if (WARN_ON_ONCE(index >= RMAP_LOG_SIZE)) + index = RMAP_LOG_SIZE - 1; + cur[index]++; + } + } + } + } + + write_unlock(&kvm->mmu_lock); + mutex_unlock(&kvm->slots_lock); + + /* index=0 counts no rmap; index=1 counts 1 rmap */ + seq_printf(m, "Rmap_Count:\t0\t1\t"); + for (i = 2; i < RMAP_LOG_SIZE; i++) { + j = 1 << (i - 1); + k = (1 << i) - 1; + seq_printf(m, "%d-%d\t", j, k); + } + seq_printf(m, "\n"); + + for (i = 0; i < KVM_NR_PAGE_SIZES; i++) { + seq_printf(m, "Level=%s:\t", kvm_lpage_str[i]); + cur = log[i]; + for (j = 0; j < RMAP_LOG_SIZE; j++) + seq_printf(m, "%d\t", cur[j]); + seq_printf(m, "\n"); + } + + ret = 0; +out: + for (i = 0; i < KVM_NR_PAGE_SIZES; i++) + if (log[i]) + kfree(log[i]); + + return ret; +} + +static int kvm_mmu_rmaps_stat_open(struct inode *inode, struct file *file) +{ + struct kvm *kvm = inode->i_private; + + if (!kvm_get_kvm_safe(kvm)) + return -ENOENT; + + return single_open(file, kvm_mmu_rmaps_stat_show, kvm); +} + +static int kvm_mmu_rmaps_stat_release(struct inode *inode, struct file *file) +{ + struct kvm *kvm = inode->i_private; + + kvm_put_kvm(kvm); + + return single_release(inode, file); +} + +static const struct file_operations mmu_rmaps_stat_fops = { + .open = kvm_mmu_rmaps_stat_open, + .read = seq_read, + .llseek = seq_lseek, + .release = kvm_mmu_rmaps_stat_release, +}; + +int kvm_arch_create_vm_debugfs(struct kvm *kvm) +{ + debugfs_create_file("mmu_rmaps_stat", 0644, kvm->debugfs_dentry, kvm, + &mmu_rmaps_stat_fops); + return 0; +} + static void kvm_unload_vcpu_mmu(struct kvm_vcpu *vcpu) { vcpu_load(vcpu); From patchwork Fri Jul 30 22:04:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12412029 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-24.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CBC1C43214 for ; Fri, 30 Jul 2021 22:05:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7B7B360F01 for ; Fri, 30 Jul 2021 22:05:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233254AbhG3WFR (ORCPT ); Fri, 30 Jul 2021 18:05:17 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:35615 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233247AbhG3WFO (ORCPT ); Fri, 30 Jul 2021 18:05:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1627682708; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nJ5kIz6o1OQd9QTNvIHXfvS8tA6xw4680ifwk1fWPhY=; b=OiBcf8ItaceCmzR9F5B24Vwh30MEgE8kp9gYAIsHzLNphdvLk55YjWqKXBTPByZcHVzKjc AYrGeAfKOVM1gaY5TA7IcpY0YYdxl+KuzIAe9FvxUVE5ZbWD6XLuYrjBHVhX5PNqTsRBGP bfaYISSb1IcxMkF33ViPCTMX5aCDy/U= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-527-KEl4-E6wPJGnyFiEsjj8oA-1; Fri, 30 Jul 2021 18:05:07 -0400 X-MC-Unique: KEl4-E6wPJGnyFiEsjj8oA-1 Received: by mail-qt1-f199.google.com with SMTP id w19-20020ac87e930000b029025a2609eb04so5104392qtj.17 for ; Fri, 30 Jul 2021 15:05:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nJ5kIz6o1OQd9QTNvIHXfvS8tA6xw4680ifwk1fWPhY=; b=JCni+2vOUJ098pbnY/uu4XrIAUp7LUmT0AGk0v2HQzCUF2yA7t0myUweIgWdQxuexL uDnbU25cGsEb5C9kZ2XCru2tlqPs2PdXeV/faNn7hHzMQ+9ALAPQApu1uYOE4IpKgx2O ifPQ2eoOTjtmexzEZjHqJSkPvtDZa5uSfuGlKeydi2jdhpgTx74PJqDSZI5ybaV/ZMqz b3nx/pdWhXFuPnEI/Z3abY5R0aMAEQwSfDVEw+Tf28SvZinvmD5wlzg6LNoTydoTs/O1 K3ctsXUbSADbp85rxfWvrMerjuKelWDxHndQ4XItKrHDLvI+hsxtm7+9DLXZiK4q0hwz xhvw== X-Gm-Message-State: AOAM532d4WveYPiIrvpxzHghsr15m2jf6mUe2k5tpXYNJUw4vCFLJbKm 82eAb/4Tix8EuC3AdSu/+5hkPaBHzCwzPvaCUeimQws0uoEzPV802uo9GXI/gU/IgYKGCDmbDBe dhvaaDePTcX426JjkUK8K8HJlmFUnrSj8fe4QlWR3Qea/ajNdavWUZaIRnoYrCg== X-Received: by 2002:a37:9c06:: with SMTP id f6mr4344540qke.86.1627682705671; Fri, 30 Jul 2021 15:05:05 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz2+zyiyehuJ7kqaZ8WuhUtBpixZqcJNaSYno9mpuUyv7sxWUQmjmlZLJ6h4xEDlsaUXutyew== X-Received: by 2002:a37:9c06:: with SMTP id f6mr4344517qke.86.1627682705409; Fri, 30 Jul 2021 15:05:05 -0700 (PDT) Received: from t490s.. (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id l12sm1199651qtx.45.2021.07.30.15.05.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Jul 2021 15:05:04 -0700 (PDT) From: Peter Xu To: kvm@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Vitaly Kuznetsov , Sean Christopherson , peterx@redhat.com, Maxim Levitsky , Paolo Bonzini Subject: [PATCH v3 5/7] KVM: X86: MMU: Tune PTE_LIST_EXT to be bigger Date: Fri, 30 Jul 2021 18:04:53 -0400 Message-Id: <20210730220455.26054-6-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210730220455.26054-1-peterx@redhat.com> References: <20210730220455.26054-1-peterx@redhat.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Currently rmap array element only contains 3 entries. However for EPT=N there could have a lot of guest pages that got tens of even hundreds of rmap entry. A normal distribution of a 6G guest (even if idle) shows this with rmap count statistics: Rmap_Count: 0 1 2-3 4-7 8-15 16-31 32-63 64-127 128-255 256-511 512-1023 Level=4K: 3089171 49005 14016 1363 235 212 15 7 0 0 0 Level=2M: 5951 227 0 0 0 0 0 0 0 0 0 Level=1G: 32 0 0 0 0 0 0 0 0 0 0 If we do some more fork some pages will grow even larger rmap counts. This patch makes PTE_LIST_EXT bigger so it'll be more efficient for the general use case of EPT=N as we do list reference less and the loops over PTE_LIST_EXT will be slightly more efficient; but still not too large so less waste when array not full. It should not affecting EPT=Y since EPT normally only has zero or one rmap entry for each page, so no array is even allocated. With a test case to fork 500 child and recycle them ("./rmap_fork 500" [1]), this patch speeds up fork time of about 29%. Before: 473.90 (+-5.93%) After: 366.10 (+-4.94%) [1] https://github.com/xzpeter/clibs/commit/825436f825453de2ea5aaee4bdb1c92281efe5b3 Signed-off-by: Peter Xu --- arch/x86/kvm/mmu/mmu.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 16c99f771c9e..c0b452bb5dd9 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -137,8 +137,8 @@ module_param(dbg, bool, 0644); #include -/* make pte_list_desc fit well in cache line */ -#define PTE_LIST_EXT 3 +/* make pte_list_desc fit well in cache lines */ +#define PTE_LIST_EXT 15 struct pte_list_desc { u64 *sptes[PTE_LIST_EXT]; From patchwork Fri Jul 30 22:06:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12412031 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-24.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 032E5C4338F for ; Fri, 30 Jul 2021 22:06:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DFD6D60F01 for ; Fri, 30 Jul 2021 22:06:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233065AbhG3WGM (ORCPT ); Fri, 30 Jul 2021 18:06:12 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:43211 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232796AbhG3WGM (ORCPT ); Fri, 30 Jul 2021 18:06:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1627682766; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=k6/TIj2oL2PuBNwoGj29exdxyYSlXcamNRrMDM9CcnI=; b=F8H/jvahyjTGlrmq06swTtRxjVDQx1xpoIi8dTpwAfrMBTUIdVM19mzW2uqNkQ1Pd+1QwP EJ9pa7Bs4XcwS8Spij1w2qeQ8u+vZ0j+NwceOzHNAfW3YoSxD/QA9T3NUNU4eTnSSKvqde pAoSR91U8ukc35R4Uq/YhMLjylQQtmo= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-457-lZzK_85_PWGAKuhdPZzuvA-1; Fri, 30 Jul 2021 18:06:05 -0400 X-MC-Unique: lZzK_85_PWGAKuhdPZzuvA-1 Received: by mail-qv1-f69.google.com with SMTP id w10-20020a0cfc4a0000b0290335dd22451dso4754147qvp.5 for ; Fri, 30 Jul 2021 15:06:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=k6/TIj2oL2PuBNwoGj29exdxyYSlXcamNRrMDM9CcnI=; b=dxOadgBZoHJxXVwqxfKM/A1uFASEAuM5zh6uPOAoGD1+Ynt8Wh8Uf/3viGwdq332OT Zwl9BsYWZ19lYMCtGjwS7PvQ1Hslz4JRW17THmB50gZLt+j2l65KnF9WD33ZlLmx1b/A 4bGyMeGVIaBuhk2nZf14GrbzKdbV3b/w7gJlzkgVK/UI5cgqbCYBAElyE07RsFiaTclE F+n4RBGj3ir9QBvtLlAEoA0i51v0If8YPG9syEwqUu5JOC8eyVXjwyR0xSpmdMMj0/2a 1gCOmymLvDFySDBSdOqRYDfdeVRD+pHexVaK7ST64RU6fSw0pOT/0BvBKDQ49RkG0jTM NH2A== X-Gm-Message-State: AOAM533rB28EdJnMctoOxagSFeqDUkoaLEZjcVWSy1BJC49zznh+CG+0 WGS8Ibh9oO+L82SD+NhTt85F5x4Wi+G02wLfZ5VQlpMoAwI0XNTMLulAhAMLThsDbmPZm36E0fO PW61SaX5IPMgs X-Received: by 2002:a0c:be8e:: with SMTP id n14mr5098409qvi.16.1627682764545; Fri, 30 Jul 2021 15:06:04 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwcw83DgDJNDOwVMK+xRgEPO3B9UIXdU/pUAOXMNnInpKpTdAOBSEDfC1apcAufJ0N+exEV0Q== X-Received: by 2002:a0c:be8e:: with SMTP id n14mr5098393qvi.16.1627682764334; Fri, 30 Jul 2021 15:06:04 -0700 (PDT) Received: from t490s.. (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id o186sm1622893qke.44.2021.07.30.15.06.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Jul 2021 15:06:03 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Maxim Levitsky , Sean Christopherson , Vitaly Kuznetsov , peterx@redhat.com, Paolo Bonzini Subject: [PATCH v3 6/7] KVM: X86: Optimize pte_list_desc with per-array counter Date: Fri, 30 Jul 2021 18:06:02 -0400 Message-Id: <20210730220602.26327-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210730220455.26054-1-peterx@redhat.com> References: <20210730220455.26054-1-peterx@redhat.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add a counter field into pte_list_desc, so as to simplify the add/remove/loop logic. E.g., we don't need to loop over the array any more for most reasons. This will make more sense after we've switched the array size to be larger otherwise the counter will be a waste. Initially I wanted to store a tail pointer at the head of the array list so we don't need to traverse the list at least for pushing new ones (if without the counter we traverse both the list and the array). However that'll need slightly more change without a huge lot benefit, e.g., after we grow entry numbers per array the list traversing is not so expensive. So let's be simple but still try to get as much benefit as we can with just these extra few lines of changes (not to mention the code looks easier too without looping over arrays). I used the same a test case to fork 500 child and recycle them ("./rmap_fork 500" [1]), this patch further speeds up the total fork time of about 4%, which is a total of 33% of vanilla kernel: Vanilla: 473.90 (+-5.93%) 3->15 slots: 366.10 (+-4.94%) Add counter: 351.00 (+-3.70%) [1] https://github.com/xzpeter/clibs/commit/825436f825453de2ea5aaee4bdb1c92281efe5b3 Signed-off-by: Peter Xu --- arch/x86/kvm/mmu/mmu.c | 40 ++++++++++++++++++++++++---------------- 1 file changed, 24 insertions(+), 16 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index c0b452bb5dd9..111c37141dbe 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -138,11 +138,21 @@ module_param(dbg, bool, 0644); #include /* make pte_list_desc fit well in cache lines */ -#define PTE_LIST_EXT 15 +#define PTE_LIST_EXT 14 +/* + * Slight optimization of cacheline layout, by putting `more' and `spte_count' + * at the start; then accessing it will only use one single cacheline for + * either full (entries==PTE_LIST_EXT) case or entries<=6. + */ struct pte_list_desc { - u64 *sptes[PTE_LIST_EXT]; struct pte_list_desc *more; + /* + * Stores number of entries stored in the pte_list_desc. No need to be + * u64 but just for easier alignment. When PTE_LIST_EXT, means full. + */ + u64 spte_count; + u64 *sptes[PTE_LIST_EXT]; }; struct kvm_shadow_walk_iterator { @@ -901,7 +911,7 @@ static int pte_list_add(struct kvm_vcpu *vcpu, u64 *spte, struct kvm_rmap_head *rmap_head) { struct pte_list_desc *desc; - int i, count = 0; + int count = 0; if (!rmap_head->val) { rmap_printk("%p %llx 0->1\n", spte, *spte); @@ -911,24 +921,24 @@ static int pte_list_add(struct kvm_vcpu *vcpu, u64 *spte, desc = mmu_alloc_pte_list_desc(vcpu); desc->sptes[0] = (u64 *)rmap_head->val; desc->sptes[1] = spte; + desc->spte_count = 2; rmap_head->val = (unsigned long)desc | 1; ++count; } else { rmap_printk("%p %llx many->many\n", spte, *spte); desc = (struct pte_list_desc *)(rmap_head->val & ~1ul); - while (desc->sptes[PTE_LIST_EXT-1]) { + while (desc->spte_count == PTE_LIST_EXT) { count += PTE_LIST_EXT; - if (!desc->more) { desc->more = mmu_alloc_pte_list_desc(vcpu); desc = desc->more; + desc->spte_count = 0; break; } desc = desc->more; } - for (i = 0; desc->sptes[i]; ++i) - ++count; - desc->sptes[i] = spte; + count += desc->spte_count; + desc->sptes[desc->spte_count++] = spte; } return count; } @@ -938,13 +948,12 @@ pte_list_desc_remove_entry(struct kvm_rmap_head *rmap_head, struct pte_list_desc *desc, int i, struct pte_list_desc *prev_desc) { - int j; + int j = desc->spte_count - 1; - for (j = PTE_LIST_EXT - 1; !desc->sptes[j] && j > i; --j) - ; desc->sptes[i] = desc->sptes[j]; desc->sptes[j] = NULL; - if (j != 0) + desc->spte_count--; + if (desc->spte_count) return; if (!prev_desc && !desc->more) rmap_head->val = 0; @@ -977,7 +986,7 @@ static void __pte_list_remove(u64 *spte, struct kvm_rmap_head *rmap_head) desc = (struct pte_list_desc *)(rmap_head->val & ~1ul); prev_desc = NULL; while (desc) { - for (i = 0; i < PTE_LIST_EXT && desc->sptes[i]; ++i) { + for (i = 0; i < desc->spte_count; ++i) { if (desc->sptes[i] == spte) { pte_list_desc_remove_entry(rmap_head, desc, i, prev_desc); @@ -1001,7 +1010,7 @@ static void pte_list_remove(struct kvm_rmap_head *rmap_head, u64 *sptep) unsigned int pte_list_count(struct kvm_rmap_head *rmap_head) { struct pte_list_desc *desc; - unsigned int i, count = 0; + unsigned int count = 0; if (!rmap_head->val) return 0; @@ -1011,8 +1020,7 @@ unsigned int pte_list_count(struct kvm_rmap_head *rmap_head) desc = (struct pte_list_desc *)(rmap_head->val & ~1ul); while (desc) { - for (i = 0; (i < PTE_LIST_EXT) && desc->sptes[i]; i++) - count++; + count += desc->spte_count; desc = desc->more; } From patchwork Fri Jul 30 22:06:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12412033 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9816BC432BE for ; Fri, 30 Jul 2021 22:06:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8144860E93 for ; Fri, 30 Jul 2021 22:06:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233247AbhG3WGR (ORCPT ); Fri, 30 Jul 2021 18:06:17 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:35492 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233146AbhG3WGQ (ORCPT ); Fri, 30 Jul 2021 18:06:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1627682770; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=o4BBOvgzxNUCJGeElx/7CJIgRVcb9Llih1duOoDbR4M=; b=iq5n5/mA1hd3AeNX0RyRTMn4v+PMwDrBLWbaLiWyYiboJ+G6busUmmS9hwHkvLGo8clXeI XA+ZDQs0rEXwGAnmuuJ5MDrdpE+dVMR07WAp0UiZPf/i+lR0AFSuuoc6dUVDjziwXnpD12 06o7r9BpGbQ0I/qR86pjtH2VgWJJWe0= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-205-7tBJQCaWPvS0xt48LluGsQ-1; Fri, 30 Jul 2021 18:06:07 -0400 X-MC-Unique: 7tBJQCaWPvS0xt48LluGsQ-1 Received: by mail-qk1-f198.google.com with SMTP id e11-20020a05620a208bb02903b854c43335so5986724qka.21 for ; Fri, 30 Jul 2021 15:06:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=o4BBOvgzxNUCJGeElx/7CJIgRVcb9Llih1duOoDbR4M=; b=iapL9uUcTkNBAyVNnkdmF3dsRZGClgU+pAHg4OzXTjQ51kqpLoaIuvRi/a6c/IqIHV gvUbIXkekNtZtMW40p5XVPGXSsxyoO6AoTAMOIo8KnISHDAK/8b5P5yWprNhwqzc2ga1 6ttGKndYmKEC5K/9LLjs2IbJiJb/R9E1AiPLYCL+PQv4CKsTQUGbk+CKQQ07C/X6SDN+ 9x2cq9QTZAfjZbX9Mf8dtPKow2fRYumhkiEgogEf0N88EkkpyjJzl3AnJchdGIG9PnrL hsbkMVlb3CrMCmyy3TvKvuUTYlrriP6rhZV7WY9WLQNdpRjxBrnKiz8V11ufR9ge6rHd 8dpQ== X-Gm-Message-State: AOAM531CCL6U6BzMg0nuK3z67czVW69MVvoPXToAwcSahxp/bfzuD31b Xo11R61r7TU6ftwWbxhjuI+7f/Zno5kkc0TPc8ain0DpYKSbiywxP4Vefo31/q1xZzwVihIfnwx r4MunaAVwnsiQ X-Received: by 2002:a0c:df09:: with SMTP id g9mr5152069qvl.30.1627682766745; Fri, 30 Jul 2021 15:06:06 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz8B8Fu0lZQSprSiVyqxCrIMRlql3zmCcV7iN3gkLx6KiU458PV2fSBpow9iSveEnl956h6FQ== X-Received: by 2002:a0c:df09:: with SMTP id g9mr5152051qvl.30.1627682766549; Fri, 30 Jul 2021 15:06:06 -0700 (PDT) Received: from t490s.. (bras-base-toroon474qw-grc-65-184-144-111-238.dsl.bell.ca. [184.144.111.238]) by smtp.gmail.com with ESMTPSA id d16sm1154974qtj.69.2021.07.30.15.06.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 30 Jul 2021 15:06:06 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: Maxim Levitsky , Sean Christopherson , Vitaly Kuznetsov , peterx@redhat.com, Paolo Bonzini Subject: [PATCH v3 7/7] KVM: X86: Optimize zapping rmap Date: Fri, 30 Jul 2021 18:06:05 -0400 Message-Id: <20210730220605.26377-1-peterx@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210730220455.26054-1-peterx@redhat.com> References: <20210730220455.26054-1-peterx@redhat.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Using rmap_get_first() and rmap_remove() for zapping a huge rmap list could be slow. The easy way is to travers the rmap list, collecting the a/d bits and free the slots along the way. Provide a pte_list_destroy() and do exactly that. Signed-off-by: Peter Xu --- arch/x86/kvm/mmu/mmu.c | 41 +++++++++++++++++++++++++++++------------ 1 file changed, 29 insertions(+), 12 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 111c37141dbe..9b2616760e23 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1027,6 +1027,34 @@ unsigned int pte_list_count(struct kvm_rmap_head *rmap_head) return count; } +/* Return true if rmap existed, false otherwise */ +static bool pte_list_destroy(struct kvm_rmap_head *rmap_head) +{ + struct pte_list_desc *desc, *next; + int i; + + if (!rmap_head->val) + return false; + + if (!(rmap_head->val & 1)) { + mmu_spte_clear_track_bits((u64 *)rmap_head->val); + goto out; + } + + desc = (struct pte_list_desc *)(rmap_head->val & ~1ul); + + for (; desc; desc = next) { + for (i = 0; i < desc->spte_count; i++) + mmu_spte_clear_track_bits(desc->sptes[i]); + next = desc->more; + mmu_free_pte_list_desc(desc); + } +out: + /* rmap_head is meaningless now, remember to reset it */ + rmap_head->val = 0; + return true; +} + static struct kvm_rmap_head *__gfn_to_rmap(gfn_t gfn, int level, const struct kvm_memory_slot *slot) { @@ -1418,18 +1446,7 @@ static bool rmap_write_protect(struct kvm_vcpu *vcpu, u64 gfn) static bool kvm_zap_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head, const struct kvm_memory_slot *slot) { - u64 *sptep; - struct rmap_iterator iter; - bool flush = false; - - while ((sptep = rmap_get_first(rmap_head, &iter))) { - rmap_printk("spte %p %llx.\n", sptep, *sptep); - - pte_list_remove(rmap_head, sptep); - flush = true; - } - - return flush; + return pte_list_destroy(rmap_head); } static bool kvm_unmap_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head,