From patchwork Fri Jul 30 22:37:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12412077 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C29BC432BE for ; Fri, 30 Jul 2021 22:37:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2183760F42 for ; Fri, 30 Jul 2021 22:37:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233273AbhG3WhW (ORCPT ); Fri, 30 Jul 2021 18:37:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52438 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232817AbhG3WhV (ORCPT ); Fri, 30 Jul 2021 18:37:21 -0400 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B799EC06175F for ; Fri, 30 Jul 2021 15:37:15 -0700 (PDT) Received: by mail-pl1-x649.google.com with SMTP id f9-20020a1709028609b0290128bcba6be7so8699917plo.18 for ; Fri, 30 Jul 2021 15:37:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=dMwap4hObUgoDwGKFMQjZvRZTCSpSBJpDZ6SHzoUej4=; b=px4bUOZfcOhRJsEm/Sr8g4k95JNeIzM6JiOzIBpBY6IM6kuDZGh92zuj7W+WNPQuyL 7VNqO3x8d2aBGs+rWxrd4v8KgYIEM9pxJeEM//MAaQgrV3jGrwYhYKeevY5tiIavyOYu cS3RgbFZG/ZMyct9IdRjJEVUE1U7rlFxzy3TFqAmuUZP5aXmxh0S0nycIamkztVy+dz/ 6XSBD2Z7hpr5rb/mvZEY12vh9SJApojYBRNEM6F6Tp1JId/lWyOHPzy1XXQneGqcP3SB nvVxFBkRp2IFOZ9v+078asAjbU3oMK7qtoN3lOpYXy2aUVQHStTZyABRll1ywSySLPUG pdxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=dMwap4hObUgoDwGKFMQjZvRZTCSpSBJpDZ6SHzoUej4=; b=LiiwUKeeZRC+nDt/hTc64pWD+NPR17INW6FwfkEui9jgR8WNdBNzbyMhc6v1332G1P vDKm89A5BRrBsKEMqWBZfr37Qw27aMaBhvVb+n+iDCtd5Ns6P7qIGf5GFd8p7tN4ZjxJ wkjjJEfQAsg+qIPe7HMeUofa7Lmvg5ljOdmrjZ0AQBoowOAbdYNs1B9pLx421D+I0Nk+ TQqDbZLh/k1J/XyiEwBmpcxdJKaoyLFc2Zvx/WlxRUQjxwbxZs3Gd7Cgeoc98xVyTLUN K9i06Ui+vqCnnJh+O1ZP9JNgboWyw7GBEWYzQASdnDtQyFhtmy5oHa3v0IEfH2mOwaFZ m5JA== X-Gm-Message-State: AOAM5305Q/cvwtpCxy6L57XfF4cxGbr4OHbyJQTLCFNnzkCzKCLdHSrh RfXZWOgYCSga5fKYCKBjXsb1gtEU6YQ+DUaFjcAco04VVbUIeA3CW9pezJYt2yXo6djYR3/TK7J GWvlF6i+fkQb93O6W4lw3AzLaTC4oKRLKe0tj117+WEJJusUaRNPaIegcW/S44gE= X-Google-Smtp-Source: ABdhPJxBfbsygN5WdnwHHjaO9B0TAkPrvBKH8QFlHhGFSBhwXNnLD7yI53Ixn1v9pk+s3wtGmjh2mhXJRQXdhA== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a05:6a00:1394:b029:3ac:c1a3:f988 with SMTP id t20-20020a056a001394b02903acc1a3f988mr5137051pfg.37.1627684635120; Fri, 30 Jul 2021 15:37:15 -0700 (PDT) Date: Fri, 30 Jul 2021 22:37:02 +0000 In-Reply-To: <20210730223707.4083785-1-dmatlack@google.com> Message-Id: <20210730223707.4083785-2-dmatlack@google.com> Mime-Version: 1.0 References: <20210730223707.4083785-1-dmatlack@google.com> X-Mailer: git-send-email 2.32.0.554.ge1b32706d8-goog Subject: [PATCH 1/6] KVM: Cache the least recently used slot index per vCPU From: David Matlack To: kvm@vger.kernel.org Cc: Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Paolo Bonzini , Junaid Shahid , Andrew Jones , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The memslot for a given gfn is looked up multiple times during page fault handling. Avoid binary searching for it multiple times by caching the least recently used slot. There is an existing VM-wide LRU slot but that does not work well for cases where vCPUs are accessing memory in different slots (see performance data below). Another benefit of caching the least recently use slot (versus looking up the slot once and passing around a pointer) is speeding up memslot lookups *across* faults and during spte prefetching. To measure the performance of this change I ran dirty_log_perf_test with 64 vCPUs and 64 memslots and measured "Populate memory time" and "Iteration 2 dirty memory time". Tests were ran with eptad=N to force dirty logging to use fast_page_fault so its performance could be measured. Config | Metric | Before | After ---------- | ----------------------------- | ------ | ------ tdp_mmu=Y | Populate memory time | 6.76s | 5.47s tdp_mmu=Y | Iteration 2 dirty memory time | 2.83s | 0.31s tdp_mmu=N | Populate memory time | 20.4s | 18.7s tdp_mmu=N | Iteration 2 dirty memory time | 2.65s | 0.30s The "Iteration 2 dirty memory time" results are especially compelling because they are equivalent to running the same test with a single memslot. In other words, fast_page_fault performance no longer scales with the number of memslots. Signed-off-by: David Matlack --- include/linux/kvm_host.h | 61 ++++++++++++++++++++++++++++++---------- virt/kvm/kvm_main.c | 21 +++++++++++++- 2 files changed, 66 insertions(+), 16 deletions(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 9d6b4ad407b8..320090d5a124 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -354,6 +354,13 @@ struct kvm_vcpu { struct kvm_vcpu_stat stat; char stats_id[KVM_STATS_NAME_SIZE]; struct kvm_dirty_ring dirty_ring; + + /* + * The index of the least recently used memslot by this vCPU. It's ok + * if this becomes stale due to memslot changes since we always check + * it is a valid slot. + */ + int lru_slot_index; }; /* must be called with irqs disabled */ @@ -1189,27 +1196,38 @@ int kvm_request_irq_source_id(struct kvm *kvm); void kvm_free_irq_source_id(struct kvm *kvm, int irq_source_id); bool kvm_arch_irqfd_allowed(struct kvm *kvm, struct kvm_irqfd *args); +static inline struct kvm_memory_slot *get_slot(struct kvm_memslots *slots, int slot_index) +{ + if (slot_index < 0 || slot_index >= slots->used_slots) + return NULL; + + return &slots->memslots[slot_index]; +} + +static inline bool slot_contains_gfn(struct kvm_memslots *slots, int slot_index, gfn_t gfn) +{ + struct kvm_memory_slot *memslot = get_slot(slots, slot_index); + + if (!memslot) + return false; + + return gfn >= memslot->base_gfn && gfn < memslot->base_gfn + memslot->npages; +} + /* - * search_memslots() and __gfn_to_memslot() are here because they are - * used in non-modular code in arch/powerpc/kvm/book3s_hv_rm_mmu.c. - * gfn_to_memslot() itself isn't here as an inline because that would - * bloat other code too much. - * * IMPORTANT: Slots are sorted from highest GFN to lowest GFN! */ -static inline struct kvm_memory_slot * -search_memslots(struct kvm_memslots *slots, gfn_t gfn) +static inline int __search_memslots(struct kvm_memslots *slots, gfn_t gfn) { int start = 0, end = slots->used_slots; int slot = atomic_read(&slots->lru_slot); struct kvm_memory_slot *memslots = slots->memslots; if (unlikely(!slots->used_slots)) - return NULL; + return -1; - if (gfn >= memslots[slot].base_gfn && - gfn < memslots[slot].base_gfn + memslots[slot].npages) - return &memslots[slot]; + if (slot_contains_gfn(slots, slot, gfn)) + return slot; while (start < end) { slot = start + (end - start) / 2; @@ -1220,13 +1238,26 @@ search_memslots(struct kvm_memslots *slots, gfn_t gfn) start = slot + 1; } - if (start < slots->used_slots && gfn >= memslots[start].base_gfn && - gfn < memslots[start].base_gfn + memslots[start].npages) { + if (slot_contains_gfn(slots, start, gfn)) { atomic_set(&slots->lru_slot, start); - return &memslots[start]; + return start; } - return NULL; + return -1; +} + +/* + * search_memslots() and __gfn_to_memslot() are here because they are + * used in non-modular code in arch/powerpc/kvm/book3s_hv_rm_mmu.c. + * gfn_to_memslot() itself isn't here as an inline because that would + * bloat other code too much. + */ +static inline struct kvm_memory_slot * +search_memslots(struct kvm_memslots *slots, gfn_t gfn) +{ + int slot_index = __search_memslots(slots, gfn); + + return get_slot(slots, slot_index); } static inline struct kvm_memory_slot * diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index a96cbe24c688..9307594bda0c 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -415,6 +415,7 @@ static void kvm_vcpu_init(struct kvm_vcpu *vcpu, struct kvm *kvm, unsigned id) vcpu->preempted = false; vcpu->ready = false; preempt_notifier_init(&vcpu->preempt_notifier, &kvm_preempt_ops); + vcpu->lru_slot_index = 0; } void kvm_vcpu_destroy(struct kvm_vcpu *vcpu) @@ -2024,7 +2025,25 @@ EXPORT_SYMBOL_GPL(gfn_to_memslot); struct kvm_memory_slot *kvm_vcpu_gfn_to_memslot(struct kvm_vcpu *vcpu, gfn_t gfn) { - return __gfn_to_memslot(kvm_vcpu_memslots(vcpu), gfn); + struct kvm_memslots *slots = kvm_vcpu_memslots(vcpu); + int slot_index = vcpu->lru_slot_index; + struct kvm_memory_slot *slot; + + if (!slot_contains_gfn(slots, slot_index, gfn)) + slot_index = __search_memslots(slots, gfn); + + slot = get_slot(slots, slot_index); + + /* + * Purposely avoid updating vcpu->lru_slot_index if the gfn is not + * backed by memslot as that will guarantee a cache miss on the next + * try. By leaving vcpu->lru_slot_index untouched we have a chance of + * a hit on the next lookup. + */ + if (slot) + vcpu->lru_slot_index = slot_index; + + return slot; } EXPORT_SYMBOL_GPL(kvm_vcpu_gfn_to_memslot); From patchwork Fri Jul 30 22:37:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12412079 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14EAFC4338F for ; Fri, 30 Jul 2021 22:37:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E79CF60F42 for ; Fri, 30 Jul 2021 22:37:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232817AbhG3WhY (ORCPT ); Fri, 30 Jul 2021 18:37:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52446 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233310AbhG3WhX (ORCPT ); Fri, 30 Jul 2021 18:37:23 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB801C06175F for ; Fri, 30 Jul 2021 15:37:17 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id j9-20020a2581490000b02905897d81c63fso6553844ybm.8 for ; Fri, 30 Jul 2021 15:37:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Fo9NvED0lpMkXN/JaIKCFquwO1thPwina4O0gbID30c=; b=Ayf8xXcekBEC/WtWhsL8b0fELKIo/kWYg2fpL1gfO/i1K1fxcCofieXaNyLbIv4zpq BmM4OVMuLdKosyOSvKtiXcYK2U8mkscxyMiANttRwiax8h0KX9/pc7l1hxfRV6CEBVwk 0IwRBzGz1b3Pjb5gC+rmzjmWV8pzix7HcOp5xRJ1KxJf/btpUiG03GNyXX6SCzONglJG dWQTu2DVrCzUPmDSt3MB4L1V0N7+BHfNxEmlVE6q3YK+tPDOaJAheQ1nxHvXmbEqc66A lvgGbUjNxCx2H00uFvDBSt5BXLXpEiSuRdJ/CYuJVoGAd+uOeBWnOm1o5+xcw+CHJeBP VYag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Fo9NvED0lpMkXN/JaIKCFquwO1thPwina4O0gbID30c=; b=o+sOXZDFDx+PPlzhwx8fcZ+N+1SGV3uG1FldcjDtamp9uMU8Cskba6uEoIdjMWVLVa BzcyXqSlZm9gwDc02wubWY91wECEX1J4wOg6Jk/m07OxCTRvQI8Y218+8Kt2YCOsyT0Q kOKezr8Bb7GCmkgODuGwW5XqWJyDTHb396s5aJJTRQg9MgE0iNJIyKKIz8TXWtGAc7rf x5vapovC3PK4ZAlAluAJjZG/ffdvIu2/CRgMgOSe/lRJY8x23krdmJHiTiXwOV9fJMnI POn/XtCbdvKzo8dNB6wvFqtt/bnMQhfDfhGeFP0qWlQkDOCG3H2iFIci3P7z/ak0P3q0 gvyQ== X-Gm-Message-State: AOAM532CrZundFM6/bbIhW8BuCbwW/NiDHP9re5+X2jpN+cJVu/seVTt EddKZPddsHtNO5P9S0UnXo+rs4TRuG/xLr2mJ0pMDmjqmcfYmjNFZvbqQfva67AknRqbaKI5HOc yFGBFzfd3UlbYLiG5NdDTRPQGayNAFJcCcMV8BFY9GZzbM7aLeIkXev4li0QNrRU= X-Google-Smtp-Source: ABdhPJwMnskhDblL6E8qFgZaYyvNXA8wYcJUVlWu79Mi0Q3msbXcQaS3Uf4RpviCGRUDy+BKatABfYafDH7KkQ== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a25:b810:: with SMTP id v16mr6539371ybj.357.1627684636816; Fri, 30 Jul 2021 15:37:16 -0700 (PDT) Date: Fri, 30 Jul 2021 22:37:03 +0000 In-Reply-To: <20210730223707.4083785-1-dmatlack@google.com> Message-Id: <20210730223707.4083785-3-dmatlack@google.com> Mime-Version: 1.0 References: <20210730223707.4083785-1-dmatlack@google.com> X-Mailer: git-send-email 2.32.0.554.ge1b32706d8-goog Subject: [PATCH 2/6] KVM: Avoid VM-wide lru_slot lookup in kvm_vcpu_gfn_to_memslot From: David Matlack To: kvm@vger.kernel.org Cc: Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Paolo Bonzini , Junaid Shahid , Andrew Jones , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Now that vCPUs keep track of their own LRU slot, there's no good reason to have them check and update the VM-wide LRU slot. There's no performance data to motivate this change however there are two rationals: 1. Now that vCPUs have their own LRU slot, there's a potential for a double miss (miss the vCPU LRU slot and then miss the VM-wide LRU slot). By avoiding the VM-wide LRU slot check we keep the worst case to a single miss. 2. Large VMs are likely to have multiple memslots and vCPUs accessing different slots. Intuitively, vCPUs will end up thrashing the VM-wide LRU slot, decreasing the LRU hit rate for VM-wide operations such as mmu notifiers and VM ioctls. Signed-off-by: David Matlack --- include/linux/kvm_host.h | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 320090d5a124..870e1e6fb771 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -1220,17 +1220,13 @@ static inline bool slot_contains_gfn(struct kvm_memslots *slots, int slot_index, static inline int __search_memslots(struct kvm_memslots *slots, gfn_t gfn) { int start = 0, end = slots->used_slots; - int slot = atomic_read(&slots->lru_slot); struct kvm_memory_slot *memslots = slots->memslots; if (unlikely(!slots->used_slots)) return -1; - if (slot_contains_gfn(slots, slot, gfn)) - return slot; - while (start < end) { - slot = start + (end - start) / 2; + int slot = start + (end - start) / 2; if (gfn >= memslots[slot].base_gfn) end = slot; @@ -1238,10 +1234,8 @@ static inline int __search_memslots(struct kvm_memslots *slots, gfn_t gfn) start = slot + 1; } - if (slot_contains_gfn(slots, start, gfn)) { - atomic_set(&slots->lru_slot, start); + if (slot_contains_gfn(slots, start, gfn)) return start; - } return -1; } @@ -1255,8 +1249,16 @@ static inline int __search_memslots(struct kvm_memslots *slots, gfn_t gfn) static inline struct kvm_memory_slot * search_memslots(struct kvm_memslots *slots, gfn_t gfn) { - int slot_index = __search_memslots(slots, gfn); + int slot_index = atomic_read(&slots->lru_slot); + + if (slot_contains_gfn(slots, slot_index, gfn)) + return get_slot(slots, slot_index); + + slot_index = __search_memslots(slots, gfn); + if (slot_index < 0) + return NULL; + atomic_set(&slots->lru_slot, slot_index); return get_slot(slots, slot_index); } From patchwork Fri Jul 30 22:37:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12412081 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5CCAC432BE for ; Fri, 30 Jul 2021 22:37:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A0F3F60FE7 for ; Fri, 30 Jul 2021 22:37:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233500AbhG3Wh0 (ORCPT ); Fri, 30 Jul 2021 18:37:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52458 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233322AbhG3WhZ (ORCPT ); Fri, 30 Jul 2021 18:37:25 -0400 Received: from mail-qk1-x749.google.com (mail-qk1-x749.google.com [IPv6:2607:f8b0:4864:20::749]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3DCBDC06175F for ; Fri, 30 Jul 2021 15:37:19 -0700 (PDT) Received: by mail-qk1-x749.google.com with SMTP id x12-20020a05620a14acb02903b8f9d28c19so6550062qkj.23 for ; Fri, 30 Jul 2021 15:37:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=DVeQW96yDje60l1vMNxOEQha2Pu/nzO8XHDKxp1dy1o=; b=LhEp5DVrxD5oHu/gzk50OVffOBhsi4k+j8Lc1A3h639Z07Be9R02s/XJbWwN+r0hJm XST7lu9uGQzNaqPWzPuVY5cdmDmhYI67Ro6yELPT+ZUSzSL5tjifgj3i0CSAk0j2g8qK ZeSrIR5F3OO4HKNtb3SJrjYDgcrjg6OsGzmRlMjmQhl5ud2tTI8TYssBmrQbhEmXFJpY 6zHzDcYHuS6GJTN1P9ydGxY3u8VXCK5FY3lebP9WDVSLNDuez4aZt3BkYIuDol2mh8v7 IViDBYfkz5D7KeikNiTV4/Avo/6VNR4zY8o6tGCoyE6xxLqETSEhBCh0PO8/JzS60ioC gd1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=DVeQW96yDje60l1vMNxOEQha2Pu/nzO8XHDKxp1dy1o=; b=nWPKeE/lwtOyla0lT8ByzjYMakjxj3PVUnTWb+hUIWbwXU94Vb1tIQZpk/GNsa8vJQ 2wB80mNO3PQZp60gid6zSjFOjsjwLRCiBizMlvF3kACDFx+ceCvSEPguS3zRPjMwwnAh qst7oxsFrSwfKZntLzD+MBb9RsMuTumAwjcyB0YGMxBbk4aT/WctGLm9aqvhdpIL2V6o SLzez4tD03qk3USwRpYThtqf775GxzzxEdiLLpdyk+igySyqqIdCu5lVNYqcif4NQiJD xgLksOy0U+bq0RPzAvie4VY2jbcryvvsaPgqJgKLK5glu51ZoHwS+dEQrdh4K8IM4QLZ uprQ== X-Gm-Message-State: AOAM533dr17lK20ZCZtRE/hCEeP9q5ZIBcrVa4FGlUmMma/xxyXFCBms upBkKiXQY4FoxAhhY84XN9HCGY8FUKq0vkhUBdwshQZU/VlXkYkkosTLAdIl71nTQEeYIESUzEP mcRFqkMEi9mP1J5QdA1VqfDTQUQs59XS/t2Y9MweGa3/vopL1nqrTGwHDZ5Hg/bU= X-Google-Smtp-Source: ABdhPJwC9qCk9bZ/jZxxOerVx4mYD76+MVxUaByt8pYh/M2A2qsqnHeJcnFBoPMQVZAGsT/opI4NbN4g89Upog== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a0c:d801:: with SMTP id h1mr5090968qvj.60.1627684638295; Fri, 30 Jul 2021 15:37:18 -0700 (PDT) Date: Fri, 30 Jul 2021 22:37:04 +0000 In-Reply-To: <20210730223707.4083785-1-dmatlack@google.com> Message-Id: <20210730223707.4083785-4-dmatlack@google.com> Mime-Version: 1.0 References: <20210730223707.4083785-1-dmatlack@google.com> X-Mailer: git-send-email 2.32.0.554.ge1b32706d8-goog Subject: [PATCH 3/6] KVM: x86/mmu: Speed up dirty logging in tdp_mmu_map_handle_target_level From: David Matlack To: kvm@vger.kernel.org Cc: Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Paolo Bonzini , Junaid Shahid , Andrew Jones , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The existing TDP MMU methods to handle dirty logging are vcpu-agnostic since they can be driven by MMU notifiers and other non-vcpu-specific events in addition to page faults. However this means that the TDP MMU is not benefiting from the new vcpu->lru_slot_index. Fix that by special casing dirty logging in tdp_mmu_map_handle_target_level. This improves "Populate memory time" in dirty_log_perf_test by 5%: Command | Before | After ------------------------------- | ---------------- | ------------- ./dirty_log_perf_test -v64 -x64 | 5.472321072s | 5.169832886s Signed-off-by: David Matlack --- arch/x86/kvm/mmu/tdp_mmu.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 43f12f5d12c0..1467f99c846d 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -929,10 +929,19 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu, int write, map_writable, !shadow_accessed_mask, &new_spte); - if (new_spte == iter->old_spte) + if (new_spte == iter->old_spte) { ret = RET_PF_SPURIOUS; - else if (!tdp_mmu_set_spte_atomic(vcpu->kvm, iter, new_spte)) - return RET_PF_RETRY; + } else { + if (!tdp_mmu_set_spte_atomic_no_dirty_log(vcpu->kvm, iter, new_spte)) + return RET_PF_RETRY; + + /* + * Mark the gfn dirty here rather that through the vcpu-agnostic + * handle_changed_spte_dirty_log to leverage vcpu->lru_slot_index. + */ + if (is_writable_pte(new_spte)) + kvm_vcpu_mark_page_dirty(vcpu, iter->gfn); + } /* * If the page fault was caused by a write but the page is write From patchwork Fri Jul 30 22:37:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12412083 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1EC68C4338F for ; Fri, 30 Jul 2021 22:37:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0923560FE7 for ; Fri, 30 Jul 2021 22:37:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233516AbhG3Wh1 (ORCPT ); Fri, 30 Jul 2021 18:37:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52462 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233460AbhG3Wh0 (ORCPT ); Fri, 30 Jul 2021 18:37:26 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C6EC0C06175F for ; Fri, 30 Jul 2021 15:37:20 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id s123-20020a2577810000b02904f84a5c5297so12132027ybc.16 for ; Fri, 30 Jul 2021 15:37:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=dT/LFuHRBf+C472YuFJCuy5RtgpdJJjWKkeLjLnxKxA=; b=RNNXXejJfclzGPg5C/Iq75evsn6AF6dYeRc40IEs9g3Obv6ocIozNA8m4JXs/D+x/B kkDFMyvCIM4s6KCHC6j6hTRSrHn5OpxGnJgQz2HqgEgXXUEYbdaBwVrRSXlB9TfcKY2W E6kBBrdQHJWIpyNzz+FVh0GCSR7DrJNy+SMpg4zDpd4/kdpGjncESgbtbisg/x7RqWwv s+1bwp5OLztv3aZaxMDlQc6w3qV03grDqo9M6ZP+MjCdx2QwOZSLle9gQBAvz065FSiM RVqArbSsPunOnq/PBomgGRhqNqJtsmXABmWMymEq1Dhpv1RwKs9VWB+WD2NfLXsJGHG7 FpEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=dT/LFuHRBf+C472YuFJCuy5RtgpdJJjWKkeLjLnxKxA=; b=MtLzw3BftWXSbBzda0iN9QY8M9SSpp1MhqOVCAt1bz6zG6Zk4W7Hcfq8G/S63RQAxw jv12R0NwpjMCFv/HmEaAHS4O6Es8ONd91Y+zIMVIQlJiRAXwF/q3sN77NJizkCH9is7S IFM/BDwWjEF8aXB4oeCJBkq2fy4bZe59943LcSMoWdQOXGM9Df12V7VnO5jIW5knc/uw mMjQ4geysOpKtbQmEaL7lnEm5LLKR7uFYwKwhWH8hlHc8Qi4drTYV4eGmQaepr5hSWqf xBTgtVEJyBaHphuWBgPqm7+rU/Mt13RS1uDc+eqsoakwUYf0IASLhiwP/YU0Wq+736PG /z1Q== X-Gm-Message-State: AOAM530v6geBgAYtjgyZ3mRz+BZX9EnVJ0AfSfxjVao+/+Pir+Rj7t9Y gVNxxZWfWzb0VPwm3XlsDw2PDjXkaacJJi8TSxiFHu/VSHVQo+Invxb9zcogg+0EVFhkdSyoqoM RoKtVZ6gfXKWDqtcvLquITexDpBm8o+TQVI7COGP/QPnZP9gq61Idk2UcOp5YBSw= X-Google-Smtp-Source: ABdhPJxAMkvHC5nm2B+doIn5QGVay8C3OhySnZMSfi4ETddhbuY33oRb3sbBOAUpF8FnTHCw8B/9jJTycdG9Jw== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a25:e6d1:: with SMTP id d200mr5962463ybh.451.1627684639957; Fri, 30 Jul 2021 15:37:19 -0700 (PDT) Date: Fri, 30 Jul 2021 22:37:05 +0000 In-Reply-To: <20210730223707.4083785-1-dmatlack@google.com> Message-Id: <20210730223707.4083785-5-dmatlack@google.com> Mime-Version: 1.0 References: <20210730223707.4083785-1-dmatlack@google.com> X-Mailer: git-send-email 2.32.0.554.ge1b32706d8-goog Subject: [PATCH 4/6] KVM: x86/mmu: Leverage vcpu->lru_slot_index for rmap_add and rmap_recycle From: David Matlack To: kvm@vger.kernel.org Cc: Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Paolo Bonzini , Junaid Shahid , Andrew Jones , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org rmap_add() and rmap_recycle() both run in the context of the vCPU and thus we can use kvm_vcpu_gfn_to_memslot() to look up the memslot. This enables rmap_add() and rmap_recycle() to take advantage of vcpu->lru_slot_index and avoid expensive memslot searching. This change improves the performance of "Populate memory time" in dirty_log_perf_test with tdp_mmu=N. In addition to improving the performance, "Populate memory time" no longer scales with the number of memslots in the VM. Command | Before | After ------------------------------- | ---------------- | ------------- ./dirty_log_perf_test -v64 -x1 | 15.18001570s | 14.99469366s ./dirty_log_perf_test -v64 -x64 | 18.71336392s | 14.98675076s Signed-off-by: David Matlack Reviewed-by: Paolo Bonzini --- arch/x86/kvm/mmu/mmu.c | 35 ++++++++++++++++++++--------------- 1 file changed, 20 insertions(+), 15 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index a8cdfd8d45c4..370a6ebc2ede 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1043,17 +1043,6 @@ static struct kvm_rmap_head *__gfn_to_rmap(gfn_t gfn, int level, return &slot->arch.rmap[level - PG_LEVEL_4K][idx]; } -static struct kvm_rmap_head *gfn_to_rmap(struct kvm *kvm, gfn_t gfn, - struct kvm_mmu_page *sp) -{ - struct kvm_memslots *slots; - struct kvm_memory_slot *slot; - - slots = kvm_memslots_for_spte_role(kvm, sp->role); - slot = __gfn_to_memslot(slots, gfn); - return __gfn_to_rmap(gfn, sp->role.level, slot); -} - static bool rmap_can_add(struct kvm_vcpu *vcpu) { struct kvm_mmu_memory_cache *mc; @@ -1064,24 +1053,39 @@ static bool rmap_can_add(struct kvm_vcpu *vcpu) static int rmap_add(struct kvm_vcpu *vcpu, u64 *spte, gfn_t gfn) { + struct kvm_memory_slot *slot; struct kvm_mmu_page *sp; struct kvm_rmap_head *rmap_head; sp = sptep_to_sp(spte); kvm_mmu_page_set_gfn(sp, spte - sp->spt, gfn); - rmap_head = gfn_to_rmap(vcpu->kvm, gfn, sp); + slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn); + rmap_head = __gfn_to_rmap(gfn, sp->role.level, slot); return pte_list_add(vcpu, spte, rmap_head); } + static void rmap_remove(struct kvm *kvm, u64 *spte) { + struct kvm_memslots *slots; + struct kvm_memory_slot *slot; struct kvm_mmu_page *sp; gfn_t gfn; struct kvm_rmap_head *rmap_head; sp = sptep_to_sp(spte); gfn = kvm_mmu_page_get_gfn(sp, spte - sp->spt); - rmap_head = gfn_to_rmap(kvm, gfn, sp); + + /* + * Unlike rmap_add and rmap_recycle, rmap_remove does not run in the + * context of a vCPU so have to determine which memslots to use based + * on context information in sp->role. + */ + slots = kvm_memslots_for_spte_role(kvm, sp->role); + + slot = __gfn_to_memslot(slots, gfn); + rmap_head = __gfn_to_rmap(gfn, sp->role.level, slot); + __pte_list_remove(spte, rmap_head); } @@ -1628,12 +1632,13 @@ static bool kvm_test_age_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head, static void rmap_recycle(struct kvm_vcpu *vcpu, u64 *spte, gfn_t gfn) { + struct kvm_memory_slot *slot; struct kvm_rmap_head *rmap_head; struct kvm_mmu_page *sp; sp = sptep_to_sp(spte); - - rmap_head = gfn_to_rmap(vcpu->kvm, gfn, sp); + slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn); + rmap_head = __gfn_to_rmap(gfn, sp->role.level, slot); kvm_unmap_rmapp(vcpu->kvm, rmap_head, NULL, gfn, sp->role.level, __pte(0)); kvm_flush_remote_tlbs_with_address(vcpu->kvm, sp->gfn, From patchwork Fri Jul 30 22:37:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12412085 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9360C4320A for ; Fri, 30 Jul 2021 22:37:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C99C060FE7 for ; Fri, 30 Jul 2021 22:37:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232746AbhG3Wh3 (ORCPT ); Fri, 30 Jul 2021 18:37:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52474 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233322AbhG3Wh1 (ORCPT ); Fri, 30 Jul 2021 18:37:27 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 58514C06175F for ; Fri, 30 Jul 2021 15:37:22 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id f62-20020a17090a28c4b02901733dbfa29cso15101167pjd.0 for ; Fri, 30 Jul 2021 15:37:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=C8wsi/YpXV8rtnPArvQuTmmEPgRsr+T0f2WmE+1tIKc=; b=W2tOXIonhbyUDII+EidcljCQOcSIe5iUE2ZrdgCTlFu7041m4I02VARVFkyT/GkvnF UqEBFvKWdKcId+x9qxqUd4x+lXSIpNLmDUHHgVDo01GNi3Je+dBRP01Nmi4lAkiCjzAo b/GhgAZeFernKZ2OBbYJMztIkq+/YiVttFivbm9auFs9aZOpWPzeUOQgaiHwo5w71Hys WlV/RIbbzWCxSA6IFQssu9ODOtzoHe/Wzfhx/254j00JfaU+VkLJ8JlrY1GF2tl0CE5l xwPRsqrwBkGkcgP2FeCh40ebCt/RG4Vog7J+gtrDuLQ6DaVmDucGctbYp6F02l5yWzGx dzJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=C8wsi/YpXV8rtnPArvQuTmmEPgRsr+T0f2WmE+1tIKc=; b=mQ5KS9vumYKnAZCaQzRZy9QPemvIdYEPmxjC7Hq9hfMI6jkIhx9IgKSheTjhZnSq8m M6/Dt7KmsDCnbhtduqcRov0CeCKJ9B6EbEyZGHXldxbqk467LbBM6P9uGbjLpCgzPrPN rSaZFldU+5xCQRKHWcJkm4Rt8wcXoeXJxvxDI9hZWLhIRA+n1jBoSkG+JhaN0uXfW9xY YOD5QCU4bgP8pNcBU6sSZO6e2NdKEdbB2OzesLvCWakuRti5lOxvRLUs6rSYyS37XPPU jBvYrNM7CPUI9FcZM4XMydUQ16++se+4eJdU0DpNyKDaTcxkNC5EnzE4XPA2OQ7Z/Gnp Dvxg== X-Gm-Message-State: AOAM5310jOpIE8ix0DU3scxxA7J6Xe6CHJEbVi0EQCnXY7v57kTZ0XVa F+SC8xyoGLuyNu2J/AY6Jx5us6Vjcqhe0u//oHXV0hUU0q55ACzEg8tBNSbUq7XPxzjKZp1M1Tp A8ESjq314PfK9AsSh/71t0rbNGYzuI6FS6NGj7XvMu4kUQsSITSUUt+KTi3YkyDI= X-Google-Smtp-Source: ABdhPJyykJ4sAN4ft9cJ3uqJICBj9Be2CtGVlA9sX1e+0ejfcLAS7IcyqJ1Xqz3tbezEKqlhYWlNYLWDqNGlpw== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:903:300d:b029:12c:916f:fedd with SMTP id o13-20020a170903300db029012c916ffeddmr4259981pla.19.1627684641735; Fri, 30 Jul 2021 15:37:21 -0700 (PDT) Date: Fri, 30 Jul 2021 22:37:06 +0000 In-Reply-To: <20210730223707.4083785-1-dmatlack@google.com> Message-Id: <20210730223707.4083785-6-dmatlack@google.com> Mime-Version: 1.0 References: <20210730223707.4083785-1-dmatlack@google.com> X-Mailer: git-send-email 2.32.0.554.ge1b32706d8-goog Subject: [PATCH 5/6] KVM: x86/mmu: Rename __gfn_to_rmap to gfn_to_rmap From: David Matlack To: kvm@vger.kernel.org Cc: Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Paolo Bonzini , Junaid Shahid , Andrew Jones , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org gfn_to_rmap was removed in the previous patch so there is no need to retain the double underscore on __gfn_to_rmap. Signed-off-by: David Matlack Reported-by: kernel test robot Reported-by: kernel test robot Reviewed-by: Paolo Bonzini --- arch/x86/kvm/mmu/mmu.c | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 370a6ebc2ede..df493729d86c 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1034,8 +1034,8 @@ static bool pte_list_destroy(struct kvm_rmap_head *rmap_head, return true; } -static struct kvm_rmap_head *__gfn_to_rmap(gfn_t gfn, int level, - const struct kvm_memory_slot *slot) +static struct kvm_rmap_head *gfn_to_rmap(gfn_t gfn, int level, + const struct kvm_memory_slot *slot) { unsigned long idx; @@ -1060,7 +1060,7 @@ static int rmap_add(struct kvm_vcpu *vcpu, u64 *spte, gfn_t gfn) sp = sptep_to_sp(spte); kvm_mmu_page_set_gfn(sp, spte - sp->spt, gfn); slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn); - rmap_head = __gfn_to_rmap(gfn, sp->role.level, slot); + rmap_head = gfn_to_rmap(gfn, sp->role.level, slot); return pte_list_add(vcpu, spte, rmap_head); } @@ -1084,7 +1084,7 @@ static void rmap_remove(struct kvm *kvm, u64 *spte) slots = kvm_memslots_for_spte_role(kvm, sp->role); slot = __gfn_to_memslot(slots, gfn); - rmap_head = __gfn_to_rmap(gfn, sp->role.level, slot); + rmap_head = gfn_to_rmap(gfn, sp->role.level, slot); __pte_list_remove(spte, rmap_head); } @@ -1306,8 +1306,8 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm, return; while (mask) { - rmap_head = __gfn_to_rmap(slot->base_gfn + gfn_offset + __ffs(mask), - PG_LEVEL_4K, slot); + rmap_head = gfn_to_rmap(slot->base_gfn + gfn_offset + __ffs(mask), + PG_LEVEL_4K, slot); __rmap_write_protect(kvm, rmap_head, false); /* clear the first set bit */ @@ -1339,8 +1339,8 @@ static void kvm_mmu_clear_dirty_pt_masked(struct kvm *kvm, return; while (mask) { - rmap_head = __gfn_to_rmap(slot->base_gfn + gfn_offset + __ffs(mask), - PG_LEVEL_4K, slot); + rmap_head = gfn_to_rmap(slot->base_gfn + gfn_offset + __ffs(mask), + PG_LEVEL_4K, slot); __rmap_clear_dirty(kvm, rmap_head, slot); /* clear the first set bit */ @@ -1406,7 +1406,7 @@ bool kvm_mmu_slot_gfn_write_protect(struct kvm *kvm, if (kvm_memslots_have_rmaps(kvm)) { for (i = min_level; i <= KVM_MAX_HUGEPAGE_LEVEL; ++i) { - rmap_head = __gfn_to_rmap(gfn, i, slot); + rmap_head = gfn_to_rmap(gfn, i, slot); write_protected |= __rmap_write_protect(kvm, rmap_head, true); } } @@ -1506,9 +1506,8 @@ rmap_walk_init_level(struct slot_rmap_walk_iterator *iterator, int level) { iterator->level = level; iterator->gfn = iterator->start_gfn; - iterator->rmap = __gfn_to_rmap(iterator->gfn, level, iterator->slot); - iterator->end_rmap = __gfn_to_rmap(iterator->end_gfn, level, - iterator->slot); + iterator->rmap = gfn_to_rmap(iterator->gfn, level, iterator->slot); + iterator->end_rmap = gfn_to_rmap(iterator->end_gfn, level, iterator->slot); } static void @@ -1638,7 +1637,7 @@ static void rmap_recycle(struct kvm_vcpu *vcpu, u64 *spte, gfn_t gfn) sp = sptep_to_sp(spte); slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn); - rmap_head = __gfn_to_rmap(gfn, sp->role.level, slot); + rmap_head = gfn_to_rmap(gfn, sp->role.level, slot); kvm_unmap_rmapp(vcpu->kvm, rmap_head, NULL, gfn, sp->role.level, __pte(0)); kvm_flush_remote_tlbs_with_address(vcpu->kvm, sp->gfn, From patchwork Fri Jul 30 22:37:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Matlack X-Patchwork-Id: 12412087 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 916F6C4338F for ; Fri, 30 Jul 2021 22:37:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 77ACF60F42 for ; Fri, 30 Jul 2021 22:37:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231189AbhG3Wha (ORCPT ); Fri, 30 Jul 2021 18:37:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52484 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233532AbhG3Wh3 (ORCPT ); Fri, 30 Jul 2021 18:37:29 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D0C3EC061765 for ; Fri, 30 Jul 2021 15:37:23 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id v9-20020a17090a7c09b02901778a2a8fd6so3616951pjf.3 for ; Fri, 30 Jul 2021 15:37:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=2aoVz7znREQKWV9lVbXQeoyZokiTowDuQPpHDCPdpA4=; b=mfd2dcSY+xxaFitbUK4muN+w6fmLX7dohJ22U9MPcdGpwcjshJlA0DIWffHCm0y6bB 3zhmnGby+5xCDwEPVOBey2NBI1StJ0574Crc6pQc1Y2Cq/e/YCtwSb9c880VWmftET1e R+rapN1N0CL2WuUHE/6xOKeRMi9vIpASLmwIsRZOz/LNgUSFmfHdxRLDPBODVZO0rFzV 1NPu7IbfMm8O/BeMYwN0KoOpWCHZNgteWrhCIRd9hX0eAThFua8ly5K9w+lFW/m/N9u3 mWQg6gOOnSWlr+E6OMnibs9KxXScUCuhn0NZEmcsgIt0A6yur5lrX5cmrtwvSnRfKORI 0Bgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=2aoVz7znREQKWV9lVbXQeoyZokiTowDuQPpHDCPdpA4=; b=tnNnSAxB0zMiBcZK8hZ0MTipZbrZb7JohbJ9XVtRZlMRsU0xuoFwba2gs/IeWBs3L/ JiPuu2Dvgox5n7nTrmVCQG6XLJtHi+MXcfTLn7n8GhSMzr0yWhMIROSZa0BZRtv/efbX mypHqzLOW9ykcORDZuZiSqHuc/4U0jQEQ7vWdGalW6KgbrwEE7NrO1IzniesSWaoeK7U IACaniJ55Ac561mDDyXuTAkZzEboRvdHDk6uksYph+96kDnt0B98H9YpCW2k+yQ6ZzMg JEJe7cTYLXOTm6bUG3ywd/lLtHn3oq7XncnEW+C9HF4Vwofx9L/2sivzv+h6dl77diAj 4zYA== X-Gm-Message-State: AOAM532WtSm5RBvyvysXVB+rAT3ExC2g2zMpT9yglEeIWRA/Udv151ZI RXor9TNE8EZHnCHF54PAcFvCSDg/UCFhHf0RORI+T3lqz2FSJi0Cbu/RCEYwdSKVefrukoMacwG cf8xxKy8KD9aOp0tWM31LmdkmKYxilBv9MDbweZ8JX6hxAj4PEvrPkvJNovSqTjo= X-Google-Smtp-Source: ABdhPJyZdW12btC6KWIPiwDEZZ2cNu6fj4nIlKhVG0AwtGlj/lfNiJvKiDHlfeYxcJS6sTtvF1z7g7IjRcOTzQ== X-Received: from dmatlack-heavy.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:19cd]) (user=dmatlack job=sendgmr) by 2002:a17:902:bc82:b029:12b:a074:1fae with SMTP id bb2-20020a170902bc82b029012ba0741faemr2755861plb.29.1627684643279; Fri, 30 Jul 2021 15:37:23 -0700 (PDT) Date: Fri, 30 Jul 2021 22:37:07 +0000 In-Reply-To: <20210730223707.4083785-1-dmatlack@google.com> Message-Id: <20210730223707.4083785-7-dmatlack@google.com> Mime-Version: 1.0 References: <20210730223707.4083785-1-dmatlack@google.com> X-Mailer: git-send-email 2.32.0.554.ge1b32706d8-goog Subject: [PATCH 6/6] KVM: selftests: Support multiple slots in dirty_log_perf_test From: David Matlack To: kvm@vger.kernel.org Cc: Ben Gardon , Joerg Roedel , Jim Mattson , Wanpeng Li , Vitaly Kuznetsov , Sean Christopherson , Paolo Bonzini , Junaid Shahid , Andrew Jones , David Matlack Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Introduce a new option to dirty_log_perf_test: -x number_of_slots. This causes the test to attempt to split the region of memory into the given number of slots. If the region cannot be evenly divided, the test will fail. This allows testing with more than one slot and therefore measure how performance scales with the number of memslots. Signed-off-by: David Matlack --- .../selftests/kvm/access_tracking_perf_test.c | 2 +- .../selftests/kvm/demand_paging_test.c | 2 +- .../selftests/kvm/dirty_log_perf_test.c | 76 ++++++++++++++++--- .../selftests/kvm/include/perf_test_util.h | 2 +- .../selftests/kvm/lib/perf_test_util.c | 20 +++-- .../kvm/memslot_modification_stress_test.c | 2 +- 6 files changed, 84 insertions(+), 20 deletions(-) diff --git a/tools/testing/selftests/kvm/access_tracking_perf_test.c b/tools/testing/selftests/kvm/access_tracking_perf_test.c index e2baa187a21e..3e23b2105f4b 100644 --- a/tools/testing/selftests/kvm/access_tracking_perf_test.c +++ b/tools/testing/selftests/kvm/access_tracking_perf_test.c @@ -333,7 +333,7 @@ static void run_test(enum vm_guest_mode mode, void *arg) pthread_t *vcpu_threads; int vcpus = params->vcpus; - vm = perf_test_create_vm(mode, vcpus, params->vcpu_memory_bytes, + vm = perf_test_create_vm(mode, vcpus, params->vcpu_memory_bytes, 1, params->backing_src); perf_test_setup_vcpus(vm, vcpus, params->vcpu_memory_bytes, diff --git a/tools/testing/selftests/kvm/demand_paging_test.c b/tools/testing/selftests/kvm/demand_paging_test.c index b74704305835..61266a729d88 100644 --- a/tools/testing/selftests/kvm/demand_paging_test.c +++ b/tools/testing/selftests/kvm/demand_paging_test.c @@ -293,7 +293,7 @@ static void run_test(enum vm_guest_mode mode, void *arg) int vcpu_id; int r; - vm = perf_test_create_vm(mode, nr_vcpus, guest_percpu_mem_size, + vm = perf_test_create_vm(mode, nr_vcpus, guest_percpu_mem_size, 1, p->src_type); perf_test_args.wr_fract = 1; diff --git a/tools/testing/selftests/kvm/dirty_log_perf_test.c b/tools/testing/selftests/kvm/dirty_log_perf_test.c index 80cbd3a748c0..034458dd89a2 100644 --- a/tools/testing/selftests/kvm/dirty_log_perf_test.c +++ b/tools/testing/selftests/kvm/dirty_log_perf_test.c @@ -94,8 +94,59 @@ struct test_params { int wr_fract; bool partition_vcpu_memory_access; enum vm_mem_backing_src_type backing_src; + int slots; }; +static void toggle_dirty_logging(struct kvm_vm *vm, int slots, bool enable) +{ + int i; + + for (i = 0; i < slots; i++) { + int slot = PERF_TEST_MEM_SLOT_INDEX + i; + int flags = enable ? KVM_MEM_LOG_DIRTY_PAGES : 0; + + vm_mem_region_set_flags(vm, slot, flags); + } +} + +static inline void enable_dirty_logging(struct kvm_vm *vm, int slots) +{ + toggle_dirty_logging(vm, slots, true); +} + +static inline void disable_dirty_logging(struct kvm_vm *vm, int slots) +{ + toggle_dirty_logging(vm, slots, false); +} + +static void get_dirty_log(struct kvm_vm *vm, int slots, unsigned long *bitmap, + uint64_t nr_pages) +{ + uint64_t slot_pages = nr_pages / slots; + int i; + + for (i = 0; i < slots; i++) { + int slot = PERF_TEST_MEM_SLOT_INDEX + i; + unsigned long *slot_bitmap = bitmap + i * slot_pages; + + kvm_vm_get_dirty_log(vm, slot, slot_bitmap); + } +} + +static void clear_dirty_log(struct kvm_vm *vm, int slots, unsigned long *bitmap, + uint64_t nr_pages) +{ + uint64_t slot_pages = nr_pages / slots; + int i; + + for (i = 0; i < slots; i++) { + int slot = PERF_TEST_MEM_SLOT_INDEX + i; + unsigned long *slot_bitmap = bitmap + i * slot_pages; + + kvm_vm_clear_dirty_log(vm, slot, slot_bitmap, 0, slot_pages); + } +} + static void run_test(enum vm_guest_mode mode, void *arg) { struct test_params *p = arg; @@ -114,7 +165,7 @@ static void run_test(enum vm_guest_mode mode, void *arg) struct timespec clear_dirty_log_total = (struct timespec){0}; vm = perf_test_create_vm(mode, nr_vcpus, guest_percpu_mem_size, - p->backing_src); + p->slots, p->backing_src); perf_test_args.wr_fract = p->wr_fract; @@ -163,8 +214,7 @@ static void run_test(enum vm_guest_mode mode, void *arg) /* Enable dirty logging */ clock_gettime(CLOCK_MONOTONIC, &start); - vm_mem_region_set_flags(vm, PERF_TEST_MEM_SLOT_INDEX, - KVM_MEM_LOG_DIRTY_PAGES); + enable_dirty_logging(vm, p->slots); ts_diff = timespec_elapsed(start); pr_info("Enabling dirty logging time: %ld.%.9lds\n\n", ts_diff.tv_sec, ts_diff.tv_nsec); @@ -190,8 +240,7 @@ static void run_test(enum vm_guest_mode mode, void *arg) iteration, ts_diff.tv_sec, ts_diff.tv_nsec); clock_gettime(CLOCK_MONOTONIC, &start); - kvm_vm_get_dirty_log(vm, PERF_TEST_MEM_SLOT_INDEX, bmap); - + get_dirty_log(vm, p->slots, bmap, host_num_pages); ts_diff = timespec_elapsed(start); get_dirty_log_total = timespec_add(get_dirty_log_total, ts_diff); @@ -200,9 +249,7 @@ static void run_test(enum vm_guest_mode mode, void *arg) if (dirty_log_manual_caps) { clock_gettime(CLOCK_MONOTONIC, &start); - kvm_vm_clear_dirty_log(vm, PERF_TEST_MEM_SLOT_INDEX, bmap, 0, - host_num_pages); - + clear_dirty_log(vm, p->slots, bmap, host_num_pages); ts_diff = timespec_elapsed(start); clear_dirty_log_total = timespec_add(clear_dirty_log_total, ts_diff); @@ -213,7 +260,7 @@ static void run_test(enum vm_guest_mode mode, void *arg) /* Disable dirty logging */ clock_gettime(CLOCK_MONOTONIC, &start); - vm_mem_region_set_flags(vm, PERF_TEST_MEM_SLOT_INDEX, 0); + disable_dirty_logging(vm, p->slots); ts_diff = timespec_elapsed(start); pr_info("Disabling dirty logging time: %ld.%.9lds\n", ts_diff.tv_sec, ts_diff.tv_nsec); @@ -244,7 +291,8 @@ static void help(char *name) { puts(""); printf("usage: %s [-h] [-i iterations] [-p offset] " - "[-m mode] [-b vcpu bytes] [-v vcpus] [-o] [-s mem type]\n", name); + "[-m mode] [-b vcpu bytes] [-v vcpus] [-o] [-s mem type]" + "[-x memslots]\n", name); puts(""); printf(" -i: specify iteration counts (default: %"PRIu64")\n", TEST_HOST_LOOP_N); @@ -263,6 +311,8 @@ static void help(char *name) " them into a separate region of memory for each vCPU.\n"); printf(" -s: specify the type of memory that should be used to\n" " back the guest data region.\n\n"); + printf(" -x: Split the memory region into this number of memslots.\n" + " (default: 1)"); backing_src_help(); puts(""); exit(0); @@ -276,6 +326,7 @@ int main(int argc, char *argv[]) .wr_fract = 1, .partition_vcpu_memory_access = true, .backing_src = VM_MEM_SRC_ANONYMOUS, + .slots = 1, }; int opt; @@ -286,7 +337,7 @@ int main(int argc, char *argv[]) guest_modes_append_default(); - while ((opt = getopt(argc, argv, "hi:p:m:b:f:v:os:")) != -1) { + while ((opt = getopt(argc, argv, "hi:p:m:b:f:v:os:x:")) != -1) { switch (opt) { case 'i': p.iterations = atoi(optarg); @@ -316,6 +367,9 @@ int main(int argc, char *argv[]) case 's': p.backing_src = parse_backing_src_type(optarg); break; + case 'x': + p.slots = atoi(optarg); + break; case 'h': default: help(argv[0]); diff --git a/tools/testing/selftests/kvm/include/perf_test_util.h b/tools/testing/selftests/kvm/include/perf_test_util.h index 005f2143adeb..df9f1a3a3ffb 100644 --- a/tools/testing/selftests/kvm/include/perf_test_util.h +++ b/tools/testing/selftests/kvm/include/perf_test_util.h @@ -44,7 +44,7 @@ extern struct perf_test_args perf_test_args; extern uint64_t guest_test_phys_mem; struct kvm_vm *perf_test_create_vm(enum vm_guest_mode mode, int vcpus, - uint64_t vcpu_memory_bytes, + uint64_t vcpu_memory_bytes, int slots, enum vm_mem_backing_src_type backing_src); void perf_test_destroy_vm(struct kvm_vm *vm); void perf_test_setup_vcpus(struct kvm_vm *vm, int vcpus, diff --git a/tools/testing/selftests/kvm/lib/perf_test_util.c b/tools/testing/selftests/kvm/lib/perf_test_util.c index b488f4aefea8..aebb223d34a7 100644 --- a/tools/testing/selftests/kvm/lib/perf_test_util.c +++ b/tools/testing/selftests/kvm/lib/perf_test_util.c @@ -50,11 +50,12 @@ static void guest_code(uint32_t vcpu_id) } struct kvm_vm *perf_test_create_vm(enum vm_guest_mode mode, int vcpus, - uint64_t vcpu_memory_bytes, + uint64_t vcpu_memory_bytes, int slots, enum vm_mem_backing_src_type backing_src) { struct kvm_vm *vm; uint64_t guest_num_pages; + int i; pr_info("Testing guest mode: %s\n", vm_guest_mode_string(mode)); @@ -68,6 +69,9 @@ struct kvm_vm *perf_test_create_vm(enum vm_guest_mode mode, int vcpus, "Guest memory size is not host page size aligned."); TEST_ASSERT(vcpu_memory_bytes % perf_test_args.guest_page_size == 0, "Guest memory size is not guest page size aligned."); + TEST_ASSERT(guest_num_pages % slots == 0, + "Guest memory cannot be evenly divided into %d slots.", + slots); vm = vm_create_with_vcpus(mode, vcpus, DEFAULT_GUEST_PHY_PAGES, (vcpus * vcpu_memory_bytes) / perf_test_args.guest_page_size, @@ -95,10 +99,16 @@ struct kvm_vm *perf_test_create_vm(enum vm_guest_mode mode, int vcpus, #endif pr_info("guest physical test memory offset: 0x%lx\n", guest_test_phys_mem); - /* Add an extra memory slot for testing */ - vm_userspace_mem_region_add(vm, backing_src, guest_test_phys_mem, - PERF_TEST_MEM_SLOT_INDEX, - guest_num_pages, 0); + /* Add extra memory slots for testing */ + for (i = 0; i < slots; i++) { + uint64_t region_pages = guest_num_pages / slots; + vm_paddr_t region_start = guest_test_phys_mem + + region_pages * perf_test_args.guest_page_size * i; + + vm_userspace_mem_region_add(vm, backing_src, region_start, + PERF_TEST_MEM_SLOT_INDEX + i, + region_pages, 0); + } /* Do mapping for the demand paging memory slot */ virt_map(vm, guest_test_virt_mem, guest_test_phys_mem, guest_num_pages); diff --git a/tools/testing/selftests/kvm/memslot_modification_stress_test.c b/tools/testing/selftests/kvm/memslot_modification_stress_test.c index 98351ba0933c..8a9c6ccce3ca 100644 --- a/tools/testing/selftests/kvm/memslot_modification_stress_test.c +++ b/tools/testing/selftests/kvm/memslot_modification_stress_test.c @@ -105,7 +105,7 @@ static void run_test(enum vm_guest_mode mode, void *arg) struct kvm_vm *vm; int vcpu_id; - vm = perf_test_create_vm(mode, nr_vcpus, guest_percpu_mem_size, + vm = perf_test_create_vm(mode, nr_vcpus, guest_percpu_mem_size, 1, VM_MEM_SRC_ANONYMOUS); perf_test_args.wr_fract = 1;