From patchwork Tue Jun 11 00:21:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13692684 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B41AC27C4F for ; Tue, 11 Jun 2024 00:22:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 75FE16B0083; Mon, 10 Jun 2024 20:22:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 736016B0085; Mon, 10 Jun 2024 20:22:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5D6306B0088; Mon, 10 Jun 2024 20:22:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 3E62A6B0083 for ; Mon, 10 Jun 2024 20:22:02 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id B63FE410BA for ; Tue, 11 Jun 2024 00:22:01 +0000 (UTC) X-FDA: 82216705242.09.B169AF0 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf22.hostedemail.com (Postfix) with ESMTP id 0330EC0012 for ; Tue, 11 Jun 2024 00:21:59 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=23+nGKJS; spf=pass (imf22.hostedemail.com: domain of 3p5hnZgoKCLAZjXekWXjedWeeWbU.SecbYdkn-ccalQSa.ehW@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3p5hnZgoKCLAZjXekWXjedWeeWbU.SecbYdkn-ccalQSa.ehW@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718065320; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=znnebkeVSvsFTLm00akjOWCV2leg9gyqu/Y0G491TkM=; b=c5lQJMygCvJmaHvIW5kv1iIY3VpNSaPB+yV+gHxMKekK1eLV1zMI5zcopDE269z7/a8GTd hYeF2lo9GFh1dBj17zXGzQ2x2blqvm/Upw73FFGlEuhRNzFkE3jN7VkaDo6QJuT/hnwXZv RWp3E/ZYzow63GPW178CfzQIt/QqIjU= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=23+nGKJS; spf=pass (imf22.hostedemail.com: domain of 3p5hnZgoKCLAZjXekWXjedWeeWbU.SecbYdkn-ccalQSa.ehW@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3p5hnZgoKCLAZjXekWXjedWeeWbU.SecbYdkn-ccalQSa.ehW@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718065320; a=rsa-sha256; cv=none; b=w+q0uYDFdEw/5sVlTj933dpP5/pxPxsi5+G2yfQNI9zsZAFzCycFY6DVxp7RFqBTFnlvQv jWTPgoAMYWp+X+z/XlIoP932ny8ZWEYaU4obgkTPjIvglXOQiYRVzBX5Dk41AYuw8Cu8JU +G/ZTuTYX1e9oihfBv2AWKGu874O8us= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-627f382fb97so95025847b3.3 for ; Mon, 10 Jun 2024 17:21:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1718065319; x=1718670119; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=znnebkeVSvsFTLm00akjOWCV2leg9gyqu/Y0G491TkM=; b=23+nGKJSutDx5rAbLLg3b3oNhwoyikZl2A91VyExWSvUgpQvR5jnboYpKf4MICOD+J 4qqwD2XgozOYjSIfCEohMkCMe9op7HLhWsp/ZeYAhnS4Lx9eDFZJ3Tf8oLwCgokGHpaW e9kCE3XZrqgIWovZT348ck3Mtuo1CN44Txg9IVifIME5jQR4c7GaaRzSJ2CHewo/NG5L f/kvH5W8vFsSmohByE0MaXY1mX4MKq0CjfQHH/A/dwFcyTKSxyETilZOSdkXth++5ehh 1turF+v2YrM4laiw0HncuL/8Ze4UKfgMCufWpge1Jh7+iBwtJosT47fULVI/Zv3zWdL8 2W9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718065319; x=1718670119; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=znnebkeVSvsFTLm00akjOWCV2leg9gyqu/Y0G491TkM=; b=EB40qPVSY2Cdqcr0eovoYbpd+sNa0qOyMHQjVRZss++thX31tb+vjcz+ihEjuzTwlQ tGDURJqIu5Btqrsj6V/3ZqBoExoypwi70694y8sNxsBVFAyCFtOTqgNnR1IL3fEhiJat YGFmiIUMoX6oqzoDYLDNKUeKh+AQxfucYecSvplTBp0xdZlQvSebUhWZxRen0xX7qFqn TFqAFVBKpkK4TXTWiX+fDxZiGK29rSXQeLmx5/QC52A5fHmqypJh9fkyUItTuMKSvV+D Q4fPxnDIDw/5KN5gY9Nr+CrQh6k84F05TL6Z3myazng7maM1yyJzdGXZypxGTIGu282E fRcA== X-Forwarded-Encrypted: i=1; AJvYcCU6/FmaQhFJ6YVUstb65BbqepAYqSPYK9Fe/JVMYnb/fSiKSqs3DyV+eUX0malymI/Zozzgpbe6/owpnryGCsg7gHI= X-Gm-Message-State: AOJu0YzPbxdA7PMqx1jDi6ihGLWlGcIO27LRcdcFIHlm3TuWIJEDSbEC K1Gesv3l65wrnMKPwsjFs2EquG+lWjHlEmnm/oayblIHsELyBtESXK6ZeDZql2eXcNUkQx6iTMF tPTqhFHfvq9CaacOz+g== X-Google-Smtp-Source: AGHT+IHAQpzXE/JP1O7Hg4+Kq/h9j2tVw1U1SGU5I2M7dDYM3/uT/d6pwcmk2+n0+ICjilMbsMpRbG4g7tWO2ExY X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:690c:6612:b0:62d:21:3f66 with SMTP id 00721157ae682-62d0021471dmr22261127b3.1.1718065319010; Mon, 10 Jun 2024 17:21:59 -0700 (PDT) Date: Tue, 11 Jun 2024 00:21:36 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.45.2.505.gda0bf45e8d-goog Message-ID: <20240611002145.2078921-1-jthoughton@google.com> Subject: [PATCH v5 0/9] mm: multi-gen LRU: Walk secondary MMU page tables while aging From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Ankit Agrawal , Axel Rasmussen , Catalin Marinas , David Matlack , David Rientjes , James Houghton , James Morse , Jonathan Corbet , Marc Zyngier , Oliver Upton , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Suzuki K Poulose , Wei Xu , Will Deacon , Yu Zhao , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspam-User: X-Rspamd-Queue-Id: 0330EC0012 X-Rspamd-Server: rspam01 X-Stat-Signature: jpbr7roycc95bigsiy9d6k7a1aymojss X-HE-Tag: 1718065319-351751 X-HE-Meta: U2FsdGVkX1+3ojsc9ulBk9YY/zfNnZ01tEMPEkJLsg8OSX2tk3ysE1I0PwDu1lj7KOvogsjA3YPfoV2KCBwRuRgjfADCQLrvzOy4lmXMgOOkstdUBSbGafEuY5o5c/S+vJCwn6gZm/Xq1yXo0OF5Q0OQMFOYNRUubh+jvu7hvSxli5K4K78lJywNn4NVSLOouvEmG/hLTjnTKzRWaCp8kC4xZOnk5dsLE572iweOF/UarpDmu+4+6JQp1oi0RlJVhCHTQ4gxVm9knkM2rpAauVrkcZscxli3O5dXvlf9PjffoHSqiRItndi53/qXR+P7zzXWoJcDuKmfG8yODigjtS5QBItzR4KqmhJHxz7yAoqGz1x9hq9kXaJe2I1FL7LSHoOLo7nwNOlQ3zpZ8TeuM8W3gWlRJfr002cXOsm4HJiTq7iMcPALyLRxCeyqL2KzGedfJgxV7LUgNTqRNbxLgbuMCTFI56aV8i4/ZwZK0DwYIJv9cZZBqJb6imnaXAku0w6/pLFSOQ5DuU5nsgs4ypkFcRofOYfZ304fwiPhXKv0WBuh3KOYBrabCiTh+Z70uIUbWmngXacthZ2Ibn6sPPTaa013ncXkqpfB39Nx1o0LiFdPaQ2Rg55NHYPr/CsUPIawlXKJqEohc6Z1kGoHtnEWSk0CMxjSiSNjeq5Qo+isZXQT8eYNBza2ehwLg4PeCwXSXB9tNb1eNPqWnO1bcTQx36p44i764hxgXkBhB4zlsIa6iirs2KRqN6mz7DtY3yiWtvVe4B4omnTVWylF6IxL9Pxe8wVrMfyPGEHA56oHw1zD497jkpLSPLWjladzatqmg0BTQtow4zX5mDvYOymwk8crD6Prh/MpAyv71P1dzzdU82VIg9LQEo6SkvFIzazniDcwA460nzZtPuBxPa7Tpe2/EGJKzQfKYroQrtlDQOvaY7NffKYflTqjgLYXxQhYJRnfrXyEC7mSGxD G/7wqXPa 05X/ZYu9R9RTsr6u/R8w6VZbkIlRifiCPLGxtVvH8wW0usTNihSKZOco5TByX5pilSJpaYuBF/lPOXYhyvAEUjUXHEw+IMFED2yI7hKs3oPYkDGKtGF1RrwSIDx5tv9zFWEZQpF5jpF17zVbKmAtpSxyxwfW3wDm8zZ9ADeRFsm03Ag0M+jjhjkaaGd/8KxcfG8uBY/UoVcxDPI4o0M1KqYeoFRhaFEyFqY/im/8Oh+9jDzg3J5KO5OrEHaO4xsFVKpivlpEEjPXRdVLhdwHv2Bsu5eCLt+p7cwGoSZ2b/FyTrmk61K7YXXdO1r5nktpLlBuTh/RGLS+eBg3OzcntcF/ymvM8JcIJWNVQCF9wiYXvf4KkHk/ZCdF4sGtWqOAzjYwIm2s2v6rwYGbn434uNSyti8DEuhlPoba2KLw0F1Pf40hqr0y5o55WJS3XLqHYwQovGv9rxTmV1t+P3V1/cNvCRqKIFn64b6poRCi2LPc7PTDcGihF12rRxCry3NEdbQ1ZC5RaoZEJ1xMw+ZSbYqGp1ZpcQ7iuh3N6MjDzdT6AR1BEtlT4ydJ+iWv8baQs8xKPEWVIxAqQzeLLNEr0F8OdXoxQa2bTXqvW7hP0NV05EbY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patchset makes it possible for MGLRU to consult secondary MMUs while doing aging, not just during eviction. This allows for more accurate reclaim decisions, which is especially important for proactive reclaim. This series does the following: 1. Improve locking for the existing test/clear_young notifiers for x86 and arm64. 2. Add a new notifier test_clear_young_fast_only(), implemented only by KVM/x86. 3. Incorporate test_clear_young_fast_only() into MGLRU aging. To make aging work for more than just x86, the test_clear_young_fast_only() notifier must be implemented by those other architectures. access_tracking_perf_test now has a mode (-p) to check performance of MGLRU aging while the VM is faulting memory in. See the v4 cover letter[1] for performance data collected with this test. Previous versions of this series included logic in MGLRU and KVM to support batching the updates to secondary page tables. This version removes this logic, as it was complex and not necessary to enable proactive reclaim. This optimization, as well as the additional optimizations for arm64 and powerpc, can be done in a later series. === Previous Versions === This v5 re-adds a lot of logic that was present in v3 and earlier versions of the series. There is an important difference I want to call out: - should_look_around() can sometimes require two notifiers instead of one. This is necessary if I forbid myself from modifying mmu_notifier_clear_young(). It may simply be better to do what v2/v3 did have and not have a fast-only notifier, and merge them all. This makes the API slightly more complex. I'm not sure which is better. Change log: Since v4[1]: - Removed Kconfig that controlled when aging was enabled. Aging will be done whenever the architecture supports it (thanks Yu). - Added a new MMU notifier, test_clear_young_fast_only(), specifically for MGLRU to use. - Add kvm_fast_{test_,}age_gfn, implemented by x86. - Fix locking for clear_flush_young(). - Added KVM_MMU_NOTIFIER_YOUNG_LOCKLESS to clean up locking changes (thanks Sean). - Fix WARN_ON and other cleanup for the arm64 locking changes (thanks Oliver). Since v3[2]: - Vastly simplified the series (thanks David). Removed mmu notifier batching logic entirely. - Cleaned up how locking is done for mmu_notifier_test/clear_young (thanks David). - Look-around is now only done when there are no secondary MMUs subscribed to MMU notifiers. - CONFIG_LRU_GEN_WALKS_SECONDARY_MMU has been added. - Fixed the lockless implementation of kvm_{test,}age_gfn for x86 (thanks David). - Added MGLRU functional and performance tests to access_tracking_perf_test (thanks Axel). - In v3, an mm would be completely ignored (for aging) if there was a secondary MMU but support for secondary MMU walking was missing. Now, missing secondary MMU walking support simply skips the notifier calls (except for eviction). - Added a sanity check for that range->lockless and range->on_lock are never both provided for the memslot walk. For the changes since v2[3], see v3. Based on 6.10-rc3. [1]: https://lore.kernel.org/linux-mm/20240529180510.2295118-1-jthoughton@google.com/ [2]: https://lore.kernel.org/linux-mm/20240401232946.1837665-1-jthoughton@google.com/ [3]: https://lore.kernel.org/kvmarm/20230526234435.662652-1-yuzhao@google.com/ James Houghton (8): KVM: Add lockless memslot walk to KVM KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn mm: Add test_clear_young_fast_only MMU notifier KVM: Add kvm_fast_age_gfn and kvm_fast_test_age_gfn KVM: x86: Implement kvm_fast_test_age_gfn and kvm_fast_age_gfn mm: multi-gen LRU: Have secondary MMUs participate in aging KVM: selftests: Add multi-gen LRU aging to access_tracking_perf_test Yu Zhao (1): KVM: x86: Move tdp_mmu_enabled and shadow_accessed_mask Documentation/admin-guide/mm/multigen_lru.rst | 6 +- arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/hyp/pgtable.c | 15 +- arch/arm64/kvm/mmu.c | 26 +- arch/x86/include/asm/kvm_host.h | 14 + arch/x86/kvm/Kconfig | 2 + arch/x86/kvm/mmu.h | 6 - arch/x86/kvm/mmu/mmu.c | 60 ++- arch/x86/kvm/mmu/spte.h | 1 - arch/x86/kvm/mmu/tdp_iter.h | 27 +- arch/x86/kvm/mmu/tdp_mmu.c | 67 ++- include/linux/kvm_host.h | 8 + include/linux/mmu_notifier.h | 50 +++ include/linux/mmzone.h | 6 +- include/trace/events/kvm.h | 22 + mm/mmu_notifier.c | 26 ++ mm/rmap.c | 9 +- mm/vmscan.c | 185 +++++++-- tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/access_tracking_perf_test.c | 365 ++++++++++++++-- .../selftests/kvm/include/lru_gen_util.h | 55 +++ .../testing/selftests/kvm/lib/lru_gen_util.c | 391 ++++++++++++++++++ virt/kvm/Kconfig | 7 + virt/kvm/kvm_main.c | 73 +++- 24 files changed, 1283 insertions(+), 140 deletions(-) create mode 100644 tools/testing/selftests/kvm/include/lru_gen_util.h create mode 100644 tools/testing/selftests/kvm/lib/lru_gen_util.c base-commit: 83a7eefedc9b56fe7bfeff13b6c7356688ffa670