From patchwork Wed May 29 18:05:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13679363 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52C0CC25B75 for ; Wed, 29 May 2024 18:05:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E11916B009C; Wed, 29 May 2024 14:05:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DC0FF6B009D; Wed, 29 May 2024 14:05:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CB0496B009E; Wed, 29 May 2024 14:05:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id AEA106B009C for ; Wed, 29 May 2024 14:05:20 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 3EA7C160C0B for ; Wed, 29 May 2024 18:05:20 +0000 (UTC) X-FDA: 82172210400.29.73F7A30 Received: from mail-ua1-f73.google.com (mail-ua1-f73.google.com [209.85.222.73]) by imf21.hostedemail.com (Postfix) with ESMTP id 84BAF1C000E for ; Wed, 29 May 2024 18:05:18 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=m3qDfMRK; spf=pass (imf21.hostedemail.com: domain of 3XW5XZgoKCNA5F3AG23FA92AA270.yA8749GJ-886Hwy6.AD2@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3XW5XZgoKCNA5F3AG23FA92AA270.yA8749GJ-886Hwy6.AD2@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717005918; a=rsa-sha256; cv=none; b=5JuT8EdH3IYHT3Rzvzz91C2E2SZvH+2q5yOXb05hqwtUtEPMkFJxVs4kxMo04m27FsLUkw zc/hSS52mATB9DivB52YBVV0li2SlOMJDi/RicB6iLGu+nw68KKSsjbco3gImY6VhPBw4+ dFOKqBBKAX4ybxVzUhBlAtd05A1U7uw= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=m3qDfMRK; spf=pass (imf21.hostedemail.com: domain of 3XW5XZgoKCNA5F3AG23FA92AA270.yA8749GJ-886Hwy6.AD2@flex--jthoughton.bounces.google.com designates 209.85.222.73 as permitted sender) smtp.mailfrom=3XW5XZgoKCNA5F3AG23FA92AA270.yA8749GJ-886Hwy6.AD2@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717005918; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=oSeK4HIzvhbzZ4+kBhyxeRArFmVuKpeUMlCOBrkyZKY=; b=0AbVURy8QQ6jHXA1r0bhkS3/THnBjIDG7pQXBrOtidO0FobMbGYHfMiuAhEwF+Py4giFHR 5wXYWFZ/Df/Cgaus09EEdL5XGRix9NuPLKkfhfDJVUfc8yRPIsZbdRx9TJBPZ51nvMjYz3 rbI4mQSz8Rtvf3LBA5cT9y7g5YN41ss= Received: by mail-ua1-f73.google.com with SMTP id a1e0cc1a2514c-80ac7fc2ee1so20946241.0 for ; Wed, 29 May 2024 11:05:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717005917; x=1717610717; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=oSeK4HIzvhbzZ4+kBhyxeRArFmVuKpeUMlCOBrkyZKY=; b=m3qDfMRKKfFR92JE8vLC7ElIQpYG3MWI6sC1jDaPbUa4DFfHoZ+jYGdE9VZ2kkXm0T WkkVgkiTnG8lHH6TlrVta/Sn30WFIXaIOYvGPSGGNo3OKd7IPtUe1Eq0KUhvhio6bjZT ZTYEIKf/xcDfw2/Snl+oPj9Zga32eMWqoMjIHpOcxna4rfmdvx+5KP3SfYYzIFRmOSM8 O5dXSH/FKpvr64q2VDXy+ImjKrzgKWvLKHGLeYZ5CHvELoTj7OqgqDFGEiacIU29Qy4Q qbZNKy+l4SWTpgyXommArVzKljXA2WBO1S/x6G8EBdVNwYIUNm9ETM8aQgvhCQbEuSke ynwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717005917; x=1717610717; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=oSeK4HIzvhbzZ4+kBhyxeRArFmVuKpeUMlCOBrkyZKY=; b=f/v70uOaqvKwTzu02+iPFw7je5BoVuflcek2jGLxishShGd6SLrp9YzZb5VkS+RlrR 0dH1A5eEIOZ/0Fo81K2KAW2Bccv8GmpRW8RmJ257u3ThUcWqoDSpfLFaXgvmoF6uibAn 2o5GUgC1OM+8KWIZ6ISWgg/tKVmz01LcjIiFFTUCO385aCGvcyR4Pw8QVvEgX5Cl/Ipt pNkIIZxIcFkParmjItJRZc9cKsO291lFbKWb9j/IbYstBVNFdTAzbPfb6f9coBs50Cy/ jMJXu9gCVgLmVrGDm0lYYwJodu4ONyeiq+snhBSYikYwGuhTktKATIOx8f42FHAU1zGw UqCg== X-Forwarded-Encrypted: i=1; AJvYcCUifyWK1B5qe7VD134ucueGDpC4jVwXmJtrN5D1B0i/x4bz1DrCG6VYG12Bpe14zRQKIh1xT880FEhWGemgL48KshA= X-Gm-Message-State: AOJu0Yycq0xmU1JpJTP7YdjnzKcpl+Dx9TD/uwZ1exRzytx6a/wN4v52 AeEpnJg7K8rvOudc/oro4FLB0amUQm8OtcTtoNKxnebl/ejSUGvQG57eBYNBDFZ/0gYxN022YQv tnBvAcMR+7mqu707+1g== X-Google-Smtp-Source: AGHT+IGirSTR9frt+FMY7mEAkHJfvAwcaa/uk6kZfZAYRVHTIdMTLDcjTaGUq6V+fW4RjZ9NSYV+AI/l972bzeUv X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6102:3f9e:b0:48a:5834:cea7 with SMTP id ada2fe7eead31-48a5834e759mr882345137.7.1717005917523; Wed, 29 May 2024 11:05:17 -0700 (PDT) Date: Wed, 29 May 2024 18:05:03 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.45.1.288.g0e0cd299f1-goog Message-ID: <20240529180510.2295118-1-jthoughton@google.com> Subject: [PATCH v4 0/7] mm: multi-gen LRU: Walk secondary MMU page tables while aging From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Albert Ou , Ankit Agrawal , Anup Patel , Atish Patra , Axel Rasmussen , Bibo Mao , Catalin Marinas , David Matlack , David Rientjes , Huacai Chen , James Houghton , James Morse , Jonathan Corbet , Marc Zyngier , Michael Ellerman , Nicholas Piggin , Oliver Upton , Palmer Dabbelt , Paul Walmsley , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Shuah Khan , Suzuki K Poulose , Tianrui Zhao , Will Deacon , Yu Zhao , Zenghui Yu , kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, loongarch@lists.linux.dev X-Rspamd-Queue-Id: 84BAF1C000E X-Rspam-User: X-Rspamd-Server: rspam12 X-Stat-Signature: ji9psise3tny7w4f9mo3gxb9jdpaigc8 X-HE-Tag: 1717005918-610279 X-HE-Meta: U2FsdGVkX1/wOQt1zkDaCSM+dS001dqsBwQcAXIo3b3L9zq+UqgewhkGO8eEVyiPSNPAgB1p7sjVW21HSowVOpjPSvrwYw3544tqzQrF7Rn14oXKjzDNw8lpu+8urBPYMAPHoDDT+vKhzqtLbILlup/fndW5BTxpw9S1UzCuG1m6YV1yD4OkOnUkbCYDCJ9m/EI4JCOYacF2qV0JhwMJLH1/uhK/ehhc9uGq751cx7PARXAlASFq5PbMeEK0AHR+87QCL6D3qEZGubW6nH9egUhQ9poVZV/fwcerNw5EFzOeyOPlY1GmRbTc0RHrhdOYZsil4KXpXYtanqtsjN4/qIept6NaW11gAVtTIu3STqSbt66Sx8/xFjT6jog7THQFr+5demC8F3pJXX6/Ps8ZfqF/DUkqAgTfWeIYeKbEE2wtJciCfbcJAfVMJ4iP5XSk+CU2m+NX145G0GlZviA6oISELHhp7wURWPDDpR0VOzJloKpHmRIQ4LdHalikvtXkXGVibiWF5SFNAgHGNo0OZNxv8938y21uTYqe7q1H32fGsMiWgMSop8p3AKx4ZPS0EgwN0Op9Qh4CVBM6fjLh9UPxR2JLB3uXJYbfH6rFsTUDOSSMS4XUbHgQIbbVDpxNxrS02sEoP6/dU+DytnxWBwpNdCsDhu8CiqXV5dA/KRpV+pWemi93rA/dqTmkZltL5U913u3wHARpujQ6ZfIRd/dfk9SD9nLbCsitDYiJDuF1ibOcR4XFYBBRwex7q8cpacTV60+TPvHbvBC+ZGpaXKWjvKxPO9cwYxKZgYeph08bRJPAppop8fhVJeB2eDexlX6ZcGve3xWfv6gYxkNA1YkNEYFTUIIgRs/iLRUSuAj6dgZNnUHSs5iFaXF1HX68VC0arjDi5i7l3qxHWtZcvyy7M8tmjJr4juRE5Gn95PztnZ3css6n710GuORwfr9rznKlpErhL1AkbcpEj2h hcqsoiUh QZSfXUyBs1VgojK15VC+7dn6Aw3qIoNkdOo92IYAqpwHL8UhsYmPkAXWqCj8/BEMThGaZLRGE9Jv7py6rvYo3Jg2Cp5sr5K5SBKrC8ae0oU+Sh7aK3w0Xb24XXKIKxnN7XTIjEKzsAgeez73gcf03zjrIvGZKwGSnIQHISf3bKZOzgMn5Y+b+vIPtCvGBZ0o0USrknkfNGxK0ED0brWWxnNQf9ZQysdQPPGRdyyEREw7mc+9r1WF0pRWsta+C/yfCmDoceKJGNz1Q66LKWGg2uqweg5fjNFZr/gH6QpjqPN4R+xsF/l5bzbaotuDsHY49Ly4yEColQYApmH676eYGezwtswo/ZrmvQFu6/Zyks1xZXBU5SMVRcnMfUC9/LPf2Ws8Ea8SnUEtQlZ44dFyTNYSvX57z5utP20e3osB/sS4QFVHTiWYH7waOWUxH7vAcbd63qDbx2jRk72Ux5ce3hqjR7Wf2Wlw7jOj123+E/4uXwDrAVOxDi8nZvehnjmOp5HyvgkkNDei+qKYMmBse4DXufwV1H0jAWNV87v2V/1n+shsk9QqiYFJNWNOvEdiHPQvGacWRZ1aeRvNJKK/Iqkwp3uKzWiB3cNmKeFGODrQMj0tb9enWM4LHGuBMc0SSheP5mzfZSF6JMxSCWeqOOar51de0SCW/XS5fJNmPDi0cg9eUTvXDmEoosh+sMMYuXAc+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patchset makes it possible for MGLRU to consult secondary MMUs while doing aging, not just during eviction. This allows for more accurate reclaim decisions, which is especially important for proactive reclaim. This series makes the necessary MMU notifier changes to MGLRU and then includes optimizations on top of that. This series also now includes changes to access_tracking_perf_test to verify that aging works properly for pages that are mainly used by KVM. access_tracking_perf_test also has a mode (-p) to check performance of MGLRU aging while the VM is faulting memory in. Here are some results: Testing MGLRU aging while vCPUs are faulting in memory on x86 with the TDP MMU. THPs disabled. The test results varied a decent amount from run to run. I did my best to take representative averages, but nonetheless, the big picture is the important part. Main takeaways: - With the optimizations, the workload is much less impacted by the presence of aging. - With the optimizations, MGLRU is able to do aging much more quickly, especially at 8+ vCPUs on my machine. ./access_tracking_perf_test -p -l -b 1G -v $N_VCPUS # 1G per vCPU num_vcpus vcpu wall time aging avg pass time 1 (no aging) 0.878822016 n/a 1 (no opt) 0.938250568 0.008236007 1 (opt) 0.912270190 0.007314582 2 (no aging) 0.984959659 n/a 2 (no opt) 1.057880728 0.017989741 2 (opt) 1.037735641 0.013996319 4 (no aging) 1.264881581 n/a 4 (no opt) 1.318849182 0.056164918 4 (opt) 1.314653465 0.029311993 8 (no aging) 1.473883699 n/a 8 (no opt) 1.589441079 0.227419586s 8 (opt) 1.498439592 0.063857740s 16 (no aging) 2.048766096 n/a 16 (no opt) 2.399335597 1.247142841s 16 (opt) 2.000914001 0.121628689s 32 (no aging) 3.316256321 n/a 32 (no opt) 3.955417018 4.347290433 32 (opt) 3.355274507 0.250886289 64 (no aging) 6.498958516 n/a 64 (no opt) 7.127533884 9.815592054 64 (opt) 6.442582168 1.392907010 112 (no aging) 8.498029491 n/a 112 (no opt) 10.21372495 13.47381656 112 (opt) 8.896963554 2.292223850 Previous versions of this series included logic in MGLRU and KVM to support batching the updates to secondary page tables. This version removes this logic, as it was complex and not necessary to enable proactive reclaim. This optimization, as well as the additional optimizations for arm64 and powerpc, can be done in a later series. Changes since v3[1]: - Vastly simplified the series (thanks David). Removed mmu notifier batching logic entirely. - Cleaned up how locking is done for mmu_notifier_test/clear_young (thanks David). - Look-around is now only done when there are no secondary MMUs subscribed to MMU notifiers. - CONFIG_LRU_GEN_WALKS_SECONDARY_MMU has been added. - Fixed the lockless implementation of kvm_{test,}age_gfn for x86 (thanks David). - Added MGLRU functional and performance tests to access_tracking_perf_test (thanks Axel). - In v3, an mm would be completely ignored (for aging) if there was a secondary MMU but support for secondary MMU walking was missing. Now, missing secondary MMU walking support simply skips the notifier calls (except for eviction). - Added a sanity check for that range->lockless and range->on_lock are never both provided for the memslot walk. For the changes from v2[2] to v3, see v3[1]. This series applies cleanly to mm/mm-unstable and kvm/queue. [1]: https://lore.kernel.org/linux-mm/20240401232946.1837665-1-jthoughton@google.com/ [2]: https://lore.kernel.org/kvmarm/20230526234435.662652-1-yuzhao@google.com/ James Houghton (7): mm/Kconfig: Add LRU_GEN_WALKS_SECONDARY_MMU mm: multi-gen LRU: Have secondary MMUs participate in aging KVM: Add lockless memslot walk to KVM KVM: Move MMU lock acquisition for test/clear_young to architecture KVM: x86: Relax locking for kvm_test_age_gfn and kvm_age_gfn KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn KVM: selftests: Add multi-gen LRU aging to access_tracking_perf_test Documentation/admin-guide/mm/multigen_lru.rst | 6 +- arch/arm64/kvm/hyp/pgtable.c | 9 +- arch/arm64/kvm/mmu.c | 30 +- arch/loongarch/kvm/mmu.c | 20 +- arch/mips/kvm/mmu.c | 21 +- arch/powerpc/kvm/book3s.c | 14 +- arch/riscv/kvm/mmu.c | 26 +- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/mmu/mmu.c | 10 +- arch/x86/kvm/mmu/tdp_iter.h | 27 +- arch/x86/kvm/mmu/tdp_mmu.c | 67 ++- include/linux/kvm_host.h | 1 + include/linux/mmzone.h | 6 +- mm/Kconfig | 8 + mm/rmap.c | 9 +- mm/vmscan.c | 144 +++++-- tools/testing/selftests/kvm/Makefile | 1 + .../selftests/kvm/access_tracking_perf_test.c | 365 ++++++++++++++-- .../selftests/kvm/include/lru_gen_util.h | 55 +++ .../testing/selftests/kvm/lib/lru_gen_util.c | 391 ++++++++++++++++++ virt/kvm/kvm_main.c | 38 +- 21 files changed, 1104 insertions(+), 145 deletions(-) create mode 100644 tools/testing/selftests/kvm/include/lru_gen_util.h create mode 100644 tools/testing/selftests/kvm/lib/lru_gen_util.c base-commit: e0cce98fe279b64f4a7d81b7f5c3a23d80b92fbc