From patchwork Mon Apr 1 23:29:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13613089 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55119CD128A for ; Mon, 1 Apr 2024 23:30:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C2A036B008C; Mon, 1 Apr 2024 19:29:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BDA1A6B0092; Mon, 1 Apr 2024 19:29:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA1756B0093; Mon, 1 Apr 2024 19:29:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 8BD5C6B008C for ; Mon, 1 Apr 2024 19:29:59 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3A663405F1 for ; Mon, 1 Apr 2024 23:29:59 +0000 (UTC) X-FDA: 81962558118.22.CC22CBC Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf25.hostedemail.com (Postfix) with ESMTP id 88C3FA000D for ; Mon, 1 Apr 2024 23:29:57 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=0Xm0uRlW; spf=pass (imf25.hostedemail.com: domain of 3dEMLZgoKCF8GQELRDEQLKDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--jthoughton.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3dEMLZgoKCF8GQELRDEQLKDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712014197; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=3wg9CCGpIdLpfDRqh60QLQofCQe0biAtvSj1Gdo5bk8=; b=19jWMpogS7p7/be0efdM1yfKucUFAAlp50pcedj3sS8aREMceR+eGO51Ps2GfcY3wJMITp tYFe0URbni8qtcpLcIAtjK5g2+BGtswUBLkm/zEE6yzXQSfLqXviQqaME/fHcTybEbVyM9 n+SSYFoywWml1fYQDzupup8dYKG9tsw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712014197; a=rsa-sha256; cv=none; b=yAj+UA9hddDCZHWCmCcJUBLe2b/FVIKz4kaR39XxqgHM39YX0NOc+BpgILcO1gw08MfXCV MbmS4rCj0GEpEodXcnKblP4cCRQCR/AXk5Rjya/asr/1cXn2tmVHTodJcxeQcHR81kCq1q LJmmG0mH9gtzuLnKovTVYH1+bz80240= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=0Xm0uRlW; spf=pass (imf25.hostedemail.com: domain of 3dEMLZgoKCF8GQELRDEQLKDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--jthoughton.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3dEMLZgoKCF8GQELRDEQLKDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60cc8d4e1a4so84495047b3.3 for ; Mon, 01 Apr 2024 16:29:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1712014196; x=1712618996; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=3wg9CCGpIdLpfDRqh60QLQofCQe0biAtvSj1Gdo5bk8=; b=0Xm0uRlWi3C15qb7nl0eqaiH8MBvRpwT+B18qon80WtvkZf1IJZRxEEiIguwRDtLyo xXYwlTjyiOmvc3pqM4sYFDD23KfDaDELBGq2fvjBm8hNsUTg5vWRFhIuajN7G2nA3TN9 eOrwSce2F3SE2sMAG5niR22m/ptvO04npbCW7TtTH3az0l8pImOZfsulUb8lcA6BqU60 J1ZLU/Zw5Uz+Twp+hN8cHvcNDp65gcLVO7uurU0L93ortPQZv6Y3xYcTR6zKyWoG7nm8 3tcmx+Vx+phJKAiw5/uWJu9x5GGs7D/JzuTFFDMPYZs1HDjFE1AymIAof0YOXw//zE6/ s5jA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712014196; x=1712618996; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=3wg9CCGpIdLpfDRqh60QLQofCQe0biAtvSj1Gdo5bk8=; b=LT+NXMbGgW0OO/1XppOsh6RJPFlq3po3iUvjC1SLHMRUi/+snXlIhpeP1L9iwOQQ7O WTm1WmRMledRH6mrTpjcvx2wFLDYPCD14+oZi2Q50YlVCUO3SRBTNmIpJ4E7P6ygR0An QKClkPjiblNHNyeGQCHhM+MSaxsegH3gXTedhrIrN1z+GHdtZs/o4C5KlfZJi6XEO5o5 mFe9PVHXMuf7g6pGT8I89Qf6vo5EIWtM4bBsgKxAfygCKBQB4bT/Ezi9LXZoNtxzqUzU IN4QS1C8KLC05KbNIKdTLEe3hoHVAZYMW0Z8vRv3/hswUalDfS7Qnto9N3JQorGWndL4 IAQg== X-Forwarded-Encrypted: i=1; AJvYcCX6wk6gAh7u1JRANwcpnRbwFK3CFunweEzeFuY2/eHFExtyVVciNKdmLbTDhcHKEWXIHvvAL6nG1e0MtPQoQwnUP1Y= X-Gm-Message-State: AOJu0YxuHIz9M1CtvpqSnn5c+htg8CXLt70i946w8tq0CvVEIKVsw4jm 1SUfKxXdgWAq+lEUHHCkdKqk6RqCoSoa8iGcPLiE7PxjptZkCoO2x1cIo01Ac+utHAcDU9FhjmW cRtFp8TK5DpUfpt6oQw== X-Google-Smtp-Source: AGHT+IGQKV963bGOmZkg1qJdb4wkJ3JL7Okh/SQUhgB3vOnWcX/gauefdkact4kTuSTRZOisDp+cn6EF/9Mo2eKk X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:690c:f83:b0:614:e20c:d423 with SMTP id df3-20020a05690c0f8300b00614e20cd423mr823185ywb.10.1712014196503; Mon, 01 Apr 2024 16:29:56 -0700 (PDT) Date: Mon, 1 Apr 2024 23:29:39 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240401232946.1837665-1-jthoughton@google.com> Subject: [PATCH v3 0/7] mm/kvm: Improve parallelism for access bit harvesting From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Yu Zhao , David Matlack , Marc Zyngier , Oliver Upton , Sean Christopherson , Jonathan Corbet , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Shaoqin Huang , Gavin Shan , Ricardo Koller , Raghavendra Rao Ananta , Ryan Roberts , David Rientjes , Axel Rasmussen , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, James Houghton X-Stat-Signature: gteogw6ukxjccpmwy43ty3c4c51wa8qu X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 88C3FA000D X-Rspam-User: X-HE-Tag: 1712014197-345609 X-HE-Meta: U2FsdGVkX18covzUHnZAWpiePPBf6xef1zubLE5R+5rWPMa4mqKPBSY1Mj24wp+4MmH+e1BCjPRCh+ev3EAc55QCrtDIAHcDAToYeUt0h6qqOStazPyjGkSoETWgpHRZzK26WQNX+1AQopVf8iXg8dX2FVjbbjv5w6JWc6jWPark0hdRgFqqBcg5VaBcGnG23ae8V2wCRJEF4xPcuxZyuFL2pycsa82rDJiAul5UD/GFH6wFHI6AGJDeLm0lZhQNncuy6lbytJxylJikn0MJhpliWi4K8LO2mtPijjefAyx6mnC52V38/zledzNvTyAMPM6GzRFowJ6DNGCxZ4d+O9F15jEO65ideu8w4kDKiT6+s4flLVDBgwYQfXUC2v53Nej3uCXfpznNJhmFbZ2rfROM9dNXOLVu2R++DtxpEKVazBRB2OqabIA71dfkGxn/80CgJbWQI7sBVVwKZHFj+2ZKXYYchz7Klr5HMYegeAARbZN98CXK046vUiqR5a9OQwAkyG+bHZUVVIX+WXh61rzE7yovPV+X3JHpycGP790bWuSuLjtJZDLJIZjrfCEjT93aQ5KOAPFkBX5GkxaZpULoLMQ0t/qCGif7QcLe7fcGp8NjWoq2ssQWsRpJYAZRw2Q9DMJG3C64Ynd0/DDAeBVjcYES35F7LZIxfW5vPPdYxdZsVRcQv9KxT5Y4MU/nyGav0C7kLughaa/qLcSzpTigj5KnZLf8sFQs1Pfuzz16PjmcOrSGPQEYVV6l3ghEuZ1cbdLCJFekYZ+4b9Yol7gubyLzoxyb5UXNLrw+iMFSEJzmMMvS7x064qL+OoBevHkkBYPVuxfWUQ9TQAo0v88rQnDuSOYKihdQYQWRv1XziOMCTWIt2zqwteSTrKWD1EXWSLGymfvAcRei077EuYDvL6ZrHp6TXsZhl0ncc69lr+4DwnIKAptzr6Pfoi2xybby/5JdxOLMMAiUH60 hb/OnEpL aTeY9nWIuJQFCqz7UVIMZSHQ4hb8TOgaJsCmcRz0J+NnEuM7mFCh676ifb/oDG9COsND2MZVlelakX2eD/xX3ttSVRvn1DavtrhtSS7be5/ZOCPonnso+MZv/eTTYmRi9ZioI88qQYrRsK65jicN6sbqBaChGibbvi+cnbWBTOtVTmjvjKQqLO8O8+kQHpzcfq/erzGOHkkqhT7MeQ/3gU4Z8mPonsb7Q1hRC6i/7vyWTdALMDMVui2ITDthcsikurds1jLErR6QUTmbWtEFTQ87+ny58YYIpOYdJjEKYwyCOOtLI0HZfRsfaT/5mx3Ur/nrM3sDf7kfV8X/1g29N/OczXpcUZkRaQdkG7F5T1TLKc2xWLqfLRbscOWKfOZEn3+syUu3oY9gWZizaES7XiSuhG4VoP3I1kCMXITx4IYy0E7ILgU0yZedUWBeB6+NG7JtH4VnBdWS0Pku1OP9OU9yHjRS35U/9CLj5yZdmGBEjQQey9ABGpP6q2nKx048Y3DKwtGglGunMAeiCfRUjMcF49uG/ErtbuPAEYKRiUiVho6pAUvxu0CqzD53mSLWLm4CwpEZeHHoslj5pP5CGD91hiav3Pch1rgpqHG0pvvAwGrYbuWuEKuNLjY2qqEG3iJS46xx4wFEnfIHbsg2oMTkXeukWFi/DwDT/JxNXn4xcnNJ/+KRVFMQYqlAwAJ+Pq9p2 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patchset adds a fast path in KVM to test and clear access bits on sptes without taking the mmu_lock. It also adds support for using a bitmap to (1) test the access bits for many sptes in a single call to mmu_notifier_test_young, and to (2) clear the access bits for many ptes in a single call to mmu_notifier_clear_young. With Yu's permission, I'm now working on getting this series into a mergeable state. I'm posting this as an RFC because I'm not sure if the arm64 bits are correct, and I haven't done complete performance testing. I want to do broader experimentation to see how much this improves VM performance in a cloud environment, but I want to be sure that the code is mergeable first. Yu has posted other performance results[1], [2]. This v3 shouldn't significantly change the x86 results, but the arm64 results may have changed. The most important changes since v2[3]: - Split the test_clear_young MMU notifier back into test_young and clear_young. I did this because the bitmap passed in has a distinct meaning for each of them, and I felt that this was cleaner. - The return value of test_young / clear_young now indicates if the bitmap was used. - Removed the custom spte walker to implement the lockless path. This was important for arm64 to be functionally correct (thanks Oliver), and it avoids a lot of problems brought up in review of v2 (for example[4]). - Add kvm_arch_prepare_bitmap_age and kvm_arch_finish_bitmap_age to allow for arm64 to implement its bitmap-based aging to grab the MMU lock for reading while allowing x86 to be lockless. - The powerpc changes have been dropped. - The logic to inform architectures how to use the bitmap has been cleaned up (kvm_should_clear_young has been split into kvm_gfn_should_age and kvm_gfn_record_young) (thanks Nicolas). There were some smaller changes too: - Added test_clear_young_metadata (thanks Sean). - MMU_NOTIFIER_RANGE_LOCKLESS has been renamed to MMU_NOTIFIER_YOUNG_FAST, to indicate to the caller that passing a bitmap for MGLRU look-around is likely to be beneficial. - Cleaned up comments that describe the changes to mmu_notifier_test_young / mmu_notifier_clear_young (thanks Nicolas). [1]: https://lore.kernel.org/all/20230609005943.43041-1-yuzhao@google.com/ [2]: https://lore.kernel.org/all/20230609005935.42390-1-yuzhao@google.com/ [3]: https://lore.kernel.org/kvmarm/20230526234435.662652-1-yuzhao@google.com/ [4]: https://lore.kernel.org/all/ZItX64Bbx5vdjo9M@google.com/ James Houghton (5): mm: Add a bitmap into mmu_notifier_{clear,test}_young KVM: Move MMU notifier function declarations KVM: Add basic bitmap support into kvm_mmu_notifier_test/clear_young KVM: x86: Participate in bitmap-based PTE aging KVM: arm64: Participate in bitmap-based PTE aging Yu Zhao (2): KVM: x86: Move tdp_mmu_enabled and shadow_accessed_mask mm: multi-gen LRU: use mmu_notifier_test_clear_young() Documentation/admin-guide/mm/multigen_lru.rst | 6 +- arch/arm64/include/asm/kvm_host.h | 5 + arch/arm64/include/asm/kvm_pgtable.h | 4 +- arch/arm64/kvm/hyp/pgtable.c | 21 +- arch/arm64/kvm/mmu.c | 23 ++- arch/x86/include/asm/kvm_host.h | 20 ++ arch/x86/kvm/mmu.h | 6 - arch/x86/kvm/mmu/mmu.c | 16 +- arch/x86/kvm/mmu/spte.h | 1 - arch/x86/kvm/mmu/tdp_mmu.c | 10 +- include/linux/kvm_host.h | 101 ++++++++-- include/linux/mmu_notifier.h | 93 ++++++++- include/linux/mmzone.h | 6 +- include/trace/events/kvm.h | 13 +- mm/mmu_notifier.c | 20 +- mm/rmap.c | 9 +- mm/vmscan.c | 183 ++++++++++++++---- virt/kvm/kvm_main.c | 100 +++++++--- 18 files changed, 509 insertions(+), 128 deletions(-) base-commit: 0cef2c0a2a356137b170c3cb46cb9c1dd2ca3e6b