From patchwork Mon Apr 1 23:29:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13613114 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F1578CD1288 for ; Mon, 1 Apr 2024 23:30:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: Mime-Version:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=uymUBPwjIvH5Ok193upCUFRVHZqVg34PKZohKLZ6q6Y=; b=eqj cRhhkJdPbEacUG3X4dsnukkrdN8MR7+EHvSKIhtIbv42zK5PhWw+I2ud+xr4bSQyj6sHeKQUmJJQU jOc3NII3ytg6H9JKloRPlkdjqaQ4JUByQGk/F7zTY5LbzOn3qS/ieZsP/cjWZj9Q8/1627fFjcaBu sx3q4dJ+pDibWOLSc0VrJJTPUhsRFbvL/a91+M8iaUDlih8hgV3+O4L1wWYXnTCU/nRZWziZHD9ZQ Nxf+Rd0hIELtc+ph+SAhN4pB8L/YvE8dCodbH1UIcaezMobD76vXe2vyfKl8yvr6XkcaEwxE4sXJ+ yFFmk0A3MsPy9c0xFEX4Q04myzZoATA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rrR6H-000000097Gp-2oSE; Mon, 01 Apr 2024 23:30:01 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rrR6E-000000097FC-2xek for linux-arm-kernel@lists.infradead.org; Mon, 01 Apr 2024 23:30:00 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-61517841a2eso6752727b3.2 for ; Mon, 01 Apr 2024 16:29:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1712014196; x=1712618996; darn=lists.infradead.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=3wg9CCGpIdLpfDRqh60QLQofCQe0biAtvSj1Gdo5bk8=; b=EvkCVD74dhl+ifA1gcw7Pj9jHxlGQz/qcHlJbiHMi9ekwOw5OTeXD6XJBD8YID5soJ s8GtIPpcPh++bKMWohkvj7U8o4Luq1FUmdTsOpN4bLffn9inTlw1Es7fCDdh+ZuhVRH7 hTrF9BQ+JIYMwq8FipgqvRQ3hnrzTpS2Q4VPA0YUdBVm8JIYQdtCVaNUH8xt8E8vBgxD DpzoienlEUj1QVAGmU2dItYQDCMu5k93xE8vOe0zZNQ/hfuuH0WSciHKlF66f8npZiYf reuGe4anw7kyOg8XOHmz2vUChBx4pYRx2wAVb4HYpFhWU7cTzlGRZML3GVXlKFguE8SD K/lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712014196; x=1712618996; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=3wg9CCGpIdLpfDRqh60QLQofCQe0biAtvSj1Gdo5bk8=; b=Ak/9wvVJcpCbAk5FCsC6aStfd3QFHapb0KKKQI+mBWlxlcj6c6Wk3c7jwinO0dd/2Q fSHyPqYTYukardyjg8L0BCQZhFQ8eOYGNaqT9Rv0ygcsvIPSaV9tVL+2UvKEbNj7/mfR i3kdkSXHehA9bbTD67CASePqoiBy0udqDHquYLC2BTqcSjeiDQ56doTPEEalDJgsIH5f nSKYb75S9SztTYBplYm5Eeoz06X6aHIkmlqnIgJ6EkCsAuhA/rpQ6Xn4KAecp9JksnUW NtgihNLnfouuWyqBdJ6TkKK9BIdwMiM51Y7aE0iqDJRCHdkmiY7UnH2maFc60hivaTE6 0z8Q== X-Forwarded-Encrypted: i=1; AJvYcCW1shixIMqxCDn1nfXe0hN6WNqBhLExmfp4Cubjr1ALHAd0x4OsbfBKORSUokeuSXP9GDgsuqEkp0FZLu93gdHsTdBFFGMHf8St3skUfRQhr8vSROQ= X-Gm-Message-State: AOJu0YxNe5lph26cq69cyap4QHaIJYOrUSngfm8SvZ9tZXmgZb3aIHgI 9e4eG2znNMYWllC/ILzC1HPR+rXBbIunPs0LIdxGCKxRNZWgtzBArGbYyhfm2azcNZacq51+OXr JUd5HpM4o2zwQR0IIFw== X-Google-Smtp-Source: AGHT+IGQKV963bGOmZkg1qJdb4wkJ3JL7Okh/SQUhgB3vOnWcX/gauefdkact4kTuSTRZOisDp+cn6EF/9Mo2eKk X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:690c:f83:b0:614:e20c:d423 with SMTP id df3-20020a05690c0f8300b00614e20cd423mr823185ywb.10.1712014196503; Mon, 01 Apr 2024 16:29:56 -0700 (PDT) Date: Mon, 1 Apr 2024 23:29:39 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.44.0.478.gd926399ef9-goog Message-ID: <20240401232946.1837665-1-jthoughton@google.com> Subject: [PATCH v3 0/7] mm/kvm: Improve parallelism for access bit harvesting From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Yu Zhao , David Matlack , Marc Zyngier , Oliver Upton , Sean Christopherson , Jonathan Corbet , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Shaoqin Huang , Gavin Shan , Ricardo Koller , Raghavendra Rao Ananta , Ryan Roberts , David Rientjes , Axel Rasmussen , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, James Houghton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240401_162958_780173_D2C3C69B X-CRM114-Status: GOOD ( 21.59 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This patchset adds a fast path in KVM to test and clear access bits on sptes without taking the mmu_lock. It also adds support for using a bitmap to (1) test the access bits for many sptes in a single call to mmu_notifier_test_young, and to (2) clear the access bits for many ptes in a single call to mmu_notifier_clear_young. With Yu's permission, I'm now working on getting this series into a mergeable state. I'm posting this as an RFC because I'm not sure if the arm64 bits are correct, and I haven't done complete performance testing. I want to do broader experimentation to see how much this improves VM performance in a cloud environment, but I want to be sure that the code is mergeable first. Yu has posted other performance results[1], [2]. This v3 shouldn't significantly change the x86 results, but the arm64 results may have changed. The most important changes since v2[3]: - Split the test_clear_young MMU notifier back into test_young and clear_young. I did this because the bitmap passed in has a distinct meaning for each of them, and I felt that this was cleaner. - The return value of test_young / clear_young now indicates if the bitmap was used. - Removed the custom spte walker to implement the lockless path. This was important for arm64 to be functionally correct (thanks Oliver), and it avoids a lot of problems brought up in review of v2 (for example[4]). - Add kvm_arch_prepare_bitmap_age and kvm_arch_finish_bitmap_age to allow for arm64 to implement its bitmap-based aging to grab the MMU lock for reading while allowing x86 to be lockless. - The powerpc changes have been dropped. - The logic to inform architectures how to use the bitmap has been cleaned up (kvm_should_clear_young has been split into kvm_gfn_should_age and kvm_gfn_record_young) (thanks Nicolas). There were some smaller changes too: - Added test_clear_young_metadata (thanks Sean). - MMU_NOTIFIER_RANGE_LOCKLESS has been renamed to MMU_NOTIFIER_YOUNG_FAST, to indicate to the caller that passing a bitmap for MGLRU look-around is likely to be beneficial. - Cleaned up comments that describe the changes to mmu_notifier_test_young / mmu_notifier_clear_young (thanks Nicolas). [1]: https://lore.kernel.org/all/20230609005943.43041-1-yuzhao@google.com/ [2]: https://lore.kernel.org/all/20230609005935.42390-1-yuzhao@google.com/ [3]: https://lore.kernel.org/kvmarm/20230526234435.662652-1-yuzhao@google.com/ [4]: https://lore.kernel.org/all/ZItX64Bbx5vdjo9M@google.com/ James Houghton (5): mm: Add a bitmap into mmu_notifier_{clear,test}_young KVM: Move MMU notifier function declarations KVM: Add basic bitmap support into kvm_mmu_notifier_test/clear_young KVM: x86: Participate in bitmap-based PTE aging KVM: arm64: Participate in bitmap-based PTE aging Yu Zhao (2): KVM: x86: Move tdp_mmu_enabled and shadow_accessed_mask mm: multi-gen LRU: use mmu_notifier_test_clear_young() Documentation/admin-guide/mm/multigen_lru.rst | 6 +- arch/arm64/include/asm/kvm_host.h | 5 + arch/arm64/include/asm/kvm_pgtable.h | 4 +- arch/arm64/kvm/hyp/pgtable.c | 21 +- arch/arm64/kvm/mmu.c | 23 ++- arch/x86/include/asm/kvm_host.h | 20 ++ arch/x86/kvm/mmu.h | 6 - arch/x86/kvm/mmu/mmu.c | 16 +- arch/x86/kvm/mmu/spte.h | 1 - arch/x86/kvm/mmu/tdp_mmu.c | 10 +- include/linux/kvm_host.h | 101 ++++++++-- include/linux/mmu_notifier.h | 93 ++++++++- include/linux/mmzone.h | 6 +- include/trace/events/kvm.h | 13 +- mm/mmu_notifier.c | 20 +- mm/rmap.c | 9 +- mm/vmscan.c | 183 ++++++++++++++---- virt/kvm/kvm_main.c | 100 +++++++--- 18 files changed, 509 insertions(+), 128 deletions(-) base-commit: 0cef2c0a2a356137b170c3cb46cb9c1dd2ca3e6b