From patchwork Wed Jul 24 01:10:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13740498 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE6B8C3DA64 for ; Wed, 24 Jul 2024 01:11:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9ED766B0083; Tue, 23 Jul 2024 21:11:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 99AF36B0085; Tue, 23 Jul 2024 21:11:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7F3F56B0089; Tue, 23 Jul 2024 21:11:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 5759A6B0085 for ; Tue, 23 Jul 2024 21:11:18 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id F164F40538 for ; Wed, 24 Jul 2024 01:11:17 +0000 (UTC) X-FDA: 82372867794.13.863B110 Received: from mail-vs1-f74.google.com (mail-vs1-f74.google.com [209.85.217.74]) by imf02.hostedemail.com (Postfix) with ESMTP id 3A82A8000D for ; Wed, 24 Jul 2024 01:11:15 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=lirTCjLE; spf=pass (imf02.hostedemail.com: domain of 3slSgZgoKCBk8I6DJ56IDC5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--jthoughton.bounces.google.com designates 209.85.217.74 as permitted sender) smtp.mailfrom=3slSgZgoKCBk8I6DJ56IDC5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721783438; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cbTYml8WJPEGh8qSfIdK1VudaJalVARhAdPc06J+ALw=; b=64HXllbFZluLvUgi1PcPq+iQsbZfjZjoylM0EhVZOIw9UsTOA7GwVkTZ/du+YCd25+tKsQ Oxp8O2DmsH4+6oGF8yDTq2i6b29skL85+CIAcZQO43QqkDJNYOeegf23wL41Xui81KtxPi HWV4Mi/CuDHdSad0j3+JuQVaWfiRR4M= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=lirTCjLE; spf=pass (imf02.hostedemail.com: domain of 3slSgZgoKCBk8I6DJ56IDC5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--jthoughton.bounces.google.com designates 209.85.217.74 as permitted sender) smtp.mailfrom=3slSgZgoKCBk8I6DJ56IDC5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721783438; a=rsa-sha256; cv=none; b=PZ3lBgvwpjuxjBvwyOBmDM74Sto4gGbV9goRlLevlR3snI9W8sGXx8tLTN5Zaq1YbEzW2E tEKOpEHRPzLKkWrQR5H7NlnDuFwqS+zdJatuf2xpN5KD2lDd890KZSZlE3bHPKCOYKbFvO HZwpHlCCAQ6nbblLQuMsIHhGjpJuJTE= Received: by mail-vs1-f74.google.com with SMTP id ada2fe7eead31-4928b926d38so1619415137.0 for ; Tue, 23 Jul 2024 18:11:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1721783474; x=1722388274; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=cbTYml8WJPEGh8qSfIdK1VudaJalVARhAdPc06J+ALw=; b=lirTCjLEsRIFAq4BrOnmYSEqDi+XYT9em/qUvRlny8xamJHbR7Zottm185NSx7Jm3N +3zDbwl7D8fzI47XuD5i8ovNJJbHXAWz05VBloP9LQfIUAQIN0vaWLwNTTFyV6RmVvCF ClRA6WhAULXwPrhxjE3zjv2iK6kEnbSZWIO1We9dh7lfDirlWZTbLfHJ0CJ/khs2HP87 BtXTujs/FN71jY5DhGjd+T3hk8KD/lUnMrD2bPxVK3INcnJoZduGElp73OLWEDPa2cMS HvQ4DNr5OZHvd1Yo6sb5bJAnpTxiUGBgaey77868SJog8+DwhS8qz4yVq00fXP4+Aijx HuWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721783474; x=1722388274; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=cbTYml8WJPEGh8qSfIdK1VudaJalVARhAdPc06J+ALw=; b=ZiAvgbQ57R0xuAuUn16GkmfcC5EnKWcJwf8DQ9ZrwT4u9IlxpnIItwXLzXieztPDlh pS0f6qZzLun8OszQSyevi5hjUlzR4Sw7KfULF7FZgUfr5A9WQI3FAeGycSAIWQ2YisUJ B7bw6aptLzfw3Q2pnnQny5fnLHD1nGKC8RXQmclV2kIREiSrbIjTUCloDQz0e4bxsmbT hRCwQybiYx6mGvJj/lYu9LPg7nBQo2HJ78N3AP2M1ZdGvtFyo53PK1rosdjagE5cCrCq M5eInSwhMZ0i7XorD68m2DYOnXk4w7oofkPuCHFeEgL8QO0b+rMa8NwOXb4MmfA7LqYt vzHQ== X-Forwarded-Encrypted: i=1; AJvYcCWn1Q/kR6ZPauBuhbG+W4l9eXDy7bcWbmva5S9zbkDBwPh5Z8t9ThXLI1Fkjoa8L+CD1H2M7vmqXN7p3f80BBM1a+A= X-Gm-Message-State: AOJu0Yx25FPPM4nw914tBbXVqHY55qQqp5n86xiSI/T0SrZ+KgFF6WBO 3zpqGehutNADtpW2K8hYrMHtCU7IgDZ/kJjamac7sfbjWyyoJyjeVRHK4bek3elXUqLOi+Eruyz iDkSMoimZM+W2Nak6Vg== X-Google-Smtp-Source: AGHT+IFU0dcKvF1oJPyuhM+g7ff3d/ALiIFFaW2tweOW7lf82u7TeCdfrHDEq+idZRCqJG0SfkAoO+hAK8jfDIhe X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6102:54a5:b0:492:a760:c94c with SMTP id ada2fe7eead31-493c199efb7mr42818137.4.1721783474197; Tue, 23 Jul 2024 18:11:14 -0700 (PDT) Date: Wed, 24 Jul 2024 01:10:26 +0000 In-Reply-To: <20240724011037.3671523-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240724011037.3671523-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.rc1.232.g9752f9e123-goog Message-ID: <20240724011037.3671523-2-jthoughton@google.com> Subject: [PATCH v6 01/11] KVM: Add lockless memslot walk to KVM From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Ankit Agrawal , Axel Rasmussen , Catalin Marinas , David Matlack , David Rientjes , James Houghton , James Morse , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Suzuki K Poulose , Wei Xu , Will Deacon , Yu Zhao , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Stat-Signature: ok3ewf993k5rdnj7ipbw39iym5koc8g5 X-Rspam-User: X-Rspamd-Queue-Id: 3A82A8000D X-Rspamd-Server: rspam02 X-HE-Tag: 1721783475-728939 X-HE-Meta: U2FsdGVkX1964HRqckflKNVN+myHq5+wLyU1XLNbyU/CcMlhgNVUO4Mz6jKD4OoxL2Isfo8boPnNh6wYP1ix8Xbmlyyc6OP+Gyjqnf2WwMc8j8E2kgBc3+OhMVSKmubFoDPraJTgWiqGIth9Wkj8XxVxylDFKBpdzI0eHKJZwnsrlLQi+fA2zhSwOCu4g8csRUqKHtrrV6hXaN66itSZ6EqPUCoJroBC+lEQatypkz5CgOzk4KSw5Co0kloGG1JKNo2nK/PTVgt05+h3WPgM28aqy9hTNbLK4ga9RiBKiOriE3mHYq6Cf/8txzdpakXFM6oErK9zv2qKhnmwy5+27c+cTA9xUsTrXklkF8s7iVS5umdYz72rO5vYdrGXh9GokWoRdGBYGs9NC3mwhC4UaZp21ftaWjJmYJGarUQC4QbhCXpYgtzGu6x7upkuOG5WmfZatruuY03ZBTs1y5F4/CvvoOKOPWcEMOLGNmWYPMTNvIEX8tYINxWsR/OstKW9MVJKPX2AIC9AYt73uUPn/k/Z02hG5sQAt/NNWO6ET5UeGuu+Bxn8FP1uvhpd1BafRjZI4oAziHjqqln130QP8LgKY30NVItu2XqgZWWWgWqeuGPQB4qKo241au93ldw1PWT4K6lN+hNucdRJ3ZIF3zSxr7obnfV7ypYKIkCkqwe5U4t/D7MOAnSUpe+ajfhq5Toqb5f08Xgvi/KxrtWL17E+YOheE5frrgKMQbpUnWRG+LIzai362npqvfYZM0L8up4w2tHSJ3bnUAQlAPz57baPcWxtmDxPagA8zYH/LKOK4nCyaMWaK6rCBqqnWzMCDZCExN1bCfmUiOEYL2kq9k840VkoSZV3Wc1IdK220s0bW89gSX88/lpwTWGqmHO0JMWovHgyZfgG4GWBZDyKppQlXKHrfEcgWMqEuclQue7QlX4yubfJVVLZjeASRwASvItVMFuP1KBttc2kpZm oBZ2QhWA lP2bikcFshJ504CZhIYMdcX6ympA1LPLZwtfRzHHt6UR0ZsFIgjrQqC8vViRNxd6CEC1V4wAfnX0Tn5PcZ3k3vD0ywexXmnx7wOJSJx0JaoUgU8Cc2ngIoioRDPr5OF1ptEJX15LEzNrLHvLKvSrqMS4ukpMQeDDgIYaa6USZdRqbm/1XlNGir7KGKYPS4rK0a4Cit0YZnVHHDMouVTs67eaGXWiZmOY/2vXXZuDWiMKnkEwIuaMg0XvWLb72c7IUaEKVh0XaIBczSj1iC3lLZCeb1MiltJyYa+Lay5Wab8nF2pFsJGofTZrFZxc8Jc/w4tUV3liW/uu0uvTBUdBLUYCGr52VTdLLXrZa+WC0CSkz0UCwe+UTicK0Ulam1u3b1rjp4ur6+HFli68zpSb6ZBQH48EFTK7YH4AEVaQ/k+kZKEwj86VnpAuxKbrRrrmprHjOFFQrjgOxNHzr5VsoApvUwPnfsr4T+7Gqk0x+TjnSfI5qPfKPZ0T31ezNG6pEPG8Rnr26JWL+5ILLh1xf5HOnfT7FMIlsWExKQqGXb9DUslM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Provide flexibility to the architecture to synchronize as optimally as they can instead of always taking the MMU lock for writing. Architectures that do their own locking must select CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS. The immediate application is to allow architectures to implement the test/clear_young MMU notifiers more cheaply. Suggested-by: Yu Zhao Signed-off-by: James Houghton Reviewed-by: David Matlack --- include/linux/kvm_host.h | 1 + virt/kvm/Kconfig | 3 +++ virt/kvm/kvm_main.c | 26 +++++++++++++++++++------- 3 files changed, 23 insertions(+), 7 deletions(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 689e8be873a7..8cd80f969cff 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -266,6 +266,7 @@ struct kvm_gfn_range { gfn_t end; union kvm_mmu_notifier_arg arg; bool may_block; + bool lockless; }; bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range); bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig index b14e14cdbfb9..632334861001 100644 --- a/virt/kvm/Kconfig +++ b/virt/kvm/Kconfig @@ -100,6 +100,9 @@ config KVM_GENERIC_MMU_NOTIFIER select MMU_NOTIFIER bool +config KVM_MMU_NOTIFIER_YOUNG_LOCKLESS + bool + config KVM_GENERIC_MEMORY_ATTRIBUTES depends on KVM_GENERIC_MMU_NOTIFIER bool diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index d0788d0a72cc..33f8997a5c29 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -555,6 +555,7 @@ struct kvm_mmu_notifier_range { on_lock_fn_t on_lock; bool flush_on_ret; bool may_block; + bool lockless; }; /* @@ -609,6 +610,10 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, IS_KVM_NULL_FN(range->handler))) return r; + /* on_lock will never be called for lockless walks */ + if (WARN_ON_ONCE(range->lockless && !IS_KVM_NULL_FN(range->on_lock))) + return r; + idx = srcu_read_lock(&kvm->srcu); for (i = 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) { @@ -640,15 +645,18 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, gfn_range.start = hva_to_gfn_memslot(hva_start, slot); gfn_range.end = hva_to_gfn_memslot(hva_end + PAGE_SIZE - 1, slot); gfn_range.slot = slot; + gfn_range.lockless = range->lockless; if (!r.found_memslot) { r.found_memslot = true; - KVM_MMU_LOCK(kvm); - if (!IS_KVM_NULL_FN(range->on_lock)) - range->on_lock(kvm); - - if (IS_KVM_NULL_FN(range->handler)) - goto mmu_unlock; + if (!range->lockless) { + KVM_MMU_LOCK(kvm); + if (!IS_KVM_NULL_FN(range->on_lock)) + range->on_lock(kvm); + + if (IS_KVM_NULL_FN(range->handler)) + goto mmu_unlock; + } } r.ret |= range->handler(kvm, &gfn_range); } @@ -658,7 +666,7 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, kvm_flush_remote_tlbs(kvm); mmu_unlock: - if (r.found_memslot) + if (r.found_memslot && !range->lockless) KVM_MMU_UNLOCK(kvm); srcu_read_unlock(&kvm->srcu, idx); @@ -679,6 +687,8 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn, .on_lock = (void *)kvm_null_fn, .flush_on_ret = true, .may_block = false, + .lockless = + IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), }; return __kvm_handle_hva_range(kvm, &range).ret; @@ -697,6 +707,8 @@ static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn .on_lock = (void *)kvm_null_fn, .flush_on_ret = false, .may_block = false, + .lockless = + IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), }; return __kvm_handle_hva_range(kvm, &range).ret;