From patchwork Wed Jul 24 01:10:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13740499 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01340C3DA49 for ; Wed, 24 Jul 2024 01:11:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 495A26B0085; Tue, 23 Jul 2024 21:11:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3AB5E6B0088; Tue, 23 Jul 2024 21:11:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 185E26B0089; Tue, 23 Jul 2024 21:11:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E62786B0085 for ; Tue, 23 Jul 2024 21:11:18 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 99BB9A2A5F for ; Wed, 24 Jul 2024 01:11:18 +0000 (UTC) X-FDA: 82372867836.21.40900EB Received: from mail-vs1-f74.google.com (mail-vs1-f74.google.com [209.85.217.74]) by imf26.hostedemail.com (Postfix) with ESMTP id D34F614000E for ; Wed, 24 Jul 2024 01:11:16 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=qQpgxK+g; spf=pass (imf26.hostedemail.com: domain of 3s1SgZgoKCBo9J7EK67JED6EE6B4.2ECB8DKN-CCAL02A.EH6@flex--jthoughton.bounces.google.com designates 209.85.217.74 as permitted sender) smtp.mailfrom=3s1SgZgoKCBo9J7EK67JED6EE6B4.2ECB8DKN-CCAL02A.EH6@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721783414; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lX17S5nNca5CiBEi8R7tHh84KKPLmRWb0mcnxRYOK88=; b=pNSGr2A/Js9xwVLi0tplvpKYSWf+YclfsEpTjHQHsAXVSs5OzEXKiEyZIpQ/3wpjCNXFQt CzUhsDnAQC/vRhTIL9AtL4l7/T31T7LenJIzra0V/FBuWpnCHZ37BZ3hXny96ez3on7ibx TWdI8NPv3dqOQF0pNtVvCncAXFSuEtM= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=qQpgxK+g; spf=pass (imf26.hostedemail.com: domain of 3s1SgZgoKCBo9J7EK67JED6EE6B4.2ECB8DKN-CCAL02A.EH6@flex--jthoughton.bounces.google.com designates 209.85.217.74 as permitted sender) smtp.mailfrom=3s1SgZgoKCBo9J7EK67JED6EE6B4.2ECB8DKN-CCAL02A.EH6@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721783414; a=rsa-sha256; cv=none; b=fjUT5/CP17fAucouHfpo6gox36hONN5VBVDCMqpZUBbhZPSrAv/iFUQOcUZgkOeohDB5nX iZOXBN2zHSVBq6Os5I0GRvE2NJMf9KPu1XuzHiDxa70aR5LOrsPUYSiIgK+bUhysHt1eED qCMfCl+09AdPoQTPSXolDsBS0jerFos= Received: by mail-vs1-f74.google.com with SMTP id ada2fe7eead31-4928cea3c69so1738206137.3 for ; Tue, 23 Jul 2024 18:11:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1721783476; x=1722388276; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lX17S5nNca5CiBEi8R7tHh84KKPLmRWb0mcnxRYOK88=; b=qQpgxK+gMSwjk0x/ImDPs+F0Jl5qXFOTBvTyDSjmikQ9B6HYbRgJbwz2uCsizbqHHZ yPsB1+hpEgwO5YSSEBPEUemQqef555mx0ibVU4VFHQp3qLaCvlwyIkruwh1xFMHVTOQH 0ya84Jip0XAi+OwZTsZ5IFaI0cjXB/Re+472V4qJ06ZSBPjd5Z1IMoU78VzqgR2EHV/0 RCSSuysRyn+kJey8r0681CO8aSQGdudDR/us/RnXiLVTHOAkJcRYoLnD3fwawt4T25ik wSByAPxA/VDD/EFQ/ER+dPIKSB/QvEtuyhOICRgGSVws5m9XfIFvpEQdjHsjBKGgJLIt 2+LQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721783476; x=1722388276; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lX17S5nNca5CiBEi8R7tHh84KKPLmRWb0mcnxRYOK88=; b=RKwo0BIEuqzr8AwwSEUhA9Jta90nWsizDdQWKOzzLi+Ie0QgLKMg7IthBaf4w7QQ+D UnLjYNWrWQjXK32UlTKYiuWXINChrh7I4vfxLilIUWpNciIozJEgP2qMcKdiHMXttRma Ye2es9Fw1M8QO/MnWVInGqAEwYgq50IlGBq5FukZZAgP/V9psOSXCA8LVLVHbU9EZliO fM4xWfJZ26SXES2k5E0hJVn2kR6uS5X5DoazLyfroW8xOVxUccYKbjRfodxpsXYkDwty hEPJzEXdAzvZBH/ScVjLt4MiYgeq303+OxMquMA+NH4SQVMn2RWGFipBGNoU2eNKTlXx 4PuQ== X-Forwarded-Encrypted: i=1; AJvYcCXA+5E3ym+MD7ehtqMCJ5QeEUAkV46rZ1VyqteL8JDu9Ke+bXv9RGfH4chh7TKGcDdg21azmIDNvyElltd6Whv3mXw= X-Gm-Message-State: AOJu0YxEtlEDss0Ao8hcTPzgqvyCmyQmd0dB9QIUwC6tZiVd78ZecoTl IANj5P8ZlXfF26HE0iOgq47YankqqCXG2qoNDQtWUkkw68MQvqtrwwG8aZ6PLSXa7c+Yx178sen K2o4ORABBmMcIyD8HWQ== X-Google-Smtp-Source: AGHT+IGlIuIrgkANC0pZBbk8z8He5iptT26RG1V2ZrvO9VkU9yoaN+aGYmYAd60aeXGiO8520IPOKzxK6pnBSRdv X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6102:2ac6:b0:492:9449:c33e with SMTP id ada2fe7eead31-493c19d19c5mr46187137.5.1721783475890; Tue, 23 Jul 2024 18:11:15 -0700 (PDT) Date: Wed, 24 Jul 2024 01:10:28 +0000 In-Reply-To: <20240724011037.3671523-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240724011037.3671523-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.rc1.232.g9752f9e123-goog Message-ID: <20240724011037.3671523-4-jthoughton@google.com> Subject: [PATCH v6 03/11] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Ankit Agrawal , Axel Rasmussen , Catalin Marinas , David Matlack , David Rientjes , James Houghton , James Morse , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Suzuki K Poulose , Wei Xu , Will Deacon , Yu Zhao , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: D34F614000E X-Stat-Signature: 4knu43tkydo3qut173cu4k9r64wec1op X-Rspam-User: X-HE-Tag: 1721783476-902587 X-HE-Meta: U2FsdGVkX19uof8qD5E4VH+YWkGtiLNdpihg1DhAc1Ug2Z0CJQPQeNtxxp4dbHKPMIhE+blxdsVDxomtWNH/SHbg2sDXldhbpG2YASF9XO/SYzTiJnxNM0hgYnZiZ80Q4A9wrYpcrGHRIDXrZTk5xFWT2YkBMdYYmlZhp69U6jEP6/KN0gfjzEn5PJuE9nUn6eb2/g6C7YGZXKQS7V8sPIAkoJ5ZPq7TA7NPi4J+kwt33pGtqHWaTPAEdt7VSg1bU2IKKteV0GqTX5H1/QKkU397vgI4uXIS8V0AWAu4rrpz8zCh9zwkq3mGrms7ErUIaMJbh8icB3SBe43EP/gkGDylp65+fk5CZCZGAZEDcBRwa/O0ljAUD39NryXVoyQU7oY7zaq2WX5efO4gazt05W4TpSh1cikJmu6V3WxmXR924SXMwpqr06cY1rfVa7Co5RPNwJDzy6x5Jg839Wn76yJAmW4tDKaUyjjHoSJLxYymrBDWbGhYPV8mnzGCUi7k9a5yEKgEdNfngS6Ddmi50/by8jCsojkrGHLO93SUhcD8agLkeC2vuHlBvpUGVPSDscG+ztsvjc1ajVawIHLB0aLQOQSjDyb5NYBT2ImY+ano3YJdM412rX8ciSXptTyyyGgEHouACZQnTBgG39acmXZK6ZlkvGwkcDeowpsBujJiZchSX5qJFaQwd1q2ndyxhs/ezSMpzaJDpM1ouWKMklbCOf6ZvnEgLB7kuoMcNpogsodJ0RIUV6xDYVLi460cjii7uAUxtGsVPx8DzQDKLEjHEdTZKJV9lseSPM9vymHa350ATD/A1vg2pb8WMuykh3il4LASDuMRQgBJ3GE9pTJ7tVP08ttF2tbYblTul6IY3vaGYsR1sqlNyjTvi0s6bvyk2XZqtIgSLYWkJqcD7tTpBPYfEUfz9F2H5nXOlH+fcAi2vZVYFrnHSk2jz0CjMu0i51F3SD/EhC6lH9H fW7x1zM1 lErPLBv2U+aNT8q95xh7a7Tig7quDFd+pQOCBmmznJv+tZA8mdtsUpqHV9Fi84u1tgzOsfajfg3LIaMJcTOuAz3csn3lLFccUVOL8qYHAW0+DfDmjHFjUWIeeR6k4c/28T47WJram/ygREhqNjvlTfc/f9nZ/YZpV3iQ1DMOJ03Q1fs52jYlO0P8Gp9gAVDHS1k5wq2u8k7k+69cbHKUWD5Feyn3iVol4I1zhwwv9xCg7oKWzwGIuj/B3xrypJ52xIh0hIY+8RE90hqhPwp0EyI9exyWoKsph8a26AYrFZC1Ohm5jXMZqwedtzrdXgeitKfvlf6rx8ahIMbkJYxqoqwaoyxRD6JHQeKjUpItA3E/GA4tUOXB18BmJu+aMTx77q8Y7/A9xnsTenUt7L97375vqnH2S8qB6HwYMs9Tgu9jeRlqLdst2mHYLQmM8M1U2neYQ1vdHiFQX1UAvTuiNsgamhron+pNC16mADFT8DME96tbT27zB84Y0uijdHmxy21Uq6RU0EX2LueASiuoFeL73mzRPY4jaVbg6uQ/ROfuGyEDKmJBy1MhuKleJ++wum1tK0FHUs6If0+J+hwgUWXWpT8vSQ24ceORm X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Replace the MMU write locks (taken in the memslot iteration loop) for read locks. Grabbing the read lock instead of the write lock is safe because the only requirement we have is that the stage-2 page tables do not get deallocated while we are walking them. The stage2_age_walker() callback is safe to race with itself; update the comment to reflect the synchronization change. Signed-off-by: James Houghton --- arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/hyp/pgtable.c | 15 +++++++++------ arch/arm64/kvm/mmu.c | 30 ++++++++++++++++++++++-------- 3 files changed, 32 insertions(+), 14 deletions(-) diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig index 58f09370d17e..7a1af8141c0e 100644 --- a/arch/arm64/kvm/Kconfig +++ b/arch/arm64/kvm/Kconfig @@ -22,6 +22,7 @@ menuconfig KVM select KVM_COMMON select KVM_GENERIC_HARDWARE_ENABLING select KVM_GENERIC_MMU_NOTIFIER + select KVM_MMU_NOTIFIER_YOUNG_LOCKLESS select HAVE_KVM_CPU_RELAX_INTERCEPT select KVM_MMIO select KVM_GENERIC_DIRTYLOG_READ_PROTECT diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 9e2bbee77491..a24a2a857456 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -1319,10 +1319,10 @@ static int stage2_age_walker(const struct kvm_pgtable_visit_ctx *ctx, data->young = true; /* - * stage2_age_walker() is always called while holding the MMU lock for - * write, so this will always succeed. Nonetheless, this deliberately - * follows the race detection pattern of the other stage-2 walkers in - * case the locking mechanics of the MMU notifiers is ever changed. + * This walk is not exclusive; the PTE is permitted to change from + * under us. If there is a race to update this PTE, then the GFN is + * most likely young, so failing to clear the AF is likely to be + * inconsequential. */ if (data->mkold && !stage2_try_set_pte(ctx, new)) return -EAGAIN; @@ -1345,10 +1345,13 @@ bool kvm_pgtable_stage2_test_clear_young(struct kvm_pgtable *pgt, u64 addr, struct kvm_pgtable_walker walker = { .cb = stage2_age_walker, .arg = &data, - .flags = KVM_PGTABLE_WALK_LEAF, + .flags = KVM_PGTABLE_WALK_LEAF | + KVM_PGTABLE_WALK_SHARED, }; + int r; - WARN_ON(kvm_pgtable_walk(pgt, addr, size, &walker)); + r = kvm_pgtable_walk(pgt, addr, size, &walker); + WARN_ON_ONCE(r && r != -EAGAIN); return data.young; } diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 6981b1bc0946..e37765f6f2a1 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1912,29 +1912,43 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { u64 size = (range->end - range->start) << PAGE_SHIFT; + bool young = false; + + read_lock(&kvm->mmu_lock); if (!kvm->arch.mmu.pgt) - return false; + goto out; - return kvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt, - range->start << PAGE_SHIFT, - size, true); + young = kvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt, + range->start << PAGE_SHIFT, + size, true); /* * TODO: Handle nested_mmu structures here using the reverse mapping in * a later version of patch series. */ + +out: + read_unlock(&kvm->mmu_lock); + return young; } bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { u64 size = (range->end - range->start) << PAGE_SHIFT; + bool young = false; + + read_lock(&kvm->mmu_lock); if (!kvm->arch.mmu.pgt) - return false; + goto out; - return kvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt, - range->start << PAGE_SHIFT, - size, false); + young = kvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt, + range->start << PAGE_SHIFT, + size, false); + +out: + read_unlock(&kvm->mmu_lock); + return young; } phys_addr_t kvm_mmu_get_httbr(void)