From patchwork Wed Jul 24 01:10:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Houghton X-Patchwork-Id: 13740528 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5F23CC3DA49 for ; Wed, 24 Jul 2024 01:13:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:References:Mime-Version:In-Reply-To:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=lX17S5nNca5CiBEi8R7tHh84KKPLmRWb0mcnxRYOK88=; b=VzaQeyZfiZPzx+NnLm7E4G5vcp IGgfu0ZWU6o965e0Yk8RJcIFlxZVfblap03krDMsrFRRx9JMLoYg/eWfe9KoPGtbKJCZqgorc59Qr NgPBSoezX8jUBDqhnxz3XRTxqk7foaQAcUzgHlFNpXF7cmQxiqFrXhX40yasfiQ4SQAL3OgW2jyFO w9s8Nl7Nstr1bvlwzDp67rnl7FBNd25EvnNCRV7PakDgEZmSEVl+dM8GI2JPtq6PYviC1onfYfvKj DVbiogKom1Ww1hm2AyZ81A3Cmi1qgc3QghUXbvk7j1pXjO/0LOuau/wQ71M+XjAfIjL3IMeMxpHTI 9HrdkU6g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sWQYl-0000000DxuW-2AII; Wed, 24 Jul 2024 01:12:51 +0000 Received: from mail-vs1-xe4a.google.com ([2607:f8b0:4864:20::e4a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sWQXF-0000000Dx8b-3jM2 for linux-arm-kernel@lists.infradead.org; Wed, 24 Jul 2024 01:11:19 +0000 Received: by mail-vs1-xe4a.google.com with SMTP id ada2fe7eead31-49296cd959dso1464862137.1 for ; Tue, 23 Jul 2024 18:11:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1721783476; x=1722388276; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lX17S5nNca5CiBEi8R7tHh84KKPLmRWb0mcnxRYOK88=; b=FpoZWszyXQ7hGokHqR0hFdOI5JJpVaDjBMwIu72uJMHE8tlyyF1acmsFAiWItqruru /VgFPPjI40Eg2kGtc2yF419kJ9nbIppYkYHjHw5nkWWtoXZ5JCkjhABBI7ytHcSHUrDk 4T3hynLWztkuDVuMzhkONgg6Jxe3v8QRaoawtLFrK8ArSc1tGsM6ifuGMHco399ZAeR6 6+A9b07H9nVIMq4p5v6hjzWlb85du8kCBbigNWFrRLfekiQCiYTjHVhNj5wFjGosEiuB EVMJI9A1zURkATgp087dHRjo6jz/7Q7tNsy6sBLtrOF7ZgLkN2R6u4ayWfj7KSXbMtM1 feNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721783476; x=1722388276; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lX17S5nNca5CiBEi8R7tHh84KKPLmRWb0mcnxRYOK88=; b=ZqTOC9MNotmUKVAIplrOE9qZQcbTCiAKOmUtUOwfw1ZucIQFIBuR3npHaJSkDqCvZK U+WaXxdBRPliPqJtemvqrCQdPVsM8T7aKP5UnMSfc/t7BmXp5FOQLDQu4wzr4GnsuKya +hTMGXIs1tj/g/SKrgaxAMEWcrWxr7jYbmGFY2SiayxOZXqOt0glxKF77wPZOflGBU2Z FqmczQScqIV7E0EwFXLOTeQ4Rl3WmWpKOYtrhgchU1NurKZQaBzXC8oM2xRuJRRr2Eus I3oiESnVpOYGf7qHWBAVO3eBLkl8EpmkKg2w4IfqqXf++xDqY1MxureOceknTq/fTud+ KIpA== X-Forwarded-Encrypted: i=1; AJvYcCUBb2RuY5S/t00NbLcT9PyrmblEU2DFKS2QZdOADSO2C2mvN5UdqAe+ed3AdvhW6ppL9uAI97TexXTNy+pYE4k+7k+ToT+q4pzDEnpvFzpOw8V66TI= X-Gm-Message-State: AOJu0YwyfGXpPkZv/jPfk0W8iorginmXgGe8KqxMRxhyehUERSPVmiDD NWTy69xs9drsvg77k1XNWpw3KiT/rktItRWZTUyxULLSUqGr3uC6Da/dopHrTnJfhABpqOb5Jlq iVdSVTcv1acMKvD3yJA== X-Google-Smtp-Source: AGHT+IGlIuIrgkANC0pZBbk8z8He5iptT26RG1V2ZrvO9VkU9yoaN+aGYmYAd60aeXGiO8520IPOKzxK6pnBSRdv X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6102:2ac6:b0:492:9449:c33e with SMTP id ada2fe7eead31-493c19d19c5mr46187137.5.1721783475890; Tue, 23 Jul 2024 18:11:15 -0700 (PDT) Date: Wed, 24 Jul 2024 01:10:28 +0000 In-Reply-To: <20240724011037.3671523-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240724011037.3671523-1-jthoughton@google.com> X-Mailer: git-send-email 2.46.0.rc1.232.g9752f9e123-goog Message-ID: <20240724011037.3671523-4-jthoughton@google.com> Subject: [PATCH v6 03/11] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Ankit Agrawal , Axel Rasmussen , Catalin Marinas , David Matlack , David Rientjes , James Houghton , James Morse , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Oliver Upton , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Suzuki K Poulose , Wei Xu , Will Deacon , Yu Zhao , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240723_181117_961930_91A2985A X-CRM114-Status: GOOD ( 18.11 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Replace the MMU write locks (taken in the memslot iteration loop) for read locks. Grabbing the read lock instead of the write lock is safe because the only requirement we have is that the stage-2 page tables do not get deallocated while we are walking them. The stage2_age_walker() callback is safe to race with itself; update the comment to reflect the synchronization change. Signed-off-by: James Houghton --- arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/hyp/pgtable.c | 15 +++++++++------ arch/arm64/kvm/mmu.c | 30 ++++++++++++++++++++++-------- 3 files changed, 32 insertions(+), 14 deletions(-) diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig index 58f09370d17e..7a1af8141c0e 100644 --- a/arch/arm64/kvm/Kconfig +++ b/arch/arm64/kvm/Kconfig @@ -22,6 +22,7 @@ menuconfig KVM select KVM_COMMON select KVM_GENERIC_HARDWARE_ENABLING select KVM_GENERIC_MMU_NOTIFIER + select KVM_MMU_NOTIFIER_YOUNG_LOCKLESS select HAVE_KVM_CPU_RELAX_INTERCEPT select KVM_MMIO select KVM_GENERIC_DIRTYLOG_READ_PROTECT diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 9e2bbee77491..a24a2a857456 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -1319,10 +1319,10 @@ static int stage2_age_walker(const struct kvm_pgtable_visit_ctx *ctx, data->young = true; /* - * stage2_age_walker() is always called while holding the MMU lock for - * write, so this will always succeed. Nonetheless, this deliberately - * follows the race detection pattern of the other stage-2 walkers in - * case the locking mechanics of the MMU notifiers is ever changed. + * This walk is not exclusive; the PTE is permitted to change from + * under us. If there is a race to update this PTE, then the GFN is + * most likely young, so failing to clear the AF is likely to be + * inconsequential. */ if (data->mkold && !stage2_try_set_pte(ctx, new)) return -EAGAIN; @@ -1345,10 +1345,13 @@ bool kvm_pgtable_stage2_test_clear_young(struct kvm_pgtable *pgt, u64 addr, struct kvm_pgtable_walker walker = { .cb = stage2_age_walker, .arg = &data, - .flags = KVM_PGTABLE_WALK_LEAF, + .flags = KVM_PGTABLE_WALK_LEAF | + KVM_PGTABLE_WALK_SHARED, }; + int r; - WARN_ON(kvm_pgtable_walk(pgt, addr, size, &walker)); + r = kvm_pgtable_walk(pgt, addr, size, &walker); + WARN_ON_ONCE(r && r != -EAGAIN); return data.young; } diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 6981b1bc0946..e37765f6f2a1 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1912,29 +1912,43 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { u64 size = (range->end - range->start) << PAGE_SHIFT; + bool young = false; + + read_lock(&kvm->mmu_lock); if (!kvm->arch.mmu.pgt) - return false; + goto out; - return kvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt, - range->start << PAGE_SHIFT, - size, true); + young = kvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt, + range->start << PAGE_SHIFT, + size, true); /* * TODO: Handle nested_mmu structures here using the reverse mapping in * a later version of patch series. */ + +out: + read_unlock(&kvm->mmu_lock); + return young; } bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { u64 size = (range->end - range->start) << PAGE_SHIFT; + bool young = false; + + read_lock(&kvm->mmu_lock); if (!kvm->arch.mmu.pgt) - return false; + goto out; - return kvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt, - range->start << PAGE_SHIFT, - size, false); + young = kvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt, + range->start << PAGE_SHIFT, + size, false); + +out: + read_unlock(&kvm->mmu_lock); + return young; } phys_addr_t kvm_mmu_get_httbr(void)