From patchwork Tue Sep 10 16:30:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Roy X-Patchwork-Id: 13798888 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D923ECE58A for ; Tue, 10 Sep 2024 16:31:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D92748D0093; Tue, 10 Sep 2024 12:31:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D19A18D0002; Tue, 10 Sep 2024 12:31:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B6CE38D0093; Tue, 10 Sep 2024 12:31:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 95A888D0002 for ; Tue, 10 Sep 2024 12:31:43 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 457C9A0853 for ; Tue, 10 Sep 2024 16:31:43 +0000 (UTC) X-FDA: 82549369686.21.1A89E45 Received: from smtp-fw-6002.amazon.com (smtp-fw-6002.amazon.com [52.95.49.90]) by imf18.hostedemail.com (Postfix) with ESMTP id 4490E1C000D for ; Tue, 10 Sep 2024 16:31:41 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=amazon.co.uk header.s=amazon201209 header.b="L/x6q0sZ"; spf=pass (imf18.hostedemail.com: domain of "prvs=976277991=roypat@amazon.co.uk" designates 52.95.49.90 as permitted sender) smtp.mailfrom="prvs=976277991=roypat@amazon.co.uk"; dmarc=pass (policy=quarantine) header.from=amazon.co.uk ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725985798; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nrEWCAm2NUlH3H3kjrWi9+hHd8dgkyixf/clN7dg47w=; b=3s8O2qleo7vyIhyKxRF8FQIKTJ1qokyZF9W9e6PThtjJmqDF1YMwaP7kBpRncEXn9pEi2x I8+LAWIiALKSKpcKWzrWWBVH17c0a+Pa0DTu3Hg52v1hy28qC8h3xCoWLlYniiNt94hicy 0hb59z7k51I3R3yFVEvWIx0tREEuQLs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725985798; a=rsa-sha256; cv=none; b=IP9xmBcKaVkTGgzVR/HnvifaFJ4iYJuysd063qmezkAFOjGwkqzTRUxyFXGp1rS80FhxLE 5+NWeIo8/s93qha9J1aYDdRd4ZDLsEtG5hRDeA0Ni3C9rnoJi7pRjJ88RIyV3q2DuwXMea 9z3ClQqeBmBoHb7/Df2xTKWTkh0Efg0= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=amazon.co.uk header.s=amazon201209 header.b="L/x6q0sZ"; spf=pass (imf18.hostedemail.com: domain of "prvs=976277991=roypat@amazon.co.uk" designates 52.95.49.90 as permitted sender) smtp.mailfrom="prvs=976277991=roypat@amazon.co.uk"; dmarc=pass (policy=quarantine) header.from=amazon.co.uk DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.co.uk; i=@amazon.co.uk; q=dns/txt; s=amazon201209; t=1725985902; x=1757521902; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=nrEWCAm2NUlH3H3kjrWi9+hHd8dgkyixf/clN7dg47w=; b=L/x6q0sZWZg6KJQiavFHq6sKC9wGFqBhmSWtOa6NRVqbl3vNml9+WA4X lM8idV5vgYXiojVMNEIoJiKgtO6dUQyOjHJboF6XFgyVVYDcHbBhc7/OR Dg9kK2+Yyiqpabg1xYmY4eyZy2ubj+lQN9cDkf6FtghbTma7Ses9+qLDd 4=; X-IronPort-AV: E=Sophos;i="6.10,217,1719878400"; d="scan'208";a="432478644" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO smtpout.prod.us-east-1.prod.farcaster.email.amazon.dev) ([10.43.8.6]) by smtp-border-fw-6002.iad6.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Sep 2024 16:31:39 +0000 Received: from EX19MTAUEB001.ant.amazon.com [10.0.44.209:27167] by smtpin.naws.us-east-1.prod.farcaster.email.amazon.dev [10.0.42.209:2525] with esmtp (Farcaster) id f943c050-93c0-47aa-b307-1815f2e5f7c0; Tue, 10 Sep 2024 16:31:37 +0000 (UTC) X-Farcaster-Flow-ID: f943c050-93c0-47aa-b307-1815f2e5f7c0 Received: from EX19D008UEA004.ant.amazon.com (10.252.134.191) by EX19MTAUEB001.ant.amazon.com (10.252.135.108) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34; Tue, 10 Sep 2024 16:31:32 +0000 Received: from EX19MTAUWB001.ant.amazon.com (10.250.64.248) by EX19D008UEA004.ant.amazon.com (10.252.134.191) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34; Tue, 10 Sep 2024 16:31:31 +0000 Received: from ua2d7e1a6107c5b.home (172.19.88.180) by mail-relay.amazon.com (10.250.64.254) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34 via Frontend Transport; Tue, 10 Sep 2024 16:31:27 +0000 From: Patrick Roy To: , , , , , , , , , , , , , , , , , , , , CC: Patrick Roy , , , , , Subject: [RFC PATCH v2 08/10] kvm: pfncache: Support caching gmem pfns Date: Tue, 10 Sep 2024 17:30:34 +0100 Message-ID: <20240910163038.1298452-9-roypat@amazon.co.uk> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20240910163038.1298452-1-roypat@amazon.co.uk> References: <20240910163038.1298452-1-roypat@amazon.co.uk> MIME-Version: 1.0 X-Stat-Signature: rssbzahdkcgifthoae4grq6c3ef1myrg X-Rspamd-Queue-Id: 4490E1C000D X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1725985901-439184 X-HE-Meta: U2FsdGVkX19o/c7emcRuc/A1tqU43otwLgXuOCXw7LFaJ/ZQC5RFmuRKIQc9fyYazY8AL9hQghgT0AF6eU6LF12L8sy+5q5zq5x0cu13OFLj02FAw8QpV0GehqSymLcc5YarK1tYLr1BvUPfxn6Rihln51yEEuAwNHMqDrqiZGKxBKfepqrxIrlV+MfiPZc/UZpzAK4O1UmK8F4m42VL4uFS4aAmCT+ps1qT0GAUaaKmmv8IZaZj7Zr6O2Omb9+7oZCTed472XG0dQ4aSh9dPuQ8zFdhgCOzv6XouFUIeORY3UHrgjcwJ2kuCbrWxCgHJk0Uq5ATXlRpa9EWgtFMcaxPnZ1MAsaDByvcIfR6JA7LOZnDFmznBLv/VWwrtv3AjZwCpeZxDgVUlOV857fRjhS7LVCuw09YVhjO0XIhEGXGRKxn5Jn92E4CnCr+eLekNez8tPZi0L0BFt7e66Db0FHv/7dnQnc8B6l6MIkZ76NUbfmtKu1Hl2AoXpQFfcdF2Fv9taitMfwGddDWHTg722unwfCYlLkTkRcNR7s+1RFlLkx9tBTf2pc1UeVDxPVxixO6NNuSJyEJ4moeJuV3jTH9Wjj2NYKp02SD5E3fhA37GOssZjsksdhQdlJ9gTgx3m2kzJTPudn48X7220kttnMyjl5B5zebN0XnFKIwnv805qdrv8M3IXE/R77Sv7es4kRuh/mJK9EWTU8qr2oKbKn2QMs7IgtIiELp0KgtuhBR5roVYwDhQnGSWK/9+rkenW8NdWR0Dzy16GpJBbN5ZMMa3GEFnv3wFfsqZYxifLJp6k6EVnhFO6iuP0BiuxuWicLcH5Gy6XdJCxwc0lZVEdmoXBwIkVgW7h5JCnc3uncWSjGWKGVR5zoM6lsF93B8hhqkZdZvYFr2d7uMxTlLYE+jjP3IF2McYV4rvUnId/tCaupvCWiX9Ays73Cx70d5fHfaagMrxOkwB34UuEV wMnHGdQD 4DASqW4wFSTM2KSL3qwNB12hygtg5oqOPn5jzIAfrplvNfF0lZ+JY0ISYIdl9+ukREoJHlCWKySxGsX5QYMTcVX7/n13vjYbmwXuFN+3Fy7ciN5rCT7MVTcszW4588Z1/8K+dlnj0vfrWCR2Qf9sf4jpSfPTRmqG2WCQuUJrSjblZIdQm5tQC3g9rsjnPpJuKaKabHMGugBakmJz6ruOw4wTv3S7hW/qtED74iOgofC+M8pzYsemQ+zcyyu76jdMauf2twXQuRXMyw2YJJJR9OLQ6CCm0ksPJYTf6MWRirGzfmoARLB/RU9YqAEV1/fL3jArqa+++kwzsjblLzZ8iRoh3t7DSpMIhaJCxpR1opR/0hUL62REa5Oqz0NuY6IzgrIat X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Inside the `hva_to_pfn_retry` loop, for gpa based gpcs, check whether the gpa has KVM_MEMORY_ATTRIBUTE_PRIVATE set, and if so, use `kvm_gmem_get_pfn` with `KVM_GMEM_GET_PFN_SHARED` to resolve the pfn. Ignore uhva based gpcs for now, as they are only used with Xen, and we don't have guest_memfd there (yet). Gmem pfns that are cached by a gpc have their sharing refcount elevated until the gpc gets invalidated (or rather: until it gets refreshed after invalidation) or deactivated. Since during the refresh loop the memory attributes could change between private shared, store a uhva anyway, even if it will not be used in the translation in the end. Signed-off-by: Patrick Roy --- include/linux/kvm_types.h | 1 + virt/kvm/pfncache.c | 63 ++++++++++++++++++++++++++++++++++----- 2 files changed, 56 insertions(+), 8 deletions(-) diff --git a/include/linux/kvm_types.h b/include/linux/kvm_types.h index 827ecc0b7e10a..8903b8f46cf6c 100644 --- a/include/linux/kvm_types.h +++ b/include/linux/kvm_types.h @@ -70,6 +70,7 @@ struct gfn_to_pfn_cache { kvm_pfn_t pfn; bool active; bool valid; + bool private; }; #ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c index 6de934a8a153f..a4f935e80f545 100644 --- a/virt/kvm/pfncache.c +++ b/virt/kvm/pfncache.c @@ -16,6 +16,7 @@ #include #include #include +#include #include "kvm_mm.h" @@ -145,13 +146,20 @@ static void *gpc_map(kvm_pfn_t pfn) #endif } -static void gpc_unmap(kvm_pfn_t pfn, void *khva) +static void gpc_unmap(kvm_pfn_t pfn, void *khva, bool private) { /* Unmap the old pfn/page if it was mapped before. */ if (is_error_noslot_pfn(pfn) || !khva) return; if (pfn_valid(pfn)) { + if (private) { + struct folio *folio = pfn_folio(pfn); + + folio_lock(folio); + kvm_gmem_put_shared_pfn(pfn); + folio_unlock(folio); + } kunmap(pfn_to_page(pfn)); return; } @@ -203,6 +211,7 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc) void *old_khva = (void *)PAGE_ALIGN_DOWN((uintptr_t)gpc->khva); kvm_pfn_t new_pfn = KVM_PFN_ERR_FAULT; void *new_khva = NULL; + bool private = gpc->private; unsigned long mmu_seq; lockdep_assert_held(&gpc->refresh_lock); @@ -235,17 +244,43 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc) * the existing mapping and didn't create a new one. */ if (new_khva != old_khva) - gpc_unmap(new_pfn, new_khva); + gpc_unmap(new_pfn, new_khva, private); kvm_release_pfn_clean(new_pfn); cond_resched(); } - /* We always request a writeable mapping */ - new_pfn = hva_to_pfn(gpc->uhva, false, false, NULL, true, NULL); - if (is_error_noslot_pfn(new_pfn)) - goto out_error; + /* + * If we do not have a GPA, we cannot immediately determine + * whether the area of guest memory gpc->uhva pointed to + * is currently set to shared. So assume that uhva-based gpcs + * never have their underlying guest memory switched to + * private (which we can do as uhva-based gpcs are only used + * with Xen, and guest_memfd is not supported there). + */ + if (gpc->gpa != INVALID_GPA) { + /* + * mmu_notifier events can be due to shared/private conversions, + * thus recheck this every iteration. + */ + private = kvm_mem_is_private(gpc->kvm, gpa_to_gfn(gpc->gpa)); + } else { + private = false; + } + + if (private) { + int r = kvm_gmem_get_pfn(gpc->kvm, gpc->memslot, gpa_to_gfn(gpc->gpa), + &new_pfn, NULL, KVM_GMEM_GET_PFN_SHARED); + if (r) + goto out_error; + } else { + /* We always request a writeable mapping */ + new_pfn = hva_to_pfn(gpc->uhva, false, false, NULL, + true, NULL); + if (is_error_noslot_pfn(new_pfn)) + goto out_error; + } /* * Obtain a new kernel mapping if KVM itself will access the @@ -274,6 +309,7 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc) gpc->valid = true; gpc->pfn = new_pfn; gpc->khva = new_khva + offset_in_page(gpc->uhva); + gpc->private = private; /* * Put the reference to the _new_ pfn. The pfn is now tracked by the @@ -298,6 +334,7 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned l kvm_pfn_t old_pfn; bool hva_change = false; void *old_khva; + bool old_private; int ret; /* Either gpa or uhva must be valid, but not both */ @@ -316,6 +353,7 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned l old_pfn = gpc->pfn; old_khva = (void *)PAGE_ALIGN_DOWN((uintptr_t)gpc->khva); old_uhva = PAGE_ALIGN_DOWN(gpc->uhva); + old_private = gpc->private; if (kvm_is_error_gpa(gpa)) { page_offset = offset_in_page(uhva); @@ -338,6 +376,11 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned l gpc->gpa = gpa; gpc->generation = slots->generation; gpc->memslot = __gfn_to_memslot(slots, gfn); + /* + * compute the uhva even for private memory, in case an + * invalidation event flips memory from private to + * shared while in hva_to_pfn_retry + */ gpc->uhva = gfn_to_hva_memslot(gpc->memslot, gfn); if (kvm_is_error_hva(gpc->uhva)) { @@ -395,7 +438,7 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned l write_unlock_irq(&gpc->lock); if (unmap_old) - gpc_unmap(old_pfn, old_khva); + gpc_unmap(old_pfn, old_khva, old_private); return ret; } @@ -486,6 +529,7 @@ void kvm_gpc_deactivate(struct gfn_to_pfn_cache *gpc) struct kvm *kvm = gpc->kvm; kvm_pfn_t old_pfn; void *old_khva; + bool old_private; guard(mutex)(&gpc->refresh_lock); @@ -508,6 +552,9 @@ void kvm_gpc_deactivate(struct gfn_to_pfn_cache *gpc) old_khva = gpc->khva - offset_in_page(gpc->khva); gpc->khva = NULL; + old_private = gpc->private; + gpc->private = false; + old_pfn = gpc->pfn; gpc->pfn = KVM_PFN_ERR_FAULT; write_unlock_irq(&gpc->lock); @@ -516,6 +563,6 @@ void kvm_gpc_deactivate(struct gfn_to_pfn_cache *gpc) list_del(&gpc->list); spin_unlock(&kvm->gpc_lock); - gpc_unmap(old_pfn, old_khva); + gpc_unmap(old_pfn, old_khva, old_private); } }