From patchwork Tue Jul 9 13:20:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Roy X-Patchwork-Id: 13727954 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D134CC3DA41 for ; Tue, 9 Jul 2024 13:21:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 733186B00B3; Tue, 9 Jul 2024 09:21:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6BBFB6B00B4; Tue, 9 Jul 2024 09:21:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 537DA6B00B5; Tue, 9 Jul 2024 09:21:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2BF736B00B3 for ; Tue, 9 Jul 2024 09:21:26 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id DC2EA1618C2 for ; Tue, 9 Jul 2024 13:21:25 +0000 (UTC) X-FDA: 82320275730.01.5CDF233 Received: from smtp-fw-9105.amazon.com (smtp-fw-9105.amazon.com [207.171.188.204]) by imf20.hostedemail.com (Postfix) with ESMTP id C89A11C001B for ; Tue, 9 Jul 2024 13:21:22 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=amazon.co.uk header.s=amazon201209 header.b=SJ4rfNYq; dmarc=pass (policy=quarantine) header.from=amazon.co.uk; spf=pass (imf20.hostedemail.com: domain of "prvs=913fd7204=roypat@amazon.co.uk" designates 207.171.188.204 as permitted sender) smtp.mailfrom="prvs=913fd7204=roypat@amazon.co.uk" ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720531260; a=rsa-sha256; cv=none; b=rkvWypcqIlzXy/O6UqIVyj5aRBIxSge7rcfSQoYQWMy2urG/lKNr1B8mCyaajV2as6EKdO xw2XuLdyYnYDwFeIxz8cJVRXKwsAAzvnXsoeTGL7Y67BdtYwFf+w/FwusbGaSfdziJYVe0 21OW1uyX1BBmNSX7Cf0i5PW6Um3YDMU= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=amazon.co.uk header.s=amazon201209 header.b=SJ4rfNYq; dmarc=pass (policy=quarantine) header.from=amazon.co.uk; spf=pass (imf20.hostedemail.com: domain of "prvs=913fd7204=roypat@amazon.co.uk" designates 207.171.188.204 as permitted sender) smtp.mailfrom="prvs=913fd7204=roypat@amazon.co.uk" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720531260; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+mzO007algvViExHdnpknhkipITD3cHhlfbBcPgvLYk=; b=q83OdqtzFpcXbslKGNJJmaS7vFxgA0rlMDnjk1vE81nT4xjpZUu0nrjgFpxgDB1b/aGP34 kpvYnlOmcaAcWeyelLOVgnmrVC27eXWtqSG2ydRryQslLhml5kvW0pQhfTF6TDL45wn305 wMh0h+zh6L1Yko2qadMpMglne7yQt7M= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.co.uk; i=@amazon.co.uk; q=dns/txt; s=amazon201209; t=1720531283; x=1752067283; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+mzO007algvViExHdnpknhkipITD3cHhlfbBcPgvLYk=; b=SJ4rfNYqkGBW89jP4cSGSgwSqCGGFeraGPQGashE8UVRJol6I0HjpRPE ZKIvfahZlC7ki/iojpPdOr4CVcPyZSJniPUKUatv1wdSmvy3ikqrp2QOY 0BBbE0rdpBQt+luO4kYI/0djHT7jlSBT/xH8wAyC//2eFVhNLxPzgEnTI Q=; X-IronPort-AV: E=Sophos;i="6.09,195,1716249600"; d="scan'208";a="739970201" Received: from pdx4-co-svc-p1-lb2-vlan2.amazon.com (HELO smtpout.prod.us-east-1.prod.farcaster.email.amazon.dev) ([10.25.36.210]) by smtp-border-fw-9105.sea19.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jul 2024 13:21:15 +0000 Received: from EX19MTAUEC001.ant.amazon.com [10.0.0.204:50152] by smtpin.naws.us-east-1.prod.farcaster.email.amazon.dev [10.0.92.61:2525] with esmtp (Farcaster) id 1a418e6e-d33b-4bef-8883-26ade4dd88c5; Tue, 9 Jul 2024 13:21:13 +0000 (UTC) X-Farcaster-Flow-ID: 1a418e6e-d33b-4bef-8883-26ade4dd88c5 Received: from EX19D008UEA004.ant.amazon.com (10.252.134.191) by EX19MTAUEC001.ant.amazon.com (10.252.135.222) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34; Tue, 9 Jul 2024 13:21:10 +0000 Received: from EX19MTAUEC001.ant.amazon.com (10.252.135.222) by EX19D008UEA004.ant.amazon.com (10.252.134.191) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34; Tue, 9 Jul 2024 13:21:10 +0000 Received: from ua2d7e1a6107c5b.ant.amazon.com (172.19.88.180) by mail-relay.amazon.com (10.252.135.200) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34 via Frontend Transport; Tue, 9 Jul 2024 13:21:07 +0000 From: Patrick Roy To: , , , , , CC: Patrick Roy , , , , , , , , , , , , , , , , , Subject: [RFC PATCH 5/8] kvm: gmem: add option to remove guest private memory from direct map Date: Tue, 9 Jul 2024 14:20:33 +0100 Message-ID: <20240709132041.3625501-6-roypat@amazon.co.uk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240709132041.3625501-1-roypat@amazon.co.uk> References: <20240709132041.3625501-1-roypat@amazon.co.uk> MIME-Version: 1.0 X-Rspamd-Queue-Id: C89A11C001B X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: prnsmyiyrtf36opwmz69ca9pqfxmu4r8 X-HE-Tag: 1720531282-460892 X-HE-Meta: U2FsdGVkX1/zVfGZUMzDdMrHXTRKP+hTkQVZrDKcOvA6oI6MJDFpHPCDfW3XgMscTWTy7sF8O35v3q+vNCPX9HmDohE7KJXTwcwWBAU0A2tbKX0bQyH0g2flpbsjoDdjMGv4Pqfe01VNyu6SuuVBty2Psz9od3+C0f9GNsUbVEA47AvALwwK+ZF0IC7Mxr7NnSZp5sl2lwwiCeiqpCE90oh+98ZkY6P0pP4kgrIzHycJ13hIoLgkMBQEQyCQVR/i1vHYC+UXv0QQ7qGBlRMz6yFMKp1Z7CQ+iaUwjEuuO8UAZtgtwfa0tDKe5weK/oGbboWFPt86gIy9K+Vk6QtYJ9OsxklmDtDq0pBsFCNHYylz2AqkhHCEL9TdojRT+PQhr+/8/U2DxO2qDOl+FwFzR05KlZkg++xa5egwsiH0p5E1piwy+zEdQKJpw+O9z3ILBYCdEIvuWee5A7sUi8hCpOIk0lO9tIQCzgK87lt0j6G4kQiMK4Y2itqKus8McjPXA6T2xQnVMW4Tbd1T9tl7NfOSUUmIvOGVpWRduA7KsietbWjmqSTZGUiLZHmgVTCVkXyvwcWHHB2gDuSYNQCZh3hTT/cWkb8nIiE1ZipbXSwjSVUM7qsUdobf+kD7LLIyuTh3PnyPOUMif3s0tVzYrv9/YovZN6exq7IuGnSs2ikf33fSOqE7Zmn9p33/ikwGtmTqxTIfM0I4Nk9cwHAW5XnxqncUd8XYrRba2xGvAMmXLmKEaZbqWOWgOppuVvzrCQrkTEjepiz0fYac2GB444oE0a0jTQr6NsfP7dR9CDpslPg8OGoubPQhM4cgwza0ZGo4nnD1wtR6AhVZ/OGN5aw03K6b7pnYrjxRluYHdT4kSAm0vYY6lNKBz5IfmHKKklXfpX03IZiBDpgXIGvot5IuWWa6uyEB+C8CENewRIaH5VNkHbpujOcJo2CDeiKrqDANpGlhf2tYJxv3RbB hB47oVKO ODKAokXTtJDE9VfXKnmG3iWaazGw/P7ok5xFYruv5YB+k3H8HYHoGZV8iXt8RZWTy12CeS2f/LTiIr99uwS9PhaBBX772YhbByBqeYlWb1AgpuYEAj5EadKX7sNn1VuQxWAv6xV9GdReHVbelmfDj+e1iNlDLuQ9JqD7n4Ywl6JSI8TTVVXZiMhCkZNe3hU0vofvALm6hdqis2T5OSx2+FS5iK0YWEpLe2rjU6rykLFuNEXEg1PgmnfU2kSij4DfVOwg8i0PS/GpU1E8HlcqsvNWR9aaq6Q9xA2VJxSNATBmLDqyD+vLTUXt0bzv99snnrCE3owQ/fZSsk1IPYdFYjVY+0msB6W6HQZHjU7J5AX3B7v8pdfYggTgfGxr9nURCz/qPzQIb50zJ6S1Unt4+QHItpC+HG3ydbNHY0Y2p4TmBmHE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: While guest_memfd is not available to be mapped by userspace, it is still accessible through the kernel's direct map. This means that in scenarios where guest-private memory is not hardware protected, it can be speculatively read and its contents potentially leaked through hardware side-channels. Removing guest-private memory from the direct map, thus mitigates a large class of speculative execution issues [1, Table 1]. This patch adds a flag to the `KVM_CREATE_GUEST_MEMFD` which, if set, removes the struct pages backing guest-private memory from the direct map. Should `CONFIG_HAVE_KVM_GMEM_{INVALIDATE, PREPARE}` be set, pages are removed after preparation and before invalidation, so that the prepare/invalidate routines do not have to worry about potentially absent direct map entries. Direct map removal do not reuse the `KVM_GMEM_PREPARE` machinery, since `prepare` can be called multiple time, and it is the responsibility of the preparation routine to not "prepare" the same folio twice [2]. Thus, instead explicitly check if `filemap_grab_folio` allocated a new folio, and remove the returned folio from the direct map only if this was the case. The patch uses release_folio instead of free_folio to reinsert pages back into the direct map as by the time free_folio is called, folio->mapping can already be NULL. This means that a call to folio_inode inside free_folio might deference a NULL pointer, leaving no way to access the inode which stores the flags that allow determining whether the page was removed from the direct map in the first place. Lastly, the patch uses set_direct_map_{invalid,default}_noflush instead of `set_memory_[n]p` to avoid expensive flushes of TLBs and the L*-cache hierarchy. This is especially important once KVM restores direct map entries on-demand in later patches, where simple FIO benchmarks of a virtio-blk device have shown that TLB flushes on a Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz resulted in 80% degradation in throughput compared to a non-flushing solution. Not flushing the TLB means that until TLB entries for temporarily restored direct map entries get naturally evicted, they can be used during speculative execution, and effectively "unhide" the memory for longer than intended. We consider this acceptable, as the only pages that are temporarily reinserted into the direct map like this will either hold PV data structures (kvm-clock, asyncpf, etc), or pages containing privileged instructions inside the guest kernel image (in the MMIO emulation case). [1]: https://download.vusec.net/papers/quarantine_raid23.pdf Signed-off-by: Patrick Roy --- include/uapi/linux/kvm.h | 2 ++ virt/kvm/guest_memfd.c | 52 ++++++++++++++++++++++++++++++++++------ 2 files changed, 47 insertions(+), 7 deletions(-) diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index e065d9fe7ab2..409116aa23c9 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -1563,4 +1563,6 @@ struct kvm_create_guest_memfd { __u64 reserved[6]; }; +#define KVM_GMEM_NO_DIRECT_MAP (1ULL << 0) + #endif /* __LINUX_KVM_H */ diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 9148b9679bb1..dc9b0c2d0b0e 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -4,6 +4,7 @@ #include #include #include +#include #include "kvm_mm.h" @@ -49,9 +50,16 @@ static int kvm_gmem_prepare_folio(struct inode *inode, pgoff_t index, struct fol return 0; } +static bool kvm_gmem_not_present(struct inode *inode) +{ + return ((unsigned long)inode->i_private & KVM_GMEM_NO_DIRECT_MAP) != 0; +} + static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index, bool prepare) { struct folio *folio; + bool zap_direct_map = false; + int r; /* TODO: Support huge pages. */ folio = filemap_grab_folio(inode->i_mapping, index); @@ -74,16 +82,30 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index, bool for (i = 0; i < nr_pages; i++) clear_highpage(folio_page(folio, i)); + // We need to clear the folio before calling kvm_gmem_prepare_folio, + // but can only remove it from the direct map _after_ preparation is done. + zap_direct_map = kvm_gmem_not_present(inode); + folio_mark_uptodate(folio); } if (prepare) { - int r = kvm_gmem_prepare_folio(inode, index, folio); - if (r < 0) { - folio_unlock(folio); - folio_put(folio); - return ERR_PTR(r); - } + r = kvm_gmem_prepare_folio(inode, index, folio); + if (r < 0) + goto out_err; + } + + if (zap_direct_map) { + r = set_direct_map_invalid_noflush(&folio->page); + if (r < 0) + goto out_err; + + // We use the private flag to track whether the folio has been removed + // from the direct map. This is because inside of ->free_folio, + // we do not have access to the address_space anymore, meaning we + // cannot check folio_inode(folio)->i_private to determine whether + // KVM_GMEM_NO_DIRECT_MAP was set. + folio_set_private(folio); } /* @@ -91,6 +113,10 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index, bool * unevictable and there is no storage to write back to. */ return folio; +out_err: + folio_unlock(folio); + folio_put(folio); + return ERR_PTR(r); } static void kvm_gmem_invalidate_begin(struct kvm_gmem *gmem, pgoff_t start, @@ -354,10 +380,22 @@ static void kvm_gmem_free_folio(struct folio *folio) } #endif +static void kvm_gmem_invalidate_folio(struct folio *folio, size_t start, size_t end) +{ + if (start == 0 && end == PAGE_SIZE) { + // We only get here if PG_private is set, which only happens if kvm_gmem_not_present + // returned true in kvm_gmem_get_folio. Thus no need to do that check again. + BUG_ON(set_direct_map_default_noflush(&folio->page)); + + folio_clear_private(folio); + } +} + static const struct address_space_operations kvm_gmem_aops = { .dirty_folio = noop_dirty_folio, .migrate_folio = kvm_gmem_migrate_folio, .error_remove_folio = kvm_gmem_error_folio, + .invalidate_folio = kvm_gmem_invalidate_folio, #ifdef CONFIG_HAVE_KVM_GMEM_INVALIDATE .free_folio = kvm_gmem_free_folio, #endif @@ -443,7 +481,7 @@ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args) { loff_t size = args->size; u64 flags = args->flags; - u64 valid_flags = 0; + u64 valid_flags = KVM_GMEM_NO_DIRECT_MAP; if (flags & ~valid_flags) return -EINVAL;