From patchwork Fri Jan 10 18:40:52 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13935255 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A469E7719C for ; Fri, 10 Jan 2025 18:42:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 88DE46B00D7; Fri, 10 Jan 2025 13:41:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 821206B00D8; Fri, 10 Jan 2025 13:41:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 643726B00D9; Fri, 10 Jan 2025 13:41:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 405B36B00D7 for ; Fri, 10 Jan 2025 13:41:48 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id F096EAEA4F for ; Fri, 10 Jan 2025 18:41:47 +0000 (UTC) X-FDA: 82992411054.10.B4C6DBD Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) by imf22.hostedemail.com (Postfix) with ESMTP id 15A84C0002 for ; Fri, 10 Jan 2025 18:41:45 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=dVQM8EW6; spf=pass (imf22.hostedemail.com: domain of 36GmBZwgKCAUofhprfsglttlqj.htrqnsz2-rrp0fhp.twl@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=36GmBZwgKCAUofhprfsglttlqj.htrqnsz2-rrp0fhp.twl@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736534506; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CAllTjvu+aeRJsuZfnIbaARP70yHbpTXyh5BDwNTLpw=; b=3Kd25hRRqpVHj28qrJkFNL3rIa/Y0LoZrX9g3a86FzPgdRVvDBCtE6Wp8fyF6QUeiI4kO5 qBL+keJEvTYQwkDULEeSyRZ9HTfPR502XyiEQFivvx2JqBRAlrtDmkma4XhXsvNNx9WsGm 3IM0JCHl8VpY6qzXGU5bOSRmTTxDraI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736534506; a=rsa-sha256; cv=none; b=FS5OvkVIUQ0tnQ5VvjKCJH0DRuGwuFPyAcQrnZVmQ6XBZ4zhl1yP6ya5rRJSPA9UoRCQ7M t4wN0qaNvxWO1v8SEKyiqIBIEzmm6IQC0hbfWTwNPet4kD9n0rDJ9AxbEp5XcOZV3ieZk+ 5s16HfiyXn6sVcoo8zYoPY9BOHFLK8I= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=dVQM8EW6; spf=pass (imf22.hostedemail.com: domain of 36GmBZwgKCAUofhprfsglttlqj.htrqnsz2-rrp0fhp.twl@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=36GmBZwgKCAUofhprfsglttlqj.htrqnsz2-rrp0fhp.twl@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-43621907030so19343065e9.1 for ; Fri, 10 Jan 2025 10:41:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736534505; x=1737139305; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=CAllTjvu+aeRJsuZfnIbaARP70yHbpTXyh5BDwNTLpw=; b=dVQM8EW6fyH+4Sg9DINJN24bKorjyogwln0AHFl5Ur9GcvmY/EVFxI8FTWJIjkTKGK D5oQ++CnLxRrYy75xZETOPG8ZlqBxvxM4h1GWpRj1hBOYFrbRBhDwlc0e1GztcOAZes+ ya4sgg1FclWfY6fc7LhrTIwPTj82NBht/mJnX6/KHWc58gHL0HZKydODFZHsFI5YS1Up Vz6a45pDRYtWGWXa02lGJXCCtPKo0fJGWhXvzyMH5hBZaj5BZ+Iw/dcg5uOU6WAlUBih Z2xlmkOSQTQ28KeU1KhqW2Qf/NVioQLqX9ZlVKZwAApxi4gOqaNLNgS7CtofTBjNyxbp /YuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736534505; x=1737139305; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=CAllTjvu+aeRJsuZfnIbaARP70yHbpTXyh5BDwNTLpw=; b=vn6G5X8mlobEo+yLEqKYcmLNw1clHNSp2Vm12rrl4KxJaaJflkGJn5Tb0zc/FDtXdA aKssplnOxwBUc/HFOiOGre4jd1RhgCtGMed+rQDuxvXxQ9UX83Giq8Om4xHlvcCq+TOO 0FJbRoi5avgk8HSqSZlJKFoEm9G0HK1sS66uBqyLp2uJZSu0ZdeE9j1OB3ALV4Ums7DQ 47GF6HMdBFQuNaDgTdqvo55xX9ul+ARv1l81ovKtOooebDleEDl6rdSkdsFd9zXMImCL tRVu4E0d1UYjM59CCMpMIbOBH4cueZS6cg80OBNxk8FMAWUWLsapo5BeMuboWSo0N7ff xJiQ== X-Forwarded-Encrypted: i=1; AJvYcCXGmUaTG6Aa+xUtR5AXtHwWMS0norvWJut1JVWcabttc9uGoQD0yzFmrK1Ma8GyBhqI4bHIJcVS3w==@kvack.org X-Gm-Message-State: AOJu0YyhabME9IJ8waWTUPaoNv/a+vfXrg28FlADzzS2Vlu6SvM5Wlyp C/MXEO3xQkxHy0aVaNL9Y0ZqhkEZFLCL0AVeam8C/9L9sX0OaMnfVzKcsXdW/pGmGdk0PsAC+sx GahtJ8LNWAg== X-Google-Smtp-Source: AGHT+IFbu2VfquZETrYawBwFSqjLwW/Qzimn4MDKfCetEVKQSMACYt32ELZsfcDoj+dSkmxIf50qY6QNFb/+Xg== X-Received: from wmqe1.prod.google.com ([2002:a05:600c:4e41:b0:434:a050:ddcf]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:3c85:b0:436:18d0:aa6e with SMTP id 5b1f17b1804b1-436e2679a7cmr125841955e9.5.1736534504638; Fri, 10 Jan 2025 10:41:44 -0800 (PST) Date: Fri, 10 Jan 2025 18:40:52 +0000 In-Reply-To: <20250110-asi-rfc-v2-v2-0-8419288bc805@google.com> Mime-Version: 1.0 References: <20250110-asi-rfc-v2-v2-0-8419288bc805@google.com> X-Mailer: b4 0.15-dev Message-ID: <20250110-asi-rfc-v2-v2-26-8419288bc805@google.com> Subject: [PATCH RFC v2 26/29] x86: Create library for flushing L1D for L1TF From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Richard Henderson , Matt Turner , Vineet Gupta , Russell King , Catalin Marinas , Will Deacon , Guo Ren , Brian Cain , Huacai Chen , WANG Xuerui , Geert Uytterhoeven , Michal Simek , Thomas Bogendoerfer , Dinh Nguyen , Jonas Bonn , Stefan Kristiansson , Stafford Horne , "James E.J. Bottomley" , Helge Deller , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Naveen N Rao , Madhavan Srinivasan , Paul Walmsley , Palmer Dabbelt , Albert Ou , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Yoshinori Sato , Rich Felker , John Paul Adrian Glaubitz , "David S. Miller" , Andreas Larsson , Richard Weinberger , Anton Ivanov , Johannes Berg , Chris Zankel , Max Filippov , Arnd Bergmann , Andrew Morton , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Uladzislau Rezki , Christoph Hellwig , Masami Hiramatsu , Mathieu Desnoyers , Mike Rapoport , Arnaldo Carvalho de Melo , Namhyung Kim , Mark Rutland , Alexander Shishkin , Jiri Olsa , Ian Rogers , Adrian Hunter , Dennis Zhou , Tejun Heo , Christoph Lameter , Sean Christopherson , Paolo Bonzini , Ard Biesheuvel , Josh Poimboeuf , Pawan Gupta Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-alpha@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-um@lists.infradead.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, kvm@vger.kernel.org, linux-efi@vger.kernel.org, Brendan Jackman X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 15A84C0002 X-Stat-Signature: p6q1ak47qumhjktyycmmqm5gc674nf99 X-Rspam-User: X-HE-Tag: 1736534505-499792 X-HE-Meta: U2FsdGVkX18qi/5O830bwZSCaHMU+g/AUMIjs7Cqk7/I3KLgTaxAKrwK+UOJ4voXdp7pVlHLlIBOvMpXkQT5cAhqjzdr2+PIS42keNTd6Pp//XtsQx3rzroFHIwfRcs+CSjd4Lo4zpevV3/4MjemY3Uz2dUfR72/3zfrohytL3hCvOuD8ONRXG5uD5IkXOP40ROJePdXYkuyEwZ6BQbTo897Gx/mHR4d2pWA2o3uRSSDyQ+U2xsW8BROK55iNrib/zpnk6o62c441XYLn8MHw4fhYhe05ZJEOCIE3JiOTJVuggUtX5tn0IhtmqYpRzms9VDuIB3bpyuCh3eQFcnwAZQXDhC8beVouIBBXM3PBEEjDKnlidPqJniDNNMU9wUhi9D//Q1LsV5k+NnIjnshgT5LWHvwMYn12+EcEWEEu7EtEmU74TTEErfoSP1AJaQtTWGU2H+POw12X5J6yxB9N7AH//Ub+mpVZKoNRow5XZ6ESPTyCtd96pcVbdsHrcLPwj3ooycpqfv1UypvmnZ/skEMNRe8eib1eI3BEJacaHCv1JYpxTD5mq5icr+TRigEtKXoSw8kx8tb4sBtGzJQb36XtylRzAKXWADdjSjgf2OPdsqDf3jy2lGqYPAvMMSvO7Il5LxSbJdzu7lRIGvQCLCGR6jDGJ8iFMczwG7QvFO22VCEigrffz7WN/+pk7vNlpBf4bFbi2wrwuEOkK7ova71Uny66o0SA67v6ijQC1cXLuaBguMGnaqg/w5/FEl7xW+6BhQaJdd3pGmsJuRushY3DmbopibPQXZ96S4b5jaWZSKiuh6a/jB735Y/Dy5Ntu6ok7aWcbmBQG1oiKCBrZAaJnCtVKqTlvHqIm7LgHokFVr3G4v0rW6/xg6MyNvmhti7vvjuxk6hTHkNfILXV7neKuS1EN6GPjoT5Fgcr3EajRCBxV3mApkEQzwJ1XWjbENqr4qYjJfEfkdeCC+ 9qBRLje2 aUULbGHv7mzDttipfAztLB8/uGC5m8VuiDxJ4RYLpJKcbGhXZJ++XAZCcfBPdHnIOuxLWIOHLZbGLiqQkLuozh+SUFtFY8dq8sgBjtwAmWlBGy5ox0z4u7U0ea2NwOik/yBSsOI1PWC586VJMLaXLwi1gSJnhaF8nLMvBJbUaGPllrkzQ0+85tOqL9dtAijCFF5uoKzWuvkPjMXInCyojaDwcjjXw9cfsJC70uW+8KeMPfTI+xKLELb9/y0jil2MCclHOwAiJRJLGv+nDt6D6OaNHdG38aQGkunMEjYO3oWye8MLTXbB8r35cJZY6A7ij1dWm/wcY/vaHzYGB1GUpw7ogfJsh9VtszXBlWYG3XaXh0lJeT+5oG2CwFFHoZZlW/5IoiksiBrQeYK4THGg9bt9cNl8ZWxuooVO7yW2YQoyqH6eKvThouqq1oVPm2rqu2pAYfWLkKQeVU/Thw3ryoQWNcbJWUaEamg/DiyNGinrp1CW51PUibIBKvGZ/7t4jRpUcm0htlB+k9dDnOr4uVhCdgqOV2jt2YKerlCLOz8akcmWPjTT1oxdKtzpxEblLRvP7mWxWBxeiEyGi9wZ+avQ473YGzNUT1yebiTVYm0POHT4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: ASI will need to use this L1D flushing logic so put it in a library where it can be used independently of KVM. Since we're creating this library, it starts to look messy if we don't also use it in the double-opt-in (both kernel cmdline and prctl) mm-switching flush logic which is there for mitigating Snoop-Assisted L1 Data Sampling ("SAL1DS"). However, that logic doesn't use any software-based fallback for flushing on CPUs without the L1D_FLUSH command. In that case the prctl opt-in will fail. One option would be to just start using the software fallback sequence currently done by VMX code, but Linus didn't seem happy with a similar sequence being used here [1]. CPUs affected by SAL1DS are a subset of those affected by L1TF, so it wouldn't be completely insane to assume that the same sequence works for both cases, but I'll err on the side of caution and avoid risk of giving users a false impression that the kernel has really flushed L1D for them. [1] https://lore.kernel.org/linux-kernel/CAHk-=whC4PUhErcoDhCbTOdmPPy-Pj8j9ytsdcyz9TorOb4KUw@mail.gmail.com/ Instead, create this awkward library that is scoped specifically to L1TF, which will be used only by VMX and ASI, and has an annoying "only sometimes works" doc-comment. Users of the library can then infer from that comment whether they have flushed L1D. No functional change intended. Checkpatch-args: --ignore=COMMIT_LOG_LONG_LINE Signed-off-by: Brendan Jackman --- arch/x86/Kconfig | 4 ++ arch/x86/include/asm/l1tf.h | 11 ++++++ arch/x86/kvm/Kconfig | 1 + arch/x86/kvm/vmx/vmx.c | 66 +++---------------------------- arch/x86/lib/Makefile | 1 + arch/x86/lib/l1tf.c | 94 +++++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 117 insertions(+), 60 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index ae31f36ce23d7c29d1e90b726c5a2e6ea5a63c8d..ca984dc7ee2f2b68c3ce1bcb5055047ca4f2a65d 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -2523,6 +2523,7 @@ config MITIGATION_ADDRESS_SPACE_ISOLATION bool "Allow code to run with a reduced kernel address space" default n depends on X86_64 && !PARAVIRT && !UML + select X86_L1TF_FLUSH_LIB help This feature provides the ability to run some kernel code with a reduced kernel address space. This can be used to @@ -3201,6 +3202,9 @@ config HAVE_ATOMIC_IOMAP def_bool y depends on X86_32 +config X86_L1TF_FLUSH_LIB + def_bool n + source "arch/x86/kvm/Kconfig" source "arch/x86/Kconfig.assembler" diff --git a/arch/x86/include/asm/l1tf.h b/arch/x86/include/asm/l1tf.h new file mode 100644 index 0000000000000000000000000000000000000000..e0be19c588bb5ec5c76a1861492e48b88615b4b8 --- /dev/null +++ b/arch/x86/include/asm/l1tf.h @@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_L1TF_FLUSH_H +#define _ASM_L1TF_FLUSH_H + +#ifdef CONFIG_X86_L1TF_FLUSH_LIB +int l1tf_flush_setup(void); +void l1tf_flush(void); +#endif /* CONFIG_X86_L1TF_FLUSH_LIB */ + +#endif + diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig index f09f13c01c6bbd28fa37fdf50547abf4403658c9..81c71510e33e52447882ab7b22682199c57b492e 100644 --- a/arch/x86/kvm/Kconfig +++ b/arch/x86/kvm/Kconfig @@ -92,6 +92,7 @@ config KVM_SW_PROTECTED_VM config KVM_INTEL tristate "KVM for Intel (and compatible) processors support" depends on KVM && IA32_FEAT_CTL + select X86_L1TF_FLUSH_LIB help Provides support for KVM on processors equipped with Intel's VT extensions, a.k.a. Virtual Machine Extensions (VMX). diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 0e90463f1f2183b8d716f85d5c8a8af8958fef0b..b1a02f27b3abce0ef6ac448b66bef2c653a52eef 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -42,6 +42,7 @@ #include #include #include +#include #include #include #include @@ -250,9 +251,6 @@ static void *vmx_l1d_flush_pages; static int vmx_setup_l1d_flush(enum vmx_l1d_flush_state l1tf) { - struct page *page; - unsigned int i; - if (!boot_cpu_has_bug(X86_BUG_L1TF)) { l1tf_vmx_mitigation = VMENTER_L1D_FLUSH_NOT_REQUIRED; return 0; @@ -288,26 +286,11 @@ static int vmx_setup_l1d_flush(enum vmx_l1d_flush_state l1tf) l1tf = VMENTER_L1D_FLUSH_ALWAYS; } - if (l1tf != VMENTER_L1D_FLUSH_NEVER && !vmx_l1d_flush_pages && - !boot_cpu_has(X86_FEATURE_FLUSH_L1D)) { - /* - * This allocation for vmx_l1d_flush_pages is not tied to a VM - * lifetime and so should not be charged to a memcg. - */ - page = alloc_pages(GFP_KERNEL, L1D_CACHE_ORDER); - if (!page) - return -ENOMEM; - vmx_l1d_flush_pages = page_address(page); + if (l1tf != VMENTER_L1D_FLUSH_NEVER) { + int err = l1tf_flush_setup(); - /* - * Initialize each page with a different pattern in - * order to protect against KSM in the nested - * virtualization case. - */ - for (i = 0; i < 1u << L1D_CACHE_ORDER; ++i) { - memset(vmx_l1d_flush_pages + i * PAGE_SIZE, i + 1, - PAGE_SIZE); - } + if (err) + return err; } l1tf_vmx_mitigation = l1tf; @@ -6652,20 +6635,8 @@ int vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath) return ret; } -/* - * Software based L1D cache flush which is used when microcode providing - * the cache control MSR is not loaded. - * - * The L1D cache is 32 KiB on Nehalem and later microarchitectures, but to - * flush it is required to read in 64 KiB because the replacement algorithm - * is not exactly LRU. This could be sized at runtime via topology - * information but as all relevant affected CPUs have 32KiB L1D cache size - * there is no point in doing so. - */ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) { - int size = PAGE_SIZE << L1D_CACHE_ORDER; - /* * This code is only executed when the flush mode is 'cond' or * 'always' @@ -6695,32 +6666,7 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) vcpu->stat.l1d_flush++; - if (static_cpu_has(X86_FEATURE_FLUSH_L1D)) { - native_wrmsrl(MSR_IA32_FLUSH_CMD, L1D_FLUSH); - return; - } - - asm volatile( - /* First ensure the pages are in the TLB */ - "xorl %%eax, %%eax\n" - ".Lpopulate_tlb:\n\t" - "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t" - "addl $4096, %%eax\n\t" - "cmpl %%eax, %[size]\n\t" - "jne .Lpopulate_tlb\n\t" - "xorl %%eax, %%eax\n\t" - "cpuid\n\t" - /* Now fill the cache */ - "xorl %%eax, %%eax\n" - ".Lfill_cache:\n" - "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t" - "addl $64, %%eax\n\t" - "cmpl %%eax, %[size]\n\t" - "jne .Lfill_cache\n\t" - "lfence\n" - :: [flush_pages] "r" (vmx_l1d_flush_pages), - [size] "r" (size) - : "eax", "ebx", "ecx", "edx"); + l1tf_flush(); } void vmx_update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int irr) diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile index 98583a9dbab337e09a2e58905e5200499a496a07..b0a45bd70b40743a3fccb352b9641caacac83275 100644 --- a/arch/x86/lib/Makefile +++ b/arch/x86/lib/Makefile @@ -37,6 +37,7 @@ lib-$(CONFIG_INSTRUCTION_DECODER) += insn.o inat.o insn-eval.o lib-$(CONFIG_RANDOMIZE_BASE) += kaslr.o lib-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o lib-$(CONFIG_MITIGATION_RETPOLINE) += retpoline.o +lib-$(CONFIG_X86_L1TF_FLUSH_LIB) += l1tf.o obj-y += msr.o msr-reg.o msr-reg-export.o hweight.o obj-y += iomem.o diff --git a/arch/x86/lib/l1tf.c b/arch/x86/lib/l1tf.c new file mode 100644 index 0000000000000000000000000000000000000000..c474f18ae331c8dfa7a029c457dd3cf75bebf808 --- /dev/null +++ b/arch/x86/lib/l1tf.c @@ -0,0 +1,94 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + +#include +#include +#include + +#define L1D_CACHE_ORDER 4 +static void *l1tf_flush_pages; + +int l1tf_flush_setup(void) +{ + struct page *page; + unsigned int i; + + if (l1tf_flush_pages || boot_cpu_has(X86_FEATURE_FLUSH_L1D)) + return 0; + + page = alloc_pages(GFP_KERNEL, L1D_CACHE_ORDER); + if (!page) + return -ENOMEM; + l1tf_flush_pages = page_address(page); + + /* + * Initialize each page with a different pattern in + * order to protect against KSM in the nested + * virtualization case. + */ + for (i = 0; i < 1u << L1D_CACHE_ORDER; ++i) { + memset(l1tf_flush_pages + i * PAGE_SIZE, i + 1, + PAGE_SIZE); + } + + return 0; +} +EXPORT_SYMBOL(l1tf_flush_setup); + +/* + * Flush L1D in a way that: + * + * - definitely works on CPUs X86_FEATURE_FLUSH_L1D (because the SDM says so). + * - almost definitely works on other CPUs with L1TF (because someone on LKML + * said someone from Intel said so). + * - may or may not work on other CPUs. + * + * Don't call unless l1tf_flush_setup() has returned successfully. + */ +noinstr void l1tf_flush(void) +{ + int size = PAGE_SIZE << L1D_CACHE_ORDER; + + if (static_cpu_has(X86_FEATURE_FLUSH_L1D)) { + native_wrmsrl(MSR_IA32_FLUSH_CMD, L1D_FLUSH); + return; + } + + if (WARN_ON(!l1tf_flush_pages)) + return; + + /* + * This sequence was provided by Intel for the purpose of mitigating + * L1TF on VMX. + * + * The L1D cache is 32 KiB on Nehalem and some later microarchitectures, + * but to flush it is required to read in 64 KiB because the replacement + * algorithm is not exactly LRU. This could be sized at runtime via + * topology information but as all relevant affected CPUs have 32KiB L1D + * cache size there is no point in doing so. + */ + asm volatile( + /* First ensure the pages are in the TLB */ + "xorl %%eax, %%eax\n" + ".Lpopulate_tlb:\n\t" + "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t" + "addl $4096, %%eax\n\t" + "cmpl %%eax, %[size]\n\t" + "jne .Lpopulate_tlb\n\t" + "xorl %%eax, %%eax\n\t" + "cpuid\n\t" + /* Now fill the cache */ + "xorl %%eax, %%eax\n" + ".Lfill_cache:\n" + "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t" + "addl $64, %%eax\n\t" + "cmpl %%eax, %[size]\n\t" + "jne .Lfill_cache\n\t" + "lfence\n" + :: [flush_pages] "r" (l1tf_flush_pages), + [size] "r" (size) + : "eax", "ebx", "ecx", "edx"); +} +EXPORT_SYMBOL(l1tf_flush);