From patchwork Fri Jan 10 18:40:53 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13935422 Received: from mail-lj1-f202.google.com (mail-lj1-f202.google.com [209.85.208.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6DABE24B229 for ; Fri, 10 Jan 2025 18:49:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736534970; cv=none; b=cuxXBcxhlpfGiiLIavW4wzqIVVCsMuk/3vpld/Yw+DVKszzUjNev5nDj5r/nMeE3Aq+7Gulc18EcEG7ThtxEyRLeQXAue+IUTp4U3TkK+LXOeQeuboQeyRZRT0Xg7Al5TQ5HybUQljaB+gv1mKnMLnM6l8wdbKz1GEWbuIW89DQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736534970; c=relaxed/simple; bh=Ua0Eznh6vBIqJokkBKNt1xiE4aA9cS4fZDm05XjqWbo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ZYU5yA1R9AwD8d2iFwvcM/m51bMkXU0cJKbEdG9SLBD33paj9mnkFQujzMs/DBtC7MgzxkmDUobHRmlMO3dP6EqlO6etry3RZX2ePHSoRWJUQCH0TWEfDXwJYh6+YJ3kdilO3lCsQ8/u89TY2LyZlaG+ZLHRfuWI4b2F0Jm2YAI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=HBEM+fez; arc=none smtp.client-ip=209.85.208.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="HBEM+fez" Received: by mail-lj1-f202.google.com with SMTP id 38308e7fff4ca-301e3d7adceso12194951fa.3 for ; Fri, 10 Jan 2025 10:49:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736534967; x=1737139767; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=YGrdWVEeU4Ptbj/O5LlMCmxq+nbCGI8+4U1HKBQ0dew=; b=HBEM+fezq8/YZ4D9uRhnVfou9hVbtPF61TjOo/Ux8S/YyRUfRxPyXtisLDshxlrxgJ axQVHFb7GKdkLtz2iUvwqsVy0o9kOud0+kEVgZkRAc9p76GICe8u0ASPFkUVGdBdOnEn f7RVXxDV3DVI8ZRNgy0mMCglbjWiP2+6XPUqiaLn1r2TPCHfXnYxHe8xx0KvJHmCGMOr NvFMfcnUM/oM6QaD1U0fp6IwrLDVAAYrkKDCUpBo05rXmdDvgGlcq/Zps4ZMA2HGygjN vJIKLzagQXiBQUJqMT51v4XU4KLVaJXBOOH5h3XUo01scKSWVjuZ4joLQMvL8/j4m7me spug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736534967; x=1737139767; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YGrdWVEeU4Ptbj/O5LlMCmxq+nbCGI8+4U1HKBQ0dew=; b=fr6CcOi66jqwXTBsRvL+YKMHkWB66IVFaRArqm3drFKRuKESLEF0JjrFJIdAEL1rJg 8jI7amODofMNTz5MebMLZeoYbf6ovxCbVPIt/daemDLW36zlWpL1CpOczRnvR2bLITfQ soyL3GC98zf/jbplNM50ESSYgFDyFUcbmMiN12E42EiJx3SSB5A0DpxWfFcDAICh4WYH Pw2THMLczikiZO5CCDsu+5GE1yX2eaVmMGzSTjoh9tafwGCk//tpTj9RLOBFNo3YWDVK iS3w36XLxjQsNGEVZ688giDN123L9V1JfRD0JkrAV+WYjmN6nQp624ZbLlT/So/2woEP YDHg== X-Forwarded-Encrypted: i=1; AJvYcCUpchf0mZZaQktqNCUQh2PlHowAT5GtPAGXcpKrTfsW/o76nCYHZipkDIsE80z74mdQ04U=@vger.kernel.org X-Gm-Message-State: AOJu0YyE17UdPqPuIVLZp1cJZjnM5wYGg4TdTqJzq0xqVz0dQr7KPMDd GU0GHDu2iT+pCGE8HEjprHMcOiINM+GUqASTbAW5nIp9KpU5z3Wpx2h1lAmYMwI/MHo2TlbgI3F oe6UMf0IljA== X-Google-Smtp-Source: AGHT+IEgsXbDoolH8oLFMalWBy0djT9uJM60zMy+Lo0ErgHawFZLEGOSZUiE5P0d6bxzOZZfpK4Vh92Jd6K16A== X-Received: from wmrn43.prod.google.com ([2002:a05:600c:502b:b0:434:a9bd:e68c]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4f81:b0:434:f1d5:1453 with SMTP id 5b1f17b1804b1-436e2531ec8mr120237105e9.0.1736534506703; Fri, 10 Jan 2025 10:41:46 -0800 (PST) Date: Fri, 10 Jan 2025 18:40:53 +0000 In-Reply-To: <20250110-asi-rfc-v2-v2-0-8419288bc805@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250110-asi-rfc-v2-v2-0-8419288bc805@google.com> X-Mailer: b4 0.15-dev Message-ID: <20250110-asi-rfc-v2-v2-27-8419288bc805@google.com> Subject: [PATCH RFC v2 27/29] mm: asi: Add some mitigations on address space transitions From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Richard Henderson , Matt Turner , Vineet Gupta , Russell King , Catalin Marinas , Will Deacon , Guo Ren , Brian Cain , Huacai Chen , WANG Xuerui , Geert Uytterhoeven , Michal Simek , Thomas Bogendoerfer , Dinh Nguyen , Jonas Bonn , Stefan Kristiansson , Stafford Horne , "James E.J. Bottomley" , Helge Deller , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Naveen N Rao , Madhavan Srinivasan , Paul Walmsley , Palmer Dabbelt , Albert Ou , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Yoshinori Sato , Rich Felker , John Paul Adrian Glaubitz , "David S. Miller" , Andreas Larsson , Richard Weinberger , Anton Ivanov , Johannes Berg , Chris Zankel , Max Filippov , Arnd Bergmann , Andrew Morton , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Uladzislau Rezki , Christoph Hellwig , Masami Hiramatsu , Mathieu Desnoyers , Mike Rapoport , Arnaldo Carvalho de Melo , Namhyung Kim , Mark Rutland , Alexander Shishkin , Jiri Olsa , Ian Rogers , Adrian Hunter , Dennis Zhou , Tejun Heo , Christoph Lameter , Sean Christopherson , Paolo Bonzini , Ard Biesheuvel , Josh Poimboeuf , Pawan Gupta Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-alpha@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-um@lists.infradead.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, kvm@vger.kernel.org, linux-efi@vger.kernel.org, Brendan Jackman Here we ASI actually starts becoming a real exploit mitigation, On CPUs with L1TF, flush L1D when the ASI data taints say so. On all CPUs, do some general branch predictor clearing whenever the control taints say so. This policy is very much just a starting point for discussion. Primarily it's a vague gesture at the fact that there is leeway in how ASI is used: it can be used to target CPU-specific issues (as is the case for L1TF here), or it can be used as a fairly broad mitigation (asi_maybe_flush_control() mitigates several known Spectre-style attacks and very likely also some unknown ones). Signed-off-by: Brendan Jackman --- arch/x86/include/asm/nospec-branch.h | 2 ++ arch/x86/kvm/vmx/vmx.c | 1 + arch/x86/lib/l1tf.c | 2 ++ arch/x86/lib/retpoline.S | 10 ++++++++++ arch/x86/mm/asi.c | 29 +++++++++++++++++++++-------- 5 files changed, 36 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h index 96b410b1d4e841eb02f53a4691ee794ceee4ad2c..4582fb1fb42f6fd226534012d969ed13085e943a 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -614,6 +614,8 @@ static __always_inline void mds_idle_clear_cpu_buffers(void) mds_clear_cpu_buffers(); } +extern void fill_return_buffer(void); + #endif /* __ASSEMBLY__ */ #endif /* _ASM_X86_NOSPEC_BRANCH_H_ */ diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index b1a02f27b3abce0ef6ac448b66bef2c653a52eef..a532783caaea97291cd92a2e2cac617f74f76c7e 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6635,6 +6635,7 @@ int vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath) return ret; } +/* Must be reentrant, for use by vmx_post_asi_enter. */ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) { /* diff --git a/arch/x86/lib/l1tf.c b/arch/x86/lib/l1tf.c index c474f18ae331c8dfa7a029c457dd3cf75bebf808..ffe1c3d0ef43ff8f1781f2e446aed041f4ce3179 100644 --- a/arch/x86/lib/l1tf.c +++ b/arch/x86/lib/l1tf.c @@ -46,6 +46,8 @@ EXPORT_SYMBOL(l1tf_flush_setup); * - may or may not work on other CPUs. * * Don't call unless l1tf_flush_setup() has returned successfully. + * + * Must be reentrant, for use by ASI. */ noinstr void l1tf_flush(void) { diff --git a/arch/x86/lib/retpoline.S b/arch/x86/lib/retpoline.S index 391059b2c6fbc4a571f0582c7c4654147a930cef..6d126fff6bf839889086fe21464d8af07316d7e5 100644 --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -396,3 +396,13 @@ SYM_CODE_END(__x86_return_thunk) EXPORT_SYMBOL(__x86_return_thunk) #endif /* CONFIG_MITIGATION_RETHUNK */ + +.pushsection .noinstr.text, "ax" +SYM_CODE_START(fill_return_buffer) + UNWIND_HINT_FUNC + ENDBR + __FILL_RETURN_BUFFER(%_ASM_AX,RSB_CLEAR_LOOPS) + RET +SYM_CODE_END(fill_return_buffer) +__EXPORT_THUNK(fill_return_buffer) +.popsection diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index 1e9dc568e79e8686a4dbf47f765f2c2535d025ec..f10f6614b26148e5ba423d8a44f640674573ee40 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -10,6 +10,7 @@ #include #include +#include #include #include #include @@ -38,6 +39,8 @@ struct asi __asi_global_nonsensitive = { .mm = &init_mm, }; +static bool do_l1tf_flush __ro_after_init; + static inline bool asi_class_id_valid(enum asi_class_id class_id) { return class_id >= 0 && class_id < ASI_MAX_NUM_CLASSES; @@ -361,6 +364,15 @@ static int __init asi_global_init(void) asi_clone_pgd(asi_global_nonsensitive_pgd, init_mm.pgd, VMEMMAP_START + (1UL << PGDIR_SHIFT)); + if (boot_cpu_has_bug(X86_BUG_L1TF)) { + int err = l1tf_flush_setup(); + + if (err) + pr_warn("Failed to setup L1TF flushing for ASI (%pe)", ERR_PTR(err)); + else + do_l1tf_flush = true; + } + #ifdef CONFIG_PM_SLEEP register_syscore_ops(&asi_syscore_ops); #endif @@ -512,10 +524,12 @@ static __always_inline void maybe_flush_control(struct asi *next_asi) if (!taints) return; - /* - * This is where we'll do the actual dirty work of clearing uarch state. - * For now we just pretend, clear the taints. - */ + /* Clear normal indirect branch predictions, if we haven't */ + if (cpu_feature_enabled(X86_FEATURE_IBPB)) + __wrmsr(MSR_IA32_PRED_CMD, PRED_CMD_IBPB, 0); + + fill_return_buffer(); + this_cpu_and(asi_taints, ~ASI_TAINTS_CONTROL_MASK); } @@ -536,10 +550,9 @@ static __always_inline void maybe_flush_data(struct asi *next_asi) if (!taints) return; - /* - * This is where we'll do the actual dirty work of clearing uarch state. - * For now we just pretend, clear the taints. - */ + if (do_l1tf_flush) + l1tf_flush(); + this_cpu_and(asi_taints, ~ASI_TAINTS_DATA_MASK); }