From patchwork Fri Jul 12 17:00:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13731984 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1AF09C3DA45 for ; Fri, 12 Jul 2024 17:00:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9D4E06B008C; Fri, 12 Jul 2024 13:00:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 95D9E6B0092; Fri, 12 Jul 2024 13:00:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7FE2F6B0093; Fri, 12 Jul 2024 13:00:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6220C6B008C for ; Fri, 12 Jul 2024 13:00:58 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 1B98540C14 for ; Fri, 12 Jul 2024 17:00:58 +0000 (UTC) X-FDA: 82331715396.14.A36BA1E Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) by imf14.hostedemail.com (Postfix) with ESMTP id D951F10001C for ; Fri, 12 Jul 2024 17:00:54 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=dGtyBpi6; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 3RGGRZggKCIculnvxlymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--jackmanb.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=3RGGRZggKCIculnvxlymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--jackmanb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803629; a=rsa-sha256; cv=none; b=lgt4DHH46FdGtV+E0aqcvm6lAK2bpbwZ2OW1rucfEzag8MGgbdPcXb7PSR3wh2eHY37b3U 1GBPJK6dotxfR+vS1v5Kdp4W21HSwWXSoYMwi+cfaScUUFR3QNs+OTYHZv8B7KfrloCYGr VXJprhxLxZkdU/gWjuh07ZkuK16NAyI= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=dGtyBpi6; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf14.hostedemail.com: domain of 3RGGRZggKCIculnvxlymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--jackmanb.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=3RGGRZggKCIculnvxlymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--jackmanb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803629; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XYkIsXZqE6yHwQwkHX4TOc/fbGXEyWGzvfr2StuEGo8=; b=jtpRk3h9Lm4n05sF82JdAEHxFEQw2A7WPp0zIhbiDnhK4uWnkPkvFBsk5fIBoMkPtdt9XI 15tvwRTGX14dgLThdjyiuZl/6UAR26dOy8X3ZLAOx6+JxlORLkEVtqWr0QO4nT3DmYfkJ4 eLyDkY9dRtYpKtJPQPDC3Bcs+xqkTiU= Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-4266d0183feso13489825e9.0 for ; Fri, 12 Jul 2024 10:00:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803653; x=1721408453; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=XYkIsXZqE6yHwQwkHX4TOc/fbGXEyWGzvfr2StuEGo8=; b=dGtyBpi6X5fsiGjvAv2BtAHZr6GBl8gAOu6J0KRqNw8sTWrfn2Xi+ZalUYZycnUlPf bPTj4DiDIeMlwKbJrdQ4P4rrfB45FE5jJBnHQjnS1A+tN5vqjUaSPdU/JRI9iS7Apmi1 piiroVAB8jTkX80dHqefNPKEpmtB+GeqP0/DB2E5WAc3tugTnP6Fn3uEhqtFh8ohdtK+ 0jEB03+HwUSaPK8xdJ50e/uKRKnvqetM3t8UHCHxwS7Z5ciHqQ/QLtvRkyydxmnid58g IHL+13v2dEmzWO+0Z6aNjYfdRDKuGBh31ns3P2GgFsjvovfd+cNW/7S3Dw9M4FIFGbVk j/HQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803653; x=1721408453; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XYkIsXZqE6yHwQwkHX4TOc/fbGXEyWGzvfr2StuEGo8=; b=s6vWJcXFD9GUOZzk7l9KUKY0KD76nhyEygcXcwkZ7T4IdP/d9Kd5BuHBdU/bsiszdr lXfY2tFJ6p6Vht2P0ZXCalvLVf5GLywLyCyGNCd99vO6GxHOZiDGq9EQ65BNn6QJR+5g vKLPa0YeQePIxEmvbBzj5a9HDXvm22IUkNCPXQ5kV7IQxsRA6wZI+4wjVte0vXNndxxZ HqgtgOpfKRm0w4ZQkyqRH6SyE4r8JKWwTFfRMJj2KS0qss5ZXYixKGLsYkel+ZCwk/Ha AAS3pdQtAcVn3N1EAlfASvmUoS4g1bnWGtwmuG1Z0jaRsrx1l+y2NimEf+En+N6OzUtW bfNg== X-Forwarded-Encrypted: i=1; AJvYcCUds+3fO1mmLha7GKTEozXv5edS0TLFJ2JrjMoYVZf3rxhgKydrR/b2mcAp69SiH++cu4BKBdkZ9yqEdX2pSVHkySc= X-Gm-Message-State: AOJu0Yw1wUHJL961tn/PF8kC7qCm5VX+T+DBFXSNDxUIgNhDhpYUOzPI mLTVa0I5oFykCZA2vEUA2SHRUrItgs5hF5DWiBsiFmttTULSLpbZkZJZo5p5OcD60Vm7FTKrmSk J0ccOTSzCKA== X-Google-Smtp-Source: AGHT+IHv51LtvzZcsv+Q35b3qDQSC+/JaCm1f27HCjANRBhvdVtpbFdUnaeYYxG55HOkAMdxTZXXEQFhJkEhkg== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:600c:35c6:b0:424:a4ac:561b with SMTP id 5b1f17b1804b1-426708fda54mr866675e9.7.1720803652922; Fri, 12 Jul 2024 10:00:52 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:19 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-1-144b319a40d8@google.com> Subject: [PATCH 01/26] mm: asi: Make some utility functions noinstr compatible From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Rspamd-Queue-Id: D951F10001C X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: re8whbcmb3tssdj5434qqnwsa68fgc8g X-HE-Tag: 1720803654-43709 X-HE-Meta: U2FsdGVkX1/vIGIcqevuy82gIyY5GAi6VDsaL5fn5WTHgjaj4QvVnNUAnf6XvfngNWLJ9cd9C0mryNxmUQumDBzDB8/EHSHeEM2W6YkB1ftZD6JuDHvMjKsUOCQDQ33b35DPih1q9VHRETibQj4A2y6AogM1Ir9aPwAkPu3WVv03/Q+c3Zkkaiv1BZchB1liqwoE3wo4CRi7kFMLcJLSt+SGXdt3bf0wHDwl+5zot8JQAYv/75Z+Gy8k6whGhjkJvtl1YbO+wMQq8O3/q/FZHixfcXsiI6KUH9AMk/UV5sk48ipBWnadAKV7NAYaU6Zw185L7lW6mg+LVeRcGoR8HSJSDVEjs69P3oDY2UlKwr6QHBTmAfB5+nbOM/2fLGoRwHKVc3UTUPWo6qnUbJfdkCFw2fakP7VcAI3A40B/ccV1QbSNtGe9g8L0Xe8HySYJ/KxfD/fqTNB8FSB9OTJlM6M3nGeuidqlMAXqvWlzFDfTqYpGlDalNCne5Zqs6liyRD787eBDuo7eNdp6ati+XFtSYdjq5H4qFwk7crATcVZBud8q5ee0mBPyU4q2VNbpl53T4r+xr7qvcX5ZcVoRRGbbCJdlv6rVHmCEfqrFNbp/+b7JCiSuFkNYdCYgpmF+vNQLWizXrSEGyCZs084yGdT5+DwwVLjOI8AAHRkWgd9fTwg6TKGbDEdfRdldkE5GcG/vaJ/aC3rEu717LRAyJ+vUVRsU6BXiU6D1kyb4Y1E5mE7/HbLcB08wiffeAtDGRaq1WOhEXEEwvbQT0UyZykHA/HE2BZclfFn6R0UMkWatthZ/kZZzamGkYNH5P/h+/oy7Umdc7WDL5fKk5MOWSR22MyiX49wQcP4aIvB8oD+cBI6a+PDqvogH1o7kYuLNKQZxj8yMY3Jr5pUbuUFqh3M8/b8nQRsfA1uAY5GLRedBA6bb025LKySQBksCgSTSjrTbaqXkujQ3fo3snE7 Hp0KvZZg c3STW3JbwArv9Neu8IJGjXqgDSoV68jfOZ716jJ3VC3ko0IjgaFItCmkrRc/SzPlOFsRWGdnkn4Xs8rSGY8H7Lt9r3nKQ+0+uNfqqsdUv+DzYh3rLz2tnnXKmipZCloWPsj47nY/VuP3ZUuBMwV1CKCBisZKbaWi87FwiJ4VbEs0IK+B+aBrK/47Bri4VbYc9WT5GiAKJ4lM7eF2ZkPee9y1beeYdRFowjm+5Erg/jQ+mz3bJRCJz+pBlibl+bnNo6FJjA6Z8mc9obpWzn0x2BjoOuAmrBeIB7feGiUGAN9m1gOQpvDuEAGU+ZXeKJYreGePIQUuyBEIccpwJaxy7eWIJyBdEZk085RnTHj4YJ/HuByfEivFgfuJcVbTg+yYkULQ7wO0s+kTeO2tTfRMQD/KhTle/Z54+kMAGGnwTSfenFdKYhOhdn+Nmu/nrZKNNU2lGLK/YCpvVPIxnl1cnE6DbOLUBjj2XHmmv7375nmKwek/1Mbjz44SvfRPr0gdDuWcQpq6f7YQt/4FE/2qkVJ4cxHdLAIU02iHO X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Junaid Shahid Some existing utility functions would need to be called from a noinstr context in the later patches. So mark these as either noinstr or __always_inline. Signed-off-by: Junaid Shahid Signed-off-by: Brendan Jackman --- arch/x86/include/asm/processor.h | 2 +- arch/x86/include/asm/special_insns.h | 8 ++++---- arch/x86/mm/tlb.c | 8 ++++---- include/linux/compiler_types.h | 8 ++++++++ 4 files changed, 17 insertions(+), 9 deletions(-) diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index 78e51b0d6433d..dc45d622eae4e 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -206,7 +206,7 @@ void print_cpu_msr(struct cpuinfo_x86 *); /* * Friendlier CR3 helpers. */ -static inline unsigned long read_cr3_pa(void) +static __always_inline unsigned long read_cr3_pa(void) { return __read_cr3() & CR3_ADDR_MASK; } diff --git a/arch/x86/include/asm/special_insns.h b/arch/x86/include/asm/special_insns.h index 2e9fc5c400cdc..c63433dc04d34 100644 --- a/arch/x86/include/asm/special_insns.h +++ b/arch/x86/include/asm/special_insns.h @@ -42,14 +42,14 @@ static __always_inline void native_write_cr2(unsigned long val) asm volatile("mov %0,%%cr2": : "r" (val) : "memory"); } -static inline unsigned long __native_read_cr3(void) +static __always_inline unsigned long __native_read_cr3(void) { unsigned long val; asm volatile("mov %%cr3,%0\n\t" : "=r" (val) : __FORCE_ORDER); return val; } -static inline void native_write_cr3(unsigned long val) +static __always_inline void native_write_cr3(unsigned long val) { asm volatile("mov %0,%%cr3": : "r" (val) : "memory"); } @@ -153,12 +153,12 @@ static __always_inline void write_cr2(unsigned long x) * Careful! CR3 contains more than just an address. You probably want * read_cr3_pa() instead. */ -static inline unsigned long __read_cr3(void) +static __always_inline unsigned long __read_cr3(void) { return __native_read_cr3(); } -static inline void write_cr3(unsigned long x) +static __always_inline void write_cr3(unsigned long x) { native_write_cr3(x); } diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 44ac64f3a047c..6ca18ac9058b6 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -110,7 +110,7 @@ /* * Given @asid, compute kPCID */ -static inline u16 kern_pcid(u16 asid) +static inline_or_noinstr u16 kern_pcid(u16 asid) { VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE); @@ -155,9 +155,9 @@ static inline u16 user_pcid(u16 asid) return ret; } -static inline unsigned long build_cr3(pgd_t *pgd, u16 asid, unsigned long lam) +static inline_or_noinstr unsigned long build_cr3(pgd_t *pgd, u16 asid, unsigned long lam) { - unsigned long cr3 = __sme_pa(pgd) | lam; + unsigned long cr3 = __sme_pa_nodebug(pgd) | lam; if (static_cpu_has(X86_FEATURE_PCID)) { VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE); @@ -1087,7 +1087,7 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end) * It's intended to be used for code like KVM that sneakily changes CR3 * and needs to restore it. It needs to be used very carefully. */ -unsigned long __get_current_cr3_fast(void) +inline_or_noinstr unsigned long __get_current_cr3_fast(void) { unsigned long cr3 = build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h index 8f8236317d5b1..955497335832c 100644 --- a/include/linux/compiler_types.h +++ b/include/linux/compiler_types.h @@ -320,6 +320,14 @@ struct ftrace_likely_data { */ #define __cpuidle __noinstr_section(".cpuidle.text") +/* + * Can be used for functions which themselves are not strictly noinstr, but + * may be called from noinstr code. + */ +#define inline_or_noinstr \ + inline notrace __attribute((__section__(".noinstr.text"))) \ + __no_kcsan __no_sanitize_address __no_sanitize_coverage + #endif /* __KERNEL__ */ #endif /* __ASSEMBLY__ */ From patchwork Fri Jul 12 17:00:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13731985 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72389C2BD09 for ; Fri, 12 Jul 2024 17:01:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BF99B6B0093; Fri, 12 Jul 2024 13:01:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BA7516B0095; Fri, 12 Jul 2024 13:01:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9F9DF6B0096; Fri, 12 Jul 2024 13:01:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7D1E26B0093 for ; Fri, 12 Jul 2024 13:01:02 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id F1DB91A0C40 for ; Fri, 12 Jul 2024 17:01:01 +0000 (UTC) X-FDA: 82331715522.29.65C4B56 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) by imf07.hostedemail.com (Postfix) with ESMTP id CEBF840039 for ; Fri, 12 Jul 2024 17:00:58 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=iax7Z+s3; spf=pass (imf07.hostedemail.com: domain of 3SWGRZggKCIwzqs02q3rw44w1u.s421y3AD-220Bqs0.47w@flex--jackmanb.bounces.google.com designates 209.85.221.73 as permitted sender) smtp.mailfrom=3SWGRZggKCIwzqs02q3rw44w1u.s421y3AD-220Bqs0.47w@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803642; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=27EENIWQHHAHugBK1Q6SYWc3PF1x6i/tf/vuqqwaqlI=; b=kZ8QypKV+NvEMInY8EEogipCAcaiO4psfA1cY/BPZ/OJ36/ysRww5GmqRsdNC3ExajQ50Q MvDp5u0Sv2RvIbIFBDvN4PVBT1y1NJDu4VhJcQINAmLpygWm8gWyOP3GqSMDXAa6rRsEA5 mA/Htn99iCLDaMxdV/41CqYqq7WNW8k= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=iax7Z+s3; spf=pass (imf07.hostedemail.com: domain of 3SWGRZggKCIwzqs02q3rw44w1u.s421y3AD-220Bqs0.47w@flex--jackmanb.bounces.google.com designates 209.85.221.73 as permitted sender) smtp.mailfrom=3SWGRZggKCIwzqs02q3rw44w1u.s421y3AD-220Bqs0.47w@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803642; a=rsa-sha256; cv=none; b=fXFEdXNXce15/yNDyrnGoXly9mcfzZTRMm9SbK+yBYQjDWeFUoY8Wb/KzAY/QIaJ8qBRU7 LQ2bqs/S8xvUMYt8EPc5oUJHcyMbJLJAHyHg6Zz9q9utXnEAFqGGSusQDd7YQRhyY6qcdK Flm44cdu56FMj70EgesFi95G4+/TW1c= Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-3678f403afaso1232089f8f.0 for ; Fri, 12 Jul 2024 10:00:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803657; x=1721408457; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=27EENIWQHHAHugBK1Q6SYWc3PF1x6i/tf/vuqqwaqlI=; b=iax7Z+s3AX+H0Iks5llBrUkX5vTedtTpmaiwv3S3TiMLVBiRIIGX2LQTMjIJpq6uFz 3Ye42uURDOrsHU/lXOEszbPS7+zPjNXFGYexmW/5QgQDpVKjJDzqPm11cuTFXw0opA/9 A6yt0zE8mq1zPEILnquNSv9ano40yvNZiHg+Br1hb16jvXKOgbLUSnbAsyKbz65oHhMJ ewfTGgNBijTUPvrylMolprpir48YJqSUDrKlZZkq6J2ZDsz6c+bgh09nzzhsEEaW3Ps5 4AwWzSHtvprCmBXhHYxjUM4EFBuuSVSTruq8pbrcLNLGkTR8XR7Xkvx2YIxbMWuJdf/G 4gcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803657; x=1721408457; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=27EENIWQHHAHugBK1Q6SYWc3PF1x6i/tf/vuqqwaqlI=; b=J5Dn+GkBDP7VdliIOVbRWxwclqa9m5Gi9PTk7iYjVmYqfLbka0U2tXe4tcm5X091tu vhW1pSudxPiHgEw989tCugUrRj0tsFBP06bDCjzNu9tnWO/TQpijjzy73r+5GJHKDftA KmexWHN1gPMiV5tJMsB6LXitzHm44BOmWoVAIeF9iSLfb+BtmKZBoBwyL8fRVLAB0Cxx 0JCvAVYASvOGcFkBa7EEJEylN+r7dBiJxD4LJr7nJU3t3nxK8fZnDSmugRLSgfREDhKO BkGd+Ox4WC1+FkIj4MYZIvl8qZ50PJWO2pc24nHk/HfOPQhtSB5hQyj8dOlbjJNKZVWw S30A== X-Forwarded-Encrypted: i=1; AJvYcCWmuD8WOU6TSkmtWJsGZaIHEMINoIHDHDGImitmEPk7B1GBwNJH7CpPriI8UKKk4wA8Op34xtr3qdenewqi/7Rf7EY= X-Gm-Message-State: AOJu0YyUan+Oui4Xa5/oOxph8IYAzc74X7Wc43MjwNhezwdTaukLIOrV ZQPO9fW59QgpWK8P17i3bhGxVEjtHtipmIT3632vPKM5YmDDhtEidhRHh6gHIu643WRgZDC7rSS ThhGbAfjJVw== X-Google-Smtp-Source: AGHT+IGA7T3CScQphIzrB5GwbqwR/XvQ+ZJeEz+EAB7hHAYrv99hyozzic8O5c9/DkjRZkjaSXDa+Xm3xrwxfw== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:6000:14c:b0:360:727b:8b5d with SMTP id ffacd0b85a97d-367cea738dbmr25053f8f.6.1720803657248; Fri, 12 Jul 2024 10:00:57 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:20 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-2-144b319a40d8@google.com> Subject: [PATCH 02/26] x86: Create CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: CEBF840039 X-Stat-Signature: ouqencddppiasir57tgoybse3nfknotp X-HE-Tag: 1720803658-166212 X-HE-Meta: U2FsdGVkX1+XqHkTKkYd8+Eqni8fI6nwRWsyAFLp5Fy/BYxQRdAgCwU7LMK9AQwkCGNGAZzfIs9h8rxIC9/Gx1j1QafyPayOjWoWNd1T4uvS6A9/+J+e0/sGNckLMmu/5X7gYMhNRuL63Hm22XfU83UgrGbS5d36zub4Z1+z87VBOBpkmD8q2QNWrVHn2Ghi9xL62socvcgpC9l/nienQj2Zff3WTqiAH8v02zR6/TGVI4Gi+mdLs9ZPDGferiuz5VC01q3Lxth38VZWGwqZALQox9vfUGq19jYnOuSFYJteZugNrdJiPeneToX6CRH7c7U10A9faaYasFMXLgbEsJ9cjS4AsSj/LNIPDoKhwuzdZ84LO3Aeiv/PS3zpQCThf/eQ25lvnQ5zCYLJMtiYTO/FmKzue5OnT9NC6Pk8/2i++k/JIv9v2AuLHeSoKwhYdca/nXK/hIPs0hF77UU/np0mcZAlig6u20U7ATTK2ZkMr6riG2Lhfhybb2F+vz9nXcPSCiigreYRXpFkh//4gF8Am1Cp5YleiA8+MXjtjLz0CrWt3H7j/pn5CJwNXvv0P41tF4I3S1VbftAIMG+vJFAJIVDacE6c3SUHLfggeyeVlWvwqi8j2zLKWwfVk2TEOlNgI2F92gSSDVsaS63YrFPnyNTURp1d04in8uxE/0b+2W1U4p/vjAwVrVu1/aNDhRpPfq6JSjPjVfq056tztfO3+VLTYO+WuFLAB4A2PEqywtEsAbrB6dtj1kzZ4Z9tdXAb6nIacg0ufHz8qDxoVA/Mby+Whe9Hm7mcVRNJ5SaszW1EVpAamBrCGAC9ggsP4Dg6cEgHE1yV1L0MwuRXv3kcJQGmqjCh0E36GQuppTt//GcWZVv6VeDJlG/VqYSka37WNB783+kOL0Z4TaMpynBtHlqM1HgX1LDrSWH67BYNKMn9ISv+g5XpASr3xjezErqpQrj6Dfx7npJCnHZ GKThF9UO EeqMvJC1+BoadQB24C3we7qcw5KU4dpqWlOcZEA0b5PcuZNmkivockZDqZd2P+mf83a9rjS286T9lTsbli7qxDF2BAUrTWhEipV2RvmJVFLS48WrCMKq+YSLs0GqaFb+7XvSQLP2AaqmatWz4MmfPPmwBRWzZhKvGLfyXoW3oQMMmSJF15XG7jrk+V4XCJsPSR2AVz2MvIMjcLGsZ7wJbdiAuqXKxxR+Vxgl2lwjEwpCi06dOZ6Jqst0BLlfL4Cs3I1UNlkLq/oB0X+94uiS+uXjgI7Pj/hfw10hKJpM7fWvVQA7CTzdPMoGZOLiZN6nVpmV0r6e8O/RXbQewo4od/HABon3jal8s8WiTALEqsXpU6NE77tgWKMJPxNVm5Z7pV70C0ll5XyZEa7tYWtlPQ39MIahAWLodFQheTgfOL+jVicr7/0naiCu5RhlpCnztn67/Pk1GMD6fxKrXuCS1ZTbN4F4RiGM09zwaauF7BEzgLceg7nZcdb4h/U9UN39/Mb+RWn4IEaFoIisSzB4r1LQBYF1CQVcqMO5PYPhk/H076cORVXFN+F9/NqGhISyB9OLZ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently a nop config. Keeping as a separate commit for easy review of the boring bits. Later commits will use and enable this new config. This config is only added for non-UML x86_64 as other architectures do not yet have pending implementations. It also has somewhat artificial dependencies on !PARAVIRT and !KASAN which are explained in the Kconfig file. Co-developed-by: Junaid Shahid Signed-off-by: Brendan Jackman --- arch/alpha/include/asm/Kbuild | 1 + arch/arc/include/asm/Kbuild | 1 + arch/arm/include/asm/Kbuild | 1 + arch/arm64/include/asm/Kbuild | 1 + arch/csky/include/asm/Kbuild | 1 + arch/hexagon/include/asm/Kbuild | 1 + arch/loongarch/include/asm/Kbuild | 1 + arch/m68k/include/asm/Kbuild | 1 + arch/microblaze/include/asm/Kbuild | 1 + arch/mips/include/asm/Kbuild | 1 + arch/nios2/include/asm/Kbuild | 1 + arch/openrisc/include/asm/Kbuild | 1 + arch/parisc/include/asm/Kbuild | 1 + arch/powerpc/include/asm/Kbuild | 1 + arch/riscv/include/asm/Kbuild | 1 + arch/s390/include/asm/Kbuild | 1 + arch/sh/include/asm/Kbuild | 1 + arch/sparc/include/asm/Kbuild | 1 + arch/um/include/asm/Kbuild | 1 + arch/x86/Kconfig | 19 +++++++++++++++++++ arch/xtensa/include/asm/Kbuild | 1 + include/asm-generic/asi.h | 5 +++++ 22 files changed, 44 insertions(+) diff --git a/arch/alpha/include/asm/Kbuild b/arch/alpha/include/asm/Kbuild index 396caece6d6d9..ca72ce3baca13 100644 --- a/arch/alpha/include/asm/Kbuild +++ b/arch/alpha/include/asm/Kbuild @@ -5,3 +5,4 @@ generic-y += agp.h generic-y += asm-offsets.h generic-y += kvm_para.h generic-y += mcs_spinlock.h +generic-y += asi.h diff --git a/arch/arc/include/asm/Kbuild b/arch/arc/include/asm/Kbuild index 3c1afa524b9c2..60bdeffa7c31e 100644 --- a/arch/arc/include/asm/Kbuild +++ b/arch/arc/include/asm/Kbuild @@ -4,3 +4,4 @@ generic-y += kvm_para.h generic-y += mcs_spinlock.h generic-y += parport.h generic-y += user.h +generic-y += asi.h diff --git a/arch/arm/include/asm/Kbuild b/arch/arm/include/asm/Kbuild index 03657ff8fbe3d..1e2c3d8dbbd99 100644 --- a/arch/arm/include/asm/Kbuild +++ b/arch/arm/include/asm/Kbuild @@ -6,3 +6,4 @@ generic-y += parport.h generated-y += mach-types.h generated-y += unistd-nr.h +generic-y += asi.h diff --git a/arch/arm64/include/asm/Kbuild b/arch/arm64/include/asm/Kbuild index 4b6d2d52053e4..f95699a559309 100644 --- a/arch/arm64/include/asm/Kbuild +++ b/arch/arm64/include/asm/Kbuild @@ -5,6 +5,7 @@ generic-y += qrwlock.h generic-y += qspinlock.h generic-y += parport.h generic-y += user.h +generic-y += asi.h generated-y += cpucap-defs.h generated-y += sysreg-defs.h diff --git a/arch/csky/include/asm/Kbuild b/arch/csky/include/asm/Kbuild index 1117c28cb7e8a..5e49ccb571644 100644 --- a/arch/csky/include/asm/Kbuild +++ b/arch/csky/include/asm/Kbuild @@ -10,3 +10,4 @@ generic-y += qspinlock.h generic-y += parport.h generic-y += user.h generic-y += vmlinux.lds.h +generic-y += asi.h \ No newline at end of file diff --git a/arch/hexagon/include/asm/Kbuild b/arch/hexagon/include/asm/Kbuild index 3ece3c93fe086..744ffbeeb7ae4 100644 --- a/arch/hexagon/include/asm/Kbuild +++ b/arch/hexagon/include/asm/Kbuild @@ -3,3 +3,4 @@ generic-y += extable.h generic-y += iomap.h generic-y += kvm_para.h generic-y += mcs_spinlock.h +generic-y += asi.h diff --git a/arch/loongarch/include/asm/Kbuild b/arch/loongarch/include/asm/Kbuild index 2dbec7853ae86..66fcd325d6083 100644 --- a/arch/loongarch/include/asm/Kbuild +++ b/arch/loongarch/include/asm/Kbuild @@ -27,3 +27,4 @@ generic-y += param.h generic-y += posix_types.h generic-y += resource.h generic-y += kvm_para.h +generic-y += asi.h diff --git a/arch/m68k/include/asm/Kbuild b/arch/m68k/include/asm/Kbuild index 0dbf9c5c6faeb..faf0f135df4ab 100644 --- a/arch/m68k/include/asm/Kbuild +++ b/arch/m68k/include/asm/Kbuild @@ -4,3 +4,4 @@ generic-y += extable.h generic-y += kvm_para.h generic-y += mcs_spinlock.h generic-y += spinlock.h +generic-y += asi.h diff --git a/arch/microblaze/include/asm/Kbuild b/arch/microblaze/include/asm/Kbuild index a055f5dbe00a3..012e4bf83c134 100644 --- a/arch/microblaze/include/asm/Kbuild +++ b/arch/microblaze/include/asm/Kbuild @@ -8,3 +8,4 @@ generic-y += parport.h generic-y += syscalls.h generic-y += tlb.h generic-y += user.h +generic-y += asi.h diff --git a/arch/mips/include/asm/Kbuild b/arch/mips/include/asm/Kbuild index 7ba67a0d6c97b..3191699298d80 100644 --- a/arch/mips/include/asm/Kbuild +++ b/arch/mips/include/asm/Kbuild @@ -13,3 +13,4 @@ generic-y += parport.h generic-y += qrwlock.h generic-y += qspinlock.h generic-y += user.h +generic-y += asi.h diff --git a/arch/nios2/include/asm/Kbuild b/arch/nios2/include/asm/Kbuild index 7fe7437555fb4..bfdc4026c5b16 100644 --- a/arch/nios2/include/asm/Kbuild +++ b/arch/nios2/include/asm/Kbuild @@ -5,3 +5,4 @@ generic-y += kvm_para.h generic-y += mcs_spinlock.h generic-y += spinlock.h generic-y += user.h +generic-y += asi.h diff --git a/arch/openrisc/include/asm/Kbuild b/arch/openrisc/include/asm/Kbuild index c8c99b554ca4c..d137c4e08e369 100644 --- a/arch/openrisc/include/asm/Kbuild +++ b/arch/openrisc/include/asm/Kbuild @@ -7,3 +7,4 @@ generic-y += spinlock.h generic-y += qrwlock_types.h generic-y += qrwlock.h generic-y += user.h +generic-y += asi.h diff --git a/arch/parisc/include/asm/Kbuild b/arch/parisc/include/asm/Kbuild index 4fb596d94c893..3cbb4eb14712c 100644 --- a/arch/parisc/include/asm/Kbuild +++ b/arch/parisc/include/asm/Kbuild @@ -5,3 +5,4 @@ generic-y += agp.h generic-y += kvm_para.h generic-y += mcs_spinlock.h generic-y += user.h +generic-y += asi.h diff --git a/arch/powerpc/include/asm/Kbuild b/arch/powerpc/include/asm/Kbuild index 61a8d5555cd7e..103c7e2f66987 100644 --- a/arch/powerpc/include/asm/Kbuild +++ b/arch/powerpc/include/asm/Kbuild @@ -8,3 +8,4 @@ generic-y += mcs_spinlock.h generic-y += qrwlock.h generic-y += vtime.h generic-y += early_ioremap.h +generic-y += asi.h diff --git a/arch/riscv/include/asm/Kbuild b/arch/riscv/include/asm/Kbuild index 504f8b7e72d41..08c199a56731e 100644 --- a/arch/riscv/include/asm/Kbuild +++ b/arch/riscv/include/asm/Kbuild @@ -9,3 +9,4 @@ generic-y += qrwlock.h generic-y += qrwlock_types.h generic-y += user.h generic-y += vmlinux.lds.h +generic-y += asi.h diff --git a/arch/s390/include/asm/Kbuild b/arch/s390/include/asm/Kbuild index 4b904110d27cb..b5caf77e8d955 100644 --- a/arch/s390/include/asm/Kbuild +++ b/arch/s390/include/asm/Kbuild @@ -7,3 +7,4 @@ generated-y += unistd_nr.h generic-y += asm-offsets.h generic-y += kvm_types.h generic-y += mcs_spinlock.h +generic-y += asi.h diff --git a/arch/sh/include/asm/Kbuild b/arch/sh/include/asm/Kbuild index fc44d9c88b419..ea19e45158285 100644 --- a/arch/sh/include/asm/Kbuild +++ b/arch/sh/include/asm/Kbuild @@ -3,3 +3,4 @@ generated-y += syscall_table.h generic-y += kvm_para.h generic-y += mcs_spinlock.h generic-y += parport.h +generic-y += asi.h diff --git a/arch/sparc/include/asm/Kbuild b/arch/sparc/include/asm/Kbuild index 43b0ae4c2c211..cb9062c9be17f 100644 --- a/arch/sparc/include/asm/Kbuild +++ b/arch/sparc/include/asm/Kbuild @@ -4,3 +4,4 @@ generated-y += syscall_table_64.h generic-y += agp.h generic-y += kvm_para.h generic-y += mcs_spinlock.h +generic-y += asi.h diff --git a/arch/um/include/asm/Kbuild b/arch/um/include/asm/Kbuild index b2d834a29f3a9..1bcb16b09dc49 100644 --- a/arch/um/include/asm/Kbuild +++ b/arch/um/include/asm/Kbuild @@ -28,3 +28,4 @@ generic-y += trace_clock.h generic-y += kprobes.h generic-y += mm_hooks.h generic-y += vga.h +generic-y += asi.h diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 928820e61cb50..ff74aa53842ea 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -2516,6 +2516,25 @@ config MITIGATION_PAGE_TABLE_ISOLATION See Documentation/arch/x86/pti.rst for more details. +config MITIGATION_ADDRESS_SPACE_ISOLATION + bool "Allow code to run with a reduced kernel address space" + default n + depends on X86_64 && !PARAVIRT && !KASAN && !UML + help + This feature provides the ability to run some kernel code + with a reduced kernel address space. This can be used to + mitigate some speculative execution attacks. + + The !PARAVIRT dependency is only because of lack of testing; in theory + the code is written to work under paravirtualization. In practice + there are likely to be unhandled cases, in particular concerning TLB + flushes. + + The !KASAN dependency is mainly because ASI creates a secondary + direct-map region in order to implement local-nonsensitive memory. + This dependencies will later be removed with extensions to the KASAN + implementation. + config MITIGATION_RETPOLINE bool "Avoid speculative indirect branches in kernel" select OBJTOOL if HAVE_OBJTOOL diff --git a/arch/xtensa/include/asm/Kbuild b/arch/xtensa/include/asm/Kbuild index fa07c686cbcc2..07cea6902f980 100644 --- a/arch/xtensa/include/asm/Kbuild +++ b/arch/xtensa/include/asm/Kbuild @@ -8,3 +8,4 @@ generic-y += parport.h generic-y += qrwlock.h generic-y += qspinlock.h generic-y += user.h +generic-y += asi.h diff --git a/include/asm-generic/asi.h b/include/asm-generic/asi.h new file mode 100644 index 0000000000000..c4d9a5ff860a9 --- /dev/null +++ b/include/asm-generic/asi.h @@ -0,0 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ASM_GENERIC_ASI_H +#define __ASM_GENERIC_ASI_H + +#endif From patchwork Fri Jul 12 17:00:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13731986 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73F9CC3DA4D for ; Fri, 12 Jul 2024 17:01:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BFAFF6B0095; Fri, 12 Jul 2024 13:01:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B81FB6B0096; Fri, 12 Jul 2024 13:01:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9ACD56B0098; Fri, 12 Jul 2024 13:01:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 786396B0095 for ; Fri, 12 Jul 2024 13:01:04 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id ED653140C11 for ; Fri, 12 Jul 2024 17:01:03 +0000 (UTC) X-FDA: 82331715606.12.00FB551 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) by imf02.hostedemail.com (Postfix) with ESMTP id 638C68000B for ; Fri, 12 Jul 2024 17:01:01 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=CwkMlv+i; spf=pass (imf02.hostedemail.com: domain of 3S2GRZggKCI41su24s5ty66y3w.u64305CF-442Dsu2.69y@flex--jackmanb.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=3S2GRZggKCI41su24s5ty66y3w.u64305CF-442Dsu2.69y@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803627; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PMdbHr8f+bHJyqhYwYG2FeYYV8yyAj+rEymiJJJZ+c4=; b=h+RF+tPPIeL+6WJSjfV/dbML8d8rU4olAcIsW5g4QtBP14JIGtY2NnD5eY4GZJL92sLxea dAsmAYBlljPV2bOZ7yb9s88QRmUcbpygFTMPbVi6g1R3cCyDvQZ8k/MtvCWxiGwo+FqR+b LnReAluQTdQ/LQ80zx/xiO97SAxdO34= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803627; a=rsa-sha256; cv=none; b=bq2S2ZpySdizF0HOcTIUU+K06NIyUpuQU0TH+q4dltkV858F1gimQ6Wk++FpOMO+AZsGGw CE0wLmKuFypT04zg2HQuDb+4JQeKoefSXNzP5ErQPSp4+rJf03ikpedu1pnvNqQSUpxcz8 Wk94X40HdFnUy3enH82RmmgsUM/4csc= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=CwkMlv+i; spf=pass (imf02.hostedemail.com: domain of 3S2GRZggKCI41su24s5ty66y3w.u64305CF-442Dsu2.69y@flex--jackmanb.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=3S2GRZggKCI41su24s5ty66y3w.u64305CF-442Dsu2.69y@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-42668857c38so15407125e9.1 for ; Fri, 12 Jul 2024 10:01:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803660; x=1721408460; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PMdbHr8f+bHJyqhYwYG2FeYYV8yyAj+rEymiJJJZ+c4=; b=CwkMlv+iWAp3yhZh80+H4GKsEN0VT+3pxFAOV1H617k4gtMbkQCNnFsY9Xc/pjpIUp a6RA7GyAw3EMlB6TI9K7HPiG23gWbAZlL+92PvN+Ftma4GVj/RM5iSZoRnrmE9NUeL4m QXeoru8HwBnGZT29/Jp2Iaorw6EuguDErE1xXsRf4cXPE2yhetXslS65mDh4SoR4CC/U JjxNh5f/r1+7ULmG8BO5cCEySmV0W2FBiYuWMxQSI++r8gXLXcUGMeDHwXVpd0G1S+D6 S1hBolDNa+ae/Tk4pEB5qXD1+/fVPfYofuLSBkgxMooSDZb5pEHMW/sMhscznhXGw0pl FekQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803660; x=1721408460; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PMdbHr8f+bHJyqhYwYG2FeYYV8yyAj+rEymiJJJZ+c4=; b=F76g/YRuSV9IIl0GSBq/W8Ilixf336sRDls8//hVihiD5YUehS2ZwVt9DaWzQ9fO6z +iNPYZp9WUzwlhkOQj21WtN1c2WbNKWigY9nwqbYK/wMRPJYBv8CRFyZK/2GhqQaNvjd zC9b0Fn9Id0Y0CWuQYfyuJ8XEDP67cs1nni09hNWcMmKnsZUTxKdNEZHt9b3YutOh8Xy wxTWVE7+dAZL6ixpw6cKROThN+L/If6tf4VQT1hxUOTGRy2XqhcO0E4GglkEagN17lij /M5NZ3hFMPCWdYLTg/0XPYyT5vI7Yi+GVMKnz1uVL9CARIVq/OcJ2cirkZASzxy3Ryxy CPCw== X-Forwarded-Encrypted: i=1; AJvYcCWjHM9B96CrUTkhhj6CinBqB1MmlBx9lJGhldVVMO9yCgUmvLM5+ztEUCMX9nRcMkxc3dOqHIA6Gj8scscqd3osT48= X-Gm-Message-State: AOJu0YzHnrnK94hpvIWA9Nm7bid0fA5W7x1rohoub7G6iZ1dTN1AT0ra FE3NvnCc55rYg3HDvHkUYlyZ5b1FtFUOPE5n7QQgmTHoLZ0Kq9pyzQIr1fw6pjMxn5sBv2wR/UG uFGTAYEfJ6A== X-Google-Smtp-Source: AGHT+IEVF7omOk0dAyaMWn9nP5g1PTxgYRWlYkZl2KK6tCcUkjJHKogxau8pwI0HbmROqrqrZGZvZockxDjFbQ== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:600c:1c13:b0:426:6668:d59 with SMTP id 5b1f17b1804b1-426707f91dbmr1903295e9.3.1720803659878; Fri, 12 Jul 2024 10:00:59 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:21 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-3-144b319a40d8@google.com> Subject: [PATCH 03/26] mm: asi: Introduce ASI core API From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Stat-Signature: y3k9gqwfrrjndc6w8udi4ayqdkbw81f5 X-Rspamd-Queue-Id: 638C68000B X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1720803661-321106 X-HE-Meta: U2FsdGVkX187xNY+L98C0kI2sEek2UzIGReTPKGgeZwYucaKBw41xbSKUc4noL/NlP+ydZum6tTbRvZV2WU0yQEyOGGCkiWTWppvr9ubaYCkjsZreBcSTLlSZz9fSqYBNEgWIvhENt6Pjghl9cTW7whrHFlTGvjChF1rx4hBXEc4Jd+qdttAPNsi1paADgP3NfMxepBVlfj1OQDoQLCH3vx2fdL5bSm/tnmuNfoR5EIzPdvFOO3+Rr0V2is5SLCEeRWJOWNmKhy9TXPuA+6cMUxD3pri0Xnwi2uQMaAIZv67bN+zRLUJ6TZZFRvJ9WfoVhcxwuRGzWFUE2LH4myA3DI5J0X3vEtRaHcAkMDLAi08Yy4L5Dem7DxoC3TbWJPhrNNojdZqxXyVqhqRSk2jDA/2L//bMmOx7gQxPBlVxoyr1enfU2cOHFuXs1XI5v8FcyjiN7tehsXY9sYfC28Yg4TB2Lrk+Zv8OZys545vNAIGpEHD52sZQrO7hKFbignHLuopLPZcSRSCaXKR4qga2Z8TJcWMaT5i8Zas9cua7zK8E0G2DX6RCQAfc6nMDsaoxCCoYQJxwpcrKix4k03vsbm3I3tUJYRZE/qKDH74ernSEYTFQcIYu5dXWSRTZxeXPxehbRv54YsGMe8o1ZiZbXDlbrgzT5Di8jBZITFFNr01cBPrwgw5HQrsZ/h13oxuWKp8sojDpMC7slufqZfbZUs64TKjRiPJ53Ojbol1sol2qjhnUN7Puil6MDX6YE0q95/q6BQ+I94SDYMQHD5M183iGqreAj75KXQl91pC7dHNdzsxMADlCpW9VDDMln+dH33rs4jVIfIVtWmQ6cBaUIQtTTxQfhcHQYPp8wP/fct9wja1iS9urpGT14Dol2C6eEm+WjMp95sxaLnJzEI0Ws3FgAR+F4sriUdFmibhkrnV3cCmVmK70XROrYOfGIag9GfLbRMrBGE8+Izq+qS E8mNjZls aSMiUJKCKHa+RXnMnYb7m1Yq1QM8+TTIBzK7eRqTECTC0q7ES5rSVOXt80QO6CTqNKGKxLLaPwF78y9/D341mQrmj3TFpQ+HipwLfNQeLch2XiRYr1Twc0/ftFnEgJcvLGVgcexrxBhss3vQLy6bab7bLQZJ03Q7kiSLaJyD7cPxc7e7NUv1pmaamJ7VG0DpjOWB0J9xrsXjuj+gtZv1yKUD/Nv060pZQZ6iNwgzhOWSDUW6VnEfuQXOaghfySDrWSKq+R5WVWxKb1gDyDVe34bLwPVaPluHRDuKuj6zjVXYmz+gHv67LvELekgrVcZYXHqTLVsyVC4BOC5DXEZy4rrmE2PyxASjA2MQFEgSolVGjpksyONFIzwwPGdVRORQuSTC+EuS/HEoonxAYUK92k+iNnkXS7ijwckUnOvcb9p3Ww7zzIKhAHIgXztvqa5Z/szmzwuVfYp5dF+8gSq8lb/0fFtSU0pctNG84Q1AKA3oyZDkRlk9VWB6tcmCyt3xEB5b5sr3WkhKxAXEnnSofv41bO1EKkUL+w4/puVVJ2/xpVKe3mcrV17bLMF/jOU0kW+3y5oMp8/Gt2f33nBUI0uUVWKYn/OmVQxZ4CB/cLar0ewFZ0yf1YEWYRTxrZVwuOfeXHfTNT8Qa8Uk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Junaid Shahid Introduce core API for Address Space Isolation (ASI). Kernel address space isolation provides the ability to run some kernel code with a restricted kernel address space. There can be multiple classes of such restricted kernel address spaces (e.g. KPTI, KVM-PTI etc.). Each ASI class is identified by an index. The ASI class can register some hooks to be called when entering/exiting the restricted address space. Currently, there is a fixed maximum number of ASI classes supported. In addition, each process can have at most one restricted address space from each ASI class. Neither of these are inherent limitations and are merely simplifying assumptions for the time being. (The high-level ASI API was derived from the original ASI RFC by Alexandre Chartre [0]). [0]: https://lore.kernel.org/kvm/1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com Signed-off-by: Ofir Weisse Signed-off-by: Junaid Shahid Signed-off-by: Brendan Jackman --- arch/x86/include/asm/asi.h | 175 +++++++++++++++++++++++++++++ arch/x86/include/asm/processor.h | 8 ++ arch/x86/include/asm/tlbflush.h | 2 + arch/x86/mm/Makefile | 1 + arch/x86/mm/asi.c | 234 +++++++++++++++++++++++++++++++++++++++ arch/x86/mm/init.c | 3 +- arch/x86/mm/tlb.c | 2 +- include/asm-generic/asi.h | 50 +++++++++ include/linux/mm_types.h | 7 ++ kernel/fork.c | 3 + mm/init-mm.c | 4 + 11 files changed, 487 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h new file mode 100644 index 0000000000000..a052e561b2b70 --- /dev/null +++ b/arch/x86/include/asm/asi.h @@ -0,0 +1,175 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_ASI_H +#define _ASM_X86_ASI_H + +#include + +#include +#include +#include +#include + +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + +/* + * Overview of API usage by ASI clients: + * + * Setup: First call asi_init() to create a domain. At present only one domain + * can be created per mm per class, but it's safe to asi_init() this domain + * multiple times. For each asi_init() call you must call asi_destroy() AFTER + * you are certain all CPUs have exicted the restricted address space (by + * calling asi_exit()). + * + * Runtime usage: + * + * 1. Call asi_enter() to switch to the restricted address space. This can't be + * from an interrupt or exception handler and preemption must be disabled. + * + * 2. Execute untrusted code. + * + * 3. Call asi_relax() to inform the ASI subsystem that untrusted code execution + * is finished. This doesn't cause any address space change. + * + * 4. Either: + * + * a. Go back to 1. + * + * b. Call asi_exit() before returning to userspace. This immediately + * switches to the unrestricted address space. + * + * The region between 1 and 3 is called the "ASI critical section". During the + * critical section, it is a bug to access any sensitive data, and you mustn't + * sleep. + * + * The restriction on sleeping is not really a fundamental property of ASI. + * However for performance reasons it's important that the critical section is + * absolutely as short as possible. So the ability to do sleepy things like + * taking mutexes oughtn't to confer any convenience on API users. + * + * Similarly to the issue of sleeping, the need to asi_exit in case 4b is not a + * fundamental property of the system but a limitation of the current + * implementation. With further work it is possible to context switch + * from and/or to the restricted address space, and to return to userspace + * directly from the restricted address space, or _in_ it. + * + * Note that the critical section only refers to the direct execution path from + * asi_enter to asi_relax: it's fine to access sensitive data from exceptions + * and interrupt handlers that occur during that time. ASI will re-enter the + * restricted address space before returning from the outermost + * exception/interrupt. + * + * Note: ASI does not modify KPTI behaviour; when ASI and KPTI run together + * there are 2+N address spaces per task: the unrestricted kernel address space, + * the user address space, and one restricted (kernel) address space for each of + * the N ASI classes. + */ + +#define ASI_MAX_NUM_ORDER 2 +#define ASI_MAX_NUM (1 << ASI_MAX_NUM_ORDER) + +struct asi_hooks { + /* + * Both of these functions MUST be idempotent and re-entrant. They will + * be called in no particular order and with no particular symmetry wrt. + * the number of calls. They are part of the ASI critical section, so + * they must not sleep and must not access sensitive data. + */ + void (*post_asi_enter)(void); + void (*pre_asi_exit)(void); +}; + +/* + * An ASI class is a type of isolation that can be applied to a process. A + * process may have a domain for each class. + */ +struct asi_class { + struct asi_hooks ops; + const char *name; +}; + +/* + * An ASI domain (struct asi) represents a restricted address space. The + * unrestricted address space (and user address space under PTI) are not + * represented as a domain. + */ +struct asi { + pgd_t *pgd; + struct asi_class *class; + struct mm_struct *mm; + int64_t ref_count; +}; + +DECLARE_PER_CPU_ALIGNED(struct asi *, curr_asi); + +void asi_init_mm_state(struct mm_struct *mm); + +int asi_register_class(const char *name, const struct asi_hooks *ops); +void asi_unregister_class(int index); + +int asi_init(struct mm_struct *mm, int asi_index, struct asi **out_asi); +void asi_destroy(struct asi *asi); + +/* Enter an ASI domain (restricted address space) and begin the critical section. */ +void asi_enter(struct asi *asi); + +/* + * Leave the "tense" state if we are in it, i.e. end the critical section. We + * will stay relaxed until the next asi_enter. + */ +void asi_relax(void); + +/* Immediately exit the restricted address space if in it */ +void asi_exit(void); + +/* The target is the domain we'll enter when returning to process context. */ +static __always_inline struct asi *asi_get_target(struct task_struct *p) +{ + return p->thread.asi_state.target; +} + +static __always_inline void asi_set_target(struct task_struct *p, + struct asi *target) +{ + p->thread.asi_state.target = target; +} + +static __always_inline struct asi *asi_get_current(void) +{ + return this_cpu_read(curr_asi); +} + +/* Are we currently in a restricted address space? */ +static __always_inline bool asi_is_restricted(void) +{ + return (bool)asi_get_current(); +} + +/* If we exit/have exited, can we stay that way until the next asi_enter? */ +static __always_inline bool asi_is_relaxed(void) +{ + return !asi_get_target(current); +} + +/* + * Is the current task in the critical section? + * + * This is just the inverse of !asi_is_relaxed(). We have both functions in order to + * help write intuitive client code. In particular, asi_is_tense returns false + * when ASI is disabled, which is judged to make user code more obvious. + */ +static __always_inline bool asi_is_tense(void) +{ + return !asi_is_relaxed(); +} + +static __always_inline pgd_t *asi_pgd(struct asi *asi) +{ + return asi ? asi->pgd : NULL; +} + +#define INIT_MM_ASI(init_mm) \ + .asi_init_lock = __MUTEX_INITIALIZER(init_mm.asi_init_lock), + +#endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ + +#endif diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index dc45d622eae4e..a42f03ff3edca 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -5,6 +5,7 @@ #include /* Forward declaration, a strange C thing */ +struct asi; struct task_struct; struct mm_struct; struct io_bitmap; @@ -489,6 +490,13 @@ struct thread_struct { struct thread_shstk shstk; #endif +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + struct { + /* Domain to enter when returning to process context. */ + struct asi *target; + } asi_state; +#endif + /* Floating point and extended processor state */ struct fpu fpu; /* diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 25726893c6f4d..ed847567b25de 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -391,6 +391,8 @@ static inline bool huge_pmd_needs_flush(pmd_t oldpmd, pmd_t newpmd) } #define huge_pmd_needs_flush huge_pmd_needs_flush +unsigned long build_cr3(pgd_t *pgd, u16 asid, unsigned long lam); + #ifdef CONFIG_ADDRESS_MASKING static inline u64 tlbstate_lam_cr3_mask(void) { diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile index 428048e73bd2e..499233f001dc2 100644 --- a/arch/x86/mm/Makefile +++ b/arch/x86/mm/Makefile @@ -62,6 +62,7 @@ obj-$(CONFIG_NUMA_EMU) += numa_emulation.o obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS) += pkeys.o obj-$(CONFIG_RANDOMIZE_MEMORY) += kaslr.o obj-$(CONFIG_MITIGATION_PAGE_TABLE_ISOLATION) += pti.o +obj-$(CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION) += asi.o obj-$(CONFIG_X86_MEM_ENCRYPT) += mem_encrypt.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_amd.o diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c new file mode 100644 index 0000000000000..c5979d78fdbbd --- /dev/null +++ b/arch/x86/mm/asi.c @@ -0,0 +1,234 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include + +#include +#include +#include + +static struct asi_class asi_class[ASI_MAX_NUM]; +static DEFINE_SPINLOCK(asi_class_lock); + +DEFINE_PER_CPU_ALIGNED(struct asi *, curr_asi); +EXPORT_SYMBOL(curr_asi); + +static inline bool asi_class_registered(int index) +{ + return asi_class[index].name != NULL; +} + +static inline bool asi_index_valid(int index) +{ + return index >= 0 && index < ARRAY_SIZE(asi_class); +} + +int asi_register_class(const char *name, const struct asi_hooks *ops) +{ + int i; + + VM_BUG_ON(name == NULL); + + spin_lock(&asi_class_lock); + + for (i = 0; i < ARRAY_SIZE(asi_class); i++) { + if (!asi_class_registered(i)) { + asi_class[i].name = name; + if (ops != NULL) + asi_class[i].ops = *ops; + break; + } + } + + spin_unlock(&asi_class_lock); + + if (i == ARRAY_SIZE(asi_class)) + i = -ENOSPC; + + return i; +} +EXPORT_SYMBOL_GPL(asi_register_class); + +void asi_unregister_class(int index) +{ + BUG_ON(!asi_index_valid(index)); + + spin_lock(&asi_class_lock); + + WARN_ON(asi_class[index].name == NULL); + memset(&asi_class[index], 0, sizeof(struct asi_class)); + + spin_unlock(&asi_class_lock); +} +EXPORT_SYMBOL_GPL(asi_unregister_class); + + +static void __asi_destroy(struct asi *asi) +{ + lockdep_assert_held(&asi->mm->asi_init_lock); + +} + +int asi_init(struct mm_struct *mm, int asi_index, struct asi **out_asi) +{ + struct asi *asi; + int err = 0; + + *out_asi = NULL; + + BUG_ON(!asi_index_valid(asi_index)); + + asi = &mm->asi[asi_index]; + + BUG_ON(!asi_class_registered(asi_index)); + + mutex_lock(&mm->asi_init_lock); + + if (asi->ref_count++ > 0) + goto exit_unlock; /* err is 0 */ + + BUG_ON(asi->pgd != NULL); + + /* + * For now, we allocate 2 pages to avoid any potential problems with + * KPTI code. This won't be needed once KPTI is folded into the ASI + * framework. + */ + asi->pgd = (pgd_t *)__get_free_pages( + GFP_KERNEL_ACCOUNT | __GFP_ZERO, PGD_ALLOCATION_ORDER); + if (!asi->pgd) { + err = -ENOMEM; + goto exit_unlock; + } + + asi->class = &asi_class[asi_index]; + asi->mm = mm; + +exit_unlock: + if (err) + __asi_destroy(asi); + else + *out_asi = asi; + + mutex_unlock(&mm->asi_init_lock); + + return err; +} +EXPORT_SYMBOL_GPL(asi_init); + +void asi_destroy(struct asi *asi) +{ + struct mm_struct *mm; + + if (!asi) + return; + + mm = asi->mm; + /* + * We would need this mutex even if the refcount was atomic as we need + * to block concurrent asi_init calls. + */ + mutex_lock(&mm->asi_init_lock); + WARN_ON_ONCE(asi->ref_count <= 0); + if (--(asi->ref_count) == 0) { + free_pages((ulong)asi->pgd, PGD_ALLOCATION_ORDER); + memset(asi, 0, sizeof(struct asi)); + } + mutex_unlock(&mm->asi_init_lock); +} +EXPORT_SYMBOL_GPL(asi_destroy); + +static noinstr void __asi_enter(void) +{ + u64 asi_cr3; + struct asi *target = asi_get_target(current); + + /* + * This is actually false restriction, it should be fine to be + * preemptible during the critical section. But we haven't tested it. We + * will also need to disable preemption during this function itself and + * perhaps elsewhere. This false restriction shouldn't create any + * additional burden for ASI clients anyway: the critical section has + * to be as short as possible to avoid unnecessary ASI transitions so + * disabling preemption should be fine. + */ + VM_BUG_ON(preemptible()); + + if (!target || target == this_cpu_read(curr_asi)) + return; + + VM_BUG_ON(this_cpu_read(cpu_tlbstate.loaded_mm) == + LOADED_MM_SWITCHING); + + /* + * Must update curr_asi before writing CR3 to ensure an interrupting + * asi_exit sees that it may need to switch address spaces. + */ + this_cpu_write(curr_asi, target); + + asi_cr3 = build_cr3(target->pgd, + this_cpu_read(cpu_tlbstate.loaded_mm_asid), + tlbstate_lam_cr3_mask()); + write_cr3(asi_cr3); + + if (target->class->ops.post_asi_enter) + target->class->ops.post_asi_enter(); +} + +noinstr void asi_enter(struct asi *asi) +{ + VM_WARN_ON_ONCE(!asi); + + asi_set_target(current, asi); + barrier(); + + __asi_enter(); +} +EXPORT_SYMBOL_GPL(asi_enter); + +inline_or_noinstr void asi_relax(void) +{ + barrier(); + asi_set_target(current, NULL); +} +EXPORT_SYMBOL_GPL(asi_relax); + +noinstr void asi_exit(void) +{ + u64 unrestricted_cr3; + struct asi *asi; + + preempt_disable_notrace(); + + VM_BUG_ON(this_cpu_read(cpu_tlbstate.loaded_mm) == + LOADED_MM_SWITCHING); + + asi = this_cpu_read(curr_asi); + if (asi) { + if (asi->class->ops.pre_asi_exit) + asi->class->ops.pre_asi_exit(); + + unrestricted_cr3 = + build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, + this_cpu_read(cpu_tlbstate.loaded_mm_asid), + tlbstate_lam_cr3_mask()); + + write_cr3(unrestricted_cr3); + /* + * Must not update curr_asi until after CR3 write, otherwise a + * re-entrant call might not enter this branch. (This means we + * might do unnecessary CR3 writes). + */ + this_cpu_write(curr_asi, NULL); + } + + preempt_enable_notrace(); +} +EXPORT_SYMBOL_GPL(asi_exit); + +void asi_init_mm_state(struct mm_struct *mm) +{ + memset(mm->asi, 0, sizeof(mm->asi)); + mutex_init(&mm->asi_init_lock); +} diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c index 679893ea5e687..5b06d30dee672 100644 --- a/arch/x86/mm/init.c +++ b/arch/x86/mm/init.c @@ -249,7 +249,8 @@ static void __init probe_page_size_mask(void) /* By the default is everything supported: */ __default_kernel_pte_mask = __supported_pte_mask; /* Except when with PTI where the kernel is mostly non-Global: */ - if (cpu_feature_enabled(X86_FEATURE_PTI)) + if (cpu_feature_enabled(X86_FEATURE_PTI) || + IS_ENABLED(CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION)) __default_kernel_pte_mask &= ~_PAGE_GLOBAL; /* Enable 1 GB linear kernel mappings if available: */ diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 6ca18ac9058b6..9a5afeac96547 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -155,7 +155,7 @@ static inline u16 user_pcid(u16 asid) return ret; } -static inline_or_noinstr unsigned long build_cr3(pgd_t *pgd, u16 asid, unsigned long lam) +inline_or_noinstr unsigned long build_cr3(pgd_t *pgd, u16 asid, unsigned long lam) { unsigned long cr3 = __sme_pa_nodebug(pgd) | lam; diff --git a/include/asm-generic/asi.h b/include/asm-generic/asi.h index c4d9a5ff860a9..3660fc1defe87 100644 --- a/include/asm-generic/asi.h +++ b/include/asm-generic/asi.h @@ -2,4 +2,54 @@ #ifndef __ASM_GENERIC_ASI_H #define __ASM_GENERIC_ASI_H +#ifndef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + +#define ASI_MAX_NUM_ORDER 0 +#define ASI_MAX_NUM 0 + +#ifndef _ASSEMBLY_ + +struct asi_hooks {}; +struct asi {}; + +static inline +int asi_register_class(const char *name, const struct asi_hooks *ops) +{ + return 0; +} + +static inline void asi_unregister_class(int asi_index) { } + +static inline void asi_init_mm_state(struct mm_struct *mm) { } + +static inline int asi_init(struct mm_struct *mm, int asi_index, + struct asi **asi_out) +{ + return 0; +} + +static inline void asi_destroy(struct asi *asi) { } + +static inline void asi_enter(struct asi *asi) { } + +static inline void asi_relax(void) { } + +static inline bool asi_is_relaxed(void) { return true; } + +static inline bool asi_is_tense(void) { return false; } + +static inline void asi_exit(void) { } + +static inline bool asi_is_restricted(void) { return false; } + +static inline struct asi *asi_get_current(void) { return NULL; } + +static inline struct asi *asi_get_target(struct task_struct *p) { return NULL; } + +static inline pgd_t *asi_pgd(struct asi *asi) { return NULL; } + +#endif /* !_ASSEMBLY_ */ + +#endif /* !CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ + #endif diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 5240bd7bca338..226a586ebbdca 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -19,8 +19,10 @@ #include #include #include +#include #include +#include #ifndef AT_VECTOR_SIZE_ARCH #define AT_VECTOR_SIZE_ARCH 0 @@ -802,6 +804,11 @@ struct mm_struct { atomic_t membarrier_state; #endif +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + struct asi asi[ASI_MAX_NUM]; + struct mutex asi_init_lock; +#endif + /** * @mm_users: The number of users including userspace. * diff --git a/kernel/fork.c b/kernel/fork.c index aebb3e6c96dc6..a6251d11106a6 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -109,6 +109,7 @@ #include #include #include +#include #include @@ -1292,6 +1293,8 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, mm->def_flags = 0; } + asi_init_mm_state(mm); + if (mm_alloc_pgd(mm)) goto fail_nopgd; diff --git a/mm/init-mm.c b/mm/init-mm.c index 24c8093792745..e820e1c6edd48 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -12,6 +12,7 @@ #include #include #include +#include #ifndef INIT_MM_CONTEXT #define INIT_MM_CONTEXT(name) @@ -44,6 +45,9 @@ struct mm_struct init_mm = { #endif .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE, +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + INIT_MM_ASI(init_mm) +#endif INIT_MM_CONTEXT(init_mm) }; From patchwork Fri Jul 12 17:00:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13731987 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 146D5C3DA45 for ; Fri, 12 Jul 2024 17:01:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 717CD6B0096; Fri, 12 Jul 2024 13:01:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6C5FF6B0098; Fri, 12 Jul 2024 13:01:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 36AE66B0099; Fri, 12 Jul 2024 13:01:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1ACA76B0096 for ; Fri, 12 Jul 2024 13:01:06 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C2A041C1C71 for ; Fri, 12 Jul 2024 17:01:05 +0000 (UTC) X-FDA: 82331715690.01.994B108 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf03.hostedemail.com (Postfix) with ESMTP id CFA4220021 for ; Fri, 12 Jul 2024 17:01:03 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=IjgJLBE2; spf=pass (imf03.hostedemail.com: domain of 3TmGRZggKCJE4vx57v8w19916z.x97638FI-775Gvx5.9C1@flex--jackmanb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3TmGRZggKCJE4vx57v8w19916z.x97638FI-775Gvx5.9C1@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803646; a=rsa-sha256; cv=none; b=jQjKGFRf8R8mHAAaGEd5NyDa+Li/KGkuLTsQ+vlExOGFwy5n/+SL8BWYE1ucYPqsj1Wpfb x4FEkPGB8IW+36iy551lqiUNqF3Pt3/nST1DLNarpW02dccUBjyB9JNuBhWTjqTXXpAnzT 33rGfOezuTVkyCK9LnWy4wLi0M1sxgQ= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=IjgJLBE2; spf=pass (imf03.hostedemail.com: domain of 3TmGRZggKCJE4vx57v8w19916z.x97638FI-775Gvx5.9C1@flex--jackmanb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3TmGRZggKCJE4vx57v8w19916z.x97638FI-775Gvx5.9C1@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803646; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uRdo734SZY7LgE59iZWS/AzLUQw9XRryPytUBqM/u/Y=; b=LbZYbpafe2opLtWobDJXTjFzp3wzv7QgUDs90aojPnAQatrg0jephhkLyFGMyKrjrMk0s2 h00FkFrY39EfFogcfSB50ZTdN9IygR+SSwbETH7OrpWlmYhfRjyBwphTx+5Cv5/ZiyCLD4 6S+Iz7Q/iIgR/SFKVRct2RVelKH21uI= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e02b5792baaso4161794276.2 for ; Fri, 12 Jul 2024 10:01:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803663; x=1721408463; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=uRdo734SZY7LgE59iZWS/AzLUQw9XRryPytUBqM/u/Y=; b=IjgJLBE2IE0/N8ZFIAilAMJC5dbT/EzGcHVy2o62rdU4YIkmfEJJtG6YwyIFZRg5rF vgSiYcZrM7l2PzzE5QrGPzyhKy/Uq+1pUl2xPoLiSgumykHRaGx1qrAoQgDyfZ2GRuoG rt9fW5iLpVjTft/zcW3ddgJH7hV8o3/2ogWZwXJOCBL7P/xHQw6RNHDOHlsBs+pTL8M3 PAM79dAZTKmuCGRDIXPYm8hGT0ghcAJ49j0tXmcLGvqCpdAR0FIFtEt/YtvxLU4DOKx1 IsQcxz9ylUPGMDxNeHeh1/vAlfEsN8Jr468XE6jKLlafdNmWq66hkIf7/Whh6BrW7gb6 oEkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803663; x=1721408463; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=uRdo734SZY7LgE59iZWS/AzLUQw9XRryPytUBqM/u/Y=; b=mmr2aqEEw4bv/1yng+9Tb+7Mi4VOHdQnwGU0bQJ4Gs/nkh27SGhjtDLztBlFFJB0b8 9605meF3qXAonyVqtmKRQ1yBtGQ6bhSl8uPxS3G0+VIWVSI/e1tiW8AJwTBDzKpZUdy2 FbTqfpI538+lYZMKqP8qenJ3ukokLILms/57FmLTOB5+SBE7gPV7w1+XE4Jt0It2whFS Y1PI03RiU8+UNjF5otwhDES8rilezpiIgCh4facLLnUUaEtaoDYdUh78jFat3r8Cvhjx UmW8ASEwG8TFWkWDmQdHz/Nm3MIgcDkqX3hj5pCT/OihH8zDtuVxihg//CzYyGhcXaP5 an0g== X-Forwarded-Encrypted: i=1; AJvYcCWa2t+lE5FApaKVhuktB7Yw/1SmXx6y6LehLp/b9kIKZvXwd8wZ/ExuZEKvvcjHw8UJ3wOgde+r8qpvsyOqkxX1dag= X-Gm-Message-State: AOJu0Yxvh71vC1PdtWVIS+nCZ5rl5uBDBiSRLpzdoWK2dDhTAxpPWYeY G3RUfxj9WntED/aAq4VN0cmw/gT9hXnmyJQAtyW0ObKA8WCZH/DNAgGLgzthJJUwHsLDhfMmpw0 XSJa122w4IA== X-Google-Smtp-Source: AGHT+IFYA55X1keLcQMRBGeCAFNmP+QStD5pTx8+8q+l5zrMjeIlti4CTq5gIWF/C14pV/xqXPndYQF6ybLHYg== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:6902:1b12:b0:e03:b3e8:f9a1 with SMTP id 3f1490d57ef6-e041b02fabamr791937276.2.1720803662807; Fri, 12 Jul 2024 10:01:02 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:22 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-4-144b319a40d8@google.com> Subject: [PATCH 04/26] objtool: let some noinstr functions make indirect calls From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Stat-Signature: or9hbrr9rchs5q96igdh1hek735owp6b X-Rspamd-Queue-Id: CFA4220021 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1720803663-153609 X-HE-Meta: U2FsdGVkX1+sw67fO/W7cMF6ZcLeYdlYss8NYf2Sn8oi7gjBkr2l+pONq1A3l0qos/OipUAs1ufxnZNnqvCVxEeFfW6UL4ynrpjKMzIXtGvugiGcy46qxNeky7j/LBD0D+MLCQalqr4HaanK0PGscXu7H0bPlOQMsQ3t2VMF1k0rsJootHa/MEYV1tqpfSGPXWb4RFHR++WoYVmY6CfHDO5WTP9f1T9ax2EQTxA33zbgK5R2d941vQf86aj1XHotziIDWt3GTkLbOAhME3JHNKBrip420AdLTG6wuCh2RCHH9ymXa4KL5RXOCEcs3vVuOI5pr/KrTSE+G+7JoZm4puQ8XI0i+FPGX+pJBio8mdj0KVRYRZMPvIQj7bUI5kmjnKwZVqEtn9e++wvIxvr6vPnKDRflaus72HPmFQotMU/BlvNhasZisu7nfQUZat7haDhSC2krFvQUx9VERDHL/mrsixIlwnxKaqIWop9ZVEDlKqnZlY9BUxmBdTpbI7faogLTQeV+XcaMeQBY7dWt/uBZHgCCa4TCBma99juO1wGzckHXhY3KO+FWL/stTaDiS0HP3OLGoaU90BVp65GkqhwRFfLLr6kSFdKx6/CMBjLQQNcbRCjDAUu9bfnwmZO4pf+Q6Q+8q9+HoxAHiKmkA7/Dr1cjZBaZCLo8CQZIE+kebC4Ssm5FHBGGYEpCH5lSdBXez8sNPXEmxor0ULNlSAY2nLH679lVjW9rQFdlLGszhePI63qbOwkP53eQu/Jo5s3C8e787GyeSJ8qTPaR1wb5eqqKxJP3GlfVYIkAPlwsgX2Z7Xu0PFOKmVibQtmsKbJvfTd6PzZeKo7ut0Vy0Rc5oHGAA2aPIeeaRW3REO2+IeSjXwSwZU+h+DCRNaJUGj5hV3Jq1S9kj1w4guo9XMLMuasxzi/fFUXgn9XlAuM1BRNDYxOZI7nq0ZQbNI2yDWFtwxGQg77QsI/RX6D xaqdSsna CwiZvn3qIs3yPLbTbFBnUQv9zsbYxhMKMOdAEXjb9mVKl0uAe5k1hXpSsZlZ7lPKOTVHx1Yd8uk4TirIEqXFknzvRsIr76lcUdjjN000ajQIL9qE/wX3uMLa7Y9g1oVQBrjtsjMykAwR55AYAMjUzhsx1nC4cfHBk73ectd5oqIf+EaKs0VhHj13rhiNFzjU8W4i/QI9xgOEupSNp8cTwfkKYn1Dx30KhLn1silba8aDmG1ylyxmEal6u8NboiEeu29VBsVOVO/HdiDYi01J4YYRYt6V8Mz/8030s14scqxtPuQ678cVvCEtx5iNB4CcRaL7SqRRRBrcaxFLe/MXTAg7rRJZ4+IfxsX4mx82mIS2C1A6Tp8KtkPKcD0Q8YodUdsNSkPoFMExTjJuqwlcIZVyheiPYU9dy/1CzXTSDUOjoR+47n6v61zU6ZRwQXqeCE/V3I72Tbk+2l5ZEABzGMV1UtR3sUGnCM2X1X2TuXo5jAvZXsXajh7pbFQvt89wENjuzzw09uSdwOEVT96Fpo+P/BOzr1t6n6qPE X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: As described in the comment, some noinstr functions really need to make indirect calls. Those functions could be rewritten to use static calls, but that just shifts the "assume it's instrumented" to "assume the indirect call is fine" which seems like just moving the problem around. Instead here's a way to selectively mark functions that are known to be in the danger zone, and we'll just have to be careful with them. Signed-off-by: Brendan Jackman --- tools/objtool/check.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/tools/objtool/check.c b/tools/objtool/check.c index 0a33d9195b7a9..a760a858d8aa3 100644 --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -3425,6 +3425,17 @@ static bool pv_call_dest(struct objtool_file *file, struct instruction *insn) return file->pv_ops[idx].clean; } +static inline bool allow_noinstr_indirect_call(struct symbol *func) +{ + /* + * These functions are noinstr but make indirect calls. The programmer + * solemnly promises that the target functions are noinstr too, but they + * might be in modules so we can't prove it here. + */ + return (!strcmp(func->name, "asi_exit") || + !strcmp(func->name, "__asi_enter")); +} + static inline bool noinstr_call_dest(struct objtool_file *file, struct instruction *insn, struct symbol *func) @@ -3437,6 +3448,9 @@ static inline bool noinstr_call_dest(struct objtool_file *file, if (file->pv_ops) return pv_call_dest(file, insn); + if (allow_noinstr_indirect_call(insn->sym)) + return true; + return false; } From patchwork Fri Jul 12 17:00:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13731988 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BE14C2BD09 for ; Fri, 12 Jul 2024 17:01:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A57236B009A; Fri, 12 Jul 2024 13:01:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D5426B009B; Fri, 12 Jul 2024 13:01:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 73B5F6B009C; Fri, 12 Jul 2024 13:01:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 4D83E6B009A for ; Fri, 12 Jul 2024 13:01:10 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id F03D6140BE7 for ; Fri, 12 Jul 2024 17:01:09 +0000 (UTC) X-FDA: 82331715858.26.5C4AFCC Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) by imf15.hostedemail.com (Postfix) with ESMTP id 850A4A0041 for ; Fri, 12 Jul 2024 17:01:07 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=tGm+1uMo; spf=pass (imf15.hostedemail.com: domain of 3UWGRZggKCJQ7y08AyBz4CC492.0CA96BIL-AA8Jy08.CF4@flex--jackmanb.bounces.google.com designates 209.85.221.74 as permitted sender) smtp.mailfrom=3UWGRZggKCJQ7y08AyBz4CC492.0CA96BIL-AA8Jy08.CF4@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803633; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=A+jkQL2bLgnd1dA3asyFJ85yKqjjPIZgiPP4ZLm28ds=; b=FUBEUGo4Au0vqnrbH8Axf+hpqti3Aa4uTG84OEbQISIi+CKVDXs55jcOkpYoImitR4vGAn dKOByZ0rkcv0whFhah3xNV84E2B3UJA5xE7N/QKLTs8GnHgj9TnGyjndOg86ijxvgwH/6n H6GfTiIRYrxYWhCxkKQuRYtX3u30xgA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803633; a=rsa-sha256; cv=none; b=d/LxYiR+CnDhcjoCBBkMRPQgFbfLKkwyzqtEOEzAy7s98HAfYk9kbyH9noRfG96KrZQJL3 kzD4p3l8i3z6ApDgT47ISs0YEk/jJNmC+rvkAowGNzTJjIyBFlGzOMF+Um7BpAbTEkEBW9 hISKHDAH9/JJoN1sPF9RDZwyPiqaV+0= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=tGm+1uMo; spf=pass (imf15.hostedemail.com: domain of 3UWGRZggKCJQ7y08AyBz4CC492.0CA96BIL-AA8Jy08.CF4@flex--jackmanb.bounces.google.com designates 209.85.221.74 as permitted sender) smtp.mailfrom=3UWGRZggKCJQ7y08AyBz4CC492.0CA96BIL-AA8Jy08.CF4@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-36785e6c1e6so411012f8f.3 for ; Fri, 12 Jul 2024 10:01:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803666; x=1721408466; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=A+jkQL2bLgnd1dA3asyFJ85yKqjjPIZgiPP4ZLm28ds=; b=tGm+1uMojNvOSR9u6g2CEu74tVHYlqZdyiYKgd4ooUfnCIgTdTN04FvGsHxahmCR7J 9G4ERkrAF6vCYMtVD8KF3r5HczIq8muAloXu6XadDILhQd4Ms9TWvvistJ4g0gLbuR/3 f1k2gmB198BbozZLraGYXndM+IZHoHGObRtcw+WAzxeWp1jSVWEGXDsg+FUqhrP51ugU DW49udhtmwJrNBuXT6jFmesnfy8xHMv5izNeb7Gb3b1EYYpx8Pph1syEo8nSfvgAZPRH XegKKk7JrmeXsXrX2QFYoJVbYGpWMSKRnFOSeUs1nGTcQHHWogTpmnTEdGT8Q/K14u8+ KjaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803666; x=1721408466; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=A+jkQL2bLgnd1dA3asyFJ85yKqjjPIZgiPP4ZLm28ds=; b=VjRJYJUGI+78mjbJ+p7WvLRcB28o2kwxJQmC+sFFemLKpb0Anrzb7h4OPl4nRyUebX jb7shCdEEETq3/G5hMhBxQTVjJcAXsy2cXFuKdOhTiurZWQWdpcVhduOclIqGMMY8hyu mUSJKWtoeIBYz0/u9nhqoJZYru/a8f/1DZFSAX5q016e0lp1cBoq5YPpDewIMR4ATd4v wf4m1kDEFpHGuzu54zZVagRc0Dd0P/+mSBwH9+KzAa4tu+S2V/REt6JM6ipL8xTKV06c opcNDiRRIS7yRmh6MVXzJhk6R9/La4c8fXxTgHVEoPgcBEziW853dQKeXJDP8oiITgo6 ruCA== X-Forwarded-Encrypted: i=1; AJvYcCVbZbSvoP3estjWEP76i/mstV8e5XsxSNMhCtkZo7aW4pi6+m+AsxgiiqXEIXdrcJ8NmMI3D5TDph4JPuqNyu43FIE= X-Gm-Message-State: AOJu0Yw65IU6O8+6ouTaw7IAfa22/TXbGosxf30E000/z53VlywNmPTt SouhNOvFQ3fQbyb8gUKIND+IPyrFuXKtA/UbsZvegTifRFsSSY51fpX/0CfIFTrvOmIt5DLMepC kA8ISLDeo0g== X-Google-Smtp-Source: AGHT+IHocE6UCQJXGH4jW5S1f1i1qM0rmNw/QM4OyMtcRUe5WUau/3oluaHQD6zUlZ5KVAkubOggLQncuc63kw== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:adf:ea0c:0:b0:367:8147:25c5 with SMTP id ffacd0b85a97d-367cea8ef32mr16488f8f.8.1720803665595; Fri, 12 Jul 2024 10:01:05 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:23 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-5-144b319a40d8@google.com> Subject: [PATCH 05/26] mm: asi: Add infrastructure for boot-time enablement From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Stat-Signature: npwiqtuwjs7aa1dxd5kzuprhpmpi3xwf X-Rspamd-Queue-Id: 850A4A0041 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1720803667-946999 X-HE-Meta: U2FsdGVkX1+BmKkbqP1jq6FzE3vfbo7r72bCbdr/u8ZTq114s3bAKp87Hjq2B2VjJobWPifm2KKiZFjwQYGvsLteY3x601NPxZgW3Ml0+BfjQaIsUdrVjUZassgy0M3LhvZ/iqWAmzQaf3e+WCWxUHdJETZFRyeKQ5X7A22FlSkd1gVa0aiYPRYfwyld8PE+euuEUddQ6xaHQucJvrOKeQ2cJVposuENKr5aJRxE3v5jHdgQbKXQk5Y9GImVMmvYQ6w38S4o272rzYm7WIDW7B8PD8N7/ev+8UJn1RLFqVM3/HvnKlm353z4BvATNBvR6WXzTigWUU+g9UmkrRXLNPjR4UAcT9CN1Uo49z+h5pqGrJJLnJ6TJEpAUBd2ugrGDL9qT1KSA6UsvS9+VkD66zYLgRtPwO6cAQqIO9e3lAEJHnj9GAR96widiJCIloJ4P74NCCxZNfex/uS/ED5lo22+rzkWUpKOOvI+4umpmk990loScfdaOThiDTW6m5SRo1ZXC7QlPqjYUXNGp94zm/46nbr5J7//SWN5cg3KB+VvQoLeDUGtaQari9ZfHecPLVAl1OustWNH78xmpwC7oML/Cphd1ncNkg7ua0qPmXaf7AidXYZCs3wcHDC3SA6MDrcf/vEcaRs0YOSgrWPocCE5Nda0q93bCmkioxUAUUV4SqShAsY6LOPsf9XEtnPfdevHIZ2nHUj78M0YwTQeCaS3Clk/uyEeqQY0unX2RWwM+qSYl1ggIiuv9Q7YUmoJubtxkYuIhaUChmvcxuFX2kSKjGUcyuTV7XZkWn6ShM/lO5/NdzMd5F4Sfki5gYDuR2Ubz9dgGAhADqRjHh8mZI57dsaQYDOqjBd39yss5ZysiLlTyCNMdXWa0MkZs224vhwonQ3oWVJX2r0agF7h9R0jaWIFDb91IXMbXNktDa0D8YwtIuQBpV2Y+R9kGv1bNZdWuC7wKRtzE/P/yxr SvD6Airi VFMDfYMVDwQiW22u/mOn8W+cJ4CK3VCg1a5kbtH2KGFXupWe2gAR9yVAQseek/ReJVT0yVqisniDRXTnhaVAF2iSFu0UyaL3X11CHgrVMINoTdsl0qlW60pahBo0lu/w1L5WSSjry+SnYKKREQzs6zAsS5UP7KzwfcOz/Yl7IFK05fbS7OSb2Qv/ZTAwYf/gPiZUQrF29d+gK1HFGcO74BSPOaUS19aC7brVhdKMGvAbTjs6tC7ob0yyZBCFw+h48sojq5NymbY1c9kegEMyRtcDk6yESohUYmJ+HrU18du9DeUNbNAbPD+Nyzqdf+6VH6T14Hs/tP2L3VvMXFmxtbMRk23ftrSVvJrEYJ24mQqfJsj6yaX0jy7vbyEa36mZZ0SlXWmYM8LWeMZKfuOyVIGVmFcR02fBlFejRxonVEOTVLhc8URJpkcSbF6nnxl3yJcK+1thaS4OIp+W1ySJiDnalkKMBLcoiXYTuzxtL96zmP97C79l8vgeOxAATSdDWXiGtrHkzdUQV6cvuvSIyyzcbgsOdBU66lBYY X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add a boot time parameter to control the newly added X86_FEATURE_ASI. "asi=on" or "asi=off" can be used in the kernel command line to enable or disable ASI at boot time. If not specified, ASI enablement depends on CONFIG_ADDRESS_SPACE_ISOLATION_DEFAULT_ON, which is off by default. asi_check_boottime_disable() is modeled after pti_check_boottime_disable(). The boot parameter is currently ignored until ASI is fully functional. Once we have a set of ASI features checked in that we have actually tested, we will stop ignoring the flag. But for now let's just add the infrastructure so we can implement the usage code. Co-developed-by: Junaid Shahid Co-developed-by: Yosry Ahmed Signed-off-by: Brendan Jackman --- arch/x86/Kconfig | 8 +++++ arch/x86/include/asm/asi.h | 20 +++++++++-- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/disabled-features.h | 8 ++++- arch/x86/mm/asi.c | 61 +++++++++++++++++++++++++++----- arch/x86/mm/init.c | 4 ++- include/asm-generic/asi.h | 4 +++ 7 files changed, 92 insertions(+), 14 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index ff74aa53842e..7f21de55d6ac 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -2535,6 +2535,14 @@ config MITIGATION_ADDRESS_SPACE_ISOLATION This dependencies will later be removed with extensions to the KASAN implementation. +config ADDRESS_SPACE_ISOLATION_DEFAULT_ON + bool "Enable address space isolation by default" + default n + depends on ADDRESS_SPACE_ISOLATION + help + If selected, ASI is enabled by default at boot if the asi=on or + asi=off are not specified. + config MITIGATION_RETPOLINE bool "Avoid speculative indirect branches in kernel" select OBJTOOL if HAVE_OBJTOOL diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index a052e561b2b7..04ba2ec7fd28 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -6,6 +6,7 @@ #include #include +#include #include #include @@ -64,6 +65,9 @@ * the N ASI classes. */ +/* Try to avoid this outside of hot code (see comment on _static_cpu_has). */ +#define static_asi_enabled() cpu_feature_enabled(X86_FEATURE_ASI) + #define ASI_MAX_NUM_ORDER 2 #define ASI_MAX_NUM (1 << ASI_MAX_NUM_ORDER) @@ -101,6 +105,8 @@ struct asi { DECLARE_PER_CPU_ALIGNED(struct asi *, curr_asi); +void asi_check_boottime_disable(void); + void asi_init_mm_state(struct mm_struct *mm); int asi_register_class(const char *name, const struct asi_hooks *ops); @@ -124,7 +130,9 @@ void asi_exit(void); /* The target is the domain we'll enter when returning to process context. */ static __always_inline struct asi *asi_get_target(struct task_struct *p) { - return p->thread.asi_state.target; + return static_asi_enabled() + ? p->thread.asi_state.target + : NULL; } static __always_inline void asi_set_target(struct task_struct *p, @@ -135,7 +143,9 @@ static __always_inline void asi_set_target(struct task_struct *p, static __always_inline struct asi *asi_get_current(void) { - return this_cpu_read(curr_asi); + return static_asi_enabled() + ? this_cpu_read(curr_asi) + : NULL; } /* Are we currently in a restricted address space? */ @@ -144,7 +154,11 @@ static __always_inline bool asi_is_restricted(void) return (bool)asi_get_current(); } -/* If we exit/have exited, can we stay that way until the next asi_enter? */ +/* + * If we exit/have exited, can we stay that way until the next asi_enter? + * + * When ASI is disabled, this returns true. + */ static __always_inline bool asi_is_relaxed(void) { return !asi_get_target(current); diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 3c7434329661..a6b213c7df44 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -470,6 +470,7 @@ #define X86_FEATURE_BHI_CTRL (21*32+ 2) /* "" BHI_DIS_S HW control available */ #define X86_FEATURE_CLEAR_BHB_HW (21*32+ 3) /* "" BHI_DIS_S HW control enabled */ #define X86_FEATURE_CLEAR_BHB_LOOP_ON_VMEXIT (21*32+ 4) /* "" Clear branch history at vmexit using SW loop */ +#define X86_FEATURE_ASI (21*32+5) /* Kernel Address Space Isolation */ /* * BUG word(s) diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index c492bdc97b05..c7964ed4fef8 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -50,6 +50,12 @@ # define DISABLE_PTI (1 << (X86_FEATURE_PTI & 31)) #endif +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +# define DISABLE_ASI 0 +#else +# define DISABLE_ASI (1 << (X86_FEATURE_ASI & 31)) +#endif + #ifdef CONFIG_MITIGATION_RETPOLINE # define DISABLE_RETPOLINE 0 #else @@ -154,7 +160,7 @@ #define DISABLED_MASK17 0 #define DISABLED_MASK18 (DISABLE_IBT) #define DISABLED_MASK19 (DISABLE_SEV_SNP) -#define DISABLED_MASK20 0 +#define DISABLED_MASK20 (DISABLE_ASI) #define DISABLED_MASK21 0 #define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 22) diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index c5979d78fdbb..21207a3e8b17 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -4,7 +4,9 @@ #include #include +#include #include +#include #include #include @@ -28,6 +30,9 @@ int asi_register_class(const char *name, const struct asi_hooks *ops) { int i; + if (!boot_cpu_has(X86_FEATURE_ASI)) + return 0; + VM_BUG_ON(name == NULL); spin_lock(&asi_class_lock); @@ -52,6 +57,9 @@ EXPORT_SYMBOL_GPL(asi_register_class); void asi_unregister_class(int index) { + if (!boot_cpu_has(X86_FEATURE_ASI)) + return; + BUG_ON(!asi_index_valid(index)); spin_lock(&asi_class_lock); @@ -63,11 +71,36 @@ void asi_unregister_class(int index) } EXPORT_SYMBOL_GPL(asi_unregister_class); +void __init asi_check_boottime_disable(void) +{ + bool enabled = IS_ENABLED(CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION_DEFAULT_ON); + char arg[4]; + int ret; + + ret = cmdline_find_option(boot_command_line, "asi", arg, sizeof(arg)); + if (ret == 3 && !strncmp(arg, "off", 3)) { + enabled = false; + pr_info("ASI disabled through kernel command line.\n"); + } else if (ret == 2 && !strncmp(arg, "on", 2)) { + enabled = true; + pr_info("Ignoring asi=on param while ASI implementation is incomplete.\n"); + } else { + pr_info("ASI %s by default.\n", + enabled ? "enabled" : "disabled"); + } + + if (enabled) + pr_info("ASI enablement ignored due to incomplete implementation.\n"); +} static void __asi_destroy(struct asi *asi) { - lockdep_assert_held(&asi->mm->asi_init_lock); + WARN_ON_ONCE(asi->ref_count <= 0); + if (--(asi->ref_count) > 0) + return; + free_pages((ulong)asi->pgd, PGD_ALLOCATION_ORDER); + memset(asi, 0, sizeof(struct asi)); } int asi_init(struct mm_struct *mm, int asi_index, struct asi **out_asi) @@ -77,6 +110,9 @@ int asi_init(struct mm_struct *mm, int asi_index, struct asi **out_asi) *out_asi = NULL; + if (!boot_cpu_has(X86_FEATURE_ASI)) + return 0; + BUG_ON(!asi_index_valid(asi_index)); asi = &mm->asi[asi_index]; @@ -121,7 +157,7 @@ void asi_destroy(struct asi *asi) { struct mm_struct *mm; - if (!asi) + if (!boot_cpu_has(X86_FEATURE_ASI) || !asi) return; mm = asi->mm; @@ -130,11 +166,7 @@ void asi_destroy(struct asi *asi) * to block concurrent asi_init calls. */ mutex_lock(&mm->asi_init_lock); - WARN_ON_ONCE(asi->ref_count <= 0); - if (--(asi->ref_count) == 0) { - free_pages((ulong)asi->pgd, PGD_ALLOCATION_ORDER); - memset(asi, 0, sizeof(struct asi)); - } + __asi_destroy(asi); mutex_unlock(&mm->asi_init_lock); } EXPORT_SYMBOL_GPL(asi_destroy); @@ -178,6 +210,9 @@ static noinstr void __asi_enter(void) noinstr void asi_enter(struct asi *asi) { + if (!static_asi_enabled()) + return; + VM_WARN_ON_ONCE(!asi); asi_set_target(current, asi); @@ -189,8 +224,10 @@ EXPORT_SYMBOL_GPL(asi_enter); inline_or_noinstr void asi_relax(void) { - barrier(); - asi_set_target(current, NULL); + if (static_asi_enabled()) { + barrier(); + asi_set_target(current, NULL); + } } EXPORT_SYMBOL_GPL(asi_relax); @@ -199,6 +236,9 @@ noinstr void asi_exit(void) u64 unrestricted_cr3; struct asi *asi; + if (!static_asi_enabled()) + return; + preempt_disable_notrace(); VM_BUG_ON(this_cpu_read(cpu_tlbstate.loaded_mm) == @@ -229,6 +269,9 @@ EXPORT_SYMBOL_GPL(asi_exit); void asi_init_mm_state(struct mm_struct *mm) { + if (!boot_cpu_has(X86_FEATURE_ASI)) + return; + memset(mm->asi, 0, sizeof(mm->asi)); mutex_init(&mm->asi_init_lock); } diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c index 5b06d30dee67..e2a29f6779d9 100644 --- a/arch/x86/mm/init.c +++ b/arch/x86/mm/init.c @@ -27,6 +27,7 @@ #include #include #include +#include /* * We need to define the tracepoints somewhere, and tlb.c @@ -250,7 +251,7 @@ static void __init probe_page_size_mask(void) __default_kernel_pte_mask = __supported_pte_mask; /* Except when with PTI where the kernel is mostly non-Global: */ if (cpu_feature_enabled(X86_FEATURE_PTI) || - IS_ENABLED(CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION)) + cpu_feature_enabled(X86_FEATURE_ASI)) __default_kernel_pte_mask &= ~_PAGE_GLOBAL; /* Enable 1 GB linear kernel mappings if available: */ @@ -757,6 +758,7 @@ void __init init_mem_mapping(void) unsigned long end; pti_check_boottime_disable(); + asi_check_boottime_disable(); probe_page_size_mask(); setup_pcid(); diff --git a/include/asm-generic/asi.h b/include/asm-generic/asi.h index 3660fc1defe8..d0a451f9d0b7 100644 --- a/include/asm-generic/asi.h +++ b/include/asm-generic/asi.h @@ -48,6 +48,10 @@ static inline struct asi *asi_get_target(struct task_struct *p) { return NULL; } static inline pgd_t *asi_pgd(struct asi *asi) { return NULL; } +#define static_asi_enabled() false + +static inline void asi_check_boottime_disable(void) { } + #endif /* !_ASSEMBLY_ */ #endif /* !CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ From patchwork Fri Jul 12 17:00:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13731989 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11DC0C2BD09 for ; Fri, 12 Jul 2024 17:01:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5B88F6B009B; Fri, 12 Jul 2024 13:01:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4CC9C6B009C; Fri, 12 Jul 2024 13:01:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2CFBE6B009D; Fri, 12 Jul 2024 13:01:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 0388A6B009B for ; Fri, 12 Jul 2024 13:01:12 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 7A5A9140BFA for ; Fri, 12 Jul 2024 17:01:12 +0000 (UTC) X-FDA: 82331715984.25.9D30483 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) by imf11.hostedemail.com (Postfix) with ESMTP id 6A63240034 for ; Fri, 12 Jul 2024 17:01:10 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=cBht38FA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf11.hostedemail.com: domain of 3VGGRZggKCJcA13BD1E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--jackmanb.bounces.google.com designates 209.85.221.73 as permitted sender) smtp.mailfrom=3VGGRZggKCJcA13BD1E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--jackmanb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803645; a=rsa-sha256; cv=none; b=49J69kB5fjNlfQ2DjB8gvYouqXWhu9hhvnPBqtlTmxY+MuhZagpzEnJSrVGP/eIrApxcEs 4pupx9gGkXgdQ5SZIuBqpCYAwQzI85eoCV2MKzALF6jTDA7D3r7p1x0WOYtAigOWbY2WUq ShPnb1AUaR6xt/hVQbfnhJ7+a4dSYr4= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=cBht38FA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf11.hostedemail.com: domain of 3VGGRZggKCJcA13BD1E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--jackmanb.bounces.google.com designates 209.85.221.73 as permitted sender) smtp.mailfrom=3VGGRZggKCJcA13BD1E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--jackmanb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803645; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qrAVAbVK+nVNUznV+52/ScTYsIGKlGAcY8a4rOPWm6Y=; b=eQOP2xIVg6mdxNoTWRfbf+EicMwKLDHcGziujtp+UQTno0oIl8qaAIZ7Y8BGRSZ3w8ZJx1 TdJRAQ/L6/d5OafO4Wwa5NebP6SgEeUV7GCvl5EXpQOgLiIbzMF/n39uvCfetFkQo5Q5Vx T0v7WhWF1Xs5nkco3zwk/JX7FDz658g= Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-36785e6c1e6so411038f8f.3 for ; Fri, 12 Jul 2024 10:01:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803669; x=1721408469; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=qrAVAbVK+nVNUznV+52/ScTYsIGKlGAcY8a4rOPWm6Y=; b=cBht38FA581BiWP4Ilk3xQudUL3a5rD2+SxJtAfSiPWQQZpuIqb5dJKcExb7+9XIA2 DorgaQD93EIiiHrZ/+rqp2JnqpjPNP6RNTaA6D47rsGALFdcKHJ8ir3m3YnBQ6+cHjz2 wNflVM20nmd7tXlY+u9EPvdF6lnOnu0qjvq8U/EMHugtT8hcY+JgjOt1gR1ImNKldv3i LhpNpkG5SKwsTB0FeC00fs2hkyiKXitcIcoq2oDJ3RMAyrFZiOJjE5ONZt3xIQ+MB4Yo JBfXcvNq7kO6q/5AxT3SUw2jKvNk1A5VfDYXSMSn/hNaRRoDod/xCo22vO955VlQbR3W 2+Ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803669; x=1721408469; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qrAVAbVK+nVNUznV+52/ScTYsIGKlGAcY8a4rOPWm6Y=; b=PxXJC0GIcTWQQIxduGNUhEBA4Uxi0t7ULoRuUEUYBJm+NHUlMwv3yzc7uUB09Y39Ih gIFPAAHrvhfIzF39hl5adJJ7adA6yqe03jUsPJJGX9VxNz0z3U7zYjr4DJMtqmB1jEF2 fptOo7ufOaSO1yZevNpN05gkGNv1QZv59cVst/7Rop7/j8GUorwmc6maoTmbl6TWtKCO 2khk56EAiMJKscYVLbGbRt30tTcBoSK5anoreHFR2b+OSHZJScWdin06kotCNX8c6BTw TrFesaf/Axt8yy8n2S4wBGZi72+xgaC5ydjirFSXhDvC0CwBn83V5U96fQZsYQdz9zTY q3UQ== X-Forwarded-Encrypted: i=1; AJvYcCX7WchaT1sqyJmSgUGwPrD4p7B2zpa7foYcjNw+iy9NqcKB5ETshVzX8VAvH91EuddT87detrSBG+6184Coy9oqcDo= X-Gm-Message-State: AOJu0YzHnYNYHA4L113ekt5mXIKnur29O0JZLblgiO5Zm2pRzNc3hTOU DjPPQdKZsKmPmHaRLBu/yjn/CLB1FejN6f6QN2Z8a5mbKM6CPlN4v3qj0cQj2yhTnV/U8SG7RGa YjohDLh1HUA== X-Google-Smtp-Source: AGHT+IF3kvDD67OIZ2BMm13dMbd7zSJf6yY9zVib7vZL6oHM8iz/nEJhVOpklDYifHh85TWN8ozTDRdKs7e20g== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:6000:187:b0:367:60dc:9ec4 with SMTP id ffacd0b85a97d-367cea68085mr18401f8f.6.1720803668718; Fri, 12 Jul 2024 10:01:08 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:24 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-6-144b319a40d8@google.com> Subject: [PATCH 06/26] mm: asi: ASI support in interrupts/exceptions From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Rspamd-Queue-Id: 6A63240034 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: nn45y83a7srx3j4us1t59ftbgdkr8dpj X-HE-Tag: 1720803670-252161 X-HE-Meta: U2FsdGVkX18pGIAYXNNVlIUjRXHnUNaBuWFEdWGaWpViszZYQDI6zESFJ5ioe39vRSE90w/JAeYv6FVQ09Du1BQQXWM8A1qaINSOvHWMZ0nXQyQmhzSV80oWm3xVoWzWbWtYdvugTDOwwPmTUAMS1oEEFoGqPcTl0LjFAetHARbLaxnwDrsdiFPu1ZjtfTrO6aEcllOT4lCn488VwXw7fzk/upwxE3Se9iXdSXzPXWoIR/VkaTOBwEzfowlpJeE1VWyuujGFZbY2kTittorxW0vdlMXpWHK1+WI3+sieG7+dkpmBZhsKigmo1++hDyP7/lUDInc/DEwN/iXYh3HQ0S9VIB48aKo+s3lp3odcKlgC3M5bmYl+/MVAEsNcUtQDLbYDf5qYW/K1/inzvb3hcYlqWUZzcacU75vXrgIBgpt4f7jVY5XntblQkMTrvKnq45Uu6qDmCTHhhEgNqdPNIGmVjF0tcCWVI3UTEA8B1qpTo2w2TzdBcIsQKBvyDo+wJcWOtybB1zZb6FrurGfygGA7pMMDZ4nDC7iTyXxsZ5UqCBDVYPv6CgJRcZ3BiPWE/T6Ii5bpWcvDqpB1aVOPABP+JRZ/fEF5xTyq+FJiNeLEgzupjOXxtO6GmNIMFtFX9IajNIcxLedQ436bNZu/QVrsE6ylb75zUov8WhS1ZQ5I96ub9ljThKJwJJt1KhtfgIWdP6ekwqh2xkStC44ssOUnY92oEA2/CPvn4Mwmk/8g31Ta/evXOVK9Pie1hJL/wI8nxzQ9OMqLpCMpmEkAXfxPCgJdMHSWi8J0KaEQ/6c6FEK0ry0/CX14KlEod0hDCzr7GZk0U09DO3V9ZBhW1tWv0xsm/gPAdJF7z19VwdKT6t5s9Pg2hy4BsQUY76f2fnSprEsF5Q+uFg3eQL3CI6yrO+OZYBcFPa6oGf0O+ypIJdhVIhzHh3pCV42E3N+iR855ROMKJRla6yJcAcY yPWx2NRl vsAJ1BQtRPWqP/RYPQUJgIuf1DDV0Z3SvhmQjilSZyE+w4Q+ySchXJYPpkJjRv+SGXaeBHvju5Pn21c5jbGu3kD/iCZPO38GxI3YWj2sWemw5pVA9mrds3AYbsQM0jzIMheEuFkXBlNBGT2mSzJHYBUzT/+cF/9Ek/HTTNY+xAV5pp/7KWKzJ1O3bbdr9BW/5QkdoXxhh1Qxjgc9ad8EiK+RexoCeJIJLkAixpuJplNh3Ft+Bbg50qswkM0gXsAW73PGun4v1ZPZZl8toahnXUgv+q2CN0QEnesJhk5e7HPzNL+1n5S7Xu4H3Vz6u3VA6mDbC1BN7APzppFbof6AOzJNqG/IcfsrGYyB+Ilz2PzeACvJ38YbEu410OYZ6uNZHxX4Xe77a3ngi2SyrcYxaAE/rW0jGKURKc0HeYIYIsOARJ541A/FQRJ+ZTGFoXfnsEhXbgKe7QjEVAR2kWYBPWRBOAB8184dDgsFCiKM/ooLVVZFqyxXmcj20sMYdi3+azWkm7pSgL1dvEDidExpjbLJe9Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add support for potentially switching address spaces from within interrupts/exceptions/NMIs etc. An interrupt does not automatically switch to the unrestricted address space. It can switch if needed to access some memory not available in the restricted address space, using the normal asi_exit call. On return from the outermost interrupt, if the target address space was the restricted address space (e.g. we were in the critical code path between ASI Enter and VM Enter), the restricted address space will be automatically restored. Otherwise, execution will continue in the unrestricted address space until the next explicit ASI Enter. In order to keep track of when to restore the restricted address space, an interrupt/exception nesting depth counter is maintained per-task. An alternative implementation without needing this counter is also possible, but the counter unlocks an additional nice-to-have benefit by allowing detection of whether or not we are currently executing inside an exception context, which would be useful in a later patch. Note that for KVM on SVM, this is not actually necessary as NMIs are in fact maskable via CLGI. It's not clear to me if VMX has something equivalent but we will need this infrastructure in place for userspace support anyway. Signed-off-by: Junaid Shahid Signed-off-by: Brendan Jackman --- arch/x86/include/asm/asi.h | 68 ++++++++++++++++++++++++++++++++++++++-- arch/x86/include/asm/idtentry.h | 50 ++++++++++++++++++++++++----- arch/x86/include/asm/processor.h | 5 +++ arch/x86/kernel/process.c | 2 ++ arch/x86/kernel/traps.c | 22 +++++++++++++ arch/x86/mm/asi.c | 5 ++- include/asm-generic/asi.h | 10 ++++++ 7 files changed, 151 insertions(+), 11 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index 04ba2ec7fd28..df34a8c0560b 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -127,6 +127,11 @@ void asi_relax(void); /* Immediately exit the restricted address space if in it */ void asi_exit(void); +static inline void asi_init_thread_state(struct thread_struct *thread) +{ + thread->asi_state.intr_nest_depth = 0; +} + /* The target is the domain we'll enter when returning to process context. */ static __always_inline struct asi *asi_get_target(struct task_struct *p) { @@ -167,9 +172,10 @@ static __always_inline bool asi_is_relaxed(void) /* * Is the current task in the critical section? * - * This is just the inverse of !asi_is_relaxed(). We have both functions in order to - * help write intuitive client code. In particular, asi_is_tense returns false - * when ASI is disabled, which is judged to make user code more obvious. + * This is just the inverse of !asi_is_relaxed(). We have both functions in + * order to help write intuitive client code. In particular, asi_is_tense + * returns false when ASI is disabled, which is judged to make user code more + * obvious. */ static __always_inline bool asi_is_tense(void) { @@ -181,6 +187,62 @@ static __always_inline pgd_t *asi_pgd(struct asi *asi) return asi ? asi->pgd : NULL; } +static __always_inline void asi_intr_enter(void) +{ + if (static_asi_enabled() && asi_is_tense()) { + current->thread.asi_state.intr_nest_depth++; + barrier(); + } +} + +void __asi_enter(void); + +static __always_inline void asi_intr_exit(void) +{ + if (static_asi_enabled() && asi_is_tense()) { + /* + * If an access to sensitive memory got reordered after the + * decrement, the #PF handler for that access would see a value + * of 0 for the counter and re-__asi_enter before returning to + * the faulting access, triggering an infinite PF loop. + */ + barrier(); + + if (--current->thread.asi_state.intr_nest_depth == 0) { + /* + * If the decrement got reordered after __asi_enter, an + * interrupt that came between __asi_enter and the + * decrement would always see a nonzero value for the + * counter so it wouldn't call __asi_enter again and we + * would return to process context in the wrong address + * space. + */ + barrier(); + __asi_enter(); + } + } +} + +/* + * Returns the nesting depth of interrupts/exceptions that have interrupted the + * ongoing critical section. If the current task is not in a critical section + * this is 0. + */ +static __always_inline int asi_intr_nest_depth(void) +{ + return current->thread.asi_state.intr_nest_depth; +} + +/* + * Remember that interrupts/exception don't count as the critical section. If + * you want to know if the current task is in the critical section use + * asi_is_tense(). + */ +static __always_inline bool asi_in_critical_section(void) +{ + return asi_is_tense() && !asi_intr_nest_depth(); +} + #define INIT_MM_ASI(init_mm) \ .asi_init_lock = __MUTEX_INITIALIZER(init_mm.asi_init_lock), diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentry.h index 749c7411d2f1..446aed5ebe18 100644 --- a/arch/x86/include/asm/idtentry.h +++ b/arch/x86/include/asm/idtentry.h @@ -12,6 +12,7 @@ #include #include +#include typedef void (*idtentry_t)(struct pt_regs *regs); @@ -55,12 +56,15 @@ static __always_inline void __##func(struct pt_regs *regs); \ \ __visible noinstr void func(struct pt_regs *regs) \ { \ - irqentry_state_t state = irqentry_enter(regs); \ + irqentry_state_t state; \ \ + asi_intr_enter(); \ + state = irqentry_enter(regs); \ instrumentation_begin(); \ __##func (regs); \ instrumentation_end(); \ irqentry_exit(regs, state); \ + asi_intr_exit(); \ } \ \ static __always_inline void __##func(struct pt_regs *regs) @@ -102,12 +106,15 @@ static __always_inline void __##func(struct pt_regs *regs, \ __visible noinstr void func(struct pt_regs *regs, \ unsigned long error_code) \ { \ - irqentry_state_t state = irqentry_enter(regs); \ + irqentry_state_t state; \ \ + asi_intr_enter(); \ + state = irqentry_enter(regs); \ instrumentation_begin(); \ __##func (regs, error_code); \ instrumentation_end(); \ irqentry_exit(regs, state); \ + asi_intr_exit(); \ } \ \ static __always_inline void __##func(struct pt_regs *regs, \ @@ -139,7 +146,16 @@ static __always_inline void __##func(struct pt_regs *regs, \ * is required before the enter/exit() helpers are invoked. */ #define DEFINE_IDTENTRY_RAW(func) \ -__visible noinstr void func(struct pt_regs *regs) +static __always_inline void __##func(struct pt_regs *regs); \ + \ +__visible noinstr void func(struct pt_regs *regs) \ +{ \ + asi_intr_enter(); \ + __##func (regs); \ + asi_intr_exit(); \ +} \ + \ +static __always_inline void __##func(struct pt_regs *regs) /** * DEFINE_FREDENTRY_RAW - Emit code for raw FRED entry points @@ -178,7 +194,18 @@ noinstr void fred_##func(struct pt_regs *regs) * is required before the enter/exit() helpers are invoked. */ #define DEFINE_IDTENTRY_RAW_ERRORCODE(func) \ -__visible noinstr void func(struct pt_regs *regs, unsigned long error_code) +static __always_inline void __##func(struct pt_regs *regs, \ + unsigned long error_code); \ + \ +__visible noinstr void func(struct pt_regs *regs, unsigned long error_code)\ +{ \ + asi_intr_enter(); \ + __##func (regs, error_code); \ + asi_intr_exit(); \ +} \ + \ +static __always_inline void __##func(struct pt_regs *regs, \ + unsigned long error_code) /** * DECLARE_IDTENTRY_IRQ - Declare functions for device interrupt IDT entry @@ -209,14 +236,17 @@ static void __##func(struct pt_regs *regs, u32 vector); \ __visible noinstr void func(struct pt_regs *regs, \ unsigned long error_code) \ { \ - irqentry_state_t state = irqentry_enter(regs); \ + irqentry_state_t state; \ u32 vector = (u32)(u8)error_code; \ \ + asi_intr_enter(); \ + state = irqentry_enter(regs); \ instrumentation_begin(); \ kvm_set_cpu_l1tf_flush_l1d(); \ run_irq_on_irqstack_cond(__##func, regs, vector); \ instrumentation_end(); \ irqentry_exit(regs, state); \ + asi_intr_exit(); \ } \ \ static noinline void __##func(struct pt_regs *regs, u32 vector) @@ -256,12 +286,15 @@ static __always_inline void instr_##func(struct pt_regs *regs) \ \ __visible noinstr void func(struct pt_regs *regs) \ { \ - irqentry_state_t state = irqentry_enter(regs); \ + irqentry_state_t state; \ \ + asi_intr_enter(); \ + state = irqentry_enter(regs); \ instrumentation_begin(); \ instr_##func (regs); \ instrumentation_end(); \ irqentry_exit(regs, state); \ + asi_intr_exit(); \ } \ \ void fred_##func(struct pt_regs *regs) \ @@ -295,12 +328,15 @@ static __always_inline void instr_##func(struct pt_regs *regs) \ \ __visible noinstr void func(struct pt_regs *regs) \ { \ - irqentry_state_t state = irqentry_enter(regs); \ + irqentry_state_t state; \ \ + asi_intr_enter(); \ + state = irqentry_enter(regs); \ instrumentation_begin(); \ instr_##func (regs); \ instrumentation_end(); \ irqentry_exit(regs, state); \ + asi_intr_exit(); \ } \ \ void fred_##func(struct pt_regs *regs) \ diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index a42f03ff3edc..5b10b3c09b6a 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -494,6 +494,11 @@ struct thread_struct { struct { /* Domain to enter when returning to process context. */ struct asi *target; + /* + * The depth of interrupt/exceptions interrupting an ASI + * critical section + */ + int intr_nest_depth; } asi_state; #endif diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index b8441147eb5e..ca2391079e59 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -96,6 +96,8 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) #ifdef CONFIG_VM86 dst->thread.vm86 = NULL; #endif + asi_init_thread_state(&dst->thread); + /* Drop the copied pointer to current's fpstate */ dst->thread.fpu.fpstate = NULL; diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index 4fa0b17e5043..ca0d0b9fe955 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -64,6 +64,7 @@ #include #include #include +#include #include #include #include @@ -414,6 +415,27 @@ DEFINE_IDTENTRY_DF(exc_double_fault) } #endif + /* + * Do an asi_exit() only here because a #DF usually indicates + * the system is in a really bad state, and we don't want to + * cause any additional issue that would prevent us from + * printing a correct stack trace. + * + * The additional issues are not related to a possible triple + * fault, which can only occurs if a fault is encountered while + * invoking this handler, but here we are already executing it. + * Instead, an ASI-induced #PF here could potentially end up + * getting another #DF. For example, if there was some issue in + * invoking the #PF handler. The handler for the second #DF + * could then again cause an ASI-induced #PF leading back to the + * same recursion. + * + * This is not needed in the espfix64 case above, since that + * code is about turning a #DF into a #GP which is okay to + * handle in the restricted domain. That's also why we don't + * asi_exit() in the #GP handler. + */ + asi_exit(); irqentry_nmi_enter(regs); instrumentation_begin(); notify_die(DIE_TRAP, str, regs, error_code, X86_TRAP_DF, SIGSEGV); diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index 21207a3e8b17..2cd8e93a4415 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -171,7 +171,7 @@ void asi_destroy(struct asi *asi) } EXPORT_SYMBOL_GPL(asi_destroy); -static noinstr void __asi_enter(void) +noinstr void __asi_enter(void) { u64 asi_cr3; struct asi *target = asi_get_target(current); @@ -186,6 +186,7 @@ static noinstr void __asi_enter(void) * disabling preemption should be fine. */ VM_BUG_ON(preemptible()); + VM_BUG_ON(current->thread.asi_state.intr_nest_depth != 0); if (!target || target == this_cpu_read(curr_asi)) return; @@ -246,6 +247,8 @@ noinstr void asi_exit(void) asi = this_cpu_read(curr_asi); if (asi) { + WARN_ON_ONCE(asi_in_critical_section()); + if (asi->class->ops.pre_asi_exit) asi->class->ops.pre_asi_exit(); diff --git a/include/asm-generic/asi.h b/include/asm-generic/asi.h index d0a451f9d0b7..fa0bbf899a09 100644 --- a/include/asm-generic/asi.h +++ b/include/asm-generic/asi.h @@ -38,6 +38,8 @@ static inline bool asi_is_relaxed(void) { return true; } static inline bool asi_is_tense(void) { return false; } +static inline bool asi_in_critical_section(void) { return false; } + static inline void asi_exit(void) { } static inline bool asi_is_restricted(void) { return false; } @@ -48,6 +50,14 @@ static inline struct asi *asi_get_target(struct task_struct *p) { return NULL; } static inline pgd_t *asi_pgd(struct asi *asi) { return NULL; } +static inline void asi_init_thread_state(struct thread_struct *thread) { } + +static inline void asi_intr_enter(void) { } + +static inline int asi_intr_nest_depth(void) { return 0; } + +static inline void asi_intr_exit(void) { } + #define static_asi_enabled() false static inline void asi_check_boottime_disable(void) { } From patchwork Fri Jul 12 17:00:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13731990 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB267C3DA45 for ; Fri, 12 Jul 2024 17:01:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C47776B009C; Fri, 12 Jul 2024 13:01:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BCF436B009D; Fri, 12 Jul 2024 13:01:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9D5186B009E; Fri, 12 Jul 2024 13:01:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 795796B009C for ; Fri, 12 Jul 2024 13:01:15 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 2CACA120C23 for ; Fri, 12 Jul 2024 17:01:15 +0000 (UTC) X-FDA: 82331716110.27.ECA0A06 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) by imf13.hostedemail.com (Postfix) with ESMTP id 3343120020 for ; Fri, 12 Jul 2024 17:01:12 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=hiYWvOhG; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3V2GRZggKCJoD46EG4H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--jackmanb.bounces.google.com designates 209.85.221.73 as permitted sender) smtp.mailfrom=3V2GRZggKCJoD46EG4H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--jackmanb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803647; a=rsa-sha256; cv=none; b=ncELCe8dmYY7dtndz+6N7xiWfcozbFjlUpRTbJV//+qrSAGQ2ZTGXpoLk/1w3TJz9k4WdW +6dLpgbiegZ1+jxeHZsN4uLVyMdmOOeGRELppBan/cBD55W9EIjaq96BHJU1odt5QKyhB8 H1EvjDkth1pwK2c/UaUWi1GkSPITmiM= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=hiYWvOhG; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3V2GRZggKCJoD46EG4H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--jackmanb.bounces.google.com designates 209.85.221.73 as permitted sender) smtp.mailfrom=3V2GRZggKCJoD46EG4H5AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--jackmanb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803647; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HhvK+VsrCDIF0zdCgKI0fd0HHuRdBdA78SeRsMH4q04=; b=g+awT0zFiAOagHomrzmKUIdDvBe4tTx953j3DZBYk9l1JKrkHle5dAqNppaGQYQvdwFm8U VOBZeftvHB59G3VZJKw3nEqr0FXm8Bfc625itN7fpzbDbgnGEnqB4+ntJosK40twKlw1GC O/P3omMzociv8Y/3WT+8cFlGXwSa+3k= Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-36805bfd95aso514276f8f.3 for ; Fri, 12 Jul 2024 10:01:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803672; x=1721408472; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=HhvK+VsrCDIF0zdCgKI0fd0HHuRdBdA78SeRsMH4q04=; b=hiYWvOhGTyqtcFSobfiv6gR9FV6NyolRqBJF92fIDJJaMn30AU9FC+kjCQeU8UlHdm oHNrV8XSSYjHltzxE2+kbNKbaIk+kNXrK7lQzi9NFIHLczVuLQwFnTgNf7dHPuB8N0/D QWnHFACFNxeJ1YaoE6eOLeY25S0Fx5hli/dr7IRYw5i9bw+BYTuivFBABSWlKL4oXtDE Z2SeafDi6hmTeN40Xnm5XpxM5og5P/xpsCoLyTfuivA/LvY+I6FvpFHvvkrXgfSaDhsT /KoDstpOJuzpndTvBjhpcLDyNYALhu7Fqm7YPIzyyHeIZ0m1fguyU9peK507tM/FoYhC pL+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803672; x=1721408472; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=HhvK+VsrCDIF0zdCgKI0fd0HHuRdBdA78SeRsMH4q04=; b=rEffcPeihLVzI1SBT7J/33CneEQdrI7OjQbVq6Gy/7KgOwQoTHxjRUq3nG9HbpmSex 4FwqJL80XVKwuCz2ST1vTkWRAEhWYFqJZO/VWM+qDeQntugh5YJ1TrGfO5bFlIpbAPBB TydsfOvASYC6hlahePWj/iiLyy8PHMMW+4HVxy0MPEulgqGYW8kW2wBEQCMdFLtoDiXE Fh7xxUAAu2bZIKr6hsR6inc0tvYaYvUm4b7NI8qyEFyvNsjWeKAVNh+hx/+idbJNUCrO lnX0cOHPNxHxuxzEQ8itqdyAu9BdiJDFgCaVTnjv84CNf7f3tAkjHhvI9IfJLypNjCU6 wC/g== X-Forwarded-Encrypted: i=1; AJvYcCWiCXTVjsB36AQF50i3WnScWteaAvoaLFdD+Cb/dzN4JgGcL93X5vnP9LN/EzZJwQR9Q3QOnpey534NBT1fugq2rUE= X-Gm-Message-State: AOJu0YyBaZx6xGWBlQCrOCIE7NDsCEw6BOZdArl4dBC945yQ4CX8LytR P9otGdrOS5ULkEyY59JuwiW5hpsV+Dclmzsbgl5oXOippAMMfZg0N1lgc8bhbsKEKSavf9trLf/ iDv24qcu+OA== X-Google-Smtp-Source: AGHT+IFAPE6Abc5gQhah4PjzgMavZ5HwflkhPdDu4LbVKDPqBKd9rbMZcjfutEvjASrTYbwEe+XhPDdeyFCaUQ== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:adf:e988:0:b0:367:9db7:d6d4 with SMTP id ffacd0b85a97d-367ceac39c9mr19900f8f.9.1720803671416; Fri, 12 Jul 2024 10:01:11 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:25 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-7-144b319a40d8@google.com> Subject: [PATCH 07/26] mm: asi: Switch to unrestricted address space before a context switch From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Rspamd-Queue-Id: 3343120020 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: j1577uaeu9pjbffircbane6rg4q7iqe5 X-HE-Tag: 1720803672-63084 X-HE-Meta: U2FsdGVkX1+bGdXLk8LhrKBXOM6VlPRRSDg81YjYvhADmWo19nFTuwN6T4yKdAtQPAxAz8flgefgX0JciNsW3tgYpLFXc29Fed2kqNRAc26tqSh6ZlebKTl6wFIG+B7AOUumID8NMDaIHNweERqOTwH4YtOjPjh131gD0NACl3UHiWRMLAcuzT4zggcEZd/MEeZZ5WSI1H2mgafqUg5b+BDz0UZlU9wLmTb6X+DllnXmMjTg0cGNdBuYhSJNuu6fLW+wBshrnqGD8A5EYDrFpU+qaLW98Cw0QepJNxHSh1o13vzNnSFQmy7qp9g0H39K7FmZ5geY+sGeIqvBs4xST2VdoRaMjGHmAc+m96p6NwLEhiywPAItX1wFS6OZ665P7lFAPTXrak+6r3IJTeAPTMho4b6EUvJpyYknCKyDeUrUzSrWS4MNx0TS714I7qrZQoq7YyDaojx/GgXE4bNHx/4VmTwXwNRwImFlHsfZrlz8gP0dwKCEP8UYnyzY2DH2YfLh0r8PYVOBaa1auIj0VM0oTdq1kzwp5wRxyVzVpzOfIH1eINJc2R+7BzJI9V66Pjw1O2H7/A7miqlcQP9RIAXkHDTdNPrcpwC8YWIpcINanAExvcaKNWeYBbAGeYEYZObagnPSHkzcSsRGQgkGpeZRtt1qkn/w5EzZGtfVbygf9KOLAZppwTve6OMYXBofFpuKYUt+PSSG36VbY+f0koQ8QtaI8Zlx+GGP2LB8gsHsSqn1YVt5vAXrGoWWpUH9StNtat5DkV1FojA1jji3F06foSF23JMbz30ZpJ5rjoPT3Xgq1VdYjdJG/BAVm/TITO84atCS+CRTsvWQSv6uWTOUDLEgA80TL9Gn7uyvi3N+2sgIR1NlitMDRro5dvgDPRr/RsUZeeNMFTEtgn3Kis6SSjqABx8d7QVjeB6zFQ8yaUmv6vvf0EsXzlT4OvfER2tFewfuZXvaEYFBYXG gakPyIK9 v1ckFsKFIaaCNIu1yfgWUh3Unf6pHpziAjXkpEXUCNQ9ApOv70FGKuXE+oPXFjfQTdHycE2srKTFIMbxjC4dSK1Bs+pj0NyC0r50ryKEAdgEn5UTLf+4JBQt3ioSKq6FL+ubFZFM/pz7aSerEkK0D61Ww1gvc3z4Kc6bu72BI7vDCRhU86Dv9evmHddHEZIcg7+1AKDcT20yLN4iaZLSER2hthAoN42b7ZAnKE8fg/PtCg/8SqgKK86HTsJCfFmrEa281fgo6ekvAXc4jKKXCb8+CNNXlzo6ZDyIrrLiRBICVdzd79J1JPYUlMJdLRBRVE/LEr7XWm2WiQ1IUDIGUZnmxRutu5xshNJZfpT8EU2gdOA4LogrGCroSsVu9dZM9CYYNM49wLtXZab9n4zQHRBUIMS7poZSoAOd1/hEi13oPfQnTuZIfLXY+GH+ypPwIl37QWpsIcqVECrTuWf+CdP86YwmmKnRNEAjIqeTxs/N0Nzz6R1IS+cEFIlolh/+93VG14uSDx1mERMAx4KQV6LzNMZhKY3zC5i8kvIK6Pn4crOSiRjj6VmNL9g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Junaid Shahid To keep things simpler for the time being, we disallow context switches within the restricted address space. In the future, we would be able to relax this limitation for the case of context switches to different threads within the same process (or to the idle thread and back). Signed-off-by: Junaid Shahid Signed-off-by: Brendan Jackman --- kernel/sched/core.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 7019a40457a6..e65ac22e5a28 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -77,6 +77,7 @@ #include #include #include +#include #define CREATE_TRACE_POINTS #include @@ -5353,6 +5354,8 @@ static __always_inline struct rq * context_switch(struct rq *rq, struct task_struct *prev, struct task_struct *next, struct rq_flags *rf) { + asi_exit(); + prepare_task_switch(rq, prev, next); /* From patchwork Fri Jul 12 17:00:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13731991 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDF0BC3DA45 for ; Fri, 12 Jul 2024 17:01:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9B3CA6B009E; Fri, 12 Jul 2024 13:01:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9104F6B009F; Fri, 12 Jul 2024 13:01:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7176D6B00A0; Fri, 12 Jul 2024 13:01:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4B51B6B009E for ; Fri, 12 Jul 2024 13:01:18 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D53EF80BEC for ; Fri, 12 Jul 2024 17:01:17 +0000 (UTC) X-FDA: 82331716194.05.E4BF2A0 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf24.hostedemail.com (Postfix) with ESMTP id BC77D180038 for ; Fri, 12 Jul 2024 17:01:15 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=jRZHhJcD; spf=pass (imf24.hostedemail.com: domain of 3WmGRZggKCJ0G79HJ7K8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--jackmanb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3WmGRZggKCJ0G79HJ7K8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803649; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IU+0QFZ2g2My69wAuJPkncJIr3fDXfYxT6Ca6fNVeD8=; b=Un1Sq7lrJmHx/nf5vTPdk+2T34HhXTJ6t29Z08NfsovFHg+8FiuXtMw/KxkfuhMewxWyfY EwvwVoK1jbAvHrzK2XwdGvK7Oi43GBJl6yF/xF6xTf5eGntBShPzKj+iV4+HH5JnG87hcd q8l4+w3/yxQoUXUvUdd8G9MeD47Fy9s= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=jRZHhJcD; spf=pass (imf24.hostedemail.com: domain of 3WmGRZggKCJ0G79HJ7K8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--jackmanb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3WmGRZggKCJ0G79HJ7K8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803649; a=rsa-sha256; cv=none; b=IekJ4a9gYUpPDvderVu3DWpOBja3Vv2/0l/pbjlq9ODuKRCMMm/AnZcAPnlkySLK0bByZl 9GgOn6P3LNQ3RM73GQz3lRZQPBLDZjj+p98dqjn5tEuQ9qsxggI1gMEnWP24ut6LBXpYk+ tmSyHssZRweulQXnKvzaOZMY8bRftvc= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-64b9800b377so41502337b3.0 for ; Fri, 12 Jul 2024 10:01:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803675; x=1721408475; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IU+0QFZ2g2My69wAuJPkncJIr3fDXfYxT6Ca6fNVeD8=; b=jRZHhJcDi4xULJ3u96PgeJdikVCG/kV1q/faDHOg0ydeP2xND2+d+KJxeXnih+ge3+ VVJC+i2nAADbzFLrjcY5DI1+/LXHAtCZQXpGvogQb3PsbKjL40Mu/7rcAbGGrmQId0DJ Hv/Nm4g/cVIYdvdY900Cxw7/dVn2STSaAX7O7VCI2Brqb//JEwZt/OrWOOrRkh5f0C51 88x6QOCQ3B+b/x4N+iGFq4qY0OqKlPwgTzD7obB1XUKOByw9Ekm3mdUhAZBxinwjuHBr HxwHa2cQ/nosdbnM/Jxyxp5GTyeYvbjO0uWcrVgHk7CMla0xywHmhsxkLgU/t2Bhvly9 qJMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803675; x=1721408475; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IU+0QFZ2g2My69wAuJPkncJIr3fDXfYxT6Ca6fNVeD8=; b=aMor4bWQnTLrRIac0iW+Yy/hX1ugt+blg47UzE0+Crx5zYdxj7uveTkBLxBdT60Ewm 2ECuLnizfWI0YGOfftvkY6z5FoZjn8O5dt1X7UuVRGjp7bxOqUwVPKdoqPAs8pPobQVp 3RsuE7fP6hp0KPB4vgQB6ON5UFAW6KK6RRdfbdTUa5NwArNSuo1kuQjWKyKBr1BXCaRj bktIc26Yjvn4Hs48ot0Q4zeeSotRXEjca10Gj3Pc3v2Piv+Pdwe+/7DIUg6xX1O+jNcO 7KiBUsok/3ai9rjZbzSlydblAAssXOSE4C+GEOAzc+ARmQoj/S38G1Xhxy5MF83U6X9W XLpw== X-Forwarded-Encrypted: i=1; AJvYcCWO28c/coZ+i+f4w/FMjN1N/JudnHVzdH9ef0iWH9NSsIKw0J/0hPV1NnJbqdxX7TNtrCvo5H8NtwofczcN4oM/qvo= X-Gm-Message-State: AOJu0YyABITeV0XeEH1yXBpJiP2v8bCS4b8uTsUSbRC3K8vlRqAjmvg/ HInfh/ZGSqgwjPKBz2C+XzAcujKHxK1+91ngbEgjSo2L6u1HEQYMO9kvDf5QJR7mSXiuh4Djeq1 ImwhvhJ8Blw== X-Google-Smtp-Source: AGHT+IH090AhnVnRaBqnnakPZ4/V5j0nTOpLdhxmJg4znTZivgjhg0KK45kar3tFbJqN9PhohJA5PmDXL/BkUA== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a25:945:0:b0:e05:7113:920c with SMTP id 3f1490d57ef6-e058a6cfb8dmr11142276.6.1720803674689; Fri, 12 Jul 2024 10:01:14 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:26 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-8-144b319a40d8@google.com> Subject: [PATCH 08/26] mm: asi: Use separate PCIDs for restricted address spaces From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Stat-Signature: xgien3jca1wj9gf8cbrmncm8f9hdsbg4 X-Rspam-User: X-Rspamd-Queue-Id: BC77D180038 X-Rspamd-Server: rspam02 X-HE-Tag: 1720803675-420100 X-HE-Meta: U2FsdGVkX19jZNuRUpf2vvFNq58nPPTeXuSFtf5BHMyNyEd8zM8FtnO5C91/I+rjFsZVrLeiJNn1rmKJP8F82BE+uh4BeIXzXkwn1DzMOpkoQNA/QYGxlkxMRd7TFnnolgPB5kyFKZsPtLZaAt9R+jdb1QH7sIIe5m7ERcXk1a7DqZ1lBBBegfIpUGdkc27J8ibCmBoWwrj3WjicEOuNf3XrTuBhY3r0bBbHm5GjicjfJ9nEwsVerP27sdv7Rhk9mUV3xhrkm86AvwdsG15OKEfhoNSt5wuEjVXdY6TSw0uLL1MSktq2DmvLz6bNd75SX960T+5F+52aNrdDu2a97hrynJdch/W2sf/NNvcbsO+jD3tEDjj1t492WyS10c/fNT/Qew+DFR5YrMo50rzUm4Uimv/nNSUXKPtOeyMPuXnoLw75sgDw0EcnLH5ItGmJth9vpM45WkqEkG5QoS6iFBDjyc/9/pNuXzk7WjRiKhTQ5+fVLOm0HgLUmTKyM17an1TfXlbXDGhGCsK+QOOiqNB8XLmYwl+zeM0ex6xhVmvYmOT5b+cMYfAbBlMPwdhoVGWEvaqiYPo8H3m7G0JdwJ3uIGHOb0sO45S82p3xGBBo2KebiqcNG6puwP96gO8/tcrSKkXWCl0JD+r9iS0BwgMnAIVQYF4NFhCZRsQNDDK88La0l8Z9ZxQwUWJYu85a4FxX8N1oJw15bUcrT3dL7v2DiD6Mb1+opZgRXWCBwSf1J40ZK7ej6Ub0n06rW8R87EPboGjq/iQ1gvtGNC4ysHrV7vsK2XCk4DOzGO8SYjnf1uOpyGtJN5/r7/zJROQyAEyE79iFB6ob2TnG9yX1dyZ3sxcwK0HztfNI3NZ241KMcxbcS3sJ7shapUjWmpeoCZcNX9ZcFo4WOGrqNm8Tq70nE2fEuw3IQLL3Ela4hywpX3/GJpcr3Y00WaFT30UYsNvJAQWR+Rwy86YBaMr 4+AvqVxl SJNUCzkA33rFgxgb4eQ859reEsDkKuc6nCOyeXeFl207Q9iV5Tbwh0CNqy1nBueTxuQejMs2U2dkzsod0iNES3bTfOsNCSzDgp0nxm/pHxh1o/dlzBVO/e0lPa27YzY7PjQ7m8ZOL/gZv7EAtyqc/YcH4PSaGFI+CvauHiBPgoMp4lDAPA30BC+Oq2fCNk4U3iG8pSGCbkxS0p4j4hXeXqQsnE0gyHw9y55wwRJln4OMDSHPb4OPM5YeOrCWd1xNgHHO7GCdw52E8DFbSc3OR6kbkZMQcL71G/h99XK3tUbWLHAuxk5L46qFrrub4F/C509gUQ4xryPQxJxbwUC3PvjaBKCpo4wybkMUEQ0KQeOKuC3IE4KOFVQEGdNspfseiVEjLma2oEvMNCb+FpCuMPbeFqZGTMZgaVoqSbx9bQxzu30y2/BZS96kDOQHbl99GBo8tYzkrSqeadrvNpfqnFup2pleEVGdpYNhXa1ozrU7AB4oJ+MQG4vMnpV0WIZ0aBh+2ox/2eMqASbVicTLzPFWtgbhOs7DbVRX/ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Junaid Shahid Each restricted address space is assigned a separate PCID. Since currently only one ASI instance per-class exists for a given process, the PCID is just derived from the class index. This commit only sets the appropriate PCID when switching CR3, but does not actually use the NOFLUSH bit. That will be done by later patches. Signed-off-by: Junaid Shahid Signed-off-by: Brendan Jackman --- arch/x86/include/asm/asi.h | 10 +++++++++- arch/x86/include/asm/tlbflush.h | 3 +++ arch/x86/mm/asi.c | 7 ++++--- arch/x86/mm/tlb.c | 44 +++++++++++++++++++++++++++++++++++++---- 4 files changed, 56 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index df34a8c0560b..1a19a925300c 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -69,7 +69,14 @@ #define static_asi_enabled() cpu_feature_enabled(X86_FEATURE_ASI) #define ASI_MAX_NUM_ORDER 2 -#define ASI_MAX_NUM (1 << ASI_MAX_NUM_ORDER) +/* + * We include an ASI identifier in the higher bits of PCID to use + * different PCID for restricted ASIs from non-restricted ASIs (see asi_pcid). + * The ASI identifier we use for this is asi_index + 1, as asi_index + * starts from 0. The -1 below for ASI_MAX_NUM comes from this PCID + * space availability. + */ +#define ASI_MAX_NUM ((1 << ASI_MAX_NUM_ORDER) - 1) struct asi_hooks { /* @@ -101,6 +108,7 @@ struct asi { struct asi_class *class; struct mm_struct *mm; int64_t ref_count; + u16 index; }; DECLARE_PER_CPU_ALIGNED(struct asi *, curr_asi); diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index ed847567b25d..3605f6b99da7 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -392,6 +392,9 @@ static inline bool huge_pmd_needs_flush(pmd_t oldpmd, pmd_t newpmd) #define huge_pmd_needs_flush huge_pmd_needs_flush unsigned long build_cr3(pgd_t *pgd, u16 asid, unsigned long lam); +unsigned long build_cr3_pcid(pgd_t *pgd, u16 pcid, unsigned long lam, bool noflush); + +u16 asi_pcid(struct asi *asi, u16 asid); #ifdef CONFIG_ADDRESS_MASKING static inline u64 tlbstate_lam_cr3_mask(void) diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index 2cd8e93a4415..0ba156f879d3 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -140,6 +140,7 @@ int asi_init(struct mm_struct *mm, int asi_index, struct asi **out_asi) asi->class = &asi_class[asi_index]; asi->mm = mm; + asi->index = asi_index; exit_unlock: if (err) @@ -174,6 +175,7 @@ EXPORT_SYMBOL_GPL(asi_destroy); noinstr void __asi_enter(void) { u64 asi_cr3; + u16 pcid; struct asi *target = asi_get_target(current); /* @@ -200,9 +202,8 @@ noinstr void __asi_enter(void) */ this_cpu_write(curr_asi, target); - asi_cr3 = build_cr3(target->pgd, - this_cpu_read(cpu_tlbstate.loaded_mm_asid), - tlbstate_lam_cr3_mask()); + pcid = asi_pcid(target, this_cpu_read(cpu_tlbstate.loaded_mm_asid)); + asi_cr3 = build_cr3_pcid(target->pgd, pcid, tlbstate_lam_cr3_mask(), false); write_cr3(asi_cr3); if (target->class->ops.post_asi_enter) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 9a5afeac9654..34d61b56d33f 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -98,7 +98,12 @@ # define PTI_CONSUMED_PCID_BITS 0 #endif -#define CR3_AVAIL_PCID_BITS (X86_CR3_PCID_BITS - PTI_CONSUMED_PCID_BITS) +#define ASI_CONSUMED_PCID_BITS ASI_MAX_NUM_ORDER +#define ASI_PCID_BITS_SHIFT CR3_AVAIL_PCID_BITS +#define CR3_AVAIL_PCID_BITS (X86_CR3_PCID_BITS - PTI_CONSUMED_PCID_BITS - \ + ASI_CONSUMED_PCID_BITS) + +static_assert(BIT(CR3_AVAIL_PCID_BITS) > TLB_NR_DYN_ASIDS); /* * ASIDs are zero-based: 0->MAX_AVAIL_ASID are valid. -1 below to account @@ -155,18 +160,23 @@ static inline u16 user_pcid(u16 asid) return ret; } +static inline unsigned long __build_cr3(pgd_t *pgd, u16 pcid, unsigned long lam) +{ + return __sme_pa_nodebug(pgd) | pcid | lam; +} + inline_or_noinstr unsigned long build_cr3(pgd_t *pgd, u16 asid, unsigned long lam) { - unsigned long cr3 = __sme_pa_nodebug(pgd) | lam; + u16 pcid = 0; if (static_cpu_has(X86_FEATURE_PCID)) { VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE); - cr3 |= kern_pcid(asid); + pcid = kern_pcid(asid); } else { VM_WARN_ON_ONCE(asid != 0); } - return cr3; + return __build_cr3(pgd, pcid, lam); } static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid, @@ -181,6 +191,19 @@ static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid, return build_cr3(pgd, asid, lam) | CR3_NOFLUSH; } +inline_or_noinstr unsigned long build_cr3_pcid(pgd_t *pgd, u16 pcid, + unsigned long lam, bool noflush) +{ + u64 noflush_bit = 0; + + if (!static_cpu_has(X86_FEATURE_PCID)) + pcid = 0; + else if (noflush) + noflush_bit = CR3_NOFLUSH; + + return __build_cr3(pgd, pcid, lam) | noflush_bit; +} + /* * We get here when we do something requiring a TLB invalidation * but could not go invalidate all of the contexts. We do the @@ -995,6 +1018,19 @@ static void put_flush_tlb_info(void) #endif } +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + +inline_or_noinstr u16 asi_pcid(struct asi *asi, u16 asid) +{ + return kern_pcid(asid) | ((asi->index + 1) << ASI_PCID_BITS_SHIFT); +} + +#else /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ + +u16 asi_pcid(struct asi *asi, u16 asid) { return kern_pcid(asid); } + +#endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ + void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int stride_shift, bool freed_tables) From patchwork Fri Jul 12 17:00:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13731992 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE1E0C2BD09 for ; Fri, 12 Jul 2024 17:01:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6A33E6B00A0; Fri, 12 Jul 2024 13:01:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 62C736B00A1; Fri, 12 Jul 2024 13:01:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 42FB56B00A2; Fri, 12 Jul 2024 13:01:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1487D6B00A0 for ; Fri, 12 Jul 2024 13:01:22 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 7276E140787 for ; Fri, 12 Jul 2024 17:01:21 +0000 (UTC) X-FDA: 82331716362.06.65EE14D Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) by imf03.hostedemail.com (Postfix) with ESMTP id ED78920033 for ; Fri, 12 Jul 2024 17:01:18 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=SJvoR4yd; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3XWGRZggKCKAJACKMANBGOOGLE.COMLINUX-MMKVACK.ORG@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=3XWGRZggKCKAJACKMANBGOOGLE.COMLINUX-MMKVACK.ORG@flex--jackmanb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803653; a=rsa-sha256; cv=none; b=nhGuGgFd3UC9V9Fzyxt2sh+1KEoaQre4jhe0tajg6dfIcWauAA0YdG6EvrqX1VzoS7Hxh8 V9CrgfGc7XPZdaOv8n6F8iCHi9jgShRalheXGFLkjwuDhhqJ53DjPmg7sHHjE3If4VqR+X S3XJOYzZkTQoUlRYV6+gp8Nk2CXMTqM= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=SJvoR4yd; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3XWGRZggKCKAJACKMANBGOOGLE.COMLINUX-MMKVACK.ORG@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=3XWGRZggKCKAJACKMANBGOOGLE.COMLINUX-MMKVACK.ORG@flex--jackmanb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803653; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=STKCp8fX1eTmkj5Bbc4ElFS0AmmAwYWDzWQm15HV9Io=; b=Kvhhx2Gfzof39zjYt1Wdg0G6hKyI7+LjVHSiZz9K6ytB7WoMf/S90NMi3Orl9lFTb3sfUp ZeBJ3ogRReHyirtkvfGlvnYMwUcJIfHP7SkJIGqMqs6wc9hY5/Y7xXyAuMqmI0X7bXXJh2 CWDXPkNn9Y3cFwHtfX3iGoyArqyWhsc= Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-4272718b9b0so14630545e9.1 for ; Fri, 12 Jul 2024 10:01:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803677; x=1721408477; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=STKCp8fX1eTmkj5Bbc4ElFS0AmmAwYWDzWQm15HV9Io=; b=SJvoR4ydmpHToPGYiENBBydKQIrUsYHIOHV2AkjK6Kvze9geU/ZU0aJmRwDeZdMqjK iX2RtvzuhWlQlpGmiZ27DHAJwaHE/2q7K3TD4wmDUxMBkEoUiBidiGZNnDtweKWoWLTr TvfTI/8daH9m2Umy2iMjiQQUzZS3nu8N3FxfpPu5+p16qmXk3HR4SzrgxKRjbQpcPJpM ZeGnzJZ5kBdFFWf8n2bexTvV79T8wvEm9wHrQmJQPO42aQGNrzUrXOA0HD9A0R6za+Aj Cp0VBjQEg47deP0IBbW0lPTQqK6cHwcmgVWLGx9hpf3yu+ElMRbiaLbYfYc7WFaLvA2H V6kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803677; x=1721408477; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=STKCp8fX1eTmkj5Bbc4ElFS0AmmAwYWDzWQm15HV9Io=; b=WaFX4CFKg74qxSUq66vuuIKqGR8i8Dr6jir40ssKzU2wrwdfBkePrJ8Z2kMSdDVkz4 4f7a2aKtsExn1vdEq5paVexXYjPucKl0QKbxBxfLCeX0VaAy4/LN6OoVgVyMVoTug7vn 4JkRUoxV25/n1ZPAMrzcvQLlNHwBSiblRXZxW1J/YH9t3F0pVnFEzkxW9TfotlEu3tBJ of0I2wiGPJ3XLEGcae3zDtA8r2o7xewf9A1cDIohmUsCUkJD+EOCcecM4jd7OdgbpUpa y1UXzvuu+DrIwi3Oz7aJmdqh5h+Ra8diPCLs9oiDaVPDuCcgssJ0BU7h89ANelJz6TmJ Pkhw== X-Forwarded-Encrypted: i=1; AJvYcCWCCsi/eS+KaqGN6R9H9D3JwsQxCKRY/YJPndbU+wcb9tOprWlKXFr2Tx74tU6xv1UksNuKYx+BxYPCh30yR4hhzSE= X-Gm-Message-State: AOJu0YxPjgHZ5dVFttaHxek8CdiNTqdXloJhJBUOEPHoNFE7iL2nRFu8 v9r9IgL6MOglBuV0lHfs70fMtYbVSf/RdOp1oTgoa3G1RnEcQ7v9fTKzUagvD93ZvirV4LWH3I7 wh90AlxD20g== X-Google-Smtp-Source: AGHT+IFyt7OjPNZMl6PbkdfYNDPh8L+dPRBgusnHtzyUmHnXcEp/uNI1T+enUzqfuMGxALw2f3LiWe2F0njYig== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:600c:358a:b0:426:5ff0:1b48 with SMTP id 5b1f17b1804b1-426708ef8b5mr784025e9.4.1720803677551; Fri, 12 Jul 2024 10:01:17 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:27 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-9-144b319a40d8@google.com> Subject: [PATCH 09/26] mm: asi: Make __get_current_cr3_fast() ASI-aware From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Rspamd-Queue-Id: ED78920033 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 1iipqoype7jxigohy85a8k8xa3xfky34 X-HE-Tag: 1720803678-419373 X-HE-Meta: U2FsdGVkX1/zIvsLmupD+OFhN6aySst+ygtF60K4uhw0G0Y0p5azMaiICVlAg3FGIEDlg3JM0Uhg642gsOCQVBPgjzpqj7iNHNY30rpluLRNBn+J/mBWcE8Vy3vXxo7pOll1Vgitlv7gEQDH8PnbE3JARVbDy9PqupXdtIDVeqyVXL1folyjN5gLCkjfmSVIMdsLHy132P2k9IWC8FqpSNmA9tFqspVlJOhEc3XjCJVvqlMseWRlsSkraKEaRnx2sCSlze8MYhUWXh5nuE4gnUA2f6M7ttaQkgg4LFwRB5cqnLw4yCLgMX2cApx9RulefhO5GijvfFA4g9M9jp22idLHSYc9NARiNfh8VPEbY5btkWx0dTdQIVwgaJE1vkBNQTL1g0cD3h6pv+5/KQWR1/6y46XUfk2OU3f07NaE0i1VhSinbXADNPb0aMGd1B9FxQW9TtRxG724f4nlNcIFdZ9iQI76MmKXTfe/JQrVq7GXjKDroGsjyKbp2p6dQncyN9L/VXWyjXphT39d5c+nSMD6vM9NQb6jAJ+Y3hSok7Ps43LCVS8pDITSLVWpScHgeA+y52NjmgTwwokMu3Cc9fmkR9bq8oRCO4oksRTdyzN9IUSNowMfvVcK5RL6r3236gDidkDdPkMcY9NU35Xn8K0ljKoN1AGTdSUT8iuUJhMh1NP6dVzk1wraTpoVLoYVV17lYI3eFkHYzw3bmwEvuD4z4HQnGUmhRmWqqav2DSLzPCXx/VGJpINaGDuSgI2J8dZuhAX6a2BNhFDD4dAtNmhlcyIOFpQ/E0ivveN92wqeK2x53c2xn4BOxAt3vHsuY61KtlSB5Ii0+K3eZMhuC0lpsLmbxUFkNJiJclpX3TlgS/qWyNBqVYcen++FJrTsI+XjsfaTuzUtyAHsB77bmPVaW3SvQJbFNq+4BpJx28PpxP0auKdDQ01t+kVynLZz6z6c94xsHZkWzPXoCcc 7Fdidih/ uplBhTal9+V62KAI23wN1uvPQGdfHNA+tOyojsn9/P+HOpztSUus0gi/eDdKSWgiQmO6HS4Rww3oA8FJTcq6G/VB9bvUtXShI5EuAnz/TUNYNAaqJdi7PLgQBIhEkb4NSRNbbAU8PaZR7Q4SombPA36w37H0vkYeR3Cdc7V73RmXfRTzFSsunWGQPgbHv/7PiPR/sNQYSB7NBeliqcwDXabNQkbpeXG+v4sck1uFg1qEqwPqrSDda/C8OcLGWnIFVAVu9pzrYsxaQeRJVq02VOBtANwscx/CSakV0kQR4CPy6HHj/5WAqGuiY6N9q74IurVX6VVtszHAyl7ek0oO8u4AC/Z/JDQeOBXeWhIqS40neY/n4MHsMr+CfGBNnttUJ1osm2FR9FGVh0RCDeAKdIjrrMIrVSBGlZ03X+VoHr8gRHBJklLv2Xyi2ohuwG29hTDLFiNLLT5zmaG+E/cjr6aUm69m8wlQ6FFEkp1G8UYJ1fPLQt6RUQCwxOvfcIzKsuLMHK/utYUhdr+7CrH/XByXFB4CPBgG4Qz5W9FuCUswHf9yUsdmf1vG818hThMgZ5hoiixmbVos8EbYgMq5az3gRH4EbQF+ihUaPFaWrc7i7Ae3Lj8RWX4lT9A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Junaid Shahid When ASI is active, __get_current_cr3_fast() adjusts the returned CR3 value accordingly to reflect the actual ASI CR3. Signed-off-by: Junaid Shahid Signed-off-by: Brendan Jackman --- arch/x86/mm/tlb.c | 27 +++++++++++++++++++++++---- 1 file changed, 23 insertions(+), 4 deletions(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 34d61b56d33f..02f73a71d4ea 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include "mm_internal.h" @@ -1125,14 +1126,32 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end) */ inline_or_noinstr unsigned long __get_current_cr3_fast(void) { - unsigned long cr3 = - build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, - this_cpu_read(cpu_tlbstate.loaded_mm_asid), - tlbstate_lam_cr3_mask()); + unsigned long cr3; + pgd_t *pgd; + u16 asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); + struct asi *asi = asi_get_current(); + u16 pcid; + + if (asi) { + pgd = asi_pgd(asi); + pcid = asi_pcid(asi, asid); + } else { + pgd = this_cpu_read(cpu_tlbstate.loaded_mm)->pgd; + pcid = kern_pcid(asid); + } + + cr3 = build_cr3_pcid(pgd, pcid, tlbstate_lam_cr3_mask(), false); /* For now, be very restrictive about when this can be called. */ VM_WARN_ON(in_nmi() || preemptible()); + /* + * Outside of the ASI critical section, an ASI-restricted CR3 is + * unstable because an interrupt (including an inner interrupt, if we're + * already in one) could cause a persistent asi_exit. + */ + VM_WARN_ON_ONCE(asi && (asi_is_relaxed() || asi_intr_nest_depth())); + VM_BUG_ON(cr3 != __read_cr3()); return cr3; } From patchwork Fri Jul 12 17:00:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13731993 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2A1CC2BD09 for ; Fri, 12 Jul 2024 17:01:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BAA846B00A1; Fri, 12 Jul 2024 13:01:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B358F6B00A2; Fri, 12 Jul 2024 13:01:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 937FE6B00A3; Fri, 12 Jul 2024 13:01:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 676E86B00A1 for ; Fri, 12 Jul 2024 13:01:24 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 1D8BDA3F26 for ; Fri, 12 Jul 2024 17:01:24 +0000 (UTC) X-FDA: 82331716488.12.0484F66 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) by imf16.hostedemail.com (Postfix) with ESMTP id EC53D18002A for ; Fri, 12 Jul 2024 17:01:21 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="Z/ZdD3cF"; spf=pass (imf16.hostedemail.com: domain of 3YGGRZggKCKMMDFNPDQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=3YGGRZggKCKMMDFNPDQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803647; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jms+qnUoGWR3Ok8D5PSRh1HvZeNFLLNfdQwhqdA9AN8=; b=JcQCh6+SYgkBypsp30omWSUocgwujasfYqNiM2iJqnJ6Gd4+AZH0eFkzrC1aWIb94JttNx JNQ0hLLuJ3TFxL7DWRVQ+vjyQInSpj5Rf0UfzNm2EljC771RPvtT8L3OH/pOT6OApWHubI BZB0/W+2TKD4sUMQeaXkDCr3eoVo4eY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803647; a=rsa-sha256; cv=none; b=Y+IlCYV18sOpBzseypmAO/dzz1cHo0izBn+pkahYeIqNeN9bXX6EzjKmqJc/k+PG92Xy7n NC2rdVh5zCza1/PJeS9HUI7UGG/RcoyD3jYIb0OeqISbwrM5WWS7gDKpnkfAku6tVDfVIe IcaVhpqDT3nKt5Kc4g+/AHjSOrihDas= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="Z/ZdD3cF"; spf=pass (imf16.hostedemail.com: domain of 3YGGRZggKCKMMDFNPDQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=3YGGRZggKCKMMDFNPDQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-4265464ddc9so12814175e9.1 for ; Fri, 12 Jul 2024 10:01:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803680; x=1721408480; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=jms+qnUoGWR3Ok8D5PSRh1HvZeNFLLNfdQwhqdA9AN8=; b=Z/ZdD3cFyg+wQ+WLYsPvtVddXtKR6sC5/cipvY4MVDPPdnRUA7fDx8h4s6kN/l6JLA fbHbzT/ZbTwX18lJA5WUeomu1whBa23bW5VSe7SYpx+txuVbgCeLot1YaJKrX2FcHzUF t780esivmslZf9oSfuZCmAp+3yYZFGK63/5MzXVVMfjuOPNcjl/BByw21AV3wlkDrRHM RqYhPA0x5uXRbmgVVqBzEhynMCo6ESJ3VjjdzwkGRLNlQMvkqebnCcwTpgK9CVvYoZv0 8tJmJWdKB2cMPwrJMmi820UkrQpFxf109DpcLS8aPAApNeO94rmOGF4on9f8PXKmRnh8 w4Xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803680; x=1721408480; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jms+qnUoGWR3Ok8D5PSRh1HvZeNFLLNfdQwhqdA9AN8=; b=QpK0PYk3VL6FbpelIWxYvJqmEYgKw9/edoL9IjvyQVR1Y1s26QQMd0cfGI3LaWQRo7 cu93Kl+MLbjJG8gSp3tvN7iWvjmk+NIP+NRxvsTds15hlLswlDLtEj/08yOmoo+cOUmf uuhtxkglgouwbREcYD6/hJ6w7AsM5Ik+SevbO4WJhKUnwevgeqcVINhgoXjlu+KRkPd7 /gseA+xbmwCksTV0HyqEqSdG0gNQMIimUgBe8qoQqzASlguFQ2RmyJwGUK9w/fTFJEjg 1YKgI+S+4p5gnavGsZIoaFA9gmVcyRvkLkPhYqhn1A2H6vfSA9h9T3hX4VmxzwpsOtBq 0ovA== X-Forwarded-Encrypted: i=1; AJvYcCXUynzzLzKIluF/EQ4M54sgcm5d5czGZaWx0COgB/eivDJxKhFMtS9IZdq0zKwVFwwRPDvZaJEYq3iPbAp8VlUbZ2Y= X-Gm-Message-State: AOJu0YwDBUKN/B/ki//KZMvhYi+dxd1zp5UfxPsMTK7+8hUEY5pdaL94 /EQrpg4LapYIiarl361gLf6VF5Hc7/fMB0B9eBl6e1ukwp8uYkNP/yBBC+lLKv50RJgqCZq+RF9 JXkS56nhbRw== X-Google-Smtp-Source: AGHT+IGisRm/zH5xPuc3vZntkMct9zRaY/l9dILlFDVOJcrKy+wGrjnqUwhX4l9eAthBD2p9mDlo+nx2x2+a6A== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a7b:cb11:0:b0:426:670b:20bf with SMTP id 5b1f17b1804b1-427a0a24cb6mr168765e9.0.1720803680269; Fri, 12 Jul 2024 10:01:20 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:28 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-10-144b319a40d8@google.com> Subject: [PATCH 10/26] mm: asi: Avoid warning from NMI userspace accesses in ASI context From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Stat-Signature: mbi113szfy5hc6o7rqi1oc6fiabmtj7j X-Rspamd-Queue-Id: EC53D18002A X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1720803681-575358 X-HE-Meta: U2FsdGVkX1/Yc3w/W+eXmj/VlBZR7theP9cGvrvvIE0rVMHEgTwmeSKbn2TxIKeKKlp04KL25W2LaDvWLQBRMyMsDDMMM3/kp8N6P7CwLlb83bce91tQUpIWupc5iZsU860efz1Jg3PwkW5CrRBWR7C0wqj7AIlCCj2ujGOId6eB1u2bP3B//5/nOuaRzSB9h2qE53Mpg2b7l47YHb1MbOTAx3PYp88ktSfoNfCvdNcPI90JbB5pUbgPziVPFDFhN5eo6A50EmeRU3ud2FGQdHFk28/e9ZjknuqEbOzqqHU5Fmq4Q0FTmCCmEN/FfeC7uqTWsET280mnSkqKwJrFp6dQRe5xDAf9Gwo8GaFZjZ+2EAgubbL35vwhz6D6WaGvkO422aKO1puymzP505jxXKP7NSUE/qyJZ/o7Vqx1gRbpI8fSYaPX3s+Dsy+1wix+Ij9fR4UFTxfa3KdAwX2hJjHOpEdy0afmmfTR0t+DtseshnCtIajf/f/QSTCM7DUQC9FuH3xxf1FKiSeyfHOKsXfx3cJj4DxQkF6rPfPwlx9sV1HuSsv0ePRBAIBUZ8/fqzKjKow7T59bgdDtXyLG9V9XwEMU6qpiTbF4Kj6X2V0m2+6Uzx9oV6U5vPHlorrBvPeVt3LGK3ABFeXHcgPffIlXiihDdX9f3Tv3fbOP+gHl9mLhBIc5eRMp0558nDmRKrR/o7iGr/TxrZ8vitljpVTdqzerqOnRWlT9VSJ0k2IANae0Y4qZVOGffsOpsymvcy9E9k5SMbtcFrqYNVaVJQG6OkY5YcKvbikMxn3SUo6QztxmDRkmGng7n2Z1c4jtGduDEuYX3cMMRLLAqGbQlpMuwx/eah9u+jZDr4B4zaKzgfa86ldnqHrLMSYoxufFHFvsOgbCwAhyZ8AkXlMaS9JHrHpvBb91yi0H4Jx+hR0PZgYhwRNCGE7wrOiOaFJX8LZRueZT8Zwqh5Y+xm1 kDPQ93QW YHyNU/IcfDgXAQo+LO/EApYVwm/cXsF8vyXtf6uqS6isnPGDCZF0v+kimeHlynymze791s7ILapHYPpUD+mPNNogJ+B2hbb2PfO7+3nme5b5JMFmlB1ZX33g4FNtavPxt0M9n40gD9VcsAda8i1rYup/MNCeduVTRz9X4RVfkDInDrEClDDMFtOaDmu4kr6I2SLLmlJrHhwsmAJh9OpWwKxdIH6ynwabJnNBUYz5amxzECOweFYGXo/ew5x9qYxFF9kkWOSuVXC0tD8+0c9WddaF1dtQgiDDF8qlZRuVqTBm+4yES5fEK0kUAVv9BMcZmI2q/QB6VtIYi82bhP/nsim7vUI/sY64PI/dUpVQ3Klaz66X3iVW4GBNyLtdeQXz6xv5rxcBHpmavqDtP7zP2NRmdgDYlUsRjclmHCZ6G8fNeXQFzc2roa/7SAAanGGIV1C0zOqoCT3SfAGYJZaXlSIzZ9tQ+y3YwH35Bnjd+B1e9yDp6oQFe9tZBv1VGdaIoSLvtVfa/DjaLlKtcd7Z2N9AEvFkvYp1Fc+D6 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: nmi_uaccess_okay() emits a warning if current CR3 != mm->pgd. Limit the warning to only when ASI is not active. Co-developed-by: Junaid Shahid Signed-off-by: Brendan Jackman --- arch/x86/mm/tlb.c | 28 +++++++++++++++++++++++----- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 02f73a71d4ea..e80cd67a5239 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1326,6 +1326,24 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) put_cpu(); } +static inline bool cr3_matches_current_mm(void) +{ + struct asi *asi = asi_get_current(); + pgd_t *cr3_pgd; + + /* + * Prevent read_cr3_pa -> [NMI, asi_exit] -> asi_get_current, + * otherwise we might find CR3 pointing to the ASI PGD but not + * find a current ASI domain. + */ + barrier(); + cr3_pgd = __va(read_cr3_pa()); + + if (cr3_pgd == current->mm->pgd) + return true; + return asi && (cr3_pgd == asi_pgd(asi)); +} + /* * Blindly accessing user memory from NMI context can be dangerous * if we're in the middle of switching the current user task or @@ -1341,10 +1359,10 @@ bool nmi_uaccess_okay(void) VM_WARN_ON_ONCE(!loaded_mm); /* - * The condition we want to check is - * current_mm->pgd == __va(read_cr3_pa()). This may be slow, though, - * if we're running in a VM with shadow paging, and nmi_uaccess_okay() - * is supposed to be reasonably fast. + * The condition we want to check that CR3 points to either + * current_mm->pgd or an appropriate ASI PGD. Reading CR3 may be slow, + * though, if we're running in a VM with shadow paging, and + * nmi_uaccess_okay() is supposed to be reasonably fast. * * Instead, we check the almost equivalent but somewhat conservative * condition below, and we rely on the fact that switch_mm_irqs_off() @@ -1353,7 +1371,7 @@ bool nmi_uaccess_okay(void) if (loaded_mm != current_mm) return false; - VM_WARN_ON_ONCE(current_mm->pgd != __va(read_cr3_pa())); + VM_WARN_ON_ONCE(!cr3_matches_current_mm()); return true; } From patchwork Fri Jul 12 17:00:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13731994 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD906C2BD09 for ; Fri, 12 Jul 2024 17:01:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2F8216B00A3; Fri, 12 Jul 2024 13:01:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 282516B00A4; Fri, 12 Jul 2024 13:01:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 086566B00A5; Fri, 12 Jul 2024 13:01:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id DB9846B00A3 for ; Fri, 12 Jul 2024 13:01:27 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 3A36C160C58 for ; Fri, 12 Jul 2024 17:01:27 +0000 (UTC) X-FDA: 82331716614.08.6E54B5B Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) by imf04.hostedemail.com (Postfix) with ESMTP id 4069740012 for ; Fri, 12 Jul 2024 17:01:24 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ROohRSID; spf=pass (imf04.hostedemail.com: domain of 3Y2GRZggKCKYPGIQSGTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=3Y2GRZggKCKYPGIQSGTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803669; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tzyFc6toxGAQrtxbLGqZIE4RB+JOOXOhclq2H8+llZE=; b=IhYU5mLtgQxizFsp/huhhNgwQh4hgVnKp1XYaIo2VklO9S0my3kvB05Qs/sqpzv0XEJR/K TNNf/Rqc7PjAVylryzsr2k6AWWxwGAp/CMqmXjPGnJBTC8gQ3bMHKlNF2IziGn/jI+Fu9h BLxMkt1WcHLc7MXzQTCjEdcIBEMMwW4= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ROohRSID; spf=pass (imf04.hostedemail.com: domain of 3Y2GRZggKCKYPGIQSGTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=3Y2GRZggKCKYPGIQSGTHMUUMRK.IUSROTad-SSQbGIQ.UXM@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803669; a=rsa-sha256; cv=none; b=gpbwvGOrbPSTqcIL7AO6EEBSjlRjWyRwwgzj/XzQUVs5PG4eN8VPeLMs3rfnLBKCXHJ0rm kvN9rDykp4tHfmu3oACb/bUhdDykm5jEUP8HBYEUWH9Kc6kQryvWMCBBm6Y8YNX61Um+L9 k1yTr8UhQv+m3U9rAPBwNx1smuaW8ZE= Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-4266fbae4c6so15467995e9.0 for ; Fri, 12 Jul 2024 10:01:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803684; x=1721408484; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=tzyFc6toxGAQrtxbLGqZIE4RB+JOOXOhclq2H8+llZE=; b=ROohRSIDsX0lUX6mhLpijgiXanWO/6e3v2FVn9RZisArO+vpP7dLXy3HWsahUSwA5J halAz2WsFU6k3LdoOlxgdPW7esN0PAqYUGid4Gup8CDjQVAv8gboJpBg7GdIegrGBoiG +mAQI9k3AkvsTvYi4wC1eFx67RTdWPpx/EcRl7m7khDowEO2LwMUoBLmIbR4bwqxu0MA DlRtlr/41+n20C8cN82YvOcMVs+wMDMKvpeXOsyTzKWRKEYhYJGir45ZVlZgfOfY1+D8 WPKyR/8M1Hevokvs85v6MSpxwVXVueOc6L1D8sTFnjG8j9OkUv2JeCRgmWDeyjTP+I1c MtIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803684; x=1721408484; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tzyFc6toxGAQrtxbLGqZIE4RB+JOOXOhclq2H8+llZE=; b=VMpfWkAfyPQs0Apv4f2AA6Sol0UNHCilpd9n19XdzO5h1wC5JYImpZfXPR4POkj1Sx 8KkzXqrO/HSD2vl1LifPYvdKt1/CgI3wiaMYqQIL4vILMHKZnSakZ3Udk27GBnED+NhB mtTnwxJKSMEJ8rrD48KvGxosL9BNAQypeobVuiPUv/y1JlKVKUYKeiZtJek+4dWF8ly6 WXnLj1Yfl6nqbhNYn16lhpafum0+hPXv0018abcN3w6egHIejbZUpNK8FuBay7s+jQgK a+et8TyHX+axD/EKOvUk+m9lUP5LiKS41Io/2I/XTL5/BBqbydImLaFQcMDZF6F8yO3O ClfA== X-Forwarded-Encrypted: i=1; AJvYcCUkyKu+k61vJUa/0J9gxQ7OBTfJED8GXnoMgh3U82cOsuTaKpNz0E4oR2NUmZM6dHoWoFBL7fEzxpd27cwM9bo9T5c= X-Gm-Message-State: AOJu0YzfvsAs/DNdQS8x27m+zhKZAmq0XDme4VuwFSbtkbTlqAg/TJVC NAPuIA26u2PB/K6kwdinfvmzIMkukzWbIEPhALAOy+Us3uRYgbm97W8jhqNJqrIiSUxaFJMYZ7t ETfbUctN4/w== X-Google-Smtp-Source: AGHT+IEU6XhUTfh8cq6gCvMggnXLFkSt7XlYIzIUi6HAiuQhdPaLLmoiwQpzZ2rTTEjQ37T3mDQb8/UfRpK6jw== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:600c:4f0e:b0:426:6a73:fb5f with SMTP id 5b1f17b1804b1-426708f9ab5mr1904875e9.7.1720803683507; Fri, 12 Jul 2024 10:01:23 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:29 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-11-144b319a40d8@google.com> Subject: [PATCH 11/26] mm: asi: ASI page table allocation functions From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 4069740012 X-Stat-Signature: ctks44strrwncq8s6kcq46aix5mcr11u X-HE-Tag: 1720803684-748232 X-HE-Meta: U2FsdGVkX1/lQBXojyAdueW/AqWyKEnp5SDIGEoUocd+VUlV2maBBww5AQqtj+oiiQjKhePIcAhlM+hxmL66fReyOE3eF0+JksglrUSuSPXEwbpl2hpDwdsziZSminJdh2OMt2DmeXhIlJWSrA7eq7lp3QuKGB+EAyr/C6Xm9la8f8aZbdOL8EH7DU6R4GZPyEmncn1+APL0yyB4BwizHHj9/IlsCdZTZ+7++a4gd7kt1ggFM9DYHBIddWmMDmGTcd0QO6//NRs8W2cfHN5LJBsC08Bz2Vt8WSqTY5aVfx/+84cti6zlQiigr7/ienXJeombnOrUYyMAg//ipysZ7iRnmdxIHRZRwuaaSmblNuH2XfzM1vAaNvRERh7lkim4nWE+OOtoJGjAn8hq5r50Xh/2RmfgEnKMmpvd5x23XC5gPp/6n2D3R+7CPlwfb8GDJluHRxuq+hReMeGM+Enf+E+Z+IiqoYoGuG4iLjTYDATDYIJddpIqTgdTE11q+LYJw4ioMlRCrnTFsdgFRlYW81BL5nDvP5RUGa1ntlqPsDDRrdJXu0JYlWxHJaaqtwJR+2/7I3zTiGJBeRK+Lvoxv4k87y9jkZZZckU11J5OLSsb3/sKHODIpinPNfW0iq1JqznaQfvaizGrK5DjrgxqwWJ/FOfrrr2V8w7Og4I9UnL2U3+KfeO6yIBqqXky/HEPk+WZEoZapRvTmYs1sP1uOGc+rpb1AZ699ATJFaIeJCvG+f4h0CSaGF3oLKhX4PzhpppSUo0NI7WiJP+aOX/ze9/UWoNpz4npvDZwWP+rl5p5OTs4Z865/nGRAb15UqgbKP+0TXyEahJXlUFHXHLu2iirKz2jqiUhzrzT515Pk/v4E/xflwQIS+61/vITllmSyVUSzmLq4kzAdo8kb8zf4e6xsWo8P5Z5amNi86875BtcilyKMv+gpCO7sZ36ImTxdiGce5HaGkqRynrr5ns iPxYM6wX P7IXgVH4k9s/KajSMr0tJMo5B2jKUj+LPdoYuieV6cG3C7hR04xBw7LhIZgWElfLMQZ3Cc+7IRQclvluNoAy0XBCFEJ4pdELElzSkd5B0R6KVhT/PnZ1bHCVuAC2Qs8TeL0rhAzLEBi1bwJjuaVkqaqNZlq9qFAJ7OeURjLRShR/vK7seawKZsVQthj4ilZvJ6YrsBLQ/WTgwBFGaLg1g7Mj7OkTC1mdakly82r9ul78ZBUdy95m9IfL8K7OFOZNwhRIwoNeN9NEMqfl0qZU/M04e6VQkDmcttX701wAby62IDlrQfKI5qtcCE4PxDFz7462RP+ZBygNlMmDiJK+X3h9hwE+dGB7Xf1cb5QORflOmF7TNdBFhFSVzV8lVwSfPdrYm8aQZS1whP9ZJiI5JNA93QnzVz84rb/kehjY52FM8Uy1+fyKNoxMznpXmF3oHdk+5H4zvNcGHkYlrwFz+3hGkadpEewEv1WUrgoiNHU62u7Ywb/Ill04hpZKwDglSn+A7g1Qe22neWbJBZXWLLfLuTtfNjBT0DjFG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Junaid Shahid This adds custom allocation and free functions for ASI page tables. The alloc functions support allocating memory using different GFP reclaim flags, in order to be able to support non-sensitive allocations from both standard and atomic contexts. They also install the page tables locklessly, which makes it slightly simpler to handle non-sensitive allocations from interrupts/exceptions. Signed-off-by: Junaid Shahid Signed-off-by: Brendan Jackman --- arch/x86/mm/asi.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index 0ba156f879d3..8798aab66748 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -71,6 +71,65 @@ void asi_unregister_class(int index) } EXPORT_SYMBOL_GPL(asi_unregister_class); +#ifndef mm_inc_nr_p4ds +#define mm_inc_nr_p4ds(mm) do {} while (false) +#endif + +#ifndef mm_dec_nr_p4ds +#define mm_dec_nr_p4ds(mm) do {} while (false) +#endif + +#define pte_offset pte_offset_kernel + +/* + * asi_p4d_alloc, asi_pud_alloc, asi_pmd_alloc, asi_pte_alloc. + * + * These are like the normal xxx_alloc functions, but: + * + * - They use atomic operations instead of taking a spinlock; this allows them + * to be used from interrupts. This is necessary because we use the page + * allocator from interrupts and the page allocator ultimately calls this + * code. + * - They support customizing the allocation flags. + * + * On the other hand, they do not use the normal page allocation infrastructure, + * that means that PTE pages do not have the PageTable type nor the PagePgtable + * flag and we don't increment the meminfo stat (NR_PAGETABLE) as they do. + */ +static_assert(!IS_ENABLED(CONFIG_PARAVIRT)); +#define DEFINE_ASI_PGTBL_ALLOC(base, level) \ +__maybe_unused \ +static level##_t * asi_##level##_alloc(struct asi *asi, \ + base##_t *base, ulong addr, \ + gfp_t flags) \ +{ \ + if (unlikely(base##_none(*base))) { \ + ulong pgtbl = get_zeroed_page(flags); \ + phys_addr_t pgtbl_pa; \ + \ + if (!pgtbl) \ + return NULL; \ + \ + pgtbl_pa = __pa(pgtbl); \ + \ + if (cmpxchg((ulong *)base, 0, \ + pgtbl_pa | _PAGE_TABLE) != 0) { \ + free_page(pgtbl); \ + goto out; \ + } \ + \ + mm_inc_nr_##level##s(asi->mm); \ + } \ +out: \ + VM_BUG_ON(base##_leaf(*base)); \ + return level##_offset(base, addr); \ +} + +DEFINE_ASI_PGTBL_ALLOC(pgd, p4d) +DEFINE_ASI_PGTBL_ALLOC(p4d, pud) +DEFINE_ASI_PGTBL_ALLOC(pud, pmd) +DEFINE_ASI_PGTBL_ALLOC(pmd, pte) + void __init asi_check_boottime_disable(void) { bool enabled = IS_ENABLED(CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION_DEFAULT_ON); From patchwork Fri Jul 12 17:00:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13731995 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAACAC2BD09 for ; Fri, 12 Jul 2024 17:01:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 716D56B00A5; Fri, 12 Jul 2024 13:01:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 69FCC6B00A6; Fri, 12 Jul 2024 13:01:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F2AB6B00A7; Fri, 12 Jul 2024 13:01:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2621E6B00A5 for ; Fri, 12 Jul 2024 13:01:30 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id DFC03140C10 for ; Fri, 12 Jul 2024 17:01:29 +0000 (UTC) X-FDA: 82331716698.27.CC7B6CD Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) by imf12.hostedemail.com (Postfix) with ESMTP id D18A340018 for ; Fri, 12 Jul 2024 17:01:27 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=SEtLCcsV; spf=pass (imf12.hostedemail.com: domain of 3ZmGRZggKCKkSJLTVJWKPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--jackmanb.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=3ZmGRZggKCKkSJLTVJWKPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803642; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MiTpcmi4SGpEzjuyBzeicJVJrQMm8QovFUpVk7p2Hrs=; b=CYKgEt75xQX1rncGbTQKoVFCNYjFCcEUvBufIY0yvi8tfPXdM4TT82xd7/MGNCmG0ZJ6By hwM/iQFB030XzncPN9ka1004P982HuwaIxyD6vWs0T4z2UErYtJqa/CAn0xDel0SR4ErIy P30arMj9ZZjy8XreK00+ONMSIaRZaEQ= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=SEtLCcsV; spf=pass (imf12.hostedemail.com: domain of 3ZmGRZggKCKkSJLTVJWKPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--jackmanb.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=3ZmGRZggKCKkSJLTVJWKPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803642; a=rsa-sha256; cv=none; b=yOvsDAbi1mczIZtDxaEOtpAVr5eY+cO44QSvFP/KzTxI09lAHPhu9sMQfRSmZ10ZK20Jk0 vwWaUB99nXc4F0PTfosRRDcI5OAndZT0ms9p+yTscR9Rxe3omJkHYYsdBlUjLL5sjxQ7Up sG8CTkjVV42JxcSG59eM0ukK1qqxJWo= Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-42674318a4eso20458135e9.1 for ; Fri, 12 Jul 2024 10:01:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803686; x=1721408486; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MiTpcmi4SGpEzjuyBzeicJVJrQMm8QovFUpVk7p2Hrs=; b=SEtLCcsVgT3bdS0WLC6sXaBPT3tOytqB1mbmUZw6yf9YWGq+RvTF3X1g3xKpcjjqRe Bl54ODgjE0MsSu43kaDtfuFH+mD87lHzTn93jfcLn5VCdoKhQHZMlWAipHCCeV3atrx4 vJN5qt7bPFo6jJJmUadmiMtIAFBCzROwbSP+M5z+Fnl14lD8ka9XEZ0AyZBU1ZbNBeq1 NYlFxIjFFFR3xCbZOPAwgwkhwhJAPOjLcaZC7AMSEpi3jTSFTmFG0naP8jCOpd7ACiyT AN9m3Rd//TxjmKhyPe2ISq6TwJQVdL5n9fWuE8yLXuYu25wLzKp1gf9e+ib4lr8TK5MB zMsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803686; x=1721408486; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MiTpcmi4SGpEzjuyBzeicJVJrQMm8QovFUpVk7p2Hrs=; b=FLoeXS3JUkcMtdr3WrOylItrS+xVR6U2mo38EasLuX72MH0jkPxwedAuFSipSZmZKU zenD733bGg9qAxC7wmZEWXmY7UtWPqKpaFISmvrVUCl/hStDCRH4KxnB5juqswAU8Ubt fKpH2XmUELcj/Fc6apdUtKUzYVUCMgr4ho/yG9jsSe0LA7OGMYwbqb3bHNNrEN48QMxj cqG1lx8eR4oSgIMhKqfnQsO3AaIkzHKGNsoxyVlyBZ520t0Tbs1Fe5uvOUaJBa0XDivR 2GtEQSejQLz9sbQI8VFRe1wifFSeqOySYiMu2eEZ9/CnCk/6UmytnF31ak22IObOBwov vi/Q== X-Forwarded-Encrypted: i=1; AJvYcCXtmlyOyZtIl+e9p2X2MVw08QptCYDumjnAtjTeLVRSdi79d+n5Bs+imfFJHddxEbzEQAkVDK+rql8WJYo7yAQ/1mM= X-Gm-Message-State: AOJu0YyRUQwkcOtD7HApRtAsSQA2/RddyoqTsIeFv5hUiPkuajIIngfT mZBc7ninnPsXPiZp+xQzMlsHplcSB0jsOTeezuZuQJr2BegUTkFutm1TNj9yV+bv4Sd/TFc2g0d O7vdGemLEgQ== X-Google-Smtp-Source: AGHT+IHaxca8Kt7JtoFlz5j8ZwGaa7ohvNgTm761vo7JWzSlYydrUy+c3mRwk3Ip75evYABidpu6mgorHtLpeg== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a5d:6387:0:b0:367:890e:935e with SMTP id ffacd0b85a97d-367cea67da0mr20496f8f.4.1720803686255; Fri, 12 Jul 2024 10:01:26 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:30 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-12-144b319a40d8@google.com> Subject: [PATCH 12/26] mm: asi: asi_exit() on PF, skip handling if address is accessible From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: D18A340018 X-Stat-Signature: 97rb9tnpcguue5tm68azw8yzis7q11om X-Rspam-User: X-HE-Tag: 1720803687-85867 X-HE-Meta: U2FsdGVkX1+Q7fvK5PfzdaJoYJ4an1lnK21CVaJwV/3N+1gZU64OX0fApkw/pGb0MVpbjsZblOaxkFQLhwdKlFUq1BK7yVF9Lo21JQnDkL+/wKDMzhwb6lI7hNvv+0uh7TCYg6LzBKU9E6vreftmd2Y77+/SiaLbJZYKlp4KAeLZvXYggsqOuByXh6UspqEZi/KiCtDswDx070oOCTfTQy99AMH5ebmvu5tTUqSza76WEHy37C065413mNh8KyjpcZa3VO9UzQYuwIsiYuQQeDy/BUvu+aEtFP97tDGB8qQnaNrELYTAGVnnBELPuVi17OBembRrf1ejHr8BXkYlkrzPCaSqWMUIkP+XzOnhfADlnyWcXHoQhnMWC2fPTmEHtDRKDvQSlLzm2VWskI3oJF/c6RhUIkKJhW4aQ9mweOFQnJntXJU4Lh3AOLoDWbUYtvgLUg39v2i4m1ChG8jAN1ExOEN9W5YimAfYXu65YAP9ExNq7wHWWCvuYCqEvGJeveWxYB2B/IYo3TINnvu1Gyrb1G7vUc6bzhBGG0aIF9T+guuNEo2oO6froWRFGQPQUqrlR8DSTNBLTIxHQjdpVACxdvy9rO18TKqj24WyxSDPzP4MKPzjwlm4JPNcXraO7eo9Eoc86rQkDys64DbF3yz+JSDE+G+QCqs8W2tgTRkJk2xmNhJ2qdiA4Ad25fw1f4IGybLhKXU1bc1X3XkpmWly2i3UeT7RGjklhook2FwBma7FN2TRP70o1Lpe52v/eNVnZPYupvmnG5z2I5MZ+9KleDZ8ZtMOuJopg5tInjyRsRCyOXNWB2BhqXQfuU+LWL3jpFh/REf466UzniNiqGWK1su5mXfV2GHOZ92TIJAMjHESuJeBnCKU8AnX4DZHq97IV2um9zQ5NPu71KWarvyCnUXNk8xw/L1pITWo0KqY2u6ymfWrTHO78DArFYVbex82men/ihNd0IScv2L 0vmPO5eU PgsHqbGCVNngk1ZOO0qiSjFGQ7E7lWnLszqwhJY7XQXwmIIl66P0J7VB7tH0BLlUOS7jnHnBuZwN/TGvkDgZx6uT1n5uc7zBl6KDovUBGxGpqCQVhiJQrY0zKUBqqYomF1Sy35MHJKvy8OLjmsvxMlzLBighFBE5YSZuPhy0ojpVhFkLMtzI96tE3NiYePv8GT+JYL2m8iJfdBIQPBFnB8NtzUqI+RT9nYLsAvX9CgH4jG/3BsqswvAe172QCdSh/+A7U089aRU/ZsibyGoj5F2kMwnBo+BV+wxSOpuc2b7Ls/lT2bDq2eKc4FeF4WZOuM8iqDiadyh3eY7OcGxjEgElDV9/LDPzkhWXm+z5164kRQqR7wikGq2ilRjToIYKjwmEiZQq0zv4F86i/CZmpSOnJ/AxfSeDoalDiWIWiH2yEVq9QJ+IbKuLB4Ye1dCPAJsqzGU3lUd+/dOM0Gb0AEZibG5cQx7Wk8dfODKmOE07wZypfrPiTCcm3ZkmWMCH39+hnzaX/fpD5wJuV+WPwcZom1mzpZMn5SBLQ8M7gMGhq1kMfoUNES2+c2VVEMflmAnMnpYjs0JYhvaM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Ofir Weisse On a page-fault - do asi_exit(). Then check if now after the exit the address is accessible. We do this by refactoring spurious_kernel_fault() into two parts: 1. Verify that the error code value is something that could arise from a lazy TLB update. 2. Walk the page table and verify permissions, which is now called is_address_accessible(). We also define PTE_PRESENT() and PMD_PRESENT() which are suitable for checking userspace pages. For the sake of spurious faults, pte_present() and pmd_present() are only good for kernelspace pages. This is because these macros might return true even if the present bit is 0 (only relevant for userspace). Signed-off-by: Ofir Weisse Signed-off-by: Brendan Jackman --- arch/x86/mm/fault.c | 119 +++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 104 insertions(+), 15 deletions(-) diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index bba4e020dd64..e0bc5006c371 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -942,7 +942,7 @@ do_sigbus(struct pt_regs *regs, unsigned long error_code, unsigned long address, force_sig_fault(SIGBUS, BUS_ADRERR, (void __user *)address); } -static int spurious_kernel_fault_check(unsigned long error_code, pte_t *pte) +static __always_inline int kernel_protection_ok(unsigned long error_code, pte_t *pte) { if ((error_code & X86_PF_WRITE) && !pte_write(*pte)) return 0; @@ -953,6 +953,9 @@ static int spurious_kernel_fault_check(unsigned long error_code, pte_t *pte) return 1; } +static inline_or_noinstr int kernel_access_ok( + unsigned long error_code, unsigned long address, pgd_t *pgd); + /* * Handle a spurious fault caused by a stale TLB entry. * @@ -978,11 +981,6 @@ static noinline int spurious_kernel_fault(unsigned long error_code, unsigned long address) { pgd_t *pgd; - p4d_t *p4d; - pud_t *pud; - pmd_t *pmd; - pte_t *pte; - int ret; /* * Only writes to RO or instruction fetches from NX may cause @@ -998,6 +996,50 @@ spurious_kernel_fault(unsigned long error_code, unsigned long address) return 0; pgd = init_mm.pgd + pgd_index(address); + return kernel_access_ok(error_code, address, pgd); +} +NOKPROBE_SYMBOL(spurious_kernel_fault); + +/* + * For kernel addresses, pte_present and pmd_present are sufficient for + * is_address_accessible. For user addresses these functions will return true + * even though the pte is not actually accessible by hardware (i.e _PAGE_PRESENT + * is not set). This happens in cases where the pages are physically present in + * memory, but they are not made accessible to hardware as they need software + * handling first: + * + * - ptes/pmds with _PAGE_PROTNONE need autonuma balancing (see pte_protnone(), + * change_prot_numa(), and do_numa_page()). + * + * - pmds with _PAGE_PSE & !_PAGE_PRESENT are undergoing splitting (see + * split_huge_page()). + * + * Here, we care about whether the hardware can actually access the page right + * now. + * + * These issues aren't currently present for PUD but we also have a custom + * PUD_PRESENT for a layer of future-proofing. + */ +#define PUD_PRESENT(pud) (pud_flags(pud) & _PAGE_PRESENT) +#define PMD_PRESENT(pmd) (pmd_flags(pmd) & _PAGE_PRESENT) +#define PTE_PRESENT(pte) (pte_flags(pte) & _PAGE_PRESENT) + +/* + * Check if an access by the kernel would cause a page fault. The access is + * described by a page fault error code (whether it was a write/instruction + * fetch) and address. This doesn't check for types of faults that are not + * expected to affect the kernel, e.g. PKU. The address can be user or kernel + * space, if user then we assume the access would happen via the uaccess API. + */ +static inline_or_noinstr int +kernel_access_ok(unsigned long error_code, unsigned long address, pgd_t *pgd) +{ + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + int ret; + if (!pgd_present(*pgd)) return 0; @@ -1006,27 +1048,27 @@ spurious_kernel_fault(unsigned long error_code, unsigned long address) return 0; if (p4d_leaf(*p4d)) - return spurious_kernel_fault_check(error_code, (pte_t *) p4d); + return kernel_protection_ok(error_code, (pte_t *) p4d); pud = pud_offset(p4d, address); - if (!pud_present(*pud)) + if (!PUD_PRESENT(*pud)) return 0; if (pud_leaf(*pud)) - return spurious_kernel_fault_check(error_code, (pte_t *) pud); + return kernel_protection_ok(error_code, (pte_t *) pud); pmd = pmd_offset(pud, address); - if (!pmd_present(*pmd)) + if (!PMD_PRESENT(*pmd)) return 0; if (pmd_leaf(*pmd)) - return spurious_kernel_fault_check(error_code, (pte_t *) pmd); + return kernel_protection_ok(error_code, (pte_t *) pmd); pte = pte_offset_kernel(pmd, address); - if (!pte_present(*pte)) + if (!PTE_PRESENT(*pte)) return 0; - ret = spurious_kernel_fault_check(error_code, pte); + ret = kernel_protection_ok(error_code, pte); if (!ret) return 0; @@ -1034,12 +1076,11 @@ spurious_kernel_fault(unsigned long error_code, unsigned long address) * Make sure we have permissions in PMD. * If not, then there's a bug in the page tables: */ - ret = spurious_kernel_fault_check(error_code, (pte_t *) pmd); + ret = kernel_protection_ok(error_code, (pte_t *) pmd); WARN_ONCE(!ret, "PMD has incorrect permission bits\n"); return ret; } -NOKPROBE_SYMBOL(spurious_kernel_fault); int show_unhandled_signals = 1; @@ -1483,6 +1524,29 @@ handle_page_fault(struct pt_regs *regs, unsigned long error_code, } } +static __always_inline void warn_if_bad_asi_pf( + unsigned long error_code, unsigned long address) +{ +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + struct asi *target; + + /* + * It's a bug to access sensitive data from the "critical section", i.e. + * on the path between asi_enter and asi_relax, where untrusted code + * gets run. #PF in this state sees asi_intr_nest_depth() as 1 because + * #PF increments it. We can't think of a better way to determine if + * this has happened than to check the ASI pagetables, hence we can't + * really have this check in non-debug builds unfortunately. + */ + VM_WARN_ONCE( + (target = asi_get_target(current)) != NULL && + asi_intr_nest_depth() == 1 && + !kernel_access_ok(error_code, address, asi_pgd(target)), + "ASI-sensitive data access from critical section, addr=%px error_code=%lx class=%s", + (void *) address, error_code, target->class->name); +#endif +} + DEFINE_IDTENTRY_RAW_ERRORCODE(exc_page_fault) { irqentry_state_t state; @@ -1490,6 +1554,31 @@ DEFINE_IDTENTRY_RAW_ERRORCODE(exc_page_fault) address = cpu_feature_enabled(X86_FEATURE_FRED) ? fred_event_data(regs) : read_cr2(); + if (static_asi_enabled() && !user_mode(regs)) { + pgd_t *pgd; + + /* Can be a NOP even for ASI faults, because of NMIs */ + asi_exit(); + + /* + * handle_page_fault() might oops if we run it for a kernel + * address. This might be the case if we got here due to an ASI + * fault. We avoid this case by checking whether the address is + * now, after asi_exit(), accessible by hardware. If it is - + * there's nothing to do. Note that this is a bit of a shotgun; + * we can also bail early from user-address faults here that + * weren't actually caused by ASI. So we might wanna move this + * logic later in the handler. In particular, we might be losing + * some stats here. However for now this keeps ASI page faults + * nice and fast. + */ + pgd = (pgd_t *)__va(read_cr3_pa()) + pgd_index(address); + if (kernel_access_ok(error_code, address, pgd)) { + warn_if_bad_asi_pf(error_code, address); + return; + } + } + prefetchw(¤t->mm->mmap_lock); /* From patchwork Fri Jul 12 17:00:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13731996 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE0FDC2BD09 for ; Fri, 12 Jul 2024 17:01:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A07FB6B00A8; Fri, 12 Jul 2024 13:01:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 987B86B00A9; Fri, 12 Jul 2024 13:01:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7653C6B00AA; Fri, 12 Jul 2024 13:01:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 441A16B00A8 for ; Fri, 12 Jul 2024 13:01:35 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 83433140BD1 for ; Fri, 12 Jul 2024 17:01:34 +0000 (UTC) X-FDA: 82331716908.10.DFC15FB Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) by imf05.hostedemail.com (Postfix) with ESMTP id CBFF3100041 for ; Fri, 12 Jul 2024 17:01:31 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=MkTFA6J2; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3aWGRZggKCKwVMOWYMZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--jackmanb.bounces.google.com designates 209.85.221.74 as permitted sender) smtp.mailfrom=3aWGRZggKCKwVMOWYMZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--jackmanb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803666; a=rsa-sha256; cv=none; b=X+kHu+yNGYySlCniJS2nGwRnm2MdAEide6N6c1qw3pfTeIZ9Sr+Na235EH+qQRxD+aAhgS 9UJqd1STgERUTG9XeSSg/VCZcIsJj/5t1NDjo1qGcb1ZqPKmVeYHUM/c6+doBQCkXGp6q6 amFSxfzDKjloRpEU77ZpXpEIJYyAJpA= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=MkTFA6J2; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3aWGRZggKCKwVMOWYMZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--jackmanb.bounces.google.com designates 209.85.221.74 as permitted sender) smtp.mailfrom=3aWGRZggKCKwVMOWYMZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--jackmanb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803666; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hLefymYBmXT6JRGt/rz6i03hx4/br+A6fWBCFOJPqqY=; b=RdKvKDo6dM9LonZWFyxFI6kJGw2h9P7fCZ2l2bTJ4l2vkwlnMm50RjpL0X6wUvycgRabLl qiHtanFu9eGsatnct5BJ0r3/IanZ2jPW3racC8BGYNzCMnnywGozuRinyQUZuRJ1V0Ydgc Mw7hrsvHXrKH90bfxsBXpmvCJdXWhtw= Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-367990b4beeso1258834f8f.2 for ; Fri, 12 Jul 2024 10:01:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803690; x=1721408490; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=hLefymYBmXT6JRGt/rz6i03hx4/br+A6fWBCFOJPqqY=; b=MkTFA6J2LTYe5X6lm3yWgqRgcfGAuM2Gn+coouOl34F35kcOCoDdBDQseze5Ybt4h3 D3PSU5CVk+ieDRz034Y5918bZwLVoNJcSk3LN2UAW1Dl4ejb2DuYG5bE5fbVl15s5ois mlz6Nr7+fPu6VHFmFdzTDW4YTB0bBSfOkMnSDVKAfhjBhroqap229PDmaPYGBomAL8B7 E1cUV/aI6MAu4BnGgXFIZZnPNvf76xJ58n3Uz81ATi0JBGKsgsoI/ipHxWC21G9FECbM KDYnZjkvvb2hj74IXrdMl1mhg5LK6qv/HzNXOmHCvyqjh2jt1+PWScMa6wgsKU4YqVNF mWRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803690; x=1721408490; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hLefymYBmXT6JRGt/rz6i03hx4/br+A6fWBCFOJPqqY=; b=E7I34iZ2qrciYM21nGSibPS7nTRH/Lnc1wdIw2x1uASbxCqGRDuzVyT2z9AOeiOtDV kOIoShNsAKrLbNGUCvYotf74ReFFrHN0yFCyfACP8PCnROfiQn8gsGmZ0eMFQcE7Y1Uy 7jhKcSPabU9/FBBTC2cVEDCikJ7hHe+es3TmCtoY/OJDU2HCCrwgdRn32tOKDmQAkDw/ mvkdBvj3cc8yxhnEx+W5ysppff7npiokkxcPZc/f2JVeazhsX5dTOLCDWtMRXMphAjSH xV0yLkjGOEmn8I1rkTNQ6F3aVHGxLe2HpLS0ws5R0vSZWymtDTCAkdT2cx9dEm5T8WL3 4zhQ== X-Forwarded-Encrypted: i=1; AJvYcCUw6onO6yEA6/owv+WTQ6TSqeHmMwhKbuOiRa1WgNy1UhVUopRC85L0E6b/ecYwOcuQ7cm3Np2LShoACZonG7NHw14= X-Gm-Message-State: AOJu0YzHJtXPpYkpRTrWr7FevLUFLM+ENub9fQdxQdLoU7UOrsgCYEnY UwDktJLJc5sAlVDcvxNWjl/1mF859iAagUny3zDRkGaqiyj5EvkTvKxs5C8BxRAGxpPYg/qy1Nf XneFyIwHTDQ== X-Google-Smtp-Source: AGHT+IGmoS0wFSTyq42rNb9s8Sa7gNvWrt/+iS9kYwcQcp6CEEUakdBFKNvwNThuXyaWVj/Pb47/G/lt1+tIwQ== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:adf:e692:0:b0:367:9b1f:c59b with SMTP id ffacd0b85a97d-367ceac4433mr15745f8f.9.1720803689929; Fri, 12 Jul 2024 10:01:29 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:31 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-13-144b319a40d8@google.com> Subject: [PATCH 13/26] mm: asi: Functions to map/unmap a memory range into ASI page tables From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Rspamd-Queue-Id: CBFF3100041 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: xqhf6u5prjihus1g5511b8oemw1xf7wh X-HE-Tag: 1720803691-569885 X-HE-Meta: U2FsdGVkX18GbFJkmTDamtuNPgi3m+Vpr5wWtkjNu63oVuJXMbuaA5A1OQgfdr5rzL2PB/96EKsCunbVfgsyYmk/44pDL95OYGtY5ycc4C32vQcxoifdcK83qRb9ipY8G2dnFf8u/dSESrdbBHb2kwPCAFQ6n3zyjdyPsSGTKP2mgrNrJwNGPZsHqFZaEkT3uZ27l+YnKe2DSYGrDzX9KSC+Mezbm7aPSVoVCaa0I9fV63aAp8U8LFHDggMvnp59ABRattR7Hl9jj4jHASS/rlHtViRzDKfy4o1wQ/zxHMjnfrV6O44YxWGyGQVkU4pDSXOc0PoOAj7LCXvTuHJA5Kgu19IX1CERIg96nnZ0Wsn7ZTUKLRNimfrdxXg2jcP3pRrAfwjSOAd3vckPZDzseTjUAdGI7BQRAokn6oT5tK81rHkoLv6X83MUHHiT3775PjY86diBmrkuOzOA/TmoNDDAxibP7NTvAd3QoRkXl9tG3Tl0mA8kg9MyrRecA4/3DCqD8Vir5w4ksfshqL7uXjbLOtpeAz66D3OjcHk2O7ZCIqsWjItN+t9JfEXJ2IGTG5uzf6YJugAWTm0KfVl2EgWjWIPykXbq644PJ2kLxzMU5hsPl2EKZunMh6bE6FRCZCOGzG4uUkNwQ3q4q3EPAR9lsW9PO5BaV7JwKtlJrHqWjhgEQKKcC/lBXA8ZFvBUzE1rpNBqIAa8FiHFtQeKE2cqMvu8btWJ4nXQzyDgtRvbrDEPOQrlh6rhOlgLQUkfasMNwhVjsYRHPMPBuqDaJcORtqJ/iL1QPQbQvkz12oQx/BwURu9HRFWhoZzhpJ0w8ypMzIBu5yOY05W8hExcwykRLVZXdvBxdNV0CuunA28pUmZzv+zrvwdxFMttp8ZxdjTtfAY4djFNfn/z4Idr3hKKja+n1I+ApPPShUsClQcFzq6AugMWbXCmDCvzyd1JooNM02iWn59rloXVJxJ 6go0Mwo3 2N0xjLs2IO5j9ZAvR3nMEwVXxQ/nNwDRmE+K2G45wD9eZnzagQTJeYYbHHhAz/4YAviY14ES85EQljJ8J7D2mG4ew2InQrN41kYpBatYxFWxQkPjEOfdbnGjVRZq7NMmgAQrjAseJTl5g+hjbqmLGzjp1BowjnYIqYE4LbYkpW6AU51bMXMJEvNjUmI88jDhJlATLMpaUhAQNaAOr46alQMdrYC9qhGq8oXDBwE3Q5KtDiXl6Tam7k261eRXHgjexBXaU/Wk5WTNLyKOYmTRysMAs3S9UpUWBiXUCMAVGplNpNZMPfNC1MYJU2TuIDAX+VGrq0mVVcaMCzjTEV6Ma614VuwCCG64CAegSzYWQ6R7sO8rWFNad6CV6U/EtE8qN5a2wMdS+WDQE+a9NZhME7cqxSIJgU/ffUhLBwHJzNEZ70RR3dsUyOVYUGhfi20scebRnNEtRPV5EZtDbZLoRPFrJDAcGHgd/zQ0+E8kGX2mxfivEYtEP2q/zVol+je93DUi3I8UaEi1heV6V/VgmCXqUC9sr+xKCHzpVPkfAjzRJapQSw8CA3P6PlzHiGXr4bGeKEWOc9Z0+gWcSy40xDFblNequc6j4hYpufoi5WRw+kQ0MaPk5zVeUXFaJnHV6T/8VUt7eR0i+60dBafhu2JtFUA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Junaid Shahid Two functions, asi_map() and asi_map_gfp(), are added to allow mapping memory into ASI page tables. The mapping will be identical to the one for the same virtual address in the unrestricted page tables. This is necessary to allow switching between the page tables at any arbitrary point in the kernel. Another function, asi_unmap() is added to allow unmapping memory mapped via asi_map* Signed-off-by: Junaid Shahid Signed-off-by: Brendan Jackman --- arch/x86/include/asm/asi.h | 5 + arch/x86/mm/asi.c | 238 ++++++++++++++++++++++++++++++++++++++++++++- arch/x86/mm/tlb.c | 5 + include/asm-generic/asi.h | 13 +++ include/linux/pgtable.h | 3 + mm/internal.h | 2 + mm/vmalloc.c | 32 +++--- 7 files changed, 284 insertions(+), 14 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index 1a19a925300c9..9aad843eb6dfa 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -135,6 +135,11 @@ void asi_relax(void); /* Immediately exit the restricted address space if in it */ void asi_exit(void); +int asi_map_gfp(struct asi *asi, void *addr, size_t len, gfp_t gfp_flags); +int asi_map(struct asi *asi, void *addr, size_t len); +void asi_unmap(struct asi *asi, void *addr, size_t len); +void asi_flush_tlb_range(struct asi *asi, void *addr, size_t len); + static inline void asi_init_thread_state(struct thread_struct *thread) { thread->asi_state.intr_nest_depth = 0; diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index 8798aab667489..e43b206450ad9 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -9,6 +9,9 @@ #include #include #include +#include + +#include "../../../mm/internal.h" static struct asi_class asi_class[ASI_MAX_NUM]; static DEFINE_SPINLOCK(asi_class_lock); @@ -98,7 +101,6 @@ EXPORT_SYMBOL_GPL(asi_unregister_class); */ static_assert(!IS_ENABLED(CONFIG_PARAVIRT)); #define DEFINE_ASI_PGTBL_ALLOC(base, level) \ -__maybe_unused \ static level##_t * asi_##level##_alloc(struct asi *asi, \ base##_t *base, ulong addr, \ gfp_t flags) \ @@ -338,3 +340,237 @@ void asi_init_mm_state(struct mm_struct *mm) memset(mm->asi, 0, sizeof(mm->asi)); mutex_init(&mm->asi_init_lock); } + +static bool is_page_within_range(unsigned long addr, unsigned long page_size, + unsigned long range_start, unsigned long range_end) +{ + unsigned long page_start = ALIGN_DOWN(addr, page_size); + unsigned long page_end = page_start + page_size; + + return page_start >= range_start && page_end <= range_end; +} + +static bool follow_physaddr( + pgd_t *pgd_table, unsigned long virt, + phys_addr_t *phys, unsigned long *page_size, ulong *flags) +{ + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + + /* This may be written using lookup_address_in_*, see kcl/675039. */ + + *page_size = PGDIR_SIZE; + pgd = pgd_offset_pgd(pgd_table, virt); + if (!pgd_present(*pgd)) + return false; + if (pgd_leaf(*pgd)) { + *phys = PFN_PHYS(pgd_pfn(*pgd)) | (virt & ~PGDIR_MASK); + *flags = pgd_flags(*pgd); + return true; + } + + *page_size = P4D_SIZE; + p4d = p4d_offset(pgd, virt); + if (!p4d_present(*p4d)) + return false; + if (p4d_leaf(*p4d)) { + *phys = PFN_PHYS(p4d_pfn(*p4d)) | (virt & ~P4D_MASK); + *flags = p4d_flags(*p4d); + return true; + } + + *page_size = PUD_SIZE; + pud = pud_offset(p4d, virt); + if (!pud_present(*pud)) + return false; + if (pud_leaf(*pud)) { + *phys = PFN_PHYS(pud_pfn(*pud)) | (virt & ~PUD_MASK); + *flags = pud_flags(*pud); + return true; + } + + *page_size = PMD_SIZE; + pmd = pmd_offset(pud, virt); + if (!pmd_present(*pmd)) + return false; + if (pmd_leaf(*pmd)) { + *phys = PFN_PHYS(pmd_pfn(*pmd)) | (virt & ~PMD_MASK); + *flags = pmd_flags(*pmd); + return true; + } + + *page_size = PAGE_SIZE; + pte = pte_offset_map(pmd, virt); + if (!pte) + return false; + + if (!pte_present(*pte)) { + pte_unmap(pte); + return false; + } + + *phys = PFN_PHYS(pte_pfn(*pte)) | (virt & ~PAGE_MASK); + *flags = pte_flags(*pte); + + pte_unmap(pte); + return true; +} + +/* + * Map the given range into the ASI page tables. The source of the mapping is + * the regular unrestricted page tables. Can be used to map any kernel memory. + * + * The caller MUST ensure that the source mapping will not change during this + * function. For dynamic kernel memory, this is generally ensured by mapping the + * memory within the allocator. + * + * If this fails, it may leave partial mappings behind. You must asi_unmap them, + * bearing in mind asi_unmap's requirements on the calling context. Part of the + * reason for this is that we don't want to unexpectedly undo mappings that + * weren't created by the present caller. + * + * If the source mapping is a large page and the range being mapped spans the + * entire large page, then it will be mapped as a large page in the ASI page + * tables too. If the range does not span the entire huge page, then it will be + * mapped as smaller pages. In that case, the implementation is slightly + * inefficient, as it will walk the source page tables again for each small + * destination page, but that should be ok for now, as usually in such cases, + * the range would consist of a small-ish number of pages. + * + * Note that upstream + * (https://lore.kernel.org/all/20210317155843.c15e71f966f1e4da508dea04@linux-foundation.org/) + * vmap_p4d_range supports huge mappings. It is probably possible to use that + * logic instead of custom mapping duplication logic in later versions of ASI. + */ +int __must_check asi_map_gfp(struct asi *asi, void *addr, unsigned long len, gfp_t gfp_flags) +{ + unsigned long virt; + unsigned long start = (size_t)addr; + unsigned long end = start + len; + unsigned long page_size; + + if (!static_asi_enabled()) + return 0; + + VM_BUG_ON(!IS_ALIGNED(start, PAGE_SIZE)); + VM_BUG_ON(!IS_ALIGNED(len, PAGE_SIZE)); + VM_BUG_ON(!fault_in_kernel_space(start)); /* Misnamed, ignore "fault_" */ + + gfp_flags &= GFP_RECLAIM_MASK; + + if (asi->mm != &init_mm) + gfp_flags |= __GFP_ACCOUNT; + + for (virt = start; virt < end; virt = ALIGN(virt + 1, page_size)) { + pgd_t *pgd; + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + pte_t *pte; + phys_addr_t phys; + ulong flags; + + if (!follow_physaddr(asi->mm->pgd, virt, &phys, &page_size, &flags)) + continue; + +#define MAP_AT_LEVEL(base, BASE, level, LEVEL) { \ + if (base##_leaf(*base)) { \ + if (WARN_ON_ONCE(PHYS_PFN(phys & BASE##_MASK) !=\ + base##_pfn(*base))) \ + return -EBUSY; \ + continue; \ + } \ + \ + level = asi_##level##_alloc(asi, base, virt, gfp_flags);\ + if (!level) \ + return -ENOMEM; \ + \ + if (page_size >= LEVEL##_SIZE && \ + (level##_none(*level) || level##_leaf(*level)) && \ + is_page_within_range(virt, LEVEL##_SIZE, \ + start, end)) { \ + page_size = LEVEL##_SIZE; \ + phys &= LEVEL##_MASK; \ + \ + if (!level##_none(*level)) { \ + if (WARN_ON_ONCE(level##_pfn(*level) != \ + PHYS_PFN(phys))) { \ + return -EBUSY; \ + } \ + } else { \ + set_##level(level, \ + __##level(phys | flags)); \ + } \ + continue; \ + } \ + } + + pgd = pgd_offset_pgd(asi->pgd, virt); + + MAP_AT_LEVEL(pgd, PGDIR, p4d, P4D); + MAP_AT_LEVEL(p4d, P4D, pud, PUD); + MAP_AT_LEVEL(pud, PUD, pmd, PMD); + /* + * If a large page is going to be partially mapped + * in 4k pages, convert the PSE/PAT bits. + */ + if (page_size >= PMD_SIZE) + flags = protval_large_2_4k(flags); + MAP_AT_LEVEL(pmd, PMD, pte, PAGE); + + VM_BUG_ON(true); /* Should never reach here. */ + } + + return 0; +#undef MAP_AT_LEVEL +} + +int __must_check asi_map(struct asi *asi, void *addr, unsigned long len) +{ + return asi_map_gfp(asi, addr, len, GFP_KERNEL); +} + +/* + * Unmap a kernel address range previously mapped into the ASI page tables. + * + * The area being unmapped must be a whole previously mapped region (or regions) + * Unmapping a partial subset of a previously mapped region is not supported. + * That will work, but may end up unmapping more than what was asked for, if + * the mapping contained huge pages. A later patch will remove this limitation + * by splitting the huge mapping in the ASI page table in such a case. For now, + * vunmap_pgd_range() will just emit a warning if this situation is detected. + * + * This might sleep, and cannot be called with interrupts disabled. + */ +void asi_unmap(struct asi *asi, void *addr, size_t len) +{ + size_t start = (size_t)addr; + size_t end = start + len; + pgtbl_mod_mask mask = 0; + + if (!static_asi_enabled() || !len) + return; + + VM_BUG_ON(start & ~PAGE_MASK); + VM_BUG_ON(len & ~PAGE_MASK); + VM_BUG_ON(!fault_in_kernel_space(start)); /* Misnamed, ignore "fault_" */ + + vunmap_pgd_range(asi->pgd, start, end, &mask); + + /* We don't support partial unmappings - b/270310049 */ + if (mask & PGTBL_P4D_MODIFIED) { + VM_WARN_ON(!IS_ALIGNED((ulong)addr, P4D_SIZE)); + VM_WARN_ON(!IS_ALIGNED((ulong)len, P4D_SIZE)); + } else if (mask & PGTBL_PUD_MODIFIED) { + VM_WARN_ON(!IS_ALIGNED((ulong)addr, PUD_SIZE)); + VM_WARN_ON(!IS_ALIGNED((ulong)len, PUD_SIZE)); + } else if (mask & PGTBL_PMD_MODIFIED) { + VM_WARN_ON(!IS_ALIGNED((ulong)addr, PMD_SIZE)); + VM_WARN_ON(!IS_ALIGNED((ulong)len, PMD_SIZE)); + } + + asi_flush_tlb_range(asi, addr, len); +} diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index e80cd67a5239e..36087d6238e6f 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1026,6 +1026,11 @@ inline_or_noinstr u16 asi_pcid(struct asi *asi, u16 asid) return kern_pcid(asid) | ((asi->index + 1) << ASI_PCID_BITS_SHIFT); } +void asi_flush_tlb_range(struct asi *asi, void *addr, size_t len) +{ + flush_tlb_kernel_range((ulong)addr, (ulong)addr + len); +} + #else /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ u16 asi_pcid(struct asi *asi, u16 asid) { return kern_pcid(asid); } diff --git a/include/asm-generic/asi.h b/include/asm-generic/asi.h index fa0bbf899a094..3956f995fe6a1 100644 --- a/include/asm-generic/asi.h +++ b/include/asm-generic/asi.h @@ -2,6 +2,8 @@ #ifndef __ASM_GENERIC_ASI_H #define __ASM_GENERIC_ASI_H +#include + #ifndef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION #define ASI_MAX_NUM_ORDER 0 @@ -58,6 +60,17 @@ static inline int asi_intr_nest_depth(void) { return 0; } static inline void asi_intr_exit(void) { } +static inline int asi_map(struct asi *asi, void *addr, size_t len) +{ + return 0; +} + +static inline +void asi_unmap(struct asi *asi, void *addr, size_t len) { } + +static inline +void asi_flush_tlb_range(struct asi *asi, void *addr, size_t len) { } + #define static_asi_enabled() false static inline void asi_check_boottime_disable(void) { } diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 85fc7554cd52b..4884dfc6e699b 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1788,6 +1788,9 @@ typedef unsigned int pgtbl_mod_mask; #ifndef pmd_leaf #define pmd_leaf(x) false #endif +#ifndef pte_leaf +#define pte_leaf(x) 1 +#endif #ifndef pgd_leaf_size #define pgd_leaf_size(x) (1ULL << PGDIR_SHIFT) diff --git a/mm/internal.h b/mm/internal.h index 07ad2675a88b4..8a8f98e119dfa 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -217,6 +217,8 @@ void unmap_page_range(struct mmu_gather *tlb, void page_cache_ra_order(struct readahead_control *, struct file_ra_state *, unsigned int order); void force_page_cache_ra(struct readahead_control *, unsigned long nr); +void vunmap_pgd_range(pgd_t *pgd_table, unsigned long addr, unsigned long end, + pgtbl_mod_mask *mask); static inline void force_page_cache_readahead(struct address_space *mapping, struct file *file, pgoff_t index, unsigned long nr_to_read) { diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 125427cbdb87b..7a8daf5afb7cc 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -419,6 +419,24 @@ static void vunmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, } while (p4d++, addr = next, addr != end); } +void vunmap_pgd_range(pgd_t *pgd_table, unsigned long addr, unsigned long end, + pgtbl_mod_mask *mask) +{ + unsigned long next; + pgd_t *pgd = pgd_offset_pgd(pgd_table, addr); + + BUG_ON(addr >= end); + + do { + next = pgd_addr_end(addr, end); + if (pgd_bad(*pgd)) + *mask |= PGTBL_PGD_MODIFIED; + if (pgd_none_or_clear_bad(pgd)) + continue; + vunmap_p4d_range(pgd, addr, next, mask); + } while (pgd++, addr = next, addr != end); +} + /* * vunmap_range_noflush is similar to vunmap_range, but does not * flush caches or TLBs. @@ -433,21 +451,9 @@ static void vunmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, */ void __vunmap_range_noflush(unsigned long start, unsigned long end) { - unsigned long next; - pgd_t *pgd; - unsigned long addr = start; pgtbl_mod_mask mask = 0; - BUG_ON(addr >= end); - pgd = pgd_offset_k(addr); - do { - next = pgd_addr_end(addr, end); - if (pgd_bad(*pgd)) - mask |= PGTBL_PGD_MODIFIED; - if (pgd_none_or_clear_bad(pgd)) - continue; - vunmap_p4d_range(pgd, addr, next, &mask); - } while (pgd++, addr = next, addr != end); + vunmap_pgd_range(init_mm.pgd, start, end, &mask); if (mask & ARCH_PAGE_TABLE_SYNC_MASK) arch_sync_kernel_mappings(start, end); From patchwork Fri Jul 12 17:00:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13731997 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77AF2C3DA45 for ; Fri, 12 Jul 2024 17:01:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 73EDC6B00A9; Fri, 12 Jul 2024 13:01:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6C64F6B00AA; Fri, 12 Jul 2024 13:01:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4FDED6B00AB; Fri, 12 Jul 2024 13:01:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 2B3986B00A9 for ; Fri, 12 Jul 2024 13:01:37 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id AA52A1A0C61 for ; Fri, 12 Jul 2024 17:01:36 +0000 (UTC) X-FDA: 82331716992.15.F30AA3F Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf14.hostedemail.com (Postfix) with ESMTP id 20B7C100004 for ; Fri, 12 Jul 2024 17:01:33 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=jMKbYJR2; spf=pass (imf14.hostedemail.com: domain of 3bWGRZggKCLAZQSacQdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--jackmanb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3bWGRZggKCLAZQSacQdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803659; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3PoEMkqequy1KWINMIJVW/1gdvscCRasMo3UATAb/nU=; b=kRgHdHcRvnShGU8etMpOo6OxpCZRr4JIT/1GWDnL7XpRUqiTTPfoDU3psR0D7mroN14R2G 7THPKHhPYXh5R/88VB3xPIgqlL01jrdmwluUwOyPVFTXdWJ1HsvdiPxU6K3+JQKy9j9drY b2ezUdqjhTwGYwm0cCROJkYw7VtsQSg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803659; a=rsa-sha256; cv=none; b=GtWD3iHi2M5fOrLB8KMqx5LKB+CkSD/Bh3tMX13AZwlUpce//69za/XngeEa+CrIKG20Wy DU8rfSbGFfhP5P7NyITpgqd9jRPjz6Kd2wpgLc0D4ylZ8gmQOZh2BH9A428+P3tv5B7TKX AgFv8To4DGAMJBq9uFH7YxScYKFYW7w= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=jMKbYJR2; spf=pass (imf14.hostedemail.com: domain of 3bWGRZggKCLAZQSacQdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--jackmanb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3bWGRZggKCLAZQSacQdRWeeWbU.SecbYdkn-ccalQSa.ehW@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-65194ea3d4dso35987957b3.0 for ; Fri, 12 Jul 2024 10:01:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803693; x=1721408493; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=3PoEMkqequy1KWINMIJVW/1gdvscCRasMo3UATAb/nU=; b=jMKbYJR2TFaMLSP6guMDVo9PdnQGFZnlw5UMcFOCweaP3uxGkNKap905XJaIJ26a0+ Ia23fcbNH3tlx9LDVj03Pk8q/TrpmhUoOeF79qdpNDM/PkohDtTOWSPMHXshoibsQ16O Vt6fR+T+/doplzKiQjLyCMkP0KT99xHUFbxUZ523F/tbanONJwb5oa2A4JuxMJVftvqO Xg4JUzdMP4As6J3tnkxq8F7e/CD1BGqORy/uc0yCoP5TLdvOzaKkTni/njRgVULNOedT MiF071EWw6fxqa4NCS+wZ993SvS+OsU6s8jo+CiJy4pa4i1itj2VEYW66I9F7XFIqJ72 9owg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803693; x=1721408493; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3PoEMkqequy1KWINMIJVW/1gdvscCRasMo3UATAb/nU=; b=swRVGjZeur4S8OG03/+D/DmATJPKDLX3RGeMQU/IDadUaRUCdFSN/o3xn3Bpusa2ZN 4I3VfCNk42DWtxaFScm+YXa1lma6p/Tlpcr/pcXL/S/F9x8dFVOybrPrawTIjA9xnrRM Dob6oKnvnxtR3Yx7bmqP6E/AbldUpyLMV9P42poGv4TS+L5/+dvvB3lXEGLr48lRV4B7 sgEPUPW+YouXpDhkut4Cu5FhqlBcgQTIczmya//yyedvkbF+IOKWSthainYGeY+B+rQx wwleVm7saSRQLsH3EUeEt75xRVjBgwGBrWUFfvT9Nkp5keUu6O5UDaAJjwn/k5q6bBuA 3xNg== X-Forwarded-Encrypted: i=1; AJvYcCUSeb/MDe204PRgisb/lMJxbjT4AeGc1FpZfsAaemj4VqUhKnWaTLltweIlbIAAqlmrNp5cs2wUHfQpxzWk4ReZgG4= X-Gm-Message-State: AOJu0YyTIX3fb1Sy78eWCPgSAUroqutT+fzZ69DMAMsDoDZ8NAwgcQnT FHUn7uXFA+ND2XewbkNe170DgUNjMf6JB+fMwulllW8AsnzkvTl2i1gz2bSEM0b11brqCbUD0cL uUzocdRzm3g== X-Google-Smtp-Source: AGHT+IGAsyCLoTSPrJ2N91yIZy7N9XYfi0+vS82VtwZTogujGpm4xok4sUQDdphNDQnjhHnIeN3tEJvaUVoySQ== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:690c:488a:b0:62c:ea0b:a447 with SMTP id 00721157ae682-658ee69b8aemr3280537b3.2.1720803693133; Fri, 12 Jul 2024 10:01:33 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:32 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-14-144b319a40d8@google.com> Subject: [PATCH 14/26] mm: asi: Add basic infrastructure for global non-sensitive mappings From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Stat-Signature: kryfswr7oiinwahwp96rhfkjf5cxqjtc X-Rspamd-Queue-Id: 20B7C100004 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1720803693-488382 X-HE-Meta: U2FsdGVkX1/By7RQwpBDL4Yfexa1vqelsBxAjZd7WiAYWWl83uvBq1TExNqGSUCQyPzxCoCmVT69AOgdGBWWQmJwuqubEZflOOgtT6ErXhxA2mebbElmx8NJFYc327lPMgQEhAcj8dzIBNU0ZkiHe9/f5x2WkGSkw6bN6Z1rqO/ECtdaojvRsHVx3oq1Mkumk7YakrU5w+edFH0ph1oK/gIFzITB25Eu8PU9JF3vpVVvvUW7TOyfrkiDdCksWXFFdyTzJuDGpSIRzMTMKzGZGN+WDBZ9BgJTB5Yzi2MBVfSLRR1Q/yWdzT97UFXD3CbS2VokDUxYTr/5vHDp3BZckscvHfNYqEQIfpDb2cIiPJwFXwtW+0fJWgNeHjks9ZZd2bzczCeqPC1yYyQdUpq4cJ1y7A+Ay4rMZdGzNlYASCpQufSvpOkzikXT/O0miitRr5TPf+6WbVChDTwzr/iFEGTOJTwberiEXlau+HUi05B/iGKaVxle9vyITwDxU5B1ensDbXCCbm/vTnyiW4Co4z83hrh9cLa0t0RDegY6FlXkoLNlpmURO18tZno4oAcBRCyg/CD+v/VxLjQEWzqseRlaby22ykAkNHbF9BadF1+XVTESnzGjCmgNLKjBya1QXDpNlhqcuui61WDas7yXvusbdI3+8e9md/HK4VYm9UekUjiFFffK6UF3EsQU0fwcH8t1Q+zEIEMUZAqEUGvtO/v71B5OiGa4HL9ho1yWTPxvm1+7aYsTRgfXS/P6OJVqPPvpHYhnbH01nXTIF6XZSIPDAQUZVFIItd0WwfYA63/7koW0Pps6fzdKBUKDUor3Z+pgHdTUIne4lHln5OCmIsIdwhSPv7eb/VBOsNlEru4iD1XjfzfcG9L3rg8YOSxsNTH0O4Mg/waDVrshzeUOAaHaN6WMv5ddMEyi/r55moaq8V56qFJPegF/JISzxP4sca+T7e2kk2yoNWh3uoX zWfuZ/yo inKIP1h3sKzpeTw3mxVaoa81hEgKsnEUJ2ZU4XRMQm5nR7lTwHQEezhbkUCVNMG5zOrPi6gz9WcUkfSn8YP6fS4RRQzjqIqJ9WkjoKSsGrM2d8UO+Z+hgiUzSBpNDL8LUn5e5o4Tamaf1tl2iPeltiG33LQlgRQjqqlhPEU0eosQev86+2f2ddceUJk6Ib1pkNenwZmoxV12hD4tFRs9XEsCxoK4+ul6AgNMK+miBprleBXPLbBz5QjmuHNFR3cL5y5Uf+9UIF3K4SKV+/FAWNl7aSBpw3Zdd30AYY8bpMHFMCkvT4I8jBJ98ieDEew+yQzSRrrpAeot8fH9PZ7Mr6egMetBwo/rQwbT2EKapw6X3lGZ5ovVPxECHtB4M+TwHl/5GxjJirfLok6QS3zAlkUCHHZRz/oPsexBahPjDREzVw6ZqpxR/AwKBSEP+uDo/SSBMJIdUPJ3yhl7psUHicz3xosRvRbzVCfExIiQs5UVUV7pLbeCV0OkuyvveY2CQXUUf82Y2npwcICZG/gF6sfxJMwkwro53ITy6 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Junaid Shahid A pseudo-PGD is added to store global non-sensitive ASI mappings. Actual ASI PGDs copy entries from this pseudo-PGD during asi_init(). Memory can be mapped as globally non-sensitive by calling asi_map() with ASI_GLOBAL_NONSENSITIVE. Page tables allocated for global non-sensitive mappings are never freed. While a previous version used init_mm.asi[0] as the special global nonsensitive domain, here we have tried to avoid special-casing index 0. So now we have a special global variable for that. For this to work we need to make sure that nobody assumes that asi is a member of asi->mm->asi (also that nobody assumes a struct asi is embedded in a struct mm - but that seems like a weird assumption to make anyway, when you already have the .mm pointer). I currently believe that this is worth it for the reduced level of magic in the code. Signed-off-by: Junaid Shahid Signed-off-by: Brendan Jackman --- arch/x86/include/asm/asi.h | 3 +++ arch/x86/mm/asi.c | 37 +++++++++++++++++++++++++++++++++++++ arch/x86/mm/init_64.c | 25 ++++++++++++++++--------- arch/x86/mm/mm_internal.h | 3 +++ include/asm-generic/asi.h | 2 ++ 5 files changed, 61 insertions(+), 9 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index 9aad843eb6df..2d86a5c17f2b 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -78,6 +78,9 @@ */ #define ASI_MAX_NUM ((1 << ASI_MAX_NUM_ORDER) - 1) +extern struct asi __asi_global_nonsensitive; +#define ASI_GLOBAL_NONSENSITIVE (&__asi_global_nonsensitive) + struct asi_hooks { /* * Both of these functions MUST be idempotent and re-entrant. They will diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index e43b206450ad..807d51497f43 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -11,6 +11,7 @@ #include #include +#include "mm_internal.h" #include "../../../mm/internal.h" static struct asi_class asi_class[ASI_MAX_NUM]; @@ -19,6 +20,13 @@ static DEFINE_SPINLOCK(asi_class_lock); DEFINE_PER_CPU_ALIGNED(struct asi *, curr_asi); EXPORT_SYMBOL(curr_asi); +static __aligned(PAGE_SIZE) pgd_t asi_global_nonsensitive_pgd[PTRS_PER_PGD]; + +struct asi __asi_global_nonsensitive = { + .pgd = asi_global_nonsensitive_pgd, + .mm = &init_mm, +}; + static inline bool asi_class_registered(int index) { return asi_class[index].name != NULL; @@ -154,6 +162,31 @@ void __init asi_check_boottime_disable(void) pr_info("ASI enablement ignored due to incomplete implementation.\n"); } +static int __init asi_global_init(void) +{ + if (!boot_cpu_has(X86_FEATURE_ASI)) + return 0; + + /* + * Lower-level pagetables for global nonsensitive mappings are shared, + * but the PGD has to be copied into each domain during asi_init. To + * avoid needing to synchronize new mappings into pre-existing domains + * we just pre-allocate all of the relevant level N-1 entries so that + * the global nonsensitive PGD already has pointers that can be copied + * when new domains get asi_init()ed. + */ + preallocate_sub_pgd_pages(asi_global_nonsensitive_pgd, + PAGE_OFFSET, + PAGE_OFFSET + PFN_PHYS(max_pfn) - 1, + "ASI Global Non-sensitive direct map"); + preallocate_sub_pgd_pages(asi_global_nonsensitive_pgd, + VMALLOC_START, VMALLOC_END, + "ASI Global Non-sensitive vmalloc"); + + return 0; +} +subsys_initcall(asi_global_init) + static void __asi_destroy(struct asi *asi) { WARN_ON_ONCE(asi->ref_count <= 0); @@ -168,6 +201,7 @@ int asi_init(struct mm_struct *mm, int asi_index, struct asi **out_asi) { struct asi *asi; int err = 0; + uint i; *out_asi = NULL; @@ -203,6 +237,9 @@ int asi_init(struct mm_struct *mm, int asi_index, struct asi **out_asi) asi->mm = mm; asi->index = asi_index; + for (i = KERNEL_PGD_BOUNDARY; i < PTRS_PER_PGD; i++) + set_pgd(asi->pgd + i, asi_global_nonsensitive_pgd[i]); + exit_unlock: if (err) __asi_destroy(asi); diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 7e177856ee4f..f67f4637357c 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -1278,18 +1278,15 @@ static void __init register_page_bootmem_info(void) #endif } -/* - * Pre-allocates page-table pages for the vmalloc area in the kernel page-table. - * Only the level which needs to be synchronized between all page-tables is - * allocated because the synchronization can be expensive. - */ -static void __init preallocate_vmalloc_pages(void) +/* Initialize empty pagetables at the level below PGD. */ +void __init preallocate_sub_pgd_pages(pgd_t *pgd_table, ulong start, + ulong end, const char *name) { unsigned long addr; const char *lvl; - for (addr = VMALLOC_START; addr <= VMEMORY_END; addr = ALIGN(addr + 1, PGDIR_SIZE)) { - pgd_t *pgd = pgd_offset_k(addr); + for (addr = start; addr <= end; addr = ALIGN(addr + 1, PGDIR_SIZE)) { + pgd_t *pgd = pgd_offset_pgd(pgd_table, addr); p4d_t *p4d; pud_t *pud; @@ -1325,7 +1322,17 @@ static void __init preallocate_vmalloc_pages(void) * The pages have to be there now or they will be missing in * process page-tables later. */ - panic("Failed to pre-allocate %s pages for vmalloc area\n", lvl); + panic("Failed to pre-allocate %s pages for %s area\n", lvl, name); +} + +/* + * Pre-allocates page-table pages for the vmalloc area in the kernel page-table. + * Only the level which needs to be synchronized between all page-tables is + * allocated because the synchronization can be expensive. + */ +static void __init preallocate_vmalloc_pages(void) +{ + preallocate_sub_pgd_pages(init_mm.pgd, VMALLOC_START, VMEMORY_END, "vmalloc"); } void __init mem_init(void) diff --git a/arch/x86/mm/mm_internal.h b/arch/x86/mm/mm_internal.h index 3f37b5c80bb3..1203a977edcd 100644 --- a/arch/x86/mm/mm_internal.h +++ b/arch/x86/mm/mm_internal.h @@ -25,4 +25,7 @@ void update_cache_mode_entry(unsigned entry, enum page_cache_mode cache); extern unsigned long tlb_single_page_flush_ceiling; +extern void preallocate_sub_pgd_pages(pgd_t *pgd_table, ulong start, + ulong end, const char *name); + #endif /* __X86_MM_INTERNAL_H */ diff --git a/include/asm-generic/asi.h b/include/asm-generic/asi.h index 3956f995fe6a..fd5a302e0e09 100644 --- a/include/asm-generic/asi.h +++ b/include/asm-generic/asi.h @@ -9,6 +9,8 @@ #define ASI_MAX_NUM_ORDER 0 #define ASI_MAX_NUM 0 +#define ASI_GLOBAL_NONSENSITIVE NULL + #ifndef _ASSEMBLY_ struct asi_hooks {}; From patchwork Fri Jul 12 17:00:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13731998 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12BA7C2BD09 for ; Fri, 12 Jul 2024 17:01:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EB7A76B00AC; Fri, 12 Jul 2024 13:01:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E40CC6B00AD; Fri, 12 Jul 2024 13:01:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C93926B00AE; Fri, 12 Jul 2024 13:01:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A543D6B00AC for ; Fri, 12 Jul 2024 13:01:39 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 4316F40C2B for ; Fri, 12 Jul 2024 17:01:39 +0000 (UTC) X-FDA: 82331717118.15.49AE7DE Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) by imf09.hostedemail.com (Postfix) with ESMTP id 5489A14001F for ; Fri, 12 Jul 2024 17:01:36 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1fbbXO6u; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of 3b2GRZggKCLIbSUceSfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--jackmanb.bounces.google.com designates 209.85.221.73 as permitted sender) smtp.mailfrom=3b2GRZggKCLIbSUceSfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--jackmanb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803659; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wVrvdlJuWkiD+fWXDk/FgNJ7m+hI0zlwjZ1uzkGuBuA=; b=f39T2HDa/EwJ3KjlwNftOLLYPgOK0QyEMxQ8iqP0hdkS/IrTnluBi8R+7yeyzV0Ox8k2nS IITZTibHxxs2GPOQ5zgpDzR5KhZF7n8aSrUc/jPd385Qz4aO5aHop5IO4soDh0dZWHMtkW shMrrAMotDKXkLcULaeVuI7JOcdrI7g= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803659; a=rsa-sha256; cv=none; b=x2BAVWPQAylvqkmvVWRT5TderBktCYyvF6CmEHKfpftjzIxhM31/wajNd5d/Mc9G3J/RY7 DsYpm1cuk8tn9J2D+DvmA4vpQ7U4brb2UQKBgUpEur8z0zelGmzgjZG6VaRYEzhjWZugm5 oQE9LW0Zom6HWniJ1iINMgB+j2v8WlE= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1fbbXO6u; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of 3b2GRZggKCLIbSUceSfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--jackmanb.bounces.google.com designates 209.85.221.73 as permitted sender) smtp.mailfrom=3b2GRZggKCLIbSUceSfTYggYdW.Ugedafmp-eecnSUc.gjY@flex--jackmanb.bounces.google.com Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-3678e523e32so1957629f8f.1 for ; Fri, 12 Jul 2024 10:01:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803696; x=1721408496; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wVrvdlJuWkiD+fWXDk/FgNJ7m+hI0zlwjZ1uzkGuBuA=; b=1fbbXO6uLHFE/vZBNuRmvCy883MsXJu26U1qL47iPXGv04L3FjTg0Me/I6UPU3b5lx w6WiyGhU1M0QsUR/9Sjsjz5XSijZSbCNn7u2gXdZRlSwL4TxCgizgqGvtaL1bCW/+/aq SbKPW0ZjJyTYGjU0oG1+Mi2bjfFq/6XdwH8kDOhl9C2G3JnWm0PrV5DRIdYcxfNeYE8w unZpVW2F53PIDZedC3A2YOr1migTb2BgCJ672dFtN2d67epPsEUBRfF6BMW9Y5BmQJ1S EfUyxc7+2TY0lDvw6fyBlkFc7gMXkv5+OE1zLCsYU9pOrW50wFiYW7la+0BQ26hO9iQ7 2n+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803696; x=1721408496; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wVrvdlJuWkiD+fWXDk/FgNJ7m+hI0zlwjZ1uzkGuBuA=; b=fHAWx22HjF1wnnHinc5iD4kjLSFK2zQU+8v8wOit6iraPsb/H9j/FKoaQnPcadoFpF nIMu4zP4FKwmv2tk7sbMdHsIJvwOwps9b28g59A8GNrO+RBSrYtZa6cr0uqn9SPZAKdf tMHAqYKUCZTMXqJaBrULz5KFp/Fq9O8hcd5Rt1rvVhydwP+35a3bGV4QVEfociOj2ChE vcEO6RNlEKl5PC2cGPmC/uU1rZS3hSB4P+1oMnrl6FgHTR/qlvRU/Gm1bjkm2jTu+psw 0o/tlx7e/EZq7jWjV+Y3owUwOn8WDxs0UUemD9JaxT94E67ySx/UK5iLH2cMwzg39kjJ 808g== X-Forwarded-Encrypted: i=1; AJvYcCXbD7H5kyTVoz5lny/cboUmcLRgJHCNlhpYtdKkPzrg3mHltAnJZLLM5uuTGXQErYBbUTmV1uYKv/fJRj0si7ap+QY= X-Gm-Message-State: AOJu0YxhZLa1GApnxlp737hbapEFeVrysv70bgc3ZtAUByql++P8INJ6 vpdWnlmiz8bxPQkkj0wkyo5yfO8Z7SfA0JAro5lUVxx58Oy7T8w8hxYE77+dRm4d+vZxhJkwG4/ xqkbZjcNc7w== X-Google-Smtp-Source: AGHT+IErU4rdmgQiGRDTef2HJjYT1+igKTiETpi6GcIgcGxVKuf8GnSM5HDh2rVJ+wSRQZ9KiIWprZFuMKIHlA== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:6000:400c:b0:368:5d2:9e5f with SMTP id ffacd0b85a97d-36805d29fa3mr5077f8f.0.1720803695788; Fri, 12 Jul 2024 10:01:35 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:33 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-15-144b319a40d8@google.com> Subject: [PATCH 15/26] mm: Add __PAGEFLAG_FALSE From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 5489A14001F X-Stat-Signature: udn4mwd5wir818jjzmqdh5hchruhsgpq X-Rspam-User: X-HE-Tag: 1720803696-725516 X-HE-Meta: U2FsdGVkX18pRAvW3RJuq6A0xbyfkogFm3Y6d2tt+lXFGRxbESsJnQiq3CCwhtv2+njeeci7/zMpI0o2VMMuzUUfD8o/DqEL2r7D5zaQ24CHxe+L1MjaXXXAg+Xn/xC+PrvpiEwnHYqHrx6RJUEq22gdERrrL9dSsTUUhfwD8l2mQlJ3CkH7bLHQ3Mg+1rUUpPESJua7l96KcShrXxTX0eS3mpuqcLFAS1gnjMCJvQwCBOKG06uUJ3NUNGunZ6h5hCzl9lPlh06uL7/eUoiTXICs/9tfuykp3+sG7ijc/hF9CjOBZU3CmeQ6LgYIDlbEzqP8KYSJMc+tUEKSMZc3wUWvAjIbjxazoU7F81JOwgU8KrqKK1BG/YlaaWtiPFck3mb/aQkvUTEDX7RaTLIMILC6RLBI7T9JP+twkLqiHJgwLryboURrQzEc6CFgSR5OdRbxohqc3YO6Dz1QZ1nAukfAxKgfONZa0XraRJErCqLHK5k36I67ayv2gEy9KToivXFNE3DjzQKpwNRbcsog4R8Xh8NILQ8XUdMvJXd11gLHDhdaAhmmYMniedvPWW4UGTb3Hcj6Jr0I/oZJY9FHQncB0P6xUOkBtu+QpWZM4ocVzudjp0VL+oNl9mcPg66fXDI1sMq9gK5cDUK4lEjmg/t2YSL4gkKEamZ35PtdV7SxbX4LJ1c979sNhI9zkJWRrCq2omT23mHI6SvxDzR3YsDh1EHLU6TjB1mz6KImXYuXuioCw52EJNaoR40aD/8DkUr4F3HwYiuEUH0ECulnpri//b5czSSCQieS8kFrTGRGK1pI8R/Kn86Td1blClxrER5tUY7wSKW0Xb/b0YQc7cBzocn1Fc16SQZJnjQAf04RMJb0/lcD3Dbz4bWhbaeLAWdO5OEstzvJjscSJ4q1tdZ0UVdgSLMvanEi1a/kqb1YM+sCYVAraGMxpi+CxodVrj1FRYs9xVHuMrVRRsn Hg1NLShk 5WFBGyLVMk8SceeIOg+QnXx/HLY4VHo2Siuzg8SU9t2c8yi3ln9zeWKDI97D0C3QDmTS5xDhXwNqZ/soKAyjMUP9tE1D7S2c+WkBmjbvUmBVxtCBOVHZ6KZdEdb8ZZeL5cG/GntyZkCqatYapieeZcm17hhefv7eRoED9D8PCUW4VyHYVgyVSih94jCh4U5C2osfXzC/inhZK1JVuVQ8GCZG6DIzQ/C9RqdX9TkTlxVlALotmJzlQYGCD60iTpmGOYES6JF4J+qAdmDquSwdlb8zv+84tqjWNY2Sb4MYOiJCUzcI6j3IFXK7kBetoJVhh1zrxD6NZ3+z4Uf9JLFqzpqzXx7pM+/KQUcJtHCFfgcQPwE1MtEnbCU1dy2ueQ5/7nt7VWTQlKX2lVRdU3CTSRoUzXdhMoW0H+X6RpXmWiiALzTmcCx1KlYtY75GJzaActxXDUIyoZ2rWh3He8hLhIEENlElzK1G+OIlxyY5th7UfcLqZJZW1EDXpf3acYF+OUYdo/3AuWrIeXiffy1Zpkx1B99BpZGqPG5FmBuAjLKvrKhWVV/OIxBv38wRULdCIy08IHgbGvbFhS/c= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: __PAGEFLAG_FALSE is a non-atomic equivalent of PAGEFLAG_FALSE. Signed-off-by: Brendan Jackman --- include/linux/page-flags.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 4bf1c25fd1dc5..57fa58899a661 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -488,6 +488,10 @@ static inline int Page##uname(const struct page *page) { return 0; } FOLIO_SET_FLAG_NOOP(lname) \ static inline void SetPage##uname(struct page *page) { } +#define __SETPAGEFLAG_NOOP(uname, lname) \ +static inline void __folio_set_##lname(struct folio *folio) { } \ +static inline void __SetPage##uname(struct page *page) { } + #define CLEARPAGEFLAG_NOOP(uname, lname) \ FOLIO_CLEAR_FLAG_NOOP(lname) \ static inline void ClearPage##uname(struct page *page) { } @@ -510,6 +514,9 @@ static inline int TestClearPage##uname(struct page *page) { return 0; } #define TESTSCFLAG_FALSE(uname, lname) \ TESTSETFLAG_FALSE(uname, lname) TESTCLEARFLAG_FALSE(uname, lname) +#define __PAGEFLAG_FALSE(uname, lname) TESTPAGEFLAG_FALSE(uname, lname) \ + __SETPAGEFLAG_NOOP(uname, lname) __CLEARPAGEFLAG_NOOP(uname, lname) + __PAGEFLAG(Locked, locked, PF_NO_TAIL) FOLIO_FLAG(waiters, FOLIO_HEAD_PAGE) PAGEFLAG(Error, error, PF_NO_TAIL) TESTCLEARFLAG(Error, error, PF_NO_TAIL) From patchwork Fri Jul 12 17:00:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13731999 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5DFCAC3DA45 for ; Fri, 12 Jul 2024 17:01:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E6AFB6B00AD; Fri, 12 Jul 2024 13:01:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DF2646B00AE; Fri, 12 Jul 2024 13:01:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD2B96B00AF; Fri, 12 Jul 2024 13:01:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 9094F6B00AD for ; Fri, 12 Jul 2024 13:01:42 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 47061160C47 for ; Fri, 12 Jul 2024 17:01:42 +0000 (UTC) X-FDA: 82331717244.29.6644A3D Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) by imf07.hostedemail.com (Postfix) with ESMTP id 296334002C for ; Fri, 12 Jul 2024 17:01:39 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=KUC6nPi1; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 3cmGRZggKCLUeVXfhViWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=3cmGRZggKCLUeVXfhViWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--jackmanb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803674; a=rsa-sha256; cv=none; b=qn/AIWVIpUFr9oT7V57OxWcPDXfpmRMnxbVysLSPMBEBQ0DMdEkAOLTzxDnbciCH0vuGTJ jkw0J2/V449Zmlf5JwZbPlxtCl/Q2ip89/eUNXMUs/lbVvI57Ua6TW7WpR37fWJd/7h14Z f1rUF2EbBPx0MlJ+bWwLjiZc2BnFPDo= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=KUC6nPi1; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 3cmGRZggKCLUeVXfhViWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=3cmGRZggKCLUeVXfhViWbjjbgZ.Xjhgdips-hhfqVXf.jmb@flex--jackmanb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803674; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tuve7JnQEpqqfaMLOqszce9B8olVou/+ZFnfWaWBcIQ=; b=NxD0woi7kKruZmG+taWo3HIFLWUj1E+uVDOu0D1KqsnWILhgtnpTMlttlmDNPoVMcHJu1D 2GpMECJoGujZWjAQkqRhmD2ggBhUnHyFNVSzc3rlQFT01rTlpoMdP4WzFRaCxqUscEaWV4 o5GuZVHILWGu9LcKxAGTZaSy1FbSSx8= Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-42796140873so13373425e9.2 for ; Fri, 12 Jul 2024 10:01:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803699; x=1721408499; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=tuve7JnQEpqqfaMLOqszce9B8olVou/+ZFnfWaWBcIQ=; b=KUC6nPi1Ts8/7uu9mMu9YhW4JIq3jjh66TAmqqdg+PT1IzX7ly35Gx4FdW4JFnOrYh m+OhbcBt7VyvjBBpniVuvo1Vd7xQw7a0hpu+1Bu0t68nfWLom2jqaRK6ikjCNUWz/rUa 5y3KZk0VXebuaB07NKOef9sLbYGvgWy9syOAiIqQeT0aP0DYcVBzyn1IYWGZ7LGQBmwZ 8prGj/veAX3NDqDdAjPWFd8Lb613FxviCP+tfXqiJXKZrQAkhzqJ1S4QG9ObzFyAlfKU lJXy+KOLqV0jLFhKXY5dEJ9J/Y+wwt/7x4sRNoNi/OsUwzNh0Pf2Axvej3xIW6fmPG91 Clvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803699; x=1721408499; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tuve7JnQEpqqfaMLOqszce9B8olVou/+ZFnfWaWBcIQ=; b=KkeeiMXLO7ucqhOWL7rGyPSmAXP2Ug/vqF8LXrGa1Y1/GKlI8m2TBpg75pQ4Q9CO9W 3hRdrNKqryhur7/MokrBbzUE87jV4T7Rufx/N5OmhwL/KqmBLBCmdcouO7ge/efJIRVS DvW4dzTRuhlEbuAk+Uwf6V1eojXuF/nYebTBQkd+iTtmT9QaH000O2Ka+TNsFISs5nlr qGDH4oVAF75taDcokoGSi8Xdi2L7xkIB0xQ3JX8p7ZbONK+WeH2mZTOzyVgltLx9UoZL LehZjhBIPp4ctOwZ7Eazgk3Hwm+//SXanJXWOCppVXL4t+W+2ICsBkX00fkNtgr+UhSD fKlg== X-Forwarded-Encrypted: i=1; AJvYcCWLPQdTgxOifZNfWUwdpikzYjuuNZvYtsHqBuYfmNVnSQO99qfpjy6DHF9iar3acY5HK9UWan5VfiFFxQImXoF2zIA= X-Gm-Message-State: AOJu0Yx+ODIlpbo2G8RpH7NAgW39Yz1rjxUA8N4Xs4M5+7gMjSAVWIee TsHZ1Le1VLkggWEirmB3wONJWO8qd7IQ11Q40q8BuSYZEWTA/dUADTN4uPBTVQAU51qJ32BFfBV AVnYrdBH5pA== X-Google-Smtp-Source: AGHT+IE/LeQCdWU5upInTaAHGJKI73E8a+vH6WinMVUM+08W4ltowKkHhgsLti9XyJHiOTCwn2ccsqpZeVTo4A== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:6000:14c:b0:360:727b:8b5d with SMTP id ffacd0b85a97d-367cea738dbmr25058f8f.6.1720803698587; Fri, 12 Jul 2024 10:01:38 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:34 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-16-144b319a40d8@google.com> Subject: [PATCH 16/26] mm: asi: Map non-user buddy allocations as nonsensitive From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Rspamd-Queue-Id: 296334002C X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: o1x5gcin78ehjty7fftme7tcs1sghmxn X-HE-Tag: 1720803699-816883 X-HE-Meta: U2FsdGVkX18lzbbXg1mHZrkPx3eVH6O5UWtqzjoqIykp6Q5DDhceP1HAN0I8BZwL/nIChTiwysy+ruy6lb1dYUKV8EvTq65JIFkOC6VJ8Xkl25dlNX+1ZOEwVjqZ0hoZ1QzE7W5j/Xur9zDZrDL59TwmQmlzJD1Bo12sl7wWhNTQN/pnnhwOuocyM+kDvXNL+iHU8Jp5ZhLtZgg39zqHhJmCXCMKQ5+YjsyUw+1Xtec7hJYlorX//xH1OxaZQdiJHvOiCsJvwovIXjOzlpVPP5y7iTUvijbqWfiDgBnfC6bqN2P9jE22/S0RUqxmJQpFBkv/03zWodJF+ujdpcZR6qatPWuM/7gJwUW03e9HO3L4cr3uiZRp0ONKXjVD+Lp7fU3bxXCKPGZTQ15XtBj3UKqLgud8qv2xFj+oO9oV+PLFenB51xkirrQGK+Hytj8HgXbUKGvrh7esTPfuxnu+1Q4ICm1Vu9NGvtei7AMDsmju2G0gYnxnlbZFfb1g6Zsa8B15o8OfCcvNtw/f7sxStB8vvELc7rDnRmR4osROzRACsC0b2i1E7hruW5VBQdcGOCKTO3FPOArdGTaqzEJwe+cDSEQH4lk3EQWhXuqWTQjxGznBeAphPVAdFeHjfuJhd+/mu7Eu9omFI10E3dxfHoY15xt+wXsdvfpXgh/cxL0X4/zYXZaMd9X0XhAber/cYAtFf0BzAdC89/pefjb0zGJTl4ySYDZaRlrbdQnGAT7kP5pekHH25mMUK1OF6xJGiATjnLWgHnA565wIOQxuKCHTyG5Nh1Jr3uctepsU5aBHVW6kRtzrSno7AkE+wadkt1s2trkziulXI0HKkGVCeyPyZA/H1ogujG7BUvO9TN4P0xPKE5PYo6MZHntZZTkCetv4rsQm+XgipwtOYJAaOMO+a0gWB0W31qyijkrhyemv53r8ohqVj5Jssx76/mnsy5+VwC+SEqbDSxDrqpp QeZOq1Rc n+pDxpkk5FkEqdWfITLQBLcMUMaTayWyrEX3OPALf8UaULTZnzgMdPvGhZAnpGEu+dD/Ghs3t78n2ZK4etmpfVpnC0ayctjLOv8Uw/hNTmEm/utSf5SJ4sd0YTFubJ+bQmp1oyzwtfqFibJLvBBVc7k8mWe2ahOZMX7e3IiT0S2VtGXvtzQvNfLtGZJx1LE/j3I50hR/1G++zZl4QBn5VvaM+ZjxpKkP02+EXw3JUe6NHCscqpiqPhnWoxTd+wuPw5T5Fg8m5BptrcMM1ITjKTT3gnBwgnqKCOSePLxfLVtJY9KC/dw+BZANw5cH6k//7uw4wNVciMWq7eOXfKdPxbuSvOPjFFoeJhAcHjbfRk6AqcJncqLMyNH4q3GlkRB6+kX6ogAmp7iYclxR4bZAPz2oW8QtRosvI2vU1ZkWeZRsJU5t+m84GT9Zg/yHi+GwMl1zRYJ0jcITkDyvIOYnlXVlH1o65nmE6mRAD3uIgeDt0DAk13zMktOxWwsehhgU3nhhrDOfCaEKtrMLvHwHr+wRP/AKz6VnQf+Dl+yCKz+NYSyg7f6npAi5Kjdshqw8PO0Wa2LfH7x0+m+21sKfm7MvFNKAlnB9eYTq5UQ4t4JLSdQg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is just simplest possible page_alloc patch I could come up with to demonstrate ASI working in a "denylist" mode: we map the direct map into the restricted address space, except pages allocated with GFP_USER. Pages must be asi_unmap()'d before they can be re-allocated. This requires a TLB flush, which can't generally be done from the free path (requires IRQs on), so pages that need unmapping are freed via a workqueue. This solution is silly for at least the following reasons: - If the async queue gets long, we'll run out of allocatable memory. - We don't batch the TLB flushing or worker wakeups at all. - We drop FPI flags and skip the pcplists. Internally at Google we've so far found with plenty of extra complexity we're able to make the principle work for the workloads we've tested so far, but it seems likely we'll hit a wall where tuning gets impossible. So instead for the [PATCH] version I hope to come up with an implementation that instead just makes the allocator more deeply aware of sensitivity, most likely this will look a bit like an extra "dimension" like movability etc. This was discussed at LSF/MM/BPF [1] but I haven't made time to experiment on it yet. With this smarter approach, it should also be possible to remove the pageflag, as other contextual information will let us know if a page is mapped in the restricted address space (the page tables also reflect this status...). [1] https://youtu.be/WD9-ey8LeiI The main thing in here that is "real" and may warrant discussion is __GFP_SENSITIVE (or at least, some sort of allocator switch to determine sensitivity, in an "allowlist" model we would probably have the opposite, and in future iterations we might want additional options for different "types" of sensitivity). I think we need this as an extension to the allocation API; the main alternative would be to infer from context of the allocation whether the data should be treated as sensitive; however I think we will have contexts where both sensitive and nonsensitive data needs to be allocatable. If there are concerns about __GFP flags specifically, rather than just the general problem of expanding the allocator API, we could always just provide an API like __alloc_pages_sensitive or something, implemented with ALLOC_ flags internally. Signed-off-by: Brendan Jackman --- arch/x86/mm/asi.c | 33 +++++++++- include/linux/gfp_types.h | 15 ++++- include/linux/page-flags.h | 9 +++ include/trace/events/mmflags.h | 12 +++- mm/page_alloc.c | 143 ++++++++++++++++++++++++++++++++++++++++- tools/perf/builtin-kmem.c | 1 + 6 files changed, 208 insertions(+), 5 deletions(-) diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index 807d51497f43a..6e106f25abbb9 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -5,6 +5,8 @@ #include #include +#include + #include #include #include @@ -102,10 +104,17 @@ EXPORT_SYMBOL_GPL(asi_unregister_class); * allocator from interrupts and the page allocator ultimately calls this * code. * - They support customizing the allocation flags. + * - They avoid infinite recursion when the page allocator calls back to + * asi_map * * On the other hand, they do not use the normal page allocation infrastructure, * that means that PTE pages do not have the PageTable type nor the PagePgtable * flag and we don't increment the meminfo stat (NR_PAGETABLE) as they do. + * + * As an optimisation we attempt to map the pagetables in + * ASI_GLOBAL_NONSENSITIVE, but this can fail, and for simplicity we don't do + * anything about that. This means it's invalid to access ASI pagetables from a + * critical section. */ static_assert(!IS_ENABLED(CONFIG_PARAVIRT)); #define DEFINE_ASI_PGTBL_ALLOC(base, level) \ @@ -114,8 +123,11 @@ static level##_t * asi_##level##_alloc(struct asi *asi, \ gfp_t flags) \ { \ if (unlikely(base##_none(*base))) { \ - ulong pgtbl = get_zeroed_page(flags); \ + /* Stop asi_map calls causing recursive allocation */ \ + gfp_t pgtbl_gfp = flags | __GFP_SENSITIVE; \ + ulong pgtbl = get_zeroed_page(pgtbl_gfp); \ phys_addr_t pgtbl_pa; \ + int err; \ \ if (!pgtbl) \ return NULL; \ @@ -129,6 +141,16 @@ static level##_t * asi_##level##_alloc(struct asi *asi, \ } \ \ mm_inc_nr_##level##s(asi->mm); \ + \ + err = asi_map_gfp(ASI_GLOBAL_NONSENSITIVE, \ + (void *)pgtbl, PAGE_SIZE, flags); \ + if (err) \ + /* Should be rare. Spooky. */ \ + pr_warn_ratelimited("Created sensitive ASI %s (%pK, maps %luK).\n",\ + #level, (void *)pgtbl, addr); \ + else \ + __SetPageGlobalNonSensitive(virt_to_page(pgtbl));\ + \ } \ out: \ VM_BUG_ON(base##_leaf(*base)); \ @@ -469,6 +491,9 @@ static bool follow_physaddr( * reason for this is that we don't want to unexpectedly undo mappings that * weren't created by the present caller. * + * This must not be called from the critical section, as ASI's pagetables are + * not guaranteed to be mapped in the restricted address space. + * * If the source mapping is a large page and the range being mapped spans the * entire large page, then it will be mapped as a large page in the ASI page * tables too. If the range does not span the entire huge page, then it will be @@ -492,6 +517,9 @@ int __must_check asi_map_gfp(struct asi *asi, void *addr, unsigned long len, gfp if (!static_asi_enabled()) return 0; + /* ASI pagetables might be sensitive. */ + WARN_ON_ONCE(asi_in_critical_section()); + VM_BUG_ON(!IS_ALIGNED(start, PAGE_SIZE)); VM_BUG_ON(!IS_ALIGNED(len, PAGE_SIZE)); VM_BUG_ON(!fault_in_kernel_space(start)); /* Misnamed, ignore "fault_" */ @@ -591,6 +619,9 @@ void asi_unmap(struct asi *asi, void *addr, size_t len) if (!static_asi_enabled() || !len) return; + /* ASI pagetables might be sensitive. */ + WARN_ON_ONCE(asi_in_critical_section()); + VM_BUG_ON(start & ~PAGE_MASK); VM_BUG_ON(len & ~PAGE_MASK); VM_BUG_ON(!fault_in_kernel_space(start)); /* Misnamed, ignore "fault_" */ diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index 13becafe41df0..d33953a1c9b28 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -55,6 +55,7 @@ enum { #ifdef CONFIG_LOCKDEP ___GFP_NOLOCKDEP_BIT, #endif + ___GFP_SENSITIVE_BIT, ___GFP_LAST_BIT }; @@ -95,6 +96,11 @@ enum { #else #define ___GFP_NOLOCKDEP 0 #endif +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +#define ___GFP_SENSITIVE BIT(___GFP_SENSITIVE_BIT) +#else +#define ___GFP_SENSITIVE 0 +#endif /* * Physical address zone modifiers (see linux/mmzone.h - low four bits) @@ -284,6 +290,12 @@ enum { /* Disable lockdep for GFP context tracking */ #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP) +/* + * Allocate sensitive memory, i.e. do not map it into ASI's restricted address + * space. + */ +#define __GFP_SENSITIVE ((__force gfp_t)___GFP_SENSITIVE) + /* Room for N __GFP_FOO bits */ #define __GFP_BITS_SHIFT ___GFP_LAST_BIT #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1)) @@ -365,7 +377,8 @@ enum { #define GFP_NOWAIT (__GFP_KSWAPD_RECLAIM | __GFP_NOWARN) #define GFP_NOIO (__GFP_RECLAIM) #define GFP_NOFS (__GFP_RECLAIM | __GFP_IO) -#define GFP_USER (__GFP_RECLAIM | __GFP_IO | __GFP_FS | __GFP_HARDWALL) +#define GFP_USER (__GFP_RECLAIM | __GFP_IO | __GFP_FS | \ + __GFP_HARDWALL | __GFP_SENSITIVE) #define GFP_DMA __GFP_DMA #define GFP_DMA32 __GFP_DMA32 #define GFP_HIGHUSER (GFP_USER | __GFP_HIGHMEM) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 57fa58899a661..d4842cd1fb59a 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -135,6 +135,9 @@ enum pageflags { #ifdef CONFIG_ARCH_USES_PG_ARCH_X PG_arch_2, PG_arch_3, +#endif +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + PG_global_nonsensitive, #endif __NR_PAGEFLAGS, @@ -642,6 +645,12 @@ FOLIO_TEST_CLEAR_FLAG(young, FOLIO_HEAD_PAGE) FOLIO_FLAG(idle, FOLIO_HEAD_PAGE) #endif +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +__PAGEFLAG(GlobalNonSensitive, global_nonsensitive, PF_ANY); +#else +__PAGEFLAG_FALSE(GlobalNonSensitive, global_nonsensitive); +#endif + /* * PageReported() is used to track reported free pages within the Buddy * allocator. We can use the non-atomic version of the test and set diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index d55e53ac91bd2..416a79fe1a66d 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -50,7 +50,8 @@ gfpflag_string(__GFP_RECLAIM), \ gfpflag_string(__GFP_DIRECT_RECLAIM), \ gfpflag_string(__GFP_KSWAPD_RECLAIM), \ - gfpflag_string(__GFP_ZEROTAGS) + gfpflag_string(__GFP_ZEROTAGS), \ + gfpflag_string(__GFP_SENSITIVE) #ifdef CONFIG_KASAN_HW_TAGS #define __def_gfpflag_names_kasan , \ @@ -95,6 +96,12 @@ #define IF_HAVE_PG_ARCH_X(_name) #endif +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +#define IF_HAVE_ASI(_name) ,{1UL << PG_##_name, __stringify(_name)} +#else +#define IF_HAVE_ASI(_name) +#endif + #define DEF_PAGEFLAG_NAME(_name) { 1UL << PG_##_name, __stringify(_name) } #define __def_pageflag_names \ @@ -125,7 +132,8 @@ IF_HAVE_PG_HWPOISON(hwpoison) \ IF_HAVE_PG_IDLE(idle) \ IF_HAVE_PG_IDLE(young) \ IF_HAVE_PG_ARCH_X(arch_2) \ -IF_HAVE_PG_ARCH_X(arch_3) +IF_HAVE_PG_ARCH_X(arch_3) \ +IF_HAVE_ASI(global_nonsensitive) #define show_page_flags(flags) \ (flags) ? __print_flags(flags, "|", \ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 14d39f34d3367..1e71ee9ae178c 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1081,6 +1081,8 @@ static void kernel_init_pages(struct page *page, int numpages) kasan_enable_current(); } +static bool asi_async_free_enqueue(struct page *page, unsigned int order); + __always_inline bool free_pages_prepare(struct page *page, unsigned int order) { @@ -1177,7 +1179,7 @@ __always_inline bool free_pages_prepare(struct page *page, debug_pagealloc_unmap_pages(page, 1 << order); - return true; + return !asi_async_free_enqueue(page, order); } /* @@ -4364,6 +4366,136 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, return true; } +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + +struct asi_async_free_cpu_state { + struct work_struct work; + struct list_head to_free; +}; +static DEFINE_PER_CPU(struct asi_async_free_cpu_state, asi_async_free_cpu_state); + +static bool async_free_work_initialized; + +static void asi_async_free_work_fn(struct work_struct *work) +{ + struct asi_async_free_cpu_state *cpu_state = + container_of(work, struct asi_async_free_cpu_state, work); + struct page *page, *tmp; + struct list_head to_free = LIST_HEAD_INIT(to_free); + + local_irq_disable(); + list_splice_init(&cpu_state->to_free, &to_free); + local_irq_enable(); /* IRQs must be on for asi_unmap. */ + + /* Use _safe because __free_the_page uses .lru */ + list_for_each_entry_safe(page, tmp, &to_free, lru) { + unsigned long order = page_private(page); + + asi_unmap(ASI_GLOBAL_NONSENSITIVE, page_to_virt(page), + PAGE_SIZE << order); + for (int i = 0; i < (1 << order); i++) + __ClearPageGlobalNonSensitive(page + i); + + /* + * Note weird loop-de-loop here, we might already have called + * __free_pages_ok for this page, but now we've cleared + * PageGlobalNonSensitive so it won't end up back on the queue + * again. + */ + __free_pages_ok(page, order, FPI_NONE); + cond_resched(); + } +} + +/* Returns true if the page was queued for asynchronous freeing. */ +static bool asi_async_free_enqueue(struct page *page, unsigned int order) +{ + struct asi_async_free_cpu_state *cpu_state; + unsigned long flags; + + if (!PageGlobalNonSensitive(page)) + return false; + + local_irq_save(flags); + cpu_state = this_cpu_ptr(&asi_async_free_cpu_state); + set_page_private(page, order); + list_add(&page->lru, &cpu_state->to_free); + local_irq_restore(flags); + + return true; +} + +static int __init asi_page_alloc_init(void) +{ + int cpu; + + if (!static_asi_enabled()) + return 0; + + for_each_possible_cpu(cpu) { + struct asi_async_free_cpu_state *cpu_state + = &per_cpu(asi_async_free_cpu_state, cpu); + + INIT_WORK(&cpu_state->work, asi_async_free_work_fn); + INIT_LIST_HEAD(&cpu_state->to_free); + } + + /* + * This function is called before SMP is initialized, so we can assume + * that this is the only running CPU at this point. + */ + + barrier(); + async_free_work_initialized = true; + barrier(); + + return 0; +} +early_initcall(asi_page_alloc_init); + +static int asi_map_alloced_pages(struct page *page, uint order, gfp_t gfp_mask) +{ + + if (!static_asi_enabled()) + return 0; + + if (!(gfp_mask & __GFP_SENSITIVE)) { + int err = asi_map_gfp( + ASI_GLOBAL_NONSENSITIVE, page_to_virt(page), + PAGE_SIZE * (1 << order), gfp_mask); + uint i; + + if (err) + return err; + + for (i = 0; i < (1 << order); i++) + __SetPageGlobalNonSensitive(page + i); + } + + return 0; +} + +#else /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ + +static inline +int asi_map_alloced_pages(struct page *pages, uint order, gfp_t gfp_mask) +{ + return 0; +} + +static inline +bool asi_unmap_freed_pages(struct page *page, unsigned int order) +{ + return true; +} + +static bool asi_async_free_enqueue(struct page *page, unsigned int order) +{ + return false; +} + +#endif + /* * __alloc_pages_bulk - Allocate a number of order-0 pages to a list or array * @gfp: GFP flags for the allocation @@ -4551,6 +4683,10 @@ struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, if (WARN_ON_ONCE_GFP(order > MAX_PAGE_ORDER, gfp)) return NULL; + /* Clear out old (maybe sensitive) data before reallocating as nonsensitive. */ + if (!static_asi_enabled() && !(gfp & __GFP_SENSITIVE)) + gfp |= __GFP_ZERO; + gfp &= gfp_allowed_mask; /* * Apply scoped allocation constraints. This is mainly about GFP_NOFS @@ -4597,6 +4733,11 @@ struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, trace_mm_page_alloc(page, order, alloc_gfp, ac.migratetype); kmsan_alloc_page(page, order, alloc_gfp); + if (page && unlikely(asi_map_alloced_pages(page, order, gfp))) { + __free_pages(page, order); + page = NULL; + } + return page; } EXPORT_SYMBOL(__alloc_pages); diff --git a/tools/perf/builtin-kmem.c b/tools/perf/builtin-kmem.c index 9714327fd0ead..912497b7b1c3f 100644 --- a/tools/perf/builtin-kmem.c +++ b/tools/perf/builtin-kmem.c @@ -682,6 +682,7 @@ static const struct { { "__GFP_RECLAIM", "R" }, { "__GFP_DIRECT_RECLAIM", "DR" }, { "__GFP_KSWAPD_RECLAIM", "KR" }, + { "__GFP_SENSITIVE", "S" }, }; static size_t max_gfp_len; From patchwork Fri Jul 12 17:00:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13732000 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 013DBC3DA4D for ; Fri, 12 Jul 2024 17:01:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 25E196B00AF; Fri, 12 Jul 2024 13:01:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1E6116B00B0; Fri, 12 Jul 2024 13:01:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0105E6B00B1; Fri, 12 Jul 2024 13:01:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id CDAB56B00AF for ; Fri, 12 Jul 2024 13:01:45 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 3D32F12054B for ; Fri, 12 Jul 2024 17:01:45 +0000 (UTC) X-FDA: 82331717370.12.8E93C23 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) by imf10.hostedemail.com (Postfix) with ESMTP id 010B2C0026 for ; Fri, 12 Jul 2024 17:01:42 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=rblEXR1l; spf=pass (imf10.hostedemail.com: domain of 3dWGRZggKCLghYaikYlZemmejc.amkjglsv-kkitYai.mpe@flex--jackmanb.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=3dWGRZggKCLghYaikYlZemmejc.amkjglsv-kkitYai.mpe@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803685; a=rsa-sha256; cv=none; b=rnE83kYg2qnE0iDYVJz2aAxA5Q7azrIwE0z5IAVDu3AXS5EuTSRUe/HEPdoFGLUhwTMFzX wtmLiDhxUvvzki2BVUA1baWkQX0w3GZl2ptR42Dmr9tdVqurF5wu2Vit16mLIhCAAh4Zbx 36cMJvmWJcJaW8EHWR1+KkyrBmhkQ4w= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=rblEXR1l; spf=pass (imf10.hostedemail.com: domain of 3dWGRZggKCLghYaikYlZemmejc.amkjglsv-kkitYai.mpe@flex--jackmanb.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=3dWGRZggKCLghYaikYlZemmejc.amkjglsv-kkitYai.mpe@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803685; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mWHMoCzbupyPLn4HJqmbJTZdF2QBYUbInE64x2NA2rE=; b=xfZE76rwq7CLvmOTxRAnXhRy5wyLep54e7jnDfSXui2ubN+yDh2Osz74OsA7cKUrwinVHx wAsRgdpERXGsoGKoQEujrYEBr+LPZNLegkbEe6fXdiaXwIuAaxYjrFdWm1uS3ysGohreMv 5wCYF62OtP9tnJZ/phqE+zioVLdw1EM= Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-426620721c2so15370935e9.2 for ; Fri, 12 Jul 2024 10:01:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803701; x=1721408501; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mWHMoCzbupyPLn4HJqmbJTZdF2QBYUbInE64x2NA2rE=; b=rblEXR1l7C+VrAR8tY9mzVRYbwyr+5p0eDVrCIP7jlttGLsZvv3Vl4gQpL18gcbWiq U/pbq6sdqR/2LXF1zKXX/0r99cKwfHuXZguJmA/EIH5UylYcAFNxxa4pixZVvVvbibDu mzbxBqzZCSuMPza4NYqGz+NUZj622VoKDzHsBlLrhSSpeKslIwozFhjJACKvtTVaiFiv TvZyovKiVIBd3izO17muPHCC7/urG9XDMe21CRWzRSTdusxtFnv36YMWeG+wB+nordOw 1/QLgKMmmW4sgd+f1B3UI99n//s24rPXd2T0G76NHP75I7lHKNNzpjL38Fwm0kS23N2o zATA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803701; x=1721408501; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mWHMoCzbupyPLn4HJqmbJTZdF2QBYUbInE64x2NA2rE=; b=MfmwFTHMMcdkrS+DSxEdDX6itSSOwnfz0QzsTPH1G8TgBZDWW9OTCEcds8iVGXhfJB miHll7enmddLqLBNJFKR8VLRn2gaZUzjwq/IWcboA6+XcG85+HQrn4BnyqvKXpBFGGaH r60Oxmcu7AfjDmsHJHig7WWj3bl1yMAZRSWDJxlyt/JJyLgZK23qIj76EmMk6C1z4Mku kYWaccbDI47+em4Y1pvOLPsF+zqngLTsjlWIokK+0EOlrZA4KbupAhR7adVf6q7Ok8gy QvWt7jE/0qsY/fwSsPM/kfEUf/Kqk7M5ONCOzrpfRrSvlpT/juL4EN827ZTx7qWXj73j 7+zQ== X-Forwarded-Encrypted: i=1; AJvYcCUcjMBIKV+R+J0suqzE92G1+4UM7Zsh+AdXyy6a31tEfuOJijq8uuqu7ofFfSU5u9OXmyTsn1eermnrKFy8ctF1lmE= X-Gm-Message-State: AOJu0Yw36BJ2RbSAw0ddok3+XSjTMSMQd1BUzf3JzG4G263PRndcBoVa MupwnlY4GmgcKohJ+sjIkTIBWJ+bldkPOukZqYNCvp6AUJ9V9wrjWWDITjTHbE+NtlAxdTy64WO WLaFUhwi2Mw== X-Google-Smtp-Source: AGHT+IH7bbw/Dn5jSbbHkSsXS63XrNwSXZ9nYhS09PC8vYQaS4DPRI1dYsfdXRcOP1EzG0hCXTQisDlHIVSzOg== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:600c:4f0e:b0:426:6a73:fb5f with SMTP id 5b1f17b1804b1-426708f9ab5mr1905165e9.7.1720803701319; Fri, 12 Jul 2024 10:01:41 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:35 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-17-144b319a40d8@google.com> Subject: [PATCH 17/26] mm: asi: Map kernel text and static data as nonsensitive From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Stat-Signature: 6dnkufj4amybwka6fhd9h395745dxj4k X-Rspamd-Queue-Id: 010B2C0026 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1720803702-307418 X-HE-Meta: U2FsdGVkX18afb65Esl3yCYZLwV5K5KIqO8Wa+ud+TYvdCrEDykXUtAt5xoTm4DJhmLLnE0OT78rQff7uJilb4FAaJSTZrDWtQZ4FhALwuLQTUECINOdNMXdS2OKpsyjYMcgXSg6sLEd/ZaHyi2aCZr8y0HwNKednCNzar0u1EVPbnpH4eExlbsINiMAjHIH8r5nMn7RtLbKEmv9dIdfoZ8CVrK3WK3ihWfYVkP8Do7iADbNWrq+vDT+LxXDNB7QQ/V28g6wAY9VCs3A+byc9mcy7lG1RfEhb7O1SgQdeQKRKfCXaSNEMHyH3lGYykzWPMfcFe/Vtoa5Fy9E5XEBAsRF6NmNarwjdiVhjPt47bGEzWUDF7rQvw+jqQJNcVfo4XZjvsAV0n8rCVarUb8WtepFAxXiRsUal9KAsqZTkC8Lk+t9UmvzgkPO+Oa3ioCoqgU/wXIHvJnzfSL4Tgo8JwMwqZXISQTUwcgpJQbIkaPHQypfNtU7C9drHqYkTA7cG9p+13F4KMkvUEGGbbvYKTY0y3TfuPbvdBrKVAQ5sz2XPP0vBAHBFpUr9AAxgPHR+e/fXpDQpZb4mnt2I0kwWMqUIfgmhecYmF90r8Je5ry7Cg8RtRekvyelevGG+ttlXDgBvMYo5yWJaBmHBzPDDm+j9KRA3vctAqD4cq08QDE1JFdgEH8l27s/5+BBZxa46f4eJqrnS2kkD0XcJ4qGLvYgXzXIibrYtY3x+nNv0qtQj9BZxW4w/eKUupch9xeBQOkK11JbxdE3RiZczIvNjQFb+MUGCLPqvPMgVab0xRXMWzBGEa3VidcZ4Mj7fIJZ5obs3YFoFyCm+zz3H9BK/r7Bl5mhoZ23+w2Gi5pSXS1ldA9HuA/XB7wXiCHXL7Lll2tTN7KAdkC3EBUQwG37ZYqIo7lKW6hagr3QMud/gIJemLnvJMEqpRm0oCcNORmqMkybMrathBoMQ6nGuED dxGLjYAG aWTUINO4d0GJcHog0YNHas0V9OGlNc2Oxm1ukNswECslPwC4jzK0WTV1bEuvCQuaCvULK2Y7gdv3fphxaYbIcKBJv46fPDNtLdVlKgNEBGkQJ04koJKMqTbuWoXPy7YRsOhj0pvhPNUi9oKztR8y7+oYD5V2w9DqkF3BsJxl+GmUtpNSPiBBoYzAuk2An181b2RBXMbg4+AV2JQRDbILoLstMN/K9YHyB2AIQ+c+DUTt0ir8KL69WOTdCmSR+/gbTZN48sHmpw2eNZIUIxzENu/3X8vFnkAKHsxNRl/mGOGeIpMISKBQlrTybM0BTYGSgOuCGI4hMkTIygOJiOhXqz0ExaHyebtPn5uurKIZmsksluE/Owdo60gAhpLMc9AOhyEJ9PU/Wxib/593/WxrGpAxXi0OBZ2ZYcGck7tJvhcM3SxvKccpLS7gFv44lA7Rad9+xj0smn7WvpkNS6+U9tty4qiqMHUpzA8aLXTz8InbJSz+dIXsDZXGI9eSVw47xmChDSD7wkzy/1at6ZBOqvlhNxR1chr2Fcyym X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Basically we need to map the kernel code and all its static variables. Per-CPU variables need to be treated specially as described in the comments. The cpu_entry_area is similar - this needs to be nonsensitive so that the CPU can access the GDT etc when handling a page fault. Under 5-level paging, most of the kernel memory comes under a single PGD entry (see Documentation/x86/x86_64/mm.rst. Basically, the mapping is for this big region is the same as under 4-level, just wrapped in an outer PGD entry). For that region, the "clone" logic is moved down one step of the paging hierarchy. Note that the p4d_alloc in asi_clone_p4d won't actually be used in practice; the relevant PGD entry will always have been populated by prior asi_map calls so this code would "work" if we just wrote p4d_offset (but asi_clone_p4d would be broken if viewed in isolation). The vmemmap area is not under this single PGD, it has its own 2-PGD area, so we still use asi_clone_pgd for that one. Signed-off-by: Brendan Jackman --- arch/x86/mm/asi.c | 106 +++++++++++++++++++++++++++++++++++++- include/asm-generic/vmlinux.lds.h | 11 ++++ 2 files changed, 116 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index 6e106f25abbb..891b8d351df8 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -7,8 +7,8 @@ #include #include -#include #include +#include #include #include #include @@ -184,8 +184,68 @@ void __init asi_check_boottime_disable(void) pr_info("ASI enablement ignored due to incomplete implementation.\n"); } +/* + * Map data by sharing sub-PGD pagetables with the unrestricted mapping. This is + * more efficient than asi_map, but only works when you know the whole top-level + * page needs to be mapped in the restricted tables. Note that the size of the + * mappings this creates differs between 4 and 5-level paging. + */ +static void asi_clone_pgd(pgd_t *dst_table, pgd_t *src_table, size_t addr) +{ + pgd_t *src = pgd_offset_pgd(src_table, addr); + pgd_t *dst = pgd_offset_pgd(dst_table, addr); + + if (!pgd_val(*dst)) + set_pgd(dst, *src); + else + WARN_ON_ONCE(pgd_val(*dst) != pgd_val(*src)); +} + +/* + * For 4-level paging this is exactly the same as asi_clone_pgd. For 5-level + * paging it clones one level lower. So this always creates a mapping of the + * same size. + */ +static void asi_clone_p4d(pgd_t *dst_table, pgd_t *src_table, size_t addr) +{ + pgd_t *src_pgd = pgd_offset_pgd(src_table, addr); + pgd_t *dst_pgd = pgd_offset_pgd(dst_table, addr); + p4d_t *src_p4d = p4d_alloc(&init_mm, src_pgd, addr); + p4d_t *dst_p4d = p4d_alloc(&init_mm, dst_pgd, addr); + + if (!p4d_val(*dst_p4d)) + set_p4d(dst_p4d, *src_p4d); + else + WARN_ON_ONCE(p4d_val(*dst_p4d) != p4d_val(*src_p4d)); +} + +/* + * percpu_addr is where the linker put the percpu variable. asi_map_percpu finds + * the place where the percpu allocator copied the data during boot. + * + * This is necessary even when the page allocator defaults to + * global-nonsensitive, because the percpu allocator uses the memblock allocator + * for early allocations. + */ +static int asi_map_percpu(struct asi *asi, void *percpu_addr, size_t len) +{ + int cpu, err; + void *ptr; + + for_each_possible_cpu(cpu) { + ptr = per_cpu_ptr(percpu_addr, cpu); + err = asi_map(asi, ptr, len); + if (err) + return err; + } + + return 0; +} + static int __init asi_global_init(void) { + int err; + if (!boot_cpu_has(X86_FEATURE_ASI)) return 0; @@ -205,6 +265,46 @@ static int __init asi_global_init(void) VMALLOC_START, VMALLOC_END, "ASI Global Non-sensitive vmalloc"); + /* Map all kernel text and static data */ + err = asi_map(ASI_GLOBAL_NONSENSITIVE, (void *)__START_KERNEL, + (size_t)_end - __START_KERNEL); + if (WARN_ON(err)) + return err; + err = asi_map(ASI_GLOBAL_NONSENSITIVE, (void *)FIXADDR_START, + FIXADDR_SIZE); + if (WARN_ON(err)) + return err; + /* Map all static percpu data */ + err = asi_map_percpu( + ASI_GLOBAL_NONSENSITIVE, + __per_cpu_start, __per_cpu_end - __per_cpu_start); + if (WARN_ON(err)) + return err; + + /* + * The next areas are mapped using shared sub-P4D paging structures + * (asi_clone_p4d instead of asi_map), since we know the whole P4D will + * be mapped. + */ + asi_clone_p4d(asi_global_nonsensitive_pgd, init_mm.pgd, + CPU_ENTRY_AREA_BASE); +#ifdef CONFIG_X86_ESPFIX64 + asi_clone_p4d(asi_global_nonsensitive_pgd, init_mm.pgd, + ESPFIX_BASE_ADDR); +#endif + /* + * The vmemmap area actually _must_ be cloned via shared paging + * structures, since mappings can potentially change dynamically when + * hugetlbfs pages are created or broken down. + * + * We always clone 2 PGDs, this is a corrolary of the sizes of struct + * page, a page, and the physical address space. + */ + WARN_ON(sizeof(struct page) * MAXMEM / PAGE_SIZE != 2 * (1UL << PGDIR_SHIFT)); + asi_clone_pgd(asi_global_nonsensitive_pgd, init_mm.pgd, VMEMMAP_START); + asi_clone_pgd(asi_global_nonsensitive_pgd, init_mm.pgd, + VMEMMAP_START + (1UL << PGDIR_SHIFT)); + return 0; } subsys_initcall(asi_global_init) @@ -482,6 +582,10 @@ static bool follow_physaddr( * Map the given range into the ASI page tables. The source of the mapping is * the regular unrestricted page tables. Can be used to map any kernel memory. * + * In contrast to some internal ASI logic (asi_clone_pgd and asi_clone_p4d) this + * never shares pagetables between restricted and unrestricted address spaces, + * instead it creates wholly new equivalent mappings. + * * The caller MUST ensure that the source mapping will not change during this * function. For dynamic kernel memory, this is generally ensured by mapping the * memory within the allocator. diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index f7749d0f2562..4eca33d62950 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -1021,6 +1021,16 @@ COMMON_DISCARDS \ } +/* + * ASI maps certain sections with certain sensitivity levels, so they need to + * have a page-aligned size. + */ +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +#define ASI_ALIGN() ALIGN(PAGE_SIZE) +#else +#define ASI_ALIGN() . +#endif + /** * PERCPU_INPUT - the percpu input sections * @cacheline: cacheline size @@ -1042,6 +1052,7 @@ *(.data..percpu) \ *(.data..percpu..shared_aligned) \ PERCPU_DECRYPTED_SECTION \ + . = ASI_ALIGN(); \ __per_cpu_end = .; /** From patchwork Fri Jul 12 17:00:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13732001 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DD71C2BD09 for ; Fri, 12 Jul 2024 17:02:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 93B006B00B1; Fri, 12 Jul 2024 13:01:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 875AF6B00B2; Fri, 12 Jul 2024 13:01:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 679056B00B3; Fri, 12 Jul 2024 13:01:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 43DDC6B00B1 for ; Fri, 12 Jul 2024 13:01:49 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D7B571C1F0D for ; Fri, 12 Jul 2024 17:01:48 +0000 (UTC) X-FDA: 82331717496.06.014C6F6 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) by imf22.hostedemail.com (Postfix) with ESMTP id 8F5E8C001D for ; Fri, 12 Jul 2024 17:01:46 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=d+7XX2wT; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3eGGRZggKCLskbdlnbochpphmf.dpnmjovy-nnlwbdl.psh@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=3eGGRZggKCLskbdlnbochpphmf.dpnmjovy-nnlwbdl.psh@flex--jackmanb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803668; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9uYrhf+8D7jqi5tPdexW+5CSq/ZRPpRaUw7HFMG3mLc=; b=gyxISH8DVobNB9+ENK1SBhEegoe0dByJZqwHOvbI1oeMOr7stqHAz9dLaArkxoOiNSWfCz YiCDsrJ3igXcJg/2ssq+/YNo/c2bSEKSQy/06mYNt75QU6NjBhI9Xiinizbp6J+6T9QV76 HQ6jML2f0oPxJRkAiXv9zMIL1CXrq2I= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803668; a=rsa-sha256; cv=none; b=q0Rf9dvLH1I2zItSB2q+sXA8IgBRmhmQG831K5LBPTdMmF1jIW5OyoPfBOOp85tdqhS2N/ VPaymUdJK2SYedI+soSCt4bSkARnga8duGtDfGW6V9xJBZfAQx/QgzoASMqtIU2r5Np48x Md/HBpKToxf0OL9HfYMOIKmmJNKbSYY= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=d+7XX2wT; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of 3eGGRZggKCLskbdlnbochpphmf.dpnmjovy-nnlwbdl.psh@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=3eGGRZggKCLskbdlnbochpphmf.dpnmjovy-nnlwbdl.psh@flex--jackmanb.bounces.google.com Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-426620721c2so15371235e9.2 for ; Fri, 12 Jul 2024 10:01:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803705; x=1721408505; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9uYrhf+8D7jqi5tPdexW+5CSq/ZRPpRaUw7HFMG3mLc=; b=d+7XX2wTN/SEWLNVHOCQWKyAhCzdMkRz3wHrg4c2SRfX45cgbkw+dEltlM8E0lYAiR 4oPsJurnVta8NijDYSAMvHN/v9GU4bqDDEzW9sNzdc9HH/tY8SjQUZeff6vdJ3u8a/KL pbFHzari8T5AK0jS8xC34Oqb+zsGKll+0geHYkmZFng/DlxpA8sLknyvVcLo1yyKdhwC MFFe4RCl20JM6VHyEySe83mzQf8vqsfwtDM+hna45IxOuirF4w754gEzSeie3N323dJU uJhetRSUCHTn8bq6hBRUKf2roim3mzmQ/tBR7ICOGko4OZOLNZdcpHMBdhPVtgV/0IMD mhDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803705; x=1721408505; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9uYrhf+8D7jqi5tPdexW+5CSq/ZRPpRaUw7HFMG3mLc=; b=Tz3/mhUV/xbRFZq8lDq6zmE5Ycv469PSFg2BadsTpLFtX+kKxFesFuzHkso5VwCdAw PfVXxZh5TnOsVO381iHjUfV1OnE7PupqodZIacw/EXIzsONdZypiM7qzMowhUhnyd1Pm 0kTfJfCNCsakJMYxRE/eI1d4HjyGxjhvFP8YxSqh4Aa3FtK1dpoxQqKVgCkVEnwzuQta hsnSQJ0ychjG0/ANjfMJCv1yQ+Al1Vj/8N+/HIjTWxA7iEHJq1hA6gaSajvHLyAAQYs/ IIqTpvTvckwtdc/nSgkVDxtSo0oG89uRjM6cd4Ye1LzEySNXww/xywyAhEgnZEpLWlNk 2SGA== X-Forwarded-Encrypted: i=1; AJvYcCUE/hjVczjKLjv5FmOcLChOeoBJCvU4ILmvVIYaKayT2C8xjEWwiOSTwvDkSlpy1WkIYyj6aWnbz987YHAed7+b7Ug= X-Gm-Message-State: AOJu0YzNsGakICwR0BZZo4vjV1dX52Tn+QTK9IFmQUj6G8Se6YDSjK49 BifOlo0XXyt2p0Kkn1mra9v3y9vAsphu7kVu7tY42XOwVm9Xk8qBt05+VWD6MYcKlPYcWU2ZBdV sHFHGDjLqgw== X-Google-Smtp-Source: AGHT+IEAch1YA6Hi8yZ9PqgivQRgArWS9QwTp9vfGmMIf5En6W6fVDOrfLBr4fzHwEnBua+4wRIDTzrg/vklEA== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:600c:19cc:b0:426:6c7a:3a77 with SMTP id 5b1f17b1804b1-426705ced5bmr1906465e9.1.1720803704718; Fri, 12 Jul 2024 10:01:44 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:36 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-18-144b319a40d8@google.com> Subject: [PATCH 18/26] mm: asi: Map vmalloc/vmap data as nonsesnitive From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 8F5E8C001D X-Stat-Signature: ohr4wpps5qfgg8kg7u9oz87kn8pqjqyb X-Rspam-User: X-HE-Tag: 1720803706-469418 X-HE-Meta: U2FsdGVkX1+/pvCehfLH5ulu0bPQ4puWKhOY+fw+5lpUKYWM+TlxklbHbUWHM/Jg21msDFASt0Kn4nE4omzCSJjePJkdaE5LFF8M70wHyfWYSr5WBMZEiKwj5KRewyyeKTwgfjqcFQ9w2qDYb2P2y0q5iXc2T5QZd2USIkOwb8S+z/ZqU1kNNvyAAGAG5GcnR6LrSRhA7c1sYb4w5zhQtJynooYxzg5RV5r2LA0QMD/kZ6PRde6WbNyLw5Y5jovuFp3PfxOjLWVJdfyh/D5JFUjhlIrrfawQMjrnn6aPAoSCsQBgJSUUdzLYCDsly1BFpf7RXRxqGOrGYbGJxVN/Xc2gjJ1LoWh+yn6aVhPojybVgAAl8O1f/F2aEMrchrWtSuAJx49Yphr5fBhPOSuCD9og0Fn8bVun+qODFwSf4Q8YBaFFo2cNwkzl3+XLJGhkvDNSRHctPVBpjbzpvReiPgmvjSDNHWhI3WaPqkmeCW3L1/c5Ivbd098z5MlDfmh6OdBViUOHnbDOSrZRQZAO4pOvk8ABeJe0idjN9qR+u1TW0r2QHHKfGaoDEZdOPqVB++QL0OdTGZxC+3gzzGwCHnuRnPdAyTS2fNEUmCyg+aoO0K7bUfHaXWcbNlHp517S0VAlC9nlMOZEr4Y5tOOrGsbmdFqCr6oO4PQO0YgV+7YS0T4gwQnRDZpN6EgOLxPBhVaLJal0RaWK0L83u4s27b6ePsOm7ucYP1LtxU7msLNv+8j2JlHfGdcIQk+nfXP88R1fI4nrmA31Yn/jvQRbiqgzgK9Quzu3lqIc8ycogIgzKwYgSTa1EUhtT3QeCf+WIAKxzN4fIwgklbPzMDzG8+r2CExLHmRSzIQxxXT/99puVbLsByDgOx6uGEhPnNPBXd1kACz+TGSHmlS2DcwZadM4Uu094w4xv5hr9baaSS/IlJUpVV36fGEVV+prR0aEGt+hzyPWO0D82P94gFP hZmvY9KT GkzrQPJDVimFDXZpQ6+RBGdFUAm3oNKxTL6Kg588oxRhptuTYSbJmJnWtxRdPJo6gWCztgMZ/hck/LxPZUhuZ1jJXK5OwRjb3tepKDOcOKnGaFKkSmvfRMF4EsBzPnHmauBuEQggg3nNgZINhdpijIQxbhTgnMrQtY0FYRZqin3GN1Rk+78cyvRUDwlKwd7Nt3/Tqh5QdvU13AlwBFe3t2WXv5pUEm8xXzNOOm29kXze+5qnSIgPXY9C2Gqu0RnPMYQwHNY9rDGIKOb5InmhyudmGFwSHafMu1m29Awe2t7aYj9trI6f1pr/39vBP08fgRUNLCgwE/oxLvYLF54wmhfZrtVZ0P5gQ2EzdZRTkFjlHJni1q7aiBu6UmSX7l4xFwCPCrUH1s2OkiEbXdvP0G/V0OSMniRULwtuJV/73RoKLqp8UfdcQ4f+Fiiz6GuD9DV9w+K9oqREI5ZEjHG3s+zEVt8Ml6/Fi8GD2Ldc/5W+m+O5j65LLFMjo4UrIah5GN11GSqRjGjvLLO1V/3vpM7iFkN+yj53/aDEq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We add new VM flags for sensitive and global-nonsensitive, parallel to the corresponding GFP flags. __get_vm_area_node and friends will default to creating global-nonsensitive VM areas, and vmap then calls asi_map as necessary. __vmalloc_node_range has additional logic to check and set defaults for the sensitivity of the underlying page allocation. It does this via an initial __set_asi_flags call - note that it then calls __get_vm_area_node which also calls __set_asi_flags. This second call is a NOP. By default, we mark the underlying page allocation as sensitive, even if the VM area is global-nonsensitive. This is just an optimization to avoid unnecessary asi_map etc, since presumably most code has no reason to access vmalloc'd data through the direct map. There are some details of the GFP-flag/VM-flag interaction that are not really obvious, for example: what should happen when callers of __vmalloc explicitly set GFP sensitivity flags? (That function has no VM flags argument). For the moment let's just not block on that and focus on adding the infastructure, though. At the moment, the high-level vmalloc APIs doesn't actually provide a way to conffigure sensitivity, this commit just adds the infrastructure. We'll have to decide how to expose this to allocation sites as we implement more denylist logic. vmap does already allow configuring vm flags. Signed-off-by: Brendan Jackman --- mm/vmalloc.c | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 7a8daf5afb7c..d14e2f692e42 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3189,6 +3189,7 @@ struct vm_struct *remove_vm_area(const void *addr) { struct vmap_area *va; struct vm_struct *vm; + unsigned long vm_addr; might_sleep(); @@ -3200,6 +3201,7 @@ struct vm_struct *remove_vm_area(const void *addr) if (!va || !va->vm) return NULL; vm = va->vm; + vm_addr = (unsigned long) READ_ONCE(vm->addr); debug_check_no_locks_freed(vm->addr, get_vm_area_size(vm)); debug_check_no_obj_freed(vm->addr, get_vm_area_size(vm)); @@ -3331,6 +3333,7 @@ void vfree(const void *addr) addr); return; } + asi_unmap(ASI_GLOBAL_NONSENSITIVE, vm->addr, get_vm_area_size(vm)); if (unlikely(vm->flags & VM_FLUSH_RESET_PERMS)) vm_reset_perms(vm); @@ -3370,12 +3373,14 @@ void vunmap(const void *addr) if (!addr) return; + vm = remove_vm_area(addr); if (unlikely(!vm)) { WARN(1, KERN_ERR "Trying to vunmap() nonexistent vm area (%p)\n", addr); return; } + asi_unmap(ASI_GLOBAL_NONSENSITIVE, vm->addr, get_vm_area_size(vm)); kfree(vm); } EXPORT_SYMBOL(vunmap); @@ -3424,16 +3429,21 @@ void *vmap(struct page **pages, unsigned int count, addr = (unsigned long)area->addr; if (vmap_pages_range(addr, addr + size, pgprot_nx(prot), - pages, PAGE_SHIFT) < 0) { - vunmap(area->addr); - return NULL; - } + pages, PAGE_SHIFT) < 0) + goto err; + + if (asi_map(ASI_GLOBAL_NONSENSITIVE, area->addr, + get_vm_area_size(area))) + goto err; /* The necessary asi_unmap() is in vunmap. */ if (flags & VM_MAP_PUT_PAGES) { area->pages = pages; area->nr_pages = count; } return area->addr; +err: + vunmap(area->addr); + return NULL; } EXPORT_SYMBOL(vmap); @@ -3701,6 +3711,10 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, goto fail; } + if (asi_map(ASI_GLOBAL_NONSENSITIVE, area->addr, + get_vm_area_size(area))) + goto fail; /* The necessary asi_unmap() is in vfree. */ + return area->addr; fail: @@ -3780,6 +3794,13 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, size = ALIGN(real_size, 1UL << shift); } + /* + * Assume nobody is interested in accessing these pages via the direct + * map, so there's no point in having them in ASI's global-nonsensitive + * physmap, which would just cost us a TLB flush later on. + */ + gfp_mask |= __GFP_SENSITIVE; + again: area = __get_vm_area_node(real_size, align, shift, VM_ALLOC | VM_UNINITIALIZED | vm_flags, start, end, node, From patchwork Fri Jul 12 17:00:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13732002 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4214DC3DA45 for ; Fri, 12 Jul 2024 17:02:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EE39C6B00B2; Fri, 12 Jul 2024 13:01:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E6A0A6B00B4; Fri, 12 Jul 2024 13:01:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C6C3E6B00B5; Fri, 12 Jul 2024 13:01:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9B1016B00B2 for ; Fri, 12 Jul 2024 13:01:51 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5CD3180BE3 for ; Fri, 12 Jul 2024 17:01:51 +0000 (UTC) X-FDA: 82331717622.18.F175EB9 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf15.hostedemail.com (Postfix) with ESMTP id 6A04DA0036 for ; Fri, 12 Jul 2024 17:01:48 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="3IU/E2X1"; spf=pass (imf15.hostedemail.com: domain of 3e2GRZggKCL4negoqerfksskpi.gsqpmry1-qqozego.svk@flex--jackmanb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3e2GRZggKCL4negoqerfksskpi.gsqpmry1-qqozego.svk@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803673; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=L6MmdIw12rXfqeUQQj88z73bFenIU6DgFXoL45UzNnM=; b=YngaRiTsrNSk3czKXzDDF8VHUcNHBh/KzHsEFM3LhpJHcU2T+cuYN4BNYfvYMPQvC9SLVK e/AwoPL69tE6ai5SNo4IGuc3lnRXq750wGjumqaQTXemA+xZlAMh9BLyoGBjLSmzTtZqR7 WEvh3in5PQRQNMOzpjuzgeFr34jI8Qs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803673; a=rsa-sha256; cv=none; b=TSvlaXW7hNgZP19Kbx0Nm6gUwNbu0h2z7InD2n9a6fQfFWrUwLgOBdWDgbqgBLsfxI4RdR hf9ogMnQAJ2aIzp7u8vU4+lubOi9zboV+9PLgB9hfdww82pR5KgA7pAtjv8t7YdRIBRpRE LGFmvWYfbupCnXxtv0dQEZGGc9V/668= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="3IU/E2X1"; spf=pass (imf15.hostedemail.com: domain of 3e2GRZggKCL4negoqerfksskpi.gsqpmry1-qqozego.svk@flex--jackmanb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3e2GRZggKCL4negoqerfksskpi.gsqpmry1-qqozego.svk@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e039b77a040so3804153276.2 for ; Fri, 12 Jul 2024 10:01:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803707; x=1721408507; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=L6MmdIw12rXfqeUQQj88z73bFenIU6DgFXoL45UzNnM=; b=3IU/E2X1tCYquSqpMfgAJPAfiCqt5eRgk0S86C75awrzryI22d4OMDdxwucdtaYRgN ShFvO76q+3mgc5KtvxZPBmElonyqC4CYeB3NhevtQjS7a2PijF+NNis7s0ZEuHuqI+R0 Znd5vrIDby89VhBMsb8jzwMymYl8CfASIXs+0PQMXVp4/bksXF7ntCByAxOJaWcxTO6d +z0Qj9UnA7njMpfqzMD397MA6AXnxSSd6JU55IXQLdGdzoJjz0qjFJXGXxj4TVcNpf79 FC2fh6QjSLrMU/ForjBRtjaozEvbfUfbb1uK50XbBzYJoLmyilH8tz6lfLysw5rwA/JX SfjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803707; x=1721408507; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=L6MmdIw12rXfqeUQQj88z73bFenIU6DgFXoL45UzNnM=; b=jOMaxTJwtrwubIZJbTKQn6vTZCHNDxVp+DcCnvgxzgJYHNzKUfLD42JaYKuIU/Qov9 PwfvtRwvlUbNwN5vuto9JtJtuMY503Nu2oPy7bT/vPWd6oOpDAOoyyZo9DIjY9AGrv4F SLiVwp2NqKkAPSPB8wVYKi8hEFPy3v1sbJkQTw2kPj8x2xWwsxRPXvctptEvQv5zvCeU hnrXNXNlBthudw3NASY3uXii423Mqe8lAShtTpYAjVLJv4nFUH8Icx1yrn4BN4dPfIry fbaVfL2bjR0itqRcRX2dxHGiPbIJXP+AdOAXvBCADED/06VSQtmSH2G1aTnWnb0qN0oU foyg== X-Forwarded-Encrypted: i=1; AJvYcCU55KNyu/p/0dNVnOFDkPhfxEP5Jb/2cLzSJ59zdgFmSGRm2ORgbKO5SEWVeYm/5yTlNAo/M3N/o7VRBM4Ey7BSXdk= X-Gm-Message-State: AOJu0YyOpiHnre0Bd+qe8g4ygfkfGXigpzVexROflTSo2nX/T5CRfg3s FyIl874/IaPUVEuK6EYZMXANNv1YCvu/ciqULcDAbj+99lBwPS4HITOa8zh0wfTC53QWkzOBlpT 2GjxRi8Tafw== X-Google-Smtp-Source: AGHT+IE68Lhuxh26ghjRhhLaHrJ9YduMOpsYsuWnIFTPqHiwg/OIUXbqUWhmNqr8FCCI3K44ay3Zr7IW+9N5Rw== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:6902:2e0d:b0:e03:2f90:e81d with SMTP id 3f1490d57ef6-e041b14c989mr757806276.11.1720803707241; Fri, 12 Jul 2024 10:01:47 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:37 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-19-144b319a40d8@google.com> Subject: [PATCH 19/26] percpu: clean up all mappings when pcpu_map_pages() fails From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman , Dennis Zhou X-Stat-Signature: uwa98cua3ts9i7mfmeuaqdsj4oqbyttc X-Rspamd-Queue-Id: 6A04DA0036 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1720803708-402162 X-HE-Meta: U2FsdGVkX19njWNfD+Jvf+EhIMa615d8M8ImxaDp4m7GLcwJ6o6XnmFC93uWYXohFoqe0PsokOJhCJkFKN9fMMlKHY9k3Q9q7bnzOMdQOmTZbg1M8S5zlK0jfnQv3OgWFuApg3HdUqeL7dBFSPJFHybZ6aGBpPU5Ow8/9e+YBI0wEchYOfb1fJorqStktwY38qksfaABMQlPr9Y9ADnpIN9SB8y4UX6inrVFKaq5SEGzoFsZwOjiDPiMhylOKkdeW2VIZj2dRdVXCOaGBTwMmFLFVCdjITw6cKEKSYQNcqWUf0hrWh22OyXMHe0+Kt8wHqXQYsuvoMtiXXLK84WaNwr67dpmuth5A0giMcXnC94cyB0SUbRCYXqzzDWUBXnpYstjOInDkDjde6advKyrbUqRkuVQAon2bhsgHG7KZF69HphwW3NrNS1oH+Htm/w7pXUtXLnEA3xBYlOKBmTiT1cqO83x8ilYNIB7XuaQhRCMbhhddQD9Ro4Mq94mqWco+BFob9JboxmRGUUxjbd/OL72Kq/efhXqwT+1sjr5sdSuKsOqF7I0xj+Q4C4usutBYBQODPjZw8HJSk0Xkhc+tuf2nw4DQjTCmPdKr2ldXQOZ0I6jTD3oDw7wudTq3WzQt6JpeXXLU7L/zOFKBtgBDQZKrJE3dk7bFhnKQc1s9Hjb7h2nnvSdpzdjAF/KsgDD2rYLOlKTFBz2d8REezmXy4aOszcz56j8hnwBzpYUQFkVRcYykh4JFiqOVWFPDxc5pjdTmp2KN3Ijyth2+mEU5+twRZ7mMjljYSejUdWds9hPMksqz1s5lokScKl01Ulgo9XNFHVbjW1INeMM2mMTPzrlnklqw0FEaUpfJXvENjJWXMu+ec0Dh2Ye/3JqPrwln5UVuDMCn4JQgGQ3hHjxcMNqEj6mMaQfORxb/LNbjD3sRyCM82llaUb0+v1PkiHs7MaU2qyDL77jUUguGLh uDG57XEb YF4JnOBH/skjOOsnSvZO3ukNNS3/NjCeAQ6jfAqREV9JaAmpbG34LUOlA3GWF4EyCM24Zirgr+pr8yhr0KSdot+pL34MrGp2cInax2RFcwkcTPon+XRiSgQUyYnZC0PVgVQbdZq8ljoZHVnLQHCMRdrEa7ER/q933icnwHqkzbiUJemZx0v5V0+YoqIooOfhfoMarRCO+UlnbeHGSLw5AbbCbkUgzoz7Zi0zL+FSyqzV/ny39SuQ8GCyrti4oIsbre/1ivIYrG5rwhXsVZZL1xp5b/fXkHOXQ3HPFdYzpb9vtFWx51kliimzaPOc5/C6VZSspp84VY0Wo0r3d21jecmRBd5bYXtUbhKdU5ej5HDh0tMWpOjpnds0qWEQIJNzyUURNB6qwk5X8AxyecjGRjP7CqO0RRmEY4JXTI1kwVHhn/CH1iswR359i6LF2QzjIwhNue3Kgi6MmIUsYAvcC5Ql7oeD6YF8uQgyXHHrn8SAHito5qz10C6BTNHuED6meklSO/kMCD4f3RQCsDQN1T5OcuHM7K/+VfN5S/KqEgzb4voX+XDjiJ7JDWM5aE8Y8z5upawmFd2t3rqv2MUNjgYPPnLkWJmR4J38GFaQiwCcjq2JeTqZ6n2EPeQA/UZcefKOi X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Yosry Ahmed In pcpu_map_pages(), if __pcpu_map_pages() fails on a CPU, we call __pcpu_unmap_pages() to clean up mappings on all CPUs where mappings were created, but not on the CPU where __pcpu_map_pages() fails. __pcpu_map_pages() and __pcpu_unmap_pages() are wrappers around vmap_pages_range_noflush() and vunmap_range_noflush(). All other callers of vmap_pages_range_noflush() call vunmap_range_noflush() when mapping fails, except pcpu_map_pages(). The reason could be that partial mappings may be left behind from a failed mapping attempt. Call __pcpu_unmap_pages() for the failed CPU as well in pcpu_map_pages(). This was found by code inspection, no failures or bugs were observed. Signed-off-by: Yosry Ahmed Acked-by: Dennis Zhou (am from https://lore.kernel.org/lkml/20240311194346.2291333-1-yosryahmed@google.com/) --- mm/percpu-vm.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/percpu-vm.c b/mm/percpu-vm.c index 2054c9213c43..cd69caf6aa8d 100644 --- a/mm/percpu-vm.c +++ b/mm/percpu-vm.c @@ -231,10 +231,10 @@ static int pcpu_map_pages(struct pcpu_chunk *chunk, return 0; err: for_each_possible_cpu(tcpu) { - if (tcpu == cpu) - break; __pcpu_unmap_pages(pcpu_chunk_addr(chunk, tcpu, page_start), page_end - page_start); + if (tcpu == cpu) + break; } pcpu_post_unmap_tlb_flush(chunk, page_start, page_end); return err; From patchwork Fri Jul 12 17:00:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13732003 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09026C2BD09 for ; Fri, 12 Jul 2024 17:02:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 36F876B00B5; Fri, 12 Jul 2024 13:01:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F8BF6B00B6; Fri, 12 Jul 2024 13:01:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D6436B00B7; Fri, 12 Jul 2024 13:01:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D9ABF6B00B5 for ; Fri, 12 Jul 2024 13:01:53 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 928D11A0C11 for ; Fri, 12 Jul 2024 17:01:53 +0000 (UTC) X-FDA: 82331717706.29.207236E Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf22.hostedemail.com (Postfix) with ESMTP id 9D909C0025 for ; Fri, 12 Jul 2024 17:01:51 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=aaD5WhFb; spf=pass (imf22.hostedemail.com: domain of 3fmGRZggKCMEqhjrthuinvvnsl.jvtspu14-ttr2hjr.vyn@flex--jackmanb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3fmGRZggKCMEqhjrthuinvvnsl.jvtspu14-ttr2hjr.vyn@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803666; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vODRaOCa3ahOVXhr5qs8R058obZ/hmyGBbEWzV1lmI8=; b=NfHzXVAB46TpbwxMd/aHatNvnrFdKpIH5lt0/3oIAWnDZJI4qTXKK8VBG7AKL7uNwNbECA b37LbsXfcCq/XNd6U1b4svv/Jce1Ejc3/1W+GzWog3kismhvA5TRah8mOGtEFPsax6Tn+e 9TNTd8Gov4WkKfolpGOLldDXvXUAEIM= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=aaD5WhFb; spf=pass (imf22.hostedemail.com: domain of 3fmGRZggKCMEqhjrthuinvvnsl.jvtspu14-ttr2hjr.vyn@flex--jackmanb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3fmGRZggKCMEqhjrthuinvvnsl.jvtspu14-ttr2hjr.vyn@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803666; a=rsa-sha256; cv=none; b=N2IMzfsa/Gc9w2Xzk+zO3C8EcwSc+tvqvgXDREmHIaY+XhBZLtCXNAv+avUwu31O4N55ym WG1toja9tbCX//d0IkJU6YNftHW6yy/2CfWPIm4tVW3Uw99ItS4V1TiC3Ato8p+ASmHm6m wlZS6mhEWAA+0jDzcZt0IKDmqYRozKc= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6525230ceeaso40867057b3.2 for ; Fri, 12 Jul 2024 10:01:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803710; x=1721408510; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=vODRaOCa3ahOVXhr5qs8R058obZ/hmyGBbEWzV1lmI8=; b=aaD5WhFbTyT+RQQpcLXsTkATO76oQejk2GAwaQnc5w6rjPJT/MLX8i4cpvH5x4fst1 FdJMqVl7Wx1iGaPaXqGheixxBCVyuevpf+lJ9iwrU3TauxHE0JQYrRfzXKNeqFL9DRcr YhNAvuWoP2f07lZEd4zC6p1eZH6PDjLVLIN9E1R8mCUcR3vy9EECedO6aDzD0EBOa/SN /45Mw0QqLJU8FHHmBxHHoMD8SCeGppTuXmHUMpTYip8zD/6p3i0U4R8h/fzbVDEnHXhR CjVZ/YZvpblsmzgc4cU9Td3THjNTKrzO5GxCW9dxUuepGIpwA/Nbn5Em1E4fypTvRfda WRNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803710; x=1721408510; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=vODRaOCa3ahOVXhr5qs8R058obZ/hmyGBbEWzV1lmI8=; b=njQwJ2xmiumDm97Mdr3sJwTN7aHYJL4J323SRP5uNvc0KLZ3OpW929gCs222JUtd3w kEt1TCTEd5T7tCJU4O/qxWK7dnWP3tMw1i1h+VVSZpn1LRjEOCGD567A0bzEru7JLE0y JHjssOUAvcNdq8mcpZ/Kkmj3hkHpxIN0VWMuU42uDVp8hDwmykRy//WRLynvNsF/cUXp BncJUDo20hyt25yqkCa31OXU7+D0573yMpcL1X01qd1iW+pR1ktfYFpkjR79VmNTQshL xas/DEJlSRuul2ibV22PvbvB1td3qqYHmx1Z48Jixun52j5EPuUMkMNMWj1CmicLmyyJ 0xhg== X-Forwarded-Encrypted: i=1; AJvYcCUKCTsaYbEMHrqTOJ/YKChhdfGVl2l9ERB514fSGhl1a13FBDOfXY6fwVTuSgPZxJWCj2pvlKxNgEEYvD8NAf2W1Vw= X-Gm-Message-State: AOJu0Yz6ojLZqzUqr+aW5IR+ufVi9vvHqJEOfkmqQe5RBfh3hIF/zG+A NGKfpw7o1JIq14rcCsClpREBpSoZhXsxTdoSP6R7HMz+zHpgUyOnDfJgFnVMXHXyAPKsTuuOSUk j/OUe7vINEw== X-Google-Smtp-Source: AGHT+IFZSEZj3VcvmAV2TZcIrShIr4wt15JBT1NWiP/B9JdX5vcjtoCXk1eU0WBA34EhNCZasboijrfP9ezs0Q== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:6902:70b:b0:e03:5144:1d48 with SMTP id 3f1490d57ef6-e041b142c52mr23629276.11.1720803710367; Fri, 12 Jul 2024 10:01:50 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:38 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-20-144b319a40d8@google.com> Subject: [PATCH 20/26] mm: asi: Map dynamic percpu memory as nonsensitive From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 9D909C0025 X-Stat-Signature: pwpa9ay8rho4tsyr9craperggmiw6xwb X-Rspam-User: X-HE-Tag: 1720803711-847929 X-HE-Meta: U2FsdGVkX1+AB1xwPsNYwtA3zuwO6rBdcot4vR4vdi5Iv/ERT6+mHdUZnNyc+AfW08Kjg+qYi8sJQ4lj9DtyzR4G4cdI1/GX5ZyjxkAWU3Dr5BXV8EoJ8E86qqL1k8vjpjoVqWsQ7olWK6nwOiVFIXoqJ3Vl10GXz59oAcqEyw8vfC1lvAVdCrp1RBHZvEFIJPliPQov/L8VjvrCyVWtY+WY8LUr+Zx6zPfjmy2T1TPMZFBCGj3ljtfMP0J1bDhG4ul881EUKN7mGPBTveB312vmqll2BgDaqVDqi/oJEXM0yeiNFpWK+0M/SsVxLnWreEMVVdYDbnGp3hHh/FsBeQkXFsnA+ntAz+SehBLszxVqvGD8lsO1oc4nn+fzQWf1KlyDlZk7uuSSzAx7FGGSv8o/+4Sx5/ztgUw+xNCHvODp+obkeh5ZICmv9V6iBS/rIBEW3FjcmhYQgHl5baoA5KE0c4BSowzxW3N6ujwu3H0kj9OwEANWVmb0yKfqgz//kD232zYH0OkekA5VOSE6rFsrV06lNiPlatW/pnKo6f8WmJGw89JOH9zbJRQFe4TqSSMEIrAaZqC/xgN7mgy7TdbCp9+jlH3fyyZ8AnVFODacdvSl/p5RPSR0ISQX8G7COz18ewJ0hpS6Sud1M34H4fxR3cR3UIfTPbdj/Q86DBRSCqa6vnqHggMYP/f5jhvGtZ0gq1FHjK2TSmo8E+Lm66iYdVzhysCKEziVga+x/sQLjQolkMxtfbWwLB8X+2hzS0qBoHXaZHSc1cZQdI0EgM+y9iA96iH9K4VozSyiXfJgkjWKF9S4H/uHVjDeKgl8W0js1C0nr2mZG3H2iW1C+htKgjocpyPso94szD3Cw3HW46I5hL9e56UloMicmNNect/rVPuuF+ct753L2d4ODfksR+ikPQVIgbtAT8I7vzJhGOr8QBHlky3rFtPQhuu+w2dLHUeesQlOlFyzt8O IES3Yc/P sTtKUjBbGMpreSA04HpvWqe60dh1eJ6Ha6CuoZz9zBBCI23xWQdZ0KvP+M5sY9UfWRAdFx3JgiL6EiLS7JikMiDohstHjjQX8Z8zCSYP6SeyEyv6etjaXJpOEUNNT08G4vSpIW+W9rolVf58Pp8KBNp0+W0WiguXOGQ4UCeVzhkgl1ktkS+b8vf8LBKjgKSgWa920f5geDiM+biWVLFAKC1vFz3E185SSQ/hlYP/84vE5dWOizGF1ggrezzLw8m1CqXlha7p0M/bMhKi66o0WDPpwOV/vKwqC46HECLoJFcguSUw7qw+4MNUAb1WMy6W3ygtURLiLJ8r9By0QIiv2WdRckMtz0WyOG36FoIcmxM8Gb9Lv+ssiqHPjzhx5opYQn2Mu25fZ2HG0KOPAOuKpzwbyGD8HfweLDip1rPC1U2bGMSUEaDpXdjpVRAnTuBzQZJtgxkVLoZMiBmxHkylj+rmdRhEC7361t+H4jpKtd+ZqmoOKwGtR2oa3pFjWdL6reL5Gcx8GieBjTaC4F43h24ua2bGQH/+j69F9 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Reiji Watanabe Currently, all dynamic percpu memory is implicitly (and unintentionally) treated as sensitive memory. Unconditionally map pages for dynamically allocated percpu memory as global nonsensitive memory, other than pages that are allocated for pcpu_{first,reserved}_chunk during early boot via memblock allocator (these will be taken care by the following patch). We don't support sensitive percpu memory allocation yet. Co-developed-by: Junaid Shahid Signed-off-by: Junaid Shahid Signed-off-by: Reiji Watanabe Signed-off-by: Brendan Jackman WIP: Drop VM_SENSITIVE checks from percpu code --- mm/percpu-vm.c | 50 ++++++++++++++++++++++++++++++++++++++++++++------ mm/percpu.c | 4 ++-- 2 files changed, 46 insertions(+), 8 deletions(-) diff --git a/mm/percpu-vm.c b/mm/percpu-vm.c index cd69caf6aa8d8..2935d7fbac415 100644 --- a/mm/percpu-vm.c +++ b/mm/percpu-vm.c @@ -132,11 +132,20 @@ static void pcpu_pre_unmap_flush(struct pcpu_chunk *chunk, pcpu_chunk_addr(chunk, pcpu_high_unit_cpu, page_end)); } -static void __pcpu_unmap_pages(unsigned long addr, int nr_pages) +static void ___pcpu_unmap_pages(unsigned long addr, int nr_pages) { vunmap_range_noflush(addr, addr + (nr_pages << PAGE_SHIFT)); } +static void __pcpu_unmap_pages(unsigned long addr, int nr_pages, + unsigned long vm_flags) +{ + unsigned long size = nr_pages << PAGE_SHIFT; + + asi_unmap(ASI_GLOBAL_NONSENSITIVE, (void *)addr, size); + ___pcpu_unmap_pages(addr, nr_pages); +} + /** * pcpu_unmap_pages - unmap pages out of a pcpu_chunk * @chunk: chunk of interest @@ -153,6 +162,8 @@ static void __pcpu_unmap_pages(unsigned long addr, int nr_pages) static void pcpu_unmap_pages(struct pcpu_chunk *chunk, struct page **pages, int page_start, int page_end) { + struct vm_struct **vms = (struct vm_struct **)chunk->data; + unsigned long vm_flags = vms ? vms[0]->flags : VM_ALLOC; unsigned int cpu; int i; @@ -165,7 +176,7 @@ static void pcpu_unmap_pages(struct pcpu_chunk *chunk, pages[pcpu_page_idx(cpu, i)] = page; } __pcpu_unmap_pages(pcpu_chunk_addr(chunk, cpu, page_start), - page_end - page_start); + page_end - page_start, vm_flags); } } @@ -190,13 +201,38 @@ static void pcpu_post_unmap_tlb_flush(struct pcpu_chunk *chunk, pcpu_chunk_addr(chunk, pcpu_high_unit_cpu, page_end)); } -static int __pcpu_map_pages(unsigned long addr, struct page **pages, - int nr_pages) +/* + * __pcpu_map_pages() should not be called during the percpu initialization, + * as asi_map() depends on the page allocator (which isn't available yet + * during percpu initialization). Instead, ___pcpu_map_pages() can be used + * during the percpu initialization. But, any pages that are mapped with + * ___pcpu_map_pages() will be treated as sensitive memory, unless + * they are explicitly mapped with asi_map() later. + */ +static int ___pcpu_map_pages(unsigned long addr, struct page **pages, + int nr_pages) { return vmap_pages_range_noflush(addr, addr + (nr_pages << PAGE_SHIFT), PAGE_KERNEL, pages, PAGE_SHIFT); } +static int __pcpu_map_pages(unsigned long addr, struct page **pages, + int nr_pages, unsigned long vm_flags) +{ + unsigned long size = nr_pages << PAGE_SHIFT; + int err; + + err = ___pcpu_map_pages(addr, pages, nr_pages); + if (err) + return err; + + /* + * If this fails, pcpu_map_pages()->__pcpu_unmap_pages() will call + * asi_unmap() and clean up any partial mappings. + */ + return asi_map(ASI_GLOBAL_NONSENSITIVE, (void *)addr, size); +} + /** * pcpu_map_pages - map pages into a pcpu_chunk * @chunk: chunk of interest @@ -214,13 +250,15 @@ static int __pcpu_map_pages(unsigned long addr, struct page **pages, static int pcpu_map_pages(struct pcpu_chunk *chunk, struct page **pages, int page_start, int page_end) { + struct vm_struct **vms = (struct vm_struct **)chunk->data; + unsigned long vm_flags = vms ? vms[0]->flags : VM_ALLOC; unsigned int cpu, tcpu; int i, err; for_each_possible_cpu(cpu) { err = __pcpu_map_pages(pcpu_chunk_addr(chunk, cpu, page_start), &pages[pcpu_page_idx(cpu, page_start)], - page_end - page_start); + page_end - page_start, vm_flags); if (err < 0) goto err; @@ -232,7 +270,7 @@ static int pcpu_map_pages(struct pcpu_chunk *chunk, err: for_each_possible_cpu(tcpu) { __pcpu_unmap_pages(pcpu_chunk_addr(chunk, tcpu, page_start), - page_end - page_start); + page_end - page_start, vm_flags); if (tcpu == cpu) break; } diff --git a/mm/percpu.c b/mm/percpu.c index 4e11fc1e6deff..d8309f2ea4e44 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -3328,8 +3328,8 @@ int __init pcpu_page_first_chunk(size_t reserved_size, pcpu_fc_cpu_to_node_fn_t pcpu_populate_pte(unit_addr + (i << PAGE_SHIFT)); /* pte already populated, the following shouldn't fail */ - rc = __pcpu_map_pages(unit_addr, &pages[unit * unit_pages], - unit_pages); + rc = ___pcpu_map_pages(unit_addr, &pages[unit * unit_pages], + unit_pages); if (rc < 0) panic("failed to map percpu area, err=%d\n", rc); From patchwork Fri Jul 12 17:00:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13732004 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A813AC3DA45 for ; Fri, 12 Jul 2024 17:02:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3752A6B007B; Fri, 12 Jul 2024 13:01:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F5F06B00B7; Fri, 12 Jul 2024 13:01:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 123776B00B8; Fri, 12 Jul 2024 13:01:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id DB9B66B00B7 for ; Fri, 12 Jul 2024 13:01:56 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 22319140C05 for ; Fri, 12 Jul 2024 17:01:56 +0000 (UTC) X-FDA: 82331717832.10.D832C11 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf15.hostedemail.com (Postfix) with ESMTP id 149D3A0027 for ; Fri, 12 Jul 2024 17:01:53 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=oiPm41vd; spf=pass (imf15.hostedemail.com: domain of 3gWGRZggKCMQtkmuwkxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--jackmanb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3gWGRZggKCMQtkmuwkxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803697; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6yzlvtdLlwV2xo7wLZNn52PBY89yLp9ZWddfXkgzdEU=; b=syDdReUuymO3tXlu7l6fC5v0DKaIMeU5rdrn3xQ5ppEeC3nTjUT93qDG+1SpIByL68Ccwb cFPsTAHfHdfGpkSq/P7Wts+A+3tuL11Ny32VyggaNd1bmx/PFqCMIl9XGnQSpCvoxze8zM h6xhOeQKpE56eBgA1y6GY1Ieb8dwvv0= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=oiPm41vd; spf=pass (imf15.hostedemail.com: domain of 3gWGRZggKCMQtkmuwkxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--jackmanb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3gWGRZggKCMQtkmuwkxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803697; a=rsa-sha256; cv=none; b=7/MwrRqKh5OEuovUl0KwnEl06Meb2a9Rd4z6awXvtnFRWuZEpeQA0jkJ5VHmet5wb4bI0k n8MthJYiZfQzVF/4mJXzHUbKzKs6MoU2v8tfTbxWelNv45/prmMoM4w4Fc9TuF8gyx5HK7 Ya//s/L+MYAjOcvMvPQ7MmabuKlOi4M= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e05600ade22so3606409276.1 for ; Fri, 12 Jul 2024 10:01:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803713; x=1721408513; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6yzlvtdLlwV2xo7wLZNn52PBY89yLp9ZWddfXkgzdEU=; b=oiPm41vdQvuOW1bzr8/2mdQdtrF9ZVYXjRFnNlHqoNgBu60dquP/P3zF/Z0teyqTt5 rqKd1JNq8lE/zzYrgCaNs1pKflagV1azOyuP5ntztAgom9hz0GEzfdyMAL1YPpUh+oDb FyP7ytovIWpVFarq1YEVxnDNAwVfbZYGgbfOQavp5VdxP2InoFu1Jm5blQpRE5/pGTeM eey3MCpgQ8gAKlj7RwmgY2uDHY4X7etSh17KsDjr0tApdz2wK2YCz2jvPtoUpaJ1DeYf c+NRfNn0Ta03hcIldaQyO+aVSXXzS8w4mZOUuJnZYvsqtmodHyo1bhbf7/QtMvuSQ9KX IncQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803713; x=1721408513; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6yzlvtdLlwV2xo7wLZNn52PBY89yLp9ZWddfXkgzdEU=; b=RmM8gEvmz7pQvF5InDS9n9dWHzq9VR0+NPizDWx2VqUTmnSOmi793ZIQ4ORhVba8v/ sYmBN7FTOYhLg2bOw0LlwK1WgdDTj2czowtyNtrQO6ghJgYkPk75TI27SDJbMMeqDUXr KE047/65+zsbe62/L6McYRpPoSOVYeLx8KNw5lkhxleYgiqgthbzQGeeQiwFHDfAxhzh zGqSvcb7DfCvD9nxzpuHenrNr7+4haThltf1dtDzeeNH3LipNjGXmQSREtOupP8rw9ON gZCSv2UyiG6lrdEIEcIHdgoX4t/3VFttqpS9p4zZ06QNM8p/as9Er3B6RbTls3A8p+1O 10LQ== X-Forwarded-Encrypted: i=1; AJvYcCVFk0hGCiDkxfp0dXD6YY0MshdeQHfSRdjgC7AXm6mmJwLtrvYTajj3yXN2VYDPWTg6ILavu5c4DEZgZj0p0TeBrRg= X-Gm-Message-State: AOJu0YwSLTEEO9WCOAGKhE1ZEuQlD62TsL9DkpfKRYL/VS/mWzLa+p4r 0dADCNXxEOlq+XJ7NNsPizXutjjIxYTjRI+X6OIvhLukgT8PDZVFfhe6iP4voIKAOgW9u0EJjZI 7Xt0LAVZZ5Q== X-Google-Smtp-Source: AGHT+IGnjpXnxbERxy2U1NWxsSb37YpVUWZ9S/9b1Rm5PwazcdcV13kMgwRbwaJighCTP4xDBxvKZC1Kul9XcQ== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:6902:1389:b0:e02:f35c:d398 with SMTP id 3f1490d57ef6-e058a707db8mr92172276.0.1720803713060; Fri, 12 Jul 2024 10:01:53 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:39 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-21-144b319a40d8@google.com> Subject: [PATCH 21/26] KVM: x86: asi: Restricted address space for VM execution From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 149D3A0027 X-Stat-Signature: a39yoypjdhz3iihu4bzrsipry9sppj1x X-HE-Tag: 1720803713-947317 X-HE-Meta: U2FsdGVkX18RcjSIQxBdn/ei2hiYvuktrHsN6M0B/iAM3UjgTgM8ESmKwYD+LQXHjyEkk+RlG+EQrVq/f5NpfjK9u0eWd0dPhANCXmzw//y524kF53hJYfalu7vN2y8XPTD4rAiNaxXFETXdttmr4cvXsBvbtPSkwkIIc0Fdq85VTaITclm6KW9iBtLtDVhQW8wOFD6SbMo+w44uPYpmkBLDoT5btv+N0IS6DHhGlqSCH4n0gyLim/xghGsy7GTyOGGTpUGV35NhR8F+iy4E/aP7OnYO0jeqGpG3wWmzEM2BjJQ77nc6dyNS1yTp55yy2Dacj6ocdKGPYpoG0dIbEedOz+m33DUbtbnhQjPDGwlVQ2s0HjvDcnTutBDAtTB7T0X0xnNCoPB4NPFGIwabTvL1Gpz8f4LsvjLtegNyx8ZJ/M+1O6vb/alBtmm+vlo+8xIkg827083nPWfEsWvvU96vHLjf7x4ZqB/VruLBoyzgHUPK/lnPrSQkIOVl+ammyEMLMOKh2PwtvOj6+o5IJH4bN/jy+zHxdGPdzoF12dz9siE2PRhtMMmGgAfyeyT1+YRztYlh0B19oN9Vnx/FWM629VqKSKhqDslOtCrAZsttamHdNN7g7heYoQ9voR8c1ELEgju+hEMPlrVExaX9dD60gAZK7Hwv8wn77jsqjqlxToTgCKzdqUeMHYrx/3raqvu/2rMVb7jVxCKrv7kCqwqIk7RsnalgzPhINS/YUi4zmhIMBATTwzhEWGpSTcXfjpbsj8+c+dTvcjRtcDyVBTh/+vhJ4/aJYw0SQeCOa4lrVdblmVqDknCTQQrn51aqvZz2JpEmkutPjDegY4AGx+CWHxy2OfeE9UpGIyvjzZXbPqNcrvGpH/FxKhdXPdyltS2S9KGUX4lRPDyTHhmS7wORawxL8Xz3QUb7MhvBWWPGHvpzVsAgzahhJE166x53+4sdR/h57O+zJ+6j7qT 32YTbDwz wMilQB47zPP6cV0HRU7lXcotscZOLUIQ+gYMI6tuj81/FxgUhzx93iADrSkrqHhhsLo/3Onmk5BeeE8eb6UXOgnExUyR2ZvkMILfYuRCydx4sue3A1KCNtYm4XVHSTIco28Al7XfvBK9Nwl1O8+mIPiGMG4IETc/AYzSJ1FUQ8ljLhXX7KTQfYZNChylqxS2P1DMaj6EgKCg/ppVNgBVSsZ1uxqY77O4YJQRgqs+xtYn7ILAlZXaOwXkRSHNch73wYwGjsMpGOYAHJ0M6S3YvWXslFwNCbo1TkgxVzRhfdFFd3/7PEWCzFRC2/lWH1hsGXYIEEc2u4QtrPyA+J/VIj4gsbqUPgPfbKbHq1FK4uCgl5Xm9MXQzbxnCzH7F07I7MrRCl3nnkfonVnJ4KYvdNGL0n5qKVQkbsWGaJaPvGkIUeCcL3hoE4DB3Ii9NITBxXRXcVFrVjlYU0zgy5ZbD6SoXhMwIb/uc+GLAnDH8ublwICepzp8vXCDIXWNhaYgeR/KKqFApsdGDTWZC5AgDRV4eiw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: An ASI restricted address space is added for KVM. It is currently only enabled for Intel CPUs. This change incorporates an extra asi_exit at the end of vcpu_run. We expect later iterations of ASI to drop that call as we gain the ablity to context switch within the ASI domain. Signed-off-by: Brendan Jackman --- arch/x86/include/asm/kvm_host.h | 3 +++ arch/x86/kvm/svm/svm.c | 2 ++ arch/x86/kvm/vmx/vmx.c | 36 ++++++++++++++++++++++-------------- arch/x86/kvm/x86.c | 29 +++++++++++++++++++++++++++-- 4 files changed, 54 insertions(+), 16 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 6efd1497b0263..6c3326cb8273c 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -36,6 +36,7 @@ #include #include #include +#include #define __KVM_HAVE_ARCH_VCPU_DEBUGFS @@ -1514,6 +1515,8 @@ struct kvm_arch { */ #define SPLIT_DESC_CACHE_MIN_NR_OBJECTS (SPTE_ENT_PER_PAGE + 1) struct kvm_mmu_memory_cache split_desc_cache; + + struct asi *asi; }; struct kvm_vm_stat { diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 9aaf83c8d57df..6f9a279c12dc7 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -4108,6 +4108,7 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, bool spec_ctrl_in guest_state_enter_irqoff(); amd_clear_divider(); + asi_enter(vcpu->kvm->arch.asi); if (sev_es_guest(vcpu->kvm)) __svm_sev_es_vcpu_run(svm, spec_ctrl_intercepted, @@ -4115,6 +4116,7 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, bool spec_ctrl_in else __svm_vcpu_run(svm, spec_ctrl_intercepted); + asi_relax(); guest_state_exit_irqoff(); } diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 22411f4aff530..1105d666a8ade 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -49,6 +49,7 @@ #include #include #include +#include #include @@ -7255,14 +7256,32 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, unsigned int flags) { struct vcpu_vmx *vmx = to_vmx(vcpu); + unsigned long cr3; guest_state_enter_irqoff(); + asi_enter(vcpu->kvm->arch.asi); + + /* + * Refresh vmcs.HOST_CR3 if necessary. This must be done immediately + * prior to VM-Enter, as the kernel may load a new ASID (PCID) any time + * it switches back to the current->mm, which can occur in KVM context + * when switching to a temporary mm to patch kernel code, e.g. if KVM + * toggles a static key while handling a VM-Exit. + * Also, this must be done after asi_enter(), as it changes CR3 + * when switching address spaces. + */ + cr3 = __get_current_cr3_fast(); + if (unlikely(cr3 != vmx->loaded_vmcs->host_state.cr3)) { + vmcs_writel(HOST_CR3, cr3); + vmx->loaded_vmcs->host_state.cr3 = cr3; + } /* * L1D Flush includes CPU buffer clear to mitigate MDS, but VERW * mitigation for MDS is done late in VMentry and is still * executed in spite of L1D Flush. This is because an extra VERW * should not matter much after the big hammer L1D Flush. + * This is only after asi_enter() for performance reasons. */ if (static_branch_unlikely(&vmx_l1d_should_flush)) vmx_l1d_flush(vcpu); @@ -7283,6 +7302,8 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, vmx->idt_vectoring_info = 0; + asi_relax(); + vmx_enable_fb_clear(vmx); if (unlikely(vmx->fail)) { @@ -7311,7 +7332,7 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit) { struct vcpu_vmx *vmx = to_vmx(vcpu); - unsigned long cr3, cr4; + unsigned long cr4; /* Record the guest's net vcpu time for enforced NMI injections. */ if (unlikely(!enable_vnmi && @@ -7354,19 +7375,6 @@ static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit) vmcs_writel(GUEST_RIP, vcpu->arch.regs[VCPU_REGS_RIP]); vcpu->arch.regs_dirty = 0; - /* - * Refresh vmcs.HOST_CR3 if necessary. This must be done immediately - * prior to VM-Enter, as the kernel may load a new ASID (PCID) any time - * it switches back to the current->mm, which can occur in KVM context - * when switching to a temporary mm to patch kernel code, e.g. if KVM - * toggles a static key while handling a VM-Exit. - */ - cr3 = __get_current_cr3_fast(); - if (unlikely(cr3 != vmx->loaded_vmcs->host_state.cr3)) { - vmcs_writel(HOST_CR3, cr3); - vmx->loaded_vmcs->host_state.cr3 = cr3; - } - cr4 = cr4_read_shadow(); if (unlikely(cr4 != vmx->loaded_vmcs->host_state.cr4)) { vmcs_writel(HOST_CR4, cr4); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 91478b769af08..b9947e88d4ac6 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -85,6 +85,7 @@ #include #include #include +#include #define CREATE_TRACE_POINTS #include "trace.h" @@ -318,6 +319,8 @@ u64 __read_mostly host_xcr0; static struct kmem_cache *x86_emulator_cache; +static int __read_mostly kvm_asi_index = -1; + /* * When called, it means the previous get/set msr reached an invalid msr. * Return true if we want to ignore/silent this failed msr access. @@ -9750,6 +9753,11 @@ int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops) if (r) goto out_free_percpu; + r = asi_register_class("KVM", NULL); + if (r < 0) + goto out_mmu_exit; + kvm_asi_index = r; + if (boot_cpu_has(X86_FEATURE_XSAVE)) { host_xcr0 = xgetbv(XCR_XFEATURE_ENABLED_MASK); kvm_caps.supported_xcr0 = host_xcr0 & KVM_SUPPORTED_XCR0; @@ -9767,7 +9775,7 @@ int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops) r = ops->hardware_setup(); if (r != 0) - goto out_mmu_exit; + goto out_asi_unregister; kvm_ops_update(ops); @@ -9820,6 +9828,8 @@ int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops) out_unwind_ops: kvm_x86_ops.hardware_enable = NULL; static_call(kvm_x86_hardware_unsetup)(); +out_asi_unregister: + asi_unregister_class(kvm_asi_index); out_mmu_exit: kvm_mmu_vendor_module_exit(); out_free_percpu: @@ -9851,6 +9861,7 @@ void kvm_x86_vendor_exit(void) cancel_work_sync(&pvclock_gtod_work); #endif static_call(kvm_x86_hardware_unsetup)(); + asi_unregister_class(kvm_asi_index); kvm_mmu_vendor_module_exit(); free_percpu(user_return_msrs); kmem_cache_destroy(x86_emulator_cache); @@ -11436,6 +11447,13 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) r = vcpu_run(vcpu); + /* + * At present ASI doesn't have the capability to transition directly + * from the restricted address space to the user address space. So we + * just return to the unrestricted address space in between. + */ + asi_exit(); + out: kvm_put_guest_fpu(vcpu); if (kvm_run->kvm_valid_regs) @@ -12539,10 +12557,14 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) kvm_mmu_init_vm(kvm); - ret = static_call(kvm_x86_vm_init)(kvm); + ret = asi_init(kvm->mm, kvm_asi_index, &kvm->arch.asi); if (ret) goto out_uninit_mmu; + ret = static_call(kvm_x86_vm_init)(kvm); + if (ret) + goto out_asi_destroy; + INIT_HLIST_HEAD(&kvm->arch.mask_notifier_list); atomic_set(&kvm->arch.noncoherent_dma_count, 0); @@ -12579,6 +12601,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) return 0; +out_asi_destroy: + asi_destroy(kvm->arch.asi); out_uninit_mmu: kvm_mmu_uninit_vm(kvm); kvm_page_track_cleanup(kvm); @@ -12720,6 +12744,7 @@ void kvm_arch_destroy_vm(struct kvm *kvm) kvm_destroy_vcpus(kvm); kvfree(rcu_dereference_check(kvm->arch.apic_map, 1)); kfree(srcu_dereference_check(kvm->arch.pmu_event_filter, &kvm->srcu, 1)); + asi_destroy(kvm->arch.asi); kvm_mmu_uninit_vm(kvm); kvm_page_track_cleanup(kvm); kvm_xen_destroy_vm(kvm); From patchwork Fri Jul 12 17:00:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13732005 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5D90C2BD09 for ; Fri, 12 Jul 2024 17:02:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2AB1B6B00B9; Fri, 12 Jul 2024 13:02:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 20D1E6B00BB; Fri, 12 Jul 2024 13:02:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F044F6B00BA; Fri, 12 Jul 2024 13:02:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CE98C6B00B8 for ; Fri, 12 Jul 2024 13:02:02 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4DF2FC0AC1 for ; Fri, 12 Jul 2024 17:02:01 +0000 (UTC) X-FDA: 82331718042.27.55176CE Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) by imf11.hostedemail.com (Postfix) with ESMTP id 52D494003C for ; Fri, 12 Jul 2024 17:01:59 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=IN6Fv3uM; spf=pass (imf11.hostedemail.com: domain of 3hWGRZggKCMgxoqy0o1pu22uzs.q20zw18B-00y9oqy.25u@flex--jackmanb.bounces.google.com designates 209.85.221.74 as permitted sender) smtp.mailfrom=3hWGRZggKCMgxoqy0o1pu22uzs.q20zw18B-00y9oqy.25u@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803703; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LtvK59wkpmv96YBPTwdu+UmQnFP5VkAZYiTxY4Mh4uQ=; b=wanuE70s7BEATPSm5z3M68713Rg06w0HdvhZWVC/mqj3ZtyUFqtPrsnTXYN8UgUvl17AIj vLUimMUpCSLytlfBHnFxRn+AV5jD4okizD3oT9857auT2POrMAkmEc3VD7InCy9Qpy+kOt /+qn5W7EA3uzkYp+WPPK1qtUrEhNpQo= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=IN6Fv3uM; spf=pass (imf11.hostedemail.com: domain of 3hWGRZggKCMgxoqy0o1pu22uzs.q20zw18B-00y9oqy.25u@flex--jackmanb.bounces.google.com designates 209.85.221.74 as permitted sender) smtp.mailfrom=3hWGRZggKCMgxoqy0o1pu22uzs.q20zw18B-00y9oqy.25u@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803703; a=rsa-sha256; cv=none; b=xSSgyuoQPay7VpheRrujCZFpMzZ5l4NhB/u6ikJtpfJukTCvi9dw/Fye3BP9/19H15Y/0f 2LlTkbZx3j4I+VLH8AAhA7IZ/crbq3u9XYFBijwC1CX/YifglfWRwClTOty1CHUzf7Al7T ulOMiKozDu2WDAdxh7uHJ7SYh92kldc= Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-367960f4673so1771696f8f.1 for ; Fri, 12 Jul 2024 10:01:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803718; x=1721408518; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=LtvK59wkpmv96YBPTwdu+UmQnFP5VkAZYiTxY4Mh4uQ=; b=IN6Fv3uMAS72pBSelDk0+TBW/1wiDu0DSgM80YQsO17b5kbaEqxMbHaQ6+o8fdWkmE JC4XZJ0Zb2OZqA88AzfvHWHauqLg2jYD1qZeXNUbUPGgC1EKgSZvFzw2FfCAy9+kj88U YdJj5vagCUA7/vTGzYKI6Nk60kpgknoqKjTOtnzNUhms2GXAM11z/9pxdu8nont454Dm 6bqSKFqsFAG8aRE+kKB2Ly43Zc1mpnxg9zO9gZEU4lLGxnP/+6DFNFubWXtDad4AA5a3 XndddP5Up3FWNJxCgWz3pdG+otzASADZbUwR/C4hqz7j9Z67lkBgugI1Bji/CaAInGEW 79oQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803718; x=1721408518; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=LtvK59wkpmv96YBPTwdu+UmQnFP5VkAZYiTxY4Mh4uQ=; b=ATRBLBonPQW9LY1iK4wc9RfAvBiZG9QDhnP7AnN1f2/EYmmn5N5qf7JlZLbyyfIbJH 2fVwHulCK1gjgqDwsvnkVgeTTLDx3O6TZxiDXYqeYg7A7cQpMrF0NReBTrRXAkvO2akp hTvalj9jQTmYXQ01lR5eKD1XSKzLruNV4E6HfvOB7dGlWmrd5K4hCvYdngOqo/UV1wM+ +sBqgbICcufeTOYbygmS9qi/g8w51Gx7TZTlOcsGNhgY3UuUWxGtcBt9omPLTxeXRP/5 JTE7IEQ/owfFKxB99ZcZuvieXFm6PWLZKZ7MMIaILe6TKBBlpIb7Q9VGIs1UJYlXDyR3 U2+g== X-Forwarded-Encrypted: i=1; AJvYcCWrkrfPwKdueSyuG2uibnl/+/+joeSCD6nS7vfWRmqv/wr2qWzwNZwS9601TzPyvrXGoV6OXTUZU5UxnCw8VBlDjps= X-Gm-Message-State: AOJu0YzqRI9kPjcfuzNjOv3zJEFFtW/pHRB7m50oNny/haN22p5h8S5g cmohIOtINY3NGd6cpHRNxT3nWy/BRfHYctcAOKfNVwEEjfrCa2Sd71erMWJBouiJrwxjRgTRfXT 9K6IYNRbi+Q== X-Google-Smtp-Source: AGHT+IFpeHjscoHoV+i7b46AZT6D1f9aKe2Nj5oFbdtPCtyqFA2ot3+sTCgSvjfbQ/uIAyhmtnXZfQlaZ6B1Pg== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:6000:2ae:b0:366:df3f:6f98 with SMTP id ffacd0b85a97d-367ff696f10mr6916f8f.1.1720803717468; Fri, 12 Jul 2024 10:01:57 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:40 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-22-144b319a40d8@google.com> Subject: [PATCH 22/26] KVM: x86: asi: Stabilize CR3 when potentially accessing with ASI From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 52D494003C X-Stat-Signature: 4c59easnqs98keitu5bmkopjwppnbcfp X-HE-Tag: 1720803719-284027 X-HE-Meta: U2FsdGVkX1/iTF/PztUZ53ajh4hPbOjMSxA2A1t/4pdrM2Ed/RbEL+5Pe1Y6a9c0kn3Ss+qu1fEMMgrqJot3FFxbo5iNRw2TCFUnzUsSGS6HBF+0WYkFRrpDmPAHpk3kqwEDrtAZlixA2vjtpom8rDCS2LOhE7ndj4zNTyZhwEL6dhLxrVQILRJk7iog70P5w4F4I2DhH7VDSgHDGzTbKaNz5YWVj6CRYfNil9I3OFfOIxtk5YYuhwABMhy2cl/TkTfbp9TKOZdzHD7tt9t8WCVYSlyGwq/Htzbbrc0d5syInNgzl9zFEJ1FteNefKS7PKGOrhcCPhdUjq12ZjXRlgg6euYHM8/7pI2EiXl3BHfY9iDB1DFuwx1JJim5Fal5M0Dcw2mYFNUOZ6q83GHMNftgf+Onq9NC5ohCecAAbA3AVV7VbhYc57rZ6IxqDRDE09TFrYObJQ7OrQWqcpcJHhCL6lqmEXNsL6UZLyTf4HGjIBgBpfc3Q/JQxfwCqxrjYYMY8QCs/vvDOu6WJ9JCX+/cGrP6kCo5I9DX1eCcbP9eIi+OFN+nrfp5Q6bOro1oN9dBYA0H86uGRl1LI/1QACYZWUjUDfkPpAxGWV0sLDDRT8LpYkq/M6edu8iKo1NKkCEELy7SOjoZ00CorICIjOq5jcDnGWGhLZOFQVlgut4+Yp9/O5RJIwj4vcHAMbUFAkcwPDN2uKy9Je9b/Ck1E1Do2zrZhhUYOeLn2Jg4oAGoEgNnr2KOIaFO5ZKuKZ2dCtMw8kfAMRvvIBGQe6VkH1n33nY88BTjH4PqPElvL4dFKJUZXzQDLQFoOe8RDrky7FOINfZgmEFuh8FW4ur+Wdx2OLyAdEUz6fVFk4BNsOODE33YtH7ZqQ3CRW28Yi5NBXt4B2qYxJcpxGzYcNGB9ANPg+3zqawFK6bZSFTJTOOIKffliomQm15d/nC167GyoiveW3sOxuUVTaiziGL mZHNt1ap CzLdZ36caufFyOQ8HLxsLVoMxZiKK6qR4sBRHvemwImgy3RhUM+plyX2aU87/y0rn4FRa4x0mpfAo6OtNqqPAaod7nZbMGGvVw1MeEaG4kFGyVYgjgb3fWJV1juBRTYiGjrGFfzKb/rRhkal37JY0/GP6jtFMCMewh5VNzpMrQ0GQYgYfk0n+Obwp1g/qJM7KLm0z8fJwPiOPdN6dcnN2fXtymF/hXudjr7P3TzJRk4PeEY1E4QLreKftpRg4vXILwFMVUB1K65w3LuOp5k5U16q8vgyKVxOHXvXEvgo30Wem4IlJQk3f1MtDHseLufFEOHL7hvV3Vpem5+fhGYiYbwWZF5XzjEEFMRNwfPJczt9R7gntqoVRGZtb5wtFfSuDDewhHdBbXyvaJYRzJUaf5lqMWPbE/Rwoj4wgbiIv88Ri3hskck8kg7oj13LzDsEOQwopttJ+Gw7WHDD+ENjVVnIQ6htudB8r57QCTuCWy4FBEpni3pRjnUt3AAWqBRqUWC/IA25yz1G5vCkozvjRyN69cjtdaZBO5IwV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: nested_vmx_check_vmentry_hw() does a VM Enter as a "dry run" to check the VMCS. It's important that we VM Exit back into the correct CR3 in order to avoid going out of sync with ASI state. Under ASI, CR3 is unstable even when interrupts are disabled, except a) during the ASI critical section and b) when the address space is unrestricted. We can take advantage of case b) here to make sure the VM Enter is safe. Signed-off-by: Brendan Jackman --- arch/x86/kvm/vmx/nested.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index d05ddf751491..ffca468f8197 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -3147,6 +3147,14 @@ static int nested_vmx_check_vmentry_hw(struct kvm_vcpu *vcpu) */ vmcs_writel(GUEST_RFLAGS, 0); + /* + * Stabilize CR3 to ensure the VM Exit returns to the correct address + * space. This is costly; at the expense of complexity it could be + * optimized away by instead doing an asi_enter() to create an ASI + * critical section, in the case that we are currently restricted. + */ + asi_exit(); + cr3 = __get_current_cr3_fast(); if (unlikely(cr3 != vmx->loaded_vmcs->host_state.cr3)) { vmcs_writel(HOST_CR3, cr3); From patchwork Fri Jul 12 17:00:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13732006 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90CECC3DA45 for ; Fri, 12 Jul 2024 17:02:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AE7A86B00BA; Fri, 12 Jul 2024 13:02:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A716F6B00BB; Fri, 12 Jul 2024 13:02:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C42B6B00BC; Fri, 12 Jul 2024 13:02:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 628156B00BA for ; Fri, 12 Jul 2024 13:02:03 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0D53780B88 for ; Fri, 12 Jul 2024 17:02:03 +0000 (UTC) X-FDA: 82331718126.17.1CD364E Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf16.hostedemail.com (Postfix) with ESMTP id 3750918000D for ; Fri, 12 Jul 2024 17:02:01 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Y5edYKPZ; spf=pass (imf16.hostedemail.com: domain of 3iGGRZggKCMs0rt13r4sx55x2v.t532z4BE-331Crt1.58x@flex--jackmanb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3iGGRZggKCMs0rt13r4sx55x2v.t532z4BE-331Crt1.58x@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803704; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tumXnPtU8b3obH923+Z5afsvMmdmRsZkkH9VAm+WVDk=; b=UFgaQzWAq/up8s6PGW/sVAbm0brgyOwTG7P2Er9A/G+fYAWplnNYVGB7JGTziX8Sg3mJ38 Yg+TVv8iyaLp39qHwch4yvADgUz48VBNvkalVt5ffYZX0Yzoswh/mE9GXmKIAPswDr4iz9 iib1RU5r9ar01Eur6/9jTwdKY16zAek= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Y5edYKPZ; spf=pass (imf16.hostedemail.com: domain of 3iGGRZggKCMs0rt13r4sx55x2v.t532z4BE-331Crt1.58x@flex--jackmanb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3iGGRZggKCMs0rt13r4sx55x2v.t532z4BE-331Crt1.58x@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803704; a=rsa-sha256; cv=none; b=7CHkKU9Ozh+PkICJuFm9VEh1EoRwtQmLRLh1jZwjPj08ybLundl9Pn8x7KwWh2vtw13LfG rHvl7k+0MkLrDa3o04JXhu3MMFsttZJS4uEJ4zwVZl9Sbw0XOUHcyjVo4u4wI34xqsONe5 BaoY3urNQ9CN+7lUGXtFecdMWR52P7w= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-64b9800b377so41517397b3.0 for ; Fri, 12 Jul 2024 10:02:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803720; x=1721408520; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=tumXnPtU8b3obH923+Z5afsvMmdmRsZkkH9VAm+WVDk=; b=Y5edYKPZcfX7orr5bir4eTVjnNpYu6/3/bIw331H5lycvAnDb0B4Qux5QxwJa7O1yV 8h0YX1zBDVfP8s4jFx2bN9jBX9PY3Q7LOAjFzGsmjCxJrpeh0xd4LCvUbQCPNtfx5u4c iPBYYYRA8f3yH9F4xEoIfK7+oAVbqHDOmnmLxscLoqYdsgtF8G8PCplPJPIk/toljMTf F1PEsXcKZw4Jqhq86Db1Kton9zkhG65rdP87I2RgcJFEDTxJhWOJZycpslUgbKPmyGWL 9qocYxaigf0arYKx9mNWlaRyGowa9qY0DEVuYzwd7JRa16PfJW7SBBG+J17QUgucS7oF SUvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803720; x=1721408520; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tumXnPtU8b3obH923+Z5afsvMmdmRsZkkH9VAm+WVDk=; b=hCLZhgk92UKWsq+nPOIfWONdke5gQoyITO9MET3XAxQsejvDcbWnvK5NPW8ZztlSxz ClsMJBLKL9bhuldwcVBRcv0DNMIF+zGMR+3peZT+Wrs4Jo5N7mhv9iDl4Tjl52VJMNKx SUq4cLPsfrI1mhSxq2NSgu6WqW8sDM/AieqAVgJhOsKpmbagjGc78uaXqV8M0wxZcHKc L6faLcOJWSWlEvsrUUile4W5hW7UcozyFWtVjja2gNNg25ruqYQQfMgPgJsTT7yqrZH4 5B1qOUsWsjuhuUsEtqN0BFTa6ONW0SaFrl/6JWIl2fUnoJh+DOTwccFEb/lGITY4Iknc by4g== X-Forwarded-Encrypted: i=1; AJvYcCVNq2tNLGRmE2+1puANT4KJCtSxNaLQXtpt6FMTtb0/DoHE6UrClUVh7R7V3eHSnuMFSyflOPxxyMKpjKLaDy9VmPA= X-Gm-Message-State: AOJu0YyDxtzGxoWWukGQPHxeYoxtK15DX+uVzl0uBtMc1HI/UNUVQzZT SzmZX6hz0SExMw1AYRUG6aBgSk+TzSLmtlMMLMJZ2Xmi6OFU17F1dkqIMS5uOycCFNZaf74LHlq yEpKEbw2NTQ== X-Google-Smtp-Source: AGHT+IEh5KZMBf6CBGJRGFZI0bbM06TYxceB64wgc/tD6lYzumVKZafd3yatQIPYK79nMgLcMRU8yR72H65XJA== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:6902:1892:b0:e03:3c8c:e80f with SMTP id 3f1490d57ef6-e058a86d9bamr152146276.1.1720803720133; Fri, 12 Jul 2024 10:02:00 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:41 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-23-144b319a40d8@google.com> Subject: [PATCH 23/26] mm: asi: Stabilize CR3 in switch_mm_irqs_off() From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 3750918000D X-Stat-Signature: ca3b1kjgj8suw9rsfnt3oyejtkw7fnbk X-HE-Tag: 1720803720-752784 X-HE-Meta: U2FsdGVkX1/PR9fh5NbkFAvx9x9/sJoT0JeBjTGqc5K1mQ0BknAqwzx38wiRwxez1AWgruQpbPziUCZ6B61v1y/LJGisIedLGyozjsdC5FKgLzr9dxgAeQDPIVKzLoU8v6SLlwdvjM6zUZbC4dkJkLddMyod/uKMSqnNdko7kNfwp1rl5FnOT5hFMLup8N/7fgElLlYrvFVRwAHpd+xcC1nspAYFY7i9HVXNUbtiGrhQUXbPVkaxVL4FHVmdjHO21003LTAx7Ck7z25WIemrDd1GASzGmnug+f10jnBreO2WfiE5Wm+ld1rJ1HupAQjC2UtmPDxe5YfIh3pIwnKY+P8XsPWJgFzw+qxNjfOaZXUOtaTheYmrPwihZZYoPxHN9Nouo0WWOIN1kfAy757bh4L+GaX8LKIz6MN6ES4ujwXbSVXnSoUeDctESYyuUQWt+rq46OzuH+/LWOjmrjXdyOMYcHd2rSrKQ7rU67k1aXx+nOqAYzrWEglCJXxDVuKTtydAG32B2h3I589A216oVHSF9cM0WanPMQgWz4UszaSo3LpUbrfsdunYBWbeOPG9l1KWEKXQ1S5gwDn59g3CP9nFPOjpwz5PLxEBFXODat7h2FGQtsi6K65/XRDx5TpOpNsghVP1DmpxfL399zTSB46xvGXwwtK8ScW2friLHS9UKmvChZHxyevDfvjoIOdH981ivq15KPCe0PldSM3trlS02rO6Co47yWP3qgnapj5/p1/xY5ihDyFP7FLjv8R2gWpewgiXehyzhq8j/IrcW2MOrSSJGkENqbb3rgetyw0eOg/eYdEb+7XGXK6P9HMXUBmNnhAKm/T79xaoTVrP+Y/fFvMXmgD8eCMKxL1m1Xog21PdSJxDd9rtu8bBGIhIPPiqWOCbVNYPymEqzfvGOofUwL9mx1lloiVFJsVo6QXgD9905PQHFiYhgm9W2NHajGa8RZ2ioLK2K7L5wZ3 yHoDJgve dBMque9XoVJ5ou/6pQOZLSp1jhV3a9bk382rIxRrSsv8PQrowYJOaJiaQD+7p00UFGsytm2Rxh79vfkbC5t8f5D/InKm/tFBjuC4qrXl1bs/jCPbqYoKwrNb3Km9jT5LGwn2qDArQJcoqLpEI4GIkqveRAuRtY+KrX3wxmkuorzuaREgJmXN0JVeMrSE8mqvBEiPmyQY3WE6pEv7WLB2I2MHCgt/u9svt+AlaYqwPgijv+c6uTDCuWwUjfrpRugdsVRtDdX9FHZtSbePDfBVoemhWXI/OByQxPi9VuUVE8JaBlLkArY7sz0ksUPzz1VoRd8cb2nnPnQj/eMWG29ZuOKvEpUGgmwvxjUCMc4+1Uoe7sXz5fnEN83FGQbfgNRYplgAT7nBTTYEITEGELjnmqucA4+aoE0c+5HwvH7I6326cWUYNq3nLVrV4FJn0Ps3HmCWpB9gorFTxtwPTPaPnwfVhaPYfn+15kIkh1sVkd4+ixcF56UpII8B5/2IWHsZgWwA6/ZCT0Lg/wq3CpYOABH9gqAe0SHoE+a+AvVGtDVV6HO3vQA/G54UKPXBK5pCXjlnFmARknuXDxKAMo65NJLU6WFCKUn/4fYWqk4Nv902rjRs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: An ASI-restricted CR3 is unstable as interrupts can cause ASI-exits. Although we already unconditionally ASI-exit during context-switch, and before returning from the VM-run path, it's still possible to reach switch_mm_irqs_off() in a restricted context, because KVM code updates static keys, which requires using a temporary mm. Signed-off-by: Brendan Jackman --- arch/x86/mm/tlb.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 36087d6238e6..a9804274049e 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -534,6 +534,9 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, bool need_flush; u16 new_asid; + /* Stabilize CR3, before reading or writing CR3 */ + asi_exit(); + /* We don't want flush_tlb_func() to run concurrently with us. */ if (IS_ENABLED(CONFIG_PROVE_LOCKING)) WARN_ON_ONCE(!irqs_disabled()); From patchwork Fri Jul 12 17:00:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13732007 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A3B6C2BD09 for ; Fri, 12 Jul 2024 17:02:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D17D16B00BC; Fri, 12 Jul 2024 13:02:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C97E36B00BD; Fri, 12 Jul 2024 13:02:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC0646B00BE; Fri, 12 Jul 2024 13:02:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 78C8A6B00BC for ; Fri, 12 Jul 2024 13:02:06 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 1523B80C08 for ; Fri, 12 Jul 2024 17:02:06 +0000 (UTC) X-FDA: 82331718252.07.3FB8064 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf26.hostedemail.com (Postfix) with ESMTP id D3EFE140013 for ; Fri, 12 Jul 2024 17:02:03 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JAke7Bbm; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3imGRZggKCM02tv35t6uz77z4x.v75416DG-553Etv3.7Az@flex--jackmanb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3imGRZggKCM02tv35t6uz77z4x.v75416DG-553Etv3.7Az@flex--jackmanb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803698; a=rsa-sha256; cv=none; b=kNHwBzN84GU300TWYW7x6NTbydxkfHHpeM0e2yAlqafjVBXt3Ax/qbkTLkkf8tTCt9Nbqj NI+a8r8BbGe6qtOCgkIvCk8Xm5nMqn2O9Yc9BsD8iwBZa9MP50axBbgZw14LwiVKRp6TuE v7jwu83CF1NK80QLd98U24I9q/wNFXs= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JAke7Bbm; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3imGRZggKCM02tv35t6uz77z4x.v75416DG-553Etv3.7Az@flex--jackmanb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3imGRZggKCM02tv35t6uz77z4x.v75416DG-553Etv3.7Az@flex--jackmanb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803698; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2i/EY2B7r1Db+9uC+ppZJ+s/pV+Uwv0UgdTvi6uATis=; b=bXUBO3WeLhIqMNDXq4yhr6OjjL/9CburV4VgiWn8uVB3kUv94QjFJ/REJdNuECcTTTxvoB Q2uiLTXxS8WiwVWK8uCnv5yPmhL8ctQKh0DhcnPvw0OsN+ojtGRe5Ic1vB9aSO4gkBPJaA Y2hU7NY22Rlyk5/qOCGRV3a330lCCsI= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-65be82a3241so37994247b3.0 for ; Fri, 12 Jul 2024 10:02:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803723; x=1721408523; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2i/EY2B7r1Db+9uC+ppZJ+s/pV+Uwv0UgdTvi6uATis=; b=JAke7BbmaejGHLCyVnduEaXDBK6wtNRVYUWb26hIRrGVErxEvWVzBAr7gwAUTBwkYg CqIEu8lZkOLPbXfzmrEGzebU02L3xhZ8J9lHIQfngwOVEVrOJWU2gkKTlysPP5mypmOb gCS4pfjyB+V8wF7MGR2D8BTKGTTiOnJZ6ApLW7R1N7WN9Mrgyst0rOLY4ac6mmOESWIq S1yh0Fij1ERgQ1cbNUu/jGRaRb0zyisPyqtuOHH6RiTfmwPhRdb4YgEHrxLwe1JBqGR/ L6JETlGS5olq6dbwYIHH86URvVqejlcZ5n7IRnhjZnZguYPOg0b0b5ecAjX/TNRByou4 C4DA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803723; x=1721408523; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2i/EY2B7r1Db+9uC+ppZJ+s/pV+Uwv0UgdTvi6uATis=; b=U98sAww18MQu3Y0DdPDw1sSTR89Pk4ri1yPiqpSDy9x4OZE6P5sVUfXH8l1ZjDQ0dc uDIOQ3H2cDqGe/WmrMAmFxiWQKqdneE50CbySadb7FIQ/4WC49F0MaxWVGSI1P2WmT3C fkKnkKuC/9ruUDDcI+Wqv7gSqUtCqzO+mzVFv7ev9iGXTZxoEeAcPwXAoWJZLsk+qfbs MB4oGiLQbVpHV8IKrPBLH3tRMECJ/Bmkx43l9kvJV7RsnkkS9QpLdRkypuUtwLPIWizg F/7sTderZwNh6ecBJ/pjtTr5tljofEJ3jOgfrpxmKqujKciekAEWKS6L2/wyPn4JVPjF FsJQ== X-Forwarded-Encrypted: i=1; AJvYcCVgxf4keoicM8N5ogAvwrP4mpUpZCdMnmNjN8ygfFyaBJO071GYvH/SuxMEnn09Xnq8G3iDy+xZUrXKYGkvkuWvpxI= X-Gm-Message-State: AOJu0YwgVmnQRovpTAmK86l+Uw6DWx2ZDK9aNoPmFNNy5+lxo/IZXZRX FvWgZpqZjqJ1MawuHYWJbceljSh7HxS8ptN7DtUoWrO4dAmkdZfb1XpA8llMMOmwUbTE7l7Wes1 XwvnqF7dfcg== X-Google-Smtp-Source: AGHT+IFNgdBalbrX784MJSkG7g28jv3ta6SaiIVV2y32vALYWfCkCKFBAZMdIemQeL1Zf8yLiAeIuqRY45gfTg== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:6902:154d:b0:dfe:fe5e:990a with SMTP id 3f1490d57ef6-e041b1134demr24774276.9.1720803722717; Fri, 12 Jul 2024 10:02:02 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:42 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-24-144b319a40d8@google.com> Subject: [PATCH 24/26] mm: asi: Make TLB flushing correct under ASI From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Rspamd-Queue-Id: D3EFE140013 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: ufubqdkkamsfqn6fwsn57gn1xj9xhygs X-HE-Tag: 1720803723-214610 X-HE-Meta: U2FsdGVkX1/YkLSkBTS5ALNhBFlfhFJjw9Dr/sL9sn0wAHi1goIOno+3EWQI/zVc3KYI8wM51YOXhgUwE0ukcsYk9u0CRjvvo011mUmj9ypzPN5NMeI357q6BPK2dtORlTvJIR+Nif4j/TZCyu0P/Z0R1Gj/Xn4srOhHYjnmi8gKBCp5aeZxJXOGBgTXPloVTK053TERs+Zox/1LSoKdh4iFceEEfietNZqdhEgXTCKNadbBGB/QkrM/dvIwOVmTModUiupOhbGmVvRegnipmznboF9lpLochrjA0ehbUcLz/xiycmA+SkLrGh3/VMKJZ9JcnBS6x0BOXd1ydmQ6GAWLITaBreDUF3y0S36MnBdXBuLRlVk0e6VXmrkWbkbq8Ym++iJG/ZeEFnrOjlHq6/uEBG9wD5hmJzi0TTKVLJmLjZVqrFSztdj/1WwyuGxZGBqVCtPYxibvDfZfJiTx1lc0TLZThjAeph4XAQ3Y2u/PpKwSDvm3TFI/zRoVGNvHEYP4dKb/2Smz5xVuraSZkx22z/5YvqyX/n63ekp/JUZFnHMPreBxER6sJmBRax2L/CR9zo8H+f78/fCpNULNlDhaIVvzhoA+Fdl/aDTTZRJ+5VaQt5WKtb4KOnC9wfVUSEPpZJCVX+OMBGTRkKbO+UMCHmzGpy1940EaF+yajYrTVI5/0PhD6Yglfd+vJLQLNrnjkZ8Y5DrWjQyyjrYA1htqR+OBeUzVBrqgyfNs7bSTem9jFOlz7guPMJjK2zWeyT0mmxwmFItvjf4sn/tvKjmTgtRd9yHw4jZpFql0uvbOqn3nEBhzZiBymRN+zvsVpYK3u0Wz8UCTE8nB7mAUwa9gh4GQ9ogQlDDs3uEAaTb5jcHZ7QAeugeCHiuGhQaPEFMeqclKEd50gzSIIGz7STcD+aK6Q8pFmMv8Qk2AopNV0julUmtMnYZl3sdHw5/OfLLihVzwOdquBzUmsrD aG5Kf4P9 Fyw+DwamyqRS8aoEHtq6G9NIxSNnUnwFZyCeh4j6mj38sLZRMUcm9XbYgV5kvZQ7QYJ9mReOc87cTo6zHTisMNzkOpvK6YP2WGUkhwy/+hLJseLhy9k+I5jjNlWqCmeqD5hkdx8mbof7GcLDDTE0Ooecv9K3+pWc1w/EmtJHDlxuEhOw/pgCPgJWTnQiwKUrQibvrFb0vHLlu8oSSHNPVqilmlxoR2c+6bjTyOHbSEJbhheVYibWwI9+pWHw1bCeVzjUL+cbIKmIrS8Lrr6w5z05LJx1UpvWWyTUdDp8/2ugZs6bEb7gUL0sNAa1Pb3ubz8+tXNiyOt/9hpD0QAZ/0CrszhYyrkGrSZlvyh3aGQO1JbFlK+uCp5FQ/zcjfK+TJ9GnxJGUMpMhZPAdoNm9C6fDc/DzPSfmHb9O0h3sMnYjIOeBo6NVw3FAUvXTbY82PCjky1Lga7Iy08ETeoR/ao3e0a02ZRvz3rIEebowlavnKqcGn/DJk67hQoVb/oPU4H5PYGH48Q55B7dYyltIy9z/GnNqVPKngj4e X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is the absolute minimum change for TLB flushing to be correct under ASI. There are two arguably orthogonal changes in here but they feel small enough for a single commit. .:: CR3 stabilization As noted in the comment ASI can destabilize CR3, but we can stabilize it again by calling asi_exit, this makes it safe to read CR3 and write it back. This is enough to be correct - we don't have to worry about invalidating the other ASI address space (i.e. we don't need to invalidate the restricted address space if we are currently unrestricted / vice versa) because we currently never set the noflush bit in CR3 for ASI transitions. Even without using CR3's noflush bit there are trivial optimizations still on the table here: on where invpcid_flush_single_context is available (i.e. with the INVPCID_SINGLE feature) we can use that in lieu of the CR3 read/write, and avoid the extremely costly asi_exit. .:: Invalidating kernel mappings Before ASI, with KPTI off we always either disable PCID or use global mappings for kernel memory. However ASI disables global kernel mappings regardless of factors. So we need to invalidate other address spaces to trigger a flush when we switch into them. Note that there is currently a pointless write of cpu_tlbstate.invalidate_other in the case of KPTI and !PCID. We've added another case of that (ASI, !KPTI and !PCID). I think that's preferable to expanding the conditional in flush_tlb_one_kernel. Signed-off-by: Brendan Jackman --- arch/x86/mm/tlb.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index a9804274049e..1d9a300fe788 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -219,7 +219,7 @@ static void clear_asid_other(void) * This is only expected to be set if we have disabled * kernel _PAGE_GLOBAL pages. */ - if (!static_cpu_has(X86_FEATURE_PTI)) { + if (!static_cpu_has(X86_FEATURE_PTI) && !static_cpu_has(X86_FEATURE_ASI)) { WARN_ON_ONCE(1); return; } @@ -1178,15 +1178,19 @@ void flush_tlb_one_kernel(unsigned long addr) * use PCID if we also use global PTEs for the kernel mapping, and * INVLPG flushes global translations across all address spaces. * - * If PTI is on, then the kernel is mapped with non-global PTEs, and - * __flush_tlb_one_user() will flush the given address for the current - * kernel address space and for its usermode counterpart, but it does - * not flush it for other address spaces. + * If PTI or ASI is on, then the kernel is mapped with non-global PTEs, + * and __flush_tlb_one_user() will flush the given address for the + * current kernel address space and, if PTI is on, for its usermode + * counterpart, but it does not flush it for other address spaces. */ flush_tlb_one_user(addr); - if (!static_cpu_has(X86_FEATURE_PTI)) + /* Nothing more to do if PTI and ASI are completely off. */ + if (!static_cpu_has(X86_FEATURE_PTI) && !static_cpu_has(X86_FEATURE_ASI)) { + VM_WARN_ON_ONCE(static_cpu_has(X86_FEATURE_PCID) && + !(__default_kernel_pte_mask & _PAGE_GLOBAL)); return; + } /* * See above. We need to propagate the flush to all other address @@ -1275,6 +1279,13 @@ STATIC_NOPV void native_flush_tlb_local(void) invalidate_user_asid(this_cpu_read(cpu_tlbstate.loaded_mm_asid)); + /* + * Restricted ASI CR3 is unstable outside of critical section, so we + * couldn't flush via a CR3 read/write. + */ + if (!asi_in_critical_section()) + asi_exit(); + /* If current->mm == NULL then the read_cr3() "borrows" an mm */ native_write_cr3(__native_read_cr3()); } From patchwork Fri Jul 12 17:00:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13732008 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 910E9C3DA45 for ; Fri, 12 Jul 2024 17:02:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A4CA46B00BE; Fri, 12 Jul 2024 13:02:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D5A06B00BF; Fri, 12 Jul 2024 13:02:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D9026B00C0; Fri, 12 Jul 2024 13:02:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 5642F6B00BE for ; Fri, 12 Jul 2024 13:02:11 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 10A8E40C45 for ; Fri, 12 Jul 2024 17:02:09 +0000 (UTC) X-FDA: 82331718378.02.682C665 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) by imf09.hostedemail.com (Postfix) with ESMTP id C57D214000F for ; Fri, 12 Jul 2024 17:02:06 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=38btf74X; spf=pass (imf09.hostedemail.com: domain of 3jWGRZggKCNA5wy68w9x2AA270.yA8749GJ-886Hwy6.AD2@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=3jWGRZggKCNA5wy68w9x2AA270.yA8749GJ-886Hwy6.AD2@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803700; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dYEbdVw49p7KL3LM6Ug+7tiHwUhfthVzlKoBikBuWUA=; b=4rqSRP7pUaNP0gZoNUhK3Sm+g7Qdyb8iGr4YYKxgy7cOiGQXRUFC+TuGzqtuw3GAPG2ZL/ xo2EMi6FCK3MKLsXVB0Uil9aMvZuaPYEJCA//BKUhZswfjVOPfoqqm4jK5N5MLYTQjaCDx IpFRi+l97MWi9lQ5ag+BVmK3tiNeIOI= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=38btf74X; spf=pass (imf09.hostedemail.com: domain of 3jWGRZggKCNA5wy68w9x2AA270.yA8749GJ-886Hwy6.AD2@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=3jWGRZggKCNA5wy68w9x2AA270.yA8749GJ-886Hwy6.AD2@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803700; a=rsa-sha256; cv=none; b=QjBjA8HH2VNpkN7FiMyZG1bGTYttWNXcvh/PyxJKLvVNJCTCuZxCEf4u8sEqH9wf3irek7 opFRQEoJHpXB2AuqpxtIKySjDouE8bwQa9uAxlAegtwAIPzP66RekgEKa38nSNq5RCOXWd Q+bpoZI15fWWABMIuzF1iBBpsvdzILA= Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-42725d3ae3eso15365045e9.3 for ; Fri, 12 Jul 2024 10:02:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803725; x=1721408525; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=dYEbdVw49p7KL3LM6Ug+7tiHwUhfthVzlKoBikBuWUA=; b=38btf74XSJHx4uDhwERRzBlkf8cSTWzd/iR9OdpHqUnf2Q/co7+0lFkaIEaW5MKonx ARG/6Qayc7OX792lpP6tyN5pywDywjg7L11cjsBmHIFCcYLVv0i31hcIBAsHDY/Ci09L AWgMU+bHfXikEW+6OWFjtcJcecK+MWvP47x1lF9AwSETEg8jGYi4m6RSrExRo/AH0QLI CdDtst+zNfndLQJXU79IPRd77Hgdl1XQc7kenSiWgYe0jUhnFLLPc8v10WZBVdF/zjTa VnZ17sBi227r/4hPQw6Eh7lSbdyoum6C85jPH2Ok58j7oXL2LBsLhZ7SzMvyXUs8F4Qx mj3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803725; x=1721408525; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dYEbdVw49p7KL3LM6Ug+7tiHwUhfthVzlKoBikBuWUA=; b=sOV+5pRDITiJt1kltPgkNk023+/nIOpWisyRjUMuQdqqKh+Ie9hWCnS+mTHZmDJoJh c1G517DsIhzQ/nfaqM7YuYeqCvBV0mJ+7mW/GKlEoeiV7l4VKPiDOsTPcEHiabtsQXR+ tH5jgqk6kxl/RVNjLYVfnYGKTrc/lDEQBMQB/e+xUuV05Ce+xgcrmPAK88V3AVFSt94U MW1LzMixrbQWDh7pu+AdpwRW+HaPuLmmH8rJH/1dtg7HS6oYm7w0RxZk0oNgjm5X/mZg HoCIk9WDaFFuWDUPsncv+Yu2yJ+gfIiBwGsNm9d4smfT+n6gFxGiSeBemwy72h3TRUfD 0Qdw== X-Forwarded-Encrypted: i=1; AJvYcCX5HRO19CMp7qpf8UP3ECFULa0PT33n0Qd0PaJPLfhL01VX8QeD4a101DgnGkXCTt/aYZ3aanr/1fNzD2FqeEi+7FM= X-Gm-Message-State: AOJu0YzI++5mfe9uMalDxIIJKC3NEjlunf+VwzfxL9RR8zFKmUl46frw uyco9feAzxEh/rh2szSxCHy2clOcCW70M0c4vnncIUI+FA84EUUsHV+Dd8/ZGkVpErwIoBl3XKg Mfm9C3yyWJQ== X-Google-Smtp-Source: AGHT+IEpFjrIcdozWeccrHJ18iMfCUI+HuVuPHUhyCg9CuNp5bbpeH6dXY+Lf29VqG6hp3YNWuNGMoEtsiBUuQ== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:600c:5129:b0:426:6a14:8e62 with SMTP id 5b1f17b1804b1-4266ff87a5fmr1913465e9.0.1720803725228; Fri, 12 Jul 2024 10:02:05 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:43 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-25-144b319a40d8@google.com> Subject: [PATCH 25/26] mm: asi: Stop ignoring asi=on cmdline flag From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Stat-Signature: bd4wac9ys9j7cu43rsgzq5uahkh3a719 X-Rspam-User: X-Rspamd-Queue-Id: C57D214000F X-Rspamd-Server: rspam02 X-HE-Tag: 1720803726-289523 X-HE-Meta: U2FsdGVkX18fuqCRBdZL+FreciuD1MWKyfjCD0ajmEZ1a3kKT2BxU7eRg8u08xOFjteUdc0mzKsKoLGmuWxsA6k7TlXgsAVbfSteA2pWomWcVuFrVKPkqxKgFT1o37X1SAFlOa4jOZZmccuQJA1g1FgG8l4w1ufUzGdX5sglqMP19lbFzUpDcwYFVM4GJcSaTloRjlxLB27dWKJJWopotsy5Wp4ggKGk8wLWtTyMjIHVB9+DYurHIa99FNuYbepc5PvJKTFc/UbUmatMEFnE5hg+YooTgmTH2dte2mudTAOYIsgHbU/ww067c4Cwuip8l9y96UdIRc5OEX1XucDYhVv4LRJ2YiUtf7oZV9zEc3JsVmVRut3VLxEaSp5mf+nrClPQFfBav97+93H4cjgund7K/4liLimqbnhMGZyeG7GCPDZaxvrZZdfVgNTaOPfhIDwUoZX2aS5SbJ8W56Up01gY1Nv/KP1Os/mpO5kvdHWjQdKkZbsGu8oxV2Ap538NPf7Fc9KUg4rlUjLdENWFB4x9SZ8tumNJH/Z3Z0xAzsdrJPnmh+8BQG9YdMJKCWNvAnRKwyB0NHvQTeS+5/QBAJnD2wUVWjVZ0EayHgUwhYLGpi7qKEdXr3plpW85dO8HMnko42eqdfXWnaeHJmUYv8JoI3Ezx9eNUYzPKP6Ae8es5gp4tCAiRDuSKbd7r447eWut48GT8kDvixB2AlaZaloI34UabvLwPuTSJFuRiEcoZZBehUHEZJJBYdF96updUAHVXBPNNk1LmlkDjqr2iIZRq0R01fccQ9C0wQ13hZflJM6/PUzTUlek3tp8G18o198eHsVM72ohZcQIGdwx5e7vgEjfH908qgN5yGnw332Q0VlySBr5qZGlDcwO/ouRhDBMZ26NSisAU+cB6RYUSco5yRzECz5xoFr7LKDh1qp0ii7zxb8jEUuwcD+DfeudVhPiNyc8gPUAH0h4T8R Wk2WpVe4 etxR3RueqhCpxNUIgPPP6axbL1UaMPVu2e7BJNrUotv0nbT4FOputujc3p3oSvMzvMaKREccyY8fpRNxOjaFXqoc1wccFB7gBJimH2ZdWA25o5/mcsUm/EOvfijJwmVhQZ3YqyaVONECJkmYPyYI2u2yqVH6KMkpG8fmH9Ggv7+7oeIChhQJp8CyW/HTZN60xPbRj6SuOW5ig3LKRAIKoT6D/lDcgd/p1zK1ycxOEgl1OGruoUHxXAJq5NMAG/WQBuTn2ZgCNbnWMrNu5m3k1IOpOvYnP/rC/KrqDkxqmiGZ8IwoOlIJ6WGC9L7qSNf8QrRXOgu1OcVYaGliXek1sqVEFY6T1gVhTR2lOBudo+WHkvtWdJ0GQl/7wDCnRUwucKmxogtQ6Zl4Ke9PzPA00x6hEjA9oYkPvzMuaUaDLEWr2mSQi87n1v1SmJl06Wz2qpL97MJ/7sESrHKrRE5L7mP2pSlwKs5WGW+/UOu+FQuNTDSO0KZaYP3lKHqPUGTFTVE7lDIhvmLetasPRg0Er3oJbn+byisGxzkdV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: At this point the minimum requirements are in place for the kernel to operate correctly with ASI enabled. Signed-off-by: Brendan Jackman --- arch/x86/mm/asi.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index 891b8d351df8..6cf0af1bfa07 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -174,14 +174,14 @@ void __init asi_check_boottime_disable(void) pr_info("ASI disabled through kernel command line.\n"); } else if (ret == 2 && !strncmp(arg, "on", 2)) { enabled = true; - pr_info("Ignoring asi=on param while ASI implementation is incomplete.\n"); + pr_info("ASI enabled through kernel command line.\n"); } else { pr_info("ASI %s by default.\n", enabled ? "enabled" : "disabled"); } if (enabled) - pr_info("ASI enablement ignored due to incomplete implementation.\n"); + setup_force_cpu_cap(X86_FEATURE_ASI); } /* From patchwork Fri Jul 12 17:00:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13732009 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E8A6C2BD09 for ; Fri, 12 Jul 2024 17:02:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DAB496B00C0; Fri, 12 Jul 2024 13:02:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D309D6B00C1; Fri, 12 Jul 2024 13:02:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B34A26B00C2; Fri, 12 Jul 2024 13:02:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8CF406B00C0 for ; Fri, 12 Jul 2024 13:02:12 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D3549140C2F for ; Fri, 12 Jul 2024 17:02:11 +0000 (UTC) X-FDA: 82331718462.05.9BB3D38 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf11.hostedemail.com (Postfix) with ESMTP id B871D40032 for ; Fri, 12 Jul 2024 17:02:09 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=iJTtODhz; spf=pass (imf11.hostedemail.com: domain of 3kGGRZggKCNM8z19BzC05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--jackmanb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3kGGRZggKCNM8z19BzC05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803696; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Zn579dgno7SZboGdojpY5JJU8xi+US4yr/l6Mjf8vz0=; b=L0klBhc0oe6oBWyoZ3r9yE4UL7uwCsMo5fU+pGfs1klpSABjJ6rT8JwnPQInDOT+8wh+f+ MDAkB4NsuVNj9GyHEH3SlP5y5uA3+n1V8C/9S5ElaEPbZ+K3CMK5pMdRBtFuDWupSkqrGA gc3sQ/dC2d9svbCyOWS0SQP8Dly/n+Y= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803696; a=rsa-sha256; cv=none; b=uq7oXNFw9lQs528o2+6pcIjcZTUkS1t4UUW6q2d9BUhD//2nOCMhkMIGE+ATsZhwKGTBjJ g2XmOp3boAkjCOEvDzJ0KkJUNzwX+W3RNHF2toagPacgBUt5honEzvgMk2XAv6qLfG0F5A x5HmjDu6+sBHfdfeeO10K5xe3syfGg8= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=iJTtODhz; spf=pass (imf11.hostedemail.com: domain of 3kGGRZggKCNM8z19BzC05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--jackmanb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3kGGRZggKCNM8z19BzC05DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-653306993a8so36442177b3.1 for ; Fri, 12 Jul 2024 10:02:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803728; x=1721408528; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Zn579dgno7SZboGdojpY5JJU8xi+US4yr/l6Mjf8vz0=; b=iJTtODhzl9dD6qfFJBUIh+yY1gN3IoG8jC2hZp8I5LbwEPqvCTUvaV4QF3aa0SzjgA m6dFrLzPrpZqo7Y5SnknwEODOlY7VMhRsSQf2kkJauTTnndGRggbFqZTDtjsd/CDFeq8 fD+oQFXqjoAtmHqQeUMIJDvP9WgFdCR3L+Cf6R0OaGG/hqgLTsy292dpTQU6w5OPYvrH 6eZgxCc42ds2ZkTKdD0OqcOS3ComgsHW5G3H5o5alwHUYD6+ckdgaRSAawZ7MSGgkRlN OEpee+ZgIXieCdtfBslORAs2EafUGYcUE5erABcgRzKafcfXaTYozJES1TDe1EpDMKa5 W7wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803728; x=1721408528; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Zn579dgno7SZboGdojpY5JJU8xi+US4yr/l6Mjf8vz0=; b=Ij1S1ZBMEidLXbg0TgNkHUNeCBxOeTryVBY2XBcOXSdiWuTE+ZMjHDF2xEsMyoJudn RnSPNRz4Zn1vtpxKOcgWVimzwWZQ8UfuGvmi38B8uIB8+N7xqXB7T3iK8b52EMh1ChW2 mt85o6ogCVBukGtcy+lXukaHYOfo5smKsSDt5q2q4hFeAo+Q0/A67jYYkJILkp1tc4dz k/HKjPRKfW8AMMBRoxW0bu8ROMMqYsTG8EY9zhuQiYXtgVjni7RRB4JJ4K9OVvxzC7PD hgUS7oPqDaio+5d+S2rjg6oVeQkKORXWYCtPs6pRC7jp7fHWqPpIjDd2W74Z6eyCm9V8 wAuw== X-Forwarded-Encrypted: i=1; AJvYcCWgNjXw+pbP2/elvoG6BAFrz9YJeIukzHRbFEyDoGr5RYQzgcFSROb3ADHMK0FosMitGdW4SFV2NMRrlfTjPLqARDU= X-Gm-Message-State: AOJu0YzjmY1CT36V3INsVdsjgPwyXmHnyjgm1EvGW5BF5vN97P2mQZ43 stv3lQnH5BuzixA0TZFlGpqh8OIZl9QK2anVQKsojP1+pFP5zAXiKgZk2S65yIYzVT0MK5/pHgR vPreMErHumQ== X-Google-Smtp-Source: AGHT+IGmf1CJRMIpY+h+xba+0XgEdJs1Sqn1txDmOypcZe8mfXwCEjETMiBlkTEaYdYTy3241JdukP3H8LKm6w== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:690c:46c8:b0:62c:de05:5a78 with SMTP id 00721157ae682-658f01fd061mr625407b3.6.1720803728252; Fri, 12 Jul 2024 10:02:08 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:44 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-26-144b319a40d8@google.com> Subject: [PATCH 26/26] KVM: x86: asi: Add some mitigations on address space transitions From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Rspamd-Queue-Id: B871D40032 X-Stat-Signature: 188y9wtttz1b3au7w1j7jajrijwu6ub5 X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1720803729-473933 X-HE-Meta: U2FsdGVkX18ijRvQnO3aqoWyMrRPNQiZCbhg//BrSgMy2+uEo0G8exiamuWv6/zIqVrr6eLsKYgVxdnjEiOZFAmJz0jGCDYzPuWwguh0Feegm1E5+9YxTat0ZPK0jn3YEkaZzq104+DKAHOg2b5Ej/Xc0SriEjARcJ2dpmFKrCRxeQ2PJaFktKqD/7tXbUhl8nEyJW4bsXkNzfodcETJ0nCBoI0z4DfAH847vDyj86OBqsEkcX7D/N2IqorhUwa+ybLCVjnrDENpKU/sn88QAvekFOvzM/YF78u5ubW3kUJTHooLX53cgUSLHFyuizAaJw6dj74sY1yWtFp2Pd6IemvtFEnt1kHEMHCe+JMrX3F5EErvcBZABF4XTgKe1d/AROzNAWvfvq3PdnQp+GqYY1i8fWyasfxf+KoDZtLVHbBg4s3KB/e4XvwlQkVG+WDULIS4Ofw7MeSA9/XAbJPg/1UNyZUs9kWSamxBGzbOxuix8xvZjSF2gfSdqx3iDzosYrHvWB/A5u+qADDVGBOgZGgkpc6/TcP5ytZFneITFqyVg8rgSO7kWHYiTNhrNtUqGtZUCHUiJNOlAGKXr4gDx1WiugTy5ItR6SwtdnEcRaD7htSjmNqHnMYABpUfDS8DzOm8rRwcECvlzlLy9bdJrSt8juGP5rts0MjczVUnNZ32cyTGVlLMJweY6nU7TWHwKUyHnUWlzVhUHt2+yj3N+/zYWpRV5J7hq69gJdlWjNUQXUipVxvlHanxzBiZTC1n7utdIQmuBkFncnNAmVI58UY24/nzJyhvqfm5DFukEpR4P9qE26mZtjWFX9+YhArnI569KmZydspWCZVpsRI6gWIDFBfQT1d1opKfeiHF/I9dbQBHUiDaQ7V5OaDF6qoPLFFMdP37SKOilVzjWcwy94fwGJty6vZ/GlVx+kp4RIqr8Fa5x8IlTIreAVzaMtpjoM8wIiJAGMnSV8wd0nx YBmgfd/o xCjUE7a2vFL+QERqj4cPejrNNnv1wRpJe5GRB7j/Hf3xeHKa/ItAT4UIej3xAmp9KRkBkoo7/O+eUaCNd4ghXo1Ddt/RWepxvC5Bc4g0H9c+6Kzz0lZu/43G46p69E+LjsdAdxhgRkTtti2Q//kJA7D4AQKh9xTOOamDbS8vh67k6QHg5nplj6b53Opz2g3vKIyC2Z2063GQUbfMGGQT4p3LcwM+z990krLoRbstfm8q2XcYhOhxUDHG2zTkJdcTGsQNo4ASUxdJFKeGP4BqVx1EMTigyqhcJ8t/mVphFfo4O2dlEUwxUhPcSE1C+cZbMG3FUu1xGwiK7Zldl6pOoU+/fas8RkSwF8J2Trf5pUltIWAyoypwPqqsWKjx+RvBzNwgd2h1sQWPkBw6A18jBGrFZuMP6JIUheWqvtjzYBgOg6o2u5+EZOU4sWIUVXu96b5Yq92OwJQt2IS2f4y9iZPgaRno1hHDN8bJCF4LPhwTGFY8gLilvzpIo9n5ID4l8rEaMr18Vh/3K8J7y73GbgBakbQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Here we start actually turning ASI into a real exploit mitigation. On all CPUs we attempt to obliterate any indirect branch predictor training before mapping in any secrets. We can also flush side channels on the inverse transition. So, in this iteration we flush L1D, but only on CPUs affected by L1TF. The rationale for this is: L1TF seems to have been a relative outlier in terms of its impact, and the mitigation is obviously rather devastating. On the other hand, Spectre-type attacks are continuously being found, and it's quite reasonable to assume that existing systems are vulnerable to variations that are not currently mitigated by bespoke techniques like Safe RET. This is clearly an incomplete policy, for example it probably makes sense to perform MDS mitigations in post_asi_enter, and there is clearly a wide range of alternative postures with regard to per-platform vs blanket mitigation configurations. This also ought to be integrated more intelligently with bugs.c - this will probably require a fair bit of discussion so it might warrant a patchset all to itself. For now though, this ouhgt to provide an example of the kind of thing we might do with ASI. The changes to the inline asm for L1D flushes are to avoid duplicate jump labels breaking the build in the case that vmx_l1d_flush() gets inlined at multiple locations (as it seems to do in my builds). Signed-off-by: Brendan Jackman --- arch/x86/include/asm/kvm_host.h | 2 + arch/x86/include/asm/nospec-branch.h | 2 + arch/x86/kvm/vmx/vmx.c | 88 ++++++++++++++++++++++++------------ arch/x86/kvm/x86.c | 33 +++++++++++++- arch/x86/lib/retpoline.S | 7 +++ 5 files changed, 101 insertions(+), 31 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 6c3326cb8273c..8b7226dd2e027 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1840,6 +1840,8 @@ struct kvm_x86_init_ops { struct kvm_x86_ops *runtime_ops; struct kvm_pmu_ops *pmu_ops; + + void (*post_asi_enter)(void); }; struct kvm_arch_async_pf { diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h index ff5f1ecc7d1e6..9502bdafc1edd 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -605,6 +605,8 @@ static __always_inline void mds_idle_clear_cpu_buffers(void) mds_clear_cpu_buffers(); } +extern void fill_return_buffer(void); + #endif /* __ASSEMBLY__ */ #endif /* _ASM_X86_NOSPEC_BRANCH_H_ */ diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 1105d666a8ade..6efcbddf6ce27 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6629,37 +6629,18 @@ static int vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath) * is not exactly LRU. This could be sized at runtime via topology * information but as all relevant affected CPUs have 32KiB L1D cache size * there is no point in doing so. + * + * Must be reentrant, for use by vmx_post_asi_enter. */ -static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) +static inline_or_noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) { int size = PAGE_SIZE << L1D_CACHE_ORDER; /* - * This code is only executed when the flush mode is 'cond' or - * 'always' + * In theory we lose some of these increments to reentrancy under ASI. + * We just tolerate imprecise stats rather than deal with synchronizing. + * Anyway in practice on 64 bit it's gonna be a single instruction. */ - if (static_branch_likely(&vmx_l1d_flush_cond)) { - bool flush_l1d; - - /* - * Clear the per-vcpu flush bit, it gets set again - * either from vcpu_run() or from one of the unsafe - * VMEXIT handlers. - */ - flush_l1d = vcpu->arch.l1tf_flush_l1d; - vcpu->arch.l1tf_flush_l1d = false; - - /* - * Clear the per-cpu flush bit, it gets set again from - * the interrupt handlers. - */ - flush_l1d |= kvm_get_cpu_l1tf_flush_l1d(); - kvm_clear_cpu_l1tf_flush_l1d(); - - if (!flush_l1d) - return; - } - vcpu->stat.l1d_flush++; if (static_cpu_has(X86_FEATURE_FLUSH_L1D)) { @@ -6670,26 +6651,57 @@ static noinstr void vmx_l1d_flush(struct kvm_vcpu *vcpu) asm volatile( /* First ensure the pages are in the TLB */ "xorl %%eax, %%eax\n" - ".Lpopulate_tlb:\n\t" + ".Lpopulate_tlb_%=:\n\t" "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t" "addl $4096, %%eax\n\t" "cmpl %%eax, %[size]\n\t" - "jne .Lpopulate_tlb\n\t" + "jne .Lpopulate_tlb_%=\n\t" "xorl %%eax, %%eax\n\t" "cpuid\n\t" /* Now fill the cache */ "xorl %%eax, %%eax\n" - ".Lfill_cache:\n" + ".Lfill_cache_%=:\n" "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t" "addl $64, %%eax\n\t" "cmpl %%eax, %[size]\n\t" - "jne .Lfill_cache\n\t" + "jne .Lfill_cache_%=\n\t" "lfence\n" :: [flush_pages] "r" (vmx_l1d_flush_pages), [size] "r" (size) : "eax", "ebx", "ecx", "edx"); } +static noinstr void vmx_maybe_l1d_flush(struct kvm_vcpu *vcpu) +{ + /* + * This code is only executed when the flush mode is 'cond' or + * 'always' + */ + if (static_branch_likely(&vmx_l1d_flush_cond)) { + bool flush_l1d; + + /* + * Clear the per-vcpu flush bit, it gets set again + * either from vcpu_run() or from one of the unsafe + * VMEXIT handlers. + */ + flush_l1d = vcpu->arch.l1tf_flush_l1d; + vcpu->arch.l1tf_flush_l1d = false; + + /* + * Clear the per-cpu flush bit, it gets set again from + * the interrupt handlers. + */ + flush_l1d |= kvm_get_cpu_l1tf_flush_l1d(); + kvm_clear_cpu_l1tf_flush_l1d(); + + if (!flush_l1d) + return; + } + + vmx_l1d_flush(vcpu); +} + static void vmx_update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int irr) { struct vmcs12 *vmcs12 = get_vmcs12(vcpu); @@ -7284,7 +7296,7 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, * This is only after asi_enter() for performance reasons. */ if (static_branch_unlikely(&vmx_l1d_should_flush)) - vmx_l1d_flush(vcpu); + vmx_maybe_l1d_flush(vcpu); else if (static_branch_unlikely(&mmio_stale_data_clear) && kvm_arch_has_assigned_device(vcpu->kvm)) mds_clear_cpu_buffers(); @@ -8321,6 +8333,14 @@ gva_t vmx_get_untagged_addr(struct kvm_vcpu *vcpu, gva_t gva, unsigned int flags return (sign_extend64(gva, lam_bit) & ~BIT_ULL(63)) | (gva & BIT_ULL(63)); } +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +static noinstr void vmx_post_asi_enter(void) +{ + if (boot_cpu_has_bug(X86_BUG_L1TF)) + vmx_l1d_flush(kvm_get_running_vcpu()); +} +#endif + static struct kvm_x86_ops vmx_x86_ops __initdata = { .name = KBUILD_MODNAME, @@ -8727,6 +8747,14 @@ static struct kvm_x86_init_ops vmx_init_ops __initdata = { .runtime_ops = &vmx_x86_ops, .pmu_ops = &intel_pmu_ops, + +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + /* + * Only Intel CPUs currently do anything in post-enter, so this is a + * vendor hook for now. + */ + .post_asi_enter = vmx_post_asi_enter, +#endif }; static void vmx_cleanup_l1d_flush(void) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index b9947e88d4ac6..b5e4df2aa1636 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -9695,6 +9695,36 @@ static void kvm_x86_check_cpu_compat(void *ret) *(int *)ret = kvm_x86_check_processor_compatibility(); } +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + +static noinstr void pre_asi_exit(void) +{ + /* + * Flush out prediction trainings by the guest before we go to access + * secrets. + */ + + /* Clear normal indirect branch predictions, if we haven't */ + if (cpu_feature_enabled(X86_FEATURE_IBPB) && + !cpu_feature_enabled(X86_FEATURE_IBPB_ON_VMEXIT)) + __wrmsr(MSR_IA32_PRED_CMD, PRED_CMD_IBPB, 0); + + /* Flush the RAS/RSB if we haven't already. */ + if (!IS_ENABLED(CONFIG_RETPOLINE) || + !cpu_feature_enabled(X86_FEATURE_RSB_VMEXIT)) + fill_return_buffer(); +} + +struct asi_hooks asi_hooks = { + .pre_asi_exit = pre_asi_exit, + /* post_asi_enter populated later. */ +}; + +#else /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ +struct asi_hooks asi_hooks = {}; +#endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ + + int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops) { u64 host_pat; @@ -9753,7 +9783,8 @@ int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops) if (r) goto out_free_percpu; - r = asi_register_class("KVM", NULL); + asi_hooks.post_asi_enter = ops->post_asi_enter; + r = asi_register_class("KVM", &asi_hooks); if (r < 0) goto out_mmu_exit; kvm_asi_index = r; diff --git a/arch/x86/lib/retpoline.S b/arch/x86/lib/retpoline.S index 391059b2c6fbc..db5b8ee01efeb 100644 --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -396,3 +396,10 @@ SYM_CODE_END(__x86_return_thunk) EXPORT_SYMBOL(__x86_return_thunk) #endif /* CONFIG_MITIGATION_RETHUNK */ + +.pushsection .noinstr.text, "ax" +SYM_CODE_START(fill_return_buffer) + __FILL_RETURN_BUFFER(%_ASM_AX,RSB_CLEAR_LOOPS) + RET +SYM_CODE_END(fill_return_buffer) +.popsection