From patchwork Fri Jan 10 18:40:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13935230 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94058E7719C for ; Fri, 10 Jan 2025 18:40:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 15FF86B00B1; Fri, 10 Jan 2025 13:40:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 110C66B00B4; Fri, 10 Jan 2025 13:40:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA4366B00B5; Fri, 10 Jan 2025 13:40:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D074E6B00B1 for ; Fri, 10 Jan 2025 13:40:51 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 7BF2BC0E16 for ; Fri, 10 Jan 2025 18:40:51 +0000 (UTC) X-FDA: 82992408702.24.1C1BFE3 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) by imf13.hostedemail.com (Postfix) with ESMTP id 8E5A720006 for ; Fri, 10 Jan 2025 18:40:49 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=4SQEzOur; spf=pass (imf13.hostedemail.com: domain of 3r2mBZwgKCMozqs02q3rw44w1u.s421y3AD-220Bqs0.47w@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=3r2mBZwgKCMozqs02q3rw44w1u.s421y3AD-220Bqs0.47w@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736534449; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lMV7uJ7I18OKYw8GMGFQsmzHxbOybRxQHWQLaqO2yfM=; b=tVj4cWyhT3RK7TQhz1ydoenAni7enid/fO3TTiujonGt70YZaOOq85yjUp4/OsPgA9cx+8 hb6sFbjDvowe8bnaGpySKm1HpkFFTtH2wY0u3M0+xQPeTEDiQlgcvas806dCITCBzMF/kM D3eYczl07QzLzrAsKyuZvPAu+gl5uD0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736534449; a=rsa-sha256; cv=none; b=XDcNsqaXXfxvym15UfOX6wmP6SIfINIWwAWNHY/gfaQcMXhhqag6kfaHCwRclvc28xinA1 DxXHOQ5cOYZprAHvTTE9sxHgyucvEUD4oOpnxNDGeTn0D8f6/Rt0buaAErpPjPVxNQZGTK F6aAvWKrzR2CXbJG32TLjRS6zEbG4BM= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=4SQEzOur; spf=pass (imf13.hostedemail.com: domain of 3r2mBZwgKCMozqs02q3rw44w1u.s421y3AD-220Bqs0.47w@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=3r2mBZwgKCMozqs02q3rw44w1u.s421y3AD-220Bqs0.47w@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-436289a570eso19444325e9.0 for ; Fri, 10 Jan 2025 10:40:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736534448; x=1737139248; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lMV7uJ7I18OKYw8GMGFQsmzHxbOybRxQHWQLaqO2yfM=; b=4SQEzOuru9inTNsHW+B3wnlAaEIpl03tMTiJkKvKxTEldsceho14pAAGgWjs2hCaC2 f+pKJLLMwp6ekVlMs4E8ESbpqXkyIb7wk7dknO2se0KsciKGyv6/w34OJQoFP9r6MJcQ YUYhdN3AO8bXC42u9bvWKMAhVZrH3dMWwCPiR6tMK9SSas7ihyJ0NodqJ7nQVjX9SImQ K2T01eFYGo3xL6I/zMX0OT0JY2neHid8v3L6h3t0hKxb4DanUt3mbd8PrK+fwx0N54uD IweIYIEmWx7DWGsLp33NuNKb/Ivy0dFdsRXZqnNda3NCjaZK0jqvMDgw2rVWl6iTlEG7 1opw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736534448; x=1737139248; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lMV7uJ7I18OKYw8GMGFQsmzHxbOybRxQHWQLaqO2yfM=; b=fw1OdicGfhX+beEJ8ArGPgkybC5Sk/YY5yuG+8MbW8qkVYeOzxaafXiy/nnQuzHGmQ Z3thySoT6EkEY2mGMURTfIQVnnHpuv/OSox9YhbafYMaoz0GP3w79v1NseoQRYpwevtn YvBVI48oa2d8KeqkOUWoLW1lRvy69qeX0Fffs9/8TpFjUZp/cuHxr2XwFxy7PBR6kFfd xd/gz5b39HlYWOeYgUyFD3DQK1mJrT2+M70S0zODMBJi+MNhyYpcTc/FQhKl0IrHx1M6 VH8BHrs2+pGK7lfqrImYPobnDhocEsWbg9ArmkNePnyYv8p4ey/r0fpm/RgL9i2n8u+6 2s8A== X-Forwarded-Encrypted: i=1; AJvYcCW6ulCrJbj/FiLadR8mYCnUjqb2TXzT0le1DB9ZkjNmf08wBX1Ipg8EilbpMf2l2umRZEPUSCR3dA==@kvack.org X-Gm-Message-State: AOJu0Ywb+exQQq3sqt7LpDLyeXTBuiWwNWumAFR2BypHEgmYnKBuLUmR 0bEoK/An8qnhZLPYcvRAaVGY4MjhQ9+AnldWVw3L9sdmomWfAxUov6GCb/1oywYU5taEmE9r/GZ F6mbVUUopWg== X-Google-Smtp-Source: AGHT+IFp39E9krgcILEXjwgS0eG/yaDAJpZCMdfJgN+pTbMyBclx0Pttf9qnW8OHp1hJ5TpOh65GX4riAPR9zA== X-Received: from wmbbd12.prod.google.com ([2002:a05:600c:1f0c:b0:434:fd41:173c]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:1d03:b0:434:a781:f5d9 with SMTP id 5b1f17b1804b1-436e2697b32mr62972955e9.11.1736534447848; Fri, 10 Jan 2025 10:40:47 -0800 (PST) Date: Fri, 10 Jan 2025 18:40:27 +0000 In-Reply-To: <20250110-asi-rfc-v2-v2-0-8419288bc805@google.com> Mime-Version: 1.0 References: <20250110-asi-rfc-v2-v2-0-8419288bc805@google.com> X-Mailer: b4 0.15-dev Message-ID: <20250110-asi-rfc-v2-v2-1-8419288bc805@google.com> Subject: [PATCH RFC v2 01/29] mm: asi: Make some utility functions noinstr compatible From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Richard Henderson , Matt Turner , Vineet Gupta , Russell King , Catalin Marinas , Will Deacon , Guo Ren , Brian Cain , Huacai Chen , WANG Xuerui , Geert Uytterhoeven , Michal Simek , Thomas Bogendoerfer , Dinh Nguyen , Jonas Bonn , Stefan Kristiansson , Stafford Horne , "James E.J. Bottomley" , Helge Deller , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Naveen N Rao , Madhavan Srinivasan , Paul Walmsley , Palmer Dabbelt , Albert Ou , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Yoshinori Sato , Rich Felker , John Paul Adrian Glaubitz , "David S. Miller" , Andreas Larsson , Richard Weinberger , Anton Ivanov , Johannes Berg , Chris Zankel , Max Filippov , Arnd Bergmann , Andrew Morton , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Uladzislau Rezki , Christoph Hellwig , Masami Hiramatsu , Mathieu Desnoyers , Mike Rapoport , Arnaldo Carvalho de Melo , Namhyung Kim , Mark Rutland , Alexander Shishkin , Jiri Olsa , Ian Rogers , Adrian Hunter , Dennis Zhou , Tejun Heo , Christoph Lameter , Sean Christopherson , Paolo Bonzini , Ard Biesheuvel , Josh Poimboeuf , Pawan Gupta Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-alpha@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-um@lists.infradead.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, kvm@vger.kernel.org, linux-efi@vger.kernel.org, Brendan Jackman X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 8E5A720006 X-Stat-Signature: w5wzx53h5zi1unepk4gqhe38atztzjxn X-Rspam-User: X-HE-Tag: 1736534449-244109 X-HE-Meta: U2FsdGVkX1+bDHNxUkDYEJY/PMdgYJGCg6Vm0bdc/6ypOKUFjSUIDxnkh9ZemsrvjiYEeml4LGkI/J6o+rLJhhUsmTm73KRSFqj++pOdumB0Zja8JIGTfoV8+0/NCTm8zT88YVKoXlAtrFJpJEshfvC+NU4nSdLaFckFZyq4EPSA+EVf5dW+5t0Tqj46c2J1YIVEvo8t7t9TBb7vwBajlYR0p+woZNft9CKcx8io6s6+PKCTWTcMWXS3WFjbHGZdoMt+WCwm8UdqylqdCYMLjX/LcplqeWdtb7jp6ISMn2uxIsggpbpYpT+TFEXonagkWAByNCMX0Gkr6buPb7Dv7ffvdEuxtCJl3U/DmeZ/4oLJ6UhCG6UeuxBgtsgq7rR3MkU6uf3L26V9IyvZhgTj5JTku13iQ+JsW4yZw6tTuljZKWoYjrAIdhBQcqsxheGMxfGNXjTdNi+4idPNbasQZBRuj76tBQfpZH9tuviVINNwb1m5gWi0rHhp44jVEnd+kV8VuieVu02wKaeGuqJFv4iCsbI5amYsmR6tRl6nHsLuS5uuivQJbiwx/jBPu2aWdFPD4gupGXgws+qUqWDACeWu8EVcaU4YHM+4fJ5gkDcF4hg12I4X1UR6votjtA5FyhRfyZAK6Ov7x34IhIY6RNGfXEgl/xqWBBkVf9jfRKcnHojP/vcOVLrMFXxtM/lESAsOQKURdpm/qf1pmZiIzncreGBPOGNsUKRGr8fgQeMb7L4vmeX4p5+lxGH4nwJvJyI0G8mBttq9q+4HYlURdJ6EQDseeRCWK1s9qGC0FpGkvokTwMo02dgc7ro3k/vEFnsrVee+aNpGSa5wmFJt8jPc7M6emtz+proIWlBYJYKhfKF/BJt24fm6kE9bg1IbVj506T0hcXAZXnDSZo3lqdHQjtk+H94F/6N2yTE5eNkqQ/5ByyPPfWN4mERgZ6eL99tEqtUY/rJuDz1V+Bs aN7EfnZO WFpOZv3gg7vCR8OfX15as2pi3ray6FZyS1oEYzEk+QuWOdCNiJF8xb2kMusIEnb5SRfENfwbaZDDwzx7hFdly/AJNRQGaG3F9bZVIWjE5rLTFsJv6Xa7WCtuqBQ55Tk0T6+CdWwZn3XQIQUuEP25vxayirQwfW8Hm8fY/s3pyo7RwzfD5fJuuTSYIy5CZrg+sqfBb9fe1rb/74pMq4nqvmGC1jUckdl9Kmbumza7Prg5uPXigHzHuM2ix4L7PWfBqgh721U/b6+80FwRTTsifKzya1IS25uIWGW752vq6xvdl6IiZ2KH3Yo8U6tc4pLS3teFHQi7lFSUOhnJBm2iIChkryRGV3+qlWwGrPLNyx6EOALUQnPYLsuGGd4soPF6VkyuFbN0Je9q94WPLNvh7QTVLL2hY2p9i43qCfQs8slehY1GquBnvIOPbU8uDn5p8+a1gU5fhiRlXT9Cl/oNYz67vX6NE4/BALy6TbablPObYxPW77bEcyi1PFNKfxHvlnMGzPqn6kP9++TepPkfn422R812aetKr63fUj3swEx4sJtcByuLSyY5dKti2d8Nt97JvGKTS/R412bSac8vf2pnKU9ksHIceFzALYl433cnBON8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Some existing utility functions would need to be called from a noinstr context in the later patches. So mark these as either noinstr or __always_inline. An earlier version of this by Junaid had a macro that was intended to tell the compiler "either inline this function, or call it in the noinstr section", which basically boiled down to: #define inline_or_noinstr noinline __section(".noinstr.text") Unfortunately Thomas pointed out this will prevent the function from being inlined at call sites in .text. So far I haven't been able[1] to find a formulation that lets us : 1. avoid calls from .noinstr.text -> .text, 2. while also letting the compiler freely decide what to inline. 1 is a functional requirement so here I'm just giving up on 2. Existing callsites of this code are just forced inline. For the incoming code that needs to call it from noinstr, they will be out-of-line calls. [1] https://lore.kernel.org/lkml/CA+i-1C1z35M8wA_4AwMq7--c1OgjNoLGTkn4+Td5gKg7QQAzWw@mail.gmail.com/ Checkpatch-args: --ignore=COMMIT_LOG_LONG_LINE Signed-off-by: Brendan Jackman --- arch/x86/include/asm/processor.h | 2 +- arch/x86/include/asm/special_insns.h | 8 ++++---- arch/x86/include/asm/tlbflush.h | 3 +++ arch/x86/mm/tlb.c | 13 +++++++++---- 4 files changed, 17 insertions(+), 9 deletions(-) diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index 4a686f0e5dbf6d906ed38276148b186e920927b3..1a1b7ea5d7d32a47d783d9d62cd2a53672addd6f 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -220,7 +220,7 @@ void print_cpu_msr(struct cpuinfo_x86 *); /* * Friendlier CR3 helpers. */ -static inline unsigned long read_cr3_pa(void) +static __always_inline unsigned long read_cr3_pa(void) { return __read_cr3() & CR3_ADDR_MASK; } diff --git a/arch/x86/include/asm/special_insns.h b/arch/x86/include/asm/special_insns.h index aec6e2d3aa1d52e5c8f513e188015a45e9eeaeb2..6e103358966f6f1333aa07be97aec5f8af794120 100644 --- a/arch/x86/include/asm/special_insns.h +++ b/arch/x86/include/asm/special_insns.h @@ -42,14 +42,14 @@ static __always_inline void native_write_cr2(unsigned long val) asm volatile("mov %0,%%cr2": : "r" (val) : "memory"); } -static inline unsigned long __native_read_cr3(void) +static __always_inline unsigned long __native_read_cr3(void) { unsigned long val; asm volatile("mov %%cr3,%0\n\t" : "=r" (val) : __FORCE_ORDER); return val; } -static inline void native_write_cr3(unsigned long val) +static __always_inline void native_write_cr3(unsigned long val) { asm volatile("mov %0,%%cr3": : "r" (val) : "memory"); } @@ -153,12 +153,12 @@ static __always_inline void write_cr2(unsigned long x) * Careful! CR3 contains more than just an address. You probably want * read_cr3_pa() instead. */ -static inline unsigned long __read_cr3(void) +static __always_inline unsigned long __read_cr3(void) { return __native_read_cr3(); } -static inline void write_cr3(unsigned long x) +static __always_inline void write_cr3(unsigned long x) { native_write_cr3(x); } diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 69e79fff41b800a0a138bcbf548dde9d72993105..c884174a44e119a3c027c44ada6c5cdba14d1282 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -423,4 +423,7 @@ static inline void __native_tlb_flush_global(unsigned long cr4) native_write_cr4(cr4 ^ X86_CR4_PGE); native_write_cr4(cr4); } + +unsigned long build_cr3_noinstr(pgd_t *pgd, u16 asid, unsigned long lam); + #endif /* _ASM_X86_TLBFLUSH_H */ diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 86593d1b787d8a5b9fa4bd492356898ec8870938..f0428e5e1f1947903ee87c4c6444844ee11b45c3 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -108,7 +108,7 @@ /* * Given @asid, compute kPCID */ -static inline u16 kern_pcid(u16 asid) +static __always_inline u16 kern_pcid(u16 asid) { VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE); @@ -153,9 +153,9 @@ static inline u16 user_pcid(u16 asid) return ret; } -static inline unsigned long build_cr3(pgd_t *pgd, u16 asid, unsigned long lam) +static __always_inline unsigned long build_cr3(pgd_t *pgd, u16 asid, unsigned long lam) { - unsigned long cr3 = __sme_pa(pgd) | lam; + unsigned long cr3 = __sme_pa_nodebug(pgd) | lam; if (static_cpu_has(X86_FEATURE_PCID)) { cr3 |= kern_pcid(asid); @@ -166,6 +166,11 @@ static inline unsigned long build_cr3(pgd_t *pgd, u16 asid, unsigned long lam) return cr3; } +noinstr unsigned long build_cr3_noinstr(pgd_t *pgd, u16 asid, unsigned long lam) +{ + return build_cr3(pgd, asid, lam); +} + static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid, unsigned long lam) { @@ -1084,7 +1089,7 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end) * It's intended to be used for code like KVM that sneakily changes CR3 * and needs to restore it. It needs to be used very carefully. */ -unsigned long __get_current_cr3_fast(void) +noinstr unsigned long __get_current_cr3_fast(void) { unsigned long cr3 = build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd,