From patchwork Tue Mar 4 13:58:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Borislav Petkov X-Patchwork-Id: 14000778 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF818C282D2 for ; Tue, 4 Mar 2025 13:58:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C5F7280001; Tue, 4 Mar 2025 08:58:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 275BF6B008A; Tue, 4 Mar 2025 08:58:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 116DE280001; Tue, 4 Mar 2025 08:58:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E62B96B0089 for ; Tue, 4 Mar 2025 08:58:32 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 5E690161D70 for ; Tue, 4 Mar 2025 13:58:32 +0000 (UTC) X-FDA: 83184023664.11.5919AFF Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf30.hostedemail.com (Postfix) with ESMTP id B0BF68001C for ; Tue, 4 Mar 2025 13:58:30 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=OZIgd2wB; spf=pass (imf30.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741096710; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oTl3NgJT+1T/zGUp4F3/SyhLl/ksYXb28F9MHJXx7q8=; b=yweZFLgpfRWYoOa7CrGBwJa/Qz3j5m4k3rx0S1w/ETCu9H/pQBP+CVWxNop9IFrIvej2p+ 65DxTnwTc8VbZMHGm0rnacM8SUnepk6ygUCRo2hJ/g81Ep3pIiSEO3BD2qnboX4Xo+aZVf wEsvo7V2yDjrLKYm39a9mLmkcDqHbkQ= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=OZIgd2wB; spf=pass (imf30.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741096710; a=rsa-sha256; cv=none; b=Lb8jn0uyGG3uhZmwkeOGw0O+KorU+5TbUry99GH2PN9ZoOePZTz8QvE/MFX3wOsjLmqJMR Qv9/lc0oAm4IFYn46GdY3FOXfsKowrJEhPKsbsOWtYqfNzmzPLe1Vsqvgc6cFzwGG4JP/R mgy1KTG+Adwt6jOaSB6Y71JYqZJM2Vk= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 2E3355C5D54; Tue, 4 Mar 2025 13:56:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D06C1C4CEE5; Tue, 4 Mar 2025 13:58:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741096709; bh=RTPSleP7Ih9cuOhakawnif+qFen0hQD2ElBjJf/R2uo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OZIgd2wB3QlNyLuuLAxLvQkYbI3bmfHrOI1t5jIwzoVCi+tTxuy/UGkhYq/jlZt4L 28ZJV3wE2+RmDHPbX3VGbOG6rF+xcIOp+QX2XWIFMii1TdPtzyu3iVjxbqX5tPed83 nTj6XLNc+HT049+p4GrntMUWDjtpHP0PfvInJPB0BpQS1S/gByelbiR1Vs/h3pWme0 aFL/oYfdxO0b25B1iYW/5amNk1nOO5VG8hcpCXucZVs8oUsp66yJg+VrG+BCuv8Sj1 VuIlrn1ixV9FAzTp+YFM/4pi8jPMbqyCjDthffiyEgv7GQJgmda+06qO3pqH8o9+kH JaLL+fZtldtew== From: Borislav Petkov To: riel@surriel.com Cc: Manali.Shukla@amd.com, akpm@linux-foundation.org, andrew.cooper3@citrix.com, jackmanb@google.com, jannh@google.com, kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, mhklinux@outlook.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, x86@kernel.org, zhengqi.arch@bytedance.com, Borislav Petkov Subject: [PATCH v15 02/11] x86/mm: Add INVLPGB feature and Kconfig entry Date: Tue, 4 Mar 2025 14:58:07 +0100 Message-ID: <20250304135816.12356-3-bp@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250304135816.12356-1-bp@kernel.org> References: <20250304135816.12356-1-bp@kernel.org> MIME-Version: 1.0 X-Stat-Signature: jku9fduqmpi65fcbysycue9nmnimftd7 X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: B0BF68001C X-Rspam-User: X-HE-Tag: 1741096710-777094 X-HE-Meta: U2FsdGVkX1+axrdu/5kRYV+h/+iq9c6nXrOAtcMjXnkFM3PEVek53LZNLEEcg2p4mrNdXsbQ2vtmTN1WSXHyyX/BOg9Sq4NQe4EPui+CyoavV8NwVGWLCTyTK4mNoSudlHcrKxQb21qKqVZkkKc077qO1HI4v2VZPPAHzs+nnapqiG+7AIX+1Cf9GDIC229tJLBP4YreCCnBWNVx3IIKIpqIZKalA5DYAxunHHjKvCatEdzec2s/XGAuInohYpPWwNktkzDv1LA+jqannCOUDTC0pVgao2/B8Gw55NIm4YqosWI34TMedIMIa9YgROwZnuxC2hE+f0Vt23bqWGo018ESgV25Auy0TMz4DLZUTT7Vgql6r3C7GPsGxNADfo75UpB3ZHPm2vXeixo+K3ZsRcmSbhlcrexS0gUjk2JUJt7Nd39nCF4umEY+nmGvEOH+bOzzOjpJmqCMv53S8YAed9gWSZ8zX9rJRA+ysQZq45nVo+1pG1yt9cXwWOMS1n5unJyYQdMInhxIwiV6+gveGzAeNBhi7PinYwArNUXzKDIB3vTpgy1QNSk2ICdYJwLLuOYy2qBZzNVI+4AFBnRaHrdIWqvKAtofHfIwk8nqaPtJScQwiwK+vNKac7F2AVtygfzwZYwKMebLSMa+pi6CbxhgUL7wwZtyu9nd6evjwYXvCI2YY50J12wcKDF+3//6wbAmfazzSvXlY4J5ZsIqpg3JWWNjhjl61XOHi1Kwdp+0Gm1IMgTVqbbrUHgPKGflH+09iTZYVhmETn8be8lRIIVoqARuriKdwHjWn9Uvk9TOcu+uckLRY7bSpmDLh01BB0C8XKKb4Ht0+WaMxVDNefQupq2va0lMwSNG9GK6Ny4Ww7ZVq++N1PLFhb3dHqyBBGvyMmssEBoiVO3NanC1J4pFbeLymZgblicb4sc2aCqwpfuM19nfdCp0t9zhQ/MdgDmfCj0V5qjlLhlgalb 6gkEt69M brE2TdSjHL2uTU/ADE57ovaaY+vKbbzQFhc0PE4OkkvPHnBNEvpS+YJDUgdhiKotX1s00w1qqSRIwTWQ7W//bP+0rAv6Z9k0WQmXCsAGi8eWoN1rISaYKE3na5nEmhhxIxEAX3c5wjsIh8vlL5nKtqjzzmAaW3EPkJwz4r9P9Z7aP0Q2oP/qHCGI9kVT4pRs94UMhfRzWV/Vpi61Y3cJpPTzGGS7R4Jn+94A8EcP/gmd88apzCcFnj+KcDw7fzvXhL7Zz6obC8jpcMap9RAEglmO9fzbKKo4DqdjTdV8/oHpldRvNg/AqE9NrEiPVqVy70f9QlA2nNaFcOIvs1VV2klVxo2f07ljoqa+9BprGJtbcBx4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Rik van Riel In addition, the CPU advertises the maximum number of pages that can be shot down with one INVLPGB instruction in CPUID. Save that information for later use. [ bp: use cpu_has(), typos, massage. ] Signed-off-by: Rik van Riel Signed-off-by: Borislav Petkov (AMD) Link: https://lore.kernel.org/r/20250226030129.530345-3-riel@surriel.com --- arch/x86/Kconfig.cpu | 4 ++++ arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/disabled-features.h | 8 +++++++- arch/x86/include/asm/tlbflush.h | 3 +++ arch/x86/kernel/cpu/amd.c | 6 ++++++ 5 files changed, 21 insertions(+), 1 deletion(-) diff --git a/arch/x86/Kconfig.cpu b/arch/x86/Kconfig.cpu index 2a7279d80460..25c55cc17c5e 100644 --- a/arch/x86/Kconfig.cpu +++ b/arch/x86/Kconfig.cpu @@ -401,6 +401,10 @@ menuconfig PROCESSOR_SELECT This lets you choose what x86 vendor support code your kernel will include. +config BROADCAST_TLB_FLUSH + def_bool y + depends on CPU_SUP_AMD && 64BIT + config CPU_SUP_INTEL default y bool "Support Intel processors" if PROCESSOR_SELECT diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 508c0dad116b..8770dc185fe9 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -338,6 +338,7 @@ #define X86_FEATURE_CLZERO (13*32+ 0) /* "clzero" CLZERO instruction */ #define X86_FEATURE_IRPERF (13*32+ 1) /* "irperf" Instructions Retired Count */ #define X86_FEATURE_XSAVEERPTR (13*32+ 2) /* "xsaveerptr" Always save/restore FP error pointers */ +#define X86_FEATURE_INVLPGB (13*32+ 3) /* INVLPGB and TLBSYNC instructions supported */ #define X86_FEATURE_RDPRU (13*32+ 4) /* "rdpru" Read processor register at user level */ #define X86_FEATURE_WBNOINVD (13*32+ 9) /* "wbnoinvd" WBNOINVD instruction */ #define X86_FEATURE_AMD_IBPB (13*32+12) /* Indirect Branch Prediction Barrier */ diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index c492bdc97b05..625a89259968 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -129,6 +129,12 @@ #define DISABLE_SEV_SNP (1 << (X86_FEATURE_SEV_SNP & 31)) #endif +#ifdef CONFIG_X86_BROADCAST_TLB_FLUSH +#define DISABLE_INVLPGB 0 +#else +#define DISABLE_INVLPGB (1 << (X86_FEATURE_INVLPGB & 31)) +#endif + /* * Make sure to add features to the correct mask */ @@ -146,7 +152,7 @@ #define DISABLED_MASK11 (DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET| \ DISABLE_CALL_DEPTH_TRACKING|DISABLE_USER_SHSTK) #define DISABLED_MASK12 (DISABLE_FRED|DISABLE_LAM) -#define DISABLED_MASK13 0 +#define DISABLED_MASK13 (DISABLE_INVLPGB) #define DISABLED_MASK14 0 #define DISABLED_MASK15 0 #define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \ diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 3da645139748..855c13da2045 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -183,6 +183,9 @@ static inline void cr4_init_shadow(void) extern unsigned long mmu_cr4_features; extern u32 *trampoline_cr4_features; +/* How many pages can be invalidated with one INVLPGB. */ +extern u16 invlpgb_count_max; + extern void initialize_tlbstate_and_flush(void); /* diff --git a/arch/x86/kernel/cpu/amd.c b/arch/x86/kernel/cpu/amd.c index 54194f5995de..7a72ef47a983 100644 --- a/arch/x86/kernel/cpu/amd.c +++ b/arch/x86/kernel/cpu/amd.c @@ -29,6 +29,8 @@ #include "cpu.h" +u16 invlpgb_count_max __ro_after_init; + static inline int rdmsrl_amd_safe(unsigned msr, unsigned long long *p) { u32 gprs[8] = { 0 }; @@ -1139,6 +1141,10 @@ static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c) tlb_lli_2m[ENTRIES] = eax & mask; tlb_lli_4m[ENTRIES] = tlb_lli_2m[ENTRIES] >> 1; + + /* Max number of pages INVLPGB can invalidate in one shot */ + if (cpu_has(c, X86_FEATURE_INVLPGB)) + invlpgb_count_max = (cpuid_edx(0x80000008) & 0xffff) + 1; } static const struct cpu_dev amd_cpu_dev = {