From patchwork Wed Feb 5 01:39:55 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rik van Riel X-Patchwork-Id: 13960406 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9EE76C02197 for ; Wed, 5 Feb 2025 01:42:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DC9B5280009; Tue, 4 Feb 2025 20:41:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D797E280029; Tue, 4 Feb 2025 20:41:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B5518280009; Tue, 4 Feb 2025 20:41:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 8F477280029 for ; Tue, 4 Feb 2025 20:41:58 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 4549647CD9 for ; Wed, 5 Feb 2025 01:41:58 +0000 (UTC) X-FDA: 83084189916.06.E14BC88 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) by imf02.hostedemail.com (Postfix) with ESMTP id A67C08000C for ; Wed, 5 Feb 2025 01:41:56 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738719716; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HRG2Bbm1sWj6J9i73OhV5GaNFdEzDwFt59ntbbNT8+4=; b=A0JAYaqeBtWW7izZOGCy7VyoEmmgXIoFQwb+6FZQ5s1AlbI3H6qnsYxuOtS2SL1IVFCfav /iTm5o6nJgPVGfMusd3poanQFZN1pTjiIEQau/NQ6NTWE23vzn0Fr4QJm+rOOJBtcQxjeK g8Oy+46DsBWo+WGs4D8/F8hsKV8dNsk= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738719716; a=rsa-sha256; cv=none; b=3mPEY6+DCTKQ8RsBImpqevbN+ncY6GJeYb475FTs1aXxFmYoXTjRj9AXe5w8fNTwnn/0J3 ssfz31mUWwMvSbDcgLwCV7m2nuVB8rsj/AUGX4qlESIC5l0LssDAN6Vgzasm60iDVbptXK 5XilptHLOCoARD7AHF+O41Ti4xrS2CQ= Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1tfUP4-000000004Cs-2ubK; Tue, 04 Feb 2025 20:40:34 -0500 From: Rik van Riel To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, bp@alien8.de, peterz@infradead.org, dave.hansen@linux.intel.com, zhengqi.arch@bytedance.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, kernel-team@meta.com, linux-mm@kvack.org, akpm@linux-foundation.org, jannh@google.com, mhklinux@outlook.com, andrew.cooper3@citrix.com, Rik van Riel , Manali Shukla Subject: [PATCH v8 06/12] x86/mm: use INVLPGB for kernel TLB flushes Date: Tue, 4 Feb 2025 20:39:55 -0500 Message-ID: <20250205014033.3626204-7-riel@surriel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250205014033.3626204-1-riel@surriel.com> References: <20250205014033.3626204-1-riel@surriel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: A67C08000C X-Stat-Signature: dpw9jz3pz6dcojie5ogpz4tit93byncz X-HE-Tag: 1738719716-354838 X-HE-Meta: U2FsdGVkX1+65bYy7VqxSroT1xgK6g0gq0qjMkjMFSC/Q2x9gUCle3iLM5AEscUjnOYonyXbXo3ZpPzWu68kA2ghpCF5fhEtVqMWsTzsazBrto16uU2mR3JyZRfY4jgSG4UV+zaBd4rmZ1gKMvHVKjzURowoUWGG6c/nKrLfhh0++u1Zw2vakqVwZOD+8hWBgVHpwyGpibo7qSe1AzQyongEFM7+dIgYM6AjcQWz801s9X4ZVnCzZxfgk9IpEF7L2bpM3M4isQolU9GcegsIH7s+cwOVHZEb17piHY0HRwK0nL6g7MY+n7zU2mz44HCAD1pLA+YLnUJ1JH/W0xR1WALQxlVQD8ZsNCsBFbSGG28TxdEn2MXihQk1xKEVwUQTryHK7AYnJsoEYawWMKV25uyIH+uMstInlFR1hftcmNATnJ2lrcT+dlzoDqCKzrlcuONwol8zM7nEqkT0/fJ3JAXds6Zg2XhPOCqEnf973zsocMfhbZT+EvjCWIZGhnsfuMaE/c/WE/ugaTQwL9D3DaSAsptYoJlDqsAiLCpdO9pw8n2FL8SLI/AK+jAVw6bBIF5VLNqe+lo5Z0wFGHNXmhMjgFdcZ62NcjeuNBMwZJSLFYQ0rcJ/sw3eqljJUKfFl2izlbugvE8d3QnEOBSYvJXB27ktzzd08Pv/3Z4bAujrsFfSaRgpAtgLqfjIzQBFsbNP5GYyO6U6Eueqc+VG0ZifXFg6al2lo1pWAi6RDeSaTALGKfIEoEfqGsVMYHozScofJmdc7LwiFw0axm8dbx6UCX/ycfruh5RHC1dw+KZey7lLjCqBKATWUkrSSw/ucRTR7P7ylRYBtFyShRE5niG0jUaHIa/XjFXalY0QqL73AMfGy+e/K1RaUhJSAepND9nBEqMv8WMGOFlMSxNZqGNN2v6I+TY8b9OWSQw4n/s/OlrkR8VzhCC7v5d6HRlXqeQ68q/C/JdZRRU4Id7 xQ9syHvY WcYElPUlAdVb/QoMnBMxel9KHWpyBxoVFXhYvJQ0vSIBWNa3vFAie+tJsm8ygQgsB5bfqhno2N1XJKSFLLB4NN+28guXVMbrod/wy2r7tTIXf5JEL4l/SDrfbm0yUuwB0zKRPgBwgtW3hhLlmXsxwje+MApu2MijX7ll5nR0W1pZJDelo/kwiIyUues6EQPV2qd5krpa67GUXu1n1m4vgmD1l5CkMKZJp61HNn5/2l8E5VJh4gWeRMBEWnqc3XTbx+jyB/KUHH8M/PFbB0I9P8/lmilIeIHAJLHwoQovIqubRjp/s8D+aAm+/wLvjpvvpurYJHxx9N7k4yIr+Y1LeTHjL7NJXp9+D6FUQlk9jnWPFG1wnzUkFB5RP4NmRGSVDr3lqrlICFRJNkhvrI6T3eCE8czea6dxsodpy6PGDq87efstWcV59KGYQ0vmyVHW8ZhPHk29kpZVSevvOaPrXR02QXogpQEipd2qNgrt04PsBT6BfUuCzAfZnJQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use broadcast TLB invalidation for kernel addresses when available. Remove the need to send IPIs for kernel TLB flushes. Signed-off-by: Rik van Riel Tested-by: Manali Shukla --- arch/x86/mm/tlb.c | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 02e1f5c5bca3..7945624afc98 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1082,6 +1082,30 @@ void flush_tlb_all(void) on_each_cpu(do_flush_tlb_all, NULL, 1); } +static bool broadcast_kernel_range_flush(struct flush_tlb_info *info) +{ + unsigned long addr; + unsigned long nr; + + if (!IS_ENABLED(CONFIG_X86_BROADCAST_TLB_FLUSH)) + return false; + + if (!cpu_feature_enabled(X86_FEATURE_INVLPGB)) + return false; + + if (info->end == TLB_FLUSH_ALL) { + invlpgb_flush_all(); + return true; + } + + for (addr = info->start; addr < info->end; addr += nr << PAGE_SHIFT) { + nr = min((info->end - addr) >> PAGE_SHIFT, invlpgb_count_max); + invlpgb_flush_addr_nosync(addr, nr); + } + tlbsync(); + return true; +} + static void do_kernel_range_flush(void *info) { struct flush_tlb_info *f = info; @@ -1101,7 +1125,9 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end) info = get_flush_tlb_info(NULL, start, end, PAGE_SHIFT, false, TLB_GENERATION_INVALID); - if (info->end == TLB_FLUSH_ALL) + if (broadcast_kernel_range_flush(info)) + ; /* Fall through. */ + else if (info->end == TLB_FLUSH_ALL) on_each_cpu(do_flush_tlb_all, NULL, 1); else on_each_cpu(do_kernel_range_flush, info, 1);