From patchwork Tue Mar 4 13:58:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Borislav Petkov X-Patchwork-Id: 14000777 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 179DBC282C6 for ; Tue, 4 Mar 2025 13:58:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 92DF66B0085; Tue, 4 Mar 2025 08:58:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8B6F6280001; Tue, 4 Mar 2025 08:58:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E1DB6B0089; Tue, 4 Mar 2025 08:58:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 4D50D6B0085 for ; Tue, 4 Mar 2025 08:58:29 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 06324141CB9 for ; Tue, 4 Mar 2025 13:58:29 +0000 (UTC) X-FDA: 83184023538.05.2C24610 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf20.hostedemail.com (Postfix) with ESMTP id 5C13E1C001B for ; Tue, 4 Mar 2025 13:58:27 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=h8kbxrx7; spf=pass (imf20.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741096707; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sarqgyLf7PRHyIouFMRKv/vMk4RKJdq/9QGwcqMirTg=; b=hdMghrPacCnXiL6Gp1/B9oiKcPjIBmOaMX4Z+Oykfhrr43NWCRLdo6BCLPLfBcbsusWFia T/QtQKAzH8oq7PsLVLtFZidyQKywl1qsbmTFRKJekYrPLzslFeu25Sqp0vLbTy+pNkCK67 5r6vIDG0VMJsF79/NLS9GO1sd6dGF6E= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=h8kbxrx7; spf=pass (imf20.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741096707; a=rsa-sha256; cv=none; b=Mv598ai+a3dBbtDypK0AlFHXMps6NAWgJ9MssvqYFsbSj4whlME5s3Oi3K3fd+DOe9DOft VxQy+88IHw+r5wDHmIeqkb4/fOUgXIhLynoAOniwV8yU8Er2dHWFQE0CxTbd3zWgXAKOek fZwWF41b9VZz/06wFAbFzL/dGDQEFG8= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id C01465C5D48; Tue, 4 Mar 2025 13:56:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4144AC4CEE9; Tue, 4 Mar 2025 13:58:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741096706; bh=vdVKhRglpt9Vosn6QZofV8P5uPpS3iho3gB/iAAuoYE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=h8kbxrx7XK4Q+NvxBwsqKAlUsRBlNusvYOQqa0kf5IrWbDf80uhG5BGtg+q9zejMf oAqZYkG9/qjpExATxHbmuTpbZK2dfkB7e6NzRlJsNoDhm2Mlg8TE+bEZz5S/zi/QOR /Zw0xLHN3jXCTS2tzLM1RblDh0T7FFfcDnBF7wlZt7FWicx8bsydT8MZoHrPcWDSYV t/u5zBdrjbIS1pMarOZxjDvEt/zd9iNJo/N/qn4PRfEBtEHfm1mrO9nnV1a+HuIXfv WtAWNU17nNtTk9VdJaXMcKNm9IwQcRzKR5IQ3yE6ntI9ZLYVjMCJhWyICrHRxL689W 4CHBTPC+GBXiw== From: Borislav Petkov To: riel@surriel.com Cc: Manali.Shukla@amd.com, akpm@linux-foundation.org, andrew.cooper3@citrix.com, jackmanb@google.com, jannh@google.com, kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, mhklinux@outlook.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, x86@kernel.org, zhengqi.arch@bytedance.com, Dave Hansen , Borislav Petkov Subject: [PATCH v15 01/11] x86/mm: Consolidate full flush threshold decision Date: Tue, 4 Mar 2025 14:58:06 +0100 Message-ID: <20250304135816.12356-2-bp@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250304135816.12356-1-bp@kernel.org> References: <20250304135816.12356-1-bp@kernel.org> MIME-Version: 1.0 X-Stat-Signature: t1q6hapr99khtqw88knxhdpj5p56y1dw X-Rspamd-Queue-Id: 5C13E1C001B X-Rspamd-Server: rspam06 X-Rspam-User: X-HE-Tag: 1741096707-46387 X-HE-Meta: U2FsdGVkX189YrkQmbsMRDDmuLdNm9KVKNF6XNKWqLrQ7ZzdiHvAleBvc1po/bOFiMR9JdM89Sk7taJhQIsaPUck4gHP+nvUolw7o+PLpQxebLObRbMz2JDEwhqG7zg30BCfOZ2POZzXYyTrQ/KuoBqzbQkiqJt1bv4xdmeqMdLTpx9q826PyOwLHMF3xOaRagMD3YJeucAL0WgKA0OawL5L7bra75C5QuRaDRxrNXQ+NACwu9D2iUcQk5BHIxf0AS6qtRxqXSrczEkirO0xOxjLjoFn77OeIWQrrvKubwQfKWe6ummLsmtBq2oBjP61PaRI+PdZmiwsrl4qekYm8pG/4cah/BbqGg+4H3Da2eWlvXQrtc6UraxzN6Pg/OPNIRP+rW5xXuFYyTY8lfgsqX0/b8OMwU6nuxMjNzMr6nPhTgnzqJqdeoVnj3Is156T3MnhGBhZe24AaEWfIGyszvCWVTksB4AbVYwViUJhzYCKAlxbLMxhbbpmyCx1zjU3apM9mBABT1PbR8iOD7losv5Dpp9KxgPa0QB/M4K7kv35uwI2jC7a/dk0z6IwY1BCrq0IJ2C1/IOCuOARU8v+kvKNyechRZYbIc3F4Yk+qkdiHj064PufYA0KqGTTm+qot1uyPiAokfxr3XdvlgXVkycaUZA3ub4Ih7pqjCN542kEHlmWzlsnyYfZfS7qBQaOO0krw9PSM99od5t+xpGqz/lIz0C4NvYQxm5LyZtvylV3GuZZXOaBI9NyyyQv6AgMuuzDbC5HiSmO150qJp7s8KJDfn/x5Uph//q2ooQFiADijbV/jQKlfwskVSXhH422oYIW+9K9nSBsNDZvpUA7slx32sG6qoovZccSPKlQ60ieJT6DFViCIJOfisL9rRnTvw18KZWRitACYvoR6NmVzg0VHDgsoVn4+EJn9HE3D2b1L6puZG3XBkQuTSncCBcWnNlAn9ApgKIL2J4V/NL /4vBYvGp tdcfGfS0otKn8jvqRbvQGRrQffES27yaLR/peMwBeGCnzgIsR3k9DCOf7Q6D5dh2OSGcrrj4O5U3uHpgXSfjF+ysnFZs/rqtH871RaDJJIUI1HawJY4F90bKsT51G6PxEUhoSMtDb58eZ7pWbxJLE0yA30cf/rhcjQWIQ2GQzp66aair1h+XHmB5X5HQN2n3mrZb8tiq/Lx4Qa4ZEea7+l9zQdXUkPOEl+MBofWMlzdPIZxCiK6MJvY7G1fLEvH97xZefRcGnsYWBXgA4w0LxIjhhpeCp2Wxj/bxvM8rA+2Q8RSgkfee40nFaPEB8TkVicjLY7ve8hqVEF09w9SoIJCdco4/rHzCJQBUQeS+7bgrdGOBGOuiXdj4TTo4bVjiJlNZY X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Rik van Riel Reduce code duplication by consolidating the decision point for whether to do individual invalidations or a full flush inside get_flush_tlb_info(). Suggested-by: Dave Hansen Signed-off-by: Rik van Riel Signed-off-by: Borislav Petkov (AMD) Reviewed-by: Borislav Petkov (AMD) Acked-by: Dave Hansen Link: https://lore.kernel.org/r/20250226030129.530345-2-riel@surriel.com --- arch/x86/mm/tlb.c | 41 +++++++++++++++++++---------------------- 1 file changed, 19 insertions(+), 22 deletions(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index ffc25b348041..dbcb5c968ff9 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1000,6 +1000,15 @@ static struct flush_tlb_info *get_flush_tlb_info(struct mm_struct *mm, BUG_ON(this_cpu_inc_return(flush_tlb_info_idx) != 1); #endif + /* + * If the number of flushes is so large that a full flush + * would be faster, do a full flush. + */ + if ((end - start) >> stride_shift > tlb_single_page_flush_ceiling) { + start = 0; + end = TLB_FLUSH_ALL; + } + info->start = start; info->end = end; info->mm = mm; @@ -1026,17 +1035,8 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, bool freed_tables) { struct flush_tlb_info *info; + int cpu = get_cpu(); u64 new_tlb_gen; - int cpu; - - cpu = get_cpu(); - - /* Should we flush just the requested range? */ - if ((end == TLB_FLUSH_ALL) || - ((end - start) >> stride_shift) > tlb_single_page_flush_ceiling) { - start = 0; - end = TLB_FLUSH_ALL; - } /* This is also a barrier that synchronizes with switch_mm(). */ new_tlb_gen = inc_mm_tlb_gen(mm); @@ -1089,22 +1089,19 @@ static void do_kernel_range_flush(void *info) void flush_tlb_kernel_range(unsigned long start, unsigned long end) { - /* Balance as user space task's flush, a bit conservative */ - if (end == TLB_FLUSH_ALL || - (end - start) > tlb_single_page_flush_ceiling << PAGE_SHIFT) { - on_each_cpu(do_flush_tlb_all, NULL, 1); - } else { - struct flush_tlb_info *info; + struct flush_tlb_info *info; + + guard(preempt)(); - preempt_disable(); - info = get_flush_tlb_info(NULL, start, end, 0, false, - TLB_GENERATION_INVALID); + info = get_flush_tlb_info(NULL, start, end, PAGE_SHIFT, false, + TLB_GENERATION_INVALID); + if (info->end == TLB_FLUSH_ALL) + on_each_cpu(do_flush_tlb_all, NULL, 1); + else on_each_cpu(do_kernel_range_flush, info, 1); - put_flush_tlb_info(); - preempt_enable(); - } + put_flush_tlb_info(); } /* From patchwork Tue Mar 4 13:58:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Borislav Petkov X-Patchwork-Id: 14000778 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF818C282D2 for ; Tue, 4 Mar 2025 13:58:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C5F7280001; Tue, 4 Mar 2025 08:58:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 275BF6B008A; Tue, 4 Mar 2025 08:58:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 116DE280001; Tue, 4 Mar 2025 08:58:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E62B96B0089 for ; Tue, 4 Mar 2025 08:58:32 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 5E690161D70 for ; Tue, 4 Mar 2025 13:58:32 +0000 (UTC) X-FDA: 83184023664.11.5919AFF Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf30.hostedemail.com (Postfix) with ESMTP id B0BF68001C for ; Tue, 4 Mar 2025 13:58:30 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=OZIgd2wB; spf=pass (imf30.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741096710; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oTl3NgJT+1T/zGUp4F3/SyhLl/ksYXb28F9MHJXx7q8=; b=yweZFLgpfRWYoOa7CrGBwJa/Qz3j5m4k3rx0S1w/ETCu9H/pQBP+CVWxNop9IFrIvej2p+ 65DxTnwTc8VbZMHGm0rnacM8SUnepk6ygUCRo2hJ/g81Ep3pIiSEO3BD2qnboX4Xo+aZVf wEsvo7V2yDjrLKYm39a9mLmkcDqHbkQ= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=OZIgd2wB; spf=pass (imf30.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741096710; a=rsa-sha256; cv=none; b=Lb8jn0uyGG3uhZmwkeOGw0O+KorU+5TbUry99GH2PN9ZoOePZTz8QvE/MFX3wOsjLmqJMR Qv9/lc0oAm4IFYn46GdY3FOXfsKowrJEhPKsbsOWtYqfNzmzPLe1Vsqvgc6cFzwGG4JP/R mgy1KTG+Adwt6jOaSB6Y71JYqZJM2Vk= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 2E3355C5D54; Tue, 4 Mar 2025 13:56:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D06C1C4CEE5; Tue, 4 Mar 2025 13:58:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741096709; bh=RTPSleP7Ih9cuOhakawnif+qFen0hQD2ElBjJf/R2uo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OZIgd2wB3QlNyLuuLAxLvQkYbI3bmfHrOI1t5jIwzoVCi+tTxuy/UGkhYq/jlZt4L 28ZJV3wE2+RmDHPbX3VGbOG6rF+xcIOp+QX2XWIFMii1TdPtzyu3iVjxbqX5tPed83 nTj6XLNc+HT049+p4GrntMUWDjtpHP0PfvInJPB0BpQS1S/gByelbiR1Vs/h3pWme0 aFL/oYfdxO0b25B1iYW/5amNk1nOO5VG8hcpCXucZVs8oUsp66yJg+VrG+BCuv8Sj1 VuIlrn1ixV9FAzTp+YFM/4pi8jPMbqyCjDthffiyEgv7GQJgmda+06qO3pqH8o9+kH JaLL+fZtldtew== From: Borislav Petkov To: riel@surriel.com Cc: Manali.Shukla@amd.com, akpm@linux-foundation.org, andrew.cooper3@citrix.com, jackmanb@google.com, jannh@google.com, kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, mhklinux@outlook.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, x86@kernel.org, zhengqi.arch@bytedance.com, Borislav Petkov Subject: [PATCH v15 02/11] x86/mm: Add INVLPGB feature and Kconfig entry Date: Tue, 4 Mar 2025 14:58:07 +0100 Message-ID: <20250304135816.12356-3-bp@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250304135816.12356-1-bp@kernel.org> References: <20250304135816.12356-1-bp@kernel.org> MIME-Version: 1.0 X-Stat-Signature: jku9fduqmpi65fcbysycue9nmnimftd7 X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: B0BF68001C X-Rspam-User: X-HE-Tag: 1741096710-777094 X-HE-Meta: U2FsdGVkX1+axrdu/5kRYV+h/+iq9c6nXrOAtcMjXnkFM3PEVek53LZNLEEcg2p4mrNdXsbQ2vtmTN1WSXHyyX/BOg9Sq4NQe4EPui+CyoavV8NwVGWLCTyTK4mNoSudlHcrKxQb21qKqVZkkKc077qO1HI4v2VZPPAHzs+nnapqiG+7AIX+1Cf9GDIC229tJLBP4YreCCnBWNVx3IIKIpqIZKalA5DYAxunHHjKvCatEdzec2s/XGAuInohYpPWwNktkzDv1LA+jqannCOUDTC0pVgao2/B8Gw55NIm4YqosWI34TMedIMIa9YgROwZnuxC2hE+f0Vt23bqWGo018ESgV25Auy0TMz4DLZUTT7Vgql6r3C7GPsGxNADfo75UpB3ZHPm2vXeixo+K3ZsRcmSbhlcrexS0gUjk2JUJt7Nd39nCF4umEY+nmGvEOH+bOzzOjpJmqCMv53S8YAed9gWSZ8zX9rJRA+ysQZq45nVo+1pG1yt9cXwWOMS1n5unJyYQdMInhxIwiV6+gveGzAeNBhi7PinYwArNUXzKDIB3vTpgy1QNSk2ICdYJwLLuOYy2qBZzNVI+4AFBnRaHrdIWqvKAtofHfIwk8nqaPtJScQwiwK+vNKac7F2AVtygfzwZYwKMebLSMa+pi6CbxhgUL7wwZtyu9nd6evjwYXvCI2YY50J12wcKDF+3//6wbAmfazzSvXlY4J5ZsIqpg3JWWNjhjl61XOHi1Kwdp+0Gm1IMgTVqbbrUHgPKGflH+09iTZYVhmETn8be8lRIIVoqARuriKdwHjWn9Uvk9TOcu+uckLRY7bSpmDLh01BB0C8XKKb4Ht0+WaMxVDNefQupq2va0lMwSNG9GK6Ny4Ww7ZVq++N1PLFhb3dHqyBBGvyMmssEBoiVO3NanC1J4pFbeLymZgblicb4sc2aCqwpfuM19nfdCp0t9zhQ/MdgDmfCj0V5qjlLhlgalb 6gkEt69M brE2TdSjHL2uTU/ADE57ovaaY+vKbbzQFhc0PE4OkkvPHnBNEvpS+YJDUgdhiKotX1s00w1qqSRIwTWQ7W//bP+0rAv6Z9k0WQmXCsAGi8eWoN1rISaYKE3na5nEmhhxIxEAX3c5wjsIh8vlL5nKtqjzzmAaW3EPkJwz4r9P9Z7aP0Q2oP/qHCGI9kVT4pRs94UMhfRzWV/Vpi61Y3cJpPTzGGS7R4Jn+94A8EcP/gmd88apzCcFnj+KcDw7fzvXhL7Zz6obC8jpcMap9RAEglmO9fzbKKo4DqdjTdV8/oHpldRvNg/AqE9NrEiPVqVy70f9QlA2nNaFcOIvs1VV2klVxo2f07ljoqa+9BprGJtbcBx4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Rik van Riel In addition, the CPU advertises the maximum number of pages that can be shot down with one INVLPGB instruction in CPUID. Save that information for later use. [ bp: use cpu_has(), typos, massage. ] Signed-off-by: Rik van Riel Signed-off-by: Borislav Petkov (AMD) Link: https://lore.kernel.org/r/20250226030129.530345-3-riel@surriel.com --- arch/x86/Kconfig.cpu | 4 ++++ arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/disabled-features.h | 8 +++++++- arch/x86/include/asm/tlbflush.h | 3 +++ arch/x86/kernel/cpu/amd.c | 6 ++++++ 5 files changed, 21 insertions(+), 1 deletion(-) diff --git a/arch/x86/Kconfig.cpu b/arch/x86/Kconfig.cpu index 2a7279d80460..25c55cc17c5e 100644 --- a/arch/x86/Kconfig.cpu +++ b/arch/x86/Kconfig.cpu @@ -401,6 +401,10 @@ menuconfig PROCESSOR_SELECT This lets you choose what x86 vendor support code your kernel will include. +config BROADCAST_TLB_FLUSH + def_bool y + depends on CPU_SUP_AMD && 64BIT + config CPU_SUP_INTEL default y bool "Support Intel processors" if PROCESSOR_SELECT diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 508c0dad116b..8770dc185fe9 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -338,6 +338,7 @@ #define X86_FEATURE_CLZERO (13*32+ 0) /* "clzero" CLZERO instruction */ #define X86_FEATURE_IRPERF (13*32+ 1) /* "irperf" Instructions Retired Count */ #define X86_FEATURE_XSAVEERPTR (13*32+ 2) /* "xsaveerptr" Always save/restore FP error pointers */ +#define X86_FEATURE_INVLPGB (13*32+ 3) /* INVLPGB and TLBSYNC instructions supported */ #define X86_FEATURE_RDPRU (13*32+ 4) /* "rdpru" Read processor register at user level */ #define X86_FEATURE_WBNOINVD (13*32+ 9) /* "wbnoinvd" WBNOINVD instruction */ #define X86_FEATURE_AMD_IBPB (13*32+12) /* Indirect Branch Prediction Barrier */ diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index c492bdc97b05..625a89259968 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -129,6 +129,12 @@ #define DISABLE_SEV_SNP (1 << (X86_FEATURE_SEV_SNP & 31)) #endif +#ifdef CONFIG_X86_BROADCAST_TLB_FLUSH +#define DISABLE_INVLPGB 0 +#else +#define DISABLE_INVLPGB (1 << (X86_FEATURE_INVLPGB & 31)) +#endif + /* * Make sure to add features to the correct mask */ @@ -146,7 +152,7 @@ #define DISABLED_MASK11 (DISABLE_RETPOLINE|DISABLE_RETHUNK|DISABLE_UNRET| \ DISABLE_CALL_DEPTH_TRACKING|DISABLE_USER_SHSTK) #define DISABLED_MASK12 (DISABLE_FRED|DISABLE_LAM) -#define DISABLED_MASK13 0 +#define DISABLED_MASK13 (DISABLE_INVLPGB) #define DISABLED_MASK14 0 #define DISABLED_MASK15 0 #define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \ diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 3da645139748..855c13da2045 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -183,6 +183,9 @@ static inline void cr4_init_shadow(void) extern unsigned long mmu_cr4_features; extern u32 *trampoline_cr4_features; +/* How many pages can be invalidated with one INVLPGB. */ +extern u16 invlpgb_count_max; + extern void initialize_tlbstate_and_flush(void); /* diff --git a/arch/x86/kernel/cpu/amd.c b/arch/x86/kernel/cpu/amd.c index 54194f5995de..7a72ef47a983 100644 --- a/arch/x86/kernel/cpu/amd.c +++ b/arch/x86/kernel/cpu/amd.c @@ -29,6 +29,8 @@ #include "cpu.h" +u16 invlpgb_count_max __ro_after_init; + static inline int rdmsrl_amd_safe(unsigned msr, unsigned long long *p) { u32 gprs[8] = { 0 }; @@ -1139,6 +1141,10 @@ static void cpu_detect_tlb_amd(struct cpuinfo_x86 *c) tlb_lli_2m[ENTRIES] = eax & mask; tlb_lli_4m[ENTRIES] = tlb_lli_2m[ENTRIES] >> 1; + + /* Max number of pages INVLPGB can invalidate in one shot */ + if (cpu_has(c, X86_FEATURE_INVLPGB)) + invlpgb_count_max = (cpuid_edx(0x80000008) & 0xffff) + 1; } static const struct cpu_dev amd_cpu_dev = { From patchwork Tue Mar 4 13:58:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Borislav Petkov X-Patchwork-Id: 14000779 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 143FCC021B8 for ; Tue, 4 Mar 2025 13:58:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 86A05280002; Tue, 4 Mar 2025 08:58:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 81A466B008C; Tue, 4 Mar 2025 08:58:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E6FF280002; Tue, 4 Mar 2025 08:58:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4BA286B008A for ; Tue, 4 Mar 2025 08:58:36 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id DFF885532E for ; Tue, 4 Mar 2025 13:58:35 +0000 (UTC) X-FDA: 83184023790.19.4846EEB Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf12.hostedemail.com (Postfix) with ESMTP id 4433F40018 for ; Tue, 4 Mar 2025 13:58:34 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=dyCCJ2el; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf12.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741096714; a=rsa-sha256; cv=none; b=WD6b4D3FcqZ9Qvo+vAOlotz4tCaKhBIH7A3XmgQbA9HXv850Hx24OqiEXQvDevTfqjOxdp UWsJmUfRcPjUQ8Wa6kVXqoRYBTvB+2Nl1ZNBbl9zfn0KwpD/LLHpIPgFxQQiCL+ucohunq K+s5vP1/heP+AG/Ns7cuA9QJ3FflMtc= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=dyCCJ2el; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf12.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741096714; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xEsQkPYQlYiTKuL2teGu/PlbVN6Z5HqX0RAu700uAWg=; b=qyhGPLUAldLvWw4GiFWD8djCxUlzPZiZW1tKehWjG0o7O+vriVVYxNLfU7JurwToa4Ebnq vvMuEmjTM5GHycr8LxR1St1AbqzKYy0Wl6MRGDhfHvPI9lnlid2TWI0PQofiIo4JqUmo2F aLelZLIkno4QegL6Ur5ShOa/FGpVOUY= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 941455C5D33; Tue, 4 Mar 2025 13:56:16 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4164FC4CEE7; Tue, 4 Mar 2025 13:58:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741096713; bh=X1huOsN5W4jKLuaRwLL36yDmkKY/JDVc78p3WqGQVBs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dyCCJ2el7Tll84ljQirbpN9UXDOr1tBvbVY6JChmK1K/KX8vxcYbzjQMOxB8YaB2g jn400jK3VU/APZOnikrergnf8HR5i6vNVfPZBTW/+1jFlsYjsbsgHj6wDQ8vcD3gH0 O9VpzREeaAaqFN8/MCnbe9k9BYbaI1JXYVSS1v+leMLf81F3zO1p2SN374YLDeRSjQ E6SK009MqxA+V8wUDnhUKbt/MtJxM9OzER6wgXmMoPuEITmoIFkZDYYqeTUqcKEcQI 7blFfCoHOzK8s3V89XKBFzaEC0NTe9Bjd05+qIorFP1czuA4GDJwToJ29cx/KPdJm0 GneKxW3yoWLxw== From: Borislav Petkov To: riel@surriel.com Cc: Manali.Shukla@amd.com, akpm@linux-foundation.org, andrew.cooper3@citrix.com, jackmanb@google.com, jannh@google.com, kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, mhklinux@outlook.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, x86@kernel.org, zhengqi.arch@bytedance.com, Borislav Petkov Subject: [PATCH v15 03/11] x86/mm: Add INVLPGB support code Date: Tue, 4 Mar 2025 14:58:08 +0100 Message-ID: <20250304135816.12356-4-bp@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250304135816.12356-1-bp@kernel.org> References: <20250304135816.12356-1-bp@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 4433F40018 X-Stat-Signature: y58q1gcpepz5ukcazzwgdy3h9jz9xwyp X-HE-Tag: 1741096714-880281 X-HE-Meta: U2FsdGVkX1+rI1vLj0yxPqvp1wU+nYPXpx3YkR3jN1SJ4lsX7wLRG6n3H1Pi8apYlzLy83QsqctAjzsi/CYNos/5dwbR923x1rh7UQnh4wqf+oTvuhSZSC48rYS8La4g3m+JqmpbFPxz1FSofrjhxSd0k/QkAU4NhIC1rTsT6xgyNkDxoigS+QcEzdsiDHCc4L/2va6mC+ZK1BQaKnSQzEo+uf3T3o6NnZ5hVjISy42fyoihjM5ERScjgdIy1+st7egZeYb5TmBdd0H4+n3mb97CcTNqIRx3IeB+Rv8tErtFK+EBbaiZY11qIDWDmXRBRacspUYa3cw1maYMmQFjghFSvq+cafNtoKCZgd5+6CR+G1EDfmioQmbFPt1i3bweMBxUCZyDV0c2Lo3szmzo/cRiKXgKxBg9XVDzqXedCc8lXZVYC5I6KZAPnChT67nUkPrJ9oSx5sOur/U9EnzcJs6mmtLCgmFiKK3FiYuwGMj0ndYqtVaoW0XcJAcGvj/PjhtKHV15REdvjPr0fM3yLNt1vTNeEzEJb2iVikiWmkRfU+c2RxRTlFGn8ljvqHjCWmvDhSvqF9rCGKxpG4qS0P81QJ9DTDBETNUzztmEt6+OMCaamcY6My31hIgd52Mhsy38j8xizZ92Nnf61/EHwDM164M4AU9rD+lWHgdYer0WVZizFVqW9y0L+Fbv93A/hrlanO8N2+las7Lt3OuX/LAQ+/7hU8sTt9nOHYl43i6nD8VltioT9xg17QIxr+Cjq4uH3zRxEayyvBGYjCCEyJ9nQBUr8sNBSUrG5stmIgCwZwniVH6NSfA340RZniY7l5oBEJ5CDirl1FBLREAIz26a85+Au/XfjPcQEbLu426GkStW6VLmV2+xaLBlXQRBPOkuZi/UZGBAeBAGfEc3WbgovpGhqGFIUq+Nra0Fr5JqsNs67di1OB746IfHIzH19kIDF0nAI14G2ELwE2T eERrqG7X lnLuW69e38he/0WbXDkimrLix7dFQyDutCx8gyptIIYsVSPAokWQpR5KxacNWb4vQIeweLAhev9YW/9rSnNWuv4QsoBWV/8dmnSoETkyAwxXq9uVUlyTSCHag9OSjoVdHdZA7CcKUmihcT+3DBZw3tE736mYPUgKm0WDtTMHSkjG1aZb7PHKPWKdO77DMXHpDiKPm1ftYDgY+5mMqlMfL97BS539AnTMiF2dYHOleJNQu+/zaqVMRsrwLnpb3OHXN+6LWFZl4JrlLdsQt2ne5OwPnB4p/cUbpTR6FLH2WJmnAY4Mi0K9JbzoMv/WlyuZBILaO3eDy70f8T1BHCz7YSrV04p3Fn7UqqQR5UZeptXrbw3A= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Rik van Riel Add helper functions and definitions needed to use broadcast TLB invalidation on AMD CPUs. [ bp: - Cleanup commit message - Improve and expand comments - push the preemption guards inside the invlpgb* helpers - merge improvements from dhansen - add !CONFIG_BROADCAST_TLB_FLUSH function stubs because Clang can't do DCE properly yet and looks at the inline asm and complains about it getting a u64 argument on 32-bit code ] Signed-off-by: Rik van Riel Signed-off-by: Borislav Petkov (AMD) Link: https://lore.kernel.org/r/20250226030129.530345-4-riel@surriel.com --- arch/x86/include/asm/tlb.h | 126 +++++++++++++++++++++++++++++++++++++ 1 file changed, 126 insertions(+) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 77f52bc1578a..8ffcae7beb55 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -6,6 +6,9 @@ static inline void tlb_flush(struct mmu_gather *tlb); #include +#include +#include +#include static inline void tlb_flush(struct mmu_gather *tlb) { @@ -25,4 +28,127 @@ static inline void invlpg(unsigned long addr) asm volatile("invlpg (%0)" ::"r" (addr) : "memory"); } +enum addr_stride { + PTE_STRIDE = 0, + PMD_STRIDE = 1 +}; + +#ifdef CONFIG_BROADCAST_TLB_FLUSH +/* + * INVLPGB does broadcast TLB invalidation across all the CPUs in the system. + * + * The INVLPGB instruction is weakly ordered, and a batch of invalidations can + * be done in a parallel fashion. + * + * The instruction takes the number of extra pages to invalidate, beyond + * the first page, while __invlpgb gets the more human readable number of + * pages to invalidate. + * + * The bits in rax[0:2] determine respectively which components of the address + * (VA, PCID, ASID) get compared when flushing. If neither bits are set, *any* + * address in the specified range matches. + * + * TLBSYNC is used to ensure that pending INVLPGB invalidations initiated from + * this CPU have completed. + */ +static inline void __invlpgb(unsigned long asid, unsigned long pcid, + unsigned long addr, u16 nr_pages, + enum addr_stride stride, u8 flags) +{ + u32 edx = (pcid << 16) | asid; + u32 ecx = (stride << 31) | (nr_pages - 1); + u64 rax = addr | flags; + + /* The low bits in rax are for flags. Verify addr is clean. */ + VM_WARN_ON_ONCE(addr & ~PAGE_MASK); + + /* INVLPGB; supported in binutils >= 2.36. */ + asm volatile(".byte 0x0f, 0x01, 0xfe" :: "a" (rax), "c" (ecx), "d" (edx)); +} + +static inline void __tlbsync(void) +{ + /* + * TLBSYNC waits for INVLPGB instructions originating on the same CPU + * to have completed. Print a warning if the task has been migrated, + * and might not be waiting on all the INVLPGBs issued during this TLB + * invalidation sequence. + */ + cant_migrate(); + + /* TLBSYNC: supported in binutils >= 0.36. */ + asm volatile(".byte 0x0f, 0x01, 0xff" ::: "memory"); +} +#else +/* Some compilers (I'm looking at you clang!) simply can't do DCE */ +static inline void __invlpgb(unsigned long asid, unsigned long pcid, + unsigned long addr, u16 nr_pages, + enum addr_stride s, u8 flags) { } +static inline void __tlbsync(void) { } +#endif + +/* + * INVLPGB can be targeted by virtual address, PCID, ASID, or any combination + * of the three. For example: + * - FLAG_VA | FLAG_INCLUDE_GLOBAL: invalidate all TLB entries at the address + * - FLAG_PCID: invalidate all TLB entries matching the PCID + * + * The first is used to invalidate (kernel) mappings at a particular + * address across all processes. + * + * The latter invalidates all TLB entries matching a PCID. + */ +#define INVLPGB_FLAG_VA BIT(0) +#define INVLPGB_FLAG_PCID BIT(1) +#define INVLPGB_FLAG_ASID BIT(2) +#define INVLPGB_FLAG_INCLUDE_GLOBAL BIT(3) +#define INVLPGB_FLAG_FINAL_ONLY BIT(4) +#define INVLPGB_FLAG_INCLUDE_NESTED BIT(5) + +/* The implied mode when all bits are clear: */ +#define INVLPGB_MODE_ALL_NONGLOBALS 0UL + +static inline void invlpgb_flush_user_nr_nosync(unsigned long pcid, + unsigned long addr, + u16 nr, bool stride) +{ + enum addr_stride str = stride ? PMD_STRIDE : PTE_STRIDE; + u8 flags = INVLPGB_FLAG_PCID | INVLPGB_FLAG_VA; + + __invlpgb(0, pcid, addr, nr, str, flags); +} + +/* Flush all mappings for a given PCID, not including globals. */ +static inline void invlpgb_flush_single_pcid_nosync(unsigned long pcid) +{ + __invlpgb(0, pcid, 0, 1, PTE_STRIDE, INVLPGB_FLAG_PCID); +} + +/* Flush all mappings, including globals, for all PCIDs. */ +static inline void invlpgb_flush_all(void) +{ + /* + * TLBSYNC at the end needs to make sure all flushes done on the + * current CPU have been executed system-wide. Therefore, make + * sure nothing gets migrated in-between but disable preemption + * as it is cheaper. + */ + guard(preempt)(); + __invlpgb(0, 0, 0, 1, PTE_STRIDE, INVLPGB_FLAG_INCLUDE_GLOBAL); + __tlbsync(); +} + +/* Flush addr, including globals, for all PCIDs. */ +static inline void invlpgb_flush_addr_nosync(unsigned long addr, u16 nr) +{ + __invlpgb(0, 0, addr, nr, PTE_STRIDE, INVLPGB_FLAG_INCLUDE_GLOBAL); +} + +/* Flush all mappings for all PCIDs except globals. */ +static inline void invlpgb_flush_all_nonglobals(void) +{ + guard(preempt)(); + __invlpgb(0, 0, 0, 1, PTE_STRIDE, INVLPGB_MODE_ALL_NONGLOBALS); + __tlbsync(); +} #endif /* _ASM_X86_TLB_H */ From patchwork Tue Mar 4 13:58:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Borislav Petkov X-Patchwork-Id: 14000780 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24E56C282D2 for ; Tue, 4 Mar 2025 13:58:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B02E3280003; Tue, 4 Mar 2025 08:58:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AB372280004; Tue, 4 Mar 2025 08:58:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 95434280003; Tue, 4 Mar 2025 08:58:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 6E8C86B008A for ; Tue, 4 Mar 2025 08:58:39 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 306411C53A8 for ; Tue, 4 Mar 2025 13:58:39 +0000 (UTC) X-FDA: 83184023958.24.B08941A Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf23.hostedemail.com (Postfix) with ESMTP id 8B3B0140015 for ; Tue, 4 Mar 2025 13:58:37 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=A7wXgql1; spf=pass (imf23.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741096717; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UauOJjf3CILBed2OKV/Xg7yN/eQK6t+qTH1V1u1wbJA=; b=bxEncVwG0G+On4PnlPa/YhT3ohWnww50of3iM4QQuaqEFvL9yk/ZheWPmdb4HqPpA20SNs jay0/uz8UqAGcbSKEyG6jBONyrCF2iola8NA+t3pju1h+Z+KLoTIFl3rZPlSh6/AryU1dE rYxi8N81IUNFDvjgk7Gbeh3OaiMw0jo= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=A7wXgql1; spf=pass (imf23.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741096717; a=rsa-sha256; cv=none; b=vx7dXd/rGtTysFpTT5or2Kqhe0xHeELhL6VYpceIAMJj8fA6cMDZYBGAV7mOQ/Z/xo7bwo tnY4laVcEx+dyZTOnaT/L4Uj9c9IObjATsURZjiOfWNzKi6V19K9uQ2iOOBo7SdYnb/sdu XqTjksbyQMRvdY07JWKLENYSMxnX+to= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 0504E5C5D57; Tue, 4 Mar 2025 13:56:20 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A76C2C4CEE9; Tue, 4 Mar 2025 13:58:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741096716; bh=SCBkxqxjfa8HkA0BTRuWWUnRPjL2osKJM/O5wsn1zgg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=A7wXgql127OzEvUasxqBMsRN6H4aOQJkUBD9QFFp0MlWYDxeAMoemW1MSkD4DdFqs Y+dzSoFejSn2qTJdDM/MMYQzqDcJHWXSkBLHFhvyJlXZnqTc+q9qU4EfWlJiPxNMri 4Ub42ppW+HYV/33X/CNhXK9xRzE0lEcRMQ2a+lFHZggSFPa8OWGKesGc7na8LhUotY 13TM2ptAtVMkYvywbkYqXhdYkHXtIFNDSsoD1E3T7O4ctYWFvG24yllC8qH3EJqjd5 ScPHoLs5KpaTir7CU2nrRL173YHIC5U/nkdibqRywUCjXPphsc/MxLuXvYg+Gj3Uqi 61rZwEDraemlg== From: Borislav Petkov To: riel@surriel.com Cc: Manali.Shukla@amd.com, akpm@linux-foundation.org, andrew.cooper3@citrix.com, jackmanb@google.com, jannh@google.com, kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, mhklinux@outlook.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, x86@kernel.org, zhengqi.arch@bytedance.com, Borislav Petkov Subject: [PATCH v15 04/11] x86/mm: Use INVLPGB for kernel TLB flushes Date: Tue, 4 Mar 2025 14:58:09 +0100 Message-ID: <20250304135816.12356-5-bp@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250304135816.12356-1-bp@kernel.org> References: <20250304135816.12356-1-bp@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: wzm8n8bhuya7rzajnjsa41q6tyk4qcya X-Rspamd-Queue-Id: 8B3B0140015 X-Rspamd-Server: rspam07 X-HE-Tag: 1741096717-51723 X-HE-Meta: U2FsdGVkX18KnDsaTbthlUd0/R5F8MOpMBv1g//H3nFNTWA0kQkuJizGEMmaqGW94Fgiqpu6172IXIfweUlICWM9J0kxDN+tMa66hTJzYeI5H2IHC3QxVY06r1saZdfgDnvBMt9yaMq0vRqFa4v0nuUB9UVADVuWt8iFChu4nVu1i1JvNklG0zSvGC0vXycyxeKZfIOua2QIkF7pe5CELkCWDP+kKx4zKdZ5F4eXsXRIqtUuN2MW5aY06oYqnbK8rQLPuM778uEqFLXMrlF+yzYUQ/JgI668h1eF1R4MWqPHiHFFPG05Jcn7mSB03JcA/lrPXGb5URbKyuBW9H9B5jG7Aldc7ViZUYUgYWqIFZsBhMFD4jZuOMlXIxyrHkl8t7SgiHyf/6fTp3qbJKI6H3T7j82LlUgKh1D4dgOh3gFZlB1mFYDAUM/k2C+sGCZsdCcW8Mc8EXLeFtno9TxVpDoouDV3Oh05/EMyxWfcm14FIX76hgFl+idwLDJWtvieEMKEv+CH/sEzq7MjFM1WSRIxUOJ0gS9GFcVFymt5mzwRhZ4XMupRl2rOW74tcVGVTp0kLHqKMW/FrNpCH1jQKdlWJ/StgLJ4lZ1qVL02x+GJxwG4vKsCRsnOOYewVA8U3F1YxzAh2kXbokzUwyjitI0Qd+MLh1GAsw9Wkge/PGCsNcEQT00937YcZ+5TsMQNP5qw0k4ZFPwiOPa2ejgcyMalkQyCYnx0tb3/HxmEKZsWb4pJJJRr/mszgDewxa+wLnomNDxV4L6grTs2c/TL5WhBCpBaycrPAV/zeF1p6OyjrGyU+7A7mBetz7yLIN9fxKwhfQkmUqDuCzthCcZhr6+2/Q0+aZbbe8a6Ecxe/4Bl/QfftiwfeY1stsRVjJq5++5D4Z0cYwrWz1iarAJBJmSb4kF1j7oGpDHHZMOhvN8eQ42+37KJtpxYmpomp1QoYuKDXP3os3y6PnnypCP EKOW9y2R NodiIgyEMTgqIjJJ9w0+559d+jpULrwesaF8qqWcgHDsgETZopKPdd8/Dc3NrSyQE9CC2fbp0kzNPyNQ9W+d8PLf6WiC3TsPEv4uuqOUsTVU+evWhdQq8SVRmrXhKvoqtPiBLtXM0Bf1rQbvgaaRiYQ8QZsh1FiIuzvWJzC+zinxxTQ3qciZAlsCI8cFtmZiOk8WiNoAXSmlTwxtSUfyT1PfLAMJG1dBvqfVtHXvYmxTiU76/Vc37DwwXrQeM4/7N3pY8a/JsMN1TuKjBUi7Q070l8t8uZYZ67tGNgSIwoJMe4qS3WAyUpU0TWIaVfhBKw35nO+G2shH2IecJ3GByuAYQbewY0FO1JxXeFSoIVVlJqMw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Rik van Riel Use broadcast TLB invalidation for kernel addresses when available. Remove the need to send IPIs for kernel TLB flushes. [ bp: Integrate dhansen's comments additions, merge the flush_tlb_all() change into this one too. ] Signed-off-by: Rik van Riel Signed-off-by: Borislav Petkov (AMD) Link: https://lore.kernel.org/r/20250226030129.530345-5-riel@surriel.com --- arch/x86/mm/tlb.c | 48 +++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 44 insertions(+), 4 deletions(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index dbcb5c968ff9..8cd084bc3d98 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1064,7 +1064,6 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, mmu_notifier_arch_invalidate_secondary_tlbs(mm, start, end); } - static void do_flush_tlb_all(void *info) { count_vm_tlb_event(NR_TLB_REMOTE_FLUSH_RECEIVED); @@ -1074,7 +1073,32 @@ static void do_flush_tlb_all(void *info) void flush_tlb_all(void) { count_vm_tlb_event(NR_TLB_REMOTE_FLUSH); - on_each_cpu(do_flush_tlb_all, NULL, 1); + + /* First try (faster) hardware-assisted TLB invalidation. */ + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) + invlpgb_flush_all(); + else + /* Fall back to the IPI-based invalidation. */ + on_each_cpu(do_flush_tlb_all, NULL, 1); +} + +/* Flush an arbitrarily large range of memory with INVLPGB. */ +static void invlpgb_kernel_range_flush(struct flush_tlb_info *info) +{ + unsigned long addr, nr; + + for (addr = info->start; addr < info->end; addr += nr << PAGE_SHIFT) { + nr = (info->end - addr) >> PAGE_SHIFT; + + /* + * INVLPGB has a limit on the size of ranges it can + * flush. Break up large flushes. + */ + nr = clamp_val(nr, 1, invlpgb_count_max); + + invlpgb_flush_addr_nosync(addr, nr); + } + __tlbsync(); } static void do_kernel_range_flush(void *info) @@ -1087,6 +1111,22 @@ static void do_kernel_range_flush(void *info) flush_tlb_one_kernel(addr); } +static void kernel_tlb_flush_all(struct flush_tlb_info *info) +{ + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) + invlpgb_flush_all(); + else + on_each_cpu(do_flush_tlb_all, NULL, 1); +} + +static void kernel_tlb_flush_range(struct flush_tlb_info *info) +{ + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) + invlpgb_kernel_range_flush(info); + else + on_each_cpu(do_kernel_range_flush, info, 1); +} + void flush_tlb_kernel_range(unsigned long start, unsigned long end) { struct flush_tlb_info *info; @@ -1097,9 +1137,9 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end) TLB_GENERATION_INVALID); if (info->end == TLB_FLUSH_ALL) - on_each_cpu(do_flush_tlb_all, NULL, 1); + kernel_tlb_flush_all(info); else - on_each_cpu(do_kernel_range_flush, info, 1); + kernel_tlb_flush_range(info); put_flush_tlb_info(); } From patchwork Tue Mar 4 13:58:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Borislav Petkov X-Patchwork-Id: 14000781 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20805C021B8 for ; Tue, 4 Mar 2025 13:58:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A66116B008C; Tue, 4 Mar 2025 08:58:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A21BA280005; Tue, 4 Mar 2025 08:58:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8DE09280004; Tue, 4 Mar 2025 08:58:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 7172D6B008C for ; Tue, 4 Mar 2025 08:58:43 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1D8A4141D06 for ; Tue, 4 Mar 2025 13:58:43 +0000 (UTC) X-FDA: 83184024126.05.8A179B3 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf08.hostedemail.com (Postfix) with ESMTP id 8272416000C for ; Tue, 4 Mar 2025 13:58:41 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=kd8Yw7Lw; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf08.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741096721; a=rsa-sha256; cv=none; b=QRPj8+xg8SZCRUNIL6PHLSGbCOdLqFn2ZFWw4avT6M3qXldZ0pQYFGVhAAZOws7TISaXW4 tpTqjRK+PbOa0wYy7y5tiUJonm5VJUQbT8apfDhCypBp8IiJ94/yMqxvbHUzXDSb1GyDr8 YwMXCEHbkEIXRgx/DglzZebM/dramoQ= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=kd8Yw7Lw; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf08.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741096721; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=omTic7yHPg2GYmfJbHtpwTPhyEj1T7bmPvF7pe9loYw=; b=0cFdiTWQxvZ4S8IwbuikAQ3iywMZ3VBwKU6xfFuUK2QA8qwnT6SJgsCtRpzsEclDML1DJf /MjH+dZeREe1pwms/tb4nd66Bi+nbzW5w3tQLf4Zkw9zSHKc6EnpKEuwK6ztJlVIITgrtk gm50WiYpxZwLviWk6s3KSCuX9ZVFR/8= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id E27865C5CF5; Tue, 4 Mar 2025 13:56:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 174FDC4CEE5; Tue, 4 Mar 2025 13:58:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741096720; bh=txoN3V9rWwXFEyWDRXIoMCIFbMoUCiBnnRfSGjS9m3M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kd8Yw7Lwr4sjOpYWFXwXQSEC5xzprBOCIBE7bXwYr8GTXC+xFDjpLzy06CmfMPBne u6ZdRClLHEfy/M+wAKbra6zqvNXfIiC4v94rdgan9Rd1Z/UVXIeuRLeQOB0w8w0Xlu 5D8xfVyMQnYo7qU7qdPIZZsF1BViGBgk33U39jmiZdbrGUj9z3t4C8bt3CBSbFT5l9 /2r7w6F9gxN5CfJM9sClY0EMsUCrLtI0k6as50Bjxm6gCLyKhbZy2BdTMpSEFVRycd o0c3oY9PoYlFvhAh3shml2NRuxEbfXoD8Rlj1YvT24ywiz2F33RnzIx9Taqe6dint8 rD4d7P/Jk/CkA== From: Borislav Petkov To: riel@surriel.com Cc: Manali.Shukla@amd.com, akpm@linux-foundation.org, andrew.cooper3@citrix.com, jackmanb@google.com, jannh@google.com, kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, mhklinux@outlook.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, x86@kernel.org, zhengqi.arch@bytedance.com, Borislav Petkov Subject: [PATCH v15 05/11] x86/mm: Use broadcast TLB flushing in page reclaim Date: Tue, 4 Mar 2025 14:58:10 +0100 Message-ID: <20250304135816.12356-6-bp@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250304135816.12356-1-bp@kernel.org> References: <20250304135816.12356-1-bp@kernel.org> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 8272416000C X-Stat-Signature: aa8bg7s9ssfy943asft1x7d8htc44deh X-Rspam-User: X-HE-Tag: 1741096721-868233 X-HE-Meta: U2FsdGVkX180L4Ol8T1lL9wR978FT29qP0BDSMARdEEqjYLXpEMEzykS6QpDuV/6tD2G2EetKIvoH5bawfRCRPBx1iwBCE2F0kCTtYXNmwrnJMFE0sD4y603M0UogkWyhtBEa4IN1jMPH/m4yiMVs+UcQYymRV3mNsb7ZPST8XyoeP68/HeVIH0z9LKYpdl3vkbmyU6RYRrkXvvThCSwlbTNrRqqxgjhFSrqp1Bo08sbf0ExOReCr1oD5dlrSeqYEZiyeZiflZnR/Uia708RjimBAKDVmi0+9lQqVz45EcTNAP0C0RDpflto6OmbvF9jewGEO/0fbaZw5LYjW8pMU6w0WSKLBKEPKdOtXIBjiToR0YLS60v09jOBVglaefY6LUp9WOC7i0AHkuhIgkzTrRS3tu8jygGhNTv+XVW8NyUB5yi3kc9+L7laJIaEMGIvnlWDtAhSS1oEWWxotpqapZMsTP6jgCUCux+x9WXoAnlendPUFEpkL5MeSTilcJXDm9xZNHTM6VEMbA6+D2WVab7o5QtfXWD1jmQ5I48ARNyuDRi1OhrdIJoxKpygsq0Zwn7lHM3myFqEuUkYIL7fxqsJf7EnQqbpwYMIbclQTEgcqDILKpEswDKFP+7iP2cZOkwTRDxocKo5+V7cpJ3N/nviIK/bvDT0AgEqnW714wfklLYQ4HSluApLjCmUgUWjAQU+AmIyKrpqzH6Wyw2xqVLUpdcAxtKFNFMvwS7OGgn+qzuZLELHafvOxDcGMK21q1SdqblVJsi/7chdD6dXarTk9whAcbj2/2pwAnJU0w4l5Sh/yf8/MjMklHaFlNK9gq+O0K9PACgygp5veyotzv18q6sZfS+d60vHlcEJiqEDYNLjawo9dXP76hcjvDmP9jatuvWJdYdFOB22SbMYUm4CcavrlTiVImM/Pj7dWC4sVUXbpPmCUHkiDLHbx6QCij0/vpuq75EBd+SKmmR NADe2qXC 1t8hijquCiT0vKSWTemS/wvGeKppt6Em2yAVoiNs1Uj1/tqueP+Ad0nhTsTz6R7G3KmySOnB+eDQTedJTKXfIirIHX1tWkcNShkzlmPdHZMjFFL67vCHCfO8Xr9FiwbAHEzvHb41Ia8qVVOGEErqOGlm7U7zaciEFe1occxoiftbxWUTDFtGtF9qjyrEqzCIX/Deyq4L/lYRyDQCUtyLnlq9NprKEey4YbWo3C9LyV12tWuGjAiN9XNHk+i94QIy6PidS6RcDcDoVQuUu57erQtZiZR25wODJhrTqno6kHpBAw2jaPNk02OznSGKFMZeYyQwZqlhZvpKKxbX0I1WCoTaxlxK1qmMjxpuaWjTS1sNNXms= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Rik van Riel Page reclaim tracks only the CPU(s) where the TLB needs to be flushed, rather than all the individual mappings that may be getting invalidated. Use broadcast TLB flushing when that is available. [ bp: Massage commit message. ] Signed-off-by: Rik van Riel Signed-off-by: Borislav Petkov (AMD) Link: https://lore.kernel.org/r/20250226030129.530345-7-riel@surriel.com --- arch/x86/mm/tlb.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 8cd084bc3d98..76b4a88afb56 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1320,7 +1320,9 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) * a local TLB flush is needed. Optimize this use-case by calling * flush_tlb_func_local() directly in this case. */ - if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) { + invlpgb_flush_all_nonglobals(); + } else if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { flush_tlb_multi(&batch->cpumask, info); } else if (cpumask_test_cpu(cpu, &batch->cpumask)) { lockdep_assert_irqs_enabled(); From patchwork Tue Mar 4 13:58:11 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Borislav Petkov X-Patchwork-Id: 14000782 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75934C021B8 for ; Tue, 4 Mar 2025 13:58:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A367280005; Tue, 4 Mar 2025 08:58:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 02E4F280004; Tue, 4 Mar 2025 08:58:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE887280005; Tue, 4 Mar 2025 08:58:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id BAFB0280004 for ; Tue, 4 Mar 2025 08:58:46 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 37B75B844C for ; Tue, 4 Mar 2025 13:58:46 +0000 (UTC) X-FDA: 83184024252.06.3ECBAF4 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf16.hostedemail.com (Postfix) with ESMTP id 7EF5018000C for ; Tue, 4 Mar 2025 13:58:44 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=JP9sY8U5; spf=pass (imf16.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741096724; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RmYDNuIaKqfregEo7bu+oqZ4mQLzeqeyfM+8dFpG3NY=; b=hLEL5OGUqYZBxmJmTp/z81AxceTkinx56blWJPoLAojjTkCWbQ4XPdQa6gbA/LF/CxQMvE LSPqhb6dYtgWhMNmG6qmADUcwUBsjlp9671hXVTymOP3Ark2S/bgCgTD2x0SmxYIW1HBB0 apbw1vnCCoafcvT+O6W209Nmk0W2z34= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=JP9sY8U5; spf=pass (imf16.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741096724; a=rsa-sha256; cv=none; b=UVEp1gjpMWcsajfHrN9Us/CEibxkdmuVBJJruMzEVWHKKLs+kxXyCAZdQh9pAOCLeklCfO yoVvD7EJ9HXEW0491xjK0Y1ivknmD41su2GrzfAYYTiOc7Tojuh+Qs7LpEe5mZA9+WA3yo JZ+t8wHiiOOV8bIdA4STj++dMNUf+BU= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id F154A5C5D4C; Tue, 4 Mar 2025 13:56:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7DE90C4CEEB; Tue, 4 Mar 2025 13:58:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741096723; bh=AVkerff15j+dPRSSgj4ss5DTO/4GX95JT9jOuxMQPW4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JP9sY8U5pmzL0VXVtQ8DDXVIcr5nWRhexUni6oc/fUrknFMJtHLFHKJRB1st1emWQ kU5CCsTVKjRpi6pn/M9QIA9yFglTQfOdkolgWfCWMMUZO434DUnu3IRQ4hoF7bEvMq dABsMZf1WIAzxP0r25tMwC1VddoeVqG0FwTLJR9iSLjJy4ECVG7F2H7qe9VlML8JNc saYG1dibzGIgBoZg3HVjRejXpCqBiYYKVorKP2ltAhoO1jbrgfkPEMJhLZm/vbHfQn 9hMn+gaJCyiYpVVD5KEqdLlbxqxt8q2+eTMHWyZHf/si6HoKkBa+442e09197SgHqA wup9zdwT5oiKQ== From: Borislav Petkov To: riel@surriel.com Cc: Manali.Shukla@amd.com, akpm@linux-foundation.org, andrew.cooper3@citrix.com, jackmanb@google.com, jannh@google.com, kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, mhklinux@outlook.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, x86@kernel.org, zhengqi.arch@bytedance.com, Borislav Petkov Subject: [PATCH v15 06/11] x86/mm: Add global ASID allocation helper functions Date: Tue, 4 Mar 2025 14:58:11 +0100 Message-ID: <20250304135816.12356-7-bp@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250304135816.12356-1-bp@kernel.org> References: <20250304135816.12356-1-bp@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 7EF5018000C X-Rspamd-Server: rspam09 X-Stat-Signature: stqbcguo7xfuw4zxo5o4jei1j65fcqpg X-HE-Tag: 1741096724-123550 X-HE-Meta: U2FsdGVkX19N4EjqMoACOJQbJ8raAO2LlGfwHN4hZB/hoC2GyO6qYMQQdj8jEKbz0R0UfCuxSMvArxvVdEuIt5tBn4OX6J75r5cTuEAXMXeEBubOxck3Szq0h2FG2Nw+6t5OJMt2sSad7pCuPcwljW61SEbanVtLd4IlPg3B97q5YxB29/4Xuk95ylNl4YqXH6s2MUSpLTBQF3RpjKgRbTcIYal6tJnyKQ1/X9mYvtBTi2k8MLUFt1L2ofQd7rr/IE1gZqU8StmHzqvGMpMbvbRAnxcg+JY1VImOWDEiO3MOTE0HD61qt7BrB+Nbu75gYCQLHuJFET5UkQIYWdmmVp2AkDQZ5QBOdahe8D8Jr5cxdcgASccR2+uSFjGt65oP20xXKGAKpnl4qAqed22ro7bcI5/da4tR+7DlFHL5F3QJ/5RU9WCnkdQz52NZxYP+BmC10InwquPLcQQvOLcAmK4AEpxgBhkMMCdGdjl9QG1AgGVBl5qdTzm77Jncuj8VhEyvs+EAL6gXDnYYG74NO4nagqP0XVkXAHUS41Uj8N2xXycXCIx5eCP+r7m+Wzzs4+Hp1l7p69OJpXWB6aWbLTBpZ/9U4Ut7XAkwvrGimuaMo6L/6/YpSoqB6nGjZ2PakpZ2uY6WAlFSXwsJLbRFtOnP9+8aVFKmC6/EPeYGZUwJK1V21k5CkKUcOmvJTlScOjIDJmi8imkjUlBM9hIUWYykOd7E8IWp1R5QM0LbdsTHeHl8KQasjSbV5OIwxN3SAD8M5+ojW5BFL10IogCymFfQjLRMFfIUyXKiaHtrWzABE/hVGj1oWjTgGQqpiD1BFwvVtiVnPYcu1vsaYBRrsA7egU0YuaH4t4xZU1AXbYaBPX12lxsaMEKzi8MyTH0s+qbVbUEpPJLUDQbYJXbovCXav+Gu3MHfZLZZ2fuTD8j6yUBk5szR4RaKwvVGwcLy8r5N7Dx2wTGewKr9NLq l4qCwGtg sHJdG2jWaB1q1NAJdS4rbvjlkx9xIq3un7a5LM5i1SmyMwEWPc9Bh8Sbpu7MyAL1YL0rsbefCuzBCvumeW/RQR7T2DDUzCjrNLrxqhqjGXjciVIEpObA60wTs7dMQL2vNL3REDgNu02gmZt3Hqn2MXKmTvDucsQDLkE0uhnQYUffIejhN5rUw+Uc8CHRVIQbIc2Sca1nTMOVNDnnyzDTCW2bZR0YDeMwRwoCj8tceFpn93rxtjvMOuOFtCdIZt2VFZqDufuXCM9XHY5QpwTZ+fMb1DW8vWY0ZcHhmSfrCsVpUN4u82LwwTF0aig5m/s2Wt5IoUw2YiwD9EQqI1e4PZx5NZ0ox3hTNht75nYPh5Z6QimY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Rik van Riel Add functions to manage global ASID space. Multithreaded processes that are simultaneously active on 4 or more CPUs can get a global ASID, resulting in the same PCID being used for that process on every CPU. This in turn will allow the kernel to use hardware-assisted TLB flushing through AMD INVLPGB or Intel RAR for these processes. [ bp: - Extend use_global_asid() comment - s/X86_BROADCAST_TLB_FLUSH/BROADCAST_TLB_FLUSH/g - other touchups ] Signed-off-by: Rik van Riel Signed-off-by: Borislav Petkov (AMD) Link: https://lore.kernel.org/r/20250226030129.530345-8-riel@surriel.com --- arch/x86/include/asm/mmu.h | 12 +++ arch/x86/include/asm/mmu_context.h | 2 + arch/x86/include/asm/tlbflush.h | 37 +++++++ arch/x86/mm/tlb.c | 154 ++++++++++++++++++++++++++++- 4 files changed, 202 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index 3b496cdcb74b..8b8055a8eb9e 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -69,6 +69,18 @@ typedef struct { u16 pkey_allocation_map; s16 execute_only_pkey; #endif + +#ifdef CONFIG_BROADCAST_TLB_FLUSH + /* + * The global ASID will be a non-zero value when the process has + * the same ASID across all CPUs, allowing it to make use of + * hardware-assisted remote TLB invalidation like AMD INVLPGB. + */ + u16 global_asid; + + /* The process is transitioning to a new global ASID number. */ + bool asid_transition; +#endif } mm_context_t; #define INIT_MM_CONTEXT(mm) \ diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 795fdd53bd0a..a2c70e495b1b 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -139,6 +139,8 @@ static inline void mm_reset_untag_mask(struct mm_struct *mm) #define enter_lazy_tlb enter_lazy_tlb extern void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk); +extern void mm_free_global_asid(struct mm_struct *mm); + /* * Init a new mm. Used on mm copies, like at fork() * and on mm's that are brand-new, like at execve(). diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 855c13da2045..f7b374bcdc0b 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -6,6 +6,7 @@ #include #include +#include #include #include #include @@ -234,6 +235,42 @@ void flush_tlb_one_kernel(unsigned long addr); void flush_tlb_multi(const struct cpumask *cpumask, const struct flush_tlb_info *info); +static inline bool is_dyn_asid(u16 asid) +{ + return asid < TLB_NR_DYN_ASIDS; +} + +#ifdef CONFIG_BROADCAST_TLB_FLUSH +static inline u16 mm_global_asid(struct mm_struct *mm) +{ + u16 asid; + + if (!cpu_feature_enabled(X86_FEATURE_INVLPGB)) + return 0; + + asid = smp_load_acquire(&mm->context.global_asid); + + /* mm->context.global_asid is either 0, or a global ASID */ + VM_WARN_ON_ONCE(asid && is_dyn_asid(asid)); + + return asid; +} + +static inline void mm_assign_global_asid(struct mm_struct *mm, u16 asid) +{ + /* + * Notably flush_tlb_mm_range() -> broadcast_tlb_flush() -> + * finish_asid_transition() needs to observe asid_transition = true + * once it observes global_asid. + */ + mm->context.asid_transition = true; + smp_store_release(&mm->context.global_asid, asid); +} +#else +static inline u16 mm_global_asid(struct mm_struct *mm) { return 0; } +static inline void mm_assign_global_asid(struct mm_struct *mm, u16 asid) { } +#endif /* CONFIG_BROADCAST_TLB_FLUSH */ + #ifdef CONFIG_PARAVIRT #include #endif diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 76b4a88afb56..6c24d967b77d 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -74,13 +74,15 @@ * use different names for each of them: * * ASID - [0, TLB_NR_DYN_ASIDS-1] - * the canonical identifier for an mm + * the canonical identifier for an mm, dynamically allocated on each CPU + * [TLB_NR_DYN_ASIDS, MAX_ASID_AVAILABLE-1] + * the canonical, global identifier for an mm, identical across all CPUs * - * kPCID - [1, TLB_NR_DYN_ASIDS] + * kPCID - [1, MAX_ASID_AVAILABLE] * the value we write into the PCID part of CR3; corresponds to the * ASID+1, because PCID 0 is special. * - * uPCID - [2048 + 1, 2048 + TLB_NR_DYN_ASIDS] + * uPCID - [2048 + 1, 2048 + MAX_ASID_AVAILABLE] * for KPTI each mm has two address spaces and thus needs two * PCID values, but we can still do with a single ASID denomination * for each mm. Corresponds to kPCID + 2048. @@ -251,6 +253,152 @@ static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, *need_flush = true; } +/* + * Global ASIDs are allocated for multi-threaded processes that are + * active on multiple CPUs simultaneously, giving each of those + * processes the same PCID on every CPU, for use with hardware-assisted + * TLB shootdown on remote CPUs, like AMD INVLPGB or Intel RAR. + * + * These global ASIDs are held for the lifetime of the process. + */ +static DEFINE_RAW_SPINLOCK(global_asid_lock); +static u16 last_global_asid = MAX_ASID_AVAILABLE; +static DECLARE_BITMAP(global_asid_used, MAX_ASID_AVAILABLE); +static DECLARE_BITMAP(global_asid_freed, MAX_ASID_AVAILABLE); +static int global_asid_available = MAX_ASID_AVAILABLE - TLB_NR_DYN_ASIDS - 1; + +/* + * When the search for a free ASID in the global ASID space reaches + * MAX_ASID_AVAILABLE, a global TLB flush guarantees that previously + * freed global ASIDs are safe to re-use. + * + * This way the global flush only needs to happen at ASID rollover + * time, and not at ASID allocation time. + */ +static void reset_global_asid_space(void) +{ + lockdep_assert_held(&global_asid_lock); + + invlpgb_flush_all_nonglobals(); + + /* + * The TLB flush above makes it safe to re-use the previously + * freed global ASIDs. + */ + bitmap_andnot(global_asid_used, global_asid_used, + global_asid_freed, MAX_ASID_AVAILABLE); + bitmap_clear(global_asid_freed, 0, MAX_ASID_AVAILABLE); + + /* Restart the search from the start of global ASID space. */ + last_global_asid = TLB_NR_DYN_ASIDS; +} + +static u16 allocate_global_asid(void) +{ + u16 asid; + + lockdep_assert_held(&global_asid_lock); + + /* The previous allocation hit the edge of available address space */ + if (last_global_asid >= MAX_ASID_AVAILABLE - 1) + reset_global_asid_space(); + + asid = find_next_zero_bit(global_asid_used, MAX_ASID_AVAILABLE, last_global_asid); + + if (asid >= MAX_ASID_AVAILABLE && !global_asid_available) { + /* This should never happen. */ + VM_WARN_ONCE(1, "Unable to allocate global ASID despite %d available\n", + global_asid_available); + return 0; + } + + /* Claim this global ASID. */ + __set_bit(asid, global_asid_used); + last_global_asid = asid; + global_asid_available--; + return asid; +} + +/* + * Check whether a process is currently active on more than @threshold CPUs. + * This is a cheap estimation on whether or not it may make sense to assign + * a global ASID to this process, and use broadcast TLB invalidation. + */ +static bool mm_active_cpus_exceeds(struct mm_struct *mm, int threshold) +{ + int count = 0; + int cpu; + + /* This quick check should eliminate most single threaded programs. */ + if (cpumask_weight(mm_cpumask(mm)) <= threshold) + return false; + + /* Slower check to make sure. */ + for_each_cpu(cpu, mm_cpumask(mm)) { + /* Skip the CPUs that aren't really running this process. */ + if (per_cpu(cpu_tlbstate.loaded_mm, cpu) != mm) + continue; + + if (per_cpu(cpu_tlbstate_shared.is_lazy, cpu)) + continue; + + if (++count > threshold) + return true; + } + return false; +} + +/* + * Assign a global ASID to the current process, protecting against + * races between multiple threads in the process. + */ +static void use_global_asid(struct mm_struct *mm) +{ + u16 asid; + + guard(raw_spinlock_irqsave)(&global_asid_lock); + + /* This process is already using broadcast TLB invalidation. */ + if (mm_global_asid(mm)) + return; + + /* + * The last global ASID was consumed while waiting for the lock. + * + * If this fires, a more aggressive ASID reuse scheme might be + * needed. + */ + if (!global_asid_available) { + VM_WARN_ONCE(1, "Ran out of global ASIDs\n"); + return; + } + + asid = allocate_global_asid(); + if (!asid) + return; + + mm_assign_global_asid(mm, asid); +} + +void mm_free_global_asid(struct mm_struct *mm) +{ + if (!cpu_feature_enabled(X86_FEATURE_INVLPGB)) + return; + + if (!mm_global_asid(mm)) + return; + + guard(raw_spinlock_irqsave)(&global_asid_lock); + + /* The global ASID can be re-used only after flush at wrap-around. */ +#ifdef CONFIG_BROADCAST_TLB_FLUSH + __set_bit(mm->context.global_asid, global_asid_freed); + + mm->context.global_asid = 0; + global_asid_available++; +#endif +} + /* * Given an ASID, flush the corresponding user ASID. We can delay this * until the next time we switch to it. From patchwork Tue Mar 4 13:58:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Borislav Petkov X-Patchwork-Id: 14000783 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94700C282C6 for ; Tue, 4 Mar 2025 13:58:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 25182280006; Tue, 4 Mar 2025 08:58:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 202C3280004; Tue, 4 Mar 2025 08:58:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 05475280006; Tue, 4 Mar 2025 08:58:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D75FE280004 for ; Tue, 4 Mar 2025 08:58:49 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 957A5C1D8B for ; Tue, 4 Mar 2025 13:58:49 +0000 (UTC) X-FDA: 83184024378.11.75B1E57 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf17.hostedemail.com (Postfix) with ESMTP id EFA5A4000F for ; Tue, 4 Mar 2025 13:58:47 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="i7QlZWM/"; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf17.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741096728; a=rsa-sha256; cv=none; b=PhB7JCP5s+ceYTgCDLW0WGlg3AZ+bpemq16Z2BqYXNPCeYPfCJUJYxRZ2uToIY5oGxexUn Y6raN6EA6Y9TlFG/7OzCv3BeC0aRfrWzyDJdB8t0KoWC3th9Z7/DWDc/6GEZl4BtTuTQ0U gSeBBZw9vsfqBuCt4AYUipP8QUHLelM= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="i7QlZWM/"; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf17.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741096728; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tRO3ZdI0vGr6ql9kI7Cnzh5EjxtVoL8FFEwty0575Tg=; b=4nTVUW/841imTBlDcpNj7d748gNit0dReoIir/5wwmlfaQfNpsIJ68umN1wdnJPHxXZKdK NP3Q6d0QK2gzUFvPmF9OxXiTMhT7K5Pxl9OKpqizhI1b5sQ5id6L0WJpAPus/8ijgAuSt2 NtzUgDyNGU+pFMLUjQLQkmyK0EOqY/8= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 685D85C5D36; Tue, 4 Mar 2025 13:56:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E4F4FC4CEE9; Tue, 4 Mar 2025 13:58:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741096726; bh=peFXxf6pvLf5YEyvcQ+VbBtK2U9T/s+L+AAUIiO804s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=i7QlZWM/CvKDjp1KSyaZW0Gg+U5tDAu/tniChxnze6F0p3R4UsBk4jH0EF/Kzs3Mx 0moEMEQ3opI+uD/pw7uBzKAS6sbhHhSJkBE4pKAC4hIiyOQ1I6bV7Ae63yPKmdKtNh EtB/ZqyF/dPQKdEpvFBzOZkzsiRzXZcD+vXEUXJhYa2R72hMWWFqkkmB67ZwlqnTQn R512uPWnT7sVtlM3yK4XhbSY1taGfbn8u8PdmACuqr8Kn2V4o7m3/G28tHK6YZBDx8 uQU61uZPanCTq0lOqmnhjIkUF1YceJP0PeCCPXUZaTWx8qKKsnlzwJKmxmTvA+CC84 CkqRKyss0jfHA== From: Borislav Petkov To: riel@surriel.com Cc: Manali.Shukla@amd.com, akpm@linux-foundation.org, andrew.cooper3@citrix.com, jackmanb@google.com, jannh@google.com, kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, mhklinux@outlook.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, x86@kernel.org, zhengqi.arch@bytedance.com, Borislav Petkov Subject: [PATCH v15 07/11] x86/mm: Handle global ASID context switch and TLB flush Date: Tue, 4 Mar 2025 14:58:12 +0100 Message-ID: <20250304135816.12356-8-bp@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250304135816.12356-1-bp@kernel.org> References: <20250304135816.12356-1-bp@kernel.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: EFA5A4000F X-Stat-Signature: rh7zpdrrrz8kjiibcfpn34jsno9494k8 X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1741096727-200870 X-HE-Meta: U2FsdGVkX18VbNkta/SYK2/eskbCX9fusVRPlFc8gkOd9GPqMgcRIizGvGidxBLV+Yr9n1Rd7BBHRiSQe7SBwY0pxB5UIEFWluHh31th0haW4ERKB1anhZCm2NbGIxv8g72fzY1IMRpdP+52Yei2n62ZXWOkrjdH3HyHB2l8bL5K6wZKEnAVFAXCVDKuyRzRJ+e6BrVEP/OXx6b/fMdiWBFRFSEoHOII9uW3UbLKrSfEUXNm428KpNHvcfAhKIWbuT3butoeCWRKlDcRv/qn1XcrkjwelvIZAmFVV8FbgVr6jhd2UUzRnZNnBA7CvFvyY7XQB07nDW1dVTYmqShuC3LF754P7ATHzyoKh3LrllVQnSk4udk/PVM+D5FIA2Uey7yQ7id2WJDyLEVMmW1mfyUSk0eMpiTCbgi8+4LfZQGDrgFYqjN306IvYGw916xNPPSnOq6EvqKiYOZmTCtZFZpRQbx/xvgR1TfyJlQWVz2O+pOMi8aYjo27186x2Jo89f8MDL15oiOhl1wMgaplGIiyzwS4+x06zrwCfTGLc/0ThGSkW9Gv34JXIp584pcFODMDlqYYjFDus9niPQKVV3yDkKGUyNbzCvnrkkZvF6uv6VM20/IhXD51e1Gd4sCZ9/NklbCa8gW8taWMPEyPbbE+J3XcKH5oEBIgCeaL1fHjkJQzeouGoF7O+Fw+TVCE03jtAy9byEywtpR7VxIMupr3YOjgVMHWFknp2CZ9l98Oq4yxyEe/ED26JkSuYHmpBbupNkKb8evddFyTUOL5sYpAwWq5kmFMDNHRN8l9BH7d771ZKEVRLubQkY5gdqReph0OZd4LrWoWEpq6/89i00ZjGgHdHXz3rcvIpIf9HXUr3cuvTrdEqaat4ABx/roFjzvBbIuNsUGIrqkLRAfod495ib1LK4cvDqMnMq8Gtb3QN0FOgZfnF6xajuSdMpt/EtTdF8UxIiBzUqwxkcu 959ZobhB 5V+BUDpcqkm8v1OEHjTAo/odhjehkz1fRcs9Us7NhT16lRUdeOTAbBTowAenMmhziYfmAPVpL4W+0dPX4q4zSZsdsZ2rUw10of9bEHIdmgPLgE9HOcUAo9nJnWqlxAXET7bJScujCVdy/P1fTRDVkEGiJMBQwWRiR+Yazji6fsGTqA9Zf2gOlSPOgftSU/tBVD19LjxbEp83eojoCWZAjqfzz2GRjVZ+6SsHe0upY6VlwrK6eFt8A03VaGXGuuCF0LXQqV2/N7Ir0pNF+CS3vg33OibsxELnLUFvnHbSJqIBc91qBQIQbm3GZUbLvfmy6vImc5B1cfW58iAD0snORnYMk4dtm0tw8CV0rNRJUXNiLhQI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Rik van Riel Do context switch and TLB flush support for processes that use a global ASID and PCID across all CPUs. At both context switch time and TLB flush time, it needs to be checked whether a task is switching to a global ASID, and, if so, reload the TLB with the new ASID as appropriate. In both code paths, the TLB flush is avoided if a global ASID is used, because the global ASIDs are always kept up to date across CPUs, even when the process is not running on a CPU. [ bp: - Massage - :%s/\/cpu_feature_enabled/cgi ] Signed-off-by: Rik van Riel Signed-off-by: Borislav Petkov (AMD) Link: https://lore.kernel.org/r/20250226030129.530345-9-riel@surriel.com --- arch/x86/include/asm/tlbflush.h | 14 ++++++ arch/x86/mm/tlb.c | 77 ++++++++++++++++++++++++++++++--- 2 files changed, 84 insertions(+), 7 deletions(-) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index f7b374bcdc0b..1f61a39a8776 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -240,6 +240,11 @@ static inline bool is_dyn_asid(u16 asid) return asid < TLB_NR_DYN_ASIDS; } +static inline bool is_global_asid(u16 asid) +{ + return !is_dyn_asid(asid); +} + #ifdef CONFIG_BROADCAST_TLB_FLUSH static inline u16 mm_global_asid(struct mm_struct *mm) { @@ -266,9 +271,18 @@ static inline void mm_assign_global_asid(struct mm_struct *mm, u16 asid) mm->context.asid_transition = true; smp_store_release(&mm->context.global_asid, asid); } + +static inline bool mm_in_asid_transition(struct mm_struct *mm) +{ + if (!cpu_feature_enabled(X86_FEATURE_INVLPGB)) + return false; + + return mm && READ_ONCE(mm->context.asid_transition); +} #else static inline u16 mm_global_asid(struct mm_struct *mm) { return 0; } static inline void mm_assign_global_asid(struct mm_struct *mm, u16 asid) { } +static inline bool mm_in_asid_transition(struct mm_struct *mm) { return false; } #endif /* CONFIG_BROADCAST_TLB_FLUSH */ #ifdef CONFIG_PARAVIRT diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 6c24d967b77d..b5681e6f2333 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -227,6 +227,20 @@ static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, return; } + /* + * TLB consistency for global ASIDs is maintained with hardware assisted + * remote TLB flushing. Global ASIDs are always up to date. + */ + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) { + u16 global_asid = mm_global_asid(next); + + if (global_asid) { + *new_asid = global_asid; + *need_flush = false; + return; + } + } + if (this_cpu_read(cpu_tlbstate.invalidate_other)) clear_asid_other(); @@ -399,6 +413,23 @@ void mm_free_global_asid(struct mm_struct *mm) #endif } +/* + * Is the mm transitioning from a CPU-local ASID to a global ASID? + */ +static bool mm_needs_global_asid(struct mm_struct *mm, u16 asid) +{ + u16 global_asid = mm_global_asid(mm); + + if (!cpu_feature_enabled(X86_FEATURE_INVLPGB)) + return false; + + /* Process is transitioning to a global ASID */ + if (global_asid && asid != global_asid) + return true; + + return false; +} + /* * Given an ASID, flush the corresponding user ASID. We can delay this * until the next time we switch to it. @@ -704,7 +735,8 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, */ if (prev == next) { /* Not actually switching mm's */ - VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs[prev_asid].ctx_id) != + VM_WARN_ON(is_dyn_asid(prev_asid) && + this_cpu_read(cpu_tlbstate.ctxs[prev_asid].ctx_id) != next->context.ctx_id); /* @@ -721,6 +753,20 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, !cpumask_test_cpu(cpu, mm_cpumask(next)))) cpumask_set_cpu(cpu, mm_cpumask(next)); + /* Check if the current mm is transitioning to a global ASID */ + if (mm_needs_global_asid(next, prev_asid)) { + next_tlb_gen = atomic64_read(&next->context.tlb_gen); + choose_new_asid(next, next_tlb_gen, &new_asid, &need_flush); + goto reload_tlb; + } + + /* + * Broadcast TLB invalidation keeps this ASID up to date + * all the time. + */ + if (is_global_asid(prev_asid)) + return; + /* * If the CPU is not in lazy TLB mode, we are just switching * from one thread in a process to another thread in the same @@ -754,6 +800,13 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, */ cond_mitigation(tsk); + /* + * Let nmi_uaccess_okay() and finish_asid_transition() + * know that CR3 is changing. + */ + this_cpu_write(cpu_tlbstate.loaded_mm, LOADED_MM_SWITCHING); + barrier(); + /* * Leave this CPU in prev's mm_cpumask. Atomic writes to * mm_cpumask can be expensive under contention. The CPU @@ -768,14 +821,12 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, next_tlb_gen = atomic64_read(&next->context.tlb_gen); choose_new_asid(next, next_tlb_gen, &new_asid, &need_flush); - - /* Let nmi_uaccess_okay() know that we're changing CR3. */ - this_cpu_write(cpu_tlbstate.loaded_mm, LOADED_MM_SWITCHING); - barrier(); } +reload_tlb: new_lam = mm_lam_cr3_mask(next); if (need_flush) { + VM_WARN_ON_ONCE(is_global_asid(new_asid)); this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id); this_cpu_write(cpu_tlbstate.ctxs[new_asid].tlb_gen, next_tlb_gen); load_new_mm_cr3(next->pgd, new_asid, new_lam, true); @@ -894,7 +945,7 @@ static void flush_tlb_func(void *info) const struct flush_tlb_info *f = info; struct mm_struct *loaded_mm = this_cpu_read(cpu_tlbstate.loaded_mm); u32 loaded_mm_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); - u64 local_tlb_gen = this_cpu_read(cpu_tlbstate.ctxs[loaded_mm_asid].tlb_gen); + u64 local_tlb_gen; bool local = smp_processor_id() == f->initiating_cpu; unsigned long nr_invalidate = 0; u64 mm_tlb_gen; @@ -917,6 +968,16 @@ static void flush_tlb_func(void *info) if (unlikely(loaded_mm == &init_mm)) return; + /* Reload the ASID if transitioning into or out of a global ASID */ + if (mm_needs_global_asid(loaded_mm, loaded_mm_asid)) { + switch_mm_irqs_off(NULL, loaded_mm, NULL); + loaded_mm_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); + } + + /* Broadcast ASIDs are always kept up to date with INVLPGB. */ + if (is_global_asid(loaded_mm_asid)) + return; + VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs[loaded_mm_asid].ctx_id) != loaded_mm->context.ctx_id); @@ -934,6 +995,8 @@ static void flush_tlb_func(void *info) return; } + local_tlb_gen = this_cpu_read(cpu_tlbstate.ctxs[loaded_mm_asid].tlb_gen); + if (unlikely(f->new_tlb_gen != TLB_GENERATION_INVALID && f->new_tlb_gen <= local_tlb_gen)) { /* @@ -1101,7 +1164,7 @@ STATIC_NOPV void native_flush_tlb_multi(const struct cpumask *cpumask, * up on the new contents of what used to be page tables, while * doing a speculative memory access. */ - if (info->freed_tables) + if (info->freed_tables || mm_in_asid_transition(info->mm)) on_each_cpu_mask(cpumask, flush_tlb_func, (void *)info, true); else on_each_cpu_cond_mask(should_flush_tlb, flush_tlb_func, From patchwork Tue Mar 4 13:58:13 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Borislav Petkov X-Patchwork-Id: 14000784 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 049E9C021B8 for ; Tue, 4 Mar 2025 13:58:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8E110280007; Tue, 4 Mar 2025 08:58:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 89020280004; Tue, 4 Mar 2025 08:58:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7348C280007; Tue, 4 Mar 2025 08:58:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4A16A280004 for ; Tue, 4 Mar 2025 08:58:53 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 0B2891A00FB for ; Tue, 4 Mar 2025 13:58:53 +0000 (UTC) X-FDA: 83184024546.30.7A21830 Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf01.hostedemail.com (Postfix) with ESMTP id 6F06440005 for ; Tue, 4 Mar 2025 13:58:51 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=vF23nCnL; spf=pass (imf01.hostedemail.com: domain of bp@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=bp@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741096731; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xgJGz8LK+XXFuSSYnj1ItMez95xgr3OW/atRa/PU/o0=; b=dtyuo5d6Zc2iCaReqQOkZQFL0VVafrScHZxVEYMGHpLBhrBfvjz5nBSz1F6eq8s8ABR0Ai UqfQ03mRnkJFINP5ftXR82VClucIocIEFxPr3/Py0VsfG62sMkl6rAMt283X9CuquYkwHU M6Xfec0L6qR63e7wruU5NZ9uuozXW8M= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741096731; a=rsa-sha256; cv=none; b=AJlLPXH9dtNB95BZe4ggXHqoHVCJOB+WOyA01m8bvOVWTxp4lDdRXV9LPrxUBoxTNMpe1A +5A8uJBFwABxlsLbml8uO3Z7ukO/7eNCDQx/pZNf7a5HAMW13A+92qBEePfEv8QoqOu4t1 5gt/BJEODoIEhEHhPGX78uKODYBUQpI= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=vF23nCnL; spf=pass (imf01.hostedemail.com: domain of bp@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=bp@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id C988AA447F1; Tue, 4 Mar 2025 13:53:19 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5A1BEC4CEE5; Tue, 4 Mar 2025 13:58:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741096730; bh=zGmupePrg6X0HyaMkp7BsDdleTUSKa3KpW1HRLcE7Sc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=vF23nCnLD/EN0hiNA2S0Dqw90+rhjkP7uGC0lVh/1ZHuOtrBMANtrepOA+aV8+MSg xgQ3p8MST+8/lt1MLO2MGozYt0O0K8QxxfiQwlnoblC+mjaG2PxViOVq2Ey1bn1u1W LST5ksnoIkQCTZetTqtr14iX9Ge96BNiyflO2nZlNIdba3YhxfHu6uorNEOYk8Q1ZN VA4+d0KCUxq25Nf042VOC2jEWSMDGciSFvhrCSNt7DNEHCiNNtyerc8nn3+d0/WKE7 np3qfANNc6IBDPhbUPt2PdJtVlJtsvBd4o0khHeqErwRCkbI7idmpiS3cAkqF1e4yW jzYPtG1e7BX6Q== From: Borislav Petkov To: riel@surriel.com Cc: Manali.Shukla@amd.com, akpm@linux-foundation.org, andrew.cooper3@citrix.com, jackmanb@google.com, jannh@google.com, kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, mhklinux@outlook.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, x86@kernel.org, zhengqi.arch@bytedance.com, Borislav Petkov Subject: [PATCH v15 08/11] x86/mm: Add global ASID process exit helpers Date: Tue, 4 Mar 2025 14:58:13 +0100 Message-ID: <20250304135816.12356-9-bp@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250304135816.12356-1-bp@kernel.org> References: <20250304135816.12356-1-bp@kernel.org> MIME-Version: 1.0 X-Stat-Signature: xc513fcs9brzarzhn3x544e8p8o6rbhh X-Rspamd-Queue-Id: 6F06440005 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1741096731-375754 X-HE-Meta: U2FsdGVkX1/4lTX9lukCYGVHpK5evd/UDiL63WZnHD3ffIIJdUNFwk2nwybMk1OPr4KgMSw1uI2Fu3j41SjnXfLL2KmhozT+qxwTf25DyBKVi1Lu26jSXOOAD+LWvIbhovbq/q69nfDooicUj5Tkc1GErH6xIQHr9cbnLQz+24XlhsDKvyAyqctuG9FeY3/P7OcKVIy/PQ92PgITpS2y9UiK3AhvnP2P912tebtuUh2sxCAgdheSQAswdP6itsVMXiTCuiJ0UIIRpH2l6uYFp2SstqxrGyN7mVq0zmdTd3POslH+0Yhp7EI7N+UYqaIyD/5mkjJP3HS7ww+kJPs/LGI9GB+t1V74yVvJefu1KO+5s/wiYti8b4K5m747adzlEPSYP277SxcrXU5yLu6LrLrkL0D1IbYnmXi/WF7T5nURXo0jOydx2zJODENGhkPpLWdF6wH3n/ERK6kCyHkishmp/MJ7EHr95X0/PujuMm6xvIzZ+MrCZoumr1S/wSAHYh4Cf1eX9kPHeGVdTSDe/gTQY5nWGUk7bczp2eqbSZ4ZtDb8loIuf//aspiGh/09AzdUtTTdm0KAfVq8WuAMxW5REdeTVCCxZRRu378NUTi5GdhBq1NrRuPlQfewbNZHUuq5vY7yrpaNpJmmR4NxJbLsnbH1q3kKp6YOdcd66fEyMDh41pWsJi/EUefy52hlFilDxiq7XwgC4lIuff7Z1AZSRh1ElhxprZB+3Gq+34Ey0qAKeHHeefHarK6Jr4ktWSUyit4RBUZZFEAM3rZTpXKw/59PLGVGuTZrh8McRHVXakU0CaW2+OGXDW0I6vBQIQ1PGUJvWlsjJL0lQff8owOG21UwMEQO9xqDuaiPW02CAFYPrehX7IwviHP5p973WzcoZ9HfD97Rik2BrFtcT+yGuj8UITOZPLx41lxGUulmniTcv50voRm9IcmcgWtm/qapt73YghEqmM/Mhkw tshZPfzZ RflkdMBwOCDp2/apSo8tRylFSxF1uTPEQWAWrQWKkx9+6K/PznNieVkmVdxvKYHl9RKX+H+Z3TWBqjFw1W6P2UdtbEXuCmU+WiSnbFPHdwds+wcAcz8JOgQxKuDNiIy2Oi5P8m7ucE0u388cd7+PqL6aNKjbxO4s4vCOV4K1EqrOhFWAt1YTCZUG25Pm+5V7gPY6HR1mGjE1ZbwK6+NNOUdf8ek21dq7xPFvEBC1U+G+EblcwlfJGwMGyEJYhs08ITJ3vgvMv6zCJ0nVp/jv6j2o+eWFKe6KGMI04AcqGupIFXNEEJueOO2eylJYKf+l2h4L3RhWFfg8QWU1Um3CXw8fyTt4O1ItV9nPCTEAGrhEm2B0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Rik van Riel A global ASID is allocated for the lifetime of a process. Free the global ASID at process exit time. [ bp: Massage, create helpers, hide details inside them. ] Signed-off-by: Rik van Riel Signed-off-by: Borislav Petkov (AMD) Link: https://lore.kernel.org/r/20250226030129.530345-10-riel@surriel.com --- arch/x86/include/asm/mmu_context.h | 8 +++++++- arch/x86/include/asm/tlbflush.h | 9 +++++++++ 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index a2c70e495b1b..2398058b6e83 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -2,7 +2,6 @@ #ifndef _ASM_X86_MMU_CONTEXT_H #define _ASM_X86_MMU_CONTEXT_H -#include #include #include #include @@ -13,6 +12,7 @@ #include #include #include +#include extern atomic64_t last_mm_ctx_id; @@ -139,6 +139,9 @@ static inline void mm_reset_untag_mask(struct mm_struct *mm) #define enter_lazy_tlb enter_lazy_tlb extern void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk); +#define mm_init_global_asid mm_init_global_asid +extern void mm_init_global_asid(struct mm_struct *mm); + extern void mm_free_global_asid(struct mm_struct *mm); /* @@ -163,6 +166,8 @@ static inline int init_new_context(struct task_struct *tsk, mm->context.execute_only_pkey = -1; } #endif + + mm_init_global_asid(mm); mm_reset_untag_mask(mm); init_new_context_ldt(mm); return 0; @@ -172,6 +177,7 @@ static inline int init_new_context(struct task_struct *tsk, static inline void destroy_context(struct mm_struct *mm) { destroy_context_ldt(mm); + mm_free_global_asid(mm); } extern void switch_mm(struct mm_struct *prev, struct mm_struct *next, diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 1f61a39a8776..e6c3be06dd21 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -261,6 +261,14 @@ static inline u16 mm_global_asid(struct mm_struct *mm) return asid; } +static inline void mm_init_global_asid(struct mm_struct *mm) +{ + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) { + mm->context.global_asid = 0; + mm->context.asid_transition = false; + } +} + static inline void mm_assign_global_asid(struct mm_struct *mm, u16 asid) { /* @@ -281,6 +289,7 @@ static inline bool mm_in_asid_transition(struct mm_struct *mm) } #else static inline u16 mm_global_asid(struct mm_struct *mm) { return 0; } +static inline void mm_init_global_asid(struct mm_struct *mm) { } static inline void mm_assign_global_asid(struct mm_struct *mm, u16 asid) { } static inline bool mm_in_asid_transition(struct mm_struct *mm) { return false; } #endif /* CONFIG_BROADCAST_TLB_FLUSH */ From patchwork Tue Mar 4 13:58:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Borislav Petkov X-Patchwork-Id: 14000785 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49280C021B8 for ; Tue, 4 Mar 2025 13:58:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C64C1280008; Tue, 4 Mar 2025 08:58:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C151C280004; Tue, 4 Mar 2025 08:58:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A8CAC280008; Tue, 4 Mar 2025 08:58:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 86CDF280004 for ; Tue, 4 Mar 2025 08:58:56 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4B713120112 for ; Tue, 4 Mar 2025 13:58:56 +0000 (UTC) X-FDA: 83184024672.13.378501E Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf09.hostedemail.com (Postfix) with ESMTP id A2263140005 for ; Tue, 4 Mar 2025 13:58:54 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=S8+HPt3i; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf09.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741096734; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1HN5gYQTXAR2bWO25eG78FcVaGvIZPTU3+5WZ5eDXLI=; b=iqBhC1E5+KiWpqxTwWltTmtC/WctJsJMbxnqJyUZcJ41KMmI+ArSD25mvzfWPLX4BQY5Q0 pun7S8LM9Btam9SaOsij2aqPgcNuun8sLw/cDeqifnV26ZnspRUFAWWmmHFf09ufQ607Bg eCT+jkr3HquHVEn4ZT8F5Fo9rr9E8II= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=S8+HPt3i; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf09.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741096734; a=rsa-sha256; cv=none; b=cGsnpIBa4US7D9kravybYsdEr/eeEuuGmCpyf9MjeigGIUHjVjHrhSN4ucNtF3eAr5kIYf G/7oMROMaIj2vCZpYd2XFfSAtCk3wca0VIxNRgX3tNY9KkGIk6jZa9gaGM8PVAW7RF3Y8+ O/+BcrE/0HINbm0is7USuzDcJJsT8f8= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 1DA915C5D5E; Tue, 4 Mar 2025 13:56:37 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C0033C4CEEC; Tue, 4 Mar 2025 13:58:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741096733; bh=z2DELLe0Mw9azg4Y88YeuBgadHUuT0vo+TmJ6yIo+tY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=S8+HPt3ijNm2idGQ/KNL7I1fl9hwX2WAjFZuhtSua12qfxgd234GwBSdBWD6AyXtg ObChu36YdR+j1y4KCTaF5rn9itEJV5UDlYJnSmmgLMmxDIIi2uHpGyL02ePbMsC3YT QmSjTvacRbODazBwh56pbdrU4Eni0Fuzq8mGQiVhKYg7xRVPMsT93Jf/mEq1UzfdGF M8DFz00+8x4Ie8wJJt3bXAhktL41fswds+SatUToUA/fDxcVilxw5cUWT2hGUArq1D aejw6+SS10aNx0LubbfUGq7u/IuNXsx1ukEfyvbgA4VzS8QGAkqkQoh3RBABvT3QiT 0GC5k8AvFNZWQ== From: Borislav Petkov To: riel@surriel.com Cc: Manali.Shukla@amd.com, akpm@linux-foundation.org, andrew.cooper3@citrix.com, jackmanb@google.com, jannh@google.com, kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, mhklinux@outlook.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, x86@kernel.org, zhengqi.arch@bytedance.com, Borislav Petkov Subject: [PATCH v15 09/11] x86/mm: Enable broadcast TLB invalidation for multi-threaded processes Date: Tue, 4 Mar 2025 14:58:14 +0100 Message-ID: <20250304135816.12356-10-bp@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250304135816.12356-1-bp@kernel.org> References: <20250304135816.12356-1-bp@kernel.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: A2263140005 X-Rspam-User: X-Stat-Signature: xnmqh7fxtt6wihdzsurfaprauddonuxw X-HE-Tag: 1741096734-358964 X-HE-Meta: U2FsdGVkX1/EtLQGmpP8NTaUnejYiyIp86jBwSOXxBnC5w2SrWkZbSQUqhC8YG5SvPia0cwFWc9Xxt5cBICDuEbHBeoUiHCnlmd7BdQ0WBonU3gNBFBAnpOPT7z2qyUP8R+7L1qM+3tOKCutnfjNoZpoRFJcx5++54AzsScoc2KaRCMw0QsHFGPSBGy0zHkE23iJF6JFN6LMN3Qjatyd6VJ8Enl77wbfc+LlGnmiTdKfgV+oupf2BJOuX0wkMyCVdLvcxcM+BvEEXiRdQ/1C7lv6lf9XUHDy6fEsabE7PzCOy6ujsBSiIaPkCz7oy5HTZcqyIjc80KGv1M50O8d9eFFiJZ+KKqMMKlsdDjQT7H/ueWkrAqqJX4oWoGLYlIpUDOszIJRBUSZe0XJLk9cZVhs9nqR/r5iawD2FQdm17x7a73mQ0h1r4djP8tpF3UytKfbH8E9m/4WfjsnsKLBlu0GdZIMFFp5yUm2nvLeI95sWsl9O5zWb0GSRjrUwk4Ptw1K4Rng+LmuWDXgqgyJZLNEs6NcOi5AasshW9y5kG1Ej/SJ18SlqfE47Hzl/9TpNkGxPEBtk3BVaHt6EnIuvzlK5Uai0q0DrJ5nKqXtQldJbqUqNX7vPc9yDtwM2UBWkKL+tu4cNM9LYENZmqa38oRXdtmOGft4TQp/TvJyUjc2D6dIKdYn60AUq6GggdV/fiZhgNS61CLWwkQtsVxo58GfAEgh28rDisvz73A3lKfLzE4La/dOHRJevcnQPk2JvrJPQe9GSSHEXKNXm9j/MhBt05UbN6phJafDCEjrCKEZzr1XHwtx/tgTnYVzjrTMtV1O5Yix97hQqeBq7ujhgTDJhtZnW09j4tk6PPTo1y3Uw1hmwv9KlIMGuLuYr4sRDSs7fbX+XsjN7Z08wdXDiylHWI7Iy1T9eZnJXWEQkWvt3HGeqM+o03BIrQ4FN/iTJmnnMUUgQX7WuEK3VQs7 W8ZZHwB9 ETcmwiTyg5H8wsLLga2K+YSgYfL33+eJvuKZFFVDIJuxfmoxQdAg8CUqWF8GhvkDoqGsR+f+w7cJiiRM5vlsq1XWst0SlPCbwz9HDd2DWSN0j6AzdSdxfsj2hIJmDjSNfKftG5pDS5gk4zUVx6R1XAI1caBhEai+td7TtMwjrm5L9+9BT5mC+O/lOdQpMIKeuGEo7uF9v443z7NguovyufvcqzrkjW/9q0mj16OAz5U4tLNjRCd5VOTzjZTmMP8zt8rrX1chQOYhExlN0Q3wAkZDCeGgGh2aX/BTLI4uQyu5vsAV+KvDKELuyk2yjswK4Yb4N9FMOxRC/ql2RJqIImZSFszaXBnHtVdTcZsHfVjCfsxqmFKVq7t9iWXnV5C+0zm8nfU6+BEqxZp+YrsxcO16MTO4E22dYtebCG4M6QqOql7p14EbnTcuuSkBEuJfVx115mjD7oquQIsaT48r2D6cpqA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Rik van Riel There is not enough room in the 12-bit ASID address space to hand out broadcast ASIDs to every process. Only hand out broadcast ASIDs to processes when they are observed to be simultaneously running on 4 or more CPUs. This also allows single threaded process to continue using the cheaper, local TLB invalidation instructions like INVLPGB. Due to the structure of flush_tlb_mm_range(), the INVLPGB flushing is done in a generically named broadcast_tlb_flush() function which can later also be used for Intel RAR. Combined with the removal of unnecessary lru_add_drain calls() (see https://lore.kernel.org/r/20241219153253.3da9e8aa@fangorn) this results in a nice performance boost for the will-it-scale tlb_flush2_threads test on an AMD Milan system with 36 cores: - vanilla kernel: 527k loops/second - lru_add_drain removal: 731k loops/second - only INVLPGB: 527k loops/second - lru_add_drain + INVLPGB: 1157k loops/second Profiling with only the INVLPGB changes showed while TLB invalidation went down from 40% of the total CPU time to only around 4% of CPU time, the contention simply moved to the LRU lock. Fixing both at the same time about doubles the number of iterations per second from this case. Comparing will-it-scale tlb_flush2_threads with several different numbers of threads on a 72 CPU AMD Milan shows similar results. The number represents the total number of loops per second across all the threads: threads tip INVLPGB 1 315k 304k 2 423k 424k 4 644k 1032k 8 652k 1267k 16 737k 1368k 32 759k 1199k 64 636k 1094k 72 609k 993k 1 and 2 thread performance is similar with and without INVLPGB, because INVLPGB is only used on processes using 4 or more CPUs simultaneously. The number is the median across 5 runs. Some numbers closer to real world performance can be found at Phoronix, thanks to Michael: https://www.phoronix.com/news/AMD-INVLPGB-Linux-Benefits [ bp: - Massage - :%s/\/cpu_feature_enabled/cgi - :%s/\/mm_clear_asid_transition/cgi - Fold in a 0day bot fix: https://lore.kernel.org/oe-kbuild-all/202503040000.GtiWUsBm-lkp@intel.com ] Signed-off-by: Rik van Riel Signed-off-by: Borislav Petkov (AMD) Reviewed-by: Nadav Amit Link: https://lore.kernel.org/r/20250226030129.530345-11-riel@surriel.com WIP Signed-off-by: Borislav Petkov (AMD) --- arch/x86/include/asm/tlbflush.h | 6 ++ arch/x86/mm/tlb.c | 104 +++++++++++++++++++++++++++++++- 2 files changed, 109 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index e6c3be06dd21..7cad283d502d 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -280,6 +280,11 @@ static inline void mm_assign_global_asid(struct mm_struct *mm, u16 asid) smp_store_release(&mm->context.global_asid, asid); } +static inline void mm_clear_asid_transition(struct mm_struct *mm) +{ + WRITE_ONCE(mm->context.asid_transition, false); +} + static inline bool mm_in_asid_transition(struct mm_struct *mm) { if (!cpu_feature_enabled(X86_FEATURE_INVLPGB)) @@ -291,6 +296,7 @@ static inline bool mm_in_asid_transition(struct mm_struct *mm) static inline u16 mm_global_asid(struct mm_struct *mm) { return 0; } static inline void mm_init_global_asid(struct mm_struct *mm) { } static inline void mm_assign_global_asid(struct mm_struct *mm, u16 asid) { } +static inline void mm_clear_asid_transition(struct mm_struct *mm) { } static inline bool mm_in_asid_transition(struct mm_struct *mm) { return false; } #endif /* CONFIG_BROADCAST_TLB_FLUSH */ diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index b5681e6f2333..0efd99053c09 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -430,6 +430,105 @@ static bool mm_needs_global_asid(struct mm_struct *mm, u16 asid) return false; } +/* + * x86 has 4k ASIDs (2k when compiled with KPTI), but the largest x86 + * systems have over 8k CPUs. Because of this potential ASID shortage, + * global ASIDs are handed out to processes that have frequent TLB + * flushes and are active on 4 or more CPUs simultaneously. + */ +static void consider_global_asid(struct mm_struct *mm) +{ + if (!cpu_feature_enabled(X86_FEATURE_INVLPGB)) + return; + + /* Check every once in a while. */ + if ((current->pid & 0x1f) != (jiffies & 0x1f)) + return; + + /* + * Assign a global ASID if the process is active on + * 4 or more CPUs simultaneously. + */ + if (mm_active_cpus_exceeds(mm, 3)) + use_global_asid(mm); +} + +static void finish_asid_transition(struct flush_tlb_info *info) +{ + struct mm_struct *mm = info->mm; + int bc_asid = mm_global_asid(mm); + int cpu; + + if (!mm_in_asid_transition(mm)) + return; + + for_each_cpu(cpu, mm_cpumask(mm)) { + /* + * The remote CPU is context switching. Wait for that to + * finish, to catch the unlikely case of it switching to + * the target mm with an out of date ASID. + */ + while (READ_ONCE(per_cpu(cpu_tlbstate.loaded_mm, cpu)) == LOADED_MM_SWITCHING) + cpu_relax(); + + if (READ_ONCE(per_cpu(cpu_tlbstate.loaded_mm, cpu)) != mm) + continue; + + /* + * If at least one CPU is not using the global ASID yet, + * send a TLB flush IPI. The IPI should cause stragglers + * to transition soon. + * + * This can race with the CPU switching to another task; + * that results in a (harmless) extra IPI. + */ + if (READ_ONCE(per_cpu(cpu_tlbstate.loaded_mm_asid, cpu)) != bc_asid) { + flush_tlb_multi(mm_cpumask(info->mm), info); + return; + } + } + + /* All the CPUs running this process are using the global ASID. */ + mm_clear_asid_transition(mm); +} + +static void broadcast_tlb_flush(struct flush_tlb_info *info) +{ + bool pmd = info->stride_shift == PMD_SHIFT; + unsigned long asid = mm_global_asid(info->mm); + unsigned long addr = info->start; + + /* + * TLB flushes with INVLPGB are kicked off asynchronously. + * The inc_mm_tlb_gen() guarantees page table updates are done + * before these TLB flushes happen. + */ + if (info->end == TLB_FLUSH_ALL) { + invlpgb_flush_single_pcid_nosync(kern_pcid(asid)); + /* Do any CPUs supporting INVLPGB need PTI? */ + if (cpu_feature_enabled(X86_FEATURE_PTI)) + invlpgb_flush_single_pcid_nosync(user_pcid(asid)); + } else do { + unsigned long nr = 1; + + if (info->stride_shift <= PMD_SHIFT) { + nr = (info->end - addr) >> info->stride_shift; + nr = clamp_val(nr, 1, invlpgb_count_max); + } + + invlpgb_flush_user_nr_nosync(kern_pcid(asid), addr, nr, pmd); + if (cpu_feature_enabled(X86_FEATURE_PTI)) + invlpgb_flush_user_nr_nosync(user_pcid(asid), addr, nr, pmd); + + addr += nr << info->stride_shift; + } while (addr < info->end); + + finish_asid_transition(info); + + /* Wait for the INVLPGBs kicked off above to finish. */ + __tlbsync(); +} + /* * Given an ASID, flush the corresponding user ASID. We can delay this * until the next time we switch to it. @@ -1260,9 +1359,12 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, * a local TLB flush is needed. Optimize this use-case by calling * flush_tlb_func_local() directly in this case. */ - if (cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids) { + if (mm_global_asid(mm)) { + broadcast_tlb_flush(info); + } else if (cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids) { info->trim_cpumask = should_trim_cpumask(mm); flush_tlb_multi(mm_cpumask(mm), info); + consider_global_asid(mm); } else if (mm == this_cpu_read(cpu_tlbstate.loaded_mm)) { lockdep_assert_irqs_enabled(); local_irq_disable(); From patchwork Tue Mar 4 13:58:15 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Borislav Petkov X-Patchwork-Id: 14000786 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12EC2C282C6 for ; Tue, 4 Mar 2025 13:59:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 980FF280009; Tue, 4 Mar 2025 08:59:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 92CAA280004; Tue, 4 Mar 2025 08:59:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7CF6D280009; Tue, 4 Mar 2025 08:59:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 5E0C8280004 for ; Tue, 4 Mar 2025 08:59:00 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id DC352C1D9D for ; Tue, 4 Mar 2025 13:58:59 +0000 (UTC) X-FDA: 83184024798.03.3092F2D Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf07.hostedemail.com (Postfix) with ESMTP id 3184840015 for ; Tue, 4 Mar 2025 13:58:58 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="lwH/QIJc"; spf=pass (imf07.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741096738; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PHunenwCcu8fCU1fxEi1+mySKBOkJ4IBXxnE72H1GQM=; b=SmQv6onguOx7F9+m4GcYV5fV6EWm5tlROrb+6SMFD++LDNy5snwCd482lBWzf1QGonfczV lazNosrZEmmx/4XHPDlC0tZvSSGg8XnToGo7cZhK8eKfWQlesN7vOVQ7hy69NAs9EVdnPs Y2C8C+VScIk9lZkJnAlp539jSYpqid0= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="lwH/QIJc"; spf=pass (imf07.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741096738; a=rsa-sha256; cv=none; b=rnnXZWH4OCziOWieKD07dzldIkNQmVYK2u2ASpcPW1+Qc11gfy1/EEoRI3snWu0ErVUX8Y QtvY54GHmVRB25Yz9H+pTomO9rakNkYdMg5J5vSgPJxqI4CY9Dd3j6q8pJ22hpjUWv2qKX hI9zzrSc4husX5KT1/9dzSNpbMFBLF4= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id A3D0A5C5B8F; Tue, 4 Mar 2025 13:56:40 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 30815C4CEE5; Tue, 4 Mar 2025 13:58:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741096737; bh=stV7xxxBalUUJGGfdqVe/j6rs+E8JtA24+pOHC7n1qU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lwH/QIJcNEZuv14gJ1GlBIj0C24z2Il17tn5+VNrDVbF78IHCNS8RRmVhVrnoHm1u MvuOAMxjYV8ex3ydzFufLf4b883+FR9Ppd0ORuMOjG0SyOmLlv1BIqPTGlZc2+h+Xo fzoPx2i5uqIDcGTcSfRiJ7K54l1/wxAED8x2dSZ0UqDXwyvkKFI5n+tPM7hHaAy/0g bZwc3GyP0h45riZXdUset9sogaMUvybd11rtIYjdNdmh0Zz7XuSL36e8n0hePF8CIi 02APEkK9B8DT91ndAFC3ou06Mooc3lFi11klMfxsAqGnuVa7NoAYlGtuhywJ537vwK kL/63Oj2YACEw== From: Borislav Petkov To: riel@surriel.com Cc: Manali.Shukla@amd.com, akpm@linux-foundation.org, andrew.cooper3@citrix.com, jackmanb@google.com, jannh@google.com, kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, mhklinux@outlook.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, x86@kernel.org, zhengqi.arch@bytedance.com, Borislav Petkov Subject: [PATCH v15 10/11] x86/mm: Do targeted broadcast flushing from tlbbatch code Date: Tue, 4 Mar 2025 14:58:15 +0100 Message-ID: <20250304135816.12356-11-bp@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250304135816.12356-1-bp@kernel.org> References: <20250304135816.12356-1-bp@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: w3st8sk3swofkf5qr5merkdr87xj81hj X-Rspamd-Queue-Id: 3184840015 X-Rspamd-Server: rspam07 X-HE-Tag: 1741096738-87119 X-HE-Meta: U2FsdGVkX19QGHUu4uPFwm9wMf3UM/+bR85FJxTMhFHsjB50kTZu1FEGwXoA0G9Ks3Q5pgFvgTZPCgevUotNuBvPF+ATyvCSUK8GauZpUkTUCHl+2CR8hqa8NWpJRc9gg6UtezuhwYD2p73LIWq1rVQbl+tB+TP6oiRJ1NAeP9ZCdxZlQBrYPGjLGBXAWkYsr67ynaY1y8f+QaFLw0wWRVDD7wyhh/3O1XYT1tlcqDpxa9H2upCHgXI2hO6OKyBJtLC28O3qRiIsENkrYRZrW/gJQjhfhcDfotsa82ki+v0pLWTtu5280YTVYR/OTE5dC36MaKs9U+43yzxl8+5CbbYrI7eMuPSHtJfZBFRg0KP7k3xzqMHiYAVnIURJcQ9Be/scaiuDsRFMqciLv/crPB5nRAJdyA4iNAlgoecG0LPfh33HTm9XRZxMWTKev5p7soBF0hNf55/lFSIVYrj8HMnKl8Y3e6wp/lhKCpFj7LvyqQ75vjrIW+h2KUhzWcDP/vPEUXTZvDNlyiZvNgwR+Mt+FXOv+4n+bXrpDa2ktI0D5zYwpI1CPHVYi/fDdzNpXdKrTLS/Z7X4KIubBXVIIH0izcbllydCMkelBYpI4nWRYdpiJhaDQnqikA3lYURWQgujZrGNtUPRWVUYHyn5g+2LvpfbsnvJm4y3NEJreQRk/xHwDeZ+yTgoXpOvMdgEvpckSYTCnQ+ayzmnf0Zm2hMzYkyvDqy4ttHRFENuJkU5Vl/mlnBprMxCC1ePg+SSpeEerFfq/Pj4vRUPpx5nrFAe+hedzclXoEY9LCgQnux/THRUFmrKxMm7p4VjsErkvLeb5hgkhX9bH+5yETbJQn/BCbz5zw5X5IR1KxC4VYGisH1VZEG8KmAUE5G4R2L37CEj8ij/+usb8zuZtF8jJc/36XdddOGfvA1LmJJTc4gxXo/opZuO8kVh2A4snhuKIY0nYAUH3kQiYEvcsVM O0Caz831 tjnqFnJPer0ztHDeokbJLaCfNk63otyk9Wlc+/5RPMPzsjtTRVqJm/9klC/XYsr2ANuiMDncYHk3EbAskbYBSz8BHdZS3qdllLxAUAFIhJWbpDBNtqeDiRTbWCW0M2VCImzj6otItM4Efkl+mbivQmAJIBrnn5FcR7ISAWEHn94sIigDqR/qI2B4Smm8tiCEveMf363vBqZPNE8VBQ1FjSOH7JYmq1VAWgx5Eictre7tpQ74a3/aLrizaiNMehq/mFWgHaCuyMX7C2j8DBHmTXemdvENjAUS36rBtHNiJt/qQ74Hrut7+CwZ18Ewxp3i8zPf98Vc8piozDuNSHJTvSEIBi6ORD7veUz1ozYy0G7g6tnE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Rik van Riel Instead of doing a system-wide TLB flush from arch_tlbbatch_flush(), queue up asynchronous, targeted flushes from arch_tlbbatch_add_pending(). This also allows to avoid adding the CPUs of processes using broadcast flushing to the batch->cpumask, and will hopefully further reduce TLB flushing from the reclaim and compaction paths. [ bp: - Massage - :%s/\/cpu_feature_enabled/cgi - merge in improvements from dhansen ] Signed-off-by: Rik van Riel Signed-off-by: Borislav Petkov (AMD) Link: https://lore.kernel.org/r/20250226030129.530345-12-riel@surriel.com --- arch/x86/include/asm/tlb.h | 10 ++-- arch/x86/include/asm/tlbflush.h | 27 ++++++---- arch/x86/mm/tlb.c | 88 +++++++++++++++++++++++++++++++-- 3 files changed, 108 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 8ffcae7beb55..e8561a846754 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -108,9 +108,9 @@ static inline void __tlbsync(void) { } /* The implied mode when all bits are clear: */ #define INVLPGB_MODE_ALL_NONGLOBALS 0UL -static inline void invlpgb_flush_user_nr_nosync(unsigned long pcid, - unsigned long addr, - u16 nr, bool stride) +static inline void __invlpgb_flush_user_nr_nosync(unsigned long pcid, + unsigned long addr, + u16 nr, bool stride) { enum addr_stride str = stride ? PMD_STRIDE : PTE_STRIDE; u8 flags = INVLPGB_FLAG_PCID | INVLPGB_FLAG_VA; @@ -119,7 +119,7 @@ static inline void invlpgb_flush_user_nr_nosync(unsigned long pcid, } /* Flush all mappings for a given PCID, not including globals. */ -static inline void invlpgb_flush_single_pcid_nosync(unsigned long pcid) +static inline void __invlpgb_flush_single_pcid_nosync(unsigned long pcid) { __invlpgb(0, pcid, 0, 1, PTE_STRIDE, INVLPGB_FLAG_PCID); } @@ -139,7 +139,7 @@ static inline void invlpgb_flush_all(void) } /* Flush addr, including globals, for all PCIDs. */ -static inline void invlpgb_flush_addr_nosync(unsigned long addr, u16 nr) +static inline void __invlpgb_flush_addr_nosync(unsigned long addr, u16 nr) { __invlpgb(0, 0, addr, nr, PTE_STRIDE, INVLPGB_FLAG_INCLUDE_GLOBAL); } diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 7cad283d502d..214d912ac148 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -105,6 +105,9 @@ struct tlb_state { * need to be invalidated. */ bool invalidate_other; +#ifdef CONFIG_BROADCAST_TLB_FLUSH + bool need_tlbsync; +#endif #ifdef CONFIG_ADDRESS_MASKING /* @@ -292,12 +295,24 @@ static inline bool mm_in_asid_transition(struct mm_struct *mm) return mm && READ_ONCE(mm->context.asid_transition); } + +static inline bool cpu_need_tlbsync(void) +{ + return this_cpu_read(cpu_tlbstate.need_tlbsync); +} + +static inline void cpu_set_tlbsync(bool state) +{ + this_cpu_write(cpu_tlbstate.need_tlbsync, state); +} #else static inline u16 mm_global_asid(struct mm_struct *mm) { return 0; } static inline void mm_init_global_asid(struct mm_struct *mm) { } static inline void mm_assign_global_asid(struct mm_struct *mm, u16 asid) { } static inline void mm_clear_asid_transition(struct mm_struct *mm) { } static inline bool mm_in_asid_transition(struct mm_struct *mm) { return false; } +static inline bool cpu_need_tlbsync(void) { return false; } +static inline void cpu_set_tlbsync(bool state) { } #endif /* CONFIG_BROADCAST_TLB_FLUSH */ #ifdef CONFIG_PARAVIRT @@ -347,21 +362,15 @@ static inline u64 inc_mm_tlb_gen(struct mm_struct *mm) return atomic64_inc_return(&mm->context.tlb_gen); } -static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, - struct mm_struct *mm, - unsigned long uaddr) -{ - inc_mm_tlb_gen(mm); - cpumask_or(&batch->cpumask, &batch->cpumask, mm_cpumask(mm)); - mmu_notifier_arch_invalidate_secondary_tlbs(mm, 0, -1UL); -} - static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm) { flush_tlb_mm(mm); } extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); +extern void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, + struct mm_struct *mm, + unsigned long uaddr); static inline bool pte_flags_need_flush(unsigned long oldflags, unsigned long newflags, diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 0efd99053c09..61065975c139 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -492,6 +492,37 @@ static void finish_asid_transition(struct flush_tlb_info *info) mm_clear_asid_transition(mm); } +static inline void tlbsync(void) +{ + if (cpu_need_tlbsync()) { + __tlbsync(); + cpu_set_tlbsync(false); + } +} + +static inline void invlpgb_flush_user_nr_nosync(unsigned long pcid, + unsigned long addr, + u16 nr, bool pmd_stride) +{ + __invlpgb_flush_user_nr_nosync(pcid, addr, nr, pmd_stride); + if (!cpu_need_tlbsync()) + cpu_set_tlbsync(true); +} + +static inline void invlpgb_flush_single_pcid_nosync(unsigned long pcid) +{ + __invlpgb_flush_single_pcid_nosync(pcid); + if (!cpu_need_tlbsync()) + cpu_set_tlbsync(true); +} + +static inline void invlpgb_flush_addr_nosync(unsigned long addr, u16 nr) +{ + __invlpgb_flush_addr_nosync(addr, nr); + if (!cpu_need_tlbsync()) + cpu_set_tlbsync(true); +} + static void broadcast_tlb_flush(struct flush_tlb_info *info) { bool pmd = info->stride_shift == PMD_SHIFT; @@ -790,6 +821,8 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, if (IS_ENABLED(CONFIG_PROVE_LOCKING)) WARN_ON_ONCE(!irqs_disabled()); + tlbsync(); + /* * Verify that CR3 is what we think it is. This will catch * hypothetical buggy code that directly switches to swapper_pg_dir @@ -966,6 +999,8 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, */ void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk) { + tlbsync(); + if (this_cpu_read(cpu_tlbstate.loaded_mm) == &init_mm) return; @@ -1633,9 +1668,7 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) * a local TLB flush is needed. Optimize this use-case by calling * flush_tlb_func_local() directly in this case. */ - if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) { - invlpgb_flush_all_nonglobals(); - } else if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { + if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { flush_tlb_multi(&batch->cpumask, info); } else if (cpumask_test_cpu(cpu, &batch->cpumask)) { lockdep_assert_irqs_enabled(); @@ -1644,12 +1677,61 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) local_irq_enable(); } + /* + * Wait for outstanding INVLPGB flushes. batch->cpumask will + * be empty when the batch was handled completely by INVLPGB. + * Note that mm_in_asid_transition() mm's may use INVLPGB and + * the flush_tlb_multi() IPIs at the same time. + */ + tlbsync(); + cpumask_clear(&batch->cpumask); put_flush_tlb_info(); put_cpu(); } +void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, + struct mm_struct *mm, unsigned long uaddr) +{ + u16 global_asid = mm_global_asid(mm); + + if (global_asid) { + /* + * Global ASIDs can be flushed with INVLPGB. Flush + * now instead of batching them for later. A later + * tlbsync() is required to ensure these completed. + */ + invlpgb_flush_user_nr_nosync(kern_pcid(global_asid), uaddr, 1, false); + /* Do any CPUs supporting INVLPGB need PTI? */ + if (cpu_feature_enabled(X86_FEATURE_PTI)) + invlpgb_flush_user_nr_nosync(user_pcid(global_asid), uaddr, 1, false); + + /* + * Some CPUs might still be using a local ASID for this + * process, and require IPIs, while others are using the + * global ASID. + * + * In this corner case, both broadcast TLB invalidation + * and IPIs need to be sent. The IPIs will help + * stragglers transition to the broadcast ASID. + */ + if (mm_in_asid_transition(mm)) + global_asid = 0; + } + + if (!global_asid) { + /* + * Mark the mm and the CPU so that + * the TLB gets flushed later. + */ + inc_mm_tlb_gen(mm); + cpumask_or(&batch->cpumask, &batch->cpumask, mm_cpumask(mm)); + } + + mmu_notifier_arch_invalidate_secondary_tlbs(mm, 0, -1UL); +} + /* * Blindly accessing user memory from NMI context can be dangerous * if we're in the middle of switching the current user task or From patchwork Tue Mar 4 13:58:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Borislav Petkov X-Patchwork-Id: 14000787 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02455C021B8 for ; Tue, 4 Mar 2025 13:59:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C17A828000A; Tue, 4 Mar 2025 08:59:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BC4EF280004; Tue, 4 Mar 2025 08:59:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A3EE128000A; Tue, 4 Mar 2025 08:59:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 85CDC280004 for ; Tue, 4 Mar 2025 08:59:03 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 3CC7B141CD9 for ; Tue, 4 Mar 2025 13:59:03 +0000 (UTC) X-FDA: 83184024966.02.9B5B7EE Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf09.hostedemail.com (Postfix) with ESMTP id 9895A140007 for ; Tue, 4 Mar 2025 13:59:01 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=li36mmNf; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf09.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741096741; a=rsa-sha256; cv=none; b=XjKKfLwyfcos0/SDsFimQZodlsft0rqJywJ7R9Nfwcm2/CcYzVwPmbgK22mhyknPZg6LHH dguuHzyqUoYErMifxwRrzNPTtqTPcplkQ/yg5BM7UBtqIz1ISDx0gzpil0kqXghkRVgV4E 0/dyj97KeMdq38XiMrAIMgOF1mGau/Y= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=li36mmNf; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf09.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741096741; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hOF/oNewM9EWYwLL2Lq6iBUPtoytGFygnrlu4K7b4n0=; b=uo+G+8Ub7JyWhx1PWxcKV8pRl9HJ0MCxDUVYiCyIgbq31M3/3lyeHNBsd8Gh9mwOBpFBDc 2KiAC7T5ZYsYhB1g1tezG64K+dQKl8e4vacq3w+gD0ScveAnbXsxqx9O09EVFlMnOjB/Vj ffuXqRZiW3Ahja/nzaX4dAhLEGIwA0E= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 1A7F35C5D33; Tue, 4 Mar 2025 13:56:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 95CCAC4CEE5; Tue, 4 Mar 2025 13:58:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741096740; bh=9c2Sjy45ucPykbiLx09DjVwbvVa+jkeWfLgJg4HPpMw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=li36mmNfukBpxBkysoR3bxmyUpKSu6bxn6vCrX/WOwfkw6/ZMkyU2UBjUbzp9EIdO 0RKSgZ1sUYMKzoBN7WAh8ToBmY7S0k/5vfOnWOlJg6eXNl76M3spdD/h6G0WzMK4EL iuWOdPCneAD/eG7wnFtqJj8/DaY0dRcjc9NnfsT73A0Fj/t5WLW8blOsynLGlVKQNs q3aQELgkEn3cO2RBmwos/wAVpTSMjZhsQdVtP6n+25lF/6MA865p14lSRUdhus28BL DJcPDIszXJBAyDtNlg8Gd1H7/DkNUGyDnOVXtwiU1HhwLy7S6vbh8eMthWNqmqePmC uSSJ1Ogw0mwDw== From: Borislav Petkov To: riel@surriel.com Cc: Manali.Shukla@amd.com, akpm@linux-foundation.org, andrew.cooper3@citrix.com, jackmanb@google.com, jannh@google.com, kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, mhklinux@outlook.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, x86@kernel.org, zhengqi.arch@bytedance.com, Borislav Petkov Subject: [PATCH v15 11/11] x86/mm: Enable AMD translation cache extensions Date: Tue, 4 Mar 2025 14:58:16 +0100 Message-ID: <20250304135816.12356-12-bp@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250304135816.12356-1-bp@kernel.org> References: <20250304135816.12356-1-bp@kernel.org> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 9895A140007 X-Stat-Signature: msbhbjfopicgsr7dg3kt5f4966uxxh3p X-Rspam-User: X-HE-Tag: 1741096741-534384 X-HE-Meta: U2FsdGVkX1/gNxbvYojBopIHaWow/e4FDi+1HsCQtXuCPw1+3z+VpxajCNZtZiZZoTHGEJpawVM6xxP3WgcwG3mcT8zrB0g/3z9d46GWd9ZgI/9HKH339Ndps4EqYJClj4eFZno5OCQ9BOjh12uQAQ/SU6MHBgZj5BuJbrD0zOl6MxxZDIumF3A/Bl+MNMPu+pSzY5tKC/nr9ojjHhxhTg3Q8BFoSvFyQhsw/EKQ/OGBGALyhL0f5ewhwG4AHOqf3myqAWQV/MhzLpWX+hOXnmQCwJ9pHMmd06uTrEcz+kKzbvaFlvMdJO6XOtodprhVgbAd9KdMJ/obhp9L8QMXGaDQkoqyEVSjmbs2LHgFy5GDWeRTobh5SeIiynR9wqIzyZV6cp0KHb0uJZnxIs15q0/gZgjEBBimga1xrX2aoEJPr6P3e0z1wPkTGj9FUz6sDiDzbwU/MOriFU5/7FxpIunIXsuLXIHsQocO+uTBcHFFRSHN93n49aXvt/xkfRoi8MEzBY/t8nXS4d96lMrAF5pJ97mGd8q+WNdhCeD9/H3eQwTlSbNYvzQJSx3x8gynHyCxWQjE7jtSahiMSo11lY8/ic3+kc9fILsHDFGAAO6HYz5/B+dX7dphjwEK07whclA4yxtqX+KMl8xAAP3raUW492FzuCPagP3mVnfwLXTWYdRx5CmpKRaGuvelAv2EP2iWnSpeNxnGucq4MA9qCO4haAHpX6+wqOkg1yiiBgQegIzN55VTgMpuGh5OaZnGjKGoheOoOtKUs//eJEEUl9hK4CjWyeXuaIeFfwQXlxTCTTQ/Gmqtgw8CQ81Cu1ygs5tuf3mtKfhn4h66l5eUvU18eSCZ90gP+/193Dt4GfLCJBxYRm9PggszwpMd3niTN6SM/bXTzt9F9jhlnKn9PSYKKivTxRAZRq/SHMa+X/NEMrqcHyVTC8WlivBICHHNCJMLCNQhH8rN2VXpqMo w6xUvaPv a3mT2rfTfRVTKJEtHLZtvi1I9Z5ArjAYzMInyUh1bNX9GhvmGe9cFJ2X7U7YviYF61DbqdAnbnV7Jx3sSj2EunHzkaYDu0Vbr8CDLpO5qOjGku86ZzrfS3oXF9Qp1/7gTqFq20a5A2h5JgfRW4d/gs0IB9n5x7KWFl1N1F/RmQGgsK/N6xJvBgqZTKpAN2U9ycaYejh1ZbnrLiwYwJk+vfCR0J5ofgbuTSPgIu/NkGXFzwAgF01Ud9BzIumB9gp/GqIJcjuwufEC5NKkSR471Zrp6Aml0b73rsytsOXGT1MbJ/dXWAsVRMzc1xEh4Eof+KlOKfchmNKHyS785cP7aGmnYCMHNlKUwo2bbkZ/fOHH3RHw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Rik van Riel With AMD TCE (translation cache extensions) only the intermediate mappings that cover the address range zapped by INVLPG / INVLPGB get invalidated, rather than all intermediate mappings getting zapped at every TLB invalidation. This can help reduce the TLB miss rate, by keeping more intermediate mappings in the cache. From the AMD manual: Translation Cache Extension (TCE) Bit. Bit 15, read/write. Setting this bit to 1 changes how the INVLPG, INVLPGB, and INVPCID instructions operate on TLB entries. When this bit is 0, these instructions remove the target PTE from the TLB as well as all upper-level table entries that are cached in the TLB, whether or not they are associated with the target PTE. When this bit is set, these instructions will remove the target PTE and only those upper-level entries that lead to the target PTE in the page table hierarchy, leaving unrelated upper-level entries intact. [ bp: use cpu_has()... I know, it is a mess. ] Signed-off-by: Rik van Riel Signed-off-by: Borislav Petkov (AMD) Link: https://lore.kernel.org/r/20250226030129.530345-13-riel@surriel.com --- arch/x86/include/asm/msr-index.h | 2 ++ arch/x86/kernel/cpu/amd.c | 4 ++++ tools/arch/x86/include/asm/msr-index.h | 2 ++ 3 files changed, 8 insertions(+) diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h index 72765b2fe0d8..1aacd6b68fab 100644 --- a/arch/x86/include/asm/msr-index.h +++ b/arch/x86/include/asm/msr-index.h @@ -25,6 +25,7 @@ #define _EFER_SVME 12 /* Enable virtualization */ #define _EFER_LMSLE 13 /* Long Mode Segment Limit Enable */ #define _EFER_FFXSR 14 /* Enable Fast FXSAVE/FXRSTOR */ +#define _EFER_TCE 15 /* Enable Translation Cache Extensions */ #define _EFER_AUTOIBRS 21 /* Enable Automatic IBRS */ #define EFER_SCE (1<<_EFER_SCE) @@ -34,6 +35,7 @@ #define EFER_SVME (1<<_EFER_SVME) #define EFER_LMSLE (1<<_EFER_LMSLE) #define EFER_FFXSR (1<<_EFER_FFXSR) +#define EFER_TCE (1<<_EFER_TCE) #define EFER_AUTOIBRS (1<<_EFER_AUTOIBRS) /* diff --git a/arch/x86/kernel/cpu/amd.c b/arch/x86/kernel/cpu/amd.c index 7a72ef47a983..705853315c0d 100644 --- a/arch/x86/kernel/cpu/amd.c +++ b/arch/x86/kernel/cpu/amd.c @@ -1075,6 +1075,10 @@ static void init_amd(struct cpuinfo_x86 *c) /* AMD CPUs don't need fencing after x2APIC/TSC_DEADLINE MSR writes. */ clear_cpu_cap(c, X86_FEATURE_APIC_MSRS_FENCE); + + /* Enable Translation Cache Extension */ + if (cpu_has(c, X86_FEATURE_TCE)) + msr_set_bit(MSR_EFER, _EFER_TCE); } #ifdef CONFIG_X86_32 diff --git a/tools/arch/x86/include/asm/msr-index.h b/tools/arch/x86/include/asm/msr-index.h index 3ae84c3b8e6d..dc1c1057f26e 100644 --- a/tools/arch/x86/include/asm/msr-index.h +++ b/tools/arch/x86/include/asm/msr-index.h @@ -25,6 +25,7 @@ #define _EFER_SVME 12 /* Enable virtualization */ #define _EFER_LMSLE 13 /* Long Mode Segment Limit Enable */ #define _EFER_FFXSR 14 /* Enable Fast FXSAVE/FXRSTOR */ +#define _EFER_TCE 15 /* Enable Translation Cache Extensions */ #define _EFER_AUTOIBRS 21 /* Enable Automatic IBRS */ #define EFER_SCE (1<<_EFER_SCE) @@ -34,6 +35,7 @@ #define EFER_SVME (1<<_EFER_SVME) #define EFER_LMSLE (1<<_EFER_LMSLE) #define EFER_FFXSR (1<<_EFER_FFXSR) +#define EFER_TCE (1<<_EFER_TCE) #define EFER_AUTOIBRS (1<<_EFER_AUTOIBRS) /*