From patchwork Tue Mar 4 13:58:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Borislav Petkov X-Patchwork-Id: 14000783 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94700C282C6 for ; Tue, 4 Mar 2025 13:58:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 25182280006; Tue, 4 Mar 2025 08:58:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 202C3280004; Tue, 4 Mar 2025 08:58:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 05475280006; Tue, 4 Mar 2025 08:58:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D75FE280004 for ; Tue, 4 Mar 2025 08:58:49 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 957A5C1D8B for ; Tue, 4 Mar 2025 13:58:49 +0000 (UTC) X-FDA: 83184024378.11.75B1E57 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf17.hostedemail.com (Postfix) with ESMTP id EFA5A4000F for ; Tue, 4 Mar 2025 13:58:47 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="i7QlZWM/"; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf17.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741096728; a=rsa-sha256; cv=none; b=PhB7JCP5s+ceYTgCDLW0WGlg3AZ+bpemq16Z2BqYXNPCeYPfCJUJYxRZ2uToIY5oGxexUn Y6raN6EA6Y9TlFG/7OzCv3BeC0aRfrWzyDJdB8t0KoWC3th9Z7/DWDc/6GEZl4BtTuTQ0U gSeBBZw9vsfqBuCt4AYUipP8QUHLelM= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="i7QlZWM/"; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf17.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741096728; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tRO3ZdI0vGr6ql9kI7Cnzh5EjxtVoL8FFEwty0575Tg=; b=4nTVUW/841imTBlDcpNj7d748gNit0dReoIir/5wwmlfaQfNpsIJ68umN1wdnJPHxXZKdK NP3Q6d0QK2gzUFvPmF9OxXiTMhT7K5Pxl9OKpqizhI1b5sQ5id6L0WJpAPus/8ijgAuSt2 NtzUgDyNGU+pFMLUjQLQkmyK0EOqY/8= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 685D85C5D36; Tue, 4 Mar 2025 13:56:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E4F4FC4CEE9; Tue, 4 Mar 2025 13:58:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741096726; bh=peFXxf6pvLf5YEyvcQ+VbBtK2U9T/s+L+AAUIiO804s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=i7QlZWM/CvKDjp1KSyaZW0Gg+U5tDAu/tniChxnze6F0p3R4UsBk4jH0EF/Kzs3Mx 0moEMEQ3opI+uD/pw7uBzKAS6sbhHhSJkBE4pKAC4hIiyOQ1I6bV7Ae63yPKmdKtNh EtB/ZqyF/dPQKdEpvFBzOZkzsiRzXZcD+vXEUXJhYa2R72hMWWFqkkmB67ZwlqnTQn R512uPWnT7sVtlM3yK4XhbSY1taGfbn8u8PdmACuqr8Kn2V4o7m3/G28tHK6YZBDx8 uQU61uZPanCTq0lOqmnhjIkUF1YceJP0PeCCPXUZaTWx8qKKsnlzwJKmxmTvA+CC84 CkqRKyss0jfHA== From: Borislav Petkov To: riel@surriel.com Cc: Manali.Shukla@amd.com, akpm@linux-foundation.org, andrew.cooper3@citrix.com, jackmanb@google.com, jannh@google.com, kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, mhklinux@outlook.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, x86@kernel.org, zhengqi.arch@bytedance.com, Borislav Petkov Subject: [PATCH v15 07/11] x86/mm: Handle global ASID context switch and TLB flush Date: Tue, 4 Mar 2025 14:58:12 +0100 Message-ID: <20250304135816.12356-8-bp@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250304135816.12356-1-bp@kernel.org> References: <20250304135816.12356-1-bp@kernel.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: EFA5A4000F X-Stat-Signature: rh7zpdrrrz8kjiibcfpn34jsno9494k8 X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1741096727-200870 X-HE-Meta: U2FsdGVkX18VbNkta/SYK2/eskbCX9fusVRPlFc8gkOd9GPqMgcRIizGvGidxBLV+Yr9n1Rd7BBHRiSQe7SBwY0pxB5UIEFWluHh31th0haW4ERKB1anhZCm2NbGIxv8g72fzY1IMRpdP+52Yei2n62ZXWOkrjdH3HyHB2l8bL5K6wZKEnAVFAXCVDKuyRzRJ+e6BrVEP/OXx6b/fMdiWBFRFSEoHOII9uW3UbLKrSfEUXNm428KpNHvcfAhKIWbuT3butoeCWRKlDcRv/qn1XcrkjwelvIZAmFVV8FbgVr6jhd2UUzRnZNnBA7CvFvyY7XQB07nDW1dVTYmqShuC3LF754P7ATHzyoKh3LrllVQnSk4udk/PVM+D5FIA2Uey7yQ7id2WJDyLEVMmW1mfyUSk0eMpiTCbgi8+4LfZQGDrgFYqjN306IvYGw916xNPPSnOq6EvqKiYOZmTCtZFZpRQbx/xvgR1TfyJlQWVz2O+pOMi8aYjo27186x2Jo89f8MDL15oiOhl1wMgaplGIiyzwS4+x06zrwCfTGLc/0ThGSkW9Gv34JXIp584pcFODMDlqYYjFDus9niPQKVV3yDkKGUyNbzCvnrkkZvF6uv6VM20/IhXD51e1Gd4sCZ9/NklbCa8gW8taWMPEyPbbE+J3XcKH5oEBIgCeaL1fHjkJQzeouGoF7O+Fw+TVCE03jtAy9byEywtpR7VxIMupr3YOjgVMHWFknp2CZ9l98Oq4yxyEe/ED26JkSuYHmpBbupNkKb8evddFyTUOL5sYpAwWq5kmFMDNHRN8l9BH7d771ZKEVRLubQkY5gdqReph0OZd4LrWoWEpq6/89i00ZjGgHdHXz3rcvIpIf9HXUr3cuvTrdEqaat4ABx/roFjzvBbIuNsUGIrqkLRAfod495ib1LK4cvDqMnMq8Gtb3QN0FOgZfnF6xajuSdMpt/EtTdF8UxIiBzUqwxkcu 959ZobhB 5V+BUDpcqkm8v1OEHjTAo/odhjehkz1fRcs9Us7NhT16lRUdeOTAbBTowAenMmhziYfmAPVpL4W+0dPX4q4zSZsdsZ2rUw10of9bEHIdmgPLgE9HOcUAo9nJnWqlxAXET7bJScujCVdy/P1fTRDVkEGiJMBQwWRiR+Yazji6fsGTqA9Zf2gOlSPOgftSU/tBVD19LjxbEp83eojoCWZAjqfzz2GRjVZ+6SsHe0upY6VlwrK6eFt8A03VaGXGuuCF0LXQqV2/N7Ir0pNF+CS3vg33OibsxELnLUFvnHbSJqIBc91qBQIQbm3GZUbLvfmy6vImc5B1cfW58iAD0snORnYMk4dtm0tw8CV0rNRJUXNiLhQI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Rik van Riel Do context switch and TLB flush support for processes that use a global ASID and PCID across all CPUs. At both context switch time and TLB flush time, it needs to be checked whether a task is switching to a global ASID, and, if so, reload the TLB with the new ASID as appropriate. In both code paths, the TLB flush is avoided if a global ASID is used, because the global ASIDs are always kept up to date across CPUs, even when the process is not running on a CPU. [ bp: - Massage - :%s/\/cpu_feature_enabled/cgi ] Signed-off-by: Rik van Riel Signed-off-by: Borislav Petkov (AMD) Link: https://lore.kernel.org/r/20250226030129.530345-9-riel@surriel.com --- arch/x86/include/asm/tlbflush.h | 14 ++++++ arch/x86/mm/tlb.c | 77 ++++++++++++++++++++++++++++++--- 2 files changed, 84 insertions(+), 7 deletions(-) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index f7b374bcdc0b..1f61a39a8776 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -240,6 +240,11 @@ static inline bool is_dyn_asid(u16 asid) return asid < TLB_NR_DYN_ASIDS; } +static inline bool is_global_asid(u16 asid) +{ + return !is_dyn_asid(asid); +} + #ifdef CONFIG_BROADCAST_TLB_FLUSH static inline u16 mm_global_asid(struct mm_struct *mm) { @@ -266,9 +271,18 @@ static inline void mm_assign_global_asid(struct mm_struct *mm, u16 asid) mm->context.asid_transition = true; smp_store_release(&mm->context.global_asid, asid); } + +static inline bool mm_in_asid_transition(struct mm_struct *mm) +{ + if (!cpu_feature_enabled(X86_FEATURE_INVLPGB)) + return false; + + return mm && READ_ONCE(mm->context.asid_transition); +} #else static inline u16 mm_global_asid(struct mm_struct *mm) { return 0; } static inline void mm_assign_global_asid(struct mm_struct *mm, u16 asid) { } +static inline bool mm_in_asid_transition(struct mm_struct *mm) { return false; } #endif /* CONFIG_BROADCAST_TLB_FLUSH */ #ifdef CONFIG_PARAVIRT diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 6c24d967b77d..b5681e6f2333 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -227,6 +227,20 @@ static void choose_new_asid(struct mm_struct *next, u64 next_tlb_gen, return; } + /* + * TLB consistency for global ASIDs is maintained with hardware assisted + * remote TLB flushing. Global ASIDs are always up to date. + */ + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) { + u16 global_asid = mm_global_asid(next); + + if (global_asid) { + *new_asid = global_asid; + *need_flush = false; + return; + } + } + if (this_cpu_read(cpu_tlbstate.invalidate_other)) clear_asid_other(); @@ -399,6 +413,23 @@ void mm_free_global_asid(struct mm_struct *mm) #endif } +/* + * Is the mm transitioning from a CPU-local ASID to a global ASID? + */ +static bool mm_needs_global_asid(struct mm_struct *mm, u16 asid) +{ + u16 global_asid = mm_global_asid(mm); + + if (!cpu_feature_enabled(X86_FEATURE_INVLPGB)) + return false; + + /* Process is transitioning to a global ASID */ + if (global_asid && asid != global_asid) + return true; + + return false; +} + /* * Given an ASID, flush the corresponding user ASID. We can delay this * until the next time we switch to it. @@ -704,7 +735,8 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, */ if (prev == next) { /* Not actually switching mm's */ - VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs[prev_asid].ctx_id) != + VM_WARN_ON(is_dyn_asid(prev_asid) && + this_cpu_read(cpu_tlbstate.ctxs[prev_asid].ctx_id) != next->context.ctx_id); /* @@ -721,6 +753,20 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, !cpumask_test_cpu(cpu, mm_cpumask(next)))) cpumask_set_cpu(cpu, mm_cpumask(next)); + /* Check if the current mm is transitioning to a global ASID */ + if (mm_needs_global_asid(next, prev_asid)) { + next_tlb_gen = atomic64_read(&next->context.tlb_gen); + choose_new_asid(next, next_tlb_gen, &new_asid, &need_flush); + goto reload_tlb; + } + + /* + * Broadcast TLB invalidation keeps this ASID up to date + * all the time. + */ + if (is_global_asid(prev_asid)) + return; + /* * If the CPU is not in lazy TLB mode, we are just switching * from one thread in a process to another thread in the same @@ -754,6 +800,13 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, */ cond_mitigation(tsk); + /* + * Let nmi_uaccess_okay() and finish_asid_transition() + * know that CR3 is changing. + */ + this_cpu_write(cpu_tlbstate.loaded_mm, LOADED_MM_SWITCHING); + barrier(); + /* * Leave this CPU in prev's mm_cpumask. Atomic writes to * mm_cpumask can be expensive under contention. The CPU @@ -768,14 +821,12 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, next_tlb_gen = atomic64_read(&next->context.tlb_gen); choose_new_asid(next, next_tlb_gen, &new_asid, &need_flush); - - /* Let nmi_uaccess_okay() know that we're changing CR3. */ - this_cpu_write(cpu_tlbstate.loaded_mm, LOADED_MM_SWITCHING); - barrier(); } +reload_tlb: new_lam = mm_lam_cr3_mask(next); if (need_flush) { + VM_WARN_ON_ONCE(is_global_asid(new_asid)); this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id); this_cpu_write(cpu_tlbstate.ctxs[new_asid].tlb_gen, next_tlb_gen); load_new_mm_cr3(next->pgd, new_asid, new_lam, true); @@ -894,7 +945,7 @@ static void flush_tlb_func(void *info) const struct flush_tlb_info *f = info; struct mm_struct *loaded_mm = this_cpu_read(cpu_tlbstate.loaded_mm); u32 loaded_mm_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); - u64 local_tlb_gen = this_cpu_read(cpu_tlbstate.ctxs[loaded_mm_asid].tlb_gen); + u64 local_tlb_gen; bool local = smp_processor_id() == f->initiating_cpu; unsigned long nr_invalidate = 0; u64 mm_tlb_gen; @@ -917,6 +968,16 @@ static void flush_tlb_func(void *info) if (unlikely(loaded_mm == &init_mm)) return; + /* Reload the ASID if transitioning into or out of a global ASID */ + if (mm_needs_global_asid(loaded_mm, loaded_mm_asid)) { + switch_mm_irqs_off(NULL, loaded_mm, NULL); + loaded_mm_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); + } + + /* Broadcast ASIDs are always kept up to date with INVLPGB. */ + if (is_global_asid(loaded_mm_asid)) + return; + VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs[loaded_mm_asid].ctx_id) != loaded_mm->context.ctx_id); @@ -934,6 +995,8 @@ static void flush_tlb_func(void *info) return; } + local_tlb_gen = this_cpu_read(cpu_tlbstate.ctxs[loaded_mm_asid].tlb_gen); + if (unlikely(f->new_tlb_gen != TLB_GENERATION_INVALID && f->new_tlb_gen <= local_tlb_gen)) { /* @@ -1101,7 +1164,7 @@ STATIC_NOPV void native_flush_tlb_multi(const struct cpumask *cpumask, * up on the new contents of what used to be page tables, while * doing a speculative memory access. */ - if (info->freed_tables) + if (info->freed_tables || mm_in_asid_transition(info->mm)) on_each_cpu_mask(cpumask, flush_tlb_func, (void *)info, true); else on_each_cpu_cond_mask(should_flush_tlb, flush_tlb_func,