From patchwork Tue Mar 4 13:58:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Borislav Petkov X-Patchwork-Id: 14000779 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 143FCC021B8 for ; Tue, 4 Mar 2025 13:58:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 86A05280002; Tue, 4 Mar 2025 08:58:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 81A466B008C; Tue, 4 Mar 2025 08:58:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E6FF280002; Tue, 4 Mar 2025 08:58:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4BA286B008A for ; Tue, 4 Mar 2025 08:58:36 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id DFF885532E for ; Tue, 4 Mar 2025 13:58:35 +0000 (UTC) X-FDA: 83184023790.19.4846EEB Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf12.hostedemail.com (Postfix) with ESMTP id 4433F40018 for ; Tue, 4 Mar 2025 13:58:34 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=dyCCJ2el; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf12.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741096714; a=rsa-sha256; cv=none; b=WD6b4D3FcqZ9Qvo+vAOlotz4tCaKhBIH7A3XmgQbA9HXv850Hx24OqiEXQvDevTfqjOxdp UWsJmUfRcPjUQ8Wa6kVXqoRYBTvB+2Nl1ZNBbl9zfn0KwpD/LLHpIPgFxQQiCL+ucohunq K+s5vP1/heP+AG/Ns7cuA9QJ3FflMtc= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=dyCCJ2el; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf12.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741096714; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xEsQkPYQlYiTKuL2teGu/PlbVN6Z5HqX0RAu700uAWg=; b=qyhGPLUAldLvWw4GiFWD8djCxUlzPZiZW1tKehWjG0o7O+vriVVYxNLfU7JurwToa4Ebnq vvMuEmjTM5GHycr8LxR1St1AbqzKYy0Wl6MRGDhfHvPI9lnlid2TWI0PQofiIo4JqUmo2F aLelZLIkno4QegL6Ur5ShOa/FGpVOUY= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 941455C5D33; Tue, 4 Mar 2025 13:56:16 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4164FC4CEE7; Tue, 4 Mar 2025 13:58:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741096713; bh=X1huOsN5W4jKLuaRwLL36yDmkKY/JDVc78p3WqGQVBs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dyCCJ2el7Tll84ljQirbpN9UXDOr1tBvbVY6JChmK1K/KX8vxcYbzjQMOxB8YaB2g jn400jK3VU/APZOnikrergnf8HR5i6vNVfPZBTW/+1jFlsYjsbsgHj6wDQ8vcD3gH0 O9VpzREeaAaqFN8/MCnbe9k9BYbaI1JXYVSS1v+leMLf81F3zO1p2SN374YLDeRSjQ E6SK009MqxA+V8wUDnhUKbt/MtJxM9OzER6wgXmMoPuEITmoIFkZDYYqeTUqcKEcQI 7blFfCoHOzK8s3V89XKBFzaEC0NTe9Bjd05+qIorFP1czuA4GDJwToJ29cx/KPdJm0 GneKxW3yoWLxw== From: Borislav Petkov To: riel@surriel.com Cc: Manali.Shukla@amd.com, akpm@linux-foundation.org, andrew.cooper3@citrix.com, jackmanb@google.com, jannh@google.com, kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, mhklinux@outlook.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, x86@kernel.org, zhengqi.arch@bytedance.com, Borislav Petkov Subject: [PATCH v15 03/11] x86/mm: Add INVLPGB support code Date: Tue, 4 Mar 2025 14:58:08 +0100 Message-ID: <20250304135816.12356-4-bp@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250304135816.12356-1-bp@kernel.org> References: <20250304135816.12356-1-bp@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 4433F40018 X-Stat-Signature: y58q1gcpepz5ukcazzwgdy3h9jz9xwyp X-HE-Tag: 1741096714-880281 X-HE-Meta: U2FsdGVkX1+rI1vLj0yxPqvp1wU+nYPXpx3YkR3jN1SJ4lsX7wLRG6n3H1Pi8apYlzLy83QsqctAjzsi/CYNos/5dwbR923x1rh7UQnh4wqf+oTvuhSZSC48rYS8La4g3m+JqmpbFPxz1FSofrjhxSd0k/QkAU4NhIC1rTsT6xgyNkDxoigS+QcEzdsiDHCc4L/2va6mC+ZK1BQaKnSQzEo+uf3T3o6NnZ5hVjISy42fyoihjM5ERScjgdIy1+st7egZeYb5TmBdd0H4+n3mb97CcTNqIRx3IeB+Rv8tErtFK+EBbaiZY11qIDWDmXRBRacspUYa3cw1maYMmQFjghFSvq+cafNtoKCZgd5+6CR+G1EDfmioQmbFPt1i3bweMBxUCZyDV0c2Lo3szmzo/cRiKXgKxBg9XVDzqXedCc8lXZVYC5I6KZAPnChT67nUkPrJ9oSx5sOur/U9EnzcJs6mmtLCgmFiKK3FiYuwGMj0ndYqtVaoW0XcJAcGvj/PjhtKHV15REdvjPr0fM3yLNt1vTNeEzEJb2iVikiWmkRfU+c2RxRTlFGn8ljvqHjCWmvDhSvqF9rCGKxpG4qS0P81QJ9DTDBETNUzztmEt6+OMCaamcY6My31hIgd52Mhsy38j8xizZ92Nnf61/EHwDM164M4AU9rD+lWHgdYer0WVZizFVqW9y0L+Fbv93A/hrlanO8N2+las7Lt3OuX/LAQ+/7hU8sTt9nOHYl43i6nD8VltioT9xg17QIxr+Cjq4uH3zRxEayyvBGYjCCEyJ9nQBUr8sNBSUrG5stmIgCwZwniVH6NSfA340RZniY7l5oBEJ5CDirl1FBLREAIz26a85+Au/XfjPcQEbLu426GkStW6VLmV2+xaLBlXQRBPOkuZi/UZGBAeBAGfEc3WbgovpGhqGFIUq+Nra0Fr5JqsNs67di1OB746IfHIzH19kIDF0nAI14G2ELwE2T eERrqG7X lnLuW69e38he/0WbXDkimrLix7dFQyDutCx8gyptIIYsVSPAokWQpR5KxacNWb4vQIeweLAhev9YW/9rSnNWuv4QsoBWV/8dmnSoETkyAwxXq9uVUlyTSCHag9OSjoVdHdZA7CcKUmihcT+3DBZw3tE736mYPUgKm0WDtTMHSkjG1aZb7PHKPWKdO77DMXHpDiKPm1ftYDgY+5mMqlMfL97BS539AnTMiF2dYHOleJNQu+/zaqVMRsrwLnpb3OHXN+6LWFZl4JrlLdsQt2ne5OwPnB4p/cUbpTR6FLH2WJmnAY4Mi0K9JbzoMv/WlyuZBILaO3eDy70f8T1BHCz7YSrV04p3Fn7UqqQR5UZeptXrbw3A= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Rik van Riel Add helper functions and definitions needed to use broadcast TLB invalidation on AMD CPUs. [ bp: - Cleanup commit message - Improve and expand comments - push the preemption guards inside the invlpgb* helpers - merge improvements from dhansen - add !CONFIG_BROADCAST_TLB_FLUSH function stubs because Clang can't do DCE properly yet and looks at the inline asm and complains about it getting a u64 argument on 32-bit code ] Signed-off-by: Rik van Riel Signed-off-by: Borislav Petkov (AMD) Link: https://lore.kernel.org/r/20250226030129.530345-4-riel@surriel.com --- arch/x86/include/asm/tlb.h | 126 +++++++++++++++++++++++++++++++++++++ 1 file changed, 126 insertions(+) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 77f52bc1578a..8ffcae7beb55 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -6,6 +6,9 @@ static inline void tlb_flush(struct mmu_gather *tlb); #include +#include +#include +#include static inline void tlb_flush(struct mmu_gather *tlb) { @@ -25,4 +28,127 @@ static inline void invlpg(unsigned long addr) asm volatile("invlpg (%0)" ::"r" (addr) : "memory"); } +enum addr_stride { + PTE_STRIDE = 0, + PMD_STRIDE = 1 +}; + +#ifdef CONFIG_BROADCAST_TLB_FLUSH +/* + * INVLPGB does broadcast TLB invalidation across all the CPUs in the system. + * + * The INVLPGB instruction is weakly ordered, and a batch of invalidations can + * be done in a parallel fashion. + * + * The instruction takes the number of extra pages to invalidate, beyond + * the first page, while __invlpgb gets the more human readable number of + * pages to invalidate. + * + * The bits in rax[0:2] determine respectively which components of the address + * (VA, PCID, ASID) get compared when flushing. If neither bits are set, *any* + * address in the specified range matches. + * + * TLBSYNC is used to ensure that pending INVLPGB invalidations initiated from + * this CPU have completed. + */ +static inline void __invlpgb(unsigned long asid, unsigned long pcid, + unsigned long addr, u16 nr_pages, + enum addr_stride stride, u8 flags) +{ + u32 edx = (pcid << 16) | asid; + u32 ecx = (stride << 31) | (nr_pages - 1); + u64 rax = addr | flags; + + /* The low bits in rax are for flags. Verify addr is clean. */ + VM_WARN_ON_ONCE(addr & ~PAGE_MASK); + + /* INVLPGB; supported in binutils >= 2.36. */ + asm volatile(".byte 0x0f, 0x01, 0xfe" :: "a" (rax), "c" (ecx), "d" (edx)); +} + +static inline void __tlbsync(void) +{ + /* + * TLBSYNC waits for INVLPGB instructions originating on the same CPU + * to have completed. Print a warning if the task has been migrated, + * and might not be waiting on all the INVLPGBs issued during this TLB + * invalidation sequence. + */ + cant_migrate(); + + /* TLBSYNC: supported in binutils >= 0.36. */ + asm volatile(".byte 0x0f, 0x01, 0xff" ::: "memory"); +} +#else +/* Some compilers (I'm looking at you clang!) simply can't do DCE */ +static inline void __invlpgb(unsigned long asid, unsigned long pcid, + unsigned long addr, u16 nr_pages, + enum addr_stride s, u8 flags) { } +static inline void __tlbsync(void) { } +#endif + +/* + * INVLPGB can be targeted by virtual address, PCID, ASID, or any combination + * of the three. For example: + * - FLAG_VA | FLAG_INCLUDE_GLOBAL: invalidate all TLB entries at the address + * - FLAG_PCID: invalidate all TLB entries matching the PCID + * + * The first is used to invalidate (kernel) mappings at a particular + * address across all processes. + * + * The latter invalidates all TLB entries matching a PCID. + */ +#define INVLPGB_FLAG_VA BIT(0) +#define INVLPGB_FLAG_PCID BIT(1) +#define INVLPGB_FLAG_ASID BIT(2) +#define INVLPGB_FLAG_INCLUDE_GLOBAL BIT(3) +#define INVLPGB_FLAG_FINAL_ONLY BIT(4) +#define INVLPGB_FLAG_INCLUDE_NESTED BIT(5) + +/* The implied mode when all bits are clear: */ +#define INVLPGB_MODE_ALL_NONGLOBALS 0UL + +static inline void invlpgb_flush_user_nr_nosync(unsigned long pcid, + unsigned long addr, + u16 nr, bool stride) +{ + enum addr_stride str = stride ? PMD_STRIDE : PTE_STRIDE; + u8 flags = INVLPGB_FLAG_PCID | INVLPGB_FLAG_VA; + + __invlpgb(0, pcid, addr, nr, str, flags); +} + +/* Flush all mappings for a given PCID, not including globals. */ +static inline void invlpgb_flush_single_pcid_nosync(unsigned long pcid) +{ + __invlpgb(0, pcid, 0, 1, PTE_STRIDE, INVLPGB_FLAG_PCID); +} + +/* Flush all mappings, including globals, for all PCIDs. */ +static inline void invlpgb_flush_all(void) +{ + /* + * TLBSYNC at the end needs to make sure all flushes done on the + * current CPU have been executed system-wide. Therefore, make + * sure nothing gets migrated in-between but disable preemption + * as it is cheaper. + */ + guard(preempt)(); + __invlpgb(0, 0, 0, 1, PTE_STRIDE, INVLPGB_FLAG_INCLUDE_GLOBAL); + __tlbsync(); +} + +/* Flush addr, including globals, for all PCIDs. */ +static inline void invlpgb_flush_addr_nosync(unsigned long addr, u16 nr) +{ + __invlpgb(0, 0, addr, nr, PTE_STRIDE, INVLPGB_FLAG_INCLUDE_GLOBAL); +} + +/* Flush all mappings for all PCIDs except globals. */ +static inline void invlpgb_flush_all_nonglobals(void) +{ + guard(preempt)(); + __invlpgb(0, 0, 0, 1, PTE_STRIDE, INVLPGB_MODE_ALL_NONGLOBALS); + __tlbsync(); +} #endif /* _ASM_X86_TLB_H */