From patchwork Wed Feb 5 01:39:57 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rik van Riel X-Patchwork-Id: 13960404 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C642C02196 for ; Wed, 5 Feb 2025 01:41:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DEE8D280016; Tue, 4 Feb 2025 20:41:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DA106280009; Tue, 4 Feb 2025 20:41:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C673D280016; Tue, 4 Feb 2025 20:41:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A5EF1280009 for ; Tue, 4 Feb 2025 20:41:57 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 529BDC05F0 for ; Wed, 5 Feb 2025 01:41:57 +0000 (UTC) X-FDA: 83084189874.08.88AEBE8 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) by imf27.hostedemail.com (Postfix) with ESMTP id 963FE40003 for ; Wed, 5 Feb 2025 01:41:55 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf27.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738719715; a=rsa-sha256; cv=none; b=LLK31Mj5hxojVyo3J1xeGfHThLa66WkYCXve0qeo1EXNNXM1G8CBo7n7KU9L9hsW22OYII OI1iqYxfCEpZ8k//B9k1VOJ2IAUyYAGPnrU6Q3YvKLPMse/xI36P86kFPaaglllFQWdU6c EBpdHueRWJhTsp4qQvwKADdUvOOEg5M= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf27.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738719715; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RJvH0dEjzsESgMcFRzd9556aYq/w5MJOy0P94uuYdRE=; b=4Lzm32WJ3Z6xuiJvZpPft0+wSIU692gx/qLZBsHcppp/4OmhDSHvfiZqQ4cSJ/dKjXizF7 BvQX7lMCKzsEZPdRvFuMk6v7E7kx5hSUCQ1gM9P5OdUC904qwNg8E5RNBIfyx0/BBl7YSP lM5z7bYlpTMIe1QEz3/CGIFuVeTfu+g= Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1tfUP4-000000004Cs-35hO; Tue, 04 Feb 2025 20:40:34 -0500 From: Rik van Riel To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, bp@alien8.de, peterz@infradead.org, dave.hansen@linux.intel.com, zhengqi.arch@bytedance.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, kernel-team@meta.com, linux-mm@kvack.org, akpm@linux-foundation.org, jannh@google.com, mhklinux@outlook.com, andrew.cooper3@citrix.com, Rik van Riel , Manali Shukla Subject: [PATCH v8 08/12] x86/mm: use broadcast TLB flushing for page reclaim TLB flushing Date: Tue, 4 Feb 2025 20:39:57 -0500 Message-ID: <20250205014033.3626204-9-riel@surriel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250205014033.3626204-1-riel@surriel.com> References: <20250205014033.3626204-1-riel@surriel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 963FE40003 X-Stat-Signature: qjrsemwyxmtus8yyn3p9mkstmqxmpeos X-HE-Tag: 1738719715-206273 X-HE-Meta: U2FsdGVkX1/ogxMZ93titD7ECU5bgPW/4ntfKuAI0AIvKM6T5Q4+sG6WRVlcs5+QnJ+C+R1naFLx7aSJJnkOdDX28ANrHUQy1H0vhjTiUD8WgOfEmJxc0LlRuINc57c2fKQl0FKOBMHD7YvGKrmrR/aX3GVhqTWNCOD4WL0mexVeTZSw4rj59nsIBcw1wJdHCJbgcR+fII5qkvDMUiXL45dATwIw8buqTZwKGEIshZkDXPYUThM+q9XbAqqQnGvMElDlnfhooKvdCoWhn1R+kMjl8uLHtxLkeAFImlkanaajEQ3UFmzWdL5xEf219l94wV+EsVvyUsZUG0or75yCMmEosyvHvJHG1VjEp6Vs9nu4hUDgXn4tjjlwW6p2r1Poy7x+ZjN65+GQB3sCCmrTu39Q9pG0MDsmtCZoRx5MhW72zxfdEM95yJAkTUYf7Mdj3+c0etZ/izSe2dYsJKtbCPDLqOJQR5fV35MSEYvLQG20t58V7Rax2Yvi5J26PbDUAlN28BW3JSODtCgHR/NcfXbJVxiOH25AU3pbbMVo7nVSu7R15PjTNrVugcgcVT9vwLvvmZxqYgQ7LJ4UluilSWAD3k2Dx6a+dC8HNVZ0txHQleNe599Mx/W8RuWBnyS289MVLXLIURQvhhAHC56VeRpSswR0kgW93WdsCJRadol2BgoNklc4hxedSwR6pddFXVgYKqMZbAdd+1GanaeuDp8j/4HjWFICQjaU/OOqzmBkpuek2uXG9B3zZAG7v5gPNhBxjtHuBG2M20xt9dVtKKBY3qx2uvFz6XY/4leKFgPcfF7Wa4XaJ614uipsESkOVmpYdpkuS4ClCAJ0TGiar6l6CWEclVWypZjHruXGXxu+SZIvSXwJuCBmQqzmbDEdPHoxBHYckPPhMdSPL4lCbr3YSbtHy/h4apk1XNrfE0IbLhiZMU8TKtn5h5s0Y/Lkl8UelobHJrUEn6SpmFp da+uh7GE d6wiA9m1OwCMhC7kPuyC50W7/EAUZYZlE+RwUWzYsVGEqrLIiZPEl9KzB4cBY2QrcTceITytVnmDN+2sNXkyiPnoKTGRsMtOwY8Ptqg3eVyWQucGcGEJr54JW+mn0jXvhmWMxTS/zYLput0Ep4hrhxmSuomxbcy6vuwHSRjNxNvFyT77Pxa95buvQH4TOvwvQDtz4CvPkEjG9uPFq9MKX4jSsGV5onN7ZwjQNMpZ+uBH+QQhBUfUdLV8LKiDKwXsWeKybfFJmhBRREc1qGkqfop45dcWI6ygKcTPvrlq5jg/CiT2kUoi9Ct7WQH/iLem+EGGZuJoZeQiPERBKrEB/PViSu85Z6Tm0YNJT1qQJ8vQ3dVwg2hBaTsTtYA7VsHooEcici+vx7uiKjSQ1YzQlJKusORIZNQlo042H9Lzekeq9AD82y/SoiHp+OPYoBqNA6OsmYqzo++Xbkk1PPYR80yoU0pBAx8ciz0+Jinh+TwlZ3i5SYpASuBt1iw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the page reclaim code, we only track the CPU(s) where the TLB needs to be flushed, rather than all the individual mappings that may be getting invalidated. Use broadcast TLB flushing when that is available. Signed-off-by: Rik van Riel Tested-by: Manali Shukla --- arch/x86/mm/tlb.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 43e1abd4bcb0..25c13416a0e8 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1326,7 +1326,9 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) * a local TLB flush is needed. Optimize this use-case by calling * flush_tlb_func_local() directly in this case. */ - if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) { + invlpgb_flush_all_nonglobals(); + } else if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { flush_tlb_multi(&batch->cpumask, info); } else if (cpumask_test_cpu(cpu, &batch->cpumask)) { lockdep_assert_irqs_enabled();