From patchwork Tue Mar 4 13:58:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Borislav Petkov X-Patchwork-Id: 14000781 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20805C021B8 for ; Tue, 4 Mar 2025 13:58:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A66116B008C; Tue, 4 Mar 2025 08:58:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A21BA280005; Tue, 4 Mar 2025 08:58:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8DE09280004; Tue, 4 Mar 2025 08:58:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 7172D6B008C for ; Tue, 4 Mar 2025 08:58:43 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1D8A4141D06 for ; Tue, 4 Mar 2025 13:58:43 +0000 (UTC) X-FDA: 83184024126.05.8A179B3 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf08.hostedemail.com (Postfix) with ESMTP id 8272416000C for ; Tue, 4 Mar 2025 13:58:41 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=kd8Yw7Lw; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf08.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741096721; a=rsa-sha256; cv=none; b=QRPj8+xg8SZCRUNIL6PHLSGbCOdLqFn2ZFWw4avT6M3qXldZ0pQYFGVhAAZOws7TISaXW4 tpTqjRK+PbOa0wYy7y5tiUJonm5VJUQbT8apfDhCypBp8IiJ94/yMqxvbHUzXDSb1GyDr8 YwMXCEHbkEIXRgx/DglzZebM/dramoQ= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=kd8Yw7Lw; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf08.hostedemail.com: domain of bp@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=bp@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741096721; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=omTic7yHPg2GYmfJbHtpwTPhyEj1T7bmPvF7pe9loYw=; b=0cFdiTWQxvZ4S8IwbuikAQ3iywMZ3VBwKU6xfFuUK2QA8qwnT6SJgsCtRpzsEclDML1DJf /MjH+dZeREe1pwms/tb4nd66Bi+nbzW5w3tQLf4Zkw9zSHKc6EnpKEuwK6ztJlVIITgrtk gm50WiYpxZwLviWk6s3KSCuX9ZVFR/8= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id E27865C5CF5; Tue, 4 Mar 2025 13:56:23 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 174FDC4CEE5; Tue, 4 Mar 2025 13:58:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741096720; bh=txoN3V9rWwXFEyWDRXIoMCIFbMoUCiBnnRfSGjS9m3M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kd8Yw7Lwr4sjOpYWFXwXQSEC5xzprBOCIBE7bXwYr8GTXC+xFDjpLzy06CmfMPBne u6ZdRClLHEfy/M+wAKbra6zqvNXfIiC4v94rdgan9Rd1Z/UVXIeuRLeQOB0w8w0Xlu 5D8xfVyMQnYo7qU7qdPIZZsF1BViGBgk33U39jmiZdbrGUj9z3t4C8bt3CBSbFT5l9 /2r7w6F9gxN5CfJM9sClY0EMsUCrLtI0k6as50Bjxm6gCLyKhbZy2BdTMpSEFVRycd o0c3oY9PoYlFvhAh3shml2NRuxEbfXoD8Rlj1YvT24ywiz2F33RnzIx9Taqe6dint8 rD4d7P/Jk/CkA== From: Borislav Petkov To: riel@surriel.com Cc: Manali.Shukla@amd.com, akpm@linux-foundation.org, andrew.cooper3@citrix.com, jackmanb@google.com, jannh@google.com, kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, mhklinux@outlook.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, x86@kernel.org, zhengqi.arch@bytedance.com, Borislav Petkov Subject: [PATCH v15 05/11] x86/mm: Use broadcast TLB flushing in page reclaim Date: Tue, 4 Mar 2025 14:58:10 +0100 Message-ID: <20250304135816.12356-6-bp@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250304135816.12356-1-bp@kernel.org> References: <20250304135816.12356-1-bp@kernel.org> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 8272416000C X-Stat-Signature: aa8bg7s9ssfy943asft1x7d8htc44deh X-Rspam-User: X-HE-Tag: 1741096721-868233 X-HE-Meta: U2FsdGVkX180L4Ol8T1lL9wR978FT29qP0BDSMARdEEqjYLXpEMEzykS6QpDuV/6tD2G2EetKIvoH5bawfRCRPBx1iwBCE2F0kCTtYXNmwrnJMFE0sD4y603M0UogkWyhtBEa4IN1jMPH/m4yiMVs+UcQYymRV3mNsb7ZPST8XyoeP68/HeVIH0z9LKYpdl3vkbmyU6RYRrkXvvThCSwlbTNrRqqxgjhFSrqp1Bo08sbf0ExOReCr1oD5dlrSeqYEZiyeZiflZnR/Uia708RjimBAKDVmi0+9lQqVz45EcTNAP0C0RDpflto6OmbvF9jewGEO/0fbaZw5LYjW8pMU6w0WSKLBKEPKdOtXIBjiToR0YLS60v09jOBVglaefY6LUp9WOC7i0AHkuhIgkzTrRS3tu8jygGhNTv+XVW8NyUB5yi3kc9+L7laJIaEMGIvnlWDtAhSS1oEWWxotpqapZMsTP6jgCUCux+x9WXoAnlendPUFEpkL5MeSTilcJXDm9xZNHTM6VEMbA6+D2WVab7o5QtfXWD1jmQ5I48ARNyuDRi1OhrdIJoxKpygsq0Zwn7lHM3myFqEuUkYIL7fxqsJf7EnQqbpwYMIbclQTEgcqDILKpEswDKFP+7iP2cZOkwTRDxocKo5+V7cpJ3N/nviIK/bvDT0AgEqnW714wfklLYQ4HSluApLjCmUgUWjAQU+AmIyKrpqzH6Wyw2xqVLUpdcAxtKFNFMvwS7OGgn+qzuZLELHafvOxDcGMK21q1SdqblVJsi/7chdD6dXarTk9whAcbj2/2pwAnJU0w4l5Sh/yf8/MjMklHaFlNK9gq+O0K9PACgygp5veyotzv18q6sZfS+d60vHlcEJiqEDYNLjawo9dXP76hcjvDmP9jatuvWJdYdFOB22SbMYUm4CcavrlTiVImM/Pj7dWC4sVUXbpPmCUHkiDLHbx6QCij0/vpuq75EBd+SKmmR NADe2qXC 1t8hijquCiT0vKSWTemS/wvGeKppt6Em2yAVoiNs1Uj1/tqueP+Ad0nhTsTz6R7G3KmySOnB+eDQTedJTKXfIirIHX1tWkcNShkzlmPdHZMjFFL67vCHCfO8Xr9FiwbAHEzvHb41Ia8qVVOGEErqOGlm7U7zaciEFe1occxoiftbxWUTDFtGtF9qjyrEqzCIX/Deyq4L/lYRyDQCUtyLnlq9NprKEey4YbWo3C9LyV12tWuGjAiN9XNHk+i94QIy6PidS6RcDcDoVQuUu57erQtZiZR25wODJhrTqno6kHpBAw2jaPNk02OznSGKFMZeYyQwZqlhZvpKKxbX0I1WCoTaxlxK1qmMjxpuaWjTS1sNNXms= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Rik van Riel Page reclaim tracks only the CPU(s) where the TLB needs to be flushed, rather than all the individual mappings that may be getting invalidated. Use broadcast TLB flushing when that is available. [ bp: Massage commit message. ] Signed-off-by: Rik van Riel Signed-off-by: Borislav Petkov (AMD) Link: https://lore.kernel.org/r/20250226030129.530345-7-riel@surriel.com --- arch/x86/mm/tlb.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 8cd084bc3d98..76b4a88afb56 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1320,7 +1320,9 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) * a local TLB flush is needed. Optimize this use-case by calling * flush_tlb_func_local() directly in this case. */ - if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) { + invlpgb_flush_all_nonglobals(); + } else if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { flush_tlb_multi(&batch->cpumask, info); } else if (cpumask_test_cpu(cpu, &batch->cpumask)) { lockdep_assert_irqs_enabled();