From patchwork Thu Feb 6 04:43:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rik van Riel X-Patchwork-Id: 13962192 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 348EEC02196 for ; Thu, 6 Feb 2025 05:05:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BED356B0085; Thu, 6 Feb 2025 00:05:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B74FD6B0088; Thu, 6 Feb 2025 00:05:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A14F36B0089; Thu, 6 Feb 2025 00:05:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 836C16B0085 for ; Thu, 6 Feb 2025 00:05:04 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 3775A1A0E75 for ; Thu, 6 Feb 2025 05:05:04 +0000 (UTC) X-FDA: 83088330528.15.8614C83 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) by imf21.hostedemail.com (Postfix) with ESMTP id A80F41C0003 for ; Thu, 6 Feb 2025 05:05:02 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf21.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738818302; a=rsa-sha256; cv=none; b=ixD0BqhJ7APBRMZii61u0sf3jk6/hIjgkwAW+ktTmrUC6njzjUg5yxQ5ifq3uVJ4/vUt8M DFYpRt8TdsJAvqIUJ1plCqpcuOAzyoVsZZVjqknKBjHIShwuyfQ4pkrVZpb8ngL03Ntsfp YIUmDYpA6fESVZD/vUaNs3nCge36Fso= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf21.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738818302; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WowDIwnpgIEpwOsuWS97CLarZHxjKmlMZS46pitOn9s=; b=7pN+P8bT9SjxT8TOiqPpkOGRZuehedLCqVZzjHQqU6Mjiv1EDoP56yXxnxs3UEZsX2BeZA nZu2UCBZRHbzMF7DPQJSbLmBznVqWkwNTgISD5pgU4Mvt+a+a7ap8+u6+I7ovMpGzxvgXe /c/8O1T7qH/xzy9bXLJ4QV/TNSR7l1Y= Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1tftjw-000000004tQ-2Tcd; Wed, 05 Feb 2025 23:43:48 -0500 From: Rik van Riel To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, bp@alien8.de, peterz@infradead.org, dave.hansen@linux.intel.com, zhengqi.arch@bytedance.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, kernel-team@meta.com, linux-mm@kvack.org, akpm@linux-foundation.org, jannh@google.com, mhklinux@outlook.com, andrew.cooper3@citrix.com, Rik van Riel , Manali Shukla Subject: [PATCH v9 08/12] x86/mm: use broadcast TLB flushing for page reclaim TLB flushing Date: Wed, 5 Feb 2025 23:43:27 -0500 Message-ID: <20250206044346.3810242-9-riel@surriel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250206044346.3810242-1-riel@surriel.com> References: <20250206044346.3810242-1-riel@surriel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: A80F41C0003 X-Stat-Signature: ocqsnjs7gp6156jn7b55bysdwuitozng X-HE-Tag: 1738818302-328216 X-HE-Meta: U2FsdGVkX181DLWPRTPtAVZKFZNP30sSpS3T0OMQG82nQLsLUCIS3pWKsr+QV8O4i5WMcS48W5DBvOL3wuHa94w61rMyC61RVw00jjaECsGs1g8ucbp3wJu8Sh8OGAox8IjMbOaASzUUk/W8k5pdbbA4fzux9qwUfTYicAzI4Tk/y/4jw4OWGqWtQhjcPT7e+ObwDkfv5pWIiYmuiKMcJX0qQSqR6ttUf+N32fNoIxyz+iIqNLxRqwXY6TqaBlYMjrr5DrlwwhGoUONU4wPlnCX9MwJ/+QK7mL823oJoPoyBw3aUvEJ6ZV/JxIhgNUgrgpC9u74Sse/Ix23d1pv+BZ+bDzXIGuLi3/L3YHNzxiQAjMV9JmaK49jI/jP36emEzfxVroKQEPq+//+hZNdhDtYPBT3K0/WSHMpd7FoRjxanWtfa/9ZzssP5DA0IGo3Jqd8MhZ03hfGwHEyM64f5IpuAfAxDURKJq+WUvGb4mUpr92K/aCaJqs/FrExgMbdbnMraDJaGhbmCaVebxtQ+YLj9pUZHljOK64uhDL87ZJfz6trlER6EsC05n/Ts/G1a8tA4b+lbJG2SVTvG9WUUxYaR5DgzmVITug3KhMPezIXEX3Q53rhtRdgUg7mQxMf9MrfS2U6Pt9PS6ig45T/NN2Z3FGwF5EDLRkGG1jdFOq+vtpgjXoXiIsvHmDNt7pyyZLPeyOzTH0ojLHMYVCyww/9MAG/j0OdY3Munli5mzTEjAgPI+kllNMQ1fZE4pNnfKx+efSyXHzVQYFvmX71Ge7rSuMSlnRBOexQvadlExUJU062hGj74TLkqcorbx4cy7htaTERRIsBYeXvyR5ezb+i4PCsBgdo+lLORvD5Hpj1snCBy0IxZ/NCfNdQ48/iq4//fu51fLcRnBqIUijBRlDW+J0ZBYdFvZDcDT+L4VRsXRuud96TumGFvai88+7Aqw/NPCarUlpfP5EL5/XX uBIjP8Gc oaj9nEH+lFdacslaXdfWLD9hho42ksXrbDqjSOX+q+N5vPjeF9LrtKfC7dUT0f9GIX3NOyg5HcFxkF5gqa7kUlg5qnSMW5NHhsg0wk/h1YKToV1i9YP23d6Rsn4ukKnk26nLQA2LQKHnKFaGxgTJYwS3pCt1dg27v1WAHtxOiTtjl88oKZecCKmXkZOm3hM6jvS7Y6Txt8hbGd5R6zy2c/RWV2qEbZTGKOpBIDQ3IwM+kwkJi8DIeEqtaPI0VIgGiGSzFR/5t5XsWjYwaTjFkwbq7ZeclbMtv4WK8sJ1CDv8tCYl8i6BmpBdWOrWUclpyX15SW+QAz+tsoAMFCL1FobCPhRWgbxKd8eXv/J3O26sRxx7fk9rvCOcLM4ldoy9BGrFEGsRBxyPVIHIAFWSAhTkSM1cuo3GakiFs1rekwJFPB5DgOUKLNVPW1TQY2BKaXTrqzU3x+sairfRsTbOEs/hkoByVIT13z6GZAe1V/JegQRA7B7MkHF49uQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the page reclaim code, we only track the CPU(s) where the TLB needs to be flushed, rather than all the individual mappings that may be getting invalidated. Use broadcast TLB flushing when that is available. Signed-off-by: Rik van Riel Tested-by: Manali Shukla --- arch/x86/mm/tlb.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index d225a3df9aa7..f2d18f16d76f 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1330,7 +1330,9 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) * a local TLB flush is needed. Optimize this use-case by calling * flush_tlb_func_local() directly in this case. */ - if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) { + invlpgb_flush_all_nonglobals(); + } else if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { flush_tlb_multi(&batch->cpumask, info); } else if (cpumask_test_cpu(cpu, &batch->cpumask)) { lockdep_assert_irqs_enabled();