From patchwork Mon Jan 20 02:40:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rik van Riel X-Patchwork-Id: 13944700 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6786C02187 for ; Mon, 20 Jan 2025 02:42:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3CDAD280001; Sun, 19 Jan 2025 21:42:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1C0A86B0098; Sun, 19 Jan 2025 21:42:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F41316B0093; Sun, 19 Jan 2025 21:42:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C67D26B0088 for ; Sun, 19 Jan 2025 21:42:29 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 7B97EC2643 for ; Mon, 20 Jan 2025 02:42:29 +0000 (UTC) X-FDA: 83026281618.24.E6DB113 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) by imf20.hostedemail.com (Postfix) with ESMTP id E8FBF1C0006 for ; Mon, 20 Jan 2025 02:42:27 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf20.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737340948; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hYZKHKN33BrkcN2hT6qEtBNKXUt3RzEMGWzOtIbwDXY=; b=GbVwi0PNlt69jFLWfkf7kxsMSsDTFQMxoFJXWqv0t7KSBttfey9ZPzws3kFv27GEx73BCx p7rlHwRsU5Mh5QhRFSD1jslq2+b6ubrY6Cls4kW9o9UYDC9Sw6XdWWBqr5vl7w5WblM0zG Sz7ZmG2GNvKrs0TGEFJkT/JDJ/U8Yf8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737340948; a=rsa-sha256; cv=none; b=jL8/1Kv+UBPgMt8DSfXlOEzmpiMDNTXYlMsbIws5SKGzpc1dsiUaurBthP3ltsYTo2n7yV AUvfsx4bZ/zz037fNcWvQG925VCA7nrAWD8FQzfosvP9/qFIkhZQA/mCKtnDdA+CUdE3Qm RanGC+FYwV1UL3yul3hYZo3Yu3xmF8g= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf20.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1tZhis-000000002w5-1ISQ; Sun, 19 Jan 2025 21:41:06 -0500 From: Rik van Riel To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, bp@alien8.de, peterz@infradead.org, dave.hansen@linux.intel.com, zhengqi.arch@bytedance.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, kernel-team@meta.com, linux-mm@kvack.org, akpm@linux-foundation.org, jannh@google.com, mhklinux@outlook.com, andrew.cooper3@citrix.com, Rik van Riel Subject: [PATCH v6 08/12] x86/mm: use broadcast TLB flushing for page reclaim TLB flushing Date: Sun, 19 Jan 2025 21:40:16 -0500 Message-ID: <20250120024104.1924753-9-riel@surriel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250120024104.1924753-1-riel@surriel.com> References: <20250120024104.1924753-1-riel@surriel.com> MIME-Version: 1.0 X-Stat-Signature: 4t1rocj7hsfufomsen7ath65ei6trrd7 X-Rspam-User: X-Rspamd-Queue-Id: E8FBF1C0006 X-Rspamd-Server: rspam03 X-HE-Tag: 1737340947-264176 X-HE-Meta: U2FsdGVkX1+gDpZOo+o0u6Am7y0eKcDaBUb3Dq6C0uq/J4ov+IPOBYII5/OCpKZMTMbosK73ajPQ2tFOCd+mlhh2fIsNu9X2N4je5Wwpx4jGw2KKLk1D28sVaio/giHcI6q7sBGy6Ea+6ZeOp2pl+lRb4YU2nI+R19gH+UJ7mmv0K72kVuSXrXMfnFP/fWDjDmRriuyESq6SMJZ5bnY+EFIBE5tLclScVnRqMH2Pp3YWwF79j3ODW2Ig2izALAipl7WoaQLGac9ZPaXD4Rhda2zkERSps4Tx719UMjqIFDvUEK4/MXkg2KJhx0vj/vygQXU5uS6OFURgzzOx0tSO7VETdR7WF8Sxgdv0if8wXRtnc3BHfbio2kfDXqZgqM0oP9yGTCJZz+sDbpldbn73ukyFLi8qLAkvVS/8A3Npl5zesHnHJeAf6rHhixhSuNiBLSEiAW4WUbnHq/EUaXDvEQQIrxTkErPnmXdNbB+1YCw1lJzVfXGZSSjG80/3H4onjYm1tPszCngNM+05qU4OXdkx01az3+T+CeUUT7fWDnHafJXqbbQaB4G13J01bBKA/rO5smCY6Qt67uf97tFulVAiCcP/RNuQHUocoBkwrom41iQsEnIySHvWyF0Gxc7omYIqiuNrYnroFFV17YpBF9aHBPhhKvYWekVCxh/CyNk7oQivjl3jjTeaTp0gaznqrSlyRtKGEeTqFipakPuBLivukkhkAgn51E5kILJQPrGoPgbWFQe5iasZjdQmJtDjBWfQBv5h55rSVRSGsuLO5NeFFx5pBd65vRtGBP3F9IH16yT+LcLgEIcdXATWn6DUvh1zUXGyt7C/izBtCKMJBJ0gD6kg7h4zWcK0EdGmFQqe6y1/0wx83yiQVhnsdGMlMlIOPv2RfAPJpXFdhxO12bsyPVmUud3vnXacTND3TWvdR+sK8Nq4tE3nmOuZ9YvV3gHvvqyGdGjqOVLczxv Qg9K3EZc 37etcnjJKuA6oVfwAGt8NTSM0QyZr7P+h7LnsfXrKdYaegxReaslQc0uIETtjnztkHQL2jSe3pxSni9n2EIPxIvBOCCXXuJwr4n4MzdXrJjhgWoyKxwEBeFAVSTxJsII6UbwZHh/ZRtEExEOFfDcE8tXPMzIdDCzOwTHu16xKjhdDPUAiIMP5nmQr3h8BWd/tv/xUu3KBC89YcSQH5EIKFFIB4UPAuowouIMMki727ugbGhI+K2NgNoZ56CNtROuNz7GfiEPObJr9aMx2jR97vqbS8V/UDLwXwROBEOSj20OSzZ1NTWNgIOGcJIZyw8r1Uwmnl3eGeCUE3F9y/ayQNQxlzrL+miCxo8tLcrSz9GdQfGESLMD8YIdPi+Sbpv0MldiDSvL+n7crPCwAEZ+n0x5c8VIlWiiPZi948R8w+HOEF8w= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the page reclaim code, we only track the CPU(s) where the TLB needs to be flushed, rather than all the individual mappings that may be getting invalidated. Use broadcast TLB flushing when that is available. Signed-off-by: Rik van Riel --- arch/x86/mm/tlb.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index e2a0b7fc5fed..9d4864db5720 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1321,7 +1321,9 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) * a local TLB flush is needed. Optimize this use-case by calling * flush_tlb_func_local() directly in this case. */ - if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) { + invlpgb_flush_all_nonglobals(); + } else if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { flush_tlb_multi(&batch->cpumask, info); } else if (cpumask_test_cpu(cpu, &batch->cpumask)) { lockdep_assert_irqs_enabled();