From patchwork Thu Jan 23 04:23:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rik van Riel X-Patchwork-Id: 13947928 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90A79C02182 for ; Thu, 23 Jan 2025 04:26:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2EEC36B008A; Wed, 22 Jan 2025 23:26:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 276946B008C; Wed, 22 Jan 2025 23:26:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 117F5280002; Wed, 22 Jan 2025 23:26:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E42AE6B008A for ; Wed, 22 Jan 2025 23:26:42 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 9642C1A0AC8 for ; Thu, 23 Jan 2025 04:26:42 +0000 (UTC) X-FDA: 83037430644.14.D76EEA7 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) by imf28.hostedemail.com (Postfix) with ESMTP id 1F4F6C0004 for ; Thu, 23 Jan 2025 04:26:40 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf28.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737606401; a=rsa-sha256; cv=none; b=308sFyXol9cwMOpxqqsN1QNxzJg6rufKQi4WR6QGpXMQxQosQ48eHW9UwUOl5Z6tr5UDs/ q+uVkSjgQCR99VU3+wvGQwrjZWIL5q0+1Olh2prhXBrDC1y6r/HYohp7U4Ek4xx2oSPiq8 vG+3rvnx6niydFHCFEDMBQhxp0j9d/A= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf28.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737606401; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hYZKHKN33BrkcN2hT6qEtBNKXUt3RzEMGWzOtIbwDXY=; b=Xpp2qN1D52gCX92Ftq39kqx/1P3EloCZDNR1zM/lYswFF1JuFq/grKILpj6RGEeYKcMYAw jSY+w6dFfz0nfjY5FizD5eXOZZSKfByuAp3FLyVfmwaXHKnjn5V7zG4mN88CuA8W4fTAth QeG0hHFnAwxO2Q+vWmPtSo2MFQcEXqk= Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1taols-000000005uH-396n; Wed, 22 Jan 2025 23:24:48 -0500 From: Rik van Riel To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, bp@alien8.de, peterz@infradead.org, dave.hansen@linux.intel.com, zhengqi.arch@bytedance.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, kernel-team@meta.com, linux-mm@kvack.org, akpm@linux-foundation.org, jannh@google.com, mhklinux@outlook.com, andrew.cooper3@citrix.com, Rik van Riel Subject: [PATCH v7 08/12] x86/mm: use broadcast TLB flushing for page reclaim TLB flushing Date: Wed, 22 Jan 2025 23:23:27 -0500 Message-ID: <20250123042447.2259648-9-riel@surriel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250123042447.2259648-1-riel@surriel.com> References: <20250123042447.2259648-1-riel@surriel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 1F4F6C0004 X-Rspamd-Server: rspam10 X-Stat-Signature: 8y5zckyd33yayey65oacg5seozacz6f3 X-HE-Tag: 1737606400-486586 X-HE-Meta: U2FsdGVkX19v1g2XeG2tdcsfRCgadlvqG6peFMMbMXPTkrt4JacW+eyRghgN7eX/Q6ymAoTLgkgNUDGysDiQ5YO9wsxcw25bMj0mfwXoEFjMvBsBz0uoAyBwG155pV5JVUom3L1RO3Y5yWbLmm66fgaPqLMI360u7Th5dyyjpWxmR4ufK7Yl5gC6IZwL31m/mY2mhgnEYsz6WDuMoO4DQq9Xb9JpPzl9rvMzu9h4mWdQJZ2jxAgNC7ACTizaGGmTSC+pJzSVod9IS9s7pGejrovyROkU9vWQFImZY7RQK7XU47kBRY3oGwWgciN/SO9d3KqJJ4Yc23HTzJlEeHOFztsx0IBK4rWi5NYOL8LDoroKBS50YrfzHjTmfZlNy0KVTyBbb9X5TzLHkuPReGl6KbEN29PS2syohjAi8pWNXcjUJK85QQOwfMqvb78Ng5C9mwdYdvBNU9Wr3MZWfLogXIt8ct0AysGqExUdc6PgWgEEDJPAeQSroptalUqIv01vcYve4gw1lECsgAOWhs/pWBZjUlCdjCpkpn8ZzNEXh6KDgxCH+eadkuho7jTgz3zgJ/KjJZlV5isEz1FxV6QUkuu3tbcIr9PlLqoocMYnYbrA7kT84+qPmDmlIFDniE31yuWccgwdW6YvxIETqXvyZrt2Enyss4Bzab99qRiWCVJcegqg+HHdyHH4BQtvqIdhlNugxz1N4zoDFgUvVD0iSDqgz26J+h7C2Gzraww8Tm9f22p6paMt1r1/jDIEbMAeRpwxyMKYDdeHdZdJq2LemeDI8GT4ua5icKbyl0fI4FLcwIGlPkmXwWdcFV/cnJ8HF7ErWDMSHdUHnXRPYenWpNoOs8Wzzjafx62AwqOGoGyL0Z7SfG3dFArteY4lG7f3lLENXbh1yuNpntT/j5OMv0iK8eCD/99byMTu0BDSYAjC4BEbqLRGcJzBbxPcHBnjUOmYr6MJdtnEdArIDh2 1AV+kPcD ZhkkeAt13Zb+x8JR6DYci0nvU8S1MiuMxGO1YRkWBxOGzy6IwMs38UdEEtG3+HXxf5iI8cC69DW7vQJAiFFO//h2D9mwgg2w+qjX59dQ15KXaZlX0+HyLxkY9kH+aNNbKSHeCxJEOBysJyF5iwuI/14pjil0iOtqHfYayNDhVds1i5wR9MgshQe9Xeld8tU3vab+J84gY77r3uidVqcTIeH5dTseCnQe9aYShSCkcbhgdbJABEgNFIGpr9XR2m6R4R+AwZsl/TEsBzgISrYW6wm51rGConSZMLoOmXyEbSIf58/7IPkavlLXg4vKiwzJeooH6/XH/c3/yZTpo8B3KeKohRjjZZvO1D4UWlxJgog2hqhePAvLcesEL2RWy202DrXIsr4BR5A0jjR5MAcLUZiQfLqY/an4ox4jZdtl0YjVBhq8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the page reclaim code, we only track the CPU(s) where the TLB needs to be flushed, rather than all the individual mappings that may be getting invalidated. Use broadcast TLB flushing when that is available. Signed-off-by: Rik van Riel --- arch/x86/mm/tlb.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index e2a0b7fc5fed..9d4864db5720 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1321,7 +1321,9 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) * a local TLB flush is needed. Optimize this use-case by calling * flush_tlb_func_local() directly in this case. */ - if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) { + invlpgb_flush_all_nonglobals(); + } else if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { flush_tlb_multi(&batch->cpumask, info); } else if (cpumask_test_cpu(cpu, &batch->cpumask)) { lockdep_assert_irqs_enabled();