From patchwork Thu Feb 20 05:20:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13983321 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9FCA7C021B1 for ; Thu, 20 Feb 2025 05:20:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 749BA280298; Thu, 20 Feb 2025 00:20:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6F8D1280296; Thu, 20 Feb 2025 00:20:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 57814280298; Thu, 20 Feb 2025 00:20:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 2A498280294 for ; Thu, 20 Feb 2025 00:20:44 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id C989C1A0B57 for ; Thu, 20 Feb 2025 05:20:43 +0000 (UTC) X-FDA: 83139173166.04.602A9C6 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf11.hostedemail.com (Postfix) with ESMTP id E7FC04000C for ; Thu, 20 Feb 2025 05:20:41 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740028842; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=7Xkw5cOHbRQYx+GyGckI9nl+nSqwvdWlBwx/zaa9ipo=; b=pzPcdJ1EMLF+GM65L5UuINatZrSMgfHvkkAUZ0f9YxRHUjT8I58MGCzfOorMw2HHxO4xSM +Z0Jl+s+hjWU/DwlFxO3B5zqBfGPWICdYyrE/Z85HM0RN/szjdkqUCJbMZ20FDKJU/VuXS Cdzo9ZOGJsZIBhOx0fvV4Sgc8/cyarw= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740028842; a=rsa-sha256; cv=none; b=inAZmNwh+vBp91nEeo0kga6YlhVSYjwy4FryKaZP1Gat/O/MYuggAwOFIAJzW4RxDCSdjd ek9R5nWsEKYP+G1cOmeETc6JGD3fhhjHTNBVG+C+ypFpQn74I8m5yHRW3mqvtciWCmuTJi 84Luk4GruGPcgKuBayovL/Qpo8tK33M= X-AuditID: a67dfc5b-3c9ff7000001d7ae-b0-67b6bba6f406 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RFC PATCH v12 01/26] x86/tlb: add APIs manipulating tlb batch's arch data Date: Thu, 20 Feb 2025 14:20:02 +0900 Message-Id: <20250220052027.58847-2-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20250220052027.58847-1-byungchul@sk.com> References: <20250220052027.58847-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrGLMWRmVeSWpSXmKPExsXC9ZZnoe6y3dvSDZ6fFbSYs34Nm8XnDf/Y LF5saGe0+Lr+F7PF0099LBaXd81hs7i35j+rxflda1ktdizdx2Rx6cACJovjvQeYLObf+8xm sXnTVGaL41OmMlr8/gFUfHLWZBYHAY/vrX0sHjtn3WX3WLCp1GPzCi2PxXteMnlsWtXJ5rHp 0yR2j3fnzrF7nJjxm8Vj3slAj/f7rrJ5bP1l59E49Rqbx+dNcgF8UVw2Kak5mWWpRfp2CVwZ 85edZCpYLVTxYeVs1gbGw/xdjJwcEgImEu97FjDD2DO/TWAHsdkE1CVu3PgJFhcRMJM42PoH LM4scJdJ4kA/G4gtLBAsMenaXbA4i4CqxKfj58HqeQVMJdZd/wk1U15i9YYDYDYn0JwfM3rB eoWAat4tuMTUxcgFVPOeTeL6ud1sEA2SEgdX3GCZwMi7gJFhFaNQZl5ZbmJmjoleRmVeZoVe cn7uJkZg6C+r/RO9g/HTheBDjAIcjEo8vDNat6ULsSaWFVfmHmKU4GBWEuFtq9+SLsSbklhZ lVqUH19UmpNafIhRmoNFSZzX6Ft5ipBAemJJanZqakFqEUyWiYNTqoExoOvliaW8hz33Pz6v zD9x0r+mSVLV2nx/f349Ndd0tbjFZKNN7r9Pii2qjF4dev9q7IzC9n5OcXtVBS+vqf5tvvLe aY9/SF+vzzosk/Gk433kss8pfGG+hwuapeS01vLXSRu5+/ou/uFwcfI/JS7p7q+zeGQEXz8p DpCZULx3Neu9HzMKqrqVWIozEg21mIuKEwEGxIPeeQIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrLLMWRmVeSWpSXmKPExsXC5WfdrLts97Z0g1dn2C3mrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XhuSdZLS7vmsNmcW/Nf1aL87vWslrsWLqPyeLSgQVMFsd7DzBZ zL/3mc1i86apzBbHp0xltPj9A6j45KzJLA6CHt9b+1g8ds66y+6xYFOpx+YVWh6L97xk8ti0 qpPNY9OnSewe786dY/c4MeM3i8e8k4Ee7/ddZfNY/OIDk8fWX3YejVOvsXl83iQXwB/FZZOS mpNZllqkb5fAlTF/2UmmgtVCFR9WzmZtYDzM38XIySEhYCIx89sEdhCbTUBd4saNn8wgtoiA mcTB1j9gcWaBu0wSB/rZQGxhgWCJSdfugsVZBFQlPh0/D1bPK2Aqse46RK+EgLzE6g0HwGxO oDk/ZvSC9QoB1bxbcIlpAiPXAkaGVYwimXlluYmZOaZ6xdkZlXmZFXrJ+bmbGIGBvKz2z8Qd jF8uux9iFOBgVOLhffB4a7oQa2JZcWXuIUYJDmYlEd62+i3pQrwpiZVVqUX58UWlOanFhxil OViUxHm9wlMThATSE0tSs1NTC1KLYLJMHJxSDYyLDJd0m/osX1s1z2FRbiejiZPpyu//m6Qf b+s8WVDScbZqitCFH5t0O/bFyD5p/Z7PPHXx7NKgdqX1vdkXlfoCnLOV1ha4F6Z3Cz3Mvrq2 y+HEtc5DT2x/HtB5VTm3JC7j9X7nW9uXsn/YKxb55rmm75PMQyrCYlxSAsfc7+v2sKl0p5sv ilRiKc5INNRiLipOBAAe/tirYAIAAA== X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: E7FC04000C X-Stat-Signature: p8k3ac7pu9rz7hbnqx3jdrcxkf7he34y X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1740028841-147504 X-HE-Meta: U2FsdGVkX19rCB/fU80qziCo15ELkkV/UEYFRlzfLnyk/dK48hZaRhHjHaXwlrCI1WLY/tQJnVT3mADitPEt3AmIuU2/hgeQoWYiPllaAI6MMCgCuKGFjAK20hJ2Cy5j2b8Lyrbx1npdiYy1OMZgbW5fmLbv6TwU658B9zIgamR80YBdm7Ud7i4N88or8srudkC84C1TcEigGwNbfl9CoW0KqJSqy8Yae9GLB4zoxNKvSGq42CEt660UK5jgS1N6J8ygfOEl1oe2yiJ/M97o4Ft/Ae9H2cYwJRfSNpelEVrz4F0ON2VQXPGMx00UwmXZ5CbTblFzQwZvstxkopXT3UVyrb+Kgg5JYBO/nbqOXv8FAflXsP8awhkwMmIBJS/AWH9N35UHs3SrWIoLKRP51qmPjJXK7P+OupHXvevs4OzKlH+y1bSKaWPgcN5blHFcbLNORQXnqwEmkvAEgfeE2uqgLD2c1UgfDA6kLCB6MUD4isB584wcPdffJQAIPGhDUHOKWTm0HKTLjcK7QEgQZ34ZnYgBCEcXKB+qAkjqTypXdQa54OINn/YvF4CxwvjQjkvjL+SK91IuMe+TNm6dIxWNKP0QyjPVcN83ro0xsc1bvoQlpbyqWUwIrsOBRoOGWWuOKxcFP/8x41oeS81waXO6ZKwHhR4ChOxSn1ZU3dR3jkBS03cD/qlyPIle/FMwNTSX4m2ZJ0HTtD/z1JwDDgwgEqyUeIGDm4b9HSRuvg/OuzmtL457rLemi4d/0y7y+ym5BlyDVUVRjK2x66sQ7c1fLTssn+qQgz4IV4x+m0DSQe/zx/MvwzLhsK81cRoWKe6BvG4TF9N+2Br9ud6A4GP9KpbY92UijXeGGxPAMSACz0BHzqOOmn2GajlWqiSKv2vEpyP7IePuNbfFwI924u+d2207egkUURFXGVTx7v/RcAh3SZ7xdnOg0cWFRc1boG6peH7f5QBzEswOeFf A6+Y7AYC 0G0SroIOxQlGT9lmcHrEVSZ9Bqg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read-only and were unmapped, since the contents of the folios wouldn't change while staying in pcp or buddy so we can still read the data through the stale tlb entries. This is a preparation for the mechanism that needs to recognize read-only tlb entries by separating tlb batch arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed. It also optimizes tlb shootdown by skipping CPUs that have already performed tlb flush needed since. To support it, added APIs manipulating arch data for x86. Signed-off-by: Byungchul Park --- arch/x86/include/asm/tlbflush.h | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 69e79fff41b80..0ae9564c7301e 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -293,6 +294,29 @@ static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm) extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); +static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch) +{ + cpumask_clear(&batch->cpumask); +} + +static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + cpumask_or(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + +static inline bool arch_tlbbatch_need_fold(struct arch_tlbflush_unmap_batch *batch, + struct mm_struct *mm) +{ + return !cpumask_subset(mm_cpumask(mm), &batch->cpumask); +} + +static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + return !cpumask_andnot(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + static inline bool pte_flags_need_flush(unsigned long oldflags, unsigned long newflags, bool ignore_access)