From patchwork Fri May 10 06:51:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13660935 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 317F6C25B10 for ; Fri, 10 May 2024 06:52:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8CC616B0096; Fri, 10 May 2024 02:52:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8560F6B0099; Fri, 10 May 2024 02:52:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 524056B009A; Fri, 10 May 2024 02:52:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 279A26B0096 for ; Fri, 10 May 2024 02:52:25 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id BF80FC04CE for ; Fri, 10 May 2024 06:52:24 +0000 (UTC) X-FDA: 82101567408.21.68AF495 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf17.hostedemail.com (Postfix) with ESMTP id E0F544001F for ; Fri, 10 May 2024 06:52:22 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf17.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715323943; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=Ilpey0YHF3hm79X1vBIhnNe8zbnRWNG/60DUNWlxvcE=; b=GxWK+KpzcpoCa3rPMu9wTDSh5MsfmjHhQ8z+l8oVyMFwfTelkBT6PxsQb96bD5N5/6ZcWj Pbo54OpdFGkhdg55LT8aZ9QyJaBw35heWAibReVE236QolKWZEuzbRiGTSZq/1RCvdDJcZ VHLcEJ7zNYqS1nxGet7pucyJTPiYEsc= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf17.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715323943; a=rsa-sha256; cv=none; b=fJbOcxQLPIYK2UwwSBucN1PuNrhQIGw8plVVA7BrdeUQPg18Zc0/vatK9FIFO6dO0HGUS9 bcA6k+IYL1n96D4xezS95xQtQURvm+RDd/ejX8jQ2Ry8yTECJzg5D5m/789vJUwaBgBMlb +OMAdxPbBDtaOPp398fSwm1+taSexAw= X-AuditID: a67dfc5b-d6dff70000001748-ce-663dc42165ab From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v10 03/12] riscv, tlb: add APIs manipulating tlb batch's arch data Date: Fri, 10 May 2024 15:51:57 +0900 Message-Id: <20240510065206.76078-4-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240510065206.76078-1-byungchul@sk.com> References: <20240510065206.76078-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrGLMWRmVeSWpSXmKPExsXC9ZZnka7iEds0g3MP5SzmrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG 8nX3GAs+ClQc/beKvYFxGV8XIyeHhICJxJ0bjaww9vzjvxhBbDYBdYkbN34yg9giAmYSB1v/ sIPYzAJ3mSQO9LOB2MICQRL/bn0Ei7MIqEpc7r4GFucVMJX40/SEHWKmvMTqDQfA5nACzfmw bAYTiC0EVNPxYCLQLi6gmvdsEu9n/4BqkJQ4uOIGywRG3gWMDKsYhTLzynITM3NM9DIq8zIr 9JLzczcxAkN/We2f6B2Mny4EH2IU4GBU4uHdsdkmTYg1say4MvcQowQHs5IIb1WNdZoQb0pi ZVVqUX58UWlOavEhRmkOFiVxXqNv5SlCAumJJanZqakFqUUwWSYOTqkGRsbqeR9+rt6wsrS+ tiZp0caE2BNhIRPnnlY+4OUhJXZ7w0UJq81fhadflTkZIGZm51Nx9Eb8Ro7fM9pXs5eyZKxR WNHvKfZZQ0qatfyDmfLZjonFJWdDPLeu00n7t7n5Tcyu2++Tq0OCj/SL3Sn8wWb35uyGROa4 GbsUDlQffya542OWCVNqoRJLcUaioRZzUXEiAN+vQop5AgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrLLMWRmVeSWpSXmKPExsXC5WfdrKt4xDbN4OFhIYs569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZSxfd4+x4KNAxdF/q9gbGJfxdTFyckgImEjMP/6LEcRmE1CXuHHjJzOILSJg JnGw9Q87iM0scJdJ4kA/G4gtLBAk8e/WR7A4i4CqxOXua2BxXgFTiT9NT9ghZspLrN5wAGwO J9CcD8tmMIHYQkA1HQ8mMk5g5FrAyLCKUSQzryw3MTPHVK84O6MyL7NCLzk/dxMjMJCX1f6Z uIPxy2X3Q4wCHIxKPLw7NtukCbEmlhVX5h5ilOBgVhLhraqxThPiTUmsrEotyo8vKs1JLT7E KM3BoiTO6xWemiAkkJ5YkpqdmlqQWgSTZeLglGpgNNC4p2bbwPvpx8oDdYE3Pv7J/r9wzkeF 5vlcp60juLkDmWYfTPFY//KYhsOnQM0N3x6K3X7t+YfxwIS7CbvVW3Wv6jTu9VmlKLE9tLtk 350fvw/svB+meUdhW9a/qclF0pI9zNZ6q3XNDhZtzItqOc3g7bxR9XztgenrLN6KTxN2NVp5 4T7jSyWW4oxEQy3mouJEAMaRpBZgAgAA X-CFilter-Loop: Reflected X-Rspamd-Server: rspam01 X-Stat-Signature: 8rqzcu4exutusueybw7y77d78ne84sem X-Rspam-User: X-Rspamd-Queue-Id: E0F544001F X-HE-Tag: 1715323942-720471 X-HE-Meta: U2FsdGVkX1+CzbIqzwW8IWktpZ4/ADfSZxQreVenCf8swY3JoCsGSIkvnHnAaJu5YiAXchSOgBoztX7uh5daKsVLmlkEtv7kPnIrOCyep0CzEzbnwOsYyxbkeMzL/xf5q/uzj6Ws+AVaUtm4gb+hQpXGePmhZGvDV6VUho27OJom3EGJEstJ/bRsjT5NVVX6GuVoKBn3octdSDRj+4DUa4r+8fOZhQBZUat9ZRWadKF1Wo4wTydTip0gvOgN0uBIJtC8GYBON4t3gz7VYk2mJJqIv2DKJWNdDwUM81dQmFKju53mFSoHUSmo0nkTTxJRkanGocdimp3m/7nohgY+gG/aHbZlWkovqopv5PZFPS+lhFg7mx771cvx+w1va7o0HHBehnYPwmx0eiw8x/9oPhsV6k3gup0NNGOOpRTNYp08O40CypcA945hUfba8J4qf/wMnn+ocKXKXAKtpua929wn9smZZjGmd12Iv2/jxLKtTMhVjWnbJ1iqr8ypQKvWB+Tpw3tOpM3d97iWTpvlGahUA0YWNFeIRn+affLuytqXFLaHbMTMEErxqEThWVJZuNW0qXa/Wkvmiij7MWhl3TjHiVhnasEZ4kupBQl5jvpwN/mNT1N/uOUMEoa3JIpDHNG42SYpHgA6QqGO02oDw8WwBV45n/2UBYmu9E/KdsIOm7sTcHP7Dex6r3uv+4v6vtyCsKuZRm2DVvl+xG/Kv6L0j0Dm0ZvGF8sgIKqubCLik5WKgrlVbd8TeYrkLMnHPfckSI2ME8KSy0C1yb53pucUKyG9BVOiXt2GMOsFSIKOXVyInjAxU9/VRH8DFq9BE1lKCBn3kw03DkB41j6GyTVHd4FS82JkXAkvRx2OG13i1riNB6UjDmTps9RgIQaltCdkqVFfD8BrCFaG3Ndrv8SoBCDPxDh8fprI8faLCHsYurK3aTvS+z4Z9KcfZX56xNG7F1JnnQk5KjpTD7Y eQHB2fGX QVAASZ/vEvFF60bYpMm/Jbblkaw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries. This is a preparation for the mechanism that needs to recognize read-only tlb entries by separating tlb batch arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed. It also optimizes tlb shootdown by skipping CPUs that have already performed tlb flush needed since. To support it, added APIs manipulating arch data for riscv. Signed-off-by: Byungchul Park --- arch/riscv/include/asm/tlbflush.h | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 4112cc8d1d69..480c082ccde3 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -8,6 +8,7 @@ #define _ASM_RISCV_TLBFLUSH_H #include +#include #include #include @@ -55,6 +56,26 @@ void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, void arch_flush_tlb_batched_pending(struct mm_struct *mm); void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); +static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch) +{ + cpumask_clear(&batch->cpumask); + +} + +static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + cpumask_or(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); + +} + +static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + return !cpumask_andnot(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); + +} + #else /* CONFIG_SMP && CONFIG_MMU */ #define flush_tlb_all() local_flush_tlb_all()