From patchwork Mon May 20 02:17:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13667998 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8E04C25B79 for ; Mon, 20 May 2024 02:17:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 422276B0095; Sun, 19 May 2024 22:17:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2C0CD6B0099; Sun, 19 May 2024 22:17:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA2266B0095; Sun, 19 May 2024 22:17:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B15276B0096 for ; Sun, 19 May 2024 22:17:50 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 68321140854 for ; Mon, 20 May 2024 02:17:50 +0000 (UTC) X-FDA: 82137163500.19.D6F5E5A Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf21.hostedemail.com (Postfix) with ESMTP id 5E8A81C0008 for ; Mon, 20 May 2024 02:17:47 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf21.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1716171468; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=lYfFMbAEM1F2cBEc4miPa+Jx1WoZRiG7k2wKNFJlSqA=; b=OsAOw2PX1/GUsSrSnfv+B8duNMJFgBDBXsXTS5O4rdHCEuOJFP2ggtGciz8XHPoOnJpV+N 9Rh+QW2fE+Q0sencyAlQLZ4QrwXgvLfR7/HRwPE42sQjJ41XtU23r03nWq9RBFOfJ+TMYb LZYVommYmmE5+TtShsma+gQmPh9Jg2o= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf21.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1716171468; a=rsa-sha256; cv=none; b=AbRLCmCxbBe4om0NYbRXIPUlwn8d9zKQY6ASWPzcaXFFfxx+6vkPmnTtnfaAvklxlYQkCq 86NeOEkiSX0PIQoq5ZRBqEX4hmh+IDn741xhH9rlA0QlD5dO7ZT6lX06GGZsLh6q4itHVE XF41B/e/sjQsTO3EkY8oaZQdGuZhoVU= X-AuditID: a67dfc5b-d6dff70000001748-9a-664ab2c8f665 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v10 01/12] x86/tlb: add APIs manipulating tlb batch's arch data Date: Mon, 20 May 2024 11:17:23 +0900 Message-Id: <20240520021734.21527-2-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240520021734.21527-1-byungchul@sk.com> References: <20240520021734.21527-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrCLMWRmVeSWpSXmKPExsXC9ZZnoe6JTV5pBlP3WlrMWb+GzeLzhn9s Fi82tDNafF3/i9ni6ac+FovLu+awWdxb85/V4vyutawWO5buY7K4dGABk8Xx3gNMFvPvfWaz 2LxpKrPF8SlTGS1+/wAqPjlrMouDgMf31j4Wj52z7rJ7LNhU6rF5hZbH4j0vmTw2repk89j0 aRK7x7tz59g9Tsz4zeIx72Sgx/t9V9k8tv6y82iceo3N4/MmuQC+KC6blNSczLLUIn27BK6M k/0bmAreCFSceTqPvYFxDl8XIyeHhICJxLfry5hh7Dv7V7GD2GwC6hI3bvwEi4sImEkcbP0D FmcWuMskcaCfrYuRg0NYIEyia68kSJhFQFXi595/7CBhXgFTiT0nlSEmykus3nAAbAon0JQZ q3aygNhCQCU/jn4FmsIFVPOeTeLd/C4WiAZJiYMrbrBMYORdwMiwilEoM68sNzEzx0QvozIv s0IvOT93EyMw7JfV/onewfjpQvAhRgEORiUe3h2PPNOEWBPLiitzDzFKcDArifBu2gIU4k1J rKxKLcqPLyrNSS0+xCjNwaIkzmv0rTxFSCA9sSQ1OzW1ILUIJsvEwSnVwLh88623XLuf1xx+ 4nCgb8Zh2+6ieEm1B7kiuyeuc/+d+z8/7JJ8Q+duthmZ+aLBV988WFP8T7D4OrOS+j0nnbLX 6yLcdE/tOhPe/cXKct5bE0Mf7v7drku2aEVbiNmuv5F3mpv7+LcFh1ZdaZnuFlQseI2TrYxb qTb5R37So2eJu38LWj090KLEUpyRaKjFXFScCAC3hbI8dwIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrNLMWRmVeSWpSXmKPExsXC5WfdrHtik1eawap7OhZz1q9hs/i84R+b xYsN7YwWX9f/YrZ4+qmPxeLw3JOsFpd3zWGzuLfmP6vF+V1rWS12LN3HZHHpwAImi+O9B5gs 5t/7zGaxedNUZovjU6YyWvz+AVR8ctZkFgdBj++tfSweO2fdZfdYsKnUY/MKLY/Fe14yeWxa 1cnmsenTJHaPd+fOsXucmPGbxWPeyUCP9/uusnksfvGByWPrLzuPxqnX2Dw+b5IL4I/isklJ zcksSy3St0vgyjjZv4Gp4I1AxZmn89gbGOfwdTFyckgImEjc2b+KHcRmE1CXuHHjJzOILSJg JnGw9Q9YnFngLpPEgX62LkYODmGBMImuvZIgYRYBVYmfe/+xg4R5BUwl9pxUhpgoL7F6wwGw KZxAU2as2skCYgsBlfw4+pVtAiPXAkaGVYwimXlluYmZOaZ6xdkZlXmZFXrJ+bmbGIFBvKz2 z8QdjF8uux9iFOBgVOLh3XDbM02INbGsuDL3EKMEB7OSCO+mLUAh3pTEyqrUovz4otKc1OJD jNIcLErivF7hqQlCAumJJanZqakFqUUwWSYOTqkGxrx9Rwvbpp6yK+u6cTZo77fTcRGJemr6 F3U46vztUzd/qX3Z+HGV6pMryQ/fskm937CFtzHfKl4+8gRPd2im8Gv+unOOX+78boi3Yiwp kdr78+/N/cEHf0rvi26Iu1b22/nEQ+ZN36OYkit5O35FzjcqUy3dZChcmLX3z3KZg5F+OxJX XlSfqsRSnJFoqMVcVJwIABWbBh5eAgAA X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 5E8A81C0008 X-Stat-Signature: kgkm9bduxp3ky3ijjta7ehs158maz3m1 X-Rspam-User: X-Rspamd-Server: rspam04 X-HE-Tag: 1716171467-879970 X-HE-Meta: U2FsdGVkX1/E+CdDgY4vLjVEYxNE2o6OkQiWFfaPpaeuWmr7Oqe6BNpqinPMV51FYxx1XTko+WI3LNSmK7yl/ife8IbPduE4aWyC3QX9Cw0UeBWxSFPzFy/Z0cCDs3ULjH2eyxpJIxACRGP3DHH8sBwZNnBicOB+eZ1bLCXTfymmjBtwB8C71Tp/qVGxLfcoKKBjyj7xBVe2VBwgYBsXWg8HhvApxFk9/0wqsFDyQOkEqB1SAadxyk6h0KwNdUlhLoMQDe/QzZNC+mEGEdJoKW3Wi28f0oidJjI8wrnL62+uuPP724PtEf2kMrr1bANgvmwILw8roO69Y1sZ8SnmFL5nQHfPPYD5iJxNOlkTvpcFCVVgVblNHE3V28v/Kam0mR914r/6VaOeieaFrd5RlIGMHbe6bV4FQFZplnW4UW5b96ix14SwYnmQ7eAbVij4C5YshIKv/ClbPL13Zv/qbvxUz59BgXnDdhnh/kHK6G2umyA+0s8vNECdoZ59657S/7G8pg5NnS21hq3xQFwwKg58+zOz/sOZzb8atx7hoRKCYqHiPGDHc8Uvnv8vfytCUYWhdWv2UOD5B9pNsRAgYfpVlabd0Hlp3sLy2IMo0mMYPbVPOvGelO7pbuEaSfnGGQa4Mt1FzqcUqwoTwTuRABkNvsQRTtIYDKWabsnNXkwVOg0yA7UV4ngWemvHCZuNSI2zvUYvKOOtjeFunAnnoieKnZneZhqkrnPZTxRCe3b3LRP7GuRVPiFQB54B6+R9LmjZlBcAdQ0pp1ackBaWPpyBWwFhYQJm8E6m24mEInEQ1nXY+8dMol54K+vrC8SJeRcN9uZOr9TJG28KL273yNwap6ucqnerZQNrs7kNPRtJTeHLFljYT8OW868yI7PJwBTsfAIGTnM41nofmrlO6yIiXPYK38OhneiupMfgNtnjQ2YvwKDcgApx9DnglVt6AFZCz/of8nTQF1/aLyA NfiQpTk7 1miiw7WGJq8Cnq+vxtKk5900B6Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read-only and were unmapped, since the contents of the folios wouldn't change while staying in pcp or buddy so we can still read the data through the stale tlb entries. This is a preparation for the mechanism that needs to recognize read-only tlb entries by separating tlb batch arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed. It also optimizes tlb shootdown by skipping CPUs that have already performed tlb flush needed since. To support it, added APIs manipulating arch data for x86. Signed-off-by: Byungchul Park --- arch/x86/include/asm/tlbflush.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 25726893c6f4..a14f77c5cdde 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -293,6 +294,23 @@ static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm) extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); +static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch) +{ + cpumask_clear(&batch->cpumask); +} + +static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + cpumask_or(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + +static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + return !cpumask_andnot(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + static inline bool pte_flags_need_flush(unsigned long oldflags, unsigned long newflags, bool ignore_access)