From patchwork Mon Feb 26 03:06:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13571186 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6D78C54798 for ; Mon, 26 Feb 2024 03:06:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EA4C16B0143; Sun, 25 Feb 2024 22:06:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D82AC6B0146; Sun, 25 Feb 2024 22:06:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC81C6B0145; Sun, 25 Feb 2024 22:06:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 983436B0143 for ; Sun, 25 Feb 2024 22:06:31 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 68E37A07D9 for ; Mon, 26 Feb 2024 03:06:31 +0000 (UTC) X-FDA: 81832466982.02.CAE71CC Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf05.hostedemail.com (Postfix) with ESMTP id 45AFA10000A for ; Mon, 26 Feb 2024 03:06:28 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=none; spf=pass (imf05.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708916789; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=eRD1/8o+qtEOTAKMY7ucioDJJtMI3+WpTL+ASvHX4To=; b=GVnlKyC9+hdR5dbMeaVG+LiNIgAVWbma9qZYIdIe6YH0gnoml6RdsamFKP98hhu7V34j/K uA+c4JHm4m9Z30n3uNPwkh2Y2qMaOoYRS4oNBRqBRg/ZzNrs6Hci05r+KHOKG/czRQHfj8 BHLQz9zPWGMYoXhd1Cs899t6J159F68= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708916789; a=rsa-sha256; cv=none; b=WZvkpo9cgarXy8T0oeuxk6PK1/pDrNjUKfUv8kaR4QBFq/UDDzEtaj2l9INQgr1cr6psWl Rk3tptYPMbp7UT1AT2k+bf+xLN7P+tjyKTwXjO+7Rz+MGT40BrCTC/mtKXiHQBohboZnHp hyEy0VNRY2moCWDrOEeRX1TDPi8Nmk8= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=none; spf=pass (imf05.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none X-AuditID: a67dfc5b-d6dff70000001748-27-65dc002fde00 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v8 1/8] x86/tlb: Add APIs manipulating tlb batch's arch data Date: Mon, 26 Feb 2024 12:06:06 +0900 Message-Id: <20240226030613.22366-2-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240226030613.22366-1-byungchul@sk.com> References: <20240226030613.22366-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrKLMWRmVeSWpSXmKPExsXC9ZZnka4+w51UgxdrDS3mrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG 7dsvmQpW8VUs+dHO1sA4jaeLkZNDQsBEYvfzC0ww9tqN61lBbDYBdYkbN34yg9giAmYSB1v/ sIPYzAJ3mSQO9LOB2MICwRIvLqwCq2ERUJXYf+QMI4jNK2Aqsf7lfKiZ8hKrNxwAquHg4ASa 8/6/IUhYCKjkb+9VoHIuoJLPbBJP3k9lhaiXlDi44gbLBEbeBYwMqxiFMvPKchMzc0z0Mirz Miv0kvNzNzECA39Z7Z/oHYyfLgQfYhTgYFTi4V3w4XaqEGtiWXFl7iFGCQ5mJRHecJmbqUK8 KYmVValF+fFFpTmpxYcYpTlYlMR5jb6VpwgJpCeWpGanphakFsFkmTg4pRoYuVz6bB9Jzriy X8VMq9yz6u+fJIHjl14utgvvvFLz+vG1NI32YqXOFV2vJ50/LLDgY+mTsIK2WZfiXQ9Ylqms VXUwXRJdkpDhGPPLS7Y//Rjzt6/fqvSSU7dythmc7Z21dKH+x6TGK6rHZ768+PTCrp+Xt1SI 7mS0ckp/fjTwS0lLMWtOttYjJZbijERDLeai4kQANQUZ3XgCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrDLMWRmVeSWpSXmKPExsXC5WfdrKvPcCfV4EKrusWc9WvYLD5v+Mdm 8WJDO6PF1/W/mC2efupjsTg89ySrxeVdc9gs7q35z2pxftdaVosdS/cxWVw6sIDJ4njvASaL +fc+s1ls3jSV2eL4lKmMFr9/ABWfnDWZxUHQ43trH4vHzll32T0WbCr12LxCy2PxnpdMHptW dbJ5bPo0id3j3blz7B4nZvxm8Zh3MtDj/b6rbB6LX3xg8tj6y86jceo1No/Pm+QC+KO4bFJS czLLUov07RK4Mm7ffslUsIqvYsmPdrYGxmk8XYycHBICJhJrN65nBbHZBNQlbtz4yQxiiwiY SRxs/cMOYjML3GWSONDPBmILCwRLvLiwCqyGRUBVYv+RM4wgNq+AqcT6l/OZIGbKS6zecACo hoODE2jO+/+GIGEhoJK/vVcZJzByLWBkWMUokplXlpuYmWOqV5ydUZmXWaGXnJ+7iREYxstq /0zcwfjlsvshRgEORiUe3gUfbqcKsSaWFVfmHmKU4GBWEuENl7mZKsSbklhZlVqUH19UmpNa fIhRmoNFSZzXKzw1QUggPbEkNTs1tSC1CCbLxMEp1cA475ylarTBfS6DxKkLHoZMUPNqZ6s3 WRz5f/LmNdb/N0pH8RdsrHrgPX93wpIshQkpPFs3NSscv3LOdKJmk32G8v6Ie4zpVY93Fs+p VWzeqnklMXD2Go0du2Zfr86Snpe0Lv3CTd5bydZLFm+Ry/BRTzY/NyfwyaOXi75ZlzX8/qAU UhH7KyVciaU4I9FQi7moOBEA6KTWhl8CAAA= X-CFilter-Loop: Reflected X-Stat-Signature: ynb8q6bkhe6itwg6u5imyk7n7hzuk7po X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 45AFA10000A X-Rspam-User: X-HE-Tag: 1708916788-95388 X-HE-Meta: U2FsdGVkX1+9h2s+SCgwMg/vHXxzcQgU1uVJt+0vsQt7LsC9X7i7KxiCA1tkKEAjt9Xq/VF0fgWUBX27WMw0bvF9y1IO9QBni/N0Q6bbzJuS3enunTNU7nWLK7WPHtODP05AgR5UhQQUm9BiyJpGnctAbakkxFL10Wsh7rtbc0SA0/iYwS8UE+16PQXDIuF48oEl4F8RUgUyR7qGyJg8w0Km6tGcOxFuSC5VN3ssvEyRMXXfCr/P7YtAHi1NX9b976O1C6ZLOPlW/cssds2/w/8eDcdS5IBEAZlhEQG++pktQ/ri00GfVutwSrQV/kHAiM6RTbM4S1GnY7WJ3xk0VQjOYJTKPwKWPKUUNK96t/Mtfor92GBkj5Y2v4FsvG/sBvej6PV8vq5aGcz64yqPaIXf6Q4g+qSMYCn1MbSKFM/zJMTc8WDRMH7fUBcCOQ2Ac2yaM3CG4x5yVTFnvRDcbh094Yg2nlpvoNhKZUGJlFvsQNfVqWJMmzIMjnVkX73o4fyzJtR+y0clpUQ5Twu0Fw/2tr9Y/HaDkABOTl0UtCpv9erbfV7evlrO/z5FuJZg+/bLFMFHfsUuTsalkaCUpjKP5LvnhfMTsyFpC0uJ0q4ZF1KSRjvw4v0Ub8mzFGr96fAzvm7mMpvkm4k4ncr7ThFjSv9lyCCwk9J/Sgw2yekQ+nqSxNbEitj0zOhaACLGfym9DBkRvlZSAz+LSolINpTRHSnRWsSMLfj67YRODG+5w2i9keH9Z1D5zq5v/B87irwrS4ovg2/RTwdju/ETiw2OivsTsqx82pbADCn1tpeNw209i1HAHxMtr5F5IpcvjNVM8b/cL2wMsglGVXHQO4TFuFx4MF0pZUNuCe+Z0MNCnz3e0TobfyVDaE4rlF5fcNc8Ndkmvky3uUm0MBplgGVm327NB4tRjjYTubdNWMnPSns4dioovFhAuaCxYjXwqPwa2A6uNGQTgD+iWEw M/Q0yUmw RAC6nelzrcBj34qrqKYWxwRWprA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparation for migrc mechanism that needs to recognize read-only TLB entries during batched TLB flush by separating tlb batch's arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed. Migrc also needs to optimize CPUs to flush by clearing ones that have already performed TLB flush needed. To support it, added APIs manipulating arch data for x86. Signed-off-by: Byungchul Park --- arch/x86/include/asm/tlbflush.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 25726893c6f4..fa7e16dbeb44 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -293,6 +294,23 @@ static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm) extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); +static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch) +{ + cpumask_clear(&batch->cpumask); +} + +static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + cpumask_or(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + +static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + return cpumask_andnot(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + static inline bool pte_flags_need_flush(unsigned long oldflags, unsigned long newflags, bool ignore_access) From patchwork Mon Feb 26 03:06:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13571185 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50023C54798 for ; Mon, 26 Feb 2024 03:06:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC03C6B0142; Sun, 25 Feb 2024 22:06:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C70E26B0143; Sun, 25 Feb 2024 22:06:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A74F46B0146; Sun, 25 Feb 2024 22:06:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 831C36B0142 for ; Sun, 25 Feb 2024 22:06:31 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 663DDA07BF for ; Mon, 26 Feb 2024 03:06:31 +0000 (UTC) X-FDA: 81832466982.02.9488C1A Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf30.hostedemail.com (Postfix) with ESMTP id E1E188000E for ; Mon, 26 Feb 2024 03:06:27 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708916789; a=rsa-sha256; cv=none; b=YR7vLweDHgf7A7whEqx3KUXGmqPVWwLQA2DCXIZzvUiAVjwlkWjLzKEc534jcbrSGcVG6d bt0HjMdE+8ZGeuzAjQP/GqK4vxjbecmYPFMH5G8YkLrIidN8Qu4J1QH67/f9yxg1hue8X2 aG6pi0WnGs5vRsdrXR42Xz1argEjWSg= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708916789; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=23+JjFD9uRPbyMWGqJTLiQYjg6Gkh8EQ/TB1jpahIpQ=; b=NqNl/A+FGfZb93pH6X1dslol8XoifS5LtdUBCW5WZNGeY/hFlwPlQjXAxY3O67kFfADdf6 T46ru44+YJBiB9eNfES39S9buw/LFfvuFugm/DZ7oQpdSd8dD9oESJOFph+pnwkrU5duuS OFM9MizkcaCLenRMQ0xrC9SE7WQ4N6k= X-AuditID: a67dfc5b-d6dff70000001748-2d-65dc002fe326 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v8 2/8] arm64: tlbflush: Add APIs manipulating tlb batch's arch data Date: Mon, 26 Feb 2024 12:06:07 +0900 Message-Id: <20240226030613.22366-3-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240226030613.22366-1-byungchul@sk.com> References: <20240226030613.22366-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrCLMWRmVeSWpSXmKPExsXC9ZZnoa4+w51Ug5Z+c4s569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF51xw2i3tr/rNanN+1ltVix9J9TBaXDixgsjjee4DJYv69z2wW mzdNZbY4PmUqo8XvH0DFJ2dNZnEQ8Pje2sfisXPWXXaPBZtKPTav0PJYvOclk8emVZ1sHps+ TWL3eHfuHLvHiRm/WTzmnQz0eL/vKpvH1l92Ho1Tr7F5fN4kF8AXxWWTkpqTWZZapG+XwJWx p+8he8FsropN79vZGhj3cnQxcnJICJhI3Ph3i6mLkQPM/jTZByTMJqAucePGT2YQW0TATOJg 6x92EJtZ4C6TxIF+NhBbWCBa4umBvWA2i4CqxI9ZE8FsXgFTict/D7BCjJeXWL3hADPIeE6g Oe//G4KEhYBK/vZeZexi5AIq+cwmcX7rOkaIekmJgytusExg5F3AyLCKUSgzryw3MTPHRC+j Mi+zQi85P3cTIzDsl9X+id7B+OlC8CFGAQ5GJR7eBR9upwqxJpYVV+YeYpTgYFYS4Q2XuZkq xJuSWFmVWpQfX1Sak1p8iFGag0VJnNfoW3mKkEB6YklqdmpqQWoRTJaJg1OqgbH8WUAjB4/K pUTN84drxJa9qvKJkwto/LJfe9OESZlTM89eUDnZtPN+8MVLv53und/BNE+FaWPktqmzWvLm P5cJrHsYckXSVjpGtkQuM0ZQgSHq+s1tMQptAl9nni1YbyNauPLK04TrlnOLvz4wtLp8yHWh kqz6jhusn3ii57kJiHndyyg5+kWJpTgj0VCLuag4EQCxiRetdwIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrNLMWRmVeSWpSXmKPExsXC5WfdrKvPcCfV4Nh/bYs569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZezpe8heMJurYtP7drYGxr0cXYwcHBICJhKfJvt0MXJysAmoS9y48ZMZxBYR MJM42PqHHcRmFrjLJHGgnw3EFhaIlnh6YC+YzSKgKvFj1kQwm1fAVOLy3wOsILaEgLzE6g0H mEHGcwLNef/fECQsBFTyt/cq4wRGrgWMDKsYRTLzynITM3NM9YqzMyrzMiv0kvNzNzECg3hZ 7Z+JOxi/XHY/xCjAwajEw7vgw+1UIdbEsuLK3EOMEhzMSiK84TI3U4V4UxIrq1KL8uOLSnNS iw8xSnOwKInzeoWnJggJpCeWpGanphakFsFkmTg4pRoYbZdFV7/+3CLisHb1f6N//JvnLtg7 8fL8E5WxHUc2Xb/Vk1retuX/8+tGqjkl/moNHc/2Hn69e8evYI6NrVU8Oj+sXV7y9r0VMQo2 OqHe2u1qXuQqz/7u5OoL/wXmfNh/+pX1I6Xn7SUB8oX37i9q16jYrT3137HYE1/aNRU8LKOu Png5ZWb4ZyWW4oxEQy3mouJEALEqOqReAgAA X-CFilter-Loop: Reflected X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: E1E188000E X-Stat-Signature: pzrj57ijtzy4c5cub63g3rgadte5fpz3 X-HE-Tag: 1708916787-651838 X-HE-Meta: U2FsdGVkX18pC/cvCtTUTAP9TXTbIFd3DLvCI+911PiDynk3WoF8NDy0GNwNInJcVD+0ctRstldT8DKLRtryqM0+aUXfzGmX7XpYaaAPpZHzAUd5dCCJrXZXBQL3OSeVD0rFjoe6SMoP2jLi5lgiczheb4AgM7yCB7jycmzKvb5hrAvrGCYkEiKfDH0xL10ScUrlKTgMgRBWSd9rGmbwY2ObcH+p3rGpEHCm6cDVLZBNGk/Lg+nuDh8C0a4QL2fJ+4ts9xInbRNrznLdSDg/aqcr+/gBgWNy/VsrEqDm7DCzpytL55qV0jIT/LmCgcsWxKTO1Yb0pVm3XkMzd/q5PVzAVrl3IcgCsKzciVr7jP2ShrtSTX4x7l36sNO1+s3glZHGLwr36Y8AHM8zqCqEifslmyzUiFX3pCRU4cgpiT/bimMg5Bkl/pGpX710Dh1i4l1PS79x1wSbsFUKTew3xXvhM6WweROLdZtAB8YQFeI0WaHzV3tRmDL+U7gAcds+aSdcMsYc0yDqutKK1jc9nY9SHEFTy5iquY/vQ+V6bb+hMoo9yYRg9bs9vTifh22mUAzbXH48tLLtPjy8ypyywVlSLwF/IYRgm035k2jI8PVnojk3NTABmGSBQLzIb6Yte6dsKlKgDK3a76C2zn5AqHTny4iAaAuIm01AUnO7+1sFO32Y9YPN+ciF6WbZ3Zi7FN0lE/z5KQtxh44LPrXDIxIgOvAsocKRk7ON7QdaPTRW4ydRi66gW/qBzXm0UfBewqkP1evDUmpCiMman9Kv6DjBXatm1pdRMRRyfJNspHUk62I2himWHwdGp77s3nPhDF7I49XksnpbfqGdfuyiB6SG0o3Ly+Vw3bz6mrPyHLpo/F84HPZXsBXcX2qflAah3CM6Ic1oaRZ678vWG3mqBhOJ9xlA6vQhK+YeIjkYDhnOueN1jHbzVNiKSMIa8lRhnkf8u7hKqm8BXibmxyr pOA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparation for migrc mechanism that requires to manipulate tlb batch's arch data. Even though arm64 does nothing with it, arch with CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH should provide the APIs. Signed-off-by: Byungchul Park --- arch/arm64/include/asm/tlbflush.h | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index bb2c2833a987..4f2094843e7a 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -328,6 +328,25 @@ static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) dsb(ish); } +static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch) +{ + /* nothing to do */ +} + +static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + /* nothing to do */ +} + +static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + /* nothing to do */ + + return false; +} + /* * This is meant to avoid soft lock-ups on large TLB flushing ranges and not * necessarily a performance improvement. From patchwork Mon Feb 26 03:06:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13571187 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B37DC47DD9 for ; Mon, 26 Feb 2024 03:06:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A796D6B0146; Sun, 25 Feb 2024 22:06:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A29676B0148; Sun, 25 Feb 2024 22:06:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C83B6B0149; Sun, 25 Feb 2024 22:06:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 70D916B0146 for ; Sun, 25 Feb 2024 22:06:33 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4C784C06C6 for ; Mon, 26 Feb 2024 03:06:33 +0000 (UTC) X-FDA: 81832467066.23.2885B32 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf01.hostedemail.com (Postfix) with ESMTP id 8A98340005 for ; Mon, 26 Feb 2024 03:06:31 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708916791; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=deHIYqkuaMMowWAwSsScVOmIndLCnwrqbZwxQiYz+QA=; b=HPy/9qEA10DFpzl9azLs/Q1QCYJ178mgmdZA6S8Fw+fp90RykPA2SOjHQkdtZYXMxbNrk+ peGVTqncBE+LeESnI4xvb7JV/1QAdwwcpfBK5F6h411sJt+wpWUkg/qO6Et7xOW2EsuzYx gwbfVWPjxYgEQgkSB/cUa9Pjc9Xwk+o= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708916791; a=rsa-sha256; cv=none; b=bi/aQrX6VmhZm+1p9+g56rBn4YD5SKwFx1Wjlkdn6r+4da8q2LdmnonIr/sVbOHSxez5ZS 7QUxs9ONFf+rXQK9rISarSj60m0pBx6ExhcNGeQSTpE9qirw2iZwvJoSXbXr8nD14x/ugw te9XDLf2x3ZJL9zet2K5F1HdjGC746w= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none X-AuditID: a67dfc5b-d6dff70000001748-32-65dc002f6be1 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v8 3/8] mm/rmap: Recognize read-only TLB entries during batched TLB flush Date: Mon, 26 Feb 2024 12:06:08 +0900 Message-Id: <20240226030613.22366-4-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240226030613.22366-1-byungchul@sk.com> References: <20240226030613.22366-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrCLMWRmVeSWpSXmKPExsXC9ZZnoa4+w51Ug8NXLS3mrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG 13lOBcelKz582sHYwPhArIuRk0NCwETiy66TrDD2mZe/GUFsNgF1iRs3fjKD2CICZhIHW/+w g9jMAneZJA70s4HYwgIJEg+nzWMCsVkEVCUOn4Go5xUwlTg8YSHUTHmJ1RsOAMU5ODiB5rz/ bwgSFgIq+dt7FWgVF1DJezaJE38OMELUS0ocXHGDZQIj7wJGhlWMQpl5ZbmJmTkmehmVeZkV esn5uZsYgWG/rPZP9A7GTxeCDzEKcDAq8fAu+HA7VYg1say4MvcQowQHs5IIb7jMzVQh3pTE yqrUovz4otKc1OJDjNIcLErivEbfylOEBNITS1KzU1MLUotgskwcnFINjDVc17fHpHDM1vip 1c0s8nhT0EpZt02LFl3InXXdeEMsC/f+t82Fu1aonZkVb8v5j6vsV57bUY6Fwc8sGdZGfbZf 67JQoGOSxJMTjK/S174xDou/mSInM0+fadZjNdZpYaq7p9u0lO65sqdb2MLRuNIzdU/+LsUa jfOWLlrblJwULwpN0GEMUWIpzkg01GIuKk4EALCYTpR3AgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrNLMWRmVeSWpSXmKPExsXC5WfdrKvPcCfV4PZ3PYs569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZXyd51RwXLriw6cdjA2MD8S6GDk5JARMJM68/M0IYrMJqEvcuPGTGcQWETCT ONj6hx3EZha4yyRxoJ8NxBYWSJB4OG0eE4jNIqAqcfgMRD2vgKnE4QkLWSFmykus3nAAKM7B wQk05/1/Q5CwEFDJ396rjBMYuRYwMqxiFMnMK8tNzMwx1SvOzqjMy6zQS87P3cQIDOJltX8m 7mD8ctn9EKMAB6MSD++CD7dThVgTy4orcw8xSnAwK4nwhsvcTBXiTUmsrEotyo8vKs1JLT7E KM3BoiTO6xWemiAkkJ5YkpqdmlqQWgSTZeLglGpgDLfQf3L+fFbP6oyusxLzttvzPdJcJr97 ScDHgA/qEcmONkwr7/z9cOH7/fm1pZfvz/sgFbvYeTG3S/t1QbmXp82FA56ceN7pYWzxU3fH /6YgsQkdSlmvwkVXvJ8iMtPoStaDkj7+locqHgmzLln5O58/HNJVepZVq0Q2YNnvJ8/cDc5M Wqv7U4mlOCPRUIu5qDgRANgmFc1eAgAA X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 8A98340005 X-Rspam-User: X-Stat-Signature: s198urannygcrkrjb5m7386k4hetu7ns X-Rspamd-Server: rspam03 X-HE-Tag: 1708916791-520159 X-HE-Meta: U2FsdGVkX1+ymFloTNiKXCkYvn8QftL//Xq6KrYHg6Hm+oW/IN1vEBoG65HT/mvuubf1lC6m5pdMzrMuf8qHenZlMTOk4GDiAl2f77FgFG7BsoOwN4HsRy4wRwVh+n9Qo2GFiqsjJf3chgYOwz0WJmpv9+G0wmqBmoph4hpFrgPyJN8jXcpiNJ3YzJ42mjJJs6nb4UYxGIQWdNI1+D6TRQXGLq0udCOhe0k3esJLNaTk+/S/TRpVwB36DomggfFpOvasDcchIpg4vAPuWxF0wR6S5B2Wy/PQZPKxoHuvOTFazsJkvcJ+p/FFoLL+dstqKqu6WrXylJptSPNDdMji5MZ/q9Ajv0GN5PDHuqTZtbiv3GYl0UX19M9BrHwIf9lVNG4CfJ8RLSqSTqc++PkxKWqJmeUgNVirqT4ODl+aHjTmoSQ3Zh2cJbinztuKebWPAKMT2rXOskFPhOYrX0Y69oi0N71hhmSFbE6ExnTVxwIx5NGfsx+tC5T1g3eLrrkLMI6iGq/x+tNT6chYUI6AUcKiTElGrJzAqmx8kEpBvEVsG93vrtVJTuNzY8GHCxFRQPJvb0yA135bDC3SfQHXUMUfSJgmLvgjIBya/708nHYPUwp/TS6jCS2rg4Xj7jRI36UggVejKONkwIGzWfYbxbLXfb5DxPtpNngi1HzeNvjpAbnTx5l5gowNEREOgY9l4cjtpFLkda5FArJMEcrfgg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Functionally, no change. This is a preparation for migrc mechanism that requires to recognize read-only TLB entries and makes use of them to batch more aggressively. Plus, the newly introduced API, fold_ubc() will be used by migrc mechanism when manipulating tlb batch data. Signed-off-by: Byungchul Park --- include/linux/sched.h | 1 + mm/internal.h | 4 ++++ mm/rmap.c | 31 ++++++++++++++++++++++++++++++- 3 files changed, 35 insertions(+), 1 deletion(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 292c31697248..0317e7a65151 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1328,6 +1328,7 @@ struct task_struct { #endif struct tlbflush_unmap_batch tlb_ubc; + struct tlbflush_unmap_batch tlb_ubc_ro; /* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe; diff --git a/mm/internal.h b/mm/internal.h index b61034bd50f5..b880f1e78700 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -923,6 +923,7 @@ extern struct workqueue_struct *mm_percpu_wq; void try_to_unmap_flush(void); void try_to_unmap_flush_dirty(void); void flush_tlb_batched_pending(struct mm_struct *mm); +void fold_ubc(struct tlbflush_unmap_batch *dst, struct tlbflush_unmap_batch *src); #else static inline void try_to_unmap_flush(void) { @@ -933,6 +934,9 @@ static inline void try_to_unmap_flush_dirty(void) static inline void flush_tlb_batched_pending(struct mm_struct *mm) { } +static inline void fold_ubc(struct tlbflush_unmap_batch *dst, struct tlbflush_unmap_batch *src) +{ +} #endif /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ extern const struct trace_print_flags pageflag_names[]; diff --git a/mm/rmap.c b/mm/rmap.c index 7a27a2b41802..da36f23ff7b0 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -605,6 +605,28 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, } #ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH + +void fold_ubc(struct tlbflush_unmap_batch *dst, + struct tlbflush_unmap_batch *src) +{ + if (!src->flush_required) + return; + + /* + * Fold src to dst. + */ + arch_tlbbatch_fold(&dst->arch, &src->arch); + dst->writable = dst->writable || src->writable; + dst->flush_required = true; + + /* + * Reset src. + */ + arch_tlbbatch_clear(&src->arch); + src->flush_required = false; + src->writable = false; +} + /* * Flush TLB entries for recently unmapped pages from remote CPUs. It is * important if a PTE was dirty when it was unmapped that it's flushed @@ -614,7 +636,9 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, void try_to_unmap_flush(void) { struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc_ro = ¤t->tlb_ubc_ro; + fold_ubc(tlb_ubc, tlb_ubc_ro); if (!tlb_ubc->flush_required) return; @@ -645,13 +669,18 @@ void try_to_unmap_flush_dirty(void) static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, unsigned long uaddr) { - struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc; int batch; bool writable = pte_dirty(pteval); if (!pte_accessible(mm, pteval)) return; + if (pte_write(pteval) || writable) + tlb_ubc = ¤t->tlb_ubc; + else + tlb_ubc = ¤t->tlb_ubc_ro; + arch_tlbbatch_add_pending(&tlb_ubc->arch, mm, uaddr); tlb_ubc->flush_required = true; From patchwork Mon Feb 26 03:06:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13571188 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72AD7C54E41 for ; Mon, 26 Feb 2024 03:06:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 34CB56B014B; Sun, 25 Feb 2024 22:06:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3089D6B014D; Sun, 25 Feb 2024 22:06:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 19E8E6B014C; Sun, 25 Feb 2024 22:06:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id ED1626B0148 for ; Sun, 25 Feb 2024 22:06:33 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C526E8067C for ; Mon, 26 Feb 2024 03:06:33 +0000 (UTC) X-FDA: 81832467066.24.7B746F8 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf05.hostedemail.com (Postfix) with ESMTP id EC7F710000B for ; Mon, 26 Feb 2024 03:06:31 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=none; spf=pass (imf05.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708916792; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=JD5mtfEWkRjuaUV9XVtob8Xf345yQ1B8wbzU19SAuqU=; b=vzLJ4DZA9KptWHIOojwge3xrpXCZ8W3anTTOE1VGknceXW3H/+r6iC63zSLwKomPSZh5ka S4EODhDyDAOBPE3qjZisefC0bpfdeQqJGrgzEtxgnNFwpu2JG2PpF5Y6kmOBRP2UtUAmGM YnNtWNROh5L0LPDBez5qWX8e4soWR+4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708916792; a=rsa-sha256; cv=none; b=KXTuwm4OkLGJ0v1CBx1QrPqGR1QmhJhh0F4mAl6lyeh4qCikihU/c4VF4uQgk90MG33nff n9SC1BJepBE4E7IHuMsHSyrUgM9hXGAogMvL6IrJEEMzlTZu8u2YE0OvXkCHk7uyyLG+yp rqRD8W6dg+WCd1MFhwB6x608EzQ1T0s= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=none; spf=pass (imf05.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none X-AuditID: a67dfc5b-d6dff70000001748-37-65dc002fa99c From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v8 4/8] x86/tlb, mm/rmap: Separate arch_tlbbatch_clear() out of arch_tlbbatch_flush() Date: Mon, 26 Feb 2024 12:06:09 +0900 Message-Id: <20240226030613.22366-5-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240226030613.22366-1-byungchul@sk.com> References: <20240226030613.22366-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrMLMWRmVeSWpSXmKPExsXC9ZZnka4Bw51Ug69hFnPWr2Gz+LzhH5vF iw3tjBZf1/9itnj6qY/F4vKuOWwW99b8Z7U4v2stq8WOpfuYLC4dWMBkcbz3AJPF/Huf2Sw2 b5rKbHF8ylRGi98/gIpPzprM4iDg8b21j8Vj56y77B4LNpV6bF6h5bF4z0smj02rOtk8Nn2a xO7x7tw5do8TM36zeMw7Gejxft9VNo+tv+w8GqdeY/P4vEkugC+KyyYlNSezLLVI3y6BK+Po nXtsBXM4K87/msrUwHiJvYuRk0NCwETiy4WXrDD2i387wOJsAuoSN278ZAaxRQTMJA62/gGL MwvcZZI40M/WxcjBISyQI7HvaxpImEVAVWL3z6dgJbwCphJrX55mgRgpL7F6wwFmkHJOoDHv /xuChIWASv72XmXsYuQCKvnMJrGqfSYjRL2kxMEVN1gmMPIuYGRYxSiUmVeWm5iZY6KXUZmX WaGXnJ+7iREY9Mtq/0TvYPx0IfgQowAHoxIP74IPt1OFWBPLiitzDzFKcDArifCGy9xMFeJN SaysSi3Kjy8qzUktPsQozcGiJM5r9K08RUggPbEkNTs1tSC1CCbLxMEp1cCoaNMdsOrB5+U9 3HeTbrvb1c+c+/aLqmXDNIFjH5Wu+Nz++fVNOtPBeha2+pf9+rLaV7a+fzFj9s3EELXrDVzM olvN17zXLw336ExxtExueXIyNeHsqeCHBQKTDk2+23kygy1r8/J2MU/ti5uyIj5vuc1h/90l c23cTyN1/5OLHoW2hc5++3y/EktxRqKhFnNRcSIATFW9xnYCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrNLMWRmVeSWpSXmKPExsXC5WfdrKvPcCfVYP09Y4s569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZRy9c4+tYA5nxflfU5kaGC+xdzFyckgImEi8+LcDzGYTUJe4ceMnM4gtImAm cbD1D1icWeAuk8SBfrYuRg4OYYEciX1f00DCLAKqErt/PgUr4RUwlVj78jQLxEh5idUbDjCD lHMCjXn/3xAkLARU8rf3KuMERq4FjAyrGEUy88pyEzNzTPWKszMq8zIr9JLzczcxAoN4We2f iTsYv1x2P8QowMGoxMO74MPtVCHWxLLiytxDjBIczEoivOEyN1OFeFMSK6tSi/Lji0pzUosP MUpzsCiJ83qFpyYICaQnlqRmp6YWpBbBZJk4OKUaGL209/65XXXWKmDGn+p/C/+LPHXn3rCp uvOp7YIZoYk8inLKSj33ZebM2cfpl5Hhf+inK1/xR+v5ZzjOXA/q3Gd443yZlcOa84uuLThy LO6y++o6FgVZgcKfLGJn7dccLRXedev7mYx3Z2bZTU+WL1zryTJlRirjrd9rBWaGRa5+lutY wbx7ursSS3FGoqEWc1FxIgA7DuZGXgIAAA== X-CFilter-Loop: Reflected X-Stat-Signature: a3pi9jdhti7jroexop8bysqgdm4smqey X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: EC7F710000B X-Rspam-User: X-HE-Tag: 1708916791-97485 X-HE-Meta: U2FsdGVkX19Bo+L7YjOXvLvUlCzzrS3H/5yy1qccNZLNFpaGhWHCceipDz/IOmrHOKqc8F3mT5+QlQOQguf3E4+6R6zFvojqO6AmUz9ypaehfuOD7++puuu+J3KBWW5MHk7erd7snpme/8e50UEQVMbh8S58iSLdMJxMGWza3xsx4XPI6ZnJ4Ev6pemrfetw3+t+gWmff+4OsnsOxdUyeOiQWXRHUmIHVJGcSlkaw4DnNo2ucQ2dav1eVKzKJIY4CKJMHBupjuKEWTDvNI1rDS+Ka5bXnMFXeuQmwDe63Cg15M1v8pBiVjLNieZAsk/4HkQZUKvPoYtdFsq37ALEfThICdbB7Bff5X9F7/QkkqSAf1PWz9Pn7h/HxlbPpa4pw0cYdBuLmEjOe5SggemjHncfEkaT3oUPArM8adg5SBsVemwDK7wfs9pPUF4KYCg/kkEYYHMeadNHNOyLM3hqtK+o9yMidHeORIaSQAE5oCiVN5bNx/frpq2sAJQplEXB+1Z/ybjTBG+/wfNGJ43c7o96Pbe2hsnQz+QNmCgVCmnH4jAtEtjx3qnIP30SrIXgwcgxLsmKFivMXjEFeSFyJnZeALQcvhcVnaMuNVu1tMzgbBo9CQJOyMfXQ2E/kJnsYLl6yvLNW6ATslhbaBH0C4L97GDE4V21VwrCAXi6HxZpTcLqDwgrz637/Mu5/GOnl/TuY7pHWS+xAN/RJTIA5qdRK9Mhf18fm1uO9z4lXy1At4wjwCJGfjoZ2XFN2e8iSucbcIkO7mgOMI9KR/jeaPJlddxl+B8I95kh1YtKL9kRpma9rYGe17xoC5DzZJRQmdmkTVRbvXE6TaIXjsd5vQQJiDftk/N1kXImfRXVQD4EBkIQVu2cWhP/ytHGtWGkw4S0bGi0tsxoLrba8FY1OrdZShHP/ByMDNLbZ+K+KiB0CfOz5UGjK8ZtczM0kpxNE81gh3wfeerelATRZsV +vTFkPze 8yjOvl3mAkOjOqD7etiDXDt8kRK3iGAuCSOPj X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparation for migrc mechanism that requires to avoid redundant TLB flushes by manipulating tlb batch's arch data after arch_tlbbatch_flush(). However, we cannot because the data is getting cleared inside arch_tlbbatch_flush(). So separated the part clearing the tlb batch's arch data out of arch_tlbbatch_flush(). Signed-off-by: Byungchul Park --- arch/x86/mm/tlb.c | 2 -- mm/rmap.c | 1 + 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 453ea95b667d..941f41df02f3 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1268,8 +1268,6 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) local_irq_enable(); } - cpumask_clear(&batch->cpumask); - put_flush_tlb_info(); put_cpu(); } diff --git a/mm/rmap.c b/mm/rmap.c index da36f23ff7b0..b484d659d0c1 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -643,6 +643,7 @@ void try_to_unmap_flush(void) return; arch_tlbbatch_flush(&tlb_ubc->arch); + arch_tlbbatch_clear(&tlb_ubc->arch); tlb_ubc->flush_required = false; tlb_ubc->writable = false; } From patchwork Mon Feb 26 03:06:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13571189 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 700E3C47DD9 for ; Mon, 26 Feb 2024 03:06:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 792D16B0148; Sun, 25 Feb 2024 22:06:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6F39E6B014D; Sun, 25 Feb 2024 22:06:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 34BD86B0148; Sun, 25 Feb 2024 22:06:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 1632B6B014B for ; Sun, 25 Feb 2024 22:06:34 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id BDDF9A0742 for ; Mon, 26 Feb 2024 03:06:33 +0000 (UTC) X-FDA: 81832467066.18.3E57A1C Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf30.hostedemail.com (Postfix) with ESMTP id EAE5C80003 for ; Mon, 26 Feb 2024 03:06:31 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708916792; a=rsa-sha256; cv=none; b=F7nB6wfw4ov2KgqP/H8QO/l3kkp9ijCpAD5bgEcTCfffpe3rFMLO/FEOW1p68Vw8U3Fo9T QVXxXUj22kKsNU7kxN9e6rs2ckYyOKXbl4f7ftMExifW5w3ISVMHa8423ydcekUAj4ubOP GYlm94dNTNg70vBcg7n7ddHF5rJAhm8= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708916792; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=hj/OU3wk3OzsqULfJnD2mvtgwTJyJb/ndDOd9gzDe6E=; b=AHucgbMR47qudf0wSxS3B8Wv8Q81wka/67JBJlvhAYlpL6u9vaAWwRoXGs3emk7RNv3TZE JpnfKprAvmW8IpI3OayqwL7cRq6RewY8qpjvL7Bo/stnjSJ9vlg4rTTZaSSg6Vwqw14Fhn n+e1eZqF03x5pWU4JZoWvKYrFJ8m12w= X-AuditID: a67dfc5b-d6dff70000001748-3c-65dc002ffdf6 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v8 5/8] mm: Separate move/undo doing on folio list from migrate_pages_batch() Date: Mon, 26 Feb 2024 12:06:10 +0900 Message-Id: <20240226030613.22366-6-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240226030613.22366-1-byungchul@sk.com> References: <20240226030613.22366-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrKLMWRmVeSWpSXmKPExsXC9ZZnoa4Bw51UgytbWSzmrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG jfd32Qv6dSoWzXnI3sC4UrmLkZNDQsBEYkF3ByOM3dt7hg3EZhNQl7hx4ycziC0iYCZxsPUP O4jNLHCXSeJAP1iNsECKxN3J38FsFgFViYP37oHV8AqYSsxe8ZYFYqa8xOoNB4DmcHBwAs15 /98QJCwEVPK39yrQWi6gkvdsElc6trBC1EtKHFxxg2UCI+8CRoZVjEKZeWW5iZk5JnoZlXmZ FXrJ+bmbGIGBv6z2T/QOxk8Xgg8xCnAwKvHwLvhwO1WINbGsuDL3EKMEB7OSCG+4zM1UId6U xMqq1KL8+KLSnNTiQ4zSHCxK4rxG38pThATSE0tSs1NTC1KLYLJMHJxSDYxTrXaYzOrinGnB +X7XlfuLHN81fbstrP787pK/m45/Eet+c2XzvCshq7vO5H7O+m9Vs1fYqW7vos7J3lqrWjtl H+px1wY2GF2+98WyuXvaZ6V1+ovasvLDwqaGv5pb++xnzoqE3ckCH1qtjtS6O5wU5RZcUbzo 4R21jdvLfVLfi64+ONsnWfSTEktxRqKhFnNRcSIAlFm963gCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrDLMWRmVeSWpSXmKPExsXC5WfdrKvPcCfV4OQlM4s569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZdx4f5e9oF+nYtGch+wNjCuVuxg5OSQETCR6e8+wgdhsAuoSN278ZAaxRQTM JA62/mEHsZkF7jJJHOgHqxEWSJG4O/k7mM0ioCpx8N49sBpeAVOJ2SveskDMlJdYveEA0BwO Dk6gOe//G4KEhYBK/vZeZZzAyLWAkWEVo0hmXlluYmaOqV5xdkZlXmaFXnJ+7iZGYBgvq/0z cQfjl8vuhxgFOBiVeHgXfLidKsSaWFZcmXuIUYKDWUmEN1zmZqoQb0piZVVqUX58UWlOavEh RmkOFiVxXq/w1AQhgfTEktTs1NSC1CKYLBMHp1QD4xnnJfa9v8VUskR+vYviY+g12GM8qy2y qkFCPttR5o+kxdn4ogsvDaW/7btgmLZz7mrO7YHbuNkmCiiFmAQWqFy2X2F01DXWdu1rk6bf L5SrT6xdrzzVL+zI8rUb3BcW2TAlzK/cIrBT+ku186IHK1iStiz28BSrvlkRc/ajwKErieLr vZknK7EUZyQaajEXFScCAI1XUVNfAgAA X-CFilter-Loop: Reflected X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: EAE5C80003 X-Stat-Signature: eokdyu3fdorogx1cadnmmcj5fm1txsnt X-HE-Tag: 1708916791-897652 X-HE-Meta: U2FsdGVkX18z5zmB9/IJWz7KgSPuvekEDDtJdyvPwiYf56j30Y3H7qLH4dpBfuFC85DuftyYs5c4nCg2LTSyrI78Kw4OrNpDMLoBAe531bTlz4oeV5lqe3kg8u+rZPNn9VOjlt2vs77t6fUCQIV8bioAyvLaGdxCNLviuZThGt8MV2HSRE60DF7/WN8/RExBA6oOTKy7SSImun9WDPuxqI4Q1bZbU4WEPEP0NPkpJLVFBCxPy6VdJd/AVNKXVSKvBBJoc+d7CU7PYC1M1Z8RE5XhOsOAuD1v8315vtMeT85j2cu91hpVugjdUcboQ+/FnmY6wG7mc13xU6qGI0idHABYfKXGOJyh7gGT7VvF28377CJo3YDsRkgeic0lq5ISDEjGxxPKcCUvh6spr4//KvsFDmmjmskob3Mr7MfMvq36FDGrY3epTCw8HeEHeEvdQRCPYQ//nIGLcYyfQpBIBrutY0FWhg7JS48DwIWGviq8uhYTspzHjjjyvQaRuz6cORNjm2ud1OTrF6Jjq04qV9x9xfV2LE+YTyuIGm5OxyuTtzM17K671qmFM/lLh8tbcIMClGcZWAThdoHGYBQ9mF/uUOP4yEoNFG/g0YoDCn708Pp9JW+7dhNtDE5cMc2Rs6laJ+U564TTdpo/c3/Y4PtWNJvMc87hFoHSTFlnLwSIe7Ux4Jd0tYisHV6NWiwWZ/8mehF7bJAeMoC9nScqaJ7VLq5QolnrVl2Xbg67qOMKy8HWPbEbXXn+p6nEMkAajjzdu402+caRSHLhpJUtZTXO+/aMXnPvLi5Er8oOQioS3KrnbxF9hul682oL1Sqi1qeJXw/GVT4dCu2NPu6eLpwkW8/BQ+cFUPsikfZstXJsxVv5GR9Q+0B/rrWa1wNwUIWEFoyN2l+lm1Qu7pj6YfaKRj1yA3gk9ym5c1UfTKpoRFntBPLG1G5C3iJ5Mf9gMfsNj4ZM7MPUcGX6SB8 uvgAdzq/ enDAfz+UZIxcjAPAvQuoh1bJ5eZdVQNXnUlwY X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Functionally, no change. This is a preparation for migrc mechanism that requires to use separate folio lists for its own handling at migration. Refactored migrate_pages_batch() and separated move and undo parts operating on folio list, from migrate_pages_batch(). Signed-off-by: Byungchul Park --- mm/migrate.c | 134 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 83 insertions(+), 51 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 397f2a6e34cb..bbe1ecef4956 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1611,6 +1611,81 @@ static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio, return nr_failed; } +static void migrate_folios_move(struct list_head *src_folios, + struct list_head *dst_folios, + free_folio_t put_new_folio, unsigned long private, + enum migrate_mode mode, int reason, + struct list_head *ret_folios, + struct migrate_pages_stats *stats, + int *retry, int *thp_retry, int *nr_failed, + int *nr_retry_pages) +{ + struct folio *folio, *folio2, *dst, *dst2; + bool is_thp; + int nr_pages; + int rc; + + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio); + nr_pages = folio_nr_pages(folio); + + cond_resched(); + + rc = migrate_folio_move(put_new_folio, private, + folio, dst, mode, + reason, ret_folios); + /* + * The rules are: + * Success: folio will be freed + * -EAGAIN: stay on the unmap_folios list + * Other errno: put on ret_folios list + */ + switch(rc) { + case -EAGAIN: + *retry += 1; + *thp_retry += is_thp; + *nr_retry_pages += nr_pages; + break; + case MIGRATEPAGE_SUCCESS: + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; + break; + default: + *nr_failed += 1; + stats->nr_thp_failed += is_thp; + stats->nr_failed_pages += nr_pages; + break; + } + dst = dst2; + dst2 = list_next_entry(dst, lru); + } +} + +static void migrate_folios_undo(struct list_head *src_folios, + struct list_head *dst_folios, + free_folio_t put_new_folio, unsigned long private, + struct list_head *ret_folios) +{ + struct folio *folio, *folio2, *dst, *dst2; + + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + int old_page_state = 0; + struct anon_vma *anon_vma = NULL; + + __migrate_folio_extract(dst, &old_page_state, &anon_vma); + migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED, + anon_vma, true, ret_folios); + list_del(&dst->lru); + migrate_folio_undo_dst(dst, true, put_new_folio, private); + dst = dst2; + dst2 = list_next_entry(dst, lru); + } +} + /* * migrate_pages_batch() first unmaps folios in the from list as many as * possible, then move the unmapped folios. @@ -1633,7 +1708,7 @@ static int migrate_pages_batch(struct list_head *from, int pass = 0; bool is_thp = false; bool is_large = false; - struct folio *folio, *folio2, *dst = NULL, *dst2; + struct folio *folio, *folio2, *dst = NULL; int rc, rc_saved = 0, nr_pages; LIST_HEAD(unmap_folios); LIST_HEAD(dst_folios); @@ -1769,42 +1844,11 @@ static int migrate_pages_batch(struct list_head *from, thp_retry = 0; nr_retry_pages = 0; - dst = list_first_entry(&dst_folios, struct folio, lru); - dst2 = list_next_entry(dst, lru); - list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { - is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio); - nr_pages = folio_nr_pages(folio); - - cond_resched(); - - rc = migrate_folio_move(put_new_folio, private, - folio, dst, mode, - reason, ret_folios); - /* - * The rules are: - * Success: folio will be freed - * -EAGAIN: stay on the unmap_folios list - * Other errno: put on ret_folios list - */ - switch(rc) { - case -EAGAIN: - retry++; - thp_retry += is_thp; - nr_retry_pages += nr_pages; - break; - case MIGRATEPAGE_SUCCESS: - stats->nr_succeeded += nr_pages; - stats->nr_thp_succeeded += is_thp; - break; - default: - nr_failed++; - stats->nr_thp_failed += is_thp; - stats->nr_failed_pages += nr_pages; - break; - } - dst = dst2; - dst2 = list_next_entry(dst, lru); - } + /* Move the unmapped folios */ + migrate_folios_move(&unmap_folios, &dst_folios, + put_new_folio, private, mode, reason, + ret_folios, stats, &retry, &thp_retry, + &nr_failed, &nr_retry_pages); } nr_failed += retry; stats->nr_thp_failed += thp_retry; @@ -1813,20 +1857,8 @@ static int migrate_pages_batch(struct list_head *from, rc = rc_saved ? : nr_failed; out: /* Cleanup remaining folios */ - dst = list_first_entry(&dst_folios, struct folio, lru); - dst2 = list_next_entry(dst, lru); - list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { - int old_page_state = 0; - struct anon_vma *anon_vma = NULL; - - __migrate_folio_extract(dst, &old_page_state, &anon_vma); - migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED, - anon_vma, true, ret_folios); - list_del(&dst->lru); - migrate_folio_undo_dst(dst, true, put_new_folio, private); - dst = dst2; - dst2 = list_next_entry(dst, lru); - } + migrate_folios_undo(&unmap_folios, &dst_folios, + put_new_folio, private, ret_folios); return rc; } From patchwork Mon Feb 26 03:06:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13571190 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2823C54798 for ; Mon, 26 Feb 2024 03:06:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C9ED56B014D; Sun, 25 Feb 2024 22:06:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B895C6B014E; Sun, 25 Feb 2024 22:06:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9D9F96B014F; Sun, 25 Feb 2024 22:06:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8F33A6B014D for ; Sun, 25 Feb 2024 22:06:35 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 575D84067B for ; Mon, 26 Feb 2024 03:06:35 +0000 (UTC) X-FDA: 81832467150.13.1D11DC9 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf01.hostedemail.com (Postfix) with ESMTP id 9D29340005 for ; Mon, 26 Feb 2024 03:06:33 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708916794; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=sYa6ZVoMA5/dASQ+t66FETLgCEz/UfospKSHXnWzbLI=; b=fZ20++BpYY5/jqF0Tci/J+Wtenf1wrmRMcxRhGW1w5IcHIejRnH+UQXA0ILZF4EtPbVDc5 LJ+SsxLY1SLUvKXPZiRmiPKZ4w2liVhKywUO43TNVtnVQpesr9OUTPF7CboafuPEIQMb3s OQvqPi0vuPtXnDcK9gYXu6UgyAbypjc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708916794; a=rsa-sha256; cv=none; b=cP+RMgYnWTBqVCK20foD1Q1cJn8fRrIjzxMI7vgiXn8OgFUIUqvfqBC30hnI1Sg5NA41oq UHfGH4mz/dIyXtv69nWinha1MNDrOxP5MOi5G+4NOQKjZqPoef5Jqo2eRvh7W92V+t85Wy hRiaruPlZELuYGX3OpKgpCIW5fgPKlc= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none X-AuditID: a67dfc5b-d6dff70000001748-41-65dc002f0738 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v8 6/8] mm: Add APIs to free a folio directly to the buddy bypassing pcp Date: Mon, 26 Feb 2024 12:06:11 +0900 Message-Id: <20240226030613.22366-7-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240226030613.22366-1-byungchul@sk.com> References: <20240226030613.22366-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrCLMWRmVeSWpSXmKPExsXC9ZZnoa4Bw51Ug/Pn2CzmrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG qfuL2AuuyVb8f/uPtYFxskQXIweHhICJxNZHGV2MnGDm0zntLCA2m4C6xI0bP5lBbBEBM4mD rX/YQWxmgbtMEgf62UBsYYF4if2X9oDVsAioSux98QOsl1fAVGLG58lsEDPlJVZvOMAMsooT aM77/4YgYSGgkr+9Vxm7GLmASj6zSWzZepgdol5S4uCKGywTGHkXMDKsYhTKzCvLTczMMdHL qMzLrNBLzs/dxAgM+2W1f6J3MH66EHyIUYCDUYmHd8GH26lCrIllxZW5hxglOJiVRHjDZW6m CvGmJFZWpRblxxeV5qQWH2KU5mBREuc1+laeIiSQnliSmp2aWpBaBJNl4uCUamBs/fwpueHU +a9tKw+2sUW6Xbrza+XE5pfOx8KLlTklC3pe7Dl5W/xH0iLN+GtZapmOzp+F9zow/WKaJquq t/Td3RgtJZXY6nUBOjITfu//bnl8389p/695R/1hzOyZG64y9XTs34X8TNqa66csEjSdH9Zm u73wwNTLs17mLVOZ/P3rydCGyGUflViKMxINtZiLihMBNdP+sncCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrDLMWRmVeSWpSXmKPExsXC5WfdrKvPcCfVYOcJS4s569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZZy6v4i94Jpsxf+3/1gbGCdLdDFyckgImEg8ndPOAmKzCahL3LjxkxnEFhEw kzjY+ocdxGYWuMskcaCfDcQWFoiX2H9pD1gNi4CqxN4XP8B6eQVMJWZ8nswGMVNeYvWGA0A1 HBycQHPe/zcECQsBlfztvco4gZFrASPDKkaRzLyy3MTMHFO94uyMyrzMCr3k/NxNjMAwXlb7 Z+IOxi+X3Q8xCnAwKvHwLvhwO1WINbGsuDL3EKMEB7OSCG+4zM1UId6UxMqq1KL8+KLSnNTi Q4zSHCxK4rxe4akJQgLpiSWp2ampBalFMFkmDk6pBkadtKp9a39EpfhfkeG/yl7+RWGKU8Ar h7QetZ0O31+/at9+Y9qPHSdCr54z+NdVrbB9evJcp3jNm4/Z5V709a477P5QvGKyx1WtmQw9 zwzOBq15tV137Ycfy9IvnZa4HZEYYGsSVuHG98zS5ZL1gbK2ZSzHJ3fMO6gpEPyQ5yTnZY24 2xMfZzEosRRnJBpqMRcVJwIAuYNFwF8CAAA= X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 9D29340005 X-Rspam-User: X-Stat-Signature: tpm56n6nk6fbnw99fohyb4mubp3bawje X-Rspamd-Server: rspam03 X-HE-Tag: 1708916793-235151 X-HE-Meta: U2FsdGVkX19PTGhVDAK538BcYZJhgvK25bU3xBjwq7Whcx1JAa8iP0nH2YqD8Y8rU2fwSj1osnDZSGpqKddiO2oKlSw57q4QVO1me4LkkSBMscBxtFkoUWrR1oXA3Xo6HFp1VAy5QVH/gC4iy42+sJl6LmQGKqz00+3gtRXYb5yKVeM1wK7OTt8TSj3gw09gj8jLLbnsQx508M0S7jE+lDiAWTnvdkG61N/MOE5V/39/BQLfuzsdiqsrb72gML0U5WsnpFjYISORb5Zc8aXJ+kHcqxFtwlG8Ov0bj9msk2nAHxPqCmPR4ws4EbgKX3U8FkM8QFZbaN5OsfNM/xrUijlStRCtRYRE3OyQh0nn6TetF2V3PyHdfHMVmfwqA46W4hcdbf0hCzeTNVROIFUN3Jk6mqNtav7NuGADAGoooqEpPECe9dav3KrVz16cImNkWVbnLsXesmLT/apgBHkyzwKtv7kc0kwwrGtLMi9JupOR83lOY1tjlur0u/fcI6LeJBKnQqhCsWoQzkmagFAXKuAhGHsuXeqQiv5+68j5gGSaTdpxlQOQh309W1zaAqLNhSw42e1XFQ0k0IDTb2BJhySfJZT/zgwYNbQUFwpM4I4O5pSabO2ESRGfgzmwSQ1rwZekkx5g37znN7bao6Vu3NewOoYu2wRLikTCKXoVBlNPdwjRsUl5k1HiQxan1Ul8P+u+NIP9RJJlDD68XgILDw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparation for migrc mechanism that frees folios at a better time. The mechanism will defer the use of folio_put*() for source folios of migration, that are unlikely to be used and a group of folios will be freed at once at a later time. However, this will pollute pcp so as to inexpectedly free_pcppages_bulk() fresher folios and make pcp get unstable. To facilitate this new mechanism, an additional API has been added that allows folios under migrc's control to be freed directly to buddy bypassing pcp. Signed-off-by: Byungchul Park --- include/linux/mm.h | 23 +++++++++++++++++++++++ mm/internal.h | 1 + mm/page_alloc.c | 10 ++++++++++ mm/swap.c | 7 +++++++ 4 files changed, 41 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index da5219b48d52..fc0581cce3a7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1284,6 +1284,7 @@ static inline struct folio *virt_to_folio(const void *x) } void __folio_put(struct folio *folio); +void __folio_put_small_nopcp(struct folio *folio); void put_pages_list(struct list_head *pages); @@ -1483,6 +1484,28 @@ static inline void folio_put(struct folio *folio) __folio_put(folio); } +/** + * folio_put_small_nopcp - Decrement the reference count on a folio. + * @folio: The folio. + * + * This is only for a single page folio to release directly to the buddy + * allocator bypassing pcp. + * + * If the folio's reference count reaches zero, the memory will be + * released back to the page allocator and may be used by another + * allocation immediately. Do not access the memory or the struct folio + * after calling folio_put_small_nopcp() unless you can be sure that it + * wasn't the last reference. + * + * Context: May be called in process or interrupt context, but not in NMI + * context. May be called while holding a spinlock. + */ +static inline void folio_put_small_nopcp(struct folio *folio) +{ + if (folio_put_testzero(folio)) + __folio_put_small_nopcp(folio); +} + /** * folio_put_refs - Reduce the reference count on a folio. * @folio: The folio. diff --git a/mm/internal.h b/mm/internal.h index b880f1e78700..3be8fd5604e8 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -451,6 +451,7 @@ extern int user_min_free_kbytes; extern void free_unref_page(struct page *page, unsigned int order); extern void free_unref_page_list(struct list_head *list); +extern void free_pages_nopcp(struct page *page, unsigned int order); extern void zone_pcp_reset(struct zone *zone); extern void zone_pcp_disable(struct zone *zone); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 733732e7e0ba..21b8c8cd1673 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -565,6 +565,16 @@ static inline void free_the_page(struct page *page, unsigned int order) __free_pages_ok(page, order, FPI_NONE); } +void free_pages_nopcp(struct page *page, unsigned int order) +{ + /* + * This function will be used in case that the pages are too + * cold to keep in pcp e.g. migrc mechanism. So it'd better + * release the pages to the tail. + */ + __free_pages_ok(page, order, FPI_TO_TAIL); +} + /* * Higher-order pages are called "compound pages". They are structured thusly: * diff --git a/mm/swap.c b/mm/swap.c index cd8f0150ba3a..3f37496a1184 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -106,6 +106,13 @@ static void __folio_put_small(struct folio *folio) free_unref_page(&folio->page, 0); } +void __folio_put_small_nopcp(struct folio *folio) +{ + __page_cache_release(folio); + mem_cgroup_uncharge(folio); + free_pages_nopcp(&folio->page, 0); +} + static void __folio_put_large(struct folio *folio) { /* From patchwork Mon Feb 26 03:06:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13571192 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2372C54798 for ; Mon, 26 Feb 2024 03:06:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A28FE6B0150; Sun, 25 Feb 2024 22:06:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9DB896B0152; Sun, 25 Feb 2024 22:06:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 596CF6B0150; Sun, 25 Feb 2024 22:06:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 41B7B6B014E for ; Sun, 25 Feb 2024 22:06:36 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 1CD811C071A for ; Mon, 26 Feb 2024 03:06:36 +0000 (UTC) X-FDA: 81832467192.08.15F8342 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf05.hostedemail.com (Postfix) with ESMTP id 1A5FC10000B for ; Mon, 26 Feb 2024 03:06:33 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=none; spf=pass (imf05.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708916794; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=fZXGApSJ6yOmQMw6O9Qn2JVVKDcgwybFt35sixwjZS8=; b=ffj1stzWGuLX8F0cuYggsqDbAbnNqZspxLPig+za36CvcUYywK8uP2l9JQVhqdRYn/yukW MqRx6aL6ts0WaLQUtEptrV7cLhQTy2HbPEb/34071vsgehlrPfX4ttg7ftsLGvfcOlP/y1 7Ie8dGcOxm+4iXcGgs7GBE67kuB2Dgc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708916794; a=rsa-sha256; cv=none; b=A2/ZGsiThWKDtH8H9qc4DZlHDUUoqp5ruxXoBNZxc3Hw8doNZYbR9QFcCYrJu/MfN7/j04 eAEp4HVTIGvqXunjSjcwW6hRbjV8kCuF9d/nkRATd68i161W6sK2lREUSyYJw6Lghumw7y QOXkLRlJgV+DGZObYrw7lgh3LUbjeHQ= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=none; spf=pass (imf05.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none X-AuditID: a67dfc5b-d6dff70000001748-46-65dc00307553 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v8 7/8] mm: Defer TLB flush by keeping both src and dst folios at migration Date: Mon, 26 Feb 2024 12:06:12 +0900 Message-Id: <20240226030613.22366-8-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240226030613.22366-1-byungchul@sk.com> References: <20240226030613.22366-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrOLMWRmVeSWpSXmKPExsXC9ZZnoa4Bw51Ug9NP+CzmrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG jEVnWAuerGSsuNpxjL2BcU0bYxcjJ4eEgInEjUWtrDD2z94vzCA2m4C6xI0bP8FsEQEziYOt f9hBbGaBu0wSB/rZQGxhgSSJwzt7mUBsFgFViUsnF4PV8wqYSjzZ/5ANYqa8xOoNB4DiHByc QHPe/zcECQsBlfztvQp0AhdQyWc2iZf73jND1EtKHFxxg2UCI+8CRoZVjEKZeWW5iZk5JnoZ lXmZFXrJ+bmbGIHBv6z2T/QOxk8Xgg8xCnAwKvHwLvhwO1WINbGsuDL3EKMEB7OSCG+4zM1U Id6UxMqq1KL8+KLSnNTiQ4zSHCxK4rxG38pThATSE0tSs1NTC1KLYLJMHJxSDYyuGZkJ0hNc 3c9aPc55qVo4dT6fw4wNB332Fm3pyn+0a3qV6FG2yoodTy3Pe/MvX+M4qT9+0xk/qTqmS64P 5ARUROuDWyvf7jjfqGZy8azqiojd+24cS7/8OtCMa9mXq3zHt97i8jVv3nBxmkjS6eC0pCfb qybsvWVlv3PJCqWwX5+SotbZxaQrsRRnJBpqMRcVJwIAihqzTnoCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrHLMWRmVeSWpSXmKPExsXC5WfdrGvAcCfVYN4hJos569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZcxYdIa14MlKxoqrHcfYGxjXtDF2MXJySAiYSPzs/cIMYrMJqEvcuPETzBYR MJM42PqHHcRmFrjLJHGgnw3EFhZIkji8s5cJxGYRUJW4dHIxWD2vgKnEk/0P2SBmykus3nAA KM7BwQk05/1/Q5CwEFDJ396rjBMYuRYwMqxiFMnMK8tNzMwx1SvOzqjMy6zQS87P3cQIDOVl tX8m7mD8ctn9EKMAB6MSD++CD7dThVgTy4orcw8xSnAwK4nwhsvcTBXiTUmsrEotyo8vKs1J LT7EKM3BoiTO6xWemiAkkJ5YkpqdmlqQWgSTZeLglGpgVGlNtTocvEB9xl8pXhWjjDvbrTJk GvmFZtYK+j9Wzef5GHD7lNzute23ulJZTWcpFvwqbJvWv1NgTb+OYXnXu4/m7zQqXV+s6Jtg dKPf39g123fmt/xvtxqMeT6fNJmZcWfeh9MbTHjMt8wov3vGt+7chpkSH5rvfWBKjhd7z542 Nc3bS3O+EktxRqKhFnNRcSIAWMK1WWECAAA= X-CFilter-Loop: Reflected X-Stat-Signature: cj9t6fugk3swqtybtodrrqpd8wh96rdf X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 1A5FC10000B X-Rspam-User: X-HE-Tag: 1708916793-501438 X-HE-Meta: U2FsdGVkX1+ci6yPi0UpzyBZzNAxgLQ/zgbRC5owqR6iTjhPAMmYMGN5gKmyVVOrveVx/u2HbrbEkROs8Pr7z4LYAzRB8WbaZemKY05piXYErRnOhwR3ov8mwl1wXWAqFn67cAfcQVa86wmedJagtvPhu1GMj1JfAUUSRj7Aunw7rC6gVowN1n/r48gTebG1B3yx9DgATPiHrLr7gw/aovMLdlgya5pmBPx0vmBuWJTqJmGuagHS1kwxKVSOR8I6+Azc+cO+S6PXx7xv+XfDXXcND2QLqpy30dKREcYKXALUZntUuMEEV/E76MT5DFQVBuK5LabJ61YLSI4BJI4CtLxfbZ5XzVLE9oKLXNUsA3zn4xZJZnOlK9Z2creGZX9KFVvUVwBQdP5ct2PrJbxQ9XTt735hj3HH1uS5xMrZ3RZ3Bh9hMz0xXY+JcOFWG4KV9DKdUh0yPxWAshYJ3sJJZZAp2MPLgiRrAHHheI93P0eQyZ5TrnDDhGWoUGje2AQwp92dl8d/peYkiEV6kkMuamnQzS6kwJ9AEGtXxdKB5kqtDDQ1vEdkGQvPEmUc8ecf2eEhaxXvuA2tZFqUQ/R8QZplOtqwsqW/3gnpoiGsytkF5FIiJ18wms3pTbRSwYuUnlQBplAOg+p4GMCpZ5Lz57tW3wAJ0dcqj8TCohJ27OJVls4GG3IQVwhx45Vt45hfFTDMFbH067lW/s6weIhJDp+iOYd8e2D8z6pLhN1T7/5LP9iGQeDSAxpdOBqOy8/Gn0am1IHCOx0Jtp0MIJ1kX6WkJNMEd8x72kf8yG9w9W/O5oRcikhxaz0myISL4cu5Fuq4q/Iw0l6J0sSSkyO4ZnbiOKGeUM9DdLVSEhoN/CLG/oWAFgWWiLClJKqpJ3NUWDZ5UcdpXFwORU5/jIjmJ9yGo1UcdZBceZ6ThYGltR37iSC5p4N3AdLs8L1p4gUs9XyfCBwrWDaZByL63TL Opw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Implementation of MIGRC mechanism that stands for 'Migration Read Copy'. We always face the migration overhead at either promotion or demotion, while working with tiered memory e.g. CXL memory and found out TLB shootdown is a quite big one that is needed to get rid of if possible. Fortunately, TLB flush can be defered if both source and destination of folios during migration are kept until all TLB flushes required will have been done, of course, only if the target PTE entries have read-only permission, more precisely speaking, don't have write permission. Otherwise, no doubt the folio might get messed up. To achieve that: 1. For the folios that map only to non-writable TLB entries, prevent TLB flush at migration by keeping both source and destination folios, which will be handled later at a better time. 2. When any non-writable TLB entry changes to writable e.g. through fault handler, give up migrc mechanism so as to perform TLB flush required right away. The following evaluation using XSBench shows the improvement like: 1. itlb flush was reduced by 93.9%. 2. dtlb thread was reduced by 43.5%. 3. stlb flush was reduced by 24.9%. 4. dtlb store misses was reduced by 34.2%. 5. itlb load misses was reduced by 45.5%. 6. The runtime was reduced by 3.5%. The measurement result: Architecture - x86_64 QEMU - kvm enabled, host cpu Numa - 2 nodes (16 CPUs 1GB, no CPUs 8GB) Linux Kernel - v6.7, numa balancing tiering on, demotion enabled Benchmark - XSBench -p 100000000 (-p option makes the runtime longer) run 'perf stat' using events: 1) itlb.itlb_flush 2) tlb_flush.dtlb_thread 3) tlb_flush.stlb_any 4) dTLB-load-misses 5) dTLB-store-misses 6) iTLB-load-misses run 'cat /proc/vmstat' and pick: 1) numa_pages_migrated 2) pgmigrate_success 3) nr_tlb_remote_flush 4) nr_tlb_remote_flush_received 5) nr_tlb_local_flush_all 6) nr_tlb_local_flush_one BEFORE - mainline v6.7 ---------------------- $ perf stat -a \ -e itlb.itlb_flush \ -e tlb_flush.dtlb_thread \ -e tlb_flush.stlb_any \ -e dTLB-load-misses \ -e dTLB-store-misses \ -e iTLB-load-misses \ ./XSBench -p 100000000 Performance counter stats for 'system wide': 85647229 itlb.itlb_flush 480981504 tlb_flush.dtlb_thread 323937200 tlb_flush.stlb_any 238381632579 dTLB-load-misses 601514255 dTLB-store-misses 2974157461 iTLB-load-misses 2252.883892112 seconds time elapsed $ cat /proc/vmstat ... numa_pages_migrated 12790664 pgmigrate_success 26835314 nr_tlb_remote_flush 3031412 nr_tlb_remote_flush_received 45234862 nr_tlb_local_flush_all 216584 nr_tlb_local_flush_one 740940 ... AFTER - mainline v6.7 + migrc ----------------------------- $ perf stat -a \ -e itlb.itlb_flush \ -e tlb_flush.dtlb_thread \ -e tlb_flush.stlb_any \ -e dTLB-load-misses \ -e dTLB-store-misses \ -e iTLB-load-misses \ ./XSBench -p 100000000 Performance counter stats for 'system wide': 5240261 itlb.itlb_flush 271581774 tlb_flush.dtlb_thread 243149389 tlb_flush.stlb_any 234502983364 dTLB-load-misses 395673680 dTLB-store-misses 1620215163 iTLB-load-misses 2172.283436287 seconds time elapsed $ cat /proc/vmstat ... numa_pages_migrated 14897064 pgmigrate_success 30825530 nr_tlb_remote_flush 198290 nr_tlb_remote_flush_received 2820156 nr_tlb_local_flush_all 92048 nr_tlb_local_flush_one 741401 ... Signed-off-by: Byungchul Park --- include/linux/mmzone.h | 7 ++ include/linux/sched.h | 8 ++ mm/internal.h | 53 ++++++++ mm/memory.c | 8 ++ mm/migrate.c | 271 +++++++++++++++++++++++++++++++++++++++-- mm/page_alloc.c | 11 +- mm/rmap.c | 12 +- 7 files changed, 358 insertions(+), 12 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 9db36e197712..492111cd1176 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1002,6 +1002,13 @@ struct zone { /* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS]; + +#if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) + /* + * the number of folios pending for TLB flush in the zone + */ + atomic_t migrc_pending_nr; +#endif } ____cacheline_internodealigned_in_smp; enum pgdat_flags { diff --git a/include/linux/sched.h b/include/linux/sched.h index 0317e7a65151..d8c285309a8f 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1330,6 +1330,14 @@ struct task_struct { struct tlbflush_unmap_batch tlb_ubc; struct tlbflush_unmap_batch tlb_ubc_ro; +#if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) + /* + * whether all the mappings of a folio during unmap are read-only + * so that migrc can work on the folio + */ + bool can_migrc; +#endif + /* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe; diff --git a/mm/internal.h b/mm/internal.h index 3be8fd5604e8..ab02cb8306e2 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1284,4 +1284,57 @@ static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry, } #endif /* CONFIG_SHRINKER_DEBUG */ +#if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) +/* + * Reset the indicator indicating there are no writable mappings at the + * beginning of every rmap traverse for unmap. Migrc can work only when + * all the mappings are read-only. + */ +static inline void can_migrc_init(void) +{ + current->can_migrc = true; +} + +/* + * Mark the folio is not applicable to migrc, once it found a writble or + * dirty pte during rmap traverse for unmap. + */ +static inline void can_migrc_fail(void) +{ + current->can_migrc = false; +} + +/* + * Check if all the mappings are read-only and read-only mappings even + * exist. + */ +static inline bool can_migrc_test(void) +{ + return current->can_migrc && current->tlb_ubc_ro.flush_required; +} + +/* + * Return the number of folios pending TLB flush that have yet to get + * freed in the zone. + */ +static inline int migrc_pending_nr_in_zone(struct zone *z) +{ + return atomic_read(&z->migrc_pending_nr); +} + +/* + * Perform TLB flush needed and free the folios under migrc's control. + */ +bool migrc_flush_free_folios(void); +void migrc_flush_start(void); +void migrc_flush_end(struct tlbflush_unmap_batch *batch); +#else /* CONFIG_MIGRATION && CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ +static inline void can_migrc_init(void) {} +static inline void can_migrc_fail(void) {} +static inline bool can_migrc_test(void) { return false; } +static inline int migrc_pending_nr_in_zone(struct zone *z) { return 0; } +static inline bool migrc_flush_free_folios(void) { return false; } +static inline void migrc_flush_start(void) {} +static inline void migrc_flush_end(struct tlbflush_unmap_batch *batch) {} +#endif #endif /* __MM_INTERNAL_H */ diff --git a/mm/memory.c b/mm/memory.c index 6e0712d06cd4..e67de161da8b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3462,6 +3462,14 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) if (vmf->page) folio = page_folio(vmf->page); + /* + * The folio may or may not be one that is under migrc's control + * and about to change its permission from read-only to writable. + * Conservatively give up deferring TLB flush just in case. + */ + if (folio) + migrc_flush_free_folios(); + /* * Shared mapping: we are guaranteed to have VM_WRITE and * FAULT_FLAG_WRITE set at this point. diff --git a/mm/migrate.c b/mm/migrate.c index bbe1ecef4956..cbe5372f159e 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -57,6 +57,194 @@ #include "internal.h" +#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH +static struct tlbflush_unmap_batch migrc_ubc; +static LIST_HEAD(migrc_folios); +static DEFINE_SPINLOCK(migrc_lock); + +static void init_tlb_ubc(struct tlbflush_unmap_batch *ubc) +{ + arch_tlbbatch_clear(&ubc->arch); + ubc->flush_required = false; + ubc->writable = false; +} + +static void migrc_keep_folio(struct folio *f, struct list_head *h) +{ + list_move_tail(&f->lru, h); + folio_get(f); + atomic_inc(&folio_zone(f)->migrc_pending_nr); +} + +static void migrc_undo_folio(struct folio *f) +{ + list_del(&f->lru); + folio_put(f); + atomic_dec(&folio_zone(f)->migrc_pending_nr); +} + +static void migrc_release_folio(struct folio *f) +{ + folio_put_small_nopcp(f); + atomic_dec(&folio_zone(f)->migrc_pending_nr); +} + +/* + * Need to synchronize between TLB flush and managing pending CPUs in + * migrc_ubc. Take a look at the following scenario: + * + * CPU0 CPU1 + * ---- ---- + * TLB flush + * Unmap folios (needing TLB flush) + * Add pending CPUs to migrc_ubc + * Clear the CPUs from migrc_ubc + * + * The pending CPUs added in CPU1 should not be cleared from migrc_ubc + * in CPU0 because the TLB flush for migrc_ubc added in CPU1 has not + * been performed this turn. To avoid this, using 'migrc_flushing' + * variable, prevent adding pending CPUs to migrc_ubc and give up migrc + * mechanism if others are in the middle of TLB flush, like: + * + * CPU0 CPU1 + * ---- ---- + * migrc_flushing++ + * TLB flush + * Unmap folios (needing TLB flush) + * If migrc_flushing == 0: + * Add pending CPUs to migrc_ubc + * Else: <--- hit + * Give up migrc mechanism + * Clear the CPUs from migrc_ubc + * migrc_flush-- + * + * Only the following case would be allowed for migrc mechanism to work: + * + * CPU0 CPU1 + * ---- ---- + * Unmap folios (needing TLB flush) + * If migrc_flushing == 0: <--- hit + * Add pending CPUs to migrc_ubc + * Else: + * Give up migrc mechanism + * migrc_flushing++ + * TLB flush + * Clear the CPUs from migrc_ubc + * migrc_flush-- + */ +static int migrc_flushing; + +static bool migrc_add_pending_ubc(struct tlbflush_unmap_batch *ubc) +{ + struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + if (migrc_flushing) { + spin_unlock_irqrestore(&migrc_lock, flags); + + /* + * Give up migrc mechanism. Just let TLB flush needed + * handled by try_to_unmap_flush() at the caller side. + */ + fold_ubc(tlb_ubc, ubc); + return false; + } + fold_ubc(&migrc_ubc, ubc); + spin_unlock_irqrestore(&migrc_lock, flags); + return true; +} + +static bool migrc_add_pending_folios(struct list_head *folios) +{ + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + if (migrc_flushing) { + spin_unlock_irqrestore(&migrc_lock, flags); + + /* + * Give up migrc mechanism. The caller should perform + * TLB flush needed using migrc_flush_free_folios() and + * undo some on the folios e.g. restore folios' + * reference count increased by migrc and more. + */ + return false; + } + list_splice(folios, &migrc_folios); + spin_unlock_irqrestore(&migrc_lock, flags); + return true; +} + +void migrc_flush_start(void) +{ + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + migrc_flushing++; + spin_unlock_irqrestore(&migrc_lock, flags); +} + +void migrc_flush_end(struct tlbflush_unmap_batch *batch) +{ + LIST_HEAD(folios); + struct folio *f, *f2; + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + if (!arch_tlbbatch_done(&migrc_ubc.arch, &batch->arch)) { + list_splice_init(&migrc_folios, &folios); + migrc_ubc.flush_required = false; + migrc_ubc.writable = false; + } + migrc_flushing--; + spin_unlock_irqrestore(&migrc_lock, flags); + + list_for_each_entry_safe(f, f2, &folios, lru) + migrc_release_folio(f); +} + +bool migrc_flush_free_folios(void) +{ + struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + LIST_HEAD(folios); + struct folio *f, *f2; + unsigned long flags; + bool ret = true; + + spin_lock_irqsave(&migrc_lock, flags); + list_splice_init(&migrc_folios, &folios); + fold_ubc(tlb_ubc, &migrc_ubc); + spin_unlock_irqrestore(&migrc_lock, flags); + + if (list_empty(&folios)) + ret = false; + + try_to_unmap_flush(); + list_for_each_entry_safe(f, f2, &folios, lru) + migrc_release_folio(f); + return ret; +} +#else /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ +static void init_tlb_ubc(struct tlbflush_unmap_batch *ubc) +{ +} +static void migrc_keep_folio(struct folio *f, struct list_head *h) +{ +} +static void migrc_undo_folio(struct folio *f) +{ +} +static bool migrc_add_pending_ubc(struct tlbflush_unmap_batch *ubc) +{ + return false; +} +static bool migrc_add_pending_folios(struct list_head *folios) +{ + return false; +} +#endif + bool isolate_movable_page(struct page *page, isolate_mode_t mode) { struct folio *folio = folio_get_nontail_page(page); @@ -1274,7 +1462,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, struct folio *src, struct folio *dst, enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) + struct list_head *ret, struct list_head *move_succ) { int rc; int old_page_state = 0; @@ -1321,9 +1509,13 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, /* * A folio that has been migrated has all references removed - * and will be freed. + * and will be freed, unless it's under migrc's control. */ - list_del(&src->lru); + if (move_succ) + migrc_keep_folio(src, move_succ); + else + list_del(&src->lru); + /* Drop an anon_vma reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); @@ -1618,7 +1810,7 @@ static void migrate_folios_move(struct list_head *src_folios, struct list_head *ret_folios, struct migrate_pages_stats *stats, int *retry, int *thp_retry, int *nr_failed, - int *nr_retry_pages) + int *nr_retry_pages, struct list_head *move_succ) { struct folio *folio, *folio2, *dst, *dst2; bool is_thp; @@ -1635,7 +1827,7 @@ static void migrate_folios_move(struct list_head *src_folios, rc = migrate_folio_move(put_new_folio, private, folio, dst, mode, - reason, ret_folios); + reason, ret_folios, move_succ); /* * The rules are: * Success: folio will be freed @@ -1712,17 +1904,34 @@ static int migrate_pages_batch(struct list_head *from, int rc, rc_saved = 0, nr_pages; LIST_HEAD(unmap_folios); LIST_HEAD(dst_folios); + LIST_HEAD(unmap_folios_migrc); + LIST_HEAD(dst_folios_migrc); + LIST_HEAD(move_succ); bool nosplit = (reason == MR_NUMA_MISPLACED); + struct tlbflush_unmap_batch pending_ubc; + struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc_ro = ¤t->tlb_ubc_ro; + bool do_migrc; + bool migrc_ubc_succ; VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC && !list_empty(from) && !list_is_singular(from)); + /* + * Apply migrc only to numa migration for now. + */ + init_tlb_ubc(&pending_ubc); + do_migrc = IS_ENABLED(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) && + (reason == MR_DEMOTION || reason == MR_NUMA_MISPLACED); + for (pass = 0; pass < nr_pass && retry; pass++) { retry = 0; thp_retry = 0; nr_retry_pages = 0; list_for_each_entry_safe(folio, folio2, from, lru) { + bool can_migrc; + is_large = folio_test_large(folio); is_thp = is_large && folio_test_pmd_mappable(folio); nr_pages = folio_nr_pages(folio); @@ -1752,9 +1961,12 @@ static int migrate_pages_batch(struct list_head *from, continue; } + can_migrc_init(); rc = migrate_folio_unmap(get_new_folio, put_new_folio, private, folio, &dst, mode, reason, ret_folios); + can_migrc = do_migrc && can_migrc_test() && !is_large; + /* * The rules are: * Success: folio will be freed @@ -1800,7 +2012,8 @@ static int migrate_pages_batch(struct list_head *from, /* nr_failed isn't updated for not used */ stats->nr_thp_failed += thp_retry; rc_saved = rc; - if (list_empty(&unmap_folios)) + if (list_empty(&unmap_folios) && + list_empty(&unmap_folios_migrc)) goto out; else goto move; @@ -1814,8 +2027,19 @@ static int migrate_pages_batch(struct list_head *from, stats->nr_thp_succeeded += is_thp; break; case MIGRATEPAGE_UNMAP: - list_move_tail(&folio->lru, &unmap_folios); - list_add_tail(&dst->lru, &dst_folios); + if (can_migrc) { + list_move_tail(&folio->lru, &unmap_folios_migrc); + list_add_tail(&dst->lru, &dst_folios_migrc); + + /* + * Gather ro batch data to add + * to migrc_ubc after unmap. + */ + fold_ubc(&pending_ubc, tlb_ubc_ro); + } else { + list_move_tail(&folio->lru, &unmap_folios); + list_add_tail(&dst->lru, &dst_folios); + } break; default: /* @@ -1829,12 +2053,19 @@ static int migrate_pages_batch(struct list_head *from, stats->nr_failed_pages += nr_pages; break; } + /* + * Done with the current folio. Fold the ro + * batch data gathered, to the normal batch. + */ + fold_ubc(tlb_ubc, tlb_ubc_ro); } } nr_failed += retry; stats->nr_thp_failed += thp_retry; stats->nr_failed_pages += nr_retry_pages; move: + /* Should be before try_to_unmap_flush() */ + migrc_ubc_succ = do_migrc && migrc_add_pending_ubc(&pending_ubc); /* Flush TLBs for all unmapped folios */ try_to_unmap_flush(); @@ -1848,7 +2079,27 @@ static int migrate_pages_batch(struct list_head *from, migrate_folios_move(&unmap_folios, &dst_folios, put_new_folio, private, mode, reason, ret_folios, stats, &retry, &thp_retry, - &nr_failed, &nr_retry_pages); + &nr_failed, &nr_retry_pages, NULL); + migrate_folios_move(&unmap_folios_migrc, &dst_folios_migrc, + put_new_folio, private, mode, reason, + ret_folios, stats, &retry, &thp_retry, + &nr_failed, &nr_retry_pages, migrc_ubc_succ ? + &move_succ : NULL); + } + + /* + * In case that migrc_add_pending_ubc() has been added + * successfully but migrc_add_pending_folios() does not. + */ + if (migrc_ubc_succ && !migrc_add_pending_folios(&move_succ)) { + migrc_flush_free_folios(); + + /* + * Undo src folios that have been successfully added to + * move_succ. + */ + list_for_each_entry_safe(folio, folio2, &move_succ, lru) + migrc_undo_folio(folio); } nr_failed += retry; stats->nr_thp_failed += thp_retry; @@ -1859,6 +2110,8 @@ static int migrate_pages_batch(struct list_head *from, /* Cleanup remaining folios */ migrate_folios_undo(&unmap_folios, &dst_folios, put_new_folio, private, ret_folios); + migrate_folios_undo(&unmap_folios_migrc, &dst_folios_migrc, + put_new_folio, private, ret_folios); return rc; } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 21b8c8cd1673..6ef0c22b1109 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2972,6 +2972,8 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark, long min = mark; int o; + free_pages += migrc_pending_nr_in_zone(z); + /* free_pages may go negative - that's OK */ free_pages -= __zone_watermark_unusable_free(z, order, alloc_flags); @@ -3066,7 +3068,7 @@ static inline bool zone_watermark_fast(struct zone *z, unsigned int order, long usable_free; long reserved; - usable_free = free_pages; + usable_free = free_pages + migrc_pending_nr_in_zone(z); reserved = __zone_watermark_unusable_free(z, 0, alloc_flags); /* reserved may over estimate high-atomic reserves. */ @@ -3273,6 +3275,13 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, gfp_mask)) { int ret; + if (migrc_pending_nr_in_zone(zone) && + migrc_flush_free_folios() && + zone_watermark_fast(zone, order, mark, + ac->highest_zoneidx, + alloc_flags, gfp_mask)) + goto try_this_zone; + if (has_unaccepted_memory()) { if (try_to_accept_memory(zone, order)) goto try_this_zone; diff --git a/mm/rmap.c b/mm/rmap.c index b484d659d0c1..39ab0d64665a 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -642,7 +642,9 @@ void try_to_unmap_flush(void) if (!tlb_ubc->flush_required) return; + migrc_flush_start(); arch_tlbbatch_flush(&tlb_ubc->arch); + migrc_flush_end(tlb_ubc); arch_tlbbatch_clear(&tlb_ubc->arch); tlb_ubc->flush_required = false; tlb_ubc->writable = false; @@ -677,9 +679,15 @@ static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, if (!pte_accessible(mm, pteval)) return; - if (pte_write(pteval) || writable) + if (pte_write(pteval) || writable) { tlb_ubc = ¤t->tlb_ubc; - else + + /* + * Migrc cannot work with the folio, once it found a + * writable or dirty mapping on it. + */ + can_migrc_fail(); + } else tlb_ubc = ¤t->tlb_ubc_ro; arch_tlbbatch_add_pending(&tlb_ubc->arch, mm, uaddr); From patchwork Mon Feb 26 03:06:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13571191 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEC06C54E41 for ; Mon, 26 Feb 2024 03:06:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 59B826B0153; Sun, 25 Feb 2024 22:06:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D3806B0151; Sun, 25 Feb 2024 22:06:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D600940008; Sun, 25 Feb 2024 22:06:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2019F6B014E for ; Sun, 25 Feb 2024 22:06:36 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id EA9B01C070A for ; Mon, 26 Feb 2024 03:06:35 +0000 (UTC) X-FDA: 81832467150.09.7D3E6E5 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf30.hostedemail.com (Postfix) with ESMTP id 22F3D80010 for ; Mon, 26 Feb 2024 03:06:33 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708916794; a=rsa-sha256; cv=none; b=0BVrpI90A6LOM1TwnShKQBsDz2loU/TfLWaVPVFtuF7k3p9OjcuVf33WMaVUdqHrwzn+7R h7wS1JCaLR1S/4Vam6ujkJhUdmba2Iz8Zi6c6sNAhlyS4EEJDngyJBGDefYS+pP0YXAqhy NvANLjybDLSP+CbM+FHuGz0hs6ChqWc= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708916794; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=z8N3K7cEzP29wp8OmV1x4SKhsG69RUNKzzZ3XrSlbEU=; b=7JVlM9eV6crc2uDckPKy170ylqNr7W/SimNscfhaHK8reXHCnaiA9YAsMtIdjbjeemZx5L Wz0/C0Rwxt51IfhpXPsTxqHYhYUInEHxsCk9V427tdOP2hywVsJSSboyFKqL4GgbdKmUok ufcSPzIpuZtulwnxBRwf1tmG2KMzWms= X-AuditID: a67dfc5b-d6dff70000001748-4b-65dc00305618 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v8 8/8] mm: Pause migrc mechanism at high memory pressure Date: Mon, 26 Feb 2024 12:06:13 +0900 Message-Id: <20240226030613.22366-9-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240226030613.22366-1-byungchul@sk.com> References: <20240226030613.22366-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrCLMWRmVeSWpSXmKPExsXC9ZZnoa4Bw51Ug5PTRS3mrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAldG V9t/toKpKhW/N1k1MO6X7WLk5JAQMJH42X+DEcZu+neHCcRmE1CXuHHjJzOILSJgJnGw9Q87 iM0scJdJ4kA/G4gtLBAgsbDzMiuIzSKgKnFmzjUwm1fAVOLOjgvsEDPlJVZvOAA0h4ODE2jO +/+GIGEhoJK/vVeB1nIBlXxmkzjyfB0LRL2kxMEVN1gmMPIuYGRYxSiUmVeWm5iZY6KXUZmX WaGXnJ+7iREY9stq/0TvYPx0IfgQowAHoxIP74IPt1OFWBPLiitzDzFKcDArifCGy9xMFeJN SaysSi3Kjy8qzUktPsQozcGiJM5r9K08RUggPbEkNTs1tSC1CCbLxMEp1cCokaC2g9Mq8Ixo ypMFAUuX3Dtj+C6pXKu1uPzwVqaOQ5+uGCTEH5Xt5FaX/M1ZUG8b1f9egfPBX3ahxQ21f3kv 31ro4dambOOnIjA7eVX+30BW+ZfbQi+ErRCwWd73wEfz0OqEsOe/3M+oXOdf9W6txuklXGms /Z41U7YauOWI9KisY5nPcl+JpTgj0VCLuag4EQCWNApudwIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrNLMWRmVeSWpSXmKPExsXC5WfdrGvAcCfVYMJNdos569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZXS1/WcrmKpS8XuTVQPjftkuRk4OCQETiaZ/d5hAbDYBdYkbN34yg9giAmYS B1v/sIPYzAJ3mSQO9LOB2MICARILOy+zgtgsAqoSZ+ZcA7N5BUwl7uy4wA4xU15i9YYDQHM4 ODiB5rz/bwgSFgIq+dt7lXECI9cCRoZVjCKZeWW5iZk5pnrF2RmVeZkVesn5uZsYgUG8rPbP xB2MXy67H2IU4GBU4uFd8OF2qhBrYllxZe4hRgkOZiUR3nCZm6lCvCmJlVWpRfnxRaU5qcWH GKU5WJTEeb3CUxOEBNITS1KzU1MLUotgskwcnFINjOZfFouliRezzvW/IjXh9MJlbuUnPmo1 9K0T/nN3ysVdU24d335DOrfyjYaUyipdoWO8lzOEqvZ0LuR0XbJrWuzpeYELJKVm61zXW8Um sz14af3MZ/bTebLOu/78Eb/f3/q+2RZTqfrutEnOeu8Cb5yrFtlh0HHpa5gYQ4Dhkuw4g6bv Suf+ciuxFGckGmoxFxUnAgAB/uHHXgIAAA== X-CFilter-Loop: Reflected X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 22F3D80010 X-Stat-Signature: dbc4d7umrf7r1f9o3n19j3qxztahdz9j X-HE-Tag: 1708916793-401397 X-HE-Meta: U2FsdGVkX18nYgOX5hpq19dauXYDcEfbrFCdyRCShfFGQbi9z9L3qEikfOid+ESf4WFx9rWEMc/5TuB/AQEJnB3vd40ZLXfznPRZ68KgkP7MpqN+prmBn8PoqRVWEqxfudz5Poun9/IMb9OTpwgEqT6rBj1kiC6XkOqRVWUv1GRgCcBF4ChaUlDyR3PL95qxHZL79/qxM/hQqS+d5vmFmJerwncGLPMGpM6MTRFDr7YKoEJEgIaJQzAjZ8L7vhM/nl757BIq64t6g13c45jG+7hYnjiqVRevl34WPQT6yPmxb3CWOR1iNphCHz8aRuGmhduFIH7lv0Qxfo3DD7ZtHAvWwmLeMaTKW9bACDJtkLiK7m4qYt0UnYsn2DxFQtqQCtHWTdEgWBByhI4n4uiBnbhQq9JnyGYrFWkD4tY1QAVuqtSF25y/WP6B9syBqakGggcnmzwQnZktFgBs1u7bFeZ2pR8DRQbxyIGaXEbGx6qv5mbf2OSKyKra/EZET1RvfHwaZvt8ukX0uoyJ1PYSZ3ZVU0nnODeFnAGoIBYpUpc7AvuwgPylCtuH/czxlstDdA+QM0WTWfGPK9ctdza2/M6pAxZi68AtUM4nD6rwOLg5NNmByMfzMMnqEjPZMeR8mDJLkTp70B1eLse6IxptXLDOwjn9l9nnll4lXsfArmsMYbAY2a17yevhEms84qI2DAR53wMqxAYhR90+ufBj08EFfxRMjGLzfmN47SmRYSvQ8zYnIRTyEQV08RK/JeWAzJlite1k7dyXSNfHAfF7NgMaYbvOE1UM6IInBrRPCFlIGDaFymXUBVGtqpkwbOL3A1HWplQvUgOmlOuNdzjTQSyz/0JWtVseBs7CxJDE7/KrDGPMWkZyPPdnL3qlh1u1pjolazAbl6gTsGE4SGGKvIBbRIrUWKwi9NKrv1pwbF2UnA2La1xznNolIKJ5hMW/jnmZKTK8dc9VNIJo9pt y0A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Regression was observed when the system is in high memory pressure with swap on, where migrc might keep a number of folios in its pending queue, which possibly makes it worse. So temporarily prevented migrc from working on that condition. Signed-off-by: Byungchul Park --- mm/internal.h | 20 ++++++++++++++++++++ mm/migrate.c | 18 +++++++++++++++++- mm/page_alloc.c | 13 +++++++++++++ 3 files changed, 50 insertions(+), 1 deletion(-) diff --git a/mm/internal.h b/mm/internal.h index ab02cb8306e2..55781f879fb2 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1285,6 +1285,8 @@ static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry, #endif /* CONFIG_SHRINKER_DEBUG */ #if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) +extern atomic_t migrc_pause_cnt; + /* * Reset the indicator indicating there are no writable mappings at the * beginning of every rmap traverse for unmap. Migrc can work only when @@ -1313,6 +1315,21 @@ static inline bool can_migrc_test(void) return current->can_migrc && current->tlb_ubc_ro.flush_required; } +static inline void migrc_pause(void) +{ + atomic_inc(&migrc_pause_cnt); +} + +static inline void migrc_resume(void) +{ + atomic_dec(&migrc_pause_cnt); +} + +static inline bool migrc_paused(void) +{ + return !!atomic_read(&migrc_pause_cnt); +} + /* * Return the number of folios pending TLB flush that have yet to get * freed in the zone. @@ -1332,6 +1349,9 @@ void migrc_flush_end(struct tlbflush_unmap_batch *batch); static inline void can_migrc_init(void) {} static inline void can_migrc_fail(void) {} static inline bool can_migrc_test(void) { return false; } +static inline void migrc_pause(void) {} +static inline void migrc_resume(void) {} +static inline bool migrc_paused(void) { return false; } static inline int migrc_pending_nr_in_zone(struct zone *z) { return 0; } static inline bool migrc_flush_free_folios(void) { return false; } static inline void migrc_flush_start(void) {} diff --git a/mm/migrate.c b/mm/migrate.c index cbe5372f159e..fbc8586ed735 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -62,6 +62,12 @@ static struct tlbflush_unmap_batch migrc_ubc; static LIST_HEAD(migrc_folios); static DEFINE_SPINLOCK(migrc_lock); +/* + * Increase on entry of handling high memory pressure e.g. direct + * reclaim, decrease on the exit. See __alloc_pages_slowpath(). + */ +atomic_t migrc_pause_cnt = ATOMIC_INIT(0); + static void init_tlb_ubc(struct tlbflush_unmap_batch *ubc) { arch_tlbbatch_clear(&ubc->arch); @@ -1922,7 +1928,8 @@ static int migrate_pages_batch(struct list_head *from, */ init_tlb_ubc(&pending_ubc); do_migrc = IS_ENABLED(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) && - (reason == MR_DEMOTION || reason == MR_NUMA_MISPLACED); + (reason == MR_DEMOTION || reason == MR_NUMA_MISPLACED) && + !migrc_paused(); for (pass = 0; pass < nr_pass && retry; pass++) { retry = 0; @@ -1961,6 +1968,15 @@ static int migrate_pages_batch(struct list_head *from, continue; } + /* + * In case that the system is in high memory + * pressure, give up migrc mechanism this turn. + */ + if (unlikely(do_migrc && migrc_paused())) { + fold_ubc(tlb_ubc, &pending_ubc); + do_migrc = false; + } + can_migrc_init(); rc = migrate_folio_unmap(get_new_folio, put_new_folio, private, folio, &dst, mode, reason, diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 6ef0c22b1109..366777afce7f 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4072,6 +4072,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, unsigned int cpuset_mems_cookie; unsigned int zonelist_iter_cookie; int reserve_flags; + bool migrc_paused = false; restart: compaction_retries = 0; @@ -4203,6 +4204,16 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, if (page) goto got_pg; + /* + * The system is in very high memory pressure. Pause migrc from + * expanding its pending queue temporarily. + */ + if (!migrc_paused) { + migrc_pause(); + migrc_paused = true; + migrc_flush_free_folios(); + } + /* Caller is not willing to reclaim, we can't balance anything */ if (!can_direct_reclaim) goto nopage; @@ -4330,6 +4341,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, warn_alloc(gfp_mask, ac->nodemask, "page allocation failure: order:%u", order); got_pg: + if (migrc_paused) + migrc_resume(); return page; }