From patchwork Mon Jan 15 08:19:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13519379 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4EDDBC3DA79 for ; Mon, 15 Jan 2024 08:20:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BF67D6B008C; Mon, 15 Jan 2024 03:20:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BA82E6B0095; Mon, 15 Jan 2024 03:20:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A47FD6B0096; Mon, 15 Jan 2024 03:20:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 92B756B008C for ; Mon, 15 Jan 2024 03:20:08 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 61D36407F8 for ; Mon, 15 Jan 2024 08:20:08 +0000 (UTC) X-FDA: 81680847696.15.122806E Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf10.hostedemail.com (Postfix) with ESMTP id D50CDC0018 for ; Mon, 15 Jan 2024 08:20:05 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf10.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705306806; a=rsa-sha256; cv=none; b=wqlUpJVzd3nL5/hAs0cjkVqxN0nbQ3Nj+ROXuXjlcvkyblA2BktP7PblQP8WbIpXmip4aQ 1LC/axXkHtYYaj6WBU/YKXmjlRockzFF+69X79Zl8Z9liaXnmTah6S6V78eDYdPx6jO+wT b0z7lJdRIuHnTKvSqq2XUECFIPxKUw0= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf10.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705306806; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=eRD1/8o+qtEOTAKMY7ucioDJJtMI3+WpTL+ASvHX4To=; b=1PizzkrqU4xFY8AIWsdguQnyJGWltW4M8qol6Vvd9sw/Jtq5wcSmtz0fL+rIZtc3b7ZWog 5Ogtb/vFUrru3TogQLxImRUUZ8E8/sEhRJiG4LacPeK4VeGDSXgQ8Mdu1es6UNTKNCWzyX KE7Zre8RDaIMrdTAX6XG9RyAHFvJU2M= X-AuditID: a67dfc5b-d85ff70000001748-75-65a4eab4da69 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v6 1/7] x86/tlb: Add APIs manipulating tlb batch's arch data Date: Mon, 15 Jan 2024 17:19:47 +0900 Message-Id: <20240115081953.2521-2-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240115081953.2521-1-byungchul@sk.com> References: <20240115081953.2521-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrALMWRmVeSWpSXmKPExsXC9ZZnoe6WV0tSDXZN5LGYs34Nm8XnDf/Y LF5saGe0+Lr+F7PF0099LBaXd81hs7i35j+rxflda1ktdizdx2Rx6cACJovrux4yWhzvPcBk Mf/eZzaLzZumMlscnzKV0eL3D6COk7MmszgIenxv7WPx2DnrLrvHgk2lHptXaHks3vOSyWPT qk42j02fJrF7vDt3jt3jxIzfLB7zTgZ6vN93lc1j6y87j8ap19g8Pm+S83g3/y1bAH8Ul01K ak5mWWqRvl0CV8bt2y+ZClbxVSz50c7WwDiNp4uRg0NCwERiU4NxFyMnmLns9nJGEJtNQF3i xo2fzCC2iICZxMHWP+xdjFwczAIfmSRWf+9gAUkIC/hIvG16CVbEIqAq8fvGSTYQmxdo0Jxr DSwQQ+UlVm84AFbDKWAq8e7jcXYQWwiopuXuOlaQoRIC7ewST1adYodokJQ4uOIGywRG3gWM DKsYhTLzynITM3NM9DIq8zIr9JLzczcxAuNhWe2f6B2Mny4EH2IU4GBU4uH98XdxqhBrYllx Ze4hRgkOZiUR3uo7S1KFeFMSK6tSi/Lji0pzUosPMUpzsCiJ8xp9K08REkhPLEnNTk0tSC2C yTJxcEo1MHKmheSanGLfknzfh89jQ/eZsjpmuQmx9eud/hVnJZtsZ/kQe2OFs8FZ9mXLO9t5 1ut6+O5dG1KaLrp01cxoG0b7I23TWraZePttCe6JLrdg+WR3j9lw77Q5v7fFqU/82enPlnfM aG1vteG3mpMHBY5V8nhJ5Qp9dXwoLfizxfNee7/Du5PnlViKMxINtZiLihMBByu8QYMCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrBLMWRmVeSWpSXmKPExsXC5WfdrLvl1ZJUg9WvGC3mrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XhuSdZLS7vmsNmcW/Nf1aL87vWslrsWLqPyeLSgQVMFtd3PWS0 ON57gMli/r3PbBabN01ltjg+ZSqjxe8fQB0nZ01mcRDy+N7ax+Kxc9Zddo8Fm0o9Nq/Q8li8 5yWTx6ZVnWwemz5NYvd4d+4cu8eJGb9ZPOadDPR4v+8qm8fiFx+YPLb+svNonHqNzePzJjmP d/PfsgUIRHHZpKTmZJalFunbJXBl3L79kqlgFV/Fkh/tbA2M03i6GDk5JARMJJbdXs4IYrMJ qEvcuPGTGcQWETCTONj6h72LkYuDWeAjk8Tq7x0sIAlhAR+Jt00vwYpYBFQlft84yQZi8wIN mnOtgQViqLzE6g0HwGo4BUwl3n08zg5iCwHVtNxdxzqBkWsBI8MqRpHMvLLcxMwcU73i7IzK vMwKveT83E2MwPBeVvtn4g7GL5fdDzEKcDAq8fD++Ls4VYg1say4MvcQowQHs5IIb/WdJalC vCmJlVWpRfnxRaU5qcWHGKU5WJTEeb3CUxOEBNITS1KzU1MLUotgskwcnFINjLtmvf6U1GIv 4rvzC98p3/BJJ19arL37uFbRSl/dp/zHu6SePwxO4t9nhM2yOzQtaFtS8LvHJpqbfPhV7/jN 4vq65n7Nhgdfoict2ypnmriMt+8y9+NrTLv6N34rW26r5bxvadM6F/l1K/N3b+C5+Pr6GsmT jyZ/Y51lsswy8s37qunmGq51bvFKLMUZiYZazEXFiQBY1mM4awIAAA== X-CFilter-Loop: Reflected X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: D50CDC0018 X-Stat-Signature: yn3cbogw51o3fjke99d7tzj4jon5prmd X-Rspam-User: X-HE-Tag: 1705306805-900750 X-HE-Meta: U2FsdGVkX1+sTLR9TrOHYnY/zjhATIw48FyOGrXt7KR3Wv1J5OiROiO4i92NY1VMqJWEocrpuN4IJ+vNL6vM4LJS0AZuOTNMyv/aNSh9WAn0+WBz9aK7tnkXnuwTjAgJogH9c3RioOTXLpe2qwK6weYHpF+QQ1HHVtv7TgXLxOOLzfVwFB0Xc/J/kgPUNwnS9PKua33dts0z+IY4xQVcJPc3l3Pkj0NV2v1MNDg4zcJW6vpiKQz+Dn7SF88GKmYmf0FmaUfBoec4pv8sRtkTDfb4visj/LQCjAeFW5uvhp2rCG05y4xGua3lALv6M58Hpy4yYNQnpf2GZYZLvF35WqAtVvbPdLA/yPoBExMrgrBipiOU+KmbT661Fx7RjtH9Tolyijq2uN6dmNsBUGLA+soctbhBlEiS27H3A2p+CRbDhGO53lSSRyCmCKIO5qRVLWPYvj0Mg+CQTN7dZMhtddH0fKgw8U5rpZjJaXJ2pTGA0L01G+0sOaqn77Iom1sbXn//c7JwDnCxK6Ht/rNqmTsb/LqHSMWKaEWgq/Wn0THOMk0+7ofMZZmAWs04Jm+c/4ZxNno5Sza4gwY5PicAfMlQH+FMo9MukzsnhcZEnf8XV+70OKfon92Zx/sNnNujpvrkMnbr58vYyCWlyG3sxDxlaOrTFNelnyM9NZPYDQ60DOS4OFcu626T43JcJ7VRrRL6feOb8iQj3QN79dQxugewi+y76FLu/Ev8+Jpz/i0p0V2Z4t15KtZlzrN9F/QYxu102BwFedx3DlpJEluXuxTyYd6fXgVtPZ2RXlsUSS+Divay7MvmGod+15gZrr37+53zjWeD4cPNgxEHnJ1LNq2WcyqLbKU+ODpfY+EHMhSNEODa58ItMbKYYIGBZdyt32xfISGKQhUnfspWuhZp66ctUmeMtWSwsQdwmtfiSZUPF1HodgVIhhLiYxnitnmvPKEYjHjfm5mikV9yFa7 hr4af4Cr IJw/vx3UVpQWs6L3BVa0/MpLHaA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparation for migrc mechanism that needs to recognize read-only TLB entries during batched TLB flush by separating tlb batch's arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed. Migrc also needs to optimize CPUs to flush by clearing ones that have already performed TLB flush needed. To support it, added APIs manipulating arch data for x86. Signed-off-by: Byungchul Park --- arch/x86/include/asm/tlbflush.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 25726893c6f4..fa7e16dbeb44 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -293,6 +294,23 @@ static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm) extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); +static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch) +{ + cpumask_clear(&batch->cpumask); +} + +static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + cpumask_or(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + +static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + return cpumask_andnot(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + static inline bool pte_flags_need_flush(unsigned long oldflags, unsigned long newflags, bool ignore_access) From patchwork Mon Jan 15 08:19:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13519381 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14C57C3DA79 for ; Mon, 15 Jan 2024 08:20:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2CFF36B0096; Mon, 15 Jan 2024 03:20:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 280BE6B0099; Mon, 15 Jan 2024 03:20:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 121376B009A; Mon, 15 Jan 2024 03:20:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id EF48D6B0096 for ; Mon, 15 Jan 2024 03:20:10 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C9A3F80830 for ; Mon, 15 Jan 2024 08:20:10 +0000 (UTC) X-FDA: 81680847780.01.2E12866 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf10.hostedemail.com (Postfix) with ESMTP id E56EAC0026 for ; Mon, 15 Jan 2024 08:20:08 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf10.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705306809; a=rsa-sha256; cv=none; b=BOk0kY5cAbrEoO6C3NjsPSq4NTWkvXUWzfrTL1PGwZXbxYPliYPaalwrpEMdp/I5qJpVwg oBVF+g+D27v+Dk52j0umCCRoIEvMASmZwe4qHOFHhcuMSX4El1v0A9CxwcpT7J/yFae8v3 +zHxjEu9LXDnW9LJSux7zDIQrkHR6ek= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf10.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705306809; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=23+JjFD9uRPbyMWGqJTLiQYjg6Gkh8EQ/TB1jpahIpQ=; b=wvhFMrhIhiJOYueyQLm3mx4JWsGVFXERXqNwzLiNC9XZBu8zqrnsEJ87pAupOMZog1C/xB 96OuFEbVfm3NpgOw6rM9c5DG5iWNky7RYI5DqDGGmKxb7yaJzffvo638ItBpvTdWbSCzL5 qvopp8aqDGNdjPf9QvzcbvBCzgkPE20= X-AuditID: a67dfc5b-d85ff70000001748-7a-65a4eab45555 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v6 2/7] arm64: tlbflush: Add APIs manipulating tlb batch's arch data Date: Mon, 15 Jan 2024 17:19:48 +0900 Message-Id: <20240115081953.2521-3-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240115081953.2521-1-byungchul@sk.com> References: <20240115081953.2521-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrALMWRmVeSWpSXmKPExsXC9ZZnke6WV0tSDR4tELWYs34Nm8XnDf/Y LF5saGe0+Lr+F7PF0099LBaXd81hs7i35j+rxflda1ktdizdx2Rx6cACJovrux4yWhzvPcBk Mf/eZzaLzZumMlscnzKV0eL3D6COk7MmszgIenxv7WPx2DnrLrvHgk2lHptXaHks3vOSyWPT qk42j02fJrF7vDt3jt3jxIzfLB7zTgZ6vN93lc1j6y87j8ap19g8Pm+S83g3/y1bAH8Ul01K ak5mWWqRvl0CV8aevofsBbO5Kja9b2drYNzL0cXIySEhYCLx5+sEFhj75OkWZhCbTUBd4saN n2C2iICZxMHWP+xdjFwczAIfmSRWf+8AauDgEBYIkTi2WQakhkVAVeLX5nZGEJsXaM62nu/M EDPlJVZvOABmcwqYSrz7eJwdxBYCqmm5u44VZKaEQDu7xIfZh1khGiQlDq64wTKBkXcBI8Mq RqHMvLLcxMwcE72MyrzMCr3k/NxNjMB4WFb7J3oH46cLwYcYBTgYlXh4f/xdnCrEmlhWXJl7 iFGCg1lJhLf6zpJUId6UxMqq1KL8+KLSnNTiQ4zSHCxK4rxG38pThATSE0tSs1NTC1KLYLJM HJxSDYyeJU89j3v1/La5sGbRvAUqB1wEc2aphZgc+bJ6vneVnq7Fjf08s4oiunM4vTco6UnP 4Pa2en18wqR679BEvR9ZTay9fMn6KtP8ff/w8hWUqdxQs95W6xPRyW+3JOPdN2G5qzMrBB1D rzltr15xZc5NLRM3pZfrThcYTz37tnvirp+ftPdwrFFiKc5INNRiLipOBACmt96qgwIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrOLMWRmVeSWpSXmKPExsXC5WfdrLvl1ZJUgyXv2SzmrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XhuSdZLS7vmsNmcW/Nf1aL87vWslrsWLqPyeLSgQVMFtd3PWS0 ON57gMli/r3PbBabN01ltjg+ZSqjxe8fQB0nZ01mcRDy+N7ax+Kxc9Zddo8Fm0o9Nq/Q8li8 5yWTx6ZVnWwemz5NYvd4d+4cu8eJGb9ZPOadDPR4v+8qm8fiFx+YPLb+svNonHqNzePzJjmP d/PfsgUIRHHZpKTmZJalFunbJXBl7Ol7yF4wm6ti0/t2tgbGvRxdjJwcEgImEidPtzCD2GwC 6hI3bvwEs0UEzCQOtv5h72Lk4mAW+Mgksfp7B0sXIweHsECIxLHNMiA1LAKqEr82tzOC2LxA c7b1fGeGmCkvsXrDATCbU8BU4t3H4+wgthBQTcvddawTGLkWMDKsYhTJzCvLTczMMdUrzs6o zMus0EvOz93ECAzuZbV/Ju5g/HLZ/RCjAAejEg/vj7+LU4VYE8uKK3MPMUpwMCuJ8FbfWZIq xJuSWFmVWpQfX1Sak1p8iFGag0VJnNcrPDVBSCA9sSQ1OzW1ILUIJsvEwSnVwFj5aoFW5esc p8jDSq8/e2hqtpsb33t5OrMqRbH7jOyq7KrUvI3dHUXCDGbrxK493BEyV+31+r9PnnrFbtqm 8vy9wEtFk49lh7YY3D/cwmyzpcJqW7DV6YL3DqrOnJsThROWbFuyaX5VlIHa9dzNPy7VKmk3 qRTUzK1xP554/fWf+iVHym57bFZiKc5INNRiLipOBACps5EaagIAAA== X-CFilter-Loop: Reflected X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: E56EAC0026 X-Stat-Signature: x56stp36ohr9j133otbqhguzib39tqsm X-Rspam-User: X-HE-Tag: 1705306808-562383 X-HE-Meta: U2FsdGVkX1/+z+7rRl0AXCW+Xmktm86vDq1Hd9XtMSaUx8pNgAmkpHUNPkcc2w4aFU5db6APJYT0DwmTORksrmeFzvzhG+LLQvVxHTkvC+DciPLUkbDyJL0vixz/Ut2XXxp/yHqe/PBd6ogYH3eid6gaQsTOThHRMHaI3VlPxVUW9N1HGhgK0lzyfYPUTFp9FcBmdu4eMlOnSAufBTB2Li7k4eN/fsrqzP3SZ2lHip3C0PyX7W0UEoBkN+CRz0H38Bq83EcXK/onGUZe9Oii6ii/TG0vtQdsszOmIkddncjUCJACSSLUCJKILoojU8BzhS9QdGWzVh3M3Cj5BZxjBAR3Jn5y5lmkhn/5dlLH1AxIEW5mIVIbuBuYk9OatfsVpJ4+Dupg1NnNKbdEwUQ14dZtQwT9/56eVjAb6VpZPiAyzA7j1v+PX3y+cHR2okcEWCRGOrXQIhoJNZtLeKinSEj1TdfNnB3OYjKPczHdAGYTdClLU7dik2Cz2iNuchVPJ3gCtGcdquox4SVPYrnaAux4TxF5oevwqwm4JPAarBAjjZOvFfzeSeQKvQBm4vqnWg9z8eYYVqLJLwkQq7sGHLUNqCfNsRBvS+ggqXw1KrwvxTXx1ou460s82ubCsv/MuscC8qIUROCsOjQWfbKX3Lmgt7cN933p2lQ1cv3Yg1axpPkr8V3lXE3IA09oR+5ESjWyA0mNQlSFU/pcE3PuVFHW73y+tRJ6jsUesqeabIU9qWnB2oaypVUlr84rYqfzKoVsQTi82Gjy7M5gFV0YzvCDzE85liHptSLTF5czL1OoPUO4ze513yQcSxMiHHR7cemmxCYS+sq7JDKvhYRevT/tYReagnyYTVoAVgTrJG55Oz2JZiO6u8ptaPF1SMwkbRTo9D7pLTeUSSqG9tGD2npBZF7OgSaKE1XhhW6WOGIsm+TWZGBhdYbXX1UFn2FkMKIkMh8CAEitNY7ir3f B1Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparation for migrc mechanism that requires to manipulate tlb batch's arch data. Even though arm64 does nothing with it, arch with CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH should provide the APIs. Signed-off-by: Byungchul Park --- arch/arm64/include/asm/tlbflush.h | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index bb2c2833a987..4f2094843e7a 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -328,6 +328,25 @@ static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) dsb(ish); } +static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch) +{ + /* nothing to do */ +} + +static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + /* nothing to do */ +} + +static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + /* nothing to do */ + + return false; +} + /* * This is meant to avoid soft lock-ups on large TLB flushing ranges and not * necessarily a performance improvement. From patchwork Mon Jan 15 08:19:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13519382 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2856FC47258 for ; Mon, 15 Jan 2024 08:20:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4CB136B0099; Mon, 15 Jan 2024 03:20:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 408176B009A; Mon, 15 Jan 2024 03:20:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 27FDE6B009B; Mon, 15 Jan 2024 03:20:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 187E66B0099 for ; Mon, 15 Jan 2024 03:20:12 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id DF497407FF for ; Mon, 15 Jan 2024 08:20:11 +0000 (UTC) X-FDA: 81680847822.25.F2B45CC Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf02.hostedemail.com (Postfix) with ESMTP id 075F380011 for ; Mon, 15 Jan 2024 08:20:09 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf02.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705306810; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=deHIYqkuaMMowWAwSsScVOmIndLCnwrqbZwxQiYz+QA=; b=u7GY04k7TA88lgkG872ExUI0fAlPeafJt1nrODFX7uXW+W0bwko6nvKiZcY6Okw9cudl4i qyz991gn6i+62Y5763GNRtsNApN0Uc+wZauBhXNzke02rY4Jemv3cxj9W+bIlf0vffLO2p 2COJ89QNGJhLi+ETmaDZsI0oyV9+XUw= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf02.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705306810; a=rsa-sha256; cv=none; b=5QAwkGG20IaJmGyk3+y/9PSCV0GW6BEH7oHgk4FiGFIF+Plt+2k+6Ncwg3XZFvWiYYWvCy TooCF6g+qveUjJbQueMf/Zsq9mKXdMmfAYu8pDSF9ZXJ4EvrvkzGc9Mu4v3r56WaLvYCdQ hC11KPInhrLXAE0xC5VM0gobGsIsbSU= X-AuditID: a67dfc5b-d85ff70000001748-7f-65a4eab48a36 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v6 3/7] mm/rmap: Recognize read-only TLB entries during batched TLB flush Date: Mon, 15 Jan 2024 17:19:49 +0900 Message-Id: <20240115081953.2521-4-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240115081953.2521-1-byungchul@sk.com> References: <20240115081953.2521-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrALMWRmVeSWpSXmKPExsXC9ZZnoe6WV0tSDboXi1nMWb+GzeLzhn9s Fi82tDNafF3/i9ni6ac+FovLu+awWdxb85/V4vyutawWO5buY7K4dGABk8X1XQ8ZLY73HmCy mH/vM5vF5k1TmS2OT5nKaPH7B1DHyVmTWRwEPb639rF47Jx1l91jwaZSj80rtDwW73nJ5LFp VSebx6ZPk9g93p07x+5xYsZvFo95JwM93u+7yuax9ZedR+PUa2wenzfJebyb/5YtgD+KyyYl NSezLLVI3y6BK+PrPKeC49IVHz7tYGxgfCDWxcjJISFgIvHl7ERWGPvasXWMIDabgLrEjRs/ mUFsEQEziYOtf9i7GLk4mAU+Mkms/t7BApIQFoiUaFixF6yBRUBV4tjmN2ANvECDtm2dATVU XmL1hgNgcU4BU4l3H4+zg9hCQDUtd9exggyVEGhml7j+aAlUg6TEwRU3WCYw8i5gZFjFKJSZ V5abmJljopdRmZdZoZecn7uJERgPy2r/RO9g/HQh+BCjAAejEg/vj7+LU4VYE8uKK3MPMUpw MCuJ8FbfWZIqxJuSWFmVWpQfX1Sak1p8iFGag0VJnNfoW3mKkEB6YklqdmpqQWoRTJaJg1Oq gVHzRf2p6w7TWh/tPSadJTRDa14q24VzVforrpY+Vrrp+DqTIX6WrdMFDcMVM+reZp698f+z y5rapKVWeyYutX2xjbEm9UWMjt/ZK7LyMflerzS61bnmO0371bj32FfNCNeLNgoLD6feu3tC 0uKxs+aE5E17k1QXJ0p8XcqwVSawVOj5Zp53nPJKLMUZiYZazEXFiQBlepJngwIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrOLMWRmVeSWpSXmKPExsXC5WfdrLvl1ZJUg+5VnBZz1q9hs/i84R+b xYsN7YwWX9f/YrZ4+qmPxeLw3JOsFpd3zWGzuLfmP6vF+V1rWS12LN3HZHHpwAImi+u7HjJa HO89wGQx/95nNovNm6YyWxyfMpXR4vcPoI6TsyazOAh5fG/tY/HYOesuu8eCTaUem1doeSze 85LJY9OqTjaPTZ8msXu8O3eO3ePEjN8sHvNOBnq833eVzWPxiw9MHlt/2Xk0Tr3G5vF5k5zH u/lv2QIEorhsUlJzMstSi/TtErgyvs5zKjguXfHh0w7GBsYHYl2MnBwSAiYS146tYwSx2QTU JW7c+MkMYosImEkcbP3D3sXIxcEs8JFJYvX3DhaQhLBApETDir1gDSwCqhLHNr8Ba+AFGrRt 6wxWiKHyEqs3HACLcwqYSrz7eJwdxBYCqmm5u451AiPXAkaGVYwimXlluYmZOaZ6xdkZlXmZ FXrJ+bmbGIHBvaz2z8QdjF8uux9iFOBgVOLh/fF3caoQa2JZcWXuIUYJDmYlEd7qO0tShXhT EiurUovy44tKc1KLDzFKc7AoifN6hacmCAmkJ5akZqemFqQWwWSZODilGhgbr0o4HFm85Kfq gQu/F6bd3ZnyYU7heQ/jhRv+XX/i9v7Dyplc1+Z/TvJpuKbSkDZ5h9VL/718xgfjHHtifQ5k /o0IP6rZ99dde3+x3I05j08dDlsUkhimK2t0VpypRvMU37q+NBd37ZSGyGMPb7tXfjdXm3BL WPDfoRmnyy+5T75r/Gu++ywtJZbijERDLeai4kQAJQwUJmoCAAA= X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 075F380011 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: burdypz3bhwru95wdjzcthpp68o9xktx X-HE-Tag: 1705306809-382790 X-HE-Meta: U2FsdGVkX19fg5chTQu5cn1Xg6Xi0RUyJy5tQe8IccFOFUAQp/8l0V7w6bjoH3aHJf1E8s5+xk83eT7xWgFnyOJfSQuO5wGdZ1+Qt0ky4Yf6h2+ZJS7gHK5MOKgLyeDAvolBK/uuvkDM29LwvWj2xq0fNniZwKaa7LN8oBtzsUmfKAzY6averYlNmlFn4jBe8dS6fjY0FBYYWJ6eGqDz2I2n3yBsO3GullzOhPNX4ZVH94Ihmc+QUdC+vrS9bg7TucOeioS80auPEwF8uyIacF5c6aNo4AG4TxDOoLoCYdeRoW0+tebeVRac6M9zyyZ+rSshZhh1Z4fBlJ52bt2IYT4inmr/tWbHsVvexQYEoplwrC23bT781dvWLxqLrwiqzidnjHIH7obpTRd48eadxDnrwGSePPDU3mvuF5hZrYNs59zJlZwa9pvBV/MDBnXKfv/RsgEfLIHi1zPOvhZH4tvK9ctYTCkDfnmUfhm3QaVt6EUcQtjgtGPzNInjccwh4WGtvRiAKDZ658wksTDcpUWMvgbhog7H8rmA5c8/62G6QgOe/0F+JA/BF5whKGJeBXwjicnTB2oCJu9js/xEjrfQ7h0wXzwSKXX0iWStO5QZSDdk02wQuuTxaVLeFfhsWuwHa1Ezt3fSw4DgAr0wCx78DzaEAG+Tb0XgfGQDLWhGblfRuNTUxWF6DxxgAlG3lVSZU1j1uPGqFMgZataLiCpUQYZjESKeGKq3aKSWX5VLPlMXRHM5UooMzPvMTxf3f37bnIVECNcvym+vnODdY5V6Y55/Sk0oTl0v89aMF4BJ+KIFE/SfbTx0HE2s+f95KNi8Z0IktkhJmrmyKkQ4Whyc1GHZYdvn4/ux3+KhDP9n3HoRa0u62D9PU4rV8DuiHDQGOnVQAy/ZQrJ6U0exP2TFsm1AGeiDumYgjorx4t99xVFu2MQ20jfGnqurmWrp3e66bRapoxjIHTaMyvE Hzg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Functionally, no change. This is a preparation for migrc mechanism that requires to recognize read-only TLB entries and makes use of them to batch more aggressively. Plus, the newly introduced API, fold_ubc() will be used by migrc mechanism when manipulating tlb batch data. Signed-off-by: Byungchul Park --- include/linux/sched.h | 1 + mm/internal.h | 4 ++++ mm/rmap.c | 31 ++++++++++++++++++++++++++++++- 3 files changed, 35 insertions(+), 1 deletion(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 292c31697248..0317e7a65151 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1328,6 +1328,7 @@ struct task_struct { #endif struct tlbflush_unmap_batch tlb_ubc; + struct tlbflush_unmap_batch tlb_ubc_ro; /* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe; diff --git a/mm/internal.h b/mm/internal.h index b61034bd50f5..b880f1e78700 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -923,6 +923,7 @@ extern struct workqueue_struct *mm_percpu_wq; void try_to_unmap_flush(void); void try_to_unmap_flush_dirty(void); void flush_tlb_batched_pending(struct mm_struct *mm); +void fold_ubc(struct tlbflush_unmap_batch *dst, struct tlbflush_unmap_batch *src); #else static inline void try_to_unmap_flush(void) { @@ -933,6 +934,9 @@ static inline void try_to_unmap_flush_dirty(void) static inline void flush_tlb_batched_pending(struct mm_struct *mm) { } +static inline void fold_ubc(struct tlbflush_unmap_batch *dst, struct tlbflush_unmap_batch *src) +{ +} #endif /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ extern const struct trace_print_flags pageflag_names[]; diff --git a/mm/rmap.c b/mm/rmap.c index 7a27a2b41802..da36f23ff7b0 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -605,6 +605,28 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, } #ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH + +void fold_ubc(struct tlbflush_unmap_batch *dst, + struct tlbflush_unmap_batch *src) +{ + if (!src->flush_required) + return; + + /* + * Fold src to dst. + */ + arch_tlbbatch_fold(&dst->arch, &src->arch); + dst->writable = dst->writable || src->writable; + dst->flush_required = true; + + /* + * Reset src. + */ + arch_tlbbatch_clear(&src->arch); + src->flush_required = false; + src->writable = false; +} + /* * Flush TLB entries for recently unmapped pages from remote CPUs. It is * important if a PTE was dirty when it was unmapped that it's flushed @@ -614,7 +636,9 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, void try_to_unmap_flush(void) { struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc_ro = ¤t->tlb_ubc_ro; + fold_ubc(tlb_ubc, tlb_ubc_ro); if (!tlb_ubc->flush_required) return; @@ -645,13 +669,18 @@ void try_to_unmap_flush_dirty(void) static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, unsigned long uaddr) { - struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc; int batch; bool writable = pte_dirty(pteval); if (!pte_accessible(mm, pteval)) return; + if (pte_write(pteval) || writable) + tlb_ubc = ¤t->tlb_ubc; + else + tlb_ubc = ¤t->tlb_ubc_ro; + arch_tlbbatch_add_pending(&tlb_ubc->arch, mm, uaddr); tlb_ubc->flush_required = true; From patchwork Mon Jan 15 08:19:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13519383 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0389FC47422 for ; Mon, 15 Jan 2024 08:20:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 84D526B009A; Mon, 15 Jan 2024 03:20:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7FE8A6B009B; Mon, 15 Jan 2024 03:20:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 677886B009C; Mon, 15 Jan 2024 03:20:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 57E536B009A for ; Mon, 15 Jan 2024 03:20:13 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 1E58E120217 for ; Mon, 15 Jan 2024 08:20:13 +0000 (UTC) X-FDA: 81680847906.19.E777CC9 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf10.hostedemail.com (Postfix) with ESMTP id 2FA51C0028 for ; Mon, 15 Jan 2024 08:20:10 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf10.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705306811; a=rsa-sha256; cv=none; b=oC+hGVWJUzrmyVIC0PFSYhNDB16R9CG+QyxMq5ombLfSRt7k4/hN0hN+qcUpx/wSNuoWhi JCHjPlVa3wI+Blx832qLk9cFik6LAx7lLqFqGJzNm3E6+2xqPTtlsXnjWfNOurXSu3R4KR 3YShAIMpIvQ2JFwqW+kVPVO2ISaW7QM= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf10.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705306811; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=hj/OU3wk3OzsqULfJnD2mvtgwTJyJb/ndDOd9gzDe6E=; b=nj8AslgA2DEd8JNlnJc86AoRNbUmL3PKSlbzzWgpZNe0EAEwUx5t1uqhxbNNhTOIOywtpW k3rCCm9kA+yyuo+fQLLc0IXYXbuH5s/x3+/OXGuuk3np3fVsQRhcrffiRdtLVYApoeb0pM QTlb/qcHTOizXXesNESXQJy1bg7DDKI= X-AuditID: a67dfc5b-d85ff70000001748-84-65a4eab4f79b From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v6 4/7] mm: Separate move/undo doing on folio list from migrate_pages_batch() Date: Mon, 15 Jan 2024 17:19:50 +0900 Message-Id: <20240115081953.2521-5-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240115081953.2521-1-byungchul@sk.com> References: <20240115081953.2521-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrPLMWRmVeSWpSXmKPExsXC9ZZnoe6WV0tSDbZfFbOYs34Nm8XnDf/Y LF5saGe0+Lr+F7PF0099LBaXd81hs7i35j+rxflda1ktdizdx2Rx6cACJovrux4yWhzvPcBk Mf/eZzaLzZumMlscnzKV0eL3D6COk7MmszgIenxv7WPx2DnrLrvHgk2lHptXaHks3vOSyWPT qk42j02fJrF7vDt3jt3jxIzfLB7zTgZ6vN93lc1j6y87j8ap19g8Pm+S83g3/y1bAH8Ul01K ak5mWWqRvl0CV8aN93fZC/p1KhbNecjewLhSuYuRk0NCwERiS/8KRhh76co1zCA2m4C6xI0b P8FsEQEziYOtf9i7GLk4mAU+Mkms/t7B0sXIwSEsECvR0q4JUsMioCoxf/1+sHpeoDlr7jUz QcyUl1i94QBYnFPAVOLdx+PsILYQUE3L3XWsEDXN7BJTngZA2JISB1fcYJnAyLuAkWEVo1Bm XlluYmaOiV5GZV5mhV5yfu4mRmA0LKv9E72D8dOF4EOMAhyMSjy8P/4uThViTSwrrsw9xCjB wawkwlt9Z0mqEG9KYmVValF+fFFpTmrxIUZpDhYlcV6jb+UpQgLpiSWp2ampBalFMFkmDk6p BsayDdmKIvZJfT1vmndotTML1bskNc9m3rehRtL8XVaftK/b6Q0Tj2Zwz+HfX7PJf/VKltXT s9bGHk+XjF4QfyX70aTtipv3PHT/PIN9n63qjsaXL/95blZcsr5hs5L5TqE2I0f+Peddd88S 6HANWj939qsDm79IPop6plt3XtJoSoiqyvM5HNFKLMUZiYZazEXFiQAY5wXvggIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrOLMWRmVeSWpSXmKPExsXC5WfdrLvl1ZJUg3tPuC3mrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XhuSdZLS7vmsNmcW/Nf1aL87vWslrsWLqPyeLSgQVMFtd3PWS0 ON57gMli/r3PbBabN01ltjg+ZSqjxe8fQB0nZ01mcRDy+N7ax+Kxc9Zddo8Fm0o9Nq/Q8li8 5yWTx6ZVnWwemz5NYvd4d+4cu8eJGb9ZPOadDPR4v+8qm8fiFx+YPLb+svNonHqNzePzJjmP d/PfsgUIRHHZpKTmZJalFunbJXBl3Hh/l72gX6di0ZyH7A2MK5W7GDk5JARMJJauXMMMYrMJ qEvcuPETzBYRMJM42PqHvYuRi4NZ4COTxOrvHSxdjBwcwgKxEi3tmiA1LAKqEvPX7wer5wWa s+ZeMxPETHmJ1RsOgMU5BUwl3n08zg5iCwHVtNxdxzqBkWsBI8MqRpHMvLLcxMwcU73i7IzK vMwKveT83E2MwOBeVvtn4g7GL5fdDzEKcDAq8fD++Ls4VYg1say4MvcQowQHs5IIb/WdJalC vCmJlVWpRfnxRaU5qcWHGKU5WJTEeb3CUxOEBNITS1KzU1MLUotgskwcnFINjDM3VMxZ7fhZ I2sjq+uKlefV+7Nn/+x6//RNsMK6hi3ciYnzD2/31Xc1/bPJP+1uy9HdSVJ1QnW5jEcmCNV9 WeL46ce13/2BUyfeCK5MSTepqb7CKKJn9cj528e4+m9yYqp81zR/vzHdG9a2ffu1e2qqz80b F/w4nOX/r0w4WP8Al21PhjT3LyWW4oxEQy3mouJEAKF8z8BqAgAA X-CFilter-Loop: Reflected X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 2FA51C0028 X-Stat-Signature: hewq6ycmqzk7purpohzgr1q59e81ci33 X-Rspam-User: X-HE-Tag: 1705306810-915934 X-HE-Meta: U2FsdGVkX1+gagxj56zh8m4/NqM5qBWwfQN4FBhqBbvIOn+6YM9WXnJp96ZQcLwf6xWBFEEHzXo7I1pcja8He2k9YIsTrDvumkHOJX91tZFjehhyBZE1sPZKJxDSVb+5mWqBXTdkWYInAHj6/V6ctjC2yJsjhHl6qZErfOZoGJC6Hxqtf9m64SbDg65HqqWHPhP8Vqq5lBoZiVI+bGwaN3MLI+SYn9n85Y8ikcGu+KE6bS0OGBtqikTTBmSkxX8HhMbOB/vlzcp2uwbgUBB5n2MEk44RD+Scy9ZFVHTrhGcVpMGFVmJnZ9m9anizmcdCLPYqPZk2Gxk12XEYQtrsJxiDxLN/lPWulbqMRKhtxtVz+Gdj23fflCBRRSf7wwqef/vyIzmnRyao31dtKtzP2fx2eGs6+EML4anvdc4AqNa/cNQffdKZwLqb70JZltD4xTrVsV8o79t9i8zkt+V36ItoR5Aa+NJZRRz7DFV8GzioKd8LzCCKdWLe4Igvb4eCdEfwnFLvU6CA/9EAgjtAEXYPkam2dU8SyOqc40v/mA5+aFCOG7zhLrTBpobk1SF3xy9ubo+Jhrmjz/EA8rf3h3uJ50zAQNOy4bL817y75ASGPFOtv7LHEpuHioy+2zEBSOyz9lCTJQsj6BBTurYFiKI/e4g4Myj3HdB+fWGcEfdz961K0lfrAmSNXoOXucF8SRXdv/XMhTHjgbAJUCuG7zgTw7IfENjz7J7WTBnejMrMAp4kHVcN+WBUXWsAYAv5DbLK1idS28NeElunf7l59fCMzT+Hm2jUF19p/1A4ddTASnz9YvsHl/6pYZ9MCEZ1RZEttHzoG32Arlqaaoinmntnxt6XIzjSZ4gEBqaXnvckjq1QTaOiJTpLqGVu4S7sTFX7LQgG+5TAfm7HD1RE25pIzlXhgkRq3ptUeIjgoDGto6wmvMyThyAC45VwQvL89EOcU4b7Y8HOK5+3DO+ 0JafUmkQ o1MNcpj4BH1fzQGNWEhnKxSfqZlg9P5uKWPk8 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Functionally, no change. This is a preparation for migrc mechanism that requires to use separate folio lists for its own handling at migration. Refactored migrate_pages_batch() and separated move and undo parts operating on folio list, from migrate_pages_batch(). Signed-off-by: Byungchul Park --- mm/migrate.c | 134 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 83 insertions(+), 51 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 397f2a6e34cb..bbe1ecef4956 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1611,6 +1611,81 @@ static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio, return nr_failed; } +static void migrate_folios_move(struct list_head *src_folios, + struct list_head *dst_folios, + free_folio_t put_new_folio, unsigned long private, + enum migrate_mode mode, int reason, + struct list_head *ret_folios, + struct migrate_pages_stats *stats, + int *retry, int *thp_retry, int *nr_failed, + int *nr_retry_pages) +{ + struct folio *folio, *folio2, *dst, *dst2; + bool is_thp; + int nr_pages; + int rc; + + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio); + nr_pages = folio_nr_pages(folio); + + cond_resched(); + + rc = migrate_folio_move(put_new_folio, private, + folio, dst, mode, + reason, ret_folios); + /* + * The rules are: + * Success: folio will be freed + * -EAGAIN: stay on the unmap_folios list + * Other errno: put on ret_folios list + */ + switch(rc) { + case -EAGAIN: + *retry += 1; + *thp_retry += is_thp; + *nr_retry_pages += nr_pages; + break; + case MIGRATEPAGE_SUCCESS: + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; + break; + default: + *nr_failed += 1; + stats->nr_thp_failed += is_thp; + stats->nr_failed_pages += nr_pages; + break; + } + dst = dst2; + dst2 = list_next_entry(dst, lru); + } +} + +static void migrate_folios_undo(struct list_head *src_folios, + struct list_head *dst_folios, + free_folio_t put_new_folio, unsigned long private, + struct list_head *ret_folios) +{ + struct folio *folio, *folio2, *dst, *dst2; + + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + int old_page_state = 0; + struct anon_vma *anon_vma = NULL; + + __migrate_folio_extract(dst, &old_page_state, &anon_vma); + migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED, + anon_vma, true, ret_folios); + list_del(&dst->lru); + migrate_folio_undo_dst(dst, true, put_new_folio, private); + dst = dst2; + dst2 = list_next_entry(dst, lru); + } +} + /* * migrate_pages_batch() first unmaps folios in the from list as many as * possible, then move the unmapped folios. @@ -1633,7 +1708,7 @@ static int migrate_pages_batch(struct list_head *from, int pass = 0; bool is_thp = false; bool is_large = false; - struct folio *folio, *folio2, *dst = NULL, *dst2; + struct folio *folio, *folio2, *dst = NULL; int rc, rc_saved = 0, nr_pages; LIST_HEAD(unmap_folios); LIST_HEAD(dst_folios); @@ -1769,42 +1844,11 @@ static int migrate_pages_batch(struct list_head *from, thp_retry = 0; nr_retry_pages = 0; - dst = list_first_entry(&dst_folios, struct folio, lru); - dst2 = list_next_entry(dst, lru); - list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { - is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio); - nr_pages = folio_nr_pages(folio); - - cond_resched(); - - rc = migrate_folio_move(put_new_folio, private, - folio, dst, mode, - reason, ret_folios); - /* - * The rules are: - * Success: folio will be freed - * -EAGAIN: stay on the unmap_folios list - * Other errno: put on ret_folios list - */ - switch(rc) { - case -EAGAIN: - retry++; - thp_retry += is_thp; - nr_retry_pages += nr_pages; - break; - case MIGRATEPAGE_SUCCESS: - stats->nr_succeeded += nr_pages; - stats->nr_thp_succeeded += is_thp; - break; - default: - nr_failed++; - stats->nr_thp_failed += is_thp; - stats->nr_failed_pages += nr_pages; - break; - } - dst = dst2; - dst2 = list_next_entry(dst, lru); - } + /* Move the unmapped folios */ + migrate_folios_move(&unmap_folios, &dst_folios, + put_new_folio, private, mode, reason, + ret_folios, stats, &retry, &thp_retry, + &nr_failed, &nr_retry_pages); } nr_failed += retry; stats->nr_thp_failed += thp_retry; @@ -1813,20 +1857,8 @@ static int migrate_pages_batch(struct list_head *from, rc = rc_saved ? : nr_failed; out: /* Cleanup remaining folios */ - dst = list_first_entry(&dst_folios, struct folio, lru); - dst2 = list_next_entry(dst, lru); - list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { - int old_page_state = 0; - struct anon_vma *anon_vma = NULL; - - __migrate_folio_extract(dst, &old_page_state, &anon_vma); - migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED, - anon_vma, true, ret_folios); - list_del(&dst->lru); - migrate_folio_undo_dst(dst, true, put_new_folio, private); - dst = dst2; - dst2 = list_next_entry(dst, lru); - } + migrate_folios_undo(&unmap_folios, &dst_folios, + put_new_folio, private, ret_folios); return rc; } From patchwork Mon Jan 15 08:19:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13519384 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 169CDC3DA79 for ; Mon, 15 Jan 2024 08:20:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7D9C26B009B; Mon, 15 Jan 2024 03:20:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 78A7E6B009C; Mon, 15 Jan 2024 03:20:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5DBC66B009D; Mon, 15 Jan 2024 03:20:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4CF636B009B for ; Mon, 15 Jan 2024 03:20:14 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 1E18916045C for ; Mon, 15 Jan 2024 08:20:14 +0000 (UTC) X-FDA: 81680847948.09.4497AB6 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf02.hostedemail.com (Postfix) with ESMTP id 388CB80011 for ; Mon, 15 Jan 2024 08:20:11 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf02.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705306812; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=xyww7pMUuLyPBfG9ZuK6U6g9ZtqJWjn+tCNK2s25zA8=; b=dG8Uph3MxPgeztE8KaFCXDmudyRrjWTnBrF4eOMB9HSQLZP6z6jIdcYyt64Ip5H+3+jYda 0VKISR6z3tmN29TpMB/DNGouUfJ+YixL8h5fhnpKzPGwpsh3Oc1CXjDdwAusIfTLmOxEe1 T22w3fmP9rOy0mz7/Q00AKE/HPTY4SI= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf02.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705306812; a=rsa-sha256; cv=none; b=X+A0Vf0Kg0TH3Nmu7rVKMFjGvqIc4ADLm2obosTaciEhiQzK63QP1UcXE3ydhkluBWat9Z oKmDEIZWd8iSrtjxDRk79apvfNis6ttOEGgMux0qP3TSSYVbJwpG2oFcucTPM0qyXgmrTY +9bDn4sPcgPHZWKkqYdBRnzjXs/Qgwo= X-AuditID: a67dfc5b-d85ff70000001748-89-65a4eab45724 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v6 5/7] mm: Add APIs to free a folio directly to the buddy bypassing pcp Date: Mon, 15 Jan 2024 17:19:51 +0900 Message-Id: <20240115081953.2521-6-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240115081953.2521-1-byungchul@sk.com> References: <20240115081953.2521-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrPLMWRmVeSWpSXmKPExsXC9ZZnke6WV0tSDebPkrWYs34Nm8XnDf/Y LF5saGe0+Lr+F7PF0099LBaXd81hs7i35j+rxflda1ktdizdx2Rx6cACJovrux4yWhzvPcBk Mf/eZzaLzZumMlscnzKV0eL3D6COk7MmszgIenxv7WPx2DnrLrvHgk2lHptXaHks3vOSyWPT qk42j02fJrF7vDt3jt3jxIzfLB7zTgZ6vN93lc1j6y87j8ap19g8Pm+S83g3/y1bAH8Ul01K ak5mWWqRvl0CV0bHdbaCP1IVh4/vZm5g/CHaxcjJISFgIrHo02N2GHvDsR1MIDabgLrEjRs/ mUFsEQEziYOtf4BquDiYBT4ySaz+3sECkhAWiJD4c/ksWBGLgKpE5789bF2MHBy8QIOWnayH mCkvsXrDAbASTgFTiXcfj4PtEgIqabm7jhVkpoRAM7vE8v9rWSEaJCUOrrjBMoGRdwEjwypG ocy8stzEzBwTvYzKvMwKveT83E2MwGhYVvsnegfjpwvBhxgFOBiVeHh//F2cKsSaWFZcmXuI UYKDWUmEt/rOklQh3pTEyqrUovz4otKc1OJDjNIcLErivEbfylOEBNITS1KzU1MLUotgskwc nFINjE4zODo3GK5az23z2HHVzVXObm9F1jPF3+1zr/slsXj3tvwk8SVl06eFxX00F1BsyWry WlclvijK22yjxDZPb6YvPh0JfW6zBSNPhXDE3jhgWPJUnFE7r/L2YftoHnPV1XfX7LxpJfTm 75PUDy9ldvD+ruW79sRh6+IdmcJmZ4+crwlfyljzUImlOCPRUIu5qDgRAEaPGL6CAgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrGLMWRmVeSWpSXmKPExsXC5WfdrLvl1ZJUg9ZmQYs569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfVdDxkt jvceYLKYf+8zm8XmTVOZLY5Pmcpo8fsHUMfJWZNZHIQ8vrf2sXjsnHWX3WPBplKPzSu0PBbv ecnksWlVJ5vHpk+T2D3enTvH7nFixm8Wj3knAz3e77vK5rH4xQcmj62/7Dwap15j8/i8Sc7j 3fy3bAECUVw2Kak5mWWpRfp2CVwZHdfZCv5IVRw+vpu5gfGHaBcjJ4eEgInEhmM7mEBsNgF1 iRs3fjKD2CICZhIHW/+wdzFycTALfGSSWP29gwUkISwQIfHn8lmwIhYBVYnOf3vYuhg5OHiB Bi07WQ8xU15i9YYDYCWcAqYS7z4eZwexhYBKWu6uY53AyLWAkWEVo0hmXlluYmaOqV5xdkZl XmaFXnJ+7iZGYGgvq/0zcQfjl8vuhxgFOBiVeHh//F2cKsSaWFZcmXuIUYKDWUmEt/rOklQh 3pTEyqrUovz4otKc1OJDjNIcLErivF7hqQlCAumJJanZqakFqUUwWSYOTqkGRvX/9/84uPN9 eJAfK+R/+bb8Bbfsb9eaXhVMvvw3UlknWe7jkliWlntncvxVLkmfl1NY09grNDWU/WmJ7uUU J6WJPhpfz/S6HYsPaGeT2154+35Lt2PSgpyJa9dkeMn/9w1Otk7fMrtuM0/Ij3mRS3fMKnst YPzwS6dvt66K8+QTU5ZOXHi2TImlOCPRUIu5qDgRALgW8+dpAgAA X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 388CB80011 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: w6mh9d47ozpe5zoqnzswe4zn6m4s71n7 X-HE-Tag: 1705306811-494182 X-HE-Meta: U2FsdGVkX18hJXmrrAovllGvOuD6ZfeJY3bj8Wrc5b0oZj/VCc3zKMGvb5Kiz676AVNbvb716QvHukqv4C5Dmd8njDtyzrU4vntpp7qitsIbkMJnZycJ2Vmf5W4gli4nAnvKC8ibteZjT5qRfB8MXxYkKNYdp6cO7dfJJyewBTP90oiapw+ITKMuW47IEPpBRKMfIZoHJsxj/BmTqFmG2xVTzraq7iqbZSDWpMY7iMxEkjF4Jdu2QTtzr2I8uY9PSwJj48Y4z7PVq6JsCSPrImj4FfPTNS325pl11JyxmvK/tXjs5N6vRqHSUMBlupv68eylCsJlx8X7U51tFec8ZruxKMGu3ZT1W6jM6xn1Iv97OkFzPZZ6M5E8CvoNxekOSMdG8mQWJk/FLPvkrh2seN42EVqwlpS/OZ6WipR1E+7tB13b7cJpcrW2Af/XGBEomW0sJqbolGLFLiHO3xUR866YOBVFqP6O1qXTi2wn2k/nagl/NtI+qCNiMU9sW3Crd2V3QyXhnkFVNj0XY3gWfRPeo1qM5nvTGtsHog0kRAKt3BEqe/r/iQxd0e920Foy/IPs28lxAyCLVtFuPPFprtdtgWf6VOmCWOJS85bY8iFi6cdOAfqXXDUg+61DNg17knPWAkNh3lAHnilesrw+pF0QYSQTeRZmRYY43TKxsShP90UTLHcE/uVhqgxeDWhAm0uRMgYUhFA8Z5TJOhC0rqp/z55fRf+0dYE4R4aJsWUHmEp2V/0T+y6owjD8KLGxilf/ZDwNUUmpXdNpI5B1uFvi9ugtqHcJRJoMSHveZerth/Fx9Wf4SXuz/PvASG62nVMTimaDtfoX2OoBspP6s4VJIiRw5ojC+g22qY3lclUHzP9sb5rY8kJYGLnLnI8Fi7mBphDpuwVKARRU5KPISn6BEQ6oMHCzTA76Iyn+Fh/04BhR7y7y19UKGamj5mfSGYb+SH3v0P0PehCV6MM kVTtWTA8 +oNWO2ESYB48i5vNS6iuhbTulAQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparation for migrc mechanism that frees folios at a better time later, rather than the moment migrating folios. The folios freed by migrc are too old to keep in pcp. Signed-off-by: Byungchul Park --- include/linux/mm.h | 23 +++++++++++++++++++++++ mm/internal.h | 1 + mm/page_alloc.c | 10 ++++++++++ mm/swap.c | 7 +++++++ 4 files changed, 41 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index da5219b48d52..fc0581cce3a7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1284,6 +1284,7 @@ static inline struct folio *virt_to_folio(const void *x) } void __folio_put(struct folio *folio); +void __folio_put_small_nopcp(struct folio *folio); void put_pages_list(struct list_head *pages); @@ -1483,6 +1484,28 @@ static inline void folio_put(struct folio *folio) __folio_put(folio); } +/** + * folio_put_small_nopcp - Decrement the reference count on a folio. + * @folio: The folio. + * + * This is only for a single page folio to release directly to the buddy + * allocator bypassing pcp. + * + * If the folio's reference count reaches zero, the memory will be + * released back to the page allocator and may be used by another + * allocation immediately. Do not access the memory or the struct folio + * after calling folio_put_small_nopcp() unless you can be sure that it + * wasn't the last reference. + * + * Context: May be called in process or interrupt context, but not in NMI + * context. May be called while holding a spinlock. + */ +static inline void folio_put_small_nopcp(struct folio *folio) +{ + if (folio_put_testzero(folio)) + __folio_put_small_nopcp(folio); +} + /** * folio_put_refs - Reduce the reference count on a folio. * @folio: The folio. diff --git a/mm/internal.h b/mm/internal.h index b880f1e78700..3be8fd5604e8 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -451,6 +451,7 @@ extern int user_min_free_kbytes; extern void free_unref_page(struct page *page, unsigned int order); extern void free_unref_page_list(struct list_head *list); +extern void free_pages_nopcp(struct page *page, unsigned int order); extern void zone_pcp_reset(struct zone *zone); extern void zone_pcp_disable(struct zone *zone); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 733732e7e0ba..21b8c8cd1673 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -565,6 +565,16 @@ static inline void free_the_page(struct page *page, unsigned int order) __free_pages_ok(page, order, FPI_NONE); } +void free_pages_nopcp(struct page *page, unsigned int order) +{ + /* + * This function will be used in case that the pages are too + * cold to keep in pcp e.g. migrc mechanism. So it'd better + * release the pages to the tail. + */ + __free_pages_ok(page, order, FPI_TO_TAIL); +} + /* * Higher-order pages are called "compound pages". They are structured thusly: * diff --git a/mm/swap.c b/mm/swap.c index cd8f0150ba3a..3f37496a1184 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -106,6 +106,13 @@ static void __folio_put_small(struct folio *folio) free_unref_page(&folio->page, 0); } +void __folio_put_small_nopcp(struct folio *folio) +{ + __page_cache_release(folio); + mem_cgroup_uncharge(folio); + free_pages_nopcp(&folio->page, 0); +} + static void __folio_put_large(struct folio *folio) { /* From patchwork Mon Jan 15 08:19:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13519385 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0EB32C47258 for ; Mon, 15 Jan 2024 08:20:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 088036B009C; Mon, 15 Jan 2024 03:20:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F2CEF6B009D; Mon, 15 Jan 2024 03:20:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE0136B009E; Mon, 15 Jan 2024 03:20:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id B4F296B009C for ; Mon, 15 Jan 2024 03:20:15 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 8DE5AA04AC for ; Mon, 15 Jan 2024 08:20:15 +0000 (UTC) X-FDA: 81680847990.28.8BB4E6E Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf10.hostedemail.com (Postfix) with ESMTP id 6F55FC0018 for ; Mon, 15 Jan 2024 08:20:13 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf10.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705306814; a=rsa-sha256; cv=none; b=sWTFXPsidtGf0KuvlYO4llf1Lf2GPB9HdqE5XWE+wx5aeP/mg76I1Bq9HVFE+9+QH5X+VN Gc6eUsnv/0zYg9S4OlXS42BDr7yGyPtQy1P1nDgSkAnsY4ryK/8w4PovhPyCCoMewUqMV8 7fJja3b9a/p89O8v1dQH8kKqd/YfkT4= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf10.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705306814; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=hz9is1xKiIeQpY3xAWk3NcXXC/19leLnQzGKE93Yxnw=; b=0Umyczu0y/vvj4jiQRwKvgZQ9jS23FTw7s4U34Lz0eLvvfabZe+F1LqL439Hm7P6cfXbcT yWbEqdcdkK0SCKReKU040XdytaGeazaWnnSjBU2Bcu8N1tsCgijzgBC/Oa0AzOagReGIAq tRqc6sposMUjiBvqreK8NVbv3TimgSk= X-AuditID: a67dfc5b-d85ff70000001748-8f-65a4eab48500 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v6 6/7] mm: Defer TLB flush by keeping both src and dst folios at migration Date: Mon, 15 Jan 2024 17:19:52 +0900 Message-Id: <20240115081953.2521-7-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240115081953.2521-1-byungchul@sk.com> References: <20240115081953.2521-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrELMWRmVeSWpSXmKPExsXC9ZZnoe6WV0tSDdZ2qFnMWb+GzeLzhn9s Fi82tDNafF3/i9ni6ac+FovLu+awWdxb85/V4vyutawWO5buY7K4dGABk8X1XQ8ZLY73HmCy mH/vM5vF5k1TmS2OT5nKaPH7B1DHyVmTWRwEPb639rF47Jx1l91jwaZSj80rtDwW73nJ5LFp VSebx6ZPk9g93p07x+5xYsZvFo95JwM93u+7yuax9ZedR+PUa2wenzfJebyb/5YtgD+KyyYl NSezLLVI3y6BK2PV37qCpZsYK77MbWZrYNzay9jFyMkhIWAiceBqOyuM/WLub7A4m4C6xI0b P5lBbBEBM4mDrX/Yuxi5OJgFPjJJrP7ewQKSEBaIlnjV/YMNxGYRUJXYsPkAE4jNCzTo4KJb UAvkJVZvOAA2iFPAVOLdx+PsILYQUE3L3XWsIEMlBJrZJdZtaGWCaJCUOLjiBssERt4FjAyr GIUy88pyEzNzTPQyKvMyK/SS83M3MQJjYlntn+gdjJ8uBB9iFOBgVOLh/fF3caoQa2JZcWXu IUYJDmYlEd7qO0tShXhTEiurUovy44tKc1KLDzFKc7AoifMafStPERJITyxJzU5NLUgtgsky cXBKNTBq6NU4qgfIvb1SuWZyxPaTsVME9z5mUyheKdG5/tHOUwu22QszvrMRmbLaVnRr8bJ1 m7Z9j/XYG7m6Q0Vy3xx5u86716wye280ty6yj84tnJS9tK+ydPu7lW82b4jf7O0tkcjYsWzG +efSe+R3PNi39bPC78diK4+ePz17m6kis51C+xezEtEaJZbijERDLeai4kQA8qGgAoUCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrJLMWRmVeSWpSXmKPExsXC5WfdrLvl1ZJUg4654hZz1q9hs/i84R+b xYsN7YwWX9f/YrZ4+qmPxeLw3JOsFpd3zWGzuLfmP6vF+V1rWS12LN3HZHHpwAImi+u7HjJa HO89wGQx/95nNovNm6YyWxyfMpXR4vcPoI6TsyazOAh5fG/tY/HYOesuu8eCTaUem1doeSze 85LJY9OqTjaPTZ8msXu8O3eO3ePEjN8sHvNOBnq833eVzWPxiw9MHlt/2Xk0Tr3G5vF5k5zH u/lv2QIEorhsUlJzMstSi/TtErgyVv2tK1i6ibHiy9xmtgbGrb2MXYycHBICJhIv5v4Gs9kE 1CVu3PjJDGKLCJhJHGz9w97FyMXBLPCRSWL19w4WkISwQLTEq+4fbCA2i4CqxIbNB5hAbF6g QQcX3YIaKi+xesMBsEGcAqYS7z4eZwexhYBqWu6uY53AyLWAkWEVo0hmXlluYmaOqV5xdkZl XmaFXnJ+7iZGYIAvq/0zcQfjl8vuhxgFOBiVeHh//F2cKsSaWFZcmXuIUYKDWUmEt/rOklQh 3pTEyqrUovz4otKc1OJDjNIcLErivF7hqQlCAumJJanZqakFqUUwWSYOTqkGxjV3Q5K8XB4c Uaif3jGr+bF7bvqxurr7tnFyXn8nncycNvf9LoHGW8KXFp9NjFqfbrGMu2O7QdGLpL9dbv2/ eGIrnPaG7/wjPH19pf5Uq3NxPxv/vGNh+76UeRnzh3lBN6MmxLtk8y/JO+VknvG/RfZOx5Te qU46CbFCduf8FEo/FLgEBaZfVGIpzkg01GIuKk4EADoJu81sAgAA X-CFilter-Loop: Reflected X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 6F55FC0018 X-Stat-Signature: mb7j1cob1fnyuat9b1mygjoqymbp8qr1 X-Rspam-User: X-HE-Tag: 1705306813-947835 X-HE-Meta: U2FsdGVkX1+oioTsNJybbhjnkfH7sREOwOvKICxSEGQekiINmCKDKPRToK3bAF0IwF4QKLcZkaPgnaUxIsagtIvWv/jTC2EaX6i4zE1YTS0Cawed9OW0p5FtPQEBeJhj0SBJ4rVK88b6m0cNg2tI5YUNoUhKqROJcp/FDa62A1F01UFUM6cx3+zB+GZtSm/umiKOchFjJkqeZ2wYzJGX7P3XxRGNypjdbx5PVN7ybeHnHso+evmi6Sxkn0AaZcUSZozXLR/T4df7LCLDw3WBN33zbEf5623LVaaEvlGweXjmgRXRMMRQ6kiMWNrsug6SDJo2k2PEZiKX2+JBoGbO98q/nH3fDt8pmDIO2YR2/ZmHgNJg6QtRJh/66plZfEYtWRxj+7biAqdalJMpvkX4ifFER4csJXqYkC0LI1SXc6ixcwWb0UQPckwJeUhzq7b4UQHWeZESHdrJuy05noNTDNav8UEzHKgUjcMA9znPNycP0eIRAgrqPmGsFNRZiw/393nhLrO7ZZ9Ria4qo75Bd5soOOQr9seL+fJqqk61PT+v167fAJXnkL7Ri2KtrxJ4n1huN63QLU6Yluco79no5kvuiyUZf4eJSSSlTca29hPNseBc43KKo8id0q/jD+LyS5JaZXnSkdQxgqksdg3ufgGw7IggGLpnzLWVU994XE6gX0hGNUJy2RpNmYt9Z6URh52Q1hIKhw8+jvh9E81hg4pMRbwHb/As0w7QvMfu0pyflrVjiuxwRHaqJdSLSwBRIOoz2ndM/VZKhadZfR+x5MhnuOvEHoLW4AYATo3zLaW1+6iMfKUIBxKEjDs/9C/4PCMzwNJSoLM7zzdxXgOBRUZJTo6BpiZ36DzD9eRs+9MO5TUg3N0Re0uWAhorPd1N1oZti5yJW0yjA1DwDS9KG9Y2ZXE8SJj4K0K54/on6cTjq6zDH4FVj1xzQ9j3CFGbXlBypLhaQ6SPBy6aIry 0Qg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Implementation of MIGRC mechanism that stands for 'Migration Read Copy'. We always face the migration overhead at either promotion or demotion, while working with tiered memory e.g. CXL memory and found out TLB shootdown is a quite big one that is needed to get rid of if possible. Fortunately, TLB flush can be defered if both source and destination of folios during migration are kept until all TLB flushes required will have been done, of course, only if the target PTE entries have read-only permission, more precisely speaking, don't have write permission. Otherwise, no doubt the folio might get messed up. To achieve that: 1. For the folios that map only to non-writable TLB entries, prevent TLB flush at migration by keeping both source and destination folios, which will be handled later at a better time. 2. When any non-writable TLB entry changes to writable e.g. through fault handler, give up migrc mechanism so as to perform TLB flush required right away. The following evaluation using XSBench shows the improvement like: 1. itlb flush was reduced by 93.9%. 2. dtlb thread was reduced by 43.5%. 3. stlb flush was reduced by 24.9%. 4. dtlb store misses was reduced by 34.2%. 5. itlb load misses was reduced by 45.5%. 6. The runtime was reduced by 3.5%. --- The measurement result: Architecture - x86_64 QEMU - kvm enabled, host cpu Numa - 2 nodes (16 CPUs 1GB, no CPUs 8GB) Linux Kernel - v6.7, numa balancing tiering on, demotion enabled Benchmark - XSBench -p 100000000 (-p option makes the runtime longer) run 'perf stat' using events: 1) itlb.itlb_flush 2) tlb_flush.dtlb_thread 3) tlb_flush.stlb_any 4) dTLB-load-misses 5) dTLB-store-misses 6) iTLB-load-misses run 'cat /proc/vmstat' and pick: 1) numa_pages_migrated 2) pgmigrate_success 3) nr_tlb_remote_flush 4) nr_tlb_remote_flush_received 5) nr_tlb_local_flush_all 6) nr_tlb_local_flush_one BEFORE - mainline v6.7 ------------------------------------------ $ perf stat -a \ -e itlb.itlb_flush \ -e tlb_flush.dtlb_thread \ -e tlb_flush.stlb_any \ -e dTLB-load-misses \ -e dTLB-store-misses \ -e iTLB-load-misses \ ./XSBench -p 100000000 Performance counter stats for 'system wide': 85647229 itlb.itlb_flush 480981504 tlb_flush.dtlb_thread 323937200 tlb_flush.stlb_any 238381632579 dTLB-load-misses 601514255 dTLB-store-misses 2974157461 iTLB-load-misses 2252.883892112 seconds time elapsed $ cat /proc/vmstat ... numa_pages_migrated 12790664 pgmigrate_success 26835314 nr_tlb_remote_flush 3031412 nr_tlb_remote_flush_received 45234862 nr_tlb_local_flush_all 216584 nr_tlb_local_flush_one 740940 ... AFTER - mainline v6.7 + migrc ------------------------------------------ $ perf stat -a \ -e itlb.itlb_flush \ -e tlb_flush.dtlb_thread \ -e tlb_flush.stlb_any \ -e dTLB-load-misses \ -e dTLB-store-misses \ -e iTLB-load-misses \ ./XSBench -p 100000000 Performance counter stats for 'system wide': 5240261 itlb.itlb_flush 271581774 tlb_flush.dtlb_thread 243149389 tlb_flush.stlb_any 234502983364 dTLB-load-misses 395673680 dTLB-store-misses 1620215163 iTLB-load-misses 2172.283436287 seconds time elapsed $ cat /proc/vmstat ... numa_pages_migrated 14897064 pgmigrate_success 30825530 nr_tlb_remote_flush 198290 nr_tlb_remote_flush_received 2820156 nr_tlb_local_flush_all 92048 nr_tlb_local_flush_one 741401 ... Signed-off-by: Byungchul Park Signed-off-by: Byungchul Park --- arch/x86/mm/tlb.c | 4 + include/linux/mm.h | 8 ++ include/linux/mmzone.h | 7 ++ include/linux/sched.h | 8 ++ mm/internal.h | 49 ++++++++ mm/memory.c | 8 ++ mm/migrate.c | 271 +++++++++++++++++++++++++++++++++++++++-- mm/page_alloc.c | 11 +- mm/rmap.c | 10 +- 9 files changed, 364 insertions(+), 12 deletions(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 453ea95b667d..9164fd7d5973 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1254,6 +1254,9 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) info = get_flush_tlb_info(NULL, 0, TLB_FLUSH_ALL, 0, false, TLB_GENERATION_INVALID); + + migrc_flush_start(); + /* * flush_tlb_multi() is not optimized for the common case in which only * a local TLB flush is needed. Optimize this use-case by calling @@ -1268,6 +1271,7 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) local_irq_enable(); } + migrc_flush_end(batch); cpumask_clear(&batch->cpumask); put_flush_tlb_info(); diff --git a/include/linux/mm.h b/include/linux/mm.h index fc0581cce3a7..777b6904923c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2038,6 +2038,14 @@ static inline bool folio_is_longterm_pinnable(struct folio *folio) } #endif +#if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) +extern void migrc_flush_start(void); +extern void migrc_flush_end(struct arch_tlbflush_unmap_batch *arch); +#else +static inline void migrc_flush_start(void) {} +static inline void migrc_flush_end(struct arch_tlbflush_unmap_batch *arch) {} +#endif + static inline void set_page_zone(struct page *page, enum zone_type zone) { page->flags &= ~(ZONES_MASK << ZONES_PGSHIFT); diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 9db36e197712..492111cd1176 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1002,6 +1002,13 @@ struct zone { /* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS]; + +#if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) + /* + * the number of folios pending for TLB flush in the zone + */ + atomic_t migrc_pending_nr; +#endif } ____cacheline_internodealigned_in_smp; enum pgdat_flags { diff --git a/include/linux/sched.h b/include/linux/sched.h index 0317e7a65151..d8c285309a8f 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1330,6 +1330,14 @@ struct task_struct { struct tlbflush_unmap_batch tlb_ubc; struct tlbflush_unmap_batch tlb_ubc_ro; +#if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) + /* + * whether all the mappings of a folio during unmap are read-only + * so that migrc can work on the folio + */ + bool can_migrc; +#endif + /* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe; diff --git a/mm/internal.h b/mm/internal.h index 3be8fd5604e8..5b93e5abf953 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1284,4 +1284,53 @@ static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry, } #endif /* CONFIG_SHRINKER_DEBUG */ +#if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) +/* + * Reset the indicator indicating there are no writable mappings at the + * beginning of every rmap traverse for unmap. Migrc can work only when + * all the mappings are read-only. + */ +static inline void can_migrc_init(void) +{ + current->can_migrc = true; +} + +/* + * Mark the folio is not applicable to migrc, once it found a writble or + * dirty pte during rmap traverse for unmap. + */ +static inline void can_migrc_fail(void) +{ + current->can_migrc = false; +} + +/* + * Check if all the mappings are read-only and read-only mappings even + * exist. + */ +static inline bool can_migrc_test(void) +{ + return current->can_migrc && current->tlb_ubc_ro.flush_required; +} + +/* + * Return the number of folios pending TLB flush that have yet to get + * freed in the zone. + */ +static inline int migrc_pending_nr_in_zone(struct zone *z) +{ + return atomic_read(&z->migrc_pending_nr); +} + +/* + * Perform TLB flush needed and free the folios under migrc's control. + */ +bool migrc_flush_free_folios(void); +#else /* CONFIG_MIGRATION && CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ +static inline void can_migrc_init(void) {} +static inline void can_migrc_fail(void) {} +static inline bool can_migrc_test(void) { return false; } +static inline int migrc_pending_nr_in_zone(struct zone *z) { return 0; } +static inline bool migrc_flush_free_folios(void) { return false; } +#endif #endif /* __MM_INTERNAL_H */ diff --git a/mm/memory.c b/mm/memory.c index 6e0712d06cd4..e67de161da8b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3462,6 +3462,14 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) if (vmf->page) folio = page_folio(vmf->page); + /* + * The folio may or may not be one that is under migrc's control + * and about to change its permission from read-only to writable. + * Conservatively give up deferring TLB flush just in case. + */ + if (folio) + migrc_flush_free_folios(); + /* * Shared mapping: we are guaranteed to have VM_WRITE and * FAULT_FLAG_WRITE set at this point. diff --git a/mm/migrate.c b/mm/migrate.c index bbe1ecef4956..da4d43ac2622 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -57,6 +57,194 @@ #include "internal.h" +#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH +static struct tlbflush_unmap_batch migrc_ubc; +static LIST_HEAD(migrc_folios); +static DEFINE_SPINLOCK(migrc_lock); + +static void init_tlb_ubc(struct tlbflush_unmap_batch *ubc) +{ + arch_tlbbatch_clear(&ubc->arch); + ubc->flush_required = false; + ubc->writable = false; +} + +static void migrc_keep_folio(struct folio *f, struct list_head *h) +{ + list_move_tail(&f->lru, h); + folio_get(f); + atomic_inc(&folio_zone(f)->migrc_pending_nr); +} + +static void migrc_undo_folio(struct folio *f) +{ + list_del(&f->lru); + folio_put(f); + atomic_dec(&folio_zone(f)->migrc_pending_nr); +} + +static void migrc_release_folio(struct folio *f) +{ + folio_put_small_nopcp(f); + atomic_dec(&folio_zone(f)->migrc_pending_nr); +} + +/* + * Need to synchronize between TLB flush and managing pending CPUs in + * migrc_ubc. Take a look at the following scenario: + * + * CPU0 CPU1 + * ---- ---- + * TLB flush + * Unmap folios (needing TLB flush) + * Add pending CPUs to migrc_ubc + * Clear the CPUs from migrc_ubc + * + * The pending CPUs added in CPU1 should not be cleared from migrc_ubc + * in CPU0 because the TLB flush for migrc_ubc added in CPU1 has not + * been performed this turn. To avoid this, using 'migrc_flushing' + * variable, prevent adding pending CPUs to migrc_ubc and give up migrc + * mechanism if others are in the middle of TLB flush, like: + * + * CPU0 CPU1 + * ---- ---- + * migrc_flushing++ + * TLB flush + * Unmap folios (needing TLB flush) + * If migrc_flushing == 0: + * Add pending CPUs to migrc_ubc + * Else: <--- hit + * Give up migrc mechanism + * Clear the CPUs from migrc_ubc + * migrc_flush-- + * + * Only the following case would be allowed for migrc mechanism to work: + * + * CPU0 CPU1 + * ---- ---- + * Unmap folios (needing TLB flush) + * If migrc_flushing == 0: <--- hit + * Add pending CPUs to migrc_ubc + * Else: + * Give up migrc mechanism + * migrc_flushing++ + * TLB flush + * Clear the CPUs from migrc_ubc + * migrc_flush-- + */ +static int migrc_flushing; + +static bool migrc_add_pending_ubc(struct tlbflush_unmap_batch *ubc) +{ + struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + if (migrc_flushing) { + spin_unlock_irqrestore(&migrc_lock, flags); + + /* + * Give up migrc mechanism. Just let TLB flush needed + * handled by try_to_unmap_flush() at the caller side. + */ + fold_ubc(tlb_ubc, ubc); + return false; + } + fold_ubc(&migrc_ubc, ubc); + spin_unlock_irqrestore(&migrc_lock, flags); + return true; +} + +static bool migrc_add_pending_folios(struct list_head *folios) +{ + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + if (migrc_flushing) { + spin_unlock_irqrestore(&migrc_lock, flags); + + /* + * Give up migrc mechanism. The caller should perform + * TLB flush needed using migrc_flush_free_folios() and + * undo some on the folios e.g. restore folios' + * reference count increased by migrc and more. + */ + return false; + } + list_splice(folios, &migrc_folios); + spin_unlock_irqrestore(&migrc_lock, flags); + return true; +} + +void migrc_flush_start(void) +{ + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + migrc_flushing++; + spin_unlock_irqrestore(&migrc_lock, flags); +} + +void migrc_flush_end(struct arch_tlbflush_unmap_batch *arch) +{ + LIST_HEAD(folios); + struct folio *f, *f2; + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + if (!arch_tlbbatch_done(&migrc_ubc.arch, arch)) { + list_splice_init(&migrc_folios, &folios); + migrc_ubc.flush_required = false; + migrc_ubc.writable = false; + } + migrc_flushing--; + spin_unlock_irqrestore(&migrc_lock, flags); + + list_for_each_entry_safe(f, f2, &folios, lru) + migrc_release_folio(f); +} + +bool migrc_flush_free_folios(void) +{ + struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + LIST_HEAD(folios); + struct folio *f, *f2; + unsigned long flags; + bool ret = true; + + spin_lock_irqsave(&migrc_lock, flags); + list_splice_init(&migrc_folios, &folios); + fold_ubc(tlb_ubc, &migrc_ubc); + spin_unlock_irqrestore(&migrc_lock, flags); + + if (list_empty(&folios)) + ret = false; + + try_to_unmap_flush(); + list_for_each_entry_safe(f, f2, &folios, lru) + migrc_release_folio(f); + return ret; +} +#else /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ +static void init_tlb_ubc(struct tlbflush_unmap_batch *ubc) +{ +} +static void migrc_keep_folio(struct folio *f, struct list_head *h) +{ +} +static void migrc_undo_folio(struct folio *f) +{ +} +static bool migrc_add_pending_ubc(struct tlbflush_unmap_batch *ubc) +{ + return false; +} +static bool migrc_add_pending_folios(struct list_head *folios) +{ + return false; +} +#endif + bool isolate_movable_page(struct page *page, isolate_mode_t mode) { struct folio *folio = folio_get_nontail_page(page); @@ -1274,7 +1462,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, struct folio *src, struct folio *dst, enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) + struct list_head *ret, struct list_head *move_succ) { int rc; int old_page_state = 0; @@ -1321,9 +1509,13 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, /* * A folio that has been migrated has all references removed - * and will be freed. + * and will be freed, unless it's under migrc's control. */ - list_del(&src->lru); + if (move_succ) + migrc_keep_folio(src, move_succ); + else + list_del(&src->lru); + /* Drop an anon_vma reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); @@ -1618,7 +1810,7 @@ static void migrate_folios_move(struct list_head *src_folios, struct list_head *ret_folios, struct migrate_pages_stats *stats, int *retry, int *thp_retry, int *nr_failed, - int *nr_retry_pages) + int *nr_retry_pages, struct list_head *move_succ) { struct folio *folio, *folio2, *dst, *dst2; bool is_thp; @@ -1635,7 +1827,7 @@ static void migrate_folios_move(struct list_head *src_folios, rc = migrate_folio_move(put_new_folio, private, folio, dst, mode, - reason, ret_folios); + reason, ret_folios, move_succ); /* * The rules are: * Success: folio will be freed @@ -1712,17 +1904,34 @@ static int migrate_pages_batch(struct list_head *from, int rc, rc_saved = 0, nr_pages; LIST_HEAD(unmap_folios); LIST_HEAD(dst_folios); + LIST_HEAD(unmap_folios_migrc); + LIST_HEAD(dst_folios_migrc); + LIST_HEAD(move_succ); bool nosplit = (reason == MR_NUMA_MISPLACED); + struct tlbflush_unmap_batch pending_ubc; + struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc_ro = ¤t->tlb_ubc_ro; + bool do_migrc; + bool migrc_ubc_succ; VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC && !list_empty(from) && !list_is_singular(from)); + /* + * Apply migrc only to numa migration for now. + */ + init_tlb_ubc(&pending_ubc); + do_migrc = IS_ENABLED(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) && + (reason == MR_DEMOTION || reason == MR_NUMA_MISPLACED); + for (pass = 0; pass < nr_pass && retry; pass++) { retry = 0; thp_retry = 0; nr_retry_pages = 0; list_for_each_entry_safe(folio, folio2, from, lru) { + bool can_migrc; + is_large = folio_test_large(folio); is_thp = is_large && folio_test_pmd_mappable(folio); nr_pages = folio_nr_pages(folio); @@ -1752,9 +1961,12 @@ static int migrate_pages_batch(struct list_head *from, continue; } + can_migrc_init(); rc = migrate_folio_unmap(get_new_folio, put_new_folio, private, folio, &dst, mode, reason, ret_folios); + can_migrc = do_migrc && can_migrc_test() && !is_large; + /* * The rules are: * Success: folio will be freed @@ -1800,7 +2012,8 @@ static int migrate_pages_batch(struct list_head *from, /* nr_failed isn't updated for not used */ stats->nr_thp_failed += thp_retry; rc_saved = rc; - if (list_empty(&unmap_folios)) + if (list_empty(&unmap_folios) && + list_empty(&unmap_folios_migrc)) goto out; else goto move; @@ -1814,8 +2027,19 @@ static int migrate_pages_batch(struct list_head *from, stats->nr_thp_succeeded += is_thp; break; case MIGRATEPAGE_UNMAP: - list_move_tail(&folio->lru, &unmap_folios); - list_add_tail(&dst->lru, &dst_folios); + if (can_migrc) { + list_move_tail(&folio->lru, &unmap_folios_migrc); + list_add_tail(&dst->lru, &dst_folios_migrc); + + /* + * Gather ro batch data to add + * to migrc_ubc after unmap. + */ + fold_ubc(&pending_ubc, tlb_ubc_ro); + } else { + list_move_tail(&folio->lru, &unmap_folios); + list_add_tail(&dst->lru, &dst_folios); + } break; default: /* @@ -1829,12 +2053,19 @@ static int migrate_pages_batch(struct list_head *from, stats->nr_failed_pages += nr_pages; break; } + /* + * Done with the current folio. Fold the ro + * batch data gathered, to the normal batch. + */ + fold_ubc(tlb_ubc, tlb_ubc_ro); } } nr_failed += retry; stats->nr_thp_failed += thp_retry; stats->nr_failed_pages += nr_retry_pages; move: + /* Should be before try_to_unmap_flush() */ + migrc_ubc_succ = do_migrc && migrc_add_pending_ubc(&pending_ubc); /* Flush TLBs for all unmapped folios */ try_to_unmap_flush(); @@ -1848,7 +2079,27 @@ static int migrate_pages_batch(struct list_head *from, migrate_folios_move(&unmap_folios, &dst_folios, put_new_folio, private, mode, reason, ret_folios, stats, &retry, &thp_retry, - &nr_failed, &nr_retry_pages); + &nr_failed, &nr_retry_pages, NULL); + migrate_folios_move(&unmap_folios_migrc, &dst_folios_migrc, + put_new_folio, private, mode, reason, + ret_folios, stats, &retry, &thp_retry, + &nr_failed, &nr_retry_pages, migrc_ubc_succ ? + &move_succ : NULL); + } + + /* + * In case that migrc_add_pending_ubc() has been added + * successfully but migrc_add_pending_folios() does not. + */ + if (migrc_ubc_succ && !migrc_add_pending_folios(&move_succ)) { + migrc_flush_free_folios(); + + /* + * Undo src folios that have been successfully added to + * move_succ. + */ + list_for_each_entry_safe(folio, folio2, &move_succ, lru) + migrc_undo_folio(folio); } nr_failed += retry; stats->nr_thp_failed += thp_retry; @@ -1859,6 +2110,8 @@ static int migrate_pages_batch(struct list_head *from, /* Cleanup remaining folios */ migrate_folios_undo(&unmap_folios, &dst_folios, put_new_folio, private, ret_folios); + migrate_folios_undo(&unmap_folios_migrc, &dst_folios_migrc, + put_new_folio, private, ret_folios); return rc; } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 21b8c8cd1673..6ef0c22b1109 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2972,6 +2972,8 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark, long min = mark; int o; + free_pages += migrc_pending_nr_in_zone(z); + /* free_pages may go negative - that's OK */ free_pages -= __zone_watermark_unusable_free(z, order, alloc_flags); @@ -3066,7 +3068,7 @@ static inline bool zone_watermark_fast(struct zone *z, unsigned int order, long usable_free; long reserved; - usable_free = free_pages; + usable_free = free_pages + migrc_pending_nr_in_zone(z); reserved = __zone_watermark_unusable_free(z, 0, alloc_flags); /* reserved may over estimate high-atomic reserves. */ @@ -3273,6 +3275,13 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, gfp_mask)) { int ret; + if (migrc_pending_nr_in_zone(zone) && + migrc_flush_free_folios() && + zone_watermark_fast(zone, order, mark, + ac->highest_zoneidx, + alloc_flags, gfp_mask)) + goto try_this_zone; + if (has_unaccepted_memory()) { if (try_to_accept_memory(zone, order)) goto try_this_zone; diff --git a/mm/rmap.c b/mm/rmap.c index da36f23ff7b0..79e1827dec89 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -676,9 +676,15 @@ static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, if (!pte_accessible(mm, pteval)) return; - if (pte_write(pteval) || writable) + if (pte_write(pteval) || writable) { tlb_ubc = ¤t->tlb_ubc; - else + + /* + * Migrc cannot work with the folio, once it found a + * writable or dirty mapping on it. + */ + can_migrc_fail(); + } else tlb_ubc = ¤t->tlb_ubc_ro; arch_tlbbatch_add_pending(&tlb_ubc->arch, mm, uaddr); From patchwork Mon Jan 15 08:19:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13519386 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3268C47422 for ; Mon, 15 Jan 2024 08:20:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C1F2C6B009D; Mon, 15 Jan 2024 03:20:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BA7A56B009E; Mon, 15 Jan 2024 03:20:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9FA0E6B009F; Mon, 15 Jan 2024 03:20:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8B5106B009D for ; Mon, 15 Jan 2024 03:20:16 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 4AF501C066B for ; Mon, 15 Jan 2024 08:20:16 +0000 (UTC) X-FDA: 81680848032.23.67B90F6 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf02.hostedemail.com (Postfix) with ESMTP id 6F0C380011 for ; Mon, 15 Jan 2024 08:20:14 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf02.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705306814; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=OT2fzduGNK1MPqy8u5PjRuQqdMY+Q2D+YpYwYXpRlo0=; b=gA16qFlyeiIjUMGxlHBuuLSdReGSqmcM0UfBKgmFduoOnxrf7izQeZCI5p+yKz2lYEUWfZ 3KYSRnjfLgwo3Yt71VwOLJq0fWsyHsBTKH7ccXHrDCwQV0C49sUXuOuP3nCiLVNi6rtBeh pZC1qx17AockHxFhVXCjkPpJFP7mTr8= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf02.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705306814; a=rsa-sha256; cv=none; b=sf+Fk9Sdot+iFTvg6i6oNnA5/jTpYnNHRdwPnlpZg4uakxUKKzDdKhMCOwXO9jObVBsZSp O9GI9/vrKimcpfekzEh7AZGTw0x2TBdaQ1GsY4kra98ATbi4PqH+g7hHothMKS5vxeXFqY AF6A8ohogrIyezDz1fESk4728SMN3Ds= X-AuditID: a67dfc5b-d85ff70000001748-95-65a4eab45f59 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v6 7/7] mm: Pause migrc mechanism at high memory pressure Date: Mon, 15 Jan 2024 17:19:53 +0900 Message-Id: <20240115081953.2521-8-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240115081953.2521-1-byungchul@sk.com> References: <20240115081953.2521-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrILMWRmVeSWpSXmKPExsXC9ZZnoe6WV0tSDV5cVbeYs34Nm8XnDf/Y LF5saGe0+Lr+F7PF0099LBaXd81hs7i35j+rxflda1ktdizdx2Rx6cACJovrux4yWhzvPcBk Mf/eZzaLzZumMlscnzKV0eL3D6COk7MmszgIenxv7WPx2DnrLrvHgk2lHptXaHks3vOSyWPT qk42j02fJrF7vDt3jt3jxIzfLB7zTgZ6vN93lc1j6y87j8ap19g8Pm+S83g3/y1bAH8Ul01K ak5mWWqRvl0CV8b5NZOZC24oVzxu3cjawNgh28XIySEhYCIxrWkJK4zd0HKLGcRmE1CXuHHj J5gtImAmcbD1D3sXIxcHs8BHJonV3ztYQBLCAp4SPQu3MoLYLAKqEhPetIHFeYEGzfzyngli qLzE6g0HwAZxCphKvPt4nB3EFgKqabm7jhVkqIRAM7vE0vP9bBANkhIHV9xgmcDIu4CRYRWj UGZeWW5iZo6JXkZlXmaFXnJ+7iZGYEQsq/0TvYPx04XgQ4wCHIxKPLw//i5OFWJNLCuuzD3E KMHBrCTCW31nSaoQb0piZVVqUX58UWlOavEhRmkOFiVxXqNv5SlCAumJJanZqakFqUUwWSYO TqkGRs+pWc4KGU5aTx5lljxscJr4MEDDk9V/VXL8lfwTG5doHPjBdmv3Ydu5DFc5mlWORsfz PZv1OpO1QnHh19VXXTtm+6RH3N7rcOKCU5qP3mu30y9XWW47fKjctHNdquGKbeLdWy6+bd/Q 9dlnxtW7qx7q9ezsnVYwI7LA3PCJltDDrlWJD1yUHiuxFGckGmoxFxUnAgAEz+8chAIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrBLMWRmVeSWpSXmKPExsXC5WfdrLvl1ZJUg+5NUhZz1q9hs/i84R+b xYsN7YwWX9f/YrZ4+qmPxeLw3JOsFpd3zWGzuLfmP6vF+V1rWS12LN3HZHHpwAImi+u7HjJa HO89wGQx/95nNovNm6YyWxyfMpXR4vcPoI6TsyazOAh5fG/tY/HYOesuu8eCTaUem1doeSze 85LJY9OqTjaPTZ8msXu8O3eO3ePEjN8sHvNOBnq833eVzWPxiw9MHlt/2Xk0Tr3G5vF5k5zH u/lv2QIEorhsUlJzMstSi/TtErgyzq+ZzFxwQ7nicetG1gbGDtkuRk4OCQETiYaWW8wgNpuA usSNGz/BbBEBM4mDrX/Yuxi5OJgFPjJJrP7ewQKSEBbwlOhZuJURxGYRUJWY8KYNLM4LNGjm l/dMEEPlJVZvOAA2iFPAVOLdx+PsILYQUE3L3XWsExi5FjAyrGIUycwry03MzDHVK87OqMzL rNBLzs/dxAgM72W1fybuYPxy2f0QowAHoxIP74+/i1OFWBPLiitzDzFKcDArifBW31mSKsSb klhZlVqUH19UmpNafIhRmoNFSZzXKzw1QUggPbEkNTs1tSC1CCbLxMEp1cC47ufENp675cE3 ReXfOTiEHTobu9Z2q05YxfvIsGfMtadztlRtuL9Kyv3IjYRiec5fnpJ/1tee75gS/efcq0lB cWomzS6HnwfwVwn7NqT4Tg5ZVfhycap6vpr7DNWtnheLf4Ztf768Yu/DgMVvBCZN5X2ozy5S +Cpe2+V2a9El873Sl/2PTexUYinOSDTUYi4qTgQA60wLeWsCAAA= X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 6F0C380011 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: r3i5fahazegnpdk8wkfxmwwgj7g45tuc X-HE-Tag: 1705306814-378066 X-HE-Meta: U2FsdGVkX189R4iBL3LyEP7TwAXKMWZ6jhxGOLWeFSfPTD67Um94Bz2PUbxjj5BQmKadNeoXPY8dbhioYiydgfQKpPklKn357bba391bwfIGtYYaksAXdmNtgM28n0CyymQuQ6isWOksnThY0albHRv1Pj6OEIW++gLOsnLXwZiWW0DtO1L9CSCcM6AANv6uc2Yj9q/jiJ5dZFNRGJZJe4EYxzJGTJlgPsYB9Fg19p8yKBDDphlSOex4nrEIVHagHPMZUNCWxrgFPul0/1z7IRRoW8pINvodGTH1tIMPtlZ+//RU4Y5zRVjROXqvOpiwfGi3DMIbj2skgAsw5MMI1Nhdi0Yt/6er4AvcpexyVm+tWPvQmRtd3el0+H6JReW3cBMJWYioIpL0h6CtUymevAxfJ/3uapttx7ouk93fJ1j61iWRBbhX5XD6mcM7mEC2VjAmcMR/p0INMhugLCtHYuS+4eEGu/lrsDGQm0EvjWCO+PEmOhUJz4tikjxeZLutHDcmDNJ/vbumF6vZ4md8/C9QnrZzX3Sno9Dk5YzrISygMh34zPehFtv7R3m/liJLd9d4oXjC0miEnKk+d9TnwgORF1A9q3xxO+WnHlNP0vCkDuvrJTof63GlqeNlKFtHl89duFyY5uV47iLVgUdVmm4KNAgBKXBp3M47Er8shPwVqaouHriK+rLaUbieYOI0tfVY87TkUFaUm2VaRMWMlTyJYGDgEdTCHsvF/YZu536fevQamF5e3p8/vei1s3Eru7OhaGhrTC2VWwRXzPWGUhI74JAAWSqRWYBbDJtHaNUDRPlxApJ157R/Zuf9e5h6OnpeZUdgtGzNnOn+JMlB44YtAJU4IrNlpgkNzJs64foeAtYFiO7ntlqnrRSXybFqgb+zEYJiFiWyFqvBCohrHJ/69fAQaiJkSDfo2tSBidxRPpiwYGRyOGB7HMN//mzRzP1CCxWxG4nGw78ukkP 2ww== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Regression was observed when the system is in high memory pressure with swap on, where migrc might keep a number of folios in its pending queue, which possibly makes it worse. So temporarily prevented migrc from working on that condition. Signed-off-by: Byungchul Park --- mm/internal.h | 20 ++++++++++++++++++++ mm/migrate.c | 18 +++++++++++++++++- mm/page_alloc.c | 13 +++++++++++++ 3 files changed, 50 insertions(+), 1 deletion(-) diff --git a/mm/internal.h b/mm/internal.h index 5b93e5abf953..7643ccb7c156 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1285,6 +1285,8 @@ static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry, #endif /* CONFIG_SHRINKER_DEBUG */ #if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) +extern atomic_t migrc_pause_cnt; + /* * Reset the indicator indicating there are no writable mappings at the * beginning of every rmap traverse for unmap. Migrc can work only when @@ -1313,6 +1315,21 @@ static inline bool can_migrc_test(void) return current->can_migrc && current->tlb_ubc_ro.flush_required; } +static inline void migrc_pause(void) +{ + atomic_inc(&migrc_pause_cnt); +} + +static inline void migrc_resume(void) +{ + atomic_dec(&migrc_pause_cnt); +} + +static inline bool migrc_paused(void) +{ + return !!atomic_read(&migrc_pause_cnt); +} + /* * Return the number of folios pending TLB flush that have yet to get * freed in the zone. @@ -1330,6 +1347,9 @@ bool migrc_flush_free_folios(void); static inline void can_migrc_init(void) {} static inline void can_migrc_fail(void) {} static inline bool can_migrc_test(void) { return false; } +static inline void migrc_pause(void) {} +static inline void migrc_resume(void) {} +static inline bool migrc_paused(void) { return false; } static inline int migrc_pending_nr_in_zone(struct zone *z) { return 0; } static inline bool migrc_flush_free_folios(void) { return false; } #endif diff --git a/mm/migrate.c b/mm/migrate.c index da4d43ac2622..b838f279f050 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -62,6 +62,12 @@ static struct tlbflush_unmap_batch migrc_ubc; static LIST_HEAD(migrc_folios); static DEFINE_SPINLOCK(migrc_lock); +/* + * Increase on entry of handling high memory pressure e.g. direct + * reclaim, decrease on the exit. See __alloc_pages_slowpath(). + */ +atomic_t migrc_pause_cnt = ATOMIC_INIT(0); + static void init_tlb_ubc(struct tlbflush_unmap_batch *ubc) { arch_tlbbatch_clear(&ubc->arch); @@ -1922,7 +1928,8 @@ static int migrate_pages_batch(struct list_head *from, */ init_tlb_ubc(&pending_ubc); do_migrc = IS_ENABLED(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) && - (reason == MR_DEMOTION || reason == MR_NUMA_MISPLACED); + (reason == MR_DEMOTION || reason == MR_NUMA_MISPLACED) && + !migrc_paused(); for (pass = 0; pass < nr_pass && retry; pass++) { retry = 0; @@ -1961,6 +1968,15 @@ static int migrate_pages_batch(struct list_head *from, continue; } + /* + * In case that the system is in high memory + * pressure, give up migrc mechanism this turn. + */ + if (unlikely(do_migrc && migrc_paused())) { + fold_ubc(tlb_ubc, &pending_ubc); + do_migrc = false; + } + can_migrc_init(); rc = migrate_folio_unmap(get_new_folio, put_new_folio, private, folio, &dst, mode, reason, diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 6ef0c22b1109..366777afce7f 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4072,6 +4072,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, unsigned int cpuset_mems_cookie; unsigned int zonelist_iter_cookie; int reserve_flags; + bool migrc_paused = false; restart: compaction_retries = 0; @@ -4203,6 +4204,16 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, if (page) goto got_pg; + /* + * The system is in very high memory pressure. Pause migrc from + * expanding its pending queue temporarily. + */ + if (!migrc_paused) { + migrc_pause(); + migrc_paused = true; + migrc_flush_free_folios(); + } + /* Caller is not willing to reclaim, we can't balance anything */ if (!can_direct_reclaim) goto nopage; @@ -4330,6 +4341,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, warn_alloc(gfp_mask, ac->nodemask, "page allocation failure: order:%u", order); got_pg: + if (migrc_paused) + migrc_resume(); return page; }