From patchwork Thu Feb 8 06:26:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13549355 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C73EC4829C for ; Thu, 8 Feb 2024 06:26:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E54CD6B009A; Thu, 8 Feb 2024 01:26:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DDCA86B009B; Thu, 8 Feb 2024 01:26:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C33016B009F; Thu, 8 Feb 2024 01:26:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B036B6B009A for ; Thu, 8 Feb 2024 01:26:26 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 924571C1212 for ; Thu, 8 Feb 2024 06:26:26 +0000 (UTC) X-FDA: 81767652372.27.6539787 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf02.hostedemail.com (Postfix) with ESMTP id A08838000B for ; Thu, 8 Feb 2024 06:26:24 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf02.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707373585; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=eRD1/8o+qtEOTAKMY7ucioDJJtMI3+WpTL+ASvHX4To=; b=xDSzn/yWmDZQlmV11FTR72zRs0SUZgwjbt09KEr6RCGJIWZofNU3OrdG1SVM369AldxpQM mWShdgRx/zZ935/Yhek0ODnq8ia6zb7PxojO0AMTjwRYK7+ZACGWLTuypKiAG327gIG52d Ge+eLTs+Gsh3w+iAFcA14YS8EPwFUTs= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf02.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707373585; a=rsa-sha256; cv=none; b=2gIVKiQAhIwsOuLuh8SA4qMjQLl/JtM5nONOAdDkfzAaJu+tmt4/XRhoHwjJ1czq0pdjNl DUfTx6C2AU+2o2iZRJR1yeOW5myKFO4EWvkEfLdtwZHr1VFx4IGGspl5JHehY1owSzNd3+ a3R4CEJCMF5Qj3eECv3L0lsz/jgwm4g= X-AuditID: a67dfc5b-d85ff70000001748-74-65c4740a3ee5 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v7 1/8] x86/tlb: Add APIs manipulating tlb batch's arch data Date: Thu, 8 Feb 2024 15:26:01 +0900 Message-Id: <20240208062608.44351-2-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240208062608.44351-1-byungchul@sk.com> References: <20240208062608.44351-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrILMWRmVeSWpSXmKPExsXC9ZZnoS53yZFUg23r2CzmrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeL6roeMFsd7DzBZ zL/3mc1i86apzBbHp0xltPj9A6jj5KzJLA6CHt9b+1g8ds66y+6xYFOpx+YVWh6L97xk8ti0 qpPNY9OnSewe786dY/c4MeM3i8e8k4Ee7/ddZfPY+svOo3HqNTaPz5vkPN7Nf8sWwB/FZZOS mpNZllqkb5fAlXH79kumglV8FUt+tLM1ME7j6WLk5JAQMJGYuHUdM4y97csFRhCbTUBd4saN n2BxEQEziYOtf9i7GLk4mAU+Mkms/t7BApIQFgiWuL7qExuIzSKgKrF5zx8mEJtXwFRi55T9 LBBD5SVWbzgANogTaNDObadYQWwhoJoXsyezgQyVEGhnl1g57wYjRIOkxMEVN1gmMPIuYGRY xSiUmVeWm5iZY6KXUZmXWaGXnJ+7iREYEctq/0TvYPx0IfgQowAHoxIP74nyw6lCrIllxZW5 hxglOJiVRHjNdhxIFeJNSaysSi3Kjy8qzUktPsQozcGiJM5r9K08RUggPbEkNTs1tSC1CCbL xMEp1cAYub/cWXVhd03Z9S05PF7Xdm3dYl1vu2DF2RQjw4scU14tneW8if863wu9GczeKjN+ +aVN4VOseqx8V0WQ31JV0Puyxo2LHOy9T+6/e3dQfVV95bJ2+wdJH07Ybn3pwKLpVvFeuvNM dujHoEmHD6r9Dcq4J/CA6VdausbTXccb+fSjWQoW6lxXYinOSDTUYi4qTgQAo4YPLYQCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrBLMWRmVeSWpSXmKPExsXC5WfdrMtVciTVoPW6qcWc9WvYLD5v+Mdm 8WJDO6PF1/W/mC2efupjsTg89ySrxeVdc9gs7q35z2pxftdaVosdS/cxWVw6sIDJ4vquh4wW x3sPMFnMv/eZzWLzpqnMFsenTGW0+P0DqOPkrMksDkIe31v7WDx2zrrL7rFgU6nH5hVaHov3 vGTy2LSqk81j06dJ7B7vzp1j9zgx4zeLx7yTgR7v911l81j84gOTx9Zfdh6NU6+xeXzeJOfx bv5btgCBKC6blNSczLLUIn27BK6M27dfMhWs4qtY8qOdrYFxGk8XIyeHhICJxLYvFxhBbDYB dYkbN34yg9giAmYSB1v/sHcxcnEwC3xkklj9vYMFJCEsECxxfdUnNhCbRUBVYvOeP0wgNq+A qcTOKftZIIbKS6zecABsECfQoJ3bTrGC2EJANS9mT2abwMi1gJFhFaNIZl5ZbmJmjqlecXZG ZV5mhV5yfu4mRmB4L6v9M3EH45fL7ocYBTgYlXh4T5QfThViTSwrrsw9xCjBwawkwmu240Cq EG9KYmVValF+fFFpTmrxIUZpDhYlcV6v8NQEIYH0xJLU7NTUgtQimCwTB6dUAyN76BPlyY/y tq+8Iv+2ci1/eYl90+u8zrdxLNd364df1HT9bzPRaeGy75+m/nJ49pt3JXe9XwK/8srQ1TNW uZ0/YX70wHrVR2o93nc25UhOXFWtsKj6z8FJEcGTbu70XDtJwMpI6+aOxQ2/31o+f/LgxWvW h1u+bn6iONPhb9/6rc92GdWqnJifrsRSnJFoqMVcVJwIAFYAZeNrAgAA X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: A08838000B X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 4zx5egr7eo6ik5pi6tq74cqfhguyqt4b X-HE-Tag: 1707373584-815228 X-HE-Meta: U2FsdGVkX1/rb5kod265hQb/jytkZeZAkIHqFY9qUWc+5BXsRX7JsVlkEqyZtGp/X/nehKhsINbB1/uko32+YUw2JOQlhpPUNY3FOOaKinjjKUJl62N/KzepbH9KSXZbVrMfX8j2xsDpfcE2aAWgsyswhOcLedRqN7N0lNtxjAkIVhBsq57yy24RKSVSsrXb3AeUnBxf7dUH0tc++v66i5pkEKfKct/WXPOcgcTXF6SgFZpjI7pINBk/uj+6AcfO+1C36G3RdzcWkpaMFqjvtkiWzepq3q3Io6ZiHooNA20AtGA6GxmTSF1q4o9kbKo3MzPRdXVdcAxv45VY39BHm0Vu6jHUDGrq1gqOoI6w8RNW/txRWlXReeJyj10c0U8Ggn0KiDqLjnJ/QoUQRWF7pOthg2fLa9ENBUF6chtg07/ZzFewu37o8xVaPVFtqq7pUK0JAOLxjbpQeYwoUoJhK2hAqfKB+UUcbJbj7wIv3iq48p4xMu94HDZzzBWrkC5YxN+OfwfeAnoSsIRZLbDq9OrkhQ26znGtmzwGhdvfBABYDbQDho2slPCkRriSgosR4zUunxugBV0rJoaU5/KWvNBvoj5vcdy8qrfUwLIM37ZNb3htIMQGUNuER497M+337Ol5TbAGkDMx6/qT0RRd6SleJFlhZpo68/jwOeYfhJ75akVM0yHkdquWcwsGqmfCRBwkfkpBPJQUKkjT5M7fqLddreBZwdp1uHoo8wyOTKPkS2zutsommjNTBy+hjkcohAh0sXS15NzL+ZLyCtWaYVgfoXlHf2eWw57p3XVPdOqycgzMWQRIoeqUaJqdJQhxYQu+WnQujiEnZuYmJdj9nwAFx+LucUBXjdlBZ3sMN/CNGesALaVk5aBIhezzojAXlUtjBE9az3ufnBXVJbKgdmVTO+fnRS2jgJdL1LaIcjqyyDHgqB6tdPdGvuum6ax9FcTuXLa4mtnEgT8e5m8 9UWgFvgw RqrjVw533bSga/UBfu5/xqYpsIw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparation for migrc mechanism that needs to recognize read-only TLB entries during batched TLB flush by separating tlb batch's arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed. Migrc also needs to optimize CPUs to flush by clearing ones that have already performed TLB flush needed. To support it, added APIs manipulating arch data for x86. Signed-off-by: Byungchul Park --- arch/x86/include/asm/tlbflush.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 25726893c6f4..fa7e16dbeb44 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -293,6 +294,23 @@ static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm) extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); +static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch) +{ + cpumask_clear(&batch->cpumask); +} + +static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + cpumask_or(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + +static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + return cpumask_andnot(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + static inline bool pte_flags_need_flush(unsigned long oldflags, unsigned long newflags, bool ignore_access) From patchwork Thu Feb 8 06:26:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13549354 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 001B8C4828F for ; Thu, 8 Feb 2024 06:26:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9E4D76B0098; Thu, 8 Feb 2024 01:26:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 96E306B009B; Thu, 8 Feb 2024 01:26:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 612FA6B009A; Thu, 8 Feb 2024 01:26:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4B6CD6B0098 for ; Thu, 8 Feb 2024 01:26:26 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 250651209A9 for ; Thu, 8 Feb 2024 06:26:26 +0000 (UTC) X-FDA: 81767652372.28.53CE5DE Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf08.hostedemail.com (Postfix) with ESMTP id C6536160002 for ; Thu, 8 Feb 2024 06:26:20 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf08.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707373584; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=23+JjFD9uRPbyMWGqJTLiQYjg6Gkh8EQ/TB1jpahIpQ=; b=L/A6GPx/xJNa3wpfJN68tS+JkPNYMQEhJych8fWCuMRH4g6/ZzkEFF2V94QQlBdLO4/utG ltuZE3GVNpN2jZ8HUP//BGF7RxlfUXSjZkJ7Z7Wpk3UR1yYtu3x0/HUtQSsyHnso9hN1Aw WJ+pD+soldC+LFOlWhQVRfAQsaGkPrI= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf08.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707373584; a=rsa-sha256; cv=none; b=Rs9q1ZjZ/iPx+QS2xWR7sjrM7t3RtKEv0mEl/FVxxIBnXCt2922J4QNbOOtc4Hy5wH2iHA B1wsjoWGP2gtflGc81tijWDOziwn31CV+YFRc5V3ESp++vIfP0+fGgrGcmdnv7U2UXsKAs WQIiDVb55OmHsOHuDp/A3gD3Jm46h2U= X-AuditID: a67dfc5b-d85ff70000001748-78-65c4740bfdae From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v7 2/8] arm64: tlbflush: Add APIs manipulating tlb batch's arch data Date: Thu, 8 Feb 2024 15:26:02 +0900 Message-Id: <20240208062608.44351-3-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240208062608.44351-1-byungchul@sk.com> References: <20240208062608.44351-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrILMWRmVeSWpSXmKPExsXC9ZZnkS53yZFUgyktnBZz1q9hs/i84R+b xYsN7YwWX9f/YrZ4+qmPxeLyrjlsFvfW/Ge1OL9rLavFjqX7mCwuHVjAZHF910NGi+O9B5gs 5t/7zGaxedNUZovjU6YyWvz+AdRxctZkFgdBj++tfSweO2fdZfdYsKnUY/MKLY/Fe14yeWxa 1cnmsenTJHaPd+fOsXucmPGbxWPeyUCP9/uusnls/WXn0Tj1GpvH501yHu/mv2UL4I/isklJ zcksSy3St0vgytjT95C9YDZXxab37WwNjHs5uhg5OSQETCTOTJ/ECGOv7ehjB7HZBNQlbtz4 yQxiiwiYSRxs/QMU5+JgFvjIJLH6ewcLSEJYIFri06v5YA0sAqoS3etWsIHYvAKmEjuv7GKD GCovsXrDAbBBnECDdm47xQpiCwHVvJg9mQ1kqIRAM7vExZYZTBANkhIHV9xgmcDIu4CRYRWj UGZeWW5iZo6JXkZlXmaFXnJ+7iZGYEQsq/0TvYPx04XgQ4wCHIxKPLwnyg+nCrEmlhVX5h5i lOBgVhLhNdtxIFWINyWxsiq1KD++qDQntfgQozQHi5I4r9G38hQhgfTEktTs1NSC1CKYLBMH p1QDY6XuocZDPQ9WCp7qPbe6OWXZyXuH9zdyq5T6fI459VHlsJPyi/XmATz7t7M++nDgpHNn fPCcW9Pf3DI/VqNjXlT5+keX6Pf1aad/h4jP4o/mznz9pkf4YcJfttd/CplixRtkpTZ3zDY5 WtabJRNgqOqh1zLh8aoIM6O2L9evl71+yu83P/9ntBJLcUaioRZzUXEiAC+gwCiEAgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrBLMWRmVeSWpSXmKPExsXC5WfdrMtVciTVoKHB2mLO+jVsFp83/GOz eLGhndHi6/pfzBZPP/WxWByee5LV4vKuOWwW99b8Z7U4v2stq8WOpfuYLC4dWMBkcX3XQ0aL 470HmCzm3/vMZrF501Rmi+NTpjJa/P4B1HFy1mQWByGP7619LB47Z91l91iwqdRj8wotj8V7 XjJ5bFrVyeax6dMkdo93586xe5yY8ZvFY97JQI/3+66yeSx+8YHJY+svO4/GqdfYPD5vkvN4 N/8tW4BAFJdNSmpOZllqkb5dAlfGnr6H7AWzuSo2vW9na2Dcy9HFyMkhIWAisbajjx3EZhNQ l7hx4ycziC0iYCZxsPUPUJyLg1ngI5PE6u8dLCAJYYFoiU+v5oM1sAioSnSvW8EGYvMKmErs vLKLDWKovMTqDQfABnECDdq57RQriC0EVPNi9mS2CYxcCxgZVjGKZOaV5SZm5pjqFWdnVOZl Vugl5+duYgSG97LaPxN3MH657H6IUYCDUYmH90T54VQh1sSy4srcQ4wSHMxKIrxmOw6kCvGm JFZWpRblxxeV5qQWH2KU5mBREuf1Ck9NEBJITyxJzU5NLUgtgskycXBKNTA+9LU/HLTn3JX+ pOi5gd9Smm6+Vpi8f+b69xP2tb+0rq9Mdc76xquX+GrCe+FWFnPlz0Ib83LcuQSYdjnmsxyQ 17bxUWM/1n84PbRs8sXXTPWpau23m4USPTZVCpcKGRsGzbxqeSFti5GukNKb5JSaR2diEifO /Rp+Y1fF+3uNZd/+v7nq9kqJpTgj0VCLuag4EQDAObukawIAAA== X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: C6536160002 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: mpc11xqnugbnatwq1mhd47o6agzzefx5 X-HE-Tag: 1707373580-295948 X-HE-Meta: U2FsdGVkX18V51RVQh6t9zPoH4qu4ZBvGFSurMJdBr5upXrcTmFXIjGQMkItZljjkZTye3Tue2DfeO4oSnV0e/eyA5Y086qgR4SXDQSyh1ZoG/1XD8sibESuv+PEDOy4NRA3bBWPVKNlKNJjCxBCzHpv6CCkPiLUYxYAI/OdQd2G/hJwe0t2KnZaFSjqp2ZRtcwPyrFi5GBnTeLhlkNUkJ+AAylbAxmiAD/YKEGC9z7tZ+iud3BLS1v/wPn5WtHF++lrRhxS6xxSVCEtRwkNWhH3Wd5rmduDy0pckydxLFj5wG7v0wUPFWDJi3UafvZIhueIuiUNo0CUAgDCLUGLhkpArFM2DO1dx4p5qe8R0AipjQUCV+lgh2D0Gpd0qMe5HsUg+w0lQ9GGvSyGpRS1//8lrAQY3Y5VQ0PiutBjXyMX4A3i5wyAw7iaNNpA8iGxTMhHfETTdeFMcwj0XsapvgUqWPzW/BfYGbQHMvsXInVhg6b8dc7GGpH4PmcWX3hZ29/AyBdGY9FLNkhyhmBuGgG7zm1u0IV57upZvlmGUYOVipoQLfJpv397M6jnNos6sAnfc7crGlD0ct/K0FfxL/GZyHrWrul69XIeyKRqvepxnxCUcKFHmoiqNX3IZcInOb76g3Sti/aH9DsbMu5VB/m6CcTKZuqyGOo3FKZegdn6FUTiPhlHfBdNZ2hD/43YeFHh12+MHd19hgHq39ON+IljNylhKFBHBO+46+ELWXr9EAkvCe/Jes67vZoVil4xBCXImNBX2dTTRq2OnH2JdksCZgFZIWKyLdFYtSaYix9AhkV06tzbW1vRbXLBCKZII0BchySLKo3yL4DjPBgwdsmwKreqRvS+mxV9legr8e6G8JZrHGkf5Eyr9tE9nHH3hJ5zWLJse7XIpreEhCCQV5G2BVvtjBWe1uj/sGJLhvA030prO34arvAlm8+MyWtNEzuYP2K6RoGfF58m4eS ajw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparation for migrc mechanism that requires to manipulate tlb batch's arch data. Even though arm64 does nothing with it, arch with CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH should provide the APIs. Signed-off-by: Byungchul Park --- arch/arm64/include/asm/tlbflush.h | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index bb2c2833a987..4f2094843e7a 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -328,6 +328,25 @@ static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) dsb(ish); } +static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch) +{ + /* nothing to do */ +} + +static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + /* nothing to do */ +} + +static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + /* nothing to do */ + + return false; +} + /* * This is meant to avoid soft lock-ups on large TLB flushing ranges and not * necessarily a performance improvement. From patchwork Thu Feb 8 06:26:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13549357 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62286C48260 for ; Thu, 8 Feb 2024 06:26:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1FBA66B009B; Thu, 8 Feb 2024 01:26:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E2A1D6B009F; Thu, 8 Feb 2024 01:26:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C075D6B00A1; Thu, 8 Feb 2024 01:26:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 8515C6B009E for ; Thu, 8 Feb 2024 01:26:28 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 45857C05C8 for ; Thu, 8 Feb 2024 06:26:28 +0000 (UTC) X-FDA: 81767652456.16.5E813D3 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf14.hostedemail.com (Postfix) with ESMTP id 5ADAB100010 for ; Thu, 8 Feb 2024 06:26:26 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707373586; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=deHIYqkuaMMowWAwSsScVOmIndLCnwrqbZwxQiYz+QA=; b=DJ3TTM3rHo7I5Ww6MPomCAfqZ46zBppxmjTTrmrNClNgy5gRVJ2azWPNdgF4hl1OLHKdi1 RoT1qBiizXr8fX5Yh0bfwFlweJJLOrxe7lfqwsNAeeEDKc3zCJNNBjMaij0Qu/11bHMbnp b17O4vZsMmy5qYe3zf9CiyGVnVP165k= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707373586; a=rsa-sha256; cv=none; b=6rPDurBOPGHOmhiMC/0Al1S3loCFDjkqYXVpg48EdCI02x9UdHhL9Djt8B4wi9NUQg1qgb OuZxqnH83Q70eYPRY5cXTOsF4CZumqghFGaSQIpvgdlWMHncCS07iNC/FTmZlVwL06SyfC /Y7nspsQNOE+UTguWkTINTFLyjNnzTA= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none X-AuditID: a67dfc5b-d85ff70000001748-7d-65c4740bc993 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v7 3/8] mm/rmap: Recognize read-only TLB entries during batched TLB flush Date: Thu, 8 Feb 2024 15:26:03 +0900 Message-Id: <20240208062608.44351-4-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240208062608.44351-1-byungchul@sk.com> References: <20240208062608.44351-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrALMWRmVeSWpSXmKPExsXC9ZZnoS53yZFUg6bNAhZz1q9hs/i84R+b xYsN7YwWX9f/YrZ4+qmPxeLyrjlsFvfW/Ge1OL9rLavFjqX7mCwuHVjAZHF910NGi+O9B5gs 5t/7zGaxedNUZovjU6YyWvz+AdRxctZkFgdBj++tfSweO2fdZfdYsKnUY/MKLY/Fe14yeWxa 1cnmsenTJHaPd+fOsXucmPGbxWPeyUCP9/uusnls/WXn0Tj1GpvH501yHu/mv2UL4I/isklJ zcksSy3St0vgyvg6z6nguHTFh087GBsYH4h1MXJySAiYSHx+9JcFxu75PZURxGYTUJe4ceMn M4gtImAmcbD1D3sXIxcHs8BHJonV3zvAGoQFEiT6zyxkBbFZBFQlmv90gDXzCphKzN2xkhFi qLzE6g0HwAZxAg3aue0UWL0QUM2L2ZPZQIZKCPxnk5jbv50dokFS4uCKGywTGHkXMDKsYhTK zCvLTczMMdHLqMzLrNBLzs/dxAiMh2W1f6J3MH66EHyIUYCDUYmH90T54VQh1sSy4srcQ4wS HMxKIrxmOw6kCvGmJFZWpRblxxeV5qQWH2KU5mBREuc1+laeIiSQnliSmp2aWpBaBJNl4uCU amAMk/hgllfEfM3ljXTHGzsDk7yyw2KLzQRM34RWCbRbBc87bbdnhrGmfvHHgO27fqWeiuOe vYSf6+YE6fXXJFdPcYz5c+Wp3KOPwva7HnTzL3rZWBClty15kXkzm5POtis3d7+2Wzf95Qa+ Z1s88v1aZ4jsm9rL/1kzwbb3Cks+s+aFTTH8/P1KLMUZiYZazEXFiQB5E/uagwIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrOLMWRmVeSWpSXmKPExsXC5WfdrMtdciTVYPkEZos569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfVdDxkt jvceYLKYf+8zm8XmTVOZLY5Pmcpo8fsHUMfJWZNZHIQ8vrf2sXjsnHWX3WPBplKPzSu0PBbv ecnksWlVJ5vHpk+T2D3enTvH7nFixm8Wj3knAz3e77vK5rH4xQcmj62/7Dwap15j8/i8Sc7j 3fy3bAECUVw2Kak5mWWpRfp2CVwZX+c5FRyXrvjwaQdjA+MDsS5GTg4JAROJnt9TGUFsNgF1 iRs3fjKD2CICZhIHW/+wdzFycTALfGSSWP29gwUkISyQINF/ZiEriM0ioCrR/KcDrJlXwFRi 7o6VjBBD5SVWbzgANogTaNDObafA6oWAal7Mnsw2gZFrASPDKkaRzLyy3MTMHFO94uyMyrzM Cr3k/NxNjMDgXlb7Z+IOxi+X3Q8xCnAwKvHwnig/nCrEmlhWXJl7iFGCg1lJhNdsx4FUId6U xMqq1KL8+KLSnNTiQ4zSHCxK4rxe4akJQgLpiSWp2ampBalFMFkmDk6pBsbT6Qs2PjP8zBge GnD61x6zpztS2dVYI/78MDKWPyPr0e73o/j8M9srfp8+2R5ee/PFGmZbs31Wwgsdznr/WiIf spbRtHRVS+M7d89Mn+bwqU9M7ex3fI70eij2oMR7bbU3R5PUlJvi3Bcy9nL9jG3xyUpr0FGe PZ/PNfi0/oxJHG86vXQbTZRYijMSDbWYi4oTAeOGLtNqAgAA X-CFilter-Loop: Reflected X-Stat-Signature: w3tgdht67w78ae5o5ffq4dpy8fgw1xxr X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 5ADAB100010 X-Rspam-User: X-HE-Tag: 1707373586-920585 X-HE-Meta: U2FsdGVkX1+qYD/U8zRl7labyAfFk8sEZue76MtKt2ngBo+zQE2LJQUikoMRrdy5Y6db9lGqtlU9jgahLHjIFC/gWDVIr+u02C2xV78BTo2JNPWeqGkqsi65tXibErl8Kl+LcgdldNmFBNgotuaKtKUx6/MUPXj5VHpqaWelRtc+S5Ogtm6jfXnvYFIoXGmzIsU9Ym/EC2zNgo1m8ucxlkGIDxuYGbbEaPaItoFaz2kGV2hr3ebF+m6WG2rAWUsohc3vEsRWkJRVjc1nwZIzcDOXkaxZ1zLNckWBUFS0JuBh0aZhHnsKqX/zzqwJndPkJ9QnpKG/RC12b+5/2wbmCW8xH2g5gw/5q4Nrq0n4aUV62RwpVO2rQ9oqsJEBEYX53vT024IdcaQn/dzeFKzXoNvvges1+fKhbwOhTlbYcuf5ycBumIThS15vW5rJyRZR6nPmzwJ848WG+zbsk9fiof/L66KgsaFnx/81fdF/q9r516d1os2y4vC8DYplrAUjFX0Jmi0mNkykhc/wIQyZaAfUDbxyf5POI5uUXp1HbJU0OvKFcchrzZ78Cnn6dQc2pjGNT+9G9xw5sFsZmfWzjk8+vt+1eD5z4s/8qlReXHu9l1RfX5AC1jcH/EALZ3/ruQzgFmkLm9YAD8Be532a6b0jBNlWjrcmHS90tc7K8zsnfprbZA50t+MM3s9x6H7S5zBtHuSdZOt/cAzF3CLFEApYMx7h2AgIYgBTfkWLbC+CO8dYtb4DmW9cMQADGdT/ldxOo6D+CfxVN7gc47GVH9kR7FmToyzRY1S8T9FfWLnnU0WeZQgwwJVd446/l1A6a7XxZwYrhpZkbnjRQMtjsWWaUjniiuGe12/p/9ZBj5ZkMEincnxjtnqbsglnqLNL5Anr84kf3n2jzNlS2xeaHLEfpjH8p8xAlxc8u54wEktmFzRqIM8KsSJk4W5cPWdbvbb+sbI13tPLbBfovhZ Znw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Functionally, no change. This is a preparation for migrc mechanism that requires to recognize read-only TLB entries and makes use of them to batch more aggressively. Plus, the newly introduced API, fold_ubc() will be used by migrc mechanism when manipulating tlb batch data. Signed-off-by: Byungchul Park --- include/linux/sched.h | 1 + mm/internal.h | 4 ++++ mm/rmap.c | 31 ++++++++++++++++++++++++++++++- 3 files changed, 35 insertions(+), 1 deletion(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 292c31697248..0317e7a65151 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1328,6 +1328,7 @@ struct task_struct { #endif struct tlbflush_unmap_batch tlb_ubc; + struct tlbflush_unmap_batch tlb_ubc_ro; /* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe; diff --git a/mm/internal.h b/mm/internal.h index b61034bd50f5..b880f1e78700 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -923,6 +923,7 @@ extern struct workqueue_struct *mm_percpu_wq; void try_to_unmap_flush(void); void try_to_unmap_flush_dirty(void); void flush_tlb_batched_pending(struct mm_struct *mm); +void fold_ubc(struct tlbflush_unmap_batch *dst, struct tlbflush_unmap_batch *src); #else static inline void try_to_unmap_flush(void) { @@ -933,6 +934,9 @@ static inline void try_to_unmap_flush_dirty(void) static inline void flush_tlb_batched_pending(struct mm_struct *mm) { } +static inline void fold_ubc(struct tlbflush_unmap_batch *dst, struct tlbflush_unmap_batch *src) +{ +} #endif /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ extern const struct trace_print_flags pageflag_names[]; diff --git a/mm/rmap.c b/mm/rmap.c index 7a27a2b41802..da36f23ff7b0 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -605,6 +605,28 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, } #ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH + +void fold_ubc(struct tlbflush_unmap_batch *dst, + struct tlbflush_unmap_batch *src) +{ + if (!src->flush_required) + return; + + /* + * Fold src to dst. + */ + arch_tlbbatch_fold(&dst->arch, &src->arch); + dst->writable = dst->writable || src->writable; + dst->flush_required = true; + + /* + * Reset src. + */ + arch_tlbbatch_clear(&src->arch); + src->flush_required = false; + src->writable = false; +} + /* * Flush TLB entries for recently unmapped pages from remote CPUs. It is * important if a PTE was dirty when it was unmapped that it's flushed @@ -614,7 +636,9 @@ struct anon_vma *folio_lock_anon_vma_read(struct folio *folio, void try_to_unmap_flush(void) { struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc_ro = ¤t->tlb_ubc_ro; + fold_ubc(tlb_ubc, tlb_ubc_ro); if (!tlb_ubc->flush_required) return; @@ -645,13 +669,18 @@ void try_to_unmap_flush_dirty(void) static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, unsigned long uaddr) { - struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc; int batch; bool writable = pte_dirty(pteval); if (!pte_accessible(mm, pteval)) return; + if (pte_write(pteval) || writable) + tlb_ubc = ¤t->tlb_ubc; + else + tlb_ubc = ¤t->tlb_ubc_ro; + arch_tlbbatch_add_pending(&tlb_ubc->arch, mm, uaddr); tlb_ubc->flush_required = true; From patchwork Thu Feb 8 06:26:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13549356 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0615EC48260 for ; Thu, 8 Feb 2024 06:26:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E2D436B00A0; Thu, 8 Feb 2024 01:26:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CA4416B009B; Thu, 8 Feb 2024 01:26:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A0AFB6B00A0; Thu, 8 Feb 2024 01:26:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 820D16B009B for ; Thu, 8 Feb 2024 01:26:28 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 5687516070B for ; Thu, 8 Feb 2024 06:26:28 +0000 (UTC) X-FDA: 81767652456.16.DA207E7 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf08.hostedemail.com (Postfix) with ESMTP id 77F1D160002 for ; Thu, 8 Feb 2024 06:26:26 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf08.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707373586; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=JD5mtfEWkRjuaUV9XVtob8Xf345yQ1B8wbzU19SAuqU=; b=UnhajxlhEKXXypR4P7ftewUHT24FZB8C0JRDyHgppEzpnxZQKD5MfKn1v1dq9FbCOU6s58 RYQoahdIEY6lBgblcs+POxchJFTCybtQsrgZEHnNVds7kZNJeN8OdEprCfWdal2VBUd3CD CcQhEh9nDbmlNsy3HdXxwKBUp6Nw6gw= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf08.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707373586; a=rsa-sha256; cv=none; b=xzNFpK02FNgDx7yiPQb2KnR5mkD/0cPCGmdUTZHWLZYL37M+VuIErwNPU4ppOO52fZiReu C6wbrlDFdeWMDmO+1Q+euOrMNadFwwytWDQYz+LF6L5g3w/ZHAtW/s+r+25FB2gpFBBjGU elVCCi9m0Liz0lw7vXfDGb0DjNcRIaY= X-AuditID: a67dfc5b-d85ff70000001748-81-65c4740bb40b From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v7 4/8] x86/tlb, mm/rmap: Separate arch_tlbbatch_clear() out of arch_tlbbatch_flush() Date: Thu, 8 Feb 2024 15:26:04 +0900 Message-Id: <20240208062608.44351-5-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240208062608.44351-1-byungchul@sk.com> References: <20240208062608.44351-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrILMWRmVeSWpSXmKPExsXC9ZZnkS53yZFUg6sTRCzmrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeL6roeMFsd7DzBZ zL/3mc1i86apzBbHp0xltPj9A6jj5KzJLA6CHt9b+1g8ds66y+6xYFOpx+YVWh6L97xk8ti0 qpPNY9OnSewe786dY/c4MeM3i8e8k4Ee7/ddZfPY+svOo3HqNTaPz5vkPN7Nf8sWwB/FZZOS mpNZllqkb5fAlXH0zj22gjmcFed/TWVqYLzE3sXIySEhYCJx+tMfJhj7V/cfRhCbTUBd4saN n8wgtoiAmcTB1j9A9VwczAIfmSRWf+9gAUkIC+RI3GxfzQZiswioSsy8/wVsEK+AqUTr4llQ C+QlVm84ADaIE2jQzm2nWEFsIaCaF7Mns4EMlRBoZpc43/2AFaJBUuLgihssExh5FzAyrGIU yswry03MzDHRy6jMy6zQS87P3cQIjIhltX+idzB+uhB8iFGAg1GJh/dE+eFUIdbEsuLK3EOM EhzMSiK8ZjsOpArxpiRWVqUW5ccXleakFh9ilOZgURLnNfpWniIkkJ5YkpqdmlqQWgSTZeLg lGpgrBbt0RY0jZkcrlj/5bQdd4ZCSlPp27eLvBNfZcySEfFXjprSckDqmj13kNHrNZN28/m1 rY8wb/36tfgux8cff+ax7LAXLqt6tFBGrTlFQW311FdR5eYKV6Ynpe273VAZ5NL7WvrOyf6U Wyzf/Uw0xHavtjhb9fnhM9UqSd2bL6QvBXTv1lurxFKckWioxVxUnAgAhBx77oQCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrBLMWRmVeSWpSXmKPExsXC5WfdrMtdciTVYOs8Dos569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfVdDxkt jvceYLKYf+8zm8XmTVOZLY5Pmcpo8fsHUMfJWZNZHIQ8vrf2sXjsnHWX3WPBplKPzSu0PBbv ecnksWlVJ5vHpk+T2D3enTvH7nFixm8Wj3knAz3e77vK5rH4xQcmj62/7Dwap15j8/i8Sc7j 3fy3bAECUVw2Kak5mWWpRfp2CVwZR+/cYyuYw1lx/tdUpgbGS+xdjJwcEgImEr+6/zCC2GwC 6hI3bvxkBrFFBMwkDrb+Aarh4mAW+Mgksfp7BwtIQlggR+Jm+2o2EJtFQFVi5v0vTCA2r4Cp ROviWVBD5SVWbzgANogTaNDObadYQWwhoJoXsyezTWDkWsDIsIpRJDOvLDcxM8dUrzg7ozIv s0IvOT93EyMwvJfV/pm4g/HLZfdDjAIcjEo8vCfKD6cKsSaWFVfmHmKU4GBWEuE123EgVYg3 JbGyKrUoP76oNCe1+BCjNAeLkjivV3hqgpBAemJJanZqakFqEUyWiYNTqoFRTPr95sd1S8w9 LZq/r75bdG7CWt+PyyTdnzM9n83+wamJT/v/NuMazwTNLbtd1wp22HY/fvghbF7crN8mU+9m Oq99Lzvv1Lo97Iea9mW8TUnIKVAN276tXve9odtFlun84o9f/smb82Vl6leeQrenDzqXeW0x r1kxO6y9fY1QpDXPdb0m2zdMSizFGYmGWsxFxYkAJNj2imsCAAA= X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 77F1D160002 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: xsg7pja3eyhd81fawpgoetwkajmes4n3 X-HE-Tag: 1707373586-570056 X-HE-Meta: U2FsdGVkX18K8sKNgg8loyBPPCdRw+czSkeotIJJbzZZt+byqprarsXa5BVkilojlviAAshdrYZrCn5pW9AGcUlSSCkamoUynk6eqAVmbghpux7cJATCDOwdYMmCOuifGp0vvr/jXYTX3GctsnsmYSfpZRJbyn5ml8YLoVR6SJAP4Bl7jcJNmG7ONsTS7sjLbTbG8dka2bsVmJjsFpbAHhLgZrxz+XOD8a8RGWvCKZpuFvbc6rDH6chvrmJXIaJ+ae6XwKzcQFPFPDxy2OEO6QJZAzaZNrF6mSEVoJDcKSIy7e/zZ1fJeeebjbvTY/0B7mxl4El/tGLzvpGG94af9JZHkm818pI/asuh+1nobnPuASD4RdL5zuc8wdGRUml4HY6XeDe3R/pzacJuxPio1ospvkyHXN5AWO86Ttyh63VfB03ykNWHPpdM7i+Qr4Yk4057PtSFGToIgw5LAX64uH6e4xzaADrSZDogR9qSsInGGjUajqsFRw+RE/4VT87Lc3f/L+OGd4OrXBeNNQthuZNDUyLnpyfjsQJqBpQzpfDkZAshlXCtubAyLdYgMeIAylXfppwGqeUVd2clN/ibjmOFlFPZBaaxcjNkJlTbLPMF3cextD2RpbUS4Bqt+f1g8X7MQW/19OGmC1D0qLUjqxwR9IGVZT5O0GbOcBqk3otGjc6A6C1TfZW5+4kr4LbahsSlX9a+xDv9YqlpzNA7oYklbQ8gI2q0xGsf4ewH5nBA3gtnBgHQTPBYwSOXSUnNPm3cgqGmHV9qWrgQ6AapFvAwLJT72FojWJrpc763mZQ0wuzmed5KPiYeG1OKVqEXYb0AHtbZwLF3PzpMf+GV4IoDPmxhF6mez5n4Sopv5Xx+bV+d3+MqQLZyeM2TxS2WOy2xDCg+l4zNa2Zur9/ZSZ6tYmcNTmWqBP0g52QmQgOppoPdBNUOMd7Ptbmw8JaVvNLsJw2f8YWYnDKnHBX i/RQrDr6 6mzPXy+309FQr6rPpZzO2rT0xIi17vxz22vOW X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparation for migrc mechanism that requires to avoid redundant TLB flushes by manipulating tlb batch's arch data after arch_tlbbatch_flush(). However, we cannot because the data is getting cleared inside arch_tlbbatch_flush(). So separated the part clearing the tlb batch's arch data out of arch_tlbbatch_flush(). Signed-off-by: Byungchul Park --- arch/x86/mm/tlb.c | 2 -- mm/rmap.c | 1 + 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 453ea95b667d..941f41df02f3 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1268,8 +1268,6 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) local_irq_enable(); } - cpumask_clear(&batch->cpumask); - put_flush_tlb_info(); put_cpu(); } diff --git a/mm/rmap.c b/mm/rmap.c index da36f23ff7b0..b484d659d0c1 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -643,6 +643,7 @@ void try_to_unmap_flush(void) return; arch_tlbbatch_flush(&tlb_ubc->arch); + arch_tlbbatch_clear(&tlb_ubc->arch); tlb_ubc->flush_required = false; tlb_ubc->writable = false; } From patchwork Thu Feb 8 06:26:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13549358 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7CC60C4828F for ; Thu, 8 Feb 2024 06:26:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5EE1E6B009E; Thu, 8 Feb 2024 01:26:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5775E6B009F; Thu, 8 Feb 2024 01:26:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 415206B00A2; Thu, 8 Feb 2024 01:26:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0C8B86B009E for ; Thu, 8 Feb 2024 01:26:29 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D4986140282 for ; Thu, 8 Feb 2024 06:26:28 +0000 (UTC) X-FDA: 81767652456.13.9EDF435 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf02.hostedemail.com (Postfix) with ESMTP id E392B80024 for ; Thu, 8 Feb 2024 06:26:26 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf02.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707373587; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=hj/OU3wk3OzsqULfJnD2mvtgwTJyJb/ndDOd9gzDe6E=; b=spqFjy3T0gpP0JlWZDv37LH7oDKTIyxcKtVFEpfUZ1FOC2BlIkrkMkN8GuWsoHEN4Sl9lu ZsyUxW6OMO7y57lgNMp+YqUXvdP687MCzLHXfQeX+X0vWJy6t4brNdwA4NwZAugiD80IUU b4PRSj8CMLHyrkB9T/Me1Q1Ip+v7ELk= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf02.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707373587; a=rsa-sha256; cv=none; b=mMgQ9CJEQx0SQo/P66ywiZFLySuqUwLbgf+zBCVKGXQKPxRhJe0RpacpT9OHtgy5SMIvYP 4x9VRpD/3WSQUlJrY8jaJwpuLSYxN1wJGh7LM6lnzndZ1VdUjcmwHuzh0G3Co+N0ToW2hq tjTGj+kgd//WZK8GViodrgBHNNFkpJs= X-AuditID: a67dfc5b-d85ff70000001748-86-65c4740be438 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v7 5/8] mm: Separate move/undo doing on folio list from migrate_pages_batch() Date: Thu, 8 Feb 2024 15:26:05 +0900 Message-Id: <20240208062608.44351-6-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240208062608.44351-1-byungchul@sk.com> References: <20240208062608.44351-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrPLMWRmVeSWpSXmKPExsXC9ZZnoS53yZFUg99LJSzmrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeL6roeMFsd7DzBZ zL/3mc1i86apzBbHp0xltPj9A6jj5KzJLA6CHt9b+1g8ds66y+6xYFOpx+YVWh6L97xk8ti0 qpPNY9OnSewe786dY/c4MeM3i8e8k4Ee7/ddZfPY+svOo3HqNTaPz5vkPN7Nf8sWwB/FZZOS mpNZllqkb5fAlXHj/V32gn6dikVzHrI3MK5U7mLk4JAQMJFY9Y2ri5ETzDw7/x0riM0moC5x 48ZPZhBbRMBM4mDrH/YuRi4OZoGPTBKrv3ewgCSEBVIkvp5uZgOxWQRUJbb+u80IYvMKmEq8 f7KSHWKovMTqDQfABnECDdq57RTYAiGgmhezJ7NB1LSzSyy9YwZhS0ocXHGDZQIj7wJGhlWM Qpl5ZbmJmTkmehmVeZkVesn5uZsYgdGwrPZP9A7GTxeCDzEKcDAq8fCeKD+cKsSaWFZcmXuI UYKDWUmE12zHgVQh3pTEyqrUovz4otKc1OJDjNIcLErivEbfylOEBNITS1KzU1MLUotgskwc nFINjOvUp+aIq0mc+5SbIprtwXXA6fDf64oWs197LtvJUbt9h+bLjW189yYEMXmZMnQYybUK V5SKdUn87TK/M2HjWu/lJ/dvWKs0328yY/xElVuT/v5+/vJClG7KwuJtrzdZL7KtKjdoNnsl ZshQY/TGffm8clZTkQiHEr+rZRZ3OSM6lx+SkPNVVWIpzkg01GIuKk4EAIuevTiCAgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrBLMWRmVeSWpSXmKPExsXC5WfdrMtdciTV4MAKbos569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfVdDxkt jvceYLKYf+8zm8XmTVOZLY5Pmcpo8fsHUMfJWZNZHIQ8vrf2sXjsnHWX3WPBplKPzSu0PBbv ecnksWlVJ5vHpk+T2D3enTvH7nFixm8Wj3knAz3e77vK5rH4xQcmj62/7Dwap15j8/i8Sc7j 3fy3bAECUVw2Kak5mWWpRfp2CVwZN97fZS/o16lYNOchewPjSuUuRk4OCQETibPz37GC2GwC 6hI3bvxkBrFFBMwkDrb+Ye9i5OJgFvjIJLH6ewcLSEJYIEXi6+lmNhCbRUBVYuu/24wgNq+A qcT7JyvZIYbKS6zecABsECfQoJ3bToEtEAKqeTF7MtsERq4FjAyrGEUy88pyEzNzTPWKszMq 8zIr9JLzczcxAsN7We2fiTsYv1x2P8QowMGoxMN7ovxwqhBrYllxZe4hRgkOZiURXrMdB1KF eFMSK6tSi/Lji0pzUosPMUpzsCiJ83qFpyYICaQnlqRmp6YWpBbBZJk4OKUaGPVl5+yQcXTs e6p7f5FPv0Xi/QeMy/bW2y032XDl8e5t2UKOfnN3+XmyCCltOL2rp4o5Q/bb/NLNGic+el44 zs23pmP7nMJPMfXn2WrMr75WnqzV/5WDZRrTz3vbDS8cPpZwfuMj7tpo0eosDtsUH639kdwm n0puic3iyj/ldXTihb9TlvfyuSqxFGckGmoxFxUnAgBFmRGYawIAAA== X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: E392B80024 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 5bn99bd14p99pmi3juope11t765999cg X-HE-Tag: 1707373586-784905 X-HE-Meta: U2FsdGVkX19lQTbQf98gEsuUXKD/2TnC/2ic/Y5NEJLssKJkR4FspuDJw3LdTmvh4Z7GlggmuW+v2ePiIjJG4Be5H8wSMR3jU1B9npktTbanAM39nA9dYCyp/M/e9e1vStdzO3cXL1J1OGI2xkfVVdqoSr5T4ocNPp0IKOVR5/UwhY861O+yUl8tnpvieOAtcWGhdugQR/2UpdOjS8KObYIHoid62K+ZQFPcGwfBC81+TT/LRTXRNohkTYPtJRgTebYgph7IE3Cu34bdB2pLTxNSWbo3P4idhJHny+4UhLaf8j395pun9Iprdv2Bk5/iWgCfdGemHtGh91TAxicbvo1FUka1WK0KYbN15Yy4Jhe1zp/UqTDQisSFzsuvn66Po+GLdnd+1NFSw8AmiWu1piB+DLTCBWiH5WVG1mAmpv92XyZsR7a4AWp4NFf3uvB/+ElKAbaonzZEVH5mabZzJE0CUrFi2CA4GdpN+a3bs369TbTufQUuEMTBKQ0EZlcWzjUQvuNLfQc+rnzGQPztXo4+I3TxYvFr89yyHMk5oc04ifOO9mFsxKVvRp7WlcO6Ux0AbvCtB9DSYHl4k6wNUfDOmRilpyiN7pdW5omWgbgVMr+yVgKMym3ej5Pk8DdXjwCy+4MzOhKFT7I3ms4Yx0mjUXC5bn50sg4+6sNGRJlCO6zDrfeZhFVPi6uErXwe2Lf/iJibioT0ImNY78I+9LtgZwtd46sq91CZEDdMbNbeoPzdiNsZWKP+7dK0+6PrlCrgZ9dpB4zrRJLt8FJzUxGiI5IGzCiGZEXCcHaXuepw/c+8MB3o3jdqm82+PPdfqRsCx3GLs1dgB1QVJceWUHjjyMs1sR2JXZfiT9IkdBmMKGMTPR70D8sThACHdBalKjElF93pIcTJQiZA8vzEcPqHuBmuG/FRPYRlOPR86CY3fz4i6XpyHQao1aHYJnBrBVAe02YWaIDKfFZBqyc 4wUiNf4+ 3z7J+6xAzVDS8Lx18Go65HeQ+gQB5QIQYzVhw X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Functionally, no change. This is a preparation for migrc mechanism that requires to use separate folio lists for its own handling at migration. Refactored migrate_pages_batch() and separated move and undo parts operating on folio list, from migrate_pages_batch(). Signed-off-by: Byungchul Park --- mm/migrate.c | 134 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 83 insertions(+), 51 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 397f2a6e34cb..bbe1ecef4956 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1611,6 +1611,81 @@ static int migrate_hugetlbs(struct list_head *from, new_folio_t get_new_folio, return nr_failed; } +static void migrate_folios_move(struct list_head *src_folios, + struct list_head *dst_folios, + free_folio_t put_new_folio, unsigned long private, + enum migrate_mode mode, int reason, + struct list_head *ret_folios, + struct migrate_pages_stats *stats, + int *retry, int *thp_retry, int *nr_failed, + int *nr_retry_pages) +{ + struct folio *folio, *folio2, *dst, *dst2; + bool is_thp; + int nr_pages; + int rc; + + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio); + nr_pages = folio_nr_pages(folio); + + cond_resched(); + + rc = migrate_folio_move(put_new_folio, private, + folio, dst, mode, + reason, ret_folios); + /* + * The rules are: + * Success: folio will be freed + * -EAGAIN: stay on the unmap_folios list + * Other errno: put on ret_folios list + */ + switch(rc) { + case -EAGAIN: + *retry += 1; + *thp_retry += is_thp; + *nr_retry_pages += nr_pages; + break; + case MIGRATEPAGE_SUCCESS: + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; + break; + default: + *nr_failed += 1; + stats->nr_thp_failed += is_thp; + stats->nr_failed_pages += nr_pages; + break; + } + dst = dst2; + dst2 = list_next_entry(dst, lru); + } +} + +static void migrate_folios_undo(struct list_head *src_folios, + struct list_head *dst_folios, + free_folio_t put_new_folio, unsigned long private, + struct list_head *ret_folios) +{ + struct folio *folio, *folio2, *dst, *dst2; + + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + int old_page_state = 0; + struct anon_vma *anon_vma = NULL; + + __migrate_folio_extract(dst, &old_page_state, &anon_vma); + migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED, + anon_vma, true, ret_folios); + list_del(&dst->lru); + migrate_folio_undo_dst(dst, true, put_new_folio, private); + dst = dst2; + dst2 = list_next_entry(dst, lru); + } +} + /* * migrate_pages_batch() first unmaps folios in the from list as many as * possible, then move the unmapped folios. @@ -1633,7 +1708,7 @@ static int migrate_pages_batch(struct list_head *from, int pass = 0; bool is_thp = false; bool is_large = false; - struct folio *folio, *folio2, *dst = NULL, *dst2; + struct folio *folio, *folio2, *dst = NULL; int rc, rc_saved = 0, nr_pages; LIST_HEAD(unmap_folios); LIST_HEAD(dst_folios); @@ -1769,42 +1844,11 @@ static int migrate_pages_batch(struct list_head *from, thp_retry = 0; nr_retry_pages = 0; - dst = list_first_entry(&dst_folios, struct folio, lru); - dst2 = list_next_entry(dst, lru); - list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { - is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio); - nr_pages = folio_nr_pages(folio); - - cond_resched(); - - rc = migrate_folio_move(put_new_folio, private, - folio, dst, mode, - reason, ret_folios); - /* - * The rules are: - * Success: folio will be freed - * -EAGAIN: stay on the unmap_folios list - * Other errno: put on ret_folios list - */ - switch(rc) { - case -EAGAIN: - retry++; - thp_retry += is_thp; - nr_retry_pages += nr_pages; - break; - case MIGRATEPAGE_SUCCESS: - stats->nr_succeeded += nr_pages; - stats->nr_thp_succeeded += is_thp; - break; - default: - nr_failed++; - stats->nr_thp_failed += is_thp; - stats->nr_failed_pages += nr_pages; - break; - } - dst = dst2; - dst2 = list_next_entry(dst, lru); - } + /* Move the unmapped folios */ + migrate_folios_move(&unmap_folios, &dst_folios, + put_new_folio, private, mode, reason, + ret_folios, stats, &retry, &thp_retry, + &nr_failed, &nr_retry_pages); } nr_failed += retry; stats->nr_thp_failed += thp_retry; @@ -1813,20 +1857,8 @@ static int migrate_pages_batch(struct list_head *from, rc = rc_saved ? : nr_failed; out: /* Cleanup remaining folios */ - dst = list_first_entry(&dst_folios, struct folio, lru); - dst2 = list_next_entry(dst, lru); - list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { - int old_page_state = 0; - struct anon_vma *anon_vma = NULL; - - __migrate_folio_extract(dst, &old_page_state, &anon_vma); - migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED, - anon_vma, true, ret_folios); - list_del(&dst->lru); - migrate_folio_undo_dst(dst, true, put_new_folio, private); - dst = dst2; - dst2 = list_next_entry(dst, lru); - } + migrate_folios_undo(&unmap_folios, &dst_folios, + put_new_folio, private, ret_folios); return rc; } From patchwork Thu Feb 8 06:26:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13549359 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC37AC4829C for ; Thu, 8 Feb 2024 06:26:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 098E26B00A2; Thu, 8 Feb 2024 01:26:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F3DAB6B009F; Thu, 8 Feb 2024 01:26:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D54716B00A3; Thu, 8 Feb 2024 01:26:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9F7706B009F for ; Thu, 8 Feb 2024 01:26:30 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 79E08C01F3 for ; Thu, 8 Feb 2024 06:26:30 +0000 (UTC) X-FDA: 81767652540.04.2B2FBA5 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf14.hostedemail.com (Postfix) with ESMTP id 9606B10000C for ; Thu, 8 Feb 2024 06:26:28 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707373589; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=xyww7pMUuLyPBfG9ZuK6U6g9ZtqJWjn+tCNK2s25zA8=; b=K43rOcWm2yEB0yXv7RWxLQdOkcr+vcjA6Ok+0vpbWNHftftkd0TlD0bC+tDWeNsTgNaPIk ZbNLqPW6Dha2Y7xzIjlSUKNBDDtV8pY4IEtYTh3vwsu3rvQMqXX6GxodC5LsJYNOajYY7U UHNciX+41vYv0hPvzX/wLK5bLvUyS7s= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707373589; a=rsa-sha256; cv=none; b=lNt98Xi5JypLMcQOtvpH2n/r1gND6PMVwtpEjm356q5QLIumQ9uYFVlhiyDwJxayqMfRyI 6oSbZjbPH0Wl+jPJZYiyznT0d8Rx6Lirknmo2vK1hQh8rrRNhZ6gdIiNQY1LwnkQ5cMkuS rh78aJbJ6L+2fvD29X9OMc1wH0SuI2A= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none X-AuditID: a67dfc5b-d85ff70000001748-8a-65c4740b3aca From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v7 6/8] mm: Add APIs to free a folio directly to the buddy bypassing pcp Date: Thu, 8 Feb 2024 15:26:06 +0900 Message-Id: <20240208062608.44351-7-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240208062608.44351-1-byungchul@sk.com> References: <20240208062608.44351-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrALMWRmVeSWpSXmKPExsXC9ZZnkS53yZFUg6X3pS3mrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeL6roeMFsd7DzBZ zL/3mc1i86apzBbHp0xltPj9A6jj5KzJLA6CHt9b+1g8ds66y+6xYFOpx+YVWh6L97xk8ti0 qpPNY9OnSewe786dY/c4MeM3i8e8k4Ee7/ddZfPY+svOo3HqNTaPz5vkPN7Nf8sWwB/FZZOS mpNZllqkb5fAldFxna3gj1TF4eO7mRsYf4h2MXJySAiYSPx9vIQRxr508xwLiM0moC5x48ZP ZhBbRMBM4mDrH/YuRi4OZoGPTBKrv3eAFQkLxEvMOLAazGYRUJW4cewdWAOvgKnElH2fmSCG ykus3nAALM4JNGjntlOsILYQUM2L2ZPZQIZKCDSzS0x/OIMFokFS4uCKGywTGHkXMDKsYhTK zCvLTczMMdHLqMzLrNBLzs/dxAiMh2W1f6J3MH66EHyIUYCDUYmH90T54VQh1sSy4srcQ4wS HMxKIrxmOw6kCvGmJFZWpRblxxeV5qQWH2KU5mBREuc1+laeIiSQnliSmp2aWpBaBJNl4uCU amAsn+fk1X/zMVvXtPUi95U7J/3kuNDX9+fysunXT17TeHYnUUK0tzzilaIku+vF/1V9v1+x SJjYOqrMn8YbWKfE53t99UzDbPO2Kewd631afBoeTvYS7rU2+6j7qftehvVz3r2nKk7euP9M qizDu87t4PP+AqUkee3tkl4Cx16/jep1lnk3Y44SS3FGoqEWc1FxIgCrGM6WgwIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrOLMWRmVeSWpSXmKPExsXC5WfdrMtdciTVoPu8gMWc9WvYLD5v+Mdm 8WJDO6PF1/W/mC2efupjsTg89ySrxeVdc9gs7q35z2pxftdaVosdS/cxWVw6sIDJ4vquh4wW x3sPMFnMv/eZzWLzpqnMFsenTGW0+P0DqOPkrMksDkIe31v7WDx2zrrL7rFgU6nH5hVaHov3 vGTy2LSqk81j06dJ7B7vzp1j9zgx4zeLx7yTgR7v911l81j84gOTx9Zfdh6NU6+xeXzeJOfx bv5btgCBKC6blNSczLLUIn27BK6MjutsBX+kKg4f383cwPhDtIuRk0NCwETi0s1zLCA2m4C6 xI0bP5lBbBEBM4mDrX/Yuxi5OJgFPjJJrP7eAVYkLBAvMePAajCbRUBV4saxd2ANvAKmElP2 fWaCGCovsXrDAbA4J9CgndtOsYLYQkA1L2ZPZpvAyLWAkWEVo0hmXlluYmaOqV5xdkZlXmaF XnJ+7iZGYHAvq/0zcQfjl8vuhxgFOBiVeHhPlB9OFWJNLCuuzD3EKMHBrCTCa7bjQKoQb0pi ZVVqUX58UWlOavEhRmkOFiVxXq/w1AQhgfTEktTs1NSC1CKYLBMHp1QD492z8W6P5Z95z+n/ MOfwUXGW1feapvXGXpUz+b8t6qzOxv8flk+PEX59PfL1TdWn/meNV9v7xrblizvNa42PP/o0 actxlspLuRwMtosnvm3b/+f8S4mnvNuONGfqZ/hYOj0wa/k+veKI3rPpa6I3L3Jibdl2UO2t lOepdz4z2C63dvZPOWf6lUmJpTgj0VCLuag4EQAsrNOuagIAAA== X-CFilter-Loop: Reflected X-Stat-Signature: pozsak5ju7pa7p9o1jm579kmpqxghhyc X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 9606B10000C X-Rspam-User: X-HE-Tag: 1707373588-256095 X-HE-Meta: U2FsdGVkX18P/ykaKgxx8zGIpNKQMfvWyv7tObsUR+iKzBQeEiXGQPPUbcvYBov+qYFDg1jy2/TjDQcS6GxxMScBDVQKOkObSK0beLF9Fmm8fGQ7bMQdNZCYsMZzOpg3FSJrlirW45UxGTeTzZzTZw9RVps7wDoJZGP2LwWQAOzFkZca6+ZCG9UecLWM/Tf/DfIXMXvW1jodiLnwWEjVWWdfIZPQ0OLvG5C8aXixj7oDgr3tG9MINrTq4JWYhdVeWz1cPCRfLzx2W/WwnrxBfyUZzyq3iNMulwCU42oR1Zcbsz12MhyU3UAyF94wkF0iocCRBf1splBkN5Fq33HR5SbaHM7Z4o4jf1NvGaGghEN5DuL4BNK1bhD75+HZ/eUY9dus18rf7bn8YFKZLwUK0ccs2OcyZ5EV/h8famOlHe7voMXhK8s+WZUmVtllsmd58dOyd4/5IFM32HExtnknCc3HNTsrffkOPsWDPgVOkieIBLYTzqGM69OitMipRlEI5tEF9FvIYMPuQMdrgK/aJHsYQp9vyOAbTaswtvP3h4UzTOBpWpK4erC5bPJoHnzEd1tsdEjj5NQLWtWFo0GdnqMAE8WDi3i5GBWYAji7N6B3SNpMd2VH28Gsv6uhcFeB/K5vcuhoZsI2JkmNNXTYecMG7FNB/Q9utS431yZUm6WzuNkE27NSJHslq5sMH3cGZqQsCnYYNC+yoodYLg06GkE64Ul0tNKzWrzTRWuAL6+J2wTb2dsK2BZJSFEZNxhvOx3Ht4T79HuPySGXiBvl/FiLiGnXdieD7FrZsbDHDgjfUC/CJSIuOdrmdn9kDlo0S7d0955GRRNPoEnivO3A1E3zgE27FmIyCK8lFB7KzP0pKvs+gdCaK1rAmYPH48qV/crY6lVb2pOpJ7cI78o1WckXi2xeKm1LuPz+oJaUD6mab3zDVK4b3vr2iBt1Nr/AjEUgxvlP8Hjw9ULV/Qs ye6xb7KV NOnJ8DVzEGtURErckKcZCmhbj9Yxo/oViEBhDUBELLULdrRI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparation for migrc mechanism that frees folios at a better time later, rather than the moment migrating folios. The folios freed by migrc are too old to keep in pcp. Signed-off-by: Byungchul Park --- include/linux/mm.h | 23 +++++++++++++++++++++++ mm/internal.h | 1 + mm/page_alloc.c | 10 ++++++++++ mm/swap.c | 7 +++++++ 4 files changed, 41 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index da5219b48d52..fc0581cce3a7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1284,6 +1284,7 @@ static inline struct folio *virt_to_folio(const void *x) } void __folio_put(struct folio *folio); +void __folio_put_small_nopcp(struct folio *folio); void put_pages_list(struct list_head *pages); @@ -1483,6 +1484,28 @@ static inline void folio_put(struct folio *folio) __folio_put(folio); } +/** + * folio_put_small_nopcp - Decrement the reference count on a folio. + * @folio: The folio. + * + * This is only for a single page folio to release directly to the buddy + * allocator bypassing pcp. + * + * If the folio's reference count reaches zero, the memory will be + * released back to the page allocator and may be used by another + * allocation immediately. Do not access the memory or the struct folio + * after calling folio_put_small_nopcp() unless you can be sure that it + * wasn't the last reference. + * + * Context: May be called in process or interrupt context, but not in NMI + * context. May be called while holding a spinlock. + */ +static inline void folio_put_small_nopcp(struct folio *folio) +{ + if (folio_put_testzero(folio)) + __folio_put_small_nopcp(folio); +} + /** * folio_put_refs - Reduce the reference count on a folio. * @folio: The folio. diff --git a/mm/internal.h b/mm/internal.h index b880f1e78700..3be8fd5604e8 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -451,6 +451,7 @@ extern int user_min_free_kbytes; extern void free_unref_page(struct page *page, unsigned int order); extern void free_unref_page_list(struct list_head *list); +extern void free_pages_nopcp(struct page *page, unsigned int order); extern void zone_pcp_reset(struct zone *zone); extern void zone_pcp_disable(struct zone *zone); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 733732e7e0ba..21b8c8cd1673 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -565,6 +565,16 @@ static inline void free_the_page(struct page *page, unsigned int order) __free_pages_ok(page, order, FPI_NONE); } +void free_pages_nopcp(struct page *page, unsigned int order) +{ + /* + * This function will be used in case that the pages are too + * cold to keep in pcp e.g. migrc mechanism. So it'd better + * release the pages to the tail. + */ + __free_pages_ok(page, order, FPI_TO_TAIL); +} + /* * Higher-order pages are called "compound pages". They are structured thusly: * diff --git a/mm/swap.c b/mm/swap.c index cd8f0150ba3a..3f37496a1184 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -106,6 +106,13 @@ static void __folio_put_small(struct folio *folio) free_unref_page(&folio->page, 0); } +void __folio_put_small_nopcp(struct folio *folio) +{ + __page_cache_release(folio); + mem_cgroup_uncharge(folio); + free_pages_nopcp(&folio->page, 0); +} + static void __folio_put_large(struct folio *folio) { /* From patchwork Thu Feb 8 06:26:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13549361 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C9ECC4829C for ; Thu, 8 Feb 2024 06:26:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B21836B00A4; Thu, 8 Feb 2024 01:26:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A33106B00A3; Thu, 8 Feb 2024 01:26:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 80EFA6B00A4; Thu, 8 Feb 2024 01:26:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 68A856B00A1 for ; Thu, 8 Feb 2024 01:26:31 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 473074072F for ; Thu, 8 Feb 2024 06:26:31 +0000 (UTC) X-FDA: 81767652582.05.94642E0 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf02.hostedemail.com (Postfix) with ESMTP id 39BD580006 for ; Thu, 8 Feb 2024 06:26:28 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf02.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707373589; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=FKGmg+DAz0FrX3KjM/m3iaGERs+2p+YP5q2ZhuB76+k=; b=n2MJ2b3SVXmK77/bmfvuXO30EXMQm/hCYFej/XyOF1BKOYPQNGvK73KDZmCtl+h6ijuqnY xFjiMFUsobEvvzL2PzSfY05pvLahq3J/4iwcHYBUa/ZTe66T3IbYHx1lGcE5KhhPg+OPHJ gxEiNF2fvpn+hI5jH3BdSMxMYqJK5V4= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf02.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707373589; a=rsa-sha256; cv=none; b=JfupFlmBnxFk5quuxC5Yad9FivIlF+qhgBSZY51BIQ0XIDpzyZyOhO8Vq+d8rWC8GfiPim mYZEZNUHuTt5eJr8Y0SuehxRTgclKtKx9Sw8LTj1ZUgMwIjI9Xb3kXIOyrRZsCzj+YkZH3 VfoKsZBaurvQt7EhzbkYiUnaQuXIxf8= X-AuditID: a67dfc5b-d85ff70000001748-8e-65c4740bd871 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v7 7/8] mm: Defer TLB flush by keeping both src and dst folios at migration Date: Thu, 8 Feb 2024 15:26:07 +0900 Message-Id: <20240208062608.44351-8-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240208062608.44351-1-byungchul@sk.com> References: <20240208062608.44351-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrILMWRmVeSWpSXmKPExsXC9ZZnoS53yZFUg19TlCzmrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeL6roeMFsd7DzBZ zL/3mc1i86apzBbHp0xltPj9A6jj5KzJLA6CHt9b+1g8ds66y+6xYFOpx+YVWh6L97xk8ti0 qpPNY9OnSewe786dY/c4MeM3i8e8k4Ee7/ddZfPY+svOo3HqNTaPz5vkPN7Nf8sWwB/FZZOS mpNZllqkb5fAlfHgBHtBzyrGiv/H/jI3MF5oY+xi5OSQEDCR2NNwnRXGvnRgPwuIzSagLnHj xk9mEFtEwEziYOsf9i5GLg5mgY9MEqu/d4AVCQskSXTuOAXUzMHBIqAqsfa/NEiYV8BUYsH3 0ywQM+UlVm84ADaHE2jOzm2nwHYJAdW8mD2ZDWSmhEA7u8T9i8ehjpCUOLjiBssERt4FjAyr GIUy88pyEzNzTPQyKvMyK/SS83M3MQIjYlntn+gdjJ8uBB9iFOBgVOLhPVF+OFWINbGsuDL3 EKMEB7OSCK/ZjgOpQrwpiZVVqUX58UWlOanFhxilOViUxHmNvpWnCAmkJ5akZqemFqQWwWSZ ODilGhjnR37bMKXmTdi0959WS/WwiH0wEre7EPBTQyzooXlLwnWhXEE2vWtPzyQ9XPwvjNX5 cMDjSdufzfFWU63ZfebdNbbpnd/WL5I6avNDYEVr8iUn9dYNftGqky59TNly25R7yskfKz6r Pv1ksHi565E7fDXtRRpTU3nrjrC45HCtmBFr9eSS/L4jSizFGYmGWsxFxYkAlBLKBoQCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrBLMWRmVeSWpSXmKPExsXC5WfdrMtdciTV4N8UEYs569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfVdDxkt jvceYLKYf+8zm8XmTVOZLY5Pmcpo8fsHUMfJWZNZHIQ8vrf2sXjsnHWX3WPBplKPzSu0PBbv ecnksWlVJ5vHpk+T2D3enTvH7nFixm8Wj3knAz3e77vK5rH4xQcmj62/7Dwap15j8/i8Sc7j 3fy3bAECUVw2Kak5mWWpRfp2CVwZD06wF/SsYqz4f+wvcwPjhTbGLkZODgkBE4lLB/azgNhs AuoSN278ZAaxRQTMJA62/mHvYuTiYBb4yCSx+nsHWJGwQJJE545TrF2MHBwsAqoSa/9Lg4R5 BUwlFnw/zQIxU15i9YYDYHM4gebs3AZSzskhBFTzYvZktgmMXAsYGVYximTmleUmZuaY6hVn Z1TmZVboJefnbmIEhvey2j8TdzB+uex+iFGAg1GJh/dE+eFUIdbEsuLK3EOMEhzMSiK8ZjsO pArxpiRWVqUW5ccXleakFh9ilOZgURLn9QpPTRASSE8sSc1OTS1ILYLJMnFwSjUwlr6Mmv1B c0m91Xm79OssFwOrbybmhk74LVxr+/Tjs8Mcvc3ezetnnp1mxPX/DY8dl3FbzawAwciObasm 3D587yL3f6OFKo9M7U6HfFsVt+t4Qo/9JP53FktWOQkEbFP7+FyHmT/W+oLEK7ay5Us44kNs o5e8/yO2qUfi5pUm34d/Ms4JTE4yVGIpzkg01GIuKk4EACPO4/JrAgAA X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 39BD580006 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: rkh3s6ibkrhe94eetriq3wieqeg96mes X-HE-Tag: 1707373588-108729 X-HE-Meta: U2FsdGVkX1+JK5QYSHVw1Skxl8NdUoLn8UAvVv76SNUDTdoKCMx7fRpuzdIbbGfw4YUR3/v1xENKqg3X/1bUGqTiCB0tXu7d+HjmiEPOe4z79vCzw1lFvHhSJNiqYlRu3MYJhDMU0Em1mbjsrVNyu30mM7SyYi/92396RUP2uLf8EeXOoBiMFjj790tkXRiBjjO3RFu7gRPOgjQbMZbdi5sKwemFOuisAzcZdloj2oneHEjuAQaf17LeSD8A1Qm/b9+/BEGcz6e3bHWZNNue7TpzQubWFk5lsmFGd9Jx0ujymIPzXTmqsiHSjpu6Xuv8BIIbALBqCwW5r0WYNwO+Wz7VuHicThkpYIUT9XSEhmq2gEeDzTditmoLYKp2LGZiMXhoRgrtdvFObm8ue6oWQBZtfZ/cwndK83DiT2coSju1D+P96TfXRpplck7G8efevkKnQPaT2RyjXqI5MBRVoZLvFRiHzpheTZ19tuLk+kPHXKCXm/W282eYhn4vgmsH+IeCanTqmYr/awUWrsZtmlTQ3jtEksOLtMnMu2iZ5YmermNLul6Pw1j2XEyjm/VB8ALRt+BtwEj1DBA/kG+9hineqROwUXlERPB7T2Yl9Z5jGWYFq1SSLRaYlREJUlKmQJ5rdwzN8sLeJLbjm73DQrRrmb/0BhdzUCokFqrJvqSG0ztJNJBFkg5IzOU/fXVIqVN+rMe01/bz2xIkuLrm9KPMyM3TOruOWsOd5WlFPXz/qCXAFFDdNBVFDukzxMw6AmrBIVHE5274fgY44EyqFs24T4mb6/yF6+PooXAF8DFd7aD78W44FtJNkmALZenL+ZVI+sIzCI/yn9YBz/UX8ygdEQMkgC6vEELEp2CLESdjYZnBPTGBrtskhaq8pYtX+3ZrMYVItPYdCOnIMlw5FiusmTkIXZ//4Bzb36sO1X5+GCxV5vKZxISaqYw/GUqWjUbDseTvFM0eFoAA5HN uBw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Implementation of MIGRC mechanism that stands for 'Migration Read Copy'. We always face the migration overhead at either promotion or demotion, while working with tiered memory e.g. CXL memory and found out TLB shootdown is a quite big one that is needed to get rid of if possible. Fortunately, TLB flush can be defered if both source and destination of folios during migration are kept until all TLB flushes required will have been done, of course, only if the target PTE entries have read-only permission, more precisely speaking, don't have write permission. Otherwise, no doubt the folio might get messed up. To achieve that: 1. For the folios that map only to non-writable TLB entries, prevent TLB flush at migration by keeping both source and destination folios, which will be handled later at a better time. 2. When any non-writable TLB entry changes to writable e.g. through fault handler, give up migrc mechanism so as to perform TLB flush required right away. The following evaluation using XSBench shows the improvement like: 1. itlb flush was reduced by 93.9%. 2. dtlb thread was reduced by 43.5%. 3. stlb flush was reduced by 24.9%. 4. dtlb store misses was reduced by 34.2%. 5. itlb load misses was reduced by 45.5%. 6. The runtime was reduced by 3.5%. --- The measurement result: Architecture - x86_64 QEMU - kvm enabled, host cpu Numa - 2 nodes (16 CPUs 1GB, no CPUs 8GB) Linux Kernel - v6.7, numa balancing tiering on, demotion enabled Benchmark - XSBench -p 100000000 (-p option makes the runtime longer) run 'perf stat' using events: 1) itlb.itlb_flush 2) tlb_flush.dtlb_thread 3) tlb_flush.stlb_any 4) dTLB-load-misses 5) dTLB-store-misses 6) iTLB-load-misses run 'cat /proc/vmstat' and pick: 1) numa_pages_migrated 2) pgmigrate_success 3) nr_tlb_remote_flush 4) nr_tlb_remote_flush_received 5) nr_tlb_local_flush_all 6) nr_tlb_local_flush_one BEFORE - mainline v6.7 ------------------------------------------ $ perf stat -a \ -e itlb.itlb_flush \ -e tlb_flush.dtlb_thread \ -e tlb_flush.stlb_any \ -e dTLB-load-misses \ -e dTLB-store-misses \ -e iTLB-load-misses \ ./XSBench -p 100000000 Performance counter stats for 'system wide': 85647229 itlb.itlb_flush 480981504 tlb_flush.dtlb_thread 323937200 tlb_flush.stlb_any 238381632579 dTLB-load-misses 601514255 dTLB-store-misses 2974157461 iTLB-load-misses 2252.883892112 seconds time elapsed $ cat /proc/vmstat ... numa_pages_migrated 12790664 pgmigrate_success 26835314 nr_tlb_remote_flush 3031412 nr_tlb_remote_flush_received 45234862 nr_tlb_local_flush_all 216584 nr_tlb_local_flush_one 740940 ... AFTER - mainline v6.7 + migrc ------------------------------------------ $ perf stat -a \ -e itlb.itlb_flush \ -e tlb_flush.dtlb_thread \ -e tlb_flush.stlb_any \ -e dTLB-load-misses \ -e dTLB-store-misses \ -e iTLB-load-misses \ ./XSBench -p 100000000 Performance counter stats for 'system wide': 5240261 itlb.itlb_flush 271581774 tlb_flush.dtlb_thread 243149389 tlb_flush.stlb_any 234502983364 dTLB-load-misses 395673680 dTLB-store-misses 1620215163 iTLB-load-misses 2172.283436287 seconds time elapsed $ cat /proc/vmstat ... numa_pages_migrated 14897064 pgmigrate_success 30825530 nr_tlb_remote_flush 198290 nr_tlb_remote_flush_received 2820156 nr_tlb_local_flush_all 92048 nr_tlb_local_flush_one 741401 ... Signed-off-by: Byungchul Park --- include/linux/mmzone.h | 7 ++ include/linux/sched.h | 8 ++ mm/internal.h | 53 ++++++++ mm/memory.c | 8 ++ mm/migrate.c | 271 +++++++++++++++++++++++++++++++++++++++-- mm/page_alloc.c | 11 +- mm/rmap.c | 12 +- 7 files changed, 358 insertions(+), 12 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 9db36e197712..492111cd1176 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1002,6 +1002,13 @@ struct zone { /* Zone statistics */ atomic_long_t vm_stat[NR_VM_ZONE_STAT_ITEMS]; atomic_long_t vm_numa_event[NR_VM_NUMA_EVENT_ITEMS]; + +#if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) + /* + * the number of folios pending for TLB flush in the zone + */ + atomic_t migrc_pending_nr; +#endif } ____cacheline_internodealigned_in_smp; enum pgdat_flags { diff --git a/include/linux/sched.h b/include/linux/sched.h index 0317e7a65151..d8c285309a8f 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1330,6 +1330,14 @@ struct task_struct { struct tlbflush_unmap_batch tlb_ubc; struct tlbflush_unmap_batch tlb_ubc_ro; +#if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) + /* + * whether all the mappings of a folio during unmap are read-only + * so that migrc can work on the folio + */ + bool can_migrc; +#endif + /* Cache last used pipe for splice(): */ struct pipe_inode_info *splice_pipe; diff --git a/mm/internal.h b/mm/internal.h index 3be8fd5604e8..ab02cb8306e2 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1284,4 +1284,57 @@ static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry, } #endif /* CONFIG_SHRINKER_DEBUG */ +#if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) +/* + * Reset the indicator indicating there are no writable mappings at the + * beginning of every rmap traverse for unmap. Migrc can work only when + * all the mappings are read-only. + */ +static inline void can_migrc_init(void) +{ + current->can_migrc = true; +} + +/* + * Mark the folio is not applicable to migrc, once it found a writble or + * dirty pte during rmap traverse for unmap. + */ +static inline void can_migrc_fail(void) +{ + current->can_migrc = false; +} + +/* + * Check if all the mappings are read-only and read-only mappings even + * exist. + */ +static inline bool can_migrc_test(void) +{ + return current->can_migrc && current->tlb_ubc_ro.flush_required; +} + +/* + * Return the number of folios pending TLB flush that have yet to get + * freed in the zone. + */ +static inline int migrc_pending_nr_in_zone(struct zone *z) +{ + return atomic_read(&z->migrc_pending_nr); +} + +/* + * Perform TLB flush needed and free the folios under migrc's control. + */ +bool migrc_flush_free_folios(void); +void migrc_flush_start(void); +void migrc_flush_end(struct tlbflush_unmap_batch *batch); +#else /* CONFIG_MIGRATION && CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ +static inline void can_migrc_init(void) {} +static inline void can_migrc_fail(void) {} +static inline bool can_migrc_test(void) { return false; } +static inline int migrc_pending_nr_in_zone(struct zone *z) { return 0; } +static inline bool migrc_flush_free_folios(void) { return false; } +static inline void migrc_flush_start(void) {} +static inline void migrc_flush_end(struct tlbflush_unmap_batch *batch) {} +#endif #endif /* __MM_INTERNAL_H */ diff --git a/mm/memory.c b/mm/memory.c index 6e0712d06cd4..e67de161da8b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3462,6 +3462,14 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) if (vmf->page) folio = page_folio(vmf->page); + /* + * The folio may or may not be one that is under migrc's control + * and about to change its permission from read-only to writable. + * Conservatively give up deferring TLB flush just in case. + */ + if (folio) + migrc_flush_free_folios(); + /* * Shared mapping: we are guaranteed to have VM_WRITE and * FAULT_FLAG_WRITE set at this point. diff --git a/mm/migrate.c b/mm/migrate.c index bbe1ecef4956..cbe5372f159e 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -57,6 +57,194 @@ #include "internal.h" +#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH +static struct tlbflush_unmap_batch migrc_ubc; +static LIST_HEAD(migrc_folios); +static DEFINE_SPINLOCK(migrc_lock); + +static void init_tlb_ubc(struct tlbflush_unmap_batch *ubc) +{ + arch_tlbbatch_clear(&ubc->arch); + ubc->flush_required = false; + ubc->writable = false; +} + +static void migrc_keep_folio(struct folio *f, struct list_head *h) +{ + list_move_tail(&f->lru, h); + folio_get(f); + atomic_inc(&folio_zone(f)->migrc_pending_nr); +} + +static void migrc_undo_folio(struct folio *f) +{ + list_del(&f->lru); + folio_put(f); + atomic_dec(&folio_zone(f)->migrc_pending_nr); +} + +static void migrc_release_folio(struct folio *f) +{ + folio_put_small_nopcp(f); + atomic_dec(&folio_zone(f)->migrc_pending_nr); +} + +/* + * Need to synchronize between TLB flush and managing pending CPUs in + * migrc_ubc. Take a look at the following scenario: + * + * CPU0 CPU1 + * ---- ---- + * TLB flush + * Unmap folios (needing TLB flush) + * Add pending CPUs to migrc_ubc + * Clear the CPUs from migrc_ubc + * + * The pending CPUs added in CPU1 should not be cleared from migrc_ubc + * in CPU0 because the TLB flush for migrc_ubc added in CPU1 has not + * been performed this turn. To avoid this, using 'migrc_flushing' + * variable, prevent adding pending CPUs to migrc_ubc and give up migrc + * mechanism if others are in the middle of TLB flush, like: + * + * CPU0 CPU1 + * ---- ---- + * migrc_flushing++ + * TLB flush + * Unmap folios (needing TLB flush) + * If migrc_flushing == 0: + * Add pending CPUs to migrc_ubc + * Else: <--- hit + * Give up migrc mechanism + * Clear the CPUs from migrc_ubc + * migrc_flush-- + * + * Only the following case would be allowed for migrc mechanism to work: + * + * CPU0 CPU1 + * ---- ---- + * Unmap folios (needing TLB flush) + * If migrc_flushing == 0: <--- hit + * Add pending CPUs to migrc_ubc + * Else: + * Give up migrc mechanism + * migrc_flushing++ + * TLB flush + * Clear the CPUs from migrc_ubc + * migrc_flush-- + */ +static int migrc_flushing; + +static bool migrc_add_pending_ubc(struct tlbflush_unmap_batch *ubc) +{ + struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + if (migrc_flushing) { + spin_unlock_irqrestore(&migrc_lock, flags); + + /* + * Give up migrc mechanism. Just let TLB flush needed + * handled by try_to_unmap_flush() at the caller side. + */ + fold_ubc(tlb_ubc, ubc); + return false; + } + fold_ubc(&migrc_ubc, ubc); + spin_unlock_irqrestore(&migrc_lock, flags); + return true; +} + +static bool migrc_add_pending_folios(struct list_head *folios) +{ + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + if (migrc_flushing) { + spin_unlock_irqrestore(&migrc_lock, flags); + + /* + * Give up migrc mechanism. The caller should perform + * TLB flush needed using migrc_flush_free_folios() and + * undo some on the folios e.g. restore folios' + * reference count increased by migrc and more. + */ + return false; + } + list_splice(folios, &migrc_folios); + spin_unlock_irqrestore(&migrc_lock, flags); + return true; +} + +void migrc_flush_start(void) +{ + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + migrc_flushing++; + spin_unlock_irqrestore(&migrc_lock, flags); +} + +void migrc_flush_end(struct tlbflush_unmap_batch *batch) +{ + LIST_HEAD(folios); + struct folio *f, *f2; + unsigned long flags; + + spin_lock_irqsave(&migrc_lock, flags); + if (!arch_tlbbatch_done(&migrc_ubc.arch, &batch->arch)) { + list_splice_init(&migrc_folios, &folios); + migrc_ubc.flush_required = false; + migrc_ubc.writable = false; + } + migrc_flushing--; + spin_unlock_irqrestore(&migrc_lock, flags); + + list_for_each_entry_safe(f, f2, &folios, lru) + migrc_release_folio(f); +} + +bool migrc_flush_free_folios(void) +{ + struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + LIST_HEAD(folios); + struct folio *f, *f2; + unsigned long flags; + bool ret = true; + + spin_lock_irqsave(&migrc_lock, flags); + list_splice_init(&migrc_folios, &folios); + fold_ubc(tlb_ubc, &migrc_ubc); + spin_unlock_irqrestore(&migrc_lock, flags); + + if (list_empty(&folios)) + ret = false; + + try_to_unmap_flush(); + list_for_each_entry_safe(f, f2, &folios, lru) + migrc_release_folio(f); + return ret; +} +#else /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */ +static void init_tlb_ubc(struct tlbflush_unmap_batch *ubc) +{ +} +static void migrc_keep_folio(struct folio *f, struct list_head *h) +{ +} +static void migrc_undo_folio(struct folio *f) +{ +} +static bool migrc_add_pending_ubc(struct tlbflush_unmap_batch *ubc) +{ + return false; +} +static bool migrc_add_pending_folios(struct list_head *folios) +{ + return false; +} +#endif + bool isolate_movable_page(struct page *page, isolate_mode_t mode) { struct folio *folio = folio_get_nontail_page(page); @@ -1274,7 +1462,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, struct folio *src, struct folio *dst, enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) + struct list_head *ret, struct list_head *move_succ) { int rc; int old_page_state = 0; @@ -1321,9 +1509,13 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, /* * A folio that has been migrated has all references removed - * and will be freed. + * and will be freed, unless it's under migrc's control. */ - list_del(&src->lru); + if (move_succ) + migrc_keep_folio(src, move_succ); + else + list_del(&src->lru); + /* Drop an anon_vma reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); @@ -1618,7 +1810,7 @@ static void migrate_folios_move(struct list_head *src_folios, struct list_head *ret_folios, struct migrate_pages_stats *stats, int *retry, int *thp_retry, int *nr_failed, - int *nr_retry_pages) + int *nr_retry_pages, struct list_head *move_succ) { struct folio *folio, *folio2, *dst, *dst2; bool is_thp; @@ -1635,7 +1827,7 @@ static void migrate_folios_move(struct list_head *src_folios, rc = migrate_folio_move(put_new_folio, private, folio, dst, mode, - reason, ret_folios); + reason, ret_folios, move_succ); /* * The rules are: * Success: folio will be freed @@ -1712,17 +1904,34 @@ static int migrate_pages_batch(struct list_head *from, int rc, rc_saved = 0, nr_pages; LIST_HEAD(unmap_folios); LIST_HEAD(dst_folios); + LIST_HEAD(unmap_folios_migrc); + LIST_HEAD(dst_folios_migrc); + LIST_HEAD(move_succ); bool nosplit = (reason == MR_NUMA_MISPLACED); + struct tlbflush_unmap_batch pending_ubc; + struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + struct tlbflush_unmap_batch *tlb_ubc_ro = ¤t->tlb_ubc_ro; + bool do_migrc; + bool migrc_ubc_succ; VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC && !list_empty(from) && !list_is_singular(from)); + /* + * Apply migrc only to numa migration for now. + */ + init_tlb_ubc(&pending_ubc); + do_migrc = IS_ENABLED(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) && + (reason == MR_DEMOTION || reason == MR_NUMA_MISPLACED); + for (pass = 0; pass < nr_pass && retry; pass++) { retry = 0; thp_retry = 0; nr_retry_pages = 0; list_for_each_entry_safe(folio, folio2, from, lru) { + bool can_migrc; + is_large = folio_test_large(folio); is_thp = is_large && folio_test_pmd_mappable(folio); nr_pages = folio_nr_pages(folio); @@ -1752,9 +1961,12 @@ static int migrate_pages_batch(struct list_head *from, continue; } + can_migrc_init(); rc = migrate_folio_unmap(get_new_folio, put_new_folio, private, folio, &dst, mode, reason, ret_folios); + can_migrc = do_migrc && can_migrc_test() && !is_large; + /* * The rules are: * Success: folio will be freed @@ -1800,7 +2012,8 @@ static int migrate_pages_batch(struct list_head *from, /* nr_failed isn't updated for not used */ stats->nr_thp_failed += thp_retry; rc_saved = rc; - if (list_empty(&unmap_folios)) + if (list_empty(&unmap_folios) && + list_empty(&unmap_folios_migrc)) goto out; else goto move; @@ -1814,8 +2027,19 @@ static int migrate_pages_batch(struct list_head *from, stats->nr_thp_succeeded += is_thp; break; case MIGRATEPAGE_UNMAP: - list_move_tail(&folio->lru, &unmap_folios); - list_add_tail(&dst->lru, &dst_folios); + if (can_migrc) { + list_move_tail(&folio->lru, &unmap_folios_migrc); + list_add_tail(&dst->lru, &dst_folios_migrc); + + /* + * Gather ro batch data to add + * to migrc_ubc after unmap. + */ + fold_ubc(&pending_ubc, tlb_ubc_ro); + } else { + list_move_tail(&folio->lru, &unmap_folios); + list_add_tail(&dst->lru, &dst_folios); + } break; default: /* @@ -1829,12 +2053,19 @@ static int migrate_pages_batch(struct list_head *from, stats->nr_failed_pages += nr_pages; break; } + /* + * Done with the current folio. Fold the ro + * batch data gathered, to the normal batch. + */ + fold_ubc(tlb_ubc, tlb_ubc_ro); } } nr_failed += retry; stats->nr_thp_failed += thp_retry; stats->nr_failed_pages += nr_retry_pages; move: + /* Should be before try_to_unmap_flush() */ + migrc_ubc_succ = do_migrc && migrc_add_pending_ubc(&pending_ubc); /* Flush TLBs for all unmapped folios */ try_to_unmap_flush(); @@ -1848,7 +2079,27 @@ static int migrate_pages_batch(struct list_head *from, migrate_folios_move(&unmap_folios, &dst_folios, put_new_folio, private, mode, reason, ret_folios, stats, &retry, &thp_retry, - &nr_failed, &nr_retry_pages); + &nr_failed, &nr_retry_pages, NULL); + migrate_folios_move(&unmap_folios_migrc, &dst_folios_migrc, + put_new_folio, private, mode, reason, + ret_folios, stats, &retry, &thp_retry, + &nr_failed, &nr_retry_pages, migrc_ubc_succ ? + &move_succ : NULL); + } + + /* + * In case that migrc_add_pending_ubc() has been added + * successfully but migrc_add_pending_folios() does not. + */ + if (migrc_ubc_succ && !migrc_add_pending_folios(&move_succ)) { + migrc_flush_free_folios(); + + /* + * Undo src folios that have been successfully added to + * move_succ. + */ + list_for_each_entry_safe(folio, folio2, &move_succ, lru) + migrc_undo_folio(folio); } nr_failed += retry; stats->nr_thp_failed += thp_retry; @@ -1859,6 +2110,8 @@ static int migrate_pages_batch(struct list_head *from, /* Cleanup remaining folios */ migrate_folios_undo(&unmap_folios, &dst_folios, put_new_folio, private, ret_folios); + migrate_folios_undo(&unmap_folios_migrc, &dst_folios_migrc, + put_new_folio, private, ret_folios); return rc; } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 21b8c8cd1673..6ef0c22b1109 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2972,6 +2972,8 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark, long min = mark; int o; + free_pages += migrc_pending_nr_in_zone(z); + /* free_pages may go negative - that's OK */ free_pages -= __zone_watermark_unusable_free(z, order, alloc_flags); @@ -3066,7 +3068,7 @@ static inline bool zone_watermark_fast(struct zone *z, unsigned int order, long usable_free; long reserved; - usable_free = free_pages; + usable_free = free_pages + migrc_pending_nr_in_zone(z); reserved = __zone_watermark_unusable_free(z, 0, alloc_flags); /* reserved may over estimate high-atomic reserves. */ @@ -3273,6 +3275,13 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, gfp_mask)) { int ret; + if (migrc_pending_nr_in_zone(zone) && + migrc_flush_free_folios() && + zone_watermark_fast(zone, order, mark, + ac->highest_zoneidx, + alloc_flags, gfp_mask)) + goto try_this_zone; + if (has_unaccepted_memory()) { if (try_to_accept_memory(zone, order)) goto try_this_zone; diff --git a/mm/rmap.c b/mm/rmap.c index b484d659d0c1..39ab0d64665a 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -642,7 +642,9 @@ void try_to_unmap_flush(void) if (!tlb_ubc->flush_required) return; + migrc_flush_start(); arch_tlbbatch_flush(&tlb_ubc->arch); + migrc_flush_end(tlb_ubc); arch_tlbbatch_clear(&tlb_ubc->arch); tlb_ubc->flush_required = false; tlb_ubc->writable = false; @@ -677,9 +679,15 @@ static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, if (!pte_accessible(mm, pteval)) return; - if (pte_write(pteval) || writable) + if (pte_write(pteval) || writable) { tlb_ubc = ¤t->tlb_ubc; - else + + /* + * Migrc cannot work with the folio, once it found a + * writable or dirty mapping on it. + */ + can_migrc_fail(); + } else tlb_ubc = ¤t->tlb_ubc_ro; arch_tlbbatch_add_pending(&tlb_ubc->arch, mm, uaddr); From patchwork Thu Feb 8 06:26:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13549360 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01DECC48260 for ; Thu, 8 Feb 2024 06:26:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 359206B009F; Thu, 8 Feb 2024 01:26:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B5A26B00A4; Thu, 8 Feb 2024 01:26:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E04B36B00A1; Thu, 8 Feb 2024 01:26:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C90E86B00A2 for ; Thu, 8 Feb 2024 01:26:30 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 9B417A0915 for ; Thu, 8 Feb 2024 06:26:30 +0000 (UTC) X-FDA: 81767652540.04.E1CE898 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf08.hostedemail.com (Postfix) with ESMTP id A86A4160002 for ; Thu, 8 Feb 2024 06:26:28 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf08.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707373589; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=z8N3K7cEzP29wp8OmV1x4SKhsG69RUNKzzZ3XrSlbEU=; b=qaFKpYg3sbPEtqq/RjsPaCBxNOeDm7UAw5nyLYvpiTMCsPFPso61qMfIgn0lFNCrtavgGj n5Kz2hEdFxPQvTNUtSUPPNpMKV/Qbqfxxxx+SDVS6U6vF2KKtqt99yc9cKzXmVINTcejfw p+r5Se8ofDCw1iG7igKb9iuo9DBd8qc= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf08.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707373589; a=rsa-sha256; cv=none; b=pSweWMMHGLc2+jPsGvkg/z6d3h+2tePlh2AZi+xeKocWblcNgXSpk/37cCEIUL/MJ/I4+v pfTreZAvqZjLd+Z2PNTRRYpY52XwNreiZ9+DWkaHIbcDRhcRUUPnSPQsqlQ2etxZ6dQsWr /QjmegmMhvYcCevrpciHZiZ4KESPbOg= X-AuditID: a67dfc5b-d85ff70000001748-92-65c4740b9853 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v7 8/8] mm: Pause migrc mechanism at high memory pressure Date: Thu, 8 Feb 2024 15:26:08 +0900 Message-Id: <20240208062608.44351-9-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240208062608.44351-1-byungchul@sk.com> References: <20240208062608.44351-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrALMWRmVeSWpSXmKPExsXC9ZZnkS53yZFUg2OLFS3mrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeL6roeMFsd7DzBZ zL/3mc1i86apzBbHp0xltPj9A6jj5KzJLA6CHt9b+1g8ds66y+6xYFOpx+YVWh6L97xk8ti0 qpPNY9OnSewe786dY/c4MeM3i8e8k4Ee7/ddZfPY+svOo3HqNTaPz5vkPN7Nf8sWwB/FZZOS mpNZllqkb5fAldHV9p+tYKpKxe9NVg2M+2W7GDk5JARMJD72/2eFsS/3bmUEsdkE1CVu3PjJ DGKLCJhJHGz9w97FyMXBLPCRSWL19w4WkISwQIDEkdZmsGYWAVWJ241nwJp5BUwlJp6eyQIx VF5i9YYDYIM4gQbt3HYKrF4IqObF7MlsIEMlBJrZJd5f3s0M0SApcXDFDZYJjLwLGBlWMQpl 5pXlJmbmmOhlVOZlVugl5+duYgTGw7LaP9E7GD9dCD7EKMDBqMTDe6L8cKoQa2JZcWXuIUYJ DmYlEV6zHQdShXhTEiurUovy44tKc1KLDzFKc7AoifMafStPERJITyxJzU5NLUgtgskycXBK NTCKF61wktiiEf395IUrt657vQ5Oe3t1QRl3RVXvm5LSHWXhd7d4bpskdTtn4Q/3Oxx2H9IK 1ugW3w85vO+9y2JN+WpuhgUX0xdfqCnesCMyIdth6xYOO4drF+5vF/y5+8DpzeGOs3pUTy5b USd2YuWx+6w2JeqVrr/Fdurrvj//+NvJDVFRa/WtlViKMxINtZiLihMBNpCA8oMCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrOLMWRmVeSWpSXmKPExsXC5WfdrMtdciTV4PUrMYs569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfVdDxkt jvceYLKYf+8zm8XmTVOZLY5Pmcpo8fsHUMfJWZNZHIQ8vrf2sXjsnHWX3WPBplKPzSu0PBbv ecnksWlVJ5vHpk+T2D3enTvH7nFixm8Wj3knAz3e77vK5rH4xQcmj62/7Dwap15j8/i8Sc7j 3fy3bAECUVw2Kak5mWWpRfp2CVwZXW3/2QqmqlT83mTVwLhftouRk0NCwETicu9WRhCbTUBd 4saNn8wgtoiAmcTB1j/sXYxcHMwCH5kkVn/vYAFJCAsESBxpbWYFsVkEVCVuN54Ba+YVMJWY eHomC8RQeYnVGw6ADeIEGrRz2ymweiGgmhezJ7NNYORawMiwilEkM68sNzEzx1SvODujMi+z Qi85P3cTIzC4l9X+mbiD8ctl90OMAhyMSjy8J8oPpwqxJpYVV+YeYpTgYFYS4TXbcSBViDcl sbIqtSg/vqg0J7X4EKM0B4uSOK9XeGqCkEB6YklqdmpqQWoRTJaJg1OqgbFAmtFb6Hr81b3/ kk53HOzf6fH6/K6CgIjdHJf5br+stGBccGvCGnHFB4HnT2p/3bvLg+ugQP7kCa+vnWA4szjB ZU5w6qyXe3dxHNpzd0/B780s1TKnV3Tymu15N/nYvfWzjFtrtz8r3PRXuPtTVVFn2aN7krL2 X1UY1Uumqu8vP6Uhcy15x7SPSizFGYmGWsxFxYkAeC9GyWoCAAA= X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: A86A4160002 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: imc7f6iotcbk866d817pcppgc8f4f3p9 X-HE-Tag: 1707373588-568703 X-HE-Meta: U2FsdGVkX196lBj+sCwnTsOoeXJNuGjeRukrPEBvkJRhITLwUqk0QzvcC505bSF2kad7CCEN1rG8Nl/ow+a9JUyX7NA3ZMCMynoEaIi3fpxYIkEvl3i0sJ8TeDd8nOakQfOyj/CJVN9g5Eqc282RIHRJzSGPcFSL7jNj0bndLtH8NQo155GIZHdl++rZPrmQe7csLdBJofA4dbot9P5Q0fE2G5AXgvHdZWQD51dlX7QidaE6lLcmu9fhx9/HelNeYD9nEmYHzv34//XShF2fAD4+BXBf3bfs799TrLGIoZQsCN9YUrVc9LH5UAwExVHoLYigbRghncQXLcMfS6p7EknQuxEVjkK/cYcqdxtRCKQExKkJofhFIyB1gAQ4QqNuStItE69O+PxkSu3bm7bDCPKWlI/2t377o/9sEUHUzuskYTet4bcneHWDfXiGPGphWsN3A7r7vkT+TPHHBkBxz4cG26bY7mHKenDmf301lfs7gj+Cf4c1+vVOfeKPqyEyzLRwNh5ltwgiXm60HA/R0Nwk9885qmrlQWzLb4y/cF+xQomQrBlAyksUHdZFCe5FIUVJNjEJroYgC6NL1r8czzlUAqdClRbllgQDBHr0l09tjWXb/gyPXceF1NhEoJ5KNO696j6BIdNfxspS3J0IrS+rqmf2Tyf17nGX5U+0yFiL2abA/2NKR+cfw+d5KC4ktbn/Vmr9qfY6UI4w9871moXi0b5IurJa1sWDenYa4F9OlVuC1+MUokr3LBCzlnUQmuMxCF3y5bA0JjbBLxipMz51jm15yI2TEAKtroXE2cb+1FJpoHkfgFIS1sqNK6ZfcsOE1sUM3bVDOYAP6EM6F1gdB39f9p47Six78OIeYwaQ/cx2BG+WDYJOCZRgmBEdobBUCiANgGrguFnpjyNK/hycBEf2s7dkSqieTjpZvPLaj5uYT7MCt/+ZSQ6k3Brm+HDJTof0a6cd666Kdj+ OLg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Regression was observed when the system is in high memory pressure with swap on, where migrc might keep a number of folios in its pending queue, which possibly makes it worse. So temporarily prevented migrc from working on that condition. Signed-off-by: Byungchul Park --- mm/internal.h | 20 ++++++++++++++++++++ mm/migrate.c | 18 +++++++++++++++++- mm/page_alloc.c | 13 +++++++++++++ 3 files changed, 50 insertions(+), 1 deletion(-) diff --git a/mm/internal.h b/mm/internal.h index ab02cb8306e2..55781f879fb2 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1285,6 +1285,8 @@ static inline void shrinker_debugfs_remove(struct dentry *debugfs_entry, #endif /* CONFIG_SHRINKER_DEBUG */ #if defined(CONFIG_MIGRATION) && defined(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) +extern atomic_t migrc_pause_cnt; + /* * Reset the indicator indicating there are no writable mappings at the * beginning of every rmap traverse for unmap. Migrc can work only when @@ -1313,6 +1315,21 @@ static inline bool can_migrc_test(void) return current->can_migrc && current->tlb_ubc_ro.flush_required; } +static inline void migrc_pause(void) +{ + atomic_inc(&migrc_pause_cnt); +} + +static inline void migrc_resume(void) +{ + atomic_dec(&migrc_pause_cnt); +} + +static inline bool migrc_paused(void) +{ + return !!atomic_read(&migrc_pause_cnt); +} + /* * Return the number of folios pending TLB flush that have yet to get * freed in the zone. @@ -1332,6 +1349,9 @@ void migrc_flush_end(struct tlbflush_unmap_batch *batch); static inline void can_migrc_init(void) {} static inline void can_migrc_fail(void) {} static inline bool can_migrc_test(void) { return false; } +static inline void migrc_pause(void) {} +static inline void migrc_resume(void) {} +static inline bool migrc_paused(void) { return false; } static inline int migrc_pending_nr_in_zone(struct zone *z) { return 0; } static inline bool migrc_flush_free_folios(void) { return false; } static inline void migrc_flush_start(void) {} diff --git a/mm/migrate.c b/mm/migrate.c index cbe5372f159e..fbc8586ed735 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -62,6 +62,12 @@ static struct tlbflush_unmap_batch migrc_ubc; static LIST_HEAD(migrc_folios); static DEFINE_SPINLOCK(migrc_lock); +/* + * Increase on entry of handling high memory pressure e.g. direct + * reclaim, decrease on the exit. See __alloc_pages_slowpath(). + */ +atomic_t migrc_pause_cnt = ATOMIC_INIT(0); + static void init_tlb_ubc(struct tlbflush_unmap_batch *ubc) { arch_tlbbatch_clear(&ubc->arch); @@ -1922,7 +1928,8 @@ static int migrate_pages_batch(struct list_head *from, */ init_tlb_ubc(&pending_ubc); do_migrc = IS_ENABLED(CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH) && - (reason == MR_DEMOTION || reason == MR_NUMA_MISPLACED); + (reason == MR_DEMOTION || reason == MR_NUMA_MISPLACED) && + !migrc_paused(); for (pass = 0; pass < nr_pass && retry; pass++) { retry = 0; @@ -1961,6 +1968,15 @@ static int migrate_pages_batch(struct list_head *from, continue; } + /* + * In case that the system is in high memory + * pressure, give up migrc mechanism this turn. + */ + if (unlikely(do_migrc && migrc_paused())) { + fold_ubc(tlb_ubc, &pending_ubc); + do_migrc = false; + } + can_migrc_init(); rc = migrate_folio_unmap(get_new_folio, put_new_folio, private, folio, &dst, mode, reason, diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 6ef0c22b1109..366777afce7f 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4072,6 +4072,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, unsigned int cpuset_mems_cookie; unsigned int zonelist_iter_cookie; int reserve_flags; + bool migrc_paused = false; restart: compaction_retries = 0; @@ -4203,6 +4204,16 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, if (page) goto got_pg; + /* + * The system is in very high memory pressure. Pause migrc from + * expanding its pending queue temporarily. + */ + if (!migrc_paused) { + migrc_pause(); + migrc_paused = true; + migrc_flush_free_folios(); + } + /* Caller is not willing to reclaim, we can't balance anything */ if (!can_direct_reclaim) goto nopage; @@ -4330,6 +4341,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, warn_alloc(gfp_mask, ac->nodemask, "page allocation failure: order:%u", order); got_pg: + if (migrc_paused) + migrc_resume(); return page; }