From patchwork Fri Jan 3 17:24:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13925730 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C054E77188 for ; Fri, 3 Jan 2025 17:26:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E7ED86B008A; Fri, 3 Jan 2025 12:26:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E2F896B008C; Fri, 3 Jan 2025 12:26:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA94C6B0092; Fri, 3 Jan 2025 12:26:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id AB5EC6B008A for ; Fri, 3 Jan 2025 12:26:07 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 291CE4418C for ; Fri, 3 Jan 2025 17:26:07 +0000 (UTC) X-FDA: 82966817724.16.2188E8C Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam02on2055.outbound.protection.outlook.com [40.107.212.55]) by imf24.hostedemail.com (Postfix) with ESMTP id 56951180014 for ; Fri, 3 Jan 2025 17:25:57 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=DzjP4foA; spf=pass (imf24.hostedemail.com: domain of ziy@nvidia.com designates 40.107.212.55 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1735925115; a=rsa-sha256; cv=pass; b=qTE8ZiRIPXhw/9uS9xrXkpoC8yLbdJXjUkRXpwG27SkTnXl8Ia9FiT14GpOFxgZKlAWREI cJRO8JEYl91orU/O6nJOt0AFHZ5ljMgQWFjYcwNOFXLYIhJvuiQBqZLOXse8KZH0zyL1A7 nQNS9K7mG8aDlrzm5c0r/eSPCNFnJd0= ARC-Authentication-Results: i=2; imf24.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=DzjP4foA; spf=pass (imf24.hostedemail.com: domain of ziy@nvidia.com designates 40.107.212.55 as permitted sender) smtp.mailfrom=ziy@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735925115; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vGh53MmAsD9e3UWlUB2RQAs903jeDpCr5tqEtYEh+TY=; b=K1w20PHmRr6rAZ/3OliVw7j7zeYYDQKjvsPBo//t9tlRf0c/bVZxAWXDo2TJ7nCPtljIEw st2OH6tHGyEATmydPz+XH6Mm2ivZH5sS0DW7LK7bhXb6/ucGIuIoQF/6X4HCAnCCb9ZGa6 PPUXmCICEMAiXRj3Gg6YGBul2021rFo= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=fo2eRtsUAzYhjQpNrQPewosTszzUg0xUssm3q5hnf+8wOgG75NDOqzVGsUy/U13TeQyF8cN+RIws1vQOOc6CU+fF8sopSnbjGQQAMVUtOxE5P2+jY4cja/KYOAAvQBanldrjsXnhIvxBSWMYMMuIULfAmPrM5omi1qgms8IEOWD6iE9U1+Y2ebNhkbu+YdybbxJMvDgZtUv9quNs9kNknqeV4hWTA+qJHrNUmdP0LEZSweVz5POnGY1+MIJsCk/Mgk/va+msSkceWMWQwKV5YqUdW3Pt7fDnRixBIn9V2WFFs7PlKsUmKjd8J/D0EjVnryS1/ObVr1uIpXwLSeqkTw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vGh53MmAsD9e3UWlUB2RQAs903jeDpCr5tqEtYEh+TY=; b=FQr+ZA6mgMOK/eHPBbijpb3+yfo04P+5nO+qTEHFh+ENLcbnKFV647/m3lc6EYEq7pvsNvjZnWaLRLqX3ob4mA0v/WJ/ZLX7GAdePS8ZoqpeBqxUMMbNprqk5i3V121I+GSMkMmzEqhOvXlJUUA4J0pTBqsNzD9aCLoOBUZqulx80dd4PIu/a2bZlNyqY7GsgbqX9YMTeq2i99xRKZ0GHuFwt+jsx/Z+TL2xaY0WJkiPzwrz18TNaXZ0WcF6O+JjJNesdIJGXh9cINVvXg35+wekPCUZ9eKV7gAQexKw4+9kC2kMsfpTHh1WxXvV8rscPZCSw3DUBQYzb01ItQHp9g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vGh53MmAsD9e3UWlUB2RQAs903jeDpCr5tqEtYEh+TY=; b=DzjP4foAOhmkux48pPlpOXsCIVcbjWE5XW7m4ZkqlDY+Pkc1c7HnNH1B9M7KSuKcWeM1LmUP/hW5VJKrELCCo7nFn8IcbPX6OhrkTyUFi79Ck6nCNglnN+LFWwC//KMEEFhI0u+G0n7M7E+/4fw0UWb+XrkurZ7LJ1/weeCM7DCknZjeT6Nxd77rJhi/RTlDcKbfU/OlI6xLwXaRDISJldEgc0fTAUfZpTabBSU+RAxTnutH9HEdeDj7xW+EbIgWX2SxUXea80HGw80bgoARC5f43eQXRGCNGzTpcYqXXTUnkUn+sAlZxQnt5UbT6s+eVlH5LDgO68+tX3us2uTcRg== Received: from DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) by CY8PR12MB8066.namprd12.prod.outlook.com (2603:10b6:930:70::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8314.11; Fri, 3 Jan 2025 17:24:37 +0000 Received: from DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a]) by DS7PR12MB9473.namprd12.prod.outlook.com ([fe80::5189:ecec:d84a:133a%3]) with mapi id 15.20.8314.013; Fri, 3 Jan 2025 17:24:37 +0000 From: Zi Yan To: linux-mm@kvack.org Cc: David Rientjes , Shivank Garg , Aneesh Kumar , David Hildenbrand , John Hubbard , Kirill Shutemov , Matthew Wilcox , Mel Gorman , "Rao, Bharata Bhasker" , Rik van Riel , RaghavendraKT , Wei Xu , Suyeon Lee , Lei Chen , "Shukla, Santosh" , "Grimm, Jon" , sj@kernel.org, shy828301@gmail.com, Liam Howlett , Gregory Price , "Huang, Ying" , Zi Yan Subject: [RFC PATCH 5/5] test: add sysctl for folio copy tests and adjust NR_MAX_BATCHED_MIGRATION Date: Fri, 3 Jan 2025 12:24:19 -0500 Message-ID: <20250103172419.4148674-6-ziy@nvidia.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20250103172419.4148674-1-ziy@nvidia.com> References: <20250103172419.4148674-1-ziy@nvidia.com> X-ClientProxiedBy: BL1PR13CA0187.namprd13.prod.outlook.com (2603:10b6:208:2be::12) To DS7PR12MB9473.namprd12.prod.outlook.com (2603:10b6:8:252::5) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS7PR12MB9473:EE_|CY8PR12MB8066:EE_ X-MS-Office365-Filtering-Correlation-Id: aa4d26f3-5c87-47e1-37b2-08dd2c1b7fb0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: skQen7OtqsoNThIXNUrzMgLpqxB9dVfdItRrK9FXt/TS+IcibzEzZShLL4dnxGzZtYk+zpWEjlJyARkFysuLYmGOITq8KnD31kA3yQQwJ7kDdoZdNZSwEaAsQJD/DQiSz3Vga+4XRrKhwcSo6KY9xlH6GSMPvzHtkbRaofWDy96TzkztdID9fIQ92Ayy2hzZj8HZ08ywGarC/4Da13sI7pHf89fyZHZtObuULIQdCF1aNnPBAvZKRmImLaTUI402RBy92uJSZKPxWXF1e73V+I3kH8mTqv8AQ5SNMWxdQtqOCJTzsNLCocYhXwDo/Kicy+ds1B9myCuxhz9c6F8NI+/ISC+KbBecaROmni6KMpNN36+LExKHKginwuMhI4fnBgvnuKAs09IDLa+FBq1rGFNbmvKxvJm42GdABpEhPHWVpyhZ3zbeSJUHIz/1YpifWHo9PuQaNAbuCz7rbclh3iHw/WRU1Xql8AuNgh16qs6pA63dnDb+6pywnx332dPErKeRJtCX7oTZe+O9OqTA0iQcURKYfoPt9o2x8KHUxLlrng9p+RqOQJMUyQS1H8SBvaVsH2ayMpdtSxFsFzmMFW27SIRWKn+SNaBFQ/OfaOQWIftSZgdFSxfwZxdH1kGvuvQFqRAnB1dufmh/f3DvrRqWpFQ962HYXY74C0hsYVwijD+su3/km7MujlqBQwKubevt9GFqoQKjdrpgD0h2yTnJ/07qD6QKlsDYFEcIDy0rYlhBqa/nmzAluWhKKNomfZujICqYRd9GcbXshhl7R9IaKZtDJyWcb+ZsETcFG7X1JOvf0g+5/ryjNqtL9farfiCDCTsV59Dxq838jzH8sZtpKnW+OToynnX/3kh/rKS9ataDY8FWDHxjnh/oRavaON0y/P9G5PEKT2ELYZm69uA/9t0lO0ipWyuc1cpaOHGlKn62Z5J19u3OphWDGaRCn45guReJ4mgfUCcQaC+ugMxCW8T4NUFokTFkKA1E86PmxTlJQf5TZmlVPRvn4z3aVdRNPoxahD1mt2da7RxaoIs+HW08XUrztmdzauXJJBL1DGIJFIc/FgaNnrsNsQtbz7LQAzP8Su2jYRT3JsnzAS3DEVhs7yr3xrrYUHWDhq7h9nh2CMVZiuauLhTm8fd/qbgWqBV9zkiEHraAHNq7zYIw1mOp7q6YCdT3oqud1lLmUeDV0BX8E4IUD60OwNoP6qTJR/wr0AK+1/hK6vOCTf1SSTm8+4liuC7BLrX16rp7P8E6wJHEam+LS/bGUIJlNPMlLW52vYAInTRtfTPIiAEi9x0mTYk5DOetvOAY7FBGuWBTJPyPtYLSmhUsh3gu0OmP5L6yqEWVsFwPnjk428AewblVsnASwpi8nK7SgHyeMer0sIaGsy2mrO0RmjIz X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS7PR12MB9473.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: +25rtkqRGgj0/l9/eIrgJRvdGKpIWukW7IGp2jYCJWP29x80IbpoEDD9ogj048J8q93XcmOwOJCyFTPvY7s3+bk/5oidz3224tl5OeqJDxhS32nqXUXo6rbpQY2I3wCBOvzPuKLimWkZ2hYe/o2QYCG1i84Kh2mCjxuY+KLCwHoZlCS/1gLCi7ph96nkHfq8mN4ITGG38bmeIch2mjJqRRZRolDPiX9eG8jJcw46dPZXYMjxyBXGVME99vuwCXzZpVPaaUBgZn2Blfw6eU1palq5nEY1jJQX/FEbaIsLM5uO59d/tX3eTJEoVsf6IcH1i00sic5rwiWvXBen2EBjhTeQGblMvgt67E39sal7Al7jpLZ7FN0ncDWoS96Z8OdPq1Gn8NtOFdlQTrH7AyUMxRy3Tfph2Y+QPKxY02uFFnzZxo4mOqP9MCWQh+D9+bKN95s5ryZjKAtu+7Im4kyTau4+7fr2V9sqFCcNYsjImuu7VMyo1BAkXrbQ3xznOrvRjdoUEUYWff3q9syiUzm4T+Wn67Tw8PaWDvq0WociKfhWSCw6X6ZI1N9o2A4KYmft3uol2M/GWg92efqb1jcAmyafpUrnNcu9a5s5QfDDMaJCOS1wnOhz/IVPau4eeuDRYic5CiNDbfZEkYiQxfsMmn4PYqv9Pd1Z1iAcYLe/a771okrs/CWFi+ZcNLovmo6XmJB1ijCoqXLrpYNgGyzX6uIdLzDUiF7J1TM1sfivXHt/RsdodkJ58rScb0tdYCH4OvIT1rT3KSZEjOqqRTXx8pJivPuPJ3hu4/pFAjIU4RqpKqmm1jDZHAURqQdrAL731PDqbZAxcbYRQh0uEPllMwVaIWwcGGb8zH/eT10StHVhqI6HGIv6f31qYaH3CeZm2FRluyXcyP+cMnvtIgw548O/GUQtR8CFs3QAmSDYKvuUTYQ3X+RZLNkNz09vFKl6gTvi2bAcjcNgTmitfR54pMVj1TusqLI7it8RWPZVWlpID0k1MJB+7xfVQ4TWBmaM0CzpOTSyD2Jcpej7v8QUIzhB2VJcH2JDfGY4qY02BnAxTc/XDJfB0Q8dNXs12In5fI9oQasNqDWfJGOb2TYmfX9fDJBP7zs97v39aGcMUerG9ZxqiGGnPuiLQKtscw/Pq83bnHm9BuXRTZJnuOsi+LCP0izoszL8Hg/PC+1fK3JBPNIb6+/ayLIwXHQ6KGYpXo0SFoyGd6B7IRtt44wwaH6gf+Qd4fAsyVQUqMi0Wkze/Ulouq+fGrui4OL86te6hp8lhmUKKHCBvJtedAFGs0V9dNAPGBmqwJSn/r9MZCqh+BUbTTLIaQ+96f+c7R/Q1EFmHuG2mHJ8fMTofoDgHWpSWgCIHrk9VoXowVcpJMw80uf++8HiYwKHzqF9D49ifBaHLuKHVwI6jWguH7AT0iGn4wvIzsZ22lywdVibwMZbkdF8wuzdD8/emLAuKvfpDD5WADwfSVK0euE6+WE9wy0Ane1b/xdYhTcOk4a4Re0dvkU8/pg9eChupb89pr1RR68+Yd6Siym5K5hAHQcT2KXpfXWiLe0I4d/Ht0ujt8g= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: aa4d26f3-5c87-47e1-37b2-08dd2c1b7fb0 X-MS-Exchange-CrossTenant-AuthSource: DS7PR12MB9473.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jan 2025 17:24:37.5846 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: mIhyGzW5QmDJWHBBA/BoHCfJEl9X3jAbzMlJsjEQwbE9l5eif5BPew19Ibs0pY3v X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8066 X-Rspamd-Queue-Id: 56951180014 X-Stat-Signature: j7gmmxyijt4pw3ce9y4ij83sxxi3ayno X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1735925157-850574 X-HE-Meta: U2FsdGVkX193bZe2vDyR7Z720GK13Ja+11Jw5TwFlL9hbJirq4FO8dSiToMmvWPe4KsxKuP3SF8mDmylzvqIDgMnueigYphsvjML2beI6w8bk1yirh+ISElAKWbwWuCHfMf8+Ff3AECj0LXBMAL7cJ4dOaF3kIeSRMMCBHgdetHYb3cSD+r4V/jTekoNftLFHiTfofMH4AEardDCQtvGLQvDjDIKpj0OejMlFDK1pw2UmWxFAAqtviD89IE8p0gmP0SQq/s9te46y4aLCil8uwkaa6gHFASvSYug/WY9qg+WU8OrJqXF+7E5rDIwcGzfS5SGPOa7cK3YGXeqSZnKheEBiW7s2OrcTt4RkKzgSP5HEJMDQnb+pRR3GfDXwY2jkwSSxbSHAPN/qZXNXpsSpnc3Vk16ienTRjnE879vRTcyZdj/wPonngBPoTe9gTfRtAhYU+MnvkT708rdOQwQPG9b6A0jSdb9ON3CiDdXEecLHR0oiUeGjfhNnXQQN8HnGwUaOCzZjwq9CrTWsidCYJLFG7claqnrCP6U/XReUObbaVcCDRPmdYpkJwlje5r4YpJPFYzLuPBbX3y8HQJGitCesJx9H6J+mGApLpcrUt7dqrUNYRymqx9GBlCvF0BBeRrgeUKz6QGiufzTbwdDSPOyIx5uE257uxLZ4MiTlAn+Hr9cnOsPmJZm3qCqNU3BtlD1zMMgEU3ZeQYP+0LhhExnF88lLOa386HpJDcmT1FC63DeVh3eAoW+vIFQdsWbN68BKtdgRrgpxRD/wPsjIpZQNpgnl8FGlEpapmWj+j47h/bUavywSGFgWvkKmxtv+UDD5BwVY8t0Z2lJcTbVtIhRaKDK8XDUy7CXo4R7A8hCGccrg0Yh2FQuPmGy/Wn7uViOnDNAlFv2iQ56SB3Xk8QZdbf0+PXDn3MBjWZuTeC4qWN3/MvsMI1rbiaycu6ovQbVNU4uv5kTxdRs5RT gQoopC82 2PXz3yAIj4wVDs7f/kqgqRf537UVBF5C5g11MpKtG8ha3PE/O8ZRVgNmDYHtY/zSwhIWiQQmGac9OS81U/4YBeP5SMkxje9QmBYde+38bA2Vor15tgtxEQ+EXLyPLEplAB4ZP7VcHMEEdtJ3pMKOiqhUW0Qi/fFDRWjs6LkfghOBBWrd1vfqLD1Gmk5SIPd59qBMhGFL8eqgziOtagH10oegOrHyMp75adPTMY0d6Fxww+NxPliwx1DB5HZ/gIUWxYjv3SVI4sC6NNgLsMZMwo52FhAL7T7sVvw7bKto1IxODLvlvMaegY3plMQ/s30BJUO/vPxv/TFV6OHqxRKou9WZWeuIPvGZUFHVZy8rI6YUF6EI+dDNlSmJexN5xorJrq/ZEwjStawL5ze7gghvwZrMLIEdlgae+OOPMKD87eI1mSdJUrBChm2VYHz/eKonw2hXLqNTN8vMRmixZLwAk6v+IqQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: 1. enable multi-threaded copy 2. specify how many CPU threads to use 3. push from local CPUs or pull from remote CPUs 4. change NR_MAX_BATCHED_MIGRATION to HPAGE_PUD_NR to allow batching THP copies. These are for testing purpose only. Signed-off-by: Zi Yan --- include/linux/mm.h | 4 ++++ include/linux/sysctl.h | 1 + kernel/sysctl.c | 29 ++++++++++++++++++++++++++++- mm/copy_pages.c | 10 +++++++--- mm/migrate.c | 6 ++++-- 5 files changed, 44 insertions(+), 6 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 1a11f9df5c2d..277b12b9ef0d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -42,6 +42,10 @@ struct pt_regs; struct folio_batch; extern int sysctl_page_lock_unfairness; +extern int sysctl_use_mt_copy; +extern unsigned int sysctl_limit_mt_num; +extern unsigned int sysctl_push_0_pull_1; + void mm_core_init(void); void init_mm_internals(void); diff --git a/include/linux/sysctl.h b/include/linux/sysctl.h index 40a6ac6c9713..f33dafea2533 100644 --- a/include/linux/sysctl.h +++ b/include/linux/sysctl.h @@ -52,6 +52,7 @@ struct ctl_dir; /* this is needed for the proc_dointvec_minmax for [fs_]overflow UID and GID */ #define SYSCTL_MAXOLDUID ((void *)&sysctl_vals[10]) #define SYSCTL_NEG_ONE ((void *)&sysctl_vals[11]) +#define SYSCTL_32 ((void *)&sysctl_vals[12]) extern const int sysctl_vals[]; diff --git a/kernel/sysctl.c b/kernel/sysctl.c index 5c9202cb8f59..f9ba48cd6e09 100644 --- a/kernel/sysctl.c +++ b/kernel/sysctl.c @@ -82,7 +82,7 @@ #endif /* shared constants to be used in various sysctls */ -const int sysctl_vals[] = { 0, 1, 2, 3, 4, 100, 200, 1000, 3000, INT_MAX, 65535, -1 }; +const int sysctl_vals[] = { 0, 1, 2, 3, 4, 100, 200, 1000, 3000, INT_MAX, 65535, -1, 32 }; EXPORT_SYMBOL(sysctl_vals); const unsigned long sysctl_long_vals[] = { 0, 1, LONG_MAX }; @@ -2091,6 +2091,33 @@ static struct ctl_table vm_table[] = { .extra2 = SYSCTL_ONE, }, #endif + { + .procname = "use_mt_copy", + .data = &use_mt_copy, + .maxlen = sizeof(use_mt_copy), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = SYSCTL_ONE, + }, + { + .procname = "limit_mt_num", + .data = &limit_mt_num, + .maxlen = sizeof(limit_mt_num), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ONE, + .extra2 = SYSCTL_32, + }, + { + .procname = "push_0_pull_1", + .data = &push_0_pull_1, + .maxlen = sizeof(push_0_pull_1), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = SYSCTL_ONE, + }, { .procname = "drop_caches", .data = &sysctl_drop_caches, diff --git a/mm/copy_pages.c b/mm/copy_pages.c index 0e2231199f66..257034550c86 100644 --- a/mm/copy_pages.c +++ b/mm/copy_pages.c @@ -10,7 +10,9 @@ #include -unsigned int limit_mt_num = 4; +unsigned int sysctl_limit_mt_num = 4; +/* push by default */ +unsigned int sysctl_push_0_pull_1; struct copy_item { char *to; @@ -45,11 +47,13 @@ int copy_page_lists_mt(struct list_head *dst_folios, struct list_head *src_folios, int nr_items) { int err = 0; - unsigned int total_mt_num = limit_mt_num; + unsigned int total_mt_num = sysctl_limit_mt_num; int to_node = folio_nid(list_first_entry(dst_folios, struct folio, lru)); + int from_node = folio_nid(list_first_entry(src_folios, struct folio, lru)); int i; struct copy_page_info *work_items[32] = {0}; - const struct cpumask *per_node_cpumask = cpumask_of_node(to_node); + const struct cpumask *per_node_cpumask = + cpumask_of_node(sysctl_push_0_pull_1 ? to_node : from_node); int cpu_id_list[32] = {0}; int cpu; int max_items_per_thread; diff --git a/mm/migrate.c b/mm/migrate.c index 18440180d747..0f7a4b09acda 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -51,6 +51,7 @@ #include "internal.h" +int sysctl_use_mt_copy; bool isolate_movable_page(struct page *page, isolate_mode_t mode) { @@ -1621,7 +1622,7 @@ static inline int try_split_folio(struct folio *folio, struct list_head *split_f } #ifdef CONFIG_TRANSPARENT_HUGEPAGE -#define NR_MAX_BATCHED_MIGRATION HPAGE_PMD_NR +#define NR_MAX_BATCHED_MIGRATION HPAGE_PUD_NR #else #define NR_MAX_BATCHED_MIGRATION 512 #endif @@ -1868,7 +1869,8 @@ static void migrate_folios_batch_move(struct list_head *src_folios, goto out; /* Batch copy the folios */ - if (total_nr_pages > 32) { + /* if (total_nr_pages > 32) { */ + if (sysctl_use_mt_copy) { copy_page_lists_mt(dst_folios, src_folios, total_nr_folios); } else { dst = list_first_entry(dst_folios, struct folio, lru);