From patchwork Tue Aug 29 06:06:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 13368616 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83E46C71153 for ; Tue, 29 Aug 2023 06:07:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 11432280035; Tue, 29 Aug 2023 02:07:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0C57A8E001E; Tue, 29 Aug 2023 02:07:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E59D8280035; Tue, 29 Aug 2023 02:07:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D15B88E001E for ; Tue, 29 Aug 2023 02:07:42 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8E56A12042F for ; Tue, 29 Aug 2023 06:07:42 +0000 (UTC) X-FDA: 81176110764.04.5B701C7 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2047.outbound.protection.outlook.com [40.107.243.47]) by imf01.hostedemail.com (Postfix) with ESMTP id A7B3C40020 for ; Tue, 29 Aug 2023 06:07:39 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=I+hzyMqD; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf01.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.243.47 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693289259; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9iOx8gTc3LVF93/VuEpUYYY6Fy2SDNkAdxbQ7DX8kM4=; b=hBPbwNj07SBk17SWVt7K7Hz4YmqOSOiHxLXdZFdeYVfDqGwl1/qcNoZDxR0KqVaDRVhKAJ 93gUxwIRoGVnKaXRNSqddotEmG22lW6l7+/ZkjrlgCXc6diizqQ+mrjf8zf7e+42sp74o3 rzwSQrJ5NYHUz8c4i6+SBmj2nz5JGjA= ARC-Authentication-Results: i=2; imf01.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=I+hzyMqD; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf01.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.243.47 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1693289259; a=rsa-sha256; cv=pass; b=DpXyziKeqBuLD/SUAuJHyYcg5anNJmVgNKbdUk99IDBxR39fZJf8xuFYB4agB5zSQfT/5X 1Ew5vnTNMTqISaoJarmjQ/ItH2Jho1vsfVeX32CwJuLSCbntzJzefsI0U0qEvsmD/rUmya tnKms/J2bffWBdNokZ84DkGsKykzr3s= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BAOiNNLWGyleLNWJJDXS3p2/6HD4eocIA4UNvYWXxveSFHPgpBL/8vrZEfrJvxDqYKUCch8woOM9n9K7feMaamFaiVvL/BL7bRecxtwitWM4e7Xd0NjE3tZ2DzJI79l42JUVbnDKI/Kru3zOwFyCflmRWAN9/d5Ip8FNggSTJcWMA/3/QyABV/n4vmgw9C0j7TZ1usJ4Ep1btnJkt8aW76jp5yQxrHBWzWbqZMUjcEJlJ1QhuL88aVSzYFyngmHMcOvymNNemJFJb2k1rsC9dHv95Tsf2um1gfgxTiwRefIPd+3xMz7RiVF6elQBbaRwSeBrjpsb50D8AxzpPlpT6w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9iOx8gTc3LVF93/VuEpUYYY6Fy2SDNkAdxbQ7DX8kM4=; b=GfYF3E2tmBT4ISMKsuKeAJJ/sJ657zv/M/GElu3e2Z5SSGwhRYHOJYzuzyBbAIve3k6RLGkaf5EelrfUse2dxs0T8srtIaRkJttyhTHVd6VQsCrEKE//rYMPQvMZLCksdD9Idf7i3j8R0pP8Tm4gsIJhdCdnmnQyZCkQjWT5rSUh2vFIRK0+qvZU1DFSrFLiYvT7bLLphPND2Fs/dVvS5VCud3/GQUJtTIUE/+57GYTlvUwymMcJJjxOOKZm/yi9JXSswKHVfvhPA+D8zzdUJRnVJ5qA6+HmRayI+Z84JztjOl8MnFg3tSrLHsOmXYxpHEL3hzBFuMJEr0GxIlXt/Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9iOx8gTc3LVF93/VuEpUYYY6Fy2SDNkAdxbQ7DX8kM4=; b=I+hzyMqDNxeglC+xskhC0Q419Nv07Bsas0yQXEniIauKMLlm5J31N9APHICHpMMJN6AxYCedU4Zf3vKHwUsQHpHC9hAnIiClVF6MrhzYeDc7kHArPrlnDSvQoh8YvocxgGFh568LvRDY2zLEZcTuegm1QCwB0ClL1dGtEXEbM/k= Received: from SA0PR11CA0157.namprd11.prod.outlook.com (2603:10b6:806:1bb::12) by PH8PR12MB7134.namprd12.prod.outlook.com (2603:10b6:510:22d::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.34; Tue, 29 Aug 2023 06:07:37 +0000 Received: from SN1PEPF0002636A.namprd02.prod.outlook.com (2603:10b6:806:1bb:cafe::44) by SA0PR11CA0157.outlook.office365.com (2603:10b6:806:1bb::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.35 via Frontend Transport; Tue, 29 Aug 2023 06:07:37 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF0002636A.mail.protection.outlook.com (10.167.241.135) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.6745.17 via Frontend Transport; Tue, 29 Aug 2023 06:07:36 +0000 Received: from BLR-L-RKODSARA.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Tue, 29 Aug 2023 01:07:31 -0500 From: Raghavendra K T To: , CC: Ingo Molnar , Peter Zijlstra , "Mel Gorman" , Andrew Morton , "David Hildenbrand" , , Juri Lelli , Vincent Guittot , Bharata B Rao , Aithal Srikanth , "kernel test robot" , Raghavendra K T , Sapkal Swapnil , K Prateek Nayak Subject: [RFC PATCH V1 4/6] sched/numa: Increase tasks' access history Date: Tue, 29 Aug 2023 11:36:12 +0530 Message-ID: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002636A:EE_|PH8PR12MB7134:EE_ X-MS-Office365-Filtering-Correlation-Id: 5d40630d-9702-4a44-e3fd-08dba8563e61 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: o26tZ2K2wPyu5XQPs6GIN712exnTqXPHVdz8k+Q06SMahzFmB38S4lZNlOizRryoD2cMWd69QxOJJPzWKMCcVTSLJxAukEc4qs4N9jTGNic50eBPeYxl17R9zphhQZF5iXy50ipzqm078ZTKq83i1Ua/bP244smTXuBH7s3039wUDMvj4+0pp9BUad9s3ZyR4mFnkuOflPk+VQLSFwOi6rEZN/pyymZM56cSyzR6tIVmVZpod97x0QYPPhN3eCZtSM+sbtOZngtdlkDAlKVlaMPofeZNhRkhxMefjdj/u+GwBtLe38BtSCZCdPjHoFz0LoeXQBUp4CYEN1kDbGmshbnlJxUbapPPX+lKvizATeB+fGkSaUUkDlon1b8+/WxD9XykqWZTrfX3FBS1ZC5TYxBT6jnUxsUkKUesyAPn/6R8zAwoIIfC5CqlPsDTeCUKkX/N5IiJF+Ya7RTLjuPC191svbBvmU/h6KfYpIvLSLnRRNKn1a6nDaX2d1pZBJu4F+MYaHO80qVEOHQ8DvKKa9nUM92HPvRmxv0dq51rnrDvQnfiMcGEaEh1NU9f7qwMrUCI3nz1ZoQgLOidy8X0SlKJPBkhVKphFwoY87AU8TgHtF/s5HhqkQdQpkSK2b53tzFQamzJTMInIOPGDupM8J7HEuTWTxMx/SvcfmYYIgLiojiW+66HqN/67mNihShNFE3i1h+v2Ub1iULYP3luQ6pmX+vY/vsiPVCuDUSZyqZotwkabvv11SyHvU272tfVw44TVFQ2R1a0ZxGxMoS7R+R0x2nz0q8aANFiWaoy1CI= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230031)(4636009)(346002)(39860400002)(396003)(376002)(136003)(1800799009)(451199024)(82310400011)(186009)(40470700004)(36840700001)(46966006)(8676002)(8936002)(4326008)(54906003)(110136005)(36756003)(316002)(70206006)(70586007)(5660300002)(2906002)(7416002)(336012)(40460700003)(41300700001)(36860700001)(7696005)(2616005)(426003)(16526019)(26005)(40480700001)(81166007)(82740400003)(356005)(478600001)(83380400001)(6666004)(47076005)(36900700001)(2101003);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Aug 2023 06:07:36.9532 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5d40630d-9702-4a44-e3fd-08dba8563e61 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002636A.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB7134 X-Rspam-User: X-Stat-Signature: yd8msuhioodkni1im8uh6g7aryc61pzb X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: A7B3C40020 X-HE-Tag: 1693289259-136590 X-HE-Meta: U2FsdGVkX1/DbN7MzXqqZMhgzcrCfHgbqUl2mDoHSKfgGvmNBNLsf6+SPScBxUxQ7fKaCxNhQa3+sTBexiUeVLfjEKeUpgV/Ra7kOCrQwsGl76pakreuD/SGmcJKZnh5D95/6zq1MADAxxjVM4Mp5or5QlCGWhRBnebkWfwKS2y+aUqQ42dB8+/moRBfCCCRVM5NX3hTj1V2W1K3qfyuR9te7mrHVn0b+nx2niJd9i0xlqJd7SnQBRPiLURWK3jmO8OXn4FZjhvSc1PW8FdOGX2PT+tpYBCLcqhKYV7JqyrbP9REhSl2OSDD/toWZQ+JrpL65rWEWaF9KH5aLHDCDKdDr3ByMgpeClNMunw+hzLQ3+y9OuyD84PaszpLoB8z3vNUhfxCrvys9hb6NMyE/YoigWbFKeJ9aIT0SlSg1Y2ed8iTb00dQbsc0Mw+QrNXzOnprcEt7NG8lM1vZsVPg7b8uGpKMjgJ2XGURwIn7yyJX7mRJ3e/7B9d+8Oqp+Uaui8l4b0tGIBXmniALAWLiJOqb4s1CJFPHWHqfmgvwjLD7+hZzmYh4ckGNUdYkDO/kFcci1+07PukgrHazTkNtF3LxhbbCvQzSwg7cw6FbdZL+Cc4WY0iCiNkjsf6bSgWiJCtCtmO9JhVanVPFHL22TDUMcZGusa+ZAinuDyb0Q5KZgrc+wC7z0mhAlbmLK9Y46D+zNWBDSZZeYxeXA8w53fskGQAA7NIDYt+UQuh4UGVfCc8qxpiPk53JsQ263hMmsWhYYTz6y8in/rFyIRAifnqwPp8912yZlN9Kdva95nt1ElKDM4T5ih7nDU6FbwBCwdkwkgrbsDmES4uua3w6dsMKagqemYGZ6nrqx7fyNChGF3CcYl9bj1tq6NE7wzlxXB637+KBOlvzgKXQCJPHx1clME2Wsp9gJYeITKvg23nYi1yDN0UXyoTpEVis1TmgPJ6c/PYVmDIDBW6O37 alVINALj RkhGyYaUxiREVWDXDXcaVsxRMJN7iiO3x0vSseAPr2UkYCFACYdt+7z5SQGX9IgRvIX5bBiDK4owN0bvLg2RupAy/DsLL7FJ0efUHlJvugs0fCCfxygL5h4Ud4Y+kikRf/nVvCD1Z5RdZEg0QX/9qnUTVQSUKpYjbsGrFJU+BcWRzri4A1UUkEdd6qWL53a/EmgZVDKfBNU3BvFiCV1cRDXZIyFxxNXCsW99ntiDK6cpjpwdP+fu540m9ZUPQ9HIKrRsZbaPfzuiwD8SfKmeg7ilaYtVV0KUT53D4ku3eeOVbPEY061qjU3m+jqzpjHF/KkSbf7hToEW1OliJqG1LDMdXpHb/5hLEtl1J5sWZ44HCjY4HeeWRCTBxTMH5k8fJgeiaTwDphTqVzbU+CNQhPu8aXO6n15eDamEs239MV29k3cfW1/Xp7ybgZi85AcsYH8svViIbZ5XsEWEdm9qSgVJ3yjO/304xRSn30b2mmoLEPqmilzCxQB4Y4DGiI5WT4T+dRjBu29GKi4hV2V/vJXNffVtXR31c7xRlpsWC/kAYlUetlTQorWVAkA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Peter Zijlstra from two to four. This prepares for optimizations based on tasks' VMA access history. Signed-off-by: Peter Zijlstra Signed-off-by: Raghavendra K T Tested-by: kernel test robot --- include/linux/mm.h | 12 ++++++++---- include/linux/mm_types.h | 4 +++- kernel/sched/fair.c | 29 ++++++++++++++++++++++++----- 3 files changed, 35 insertions(+), 10 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 406ab9ea818f..7794dc91c50f 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1689,10 +1689,14 @@ static inline int xchg_page_access_time(struct page *page, int time) static inline void vma_set_access_pid_bit(struct vm_area_struct *vma) { unsigned int pid_bit; - - pid_bit = hash_32(current->pid, ilog2(BITS_PER_LONG)); - if (vma->numab_state && !test_bit(pid_bit, &vma->numab_state->access_pids[1])) { - __set_bit(pid_bit, &vma->numab_state->access_pids[1]); + unsigned long *pids, pid_idx; + + if (vma->numab_state) { + pid_bit = hash_32(current->pid, ilog2(BITS_PER_LONG)); + pid_idx = READ_ONCE(vma->numab_state->access_pid_idx); + pids = vma->numab_state->access_pids + pid_idx; + if (!test_bit(pid_bit, pids)) + __set_bit(pid_bit, pids); } } #else /* !CONFIG_NUMA_BALANCING */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 647d9fc5da8d..676afa9e497c 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -475,10 +475,12 @@ struct vma_lock { struct rw_semaphore lock; }; +#define NR_ACCESS_PID_HIST 4 struct vma_numab_state { unsigned long next_scan; unsigned long next_pid_reset; - unsigned long access_pids[2]; + unsigned long access_pids[NR_ACCESS_PID_HIST]; + unsigned long access_pid_idx; unsigned long vma_scan_select; }; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index e26e847a8e26..3ae2a1a3ef5c 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -2958,12 +2958,26 @@ static bool task_disjoint_vma_select(struct vm_area_struct *vma) return true; } +static inline bool vma_test_access_pid_history(struct vm_area_struct *vma) +{ + unsigned int i, pid_bit; + unsigned long pids = 0; + + pid_bit = hash_32(current->pid, ilog2(BITS_PER_LONG)); + + for (i = 0; i < NR_ACCESS_PID_HIST; i++) + pids |= vma->numab_state->access_pids[i]; + + return test_bit(pid_bit, &pids); +} + static bool vma_is_accessed(struct vm_area_struct *vma) { - unsigned long pids; + /* Check if the current task had historically accessed VMA. */ + if (vma_test_access_pid_history(vma)) + return true; - pids = vma->numab_state->access_pids[0] | vma->numab_state->access_pids[1]; - return test_bit(hash_32(current->pid, ilog2(BITS_PER_LONG)), &pids); + return false; } #define VMA_PID_RESET_PERIOD (4 * sysctl_numa_balancing_scan_delay) @@ -2983,6 +2997,7 @@ static void task_numa_work(struct callback_head *work) unsigned long nr_pte_updates = 0; long pages, virtpages; struct vma_iterator vmi; + unsigned long pid_idx; SCHED_WARN_ON(p != container_of(work, struct task_struct, numa_work)); @@ -3097,8 +3112,12 @@ static void task_numa_work(struct callback_head *work) time_after(jiffies, vma->numab_state->next_pid_reset)) { vma->numab_state->next_pid_reset = vma->numab_state->next_pid_reset + msecs_to_jiffies(VMA_PID_RESET_PERIOD); - vma->numab_state->access_pids[0] = READ_ONCE(vma->numab_state->access_pids[1]); - vma->numab_state->access_pids[1] = 0; + + pid_idx = vma->numab_state->access_pid_idx; + pid_idx = (pid_idx + 1) % NR_ACCESS_PID_HIST; + + vma->numab_state->access_pid_idx = pid_idx; + vma->numab_state->access_pids[pid_idx] = 0; } /*