From patchwork Fri Nov 1 07:57:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 11222561 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9CD5714DB for ; Fri, 1 Nov 2019 07:58:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 68D09208CB for ; Fri, 1 Nov 2019 07:58:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 68D09208CB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A29D56B0272; Fri, 1 Nov 2019 03:58:47 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9B3366B0274; Fri, 1 Nov 2019 03:58:47 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 82C3C6B0275; Fri, 1 Nov 2019 03:58:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0204.hostedemail.com [216.40.44.204]) by kanga.kvack.org (Postfix) with ESMTP id 58B816B0272 for ; Fri, 1 Nov 2019 03:58:47 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with SMTP id E5EDF181AEF1E for ; Fri, 1 Nov 2019 07:58:46 +0000 (UTC) X-FDA: 76106957052.28.wren75_126259b77c520 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,ying.huang@intel.com,:peterz@infradead.org::linux-kernel@vger.kernel.org:ying.huang@intel.com:akpm@linux-foundation.org:mhocko@suse.com:riel@redhat.com:mgorman@suse.de:mingo@kernel.org:dave.hansen@linux.intel.com:dan.j.williams@intel.com:fengguang.wu@intel.com,RULES_HIT:30003:30034:30054:30055:30064:30070,0,RBL:134.134.136.20:@intel.com:.lbl8.mailshell.net-62.50.0.100 64.95.201.95,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:2,LUA_SUMMARY:none X-HE-Tag: wren75_126259b77c520 X-Filterd-Recvd-Size: 7481 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by imf24.hostedemail.com (Postfix) with ESMTP for ; Fri, 1 Nov 2019 07:58:46 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 01 Nov 2019 00:58:45 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.68,254,1569308400"; d="scan'208";a="225962518" Received: from yhuang-dev.sh.intel.com ([10.239.159.29]) by fmsmga004.fm.intel.com with ESMTP; 01 Nov 2019 00:58:43 -0700 From: "Huang, Ying" To: Peter Zijlstra Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Andrew Morton , Michal Hocko , Rik van Riel , Mel Gorman , Ingo Molnar , Dave Hansen , Dan Williams , Fengguang Wu Subject: [RFC 10/10] autonuma, memory tiering: Adjust hot threshold automatically Date: Fri, 1 Nov 2019 15:57:27 +0800 Message-Id: <20191101075727.26683-11-ying.huang@intel.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191101075727.26683-1-ying.huang@intel.com> References: <20191101075727.26683-1-ying.huang@intel.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Huang Ying It isn't easy for the administrator to determine the hot threshold. So in this patch, a method to adjust the hot threshold automatically is implemented. The basic idea is to control the number of the candidate promotion pages to match the promotion rate limit. If the hint page fault latency of a page is less than the hot threshold, we will try to promote the page, that is, the page is the candidate promotion page. If the number of the candidate promotion pages in the statistics interval is much higher than the promotion rate limit, the hot threshold will be lowered to reduce the number of the candidate promotion pages. Otherwise, the hot threshold will be raised to increase the number of the candidate promotion pages. To make the above method works, in each statistics interval, the total number of the pages to check (on which the hint page faults occur) and the hot/cold distribution need to be stable. Because the page tables are scanned linearly in autonuma, but the hot/cold distribution isn't uniform along the address. The statistics interval should be larger than the autonuma scan period. So in the patch, the max scan period is used as statistics interval and it works well in our tests. The sysctl knob kernel.numa_balancing_hot_threshold_ms becomes the initial value and max value of the hot threshold. The patch improves the score of pmbench memory accessing benchmark with 80:20 read/write ratio and normal access address distribution by 5.5% with 24.6% fewer NUMA page migrations on a 2 socket Intel server with Optance DC Persistent Memory. Because it improves the accuracy of the hot page selection. Signed-off-by: "Huang, Ying" Cc: Andrew Morton Cc: Michal Hocko Cc: Rik van Riel Cc: Mel Gorman Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Dave Hansen Cc: Dan Williams Cc: Fengguang Wu Cc: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org --- include/linux/mmzone.h | 3 +++ kernel/sched/fair.c | 48 ++++++++++++++++++++++++++++++++++++++---- 2 files changed, 47 insertions(+), 4 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 46382b058546..afd56541252c 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -772,6 +772,9 @@ typedef struct pglist_data { #ifdef CONFIG_NUMA_BALANCING unsigned long autonuma_jiffies; unsigned long autonuma_try_migrate; + unsigned long autonuma_threshold_jiffies; + unsigned long autonuma_threshold_try_migrate; + unsigned long autonuma_threshold; #endif /* Fields commonly accessed by the page reclaim scanner */ struct lruvec lruvec; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 0a83e9cf6685..22bdbb7afac2 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1486,6 +1486,41 @@ static bool numa_migration_check_rate_limit(struct pglist_data *pgdat, return true; } +#define NUMA_MIGRATION_ADJUST_STEPS 16 + +static void numa_migration_adjust_threshold(struct pglist_data *pgdat, + unsigned long rate_limit, + unsigned long ref_threshold) +{ + unsigned long now = jiffies, last_threshold_jiffies; + unsigned long unit_threshold, threshold; + unsigned long try_migrate, ref_try_migrate, mdiff; + + last_threshold_jiffies = pgdat->autonuma_threshold_jiffies; + if (now > last_threshold_jiffies + + msecs_to_jiffies(sysctl_numa_balancing_scan_period_max) && + cmpxchg(&pgdat->autonuma_threshold_jiffies, + last_threshold_jiffies, now) == last_threshold_jiffies) { + + ref_try_migrate = rate_limit * + sysctl_numa_balancing_scan_period_max / 1000; + try_migrate = node_page_state(pgdat, NUMA_TRY_MIGRATE); + mdiff = try_migrate - pgdat->autonuma_threshold_try_migrate; + unit_threshold = ref_threshold / NUMA_MIGRATION_ADJUST_STEPS; + threshold = pgdat->autonuma_threshold; + if (!threshold) + threshold = ref_threshold; + if (mdiff > ref_try_migrate * 11 / 10) + threshold = max(threshold - unit_threshold, + unit_threshold); + else if (mdiff < ref_try_migrate * 9 / 10) + threshold = min(threshold + unit_threshold, + ref_threshold); + pgdat->autonuma_threshold_try_migrate = try_migrate; + pgdat->autonuma_threshold = threshold; + } +} + bool should_numa_migrate_memory(struct task_struct *p, struct page * page, int src_nid, int dst_cpu, unsigned long addr, int flags) @@ -1501,7 +1536,7 @@ bool should_numa_migrate_memory(struct task_struct *p, struct page * page, if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && next_promotion_node(src_nid) != -1) { struct pglist_data *pgdat; - unsigned long rate_limit, latency, threshold; + unsigned long rate_limit, latency, threshold, def_threshold; pgdat = NODE_DATA(dst_nid); if (pgdat_free_space_enough(pgdat)) @@ -1511,16 +1546,21 @@ bool should_numa_migrate_memory(struct task_struct *p, struct page * page, if (!(flags & TNF_YOUNG)) return false; - threshold = msecs_to_jiffies( + def_threshold = msecs_to_jiffies( sysctl_numa_balancing_hot_threshold); + rate_limit = sysctl_numa_balancing_rate_limit << + (20 - PAGE_SHIFT); + numa_migration_adjust_threshold(pgdat, rate_limit, + def_threshold); + + threshold = pgdat->autonuma_threshold; + threshold = threshold ? : def_threshold; if (flags & TNF_WRITE) threshold *= 2; latency = numa_hint_fault_latency(p, addr); if (latency > threshold) return false; - rate_limit = sysctl_numa_balancing_rate_limit << - (20 - PAGE_SHIFT); return numa_migration_check_rate_limit(pgdat, rate_limit, hpage_nr_pages(page)); }