From patchwork Fri Sep 24 16:12:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Hansen X-Patchwork-Id: 12515895 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83626C433FE for ; Fri, 24 Sep 2021 16:13:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0F6196124C for ; Fri, 24 Sep 2021 16:13:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 0F6196124C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id EF17B6B0075; Fri, 24 Sep 2021 12:12:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E578E6B0072; Fri, 24 Sep 2021 12:12:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D6C70900002; Fri, 24 Sep 2021 12:12:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0251.hostedemail.com [216.40.44.251]) by kanga.kvack.org (Postfix) with ESMTP id C62906B0072 for ; Fri, 24 Sep 2021 12:12:58 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 7E7828249980 for ; Fri, 24 Sep 2021 16:12:58 +0000 (UTC) X-FDA: 78622960836.06.422468F Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf14.hostedemail.com (Postfix) with ESMTP id DD1826001E6E for ; Fri, 24 Sep 2021 16:12:57 +0000 (UTC) X-IronPort-AV: E=McAfee;i="6200,9189,10116"; a="203601573" X-IronPort-AV: E=Sophos;i="5.85,320,1624345200"; d="scan'208";a="203601573" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Sep 2021 09:12:55 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,320,1624345200"; d="scan'208";a="551865372" Received: from davehans-spike.ostc.intel.com (HELO localhost.localdomain) ([10.165.28.105]) by FMSMGA003.fm.intel.com with ESMTP; 24 Sep 2021 09:12:53 -0700 Subject: [PATCH 1/2] mm/migrate: optimize hotplug-time demotion order updates To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org,Dave Hansen ,oliver.sang@intel.com,ying.huang@intel.com,mhocko@suse.com,weixugc@google.com,osalvador@suse.de,rientjes@google.com,dan.j.williams@intel.com,david@redhat.com,gthelen@google.com,yang.shi@linux.alibaba.com,akpm@linux-foundation.org From: Dave Hansen Date: Fri, 24 Sep 2021 09:12:53 -0700 References: <20210924161251.093CCD06@davehans-spike.ostc.intel.com> In-Reply-To: <20210924161251.093CCD06@davehans-spike.ostc.intel.com> Message-Id: <20210924161253.D7673E31@davehans-spike.ostc.intel.com> Authentication-Results: imf14.hostedemail.com; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=intel.com (policy=none); spf=none (imf14.hostedemail.com: domain of dave.hansen@linux.intel.com has no SPF policy when checking 192.55.52.136) smtp.mailfrom=dave.hansen@linux.intel.com X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: DD1826001E6E X-Stat-Signature: 6np1kbqmyyhyn8dagb7417h8qe5r43io X-HE-Tag: 1632499977-496803 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Dave Hansen == tl;dr == Automatic demotion opted for a simple, lazy approach to handling hotplug events. This noticeably slows down memory hotplug[1]. Optimize away updates to the demotion order when memory hotplug events should have no effect. This has no effect on CPU hotplug. There is no known problem on the CPU side and any work there will be in a separate series. == Background == Automatic demotion is a memory migration strategy to ensure that new allocations have room in faster memory tiers on tiered memory systems. The kernel maintains an array (node_demotion[]) to drive these migrations. The node_demotion[] path is calculated by starting at nodes with CPUs and then "walking" to nodes with memory. Only hotplug events which online or offline a node with memory (N_ONLINE) or CPUs (N_CPU) will actually affect the migration order. == Problem == However, the current code is lazy. It completely regenerates the migration order on *any* CPU or memory hotplug event. The logic was that these events are extremely rare and that the overhead from indiscriminate order regeneration is minimal. Part of the update logic involves a synchronize_rcu(), which is a pretty big hammer. Its overhead was large enough to be detected by some 0day tests that watch memory hotplug performance[1]. == Solution == Add a new helper (node_demotion_topo_changed()) which can differentiate between superfluous and impactful hotplug events. Skip the expensive update operation for superfluous events. == Aside: Locking == It took me a few moments to declare the locking to be safe enough for node_demotion_topo_changed() to work. It all hinges on the memory hotplug lock: During memory hotplug events, 'mem_hotplug_lock' is held for write. This ensures that two memory hotplug events can not be called simultaneously. CPU hotplug has a similar lock (cpuhp_state_mutex) which also provides mutual exclusion between CPU hotplug events. In addition, the demotion code acquire and hold the mem_hotplug_lock for read during its CPU hotplug handlers. This provides mutual exclusion between the demotion memory hotplug callbacks and the CPU hotplug callbacks. This effectively allows treating the migration target generation code to act as if it is single-threaded. 1. https://lore.kernel.org/all/20210905135932.GE15026@xsang-OptiPlex-9020/ Fixes: 884a6e5d1f93 ("mm/migrate: update node demotion order on hotplug events") Reported-by: kernel test robot Signed-off-by: Dave Hansen Cc: "Huang, Ying" Cc: Michal Hocko Cc: Wei Xu Cc: Oscar Salvador Cc: David Rientjes Cc: Dan Williams Cc: David Hildenbrand Cc: Greg Thelen Cc: Yang Shi Cc: Andrew Morton Reviewed-by: David Hildenbrand --- b/mm/migrate.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff -puN mm/migrate.c~faster-node-order mm/migrate.c --- a/mm/migrate.c~faster-node-order 2021-09-24 09:12:30.988377798 -0700 +++ b/mm/migrate.c 2021-09-24 09:12:30.988377798 -0700 @@ -3239,8 +3239,18 @@ static int migration_offline_cpu(unsigne * set_migration_target_nodes(). */ static int __meminit migrate_on_reclaim_callback(struct notifier_block *self, - unsigned long action, void *arg) + unsigned long action, void *_arg) { + struct memory_notify *arg = _arg; + + /* + * Only update the node migration order when a node is + * changing status, like online->offline. This avoids + * the overhead of synchronize_rcu() in most cases. + */ + if (arg->status_change_nid < 0) + return notifier_from_errno(0); + switch (action) { case MEM_GOING_OFFLINE: /* From patchwork Fri Sep 24 16:12:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Hansen X-Patchwork-Id: 12515897 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 58E07C433EF for ; Fri, 24 Sep 2021 16:13:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F320561212 for ; Fri, 24 Sep 2021 16:13:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org F320561212 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 1CD476B0072; Fri, 24 Sep 2021 12:12:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 000656B0078; Fri, 24 Sep 2021 12:12:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE35D6B0074; Fri, 24 Sep 2021 12:12:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0088.hostedemail.com [216.40.44.88]) by kanga.kvack.org (Postfix) with ESMTP id C8A6E6B0073 for ; Fri, 24 Sep 2021 12:12:58 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 8D1FD82499A8 for ; Fri, 24 Sep 2021 16:12:58 +0000 (UTC) X-FDA: 78622960836.13.7C310DA Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf12.hostedemail.com (Postfix) with ESMTP id 15DB010000A0 for ; Fri, 24 Sep 2021 16:12:57 +0000 (UTC) X-IronPort-AV: E=McAfee;i="6200,9189,10116"; a="203601577" X-IronPort-AV: E=Sophos;i="5.85,320,1624345200"; d="scan'208";a="203601577" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Sep 2021 09:12:56 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,320,1624345200"; d="scan'208";a="551865386" Received: from davehans-spike.ostc.intel.com (HELO localhost.localdomain) ([10.165.28.105]) by FMSMGA003.fm.intel.com with ESMTP; 24 Sep 2021 09:12:55 -0700 Subject: [PATCH 2/2] mm/migrate: add CPU hotplug to demotion #ifdef To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org,Dave Hansen ,ying.huang@intel.com,mhocko@suse.com,weixugc@google.com,osalvador@suse.de,rientjes@google.com,dan.j.williams@intel.com,david@redhat.com,gthelen@google.com,yang.shi@linux.alibaba.com,akpm@linux-foundation.org From: Dave Hansen Date: Fri, 24 Sep 2021 09:12:55 -0700 References: <20210924161251.093CCD06@davehans-spike.ostc.intel.com> In-Reply-To: <20210924161251.093CCD06@davehans-spike.ostc.intel.com> Message-Id: <20210924161255.E5FE8F7E@davehans-spike.ostc.intel.com> X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 15DB010000A0 X-Stat-Signature: gsixp6rgcxw1sumfmaq97f1qg5x33npz Authentication-Results: imf12.hostedemail.com; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=intel.com (policy=none); spf=none (imf12.hostedemail.com: domain of dave.hansen@linux.intel.com has no SPF policy when checking 192.55.52.136) smtp.mailfrom=dave.hansen@linux.intel.com X-HE-Tag: 1632499977-472300 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Dave Hansen Once upon a time, the node demotion updates were driven solely by memory hotplug events. But now, there are handlers for both CPU and memory hotplug. However, the #ifdef around the code checks only memory hotplug. A system that has HOTPLUG_CPU=y but MEMORY_HOTPLUG=n would miss CPU hotplug events. Update the #ifdef around the common code. Add memory and CPU-specific #ifdefs for their handlers. These memory/CPU #ifdefs avoid unused function warnings when their Kconfig option is off. Fixes: 884a6e5d1f93 ("mm/migrate: update node demotion order on hotplug events") Signed-off-by: Dave Hansen Cc: "Huang, Ying" Cc: Michal Hocko Cc: Wei Xu Cc: Oscar Salvador Cc: David Rientjes Cc: Dan Williams Cc: David Hildenbrand Cc: Greg Thelen Cc: Yang Shi Cc: Andrew Morton --- b/mm/migrate.c | 46 +++++++++++++++++++++++++--------------------- 1 file changed, 25 insertions(+), 21 deletions(-) diff -puN mm/migrate.c~add-cpu-hotplug-config mm/migrate.c --- a/mm/migrate.c~add-cpu-hotplug-config 2021-09-24 09:12:31.308377810 -0700 +++ b/mm/migrate.c 2021-09-24 09:12:31.308377810 -0700 @@ -3066,7 +3066,7 @@ void migrate_vma_finalize(struct migrate EXPORT_SYMBOL(migrate_vma_finalize); #endif /* CONFIG_DEVICE_PRIVATE */ -#if defined(CONFIG_MEMORY_HOTPLUG) +#if defined(CONFIG_MEMORY_HOTPLUG) || defined(CONFIG_HOTPLUG_CPU) /* Disable reclaim-based migration. */ static void __disable_all_migrate_targets(void) { @@ -3208,25 +3208,7 @@ static void set_migration_target_nodes(v put_online_mems(); } -/* - * React to hotplug events that might affect the migration targets - * like events that online or offline NUMA nodes. - * - * The ordering is also currently dependent on which nodes have - * CPUs. That means we need CPU on/offline notification too. - */ -static int migration_online_cpu(unsigned int cpu) -{ - set_migration_target_nodes(); - return 0; -} - -static int migration_offline_cpu(unsigned int cpu) -{ - set_migration_target_nodes(); - return 0; -} - +#if defined(CONFIG_MEMORY_HOTPLUG) /* * This leaves migrate-on-reclaim transiently disabled between * the MEM_GOING_OFFLINE and MEM_OFFLINE events. This runs @@ -3283,6 +3265,27 @@ static int __meminit migrate_on_reclaim_ return notifier_from_errno(0); } +#endif /* CONFIG_MEMORY_HOTPLUG */ + +#ifdef CONFIG_HOTPLUG_CPU +/* + * React to hotplug events that might affect the migration targets + * like events that online or offline NUMA nodes. + * + * The ordering is also currently dependent on which nodes have + * CPUs. That means we need CPU on/offline notification too. + */ +static int migration_online_cpu(unsigned int cpu) +{ + set_migration_target_nodes(); + return 0; +} + +static int migration_offline_cpu(unsigned int cpu) +{ + set_migration_target_nodes(); + return 0; +} static int __init migrate_on_reclaim_init(void) { @@ -3303,4 +3306,5 @@ static int __init migrate_on_reclaim_ini return 0; } late_initcall(migrate_on_reclaim_init); -#endif /* CONFIG_MEMORY_HOTPLUG */ +#endif /* CONFIG_HOTPLUG_CPU */ +#endif /* CONFIG_MEMORY_HOTPLUG || CONFIG_HOTPLUG_CPU */