From patchwork Thu Jan 7 17:23:41 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vitaly Kuznetsov X-Patchwork-Id: 7978471 Return-Path: X-Original-To: patchwork-linux-acpi@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id A8C22BEEE5 for ; Thu, 7 Jan 2016 17:24:18 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 511FF20172 for ; Thu, 7 Jan 2016 17:24:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id EDD4E2011E for ; Thu, 7 Jan 2016 17:24:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752073AbcAGRXv (ORCPT ); Thu, 7 Jan 2016 12:23:51 -0500 Received: from mx1.redhat.com ([209.132.183.28]:53844 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751941AbcAGRXs (ORCPT ); Thu, 7 Jan 2016 12:23:48 -0500 Received: from int-mx14.intmail.prod.int.phx2.redhat.com (int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27]) by mx1.redhat.com (Postfix) with ESMTPS id 32F50C07584D; Thu, 7 Jan 2016 17:23:48 +0000 (UTC) Received: from vitty.brq.redhat.com (vitty.brq.redhat.com [10.34.26.3]) by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u07HNhI9007007; Thu, 7 Jan 2016 12:23:43 -0500 From: Vitaly Kuznetsov To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-acpi@vger.kernel.org, Jonathan Corbet , Greg Kroah-Hartman , Daniel Kiper , Dan Williams , Tang Chen , David Vrabel , David Rientjes , Andrew Morton , Naoya Horiguchi , Xishi Qiu , Mel Gorman , "K. Y. Srinivasan" , Igor Mammedov , Kay Sievers , Konrad Rzeszutek Wilk , Boris Ostrovsky Subject: [PATCH v3] memory-hotplug: add automatic onlining policy for the newly added memory Date: Thu, 7 Jan 2016 18:23:41 +0100 Message-Id: <1452187421-15747-1-git-send-email-vkuznets@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27 Sender: linux-acpi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Currently, all newly added memory blocks remain in 'offline' state unless someone onlines them, some linux distributions carry special udev rules like: SUBSYSTEM=="memory", ACTION=="add", ATTR{state}=="offline", ATTR{state}="online" to make this happen automatically. This is not a great solution for virtual machines where memory hotplug is being used to address high memory pressure situations as such onlining is slow and a userspace process doing this (udev) has a chance of being killed by the OOM killer as it will probably require to allocate some memory. Introduce default policy for the newly added memory blocks in /sys/devices/system/memory/auto_online_blocks file with two possible values: "offline" which preserves the current behavior and "online" which causes all newly added memory blocks to go online as soon as they're added. The default is "offline". Cc: Jonathan Corbet Cc: Greg Kroah-Hartman Cc: Daniel Kiper Cc: Dan Williams Cc: Tang Chen Cc: David Vrabel Cc: David Rientjes Cc: Andrew Morton Cc: Naoya Horiguchi Cc: Xishi Qiu Cc: Mel Gorman Cc: "K. Y. Srinivasan" Cc: Igor Mammedov Cc: Kay Sievers Cc: Konrad Rzeszutek Wilk Cc: Boris Ostrovsky Signed-off-by: Vitaly Kuznetsov --- - Changes since 'v2': - Remove config option, revert to 'offline' by default [Andrew Morton, David Rientjes] - Rename 'hotplug_autoonline' to 'auto_online_blocks' [David Rientjes] - Changes since 'v1': Add 'online' parameter to add_memory_resource() as it is being used by xen ballon driver and it adds "empty" memory pages [David Vrabel]. (I don't completely understand what prevents manual onlining in this case as we still have all newly added blocks in sysfs ... this is the discussion point.) - Changes since 'RFC': It seems nobody is strongly opposed to the idea, thus non-RFC. Change memhp_autoonline to bool, we support only MMOP_ONLINE_KEEP and MMOP_OFFLINE for the auto-onlining policy, eliminate 'unknown' from show_memhp_autoonline(). [Daniel Kiper] Put everything under CONFIG_MEMORY_HOTPLUG_AUTOONLINE, enable the feature by default (when the config option is selected) and add kernel parameter (nomemhp_autoonline) to disable the functionality upon boot when needed. - RFC: I was able to find previous attempts to fix the issue, e.g.: http://marc.info/?l=linux-kernel&m=137425951924598&w=2 http://marc.info/?l=linux-acpi&m=127186488905382 but I'm not completely sure why it didn't work out and the solution I suggest is not 'smart enough', thus 'RFC'. --- Documentation/memory-hotplug.txt | 19 +++++++++++++++---- drivers/base/memory.c | 32 ++++++++++++++++++++++++++++++++ drivers/xen/balloon.c | 2 +- include/linux/memory_hotplug.h | 4 +++- mm/memory_hotplug.c | 12 ++++++++++-- 5 files changed, 61 insertions(+), 8 deletions(-) diff --git a/Documentation/memory-hotplug.txt b/Documentation/memory-hotplug.txt index ce2cfcf..ceaf40c 100644 --- a/Documentation/memory-hotplug.txt +++ b/Documentation/memory-hotplug.txt @@ -254,12 +254,23 @@ If the memory block is online, you'll read "online". If the memory block is offline, you'll read "offline". -5.2. How to online memory +5.2. Memory onlining ------------ -Even if the memory is hot-added, it is not at ready-to-use state. -For using newly added memory, you have to "online" the memory block. +When the memory is hot-added, the kernel decides whether or not to "online" +it according to the policy which can be read from "auto_online_blocks" file: -For onlining, you have to write "online" to the memory block's state file as: +% cat /sys/devices/system/memory/auto_online_blocks + +The default is "offline" which means the newly added memory is not in a +ready-to-use state and you have to "online" the newly added memory blocks +manually. Automatic onlining can be requested by writing "online" to +"auto_online_blocks" file: + +% echo online > /sys/devices/system/memory/auto_online_blocks + +If the automatic onlining wasn't requested or some memory block was offlined +it is possible to change the individual block's state by writing to the "state" +file: % echo online > /sys/devices/system/memory/memoryXXX/state diff --git a/drivers/base/memory.c b/drivers/base/memory.c index 25425d3..44a618d 100644 --- a/drivers/base/memory.c +++ b/drivers/base/memory.c @@ -439,6 +439,37 @@ print_block_size(struct device *dev, struct device_attribute *attr, static DEVICE_ATTR(block_size_bytes, 0444, print_block_size, NULL); /* + * Memory auto online policy. + */ + +static ssize_t +show_auto_online_blocks(struct device *dev, struct device_attribute *attr, + char *buf) +{ + if (memhp_auto_online) + return sprintf(buf, "online\n"); + else + return sprintf(buf, "offline\n"); +} + +static ssize_t +store_auto_online_blocks(struct device *dev, struct device_attribute *attr, + const char *buf, size_t count) +{ + if (sysfs_streq(buf, "online")) + memhp_auto_online = true; + else if (sysfs_streq(buf, "offline")) + memhp_auto_online = false; + else + return -EINVAL; + + return count; +} + +static DEVICE_ATTR(auto_online_blocks, 0644, show_auto_online_blocks, + store_auto_online_blocks); + +/* * Some architectures will have custom drivers to do this, and * will not need to do it from userspace. The fake hot-add code * as well as ppc64 will do all of their discovery in userspace @@ -737,6 +768,7 @@ static struct attribute *memory_root_attrs[] = { #endif &dev_attr_block_size_bytes.attr, + &dev_attr_auto_online_blocks.attr, NULL }; diff --git a/drivers/xen/balloon.c b/drivers/xen/balloon.c index 12eab50..890c3b5 100644 --- a/drivers/xen/balloon.c +++ b/drivers/xen/balloon.c @@ -338,7 +338,7 @@ static enum bp_state reserve_additional_memory(void) } #endif - rc = add_memory_resource(nid, resource); + rc = add_memory_resource(nid, resource, false); if (rc) { pr_warn("Cannot add additional memory (%i)\n", rc); goto err; diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h index 2ea574f..4b7949a 100644 --- a/include/linux/memory_hotplug.h +++ b/include/linux/memory_hotplug.h @@ -99,6 +99,8 @@ extern void __online_page_free(struct page *page); extern int try_online_node(int nid); +extern bool memhp_auto_online; + #ifdef CONFIG_MEMORY_HOTREMOVE extern bool is_pageblock_removable_nolock(struct page *page); extern int arch_remove_memory(u64 start, u64 size); @@ -267,7 +269,7 @@ static inline void remove_memory(int nid, u64 start, u64 size) {} extern int walk_memory_range(unsigned long start_pfn, unsigned long end_pfn, void *arg, int (*func)(struct memory_block *, void *)); extern int add_memory(int nid, u64 start, u64 size); -extern int add_memory_resource(int nid, struct resource *resource); +extern int add_memory_resource(int nid, struct resource *resource, bool online); extern int zone_for_memory(int nid, u64 start, u64 size, int zone_default, bool for_device); extern int arch_add_memory(int nid, u64 start, u64 size, bool for_device); diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index a042a9d..0ecf860 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -76,6 +76,9 @@ static struct { #define memhp_lock_acquire() lock_map_acquire(&mem_hotplug.dep_map) #define memhp_lock_release() lock_map_release(&mem_hotplug.dep_map) +bool memhp_auto_online; +EXPORT_SYMBOL_GPL(memhp_auto_online); + void get_online_mems(void) { might_sleep(); @@ -1232,7 +1235,7 @@ int zone_for_memory(int nid, u64 start, u64 size, int zone_default, } /* we are OK calling __meminit stuff here - we have CONFIG_MEMORY_HOTPLUG */ -int __ref add_memory_resource(int nid, struct resource *res) +int __ref add_memory_resource(int nid, struct resource *res, bool online) { u64 start, size; pg_data_t *pgdat = NULL; @@ -1292,6 +1295,11 @@ int __ref add_memory_resource(int nid, struct resource *res) /* create new memmap entry */ firmware_map_add_hotplug(start, start + size, "System RAM"); + /* online pages if requested */ + if (online) + online_pages(start >> PAGE_SHIFT, size >> PAGE_SHIFT, + MMOP_ONLINE_KEEP); + goto out; error: @@ -1315,7 +1323,7 @@ int __ref add_memory(int nid, u64 start, u64 size) if (!res) return -EEXIST; - ret = add_memory_resource(nid, res); + ret = add_memory_resource(nid, res, memhp_auto_online); if (ret < 0) release_memory_resource(res); return ret;