From patchwork Thu Sep 20 22:26:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10608801 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4C6F915E8 for ; Thu, 20 Sep 2018 22:26:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3E0402DEFC for ; Thu, 20 Sep 2018 22:26:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 31E042DF14; Thu, 20 Sep 2018 22:26:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A08772DEFC for ; Thu, 20 Sep 2018 22:26:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 63E8A8E0003; Thu, 20 Sep 2018 18:26:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5EC688E0001; Thu, 20 Sep 2018 18:26:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4DCD18E0003; Thu, 20 Sep 2018 18:26:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id 0BF278E0001 for ; Thu, 20 Sep 2018 18:26:54 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id b69-v6so5438806pfc.20 for ; Thu, 20 Sep 2018 15:26:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:from :to:cc:date:message-id:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=j6g+gX5p29AHCR+jAj2W2G6ghXucCVUNEUSc4EVcWrk=; b=lGwt1K70zVcmd8v9mnjY4mclZiMEo7IUW4bAVfqW3gjVTo0Oo//CfilJO1xv2v/UW3 EYDL6wzP9XfSlNN3WW7ZsPadAtpB3OTEA7+ynuo4e0LtvxK4lC7BK8DaCmEDXTdb95/L bEr80oTwujwJXXEDYIfeFQfRzU1WBYt+H6lRUkD/xS1PeV0/UT2+Lv9nqgWgRWY1T1hY X0nRnD5QcbruUeRwK340VuedJ0X471oP+BKgb/sRA6vM6ZUiKrC1H89f8NwYZlbTI8wv lXISy/l6anv24GghCli2mbWISCtTBJ07o+/miFVyz7dpT1wc7dPcd7K+REpvz+YIrF2F PCdQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51A9fIPLv0yzLspxlLpnD+yyLlVk3PMtY+LPnpKWOBMffHeDMMz5 Wwz28mAWhcyZInck7zUrAOvLc9XeCuMxFm1PY/MUnQ1cGjbLGiT1bRYHJ975yg0lQNVoo5xIFgw y1r8VZlhyJRHwT9wRfeuc9+/fhJ48uL+5DTmJX05b5lGL4Zr8VTAd9PIlCaBljWoegA== X-Received: by 2002:a17:902:280a:: with SMTP id e10-v6mr41299456plb.187.1537482413711; Thu, 20 Sep 2018 15:26:53 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYpx4cgqJXRQdYM/ogccG/Bu/1zbIEmWwIJpNlr8H3UJ9MaRDPUBFowreU5/EASogBL/h8C X-Received: by 2002:a17:902:280a:: with SMTP id e10-v6mr41299416plb.187.1537482412629; Thu, 20 Sep 2018 15:26:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537482412; cv=none; d=google.com; s=arc-20160816; b=grDZb/9fWW+JGJdfMrL46dR4I6KNc+J2ivvi32P4dv2rHeG5h+ZvpApPtzZE6rqkl0 J/4ZB/wgQMr4ZlPLdXnH20Fh2JNvU/J4KHnW5wRfFgWnIOqN4CGQXeB++lODePFJb9vs YUxrsWH/TIObt4N6FU6JnctpqImyhBQHLUjQ2oybqMAzPt7sfFstZ+ImP5sWN9Xn52Jl hRbpdlr1e/TZfSPq6Nb6u7Qx2dCPQ2waL/8iOdZNLj72uJGMSPAcroqPKp7BPxddm4ya iJrXBkNshQR6+8p2SFprEmaTb6AgKv55Sd3i/G+mUJbP+5D2KJAn0DSh/Jh4eOzTEqBN TujA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject; bh=j6g+gX5p29AHCR+jAj2W2G6ghXucCVUNEUSc4EVcWrk=; b=akyMBHBMQH2QfZg/usYtmK9YVWOjigrIcv10h9Ry+mD2i9QGec0KKxc0AEW1TBDFjN UPIcvTYrOMzMX2RwO0eDFyXnjcF5U2OD0Lq8015dq2UtLEf4bEo2DCY28gMO+GF/SvVF JgVKUEPkXU76+KvBmcGVOvrdq3x7k+27n/4C4P37RuL9xsI8H//yD2I9IRRGQhgz7dv9 SldvapHLVu+6kED9E7WQauiv9XW6IgOE00NOvEIom10xQoWCp5zoxPwRTFihSGO7KN/a wDjRZoiRCM/tC55BPnOImdDXFWIXEZxMt6ueD1+jWigCH6POmcUXo7+YRbBL3xZZlKjG Z3HA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga12.intel.com (mga12.intel.com. [192.55.52.136]) by mx.google.com with ESMTPS id w1-v6si25923139pgt.629.2018.09.20.15.26.52 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 20 Sep 2018 15:26:52 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 192.55.52.136 as permitted sender) client-ip=192.55.52.136; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 20 Sep 2018 15:26:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,282,1534834800"; d="scan'208";a="85285550" Received: from ahduyck-mobl.amr.corp.intel.com (HELO localhost.localdomain) ([10.7.198.152]) by orsmga003.jf.intel.com with ESMTP; 20 Sep 2018 15:26:36 -0700 Subject: [PATCH v4 1/5] mm: Provide kernel parameter to allow disabling page init poisoning From: Alexander Duyck To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org Cc: pavel.tatashin@microsoft.com, mhocko@suse.com, dave.jiang@intel.com, mingo@kernel.org, dave.hansen@intel.com, jglisse@redhat.com, akpm@linux-foundation.org, logang@deltatee.com, dan.j.williams@intel.com, kirill.shutemov@linux.intel.com Date: Thu, 20 Sep 2018 15:26:36 -0700 Message-ID: <20180920222415.19464.38400.stgit@localhost.localdomain> In-Reply-To: <20180920215824.19464.8884.stgit@localhost.localdomain> References: <20180920215824.19464.8884.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP On systems with a large amount of memory it can take a significant amount of time to initialize all of the page structs with the PAGE_POISON_PATTERN value. I have seen it take over 2 minutes to initialize a system with over 12TB of RAM. In order to work around the issue I had to disable CONFIG_DEBUG_VM and then the boot time returned to something much more reasonable as the arch_add_memory call completed in milliseconds versus seconds. However in doing that I had to disable all of the other VM debugging on the system. In order to work around a kernel that might have CONFIG_DEBUG_VM enabled on a system that has a large amount of memory I have added a new kernel parameter named "vm_debug" that can be set to "-" in order to disable it. Signed-off-by: Alexander Duyck Reviewed-by: Pavel Tatashin --- v3: Switched from kernel config option to parameter v4: Added comment to parameter handler to record when option is disabled Updated parameter description based on feedback from Michal Hocko Fixed GB vs TB typo in patch description. Switch to vm_debug option similar to slub_debug Documentation/admin-guide/kernel-parameters.txt | 12 ++++++ include/linux/page-flags.h | 8 ++++ mm/debug.c | 46 +++++++++++++++++++++++ mm/memblock.c | 5 +-- mm/sparse.c | 4 +- 5 files changed, 69 insertions(+), 6 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index dfe3d7b99abf..ee257b5b584f 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -4811,6 +4811,18 @@ This is actually a boot loader parameter; the value is passed to the kernel using a special protocol. + vm_debug[=options] [KNL] Available with CONFIG_DEBUG_VM=y. + May slow down system boot speed, especially when + enabled on systems with a large amount of memory. + All options are enabled by default, and this + interface is meant to allow for selectively + enabling or disabling specific virtual memory + debugging features. + + Available options are: + P Enable page structure init time poisoning + - Disable all of the above options + vmalloc=nn[KMG] [KNL,BOOT] Forces the vmalloc area to have an exact size of . This can be used to increase the minimum size (128MB on x86). It can also be used to diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 4d99504f6496..934f91ef3f54 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -163,6 +163,14 @@ static inline int PagePoisoned(const struct page *page) return page->flags == PAGE_POISON_PATTERN; } +#ifdef CONFIG_DEBUG_VM +void page_init_poison(struct page *page, size_t size); +#else +static inline void page_init_poison(struct page *page, size_t size) +{ +} +#endif + /* * Page flags policies wrt compound pages * diff --git a/mm/debug.c b/mm/debug.c index bd10aad8539a..cdacba12e09a 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -13,6 +13,7 @@ #include #include #include +#include #include "internal.h" @@ -175,4 +176,49 @@ void dump_mm(const struct mm_struct *mm) ); } +static bool page_init_poisoning __read_mostly = true; + +static int __init setup_vm_debug(char *str) +{ + bool __page_init_poisoning = true; + + /* + * Calling vm_debug with no arguments is equivalent to requesting + * to enable all debugging options we can control. + */ + if (*str++ != '=' || !*str) + goto out; + + __page_init_poisoning = false; + if (*str == '-') + goto out; + + while (*str) { + switch (tolower(*str)) { + case'p': + __page_init_poisoning = true; + break; + default: + pr_err("vm_debug option '%c' unknown. skipped\n", + *str); + } + + str++; + } +out: + if (page_init_poisoning && !__page_init_poisoning) + pr_warn("Page struct poisoning disabled by kernel command line option 'vm_debug'\n"); + + page_init_poisoning = __page_init_poisoning; + + return 1; +} +__setup("vm_debug", setup_vm_debug); + +void page_init_poison(struct page *page, size_t size) +{ + if (page_init_poisoning) + memset(page, PAGE_POISON_PATTERN, size); +} +EXPORT_SYMBOL_GPL(page_init_poison); #endif /* CONFIG_DEBUG_VM */ diff --git a/mm/memblock.c b/mm/memblock.c index f7981098537b..b1017ec1b167 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -1495,10 +1495,9 @@ void * __init memblock_virt_alloc_try_nid_raw( ptr = memblock_virt_alloc_internal(size, align, min_addr, max_addr, nid); -#ifdef CONFIG_DEBUG_VM if (ptr && size > 0) - memset(ptr, PAGE_POISON_PATTERN, size); -#endif + page_init_poison(ptr, size); + return ptr; } diff --git a/mm/sparse.c b/mm/sparse.c index 10b07eea9a6e..67ad061f7fb8 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -696,13 +696,11 @@ int __meminit sparse_add_one_section(struct pglist_data *pgdat, goto out; } -#ifdef CONFIG_DEBUG_VM /* * Poison uninitialized struct pages in order to catch invalid flags * combinations. */ - memset(memmap, PAGE_POISON_PATTERN, sizeof(struct page) * PAGES_PER_SECTION); -#endif + page_init_poison(memmap, sizeof(struct page) * PAGES_PER_SECTION); section_mark_present(ms); sparse_init_one_section(ms, section_nr, memmap, usemap); From patchwork Thu Sep 20 22:27:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10608805 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 006E315A6 for ; Thu, 20 Sep 2018 22:28:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E0C9728816 for ; Thu, 20 Sep 2018 22:28:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CEE3F286EC; Thu, 20 Sep 2018 22:28:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 00CF92DF06 for ; Thu, 20 Sep 2018 22:27:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1F5C28E0004; Thu, 20 Sep 2018 18:27:58 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1A5A58E0001; Thu, 20 Sep 2018 18:27:58 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 094A98E0004; Thu, 20 Sep 2018 18:27:58 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id BFD768E0001 for ; Thu, 20 Sep 2018 18:27:57 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id v186-v6so3668312pgb.14 for ; Thu, 20 Sep 2018 15:27:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:from :to:cc:date:message-id:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=rtBDAMbGmBlNQ1vdQfjvBiLqtdFJw4/gYxGBxmLdmPM=; b=TVj7MZ1+97UTXre3PFKgXeNzDCFBunisteWtka0L5qcCTg47z++9wM6P3cEAB94Yuo hbxghD8y0uDQSaf/jAYo+P41MucGvVbo9BxxQKIGrKrDCtaYo4/Qxl/rur9wwqN2YCuc qLREXz6JnyUv9oiU8CT12kCs6qhfHKd+65F+2QvvLjGK/RlVMWtFrJWsthCmnzwiHi3u 3ctE5vqZSCxltulAeabv28rRHr/4tfs0+1jeWAO05NlKsb1XjfSj6YOIkxyh5mVDsoPh I/YmI4UXS9JVOkuHH2vjRIx/T1/zlj3A82b+Gi02P1+UCE3Exr90WQuAFUdNSJMKEyPP tVYg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51D/AuN6SvRXo+k7SczpcC/BuG5qdttSfKa1zxVhqK5amxJ9BPHt SLU/1NoAFGioFZ0oXy3CM3s5z+O5iB4pwmxpE6GcH5XkIBjsMzvtZRa7k1qi3RRx9jsPRbpulK8 RGRoW6h37Et7yLW8xhJ+R31XKAtRz4A1zBCT0NULfANMgQXRvuQ5h/BgCVhvn6nr1yA== X-Received: by 2002:a17:902:bd4a:: with SMTP id b10-v6mr41545728plx.209.1537482477431; Thu, 20 Sep 2018 15:27:57 -0700 (PDT) X-Google-Smtp-Source: ANB0VdaQNGF4sIS0soN+oJ4ksnYP6gOcf1eDsXtsNgz35spDTXJ2WEJX2kNhtxZH29EmMDnumj4p X-Received: by 2002:a17:902:bd4a:: with SMTP id b10-v6mr41545689plx.209.1537482476529; Thu, 20 Sep 2018 15:27:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537482476; cv=none; d=google.com; s=arc-20160816; b=dzQ938xBsJFC6I37xxXMBXguX59Sg4fR36S1mubd6av0Ql47ETyolPEyZUEXGRTwxC UFLIvc8rAVpWHuIyqqSaQioA1fVCDmocgRNU8iBBuebJ05u0U1RmFby1edLDolPNVHse wF3e3D17TwOjtRgnr1go7kGC0pdfU5UW8eOm9w4amy5XOCuOhakV/8swc2ujYHycBJ5j WEPzJBd/YFt0hVovD0rN5WG3INCtUPHPmuXxDJPFECn40XEOo96EvTDE7lYxXYSifUEm StSoQ3xnWr35LCyC9DRL9GMLa65lI2Pr2ZFFDST13LbfuwBEGVUPs/hQ34AGrHkf+A+m EOqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject; bh=rtBDAMbGmBlNQ1vdQfjvBiLqtdFJw4/gYxGBxmLdmPM=; b=KXdjAEUskAslhkMvnjeXtKG6VsVopk1kb/Ceyta6d4X8xHyWGFChNC0xGa97TObXL0 rpSN/977WOFUc38U+49MLLeDhINtOk9cGyEBftwz+lFlIxIEyYErySStiNtrEnzSKX/s O6TuPgf+2XGMpORpyMYIVDOxTI0jPzIXIcVftv/ifuMg8T2oPGu1KHa6XHsF4L5U1bay 5aTjcmR48q6AdSobtVuqt+PI55B+c3syUw4lahm1EEFTsSOeliU1cBi9tXyT+RlY+HJm PY6CYrL/bGyyth/UnXeEDsoNVKzz9zlwh3clzAQPNuGAAL9DOqnpNpvTjlmGrRbP57lH NPLw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga01.intel.com (mga01.intel.com. [192.55.52.88]) by mx.google.com with ESMTPS id p7-v6si24646254plo.159.2018.09.20.15.27.56 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 20 Sep 2018 15:27:56 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 192.55.52.88 as permitted sender) client-ip=192.55.52.88; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 20 Sep 2018 15:27:55 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,282,1534834800"; d="scan'208";a="85285823" Received: from ahduyck-mobl.amr.corp.intel.com (HELO localhost.localdomain) ([10.7.198.152]) by orsmga003.jf.intel.com with ESMTP; 20 Sep 2018 15:27:53 -0700 Subject: [PATCH v4 2/5] mm: Create non-atomic version of SetPageReserved for init use From: Alexander Duyck To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org Cc: pavel.tatashin@microsoft.com, mhocko@suse.com, dave.jiang@intel.com, mingo@kernel.org, dave.hansen@intel.com, jglisse@redhat.com, akpm@linux-foundation.org, logang@deltatee.com, dan.j.williams@intel.com, kirill.shutemov@linux.intel.com Date: Thu, 20 Sep 2018 15:27:53 -0700 Message-ID: <20180920222641.19464.75787.stgit@localhost.localdomain> In-Reply-To: <20180920215824.19464.8884.stgit@localhost.localdomain> References: <20180920215824.19464.8884.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP It doesn't make much sense to use the atomic SetPageReserved at init time when we are using memset to clear the memory and manipulating the page flags via simple "&=" and "|=" operations in __init_single_page. This patch adds a non-atomic version __SetPageReserved that can be used during page init and shows about a 10% improvement in initialization times on the systems I have available for testing. On those systems I saw initialization times drop from around 35 seconds to around 32 seconds to initialize a 3TB block of persistent memory. I believe the main advantage of this is that it allows for more compiler optimization as the __set_bit operation can be reordered whereas the atomic version cannot. I tried adding a bit of documentation based on commit ("mm, memory_hotplug: do not associate hotadded memory to zones until online"). Ideally the reserved flag should be set earlier since there is a brief window where the page is initialization via __init_single_page and we have not set the PG_Reserved flag. I'm leaving that for a future patch set as that will require a more significant refactor. Acked-by: Michal Hocko Signed-off-by: Alexander Duyck Reviewed-by: Pavel Tatashin --- v4: Added comment about __set_bit vs set_bit to the patch description include/linux/page-flags.h | 1 + mm/page_alloc.c | 9 +++++++-- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 934f91ef3f54..50ce1bddaf56 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -303,6 +303,7 @@ static inline void page_init_poison(struct page *page, size_t size) PAGEFLAG(Reserved, reserved, PF_NO_COMPOUND) __CLEARPAGEFLAG(Reserved, reserved, PF_NO_COMPOUND) + __SETPAGEFLAG(Reserved, reserved, PF_NO_COMPOUND) PAGEFLAG(SwapBacked, swapbacked, PF_NO_TAIL) __CLEARPAGEFLAG(SwapBacked, swapbacked, PF_NO_TAIL) __SETPAGEFLAG(SwapBacked, swapbacked, PF_NO_TAIL) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 712cab17f86f..29bd662fffd7 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1239,7 +1239,12 @@ void __meminit reserve_bootmem_region(phys_addr_t start, phys_addr_t end) /* Avoid false-positive PageTail() */ INIT_LIST_HEAD(&page->lru); - SetPageReserved(page); + /* + * no need for atomic set_bit because the struct + * page is not visible yet so nobody should + * access it yet. + */ + __SetPageReserved(page); } } } @@ -5513,7 +5518,7 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone, page = pfn_to_page(pfn); __init_single_page(page, pfn, zone, nid); if (context == MEMMAP_HOTPLUG) - SetPageReserved(page); + __SetPageReserved(page); /* * Mark the block movable so that blocks are reserved for From patchwork Thu Sep 20 22:29:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10608811 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1687D174F for ; Thu, 20 Sep 2018 22:29:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0850A2DF19 for ; Thu, 20 Sep 2018 22:29:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F061B2E002; Thu, 20 Sep 2018 22:29:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 52F812DF19 for ; Thu, 20 Sep 2018 22:29:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 466128E0005; Thu, 20 Sep 2018 18:29:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 414938E0001; Thu, 20 Sep 2018 18:29:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3053D8E0005; Thu, 20 Sep 2018 18:29:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by kanga.kvack.org (Postfix) with ESMTP id E6E458E0001 for ; Thu, 20 Sep 2018 18:29:47 -0400 (EDT) Received: by mail-pf1-f198.google.com with SMTP id q21-v6so5429931pff.21 for ; Thu, 20 Sep 2018 15:29:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:from :to:cc:date:message-id:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=vPEUaldZTN2zcegkQF7YPN+mqkSWAKD1fuMsCcAJKB4=; b=ukKasY411sVa6xxfbSJW5Gmt2jDgBYi7EawAMFtsLRpJ664k/3iLw49StWXecgpvAC RxVg3mho3yocgxDAZc12ycxU3LGIa9+ydR3WaMpTkPBbLvqzXjaEmb6HIcfA7uPh4C5z dESu5fAxkjvEx0ZKoUahAoYGiFS/79a/+8qUQWWPYMtHcaeifWzCDkQ0VwBNNoPTBGd8 TzQlfIt1/x2xnH24RiFBMTlu14luaxY9CMVOb0Ee5kB0/9mmTla1YHWO/wmjlvgTUZW/ xNX5Mf8mHXGRkDbMODV6LcYQWyhcw5xhIA+6cEmscaBMrY7gOYx7qziSfJlcUnQPN9/G n0zA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51CBYK7mSK9jgTA0ahkGchCN8ELtFirgqn2zjSseWSYU5depqxsl bA2jJE/wj/DGbSE/zxcFSUuVNn4VZ3FIMeMtaDzrAL+i3Z0hLvBVeRk/9qGrENyelSdWnQq4122 YdHAxdNHNYpYaUo2alamdNhiXXsJBc9QHzugfnJ4i+/PhUlxnQ5Uqv2Kj5nyB9Ien2w== X-Received: by 2002:a63:9857:: with SMTP id l23-v6mr4805347pgo.321.1537482587560; Thu, 20 Sep 2018 15:29:47 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYd9Qm9yyMbNTZo2i2HKPhecgqxpBFud4EC6X/QWE0R+dDmnkT/l2RHG1mgcCo2U1ww/ki2 X-Received: by 2002:a63:9857:: with SMTP id l23-v6mr4805296pgo.321.1537482586402; Thu, 20 Sep 2018 15:29:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537482586; cv=none; d=google.com; s=arc-20160816; b=T9dM8Iiad4RsudSr1HZa499uRbjhw++MDMv047fhHFn94Mqex9rWCVo6sNDcvj9lzV biKMW0cvpiS/WOnPAigLy2BFiVjNgBNyKfG3rlTFY4Gp/J5wQF8dd5zcO5vLPZg5SEWI mcjVBIxlzdKmzy2kayQvDC9lsOs2mgNTVnRc3Iwsy6jPlcDnGfMcyecsd3j3g+fzraIK BCS8N3IfT4YhiekmvabrDDIMFO8/5CkG+h338WC1rt3rPZ5sBRVZGkNxQiZNdq4MioZ2 /38QLHbC033JHVmQwZ4NONMne7esJVZM0qgQwbwzqe8DdV+NiOkFnMsXwuONoN/ctDU0 TSjQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject; bh=vPEUaldZTN2zcegkQF7YPN+mqkSWAKD1fuMsCcAJKB4=; b=aym+6Ok++blVVTqNt3kd7ON278VNWmLRNMT+EkVS0eDciDuXnDZf8uyZ6yxTBWgsOc S8F+h4gmN3XR9vocjYe9iRoWtt4xElWYsefdh9M/R3k5UFtWotB2pm6ZDjmPT+WQhVg3 kxDFd3SIjxvXgFtFIeMnsvXWq/NyaYfQN5mHcX2kx0bmyRd4euH8RKq84F+KmuqkzHP5 am7qQtjaQkK0c6QwZUpcA79W16gviMw0g/bwIbjf7qBeF+vn1ON5pnZ/z4xbfmceYvcN h9/6j9vxFflNE7m6J5IvRkzMltHr2uLfrXCoHDNgiHJrDgCOF6plsy1XcmRcs0KpJW/s JHAQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga07.intel.com (mga07.intel.com. [134.134.136.100]) by mx.google.com with ESMTPS id t9-v6si23641437pgo.68.2018.09.20.15.29.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 20 Sep 2018 15:29:46 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.100 as permitted sender) client-ip=134.134.136.100; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 20 Sep 2018 15:29:45 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,282,1534834800"; d="scan'208";a="85286839" Received: from ahduyck-mobl.amr.corp.intel.com (HELO localhost.localdomain) ([10.7.198.152]) by orsmga003.jf.intel.com with ESMTP; 20 Sep 2018 15:29:33 -0700 Subject: [PATCH v4 3/5] mm: Defer ZONE_DEVICE page initialization to the point where we init pgmap From: Alexander Duyck To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org Cc: pavel.tatashin@microsoft.com, mhocko@suse.com, dave.jiang@intel.com, mingo@kernel.org, dave.hansen@intel.com, jglisse@redhat.com, akpm@linux-foundation.org, logang@deltatee.com, dan.j.williams@intel.com, kirill.shutemov@linux.intel.com Date: Thu, 20 Sep 2018 15:29:33 -0700 Message-ID: <20180920222758.19464.83992.stgit@localhost.localdomain> In-Reply-To: <20180920215824.19464.8884.stgit@localhost.localdomain> References: <20180920215824.19464.8884.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The ZONE_DEVICE pages were being initialized in two locations. One was with the memory_hotplug lock held and another was outside of that lock. The problem with this is that it was nearly doubling the memory initialization time. Instead of doing this twice, once while holding a global lock and once without, I am opting to defer the initialization to the one outside of the lock. This allows us to avoid serializing the overhead for memory init and we can instead focus on per-node init times. One issue I encountered is that devm_memremap_pages and hmm_devmmem_pages_create were initializing only the pgmap field the same way. One wasn't initializing hmm_data, and the other was initializing it to a poison value. Since this is something that is exposed to the driver in the case of hmm I am opting for a third option and just initializing hmm_data to 0 since this is going to be exposed to unknown third party drivers. Signed-off-by: Alexander Duyck Reviewed-by: Pavel Tatashin --- v4: Moved moved memmap_init_zone_device to below memmmap_init_zone to avoid merge conflicts with other changes in the kernel. include/linux/mm.h | 2 + kernel/memremap.c | 24 +++++--------- mm/hmm.c | 12 ++++--- mm/page_alloc.c | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 4 files changed, 107 insertions(+), 23 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d63d163f341d..25c89615d303 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -848,6 +848,8 @@ static inline bool is_zone_device_page(const struct page *page) { return page_zonenum(page) == ZONE_DEVICE; } +extern void memmap_init_zone_device(struct zone *, unsigned long, + unsigned long, struct dev_pagemap *); #else static inline bool is_zone_device_page(const struct page *page) { diff --git a/kernel/memremap.c b/kernel/memremap.c index 5b8600d39931..d0c32e473f82 100644 --- a/kernel/memremap.c +++ b/kernel/memremap.c @@ -175,10 +175,10 @@ void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap) struct vmem_altmap *altmap = pgmap->altmap_valid ? &pgmap->altmap : NULL; struct resource *res = &pgmap->res; - unsigned long pfn, pgoff, order; + struct dev_pagemap *conflict_pgmap; pgprot_t pgprot = PAGE_KERNEL; + unsigned long pgoff, order; int error, nid, is_ram; - struct dev_pagemap *conflict_pgmap; align_start = res->start & ~(SECTION_SIZE - 1); align_size = ALIGN(res->start + resource_size(res), SECTION_SIZE) @@ -256,19 +256,13 @@ void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap) if (error) goto err_add_memory; - for_each_device_pfn(pfn, pgmap) { - struct page *page = pfn_to_page(pfn); - - /* - * ZONE_DEVICE pages union ->lru with a ->pgmap back - * pointer. It is a bug if a ZONE_DEVICE page is ever - * freed or placed on a driver-private list. Seed the - * storage with LIST_POISON* values. - */ - list_del(&page->lru); - page->pgmap = pgmap; - percpu_ref_get(pgmap->ref); - } + /* + * Initialization of the pages has been deferred until now in order + * to allow us to do the work while not holding the hotplug lock. + */ + memmap_init_zone_device(&NODE_DATA(nid)->node_zones[ZONE_DEVICE], + align_start >> PAGE_SHIFT, + align_size >> PAGE_SHIFT, pgmap); devm_add_action(dev, devm_memremap_pages_release, pgmap); diff --git a/mm/hmm.c b/mm/hmm.c index c968e49f7a0c..774d684fa2b4 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -1024,7 +1024,6 @@ static int hmm_devmem_pages_create(struct hmm_devmem *devmem) resource_size_t key, align_start, align_size, align_end; struct device *device = devmem->device; int ret, nid, is_ram; - unsigned long pfn; align_start = devmem->resource->start & ~(PA_SECTION_SIZE - 1); align_size = ALIGN(devmem->resource->start + @@ -1109,11 +1108,14 @@ static int hmm_devmem_pages_create(struct hmm_devmem *devmem) align_size >> PAGE_SHIFT, NULL); mem_hotplug_done(); - for (pfn = devmem->pfn_first; pfn < devmem->pfn_last; pfn++) { - struct page *page = pfn_to_page(pfn); + /* + * Initialization of the pages has been deferred until now in order + * to allow us to do the work while not holding the hotplug lock. + */ + memmap_init_zone_device(&NODE_DATA(nid)->node_zones[ZONE_DEVICE], + align_start >> PAGE_SHIFT, + align_size >> PAGE_SHIFT, &devmem->pagemap); - page->pgmap = &devmem->pagemap; - } return 0; error_add_memory: diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 29bd662fffd7..ac1fa0efdea0 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -5490,12 +5490,23 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone, if (highest_memmap_pfn < end_pfn - 1) highest_memmap_pfn = end_pfn - 1; +#ifdef CONFIG_ZONE_DEVICE /* * Honor reservation requested by the driver for this ZONE_DEVICE - * memory + * memory. We limit the total number of pages to initialize to just + * those that might contain the memory mapping. We will defer the + * ZONE_DEVICE page initialization until after we have released + * the hotplug lock. */ - if (altmap && start_pfn == altmap->base_pfn) - start_pfn += altmap->reserve; + if (zone == ZONE_DEVICE) { + if (!altmap) + return; + + if (start_pfn == altmap->base_pfn) + start_pfn += altmap->reserve; + end_pfn = altmap->base_pfn + vmem_altmap_offset(altmap); + } +#endif for (pfn = start_pfn; pfn < end_pfn; pfn++) { /* @@ -5539,6 +5550,81 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone, } } +#ifdef CONFIG_ZONE_DEVICE +void __ref memmap_init_zone_device(struct zone *zone, + unsigned long start_pfn, + unsigned long size, + struct dev_pagemap *pgmap) +{ + unsigned long pfn, end_pfn = start_pfn + size; + struct pglist_data *pgdat = zone->zone_pgdat; + unsigned long zone_idx = zone_idx(zone); + unsigned long start = jiffies; + int nid = pgdat->node_id; + + if (WARN_ON_ONCE(!pgmap || !is_dev_zone(zone))) + return; + + /* + * The call to memmap_init_zone should have already taken care + * of the pages reserved for the memmap, so we can just jump to + * the end of that region and start processing the device pages. + */ + if (pgmap->altmap_valid) { + struct vmem_altmap *altmap = &pgmap->altmap; + + start_pfn = altmap->base_pfn + vmem_altmap_offset(altmap); + size = end_pfn - start_pfn; + } + + for (pfn = start_pfn; pfn < end_pfn; pfn++) { + struct page *page = pfn_to_page(pfn); + + __init_single_page(page, pfn, zone_idx, nid); + + /* + * Mark page reserved as it will need to wait for onlining + * phase for it to be fully associated with a zone. + * + * We can use the non-atomic __set_bit operation for setting + * the flag as we are still initializing the pages. + */ + __SetPageReserved(page); + + /* + * ZONE_DEVICE pages union ->lru with a ->pgmap back + * pointer and hmm_data. It is a bug if a ZONE_DEVICE + * page is ever freed or placed on a driver-private list. + */ + page->pgmap = pgmap; + page->hmm_data = 0; + + /* + * Mark the block movable so that blocks are reserved for + * movable at startup. This will force kernel allocations + * to reserve their blocks rather than leaking throughout + * the address space during boot when many long-lived + * kernel allocations are made. + * + * bitmap is created for zone's valid pfn range. but memmap + * can be created for invalid pages (for alignment) + * check here not to call set_pageblock_migratetype() against + * pfn out of zone. + * + * Please note that MEMMAP_HOTPLUG path doesn't clear memmap + * because this is done early in sparse_add_one_section + */ + if (!(pfn & (pageblock_nr_pages - 1))) { + set_pageblock_migratetype(page, MIGRATE_MOVABLE); + cond_resched(); + } + } + + pr_info("%s initialised, %lu pages in %ums\n", dev_name(pgmap->dev), + size, jiffies_to_msecs(jiffies - start)); +} + +#endif static void __meminit zone_init_free_lists(struct zone *zone) { unsigned int order, t; From patchwork Thu Sep 20 22:29:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10608813 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3315415A6 for ; Thu, 20 Sep 2018 22:29:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 24A372DF19 for ; Thu, 20 Sep 2018 22:29:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 188D12E002; Thu, 20 Sep 2018 22:29:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 85DB22DF19 for ; Thu, 20 Sep 2018 22:29:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 26B728E0006; Thu, 20 Sep 2018 18:29:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1F4BE8E0001; Thu, 20 Sep 2018 18:29:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0E3F68E0006; Thu, 20 Sep 2018 18:29:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id C420D8E0001 for ; Thu, 20 Sep 2018 18:29:48 -0400 (EDT) Received: by mail-pl1-f200.google.com with SMTP id 90-v6so4980158pla.18 for ; Thu, 20 Sep 2018 15:29:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:from :to:cc:date:message-id:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=PLvctc37G0Nx2a3mlCkOPpDTxZJGpZA6T4lg+7tnz6M=; b=h42bxWOFj3ki6jyudUTnG22StdGzLtw1Gol+fF/5Q7D4IcreWgje/rhvsy5FOK9Iad pWPSMLb4HmPaA85vbSrIqjzJ8DMwIp+wFAORaBrtJLpWR7q4TlYZj1/CyHt29io7twPN yvc7OMPWPZXBUCyEgZvipSkqnAGbX0YheugbXAeWz2j7nQ+txC3Y0h5K6kjaig5Igv55 P2ZjsJJAfHCtTmN/1NzTOdNpz9U1e8jYJM9wm5gRGWWIT5Fpkrw97HTIukzo+78JE9qj 2DIB/mV9wWMpFWosigNuvWBDZ/FvsQaB7mf6PIEz3mayFjoQsnCB/2H3MXg0HY9Q3U28 coqg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51CapGHA50j8myM0GX8E5Wd3LxtAgmZ1ry7pgPa3j6TIe9+yZR3a iXiH2eSlkDO6R+0ko+1Q+qfQxkmzDRF+8eFNWzcycli0l5xUUJgT0g01h+pafES00KBLHzZ5q5t /baiPJop5egQ7bfRBWRrr+LkcrktD/JPPNioE3ZlQGLqDOkDyC6Wi5XK2+y4dtGnIJA== X-Received: by 2002:a63:904a:: with SMTP id a71-v6mr38313564pge.339.1537482588449; Thu, 20 Sep 2018 15:29:48 -0700 (PDT) X-Google-Smtp-Source: ANB0VdYyL7jh/p50+EHyLq9x/EYWLvbu9JAWLzqK7PyIeYNgRxigMwFJYc+mc0K2QxDPi9MMha7a X-Received: by 2002:a63:904a:: with SMTP id a71-v6mr38313513pge.339.1537482587160; Thu, 20 Sep 2018 15:29:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537482587; cv=none; d=google.com; s=arc-20160816; b=n8Y7O8jbU2TLv3YhDl5donJ3Ci85nx6X29dUSpPYyE0gT6qR7dUT8ObS9eBEbKoQEz A8MKOqDmubY/pz7rmZunszAfnK4vRnCmEs1KuS9bUyvce1ErFw+pDEfSUleicOVkJhQL uAO2N84T24owsTOHS5ihkqsQ5ZtRlgndM4RMuGyUAC+eKdF6jucryJ6QkdAiqgLttqY7 riRDKp/Wazlal3oKUlnDJIlVQIpv+VFMBBPDXijoSaQsq3V80o3PapVCGcA63Q7+d3f9 bA5rVSTHaSyMHk0/jH+A2ehozQOvzsGd0CNLDXWseIzls4ckFiS74oAj91u9KhAnwmba 5lDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject; bh=PLvctc37G0Nx2a3mlCkOPpDTxZJGpZA6T4lg+7tnz6M=; b=c+LASBHUMClZya1tmabXFQcANY97uXrbi7wAYc1dI2qXdGaRY4gjWNbI66rjVcOxzk 9SmOJbB6EUkr5D07rlxpslo7nkM5U967z0ppz3P4e9otjv8QCaW2WQE83jX9dVyrv0/i LapEVTabXLV9hyWsO68Xwdat/bSJLN78oxWB8UOWL9EjAd1hwg0uiMdQbnkUqNY4+T0i oEhTfEYhBy2967M188StgKL+gvI/DApGiodqeDRNqWY94YgYbia+qCoBD9p3qJtFpguT 4o/B4fCU+ud/Lfl8l+PERuYnttOtft0EMX81Ujxw2DcMBFQ7tzSBKrC+i1ZtRQdFISur 6SmA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga07.intel.com (mga07.intel.com. [134.134.136.100]) by mx.google.com with ESMTPS id t9-v6si23641437pgo.68.2018.09.20.15.29.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 20 Sep 2018 15:29:47 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.100 as permitted sender) client-ip=134.134.136.100; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 20 Sep 2018 15:29:46 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,282,1534834800"; d="scan'208";a="85286849" Received: from ahduyck-mobl.amr.corp.intel.com (HELO localhost.localdomain) ([10.7.198.152]) by orsmga003.jf.intel.com with ESMTP; 20 Sep 2018 15:29:45 -0700 Subject: [PATCH v4 4/5] async: Add support for queueing on specific node From: Alexander Duyck To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org Cc: pavel.tatashin@microsoft.com, mhocko@suse.com, dave.jiang@intel.com, mingo@kernel.org, dave.hansen@intel.com, jglisse@redhat.com, akpm@linux-foundation.org, logang@deltatee.com, dan.j.williams@intel.com, kirill.shutemov@linux.intel.com Date: Thu, 20 Sep 2018 15:29:45 -0700 Message-ID: <20180920222938.19464.34102.stgit@localhost.localdomain> In-Reply-To: <20180920215824.19464.8884.stgit@localhost.localdomain> References: <20180920215824.19464.8884.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch introduces two new variants of the async_schedule_ functions that allow scheduling on a specific node. These functions are async_schedule_on and async_schedule_on_domain which end up mapping to async_schedule and async_schedule_domain but provide NUMA node specific functionality. The original functions were moved to inline function definitions that call the new functions while passing NUMA_NO_NODE. The main motivation behind this is to address the need to be able to schedule NVDIMM init work on specific NUMA nodes in order to improve performance of memory initialization. One additional change I made is I dropped the "extern" from the function prototypes in the async.h kernel header since they aren't needed. Signed-off-by: Alexander Duyck --- include/linux/async.h | 20 +++++++++++++++++--- kernel/async.c | 36 +++++++++++++++++++++++++----------- 2 files changed, 42 insertions(+), 14 deletions(-) diff --git a/include/linux/async.h b/include/linux/async.h index 6b0226bdaadc..9878b99cbb01 100644 --- a/include/linux/async.h +++ b/include/linux/async.h @@ -14,6 +14,7 @@ #include #include +#include typedef u64 async_cookie_t; typedef void (*async_func_t) (void *data, async_cookie_t cookie); @@ -37,9 +38,22 @@ struct async_domain { struct async_domain _name = { .pending = LIST_HEAD_INIT(_name.pending), \ .registered = 0 } -extern async_cookie_t async_schedule(async_func_t func, void *data); -extern async_cookie_t async_schedule_domain(async_func_t func, void *data, - struct async_domain *domain); +async_cookie_t async_schedule_on(async_func_t func, void *data, int node); +async_cookie_t async_schedule_on_domain(async_func_t func, void *data, int node, + struct async_domain *domain); + +static inline async_cookie_t async_schedule(async_func_t func, void *data) +{ + return async_schedule_on(func, data, NUMA_NO_NODE); +} + +static inline async_cookie_t +async_schedule_domain(async_func_t func, void *data, + struct async_domain *domain) +{ + return async_schedule_on_domain(func, data, NUMA_NO_NODE, domain); +} + void async_unregister_domain(struct async_domain *domain); extern void async_synchronize_full(void); extern void async_synchronize_full_domain(struct async_domain *domain); diff --git a/kernel/async.c b/kernel/async.c index a893d6170944..1d7ce81c1949 100644 --- a/kernel/async.c +++ b/kernel/async.c @@ -56,6 +56,7 @@ synchronization with the async_synchronize_full() function, before returning #include #include #include +#include #include "workqueue_internal.h" @@ -149,8 +150,11 @@ static void async_run_entry_fn(struct work_struct *work) wake_up(&async_done); } -static async_cookie_t __async_schedule(async_func_t func, void *data, struct async_domain *domain) +static async_cookie_t __async_schedule(async_func_t func, void *data, + struct async_domain *domain, + int node) { + int cpu = WORK_CPU_UNBOUND; struct async_entry *entry; unsigned long flags; async_cookie_t newcookie; @@ -194,30 +198,40 @@ static async_cookie_t __async_schedule(async_func_t func, void *data, struct asy /* mark that this task has queued an async job, used by module init */ current->flags |= PF_USED_ASYNC; + /* guarantee cpu_online_mask doesn't change during scheduling */ + get_online_cpus(); + + if (node >= 0 && node < MAX_NUMNODES && node_online(node)) + cpu = cpumask_any_and(cpumask_of_node(node), cpu_online_mask); + /* schedule for execution */ - queue_work(system_unbound_wq, &entry->work); + queue_work_on(cpu, system_unbound_wq, &entry->work); + + put_online_cpus(); return newcookie; } /** - * async_schedule - schedule a function for asynchronous execution + * async_schedule_on - schedule a function for asynchronous execution * @func: function to execute asynchronously * @data: data pointer to pass to the function + * @node: NUMA node to complete the work on * * Returns an async_cookie_t that may be used for checkpointing later. * Note: This function may be called from atomic or non-atomic contexts. */ -async_cookie_t async_schedule(async_func_t func, void *data) +async_cookie_t async_schedule_on(async_func_t func, void *data, int node) { - return __async_schedule(func, data, &async_dfl_domain); + return __async_schedule(func, data, &async_dfl_domain, node); } -EXPORT_SYMBOL_GPL(async_schedule); +EXPORT_SYMBOL_GPL(async_schedule_on); /** - * async_schedule_domain - schedule a function for asynchronous execution within a certain domain + * async_schedule_on_domain - schedule a function for asynchronous execution within a certain domain * @func: function to execute asynchronously * @data: data pointer to pass to the function + * @node: NUMA node to complete the work on * @domain: the domain * * Returns an async_cookie_t that may be used for checkpointing later. @@ -226,12 +240,12 @@ async_cookie_t async_schedule(async_func_t func, void *data) * synchronization domain is specified via @domain. Note: This function * may be called from atomic or non-atomic contexts. */ -async_cookie_t async_schedule_domain(async_func_t func, void *data, - struct async_domain *domain) +async_cookie_t async_schedule_on_domain(async_func_t func, void *data, int node, + struct async_domain *domain) { - return __async_schedule(func, data, domain); + return __async_schedule(func, data, domain, node); } -EXPORT_SYMBOL_GPL(async_schedule_domain); +EXPORT_SYMBOL_GPL(async_schedule_on_domain); /** * async_synchronize_full - synchronize all asynchronous function calls From patchwork Thu Sep 20 22:29:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10608815 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5E52215A6 for ; Thu, 20 Sep 2018 22:30:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 500162E002 for ; Thu, 20 Sep 2018 22:30:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 442182E00C; Thu, 20 Sep 2018 22:30:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B4F732E002 for ; Thu, 20 Sep 2018 22:30:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D1B948E0007; Thu, 20 Sep 2018 18:30:18 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CA11D8E0001; Thu, 20 Sep 2018 18:30:18 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B440F8E0007; Thu, 20 Sep 2018 18:30:18 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by kanga.kvack.org (Postfix) with ESMTP id 704DD8E0001 for ; Thu, 20 Sep 2018 18:30:18 -0400 (EDT) Received: by mail-pl1-f200.google.com with SMTP id h1-v6so341711pld.21 for ; Thu, 20 Sep 2018 15:30:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:from :to:cc:date:message-id:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=GrqJH7t83fWCDPoPPhyEVm2eAu2FCN5vFCJtp0C7hwU=; b=Lc3P7NRH4mg+iDx22IrZK+eGI9PuIQ+YEELAk6c17EkgSr4Ww2Mo95SlVVAYQL47FG cs0qdXfLc1iavoh3C6EC99TWRI1fWuwWw6Iw7pwLtwqfCDmoIwsBJ3/ty9f8D8UU6c54 R5S8GPFmDa7K1k3cf5e6tHCBID9saGL73cChldIc5h2fTTmMfweclehRNEyi3Ag/fBZN 9hl69t9JJrk/9lPXz8WCzSi+FQRdCZvmyv5RuzNu87Xsx5WQCjX6ZAsbvq+FX0r+xB99 XwfzNyZEkaCnkEb/+c9dJXCSt9kCwyloxl/llQgmUlC7eaSqbuJMNx/CROkCkFzWVOkN oAUw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: APzg51Bz/V8H4KYqt/i5dRIJ20Qh4fQZeS54db7rro6P6a0VmCImkrCP j0EC3+w4qmMw/tcMVExy+gMi9Nu78ubu0D69NYtfpUZihdCei86RH45kfb4wshGqqZutBYrIJU7 Et9jPhHnFhvhXj4ZYldTD8RuPr886p9UG56WLi4QdRN3NyEwxLSvhgr/7MkEXte8yMQ== X-Received: by 2002:a63:1a5a:: with SMTP id a26-v6mr38994085pgm.9.1537482618115; Thu, 20 Sep 2018 15:30:18 -0700 (PDT) X-Google-Smtp-Source: ANB0VdbOT8BqwgzI3ZJtDgo/scmz8mpIskeRQDRPOFo2bZe7ECJOwTQ2dRZ1LK8rpxWzVByyES4f X-Received: by 2002:a63:1a5a:: with SMTP id a26-v6mr38994037pgm.9.1537482617255; Thu, 20 Sep 2018 15:30:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1537482617; cv=none; d=google.com; s=arc-20160816; b=nwTjDXkInPmh4qJoqXPoWxjdtaamr+BMa9XNKGW7budcfs35q4TFRjGZhHKaWGngeM Yhv9AQTR4ij6Vo1A1fFIbTdYKyHMzo9EligkwI/U2jgarNRiQAZol7uXqi/K3RCHVVa1 k0CE7QoBoMUOjJiR5gQnZV1WB4vKd5RJPfxg7srHuGoKOGlmzo0C1a9nM3m3R0xc95fh 2dFRt2NQHy66rWs3azvDsgN75MPKanFb5iZ9Ilin1yXDu/X0dbEksMK1BBy8mFL3w1LR q1p1V7feskPkRvUrdzfZLXIk1U1zRI4XqYKbMM+eM/vJlKcYhLN9sdylN1RMHDEjcLiZ HKwg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject; bh=GrqJH7t83fWCDPoPPhyEVm2eAu2FCN5vFCJtp0C7hwU=; b=VaVyIOIiGn9HgiHlTQhGHRK8IVlKVuA2h34SDh5NgCSsw6PJ6x409PDp55MB8miYkr nga1jRZDecSxWqHq87wPE3x3b0zLVXITcgg2c9IyWZxojWsOr4+aubFk5GsIjatyAxnO WSo7QP//xEqDWF4i94otK6sVtUC4Vltpqe22HFMri/57EffynIUda69fMRU56TyvzgoE 3k0alswApaROdiSM2a3vx+6FFbfZ98dVRAxVLA62Yl+NfeKnrLJEvY081QP0Gr+8uing HcX3IIZcXetkKD5faI6LWNpXNaZYC9HLIXpOjV7TeP0a439/BRcDoGPm1xtMkyK+UThB d+6g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga07.intel.com (mga07.intel.com. [134.134.136.100]) by mx.google.com with ESMTPS id k8-v6si3030247plt.176.2018.09.20.15.30.16 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 20 Sep 2018 15:30:17 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.100 as permitted sender) client-ip=134.134.136.100; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 20 Sep 2018 15:30:16 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,282,1534834800"; d="scan'208";a="85287175" Received: from ahduyck-mobl.amr.corp.intel.com (HELO localhost.localdomain) ([10.7.198.152]) by orsmga003.jf.intel.com with ESMTP; 20 Sep 2018 15:29:58 -0700 Subject: [PATCH v4 5/5] nvdimm: Schedule device registration on node local to the device From: Alexander Duyck To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org Cc: pavel.tatashin@microsoft.com, mhocko@suse.com, dave.jiang@intel.com, mingo@kernel.org, dave.hansen@intel.com, jglisse@redhat.com, akpm@linux-foundation.org, logang@deltatee.com, dan.j.williams@intel.com, kirill.shutemov@linux.intel.com Date: Thu, 20 Sep 2018 15:29:57 -0700 Message-ID: <20180920222951.19464.39241.stgit@localhost.localdomain> In-Reply-To: <20180920215824.19464.8884.stgit@localhost.localdomain> References: <20180920215824.19464.8884.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch is meant to force the device registration for nvdimm devices to be closer to the actual device. This is achieved by using either the NUMA node ID of the region, or of the parent. By doing this we can have everything above the region based on the region, and everything below the region based on the nvdimm bus. One additional change I made is that we hold onto a reference to the parent while we are going through registration. By doing this we can guarantee we can complete the registration before we have the parent device removed. By guaranteeing NUMA locality I see an improvement of as high as 25% for per-node init of a system with 12TB of persistent memory. Signed-off-by: Alexander Duyck --- drivers/nvdimm/bus.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/drivers/nvdimm/bus.c b/drivers/nvdimm/bus.c index 8aae6dcc839f..ca935296d55e 100644 --- a/drivers/nvdimm/bus.c +++ b/drivers/nvdimm/bus.c @@ -487,7 +487,9 @@ static void nd_async_device_register(void *d, async_cookie_t cookie) dev_err(dev, "%s: failed\n", __func__); put_device(dev); } + put_device(dev); + put_device(dev->parent); } static void nd_async_device_unregister(void *d, async_cookie_t cookie) @@ -504,12 +506,25 @@ static void nd_async_device_unregister(void *d, async_cookie_t cookie) void __nd_device_register(struct device *dev) { + int node; + if (!dev) return; + dev->bus = &nvdimm_bus_type; + get_device(dev->parent); get_device(dev); - async_schedule_domain(nd_async_device_register, dev, - &nd_async_domain); + + /* + * For a region we can break away from the parent node, + * otherwise for all other devices we just inherit the node from + * the parent. + */ + node = is_nd_region(dev) ? to_nd_region(dev)->numa_node : + dev_to_node(dev->parent); + + async_schedule_on_domain(nd_async_device_register, dev, node, + &nd_async_domain); } void nd_device_register(struct device *dev)