From patchwork Mon Sep 10 23:43:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10594907 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9F1316CB for ; Mon, 10 Sep 2018 23:43:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8D1AC291BB for ; Mon, 10 Sep 2018 23:43:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 807A1291F4; Mon, 10 Sep 2018 23:43:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 07131291BB for ; Mon, 10 Sep 2018 23:43:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 040B28E0005; Mon, 10 Sep 2018 19:43:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id F0B2D8E0001; Mon, 10 Sep 2018 19:43:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD5688E0005; Mon, 10 Sep 2018 19:43:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 9A4E98E0001 for ; Mon, 10 Sep 2018 19:43:44 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id 2-v6so10572529plc.11 for ; Mon, 10 Sep 2018 16:43:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:subject:from:to:cc:date :message-id:in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=mK1ugpb/LCNhaDTIyoSauvWmhfcRUffOy8mLsBZJR5A=; b=KAYLBnp/KmaK6Rlhr0NBH+u1FUAjeRzC1UEvMu1UfC2jQfVtvPAXWJ8PNueajpxV/i AA8bly06qAc9/eqnZcYeNrHOPRVSwvlbpJHLWFVMgJH92f/GpDx8BxBT97tuacH0ysyR grI2JOM7lm69MQFk9hhKLUHx/7KbK/neg0FQCjKvruhfOYpipwTy3oai3XbGPfQj8Ww5 dZA+1HtsfF4gBleh2waIyGdQPAAPH9ltFua47Rzc36lXjdPXNa8dNdUqTjCsrbYSW9nc Idiifa/XtY7nBOaf4QT3VNowN4QQgMnG99gCoj/Qxn/HENboQMoWsIR3SR0BnJCDpSsb wd9g== X-Gm-Message-State: APzg51CT0hBkT40qDLDl7rg2aIClHO7Ip4UcWws50uPtHS+jXpHAYkGF mwCIn71yG7Z2RpsLoIQV3b7l6nqLMhe8Py7q3byvEoDQBlbCzVxb0+fFQ34XiGjoXB5y9zFAGYO HdfX+KgIzCvH2pKrtsrBdiAa1KcJHdU/0AX2jHzHQ5pEb94cKYPwZHxxoZWS88cm02uuO/TBN65 C0nwfhtDqSmHHI0vvWnTS8tynGujtB0dL7111UfTzXrLNlcmGQcu6j2QZNGWbbwRPeugzn/5wGZ blYRzDYGlgczhgbHtzcQ/xaas89Bx24dITSxaD/i4XrQ0ZQMzbPgq9axb4ZJ523Pz8IjKVr6GJ8 ttT8pP07Ddj/QJx4feSm91rtwBVDUxe/ERzeHT5s7WY+orRG+jc+Lsdj1ONtQkKSH4/1MPI0ldK M X-Received: by 2002:a17:902:1d4a:: with SMTP id u10-v6mr24374607plu.267.1536623024271; Mon, 10 Sep 2018 16:43:44 -0700 (PDT) X-Received: by 2002:a17:902:1d4a:: with SMTP id u10-v6mr24374580plu.267.1536623023411; Mon, 10 Sep 2018 16:43:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536623023; cv=none; d=google.com; s=arc-20160816; b=qmnCWDj5lT8BVaH5DV5VnpxlG8+hEE+ogyz9Xruarp8+o73E1j98NyjgjOf5sfJOMN aP3UCyJkoo53kimQWe5MuMzD+4ZOjocVxvp70fQyjVkt2LstLETSozFP3VFyHr50TMEk s6EIClFYTpqsluXCRZfXE2AY2IKU3ZIMLH6RDYmKB/edrQPmElautGjW3VFIzNBW/RXb LYsSLwrYC5mR/xsX61CH936QWVT61JE/D2iZt/dp2F0KWhO6NGX0LxcdEcZRh+Z2LO8+ KjNSX20bcAV1UxHWsOzpTVhRQ5AdS/mYVFsN9ZYulXV4kmXDTWYZ+hyvu+6ZuR5/OKn2 5dGA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject:dkim-signature; bh=mK1ugpb/LCNhaDTIyoSauvWmhfcRUffOy8mLsBZJR5A=; b=yfB2mVa5h2d5XQlSVKvWWwuHM7EZZHTLrXgxq36DNMaAfegpBsbbzAdFTjBTwVQIau U3QdmJfLKv2oGErkrbp58TfudCUqx8Xwm2E//oybjj+V3PZg7b3x1tmBkZ7OQuQm66ZW 4ds+FrCLavPwyHWKtaxhrSatoKJW1rkO4Ff8aZmIn9HHCoHGMfJB0meWlfh3l/Wmrpp4 JY0/UqJ6Fr8Xcu/BBLmx5K7sTZ85iWkbh5PdhI/9ZKpAu50gr/GNq28z1KVweabVLcz9 Emn21q/SJvwa9xljizO6zNw60xLNzqGHdFCWyOAwkYXxP1Ex/9M+5/fpGqWH+EwQY5Mg Cdow== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=gXMFNcn6; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id 69-v6sor3137841pla.99.2018.09.10.16.43.43 for (Google Transport Security); Mon, 10 Sep 2018 16:43:43 -0700 (PDT) Received-SPF: pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=gXMFNcn6; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=mK1ugpb/LCNhaDTIyoSauvWmhfcRUffOy8mLsBZJR5A=; b=gXMFNcn6Uhsq1Poo1ZFAUqIaW2IbJ/INlomS9949qdQEdHq+HaXa37/FmPUrchHLP9 XsgeeVjxvdF+z/23Or2XnohMb5J1aPlYTfgh8SbJMW+hh7WpRrefoNZ6Pz3bfQTrXJ4D iuc88bA0mRkvfIR7kSBdx2DcQX6dWSd5Yu3I2X6W38GHLys7ka4WnIeQqmFU0vkV6z7H z/QU+FnT7DQZz7Pwpk66ztExDL6eQLrl5xZd0STWmRrUsrppfUPJIJlYb1j68PBtxzHH qeCB2tvhUlhqZ6RP71JTIU0PpoGYdesLPYQ5tYGmQKqc6cdiJr8dQX3oj4TsNlJ2JGEA 9lCA== X-Google-Smtp-Source: ANB0VdaVXpJqV+58IBdRBFehwoJtL+fd7tHuuYHv5Y+GQ65svTqcpB8Q60RFqHEI56PxpPTtNWOn8w== X-Received: by 2002:a17:902:3c5:: with SMTP id d63-v6mr24207713pld.145.1536623022977; Mon, 10 Sep 2018 16:43:42 -0700 (PDT) Received: from localhost.localdomain (static-50-53-21-37.bvtn.or.frontiernet.net. [50.53.21.37]) by smtp.gmail.com with ESMTPSA id d12-v6sm24980289pfk.69.2018.09.10.16.43.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Sep 2018 16:43:42 -0700 (PDT) Subject: [PATCH 1/4] mm: Provide kernel parameter to allow disabling page init poisoning From: Alexander Duyck To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org Cc: pavel.tatashin@microsoft.com, mhocko@suse.com, dave.jiang@intel.com, mingo@kernel.org, dave.hansen@intel.com, jglisse@redhat.com, akpm@linux-foundation.org, logang@deltatee.com, dan.j.williams@intel.com, kirill.shutemov@linux.intel.com Date: Mon, 10 Sep 2018 16:43:41 -0700 Message-ID: <20180910234341.4068.26882.stgit@localhost.localdomain> In-Reply-To: <20180910232615.4068.29155.stgit@localhost.localdomain> References: <20180910232615.4068.29155.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Alexander Duyck On systems with a large amount of memory it can take a significant amount of time to initialize all of the page structs with the PAGE_POISON_PATTERN value. I have seen it take over 2 minutes to initialize a system with over 12GB of RAM. In order to work around the issue I had to disable CONFIG_DEBUG_VM and then the boot time returned to something much more reasonable as the arch_add_memory call completed in milliseconds versus seconds. However in doing that I had to disable all of the other VM debugging on the system. In order to work around a kernel that might have CONFIG_DEBUG_VM enabled on a system that has a large amount of memory I have added a new kernel parameter named "page_init_poison" that can be set to "off" in order to disable it. Signed-off-by: Alexander Duyck Reviewed-by: Pavel Tatashin --- Documentation/admin-guide/kernel-parameters.txt | 8 ++++++++ include/linux/page-flags.h | 8 ++++++++ mm/debug.c | 16 ++++++++++++++++ mm/memblock.c | 5 ++--- mm/sparse.c | 4 +--- 5 files changed, 35 insertions(+), 6 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 64a3bf54b974..7b21e0b9c394 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3047,6 +3047,14 @@ off: turn off poisoning (default) on: turn on poisoning + page_init_poison= [KNL] Boot-time parameter changing the + state of poisoning of page structures during early + boot. Used to verify page metadata is not accessed + prior to initialization. Available with + CONFIG_DEBUG_VM=y. + off: turn off poisoning + on: turn on poisoning (default) + panic= [KNL] Kernel behaviour on panic: delay timeout > 0: seconds before rebooting timeout = 0: wait forever diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 74bee8cecf4c..d00216cf00f8 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -162,6 +162,14 @@ static inline int PagePoisoned(const struct page *page) return page->flags == PAGE_POISON_PATTERN; } +#ifdef CONFIG_DEBUG_VM +void page_init_poison(struct page *page, size_t size); +#else +static inline void page_init_poison(struct page *page, size_t size) +{ +} +#endif + /* * Page flags policies wrt compound pages * diff --git a/mm/debug.c b/mm/debug.c index 38c926520c97..c5420422c0b5 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -175,4 +175,20 @@ void dump_mm(const struct mm_struct *mm) ); } +static bool page_init_poisoning __read_mostly = true; + +static int __init page_init_poison_param(char *buf) +{ + if (!buf) + return -EINVAL; + return strtobool(buf, &page_init_poisoning); +} +early_param("page_init_poison", page_init_poison_param); + +void page_init_poison(struct page *page, size_t size) +{ + if (page_init_poisoning) + memset(page, PAGE_POISON_PATTERN, size); +} +EXPORT_SYMBOL_GPL(page_init_poison); #endif /* CONFIG_DEBUG_VM */ diff --git a/mm/memblock.c b/mm/memblock.c index 237944479d25..a85315083b5a 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -1444,10 +1444,9 @@ void * __init memblock_virt_alloc_try_nid_raw( ptr = memblock_virt_alloc_internal(size, align, min_addr, max_addr, nid); -#ifdef CONFIG_DEBUG_VM if (ptr && size > 0) - memset(ptr, PAGE_POISON_PATTERN, size); -#endif + page_init_poison(ptr, size); + return ptr; } diff --git a/mm/sparse.c b/mm/sparse.c index 10b07eea9a6e..67ad061f7fb8 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -696,13 +696,11 @@ int __meminit sparse_add_one_section(struct pglist_data *pgdat, goto out; } -#ifdef CONFIG_DEBUG_VM /* * Poison uninitialized struct pages in order to catch invalid flags * combinations. */ - memset(memmap, PAGE_POISON_PATTERN, sizeof(struct page) * PAGES_PER_SECTION); -#endif + page_init_poison(memmap, sizeof(struct page) * PAGES_PER_SECTION); section_mark_present(ms); sparse_init_one_section(ms, section_nr, memmap, usemap); From patchwork Mon Sep 10 23:43:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10594911 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A7E5214E0 for ; Mon, 10 Sep 2018 23:43:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 962BD291BB for ; Mon, 10 Sep 2018 23:43:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8A751291F4; Mon, 10 Sep 2018 23:43:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1C547291BB for ; Mon, 10 Sep 2018 23:43:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 300718E0006; Mon, 10 Sep 2018 19:43:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2884D8E0001; Mon, 10 Sep 2018 19:43:51 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 14F928E0006; Mon, 10 Sep 2018 19:43:51 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id C92A18E0001 for ; Mon, 10 Sep 2018 19:43:50 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id f13-v6so11374655pgs.15 for ; Mon, 10 Sep 2018 16:43:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:subject:from:to:cc:date :message-id:in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=lhFr3T5RF4ix/HN80RMUA8xymXKjrz1rlCqR/lwO+vo=; b=DzoQRO0g/M04CGo1m053bDDSLGjX+qNFNkv9OOu1EhmdUEqpDL7tmBgCe3aTxQ50JW YLiWzg+Kqpw9ffsdGiFX9+rmg/egfBQHZQ3vCCf/Uh9IUTI7eZFfdB+lh5tALZ7d1aqR tH9IE5sXWH1LDblFr+/ZZOMaO/6ur6Wqv+mlpus0Zdm1Y8z9PWdk2hxrm+E/n0BioCjI Zo0Tgt7bnTQ2+TwinsM/jrbnlsa5gZiU1yFZkl0DIs+2eMZ9ZLQLS+xF8lojOScdgxdk VrX6yzcCopl5OAjqp0vlg/OXDYdrdBZKTheyl3OnioVnnvg9R3UjAOgjNPdDdT2QFZ+H oXyw== X-Gm-Message-State: APzg51B4v5Q5JquY+ff5j7mzy068WSA3/N+62SKCh47ysc019Jd2gAf+ E8mL3CzSBWNHjwJhq7yAWm5Jwq2PrqtCqHsCpo3MhMhN2UNKj8lwcz4/APEoaXMPlcnqZ/yEN7b TQrVBG1pyrXlCzvXhUUo4nWbpTQ/SuMSeuehf0moxoVFmHu4iCv923NYxtyAwwOELLDDLBsdIf1 VhMX0LaVOWIblzM5UUxM4KWFe7S2JTWWlzSRSEGa+z7DX/6/dsOyTe9SrtWFOQPlDHF2j31q1GD SHxQjAu3ONxoHbm/iPj/src+zix58KrXQ1X+FXj3kBRDbNBH8pBQr93KCxjkjWBXrGyY5wKMrHD ij2aUkwY24+T0J4wtrZzaLaweUF9fA6rd0jAkXZa2PYWfkaeUngD/DmG0dMKF3SQQOUT6Dv0Pn+ Q X-Received: by 2002:a63:610:: with SMTP id 16-v6mr24746687pgg.96.1536623030483; Mon, 10 Sep 2018 16:43:50 -0700 (PDT) X-Received: by 2002:a63:610:: with SMTP id 16-v6mr24746655pgg.96.1536623029707; Mon, 10 Sep 2018 16:43:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536623029; cv=none; d=google.com; s=arc-20160816; b=pCn8FlYNxKvR70F9ebw9Xj391MsTKEQL7kTDCgh9Oo8VpaI2NhxlgAMlVN3HDytX92 GEgOwGTWm+dB0i3uOyPBaxyfBvFaMSG48aZRFrpDHMoidM6X3kEDqnf2AX53lijE6+Qe NO3Xa1utsQInSzjFncff3k2JgwmCF/5a/wduVhPqtOGLCV5eqDHXNbIu3LmzSK1CmU2b MNLi2mRjYefGPPCwTAsIOv+FsvJk1bOSu3WhNhntZ3a7xP7T2YTwqjZj5Ag4y9XcAlyl vHk3LZkfb3JHTuxYJZJvfJP4IFC7QDXzOPB9g+qPqlyorZGJNyY9gNkI3mzbMHBr+ARs p+LQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject:dkim-signature; bh=lhFr3T5RF4ix/HN80RMUA8xymXKjrz1rlCqR/lwO+vo=; b=ExNk0zgP7nwi5Qtvpzw1SKIPvqqPZAdh/5eJ0juU7YCYt/X3curfPNHjpm6ReWm99X SB/jkiXIbtYptzdbT5xM/p3VZvbVd6OMmHeYfQ3CKThfBVmEE+ewAbWOh7gma0UFeZKs xFEEOtjIxsQGeuCG7lyiu6wLj7HZScywkWiPTxK+MRLCKkUi74gzS599HKT+mn/3WuEK QzffSsxY39QUZ6PWERfBFfwoW3ZAlZ9/Vqb2lBx+vh/JKkkEMfrLW2PIBg6MJRbKHdUr OGsUk3xjr+HdAwCEKo7KwNZiLvEeFX3cIQiVdQNmKvJFFCL2u5skRto4qh5ypoG6ct9h 4ntw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=M5jUdHnp; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id t8-v6sor3072510plz.147.2018.09.10.16.43.49 for (Google Transport Security); Mon, 10 Sep 2018 16:43:49 -0700 (PDT) Received-SPF: pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=M5jUdHnp; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=lhFr3T5RF4ix/HN80RMUA8xymXKjrz1rlCqR/lwO+vo=; b=M5jUdHnplNrgx3jtLSwG+/FiSgIS5lu66K/26ReK6zbipcdAmf2HbSKpXwL++aiju+ BBPLh+5tfixgC9jksDYNDyOzwlV9qKrTrb3wq1Avw2EtcbsmSbb5ib0kltUcSCuZkk1L tUJM4vwwtCW1eWbbOxG5/eM08A/nR81VfKIwm1GYNN9jiWxeqdtUkTee2LITfK3kV+oX bQyuH29EMKTNsd9GC774DuOq9xIPkqxxgsQSh26nIvJHbXrSG0u2q385g0pcmvaJKEfQ 9nFUXSV5uLTUIsICYY0eagbcIS8YZ9EPAqXl4PMy2TeTZ2GAVKo9jxcEUWk3UUtKBzn5 PotA== X-Google-Smtp-Source: ANB0VdZ9nofLh3IK/Vq5oODEvu42HmX3XMswWPrs5u5Htg1ryZEmW2HxTof/tUVek3Nw+LtN1MGgcw== X-Received: by 2002:a17:902:344:: with SMTP id 62-v6mr24533133pld.164.1536623029274; Mon, 10 Sep 2018 16:43:49 -0700 (PDT) Received: from localhost.localdomain (static-50-53-21-37.bvtn.or.frontiernet.net. [50.53.21.37]) by smtp.gmail.com with ESMTPSA id v20-v6sm33973962pfk.12.2018.09.10.16.43.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Sep 2018 16:43:48 -0700 (PDT) Subject: [PATCH 2/4] mm: Create non-atomic version of SetPageReserved for init use From: Alexander Duyck To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org Cc: pavel.tatashin@microsoft.com, mhocko@suse.com, dave.jiang@intel.com, mingo@kernel.org, dave.hansen@intel.com, jglisse@redhat.com, akpm@linux-foundation.org, logang@deltatee.com, dan.j.williams@intel.com, kirill.shutemov@linux.intel.com Date: Mon, 10 Sep 2018 16:43:48 -0700 Message-ID: <20180910234348.4068.92164.stgit@localhost.localdomain> In-Reply-To: <20180910232615.4068.29155.stgit@localhost.localdomain> References: <20180910232615.4068.29155.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Alexander Duyck It doesn't make much sense to use the atomic SetPageReserved at init time when we are using memset to clear the memory and manipulating the page flags via simple "&=" and "|=" operations in __init_single_page. This patch adds a non-atomic version __SetPageReserved that can be used during page init and shows about a 10% improvement in initialization times on the systems I have available for testing. On those systems I saw initialization times drop from around 35 seconds to around 32 seconds to initialize a 3TB block of persistent memory. I tried adding a bit of documentation based on commit ("mm, memory_hotplug: do not associate hotadded memory to zones until online"). Ideally the reserved flag should be set earlier since there is a brief window where the page is initialization via __init_single_page and we have not set the PG_Reserved flag. I'm leaving that for a future patch set as that will require a more significant refactor. Acked-by: Michal Hocko Signed-off-by: Alexander Duyck Reviewed-by: Pavel Tatashin --- include/linux/page-flags.h | 1 + mm/page_alloc.c | 17 +++++++++++++++-- 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index d00216cf00f8..1b1f8e0378ae 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -300,6 +300,7 @@ static inline void page_init_poison(struct page *page, size_t size) PAGEFLAG(Reserved, reserved, PF_NO_COMPOUND) __CLEARPAGEFLAG(Reserved, reserved, PF_NO_COMPOUND) + __SETPAGEFLAG(Reserved, reserved, PF_NO_COMPOUND) PAGEFLAG(SwapBacked, swapbacked, PF_NO_TAIL) __CLEARPAGEFLAG(SwapBacked, swapbacked, PF_NO_TAIL) __SETPAGEFLAG(SwapBacked, swapbacked, PF_NO_TAIL) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 89d2a2ab3fe6..a9b095a72fd9 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1231,7 +1231,12 @@ void __meminit reserve_bootmem_region(phys_addr_t start, phys_addr_t end) /* Avoid false-positive PageTail() */ INIT_LIST_HEAD(&page->lru); - SetPageReserved(page); + /* + * no need for atomic set_bit because the struct + * page is not visible yet so nobody should + * access it yet. + */ + __SetPageReserved(page); } } } @@ -5517,8 +5522,16 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone, not_early: page = pfn_to_page(pfn); __init_single_page(page, pfn, zone, nid); + + /* + * Mark page reserved as it will need to wait for onlining + * phase for it to be fully associated with a zone. + * + * We can use the non-atomic __set_bit operation for setting + * the flag as we are still initializing the pages. + */ if (context == MEMMAP_HOTPLUG) - SetPageReserved(page); + __SetPageReserved(page); /* * Mark the block movable so that blocks are reserved for From patchwork Mon Sep 10 23:43:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10594915 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 868C96CB for ; Mon, 10 Sep 2018 23:43:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 730E1291BB for ; Mon, 10 Sep 2018 23:43:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6705F291F4; Mon, 10 Sep 2018 23:43:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BCFD6291BB for ; Mon, 10 Sep 2018 23:43:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C4E1B8E0007; Mon, 10 Sep 2018 19:43:57 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BD6F68E0001; Mon, 10 Sep 2018 19:43:57 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A79618E0007; Mon, 10 Sep 2018 19:43:57 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id 61DC98E0001 for ; Mon, 10 Sep 2018 19:43:57 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id 3-v6so10636084plq.6 for ; Mon, 10 Sep 2018 16:43:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:subject:from:to:cc:date :message-id:in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=JQupwLl2rKeiSvMZVMaWUdsg3iLwHjoTCWaGtKuBCO0=; b=kG+zBLmJeDWl6QxqEF2fTkBzRVPrpg2q3pq0SwLhmNKDXW6/JYDPsKORBkAVXQWWnL TdhtBQNyI1drarfMsoAhoAiWG8z/TIke8qjVj8J40OgnmZfMovjIiakqvh2+hZYJtSbU hEd8M//38xWoqLQqTLRHbDnmXg5rti56T1fMh92M7GOg3a2IaNWZbfCtYEgBGKRJiNb7 ASgJ3HUZvH+C0rpqFmozhe61RphEVS9HnXHLUu81pxgg8ygWKnw6Tsl5WRIyfeRKM497 buL6Q+pCTnOYBe7iixl42bMpM4xzSEy9d1h41kNNEqbmCyU5SxPAuydKoD3buQD6JrJp QoVQ== X-Gm-Message-State: APzg51Cdl8Wm/g5M3ofbeRMiI5zMFdAkKUdqXMNhEiwLbRK2oe+d5C9y zHNP9onbUx5LlCWov7gKyIq10XCI5FM2PDHqmhMq8nGnClIaEJcRVPHLVON39S7ThjD0+6lFU2d ZZJRTCfWuWhzDBPSrLClry7hVmq2ts/HdaFmg+oV+GMZD3ZqOQp56YqktHIHNeyvTfrdK3gQlyr hYTu+oturjGxWSAcwG34ML+U5HDm2DX+wUc3iPmN3cEvS6ZWQMbECQfQscduVXYL5Hep+WBHDzg ZDciIKjK43wBrK4nQI/nvyQtOUpEQIPTOrfIsxEWjnyQw9gXOz5BQ3Gu24l2F+hCNhCcCa9WXdH MW/mfKDA3/FN4yzb1Ik9e7oSP+OyIx+3lIz8thBGNoc/McyOdKPSHzKhavJ5FWWqDmaxOajIgfD i X-Received: by 2002:a17:902:740a:: with SMTP id g10-v6mr24828047pll.22.1536623037048; Mon, 10 Sep 2018 16:43:57 -0700 (PDT) X-Received: by 2002:a17:902:740a:: with SMTP id g10-v6mr24828004pll.22.1536623035962; Mon, 10 Sep 2018 16:43:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536623035; cv=none; d=google.com; s=arc-20160816; b=R3whBi7t0yiKtPplncwGkE+UpgOl+EBUFAyfPbgrZTmVYlTK0yaJ6CkI9rMVufzeeg IC1dxogKejtWbetcZrRtLee78GtagHKDEGyYrUCl5d6qzt4v7M249mijkCRhzJI9fvaX bp+Ee7nsU0gDXKrBzi5Ss3/Yn4k/nnDz6I8d/XhrceESMCYaZIdOX7lihDkjDIXxkkXv tF5R+3cLyHaGSkZV4NIb8fQ7ZQ0m2H5muv0b5kTtKsnbGeZ+G0Tce8hr2asC4r8xfaAu H61Up5PVSau5O0anfObG+y8WAEHXq5KhvHA2ONLtnWmxVvFMpC4MB+viL8Oz5n/NEpqL k51A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject:dkim-signature; bh=JQupwLl2rKeiSvMZVMaWUdsg3iLwHjoTCWaGtKuBCO0=; b=xHW/XGlbiUGPftrCHS53kn0TkHzA575uGzt+pqHzjtvIPh1NO3GDACk6zMHcOIaRfn +f7D+Jz1p7/BBWageFvyrPt6spnQLNf40zrlUxyU19zdp/49P8y7FmIO0j8ku/WGQqqW EqrIWkcNQnFhN6vq3hCS5ElBIiERcg9SnjGMBzTeJW6CJ91b9mYYscW1giUPLxtoQVqY FDrPx77HL3jl9BJJ0563ilwhA1A454DmuvOtgORk+qlX7OqsAnexYcqqZ5auj8SbEPO8 TovIVwCPtAF/hrv93to7cZDJwBTxTLz4xf0g867zeH+8giZu0KQmke6UjHJpPJasWhES I2EA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=AT9eVUmg; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id x21-v6sor2360929pgx.300.2018.09.10.16.43.55 for (Google Transport Security); Mon, 10 Sep 2018 16:43:55 -0700 (PDT) Received-SPF: pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=AT9eVUmg; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=JQupwLl2rKeiSvMZVMaWUdsg3iLwHjoTCWaGtKuBCO0=; b=AT9eVUmgpdmWtSo+Fxw9TA03wtkozOaVxProH2Taj3iWVr/og1kR7XXfBECGTDYY+s Y0zD1mkjqd5fnLSY1Qg0z8MCXG2L3oWzYZ8aktgat2oa1XJQ7vBxCDZUhymTKhAV12hB uIvW/0hQ8EhvyKNM2Z997gfjitnq2c39IP6a4ZWxQvy+ho/VuGIzyhbR5idZ22Dc8Xyk hP2DyK6qcsLDjJh1zTraqqPHZpC/4Y/uUJLp7RqwNcCGF7N65+nfG0pWqX0ZAGveggjP cq4G+k8LUg2g0MZp+jZ3aRAU9Hi8zZRMIoRRy4nzYFX7GnUawPeh5lKd6oi/Cnfl8Sjg MOPQ== X-Google-Smtp-Source: ANB0VdajruHbt2gGQL2PIS6t3FblvkV2GDRVuThV7Bt+mSUv+GUqIFVQziEeAQLM4CcmkVf0yz6DAg== X-Received: by 2002:a63:a112:: with SMTP id b18-v6mr25197451pgf.384.1536623035513; Mon, 10 Sep 2018 16:43:55 -0700 (PDT) Received: from localhost.localdomain (static-50-53-21-37.bvtn.or.frontiernet.net. [50.53.21.37]) by smtp.gmail.com with ESMTPSA id i75-v6sm25134928pgc.20.2018.09.10.16.43.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Sep 2018 16:43:55 -0700 (PDT) Subject: [PATCH 3/4] mm: Defer ZONE_DEVICE page initialization to the point where we init pgmap From: Alexander Duyck To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org Cc: pavel.tatashin@microsoft.com, mhocko@suse.com, dave.jiang@intel.com, mingo@kernel.org, dave.hansen@intel.com, jglisse@redhat.com, akpm@linux-foundation.org, logang@deltatee.com, dan.j.williams@intel.com, kirill.shutemov@linux.intel.com Date: Mon, 10 Sep 2018 16:43:54 -0700 Message-ID: <20180910234354.4068.65260.stgit@localhost.localdomain> In-Reply-To: <20180910232615.4068.29155.stgit@localhost.localdomain> References: <20180910232615.4068.29155.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Alexander Duyck The ZONE_DEVICE pages were being initialized in two locations. One was with the memory_hotplug lock held and another was outside of that lock. The problem with this is that it was nearly doubling the memory initialization time. Instead of doing this twice, once while holding a global lock and once without, I am opting to defer the initialization to the one outside of the lock. This allows us to avoid serializing the overhead for memory init and we can instead focus on per-node init times. One issue I encountered is that devm_memremap_pages and hmm_devmmem_pages_create were initializing only the pgmap field the same way. One wasn't initializing hmm_data, and the other was initializing it to a poison value. Since this is something that is exposed to the driver in the case of hmm I am opting for a third option and just initializing hmm_data to 0 since this is going to be exposed to unknown third party drivers. Signed-off-by: Alexander Duyck --- include/linux/mm.h | 2 + kernel/memremap.c | 24 +++++--------- mm/hmm.c | 12 ++++--- mm/page_alloc.c | 89 +++++++++++++++++++++++++++++++++++++++++++++++++++- 4 files changed, 105 insertions(+), 22 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a61ebe8ad4ca..47b440bb3050 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -848,6 +848,8 @@ static inline bool is_zone_device_page(const struct page *page) { return page_zonenum(page) == ZONE_DEVICE; } +extern void memmap_init_zone_device(struct zone *, unsigned long, + unsigned long, struct dev_pagemap *); #else static inline bool is_zone_device_page(const struct page *page) { diff --git a/kernel/memremap.c b/kernel/memremap.c index 5b8600d39931..d0c32e473f82 100644 --- a/kernel/memremap.c +++ b/kernel/memremap.c @@ -175,10 +175,10 @@ void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap) struct vmem_altmap *altmap = pgmap->altmap_valid ? &pgmap->altmap : NULL; struct resource *res = &pgmap->res; - unsigned long pfn, pgoff, order; + struct dev_pagemap *conflict_pgmap; pgprot_t pgprot = PAGE_KERNEL; + unsigned long pgoff, order; int error, nid, is_ram; - struct dev_pagemap *conflict_pgmap; align_start = res->start & ~(SECTION_SIZE - 1); align_size = ALIGN(res->start + resource_size(res), SECTION_SIZE) @@ -256,19 +256,13 @@ void *devm_memremap_pages(struct device *dev, struct dev_pagemap *pgmap) if (error) goto err_add_memory; - for_each_device_pfn(pfn, pgmap) { - struct page *page = pfn_to_page(pfn); - - /* - * ZONE_DEVICE pages union ->lru with a ->pgmap back - * pointer. It is a bug if a ZONE_DEVICE page is ever - * freed or placed on a driver-private list. Seed the - * storage with LIST_POISON* values. - */ - list_del(&page->lru); - page->pgmap = pgmap; - percpu_ref_get(pgmap->ref); - } + /* + * Initialization of the pages has been deferred until now in order + * to allow us to do the work while not holding the hotplug lock. + */ + memmap_init_zone_device(&NODE_DATA(nid)->node_zones[ZONE_DEVICE], + align_start >> PAGE_SHIFT, + align_size >> PAGE_SHIFT, pgmap); devm_add_action(dev, devm_memremap_pages_release, pgmap); diff --git a/mm/hmm.c b/mm/hmm.c index c968e49f7a0c..774d684fa2b4 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -1024,7 +1024,6 @@ static int hmm_devmem_pages_create(struct hmm_devmem *devmem) resource_size_t key, align_start, align_size, align_end; struct device *device = devmem->device; int ret, nid, is_ram; - unsigned long pfn; align_start = devmem->resource->start & ~(PA_SECTION_SIZE - 1); align_size = ALIGN(devmem->resource->start + @@ -1109,11 +1108,14 @@ static int hmm_devmem_pages_create(struct hmm_devmem *devmem) align_size >> PAGE_SHIFT, NULL); mem_hotplug_done(); - for (pfn = devmem->pfn_first; pfn < devmem->pfn_last; pfn++) { - struct page *page = pfn_to_page(pfn); + /* + * Initialization of the pages has been deferred until now in order + * to allow us to do the work while not holding the hotplug lock. + */ + memmap_init_zone_device(&NODE_DATA(nid)->node_zones[ZONE_DEVICE], + align_start >> PAGE_SHIFT, + align_size >> PAGE_SHIFT, &devmem->pagemap); - page->pgmap = &devmem->pagemap; - } return 0; error_add_memory: diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a9b095a72fd9..81a3fd942c45 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -5454,6 +5454,83 @@ void __ref build_all_zonelists(pg_data_t *pgdat) #endif } +#ifdef CONFIG_ZONE_DEVICE +void __ref memmap_init_zone_device(struct zone *zone, unsigned long pfn, + unsigned long size, + struct dev_pagemap *pgmap) +{ + struct pglist_data *pgdat = zone->zone_pgdat; + unsigned long zone_idx = zone_idx(zone); + unsigned long end_pfn = pfn + size; + unsigned long start = jiffies; + int nid = pgdat->node_id; + unsigned long nr_pages; + + if (WARN_ON_ONCE(!pgmap || !is_dev_zone(zone))) + return; + + /* + * The call to memmap_init_zone should have already taken care + * of the pages reserved for the memmap, so we can just jump to + * the end of that region and start processing the device pages. + */ + if (pgmap->altmap_valid) { + struct vmem_altmap *altmap = &pgmap->altmap; + + pfn = altmap->base_pfn + vmem_altmap_offset(altmap); + } + + /* Record the number of pages we are about to initialize */ + nr_pages = end_pfn - pfn; + + for (; pfn < end_pfn; pfn++) { + struct page *page = pfn_to_page(pfn); + + __init_single_page(page, pfn, zone_idx, nid); + + /* + * Mark page reserved as it will need to wait for onlining + * phase for it to be fully associated with a zone. + * + * We can use the non-atomic __set_bit operation for setting + * the flag as we are still initializing the pages. + */ + __SetPageReserved(page); + + /* + * ZONE_DEVICE pages union ->lru with a ->pgmap back + * pointer and hmm_data. It is a bug if a ZONE_DEVICE + * page is ever freed or placed on a driver-private list. + */ + page->pgmap = pgmap; + page->hmm_data = 0; + + /* + * Mark the block movable so that blocks are reserved for + * movable at startup. This will force kernel allocations + * to reserve their blocks rather than leaking throughout + * the address space during boot when many long-lived + * kernel allocations are made. + * + * bitmap is created for zone's valid pfn range. but memmap + * can be created for invalid pages (for alignment) + * check here not to call set_pageblock_migratetype() against + * pfn out of zone. + * + * Please note that MEMMAP_HOTPLUG path doesn't clear memmap + * because this is done early in sparse_add_one_section + */ + if (!(pfn & (pageblock_nr_pages - 1))) { + set_pageblock_migratetype(page, MIGRATE_MOVABLE); + cond_resched(); + } + } + + pr_info("%s initialised, %lu pages in %ums\n", dev_name(pgmap->dev), + nr_pages, jiffies_to_msecs(jiffies - start)); +} + +#endif /* * Initially all pages are reserved - free ones are freed * up by free_all_bootmem() once the early boot process is @@ -5477,10 +5554,18 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone, /* * Honor reservation requested by the driver for this ZONE_DEVICE - * memory + * memory. We limit the total number of pages to initialize to just + * those that might contain the memory mapping. We will defer the + * ZONE_DEVICE page initialization until after we have released + * the hotplug lock. */ - if (altmap && start_pfn == altmap->base_pfn) + if (altmap && start_pfn == altmap->base_pfn) { start_pfn += altmap->reserve; + end_pfn = altmap->base_pfn + + vmem_altmap_offset(altmap); + } else if (zone == ZONE_DEVICE) { + end_pfn = start_pfn; + } for (pfn = start_pfn; pfn < end_pfn; pfn++) { /* From patchwork Mon Sep 10 23:44:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10594921 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5B3DE6CB for ; Mon, 10 Sep 2018 23:44:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 49012291BB for ; Mon, 10 Sep 2018 23:44:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3D043291F4; Mon, 10 Sep 2018 23:44:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5C989291BB for ; Mon, 10 Sep 2018 23:44:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6F6F18E0008; Mon, 10 Sep 2018 19:44:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 67FAF8E0001; Mon, 10 Sep 2018 19:44:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5217D8E0008; Mon, 10 Sep 2018 19:44:04 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id 0DF848E0001 for ; Mon, 10 Sep 2018 19:44:04 -0400 (EDT) Received: by mail-pl1-f198.google.com with SMTP id 2-v6so10572786plc.11 for ; Mon, 10 Sep 2018 16:44:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:subject:from:to:cc:date :message-id:in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=MJv9XTQcitO3gy5LNlz5Iv9gvth5EHKXzc6g36JI3Fc=; b=byVfWoXOvooddAYHsvZt5KPfd48mAJq7EVZNPTXt5kYErCbFJ6FE5YzkFQRDFTwpp2 Op66bmVjSk1PCQgY4oK3vw866LZkfAT2Oo8UO2ZTRZbhBKL6eMpRbNnCF8ASqk7mAZkj YOp9ImvYL0UQeurt9ewo7r2ER0csseFHhCCpGPK2HghF0R/YlMBny4VF5cyEx9Bhqmb9 555H8Din5mACZaU1hrf0EarB8cXxHgYtZnzNi8sPSCwi97kk/UVFQg0htXtPIjFO9IR/ Mphr8agfRgGSbsaeRU5VcHNRWS21/tXx4ixTUnbX+bogB70yduhNvEx/pHAF5NcaFds2 bs/g== X-Gm-Message-State: APzg51BalOnB5LDc0hM+jcAAnErhArLkKx87fv14ItJJIMdTKHPmadZ8 Ckulgg3kIfV1V+2zZd/U0j43keeD68TdxL0YBuEtvUNDurwgwz634u1jA40/IB2LAI9N3pTYdCE EdkCmcMazD9vddYEPnLYy4jarQqkMro6b9VZIRuZC7yCh4kZaG1ZlauX+6hDNBbTKP3ndNAsi+P EZaHPjjmukBpmaWgeIvw9NMp8UfLT7lftteLKfJ6kwHDRmZ6hfBjMMoAJ9hARtZbevfIXT5RNtK vdhdi6hLDicbNIJq4gkCGxSf8Cpp7QIPFhsC43zHB4YKs9JqSJIZYPF7vTO+55aAHd4NQoBeawL tY130+E2/a4ZMMVzpvLlSiASKePKBY/deyEG3WtEILOiyy9GD0WRrz99BGsdnJRFyNN2hjmuDL0 r X-Received: by 2002:a63:ff1f:: with SMTP id k31-v6mr24770616pgi.20.1536623043718; Mon, 10 Sep 2018 16:44:03 -0700 (PDT) X-Received: by 2002:a63:ff1f:: with SMTP id k31-v6mr24770559pgi.20.1536623042302; Mon, 10 Sep 2018 16:44:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536623042; cv=none; d=google.com; s=arc-20160816; b=dmZZHI4c5Dxb8m9Uz6Y65PhlQTcTmDWzwGw0sz1YO7sGqW6knLjAEi28exoBEzfLTn wGba+4AboioYsNUuzzc2ASPVfarxiWqSS5eTJIgpgP+XuxoBrGmoAFK5M5lqHYAQWoum 1TrAtO8ezB8aF9hvzxObWRhsjTInR8S9CZZrWiVFXLi87F6FuKV1HxvfhDILgaBj+di2 HpBYUoiolhWzQVSg/ECxCDVTRbLCO5Y7cXWBh8Vde7kitVqHcYuvoPTHxHeg0pk6Rlg9 ohm84FuZN5hxsA8cckC+5kRbiGHBhSQkItZwAij1sY5xw2Z5FEvC2QEc+zyrh/d34yk7 OH2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject:dkim-signature; bh=MJv9XTQcitO3gy5LNlz5Iv9gvth5EHKXzc6g36JI3Fc=; b=r0uHMN9QQzdtRKtTEI08377mRuN/bUhC2WlgRilxfWGHgnJHkLqM9GmHSn7qm5rsAj cTOO2hU0rInY3Av72Py1XlrZ73afl7Y6xQv07002BxotykmpgDIxCRkQkvuv8Ma+1la6 36pjvj57iy8r+pd8M7TshRVPV6KFacrhrzLfuwBiM4/B8V4AGIRVz/yvRyIEvEJxj+LB qDYZ6AebRfrcooVYBrL4Zn6y5mye3u5jQ6q3AsUMkcg0gzmJhvJZfg8N2xyzDB/THQJJ lwmbAeFP/T3N8oaWMD01p5yK7DmpV7j52YiE7+Z558a095a5gb8XGiZ7A2povjZ68Ye5 ZRQw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="PO+/yHzK"; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id u9-v6sor3090814plz.78.2018.09.10.16.44.02 for (Google Transport Security); Mon, 10 Sep 2018 16:44:02 -0700 (PDT) Received-SPF: pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b="PO+/yHzK"; spf=pass (google.com: domain of alexander.duyck@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=alexander.duyck@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=MJv9XTQcitO3gy5LNlz5Iv9gvth5EHKXzc6g36JI3Fc=; b=PO+/yHzKVe6BrtyAYDCrPoH1MS/MgG2SdH2MWE4Pu16baBk2jwxUqndfwe7rrtFeej oNi0xoLwWDoh/gd29YEKhez5Fy1B0bGTUGlQS4awi43cRSQ8iOBoIPq5oivMRFj72UY5 f/VQa99Utf8IKzhwwIc7PqOHfuuG7xBmEB/exPT+8K0id8EJdHbVG4JJatsFCVZDMBi5 2gMw6sr3IyU1oH20TXT94TN7u7jJ3We/C7qEycP6m4tSiuK2GvyPZIZu3sCMOBGm/r0U e8ToHPOnpYI6rp3t/sHjxch8Nt0PHUEgdYOTrCXwZ1s33o4RIGLboXtgZK6qo7kLre/9 vMFw== X-Google-Smtp-Source: ANB0Vdae3zS05W6i/7y646GBZXtkkHI+KeQ/8novIzql8yYtiMRq7VA37016jMUlgSw8jVTKp/3lWw== X-Received: by 2002:a17:902:7402:: with SMTP id g2-v6mr24285976pll.321.1536623041876; Mon, 10 Sep 2018 16:44:01 -0700 (PDT) Received: from localhost.localdomain (static-50-53-21-37.bvtn.or.frontiernet.net. [50.53.21.37]) by smtp.gmail.com with ESMTPSA id u184-v6sm21581039pgd.46.2018.09.10.16.44.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Sep 2018 16:44:01 -0700 (PDT) Subject: [PATCH 4/4] nvdimm: Trigger the device probe on a cpu local to the device From: Alexander Duyck To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org Cc: pavel.tatashin@microsoft.com, mhocko@suse.com, dave.jiang@intel.com, mingo@kernel.org, dave.hansen@intel.com, jglisse@redhat.com, akpm@linux-foundation.org, logang@deltatee.com, dan.j.williams@intel.com, kirill.shutemov@linux.intel.com Date: Mon, 10 Sep 2018 16:44:00 -0700 Message-ID: <20180910234400.4068.15541.stgit@localhost.localdomain> In-Reply-To: <20180910232615.4068.29155.stgit@localhost.localdomain> References: <20180910232615.4068.29155.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Alexander Duyck This patch is based off of the pci_call_probe function used to initialize PCI devices. The general idea here is to move the probe call to a location that is local to the memory being initialized. By doing this we can shave significant time off of the total time needed for initialization. With this patch applied I see a significant reduction in overall init time as without it the init varied between 23 and 37 seconds to initialize a 3GB node. With this patch applied the variance is only between 23 and 26 seconds to initialize each node. I hope to refine this further in the future by combining this logic into the async_schedule_domain code that is already in use. By doing that it would likely make this functionality redundant. Signed-off-by: Alexander Duyck Reviewed-by: Pavel Tatashin --- drivers/nvdimm/bus.c | 45 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/drivers/nvdimm/bus.c b/drivers/nvdimm/bus.c index 8aae6dcc839f..5b73953176b1 100644 --- a/drivers/nvdimm/bus.c +++ b/drivers/nvdimm/bus.c @@ -27,6 +27,7 @@ #include #include #include +#include #include "nd-core.h" #include "nd.h" #include "pfn.h" @@ -90,6 +91,48 @@ static void nvdimm_bus_probe_end(struct nvdimm_bus *nvdimm_bus) nvdimm_bus_unlock(&nvdimm_bus->dev); } +struct nvdimm_drv_dev { + struct nd_device_driver *nd_drv; + struct device *dev; +}; + +static long __nvdimm_call_probe(void *_nddd) +{ + struct nvdimm_drv_dev *nddd = _nddd; + struct nd_device_driver *nd_drv = nddd->nd_drv; + + return nd_drv->probe(nddd->dev); +} + +static int nvdimm_call_probe(struct nd_device_driver *nd_drv, + struct device *dev) +{ + struct nvdimm_drv_dev nddd = { nd_drv, dev }; + int rc, node, cpu; + + /* + * Execute driver initialization on node where the device is + * attached. This way the driver will be able to access local + * memory instead of having to initialize memory across nodes. + */ + node = dev_to_node(dev); + + cpu_hotplug_disable(); + + if (node < 0 || node >= MAX_NUMNODES || !node_online(node)) + cpu = nr_cpu_ids; + else + cpu = cpumask_any_and(cpumask_of_node(node), cpu_online_mask); + + if (cpu < nr_cpu_ids) + rc = work_on_cpu(cpu, __nvdimm_call_probe, &nddd); + else + rc = __nvdimm_call_probe(&nddd); + + cpu_hotplug_enable(); + return rc; +} + static int nvdimm_bus_probe(struct device *dev) { struct nd_device_driver *nd_drv = to_nd_device_driver(dev->driver); @@ -104,7 +147,7 @@ static int nvdimm_bus_probe(struct device *dev) dev->driver->name, dev_name(dev)); nvdimm_bus_probe_start(nvdimm_bus); - rc = nd_drv->probe(dev); + rc = nvdimm_call_probe(nd_drv, dev); if (rc == 0) nd_region_probe_success(nvdimm_bus, dev); else