From patchwork Fri Nov 30 21:53:23 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10707271 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 327D614E2 for ; Fri, 30 Nov 2018 21:53:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2311630625 for ; Fri, 30 Nov 2018 21:53:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 178EA30065; Fri, 30 Nov 2018 21:53:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 779FD30620 for ; Fri, 30 Nov 2018 21:53:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3188A6B5A79; Fri, 30 Nov 2018 16:53:26 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2C89F6B5A7A; Fri, 30 Nov 2018 16:53:26 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 190976B5A7B; Fri, 30 Nov 2018 16:53:26 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id C7E9A6B5A79 for ; Fri, 30 Nov 2018 16:53:25 -0500 (EST) Received: by mail-pf1-f200.google.com with SMTP id f69so5520200pff.5 for ; Fri, 30 Nov 2018 13:53:25 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:from :to:cc:date:message-id:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=NJWMWnGqc5wXx77X2p+emxeUcX1MEi7E0F8HR0yF8tM=; b=fTT09+14wE7WERqqFpaLG4F04ZMmz5AWPf+QmlTrNkTsU1KX5AWG5gakEim/AEk1YI ZdjN1yJIPRAkUoMHgeGMR+5xiXlmupxP5dof9wAHJO7iJg/7FrRk2csQXgbh2+W3sDFP k0da7aKOb7IB5EqW7IArPDXIGr19rIwtHhfruNAs5XNULXe6ZYXW47B2zDe8Nx85Cxyk WaLrcYN0n7Iwp7LZk4IXDNcr+OKE9TA4sgO/pdHofD0svrgEQjgm+8BEkxN13xxAm1Ck +5hDJkHJVpwrboHHnM9UpiS4y8z3/NsE10dvBw8C1FbxetyOBBaPVG7CQ5h4ubqwnyzU lEtw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AA+aEWYLIyjDBQvpGCHaYOOpRycyx/yGiBQisIdGFUS/6UeRuTGKxVqi GpJgVrIfJDrku6iRAZDtJW1LiCThYbUpuGIJ07XD9c0M9YSny/hlMiFcoyoLFN7Mxnlf4rugxhj M4vmDbuHmeyeMT9nc2Vxkuad0eDKkkaN320Z9ZnmlnRS955O77omuwTZNwCnosg8sEA== X-Received: by 2002:a17:902:f01:: with SMTP id 1mr7016757ply.143.1543614805477; Fri, 30 Nov 2018 13:53:25 -0800 (PST) X-Google-Smtp-Source: AFSGD/X4JvRKSGQuGcUOSYYMoKccKyZhtf00ch/PQ0Jy2k+2Q45ekQulJBB7w4l7B/3wbhlB2F5f X-Received: by 2002:a17:902:f01:: with SMTP id 1mr7016713ply.143.1543614804482; Fri, 30 Nov 2018 13:53:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1543614804; cv=none; d=google.com; s=arc-20160816; b=y29hZ83gAst9DiH9PPHdPvkRmQ4WSuhPS9QRpLxMBCIog6g8yEe/1bJYRQCU1qBzz6 FCh1qXj//Re72AnJEgYR02Hc3ntNSpSAn9+FgmfYGin/Gx7i0XXa7Ws5Q5r1vAbba6Mj 05dDjM0YlqtkYqdC/oTk5P0KFn8aA4BhMmO48S7/jn9iLjAUgQggdYg1MfoiUqF+YFtI BiWnYH0gH2nYtvPqqaMpF9NRu38JZl7dWbOW9bn3cLTOAg/JdXEeC6AKhi/ErGjH5TRk MqN/RCb6HMkpWJ6DrDOM/c1zqP39vmt3fUK/qzAoog2amirYh99Mul6eeut1duqdJH88 vsbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject; bh=NJWMWnGqc5wXx77X2p+emxeUcX1MEi7E0F8HR0yF8tM=; b=jcREmDoWgr45tjejTusFhN2XpyKzQn9cztJ2uKftLX3XeGnQ5o5p9mYSg2n0Lne4KX s2WPYOHVjozf0Ewn575wwCyr7Cjr0eyGpg0HYFyCxfIWchG9nnq/sdfKf8SR5qG3W8vs YLzZFxky/tS5Ot28FkK7xAKR5v4LZd3sMXNvRWBubIFguASI4+AmY2iLB3Ot56ixeVDo sn+UR5l2KzEwn0e6D7r6SFps3ej1ydRHk0lrdpbAK6eUjOUGOeXooqF0JOZ1VcLr0KOP mFUDX8ACoDk4VuO4iqXizawzCsU3hM/eFzYQzJqnbm+YOCf5BpUm/1cKbSiBiKWGmc2A +lYw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga18.intel.com (mga18.intel.com. [134.134.136.126]) by mx.google.com with ESMTPS id t6si6239072pgn.258.2018.11.30.13.53.24 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 30 Nov 2018 13:53:24 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.126 as permitted sender) client-ip=134.134.136.126; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga106.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Nov 2018 13:53:24 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,300,1539673200"; d="scan'208";a="96500691" Received: from ahduyck-desk1.jf.intel.com (HELO ahduyck-desk1.amr.corp.intel.com) ([10.7.198.76]) by orsmga006.jf.intel.com with ESMTP; 30 Nov 2018 13:53:23 -0800 Subject: [mm PATCH v6 7/7] mm: Use common iterator for deferred_init_pages and deferred_free_pages From: Alexander Duyck To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org, davem@davemloft.net, pavel.tatashin@microsoft.com, mhocko@suse.com, mingo@kernel.org, kirill.shutemov@linux.intel.com, dan.j.williams@intel.com, dave.jiang@intel.com, alexander.h.duyck@linux.intel.com, rppt@linux.vnet.ibm.com, willy@infradead.org, vbabka@suse.cz, khalid.aziz@oracle.com, ldufour@linux.vnet.ibm.com, mgorman@techsingularity.net, yi.z.zhang@linux.intel.com, alexander.h.duyck@linux.intel.com Date: Fri, 30 Nov 2018 13:53:23 -0800 Message-ID: <154361480390.7497.9730184349746888133.stgit@ahduyck-desk1.amr.corp.intel.com> In-Reply-To: <154361452447.7497.1348692079883153517.stgit@ahduyck-desk1.amr.corp.intel.com> References: <154361452447.7497.1348692079883153517.stgit@ahduyck-desk1.amr.corp.intel.com> User-Agent: StGit/unknown-version MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Create a common iterator to be used by both deferred_init_pages and deferred_free_pages. By doing this we can cut down a bit on code overhead as they will likely both be inlined into the same function anyway. This new approach allows deferred_init_pages to make use of __init_pageblock. By doing this we can cut down on the code size by sharing code between both the hotplug and deferred memory init code paths. An additional benefit to this approach is that we improve in cache locality of the memory init as we can focus on the memory areas related to identifying if a given PFN is valid and keep that warm in the cache until we transition to a region of a different type. So we will stream through a chunk of valid blocks before we turn to initializing page structs. On my x86_64 test system with 384GB of memory per node I saw a reduction in initialization time from 1.38s to 1.06s as a result of this patch. Reviewed-by: Pavel Tatashin Signed-off-by: Alexander Duyck --- mm/page_alloc.c | 146 +++++++++++++++++++++++++++++-------------------------- 1 file changed, 77 insertions(+), 69 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 61eb9945d805..48c6fc73a70d 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1481,32 +1481,6 @@ void clear_zone_contiguous(struct zone *zone) } #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT -static void __init deferred_free_range(unsigned long pfn, - unsigned long nr_pages) -{ - struct page *page; - unsigned long i; - - if (!nr_pages) - return; - - page = pfn_to_page(pfn); - - /* Free a large naturally-aligned chunk if possible */ - if (nr_pages == pageblock_nr_pages && - (pfn & (pageblock_nr_pages - 1)) == 0) { - set_pageblock_migratetype(page, MIGRATE_MOVABLE); - __free_pages_core(page, pageblock_order); - return; - } - - for (i = 0; i < nr_pages; i++, page++, pfn++) { - if ((pfn & (pageblock_nr_pages - 1)) == 0) - set_pageblock_migratetype(page, MIGRATE_MOVABLE); - __free_pages_core(page, 0); - } -} - /* Completion tracking for deferred_init_memmap() threads */ static atomic_t pgdat_init_n_undone __initdata; static __initdata DECLARE_COMPLETION(pgdat_init_all_done_comp); @@ -1518,48 +1492,89 @@ static inline void __init pgdat_init_report_one_done(void) } /* - * Returns true if page needs to be initialized or freed to buddy allocator. + * Returns count if page range needs to be initialized or freed * - * First we check if pfn is valid on architectures where it is possible to have - * holes within pageblock_nr_pages. On systems where it is not possible, this - * function is optimized out. + * First we check if the contiguous pfns are valid on architectures where it + * is possible to have holes within pageblock_nr_pages. On systems where it + * is not possible, this function is optimized out. + * + * Then, we check if a current large page is valid by only checking the + * validity of the head pfn. * - * Then, we check if a current large page is valid by only checking the validity - * of the head pfn. */ -static inline bool __init deferred_pfn_valid(unsigned long pfn) +static unsigned long __next_pfn_valid_range(unsigned long *pfn, + unsigned long *i, + unsigned long end_pfn) { - if (!pfn_valid_within(pfn)) - return false; - if (!(pfn & (pageblock_nr_pages - 1)) && !pfn_valid(pfn)) - return false; - return true; + unsigned long start_pfn = *i; + + while (start_pfn < end_pfn) { + unsigned long t = ALIGN(start_pfn + 1, pageblock_nr_pages); + unsigned long pageblock_pfn = min(t, end_pfn); + unsigned long count = 0; + +#ifndef CONFIG_HOLES_IN_ZONE + if (pfn_valid(start_pfn)) + count = pageblock_pfn - start_pfn; + start_pfn = pageblock_pfn; +#else + while (start_pfn < pageblock_pfn) { + if (pfn_valid(start_pfn++)) { + count++; + continue; + } + + if (!count) + continue; + + /* + * The last PFN was invalid, report the block of + * PFNs we currently have available and skip over + * the invalid one. + */ + *pfn = start_pfn - (count + 1); + *i = start_pfn; + return count; + } +#endif + if (!count) + continue; + + *pfn = start_pfn - count; + *i = start_pfn; + return count; + } + + return 0; } +#define for_each_deferred_pfn_valid_range(pfn, count, i, start_pfn, end_pfn) \ + for (i = (start_pfn), \ + count = __next_pfn_valid_range(&pfn, &i, (end_pfn)); \ + count; \ + count = __next_pfn_valid_range(&pfn, &i, (end_pfn))) + /* * Free pages to buddy allocator. Try to free aligned pages in * pageblock_nr_pages sizes. */ -static void __init deferred_free_pages(unsigned long pfn, +static void __init deferred_free_pages(unsigned long start_pfn, unsigned long end_pfn) { - unsigned long nr_pgmask = pageblock_nr_pages - 1; - unsigned long nr_free = 0; - - for (; pfn < end_pfn; pfn++) { - if (!deferred_pfn_valid(pfn)) { - deferred_free_range(pfn - nr_free, nr_free); - nr_free = 0; - } else if (!(pfn & nr_pgmask)) { - deferred_free_range(pfn - nr_free, nr_free); - nr_free = 1; - touch_nmi_watchdog(); + unsigned long i, pfn, count; + + for_each_deferred_pfn_valid_range(pfn, count, i, start_pfn, end_pfn) { + struct page *page = pfn_to_page(pfn); + + if (count == pageblock_nr_pages) { + __free_pages_core(page, pageblock_order); } else { - nr_free++; + while (count--) + __free_pages_core(page++, 0); } + + touch_nmi_watchdog(); } - /* Free the last block of pages to allocator */ - deferred_free_range(pfn - nr_free, nr_free); } /* @@ -1568,29 +1583,22 @@ static void __init deferred_free_pages(unsigned long pfn, * Return number of pages initialized. */ static unsigned long __init deferred_init_pages(struct zone *zone, - unsigned long pfn, + unsigned long start_pfn, unsigned long end_pfn) { - unsigned long nr_pgmask = pageblock_nr_pages - 1; int nid = zone_to_nid(zone); + unsigned long i, pfn, count; unsigned long nr_pages = 0; int zid = zone_idx(zone); - struct page *page = NULL; - for (; pfn < end_pfn; pfn++) { - if (!deferred_pfn_valid(pfn)) { - page = NULL; - continue; - } else if (!page || !(pfn & nr_pgmask)) { - page = pfn_to_page(pfn); - touch_nmi_watchdog(); - } else { - page++; - } - __init_single_page(page, pfn, zid, nid); - nr_pages++; + for_each_deferred_pfn_valid_range(pfn, count, i, start_pfn, end_pfn) { + nr_pages += count; + __init_pageblock(pfn, count, zid, nid, NULL, false); + + touch_nmi_watchdog(); } - return (nr_pages); + + return nr_pages; } /*