From patchwork Fri Oct 5 15:12:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10628297 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6D877112B for ; Fri, 5 Oct 2018 15:12:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 575CA297EE for ; Fri, 5 Oct 2018 15:12:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 55A172980C; Fri, 5 Oct 2018 15:12:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A878B29825 for ; Fri, 5 Oct 2018 15:12:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 75DD26B0271; Fri, 5 Oct 2018 11:12:34 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 70D5F6B0272; Fri, 5 Oct 2018 11:12:34 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5D7AC6B0273; Fri, 5 Oct 2018 11:12:34 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) by kanga.kvack.org (Postfix) with ESMTP id 170986B0271 for ; Fri, 5 Oct 2018 11:12:34 -0400 (EDT) Received: by mail-pg1-f200.google.com with SMTP id e6-v6so6830647pge.5 for ; Fri, 05 Oct 2018 08:12:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:from :to:cc:date:message-id:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=VbI0oypiNuBd2J+fDxXjgmQgjmKW+0JuD3bnq1+BaS4=; b=j/76T2lCzx53hoeH11/oXPvx0463tRDO9z7gIKBc8JNuh0hcb7NCDoqTSz8RmoPdAF H0g9Pch3Q3Dis/cf6IYatAvdM6J0ML2lEBK/cGxnNZA5G1GF2fFKg+C2arwKpvCkklHu 5VgF2wZz7d/XjL2U9dOROEHjZHjCZ/CwkQ9hPB5eoriT9M95xnThy35K3U2W37qh7C+m RbZeFaFb/ZEJZCA3tGN3/RTc9fKW6nQVgByUpFSjQE89GQv9HTuMODe5sGDsgvY7ZFiT SimgRS7CdwVRVooBQQKfbC/mxjNu0fSiAvYPfoSFFhAPKpyxUPXWPZ48eQUaswYWfQ4Z idhg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfohsyhjvSKbrGXyYY/UTy8wfgCL84oBHboKQ/QYCzyiTMAarmoiF uYMAiS/1XdAoZaWeP5Vaf+zfk9/75IpEqf3oZfhFvzbGB3JTTudVKd0NtjOOW13PXhz8jw/K5za fNtmPbt5yhPaXC0hxG6pY9ICbjsUrCNRjUWfYPvSLnQdkztMNNm2m6Z+jfThMGrjdkA== X-Received: by 2002:a17:902:bcc3:: with SMTP id o3-v6mr12361897pls.202.1538752353740; Fri, 05 Oct 2018 08:12:33 -0700 (PDT) X-Google-Smtp-Source: ACcGV61atB6MVz7HUZngWjApF6rGwkNp6Vz09cyvkRW7a5cM9dfEdx+2qnyPWFp4gc9Nst0R3xH+ X-Received: by 2002:a17:902:bcc3:: with SMTP id o3-v6mr12361829pls.202.1538752352787; Fri, 05 Oct 2018 08:12:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1538752352; cv=none; d=google.com; s=arc-20160816; b=Q1pA7tMhegtSL2Qs/aduYRCxrgJZlUkHaaC0i+veTD+lhAxJJu12v7kP1F3y9ldI75 Fkd+0f9P0fXS1832IYeGCNq3Kbfuxv267fx6fMN7o55AwcW9O0xIp2v8ngc7LlOVOc2D tG3yIScXPMW06Tipljgn2tlJzEdchkvhm6ZK5/UtJsGOrTdlEZQGXEkYXx+c0BmfRTZ7 M7Z9GTzglFKeoCHQq3UXA1UMZn1AlCVaKHVp3nPNzUIeiHnYiPpYhfKkhf0SUcEz5Uuj r9fg1ixNckr1FbKc1rsvXjj6FigcVL4Uoe2Kp/j8Pm7rbxD/qelV+V9u7KzwhYkTiL9r DRlg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject; bh=VbI0oypiNuBd2J+fDxXjgmQgjmKW+0JuD3bnq1+BaS4=; b=OVqu/wft4rpBNPVWBZZiibBmzHidy4y4F25x9dE2WAbJnFjpHEiR5E3EztjO4OVjUi GaQGTqN9VWnJUOMGJxgNiBoHzQr4liD70z3/Qph2DroZ2Oi5/AVOYp0yguH8ZREBrZ+C f9kCPbzOjBxaW1xzigcv9PDxklaAb01B6yRFaFS3Jgr8+S5Z/N84uNW/ZijD8WP8i1Kv RKtly/GqwUtQzZ3ZAPkYO+RWsWJuOLsu70S0iNyDQPsrIOZgEWKMlmavIu7RFMqrmW3D L2DtIeaUU32C4/gFvOq+50VSFbw+8A6Z1XALVW+X+Mb3rXG4FZl33ejgbvMK4cYtPQC5 ICyg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga06.intel.com (mga06.intel.com. [134.134.136.31]) by mx.google.com with ESMTPS id d35-v6si9294185pla.116.2018.10.05.08.12.32 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 05 Oct 2018 08:12:32 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.31 as permitted sender) client-ip=134.134.136.31; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Oct 2018 08:12:32 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,344,1534834800"; d="scan'208";a="79013196" Received: from ahduyck-mobl.amr.corp.intel.com (HELO localhost.localdomain) ([10.7.198.166]) by orsmga008.jf.intel.com with ESMTP; 05 Oct 2018 08:12:31 -0700 Subject: [mm PATCH 5/5] mm: Use common iterator for deferred_init_pages and deferred_free_pages From: Alexander Duyck To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: pavel.tatashin@microsoft.com, mhocko@suse.com, dave.jiang@intel.com, alexander.h.duyck@linux.intel.com, linux-kernel@vger.kernel.org, willy@infradead.org, davem@davemloft.net, khalid.aziz@oracle.com, rppt@linux.vnet.ibm.com, vbabka@suse.cz, sparclinux@vger.kernel.org, dan.j.williams@intel.com, ldufour@linux.vnet.ibm.com, mgorman@techsingularity.net, mingo@kernel.org, kirill.shutemov@linux.intel.com Date: Fri, 05 Oct 2018 08:12:30 -0700 Message-ID: <20181005151230.17473.88155.stgit@localhost.localdomain> In-Reply-To: <20181005151006.17473.83040.stgit@localhost.localdomain> References: <20181005151006.17473.83040.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch creates a common iterator to be used by both deferred_init_pages and deferred_free_pages. By doing this we can cut down a bit on code overhead as they will likely both be inlined into the same function anyway. This new approach allows deferred_init_pages to make use of __init_pageblock. By doing this we can cut down on the code size by sharing code between both the hotplug and deferred memory init code paths. An additional benefit to this approach is that we improve in cache locality of the memory init as we can focus on the memory areas related to identifying if a given PFN is valid and keep that warm in the cache until we transition to a region of a different type. So we will stream through a chunk of valid blocks before we turn to initializing page structs. Signed-off-by: Alexander Duyck --- mm/page_alloc.c | 134 +++++++++++++++++++++++++++---------------------------- 1 file changed, 65 insertions(+), 69 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 1baea475f296..815ce793c73d 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1480,32 +1480,6 @@ void clear_zone_contiguous(struct zone *zone) } #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT -static void __init deferred_free_range(unsigned long pfn, - unsigned long nr_pages) -{ - struct page *page; - unsigned long i; - - if (!nr_pages) - return; - - page = pfn_to_page(pfn); - - /* Free a large naturally-aligned chunk if possible */ - if (nr_pages == pageblock_nr_pages && - (pfn & (pageblock_nr_pages - 1)) == 0) { - set_pageblock_migratetype(page, MIGRATE_MOVABLE); - __free_pages_boot_core(page, pageblock_order); - return; - } - - for (i = 0; i < nr_pages; i++, page++, pfn++) { - if ((pfn & (pageblock_nr_pages - 1)) == 0) - set_pageblock_migratetype(page, MIGRATE_MOVABLE); - __free_pages_boot_core(page, 0); - } -} - /* Completion tracking for deferred_init_memmap() threads */ static atomic_t pgdat_init_n_undone __initdata; static __initdata DECLARE_COMPLETION(pgdat_init_all_done_comp); @@ -1517,48 +1491,77 @@ static inline void __init pgdat_init_report_one_done(void) } /* - * Returns true if page needs to be initialized or freed to buddy allocator. + * Returns count if page range needs to be initialized or freed * - * First we check if pfn is valid on architectures where it is possible to have - * holes within pageblock_nr_pages. On systems where it is not possible, this - * function is optimized out. + * First, we check if a current large page is valid by only checking the + * validity of the head pfn. * - * Then, we check if a current large page is valid by only checking the validity - * of the head pfn. + * Then we check if the contiguous pfns are valid on architectures where it + * is possible to have holes within pageblock_nr_pages. On systems where it + * is not possible, this function is optimized out. */ -static inline bool __init deferred_pfn_valid(unsigned long pfn) +static unsigned long __next_pfn_valid_range(unsigned long *i, + unsigned long end_pfn) { - if (!pfn_valid_within(pfn)) - return false; - if (!(pfn & (pageblock_nr_pages - 1)) && !pfn_valid(pfn)) - return false; - return true; + unsigned long pfn = *i; + unsigned long count; + + while (pfn < end_pfn) { + unsigned long t = ALIGN(pfn + 1, pageblock_nr_pages); + unsigned long pageblock_pfn = min(t, end_pfn); + +#ifndef CONFIG_HOLES_IN_ZONE + count = pageblock_pfn - pfn; + pfn = pageblock_pfn; + if (!pfn_valid(pfn)) + continue; +#else + for (count = 0; pfn < pageblock_pfn; pfn++) { + if (pfn_valid_within(pfn)) { + count++; + continue; + } + + if (count) + break; + } + + if (!count) + continue; +#endif + *i = pfn; + return count; + } + + return 0; } +#define for_each_deferred_pfn_valid_range(i, start_pfn, end_pfn, pfn, count) \ + for (i = (start_pfn), \ + count = __next_pfn_valid_range(&i, (end_pfn)); \ + count && ({ pfn = i - count; 1; }); \ + count = __next_pfn_valid_range(&i, (end_pfn))) /* * Free pages to buddy allocator. Try to free aligned pages in * pageblock_nr_pages sizes. */ -static void __init deferred_free_pages(unsigned long pfn, +static void __init deferred_free_pages(unsigned long start_pfn, unsigned long end_pfn) { - unsigned long nr_pgmask = pageblock_nr_pages - 1; - unsigned long nr_free = 0; - - for (; pfn < end_pfn; pfn++) { - if (!deferred_pfn_valid(pfn)) { - deferred_free_range(pfn - nr_free, nr_free); - nr_free = 0; - } else if (!(pfn & nr_pgmask)) { - deferred_free_range(pfn - nr_free, nr_free); - nr_free = 1; - touch_nmi_watchdog(); + unsigned long i, pfn, count; + + for_each_deferred_pfn_valid_range(i, start_pfn, end_pfn, pfn, count) { + struct page *page = pfn_to_page(pfn); + + if (count == pageblock_nr_pages) { + __free_pages_boot_core(page, pageblock_order); } else { - nr_free++; + while (count--) + __free_pages_boot_core(page++, 0); } + + touch_nmi_watchdog(); } - /* Free the last block of pages to allocator */ - deferred_free_range(pfn - nr_free, nr_free); } /* @@ -1567,29 +1570,22 @@ static void __init deferred_free_pages(unsigned long pfn, * Return number of pages initialized. */ static unsigned long __init deferred_init_pages(struct zone *zone, - unsigned long pfn, + unsigned long start_pfn, unsigned long end_pfn) { - unsigned long nr_pgmask = pageblock_nr_pages - 1; + unsigned long i, pfn, count; int nid = zone_to_nid(zone); unsigned long nr_pages = 0; int zid = zone_idx(zone); - struct page *page = NULL; - for (; pfn < end_pfn; pfn++) { - if (!deferred_pfn_valid(pfn)) { - page = NULL; - continue; - } else if (!page || !(pfn & nr_pgmask)) { - page = pfn_to_page(pfn); - touch_nmi_watchdog(); - } else { - page++; - } - __init_single_page(page, pfn, zid, nid); - nr_pages++; + for_each_deferred_pfn_valid_range(i, start_pfn, end_pfn, pfn, count) { + nr_pages += count; + __init_pageblock(pfn, count, zid, nid, NULL, false); + + touch_nmi_watchdog(); } - return (nr_pages); + + return nr_pages; } /*