From patchwork Thu Oct 11 22:13:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10637531 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B99FE157A for ; Thu, 11 Oct 2018 22:17:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A49542C1FB for ; Thu, 11 Oct 2018 22:17:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 98A332C206; Thu, 11 Oct 2018 22:17:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D18332C1FB for ; Thu, 11 Oct 2018 22:17:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 81C436B000C; Thu, 11 Oct 2018 18:17:00 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5FD1D6B0269; Thu, 11 Oct 2018 18:17:00 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 478946B000E; Thu, 11 Oct 2018 18:17:00 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by kanga.kvack.org (Postfix) with ESMTP id E46106B000C for ; Thu, 11 Oct 2018 18:16:59 -0400 (EDT) Received: by mail-pl1-f199.google.com with SMTP id f17-v6so7918009plr.1 for ; Thu, 11 Oct 2018 15:16:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:from :to:cc:date:message-id:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=eNd6I70UP7FktykxSCJvVzXKcLm6++R1rOpUNF8D0mI=; b=f4n22ZiG0PRxoGqoJ6ivTHGdIck7khPAV1DcV+Rc+y/1f13+0DnsAYM+cviipK4Jkb 8+7JFDIrmmxjrCmeqqvmagN6zGQSnnaoWDSGKaNlmowhGpT0Rancft5CUTywwjW4+DR6 akt37p0pY8YE0fY7T1JevVBatCO32WgcoBIPRZrbkQ4fvA6k8OgwUnLhW1bNsOb+bZ29 lMosPDNe9p4g60VaP/U3WOY4RuHtJhr1qJTbkv4PMXGgLb6j8XgwQIGBNiAYrjlA7dkx JqCm6oZH4u/+2gYEUfrkstuOc8leJ7uSl0+aTWUN14W4Q+iNzWpwb8S9hDwxcb4mcjLh cqag== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfoiwfRks4I0sZvkza7OPPnmSt5nmKusV4evjl2U0GSXl409paPMD o1FSRcqS7EWeBeEdvmNfdz6wKZ+R+IFX4Bcl/qGQHFSkMWYMZhdZZACPd/XtgbLDIYaw8b58qWV vK4XFnvBps5GXDw0Jx7S2rrKgFNow2E3SKgWcZtUIlqIzDg/7r/tiiC++83eVD+iHYw== X-Received: by 2002:a63:680a:: with SMTP id d10-v6mr3004258pgc.7.1539296219583; Thu, 11 Oct 2018 15:16:59 -0700 (PDT) X-Google-Smtp-Source: ACcGV62fA3izl4aPEGOr7m+JnwuHxU5YDP3IdxH82hlKjY1zjb3tt5iYFGUNbMAcZOKRVtQyhkmv X-Received: by 2002:a63:680a:: with SMTP id d10-v6mr3004195pgc.7.1539296218259; Thu, 11 Oct 2018 15:16:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1539296218; cv=none; d=google.com; s=arc-20160816; b=N9Btn88nTtNr6TF8ksBjf8+xdpnh8AyViX8tizcculCCdgg0HUBM3Y7PrD/5vmqXR2 QbbUm0aA5uY+y8THR6Ayp8eOvxZCG8qPXLef0E1HUkqqCvNCplCT2bcnb3hzYyuF7Fn5 HDt4IbVPlACHo1X1iN9D3kiHJIytuhNojol/JkUK+VNHJl4CcODbEaxr4E+HFT8u/CR6 Cpb+b6PyGDEg40wo7NAchBGBKyxrTPWR7WWIhxByKn0j24azaU5HXc4TeRt2far6IWHG 58IjsaTr5WyC7Zi0rMv+2P44SbQ/rEwkv+XJL9irwy/8LEDz5Y+1wyDBpxO6bO1pdufV 4o5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject; bh=eNd6I70UP7FktykxSCJvVzXKcLm6++R1rOpUNF8D0mI=; b=aBWWvCzNs+S5uMIQsaEbBiitldgBDdJjI8y56Q6XhsafqrVJLpHC58DwI6DKxYa2vo W/8M6hxl/p2PHPpOfbGeknYErL2VPQH1dic2+mfTBxWarkEE0lE5XGuZrL3PK2sUscH9 8/qs9rsSbmimP8xfv2JJIAOHHazH5gZkVDrzMpSCQy73VXzu9dDGemw+QQ4lJj4eqQkQ VF3IoRpcolQPEdgLzc10jwqRKp0VE70HJDsoTzDsJJKuXtxZfp/zh1G2QCZ4j8AI4mUc IRt8AlONx7dYc78lsxWFE1qFdDl+ReGj5j0u2NdREjdnp2sxa9yEbzV+BO+WGC0UNjBS JhSQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga18.intel.com (mga18.intel.com. [134.134.136.126]) by mx.google.com with ESMTPS id r29-v6si24916614pff.262.2018.10.11.15.16.58 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Oct 2018 15:16:58 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.126 as permitted sender) client-ip=134.134.136.126; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga106.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Oct 2018 15:16:57 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,370,1534834800"; d="scan'208";a="80765708" Received: from ahduyck-mobl.amr.corp.intel.com (HELO localhost.localdomain) ([10.7.198.157]) by orsmga008.jf.intel.com with ESMTP; 11 Oct 2018 15:13:57 -0700 Subject: [mm PATCH v2 5/6] mm: Move hot-plug specific memory init into separate functions and optimize From: Alexander Duyck To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: pavel.tatashin@microsoft.com, mhocko@suse.com, dave.jiang@intel.com, alexander.h.duyck@linux.intel.com, linux-kernel@vger.kernel.org, willy@infradead.org, davem@davemloft.net, yi.z.zhang@linux.intel.com, khalid.aziz@oracle.com, rppt@linux.vnet.ibm.com, vbabka@suse.cz, sparclinux@vger.kernel.org, dan.j.williams@intel.com, ldufour@linux.vnet.ibm.com, mgorman@techsingularity.net, mingo@kernel.org, kirill.shutemov@linux.intel.com Date: Thu, 11 Oct 2018 15:13:57 -0700 Message-ID: <20181011221357.1925.5661.stgit@localhost.localdomain> In-Reply-To: <20181011221237.1925.85591.stgit@localhost.localdomain> References: <20181011221237.1925.85591.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch is going through and combining the bits in memmap_init_zone and memmap_init_zone_device that are related to hotplug into a single function called __memmap_init_hotplug. I also took the opportunity to integrate __init_single_page's functionality into this function. In doing so I can get rid of some of the redundancy such as the LRU pointers versus the pgmap. Signed-off-by: Alexander Duyck --- mm/page_alloc.c | 199 ++++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 137 insertions(+), 62 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e435223e2ddb..5987c859676b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1192,6 +1192,79 @@ static void __meminit __init_single_page(struct page *page, unsigned long pfn, #endif } +static void __meminit __init_pageblock(unsigned long start_pfn, + unsigned long nr_pages, + unsigned long zone, int nid, + struct dev_pagemap *pgmap) +{ + unsigned long nr_pgmask = pageblock_nr_pages - 1; + struct page *start_page = pfn_to_page(start_pfn); + unsigned long pfn = start_pfn + nr_pages - 1; +#ifdef WANT_PAGE_VIRTUAL + bool is_highmem = is_highmem_idx(zone); +#endif + struct page *page; + + /* + * Enforce the following requirements: + * size > 0 + * size < pageblock_nr_pages + * start_pfn -> pfn does not cross pageblock_nr_pages boundary + */ + VM_BUG_ON(((start_pfn ^ pfn) | (nr_pages - 1)) > nr_pgmask); + + /* + * Work from highest page to lowest, this way we will still be + * warm in the cache when we call set_pageblock_migratetype + * below. + * + * The loop is based around the page pointer as the main index + * instead of the pfn because pfn is not used inside the loop if + * the section number is not in page flags and WANT_PAGE_VIRTUAL + * is not defined. + */ + for (page = start_page + nr_pages; page-- != start_page; pfn--) { + mm_zero_struct_page(page); + set_page_links(page, zone, nid, pfn); + init_page_count(page); + page_mapcount_reset(page); + page_cpupid_reset_last(page); + + /* + * ZONE_DEVICE pages union ->lru with a ->pgmap back + * pointer and hmm_data. It is a bug if a ZONE_DEVICE + * page is ever freed or placed on a driver-private list. + */ + if (pgmap) + page->pgmap = pgmap; + else + INIT_LIST_HEAD(&page->lru); +#ifdef WANT_PAGE_VIRTUAL + /* The shift won't overflow because ZONE_NORMAL is below 4G. */ + if (!is_highmem) + set_page_address(page, __va(pfn << PAGE_SHIFT)); +#endif + } + + /* + * Mark the block movable so that blocks are reserved for + * movable at startup. This will force kernel allocations + * to reserve their blocks rather than leaking throughout + * the address space during boot when many long-lived + * kernel allocations are made. + * + * bitmap is created for zone's valid pfn range. but memmap + * can be created for invalid pages (for alignment) + * check here not to call set_pageblock_migratetype() against + * pfn out of zone. + * + * Please note that MEMMAP_HOTPLUG path doesn't clear memmap + * because this is done early in sparse_add_one_section + */ + if (!(start_pfn & nr_pgmask)) + set_pageblock_migratetype(start_page, MIGRATE_MOVABLE); +} + #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT static void __meminit init_reserved_page(unsigned long pfn) { @@ -5518,6 +5591,30 @@ void __ref build_all_zonelists(pg_data_t *pgdat) return false; } +static void __meminit __memmap_init_hotplug(unsigned long size, int nid, + unsigned long zone, + unsigned long start_pfn, + struct dev_pagemap *pgmap) +{ + unsigned long pfn = start_pfn + size; + + while (pfn != start_pfn) { + unsigned long stride = pfn; + + pfn = max(ALIGN_DOWN(pfn - 1, pageblock_nr_pages), start_pfn); + stride -= pfn; + + /* + * Mark page reserved as it will need to wait for + * onlining phase for it to be fully associated with + * a zone. + */ + __init_pageblock(pfn, stride, zone, nid, pgmap); + + cond_resched(); + } +} + /* * Initially all pages are reserved - free ones are freed * up by memblock_free_all() once the early boot process is @@ -5528,46 +5625,57 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone, struct vmem_altmap *altmap) { unsigned long pfn, end_pfn = start_pfn + size; - struct page *page; if (highest_memmap_pfn < end_pfn - 1) highest_memmap_pfn = end_pfn - 1; + if (context == MEMMAP_HOTPLUG) { #ifdef CONFIG_ZONE_DEVICE - /* - * Honor reservation requested by the driver for this ZONE_DEVICE - * memory. We limit the total number of pages to initialize to just - * those that might contain the memory mapping. We will defer the - * ZONE_DEVICE page initialization until after we have released - * the hotplug lock. - */ - if (zone == ZONE_DEVICE) { - if (!altmap) - return; + /* + * Honor reservation requested by the driver for this + * ZONE_DEVICE memory. We limit the total number of pages to + * initialize to just those that might contain the memory + * mapping. We will defer the ZONE_DEVICE page initialization + * until after we have released the hotplug lock. + */ + if (zone == ZONE_DEVICE) { + if (!altmap) + return; + + if (start_pfn == altmap->base_pfn) + start_pfn += altmap->reserve; + end_pfn = altmap->base_pfn + + vmem_altmap_offset(altmap); + } +#endif + /* + * For these pages we don't need to record the pgmap as they + * should represent only those pages used to store the memory + * map. The actual ZONE_DEVICE pages will be initialized later. + */ + __memmap_init_hotplug(end_pfn - start_pfn, nid, zone, + start_pfn, NULL); - if (start_pfn == altmap->base_pfn) - start_pfn += altmap->reserve; - end_pfn = altmap->base_pfn + vmem_altmap_offset(altmap); + return; } -#endif for (pfn = start_pfn; pfn < end_pfn; pfn++) { + struct page *page; + /* * There can be holes in boot-time mem_map[]s handed to this * function. They do not exist on hotplugged memory. */ - if (context == MEMMAP_EARLY) { - if (!early_pfn_valid(pfn)) { - pfn = next_valid_pfn(pfn) - 1; - continue; - } - if (!early_pfn_in_nid(pfn, nid)) - continue; - if (overlap_memmap_init(zone, &pfn)) - continue; - if (defer_init(nid, pfn, end_pfn)) - break; + if (!early_pfn_valid(pfn)) { + pfn = next_valid_pfn(pfn) - 1; + continue; } + if (!early_pfn_in_nid(pfn, nid)) + continue; + if (overlap_memmap_init(zone, &pfn)) + continue; + if (defer_init(nid, pfn, end_pfn)) + break; page = pfn_to_page(pfn); __init_single_page(page, pfn, zone, nid); @@ -5597,14 +5705,12 @@ void __ref memmap_init_zone_device(struct zone *zone, unsigned long size, struct dev_pagemap *pgmap) { - unsigned long pfn, end_pfn = start_pfn + size; struct pglist_data *pgdat = zone->zone_pgdat; unsigned long zone_idx = zone_idx(zone); unsigned long start = jiffies; int nid = pgdat->node_id; - if (WARN_ON_ONCE(!pgmap || !is_dev_zone(zone))) - return; + VM_BUG_ON(!is_dev_zone(zone)); /* * The call to memmap_init_zone should have already taken care @@ -5613,44 +5719,13 @@ void __ref memmap_init_zone_device(struct zone *zone, */ if (pgmap->altmap_valid) { struct vmem_altmap *altmap = &pgmap->altmap; + unsigned long end_pfn = start_pfn + size; start_pfn = altmap->base_pfn + vmem_altmap_offset(altmap); size = end_pfn - start_pfn; } - for (pfn = start_pfn; pfn < end_pfn; pfn++) { - struct page *page = pfn_to_page(pfn); - - __init_single_page(page, pfn, zone_idx, nid); - - /* - * ZONE_DEVICE pages union ->lru with a ->pgmap back - * pointer and hmm_data. It is a bug if a ZONE_DEVICE - * page is ever freed or placed on a driver-private list. - */ - page->pgmap = pgmap; - page->hmm_data = 0; - - /* - * Mark the block movable so that blocks are reserved for - * movable at startup. This will force kernel allocations - * to reserve their blocks rather than leaking throughout - * the address space during boot when many long-lived - * kernel allocations are made. - * - * bitmap is created for zone's valid pfn range. but memmap - * can be created for invalid pages (for alignment) - * check here not to call set_pageblock_migratetype() against - * pfn out of zone. - * - * Please note that MEMMAP_HOTPLUG path doesn't clear memmap - * because this is done early in sparse_add_one_section - */ - if (!(pfn & (pageblock_nr_pages - 1))) { - set_pageblock_migratetype(page, MIGRATE_MOVABLE); - cond_resched(); - } - } + __memmap_init_hotplug(size, nid, zone_idx, start_pfn, pgmap); pr_info("%s initialised, %lu pages in %ums\n", dev_name(pgmap->dev), size, jiffies_to_msecs(jiffies - start));