From patchwork Fri Oct 5 15:12:24 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10628295 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 462A6112B for ; Fri, 5 Oct 2018 15:12:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2CDBA29781 for ; Fri, 5 Oct 2018 15:12:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2AAC02982E; Fri, 5 Oct 2018 15:12:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3B9CA2981F for ; Fri, 5 Oct 2018 15:12:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 16B186B026E; Fri, 5 Oct 2018 11:12:28 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0F5B26B0270; Fri, 5 Oct 2018 11:12:28 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ED7776B0271; Fri, 5 Oct 2018 11:12:27 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id A70136B026E for ; Fri, 5 Oct 2018 11:12:27 -0400 (EDT) Received: by mail-pf1-f200.google.com with SMTP id x19-v6so9187326pfh.15 for ; Fri, 05 Oct 2018 08:12:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:from :to:cc:date:message-id:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=z6Dhgy5ty4KK/Z5zIrtnjFt4Ugz+jJaNg0niAIfRotc=; b=A71zLsQRvgjbns6tXQGopcdaRH+IXgeiAu/0vEyaRQ26maSALBumpDXr5MgjAz8vve kqdQsUssR9YpaH2FgIt1nQgu7j/jk1DUMyBkPzVqqFhqpyuG0lHTP5LwHsnVQb9ai19W I60FbULcBOEN2KEoH5Y/cmd0uF0lzyKgTZUF0Wm2q03wNwsysRHdPnoYdMKur4QRTraG mg31ac7x6NDLY+pQwBC1HabBnRkvWwX3UEPwYGLJC3QZRNhW6fn/B42vmHOUJP/J6/+Q fYXA0MVbjboyQmT7sBZOYLWkdbSZcgI9afSBevfNixEah5ROTtszLgGteo56WopSStVi BmfA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: ABuFfojfwJS/b+47gbt3thbr5fmqA7TCyiLyHKOtXZsYBd7eEa7nN9cH GQIRBXVZBn9DW1zxBRjxdlHVWmABuIvDAPuPdxyCyyWIlGLc2Ulm15NfQL3Bx1aGMyXAhPNY7rY KIZ1wqjcKHdH2HvkiaN0oR/O60jsSA4fET2VJQ25KuOMCJ+/7+DbFVYNaoRS6lx0Hhg== X-Received: by 2002:aa7:87d4:: with SMTP id i20-v6mr12520337pfo.251.1538752347326; Fri, 05 Oct 2018 08:12:27 -0700 (PDT) X-Google-Smtp-Source: ACcGV63AVb2G1pmH06UbsIhMAjHG57yACoHStf/Ze/hOkmLezPMXt1F1RN6o+6h290YVRDSJBPIk X-Received: by 2002:aa7:87d4:: with SMTP id i20-v6mr12520234pfo.251.1538752346033; Fri, 05 Oct 2018 08:12:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1538752345; cv=none; d=google.com; s=arc-20160816; b=MS8UfCwQEFqerhy8064SyNPnpxjJiymNGw5FmWBqQwntq3RGO9BLNJeyoxh460Tug9 mJNVumNhixiKcTyGS2eH5XWn82CZ11wo3H4EdwECTxDEK3HJ+5Z18fnqJmsr19k2WxT/ dnd3+GoiatKVYs09XAMibN3bHRFVEzkLvsGExLcMVpt672OYIyjXcXcy/SF8vxhf2fqk w+jDkZ4NvwpDfyUxxA7DyfCDbqnR0CYPFlYt21oDGfBVV91w1hlC8OD+W9rQIsGRWDnN nontdYOXBqk6Zn8O5EWmLxhUJ6/5afjF18BHPS6SGt9Ve9DSiOhzlEZmf+VKypZW3hwV ltUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject; bh=z6Dhgy5ty4KK/Z5zIrtnjFt4Ugz+jJaNg0niAIfRotc=; b=yV86owJiLvciFD97lwWyP33MxTbnmu69Z9I+jHDX+5bECS9RLY5v58KbgVx0lZFKBi 4McUGDDOZ5i8kHtZnmfoNs0pvN0nazDxycgoTPVaQEeCarVy+dYKx4o7yMyDkc0FbExL 15lI6AQexQ3a/w43h2ugAZN+RRruCX6afzkheIW9LWqkJc8mxJVif8oLYQTLEEUTjKqE Tkp32sYb+vDTX02KMW1Bp1+FCEyTjdi6ANMv8G0+NM2QbKrnmg5wDYzslONksZCNXiY5 QsYRKJ/JVlznO3EuuDpfpCJbkHsjIam+6KFVXe4ucZ0c0dY0G/dD8ftj9BWMc/ChZ3cs Sdkw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga09.intel.com (mga09.intel.com. [134.134.136.24]) by mx.google.com with ESMTPS id w20-v6si8879454pll.89.2018.10.05.08.12.25 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 05 Oct 2018 08:12:25 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.24 as permitted sender) client-ip=134.134.136.24; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.24 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Oct 2018 08:12:25 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,344,1534834800"; d="scan'208";a="79013173" Received: from ahduyck-mobl.amr.corp.intel.com (HELO localhost.localdomain) ([10.7.198.166]) by orsmga008.jf.intel.com with ESMTP; 05 Oct 2018 08:12:24 -0700 Subject: [mm PATCH 4/5] mm: Move hot-plug specific memory init into separate functions and optimize From: Alexander Duyck To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: pavel.tatashin@microsoft.com, mhocko@suse.com, dave.jiang@intel.com, alexander.h.duyck@linux.intel.com, linux-kernel@vger.kernel.org, willy@infradead.org, davem@davemloft.net, khalid.aziz@oracle.com, rppt@linux.vnet.ibm.com, vbabka@suse.cz, sparclinux@vger.kernel.org, dan.j.williams@intel.com, ldufour@linux.vnet.ibm.com, mgorman@techsingularity.net, mingo@kernel.org, kirill.shutemov@linux.intel.com Date: Fri, 05 Oct 2018 08:12:24 -0700 Message-ID: <20181005151224.17473.53398.stgit@localhost.localdomain> In-Reply-To: <20181005151006.17473.83040.stgit@localhost.localdomain> References: <20181005151006.17473.83040.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch is going through and combining the bits in memmap_init_zone and memmap_init_zone_device that are related to hotplug into a single function called __memmap_init_hotplug. I also took the opportunity to integrate __init_single_page's functionality into this function. In doing so I can get rid of some of the redundancy such as the LRU pointers versus the pgmap. Signed-off-by: Alexander Duyck --- mm/page_alloc.c | 213 ++++++++++++++++++++++++++++++++++++------------------- 1 file changed, 140 insertions(+), 73 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 247b1f2573e4..1baea475f296 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1192,6 +1192,82 @@ static void __meminit __init_single_page(struct page *page, unsigned long pfn, #endif } +static void __meminit __init_pageblock(unsigned long start_pfn, + unsigned long nr_pages, + unsigned long zone, int nid, + struct dev_pagemap *pgmap, + bool is_reserved) +{ + unsigned long nr_pgmask = pageblock_nr_pages - 1; + struct page *start_page = pfn_to_page(start_pfn); + unsigned long pfn = start_pfn + nr_pages - 1; +#ifdef WANT_PAGE_VIRTUAL + bool is_highmem = is_highmem_idx(zone); +#endif + struct page *page; + + /* + * Enforce the following requirements: + * size > 0 + * size < pageblock_nr_pages + * start_pfn -> pfn does not cross pageblock_nr_pages boundary + */ + VM_BUG_ON(((start_pfn ^ pfn) | (nr_pages - 1)) > nr_pgmask); + + /* + * Work from highest page to lowest, this way we will still be + * warm in the cache when we call set_pageblock_migratetype + * below. + * + * The loop is based around the page pointer as the main index + * instead of the pfn because pfn is not used inside the loop if + * the section number is not in page flags and WANT_PAGE_VIRTUAL + * is not defined. + */ + for (page = start_page + nr_pages; page-- != start_page; pfn--) { + mm_zero_struct_page(page); + set_page_links(page, zone, nid, pfn); + init_page_count(page); + page_mapcount_reset(page); + page_cpupid_reset_last(page); + + if (is_reserved) + __SetPageReserved(page); + /* + * ZONE_DEVICE pages union ->lru with a ->pgmap back + * pointer and hmm_data. It is a bug if a ZONE_DEVICE + * page is ever freed or placed on a driver-private list. + */ + if (pgmap) + page->pgmap = pgmap; + else + INIT_LIST_HEAD(&page->lru); +#ifdef WANT_PAGE_VIRTUAL + /* The shift won't overflow because ZONE_NORMAL is below 4G. */ + if (!is_highmem) + set_page_address(page, __va(pfn << PAGE_SHIFT)); +#endif + } + + /* + * Mark the block movable so that blocks are reserved for + * movable at startup. This will force kernel allocations + * to reserve their blocks rather than leaking throughout + * the address space during boot when many long-lived + * kernel allocations are made. + * + * bitmap is created for zone's valid pfn range. but memmap + * can be created for invalid pages (for alignment) + * check here not to call set_pageblock_migratetype() against + * pfn out of zone. + * + * Please note that MEMMAP_HOTPLUG path doesn't clear memmap + * because this is done early in sparse_add_one_section + */ + if (!(start_pfn & nr_pgmask)) + set_pageblock_migratetype(start_page, MIGRATE_MOVABLE); +} + #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT static void __meminit init_reserved_page(unsigned long pfn) { @@ -5518,6 +5594,30 @@ void __ref build_all_zonelists(pg_data_t *pgdat) return false; } +static void __meminit __memmap_init_hotplug(unsigned long size, int nid, + unsigned long zone, + unsigned long start_pfn, + struct dev_pagemap *pgmap) +{ + unsigned long pfn = start_pfn + size; + + while (pfn != start_pfn) { + unsigned long stride = pfn; + + pfn = max(ALIGN_DOWN(pfn - 1, pageblock_nr_pages), start_pfn); + stride -= pfn; + + /* + * Mark page reserved as it will need to wait for + * onlining phase for it to be fully associated with + * a zone. + */ + __init_pageblock(pfn, stride, zone, nid, pgmap, true); + + cond_resched(); + } +} + /* * Initially all pages are reserved - free ones are freed * up by memblock_free_all() once the early boot process is @@ -5528,51 +5628,60 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone, struct vmem_altmap *altmap) { unsigned long pfn, end_pfn = start_pfn + size; - struct page *page; if (highest_memmap_pfn < end_pfn - 1) highest_memmap_pfn = end_pfn - 1; + if (context == MEMMAP_HOTPLUG) { #ifdef CONFIG_ZONE_DEVICE - /* - * Honor reservation requested by the driver for this ZONE_DEVICE - * memory. We limit the total number of pages to initialize to just - * those that might contain the memory mapping. We will defer the - * ZONE_DEVICE page initialization until after we have released - * the hotplug lock. - */ - if (zone == ZONE_DEVICE) { - if (!altmap) - return; + /* + * Honor reservation requested by the driver for this + * ZONE_DEVICE memory. We limit the total number of pages to + * initialize to just those that might contain the memory + * mapping. We will defer the ZONE_DEVICE page initialization + * until after we have released the hotplug lock. + */ + if (zone == ZONE_DEVICE) { + if (!altmap) + return; + + if (start_pfn == altmap->base_pfn) + start_pfn += altmap->reserve; + end_pfn = altmap->base_pfn + + vmem_altmap_offset(altmap); + } +#endif + /* + * For these pages we don't need to record the pgmap as they + * should represent only those pages used to store the memory + * map. The actual ZONE_DEVICE pages will be initialized later. + */ + __memmap_init_hotplug(end_pfn - start_pfn, nid, zone, + start_pfn, NULL); - if (start_pfn == altmap->base_pfn) - start_pfn += altmap->reserve; - end_pfn = altmap->base_pfn + vmem_altmap_offset(altmap); + return; } -#endif for (pfn = start_pfn; pfn < end_pfn; pfn++) { + struct page *page; + /* * There can be holes in boot-time mem_map[]s handed to this * function. They do not exist on hotplugged memory. */ - if (context == MEMMAP_EARLY) { - if (!early_pfn_valid(pfn)) { - pfn = next_valid_pfn(pfn) - 1; - continue; - } - if (!early_pfn_in_nid(pfn, nid)) - continue; - if (overlap_memmap_init(zone, &pfn)) - continue; - if (defer_init(nid, pfn, end_pfn)) - break; + if (!early_pfn_valid(pfn)) { + pfn = next_valid_pfn(pfn) - 1; + continue; } + if (!early_pfn_in_nid(pfn, nid)) + continue; + if (overlap_memmap_init(zone, &pfn)) + continue; + if (defer_init(nid, pfn, end_pfn)) + break; page = pfn_to_page(pfn); __init_single_page(page, pfn, zone, nid); - if (context == MEMMAP_HOTPLUG) - __SetPageReserved(page); /* * Mark the block movable so that blocks are reserved for @@ -5599,14 +5708,12 @@ void __ref memmap_init_zone_device(struct zone *zone, unsigned long size, struct dev_pagemap *pgmap) { - unsigned long pfn, end_pfn = start_pfn + size; struct pglist_data *pgdat = zone->zone_pgdat; unsigned long zone_idx = zone_idx(zone); unsigned long start = jiffies; int nid = pgdat->node_id; - if (WARN_ON_ONCE(!pgmap || !is_dev_zone(zone))) - return; + VM_BUG_ON(!is_dev_zone(zone)); /* * The call to memmap_init_zone should have already taken care @@ -5615,53 +5722,13 @@ void __ref memmap_init_zone_device(struct zone *zone, */ if (pgmap->altmap_valid) { struct vmem_altmap *altmap = &pgmap->altmap; + unsigned long end_pfn = start_pfn + size; start_pfn = altmap->base_pfn + vmem_altmap_offset(altmap); size = end_pfn - start_pfn; } - for (pfn = start_pfn; pfn < end_pfn; pfn++) { - struct page *page = pfn_to_page(pfn); - - __init_single_page(page, pfn, zone_idx, nid); - - /* - * Mark page reserved as it will need to wait for onlining - * phase for it to be fully associated with a zone. - * - * We can use the non-atomic __set_bit operation for setting - * the flag as we are still initializing the pages. - */ - __SetPageReserved(page); - - /* - * ZONE_DEVICE pages union ->lru with a ->pgmap back - * pointer and hmm_data. It is a bug if a ZONE_DEVICE - * page is ever freed or placed on a driver-private list. - */ - page->pgmap = pgmap; - page->hmm_data = 0; - - /* - * Mark the block movable so that blocks are reserved for - * movable at startup. This will force kernel allocations - * to reserve their blocks rather than leaking throughout - * the address space during boot when many long-lived - * kernel allocations are made. - * - * bitmap is created for zone's valid pfn range. but memmap - * can be created for invalid pages (for alignment) - * check here not to call set_pageblock_migratetype() against - * pfn out of zone. - * - * Please note that MEMMAP_HOTPLUG path doesn't clear memmap - * because this is done early in sparse_add_one_section - */ - if (!(pfn & (pageblock_nr_pages - 1))) { - set_pageblock_migratetype(page, MIGRATE_MOVABLE); - cond_resched(); - } - } + __memmap_init_hotplug(size, nid, zone_idx, start_pfn, pgmap); pr_info("%s initialised, %lu pages in %ums\n", dev_name(pgmap->dev), size, jiffies_to_msecs(jiffies - start));