From patchwork Mon Nov 5 21:19:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 10669195 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D4D2C1751 for ; Mon, 5 Nov 2018 21:19:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CBE0929C30 for ; Mon, 5 Nov 2018 21:19:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BF5FC29C5E; Mon, 5 Nov 2018 21:19:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0D2EB29C30 for ; Mon, 5 Nov 2018 21:19:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 66FC26B0271; Mon, 5 Nov 2018 16:19:53 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 623746B0272; Mon, 5 Nov 2018 16:19:53 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E9B96B0273; Mon, 5 Nov 2018 16:19:53 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id 0C40D6B0271 for ; Mon, 5 Nov 2018 16:19:53 -0500 (EST) Received: by mail-pg1-f199.google.com with SMTP id 134-v6so9798200pga.1 for ; Mon, 05 Nov 2018 13:19:52 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:from :to:cc:date:message-id:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=80JIapUxHszk8aavLj+kZ1oAph9NS5Bu/DewSKjr+2M=; b=fxr8JKtr5eHf7OZKtsnKjpQ0iK30St1S7ldp2dvw3EgPsEEQv67IK9CloKQ5oAEOaf eaVNl0ASBDUlSqWxpxca8SOtcY31jkZ70rCKXOJwPbXN1rVqlrb0dbmLznB5zVlvEo0y UZeyCKLuXwk3Nzc2+ZH/ppWlPmRHkF4ru3zFUJNtQGDFO9WPxNuauAtcwwRwc8J6+RwP ZJzgL2mK7jN2LMILrzOhsL9Tcw73pQwC64SqZiKZfFEmXP4Gws2fgnchx3th8L7aPYOI PEqiMkiHehqdJrOpszL6N6NjNfCsRC2LWyQ/QiIt65OwWInth75UvBQjPrEhu55ZfTgO jjSw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Gm-Message-State: AGRZ1gLyxhG6NC1OYSg+8ZF4iUK1Afr6/jS3MYl9JfOI41Rcq4Fv2NtK 6MSjy6ryfbYXfGa433xyNzzfrYT8dhWrqK6tLwyLBq1rOgQt8CvaIFT9GVtXSjhnlB8//3W41VC 6gVxTsf+QBsBTKNF5Exk118s5rTdKWr152dDqkMSr6GTQOL3Xj1KOW+48f+c9qw/Hpg== X-Received: by 2002:a63:5c61:: with SMTP id n33-v6mr20611909pgm.1.1541452792602; Mon, 05 Nov 2018 13:19:52 -0800 (PST) X-Google-Smtp-Source: AJdET5eTftPdLrCQELchzlZ8ulU+1FhSxE4AYKDXDmTDqeekZsyahrvI74O8RKxd00GuXBL2AE0g X-Received: by 2002:a63:5c61:: with SMTP id n33-v6mr20611852pgm.1.1541452791528; Mon, 05 Nov 2018 13:19:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1541452791; cv=none; d=google.com; s=arc-20160816; b=g/YgprEZebALQjNRHiDtQh9VWLM3eXIFNIdZ1BrPPOjZF9bCzli9b+2ekK5qIYvClg XCgv+LQUDgtyrXKlTAbw6azIBWlzuAWYlIuWQSJFip526VTyFRYGuehFTQ2zxFfsUy9y hh3RvoxFHVO7AiTogL4YPcGQdkk64/WgNWSy/f8VYjeNH00eN1DimvKj9pZvdDwfVmKg 6BmstsKnYAEXGe/HLgoa1dFsRoqTHzzkN6oznCBG4D+So1Mavj5Q93C04JfUFeAbPYn9 CDKpPELAryEOS0OvpQHGUSFDqcLlkXXouNb2ZQJNYQYRWn7bRvd+bsh1TMx+SBgl1C1j 1PcQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:user-agent:references :in-reply-to:message-id:date:cc:to:from:subject; bh=80JIapUxHszk8aavLj+kZ1oAph9NS5Bu/DewSKjr+2M=; b=Zq28xMrg7sjZ6gwWUbIdlorvkz0sdRIe62cepQicUN1faXAz3F6fHEfDOjg3rwSXDl Zb06sMu68hApJxEnHw2isV4Yn2XBds01TZyCMcCZyVY/ndb9DICuHpjP4WgfLlC9drrj yvQ3U5P1+qDjGx9TsE8F98fXl3l3uO9AXBUHwZ9H2+PwABK2v3BgvORHr034zFxzPV/K oPnUCJRqz1mD/BT8KC2QAsgGUUskIhICYkPRcL0rm0dRwnzfV8KP8QWV33tDl4DYiRrV YupO7i46tuT1fyVXJL2zeViUkxRbSWmR9RuThWFQgnXf8rKZgwIstyVEG9IHcKW0Wfo8 uZKg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from mga03.intel.com (mga03.intel.com. [134.134.136.65]) by mx.google.com with ESMTPS id a93-v6si42301254pla.226.2018.11.05.13.19.51 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 05 Nov 2018 13:19:51 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.65 as permitted sender) client-ip=134.134.136.65; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of alexander.h.duyck@linux.intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=alexander.h.duyck@linux.intel.com; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Nov 2018 13:19:51 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,469,1534834800"; d="scan'208";a="106132927" Received: from ahduyck-desk1.jf.intel.com ([10.7.198.76]) by orsmga002.jf.intel.com with ESMTP; 05 Nov 2018 13:19:50 -0800 Subject: [mm PATCH v5 5/7] mm: Move hot-plug specific memory init into separate functions and optimize From: Alexander Duyck To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvdimm@lists.01.org, davem@davemloft.net, pavel.tatashin@microsoft.com, mhocko@suse.com, mingo@kernel.org, kirill.shutemov@linux.intel.com, dan.j.williams@intel.com, dave.jiang@intel.com, alexander.h.duyck@linux.intel.com, rppt@linux.vnet.ibm.com, willy@infradead.org, vbabka@suse.cz, khalid.aziz@oracle.com, ldufour@linux.vnet.ibm.com, mgorman@techsingularity.net, yi.z.zhang@linux.intel.com, alexander.h.duyck@linux.intel.com Date: Mon, 05 Nov 2018 13:19:50 -0800 Message-ID: <154145279094.30046.504725873397414096.stgit@ahduyck-desk1.jf.intel.com> In-Reply-To: <154145268025.30046.11742652345962594283.stgit@ahduyck-desk1.jf.intel.com> References: <154145268025.30046.11742652345962594283.stgit@ahduyck-desk1.jf.intel.com> User-Agent: StGit/unknown-version MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch is going through and combining the bits in memmap_init_zone and memmap_init_zone_device that are related to hotplug into a single function called __memmap_init_hotplug. I also took the opportunity to integrate __init_single_page's functionality into this function. In doing so I can get rid of some of the redundancy such as the LRU pointers versus the pgmap. Signed-off-by: Alexander Duyck Reviewed-by: Pavel Tatashin --- mm/page_alloc.c | 214 +++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 144 insertions(+), 70 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3466a01ed90a..dbe00c1a0e23 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1192,6 +1192,92 @@ static void __meminit __init_single_page(struct page *page, unsigned long pfn, #endif } +static void __meminit __init_pageblock(unsigned long start_pfn, + unsigned long nr_pages, + unsigned long zone, int nid, + struct dev_pagemap *pgmap) +{ + unsigned long nr_pgmask = pageblock_nr_pages - 1; + struct page *start_page = pfn_to_page(start_pfn); + unsigned long pfn = start_pfn + nr_pages - 1; +#ifdef WANT_PAGE_VIRTUAL + bool is_highmem = is_highmem_idx(zone); +#endif + struct page *page; + + /* + * Enforce the following requirements: + * size > 0 + * size < pageblock_nr_pages + * start_pfn -> pfn does not cross pageblock_nr_pages boundary + */ + VM_BUG_ON(((start_pfn ^ pfn) | (nr_pages - 1)) > nr_pgmask); + + /* + * Work from highest page to lowest, this way we will still be + * warm in the cache when we call set_pageblock_migratetype + * below. + * + * The loop is based around the page pointer as the main index + * instead of the pfn because pfn is not used inside the loop if + * the section number is not in page flags and WANT_PAGE_VIRTUAL + * is not defined. + */ + for (page = start_page + nr_pages; page-- != start_page; pfn--) { + mm_zero_struct_page(page); + + /* + * We use the start_pfn instead of pfn in the set_page_links + * call because of the fact that the pfn number is used to + * get the section_nr and this function should not be + * spanning more than a single section. + */ + set_page_links(page, zone, nid, start_pfn); + init_page_count(page); + page_mapcount_reset(page); + page_cpupid_reset_last(page); + + /* + * We can use the non-atomic __set_bit operation for setting + * the flag as we are still initializing the pages. + */ + __SetPageReserved(page); + + /* + * ZONE_DEVICE pages union ->lru with a ->pgmap back + * pointer and hmm_data. It is a bug if a ZONE_DEVICE + * page is ever freed or placed on a driver-private list. + */ + page->pgmap = pgmap; + if (!pgmap) + INIT_LIST_HEAD(&page->lru); + +#ifdef WANT_PAGE_VIRTUAL + /* The shift won't overflow because ZONE_NORMAL is below 4G. */ + if (!is_highmem) + set_page_address(page, __va(pfn << PAGE_SHIFT)); +#endif + } + + /* + * Mark the block movable so that blocks are reserved for + * movable at startup. This will force kernel allocations + * to reserve their blocks rather than leaking throughout + * the address space during boot when many long-lived + * kernel allocations are made. + * + * bitmap is created for zone's valid pfn range. but memmap + * can be created for invalid pages (for alignment) + * check here not to call set_pageblock_migratetype() against + * pfn out of zone. + * + * Please note that MEMMAP_HOTPLUG path doesn't clear memmap + * because this is done early in sparse_add_one_section + */ + if (!(start_pfn & nr_pgmask)) + set_pageblock_migratetype(start_page, MIGRATE_MOVABLE); +} + #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT static void __meminit init_reserved_page(unsigned long pfn) { @@ -5513,6 +5599,25 @@ overlap_memmap_init(unsigned long zone, unsigned long *pfn) return false; } +static void __meminit __memmap_init_hotplug(unsigned long size, int nid, + unsigned long zone, + unsigned long start_pfn, + struct dev_pagemap *pgmap) +{ + unsigned long pfn = start_pfn + size; + + while (pfn != start_pfn) { + unsigned long stride = pfn; + + pfn = max(ALIGN_DOWN(pfn - 1, pageblock_nr_pages), start_pfn); + stride -= pfn; + + __init_pageblock(pfn, stride, zone, nid, pgmap); + + cond_resched(); + } +} + /* * Initially all pages are reserved - free ones are freed * up by memblock_free_all() once the early boot process is @@ -5523,49 +5628,59 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone, struct vmem_altmap *altmap) { unsigned long pfn, end_pfn = start_pfn + size; - struct page *page; if (highest_memmap_pfn < end_pfn - 1) highest_memmap_pfn = end_pfn - 1; + if (context == MEMMAP_HOTPLUG) { #ifdef CONFIG_ZONE_DEVICE - /* - * Honor reservation requested by the driver for this ZONE_DEVICE - * memory. We limit the total number of pages to initialize to just - * those that might contain the memory mapping. We will defer the - * ZONE_DEVICE page initialization until after we have released - * the hotplug lock. - */ - if (zone == ZONE_DEVICE) { - if (!altmap) - return; + /* + * Honor reservation requested by the driver for this + * ZONE_DEVICE memory. We limit the total number of pages to + * initialize to just those that might contain the memory + * mapping. We will defer the ZONE_DEVICE page initialization + * until after we have released the hotplug lock. + */ + if (zone == ZONE_DEVICE) { + if (!altmap) + return; + + if (start_pfn == altmap->base_pfn) + start_pfn += altmap->reserve; + end_pfn = altmap->base_pfn + + vmem_altmap_offset(altmap); + } +#endif + /* + * For these ZONE_DEVICE pages we don't need to record the + * pgmap as they should represent only those pages used to + * store the memory map. The actual ZONE_DEVICE pages will + * be initialized later. + */ + __memmap_init_hotplug(end_pfn - start_pfn, nid, zone, + start_pfn, NULL); - if (start_pfn == altmap->base_pfn) - start_pfn += altmap->reserve; - end_pfn = altmap->base_pfn + vmem_altmap_offset(altmap); + return; } -#endif for (pfn = start_pfn; pfn < end_pfn; pfn++) { + struct page *page; + /* * There can be holes in boot-time mem_map[]s handed to this * function. They do not exist on hotplugged memory. */ - if (context == MEMMAP_EARLY) { - if (!early_pfn_valid(pfn)) - continue; - if (!early_pfn_in_nid(pfn, nid)) - continue; - if (overlap_memmap_init(zone, &pfn)) - continue; - if (defer_init(nid, pfn, end_pfn)) - break; - } + if (!early_pfn_valid(pfn)) + continue; + if (!early_pfn_in_nid(pfn, nid)) + continue; + if (overlap_memmap_init(zone, &pfn)) + continue; + if (defer_init(nid, pfn, end_pfn)) + break; page = pfn_to_page(pfn); __init_single_page(page, pfn, zone, nid); - if (context == MEMMAP_HOTPLUG) - __SetPageReserved(page); /* * Mark the block movable so that blocks are reserved for @@ -5592,7 +5707,6 @@ void __ref memmap_init_zone_device(struct zone *zone, unsigned long size, struct dev_pagemap *pgmap) { - unsigned long pfn, end_pfn = start_pfn + size; struct pglist_data *pgdat = zone->zone_pgdat; unsigned long zone_idx = zone_idx(zone); unsigned long start = jiffies; @@ -5608,53 +5722,13 @@ void __ref memmap_init_zone_device(struct zone *zone, */ if (pgmap->altmap_valid) { struct vmem_altmap *altmap = &pgmap->altmap; + unsigned long end_pfn = start_pfn + size; start_pfn = altmap->base_pfn + vmem_altmap_offset(altmap); size = end_pfn - start_pfn; } - for (pfn = start_pfn; pfn < end_pfn; pfn++) { - struct page *page = pfn_to_page(pfn); - - __init_single_page(page, pfn, zone_idx, nid); - - /* - * Mark page reserved as it will need to wait for onlining - * phase for it to be fully associated with a zone. - * - * We can use the non-atomic __set_bit operation for setting - * the flag as we are still initializing the pages. - */ - __SetPageReserved(page); - - /* - * ZONE_DEVICE pages union ->lru with a ->pgmap back - * pointer and hmm_data. It is a bug if a ZONE_DEVICE - * page is ever freed or placed on a driver-private list. - */ - page->pgmap = pgmap; - page->hmm_data = 0; - - /* - * Mark the block movable so that blocks are reserved for - * movable at startup. This will force kernel allocations - * to reserve their blocks rather than leaking throughout - * the address space during boot when many long-lived - * kernel allocations are made. - * - * bitmap is created for zone's valid pfn range. but memmap - * can be created for invalid pages (for alignment) - * check here not to call set_pageblock_migratetype() against - * pfn out of zone. - * - * Please note that MEMMAP_HOTPLUG path doesn't clear memmap - * because this is done early in sparse_add_one_section - */ - if (!(pfn & (pageblock_nr_pages - 1))) { - set_pageblock_migratetype(page, MIGRATE_MOVABLE); - cond_resched(); - } - } + __memmap_init_hotplug(size, nid, zone_idx, start_pfn, pgmap); pr_info("%s initialised, %lu pages in %ums\n", dev_name(pgmap->dev), size, jiffies_to_msecs(jiffies - start));