From patchwork Thu Jun 15 11:03:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yajun Deng X-Patchwork-Id: 13281065 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 470A0EB64DC for ; Thu, 15 Jun 2023 11:04:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 958C46B0072; Thu, 15 Jun 2023 07:04:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 908C88E0002; Thu, 15 Jun 2023 07:04:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7F8CC8E0001; Thu, 15 Jun 2023 07:04:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 71A0D6B0072 for ; Thu, 15 Jun 2023 07:04:18 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 15581120BF3 for ; Thu, 15 Jun 2023 11:04:18 +0000 (UTC) X-FDA: 80904698196.14.B337637 Received: from out-38.mta0.migadu.com (out-38.mta0.migadu.com [91.218.175.38]) by imf15.hostedemail.com (Postfix) with ESMTP id BF6DAA002F for ; Thu, 15 Jun 2023 11:04:14 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=IwVTiMVl; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf15.hostedemail.com: domain of yajun.deng@linux.dev designates 91.218.175.38 as permitted sender) smtp.mailfrom=yajun.deng@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686827055; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=HXlRY/NnvVbWtv6wBhmhQHVgYiXefGsRRYFYStrDpdY=; b=EDQpROxKLGze7kiPEYAltB5d9SbMTLB4wENRLmEZtSHwOh1CzgwtL3BDUHhyg6YhZfWBy5 u7b4mXmimplPB/sbfS+BV8LHKK6l6IDTSHv9/rZ3I4mfYjKP0NWJzihhAfpzTpcKdwC0n1 pri2SYaMnnwVzugbuUQvn3HqdpbyfCo= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=IwVTiMVl; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf15.hostedemail.com: domain of yajun.deng@linux.dev designates 91.218.175.38 as permitted sender) smtp.mailfrom=yajun.deng@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686827055; a=rsa-sha256; cv=none; b=GDVs0r/OSAyAUhxg2Hlke8y4VqwOPdDsixyGejC2y0erigRqUT0b5zRxKr+rF9BHRBa3fg 4lW476GQHBZF4c3jUjoBvcaY2AaDkEUuESEXgNXyhb2J0Nf5YDKfnDrO4Ck2G9N56GduSi wifbBwjz6TtxrYRisbgsyfbDi4Blj24= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686827047; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=HXlRY/NnvVbWtv6wBhmhQHVgYiXefGsRRYFYStrDpdY=; b=IwVTiMVlmdoBf+k0CgBzwHvE5l+QAJfdPLPFzzIto9INy4+dzmm8mVOLpIgIe935EyPXo5 Eni9VHiT3AR3SV5BhfM8g9s0PLUHo26qpRkOgcK4A6KcE8Q3IXe2PyIx4K5mGmYh+Xz3Ys SdiZ2JNbST6+KC80ypO+aVW0mi3VimQ= From: Yajun Deng To: akpm@linux-foundation.org, rppt@kernel.org Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yajun Deng Subject: [PATCH] mm: pass nid to reserve_bootmem_region() Date: Thu, 15 Jun 2023 19:03:55 +0800 Message-Id: <20230615110355.1694461-1-yajun.deng@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: BF6DAA002F X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 9y34qxup5n7uomm9db8knf1edq3m8n3r X-HE-Tag: 1686827054-194602 X-HE-Meta: U2FsdGVkX1+HToSViB8s8te7WNKpkOpEKfqwEnLO5RZart/WoTM4J2T51X4/IQDPvcqmZmQVTaaOV25ZHet7XHusMlzyJZymB/nOU7ZB5RXq3JKqA5p9O+mUxYgR01dSgkFroJSo+9naK63svsZoRROkZN7gKTFSExDCaWBPdg14eo51j8vWo6B304tL29tYZLJnbZSQ9JVlzXo06TZxLahOwnNyuGJ8WKvJUylvZkol+EjhAj4RRxckRCSYrGN9wODIIjN9XcmRoXOXQkDrIKkXb+C2sOnZ4yvxmKRsXNNCfECWmsEVmSsKKAUbtCjYqw2+k8xiSatJ6771CXEHfk96oTR/4M0YkWLf4R5vJKUetgI3cuppNDLXylIQ1sKMRVBlnnCkGjTo00bZSVXvnFj2npxH3FCu9vs5arpSTTB32kg8yOqEzAJeSiFoQvi7/jdmxIfNkKdAeuxgzS/XYUzUegr0gpasFOEDqrVt1875rWCeEN4SFRHSHPxc4JRlLOGA+RAVN2xga1HQOubw/0ugPgVg9UwLooorxLttNuOqc4ncG/QfxuQq6/m8jCCJNUBFvzZCfrO1SVsH2b9qvQpkEG310wbMZuKm5gMQVY8Uha6orqCXVJ/GfB+c0zRhl2j71yuL9avynQUfAoT4CXEPxCwd+Pbg5g2of56tHhzAFIxGkdgqhDeX9AJ9YmzrslfpHyZKq4LhXxRygRjVRKd0G+NMRlEnQEtl4LcE6hTeqMsVS4QqHueXXuytPLHmzGKWN14wnsx7vZ0ArRS2jg51MQCIklAAc1ufIx75UYqGxNcEihEEUQmZZAC0MJccftwIGVHgku1dmw4JLgRFjAYyog8ew4vqXmy/kJYUqOlKGhrBxzTqNfNukzSFzUpJivzot69R831LkIEPzxJyXq41QKjRxM5p1x0GaSXrb+AWl35URlpkCyIxwVG2xAQ9BXOPK2Q2hAoZm0Rm3dW AvQjrjgx faAtz3RVrlI4lsSSVXT9Vs8lWs71ap9xHAiM+aiNd1/AkDBaytn/cDHOEck8WHOlAPUCwUyYCUPpv3G5q5Kxi2OcP24cjtHOc98+bn8awFbj+YRRtDJmL7A09G0ShhTXWOrku X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: early_pfn_to_nid() is called frequently in init_reserved_page(), it is used to convert PFN to node id. These PFN may belong to the same memory region, they have the same node id. It is not necessary to call early_pfn_to_nid() for each PFN. Pass nid to reserve_bootmem_region() and drop the call to early_pfn_to_nid() in init_reserved_page(). The following data was tested on my machine, it has 190GB RAM, and enable CONFIG_DEFERRED_STRUCT_PAGE_INIT. before the patch: memmap_init_reserved_pages() 67ms after the patch: memmap_init_reserved_pages() 20ms Signed-off-by: Yajun Deng --- include/linux/mm.h | 3 ++- mm/memblock.c | 9 ++++++--- mm/mm_init.c | 31 +++++++++++++++++++------------ 3 files changed, 27 insertions(+), 16 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 17317b1673b0..39e72ca6bf22 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2964,7 +2964,8 @@ extern unsigned long free_reserved_area(void *start, void *end, extern void adjust_managed_page_count(struct page *page, long count); -extern void reserve_bootmem_region(phys_addr_t start, phys_addr_t end); +extern void reserve_bootmem_region(phys_addr_t start, + phys_addr_t end, int nid); /* Free the reserved page into the buddy system, so it gets managed. */ static inline void free_reserved_page(struct page *page) diff --git a/mm/memblock.c b/mm/memblock.c index ff0da1858778..9e9b9a3473f9 100644 --- a/mm/memblock.c +++ b/mm/memblock.c @@ -2091,18 +2091,21 @@ static void __init memmap_init_reserved_pages(void) { struct memblock_region *region; phys_addr_t start, end; + int nid; u64 i; /* initialize struct pages for the reserved regions */ - for_each_reserved_mem_range(i, &start, &end) - reserve_bootmem_region(start, end); + __for_each_mem_range(i, &memblock.reserved, NULL, NUMA_NO_NODE, + MEMBLOCK_NONE, &start, &end, &nid) + reserve_bootmem_region(start, end, nid); /* and also treat struct pages for the NOMAP regions as PageReserved */ for_each_mem_region(region) { if (memblock_is_nomap(region)) { start = region->base; end = start + region->size; - reserve_bootmem_region(start, end); + nid = region->nid; + reserve_bootmem_region(start, end, nid); } } } diff --git a/mm/mm_init.c b/mm/mm_init.c index d393631599a7..1499efbebc6f 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -646,10 +646,8 @@ static inline void pgdat_set_deferred_range(pg_data_t *pgdat) } /* Returns true if the struct page for the pfn is initialised */ -static inline bool __meminit early_page_initialised(unsigned long pfn) +static inline bool __meminit early_page_initialised(unsigned long pfn, int nid) { - int nid = early_pfn_to_nid(pfn); - if (node_online(nid) && pfn >= NODE_DATA(nid)->first_deferred_pfn) return false; @@ -695,15 +693,14 @@ defer_init(int nid, unsigned long pfn, unsigned long end_pfn) return false; } -static void __meminit init_reserved_page(unsigned long pfn) +static void __meminit init_reserved_page(unsigned long pfn, int nid) { pg_data_t *pgdat; - int nid, zid; + int zid; - if (early_page_initialised(pfn)) + if (early_page_initialised(pfn, nid)) return; - nid = early_pfn_to_nid(pfn); pgdat = NODE_DATA(nid); for (zid = 0; zid < MAX_NR_ZONES; zid++) { @@ -717,7 +714,7 @@ static void __meminit init_reserved_page(unsigned long pfn) #else static inline void pgdat_set_deferred_range(pg_data_t *pgdat) {} -static inline bool early_page_initialised(unsigned long pfn) +static inline bool early_page_initialised(unsigned long pfn, int nid) { return true; } @@ -727,7 +724,7 @@ static inline bool defer_init(int nid, unsigned long pfn, unsigned long end_pfn) return false; } -static inline void init_reserved_page(unsigned long pfn) +static inline void init_reserved_page(unsigned long pfn, int nid) { } #endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ @@ -738,16 +735,20 @@ static inline void init_reserved_page(unsigned long pfn) * marks the pages PageReserved. The remaining valid pages are later * sent to the buddy page allocator. */ -void __meminit reserve_bootmem_region(phys_addr_t start, phys_addr_t end) +void __meminit reserve_bootmem_region(phys_addr_t start, + phys_addr_t end, int nid) { unsigned long start_pfn = PFN_DOWN(start); unsigned long end_pfn = PFN_UP(end); + if (nid == MAX_NUMNODES) + nid = first_online_node; + for (; start_pfn < end_pfn; start_pfn++) { if (pfn_valid(start_pfn)) { struct page *page = pfn_to_page(start_pfn); - init_reserved_page(start_pfn); + init_reserved_page(start_pfn, nid); /* Avoid false-positive PageTail() */ INIT_LIST_HEAD(&page->lru); @@ -2579,7 +2580,13 @@ void __init set_dma_reserve(unsigned long new_dma_reserve) void __init memblock_free_pages(struct page *page, unsigned long pfn, unsigned int order) { - if (!early_page_initialised(pfn)) + int nid = 0; + +#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT + nid = early_pfn_to_nid(pfn); +#endif + + if (!early_page_initialised(pfn, nid)) return; if (!kmsan_memblock_free_pages(page, order)) { /* KMSAN will take care of these pages. */