From patchwork Sun Mar 19 21:59:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13180630 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C95FC7618A for ; Sun, 19 Mar 2023 22:00:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 79F62900004; Sun, 19 Mar 2023 18:00:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 74F22900002; Sun, 19 Mar 2023 18:00:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 61693900004; Sun, 19 Mar 2023 18:00:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5293F900002 for ; Sun, 19 Mar 2023 18:00:30 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 18D25AAC96 for ; Sun, 19 Mar 2023 22:00:30 +0000 (UTC) X-FDA: 80587017420.23.C3A69C8 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by imf18.hostedemail.com (Postfix) with ESMTP id 595841C0003 for ; Sun, 19 Mar 2023 22:00:28 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=SIhxwyq+; spf=pass (imf18.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679263228; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jZHqPuHVpfoZUyjzDHVtR0tT/F6Ir3USKAq71EN+Dvg=; b=goO2XdWAK34ipBx5mfNsCbqhsD3T55W9BzzT3RKAr3LYayeC7gW9JTGvcKxIGZ8y2e1V5D yBxCzrfqPMIdQ3R7UGCn67p6gTYgCC8YsIWCLvA2rvIJQ8nZjZR8gims/G9iQQn5wQq4U9 kddPauL2QGp6QJx8Xt9CFvXKyDQOfxQ= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=SIhxwyq+; spf=pass (imf18.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679263228; a=rsa-sha256; cv=none; b=JzcSFEYAjSArKLCeTtLVaZJOlDWR/3dXv7TSQj5mqgSlHZw7sDiDnef6GvtZgaLVuuHz33 XjPFjN5PECtl8SqEeCct8jIyWUtHi4qVKv+w+CkgeU28BRgBkSFCXPzGKjrjMsGgUzZEwR SSe+XlwVd8DdcCdPBTRDiPstsjJJ320= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id F0CBFB80D2A; Sun, 19 Mar 2023 22:00:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B5F0AC433EF; Sun, 19 Mar 2023 22:00:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679263225; bh=tJwEQcdAqnhTzTScV6Pn/bPL/i/f+PX1TkWeIm6dtaQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SIhxwyq+GXe8QtD19WP50iL/3TUxA3hcV1snjRx1EWasCnQvoYx+gXOBmXwm96sg5 idENCYSVuI/0QLkBoOaZNj9tcO/gbZY2CUvMAMKXzcdwumPoeqJyR1Y6YW7txiLZ5J jat66Riix/3x5oYZY3cKZZWbpV692n4fK9XW1YizM20xKlzR1YiSKExvkC/GQYJWsJ peMbK1qKOqSDBP7Eud05E+83vXAaT2/44N/PYn6gkj1wMfDX5vZEa3gxH1zlv1a7K/ Ysj8PQvzdUTkMugRPxoUWcwQZ1qsM3Nf3O1tRTLj9G78Yr8WO3sYXC/zEmy9SPa3j2 Hx+n+hHWWZHwA== From: Mike Rapoport To: Andrew Morton Cc: David Hildenbrand , Matthew Wilcox , Mel Gorman , Michal Hocko , Mike Rapoport , Thomas Bogendoerfer , Vlastimil Babka , linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 01/15] mips: fix comment about pgtable_init() Date: Sun, 19 Mar 2023 23:59:54 +0200 Message-Id: <20230319220008.2138576-2-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230319220008.2138576-1-rppt@kernel.org> References: <20230319220008.2138576-1-rppt@kernel.org> MIME-Version: 1.0 X-Stat-Signature: f4yatnqixaie3zrzz5seiktctuz4xzbg X-Rspam-User: X-Rspamd-Queue-Id: 595841C0003 X-Rspamd-Server: rspam06 X-HE-Tag: 1679263228-970158 X-HE-Meta: U2FsdGVkX19ceQ5PyVScUqw30CeZerWko/gC+Qbys709K8zdAPrJXvbewvkUsWf/SOnnLVvrbPNzY848SS6bIT640f8FZtYkn/EAsH9uQHh/SwRVgoia10a7x4xckSsSe9MoRWjMheMNyMmrgZ0Kgr7iVOElLFn35bO1NGeM/WSXWigWGwZiXiTHqs0F8OdV3PNfDkLA4nJTQsTQiDadbguNyFFV+0auwrhsGzq4VPPkJC1+0/Dvj22jMwvL5cYqDC0V1t7GdB+1aaPZeGF6EvVuNT0YA2L+LUOAWAAdgtgoBh2W6k/s7n8347QStQ4JdKhZ9JFaaduo80wQL42daIfTS4ulyd6P77/4fpOO14kxTcVy431MLwAFbH7xvO9NQWJojfOgsvoiluPMYRffFsiI8a5I2ElzbfqPhUdNraJOFBcQ0iTMJsXVe7wtF/UleskAsyVOEF1gD6vW66sZviQJwZVs+YEj32LauND4n/WKUITG4p8ZgjCZHqfT2cjd1S4rBxQJ3W+1YEHUvPn5eTc0GWtOdWEYqbuvgwvx2SJ/ckj1ENdlMjNUGLA5QKB24Wmj9yGIyzaNQm7MPhRFYbeDqK/GbYfIJzxHkdrb0aZ7gUlj+VdPVCVSkdmeIWpPLLpsc/yU1Xl/2ILe6z5SUDaiqZoTFG/TBIe/oeVUA6cvrhPRSmFEL9rv8zXDdKU6N5Z2/mugxSj3Vz2dJiSlNZTGoEzLrrjrmzj9AyWTxiz9ynglRVMICpx7dR3wLv4ka1rpl7nTvVFW+Z4ZJiDk9nuf3SzxcTdyBbUsaMbtxFwIBG4ic1yo98N2Lxn15j8BAq2X5iCN69MfzhcXGjSW3rKei+jl1rHIOdTSrxbeHtjkZm8kMdfq6rQY+WCunFIZ1mzfwbQEzCewWuUbKE4CUs3ZhLNl3BZRQt3l5AO1A6EP/eIJ+lBnEeFODO5NItSuXYVfj9mdBmrcrdbc2rG sezzMkFH NqXRuPaNB095S/2Ws6HfyUTf2ZXifAx4ra3CJDskm5GeIWkuj6UlfAIQl0a5RlYOMSjgHrSaElfnPo7v9VV5n7cghMtb1wLdm+KSeWlaDtT6mKz20pJf//z708xbVebYzjNRqkdKk2TEGnI8cpuDMVZvWB6JFZJHm36WGz2i/zyKH5rPvVEzXptFOLu+KPBaNZDYqZBcZiG72fIrmBGgXnWxod/NVQcKiOnCaq6SvMhVfdRccV6c05gi90ML6ATHHfjYh8GIazaA8h55vazy6lRwytmLkZ/5DewlGiNpcei+McTQu47yqDCQ9WBd19UQXn6nTSc1RMAy2PMEhvOHC0LhYSYSp/xUBujhfLWBoBdx2pkCYUJNa0Tw+8rRwrOEh84tc7YhkZ0fGXjEiD07pB2LROQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Mike Rapoport (IBM)" Comment about fixrange_init() says that its called from pgtable_init() while the actual caller is pagetabe_init(). Update comment to match the code. Signed-off-by: Mike Rapoport (IBM) Reviewed-by: David Hildenbrand --- arch/mips/include/asm/fixmap.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/mips/include/asm/fixmap.h b/arch/mips/include/asm/fixmap.h index beea14761cef..b037718d7e8b 100644 --- a/arch/mips/include/asm/fixmap.h +++ b/arch/mips/include/asm/fixmap.h @@ -70,7 +70,7 @@ enum fixed_addresses { #include /* - * Called from pgtable_init() + * Called from pagetable_init() */ extern void fixrange_init(unsigned long start, unsigned long end, pgd_t *pgd_base); From patchwork Sun Mar 19 21:59:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13180631 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3142C76195 for ; Sun, 19 Mar 2023 22:00:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 52D9D900005; Sun, 19 Mar 2023 18:00:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4DDBC900002; Sun, 19 Mar 2023 18:00:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3A5E5900005; Sun, 19 Mar 2023 18:00:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 2B39A900002 for ; Sun, 19 Mar 2023 18:00:34 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C18C340460 for ; Sun, 19 Mar 2023 22:00:33 +0000 (UTC) X-FDA: 80587017546.19.F3877D1 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by imf29.hostedemail.com (Postfix) with ESMTP id E5B07120013 for ; Sun, 19 Mar 2023 22:00:31 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=oXkUKnzy; spf=pass (imf29.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679263232; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=26pZEosbU4xGB6qgXEMEJK0JR82wYmZqc6u/6bp3w5o=; b=DwPGz+spF2RKtYgpR74VZcMPBMmzpK7BvVLjR4MsKy6TgCRU+/aYLoFHkleiHrxmlWbn9o f6d+SMO1PtIMva779WaA8jEkS6vqu6uLy8wQqQze/feAtLNfQg5BmlOZ7vuzO5Dliz7ANf McnpAF8CUdzdrpj/RDIOsQ3kZQiR6vM= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=oXkUKnzy; spf=pass (imf29.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679263232; a=rsa-sha256; cv=none; b=0Tp/qH1JKlzaglOJjsLUp4fhoPAJEkhV6eKDwSIY3n3o2hdBhBFd/1XSfXN6sMYVV7+7qQ S9bLQEP4dO9gnKTugokFqIaeNWti1GzLJDRPpRJMluLxtRPikZLiOMS+gTS5dkwq9S+avR 2istSXYtqB+oWgVw4zW92MsOHe45sDA= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 8C56CB80D28; Sun, 19 Mar 2023 22:00:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4C11FC433A0; Sun, 19 Mar 2023 22:00:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679263229; bh=zekLhry5fbzdsyyFVJESg5+PbdXiqA638EXDFWo0PPs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oXkUKnzydDJ71mPtZtGDkbDpHakn4+vm9e5hsj9ZXIp26esqmpKA0HKya3tWcGwU9 hHaJ6/V57EkzRpGi8lnjrEYAjO5VItMs/Md0bbdadjwge+a9/WTyHI1abD1pUrKKYQ h9K7yRauBXU7Tw8Mq2jgvBEw9AE21CER4YwrQv5CWjBxfQ9UHdb/tQXfSG95r6LaJQ vBhjliYse44G7HvTmgehhaqQXcvX1kGezbx65UwsYMW0fffJYTYM3h1pRApRzerWw0 c/X95FgbZkXr+iJhGJ6iUvnfoAIV/WrZXrD5FY4iZUFuWW7Tdw/Lw0s9FGHu1TLONU lRE/6FdnqUlYQ== From: Mike Rapoport To: Andrew Morton Cc: David Hildenbrand , Matthew Wilcox , Mel Gorman , Michal Hocko , Mike Rapoport , Thomas Bogendoerfer , Vlastimil Babka , linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 02/15] mm/cma: move init_cma_reserved_pages() to cma.c and make it static Date: Sun, 19 Mar 2023 23:59:55 +0200 Message-Id: <20230319220008.2138576-3-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230319220008.2138576-1-rppt@kernel.org> References: <20230319220008.2138576-1-rppt@kernel.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: E5B07120013 X-Stat-Signature: n85d3eqo7q1qsesdz88uuypcgspt7xhy X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1679263231-637754 X-HE-Meta: U2FsdGVkX18czncFOtjsoBLwgdGVitvFGyuTHHqkNreLV6W+cCI9/uRwHhC8HGPWTPjJHMdaNaze5tR72765KItwISveK1mP7RNqvUKSGorDSssT0Lz0oycd4+Io78ajemZRDnRZOKwDgIJDY28z2tMTRcgluvzBGCJVWXti3a0XRGcFqINIhMlQkTnW8zGsKiZMz/33F3HZIOGkn8X6n0GYarfjLu2hGG7MW8YKcb3C1KxViIBO5AwcXczK2L4wkTFxhNX118HZ+3I0O7vYKHRCyrYVHaKoclLOgGm9KSFiB/UeQdNrRfocYsl7y0uiYxRDgD6zrItLzgcWueduLM1JEgtS7Z6wHvglzE/v6NVHaRmkqMwS08GMiqKTR1qGwA7Co8SxVr6PDK4xneq202n2cB2OxTJKe05bCs9t/0OTTdsNccXuEdMa78Efh5Dl5dvrhFsXueX5m+fpRVjyPGDtAir8F4WapARivX12hiOe3AvM3ZQFWS7U/w6isNjPrAVpN+VruVRodxIpn+rePLqdTjDdLdQKD/2BXycJRLAct01Tvg38Nur72A7hqndXByQ7Fulb3D/SlpetxUAVwDPNT8XxvPahK6T5wjSMg8Wox/01sK+4a1DtHwuJ7O/N/gnrVeYWcdP5iDUDRkfVYmzOGpPiudlX4XZMDpR9Wed9TxF17CqfARdvsArCpI/07oV6rlMKVQx17bEiJp3dcK5ua5WxClfRHhD1G0HchWvrbzXDdbnt5rplxn6rrxdTF5RnZZpzskGCgc5Her6UywcADkeIUZesaCvij9p2NKAMJNqb0xBljx0cy4n/gBP9qqUtnO4fgD4qa/zyDpxeBxTOt1doZc7Ir0A8dduhR2fFh3kcxYaBYSfGGhpROEaqtnHF2b/AFWKQkX9Z20dGbWm8FA/TIrf3dZLhRUNON/jfNW7cZBLKL6KLyyS+1+XcTMFKKUGdOgpWm7k1xoi liUueMf+ V8yKRUCQrvgnIefXNWMjpsgmBNfccsYy2wyhbzJRA1Biux2nwzJixR82AQIAv5z3Fu09Sf0mieX+RsgCU8KsR203aXsoQ8Z8WHt/SGGIBGLjiVUt5p+0lutgaMf0mBgIzv9/ZrKibLeUV1ORqqG8V9mOUq0D4TyhvSedhSsjMxzl6C9mlg0wGaBYmzMkiLCQS/p0sDzSVCEwH/284UbqIiEaHyr5I8kF6rV4aENWtCiQhNACzQaBGpGDb1J1J1We+x2hw8tDWSMLRXU75hR9Tiv06KEGRUlulHEBwS92RbgyPOGj64P3hWq0fpGVXyrVAgPKQswz32AB8V6o2sk1jDlUgWdHFqMs4uM/+lvAePrBFk4qUNLp6acpAr7TykY8mBM5XdHE+Pov9Y+jF9+QvNf+zyVzsp/x509q2GzgCE8ZW9EshUg5CIr9O+w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Mike Rapoport (IBM)" init_cma_reserved_pages() only used in cma.c, no point of having it in page_alloc.c. Move init_cma_reserved_pages() to cma.c and make it static. Signed-off-by: Mike Rapoport (IBM) --- include/linux/gfp.h | 5 ----- mm/cma.c | 21 +++++++++++++++++++++ mm/page_alloc.c | 21 --------------------- 3 files changed, 21 insertions(+), 26 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 65a78773dcca..7c554e4bd49f 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -361,9 +361,4 @@ extern struct page *alloc_contig_pages(unsigned long nr_pages, gfp_t gfp_mask, #endif void free_contig_range(unsigned long pfn, unsigned long nr_pages); -#ifdef CONFIG_CMA -/* CMA stuff */ -extern void init_cma_reserved_pageblock(struct page *page); -#endif - #endif /* __LINUX_GFP_H */ diff --git a/mm/cma.c b/mm/cma.c index a7263aa02c92..ce08fb9825b4 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -31,8 +31,10 @@ #include #include #include +#include #include +#include "internal.h" #include "cma.h" struct cma cma_areas[MAX_CMA_AREAS]; @@ -93,6 +95,25 @@ static void cma_clear_bitmap(struct cma *cma, unsigned long pfn, spin_unlock_irqrestore(&cma->lock, flags); } +/* Free whole pageblock and set its migration type to MIGRATE_CMA. */ +static void init_cma_reserved_pageblock(struct page *page) +{ + unsigned i = pageblock_nr_pages; + struct page *p = page; + + do { + __ClearPageReserved(p); + set_page_count(p, 0); + } while (++p, --i); + + set_pageblock_migratetype(page, MIGRATE_CMA); + set_page_refcounted(page); + __free_pages(page, pageblock_order); + + adjust_managed_page_count(page, pageblock_nr_pages); + page_zone(page)->cma_pages += pageblock_nr_pages; +} + static void __init cma_activate_area(struct cma *cma) { unsigned long base_pfn = cma->base_pfn, pfn; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 87d760236dba..22e3da842e3f 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2280,27 +2280,6 @@ void __init page_alloc_init_late(void) set_zone_contiguous(zone); } -#ifdef CONFIG_CMA -/* Free whole pageblock and set its migration type to MIGRATE_CMA. */ -void __init init_cma_reserved_pageblock(struct page *page) -{ - unsigned i = pageblock_nr_pages; - struct page *p = page; - - do { - __ClearPageReserved(p); - set_page_count(p, 0); - } while (++p, --i); - - set_pageblock_migratetype(page, MIGRATE_CMA); - set_page_refcounted(page); - __free_pages(page, pageblock_order); - - adjust_managed_page_count(page, pageblock_nr_pages); - page_zone(page)->cma_pages += pageblock_nr_pages; -} -#endif - /* * The order of subdivision here is critical for the IO subsystem. * Please do not alter this order without good reasons and regression From patchwork Sun Mar 19 21:59:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13180632 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14963C6FD1F for ; Sun, 19 Mar 2023 22:00:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A1186900006; Sun, 19 Mar 2023 18:00:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8FC33900002; Sun, 19 Mar 2023 18:00:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 79EF5900006; Sun, 19 Mar 2023 18:00:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6B634900002 for ; Sun, 19 Mar 2023 18:00:36 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 4BAB0A0478 for ; Sun, 19 Mar 2023 22:00:36 +0000 (UTC) X-FDA: 80587017672.07.984BB26 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf02.hostedemail.com (Postfix) with ESMTP id 8F03B80019 for ; Sun, 19 Mar 2023 22:00:34 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=UBpAFCMO; spf=pass (imf02.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679263234; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gqHapehoVevycqqSvfloiB2KJWOM/j3t+5M1U1JWUoc=; b=FVdJVlA4IWGfdJcHMyFwXfzSoZJ9n3rxMdmgh7HcP8Fkxm6vqxGn6HJ9ukJO5TMyodP3ok xcfhv6JoKMMY5msRCWu/Xaws6SXgUjh4C0MJ7IJXSg2SZ9+lDtDdzGH6Me6jwSWwIEr7o7 EbW6bLzhkYlzVq/tEm4MxMVRySj1qDQ= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=UBpAFCMO; spf=pass (imf02.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679263234; a=rsa-sha256; cv=none; b=og63ojITcavYx8DYI9sfL1B4t3BWaf3k1iyiP5Jv+iCBxQRYv4usIPa6wz9PEsu4ggkmBy /g3ZNXQJCa1ozp9knVfySDBohVKFwCVQcY4FGyiwHthVjWP4k2M6uGPqzrThFFBVBekevv JT270pD3CJqHqJbEb2p8ohDVNYeRDSs= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7880A611A9; Sun, 19 Mar 2023 22:00:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D5B8AC4339B; Sun, 19 Mar 2023 22:00:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679263232; bh=hdgTCF3AluRT54E9MLIUV0kdVH4EuuQ6uLe4WMnOnfE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UBpAFCMO2EPuJQ3DceQO11KrNG1Xa7ss1NwTGDl2lsm+DZvOGWjpBOdQuIwm3VlAR YiaL0FwisP5hqGIHrOB0A0C58SWIc+Kr9de4Fb2Co9kcLSMuF4+YLkruSuuiF53opk QcVgTdrUsMaauwl8da/sCVuB3R+2qsDeSEAjjOJ9poCOOx+3RZJtudqLmOr7jNrtn/ d2MLLOWir2J8tvaSn6KkJAawHeUVjENMXHRwOT0QTmY8YBGVV5EznK0U84MAzrILLh 5qR/7Lv83L7RgyzqAdFgqJFQDybFQTrUQ3ysGBg/Mu/UDsJdOJEl4YvmCEn3CKJbgf tP8fiOTbLMnLA== From: Mike Rapoport To: Andrew Morton Cc: David Hildenbrand , Matthew Wilcox , Mel Gorman , Michal Hocko , Mike Rapoport , Thomas Bogendoerfer , Vlastimil Babka , linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 03/15] mm/page_alloc: add helper for checking if check_pages_enabled Date: Sun, 19 Mar 2023 23:59:56 +0200 Message-Id: <20230319220008.2138576-4-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230319220008.2138576-1-rppt@kernel.org> References: <20230319220008.2138576-1-rppt@kernel.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 8F03B80019 X-Rspam-User: X-Stat-Signature: iwp3nsd3t8bndgnho1x16nstst977zc4 X-HE-Tag: 1679263234-358971 X-HE-Meta: U2FsdGVkX18+JWuaEJ+LFucr0vwWJxF3+XpIwI/TR4O9YK+NNY3Np1kkOHfLOTB0FpCeFjKnOmPwS25Hmpnw3guoeGoLWoUgmSZlZ328M6g2RZxoJjk0oXJZG0RmKPx+sFaZX5tyvtevOvhrGc/vOl0q8KabAdxdmopi464uo8Gqz2W7j9bMfOscPY7MjHL+TmSFM0j1rtmN0HUH/Jm6vdlMZR3Wlc8Kwb0mhoceQUExnogRNCCTj7k57d9nhzJTEdQav2fIJoO49lmGQSyvNLSr1armd5w7/vr/Qh9otFMiZMXyBPaq+yclLAwCMXUBxVrzNkotfJCsL/cBLC+POunXo0AOrkqUpSdHXbvLIcN9kllpUl70rOybTzEpMHjf46T65oplrhUcflFqJeKa5osWJNN0WytCAyHk8BCQaRfWBIEIF3KM2ZBiag6xU6waGLuQuNzxNTqA43EZMZvaWTsUH3MNYczf0lb/bWo7gne9mbZ5luMaBpUqIwhpVAACW/l0Ya3ApMqhhDKstoNjaBWkorvqfYHkDvdlmEaS51vDO+ZHzcoVOPdZ4b/sLzMZxyZFs4+yni9IMKE4tHffdPXT7b1+EkywasYhfHnif+L4Dk6j4rgc+q/9mJTGtdTk0OOT+ahQHI3t73kzhtA2LS7ZPlyiftPVWzDbq36io/SREu8zO9QZ+ytS7cgDQnhIZs8VLoitg50BUny8dqRKE34eAt7Qk5Jel1DKU5Y9Hs4cEqp6wlER5s6oAwsaEeazdaC2+/DMfQqSyQwW+wFC7WD25NgHaVm92/DvN+CuT8qzuS9d/JBYv8ST8anq+KliSP7h1DIhL3N6agHa0+TMfs9MAB0OZ8Yfxtpt/Nm/qZXfF3sFOhtfhSjtO9/KRa43q0fyYVkwG+yHWdSJFw6gqcC05VFtBr8XVk24dBw3VG2rb761MgTlFvYefKx2uUqepjM8GEDTt57FwsuT/lh X0BjQ6FV l0aYg4bwnsZAkiph4ddyLuO7zjeenEJVLWApM7adAwGAQGVfcOT2ulzLvkYiOOEZf8TvqpADXegODBtCOtGSViE5KwiIAYRDtLPxIryESE1p3D0IOF3LthfU3J+D3EpPhg/mwVXxvASljKLq+J7rjtSjbFo23DnGNrhtQm15sOaiDEj9xp9a2rcGUaHPsWg6LCF4hvIRtIyJ0I50y36LOIYkU4oLRlhD60j17siEL8EksHpVAmHvMbYWWV9cBKhvz9oK5nk2nPe+oxg9BJ7b0QrNW9sRtdlgwOVeOB1vITk9yig7fyjyIH9addFaYs5UeLQ2GO2aiWQpgAy/+KtLQAWn/H3OBo0CPzcxXtnIxRQrwhWwNl8r9EmXlGp5986m0ooWsmsMA6Fy9PiLACFeolgSYzg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Mike Rapoport (IBM)" Instead of duplicating long static_branch_enabled(&check_pages_enabled) wrap it in a helper function is_check_pages_enabled() Signed-off-by: Mike Rapoport (IBM) Reviewed-by: David Hildenbrand --- mm/page_alloc.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 22e3da842e3f..e52f90d5d6a3 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -245,6 +245,11 @@ EXPORT_SYMBOL(init_on_free); /* perform sanity checks on struct pages being allocated or freed */ static DEFINE_STATIC_KEY_MAYBE(CONFIG_DEBUG_VM, check_pages_enabled); +static inline bool is_check_pages_enabled(void) +{ + return static_branch_unlikely(&check_pages_enabled); +} + static bool _init_on_alloc_enabled_early __read_mostly = IS_ENABLED(CONFIG_INIT_ON_ALLOC_DEFAULT_ON); static int __init early_init_on_alloc(char *buf) @@ -1443,7 +1448,7 @@ static __always_inline bool free_pages_prepare(struct page *page, for (i = 1; i < (1 << order); i++) { if (compound) bad += free_tail_pages_check(page, page + i); - if (static_branch_unlikely(&check_pages_enabled)) { + if (is_check_pages_enabled()) { if (unlikely(free_page_is_bad(page + i))) { bad++; continue; @@ -1456,7 +1461,7 @@ static __always_inline bool free_pages_prepare(struct page *page, page->mapping = NULL; if (memcg_kmem_online() && PageMemcgKmem(page)) __memcg_kmem_uncharge_page(page, order); - if (static_branch_unlikely(&check_pages_enabled)) { + if (is_check_pages_enabled()) { if (free_page_is_bad(page)) bad++; if (bad) @@ -2345,7 +2350,7 @@ static int check_new_page(struct page *page) static inline bool check_new_pages(struct page *page, unsigned int order) { - if (static_branch_unlikely(&check_pages_enabled)) { + if (is_check_pages_enabled()) { for (int i = 0; i < (1 << order); i++) { struct page *p = page + i; From patchwork Sun Mar 19 21:59:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13180635 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5173EC6FD1F for ; Sun, 19 Mar 2023 22:00:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BE442900007; Sun, 19 Mar 2023 18:00:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B94B6900002; Sun, 19 Mar 2023 18:00:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7EB63900007; Sun, 19 Mar 2023 18:00:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 62281900002 for ; Sun, 19 Mar 2023 18:00:43 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 238DD80500 for ; Sun, 19 Mar 2023 22:00:43 +0000 (UTC) X-FDA: 80587017966.22.D8B18CF Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by imf01.hostedemail.com (Postfix) with ESMTP id 8E9F840005 for ; Sun, 19 Mar 2023 22:00:40 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=aINTGoYk; spf=pass (imf01.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679263241; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gIHHsYH8hIDCOAyyImUe4Ki9PYAovDgjYaGFyfzYNw4=; b=Cu/EIjnEWo3YAkPMaE1YC3DsC1MKZw8TbRNWcs9TekRFI2sG8Yh2HCRHQtQbc3UvuEnvJd 5ZE8jwqE+D/OIVEDowsBOn2je+7j2ZPnkFh9aNw8e1jSjw56/sTyCxXQAjb+513JlFgbYj GGhIy+k2iQSsdzIfxKOLY+gVcFzKNfI= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=aINTGoYk; spf=pass (imf01.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679263241; a=rsa-sha256; cv=none; b=wxsBeXsfNLyhB12yHcmjVqOiZAwoKdRnm2rhS0/eakTQq+pWZsZq1BqH/bYRy0OXU+aKMA Qwinh+yJCsVHUg531CTSFaqO1XN1yVMGbZQxWCJl+Ae69KiDJ6x9naGqUJUq3UFUq+FzzF 7X+Ib+F0qp++8Nv+RMD9a0xiQ8Qkod8= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id B55E0B80D29; Sun, 19 Mar 2023 22:00:38 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6E49BC433D2; Sun, 19 Mar 2023 22:00:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679263237; bh=LX/WB+L2l8RaBlaCBNOGAUYGZsP9hZuEDZ3R11ldiLY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=aINTGoYkB9+hXnga9rN8g8CJeYb1RV0tIFgGv3lF+22cOTwZfP8SxLoxnaRrUOKdM IqXM3zrYtn4pN206GZ5T3GvGwL5CVTrxgLGtqNquNLo0PPSkZIOoWeHRlos0plkErj AyEPgD9FyPXvszzclCD//CY4qLC+ETDQPqfrJ0XFCaH1DZOxHBpjPYdhQanaYYMzW/ tyxWMmZWo6bWDj7V9ctAOLNhU9RoV8kCWbfH+wlbgZ5/ZYwK74gaAoy3/TbLkkF6Yl Nt6KKJEOvYvcN/qLhP5nBlN8yfAgizUfwZhkHA7nKBLJW0Yx9Af9pmeinpZ5Gw22y4 yLb+onGutKi9g== From: Mike Rapoport To: Andrew Morton Cc: David Hildenbrand , Matthew Wilcox , Mel Gorman , Michal Hocko , Mike Rapoport , Thomas Bogendoerfer , Vlastimil Babka , linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 04/15] mm: move most of core MM initialization to mm/mm_init.c Date: Sun, 19 Mar 2023 23:59:57 +0200 Message-Id: <20230319220008.2138576-5-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230319220008.2138576-1-rppt@kernel.org> References: <20230319220008.2138576-1-rppt@kernel.org> MIME-Version: 1.0 X-Stat-Signature: uyoske81374o1pyrgsh59od48qsaqq5w X-Rspam-User: X-Rspamd-Queue-Id: 8E9F840005 X-Rspamd-Server: rspam06 X-HE-Tag: 1679263240-831622 X-HE-Meta: U2FsdGVkX1//v1VWn4iRNhcS87MFrEO2GUV0szztbOSKLJeBQvk0lbcVpoVLqKL/lWzMfGBapWc5iUoQH0iDNLYk3TVO3fsz03BUjkvKsbRrp5oNnyl7IV+gMVugfM6HTElueTCvE1EP0Qc71OrdAQv1zZmTae+TOtJgiHDhg9+5dLwbyPuK9Ff0tC5PPvgXu6oKe3vUp2ReU65h3mC2xW0cAhzuEVD1KtJ9Tb9fycmAop8d9hmX2ep1m2nTrJxbD/muCszk/V7VEyfmN7y4aFm1nKvfUjiQr0pcQFsaKrdOYLcnGQ1VCFw1Q+mpZr6cTV9OSVu1iXuNW/jbwocuqTXguBYI2T4EN8UK6AzKHi17hhmgeleUtx5IbTiFUWWeVYyEkYENOyxoQ9G1p68bf2N2hg4DiFnm3vKWdpH2UQPFdzWunoZ2GqTBPJR+538RlcOd/6wpkFWGIb01YBoGEq5MPYJbn8sDnUXSAkmk4COR2bEmbjflRejtWTctSbykYQ0c0ACM0jYr4WWDVFij8sf7TbkvjJFoA/SAuvi0egJMyt4eOKz4mHdBN5Thi0A5ufYfwqyGlEkbgWcYs7sJ7fZoL3Lwt/GK0/zyi65f9T93dtD4SFGRtNX9+x2MT4db7Luhx83nyiY9arpL5klX/VlMDPV4GFWKDj6Eq1ku/5kZOK9qBMJdGGwZVxQo8jDjAmu3eNIF0iEYCGlUfpiZZYBHAsXyDZ8xHMGJzV0rEGYg6ghPZg6Fs96qEHkqNRt9NIJ0M3vpsouu8Z9Z0hdkL7KzB07sq0LuiL1hws1JzNt6a6VdqJf50JJtuNs14cdSapyA52xNS+mI9WNdEEonGmBY92eE1xdCRwG1ZIOXT1L+jPdT2OPGRFCBf2swcM3vUVw28lV3sp6Hk7NuhZU+Mw+EYWLhaDZeN7/1Vl3x7dOjN82DnyLTLY2W33FADXwloQKSyUlY8q5SyxJ9NDS 7/qeuqCl YNEZOyHh924mrgYtQm+1S9dleDTm20/abuPDHF3NVR3AfW/Cu15FYDI8WjlsSeDqm5Clu90rT/NgYL9sE1nYUNVJ4yJ2MTYhx0qHilPtOZdqHZIgjaKMCMqQrKyuyadFKPGHTiUTAlMdVg7ptTP89L+cWFaX2wx4u8rfZfvozBF34OQckEFeCx6MCTvDCS+toP/2BQc91VtvXAEfPE5mloYPxqnOjUwn9IXWrJAABH/5t2EONsDvot3atACxx7vUgdC2SRbktINs1L9HdazRcx02oisabqisqjwfb7cRTViAcO5yCInRy6T2CaXC9r7hwgASGIrFU0uiftfjgGArZDLluGShZPwLx3SQWWIEnXMaSOA1ys5oqMd+ZThCrSlvUar9ILgSFI8BO7gHBsR0d5tmDd0DssXyhf38r71VBmUp1Th6f3VCQ5PrHsqTk5JMs28Qt9ufsHmvIk0k= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Mike Rapoport (IBM)" The bulk of memory management initialization code is spread all over mm/page_alloc.c and makes navigating through page allocator functionality difficult. Move most of the functions marked __init and __meminit to mm/mm_init.c to make it better localized and allow some more spare room before mm/page_alloc.c reaches 10k lines. No functional changes. Signed-off-by: Mike Rapoport (IBM) Acked-by: David Hildenbrand --- mm/internal.h | 31 + mm/mm_init.c | 2284 ++++++++++++++++++++++++++++++++++++++ mm/page_alloc.c | 2837 +++++------------------------------------------ 3 files changed, 2583 insertions(+), 2569 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index fce94775819c..6b154b4a538f 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -202,6 +202,8 @@ pmd_t *mm_find_pmd(struct mm_struct *mm, unsigned long address); * in mm/page_alloc.c */ +extern char * const zone_names[MAX_NR_ZONES]; + /* * Structure for holding the mostly immutable allocation parameters passed * between functions involved in allocations, including the alloc_pages* @@ -366,7 +368,29 @@ extern void __putback_isolated_page(struct page *page, unsigned int order, extern void memblock_free_pages(struct page *page, unsigned long pfn, unsigned int order); extern void __free_pages_core(struct page *page, unsigned int order); + +static inline void prep_compound_head(struct page *page, unsigned int order) +{ + struct folio *folio = (struct folio *)page; + + set_compound_page_dtor(page, COMPOUND_PAGE_DTOR); + set_compound_order(page, order); + atomic_set(&folio->_entire_mapcount, -1); + atomic_set(&folio->_nr_pages_mapped, 0); + atomic_set(&folio->_pincount, 0); +} + +static inline void prep_compound_tail(struct page *head, int tail_idx) +{ + struct page *p = head + tail_idx; + + p->mapping = TAIL_MAPPING; + set_compound_head(p, head); + set_page_private(p, 0); +} + extern void prep_compound_page(struct page *page, unsigned int order); + extern void post_alloc_hook(struct page *page, unsigned int order, gfp_t gfp_flags); extern int user_min_free_kbytes; @@ -377,6 +401,7 @@ extern void free_unref_page_list(struct list_head *list); extern void zone_pcp_reset(struct zone *zone); extern void zone_pcp_disable(struct zone *zone); extern void zone_pcp_enable(struct zone *zone); +extern void zone_pcp_init(struct zone *zone); extern void *memmap_alloc(phys_addr_t size, phys_addr_t align, phys_addr_t min_addr, @@ -679,6 +704,12 @@ static inline loff_t fadvise_calc_endbyte(loff_t offset, loff_t len) } /* Memory initialisation debug and verification */ +#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT +DECLARE_STATIC_KEY_TRUE(deferred_pages); + +bool __init deferred_grow_zone(struct zone *zone, unsigned int order); +#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ + enum mminit_level { MMINIT_WARNING, MMINIT_VERIFY, diff --git a/mm/mm_init.c b/mm/mm_init.c index c1883362e71d..63aa7b6b2880 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -14,7 +14,14 @@ #include #include #include +#include +#include +#include +#include +#include +#include #include "internal.h" +#include "shuffle.h" #ifdef CONFIG_DEBUG_MEMORY_INIT int __meminitdata mminit_loglevel; @@ -198,3 +205,2280 @@ static int __init mm_sysfs_init(void) return 0; } postcore_initcall(mm_sysfs_init); + +static unsigned long arch_zone_lowest_possible_pfn[MAX_NR_ZONES] __initdata; +static unsigned long arch_zone_highest_possible_pfn[MAX_NR_ZONES] __initdata; +static unsigned long zone_movable_pfn[MAX_NUMNODES] __initdata; + +static unsigned long required_kernelcore __initdata; +static unsigned long required_kernelcore_percent __initdata; +static unsigned long required_movablecore __initdata; +static unsigned long required_movablecore_percent __initdata; + +static unsigned long nr_kernel_pages __initdata; +static unsigned long nr_all_pages __initdata; +static unsigned long dma_reserve __initdata; + +bool deferred_struct_pages __meminitdata; + +static DEFINE_PER_CPU(struct per_cpu_nodestat, boot_nodestats); + +static int __init cmdline_parse_core(char *p, unsigned long *core, + unsigned long *percent) +{ + unsigned long long coremem; + char *endptr; + + if (!p) + return -EINVAL; + + /* Value may be a percentage of total memory, otherwise bytes */ + coremem = simple_strtoull(p, &endptr, 0); + if (*endptr == '%') { + /* Paranoid check for percent values greater than 100 */ + WARN_ON(coremem > 100); + + *percent = coremem; + } else { + coremem = memparse(p, &p); + /* Paranoid check that UL is enough for the coremem value */ + WARN_ON((coremem >> PAGE_SHIFT) > ULONG_MAX); + + *core = coremem >> PAGE_SHIFT; + *percent = 0UL; + } + return 0; +} + +/* + * kernelcore=size sets the amount of memory for use for allocations that + * cannot be reclaimed or migrated. + */ +static int __init cmdline_parse_kernelcore(char *p) +{ + /* parse kernelcore=mirror */ + if (parse_option_str(p, "mirror")) { + mirrored_kernelcore = true; + return 0; + } + + return cmdline_parse_core(p, &required_kernelcore, + &required_kernelcore_percent); +} +early_param("kernelcore", cmdline_parse_kernelcore); + +/* + * movablecore=size sets the amount of memory for use for allocations that + * can be reclaimed or migrated. + */ +static int __init cmdline_parse_movablecore(char *p) +{ + return cmdline_parse_core(p, &required_movablecore, + &required_movablecore_percent); +} +early_param("movablecore", cmdline_parse_movablecore); + +/* + * early_calculate_totalpages() + * Sum pages in active regions for movable zone. + * Populate N_MEMORY for calculating usable_nodes. + */ +static unsigned long __init early_calculate_totalpages(void) +{ + unsigned long totalpages = 0; + unsigned long start_pfn, end_pfn; + int i, nid; + + for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, &nid) { + unsigned long pages = end_pfn - start_pfn; + + totalpages += pages; + if (pages) + node_set_state(nid, N_MEMORY); + } + return totalpages; +} + +/* + * This finds a zone that can be used for ZONE_MOVABLE pages. The + * assumption is made that zones within a node are ordered in monotonic + * increasing memory addresses so that the "highest" populated zone is used + */ +static void __init find_usable_zone_for_movable(void) +{ + int zone_index; + for (zone_index = MAX_NR_ZONES - 1; zone_index >= 0; zone_index--) { + if (zone_index == ZONE_MOVABLE) + continue; + + if (arch_zone_highest_possible_pfn[zone_index] > + arch_zone_lowest_possible_pfn[zone_index]) + break; + } + + VM_BUG_ON(zone_index == -1); + movable_zone = zone_index; +} + +/* + * Find the PFN the Movable zone begins in each node. Kernel memory + * is spread evenly between nodes as long as the nodes have enough + * memory. When they don't, some nodes will have more kernelcore than + * others + */ +static void __init find_zone_movable_pfns_for_nodes(void) +{ + int i, nid; + unsigned long usable_startpfn; + unsigned long kernelcore_node, kernelcore_remaining; + /* save the state before borrow the nodemask */ + nodemask_t saved_node_state = node_states[N_MEMORY]; + unsigned long totalpages = early_calculate_totalpages(); + int usable_nodes = nodes_weight(node_states[N_MEMORY]); + struct memblock_region *r; + + /* Need to find movable_zone earlier when movable_node is specified. */ + find_usable_zone_for_movable(); + + /* + * If movable_node is specified, ignore kernelcore and movablecore + * options. + */ + if (movable_node_is_enabled()) { + for_each_mem_region(r) { + if (!memblock_is_hotpluggable(r)) + continue; + + nid = memblock_get_region_node(r); + + usable_startpfn = PFN_DOWN(r->base); + zone_movable_pfn[nid] = zone_movable_pfn[nid] ? + min(usable_startpfn, zone_movable_pfn[nid]) : + usable_startpfn; + } + + goto out2; + } + + /* + * If kernelcore=mirror is specified, ignore movablecore option + */ + if (mirrored_kernelcore) { + bool mem_below_4gb_not_mirrored = false; + + for_each_mem_region(r) { + if (memblock_is_mirror(r)) + continue; + + nid = memblock_get_region_node(r); + + usable_startpfn = memblock_region_memory_base_pfn(r); + + if (usable_startpfn < PHYS_PFN(SZ_4G)) { + mem_below_4gb_not_mirrored = true; + continue; + } + + zone_movable_pfn[nid] = zone_movable_pfn[nid] ? + min(usable_startpfn, zone_movable_pfn[nid]) : + usable_startpfn; + } + + if (mem_below_4gb_not_mirrored) + pr_warn("This configuration results in unmirrored kernel memory.\n"); + + goto out2; + } + + /* + * If kernelcore=nn% or movablecore=nn% was specified, calculate the + * amount of necessary memory. + */ + if (required_kernelcore_percent) + required_kernelcore = (totalpages * 100 * required_kernelcore_percent) / + 10000UL; + if (required_movablecore_percent) + required_movablecore = (totalpages * 100 * required_movablecore_percent) / + 10000UL; + + /* + * If movablecore= was specified, calculate what size of + * kernelcore that corresponds so that memory usable for + * any allocation type is evenly spread. If both kernelcore + * and movablecore are specified, then the value of kernelcore + * will be used for required_kernelcore if it's greater than + * what movablecore would have allowed. + */ + if (required_movablecore) { + unsigned long corepages; + + /* + * Round-up so that ZONE_MOVABLE is at least as large as what + * was requested by the user + */ + required_movablecore = + roundup(required_movablecore, MAX_ORDER_NR_PAGES); + required_movablecore = min(totalpages, required_movablecore); + corepages = totalpages - required_movablecore; + + required_kernelcore = max(required_kernelcore, corepages); + } + + /* + * If kernelcore was not specified or kernelcore size is larger + * than totalpages, there is no ZONE_MOVABLE. + */ + if (!required_kernelcore || required_kernelcore >= totalpages) + goto out; + + /* usable_startpfn is the lowest possible pfn ZONE_MOVABLE can be at */ + usable_startpfn = arch_zone_lowest_possible_pfn[movable_zone]; + +restart: + /* Spread kernelcore memory as evenly as possible throughout nodes */ + kernelcore_node = required_kernelcore / usable_nodes; + for_each_node_state(nid, N_MEMORY) { + unsigned long start_pfn, end_pfn; + + /* + * Recalculate kernelcore_node if the division per node + * now exceeds what is necessary to satisfy the requested + * amount of memory for the kernel + */ + if (required_kernelcore < kernelcore_node) + kernelcore_node = required_kernelcore / usable_nodes; + + /* + * As the map is walked, we track how much memory is usable + * by the kernel using kernelcore_remaining. When it is + * 0, the rest of the node is usable by ZONE_MOVABLE + */ + kernelcore_remaining = kernelcore_node; + + /* Go through each range of PFNs within this node */ + for_each_mem_pfn_range(i, nid, &start_pfn, &end_pfn, NULL) { + unsigned long size_pages; + + start_pfn = max(start_pfn, zone_movable_pfn[nid]); + if (start_pfn >= end_pfn) + continue; + + /* Account for what is only usable for kernelcore */ + if (start_pfn < usable_startpfn) { + unsigned long kernel_pages; + kernel_pages = min(end_pfn, usable_startpfn) + - start_pfn; + + kernelcore_remaining -= min(kernel_pages, + kernelcore_remaining); + required_kernelcore -= min(kernel_pages, + required_kernelcore); + + /* Continue if range is now fully accounted */ + if (end_pfn <= usable_startpfn) { + + /* + * Push zone_movable_pfn to the end so + * that if we have to rebalance + * kernelcore across nodes, we will + * not double account here + */ + zone_movable_pfn[nid] = end_pfn; + continue; + } + start_pfn = usable_startpfn; + } + + /* + * The usable PFN range for ZONE_MOVABLE is from + * start_pfn->end_pfn. Calculate size_pages as the + * number of pages used as kernelcore + */ + size_pages = end_pfn - start_pfn; + if (size_pages > kernelcore_remaining) + size_pages = kernelcore_remaining; + zone_movable_pfn[nid] = start_pfn + size_pages; + + /* + * Some kernelcore has been met, update counts and + * break if the kernelcore for this node has been + * satisfied + */ + required_kernelcore -= min(required_kernelcore, + size_pages); + kernelcore_remaining -= size_pages; + if (!kernelcore_remaining) + break; + } + } + + /* + * If there is still required_kernelcore, we do another pass with one + * less node in the count. This will push zone_movable_pfn[nid] further + * along on the nodes that still have memory until kernelcore is + * satisfied + */ + usable_nodes--; + if (usable_nodes && required_kernelcore > usable_nodes) + goto restart; + +out2: + /* Align start of ZONE_MOVABLE on all nids to MAX_ORDER_NR_PAGES */ + for (nid = 0; nid < MAX_NUMNODES; nid++) { + unsigned long start_pfn, end_pfn; + + zone_movable_pfn[nid] = + roundup(zone_movable_pfn[nid], MAX_ORDER_NR_PAGES); + + get_pfn_range_for_nid(nid, &start_pfn, &end_pfn); + if (zone_movable_pfn[nid] >= end_pfn) + zone_movable_pfn[nid] = 0; + } + +out: + /* restore the node_state */ + node_states[N_MEMORY] = saved_node_state; +} + +static void __meminit __init_single_page(struct page *page, unsigned long pfn, + unsigned long zone, int nid) +{ + mm_zero_struct_page(page); + set_page_links(page, zone, nid, pfn); + init_page_count(page); + page_mapcount_reset(page); + page_cpupid_reset_last(page); + page_kasan_tag_reset(page); + + INIT_LIST_HEAD(&page->lru); +#ifdef WANT_PAGE_VIRTUAL + /* The shift won't overflow because ZONE_NORMAL is below 4G. */ + if (!is_highmem_idx(zone)) + set_page_address(page, __va(pfn << PAGE_SHIFT)); +#endif +} + +#ifdef CONFIG_NUMA +/* + * During memory init memblocks map pfns to nids. The search is expensive and + * this caches recent lookups. The implementation of __early_pfn_to_nid + * treats start/end as pfns. + */ +struct mminit_pfnnid_cache { + unsigned long last_start; + unsigned long last_end; + int last_nid; +}; + +static struct mminit_pfnnid_cache early_pfnnid_cache __meminitdata; + +/* + * Required by SPARSEMEM. Given a PFN, return what node the PFN is on. + */ +static int __meminit __early_pfn_to_nid(unsigned long pfn, + struct mminit_pfnnid_cache *state) +{ + unsigned long start_pfn, end_pfn; + int nid; + + if (state->last_start <= pfn && pfn < state->last_end) + return state->last_nid; + + nid = memblock_search_pfn_nid(pfn, &start_pfn, &end_pfn); + if (nid != NUMA_NO_NODE) { + state->last_start = start_pfn; + state->last_end = end_pfn; + state->last_nid = nid; + } + + return nid; +} + +int __meminit early_pfn_to_nid(unsigned long pfn) +{ + static DEFINE_SPINLOCK(early_pfn_lock); + int nid; + + spin_lock(&early_pfn_lock); + nid = __early_pfn_to_nid(pfn, &early_pfnnid_cache); + if (nid < 0) + nid = first_online_node; + spin_unlock(&early_pfn_lock); + + return nid; +} +#endif /* CONFIG_NUMA */ + +#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT +static inline void pgdat_set_deferred_range(pg_data_t *pgdat) +{ + pgdat->first_deferred_pfn = ULONG_MAX; +} + +/* Returns true if the struct page for the pfn is initialised */ +static inline bool __meminit early_page_initialised(unsigned long pfn) +{ + int nid = early_pfn_to_nid(pfn); + + if (node_online(nid) && pfn >= NODE_DATA(nid)->first_deferred_pfn) + return false; + + return true; +} + +/* + * Returns true when the remaining initialisation should be deferred until + * later in the boot cycle when it can be parallelised. + */ +static bool __meminit +defer_init(int nid, unsigned long pfn, unsigned long end_pfn) +{ + static unsigned long prev_end_pfn, nr_initialised; + + if (early_page_ext_enabled()) + return false; + /* + * prev_end_pfn static that contains the end of previous zone + * No need to protect because called very early in boot before smp_init. + */ + if (prev_end_pfn != end_pfn) { + prev_end_pfn = end_pfn; + nr_initialised = 0; + } + + /* Always populate low zones for address-constrained allocations */ + if (end_pfn < pgdat_end_pfn(NODE_DATA(nid))) + return false; + + if (NODE_DATA(nid)->first_deferred_pfn != ULONG_MAX) + return true; + /* + * We start only with one section of pages, more pages are added as + * needed until the rest of deferred pages are initialized. + */ + nr_initialised++; + if ((nr_initialised > PAGES_PER_SECTION) && + (pfn & (PAGES_PER_SECTION - 1)) == 0) { + NODE_DATA(nid)->first_deferred_pfn = pfn; + return true; + } + return false; +} + +static void __meminit init_reserved_page(unsigned long pfn) +{ + pg_data_t *pgdat; + int nid, zid; + + if (early_page_initialised(pfn)) + return; + + nid = early_pfn_to_nid(pfn); + pgdat = NODE_DATA(nid); + + for (zid = 0; zid < MAX_NR_ZONES; zid++) { + struct zone *zone = &pgdat->node_zones[zid]; + + if (zone_spans_pfn(zone, pfn)) + break; + } + __init_single_page(pfn_to_page(pfn), pfn, zid, nid); +} +#else +static inline void pgdat_set_deferred_range(pg_data_t *pgdat) {} + +static inline bool early_page_initialised(unsigned long pfn) +{ + return true; +} + +static inline bool defer_init(int nid, unsigned long pfn, unsigned long end_pfn) +{ + return false; +} + +static inline void init_reserved_page(unsigned long pfn) +{ +} +#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ + +/* + * Initialised pages do not have PageReserved set. This function is + * called for each range allocated by the bootmem allocator and + * marks the pages PageReserved. The remaining valid pages are later + * sent to the buddy page allocator. + */ +void __meminit reserve_bootmem_region(phys_addr_t start, phys_addr_t end) +{ + unsigned long start_pfn = PFN_DOWN(start); + unsigned long end_pfn = PFN_UP(end); + + for (; start_pfn < end_pfn; start_pfn++) { + if (pfn_valid(start_pfn)) { + struct page *page = pfn_to_page(start_pfn); + + init_reserved_page(start_pfn); + + /* Avoid false-positive PageTail() */ + INIT_LIST_HEAD(&page->lru); + + /* + * no need for atomic set_bit because the struct + * page is not visible yet so nobody should + * access it yet. + */ + __SetPageReserved(page); + } + } +} + +/* If zone is ZONE_MOVABLE but memory is mirrored, it is an overlapped init */ +static bool __meminit +overlap_memmap_init(unsigned long zone, unsigned long *pfn) +{ + static struct memblock_region *r; + + if (mirrored_kernelcore && zone == ZONE_MOVABLE) { + if (!r || *pfn >= memblock_region_memory_end_pfn(r)) { + for_each_mem_region(r) { + if (*pfn < memblock_region_memory_end_pfn(r)) + break; + } + } + if (*pfn >= memblock_region_memory_base_pfn(r) && + memblock_is_mirror(r)) { + *pfn = memblock_region_memory_end_pfn(r); + return true; + } + } + return false; +} + +/* + * Only struct pages that correspond to ranges defined by memblock.memory + * are zeroed and initialized by going through __init_single_page() during + * memmap_init_zone_range(). + * + * But, there could be struct pages that correspond to holes in + * memblock.memory. This can happen because of the following reasons: + * - physical memory bank size is not necessarily the exact multiple of the + * arbitrary section size + * - early reserved memory may not be listed in memblock.memory + * - memory layouts defined with memmap= kernel parameter may not align + * nicely with memmap sections + * + * Explicitly initialize those struct pages so that: + * - PG_Reserved is set + * - zone and node links point to zone and node that span the page if the + * hole is in the middle of a zone + * - zone and node links point to adjacent zone/node if the hole falls on + * the zone boundary; the pages in such holes will be prepended to the + * zone/node above the hole except for the trailing pages in the last + * section that will be appended to the zone/node below. + */ +static void __init init_unavailable_range(unsigned long spfn, + unsigned long epfn, + int zone, int node) +{ + unsigned long pfn; + u64 pgcnt = 0; + + for (pfn = spfn; pfn < epfn; pfn++) { + if (!pfn_valid(pageblock_start_pfn(pfn))) { + pfn = pageblock_end_pfn(pfn) - 1; + continue; + } + __init_single_page(pfn_to_page(pfn), pfn, zone, node); + __SetPageReserved(pfn_to_page(pfn)); + pgcnt++; + } + + if (pgcnt) + pr_info("On node %d, zone %s: %lld pages in unavailable ranges", + node, zone_names[zone], pgcnt); +} + +/* + * Initially all pages are reserved - free ones are freed + * up by memblock_free_all() once the early boot process is + * done. Non-atomic initialization, single-pass. + * + * All aligned pageblocks are initialized to the specified migratetype + * (usually MIGRATE_MOVABLE). Besides setting the migratetype, no related + * zone stats (e.g., nr_isolate_pageblock) are touched. + */ +void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone, + unsigned long start_pfn, unsigned long zone_end_pfn, + enum meminit_context context, + struct vmem_altmap *altmap, int migratetype) +{ + unsigned long pfn, end_pfn = start_pfn + size; + struct page *page; + + if (highest_memmap_pfn < end_pfn - 1) + highest_memmap_pfn = end_pfn - 1; + +#ifdef CONFIG_ZONE_DEVICE + /* + * Honor reservation requested by the driver for this ZONE_DEVICE + * memory. We limit the total number of pages to initialize to just + * those that might contain the memory mapping. We will defer the + * ZONE_DEVICE page initialization until after we have released + * the hotplug lock. + */ + if (zone == ZONE_DEVICE) { + if (!altmap) + return; + + if (start_pfn == altmap->base_pfn) + start_pfn += altmap->reserve; + end_pfn = altmap->base_pfn + vmem_altmap_offset(altmap); + } +#endif + + for (pfn = start_pfn; pfn < end_pfn; ) { + /* + * There can be holes in boot-time mem_map[]s handed to this + * function. They do not exist on hotplugged memory. + */ + if (context == MEMINIT_EARLY) { + if (overlap_memmap_init(zone, &pfn)) + continue; + if (defer_init(nid, pfn, zone_end_pfn)) { + deferred_struct_pages = true; + break; + } + } + + page = pfn_to_page(pfn); + __init_single_page(page, pfn, zone, nid); + if (context == MEMINIT_HOTPLUG) + __SetPageReserved(page); + + /* + * Usually, we want to mark the pageblock MIGRATE_MOVABLE, + * such that unmovable allocations won't be scattered all + * over the place during system boot. + */ + if (pageblock_aligned(pfn)) { + set_pageblock_migratetype(page, migratetype); + cond_resched(); + } + pfn++; + } +} + +static void __init memmap_init_zone_range(struct zone *zone, + unsigned long start_pfn, + unsigned long end_pfn, + unsigned long *hole_pfn) +{ + unsigned long zone_start_pfn = zone->zone_start_pfn; + unsigned long zone_end_pfn = zone_start_pfn + zone->spanned_pages; + int nid = zone_to_nid(zone), zone_id = zone_idx(zone); + + start_pfn = clamp(start_pfn, zone_start_pfn, zone_end_pfn); + end_pfn = clamp(end_pfn, zone_start_pfn, zone_end_pfn); + + if (start_pfn >= end_pfn) + return; + + memmap_init_range(end_pfn - start_pfn, nid, zone_id, start_pfn, + zone_end_pfn, MEMINIT_EARLY, NULL, MIGRATE_MOVABLE); + + if (*hole_pfn < start_pfn) + init_unavailable_range(*hole_pfn, start_pfn, zone_id, nid); + + *hole_pfn = end_pfn; +} + +static void __init memmap_init(void) +{ + unsigned long start_pfn, end_pfn; + unsigned long hole_pfn = 0; + int i, j, zone_id = 0, nid; + + for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, &nid) { + struct pglist_data *node = NODE_DATA(nid); + + for (j = 0; j < MAX_NR_ZONES; j++) { + struct zone *zone = node->node_zones + j; + + if (!populated_zone(zone)) + continue; + + memmap_init_zone_range(zone, start_pfn, end_pfn, + &hole_pfn); + zone_id = j; + } + } + +#ifdef CONFIG_SPARSEMEM + /* + * Initialize the memory map for hole in the range [memory_end, + * section_end]. + * Append the pages in this hole to the highest zone in the last + * node. + * The call to init_unavailable_range() is outside the ifdef to + * silence the compiler warining about zone_id set but not used; + * for FLATMEM it is a nop anyway + */ + end_pfn = round_up(end_pfn, PAGES_PER_SECTION); + if (hole_pfn < end_pfn) +#endif + init_unavailable_range(hole_pfn, end_pfn, zone_id, nid); +} + +#ifdef CONFIG_ZONE_DEVICE +static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, + unsigned long zone_idx, int nid, + struct dev_pagemap *pgmap) +{ + + __init_single_page(page, pfn, zone_idx, nid); + + /* + * Mark page reserved as it will need to wait for onlining + * phase for it to be fully associated with a zone. + * + * We can use the non-atomic __set_bit operation for setting + * the flag as we are still initializing the pages. + */ + __SetPageReserved(page); + + /* + * ZONE_DEVICE pages union ->lru with a ->pgmap back pointer + * and zone_device_data. It is a bug if a ZONE_DEVICE page is + * ever freed or placed on a driver-private list. + */ + page->pgmap = pgmap; + page->zone_device_data = NULL; + + /* + * Mark the block movable so that blocks are reserved for + * movable at startup. This will force kernel allocations + * to reserve their blocks rather than leaking throughout + * the address space during boot when many long-lived + * kernel allocations are made. + * + * Please note that MEMINIT_HOTPLUG path doesn't clear memmap + * because this is done early in section_activate() + */ + if (pageblock_aligned(pfn)) { + set_pageblock_migratetype(page, MIGRATE_MOVABLE); + cond_resched(); + } + + /* + * ZONE_DEVICE pages are released directly to the driver page allocator + * which will set the page count to 1 when allocating the page. + */ + if (pgmap->type == MEMORY_DEVICE_PRIVATE || + pgmap->type == MEMORY_DEVICE_COHERENT) + set_page_count(page, 0); +} + +/* + * With compound page geometry and when struct pages are stored in ram most + * tail pages are reused. Consequently, the amount of unique struct pages to + * initialize is a lot smaller that the total amount of struct pages being + * mapped. This is a paired / mild layering violation with explicit knowledge + * of how the sparse_vmemmap internals handle compound pages in the lack + * of an altmap. See vmemmap_populate_compound_pages(). + */ +static inline unsigned long compound_nr_pages(struct vmem_altmap *altmap, + unsigned long nr_pages) +{ + return is_power_of_2(sizeof(struct page)) && + !altmap ? 2 * (PAGE_SIZE / sizeof(struct page)) : nr_pages; +} + +static void __ref memmap_init_compound(struct page *head, + unsigned long head_pfn, + unsigned long zone_idx, int nid, + struct dev_pagemap *pgmap, + unsigned long nr_pages) +{ + unsigned long pfn, end_pfn = head_pfn + nr_pages; + unsigned int order = pgmap->vmemmap_shift; + + __SetPageHead(head); + for (pfn = head_pfn + 1; pfn < end_pfn; pfn++) { + struct page *page = pfn_to_page(pfn); + + __init_zone_device_page(page, pfn, zone_idx, nid, pgmap); + prep_compound_tail(head, pfn - head_pfn); + set_page_count(page, 0); + + /* + * The first tail page stores important compound page info. + * Call prep_compound_head() after the first tail page has + * been initialized, to not have the data overwritten. + */ + if (pfn == head_pfn + 1) + prep_compound_head(head, order); + } +} + +void __ref memmap_init_zone_device(struct zone *zone, + unsigned long start_pfn, + unsigned long nr_pages, + struct dev_pagemap *pgmap) +{ + unsigned long pfn, end_pfn = start_pfn + nr_pages; + struct pglist_data *pgdat = zone->zone_pgdat; + struct vmem_altmap *altmap = pgmap_altmap(pgmap); + unsigned int pfns_per_compound = pgmap_vmemmap_nr(pgmap); + unsigned long zone_idx = zone_idx(zone); + unsigned long start = jiffies; + int nid = pgdat->node_id; + + if (WARN_ON_ONCE(!pgmap || zone_idx != ZONE_DEVICE)) + return; + + /* + * The call to memmap_init should have already taken care + * of the pages reserved for the memmap, so we can just jump to + * the end of that region and start processing the device pages. + */ + if (altmap) { + start_pfn = altmap->base_pfn + vmem_altmap_offset(altmap); + nr_pages = end_pfn - start_pfn; + } + + for (pfn = start_pfn; pfn < end_pfn; pfn += pfns_per_compound) { + struct page *page = pfn_to_page(pfn); + + __init_zone_device_page(page, pfn, zone_idx, nid, pgmap); + + if (pfns_per_compound == 1) + continue; + + memmap_init_compound(page, pfn, zone_idx, nid, pgmap, + compound_nr_pages(altmap, pfns_per_compound)); + } + + pr_info("%s initialised %lu pages in %ums\n", __func__, + nr_pages, jiffies_to_msecs(jiffies - start)); +} +#endif + +/* + * The zone ranges provided by the architecture do not include ZONE_MOVABLE + * because it is sized independent of architecture. Unlike the other zones, + * the starting point for ZONE_MOVABLE is not fixed. It may be different + * in each node depending on the size of each node and how evenly kernelcore + * is distributed. This helper function adjusts the zone ranges + * provided by the architecture for a given node by using the end of the + * highest usable zone for ZONE_MOVABLE. This preserves the assumption that + * zones within a node are in order of monotonic increases memory addresses + */ +static void __init adjust_zone_range_for_zone_movable(int nid, + unsigned long zone_type, + unsigned long node_start_pfn, + unsigned long node_end_pfn, + unsigned long *zone_start_pfn, + unsigned long *zone_end_pfn) +{ + /* Only adjust if ZONE_MOVABLE is on this node */ + if (zone_movable_pfn[nid]) { + /* Size ZONE_MOVABLE */ + if (zone_type == ZONE_MOVABLE) { + *zone_start_pfn = zone_movable_pfn[nid]; + *zone_end_pfn = min(node_end_pfn, + arch_zone_highest_possible_pfn[movable_zone]); + + /* Adjust for ZONE_MOVABLE starting within this range */ + } else if (!mirrored_kernelcore && + *zone_start_pfn < zone_movable_pfn[nid] && + *zone_end_pfn > zone_movable_pfn[nid]) { + *zone_end_pfn = zone_movable_pfn[nid]; + + /* Check if this whole range is within ZONE_MOVABLE */ + } else if (*zone_start_pfn >= zone_movable_pfn[nid]) + *zone_start_pfn = *zone_end_pfn; + } +} + +/* + * Return the number of holes in a range on a node. If nid is MAX_NUMNODES, + * then all holes in the requested range will be accounted for. + */ +unsigned long __init __absent_pages_in_range(int nid, + unsigned long range_start_pfn, + unsigned long range_end_pfn) +{ + unsigned long nr_absent = range_end_pfn - range_start_pfn; + unsigned long start_pfn, end_pfn; + int i; + + for_each_mem_pfn_range(i, nid, &start_pfn, &end_pfn, NULL) { + start_pfn = clamp(start_pfn, range_start_pfn, range_end_pfn); + end_pfn = clamp(end_pfn, range_start_pfn, range_end_pfn); + nr_absent -= end_pfn - start_pfn; + } + return nr_absent; +} + +/** + * absent_pages_in_range - Return number of page frames in holes within a range + * @start_pfn: The start PFN to start searching for holes + * @end_pfn: The end PFN to stop searching for holes + * + * Return: the number of pages frames in memory holes within a range. + */ +unsigned long __init absent_pages_in_range(unsigned long start_pfn, + unsigned long end_pfn) +{ + return __absent_pages_in_range(MAX_NUMNODES, start_pfn, end_pfn); +} + +/* Return the number of page frames in holes in a zone on a node */ +static unsigned long __init zone_absent_pages_in_node(int nid, + unsigned long zone_type, + unsigned long node_start_pfn, + unsigned long node_end_pfn) +{ + unsigned long zone_low = arch_zone_lowest_possible_pfn[zone_type]; + unsigned long zone_high = arch_zone_highest_possible_pfn[zone_type]; + unsigned long zone_start_pfn, zone_end_pfn; + unsigned long nr_absent; + + /* When hotadd a new node from cpu_up(), the node should be empty */ + if (!node_start_pfn && !node_end_pfn) + return 0; + + zone_start_pfn = clamp(node_start_pfn, zone_low, zone_high); + zone_end_pfn = clamp(node_end_pfn, zone_low, zone_high); + + adjust_zone_range_for_zone_movable(nid, zone_type, + node_start_pfn, node_end_pfn, + &zone_start_pfn, &zone_end_pfn); + nr_absent = __absent_pages_in_range(nid, zone_start_pfn, zone_end_pfn); + + /* + * ZONE_MOVABLE handling. + * Treat pages to be ZONE_MOVABLE in ZONE_NORMAL as absent pages + * and vice versa. + */ + if (mirrored_kernelcore && zone_movable_pfn[nid]) { + unsigned long start_pfn, end_pfn; + struct memblock_region *r; + + for_each_mem_region(r) { + start_pfn = clamp(memblock_region_memory_base_pfn(r), + zone_start_pfn, zone_end_pfn); + end_pfn = clamp(memblock_region_memory_end_pfn(r), + zone_start_pfn, zone_end_pfn); + + if (zone_type == ZONE_MOVABLE && + memblock_is_mirror(r)) + nr_absent += end_pfn - start_pfn; + + if (zone_type == ZONE_NORMAL && + !memblock_is_mirror(r)) + nr_absent += end_pfn - start_pfn; + } + } + + return nr_absent; +} + +/* + * Return the number of pages a zone spans in a node, including holes + * present_pages = zone_spanned_pages_in_node() - zone_absent_pages_in_node() + */ +static unsigned long __init zone_spanned_pages_in_node(int nid, + unsigned long zone_type, + unsigned long node_start_pfn, + unsigned long node_end_pfn, + unsigned long *zone_start_pfn, + unsigned long *zone_end_pfn) +{ + unsigned long zone_low = arch_zone_lowest_possible_pfn[zone_type]; + unsigned long zone_high = arch_zone_highest_possible_pfn[zone_type]; + /* When hotadd a new node from cpu_up(), the node should be empty */ + if (!node_start_pfn && !node_end_pfn) + return 0; + + /* Get the start and end of the zone */ + *zone_start_pfn = clamp(node_start_pfn, zone_low, zone_high); + *zone_end_pfn = clamp(node_end_pfn, zone_low, zone_high); + adjust_zone_range_for_zone_movable(nid, zone_type, + node_start_pfn, node_end_pfn, + zone_start_pfn, zone_end_pfn); + + /* Check that this node has pages within the zone's required range */ + if (*zone_end_pfn < node_start_pfn || *zone_start_pfn > node_end_pfn) + return 0; + + /* Move the zone boundaries inside the node if necessary */ + *zone_end_pfn = min(*zone_end_pfn, node_end_pfn); + *zone_start_pfn = max(*zone_start_pfn, node_start_pfn); + + /* Return the spanned pages */ + return *zone_end_pfn - *zone_start_pfn; +} + +static void __init calculate_node_totalpages(struct pglist_data *pgdat, + unsigned long node_start_pfn, + unsigned long node_end_pfn) +{ + unsigned long realtotalpages = 0, totalpages = 0; + enum zone_type i; + + for (i = 0; i < MAX_NR_ZONES; i++) { + struct zone *zone = pgdat->node_zones + i; + unsigned long zone_start_pfn, zone_end_pfn; + unsigned long spanned, absent; + unsigned long size, real_size; + + spanned = zone_spanned_pages_in_node(pgdat->node_id, i, + node_start_pfn, + node_end_pfn, + &zone_start_pfn, + &zone_end_pfn); + absent = zone_absent_pages_in_node(pgdat->node_id, i, + node_start_pfn, + node_end_pfn); + + size = spanned; + real_size = size - absent; + + if (size) + zone->zone_start_pfn = zone_start_pfn; + else + zone->zone_start_pfn = 0; + zone->spanned_pages = size; + zone->present_pages = real_size; +#if defined(CONFIG_MEMORY_HOTPLUG) + zone->present_early_pages = real_size; +#endif + + totalpages += size; + realtotalpages += real_size; + } + + pgdat->node_spanned_pages = totalpages; + pgdat->node_present_pages = realtotalpages; + pr_debug("On node %d totalpages: %lu\n", pgdat->node_id, realtotalpages); +} + +static unsigned long __init calc_memmap_size(unsigned long spanned_pages, + unsigned long present_pages) +{ + unsigned long pages = spanned_pages; + + /* + * Provide a more accurate estimation if there are holes within + * the zone and SPARSEMEM is in use. If there are holes within the + * zone, each populated memory region may cost us one or two extra + * memmap pages due to alignment because memmap pages for each + * populated regions may not be naturally aligned on page boundary. + * So the (present_pages >> 4) heuristic is a tradeoff for that. + */ + if (spanned_pages > present_pages + (present_pages >> 4) && + IS_ENABLED(CONFIG_SPARSEMEM)) + pages = present_pages; + + return PAGE_ALIGN(pages * sizeof(struct page)) >> PAGE_SHIFT; +} + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static void pgdat_init_split_queue(struct pglist_data *pgdat) +{ + struct deferred_split *ds_queue = &pgdat->deferred_split_queue; + + spin_lock_init(&ds_queue->split_queue_lock); + INIT_LIST_HEAD(&ds_queue->split_queue); + ds_queue->split_queue_len = 0; +} +#else +static void pgdat_init_split_queue(struct pglist_data *pgdat) {} +#endif + +#ifdef CONFIG_COMPACTION +static void pgdat_init_kcompactd(struct pglist_data *pgdat) +{ + init_waitqueue_head(&pgdat->kcompactd_wait); +} +#else +static void pgdat_init_kcompactd(struct pglist_data *pgdat) {} +#endif + +static void __meminit pgdat_init_internals(struct pglist_data *pgdat) +{ + int i; + + pgdat_resize_init(pgdat); + pgdat_kswapd_lock_init(pgdat); + + pgdat_init_split_queue(pgdat); + pgdat_init_kcompactd(pgdat); + + init_waitqueue_head(&pgdat->kswapd_wait); + init_waitqueue_head(&pgdat->pfmemalloc_wait); + + for (i = 0; i < NR_VMSCAN_THROTTLE; i++) + init_waitqueue_head(&pgdat->reclaim_wait[i]); + + pgdat_page_ext_init(pgdat); + lruvec_init(&pgdat->__lruvec); +} + +static void __meminit zone_init_internals(struct zone *zone, enum zone_type idx, int nid, + unsigned long remaining_pages) +{ + atomic_long_set(&zone->managed_pages, remaining_pages); + zone_set_nid(zone, nid); + zone->name = zone_names[idx]; + zone->zone_pgdat = NODE_DATA(nid); + spin_lock_init(&zone->lock); + zone_seqlock_init(zone); + zone_pcp_init(zone); +} + +static void __meminit zone_init_free_lists(struct zone *zone) +{ + unsigned int order, t; + for_each_migratetype_order(order, t) { + INIT_LIST_HEAD(&zone->free_area[order].free_list[t]); + zone->free_area[order].nr_free = 0; + } +} + +void __meminit init_currently_empty_zone(struct zone *zone, + unsigned long zone_start_pfn, + unsigned long size) +{ + struct pglist_data *pgdat = zone->zone_pgdat; + int zone_idx = zone_idx(zone) + 1; + + if (zone_idx > pgdat->nr_zones) + pgdat->nr_zones = zone_idx; + + zone->zone_start_pfn = zone_start_pfn; + + mminit_dprintk(MMINIT_TRACE, "memmap_init", + "Initialising map node %d zone %lu pfns %lu -> %lu\n", + pgdat->node_id, + (unsigned long)zone_idx(zone), + zone_start_pfn, (zone_start_pfn + size)); + + zone_init_free_lists(zone); + zone->initialized = 1; +} + +#ifndef CONFIG_SPARSEMEM +/* + * Calculate the size of the zone->blockflags rounded to an unsigned long + * Start by making sure zonesize is a multiple of pageblock_order by rounding + * up. Then use 1 NR_PAGEBLOCK_BITS worth of bits per pageblock, finally + * round what is now in bits to nearest long in bits, then return it in + * bytes. + */ +static unsigned long __init usemap_size(unsigned long zone_start_pfn, unsigned long zonesize) +{ + unsigned long usemapsize; + + zonesize += zone_start_pfn & (pageblock_nr_pages-1); + usemapsize = roundup(zonesize, pageblock_nr_pages); + usemapsize = usemapsize >> pageblock_order; + usemapsize *= NR_PAGEBLOCK_BITS; + usemapsize = roundup(usemapsize, 8 * sizeof(unsigned long)); + + return usemapsize / 8; +} + +static void __ref setup_usemap(struct zone *zone) +{ + unsigned long usemapsize = usemap_size(zone->zone_start_pfn, + zone->spanned_pages); + zone->pageblock_flags = NULL; + if (usemapsize) { + zone->pageblock_flags = + memblock_alloc_node(usemapsize, SMP_CACHE_BYTES, + zone_to_nid(zone)); + if (!zone->pageblock_flags) + panic("Failed to allocate %ld bytes for zone %s pageblock flags on node %d\n", + usemapsize, zone->name, zone_to_nid(zone)); + } +} +#else +static inline void setup_usemap(struct zone *zone) {} +#endif /* CONFIG_SPARSEMEM */ + +#ifdef CONFIG_HUGETLB_PAGE_SIZE_VARIABLE + +/* Initialise the number of pages represented by NR_PAGEBLOCK_BITS */ +void __init set_pageblock_order(void) +{ + unsigned int order = MAX_ORDER; + + /* Check that pageblock_nr_pages has not already been setup */ + if (pageblock_order) + return; + + /* Don't let pageblocks exceed the maximum allocation granularity. */ + if (HPAGE_SHIFT > PAGE_SHIFT && HUGETLB_PAGE_ORDER < order) + order = HUGETLB_PAGE_ORDER; + + /* + * Assume the largest contiguous order of interest is a huge page. + * This value may be variable depending on boot parameters on IA64 and + * powerpc. + */ + pageblock_order = order; +} +#else /* CONFIG_HUGETLB_PAGE_SIZE_VARIABLE */ + +/* + * When CONFIG_HUGETLB_PAGE_SIZE_VARIABLE is not set, set_pageblock_order() + * is unused as pageblock_order is set at compile-time. See + * include/linux/pageblock-flags.h for the values of pageblock_order based on + * the kernel config + */ +void __init set_pageblock_order(void) +{ +} + +#endif /* CONFIG_HUGETLB_PAGE_SIZE_VARIABLE */ + +/* + * Set up the zone data structures + * - init pgdat internals + * - init all zones belonging to this node + * + * NOTE: this function is only called during memory hotplug + */ +#ifdef CONFIG_MEMORY_HOTPLUG +void __ref free_area_init_core_hotplug(struct pglist_data *pgdat) +{ + int nid = pgdat->node_id; + enum zone_type z; + int cpu; + + pgdat_init_internals(pgdat); + + if (pgdat->per_cpu_nodestats == &boot_nodestats) + pgdat->per_cpu_nodestats = alloc_percpu(struct per_cpu_nodestat); + + /* + * Reset the nr_zones, order and highest_zoneidx before reuse. + * Note that kswapd will init kswapd_highest_zoneidx properly + * when it starts in the near future. + */ + pgdat->nr_zones = 0; + pgdat->kswapd_order = 0; + pgdat->kswapd_highest_zoneidx = 0; + pgdat->node_start_pfn = 0; + for_each_online_cpu(cpu) { + struct per_cpu_nodestat *p; + + p = per_cpu_ptr(pgdat->per_cpu_nodestats, cpu); + memset(p, 0, sizeof(*p)); + } + + for (z = 0; z < MAX_NR_ZONES; z++) + zone_init_internals(&pgdat->node_zones[z], z, nid, 0); +} +#endif + +/* + * Set up the zone data structures: + * - mark all pages reserved + * - mark all memory queues empty + * - clear the memory bitmaps + * + * NOTE: pgdat should get zeroed by caller. + * NOTE: this function is only called during early init. + */ +static void __init free_area_init_core(struct pglist_data *pgdat) +{ + enum zone_type j; + int nid = pgdat->node_id; + + pgdat_init_internals(pgdat); + pgdat->per_cpu_nodestats = &boot_nodestats; + + for (j = 0; j < MAX_NR_ZONES; j++) { + struct zone *zone = pgdat->node_zones + j; + unsigned long size, freesize, memmap_pages; + + size = zone->spanned_pages; + freesize = zone->present_pages; + + /* + * Adjust freesize so that it accounts for how much memory + * is used by this zone for memmap. This affects the watermark + * and per-cpu initialisations + */ + memmap_pages = calc_memmap_size(size, freesize); + if (!is_highmem_idx(j)) { + if (freesize >= memmap_pages) { + freesize -= memmap_pages; + if (memmap_pages) + pr_debug(" %s zone: %lu pages used for memmap\n", + zone_names[j], memmap_pages); + } else + pr_warn(" %s zone: %lu memmap pages exceeds freesize %lu\n", + zone_names[j], memmap_pages, freesize); + } + + /* Account for reserved pages */ + if (j == 0 && freesize > dma_reserve) { + freesize -= dma_reserve; + pr_debug(" %s zone: %lu pages reserved\n", zone_names[0], dma_reserve); + } + + if (!is_highmem_idx(j)) + nr_kernel_pages += freesize; + /* Charge for highmem memmap if there are enough kernel pages */ + else if (nr_kernel_pages > memmap_pages * 2) + nr_kernel_pages -= memmap_pages; + nr_all_pages += freesize; + + /* + * Set an approximate value for lowmem here, it will be adjusted + * when the bootmem allocator frees pages into the buddy system. + * And all highmem pages will be managed by the buddy system. + */ + zone_init_internals(zone, j, nid, freesize); + + if (!size) + continue; + + set_pageblock_order(); + setup_usemap(zone); + init_currently_empty_zone(zone, zone->zone_start_pfn, size); + } +} + +void __init *memmap_alloc(phys_addr_t size, phys_addr_t align, + phys_addr_t min_addr, int nid, bool exact_nid) +{ + void *ptr; + + if (exact_nid) + ptr = memblock_alloc_exact_nid_raw(size, align, min_addr, + MEMBLOCK_ALLOC_ACCESSIBLE, + nid); + else + ptr = memblock_alloc_try_nid_raw(size, align, min_addr, + MEMBLOCK_ALLOC_ACCESSIBLE, + nid); + + if (ptr && size > 0) + page_init_poison(ptr, size); + + return ptr; +} + +#ifdef CONFIG_FLATMEM +static void __init alloc_node_mem_map(struct pglist_data *pgdat) +{ + unsigned long __maybe_unused start = 0; + unsigned long __maybe_unused offset = 0; + + /* Skip empty nodes */ + if (!pgdat->node_spanned_pages) + return; + + start = pgdat->node_start_pfn & ~(MAX_ORDER_NR_PAGES - 1); + offset = pgdat->node_start_pfn - start; + /* ia64 gets its own node_mem_map, before this, without bootmem */ + if (!pgdat->node_mem_map) { + unsigned long size, end; + struct page *map; + + /* + * The zone's endpoints aren't required to be MAX_ORDER + * aligned but the node_mem_map endpoints must be in order + * for the buddy allocator to function correctly. + */ + end = pgdat_end_pfn(pgdat); + end = ALIGN(end, MAX_ORDER_NR_PAGES); + size = (end - start) * sizeof(struct page); + map = memmap_alloc(size, SMP_CACHE_BYTES, MEMBLOCK_LOW_LIMIT, + pgdat->node_id, false); + if (!map) + panic("Failed to allocate %ld bytes for node %d memory map\n", + size, pgdat->node_id); + pgdat->node_mem_map = map + offset; + } + pr_debug("%s: node %d, pgdat %08lx, node_mem_map %08lx\n", + __func__, pgdat->node_id, (unsigned long)pgdat, + (unsigned long)pgdat->node_mem_map); +#ifndef CONFIG_NUMA + /* + * With no DISCONTIG, the global mem_map is just set as node 0's + */ + if (pgdat == NODE_DATA(0)) { + mem_map = NODE_DATA(0)->node_mem_map; + if (page_to_pfn(mem_map) != pgdat->node_start_pfn) + mem_map -= offset; + } +#endif +} +#else +static inline void alloc_node_mem_map(struct pglist_data *pgdat) { } +#endif /* CONFIG_FLATMEM */ + +/** + * get_pfn_range_for_nid - Return the start and end page frames for a node + * @nid: The nid to return the range for. If MAX_NUMNODES, the min and max PFN are returned. + * @start_pfn: Passed by reference. On return, it will have the node start_pfn. + * @end_pfn: Passed by reference. On return, it will have the node end_pfn. + * + * It returns the start and end page frame of a node based on information + * provided by memblock_set_node(). If called for a node + * with no available memory, a warning is printed and the start and end + * PFNs will be 0. + */ +void __init get_pfn_range_for_nid(unsigned int nid, + unsigned long *start_pfn, unsigned long *end_pfn) +{ + unsigned long this_start_pfn, this_end_pfn; + int i; + + *start_pfn = -1UL; + *end_pfn = 0; + + for_each_mem_pfn_range(i, nid, &this_start_pfn, &this_end_pfn, NULL) { + *start_pfn = min(*start_pfn, this_start_pfn); + *end_pfn = max(*end_pfn, this_end_pfn); + } + + if (*start_pfn == -1UL) + *start_pfn = 0; +} + +static void __init free_area_init_node(int nid) +{ + pg_data_t *pgdat = NODE_DATA(nid); + unsigned long start_pfn = 0; + unsigned long end_pfn = 0; + + /* pg_data_t should be reset to zero when it's allocated */ + WARN_ON(pgdat->nr_zones || pgdat->kswapd_highest_zoneidx); + + get_pfn_range_for_nid(nid, &start_pfn, &end_pfn); + + pgdat->node_id = nid; + pgdat->node_start_pfn = start_pfn; + pgdat->per_cpu_nodestats = NULL; + + if (start_pfn != end_pfn) { + pr_info("Initmem setup node %d [mem %#018Lx-%#018Lx]\n", nid, + (u64)start_pfn << PAGE_SHIFT, + end_pfn ? ((u64)end_pfn << PAGE_SHIFT) - 1 : 0); + } else { + pr_info("Initmem setup node %d as memoryless\n", nid); + } + + calculate_node_totalpages(pgdat, start_pfn, end_pfn); + + alloc_node_mem_map(pgdat); + pgdat_set_deferred_range(pgdat); + + free_area_init_core(pgdat); + lru_gen_init_pgdat(pgdat); +} + +/* Any regular or high memory on that node ? */ +static void check_for_memory(pg_data_t *pgdat, int nid) +{ + enum zone_type zone_type; + + for (zone_type = 0; zone_type <= ZONE_MOVABLE - 1; zone_type++) { + struct zone *zone = &pgdat->node_zones[zone_type]; + if (populated_zone(zone)) { + if (IS_ENABLED(CONFIG_HIGHMEM)) + node_set_state(nid, N_HIGH_MEMORY); + if (zone_type <= ZONE_NORMAL) + node_set_state(nid, N_NORMAL_MEMORY); + break; + } + } +} + +#if MAX_NUMNODES > 1 +/* + * Figure out the number of possible node ids. + */ +void __init setup_nr_node_ids(void) +{ + unsigned int highest; + + highest = find_last_bit(node_possible_map.bits, MAX_NUMNODES); + nr_node_ids = highest + 1; +} +#endif + +static void __init free_area_init_memoryless_node(int nid) +{ + free_area_init_node(nid); +} + +/* + * Some architectures, e.g. ARC may have ZONE_HIGHMEM below ZONE_NORMAL. For + * such cases we allow max_zone_pfn sorted in the descending order + */ +bool __weak arch_has_descending_max_zone_pfns(void) +{ + return false; +} + +/** + * free_area_init - Initialise all pg_data_t and zone data + * @max_zone_pfn: an array of max PFNs for each zone + * + * This will call free_area_init_node() for each active node in the system. + * Using the page ranges provided by memblock_set_node(), the size of each + * zone in each node and their holes is calculated. If the maximum PFN + * between two adjacent zones match, it is assumed that the zone is empty. + * For example, if arch_max_dma_pfn == arch_max_dma32_pfn, it is assumed + * that arch_max_dma32_pfn has no pages. It is also assumed that a zone + * starts where the previous one ended. For example, ZONE_DMA32 starts + * at arch_max_dma_pfn. + */ +void __init free_area_init(unsigned long *max_zone_pfn) +{ + unsigned long start_pfn, end_pfn; + int i, nid, zone; + bool descending; + + /* Record where the zone boundaries are */ + memset(arch_zone_lowest_possible_pfn, 0, + sizeof(arch_zone_lowest_possible_pfn)); + memset(arch_zone_highest_possible_pfn, 0, + sizeof(arch_zone_highest_possible_pfn)); + + start_pfn = PHYS_PFN(memblock_start_of_DRAM()); + descending = arch_has_descending_max_zone_pfns(); + + for (i = 0; i < MAX_NR_ZONES; i++) { + if (descending) + zone = MAX_NR_ZONES - i - 1; + else + zone = i; + + if (zone == ZONE_MOVABLE) + continue; + + end_pfn = max(max_zone_pfn[zone], start_pfn); + arch_zone_lowest_possible_pfn[zone] = start_pfn; + arch_zone_highest_possible_pfn[zone] = end_pfn; + + start_pfn = end_pfn; + } + + /* Find the PFNs that ZONE_MOVABLE begins at in each node */ + memset(zone_movable_pfn, 0, sizeof(zone_movable_pfn)); + find_zone_movable_pfns_for_nodes(); + + /* Print out the zone ranges */ + pr_info("Zone ranges:\n"); + for (i = 0; i < MAX_NR_ZONES; i++) { + if (i == ZONE_MOVABLE) + continue; + pr_info(" %-8s ", zone_names[i]); + if (arch_zone_lowest_possible_pfn[i] == + arch_zone_highest_possible_pfn[i]) + pr_cont("empty\n"); + else + pr_cont("[mem %#018Lx-%#018Lx]\n", + (u64)arch_zone_lowest_possible_pfn[i] + << PAGE_SHIFT, + ((u64)arch_zone_highest_possible_pfn[i] + << PAGE_SHIFT) - 1); + } + + /* Print out the PFNs ZONE_MOVABLE begins at in each node */ + pr_info("Movable zone start for each node\n"); + for (i = 0; i < MAX_NUMNODES; i++) { + if (zone_movable_pfn[i]) + pr_info(" Node %d: %#018Lx\n", i, + (u64)zone_movable_pfn[i] << PAGE_SHIFT); + } + + /* + * Print out the early node map, and initialize the + * subsection-map relative to active online memory ranges to + * enable future "sub-section" extensions of the memory map. + */ + pr_info("Early memory node ranges\n"); + for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, &nid) { + pr_info(" node %3d: [mem %#018Lx-%#018Lx]\n", nid, + (u64)start_pfn << PAGE_SHIFT, + ((u64)end_pfn << PAGE_SHIFT) - 1); + subsection_map_init(start_pfn, end_pfn - start_pfn); + } + + /* Initialise every node */ + mminit_verify_pageflags_layout(); + setup_nr_node_ids(); + for_each_node(nid) { + pg_data_t *pgdat; + + if (!node_online(nid)) { + pr_info("Initializing node %d as memoryless\n", nid); + + /* Allocator not initialized yet */ + pgdat = arch_alloc_nodedata(nid); + if (!pgdat) + panic("Cannot allocate %zuB for node %d.\n", + sizeof(*pgdat), nid); + arch_refresh_nodedata(nid, pgdat); + free_area_init_memoryless_node(nid); + + /* + * We do not want to confuse userspace by sysfs + * files/directories for node without any memory + * attached to it, so this node is not marked as + * N_MEMORY and not marked online so that no sysfs + * hierarchy will be created via register_one_node for + * it. The pgdat will get fully initialized by + * hotadd_init_pgdat() when memory is hotplugged into + * this node. + */ + continue; + } + + pgdat = NODE_DATA(nid); + free_area_init_node(nid); + + /* Any memory on that node */ + if (pgdat->node_present_pages) + node_set_state(nid, N_MEMORY); + check_for_memory(pgdat, nid); + } + + memmap_init(); +} + +/** + * node_map_pfn_alignment - determine the maximum internode alignment + * + * This function should be called after node map is populated and sorted. + * It calculates the maximum power of two alignment which can distinguish + * all the nodes. + * + * For example, if all nodes are 1GiB and aligned to 1GiB, the return value + * would indicate 1GiB alignment with (1 << (30 - PAGE_SHIFT)). If the + * nodes are shifted by 256MiB, 256MiB. Note that if only the last node is + * shifted, 1GiB is enough and this function will indicate so. + * + * This is used to test whether pfn -> nid mapping of the chosen memory + * model has fine enough granularity to avoid incorrect mapping for the + * populated node map. + * + * Return: the determined alignment in pfn's. 0 if there is no alignment + * requirement (single node). + */ +unsigned long __init node_map_pfn_alignment(void) +{ + unsigned long accl_mask = 0, last_end = 0; + unsigned long start, end, mask; + int last_nid = NUMA_NO_NODE; + int i, nid; + + for_each_mem_pfn_range(i, MAX_NUMNODES, &start, &end, &nid) { + if (!start || last_nid < 0 || last_nid == nid) { + last_nid = nid; + last_end = end; + continue; + } + + /* + * Start with a mask granular enough to pin-point to the + * start pfn and tick off bits one-by-one until it becomes + * too coarse to separate the current node from the last. + */ + mask = ~((1 << __ffs(start)) - 1); + while (mask && last_end <= (start & (mask << 1))) + mask <<= 1; + + /* accumulate all internode masks */ + accl_mask |= mask; + } + + /* convert mask to number of pages */ + return ~accl_mask + 1; +} + +#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT +static void __init deferred_free_range(unsigned long pfn, + unsigned long nr_pages) +{ + struct page *page; + unsigned long i; + + if (!nr_pages) + return; + + page = pfn_to_page(pfn); + + /* Free a large naturally-aligned chunk if possible */ + if (nr_pages == pageblock_nr_pages && pageblock_aligned(pfn)) { + set_pageblock_migratetype(page, MIGRATE_MOVABLE); + __free_pages_core(page, pageblock_order); + return; + } + + for (i = 0; i < nr_pages; i++, page++, pfn++) { + if (pageblock_aligned(pfn)) + set_pageblock_migratetype(page, MIGRATE_MOVABLE); + __free_pages_core(page, 0); + } +} + +/* Completion tracking for deferred_init_memmap() threads */ +static atomic_t pgdat_init_n_undone __initdata; +static __initdata DECLARE_COMPLETION(pgdat_init_all_done_comp); + +static inline void __init pgdat_init_report_one_done(void) +{ + if (atomic_dec_and_test(&pgdat_init_n_undone)) + complete(&pgdat_init_all_done_comp); +} + +/* + * Returns true if page needs to be initialized or freed to buddy allocator. + * + * We check if a current large page is valid by only checking the validity + * of the head pfn. + */ +static inline bool __init deferred_pfn_valid(unsigned long pfn) +{ + if (pageblock_aligned(pfn) && !pfn_valid(pfn)) + return false; + return true; +} + +/* + * Free pages to buddy allocator. Try to free aligned pages in + * pageblock_nr_pages sizes. + */ +static void __init deferred_free_pages(unsigned long pfn, + unsigned long end_pfn) +{ + unsigned long nr_free = 0; + + for (; pfn < end_pfn; pfn++) { + if (!deferred_pfn_valid(pfn)) { + deferred_free_range(pfn - nr_free, nr_free); + nr_free = 0; + } else if (pageblock_aligned(pfn)) { + deferred_free_range(pfn - nr_free, nr_free); + nr_free = 1; + } else { + nr_free++; + } + } + /* Free the last block of pages to allocator */ + deferred_free_range(pfn - nr_free, nr_free); +} + +/* + * Initialize struct pages. We minimize pfn page lookups and scheduler checks + * by performing it only once every pageblock_nr_pages. + * Return number of pages initialized. + */ +static unsigned long __init deferred_init_pages(struct zone *zone, + unsigned long pfn, + unsigned long end_pfn) +{ + int nid = zone_to_nid(zone); + unsigned long nr_pages = 0; + int zid = zone_idx(zone); + struct page *page = NULL; + + for (; pfn < end_pfn; pfn++) { + if (!deferred_pfn_valid(pfn)) { + page = NULL; + continue; + } else if (!page || pageblock_aligned(pfn)) { + page = pfn_to_page(pfn); + } else { + page++; + } + __init_single_page(page, pfn, zid, nid); + nr_pages++; + } + return (nr_pages); +} + +/* + * This function is meant to pre-load the iterator for the zone init. + * Specifically it walks through the ranges until we are caught up to the + * first_init_pfn value and exits there. If we never encounter the value we + * return false indicating there are no valid ranges left. + */ +static bool __init +deferred_init_mem_pfn_range_in_zone(u64 *i, struct zone *zone, + unsigned long *spfn, unsigned long *epfn, + unsigned long first_init_pfn) +{ + u64 j; + + /* + * Start out by walking through the ranges in this zone that have + * already been initialized. We don't need to do anything with them + * so we just need to flush them out of the system. + */ + for_each_free_mem_pfn_range_in_zone(j, zone, spfn, epfn) { + if (*epfn <= first_init_pfn) + continue; + if (*spfn < first_init_pfn) + *spfn = first_init_pfn; + *i = j; + return true; + } + + return false; +} + +/* + * Initialize and free pages. We do it in two loops: first we initialize + * struct page, then free to buddy allocator, because while we are + * freeing pages we can access pages that are ahead (computing buddy + * page in __free_one_page()). + * + * In order to try and keep some memory in the cache we have the loop + * broken along max page order boundaries. This way we will not cause + * any issues with the buddy page computation. + */ +static unsigned long __init +deferred_init_maxorder(u64 *i, struct zone *zone, unsigned long *start_pfn, + unsigned long *end_pfn) +{ + unsigned long mo_pfn = ALIGN(*start_pfn + 1, MAX_ORDER_NR_PAGES); + unsigned long spfn = *start_pfn, epfn = *end_pfn; + unsigned long nr_pages = 0; + u64 j = *i; + + /* First we loop through and initialize the page values */ + for_each_free_mem_pfn_range_in_zone_from(j, zone, start_pfn, end_pfn) { + unsigned long t; + + if (mo_pfn <= *start_pfn) + break; + + t = min(mo_pfn, *end_pfn); + nr_pages += deferred_init_pages(zone, *start_pfn, t); + + if (mo_pfn < *end_pfn) { + *start_pfn = mo_pfn; + break; + } + } + + /* Reset values and now loop through freeing pages as needed */ + swap(j, *i); + + for_each_free_mem_pfn_range_in_zone_from(j, zone, &spfn, &epfn) { + unsigned long t; + + if (mo_pfn <= spfn) + break; + + t = min(mo_pfn, epfn); + deferred_free_pages(spfn, t); + + if (mo_pfn <= epfn) + break; + } + + return nr_pages; +} + +static void __init +deferred_init_memmap_chunk(unsigned long start_pfn, unsigned long end_pfn, + void *arg) +{ + unsigned long spfn, epfn; + struct zone *zone = arg; + u64 i; + + deferred_init_mem_pfn_range_in_zone(&i, zone, &spfn, &epfn, start_pfn); + + /* + * Initialize and free pages in MAX_ORDER sized increments so that we + * can avoid introducing any issues with the buddy allocator. + */ + while (spfn < end_pfn) { + deferred_init_maxorder(&i, zone, &spfn, &epfn); + cond_resched(); + } +} + +/* An arch may override for more concurrency. */ +__weak int __init +deferred_page_init_max_threads(const struct cpumask *node_cpumask) +{ + return 1; +} + +/* Initialise remaining memory on a node */ +static int __init deferred_init_memmap(void *data) +{ + pg_data_t *pgdat = data; + const struct cpumask *cpumask = cpumask_of_node(pgdat->node_id); + unsigned long spfn = 0, epfn = 0; + unsigned long first_init_pfn, flags; + unsigned long start = jiffies; + struct zone *zone; + int zid, max_threads; + u64 i; + + /* Bind memory initialisation thread to a local node if possible */ + if (!cpumask_empty(cpumask)) + set_cpus_allowed_ptr(current, cpumask); + + pgdat_resize_lock(pgdat, &flags); + first_init_pfn = pgdat->first_deferred_pfn; + if (first_init_pfn == ULONG_MAX) { + pgdat_resize_unlock(pgdat, &flags); + pgdat_init_report_one_done(); + return 0; + } + + /* Sanity check boundaries */ + BUG_ON(pgdat->first_deferred_pfn < pgdat->node_start_pfn); + BUG_ON(pgdat->first_deferred_pfn > pgdat_end_pfn(pgdat)); + pgdat->first_deferred_pfn = ULONG_MAX; + + /* + * Once we unlock here, the zone cannot be grown anymore, thus if an + * interrupt thread must allocate this early in boot, zone must be + * pre-grown prior to start of deferred page initialization. + */ + pgdat_resize_unlock(pgdat, &flags); + + /* Only the highest zone is deferred so find it */ + for (zid = 0; zid < MAX_NR_ZONES; zid++) { + zone = pgdat->node_zones + zid; + if (first_init_pfn < zone_end_pfn(zone)) + break; + } + + /* If the zone is empty somebody else may have cleared out the zone */ + if (!deferred_init_mem_pfn_range_in_zone(&i, zone, &spfn, &epfn, + first_init_pfn)) + goto zone_empty; + + max_threads = deferred_page_init_max_threads(cpumask); + + while (spfn < epfn) { + unsigned long epfn_align = ALIGN(epfn, PAGES_PER_SECTION); + struct padata_mt_job job = { + .thread_fn = deferred_init_memmap_chunk, + .fn_arg = zone, + .start = spfn, + .size = epfn_align - spfn, + .align = PAGES_PER_SECTION, + .min_chunk = PAGES_PER_SECTION, + .max_threads = max_threads, + }; + + padata_do_multithreaded(&job); + deferred_init_mem_pfn_range_in_zone(&i, zone, &spfn, &epfn, + epfn_align); + } +zone_empty: + /* Sanity check that the next zone really is unpopulated */ + WARN_ON(++zid < MAX_NR_ZONES && populated_zone(++zone)); + + pr_info("node %d deferred pages initialised in %ums\n", + pgdat->node_id, jiffies_to_msecs(jiffies - start)); + + pgdat_init_report_one_done(); + return 0; +} + +/* + * If this zone has deferred pages, try to grow it by initializing enough + * deferred pages to satisfy the allocation specified by order, rounded up to + * the nearest PAGES_PER_SECTION boundary. So we're adding memory in increments + * of SECTION_SIZE bytes by initializing struct pages in increments of + * PAGES_PER_SECTION * sizeof(struct page) bytes. + * + * Return true when zone was grown, otherwise return false. We return true even + * when we grow less than requested, to let the caller decide if there are + * enough pages to satisfy the allocation. + * + * Note: We use noinline because this function is needed only during boot, and + * it is called from a __ref function _deferred_grow_zone. This way we are + * making sure that it is not inlined into permanent text section. + */ +bool __init deferred_grow_zone(struct zone *zone, unsigned int order) +{ + unsigned long nr_pages_needed = ALIGN(1 << order, PAGES_PER_SECTION); + pg_data_t *pgdat = zone->zone_pgdat; + unsigned long first_deferred_pfn = pgdat->first_deferred_pfn; + unsigned long spfn, epfn, flags; + unsigned long nr_pages = 0; + u64 i; + + /* Only the last zone may have deferred pages */ + if (zone_end_pfn(zone) != pgdat_end_pfn(pgdat)) + return false; + + pgdat_resize_lock(pgdat, &flags); + + /* + * If someone grew this zone while we were waiting for spinlock, return + * true, as there might be enough pages already. + */ + if (first_deferred_pfn != pgdat->first_deferred_pfn) { + pgdat_resize_unlock(pgdat, &flags); + return true; + } + + /* If the zone is empty somebody else may have cleared out the zone */ + if (!deferred_init_mem_pfn_range_in_zone(&i, zone, &spfn, &epfn, + first_deferred_pfn)) { + pgdat->first_deferred_pfn = ULONG_MAX; + pgdat_resize_unlock(pgdat, &flags); + /* Retry only once. */ + return first_deferred_pfn != ULONG_MAX; + } + + /* + * Initialize and free pages in MAX_ORDER sized increments so + * that we can avoid introducing any issues with the buddy + * allocator. + */ + while (spfn < epfn) { + /* update our first deferred PFN for this section */ + first_deferred_pfn = spfn; + + nr_pages += deferred_init_maxorder(&i, zone, &spfn, &epfn); + touch_nmi_watchdog(); + + /* We should only stop along section boundaries */ + if ((first_deferred_pfn ^ spfn) < PAGES_PER_SECTION) + continue; + + /* If our quota has been met we can stop here */ + if (nr_pages >= nr_pages_needed) + break; + } + + pgdat->first_deferred_pfn = spfn; + pgdat_resize_unlock(pgdat, &flags); + + return nr_pages > 0; +} + +#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ + +void __init page_alloc_init_late(void) +{ + struct zone *zone; + int nid; + +#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT + + /* There will be num_node_state(N_MEMORY) threads */ + atomic_set(&pgdat_init_n_undone, num_node_state(N_MEMORY)); + for_each_node_state(nid, N_MEMORY) { + kthread_run(deferred_init_memmap, NODE_DATA(nid), "pgdatinit%d", nid); + } + + /* Block until all are initialised */ + wait_for_completion(&pgdat_init_all_done_comp); + + /* + * We initialized the rest of the deferred pages. Permanently disable + * on-demand struct page initialization. + */ + static_branch_disable(&deferred_pages); + + /* Reinit limits that are based on free pages after the kernel is up */ + files_maxfiles_init(); +#endif + + buffer_init(); + + /* Discard memblock private memory */ + memblock_discard(); + + for_each_node_state(nid, N_MEMORY) + shuffle_free_memory(NODE_DATA(nid)); + + for_each_populated_zone(zone) + set_zone_contiguous(zone); +} + +#ifndef __HAVE_ARCH_RESERVED_KERNEL_PAGES +/* + * Returns the number of pages that arch has reserved but + * is not known to alloc_large_system_hash(). + */ +static unsigned long __init arch_reserved_kernel_pages(void) +{ + return 0; +} +#endif + +/* + * Adaptive scale is meant to reduce sizes of hash tables on large memory + * machines. As memory size is increased the scale is also increased but at + * slower pace. Starting from ADAPT_SCALE_BASE (64G), every time memory + * quadruples the scale is increased by one, which means the size of hash table + * only doubles, instead of quadrupling as well. + * Because 32-bit systems cannot have large physical memory, where this scaling + * makes sense, it is disabled on such platforms. + */ +#if __BITS_PER_LONG > 32 +#define ADAPT_SCALE_BASE (64ul << 30) +#define ADAPT_SCALE_SHIFT 2 +#define ADAPT_SCALE_NPAGES (ADAPT_SCALE_BASE >> PAGE_SHIFT) +#endif + +/* + * allocate a large system hash table from bootmem + * - it is assumed that the hash table must contain an exact power-of-2 + * quantity of entries + * - limit is the number of hash buckets, not the total allocation size + */ +void *__init alloc_large_system_hash(const char *tablename, + unsigned long bucketsize, + unsigned long numentries, + int scale, + int flags, + unsigned int *_hash_shift, + unsigned int *_hash_mask, + unsigned long low_limit, + unsigned long high_limit) +{ + unsigned long long max = high_limit; + unsigned long log2qty, size; + void *table; + gfp_t gfp_flags; + bool virt; + bool huge; + + /* allow the kernel cmdline to have a say */ + if (!numentries) { + /* round applicable memory size up to nearest megabyte */ + numentries = nr_kernel_pages; + numentries -= arch_reserved_kernel_pages(); + + /* It isn't necessary when PAGE_SIZE >= 1MB */ + if (PAGE_SIZE < SZ_1M) + numentries = round_up(numentries, SZ_1M / PAGE_SIZE); + +#if __BITS_PER_LONG > 32 + if (!high_limit) { + unsigned long adapt; + + for (adapt = ADAPT_SCALE_NPAGES; adapt < numentries; + adapt <<= ADAPT_SCALE_SHIFT) + scale++; + } +#endif + + /* limit to 1 bucket per 2^scale bytes of low memory */ + if (scale > PAGE_SHIFT) + numentries >>= (scale - PAGE_SHIFT); + else + numentries <<= (PAGE_SHIFT - scale); + + /* Make sure we've got at least a 0-order allocation.. */ + if (unlikely(flags & HASH_SMALL)) { + /* Makes no sense without HASH_EARLY */ + WARN_ON(!(flags & HASH_EARLY)); + if (!(numentries >> *_hash_shift)) { + numentries = 1UL << *_hash_shift; + BUG_ON(!numentries); + } + } else if (unlikely((numentries * bucketsize) < PAGE_SIZE)) + numentries = PAGE_SIZE / bucketsize; + } + numentries = roundup_pow_of_two(numentries); + + /* limit allocation size to 1/16 total memory by default */ + if (max == 0) { + max = ((unsigned long long)nr_all_pages << PAGE_SHIFT) >> 4; + do_div(max, bucketsize); + } + max = min(max, 0x80000000ULL); + + if (numentries < low_limit) + numentries = low_limit; + if (numentries > max) + numentries = max; + + log2qty = ilog2(numentries); + + gfp_flags = (flags & HASH_ZERO) ? GFP_ATOMIC | __GFP_ZERO : GFP_ATOMIC; + do { + virt = false; + size = bucketsize << log2qty; + if (flags & HASH_EARLY) { + if (flags & HASH_ZERO) + table = memblock_alloc(size, SMP_CACHE_BYTES); + else + table = memblock_alloc_raw(size, + SMP_CACHE_BYTES); + } else if (get_order(size) > MAX_ORDER || hashdist) { + table = vmalloc_huge(size, gfp_flags); + virt = true; + if (table) + huge = is_vm_area_hugepages(table); + } else { + /* + * If bucketsize is not a power-of-two, we may free + * some pages at the end of hash table which + * alloc_pages_exact() automatically does + */ + table = alloc_pages_exact(size, gfp_flags); + kmemleak_alloc(table, size, 1, gfp_flags); + } + } while (!table && size > PAGE_SIZE && --log2qty); + + if (!table) + panic("Failed to allocate %s hash table\n", tablename); + + pr_info("%s hash table entries: %ld (order: %d, %lu bytes, %s)\n", + tablename, 1UL << log2qty, ilog2(size) - PAGE_SHIFT, size, + virt ? (huge ? "vmalloc hugepage" : "vmalloc") : "linear"); + + if (_hash_shift) + *_hash_shift = log2qty; + if (_hash_mask) + *_hash_mask = (1 << log2qty) - 1; + + return table; +} + +/** + * set_dma_reserve - set the specified number of pages reserved in the first zone + * @new_dma_reserve: The number of pages to mark reserved + * + * The per-cpu batchsize and zone watermarks are determined by managed_pages. + * In the DMA zone, a significant percentage may be consumed by kernel image + * and other unfreeable allocations which can skew the watermarks badly. This + * function may optionally be used to account for unfreeable pages in the + * first zone (e.g., ZONE_DMA). The effect will be lower watermarks and + * smaller per-cpu batchsize. + */ +void __init set_dma_reserve(unsigned long new_dma_reserve) +{ + dma_reserve = new_dma_reserve; +} + +void __init memblock_free_pages(struct page *page, unsigned long pfn, + unsigned int order) +{ + if (!early_page_initialised(pfn)) + return; + if (!kmsan_memblock_free_pages(page, order)) { + /* KMSAN will take care of these pages. */ + return; + } + __free_pages_core(page, order); +} diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e52f90d5d6a3..c56c147bdf27 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -72,9 +72,7 @@ #include #include #include -#include #include -#include #include #include #include @@ -355,7 +353,7 @@ int sysctl_lowmem_reserve_ratio[MAX_NR_ZONES] = { [ZONE_MOVABLE] = 0, }; -static char * const zone_names[MAX_NR_ZONES] = { +char * const zone_names[MAX_NR_ZONES] = { #ifdef CONFIG_ZONE_DMA "DMA", #endif @@ -401,17 +399,6 @@ int user_min_free_kbytes = -1; int watermark_boost_factor __read_mostly = 15000; int watermark_scale_factor = 10; -static unsigned long nr_kernel_pages __initdata; -static unsigned long nr_all_pages __initdata; -static unsigned long dma_reserve __initdata; - -static unsigned long arch_zone_lowest_possible_pfn[MAX_NR_ZONES] __initdata; -static unsigned long arch_zone_highest_possible_pfn[MAX_NR_ZONES] __initdata; -static unsigned long required_kernelcore __initdata; -static unsigned long required_kernelcore_percent __initdata; -static unsigned long required_movablecore __initdata; -static unsigned long required_movablecore_percent __initdata; -static unsigned long zone_movable_pfn[MAX_NUMNODES] __initdata; bool mirrored_kernelcore __initdata_memblock; /* movable_zone is the "real" zone pages in ZONE_MOVABLE are taken from */ @@ -427,86 +414,36 @@ EXPORT_SYMBOL(nr_online_nodes); int page_group_by_mobility_disabled __read_mostly; -bool deferred_struct_pages __meminitdata; - #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT /* * During boot we initialize deferred pages on-demand, as needed, but once * page_alloc_init_late() has finished, the deferred pages are all initialized, * and we can permanently disable that path. */ -static DEFINE_STATIC_KEY_TRUE(deferred_pages); +DEFINE_STATIC_KEY_TRUE(deferred_pages); static inline bool deferred_pages_enabled(void) { return static_branch_unlikely(&deferred_pages); } -/* Returns true if the struct page for the pfn is initialised */ -static inline bool __meminit early_page_initialised(unsigned long pfn) -{ - int nid = early_pfn_to_nid(pfn); - - if (node_online(nid) && pfn >= NODE_DATA(nid)->first_deferred_pfn) - return false; - - return true; -} - /* - * Returns true when the remaining initialisation should be deferred until - * later in the boot cycle when it can be parallelised. + * deferred_grow_zone() is __init, but it is called from + * get_page_from_freelist() during early boot until deferred_pages permanently + * disables this call. This is why we have refdata wrapper to avoid warning, + * and to ensure that the function body gets unloaded. */ -static bool __meminit -defer_init(int nid, unsigned long pfn, unsigned long end_pfn) +static bool __ref +_deferred_grow_zone(struct zone *zone, unsigned int order) { - static unsigned long prev_end_pfn, nr_initialised; - - if (early_page_ext_enabled()) - return false; - /* - * prev_end_pfn static that contains the end of previous zone - * No need to protect because called very early in boot before smp_init. - */ - if (prev_end_pfn != end_pfn) { - prev_end_pfn = end_pfn; - nr_initialised = 0; - } - - /* Always populate low zones for address-constrained allocations */ - if (end_pfn < pgdat_end_pfn(NODE_DATA(nid))) - return false; - - if (NODE_DATA(nid)->first_deferred_pfn != ULONG_MAX) - return true; - /* - * We start only with one section of pages, more pages are added as - * needed until the rest of deferred pages are initialized. - */ - nr_initialised++; - if ((nr_initialised > PAGES_PER_SECTION) && - (pfn & (PAGES_PER_SECTION - 1)) == 0) { - NODE_DATA(nid)->first_deferred_pfn = pfn; - return true; - } - return false; + return deferred_grow_zone(zone, order); } #else static inline bool deferred_pages_enabled(void) { return false; } - -static inline bool early_page_initialised(unsigned long pfn) -{ - return true; -} - -static inline bool defer_init(int nid, unsigned long pfn, unsigned long end_pfn) -{ - return false; -} -#endif +#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ /* Return a pointer to the bitmap storing bits affecting a block of pages */ static inline unsigned long *get_pageblock_bitmap(const struct page *page, @@ -772,26 +709,6 @@ void free_compound_page(struct page *page) free_the_page(page, compound_order(page)); } -static void prep_compound_head(struct page *page, unsigned int order) -{ - struct folio *folio = (struct folio *)page; - - set_compound_page_dtor(page, COMPOUND_PAGE_DTOR); - set_compound_order(page, order); - atomic_set(&folio->_entire_mapcount, -1); - atomic_set(&folio->_nr_pages_mapped, 0); - atomic_set(&folio->_pincount, 0); -} - -static void prep_compound_tail(struct page *head, int tail_idx) -{ - struct page *p = head + tail_idx; - - p->mapping = TAIL_MAPPING; - set_compound_head(p, head); - set_page_private(p, 0); -} - void prep_compound_page(struct page *page, unsigned int order) { int i; @@ -1601,80 +1518,6 @@ static void free_one_page(struct zone *zone, spin_unlock_irqrestore(&zone->lock, flags); } -static void __meminit __init_single_page(struct page *page, unsigned long pfn, - unsigned long zone, int nid) -{ - mm_zero_struct_page(page); - set_page_links(page, zone, nid, pfn); - init_page_count(page); - page_mapcount_reset(page); - page_cpupid_reset_last(page); - page_kasan_tag_reset(page); - - INIT_LIST_HEAD(&page->lru); -#ifdef WANT_PAGE_VIRTUAL - /* The shift won't overflow because ZONE_NORMAL is below 4G. */ - if (!is_highmem_idx(zone)) - set_page_address(page, __va(pfn << PAGE_SHIFT)); -#endif -} - -#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT -static void __meminit init_reserved_page(unsigned long pfn) -{ - pg_data_t *pgdat; - int nid, zid; - - if (early_page_initialised(pfn)) - return; - - nid = early_pfn_to_nid(pfn); - pgdat = NODE_DATA(nid); - - for (zid = 0; zid < MAX_NR_ZONES; zid++) { - struct zone *zone = &pgdat->node_zones[zid]; - - if (zone_spans_pfn(zone, pfn)) - break; - } - __init_single_page(pfn_to_page(pfn), pfn, zid, nid); -} -#else -static inline void init_reserved_page(unsigned long pfn) -{ -} -#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ - -/* - * Initialised pages do not have PageReserved set. This function is - * called for each range allocated by the bootmem allocator and - * marks the pages PageReserved. The remaining valid pages are later - * sent to the buddy page allocator. - */ -void __meminit reserve_bootmem_region(phys_addr_t start, phys_addr_t end) -{ - unsigned long start_pfn = PFN_DOWN(start); - unsigned long end_pfn = PFN_UP(end); - - for (; start_pfn < end_pfn; start_pfn++) { - if (pfn_valid(start_pfn)) { - struct page *page = pfn_to_page(start_pfn); - - init_reserved_page(start_pfn); - - /* Avoid false-positive PageTail() */ - INIT_LIST_HEAD(&page->lru); - - /* - * no need for atomic set_bit because the struct - * page is not visible yet so nobody should - * access it yet. - */ - __SetPageReserved(page); - } - } -} - static void __free_pages_ok(struct page *page, unsigned int order, fpi_t fpi_flags) { @@ -1733,70 +1576,6 @@ void __free_pages_core(struct page *page, unsigned int order) __free_pages_ok(page, order, FPI_TO_TAIL); } -#ifdef CONFIG_NUMA - -/* - * During memory init memblocks map pfns to nids. The search is expensive and - * this caches recent lookups. The implementation of __early_pfn_to_nid - * treats start/end as pfns. - */ -struct mminit_pfnnid_cache { - unsigned long last_start; - unsigned long last_end; - int last_nid; -}; - -static struct mminit_pfnnid_cache early_pfnnid_cache __meminitdata; - -/* - * Required by SPARSEMEM. Given a PFN, return what node the PFN is on. - */ -static int __meminit __early_pfn_to_nid(unsigned long pfn, - struct mminit_pfnnid_cache *state) -{ - unsigned long start_pfn, end_pfn; - int nid; - - if (state->last_start <= pfn && pfn < state->last_end) - return state->last_nid; - - nid = memblock_search_pfn_nid(pfn, &start_pfn, &end_pfn); - if (nid != NUMA_NO_NODE) { - state->last_start = start_pfn; - state->last_end = end_pfn; - state->last_nid = nid; - } - - return nid; -} - -int __meminit early_pfn_to_nid(unsigned long pfn) -{ - static DEFINE_SPINLOCK(early_pfn_lock); - int nid; - - spin_lock(&early_pfn_lock); - nid = __early_pfn_to_nid(pfn, &early_pfnnid_cache); - if (nid < 0) - nid = first_online_node; - spin_unlock(&early_pfn_lock); - - return nid; -} -#endif /* CONFIG_NUMA */ - -void __init memblock_free_pages(struct page *page, unsigned long pfn, - unsigned int order) -{ - if (!early_page_initialised(pfn)) - return; - if (!kmsan_memblock_free_pages(page, order)) { - /* KMSAN will take care of these pages. */ - return; - } - __free_pages_core(page, order); -} - /* * Check that the whole (or subset of) a pageblock given by the interval of * [start_pfn, end_pfn) is valid and within the same zone, before scanning it @@ -1867,549 +1646,131 @@ void clear_zone_contiguous(struct zone *zone) zone->contiguous = false; } -#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT -static void __init deferred_free_range(unsigned long pfn, - unsigned long nr_pages) -{ - struct page *page; - unsigned long i; - - if (!nr_pages) - return; - - page = pfn_to_page(pfn); - - /* Free a large naturally-aligned chunk if possible */ - if (nr_pages == pageblock_nr_pages && pageblock_aligned(pfn)) { - set_pageblock_migratetype(page, MIGRATE_MOVABLE); - __free_pages_core(page, pageblock_order); - return; - } - - for (i = 0; i < nr_pages; i++, page++, pfn++) { - if (pageblock_aligned(pfn)) - set_pageblock_migratetype(page, MIGRATE_MOVABLE); - __free_pages_core(page, 0); - } -} - -/* Completion tracking for deferred_init_memmap() threads */ -static atomic_t pgdat_init_n_undone __initdata; -static __initdata DECLARE_COMPLETION(pgdat_init_all_done_comp); - -static inline void __init pgdat_init_report_one_done(void) -{ - if (atomic_dec_and_test(&pgdat_init_n_undone)) - complete(&pgdat_init_all_done_comp); -} - /* - * Returns true if page needs to be initialized or freed to buddy allocator. + * The order of subdivision here is critical for the IO subsystem. + * Please do not alter this order without good reasons and regression + * testing. Specifically, as large blocks of memory are subdivided, + * the order in which smaller blocks are delivered depends on the order + * they're subdivided in this function. This is the primary factor + * influencing the order in which pages are delivered to the IO + * subsystem according to empirical testing, and this is also justified + * by considering the behavior of a buddy system containing a single + * large block of memory acted on by a series of small allocations. + * This behavior is a critical factor in sglist merging's success. * - * We check if a current large page is valid by only checking the validity - * of the head pfn. + * -- nyc */ -static inline bool __init deferred_pfn_valid(unsigned long pfn) +static inline void expand(struct zone *zone, struct page *page, + int low, int high, int migratetype) { - if (pageblock_aligned(pfn) && !pfn_valid(pfn)) - return false; - return true; -} - -/* - * Free pages to buddy allocator. Try to free aligned pages in - * pageblock_nr_pages sizes. - */ -static void __init deferred_free_pages(unsigned long pfn, - unsigned long end_pfn) -{ - unsigned long nr_free = 0; - - for (; pfn < end_pfn; pfn++) { - if (!deferred_pfn_valid(pfn)) { - deferred_free_range(pfn - nr_free, nr_free); - nr_free = 0; - } else if (pageblock_aligned(pfn)) { - deferred_free_range(pfn - nr_free, nr_free); - nr_free = 1; - } else { - nr_free++; - } - } - /* Free the last block of pages to allocator */ - deferred_free_range(pfn - nr_free, nr_free); -} + unsigned long size = 1 << high; -/* - * Initialize struct pages. We minimize pfn page lookups and scheduler checks - * by performing it only once every pageblock_nr_pages. - * Return number of pages initialized. - */ -static unsigned long __init deferred_init_pages(struct zone *zone, - unsigned long pfn, - unsigned long end_pfn) -{ - int nid = zone_to_nid(zone); - unsigned long nr_pages = 0; - int zid = zone_idx(zone); - struct page *page = NULL; + while (high > low) { + high--; + size >>= 1; + VM_BUG_ON_PAGE(bad_range(zone, &page[size]), &page[size]); - for (; pfn < end_pfn; pfn++) { - if (!deferred_pfn_valid(pfn)) { - page = NULL; + /* + * Mark as guard pages (or page), that will allow to + * merge back to allocator when buddy will be freed. + * Corresponding page table entries will not be touched, + * pages will stay not present in virtual address space + */ + if (set_page_guard(zone, &page[size], high, migratetype)) continue; - } else if (!page || pageblock_aligned(pfn)) { - page = pfn_to_page(pfn); - } else { - page++; - } - __init_single_page(page, pfn, zid, nid); - nr_pages++; + + add_to_free_list(&page[size], zone, high, migratetype); + set_buddy_order(&page[size], high); } - return (nr_pages); } -/* - * This function is meant to pre-load the iterator for the zone init. - * Specifically it walks through the ranges until we are caught up to the - * first_init_pfn value and exits there. If we never encounter the value we - * return false indicating there are no valid ranges left. - */ -static bool __init -deferred_init_mem_pfn_range_in_zone(u64 *i, struct zone *zone, - unsigned long *spfn, unsigned long *epfn, - unsigned long first_init_pfn) +static void check_new_page_bad(struct page *page) { - u64 j; - - /* - * Start out by walking through the ranges in this zone that have - * already been initialized. We don't need to do anything with them - * so we just need to flush them out of the system. - */ - for_each_free_mem_pfn_range_in_zone(j, zone, spfn, epfn) { - if (*epfn <= first_init_pfn) - continue; - if (*spfn < first_init_pfn) - *spfn = first_init_pfn; - *i = j; - return true; + if (unlikely(page->flags & __PG_HWPOISON)) { + /* Don't complain about hwpoisoned pages */ + page_mapcount_reset(page); /* remove PageBuddy */ + return; } - return false; + bad_page(page, + page_bad_reason(page, PAGE_FLAGS_CHECK_AT_PREP)); } /* - * Initialize and free pages. We do it in two loops: first we initialize - * struct page, then free to buddy allocator, because while we are - * freeing pages we can access pages that are ahead (computing buddy - * page in __free_one_page()). - * - * In order to try and keep some memory in the cache we have the loop - * broken along max page order boundaries. This way we will not cause - * any issues with the buddy page computation. + * This page is about to be returned from the page allocator */ -static unsigned long __init -deferred_init_maxorder(u64 *i, struct zone *zone, unsigned long *start_pfn, - unsigned long *end_pfn) +static int check_new_page(struct page *page) { - unsigned long mo_pfn = ALIGN(*start_pfn + 1, MAX_ORDER_NR_PAGES); - unsigned long spfn = *start_pfn, epfn = *end_pfn; - unsigned long nr_pages = 0; - u64 j = *i; - - /* First we loop through and initialize the page values */ - for_each_free_mem_pfn_range_in_zone_from(j, zone, start_pfn, end_pfn) { - unsigned long t; + if (likely(page_expected_state(page, + PAGE_FLAGS_CHECK_AT_PREP|__PG_HWPOISON))) + return 0; - if (mo_pfn <= *start_pfn) - break; + check_new_page_bad(page); + return 1; +} - t = min(mo_pfn, *end_pfn); - nr_pages += deferred_init_pages(zone, *start_pfn, t); +static inline bool check_new_pages(struct page *page, unsigned int order) +{ + if (is_check_pages_enabled()) { + for (int i = 0; i < (1 << order); i++) { + struct page *p = page + i; - if (mo_pfn < *end_pfn) { - *start_pfn = mo_pfn; - break; + if (unlikely(check_new_page(p))) + return true; } } - /* Reset values and now loop through freeing pages as needed */ - swap(j, *i); - - for_each_free_mem_pfn_range_in_zone_from(j, zone, &spfn, &epfn) { - unsigned long t; - - if (mo_pfn <= spfn) - break; - - t = min(mo_pfn, epfn); - deferred_free_pages(spfn, t); - - if (mo_pfn <= epfn) - break; - } - - return nr_pages; + return false; } -static void __init -deferred_init_memmap_chunk(unsigned long start_pfn, unsigned long end_pfn, - void *arg) +static inline bool should_skip_kasan_unpoison(gfp_t flags) { - unsigned long spfn, epfn; - struct zone *zone = arg; - u64 i; + /* Don't skip if a software KASAN mode is enabled. */ + if (IS_ENABLED(CONFIG_KASAN_GENERIC) || + IS_ENABLED(CONFIG_KASAN_SW_TAGS)) + return false; - deferred_init_mem_pfn_range_in_zone(&i, zone, &spfn, &epfn, start_pfn); + /* Skip, if hardware tag-based KASAN is not enabled. */ + if (!kasan_hw_tags_enabled()) + return true; /* - * Initialize and free pages in MAX_ORDER sized increments so that we - * can avoid introducing any issues with the buddy allocator. + * With hardware tag-based KASAN enabled, skip if this has been + * requested via __GFP_SKIP_KASAN. */ - while (spfn < end_pfn) { - deferred_init_maxorder(&i, zone, &spfn, &epfn); - cond_resched(); - } + return flags & __GFP_SKIP_KASAN; } -/* An arch may override for more concurrency. */ -__weak int __init -deferred_page_init_max_threads(const struct cpumask *node_cpumask) +static inline bool should_skip_init(gfp_t flags) { - return 1; + /* Don't skip, if hardware tag-based KASAN is not enabled. */ + if (!kasan_hw_tags_enabled()) + return false; + + /* For hardware tag-based KASAN, skip if requested. */ + return (flags & __GFP_SKIP_ZERO); } -/* Initialise remaining memory on a node */ -static int __init deferred_init_memmap(void *data) +inline void post_alloc_hook(struct page *page, unsigned int order, + gfp_t gfp_flags) { - pg_data_t *pgdat = data; - const struct cpumask *cpumask = cpumask_of_node(pgdat->node_id); - unsigned long spfn = 0, epfn = 0; - unsigned long first_init_pfn, flags; - unsigned long start = jiffies; - struct zone *zone; - int zid, max_threads; - u64 i; - - /* Bind memory initialisation thread to a local node if possible */ - if (!cpumask_empty(cpumask)) - set_cpus_allowed_ptr(current, cpumask); - - pgdat_resize_lock(pgdat, &flags); - first_init_pfn = pgdat->first_deferred_pfn; - if (first_init_pfn == ULONG_MAX) { - pgdat_resize_unlock(pgdat, &flags); - pgdat_init_report_one_done(); - return 0; - } + bool init = !want_init_on_free() && want_init_on_alloc(gfp_flags) && + !should_skip_init(gfp_flags); + bool zero_tags = init && (gfp_flags & __GFP_ZEROTAGS); + int i; + + set_page_private(page, 0); + set_page_refcounted(page); - /* Sanity check boundaries */ - BUG_ON(pgdat->first_deferred_pfn < pgdat->node_start_pfn); - BUG_ON(pgdat->first_deferred_pfn > pgdat_end_pfn(pgdat)); - pgdat->first_deferred_pfn = ULONG_MAX; + arch_alloc_page(page, order); + debug_pagealloc_map_pages(page, 1 << order); /* - * Once we unlock here, the zone cannot be grown anymore, thus if an - * interrupt thread must allocate this early in boot, zone must be - * pre-grown prior to start of deferred page initialization. + * Page unpoisoning must happen before memory initialization. + * Otherwise, the poison pattern will be overwritten for __GFP_ZERO + * allocations and the page unpoisoning code will complain. */ - pgdat_resize_unlock(pgdat, &flags); - - /* Only the highest zone is deferred so find it */ - for (zid = 0; zid < MAX_NR_ZONES; zid++) { - zone = pgdat->node_zones + zid; - if (first_init_pfn < zone_end_pfn(zone)) - break; - } - - /* If the zone is empty somebody else may have cleared out the zone */ - if (!deferred_init_mem_pfn_range_in_zone(&i, zone, &spfn, &epfn, - first_init_pfn)) - goto zone_empty; - - max_threads = deferred_page_init_max_threads(cpumask); - - while (spfn < epfn) { - unsigned long epfn_align = ALIGN(epfn, PAGES_PER_SECTION); - struct padata_mt_job job = { - .thread_fn = deferred_init_memmap_chunk, - .fn_arg = zone, - .start = spfn, - .size = epfn_align - spfn, - .align = PAGES_PER_SECTION, - .min_chunk = PAGES_PER_SECTION, - .max_threads = max_threads, - }; - - padata_do_multithreaded(&job); - deferred_init_mem_pfn_range_in_zone(&i, zone, &spfn, &epfn, - epfn_align); - } -zone_empty: - /* Sanity check that the next zone really is unpopulated */ - WARN_ON(++zid < MAX_NR_ZONES && populated_zone(++zone)); - - pr_info("node %d deferred pages initialised in %ums\n", - pgdat->node_id, jiffies_to_msecs(jiffies - start)); - - pgdat_init_report_one_done(); - return 0; -} - -/* - * If this zone has deferred pages, try to grow it by initializing enough - * deferred pages to satisfy the allocation specified by order, rounded up to - * the nearest PAGES_PER_SECTION boundary. So we're adding memory in increments - * of SECTION_SIZE bytes by initializing struct pages in increments of - * PAGES_PER_SECTION * sizeof(struct page) bytes. - * - * Return true when zone was grown, otherwise return false. We return true even - * when we grow less than requested, to let the caller decide if there are - * enough pages to satisfy the allocation. - * - * Note: We use noinline because this function is needed only during boot, and - * it is called from a __ref function _deferred_grow_zone. This way we are - * making sure that it is not inlined into permanent text section. - */ -static noinline bool __init -deferred_grow_zone(struct zone *zone, unsigned int order) -{ - unsigned long nr_pages_needed = ALIGN(1 << order, PAGES_PER_SECTION); - pg_data_t *pgdat = zone->zone_pgdat; - unsigned long first_deferred_pfn = pgdat->first_deferred_pfn; - unsigned long spfn, epfn, flags; - unsigned long nr_pages = 0; - u64 i; - - /* Only the last zone may have deferred pages */ - if (zone_end_pfn(zone) != pgdat_end_pfn(pgdat)) - return false; - - pgdat_resize_lock(pgdat, &flags); - - /* - * If someone grew this zone while we were waiting for spinlock, return - * true, as there might be enough pages already. - */ - if (first_deferred_pfn != pgdat->first_deferred_pfn) { - pgdat_resize_unlock(pgdat, &flags); - return true; - } - - /* If the zone is empty somebody else may have cleared out the zone */ - if (!deferred_init_mem_pfn_range_in_zone(&i, zone, &spfn, &epfn, - first_deferred_pfn)) { - pgdat->first_deferred_pfn = ULONG_MAX; - pgdat_resize_unlock(pgdat, &flags); - /* Retry only once. */ - return first_deferred_pfn != ULONG_MAX; - } - - /* - * Initialize and free pages in MAX_ORDER sized increments so - * that we can avoid introducing any issues with the buddy - * allocator. - */ - while (spfn < epfn) { - /* update our first deferred PFN for this section */ - first_deferred_pfn = spfn; - - nr_pages += deferred_init_maxorder(&i, zone, &spfn, &epfn); - touch_nmi_watchdog(); - - /* We should only stop along section boundaries */ - if ((first_deferred_pfn ^ spfn) < PAGES_PER_SECTION) - continue; - - /* If our quota has been met we can stop here */ - if (nr_pages >= nr_pages_needed) - break; - } - - pgdat->first_deferred_pfn = spfn; - pgdat_resize_unlock(pgdat, &flags); - - return nr_pages > 0; -} - -/* - * deferred_grow_zone() is __init, but it is called from - * get_page_from_freelist() during early boot until deferred_pages permanently - * disables this call. This is why we have refdata wrapper to avoid warning, - * and to ensure that the function body gets unloaded. - */ -static bool __ref -_deferred_grow_zone(struct zone *zone, unsigned int order) -{ - return deferred_grow_zone(zone, order); -} - -#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ - -void __init page_alloc_init_late(void) -{ - struct zone *zone; - int nid; - -#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT - - /* There will be num_node_state(N_MEMORY) threads */ - atomic_set(&pgdat_init_n_undone, num_node_state(N_MEMORY)); - for_each_node_state(nid, N_MEMORY) { - kthread_run(deferred_init_memmap, NODE_DATA(nid), "pgdatinit%d", nid); - } - - /* Block until all are initialised */ - wait_for_completion(&pgdat_init_all_done_comp); - - /* - * We initialized the rest of the deferred pages. Permanently disable - * on-demand struct page initialization. - */ - static_branch_disable(&deferred_pages); - - /* Reinit limits that are based on free pages after the kernel is up */ - files_maxfiles_init(); -#endif - - buffer_init(); - - /* Discard memblock private memory */ - memblock_discard(); - - for_each_node_state(nid, N_MEMORY) - shuffle_free_memory(NODE_DATA(nid)); - - for_each_populated_zone(zone) - set_zone_contiguous(zone); -} - -/* - * The order of subdivision here is critical for the IO subsystem. - * Please do not alter this order without good reasons and regression - * testing. Specifically, as large blocks of memory are subdivided, - * the order in which smaller blocks are delivered depends on the order - * they're subdivided in this function. This is the primary factor - * influencing the order in which pages are delivered to the IO - * subsystem according to empirical testing, and this is also justified - * by considering the behavior of a buddy system containing a single - * large block of memory acted on by a series of small allocations. - * This behavior is a critical factor in sglist merging's success. - * - * -- nyc - */ -static inline void expand(struct zone *zone, struct page *page, - int low, int high, int migratetype) -{ - unsigned long size = 1 << high; - - while (high > low) { - high--; - size >>= 1; - VM_BUG_ON_PAGE(bad_range(zone, &page[size]), &page[size]); - - /* - * Mark as guard pages (or page), that will allow to - * merge back to allocator when buddy will be freed. - * Corresponding page table entries will not be touched, - * pages will stay not present in virtual address space - */ - if (set_page_guard(zone, &page[size], high, migratetype)) - continue; - - add_to_free_list(&page[size], zone, high, migratetype); - set_buddy_order(&page[size], high); - } -} - -static void check_new_page_bad(struct page *page) -{ - if (unlikely(page->flags & __PG_HWPOISON)) { - /* Don't complain about hwpoisoned pages */ - page_mapcount_reset(page); /* remove PageBuddy */ - return; - } - - bad_page(page, - page_bad_reason(page, PAGE_FLAGS_CHECK_AT_PREP)); -} - -/* - * This page is about to be returned from the page allocator - */ -static int check_new_page(struct page *page) -{ - if (likely(page_expected_state(page, - PAGE_FLAGS_CHECK_AT_PREP|__PG_HWPOISON))) - return 0; - - check_new_page_bad(page); - return 1; -} - -static inline bool check_new_pages(struct page *page, unsigned int order) -{ - if (is_check_pages_enabled()) { - for (int i = 0; i < (1 << order); i++) { - struct page *p = page + i; - - if (unlikely(check_new_page(p))) - return true; - } - } - - return false; -} - -static inline bool should_skip_kasan_unpoison(gfp_t flags) -{ - /* Don't skip if a software KASAN mode is enabled. */ - if (IS_ENABLED(CONFIG_KASAN_GENERIC) || - IS_ENABLED(CONFIG_KASAN_SW_TAGS)) - return false; - - /* Skip, if hardware tag-based KASAN is not enabled. */ - if (!kasan_hw_tags_enabled()) - return true; - - /* - * With hardware tag-based KASAN enabled, skip if this has been - * requested via __GFP_SKIP_KASAN. - */ - return flags & __GFP_SKIP_KASAN; -} - -static inline bool should_skip_init(gfp_t flags) -{ - /* Don't skip, if hardware tag-based KASAN is not enabled. */ - if (!kasan_hw_tags_enabled()) - return false; - - /* For hardware tag-based KASAN, skip if requested. */ - return (flags & __GFP_SKIP_ZERO); -} - -inline void post_alloc_hook(struct page *page, unsigned int order, - gfp_t gfp_flags) -{ - bool init = !want_init_on_free() && want_init_on_alloc(gfp_flags) && - !should_skip_init(gfp_flags); - bool zero_tags = init && (gfp_flags & __GFP_ZEROTAGS); - int i; - - set_page_private(page, 0); - set_page_refcounted(page); - - arch_alloc_page(page, order); - debug_pagealloc_map_pages(page, 1 << order); - - /* - * Page unpoisoning must happen before memory initialization. - * Otherwise, the poison pattern will be overwritten for __GFP_ZERO - * allocations and the page unpoisoning code will complain. - */ - kernel_unpoison_pages(page, 1 << order); + kernel_unpoison_pages(page, 1 << order); /* * As memory initialization might be integrated into KASAN, @@ -6519,7 +5880,6 @@ static void per_cpu_pages_init(struct per_cpu_pages *pcp, struct per_cpu_zonesta #define BOOT_PAGESET_BATCH 1 static DEFINE_PER_CPU(struct per_cpu_pages, boot_pageset); static DEFINE_PER_CPU(struct per_cpu_zonestat, boot_zonestats); -static DEFINE_PER_CPU(struct per_cpu_nodestat, boot_nodestats); static void __build_all_zonelists(void *data) { @@ -6633,393 +5993,33 @@ void __ref build_all_zonelists(pg_data_t *pgdat) #endif } -/* If zone is ZONE_MOVABLE but memory is mirrored, it is an overlapped init */ -static bool __meminit -overlap_memmap_init(unsigned long zone, unsigned long *pfn) -{ - static struct memblock_region *r; - - if (mirrored_kernelcore && zone == ZONE_MOVABLE) { - if (!r || *pfn >= memblock_region_memory_end_pfn(r)) { - for_each_mem_region(r) { - if (*pfn < memblock_region_memory_end_pfn(r)) - break; - } - } - if (*pfn >= memblock_region_memory_base_pfn(r) && - memblock_is_mirror(r)) { - *pfn = memblock_region_memory_end_pfn(r); - return true; - } - } - return false; -} - -/* - * Initially all pages are reserved - free ones are freed - * up by memblock_free_all() once the early boot process is - * done. Non-atomic initialization, single-pass. - * - * All aligned pageblocks are initialized to the specified migratetype - * (usually MIGRATE_MOVABLE). Besides setting the migratetype, no related - * zone stats (e.g., nr_isolate_pageblock) are touched. - */ -void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone, - unsigned long start_pfn, unsigned long zone_end_pfn, - enum meminit_context context, - struct vmem_altmap *altmap, int migratetype) +static int zone_batchsize(struct zone *zone) { - unsigned long pfn, end_pfn = start_pfn + size; - struct page *page; - - if (highest_memmap_pfn < end_pfn - 1) - highest_memmap_pfn = end_pfn - 1; +#ifdef CONFIG_MMU + int batch; -#ifdef CONFIG_ZONE_DEVICE /* - * Honor reservation requested by the driver for this ZONE_DEVICE - * memory. We limit the total number of pages to initialize to just - * those that might contain the memory mapping. We will defer the - * ZONE_DEVICE page initialization until after we have released - * the hotplug lock. + * The number of pages to batch allocate is either ~0.1% + * of the zone or 1MB, whichever is smaller. The batch + * size is striking a balance between allocation latency + * and zone lock contention. */ - if (zone == ZONE_DEVICE) { - if (!altmap) - return; - - if (start_pfn == altmap->base_pfn) - start_pfn += altmap->reserve; - end_pfn = altmap->base_pfn + vmem_altmap_offset(altmap); - } -#endif - - for (pfn = start_pfn; pfn < end_pfn; ) { - /* - * There can be holes in boot-time mem_map[]s handed to this - * function. They do not exist on hotplugged memory. - */ - if (context == MEMINIT_EARLY) { - if (overlap_memmap_init(zone, &pfn)) - continue; - if (defer_init(nid, pfn, zone_end_pfn)) { - deferred_struct_pages = true; - break; - } - } - - page = pfn_to_page(pfn); - __init_single_page(page, pfn, zone, nid); - if (context == MEMINIT_HOTPLUG) - __SetPageReserved(page); - - /* - * Usually, we want to mark the pageblock MIGRATE_MOVABLE, - * such that unmovable allocations won't be scattered all - * over the place during system boot. - */ - if (pageblock_aligned(pfn)) { - set_pageblock_migratetype(page, migratetype); - cond_resched(); - } - pfn++; - } -} - -#ifdef CONFIG_ZONE_DEVICE -static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, - unsigned long zone_idx, int nid, - struct dev_pagemap *pgmap) -{ - - __init_single_page(page, pfn, zone_idx, nid); + batch = min(zone_managed_pages(zone) >> 10, SZ_1M / PAGE_SIZE); + batch /= 4; /* We effectively *= 4 below */ + if (batch < 1) + batch = 1; /* - * Mark page reserved as it will need to wait for onlining - * phase for it to be fully associated with a zone. + * Clamp the batch to a 2^n - 1 value. Having a power + * of 2 value was found to be more likely to have + * suboptimal cache aliasing properties in some cases. * - * We can use the non-atomic __set_bit operation for setting - * the flag as we are still initializing the pages. + * For example if 2 tasks are alternately allocating + * batches of pages, one task can end up with a lot + * of pages of one half of the possible page colors + * and the other with pages of the other colors. */ - __SetPageReserved(page); - - /* - * ZONE_DEVICE pages union ->lru with a ->pgmap back pointer - * and zone_device_data. It is a bug if a ZONE_DEVICE page is - * ever freed or placed on a driver-private list. - */ - page->pgmap = pgmap; - page->zone_device_data = NULL; - - /* - * Mark the block movable so that blocks are reserved for - * movable at startup. This will force kernel allocations - * to reserve their blocks rather than leaking throughout - * the address space during boot when many long-lived - * kernel allocations are made. - * - * Please note that MEMINIT_HOTPLUG path doesn't clear memmap - * because this is done early in section_activate() - */ - if (pageblock_aligned(pfn)) { - set_pageblock_migratetype(page, MIGRATE_MOVABLE); - cond_resched(); - } - - /* - * ZONE_DEVICE pages are released directly to the driver page allocator - * which will set the page count to 1 when allocating the page. - */ - if (pgmap->type == MEMORY_DEVICE_PRIVATE || - pgmap->type == MEMORY_DEVICE_COHERENT) - set_page_count(page, 0); -} - -/* - * With compound page geometry and when struct pages are stored in ram most - * tail pages are reused. Consequently, the amount of unique struct pages to - * initialize is a lot smaller that the total amount of struct pages being - * mapped. This is a paired / mild layering violation with explicit knowledge - * of how the sparse_vmemmap internals handle compound pages in the lack - * of an altmap. See vmemmap_populate_compound_pages(). - */ -static inline unsigned long compound_nr_pages(struct vmem_altmap *altmap, - unsigned long nr_pages) -{ - return is_power_of_2(sizeof(struct page)) && - !altmap ? 2 * (PAGE_SIZE / sizeof(struct page)) : nr_pages; -} - -static void __ref memmap_init_compound(struct page *head, - unsigned long head_pfn, - unsigned long zone_idx, int nid, - struct dev_pagemap *pgmap, - unsigned long nr_pages) -{ - unsigned long pfn, end_pfn = head_pfn + nr_pages; - unsigned int order = pgmap->vmemmap_shift; - - __SetPageHead(head); - for (pfn = head_pfn + 1; pfn < end_pfn; pfn++) { - struct page *page = pfn_to_page(pfn); - - __init_zone_device_page(page, pfn, zone_idx, nid, pgmap); - prep_compound_tail(head, pfn - head_pfn); - set_page_count(page, 0); - - /* - * The first tail page stores important compound page info. - * Call prep_compound_head() after the first tail page has - * been initialized, to not have the data overwritten. - */ - if (pfn == head_pfn + 1) - prep_compound_head(head, order); - } -} - -void __ref memmap_init_zone_device(struct zone *zone, - unsigned long start_pfn, - unsigned long nr_pages, - struct dev_pagemap *pgmap) -{ - unsigned long pfn, end_pfn = start_pfn + nr_pages; - struct pglist_data *pgdat = zone->zone_pgdat; - struct vmem_altmap *altmap = pgmap_altmap(pgmap); - unsigned int pfns_per_compound = pgmap_vmemmap_nr(pgmap); - unsigned long zone_idx = zone_idx(zone); - unsigned long start = jiffies; - int nid = pgdat->node_id; - - if (WARN_ON_ONCE(!pgmap || zone_idx != ZONE_DEVICE)) - return; - - /* - * The call to memmap_init should have already taken care - * of the pages reserved for the memmap, so we can just jump to - * the end of that region and start processing the device pages. - */ - if (altmap) { - start_pfn = altmap->base_pfn + vmem_altmap_offset(altmap); - nr_pages = end_pfn - start_pfn; - } - - for (pfn = start_pfn; pfn < end_pfn; pfn += pfns_per_compound) { - struct page *page = pfn_to_page(pfn); - - __init_zone_device_page(page, pfn, zone_idx, nid, pgmap); - - if (pfns_per_compound == 1) - continue; - - memmap_init_compound(page, pfn, zone_idx, nid, pgmap, - compound_nr_pages(altmap, pfns_per_compound)); - } - - pr_info("%s initialised %lu pages in %ums\n", __func__, - nr_pages, jiffies_to_msecs(jiffies - start)); -} - -#endif -static void __meminit zone_init_free_lists(struct zone *zone) -{ - unsigned int order, t; - for_each_migratetype_order(order, t) { - INIT_LIST_HEAD(&zone->free_area[order].free_list[t]); - zone->free_area[order].nr_free = 0; - } -} - -/* - * Only struct pages that correspond to ranges defined by memblock.memory - * are zeroed and initialized by going through __init_single_page() during - * memmap_init_zone_range(). - * - * But, there could be struct pages that correspond to holes in - * memblock.memory. This can happen because of the following reasons: - * - physical memory bank size is not necessarily the exact multiple of the - * arbitrary section size - * - early reserved memory may not be listed in memblock.memory - * - memory layouts defined with memmap= kernel parameter may not align - * nicely with memmap sections - * - * Explicitly initialize those struct pages so that: - * - PG_Reserved is set - * - zone and node links point to zone and node that span the page if the - * hole is in the middle of a zone - * - zone and node links point to adjacent zone/node if the hole falls on - * the zone boundary; the pages in such holes will be prepended to the - * zone/node above the hole except for the trailing pages in the last - * section that will be appended to the zone/node below. - */ -static void __init init_unavailable_range(unsigned long spfn, - unsigned long epfn, - int zone, int node) -{ - unsigned long pfn; - u64 pgcnt = 0; - - for (pfn = spfn; pfn < epfn; pfn++) { - if (!pfn_valid(pageblock_start_pfn(pfn))) { - pfn = pageblock_end_pfn(pfn) - 1; - continue; - } - __init_single_page(pfn_to_page(pfn), pfn, zone, node); - __SetPageReserved(pfn_to_page(pfn)); - pgcnt++; - } - - if (pgcnt) - pr_info("On node %d, zone %s: %lld pages in unavailable ranges", - node, zone_names[zone], pgcnt); -} - -static void __init memmap_init_zone_range(struct zone *zone, - unsigned long start_pfn, - unsigned long end_pfn, - unsigned long *hole_pfn) -{ - unsigned long zone_start_pfn = zone->zone_start_pfn; - unsigned long zone_end_pfn = zone_start_pfn + zone->spanned_pages; - int nid = zone_to_nid(zone), zone_id = zone_idx(zone); - - start_pfn = clamp(start_pfn, zone_start_pfn, zone_end_pfn); - end_pfn = clamp(end_pfn, zone_start_pfn, zone_end_pfn); - - if (start_pfn >= end_pfn) - return; - - memmap_init_range(end_pfn - start_pfn, nid, zone_id, start_pfn, - zone_end_pfn, MEMINIT_EARLY, NULL, MIGRATE_MOVABLE); - - if (*hole_pfn < start_pfn) - init_unavailable_range(*hole_pfn, start_pfn, zone_id, nid); - - *hole_pfn = end_pfn; -} - -static void __init memmap_init(void) -{ - unsigned long start_pfn, end_pfn; - unsigned long hole_pfn = 0; - int i, j, zone_id = 0, nid; - - for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, &nid) { - struct pglist_data *node = NODE_DATA(nid); - - for (j = 0; j < MAX_NR_ZONES; j++) { - struct zone *zone = node->node_zones + j; - - if (!populated_zone(zone)) - continue; - - memmap_init_zone_range(zone, start_pfn, end_pfn, - &hole_pfn); - zone_id = j; - } - } - -#ifdef CONFIG_SPARSEMEM - /* - * Initialize the memory map for hole in the range [memory_end, - * section_end]. - * Append the pages in this hole to the highest zone in the last - * node. - * The call to init_unavailable_range() is outside the ifdef to - * silence the compiler warining about zone_id set but not used; - * for FLATMEM it is a nop anyway - */ - end_pfn = round_up(end_pfn, PAGES_PER_SECTION); - if (hole_pfn < end_pfn) -#endif - init_unavailable_range(hole_pfn, end_pfn, zone_id, nid); -} - -void __init *memmap_alloc(phys_addr_t size, phys_addr_t align, - phys_addr_t min_addr, int nid, bool exact_nid) -{ - void *ptr; - - if (exact_nid) - ptr = memblock_alloc_exact_nid_raw(size, align, min_addr, - MEMBLOCK_ALLOC_ACCESSIBLE, - nid); - else - ptr = memblock_alloc_try_nid_raw(size, align, min_addr, - MEMBLOCK_ALLOC_ACCESSIBLE, - nid); - - if (ptr && size > 0) - page_init_poison(ptr, size); - - return ptr; -} - -static int zone_batchsize(struct zone *zone) -{ -#ifdef CONFIG_MMU - int batch; - - /* - * The number of pages to batch allocate is either ~0.1% - * of the zone or 1MB, whichever is smaller. The batch - * size is striking a balance between allocation latency - * and zone lock contention. - */ - batch = min(zone_managed_pages(zone) >> 10, SZ_1M / PAGE_SIZE); - batch /= 4; /* We effectively *= 4 below */ - if (batch < 1) - batch = 1; - - /* - * Clamp the batch to a 2^n - 1 value. Having a power - * of 2 value was found to be more likely to have - * suboptimal cache aliasing properties in some cases. - * - * For example if 2 tasks are alternately allocating - * batches of pages, one task can end up with a lot - * of pages of one half of the possible page colors - * and the other with pages of the other colors. - */ - batch = rounddown_pow_of_two(batch + batch/2) - 1; + batch = rounddown_pow_of_two(batch + batch/2) - 1; return batch; @@ -7043,1352 +6043,210 @@ static int zone_batchsize(struct zone *zone) static int zone_highsize(struct zone *zone, int batch, int cpu_online) { -#ifdef CONFIG_MMU - int high; - int nr_split_cpus; - unsigned long total_pages; - - if (!percpu_pagelist_high_fraction) { - /* - * By default, the high value of the pcp is based on the zone - * low watermark so that if they are full then background - * reclaim will not be started prematurely. - */ - total_pages = low_wmark_pages(zone); - } else { - /* - * If percpu_pagelist_high_fraction is configured, the high - * value is based on a fraction of the managed pages in the - * zone. - */ - total_pages = zone_managed_pages(zone) / percpu_pagelist_high_fraction; - } - - /* - * Split the high value across all online CPUs local to the zone. Note - * that early in boot that CPUs may not be online yet and that during - * CPU hotplug that the cpumask is not yet updated when a CPU is being - * onlined. For memory nodes that have no CPUs, split pcp->high across - * all online CPUs to mitigate the risk that reclaim is triggered - * prematurely due to pages stored on pcp lists. - */ - nr_split_cpus = cpumask_weight(cpumask_of_node(zone_to_nid(zone))) + cpu_online; - if (!nr_split_cpus) - nr_split_cpus = num_online_cpus(); - high = total_pages / nr_split_cpus; - - /* - * Ensure high is at least batch*4. The multiple is based on the - * historical relationship between high and batch. - */ - high = max(high, batch << 2); - - return high; -#else - return 0; -#endif -} - -/* - * pcp->high and pcp->batch values are related and generally batch is lower - * than high. They are also related to pcp->count such that count is lower - * than high, and as soon as it reaches high, the pcplist is flushed. - * - * However, guaranteeing these relations at all times would require e.g. write - * barriers here but also careful usage of read barriers at the read side, and - * thus be prone to error and bad for performance. Thus the update only prevents - * store tearing. Any new users of pcp->batch and pcp->high should ensure they - * can cope with those fields changing asynchronously, and fully trust only the - * pcp->count field on the local CPU with interrupts disabled. - * - * mutex_is_locked(&pcp_batch_high_lock) required when calling this function - * outside of boot time (or some other assurance that no concurrent updaters - * exist). - */ -static void pageset_update(struct per_cpu_pages *pcp, unsigned long high, - unsigned long batch) -{ - WRITE_ONCE(pcp->batch, batch); - WRITE_ONCE(pcp->high, high); -} - -static void per_cpu_pages_init(struct per_cpu_pages *pcp, struct per_cpu_zonestat *pzstats) -{ - int pindex; - - memset(pcp, 0, sizeof(*pcp)); - memset(pzstats, 0, sizeof(*pzstats)); - - spin_lock_init(&pcp->lock); - for (pindex = 0; pindex < NR_PCP_LISTS; pindex++) - INIT_LIST_HEAD(&pcp->lists[pindex]); - - /* - * Set batch and high values safe for a boot pageset. A true percpu - * pageset's initialization will update them subsequently. Here we don't - * need to be as careful as pageset_update() as nobody can access the - * pageset yet. - */ - pcp->high = BOOT_PAGESET_HIGH; - pcp->batch = BOOT_PAGESET_BATCH; - pcp->free_factor = 0; -} - -static void __zone_set_pageset_high_and_batch(struct zone *zone, unsigned long high, - unsigned long batch) -{ - struct per_cpu_pages *pcp; - int cpu; - - for_each_possible_cpu(cpu) { - pcp = per_cpu_ptr(zone->per_cpu_pageset, cpu); - pageset_update(pcp, high, batch); - } -} - -/* - * Calculate and set new high and batch values for all per-cpu pagesets of a - * zone based on the zone's size. - */ -static void zone_set_pageset_high_and_batch(struct zone *zone, int cpu_online) -{ - int new_high, new_batch; - - new_batch = max(1, zone_batchsize(zone)); - new_high = zone_highsize(zone, new_batch, cpu_online); - - if (zone->pageset_high == new_high && - zone->pageset_batch == new_batch) - return; - - zone->pageset_high = new_high; - zone->pageset_batch = new_batch; - - __zone_set_pageset_high_and_batch(zone, new_high, new_batch); -} - -void __meminit setup_zone_pageset(struct zone *zone) -{ - int cpu; - - /* Size may be 0 on !SMP && !NUMA */ - if (sizeof(struct per_cpu_zonestat) > 0) - zone->per_cpu_zonestats = alloc_percpu(struct per_cpu_zonestat); - - zone->per_cpu_pageset = alloc_percpu(struct per_cpu_pages); - for_each_possible_cpu(cpu) { - struct per_cpu_pages *pcp; - struct per_cpu_zonestat *pzstats; - - pcp = per_cpu_ptr(zone->per_cpu_pageset, cpu); - pzstats = per_cpu_ptr(zone->per_cpu_zonestats, cpu); - per_cpu_pages_init(pcp, pzstats); - } - - zone_set_pageset_high_and_batch(zone, 0); -} - -/* - * The zone indicated has a new number of managed_pages; batch sizes and percpu - * page high values need to be recalculated. - */ -static void zone_pcp_update(struct zone *zone, int cpu_online) -{ - mutex_lock(&pcp_batch_high_lock); - zone_set_pageset_high_and_batch(zone, cpu_online); - mutex_unlock(&pcp_batch_high_lock); -} - -/* - * Allocate per cpu pagesets and initialize them. - * Before this call only boot pagesets were available. - */ -void __init setup_per_cpu_pageset(void) -{ - struct pglist_data *pgdat; - struct zone *zone; - int __maybe_unused cpu; - - for_each_populated_zone(zone) - setup_zone_pageset(zone); - -#ifdef CONFIG_NUMA - /* - * Unpopulated zones continue using the boot pagesets. - * The numa stats for these pagesets need to be reset. - * Otherwise, they will end up skewing the stats of - * the nodes these zones are associated with. - */ - for_each_possible_cpu(cpu) { - struct per_cpu_zonestat *pzstats = &per_cpu(boot_zonestats, cpu); - memset(pzstats->vm_numa_event, 0, - sizeof(pzstats->vm_numa_event)); - } -#endif - - for_each_online_pgdat(pgdat) - pgdat->per_cpu_nodestats = - alloc_percpu(struct per_cpu_nodestat); -} - -static __meminit void zone_pcp_init(struct zone *zone) -{ - /* - * per cpu subsystem is not up at this point. The following code - * relies on the ability of the linker to provide the - * offset of a (static) per cpu variable into the per cpu area. - */ - zone->per_cpu_pageset = &boot_pageset; - zone->per_cpu_zonestats = &boot_zonestats; - zone->pageset_high = BOOT_PAGESET_HIGH; - zone->pageset_batch = BOOT_PAGESET_BATCH; - - if (populated_zone(zone)) - pr_debug(" %s zone: %lu pages, LIFO batch:%u\n", zone->name, - zone->present_pages, zone_batchsize(zone)); -} - -void __meminit init_currently_empty_zone(struct zone *zone, - unsigned long zone_start_pfn, - unsigned long size) -{ - struct pglist_data *pgdat = zone->zone_pgdat; - int zone_idx = zone_idx(zone) + 1; - - if (zone_idx > pgdat->nr_zones) - pgdat->nr_zones = zone_idx; - - zone->zone_start_pfn = zone_start_pfn; - - mminit_dprintk(MMINIT_TRACE, "memmap_init", - "Initialising map node %d zone %lu pfns %lu -> %lu\n", - pgdat->node_id, - (unsigned long)zone_idx(zone), - zone_start_pfn, (zone_start_pfn + size)); - - zone_init_free_lists(zone); - zone->initialized = 1; -} - -/** - * get_pfn_range_for_nid - Return the start and end page frames for a node - * @nid: The nid to return the range for. If MAX_NUMNODES, the min and max PFN are returned. - * @start_pfn: Passed by reference. On return, it will have the node start_pfn. - * @end_pfn: Passed by reference. On return, it will have the node end_pfn. - * - * It returns the start and end page frame of a node based on information - * provided by memblock_set_node(). If called for a node - * with no available memory, a warning is printed and the start and end - * PFNs will be 0. - */ -void __init get_pfn_range_for_nid(unsigned int nid, - unsigned long *start_pfn, unsigned long *end_pfn) -{ - unsigned long this_start_pfn, this_end_pfn; - int i; - - *start_pfn = -1UL; - *end_pfn = 0; - - for_each_mem_pfn_range(i, nid, &this_start_pfn, &this_end_pfn, NULL) { - *start_pfn = min(*start_pfn, this_start_pfn); - *end_pfn = max(*end_pfn, this_end_pfn); - } - - if (*start_pfn == -1UL) - *start_pfn = 0; -} - -/* - * This finds a zone that can be used for ZONE_MOVABLE pages. The - * assumption is made that zones within a node are ordered in monotonic - * increasing memory addresses so that the "highest" populated zone is used - */ -static void __init find_usable_zone_for_movable(void) -{ - int zone_index; - for (zone_index = MAX_NR_ZONES - 1; zone_index >= 0; zone_index--) { - if (zone_index == ZONE_MOVABLE) - continue; - - if (arch_zone_highest_possible_pfn[zone_index] > - arch_zone_lowest_possible_pfn[zone_index]) - break; - } - - VM_BUG_ON(zone_index == -1); - movable_zone = zone_index; -} - -/* - * The zone ranges provided by the architecture do not include ZONE_MOVABLE - * because it is sized independent of architecture. Unlike the other zones, - * the starting point for ZONE_MOVABLE is not fixed. It may be different - * in each node depending on the size of each node and how evenly kernelcore - * is distributed. This helper function adjusts the zone ranges - * provided by the architecture for a given node by using the end of the - * highest usable zone for ZONE_MOVABLE. This preserves the assumption that - * zones within a node are in order of monotonic increases memory addresses - */ -static void __init adjust_zone_range_for_zone_movable(int nid, - unsigned long zone_type, - unsigned long node_start_pfn, - unsigned long node_end_pfn, - unsigned long *zone_start_pfn, - unsigned long *zone_end_pfn) -{ - /* Only adjust if ZONE_MOVABLE is on this node */ - if (zone_movable_pfn[nid]) { - /* Size ZONE_MOVABLE */ - if (zone_type == ZONE_MOVABLE) { - *zone_start_pfn = zone_movable_pfn[nid]; - *zone_end_pfn = min(node_end_pfn, - arch_zone_highest_possible_pfn[movable_zone]); - - /* Adjust for ZONE_MOVABLE starting within this range */ - } else if (!mirrored_kernelcore && - *zone_start_pfn < zone_movable_pfn[nid] && - *zone_end_pfn > zone_movable_pfn[nid]) { - *zone_end_pfn = zone_movable_pfn[nid]; - - /* Check if this whole range is within ZONE_MOVABLE */ - } else if (*zone_start_pfn >= zone_movable_pfn[nid]) - *zone_start_pfn = *zone_end_pfn; - } -} - -/* - * Return the number of pages a zone spans in a node, including holes - * present_pages = zone_spanned_pages_in_node() - zone_absent_pages_in_node() - */ -static unsigned long __init zone_spanned_pages_in_node(int nid, - unsigned long zone_type, - unsigned long node_start_pfn, - unsigned long node_end_pfn, - unsigned long *zone_start_pfn, - unsigned long *zone_end_pfn) -{ - unsigned long zone_low = arch_zone_lowest_possible_pfn[zone_type]; - unsigned long zone_high = arch_zone_highest_possible_pfn[zone_type]; - /* When hotadd a new node from cpu_up(), the node should be empty */ - if (!node_start_pfn && !node_end_pfn) - return 0; - - /* Get the start and end of the zone */ - *zone_start_pfn = clamp(node_start_pfn, zone_low, zone_high); - *zone_end_pfn = clamp(node_end_pfn, zone_low, zone_high); - adjust_zone_range_for_zone_movable(nid, zone_type, - node_start_pfn, node_end_pfn, - zone_start_pfn, zone_end_pfn); - - /* Check that this node has pages within the zone's required range */ - if (*zone_end_pfn < node_start_pfn || *zone_start_pfn > node_end_pfn) - return 0; - - /* Move the zone boundaries inside the node if necessary */ - *zone_end_pfn = min(*zone_end_pfn, node_end_pfn); - *zone_start_pfn = max(*zone_start_pfn, node_start_pfn); - - /* Return the spanned pages */ - return *zone_end_pfn - *zone_start_pfn; -} - -/* - * Return the number of holes in a range on a node. If nid is MAX_NUMNODES, - * then all holes in the requested range will be accounted for. - */ -unsigned long __init __absent_pages_in_range(int nid, - unsigned long range_start_pfn, - unsigned long range_end_pfn) -{ - unsigned long nr_absent = range_end_pfn - range_start_pfn; - unsigned long start_pfn, end_pfn; - int i; - - for_each_mem_pfn_range(i, nid, &start_pfn, &end_pfn, NULL) { - start_pfn = clamp(start_pfn, range_start_pfn, range_end_pfn); - end_pfn = clamp(end_pfn, range_start_pfn, range_end_pfn); - nr_absent -= end_pfn - start_pfn; - } - return nr_absent; -} - -/** - * absent_pages_in_range - Return number of page frames in holes within a range - * @start_pfn: The start PFN to start searching for holes - * @end_pfn: The end PFN to stop searching for holes - * - * Return: the number of pages frames in memory holes within a range. - */ -unsigned long __init absent_pages_in_range(unsigned long start_pfn, - unsigned long end_pfn) -{ - return __absent_pages_in_range(MAX_NUMNODES, start_pfn, end_pfn); -} - -/* Return the number of page frames in holes in a zone on a node */ -static unsigned long __init zone_absent_pages_in_node(int nid, - unsigned long zone_type, - unsigned long node_start_pfn, - unsigned long node_end_pfn) -{ - unsigned long zone_low = arch_zone_lowest_possible_pfn[zone_type]; - unsigned long zone_high = arch_zone_highest_possible_pfn[zone_type]; - unsigned long zone_start_pfn, zone_end_pfn; - unsigned long nr_absent; - - /* When hotadd a new node from cpu_up(), the node should be empty */ - if (!node_start_pfn && !node_end_pfn) - return 0; - - zone_start_pfn = clamp(node_start_pfn, zone_low, zone_high); - zone_end_pfn = clamp(node_end_pfn, zone_low, zone_high); - - adjust_zone_range_for_zone_movable(nid, zone_type, - node_start_pfn, node_end_pfn, - &zone_start_pfn, &zone_end_pfn); - nr_absent = __absent_pages_in_range(nid, zone_start_pfn, zone_end_pfn); - - /* - * ZONE_MOVABLE handling. - * Treat pages to be ZONE_MOVABLE in ZONE_NORMAL as absent pages - * and vice versa. - */ - if (mirrored_kernelcore && zone_movable_pfn[nid]) { - unsigned long start_pfn, end_pfn; - struct memblock_region *r; - - for_each_mem_region(r) { - start_pfn = clamp(memblock_region_memory_base_pfn(r), - zone_start_pfn, zone_end_pfn); - end_pfn = clamp(memblock_region_memory_end_pfn(r), - zone_start_pfn, zone_end_pfn); - - if (zone_type == ZONE_MOVABLE && - memblock_is_mirror(r)) - nr_absent += end_pfn - start_pfn; - - if (zone_type == ZONE_NORMAL && - !memblock_is_mirror(r)) - nr_absent += end_pfn - start_pfn; - } - } - - return nr_absent; -} - -static void __init calculate_node_totalpages(struct pglist_data *pgdat, - unsigned long node_start_pfn, - unsigned long node_end_pfn) -{ - unsigned long realtotalpages = 0, totalpages = 0; - enum zone_type i; - - for (i = 0; i < MAX_NR_ZONES; i++) { - struct zone *zone = pgdat->node_zones + i; - unsigned long zone_start_pfn, zone_end_pfn; - unsigned long spanned, absent; - unsigned long size, real_size; - - spanned = zone_spanned_pages_in_node(pgdat->node_id, i, - node_start_pfn, - node_end_pfn, - &zone_start_pfn, - &zone_end_pfn); - absent = zone_absent_pages_in_node(pgdat->node_id, i, - node_start_pfn, - node_end_pfn); - - size = spanned; - real_size = size - absent; - - if (size) - zone->zone_start_pfn = zone_start_pfn; - else - zone->zone_start_pfn = 0; - zone->spanned_pages = size; - zone->present_pages = real_size; -#if defined(CONFIG_MEMORY_HOTPLUG) - zone->present_early_pages = real_size; -#endif - - totalpages += size; - realtotalpages += real_size; - } - - pgdat->node_spanned_pages = totalpages; - pgdat->node_present_pages = realtotalpages; - pr_debug("On node %d totalpages: %lu\n", pgdat->node_id, realtotalpages); -} - -#ifndef CONFIG_SPARSEMEM -/* - * Calculate the size of the zone->blockflags rounded to an unsigned long - * Start by making sure zonesize is a multiple of pageblock_order by rounding - * up. Then use 1 NR_PAGEBLOCK_BITS worth of bits per pageblock, finally - * round what is now in bits to nearest long in bits, then return it in - * bytes. - */ -static unsigned long __init usemap_size(unsigned long zone_start_pfn, unsigned long zonesize) -{ - unsigned long usemapsize; - - zonesize += zone_start_pfn & (pageblock_nr_pages-1); - usemapsize = roundup(zonesize, pageblock_nr_pages); - usemapsize = usemapsize >> pageblock_order; - usemapsize *= NR_PAGEBLOCK_BITS; - usemapsize = roundup(usemapsize, 8 * sizeof(unsigned long)); - - return usemapsize / 8; -} - -static void __ref setup_usemap(struct zone *zone) -{ - unsigned long usemapsize = usemap_size(zone->zone_start_pfn, - zone->spanned_pages); - zone->pageblock_flags = NULL; - if (usemapsize) { - zone->pageblock_flags = - memblock_alloc_node(usemapsize, SMP_CACHE_BYTES, - zone_to_nid(zone)); - if (!zone->pageblock_flags) - panic("Failed to allocate %ld bytes for zone %s pageblock flags on node %d\n", - usemapsize, zone->name, zone_to_nid(zone)); - } -} -#else -static inline void setup_usemap(struct zone *zone) {} -#endif /* CONFIG_SPARSEMEM */ - -#ifdef CONFIG_HUGETLB_PAGE_SIZE_VARIABLE - -/* Initialise the number of pages represented by NR_PAGEBLOCK_BITS */ -void __init set_pageblock_order(void) -{ - unsigned int order = MAX_ORDER; - - /* Check that pageblock_nr_pages has not already been setup */ - if (pageblock_order) - return; - - /* Don't let pageblocks exceed the maximum allocation granularity. */ - if (HPAGE_SHIFT > PAGE_SHIFT && HUGETLB_PAGE_ORDER < order) - order = HUGETLB_PAGE_ORDER; - - /* - * Assume the largest contiguous order of interest is a huge page. - * This value may be variable depending on boot parameters on IA64 and - * powerpc. - */ - pageblock_order = order; -} -#else /* CONFIG_HUGETLB_PAGE_SIZE_VARIABLE */ - -/* - * When CONFIG_HUGETLB_PAGE_SIZE_VARIABLE is not set, set_pageblock_order() - * is unused as pageblock_order is set at compile-time. See - * include/linux/pageblock-flags.h for the values of pageblock_order based on - * the kernel config - */ -void __init set_pageblock_order(void) -{ -} - -#endif /* CONFIG_HUGETLB_PAGE_SIZE_VARIABLE */ - -static unsigned long __init calc_memmap_size(unsigned long spanned_pages, - unsigned long present_pages) -{ - unsigned long pages = spanned_pages; - - /* - * Provide a more accurate estimation if there are holes within - * the zone and SPARSEMEM is in use. If there are holes within the - * zone, each populated memory region may cost us one or two extra - * memmap pages due to alignment because memmap pages for each - * populated regions may not be naturally aligned on page boundary. - * So the (present_pages >> 4) heuristic is a tradeoff for that. - */ - if (spanned_pages > present_pages + (present_pages >> 4) && - IS_ENABLED(CONFIG_SPARSEMEM)) - pages = present_pages; - - return PAGE_ALIGN(pages * sizeof(struct page)) >> PAGE_SHIFT; -} - -#ifdef CONFIG_TRANSPARENT_HUGEPAGE -static void pgdat_init_split_queue(struct pglist_data *pgdat) -{ - struct deferred_split *ds_queue = &pgdat->deferred_split_queue; - - spin_lock_init(&ds_queue->split_queue_lock); - INIT_LIST_HEAD(&ds_queue->split_queue); - ds_queue->split_queue_len = 0; -} -#else -static void pgdat_init_split_queue(struct pglist_data *pgdat) {} -#endif - -#ifdef CONFIG_COMPACTION -static void pgdat_init_kcompactd(struct pglist_data *pgdat) -{ - init_waitqueue_head(&pgdat->kcompactd_wait); -} -#else -static void pgdat_init_kcompactd(struct pglist_data *pgdat) {} -#endif - -static void __meminit pgdat_init_internals(struct pglist_data *pgdat) -{ - int i; - - pgdat_resize_init(pgdat); - pgdat_kswapd_lock_init(pgdat); - - pgdat_init_split_queue(pgdat); - pgdat_init_kcompactd(pgdat); - - init_waitqueue_head(&pgdat->kswapd_wait); - init_waitqueue_head(&pgdat->pfmemalloc_wait); - - for (i = 0; i < NR_VMSCAN_THROTTLE; i++) - init_waitqueue_head(&pgdat->reclaim_wait[i]); - - pgdat_page_ext_init(pgdat); - lruvec_init(&pgdat->__lruvec); -} - -static void __meminit zone_init_internals(struct zone *zone, enum zone_type idx, int nid, - unsigned long remaining_pages) -{ - atomic_long_set(&zone->managed_pages, remaining_pages); - zone_set_nid(zone, nid); - zone->name = zone_names[idx]; - zone->zone_pgdat = NODE_DATA(nid); - spin_lock_init(&zone->lock); - zone_seqlock_init(zone); - zone_pcp_init(zone); -} - -/* - * Set up the zone data structures - * - init pgdat internals - * - init all zones belonging to this node - * - * NOTE: this function is only called during memory hotplug - */ -#ifdef CONFIG_MEMORY_HOTPLUG -void __ref free_area_init_core_hotplug(struct pglist_data *pgdat) -{ - int nid = pgdat->node_id; - enum zone_type z; - int cpu; - - pgdat_init_internals(pgdat); - - if (pgdat->per_cpu_nodestats == &boot_nodestats) - pgdat->per_cpu_nodestats = alloc_percpu(struct per_cpu_nodestat); - - /* - * Reset the nr_zones, order and highest_zoneidx before reuse. - * Note that kswapd will init kswapd_highest_zoneidx properly - * when it starts in the near future. - */ - pgdat->nr_zones = 0; - pgdat->kswapd_order = 0; - pgdat->kswapd_highest_zoneidx = 0; - pgdat->node_start_pfn = 0; - for_each_online_cpu(cpu) { - struct per_cpu_nodestat *p; - - p = per_cpu_ptr(pgdat->per_cpu_nodestats, cpu); - memset(p, 0, sizeof(*p)); - } - - for (z = 0; z < MAX_NR_ZONES; z++) - zone_init_internals(&pgdat->node_zones[z], z, nid, 0); -} -#endif - -/* - * Set up the zone data structures: - * - mark all pages reserved - * - mark all memory queues empty - * - clear the memory bitmaps - * - * NOTE: pgdat should get zeroed by caller. - * NOTE: this function is only called during early init. - */ -static void __init free_area_init_core(struct pglist_data *pgdat) -{ - enum zone_type j; - int nid = pgdat->node_id; - - pgdat_init_internals(pgdat); - pgdat->per_cpu_nodestats = &boot_nodestats; - - for (j = 0; j < MAX_NR_ZONES; j++) { - struct zone *zone = pgdat->node_zones + j; - unsigned long size, freesize, memmap_pages; - - size = zone->spanned_pages; - freesize = zone->present_pages; - - /* - * Adjust freesize so that it accounts for how much memory - * is used by this zone for memmap. This affects the watermark - * and per-cpu initialisations - */ - memmap_pages = calc_memmap_size(size, freesize); - if (!is_highmem_idx(j)) { - if (freesize >= memmap_pages) { - freesize -= memmap_pages; - if (memmap_pages) - pr_debug(" %s zone: %lu pages used for memmap\n", - zone_names[j], memmap_pages); - } else - pr_warn(" %s zone: %lu memmap pages exceeds freesize %lu\n", - zone_names[j], memmap_pages, freesize); - } - - /* Account for reserved pages */ - if (j == 0 && freesize > dma_reserve) { - freesize -= dma_reserve; - pr_debug(" %s zone: %lu pages reserved\n", zone_names[0], dma_reserve); - } - - if (!is_highmem_idx(j)) - nr_kernel_pages += freesize; - /* Charge for highmem memmap if there are enough kernel pages */ - else if (nr_kernel_pages > memmap_pages * 2) - nr_kernel_pages -= memmap_pages; - nr_all_pages += freesize; - - /* - * Set an approximate value for lowmem here, it will be adjusted - * when the bootmem allocator frees pages into the buddy system. - * And all highmem pages will be managed by the buddy system. - */ - zone_init_internals(zone, j, nid, freesize); - - if (!size) - continue; - - set_pageblock_order(); - setup_usemap(zone); - init_currently_empty_zone(zone, zone->zone_start_pfn, size); - } -} - -#ifdef CONFIG_FLATMEM -static void __init alloc_node_mem_map(struct pglist_data *pgdat) -{ - unsigned long __maybe_unused start = 0; - unsigned long __maybe_unused offset = 0; - - /* Skip empty nodes */ - if (!pgdat->node_spanned_pages) - return; - - start = pgdat->node_start_pfn & ~(MAX_ORDER_NR_PAGES - 1); - offset = pgdat->node_start_pfn - start; - /* ia64 gets its own node_mem_map, before this, without bootmem */ - if (!pgdat->node_mem_map) { - unsigned long size, end; - struct page *map; - - /* - * The zone's endpoints aren't required to be MAX_ORDER - * aligned but the node_mem_map endpoints must be in order - * for the buddy allocator to function correctly. - */ - end = pgdat_end_pfn(pgdat); - end = ALIGN(end, MAX_ORDER_NR_PAGES); - size = (end - start) * sizeof(struct page); - map = memmap_alloc(size, SMP_CACHE_BYTES, MEMBLOCK_LOW_LIMIT, - pgdat->node_id, false); - if (!map) - panic("Failed to allocate %ld bytes for node %d memory map\n", - size, pgdat->node_id); - pgdat->node_mem_map = map + offset; - } - pr_debug("%s: node %d, pgdat %08lx, node_mem_map %08lx\n", - __func__, pgdat->node_id, (unsigned long)pgdat, - (unsigned long)pgdat->node_mem_map); -#ifndef CONFIG_NUMA - /* - * With no DISCONTIG, the global mem_map is just set as node 0's - */ - if (pgdat == NODE_DATA(0)) { - mem_map = NODE_DATA(0)->node_mem_map; - if (page_to_pfn(mem_map) != pgdat->node_start_pfn) - mem_map -= offset; - } -#endif -} -#else -static inline void alloc_node_mem_map(struct pglist_data *pgdat) { } -#endif /* CONFIG_FLATMEM */ - -#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT -static inline void pgdat_set_deferred_range(pg_data_t *pgdat) -{ - pgdat->first_deferred_pfn = ULONG_MAX; -} -#else -static inline void pgdat_set_deferred_range(pg_data_t *pgdat) {} -#endif - -static void __init free_area_init_node(int nid) -{ - pg_data_t *pgdat = NODE_DATA(nid); - unsigned long start_pfn = 0; - unsigned long end_pfn = 0; - - /* pg_data_t should be reset to zero when it's allocated */ - WARN_ON(pgdat->nr_zones || pgdat->kswapd_highest_zoneidx); - - get_pfn_range_for_nid(nid, &start_pfn, &end_pfn); - - pgdat->node_id = nid; - pgdat->node_start_pfn = start_pfn; - pgdat->per_cpu_nodestats = NULL; - - if (start_pfn != end_pfn) { - pr_info("Initmem setup node %d [mem %#018Lx-%#018Lx]\n", nid, - (u64)start_pfn << PAGE_SHIFT, - end_pfn ? ((u64)end_pfn << PAGE_SHIFT) - 1 : 0); - } else { - pr_info("Initmem setup node %d as memoryless\n", nid); - } - - calculate_node_totalpages(pgdat, start_pfn, end_pfn); - - alloc_node_mem_map(pgdat); - pgdat_set_deferred_range(pgdat); - - free_area_init_core(pgdat); - lru_gen_init_pgdat(pgdat); -} - -static void __init free_area_init_memoryless_node(int nid) -{ - free_area_init_node(nid); -} - -#if MAX_NUMNODES > 1 -/* - * Figure out the number of possible node ids. - */ -void __init setup_nr_node_ids(void) -{ - unsigned int highest; - - highest = find_last_bit(node_possible_map.bits, MAX_NUMNODES); - nr_node_ids = highest + 1; -} -#endif - -/** - * node_map_pfn_alignment - determine the maximum internode alignment - * - * This function should be called after node map is populated and sorted. - * It calculates the maximum power of two alignment which can distinguish - * all the nodes. - * - * For example, if all nodes are 1GiB and aligned to 1GiB, the return value - * would indicate 1GiB alignment with (1 << (30 - PAGE_SHIFT)). If the - * nodes are shifted by 256MiB, 256MiB. Note that if only the last node is - * shifted, 1GiB is enough and this function will indicate so. - * - * This is used to test whether pfn -> nid mapping of the chosen memory - * model has fine enough granularity to avoid incorrect mapping for the - * populated node map. - * - * Return: the determined alignment in pfn's. 0 if there is no alignment - * requirement (single node). - */ -unsigned long __init node_map_pfn_alignment(void) -{ - unsigned long accl_mask = 0, last_end = 0; - unsigned long start, end, mask; - int last_nid = NUMA_NO_NODE; - int i, nid; - - for_each_mem_pfn_range(i, MAX_NUMNODES, &start, &end, &nid) { - if (!start || last_nid < 0 || last_nid == nid) { - last_nid = nid; - last_end = end; - continue; - } - - /* - * Start with a mask granular enough to pin-point to the - * start pfn and tick off bits one-by-one until it becomes - * too coarse to separate the current node from the last. - */ - mask = ~((1 << __ffs(start)) - 1); - while (mask && last_end <= (start & (mask << 1))) - mask <<= 1; - - /* accumulate all internode masks */ - accl_mask |= mask; - } - - /* convert mask to number of pages */ - return ~accl_mask + 1; -} - -/* - * early_calculate_totalpages() - * Sum pages in active regions for movable zone. - * Populate N_MEMORY for calculating usable_nodes. - */ -static unsigned long __init early_calculate_totalpages(void) -{ - unsigned long totalpages = 0; - unsigned long start_pfn, end_pfn; - int i, nid; - - for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, &nid) { - unsigned long pages = end_pfn - start_pfn; - - totalpages += pages; - if (pages) - node_set_state(nid, N_MEMORY); - } - return totalpages; -} - -/* - * Find the PFN the Movable zone begins in each node. Kernel memory - * is spread evenly between nodes as long as the nodes have enough - * memory. When they don't, some nodes will have more kernelcore than - * others - */ -static void __init find_zone_movable_pfns_for_nodes(void) -{ - int i, nid; - unsigned long usable_startpfn; - unsigned long kernelcore_node, kernelcore_remaining; - /* save the state before borrow the nodemask */ - nodemask_t saved_node_state = node_states[N_MEMORY]; - unsigned long totalpages = early_calculate_totalpages(); - int usable_nodes = nodes_weight(node_states[N_MEMORY]); - struct memblock_region *r; - - /* Need to find movable_zone earlier when movable_node is specified. */ - find_usable_zone_for_movable(); - - /* - * If movable_node is specified, ignore kernelcore and movablecore - * options. - */ - if (movable_node_is_enabled()) { - for_each_mem_region(r) { - if (!memblock_is_hotpluggable(r)) - continue; - - nid = memblock_get_region_node(r); - - usable_startpfn = PFN_DOWN(r->base); - zone_movable_pfn[nid] = zone_movable_pfn[nid] ? - min(usable_startpfn, zone_movable_pfn[nid]) : - usable_startpfn; - } - - goto out2; - } - - /* - * If kernelcore=mirror is specified, ignore movablecore option - */ - if (mirrored_kernelcore) { - bool mem_below_4gb_not_mirrored = false; - - for_each_mem_region(r) { - if (memblock_is_mirror(r)) - continue; - - nid = memblock_get_region_node(r); - - usable_startpfn = memblock_region_memory_base_pfn(r); - - if (usable_startpfn < PHYS_PFN(SZ_4G)) { - mem_below_4gb_not_mirrored = true; - continue; - } - - zone_movable_pfn[nid] = zone_movable_pfn[nid] ? - min(usable_startpfn, zone_movable_pfn[nid]) : - usable_startpfn; - } - - if (mem_below_4gb_not_mirrored) - pr_warn("This configuration results in unmirrored kernel memory.\n"); - - goto out2; - } - - /* - * If kernelcore=nn% or movablecore=nn% was specified, calculate the - * amount of necessary memory. - */ - if (required_kernelcore_percent) - required_kernelcore = (totalpages * 100 * required_kernelcore_percent) / - 10000UL; - if (required_movablecore_percent) - required_movablecore = (totalpages * 100 * required_movablecore_percent) / - 10000UL; - - /* - * If movablecore= was specified, calculate what size of - * kernelcore that corresponds so that memory usable for - * any allocation type is evenly spread. If both kernelcore - * and movablecore are specified, then the value of kernelcore - * will be used for required_kernelcore if it's greater than - * what movablecore would have allowed. - */ - if (required_movablecore) { - unsigned long corepages; - - /* - * Round-up so that ZONE_MOVABLE is at least as large as what - * was requested by the user - */ - required_movablecore = - roundup(required_movablecore, MAX_ORDER_NR_PAGES); - required_movablecore = min(totalpages, required_movablecore); - corepages = totalpages - required_movablecore; - - required_kernelcore = max(required_kernelcore, corepages); - } - - /* - * If kernelcore was not specified or kernelcore size is larger - * than totalpages, there is no ZONE_MOVABLE. - */ - if (!required_kernelcore || required_kernelcore >= totalpages) - goto out; - - /* usable_startpfn is the lowest possible pfn ZONE_MOVABLE can be at */ - usable_startpfn = arch_zone_lowest_possible_pfn[movable_zone]; - -restart: - /* Spread kernelcore memory as evenly as possible throughout nodes */ - kernelcore_node = required_kernelcore / usable_nodes; - for_each_node_state(nid, N_MEMORY) { - unsigned long start_pfn, end_pfn; +#ifdef CONFIG_MMU + int high; + int nr_split_cpus; + unsigned long total_pages; + if (!percpu_pagelist_high_fraction) { /* - * Recalculate kernelcore_node if the division per node - * now exceeds what is necessary to satisfy the requested - * amount of memory for the kernel + * By default, the high value of the pcp is based on the zone + * low watermark so that if they are full then background + * reclaim will not be started prematurely. */ - if (required_kernelcore < kernelcore_node) - kernelcore_node = required_kernelcore / usable_nodes; - + total_pages = low_wmark_pages(zone); + } else { /* - * As the map is walked, we track how much memory is usable - * by the kernel using kernelcore_remaining. When it is - * 0, the rest of the node is usable by ZONE_MOVABLE + * If percpu_pagelist_high_fraction is configured, the high + * value is based on a fraction of the managed pages in the + * zone. */ - kernelcore_remaining = kernelcore_node; - - /* Go through each range of PFNs within this node */ - for_each_mem_pfn_range(i, nid, &start_pfn, &end_pfn, NULL) { - unsigned long size_pages; - - start_pfn = max(start_pfn, zone_movable_pfn[nid]); - if (start_pfn >= end_pfn) - continue; - - /* Account for what is only usable for kernelcore */ - if (start_pfn < usable_startpfn) { - unsigned long kernel_pages; - kernel_pages = min(end_pfn, usable_startpfn) - - start_pfn; - - kernelcore_remaining -= min(kernel_pages, - kernelcore_remaining); - required_kernelcore -= min(kernel_pages, - required_kernelcore); - - /* Continue if range is now fully accounted */ - if (end_pfn <= usable_startpfn) { - - /* - * Push zone_movable_pfn to the end so - * that if we have to rebalance - * kernelcore across nodes, we will - * not double account here - */ - zone_movable_pfn[nid] = end_pfn; - continue; - } - start_pfn = usable_startpfn; - } - - /* - * The usable PFN range for ZONE_MOVABLE is from - * start_pfn->end_pfn. Calculate size_pages as the - * number of pages used as kernelcore - */ - size_pages = end_pfn - start_pfn; - if (size_pages > kernelcore_remaining) - size_pages = kernelcore_remaining; - zone_movable_pfn[nid] = start_pfn + size_pages; - - /* - * Some kernelcore has been met, update counts and - * break if the kernelcore for this node has been - * satisfied - */ - required_kernelcore -= min(required_kernelcore, - size_pages); - kernelcore_remaining -= size_pages; - if (!kernelcore_remaining) - break; - } + total_pages = zone_managed_pages(zone) / percpu_pagelist_high_fraction; } /* - * If there is still required_kernelcore, we do another pass with one - * less node in the count. This will push zone_movable_pfn[nid] further - * along on the nodes that still have memory until kernelcore is - * satisfied + * Split the high value across all online CPUs local to the zone. Note + * that early in boot that CPUs may not be online yet and that during + * CPU hotplug that the cpumask is not yet updated when a CPU is being + * onlined. For memory nodes that have no CPUs, split pcp->high across + * all online CPUs to mitigate the risk that reclaim is triggered + * prematurely due to pages stored on pcp lists. */ - usable_nodes--; - if (usable_nodes && required_kernelcore > usable_nodes) - goto restart; - -out2: - /* Align start of ZONE_MOVABLE on all nids to MAX_ORDER_NR_PAGES */ - for (nid = 0; nid < MAX_NUMNODES; nid++) { - unsigned long start_pfn, end_pfn; - - zone_movable_pfn[nid] = - roundup(zone_movable_pfn[nid], MAX_ORDER_NR_PAGES); - - get_pfn_range_for_nid(nid, &start_pfn, &end_pfn); - if (zone_movable_pfn[nid] >= end_pfn) - zone_movable_pfn[nid] = 0; - } - -out: - /* restore the node_state */ - node_states[N_MEMORY] = saved_node_state; -} + nr_split_cpus = cpumask_weight(cpumask_of_node(zone_to_nid(zone))) + cpu_online; + if (!nr_split_cpus) + nr_split_cpus = num_online_cpus(); + high = total_pages / nr_split_cpus; -/* Any regular or high memory on that node ? */ -static void check_for_memory(pg_data_t *pgdat, int nid) -{ - enum zone_type zone_type; + /* + * Ensure high is at least batch*4. The multiple is based on the + * historical relationship between high and batch. + */ + high = max(high, batch << 2); - for (zone_type = 0; zone_type <= ZONE_MOVABLE - 1; zone_type++) { - struct zone *zone = &pgdat->node_zones[zone_type]; - if (populated_zone(zone)) { - if (IS_ENABLED(CONFIG_HIGHMEM)) - node_set_state(nid, N_HIGH_MEMORY); - if (zone_type <= ZONE_NORMAL) - node_set_state(nid, N_NORMAL_MEMORY); - break; - } - } + return high; +#else + return 0; +#endif } /* - * Some architectures, e.g. ARC may have ZONE_HIGHMEM below ZONE_NORMAL. For - * such cases we allow max_zone_pfn sorted in the descending order + * pcp->high and pcp->batch values are related and generally batch is lower + * than high. They are also related to pcp->count such that count is lower + * than high, and as soon as it reaches high, the pcplist is flushed. + * + * However, guaranteeing these relations at all times would require e.g. write + * barriers here but also careful usage of read barriers at the read side, and + * thus be prone to error and bad for performance. Thus the update only prevents + * store tearing. Any new users of pcp->batch and pcp->high should ensure they + * can cope with those fields changing asynchronously, and fully trust only the + * pcp->count field on the local CPU with interrupts disabled. + * + * mutex_is_locked(&pcp_batch_high_lock) required when calling this function + * outside of boot time (or some other assurance that no concurrent updaters + * exist). */ -bool __weak arch_has_descending_max_zone_pfns(void) +static void pageset_update(struct per_cpu_pages *pcp, unsigned long high, + unsigned long batch) { - return false; + WRITE_ONCE(pcp->batch, batch); + WRITE_ONCE(pcp->high, high); } -/** - * free_area_init - Initialise all pg_data_t and zone data - * @max_zone_pfn: an array of max PFNs for each zone - * - * This will call free_area_init_node() for each active node in the system. - * Using the page ranges provided by memblock_set_node(), the size of each - * zone in each node and their holes is calculated. If the maximum PFN - * between two adjacent zones match, it is assumed that the zone is empty. - * For example, if arch_max_dma_pfn == arch_max_dma32_pfn, it is assumed - * that arch_max_dma32_pfn has no pages. It is also assumed that a zone - * starts where the previous one ended. For example, ZONE_DMA32 starts - * at arch_max_dma_pfn. - */ -void __init free_area_init(unsigned long *max_zone_pfn) +static void per_cpu_pages_init(struct per_cpu_pages *pcp, struct per_cpu_zonestat *pzstats) { - unsigned long start_pfn, end_pfn; - int i, nid, zone; - bool descending; - - /* Record where the zone boundaries are */ - memset(arch_zone_lowest_possible_pfn, 0, - sizeof(arch_zone_lowest_possible_pfn)); - memset(arch_zone_highest_possible_pfn, 0, - sizeof(arch_zone_highest_possible_pfn)); - - start_pfn = PHYS_PFN(memblock_start_of_DRAM()); - descending = arch_has_descending_max_zone_pfns(); - - for (i = 0; i < MAX_NR_ZONES; i++) { - if (descending) - zone = MAX_NR_ZONES - i - 1; - else - zone = i; - - if (zone == ZONE_MOVABLE) - continue; + int pindex; - end_pfn = max(max_zone_pfn[zone], start_pfn); - arch_zone_lowest_possible_pfn[zone] = start_pfn; - arch_zone_highest_possible_pfn[zone] = end_pfn; + memset(pcp, 0, sizeof(*pcp)); + memset(pzstats, 0, sizeof(*pzstats)); - start_pfn = end_pfn; - } + spin_lock_init(&pcp->lock); + for (pindex = 0; pindex < NR_PCP_LISTS; pindex++) + INIT_LIST_HEAD(&pcp->lists[pindex]); - /* Find the PFNs that ZONE_MOVABLE begins at in each node */ - memset(zone_movable_pfn, 0, sizeof(zone_movable_pfn)); - find_zone_movable_pfns_for_nodes(); + /* + * Set batch and high values safe for a boot pageset. A true percpu + * pageset's initialization will update them subsequently. Here we don't + * need to be as careful as pageset_update() as nobody can access the + * pageset yet. + */ + pcp->high = BOOT_PAGESET_HIGH; + pcp->batch = BOOT_PAGESET_BATCH; + pcp->free_factor = 0; +} - /* Print out the zone ranges */ - pr_info("Zone ranges:\n"); - for (i = 0; i < MAX_NR_ZONES; i++) { - if (i == ZONE_MOVABLE) - continue; - pr_info(" %-8s ", zone_names[i]); - if (arch_zone_lowest_possible_pfn[i] == - arch_zone_highest_possible_pfn[i]) - pr_cont("empty\n"); - else - pr_cont("[mem %#018Lx-%#018Lx]\n", - (u64)arch_zone_lowest_possible_pfn[i] - << PAGE_SHIFT, - ((u64)arch_zone_highest_possible_pfn[i] - << PAGE_SHIFT) - 1); - } +static void __zone_set_pageset_high_and_batch(struct zone *zone, unsigned long high, + unsigned long batch) +{ + struct per_cpu_pages *pcp; + int cpu; - /* Print out the PFNs ZONE_MOVABLE begins at in each node */ - pr_info("Movable zone start for each node\n"); - for (i = 0; i < MAX_NUMNODES; i++) { - if (zone_movable_pfn[i]) - pr_info(" Node %d: %#018Lx\n", i, - (u64)zone_movable_pfn[i] << PAGE_SHIFT); + for_each_possible_cpu(cpu) { + pcp = per_cpu_ptr(zone->per_cpu_pageset, cpu); + pageset_update(pcp, high, batch); } +} - /* - * Print out the early node map, and initialize the - * subsection-map relative to active online memory ranges to - * enable future "sub-section" extensions of the memory map. - */ - pr_info("Early memory node ranges\n"); - for_each_mem_pfn_range(i, MAX_NUMNODES, &start_pfn, &end_pfn, &nid) { - pr_info(" node %3d: [mem %#018Lx-%#018Lx]\n", nid, - (u64)start_pfn << PAGE_SHIFT, - ((u64)end_pfn << PAGE_SHIFT) - 1); - subsection_map_init(start_pfn, end_pfn - start_pfn); - } - - /* Initialise every node */ - mminit_verify_pageflags_layout(); - setup_nr_node_ids(); - for_each_node(nid) { - pg_data_t *pgdat; - - if (!node_online(nid)) { - pr_info("Initializing node %d as memoryless\n", nid); - - /* Allocator not initialized yet */ - pgdat = arch_alloc_nodedata(nid); - if (!pgdat) - panic("Cannot allocate %zuB for node %d.\n", - sizeof(*pgdat), nid); - arch_refresh_nodedata(nid, pgdat); - free_area_init_memoryless_node(nid); +/* + * Calculate and set new high and batch values for all per-cpu pagesets of a + * zone based on the zone's size. + */ +static void zone_set_pageset_high_and_batch(struct zone *zone, int cpu_online) +{ + int new_high, new_batch; - /* - * We do not want to confuse userspace by sysfs - * files/directories for node without any memory - * attached to it, so this node is not marked as - * N_MEMORY and not marked online so that no sysfs - * hierarchy will be created via register_one_node for - * it. The pgdat will get fully initialized by - * hotadd_init_pgdat() when memory is hotplugged into - * this node. - */ - continue; - } + new_batch = max(1, zone_batchsize(zone)); + new_high = zone_highsize(zone, new_batch, cpu_online); - pgdat = NODE_DATA(nid); - free_area_init_node(nid); + if (zone->pageset_high == new_high && + zone->pageset_batch == new_batch) + return; - /* Any memory on that node */ - if (pgdat->node_present_pages) - node_set_state(nid, N_MEMORY); - check_for_memory(pgdat, nid); - } + zone->pageset_high = new_high; + zone->pageset_batch = new_batch; - memmap_init(); + __zone_set_pageset_high_and_batch(zone, new_high, new_batch); } -static int __init cmdline_parse_core(char *p, unsigned long *core, - unsigned long *percent) +void __meminit setup_zone_pageset(struct zone *zone) { - unsigned long long coremem; - char *endptr; - - if (!p) - return -EINVAL; + int cpu; - /* Value may be a percentage of total memory, otherwise bytes */ - coremem = simple_strtoull(p, &endptr, 0); - if (*endptr == '%') { - /* Paranoid check for percent values greater than 100 */ - WARN_ON(coremem > 100); + /* Size may be 0 on !SMP && !NUMA */ + if (sizeof(struct per_cpu_zonestat) > 0) + zone->per_cpu_zonestats = alloc_percpu(struct per_cpu_zonestat); - *percent = coremem; - } else { - coremem = memparse(p, &p); - /* Paranoid check that UL is enough for the coremem value */ - WARN_ON((coremem >> PAGE_SHIFT) > ULONG_MAX); + zone->per_cpu_pageset = alloc_percpu(struct per_cpu_pages); + for_each_possible_cpu(cpu) { + struct per_cpu_pages *pcp; + struct per_cpu_zonestat *pzstats; - *core = coremem >> PAGE_SHIFT; - *percent = 0UL; + pcp = per_cpu_ptr(zone->per_cpu_pageset, cpu); + pzstats = per_cpu_ptr(zone->per_cpu_zonestats, cpu); + per_cpu_pages_init(pcp, pzstats); } - return 0; + + zone_set_pageset_high_and_batch(zone, 0); } /* - * kernelcore=size sets the amount of memory for use for allocations that - * cannot be reclaimed or migrated. + * The zone indicated has a new number of managed_pages; batch sizes and percpu + * page high values need to be recalculated. */ -static int __init cmdline_parse_kernelcore(char *p) +static void zone_pcp_update(struct zone *zone, int cpu_online) { - /* parse kernelcore=mirror */ - if (parse_option_str(p, "mirror")) { - mirrored_kernelcore = true; - return 0; - } - - return cmdline_parse_core(p, &required_kernelcore, - &required_kernelcore_percent); + mutex_lock(&pcp_batch_high_lock); + zone_set_pageset_high_and_batch(zone, cpu_online); + mutex_unlock(&pcp_batch_high_lock); } /* - * movablecore=size sets the amount of memory for use for allocations that - * can be reclaimed or migrated. + * Allocate per cpu pagesets and initialize them. + * Before this call only boot pagesets were available. */ -static int __init cmdline_parse_movablecore(char *p) +void __init setup_per_cpu_pageset(void) { - return cmdline_parse_core(p, &required_movablecore, - &required_movablecore_percent); + struct pglist_data *pgdat; + struct zone *zone; + int __maybe_unused cpu; + + for_each_populated_zone(zone) + setup_zone_pageset(zone); + +#ifdef CONFIG_NUMA + /* + * Unpopulated zones continue using the boot pagesets. + * The numa stats for these pagesets need to be reset. + * Otherwise, they will end up skewing the stats of + * the nodes these zones are associated with. + */ + for_each_possible_cpu(cpu) { + struct per_cpu_zonestat *pzstats = &per_cpu(boot_zonestats, cpu); + memset(pzstats->vm_numa_event, 0, + sizeof(pzstats->vm_numa_event)); + } +#endif + + for_each_online_pgdat(pgdat) + pgdat->per_cpu_nodestats = + alloc_percpu(struct per_cpu_nodestat); } -early_param("kernelcore", cmdline_parse_kernelcore); -early_param("movablecore", cmdline_parse_movablecore); +__meminit void zone_pcp_init(struct zone *zone) +{ + /* + * per cpu subsystem is not up at this point. The following code + * relies on the ability of the linker to provide the + * offset of a (static) per cpu variable into the per cpu area. + */ + zone->per_cpu_pageset = &boot_pageset; + zone->per_cpu_zonestats = &boot_zonestats; + zone->pageset_high = BOOT_PAGESET_HIGH; + zone->pageset_batch = BOOT_PAGESET_BATCH; + + if (populated_zone(zone)) + pr_debug(" %s zone: %lu pages, LIFO batch:%u\n", zone->name, + zone->present_pages, zone_batchsize(zone)); +} void adjust_managed_page_count(struct page *page, long count) { @@ -8488,22 +6346,6 @@ void __init mem_init_print_info(void) ); } -/** - * set_dma_reserve - set the specified number of pages reserved in the first zone - * @new_dma_reserve: The number of pages to mark reserved - * - * The per-cpu batchsize and zone watermarks are determined by managed_pages. - * In the DMA zone, a significant percentage may be consumed by kernel image - * and other unfreeable allocations which can skew the watermarks badly. This - * function may optionally be used to account for unfreeable pages in the - * first zone (e.g., ZONE_DMA). The effect will be lower watermarks and - * smaller per-cpu batchsize. - */ -void __init set_dma_reserve(unsigned long new_dma_reserve) -{ - dma_reserve = new_dma_reserve; -} - static int page_alloc_cpu_dead(unsigned int cpu) { struct zone *zone; @@ -8945,149 +6787,6 @@ int percpu_pagelist_high_fraction_sysctl_handler(struct ctl_table *table, return ret; } -#ifndef __HAVE_ARCH_RESERVED_KERNEL_PAGES -/* - * Returns the number of pages that arch has reserved but - * is not known to alloc_large_system_hash(). - */ -static unsigned long __init arch_reserved_kernel_pages(void) -{ - return 0; -} -#endif - -/* - * Adaptive scale is meant to reduce sizes of hash tables on large memory - * machines. As memory size is increased the scale is also increased but at - * slower pace. Starting from ADAPT_SCALE_BASE (64G), every time memory - * quadruples the scale is increased by one, which means the size of hash table - * only doubles, instead of quadrupling as well. - * Because 32-bit systems cannot have large physical memory, where this scaling - * makes sense, it is disabled on such platforms. - */ -#if __BITS_PER_LONG > 32 -#define ADAPT_SCALE_BASE (64ul << 30) -#define ADAPT_SCALE_SHIFT 2 -#define ADAPT_SCALE_NPAGES (ADAPT_SCALE_BASE >> PAGE_SHIFT) -#endif - -/* - * allocate a large system hash table from bootmem - * - it is assumed that the hash table must contain an exact power-of-2 - * quantity of entries - * - limit is the number of hash buckets, not the total allocation size - */ -void *__init alloc_large_system_hash(const char *tablename, - unsigned long bucketsize, - unsigned long numentries, - int scale, - int flags, - unsigned int *_hash_shift, - unsigned int *_hash_mask, - unsigned long low_limit, - unsigned long high_limit) -{ - unsigned long long max = high_limit; - unsigned long log2qty, size; - void *table; - gfp_t gfp_flags; - bool virt; - bool huge; - - /* allow the kernel cmdline to have a say */ - if (!numentries) { - /* round applicable memory size up to nearest megabyte */ - numentries = nr_kernel_pages; - numentries -= arch_reserved_kernel_pages(); - - /* It isn't necessary when PAGE_SIZE >= 1MB */ - if (PAGE_SIZE < SZ_1M) - numentries = round_up(numentries, SZ_1M / PAGE_SIZE); - -#if __BITS_PER_LONG > 32 - if (!high_limit) { - unsigned long adapt; - - for (adapt = ADAPT_SCALE_NPAGES; adapt < numentries; - adapt <<= ADAPT_SCALE_SHIFT) - scale++; - } -#endif - - /* limit to 1 bucket per 2^scale bytes of low memory */ - if (scale > PAGE_SHIFT) - numentries >>= (scale - PAGE_SHIFT); - else - numentries <<= (PAGE_SHIFT - scale); - - /* Make sure we've got at least a 0-order allocation.. */ - if (unlikely(flags & HASH_SMALL)) { - /* Makes no sense without HASH_EARLY */ - WARN_ON(!(flags & HASH_EARLY)); - if (!(numentries >> *_hash_shift)) { - numentries = 1UL << *_hash_shift; - BUG_ON(!numentries); - } - } else if (unlikely((numentries * bucketsize) < PAGE_SIZE)) - numentries = PAGE_SIZE / bucketsize; - } - numentries = roundup_pow_of_two(numentries); - - /* limit allocation size to 1/16 total memory by default */ - if (max == 0) { - max = ((unsigned long long)nr_all_pages << PAGE_SHIFT) >> 4; - do_div(max, bucketsize); - } - max = min(max, 0x80000000ULL); - - if (numentries < low_limit) - numentries = low_limit; - if (numentries > max) - numentries = max; - - log2qty = ilog2(numentries); - - gfp_flags = (flags & HASH_ZERO) ? GFP_ATOMIC | __GFP_ZERO : GFP_ATOMIC; - do { - virt = false; - size = bucketsize << log2qty; - if (flags & HASH_EARLY) { - if (flags & HASH_ZERO) - table = memblock_alloc(size, SMP_CACHE_BYTES); - else - table = memblock_alloc_raw(size, - SMP_CACHE_BYTES); - } else if (get_order(size) > MAX_ORDER || hashdist) { - table = vmalloc_huge(size, gfp_flags); - virt = true; - if (table) - huge = is_vm_area_hugepages(table); - } else { - /* - * If bucketsize is not a power-of-two, we may free - * some pages at the end of hash table which - * alloc_pages_exact() automatically does - */ - table = alloc_pages_exact(size, gfp_flags); - kmemleak_alloc(table, size, 1, gfp_flags); - } - } while (!table && size > PAGE_SIZE && --log2qty); - - if (!table) - panic("Failed to allocate %s hash table\n", tablename); - - pr_info("%s hash table entries: %ld (order: %d, %lu bytes, %s)\n", - tablename, 1UL << log2qty, ilog2(size) - PAGE_SHIFT, size, - virt ? (huge ? "vmalloc hugepage" : "vmalloc") : "linear"); - - if (_hash_shift) - *_hash_shift = log2qty; - if (_hash_mask) - *_hash_mask = (1 << log2qty) - 1; - - return table; -} - #ifdef CONFIG_CONTIG_ALLOC #if defined(CONFIG_DYNAMIC_DEBUG) || \ (defined(CONFIG_DYNAMIC_DEBUG_CORE) && defined(DYNAMIC_DEBUG_MODULE)) From patchwork Sun Mar 19 21:59:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13180633 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85381C7618A for ; Sun, 19 Mar 2023 22:00:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5F229900008; Sun, 19 Mar 2023 18:00:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5A12C900002; Sun, 19 Mar 2023 18:00:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 48F59900008; Sun, 19 Mar 2023 18:00:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 3A5E5900002 for ; Sun, 19 Mar 2023 18:00:44 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 12EAEC0388 for ; Sun, 19 Mar 2023 22:00:44 +0000 (UTC) X-FDA: 80587018008.30.C23FF8A Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf03.hostedemail.com (Postfix) with ESMTP id 562F520012 for ; Sun, 19 Mar 2023 22:00:42 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=cKXnkkoH; spf=pass (imf03.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679263242; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HGST2anScb4FjWhK/kqFOnDwJJopTO7Qtn7SPmlR7lU=; b=fTCXqHxooY6kmszdch3JO8lmaEosJ1kvtvhueWsiNhWzMKcUaYfvSo58Rj1FoA/HCtJ4O0 KT+5c+4AxsWUiCHSoqrh6RTAQ+Koz1E7u3ggI7JQapLRhY5BxA+Ksgk662lACADRmERYps 7tZXOxl0EVEDKqV8+BHLid/V8Le7O6o= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=cKXnkkoH; spf=pass (imf03.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679263242; a=rsa-sha256; cv=none; b=bPgzSg043sJKM5W6613PZdp54bk/ZcCASe/faTq3gM9mAHfwtGf34xdo/wSl+L3U1DTOX3 k4pm3fXJfTzc0ufuD7Ky4BFJXrEC/M8r+q4ZZ+bhHDVeZQV401gAVexOvyXW/dWQjIhGqW Tr0DsYXD/d6pT4idv2x/f9J3C/alpxY= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 95877611A7; Sun, 19 Mar 2023 22:00:41 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EB2E8C433A0; Sun, 19 Mar 2023 22:00:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679263241; bh=P8FXnqE0dz8+EFTBTuViTvRWyj1tIPqZHteVCKqB69c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cKXnkkoHgl0euhog4fZoR5wa1c1Sd1phnsl14UWKcX5nkks8OeN3Jxnh6Fc9iJydM DzTs8rBsC1k3D1xz99rAr1FAM4G1QaAtbwlmRgyLhloYNQdUTrdTseiSply+39hlVB J7unwXUSRZjkZBtSLNT46Su7PVQfoEmN4alECNByK10OBwwFvM4I4brCN67bkGyqdw +8Axk9QL2Ld8K7ysWax2H47g1RkxYxAqYMYxygOXB79+RlEsaM7B5HbLbhZmUsZ62f 4d4yuRcfRDuNKFHgp5qUQzLj6gM3u83AijuLjnnZvEjGwLzkSxFOwg7KDjRtZGLLBf IiyMGiuxlsp/A== From: Mike Rapoport To: Andrew Morton Cc: David Hildenbrand , Matthew Wilcox , Mel Gorman , Michal Hocko , Mike Rapoport , Thomas Bogendoerfer , Vlastimil Babka , linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 05/15] mm: handle hashdist initialization in mm/mm_init.c Date: Sun, 19 Mar 2023 23:59:58 +0200 Message-Id: <20230319220008.2138576-6-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230319220008.2138576-1-rppt@kernel.org> References: <20230319220008.2138576-1-rppt@kernel.org> MIME-Version: 1.0 X-Stat-Signature: f3epkic1mn5xj1a8x3wfputn9jnjn8w3 X-Rspam-User: X-Rspamd-Queue-Id: 562F520012 X-Rspamd-Server: rspam06 X-HE-Tag: 1679263242-150808 X-HE-Meta: U2FsdGVkX1+ylKaIaQbttMeSPOE2BBYDiPajtscd/1x/o0/QrFv5H58lVuNyl3K4tV+vvzP/Hj1cIBe6nbWz32HLlhvkDnbGXp2goRagsiay8S6yHIdd8ZxPx+JcSleCxzfsC7TnIC5AKmEqdbuUtvvsHHuThYfVGVOIfs2QNiX9pD7Nv/cENfbeW7k/D/o8LreQNoSE8ZoeT53cSOUyiBa9rHwTolOT08QVLBranMMWuYs4ecF14LSYjjofbhYxRa2n/kmQu3igaXiwT8wYGJibeHygM+lSxWyt+Jz6GROF9aIhV9BiSMxTWhbOPhEjb6rBYwzQ62p3QyzXz14NPXbW8cii4X7aGPHtMqPFAg+ie42cME7M5covokWPLTG4mKdmJb2FdNE/bUXoOmmlUG1fntEPwXZ0OeljT5/m4xKdbuyWL9YDqRKusYy+feu2o/fQrQY2d1IJOAHEQay/u/a65sK0xSap/HrqDCqG0vEc65l9ECjkCejPfQshcLftp1/YgHjVv8fi+V0CVwP0X4Ts7yVnfcpOgRMQGdxtOu+PdCvtRmN53T1IwXp/jiaGfA138SPd3isSJPuVajsSsBaG/VEGzEURxD9IuRVxv2lw0UQCVdDvM7TwuDJh/Ce1Co11FXWKddMRc2l5ycYTNAO2Cw1Y+EUomwbUQuCIwrXo2yh/CRSY3r5Gz38RfbOrfWAKGzJ/2TCUt0G7YyPz7J9rXWNQso5LJyEbGe20TOlHiz0StpuN24xZMkjZPtMoQvsSIZ8eUhWKoSWMTc8QiS8rVvWMHTDfYx4efTQeIooiBjossx4l+aPA/+yQBnf/A9zmIOu9+ehooLjJHTqPLPnMGVPhpA1VuPFnFoNf3ec/2RKgiHU8ay8zu2GipF3m8nb0e1OY8tCsP6kOCDNk0oAuE3mimRJjeGq1uBKoYKcY0ncb4U/0i8aFgVY/gNXH1SLnxcAnt5Y0EvG9P6N Lt/4JUyN gONe+sNCtC7fasfTzu9oYR43l7fFp6Nh7+V8fN8BQnZ4yVxX5sx9reiw40dPXjy9IGKNZTHVi6+zw+/OQnD1c7TdTSBPfNI/v/XLzE4g+/BmnyvykZY7ysZuJtVZurTFX6rLJyIyM6gmsBAceY9m0425FFsNQwbvjJAUaruL3Zh1uwWW61KRi7nMXtx9Wrjg7VWydPwp/G+4f5ois0Biv5m6b5j9XZE6YMJzY7XWXo9er69rJgdgqtiHt85+qexUJuXcb4iPDx046cUZMesfaat1Qn4l9HY9sSlg+m6Rlog5qh8Owjxlkw7ylCln2sR8ZhVu+uVQ86CN+QxJ0mTpxSmjGvGcohLr7AIxqev7YOzerSMBMqBXUtrFyaALTDrqOigQSzpaAlP8jyqQ38MUQFom7IQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Mike Rapoport (IBM)" The hashdist variable must be initialized before the first call to alloc_large_system_hash() and free_area_init() looks like a better place for it than page_alloc_init(). Move hashdist handling to mm/mm_init.c Signed-off-by: Mike Rapoport (IBM) Acked-by: David Hildenbrand --- mm/mm_init.c | 22 ++++++++++++++++++++++ mm/page_alloc.c | 18 ------------------ 2 files changed, 22 insertions(+), 18 deletions(-) diff --git a/mm/mm_init.c b/mm/mm_init.c index 63aa7b6b2880..8aaaddd13a20 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -607,6 +607,25 @@ int __meminit early_pfn_to_nid(unsigned long pfn) return nid; } + +int hashdist = HASHDIST_DEFAULT; + +static int __init set_hashdist(char *str) +{ + if (!str) + return 0; + hashdist = simple_strtoul(str, &str, 0); + return 1; +} +__setup("hashdist=", set_hashdist); + +static inline void fixup_hashdist(void) +{ + if (num_node_state(N_MEMORY) == 1) + hashdist = 0; +} +#else +static inline void fixup_hashdist(void) {} #endif /* CONFIG_NUMA */ #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT @@ -1855,6 +1874,9 @@ void __init free_area_init(unsigned long *max_zone_pfn) } memmap_init(); + + /* disable hash distribution for systems with a single node */ + fixup_hashdist(); } /** diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c56c147bdf27..ff6a2fff2880 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6383,28 +6383,10 @@ static int page_alloc_cpu_online(unsigned int cpu) return 0; } -#ifdef CONFIG_NUMA -int hashdist = HASHDIST_DEFAULT; - -static int __init set_hashdist(char *str) -{ - if (!str) - return 0; - hashdist = simple_strtoul(str, &str, 0); - return 1; -} -__setup("hashdist=", set_hashdist); -#endif - void __init page_alloc_init(void) { int ret; -#ifdef CONFIG_NUMA - if (num_node_state(N_MEMORY) == 1) - hashdist = 0; -#endif - ret = cpuhp_setup_state_nocalls(CPUHP_PAGE_ALLOC, "mm/page_alloc:pcp", page_alloc_cpu_online, From patchwork Sun Mar 19 21:59:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13180634 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F15DC76195 for ; Sun, 19 Mar 2023 22:00:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D1C2F900009; Sun, 19 Mar 2023 18:00:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CCD0B900002; Sun, 19 Mar 2023 18:00:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B9489900009; Sun, 19 Mar 2023 18:00:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A5C15900002 for ; Sun, 19 Mar 2023 18:00:47 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 62BFC1603CE for ; Sun, 19 Mar 2023 22:00:47 +0000 (UTC) X-FDA: 80587018134.03.8C95737 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf19.hostedemail.com (Postfix) with ESMTP id ACCA61A0010 for ; Sun, 19 Mar 2023 22:00:45 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=fTcFp2H2; spf=pass (imf19.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679263245; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=p9nYhEji8q09j2pqGE1YEx7t+yWFyRXv7pJkAKXalmU=; b=AMwerHt87yOZNUaWIPKvxOhw8WwbLctfJKYISOulmmuvluLG7yapmmj0p2hgXCZUMfo/CT PjdHtWFs6qHLsmp4xKTEwUX5v5J09nBO7V4xeAbZnFCfQfljaA5CRg43k7afAaqrw2YTO5 NI1/aB4ndDkqNJDqecNIOxW1u06k9c4= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=fTcFp2H2; spf=pass (imf19.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679263245; a=rsa-sha256; cv=none; b=qLMxRFa0Q/9Xepo8w39RaRa8i/gOfrKLqCR3ZRlypByTy2fyllUrU3BppmFOvUqtuuzFqI t1omus7or4ze3A4TY75hqUcqguU0TKcm1fSYzwTG/5V58Ms6KQAV0zaCXzMxly+/BT4y9l RsBco5mEyk/qhK3ej+SJK4dSOLtJ7+g= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id AF0DE611D4; Sun, 19 Mar 2023 22:00:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 80A58C4339C; Sun, 19 Mar 2023 22:00:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679263244; bh=IbslJwQtSp0K62Trj6TOitHikiCkl9yMUWR56UtzbLs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fTcFp2H2uu/R1vYPaYke53LsDbKWDgvSDvKi6CPldoQ/4qufDzX2VtUQ0zZ1Z/CFN NPPheSGYydZPKT/k0QdXU4FJq87qVBloqwv6g+zJzdsOpy64g6V1HlZIqFHbu5kUo6 MayyWZ9S797/6gOCwr9bEPdKm5KJEZG/tbH4xglxgNkQtyk2r12nS9dYcTgCRKmoHJ /CvzMpSJyTiyHCmEp2VDwtwLtMIjMial4FQA+bZYa9DcemqT0CXTyvjrnGzrs+2T+3 9+Bg3zKcIhgO0QtCAMyQkyUVGz+4V8M3zxYZCvAmlLvmuz8EbA26S/KcwArdLNVJ2r s5tv4IVeWfZ4g== From: Mike Rapoport To: Andrew Morton Cc: David Hildenbrand , Matthew Wilcox , Mel Gorman , Michal Hocko , Mike Rapoport , Thomas Bogendoerfer , Vlastimil Babka , linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 06/15] mm/page_alloc: rename page_alloc_init() to page_alloc_init_cpuhp() Date: Sun, 19 Mar 2023 23:59:59 +0200 Message-Id: <20230319220008.2138576-7-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230319220008.2138576-1-rppt@kernel.org> References: <20230319220008.2138576-1-rppt@kernel.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: ACCA61A0010 X-Stat-Signature: 9nc4u3rmjc91eoabe4zcwmwtpxpuk43h X-Rspam-User: X-HE-Tag: 1679263245-753617 X-HE-Meta: U2FsdGVkX1+euMMWhjAYViXWOiAp2mDGAtpEU1uDAjqGlg+42xNm6JfnFxv0vbUc59lGMXSfP5g/2pO021FXNWtvZnJr7q1XXyOlo3Xsfo/6QLFtGqouJpnpPeicdmggGzfy0M2iHl/5wz4Uxb8Wfnov6a39NW7fm8BcklU49gaF+OAjOPCJUc21OB1O5Mi0lcpHZptqxVW+SrpbX1hTd/J3yBfT3Yt7VL0vo2OYy2R6Aj5hHulBOyXicbVaYIjVovijQD+8cNa1W6MPDqMxpBHtNDRDL4UqlfSgkipm8revgHanoqgUcoeMZD7gkImlMMMHW1swBX+4j+avhqJ1ZdRe2cLv2M3Te/txiwBXf+aZ+Nr2TyQmqZz5wCkVpA4ylV4ARVIek+3BIQ/VbFkKVqIxDnafPP6wjDBSK7RhZENcWz46oWGpHPcGsIMx0vmNGcQ9I/2sKa6jy85hp4ulGjzjKIyLUgnW5qxtqTtmjuTQaTMRVnar/lXAyY+GvRo+ThdtMrokodplVP0kv7rQzt/YZWFMWFAiapc9WuTp96qufgY+WhmM1p41wKw+F0sAU7ueEYIwyoYHv+jJ1syQCLWzaw9gJ7MwqI1Hnj9s89I1utwyaY2dYuuY6wMEfCTpep81n+7zjuqvS5XanNPt0XAgvcb92aMlPy7xsI0HXmCDD8x3WWaVsQMPJ4syKU6FcURCoDx3FVcRBUQIdou1GveSFV6V9cgmEL1PfakOkDTgQ6EC94kvPlkuasdQRu0UhWQ9aZGtggF/NSLnIeHqQ+QYwAQm/qrswHdVwogrsmEmwu7RRYaUAWitXi99/2hfu39q6ZIFdOyV3kIs3pNmCFxG2y16bXwWkqk766NMPBm3qtJ1KzGmveML63n+1q/RBh9r2ge76+tRqPO5LEgelcF5OZI0nUtGMco48s2eiUH5go3V+c1tjuPWw7wbFtQK+k0JeflAYwVBkteyxjP OPG25nB1 rCiTbarb7YCzxcsGfcaXkqHgaZnCZgSVrWW5uaMPlP2abtGn1mfnrM60uEiyXMO5yGCUmK5wmf8L0Z6HKhpcO0zJqKnqsYFNM3+1PTxtgx5msS+OoHEIUMFh7dAIa2adEYBXbptYZzz+R4rvAix4ywnLaX8YbF+8VidKWotk9P1TRAFeBEijRYWDzohX8mUvAGwTosdoTdyDJgwkjegbYRQAbTZWYAwoRPDXZO1tgKpwo8k8OsAPY8oUYZuF7JwAbz9VUv7LpEmKLcSs0q79bGjzK4g7z8t5go7beykxC5A1IRNCbYtV21zgcwKJBt4FyWEhlqybcTFwDjcJdS5nxTN/RTLhGzxVTReq6oHAeyKMvc0VHn/ffQD652KEUOMYRqhHtRJ1viKB45nWz2x097HpV/w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Mike Rapoport (IBM)" The page_alloc_init() name is really misleading because all this function does is sets up CPU hotplug callbacks for the page allocator. Rename it to page_alloc_init_cpuhp() so that name will reflect what the function does. Signed-off-by: Mike Rapoport (IBM) Reviewed-by: David Hildenbrand --- include/linux/gfp.h | 2 +- init/main.c | 2 +- mm/page_alloc.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 7c554e4bd49f..ed8cb537c6a7 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -319,7 +319,7 @@ extern void page_frag_free(void *addr); #define __free_page(page) __free_pages((page), 0) #define free_page(addr) free_pages((addr), 0) -void page_alloc_init(void); +void page_alloc_init_cpuhp(void); void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp); void drain_all_pages(struct zone *zone); void drain_local_pages(struct zone *zone); diff --git a/init/main.c b/init/main.c index 4425d1783d5c..b2499bee7a3c 100644 --- a/init/main.c +++ b/init/main.c @@ -969,7 +969,7 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void) boot_cpu_hotplug_init(); build_all_zonelists(NULL); - page_alloc_init(); + page_alloc_init_cpuhp(); pr_notice("Kernel command line: %s\n", saved_command_line); /* parameters may set static keys */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ff6a2fff2880..d1276bfe7a30 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6383,7 +6383,7 @@ static int page_alloc_cpu_online(unsigned int cpu) return 0; } -void __init page_alloc_init(void) +void __init page_alloc_init_cpuhp(void) { int ret; From patchwork Sun Mar 19 22:00:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13180636 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67AF1C7618A for ; Sun, 19 Mar 2023 22:00:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0995A90000A; Sun, 19 Mar 2023 18:00:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0491B900002; Sun, 19 Mar 2023 18:00:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E538590000A; Sun, 19 Mar 2023 18:00:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D7A21900002 for ; Sun, 19 Mar 2023 18:00:52 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 7C9DFAAC81 for ; Sun, 19 Mar 2023 22:00:52 +0000 (UTC) X-FDA: 80587018344.23.72B1ADA Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by imf02.hostedemail.com (Postfix) with ESMTP id ADF9C8001E for ; Sun, 19 Mar 2023 22:00:50 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=SGon4fha; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf02.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679263250; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=z6BzX5RSxqlEnHYkDBKrxJnZmz/tx4Z0P6dotv8/xaQ=; b=BbCaNeuGyyAUUgySDGjq043j4GyktJRyMFxEwQMMVYZ3KhK/WxR3GY/Mazy3cwfaNdVcnT 6CN/2F5TQTFXdkDFEGLaIpHGM6zqVeIaWax3UAUB42Y2EOls+KVSYbKDO5wKAOJSfznZP3 u6mGBiQgm1kOnZPjXlW4CPSLCcCY8BE= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=SGon4fha; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf02.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679263250; a=rsa-sha256; cv=none; b=URxYoYrUhr8FczVw4qEIg4UqEvTSFCsKtxkkpAKd6Siqvo9I9tksYMPBnsaGC+SgVSXN9n b7QBZSqh3oHiVDh2SEJiYLz2YV6hnrd7WnxqdpFNW0SMmfo2Dqv47kWWN5blBGt3kB9YTa nCF4T5n6j8TXRa8QN3DTh9Tiz18+UNM= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 6980FB80D29; Sun, 19 Mar 2023 22:00:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 17416C4339E; Sun, 19 Mar 2023 22:00:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679263248; bh=EYqScWvKSkV7BdC+HRlB0XJbRdBQ734VVz9MaIm+3GE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SGon4fhaJq517bPBRcHKmjkvN4o0CTIjBqWQ7HGujGdtR6/e1N65HwBjOyIcIUVW9 tDfA6jl/oGozUz9exf6JhuiAK1fRBaCBLBZQwLiZ6h6cBOSPk6EgNww8ffo54jFMCk 99TUCf1bgX4TdLghfR4z3OIXg3Iv6YP91TQyYK/LW3xkr0CljUk1W0NbwqGx4JNsKf IMjabdz34G4b94oIhDLS1kgx6k3Ha6F8ooBAOwkWk8rfbIj+e/NrR0W6thXB0OF3U1 lm9woc3T1itzI8f9Cw7TK6yWn0iU506wYbkiU34TTT9y3cf4qZIVRPpsikb9koDCx5 1PVwJYaQ236rA== From: Mike Rapoport To: Andrew Morton Cc: David Hildenbrand , Matthew Wilcox , Mel Gorman , Michal Hocko , Mike Rapoport , Thomas Bogendoerfer , Vlastimil Babka , linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 07/15] init: fold build_all_zonelists() and page_alloc_init_cpuhp() to mm_init() Date: Mon, 20 Mar 2023 00:00:00 +0200 Message-Id: <20230319220008.2138576-8-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230319220008.2138576-1-rppt@kernel.org> References: <20230319220008.2138576-1-rppt@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: ADF9C8001E X-Stat-Signature: m4157i5gg1bjnb3385i74miyip5m4kb5 X-HE-Tag: 1679263250-348993 X-HE-Meta: U2FsdGVkX19+UTEusIu9HMdyuue1mG5VyHUKe8qzEOGRDobohASiKNB190gAdGBGPJw9aw4ooiCts1CCBt/7oI71esRLa0gHzwyncpXY9OOO4ACBk7pf/9g+NVdUZHS0SOZTEgW2/JclUbB4wU+wkDxecQsM3wmLWciz7Y5eyiPuAiWElrSxjb+VHGWwqFAg5vbsxWiHVhyc6biBGE5b6BEUIQx3ff4OWE9q8vZNv+TSoMU0DqRhVo9e9nwuJfbcb0I0fgFV/N/9YcsP23FxFsPI3MK4uXjVqOzvX/Jx6eyGlt3PXY8C0PpnlQ9SQb7nh3/f0qMonC/ze5SlLWYe3w4vfZIgBFsmzRO2sTuUpqbIkdOp8KQVp2k2bo3KmEBiaPmLnf8tNb9W8dzEpQ29r9qYrwp7IrOIFn1XqQA7WzVPuOOaqPvzbt4j+fExwgCetWlDIWd5PwZ4y8PzSRoe5qvzHZByYDtTNdXOovSNbjqW8S0dCqCvjKOgjlipgJzYJO1asF/M02E3b9IBR7QTwGHuj0oW0kXnBpfMVS3ItXW9BH0/beDyH6cdMf7UMk1uWfi2i1jM4Q+OnY3ndpVxwoiXSLRAdcQl9M/rxoaC2wDRxo7pvT7RA0rHb3uH5HmbSRMpCCv9gP97h09wHRFotScbdCAcsIHhD68IFSAmCGZdiszLWZieymA5wA0DRRAT5R+YPbfaUNNW2NeanAOX0Dpu+1j16WNdb3hxIoJpxuZmbYSPYVhRrTYPwiJGfA0z0QVqpIf8DlDKX+InaZTOHWA3ZAwJXvLd/LVJ1DWzMUturjMD3DyV4ELkDCIyrywlZwpn/Gdu1j4A2+7iiBqNlGVSP6YknxVyrJ7zRrB/D5HP7UgKjvIdCgA/mmtesVUZffIulG38hpPvEE+yvzLTZ8KATzCs4RzDBXs3k/8he+/DgdUtXSksE5EutbsWswFpMQdt43GjfPCE4XZ2xOM iWJXoy+d 12uQ17I3I7uKzAKpAIfq2FJDx+5dmf+cqwwgASEPG6W4Viq7DrwBqPU037YcM3X15Z/wX6AfVe0IByYe75W6EJgjOYXJwvUjAUs+qfuq58Fa8WZhD5bx4vG0TaOO4sI19UsPLfT4rsG9veM2ukhbEOdvXizgsYSWLFbyfnL0BMHX4hDvGMf4Ust/OqZ+HZVrbE9Bb/yf5ul/JvQm3fEDmgjybebWDlnZuBXC/fBiLvoB6s6LgsWjji3JuB8Ltj6cqCNNL1rVsC93043b/I82DAH6LVxDT71t0KvD2Q4Hh0V+MVRkRqIg8XlVsfN3uh7FKMekk4lbUlqkpr0P1qnqSHM4Wz83uEyV4OUSdJkD/sShkvfLGqAxiDNRpek3BHtEQ0NVnCBqTz+IOKQitXCPjx1wmcg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Mike Rapoport (IBM)" Both build_all_zonelists() and page_alloc_init_cpuhp() must be called after SMP setup is complete but before the page allocator is set up. Still, they both are a part of memory management initialization, so move them to mm_init(). Signed-off-by: Mike Rapoport (IBM) Acked-by: David Hildenbrand --- init/main.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/init/main.c b/init/main.c index b2499bee7a3c..4423906177c1 100644 --- a/init/main.c +++ b/init/main.c @@ -833,6 +833,10 @@ static void __init report_meminit(void) */ static void __init mm_init(void) { + /* Initializations relying on SMP setup */ + build_all_zonelists(NULL); + page_alloc_init_cpuhp(); + /* * page_ext requires contiguous pages, * bigger than MAX_ORDER unless SPARSEMEM. @@ -968,9 +972,6 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void) smp_prepare_boot_cpu(); /* arch-specific boot-cpu hooks */ boot_cpu_hotplug_init(); - build_all_zonelists(NULL); - page_alloc_init_cpuhp(); - pr_notice("Kernel command line: %s\n", saved_command_line); /* parameters may set static keys */ jump_label_init(); From patchwork Sun Mar 19 22:00:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13180637 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3D9BC76195 for ; Sun, 19 Mar 2023 22:00:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8FD4190000B; Sun, 19 Mar 2023 18:00:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8AE59900002; Sun, 19 Mar 2023 18:00:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 74EC490000B; Sun, 19 Mar 2023 18:00:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 6827C900002 for ; Sun, 19 Mar 2023 18:00:55 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 1D1E01C5D9E for ; Sun, 19 Mar 2023 22:00:55 +0000 (UTC) X-FDA: 80587018470.24.1F2BFF2 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf25.hostedemail.com (Postfix) with ESMTP id 26EDFA001B for ; Sun, 19 Mar 2023 22:00:52 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="CQj36/c5"; spf=pass (imf25.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679263253; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QlBbim3xnh0fjP4Sc4K6+MacYz8bYA1jMrCNMhhDe04=; b=fzLca9dbQtuOv9mc6EsmhpH59O1DvftNvKp3xR2Bk5tdQJCNO+ZMAnpu8Tepu9s71N5kyp pjZiwWEjhVq6XleJ1atYbrCXpzC40RdmEdeRtk8bx/lXh1Df6zqwFac7CpoeU5Q18e1AyA 1/KQqK3VKWvqYxbJKjqM5c66CJKAioI= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="CQj36/c5"; spf=pass (imf25.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679263253; a=rsa-sha256; cv=none; b=4TR0VxZgciiFZatuGkyAhXty2ROpp3n+X9dK0DZBDkrnf1KqiOWKNjYRGY/kHD9hnQ+4uw zqgsoqTtW07UJEygm8zRtgJ/ppziTSnxvba0NIMMn5gpYaXEkTYsaHok2SNjYlaBIFRplv htrFTS68gIbRn7CL3firLCRYakMNP5o= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 41D38611D2; Sun, 19 Mar 2023 22:00:52 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A29E8C433A4; Sun, 19 Mar 2023 22:00:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679263251; bh=uV4gJFJoIVTV9htd8RCrOaAU9KLl8DLHB7MDbKZkC0g=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CQj36/c5qv8IcPiOqJ0xZNbpyBo/SK2XmaGkWi+IO6y2dPZ2tG4ua/h6o0SS/c+iF 0scf0+O+3sWRH8CDsPGEWYBh0quOjdP03gCVFOwKETutJR55v2nNmNqpEHFpR5zChI 1oUfHaGThm+jLhVeBkiV9A1P7wUoSIpSKGDiKp0ke6ltYYwZr5j00FKAoLikFoEcU1 wnkOmpjfpc0dCyEUH9/+HUtbM6Yo7gNsfxaQaZ2QWOT9/HoSzsa2m9O+gLhzRdh2a5 xckfYrHXcat9vF3ZRBqqDW3w9roV5Vk7ifLGQNVxGpAr/7YgP5GHwymALkop8292BD 3SB+xrnbB62mw== From: Mike Rapoport To: Andrew Morton Cc: David Hildenbrand , Matthew Wilcox , Mel Gorman , Michal Hocko , Mike Rapoport , Thomas Bogendoerfer , Vlastimil Babka , linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 08/15] init,mm: move mm_init() to mm/mm_init.c and rename it to mm_core_init() Date: Mon, 20 Mar 2023 00:00:01 +0200 Message-Id: <20230319220008.2138576-9-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230319220008.2138576-1-rppt@kernel.org> References: <20230319220008.2138576-1-rppt@kernel.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 26EDFA001B X-Stat-Signature: c7gt6f516mmfzbdjz1zp8oyqy1zc4j8s X-Rspam-User: X-HE-Tag: 1679263252-933086 X-HE-Meta: U2FsdGVkX1/WJxmU+UTHGRglozmZIo2ekdu+0/GEQLo0jtE2amQUmbng0yiFTkM3ouHk2Uwap9ayYPykhYJxgLVrk2+XtR69hX32uoK14WU68Pz63QpdLpkdaHKurGM/Dmp9X/9ARc/o4AdoZl5ZKlJlw3po4/zUbE9kbfHMf+Q1uyQjVgJs6ZzSthmY5DNtU+CL8WlGJRwHz2uhmOtL9RTP4i0hmIC6ObIHCBsrfAGpM351hVmgPwOAAeDXnwHNd4hgwKNm2F0sH1a+GOawd0u777vo2j1bsPbNVNWX/MIln0giWjjOwVyL+hnyu4XYXw0vRzw0rtwGVFHau7MIkh6Wr+zhqeS4ZabofUzXHbtzFYeJr1oqdUqoVJkkOFwy+AdpgOAbNto2LmsBrjkCn37Id+YKIcFwGfV6chzkJFdSYbj+cUBXdVDWYTKjxsY/1VcDG6GpwZJsv++jZYMikCC5goTMDNgJG9urp5piavZE9VHa5IhXrAUT8NmTY7jEX+Rc0duzEyfmrNnREnnZ5NoGqobcLNix2LQFr8K6ZjTMf7k9YVPr7Jsw/LhEX/WpognymQDX5NxXbCFoe5YaOIHk1vzfoP5HMz5KUqlAu6VlaTmNKbv9yppu+97wvZAUvAWJfmJdekvS5leqaNhXqCTNi4UWXswGE6611yYsSpljMCB6r3w8d5piJWpjVZViQgiW3Vap+loo86dpz6RlPDsBiEsjBb2jyUs4yBETDFPR7BqVbkAOcZ+d2kGZHEoIc418CPblxwdUjPJJIVcS+hO8lAtfOOMjepM9jsnFGkW4eYp9iXwaK+PnnQ5CorfG8PW/RXJIlNUdZFUNUuUH7rjElX2yghQS4L5Z44aAFN7y9p9hrCZRszd1ki6j97kN8Ys8Vz8a8x4S6l8uHQEuXXcxKJeniANmVrSCJ6ak+cQBsMvfRPvbh3Kor0pOjsoPHz2GBgrq8CWu5KxhD+h WOhEa9KJ FHixkCjIAEwtRNP6yZtQa8D7BtWAQMG+LAayIluOnY1bxXc8/NslRHvyhRyp4Y9NvrEf/DSOZbl4fsEg10f81AzLgGVB7fG5lUzsVfuF/JeAJX7ySnjlsDDV3+hLU8HDab4qXTOtHHraf9NmQ59Ac91IRQgEGxrmiAPdutw51VGPd3PEpPRSEvKIugYDJQpjZKfAaeUmLOIvavZZkBtk5ukG0JOe6I6jMlaF9cCOy9g2X1671SvoRXKM8uNeMNQ96mnrr95Vm+aqjUIYkbJK7G7ky6pGGVnC5dHnG8kSDI7mDwkE7KeTv64hHnlXMPHREZEqzknsr4NDKB/Gzh+BKGEJCQAJLdvKujMmXZ702Z6DbMEYc0B17w1nndUivrMgElXfbpIn9J9EgoVHVleVWcMgxNw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Mike Rapoport (IBM)" Make mm_init() a part of mm/ codebase. mm_core_init() better describes what the function does and does not clash with mm_init() in kernel/fork.c Signed-off-by: Mike Rapoport (IBM) Acked-by: David Hildenbrand --- include/linux/mm.h | 1 + init/main.c | 71 ++------------------------------------------ mm/mm_init.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 76 insertions(+), 69 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index ee755bb4e1c1..2d7f095136fc 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -39,6 +39,7 @@ struct pt_regs; extern int sysctl_page_lock_unfairness; +void mm_core_init(void); void init_mm_internals(void); #ifndef CONFIG_NUMA /* Don't use mapnrs, do it properly */ diff --git a/init/main.c b/init/main.c index 4423906177c1..8a20b4c25f24 100644 --- a/init/main.c +++ b/init/main.c @@ -803,73 +803,6 @@ static inline void initcall_debug_enable(void) } #endif -/* Report memory auto-initialization states for this boot. */ -static void __init report_meminit(void) -{ - const char *stack; - - if (IS_ENABLED(CONFIG_INIT_STACK_ALL_PATTERN)) - stack = "all(pattern)"; - else if (IS_ENABLED(CONFIG_INIT_STACK_ALL_ZERO)) - stack = "all(zero)"; - else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL)) - stack = "byref_all(zero)"; - else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF)) - stack = "byref(zero)"; - else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_USER)) - stack = "__user(zero)"; - else - stack = "off"; - - pr_info("mem auto-init: stack:%s, heap alloc:%s, heap free:%s\n", - stack, want_init_on_alloc(GFP_KERNEL) ? "on" : "off", - want_init_on_free() ? "on" : "off"); - if (want_init_on_free()) - pr_info("mem auto-init: clearing system memory may take some time...\n"); -} - -/* - * Set up kernel memory allocators - */ -static void __init mm_init(void) -{ - /* Initializations relying on SMP setup */ - build_all_zonelists(NULL); - page_alloc_init_cpuhp(); - - /* - * page_ext requires contiguous pages, - * bigger than MAX_ORDER unless SPARSEMEM. - */ - page_ext_init_flatmem(); - init_mem_debugging_and_hardening(); - kfence_alloc_pool(); - report_meminit(); - kmsan_init_shadow(); - stack_depot_early_init(); - mem_init(); - mem_init_print_info(); - kmem_cache_init(); - /* - * page_owner must be initialized after buddy is ready, and also after - * slab is ready so that stack_depot_init() works properly - */ - page_ext_init_flatmem_late(); - kmemleak_init(); - pgtable_init(); - debug_objects_mem_init(); - vmalloc_init(); - /* If no deferred init page_ext now, as vmap is fully initialized */ - if (!deferred_struct_pages) - page_ext_init(); - /* Should be run before the first non-init thread is created */ - init_espfix_bsp(); - /* Should be run after espfix64 is set up. */ - pti_init(); - kmsan_init_runtime(); - mm_cache_init(); -} - #ifdef CONFIG_RANDOMIZE_KSTACK_OFFSET DEFINE_STATIC_KEY_MAYBE_RO(CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT, randomize_kstack_offset); @@ -993,13 +926,13 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void) /* * These use large bootmem allocations and must precede - * kmem_cache_init() + * initalization of page allocator */ setup_log_buf(0); vfs_caches_init_early(); sort_main_extable(); trap_init(); - mm_init(); + mm_core_init(); poking_init(); ftrace_init(); diff --git a/mm/mm_init.c b/mm/mm_init.c index 8aaaddd13a20..1da48762e4a2 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -20,9 +20,15 @@ #include #include #include +#include +#include +#include +#include #include "internal.h" #include "shuffle.h" +#include + #ifdef CONFIG_DEBUG_MEMORY_INIT int __meminitdata mminit_loglevel; @@ -2504,3 +2510,70 @@ void __init memblock_free_pages(struct page *page, unsigned long pfn, } __free_pages_core(page, order); } + +/* Report memory auto-initialization states for this boot. */ +static void __init report_meminit(void) +{ + const char *stack; + + if (IS_ENABLED(CONFIG_INIT_STACK_ALL_PATTERN)) + stack = "all(pattern)"; + else if (IS_ENABLED(CONFIG_INIT_STACK_ALL_ZERO)) + stack = "all(zero)"; + else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL)) + stack = "byref_all(zero)"; + else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF)) + stack = "byref(zero)"; + else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_USER)) + stack = "__user(zero)"; + else + stack = "off"; + + pr_info("mem auto-init: stack:%s, heap alloc:%s, heap free:%s\n", + stack, want_init_on_alloc(GFP_KERNEL) ? "on" : "off", + want_init_on_free() ? "on" : "off"); + if (want_init_on_free()) + pr_info("mem auto-init: clearing system memory may take some time...\n"); +} + +/* + * Set up kernel memory allocators + */ +void __init mm_core_init(void) +{ + /* Initializations relying on SMP setup */ + build_all_zonelists(NULL); + page_alloc_init_cpuhp(); + + /* + * page_ext requires contiguous pages, + * bigger than MAX_ORDER unless SPARSEMEM. + */ + page_ext_init_flatmem(); + init_mem_debugging_and_hardening(); + kfence_alloc_pool(); + report_meminit(); + kmsan_init_shadow(); + stack_depot_early_init(); + mem_init(); + mem_init_print_info(); + kmem_cache_init(); + /* + * page_owner must be initialized after buddy is ready, and also after + * slab is ready so that stack_depot_init() works properly + */ + page_ext_init_flatmem_late(); + kmemleak_init(); + pgtable_init(); + debug_objects_mem_init(); + vmalloc_init(); + /* If no deferred init page_ext now, as vmap is fully initialized */ + if (!deferred_struct_pages) + page_ext_init(); + /* Should be run before the first non-init thread is created */ + init_espfix_bsp(); + /* Should be run after espfix64 is set up. */ + pti_init(); + kmsan_init_runtime(); + mm_cache_init(); +} From patchwork Sun Mar 19 22:00:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13180638 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA09DC7618A for ; Sun, 19 Mar 2023 22:00:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 68CA690000C; Sun, 19 Mar 2023 18:00:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 616A4900002; Sun, 19 Mar 2023 18:00:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 48F6590000C; Sun, 19 Mar 2023 18:00:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 39D29900002 for ; Sun, 19 Mar 2023 18:00:58 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 004781203EB for ; Sun, 19 Mar 2023 22:00:57 +0000 (UTC) X-FDA: 80587018596.27.FD9FBE5 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf26.hostedemail.com (Postfix) with ESMTP id 38999140021 for ; Sun, 19 Mar 2023 22:00:56 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=NyEJQwJp; spf=pass (imf26.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679263256; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IoX+IRqK+H09jdYfad/gWP6Mte+ccY5x17RdUvvK5zE=; b=Ig4aZkxUzSQqNeWOi7CF9FSSs6/iGbdlPejWazYGpUHsMAv9czrjcgAyynxBTEU77yldIX x2n81DW+CIKmN0YnAN7noBrpE/E9tUiUpB36vd/7YpSKEKA8NhwXT7Ll5Qqk0O8rYvBUR8 9Q5H/NRM2MX4GpCdoccUxRWK474r46s= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=NyEJQwJp; spf=pass (imf26.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679263256; a=rsa-sha256; cv=none; b=S3ls94gr4T3ea3romvHp2zJTCQ5Z7stDAPos7xO1U8FHpLDCrMHkotfReOrTdeMPKRYpFQ ilC+2S5r9urYyqnHA4xxHhBZvXMpraQcngHlpkq9NHSuN9TH3sChrZPob8O3i3fftfBFgs VubeG4QoIkXCZRIzj2ZaC7zR/8MhE2Q= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6CAE9611D3; Sun, 19 Mar 2023 22:00:55 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 396ADC4339C; Sun, 19 Mar 2023 22:00:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679263255; bh=Y2ZZ0QRBPTa6+gDvoBKxwGRBX0o95+AmHfq1WGe72cg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NyEJQwJpkU2SMv4Y3ZZ45lRC4TFwcsj5YD9lknG7TIGUO6XHl7Tsz5bSI+MiTUkVj CLIkYngq4kK29V2ziLnjs/6syWGxq+b5U+Zq0OqwNgyVx22M/GMBKXltpZ+pjGAH9W gZtDiEsLitP9aR+nNb8lJ0kk5Kdw7vqWjGaV1Wa6EmugEersPCV/IF8uZoCnloJ2OT GwOkl1T42rTo4uq0GPZVgA8gnWyneA5YN4+FIZRVcxYPr9bnrk33nluhaurvCn9GEa q77ullFT+nmwLLP+mQ8+DUqLF1x/kNgJNyy9+TLFNiIGWc4SwzeW8WtpoVr7210dDx 45oeEcYLogZJQ== From: Mike Rapoport To: Andrew Morton Cc: David Hildenbrand , Matthew Wilcox , Mel Gorman , Michal Hocko , Mike Rapoport , Thomas Bogendoerfer , Vlastimil Babka , linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 09/15] mm: move pgtable_init() to mm/mm_init.c and make it static Date: Mon, 20 Mar 2023 00:00:02 +0200 Message-Id: <20230319220008.2138576-10-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230319220008.2138576-1-rppt@kernel.org> References: <20230319220008.2138576-1-rppt@kernel.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 38999140021 X-Stat-Signature: 3xck5n8f33mw8bgecsy78844s7wxzkkc X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1679263256-63518 X-HE-Meta: U2FsdGVkX1/EbcJRN2qwggfMH3V4Y0YKJO31F1Pu7cvbhxdOiCHPj+VW7HE5NGA8i+LTvmPhQfvdmRVUtIaO6y+Dt7JnY/p7XXerCZeC0Qo8o7M/vD5NECF5WdlmiGzEG4IusOqLZZVfOB2jqXXAVoGHmIVEYxPJJ/bG8okV97XZ+b2ceFx1EzN1S/MU8E/n6zLuTCG/uPqvEjyCOt6gkzsmLkomHj3lEEwlbNgPAV4SU5CM+Lbj7H5IS/QgsD3DcToasdm415x3XS/RZo2tClfrfS/k2ScGdy6uYpvlSZPtr5W7Inrrffsw32/G93eroVr0Y1zYXJ0QHzR3oQ8Cehwilhp0AGmXTXgdMzC/JIoFTx6R+h23MRkuKQ/belU8lyolTuRCbVLQh5NaHrsSMRoA8BeqphdwanN6RE23qXhh5Z8+QPSxLZ4+CWIv3+mmfEPQJLNYZyj0mgc60gi2aHMq59V9J+Ca6sj+UzaRdje6jC5duJfPiFAhP+aBAr3MHr19vG0WTfG7lB6MU5A6Lqzh7WrQSZyBa/+yxVscEz1nS1IBKAZ8rKFB7O2AOFX/dKs3v1qnfltx7UbieUGB8Xz2e5ob/0bk13O6Lwz0TegdgkEXTPo1yCkSTLr19PljDkOKvcnctytSQFk6mXlgVpUijWp8w23NNxy1XgVXZFh7KwW7D7kO9qJibNj9qIp7nVrmUrLe6RXiFbWPtd6AYIH9Ogjf6HKu9iOtdPklNOTEOa2/9suMcQE0cxZf3epYPoUR2dAyaN1XD8io0qsOruNjfAVroTDR7Qpp323BUIVk2uBDMY2er8DdPdiYu0jN6WVdzYGRvccD2iGfEtBhcYshEPCx2U0hZf/mahL5Upd+MaAuEZGYyD9iBDVZylLthm007E5UN+yIWy7Lo7n+SJJ1j+XFI+GgE6DATxhzLMdPzk7RiL3Tv+6noi4vPbd8dWn8LvSS8jkuGFLRl94 RMDXRawo OTPI7QxcHVm0afQ+UZ60QWCmw3W+zIkYyE+2nHb2gq6kk9Lg/gaR9ypENe+av4WxIjzGFvBxMb5ME+PI1M6dzabWcwboJL8i8VL6lVPzYSCylLGLeFzOX/VxI5nX8P8D7dwLoOom+OY8hLor3ohPdL6KF+6/9WCLNyC05hQquc+A/FKWU5AzjWEp2AxeNua69VnXfJBqTX1ac0lO5o+AFK0azvF6xpTJxXAa7cs3zu/0TRBEytyFTXevbGbKMKg0dDP2GoWCn0ip9mjOk0wfn1EV1Xk2OO0FRwD69a1ywwF60VQvUXXEsEy729ONXvdtf8i0gGNTBJfEhbbO2fiXLmmSajG6J/RFQTc31JLmdxjp8MkM50xhGtTDNgLLrZB4kfEDYASWkns6dnPMGIVvgYPr+yQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Mike Rapoport (IBM)" pgtable_init() is only called from mm_core_init(). Move it close to the caller and make it static. Signed-off-by: Mike Rapoport (IBM) --- include/linux/mm.h | 6 ------ mm/mm_init.c | 6 ++++++ 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 2d7f095136fc..c3c67d8bc833 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2782,12 +2782,6 @@ static inline bool ptlock_init(struct page *page) { return true; } static inline void ptlock_free(struct page *page) {} #endif /* USE_SPLIT_PTE_PTLOCKS */ -static inline void pgtable_init(void) -{ - ptlock_cache_init(); - pgtable_cache_init(); -} - static inline bool pgtable_pte_page_ctor(struct page *page) { if (!ptlock_init(page)) diff --git a/mm/mm_init.c b/mm/mm_init.c index 1da48762e4a2..a91fbb57c4cc 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2511,6 +2511,12 @@ void __init memblock_free_pages(struct page *page, unsigned long pfn, __free_pages_core(page, order); } +static void __init pgtable_init(void) +{ + ptlock_cache_init(); + pgtable_cache_init(); +} + /* Report memory auto-initialization states for this boot. */ static void __init report_meminit(void) { From patchwork Sun Mar 19 22:00:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13180639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C289FC6FD1F for ; Sun, 19 Mar 2023 22:01:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5CC0990000D; Sun, 19 Mar 2023 18:01:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5793E900002; Sun, 19 Mar 2023 18:01:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4419490000D; Sun, 19 Mar 2023 18:01:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 312E7900002 for ; Sun, 19 Mar 2023 18:01:02 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 05E921A038F for ; Sun, 19 Mar 2023 22:01:02 +0000 (UTC) X-FDA: 80587018764.05.8AFA439 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf13.hostedemail.com (Postfix) with ESMTP id 55C2120003 for ; Sun, 19 Mar 2023 22:01:00 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=N5K4QQ46; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf13.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679263260; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dskQn75DzrxnhRwhxI4mFwsVHjHCHdlqm67gcRhARlY=; b=QQOF+SxjsZpWfz+/Xj9YbXnsBwAmhJpkw6fbzUVd5eLCIneIkYzBy6QAbu6+3FMqMWIRIl 65v+iJh4Xt34bYZIzBquYd9q+rnvgYFlB5yC9wZLdUlSbhqHqfjPwDYIupPsKAF6A4YL3D 4Ii17J1a7+pkFTfHYZVAzhb4Ag1ErzE= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=N5K4QQ46; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf13.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679263260; a=rsa-sha256; cv=none; b=d/Tn1mHBS8nv9MWQuTBGoVmetsHcwYujdAb0+Pwkm8Q5ERkThEjyK55Vct+e/fzaDdTMrk T/ieOtaj6Io9u3/KrvV+HNjX1sIKjFxb8CYjrzNM9JGqOLhEHFOQEVJbsFVBpwliKv8d38 zNLAVdX04qbBL9gSto+ZCyaFUKWtXdA= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 64A1F611CC; Sun, 19 Mar 2023 22:00:59 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C56DEC4339B; Sun, 19 Mar 2023 22:00:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679263258; bh=Ku/Uu0hCGZmfa3FpxlgXCqadz2KVOyqdqD9q1uKHmwo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=N5K4QQ46qKwNSCgWcOY2eoUQDG6tMOpzrVlfzMqd5KxSDU8D54PeXg7BVDbq88NAy 7qTkcbcGwgHcOiLDqN8zW2dwEl9CQ2/ZNPFDPQEgyWthnciohBW/LqLaGTv5rcaOxa 1nFGfP3zT3Kxeqh8smJtVZwsXNG4Z5pDoKozcP4ncrAfv9xOF6FlBA/J45wfO58IB6 OXsJRt6r7PccybHaqWRB+ls1lMAEIO9qlvxm8l8I46JCYdZ3H/RaViKbjvWqL1pKcP JQTWfTfu4giRjda+PXnQooYJl3x/nDL/M7LYoJjobqh4D/jU1/N5DVZzAEHxc6znBz hS5OotitB1CUw== From: Mike Rapoport To: Andrew Morton Cc: David Hildenbrand , Matthew Wilcox , Mel Gorman , Michal Hocko , Mike Rapoport , Thomas Bogendoerfer , Vlastimil Babka , linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 10/15] mm: move init_mem_debugging_and_hardening() to mm/mm_init.c Date: Mon, 20 Mar 2023 00:00:03 +0200 Message-Id: <20230319220008.2138576-11-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230319220008.2138576-1-rppt@kernel.org> References: <20230319220008.2138576-1-rppt@kernel.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 55C2120003 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 6aniamfaxn9d1pcq57i9iqjrqbpax9fo X-HE-Tag: 1679263260-806340 X-HE-Meta: U2FsdGVkX19TUI3tHw0zyvzXj+rgt35Ox/+NPcL7MIXGXN77DcmmCLsDts4rOU81zGrbarUnQ/nv9hqRfbSosT2yrvSYuKECHqhGLarjZBG8F6eDjbm34+Us6FAdSxKYmnF5x4HjV7FUkmwqaaFn1i5r1zHp3ys2gBz3PPk5/JSMgIc0KEUZMXq7amZnSUNK3FDWDFlBj49swmHeWEgNXun28Uih0o9SQxpDECvEDHAKjsfgpXNry5jetbXAYinbPGSApvEWNdkRw9wdUnBqIu3vRrdUALpbjseLa7duIA5XwD7PH2P+iRjFNALqGlexjNiLKH5tpRXh2+qs12gYDypOt1UCFbGdOYKkMd0ET9pLstd3FYkohExsDg0FTn3fuhiwaFhhFmjkX2aW9Qe5C2EfKMDjSe+Ym6zRjZ8XhLCnJ9GTQIBKin9K2R2sRilSphNYOPZnIB6yj7NEMtrWzG84g5bgW82mrWpzPvmD2LLyy6EtDkkYhbVkA0/vWRKLzc4Y3KaTYYl7e4ov5oFruODkVkgzAa5GU6vd4kGsk0fmupKiALMQh59jPEiBNshr0lTAuefqsXLvN26YaVa2i4l+RMeR3NRua/SYm7q7iPq7lz/PrzkORn/Vv5azamiugNPJeBn1g5Pyb0BKb/SOrdxNzplls6ozrrIvEQlBNk8QrnApUe61WfLmTkahgvNKCrD4cFDW9C0hO4Vrxe7QDy5R9qVse4qj4kKa4DqAao0t2EtocN40AXif9/LxQGLJP6KtOQ8w9K+3UIUBDEXziY5jPtmHF3we1/L99JDqcAVrTi2F7Al0iZVEC8OKAW1I0jzkRPUEEabxmePvhJUGGJPmscr2j3pLx5K1wkB21WASR25PumBmD7sc3S2mazdWvKMvHsutqTlmJL2p/2rS3T7R3Kk+iiPHzelQVVe2eqOHsE7viqIRjDekKaU47w2yFYuAWUVWka4g/WSvzCI wvLap9xS QE3MNpwQSeIPno/slm/DjBXUB//dIL7rO1UE3zzFCML4i7TAaZiKR8B0SWr0ikZzJGJpjkf/fx6VD9MidGZTVnMnqOnqSizFFXL/4GQkzIyUTwV+EwlZpuzW+Ic9nUEHg3vcogg89QMG9cSHG7OAPQxQYbcBHrTYZGVqSwYbLmi8CSllH42UqDEhDFZ4+nU9VZWAM46DOmdhBYzP1qtnv5t+ip/pR62Cm098jpNt46k/wHy0QD7YXmk9LqrmyeD/hJCrYwq/6XrX5F8D/uu4rTydg45vkLConcm2bGRxhvYbR+wiTppPrjdQzia7La/8OHY230G7pgEMXeU/0+zS9NPwA1Lgk6F57P1r3y06qbosGcSUVOk/ZrQEBVAOkEOpHWLq7tCDmd7szm7uj4n/E10mhHg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Mike Rapoport (IBM)" init_mem_debugging_and_hardening() is only called from mm_core_init(). Move it close to the caller and make it static. Signed-off-by: Mike Rapoport (IBM) Acked-by: David Hildenbrand --- include/linux/mm.h | 1 - mm/internal.h | 8 ++++ mm/mm_init.c | 89 +++++++++++++++++++++++++++++++++++++++++++ mm/page_alloc.c | 95 ---------------------------------------------- 4 files changed, 97 insertions(+), 96 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index c3c67d8bc833..2fecabb1a328 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3394,7 +3394,6 @@ extern int apply_to_existing_page_range(struct mm_struct *mm, unsigned long address, unsigned long size, pte_fn_t fn, void *data); -extern void __init init_mem_debugging_and_hardening(void); #ifdef CONFIG_PAGE_POISONING extern void __kernel_poison_pages(struct page *page, int numpages); extern void __kernel_unpoison_pages(struct page *page, int numpages); diff --git a/mm/internal.h b/mm/internal.h index 6b154b4a538f..827499e39d78 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -204,6 +204,14 @@ pmd_t *mm_find_pmd(struct mm_struct *mm, unsigned long address); extern char * const zone_names[MAX_NR_ZONES]; +/* perform sanity checks on struct pages being allocated or freed */ +DECLARE_STATIC_KEY_MAYBE(CONFIG_DEBUG_VM, check_pages_enabled); + +static inline bool is_check_pages_enabled(void) +{ + return static_branch_unlikely(&check_pages_enabled); +} + /* * Structure for holding the mostly immutable allocation parameters passed * between functions involved in allocations, including the alloc_pages* diff --git a/mm/mm_init.c b/mm/mm_init.c index a91fbb57c4cc..ae6bd26cf5a2 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2517,6 +2517,95 @@ static void __init pgtable_init(void) pgtable_cache_init(); } +static bool _init_on_alloc_enabled_early __read_mostly + = IS_ENABLED(CONFIG_INIT_ON_ALLOC_DEFAULT_ON); +static int __init early_init_on_alloc(char *buf) +{ + + return kstrtobool(buf, &_init_on_alloc_enabled_early); +} +early_param("init_on_alloc", early_init_on_alloc); + +static bool _init_on_free_enabled_early __read_mostly + = IS_ENABLED(CONFIG_INIT_ON_FREE_DEFAULT_ON); +static int __init early_init_on_free(char *buf) +{ + return kstrtobool(buf, &_init_on_free_enabled_early); +} +early_param("init_on_free", early_init_on_free); + +DEFINE_STATIC_KEY_MAYBE(CONFIG_DEBUG_VM, check_pages_enabled); + +/* + * Enable static keys related to various memory debugging and hardening options. + * Some override others, and depend on early params that are evaluated in the + * order of appearance. So we need to first gather the full picture of what was + * enabled, and then make decisions. + */ +static void __init init_mem_debugging_and_hardening(void) +{ + bool page_poisoning_requested = false; + bool want_check_pages = false; + +#ifdef CONFIG_PAGE_POISONING + /* + * Page poisoning is debug page alloc for some arches. If + * either of those options are enabled, enable poisoning. + */ + if (page_poisoning_enabled() || + (!IS_ENABLED(CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC) && + debug_pagealloc_enabled())) { + static_branch_enable(&_page_poisoning_enabled); + page_poisoning_requested = true; + want_check_pages = true; + } +#endif + + if ((_init_on_alloc_enabled_early || _init_on_free_enabled_early) && + page_poisoning_requested) { + pr_info("mem auto-init: CONFIG_PAGE_POISONING is on, " + "will take precedence over init_on_alloc and init_on_free\n"); + _init_on_alloc_enabled_early = false; + _init_on_free_enabled_early = false; + } + + if (_init_on_alloc_enabled_early) { + want_check_pages = true; + static_branch_enable(&init_on_alloc); + } else { + static_branch_disable(&init_on_alloc); + } + + if (_init_on_free_enabled_early) { + want_check_pages = true; + static_branch_enable(&init_on_free); + } else { + static_branch_disable(&init_on_free); + } + + if (IS_ENABLED(CONFIG_KMSAN) && + (_init_on_alloc_enabled_early || _init_on_free_enabled_early)) + pr_info("mem auto-init: please make sure init_on_alloc and init_on_free are disabled when running KMSAN\n"); + +#ifdef CONFIG_DEBUG_PAGEALLOC + if (debug_pagealloc_enabled()) { + want_check_pages = true; + static_branch_enable(&_debug_pagealloc_enabled); + + if (debug_guardpage_minorder()) + static_branch_enable(&_debug_guardpage_enabled); + } +#endif + + /* + * Any page debugging or hardening option also enables sanity checking + * of struct pages being allocated or freed. With CONFIG_DEBUG_VM it's + * enabled already. + */ + if (!IS_ENABLED(CONFIG_DEBUG_VM) && want_check_pages) + static_branch_enable(&check_pages_enabled); +} + /* Report memory auto-initialization states for this boot. */ static void __init report_meminit(void) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d1276bfe7a30..2f333c26170c 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -240,31 +240,6 @@ EXPORT_SYMBOL(init_on_alloc); DEFINE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_FREE_DEFAULT_ON, init_on_free); EXPORT_SYMBOL(init_on_free); -/* perform sanity checks on struct pages being allocated or freed */ -static DEFINE_STATIC_KEY_MAYBE(CONFIG_DEBUG_VM, check_pages_enabled); - -static inline bool is_check_pages_enabled(void) -{ - return static_branch_unlikely(&check_pages_enabled); -} - -static bool _init_on_alloc_enabled_early __read_mostly - = IS_ENABLED(CONFIG_INIT_ON_ALLOC_DEFAULT_ON); -static int __init early_init_on_alloc(char *buf) -{ - - return kstrtobool(buf, &_init_on_alloc_enabled_early); -} -early_param("init_on_alloc", early_init_on_alloc); - -static bool _init_on_free_enabled_early __read_mostly - = IS_ENABLED(CONFIG_INIT_ON_FREE_DEFAULT_ON); -static int __init early_init_on_free(char *buf) -{ - return kstrtobool(buf, &_init_on_free_enabled_early); -} -early_param("init_on_free", early_init_on_free); - /* * A cached value of the page's pageblock's migratetype, used when the page is * put on a pcplist. Used to avoid the pageblock migratetype lookup when @@ -798,76 +773,6 @@ static inline void clear_page_guard(struct zone *zone, struct page *page, unsigned int order, int migratetype) {} #endif -/* - * Enable static keys related to various memory debugging and hardening options. - * Some override others, and depend on early params that are evaluated in the - * order of appearance. So we need to first gather the full picture of what was - * enabled, and then make decisions. - */ -void __init init_mem_debugging_and_hardening(void) -{ - bool page_poisoning_requested = false; - bool want_check_pages = false; - -#ifdef CONFIG_PAGE_POISONING - /* - * Page poisoning is debug page alloc for some arches. If - * either of those options are enabled, enable poisoning. - */ - if (page_poisoning_enabled() || - (!IS_ENABLED(CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC) && - debug_pagealloc_enabled())) { - static_branch_enable(&_page_poisoning_enabled); - page_poisoning_requested = true; - want_check_pages = true; - } -#endif - - if ((_init_on_alloc_enabled_early || _init_on_free_enabled_early) && - page_poisoning_requested) { - pr_info("mem auto-init: CONFIG_PAGE_POISONING is on, " - "will take precedence over init_on_alloc and init_on_free\n"); - _init_on_alloc_enabled_early = false; - _init_on_free_enabled_early = false; - } - - if (_init_on_alloc_enabled_early) { - want_check_pages = true; - static_branch_enable(&init_on_alloc); - } else { - static_branch_disable(&init_on_alloc); - } - - if (_init_on_free_enabled_early) { - want_check_pages = true; - static_branch_enable(&init_on_free); - } else { - static_branch_disable(&init_on_free); - } - - if (IS_ENABLED(CONFIG_KMSAN) && - (_init_on_alloc_enabled_early || _init_on_free_enabled_early)) - pr_info("mem auto-init: please make sure init_on_alloc and init_on_free are disabled when running KMSAN\n"); - -#ifdef CONFIG_DEBUG_PAGEALLOC - if (debug_pagealloc_enabled()) { - want_check_pages = true; - static_branch_enable(&_debug_pagealloc_enabled); - - if (debug_guardpage_minorder()) - static_branch_enable(&_debug_guardpage_enabled); - } -#endif - - /* - * Any page debugging or hardening option also enables sanity checking - * of struct pages being allocated or freed. With CONFIG_DEBUG_VM it's - * enabled already. - */ - if (!IS_ENABLED(CONFIG_DEBUG_VM) && want_check_pages) - static_branch_enable(&check_pages_enabled); -} - static inline void set_buddy_order(struct page *page, unsigned int order) { set_page_private(page, order); From patchwork Sun Mar 19 22:00:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13180640 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F5C4C7618A for ; Sun, 19 Mar 2023 22:01:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BBC5F90000E; Sun, 19 Mar 2023 18:01:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B1E81900002; Sun, 19 Mar 2023 18:01:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 998B890000E; Sun, 19 Mar 2023 18:01:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8CE6E900002 for ; Sun, 19 Mar 2023 18:01:07 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 58F2C1C5D9E for ; Sun, 19 Mar 2023 22:01:07 +0000 (UTC) X-FDA: 80587018974.05.1623D34 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf17.hostedemail.com (Postfix) with ESMTP id E3D5740016 for ; Sun, 19 Mar 2023 22:01:03 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=kOd7Gv5m; spf=pass (imf17.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679263264; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NoiMHve7UWm4zfnV54oLaYQeRCeC+o6mPLSSjHXn11s=; b=ZKQcBxvvP6miHk4YTd/OHhmUNo/4PQ9iv7OotERTWWIO6IOhEPHJuEO5i0yACXLDPKAdyx PZ51yCDLQCB2PkY7DY89bqqiYaid0M4jBA40Xw6yblScfNyBhajkfowzNOaei0FT5j4Inq dyk7JaNDam2bm3RVp56bE4Jrw4+/f+0= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=kOd7Gv5m; spf=pass (imf17.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679263264; a=rsa-sha256; cv=none; b=7BhQVYSEyJooJlwHHlIglT28RkKbgCYx27aIY5zD3v/H5FUUvqilP9KpYXocTAS34ucemZ Pi3bDfLbw11oGHdRDSIkXDavlkmBTc2gUa7Cj7/m11Y2Vmf4e63E53wEiTesxuVzIAQgJU 8vXT+FeCbkUEjAkoe0cjoHRn9wzgOPk= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id EFBCE611CC; Sun, 19 Mar 2023 22:01:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5AD33C4339E; Sun, 19 Mar 2023 22:00:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679263262; bh=967+3O+np6cjBUo4maqL5M/MoNfM+0xG04PeQyxPgnI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kOd7Gv5m/2xJyFfcUY6qVcv/DS05Vhxg28CbZOm1yHQ8fQBZrEnyLXI8A/BQZeAVZ PiLJw4jUvUeh7VaCBe65lOjWv7ahlTb+Fn1ZrLZwGyD378pSK1SbVtsoc0an9Lh4To FkUFTbGuvHM1w9iM5uRBWUoWg1PHTZSbtCAaLguK2MNva5sczzZzcZVgudK4IbAyxE rS8rd/XjLwVGlHY+lX6zVm16lXK5HMqkj9j5FNiJbWbTXEHmTe4jWaAK+8sQPvHZpy F99UJ4JCko9T5QC93h8zgGRLd+q32X01F9a2FQSVyrnb+W9Sx+cHhHLpcMo2c97KgO MOV+qAjnx9eJg== From: Mike Rapoport To: Andrew Morton Cc: David Hildenbrand , Matthew Wilcox , Mel Gorman , Michal Hocko , Mike Rapoport , Thomas Bogendoerfer , Vlastimil Babka , linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 11/15] init,mm: fold late call to page_ext_init() to page_alloc_init_late() Date: Mon, 20 Mar 2023 00:00:04 +0200 Message-Id: <20230319220008.2138576-12-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230319220008.2138576-1-rppt@kernel.org> References: <20230319220008.2138576-1-rppt@kernel.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: E3D5740016 X-Stat-Signature: ja6iiq5bgu57as5sn5amb5drc5wb71ot X-Rspam-User: X-HE-Tag: 1679263263-703024 X-HE-Meta: U2FsdGVkX1/85yKi/KK0MIzd0vWk2AD5i7DElBxFHUjyT/BSJWntn3FW+qrQhzEj84XMOoUEUVFhaatOMLIdx+We2Qr7xPJS4g62hoFqr+S3rkdCZGhmfmumQzRk0Ws1tAVD1Dnz9Ubj+gw/cRVYO63N8MjspARmveJEru2+aVEWs5K8zrB11OSPgg56Rlsi1G4dWL1DpXM+UlfE4SFA4s6ydFjd5K23voTHrsanA/zqwfj0y9uo9+cPbd/8vTNylIMELLprdsL1i4EaGCiJQQrkdqQp31jIoobHbr7NDqrAjTlB3N6S071DO3sIEQ1XBMGbUl0M4NlhfazfejJor3IhDo3iO8IZbtMPN50G6SkwgafHMnwFK5+Y824Ia/1fT8f02kH+wglur0vbwRdIpDi9hFvQOUUQdiL8jTev+NZHC84lffWUlPWZkeiy28AGiutEVkQM3irMx9PxaK6cYxuxgZ59oA7eUsykgrxzkx1Wb99H5L5t8XpFeV7Wkma2z6GsizU+RNo2l9Y89r4HNkGrY1R36sZfylhHQDsmxoYEg6SE8ure/L9HkpcDrROIyqOT2hzk97fX+y0n/FLoHyWkluY879gmgvUu6D5gNJlt/7n40z++cwn6I8zVLXAsuE93Z47QIKKGLjIzlzpeq4AFrVYOCRA9naxn0RLWhIzmHkD8bxi2lQwomVPWLxp76VhhmZAkcPNW1q/yT37KtLVqHSLwNvirsHIqDPQ6IypYMEDn2HLQnDNGXfjBEsox3OrE3lWlrLM1rFpWdLnWjUwgzG4g6xIJUqA+zUf36svCnD7GT4vhAqhCONzP/LYuJ9KUIuVB2m4/E5ZLIhNWwSw9Soox0DY2GQM9rVkeWK1dEZRztgNN0O0BWVvlohlfXB6YWkxk6ONzbZ8gaOcku8ufiuTsBGFW9BcRcPLh9Itrp4jnbILOmkL7juqxWaV8S4SR9fsRw67pO24f9F4 5lkIiw4C 6Cj+y58kFKvaI8PKWyjYIyPGti8znzWHEwlDzPJIqLXDxhnXW/+XTiknZLAmiLa5u6JA5op0UdurBKKJcQ19JFa3VGMMdx1Y62aNMR/1cYFEA+HtyZIhUWDVd0L+Ys8EUueSzHxTjzKHj9tbAuLP11Vu8QXJ0kqJzRnqnaTXcRKRfiTdnNoCIJ8hGYfTqqvWeKIh+hdG9OIhxWdNNY2a/DnCyjHAZej5ycXPO06BJyMc8SbyR+PBYpw+qG4AjB1tjXVh/8VskUhkmPZnVRaTrPyVUdhVKe/W8ljdEjcjyYhWkbqT0x1R/P9Q/95kLXwVdKVCDiK7ZtVw3zOpK9CTniqSX4dIMY9OoFN29NM8GBIUVGTpGzdR1T6TuGE+0DxbezIftAPIUg673y9kCpuW9hAsKqiVFvLCXuYMk2WPhkAN0wmFDgTEC97hd5g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Mike Rapoport (IBM)" When deferred initialization of struct pages is enabled, page_ext_init() must be called after all the deferred initialization is done, but there is no point to keep it a separate call from kernel_init_freeable() right after page_alloc_init_late(). Fold the call to page_ext_init() into page_alloc_init_late() and localize deferred_struct_pages variable. Signed-off-by: Mike Rapoport (IBM) Reviewed-by: David Hildenbrand --- include/linux/page_ext.h | 2 -- init/main.c | 4 ---- mm/mm_init.c | 6 +++++- 3 files changed, 5 insertions(+), 7 deletions(-) diff --git a/include/linux/page_ext.h b/include/linux/page_ext.h index bc2e39090a1f..67314f648aeb 100644 --- a/include/linux/page_ext.h +++ b/include/linux/page_ext.h @@ -29,8 +29,6 @@ struct page_ext_operations { bool need_shared_flags; }; -extern bool deferred_struct_pages; - #ifdef CONFIG_PAGE_EXTENSION /* diff --git a/init/main.c b/init/main.c index 8a20b4c25f24..04113514e56a 100644 --- a/init/main.c +++ b/init/main.c @@ -62,7 +62,6 @@ #include #include #include -#include #include #include #include @@ -1561,9 +1560,6 @@ static noinline void __init kernel_init_freeable(void) padata_init(); page_alloc_init_late(); - /* Initialize page ext after all struct pages are initialized. */ - if (deferred_struct_pages) - page_ext_init(); do_basic_setup(); diff --git a/mm/mm_init.c b/mm/mm_init.c index ae6bd26cf5a2..2d73d8b05a69 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -225,7 +225,7 @@ static unsigned long nr_kernel_pages __initdata; static unsigned long nr_all_pages __initdata; static unsigned long dma_reserve __initdata; -bool deferred_struct_pages __meminitdata; +static bool deferred_struct_pages __meminitdata; static DEFINE_PER_CPU(struct per_cpu_nodestat, boot_nodestats); @@ -2338,6 +2338,10 @@ void __init page_alloc_init_late(void) for_each_populated_zone(zone) set_zone_contiguous(zone); + + /* Initialize page ext after all struct pages are initialized. */ + if (deferred_struct_pages) + page_ext_init(); } #ifndef __HAVE_ARCH_RESERVED_KERNEL_PAGES From patchwork Sun Mar 19 22:00:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13180641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6197C7618A for ; Sun, 19 Mar 2023 22:01:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4905190000F; Sun, 19 Mar 2023 18:01:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 440F3900002; Sun, 19 Mar 2023 18:01:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3087A90000F; Sun, 19 Mar 2023 18:01:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 22F8B900002 for ; Sun, 19 Mar 2023 18:01:12 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id F307480382 for ; Sun, 19 Mar 2023 22:01:11 +0000 (UTC) X-FDA: 80587019142.27.6E92633 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by imf09.hostedemail.com (Postfix) with ESMTP id 23800140025 for ; Sun, 19 Mar 2023 22:01:09 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=mM1Vr2+w; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf09.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679263270; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jb5UHswS4WDrIMdMzQva9LfRH34XPC2+tjloBOHlJyY=; b=xAGw/FxoAbXxvpDJ+VwjwtFCN/9GKvPJdxp7JBYmFGBQZtzB++Nve+7wgPPhBHZa3yiUY2 7/5SKgokr0VDQaKq/q3kZs3akknz6mPo5sI4mmNp9pz8meQ0ZtiMq3gwPQaNNYz7KrbuV2 x3GhCasnlun3J2n8OKL277Mg9fPMNKY= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=mM1Vr2+w; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf09.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679263270; a=rsa-sha256; cv=none; b=YsE+Gjb20QivBPoQafXEgLkt4sP9hEDWOurNd/hZT+pKetH93VvBrOsAVz8VqEK+SQmeRC wBIpbCixyiAlCkBCvG4HzHbjE0bnrJOJkIWyA7wPHLyT78AjpJQuGBj1upqeHETZreP4AC ZWHzBsZY6lshsVms7TRCartxxLzSd2k= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 384E2B80D2E; Sun, 19 Mar 2023 22:01:07 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E4F09C433A1; Sun, 19 Mar 2023 22:01:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679263265; bh=BYCuSoeoE+R0Fgq41f+eIMZR+J4M0f4Ilxwf9ZY18IE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mM1Vr2+we//dVSbb0oJd235ZDrn59kE4Qyq71QnwosOv75bF+QcOee7a+KPpNC7WO Na8TF9Jaeb5Q+StXUw2tVLuOQHi0l3kByL9vKvQ7jkkqPlcWuomNFNL6bFb7ITNYQF zx3mhBvD9OuvGe8YST4IX+xf6LnKx5RtAh+ovZ9sPxyxj0LEvC+mgaSEUm7Pjn3Otr UJjubGuY50Drn59r4WjnLnAhSeI0n5tXB45LpjueAEkrYTfiktcYKyevHeQJfcf24d rSSyRMPXpbh1eGjlWOoKM8sLRKYQmbPZ1N8GydCPthI2SmRA189raVXHEk668HDNYi XwSOrnh+AObdA== From: Mike Rapoport To: Andrew Morton Cc: David Hildenbrand , Matthew Wilcox , Mel Gorman , Michal Hocko , Mike Rapoport , Thomas Bogendoerfer , Vlastimil Babka , linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 12/15] mm: move mem_init_print_info() to mm_init.c Date: Mon, 20 Mar 2023 00:00:05 +0200 Message-Id: <20230319220008.2138576-13-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230319220008.2138576-1-rppt@kernel.org> References: <20230319220008.2138576-1-rppt@kernel.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 23800140025 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: c4ms61t38pcz1ru5i946m8d8dfb96hsr X-HE-Tag: 1679263269-206119 X-HE-Meta: U2FsdGVkX19DZQDEwVhcJNLJPZGOhNDCnoZCVhbKZSyvWauhuGxBs2v+o2s2vryOp7XHzPtBkdgtJhZD7EECzw8jPNQysv3VmQeFvST8BxDJgYwwL5LNycMOdKeIF1dLu9mvXGlosdkxxYq5j0V5WWU3QQRoSFi9TADgkbo27xxVn7Hsxc3o7mU6obVhVJLmHWzY1MZ0JAufxAPnhEkJHBJJXotqvOMOYsZRJSwYhMPnx9mdxMO+WhhwY2dGGrDL4xYOBVL5v9zvWHIVLdlzYOm5N77KQO17uYWcn57exkeaRq17v/IXGk7RX9GBsmI/lpVQ3dhoky6yTeAowAyzltdo1kZKl6xmlM2HDfCL+tDiSTVzZQM0qiWPEQqlBddeYT8nfT8/N82T6wdwwlxUEJDxNge/ynTdzHnHliN7pesYzPDQWr3F5XUv07jKu+xp5loeYhoC7N1I+h3PBqQAkgbzK99dakMTwJWH5DBs5HqYgILvX8WslrwjYoZ6x8thCLNI6prEmyHICuFtSjuq0SVrPSMhGGjZZ4NFgHrt9nbLFqdXgws0BAjUkqxZBjAfJhQ7LYV2KptqAFEDQKHkkiUOctlHpkAOULEY379U0Z/7BRBNm1EIrDU5jUEGCq0lB/+HXWuReNLaKBjkegYIKTkO4lEL+rx3ciUJJuCBQBvcFtuirzO6/HaDDzNOhXyM5eiT1I3dgkxohu/8dbrxXMK9jm3uib2mylVVz9icpIH9zjOJAYGSNWR23rxuo5Rp+QLvzMdM3tQbl1XXncsn+4KtYQhpS2xG287GdlPpBKhvve2XGjAwnMlcgzWuVtgqTozYo9QRUK3aQp1zPPqppoPefN+CHPduG91UwqXNQnn91TRHc/YsTUpI8j7X64JJ3/1jLGgFjQHu06AN9Ix4X3AIVYpdGuSUnRoegLUnYlLgvYTlmIFrqGDysHFpSqQuHRJTMhy+8jUgt+zM+Ik LnQj6BQf NvwfRY/8ASJpNFtLH/xinoNnleZsW5hneh8FDMw7TLiPRuxA/lkYeuAlN8G4pqFDA1119IsV2i/qcd9e6ZXRMMyVsGDNyjdw7ElucT/MzP2Z9jI4Ymz0y6LO7wWjTHHX5pw+g3yoto9W5vwiTARu7UxkZ43M7PSQ4zgmyl3k9h4dFWpMBeivcpaLGu94eqfRxaZ6Ghk7vUOZMAjdqVBmX7b1D1/DiZPIxmfJTOxxMg4XcrM9Io9v9knjjixdD2am1YmY+0O7TZn5XXBqHr9Ok81p6b4q7WFIiHyDE4V2/36HbR+I4WeGqPcszkUQfz01PJJIw5hZWPkSEiAJNx2AWlEFs+yX5m3voVqj2H3VjmiBakDd1u4tptU1HY18jSh07FjQ5Ipt2QAGCxW0rnHuwqTCuiA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Mike Rapoport (IBM)" mem_init_print_info() is only called from mm_core_init(). Move it close to the caller and make it static. Signed-off-by: Mike Rapoport (IBM) Acked-by: David Hildenbrand --- include/linux/mm.h | 1 - mm/internal.h | 1 + mm/mm_init.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++ mm/page_alloc.c | 53 ---------------------------------------------- 4 files changed, 54 insertions(+), 54 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 2fecabb1a328..e249208f8fbe 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2925,7 +2925,6 @@ extern unsigned long free_reserved_area(void *start, void *end, int poison, const char *s); extern void adjust_managed_page_count(struct page *page, long count); -extern void mem_init_print_info(void); extern void reserve_bootmem_region(phys_addr_t start, phys_addr_t end); diff --git a/mm/internal.h b/mm/internal.h index 827499e39d78..1be4278d7913 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -201,6 +201,7 @@ pmd_t *mm_find_pmd(struct mm_struct *mm, unsigned long address); /* * in mm/page_alloc.c */ +#define K(x) ((x) << (PAGE_SHIFT-10)) extern char * const zone_names[MAX_NR_ZONES]; diff --git a/mm/mm_init.c b/mm/mm_init.c index 2d73d8b05a69..73964449669e 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -24,6 +24,8 @@ #include #include #include +#include +#include #include "internal.h" #include "shuffle.h" @@ -2635,6 +2637,57 @@ static void __init report_meminit(void) pr_info("mem auto-init: clearing system memory may take some time...\n"); } +static void __init mem_init_print_info(void) +{ + unsigned long physpages, codesize, datasize, rosize, bss_size; + unsigned long init_code_size, init_data_size; + + physpages = get_num_physpages(); + codesize = _etext - _stext; + datasize = _edata - _sdata; + rosize = __end_rodata - __start_rodata; + bss_size = __bss_stop - __bss_start; + init_data_size = __init_end - __init_begin; + init_code_size = _einittext - _sinittext; + + /* + * Detect special cases and adjust section sizes accordingly: + * 1) .init.* may be embedded into .data sections + * 2) .init.text.* may be out of [__init_begin, __init_end], + * please refer to arch/tile/kernel/vmlinux.lds.S. + * 3) .rodata.* may be embedded into .text or .data sections. + */ +#define adj_init_size(start, end, size, pos, adj) \ + do { \ + if (&start[0] <= &pos[0] && &pos[0] < &end[0] && size > adj) \ + size -= adj; \ + } while (0) + + adj_init_size(__init_begin, __init_end, init_data_size, + _sinittext, init_code_size); + adj_init_size(_stext, _etext, codesize, _sinittext, init_code_size); + adj_init_size(_sdata, _edata, datasize, __init_begin, init_data_size); + adj_init_size(_stext, _etext, codesize, __start_rodata, rosize); + adj_init_size(_sdata, _edata, datasize, __start_rodata, rosize); + +#undef adj_init_size + + pr_info("Memory: %luK/%luK available (%luK kernel code, %luK rwdata, %luK rodata, %luK init, %luK bss, %luK reserved, %luK cma-reserved" +#ifdef CONFIG_HIGHMEM + ", %luK highmem" +#endif + ")\n", + K(nr_free_pages()), K(physpages), + codesize / SZ_1K, datasize / SZ_1K, rosize / SZ_1K, + (init_data_size + init_code_size) / SZ_1K, bss_size / SZ_1K, + K(physpages - totalram_pages() - totalcma_pages), + K(totalcma_pages) +#ifdef CONFIG_HIGHMEM + , K(totalhigh_pages()) +#endif + ); +} + /* * Set up kernel memory allocators */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 2f333c26170c..bb0099f7da93 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -5239,8 +5239,6 @@ static bool show_mem_node_skip(unsigned int flags, int nid, nodemask_t *nodemask return !node_isset(nid, *nodemask); } -#define K(x) ((x) << (PAGE_SHIFT-10)) - static void show_migration_types(unsigned char type) { static const char types[MIGRATE_TYPES] = { @@ -6200,57 +6198,6 @@ unsigned long free_reserved_area(void *start, void *end, int poison, const char return pages; } -void __init mem_init_print_info(void) -{ - unsigned long physpages, codesize, datasize, rosize, bss_size; - unsigned long init_code_size, init_data_size; - - physpages = get_num_physpages(); - codesize = _etext - _stext; - datasize = _edata - _sdata; - rosize = __end_rodata - __start_rodata; - bss_size = __bss_stop - __bss_start; - init_data_size = __init_end - __init_begin; - init_code_size = _einittext - _sinittext; - - /* - * Detect special cases and adjust section sizes accordingly: - * 1) .init.* may be embedded into .data sections - * 2) .init.text.* may be out of [__init_begin, __init_end], - * please refer to arch/tile/kernel/vmlinux.lds.S. - * 3) .rodata.* may be embedded into .text or .data sections. - */ -#define adj_init_size(start, end, size, pos, adj) \ - do { \ - if (&start[0] <= &pos[0] && &pos[0] < &end[0] && size > adj) \ - size -= adj; \ - } while (0) - - adj_init_size(__init_begin, __init_end, init_data_size, - _sinittext, init_code_size); - adj_init_size(_stext, _etext, codesize, _sinittext, init_code_size); - adj_init_size(_sdata, _edata, datasize, __init_begin, init_data_size); - adj_init_size(_stext, _etext, codesize, __start_rodata, rosize); - adj_init_size(_sdata, _edata, datasize, __start_rodata, rosize); - -#undef adj_init_size - - pr_info("Memory: %luK/%luK available (%luK kernel code, %luK rwdata, %luK rodata, %luK init, %luK bss, %luK reserved, %luK cma-reserved" -#ifdef CONFIG_HIGHMEM - ", %luK highmem" -#endif - ")\n", - K(nr_free_pages()), K(physpages), - codesize / SZ_1K, datasize / SZ_1K, rosize / SZ_1K, - (init_data_size + init_code_size) / SZ_1K, bss_size / SZ_1K, - K(physpages - totalram_pages() - totalcma_pages), - K(totalcma_pages) -#ifdef CONFIG_HIGHMEM - , K(totalhigh_pages()) -#endif - ); -} - static int page_alloc_cpu_dead(unsigned int cpu) { struct zone *zone; From patchwork Sun Mar 19 22:00:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13180642 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A54A1C76195 for ; Sun, 19 Mar 2023 22:01:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 44265900010; Sun, 19 Mar 2023 18:01:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3CB97900002; Sun, 19 Mar 2023 18:01:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 29328900010; Sun, 19 Mar 2023 18:01:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 17AE6900002 for ; Sun, 19 Mar 2023 18:01:14 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id E6BDBA0207 for ; Sun, 19 Mar 2023 22:01:13 +0000 (UTC) X-FDA: 80587019226.16.2AE3A24 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by imf14.hostedemail.com (Postfix) with ESMTP id 0679C100029 for ; Sun, 19 Mar 2023 22:01:11 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=dj21oSvT; spf=pass (imf14.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679263272; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AvGyKfsisscgtbAEHKoU+lyYUhws0/Tai9b1Eed9JZY=; b=T9TbfOiL/XWr0rEfYLdvFsy4gsloIP0ebkYnvQxE09yK9doqkgQzODBtwFurfINOz4I4PE eJCD24jhauvfWxihWFHuM0eyyumgOrzE+XcdFo5LT25WlGufhVl/Mp6dItiDwTAvRHlE/X ItmrRJ6Qn/n52RE52xmPfZYR5gS1k80= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=dj21oSvT; spf=pass (imf14.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679263272; a=rsa-sha256; cv=none; b=M9ASQa8zmZRHCywKSeCx4Z6KW4F7MANOdSiGE9qh0kVPXIng8SoaYbNCc2RIAAppwC0Oy3 O06jj6PYA3Yv8ZrrO6j9e5lvoUQ3BdbfXZ9HL+7Ew65HFxXaVapIR7DF//MMzWWCLcHLu8 9RdQxgwVURb+BxlesHCsyp3TXoz7CMA= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id B4924B80D30; Sun, 19 Mar 2023 22:01:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7A5F9C4339C; Sun, 19 Mar 2023 22:01:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679263269; bh=5aEejQ7EtQqthblpiGHnkd6ifM5j2LwwZNCKwxJWY2E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dj21oSvT9z3Ed9zQIOuVK5nwGGD/EqvV3Ixx7fZIDmTRlNnvvymwFxLdN5KS3CH8q HhMFUzoMLbKZVndGPuSqJws8OHtUTpsZ9y58wSjxo6Oes/hMdNBSkYmU2PprTeM6IY L0T2LHC76nu/Jf7wcAlQm8TrrzbbBtCC+k0KaMLeLH8c9syUdRBiCmhAfOQb2BeTJA nEKy78SUkn77LVx3EeUxphJS8tyk9/Qa6akui/E05DL/u7ZY60jAP5R2NT725QsSk+ 4lTdsTiMaMrCi8nhfQgQmSe0DKpKsElEDrQ/GzdER+v3axyyhzfi3QjSy3TIZF5JLB TLU2fUa0TYLUg== From: Mike Rapoport To: Andrew Morton Cc: David Hildenbrand , Matthew Wilcox , Mel Gorman , Michal Hocko , Mike Rapoport , Thomas Bogendoerfer , Vlastimil Babka , linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 13/15] mm: move kmem_cache_init() declaration to mm/slab.h Date: Mon, 20 Mar 2023 00:00:06 +0200 Message-Id: <20230319220008.2138576-14-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230319220008.2138576-1-rppt@kernel.org> References: <20230319220008.2138576-1-rppt@kernel.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 0679C100029 X-Stat-Signature: zo8ncj5rnuaxfc7xnxjhihpbqd65kbsa X-Rspam-User: X-HE-Tag: 1679263271-56791 X-HE-Meta: U2FsdGVkX1+xSl2g/gdxRtIOwWEsUshRr6yN3inQ9dcbKqsd41kahV1uRQKVWBAYqxO48m9uGHUIm1hd+Ie3YI4DW9UDxr3m0w9mM85MV0FwGHalYsflsUmRdZwx2V0178R5i58KHCHZHVFb6ln46fXk1XDO9UHpMENqxIxLO63IMU0A+edyQ5z561RlvZeMDQY8xIoaxLN3YvlEJ4HGYSAVHfkaG+N4HDF/DlOOqteZmznf33/kmzgJWGpQUyyTStQ6P3BD6nefkQHFf2IeC1msXieFsbLVaFXz8axsV1pGfHMPqeJOkzQNHh9vRGUCM59OWGUIRB09SeRcONd+vMAXA31oRQ+9BPdJCjpaFNCtv7hlni2vUOoRHzbgQDe5LRs1TL+iKr2Gk6mbXohlGLRxCSS3+NRUQTMUBjj3qu2dageI/d8iQqPjvu7gCptweEMjynDrEXPWiZdZUvryPo8y57NaLTVFvrNyXxBKMz+foy4qMcMjIJGZo5fGEZ3ntBM2VhORXBhuhBrya3kM2zwsc9AtiXOaTOu8il05idMEDFRxWSLVLabK1DLrtFHowUdKpJA6ap3uocetRx4lWxGMSUNd3P9KKTO2gWftVZkFPsG5aUbMVnNMbyoKOpfw9wRqkfhUnaYDzOD2Az3wKwFOpE7uUuIwjRGWgSpguEx+Z7L1CS9BVft8sv3PSxJO/bcVoXx5ABEk3ha55WByWFok0vfeZvSYnmw9Dak6JImpVxH6L4hTqQyQvxV4t/PTB4Xcf7Whosv08I+bYm44HSGALNUjU1uBZ1v/TU1OZNGf1t1XrX4w0u7tmuSgmwO3e612DPTtskUc+z7zQ/JbxWQXLQW1oLWRDRowqSDOBJXGRJQ34yF58PXwItvItI7GmFFX/VFa+rlprnqx17SW4SBH6V3QRYNr9RJyTX3OA6Sw4O3G1OtjXnYNJs7VJZ1B0mMuO5Q7BJfZKWRYNM7 GQwPHLQ5 3cI+Bgn7v7Xt7tAygxg9eSy2DVwMt8G0uyQo/0/UvjfXuoh5m2CiFPKAFguE8IdhTvBp+fB0puRRrWf1scv6s4Y5tUwNAlISMnLSUL03BBGOkLtS7kV7hKFBYUFLf0MxtbxcItR38fYnecsIOF9cbH4Tz7Cxdiarzt8oLuCb+UEOjRo0yfuLVwmd7lWv3zXOX/kuZ2xiA8dsCJItcos0DL5iEtegcwa5xIaY/vxMrCoorRTigFeK16aYBNUgq+FTIAVppz2WfgY0Ht/CU8R7A7B1u7idFzXe0df0fjmBOZbxlXEr3+/vvUI6apHMO4dNMihTV6tfbcrWL3AcRPp6hTG/i1/t4qXahjbeoHhXOWLyYUNANksUECbnpyrTkj7HDeQBnNMEahmu22LFn0TwjkhjUuusqIbIyU61EuJQJAMassODM9LYMJh8guw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Mike Rapoport (IBM)" kmem_cache_init() is called only from mm_core_init(), there is no need to declare it in include/linux/slab.h Move kmem_cache_init() declaration to mm/slab.h Signed-off-by: Mike Rapoport (IBM) Reviewed-by: David Hildenbrand --- include/linux/slab.h | 1 - mm/mm_init.c | 1 + mm/slab.h | 1 + 3 files changed, 2 insertions(+), 1 deletion(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index aa4575ef2965..f8b1d63c63a3 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -167,7 +167,6 @@ struct mem_cgroup; /* * struct kmem_cache related prototypes */ -void __init kmem_cache_init(void); bool slab_is_available(void); struct kmem_cache *kmem_cache_create(const char *name, unsigned int size, diff --git a/mm/mm_init.c b/mm/mm_init.c index 73964449669e..78b2366e4407 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -27,6 +27,7 @@ #include #include #include "internal.h" +#include "slab.h" #include "shuffle.h" #include diff --git a/mm/slab.h b/mm/slab.h index 43966aa5fadf..3f8df2244f5a 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -4,6 +4,7 @@ /* * Internal slab definitions */ +void __init kmem_cache_init(void); /* Reuses the bits in struct page */ struct slab { From patchwork Sun Mar 19 22:00:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13180643 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0007DC7618A for ; Sun, 19 Mar 2023 22:01:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8FDAA900011; Sun, 19 Mar 2023 18:01:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8AD81900002; Sun, 19 Mar 2023 18:01:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 77550900011; Sun, 19 Mar 2023 18:01:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 672A0900002 for ; Sun, 19 Mar 2023 18:01:16 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 38D311404B1 for ; Sun, 19 Mar 2023 22:01:16 +0000 (UTC) X-FDA: 80587019352.05.50904A9 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf26.hostedemail.com (Postfix) with ESMTP id 7411F140010 for ; Sun, 19 Mar 2023 22:01:14 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="BbRrG7/N"; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf26.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679263274; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Qx4xDHpddo9ktihFE8x/+uQnfMmrp64WSopEcMQJvJI=; b=Y6Jzc4amr6KPNOKyqkPEkzW/Lk/qvYl/9UO/gRoVOw1jRJhgTQQotZqwJi0WfMXKz+HaNA TyuNTM4FCGLmVfoXsaLKjdtiS4BD4ki6mxvRuN5/MpWqKqCpULk7Sc4JP8A9TPUQaz14jR 51z6RvsnjbhQhDSa4ZyUEN+oVjQQOb4= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="BbRrG7/N"; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf26.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679263274; a=rsa-sha256; cv=none; b=YqY/PNWfC3I3MtR0TIW1EpzfciISVHmYz31FzHUsC4z5qTipZStCIRIoUcyuVwO7VVSdLV tdf0QVBFnofvbth36aKrm4ybULcTfCi7yZwMZ26l/57Am2+jr0S4AH4Qs1Iey787YphrLf Fapp7poqLnvsc63mnCOtYS2UUNJ+U+w= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id A3779611CD; Sun, 19 Mar 2023 22:01:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0FEBEC4339B; Sun, 19 Mar 2023 22:01:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679263273; bh=PogP7wRngXQiUqmQnWvQozicSIVShAdtUNbeYC8wpS8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BbRrG7/No3s+Wxn35WykZgqumgAtgA71gdQ5J12dxWsQy8T8UYZ0xOfDgvafCU0gU juKoeAAlcxDYKBZj7ANUaIDhkirDSSOn9GQHrhjrlBMAH632EHvpMIDKl/8mm6psIy x4E9jus7CN4xPRD62XXDkHy6wYmKhH/Rrkc6uE9Pi+lX/HLxmCNzPlOAfx4mQDCypg 0IKAP5boUvNjsBQkM3+RA309/oZJHntXBHJ5Xor9n1MvaJQAHrzbMA0bHab61TmasX l+NN5p1t3GG1PEDp6Sv0t7ytroVh+moRTc2smV8hZ2vuDxm+xP/p1KRVuZ4d2A2bEy 5G44pgtQ35qMQ== From: Mike Rapoport To: Andrew Morton Cc: David Hildenbrand , Matthew Wilcox , Mel Gorman , Michal Hocko , Mike Rapoport , Thomas Bogendoerfer , Vlastimil Babka , linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 14/15] mm: move vmalloc_init() declaration to mm/internal.h Date: Mon, 20 Mar 2023 00:00:07 +0200 Message-Id: <20230319220008.2138576-15-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230319220008.2138576-1-rppt@kernel.org> References: <20230319220008.2138576-1-rppt@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 7411F140010 X-Stat-Signature: t8yz713mgbrhj71p3su6iucoaifuqppr X-HE-Tag: 1679263274-735520 X-HE-Meta: U2FsdGVkX19hU+9MFs3der7IMNgm4EtW3xhsXVg6cxiS7hwsxD2F0hJfy9H5eTciL5snRapm9+oUiopvM8qWMQ5RxORN0v0kEU+h48q+FGwaBchCp9Up26Sz80T10TvLz1K/hRHh3jkM+8hsxwSc6lHsWhroER9xJjXX6h29WEdNnNK/aecanhsP7pgQjHJfUOr3LiWfktgA8OL/3KgNZ3J0dUBlQa1iOhcsvzww2wfni+7kBoxkqqfZtXcJPXYOfRa3Jfm8mg9drcfwPiiCMlSI1qKEeA7BbAkZ5ipTB1F6ZK3J41SeD274nVR3CCp1Ioz4rmLmhTt5qkbYLdfk3FKqZBqVCEpBGSshNor0JogGfOROTAgErDYZiLdT8UAx6KRv2/kCCk8q3An9sKtrQOTrjc31rqqOwthdto3UUdyAUKvkpVk2Ko2QQ49RRddGkbYUQdAVQJN35Re3pcuyxLXnszK0MjUJHmaMufBb8lX9S2rWk4C/MWJ0+sjBMbVu1pELFP2HW8B+WbJRQRedocBvB+oXBwVtqJUrydr7AdPOhE4lg2GnR+qdq6bcKgzZENAN9V2QWnm4rWzNwm/gRRIIFretOb/lhH29XQMmXRmRc0n73j1M8XhTnlPzxnA+/sTDpTEiwl/VQ70n+tM2Wm0sL1IhdQhWiS2+j5XvrN1ZQmp7mSGwf+Z6xiSiYDNOB5B+O5aA0jux41EfN8ncUJKouxwSnEt5vgpdQesqOQPIRb+yOWbJS522fwoz+OdcI+Scr0kws8kxAinpAGEw6/XLoUxQYGi8vN/edbDqlBzs6uiZ4DwGcouooi9ygxeXHd78iyPRMs8Lto3/zkaVZYwWuOgUzToE/sHCop66ecHMVbK2irc5+haBYLdabVrMokf+cyxuNvev5spkf62kuhstfIHKmQt8NqhD3DqXLpOval+G01ML/xQHdPDdFDCE1V3hrB4JLw+JiI/gPrQ l2pN4GFG ypHnph9BI4ViOkOS//pnTinS4NLi+SS79Z4YoZEZrfn0+4vffhFOWtkjFZrmvEmMbQqYwGjvr7rhr8qKZa7LK3w4ivPJ+G/aWf7DUnX97mmPjN2IVlpw9pziqFUtiTVKPoQ8iPuI9+IoGd1uXVg6ZZokSJvda019PXFUYBtd25yan1aZxQ81shxQLwqRqPyK3pjTFGGQYSJvHK9C89KqH11OW//lfzFSfp219DtTi64X3sTEQ6EluDvQfH7tvQ3Gfc7Jpsfw1WEz07vwaftaCEpQbF9bhW0urGTlKzLKhj4ivk4/qHIaSCWjF5KoMIev01sKeNYAP+p9AGo2ZAXdKVwc4xs6FKOYTifiWzb6Cw5nsSpTcniU63QV1yypuYcG6nZ0TdLCoXoNILetD0AghYUF16w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Mike Rapoport (IBM)" vmalloc_init() is called only from mm_core_init(), there is no need to declare it in include/linux/vmalloc.h Move vmalloc_init() declaration to mm/internal.h Signed-off-by: Mike Rapoport (IBM) Reviewed-by: David Hildenbrand --- include/linux/vmalloc.h | 4 ---- mm/internal.h | 5 +++++ 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 69250efa03d1..351fc7697214 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -131,12 +131,8 @@ extern void *vm_map_ram(struct page **pages, unsigned int count, int node); extern void vm_unmap_aliases(void); #ifdef CONFIG_MMU -extern void __init vmalloc_init(void); extern unsigned long vmalloc_nr_pages(void); #else -static inline void vmalloc_init(void) -{ -} static inline unsigned long vmalloc_nr_pages(void) { return 0; } #endif diff --git a/mm/internal.h b/mm/internal.h index 1be4278d7913..7e22137b4e86 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -895,9 +895,14 @@ size_t splice_folio_into_pipe(struct pipe_inode_info *pipe, * mm/vmalloc.c */ #ifdef CONFIG_MMU +void __init vmalloc_init(void); int vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift); #else +static inline void vmalloc_init(void) +{ +} + static inline int vmap_pages_range_noflush(unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, unsigned int page_shift) From patchwork Sun Mar 19 22:00:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13180644 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB1BAC76195 for ; Sun, 19 Mar 2023 22:01:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 860C3900012; Sun, 19 Mar 2023 18:01:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8119B900002; Sun, 19 Mar 2023 18:01:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6FFDD900012; Sun, 19 Mar 2023 18:01:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 60A5C900002 for ; Sun, 19 Mar 2023 18:01:19 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 3EF3212017B for ; Sun, 19 Mar 2023 22:01:19 +0000 (UTC) X-FDA: 80587019478.13.7DCECFE Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf16.hostedemail.com (Postfix) with ESMTP id 94F4518001B for ; Sun, 19 Mar 2023 22:01:17 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=VUXzADYn; spf=pass (imf16.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679263277; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZLI3OaGCbGh/vNLMkHhIynqNYC8c9VoM1O+XA/wly8s=; b=TyWNzfdrjM0aqly1GOPyt31eCMVyRD020eye41ylJT4c9Pj387rHPX1Rx1V1EIcvlxXwVd Q4AlHAhmbmbrfkwl5q4JAIHpVSkPS+IWt08uMTMbMxReAH68AWu85nZYxGASn+6WHOdMaR d4djHurtMEvv4cQImqMTBycre6DGAI8= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=VUXzADYn; spf=pass (imf16.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679263277; a=rsa-sha256; cv=none; b=F1k7G+b7BG5ntEsIm6ZE92t6VeNJHFlrUpcpXc6DDTzxc0JKT5Hes8LtSfrPJE3bGoOIxD zp9DYBvpzM2X5JR1glt+Vd/kVXYnTHQmhKyW7IA4ZjkGjLgKWFVDKfw49eGTE/fHN7j82J HAi83x+1YFHyKRD1A9apzI1I0de3FWY= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id C71F2611D0; Sun, 19 Mar 2023 22:01:16 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9914FC433A0; Sun, 19 Mar 2023 22:01:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1679263276; bh=76mqBcXOclj70wK6vJnPXYiLxzzEBqv+7zktuDN4On8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VUXzADYnvUt8jvBwlLbVomIIwpWr3m/fWNfNtQVriQSqwUkj2ZQ+OlQip8jRy+3ow aggOLR+nBPstDeDYoCVDuML8MQZLa3SSC2CwN/E0vJj4yzNurI9Il7fULzcWCisbra c7yB8r9CFGkQKDwiNPZKiJcPeWXDbrN/71RMaDACWGeE8j2JER9CcCCibhgZKC8/Ai kbrHlM/EvN/6IV6er2DGjJCzH+6Z1ob66/8rM3N6IqLfI81LXX3uHQGJVWw+sgkSHV J9v5GLpbpWEkXzeGh4WuYYJSZn6QLtRjsE1mebhfEDeaoxlsshPaZKo0qbGN9N4NbW c0B/T0aRKxRqQ== From: Mike Rapoport To: Andrew Morton Cc: David Hildenbrand , Matthew Wilcox , Mel Gorman , Michal Hocko , Mike Rapoport , Thomas Bogendoerfer , Vlastimil Babka , linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 15/15] MAINTAINERS: extend memblock entry to include MM initialization Date: Mon, 20 Mar 2023 00:00:08 +0200 Message-Id: <20230319220008.2138576-16-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230319220008.2138576-1-rppt@kernel.org> References: <20230319220008.2138576-1-rppt@kernel.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 94F4518001B X-Rspam-User: X-Stat-Signature: yoyinscjoizk19irbj4rithbq4mietxp X-HE-Tag: 1679263277-132293 X-HE-Meta: U2FsdGVkX19kFvX/r7xw7PWZptH8iMkOEQazFT6BhpTX76U5sKp2gMiZDk6pUHyfk5hVY8SMJvsWswutz9c7Bfp4pdtAvLniAeZXiXnInkn0Qmn6MIYCDmUIcC9M95VsElFWhD5lMluJAZhC2U2HemVhOajYgEBG9YNNloF+Py7o70fC7x9RCXSya46w0cPZLZBck5NMEkCDw/msrL3tBqWHuThGQcjyxHbTs8wWhICDctR/hhBKg1UVLhRoCPhPvTmxJy1G+5X3LuXh7Lql48w++Di7DWUXI6s/qeCwI0pVv5ihlqk2i1ShgxGsIPNDYTa4+ZOxheJqypgW5R7+Nn9F22JeYerjNIhT7stH0BilcAO4/6Y5NLzLWQBBnlJ6pKL3picSGE5XKo2qB+fKuo5Ma90am2frJRNvMW+I6kYus2OdM+IEjtjsWIOb+95yoN47WCK0WNWVxY9AfNKG9n7qPqIcdJ8lMQW4NN9SMCy0CwkwlHEfGpaD++iopP74fQ179GXA5GyCyT1CrWDdNd5X9uuENGRPNYgv+NltenSABh9eSnisvvpi8GQd2Ll92psEDvlHyUJ2h0wpFcq2E/KKEo1r48Jpj+O+W9TjbvfKHsFNfBnLsvBZ3OOB5xnPzXyJDQHpJ4ZSH/ZuOlukLXdutiqG56JLdtlq0OB2mdKjwxOerq8XQ6sQkwrRVNZ+qhyS2yaMtHhZBdTvLqgRsAuFq965ipUC9BVroSfDu398zRFu/RYSa8TmZhimq6SwH01u1t3Mam1MmqlI3LCnhwkMUrp9PYVys2zJEtPk3ZrYPzxFk5/FSG7yftb7HkXhIYQE9qimheC5YxnwwPjrlbEZS3thrSvXsZ+lS1uCoL4RX/0IDP18c6tl1XmnhxnsU5qGtRFC4EEG3PIPbOhP52EyNemb2Fgu5Zct9OHYxJQYaG/tfIy43WY8cnWdPv8XOtNpdYnogoXgaRHeICa LugDfYXR 64MAcCYYGe2lX+Qf7390BGcialonDHZQ1LcJaZ4Ul7zX2lDjZorfp+ogWW4K1WVjjgJbt6SSbzclZGGsKU6hEml7xdpJTPOwQAiVBu9Jn051rMNU+kV4dMju6ZYpBOSWuj2TPnDtlcjpxppK3gh/pJ2NVeBwyyjxdkBtkRJH3sbuwDH5MUS20HvqwSuQWZzVDlRw54zDju+CCww4DwuYEjRMSNymtAuqsxUQrJ802PfcGJA6jsAy+xrz2wpycIJ+Rf/clXlQhGMa3s29YQxxbxbqjBq28k59i4qIAks5SQSmOSCAFdjAcr4brIXJFoxmbKE+qP1z5bjqLZzTGHhMEBzT0tRpyrUiFiJo1GtmVFoJtGpQGQwlNl1I3gl4FrVzrLwRToF5UMKVVDsTrgANPanF0CQ3Y4CKQvGmq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Mike Rapoport (IBM)" and add mm/mm_init.c to memblock entry in MAINTAINERS Signed-off-by: Mike Rapoport (IBM) Reviewed-by: David Hildenbrand --- MAINTAINERS | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/MAINTAINERS b/MAINTAINERS index 7002a5d3eb62..b79463ea1049 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -13368,13 +13368,14 @@ F: arch/powerpc/include/asm/membarrier.h F: include/uapi/linux/membarrier.h F: kernel/sched/membarrier.c -MEMBLOCK +MEMBLOCK AND MEMORY MANAGEMENT INITIALIZATION M: Mike Rapoport L: linux-mm@kvack.org S: Maintained F: Documentation/core-api/boot-time-mm.rst F: include/linux/memblock.h F: mm/memblock.c +F: mm/mm_init.c F: tools/testing/memblock/ MEMORY CONTROLLER DRIVERS