From patchwork Mon Nov 25 21:01:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 13885272 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBCF3D59D71 for ; Mon, 25 Nov 2024 21:02:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 659BC6B0092; Mon, 25 Nov 2024 16:01:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 426196B0093; Mon, 25 Nov 2024 16:01:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1B5BC6B0099; Mon, 25 Nov 2024 16:01:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D6A726B0089 for ; Mon, 25 Nov 2024 16:01:55 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 86590140DE0 for ; Mon, 25 Nov 2024 21:01:55 +0000 (UTC) X-FDA: 82825839264.24.FBD5989 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf23.hostedemail.com (Postfix) with ESMTP id 81018140015 for ; Mon, 25 Nov 2024 21:01:51 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=i9gH1MvY; dmarc=none; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732568511; a=rsa-sha256; cv=none; b=Y3uP/6JojPgfqb6on/PO4zyYYv7z55TR3tRHDCeDMrJKhg5WYKTxKE0DMNEFwyAY7/BmpC Im/3mV8G7rjeXqXGXGCMEj4VAZJ0jbjAJcSpP1N6Ql5nU3bg/A5tQdSPk4gfHy3dBrcjr6 CZLkwnjNE1sHAeDket4a7MiJWw+3Iew= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=i9gH1MvY; dmarc=none; spf=none (imf23.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732568511; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tZVQtxgbrEODeYoNQSSaa0wSrfaJtTpbtwjWdv648ew=; b=OC1NhesjP4GTxVIzuujyCnTdOUZeNLc//7FSneKDJ54a6bfl8HnTvCqA1PZuxfpiAg2eoq +5U265NxIQ2z/SDV5GS8gKplsedGaorMFiSHlMRcp5KuYQklbL0esOKDMJnYHB5zD6lzJt v77U5MBUGVQZEy7pERohPAbtuKsQ0TU= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=tZVQtxgbrEODeYoNQSSaa0wSrfaJtTpbtwjWdv648ew=; b=i9gH1MvYAdCi6WNbr5238wifLX N0fhzlV9u3MKVl6i+avcQIq+EKOM3DX4Zi2ZU/2FCaYRTkjzOQr8gWbhvsRX6D3yH924u6uhwA3S4 orTQo6r+m/RT+ugFJGQuB4VnRrIWGWOeukYZKaxt3Tc7irlfUQ+51SOPa65xngGSk2j7+uiTba5eY NEFqqnXwOklOw45Aszpk1+WzRqrEDByE75yTdaqddcGaGoesfsaILjBnV/5seRTNvqdYEyk/uOnDI b30bbFwua0O6bu/KnlsODxjLqomf78WdkEg78EvFyX1eTQsJXD9cHuUvikg+AbILC5moK/civbXBD gvsD+yaw==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1tFgDO-0000000CUDs-3MuH; Mon, 25 Nov 2024 21:01:50 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, David Hildenbrand , Miaohe Lin , Muchun Song , Mel Gorman Subject: [PATCH v3 01/15] mm/page_alloc: Cache page_zone() result in free_unref_page() Date: Mon, 25 Nov 2024 21:01:33 +0000 Message-ID: <20241125210149.2976098-2-willy@infradead.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241125210149.2976098-1-willy@infradead.org> References: <20241125210149.2976098-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 81018140015 X-Stat-Signature: yh38on5knyg1ijxim89twqdwc5ha57nj X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1732568511-289209 X-HE-Meta: U2FsdGVkX1/FvA/2lVUeiDqiOwhPAp5zsFFceajCgTU932LSWUIheZva9Y2MuUkaemMDaVHbPD0XXY/CLAO/ih5NAiwH2sv9vemKOK1khVviuLd1SoyDtcfXY69Dkn9NQRlFabNPKTBdqS0/+8mBzybxfW7GiUfnKeFWcV67DsWoFPTMW4/wweeNgakx62fOiGJ6m2goS/duID5MCiLskzLP0SSx/NAwPQ2a55Dl+QAX4V7Mb1yhksrb6xecXtUUafK2M48kZ+RDiD+9Ugl6fwn9YKxC6gm74LuyOCsW5Q9z0Erc/qAff5X3QeJLRAHE+TppHGC3C5qpdBn/OWR6zDGAi0Fb2duMb8Q4klYNZn8E7BKX8sZ3By70lme5scR0EZno9F88MnnYkIGB2vZgHKwM4fJN1cV1GSHCviVJeDMeBsxN0i4skZZCsVodUWF07U9uq7ULxLqAClhGz14k3lgS5kdbE77A/KTYMqOEAkDwYzQUmmcxniFz5O4394moX06DvyJIFHVoeLA+n9P7AAL9CG6YT8bJp0SzmMvB7TfL41e08beTBpdpVH6jfYZALcFmats77L5hLkP9KgVi5W6d0ISsSmmmBEaZS8pMfQG0BZ8xOw0NzI5KxStKGvvNIXDfrVN5IQJdySyKS/9Z2PHpiXt+hw1p5WkEMJO0im2UKP6k7dpHG8i5VXk0pTTp/E6fC5uDCfS+i2Bc24my4eZWz/dK9cvzcyN0qzYHIbaYKHbwsKMjtupd4tUl36oOJLYlZhh7l0I1rAiag6WhDOqTTqeSwZuaynYWntEM8DH2aHnovGZxEs2ZIEmwyCsh2d56B077UMZgU9xIylYX5/SNGvA70Nulaetvo6fIjcB93ZACQPsLqHeMK5/9k4ftxaq+6JGstZjx3hUucCdlbz5EkQkhfS5Y/0ESUg+c7YzOD/2ptG+pvOXVl3+ari52zUzDebuVQdSDqz+4WfQ F3PFcz5I c1C25OPvLFURVz9YuOGyn8/yOjf7L/mnvNGbMW46pA71yXlrPr7AY0pQm7E3qJ7ae5s1wIHJTIa34aMe4R7sXyboht9j/J0RhTMUtPyjoJosKckxVt8gO8MRJ0od/yON8jeWNv2lwuoUII4TGchLkiiOkdzt6EPthfXBOJzPEPntOFOGo2UBSs6q927CZ2UutgIsy3ZxGIs8EsWR6Wna64HhBIoUqgQCCWEI/iiaKmg4qgAIfnbjjlFKG9mGKuxETwIxXSfe7moe2KRSBhouMC2p6evdOE+uPh2AOPVdJ0f6NKGN5+TyZsmHcAkzMAsVBar2+MKO6wLffF+UU9cdPhGy5t2So181k7j17fntXicQmhfcq+0ik/VVurq+yVT5Nu+65L3Jv8bckvH0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Save 17 bytes of text by calculating page_zone() once instead of twice. Reviewed-by: Miaohe Lin Reviewed-by: Muchun Song Acked-by: Mel Gorman Signed-off-by: Matthew Wilcox (Oracle) --- mm/page_alloc.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a8cd54c02f13..c40a0c29a89c 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2700,16 +2700,16 @@ void free_unref_page(struct page *page, unsigned int order) * get those areas back if necessary. Otherwise, we may have to free * excessively into the page allocator */ + zone = page_zone(page); migratetype = get_pfnblock_migratetype(page, pfn); if (unlikely(migratetype >= MIGRATE_PCPTYPES)) { if (unlikely(is_migrate_isolate(migratetype))) { - free_one_page(page_zone(page), page, pfn, order, FPI_NONE); + free_one_page(zone, page, pfn, order, FPI_NONE); return; } migratetype = MIGRATE_MOVABLE; } - zone = page_zone(page); pcp_trylock_prepare(UP_flags); pcp = pcp_spin_trylock(zone->per_cpu_pageset); if (pcp) { From patchwork Mon Nov 25 21:01:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 13885317 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE0FED59D7E for ; Mon, 25 Nov 2024 21:58:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 27EC06B0088; Mon, 25 Nov 2024 16:58:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 22E2D6B0089; Mon, 25 Nov 2024 16:58:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0F5B26B008C; Mon, 25 Nov 2024 16:58:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E62586B0088 for ; Mon, 25 Nov 2024 16:58:20 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 98A921C7496 for ; Mon, 25 Nov 2024 21:58:20 +0000 (UTC) X-FDA: 82825981392.03.75464C5 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf07.hostedemail.com (Postfix) with ESMTP id 0C80940002 for ; Mon, 25 Nov 2024 21:58:14 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=R+4vFjM5; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732571896; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=B4MAOaCT1D+JgAGk/2nVFJpLtQ4vHb4K85arQnMGBtE=; b=tZVaaq+uWpHc/uxTDpXZ2A537IDREWjg4wr3uZOguMdG8FifsGWOPrygHzz3kzyTYWDWwg m1c2q5LZdgjypzCrhBNtM8d+XAjNj8lCe4WdGMJz3I2sIhNwHHRhSPOBHNY33Xcui2AfWE zDr4hRXhSNkOPvmfqMOkMtJvYA2+W14= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732571896; a=rsa-sha256; cv=none; b=dAbTX25VlWwLfjvVMKCkKKHFvYyy/tcGnmte1dqQNYAqeA4oPXKYeFlHC90mKQozs0meS7 Q2CCoAXFkOx2cJVhqI64BXbLnvMAgbtphYigR5FTLR1pPV+p2gbsvWrMBpLawuR69BPNUB ++NZnPijfvnJfaE8l59qE8Q3TyBnO78= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=R+4vFjM5; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=B4MAOaCT1D+JgAGk/2nVFJpLtQ4vHb4K85arQnMGBtE=; b=R+4vFjM5hId6Q+t8KqWteJ0Uy9 KefQAdkXDDr5SKD1NHa4OWq7Ouhs06TNr94wLSp5lcUf3LfmFARz/8hfRUPXQrT2c/48QKqntcdCv oMCFtLwAX60/AGft0Il1Nu7NxjoGburQlhGyd3lEczMAtfkHnmGZ7PIO/FGPLZVUmsI8cQSWcOrLz RZvbn9NME2widGTT4mCz59RRP8WLLYjcIAPHn5+IRVOj1S2avDj36tbXxxaGsTQKZkCGRKeJ8rIIx hSTWminfk/8ts8VztDh/O/9y/qntQRtCPeYjD3kjBxW+ZgpWAWo7HUlNCcy1Srb1hdEmSjmzlHnvA wLMImjXg==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1tFgDO-0000000CUDu-3n2d; Mon, 25 Nov 2024 21:01:50 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, David Hildenbrand Subject: [PATCH v3 02/15] mm: Make alloc_pages_mpol() static Date: Mon, 25 Nov 2024 21:01:34 +0000 Message-ID: <20241125210149.2976098-3-willy@infradead.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241125210149.2976098-1-willy@infradead.org> References: <20241125210149.2976098-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: 8ogn56xpuec534h4cnzqwdx3dn3z4xx1 X-Rspam-User: X-Rspamd-Queue-Id: 0C80940002 X-Rspamd-Server: rspam02 X-HE-Tag: 1732571894-572868 X-HE-Meta: U2FsdGVkX1+aYV5HkxEhjHgggnv2rdErHJyaW2ugcwSo/Afa9Djf14FcCnOY2K4GZNdcCF999LZ9CUKTqjjuZnqm/3I6aBA8gUycfzJKKGxgE0QdYKFkEbAlcA5Q7d+7MB7pt9OBOKFXXOC7XynF9JPeckADHJpbiQdFm7vOIauccSVGb87rd3ZYBvit6dCnQZVUagVE8Cey0yPn8FpqGbVWfdA185J1djmZv0c0g0li5fGfPml/bnfQ+7WTNXjZ0CzSez0x3NlkZCkUjNnQxkQaTn4esa6PpgiL9jsZF43CElhlNqk8BmjoXMEvXPNWLgI3x8xQieE8wVs+zE0vPfDfNHCL6R9GoyCGjLL090vJCrqJGt9sx0Z4wwkica9d1CsxwnYaV1Js4A710qfB2cI8Jti/1ay2Sv/5yZqWOJAOEf9eUd1+4T4ioRW1XCOAFvaMtRZlYxbtY53pRIh7SBlXl2jDKAmmjHkQ2vOOJtTMy2zDS7xNg/SPebbfTVg/7UarzOA6oiFJepCdRo19rUUmg1IATTCU+tM32JlKos8NAEWYAGRrqyF0mTsc3axxN2Wb3kyHrlUf7D2gcY4Mv/WosKE/+uJRyE3Ex80Tfeo1EgeoOKVWTsSDNkMVWVXXBZc1uo/W4I6umQIsAMCP3Pc1JPXwvHiOYPLJQ7HHpqLlpE8KG1aQpfIR+6eczdGuT93EG/BFyMbpBIhz285aFQ1KLYCTohw/OzBhLizYZyDsTd7d2X4HDV9xkOwZiainnQuiwlyDwJ3GktNcAJ5dtWrNpBtr8KooJv0jiOgkG4RjQ+tk2CSvhjK8YEtWzL9j3Rs+4O0JgZ8SdhlIwUrHlygvZG5s21cjp3rSXQC+PZlJDTzv3c3f/72ezuGTRuTGe4FvCsiFCr5zTOJCPjyENb84+ytDKFxva70EJRS54pNawzjM+F3MNcz/x4oB7zP03bUqKGuRa0mXtTaCFDR 37stp2s2 gUumVSHKvhY6/WhNnTX8D34k4Sb9mMw0xY1vRmTG+jf4cAO9xwMxzUfEpKw0VKyFeLpxobKMZzkLL9S1wn6jO7vCWgzo7NasRMgZeJ9JUw/paGu8utMYQOVUlxuTWV23ZJ8Hg9WK1Ql6+2gToeNxRppygOg1N+R3FsfTjQYUcCXJK4JPJJbUEKuXIP1Vh/Hg6ZamvH/LCSfUdQguwcX/6Ue1pG9G6nsrMmUHykurWhFpEj6WKW/vvbuPcrh7meFzJ6tSys4GZSJg3aFGfilk0fM9EqLPqdujPw5SXEBzefZgVM8Pjk0noMG79n57+IX/76ZB0jNu8C8Rz6IU2PiFsUKydInGuTviAjM4ai/i1hRAGU3yA7ZqSb8VYpgKa1v0CeFOU X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All callers outside mempolicy.c now use folio_alloc_mpol() thanks to Kefeng's cleanups, so we can remove this as a visible symbol. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/gfp.h | 8 -------- mm/mempolicy.c | 8 ++++---- 2 files changed, 4 insertions(+), 12 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index b0fe9f62d15b..c96d5d7f7b89 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -300,8 +300,6 @@ static inline struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask, #ifdef CONFIG_NUMA struct page *alloc_pages_noprof(gfp_t gfp, unsigned int order); -struct page *alloc_pages_mpol_noprof(gfp_t gfp, unsigned int order, - struct mempolicy *mpol, pgoff_t ilx, int nid); struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order); struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int order, struct mempolicy *mpol, pgoff_t ilx, int nid); @@ -312,11 +310,6 @@ static inline struct page *alloc_pages_noprof(gfp_t gfp_mask, unsigned int order { return alloc_pages_node_noprof(numa_node_id(), gfp_mask, order); } -static inline struct page *alloc_pages_mpol_noprof(gfp_t gfp, unsigned int order, - struct mempolicy *mpol, pgoff_t ilx, int nid) -{ - return alloc_pages_noprof(gfp, order); -} static inline struct folio *folio_alloc_noprof(gfp_t gfp, unsigned int order) { return __folio_alloc_node_noprof(gfp, order, numa_node_id()); @@ -331,7 +324,6 @@ static inline struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int orde #endif #define alloc_pages(...) alloc_hooks(alloc_pages_noprof(__VA_ARGS__)) -#define alloc_pages_mpol(...) alloc_hooks(alloc_pages_mpol_noprof(__VA_ARGS__)) #define folio_alloc(...) alloc_hooks(folio_alloc_noprof(__VA_ARGS__)) #define folio_alloc_mpol(...) alloc_hooks(folio_alloc_mpol_noprof(__VA_ARGS__)) #define vma_alloc_folio(...) alloc_hooks(vma_alloc_folio_noprof(__VA_ARGS__)) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index bb37cd1a51d8..cda5f56085e6 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2218,7 +2218,7 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, * * Return: The page on success or NULL if allocation fails. */ -struct page *alloc_pages_mpol_noprof(gfp_t gfp, unsigned int order, +static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, struct mempolicy *pol, pgoff_t ilx, int nid) { nodemask_t *nodemask; @@ -2280,7 +2280,7 @@ struct page *alloc_pages_mpol_noprof(gfp_t gfp, unsigned int order, struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int order, struct mempolicy *pol, pgoff_t ilx, int nid) { - return page_rmappable_folio(alloc_pages_mpol_noprof(gfp | __GFP_COMP, + return page_rmappable_folio(alloc_pages_mpol(gfp | __GFP_COMP, order, pol, ilx, nid)); } @@ -2295,7 +2295,7 @@ struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int order, * NUMA policy. The caller must hold the mmap_lock of the mm_struct of the * VMA to prevent it from going away. Should be used for all allocations * for folios that will be mapped into user space, excepting hugetlbfs, and - * excepting where direct use of alloc_pages_mpol() is more appropriate. + * excepting where direct use of folio_alloc_mpol() is more appropriate. * * Return: The folio on success or NULL if allocation fails. */ @@ -2341,7 +2341,7 @@ struct page *alloc_pages_noprof(gfp_t gfp, unsigned int order) if (!in_interrupt() && !(gfp & __GFP_THISNODE)) pol = get_task_policy(current); - return alloc_pages_mpol_noprof(gfp, order, pol, NO_INTERLEAVE_INDEX, + return alloc_pages_mpol(gfp, order, pol, NO_INTERLEAVE_INDEX, numa_node_id()); } EXPORT_SYMBOL(alloc_pages_noprof); From patchwork Mon Nov 25 21:01:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 13885331 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6114D59D7E for ; Mon, 25 Nov 2024 22:18:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2FAD06B0082; Mon, 25 Nov 2024 17:18:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2AA6D6B0083; Mon, 25 Nov 2024 17:18:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 199276B0088; Mon, 25 Nov 2024 17:18:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id EE4766B0082 for ; Mon, 25 Nov 2024 17:18:21 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 92430ADB5B for ; Mon, 25 Nov 2024 22:18:21 +0000 (UTC) X-FDA: 82826031834.19.A3ABB54 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf25.hostedemail.com (Postfix) with ESMTP id 9D718A0009 for ; Mon, 25 Nov 2024 22:18:16 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=rh3YdLCY; dmarc=none; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732573097; a=rsa-sha256; cv=none; b=Ekaw3TP0xQTfPblbK6I2QLVfW4ONDm8wIIOEdpZTLL77/TF8Wjh9Jk1zd0uFgR+3/Xb6zM 49idzW8cLoFmPy+4qxLwbgzYxVG39tqEBd7ybv/trjNX18GwanpLyEPXY7PzhvViV3CLMU iGtPSLHs7BLgA0TOI1E17bwIkL+ywPU= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=rh3YdLCY; dmarc=none; spf=none (imf25.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732573097; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Te/r5gWiFZvQzzFtah5P/5z7gMYLs5PGvFwPUfYBNaE=; b=RaWIzIyeBRjFosVv0JkmaznVcs586OclrNLnWpCryvq+jwaa6b+1noPkp5GAd3nC6Po3ge XlAzQfI6vb8/BFhT3ZEDtDt/PBdzNrwzoOgNe31tuNut0YJ28MgT1QMWlKUpwdUKtkike7 KdXLPsK2vwVdRx0o8ozwgGKyG7KSn6I= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Te/r5gWiFZvQzzFtah5P/5z7gMYLs5PGvFwPUfYBNaE=; b=rh3YdLCYE1/RnqbENJmfPiqoRc 2bBLYof7WE+1GYqJ44vNmR91o3K+4pX2YSS0UlIw+AIxVtQjjetYqnRWEm95eJ2IYdXPbLoiWIKh/ vS/BdzUFcHnKTQckU95rNoRYLZ/1cJ56RecrMoMMSzFofpwh8OAD2qYXQCZNCeFtAXt/cvHvqk+Pt gN37Aa/Lpp39ju/MXLSjYyI/rsKL6P3ORkeYEu3/1jQRjiqL3irrUYsq6t1S1SJ/69zS6ipxN5EKG PVipp/TvMQeFROFZSMjwFR2Ejp30WfckVVinfACjz0rw/r/Rc4Z9HavSfvKCEZYNa/4kXHi+55mgI x6Qkv3UQ==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1tFgDP-0000000CUDx-09rg; Mon, 25 Nov 2024 21:01:51 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, David Hildenbrand , William Kucharski , Miaohe Lin , Muchun Song Subject: [PATCH v3 03/15] mm/page_alloc: Export free_frozen_pages() instead of free_unref_page() Date: Mon, 25 Nov 2024 21:01:35 +0000 Message-ID: <20241125210149.2976098-4-willy@infradead.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241125210149.2976098-1-willy@infradead.org> References: <20241125210149.2976098-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 9D718A0009 X-Rspamd-Server: rspam11 X-Stat-Signature: 9djqx1nziyrjoknu1b11jgz8r59pw855 X-HE-Tag: 1732573096-492302 X-HE-Meta: U2FsdGVkX1+lbIoxk98LiCrs5bHGxWRhLglFhUCMZCNNKhd/GkQw+90evMPkkxi/MjgZNEP4se+XrwGY5UF1320Eq2bkWwypr+r2Xxq/OH+pO9fmEBZoIbKTlKdhR2bPnyJ4QFHnFF4LowkT6XgLAOOMdu53VvapOtVgW53S/B+8E++AP7Z0GEzoUnLMDRt71VLxsKEB4mfAWReWPsLo/kTUKej3NS5E8us1DZss3xAWiLfQdexkEfMd4wINt2C1bjRY6LVDblC+WLPk0gC+JsK6vNng/nI3nSon5W/y+W6EUb5FO+M5QCWU0WJ8ZpSbl/2M387A/uJG0K98eA/fG2AkybWn7moMcmDm/twi1lhbIQVO2GGAjMgkXFjYv0o4G38LwJdxmLwLj1GitpdtansvBiKFGD0VJEJI6iJdvkh7/iWxORryD4re+N7X4P6bHr8pHWRizx4LrXHdD/3Smlpn/eKul2XxqURksnVGu56iiMg4Df76QQiTv04z0U9tveFCsTbxFAbcLIXkb5J2lXNcm4k+wqfiUkluo1AC9TfOHwWXd4/V0uOuvaoG1ikQS22LoVsERsinx2DY3INHT/v442s03VIHZxWAEwNufsY8NH4ll7kovb0S64i/4CkPNBRLnYtsGLugIlTBZSfCebov+KY36sm44iJrXfzroU6KeM8aB07BYGAaG4GdPjwO6WZ3V4wUQJN3cFTV/x8ReFKj0Ix3GQ2dJ52TrcHdcNRyN76mHXViTtAsmcvzu1hreoKX0l5vOxyaTz0fdiCv0Xe91fT8yW58KV/kklC3M0aujSrr/5rUxz5SclL5avc4Mdy5sdRf4VDpD7hB1BJctKFh3lDknIQcHLMYF+zQwVJJ6xNjtMqZr9NSmdFT99WvZNofqtBtUZ0HxR9HikaGgCUAUfrxjmqdkK0t5dIfkCKcUB525hLTubsD6tRGlava7bw+LfnJ4n5IV8W1FCq OHw/0r7u LREHz9Gt3lBjCkbNr1uYAest2l4jkK7QVenmNOPcbRjYr1E/y2nQUX4bfentNBM8wsUuidF+q8MGFfQTZa8eGUb39p691AyVOKgfdr9+TFg486AEClItkJCttuohLA14G8NBf6c8E3CqyTlR4hE4ZTsfMFK1+qp7q0mrUFQUtuWmz1m8QbnBeH6BNn15v1yZ0kt4MgH+WW/tLvsF79Lmh9jMXVA7jZT6uVdMK/BEW31xneHPy3cFjEeTur1hXmSaGkmP+vufZJhfsLnLwIA/vqmScmmrGhk4EQgFAc9xWZH7BT0BQ1k9ikS41YtvA9tyUmconeHfSs3KzoG0k1KDukF4e6/2hU6yirfln38LrPuCQq4xQk1QffPjXl/m6PG/fpOqF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We already have the concept of "frozen pages" (eg page_ref_freeze()), so let's not complicate things by also having the concept of "unref pages". Reviewed-by: David Hildenbrand Reviewed-by: William Kucharski Reviewed-by: Miaohe Lin Reviewed-by: Muchun Song Signed-off-by: Matthew Wilcox (Oracle) --- mm/internal.h | 2 +- mm/page_alloc.c | 18 +++++++++--------- mm/page_frag_cache.c | 6 +++--- mm/swap.c | 2 +- 4 files changed, 14 insertions(+), 14 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index cb8d8e8e3ffa..ca400c70199c 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -741,7 +741,7 @@ extern bool free_pages_prepare(struct page *page, unsigned int order); extern int user_min_free_kbytes; -void free_unref_page(struct page *page, unsigned int order); +void free_frozen_pages(struct page *page, unsigned int order); void free_unref_folios(struct folio_batch *fbatch); extern void zone_pcp_reset(struct zone *zone); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c40a0c29a89c..adac485e3254 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2626,9 +2626,9 @@ static int nr_pcp_high(struct per_cpu_pages *pcp, struct zone *zone, return high; } -static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, - struct page *page, int migratetype, - unsigned int order) +static void free_frozen_page_commit(struct zone *zone, + struct per_cpu_pages *pcp, struct page *page, int migratetype, + unsigned int order) { int high, batch; int pindex; @@ -2677,7 +2677,7 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, /* * Free a pcp page */ -void free_unref_page(struct page *page, unsigned int order) +void free_frozen_pages(struct page *page, unsigned int order) { unsigned long __maybe_unused UP_flags; struct per_cpu_pages *pcp; @@ -2713,7 +2713,7 @@ void free_unref_page(struct page *page, unsigned int order) pcp_trylock_prepare(UP_flags); pcp = pcp_spin_trylock(zone->per_cpu_pageset); if (pcp) { - free_unref_page_commit(zone, pcp, page, migratetype, order); + free_frozen_page_commit(zone, pcp, page, migratetype, order); pcp_spin_unlock(pcp); } else { free_one_page(zone, page, pfn, order, FPI_NONE); @@ -2777,7 +2777,7 @@ void free_unref_folios(struct folio_batch *folios) /* * Free isolated pages directly to the - * allocator, see comment in free_unref_page. + * allocator, see comment in free_frozen_pages. */ if (is_migrate_isolate(migratetype)) { free_one_page(zone, &folio->page, pfn, @@ -2808,7 +2808,7 @@ void free_unref_folios(struct folio_batch *folios) migratetype = MIGRATE_MOVABLE; trace_mm_page_free_batched(&folio->page); - free_unref_page_commit(zone, pcp, &folio->page, migratetype, + free_frozen_page_commit(zone, pcp, &folio->page, migratetype, order); } @@ -4871,11 +4871,11 @@ void __free_pages(struct page *page, unsigned int order) struct alloc_tag *tag = pgalloc_tag_get(page); if (put_page_testzero(page)) - free_unref_page(page, order); + free_frozen_pages(page, order); else if (!head) { pgalloc_tag_sub_pages(tag, (1 << order) - 1); while (order-- > 0) - free_unref_page(page + (1 << order), order); + free_frozen_pages(page + (1 << order), order); } } EXPORT_SYMBOL(__free_pages); diff --git a/mm/page_frag_cache.c b/mm/page_frag_cache.c index 3f7a203d35c6..d2423f30577e 100644 --- a/mm/page_frag_cache.c +++ b/mm/page_frag_cache.c @@ -86,7 +86,7 @@ void __page_frag_cache_drain(struct page *page, unsigned int count) VM_BUG_ON_PAGE(page_ref_count(page) == 0, page); if (page_ref_sub_and_test(page, count)) - free_unref_page(page, compound_order(page)); + free_frozen_pages(page, compound_order(page)); } EXPORT_SYMBOL(__page_frag_cache_drain); @@ -138,7 +138,7 @@ void *__page_frag_alloc_align(struct page_frag_cache *nc, goto refill; if (unlikely(encoded_page_decode_pfmemalloc(encoded_page))) { - free_unref_page(page, + free_frozen_pages(page, encoded_page_decode_order(encoded_page)); goto refill; } @@ -166,6 +166,6 @@ void page_frag_free(void *addr) struct page *page = virt_to_head_page(addr); if (unlikely(put_page_testzero(page))) - free_unref_page(page, compound_order(page)); + free_frozen_pages(page, compound_order(page)); } EXPORT_SYMBOL(page_frag_free); diff --git a/mm/swap.c b/mm/swap.c index 10decd9dffa1..3a01acfd5a89 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -109,7 +109,7 @@ void __folio_put(struct folio *folio) page_cache_release(folio); folio_unqueue_deferred_split(folio); mem_cgroup_uncharge(folio); - free_unref_page(&folio->page, folio_order(folio)); + free_frozen_pages(&folio->page, folio_order(folio)); } EXPORT_SYMBOL(__folio_put); From patchwork Mon Nov 25 21:01:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 13885334 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1E49D59D7F for ; Mon, 25 Nov 2024 22:38:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3A5F96B0088; Mon, 25 Nov 2024 17:38:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 355B66B0089; Mon, 25 Nov 2024 17:38:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 244866B008C; Mon, 25 Nov 2024 17:38:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 0522A6B0088 for ; Mon, 25 Nov 2024 17:38:20 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 778E3140AF3 for ; Mon, 25 Nov 2024 22:38:20 +0000 (UTC) X-FDA: 82826082276.04.9E93833 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf30.hostedemail.com (Postfix) with ESMTP id B1B9C80002 for ; Mon, 25 Nov 2024 22:38:11 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=BOxJlVST; dmarc=none; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732574297; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/aWUS9CVIst+IEIjbMLK6rFw3y2dIRzWc3iSEL/kmuI=; b=Tx3Dfs3/qFYBIv+aOSX0cwN8jw/NmStlzZX2yr3EJKIz9MlxxomteUvROVf/qmNvAAWRu7 08SWJcH27DmTPBiiM6cNa8vH6wJ1dATWG32mWXml5wlxOwluIpwxD0jmVRinsnhYI2WAnK qc+SkW1dMRIQt6KYPi4jkIQD4FTjaFs= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=BOxJlVST; dmarc=none; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732574297; a=rsa-sha256; cv=none; b=mBLVGv/Oc8GUYLY100DN14pDrcfyjCJ4VcolphDLWtQYc5HUHcft7YQ51foSMqnVGHRb3r ZAkKuhzisBuOo0yU9iuSTe8Oyqb6VcsMSRjI8olI5otzigtZaql0s33C0/2ihcro68wOyV x2I9h+Sdy7zs5FO9ysr5lhZzY1Q3TUs= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=/aWUS9CVIst+IEIjbMLK6rFw3y2dIRzWc3iSEL/kmuI=; b=BOxJlVSTdII20sdq6hVAWV9/Y+ P4Iy5I+/K+aGTBgVLqpJd/9j35J4bKn9zHaMHVCNM/fndhhJZcxvSeDxFTHdBJbELI2hUIJF/dB9z Hep1IQZ1iD+TAlELeAHXJMjTNvYlHZ9N3vDV7jjNfZAOgwRKqOKYaGQRq8sWO3bZQC9uMjPpMpWSO LpBspn1IedxSfiXpk86Uee0Z/P8n5MGAP+fbpr8/PryNLp63QjnUMeQV25zNSaAXo4BafvEP/hig1 GwqKfr0UIEkROmVRHmtyd2vcDq3aNE5alj7MdjAb4dEEhnw9K3q9a8Yo3mbBF8aW8Wdsaqc4og1Sb jLmK01Gg==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1tFgDP-0000000CUDz-0ryq; Mon, 25 Nov 2024 21:01:51 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, David Hildenbrand , Miaohe Lin Subject: [PATCH v3 04/15] mm/page_alloc: Move set_page_refcounted() to callers of post_alloc_hook() Date: Mon, 25 Nov 2024 21:01:36 +0000 Message-ID: <20241125210149.2976098-5-willy@infradead.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241125210149.2976098-1-willy@infradead.org> References: <20241125210149.2976098-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: B1B9C80002 X-Stat-Signature: ga33g1du4jdaqmfxt113hu6dgw1foa84 X-HE-Tag: 1732574291-170614 X-HE-Meta: U2FsdGVkX1+SFSwqovZZHsAs8ByBjprSQhHGhs/CQQL9MWR+9+1TKMj8EAQU3aLl/kIQkY8HgHLQGtz5PeuJexKA8ToHTQdISrYQIgsmitPmQz2VeOSGo7Mne2pdaAvgrnMQjaCTPxfis70dQJcKf+QFZLI5XsCCGN/2R3hiyAWe00VeXJNcibyPUvEl9z7Z4av5OsP1OxVNOIR4D59Eql8fNSvGlF7AmL8Ee9gZUCqFv0WQAS2ids9vd5iQMj9e35HHNeAC0GUF4mhqYaH/pEQrSxudB1g3DubhgUmP4R28Qg1H4rotUfPJj+UsrTBalcoF9vNzYY9W034RRG7vZDAEQv6fmDj73jZ9rOgdPEV8ttn3yyNR4l8p1l/ZUhD6XD2DHzUk2Wyp6cmst9G6gXLoYsVpyDAU9BeVoBwu1WtqHjEXfKl9hfqeXruAIvJXp06EhQKuAVfFQYLX9qkDKtdbKCvCVlNznuaBe2/pY9n53R4naWQYjYiSykgSUb4v0UMxMpEaAzYaNUcN4SkfsSRuUOHQ6BzM7yjncm+oGdeJ9+fpe5eNR8XqmWJUSGwHbxwsnl3YLOgU0e/zHmWrTbhbdNQjfJLuS8RVV1FOT5nfcAZ+QDvXYEEY0q1ZIRjfj1KKEYXZk4KM3z49jiOUCxFigLuSGx+bBnFRnn6j3X+6jhtWKIbOxgQCE3epuYdYikXqP8bYKXo8UOdWED5NIrs/V4V7wOu/ONMwOi2KNJ+jg7G3CQMAMJL5VQKfUMcGKmAsjIbGMpli23iKabbR6zzxkxzkjmCcsoXHG7KEPUBJdGmhLILNUwIJAISm08/X/CvKlrYV2GO9bvf5e3ScdgPjV1mzho76ufN+iPtvYfAM/M1mbVYbFG2nFwnweRLRyitt0zXypmQJhSbImZr50Aiw+ABy+1op5dq0k4bS5W2sHXtUf4ZaCvwazqZPE4sgZOuYAmmgW987pVRtfwJ DDLfZnaj fQEQLdEn9aN+EygiXjkhFdAM+nK1SNfMm4t2miRAWtR1q10LOIhXPlnPY5xG/vz5MAm15NVmwR/+WVpz0950WaJCgGFCxyBKhxRrTRBgVFV1D0LFu5FJh7q3PFB0N6fFFfIS/a6OZEJlLQQnKlPIV3TVBTzVU+ZQ7sy9wQE7+J2S9Z2+l7ZQwuOAkCg4s/zyzSuyD4XaTJN0u6L9vI1rKIDoyX1zFzO6c+SF0BjF2lUleWauxCjTTb4p2d8HYKW898Vf4l9+AOwLlTBnrJrgUgbr0F6mZ1sFwzcqhw3aX7bfBDr4BrbxCeyQtAA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In preparation for allocating frozen pages, stop initialising the page refcount in post_alloc_hook(). Reviewed-by: Miaohe Lin Signed-off-by: Matthew Wilcox (Oracle) --- mm/compaction.c | 2 ++ mm/internal.h | 3 +-- mm/page_alloc.c | 3 ++- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 6009f5d1021a..2915a13b34a5 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -83,6 +83,7 @@ static inline bool is_via_compact_memory(int order) { return false; } static struct page *mark_allocated_noprof(struct page *page, unsigned int order, gfp_t gfp_flags) { post_alloc_hook(page, order, __GFP_MOVABLE); + set_page_refcounted(page); return page; } #define mark_allocated(...) alloc_hooks(mark_allocated_noprof(__VA_ARGS__)) @@ -1868,6 +1869,7 @@ static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long da dst = (struct folio *)freepage; post_alloc_hook(&dst->page, order, __GFP_MOVABLE); + set_page_refcounted(&dst->page); if (order) prep_compound_page(&dst->page, order); cc->nr_freepages -= 1 << order; diff --git a/mm/internal.h b/mm/internal.h index ca400c70199c..9cc5fdc614cf 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -735,8 +735,7 @@ static inline void prep_compound_tail(struct page *head, int tail_idx) extern void prep_compound_page(struct page *page, unsigned int order); -extern void post_alloc_hook(struct page *page, unsigned int order, - gfp_t gfp_flags); +void post_alloc_hook(struct page *page, unsigned int order, gfp_t gfp_flags); extern bool free_pages_prepare(struct page *page, unsigned int order); extern int user_min_free_kbytes; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index adac485e3254..e3a4aaf437f9 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1506,7 +1506,6 @@ inline void post_alloc_hook(struct page *page, unsigned int order, int i; set_page_private(page, 0); - set_page_refcounted(page); arch_alloc_page(page, order); debug_pagealloc_map_pages(page, 1 << order); @@ -1562,6 +1561,7 @@ static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags unsigned int alloc_flags) { post_alloc_hook(page, order, gfp_flags); + set_page_refcounted(page); if (order && (gfp_flags & __GFP_COMP)) prep_compound_page(page, order); @@ -6394,6 +6394,7 @@ static void split_free_pages(struct list_head *list) int i; post_alloc_hook(page, order, __GFP_MOVABLE); + set_page_refcounted(page); if (!order) continue; From patchwork Mon Nov 25 21:01:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 13885315 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2C08D59D73 for ; Mon, 25 Nov 2024 21:38:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 48DAA6B0088; Mon, 25 Nov 2024 16:38:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 43D826B0089; Mon, 25 Nov 2024 16:38:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 32C136B008C; Mon, 25 Nov 2024 16:38:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 150B26B0088 for ; Mon, 25 Nov 2024 16:38:21 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id BE588160EB8 for ; Mon, 25 Nov 2024 21:38:20 +0000 (UTC) X-FDA: 82825931076.18.B69AA50 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf16.hostedemail.com (Postfix) with ESMTP id 311BB180008 for ; Mon, 25 Nov 2024 21:38:14 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=dWz7GKjP; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732570696; a=rsa-sha256; cv=none; b=LvMjI+imfWA6Mk4nnaoIKc/zavhNhdDMR6aa3oscnXDyvFmet/R/cUOp02vty5BaGpOTho 21lwqCo7tBtTumCCL5aW37X91ia2z7u+D7NERL5I87QG5AOyTfZy4DPP5RxGSrZndrVrNd WUwYkuQJlTblCMJs2lIuCh+4VnIff00= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=dWz7GKjP; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732570696; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NyXU/7JDZr5zMHu3c4Kg/pLLdo4wbKi9H/MK7sGHG5A=; b=33TR+Qd7zZRrHOxvRvsgZe3dqQTjcLe8xOAVyskRwXuzmPLvkZTGVYpfdzlFh547kof2K8 rw0oWENueC8+jNRXnb+uc521xps/d/QXbduQ/QxRoFLzfRi9XVNboB+HHCTc0DA/5PMXt1 kXzFLeeHOFhBNVhSYRAnYRVNktvd+j8= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=NyXU/7JDZr5zMHu3c4Kg/pLLdo4wbKi9H/MK7sGHG5A=; b=dWz7GKjPNNF60Zr5rRZ6rKmXei xDze31MLuYvVQw4Gq6EPpGbHRnZMcDI7Jp2hLW7xior2/api14rZJhyAAQNV9mY1B9IMF0jPH7w98 OXnTv5Sx/pbM5kC6rc4Ublrq0k4JV9h3KvM1M9L0d1UPsLuqGEx8cW0eilIKB3UxUab5SPDKoutVU 4YMREb3330dMOXEz0xslJK6dK+PKDImV9/GK0d1zJz09jmhr9h2XZnm2uQf886zh8e86kVipL0mZV vmKDn3PnLhLlxeN+NahL3FTRPC813aR2cAu0L5C9f2+rYJ/WDrCMVkxJAdr1gfwuTqmLNHy+b5IR/ 5F/rXB0A==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1tFgDP-0000000CUE1-1GLR; Mon, 25 Nov 2024 21:01:51 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, David Hildenbrand Subject: [PATCH v3 05/15] mm/page_alloc: Move set_page_refcounted() to callers of prep_new_page() Date: Mon, 25 Nov 2024 21:01:37 +0000 Message-ID: <20241125210149.2976098-6-willy@infradead.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241125210149.2976098-1-willy@infradead.org> References: <20241125210149.2976098-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 311BB180008 X-Stat-Signature: kmqh8415x3fpsc6kedckfr614agogy7x X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1732570694-967033 X-HE-Meta: U2FsdGVkX1+odoMNWUPcE79Pw9YlNDYBVSoXUgPJhkgO0c7EH2+C1L8mNfOiZ4h5+0W/qDCx0w4s336e1ci4vP+86hkrlwYSd/FEfE+r5sYK/hhvZQb9NieqslGU8T8b5ymp4BeY4BLvTvD+DXZxCNUES+0jrCh1f+Ik0TgxZ5SdJhMsXQwBr/JWvmo6iBnxX7w0wJ5xM6iiER+Qf63SPElHstgM0nEN8RFUqp+WnDwCMxpVkaejgq6DVWNEzNp1iJQULIvX2P6qGZGlzSxFnEKvf4Vx6YzhD4YRhGWom+Wm/zMpeRoMKpUyH7IYLAmI4qX8zZoGYq7vNNB8JdFACqHJB/CbgwQgq533MSwr28G6YGVMakPYVIUAJJ7TmUPJsC6N8c972EKIxVxwwSs/FhtYHP4lRZMSqujuDqy7gZuqFEPKJl1euix6hgaiKzQqD1jdf5LwvZMry+ybymHyMwnCQ/fUJGq02PeZ/dd8v94f0T0NBBTCbW4U5Op7z8eujmovH17qUBScxnowD37l8HJcjkRsUhfiKIlXLiDUAKVdHCsZKeK2I7RtuM62LFeF+9PJf0snbujv4XS8v102LIZtce0BBwEDd5zZ9YDPThc7MfvgRktZ4xNazcNIj/x055xFsyAdviJohA6v3BlQtvgLx6HZ2GAEqXilQj6Bl8EMYw5/rjBr+K8ZbuJp3iy6eFuG2OQhnQguoInyWiQgrLjGUap8MzUmDLOUIkIBYfk7YCFywmZBoPHTbj638o5cogg5wg/c9AETiJ9oqS+KKT86I2RbZTFPK5CYY3f9a16TOOS02chgy2NK0WPIrQ4PoYDa0JM8hulbC+hmswCJRu9Iu1T0x0y/35NlGIBPciFXvBNJj+AhxQlVJFyDa685ebZ9weBn1Ednp4MZnkVTkDE+r4/xdq6VDOEAjlCZOpl4Qlipugpw0uqly8hkOdau2WkxXB7PHfnBFyZB+4y wJGzCfF2 2yu/p7kA79Cap7h9Glx3URlaNvDtDgKsaRsBpF0xFBYdIZgCrikhqES8LXRL3BLIM0vOA0DO6RJXU0R8D2HO6aJlu4eIBGlgpROSmxqgku4r5ZinVhRKKClyGDW5qDa5nhfXjCjRMZQC/q0381xzIOXWVYmvL6/9al2cnMqyBCdel/B/eNbhngXMrMUA9DrYoqy2N5X0DXW6XKxaFOpSKPQLCuZPZZ8fIQZbBkoFZhKkFyKAmiBjRUA0m9Sdsobeh60SVtNjEYwHFeas4sqJbYqd0ZeXPCDdGPSLa X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In preparation for allocating frozen pages, stop initialising the page refcount in prep_new_page(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/page_alloc.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e3a4aaf437f9..c2b46cdc7ffd 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1561,7 +1561,6 @@ static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags unsigned int alloc_flags) { post_alloc_hook(page, order, gfp_flags); - set_page_refcounted(page); if (order && (gfp_flags & __GFP_COMP)) prep_compound_page(page, order); @@ -3508,6 +3507,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, gfp_mask, alloc_flags, ac->migratetype); if (page) { prep_new_page(page, order, gfp_mask, alloc_flags); + set_page_refcounted(page); /* * If this is a high-order atomic allocation then check @@ -3732,8 +3732,10 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, count_vm_event(COMPACTSTALL); /* Prep a captured page if available */ - if (page) + if (page) { prep_new_page(page, order, gfp_mask, alloc_flags); + set_page_refcounted(page); + } /* Try get a page from the freelist if available */ if (!page) @@ -4712,6 +4714,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, nr_account++; prep_new_page(page, 0, gfp, 0); + set_page_refcounted(page); if (page_list) list_add(&page->lru, page_list); else @@ -6534,6 +6537,7 @@ int alloc_contig_range_noprof(unsigned long start, unsigned long end, check_new_pages(head, order); prep_new_page(head, order, gfp_mask, 0); + set_page_refcounted(head); } else { ret = -EINVAL; WARN(true, "PFN range: requested [%lu, %lu), allocated [%lu, %lu)\n", From patchwork Mon Nov 25 21:01:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 13885274 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7BB30D59D6B for ; Mon, 25 Nov 2024 21:02:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 338866B0099; Mon, 25 Nov 2024 16:01:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2251E6B009A; Mon, 25 Nov 2024 16:01:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 029EC6B009B; Mon, 25 Nov 2024 16:01:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D1D826B0099 for ; Mon, 25 Nov 2024 16:01:57 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6E15A80D9D for ; Mon, 25 Nov 2024 21:01:57 +0000 (UTC) X-FDA: 82825839306.19.72C0344 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf14.hostedemail.com (Postfix) with ESMTP id A3DD1100008 for ; Mon, 25 Nov 2024 21:01:51 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Sme2uyeU; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732568514; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=S4RU9x6aK+s9y5UC4asSHORm/FbRhVUFGtUW3BrWS7Y=; b=QMB4Em/QqDRVh+MnjeqW0JiAq61QfEW/oKG3/PxjBcs9yjpLnfSLIyMcfM9EI9dHKB+6l8 UA9LKMDlmPZglWwpBlA/Hz210co4JRVNpurDObfk/+K/+UQLOwvtogxaCSYnHNfOXwOJIc Mn7HhFOWoBjQoqmKTlCSFeqcZ2hFN6w= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732568514; a=rsa-sha256; cv=none; b=0CCGdsGXbmDT6CF1QA+EvXGxIA1jAXs4Ar+xFElaKWVuiRThAU2Veuk0GYQrYlyn9FbhgA 59S3MO8ylRnoXCzIW3+Lvr3Td2FEKaeY2mtJBQ4kec5UhcK8w43RHjHlIibOdYkN1nOlXr ecYIzuz+kLthkWtr82CAXZX72UxzOB8= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Sme2uyeU; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=S4RU9x6aK+s9y5UC4asSHORm/FbRhVUFGtUW3BrWS7Y=; b=Sme2uyeUwaOgKG2zR1UUn2qcBs K31S82iRe+oNcEeCOLSOFpzJAiTSN/MWN63cjh5eb3mu3UqlTIvliez1DDtUVqg48NHKXDQTxJ1M8 Oo7MmsnLCPsabnFyBGHyJx0FFJ3PO3vCdz4k7J4Vb+lTACSqnK+JDOyMCBFE3eljHRD1jubD5vJY9 mLImGmHxlh/SEbJ/rU/gSpfriW49KNTSr15I4+tZCpFOj/Vq/7QFKTNvloSemW1DLL7QtVE71sGCF 7d0Y5hYFADuRfSzPtLc4l+vSTQIwUmk2PpWwGrYaAumuKLJVuMm0rF+rLlD609Ukj/IFQxwX0ebuM 5SYGxWig==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1tFgDP-0000000CUE3-1fUF; Mon, 25 Nov 2024 21:01:51 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, David Hildenbrand Subject: [PATCH v3 06/15] mm/page_alloc: Move set_page_refcounted() to callers of get_page_from_freelist() Date: Mon, 25 Nov 2024 21:01:38 +0000 Message-ID: <20241125210149.2976098-7-willy@infradead.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241125210149.2976098-1-willy@infradead.org> References: <20241125210149.2976098-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam10 X-Stat-Signature: mb5jckgju9r5hzsa8yqubax3uq5d1h9k X-Rspamd-Queue-Id: A3DD1100008 X-Rspam-User: X-HE-Tag: 1732568511-199503 X-HE-Meta: U2FsdGVkX19z+gdhbO1OchSbR3IJ1xEZ9Rz/u1/x1rbfILUi0xrolzDsfqaHpxTzu9Obgxa7S6i/PaxN8r9ZYJdlCz4CEERbPlkH+3bDB//xVex6zJWBt6LSI6RhU9iecQYwmmxvneJwoFo/DljAk870EsabUv2Xwz6ddHtDXcnGXPvNTXzIJuEhIAsLCLAFVVajfpqLcHztBcaChxhSHyyt9en5p/wWjJl08rq2iOysA3OjLvnWfrPx2dXU6p7Tk3E7RBdLMQfW8MHHbMb58yzG0bEWCqtYeLQE3jqi71VHsmpZQray59mQO/ivCo6DRmt11GmX/SEKIAQPDjGBt8XWlEmpsKlFJOux3Y0gds923V7SFYGORxC1HYqasZfHoHlZsWh6DnC7l8Ck4z3cPhiDUleKZHx+fv3VfFcltda51G3fUp+ESh13T2beW3DISqkQeLIRp24y0h8DukYu3cLQcLa4G/5YjNVvUI301W7OvAxa2CAXnXXjEMwsD8wwf2oTgAmz0cj4dYaA6Go536EgBB4m/c4tEgQD5B2bFPseYoyFjxcscBEUkiYsqJt1qDNWyo28Bkrz9cR2KHbs2Yow1nwJGuFgTAxF/6HRdz+unD/OktHJQQp96pg7XkQiY0tq/RQUjS+nJekiI5E5+Kf0Ftc5NgdmUhsxM9ThHPIiszvHhHOqopqmk2rPxoNQ3Ma2X+17fWfAM756lqZDY4/GQjDpEXTAjUgsg6YAwCy4/bfvGIbY6WJvcahdSGOsEIsXSdAwMeUyAR3zemQuWpq8Sds53fHWSJKd2oTQ6EPzqK2z0F3+fvl/2zis2PgC5ISmD3KXi57pBzNVmiHw9qiWz6HN99oh41iqoDYuwfWvk1VB/bcUGfC2bnK98+rXS7Pha5O9E92qzP1+jsDUzxDaB4Bl2Fx199g5twH7JftJ1TxuE8T1mqbFljFy3A23B/ObBbj2tvZK+ZIjFUG s0RU1ODP qMzVx2Iyw+8tf0XNK7pM7QR6epKkii/qhvarKSMOjVRr5F8yLpk4gouuquXxly6N3rTYHiag2au++ViNH9eot81AJhZaK9VRoKGq4NUoLKRhgktOxo91XZ7+dsEpMJFHlj9TLT5z/zGRx6sKEwFr21TWGjVKK+0w45mpk8juTy+trrOtJMefYRA2SJ11EHjlfdn9jnwbx5HIfAWJG1SbeUqGaHWWIQawX6g4FnKorh/OzCirizQFlYmz0a6Utg6R+7aMTvlF+l8IEh8r89cvrH2u7U9Lg4/LTf2M7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In preparation for allocating frozen pages, stop initialising the page refcount in get_page_from_freelist(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/page_alloc.c | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c2b46cdc7ffd..14fa6bf7578a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3507,7 +3507,6 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, gfp_mask, alloc_flags, ac->migratetype); if (page) { prep_new_page(page, order, gfp_mask, alloc_flags); - set_page_refcounted(page); /* * If this is a high-order atomic allocation then check @@ -3602,6 +3601,8 @@ __alloc_pages_cpuset_fallback(gfp_t gfp_mask, unsigned int order, page = get_page_from_freelist(gfp_mask, order, alloc_flags, ac); + if (page) + set_page_refcounted(page); return page; } @@ -3640,8 +3641,10 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, page = get_page_from_freelist((gfp_mask | __GFP_HARDWALL) & ~__GFP_DIRECT_RECLAIM, order, ALLOC_WMARK_HIGH|ALLOC_CPUSET, ac); - if (page) + if (page) { + set_page_refcounted(page); goto out; + } /* Coredumps can quickly deplete all memory reserves */ if (current->flags & PF_DUMPCORE) @@ -3732,10 +3735,8 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, count_vm_event(COMPACTSTALL); /* Prep a captured page if available */ - if (page) { + if (page) prep_new_page(page, order, gfp_mask, alloc_flags); - set_page_refcounted(page); - } /* Try get a page from the freelist if available */ if (!page) @@ -3744,6 +3745,7 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, if (page) { struct zone *zone = page_zone(page); + set_page_refcounted(page); zone->compact_blockskip_flush = false; compaction_defer_reset(zone, order, true); count_vm_event(COMPACTSUCCESS); @@ -4002,6 +4004,7 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, drained = true; goto retry; } + set_page_refcounted(page); out: psi_memstall_leave(&pflags); @@ -4322,8 +4325,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, * that first */ page = get_page_from_freelist(gfp_mask, order, alloc_flags, ac); - if (page) + if (page) { + set_page_refcounted(page); goto got_pg; + } /* * For costly allocations, try direct compaction first, as it's likely @@ -4403,8 +4408,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, /* Attempt with potentially adjusted zonelist and alloc_flags */ page = get_page_from_freelist(gfp_mask, order, alloc_flags, ac); - if (page) + if (page) { + set_page_refcounted(page); goto got_pg; + } /* Caller is not willing to reclaim, we can't balance anything */ if (!can_direct_reclaim) @@ -4788,8 +4795,10 @@ struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, /* First allocation attempt */ page = get_page_from_freelist(alloc_gfp, order, alloc_flags, &ac); - if (likely(page)) + if (likely(page)) { + set_page_refcounted(page); goto out; + } alloc_gfp = gfp; ac.spread_dirty_pages = false; From patchwork Mon Nov 25 21:01:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 13885268 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26E40D59D72 for ; Mon, 25 Nov 2024 21:02:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B36606B0095; Mon, 25 Nov 2024 16:01:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 90D476B0099; Mon, 25 Nov 2024 16:01:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 558046B0089; Mon, 25 Nov 2024 16:01:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 0CCF56B0088 for ; Mon, 25 Nov 2024 16:01:55 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 84AAA140DE0 for ; Mon, 25 Nov 2024 21:01:54 +0000 (UTC) X-FDA: 82825839222.24.EFEBF03 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id 6394B120016 for ; Mon, 25 Nov 2024 21:01:47 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Orix3nIu; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732568511; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=d0Iig2e7urKkwRSj05cFTUrQfZB5n5HhbU15XNoSt6A=; b=6CdAzwQY7ptn4ADJv8ng3E1SRqUm1S2YEAeVndWTJjEsO+0kZ9YkcvZ4z7jzyz3Ixcnzbb JGz4MAw3VljXkdy/mlmWzZh6P0Mn3heGkT40asgyC3IKOtJXgZwcd7T7y+wRZko54hYPFF nVA/e4D5z2wxWnV32o/aOgyn9CN+tvs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732568511; a=rsa-sha256; cv=none; b=47M06GBC1f06ITZa8NyLkCEqeFdekil93sr1/vArWKTntkW17IqTeGX89H6M30R27T+dKU /5FzL/Lj8cAf0VHUciaigaErBj0IWxHioQLBcPSkE8ybi6OVrl/Oi8xah/14M/G7iJz2/4 mrsj5w5FOj4DF9fqFSdveC/mq4eclms= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Orix3nIu; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=d0Iig2e7urKkwRSj05cFTUrQfZB5n5HhbU15XNoSt6A=; b=Orix3nIuwzh7iHRJ/G3ctdHXU2 PBguElb0sS+lB3RITAuEQx0GRtZLwdQRd29ty0yu99PEl8UgK41O6bsxNiYq2xpuVVabH1+nPj+tD 0s74XzkNgMQCZCDr1mEtuzs8Nru/pbHNK/u4Yg/nZZgcqWcul3u3QRmsAtUxntglTLSH4pd22ITkj hb0zS/AZVxD0g9NcFQf1heLs75aGfZ0UnGuWzWIbghWYY7o39kBofKnnquh1OyWq7Vht+1dCWbP4/ H15escYHFF9tViGxcOT4nINNhjZ5cZoHTyIlQZGYyMvNcpSS7fhocZkm2nfn2ZzPfFgyYgrZ/60jx Njbg9E4w==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1tFgDP-0000000CUE5-22dg; Mon, 25 Nov 2024 21:01:51 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, David Hildenbrand Subject: [PATCH v3 07/15] mm/page_alloc: Move set_page_refcounted() to callers of __alloc_pages_cpuset_fallback() Date: Mon, 25 Nov 2024 21:01:39 +0000 Message-ID: <20241125210149.2976098-8-willy@infradead.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241125210149.2976098-1-willy@infradead.org> References: <20241125210149.2976098-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam10 X-Stat-Signature: gt5kaycurmiormbecdcbjjmhbqrmt7fi X-Rspamd-Queue-Id: 6394B120016 X-Rspam-User: X-HE-Tag: 1732568507-329011 X-HE-Meta: U2FsdGVkX18mt9b4geHJLQsViVP7i6dVtuX/ZQW31v9if8hM1KlykE3o1ejwSMmcmkuDUrJp0mwk2Xor7IBc9wbZ9M9oLOKpoh+FCzwKDXYB0783eEV+5RdANacLfMecytiea52CWsajtnf4VDAhsXvucb+1ZaHtpeiDl3oEhGJqic3AtsJ845bfqxIzbwD6/mmv4W+P3EsGrIXCw1r/LFS0IERECiwdmyZqHTXbF9z35EjGf/obKGeSBSFIqHChvALbeB4k3XD1cy3ZPAd4nwyU5dCabo21Yuw7VUN2aD6jakStMJSK5+cFFSjUDBkQ/p1qxkyaD2QL8mo8mhcyTz/cLe1YMBz2B36C+remDdTsypCWr5zukmDzXL/DRv3RQ3PV/fuMXPnwnDXDowoRUOcqXHWfvfpzuGwqnigM+7GKISMZoUZJjMQPHTTgapPj4/lAZ+tHLTEDA8RDVaIOcO4t3v43cgbKkHmhO96qiTeHa1foKN7PTt2fOfR6R0tZ1n9eNtcPjSXANWba1ok/aR58f1mVBMpzSDqZJ1j11Wfqh9oi/1uYlTN5Yaoq7ej0vvSRB8z9kTw+DSxCUOAc7lOAVFKpHDYKEmDMxXOciiOjRd/lXq95M+1nlAABX/sdDiktfLVucNd5DBB/atYEgkgxeUGK0Pk8CXjkK4zaKPT0C27DE4IlyCltsu5RZhZ44xDpqUG8hpivccjIaiG8uEGvR6iqtQyE/dHkyYAGiDfGCFAYLILShpfb5WalQgAfS98xK7NdtoM6/u/tmxUgmD5E4LqEJw7zJns5WS/vw8117aYXzdPuEsV6JsXypDAXgqWIA4EwnLCg3R4EEFzNdqG0UJW3+0V6ftRUpC4TKld3rHFULdCLfL6PZDmczuCt15AbNvpGzvyXmqFNIBOSVLFT2G495cKpN61mJOiLcC273L6rmbB5AcTTBK6hFAhUX22koYSsHsji1S9Nuni 88KB8s7O J8x/EUJeSh1O9EZmra9KVkWo1SilPGlkV3QMER2aHD6IO777bE6JVyFGIGcwjFHeA/5gV0SuUfO/bE1YY9YB/wLFcUIqPBAk1j2/lxYVArYrwIFhTsPznQ649CGEE3Qp0Sy9TgOjjQKtCW44XYfuXZZbKlV1kS6o1MTWIGWKh2+baThcrFOOFC8+e34jl5XcMMCLV726g381rPqaHruxXX0pYl3um73NArUA612QIJSKjc8dxMpPE7igHOc1ZIM8PM8q6XTvfRALuVDYosrBH7ZZhroWrph7jZSCR X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In preparation for allocating frozen pages, stop initialising the page refcount in __alloc_pages_cpuset_fallback(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/page_alloc.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 14fa6bf7578a..c9e5c69f0cb9 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3600,9 +3600,6 @@ __alloc_pages_cpuset_fallback(gfp_t gfp_mask, unsigned int order, if (!page) page = get_page_from_freelist(gfp_mask, order, alloc_flags, ac); - - if (page) - set_page_refcounted(page); return page; } @@ -3689,6 +3686,8 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, if (gfp_mask & __GFP_NOFAIL) page = __alloc_pages_cpuset_fallback(gfp_mask, order, ALLOC_NO_WATERMARKS, ac); + if (page) + set_page_refcounted(page); } out: mutex_unlock(&oom_lock); @@ -4517,8 +4516,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, * the situation worse. */ page = __alloc_pages_cpuset_fallback(gfp_mask, order, ALLOC_MIN_RESERVE, ac); - if (page) + if (page) { + set_page_refcounted(page); goto got_pg; + } cond_resched(); goto retry; From patchwork Mon Nov 25 21:01:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 13885269 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 932FDD59D71 for ; Mon, 25 Nov 2024 21:02:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E2B316B0098; Mon, 25 Nov 2024 16:01:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C850E6B0093; Mon, 25 Nov 2024 16:01:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9C4F36B0092; Mon, 25 Nov 2024 16:01:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 511136B0093 for ; Mon, 25 Nov 2024 16:01:55 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 0F950120EAC for ; Mon, 25 Nov 2024 21:01:55 +0000 (UTC) X-FDA: 82825839180.19.42C441D Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id 4043720017 for ; Mon, 25 Nov 2024 21:01:47 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=RkA4VhMk; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732568509; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KITuz3gfgFofNiX2ze/7jfnfQhEBGdSEA1sxkx4u09Q=; b=Ri7aM7rUXAwO8b2otjV5smSIn5cxlGLWoQrX3MlxWmQ7vmj3ShNpNVqDvGkRpvWfXRMjO+ eDRpTPkS5JgU7elot61xsgnCHnIh7WgukDQ4yX9jLqgXBSjSIrzKsevQVCwXm8cmUzLbS/ FybACeSltypfvU4Gw/bGWbczuKXcAdY= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=RkA4VhMk; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732568509; a=rsa-sha256; cv=none; b=d3ha8KXqs5TAKPw2sapUcgLd7ffwhSLYbZX46Psb4B+5+rXEKT6mq4UNvokMZby7kaj0qr qMfEVprsK3RjA6tO0/skk5+09t/3JOggoiacSGinFLsyGrojKMv3GMokdLW+yh14Jc+48T OMimDraxeQMUvrT0x9rp2TvpLywLNzA= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=KITuz3gfgFofNiX2ze/7jfnfQhEBGdSEA1sxkx4u09Q=; b=RkA4VhMkpAa6kBjjky9CQYvmNi YfYrpEp9M7YHpEYoAJDyQ1V6oyQs58tdxeULiFAvKTUc+HYYPAiRWrWiGVe+ZsnFfkxWDegDN3O8Q h+WpE8TcOmW0DrfAjLTpn39+JupH+9D3IV/I2FRosY6ZOXbhSL0X39c64rnzSFiaz7e3esjXUU3Ye zbWiexfuWIjPETDpbLSKhP4VCzyBnC3SfeRm7QttCBkMvzF7PwRafKa8blKbodj9WDQPiWTykfP4P 7SMy1bjEl33P5P+Q+4N1B4whSar4Z4h6hMOUcuz3swtQRZAf/NZpgG6p8wLt3mSTStZ22dWnagXVR YraJgY5w==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1tFgDP-0000000CUE8-2PsB; Mon, 25 Nov 2024 21:01:51 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, David Hildenbrand Subject: [PATCH v3 08/15] mm/page_alloc: Move set_page_refcounted() to callers of __alloc_pages_may_oom() Date: Mon, 25 Nov 2024 21:01:40 +0000 Message-ID: <20241125210149.2976098-9-willy@infradead.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241125210149.2976098-1-willy@infradead.org> References: <20241125210149.2976098-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 4043720017 X-Stat-Signature: ektq9n8fe7k94y135pa4yes684occ3c5 X-Rspam-User: X-HE-Tag: 1732568507-866647 X-HE-Meta: U2FsdGVkX1+hybFa8cRuG/MwLif9PgwhZpKCXswu1rM/t7PZLqYZvq6YxULrdtUp2LTS7lvZJ4/b85fjX9teMyxJREwz+vVZri6r5kaYXHmFDEZlEycpGU2NirlhXbst9hzmanaj+9KcRpo3z7YhX46oWpwFR3NwK3cyVeocdVY9fzHKv/abN//CaiSapXVb30CREEDSxejvlunDTkPIcmKv54lP76uHesQb8E4iYgXoDnTxupJNefM8f45xae6cHvTKfD4D9OREPeG5oExyv3rILVCa5PBri7RD5T2kHpa0qPH3yx70U96U/P1xezqwccTic+bSL81pC4Ml92u/vjTIc8Sf30ogr5vZc53dQcyQSQbp23wlzTzFGRuVjhNzARZTQJwrrP8Os5es9wqEnYT8GQNbfvJ9vV2WkuDCskNcoLLMrhSyU0iA/Cd9Oy+o/tAbWuDheMh8JPB5mLKxPnAvv9M1jq6dTWTt+Q2i/6ZsfZbunqXQCsYzUuFXm9hvenqZYZ6UhcWyfe1nk783U6yL7VhU2x7iYM08ODWE0ygb0wOApuxYEaF+ZGalKDOB505ECHW/6CIpfXxHnWTolZCiIT6DNX9QcpPhRxEBz0Xsn1sAbSrbYMqWLLxwup2grrg+gZ9TBsnASxy56VxU7lresnYUTfv95DlDJuL2tchLVqGPsHEII4323Wl3nknBO4UzN1rW3OJOk8lUNr4RA7TgOtGK0FeDnF+FkaIyCosAxitfyYcw6nC5Z9gI8jlKuHz64lDDZ+6+C4gOupNG/8iFZ67uDnjLQWg6fIwJxx3UYMFSN838/5ervJBIDrh4XzjKGpzCpWqvQMjwuxQznnGGbl/8OmX+ip02XDiipSbpxU0oAKPLTytcSu08YFLOVkn1ScAp9vRZ5EZClWEBWbpMi8zS5majrfKMklxyV0zV3mFpvMxasAfrLjPUqtXxq2y+bw8NzVWGk88V+jl H4uV7Wyr RNUkCAssfU8w9NxTJj073xeiSF2Us6iDVANFQe9YmtR38A6SFDCJER3io45NAxrDMdSDsI258251oFcHSgoHHCsvDZWotgqQjk9wVr2kMRoyR2BbKAJ7qx6tC60RBGsq4Gy9qCgvnzcZC5wu5MxCFO6y7/3Lq9xW1Ra3tSAGiEJ2YpqIqk7iFdTeNkm9HtMDRxUyvJyAf5jQjgKla8PxLEXl+KxHKoNGEmmeKIe07luxta74zZPC/yfh/fctXgu7h0aRdQ6hpQnBkZOAtHjhhAUD+yVVnCbHcnBF1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In preparation for allocating frozen pages, stop initialising the page refcount in __alloc_pages_may_oom(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/page_alloc.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c9e5c69f0cb9..514994cd67b8 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3638,10 +3638,8 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, page = get_page_from_freelist((gfp_mask | __GFP_HARDWALL) & ~__GFP_DIRECT_RECLAIM, order, ALLOC_WMARK_HIGH|ALLOC_CPUSET, ac); - if (page) { - set_page_refcounted(page); + if (page) goto out; - } /* Coredumps can quickly deplete all memory reserves */ if (current->flags & PF_DUMPCORE) @@ -3686,8 +3684,6 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, if (gfp_mask & __GFP_NOFAIL) page = __alloc_pages_cpuset_fallback(gfp_mask, order, ALLOC_NO_WATERMARKS, ac); - if (page) - set_page_refcounted(page); } out: mutex_unlock(&oom_lock); @@ -4471,8 +4467,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, /* Reclaim has failed us, start killing things */ page = __alloc_pages_may_oom(gfp_mask, order, ac, &did_some_progress); - if (page) + if (page) { + set_page_refcounted(page); goto got_pg; + } /* Avoid allocations with no watermarks from looping endlessly */ if (tsk_is_oom_victim(current) && From patchwork Mon Nov 25 21:01:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 13885267 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D97CD59D71 for ; Mon, 25 Nov 2024 21:01:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7F6866B008C; Mon, 25 Nov 2024 16:01:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6DFDE6B0095; Mon, 25 Nov 2024 16:01:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3AAFE6B0092; Mon, 25 Nov 2024 16:01:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 0D8196B0089 for ; Mon, 25 Nov 2024 16:01:55 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id A9CBF120EBA for ; Mon, 25 Nov 2024 21:01:54 +0000 (UTC) X-FDA: 82825839222.21.034E2F2 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf30.hostedemail.com (Postfix) with ESMTP id 2EA678002D for ; Mon, 25 Nov 2024 21:01:45 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=wBeTkPO7; dmarc=none; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732568511; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ag4W615eOv8ytJfj90k4CCqQ2bMgFGGSZEjGvrUQlnk=; b=gqTd+K7pBd0chaAYPFexrpbHuCgDT+ovSP0zhgKlWliMLRGnJ/LeeDa2mwQilXa9h0DqmR wlEXDXu6F2JaXcEg7hs7qWxTN6xgdmmdnMEMOYHQZ1vo0SB62zWzGDNB0gWYHtbKNv3hBE j2X5UdSB1jn+fyjsNg8YdDDrktuY23c= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=wBeTkPO7; dmarc=none; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732568511; a=rsa-sha256; cv=none; b=XTcfgb88KHSkNkoLIu5VE8JBf7mNjljUHeoVXM9d9eMHsrlEAxOqvRNhmMXJ4Z3pYTNLvo c47iwQBGBg6ycn2nnvUyS0P8WvJSON0R8XgorexmqS9xFOHUlDWriPmbc37x/6dURjVOrH AinEk0q82AIZLfWKGcgeB4UeLtZmFWM= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Ag4W615eOv8ytJfj90k4CCqQ2bMgFGGSZEjGvrUQlnk=; b=wBeTkPO7iJA3a1kq+yGWtIU7O/ E45K4yWJaZgnCsRBA+C0gtnd6hgNlf5Pg2OkBlPRy4O8tkpO66Vi1a1bHjiTCxZEbBygdhZiMFt8/ +ylyowTqD8jJyTiVbCf/9eAoF4sq6vnXdGhoSxJtodRzWQgJ9en+az1e56+Xa1dwBgsbTfEsJOJ+K duklq2csT0AwSzG1CMiTOyg3whUq32HvYoMb2CLUxcXYzBy1TTOS+Z3jO4QzZOpN7A7O7Xm3wVpKO UJEP6Ep8dEM805SM/jpjYIKJdG5oT1/mpfNrXlT6pXeBj9lVQqA1CGLYEwvIkCVPszaAlK0Wz2qJc O1yoXTvw==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1tFgDP-0000000CUEJ-2ybr; Mon, 25 Nov 2024 21:01:51 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, David Hildenbrand Subject: [PATCH v3 09/15] mm/page_alloc: Move set_page_refcounted() to callers of __alloc_pages_direct_compact() Date: Mon, 25 Nov 2024 21:01:41 +0000 Message-ID: <20241125210149.2976098-10-willy@infradead.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241125210149.2976098-1-willy@infradead.org> References: <20241125210149.2976098-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 2EA678002D X-Stat-Signature: poi48y71rkpiar1trdhf5ssue1zekxmc X-HE-Tag: 1732568505-488056 X-HE-Meta: U2FsdGVkX1/OAkhkpmc7nxwissE09i1rsvPnhYxP6lTMelJqN5xZLn0rH4noDK7+nJ7xh+ZxBziAreCJYFUdangE1lHvYuKpwLufXVXp3II++On96malPd5P9nXH4MhrMU5lJWEJLJwQ2vwFX1Ds3hoLCFmKscR+FJVdrI/hpRKv300h6vOqxlQUKPA/j0RneZCXX99phzH/qNHKqkhwlNj3XjGjcNUat+dWrlkhCdgiLhBaneI9mQz2yM96VQSsDvBT5KpQC3NoVt2TwUkBWdh2+J9vkGkdgPY22czM4tv8N7J/k4yRsRwtsp45tOLk2ShChZZ3MwEyo3VJ5bVlj6uWeLxln7q7rRIW1q0yPsjIi7wptHv+b2YQPFTY/nuzW6Z/vEoEjcWm2hMxMB5E8/4bVA6Gll0WEJfj9m0ee7MTXPtkUqC/9qiy3Ker281LDUxI7oIWoTQ6dScQhRSAuGivaCIlxV7eXdPmm4iPlXITNX0izL7SgKpqPkFgP21JhpV/zKzj3ISXnvyaDkC/s4aXWqSoTcgObf1gS7JOEfiQg9rZ/lSbYgi+xe+On2lJIGhax+XWy9g1xwqtRiha5ky/rsOgU9JHU2TbrtTqM+j8jB1sncCQN5hytePDWdMuTLZ5MHY23InIVMz3uA3ZW8U/YXw2RMarJPMxz0RhYGhmX1M7DNwXW7gJRcJKxEs5KKqAzv01gkt4tzvNgefvpQPm/coaPU4WpE5czKI+CJwBn8u13rwwA1qDvoB0zTscr51UlIUm3L+tvpz1u4kEv6r9fkff/wa/d0qPhJNTqtHX4baEJj9BOKn+chaxufYQ/mVt8tcHWyDNiVC0KqDtUBB6hBRStsLVXpKyeX99oZ4W/vBiz8bwmNQdTGDAoeF0x9dfhMRyGROsTwph0VcDNFCjr/WO+I4czzYnQHY3RCyDcIDLZhJTeZco61Bu6Ll0QjpXPnb0ax/jd0AqtmR vT1+gsNt +TkORIKYoex9YEni42/Umi8GSnTyii/+gj9f6rvDTb6xLb3oqBCsLq1KYS759eflbwTKM4N4pUTBJK1zQHS70i++LnD+d86yOcuyD6MJJodgJdfhCaZBDtr8ZkLVrlxSfRkeyHDLxylcUgR3tuPbYVEltfh56jDwcvIS6MtUhIYc4buVk2RpkaKCkJuGt9JCWjr8sHit/aCNhGfqFwLuqM2IixvxrIp6DBA8an4YP/BHvWyMgNrqWIpbPIjk1fPDUPty+grEAwDVoQAtPepSv8Hd5g0QvNnxT2OLd X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In preparation for allocating frozen pages, stop initialising the page refcount in __alloc_pages_direct_compact(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/page_alloc.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 514994cd67b8..0f02cb253bf5 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3740,7 +3740,6 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, if (page) { struct zone *zone = page_zone(page); - set_page_refcounted(page); zone->compact_blockskip_flush = false; compaction_defer_reset(zone, order, true); count_vm_event(COMPACTSUCCESS); @@ -4342,8 +4341,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, alloc_flags, ac, INIT_COMPACT_PRIORITY, &compact_result); - if (page) + if (page) { + set_page_refcounted(page); goto got_pg; + } /* * Checks for costly allocations with __GFP_NORETRY, which @@ -4425,8 +4426,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, /* Try direct compaction and then allocating */ page = __alloc_pages_direct_compact(gfp_mask, order, alloc_flags, ac, compact_priority, &compact_result); - if (page) + if (page) { + set_page_refcounted(page); goto got_pg; + } /* Do not loop if specifically requested */ if (gfp_mask & __GFP_NORETRY) From patchwork Mon Nov 25 21:01:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 13885266 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BF04D59D6B for ; Mon, 25 Nov 2024 21:01:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 568166B0096; Mon, 25 Nov 2024 16:01:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4BBD16B008C; Mon, 25 Nov 2024 16:01:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E73C6B0093; Mon, 25 Nov 2024 16:01:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 0E5AE6B008C for ; Mon, 25 Nov 2024 16:01:55 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id AC0B5AE6E4 for ; Mon, 25 Nov 2024 21:01:54 +0000 (UTC) X-FDA: 82825839264.12.5478EF3 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf20.hostedemail.com (Postfix) with ESMTP id 86ECE1C0011 for ; Mon, 25 Nov 2024 21:01:48 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=l9If5H5X; dmarc=none; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732568511; a=rsa-sha256; cv=none; b=zSE0mCq+wGXV2VWWZyFsxQV8nhOCzAbrcUCIVk+/mPb6meYO3hShL7DrnuwSg2Bbpp+gik b5YjuYTSUe/19uU5sydaokB5CxbCC672NAU3vCZYgDl1unseE116og0W4QhKN0LAnyKKRB GvLbDqYEcEA88FWFsbcufWgfAlxnrkw= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=l9If5H5X; dmarc=none; spf=none (imf20.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732568511; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xBcOD7+Rq7OxCsg5IggYUVCArLtnSv3WpRLtyW/sVXk=; b=KzTKFucSYAtSR9WEIxmZixWMgf/TvrMC4ei9D2f/7r0GD6k+0s2Zl86fqXFDYBCrGaojgV AqTVhV+rJd+6R/JW7Q8bAKdKlhBLa6Z4xRtIplipB56bToURGdW6u6AvfkgfyBwpKYlqan +1WAVtFqL6nlkzv8ftq94fB7PDStYlg= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=xBcOD7+Rq7OxCsg5IggYUVCArLtnSv3WpRLtyW/sVXk=; b=l9If5H5X62NQixO/foXVA2gLjd onRaneJdtB329FUKp11n1SEsdqjM2P+p4XbSxK487/6Vfz8AOaR3JMVmbNhDejJBcu4RGT8sk45lP h2l9YUEMlSac5+WVptbpT+XFQEENgQchAWUvpQz5bqNcxnQnSSgBP7OrGaWKtJcw/nn55HtfDyWl8 I5wpiVdMCbGDNmsKB13HAqMHlbxYr2jmqrnFc5FOtw5S1Dpq4bIwB3ytCHxg5rgkUNFMOX3E8mZOa dlCgOn0DGTtQDC92TNYBGaXA3RU6Xq1wSxufJo2xWT97RTuKph1HTFcpcBnKQlKxjezQoqktdVF8r gUiuT5FA==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1tFgDP-0000000CUEQ-3bYw; Mon, 25 Nov 2024 21:01:51 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, David Hildenbrand Subject: [PATCH v3 10/15] mm/page_alloc: Move set_page_refcounted() to callers of __alloc_pages_direct_reclaim() Date: Mon, 25 Nov 2024 21:01:42 +0000 Message-ID: <20241125210149.2976098-11-willy@infradead.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241125210149.2976098-1-willy@infradead.org> References: <20241125210149.2976098-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 86ECE1C0011 X-Rspamd-Server: rspam01 X-Stat-Signature: h3xkzxgt84mkbrdinie946mumfrjmzxm X-HE-Tag: 1732568508-196267 X-HE-Meta: U2FsdGVkX18ckGN1RE5iXHJrs0RNHmS8DyKUeyjL4Egg8QWP34zLyjoJg1IK0Vi03kXyKy3D+8HEzW1cMvBhpjikFOHxfODhFBk6OdSH/KPYoxoiU9h+FEg0eCHujDOCeYPoTduBbPFE6Ed31VDmgT2g8vlH2QqTGhpltEAev+Fik2Omvgw8ieRkNt0nR78+JyOxd6NSYmDghs6XReIMHvZaoc4Pam94nvc2MZKwqvz6QgADd3tg3VyeN2RxA0vM978AWDXJjjBQpPsc9WyzHWvxIkPdgIRwTKpvkhVxx/ER1PUk8GpzwyghNB4KYICsgsPKWkn4lfPISp/qJrw+Mcr4Bp+VNk72+DEZQ5cMscXyRvHoKnZDUvudavw30Y7ioz72k06RhbYiE9wjUgjRn09d/etRIGdzYTWhVZlo7/sT4PPGYBWkcVCpn/XpUza9y5zJxEojoashytEv/U7nr+laiydaqeAv81j+BVkVpTMijZrNe22mX8gp6f5am1X/KJ79zLDZzAJ7Jg+Q7y+dRcG7V5Vd+Z+zurp/gCuh8G18DTtARMbK0vRlgXv3zqUlrDu0NnYlWk18XvU3AOVJV8mFjOTr1CYBSX7htMJuQQ84UG1qQDAAbQTBMIKdAp34DZx/wG4+8a1OQ/pzTbVaviFT6t+KqMibP8zbws8nbRQo1yuImvt17uouEMAG6aDSijCKKgInTCeZXhKbJTndRWUhtqHaF7AnaHrv4eJXcqstdmgZm98eS896lBU5Tt2qLOqI8NyxouR6rJAvlgR9F8iIsyAOOqsAWaXDED+3QQVfU6jl/bgF6WeFKZMdJ2QwOTPmU5cyFw7jvYwNhAL+QCbzC/kPjA/ya9BEpq/sNKZqyi8XXacgdqagZOYtqZzN6zCYQDbKdybmoNGsdJyV/5M9MwXItQDz2KQe14X4QgfifJaBCTj482psmCKffK8xlyioYalv4YakEQLFjuc ptp5Boam w5vw6hllH+7XTO+3muxlh+hI3/ZRwoySMO0aP1E+/tIKflMy+ZxlErHuj1my31TrelmUYD0LVJ3uElnK5L5p29BmqoASfR2Rit9M2+TbCHvvsNtH3fyk7cA85CnUQvI4gOjGO3F/j1jfayvxddqzcxuTToF7C2n25dRo+75axr+ZtHh+A1WcLBnnBvwXvVbUcjTsmEZdLHCzbIzYNnhfFao2GbzTNriojZhoGRDhqCvPk2vK+0dm+D6zy11Ei6RqTU2aEXCkkz9ZPMUq6S+49wwaxsrCE8FBfMdU2 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In preparation for allocating frozen pages, stop initialising the page refcount in __alloc_pages_direct_reclaim(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/page_alloc.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 0f02cb253bf5..7acc32902fc9 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3998,7 +3998,6 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, drained = true; goto retry; } - set_page_refcounted(page); out: psi_memstall_leave(&pflags); @@ -4420,8 +4419,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, /* Try direct reclaim and then allocating */ page = __alloc_pages_direct_reclaim(gfp_mask, order, alloc_flags, ac, &did_some_progress); - if (page) + if (page) { + set_page_refcounted(page); goto got_pg; + } /* Try direct compaction and then allocating */ page = __alloc_pages_direct_compact(gfp_mask, order, alloc_flags, ac, From patchwork Mon Nov 25 21:01:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 13885271 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45474D59D72 for ; Mon, 25 Nov 2024 21:02:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 49BB36B0089; Mon, 25 Nov 2024 16:01:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0AFAA6B009A; Mon, 25 Nov 2024 16:01:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD2146B0088; Mon, 25 Nov 2024 16:01:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5A38C6B0088 for ; Mon, 25 Nov 2024 16:01:55 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 1870CA0992 for ; Mon, 25 Nov 2024 21:01:55 +0000 (UTC) X-FDA: 82825839096.17.747CBEB Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf11.hostedemail.com (Postfix) with ESMTP id BDEBC40005 for ; Mon, 25 Nov 2024 21:01:48 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=X3clOrNc; dmarc=none; spf=none (imf11.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732568511; a=rsa-sha256; cv=none; b=6D8nJL5tI9hRCxwiuIB4VVoF/d+KSYSb2I5W5Z6yjFDdRwFjMkRsy7SY9623LBgVmnXZi1 mtd/OPLwxjLYUJnc7VXjUOilNIvN5beXxgTdqY7AM3ehntAPNOOqnEwA4diWR97dwh0rB/ J6+Glox+JrZrb1JVg+6DSGqLwnEgyPo= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=X3clOrNc; dmarc=none; spf=none (imf11.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732568511; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=15NrJPSuCbNLVlEgikd4xHCNsMtTjOmIIl7SMDwjgbg=; b=Vr6mqFIFQSoQesj8bIlyqC8h/gQIbGBuklSetWNKh2b5DtD12VoYI/TSadaSSLbK8/G/fX 30a9s6xRNICkywKvBjuuF0qO9Vmy3vVT6rqxNKL6aPy/DDEZs8o81aoUCdKaVDfAl7eytr zHjK5yD0CjoylENTjj44K4QebjKzPbg= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=15NrJPSuCbNLVlEgikd4xHCNsMtTjOmIIl7SMDwjgbg=; b=X3clOrNc40T+OSY1MuXIfkyCWH YrgFiu+ojonpVG81WV/NYL2oyVlwSpR1bSQLjyImrc6jvk4dgaWMKFtat7+vf66kH1dfz7opahyfg m626qm+Zr8CGhg11URX2ZEgBbWFrMuyqoUCv2e9tZBr4ejvhoSc6U4bT8pUf7cAKD+lh3wnPy6Goz JWShP3air3YgpYdP1IvV3vNjLKh/c7PdMFbYUIrI3P/QpB5hIO4bnvPbFplb+VrsJoHDJaPuoh0mN xU4i/pmyOtxVV/nhu6UOJiuQKw8Ay6yfmnoWPAtrDsI/2m2Y9sHW1F50ddNBaUxbMJQSWl46p1d/Z 3q3baMjA==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1tFgDQ-0000000CUEX-01RA; Mon, 25 Nov 2024 21:01:52 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, David Hildenbrand Subject: [PATCH v3 11/15] mm/page_alloc: Move set_page_refcounted() to callers of __alloc_pages_slowpath() Date: Mon, 25 Nov 2024 21:01:43 +0000 Message-ID: <20241125210149.2976098-12-willy@infradead.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241125210149.2976098-1-willy@infradead.org> References: <20241125210149.2976098-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: BDEBC40005 X-Rspamd-Server: rspam01 X-Stat-Signature: yhtgxo1zdesdmx19c5ueohj6japkj7dp X-HE-Tag: 1732568508-194621 X-HE-Meta: U2FsdGVkX1/XBUencAKF2Lkq1Ucms3w1giFt8u4zazlSCOOZ1ZAPc3weM4Mqjo5Q5HuYJ99ycHpukPWTSCvdUFp/X2/pWsJcx/4gTrCeyBTVAo/uZwQtrxZ7SqMWTAjYrZbjzm621VdLkdsepB/g0C2iEQyXo6bLPufOdXnZZVxwubg3Ha6uaviIf17ZZ5jE+vLhUaWk5oHO9H9P/DQOXiYGiJxOkRlPVFZwrf2lHGmZavq9LGBeFEKeLhw7YNOje5a3fTc+DSdwMOM1jWewx3it6/7IcCYf0cusviQ+dFL5w8KWDeIiopPULc3zziybF8lRLuNcN3heDjX+uX1kSCeb9NErBztme5+ueTuqlX1MGej4633BWxUaGW20e5wcHVgIarbkx900RuF/4JyafzjrmLBq/kmYNIURw+uGjd3pfT9BhZdeM1+yFKue/MDelT8jOC9ZHjRGBimQdviuMJtn3i3lb2Swjye4TzCrO0SqzCei9F86/u+rbo//z27XpHSVHxeiwlUhc5xxeC+8MlLAIjVxIwHVVgdcRS5gIU6bsCl7C0d7/zH/rxFIl+Lqa00+8QNwi1HsyhV6H35rFg4RgNNEGCD+RZvUeWHuwRab4kLZGc2AW0CVooxh9f+QnA2AuYeIMjyPO1P8HN4ykdL22Dh6j7Agz8pyneMOcrW+mMw8Snaftx3UagL6X/EKPvDUpK49GjXaz3YlpWPubbihpFn3GWKaDH/gG2pW0LyioauB5Xkr6udBM/aGqUBGUOPTBX/lNQzLzVA4wNyd5Akrdwt/p4/U3hZ2vHFUe2VhjBSDH80J4TOaasW+OAcArmOP7siiwzoVKEokuVaUvD9im0kgc0OnO5wiTWfvuMrgmpzhs+w3tW9NRXuoDnFG4AGX01vqP86Tm3YYDGNNMWMohSdXKUko8WdLJV3aoJqjfgEwqy6qq5fCS5zglRfyO6hR3bN+BPh4qE46+BT gjIjZMs7 lvH+ZDH2ayqQ/WL1V/3X9Y2BIM8uPQAx7o8N5nOgKzhMZCv0H8obWzU99Y69kCPfkelvwUnnyIZqn5DZ12kX7TjcBNthSNWy5wRRkHvzDyEKARCxCxbbzBN1YsejFhu8DboqmVv6twLV2r6uJJYuJULR75OVQnb6BBXNUEZjEbmVtCZvpX5QjBmRGCijAVlcSIn+IwyEjKaOcNz7sxm0XsnSJ1ua8AV35TD8eQgdgs/weYy1+FE0ycoGn0/edDcR0B5GjDalyiXYqGkFtAuMTN3YfECKbswDhLgcm X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In preparation for allocating frozen pages, stop initialising the page refcount in __alloc_pages_slowpath(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/page_alloc.c | 30 +++++++++--------------------- 1 file changed, 9 insertions(+), 21 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 7acc32902fc9..c219d2471408 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4318,10 +4318,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, * that first */ page = get_page_from_freelist(gfp_mask, order, alloc_flags, ac); - if (page) { - set_page_refcounted(page); + if (page) goto got_pg; - } /* * For costly allocations, try direct compaction first, as it's likely @@ -4340,10 +4338,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, alloc_flags, ac, INIT_COMPACT_PRIORITY, &compact_result); - if (page) { - set_page_refcounted(page); + if (page) goto got_pg; - } /* * Checks for costly allocations with __GFP_NORETRY, which @@ -4403,10 +4399,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, /* Attempt with potentially adjusted zonelist and alloc_flags */ page = get_page_from_freelist(gfp_mask, order, alloc_flags, ac); - if (page) { - set_page_refcounted(page); + if (page) goto got_pg; - } /* Caller is not willing to reclaim, we can't balance anything */ if (!can_direct_reclaim) @@ -4419,18 +4413,14 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, /* Try direct reclaim and then allocating */ page = __alloc_pages_direct_reclaim(gfp_mask, order, alloc_flags, ac, &did_some_progress); - if (page) { - set_page_refcounted(page); + if (page) goto got_pg; - } /* Try direct compaction and then allocating */ page = __alloc_pages_direct_compact(gfp_mask, order, alloc_flags, ac, compact_priority, &compact_result); - if (page) { - set_page_refcounted(page); + if (page) goto got_pg; - } /* Do not loop if specifically requested */ if (gfp_mask & __GFP_NORETRY) @@ -4471,10 +4461,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, /* Reclaim has failed us, start killing things */ page = __alloc_pages_may_oom(gfp_mask, order, ac, &did_some_progress); - if (page) { - set_page_refcounted(page); + if (page) goto got_pg; - } /* Avoid allocations with no watermarks from looping endlessly */ if (tsk_is_oom_victim(current) && @@ -4518,10 +4506,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, * the situation worse. */ page = __alloc_pages_cpuset_fallback(gfp_mask, order, ALLOC_MIN_RESERVE, ac); - if (page) { - set_page_refcounted(page); + if (page) goto got_pg; - } cond_resched(); goto retry; @@ -4813,6 +4799,8 @@ struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, ac.nodemask = nodemask; page = __alloc_pages_slowpath(alloc_gfp, order, &ac); + if (page) + set_page_refcounted(page); out: if (memcg_kmem_online() && (gfp & __GFP_ACCOUNT) && page && From patchwork Mon Nov 25 21:01:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 13885270 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC575D59D6B for ; Mon, 25 Nov 2024 21:02:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 251A36B0088; Mon, 25 Nov 2024 16:01:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DBCE96B0092; Mon, 25 Nov 2024 16:01:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B73376B009A; Mon, 25 Nov 2024 16:01:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 61B326B0098 for ; Mon, 25 Nov 2024 16:01:55 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0DDB5160C67 for ; Mon, 25 Nov 2024 21:01:55 +0000 (UTC) X-FDA: 82825839348.02.218DED8 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf24.hostedemail.com (Postfix) with ESMTP id 156BB180014 for ; Mon, 25 Nov 2024 21:01:52 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=rbH5G4x6; dmarc=none; spf=none (imf24.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732568510; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3lnRAUicoNNrnu4cFq/w74O09uyTQXqJYxHWAwaxrVk=; b=w+h9pWOo58cg9E6dvnZY0t6SppcBQ154Jku4nw5iOIi1uPPTl1RKOK3yw5nb2ozhihGXTF K38uvvM2fAAPoZqVba8ExEEwbnSurO5t4yGtQzPXZEcJ8WXTi5bru263xihT0ret3QQNgd +A4uffFGnrMH6y6heRXSOPcfoRrfzA0= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=rbH5G4x6; dmarc=none; spf=none (imf24.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732568510; a=rsa-sha256; cv=none; b=m33xT8U7VLlhWgssscBLCJvI4RIp0WoYUNSW79+D6y48nHsKEBxjadW8UJmmelbGh+G/G+ GtQ6fADrOWuCNNwhWWAWWMIZdf2tTd6wnnL1ETrS/QsWe3FUxJOK9iG6rU8SRZlD8SKIgg QjeMwDaPHRb32l640F9fMWlosNk2wug= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=3lnRAUicoNNrnu4cFq/w74O09uyTQXqJYxHWAwaxrVk=; b=rbH5G4x6R+Oglp60FeWMrwaTnh w6PE5NGTSA37c7v3NOifN/UnzBCxAVKgsWCfvBybgGkahewaU5ReSmh+6rrXO9w9UNGHQgcfIhgmX Gj55amMRqL0wHc25d+V6xMhuWPzfBvTtlXmaSt2omtA7EebEBFs7jSdQMFhe0LpTws+z5imMW1mmY Z7cB+vneGZ72ymYdt9zFHPD+Bjz6z+CPyIrsWiPZbpPiCBoa6EdjcN+vkHPS22YxrcytYyKu1fmQX bvNzxmPgAwFtAND88gtNy9CEUxASjbPwQ1AwAKccX6L34W8pDYkAbGyQUvqGTynVss7rmA8Pz47tM bvZHVvwQ==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1tFgDQ-0000000CUEe-0dVy; Mon, 25 Nov 2024 21:01:52 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, David Hildenbrand Subject: [PATCH v3 12/15] mm/page_alloc: Move set_page_refcounted() to end of __alloc_pages() Date: Mon, 25 Nov 2024 21:01:44 +0000 Message-ID: <20241125210149.2976098-13-willy@infradead.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241125210149.2976098-1-willy@infradead.org> References: <20241125210149.2976098-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 156BB180014 X-Stat-Signature: fw3ppt68pqejkp3gut3m6yiue84bwpnm X-Rspam-User: X-HE-Tag: 1732568512-870490 X-HE-Meta: U2FsdGVkX1+ykpbNfH/xInVQ33CvCBhiBeouskNn8KUx5gJbsRr6MN3oM1rKEY3SHLlEDjiONKpcsdLjeYtAdMKNkZK1yYeftVBuwjGKZi19+HOMqMrJHYl7jesEiUnmKi8mTEJobxC1/pbpfDQDx/oxM1XfWOvzB1zNL22M95KycJo/uAX0dq4P+KNypC2OnpcRwYiTO+zWP8urD2vrSfdSED9pD7zBJfauqpB0TJh6d4MV4GAWW4VphdNEBPPAkSN74B4lySUgrWlMFd9JV+DX32CvwzLoaSKytmW8JzSRFMG3nV3sCbRA+9zDJ61nEpn6JDzNS/o0tRMqF5uQMC+4r5gpmW6IQfwQ1BTvUwzOw0/rrNheOSwwJygjtztfHET7YsQJNzBauqODVp4xVEqEDpyeQ4v6UVJoib1D3spttFSP9qjGy4alZC5c949poIx2svFDYURcJ6viVXnUtEAUIkazXrgHDSU1kKNOHx06jK+gnvlHzkeHK/dM+y82VBVE5QyFcZtF1d/sckpNCIeZl0s38sGZgZ3r7QaBrMhKhio1b/yT88H8GgeS06TEmbUQVY+JgtmIQ0/r00mcOT8q28gVFxBy1mA45owIWSzej+tbaxpFKphWegyVjSuUZ/vdK5TqAktYmjbDoc60p4xMdKWVML5LTF7LEcgQ92SsTGRkb4bMrWAiLjVWsYEFJEcH/bo16jNWB/bQ1zWBda4HAX8q9514Nn33Nnvt+Ysa44Gjj930Im4seG8Bc9XNJ+sBS73dy+yKaZPRgXn0o279T9z70CZScf4QP2YJkeRcoA3Mc+7jrIqVLOAMF7alfpKryPz42zKaZbxfTVsmEnfmqBgh4dp5Tn7awzD61llsrks751xxdlXVtRA6GyJlGlggCnoOp+K6Z/X2HSq4lrDpXv7fF/NvCIpl0f4wFGaGVli2vZ4cMLmXgLYWvsFoi4rMZ37rM70hjx65AVp /iQN5Bto /n63zVj+ye6j4/iOLNq0GRzYEOxkIqibFsj7vNIUP0fKdaNk/Te3ETdBYGJlJcO+elApAxyIgDtKYpMR9rvRvpSCafYk0F8OjPM+D5GC2FV/2EAE34pSGjAJOKEfPtTjC/cWikw+ZrjT4lSQrn8C2IL/+aSvKvnuNelgtCp10GakFoxILmV/NY/utkN/txTwNb6QtMdUOGVrsnv8G2YK7OqoARY65i/2v7xRlBcAIouWr47a8x622E4hUvLPKJjMU+TDzd/Zq4iP5c8AyJATUiS0uwWchut0+s6cQ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Remove some code duplication by calling set_page_refcounted() at the end of __alloc_pages() instead of after each call that can allocate a page. That means that we free a frozen page if we've exceeded the allowed memcg memory. Signed-off-by: Matthew Wilcox (Oracle) --- mm/page_alloc.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c219d2471408..35fb45b8b369 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4784,10 +4784,8 @@ struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, /* First allocation attempt */ page = get_page_from_freelist(alloc_gfp, order, alloc_flags, &ac); - if (likely(page)) { - set_page_refcounted(page); + if (likely(page)) goto out; - } alloc_gfp = gfp; ac.spread_dirty_pages = false; @@ -4799,15 +4797,15 @@ struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, ac.nodemask = nodemask; page = __alloc_pages_slowpath(alloc_gfp, order, &ac); - if (page) - set_page_refcounted(page); out: if (memcg_kmem_online() && (gfp & __GFP_ACCOUNT) && page && unlikely(__memcg_kmem_charge_page(page, gfp, order) != 0)) { - __free_pages(page, order); + free_frozen_pages(page, order); page = NULL; } + if (page) + set_page_refcounted(page); trace_mm_page_alloc(page, order, alloc_gfp, ac.migratetype); kmsan_alloc_page(page, order, alloc_gfp); From patchwork Mon Nov 25 21:01:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 13885335 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BE02D59D7E for ; Mon, 25 Nov 2024 22:38:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B6EA86B0089; Mon, 25 Nov 2024 17:38:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B1F706B008C; Mon, 25 Nov 2024 17:38:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 999076B0092; Mon, 25 Nov 2024 17:38:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7D87B6B0089 for ; Mon, 25 Nov 2024 17:38:23 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 437C1C0CDA for ; Mon, 25 Nov 2024 22:38:23 +0000 (UTC) X-FDA: 82826082360.24.F31B303 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf30.hostedemail.com (Postfix) with ESMTP id A0D0780003 for ; Mon, 25 Nov 2024 22:38:14 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=gn4cFZUi; dmarc=none; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732574300; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=T12nlbl/4Y+GQ4Uch7tugY4tn/IhU/zXrrnqkhKCI/8=; b=AaI3a9BFbb6N12wDySq+avIniGvGXcrcPfDUjrgujWB/2oqzL8mQbZLbyEFBRVIW3RUGjh tPw9+vtR+BLoxK4DloUra3qsr7BuzVnSeHQlGxVFpQL2tMrKgciOTASMoVvC2x+KI2cIvY NFUJQ/LXTF++fNkijrRBBTMTR+s8d8A= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=gn4cFZUi; dmarc=none; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732574300; a=rsa-sha256; cv=none; b=S+H4VOCbu1ZA6WnQK97gzF7qRvtb7rxIlmQISuH6kVxRetKVuhNTU+URJCG77thsCqdslE 666CmrNc8+5jx0IIp5lUaecDZNFatcu24HFmAv9SPy3h+TRCV1AlKpiwYLWIZ3qWsmGUC+ 9QHuwfJclBEVlB7wYABq3z/YMzdy9bI= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=T12nlbl/4Y+GQ4Uch7tugY4tn/IhU/zXrrnqkhKCI/8=; b=gn4cFZUiX7oShjsb4J+QpLB0tO yuI9qxl2/UzRhkQLH+5oBQMrMQ0Oetu54JMIc/TYX2UJ90bKCcisNwAAkDhey3akmQm2k58sPBBZb cS5JUUA1BNfjLNmqN2fRzFcF/47vBy35SRWkM1N26VMlpl8SDBCCLLnv3Qm7d4LAuPR1nr+US21Xg mKo0SZubGHmfRPu4cZ/ZQIoB2IrB6tGlSYjnePIKmQd4LQOJyrw5FkQgNteEm4aTU9f+hI5fidfvs QI+mkVwKV1Ly+NjcMI7YWs6Ikktpq9Hjxxp916b+CMOeKLImPoAgxceI4hL5YUlteQRO/v2gWwnzN xehEeC/Q==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1tFgDQ-0000000CUEl-1Gf5; Mon, 25 Nov 2024 21:01:52 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, David Hildenbrand Subject: [PATCH v3 13/15] mm/page_alloc: Add __alloc_frozen_pages() Date: Mon, 25 Nov 2024 21:01:45 +0000 Message-ID: <20241125210149.2976098-14-willy@infradead.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241125210149.2976098-1-willy@infradead.org> References: <20241125210149.2976098-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: A0D0780003 X-Stat-Signature: etnp5hyffamyg5htoy9hz1uu5ifbdn1z X-HE-Tag: 1732574294-65971 X-HE-Meta: U2FsdGVkX18jZCrrqnvVu1gOAESSZ3+Lx9F0xPZ2zLGeSeNpeMj0WLcVqZY9qrUMNpx6QvyXleah1Vs+Y6Ad6cZapWrfd9UqTJKNFJpcPTekHqfMvk+yexcC0XvJIRvzyON/pzVcpe1cy/hexWG43C0UTyd7Q9y942KvS5GfeH/NulPVf1UbTXp8KBzqJEhCcsH5WZqE+0ELCmX+tzs7alThZg7Iq2TOdzyTlfU0nbyvcLaukHJ/B4rZAUch+gMfAyACU3WDZJ8utHCnhvMwJNFW7E/qvABwlW64FNBfQGDEgWHeyMadiE1ZnKepcitNty+rl1rTcPR5JeY8EWBDlBtht243fPk/CtkOHrB+N6BMWBvEQmG+UIiEGMI+sE6veAm0givcR8W7WTVYAU6/NiOn2BamEIA/2kzHQWDaBdUpwGfp9Vn7C27QG1HFmu/QuS1v2CV2Ec/eUNZ0Sa1Lc6N3gFW7fp+dYn/tDvP0/MxDi17QuNS7te6vctskkb/XzzngwGpMzY2ZMxLkmdKnuMxUna/jqjNojpZXgViMsT0T0zffFy9Vjbi3DTZW4xdnqrAAyv6xUgAd7fukox71pC+RJjLpEJqKhMsL6D3sDbjWTucsdh0ZlLf3fBQiHeNQGtjog+loPk/HfAzjqwbVSUaeeXx6U9XjciIBime/j4dfCgSbMNCUWJbbI/g10phSf3HPYZeKUVUF5UYP8p06OFcBQwxDQQoVpyh/3Vnu51Y6vvKgG1UagLl7FDAu998cueuPD4wY/aTOr5Io4XasvHkdTEqk/UEySavDwuG30rdhYznZz4wusPVFT8xGESGGwIBfPPd37kXH643ZbBO42NHoqvrQ9AiMvzaksH2NruztYtUTtjKT6RPTzdL2SLpinvlakmKIRplcjq17Ujv8AF8Ii9rLV/X1Cn3YFDHCvXr9CAijIfqP6Qhhxc37T97Dpz/sToxFVSqihEw3A14 uGj1HLEp 4NO0zAhoELt+ACLWUlCNhQq5KnSjsuXl5tMIPfXrYrH4mdzHr4NtmIygofGW/fPAU0yDBHy9VweDmJVDOLHghcFG9Ke6/CW2xjAJH6mm1F7+QTQsBDjf2Y24ijMvATY/qkLAy9A0PNMGDmYi6A00rhar7qVqcJDCX+zyecif7JGh0SpQCLkaqMW+vUagZ1ANQLaDV6Wd/H3bEvc187PsP0qfsEn7PuBTnch+/oZ6EYQBhBD+TN9o3H8jOn1aBlUJJA2WbTM15ShlJDwWBeW8wgfkKamT+eSbJQ2Rv X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Defer the initialisation of the page refcount to the new __alloc_pages() wrapper and turn the old __alloc_pages() into __alloc_frozen_pages(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/internal.h | 4 ++++ mm/page_alloc.c | 18 ++++++++++++++---- 2 files changed, 18 insertions(+), 4 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 9cc5fdc614cf..55e03f8f41d9 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -740,6 +740,10 @@ extern bool free_pages_prepare(struct page *page, unsigned int order); extern int user_min_free_kbytes; +struct page *__alloc_frozen_pages_noprof(gfp_t, unsigned int order, int nid, + nodemask_t *); +#define __alloc_frozen_pages(...) \ + alloc_hooks(__alloc_frozen_pages_noprof(__VA_ARGS__)) void free_frozen_pages(struct page *page, unsigned int order); void free_unref_folios(struct folio_batch *fbatch); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 35fb45b8b369..67b1cb757def 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4747,8 +4747,8 @@ EXPORT_SYMBOL_GPL(alloc_pages_bulk_noprof); /* * This is the 'heart' of the zoned buddy allocator. */ -struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, - int preferred_nid, nodemask_t *nodemask) +struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, + int preferred_nid, nodemask_t *nodemask) { struct page *page; unsigned int alloc_flags = ALLOC_WMARK_LOW; @@ -4804,14 +4804,24 @@ struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, free_frozen_pages(page, order); page = NULL; } - if (page) - set_page_refcounted(page); trace_mm_page_alloc(page, order, alloc_gfp, ac.migratetype); kmsan_alloc_page(page, order, alloc_gfp); return page; } +EXPORT_SYMBOL(__alloc_frozen_pages_noprof); + +struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, + int preferred_nid, nodemask_t *nodemask) +{ + struct page *page; + + page = __alloc_frozen_pages_noprof(gfp, order, preferred_nid, nodemask); + if (page) + set_page_refcounted(page); + return page; +} EXPORT_SYMBOL(__alloc_pages_noprof); struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_nid, From patchwork Mon Nov 25 21:01:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 13885273 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB5E4D59D71 for ; Mon, 25 Nov 2024 21:02:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C8766B0093; Mon, 25 Nov 2024 16:01:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F1DC56B0099; Mon, 25 Nov 2024 16:01:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CFACB6B009A; Mon, 25 Nov 2024 16:01:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id A0C696B0093 for ; Mon, 25 Nov 2024 16:01:56 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 372CA80D9F for ; Mon, 25 Nov 2024 21:01:56 +0000 (UTC) X-FDA: 82825839264.25.7C36A10 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf22.hostedemail.com (Postfix) with ESMTP id EBD86C0002 for ; Mon, 25 Nov 2024 21:01:49 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=AAeUEWcS; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732568512; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Oncka15roySS4aA2OelkqOYeFHMlrdQSsvNGmD6Ch/I=; b=YLqTzfyaKgnrTr78ezcXSs17rqjYX6ENtkDuFHO6gUxATbfReY+af5k4dwU2T4Y7W+E/og N4ZvaqBSxsv69GUEph1gSvd5wRbFeB+StyXycFcmsBQV7F+Tbv8svj6iuLuMzHj0IX18wm 2Kt756GRBFw7fzL4c4g7fHRN2eVpGww= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732568512; a=rsa-sha256; cv=none; b=b2PwfJ7R3RwepxgTWNJ82ManTUezZDRlIAEwRM5nv43CsA4i8J1PJlGXlFUqIwE9m2tq5m 6mAihVsywo8MyNtAEW0ta0fqvzcR1NibyIMQKojyXfW8dqt56m47jSNovBQibNJmGuQ+ka gS+UkFjFgXEQ1gfWjtJg4pw3m2OKw2c= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=AAeUEWcS; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Oncka15roySS4aA2OelkqOYeFHMlrdQSsvNGmD6Ch/I=; b=AAeUEWcSWr8RYcKnDB05LKwKXd DWpWK2oiCpvDu8BLq8zvVWQSck/s+ZxLUZJPnUyhLoovPzyYQAryZtzZwDc0m+1PdZebLq6HXiRKT 78m3B0W4G2wQwUzGWjTVEcGUTil8VI8KjBMm1+0bEXdc8Xb4llZI2cVek1PkdRdIy1U+/VJAp8KN9 vQR8Dd/y6RTWx7RHi5sjPGJZGs6fYBFlpSlwPi0W3SZk+dKgmXoUUhBDiGh5srKfc/ybk8CUZuCaF +2apAfCx8ZE42jJcTDGwHU13AVncJ/0udRSF0NX996namHCRAYbQXHXtTcYsHkRqGAL9RnVP3Dik4 bzAb28eg==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1tFgDQ-0000000CUEs-1td0; Mon, 25 Nov 2024 21:01:52 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, David Hildenbrand , William Kucharski Subject: [PATCH v3 14/15] mm/mempolicy: Add alloc_frozen_pages() Date: Mon, 25 Nov 2024 21:01:46 +0000 Message-ID: <20241125210149.2976098-15-willy@infradead.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241125210149.2976098-1-willy@infradead.org> References: <20241125210149.2976098-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: otcsdczr7jquotbxamy6w7tdhxu6mfrg X-Rspam-User: X-Rspamd-Queue-Id: EBD86C0002 X-Rspamd-Server: rspam02 X-HE-Tag: 1732568509-998256 X-HE-Meta: U2FsdGVkX1/V6h6Zgw8beDXNuMhn23U5UZijrgf3XpYrZiM5NXAjG8Z6jT9JaD2ZUHWgw0FZzZPiklV6ZkDfL7k094JXPfow5QCiep0IgyAt1mGNDwev8nZVzHnpwxwj/MgoYCSABrjETCn85AlY56g+y7rOdiOpquF7UrirZqBNxYZISQ8+85j9xZ9fO7Y4N4jKUxCxgAdVSY1nEr7V1/1RdbwW0IxcSlfzcjNjNfWJTE1uKW3GEUnZIyji01P9PSjIYZzlB83msPExk4b9ldp44O/lqqdHsBjjUGN3BSEFnBeo0Kb6zH9zmwfmbOLJSjBpxADcMrYBzzCWJhALkIwx5f/TmhgStTt0XUqU3Axz2j4Nf26sIVWwCdrM3hQaaXyKT3oozEfjlbhl6F/lKfLGhgzGerPfxq3l5j/uZK1tPqp15dGzKDWs+avMPB7j7mF8XauyFnti+IW2em9tF2eTlhQhImmoA+SJb9GxLk6suJH8CmOK1iztjZ6OcCeww2NFYdq0RwfCOIbHFgVddt+ss1PvLeJx5CmNgT+wCCRnbXE0id6FFBjiVN2GWB8OTpQ6/AinAuR4Pk5erqaaNjWaE90e5AwZc4ZIfAeDgD7/f5jhc4/eT6WGu3nLCjdZVbOrCiTQ4HjlAD9EEy3+4Eeu7md8wn+UVEedaStbKouXniZP07ZEe0RR4m7Xa9x8JDzgBq3NDU+ghy5vylJC9cvpAe8Eb7FMWoab5irBxF0BOVsvWEBer4NCLG8jZyHLMCkmY21onqWNvgduLJU+0t+SYNvAu4WF+CClVspXEAJk25uaJrmULP1zz9Adgu+StX1meymsZgH1njBHADzqvRWYzCnm6CHiBCP4HZ+pwP2xTeBpBKbvGb+owHbxlXXeuQIjHUnvBsdQn+B34H80SmdNCGO5fmqwvwyQieXC5lJP+MXmm2u8qsc22mjmAAdLiZPEO7fmm43EnsPPfY4 SfaHkWhd QO6+jJ/SPODJzVSgRldzfbAbC7vtf8z7fm4eea5EzvQVIaEMZ2WLiwY/p3r5pApXvgaTGXctx/vh1NY/mcB8ZA65TG5pMN+NdyNzTDiGKoG899xOD0r3dYMs+GyQUu/EDbf6tHjWsnFjmFRtMqIuGq9wf/YSRve0jEvvGjOpucSwG1wETDDnHzwV0crgmLsfy2uBHT8ZkxD3THdVn0XxqFlNkOnHU7x402Z0VesrdqPAyqodjX+61vhXaxhsczuCQeXJe9F3uy4TcqlRsHoD3fG+mjRCfozuoLf43a2Um8fpfGI3kEbGVa7eqKw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Provide an interface to allocate pages from the page allocator without incrementing their refcount. This saves an atomic operation on free, which may be beneficial to some users (eg slab). Reviewed-by: William Kucharski Signed-off-by: Matthew Wilcox (Oracle) --- mm/internal.h | 12 ++++++++++++ mm/mempolicy.c | 49 ++++++++++++++++++++++++++++++++----------------- 2 files changed, 44 insertions(+), 17 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 55e03f8f41d9..74713b44bedb 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -747,6 +747,18 @@ struct page *__alloc_frozen_pages_noprof(gfp_t, unsigned int order, int nid, void free_frozen_pages(struct page *page, unsigned int order); void free_unref_folios(struct folio_batch *fbatch); +#ifdef CONFIG_NUMA +struct page *alloc_frozen_pages_noprof(gfp_t, unsigned int order); +#else +static inline struct page *alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order) +{ + return __alloc_frozen_pages_noprof(gfp, order, numa_node_id(), NULL); +} +#endif + +#define alloc_frozen_pages(...) \ + alloc_hooks(alloc_frozen_pages_noprof(__VA_ARGS__)) + extern void zone_pcp_reset(struct zone *zone); extern void zone_pcp_disable(struct zone *zone); extern void zone_pcp_enable(struct zone *zone); diff --git a/mm/mempolicy.c b/mm/mempolicy.c index cda5f56085e6..3682184993dd 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2201,9 +2201,9 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, */ preferred_gfp = gfp | __GFP_NOWARN; preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - page = __alloc_pages_noprof(preferred_gfp, order, nid, nodemask); + page = __alloc_frozen_pages_noprof(preferred_gfp, order, nid, nodemask); if (!page) - page = __alloc_pages_noprof(gfp, order, nid, NULL); + page = __alloc_frozen_pages_noprof(gfp, order, nid, NULL); return page; } @@ -2249,8 +2249,9 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, * First, try to allocate THP only on local node, but * don't reclaim unnecessarily, just compact. */ - page = __alloc_pages_node_noprof(nid, - gfp | __GFP_THISNODE | __GFP_NORETRY, order); + page = __alloc_frozen_pages_noprof( + gfp | __GFP_THISNODE | __GFP_NORETRY, order, + nid, NULL); if (page || !(gfp & __GFP_DIRECT_RECLAIM)) return page; /* @@ -2262,7 +2263,7 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, } } - page = __alloc_pages_noprof(gfp, order, nid, nodemask); + page = __alloc_frozen_pages_noprof(gfp, order, nid, nodemask); if (unlikely(pol->mode == MPOL_INTERLEAVE) && page) { /* skip NUMA_INTERLEAVE_HIT update if numa stats is disabled */ @@ -2280,8 +2281,13 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int order, struct mempolicy *pol, pgoff_t ilx, int nid) { - return page_rmappable_folio(alloc_pages_mpol(gfp | __GFP_COMP, - order, pol, ilx, nid)); + struct page *page = alloc_pages_mpol(gfp | __GFP_COMP, order, pol, + ilx, nid); + if (!page) + return NULL; + + set_page_refcounted(page); + return page_rmappable_folio(page); } /** @@ -2316,6 +2322,21 @@ struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order, struct vm_area_struct } EXPORT_SYMBOL(vma_alloc_folio_noprof); +struct page *alloc_frozen_pages_noprof(gfp_t gfp, unsigned order) +{ + struct mempolicy *pol = &default_policy; + + /* + * No reference counting needed for current->mempolicy + * nor system default_policy + */ + if (!in_interrupt() && !(gfp & __GFP_THISNODE)) + pol = get_task_policy(current); + + return alloc_pages_mpol(gfp, order, pol, NO_INTERLEAVE_INDEX, + numa_node_id()); +} + /** * alloc_pages - Allocate pages. * @gfp: GFP flags. @@ -2332,17 +2353,11 @@ EXPORT_SYMBOL(vma_alloc_folio_noprof); */ struct page *alloc_pages_noprof(gfp_t gfp, unsigned int order) { - struct mempolicy *pol = &default_policy; - - /* - * No reference counting needed for current->mempolicy - * nor system default_policy - */ - if (!in_interrupt() && !(gfp & __GFP_THISNODE)) - pol = get_task_policy(current); + struct page *page = alloc_frozen_pages_noprof(gfp, order); - return alloc_pages_mpol(gfp, order, pol, NO_INTERLEAVE_INDEX, - numa_node_id()); + if (page) + set_page_refcounted(page); + return page; } EXPORT_SYMBOL(alloc_pages_noprof); From patchwork Mon Nov 25 21:01:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 13885289 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DA99D59D71 for ; Mon, 25 Nov 2024 21:18:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E8C826B0088; Mon, 25 Nov 2024 16:18:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E3CBA6B0089; Mon, 25 Nov 2024 16:18:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D2B466B008C; Mon, 25 Nov 2024 16:18:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B57916B0088 for ; Mon, 25 Nov 2024 16:18:21 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6A83EADBAA for ; Mon, 25 Nov 2024 21:18:21 +0000 (UTC) X-FDA: 82825880550.17.EE96E79 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf16.hostedemail.com (Postfix) with ESMTP id 9E30818000E for ; Mon, 25 Nov 2024 21:18:15 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=GquQtnfs; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732569496; a=rsa-sha256; cv=none; b=uqYLA0BUafgNYNFXVuVmL5G4JiYppG8+fMNyRFKGCjmJjVgrVjmWZN7vnfwNqb2Gn/gFld SdUsPuxXzblY+RYaRZOQ4O+LAbf+B/CUTqOyK3X3EOUXhV659g/PXe/yRpAQQsuda/e4xD we/gjLLbGtu3ymgZJstSn2UTHgbj0MI= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=GquQtnfs; dmarc=none; spf=none (imf16.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732569496; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=D7AjNZcJ7LYPBNAHKmq7G2riCYIe6jFfSqMnQ3ICvDw=; b=FlmLLUvI6qNXSdi99fluvccOgds5cDg+OumsRhVEmAekZwxmX7bgbriCT7Q9RGlGK9+Lww qpG+sE4IATYb/B7K2VZL4ExPcvhNbbipV4AkBdTVegjq9tsaMJ6cA632SugXT7DrZeT+7+ V9ltacBmpAdlpkuS3pip62UUnez1jgI= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=D7AjNZcJ7LYPBNAHKmq7G2riCYIe6jFfSqMnQ3ICvDw=; b=GquQtnfsOpt/2hGkvqfdBsClva aXlHaCBMK0+g1L33DmzhGNi8QjRtT5wh9O4m6016Qy1Usqgf+pdOikJ/vKJ25KXTLFBDWnsboc7mo PNOumX53WI10RU6yUIEZj5yeg2aYfKWes2P0wiFQbw5wdh9HH1Wt1p9YTPbtgiRmOXeo3s6UyL8j7 NuVO6Y9FdQkR85JG1djqr1X+RzTjLwxkMKAj/r+s01xpmQNh3fJ5C8pKVUWdKYeglLC4pbZUguaoG hkwEU81yCb+a9sZcnKKFPFBbnRGGaTZyoI28xSR1kiF0Re1RuxSmQzxPgw+y5rB/J2t5s7MCV8gH4 HTKcdROg==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1tFgDQ-0000000CUEz-2Pwq; Mon, 25 Nov 2024 21:01:52 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, David Hildenbrand , William Kucharski Subject: [PATCH v3 15/15] slab: Allocate frozen pages Date: Mon, 25 Nov 2024 21:01:47 +0000 Message-ID: <20241125210149.2976098-16-willy@infradead.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241125210149.2976098-1-willy@infradead.org> References: <20241125210149.2976098-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: 3khr9ubehk36dy74hugerc6to5agpy9w X-Rspamd-Queue-Id: 9E30818000E X-Rspamd-Server: rspam08 X-Rspam-User: X-HE-Tag: 1732569495-17895 X-HE-Meta: U2FsdGVkX1+GIuxhV0sbd5ND7i/UOhtKAQs4W5eg/gqt8r66pcN41bXMQSqXt7pqFNniqf2+IDilqLYCeH0p1xrwqyTIx78dBrg3LAbISGxtzCAzryYDy1VYb4NMN2dc3Z88tW5xfpoqMfpl930/j1Y0NSoAxDGRzcBrqT2V47she57VBjr0I9JjgLD0QGnBGekQxNoQRFpMQrDe6n9eqN37dka3ofkKLqYEEzoNbt0TRLrVK9BpRYrm1x3XsjYWN02bjoAOoFHKDjMN4LpzQyLAXRZRN8Bm9tF/c4CXgogy0m20TQy/mqGnhOySeU5+RKfekBdSFT7X5iUb8DwHbnor4uCxeqrMNysvl3PRz3afkvmElviJxHEmsMD6ExKaxUvbv3LL9+Tu+NolAoWembUZrOkoSNKbLGl5WwLoRIjeX1oIYD/GPX7TOUW6fidklykOaIQ5QOokB84etCbghNC0VRv5rBrVyrdNt2p+Z6XoZAm3IKeacbbC4f/XbQZKHPFCl2Mj4d8U3pBkddAt0Lp45l0U4dlwSTDKJP4SNl2b4duBMZMGJVYcLbYLQxTX0BZfX7M11s62tALcyLJryMN+CESsKwSrtKBcJbpcIEXX3/9B0v0iAzjs6ed6K2334ycaaTgTb3/eTjGPWNfWIMQade1vTlHMfnPGLesU3czxWp/I1HuhmWCisvqqtxOF0GGf2A3cmsVmjFIfQMT5+3whzsRi57y3Np3g9ViPTVC3gJO2Gm3bQlwh/XVszSlcPkuTDPewZQ5vOrODFfj0L4XZg4fQVsP/0f15GNjbuKdNDawHo7fdgXQjf2RoU3MHu4y9Ud5WR1GzosGR1Bx13pYzkmEr3fDM47Pw/YkT4jiVyIf2wtY/WKs/kraLs97z4vh+lqFrpzdUXafYKSWAuXjLZBAvnQAjE3ehpehs+w5Bs2qxwXdS9B0enRcncCs5Bqw68duFLOz/zO+eVu4 uLJPC53H /peBPQp+xWJRW0W2hY60YFW+sI4U8769o3eUdZlUEe2gOhvPJjWQprbHJs5js8tgyU0ilgVzMbN7b6amvLay4dTKjHv4uS/hiPNj3axUgqLcqpbZUffaR3OIWsLhLK/kCXe7GMf0oyOJjQILNChfEsizB2ctU7KSqGyFs16f50MujRlWEHX2OAkywyNro0m31O3MZIzvr3Z8vpGFpkANj9exXFqwoAMtkYQ+R8FLD9p4IP8vchBAI65hTnkDKXs7iwhke X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Since slab does not use the page refcount, it can allocate and free frozen pages, saving one atomic operation per free. Reviewed-by: William Kucharski Signed-off-by: Matthew Wilcox (Oracle) --- mm/slub.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 19980419b176..3d0c1cd6bdf5 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2405,9 +2405,9 @@ static inline struct slab *alloc_slab_page(gfp_t flags, int node, unsigned int order = oo_order(oo); if (node == NUMA_NO_NODE) - folio = (struct folio *)alloc_pages(flags, order); + folio = (struct folio *)alloc_frozen_pages(flags, order); else - folio = (struct folio *)__alloc_pages_node(node, flags, order); + folio = (struct folio *)__alloc_frozen_pages(flags, order, node, NULL); if (!folio) return NULL; @@ -2641,7 +2641,7 @@ static void __free_slab(struct kmem_cache *s, struct slab *slab) __folio_clear_slab(folio); mm_account_reclaimed_pages(pages); unaccount_slab(slab, order, s); - __free_pages(&folio->page, order); + free_frozen_pages(&folio->page, order); } static void rcu_free_slab(struct rcu_head *h)