From patchwork Mon Nov 25 21:01:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Matthew Wilcox (Oracle)" X-Patchwork-Id: 13885273 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB5E4D59D71 for ; Mon, 25 Nov 2024 21:02:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C8766B0093; Mon, 25 Nov 2024 16:01:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F1DC56B0099; Mon, 25 Nov 2024 16:01:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CFACB6B009A; Mon, 25 Nov 2024 16:01:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id A0C696B0093 for ; Mon, 25 Nov 2024 16:01:56 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 372CA80D9F for ; Mon, 25 Nov 2024 21:01:56 +0000 (UTC) X-FDA: 82825839264.25.7C36A10 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf22.hostedemail.com (Postfix) with ESMTP id EBD86C0002 for ; Mon, 25 Nov 2024 21:01:49 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=AAeUEWcS; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732568512; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Oncka15roySS4aA2OelkqOYeFHMlrdQSsvNGmD6Ch/I=; b=YLqTzfyaKgnrTr78ezcXSs17rqjYX6ENtkDuFHO6gUxATbfReY+af5k4dwU2T4Y7W+E/og N4ZvaqBSxsv69GUEph1gSvd5wRbFeB+StyXycFcmsBQV7F+Tbv8svj6iuLuMzHj0IX18wm 2Kt756GRBFw7fzL4c4g7fHRN2eVpGww= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732568512; a=rsa-sha256; cv=none; b=b2PwfJ7R3RwepxgTWNJ82ManTUezZDRlIAEwRM5nv43CsA4i8J1PJlGXlFUqIwE9m2tq5m 6mAihVsywo8MyNtAEW0ta0fqvzcR1NibyIMQKojyXfW8dqt56m47jSNovBQibNJmGuQ+ka gS+UkFjFgXEQ1gfWjtJg4pw3m2OKw2c= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=AAeUEWcS; spf=none (imf22.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Oncka15roySS4aA2OelkqOYeFHMlrdQSsvNGmD6Ch/I=; b=AAeUEWcSWr8RYcKnDB05LKwKXd DWpWK2oiCpvDu8BLq8zvVWQSck/s+ZxLUZJPnUyhLoovPzyYQAryZtzZwDc0m+1PdZebLq6HXiRKT 78m3B0W4G2wQwUzGWjTVEcGUTil8VI8KjBMm1+0bEXdc8Xb4llZI2cVek1PkdRdIy1U+/VJAp8KN9 vQR8Dd/y6RTWx7RHi5sjPGJZGs6fYBFlpSlwPi0W3SZk+dKgmXoUUhBDiGh5srKfc/ybk8CUZuCaF +2apAfCx8ZE42jJcTDGwHU13AVncJ/0udRSF0NX996namHCRAYbQXHXtTcYsHkRqGAL9RnVP3Dik4 bzAb28eg==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1tFgDQ-0000000CUEs-1td0; Mon, 25 Nov 2024 21:01:52 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, David Hildenbrand , William Kucharski Subject: [PATCH v3 14/15] mm/mempolicy: Add alloc_frozen_pages() Date: Mon, 25 Nov 2024 21:01:46 +0000 Message-ID: <20241125210149.2976098-15-willy@infradead.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241125210149.2976098-1-willy@infradead.org> References: <20241125210149.2976098-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: otcsdczr7jquotbxamy6w7tdhxu6mfrg X-Rspam-User: X-Rspamd-Queue-Id: EBD86C0002 X-Rspamd-Server: rspam02 X-HE-Tag: 1732568509-998256 X-HE-Meta: U2FsdGVkX1/V6h6Zgw8beDXNuMhn23U5UZijrgf3XpYrZiM5NXAjG8Z6jT9JaD2ZUHWgw0FZzZPiklV6ZkDfL7k094JXPfow5QCiep0IgyAt1mGNDwev8nZVzHnpwxwj/MgoYCSABrjETCn85AlY56g+y7rOdiOpquF7UrirZqBNxYZISQ8+85j9xZ9fO7Y4N4jKUxCxgAdVSY1nEr7V1/1RdbwW0IxcSlfzcjNjNfWJTE1uKW3GEUnZIyji01P9PSjIYZzlB83msPExk4b9ldp44O/lqqdHsBjjUGN3BSEFnBeo0Kb6zH9zmwfmbOLJSjBpxADcMrYBzzCWJhALkIwx5f/TmhgStTt0XUqU3Axz2j4Nf26sIVWwCdrM3hQaaXyKT3oozEfjlbhl6F/lKfLGhgzGerPfxq3l5j/uZK1tPqp15dGzKDWs+avMPB7j7mF8XauyFnti+IW2em9tF2eTlhQhImmoA+SJb9GxLk6suJH8CmOK1iztjZ6OcCeww2NFYdq0RwfCOIbHFgVddt+ss1PvLeJx5CmNgT+wCCRnbXE0id6FFBjiVN2GWB8OTpQ6/AinAuR4Pk5erqaaNjWaE90e5AwZc4ZIfAeDgD7/f5jhc4/eT6WGu3nLCjdZVbOrCiTQ4HjlAD9EEy3+4Eeu7md8wn+UVEedaStbKouXniZP07ZEe0RR4m7Xa9x8JDzgBq3NDU+ghy5vylJC9cvpAe8Eb7FMWoab5irBxF0BOVsvWEBer4NCLG8jZyHLMCkmY21onqWNvgduLJU+0t+SYNvAu4WF+CClVspXEAJk25uaJrmULP1zz9Adgu+StX1meymsZgH1njBHADzqvRWYzCnm6CHiBCP4HZ+pwP2xTeBpBKbvGb+owHbxlXXeuQIjHUnvBsdQn+B34H80SmdNCGO5fmqwvwyQieXC5lJP+MXmm2u8qsc22mjmAAdLiZPEO7fmm43EnsPPfY4 SfaHkWhd QO6+jJ/SPODJzVSgRldzfbAbC7vtf8z7fm4eea5EzvQVIaEMZ2WLiwY/p3r5pApXvgaTGXctx/vh1NY/mcB8ZA65TG5pMN+NdyNzTDiGKoG899xOD0r3dYMs+GyQUu/EDbf6tHjWsnFjmFRtMqIuGq9wf/YSRve0jEvvGjOpucSwG1wETDDnHzwV0crgmLsfy2uBHT8ZkxD3THdVn0XxqFlNkOnHU7x402Z0VesrdqPAyqodjX+61vhXaxhsczuCQeXJe9F3uy4TcqlRsHoD3fG+mjRCfozuoLf43a2Um8fpfGI3kEbGVa7eqKw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Provide an interface to allocate pages from the page allocator without incrementing their refcount. This saves an atomic operation on free, which may be beneficial to some users (eg slab). Reviewed-by: William Kucharski Signed-off-by: Matthew Wilcox (Oracle) --- mm/internal.h | 12 ++++++++++++ mm/mempolicy.c | 49 ++++++++++++++++++++++++++++++++----------------- 2 files changed, 44 insertions(+), 17 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 55e03f8f41d9..74713b44bedb 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -747,6 +747,18 @@ struct page *__alloc_frozen_pages_noprof(gfp_t, unsigned int order, int nid, void free_frozen_pages(struct page *page, unsigned int order); void free_unref_folios(struct folio_batch *fbatch); +#ifdef CONFIG_NUMA +struct page *alloc_frozen_pages_noprof(gfp_t, unsigned int order); +#else +static inline struct page *alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order) +{ + return __alloc_frozen_pages_noprof(gfp, order, numa_node_id(), NULL); +} +#endif + +#define alloc_frozen_pages(...) \ + alloc_hooks(alloc_frozen_pages_noprof(__VA_ARGS__)) + extern void zone_pcp_reset(struct zone *zone); extern void zone_pcp_disable(struct zone *zone); extern void zone_pcp_enable(struct zone *zone); diff --git a/mm/mempolicy.c b/mm/mempolicy.c index cda5f56085e6..3682184993dd 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2201,9 +2201,9 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order, */ preferred_gfp = gfp | __GFP_NOWARN; preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - page = __alloc_pages_noprof(preferred_gfp, order, nid, nodemask); + page = __alloc_frozen_pages_noprof(preferred_gfp, order, nid, nodemask); if (!page) - page = __alloc_pages_noprof(gfp, order, nid, NULL); + page = __alloc_frozen_pages_noprof(gfp, order, nid, NULL); return page; } @@ -2249,8 +2249,9 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, * First, try to allocate THP only on local node, but * don't reclaim unnecessarily, just compact. */ - page = __alloc_pages_node_noprof(nid, - gfp | __GFP_THISNODE | __GFP_NORETRY, order); + page = __alloc_frozen_pages_noprof( + gfp | __GFP_THISNODE | __GFP_NORETRY, order, + nid, NULL); if (page || !(gfp & __GFP_DIRECT_RECLAIM)) return page; /* @@ -2262,7 +2263,7 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, } } - page = __alloc_pages_noprof(gfp, order, nid, nodemask); + page = __alloc_frozen_pages_noprof(gfp, order, nid, nodemask); if (unlikely(pol->mode == MPOL_INTERLEAVE) && page) { /* skip NUMA_INTERLEAVE_HIT update if numa stats is disabled */ @@ -2280,8 +2281,13 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, struct folio *folio_alloc_mpol_noprof(gfp_t gfp, unsigned int order, struct mempolicy *pol, pgoff_t ilx, int nid) { - return page_rmappable_folio(alloc_pages_mpol(gfp | __GFP_COMP, - order, pol, ilx, nid)); + struct page *page = alloc_pages_mpol(gfp | __GFP_COMP, order, pol, + ilx, nid); + if (!page) + return NULL; + + set_page_refcounted(page); + return page_rmappable_folio(page); } /** @@ -2316,6 +2322,21 @@ struct folio *vma_alloc_folio_noprof(gfp_t gfp, int order, struct vm_area_struct } EXPORT_SYMBOL(vma_alloc_folio_noprof); +struct page *alloc_frozen_pages_noprof(gfp_t gfp, unsigned order) +{ + struct mempolicy *pol = &default_policy; + + /* + * No reference counting needed for current->mempolicy + * nor system default_policy + */ + if (!in_interrupt() && !(gfp & __GFP_THISNODE)) + pol = get_task_policy(current); + + return alloc_pages_mpol(gfp, order, pol, NO_INTERLEAVE_INDEX, + numa_node_id()); +} + /** * alloc_pages - Allocate pages. * @gfp: GFP flags. @@ -2332,17 +2353,11 @@ EXPORT_SYMBOL(vma_alloc_folio_noprof); */ struct page *alloc_pages_noprof(gfp_t gfp, unsigned int order) { - struct mempolicy *pol = &default_policy; - - /* - * No reference counting needed for current->mempolicy - * nor system default_policy - */ - if (!in_interrupt() && !(gfp & __GFP_THISNODE)) - pol = get_task_policy(current); + struct page *page = alloc_frozen_pages_noprof(gfp, order); - return alloc_pages_mpol(gfp, order, pol, NO_INTERLEAVE_INDEX, - numa_node_id()); + if (page) + set_page_refcounted(page); + return page; } EXPORT_SYMBOL(alloc_pages_noprof);