From patchwork Wed Aug 14 03:54:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13762817 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EEC1FC531DC for ; Wed, 14 Aug 2024 03:55:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 82B1C6B0083; Tue, 13 Aug 2024 23:55:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 789686B0085; Tue, 13 Aug 2024 23:55:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5DC746B0089; Tue, 13 Aug 2024 23:55:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 383256B0083 for ; Tue, 13 Aug 2024 23:55:01 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 9EA34A7B57 for ; Wed, 14 Aug 2024 03:55:00 +0000 (UTC) X-FDA: 82449485160.20.DEF2E91 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf03.hostedemail.com (Postfix) with ESMTP id D97D220003 for ; Wed, 14 Aug 2024 03:54:58 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Ca3or8rQ; spf=pass (imf03.hostedemail.com: domain of 3kSq8ZgYKCBQIEJ1u808805y.w86527EH-664Fuw4.8B0@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3kSq8ZgYKCBQIEJ1u808805y.w86527EH-664Fuw4.8B0@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723607628; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BGqmRlGLO1UPrtqcDwqRzTwlBOD38mCYYr0sMrb4o6s=; b=1F0w/rGIzs2Zv879GMSqk3wfgPeoKLiAWk9v9mwHwFHZFGZU9izVf/YTSD/5vwuvpQzl2v JgeU4ZrQZkxCYXv+px1UzrAV7dZKxEnNOVEatQjywAiwONUpxZceBw2cav7H0UUMaWFwru vp2U02YrD4T6k6OqoX8e1vGD+GPmvP8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723607628; a=rsa-sha256; cv=none; b=Y0hC8y5YavMoFlnszHUFSBV/2jhRRCite7/mrAtAygCu46bSB0QVt+llEcAccu6o31Px4q T04SlfcsMSMS5fjUSu4rJKEIN5cBhLrWqWgOqMueLVnZv37zT+UXeNLvStMkoeqARAqaYO KFrMlf+DdMuSsDcosHgiJU9Y5sgvPvk= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Ca3or8rQ; spf=pass (imf03.hostedemail.com: domain of 3kSq8ZgYKCBQIEJ1u808805y.w86527EH-664Fuw4.8B0@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3kSq8ZgYKCBQIEJ1u808805y.w86527EH-664Fuw4.8B0@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-665a6dd38c8so133687477b3.1 for ; Tue, 13 Aug 2024 20:54:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723607698; x=1724212498; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BGqmRlGLO1UPrtqcDwqRzTwlBOD38mCYYr0sMrb4o6s=; b=Ca3or8rQpOoBy2ON6pj79kebZXN+cit8oCwt/YiYAGwjjfSmfJ6dQPDmkDmDVfD2kE OIP76Til9ooGOZsXspc7EfZP7q9XfzXOKJe4bYF+3y+2BP8YXsqRNAznkUG0VzppR3D0 OU7s/4GC4hyvwsu79P+NEANvXlVBmjBSXy8eh0DSEe23v2l79RBcoxB8+/WEb/pVsxk1 2l+rTmNmuus1XW7osspZVE+VBBQIdlh2i3msTXmmzLWii9EM/OdMjQvGiJDulW6Dk7qb 3CNWtDeLP0Rqxea7M8XMoKJGw/zqt2TCjk+/ujy/UozkV9b6gacHo67B7FDjQN5q6sNS 16Ew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723607698; x=1724212498; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BGqmRlGLO1UPrtqcDwqRzTwlBOD38mCYYr0sMrb4o6s=; b=XIofQOYTZgqUb7lw0Iqm0pjCjL1jqkicCJ//CVFTAQV1DT4Cay1aeP2skOXwRoijvq SHWP0qMnsRoeN+ETLorM1Iawfhw1JtdL/OJXgQQmY/POiaTBuF7r1uPThRwQdRrav3Yw oHIYm/IcsFcxWx648hxw2aHpol/KlMHvA3Wc2x0cTSnayX6DBsP05ITeKfTcA9s62sZu arhBbrqZYMjtxL1Uhrf5DmSIEppipnFEa8k9aQZ3DhO46ZSi2PSPzX9HUg0i3RHGkGIx vUcOA8e9YfK22BZUrH1+H9PHRnR2CtDvzsbiM5NwinpKyZublrOKYiP8RjQtGSVesgnU 6aOA== X-Forwarded-Encrypted: i=1; AJvYcCXgEHIPH3D8xgnFLJerWRll5Z4eYbSKNl51D/htZ0eUhR3PDTYOmFVPLKMMRd9VIucNCHL//0U2kA==@kvack.org X-Gm-Message-State: AOJu0Yym1jMekdq0fACwMnO3Q8+UmHZDPYNySp9+nyYakJBQYrL1jFCH 77d/wF1eQdx5UI7MnydpTJ9I1RGzpRkZ9Wh+rm+0JwJCFg0ZAbA0thlLNqh0fMWpK1OpF2wVXZk N0A== X-Google-Smtp-Source: AGHT+IHF7I7tjKFaxxVwyOf6J4cLNGcmd7dtJ8TUkzqBnzt3BcUh4JMTZUA9JOZpfjEC0zAqGoYWe8MmT6c= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:c8ad:fcc7:f7cf:f2fd]) (user=yuzhao job=sendgmr) by 2002:a05:690c:2f09:b0:689:315b:cbfc with SMTP id 00721157ae682-6ac9b7b1a2fmr44757b3.6.1723607697910; Tue, 13 Aug 2024 20:54:57 -0700 (PDT) Date: Tue, 13 Aug 2024 21:54:49 -0600 In-Reply-To: <20240814035451.773331-1-yuzhao@google.com> Mime-Version: 1.0 References: <20240814035451.773331-1-yuzhao@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240814035451.773331-2-yuzhao@google.com> Subject: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP From: Yu Zhao To: Andrew Morton , Muchun Song Cc: "Matthew Wilcox (Oracle)" , Zi Yan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao X-Rspamd-Queue-Id: D97D220003 X-Stat-Signature: sqn3r65akukrnj9mo16iipj7ns6smrc4 X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1723607698-857158 X-HE-Meta: U2FsdGVkX18phNrZdonXV68Nc3+X3AoIzkh90iqn9FM2alt7GoFs+mtnLulhfJGTVSuF2pKtaTYHjhjRz+z9YudefwI7n5gXu3l1OHyrnbfZZzYFa3H3A3eVKhHVV2/2TGhD8iMv2WjePbskHWzDwea/7olpr3Pfx7mPWBKtkBKIeNCUcpaub8oKhAX6Ri+9JKjzwGIQpw/q9nkp8i6Z3288AoDm0zsd3jylzl6jitSD6cu7V2xvMkiNy5lAgUQpUWvapNtXov01eqTgXWV7T3hufpuBwQ/UcFRLQbznG39TzpQFMTd0/oBQKSGCx0wm/cob+4jceZkvrZMu6+fhK1FxL2p+wqDw3e8bjZF+QtgIMNOcnW3w8rRJ/VsgcYz/xXoDFeO9iL6QVVSvzqGtLultZNtMylbsTHidgXH0gslr4XgWlnqX9Jqn6NZmrtsFqYXSRplu8uiN9qBZvHZumdTGgmigFMCUSurWnPmsqqXxoqdufSvAOQkN0YA7g9bUlqGfjiK2voHJpQvLET9LasN6x1vlHJ89THLaNCh6gbrgDIzpF6n2Gf19okhaDyrPmNBcFQrqCHYADDghEWP1s6A9VjN+O8IehXe1CXRlQJ+l3wYERu1qiay9lNLpvEV+Ns6yQk/aIJkvj90YlPYbYtEoyslRqicA5BMTOvsa83cbrVHguOZeSIFSnlkS22cf3gBHDi6FCHJ6T4SCKPaNWJ6O7rPvCoJonU2vPnzL02qqlI54yjRkSlT8qequ4GW7ATPxJpBEJnjFR/otchTgDz5g/2mgDgQMV61LR9ArueuGGRZffN3YdgtI4o2Rs60k1HNmFFCiZl0Ko6cNKvTaD4sjsGZuk0QO/w5DzOOh7stQaz3Nowkxwqz8/VUsP5JSeF559P8K0uqyG2R20d7uRbFXoW/1iSRGOpMSV0rr4nepE5/9LFvs6xGat7ZQnBJmUmgzmAJ+aSPKfWEqkG1 CI1LUQib 0d+lr/rxe/K/fuJVTNaoHkUMSLS5BHrqNHBFnx9r84QcMLWZI0B5tCqpqfx3ZR/+Sn8Oko32t/JHVnW5g+UYHQvmjHoWcq7ca1IutE9rJGZ9WB9A4/U1xbQTziSQPHm8JMEuyvebR662p469yNiF+CbmBEEmm+fh7kSl3uEEhiT/bp+e7GhaNcnxMnWf8EJSql53O8k5oFBFNEL1GVz3Xji9V2ubNJD0jX1zdyp9uANdFoZfDBEakF9U3jdGTp1cVEV5JqNUIj9XBwZ3pn5ygvkVGLr3INsd2WCQ695YabeOl38BQNKWsT8/iZrhXD1ooAmuIp3RkFpPCxWVddZR14gFhgTXKrgyCUSczHPB9fKzmOQBuilob3W4UKhT5me781bBu2I8iaVkjsFVdVga6ZLufWNiytHyf4h2IWrQtY243Bkm6X0YBZvgOQdp15BXEqKdYaLHNEvb7wKFYvil1tnU+5SFfu6D1WfxyGV1SK493lgucStUAqB6wdwC3Pu0oe0q1Ad528ESfpXOJCW/aYeB8lbQzk/Xkp/sXkFArq8PEOt2RY2fy9qJR/Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Support __GFP_COMP in alloc_contig_range(). When the flag is set, upon success the function returns a large folio prepared by prep_new_page(), rather than a range of order-0 pages prepared by split_free_pages() (which is renamed from split_map_pages()). alloc_contig_range() can be used to allocate folios larger than MAX_PAGE_ORDER, e.g., gigantic hugeTLB folios. So on the free path, free_one_page() needs to handle that by split_large_buddy(). Signed-off-by: Yu Zhao --- include/linux/gfp.h | 23 +++++++++ mm/compaction.c | 41 ++-------------- mm/page_alloc.c | 111 +++++++++++++++++++++++++++++++------------- 3 files changed, 108 insertions(+), 67 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index f53f76e0b17e..59266df56aeb 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -446,4 +446,27 @@ extern struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_ #endif void free_contig_range(unsigned long pfn, unsigned long nr_pages); +#ifdef CONFIG_CONTIG_ALLOC +static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp, + int nid, nodemask_t *node) +{ + struct page *page; + + if (WARN_ON(!order || !(gfp | __GFP_COMP))) + return NULL; + + page = alloc_contig_pages_noprof(1 << order, gfp, nid, node); + + return page ? page_folio(page) : NULL; +} +#else +static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp, + int nid, nodemask_t *node) +{ + return NULL; +} +#endif +/* This should be paired with folio_put() rather than free_contig_range(). */ +#define folio_alloc_gigantic(...) alloc_hooks(folio_alloc_gigantic_noprof(__VA_ARGS__)) + #endif /* __LINUX_GFP_H */ diff --git a/mm/compaction.c b/mm/compaction.c index eb95e9b435d0..d1041fbce679 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -86,33 +86,6 @@ static struct page *mark_allocated_noprof(struct page *page, unsigned int order, } #define mark_allocated(...) alloc_hooks(mark_allocated_noprof(__VA_ARGS__)) -static void split_map_pages(struct list_head *freepages) -{ - unsigned int i, order; - struct page *page, *next; - LIST_HEAD(tmp_list); - - for (order = 0; order < NR_PAGE_ORDERS; order++) { - list_for_each_entry_safe(page, next, &freepages[order], lru) { - unsigned int nr_pages; - - list_del(&page->lru); - - nr_pages = 1 << order; - - mark_allocated(page, order, __GFP_MOVABLE); - if (order) - split_page(page, order); - - for (i = 0; i < nr_pages; i++) { - list_add(&page->lru, &tmp_list); - page++; - } - } - list_splice_init(&tmp_list, &freepages[0]); - } -} - static unsigned long release_free_list(struct list_head *freepages) { int order; @@ -742,11 +715,11 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, * * Non-free pages, invalid PFNs, or zone boundaries within the * [start_pfn, end_pfn) range are considered errors, cause function to - * undo its actions and return zero. + * undo its actions and return zero. cc->freepages[] are empty. * * Otherwise, function returns one-past-the-last PFN of isolated page * (which may be greater then end_pfn if end fell in a middle of - * a free page). + * a free page). cc->freepages[] contain free pages isolated. */ unsigned long isolate_freepages_range(struct compact_control *cc, @@ -754,10 +727,9 @@ isolate_freepages_range(struct compact_control *cc, { unsigned long isolated, pfn, block_start_pfn, block_end_pfn; int order; - struct list_head tmp_freepages[NR_PAGE_ORDERS]; for (order = 0; order < NR_PAGE_ORDERS; order++) - INIT_LIST_HEAD(&tmp_freepages[order]); + INIT_LIST_HEAD(&cc->freepages[order]); pfn = start_pfn; block_start_pfn = pageblock_start_pfn(pfn); @@ -788,7 +760,7 @@ isolate_freepages_range(struct compact_control *cc, break; isolated = isolate_freepages_block(cc, &isolate_start_pfn, - block_end_pfn, tmp_freepages, 0, true); + block_end_pfn, cc->freepages, 0, true); /* * In strict mode, isolate_freepages_block() returns 0 if @@ -807,13 +779,10 @@ isolate_freepages_range(struct compact_control *cc, if (pfn < end_pfn) { /* Loop terminated early, cleanup. */ - release_free_list(tmp_freepages); + release_free_list(cc->freepages); return 0; } - /* __isolate_free_page() does not map the pages */ - split_map_pages(tmp_freepages); - /* We don't use freelists for anything. */ return pfn; } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 5841bbea482a..0a43e4ea29e4 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1197,16 +1197,36 @@ static void free_pcppages_bulk(struct zone *zone, int count, spin_unlock_irqrestore(&zone->lock, flags); } +/* Split a multi-block free page into its individual pageblocks. */ +static void split_large_buddy(struct zone *zone, struct page *page, + unsigned long pfn, int order, fpi_t fpi) +{ + unsigned long end = pfn + (1 << order); + + VM_WARN_ON_ONCE(!IS_ALIGNED(pfn, 1 << order)); + /* Caller removed page from freelist, buddy info cleared! */ + VM_WARN_ON_ONCE(PageBuddy(page)); + + if (order > pageblock_order) + order = pageblock_order; + + while (pfn != end) { + int mt = get_pfnblock_migratetype(page, pfn); + + __free_one_page(page, pfn, zone, order, mt, fpi); + pfn += 1 << order; + page = pfn_to_page(pfn); + } +} + static void free_one_page(struct zone *zone, struct page *page, unsigned long pfn, unsigned int order, fpi_t fpi_flags) { unsigned long flags; - int migratetype; spin_lock_irqsave(&zone->lock, flags); - migratetype = get_pfnblock_migratetype(page, pfn); - __free_one_page(page, pfn, zone, order, migratetype, fpi_flags); + split_large_buddy(zone, page, pfn, order, fpi_flags); spin_unlock_irqrestore(&zone->lock, flags); } @@ -1698,27 +1718,6 @@ static unsigned long find_large_buddy(unsigned long start_pfn) return start_pfn; } -/* Split a multi-block free page into its individual pageblocks */ -static void split_large_buddy(struct zone *zone, struct page *page, - unsigned long pfn, int order) -{ - unsigned long end_pfn = pfn + (1 << order); - - VM_WARN_ON_ONCE(order <= pageblock_order); - VM_WARN_ON_ONCE(pfn & (pageblock_nr_pages - 1)); - - /* Caller removed page from freelist, buddy info cleared! */ - VM_WARN_ON_ONCE(PageBuddy(page)); - - while (pfn != end_pfn) { - int mt = get_pfnblock_migratetype(page, pfn); - - __free_one_page(page, pfn, zone, pageblock_order, mt, FPI_NONE); - pfn += pageblock_nr_pages; - page = pfn_to_page(pfn); - } -} - /** * move_freepages_block_isolate - move free pages in block for page isolation * @zone: the zone @@ -1759,7 +1758,7 @@ bool move_freepages_block_isolate(struct zone *zone, struct page *page, del_page_from_free_list(buddy, zone, order, get_pfnblock_migratetype(buddy, pfn)); set_pageblock_migratetype(page, migratetype); - split_large_buddy(zone, buddy, pfn, order); + split_large_buddy(zone, buddy, pfn, order, FPI_NONE); return true; } @@ -1770,7 +1769,7 @@ bool move_freepages_block_isolate(struct zone *zone, struct page *page, del_page_from_free_list(page, zone, order, get_pfnblock_migratetype(page, pfn)); set_pageblock_migratetype(page, migratetype); - split_large_buddy(zone, page, pfn, order); + split_large_buddy(zone, page, pfn, order, FPI_NONE); return true; } move: @@ -6440,6 +6439,31 @@ int __alloc_contig_migrate_range(struct compact_control *cc, return (ret < 0) ? ret : 0; } +static void split_free_pages(struct list_head *list) +{ + int order; + + for (order = 0; order < NR_PAGE_ORDERS; order++) { + struct page *page, *next; + int nr_pages = 1 << order; + + list_for_each_entry_safe(page, next, &list[order], lru) { + int i; + + post_alloc_hook(page, order, __GFP_MOVABLE); + if (!order) + continue; + + split_page(page, order); + + /* Add all subpages to the order-0 head, in sequence. */ + list_del(&page->lru); + for (i = 0; i < nr_pages; i++) + list_add_tail(&page[i].lru, &list[0]); + } + } +} + /** * alloc_contig_range() -- tries to allocate given range of pages * @start: start PFN to allocate @@ -6552,12 +6576,25 @@ int alloc_contig_range_noprof(unsigned long start, unsigned long end, goto done; } - /* Free head and tail (if any) */ - if (start != outer_start) - free_contig_range(outer_start, start - outer_start); - if (end != outer_end) - free_contig_range(end, outer_end - end); + if (!(gfp_mask & __GFP_COMP)) { + split_free_pages(cc.freepages); + /* Free head and tail (if any) */ + if (start != outer_start) + free_contig_range(outer_start, start - outer_start); + if (end != outer_end) + free_contig_range(end, outer_end - end); + } else if (start == outer_start && end == outer_end && is_power_of_2(end - start)) { + struct page *head = pfn_to_page(start); + int order = ilog2(end - start); + + check_new_pages(head, order); + prep_new_page(head, order, gfp_mask, 0); + } else { + ret = -EINVAL; + WARN(true, "PFN range: requested [%lu, %lu), allocated [%lu, %lu)\n", + start, end, outer_start, outer_end); + } done: undo_isolate_page_range(start, end, migratetype); return ret; @@ -6666,6 +6703,18 @@ struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask, void free_contig_range(unsigned long pfn, unsigned long nr_pages) { unsigned long count = 0; + struct folio *folio = pfn_folio(pfn); + + if (folio_test_large(folio)) { + int expected = folio_nr_pages(folio); + + if (nr_pages == expected) + folio_put(folio); + else + WARN(true, "PFN %lu: nr_pages %lu != expected %d\n", + pfn, nr_pages, expected); + return; + } for (; nr_pages--; pfn++) { struct page *page = pfn_to_page(pfn); From patchwork Wed Aug 14 03:54:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13762818 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53E02C3DA4A for ; Wed, 14 Aug 2024 03:55:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DABE86B0085; Tue, 13 Aug 2024 23:55:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D33B06B0089; Tue, 13 Aug 2024 23:55:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B0E786B008A; Tue, 13 Aug 2024 23:55:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 8851B6B0085 for ; Tue, 13 Aug 2024 23:55:03 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 33C3CC0BD8 for ; Wed, 14 Aug 2024 03:55:03 +0000 (UTC) X-FDA: 82449485286.02.CCE35E1 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf07.hostedemail.com (Postfix) with ESMTP id 6C9874000E for ; Wed, 14 Aug 2024 03:55:01 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=qy4RYayM; spf=pass (imf07.hostedemail.com: domain of 3lCq8ZgYKCBcLHM4xB3BB381.zB985AHK-997Ixz7.BE3@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3lCq8ZgYKCBcLHM4xB3BB381.zB985AHK-997Ixz7.BE3@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723607631; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KXVmwVL4hZRiA8cTojFTWKVHpRNtdM/qQOsCSMq7lmE=; b=7WlROoIgw46ervClNAu03VzRGxmHhaNpIwL/KXkxP+zJNKy6bUcd841xRqmL9smj2Cwf83 IDeLUocLZWOY1AF9szeh44w4qRt6wdoAdIU8dgeuND51myO1mbbhCMUTz5gILiXXZ4cwpN hZJFSD1N8ilAN+5fBJma8DtHvrPgQN0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723607631; a=rsa-sha256; cv=none; b=tPIZIb5jWw38LKvM0Mqid/PPa6ZsnvDlsQyYKrAqnAmVkEPRqRe70VPheyzGyBvKDA7GC3 RzgEpTL5oJZ/exi4aM276UEa8IDM21LtMlw/vA196kO1twLwjD55iNC3iCmcC2mSaf7Kxz 4ZHWwMd6JAcg4ss5VtnbG5n0xSq+Af4= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=qy4RYayM; spf=pass (imf07.hostedemail.com: domain of 3lCq8ZgYKCBcLHM4xB3BB381.zB985AHK-997Ixz7.BE3@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3lCq8ZgYKCBcLHM4xB3BB381.zB985AHK-997Ixz7.BE3@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e05e3938a37so10222201276.2 for ; Tue, 13 Aug 2024 20:55:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723607700; x=1724212500; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=KXVmwVL4hZRiA8cTojFTWKVHpRNtdM/qQOsCSMq7lmE=; b=qy4RYayM0wWAER/bOasTAPGflQIxUDnAGkINKJRNQ3FokXBI8xOTyYT46fJnyPgt+O mjgiQh/vLjQs3FOCXeWg4yQ9ges7GawLY2wSe7GEq1poU3A8193nhAG4duTJI/FPIyad tAM3Of0ilWxCTjb/jJ1rqx0zxyUbYHbz68cPFQAISrmevQSh+Jjk+ot79VrzSIU3QyfR icmxIh9IsPNsqYRH9egzk5mmRfvmthDZPMVSGtqY1JyGbLYO81Pg7Iv+k32gI7HTe4f+ EFkv8jh0VRIne6IPZG+5UXXzTuU3ABW51hX9G7UMAKjPwSbTACMY5x2on3taprC/1yry 47dA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723607700; x=1724212500; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KXVmwVL4hZRiA8cTojFTWKVHpRNtdM/qQOsCSMq7lmE=; b=kzOtPoUelOgGzfExgHMmXBFyc8bXGl57ZyE2kIoTxt75rDoSRS9o5Ok2OEB5KA72gs 2JBSz/9TAMfpSSJjm5wXYAuZIodCErMdBkofOVHt0/fyH+aDdDAa7/xqMSShBf9am3zb z2VvHQpqiAvLYLkuxfODJvMZ2pQsIXan+gsyc3m+au/81O6syqYzm6PvOH9E6if0bBJH gjVyMMMihAtt2kdvRlQYqhM3uSkx2VUl4AHsnv+8XRrr8QDZnjLQHT9//0ujCWMDClCM tJhQfq9k0hNGcH3FtFruCiTGaAS4QkEurKemD10TUNYV3FoTxuWqZbcwSbjp6B2Bs/Gi l6og== X-Forwarded-Encrypted: i=1; AJvYcCVhdg4sYQkFRIsJEMkrH3WxcmW+yBq+oZx+5HPJzT+prJcsH27KX6+Dh2rN4NWz1bWwYTtNZAaSvsPO3f72Di91CoU= X-Gm-Message-State: AOJu0YySaxhZwBCwhAIDASF9g5cwweq8sjeYlopASJV6ojqUNaTUBNml cYtNhGS9bCEKTgNvrAGwiQvOmYgr8k0PZymvvjjEYeCu74xMqVzPDf3zPrwjwnoVEyZGF7+tyDo NkQ== X-Google-Smtp-Source: AGHT+IHdxjtU6GE8e6uUmdDhkDkgKFsun57md8iAQGJ8zy2FVnTlUkIoTwELvFGK+N9W879dEyGV8DeAgrM= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:c8ad:fcc7:f7cf:f2fd]) (user=yuzhao job=sendgmr) by 2002:a25:c788:0:b0:e03:62dc:63de with SMTP id 3f1490d57ef6-e1155ada202mr81659276.6.1723607700515; Tue, 13 Aug 2024 20:55:00 -0700 (PDT) Date: Tue, 13 Aug 2024 21:54:50 -0600 In-Reply-To: <20240814035451.773331-1-yuzhao@google.com> Mime-Version: 1.0 References: <20240814035451.773331-1-yuzhao@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240814035451.773331-3-yuzhao@google.com> Subject: [PATCH mm-unstable v2 2/3] mm/cma: add cma_{alloc,free}_folio() From: Yu Zhao To: Andrew Morton , Muchun Song Cc: "Matthew Wilcox (Oracle)" , Zi Yan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao X-Rspamd-Queue-Id: 6C9874000E X-Stat-Signature: a37faq3een887sq7s58p9uu1qu4kcf5e X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1723607701-217259 X-HE-Meta: U2FsdGVkX19MQItmHsmr2Rx1tXoF5u+r1pbt23aRP7GO3IaxtIdJRUotT8sb4kuw8oTzVGkTU8/pKgi/B7l8EQeuOkcrT9wGOOfWovkGkZT4sQ9+TQhB1UNY6a+OOK/CvtzvYLWz2ff6GKJuwtA5DvMckGQ2Jzp4s0uHXmKzT46DqAM2MnGaWx4/zb/jR98iFFTj4rM2NUsukomVdIN+UHxZYC9AYQph5Ep2eq9KEOg2Dm6wsL0aJnmCA7awp7Q+3lZhP2iFTs7zBQqbkrFIJIpaQCIENzS4KreJDYS6T7mivnPGtx8YUdsc654N88v5AmMEdwkQ8K6MxkiXyWk9IXscM82HoWA0aDwsKIKyT7U2YtsJnc2yZ43PFL/HEuu0GP26wb/aJxntUC0cd6X86PE/6UpbbgaoFuqyWglyHS/uE5PQpSScKGLoW3M0IbINfsKsEOlDhy5MV8a5L7HXJ0CSP3OIF2cUUmqJXWkgih8vh0kZUhyqXY0rQi1i6lfSFMc+2UAvBHCgH+M7J2BTFcKC8RxqrvtJwi56aImML7s7ePFnbRiLa4lRGjIGdM1/zRfneeQrWqVqs2Mczq5wt3LAGhLrVEMOAuqNyCKyioDw3m5wbNszszYg+FUfC3CSkMF2nN4X5MQVF/eV4HGud69qINFa8/5VfNXs3Dv3dr1cS67IHui9hCLZin74k8eTY3GOMd2qJrwk6m/sEuPHD1ZCSib97TeXHlrAQ4143CN8jpx7StjYPvNpvceaaiYleHW82tUvydKlrfA1SUecFXd7H2Ha73kiwSlXS9MVguNRcQh9l7MphzAUhK+gPth2beZ2btgnru6qkCP+aOlVhrHCYYx7rx3NHKc0q9N34937f47ROER7ZAMGSvxCX1EOHsSlKICFr3VtYqojDwoOKJBP6XqZdl7cQEediG1zDsSAmuFUJLNKjOHpUwOkFlXmccMSB3Mumv6pGOg+DPV DWCWHDaj al36Qui6TESjh5YVeJtkxJnh/ddEKqSrFvW3lB13DqtU6XZHoNicvwGhQBskSmtZD3MQZ9IcV+YLxLjRREqkXh8fiGy5oDjMDQsHz50HUHXQu+7WCKbLEQDzVFKFCbLwlv6I2shQFqe2wbnJ6R/DdOs3agloNl37RG2+Ksu5WQuZsFPRunbUQK9SN+Y467QGsPUaIpVzeAzTMCZgE3qoqRkWi3PUZdclPdCVbpc0whEAQfvIq8XkO2Apj8ImIpR+p9O3bSVFqjF5cC5sG0F6gjOX/YaW7lugFwSWdVQik7J00TIy7kdGNw8tZIq2xi/wVnzavISyF4/Y1BIVEJ3cYIfSsOWgBJjMDRnhqejxOmfJ96vFgBMyoW4EyuPjN1d4Iq0eKlkoAZNrXD6USYFh/j2yND9NaGlg/QAOV5nCItgJklU61xRwIJN8cSajqCXJ5RHVijBvClvj+3R5qykMEqy22Y0Yr8PBkhJcAD9Rtuhec4fTFY7/zQpgQk6Ui02CPAJTb7iQa7xDwDaKDuvPveHWTw/2lAUYgmKk4yr1OOvizWz5sefLmzifUZA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: With alloc_contig_range() and free_contig_range() supporting large folios, CMA can allocate and free large folios too, by cma_alloc_folio() and cma_free_folio(). Signed-off-by: Yu Zhao --- include/linux/cma.h | 16 +++++++++++++ mm/cma.c | 55 ++++++++++++++++++++++++++++++++------------- 2 files changed, 56 insertions(+), 15 deletions(-) diff --git a/include/linux/cma.h b/include/linux/cma.h index 9db877506ea8..d15b64f51336 100644 --- a/include/linux/cma.h +++ b/include/linux/cma.h @@ -52,4 +52,20 @@ extern bool cma_release(struct cma *cma, const struct page *pages, unsigned long extern int cma_for_each_area(int (*it)(struct cma *cma, void *data), void *data); extern void cma_reserve_pages_on_error(struct cma *cma); + +#ifdef CONFIG_CMA +struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp); +bool cma_free_folio(struct cma *cma, const struct folio *folio); +#else +static inline struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp) +{ + return NULL; +} + +static inline bool cma_free_folio(struct cma *cma, const struct folio *folio) +{ + return false; +} +#endif + #endif diff --git a/mm/cma.c b/mm/cma.c index 95d6950e177b..4354823d28cf 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -403,18 +403,8 @@ static void cma_debug_show_areas(struct cma *cma) spin_unlock_irq(&cma->lock); } -/** - * cma_alloc() - allocate pages from contiguous area - * @cma: Contiguous memory region for which the allocation is performed. - * @count: Requested number of pages. - * @align: Requested alignment of pages (in PAGE_SIZE order). - * @no_warn: Avoid printing message about failed allocation - * - * This function allocates part of contiguous memory on specific - * contiguous memory area. - */ -struct page *cma_alloc(struct cma *cma, unsigned long count, - unsigned int align, bool no_warn) +static struct page *__cma_alloc(struct cma *cma, unsigned long count, + unsigned int align, gfp_t gfp) { unsigned long mask, offset; unsigned long pfn = -1; @@ -463,8 +453,7 @@ struct page *cma_alloc(struct cma *cma, unsigned long count, pfn = cma->base_pfn + (bitmap_no << cma->order_per_bit); mutex_lock(&cma_mutex); - ret = alloc_contig_range(pfn, pfn + count, MIGRATE_CMA, - GFP_KERNEL | (no_warn ? __GFP_NOWARN : 0)); + ret = alloc_contig_range(pfn, pfn + count, MIGRATE_CMA, gfp); mutex_unlock(&cma_mutex); if (ret == 0) { page = pfn_to_page(pfn); @@ -494,7 +483,7 @@ struct page *cma_alloc(struct cma *cma, unsigned long count, page_kasan_tag_reset(nth_page(page, i)); } - if (ret && !no_warn) { + if (ret && !(gfp & __GFP_NOWARN)) { pr_err_ratelimited("%s: %s: alloc failed, req-size: %lu pages, ret: %d\n", __func__, cma->name, count, ret); cma_debug_show_areas(cma); @@ -513,6 +502,34 @@ struct page *cma_alloc(struct cma *cma, unsigned long count, return page; } +/** + * cma_alloc() - allocate pages from contiguous area + * @cma: Contiguous memory region for which the allocation is performed. + * @count: Requested number of pages. + * @align: Requested alignment of pages (in PAGE_SIZE order). + * @no_warn: Avoid printing message about failed allocation + * + * This function allocates part of contiguous memory on specific + * contiguous memory area. + */ +struct page *cma_alloc(struct cma *cma, unsigned long count, + unsigned int align, bool no_warn) +{ + return __cma_alloc(cma, count, align, GFP_KERNEL | (no_warn ? __GFP_NOWARN : 0)); +} + +struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp) +{ + struct page *page; + + if (WARN_ON(!order || !(gfp | __GFP_COMP))) + return NULL; + + page = __cma_alloc(cma, 1 << order, order, gfp); + + return page ? page_folio(page) : NULL; +} + bool cma_pages_valid(struct cma *cma, const struct page *pages, unsigned long count) { @@ -564,6 +581,14 @@ bool cma_release(struct cma *cma, const struct page *pages, return true; } +bool cma_free_folio(struct cma *cma, const struct folio *folio) +{ + if (WARN_ON(!folio_test_large(folio))) + return false; + + return cma_release(cma, &folio->page, folio_nr_pages(folio)); +} + int cma_for_each_area(int (*it)(struct cma *cma, void *data), void *data) { int i; From patchwork Wed Aug 14 03:54:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13762819 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46C7BC531DC for ; Wed, 14 Aug 2024 03:55:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CE36E6B008A; Tue, 13 Aug 2024 23:55:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C6BAD6B0092; Tue, 13 Aug 2024 23:55:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A48C56B0093; Tue, 13 Aug 2024 23:55:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 78AC76B008A for ; Tue, 13 Aug 2024 23:55:06 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id F040B1C4114 for ; Wed, 14 Aug 2024 03:55:05 +0000 (UTC) X-FDA: 82449485370.11.E0E8DE0 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf01.hostedemail.com (Postfix) with ESMTP id 1FD774001B for ; Wed, 14 Aug 2024 03:55:03 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="A/VYkVlb"; spf=pass (imf01.hostedemail.com: domain of 3lyq8ZgYKCBoOKP70E6EE6B4.2ECB8DKN-CCAL02A.EH6@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3lyq8ZgYKCBoOKP70E6EE6B4.2ECB8DKN-CCAL02A.EH6@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723607632; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HkJWh1CGI1FKnB7gI7WJokGF2tPUwAHyBE8neIj6WOo=; b=rFdEYxbVjCM00cL0EpmdE5fqp1noGp3VUO4UTxHpJHGKWhKSfIenrtJtuEfUcj15C9S6hR i+j9XSbxTOzbwNMdLv3o8AZla7NYOeq/bfZo441cncJjyp1aLtXuiViCKPKhhTaS6oOQPI bD6PKnmEKpqMWpdCqnrQ1ESaY54CmtM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723607632; a=rsa-sha256; cv=none; b=q0wETBxJn45Y1VhQEL/EPPwnLVR96dl10vF3M4+s6DZpxv4zBOd47K0rwzwuvwyme9SSOm TgWR5UXTS/VM2hODo5k55cgAKaGsJxRUSBtqblU3eB6SvxpICwDImczMtxXYGvepxq8NEt os+Zd80ITjSSB89vb7QzDV9xKMQOKsc= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="A/VYkVlb"; spf=pass (imf01.hostedemail.com: domain of 3lyq8ZgYKCBoOKP70E6EE6B4.2ECB8DKN-CCAL02A.EH6@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3lyq8ZgYKCBoOKP70E6EE6B4.2ECB8DKN-CCAL02A.EH6@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6ad660add0fso2378667b3.0 for ; Tue, 13 Aug 2024 20:55:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723607703; x=1724212503; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=HkJWh1CGI1FKnB7gI7WJokGF2tPUwAHyBE8neIj6WOo=; b=A/VYkVlbBNKbqNS+7CCM4USrHqgE0FByiXweGhbL89lCEfHdlS9lIhvXD780wWELZX 5Y9+oYsJi37vwWZ6Kf/ifgi1Dp/avlM6LsAFfhqiRyjYektSPKgEmn+CbxBWkP4NC9gk ygipeVffSbhBil+r/YqtAjLqIyevXafCZaCUODk3WbrUt1zNa0ArIvMZJ9fyUtXL0clf vVZO6NRG2lh8xF6V1T12D50q6j34fyw1Im0sD5ytOPZ8BNidlvnlMXk2rxBGF6iOoKr0 5kG5iVRLab9caieCeYPDA/0Ot4ssf50TKbrkqERBl6CxDcKMWdQkw0Uybr/kI/nUeBvE YnCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723607703; x=1724212503; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=HkJWh1CGI1FKnB7gI7WJokGF2tPUwAHyBE8neIj6WOo=; b=vjtVUUl5R96HmAXU6pT6SJKdEk4PiyvpQoTGMnX+fqC4TlTqFvvwSIKkYIF3y5xDfJ 8xvChqYyXAFcbTqbp4XtyoFSnSlaEB3dztr0CnlNuO8idF9haoFOSG3ms2V8oRz4QeEZ Eb+5TATxdIFGhxijYz5kfSm70X/lZB9+7Yu/LXMqYLzEYnyi07EeXaQcvCt1VqZEfowE Ihx0axpUyzcTlYLPw1rtxbTbMo5SnlpqsHSClz3x6I1WDrTRD7fTJ/u0AaFeQ1zY1y9e VSP8SwTrQQHvRfIwNO5delxGpgAQIfaJxPiFqR4qaVpk7/sKD4+H/ZwmAwBBCg2+qXky SBhg== X-Forwarded-Encrypted: i=1; AJvYcCVyGJdcIrjz6PwZklv7Zh5oKv3WKcLT1QWjR54YQ5hMHbLacWT6U0IpwCzYIGT6qoENSHy8UdJO29jBUP4r6zk2F9M= X-Gm-Message-State: AOJu0YxoTCaqOeLqt3WpypZrP1HvjIXHP19ZP8o3tqIWcX3nEbsCIFKe CMonsgYLIk1w1bua2aVtGilAC0E2yS+lKsvAqDWEtTihx5NrSo4AujdkfdWHznWy5OLGJ5lUTzc QuQ== X-Google-Smtp-Source: AGHT+IHlK/IkcXuFcqXLClLnBVvhuHrceOZTwz1/3YMEoi3KsLnVOrd8HqQm1xPKhmDUfIMOuGUHSIQI8zs= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:c8ad:fcc7:f7cf:f2fd]) (user=yuzhao job=sendgmr) by 2002:a0d:d147:0:b0:651:2eea:4dfe with SMTP id 00721157ae682-6a9df63c480mr685617b3.0.1723607703133; Tue, 13 Aug 2024 20:55:03 -0700 (PDT) Date: Tue, 13 Aug 2024 21:54:51 -0600 In-Reply-To: <20240814035451.773331-1-yuzhao@google.com> Mime-Version: 1.0 References: <20240814035451.773331-1-yuzhao@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240814035451.773331-4-yuzhao@google.com> Subject: [PATCH mm-unstable v2 3/3] mm/hugetlb: use __GFP_COMP for gigantic folios From: Yu Zhao To: Andrew Morton , Muchun Song Cc: "Matthew Wilcox (Oracle)" , Zi Yan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao , Frank van der Linden X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 1FD774001B X-Stat-Signature: y6m4xocafogmgwjkfzj6ip7cqzjhq6nz X-HE-Tag: 1723607703-582485 X-HE-Meta: U2FsdGVkX18d9xJUF8yvK1PNpV/6xKI7ELRDQhfAwhCKckB6LWmJw8tJ5c4QHfRqRs1w/+n+fGQvDVXnIPYRNQJts9XeYkNBK2vDicKUo5nXyyGPbb6NWt23Y8Vw114cQU+AsrLHsr5wX4zvrB9D6VAv3P2LySDPrtb9DccUb0yBzJeTJwHDnEucZh5kyEVZF42u5H/8/PjdrfASSB9yR0mZ1hRRhUPtbBV8u9VI7GHruAQSIcyu69FjJp85I8Owg/+GaSQzgHG40lHPw7B9KH5zOlvIDgYpVPJ8Y6iobOwAe2efd5+S8cYqJp27ZOlDN2F/XMcP3m8sGnegN42VvzJfDIcgotmsbV8ThN89svr5NvT9YB6a8SVDuxxnkF4O4m413TG99ylA2OpVIBekXB3jtXbLbm0p0WWh5dbfX8Tm1f58s7W50OOPEKpFPDbeF5c9CpaRixfmTwU27iAN8NtR8dnqtVYEzKQt/glAXS7X7jjbnpbmMlegPsutCRibDDolY6hfdT08WZ6RRfz8Y/gly50uGTVEUHvswKp+8VW9heec4e4WG5RvgldoIhqe7Tgp99ijBs58zV/AgKFctvK0wqwYH3vRYjkJjRGE1H/NeGkVPJidDubB1S1XNPcG3XB9EdLEp+5y9zcjM8VSZgTd5Jc2b5h6nelpaqp9snRIPekEvaBsEAvf62Pu8PlHB9RnVs6luRE4vEpk4ycKT/rsaFmf1Rrk/tY+yiCtCrPPxWbs7PRiT3y/Y2rtliXmpLi891G2bSTPaQOIrC4Q/T/J6L52T9m8WEub6E/KiEje4HqsS2gGxmD04jw2R+JGE4xb73pE5njND+zBpxeKuIpioepkfO7CiXQT1gtTiuwIUYukRqnaBPlyR2vfHZDQzu2RK7BDWDA4ZoV+aRisJ1n2F5Gsd8axqD33M9sVgi7Siv3uxS/nqAGSVvR3+KSAwFYbFn30RRNZHR26Vmf gI0PfR/2 qPPeoThVOmwwwK7OjGasve9YK2IfIXxm2t0Sk3ewHJEI8HcbwAz00P4IKu1rbQ4+LLjNRM/xWn0uvX50PZma7BwjLO4vDuy2ZuG7EujAsVLBCsLXBmIXTusHk1ZxlVWOqNn3imJ+GXA1ukqdxIJ52NlxctksdvtHFmzF0BEUaH1KJRsJi9404KvYjghROrthWUp3pPDzsUuJ/rQErTJtvKmeByQRMi4RX7/4iljJhdQNV3jAPfa/jhZYOaqhhywvoEh7sLgvT8a08gGyXzcc13khgnKe8D4jRziGjlRBFkbksbHeMET0/HLLISPngMBUgecDvrTGgb3RnBC0+GSkmEKnnKvCsHtgEKrxod1ziZQ0W/PO1NzLRLMwHHnLxiWBdSC4B9MxiDIRPtXI41jav4TJfZsYSP7Bnj7FZZWBK0MwM9JX9Wz45BV4CUDkyq36sokZr7lVgE0Q4imTfSjRna4GYlIXX/EMxXzrnBGUb86PwEzOmrbLcFBiJRbhJKsLmIAhl2dEP9vsMjkK7yYeTc2sSDxMica7P6KLNmM4y9fKzK8/G4OlZUC7BGg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use __GFP_COMP for gigantic folios to greatly reduce not only the amount of code but also the allocation and free time. LOC (approximately): +60, -240 Allocate and free 500 1GB hugeTLB memory without HVO by: time echo 500 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages time echo 0 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages Before After Alloc ~13s ~10s Free ~15s <1s The above magnitude generally holds for multiple x86 and arm64 CPU models. Signed-off-by: Yu Zhao Reported-by: Frank van der Linden Acked-by: Zi Yan --- include/linux/hugetlb.h | 9 +- mm/hugetlb.c | 293 ++++++++-------------------------------- 2 files changed, 62 insertions(+), 240 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 3100a52ceb73..98c47c394b89 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -896,10 +896,11 @@ static inline bool hugepage_movable_supported(struct hstate *h) /* Movability of hugepages depends on migration support. */ static inline gfp_t htlb_alloc_mask(struct hstate *h) { - if (hugepage_movable_supported(h)) - return GFP_HIGHUSER_MOVABLE; - else - return GFP_HIGHUSER; + gfp_t gfp = __GFP_COMP | __GFP_NOWARN; + + gfp |= hugepage_movable_supported(h) ? GFP_HIGHUSER_MOVABLE : GFP_HIGHUSER; + + return gfp; } static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 71d469c8e711..efa77ce87dcc 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -56,16 +56,6 @@ struct hstate hstates[HUGE_MAX_HSTATE]; #ifdef CONFIG_CMA static struct cma *hugetlb_cma[MAX_NUMNODES]; static unsigned long hugetlb_cma_size_in_node[MAX_NUMNODES] __initdata; -static bool hugetlb_cma_folio(struct folio *folio, unsigned int order) -{ - return cma_pages_valid(hugetlb_cma[folio_nid(folio)], &folio->page, - 1 << order); -} -#else -static bool hugetlb_cma_folio(struct folio *folio, unsigned int order) -{ - return false; -} #endif static unsigned long hugetlb_cma_size __initdata; @@ -100,6 +90,17 @@ static void hugetlb_unshare_pmds(struct vm_area_struct *vma, unsigned long start, unsigned long end); static struct resv_map *vma_resv_map(struct vm_area_struct *vma); +static void hugetlb_free_folio(struct folio *folio) +{ +#ifdef CONFIG_CMA + int nid = folio_nid(folio); + + if (cma_free_folio(hugetlb_cma[nid], folio)) + return; +#endif + folio_put(folio); +} + static inline bool subpool_is_free(struct hugepage_subpool *spool) { if (spool->count) @@ -1512,95 +1513,54 @@ static int hstate_next_node_to_free(struct hstate *h, nodemask_t *nodes_allowed) ((node = hstate_next_node_to_free(hs, mask)) || 1); \ nr_nodes--) -/* used to demote non-gigantic_huge pages as well */ -static void __destroy_compound_gigantic_folio(struct folio *folio, - unsigned int order, bool demote) -{ - int i; - int nr_pages = 1 << order; - struct page *p; - - atomic_set(&folio->_entire_mapcount, 0); - atomic_set(&folio->_large_mapcount, 0); - atomic_set(&folio->_pincount, 0); - - for (i = 1; i < nr_pages; i++) { - p = folio_page(folio, i); - p->flags &= ~PAGE_FLAGS_CHECK_AT_FREE; - p->mapping = NULL; - clear_compound_head(p); - if (!demote) - set_page_refcounted(p); - } - - __folio_clear_head(folio); -} - -static void destroy_compound_hugetlb_folio_for_demote(struct folio *folio, - unsigned int order) -{ - __destroy_compound_gigantic_folio(folio, order, true); -} - #ifdef CONFIG_ARCH_HAS_GIGANTIC_PAGE -static void destroy_compound_gigantic_folio(struct folio *folio, - unsigned int order) -{ - __destroy_compound_gigantic_folio(folio, order, false); -} - -static void free_gigantic_folio(struct folio *folio, unsigned int order) -{ - /* - * If the page isn't allocated using the cma allocator, - * cma_release() returns false. - */ -#ifdef CONFIG_CMA - int nid = folio_nid(folio); - - if (cma_release(hugetlb_cma[nid], &folio->page, 1 << order)) - return; -#endif - - free_contig_range(folio_pfn(folio), 1 << order); -} - #ifdef CONFIG_CONTIG_ALLOC static struct folio *alloc_gigantic_folio(struct hstate *h, gfp_t gfp_mask, int nid, nodemask_t *nodemask) { - struct page *page; - unsigned long nr_pages = pages_per_huge_page(h); + struct folio *folio; + int order = huge_page_order(h); + bool retried = false; + if (nid == NUMA_NO_NODE) nid = numa_mem_id(); - +retry: + folio = NULL; #ifdef CONFIG_CMA { int node; - if (hugetlb_cma[nid]) { - page = cma_alloc(hugetlb_cma[nid], nr_pages, - huge_page_order(h), true); - if (page) - return page_folio(page); - } + if (hugetlb_cma[nid]) + folio = cma_alloc_folio(hugetlb_cma[nid], order, gfp_mask); - if (!(gfp_mask & __GFP_THISNODE)) { + if (!folio && !(gfp_mask & __GFP_THISNODE)) { for_each_node_mask(node, *nodemask) { if (node == nid || !hugetlb_cma[node]) continue; - page = cma_alloc(hugetlb_cma[node], nr_pages, - huge_page_order(h), true); - if (page) - return page_folio(page); + folio = cma_alloc_folio(hugetlb_cma[node], order, gfp_mask); + if (folio) + break; } } } #endif + if (!folio) { + folio = folio_alloc_gigantic(order, gfp_mask, nid, nodemask); + if (!folio) + return NULL; + } - page = alloc_contig_pages(nr_pages, gfp_mask, nid, nodemask); - return page ? page_folio(page) : NULL; + if (folio_ref_freeze(folio, 1)) + return folio; + + pr_warn("HugeTLB: unexpected refcount on PFN %lu\n", folio_pfn(folio)); + hugetlb_free_folio(folio); + if (!retried) { + retried = true; + goto retry; + } + return NULL; } #else /* !CONFIG_CONTIG_ALLOC */ @@ -1617,10 +1577,6 @@ static struct folio *alloc_gigantic_folio(struct hstate *h, gfp_t gfp_mask, { return NULL; } -static inline void free_gigantic_folio(struct folio *folio, - unsigned int order) { } -static inline void destroy_compound_gigantic_folio(struct folio *folio, - unsigned int order) { } #endif /* @@ -1747,20 +1703,9 @@ static void __update_and_free_hugetlb_folio(struct hstate *h, folio_clear_hugetlb_hwpoison(folio); folio_ref_unfreeze(folio, 1); - - /* - * Non-gigantic pages demoted from CMA allocated gigantic pages - * need to be given back to CMA in free_gigantic_folio. - */ - if (hstate_is_gigantic(h) || - hugetlb_cma_folio(folio, huge_page_order(h))) { - destroy_compound_gigantic_folio(folio, huge_page_order(h)); - free_gigantic_folio(folio, huge_page_order(h)); - } else { - INIT_LIST_HEAD(&folio->_deferred_list); - folio_clear_partially_mapped(folio); - folio_put(folio); - } + INIT_LIST_HEAD(&folio->_deferred_list); + folio_clear_partially_mapped(folio); + hugetlb_free_folio(folio); } /* @@ -2033,95 +1978,6 @@ static void prep_new_hugetlb_folio(struct hstate *h, struct folio *folio, int ni spin_unlock_irq(&hugetlb_lock); } -static bool __prep_compound_gigantic_folio(struct folio *folio, - unsigned int order, bool demote) -{ - int i, j; - int nr_pages = 1 << order; - struct page *p; - - __folio_clear_reserved(folio); - for (i = 0; i < nr_pages; i++) { - p = folio_page(folio, i); - - /* - * For gigantic hugepages allocated through bootmem at - * boot, it's safer to be consistent with the not-gigantic - * hugepages and clear the PG_reserved bit from all tail pages - * too. Otherwise drivers using get_user_pages() to access tail - * pages may get the reference counting wrong if they see - * PG_reserved set on a tail page (despite the head page not - * having PG_reserved set). Enforcing this consistency between - * head and tail pages allows drivers to optimize away a check - * on the head page when they need know if put_page() is needed - * after get_user_pages(). - */ - if (i != 0) /* head page cleared above */ - __ClearPageReserved(p); - /* - * Subtle and very unlikely - * - * Gigantic 'page allocators' such as memblock or cma will - * return a set of pages with each page ref counted. We need - * to turn this set of pages into a compound page with tail - * page ref counts set to zero. Code such as speculative page - * cache adding could take a ref on a 'to be' tail page. - * We need to respect any increased ref count, and only set - * the ref count to zero if count is currently 1. If count - * is not 1, we return an error. An error return indicates - * the set of pages can not be converted to a gigantic page. - * The caller who allocated the pages should then discard the - * pages using the appropriate free interface. - * - * In the case of demote, the ref count will be zero. - */ - if (!demote) { - if (!page_ref_freeze(p, 1)) { - pr_warn("HugeTLB page can not be used due to unexpected inflated ref count\n"); - goto out_error; - } - } else { - VM_BUG_ON_PAGE(page_count(p), p); - } - if (i != 0) - set_compound_head(p, &folio->page); - } - __folio_set_head(folio); - /* we rely on prep_new_hugetlb_folio to set the hugetlb flag */ - folio_set_order(folio, order); - atomic_set(&folio->_entire_mapcount, -1); - atomic_set(&folio->_large_mapcount, -1); - atomic_set(&folio->_pincount, 0); - return true; - -out_error: - /* undo page modifications made above */ - for (j = 0; j < i; j++) { - p = folio_page(folio, j); - if (j != 0) - clear_compound_head(p); - set_page_refcounted(p); - } - /* need to clear PG_reserved on remaining tail pages */ - for (; j < nr_pages; j++) { - p = folio_page(folio, j); - __ClearPageReserved(p); - } - return false; -} - -static bool prep_compound_gigantic_folio(struct folio *folio, - unsigned int order) -{ - return __prep_compound_gigantic_folio(folio, order, false); -} - -static bool prep_compound_gigantic_folio_for_demote(struct folio *folio, - unsigned int order) -{ - return __prep_compound_gigantic_folio(folio, order, true); -} - /* * Find and lock address space (mapping) in write mode. * @@ -2160,7 +2016,6 @@ static struct folio *alloc_buddy_hugetlb_folio(struct hstate *h, */ if (node_alloc_noretry && node_isset(nid, *node_alloc_noretry)) alloc_try_hard = false; - gfp_mask |= __GFP_COMP|__GFP_NOWARN; if (alloc_try_hard) gfp_mask |= __GFP_RETRY_MAYFAIL; if (nid == NUMA_NO_NODE) @@ -2207,48 +2062,16 @@ static struct folio *alloc_buddy_hugetlb_folio(struct hstate *h, return folio; } -static struct folio *__alloc_fresh_hugetlb_folio(struct hstate *h, - gfp_t gfp_mask, int nid, nodemask_t *nmask, - nodemask_t *node_alloc_noretry) -{ - struct folio *folio; - bool retry = false; - -retry: - if (hstate_is_gigantic(h)) - folio = alloc_gigantic_folio(h, gfp_mask, nid, nmask); - else - folio = alloc_buddy_hugetlb_folio(h, gfp_mask, - nid, nmask, node_alloc_noretry); - if (!folio) - return NULL; - - if (hstate_is_gigantic(h)) { - if (!prep_compound_gigantic_folio(folio, huge_page_order(h))) { - /* - * Rare failure to convert pages to compound page. - * Free pages and try again - ONCE! - */ - free_gigantic_folio(folio, huge_page_order(h)); - if (!retry) { - retry = true; - goto retry; - } - return NULL; - } - } - - return folio; -} - static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h, gfp_t gfp_mask, int nid, nodemask_t *nmask, nodemask_t *node_alloc_noretry) { struct folio *folio; - folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, - node_alloc_noretry); + if (hstate_is_gigantic(h)) + folio = alloc_gigantic_folio(h, gfp_mask, nid, nmask); + else + folio = alloc_buddy_hugetlb_folio(h, gfp_mask, nid, nmask, node_alloc_noretry); if (folio) init_new_hugetlb_folio(h, folio); return folio; @@ -2266,7 +2089,10 @@ static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, { struct folio *folio; - folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, NULL); + if (hstate_is_gigantic(h)) + folio = alloc_gigantic_folio(h, gfp_mask, nid, nmask); + else + folio = alloc_buddy_hugetlb_folio(h, gfp_mask, nid, nmask, NULL); if (!folio) return NULL; @@ -2550,9 +2376,8 @@ struct folio *alloc_buddy_hugetlb_folio_with_mpol(struct hstate *h, nid = huge_node(vma, addr, gfp_mask, &mpol, &nodemask); if (mpol_is_preferred_many(mpol)) { - gfp_t gfp = gfp_mask | __GFP_NOWARN; + gfp_t gfp = gfp_mask & ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); folio = alloc_surplus_hugetlb_folio(h, gfp, nid, nodemask); /* Fallback to all nodes if page==NULL */ @@ -3334,6 +3159,7 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio, for (pfn = head_pfn + start_page_number; pfn < end_pfn; pfn++) { struct page *page = pfn_to_page(pfn); + __ClearPageReserved(folio_page(folio, pfn - head_pfn)); __init_single_page(page, pfn, zone, nid); prep_compound_tail((struct page *)folio, pfn - head_pfn); ret = page_ref_freeze(page, 1); @@ -3950,21 +3776,16 @@ static long demote_free_hugetlb_folios(struct hstate *src, struct hstate *dst, continue; list_del(&folio->lru); - /* - * Use destroy_compound_hugetlb_folio_for_demote for all huge page - * sizes as it will not ref count folios. - */ - destroy_compound_hugetlb_folio_for_demote(folio, huge_page_order(src)); + + split_page_owner(&folio->page, huge_page_order(src), huge_page_order(dst)); + pgalloc_tag_split(&folio->page, 1 << huge_page_order(src)); for (i = 0; i < pages_per_huge_page(src); i += pages_per_huge_page(dst)) { struct page *page = folio_page(folio, i); - if (hstate_is_gigantic(dst)) - prep_compound_gigantic_folio_for_demote(page_folio(page), - dst->order); - else - prep_compound_page(page, dst->order); - set_page_private(page, 0); + page->mapping = NULL; + clear_compound_head(page); + prep_compound_page(page, dst->order); init_new_hugetlb_folio(dst, page_folio(page)); list_add(&page->lru, &dst_list);