From patchwork Sun Aug 11 21:21:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13759888 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A8E8C531DA for ; Sun, 11 Aug 2024 21:21:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C348D6B008C; Sun, 11 Aug 2024 17:21:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B95826B0092; Sun, 11 Aug 2024 17:21:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A358B6B0098; Sun, 11 Aug 2024 17:21:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 88FF56B008C for ; Sun, 11 Aug 2024 17:21:38 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 3D9C01603C9 for ; Sun, 11 Aug 2024 21:21:38 +0000 (UTC) X-FDA: 82441236276.17.0D64E58 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf24.hostedemail.com (Postfix) with ESMTP id 74451180012 for ; Sun, 11 Aug 2024 21:21:36 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Abi73pZA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3Xyu5ZgYKCNYQMR92G8GG8D6.4GEDAFMP-EECN24C.GJ8@flex--yuzhao.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Xyu5ZgYKCNYQMR92G8GG8D6.4GEDAFMP-EECN24C.GJ8@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723411240; a=rsa-sha256; cv=none; b=v6pkn8N17Q25QIwI/JLcSWItRo1NnDSJInIxH/nFWvalSfhTQAXbEMnFg5u6XlO4UhtCjL VGygsuCu1Sg4lNxr9s0kazmhiZZp8gYzOdt0G/OLPYCNn0yble7Krurtd7rTYvu0Em2+o4 Sth6Szk6vKXvN0m62gSxqwMHbEyEWiE= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Abi73pZA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3Xyu5ZgYKCNYQMR92G8GG8D6.4GEDAFMP-EECN24C.GJ8@flex--yuzhao.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Xyu5ZgYKCNYQMR92G8GG8D6.4GEDAFMP-EECN24C.GJ8@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723411240; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=THGaYoJGBVo1NFT+ygjobh7TePYwQA+eDE4p3qbAKWk=; b=l0h+/uMr9s0aoAZkktC5k+QduslvtwuR9bpK9vWre3fgKcV83VA++qOJBS2E4BQDbEl01c XrbF/Gt1PTXOCajbaL8l8JwU7H6CwsWlqvd+esyJoD0BE4JvqI4Ttnv2AOAtAJdPh1+RGt iu3S6dz1nw944iQ3ATlC538orftYA10= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-66a2aee82a0so73081557b3.0 for ; Sun, 11 Aug 2024 14:21:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723411295; x=1724016095; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=THGaYoJGBVo1NFT+ygjobh7TePYwQA+eDE4p3qbAKWk=; b=Abi73pZAKkBtCTNqDhGYyvw0W7BLOaPcsz2DHOFGFqtfqOei2SHM6MWOj8hly3EmjY hhMMBUyjCcQr88j2oLDJanthEPI+zctX0LKtH2waE/hxIqUAMYDw/aifSNAhUGLHMc53 YtpULCLr6gMxqTIZNPDnjLcpWIU+Eizb45/Bg9KAQcfTEcSVAiIGEUwjPUfDxSVW5LHF +0g3+13FZz4IeyF7eOJyw0zl3hHYtLZdH69eQ3IMg6zCQh8dJNYqmkXqxSVzUZ3srP63 651xhT6xQgeD1MCYpIob6dCjfyOiTUiRAEIHwerTtx6ddKe/4pxDTV8olSH5CaYHZgXb BtrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723411295; x=1724016095; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=THGaYoJGBVo1NFT+ygjobh7TePYwQA+eDE4p3qbAKWk=; b=MNJwWQ+KrxFmlaX3sk17FUR/O0Sp6vOHxRtNe2KI7CP5aMJ6cMWjvVbBdDIo6z5lrZ 1S+A9GyEIeZJxOYpqyBLrCvFZfq3lH58x6O+shDoJ1vr+Z/Yr4XtpGPx2cdVrpnsC7iU z34P1BOGh4xsW7Uxwznw1mTQVCDlU3HQsnl/nmHHJ76Ms9XtapS90ua4mnIMP/wHZFVU NdxcdC1CVY+bbLk6KTai9VUArDzj/7LPnLXFNWQZPCblill4TIlS7h4BHiGCg0GS/s0H Xqv4xy4a2vBuJvNnQmemGz+ZofoTmprTQOXDtlR/FIhzTzzPlVGpwXMDC6AMpu4w+Z3F iYzQ== X-Forwarded-Encrypted: i=1; AJvYcCUr0jfuhzh5w25t42qhMgAkVNDVtACztI2Nmi/aKGZmQLoMkZlip1AcuY2qJC6l/Rg0TWyuNrTw26sPUzeHcmJMnPw= X-Gm-Message-State: AOJu0Ywy1HePSfvpb4sVR/0HvN21Oeu0uV8Y1/7vMjt/3sbGGGfM7Rc7 1jRN69yBtaUwK/GANHv+hBBaVACQaW/V2mgaCvwXUFMBhxvRsSYUrh0hYRO9YqW6Pr8XDYMgzMH QLQ== X-Google-Smtp-Source: AGHT+IGkx0nGnqUgrMKet1uE7S+7hqwGUl1ojaHvm5FZlh6Ngw6ccwePmfpItDUFlv/Deh8AXgWhV/Td5YI= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:c9c:12b4:a1e3:7f10]) (user=yuzhao job=sendgmr) by 2002:a25:dc8d:0:b0:e0e:4350:d7de with SMTP id 3f1490d57ef6-e0eb9a28207mr13988276.9.1723411295441; Sun, 11 Aug 2024 14:21:35 -0700 (PDT) Date: Sun, 11 Aug 2024 15:21:27 -0600 In-Reply-To: <20240811212129.3074314-1-yuzhao@google.com> Mime-Version: 1.0 References: <20240811212129.3074314-1-yuzhao@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240811212129.3074314-2-yuzhao@google.com> Subject: [PATCH mm-unstable v1 1/3] mm/contig_alloc: support __GFP_COMP From: Yu Zhao To: Andrew Morton , Muchun Song Cc: "Matthew Wilcox (Oracle)" , Zi Yan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 74451180012 X-Stat-Signature: wnybnqbjaumxxgxfcxzf9651sknjeujw X-Rspam-User: X-HE-Tag: 1723411296-610013 X-HE-Meta: U2FsdGVkX19T/K0XEjC9MmeJx4D8jdlk+HIqgg3hdxK+uakvvwURmT4Q7I9obt+GsKMq1/IFf/E6qhubkiCPjTZ1KE++fjKm5B33fFM6qdnDC7IXt0beLTrfiNLBxtondrchvfhep0CXAKJ/QCv3/WOcCJcp228nklk8o8nzKSazhbXXIoAvQBijN+1orGaCsw6lOiY/FsYFYUIr8Z434/pq/3vK9h3o/K2bpMqKN/tUkWQHXGfK96XMXksqmi62dgHApjIgfT0V0mmR2K+kd1FfC0AezB9loPdI1t6FEgp1X/9rRuZ1eFOO0aPW7dGQ940oaBCeTPj9hWmgZ3ip9XUYloF93ikfrKM+xDhO8ungUFQBr0qV2sVeiCfMCMfJjAASlSr+PzOfYWXyv6xs4ygLOJd5H6OgesEx+DtVZTcfKP8GFsMQNYD21dDlrDM64ZuoCWqaGZn9WMCG5FS/5dNuTr8UkkK3AQkCOszh0kdC1b2G2GwxFheTaGDNX9pPvZ68WnF/DOgbu2orHT0MHB+C3Zo+8DI1QUSrmwbAjOBE75Vd/+a9cnKynQuusFToimoSRFttDWPU0EDNa3Qwl9Jff40HDWMLnIF5uIgFH7EUT6KPLHy+CtLd5lZ8FGCBaEGWXwU97tctkVJG+ZHUmQCyrN2Qd0f61a838LPxU2TpdgtKrZpnPnJsSKsptqvNOYhmtft7o5nKxB+/X77Mc8iy1EZQRBmncb8ZPE8koI0NXvTirpmQyx72SKSmo1869yKopJP3RMBZlFRP7+r2R1Sv72jvc0DVAxwUPvaiMA9oZluPewOzG4dfNGJNgJc9mbI3R+BHAxU+sVW5bsXrrbqYXCGqSXkbebu7Q/akBVTqy9uMW1443DsnH2UU3IBRiL6KnYk4lfPljzB7upnMWgT1IlA2dXSOhPSmaVuOd05oj38c+Bu4gE1tlda7JvF5wut8yRKtzlweQ+KtABB tins1slm QT4kouQXXrge5zl5EkkqGFTR+n/IsUiflYKIxHBTO5K2L3Ac7ZVEaYzv4jp7D4x4Cle8HkSYq74BCPQMLu2DDLhBDzpX2NnIl9KpGWBRquyFwoc+V3NeFwMI4K7I02I3VyeBOFUmsY9lstjXOiZ2uZJsk5zwc0+JEMDAMEbAPi5COLkNQwRHgu778Lx1d4H7/VUXQ1JQEiizKVOeue0ouFiXLKVoGtupexhvJ978WtFvqfI/oxGkmK2/z5JlO9a8fUEhEZGUSYeXokZ5113Gtt12QM4kj57cyyChiXLzVjlCw77yI56/SW29k0c3lJZy7FbJFVo+UfubuNDL85c4YSk+7DNB/mSsuTJ5yQJhjhd53UDAFiiIpjdnj7AA7VOvH3gzfVioRKWRbjbr4Hx/0FYr6CHvTmhg458WKLBdQnhsj0NRO93l27ky+yZ6nJkblXIdA4uk9em4giJV3mwo58alRqSds35HsMCzCwf+EtmWKuAERE4+rfWHi+jJrSbSbyYPNwhrdZGAK2BSzl+TSbTePJgy1rFBHgxTYlcpRLO6CNsYCOqusQxQX7Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Support __GFP_COMP in alloc_contig_range(). When the flag is set, upon success the function returns a large folio prepared by prep_new_page(), rather than a range of order-0 pages prepared by split_free_pages() (which is renamed from split_map_pages()). alloc_contig_range() can return folios larger than MAX_PAGE_ORDER, e.g., gigantic hugeTLB folios. As a result, on the free path free_one_page() needs to handle this case by split_large_buddy(), in addition to free_contig_range() properly handling large folios by folio_put(). Signed-off-by: Yu Zhao --- mm/compaction.c | 48 +++------------------ mm/internal.h | 9 ++++ mm/page_alloc.c | 111 ++++++++++++++++++++++++++++++++++-------------- 3 files changed, 94 insertions(+), 74 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index eb95e9b435d0..1ebfef98e1d0 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -79,40 +79,6 @@ static inline bool is_via_compact_memory(int order) { return false; } #define COMPACTION_HPAGE_ORDER (PMD_SHIFT - PAGE_SHIFT) #endif -static struct page *mark_allocated_noprof(struct page *page, unsigned int order, gfp_t gfp_flags) -{ - post_alloc_hook(page, order, __GFP_MOVABLE); - return page; -} -#define mark_allocated(...) alloc_hooks(mark_allocated_noprof(__VA_ARGS__)) - -static void split_map_pages(struct list_head *freepages) -{ - unsigned int i, order; - struct page *page, *next; - LIST_HEAD(tmp_list); - - for (order = 0; order < NR_PAGE_ORDERS; order++) { - list_for_each_entry_safe(page, next, &freepages[order], lru) { - unsigned int nr_pages; - - list_del(&page->lru); - - nr_pages = 1 << order; - - mark_allocated(page, order, __GFP_MOVABLE); - if (order) - split_page(page, order); - - for (i = 0; i < nr_pages; i++) { - list_add(&page->lru, &tmp_list); - page++; - } - } - list_splice_init(&tmp_list, &freepages[0]); - } -} - static unsigned long release_free_list(struct list_head *freepages) { int order; @@ -742,11 +708,11 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, * * Non-free pages, invalid PFNs, or zone boundaries within the * [start_pfn, end_pfn) range are considered errors, cause function to - * undo its actions and return zero. + * undo its actions and return zero. cc->freepages[] are empty. * * Otherwise, function returns one-past-the-last PFN of isolated page * (which may be greater then end_pfn if end fell in a middle of - * a free page). + * a free page). cc->freepages[] contain free pages isolated. */ unsigned long isolate_freepages_range(struct compact_control *cc, @@ -754,10 +720,9 @@ isolate_freepages_range(struct compact_control *cc, { unsigned long isolated, pfn, block_start_pfn, block_end_pfn; int order; - struct list_head tmp_freepages[NR_PAGE_ORDERS]; for (order = 0; order < NR_PAGE_ORDERS; order++) - INIT_LIST_HEAD(&tmp_freepages[order]); + INIT_LIST_HEAD(&cc->freepages[order]); pfn = start_pfn; block_start_pfn = pageblock_start_pfn(pfn); @@ -788,7 +753,7 @@ isolate_freepages_range(struct compact_control *cc, break; isolated = isolate_freepages_block(cc, &isolate_start_pfn, - block_end_pfn, tmp_freepages, 0, true); + block_end_pfn, cc->freepages, 0, true); /* * In strict mode, isolate_freepages_block() returns 0 if @@ -807,13 +772,10 @@ isolate_freepages_range(struct compact_control *cc, if (pfn < end_pfn) { /* Loop terminated early, cleanup. */ - release_free_list(tmp_freepages); + release_free_list(cc->freepages); return 0; } - /* __isolate_free_page() does not map the pages */ - split_map_pages(tmp_freepages); - /* We don't use freelists for anything. */ return pfn; } diff --git a/mm/internal.h b/mm/internal.h index acda347620c6..03e795ce755f 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -679,6 +679,15 @@ extern void prep_compound_page(struct page *page, unsigned int order); extern void post_alloc_hook(struct page *page, unsigned int order, gfp_t gfp_flags); + +static inline struct page *post_alloc_hook_noprof(struct page *page, unsigned int order, + gfp_t gfp_flags) +{ + post_alloc_hook(page, order, __GFP_MOVABLE); + return page; +} +#define mark_allocated(...) alloc_hooks(post_alloc_hook_noprof(__VA_ARGS__)) + extern bool free_pages_prepare(struct page *page, unsigned int order); extern int user_min_free_kbytes; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 84a7154fde93..6c801404a108 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1196,16 +1196,36 @@ static void free_pcppages_bulk(struct zone *zone, int count, spin_unlock_irqrestore(&zone->lock, flags); } +/* Split a multi-block free page into its individual pageblocks */ +static void split_large_buddy(struct zone *zone, struct page *page, + unsigned long pfn, int order, fpi_t fpi) +{ + unsigned long end = pfn + (1 << order); + + VM_WARN_ON_ONCE(!IS_ALIGNED(pfn, 1 << order)); + /* Caller removed page from freelist, buddy info cleared! */ + VM_WARN_ON_ONCE(PageBuddy(page)); + + if (order > pageblock_order) + order = pageblock_order; + + while (pfn != end) { + int mt = get_pfnblock_migratetype(page, pfn); + + __free_one_page(page, pfn, zone, order, mt, fpi); + pfn += 1 << order; + page = pfn_to_page(pfn); + } +} + static void free_one_page(struct zone *zone, struct page *page, unsigned long pfn, unsigned int order, fpi_t fpi_flags) { unsigned long flags; - int migratetype; spin_lock_irqsave(&zone->lock, flags); - migratetype = get_pfnblock_migratetype(page, pfn); - __free_one_page(page, pfn, zone, order, migratetype, fpi_flags); + split_large_buddy(zone, page, pfn, order, fpi_flags); spin_unlock_irqrestore(&zone->lock, flags); } @@ -1697,27 +1717,6 @@ static unsigned long find_large_buddy(unsigned long start_pfn) return start_pfn; } -/* Split a multi-block free page into its individual pageblocks */ -static void split_large_buddy(struct zone *zone, struct page *page, - unsigned long pfn, int order) -{ - unsigned long end_pfn = pfn + (1 << order); - - VM_WARN_ON_ONCE(order <= pageblock_order); - VM_WARN_ON_ONCE(pfn & (pageblock_nr_pages - 1)); - - /* Caller removed page from freelist, buddy info cleared! */ - VM_WARN_ON_ONCE(PageBuddy(page)); - - while (pfn != end_pfn) { - int mt = get_pfnblock_migratetype(page, pfn); - - __free_one_page(page, pfn, zone, pageblock_order, mt, FPI_NONE); - pfn += pageblock_nr_pages; - page = pfn_to_page(pfn); - } -} - /** * move_freepages_block_isolate - move free pages in block for page isolation * @zone: the zone @@ -1758,7 +1757,7 @@ bool move_freepages_block_isolate(struct zone *zone, struct page *page, del_page_from_free_list(buddy, zone, order, get_pfnblock_migratetype(buddy, pfn)); set_pageblock_migratetype(page, migratetype); - split_large_buddy(zone, buddy, pfn, order); + split_large_buddy(zone, buddy, pfn, order, FPI_NONE); return true; } @@ -1769,7 +1768,7 @@ bool move_freepages_block_isolate(struct zone *zone, struct page *page, del_page_from_free_list(page, zone, order, get_pfnblock_migratetype(page, pfn)); set_pageblock_migratetype(page, migratetype); - split_large_buddy(zone, page, pfn, order); + split_large_buddy(zone, page, pfn, order, FPI_NONE); return true; } move: @@ -6482,6 +6481,31 @@ int __alloc_contig_migrate_range(struct compact_control *cc, return (ret < 0) ? ret : 0; } +static void split_free_pages(struct list_head *list) +{ + int order; + + for (order = 0; order < NR_PAGE_ORDERS; order++) { + struct page *page, *next; + int nr_pages = 1 << order; + + list_for_each_entry_safe(page, next, &list[order], lru) { + int i; + + mark_allocated(page, order, __GFP_MOVABLE); + if (!order) + continue; + + split_page(page, order); + + /* add all subpages to the order-0 head, in sequence */ + list_del(&page->lru); + for (i = 0; i < nr_pages; i++) + list_add_tail(&page[i].lru, &list[0]); + } + } +} + /** * alloc_contig_range() -- tries to allocate given range of pages * @start: start PFN to allocate @@ -6594,12 +6618,25 @@ int alloc_contig_range_noprof(unsigned long start, unsigned long end, goto done; } - /* Free head and tail (if any) */ - if (start != outer_start) - free_contig_range(outer_start, start - outer_start); - if (end != outer_end) - free_contig_range(end, outer_end - end); + if (!(gfp_mask & __GFP_COMP)) { + split_free_pages(cc.freepages); + /* Free head and tail (if any) */ + if (start != outer_start) + free_contig_range(outer_start, start - outer_start); + if (end != outer_end) + free_contig_range(end, outer_end - end); + } else if (start == outer_start && end == outer_end && is_power_of_2(end - start)) { + struct page *head = pfn_to_page(start); + int order = ilog2(end - start); + + check_new_pages(head, order); + prep_new_page(head, order, gfp_mask, 0); + } else { + ret = -EINVAL; + WARN(true, "PFN range: requested [%lu, %lu), allocated [%lu, %lu)\n", + start, end, outer_start, outer_end); + } done: undo_isolate_page_range(start, end, migratetype); return ret; @@ -6708,6 +6745,18 @@ struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask, void free_contig_range(unsigned long pfn, unsigned long nr_pages) { unsigned long count = 0; + struct folio *folio = pfn_folio(pfn); + + if (folio_test_large(folio)) { + int expected = folio_nr_pages(folio); + + if (nr_pages == expected) + folio_put(folio); + else + WARN(true, "PFN %lu: nr_pages %lu != expected %d\n", + pfn, nr_pages, expected); + return; + } for (; nr_pages--; pfn++) { struct page *page = pfn_to_page(pfn); From patchwork Sun Aug 11 21:21:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13759889 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B19CFC3DA4A for ; Sun, 11 Aug 2024 21:21:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3F4D66B0098; Sun, 11 Aug 2024 17:21:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3A5B36B009A; Sun, 11 Aug 2024 17:21:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 21F0F6B009E; Sun, 11 Aug 2024 17:21:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id F2B5E6B0098 for ; Sun, 11 Aug 2024 17:21:40 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id ADA06A05BC for ; Sun, 11 Aug 2024 21:21:40 +0000 (UTC) X-FDA: 82441236360.09.E9311BB Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf21.hostedemail.com (Postfix) with ESMTP id DECC71C0010 for ; Sun, 11 Aug 2024 21:21:38 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="oL/R5YbU"; spf=pass (imf21.hostedemail.com: domain of 3YSu5ZgYKCNgSOTB4IAIIAF8.6IGFCHOR-GGEP46E.ILA@flex--yuzhao.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3YSu5ZgYKCNgSOTB4IAIIAF8.6IGFCHOR-GGEP46E.ILA@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723411288; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MiWur5WPUCFOabKPS7p8sIQsgKEHOpII1M/7n2SnGyU=; b=6OtADlrH4DBXA4ct7CEjhAdsUuOQBBWnorP0UFcZGfr1BpXkw4mVmG9XesmWlwb/q/m35o j/5OWnuICWe6bfr3uTrZ6dKbkXVBi0yqqFeuhvFsi8GzHeCqrpeO4SvPlujJxpN+XfBuq1 18U0sZrxxfA2wUmgO4fc3/5mnrvwZn4= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="oL/R5YbU"; spf=pass (imf21.hostedemail.com: domain of 3YSu5ZgYKCNgSOTB4IAIIAF8.6IGFCHOR-GGEP46E.ILA@flex--yuzhao.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3YSu5ZgYKCNgSOTB4IAIIAF8.6IGFCHOR-GGEP46E.ILA@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723411288; a=rsa-sha256; cv=none; b=ABcUMia0KBMts+C+M+GhvYN7byNs6wCnYqWN3cC3sIvTNAOWlz4vOyw0mhYMLPqj6WrSd6 V5TgGLB4MeQwth0lMlHl8vUAMhPge3Hiu6osKAtPySSbg69D7pJAeDgjKtDwFrlatwufWR 2Dy3LEiVVX4PvbDbRlZtn3TdsRtFlWU= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6886cd07673so88556577b3.3 for ; Sun, 11 Aug 2024 14:21:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723411298; x=1724016098; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MiWur5WPUCFOabKPS7p8sIQsgKEHOpII1M/7n2SnGyU=; b=oL/R5YbUTk82yE+6Wp0uatGmzURJEh9Ec27HeI1r9tI4/1d1MEHUvKcLyPkfNAJsOa 3R3WFwF5EzLYWmPHtK/UO7q3qc3cOdK7mhn9GrMxr9y98AQBhYea406+hrrwxZYfp9lM fmEN9WtGUgpSuNaRDf+VG5PUEUTtvkqRLq+QX+YqqmcAqx56NGfEcQT7zXZFbwM1s3KV d45v+iq6q8m+heht1OiW2yrHSD5Qb1bdaDNFwQ8rlB58r9EwzwtfY1iw8V5Z5XLHxzD/ mxytDk5oFUw7fZiezifsPVnVid8U4P+rdmndrciEEQJTKyyFEEhYteoqOkjR0Yjp86Fc PVcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723411298; x=1724016098; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MiWur5WPUCFOabKPS7p8sIQsgKEHOpII1M/7n2SnGyU=; b=Zw4KGcle7XGPIo2oMbMCMA1YppTUGnKSo8hGb4KoeFAX0Tgubhuhf/+D48q+t9VlH0 ZpZJpt2XHPOAqpPMK5kIBAZGDIdlMbHX24T3Nh/nrnoTpjHygt7PAoHoas1TQZTyc4Vk cHWg1mBODI7iqzyUXWmiA3a5cqDkHmnxZihAqGPT/OE5ErYKkD619O31S8wIuYlU+IuH v2DXUnJppDyoDgyeDXz0e+OEl6wZdvSv/9EYgx0O2EROgeNS0fWhdGAxxO1LeQF/AK76 9MST6EC03NU4670nB7odUCwbMeX+AGQuy4OG2gBJ81C38UOXORguSY2lnNldkxkTiKiU 8+Ww== X-Forwarded-Encrypted: i=1; AJvYcCXWW+G6ASjvjGQ5YGgKE/XWA+eQ91kv4ShATk3y+zwFS+cvffTsNuD9Ytdhdt51WYV3d8obErV7KChqGCbWtmn0coY= X-Gm-Message-State: AOJu0Yw3qt+AEnJP5kzBcy7fSpb0W1u0c4eXHbxFj/U08uIBgjOaiJlM R58dL3QsRe94LzTMzIWenCQb0UTRNMRDof7zL1Wh+sZP89Dzknct3XbDByjqy6we2zglejVBeis Xiw== X-Google-Smtp-Source: AGHT+IEHCLQ7KUCLVKLYCjPKQE5Er4aNEmGlPQGmmKueq/o3fFhVzS87gYKaHPAinDZLXmdPberE0nnpI54= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:c9c:12b4:a1e3:7f10]) (user=yuzhao job=sendgmr) by 2002:a05:690c:2e13:b0:68e:edf4:7b80 with SMTP id 00721157ae682-69ec86beceamr2524307b3.5.1723411297967; Sun, 11 Aug 2024 14:21:37 -0700 (PDT) Date: Sun, 11 Aug 2024 15:21:28 -0600 In-Reply-To: <20240811212129.3074314-1-yuzhao@google.com> Mime-Version: 1.0 References: <20240811212129.3074314-1-yuzhao@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240811212129.3074314-3-yuzhao@google.com> Subject: [PATCH mm-unstable v1 2/3] mm/cma: add cma_alloc_folio() From: Yu Zhao To: Andrew Morton , Muchun Song Cc: "Matthew Wilcox (Oracle)" , Zi Yan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao X-Rspam-User: X-Stat-Signature: u38ff6q5a3i7mbbtmnb5opytt3wai6de X-Rspamd-Queue-Id: DECC71C0010 X-Rspamd-Server: rspam11 X-HE-Tag: 1723411298-63873 X-HE-Meta: U2FsdGVkX19FMdv3bcBaqgXivEecQ22nhKarr6ZYSP3fj0SFNFPBnEtyCugqVSCDAu8pSaW9vMHQFtl3Qd9oAfPF4G0qXLWbo7eQ+BFSLro7+fyOK8bmlsnTkJcksICtBExyA8c4c67Bl3DwTmrul3gM/SPMRxUAmi/5ORZOCHvL/IB1/LmlNK00dd6vNDwG90sNPd09v+6BSOj5fLtzp0nRhsx4v4vlD+kd+QAs3iPAfZYVCagtzbd9Q43ASrLdcIt7fobakBvVLF+pAROCStQGRa3GFSFjIo1/qMR1NeDt2AYtoegn2GOJhBnR3xNRiyawmGutASqlPiYA/xVo/whT3OhAvmtC26kBihFv9ZwYL3pOh5T4NHAHsmfPEcILFjyYrpoI1Uigj0IGjIED/a0LMe2r/VAdWNcO83yRylbaAvIve/4krwzi2IGVkGgXwqbq19luCuq/Zzst4BJ340va4jebBTVsaBVRVGjkRl/bGt7Yt50hZXqA/bK9B0II20/dDLOrOYZMBXLI2eZ2+gY7/UzFL98Cp2CdIggDVZn1ZzbkfJqefJf89oMAFZy2b/4QqSv1r+qbVFxWlQPbn3/yITim0QXf5kPpjVJr7Rqhbr6qpplnd91GYOYN0yjx6ofRRuUJE6SDIC6IYAiN2nFWlSVnNDBS6a/8rwWXM2K8UZmR+sJSjaAa6rzm4SWKzw9J85ss8w1JDpxETczS37BWQnmldTdmaeMb31S8LQhWocTyV15K3VS+Xh2EYVFNlgb27KsF0NgoY28SSilPWQDHdwA+HcK5QqG2NT+gQNtA2LO46zRXKI5YkN/X3gU3j7iEDITmQGf/dln9LNlrovxGvwbEsVY0Xh7iaIWfVNqH3S9ZhoyN0C2HSjmrUtoE5pD8hnixtWB77T8Ecw+6ICDu3k9Fw72sN/RIOMC5UfGJF9+LWaJCKki9OuHkIcveDg4NsqRTLJOyk14TsSR mMzAKHEg w73mZIDpeMu9C0ObzI+L1A3wwEoCfjasnTyKAcDolXr1eA8xOf2Yd96//bMAZoKMqdHRnhTNMF4A3l9OFyg2T01uyyIuRPxQReKUlp3RCg4EjXWbw8+npfTRfBQ6cYDAG/xqE2Ew+BPoEDN3hotvRK3hcUakgojm5TAB/Iv6d75MByxNHm5dLLJPKBhrkcDpdfSZ/dnub9zEcV3eOYzNQu1zRQsYsEgoPt0+P9Z97Vo+MmeKT9jd94DrPKwtpJdGN3LtNDXeEW3N7wYzePwATriScBprmIzkrCcx35Gpapb8RIMX2Gw1DdsMRVpHZMeCmT1+mFgk3qLCIC+wqAhbj7cmt3cN0V3VlXkMiFYcpPxMvI/oOZ2vndqBUA7Jp1z+KPcqX7aooqlsQW4AUs/QtbLpzH3aHVhGKNpIfZrAtj28Fb+pvQv5TI1dPeCp0dv6iFGVbZXntsNsaArG4zB9ty2wvvFPjTGCT6Pzg6SoCrm1n9HSGwwjP7RRqXBh3BIoknEe7/iCNEoU0TyHTdHOA4Grv2BW/qnZEZXcCMgIXRvK8kk/ijExJ5tHFgq0GN/F0QVgX X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: With alloc_contig_range() and free_contig_range() supporting large folios, CMA can allocate and free large folios too, by cma_alloc_folio() and cma_release(). Signed-off-by: Yu Zhao --- include/linux/cma.h | 1 + mm/cma.c | 47 ++++++++++++++++++++++++++++++--------------- 2 files changed, 33 insertions(+), 15 deletions(-) diff --git a/include/linux/cma.h b/include/linux/cma.h index 9db877506ea8..086553fbda73 100644 --- a/include/linux/cma.h +++ b/include/linux/cma.h @@ -46,6 +46,7 @@ extern int cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, struct cma **res_cma); extern struct page *cma_alloc(struct cma *cma, unsigned long count, unsigned int align, bool no_warn); +extern struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp); extern bool cma_pages_valid(struct cma *cma, const struct page *pages, unsigned long count); extern bool cma_release(struct cma *cma, const struct page *pages, unsigned long count); diff --git a/mm/cma.c b/mm/cma.c index 95d6950e177b..46feb06db8e7 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -403,18 +403,8 @@ static void cma_debug_show_areas(struct cma *cma) spin_unlock_irq(&cma->lock); } -/** - * cma_alloc() - allocate pages from contiguous area - * @cma: Contiguous memory region for which the allocation is performed. - * @count: Requested number of pages. - * @align: Requested alignment of pages (in PAGE_SIZE order). - * @no_warn: Avoid printing message about failed allocation - * - * This function allocates part of contiguous memory on specific - * contiguous memory area. - */ -struct page *cma_alloc(struct cma *cma, unsigned long count, - unsigned int align, bool no_warn) +static struct page *__cma_alloc(struct cma *cma, unsigned long count, + unsigned int align, gfp_t gfp) { unsigned long mask, offset; unsigned long pfn = -1; @@ -463,8 +453,7 @@ struct page *cma_alloc(struct cma *cma, unsigned long count, pfn = cma->base_pfn + (bitmap_no << cma->order_per_bit); mutex_lock(&cma_mutex); - ret = alloc_contig_range(pfn, pfn + count, MIGRATE_CMA, - GFP_KERNEL | (no_warn ? __GFP_NOWARN : 0)); + ret = alloc_contig_range(pfn, pfn + count, MIGRATE_CMA, gfp); mutex_unlock(&cma_mutex); if (ret == 0) { page = pfn_to_page(pfn); @@ -494,7 +483,7 @@ struct page *cma_alloc(struct cma *cma, unsigned long count, page_kasan_tag_reset(nth_page(page, i)); } - if (ret && !no_warn) { + if (ret && !(gfp & __GFP_NOWARN)) { pr_err_ratelimited("%s: %s: alloc failed, req-size: %lu pages, ret: %d\n", __func__, cma->name, count, ret); cma_debug_show_areas(cma); @@ -513,6 +502,34 @@ struct page *cma_alloc(struct cma *cma, unsigned long count, return page; } +/** + * cma_alloc() - allocate pages from contiguous area + * @cma: Contiguous memory region for which the allocation is performed. + * @count: Requested number of pages. + * @align: Requested alignment of pages (in PAGE_SIZE order). + * @no_warn: Avoid printing message about failed allocation + * + * This function allocates part of contiguous memory on specific + * contiguous memory area. + */ +struct page *cma_alloc(struct cma *cma, unsigned long count, + unsigned int align, bool no_warn) +{ + return __cma_alloc(cma, count, align, GFP_KERNEL | (no_warn ? __GFP_NOWARN : 0)); +} + +struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp) +{ + struct page *page; + + if (WARN_ON(order && !(gfp | __GFP_COMP))) + return NULL; + + page = __cma_alloc(cma, 1 << order, order, gfp); + + return page ? page_folio(page) : NULL; +} + bool cma_pages_valid(struct cma *cma, const struct page *pages, unsigned long count) { From patchwork Sun Aug 11 21:21:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13759890 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 634FCC3DA4A for ; Sun, 11 Aug 2024 21:21:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E30626B009E; Sun, 11 Aug 2024 17:21:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D6A2F6B009F; Sun, 11 Aug 2024 17:21:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BBD3E6B00A0; Sun, 11 Aug 2024 17:21:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 93F896B009E for ; Sun, 11 Aug 2024 17:21:43 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 47D0D8053E for ; Sun, 11 Aug 2024 21:21:43 +0000 (UTC) X-FDA: 82441236486.02.F8FDF87 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf05.hostedemail.com (Postfix) with ESMTP id 76D39100009 for ; Sun, 11 Aug 2024 21:21:41 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=o40zgmZf; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3ZCu5ZgYKCNsVRWE7LDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3ZCu5ZgYKCNsVRWE7LDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723411290; a=rsa-sha256; cv=none; b=WJHYBZXIHIMCMIfzrUqP20muBq7OCJLRdCe1SQcjRNO3xbX1tkTH17Tsmgqk0ihU73w6zp SxblWbi5QqLJWoQ2BDN/YbPIWcPJiO05g8ADs/8FAbRkvcCqohjhFD/JL7z9AqE8A3XGgz Hqu+JLSoWXPxZgxnqNGO/oLzJedfjw8= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=o40zgmZf; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3ZCu5ZgYKCNsVRWE7LDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yuzhao.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3ZCu5ZgYKCNsVRWE7LDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723411290; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=plPNUwSc20wvxDLG/5V4ONiycR26+90aEQUlSgRQtvI=; b=NaQpMjssoFvc7dw2uryK7M9gP+oQa6UPhuOGg/Rkp5qY0Bzgh19ek0r+a7+9lG9QRjkJiZ +k+64OcOWvJx53dp/hvJhRm11HMuajl3r6DpeZOVZOXgw+CQzBnH/7n6YmNlUbqSlB5pqm OUbxCJUah9sRV4u3xCF2+5QJSipZzfM= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-66a2aee82a0so73082727b3.0 for ; Sun, 11 Aug 2024 14:21:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723411300; x=1724016100; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=plPNUwSc20wvxDLG/5V4ONiycR26+90aEQUlSgRQtvI=; b=o40zgmZfpPUUM0DFAvfXfrIfpfzulCvnSPiveUr0Qgi4GmhQZ3ycCrXUqgXHrUh9Jy FLOG+MuiL2QAVE0uyVR/rzrZt/xzxOtr8mmOwDMfzChlt47NzKkGMos0fmqlbk+7cda6 eIiVQx+IHc3h4bYTwBLzqnCLMosh7V1DDT5ZPm51wvfIV1ddA51+JnjHLhOBj5a6rVBA QySqicihQxuhZonN9YkXbDp6oKElXrpUlpIU+ma7ST4wcDc60cob66WgG22A/F4MZL/l AMzwp1M9HsPCkY4QeFq8sr0QfKULIqlPdElHxbSmJAVmZbdIolbKDXrbVlp3b5BaSYHd m4SA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723411300; x=1724016100; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=plPNUwSc20wvxDLG/5V4ONiycR26+90aEQUlSgRQtvI=; b=xB7Z5GDo3XBAyt56KQwq+wwOex861maHnUjclBz9lf+5jtM68jJqIgP3QTwfEdfLYL 9L679arPM8mYGRdbm4om4cIC5f8PaKd62DEnPM5f/6VygqfKKbtq+Ab6OrrZXHNmApzt 5ysFCe+Uk03GcjedlDFvIyzmqTM5gSjPfsRL6mM22lkR05V0/nNjTam360jgzcDkyFck 6JARnJ36MSVb30eMLiQKr1Lb4fwBCVvVPcoMiKVUO1jzSt6cP2MASWQeO9OAfHt+15yp IuQ1p900j99/plNT1rvTPvDVPdcezIS4aacaZ+qIVw7dBgtv35EYxkZGMVAsB3SiFh+N VR/A== X-Forwarded-Encrypted: i=1; AJvYcCXhLY8FnHxjbFKsXNYRnRZzbAzIY9ZKClPNrHezB9hMgxrhNkvvbRFoMcWJtBOgzjSNTLtSASTeumciJNsiQuLbGnU= X-Gm-Message-State: AOJu0Yy+UZF53gmSqGlB4Xx5nIhOHJw8ZJNvI7cPruX5NbPMXXI6r8ky K6IyzSU8+0Y2QSzJh47GgZhwPko5haLnnlGWSTzhcmJBS21s+AZDxCz6go+G9y6XiAGO+GzTT17 n2A== X-Google-Smtp-Source: AGHT+IHabmrGk91oLaByBgnfM+5JbvP9zqcwTHpzXiVoA50OMZR/9U6IX+5qqzgzKEbG6UqMgpxrz3SgVrA= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:c9c:12b4:a1e3:7f10]) (user=yuzhao job=sendgmr) by 2002:a81:c64b:0:b0:62f:a56a:cee8 with SMTP id 00721157ae682-69ec54adf13mr3154657b3.3.1723411300470; Sun, 11 Aug 2024 14:21:40 -0700 (PDT) Date: Sun, 11 Aug 2024 15:21:29 -0600 In-Reply-To: <20240811212129.3074314-1-yuzhao@google.com> Mime-Version: 1.0 References: <20240811212129.3074314-1-yuzhao@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240811212129.3074314-4-yuzhao@google.com> Subject: [PATCH mm-unstable v1 3/3] mm/hugetlb: use __GFP_COMP for gigantic folios From: Yu Zhao To: Andrew Morton , Muchun Song Cc: "Matthew Wilcox (Oracle)" , Zi Yan , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao X-Rspam-User: X-Rspamd-Queue-Id: 76D39100009 X-Rspamd-Server: rspam01 X-Stat-Signature: hu98c17swrsr9wb4hsi5ky4fmk7xehr8 X-HE-Tag: 1723411301-762442 X-HE-Meta: U2FsdGVkX18bGe4PGXrRq4Io2Y1aqIYpcbG8+pKcKmXM0RX04cO/821b8Q/Nj8UmFQDdhebtEayaZO0QHlFZwef98uY7g4+IsAOJtMJzuLURFzXvPiZ7hL0Aobqz4nY/0dIIg9RMb4bLgZfkUjAWvgTFGZ7+Pf4wc51ft9K22JegpJOClNk0nR5HWN5mjXiuSkFwfU9YxGSpILlZI/RtJwpoycJQD5Wa3U01Da77aPEsQwtccUunVb/u8iw1bHb1+Mf9fc2zcGiNYE+oefFVgXgCq1a2Cy+KteE2F7F6OYgpGPS2FifoTgdCbf5X4yAvl2YBRNkN7lw1ThF7g1JL/EWi0L3/iS6SGs7/lmbJXcFBShgMvKU9GPbu/0J7X/FYlRhfvvLNyMQ5fXQno06iuxC9juQYkP6Q8WPz24BmBl+QTmDa33m3gdwz6oYAFYP6eHUSFXDAV2+oegH57qAAjpJdFfRcg6A0QTsiJzz16FOGVJgkUNCWmIy3/DktpbQ2rPFpCKG5zqCmBiP1bveLV/dUMJGWOO4d3GkeUWiDvC5OROu4YiZ/fUmRigBe2g4+ShlLJWTAq+jcwJUdW4LazHm5TjCLCTXHdKGgrxWfOi8EqG8tWS4VtnvwiJ+iiKzdgaEypl0oQNV4panMQocFnwMxSrh5OjZ2A+IVDil1msV9lqBTawfNjk2fFIaJYMNj4DtNkisghLTouwthZaTfdngWZcVUkqHC63njx5bvdxdRUCupAc4t3z5jyTIUOzSkqmkYFZcbSPDbGWWl3BfFkojjZ+iY1hm72JP3t2ixXk4GC8DmznCtV5NLvc6PlJDSiLDqXpa6Bbh5PekfJryOpiewR+ndP+F+q/IQZDSytV14NFsFrw7ZbdkfD0qX9j0zpZZN3NQ86BWlP4pT0q4lklyxtmesjf3G2lyIY7tfmnn9Zo0Bv5Ihxxl8zkx0n6DmdI+uNHbTV1v44BoZcQm asGDATh/ FDNLFBlrs6J4OD3rFVh5H3opSkhA4mW86uO55yAe9oeq8NQ9AJhd6niozdp+IY3V2T3Xm0iMmNYgDCxxIHDaTqZ7ac5vgAPXd7IW96Emy9VteWd5H1nWXdpff45+FDyll6DuwHIFNn8sp3zjJY1hGAVFA4O2vaRYXi25KA5WoFBqnjb5qbWLv33D1JA2+5EXJJ3cGhKeMMy0KjYJfabT6EGC92srsuueaEJzs+DsX1Chq7LmBAiIu4zrDU8Iowkmzyiw43uukNGnWQRrVWJzEU2xNzqdy5WO1tyARA8OC51ev6nkKT5P9ZNs3VJQgvtSJX/v67dYsOkumMWPo2LZ+0mOEDFKz/AQcQofObgzgm0hgkMeomub4qHQdX6gUnxuqUp1E/lVG/kKG1oTX1QVFiJ40pgZbMFLRiPcg/8eO+YBI2Zdl1flzet4MGOtdVPpbnwHcJ+R2WagKeC680NQ59Zm7+vQD3Z+9bpCXdubrk9r5CiA+ctl/N+I6AMlzGPPrjqkQRbwb2frJ1I2ry9L22g1UH2kfUuRmqX843YOJIHvHsQLK2g6Zq5beig== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use __GFP_COMP for gigantic folios to greatly reduce not only the code but also the allocation and free time. LOC (approximately): -200, +50 Allocate and free 500 1GB hugeTLB memory without HVO by: time echo 500 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages time echo 0 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages Before After Alloc ~13s ~10s Free ~15s <1s The above magnitude generally holds for multiple x86 and arm64 CPU models. Signed-off-by: Yu Zhao --- include/linux/hugetlb.h | 9 +- mm/hugetlb.c | 244 ++++++++-------------------------------- 2 files changed, 50 insertions(+), 203 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 3100a52ceb73..98c47c394b89 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -896,10 +896,11 @@ static inline bool hugepage_movable_supported(struct hstate *h) /* Movability of hugepages depends on migration support. */ static inline gfp_t htlb_alloc_mask(struct hstate *h) { - if (hugepage_movable_supported(h)) - return GFP_HIGHUSER_MOVABLE; - else - return GFP_HIGHUSER; + gfp_t gfp = __GFP_COMP | __GFP_NOWARN; + + gfp |= hugepage_movable_supported(h) ? GFP_HIGHUSER_MOVABLE : GFP_HIGHUSER; + + return gfp; } static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 1c13e65ab119..691f63408d50 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1512,43 +1512,7 @@ static int hstate_next_node_to_free(struct hstate *h, nodemask_t *nodes_allowed) ((node = hstate_next_node_to_free(hs, mask)) || 1); \ nr_nodes--) -/* used to demote non-gigantic_huge pages as well */ -static void __destroy_compound_gigantic_folio(struct folio *folio, - unsigned int order, bool demote) -{ - int i; - int nr_pages = 1 << order; - struct page *p; - - atomic_set(&folio->_entire_mapcount, 0); - atomic_set(&folio->_large_mapcount, 0); - atomic_set(&folio->_pincount, 0); - - for (i = 1; i < nr_pages; i++) { - p = folio_page(folio, i); - p->flags &= ~PAGE_FLAGS_CHECK_AT_FREE; - p->mapping = NULL; - clear_compound_head(p); - if (!demote) - set_page_refcounted(p); - } - - __folio_clear_head(folio); -} - -static void destroy_compound_hugetlb_folio_for_demote(struct folio *folio, - unsigned int order) -{ - __destroy_compound_gigantic_folio(folio, order, true); -} - #ifdef CONFIG_ARCH_HAS_GIGANTIC_PAGE -static void destroy_compound_gigantic_folio(struct folio *folio, - unsigned int order) -{ - __destroy_compound_gigantic_folio(folio, order, false); -} - static void free_gigantic_folio(struct folio *folio, unsigned int order) { /* @@ -1569,38 +1533,52 @@ static void free_gigantic_folio(struct folio *folio, unsigned int order) static struct folio *alloc_gigantic_folio(struct hstate *h, gfp_t gfp_mask, int nid, nodemask_t *nodemask) { - struct page *page; - unsigned long nr_pages = pages_per_huge_page(h); + struct folio *folio; + int order = huge_page_order(h); + bool retry = false; + if (nid == NUMA_NO_NODE) nid = numa_mem_id(); - +retry: + folio = NULL; #ifdef CONFIG_CMA { int node; - if (hugetlb_cma[nid]) { - page = cma_alloc(hugetlb_cma[nid], nr_pages, - huge_page_order(h), true); - if (page) - return page_folio(page); - } + if (hugetlb_cma[nid]) + folio = cma_alloc_folio(hugetlb_cma[nid], order, gfp_mask); - if (!(gfp_mask & __GFP_THISNODE)) { + if (!folio && !(gfp_mask & __GFP_THISNODE)) { for_each_node_mask(node, *nodemask) { if (node == nid || !hugetlb_cma[node]) continue; - page = cma_alloc(hugetlb_cma[node], nr_pages, - huge_page_order(h), true); - if (page) - return page_folio(page); + folio = cma_alloc_folio(hugetlb_cma[node], order, gfp_mask); + if (folio) + break; } } } #endif + if (!folio) { + struct page *page = alloc_contig_pages(1 << order, gfp_mask, nid, nodemask); - page = alloc_contig_pages(nr_pages, gfp_mask, nid, nodemask); - return page ? page_folio(page) : NULL; + if (!page) + return NULL; + + folio = page_folio(page); + } + + if (folio_ref_freeze(folio, 1)) + return folio; + + pr_warn("HugeTLB: unexpected refcount on PFN %lu\n", folio_pfn(folio)); + free_gigantic_folio(folio, order); + if (!retry) { + retry = true; + goto retry; + } + return NULL; } #else /* !CONFIG_CONTIG_ALLOC */ @@ -1619,8 +1597,6 @@ static struct folio *alloc_gigantic_folio(struct hstate *h, gfp_t gfp_mask, } static inline void free_gigantic_folio(struct folio *folio, unsigned int order) { } -static inline void destroy_compound_gigantic_folio(struct folio *folio, - unsigned int order) { } #endif /* @@ -1747,19 +1723,17 @@ static void __update_and_free_hugetlb_folio(struct hstate *h, folio_clear_hugetlb_hwpoison(folio); folio_ref_unfreeze(folio, 1); + INIT_LIST_HEAD(&folio->_deferred_list); /* * Non-gigantic pages demoted from CMA allocated gigantic pages * need to be given back to CMA in free_gigantic_folio. */ if (hstate_is_gigantic(h) || - hugetlb_cma_folio(folio, huge_page_order(h))) { - destroy_compound_gigantic_folio(folio, huge_page_order(h)); + hugetlb_cma_folio(folio, huge_page_order(h))) free_gigantic_folio(folio, huge_page_order(h)); - } else { - INIT_LIST_HEAD(&folio->_deferred_list); + else folio_put(folio); - } } /* @@ -2032,95 +2006,6 @@ static void prep_new_hugetlb_folio(struct hstate *h, struct folio *folio, int ni spin_unlock_irq(&hugetlb_lock); } -static bool __prep_compound_gigantic_folio(struct folio *folio, - unsigned int order, bool demote) -{ - int i, j; - int nr_pages = 1 << order; - struct page *p; - - __folio_clear_reserved(folio); - for (i = 0; i < nr_pages; i++) { - p = folio_page(folio, i); - - /* - * For gigantic hugepages allocated through bootmem at - * boot, it's safer to be consistent with the not-gigantic - * hugepages and clear the PG_reserved bit from all tail pages - * too. Otherwise drivers using get_user_pages() to access tail - * pages may get the reference counting wrong if they see - * PG_reserved set on a tail page (despite the head page not - * having PG_reserved set). Enforcing this consistency between - * head and tail pages allows drivers to optimize away a check - * on the head page when they need know if put_page() is needed - * after get_user_pages(). - */ - if (i != 0) /* head page cleared above */ - __ClearPageReserved(p); - /* - * Subtle and very unlikely - * - * Gigantic 'page allocators' such as memblock or cma will - * return a set of pages with each page ref counted. We need - * to turn this set of pages into a compound page with tail - * page ref counts set to zero. Code such as speculative page - * cache adding could take a ref on a 'to be' tail page. - * We need to respect any increased ref count, and only set - * the ref count to zero if count is currently 1. If count - * is not 1, we return an error. An error return indicates - * the set of pages can not be converted to a gigantic page. - * The caller who allocated the pages should then discard the - * pages using the appropriate free interface. - * - * In the case of demote, the ref count will be zero. - */ - if (!demote) { - if (!page_ref_freeze(p, 1)) { - pr_warn("HugeTLB page can not be used due to unexpected inflated ref count\n"); - goto out_error; - } - } else { - VM_BUG_ON_PAGE(page_count(p), p); - } - if (i != 0) - set_compound_head(p, &folio->page); - } - __folio_set_head(folio); - /* we rely on prep_new_hugetlb_folio to set the hugetlb flag */ - folio_set_order(folio, order); - atomic_set(&folio->_entire_mapcount, -1); - atomic_set(&folio->_large_mapcount, -1); - atomic_set(&folio->_pincount, 0); - return true; - -out_error: - /* undo page modifications made above */ - for (j = 0; j < i; j++) { - p = folio_page(folio, j); - if (j != 0) - clear_compound_head(p); - set_page_refcounted(p); - } - /* need to clear PG_reserved on remaining tail pages */ - for (; j < nr_pages; j++) { - p = folio_page(folio, j); - __ClearPageReserved(p); - } - return false; -} - -static bool prep_compound_gigantic_folio(struct folio *folio, - unsigned int order) -{ - return __prep_compound_gigantic_folio(folio, order, false); -} - -static bool prep_compound_gigantic_folio_for_demote(struct folio *folio, - unsigned int order) -{ - return __prep_compound_gigantic_folio(folio, order, true); -} - /* * Find and lock address space (mapping) in write mode. * @@ -2159,7 +2044,6 @@ static struct folio *alloc_buddy_hugetlb_folio(struct hstate *h, */ if (node_alloc_noretry && node_isset(nid, *node_alloc_noretry)) alloc_try_hard = false; - gfp_mask |= __GFP_COMP|__GFP_NOWARN; if (alloc_try_hard) gfp_mask |= __GFP_RETRY_MAYFAIL; if (nid == NUMA_NO_NODE) @@ -2206,48 +2090,14 @@ static struct folio *alloc_buddy_hugetlb_folio(struct hstate *h, return folio; } -static struct folio *__alloc_fresh_hugetlb_folio(struct hstate *h, - gfp_t gfp_mask, int nid, nodemask_t *nmask, - nodemask_t *node_alloc_noretry) -{ - struct folio *folio; - bool retry = false; - -retry: - if (hstate_is_gigantic(h)) - folio = alloc_gigantic_folio(h, gfp_mask, nid, nmask); - else - folio = alloc_buddy_hugetlb_folio(h, gfp_mask, - nid, nmask, node_alloc_noretry); - if (!folio) - return NULL; - - if (hstate_is_gigantic(h)) { - if (!prep_compound_gigantic_folio(folio, huge_page_order(h))) { - /* - * Rare failure to convert pages to compound page. - * Free pages and try again - ONCE! - */ - free_gigantic_folio(folio, huge_page_order(h)); - if (!retry) { - retry = true; - goto retry; - } - return NULL; - } - } - - return folio; -} - static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h, gfp_t gfp_mask, int nid, nodemask_t *nmask, nodemask_t *node_alloc_noretry) { struct folio *folio; - folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, - node_alloc_noretry); + folio = hstate_is_gigantic(h) ? alloc_gigantic_folio(h, gfp_mask, nid, nmask) : + alloc_buddy_hugetlb_folio(h, gfp_mask, nid, nmask, node_alloc_noretry); if (folio) init_new_hugetlb_folio(h, folio); return folio; @@ -2265,7 +2115,8 @@ static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, { struct folio *folio; - folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, NULL); + folio = hstate_is_gigantic(h) ? alloc_gigantic_folio(h, gfp_mask, nid, nmask) : + alloc_buddy_hugetlb_folio(h, gfp_mask, nid, nmask, NULL); if (!folio) return NULL; @@ -2549,9 +2400,8 @@ struct folio *alloc_buddy_hugetlb_folio_with_mpol(struct hstate *h, nid = huge_node(vma, addr, gfp_mask, &mpol, &nodemask); if (mpol_is_preferred_many(mpol)) { - gfp_t gfp = gfp_mask | __GFP_NOWARN; + gfp_t gfp = gfp_mask & ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); - gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); folio = alloc_surplus_hugetlb_folio(h, gfp, nid, nodemask); /* Fallback to all nodes if page==NULL */ @@ -3333,6 +3183,7 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio, for (pfn = head_pfn + start_page_number; pfn < end_pfn; pfn++) { struct page *page = pfn_to_page(pfn); + __ClearPageReserved(folio_page(folio, pfn - head_pfn)); __init_single_page(page, pfn, zone, nid); prep_compound_tail((struct page *)folio, pfn - head_pfn); ret = page_ref_freeze(page, 1); @@ -3949,21 +3800,16 @@ static long demote_free_hugetlb_folios(struct hstate *src, struct hstate *dst, continue; list_del(&folio->lru); - /* - * Use destroy_compound_hugetlb_folio_for_demote for all huge page - * sizes as it will not ref count folios. - */ - destroy_compound_hugetlb_folio_for_demote(folio, huge_page_order(src)); + + split_page_owner(&folio->page, huge_page_order(src), huge_page_order(dst)); + pgalloc_tag_split(&folio->page, 1 << huge_page_order(src)); for (i = 0; i < pages_per_huge_page(src); i += pages_per_huge_page(dst)) { struct page *page = folio_page(folio, i); - if (hstate_is_gigantic(dst)) - prep_compound_gigantic_folio_for_demote(page_folio(page), - dst->order); - else - prep_compound_page(page, dst->order); - set_page_private(page, 0); + page->mapping = NULL; + clear_compound_head(page); + prep_compound_page(page, dst->order); init_new_hugetlb_folio(dst, page_folio(page)); list_add(&page->lru, &dst_list);