From patchwork Wed Mar 20 18:02:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13598069 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E148C54E58 for ; Wed, 20 Mar 2024 18:05:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CCA986B009D; Wed, 20 Mar 2024 14:05:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B68A96B009F; Wed, 20 Mar 2024 14:05:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A1136B009D; Wed, 20 Mar 2024 14:05:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8103F6B009D for ; Wed, 20 Mar 2024 14:05:07 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4DFB7812FC for ; Wed, 20 Mar 2024 18:05:07 +0000 (UTC) X-FDA: 81918193854.29.5060E88 Received: from mail-ot1-f48.google.com (mail-ot1-f48.google.com [209.85.210.48]) by imf27.hostedemail.com (Postfix) with ESMTP id 6DCA540018 for ; Wed, 20 Mar 2024 18:05:05 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=FgH6kHVw; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf27.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.210.48 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710957905; a=rsa-sha256; cv=none; b=BVhCqAS4mQsv87SCMTsyBz4vxXLXd0IggfHGu/jDs6rwVWK/JOa6wMfXVsnbypDlQHeASS rwF5/gKO8vg4uJWkfMJzOa0ZZbTN3taL6smfPL92QR6Q6LCy8s/9F6jDcUddlde+kVr0Ih YxQxBvdbRDdJiBvlc1jzNYzPqHtb3Fc= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=FgH6kHVw; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf27.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.210.48 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710957905; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=E6EDIqJ3GpKNGgGvycU8kV/i4h8q8YkWCZkehwdTQBQ=; b=G37a+fYsoypvZt8OJZcZuJ87iWn4nn5Fi2SstvzDQa8mbfWO50WMK+2HkPCOHq8uoI2QQX AiPSi9N7p6ty+5ChYS3B6Lnp6jKNcXksu2f0H9iuDcZkQohqVGNaHlxouCabG3ZDe7WQr+ kP8snb3KTrp7/6VYHzrqD1JTqS8441Q= Received: by mail-ot1-f48.google.com with SMTP id 46e09a7af769-6e675db6fbaso37646a34.1 for ; Wed, 20 Mar 2024 11:05:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1710957904; x=1711562704; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=E6EDIqJ3GpKNGgGvycU8kV/i4h8q8YkWCZkehwdTQBQ=; b=FgH6kHVwVf66ytX/MOWz4n71OtSwBG68NtxHbBMEJvDfZ+qB2tHHbEO7IC7xrTwt02 /qxhrOSEBxcNqD8Hh/F4TPsZfE2ntyLwP+AoKgR900ZQvBQsfpaRx+4IrE3FNEgMikAX WRFV29etqhX1WlEsDxy0M7phfE4F6ErVZs/883pLrzbg4kVMcZWQXCO3YSde+TUEPlZ4 yKWLzdvynRZmibaF8y8LX2g3tCwSGxAWeOAwR0oBhiX4v+zjq8fQYlh5HrzTCoVHgC1X 5//XkyAhiwuL6mwQo+iqhojWR09FW5qYj3GFQrSf6OttwWurPffbu1pP49v4339ouIXL NnaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710957904; x=1711562704; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=E6EDIqJ3GpKNGgGvycU8kV/i4h8q8YkWCZkehwdTQBQ=; b=mB6zZ7WtG5XVwgLsHxq8RBSc4c7fnHovd1qprh+sUb5iGwNnh9ZpLEFtiA7KG2TmC0 SR9ZST4P+b6wr4oFottUmsGrO5rjh3Jdtqp419Ody2H70KH54qADnpY1jWs9lkUY7FYm 3HoJbRwqt1MDX+L2UKviiAd0OVBPmSTHa3yvGUVojIUTICmQSXkexEzjV6orwbCEmHXh KcLnGt2rU257wanqea+/0rstsNrgquZWt1NuxwIyxKrDSTA7BMKdxmZYA4n//4N5R8tM ISbMGPa7+mwc+yqep8p3oe5rtTpV4iJ+DYNoUCeO5pn1dkstStX1sHiq6zIF0ECnSeYT xCIg== X-Forwarded-Encrypted: i=1; AJvYcCWlWB9wl2joZf/MCOnZ5M+tkCRCrEWmYW8S5xkpZMkvCU8oiU2Vtyj3mq0z6CCqKT1EFh4cu/DWWEwA189OgbJ59wk= X-Gm-Message-State: AOJu0Yw1FfdJ9fPUVemw89zXiWAGSiwlYQQ7LKL7be4TRP9k46nPXYMg 793afIuggKw8x19hppnV7V6rFucH+Kx1eR/FxdjbvP1WaJEYe5TV4GLa5fR7tg8= X-Google-Smtp-Source: AGHT+IFWSemKRWWruL0QHrLgb/7FgdcOFthYCoNJGsppv1Ywv9Zl8jdZobhkbMs/1qULPD7k8iTkhQ== X-Received: by 2002:a05:6830:1548:b0:6e6:84b4:5f35 with SMTP id l8-20020a056830154800b006e684b45f35mr6856475otp.8.1710957904362; Wed, 20 Mar 2024 11:05:04 -0700 (PDT) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id gf15-20020a056214250f00b006912014b98dsm8137529qvb.129.2024.03.20.11.05.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Mar 2024 11:05:03 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 09/10] mm: page_isolation: prepare for hygienic freelists Date: Wed, 20 Mar 2024 14:02:14 -0400 Message-ID: <20240320180429.678181-10-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240320180429.678181-1-hannes@cmpxchg.org> References: <20240320180429.678181-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 6DCA540018 X-Stat-Signature: a1e6e4wdehdmnn4ypj7qfeqtx5itm7k4 X-HE-Tag: 1710957905-735827 X-HE-Meta: U2FsdGVkX19v0foEoSmKjDvpfNPYXisr154i/cnsCfv25HaNsTpzO598ULeeYbWzWw5QgkoE/wbOkOqS1Sy+aWlLinR6taTNfN14yOfR5C+zwP9Mj6zNTmejOkqCDyaFBqrCBrHNxrRUU3RVZxbvSHsWgTTq+++HugMmbeL0F+g/wqCCY18cs5lee+ljkcqGxtYfAdtAOzvq+Ga8xxJLTXMxNGLp+ZmTpwHperksRhNJLPaS063zzd50WMN+wbEhSx/DO5VZfwLYxguqaq89XKvD+YfbL8tf2Xaa/q2vVrPFvlVFA2EmwD+ZssYAyA4AlL6AzfoZX+2p8XSD8BqsDwE9vTBNk5+BAyRvew1SvLMs62VMUVXkMTgLQuCqhQJNDGaaJI3rLI8NsHTY4qsvC3r9ro9JEE7Duswr1Wk/iCWmtELH5VnPMucVzefxuP9D+cMFkcvFixNr0tJD1njCVq4LC8PPc46Y51BUVh4zsgaSfdPkJdB5KqRPH7ZKFo5AdIjTZaIfYiU+2tIE01m3KyGlJciKlI/VcKcNvburBvnmaPU9N54dkyuM9CdVXjeplghFoX4JHaBI8iRX4UNhKPYORUW5KggpJtlY6SZn7RZ9at5f+ij/z2mbej8XvRYwLzaQumCKPrCxZTyxQKOzzIikm6JvjdiyjqaeSOsHJQlMbndiJ9Hnj0KvO75FqCsof158C2GbTjQc+KIyqp3Bua0xw2E0vnX1yAIyVp7bDSaRpJy8jnqj16zFs8I0ZYzQKoSLQZExhrUSf9sFqz+ztMm41ggHUyUB5hsuoF9r2dbb70Yw9nguxFeP8TlyaU1VFCWt0iNVWy2z6f82z3EdgpPXZj7ZnF65tIiHqYnduB6pMDjxTljyUBWXvAkwt8UYwdvSFjQfrodMSMk8+7WXtWHoqkW2CT3n5J0ivIYAno/biYj3AFiJCJegLDBrA9loxi6H/SwH64PXTW/LtE6 oTTaAli4 I1EfJPaCpHvA+FjRXPGj2gopGPne+BU4UjdmfrkPeGdtdSSCLfEaooXaK/njsiKvRGKSUedhfkCyPcJ5R/EfKkdHMA4xqFt9Qcc5cZcmKSWLmDeUZIu4etnb8NfyyK7Vy6s0pnzc6JKJqya4lwBWA+0XvhIUEHi+2VlFZoE2gTUH7G/dmk+lhabdYFUzAloAL9uzahuicgU1eH24faMZsP7BQCvUMVfvsr06PzwvWPO+xlgEcm0aG2gFxhQrxOsO6/5of+ivLmjiZxG7TfTGjyXSloJZYksJn/PEbWpE9bYGM3Fu7CXJnp4fGfCAUp1vx4z2+EyPUoVDEOTEhjssd/KUrNnqmte2mWu/HBTljyQNfsY9z8oneRnkmNN2ijeb3potRxej9jky/so1SikN5AKMMYYFhXL/9AXScLakDmbpK6+SOB3hlLehi5WiLm4+/Nx8kqqTs1KKNAFziwITMGFQ/RAQEv2GbzrtDFrP4e8G0rT4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Page isolation currently sets MIGRATE_ISOLATE on a block, then drops zone->lock and scans the block for straddling buddies to split up. Because this happens non-atomically wrt the page allocator, it's possible for allocations to get a buddy whose first block is a regular pcp migratetype but whose tail is isolated. This means that in certain cases memory can still be allocated after isolation. It will also trigger the freelist type hygiene warnings in subsequent patches. start_isolate_page_range() isolate_single_pageblock() set_migratetype_isolate(tail) lock zone->lock move_freepages_block(tail) // nop set_pageblock_migratetype(tail) unlock zone->lock __rmqueue_smallest() del_page_from_freelist(head) expand(head, head_mt) WARN(head_mt != tail_mt) start_pfn = ALIGN_DOWN(MAX_ORDER_NR_PAGES) for (pfn = start_pfn, pfn < end_pfn) if (PageBuddy()) split_free_page(head) Introduce a variant of move_freepages_block() provided by the allocator specifically for page isolation; it moves free pages, converts the block, and handles the splitting of straddling buddies while holding zone->lock. The allocator knows that pageblocks and buddies are always naturally aligned, which means that buddies can only straddle blocks if they're actually >pageblock_order. This means the search-and-split part can be simplified compared to what page isolation used to do. Also tighten up the page isolation code around the expectations of which pages can be large, and how they are freed. Based on extensive discussions with and invaluable input from Zi Yan. Signed-off-by: Johannes Weiner Reviewed-by: Vlastimil Babka --- include/linux/page-isolation.h | 4 +- mm/internal.h | 4 - mm/page_alloc.c | 200 +++++++++++++++++++-------------- mm/page_isolation.c | 106 ++++++----------- 4 files changed, 151 insertions(+), 163 deletions(-) diff --git a/include/linux/page-isolation.h b/include/linux/page-isolation.h index 8550b3c91480..c16db0067090 100644 --- a/include/linux/page-isolation.h +++ b/include/linux/page-isolation.h @@ -34,7 +34,9 @@ static inline bool is_migrate_isolate(int migratetype) #define REPORT_FAILURE 0x2 void set_pageblock_migratetype(struct page *page, int migratetype); -int move_freepages_block(struct zone *zone, struct page *page, int migratetype); + +bool move_freepages_block_isolate(struct zone *zone, struct page *page, + int migratetype); int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, int migratetype, int flags, gfp_t gfp_flags); diff --git a/mm/internal.h b/mm/internal.h index f8b31234c130..d6e6c7d9f04e 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -559,10 +559,6 @@ extern void *memmap_alloc(phys_addr_t size, phys_addr_t align, void memmap_init_range(unsigned long, int, unsigned long, unsigned long, unsigned long, enum meminit_context, struct vmem_altmap *, int); - -int split_free_page(struct page *free_page, - unsigned int order, unsigned long split_pfn_offset); - #if defined CONFIG_COMPACTION || defined CONFIG_CMA /* diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d687f27d891f..efb2581ac142 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -832,64 +832,6 @@ static inline void __free_one_page(struct page *page, page_reporting_notify_free(order); } -/** - * split_free_page() -- split a free page at split_pfn_offset - * @free_page: the original free page - * @order: the order of the page - * @split_pfn_offset: split offset within the page - * - * Return -ENOENT if the free page is changed, otherwise 0 - * - * It is used when the free page crosses two pageblocks with different migratetypes - * at split_pfn_offset within the page. The split free page will be put into - * separate migratetype lists afterwards. Otherwise, the function achieves - * nothing. - */ -int split_free_page(struct page *free_page, - unsigned int order, unsigned long split_pfn_offset) -{ - struct zone *zone = page_zone(free_page); - unsigned long free_page_pfn = page_to_pfn(free_page); - unsigned long pfn; - unsigned long flags; - int free_page_order; - int mt; - int ret = 0; - - if (split_pfn_offset == 0) - return ret; - - spin_lock_irqsave(&zone->lock, flags); - - if (!PageBuddy(free_page) || buddy_order(free_page) != order) { - ret = -ENOENT; - goto out; - } - - mt = get_pfnblock_migratetype(free_page, free_page_pfn); - if (likely(!is_migrate_isolate(mt))) - __mod_zone_freepage_state(zone, -(1UL << order), mt); - - del_page_from_free_list(free_page, zone, order); - for (pfn = free_page_pfn; - pfn < free_page_pfn + (1UL << order);) { - int mt = get_pfnblock_migratetype(pfn_to_page(pfn), pfn); - - free_page_order = min_t(unsigned int, - pfn ? __ffs(pfn) : order, - __fls(split_pfn_offset)); - __free_one_page(pfn_to_page(pfn), pfn, zone, free_page_order, - mt, FPI_NONE); - pfn += 1UL << free_page_order; - split_pfn_offset -= (1UL << free_page_order); - /* we have done the first part, now switch to second part */ - if (split_pfn_offset == 0) - split_pfn_offset = (1UL << order) - (pfn - free_page_pfn); - } -out: - spin_unlock_irqrestore(&zone->lock, flags); - return ret; -} /* * A bad page could be due to a number of fields. Instead of multiple branches, * try and check multiple fields with one check. The caller must do a detailed @@ -1669,8 +1611,8 @@ static bool prep_move_freepages_block(struct zone *zone, struct page *page, return true; } -int move_freepages_block(struct zone *zone, struct page *page, - int migratetype) +static int move_freepages_block(struct zone *zone, struct page *page, + int migratetype) { unsigned long start_pfn, end_pfn; @@ -1681,6 +1623,119 @@ int move_freepages_block(struct zone *zone, struct page *page, return move_freepages(zone, start_pfn, end_pfn, migratetype); } +#ifdef CONFIG_MEMORY_ISOLATION +/* Look for a buddy that straddles start_pfn */ +static unsigned long find_large_buddy(unsigned long start_pfn) +{ + int order = 0; + struct page *page; + unsigned long pfn = start_pfn; + + while (!PageBuddy(page = pfn_to_page(pfn))) { + /* Nothing found */ + if (++order > MAX_PAGE_ORDER) + return start_pfn; + pfn &= ~0UL << order; + } + + /* + * Found a preceding buddy, but does it straddle? + */ + if (pfn + (1 << buddy_order(page)) > start_pfn) + return pfn; + + /* Nothing found */ + return start_pfn; +} + +/* Split a multi-block free page into its individual pageblocks */ +static void split_large_buddy(struct zone *zone, struct page *page, + unsigned long pfn, int order) +{ + unsigned long end_pfn = pfn + (1 << order); + + VM_WARN_ON_ONCE(order <= pageblock_order); + VM_WARN_ON_ONCE(pfn & (pageblock_nr_pages - 1)); + + /* Caller removed page from freelist, buddy info cleared! */ + VM_WARN_ON_ONCE(PageBuddy(page)); + + while (pfn != end_pfn) { + int mt = get_pfnblock_migratetype(page, pfn); + + __free_one_page(page, pfn, zone, pageblock_order, mt, FPI_NONE); + pfn += pageblock_nr_pages; + page = pfn_to_page(pfn); + } +} + +/** + * move_freepages_block_isolate - move free pages in block for page isolation + * @zone: the zone + * @page: the pageblock page + * @migratetype: migratetype to set on the pageblock + * + * This is similar to move_freepages_block(), but handles the special + * case encountered in page isolation, where the block of interest + * might be part of a larger buddy spanning multiple pageblocks. + * + * Unlike the regular page allocator path, which moves pages while + * stealing buddies off the freelist, page isolation is interested in + * arbitrary pfn ranges that may have overlapping buddies on both ends. + * + * This function handles that. Straddling buddies are split into + * individual pageblocks. Only the block of interest is moved. + * + * Returns %true if pages could be moved, %false otherwise. + */ +bool move_freepages_block_isolate(struct zone *zone, struct page *page, + int migratetype) +{ + unsigned long start_pfn, end_pfn, pfn; + int nr_moved, mt; + + if (!prep_move_freepages_block(zone, page, &start_pfn, &end_pfn, + NULL, NULL)) + return false; + + /* We're a tail block in a larger buddy */ + pfn = find_large_buddy(start_pfn); + if (pfn != start_pfn) { + struct page *buddy = pfn_to_page(pfn); + int order = buddy_order(buddy); + int mt = get_pfnblock_migratetype(buddy, pfn); + + if (!is_migrate_isolate(mt)) + __mod_zone_freepage_state(zone, -(1UL << order), mt); + del_page_from_free_list(buddy, zone, order); + set_pageblock_migratetype(page, migratetype); + split_large_buddy(zone, buddy, pfn, order); + return true; + } + + /* We're the starting block of a larger buddy */ + if (PageBuddy(page) && buddy_order(page) > pageblock_order) { + int mt = get_pfnblock_migratetype(page, pfn); + int order = buddy_order(page); + + if (!is_migrate_isolate(mt)) + __mod_zone_freepage_state(zone, -(1UL << order), mt); + del_page_from_free_list(page, zone, order); + set_pageblock_migratetype(page, migratetype); + split_large_buddy(zone, page, pfn, order); + return true; + } + + mt = get_pfnblock_migratetype(page, start_pfn); + nr_moved = move_freepages(zone, start_pfn, end_pfn, migratetype); + if (!is_migrate_isolate(mt)) + __mod_zone_freepage_state(zone, -nr_moved, mt); + else if (!is_migrate_isolate(migratetype)) + __mod_zone_freepage_state(zone, nr_moved, migratetype); + return true; +} +#endif /* CONFIG_MEMORY_ISOLATION */ + static void change_pageblock_range(struct page *pageblock_page, int start_order, int migratetype) { @@ -6390,7 +6445,6 @@ int alloc_contig_range(unsigned long start, unsigned long end, unsigned migratetype, gfp_t gfp_mask) { unsigned long outer_start, outer_end; - int order; int ret = 0; struct compact_control cc = { @@ -6463,29 +6517,7 @@ int alloc_contig_range(unsigned long start, unsigned long end, * We don't have to hold zone->lock here because the pages are * isolated thus they won't get removed from buddy. */ - - order = 0; - outer_start = start; - while (!PageBuddy(pfn_to_page(outer_start))) { - if (++order > MAX_PAGE_ORDER) { - outer_start = start; - break; - } - outer_start &= ~0UL << order; - } - - if (outer_start != start) { - order = buddy_order(pfn_to_page(outer_start)); - - /* - * outer_start page could be small order buddy page and - * it doesn't include start page. Adjust outer_start - * in this case to report failed page properly - * on tracepoint in test_pages_isolated() - */ - if (outer_start + (1UL << order) <= start) - outer_start = start; - } + outer_start = find_large_buddy(start); /* Make sure the range is really isolated. */ if (test_pages_isolated(outer_start, end, 0)) { diff --git a/mm/page_isolation.c b/mm/page_isolation.c index f84f0981b2df..042937d5abe4 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -178,16 +178,10 @@ static int set_migratetype_isolate(struct page *page, int migratetype, int isol_ unmovable = has_unmovable_pages(check_unmovable_start, check_unmovable_end, migratetype, isol_flags); if (!unmovable) { - int nr_pages; - int mt = get_pageblock_migratetype(page); - - nr_pages = move_freepages_block(zone, page, MIGRATE_ISOLATE); - /* Block spans zone boundaries? */ - if (nr_pages == -1) { + if (!move_freepages_block_isolate(zone, page, MIGRATE_ISOLATE)) { spin_unlock_irqrestore(&zone->lock, flags); return -EBUSY; } - __mod_zone_freepage_state(zone, -nr_pages, mt); zone->nr_isolate_pageblock++; spin_unlock_irqrestore(&zone->lock, flags); return 0; @@ -254,13 +248,11 @@ static void unset_migratetype_isolate(struct page *page, int migratetype) * allocation. */ if (!isolated_page) { - int nr_pages = move_freepages_block(zone, page, migratetype); /* * Isolating this block already succeeded, so this * should not fail on zone boundaries. */ - WARN_ON_ONCE(nr_pages == -1); - __mod_zone_freepage_state(zone, nr_pages, migratetype); + WARN_ON_ONCE(!move_freepages_block_isolate(zone, page, migratetype)); } else { set_pageblock_migratetype(page, migratetype); __putback_isolated_page(page, order, migratetype); @@ -374,26 +366,29 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, VM_BUG_ON(!page); pfn = page_to_pfn(page); - /* - * start_pfn is MAX_ORDER_NR_PAGES aligned, if there is any - * free pages in [start_pfn, boundary_pfn), its head page will - * always be in the range. - */ + if (PageBuddy(page)) { int order = buddy_order(page); - if (pfn + (1UL << order) > boundary_pfn) { - /* free page changed before split, check it again */ - if (split_free_page(page, order, boundary_pfn - pfn)) - continue; - } + /* move_freepages_block_isolate() handled this */ + VM_WARN_ON_ONCE(pfn + (1 << order) > boundary_pfn); pfn += 1UL << order; continue; } + /* - * migrate compound pages then let the free page handling code - * above do the rest. If migration is not possible, just fail. + * If a compound page is straddling our block, attempt + * to migrate it out of the way. + * + * We don't have to worry about this creating a large + * free page that straddles into our block: gigantic + * pages are freed as order-0 chunks, and LRU pages + * (currently) do not exceed pageblock_order. + * + * The block of interest has already been marked + * MIGRATE_ISOLATE above, so when migration is done it + * will free its pages onto the correct freelists. */ if (PageCompound(page)) { struct page *head = compound_head(page); @@ -404,16 +399,10 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, pfn = head_pfn + nr_pages; continue; } + #if defined CONFIG_COMPACTION || defined CONFIG_CMA - /* - * hugetlb, lru compound (THP), and movable compound pages - * can be migrated. Otherwise, fail the isolation. - */ - if (PageHuge(page) || PageLRU(page) || __PageMovable(page)) { - int order; - unsigned long outer_pfn; + if (PageHuge(page)) { int page_mt = get_pageblock_migratetype(page); - bool isolate_page = !is_migrate_isolate_page(page); struct compact_control cc = { .nr_migratepages = 0, .order = -1, @@ -426,56 +415,25 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, }; INIT_LIST_HEAD(&cc.migratepages); - /* - * XXX: mark the page as MIGRATE_ISOLATE so that - * no one else can grab the freed page after migration. - * Ideally, the page should be freed as two separate - * pages to be added into separate migratetype free - * lists. - */ - if (isolate_page) { - ret = set_migratetype_isolate(page, page_mt, - flags, head_pfn, head_pfn + nr_pages); - if (ret) - goto failed; - } - ret = __alloc_contig_migrate_range(&cc, head_pfn, head_pfn + nr_pages, page_mt); - - /* - * restore the page's migratetype so that it can - * be split into separate migratetype free lists - * later. - */ - if (isolate_page) - unset_migratetype_isolate(page, page_mt); - if (ret) goto failed; - /* - * reset pfn to the head of the free page, so - * that the free page handling code above can split - * the free page to the right migratetype list. - * - * head_pfn is not used here as a hugetlb page order - * can be bigger than MAX_PAGE_ORDER, but after it is - * freed, the free page order is not. Use pfn within - * the range to find the head of the free page. - */ - order = 0; - outer_pfn = pfn; - while (!PageBuddy(pfn_to_page(outer_pfn))) { - /* stop if we cannot find the free page */ - if (++order > MAX_PAGE_ORDER) - goto failed; - outer_pfn &= ~0UL << order; - } - pfn = outer_pfn; + pfn = head_pfn + nr_pages; continue; - } else + } + + /* + * These pages are movable too, but they're + * not expected to exceed pageblock_order. + * + * Let us know when they do, so we can add + * proper free and split handling for them. + */ + VM_WARN_ON_ONCE_PAGE(PageLRU(page), page); + VM_WARN_ON_ONCE_PAGE(__PageMovable(page), page); #endif - goto failed; + goto failed; } pfn++;