From patchwork Wed Mar 20 18:02:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13598061 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3DA0C54E58 for ; Wed, 20 Mar 2024 18:04:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 464996B0087; Wed, 20 Mar 2024 14:04:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 377326B0088; Wed, 20 Mar 2024 14:04:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1A3A06B0089; Wed, 20 Mar 2024 14:04:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 079C66B0087 for ; Wed, 20 Mar 2024 14:04:56 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 902DD1212F8 for ; Wed, 20 Mar 2024 18:04:55 +0000 (UTC) X-FDA: 81918193350.24.391D32E Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) by imf11.hostedemail.com (Postfix) with ESMTP id A052040028 for ; Wed, 20 Mar 2024 18:04:53 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=cq2lqv7o; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf11.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.176 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710957893; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FfocRRC1YB9gbTpQRA0qsUF6TQAVW4ud00P9WTspsIs=; b=vIeWX2LYojiROFSR5AX6tBbnqs0qAk9kpE+E8qGkZkAtYX3pIIO44XaaQNRTdRam04D5TW V6549B68NjKfQK+2MDG1SgkstWHfVLzDu/KEcP5UrZWPDFQxQR3DQIWt7FuMdfuuvoOBwl t4RNoIpb/fjT8FoGBy1HTUhgHAvmsew= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=cq2lqv7o; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf11.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.176 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710957893; a=rsa-sha256; cv=none; b=ni8oUE4Ye1Xm0CdDu57yVegrTTGJuVF2i2IV1d00rcEI+ezoclDRdmqVef4D2r3lcUBOWy dciZq1ulJdOZCzSH1uqhJLDYVEUN8/gqtzNT4VUtaSOyw2cpsj027Z0xOTNYBVN9T+8MmQ ByjmRpxeCwLKJezkszlEeEg2qwvkR+4= Received: by mail-qt1-f176.google.com with SMTP id d75a77b69052e-42e323a2e39so1642131cf.1 for ; Wed, 20 Mar 2024 11:04:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1710957892; x=1711562692; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FfocRRC1YB9gbTpQRA0qsUF6TQAVW4ud00P9WTspsIs=; b=cq2lqv7o1Wli+xcII+ZWCjaxDNsrrH+qBpYBz0wkAXOWl3g3CePdgnXpHJx+INEC5A 4cDGuzfAMksnFpo1OFqM/B0eflwPCryu9DdQMl7ZDmNL2p7B4ZjZWhcXeguY1zWrWlvZ WGlc3nAN03hC2h5utMkY39C9MwZTXUNybl+Y29h7nnQssaQtCl+zxc1bCVLVhdHIrJn9 umxh5s/NtTKYa5GAGgqv5R1aGp8i4xAMAnx+qKeciHFHfGEhskNZtp9zPi16Ma43p6JC bxZ0qRRypvH1mGtt6k2P8vMctjD0cVYTnKNDtbLVooeZEFbDdcg5VRGDCBRhiD+VJvnf Ynow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710957892; x=1711562692; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FfocRRC1YB9gbTpQRA0qsUF6TQAVW4ud00P9WTspsIs=; b=bS3gje8fs8Gp1I9IkRkrxvvkJxGjbCB30kgdynrOAWxQKVNhg2/GFhwG1PbJp9W/Ub 6I1Sxg3tc6tv5EOBiloAOIeBJOAXHHx9mYUwt2/45gMEDV8RHs+wChdTYpfFeXwqQApW 3qNv1Vt28IKbxY4PYvf2Vzf+5cN3vgRGJRR++4xLDGR4c47Ld3LL3EIUegicbTB2qvcu v1OP4YaTd+rTOZ9bcjvvmd7ZhHocNiaKL2eCvdAxmHCOMWSFdK/jeqLWPhonnJypBaw9 QxUaLwR9NubdcjCTuI4ep0qgZaXil5c/U7Atg4Leuu3feKVmVD8osTmy1gbNqIgkEhiY F24A== X-Forwarded-Encrypted: i=1; AJvYcCWW3r5cUt/v76qINZET6sMirj32hEG/ZTUmqhS9kpFqSXMSi0PjIeEnoISXMV/UuR4g6D/3xD5Scqb50me1op+z1EA= X-Gm-Message-State: AOJu0Yx93FIh/N06AlxRnSo0XtVCJDaO1kBHbVC5FTpkhj9RfAfg4dRg zGcULCv9RS1z/3CYA960MAd4mPieBElM+MjHgdHOrm8PJaHZXEc455PfLi9F72I= X-Google-Smtp-Source: AGHT+IHoamNYVBdLmp9p84PZmX7dks/ROfSNPw4orYipVjbhoiZ9qdiu/sQyu3TIfPVvxmMBsK0o2g== X-Received: by 2002:a05:622a:188a:b0:431:155e:7ef0 with SMTP id v10-20020a05622a188a00b00431155e7ef0mr750683qtc.6.1710957892574; Wed, 20 Mar 2024 11:04:52 -0700 (PDT) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id ex9-20020a05622a518900b00430bcec5432sm5506623qtb.85.2024.03.20.11.04.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Mar 2024 11:04:52 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 01/10] mm: page_alloc: remove pcppage migratetype caching Date: Wed, 20 Mar 2024 14:02:06 -0400 Message-ID: <20240320180429.678181-2-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240320180429.678181-1-hannes@cmpxchg.org> References: <20240320180429.678181-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: A052040028 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: zyquzfq9y8cuwnc1mkxzfnes3aojko7p X-HE-Tag: 1710957893-306925 X-HE-Meta: U2FsdGVkX19qkiaw8vkdc947lCY3RjepEgMZghqmZL4kwnZhUjNIsKu/LXK1+1vaCWVep06kioEkQ2uxD+f9qP3LlC1Hbc5ejX02jjOGYYj8kwlxWebCV3BFpKAKxk0UzTwufHam/9jyPK1i1t+na6gO4zEIyFqfNapRuMVcv0V3D7QuuUl5yXUy5hzXRROC1AztN2qkO2JofmeYSzRtIf6O33zGmldxRnEQ5KX3REzhnyeBPzulrZcQqupb1Qlkpz1Y/kZWAF6rnIqSGsJ2oCfTSSs+gUL41ens5McIfirZT+lxZ9AmgCK+Fy4/1UrY1/nKoc9zoGrPPAD1kuVMydEuR5v6BBl/MwYgtnPA3z5deI6kNt4moA3WWiEl1wVNKxm8m63Ulegp6GfzaWNGuyVgNackZgmfCH42eLN4SRm77Dmj64m1v/XayKVLx05Fj/ab++nTGaOUrNIojafUJIWSOXdqDCBaBg5gYGszvAG/x+3PJFiv51o/zgfSL3xgM6Mxmb5kFfSqK29kglFC1panpaGWZ6SXDvZ8pTn8W3W0AER6pRMfTd5+6dmYiCJTeKimfrJWYsEM79zQsDr8xNMmxeZUVJ4eryZwxm7Op+vYsK4P5MJFZTMPE4k79J8QeeL/toLl7QEUdpuhvaOqCi+OHXwcHU3Ve8mEVR1zPhn8JIYK09+QouVRiMOzfg0jksXZzoMLlwcBzLgZVxfbwKTrpd1bhN8ns5H77QBxkntaiEqXeMLW4KIbuT2o1Je2x18PiG9n2OY0Af+Ga+gPFXNGVvV4D4rDHHfvzVAf08jRVilmqr+Bm2ohLPTCGsuNEVohIGGVYFR2r1Z6pNrdLc6pxuWMr1A4ZO7HWEuddliUYgkgxT+M3QznqxcpaEQtSJg6QoDKg50oMD2Sda+eFSm/UMsSjnXUCv1l8TFvAdhLC2FQGbaQBZrbV3oYtRcvRLsKkPplrSBya5P1VPO z7Az8TxC 3EXAFJd7O56dZTiu1iFX8V8/9hmciwAsCvgrpwUWYlRbidgv2/j0i7TV51roxnTS87BaA1jP2fpXKcjy9MsGSWKyUCBFZByiYt/jRsaQcY+QK9V+iXQRmIxCTblJtyndIFr1HyuZLC639IGed1EEkXwod3tVBJmMWiXuhZWl4/EY6NeLP/A8ZuKpMKFa3ggUYwsrXNA4UElv4CW/hClTq7HIXVN0RsrKO35BUcXE2a46EOJE/7uv2fxCtOS/GtBlotWVKWwTx5OWn2+bZlLqmr5ZiikgW2qHIcBEEKpI2MMlDto1PvqKE7LNVS49EVTzkhoTdUm1tQs76COK4Irbk6FCDsdfCyQ/NxE48zjQJeLMAnhBULmCCrm+44bz3DojlLMrZiyt5wD8GYOEAH74iCLLhlj0TR4k8JOshDAYYFONk2GHyOcl/xD1q8iZ6JmOBRNphV2oHowqagJR8TjD+3NgH2L16ohF0FUj6Tu45jtrubo4jT5OHkFt9ilR57P82LVXjLIrwRRXLGgGjaXy++kztBxTyeUWhnlP0XILbWYe7jkkdLpHtZLTKaE0NfLIRk1EnbIEzH2R36WowOlSVl0BzRtb9W7F60EQw X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The idea behind the cache is to save get_pageblock_migratetype() lookups during bulk freeing. A microbenchmark suggests this isn't helping, though. The pcp migratetype can get stale, which means that bulk freeing has an extra branch to check if the pageblock was isolated while on the pcp. While the variance overlaps, the cache write and the branch seem to make this a net negative. The following test allocates and frees batches of 10,000 pages (~3x the pcp high marks to trigger flushing): Before: 8,668.48 msec task-clock # 99.735 CPUs utilized ( +- 2.90% ) 19 context-switches # 4.341 /sec ( +- 3.24% ) 0 cpu-migrations # 0.000 /sec 17,440 page-faults # 3.984 K/sec ( +- 2.90% ) 41,758,692,473 cycles # 9.541 GHz ( +- 2.90% ) 126,201,294,231 instructions # 5.98 insn per cycle ( +- 2.90% ) 25,348,098,335 branches # 5.791 G/sec ( +- 2.90% ) 33,436,921 branch-misses # 0.26% of all branches ( +- 2.90% ) 0.0869148 +- 0.0000302 seconds time elapsed ( +- 0.03% ) After: 8,444.81 msec task-clock # 99.726 CPUs utilized ( +- 2.90% ) 22 context-switches # 5.160 /sec ( +- 3.23% ) 0 cpu-migrations # 0.000 /sec 17,443 page-faults # 4.091 K/sec ( +- 2.90% ) 40,616,738,355 cycles # 9.527 GHz ( +- 2.90% ) 126,383,351,792 instructions # 6.16 insn per cycle ( +- 2.90% ) 25,224,985,153 branches # 5.917 G/sec ( +- 2.90% ) 32,236,793 branch-misses # 0.25% of all branches ( +- 2.90% ) 0.0846799 +- 0.0000412 seconds time elapsed ( +- 0.05% ) A side effect is that this also ensures that pages whose pageblock gets stolen while on the pcplist end up on the right freelist and we don't perform potentially type-incompatible buddy merges (or skip merges when we shouldn't), which is likely beneficial to long-term fragmentation management, although the effects would be harder to measure. Settle for simpler and faster code as justification here. v2: - remove erroneous leftover VM_BUG_ON in pcp bulk freeing (Mike) Acked-by: Zi Yan Reviewed-by: Vlastimil Babka Acked-by: Mel Gorman Tested-by: "Huang, Ying" Signed-off-by: Johannes Weiner --- mm/page_alloc.c | 66 +++++++++++-------------------------------------- 1 file changed, 14 insertions(+), 52 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 4491d0240bc6..60a632b7c9f6 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -206,24 +206,6 @@ EXPORT_SYMBOL(node_states); gfp_t gfp_allowed_mask __read_mostly = GFP_BOOT_MASK; -/* - * A cached value of the page's pageblock's migratetype, used when the page is - * put on a pcplist. Used to avoid the pageblock migratetype lookup when - * freeing from pcplists in most cases, at the cost of possibly becoming stale. - * Also the migratetype set in the page does not necessarily match the pcplist - * index, e.g. page might have MIGRATE_CMA set but be on a pcplist with any - * other index - this ensures that it will be put on the correct CMA freelist. - */ -static inline int get_pcppage_migratetype(struct page *page) -{ - return page->index; -} - -static inline void set_pcppage_migratetype(struct page *page, int migratetype) -{ - page->index = migratetype; -} - #ifdef CONFIG_HUGETLB_PAGE_SIZE_VARIABLE unsigned int pageblock_order __read_mostly; #endif @@ -1191,7 +1173,6 @@ static void free_pcppages_bulk(struct zone *zone, int count, { unsigned long flags; unsigned int order; - bool isolated_pageblocks; struct page *page; /* @@ -1204,7 +1185,6 @@ static void free_pcppages_bulk(struct zone *zone, int count, pindex = pindex - 1; spin_lock_irqsave(&zone->lock, flags); - isolated_pageblocks = has_isolate_pageblock(zone); while (count > 0) { struct list_head *list; @@ -1220,23 +1200,19 @@ static void free_pcppages_bulk(struct zone *zone, int count, order = pindex_to_order(pindex); nr_pages = 1 << order; do { + unsigned long pfn; int mt; page = list_last_entry(list, struct page, pcp_list); - mt = get_pcppage_migratetype(page); + pfn = page_to_pfn(page); + mt = get_pfnblock_migratetype(page, pfn); /* must delete to avoid corrupting pcp list */ list_del(&page->pcp_list); count -= nr_pages; pcp->count -= nr_pages; - /* MIGRATE_ISOLATE page should not go to pcplists */ - VM_BUG_ON_PAGE(is_migrate_isolate(mt), page); - /* Pageblock could have been isolated meanwhile */ - if (unlikely(isolated_pageblocks)) - mt = get_pageblock_migratetype(page); - - __free_one_page(page, page_to_pfn(page), zone, order, mt, FPI_NONE); + __free_one_page(page, pfn, zone, order, mt, FPI_NONE); trace_mm_page_pcpu_drain(page, order, mt); } while (count > 0 && !list_empty(list)); } @@ -1575,7 +1551,6 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order, continue; del_page_from_free_list(page, zone, current_order); expand(zone, page, order, current_order, migratetype); - set_pcppage_migratetype(page, migratetype); trace_mm_page_alloc_zone_locked(page, order, migratetype, pcp_allowed_order(order) && migratetype < MIGRATE_PCPTYPES); @@ -2182,7 +2157,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, * pages are ordered properly. */ list_add_tail(&page->pcp_list, list); - if (is_migrate_cma(get_pcppage_migratetype(page))) + if (is_migrate_cma(get_pageblock_migratetype(page))) __mod_zone_page_state(zone, NR_FREE_CMA_PAGES, -(1 << order)); } @@ -2378,19 +2353,6 @@ void drain_all_pages(struct zone *zone) __drain_all_pages(zone, false); } -static bool free_unref_page_prepare(struct page *page, unsigned long pfn, - unsigned int order) -{ - int migratetype; - - if (!free_pages_prepare(page, order)) - return false; - - migratetype = get_pfnblock_migratetype(page, pfn); - set_pcppage_migratetype(page, migratetype); - return true; -} - static int nr_pcp_free(struct per_cpu_pages *pcp, int batch, int high, bool free_high) { int min_nr_free, max_nr_free; @@ -2523,7 +2485,7 @@ void free_unref_page(struct page *page, unsigned int order) unsigned long pfn = page_to_pfn(page); int migratetype, pcpmigratetype; - if (!free_unref_page_prepare(page, pfn, order)) + if (!free_pages_prepare(page, order)) return; /* @@ -2533,7 +2495,7 @@ void free_unref_page(struct page *page, unsigned int order) * get those areas back if necessary. Otherwise, we may have to free * excessively into the page allocator */ - migratetype = pcpmigratetype = get_pcppage_migratetype(page); + migratetype = pcpmigratetype = get_pfnblock_migratetype(page, pfn); if (unlikely(migratetype >= MIGRATE_PCPTYPES)) { if (unlikely(is_migrate_isolate(migratetype))) { free_one_page(page_zone(page), page, pfn, order, migratetype, FPI_NONE); @@ -2572,14 +2534,14 @@ void free_unref_folios(struct folio_batch *folios) if (order > 0 && folio_test_large_rmappable(folio)) folio_undo_large_rmappable(folio); - if (!free_unref_page_prepare(&folio->page, pfn, order)) + if (!free_pages_prepare(&folio->page, order)) continue; /* * Free isolated folios and orders not handled on the PCP * directly to the allocator, see comment in free_unref_page. */ - migratetype = get_pcppage_migratetype(&folio->page); + migratetype = get_pfnblock_migratetype(&folio->page, pfn); if (!pcp_allowed_order(order) || is_migrate_isolate(migratetype)) { free_one_page(folio_zone(folio), &folio->page, pfn, @@ -2596,10 +2558,11 @@ void free_unref_folios(struct folio_batch *folios) for (i = 0; i < folios->nr; i++) { struct folio *folio = folios->folios[i]; struct zone *zone = folio_zone(folio); + unsigned long pfn = folio_pfn(folio); unsigned int order = (unsigned long)folio->private; folio->private = NULL; - migratetype = get_pcppage_migratetype(&folio->page); + migratetype = get_pfnblock_migratetype(&folio->page, pfn); /* Different zone requires a different pcp lock */ if (zone != locked_zone) { @@ -2616,9 +2579,8 @@ void free_unref_folios(struct folio_batch *folios) pcp = pcp_spin_trylock(zone->per_cpu_pageset); if (unlikely(!pcp)) { pcp_trylock_finish(UP_flags); - free_one_page(zone, &folio->page, - folio_pfn(folio), order, - migratetype, FPI_NONE); + free_one_page(zone, &folio->page, pfn, + order, migratetype, FPI_NONE); locked_zone = NULL; continue; } @@ -2787,7 +2749,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, } } __mod_zone_freepage_state(zone, -(1 << order), - get_pcppage_migratetype(page)); + get_pageblock_migratetype(page)); spin_unlock_irqrestore(&zone->lock, flags); } while (check_new_pages(page, order)); From patchwork Wed Mar 20 18:02:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13598062 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D183C6FD1F for ; Wed, 20 Mar 2024 18:04:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2908E6B0088; Wed, 20 Mar 2024 14:04:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F1DD6B0089; Wed, 20 Mar 2024 14:04:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F36EB6B008A; Wed, 20 Mar 2024 14:04:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E0B396B0088 for ; Wed, 20 Mar 2024 14:04:56 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9F819C039F for ; Wed, 20 Mar 2024 18:04:56 +0000 (UTC) X-FDA: 81918193392.07.F721262 Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) by imf24.hostedemail.com (Postfix) with ESMTP id D259E180008 for ; Wed, 20 Mar 2024 18:04:54 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=2lYvnaC0; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf24.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.175 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710957894; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HCOZaDSQS3uSogr2y2Y4MtomH1kr/Fi4b+AnkoSHYG0=; b=HapHiLN9DEcIYDYRuUXfTkxaEW2AmRK8PtLAHTYh0SEJcDwSgTmJmIFJmrdh0pH5JmcKR3 BaK4F/gD8GOyzFXoKCw9fwIICksjm/sKZji9tPNFpVRK+FfArAZpOF5Mfb0EcJlGwc1mpP wNW2ar5bzCpZ5zlaawiB6uC6tCc8f3w= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=2lYvnaC0; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf24.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.175 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710957894; a=rsa-sha256; cv=none; b=WfCuyEObqDvfjsoxG/g7tJUsCoA6quLnESSkAULUEAJMu+VSQPtzIJ5bPreNnA0uBCYA6c XV/UQAqjfp1jH2p1yqJ23Ni4lBRdYVVCjWpUbmyoWTxh1Ji+7e5A7pOGuXyL8ik+5IndcF 0XnP/O7Wq/0eK1AFOImctWFnbBrRei0= Received: by mail-qt1-f175.google.com with SMTP id d75a77b69052e-430dcd64e58so1105761cf.0 for ; Wed, 20 Mar 2024 11:04:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1710957894; x=1711562694; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HCOZaDSQS3uSogr2y2Y4MtomH1kr/Fi4b+AnkoSHYG0=; b=2lYvnaC0fqwyb2XLtX6o+qf3Vtuuec/llyGMsb930l8vfHc/1PbAZRz7LntbWv4WN6 x0xxaSrK2P21w/x37ky/2r8/tcX6ktTQl/JjxfrzkPhaOgoyNJ2XeltU4UjQ3Um6kHNx IWxpTIEuz49OVB7uf/SS4QWveDQRksUg459OSER/Mbw2CAt5gkZhkph9H+gb7HPF2BBp pirGoIIbWK+y8hFc44j4B1CGgkCW9FcwE7C+TaxL0P/VYL3Czacs8fVB0xPnJb2AKDto Hwq2B+ICYY6M0gUoT9NLItKem49QH6ClY0TzVd7M1q0t3oX3YkxE1GV/6Nc6aiWZ+Rbg JICA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710957894; x=1711562694; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HCOZaDSQS3uSogr2y2Y4MtomH1kr/Fi4b+AnkoSHYG0=; b=HsYT7dvg9+FwAtddpAW7PE5OnpV/Y5M97Hj2UlRgsVljm3/ESC8uKJSqRx11Q/x0+C PLMtzxl/Y5K3kO23Z1r/o/sUxrobNvtSeEWrqGRii1sJv/RnKyNx5bINgxkGaNewI3ZA HCGaea1i/QpHHNimpA0dHY99B6AZNitd2eL78e/U6KJbmD3t4095SLKpz+puoFTZLjrF V/vymWY7j1FU5reSNidjybi+FJ5gt3ZP2NIzLOmwJCWUtGe+YEFMlzSEwqkeJShSl94L cDycvg46L8ZCUND33BV0HEqK0h3iuNv/aFKQ+cREB+yN8Fe3XPT509f2D2vR1Ufd7Yv4 LhYw== X-Forwarded-Encrypted: i=1; AJvYcCVjLkWkOHf6zC+aBDMGfdbSa2HX/4R1WNMtrJSUv3ptgIwAgUf5jCG9KvJhipmhgaHfoKLuFDzLsyW4pV+d9aR8ynk= X-Gm-Message-State: AOJu0YwQ2AHVurf2m07ZhOiCXuq7DAgr9/CuiSzQpNSUTucW7H8lIB3A CZVfXtMiXCOnbVRhLFpBZp+WMtbfvYMUukKgZSzSWSF7eNRg+jgaE++oN9i+arE= X-Google-Smtp-Source: AGHT+IF5UfXc5lTV+1+ZbEqDKnB2S/2dMfIITfLAmk8SgkXLoh8oySBupjmVBPzQZ9ttFuOzueZSzQ== X-Received: by 2002:a05:622a:54:b0:42f:20f2:c4cf with SMTP id y20-20020a05622a005400b0042f20f2c4cfmr2979645qtw.31.1710957894003; Wed, 20 Mar 2024 11:04:54 -0700 (PDT) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id y17-20020a05622a005100b00430d8e11bebsm3231779qtw.64.2024.03.20.11.04.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Mar 2024 11:04:53 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 02/10] mm: page_alloc: optimize free_unref_folios() Date: Wed, 20 Mar 2024 14:02:07 -0400 Message-ID: <20240320180429.678181-3-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240320180429.678181-1-hannes@cmpxchg.org> References: <20240320180429.678181-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: D259E180008 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: w19k1b3ck9zr1qhn3srqf8jphkc4q63z X-HE-Tag: 1710957894-478711 X-HE-Meta: U2FsdGVkX18RIUmrQJGsUaXwQr523hu6LHQqbzabhcsmTPhCPxM0/ZDHylEy8L6djfRXWw36gA6rlAQIeE+sIXN4FnXkMvAzknBpxqle0cgr8LQhgRbtWntu1lkSmsa2ptQ45GHMQtCn7lNpG7eZ+achtyn3BkqXB0xQKxiFeDYFJIM3lfbE4JOSP+7Xe26jjgM8lc0YUbRHv0J86lAZF6oh5uCQpcwlVHiBuIqNBV1AWW1pgTOy/CqR1yQ/Nj9X+B6iHnKyni0XBJJlGkkBXlo3F4PNkkSHLdVcGy87tJYnEBi1oYTeJl3iSmIFNMn5NL+V6IEpHgGkc3Tnkad9e1Yy9snr2sYjCCQshOGhhngdP5WlZoR9xDqskHjNRdYNZ9jPvhFHBbRoiD88bs8dXmjz35X+OmVybRWdn2zRuE0Y0rZlOSGtrRrmTCIW7nwEjT8r6nIPI7W2MryBkB6uWzGYTt/fZ8oWpI/khr/+W3aU/wdKQxBGjIhRdS0PqB8msJS2AujnKC28pPs2oCcv3XKNgSk91mdcz2/JOOzB7JuwnRP28A2FG6AsNhwQ/kNwxBs+WcJzE8PxNmESUuBN8cXn7Mwei6LEuopbjGJOf+u6ikAnXYFuc5iwVcnNGDBRYDTiEYmMlytNBMLND9Spy2GfFr/ebXhWSs8qI3+WDH25P7pCk1YiZ8iYgGcbxpXhsktN021+8Kr/Sl4lD7/sR1qzbWjN5JqHQISRfL54nZ5622vtlcTIPgrtQSCXBA8Rv8r2IdpNvX9/Fgd3zz67To27RplTk5k0lKoocuppI0mK+e/QENSckdIS5Fifv8StS8Dq7OsrlrG1Y8G5BeGjZ2VqEBboMlLApw0edwuKzmEposzQbMk7cLdf5mIBy+mGIpOj5uvP78a9l/lea8yO8MRj+PuXIBI85O77tI3CnYU0GjAn7w89zMep4ZvRz57VEz6pXwjWozPuhVP8web aFC4vrQv thXvQSFgIInHixJ/ZqoKzOzHghKLT8JJbY6xwB4socI7dTFqRPC0qirvbgTFTg11rGfRTK4S1A4NKcUvLCd8ZklddrnALC8JMVHNZIZ9EWOPETbiihhm0OPj3fm2qtWfQ3BD/WsM3PgyVfpdRZjAUEX72NciCCCD3GsecSCtni6HAwL2d+q78wfjYFxLPbZ/g8qCZlgxkoftDxPwZoDezwvHkyG9C8I+6DW2Ru55IsZa8Y3TQujkSU0b0PB8ijPILrz5ptVZy+5tFLBc0Xl4+4yN3p5L3w8zi1YqQi3KdB65b9sQs7wGfhXzjkKpnyURWcTcSzEh++zjNYwh09NHjWHt62ZlDGxtg4Ob7sMlnlwj1SRQNoHmICkpFTeeh/o0M6ljDmg8+v1t91571w86bS73rHfdGMWG/WxI06wUc24YhYpnLCFh5eMEXkN6gIcoOQW+L8rsJv0Wjy9t4dWtm0exvk1Wcff2pyaWhUwNqartY3R4i20ALrBJDh0tK8K/UgRUIAxbgZW9BUh4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move direct freeing of isolated pages to the lock-breaking block in the second loop. This saves an unnecessary migratetype reassessment. Minor comment and local variable scoping cleanups. Suggested-by: Vlastimil Babka Tested-by: "Huang, Ying" Signed-off-by: Johannes Weiner Reviewed-by: Vlastimil Babka --- mm/page_alloc.c | 32 +++++++++++++++++++++++--------- 1 file changed, 23 insertions(+), 9 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 60a632b7c9f6..994e4f790e92 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2524,7 +2524,7 @@ void free_unref_folios(struct folio_batch *folios) unsigned long __maybe_unused UP_flags; struct per_cpu_pages *pcp = NULL; struct zone *locked_zone = NULL; - int i, j, migratetype; + int i, j; /* Prepare folios for freeing */ for (i = 0, j = 0; i < folios->nr; i++) { @@ -2536,14 +2536,15 @@ void free_unref_folios(struct folio_batch *folios) folio_undo_large_rmappable(folio); if (!free_pages_prepare(&folio->page, order)) continue; - /* - * Free isolated folios and orders not handled on the PCP - * directly to the allocator, see comment in free_unref_page. + * Free orders not handled on the PCP directly to the + * allocator. */ - migratetype = get_pfnblock_migratetype(&folio->page, pfn); - if (!pcp_allowed_order(order) || - is_migrate_isolate(migratetype)) { + if (!pcp_allowed_order(order)) { + int migratetype; + + migratetype = get_pfnblock_migratetype(&folio->page, + pfn); free_one_page(folio_zone(folio), &folio->page, pfn, order, migratetype, FPI_NONE); continue; @@ -2560,15 +2561,29 @@ void free_unref_folios(struct folio_batch *folios) struct zone *zone = folio_zone(folio); unsigned long pfn = folio_pfn(folio); unsigned int order = (unsigned long)folio->private; + int migratetype; folio->private = NULL; migratetype = get_pfnblock_migratetype(&folio->page, pfn); /* Different zone requires a different pcp lock */ - if (zone != locked_zone) { + if (zone != locked_zone || + is_migrate_isolate(migratetype)) { if (pcp) { pcp_spin_unlock(pcp); pcp_trylock_finish(UP_flags); + locked_zone = NULL; + pcp = NULL; + } + + /* + * Free isolated pages directly to the + * allocator, see comment in free_unref_page. + */ + if (is_migrate_isolate(migratetype)) { + free_one_page(zone, &folio->page, pfn, + order, migratetype, FPI_NONE); + continue; } /* @@ -2581,7 +2596,6 @@ void free_unref_folios(struct folio_batch *folios) pcp_trylock_finish(UP_flags); free_one_page(zone, &folio->page, pfn, order, migratetype, FPI_NONE); - locked_zone = NULL; continue; } locked_zone = zone; From patchwork Wed Mar 20 18:02:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13598063 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 349EBC6FD1F for ; Wed, 20 Mar 2024 18:05:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DD8BD6B008A; Wed, 20 Mar 2024 14:04:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D62386B008C; Wed, 20 Mar 2024 14:04:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B90136B0093; Wed, 20 Mar 2024 14:04:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9F2CD6B008A for ; Wed, 20 Mar 2024 14:04:58 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4FC89120320 for ; Wed, 20 Mar 2024 18:04:58 +0000 (UTC) X-FDA: 81918193476.18.F7F1BE3 Received: from mail-ot1-f48.google.com (mail-ot1-f48.google.com [209.85.210.48]) by imf17.hostedemail.com (Postfix) with ESMTP id 638B640016 for ; Wed, 20 Mar 2024 18:04:56 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=Uowj1WeR; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf17.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.210.48 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710957896; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=T7r0X9mNHsEvKFNd3QjZ02+rYxlZ2ijHgMC19/7k7mQ=; b=rEQjsrQBW1RmZwmEyVaVK3/VoPDn1Iirf1NlF4pElX+2QXaTwefjvTQaULGdjNcdpZRb4b tx4xJpU3/WSjdYp4P5phrAYCNreJg0YRz0U7iSGvwpTYtlncsHe0ZTdhuFT6JpLkp/azeY mfyL0UKxBo2GP47p9hyI+ne5Yghj2Mo= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=Uowj1WeR; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf17.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.210.48 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710957896; a=rsa-sha256; cv=none; b=YDcJ4Q0fg4RdjwBSzA0Ty6av+uxgTMENYcim3AdXoULuRq7MTs9qxMPPeRI3IuGfC7wfuN kJMiu8Dqkugy/KW7Cy5U3YuEjzaWTcSwJAI9n8VqClVEsnv6Kv9FV4YhDsmGtQ+s7LPQrl gnt762IS7+Enh2UG0oGuAoU1eaE+ReE= Received: by mail-ot1-f48.google.com with SMTP id 46e09a7af769-6dc8b280155so49627a34.0 for ; Wed, 20 Mar 2024 11:04:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1710957895; x=1711562695; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=T7r0X9mNHsEvKFNd3QjZ02+rYxlZ2ijHgMC19/7k7mQ=; b=Uowj1WeR3XfZ0EIFx9/OTSpAnIg5vISCYyckl+5aW75IhIfIfU2yOQiTAztoAkuANH bsVS/74uX2Onjn4m1hMnAwAoasRDJ/z3IFUuuunMxICX4vfIphYLV9thSXQMNQbuqNVq WBnggfwVqLWHgGpfW5Qw0i0lgb+S7khpmGnT1GUCiLbIXm5FSPipJ1EvwvCo/6xMT44+ 6Pqia8jv50TP5xxjd6DeUIaEzTuM6xH3PEUeY4qucYRMVw2CAoGhhAVZZrfhAg+CoGHC 878CYNGMrlC3Eg64nQVYT6aazMShZ9zCiC2S/M63W0aRcASgxOHeoNND7RYTkoxvfYHP Q3ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710957895; x=1711562695; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=T7r0X9mNHsEvKFNd3QjZ02+rYxlZ2ijHgMC19/7k7mQ=; b=B6xbYgAMptSM5pLkae8l2jsHI8dNX0HhFSSikKNmTkiu+ntL3lJzXZu5ZnuJpkf/Mx nC7nQ9t/XqXmdrgeipIK8XxOc8uhxwoo5q5UvKnqPYv68OdeDUn40H4T6m5f2/BdR/Rn 3fkGUGC8bwUJtZFQ0Md7eHQ0eDc6AX7v3r+EzeEiBg7K8iCY4JLD82K27atcILqb7Kiv FE5iX4q30lq+1R2sUaSImKzCsyyGQNXTCP/Ln5NPu33wVpZS6xGG5tR17I3nTM1uT0H4 KRHv8dNxm6UBr0u14RcTtKUuUbHWQlaMzF0x6JRa1l3wXZeEUamPSDfgHGnQ64cAWNFd RkCw== X-Forwarded-Encrypted: i=1; AJvYcCXod7EEIVE63eRpvXdGRGLHFU7NKqmnkSfW2sCtxf4eJqIRLyRFhDd4fSAqf/EKn9HlGdvuXeh+Dztef5lho5Pd1o4= X-Gm-Message-State: AOJu0YxTdD36wy0Z9XNlxQXurneOJ/U7XYaBanY076LTcHMQNe7TQ9de mEi336YPfh/f9zLC8f93dgpb2WfWfpKlp+3Mn4elaLrjBKJcGR8lHw/h+ykOBVc= X-Google-Smtp-Source: AGHT+IEjrZQWghTRcTpqpNffkDCKSOVahuvg3qnwvR8ayQXYPdrYk9RXL0q0Eb9YG2QM0kUquwP9xQ== X-Received: by 2002:a05:6808:130d:b0:3c3:8339:6ef6 with SMTP id y13-20020a056808130d00b003c383396ef6mr3008346oiv.45.1710957895377; Wed, 20 Mar 2024 11:04:55 -0700 (PDT) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id fg14-20020a05622a580e00b00430bddc75a5sm5411969qtb.23.2024.03.20.11.04.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Mar 2024 11:04:55 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 03/10] mm: page_alloc: fix up block types when merging compatible blocks Date: Wed, 20 Mar 2024 14:02:08 -0400 Message-ID: <20240320180429.678181-4-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240320180429.678181-1-hannes@cmpxchg.org> References: <20240320180429.678181-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 638B640016 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: mataxcm9nc1thdesheo3qdib8jyqz7xz X-HE-Tag: 1710957896-322565 X-HE-Meta: U2FsdGVkX1/ORuFJiadvEpULZTk0yce3etOtIJvbI/ErRRI2JnS1JqbEdSDDTckvR7JZKYtZuMxfP4zhoT9W/rlXQT8q05BXLggLlq9W6F3tDLhNH5gdlxCqylqgTxsDWvpehqEutARRxgE5fgnfUR/1mjjSpBjDO0YmxcHNJZvRsXMZRT+kuWyBudbwevZR5bKOenVnuuR/8scnjdfpBFOo+fjjmivSbMk33GBj6EYX2LViyCqA5fH4kUUGclOgbpOwdmDSGTHrzghVhRYqy3znO5dEFs3wqju5W1Qj93+0oOaV0IohMg8UKGCm4lhtBZCojlSAmmiQJSsOG8EGQAyySUypd7a96OUKDC7IDU50wRPp/C89Ckp4GdN0A5hlXAx6HWJTWUt9+rNFphcat5Jq6Xz++zCX7nsFVn025vAHe+bBGnYRuXt+K33qMYAnsoAd9AfNHHYVSkH30FHpDKCPwDreMA8NmMvBoCk7Xo2B7P4/cuF5LEdP7jl0eFM6uxM4fQBi8nxeMlOrr9tghUkX+RoLDUcnTSpcj+5vTdQ5A8XROkfz6YeKMs2rrWUwcSgo1IVPDQBPWAnn4AUlSGOJFfRNwJy8t3VfNUIDvL6fij+Oe+eAmxiaqExycvM3A2FIaSPmHxIk+yULI3xV9/R2d5+PhBkC6WVJoWKVqjiW3b+NWMDdEOPiNmtJ1Y+VgqDwcxbnEIiOAltswmHXyT4C7Nk+29JCGs/qEtmWvgmyoTFzygD6PMw9pxrUtqoVSqG9tXGScDEC/efSqt7Kv33wAsUJQFYN1YM3Uo3TeehWdrI29Esb112vNrWLFyKQhIDbtP1ZBK61fX6MtT/WW3fuevqlUaASSrBCJpp9oULChuF7hly7zM5+b9+7CIwprV+D+qkiz3uxtZv8eetANxe7wuqq8PQAmco/mthUVcRfAo1pifQ4kcZGVWacagr8fPP7ZNUheAxVts5zIGc yAfyIl8W nKvrf2MYvIWcDPcY6Yg9NyEbsWbTChxAtfLrkoi8iOpYW69+blKWCVDkmXm1sx0x80jmUQFoemzr/+orKjnEy+68JP7RPw1p+ICgBwUqfTrUTCD+FN27NwlHtU8A1c55/Ge+W7ESEKDKMP6LCRkc1bBv1L8owIXUUxVKcL4JOQ3EGbjpHnVxSEfEo92GrlUgwK840Cs4ATEBNXOKbLD7sl+9jBxhUjD83PmxlzfQi6QYOu2uaDdWOqXYVHKF4VUbKio8iXJnmkI/LC1+LyC/4fkP556WB+8UJvDRJmDuyANTCRxQe5gPGoPDJG7MysosLwIDp9ndUT4jF/5Up8ZHBU6cs55SxHmCBi/hr0pP/YQSCVQZm4L6CRLzuK8MopWVjtoVM952kksOqMWGoUF8/4IDgGY5wsjTIWnMoLlMhqkqNv2bfovTMLlLerO6h7iUgdvmnWqiLc0cs3RRLnvZL+NEXeoIQmy0alj1tdRzvfT/PdGwmxsTTNjxLQBoKcSrmma2rKppVyP+28TzCopb2H0xSmZToIwjmQkbNpRyNlW6UWR5valnYLTm6oUM2wJC7nGxlWdcMkzqfSxpPa4F8v/2k2w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The buddy allocator coalesces compatible blocks during freeing, but it doesn't update the types of the subblocks to match. When an allocation later breaks the chunk down again, its pieces will be put on freelists of the wrong type. This encourages incompatible page mixing (ask for one type, get another), and thus long-term fragmentation. Update the subblocks when merging a larger chunk, such that a later expand() will maintain freelist type hygiene. v2: - remove spurious change_pageblock_range() move (Zi Yan) Reviewed-by: Zi Yan Reviewed-by: Vlastimil Babka Acked-by: Mel Gorman Tested-by: "Huang, Ying" Signed-off-by: Johannes Weiner --- mm/page_alloc.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 994e4f790e92..4529893d9f04 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -785,10 +785,17 @@ static inline void __free_one_page(struct page *page, */ int buddy_mt = get_pfnblock_migratetype(buddy, buddy_pfn); - if (migratetype != buddy_mt - && (!migratetype_is_mergeable(migratetype) || - !migratetype_is_mergeable(buddy_mt))) - goto done_merging; + if (migratetype != buddy_mt) { + if (!migratetype_is_mergeable(migratetype) || + !migratetype_is_mergeable(buddy_mt)) + goto done_merging; + /* + * Match buddy type. This ensures that + * an expand() down the line puts the + * sub-blocks on the right freelists. + */ + set_pageblock_migratetype(buddy, migratetype); + } } /* From patchwork Wed Mar 20 18:02:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13598064 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 163D8C6FD1F for ; Wed, 20 Mar 2024 18:05:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 48C4B6B008C; Wed, 20 Mar 2024 14:05:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3ED9F6B0093; Wed, 20 Mar 2024 14:05:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 218626B0096; Wed, 20 Mar 2024 14:05:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 08EA16B008C for ; Wed, 20 Mar 2024 14:05:00 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D8A5912010F for ; Wed, 20 Mar 2024 18:04:59 +0000 (UTC) X-FDA: 81918193518.24.B395D9A Received: from mail-qv1-f49.google.com (mail-qv1-f49.google.com [209.85.219.49]) by imf13.hostedemail.com (Postfix) with ESMTP id D2D802001C for ; Wed, 20 Mar 2024 18:04:57 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=UHTqbJ9j; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf13.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.49 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710957897; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=30OLZu1BB8NMv+SdqAZeeZ3/L8O0lM0O1+2bSt3RnLI=; b=ChR8v8HHwQ6+omzAdILMZP68L47hh8LxbDiGz4O25EOTQ/zvTOU91NhAbMyUp7IYiqlQ9z WfnxI8VgRXvjT7IudaBgUVS7tfkB8kPiDGF7/Y8cTW08TIyWPvw7z5msbf3aggt738d1zV nOjc9a+NUyPBQJq5g0Tk9BY0ndMYkFc= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=UHTqbJ9j; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf13.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.49 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710957897; a=rsa-sha256; cv=none; b=V9VGADl3ye7+BXAL7NhOL9iS7FdT9MC5LEtjxgnuaIIJfbluKWE961KQQC9+o4OBADIJmi UERlYGKuc3Y64QHHdCuSyKcYgKze9poZZQRhtLTwx2L885hV7UUxjm6poNi716RDBdNNYo Yi2D6fQUAmAnVjxJPAOmV87J6jU067A= Received: by mail-qv1-f49.google.com with SMTP id 6a1803df08f44-69145fc4265so1104236d6.1 for ; Wed, 20 Mar 2024 11:04:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1710957897; x=1711562697; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=30OLZu1BB8NMv+SdqAZeeZ3/L8O0lM0O1+2bSt3RnLI=; b=UHTqbJ9j7cWib7AYpk3Y++K9+FTOEcTmqk7J9MTjJBB1KN9u+tJgWYaHA/nZR3pBFw s0XZmgNs9JOj8ZuY85w6wH1wqVBlc+XlKsRQYb5VAdL4LAml03EtKhlvu8VnwTyJLAZT y7ifGukaEc2OfjD/2LGtDGnemd/URwuoupZK1WYNoiTkWGgCGwVeWucPZ1HCv485H1kC NqPtKywFNRcxi1eVy8M2fbLczcTHZt5PPEAo5PB0xAOsS1Dd10u5J6iIATfxT41l11Y+ Yw2ay0goTeAPgWBkSqc98pfC4ytFFoxLhVVhLf46hrsCUGYfrKrVLiEBwTXGdhhqp7K2 fzAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710957897; x=1711562697; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=30OLZu1BB8NMv+SdqAZeeZ3/L8O0lM0O1+2bSt3RnLI=; b=IpDJGdUwFoeVl6n7i/cwJUvLLJnn/itAA98k4WUSwhDUirX0M/m/d84YTLssGa3yzx XFMZv0mKp4PVBMxtEG9u6TqJBWdqvgN62Gu9utTpXXxXhalP/W5gJAs1mxipE+ws2RtW LM1rG37vWJ/O3r+j8OtzuIPTGRJWK00ko0kg3MvRceW5cXPDi2Xpm53pZ34Xg+u264Xk 6GoImIyeiWlJtvO+2dgsc4IGKuqAC5iN3/AZVGnxLc/G6DclUZOc8AcGqAvqKgx54ccV heJ91rCTETKF3lbMzEj/r0rGJIOgwY4W/fSUk4SCG1hiWtKBBff5gXVTzHpf9R1XmltR q3RA== X-Forwarded-Encrypted: i=1; AJvYcCVFdLSwDh/yWuYRP4rJX5al0eQU8cqRfrEa6/B6bV8Qp4SGhSkHdjuYRglxLq42qFOEM+utsrDdFr4tE+yo1m2Fm2A= X-Gm-Message-State: AOJu0Yywu/6Kxfhc+insFQuUt5RQSWOeAuuvssi+kH20VSv297h1yypi Y+5eM+Pj67p4cu0yg7m7qGpFxqzduhDLysM64hWmjZinelkTKB8unDJRy62dWGY= X-Google-Smtp-Source: AGHT+IG5sxaRbIJ5O7l1kuKbDMw7LMa+PdnsbFz8dadgYPyg3jWsjDbKYtsMujG8RnvxMdWfiNyJxw== X-Received: by 2002:a0c:f64f:0:b0:696:442c:a659 with SMTP id s15-20020a0cf64f000000b00696442ca659mr3116723qvm.52.1710957896840; Wed, 20 Mar 2024 11:04:56 -0700 (PDT) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id 15-20020a0562140dcf00b00690cbd296fesm8183030qvt.121.2024.03.20.11.04.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Mar 2024 11:04:56 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 04/10] mm: page_alloc: move free pages when converting block during isolation Date: Wed, 20 Mar 2024 14:02:09 -0400 Message-ID: <20240320180429.678181-5-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240320180429.678181-1-hannes@cmpxchg.org> References: <20240320180429.678181-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: D2D802001C X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 9d11q7nws4ybtmu4ibdiod37wj1yqyte X-HE-Tag: 1710957897-53943 X-HE-Meta: U2FsdGVkX1+qfcApUjN+8Cvl0gef2QMvxeg/CMn0WnNfE/PC21EtqDr2HcKdGnxIHF8xMr3P6gl+SZIVGCzC9dCLJyu7153ltTCJJcrlp25LWacUZgkN5cFfB0IvGDczF8pYuI8ZA9WsCBHARAwf7xtkw7PyJ/lCenbBVfUeA86o9bwCYDMTcHgG0TYab55ZTVfvhoBMFd567Bb6Bi6XPu0Dr/jtpbjAA772/NO14MkNJT/Lcc83h6PYpToR2qKUr7E+YeJxoDkX4PnTYd4q1luokQgZhJQ5mo7bF3q3F40Vd9lwnkztdKvv2Gsdy8iTAqdJcomQq6h2ue2SOuVQl3T/IifLkFrWdVuKvPZsFAByja9L6ko95R+59cQ3gzwFX1Yxb/ZitsoujtALLATMowSPlKj1D46OwOQtBXRZmUPJFKZgsczoHx2zYszQkMoPNFhtGRQ0R7KErS7eNwCqMuD47k8EBh8GiObe63/50e5dkUsI5YgXfwkke5Bv7dsOyI3sRtzwNQ7JEJYgNWTJs2aglbnYMByyJnzFMePSHLchJPscfKsEr9tecdj6oznrX81uzBR3ljoT1OozHOQesZfVKBmCBrdJfj4pfYcuFmQXy/atX+EXZ5F1fh01uLJ/5Pw3A0HIJhnWtd9HLfZDC2O+8zagysPOVt+UI6A53nr+vdnaTg/CIhQJl4SLQ4f6az+0UATJkuXyyJB80S/DZNHx2QvBun0UYONnswszZP+gdEk64hbpodRp/vy8rE41d3jZ6sj+CUN1tshcFGhEJUPRQq2s6KgZm8sVl2jFNvlYgGaPBsktoK9L3qEO55PIgNgLdm1C0M5Wk+eXT5KKLJKHlUmUBHRieyJ58goNDKjil0nfEAhgjxgV5J97kyaRpIxCmv2mU3bSJo5ncC9s6eq0HafnVQNFy5nbzt3jwBbu9NQNjzP+R+AAn3gK5fVGKaPXXkM5rpb03//W/x2 bbYVcvaW b0QRRtkOSRtamMpN+lJDddiZCZG6pQIxpg42uPXAU2Eiqm6EtGGor+fFxWwJMMTNvuTozCJvWzQ9kL9aIS7AhfUpMS0DFIxXvgQvdmcDgja3kQa6YXc6XYgLAE6U0Dspte8XQeoFi+zIsaUTmLPfdxsgOO1v7pRiFLBey29ukertFpNdoIdjxIfGlo7ecM5t7jnd8UfU3kolk1b0OfQ3MXCoFaghvb7xssoSnxnenMJFxZgWocOM5218m7geOsGQBbiUVrctSy/az7QUxRxmFbSauJCtxyI50ssAunYXHVpS3brwGV9GQFtHqMTJNd/wuZlehJyp3BAn1bt1Uym2VDeZnRRvm/og4YCEH5EI+GfBAo2r44Ac+/ItpERzSz9AOVlVwFWXgNsfowfNsyMPOk4R7Jd15qyYcDNysmveBR+asxYbwYAy4et8qwwcnl1flPIDn3kxuHcfOFqr203aRfTOtbpcD9Ed4+7UfbOmnouxoDAUpgsMu1L8bQaocTRnaj/F5kqBdCgcdrdCOGtK8tdoiBjN9CNMuaDYvBSOKiZGNKkN8ATfxaWeAfnDF3rLMkwup6xLfuYcLFY2w+/dxF1GMxg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When claiming a block during compaction isolation, move any remaining free pages to the correct freelists as well, instead of stranding them on the wrong list. Otherwise, this encourages incompatible page mixing down the line, and thus long-term fragmentation. Reviewed-by: Zi Yan Reviewed-by: Vlastimil Babka Acked-by: Mel Gorman Tested-by: "Huang, Ying" Signed-off-by: Johannes Weiner --- mm/page_alloc.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 4529893d9f04..a1376a6fe7e4 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2683,9 +2683,12 @@ int __isolate_free_page(struct page *page, unsigned int order) * Only change normal pageblocks (i.e., they can merge * with others) */ - if (migratetype_is_mergeable(mt)) + if (migratetype_is_mergeable(mt)) { set_pageblock_migratetype(page, MIGRATE_MOVABLE); + move_freepages_block(zone, page, + MIGRATE_MOVABLE, NULL); + } } } From patchwork Wed Mar 20 18:02:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13598065 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 395A1C54E58 for ; Wed, 20 Mar 2024 18:05:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D8AAF6B0096; Wed, 20 Mar 2024 14:05:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D14776B0098; Wed, 20 Mar 2024 14:05:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B44166B0099; Wed, 20 Mar 2024 14:05:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 947166B0096 for ; Wed, 20 Mar 2024 14:05:01 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 668B84027D for ; Wed, 20 Mar 2024 18:05:01 +0000 (UTC) X-FDA: 81918193602.30.0777093 Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171]) by imf08.hostedemail.com (Postfix) with ESMTP id 64C7B160023 for ; Wed, 20 Mar 2024 18:04:59 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=VJYqKXUM; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf08.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.171 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710957899; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OZwAKBxwvX8Cu8VK/08SCQc15oSmFIOnYtXDnl3jEr8=; b=troAhqngdUDHVaX5Sf4ZJnCEjAFQ0EtLNiyemGJDZDwzjxjxsGNdT9eKCMrpFmkwdOtWjg YI/HhTCvajYWXtIYqZt1Tdzj3Hc0PkH3sLvFpkkhAXeyJCrGRM1YjwppdYIf/XraglrF2P aPuSwaDJraMEwvB9Yaor2hIUk8nsIT4= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=VJYqKXUM; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf08.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.171 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710957899; a=rsa-sha256; cv=none; b=5ioIDeNEWNBWckNu3AwAH//Oweu3svvUBND2REIVissVe7aUW2H7X6gXY2huJ/rgVHtj8u Ee6ldt0nh1eB884dD/pQwqWhkS73RI7LKxE1XzdkKrdKhIyh5kXMAW2D24o7lDW4E3zZlj Ta2T6Y9vwFIl3wzdI1FUN/lttY6UfDg= Received: by mail-qk1-f171.google.com with SMTP id af79cd13be357-78a26803f1aso9764285a.3 for ; Wed, 20 Mar 2024 11:04:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1710957898; x=1711562698; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OZwAKBxwvX8Cu8VK/08SCQc15oSmFIOnYtXDnl3jEr8=; b=VJYqKXUM3Y1Hk4ZpbGt3VhJSICBmevAx1UTTAxme28v3djiHHu74wMjnMwBOIufQET EVuju1CEPAZ4QJBmnvoFcbhBAvMkokI9L/pWUAFhN2X8DJBrZo/4cPqlSB7XQz91GZgK tARtv/znttGhOKedp8DqgMQRfjScocJiquvMsU5biMp1d4u/M3TRXZjv2LgDFJhXrWb3 4G96IvUREPAesTp7kkeuiMxc1I53Yubiu1KChoKxFf0oCioTEG5W+KQWRXxrb1tLbZFH LL3iFeG0n8Si/stpXcEsHhgNWHPGOr8aIFPy+6f2/20s83gcQFGMYbhp1+BpEiveOXNB qpyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710957898; x=1711562698; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OZwAKBxwvX8Cu8VK/08SCQc15oSmFIOnYtXDnl3jEr8=; b=jgrpI8P7n1Nrul1IrHnhyeTqSP+qxF8neciy/0F6pA+dCFzjhjOeWIZ8Vej5o8Dfo4 0zBEyeVHbllDyq0eSWUsloBEGho4gUHrUvVVBxIhALPC10b33LpdZCUcRsCrDmRHDT+5 TqmTxMTL02lM0NnHCOZbrZcB0q/BSKBlSHhLVNGYQIVnG6qrmzjxGC9Zv2XXRaVsgGAV wq/IjKV942fDzWgNqx2OKe9OCWmNjXV8GELfXpspSUtdvCWZPKOdurzyKppFevL5wGdE pC5wpiSjQtmRmg4CUn62ARrAULYYV7Z9EQKev4SXnZxwUPPph+VI0w4b5EZESlrDJ+jw 33nw== X-Forwarded-Encrypted: i=1; AJvYcCWUs50OkuOzAKw5m11ZDOegKsDKkHlO/W48V3FyuhHADweEX++9VsZYOE4jK0NSCKnWz8f9hnOC2F0iIv187MIlzjA= X-Gm-Message-State: AOJu0YxoQSAjrWtBFh+IwBFpso+eTtJvE6wzgxpf0+DQ9KXvcTmNjj6s aTnHfZJ7RT7OYpWE5YOWXngVfytAVcOhGVu1dvF+IdA74zb7dMcF9evnRl1VW19M2EHKE1v5fYP q X-Google-Smtp-Source: AGHT+IHMMH7iyyjW9zOuIWSxf38EdGRhEKTJ2fArhuWu3NYukGvTXw1ChW+Vsv6SLoiTkEyfuwoNTQ== X-Received: by 2002:a05:620a:821b:b0:78a:47e:282d with SMTP id ow27-20020a05620a821b00b0078a047e282dmr8998670qkn.56.1710957898421; Wed, 20 Mar 2024 11:04:58 -0700 (PDT) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id y1-20020ae9f401000000b00789ed16d039sm4402405qkl.54.2024.03.20.11.04.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Mar 2024 11:04:57 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 05/10] mm: page_alloc: fix move_freepages_block() range error Date: Wed, 20 Mar 2024 14:02:10 -0400 Message-ID: <20240320180429.678181-6-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240320180429.678181-1-hannes@cmpxchg.org> References: <20240320180429.678181-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 64C7B160023 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: kx7yk5qxi19gboiunb74zceoi1wdip5i X-HE-Tag: 1710957899-861360 X-HE-Meta: U2FsdGVkX19CI5NBYVktsGDKpvryh/o3E9h+zHIySy8kucXE1aw3lFqF1TwHVOOxExRI7sxx/h8iwoX+wTkDyITdq7xa/PykFNZl1V8fal32nVI4SZZFV3zQKDJDbjJsWJw56v4ECIEk7Kg/L1FBFAUqPK8Hbxj41NSRs0MeBOj/fwS6gVXj9SuEOmrW19GohZnmQCA6L6dqVZagbfj/8m9TqAyJOqXvvP7m4nPibHua5r92kFL7Gcbvp8jW8KLETuIZbVykj1BPrwyp1cDml+o87LTOgpTmp57fI+U6vI2H42TFLV2f6hThSeTgnDZp3yeTjV0jpRvhpWsPiA5UBLelZPlRTHeMp1NvnPSdqvy/st18fdJEDh8p38rTwifriyZc3LbYAWdqmq+RVC/L4V+EqibcTZinueDxyRnt2VaRsq1La4qY4Q9nMnvYgVAGVIAvjwbqIclDje1610GYqyGs457oV4/Lmx79qdr6C9KFh8Lt2bghHn9f360H+wxx7j2elei2ObY1VHXyB1GFCjXwdUTZDLXA66vVERIE3aZ9bbGXm0e/hpie3qcOrQ15CSPrSBvpKysutYSRmWQVGhdRqwxgKY3xDsZKSdoxSBpclTt0hSsdS3Dq9VmR+3NkAWW/4TcGaWrTdP3F3r340jCUPpjJ/mgD4gq6z9HBvza4myshGwM+7/FtKsGnYnnosToFEs9ruOGuHU7M7/5BiioAt19kGBTJy8o1zqV30A65qdovmaGilafnNjup5c/eW5vUGS/QiXOMDTZmqvqDjIxPywynxG6JNB6Aqu4cF+zviUOaM4XTrsI6zFT7j3NXCZvgiXt632UPDx4hdhaZ+EQITx4LL//8CuZda3EiuYNCqm3Y4BY9n2ZM+MRZHDQEWhZF4L2jofTdTNSrzzcza8uHmOZ2AmqFGR1kz3Bkhq1mjuQIz1oS2biDO1T1ycrG7I2JnYVuL/n2FCd70f4 KgTaqxLm jZWgH9TCnYp5ETyNAD9FCwkDuYbjB50aAxATZjPqKcwedPQ1F7BKwPDvWCY+T47equXt4nCCZTzZ1Xe8mTktXJ+EuWYxE4dXoIKOQIetGCjoJmFQ5++Z4zkSLZaZEGtSKHvAEEVB6IeLJBJ8xbkKSwJulXn5iikWj4ZQF2+TqBY2Ue6DLlePRhljDhMp9S9rnu2ZujReX7cClqiCEjIsTdT0i0k2zGy2P2Y8/ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a block is partially outside the zone of the cursor page, the function cuts the range to the pivot page instead of the zone start. This can leave large parts of the block behind, which encourages incompatible page mixing down the line (ask for one type, get another), and thus long-term fragmentation. This triggers reliably on the first block in the DMA zone, whose start_pfn is 1. The block is stolen, but everything before the pivot page (which was often hundreds of pages) is left on the old list. Signed-off-by: Johannes Weiner Reviewed-by: Vlastimil Babka --- mm/page_alloc.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a1376a6fe7e4..7373329763e6 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1645,9 +1645,15 @@ int move_freepages_block(struct zone *zone, struct page *page, start_pfn = pageblock_start_pfn(pfn); end_pfn = pageblock_end_pfn(pfn) - 1; - /* Do not cross zone boundaries */ + /* + * The caller only has the lock for @zone, don't touch ranges + * that straddle into other zones. While we could move part of + * the range that's inside the zone, this call is usually + * accompanied by other operations such as migratetype updates + * which also should be locked. + */ if (!zone_spans_pfn(zone, start_pfn)) - start_pfn = pfn; + return 0; if (!zone_spans_pfn(zone, end_pfn)) return 0; From patchwork Wed Mar 20 18:02:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13598066 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88608C54E58 for ; Wed, 20 Mar 2024 18:05:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3099B6B0098; Wed, 20 Mar 2024 14:05:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 28F026B009B; Wed, 20 Mar 2024 14:05:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F37396B009A; Wed, 20 Mar 2024 14:05:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D726B6B0098 for ; Wed, 20 Mar 2024 14:05:02 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 94E1F80C8F for ; Wed, 20 Mar 2024 18:05:02 +0000 (UTC) X-FDA: 81918193644.04.6E58CBA Received: from mail-qk1-f175.google.com (mail-qk1-f175.google.com [209.85.222.175]) by imf30.hostedemail.com (Postfix) with ESMTP id D75298002A for ; Wed, 20 Mar 2024 18:05:00 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=dTrTGEwJ; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf30.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.175 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710957900; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iJbaxS39OzzydjfBWSZV5WP7n3+u6yMfmgUYLy2xips=; b=asBZ7ONBKxITb7iGoTpBWPxKe6mJF4W9a5o7f8gIAxYVairKumfkiwf3/0X4QE6fpDOesX 96UINxLp7+rFymOulDIDLi4q0le4qeJvxOc+y/PwpG7wl9qQNlGvjjXRe1vVnMNwMQ6WYf bKTITjBiIJMNbUI9yYNzMQkVNo9s/Xg= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=dTrTGEwJ; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf30.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.175 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710957900; a=rsa-sha256; cv=none; b=Ki2IGq9crLZsfDH9JlU9XfFX1IQUywh2f+IX/Diw7CCF0j+2EU/smwWN3JGN6O1TnOH8jH k8/kLM6nWbLVQF2kkDXCZauchwJ2iD/+dj2OFTXwFl1Dfh1VRYBtfn/x8gG9jK21TFRQXe yJ6D/xwItycquv2rsHU6HxDjI0fdLds= Received: by mail-qk1-f175.google.com with SMTP id af79cd13be357-781753f52afso8788885a.2 for ; Wed, 20 Mar 2024 11:05:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1710957900; x=1711562700; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iJbaxS39OzzydjfBWSZV5WP7n3+u6yMfmgUYLy2xips=; b=dTrTGEwJ9al2/Q1thm2OHME16U75Y06lPqk61E6YlTdjTkqF7eCFx81Xq8NVW30MeV ph2bmKPzbuNtCLTtLM6Wgrvcwv721pI5e98qwInWV2MUYa8zo3GEbBo2wg6FGq0DO/5m whMnLMaxIyvLZQ0NX+Ss0BUGLW93nvRvvRq4/VrhhKv4xgLB4fnJ4rX51HhFLuGHR3dA XY81p1R3mzwQylzExjkgeFhTeLdcZWIL1Dfe6Wxssk7H+TkDIDzFSEcyiSK1cDqPa95T Ltf2R3YXqdsA+SXxiQ0dm8vEaz7x1cIJL6xcLKZSmNZoqJpXU4thDP/eNi+sIc0VH+e+ sIpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710957900; x=1711562700; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iJbaxS39OzzydjfBWSZV5WP7n3+u6yMfmgUYLy2xips=; b=NEAnojuJrbPyQahnDdN2DV3KPwy9IBwuNaeY43egsYtULUQG3NMREMw3a1lPAhZLgO /k42DFj1HsmY4It8c1Is76amkDhk630hYiSVmpmG7jxOIfeHcjfmQYcKli901D5fgrsV qC+HEuxtNbwt7KRTI5MCA+tqBreMcfuDkShnQvYU5a9Gmq2E7I8JKoq2OXWctCPWbDN7 5udCc0U+DwshvZAaDxONlbtnz52czgYU4rEQzN8aaIJxxmrSNUUm8AFbkbHc979T6I9L 1pIZFiIiYSIy3Zs5zVsO74CyV/uRAaqOU57DXOO3o1W1qDQU/t9pxoDUBPNpZI3o4fOl uR7Q== X-Forwarded-Encrypted: i=1; AJvYcCUI6CtrNnZSSNjqmDcA0iZjaAeslnZwpd4Pyc+0AiMOZqAmTclQBw2RIqaWkLoPBbEkK6bd5sdJlE8uLYJtAzoSZJM= X-Gm-Message-State: AOJu0YwYkG10/+2AFtmzJQYJCjgPvcDDVSSlZBTsx7dsrHN4De7/ke3X oMjmQIIxfWBq5wtBG+gDuGpRvqLvTs+IClpDqaKSGtjs1oTn+u9OIwWO4Hyjk1Q= X-Google-Smtp-Source: AGHT+IHY92ihNoxl1b7Kg9oFTjtJdyFLlDJ0yqmrH20Oav015jwylu4HFY1NbgI8MfcFHM3ggIHTBw== X-Received: by 2002:a05:620a:5587:b0:789:f9b1:e2e3 with SMTP id vq7-20020a05620a558700b00789f9b1e2e3mr2849191qkn.20.1710957899960; Wed, 20 Mar 2024 11:04:59 -0700 (PDT) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id m6-20020ae9e706000000b0078a2b5faab7sm104240qka.50.2024.03.20.11.04.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Mar 2024 11:04:59 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 06/10] mm: page_alloc: fix freelist movement during block conversion Date: Wed, 20 Mar 2024 14:02:11 -0400 Message-ID: <20240320180429.678181-7-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240320180429.678181-1-hannes@cmpxchg.org> References: <20240320180429.678181-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: gk8yhxbk6gsapanpjrqffksnff69ijqy X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: D75298002A X-HE-Tag: 1710957900-614337 X-HE-Meta: U2FsdGVkX19qjulXj/c9czicf9H5viq7UlGV0gY6hudJ+YwYgn3MJJqfR+IKkWaprKrgn6ToCtHZeDlBH2VqhARDfOneSKR3YrkDIDf9+tTAISt1Qm+CIT8vFmcfN6V74xotRQ7JfLAlVuMJn/zepIyL8t/bux5dgLiMjtUPsUrtwDjWZdVhraVtc2FFov04HeYKXw1DIBzaKZWbokHmtSomiwTz4EXImy83e+UKJP9BNihhNebJ9xBuFLueaFwiqMbkxKQywVBoFF9s9Y4ioP2uRFXKxBmGBSzZvOlG1Zc9Zb+bQBXqu0+Zatw757B3bSTvZ3nUcCjt6e5ette8OgwTd8lsWnQijeyrZBe3vhUytByYf/XVjP5m7kE0E2O6YufsyoTG0yeQPsgjK7lTTCsE9qK1DrKR6Af4IeGXTUqldgcp7XBkXjvWFvIqfwTnMwW19dRMck7m5AM0Qp7wtKnBKSbW9oIKHF7lSbtN+WQryBA+30X342P/0Cqfdk75ncseEmxFzHlhM8KyJWz51dloyf7LZYgIOJg+mQdiRqzkUGDfZG+J1WFkcmGjic1KXzbzID8Ngx/W1rpEeFrScV5AD3+OFDLGbvxBsrI1BtCVIn/vguMbiiDdNmbZZscSQl8AegBeNUYjvhskDWt/+0+9AWYhPuQ9yLfCynfEJyNnZhIFGDpdrqf4OQh7wMZj6lbnaNixoSym02F5ZSghBOvHzmI05F3ksYgOiGA7DMaah5+gjxA6s7vnfPIqUBM5vefgMD0WCZhmGRz+ch+FtjLYhYhQf0pphvCH16KqWBjI+u2xfhXKDkxRfB9XdSkee7GZEq1QCuGI+3npeE39JXgvYP5XCTioXH6eHDtTy8wkf+Gd0xjW/edqwDxxHqPLN5BS5foW6DIaWl3GmSqU3iEfIw0bjY6inMyEBg6SVafD6shQZoXVgbCiUlBHI6/2TzGiArQrFeNgRMH4GmR RGxjW7mK W2PwAU4YS1hn55oUiPnCFQd8TifE2458Fvx3nvQNisphQ+eCRVdc125xfOR2eS5We16TIQVVywLP1GdslcNiZN/W7GyjI6QTX969NfrtvlazytKXaPVBugKpX3R68IS8LtiR64ONpAgR7rHcjVMuVYJ8564ZNpAarE71ukja45y1RP8I6NKblyzeLjBnRYs0RIN/MIrx6A/1PXDOSrxMKcOKGwVxVCwcBsmWa X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently, page block type conversion during fallbacks, atomic reservations and isolation can strand various amounts of free pages on incorrect freelists. For example, fallback stealing moves free pages in the block to the new type's freelists, but then may not actually claim the block for that type if there aren't enough compatible pages already allocated. In all cases, free page moving might fail if the block straddles more than one zone, in which case no free pages are moved at all, but the block type is changed anyway. This is detrimental to type hygiene on the freelists. It encourages incompatible page mixing down the line (ask for one type, get another) and thus contributes to long-term fragmentation. Split the process into a proper transaction: check first if conversion will happen, then try to move the free pages, and only if that was successful convert the block to the new type. Tested-by: "Huang, Ying" Signed-off-by: Johannes Weiner Reviewed-by: Vlastimil Babka Acked-by: Vlastimil Babka --- include/linux/page-isolation.h | 3 +- mm/page_alloc.c | 175 ++++++++++++++++++++------------- mm/page_isolation.c | 22 +++-- 3 files changed, 121 insertions(+), 79 deletions(-) diff --git a/include/linux/page-isolation.h b/include/linux/page-isolation.h index 4ac34392823a..8550b3c91480 100644 --- a/include/linux/page-isolation.h +++ b/include/linux/page-isolation.h @@ -34,8 +34,7 @@ static inline bool is_migrate_isolate(int migratetype) #define REPORT_FAILURE 0x2 void set_pageblock_migratetype(struct page *page, int migratetype); -int move_freepages_block(struct zone *zone, struct page *page, - int migratetype, int *num_movable); +int move_freepages_block(struct zone *zone, struct page *page, int migratetype); int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, int migratetype, int flags, gfp_t gfp_flags); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 7373329763e6..e7d0d4711bdd 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1596,9 +1596,8 @@ static inline struct page *__rmqueue_cma_fallback(struct zone *zone, * Note that start_page and end_pages are not aligned on a pageblock * boundary. If alignment is required, use move_freepages_block() */ -static int move_freepages(struct zone *zone, - unsigned long start_pfn, unsigned long end_pfn, - int migratetype, int *num_movable) +static int move_freepages(struct zone *zone, unsigned long start_pfn, + unsigned long end_pfn, int migratetype) { struct page *page; unsigned long pfn; @@ -1608,14 +1607,6 @@ static int move_freepages(struct zone *zone, for (pfn = start_pfn; pfn <= end_pfn;) { page = pfn_to_page(pfn); if (!PageBuddy(page)) { - /* - * We assume that pages that could be isolated for - * migration are movable. But we don't actually try - * isolating, as that would be expensive. - */ - if (num_movable && - (PageLRU(page) || __PageMovable(page))) - (*num_movable)++; pfn++; continue; } @@ -1633,17 +1624,16 @@ static int move_freepages(struct zone *zone, return pages_moved; } -int move_freepages_block(struct zone *zone, struct page *page, - int migratetype, int *num_movable) +static bool prep_move_freepages_block(struct zone *zone, struct page *page, + unsigned long *start_pfn, + unsigned long *end_pfn, + int *num_free, int *num_movable) { - unsigned long start_pfn, end_pfn, pfn; - - if (num_movable) - *num_movable = 0; + unsigned long pfn, start, end; pfn = page_to_pfn(page); - start_pfn = pageblock_start_pfn(pfn); - end_pfn = pageblock_end_pfn(pfn) - 1; + start = pageblock_start_pfn(pfn); + end = pageblock_end_pfn(pfn) - 1; /* * The caller only has the lock for @zone, don't touch ranges @@ -1652,13 +1642,50 @@ int move_freepages_block(struct zone *zone, struct page *page, * accompanied by other operations such as migratetype updates * which also should be locked. */ - if (!zone_spans_pfn(zone, start_pfn)) - return 0; - if (!zone_spans_pfn(zone, end_pfn)) - return 0; + if (!zone_spans_pfn(zone, start)) + return false; + if (!zone_spans_pfn(zone, end)) + return false; + + *start_pfn = start; + *end_pfn = end; + + if (num_free) { + *num_free = 0; + *num_movable = 0; + for (pfn = start; pfn <= end;) { + page = pfn_to_page(pfn); + if (PageBuddy(page)) { + int nr = 1 << buddy_order(page); + + *num_free += nr; + pfn += nr; + continue; + } + /* + * We assume that pages that could be isolated for + * migration are movable. But we don't actually try + * isolating, as that would be expensive. + */ + if (PageLRU(page) || __PageMovable(page)) + (*num_movable)++; + pfn++; + } + } + + return true; +} + +int move_freepages_block(struct zone *zone, struct page *page, + int migratetype) +{ + unsigned long start_pfn, end_pfn; + + if (!prep_move_freepages_block(zone, page, &start_pfn, &end_pfn, + NULL, NULL)) + return -1; - return move_freepages(zone, start_pfn, end_pfn, migratetype, - num_movable); + return move_freepages(zone, start_pfn, end_pfn, migratetype); } static void change_pageblock_range(struct page *pageblock_page, @@ -1743,33 +1770,37 @@ static inline bool boost_watermark(struct zone *zone) } /* - * This function implements actual steal behaviour. If order is large enough, - * we can steal whole pageblock. If not, we first move freepages in this - * pageblock to our migratetype and determine how many already-allocated pages - * are there in the pageblock with a compatible migratetype. If at least half - * of pages are free or compatible, we can change migratetype of the pageblock - * itself, so pages freed in the future will be put on the correct free list. + * This function implements actual steal behaviour. If order is large enough, we + * can claim the whole pageblock for the requested migratetype. If not, we check + * the pageblock for constituent pages; if at least half of the pages are free + * or compatible, we can still claim the whole block, so pages freed in the + * future will be put on the correct free list. Otherwise, we isolate exactly + * the order we need from the fallback block and leave its migratetype alone. */ -static void steal_suitable_fallback(struct zone *zone, struct page *page, - unsigned int alloc_flags, int start_type, bool whole_block) +static struct page * +steal_suitable_fallback(struct zone *zone, struct page *page, + int current_order, int order, int start_type, + unsigned int alloc_flags, bool whole_block) { - unsigned int current_order = buddy_order(page); int free_pages, movable_pages, alike_pages; - int old_block_type; + unsigned long start_pfn, end_pfn; + int block_type; - old_block_type = get_pageblock_migratetype(page); + block_type = get_pageblock_migratetype(page); /* * This can happen due to races and we want to prevent broken * highatomic accounting. */ - if (is_migrate_highatomic(old_block_type)) + if (is_migrate_highatomic(block_type)) goto single_page; /* Take ownership for orders >= pageblock_order */ if (current_order >= pageblock_order) { + del_page_from_free_list(page, zone, current_order); change_pageblock_range(page, current_order, start_type); - goto single_page; + expand(zone, page, order, current_order, start_type); + return page; } /* @@ -1784,10 +1815,9 @@ static void steal_suitable_fallback(struct zone *zone, struct page *page, if (!whole_block) goto single_page; - free_pages = move_freepages_block(zone, page, start_type, - &movable_pages); /* moving whole block can fail due to zone boundary conditions */ - if (!free_pages) + if (!prep_move_freepages_block(zone, page, &start_pfn, &end_pfn, + &free_pages, &movable_pages)) goto single_page; /* @@ -1805,7 +1835,7 @@ static void steal_suitable_fallback(struct zone *zone, struct page *page, * vice versa, be conservative since we can't distinguish the * exact migratetype of non-movable pages. */ - if (old_block_type == MIGRATE_MOVABLE) + if (block_type == MIGRATE_MOVABLE) alike_pages = pageblock_nr_pages - (free_pages + movable_pages); else @@ -1816,13 +1846,16 @@ static void steal_suitable_fallback(struct zone *zone, struct page *page, * compatible migratability as our allocation, claim the whole block. */ if (free_pages + alike_pages >= (1 << (pageblock_order-1)) || - page_group_by_mobility_disabled) + page_group_by_mobility_disabled) { + move_freepages(zone, start_pfn, end_pfn, start_type); set_pageblock_migratetype(page, start_type); - - return; + return __rmqueue_smallest(zone, order, start_type); + } single_page: - move_to_free_list(page, zone, current_order, start_type); + del_page_from_free_list(page, zone, current_order); + expand(zone, page, order, current_order, block_type); + return page; } /* @@ -1890,9 +1923,10 @@ static void reserve_highatomic_pageblock(struct page *page, struct zone *zone) mt = get_pageblock_migratetype(page); /* Only reserve normal pageblocks (i.e., they can merge with others) */ if (migratetype_is_mergeable(mt)) { - zone->nr_reserved_highatomic += pageblock_nr_pages; - set_pageblock_migratetype(page, MIGRATE_HIGHATOMIC); - move_freepages_block(zone, page, MIGRATE_HIGHATOMIC, NULL); + if (move_freepages_block(zone, page, MIGRATE_HIGHATOMIC) != -1) { + set_pageblock_migratetype(page, MIGRATE_HIGHATOMIC); + zone->nr_reserved_highatomic += pageblock_nr_pages; + } } out_unlock: @@ -1917,7 +1951,7 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, struct zone *zone; struct page *page; int order; - bool ret; + int ret; for_each_zone_zonelist_nodemask(zone, z, zonelist, ac->highest_zoneidx, ac->nodemask) { @@ -1966,10 +2000,14 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, * of pageblocks that cannot be completely freed * may increase. */ + ret = move_freepages_block(zone, page, ac->migratetype); + /* + * Reserving this block already succeeded, so this should + * not fail on zone boundaries. + */ + WARN_ON_ONCE(ret == -1); set_pageblock_migratetype(page, ac->migratetype); - ret = move_freepages_block(zone, page, ac->migratetype, - NULL); - if (ret) { + if (ret > 0) { spin_unlock_irqrestore(&zone->lock, flags); return ret; } @@ -1990,7 +2028,7 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, * deviation from the rest of this file, to make the for loop * condition simpler. */ -static __always_inline bool +static __always_inline struct page * __rmqueue_fallback(struct zone *zone, int order, int start_migratetype, unsigned int alloc_flags) { @@ -2037,7 +2075,7 @@ __rmqueue_fallback(struct zone *zone, int order, int start_migratetype, goto do_steal; } - return false; + return NULL; find_smallest: for (current_order = order; current_order < NR_PAGE_ORDERS; current_order++) { @@ -2057,14 +2095,14 @@ __rmqueue_fallback(struct zone *zone, int order, int start_migratetype, do_steal: page = get_page_from_free_area(area, fallback_mt); - steal_suitable_fallback(zone, page, alloc_flags, start_migratetype, - can_steal); + /* take off list, maybe claim block, expand remainder */ + page = steal_suitable_fallback(zone, page, current_order, order, + start_migratetype, alloc_flags, can_steal); trace_mm_page_alloc_extfrag(page, order, current_order, start_migratetype, fallback_mt); - return true; - + return page; } #ifdef CONFIG_CMA @@ -2127,15 +2165,14 @@ __rmqueue(struct zone *zone, unsigned int order, int migratetype, return page; } } -retry: + page = __rmqueue_smallest(zone, order, migratetype); if (unlikely(!page)) { if (alloc_flags & ALLOC_CMA) page = __rmqueue_cma_fallback(zone, order); - - if (!page && __rmqueue_fallback(zone, order, migratetype, - alloc_flags)) - goto retry; + else + page = __rmqueue_fallback(zone, order, migratetype, + alloc_flags); } return page; } @@ -2689,12 +2726,10 @@ int __isolate_free_page(struct page *page, unsigned int order) * Only change normal pageblocks (i.e., they can merge * with others) */ - if (migratetype_is_mergeable(mt)) { - set_pageblock_migratetype(page, - MIGRATE_MOVABLE); - move_freepages_block(zone, page, - MIGRATE_MOVABLE, NULL); - } + if (migratetype_is_mergeable(mt) && + move_freepages_block(zone, page, + MIGRATE_MOVABLE) != -1) + set_pageblock_migratetype(page, MIGRATE_MOVABLE); } } diff --git a/mm/page_isolation.c b/mm/page_isolation.c index a5c8fa4c2a75..71539d7b96cf 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -178,15 +178,18 @@ static int set_migratetype_isolate(struct page *page, int migratetype, int isol_ unmovable = has_unmovable_pages(check_unmovable_start, check_unmovable_end, migratetype, isol_flags); if (!unmovable) { - unsigned long nr_pages; + int nr_pages; int mt = get_pageblock_migratetype(page); + nr_pages = move_freepages_block(zone, page, MIGRATE_ISOLATE); + /* Block spans zone boundaries? */ + if (nr_pages == -1) { + spin_unlock_irqrestore(&zone->lock, flags); + return -EBUSY; + } + __mod_zone_freepage_state(zone, -nr_pages, mt); set_pageblock_migratetype(page, MIGRATE_ISOLATE); zone->nr_isolate_pageblock++; - nr_pages = move_freepages_block(zone, page, MIGRATE_ISOLATE, - NULL); - - __mod_zone_freepage_state(zone, -nr_pages, mt); spin_unlock_irqrestore(&zone->lock, flags); return 0; } @@ -206,7 +209,7 @@ static int set_migratetype_isolate(struct page *page, int migratetype, int isol_ static void unset_migratetype_isolate(struct page *page, int migratetype) { struct zone *zone; - unsigned long flags, nr_pages; + unsigned long flags; bool isolated_page = false; unsigned int order; struct page *buddy; @@ -252,7 +255,12 @@ static void unset_migratetype_isolate(struct page *page, int migratetype) * allocation. */ if (!isolated_page) { - nr_pages = move_freepages_block(zone, page, migratetype, NULL); + int nr_pages = move_freepages_block(zone, page, migratetype); + /* + * Isolating this block already succeeded, so this + * should not fail on zone boundaries. + */ + WARN_ON_ONCE(nr_pages == -1); __mod_zone_freepage_state(zone, nr_pages, migratetype); } set_pageblock_migratetype(page, migratetype); From patchwork Wed Mar 20 18:02:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13598067 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 496B5C54E58 for ; Wed, 20 Mar 2024 18:05:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 37D596B009A; Wed, 20 Mar 2024 14:05:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 303946B009B; Wed, 20 Mar 2024 14:05:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 107236B009C; Wed, 20 Mar 2024 14:05:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id EC4AA6B009A for ; Wed, 20 Mar 2024 14:05:04 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A01561C15CB for ; Wed, 20 Mar 2024 18:05:04 +0000 (UTC) X-FDA: 81918193728.09.3326402 Received: from mail-qv1-f50.google.com (mail-qv1-f50.google.com [209.85.219.50]) by imf29.hostedemail.com (Postfix) with ESMTP id 6C844120017 for ; Wed, 20 Mar 2024 18:05:02 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=ob6a0R5L; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf29.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.50 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710957902; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=flNXTtbOi6l1opPzPulo8MxPNWZbuYfRY5TPRoxN0V8=; b=JTROa+pxQyYkFp1hdCPLj33LHTSuMw3Va768gg7o+pLVMaXwA6EL2OkSq57sLqiszmsvtM jPdlhId+REtNlokLF0pgOxLCcOl3fJyQsKpViCShWCy1uYMazrqQW1GgPtkybErNC/o4em ERkc73oCYIp7kMevqS35LKgBMgbP2vI= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=ob6a0R5L; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf29.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.50 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710957902; a=rsa-sha256; cv=none; b=TFkPk1gb31IV+ftColGZ6fJJ4tfpevjWkBKS2hwbHghoOdMUBFmEig1fzmVxWYYU3ZoQzK y26ckIk1r1tOqtOt7R+4JMax/iskscO+S+ZOcNqavTtvLFe0seQb4+W1waO9EXdtiAlvze Lv5j2e8O7BLeg1uvlbOMe/BxdcPLHdA= Received: by mail-qv1-f50.google.com with SMTP id 6a1803df08f44-690fed6816fso1159296d6.1 for ; Wed, 20 Mar 2024 11:05:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1710957901; x=1711562701; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=flNXTtbOi6l1opPzPulo8MxPNWZbuYfRY5TPRoxN0V8=; b=ob6a0R5L+O12jRyP600rGiAr2z0duhKxqK7DVrcmG74Xf4eoyYYqjd1pnoNTGVYx/m llIaHpJ6j8iKeUf/EnDfhGItwjaKDagDAQcUZDOAN42fX0IybrTeXsxEB56F4x9uCoSW xoZ24tv1eiGAGIbCzY0tSUgVNPuOPr1Et6+JzeKWmMZ6LiGcxwRHjrhF6YSZ2RWuT24n K+ci/UX2GElIge6c+XPrLmMexdZ2rGCVKB3Bztc743IpcrqvW8aRoCa5n+dyi/6cX2ej Q64dyK2n0xRfpuNkyQ00S67C3HKW5RYEROUGW/zLTJztSe459Y2y/WFYgqX2L9NvTDJf xh7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710957901; x=1711562701; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=flNXTtbOi6l1opPzPulo8MxPNWZbuYfRY5TPRoxN0V8=; b=gu9mP1uWJVZp55Hu1PqO6hLzMdCerQGZrZMK47SXwfsBEAtJnjQgVwZLWZALlfUQoZ izrGuHIGb7zhT7LUAfI3+VRHuMlqQ8+0SQilexbztRL2JgieknP0Btvtj4T2YuasLt8L SbWKwx4bYjblKFD1Qz43J+pKRRUSdHb7fKLhAJ+NdQ9HSs4w6TyA3A9GQyLtGjN+o+AF nfj1hZQfNdTkVehyyervPepB1i74iSGnGLsvHaPANryWCQuoWfzWOnnR/7YShNprno4z rK/spGsBn4KGdtbcIDOZWgCJ/Axp5Lv2nnZxOJq8dAnQeB5Wtgsb5ZIOAapjowwd4pY2 qvXA== X-Forwarded-Encrypted: i=1; AJvYcCUHK56dE5APMtswZxnRMjOwycO0lFQ6IVQwqeynLbFp46g+CSaCnNMo/lXTy/ffqz7/IZtwlzmPl4JWZl9Nx1nlS5c= X-Gm-Message-State: AOJu0Yz2fzoDD6F+Iy72mptzyxIir862Q8GSzOghMY2yl1lf88n79ZS2 DDiD5g1ZdDuCC52W3ieZyQxwcx7V5OOjPKLl56GbEH/WjkaqU/XwXlHAF7sQ6s0= X-Google-Smtp-Source: AGHT+IFZ8dHGGlrLtFAYw7pJUEMpCiD9DvUpZzuKYVnmjN5J3RADhgWs7NKoZjpYmw1YXPj55rjIPA== X-Received: by 2002:ad4:55d1:0:b0:696:48ca:99ad with SMTP id bt17-20020ad455d1000000b0069648ca99admr1661910qvb.14.1710957901465; Wed, 20 Mar 2024 11:05:01 -0700 (PDT) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id jm14-20020ad45ece000000b00690c5cc0ff6sm8102285qvb.124.2024.03.20.11.05.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Mar 2024 11:05:00 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 07/10] mm: page_alloc: close migratetype race between freeing and stealing Date: Wed, 20 Mar 2024 14:02:12 -0400 Message-ID: <20240320180429.678181-8-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240320180429.678181-1-hannes@cmpxchg.org> References: <20240320180429.678181-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 658mshzjbsdpxihdsr4crq4aqjkhi6ck X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 6C844120017 X-HE-Tag: 1710957902-760970 X-HE-Meta: U2FsdGVkX1+UuRrItA5YAbDyDW8KbKGgbBtQUQQwadTF/lYLrXp2vniX74DXlNgO9pZBhr13mx52gibxofQMKcqVRQpEvFQpLVLKcJScM8a8Tp4jVr0kxnuLUXYWT3/h1GQ8KjPAlOwRmEYUyqqXYV50NcwaTm8auSt3zuBDZZHcLHLeJH5SG2KwpNJdSFgwY6K5ELP616HjVEBmBeQaY90vfD+YtlEi3QsWfBYsKZZqzXORTk8Yp7ZbYx1JzMROxx1ZjVLdke3Ug0Ha/cOe4h9UzqrQO1ASW+oJwDDuGcpxYe7K+tjnBAiobzdmOLut9G6O3gSLz7HQ0aojp85HlfLfxJjeJi3A5NUZ8Mba3T/lBoL+BK82twPggKgF1bTRfKlLGAdMXTeCepB57pHJxYtFjt6be1Nv64b6K23zjECO5D1Ao7D7dUeTI7YXWu1OhviGCqr2qfOZaFsYwO/AA6o045nHmsFrz+09rQw/oWCmQJjS1G7/a69hVKZZtNIkRwYQUvXQ5nUDnjs3J+7MDrwWn6Pq7oPcNIChH1Rmk8icm3J5Ruj3p8kH4E+aGMocebs6imWGKWSQBMXP3+P1TCdn3+gZodKPVVOS6JulmWZeAYOqkBf8o95fYUkvqF3IZ6bq2YY2QYeIC2wUZrugldLVc0JKvAsW4UP/4Yl9NsC4Kjvw8vVJDk6Udc08VFaPvMgJnK4gb2PnFUVj+bcviFo+sd+0qEIwCBp2GRzEsbUgZ73urPsByz6leE8AlCxGdtlvIMhTe6duz36mvObQa1ELUudB0xqBqD0+wqVj2tZaei5bC37C27KHSpums/kyh3SgDtPOQg1on3623wG8T5xjhJCAVR8ToXM3fuVtlon95BvKad2nYPl7cQhB8xjBSWr9Djx2Sa5in/9W4tAsIDIQvgjn60EuqIa5z1IefSF0AiwnA3pTZLQCe/P29WjAMPq7xwKotwoPW9ocERF fIEapu4D xFlcVoItGcCZq5Ksl8951VT18HZMCXAsGW143KFG8KEk3uVEBYHbEh2RXYBm3AJZPKlv8n9yZqzEe1aOhTcr9PueUP0CB8se7aDBHF5+GivcNK+RWTUKMeTqYl68iB8i2DiTtQl2Hzq3EvjTOr+QvoIUDv/V8WPthPyMNH2H2VWvm6tkxyJCyTgKopxla9d86309GqOGbGb08FEQ3f9H+mJjzXRvZr0Wh/jpry6dcZnlFixVUiNspObZwMQPvi6qmqBd5zj2fVT6SvLUIVR5fgpC9mFTvtg8HMv3ce+N4VdIjcsgs6jKn2UXjbUqevUefAbxgTXqFBfzsrsMoHlhLmu4+csYO7kU3Z2H8RtzIOBafomX7FPzHXXf0/OoEPYR9EOLv0Q47KVNEUlGkYB547Z8TOIe72Wu60xsUnHUY1b8p2yYQuuv7WD/HYlAl8T77vfHlPDBgPVcfk/TfWhY+XUCft6nF751eioMD9TlpAR+Fn9k= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There are three freeing paths that read the page's migratetype optimistically before grabbing the zone lock. When this races with block stealing, those pages go on the wrong freelist. The paths in question are: - when freeing >costly orders that aren't THP - when freeing pages to the buddy upon pcp lock contention - when freeing pages that are isolated - when freeing pages initially during boot - when freeing the remainder in alloc_pages_exact() - when "accepting" unaccepted VM host memory before first use - when freeing pages during unpoisoning None of these are so hot that they would need this optimization at the cost of hampering defrag efforts. Especially when contrasted with the fact that the most common buddy freeing path - free_pcppages_bulk - is checking the migratetype under the zone->lock just fine. In addition, isolated pages need to look up the migratetype under the lock anyway, which adds branches to the locked section, and results in a double lookup when the pages are in fact isolated. Move the lookups into the lock. Reported-by: Vlastimil Babka Signed-off-by: Johannes Weiner Reviewed-by: Vlastimil Babka --- mm/page_alloc.c | 52 ++++++++++++++++++------------------------------- 1 file changed, 19 insertions(+), 33 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index e7d0d4711bdd..3f65b565eaad 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1227,18 +1227,15 @@ static void free_pcppages_bulk(struct zone *zone, int count, spin_unlock_irqrestore(&zone->lock, flags); } -static void free_one_page(struct zone *zone, - struct page *page, unsigned long pfn, - unsigned int order, - int migratetype, fpi_t fpi_flags) +static void free_one_page(struct zone *zone, struct page *page, + unsigned long pfn, unsigned int order, + fpi_t fpi_flags) { unsigned long flags; + int migratetype; spin_lock_irqsave(&zone->lock, flags); - if (unlikely(has_isolate_pageblock(zone) || - is_migrate_isolate(migratetype))) { - migratetype = get_pfnblock_migratetype(page, pfn); - } + migratetype = get_pfnblock_migratetype(page, pfn); __free_one_page(page, pfn, zone, order, migratetype, fpi_flags); spin_unlock_irqrestore(&zone->lock, flags); } @@ -1246,21 +1243,13 @@ static void free_one_page(struct zone *zone, static void __free_pages_ok(struct page *page, unsigned int order, fpi_t fpi_flags) { - int migratetype; unsigned long pfn = page_to_pfn(page); struct zone *zone = page_zone(page); if (!free_pages_prepare(page, order)) return; - /* - * Calling get_pfnblock_migratetype() without spin_lock_irqsave() here - * is used to avoid calling get_pfnblock_migratetype() under the lock. - * This will reduce the lock holding time. - */ - migratetype = get_pfnblock_migratetype(page, pfn); - - free_one_page(zone, page, pfn, order, migratetype, fpi_flags); + free_one_page(zone, page, pfn, order, fpi_flags); __count_vm_events(PGFREE, 1 << order); } @@ -2533,7 +2522,7 @@ void free_unref_page(struct page *page, unsigned int order) struct per_cpu_pages *pcp; struct zone *zone; unsigned long pfn = page_to_pfn(page); - int migratetype, pcpmigratetype; + int migratetype; if (!free_pages_prepare(page, order)) return; @@ -2545,23 +2534,23 @@ void free_unref_page(struct page *page, unsigned int order) * get those areas back if necessary. Otherwise, we may have to free * excessively into the page allocator */ - migratetype = pcpmigratetype = get_pfnblock_migratetype(page, pfn); + migratetype = get_pfnblock_migratetype(page, pfn); if (unlikely(migratetype >= MIGRATE_PCPTYPES)) { if (unlikely(is_migrate_isolate(migratetype))) { - free_one_page(page_zone(page), page, pfn, order, migratetype, FPI_NONE); + free_one_page(page_zone(page), page, pfn, order, FPI_NONE); return; } - pcpmigratetype = MIGRATE_MOVABLE; + migratetype = MIGRATE_MOVABLE; } zone = page_zone(page); pcp_trylock_prepare(UP_flags); pcp = pcp_spin_trylock(zone->per_cpu_pageset); if (pcp) { - free_unref_page_commit(zone, pcp, page, pcpmigratetype, order); + free_unref_page_commit(zone, pcp, page, migratetype, order); pcp_spin_unlock(pcp); } else { - free_one_page(zone, page, pfn, order, migratetype, FPI_NONE); + free_one_page(zone, page, pfn, order, FPI_NONE); } pcp_trylock_finish(UP_flags); } @@ -2591,12 +2580,8 @@ void free_unref_folios(struct folio_batch *folios) * allocator. */ if (!pcp_allowed_order(order)) { - int migratetype; - - migratetype = get_pfnblock_migratetype(&folio->page, - pfn); - free_one_page(folio_zone(folio), &folio->page, pfn, - order, migratetype, FPI_NONE); + free_one_page(folio_zone(folio), &folio->page, + pfn, order, FPI_NONE); continue; } folio->private = (void *)(unsigned long)order; @@ -2632,7 +2617,7 @@ void free_unref_folios(struct folio_batch *folios) */ if (is_migrate_isolate(migratetype)) { free_one_page(zone, &folio->page, pfn, - order, migratetype, FPI_NONE); + order, FPI_NONE); continue; } @@ -2645,7 +2630,7 @@ void free_unref_folios(struct folio_batch *folios) if (unlikely(!pcp)) { pcp_trylock_finish(UP_flags); free_one_page(zone, &folio->page, pfn, - order, migratetype, FPI_NONE); + order, FPI_NONE); continue; } locked_zone = zone; @@ -6823,13 +6808,14 @@ bool take_page_off_buddy(struct page *page) bool put_page_back_buddy(struct page *page) { struct zone *zone = page_zone(page); - unsigned long pfn = page_to_pfn(page); unsigned long flags; - int migratetype = get_pfnblock_migratetype(page, pfn); bool ret = false; spin_lock_irqsave(&zone->lock, flags); if (put_page_testzero(page)) { + unsigned long pfn = page_to_pfn(page); + int migratetype = get_pfnblock_migratetype(page, pfn); + ClearPageHWPoisonTakenOff(page); __free_one_page(page, pfn, zone, 0, migratetype, FPI_NONE); if (TestClearPageHWPoison(page)) { From patchwork Wed Mar 20 18:02:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13598068 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80875C54E58 for ; Wed, 20 Mar 2024 18:05:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6D90F6B009C; Wed, 20 Mar 2024 14:05:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 65EAE6B009D; Wed, 20 Mar 2024 14:05:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4899E6B009E; Wed, 20 Mar 2024 14:05:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 2C4DD6B009C for ; Wed, 20 Mar 2024 14:05:06 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id D7BA5160CCF for ; Wed, 20 Mar 2024 18:05:05 +0000 (UTC) X-FDA: 81918193770.23.C9A1D2C Received: from mail-qk1-f178.google.com (mail-qk1-f178.google.com [209.85.222.178]) by imf02.hostedemail.com (Postfix) with ESMTP id A7E388000A for ; Wed, 20 Mar 2024 18:05:03 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=RFoJQvs9; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf02.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.178 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710957903; a=rsa-sha256; cv=none; b=Yic2SJ7DYQcT82KHf2fw5omP5/alCMCNKQX69P0JfMFprrSk4fdxK1HCr+TGHdp/OM2iod wjootHXUHnZtVgtCltALyt+FzGnOz6jgk59nlMHO2xey/+DwnSZZDOz/42jkjRsu+HMEC8 Dfrd2DCjUoFskpWy+Zrza8/IswOuqdc= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=RFoJQvs9; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf02.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.178 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710957903; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/agc7Iq+rpxvE93VG+YAHYepTCDVe1OJjvDT1Xeceos=; b=8dudNTd51ypoxaoUC4oDQr1GkQrpHPU9KGh2mxF9PoLrk7NeT0cQpYeIVb2BoJgyFO7fRi LDGbibnJ5A3zhOwKBkFmI2UMY1A26dal/e7iXHoiC0rI6rBHfxVSceuC+9qDdlgV2aAiWT Jbz+dW5Oie0d0P9hyX/JEGhKuXicYvo= Received: by mail-qk1-f178.google.com with SMTP id af79cd13be357-789e6e1d96eso10281385a.3 for ; Wed, 20 Mar 2024 11:05:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1710957903; x=1711562703; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/agc7Iq+rpxvE93VG+YAHYepTCDVe1OJjvDT1Xeceos=; b=RFoJQvs96cN8iGNFpu16jpiuSovmrMryRLJOigOAYa3HFib5Rl3IStk7i8xeuL3t2E ABWpDtrngIZ/9VlR7fyHFug9ei9k8EpZuA2aE9xMxHHcW8UrILqdRP1uSlZHaZp7oRq3 0hp3wBm/uf2xD2z0VBf/be/qVJD1xaiAn77a3Ibk63nc1kbcaBBDq/aD5s0xMyhuisyy Dy7RkrGmuli+zrmF9ZsCnqbDWQyFkhM4jEwGlgg0JPlyLgrltTmg8CP9BDcUC5WWWx/s aWnDzroF6TAodLFJj8eiSqGYWkbrVZuJnAXGTNEryEVXy15B9hz7x6c3S0M7Pw/qy2rT cOeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710957903; x=1711562703; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/agc7Iq+rpxvE93VG+YAHYepTCDVe1OJjvDT1Xeceos=; b=jPIGEj8wSTFiCRn1S4D/hMS6LcdVl+5Q5x1NbNyRw8WCG1KnecUnbCePcfnigEE1C3 LqsGd56slhQx7A2KiX7QHaNVtNfgEBb86D0tR5mzrqrlsVusxfRtWYgDY+126+wYYf2v dgd591lFMxzX+kvzU8YfWvET47NiDJ3hMM+Oisa9kSVsrtmJVgWdBA5AAwoUYp+lQEvK /52pgAKVXVFUGMy7PlNcwc4AM/GvAYHFAA90aie6p5A1Xh4523ZVhRaQM5NhVp9CMIBa qvxwRlghwNy7LMutm3PSPdvwvtvWLhrxI7KSmRNTws3HCXUGy5GvTkY66vfj9nBR6k/2 tyPw== X-Forwarded-Encrypted: i=1; AJvYcCXjJWy/Kv+AJCNkvbHjJzbRP96W529hIDgYB3xsBqGWJthh2FVLzOAPWEo1g16qEbAAVtXH4ON89/cdDP1FdiuZF/c= X-Gm-Message-State: AOJu0YzWugWsjVCegeiRE7jm40Ywtqgo18kT/+qMmCZRIenirc+0QH8z euizg+vbJdIXWODMGiUp2n4k9LN/rSJl0UQ3qofwRNQFrBx8ahBKeznWde8HIX8= X-Google-Smtp-Source: AGHT+IHVDZsbT9cqnC3HSTs2AyL3KxvP5tWB415HoSJ1bTxAaB/NmMu1L5tzkR6Fc/n91vn14sa1gw== X-Received: by 2002:a37:db11:0:b0:78a:86:534f with SMTP id e17-20020a37db11000000b0078a0086534fmr2673755qki.76.1710957902810; Wed, 20 Mar 2024 11:05:02 -0700 (PDT) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id w8-20020a05620a094800b0078a210372fasm900228qkw.86.2024.03.20.11.05.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Mar 2024 11:05:02 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 08/10] mm: page_alloc: set migratetype inside move_freepages() Date: Wed, 20 Mar 2024 14:02:13 -0400 Message-ID: <20240320180429.678181-9-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240320180429.678181-1-hannes@cmpxchg.org> References: <20240320180429.678181-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: A7E388000A X-Stat-Signature: ygagc1sfte3n9c4pkcbqiuwaupttyhhc X-HE-Tag: 1710957903-508470 X-HE-Meta: U2FsdGVkX18yMiUoHHI70q9owJE1bBA2WUYNLszssfKB3kN3geuYx0cdwzSwemUkEa7FpDT5HKWwqmT2QMyeXRuBgHwkh9mYJtrlxMJgItgNIuHJDy9fZRB3wQTulPpf+NPPLmvbus4gU7/063WHTqQ44qqh4p6LdhNYI5d5bn2reU0l8FXXny0dVlkS+mGInHHJHuqckXS5UM4UvZ+jcglQwPM/5Sr0l1ORE2Ar15bVPCLi+zqNo8470yDW/eGZHbPooC6I736Js7YXiK1ivQxG2MP+uY6xgrVyVnmo7C7HopBxwQyks50cm80Ri1HtV950DPNXjI6H/Cy+eVBkYEeXiI79VnQGbSjDRVEy6oMBStjAutsI9JcFj3mxdfFLFuacUfR0WStrh6WCS1sp9f2N2AWmPnkCvcmyAeEHW3c722ok5bX+4FoBi5fD0DPQ/NK91BBQUaSucxqgCZDex0oVmCF7KeTv0W6YLR4ldQsSwhP2yLmkIBiYYwFUAR9NesosiPmbCVP+dVP07zbGOHadkz6GPJwIk6j914W7rEaxpU79WU5iegXjaY4OipeFGJ2BNg7xPd2U9tL7v4YFoCn23zBjureUPMCzxJKrvluep1cd/xVosFsHbIS+XPglSlpA8NUwq/FLLgjYudWils492GrJVY3iGF3UYzklSEnwl89usW0X7BUG7BisXfxdjYJoeAy03BKCROXSyq++5X9DWejaBfDvxDeOIOPwxi5zWaRW+aaqpLvFGMuKeOmrs1j+rLdlpn3a8T5ZstfdkkyvinK+Fp2lbxYl12v3JI7IjqXkPL/oRPi96QM8SBrCc7ZqCxY/Lx2ifjvb1gAAMYzTpEv9o1js2acjNunZZ3G1wFB50RjGqk5V56JT3ZzsNz3R12nW9XfqRtzN2sH36RL9VLyTLykbXJpbVPeP+/C/UWBrvnlg/x+LomNQoDmtYG/aRGMM7VoijsuAUqR qQE1qOQB +vYbMmEwqPQ8XEhInX/B+xtfBpGkEKucwSHfer0UwlyFo6DoUZQFWCdHmy1p8eLod7WoRVl54jg9YNPv6kyaFxigZF3fd5/wkyuKez2fcpa1Ug3UgZjUoeCqas0p6ec6KebBCBzguTu8W3SUrtf4iDlgNFSyUFUOuHqRGsOFAozOwM/OnCehy+002Ozpd/6lEWTR3oJPlOkTzsh/W/LoHJVDZBN4l7w3Vy1bW6q97O/BBD5QcsInhuVgUVS7OtMAaP5YanKwtGVCE5KttrXNlCPZ2KTRo1YtnPeIerKwmpbnCRxQysVU6TKMIOQ2d/IdeXPYTLEVOhaOFEy+BEea+hQIAQvCtp5d70qAnZMc2M1gdB9NDQld3JCb1RCnbh/fgBIFR+MQloXuxkdSUYxPNYHRHYhC6lvNi/dAxCjqq949urHGVadO0zuqHbUpAxDF86RzVZX2VeKoUhtDo6K/Kf19ejqiNEIkVVT8shkrUsi3HPl0ARFs8dZr6ZM6XHVVKGg+kcC/Y8w3xdwU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Zi Yan This avoids changing migratetype after move_freepages() or move_freepages_block(), which is error prone. It also prepares for upcoming changes to fix move_freepages() not moving free pages partially in the range. Signed-off-by: Zi Yan Signed-off-by: Johannes Weiner Reviewed-by: Vlastimil Babka --- mm/page_alloc.c | 27 +++++++++++++-------------- mm/page_isolation.c | 7 +++---- 2 files changed, 16 insertions(+), 18 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3f65b565eaad..d687f27d891f 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1581,9 +1581,8 @@ static inline struct page *__rmqueue_cma_fallback(struct zone *zone, #endif /* - * Move the free pages in a range to the freelist tail of the requested type. - * Note that start_page and end_pages are not aligned on a pageblock - * boundary. If alignment is required, use move_freepages_block() + * Change the type of a block and move all its free pages to that + * type's freelist. */ static int move_freepages(struct zone *zone, unsigned long start_pfn, unsigned long end_pfn, int migratetype) @@ -1593,6 +1592,9 @@ static int move_freepages(struct zone *zone, unsigned long start_pfn, unsigned int order; int pages_moved = 0; + VM_WARN_ON(start_pfn & (pageblock_nr_pages - 1)); + VM_WARN_ON(start_pfn + pageblock_nr_pages - 1 != end_pfn); + for (pfn = start_pfn; pfn <= end_pfn;) { page = pfn_to_page(pfn); if (!PageBuddy(page)) { @@ -1610,6 +1612,8 @@ static int move_freepages(struct zone *zone, unsigned long start_pfn, pages_moved += 1 << order; } + set_pageblock_migratetype(pfn_to_page(start_pfn), migratetype); + return pages_moved; } @@ -1837,7 +1841,6 @@ steal_suitable_fallback(struct zone *zone, struct page *page, if (free_pages + alike_pages >= (1 << (pageblock_order-1)) || page_group_by_mobility_disabled) { move_freepages(zone, start_pfn, end_pfn, start_type); - set_pageblock_migratetype(page, start_type); return __rmqueue_smallest(zone, order, start_type); } @@ -1911,12 +1914,10 @@ static void reserve_highatomic_pageblock(struct page *page, struct zone *zone) /* Yoink! */ mt = get_pageblock_migratetype(page); /* Only reserve normal pageblocks (i.e., they can merge with others) */ - if (migratetype_is_mergeable(mt)) { - if (move_freepages_block(zone, page, MIGRATE_HIGHATOMIC) != -1) { - set_pageblock_migratetype(page, MIGRATE_HIGHATOMIC); + if (migratetype_is_mergeable(mt)) + if (move_freepages_block(zone, page, + MIGRATE_HIGHATOMIC) != -1) zone->nr_reserved_highatomic += pageblock_nr_pages; - } - } out_unlock: spin_unlock_irqrestore(&zone->lock, flags); @@ -1995,7 +1996,6 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, * not fail on zone boundaries. */ WARN_ON_ONCE(ret == -1); - set_pageblock_migratetype(page, ac->migratetype); if (ret > 0) { spin_unlock_irqrestore(&zone->lock, flags); return ret; @@ -2711,10 +2711,9 @@ int __isolate_free_page(struct page *page, unsigned int order) * Only change normal pageblocks (i.e., they can merge * with others) */ - if (migratetype_is_mergeable(mt) && - move_freepages_block(zone, page, - MIGRATE_MOVABLE) != -1) - set_pageblock_migratetype(page, MIGRATE_MOVABLE); + if (migratetype_is_mergeable(mt)) + move_freepages_block(zone, page, + MIGRATE_MOVABLE); } } diff --git a/mm/page_isolation.c b/mm/page_isolation.c index 71539d7b96cf..f84f0981b2df 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -188,7 +188,6 @@ static int set_migratetype_isolate(struct page *page, int migratetype, int isol_ return -EBUSY; } __mod_zone_freepage_state(zone, -nr_pages, mt); - set_pageblock_migratetype(page, MIGRATE_ISOLATE); zone->nr_isolate_pageblock++; spin_unlock_irqrestore(&zone->lock, flags); return 0; @@ -262,10 +261,10 @@ static void unset_migratetype_isolate(struct page *page, int migratetype) */ WARN_ON_ONCE(nr_pages == -1); __mod_zone_freepage_state(zone, nr_pages, migratetype); - } - set_pageblock_migratetype(page, migratetype); - if (isolated_page) + } else { + set_pageblock_migratetype(page, migratetype); __putback_isolated_page(page, order, migratetype); + } zone->nr_isolate_pageblock--; out: spin_unlock_irqrestore(&zone->lock, flags); From patchwork Wed Mar 20 18:02:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13598069 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E148C54E58 for ; Wed, 20 Mar 2024 18:05:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CCA986B009D; Wed, 20 Mar 2024 14:05:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B68A96B009F; Wed, 20 Mar 2024 14:05:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A1136B009D; Wed, 20 Mar 2024 14:05:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8103F6B009D for ; Wed, 20 Mar 2024 14:05:07 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4DFB7812FC for ; Wed, 20 Mar 2024 18:05:07 +0000 (UTC) X-FDA: 81918193854.29.5060E88 Received: from mail-ot1-f48.google.com (mail-ot1-f48.google.com [209.85.210.48]) by imf27.hostedemail.com (Postfix) with ESMTP id 6DCA540018 for ; Wed, 20 Mar 2024 18:05:05 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=FgH6kHVw; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf27.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.210.48 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710957905; a=rsa-sha256; cv=none; b=BVhCqAS4mQsv87SCMTsyBz4vxXLXd0IggfHGu/jDs6rwVWK/JOa6wMfXVsnbypDlQHeASS rwF5/gKO8vg4uJWkfMJzOa0ZZbTN3taL6smfPL92QR6Q6LCy8s/9F6jDcUddlde+kVr0Ih YxQxBvdbRDdJiBvlc1jzNYzPqHtb3Fc= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=FgH6kHVw; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf27.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.210.48 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710957905; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=E6EDIqJ3GpKNGgGvycU8kV/i4h8q8YkWCZkehwdTQBQ=; b=G37a+fYsoypvZt8OJZcZuJ87iWn4nn5Fi2SstvzDQa8mbfWO50WMK+2HkPCOHq8uoI2QQX AiPSi9N7p6ty+5ChYS3B6Lnp6jKNcXksu2f0H9iuDcZkQohqVGNaHlxouCabG3ZDe7WQr+ kP8snb3KTrp7/6VYHzrqD1JTqS8441Q= Received: by mail-ot1-f48.google.com with SMTP id 46e09a7af769-6e675db6fbaso37646a34.1 for ; Wed, 20 Mar 2024 11:05:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1710957904; x=1711562704; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=E6EDIqJ3GpKNGgGvycU8kV/i4h8q8YkWCZkehwdTQBQ=; b=FgH6kHVwVf66ytX/MOWz4n71OtSwBG68NtxHbBMEJvDfZ+qB2tHHbEO7IC7xrTwt02 /qxhrOSEBxcNqD8Hh/F4TPsZfE2ntyLwP+AoKgR900ZQvBQsfpaRx+4IrE3FNEgMikAX WRFV29etqhX1WlEsDxy0M7phfE4F6ErVZs/883pLrzbg4kVMcZWQXCO3YSde+TUEPlZ4 yKWLzdvynRZmibaF8y8LX2g3tCwSGxAWeOAwR0oBhiX4v+zjq8fQYlh5HrzTCoVHgC1X 5//XkyAhiwuL6mwQo+iqhojWR09FW5qYj3GFQrSf6OttwWurPffbu1pP49v4339ouIXL NnaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710957904; x=1711562704; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=E6EDIqJ3GpKNGgGvycU8kV/i4h8q8YkWCZkehwdTQBQ=; b=mB6zZ7WtG5XVwgLsHxq8RBSc4c7fnHovd1qprh+sUb5iGwNnh9ZpLEFtiA7KG2TmC0 SR9ZST4P+b6wr4oFottUmsGrO5rjh3Jdtqp419Ody2H70KH54qADnpY1jWs9lkUY7FYm 3HoJbRwqt1MDX+L2UKviiAd0OVBPmSTHa3yvGUVojIUTICmQSXkexEzjV6orwbCEmHXh KcLnGt2rU257wanqea+/0rstsNrgquZWt1NuxwIyxKrDSTA7BMKdxmZYA4n//4N5R8tM ISbMGPa7+mwc+yqep8p3oe5rtTpV4iJ+DYNoUCeO5pn1dkstStX1sHiq6zIF0ECnSeYT xCIg== X-Forwarded-Encrypted: i=1; AJvYcCWlWB9wl2joZf/MCOnZ5M+tkCRCrEWmYW8S5xkpZMkvCU8oiU2Vtyj3mq0z6CCqKT1EFh4cu/DWWEwA189OgbJ59wk= X-Gm-Message-State: AOJu0Yw1FfdJ9fPUVemw89zXiWAGSiwlYQQ7LKL7be4TRP9k46nPXYMg 793afIuggKw8x19hppnV7V6rFucH+Kx1eR/FxdjbvP1WaJEYe5TV4GLa5fR7tg8= X-Google-Smtp-Source: AGHT+IFWSemKRWWruL0QHrLgb/7FgdcOFthYCoNJGsppv1Ywv9Zl8jdZobhkbMs/1qULPD7k8iTkhQ== X-Received: by 2002:a05:6830:1548:b0:6e6:84b4:5f35 with SMTP id l8-20020a056830154800b006e684b45f35mr6856475otp.8.1710957904362; Wed, 20 Mar 2024 11:05:04 -0700 (PDT) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id gf15-20020a056214250f00b006912014b98dsm8137529qvb.129.2024.03.20.11.05.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Mar 2024 11:05:03 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 09/10] mm: page_isolation: prepare for hygienic freelists Date: Wed, 20 Mar 2024 14:02:14 -0400 Message-ID: <20240320180429.678181-10-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240320180429.678181-1-hannes@cmpxchg.org> References: <20240320180429.678181-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 6DCA540018 X-Stat-Signature: a1e6e4wdehdmnn4ypj7qfeqtx5itm7k4 X-HE-Tag: 1710957905-735827 X-HE-Meta: U2FsdGVkX19v0foEoSmKjDvpfNPYXisr154i/cnsCfv25HaNsTpzO598ULeeYbWzWw5QgkoE/wbOkOqS1Sy+aWlLinR6taTNfN14yOfR5C+zwP9Mj6zNTmejOkqCDyaFBqrCBrHNxrRUU3RVZxbvSHsWgTTq+++HugMmbeL0F+g/wqCCY18cs5lee+ljkcqGxtYfAdtAOzvq+Ga8xxJLTXMxNGLp+ZmTpwHperksRhNJLPaS063zzd50WMN+wbEhSx/DO5VZfwLYxguqaq89XKvD+YfbL8tf2Xaa/q2vVrPFvlVFA2EmwD+ZssYAyA4AlL6AzfoZX+2p8XSD8BqsDwE9vTBNk5+BAyRvew1SvLMs62VMUVXkMTgLQuCqhQJNDGaaJI3rLI8NsHTY4qsvC3r9ro9JEE7Duswr1Wk/iCWmtELH5VnPMucVzefxuP9D+cMFkcvFixNr0tJD1njCVq4LC8PPc46Y51BUVh4zsgaSfdPkJdB5KqRPH7ZKFo5AdIjTZaIfYiU+2tIE01m3KyGlJciKlI/VcKcNvburBvnmaPU9N54dkyuM9CdVXjeplghFoX4JHaBI8iRX4UNhKPYORUW5KggpJtlY6SZn7RZ9at5f+ij/z2mbej8XvRYwLzaQumCKPrCxZTyxQKOzzIikm6JvjdiyjqaeSOsHJQlMbndiJ9Hnj0KvO75FqCsof158C2GbTjQc+KIyqp3Bua0xw2E0vnX1yAIyVp7bDSaRpJy8jnqj16zFs8I0ZYzQKoSLQZExhrUSf9sFqz+ztMm41ggHUyUB5hsuoF9r2dbb70Yw9nguxFeP8TlyaU1VFCWt0iNVWy2z6f82z3EdgpPXZj7ZnF65tIiHqYnduB6pMDjxTljyUBWXvAkwt8UYwdvSFjQfrodMSMk8+7WXtWHoqkW2CT3n5J0ivIYAno/biYj3AFiJCJegLDBrA9loxi6H/SwH64PXTW/LtE6 oTTaAli4 I1EfJPaCpHvA+FjRXPGj2gopGPne+BU4UjdmfrkPeGdtdSSCLfEaooXaK/njsiKvRGKSUedhfkCyPcJ5R/EfKkdHMA4xqFt9Qcc5cZcmKSWLmDeUZIu4etnb8NfyyK7Vy6s0pnzc6JKJqya4lwBWA+0XvhIUEHi+2VlFZoE2gTUH7G/dmk+lhabdYFUzAloAL9uzahuicgU1eH24faMZsP7BQCvUMVfvsr06PzwvWPO+xlgEcm0aG2gFxhQrxOsO6/5of+ivLmjiZxG7TfTGjyXSloJZYksJn/PEbWpE9bYGM3Fu7CXJnp4fGfCAUp1vx4z2+EyPUoVDEOTEhjssd/KUrNnqmte2mWu/HBTljyQNfsY9z8oneRnkmNN2ijeb3potRxej9jky/so1SikN5AKMMYYFhXL/9AXScLakDmbpK6+SOB3hlLehi5WiLm4+/Nx8kqqTs1KKNAFziwITMGFQ/RAQEv2GbzrtDFrP4e8G0rT4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Page isolation currently sets MIGRATE_ISOLATE on a block, then drops zone->lock and scans the block for straddling buddies to split up. Because this happens non-atomically wrt the page allocator, it's possible for allocations to get a buddy whose first block is a regular pcp migratetype but whose tail is isolated. This means that in certain cases memory can still be allocated after isolation. It will also trigger the freelist type hygiene warnings in subsequent patches. start_isolate_page_range() isolate_single_pageblock() set_migratetype_isolate(tail) lock zone->lock move_freepages_block(tail) // nop set_pageblock_migratetype(tail) unlock zone->lock __rmqueue_smallest() del_page_from_freelist(head) expand(head, head_mt) WARN(head_mt != tail_mt) start_pfn = ALIGN_DOWN(MAX_ORDER_NR_PAGES) for (pfn = start_pfn, pfn < end_pfn) if (PageBuddy()) split_free_page(head) Introduce a variant of move_freepages_block() provided by the allocator specifically for page isolation; it moves free pages, converts the block, and handles the splitting of straddling buddies while holding zone->lock. The allocator knows that pageblocks and buddies are always naturally aligned, which means that buddies can only straddle blocks if they're actually >pageblock_order. This means the search-and-split part can be simplified compared to what page isolation used to do. Also tighten up the page isolation code around the expectations of which pages can be large, and how they are freed. Based on extensive discussions with and invaluable input from Zi Yan. Signed-off-by: Johannes Weiner Reviewed-by: Vlastimil Babka --- include/linux/page-isolation.h | 4 +- mm/internal.h | 4 - mm/page_alloc.c | 200 +++++++++++++++++++-------------- mm/page_isolation.c | 106 ++++++----------- 4 files changed, 151 insertions(+), 163 deletions(-) diff --git a/include/linux/page-isolation.h b/include/linux/page-isolation.h index 8550b3c91480..c16db0067090 100644 --- a/include/linux/page-isolation.h +++ b/include/linux/page-isolation.h @@ -34,7 +34,9 @@ static inline bool is_migrate_isolate(int migratetype) #define REPORT_FAILURE 0x2 void set_pageblock_migratetype(struct page *page, int migratetype); -int move_freepages_block(struct zone *zone, struct page *page, int migratetype); + +bool move_freepages_block_isolate(struct zone *zone, struct page *page, + int migratetype); int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, int migratetype, int flags, gfp_t gfp_flags); diff --git a/mm/internal.h b/mm/internal.h index f8b31234c130..d6e6c7d9f04e 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -559,10 +559,6 @@ extern void *memmap_alloc(phys_addr_t size, phys_addr_t align, void memmap_init_range(unsigned long, int, unsigned long, unsigned long, unsigned long, enum meminit_context, struct vmem_altmap *, int); - -int split_free_page(struct page *free_page, - unsigned int order, unsigned long split_pfn_offset); - #if defined CONFIG_COMPACTION || defined CONFIG_CMA /* diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d687f27d891f..efb2581ac142 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -832,64 +832,6 @@ static inline void __free_one_page(struct page *page, page_reporting_notify_free(order); } -/** - * split_free_page() -- split a free page at split_pfn_offset - * @free_page: the original free page - * @order: the order of the page - * @split_pfn_offset: split offset within the page - * - * Return -ENOENT if the free page is changed, otherwise 0 - * - * It is used when the free page crosses two pageblocks with different migratetypes - * at split_pfn_offset within the page. The split free page will be put into - * separate migratetype lists afterwards. Otherwise, the function achieves - * nothing. - */ -int split_free_page(struct page *free_page, - unsigned int order, unsigned long split_pfn_offset) -{ - struct zone *zone = page_zone(free_page); - unsigned long free_page_pfn = page_to_pfn(free_page); - unsigned long pfn; - unsigned long flags; - int free_page_order; - int mt; - int ret = 0; - - if (split_pfn_offset == 0) - return ret; - - spin_lock_irqsave(&zone->lock, flags); - - if (!PageBuddy(free_page) || buddy_order(free_page) != order) { - ret = -ENOENT; - goto out; - } - - mt = get_pfnblock_migratetype(free_page, free_page_pfn); - if (likely(!is_migrate_isolate(mt))) - __mod_zone_freepage_state(zone, -(1UL << order), mt); - - del_page_from_free_list(free_page, zone, order); - for (pfn = free_page_pfn; - pfn < free_page_pfn + (1UL << order);) { - int mt = get_pfnblock_migratetype(pfn_to_page(pfn), pfn); - - free_page_order = min_t(unsigned int, - pfn ? __ffs(pfn) : order, - __fls(split_pfn_offset)); - __free_one_page(pfn_to_page(pfn), pfn, zone, free_page_order, - mt, FPI_NONE); - pfn += 1UL << free_page_order; - split_pfn_offset -= (1UL << free_page_order); - /* we have done the first part, now switch to second part */ - if (split_pfn_offset == 0) - split_pfn_offset = (1UL << order) - (pfn - free_page_pfn); - } -out: - spin_unlock_irqrestore(&zone->lock, flags); - return ret; -} /* * A bad page could be due to a number of fields. Instead of multiple branches, * try and check multiple fields with one check. The caller must do a detailed @@ -1669,8 +1611,8 @@ static bool prep_move_freepages_block(struct zone *zone, struct page *page, return true; } -int move_freepages_block(struct zone *zone, struct page *page, - int migratetype) +static int move_freepages_block(struct zone *zone, struct page *page, + int migratetype) { unsigned long start_pfn, end_pfn; @@ -1681,6 +1623,119 @@ int move_freepages_block(struct zone *zone, struct page *page, return move_freepages(zone, start_pfn, end_pfn, migratetype); } +#ifdef CONFIG_MEMORY_ISOLATION +/* Look for a buddy that straddles start_pfn */ +static unsigned long find_large_buddy(unsigned long start_pfn) +{ + int order = 0; + struct page *page; + unsigned long pfn = start_pfn; + + while (!PageBuddy(page = pfn_to_page(pfn))) { + /* Nothing found */ + if (++order > MAX_PAGE_ORDER) + return start_pfn; + pfn &= ~0UL << order; + } + + /* + * Found a preceding buddy, but does it straddle? + */ + if (pfn + (1 << buddy_order(page)) > start_pfn) + return pfn; + + /* Nothing found */ + return start_pfn; +} + +/* Split a multi-block free page into its individual pageblocks */ +static void split_large_buddy(struct zone *zone, struct page *page, + unsigned long pfn, int order) +{ + unsigned long end_pfn = pfn + (1 << order); + + VM_WARN_ON_ONCE(order <= pageblock_order); + VM_WARN_ON_ONCE(pfn & (pageblock_nr_pages - 1)); + + /* Caller removed page from freelist, buddy info cleared! */ + VM_WARN_ON_ONCE(PageBuddy(page)); + + while (pfn != end_pfn) { + int mt = get_pfnblock_migratetype(page, pfn); + + __free_one_page(page, pfn, zone, pageblock_order, mt, FPI_NONE); + pfn += pageblock_nr_pages; + page = pfn_to_page(pfn); + } +} + +/** + * move_freepages_block_isolate - move free pages in block for page isolation + * @zone: the zone + * @page: the pageblock page + * @migratetype: migratetype to set on the pageblock + * + * This is similar to move_freepages_block(), but handles the special + * case encountered in page isolation, where the block of interest + * might be part of a larger buddy spanning multiple pageblocks. + * + * Unlike the regular page allocator path, which moves pages while + * stealing buddies off the freelist, page isolation is interested in + * arbitrary pfn ranges that may have overlapping buddies on both ends. + * + * This function handles that. Straddling buddies are split into + * individual pageblocks. Only the block of interest is moved. + * + * Returns %true if pages could be moved, %false otherwise. + */ +bool move_freepages_block_isolate(struct zone *zone, struct page *page, + int migratetype) +{ + unsigned long start_pfn, end_pfn, pfn; + int nr_moved, mt; + + if (!prep_move_freepages_block(zone, page, &start_pfn, &end_pfn, + NULL, NULL)) + return false; + + /* We're a tail block in a larger buddy */ + pfn = find_large_buddy(start_pfn); + if (pfn != start_pfn) { + struct page *buddy = pfn_to_page(pfn); + int order = buddy_order(buddy); + int mt = get_pfnblock_migratetype(buddy, pfn); + + if (!is_migrate_isolate(mt)) + __mod_zone_freepage_state(zone, -(1UL << order), mt); + del_page_from_free_list(buddy, zone, order); + set_pageblock_migratetype(page, migratetype); + split_large_buddy(zone, buddy, pfn, order); + return true; + } + + /* We're the starting block of a larger buddy */ + if (PageBuddy(page) && buddy_order(page) > pageblock_order) { + int mt = get_pfnblock_migratetype(page, pfn); + int order = buddy_order(page); + + if (!is_migrate_isolate(mt)) + __mod_zone_freepage_state(zone, -(1UL << order), mt); + del_page_from_free_list(page, zone, order); + set_pageblock_migratetype(page, migratetype); + split_large_buddy(zone, page, pfn, order); + return true; + } + + mt = get_pfnblock_migratetype(page, start_pfn); + nr_moved = move_freepages(zone, start_pfn, end_pfn, migratetype); + if (!is_migrate_isolate(mt)) + __mod_zone_freepage_state(zone, -nr_moved, mt); + else if (!is_migrate_isolate(migratetype)) + __mod_zone_freepage_state(zone, nr_moved, migratetype); + return true; +} +#endif /* CONFIG_MEMORY_ISOLATION */ + static void change_pageblock_range(struct page *pageblock_page, int start_order, int migratetype) { @@ -6390,7 +6445,6 @@ int alloc_contig_range(unsigned long start, unsigned long end, unsigned migratetype, gfp_t gfp_mask) { unsigned long outer_start, outer_end; - int order; int ret = 0; struct compact_control cc = { @@ -6463,29 +6517,7 @@ int alloc_contig_range(unsigned long start, unsigned long end, * We don't have to hold zone->lock here because the pages are * isolated thus they won't get removed from buddy. */ - - order = 0; - outer_start = start; - while (!PageBuddy(pfn_to_page(outer_start))) { - if (++order > MAX_PAGE_ORDER) { - outer_start = start; - break; - } - outer_start &= ~0UL << order; - } - - if (outer_start != start) { - order = buddy_order(pfn_to_page(outer_start)); - - /* - * outer_start page could be small order buddy page and - * it doesn't include start page. Adjust outer_start - * in this case to report failed page properly - * on tracepoint in test_pages_isolated() - */ - if (outer_start + (1UL << order) <= start) - outer_start = start; - } + outer_start = find_large_buddy(start); /* Make sure the range is really isolated. */ if (test_pages_isolated(outer_start, end, 0)) { diff --git a/mm/page_isolation.c b/mm/page_isolation.c index f84f0981b2df..042937d5abe4 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -178,16 +178,10 @@ static int set_migratetype_isolate(struct page *page, int migratetype, int isol_ unmovable = has_unmovable_pages(check_unmovable_start, check_unmovable_end, migratetype, isol_flags); if (!unmovable) { - int nr_pages; - int mt = get_pageblock_migratetype(page); - - nr_pages = move_freepages_block(zone, page, MIGRATE_ISOLATE); - /* Block spans zone boundaries? */ - if (nr_pages == -1) { + if (!move_freepages_block_isolate(zone, page, MIGRATE_ISOLATE)) { spin_unlock_irqrestore(&zone->lock, flags); return -EBUSY; } - __mod_zone_freepage_state(zone, -nr_pages, mt); zone->nr_isolate_pageblock++; spin_unlock_irqrestore(&zone->lock, flags); return 0; @@ -254,13 +248,11 @@ static void unset_migratetype_isolate(struct page *page, int migratetype) * allocation. */ if (!isolated_page) { - int nr_pages = move_freepages_block(zone, page, migratetype); /* * Isolating this block already succeeded, so this * should not fail on zone boundaries. */ - WARN_ON_ONCE(nr_pages == -1); - __mod_zone_freepage_state(zone, nr_pages, migratetype); + WARN_ON_ONCE(!move_freepages_block_isolate(zone, page, migratetype)); } else { set_pageblock_migratetype(page, migratetype); __putback_isolated_page(page, order, migratetype); @@ -374,26 +366,29 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, VM_BUG_ON(!page); pfn = page_to_pfn(page); - /* - * start_pfn is MAX_ORDER_NR_PAGES aligned, if there is any - * free pages in [start_pfn, boundary_pfn), its head page will - * always be in the range. - */ + if (PageBuddy(page)) { int order = buddy_order(page); - if (pfn + (1UL << order) > boundary_pfn) { - /* free page changed before split, check it again */ - if (split_free_page(page, order, boundary_pfn - pfn)) - continue; - } + /* move_freepages_block_isolate() handled this */ + VM_WARN_ON_ONCE(pfn + (1 << order) > boundary_pfn); pfn += 1UL << order; continue; } + /* - * migrate compound pages then let the free page handling code - * above do the rest. If migration is not possible, just fail. + * If a compound page is straddling our block, attempt + * to migrate it out of the way. + * + * We don't have to worry about this creating a large + * free page that straddles into our block: gigantic + * pages are freed as order-0 chunks, and LRU pages + * (currently) do not exceed pageblock_order. + * + * The block of interest has already been marked + * MIGRATE_ISOLATE above, so when migration is done it + * will free its pages onto the correct freelists. */ if (PageCompound(page)) { struct page *head = compound_head(page); @@ -404,16 +399,10 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, pfn = head_pfn + nr_pages; continue; } + #if defined CONFIG_COMPACTION || defined CONFIG_CMA - /* - * hugetlb, lru compound (THP), and movable compound pages - * can be migrated. Otherwise, fail the isolation. - */ - if (PageHuge(page) || PageLRU(page) || __PageMovable(page)) { - int order; - unsigned long outer_pfn; + if (PageHuge(page)) { int page_mt = get_pageblock_migratetype(page); - bool isolate_page = !is_migrate_isolate_page(page); struct compact_control cc = { .nr_migratepages = 0, .order = -1, @@ -426,56 +415,25 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, }; INIT_LIST_HEAD(&cc.migratepages); - /* - * XXX: mark the page as MIGRATE_ISOLATE so that - * no one else can grab the freed page after migration. - * Ideally, the page should be freed as two separate - * pages to be added into separate migratetype free - * lists. - */ - if (isolate_page) { - ret = set_migratetype_isolate(page, page_mt, - flags, head_pfn, head_pfn + nr_pages); - if (ret) - goto failed; - } - ret = __alloc_contig_migrate_range(&cc, head_pfn, head_pfn + nr_pages, page_mt); - - /* - * restore the page's migratetype so that it can - * be split into separate migratetype free lists - * later. - */ - if (isolate_page) - unset_migratetype_isolate(page, page_mt); - if (ret) goto failed; - /* - * reset pfn to the head of the free page, so - * that the free page handling code above can split - * the free page to the right migratetype list. - * - * head_pfn is not used here as a hugetlb page order - * can be bigger than MAX_PAGE_ORDER, but after it is - * freed, the free page order is not. Use pfn within - * the range to find the head of the free page. - */ - order = 0; - outer_pfn = pfn; - while (!PageBuddy(pfn_to_page(outer_pfn))) { - /* stop if we cannot find the free page */ - if (++order > MAX_PAGE_ORDER) - goto failed; - outer_pfn &= ~0UL << order; - } - pfn = outer_pfn; + pfn = head_pfn + nr_pages; continue; - } else + } + + /* + * These pages are movable too, but they're + * not expected to exceed pageblock_order. + * + * Let us know when they do, so we can add + * proper free and split handling for them. + */ + VM_WARN_ON_ONCE_PAGE(PageLRU(page), page); + VM_WARN_ON_ONCE_PAGE(__PageMovable(page), page); #endif - goto failed; + goto failed; } pfn++; From patchwork Wed Mar 20 18:02:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13598070 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 048A9C54E58 for ; Wed, 20 Mar 2024 18:05:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 921E36B009E; Wed, 20 Mar 2024 14:05:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8A8D06B00A0; Wed, 20 Mar 2024 14:05:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D2D36B00A1; Wed, 20 Mar 2024 14:05:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 516266B009E for ; Wed, 20 Mar 2024 14:05:09 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A0E8FC023D for ; Wed, 20 Mar 2024 18:05:08 +0000 (UTC) X-FDA: 81918193896.17.7A61280 Received: from mail-qv1-f50.google.com (mail-qv1-f50.google.com [209.85.219.50]) by imf18.hostedemail.com (Postfix) with ESMTP id B433D1C001C for ; Wed, 20 Mar 2024 18:05:06 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=PLCTIjnH; spf=pass (imf18.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.50 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710957906; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7usfWeQLN0+E1Z1UvH5iS/0eHw0DMUF9yV4ujEabP2Q=; b=Hv/NvCUYCEQYqvnVV7JYa4W1dbUZwC7VcakgX3ylxqpAcD2Yfu5Wjx+d7DVGduVlSKepBK weXI1sDYAmxlwi4uFDwNvQRSDbOXKBHuPIR9lSWMwYEQfV0QpKwUSEyoBo948N8XdPqzdQ FaU/g2QlfYiClBpvnEJSI4ZM+cHcRzA= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=PLCTIjnH; spf=pass (imf18.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.50 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710957906; a=rsa-sha256; cv=none; b=f/G0EJfiq+z204fnPDw0NES+dWH2j05Wufokp3j/+N+IyZ9HTWs877oeGoTG3XzqATOJE5 46VLjH7FKsaWMqWNaH2ERhyzKVO7zAistFa3zGzTjmEV4/CLPUzPWlRgn7bwVLVZ+xm4Bd /6QYqg4MIo+UJvfWkWbRS/hZeB6+txY= Received: by mail-qv1-f50.google.com with SMTP id 6a1803df08f44-690cffa8a0bso1145456d6.0 for ; Wed, 20 Mar 2024 11:05:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1710957906; x=1711562706; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7usfWeQLN0+E1Z1UvH5iS/0eHw0DMUF9yV4ujEabP2Q=; b=PLCTIjnHGSYeYHZ5FUdWSipqXjn2t274tdJ/MJcwnyqhFho/T2f68MxKioYo9CCHFl KVKebXPzF5kHyW62tgKUReeg5IuWGotgVNxBcB5Wx04tSMw70vKZnj9t5KeHdQ0GKjiA lGzG+pVgyuQ2sGCqQ6JWaxedH2qqfrAaEyVUY9lmuYGmXxuV3UuFhDvJGLdPTOgImnxa 3qBq9gosgDYEqBNcqLmSlQdxmKggfFmRJ1ulg6q3yvF6AOtFhDE5FZiA1FPQX88l8++y aJwUO4ml+VxNuMIGJ723QWKoJTwnoJPXTJDvbrZTdA8LzLdX6PSaYU/giT4a5TDJ/SpK wpSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710957906; x=1711562706; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7usfWeQLN0+E1Z1UvH5iS/0eHw0DMUF9yV4ujEabP2Q=; b=lVlrbDqLVViqS5qS+p6PBekwyhSV2XufrYjjmCOeDhhOhmwLUgiLTVXDRGv8HFvqvS 0LA+IZG1tNJxsUAeKVAbZWDbFTcji6aoPjogXmlR+VI83oUvk4G9tlQTqvTW37BB2Z2e 2bSGgvKDkSa7LxOS98x3xvHJ2+pSd4APOWxFNqdUGXM8qujbHLeUrFQhVHhFGcEbxu24 RELiT3yrlihG64V/ZoClzhQMdCurSOgI9unOSYU1573DokJdQ0e97Dn21srpk/0uJPJn Gbp1TDl2IibRo7Wl5c9UghfYsBKme09dkyehKa8L7PN74KtsKmGv98T575Cn/y3EdeMF rrNw== X-Forwarded-Encrypted: i=1; AJvYcCULznA2NoEUHC1GH+A31KR43qNOf6p0jD0fPWerjduJLC3tMKrYnK5F6dB0dg3kh1NJfvFMrQa4q7eYYjyAjfnFwDo= X-Gm-Message-State: AOJu0YwjXCNJ9zp0xUXuj5kjFYZuphGbncveNPSPdk6WZ6oyplLINdd6 PvkuV7Ng/noLEG1JdVaQxsvC+YIjScto7/e8KDZpT1gqg24z+YsA8l6P/HmuFz0= X-Google-Smtp-Source: AGHT+IHMjYEE88Dv6WAVJtdVPq3oYrOoMOFbiUGH9pcBJQM3z1kAVXkhzhyLSrzGgEmTgy1mssmPRA== X-Received: by 2002:a0c:c20b:0:b0:696:2b3e:7813 with SMTP id l11-20020a0cc20b000000b006962b3e7813mr8058795qvh.17.1710957905785; Wed, 20 Mar 2024 11:05:05 -0700 (PDT) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id g15-20020a0562140acf00b0069150ebcc30sm8045119qvi.76.2024.03.20.11.05.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Mar 2024 11:05:05 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: Vlastimil Babka , Mel Gorman , Zi Yan , "Huang, Ying" , David Hildenbrand , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 10/10] mm: page_alloc: consolidate free page accounting Date: Wed, 20 Mar 2024 14:02:15 -0400 Message-ID: <20240320180429.678181-11-hannes@cmpxchg.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240320180429.678181-1-hannes@cmpxchg.org> References: <20240320180429.678181-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: B433D1C001C X-Rspam-User: X-Stat-Signature: imw33xxz78dj7d563ydj8ci1fnyfg4if X-Rspamd-Server: rspam01 X-HE-Tag: 1710957906-526447 X-HE-Meta: U2FsdGVkX18CEZyrqHux7fineA55yejtXEqz2Pmqs+Pn/MW8J2uyDFBGVcHXal8aV0rtu5GJn5A3F36w3IEYNr/oBIUW54IVQ0MU5HQbhQ+7yJ85kYV+bduEL/Q8FfkPVumAD8Vx9c3xe02ze4//oMhW5lEkGT5egx0TTgI9BY+0FXaJjcrn/YcmnPWDEX5Uha7MJPb8zUFrc1akwZ5OLlgudzL28i75LEA527AiWq6t+WyTVi3UxxU1o/5dqj0qMJlWAyJBt+LqndlC5MTD/62RASwZ2dmPb3FfWy68iTXMgYmPB0iuu8UK84gEukpd9kmHXSk6LLCYtOEt63VyR2vvVP+QKJfee+EGK4obFlDSPGd13fNhcBDprntteCgZIRZcmNFJBBs06Zh9f4nQe3koX1TCxG33V41p+9kEaXIO7rbMBwWBlIxFxFdjQyOahasf1REQHCYxITfNxyvPT+8CC/ZFZKqLzb5EJuStUXrcoXMK/LuHLHg1PKVgQ8a6OvF8m6qxd5dDVWOt0RZmbBBiEVnekH23Dl/WnhfTOKWhQiyzVXoDpiUElYjOgwf6UL1+rruMFVyeg0itMbSLRN4DRbhoxC9nvqeJ/Q3CpONWsPvjazX3apHoRmDYvGLS2aEP6n9cJnfEl3kCBjmREWVvUda23eL5ZkDBGkKt6xMbBN7Y/eB4qdnj6Hoj68ET9ihGCI+xnlRc3k0MGxhjn4SlwA9YgXfMqngerDAWjd5RGnico+Z7TqHR7EUD2nUmkWrQoQRYBw61QSzbtuQv1/gyEyXQfnOZRgLsfwe/WG7tG2V2N50xk2EAVO96lX7Mu6UerF9nuKEzOLOsumYNxnq7CDmIA7E1SyKz27QuaL2S/kHyfMe1SyGlBkF1mURSDZMYSfdNBCU8wHDLHbcZuock9276si69hc0iq7GICwCYbSlvGRjN9Fkz1+vHRKR+gM1hI+7v0QAjJjQ3nsD Nx0bVO4n 18mAqAUbaMVAAtofOhksZUcUnmEEfFGJyvKXbyLsKlsD7Jo+1HD7M8vaw1okZpjiBBm9NDFsi8N82mpAQCUVIUbIBPzLzMqIk6Pfi0myxtVDZKt39jL3cqGOs2gMg05h9ZofVxwUK1aE08PKSIZXSiQDYwr99M/O/YKiFG/TznID55rCr7+XF5/V/Jc+7obrPcezoI4gld3VNDgKkrKjO9BfZy6Vn5UjKQsaT1bzQ17yBJCdRkSmYhegstTlj4W5jyvl6ScDWolOs4kRKde9Q2xV4yIG8lnc3x7m67mLM55ea3AD3Y5uAo1Eg8Djm7D855tdagend564vVmD5Fao5IRtLJE1pjbe/z8Wecf0/4SU6nOdducT8ZW+AaE2U8Q29utRzD+suAKL+wkueucPXu5sSKWpWEkEpRBKrf0AZ2wQg9OIk0mCs8f1T8Jh5KgFy4ajAVb0/s/ibXswsMBvKikyrvu8E/vrnylr3UzPG7zIr6PU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Free page accounting currently happens a bit too high up the call stack, where it has to deal with guard pages, compaction capturing, block stealing and even page isolation. This is subtle and fragile, and makes it difficult to hack on the code. Now that type violations on the freelists have been fixed, push the accounting down to where pages enter and leave the freelist. Signed-off-by: Johannes Weiner Reviewed-by: Vlastimil Babka Acked-by: Johannes Weiner Acked-by: Vlastimil Babka --- include/linux/mm.h | 18 ++-- include/linux/vmstat.h | 8 -- mm/debug_page_alloc.c | 12 +-- mm/internal.h | 5 -- mm/page_alloc.c | 194 +++++++++++++++++++++++------------------ mm/page_isolation.c | 3 +- 6 files changed, 120 insertions(+), 120 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 8147b1302413..bd2e94391c7e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3781,24 +3781,22 @@ static inline bool page_is_guard(struct page *page) return PageGuard(page); } -bool __set_page_guard(struct zone *zone, struct page *page, unsigned int order, - int migratetype); +bool __set_page_guard(struct zone *zone, struct page *page, unsigned int order); static inline bool set_page_guard(struct zone *zone, struct page *page, - unsigned int order, int migratetype) + unsigned int order) { if (!debug_guardpage_enabled()) return false; - return __set_page_guard(zone, page, order, migratetype); + return __set_page_guard(zone, page, order); } -void __clear_page_guard(struct zone *zone, struct page *page, unsigned int order, - int migratetype); +void __clear_page_guard(struct zone *zone, struct page *page, unsigned int order); static inline void clear_page_guard(struct zone *zone, struct page *page, - unsigned int order, int migratetype) + unsigned int order) { if (!debug_guardpage_enabled()) return; - __clear_page_guard(zone, page, order, migratetype); + __clear_page_guard(zone, page, order); } #else /* CONFIG_DEBUG_PAGEALLOC */ @@ -3808,9 +3806,9 @@ static inline unsigned int debug_guardpage_minorder(void) { return 0; } static inline bool debug_guardpage_enabled(void) { return false; } static inline bool page_is_guard(struct page *page) { return false; } static inline bool set_page_guard(struct zone *zone, struct page *page, - unsigned int order, int migratetype) { return false; } + unsigned int order) { return false; } static inline void clear_page_guard(struct zone *zone, struct page *page, - unsigned int order, int migratetype) {} + unsigned int order) {} #endif /* CONFIG_DEBUG_PAGEALLOC */ #ifdef __HAVE_ARCH_GATE_AREA diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index 343906a98d6e..735eae6e272c 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -487,14 +487,6 @@ static inline void node_stat_sub_folio(struct folio *folio, mod_node_page_state(folio_pgdat(folio), item, -folio_nr_pages(folio)); } -static inline void __mod_zone_freepage_state(struct zone *zone, int nr_pages, - int migratetype) -{ - __mod_zone_page_state(zone, NR_FREE_PAGES, nr_pages); - if (is_migrate_cma(migratetype)) - __mod_zone_page_state(zone, NR_FREE_CMA_PAGES, nr_pages); -} - extern const char * const vmstat_text[]; static inline const char *zone_stat_name(enum zone_stat_item item) diff --git a/mm/debug_page_alloc.c b/mm/debug_page_alloc.c index 6755f0c9d4a3..d46acf989dde 100644 --- a/mm/debug_page_alloc.c +++ b/mm/debug_page_alloc.c @@ -32,8 +32,7 @@ static int __init debug_guardpage_minorder_setup(char *buf) } early_param("debug_guardpage_minorder", debug_guardpage_minorder_setup); -bool __set_page_guard(struct zone *zone, struct page *page, unsigned int order, - int migratetype) +bool __set_page_guard(struct zone *zone, struct page *page, unsigned int order) { if (order >= debug_guardpage_minorder()) return false; @@ -41,19 +40,12 @@ bool __set_page_guard(struct zone *zone, struct page *page, unsigned int order, __SetPageGuard(page); INIT_LIST_HEAD(&page->buddy_list); set_page_private(page, order); - /* Guard pages are not available for any usage */ - if (!is_migrate_isolate(migratetype)) - __mod_zone_freepage_state(zone, -(1 << order), migratetype); return true; } -void __clear_page_guard(struct zone *zone, struct page *page, unsigned int order, - int migratetype) +void __clear_page_guard(struct zone *zone, struct page *page, unsigned int order) { __ClearPageGuard(page); - set_page_private(page, 0); - if (!is_migrate_isolate(migratetype)) - __mod_zone_freepage_state(zone, (1 << order), migratetype); } diff --git a/mm/internal.h b/mm/internal.h index d6e6c7d9f04e..0a4007b03d0d 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1036,11 +1036,6 @@ static inline bool is_migrate_highatomic(enum migratetype migratetype) return migratetype == MIGRATE_HIGHATOMIC; } -static inline bool is_migrate_highatomic_page(struct page *page) -{ - return get_pageblock_migratetype(page) == MIGRATE_HIGHATOMIC; -} - void setup_zone_pageset(struct zone *zone); struct migration_target_control { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index efb2581ac142..c46491f83ac2 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -642,42 +642,72 @@ compaction_capture(struct capture_control *capc, struct page *page, } #endif /* CONFIG_COMPACTION */ -/* Used for pages not on another list */ -static inline void add_to_free_list(struct page *page, struct zone *zone, - unsigned int order, int migratetype) +static inline void account_freepages(struct page *page, struct zone *zone, + int nr_pages, int migratetype) { - struct free_area *area = &zone->free_area[order]; + if (is_migrate_isolate(migratetype)) + return; - list_add(&page->buddy_list, &area->free_list[migratetype]); - area->nr_free++; + __mod_zone_page_state(zone, NR_FREE_PAGES, nr_pages); + + if (is_migrate_cma(migratetype)) + __mod_zone_page_state(zone, NR_FREE_CMA_PAGES, nr_pages); } /* Used for pages not on another list */ -static inline void add_to_free_list_tail(struct page *page, struct zone *zone, - unsigned int order, int migratetype) +static inline void __add_to_free_list(struct page *page, struct zone *zone, + unsigned int order, int migratetype, + bool tail) { struct free_area *area = &zone->free_area[order]; - list_add_tail(&page->buddy_list, &area->free_list[migratetype]); + VM_WARN_ONCE(get_pageblock_migratetype(page) != migratetype, + "page type is %lu, passed migratetype is %d (nr=%d)\n", + get_pageblock_migratetype(page), migratetype, 1 << order); + + if (tail) + list_add_tail(&page->buddy_list, &area->free_list[migratetype]); + else + list_add(&page->buddy_list, &area->free_list[migratetype]); area->nr_free++; } +static inline void add_to_free_list(struct page *page, struct zone *zone, + unsigned int order, int migratetype, + bool tail) +{ + __add_to_free_list(page, zone, order, migratetype, tail); + account_freepages(page, zone, 1 << order, migratetype); +} + /* * Used for pages which are on another list. Move the pages to the tail * of the list - so the moved pages won't immediately be considered for * allocation again (e.g., optimization for memory onlining). */ static inline void move_to_free_list(struct page *page, struct zone *zone, - unsigned int order, int migratetype) + unsigned int order, int old_mt, int new_mt) { struct free_area *area = &zone->free_area[order]; - list_move_tail(&page->buddy_list, &area->free_list[migratetype]); + /* Free page moving can fail, so it happens before the type update */ + VM_WARN_ONCE(get_pageblock_migratetype(page) != old_mt, + "page type is %lu, passed migratetype is %d (nr=%d)\n", + get_pageblock_migratetype(page), old_mt, 1 << order); + + list_move_tail(&page->buddy_list, &area->free_list[new_mt]); + + account_freepages(page, zone, -(1 << order), old_mt); + account_freepages(page, zone, 1 << order, new_mt); } -static inline void del_page_from_free_list(struct page *page, struct zone *zone, - unsigned int order) +static inline void __del_page_from_free_list(struct page *page, struct zone *zone, + unsigned int order, int migratetype) { + VM_WARN_ONCE(get_pageblock_migratetype(page) != migratetype, + "page type is %lu, passed migratetype is %d (nr=%d)\n", + get_pageblock_migratetype(page), migratetype, 1 << order); + /* clear reported state and update reported page count */ if (page_reported(page)) __ClearPageReported(page); @@ -688,6 +718,13 @@ static inline void del_page_from_free_list(struct page *page, struct zone *zone, zone->free_area[order].nr_free--; } +static inline void del_page_from_free_list(struct page *page, struct zone *zone, + unsigned int order, int migratetype) +{ + __del_page_from_free_list(page, zone, order, migratetype); + account_freepages(page, zone, -(1 << order), migratetype); +} + static inline struct page *get_page_from_free_area(struct free_area *area, int migratetype) { @@ -759,18 +796,16 @@ static inline void __free_one_page(struct page *page, VM_BUG_ON_PAGE(page->flags & PAGE_FLAGS_CHECK_AT_PREP, page); VM_BUG_ON(migratetype == -1); - if (likely(!is_migrate_isolate(migratetype))) - __mod_zone_freepage_state(zone, 1 << order, migratetype); - VM_BUG_ON_PAGE(pfn & ((1 << order) - 1), page); VM_BUG_ON_PAGE(bad_range(zone, page), page); + account_freepages(page, zone, 1 << order, migratetype); + while (order < MAX_PAGE_ORDER) { - if (compaction_capture(capc, page, order, migratetype)) { - __mod_zone_freepage_state(zone, -(1 << order), - migratetype); + int buddy_mt = migratetype; + + if (compaction_capture(capc, page, order, migratetype)) return; - } buddy = find_buddy_page_pfn(page, pfn, order, &buddy_pfn); if (!buddy) @@ -783,19 +818,12 @@ static inline void __free_one_page(struct page *page, * pageblock isolation could cause incorrect freepage or CMA * accounting or HIGHATOMIC accounting. */ - int buddy_mt = get_pfnblock_migratetype(buddy, buddy_pfn); + buddy_mt = get_pfnblock_migratetype(buddy, buddy_pfn); - if (migratetype != buddy_mt) { - if (!migratetype_is_mergeable(migratetype) || - !migratetype_is_mergeable(buddy_mt)) - goto done_merging; - /* - * Match buddy type. This ensures that - * an expand() down the line puts the - * sub-blocks on the right freelists. - */ - set_pageblock_migratetype(buddy, migratetype); - } + if (migratetype != buddy_mt && + (!migratetype_is_mergeable(migratetype) || + !migratetype_is_mergeable(buddy_mt))) + goto done_merging; } /* @@ -803,9 +831,19 @@ static inline void __free_one_page(struct page *page, * merge with it and move up one order. */ if (page_is_guard(buddy)) - clear_page_guard(zone, buddy, order, migratetype); + clear_page_guard(zone, buddy, order); else - del_page_from_free_list(buddy, zone, order); + __del_page_from_free_list(buddy, zone, order, buddy_mt); + + if (unlikely(buddy_mt != migratetype)) { + /* + * Match buddy type. This ensures that an + * expand() down the line puts the sub-blocks + * on the right freelists. + */ + set_pageblock_migratetype(buddy, migratetype); + } + combined_pfn = buddy_pfn & pfn; page = page + (combined_pfn - pfn); pfn = combined_pfn; @@ -822,10 +860,7 @@ static inline void __free_one_page(struct page *page, else to_tail = buddy_merge_likely(pfn, buddy_pfn, page, order); - if (to_tail) - add_to_free_list_tail(page, zone, order, migratetype); - else - add_to_free_list(page, zone, order, migratetype); + __add_to_free_list(page, zone, order, migratetype, to_tail); /* Notify page reporting subsystem of freed page */ if (!(fpi_flags & FPI_SKIP_REPORT_NOTIFY)) @@ -1314,10 +1349,10 @@ static inline void expand(struct zone *zone, struct page *page, * Corresponding page table entries will not be touched, * pages will stay not present in virtual address space */ - if (set_page_guard(zone, &page[size], high, migratetype)) + if (set_page_guard(zone, &page[size], high)) continue; - add_to_free_list(&page[size], zone, high, migratetype); + add_to_free_list(&page[size], zone, high, migratetype, false); set_buddy_order(&page[size], high); } } @@ -1487,7 +1522,7 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order, page = get_page_from_free_area(area, migratetype); if (!page) continue; - del_page_from_free_list(page, zone, current_order); + del_page_from_free_list(page, zone, current_order, migratetype); expand(zone, page, order, current_order, migratetype); trace_mm_page_alloc_zone_locked(page, order, migratetype, pcp_allowed_order(order) && @@ -1527,7 +1562,7 @@ static inline struct page *__rmqueue_cma_fallback(struct zone *zone, * type's freelist. */ static int move_freepages(struct zone *zone, unsigned long start_pfn, - unsigned long end_pfn, int migratetype) + unsigned long end_pfn, int old_mt, int new_mt) { struct page *page; unsigned long pfn; @@ -1549,12 +1584,14 @@ static int move_freepages(struct zone *zone, unsigned long start_pfn, VM_BUG_ON_PAGE(page_zone(page) != zone, page); order = buddy_order(page); - move_to_free_list(page, zone, order, migratetype); + + move_to_free_list(page, zone, order, old_mt, new_mt); + pfn += 1 << order; pages_moved += 1 << order; } - set_pageblock_migratetype(pfn_to_page(start_pfn), migratetype); + set_pageblock_migratetype(pfn_to_page(start_pfn), new_mt); return pages_moved; } @@ -1612,7 +1649,7 @@ static bool prep_move_freepages_block(struct zone *zone, struct page *page, } static int move_freepages_block(struct zone *zone, struct page *page, - int migratetype) + int old_mt, int new_mt) { unsigned long start_pfn, end_pfn; @@ -1620,7 +1657,7 @@ static int move_freepages_block(struct zone *zone, struct page *page, NULL, NULL)) return -1; - return move_freepages(zone, start_pfn, end_pfn, migratetype); + return move_freepages(zone, start_pfn, end_pfn, old_mt, new_mt); } #ifdef CONFIG_MEMORY_ISOLATION @@ -1692,7 +1729,6 @@ bool move_freepages_block_isolate(struct zone *zone, struct page *page, int migratetype) { unsigned long start_pfn, end_pfn, pfn; - int nr_moved, mt; if (!prep_move_freepages_block(zone, page, &start_pfn, &end_pfn, NULL, NULL)) @@ -1703,11 +1739,9 @@ bool move_freepages_block_isolate(struct zone *zone, struct page *page, if (pfn != start_pfn) { struct page *buddy = pfn_to_page(pfn); int order = buddy_order(buddy); - int mt = get_pfnblock_migratetype(buddy, pfn); - if (!is_migrate_isolate(mt)) - __mod_zone_freepage_state(zone, -(1UL << order), mt); - del_page_from_free_list(buddy, zone, order); + del_page_from_free_list(buddy, zone, order, + get_pfnblock_migratetype(buddy, pfn)); set_pageblock_migratetype(page, migratetype); split_large_buddy(zone, buddy, pfn, order); return true; @@ -1715,23 +1749,17 @@ bool move_freepages_block_isolate(struct zone *zone, struct page *page, /* We're the starting block of a larger buddy */ if (PageBuddy(page) && buddy_order(page) > pageblock_order) { - int mt = get_pfnblock_migratetype(page, pfn); int order = buddy_order(page); - if (!is_migrate_isolate(mt)) - __mod_zone_freepage_state(zone, -(1UL << order), mt); - del_page_from_free_list(page, zone, order); + del_page_from_free_list(page, zone, order, + get_pfnblock_migratetype(page, pfn)); set_pageblock_migratetype(page, migratetype); split_large_buddy(zone, page, pfn, order); return true; } - mt = get_pfnblock_migratetype(page, start_pfn); - nr_moved = move_freepages(zone, start_pfn, end_pfn, migratetype); - if (!is_migrate_isolate(mt)) - __mod_zone_freepage_state(zone, -nr_moved, mt); - else if (!is_migrate_isolate(migratetype)) - __mod_zone_freepage_state(zone, nr_moved, migratetype); + move_freepages(zone, start_pfn, end_pfn, + get_pfnblock_migratetype(page, start_pfn), migratetype); return true; } #endif /* CONFIG_MEMORY_ISOLATION */ @@ -1845,7 +1873,7 @@ steal_suitable_fallback(struct zone *zone, struct page *page, /* Take ownership for orders >= pageblock_order */ if (current_order >= pageblock_order) { - del_page_from_free_list(page, zone, current_order); + del_page_from_free_list(page, zone, current_order, block_type); change_pageblock_range(page, current_order, start_type); expand(zone, page, order, current_order, start_type); return page; @@ -1895,12 +1923,12 @@ steal_suitable_fallback(struct zone *zone, struct page *page, */ if (free_pages + alike_pages >= (1 << (pageblock_order-1)) || page_group_by_mobility_disabled) { - move_freepages(zone, start_pfn, end_pfn, start_type); + move_freepages(zone, start_pfn, end_pfn, block_type, start_type); return __rmqueue_smallest(zone, order, start_type); } single_page: - del_page_from_free_list(page, zone, current_order); + del_page_from_free_list(page, zone, current_order, block_type); expand(zone, page, order, current_order, block_type); return page; } @@ -1970,7 +1998,7 @@ static void reserve_highatomic_pageblock(struct page *page, struct zone *zone) mt = get_pageblock_migratetype(page); /* Only reserve normal pageblocks (i.e., they can merge with others) */ if (migratetype_is_mergeable(mt)) - if (move_freepages_block(zone, page, + if (move_freepages_block(zone, page, mt, MIGRATE_HIGHATOMIC) != -1) zone->nr_reserved_highatomic += pageblock_nr_pages; @@ -2011,11 +2039,13 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, spin_lock_irqsave(&zone->lock, flags); for (order = 0; order < NR_PAGE_ORDERS; order++) { struct free_area *area = &(zone->free_area[order]); + int mt; page = get_page_from_free_area(area, MIGRATE_HIGHATOMIC); if (!page) continue; + mt = get_pageblock_migratetype(page); /* * In page freeing path, migratetype change is racy so * we can counter several free pages in a pageblock @@ -2023,7 +2053,7 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, * from highatomic to ac->migratetype. So we should * adjust the count once. */ - if (is_migrate_highatomic_page(page)) { + if (is_migrate_highatomic(mt)) { /* * It should never happen but changes to * locking could inadvertently allow a per-cpu @@ -2045,7 +2075,8 @@ static bool unreserve_highatomic_pageblock(const struct alloc_context *ac, * of pageblocks that cannot be completely freed * may increase. */ - ret = move_freepages_block(zone, page, ac->migratetype); + ret = move_freepages_block(zone, page, mt, + ac->migratetype); /* * Reserving this block already succeeded, so this should * not fail on zone boundaries. @@ -2251,12 +2282,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, * pages are ordered properly. */ list_add_tail(&page->pcp_list, list); - if (is_migrate_cma(get_pageblock_migratetype(page))) - __mod_zone_page_state(zone, NR_FREE_CMA_PAGES, - -(1 << order)); } - - __mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order)); spin_unlock_irqrestore(&zone->lock, flags); return i; @@ -2748,11 +2774,9 @@ int __isolate_free_page(struct page *page, unsigned int order) watermark = zone->_watermark[WMARK_MIN] + (1UL << order); if (!zone_watermark_ok(zone, 0, watermark, 0, ALLOC_CMA)) return 0; - - __mod_zone_freepage_state(zone, -(1UL << order), mt); } - del_page_from_free_list(page, zone, order); + del_page_from_free_list(page, zone, order, mt); /* * Set the pageblock if the isolated page is at least half of a @@ -2767,7 +2791,7 @@ int __isolate_free_page(struct page *page, unsigned int order) * with others) */ if (migratetype_is_mergeable(mt)) - move_freepages_block(zone, page, + move_freepages_block(zone, page, mt, MIGRATE_MOVABLE); } } @@ -2852,8 +2876,6 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, return NULL; } } - __mod_zone_freepage_state(zone, -(1 << order), - get_pageblock_migratetype(page)); spin_unlock_irqrestore(&zone->lock, flags); } while (check_new_pages(page, order)); @@ -6737,8 +6759,9 @@ void __offline_isolated_pages(unsigned long start_pfn, unsigned long end_pfn) BUG_ON(page_count(page)); BUG_ON(!PageBuddy(page)); + VM_WARN_ON(get_pageblock_migratetype(page) != MIGRATE_ISOLATE); order = buddy_order(page); - del_page_from_free_list(page, zone, order); + del_page_from_free_list(page, zone, order, MIGRATE_ISOLATE); pfn += (1 << order); } spin_unlock_irqrestore(&zone->lock, flags); @@ -6788,10 +6811,10 @@ static void break_down_buddy_pages(struct zone *zone, struct page *page, current_buddy = page + size; } - if (set_page_guard(zone, current_buddy, high, migratetype)) + if (set_page_guard(zone, current_buddy, high)) continue; - add_to_free_list(current_buddy, zone, high, migratetype); + add_to_free_list(current_buddy, zone, high, migratetype, false); set_buddy_order(current_buddy, high); } } @@ -6817,12 +6840,11 @@ bool take_page_off_buddy(struct page *page) int migratetype = get_pfnblock_migratetype(page_head, pfn_head); - del_page_from_free_list(page_head, zone, page_order); + del_page_from_free_list(page_head, zone, page_order, + migratetype); break_down_buddy_pages(zone, page_head, page, 0, page_order, migratetype); SetPageHWPoisonTakenOff(page); - if (!is_migrate_isolate(migratetype)) - __mod_zone_freepage_state(zone, -1, migratetype); ret = true; break; } @@ -6930,7 +6952,7 @@ static bool try_to_accept_memory_one(struct zone *zone) list_del(&page->lru); last = list_empty(&zone->unaccepted_pages); - __mod_zone_freepage_state(zone, -MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE); + account_freepages(page, zone, -MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE); __mod_zone_page_state(zone, NR_UNACCEPTED, -MAX_ORDER_NR_PAGES); spin_unlock_irqrestore(&zone->lock, flags); @@ -6982,7 +7004,7 @@ static bool __free_unaccepted(struct page *page) spin_lock_irqsave(&zone->lock, flags); first = list_empty(&zone->unaccepted_pages); list_add_tail(&page->lru, &zone->unaccepted_pages); - __mod_zone_freepage_state(zone, MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE); + account_freepages(page, zone, MAX_ORDER_NR_PAGES, MIGRATE_MOVABLE); __mod_zone_page_state(zone, NR_UNACCEPTED, MAX_ORDER_NR_PAGES); spin_unlock_irqrestore(&zone->lock, flags); diff --git a/mm/page_isolation.c b/mm/page_isolation.c index 042937d5abe4..914a71c580d8 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -252,7 +252,8 @@ static void unset_migratetype_isolate(struct page *page, int migratetype) * Isolating this block already succeeded, so this * should not fail on zone boundaries. */ - WARN_ON_ONCE(!move_freepages_block_isolate(zone, page, migratetype)); + WARN_ON_ONCE(!move_freepages_block_isolate(zone, page, + migratetype)); } else { set_pageblock_migratetype(page, migratetype); __putback_isolated_page(page, order, migratetype);