From patchwork Fri Aug 14 17:31:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Minchan Kim X-Patchwork-Id: 11715053 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2E002739 for ; Fri, 14 Aug 2020 17:31:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E12CC20768 for ; Fri, 14 Aug 2020 17:31:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="C6ftLx9s" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E12CC20768 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BE5738D0002; Fri, 14 Aug 2020 13:31:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B6C7F8D0003; Fri, 14 Aug 2020 13:31:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9BD908D0002; Fri, 14 Aug 2020 13:31:45 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0101.hostedemail.com [216.40.44.101]) by kanga.kvack.org (Postfix) with ESMTP id 749AA8D0003 for ; Fri, 14 Aug 2020 13:31:45 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 2E013181AEF23 for ; Fri, 14 Aug 2020 17:31:45 +0000 (UTC) X-FDA: 77149866570.09.jam52_531080126ffe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin09.hostedemail.com (Postfix) with ESMTP id 89454180AD81A for ; Fri, 14 Aug 2020 17:31:44 +0000 (UTC) X-Spam-Summary: 1,0,0,ec840b1cb7a268ab,d41d8cd98f00b204,minchan.kim@gmail.com,,RULES_HIT:2:41:69:355:379:541:560:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1606:1730:1747:1777:1792:2194:2196:2198:2199:2200:2201:2393:2553:2559:2562:2693:2731:2736:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:4119:4250:4321:4385:5007:6117:6119:6120:6261:6653:7875:7903:9010:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:12986:13215:13229:13894:14096:21080:21094:21323:21444:21451:21627:21795:21987:21990:30051:30054:30090,0,RBL:209.85.210.196:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.50.0.100;04yfworbywkexyznf867skcsyysbpyctdzdjrcgmxj4ujqjn5tqrobznqyxzt7j.k9b9rnh78f4tcotk4ssqiojxa6x4uq86nwmiomfm7sjw67zw8c5yf4443csdzyu.h-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: jam52_531080126ffe X-Filterd-Recvd-Size: 8417 Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) by imf23.hostedemail.com (Postfix) with ESMTP for ; Fri, 14 Aug 2020 17:31:43 +0000 (UTC) Received: by mail-pf1-f196.google.com with SMTP id m8so4885744pfh.3 for ; Fri, 14 Aug 2020 10:31:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=EkTZgMvscD2KwIBX0vZD343fn5ehrZxMsNsNzYcucec=; b=C6ftLx9sbYiyyYKgmA04wWOsnc7KvbG16gcWcNOv/hcRQw8ybQMpUwoyUXTR4dmtd0 uZe72diH9C0c9xNNh6MC0rK+V4NcOpIoVZm+yvCAf/fkIUYY5jyh5BlDOaVauQoDwEKI 4ctGErOacT8LiOg7a5hP/unoZeRiLFafHJ/dU6lGeosk0jyLsGZRcVtrzAVQglt2rg9r yPa3M8DbpvIadtJ79ZQ5Ku0NOXf9GljegFXH30kKLB2U3hAl2QXgC1un7EzyYX8sNB0A d21DpyE9mccr1JS1m/roezO+OUJaqx3ip7oPJxx216u6kZgUxQQy6tFEhkoYqKAYB8k/ G+Qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=EkTZgMvscD2KwIBX0vZD343fn5ehrZxMsNsNzYcucec=; b=ip+L71GTlT2Z9stLPEmJ5UTXPmKJ7jvQ/xFC4Xr0pBXPf8hi8aPx6+vuvR1NcjB1xx WXZWZdriva8wUg/hwzofX4qEeJq0I6DWuYkQtXFIdLFIRsVTanvmlXokD3N9PU1UuVBq hNdzVE2xrI0cqajxzaI4Vw8x77S0SeehueDwAmBmIdcmYBzpKw2s7eX5ozUGiolDRkzR +SVXPemRnClqObK2G0QGi66agAKYMbHR7JP6+BkPUpbnGFDmvG8g5ZboXO9n19L8PazQ OCnH2SK2mpCadG2EcN2W3lMmPParXRHhEVzgPV4XFg7VkD/dcCCdeP2jsbRKMrru5fi0 N4aA== X-Gm-Message-State: AOAM530MRPjRT+8BWORO2wzzjehwwGSyCpkbbukpPdN/SBoEgf13ATjY oMPjDYAIpfKie4yban55JSM= X-Google-Smtp-Source: ABdhPJzay7FypJUmNSI99B+mNjGzIjNonb2GXNqZn8e1q9by06s3lfGRmPkPuA29rxIOZolhWT6ktA== X-Received: by 2002:a65:62c3:: with SMTP id m3mr2407190pgv.338.1597426302983; Fri, 14 Aug 2020 10:31:42 -0700 (PDT) Received: from bbox-1.mtv.corp.google.com ([2620:15c:211:1:7220:84ff:fe09:5e58]) by smtp.gmail.com with ESMTPSA id n22sm8522973pjq.25.2020.08.14.10.31.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Aug 2020 10:31:41 -0700 (PDT) From: Minchan Kim To: Andrew Morton Cc: linux-mm , Joonsoo Kim , Vlastimil Babka , John Dias , Suren Baghdasaryan , pullip.cho@samsung.com, Minchan Kim Subject: [RFC 3/7] mm: compaction: deal with upcoming high-order page splitting Date: Fri, 14 Aug 2020 10:31:27 -0700 Message-Id: <20200814173131.2803002-4-minchan@kernel.org> X-Mailer: git-send-email 2.28.0.220.ged08abb693-goog In-Reply-To: <20200814173131.2803002-1-minchan@kernel.org> References: <20200814173131.2803002-1-minchan@kernel.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 89454180AD81A X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When compaction isolates free pages, it needs to consider freed pages's order and sub-page splitting to support upcoming high order page bulk allocation. Since we have primitive functions to deal with high order page splitting, this patch introduces cc->isolate_order to indicate what order pages the API user want to allocate. It isolates free pages with order greater or equal to cc->isolate_order. After isolating it splits them into sub pages of cc->isolate_order order. Signed-off-by: Minchan Kim --- mm/compaction.c | 42 ++++++++++++++++++++++++++++-------------- mm/internal.h | 1 + 2 files changed, 29 insertions(+), 14 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index f31799a841f2..76f380cb801d 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -68,7 +68,8 @@ static const unsigned int HPAGE_FRAG_CHECK_INTERVAL_MSEC = 500; #define COMPACTION_HPAGE_ORDER (PMD_SHIFT - PAGE_SHIFT) #endif -static unsigned long release_freepages(struct list_head *freelist) +static unsigned long release_freepages(struct list_head *freelist, + unsigned int order) { struct page *page, *next; unsigned long high_pfn = 0; @@ -76,7 +77,7 @@ static unsigned long release_freepages(struct list_head *freelist) list_for_each_entry_safe(page, next, freelist, lru) { unsigned long pfn = page_to_pfn(page); list_del(&page->lru); - __free_page(page); + __free_pages(page, order); if (pfn > high_pfn) high_pfn = pfn; } @@ -84,7 +85,7 @@ static unsigned long release_freepages(struct list_head *freelist) return high_pfn; } -static void split_map_pages(struct list_head *list) +static void split_map_pages(struct list_head *list, unsigned int split_order) { unsigned int i, order, nr_pages; struct page *page, *next; @@ -94,15 +95,15 @@ static void split_map_pages(struct list_head *list) list_del(&page->lru); order = page_private(page); - nr_pages = 1 << order; + nr_pages = 1 << (order - split_order); post_alloc_hook(page, order, __GFP_MOVABLE); - if (order) - split_page_by_order(page, order, 0); + if (order > split_order) + split_page_by_order(page, order, split_order); for (i = 0; i < nr_pages; i++) { list_add(&page->lru, &tmp_list); - page++; + page += 1 << split_order; } } @@ -547,8 +548,10 @@ static bool compact_unlock_should_abort(spinlock_t *lock, } /* - * Isolate free pages onto a private freelist. If @strict is true, will abort - * returning 0 on any invalid PFNs or non-free pages inside of the pageblock + * Isolate free pages onto a private freelist if order of page is greater + * or equal to cc->isolate_order. If @strict is true, will abort + * returning 0 on any invalid PFNs, pages with order lower than + * cc->isolate_order or non-free pages inside of the pageblock * (even though it may still end up isolating some pages). */ static unsigned long isolate_freepages_block(struct compact_control *cc, @@ -625,8 +628,19 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, goto isolate_fail; } - /* Found a free page, will break it into order-0 pages */ + /* + * Found a free page. will isolate and possibly split the page + * into isolate_order sub pages if the page's order is greater + * than or equal to the isolate_order. Otherwise, it will keep + * going with further pages to isolate them unless strict is + * true. + */ order = page_order(page); + if (order < cc->isolate_order) { + blockpfn += (1UL << order) - 1; + cursor += (1UL << order) - 1; + goto isolate_fail; + } isolated = __isolate_free_page(page, order); if (!isolated) break; @@ -752,11 +766,11 @@ isolate_freepages_range(struct compact_control *cc, } /* __isolate_free_page() does not map the pages */ - split_map_pages(&freelist); + split_map_pages(&freelist, cc->isolate_order); if (pfn < end_pfn) { /* Loop terminated early, cleanup. */ - release_freepages(&freelist); + release_freepages(&freelist, cc->isolate_order); return 0; } @@ -1564,7 +1578,7 @@ static void isolate_freepages(struct compact_control *cc) splitmap: /* __isolate_free_page() does not map the pages */ - split_map_pages(freelist); + split_map_pages(freelist, 0); } /* @@ -2376,7 +2390,7 @@ compact_zone(struct compact_control *cc, struct capture_control *capc) * so we don't leave any returned pages behind in the next attempt. */ if (cc->nr_freepages > 0) { - unsigned long free_pfn = release_freepages(&cc->freepages); + unsigned long free_pfn = release_freepages(&cc->freepages, 0); cc->nr_freepages = 0; VM_BUG_ON(free_pfn == 0); diff --git a/mm/internal.h b/mm/internal.h index 10c677655912..5f1e9d76a623 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -244,6 +244,7 @@ struct compact_control { bool contended; /* Signal lock or sched contention */ bool rescan; /* Rescanning the same pageblock */ bool alloc_contig; /* alloc_contig_range allocation */ + int isolate_order; /* minimum order isolated from buddy */ }; /*