From patchwork Fri Jan 18 17:51:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771391 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C7C5191E for ; Fri, 18 Jan 2019 17:55:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B3B592FF95 for ; Fri, 18 Jan 2019 17:55:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A79602FFCC; Fri, 18 Jan 2019 17:55:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0A5952FF95 for ; Fri, 18 Jan 2019 17:55:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A95C8E0023; Fri, 18 Jan 2019 12:55:13 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 062688E0002; Fri, 18 Jan 2019 12:55:13 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E3B748E0023; Fri, 18 Jan 2019 12:55:12 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id 858AE8E0002 for ; Fri, 18 Jan 2019 12:55:12 -0500 (EST) Received: by mail-ed1-f71.google.com with SMTP id m19so5262586edc.6 for ; Fri, 18 Jan 2019 09:55:12 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=/tOz8Xi3Yig60kJs65JVB2MYQvvX6oCbfn8y9xUdCPg=; b=bdIeHIhP0C/9uhNCs/a17Uq7EHoBt2/U2L6YzWT2CTuoTA+EzhrFKgg9Dhrz7zoFrC q/r/viIytHdjF293uvB+MOFusA9NfOAIan6IP6z8mVeYecZgd5nc94Mv4l/4eT+xMw3x BF1ijfI9KMozlQu6Gjsewg5iijRZ1pKN279vssdpbgxVBEvPJ+hcRSAggAvlVdeogi+a JwieS4cFLZ97dOscunfTMK8baHJ9fmDEU8wOznz9ugHgG+lHBZkMvquXSqmrsvSApv7k vDEbOj1f5rhv131ElnnWU1euHHKjCKmUgdXu5WZEWRW7dBWEY7djM4u65DI02b+IN0Mq jBkg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukf2xlcoJMp4vVkBOxCfxl09WnWfE5Y3Jbjv7FA2V3VRo7lKIMuq n8LhA2ClWGAATChAeAQa7kAWFDvEZ4dbVCVq4ibVIZ094+DO/mVZtMIMhvPwXvvutqNvTONNBVd DoccnHW/YE8Y3gDKCHds+i1FtgiDntvqSzKqfBSZuGHphu6L/A79cAvnFKJIDQskQUw== X-Received: by 2002:a05:6402:643:: with SMTP id u3mr16677495edx.276.1547834112083; Fri, 18 Jan 2019 09:55:12 -0800 (PST) X-Google-Smtp-Source: ALg8bN6yeugYPhoimoasRiItUREB5EV/v9V+dAg+RkDxWHUccfKHyss0kjjsyw9TBrYhcq5FAb8/ X-Received: by 2002:a05:6402:643:: with SMTP id u3mr16677424edx.276.1547834110799; Fri, 18 Jan 2019 09:55:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547834110; cv=none; d=google.com; s=arc-20160816; b=uMdo8aaVTISGBtLk9iEXOLTa4nrL1SoCGFpAz43jWXOdxP4cr1KxnxWodzCtNgYZkV 8ucw85oxk+cTDSWF6N5RyPMXlboeWzSMv31uPh/Sr/BeY2tT4ljElRkkkFbvz6SJi8qu vpecYKkiBzgmjZzpTdYIcvlO/pQ1dWVMxLCwBjftPTsYP3XOHXI9ykkWQrwga0tbdg4a V6G2hao7wXDDcbssTaam469LqzYpkpdTPQU9+/Venic93bbfxAwXOdZLgEgsc+12PKN5 hsPXRIzCnRp1ei3tDESkLukstrWr4m08VYjgnxdypBCs5U3XOo7vNMafoo/lVCYlBDO8 ugfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=/tOz8Xi3Yig60kJs65JVB2MYQvvX6oCbfn8y9xUdCPg=; b=OPlp54mVJWkNbJ3RJueVyefg5XZyBvyzKjjVimLOK7JLIMyBvYXyZv9ynvO/30xtow 1PPDK05Fir3KWbLs4RxO8DBsInqzxeg0XwG4LUxcsB/v5rTOLgisATz/9ryHElFf1guf ZpWcrYHKsju19w2Q4b/WEuwmm45Xnv0f3pasvayq1TncRYav3C3OQ8a+c82eiUXc+hSd jZHz5Ms11Do8Bg1l7hMOqW1r82JceoLiTk7JwxEu4u1Qdvlmm8YC7KfQJpiHlFQLEep/ lt3lG/rruEizPsfZAk19ZHip9o/X6wLgEf2CO9vC5W9dpgc3q9YluRkXq9iGHE9eYoO/ pcMA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp11.blacknight.com (outbound-smtp11.blacknight.com. [46.22.139.106]) by mx.google.com with ESMTPS id v14-v6si3876451ejw.273.2019.01.18.09.55.10 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:55:10 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) client-ip=46.22.139.106; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp11.blacknight.com (Postfix) with ESMTPS id 729001C30FC for ; Fri, 18 Jan 2019 17:55:10 +0000 (GMT) Received: (qmail 8177 invoked from network); 18 Jan 2019 17:55:10 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:55:10 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 20/22] mm, compaction: Sample pageblocks for free pages Date: Fri, 18 Jan 2019 17:51:34 +0000 Message-Id: <20190118175136.31341-21-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Once fast searching finishes, there is a possibility that the linear scanner is scanning full blocks found by the fast scanner earlier. This patch uses an adaptive stride to sample pageblocks for free pages. The more consecutive full pageblocks encountered, the larger the stride until a pageblock with free pages is found. The scanners might meet slightly sooner but it is an acceptable risk given that the search of the free lists may still encounter the pages and adjust the cached PFN of the free scanner accordingly. 5.0.0-rc1 5.0.0-rc1 roundrobin-v3r17 samplefree-v3r17 Amean fault-both-1 0.00 ( 0.00%) 0.00 * 0.00%* Amean fault-both-3 2752.37 ( 0.00%) 2729.95 ( 0.81%) Amean fault-both-5 4341.69 ( 0.00%) 4397.80 ( -1.29%) Amean fault-both-7 6308.75 ( 0.00%) 6097.61 ( 3.35%) Amean fault-both-12 10241.81 ( 0.00%) 9407.15 ( 8.15%) Amean fault-both-18 13736.09 ( 0.00%) 10857.63 * 20.96%* Amean fault-both-24 16853.95 ( 0.00%) 13323.24 * 20.95%* Amean fault-both-30 15862.61 ( 0.00%) 17345.44 ( -9.35%) Amean fault-both-32 18450.85 ( 0.00%) 16892.00 ( 8.45%) The latency is mildly improved offseting some overhead from earlier patches that are prerequisites for the rest of the series. However, a major impact is on the free scan rate with an 82% reduction. 5.0.0-rc1 5.0.0-rc1 roundrobin-v3r17 samplefree-v3r17 Compaction migrate scanned 21607271 20116887 Compaction free scanned 95336406 16668703 It's also the first time in the series where the number of pages scanned by the migration scanner is greater than the free scanner due to the increased search efficiency. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 7d39ff08269a..74bf620e3dcd 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -440,6 +440,7 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, unsigned long *start_pfn, unsigned long end_pfn, struct list_head *freelist, + unsigned int stride, bool strict) { int nr_scanned = 0, total_isolated = 0; @@ -449,10 +450,14 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, unsigned long blockpfn = *start_pfn; unsigned int order; + /* Strict mode is for isolation, speed is secondary */ + if (strict) + stride = 1; + cursor = pfn_to_page(blockpfn); /* Isolate free pages. */ - for (; blockpfn < end_pfn; blockpfn++, cursor++) { + for (; blockpfn < end_pfn; blockpfn += stride, cursor += stride) { int isolated; struct page *page = cursor; @@ -614,7 +619,7 @@ isolate_freepages_range(struct compact_control *cc, break; isolated = isolate_freepages_block(cc, &isolate_start_pfn, - block_end_pfn, &freelist, true); + block_end_pfn, &freelist, 0, true); /* * In strict mode, isolate_freepages_block() returns 0 if @@ -1132,7 +1137,7 @@ fast_isolate_around(struct compact_control *cc, unsigned long pfn, unsigned long /* Scan before */ if (start_pfn != pfn) { - isolate_freepages_block(cc, &start_pfn, pfn, &cc->freepages, false); + isolate_freepages_block(cc, &start_pfn, pfn, &cc->freepages, 1, false); if (cc->nr_freepages >= cc->nr_migratepages) return; } @@ -1140,7 +1145,7 @@ fast_isolate_around(struct compact_control *cc, unsigned long pfn, unsigned long /* Scan after */ start_pfn = pfn + nr_isolated; if (start_pfn != end_pfn) - isolate_freepages_block(cc, &start_pfn, end_pfn, &cc->freepages, false); + isolate_freepages_block(cc, &start_pfn, end_pfn, &cc->freepages, 1, false); /* Skip this pageblock in the future as it's full or nearly full */ if (cc->nr_freepages < cc->nr_migratepages) @@ -1332,6 +1337,7 @@ static void isolate_freepages(struct compact_control *cc) unsigned long block_end_pfn; /* end of current pageblock */ unsigned long low_pfn; /* lowest pfn scanner is able to scan */ struct list_head *freelist = &cc->freepages; + unsigned int stride; /* Try a small search of the free lists for a candidate */ isolate_start_pfn = fast_isolate_freepages(cc); @@ -1354,6 +1360,7 @@ static void isolate_freepages(struct compact_control *cc) block_end_pfn = min(block_start_pfn + pageblock_nr_pages, zone_end_pfn(zone)); low_pfn = pageblock_end_pfn(cc->migrate_pfn); + stride = cc->mode == MIGRATE_ASYNC ? COMPACT_CLUSTER_MAX : 1; /* * Isolate free pages until enough are available to migrate the @@ -1364,6 +1371,8 @@ static void isolate_freepages(struct compact_control *cc) block_end_pfn = block_start_pfn, block_start_pfn -= pageblock_nr_pages, isolate_start_pfn = block_start_pfn) { + unsigned long nr_isolated; + /* * This can iterate a massively long zone without finding any * suitable migration targets, so periodically check resched. @@ -1385,8 +1394,8 @@ static void isolate_freepages(struct compact_control *cc) continue; /* Found a block suitable for isolating free pages from. */ - isolate_freepages_block(cc, &isolate_start_pfn, block_end_pfn, - freelist, false); + nr_isolated = isolate_freepages_block(cc, &isolate_start_pfn, + block_end_pfn, freelist, stride, false); /* Update the skip hint if the full pageblock was scanned */ if (isolate_start_pfn == block_end_pfn) @@ -1410,6 +1419,13 @@ static void isolate_freepages(struct compact_control *cc) */ break; } + + /* Adjust stride depending on isolation */ + if (nr_isolated) { + stride = 1; + continue; + } + stride = min_t(unsigned int, COMPACT_CLUSTER_MAX, stride << 1); } /*