From patchwork Fri Jan 18 17:51:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771375 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8EDFA6C5 for ; Fri, 18 Jan 2019 17:53:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7F043285CC for ; Fri, 18 Jan 2019 17:53:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 725D1286B2; Fri, 18 Jan 2019 17:53:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BA9BB285CC for ; Fri, 18 Jan 2019 17:53:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A92CB8E001B; Fri, 18 Jan 2019 12:53:51 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A42558E0002; Fri, 18 Jan 2019 12:53:51 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 909878E001B; Fri, 18 Jan 2019 12:53:51 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 3A8ED8E0002 for ; Fri, 18 Jan 2019 12:53:51 -0500 (EST) Received: by mail-ed1-f72.google.com with SMTP id i55so5264060ede.14 for ; Fri, 18 Jan 2019 09:53:51 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=q4XbqljV4csmVEUb4HH/NDfBhps7WKZmhbq0+dowWqs=; b=oxdWuSeJG29y24d6om8HbrKdDEv0Y+PRmnh05tEBG9tYIBp6XzaT8Fp3RwLBh650fg JfzdoJkEPA/5CmQLs8GYwqGr2V3fZLHBdFmAKy8AsxDj2QHc7hqkf5zyOgJ6q8uVtKmw yI5uUFmzFnYfgxPRmtBZ+PZmNE7i5zQ3cuH4Bz1k8wzVsG9JvhNKnzTh6iwNaZMSMHpU LncQt2WAmBdLUghdVWKmLGoZjhHH0bSr4RXgWnh1Y2tyThOHlD8xD6acC8D1E9zok8Wu uiztI/5mZuh53PM3JWEic64WKn3LpC2rU3RANbQEWBuIcnf3CLgaQuTn4BUVxNmGE049 FFOg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukeH1VdZhDoioQ/ELPqtnc+EJnbAdY/gkL3p+RBKqJUpH8gmviNJ drRVrz1+a6EeRzSs6IH3G/9QVzEU9FUNWbcnu1YnZRTbZvpvvIw12fQswMfI+2xqTDFUEnIqTw2 e+9EYtk72UOW3idJaTxNS3MLSB7e7HEvqs+F9ZTsqf7IjyyTS9DzafB2Pj4jEIATgTg== X-Received: by 2002:a17:906:a44:: with SMTP id x4-v6mr15294936ejf.177.1547834030691; Fri, 18 Jan 2019 09:53:50 -0800 (PST) X-Google-Smtp-Source: ALg8bN4KTDx7d0NfnJ1NFFP8jeU+ZWBxIVxlivjWau5hnzVKfDzBRl6+g9saF0oPMoWvq+q8LX/d X-Received: by 2002:a17:906:a44:: with SMTP id x4-v6mr15294878ejf.177.1547834029482; Fri, 18 Jan 2019 09:53:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547834029; cv=none; d=google.com; s=arc-20160816; b=cbrMOdb4dhFemysUtIR5Y6BTUC1JR4Z/zWd1DggvI/4s9/Bm4nR61skgB0hYyHfjBv uBYzbwXktsebF0WzPnV7sHLcahrudnj5V9/bmDvdTGlP2nU1sYGxS7TXs+Xw/WlLVZQE NISo7wLFML4yrTxJOQbJVH4GzHXxyZGreUdmvtaFM9Du4ERinFlQnCvJKoxe6b5BYHhP 1DFUCDawuvEaaBKBy/WmK2Hr2VLH4dxNI8GNqmcs/6Nz7zE1S9305RDVmsFdKXScUjA1 LxQrb1bQmFbgg2hzPKeYu/l6hUwFLQ/Yi+gXvyDIN2xZKRSSvz1Nl2CwOXuCj3t2Re4E zGqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=q4XbqljV4csmVEUb4HH/NDfBhps7WKZmhbq0+dowWqs=; b=YXtJcvaOl5+fGYYlcVQtAMx9ANHdhvMrYG5ekF2eKcejaF0nvHDlCM+RmMXLQULBAU e2IAYnGENylO/8iAHXjsvBJ8Pt5cMbamKc1bgtWNpibb0T1BFolnwgCsHCQZkf+lEmkE 13KB/nVoAsglPGkqS9UzXGAezKUp7CqF2Ww5B0eTf3jOQ5fsYh1lbpCHvcqNyw0OHCLQ rxserqIVuULjBI2Lre2+6vy05aaRe8EjmdZo4emVTc6kLtjQOgJwTnxNXhV1IYy6GBZz hpu0Q66lypcOzvhAdvIxGT8ICFKT39i+FseMsZ+lQC8K+xXp5dIQbPwBgiU36E3xEynU viOg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp13.blacknight.com (outbound-smtp13.blacknight.com. [46.22.139.230]) by mx.google.com with ESMTPS id k13si3535974edl.377.2019.01.18.09.53.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:53:49 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) client-ip=46.22.139.230; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (unknown [81.17.254.16]) by outbound-smtp13.blacknight.com (Postfix) with ESMTPS id 2B06B1C35C8 for ; Fri, 18 Jan 2019 17:53:49 +0000 (GMT) Received: (qmail 4431 invoked from network); 18 Jan 2019 17:53:49 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:53:49 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 12/22] mm, compaction: Avoid rescanning the same pageblock multiple times Date: Fri, 18 Jan 2019 17:51:26 +0000 Message-Id: <20190118175136.31341-13-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Pageblocks are marked for skip when no pages are isolated after a scan. However, it's possible to hit corner cases where the migration scanner gets stuck near the boundary between the source and target scanner. Due to pages being migrated in blocks of COMPACT_CLUSTER_MAX, pages that are migrated can be reallocated before the pageblock is complete. The pageblock is not necessarily skipped so it can be rescanned multiple times. Similarly, a pageblock with some dirty/writeback pages may fail to migrate and be rescanned until writeback completes which is wasteful. This patch tracks if a pageblock is being rescanned. If so, then the entire pageblock will be migrated as one operation. This narrows the race window during which pages can be reallocated during migration. Secondly, if there are pages that cannot be isolated then the pageblock will still be fully scanned and marked for skipping. On the second rescan, the pageblock skip is set and the migration scanner makes progress. 5.0.0-rc1 5.0.0-rc1 findfree-v3r16 norescan-v3r16 Amean fault-both-1 0.00 ( 0.00%) 0.00 * 0.00%* Amean fault-both-3 3200.68 ( 0.00%) 3002.07 ( 6.21%) Amean fault-both-5 4847.75 ( 0.00%) 4684.47 ( 3.37%) Amean fault-both-7 6658.92 ( 0.00%) 6815.54 ( -2.35%) Amean fault-both-12 11077.62 ( 0.00%) 10864.02 ( 1.93%) Amean fault-both-18 12403.97 ( 0.00%) 12247.52 ( 1.26%) Amean fault-both-24 15607.10 ( 0.00%) 15683.99 ( -0.49%) Amean fault-both-30 18752.27 ( 0.00%) 18620.02 ( 0.71%) Amean fault-both-32 21207.54 ( 0.00%) 19250.28 * 9.23%* 5.0.0-rc1 5.0.0-rc1 findfree-v3r16 norescan-v3r16 Percentage huge-3 96.86 ( 0.00%) 95.00 ( -1.91%) Percentage huge-5 93.72 ( 0.00%) 94.22 ( 0.53%) Percentage huge-7 94.31 ( 0.00%) 92.35 ( -2.08%) Percentage huge-12 92.66 ( 0.00%) 91.90 ( -0.82%) Percentage huge-18 91.51 ( 0.00%) 89.58 ( -2.11%) Percentage huge-24 90.50 ( 0.00%) 90.03 ( -0.52%) Percentage huge-30 91.57 ( 0.00%) 89.14 ( -2.65%) Percentage huge-32 91.00 ( 0.00%) 90.58 ( -0.46%) Negligible difference but this was likely a case when the specific corner case was not hit. A previous run of the same patch based on an earlier iteration of the series showed large differences where migration rates could be halved when the corner case was hit. The specific corner case where migration scan rates go through the roof was due to a dirty/writeback pageblock located at the boundary of the migration/free scanner did not happen in this case. When it does happen, the scan rates multipled by massive margins. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 32 ++++++++++++++++++++++++++------ mm/internal.h | 1 + 2 files changed, 27 insertions(+), 6 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 19fea4a7b3f4..a31fea7b3f96 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -949,8 +949,11 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, cc->nr_migratepages++; nr_isolated++; - /* Avoid isolating too much */ - if (cc->nr_migratepages == COMPACT_CLUSTER_MAX) { + /* + * Avoid isolating too much unless this block is being + * rescanned (e.g. dirty/writeback pages, parallel allocation). + */ + if (cc->nr_migratepages == COMPACT_CLUSTER_MAX && !cc->rescan) { ++low_pfn; break; } @@ -997,11 +1000,14 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, spin_unlock_irqrestore(zone_lru_lock(zone), flags); /* - * Updated the cached scanner pfn if the pageblock was scanned - * without isolating a page. The pageblock may not be marked - * skipped already if there were no LRU pages in the block. + * Updated the cached scanner pfn once the pageblock has been scanned + * Pages will either be migrated in which case there is no point + * scanning in the near future or migration failed in which case the + * failure reason may persist. The block is marked for skipping if + * there were no pages isolated in the block or if the block is + * rescanned twice in a row. */ - if (low_pfn == end_pfn && !nr_isolated) { + if (low_pfn == end_pfn && (!nr_isolated || cc->rescan)) { if (valid_page && !skip_updated) set_pageblock_skip(valid_page); update_cached_migrate(cc, low_pfn); @@ -2033,6 +2039,20 @@ static enum compact_result compact_zone(struct compact_control *cc) int err; unsigned long start_pfn = cc->migrate_pfn; + /* + * Avoid multiple rescans which can happen if a page cannot be + * isolated (dirty/writeback in async mode) or if the migrated + * pages are being allocated before the pageblock is cleared. + * The first rescan will capture the entire pageblock for + * migration. If it fails, it'll be marked skip and scanning + * will proceed as normal. + */ + cc->rescan = false; + if (pageblock_start_pfn(last_migrated_pfn) == + pageblock_start_pfn(start_pfn)) { + cc->rescan = true; + } + switch (isolate_migratepages(cc->zone, cc)) { case ISOLATE_ABORT: ret = COMPACT_CONTENDED; diff --git a/mm/internal.h b/mm/internal.h index 983cb975545f..d5b999e5eb5f 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -205,6 +205,7 @@ struct compact_control { bool direct_compaction; /* False from kcompactd or /proc/... */ bool whole_zone; /* Whole zone should/has been scanned */ bool contended; /* Signal lock or sched contention */ + bool rescan; /* Rescanning the same pageblock */ }; unsigned long