From patchwork Fri Jan 18 17:51:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771389 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7F2A46C5 for ; Fri, 18 Jan 2019 17:55:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6A0372FF95 for ; Fri, 18 Jan 2019 17:55:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 584182FFCC; Fri, 18 Jan 2019 17:55:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CC47D2FF95 for ; Fri, 18 Jan 2019 17:55:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C8E858E0022; Fri, 18 Jan 2019 12:55:02 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C3E468E0002; Fri, 18 Jan 2019 12:55:02 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B03918E0022; Fri, 18 Jan 2019 12:55:02 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 52F598E0002 for ; Fri, 18 Jan 2019 12:55:02 -0500 (EST) Received: by mail-ed1-f72.google.com with SMTP id i55so5265197ede.14 for ; Fri, 18 Jan 2019 09:55:02 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=r+ZeJpz1lZS5QW4P1aAtJ+FjrlN8m7Grj1Z7u/uZEQM=; b=gfQMVY9z32e19yJ6xChdD9og1rUt7No1j/WikEaYLKqeDXQAMRKLNQmOrrNSzUhXCY liRjxD+S1Sa+XSGUI0Dy6vxCyaPfyGEj0a+XvjCIQM5ba/LWiaB+5V1kDWyS1tzb9hms qRwaNUGen0IcyfLkv2Zl7PsKGA4VGlLKOLajaExBQjhcExGoLkaquLJQXkREzCzCzZJz sLMD2Ds3VZG85kAYpmpfWkpL2XKiYMp4kNsSqFKTbBxWi2dTCcVq3jZ/2Cu31YAvUvXH uvYFsPPMihj7QuhlQC6heHhz9yM/QjFp5XwGPieE79Y6KtFQwk0iuyv15ye8h6560/MQ +QyA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukcCIKzLQADKHd2CgBYCqk/SEKXMiLfu40pC7pqIuGQu/0cpC2pG RGiQgXBy+Xmk600QS86hWifhxk+IzdhcvJgghT3b43/jiJrYfU54COhWDQub6IgOA1OCRyAmTIW JYjD7Der4L4Tjzx7gbr97KKPmGZrRY/2RNDgtzQWN6D8KWkPxDMUE0XicfQDOKZhAXg== X-Received: by 2002:a50:a517:: with SMTP id y23mr17051937edb.219.1547834101812; Fri, 18 Jan 2019 09:55:01 -0800 (PST) X-Google-Smtp-Source: ALg8bN5YI1UQ7CLCwxCVp5m50sqeGLUr7nKCEzksn2jLHSJp5WQq+ojIbuPAJCMeHuBxcMD1rTqD X-Received: by 2002:a50:a517:: with SMTP id y23mr17051876edb.219.1547834100645; Fri, 18 Jan 2019 09:55:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547834100; cv=none; d=google.com; s=arc-20160816; b=XX3XugCoivF3J8qQQGiiTz8BsBLgXtAaUcHZg9nN/loqq7BFahbvKkXvd2BozV2zs9 OJhXsFR1Jy8XUvpt9pwJs0Q+L7zdVgj1PGphOc/aFT+m75cnr/qz4rACqg3l/IPViIGO ooYRwwTI0Oc430h+8zWW9+YNkbSNA+MLoW6XU2KtVL/9gLRGwmx7CXBesmgtk0Tp6t7F cNPy54fdfi9aiY1U4IzhqIJcIlFg74reoOB5kcu3Zb0W4TRSGEef/4RnWgA9Txw/zjlN 6scjGoAOmIqDrZrR8NGUGh/u+n9IMpaFTgWUlgzgWKW18X+MXiHdIEWFbf/BRRaOgGVF x1nA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=r+ZeJpz1lZS5QW4P1aAtJ+FjrlN8m7Grj1Z7u/uZEQM=; b=TGDxcmT2gIN8BLKjr9Uj0lKUQMYlDO1Qc7KpUzb4pJK/S9NqdBv9oTGuow8EqZ8Hwa iBKrXu+pBndryBiMfGntYqKIL3OIZsFC8vsV9wXYEfsYShVdaGM+BOHwpUlFLFcy/1MD ClmLmnSWY2ZYUknsTNwPAse5htMIWTRcvm5vC2OodLL7sjn1v7uHZyw/JSyWqQU0/qXg 2lKZgZSQUvb4waYTy7mvTewFMWayrhpzMn7+tAigLYr/0dX2yxzeHp3xXdBoIl0ZPhPU Wm3yc9AjPeSO8+f/yEar57/i8wO6qnhWmBLVWItKDOHtXer4rooac6zW3fHAlEa1pF5Y kzmA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp13.blacknight.com (outbound-smtp13.blacknight.com. [46.22.139.230]) by mx.google.com with ESMTPS id l17si5873171edq.20.2019.01.18.09.55.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:55:00 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) client-ip=46.22.139.230; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (unknown [81.17.254.16]) by outbound-smtp13.blacknight.com (Postfix) with ESMTPS id 4B9D31C3064 for ; Fri, 18 Jan 2019 17:55:00 +0000 (GMT) Received: (qmail 7671 invoked from network); 18 Jan 2019 17:55:00 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:55:00 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 19/22] mm, compaction: Round-robin the order while searching the free lists for a target Date: Fri, 18 Jan 2019 17:51:33 +0000 Message-Id: <20190118175136.31341-20-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP As compaction proceeds and creates high-order blocks, the free list search gets less efficient as the larger blocks are used as compaction targets. Eventually, the larger blocks will be behind the migration scanner for partially migrated pageblocks and the search fails. This patch round-robins what orders are searched so that larger blocks can be ignored and find smaller blocks that can be used as migration targets. The overall impact was small on 1-socket but it avoids corner cases where the migration/free scanners meet prematurely or situations where many of the pageblocks encountered by the free scanner are almost full instead of being properly packed. Previous testing had indicated that without this patch there were occasional large spikes in the free scanner without this patch. [dan.carpenter@oracle.com: Fix static checker warning] Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 33 ++++++++++++++++++++++++++++++--- mm/internal.h | 3 ++- 2 files changed, 32 insertions(+), 4 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 04ec7d4da719..7d39ff08269a 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1147,6 +1147,24 @@ fast_isolate_around(struct compact_control *cc, unsigned long pfn, unsigned long set_pageblock_skip(page); } +/* Search orders in round-robin fashion */ +static int next_search_order(struct compact_control *cc, int order) +{ + order--; + if (order < 0) + order = cc->order - 1; + + /* Search wrapped around? */ + if (order == cc->search_order) { + cc->search_order--; + if (cc->search_order < 0) + cc->search_order = cc->order - 1; + return -1; + } + + return order; +} + static unsigned long fast_isolate_freepages(struct compact_control *cc) { @@ -1183,9 +1201,15 @@ fast_isolate_freepages(struct compact_control *cc) if (WARN_ON_ONCE(min_pfn > low_pfn)) low_pfn = min_pfn; - for (order = cc->order - 1; - order >= 0 && !page; - order--) { + /* + * Search starts from the last successful isolation order or the next + * order to search after a previous failure + */ + cc->search_order = min_t(unsigned int, cc->order - 1, cc->search_order); + + for (order = cc->search_order; + !page && order >= 0; + order = next_search_order(cc, order)) { struct free_area *area = &cc->zone->free_area[order]; struct list_head *freelist; struct page *freepage; @@ -1209,6 +1233,7 @@ fast_isolate_freepages(struct compact_control *cc) if (pfn >= low_pfn) { cc->fast_search_fail = 0; + cc->search_order = order; page = freepage; break; } @@ -2136,6 +2161,7 @@ static enum compact_result compact_zone_order(struct zone *zone, int order, .total_migrate_scanned = 0, .total_free_scanned = 0, .order = order, + .search_order = order, .gfp_mask = gfp_mask, .zone = zone, .mode = (prio == COMPACT_PRIO_ASYNC) ? @@ -2367,6 +2393,7 @@ static void kcompactd_do_work(pg_data_t *pgdat) struct zone *zone; struct compact_control cc = { .order = pgdat->kcompactd_max_order, + .search_order = pgdat->kcompactd_max_order, .total_migrate_scanned = 0, .total_free_scanned = 0, .classzone_idx = pgdat->kcompactd_classzone_idx, diff --git a/mm/internal.h b/mm/internal.h index d5b999e5eb5f..31bb0be6fd52 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -192,7 +192,8 @@ struct compact_control { struct zone *zone; unsigned long total_migrate_scanned; unsigned long total_free_scanned; - unsigned int fast_search_fail; /* failures to use free list searches */ + unsigned short fast_search_fail;/* failures to use free list searches */ + short search_order; /* order to start a fast search at */ const gfp_t gfp_mask; /* gfp mask of a direct compactor */ int order; /* order a direct compactor needs */ int migratetype; /* migratetype of direct compactor */