From patchwork Fri Jan 18 17:51:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10771353 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6896B6C5 for ; Fri, 18 Jan 2019 17:51:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 573D72FB87 for ; Fri, 18 Jan 2019 17:51:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4AD032FC0E; Fri, 18 Jan 2019 17:51:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 11EE32FB87 for ; Fri, 18 Jan 2019 17:51:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 036BF8E000F; Fri, 18 Jan 2019 12:51:50 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id F24688E0002; Fri, 18 Jan 2019 12:51:49 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E21FE8E000F; Fri, 18 Jan 2019 12:51:49 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id 88B0A8E0002 for ; Fri, 18 Jan 2019 12:51:49 -0500 (EST) Received: by mail-ed1-f71.google.com with SMTP id o21so5272371edq.4 for ; Fri, 18 Jan 2019 09:51:49 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id; bh=HykaP6BM1ZrAzK3xANMgLCkwNoiAHaQBPyZbtcAqtIs=; b=i1GjYF8fjEg1HrwbOJdGmIF7YhU5s4WSt3svqW54YxzRTKbu+/XVzayzH5rNkld4Sy 8Ktn7/rtWmpiHlkLrIUGqHum1mx0ovl2N6tsE1RZFJD3/z1ff3CYrCcQu4vNk1nPQmwx d2R1dFJZWAvsTcKZ2In0eCrSyaWRGzwHOnnL/46hfjhkIlXmy2uqoaoNBoTPUS+rpX5a 9BYd/E2hMyN8qDfNF0udHdee1cEKqWRgyDJXbz+oUURJwrvY5x/5thcq4rCM+HeVn5mr m50OK+lYzuXSgM5jSVxT0kcIX0XUqMsT7Sc7PfZ/1ICfDAEpUj/KOLFLW4VIkoVO3HP+ NPVQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukfzmZnX+kLUKRBrUtpJK8VoSIg7x8cAQQHWg7LnWdVUMvsqH02I vG8gpsZmulBiF4+qBjYfyCNRWhebwnc4ipY8TsDlBgHSB41c4LlBKZ3Qty+rMkwrsfNifDbpC/m XI34zDxle4YC99QjHeRmBw8Jfjwu1EKBewn+FJhrR78SrAfsNs4tbQNtxlI4DKWt91w== X-Received: by 2002:a17:906:1489:: with SMTP id x9-v6mr14980916ejc.222.1547833908948; Fri, 18 Jan 2019 09:51:48 -0800 (PST) X-Google-Smtp-Source: ALg8bN4NCSWZ2AvMvs/1rEQ59QpOuOhjjngXUPJl3M5bFJJtxqDne8N2uSrz+5B9utXu7El1nct2 X-Received: by 2002:a17:906:1489:: with SMTP id x9-v6mr14980859ejc.222.1547833907704; Fri, 18 Jan 2019 09:51:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547833907; cv=none; d=google.com; s=arc-20160816; b=CpWNsZRlYDs0uWOIcvw98PHf6IAhacGbu1VqE07lt7GzipTf4w6CcyMWcT5JGIsppB V0+Qi5DCs+NGwMGU99WIWGcvKGVdwYG7YNyHq9mTIiHWxmV8Zg/NvW4yQ0GQk+aPGBYT 3SWqpHOzmpKrMXqeJySRarWRUrXMuhLUhapmO2sYbQ96+ynlaNuZPIGXl776ow7VdheR 4LkYSS76leB1Bi5GY0WAzmjhUC3rcceHl8iWAhBvNk7D0NrTWl6DSP+86S4yczPDqzNI MaTbxp2qf9F98uCrL2rBkoiIWmOhy4zhS8BlL3OPkz1se2cWHtmhUeo00Bao1sGS7MJ/ /p3g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:date:subject:cc:to:from; bh=HykaP6BM1ZrAzK3xANMgLCkwNoiAHaQBPyZbtcAqtIs=; b=M4xyPXq4IepgO6sBozNh1Foq7R8oBD3drFy5SEznbW0Bm9xLLPUnEUUJFMB9jE31eo cSmYGGjrgJaOTKJqiVu+HGTQtNpbaqKhaD1no1SNLKNvMhr1jQ7LWgMVb7JnAkASmKA3 fnwU1XxqEvVdbWr+g2sdR1GBpNLdA/+c8Q3IP1lqwUuACJJJitUuVH85jcK7ikUp2txY 94t+eBs9B0+9J0QLrR2M1tf9p63Ubd0ZVoiNnP+zrlcWaBs9So2M7WgS9Lj5xfPH0lX4 WwBMlM4ggykz24abK7gq/fR4HEP/+xRbjr2xmPgNZkxPCi0Qm+FeqTi10C3pBDD5kSXm Le8Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp10.blacknight.com (outbound-smtp10.blacknight.com. [46.22.139.15]) by mx.google.com with ESMTPS id e52si333809edb.227.2019.01.18.09.51.47 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 18 Jan 2019 09:51:47 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) client-ip=46.22.139.15; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp10.blacknight.com (Postfix) with ESMTPS id 2A5241C3555 for ; Fri, 18 Jan 2019 17:51:47 +0000 (GMT) Received: (qmail 30217 invoked from network); 18 Jan 2019 17:51:47 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:51:47 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 00/22] Increase success rates and reduce latency of compaction v3 Date: Fri, 18 Jan 2019 17:51:14 +0000 Message-Id: <20190118175136.31341-1-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This is a drop-in replacement for the series currently in Andrews tree that incorporates static checking and compile warning fixes (Dan, YueHaibing) and extensive review feedback from Vlastimil. Big thanks to Vlastimil as the review was extremely detailed and a number of issues were caught. Not all the patches have been acked but I think an update is still worthwhile. Andrew, please drop the series you have and replace it with the following on the off-chance we get bug reports that are fixed already. Doing this with -fix patches would be relatively painful for little gain. Changelog since v2 o Fix static checker warnings (dan) o Fix unused variable warnings (yue) o Drop patch about PageReserved as there is some abuse of the flag outside of the mm core. (vbabka) o Drop patch using the bulk free helper as it may be vulnerable to races with compaction and gup (vbabka) o Drop patch about remote compaction. It's unnecessary at this time and unclear what the semantics should even be (vbabka) o Changelog fixes and clarifications (vbabka) o Free list management and search Confined mostly to "mm, compaction: Use free lists to quickly locate a migration source" which is arguably the most heavily modified patch in this revision (vbabka, mel) o Some minor churn, modifications, flow changes that fallout from addressing the review feedback (mel) o Minor pageblock skip changes, mostly fixing up which patch makes the changes so the patches are incremental (mel) This series reduces scan rates and success rates of compaction, primarily by using the free lists to shorten scans, better controlling of skip information and whether multiple scanners can target the same block and capturing pageblocks before being stolen by parallel requests. The series is based on mmotm from January 9th, 2019 with the previous compaction series reverted. I'm mostly using thpscale to measure the impact of the series. The benchmark creates a large file, maps it, faults it, punches holes in the mapping so that the virtual address space is fragmented and then tries to allocate THP. It re-executes for different numbers of threads. From a fragmentation perspective, the workload is relatively benign but it does stress compaction. The overall impact on latencies for a 1-socket machine is baseline patches Amean fault-both-3 3832.09 ( 0.00%) 2748.56 * 28.28%* Amean fault-both-5 4933.06 ( 0.00%) 4255.52 ( 13.73%) Amean fault-both-7 7017.75 ( 0.00%) 6586.93 ( 6.14%) Amean fault-both-12 11610.51 ( 0.00%) 9162.34 * 21.09%* Amean fault-both-18 17055.85 ( 0.00%) 11530.06 * 32.40%* Amean fault-both-24 19306.27 ( 0.00%) 17956.13 ( 6.99%) Amean fault-both-30 22516.49 ( 0.00%) 15686.47 * 30.33%* Amean fault-both-32 23442.93 ( 0.00%) 16564.83 * 29.34%* The allocation success rates are much improved baseline patches Percentage huge-3 85.99 ( 0.00%) 97.96 ( 13.92%) Percentage huge-5 88.27 ( 0.00%) 96.87 ( 9.74%) Percentage huge-7 85.87 ( 0.00%) 94.53 ( 10.09%) Percentage huge-12 82.38 ( 0.00%) 98.44 ( 19.49%) Percentage huge-18 83.29 ( 0.00%) 99.14 ( 19.04%) Percentage huge-24 81.41 ( 0.00%) 97.35 ( 19.57%) Percentage huge-30 80.98 ( 0.00%) 98.05 ( 21.08%) Percentage huge-32 80.53 ( 0.00%) 97.06 ( 20.53%) That's a nearly perfect allocation success rate. The biggest impact is on the scan rates Compaction migrate scanned 55893379 19341254 Compaction free scanned 474739990 11903963 The number of pages scanned for migration was reduced by 65% and the free scanner was reduced by 97.5%. So much less work in exchange for lower latency and better success rates. The series was also evaluated using a workload that heavily fragments memory but the benefits there are also significant, albeit not presented. It was commented that we should be rethinking scanning entirely and to a large extent I agree. However, to achieve that you need a lot of this series in place first so it's best to make the linear scanners as best as possible before ripping them out. include/linux/compaction.h | 3 +- include/linux/mmzone.h | 2 + include/linux/sched.h | 4 + kernel/sched/core.c | 3 + mm/compaction.c | 1000 ++++++++++++++++++++++++++++++++++---------- mm/internal.h | 23 +- mm/migrate.c | 2 +- mm/page_alloc.c | 76 +++- 8 files changed, 888 insertions(+), 225 deletions(-)