From patchwork Fri Jan 4 12:49:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748295 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 47A0813AD for ; Fri, 4 Jan 2019 12:50:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 357BF283C3 for ; Fri, 4 Jan 2019 12:50:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 25FAB283CF; Fri, 4 Jan 2019 12:50:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 81822283C3 for ; Fri, 4 Jan 2019 12:50:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 051CF8E00D3; Fri, 4 Jan 2019 07:50:25 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 000E98E00AE; Fri, 4 Jan 2019 07:50:24 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E32BA8E00D3; Fri, 4 Jan 2019 07:50:24 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id 88B628E00AE for ; Fri, 4 Jan 2019 07:50:24 -0500 (EST) Received: by mail-ed1-f69.google.com with SMTP id t2so34858778edb.22 for ; Fri, 04 Jan 2019 04:50:24 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id; bh=JOdD5uJbEuUsdQzkaD51hbH9nJTwND3KbQpJTUbQLIA=; b=T7VzdjMevAK34MS+btoOGNh8+eh7LfsHSbu8UckY00LElkT/STs3W7qkZd7nQa+q9e Kh3Y29IHv5h63kS/H9rkJP+e+aUhq+g3bKstlES3MuqWCBNOCheaUvQKYMsRz3Nu2Qa2 BHEgA4xYGUahX0qL4qh/qrmVjWCZJrj6w4pMBjG3tq6mEkpS16Hpi6PVjIed+YvkxWNG 98H5/4L1wJptyUyIwf1JDj33btM2tbcDBDWA1nzcreOdSpp9lq/XerJ9QAODVRncfdSz +JqHD1UJj18IUPSfvRY6lrxr4+XSeShzybcDz/duHtxrOQtMn40jC3vJE0J+8jmeOGrb 8EuA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWZ8CUa9eIV0lKf0ZqTa5ocf4bqvE4AvwezRo2OoUF/cTe5r8FVu d90ABd9SOhqrQO3wK20wGPDNnQAqf8WRYZjMB75vXh59wdr8TbIws3yX5L95jA1fbgw/Du/VKci G4tYEhxChoYuCeGpDGQI2V2rUrjrDFNwJBmlnGYp8QKhp1pGIZADTrK65gmNsbRKgIA== X-Received: by 2002:a50:924e:: with SMTP id j14mr46622874eda.142.1546606223892; Fri, 04 Jan 2019 04:50:23 -0800 (PST) X-Google-Smtp-Source: AFSGD/XqUIU1HsGw/M8dyCXbEdnVO/vJpRDw393RGWa0rBkZoLCCfH3UKa8Rz8u+3MSch0YjVl/3 X-Received: by 2002:a50:924e:: with SMTP id j14mr46622837eda.142.1546606222811; Fri, 04 Jan 2019 04:50:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606222; cv=none; d=google.com; s=arc-20160816; b=u6VUvz/TDao0xXs7HXKzZpy119oGyx6tzr3W9R5ceq+RpzKsoRqBEx/faPfPLH6Ff/ HxQRF7QhkiUKLauEozj/YXf05LA9VWYClddr7jqwwzM5vOyaQmdPOOVw7/1uSz6FmwbD 9suq3EvTjINIo/wMXT+mUyJZGAJLfjK06DcvaWm5tOjkOOBUdU/WWvWtd9aaTEDzRTuX vBmiT/qz2IeoGk5YM8du+b4qdEJu7JOnJmvVbOyK2TUXgPKpWYGhq2nDTDw5zSgtTW17 A/7oUPs2HQqa+FciH829udj9wh3L/fxCPZQnmkSrbRb8snl3+FgABGS0TVtWcZuMTCED ejtw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=message-id:date:subject:cc:to:from; bh=JOdD5uJbEuUsdQzkaD51hbH9nJTwND3KbQpJTUbQLIA=; b=bLlnP5dDa1gLIEldKQJgSNO2H+hE/zGVhgUmrQ7yY6t4ARSJSlF/A92y1wvey9NO+Y VcWG59FaG9nKCHeHvgE/08v90AiCIM8p6+LNPQYJC54v2WpW0q7xC7fSEDTVncjWOioM h2NHc89Um86FtHg06ES2iqmEaaxsrbDqWWwv+2+BeZ3nOV+EP3ixR+cFYsuIh320nSYM bNSbwa2MvFdALymO0m/tyNokmM1lRe0YNaG6pKuFY6nLqlkHKyU5CJlSasUbXS9yX1oY VUCil3WrwE18jKVp1BVH/W9Xy31whYwLF75PAwY9SuTIVtG975IAjlhbXlzyPcvTqPst GT+w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp10.blacknight.com (outbound-smtp10.blacknight.com. [46.22.139.15]) by mx.google.com with ESMTPS id n1-v6si344185eji.127.2019.01.04.04.50.22 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:50:22 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) client-ip=46.22.139.15; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp10.blacknight.com (Postfix) with ESMTPS id 3A53D1C2121 for ; Fri, 4 Jan 2019 12:50:22 +0000 (GMT) Received: (qmail 28816 invoked from network); 4 Jan 2019 12:50:22 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:50:22 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 00/25] Increase success rates and reduce latency of compaction v2 Date: Fri, 4 Jan 2019 12:49:46 +0000 Message-Id: <20190104125011.16071-1-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This series reduces scan rates and success rates of compaction, primarily by using the free lists to shorten scans, better controlling of skip information and whether multiple scanners can target the same block and capturing pageblocks before being stolen by parallel requests. The series is based on the 4.21/5.0 merge window after Andrew's tree had been merged. It's known to rebase cleanly. Primarily I'm using thpscale to measure the impact of the series. The benchmark creates a large file, maps it, faults it, punches holes in the mapping so that the virtual address space is fragmented and then tries to allocate THP. It re-executes for different numbers of threads. From a fragmentation perspective, the workload is relatively benign but it does stress compaction. The overall impact on latencies for a 1-socket machine is baseline patches Amean fault-both-3 5362.80 ( 0.00%) 4446.89 * 17.08%* Amean fault-both-5 9488.75 ( 0.00%) 5660.86 * 40.34%* Amean fault-both-7 11909.86 ( 0.00%) 8549.63 * 28.21%* Amean fault-both-12 16185.09 ( 0.00%) 11508.36 * 28.90%* Amean fault-both-18 12057.72 ( 0.00%) 19013.48 * -57.69%* Amean fault-both-24 23939.95 ( 0.00%) 19676.16 * 17.81%* Amean fault-both-30 26606.14 ( 0.00%) 27363.23 ( -2.85%) Amean fault-both-32 31677.12 ( 0.00%) 23154.09 * 26.91%* While there is a glitch at the 18-thread mark, it's known that the base page allocation latency was much lower and huge pages were taking longer -- partially due a high allocation success rate. The allocation success rates are much improved baseline patches Percentage huge-3 70.93 ( 0.00%) 98.30 ( 38.60%) Percentage huge-5 56.02 ( 0.00%) 83.36 ( 48.81%) Percentage huge-7 60.98 ( 0.00%) 89.04 ( 46.01%) Percentage huge-12 73.02 ( 0.00%) 94.36 ( 29.23%) Percentage huge-18 94.37 ( 0.00%) 95.87 ( 1.58%) Percentage huge-24 84.95 ( 0.00%) 97.41 ( 14.67%) Percentage huge-30 83.63 ( 0.00%) 96.69 ( 15.62%) Percentage huge-32 81.69 ( 0.00%) 96.10 ( 17.65%) That's a nearly perfect allocation success rate. The biggest impact is on the scan rates Compaction migrate scanned 106520811 26934599 Compaction free scanned 4180735040 26584944 The number of pages scanned for migration was reduced by 74% and the free scanner was reduced by 99.36%. So much less work in exchange for lower latency and better success rates. The series was also evaluated using a workload that heavily fragments memory but the benefits there are also significant, albeit not presented. It was commented that we should be rethinking scanning entirely and to a large extent I agree. However, to achieve that you need a lot of this series in place first so it's best to make the linear scanners as best as possible before ripping them out. include/linux/compaction.h | 3 +- include/linux/gfp.h | 7 +- include/linux/mmzone.h | 2 + include/linux/sched.h | 4 + kernel/sched/core.c | 3 + mm/compaction.c | 1031 ++++++++++++++++++++++++++++++++++---------- mm/internal.h | 23 +- mm/migrate.c | 2 +- mm/page_alloc.c | 70 ++- 9 files changed, 908 insertions(+), 237 deletions(-) Acked-by: Vlastimil Babka