From patchwork Fri Jun 15 13:52:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10466541 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id BBA376020F for ; Fri, 15 Jun 2018 13:52:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A7D2F28744 for ; Fri, 15 Jun 2018 13:52:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9C50D28D8E; Fri, 15 Jun 2018 13:52:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 513C828744 for ; Fri, 15 Jun 2018 13:52:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756211AbeFONwQ (ORCPT ); Fri, 15 Jun 2018 09:52:16 -0400 Received: from outbound-smtp12.blacknight.com ([46.22.139.17]:56627 "EHLO outbound-smtp12.blacknight.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755827AbeFONwO (ORCPT ); Fri, 15 Jun 2018 09:52:14 -0400 Received: from mail.blacknight.com (pemlinmail06.blacknight.ie [81.17.255.152]) by outbound-smtp12.blacknight.com (Postfix) with ESMTPS id 132921C1826 for ; Fri, 15 Jun 2018 14:52:13 +0100 (IST) Received: (qmail 4847 invoked from network); 15 Jun 2018 13:52:12 -0000 Received: from unknown (HELO techsingularity.net) (mgorman@techsingularity.net@[37.228.237.171]) by 81.17.254.9 with ESMTPSA (DHE-RSA-AES256-SHA encrypted, authenticated); 15 Jun 2018 13:52:12 -0000 Date: Fri, 15 Jun 2018 14:52:12 +0100 From: Mel Gorman To: Jirka Hladky Cc: Jakub Racek , linux-kernel , "Rafael J. Wysocki" , Len Brown , linux-acpi@vger.kernel.org, "kkolakow@redhat.com" Subject: Re: [4.17 regression] Performance drop on kernel-4.17 visible on Stream, Linpack and NAS parallel benchmarks Message-ID: <20180615135212.wq45co7ootvdeo2f@techsingularity.net> References: <20180611141113.pfuttg7npch3jtg6@techsingularity.net> <20180614083640.dekqhsopoefnfhb4@techsingularity.net> <20180615112522.3wujbq7bajof57qx@techsingularity.net> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: NeoMutt/20170912 (1.9.0) Sender: linux-acpi-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Fri, Jun 15, 2018 at 02:23:17PM +0200, Jirka Hladky wrote: > I added configurations that used half of the CPUs. However, that would > > mean it fits too nicely within sockets. I've added another set for one > > third of the CPUs and scheduled the tests. Unfortunately, they will not > > complete quickly as my test grid has a massive backlog of work. > > > We always use the number of threads being an integer multiple of the number > of sockets. With another number of threads, we have seen the bigger > variation in results (that's variation between subsequent runs of the same > test). > It's not immediately obvious what's special about those numbers. I did briefly recheck the variability of NAS on one of the machines but the coefficient of variance was usually quite low with occasional outliers of +/- 5% or +/- 7%. Anyway, it's a side-issue. > Nice one, thanks. It's fairly clear that rate limiting may be a major > > component and it's worth testing with the ratelimit increased. Given that > > there have been a lot of improvements on locality and corner cases since > > the rate limit was first introduced, it may also be worth considering > > elimintating the rate limiting entirely and see what falls out. > > > How can we tune mm_numa_migrate_ratelimit? It doesn't seem to be a runtime > tunable nor kernel boot parameter. Could you please share some hints on how > to change it and what value to use? I would be interested to try it out. > It's not runtime tunable I'm afraid. It's a code change and recompile. For example the following allows more pages to be migrated within a 100ms window. diff --git a/mm/migrate.c b/mm/migrate.c index 8c0af0f7cab1..edb550493f06 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1862,7 +1862,7 @@ static struct page *alloc_misplaced_dst_page(struct page *page, * window of time. Default here says do not migrate more than 1280M per second. */ static unsigned int migrate_interval_millisecs __read_mostly = 100; -static unsigned int ratelimit_pages __read_mostly = 128 << (20 - PAGE_SHIFT); +static unsigned int ratelimit_pages __read_mostly = 512 << (20 - PAGE_SHIFT); /* Returns true if the node is migrate rate-limited after the update */ static bool numamigrate_update_ratelimit(pg_data_t *pgdat,