From patchwork Fri Jan 4 12:49:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748297 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 20A3913B5 for ; Fri, 4 Jan 2019 12:50:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0FA16283C3 for ; Fri, 4 Jan 2019 12:50:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 032C5283CF; Fri, 4 Jan 2019 12:50:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 75962283C3 for ; Fri, 4 Jan 2019 12:50:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C8B28E00D4; Fri, 4 Jan 2019 07:50:34 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9768F8E00AE; Fri, 4 Jan 2019 07:50:34 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 88DFC8E00D4; Fri, 4 Jan 2019 07:50:34 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id 2F8768E00AE for ; Fri, 4 Jan 2019 07:50:34 -0500 (EST) Received: by mail-ed1-f69.google.com with SMTP id d41so34696291eda.12 for ; Fri, 04 Jan 2019 04:50:34 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=3eQJIJGK+o1ltUbhkQOc59OoBbzP/RrSV2DNYXQ5trs=; b=XQmsN4cNSLA5yHi1PJ8GBjOmMrbPJPQnpTgUpHABvhdGQEPqXzVRDK53O90mLmANds 6Go1d/PcHaq8431QP+Odt67SFxt7LdOvxwRxTiQ5zVJZ1N3ICqTulVC+iKk2rEzUEbmK thTab+E8chZSAXMNSYCZ8nJA4DT6rL5k9VLpAN6l5o1R46DFjtWb343PiV80kDjRH2NI +pj0fArUTDAuDQ/VNGP8rqJ0e9cWJHj/BB0lPx9kQdzF36uGYeq85ZrRsu/swAZ+Fbjb rmmZr4hQh6bfPgFG5P5ZmQUVAn44tVV7L8pQEV+F1Xc7OtfdfWyy+pLgLExiodnHUOiM 8bUQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWYkYUxUy9rdrEDJhHvjL8PU+8wKTliMhNpTPY4ajVHpE9+vK49R a+7zJWnqeL8F3rLfut3Sd3zoFp9tCiZ47tsi8E/6scv8HLpBBVNE3oOvxjsu8eTHqKMYSB6nkPG ID+Sn8IFKQSUe9JcGduLE+kJYZHEyBrPaDvjsy/SOHezByy8bvO/Dv4D2uoISNgawVw== X-Received: by 2002:a50:f489:: with SMTP id s9mr45665427edm.101.1546606233707; Fri, 04 Jan 2019 04:50:33 -0800 (PST) X-Google-Smtp-Source: AFSGD/UYXbCO98EKBhcO0ZR/fHfrwV7OwGR8bpX6T/HdmH21U3C0fKpKJBmQlY9lZeYyQcN2sBxO X-Received: by 2002:a50:f489:: with SMTP id s9mr45665396edm.101.1546606232868; Fri, 04 Jan 2019 04:50:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606232; cv=none; d=google.com; s=arc-20160816; b=WYW1bR7VjOEqmcHNCdp69Ak8c897Na7uPFpzU0MtVmZIC85fPd/dDhvFJa1KUIDtSw SuUB4UeojZYjX8Ika2IY9kZvk/oD+xI72mvC8gw8Fm0Ehzy4oJHIUYQCXD+k6O2hJrpJ yzcnmZBv8FqDuN3gyRoMIlOnyJ6C1s5M9JLKTz0NmKson/4hf0gNb4uH1QbAuQpoY3aV H+QgDnF2ZJVdsGfxGUMKwzelA6hF/Oj2ZPQ4iOI2VU/cNnmJI6mWbrMW5VH/E492YunR 19ERKbgIs06Qp6+2qEZYYgUz/A7NAp0YebWbDC0MB5DVpswdPVsBd5RySmgVaLar+Pfx 0LMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=3eQJIJGK+o1ltUbhkQOc59OoBbzP/RrSV2DNYXQ5trs=; b=w+9H5T2eu2TQay/0wk3kphxLJL8TkU3Yu6oPhfKoXaXGxKU9Du3NCaQ4d/93BOjW6h 4jG0AH6SiYkoFJF5PDfqDBVdy5nRxNmYfcj4cGuyCxxmsC51zcxzjHr4YFHG0Ry5Bc9o WkRYjCziey4UIYNQGNyinQYXTNda1oLyo+XVvur5zqzwVFGTBbF34e5ABnfIC4B7pKq4 r9napAXh7n753/YGkkUYVPP/bfgN6iH+mdllpBtWp6XqWB9ZzurkUNCE7kgjIKhoTRyz NmRneGstcUsneyXkhmkaVIPalHJFM2RpzL+a6jbIT7VN5XXNsPRptEKwiorLdw8LZbho KmbQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp04.blacknight.com (outbound-smtp04.blacknight.com. [81.17.249.35]) by mx.google.com with ESMTPS id y6si2193096edi.116.2019.01.04.04.50.32 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:50:32 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) client-ip=81.17.249.35; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp04.blacknight.com (Postfix) with ESMTPS id 6B74898837 for ; Fri, 4 Jan 2019 12:50:32 +0000 (UTC) Received: (qmail 29438 invoked from network); 4 Jan 2019 12:50:32 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:50:32 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 01/25] mm, compaction: Shrink compact_control Date: Fri, 4 Jan 2019 12:49:47 +0000 Message-Id: <20190104125011.16071-2-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The isolate and migrate scanners should never isolate more than a pageblock of pages so unsigned int is sufficient saving 8 bytes on a 64-bit build. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/internal.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index f4a7bb02decf..5ddf5d3771a0 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -184,8 +184,8 @@ struct compact_control { struct list_head freepages; /* List of free pages to migrate to */ struct list_head migratepages; /* List of pages being migrated */ struct zone *zone; - unsigned long nr_freepages; /* Number of isolated free pages */ - unsigned long nr_migratepages; /* Number of pages to migrate */ + unsigned int nr_freepages; /* Number of isolated free pages */ + unsigned int nr_migratepages; /* Number of pages to migrate */ unsigned long total_migrate_scanned; unsigned long total_free_scanned; unsigned long free_pfn; /* isolate_freepages search base */ From patchwork Fri Jan 4 12:49:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748299 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0D69A13B5 for ; Fri, 4 Jan 2019 12:50:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EFD72283C9 for ; Fri, 4 Jan 2019 12:50:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DE73A28434; Fri, 4 Jan 2019 12:50:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 03027283C9 for ; Fri, 4 Jan 2019 12:50:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1D8538E00D5; Fri, 4 Jan 2019 07:50:45 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 187848E00AE; Fri, 4 Jan 2019 07:50:45 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A00C8E00D5; Fri, 4 Jan 2019 07:50:45 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id A4F288E00AE for ; Fri, 4 Jan 2019 07:50:44 -0500 (EST) Received: by mail-ed1-f72.google.com with SMTP id d41so34696447eda.12 for ; Fri, 04 Jan 2019 04:50:44 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=SjNbp0igOurjj96TnaHeZWGR5AZ0iL/+N2T6YDn5F3c=; b=nZiAwIN/1FhxiafPj+MkWDK5Wt/VSIgGpy1p+0uuoQkWZNanCu3iSlIxxJdmmEZ2JK VP5p7vIpKQxWDDNQwiohaZJB/9BSfgR4ZHfaFPUTMmzOTQXIineBNUw79hKQVoczG5U8 j7kksNiGcXSwBJj2HkJCMTtwohTyJTALVrEc4h0EJXFhpUSgmirxX7KCHWig2xSodkuN hWKE9FzsWP6NkPN9antdTOjSSQGmBr/9esG6x5AY6nuh7icDF55U0uVEx66EqfX+5hhy CsLKPVoiqgfheoqeaQKMa9IXYPQ4JVc9kPWTmVRZsdej5TbyBcK0+gl48O2qlM+nGio1 1ScQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWZf3diKxODef6c7McUSMMfRStolZwHHi6hqa9WfzRe+d/zSfwCe yKChwhXtwJygMPaFevBOxfJd8xjjhvWVZnmhMa+VCvgjSrbp76cTSddWawA8ltWtFxLBWZYIJvb JlgDGfMzWQjk22SifzSg6z68/nVcr9/S4bjP5U1mzpyAolajCBpyWHhwECIuR8Ycxkw== X-Received: by 2002:a17:906:d054:: with SMTP id bo20-v6mr39001139ejb.239.1546606244171; Fri, 04 Jan 2019 04:50:44 -0800 (PST) X-Google-Smtp-Source: AFSGD/U2AxY4PQ5htpieMY2mqc6fpJrDFQdm9qNDi0QS6JbftwY0XCK8ris8kDQHjcLzqVstwORD X-Received: by 2002:a17:906:d054:: with SMTP id bo20-v6mr39001097ejb.239.1546606243058; Fri, 04 Jan 2019 04:50:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606243; cv=none; d=google.com; s=arc-20160816; b=FE0Kv02H2UxwwpwLIaVDAeGLWAkxuwsDNYRLFIHCONfDh336kf3pRdkxj89s0ENflj y0oVkmLxjmdmW3gM4Rq/MnnfgJEMDfVb8QRn3K6LOMujiHsqF+e0+pt4c0jfkdhF2o6e MCvQf7CVuA/PlCBaHZr71U+ycXHYWUjn3eQJ28WtNbKOL0N67so6/niAWsJmmLdeZjP8 rHg8dqGOde/ZG3N3AJzmdqijYKDT43BCKr0XemZfQlNaV0bZq0lw9AcVhk9uff8JfJfN gC9oHbz+qNHZhWToKSMS3pafB49TT27XT/xgoz+eNhYdulmYTshyIWZCLel3CPyvqWSk YETw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=SjNbp0igOurjj96TnaHeZWGR5AZ0iL/+N2T6YDn5F3c=; b=I/SQBm9uW+kFvVWFsSrWOGNT/RwGR/1tzjEMibbAdcalFqdwKbphCbnzQIg/i4rr2b GAqznfyz4i4zOGMAkoIhY2FI/xdm3qwppTnh8fxMP0KVROnIUxdjKYVxCMzMnQOJFJOY er3ACooXTaOEaGWtN3WX797JkHv59IYsrvrHbHV+LsITpiRgeSBv9q/p/e1hHwArONCv 28F30Exk35B0Pg3+akMTatczWWbjq4G01zNQw63A7JxB1P1h7JrmNA/LI0mwUQguMKQw EZzimS0kTu7BSlChj4vl5imCyrs1FamdrtDjnBCNBF030vDoGMQdyo+0gAowHeTlPsCl Qi0Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp02.blacknight.com (outbound-smtp02.blacknight.com. [81.17.249.8]) by mx.google.com with ESMTPS id q23si3893731eds.78.2019.01.04.04.50.42 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:50:43 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) client-ip=81.17.249.8; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp02.blacknight.com (Postfix) with ESMTPS id 9CC0D9855B for ; Fri, 4 Jan 2019 12:50:42 +0000 (UTC) Received: (qmail 29960 invoked from network); 4 Jan 2019 12:50:42 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:50:42 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 02/25] mm, compaction: Rearrange compact_control Date: Fri, 4 Jan 2019 12:49:48 +0000 Message-Id: <20190104125011.16071-3-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP compact_control spans two cache lines with write-intensive lines on both. Rearrange so the most write-intensive fields are in the same cache line. This has a negligible impact on the overall performance of compaction and is more a tidying exercise than anything. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/internal.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 5ddf5d3771a0..9437ba5791db 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -183,14 +183,14 @@ extern int user_min_free_kbytes; struct compact_control { struct list_head freepages; /* List of free pages to migrate to */ struct list_head migratepages; /* List of pages being migrated */ - struct zone *zone; unsigned int nr_freepages; /* Number of isolated free pages */ unsigned int nr_migratepages; /* Number of pages to migrate */ - unsigned long total_migrate_scanned; - unsigned long total_free_scanned; unsigned long free_pfn; /* isolate_freepages search base */ unsigned long migrate_pfn; /* isolate_migratepages search base */ unsigned long last_migrated_pfn;/* Not yet flushed page being freed */ + struct zone *zone; + unsigned long total_migrate_scanned; + unsigned long total_free_scanned; const gfp_t gfp_mask; /* gfp mask of a direct compactor */ int order; /* order a direct compactor needs */ int migratetype; /* migratetype of direct compactor */ From patchwork Fri Jan 4 12:49:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748301 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CC14A14E5 for ; Fri, 4 Jan 2019 12:50:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BC3F7283C9 for ; Fri, 4 Jan 2019 12:50:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B078E28434; Fri, 4 Jan 2019 12:50:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 38490283C9 for ; Fri, 4 Jan 2019 12:50:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 311CE8E00D6; Fri, 4 Jan 2019 07:50:55 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2E71A8E00AE; Fri, 4 Jan 2019 07:50:55 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1B03F8E00D6; Fri, 4 Jan 2019 07:50:55 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id B7DC28E00AE for ; Fri, 4 Jan 2019 07:50:54 -0500 (EST) Received: by mail-ed1-f72.google.com with SMTP id m19so35039164edc.6 for ; Fri, 04 Jan 2019 04:50:54 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=VYm1JewsY9/4gPesnVldquFFzwNaTy3YVh6A0ce8hdY=; b=eT3GJpuDnBMAajk3zlNQ4iEEWtWFoI9E+o5DU5sL4qw1gMOJZH+fSia+hLlhw/JnI/ yAppcu22qP0onSHUhegChzNXxwXUq9bfM7zXbS0uwMgh//J9u/k0HTVTdmD/PiALDj0W 4mAgMv0rymRzKDpZ5HXYytTAfpOJ4iASgzCDuQGQVPp7ZPq+vpzBYMDfo8vutdyim9G8 GkK57Xp7e3DpFjbwTfgZNBB8vSXZ1huVChFhNQp1ufPWYBKkT1WpbRdbF1kKvBnGckPu JDb/pzEsguBz9kq2WtX2Yu61w1wnJ4/ac3IJkbUjRbKXSnyKtlRJZ909rzvP5lrY0TG8 8Qnw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWZXFdIyHLVuBsXa2aGwTq2TuZS4Qp+cQP4u5K1Lx9izcBlwS5fJ ncISMKKA13yklwyl1XVYsssiRg9vs38NHKpnf/j+PuNxigb9OGfajg+5IBg6BexyUeNAlR0YOVz v/8c4hgw10Uk1gP+Y1ouDegn3/jygkkdvRiU8oN9AA5aEF9Ihv/JY3HNiMa5/nulfIg== X-Received: by 2002:a17:906:3b4b:: with SMTP id h11-v6mr38709235ejf.183.1546606254222; Fri, 04 Jan 2019 04:50:54 -0800 (PST) X-Google-Smtp-Source: AFSGD/XMREJYWiusC70j7KDx7ceSqIHkbU4yluj9xtAbO3Q94eA278hLDv/mdCLK7f/j6W4yPsh1 X-Received: by 2002:a17:906:3b4b:: with SMTP id h11-v6mr38709190ejf.183.1546606253172; Fri, 04 Jan 2019 04:50:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606253; cv=none; d=google.com; s=arc-20160816; b=bL2PXb5UgcbwCJ0i1pahSbLAgm/KXb9HQ+xyLsaMyIdDbVIak/sYJQUh3jLfS3eqqA u9Wg8dnTkIWPXNtFC0EI+JnArI/5jS4edsRGdRmvkN3oFOB7/yGt5R7RagSKWPiXYuHI fBPldZT7gIUzxIevO5jVEg46L34JXMvfwCBhGjcx4ABMEnkS3r05jBtblFlwJqfjo3Ca XPRnR03DLnQ3bBZKK15Gtkqg+x0YN7Yqs8aPoTDA1hENCX5c/l3SBq7NGau8P83tVmpN MGsHIyU/I3kABIjmQ4f/D0880hSm27UomGttqGReTk+vUpzNg/RgJPwaOHGnewQtcpEn JI8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=VYm1JewsY9/4gPesnVldquFFzwNaTy3YVh6A0ce8hdY=; b=eb+Dd2UjD/oMhxBaISQHdr0ZzW1+GiChdMYm1Lzq+5MZac5KamATkushgW4e4dJ/6Q YKEmBRXnhwAEYCcueUi77d5rHdk6jTQN9ZBBYIKm9BrhNYE71GJo0kd7fhTKksV3sNHE wu5IUSyGWzTLDYO4MQ9kzcnqK+f6vQb/arOq5sB8/jbLuvUFO4f1RxHeAkLZtf/JhBm0 H59lE3F9jObn2Ho2rnE2mSKn18tcQTTtIalO7TF1eBGs1w9Qic5LCKe/sWw7e2AwusY+ xIJXwhPyaZk5ZXNnqnt/DcfCA7wDOkLmoU/VvADCDEGcTE8f4AM9zg5wx4qTUuaOP9fA YUsw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp12.blacknight.com (outbound-smtp12.blacknight.com. [46.22.139.17]) by mx.google.com with ESMTPS id s34si4249719edb.417.2019.01.04.04.50.52 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:50:53 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) client-ip=46.22.139.17; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp12.blacknight.com (Postfix) with ESMTPS id CD9D81C1D6B for ; Fri, 4 Jan 2019 12:50:52 +0000 (GMT) Received: (qmail 30568 invoked from network); 4 Jan 2019 12:50:52 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:50:52 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 03/25] mm, compaction: Remove last_migrated_pfn from compact_control Date: Fri, 4 Jan 2019 12:49:49 +0000 Message-Id: <20190104125011.16071-4-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The last_migrated_pfn field is a bit dubious as to whether it really helps but either way, the information from it can be inferred without increasing the size of compact_control so remove the field. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 25 +++++++++---------------- mm/internal.h | 1 - 2 files changed, 9 insertions(+), 17 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index ef29490b0f46..fb4d9f52ed56 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -886,15 +886,6 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, cc->nr_migratepages++; nr_isolated++; - /* - * Record where we could have freed pages by migration and not - * yet flushed them to buddy allocator. - * - this is the lowest page that was isolated and likely be - * then freed by migration. - */ - if (!cc->last_migrated_pfn) - cc->last_migrated_pfn = low_pfn; - /* Avoid isolating too much */ if (cc->nr_migratepages == COMPACT_CLUSTER_MAX) { ++low_pfn; @@ -918,7 +909,6 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, } putback_movable_pages(&cc->migratepages); cc->nr_migratepages = 0; - cc->last_migrated_pfn = 0; nr_isolated = 0; } @@ -1539,6 +1529,7 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro enum compact_result ret; unsigned long start_pfn = zone->zone_start_pfn; unsigned long end_pfn = zone_end_pfn(zone); + unsigned long last_migrated_pfn; const bool sync = cc->mode != MIGRATE_ASYNC; cc->migratetype = gfpflags_to_migratetype(cc->gfp_mask); @@ -1584,7 +1575,7 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro cc->whole_zone = true; } - cc->last_migrated_pfn = 0; + last_migrated_pfn = 0; trace_mm_compaction_begin(start_pfn, cc->migrate_pfn, cc->free_pfn, end_pfn, sync); @@ -1593,12 +1584,14 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro while ((ret = compact_finished(zone, cc)) == COMPACT_CONTINUE) { int err; + unsigned long start_pfn = cc->migrate_pfn; switch (isolate_migratepages(zone, cc)) { case ISOLATE_ABORT: ret = COMPACT_CONTENDED; putback_movable_pages(&cc->migratepages); cc->nr_migratepages = 0; + last_migrated_pfn = 0; goto out; case ISOLATE_NONE: /* @@ -1608,6 +1601,7 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro */ goto check_drain; case ISOLATE_SUCCESS: + last_migrated_pfn = start_pfn; ; } @@ -1639,8 +1633,7 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro cc->migrate_pfn = block_end_pfn( cc->migrate_pfn - 1, cc->order); /* Draining pcplists is useless in this case */ - cc->last_migrated_pfn = 0; - + last_migrated_pfn = 0; } } @@ -1652,18 +1645,18 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro * compact_finished() can detect immediately if allocation * would succeed. */ - if (cc->order > 0 && cc->last_migrated_pfn) { + if (cc->order > 0 && last_migrated_pfn) { int cpu; unsigned long current_block_start = block_start_pfn(cc->migrate_pfn, cc->order); - if (cc->last_migrated_pfn < current_block_start) { + if (last_migrated_pfn < current_block_start) { cpu = get_cpu(); lru_add_drain_cpu(cpu); drain_local_pages(zone); put_cpu(); /* No more flushing until we migrate again */ - cc->last_migrated_pfn = 0; + last_migrated_pfn = 0; } } diff --git a/mm/internal.h b/mm/internal.h index 9437ba5791db..c6f794ad21a9 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -187,7 +187,6 @@ struct compact_control { unsigned int nr_migratepages; /* Number of pages to migrate */ unsigned long free_pfn; /* isolate_freepages search base */ unsigned long migrate_pfn; /* isolate_migratepages search base */ - unsigned long last_migrated_pfn;/* Not yet flushed page being freed */ struct zone *zone; unsigned long total_migrate_scanned; unsigned long total_free_scanned; From patchwork Fri Jan 4 12:49:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748303 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6869D13B5 for ; Fri, 4 Jan 2019 12:51:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 58A14282E8 for ; Fri, 4 Jan 2019 12:51:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4B727283C9; Fri, 4 Jan 2019 12:51:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 93CAB283C3 for ; Fri, 4 Jan 2019 12:51:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A8A308E00D7; Fri, 4 Jan 2019 07:51:05 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A395F8E00AE; Fri, 4 Jan 2019 07:51:05 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 903E58E00D7; Fri, 4 Jan 2019 07:51:05 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id 379D28E00AE for ; Fri, 4 Jan 2019 07:51:05 -0500 (EST) Received: by mail-ed1-f69.google.com with SMTP id c3so35318181eda.3 for ; Fri, 04 Jan 2019 04:51:05 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=QkeDWIo5W+IXHb2YUSNPGbgmABrlDMhjARz9jT2lisc=; b=ZZ1sEXUeZy7lCiK+u4Sxk2Q5ytBOZKfNfJ2b4StemF88sqpsEhzDPiMzkww8NSNpMN nONdcm4t3kyRZFqg2+5sZPzfAVK4zAhO+yuhipD2DLpSQVvQcIzIazu21nF/+cANaYPv ptV/JPUnow2rI8wSxcTJZGwzGdbqTNowjEQ5y4Ter4R3rSCEgKD293eRRCYMKboULwir PEVFkN6w8hmy+GAIXmv19+We38amQpIXbD+pBOWAHyj/XaMfbI6eLNLd+0/GC18SBWxy /a3OSEbpZoP66I9/DsnX9vSaMjV6Lx4i4IdmlUaO6/bHAXa/6OTKW1zZUz28u4YNqA4t oLDw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWYrPeevR0vJZBuZpoKT0sTuAWtkvGIPscZEWjmg8rKnoOGGsQKA H5Z60tLUYKwermLf/WnpRCyyRRWWiCns4SwncpMqyizNALr1httctgVyqkBaDx4nr5T2Mh85mKj ftirVQomlxXRQTHdzpV4qI+Y/XeXsbwUeg2MiYX/bzlFTv0nkbRPFWXLfZqkSxf7COg== X-Received: by 2002:a50:e0cb:: with SMTP id j11mr46258714edl.195.1546606264704; Fri, 04 Jan 2019 04:51:04 -0800 (PST) X-Google-Smtp-Source: AFSGD/WXjDh4m34f4dcK6+pOMs5UIDaBsF+5+Z+VKXPkJIDTYtsP1altrWILZrUVBSSaRll/SYHv X-Received: by 2002:a50:e0cb:: with SMTP id j11mr46258667edl.195.1546606263506; Fri, 04 Jan 2019 04:51:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606263; cv=none; d=google.com; s=arc-20160816; b=RXlgNc98tg1b+dLCGvip5VGgn0gB+tZMypnYlidXy4odueJ+6aYeymlyb9RVGU2y11 Wv2OJ5Z/1NgaVAKOPrg/teBqYMAYK6k1Ve8LD1MtzbOdTe9aZY4PHliOdafshwavjDuj QdDLmXX7b1xIrXpHY6HlmFCCdHKeZGW1R8Y23gcyvPyRuET38Ee8PQvI7TzC3/jxwSrb erBpRDVtlVsB7OOIws6qd9Qi+vlXUgWfsKi5fKtlAD2OP8cE009StBKmsjXkYSh08jx+ 41LQtCV1TR855nKCoVmVgDp7tK4o+g7eN0427K+/jzbE+TS181MRHgCYyDcgxpmIHeyi mprg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=QkeDWIo5W+IXHb2YUSNPGbgmABrlDMhjARz9jT2lisc=; b=rc/mbZwxrri9fn5fmRxeRnnwZnSYbtU5MSCNXSTdZLFiv3kGaaXd13OeA5oec6/5y2 1/zKdEbnlTSwqWf0pnU2onPn5pOsEtXr073qWIeGuQPiOpV68So95yDI4DIOG7SwUB1E +czSsNg6xo5Y6NPFPClEGzKRT838ArJqCfuOivLmITwy31UVBv+5+PzrvPQyA+amVFVQ fWs3zDSzFtYTp7CHB7yg50zbQVF/bM6Vs/d8BKVnLiAnUD+5RachmCN1PSSpyX5acZFU q/GQ2+YWSczhZQeVmY0XCJ0kBSzXLrJPsXydk5MNT3U/sDl2HGcfNUMIBLzFHwGrjkrg P2sg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp08.blacknight.com (outbound-smtp08.blacknight.com. [46.22.139.13]) by mx.google.com with ESMTPS id o21-v6si1218074ejs.199.2019.01.04.04.51.03 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:51:03 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) client-ip=46.22.139.13; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp08.blacknight.com (Postfix) with ESMTPS id 08B571C213C for ; Fri, 4 Jan 2019 12:51:03 +0000 (GMT) Received: (qmail 31287 invoked from network); 4 Jan 2019 12:51:02 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:51:02 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 04/25] mm, compaction: Remove unnecessary zone parameter in some instances Date: Fri, 4 Jan 2019 12:49:50 +0000 Message-Id: <20190104125011.16071-5-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP A zone parameter is passed into a number of top-level compaction functions despite the fact that it's already in cache_control. This is harmless but it did need an audit to check if zone actually ever changes meaningfully. This patches removes the parameter in a number of top-level functions. The change could be much deeper but this was enough to briefly clarify the flow. No functional change. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 54 ++++++++++++++++++++++++++---------------------------- 1 file changed, 26 insertions(+), 28 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index fb4d9f52ed56..7acb43f07303 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1300,8 +1300,7 @@ static inline bool is_via_compact_memory(int order) return order == -1; } -static enum compact_result __compact_finished(struct zone *zone, - struct compact_control *cc) +static enum compact_result __compact_finished(struct compact_control *cc) { unsigned int order; const int migratetype = cc->migratetype; @@ -1312,7 +1311,7 @@ static enum compact_result __compact_finished(struct zone *zone, /* Compaction run completes if the migrate and free scanner meet */ if (compact_scanners_met(cc)) { /* Let the next compaction start anew. */ - reset_cached_positions(zone); + reset_cached_positions(cc->zone); /* * Mark that the PG_migrate_skip information should be cleared @@ -1321,7 +1320,7 @@ static enum compact_result __compact_finished(struct zone *zone, * based on an allocation request. */ if (cc->direct_compaction) - zone->compact_blockskip_flush = true; + cc->zone->compact_blockskip_flush = true; if (cc->whole_zone) return COMPACT_COMPLETE; @@ -1345,7 +1344,7 @@ static enum compact_result __compact_finished(struct zone *zone, /* Direct compactor: Is a suitable page free? */ for (order = cc->order; order < MAX_ORDER; order++) { - struct free_area *area = &zone->free_area[order]; + struct free_area *area = &cc->zone->free_area[order]; bool can_steal; /* Job done if page is free of the right migratetype */ @@ -1391,13 +1390,12 @@ static enum compact_result __compact_finished(struct zone *zone, return COMPACT_NO_SUITABLE_PAGE; } -static enum compact_result compact_finished(struct zone *zone, - struct compact_control *cc) +static enum compact_result compact_finished(struct compact_control *cc) { int ret; - ret = __compact_finished(zone, cc); - trace_mm_compaction_finished(zone, cc->order, ret); + ret = __compact_finished(cc); + trace_mm_compaction_finished(cc->zone, cc->order, ret); if (ret == COMPACT_NO_SUITABLE_PAGE) ret = COMPACT_CONTINUE; @@ -1524,16 +1522,16 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, return false; } -static enum compact_result compact_zone(struct zone *zone, struct compact_control *cc) +static enum compact_result compact_zone(struct compact_control *cc) { enum compact_result ret; - unsigned long start_pfn = zone->zone_start_pfn; - unsigned long end_pfn = zone_end_pfn(zone); + unsigned long start_pfn = cc->zone->zone_start_pfn; + unsigned long end_pfn = zone_end_pfn(cc->zone); unsigned long last_migrated_pfn; const bool sync = cc->mode != MIGRATE_ASYNC; cc->migratetype = gfpflags_to_migratetype(cc->gfp_mask); - ret = compaction_suitable(zone, cc->order, cc->alloc_flags, + ret = compaction_suitable(cc->zone, cc->order, cc->alloc_flags, cc->classzone_idx); /* Compaction is likely to fail */ if (ret == COMPACT_SUCCESS || ret == COMPACT_SKIPPED) @@ -1546,8 +1544,8 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro * Clear pageblock skip if there were failures recently and compaction * is about to be retried after being deferred. */ - if (compaction_restarting(zone, cc->order)) - __reset_isolation_suitable(zone); + if (compaction_restarting(cc->zone, cc->order)) + __reset_isolation_suitable(cc->zone); /* * Setup to move all movable pages to the end of the zone. Used cached @@ -1559,16 +1557,16 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro cc->migrate_pfn = start_pfn; cc->free_pfn = pageblock_start_pfn(end_pfn - 1); } else { - cc->migrate_pfn = zone->compact_cached_migrate_pfn[sync]; - cc->free_pfn = zone->compact_cached_free_pfn; + cc->migrate_pfn = cc->zone->compact_cached_migrate_pfn[sync]; + cc->free_pfn = cc->zone->compact_cached_free_pfn; if (cc->free_pfn < start_pfn || cc->free_pfn >= end_pfn) { cc->free_pfn = pageblock_start_pfn(end_pfn - 1); - zone->compact_cached_free_pfn = cc->free_pfn; + cc->zone->compact_cached_free_pfn = cc->free_pfn; } if (cc->migrate_pfn < start_pfn || cc->migrate_pfn >= end_pfn) { cc->migrate_pfn = start_pfn; - zone->compact_cached_migrate_pfn[0] = cc->migrate_pfn; - zone->compact_cached_migrate_pfn[1] = cc->migrate_pfn; + cc->zone->compact_cached_migrate_pfn[0] = cc->migrate_pfn; + cc->zone->compact_cached_migrate_pfn[1] = cc->migrate_pfn; } if (cc->migrate_pfn == start_pfn) @@ -1582,11 +1580,11 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro migrate_prep_local(); - while ((ret = compact_finished(zone, cc)) == COMPACT_CONTINUE) { + while ((ret = compact_finished(cc)) == COMPACT_CONTINUE) { int err; unsigned long start_pfn = cc->migrate_pfn; - switch (isolate_migratepages(zone, cc)) { + switch (isolate_migratepages(cc->zone, cc)) { case ISOLATE_ABORT: ret = COMPACT_CONTENDED; putback_movable_pages(&cc->migratepages); @@ -1653,7 +1651,7 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro if (last_migrated_pfn < current_block_start) { cpu = get_cpu(); lru_add_drain_cpu(cpu); - drain_local_pages(zone); + drain_local_pages(cc->zone); put_cpu(); /* No more flushing until we migrate again */ last_migrated_pfn = 0; @@ -1678,8 +1676,8 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro * Only go back, not forward. The cached pfn might have been * already reset to zone end in compact_finished() */ - if (free_pfn > zone->compact_cached_free_pfn) - zone->compact_cached_free_pfn = free_pfn; + if (free_pfn > cc->zone->compact_cached_free_pfn) + cc->zone->compact_cached_free_pfn = free_pfn; } count_compact_events(COMPACTMIGRATE_SCANNED, cc->total_migrate_scanned); @@ -1716,7 +1714,7 @@ static enum compact_result compact_zone_order(struct zone *zone, int order, INIT_LIST_HEAD(&cc.freepages); INIT_LIST_HEAD(&cc.migratepages); - ret = compact_zone(zone, &cc); + ret = compact_zone(&cc); VM_BUG_ON(!list_empty(&cc.freepages)); VM_BUG_ON(!list_empty(&cc.migratepages)); @@ -1834,7 +1832,7 @@ static void compact_node(int nid) INIT_LIST_HEAD(&cc.freepages); INIT_LIST_HEAD(&cc.migratepages); - compact_zone(zone, &cc); + compact_zone(&cc); VM_BUG_ON(!list_empty(&cc.freepages)); VM_BUG_ON(!list_empty(&cc.migratepages)); @@ -1976,7 +1974,7 @@ static void kcompactd_do_work(pg_data_t *pgdat) if (kthread_should_stop()) return; - status = compact_zone(zone, &cc); + status = compact_zone(&cc); if (status == COMPACT_SUCCESS) { compaction_defer_reset(zone, cc.order, false); From patchwork Fri Jan 4 12:49:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748305 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 810D213B5 for ; Fri, 4 Jan 2019 12:51:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6D2E227F86 for ; Fri, 4 Jan 2019 12:51:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5B9EF283A5; Fri, 4 Jan 2019 12:51:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C262B27F86 for ; Fri, 4 Jan 2019 12:51:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 003118E00D8; Fri, 4 Jan 2019 07:51:27 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EA6B18E00AE; Fri, 4 Jan 2019 07:51:26 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D97C78E00D8; Fri, 4 Jan 2019 07:51:26 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id 7705F8E00AE for ; Fri, 4 Jan 2019 07:51:26 -0500 (EST) Received: by mail-ed1-f71.google.com with SMTP id c18so34708199edt.23 for ; Fri, 04 Jan 2019 04:51:26 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=gh9TjtpXh8D7hhgVdeb1KwN93Adcgexh1IigcGMkqmM=; b=eCdBpHzBb8hf85f+R4c/SDyRk0S6GkASRbnwLpixiLOnbUd9SWQC5f4ntyyJaCgPe5 L+FuLsYWQUNJRVca/tMK3jKh9yh2C+7ZaxrW8XTS8duRPD4Hh2DfeSwOfJKXPhMV7Rtn rDcA8age/RtAT1V9L80VmLqv8Za+SyNguwm+jcJbMoyq4kqam5hDEw0cQFxYCSYETtbF 8Rc59+cVDJ7OC7AS4AywZi4y/JgR7wcGvieQXQfQKnPWEcKbz47vK4bZWr+y0RB2EqzG b6H4/Chd6UzkjRs1rXSliXEarcLWZIEeoxPK8iKOkc6+u5BugkhviES8IhFLFfHstFN7 fsrA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.193 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWbbpm8ocAmUAa6XRZjz1R2DYZj8Td6/1Bq44FKDgsozatVASjSX jhpc2pzAplB0mWt236K9292dXTb+k6+dZFhv38F8depOwSFjvMYEbJBpxAEUS+uIsNJ6Ap7CBxJ oKzsw9k6oP9w76kPoTwZqxNdxfRJzc3QJaM4hgBO9md//6jPfRkIe57WkK4MoEgrv4w== X-Received: by 2002:a50:b559:: with SMTP id z25mr45445065edd.239.1546606285993; Fri, 04 Jan 2019 04:51:25 -0800 (PST) X-Google-Smtp-Source: AFSGD/WfKCbtkagnYmeMhsM66bu9Vneut7Sjdw0+VfHcWHMhcnH8HLJUHh7wVpnqg148SvEj5tVc X-Received: by 2002:a50:b559:: with SMTP id z25mr45445026edd.239.1546606285233; Fri, 04 Jan 2019 04:51:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606285; cv=none; d=google.com; s=arc-20160816; b=EqYwWk2FdP1Si8sWA4jIFhPOq43wY/TNIf+Xx4C2BXigGC+A3hfPuaKSTY6KMhu13G TaxJSuPl79UIL8vJ7z35GVqBfTaELu/CeFI5//fkqLusTlGv0MWuesfLcVojAJxBKJaD JuLWnxwhybJYC3zmh3XLTk0U+TmK2KCW0x2YgWT4XEGnKz4kuakUPSvAgonzrH4eUfFT y3JffJY9nA5r6ObgZdQ4/DDfOYPQH41NyhwDhxJJFbwlTq43fnRe3l1ax/XHkU45YPNQ 1Ua1JG4yonAm4g497WIb6pxMK8FYOXLUWPxTaaCMNfOggSzVe57RPNsqblFgh0+fnZMC QjIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=gh9TjtpXh8D7hhgVdeb1KwN93Adcgexh1IigcGMkqmM=; b=cKJGFq1TbWjHkply04WxGSdzlS9gPA1Jud5bPdi/mqHD++UIfrgBZtxha/fA+GXK8E bO1CgT4KXusNFhY2LTr8HgIa2ukVwel6JO/oSqJjrX/064Yd6n3yLO/8gm+PoL6koUR1 DUvDnDv3t73DTPrX898ojAwDaaOXvQI57VmgXGn7MxNsKyVQaZosKANq53GCk1Eff2fL nDHTHPc0LtPIBIE+51/Ct+OuJgui13cJqtMZSK5GGOhU61mPNEPSnMunor/YzmUZflXG zo+ZS4nVTO08mDTz6K45jAoQnvg9nsehHVZxm7XVvSE6aXU1Qgnn+V2nox6qg8K5QYug LUGw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.193 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp25.blacknight.com (outbound-smtp25.blacknight.com. [81.17.249.193]) by mx.google.com with ESMTPS id gp17-v6si1475918ejb.103.2019.01.04.04.51.25 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:51:25 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.193 as permitted sender) client-ip=81.17.249.193; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.193 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp25.blacknight.com (Postfix) with ESMTPS id 69C32B87AA for ; Fri, 4 Jan 2019 12:51:23 +0000 (GMT) Received: (qmail 32386 invoked from network); 4 Jan 2019 12:51:23 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:51:23 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 06/25] mm, compaction: Skip pageblocks with reserved pages Date: Fri, 4 Jan 2019 12:49:52 +0000 Message-Id: <20190104125011.16071-7-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Reserved pages are set at boot time, tend to be clustered and almost never become unreserved. When isolating pages for either migration sources or target, skip the entire pageblock is one PageReserved page is encountered on the grounds that it is highly probable the entire pageblock is reserved. The performance impact is relative to the number of reserved pages in the system and their location so it'll be variable but intuitively it should make sense. If the memblock allocator was ever changed to spread reserved pages throughout the address space then this patch would be impaired but it would also be considered a bug given that such a change would ruin fragmentation. On both 1-socket and 2-socket machines, scan rates are reduced slightly on workloads that intensively allocate THP while the system is fragmented. Signed-off-by: Mel Gorman --- mm/compaction.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/mm/compaction.c b/mm/compaction.c index 3afa4e9188b6..94d1e5b062ea 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -484,6 +484,15 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, goto isolate_fail; } + /* + * A reserved page is never freed and tend to be clustered in + * the same pageblock. Skip the block. + */ + if (PageReserved(page)) { + blockpfn = end_pfn; + break; + } + if (!PageBuddy(page)) goto isolate_fail; @@ -827,6 +836,13 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, goto isolate_success; } + /* + * A reserved page is never freed and tend to be + * clustered in the same pageblocks. Skip the block. + */ + if (PageReserved(page)) + low_pfn = end_pfn; + goto isolate_fail; } From patchwork Fri Jan 4 12:49:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748307 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 58B4714E5 for ; Fri, 4 Jan 2019 12:51:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 48B5D27F86 for ; Fri, 4 Jan 2019 12:51:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3AA6E283A5; Fri, 4 Jan 2019 12:51:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BCC3627F86 for ; Fri, 4 Jan 2019 12:51:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 003768E00D9; Fri, 4 Jan 2019 07:51:36 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EF58B8E00AE; Fri, 4 Jan 2019 07:51:35 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D715E8E00D9; Fri, 4 Jan 2019 07:51:35 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 752B58E00AE for ; Fri, 4 Jan 2019 07:51:35 -0500 (EST) Received: by mail-ed1-f72.google.com with SMTP id c3so35318604eda.3 for ; Fri, 04 Jan 2019 04:51:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=XBlzRH1OJ1qmwM1YNbF8hp3yWzEPL2hB+0PZirnJ9oQ=; b=FZ/yIkYFxPzqJTaj2LvN3TaiUQ5Cml/H1aVZ5t7aIRu0i3NE3FXqRzXIhigtTUvDfd CV3T8c/7MRGMp5cOxNhp9RIO8aLuLfb1tJH25892UNwskBhghmN6qT0tJ85tx8HtJyXS 3yvQyE3hvxhk3a8/CNOZMFtlPVkK7I724qnF3bO0zqMFo/PiqMyTXx6lIz85BzRqioQh 0QrE3Qaz1HOaldghwj02Qy7Neq6eeyF2L9Ybg9f5JsPnKTObaFYkUmA0nIhX2N7ofDlk LYvxmC0NVGPDZOYj4BxEmpundEzjO3XnPDzlau6zTCBvXCJMg091V97VJIgRGCuaP4YO KmSQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWb5s6p5kaGw4+vceRFovT+nsB1IvG0BSCugzveVPJAj/78UOzYA cmKEdK4UT/WRwXlNmOzgbr71glt/2eBBPxWMt3lRqEJ8u5X1lyuyZL8PcLck6ECZD9t3pnXO7Aj 9OqiZ5k/C4YoZ8OBUHrdejDKRb7erbwCf+LW3MMl5oOiEcsgTYEsMka7MS9N+ZpQshg== X-Received: by 2002:a17:906:138d:: with SMTP id f13-v6mr38462435ejc.176.1546606294948; Fri, 04 Jan 2019 04:51:34 -0800 (PST) X-Google-Smtp-Source: AFSGD/UIQtssVLI+tIxn9jEZaycHhY3qMckHRHIystWzjuD0SD9yCgzwt/dg723+uzUMwfu8f1yl X-Received: by 2002:a17:906:138d:: with SMTP id f13-v6mr38462402ejc.176.1546606293904; Fri, 04 Jan 2019 04:51:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606293; cv=none; d=google.com; s=arc-20160816; b=YNNnTJX5y8Wb2fDJGkTXUNk3lzqG1nWdzmb9RgdTe7CBtr7JCm7Z5VPFC6IkyqY2RX CAmOAEJLdrebK3iCfyGmEm3IxzaxrdKgep6ydnTMyInI4ZzOWMhttvgY7VuV0h83myWe y7yJLONzNI3pS9+ZLSpkHZKX7j4lCJNR/ooku2MyHoYU/qc45yfyIpuumXzSWXH9/1QO cgvzYL0m1XgHlwl+oEVghFuE4kCYjbhNj8OeqF/OZUjYaWwhyP+O6K22pnaIUpTfbBaB fVsmLGBFTunH18fKG2nMcSIWQDMLv6VwjaZRlvngaYWme9Vzc94EQYPyI7/c2Xo+N3jW cFKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=XBlzRH1OJ1qmwM1YNbF8hp3yWzEPL2hB+0PZirnJ9oQ=; b=GyiyJMfAbE5StGGKuQNgsprNj58S3tgms7+h4dlKFJD+YFlYPCcvI/2Jqe+AHpLd6o 7kE2XcVIoTI0ElZ9IjPBi1U8Mwg+zUjfN84QsgQjJ60dPs+AMUxHhHy0oytsd8uxFtAR fm4gqlvuFTB83k66enuBTBKZVcKMs6yJNn23qAV/wG19sPilFwNuRgUZ0U8bkJVsNcUR JHrunCLjWLhtZ5SeYRISpWC0eaWjKSvg8izTm3+KK5UT1o3XTvsnZJS8tVJSl8oUYeAm CCJK7gBTlNbhxrLSZUU6qafG1dCjUKeGrLZT+dbcUtRk/VgEAySyMFEKP0jrVISBPm5k Mk4w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp02.blacknight.com (outbound-smtp02.blacknight.com. [81.17.249.8]) by mx.google.com with ESMTPS id r24si203929edp.187.2019.01.04.04.51.33 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:51:33 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) client-ip=81.17.249.8; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.8 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp02.blacknight.com (Postfix) with ESMTPS id 94DED98837 for ; Fri, 4 Jan 2019 12:51:33 +0000 (UTC) Received: (qmail 613 invoked from network); 4 Jan 2019 12:51:33 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:51:33 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 07/25] mm, migrate: Immediately fail migration of a page with no migration handler Date: Fri, 4 Jan 2019 12:49:53 +0000 Message-Id: <20190104125011.16071-8-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Pages with no migration handler use a fallback handler which sometimes works and sometimes persistently retries. A historical example was blockdev pages but there are others such as odd refcounting when page->private is used. These are retried multiple times which is wasteful during compaction so this patch will fail migration faster unless the caller specifies MIGRATE_SYNC. This is not expected to help THP allocation success rates but it did reduce latencies very slightly in some cases. 1-socket thpfioscale 4.20.0 4.20.0 noreserved-v2r15 failfast-v2r15 Amean fault-both-1 0.00 ( 0.00%) 0.00 * 0.00%* Amean fault-both-3 3839.67 ( 0.00%) 3833.72 ( 0.15%) Amean fault-both-5 5177.47 ( 0.00%) 4967.15 ( 4.06%) Amean fault-both-7 7245.03 ( 0.00%) 7139.19 ( 1.46%) Amean fault-both-12 11534.89 ( 0.00%) 11326.30 ( 1.81%) Amean fault-both-18 16241.10 ( 0.00%) 16270.70 ( -0.18%) Amean fault-both-24 19075.91 ( 0.00%) 19839.65 ( -4.00%) Amean fault-both-30 22712.11 ( 0.00%) 21707.05 ( 4.43%) Amean fault-both-32 21692.92 ( 0.00%) 21968.16 ( -1.27%) The 2-socket results are not materially different. Scan rates are similar as expected. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/migrate.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/migrate.c b/mm/migrate.c index 5d1839a9148d..547cc1f3f3bb 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -916,7 +916,7 @@ static int fallback_migrate_page(struct address_space *mapping, */ if (page_has_private(page) && !try_to_release_page(page, GFP_KERNEL)) - return -EAGAIN; + return mode == MIGRATE_SYNC ? -EAGAIN : -EBUSY; return migrate_page(mapping, newpage, page, mode); } From patchwork Fri Jan 4 12:49:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748309 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 74EC114E5 for ; Fri, 4 Jan 2019 12:51:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 62297282E8 for ; Fri, 4 Jan 2019 12:51:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 566DF283C9; Fri, 4 Jan 2019 12:51:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D4AA2282E8 for ; Fri, 4 Jan 2019 12:51:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C7168E00DA; Fri, 4 Jan 2019 07:51:46 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 04E3B8E00AE; Fri, 4 Jan 2019 07:51:45 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E7FC78E00DA; Fri, 4 Jan 2019 07:51:45 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id 8DFAA8E00AE for ; Fri, 4 Jan 2019 07:51:45 -0500 (EST) Received: by mail-ed1-f69.google.com with SMTP id 39so35261135edq.13 for ; Fri, 04 Jan 2019 04:51:45 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=5a0bYK8uQJJ65BCn1J11l0HIP4L9v14cwxclgN4TaaY=; b=uMYNtKVJdQ2XwiPuxZsZM0nQn+0XAc5mS3loTuQxbSlfY1xm8cpTLqYznfNPQujra9 Weq+j7harJOOAVFPjsfO8QaJr7/S0MwtM3tS13hHS34T90laVAfnUxWv0CDbdpWjlN11 +yIX93szvOL1d1PktDNGuIu4UNRjoWzMU+t3rQXxOa9IOi2nUPflA44lDxvVTtJgQHqt KfGT9EWa+fXMBdBlhnS4zxOWrHkfG+7SbYd0UMmoLlBasFrnl/6VWmdyInWSDKD9JDDY hZiYTSXGraNix19Hjo7NK8bMWXKT+23egwk9JeObK/ZOMHhS9nK/IKCZ6P20edAPfTkm Av/Q== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWYUlEdSJSNoWMINy3xLSYFk4dT4ugt/7tTtj7EiaCIyaZsWc0p7 FM/8IIZ+qrC+E2SMoVqnnoM1HfcJRn7Y3CIKxfUQxYkmdCUzhSvFd6F3OxhkWFdBWhkb52vfzCr 91Gwl5FNJCjby7bfv6dpawrMhlnd7ugafqpGdnTUISFttfjZyh3ITUqCu72obzPwOqQ== X-Received: by 2002:a50:aa68:: with SMTP id p37mr45048450edc.168.1546606305071; Fri, 04 Jan 2019 04:51:45 -0800 (PST) X-Google-Smtp-Source: AFSGD/U1wLXXBzOa7ZZs5tFqkh1UuJ90o2zfymyutdkixdQD2jh+SktgnygmiR6qvYivDnsyUFbk X-Received: by 2002:a50:aa68:: with SMTP id p37mr45048405edc.168.1546606304134; Fri, 04 Jan 2019 04:51:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606304; cv=none; d=google.com; s=arc-20160816; b=Rgc0j2BfuYs4Z2QXcblLXKI/9lAZ3r4u9/Re7/F2vNfpc9dOBp7e5BHT5uApZSqsGW 2D+GMQMhMJcRLUjczWz5KRCvxvd1PoNJGWFw39NJZnV+tTKQ6aRZwtvIf+KnIBtzyHFS RmzeQh0tIHQtt6WjfRy5aFIGOUqSDwgjIyACaNpeJzoUFQnMUpj0VBgF4pIcW7tLFR2k gVqM7xMTvPg+C19wiNjYbBA4L6mnMU82dQQedGctRvmsEAM4eQTy6gW7iIbSOXGdaojU Pg0EmrSntk0BYP+oL2IFnQlFXToRiJVu1NUHzim2frRHK2gQprYTd9EPpXpQnY9POpRw H6aA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=5a0bYK8uQJJ65BCn1J11l0HIP4L9v14cwxclgN4TaaY=; b=clWS96EiZ5qubXfW3W4rzaXQFFG+mw1NYx614hnRvcnlITq5PpIGBsJa+xGKTpkSU0 UwxHQ7RKDcuvPFDMRqRLuq3QHEdpyipZfpKVw1budveB5ffvdekaU8i6e0NEKE3uLpJz x3vM+/i0/FpOGEWiSchFwxrtBvjjxeuJD8vFVXHXXBzhBwj5ZCgLbG8QJkINfwngSIGY q1y041ZNMwS1VPkysX+ubM3YnHFupfS3IngiQwdp/MGn7HvJIYLlOGTrmRxwI+1qdsIF Ovi6FEZBmKtXULjT17zukIsJN1lLbsfeTZPJS5B01eFNU20j2qO9xFiGnam03DeNDC86 fOaw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp12.blacknight.com (outbound-smtp12.blacknight.com. [46.22.139.17]) by mx.google.com with ESMTPS id w51si5062526edw.91.2019.01.04.04.51.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:51:44 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) client-ip=46.22.139.17; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp12.blacknight.com (Postfix) with ESMTPS id C8A7D1C1D6B for ; Fri, 4 Jan 2019 12:51:43 +0000 (GMT) Received: (qmail 1122 invoked from network); 4 Jan 2019 12:51:43 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:51:43 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 08/25] mm, compaction: Always finish scanning of a full pageblock Date: Fri, 4 Jan 2019 12:49:54 +0000 Message-Id: <20190104125011.16071-9-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When compaction is finishing, it uses a flag to ensure the pageblock is complete but it makes sense to always complete migration of a pageblock. Minimally, skip information is based on a pageblock and partially scanned pageblocks may incur more scanning in the future. The pageblock skip handling also becomes more strict later in the series and the hint is more useful if a complete pageblock was always scanned. The potentially impacts latency as more scanning is done but it's not a consistent win or loss as the scanning is not always a high percentage of the pageblock and sometimes it is offset by future reductions in scanning. Hence, the results are not presented this time due to a misleading mix of gains/losses without any clear pattern. However, full scanning of the pageblock is important for later patches. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 19 ++++++++----------- mm/internal.h | 1 - 2 files changed, 8 insertions(+), 12 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 94d1e5b062ea..8bf2090231a3 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1347,16 +1347,14 @@ static enum compact_result __compact_finished(struct compact_control *cc) if (is_via_compact_memory(cc->order)) return COMPACT_CONTINUE; - if (cc->finishing_block) { - /* - * We have finished the pageblock, but better check again that - * we really succeeded. - */ - if (IS_ALIGNED(cc->migrate_pfn, pageblock_nr_pages)) - cc->finishing_block = false; - else - return COMPACT_CONTINUE; - } + /* + * Always finish scanning a pageblock to reduce the possibility of + * fallbacks in the future. This is particularly important when + * migration source is unmovable/reclaimable but it's not worth + * special casing. + */ + if (!IS_ALIGNED(cc->migrate_pfn, pageblock_nr_pages)) + return COMPACT_CONTINUE; /* Direct compactor: Is a suitable page free? */ for (order = cc->order; order < MAX_ORDER; order++) { @@ -1398,7 +1396,6 @@ static enum compact_result __compact_finished(struct compact_control *cc) return COMPACT_SUCCESS; } - cc->finishing_block = true; return COMPACT_CONTINUE; } } diff --git a/mm/internal.h b/mm/internal.h index c6f794ad21a9..edb4029f64c8 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -202,7 +202,6 @@ struct compact_control { bool direct_compaction; /* False from kcompactd or /proc/... */ bool whole_zone; /* Whole zone should/has been scanned */ bool contended; /* Signal lock or sched contention */ - bool finishing_block; /* Finishing current pageblock */ }; unsigned long From patchwork Fri Jan 4 12:49:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748311 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D084D13B5 for ; Fri, 4 Jan 2019 12:51:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C11FB282E8 for ; Fri, 4 Jan 2019 12:51:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B2DB9283C9; Fri, 4 Jan 2019 12:51:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3556F282E8 for ; Fri, 4 Jan 2019 12:51:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6948A8E00DB; Fri, 4 Jan 2019 07:51:56 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 61B1F8E00AE; Fri, 4 Jan 2019 07:51:56 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 50B398E00DB; Fri, 4 Jan 2019 07:51:56 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id E6CF88E00AE for ; Fri, 4 Jan 2019 07:51:55 -0500 (EST) Received: by mail-ed1-f71.google.com with SMTP id l45so35123592edb.1 for ; Fri, 04 Jan 2019 04:51:55 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=8kl/Dp+/KzdTN0WWTLWu4uZwc5c5hUQZGkU37edu/xc=; b=sEbAUHDGgLn02Et+NQacob0kCVFuUph79Y9DyWo0i1Ku51gNLKEzZvoyx2AVz7M14S NHQZOOxnBTJ9XbUTI7Zu37Dt8ncFmBdOU9EocB506/Jj6OMAxpG+Os4tWl1ed/fvL+9J 8lubSS3f094MkpZgbs6jVy+A4yXLg9LuedAPqxXBO2YY6L05YOPmZtNGYX6L3YTH7sWz J0tcZTsgnLwjXkHfbP3cVDb7uffY/yghiKEFzg2SzD7c5BlehW9YVK7aTD3ug8jIHvbo TM+WX2jWTDyJw+FI4L1BfOHFadMg+Elu/z9CmwVe6hKkT2P25Y0ekQFyQJyCCLK/TEzW IvWQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWZY0SuQf1NAcsWexE3hctEvzCPXJWa98ommCyquXYxHU4a3idPO xQ97ZMZpKV/Zfigy8SXx5MjEh78YBzvSd/bHSeQljGbfL6+R9i+vZ4LADIncEEh6lhRzppvdSWg u48/B5BIVBGCwq8nATZLQQx8a7CEzb3Qu4uk4kXrSZkKsbxGj9hjBxsnXEcJnbDlKsQ== X-Received: by 2002:a17:906:2972:: with SMTP id x18-v6mr39434227ejd.28.1546606315406; Fri, 04 Jan 2019 04:51:55 -0800 (PST) X-Google-Smtp-Source: AFSGD/WBbqcRetPE/kOJezIOCmvk43IZYISfWTT7KbfKJd6POuDv/+TM4bZ84VDXAXsde5wCkBWs X-Received: by 2002:a17:906:2972:: with SMTP id x18-v6mr39434176ejd.28.1546606314356; Fri, 04 Jan 2019 04:51:54 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606314; cv=none; d=google.com; s=arc-20160816; b=ltpS3CJ5589zc3FM6ypyOqEcEnTigpsA/mZsekNShqp04yPaAAtJvVHxjJpqkS8xLf yu/kBNNz7TLjm7Hzrb6YzI/jz0ZY3tp3c7rKw97h8WEQ2YIfYICihjTFdfvysE+Fi8AO 8b1g1/kIi7eYNAhES0q65TlmfxenXg/Gy6SiOjQLfpnJiXnmK5Afzlswpf5brXk5I0xn ZD4+N5hjF43F727wHzbodUPIF5kETS7Io7Icqew5Irmb3Z5g4nylVzjM1Ec4PuRewqxl IhJs3PWxUcX9dEKm+1BrNJCJUJxZ6G+ORN2Hb5P1Tv7RAsjxPYntM+SUU1Nrvjlna3dS jgWA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=8kl/Dp+/KzdTN0WWTLWu4uZwc5c5hUQZGkU37edu/xc=; b=eAB3Z60xs2z9DWbsOaBin1JfciUhmisnsSbw7xGVGWzl21/cMFKJnEf+NRInC18+6s oQwQQ+JqiqmVIlSYGQhH0b8ExX6JMz4h5JZ88q8VUGUti3Q3hQsEDGcX6LUpOJ00eCZ1 1YuCIk6HbzPq1lcXx0mIbFClWkUZ/c/MH4VMAT1R9pAE63SE6EnlX0h/PeLs/iijBDxF OOrCwnWr/ijAAsMhWFB4ZlWFK1xxoiDI6RCMnYByO6PrSKjINQqImbx3pYIFrEaNr4Q2 GYCAjHQI1Vgt0v0LfDuLK9XbdcUVq0moE1HTiw+0hA4r1OikMyVtHgHRIpvVWCDve4M8 zvsQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp12.blacknight.com (outbound-smtp12.blacknight.com. [46.22.139.17]) by mx.google.com with ESMTPS id j8si668671edh.289.2019.01.04.04.51.54 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:51:54 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) client-ip=46.22.139.17; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.17 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp12.blacknight.com (Postfix) with ESMTPS id 015471C1E3F for ; Fri, 4 Jan 2019 12:51:54 +0000 (GMT) Received: (qmail 1701 invoked from network); 4 Jan 2019 12:51:53 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:51:53 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 09/25] mm, compaction: Use the page allocator bulk-free helper for lists of pages Date: Fri, 4 Jan 2019 12:49:55 +0000 Message-Id: <20190104125011.16071-10-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP release_pages() is a simpler version of free_unref_page_list() but it tracks the highest PFN for caching the restart point of the compaction free scanner. This patch optionally tracks the highest PFN in the core helper and converts compaction to use it. The performance impact is limited but it should reduce lock contention slightly in some cases. The main benefit is removing some partially duplicated code. Signed-off-by: Mel Gorman --- include/linux/gfp.h | 7 ++++++- mm/compaction.c | 12 +++--------- mm/page_alloc.c | 10 +++++++++- 3 files changed, 18 insertions(+), 11 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 5f5e25fd6149..9e58799b730f 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -543,7 +543,12 @@ void * __meminit alloc_pages_exact_nid(int nid, size_t size, gfp_t gfp_mask); extern void __free_pages(struct page *page, unsigned int order); extern void free_pages(unsigned long addr, unsigned int order); extern void free_unref_page(struct page *page); -extern void free_unref_page_list(struct list_head *list); +extern void __free_page_list(struct list_head *list, bool dropref, unsigned long *highest_pfn); + +static inline void free_unref_page_list(struct list_head *list) +{ + return __free_page_list(list, false, NULL); +} struct page_frag_cache; extern void __page_frag_cache_drain(struct page *page, unsigned int count); diff --git a/mm/compaction.c b/mm/compaction.c index 8bf2090231a3..8f0ce44dba41 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -52,16 +52,10 @@ static inline void count_compact_events(enum vm_event_item item, long delta) static unsigned long release_freepages(struct list_head *freelist) { - struct page *page, *next; - unsigned long high_pfn = 0; + unsigned long high_pfn; - list_for_each_entry_safe(page, next, freelist, lru) { - unsigned long pfn = page_to_pfn(page); - list_del(&page->lru); - __free_page(page); - if (pfn > high_pfn) - high_pfn = pfn; - } + __free_page_list(freelist, true, &high_pfn); + INIT_LIST_HEAD(freelist); return high_pfn; } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index cde5dac6229a..57ba9d1da519 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2876,18 +2876,26 @@ void free_unref_page(struct page *page) /* * Free a list of 0-order pages */ -void free_unref_page_list(struct list_head *list) +void __free_page_list(struct list_head *list, bool dropref, + unsigned long *highest_pfn) { struct page *page, *next; unsigned long flags, pfn; int batch_count = 0; + if (highest_pfn) + *highest_pfn = 0; + /* Prepare pages for freeing */ list_for_each_entry_safe(page, next, list, lru) { + if (dropref) + WARN_ON_ONCE(!put_page_testzero(page)); pfn = page_to_pfn(page); if (!free_unref_page_prepare(page, pfn)) list_del(&page->lru); set_page_private(page, pfn); + if (highest_pfn && pfn > *highest_pfn) + *highest_pfn = pfn; } local_irq_save(flags); From patchwork Fri Jan 4 12:49:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748313 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D1E1B14E5 for ; Fri, 4 Jan 2019 12:52:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C0282282E8 for ; Fri, 4 Jan 2019 12:52:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B4B45283C9; Fri, 4 Jan 2019 12:52:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 78B68282E8 for ; Fri, 4 Jan 2019 12:52:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 92CB98E00DC; Fri, 4 Jan 2019 07:52:06 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8B2A68E00AE; Fri, 4 Jan 2019 07:52:06 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 77C708E00DC; Fri, 4 Jan 2019 07:52:06 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by kanga.kvack.org (Postfix) with ESMTP id 185978E00AE for ; Fri, 4 Jan 2019 07:52:06 -0500 (EST) Received: by mail-ed1-f70.google.com with SMTP id c34so34718737edb.8 for ; Fri, 04 Jan 2019 04:52:06 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=FnmQpxMABP2Zf611yXswB2S6mBktt6aiC1/AKIq4W7A=; b=dubsv2MoGZYmzgDfUHwpZeUCW8hV52xFzXEq+iopEMAAsfBE+j67VgSTcrnWRrLn1e oZ9L313Jopcc9XaOETs3Rs6RQ6JKZecahEr7cMaA9yXUrhgPlwOlE/m6lLOCNwZ6lje3 4qFC69aO/6PXqwMRPwa/YmdcZFLGx3JIlt9il+ktE4mgd8zph2kCXnvZeYrX+9mY8do6 m0J3RnwsYpehf5pJbBtAB0j8Y0KObkMD6EWBBckmyOtlKSpw5y2fKQ9chmtiBMZGAgSx ohiHkOz4nbUGwcYcZYMsvBm4vlODCRndyyUx6BUlT3vKjvUWrA+u6yHKu+6fLFkLxuo2 dTzA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWaG9xp8XVmwUMQ+pOo4wDsjesGLNiBdPAwjkuVqJn55a73scnFA 2k+t3HfuVVrG2XRQCPJhg97u462Hs5gcdbU3QrgcLd6ErXSkMdZScDNvrsOe/2GAL8H1Nbq9TOT iMlik6bDtTgo9DTsl39546CE8qcpZHhkIC+vNQAUncTNda1wILDQ1xeH5toHnPVS6EA== X-Received: by 2002:a17:906:e9b:: with SMTP id p27-v6mr38861681ejf.29.1546606325592; Fri, 04 Jan 2019 04:52:05 -0800 (PST) X-Google-Smtp-Source: AFSGD/V0HwrQJrDJyiIFRMxxfMfeyeLyOpY8G2i7EtVAvo6FqX2fODa8E0fNKB/+D8/TXVZqxudl X-Received: by 2002:a17:906:e9b:: with SMTP id p27-v6mr38861627ejf.29.1546606324525; Fri, 04 Jan 2019 04:52:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606324; cv=none; d=google.com; s=arc-20160816; b=TR6lHgOkGTnbEKTBYIQNK6JKkfRi4/iyUIAtvjI0HHruratqq3/8nt0X9Zroa2bImk MCoRBHiXPwWrab6cSsxxopppSwi+jh86cXMpAKZVUHvb1vK1/JZCUR0AAucVPdULB00l L4GiXuI8ZHaGKdv1hoWdtabAye8/BVUmhBrP8BQKanZPP+nsp9FMDc9bchQhQJy7z5HJ V98mi/1ZO3aGQPBe4wvbJ+P4HpRho99JQOvD9vK6pTakegTqDfxqQTZUTmVgc5648See Eq3KH8vFymg7EgvaPAJrpnRvXILshr/yMM8pw8q1YIVa1omVlX+ZcP7E166rfmSLXohB tvFA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=FnmQpxMABP2Zf611yXswB2S6mBktt6aiC1/AKIq4W7A=; b=tFCP4VYHm1/9tR2Mm3gMkYHRIdlFrsb8diCqdrnUoYaf94ZCJJZsYIkunPZBOVNZUk 0owFcEaj8rDIbi4ybqt1NCZ6KB0codyCUwgx+CKAsbPkV/qruahamMK3iwA9dMXymJgv OfIzNAGU9qYcGtF6I1wq1uoQmjObse2iY1NBkhq/B5+xltliHAoj3UYHPE4psPy4ydZy l+9NODZV6rw4AG+NhG+m+H4qNgLnEd+jJY4Uosi5iuHX+jz04Y08DVuzr1k8EaKzgx7b DgTGq39ISX5oIQB5pHnlsIm9T7jghL3aV2Bj/D4TwZ4CRiWV5eOSq/Sl6ZhFrPfV5Gob wMUQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp11.blacknight.com (outbound-smtp11.blacknight.com. [46.22.139.106]) by mx.google.com with ESMTPS id a30-v6si360894ejl.130.2019.01.04.04.52.04 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:52:04 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) client-ip=46.22.139.106; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.106 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp11.blacknight.com (Postfix) with ESMTPS id 33A8A1C1B9A for ; Fri, 4 Jan 2019 12:52:04 +0000 (GMT) Received: (qmail 2494 invoked from network); 4 Jan 2019 12:52:04 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:52:04 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 10/25] mm, compaction: Ignore the fragmentation avoidance boost for isolation and compaction Date: Fri, 4 Jan 2019 12:49:56 +0000 Message-Id: <20190104125011.16071-11-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When pageblocks get fragmented, watermarks are artifically boosted to reclaim pages to avoid further fragmentation events. However, compaction is often either fragmentation-neutral or moving movable pages away from unmovable/reclaimable pages. As the true watermarks are preserved, allow compaction to ignore the boost factor. The expected impact is very slight as the main benefit is that compaction is slightly more likely to succeed when the system has been fragmented very recently. On both 1-socket and 2-socket machines for THP-intensive allocation during fragmentation the success rate was increased by less than 1% which is marginal. However, detailed tracing indicated that failure of migration due to a premature ENOMEM triggered by watermark checks were eliminated. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 57ba9d1da519..05c9a81d54ed 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2958,7 +2958,7 @@ int __isolate_free_page(struct page *page, unsigned int order) * watermark, because we already know our high-order page * exists. */ - watermark = min_wmark_pages(zone) + (1UL << order); + watermark = zone->_watermark[WMARK_MIN] + (1UL << order); if (!zone_watermark_ok(zone, 0, watermark, 0, ALLOC_CMA)) return 0; From patchwork Fri Jan 4 12:49:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748315 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 32E5B14E5 for ; Fri, 4 Jan 2019 12:52:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 20AC3283A5 for ; Fri, 4 Jan 2019 12:52:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 10E6628434; Fri, 4 Jan 2019 12:52:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 26E00283A5 for ; Fri, 4 Jan 2019 12:52:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3FCE28E00DD; Fri, 4 Jan 2019 07:52:17 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 382578E00AE; Fri, 4 Jan 2019 07:52:17 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 24BF98E00DD; Fri, 4 Jan 2019 07:52:17 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by kanga.kvack.org (Postfix) with ESMTP id BA0008E00AE for ; Fri, 4 Jan 2019 07:52:16 -0500 (EST) Received: by mail-ed1-f70.google.com with SMTP id b7so35066984eda.10 for ; Fri, 04 Jan 2019 04:52:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=sFp/AyHG7X8F455JbaLtkJ5Q4BCFrJIRG65Z8kacUsg=; b=c99y7hrvzGmBoqkKLONyFrzvx4gFLS2Sgu0Eapf6UQwSO1Z5aRTsNOTyqfhvv26eV1 E1wUWACA+XK4BMVKnt5TvJgFU1Oqz//NepxX3QGWB5rDtvxqx7XDm/OKZ8acGMPn2bHf gHgzOmJY64w4W8oKTekY6gCOQjHrg8JN7F4fo8Dz4oK+yLjmHfdt4aD4nUU8tfoTAonC 7ckPT/Hyn13gbZjusb/1BfCUfhMSPj6vIszyrUyZjJECYyO7v2/B4Skes14Z0BNLz0D4 914w0yAom10IAjP15NqA288p1epYaW3ZqJXG7XizB5Tcbxjhy2UlwTP3l4btkpp2lN5Y Ls9Q== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWZxg8+MlHVoR2i4FfuoJMh650xdbWllpbBxfdp0MBXpo3eLXmVO xyIoXoU3eodisYsCjRD+tR7bVCKbYNx48gqfLT0Pjy01spfiZqGwuK4p6Kpbz1awuojEri4KOBU CoFZW5MDUtpECwCGjYhuDmjIc/7R9cFHJ5tqCqxMJzJsfSRxMX2Wp1litzXjTugMmvg== X-Received: by 2002:a50:96c2:: with SMTP id z2mr48473731eda.11.1546606336202; Fri, 04 Jan 2019 04:52:16 -0800 (PST) X-Google-Smtp-Source: AFSGD/X8vn6JTSxi9J9wEkURPi3YyYS1xIsaMBNGDfqWfn1JteERNm6CA4/TWLZvdOWq2HkFz8RH X-Received: by 2002:a50:96c2:: with SMTP id z2mr48473665eda.11.1546606334757; Fri, 04 Jan 2019 04:52:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606334; cv=none; d=google.com; s=arc-20160816; b=ZYk/Y2YLLfJDqwW16xouSCsqlVywxwx8t9AnywbFj386kUEt5j+Dav2Igv+RLj2rI1 wCVd2rtcGfr1CVsH/vAk6rU+OxM4LKkAJPDHVd7K9PexEBkrkS4Jv8HGJs0fMmSR1Ymy 2A9tYFjzSQ+oEM1zY744hDnZQ63FuTgAyfCmvwJ0mEaa74H8taZ+EnpR0Evh1QtETC7p NqB38y+OuEffbQs3WtK/3Z4iyMzhAwqSxll5N0xvEUgNGHsjKxAIPQX4IDGwWS6dEb7K 54NGyXN3dMRkIbtKkkC0yPm5koz0eVgpMu8Dhlzt9m0m1yqot6y5/+2cr2ecf3PmavXu N4WA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=sFp/AyHG7X8F455JbaLtkJ5Q4BCFrJIRG65Z8kacUsg=; b=DxDVoWuOKUvnnNyRoSdbCmXcoxWBi+tLTibe7pyEk9+OlvkFGXOXgwlzXiH4iCq3tg 8r4vVgEEw3fR6mbumNBcX0POTaURJKe8Jy2eTnt2X9iaCNPIkH2upc7AVeQidusx6520 su9vFTZUKoOwM6UX+tLBBcXwh23Fu7f6kFZHBuP/CXzzrDu83H8lMr1k+Cox2P+2FX16 krHkZS0nqzec5BzmWTGsw4Ad2uz0z0GWjkZx8+oJDur4bLM0KZsLMaocVwj/m33AU4bQ Z+8dyI26k88KNH5covo6/ryNu7iUagvlxHHzlN6nt1cnfPwwQntbBbcW61MSMmpd34rZ sxKA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp16.blacknight.com (outbound-smtp16.blacknight.com. [46.22.139.233]) by mx.google.com with ESMTPS id o1-v6si3442106eji.22.2019.01.04.04.52.14 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:52:14 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) client-ip=46.22.139.233; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp16.blacknight.com (Postfix) with ESMTPS id 60DDF1C2178 for ; Fri, 4 Jan 2019 12:52:14 +0000 (GMT) Received: (qmail 3135 invoked from network); 4 Jan 2019 12:52:14 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:52:14 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 11/25] mm, compaction: Use free lists to quickly locate a migration source Date: Fri, 4 Jan 2019 12:49:57 +0000 Message-Id: <20190104125011.16071-12-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The migration scanner is a linear scan of a zone with a potentiall large search space. Furthermore, many pageblocks are unusable such as those filled with reserved pages or partially filled with pages that cannot migrate. These still get scanned in the common case of allocating a THP and the cost accumulates. The patch uses a partial search of the free lists to locate a migration source candidate that is marked as MOVABLE when allocating a THP. It prefers picking a block with a larger number of free pages already on the basis that there are fewer pages to migrate to free the entire block. The lowest PFN found during searches is tracked as the basis of the start for the linear search after the first search of the free list fails. After the search, the free list is shuffled so that the next search will not encounter the same page. If the search fails then the subsequent searches will be shorter and the linear scanner is used. If this search fails, or if the request is for a small or unmovable/reclaimable allocation then the linear scanner is still used. It is somewhat pointless to use the list search in those cases. Small free pages must be used for the search and there is no guarantee that movable pages are located within that block that are contiguous. 4.20.0 4.20.0 failfast-v2r15 findmig-v2r15 Amean fault-both-1 0.00 ( 0.00%) 0.00 * 0.00%* Amean fault-both-3 3833.72 ( 0.00%) 3505.69 ( 8.56%) Amean fault-both-5 4967.15 ( 0.00%) 5794.13 * -16.65%* Amean fault-both-7 7139.19 ( 0.00%) 7663.09 ( -7.34%) Amean fault-both-12 11326.30 ( 0.00%) 10983.36 ( 3.03%) Amean fault-both-18 16270.70 ( 0.00%) 13602.71 * 16.40%* Amean fault-both-24 19839.65 ( 0.00%) 16145.77 * 18.62%* Amean fault-both-30 21707.05 ( 0.00%) 19753.82 ( 9.00%) Amean fault-both-32 21968.16 ( 0.00%) 20616.16 ( 6.15%) 4.20.0 4.20.0 failfast-v2r15 findmig-v2r15 Percentage huge-1 0.00 ( 0.00%) 0.00 ( 0.00%) Percentage huge-3 84.62 ( 0.00%) 90.58 ( 7.05%) Percentage huge-5 88.43 ( 0.00%) 91.34 ( 3.29%) Percentage huge-7 88.33 ( 0.00%) 92.21 ( 4.39%) Percentage huge-12 88.74 ( 0.00%) 92.48 ( 4.21%) Percentage huge-18 86.52 ( 0.00%) 91.65 ( 5.93%) Percentage huge-24 86.42 ( 0.00%) 90.23 ( 4.41%) Percentage huge-30 86.67 ( 0.00%) 90.17 ( 4.04%) Percentage huge-32 86.00 ( 0.00%) 89.72 ( 4.32%) This shows an improvement in allocation latencies and a slight increase in allocation success rates. While not presented, there was a 13% reduction in migration scanning and a 10% reduction on system CPU usage. A 2-socket machine showed similar benefits. Signed-off-by: Mel Gorman --- mm/compaction.c | 179 +++++++++++++++++++++++++++++++++++++++++++++++++++++++- mm/internal.h | 2 + 2 files changed, 179 insertions(+), 2 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 8f0ce44dba41..137e32e8a2f5 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1050,6 +1050,12 @@ static bool suitable_migration_target(struct compact_control *cc, return false; } +static inline unsigned int +freelist_scan_limit(struct compact_control *cc) +{ + return (COMPACT_CLUSTER_MAX >> cc->fast_search_fail) + 1; +} + /* * Test whether the free scanner has reached the same or lower pageblock than * the migration scanner, and compaction should thus terminate. @@ -1060,6 +1066,19 @@ static inline bool compact_scanners_met(struct compact_control *cc) <= (cc->migrate_pfn >> pageblock_order); } +/* Reorder the free list to reduce repeated future searches */ +static void +move_freelist_tail(struct list_head *freelist, struct page *freepage) +{ + LIST_HEAD(sublist); + + if (!list_is_last(freelist, &freepage->lru)) { + list_cut_position(&sublist, freelist, &freepage->lru); + if (!list_empty(&sublist)) + list_splice_tail(&sublist, freelist); + } +} + /* * Based on information in the current compact_control, find blocks * suitable for isolating free pages from and then isolate them. @@ -1217,6 +1236,160 @@ typedef enum { */ int sysctl_compact_unevictable_allowed __read_mostly = 1; +static inline void +update_fast_start_pfn(struct compact_control *cc, unsigned long pfn) +{ + if (cc->fast_start_pfn == ULONG_MAX) + return; + + if (!cc->fast_start_pfn) + cc->fast_start_pfn = pfn; + + cc->fast_start_pfn = min(cc->fast_start_pfn, pfn); +} + +static inline void +reinit_migrate_pfn(struct compact_control *cc) +{ + if (!cc->fast_start_pfn || cc->fast_start_pfn == ULONG_MAX) + return; + + cc->migrate_pfn = cc->fast_start_pfn; + cc->fast_start_pfn = ULONG_MAX; +} + +/* + * Briefly search the free lists for a migration source that already has + * some free pages to reduce the number of pages that need migration + * before a pageblock is free. + */ +static unsigned long fast_find_migrateblock(struct compact_control *cc) +{ + unsigned int limit = freelist_scan_limit(cc); + unsigned int nr_scanned = 0; + unsigned long distance; + unsigned long pfn = cc->migrate_pfn; + unsigned long high_pfn; + int order; + + /* Skip hints are relied on to avoid repeats on the fast search */ + if (cc->ignore_skip_hint) + return pfn; + + /* + * If the migrate_pfn is not at the start of a zone or the start + * of a pageblock then assume this is a continuation of a previous + * scan restarted due to COMPACT_CLUSTER_MAX. + */ + if (pfn != cc->zone->zone_start_pfn && pfn != pageblock_start_pfn(pfn)) + return pfn; + + /* + * For smaller orders, just linearly scan as the number of pages + * to migrate should be relatively small and does not necessarily + * justify freeing up a large block for a small allocation. + */ + if (cc->order <= PAGE_ALLOC_COSTLY_ORDER) + return pfn; + + /* + * Only allow kcompactd and direct requests for movable pages to + * quickly clear out a MOVABLE pageblock for allocation. This + * reduces the risk that a large movable pageblock is freed for + * an unmovable/reclaimable small allocation. + */ + if (cc->direct_compaction && cc->migratetype != MIGRATE_MOVABLE) + return pfn; + + /* + * When starting the migration scanner, pick any pageblock within the + * first half of the search space. Otherwise try and pick a pageblock + * within the first eighth to reduce the chances that a migration + * target later becomes a source. + */ + distance = (cc->free_pfn - cc->migrate_pfn) >> 1; + if (cc->migrate_pfn != cc->zone->zone_start_pfn) + distance >>= 2; + high_pfn = pageblock_start_pfn(cc->migrate_pfn + distance); + + for (order = cc->order - 1; + order >= PAGE_ALLOC_COSTLY_ORDER && pfn == cc->migrate_pfn && nr_scanned < limit; + order--) { + struct free_area *area = &cc->zone->free_area[order]; + struct list_head *freelist; + unsigned long nr_skipped = 0; + unsigned long flags; + struct page *freepage; + + if (!area->nr_free) + continue; + + spin_lock_irqsave(&cc->zone->lock, flags); + freelist = &area->free_list[MIGRATE_MOVABLE]; + list_for_each_entry(freepage, freelist, lru) { + unsigned long free_pfn; + + nr_scanned++; + free_pfn = page_to_pfn(freepage); + if (free_pfn < high_pfn) { + update_fast_start_pfn(cc, free_pfn); + + /* + * Avoid if skipped recently. Move to the tail + * of the list so it will not be found again + * soon + */ + if (get_pageblock_skip(freepage)) { + + if (list_is_last(freelist, &freepage->lru)) + break; + + nr_skipped++; + list_del(&freepage->lru); + list_add_tail(&freepage->lru, freelist); + if (nr_skipped > 2) + break; + continue; + } + + /* Reorder to so a future search skips recent pages */ + move_freelist_tail(freelist, freepage); + + pfn = pageblock_start_pfn(free_pfn); + cc->fast_search_fail = 0; + set_pageblock_skip(freepage); + break; + } + + /* + * If low PFNs are being found and discarded then + * limit the scan as fast searching is finding + * poor candidates. + */ + if (free_pfn < cc->migrate_pfn) + limit >>= 1; + + if (nr_scanned >= limit) { + cc->fast_search_fail++; + move_freelist_tail(freelist, freepage); + break; + } + } + spin_unlock_irqrestore(&cc->zone->lock, flags); + } + + cc->total_migrate_scanned += nr_scanned; + + /* + * If fast scanning failed then use a cached entry for a page block + * that had free pages as the basis for starting a linear scan. + */ + if (pfn == cc->migrate_pfn) + reinit_migrate_pfn(cc); + + return pfn; +} + /* * Isolate all pages that can be migrated from the first suitable block, * starting at the block pointed to by the migrate scanner pfn within @@ -1235,9 +1408,10 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone, /* * Start at where we last stopped, or beginning of the zone as - * initialized by compact_zone() + * initialized by compact_zone(). The first failure will use + * the lowest PFN as the starting point for linear scanning. */ - low_pfn = cc->migrate_pfn; + low_pfn = fast_find_migrateblock(cc); block_start_pfn = pageblock_start_pfn(low_pfn); if (block_start_pfn < zone->zone_start_pfn) block_start_pfn = zone->zone_start_pfn; @@ -1560,6 +1734,7 @@ static enum compact_result compact_zone(struct compact_control *cc) * want to compact the whole zone), but check that it is initialised * by ensuring the values are within zone boundaries. */ + cc->fast_start_pfn = 0; if (cc->whole_zone) { cc->migrate_pfn = start_pfn; cc->free_pfn = pageblock_start_pfn(end_pfn - 1); diff --git a/mm/internal.h b/mm/internal.h index edb4029f64c8..b25b33c5dd80 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -187,9 +187,11 @@ struct compact_control { unsigned int nr_migratepages; /* Number of pages to migrate */ unsigned long free_pfn; /* isolate_freepages search base */ unsigned long migrate_pfn; /* isolate_migratepages search base */ + unsigned long fast_start_pfn; /* a pfn to start linear scan from */ struct zone *zone; unsigned long total_migrate_scanned; unsigned long total_free_scanned; + unsigned int fast_search_fail; /* failures to use free list searches */ const gfp_t gfp_mask; /* gfp mask of a direct compactor */ int order; /* order a direct compactor needs */ int migratetype; /* migratetype of direct compactor */ From patchwork Fri Jan 4 12:49:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748317 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7D76A13B5 for ; Fri, 4 Jan 2019 12:52:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6D29F283A5 for ; Fri, 4 Jan 2019 12:52:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 616CB28434; Fri, 4 Jan 2019 12:52:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 35DE4283C9 for ; Fri, 4 Jan 2019 12:52:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4BFD78E00DE; Fri, 4 Jan 2019 07:52:27 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 46C6C8E00AE; Fri, 4 Jan 2019 07:52:27 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3430A8E00DE; Fri, 4 Jan 2019 07:52:27 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id C88368E00AE for ; Fri, 4 Jan 2019 07:52:26 -0500 (EST) Received: by mail-ed1-f72.google.com with SMTP id i55so35179058ede.14 for ; Fri, 04 Jan 2019 04:52:26 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=AO1EIfUjY0wHJt52savv2CHgHGtRk3UkrKypX/70fsA=; b=T2ee5TF5ggeAuocN1OSY8E+3iB4HxnQSg642BnE78AopgCfZ211ftDPEFnIgsFUQ31 SVe+YciAgDuYF+Pkkl4+9AiMMBz+nOadVgXMLMRzpN3Ev/6EflgnuwUHiIhhrpd4oDRo nU1mPV69ssrCAkN71fjwj8GU5Lfk6Jh5hMHvA0zhc2l5+lRd9BED2VDB5duCTbc1W9Zu idvpPFDHNFaUy9hv8/iC3XsKzXaskT4mnFRBp9w+tGzpnszwJsFK9w4zaO1rqIwIkqOI 7A50g+ghQvMZFF34jKxgf5SgnOtpHaWk/KbVjsuMasysNe+qWWdETPcbCTM68r52bnkd B85g== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWZ/X6Olfp7e4j8pFUo1Q2F/702QEM/4bNfTH+JxwkVFkQF1V48T CuB5wGjkYeOmsPzH9M+E2MaqruHdcVEnRZSVbvhJSDhwNfpJ6peAEJ3MStGxIz3aMk9QmVPBsVo 3FBqr304aa+/Bvnfixd5tz18WZerEqTV27tjATDzvKTeX0YhBE3yPP8uTqeF0wFWHcw== X-Received: by 2002:a17:906:55a:: with SMTP id k26-v6mr39036863eja.218.1546606346232; Fri, 04 Jan 2019 04:52:26 -0800 (PST) X-Google-Smtp-Source: AFSGD/X2tEAI3FAYvHIV7VtE/ZL2tp0G3xW595OX6cmgCcYBEuPTvcuxeD2jxlQiXzFNg4Y4N1vZ X-Received: by 2002:a17:906:55a:: with SMTP id k26-v6mr39036821eja.218.1546606344965; Fri, 04 Jan 2019 04:52:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606344; cv=none; d=google.com; s=arc-20160816; b=KhZbKsOI/fiSs+XzZRuP8yw8vWfVOuo/uOSheCN2hbm5sEJXNgYABtO9hP4NxmGm3n y9hmlyMYH7I1cGjgInxkHAtjSfwCrxN7eeY1wuB5H9XE0rz3fyTZbSOPvR81OTCqfP+b xksq5tccMEqMxxAOzMw5nE5P0Plo6Gw58+nE7M+D20w03c3DotyQ8iOXMVPp34BX1XV7 B9xUhXi0+s0WSAoPJyGGAEuikfLaXBcDchYhLaivVuMopiVGdmykTiL5mal/xGPR78Qt 0cC0uF8aYsukX6lUfxy+mec0bOpx25EAv65mYtDTm88ShxT4vSVVBxFf7xuBfKmxO/bZ 2PGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=AO1EIfUjY0wHJt52savv2CHgHGtRk3UkrKypX/70fsA=; b=k0q++prq9JaXxZfultX91AvNTeA+SRoDOLMPQ0lBbEEyWG8EjjhZ4qCZXdKjdgX77S MyHPjz4LYbtqPfRjEAp23NUvL2kh5kvFFDMexRR39WKcgf/fWyVXQ2u00Mv4JVp1x+rR T09m9Nzm9CZfzntLgb11sVPA3qbmxEPUX/Ot9mapO5hnZrxhw1hFt2rWe2pYs/ti4jjp v58ViuZ2My9mNWgDxPFg+LBXasXyR4FtYGpXCnrz/WxBqXVOo8BqUXyga3ilEQ/pCL7T +4ncGub90zgV5+5ZDu5wJFnm7TD+Z8G3vbADEO0GKZmqjOgnDVvjokal2TfL2tgPkWtJ 8xpQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp10.blacknight.com (outbound-smtp10.blacknight.com. [46.22.139.15]) by mx.google.com with ESMTPS id e16-v6si384789ejk.23.2019.01.04.04.52.24 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:52:24 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) client-ip=46.22.139.15; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp10.blacknight.com (Postfix) with ESMTPS id 935591C213D for ; Fri, 4 Jan 2019 12:52:24 +0000 (GMT) Received: (qmail 3732 invoked from network); 4 Jan 2019 12:52:24 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:52:24 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 12/25] mm, compaction: Keep migration source private to a single compaction instance Date: Fri, 4 Jan 2019 12:49:58 +0000 Message-Id: <20190104125011.16071-13-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Due to either a fast search of the free list or a linear scan, it is possible for multiple compaction instances to pick the same pageblock for migration. This is lucky for one scanner and increased scanning for all the others. It also allows a race between requests on which first allocates the resulting free block. This patch tests and updates the pageblock skip for the migration scanner carefully. When isolating a block, it will check and skip if the block is already in use. Once the zone lock is acquired, it will be rechecked so that only one scanner can set the pageblock skip for exclusive use. Any scanner contending will continue with a linear scan. The skip bit is still set if no pages can be isolated in a range. While this may result in redundant scanning, it avoids unnecessarily acquiring the zone lock when there are no suitable migration sources. 1-socket thpscale 4.20.0 4.20.0 findmig-v2r15 isolmig-v2r15 Amean fault-both-1 0.00 ( 0.00%) 0.00 * 0.00%* Amean fault-both-3 3505.69 ( 0.00%) 3066.68 * 12.52%* Amean fault-both-5 5794.13 ( 0.00%) 4298.49 * 25.81%* Amean fault-both-7 7663.09 ( 0.00%) 5986.99 * 21.87%* Amean fault-both-12 10983.36 ( 0.00%) 9324.85 ( 15.10%) Amean fault-both-18 13602.71 ( 0.00%) 13350.05 ( 1.86%) Amean fault-both-24 16145.77 ( 0.00%) 13491.77 * 16.44%* Amean fault-both-30 19753.82 ( 0.00%) 15630.86 * 20.87%* Amean fault-both-32 20616.16 ( 0.00%) 17428.50 * 15.46%* This is the first patch that shows a significant reduction in latency as multiple compaction scanners do not operate on the same blocks. There is a small increase in the success rate 4.20.0-rc6 4.20.0-rc6 findmig-v1r4 isolmig-v1r4 Percentage huge-3 90.58 ( 0.00%) 95.84 ( 5.81%) Percentage huge-5 91.34 ( 0.00%) 94.19 ( 3.12%) Percentage huge-7 92.21 ( 0.00%) 93.78 ( 1.71%) Percentage huge-12 92.48 ( 0.00%) 94.33 ( 2.00%) Percentage huge-18 91.65 ( 0.00%) 94.15 ( 2.72%) Percentage huge-24 90.23 ( 0.00%) 94.23 ( 4.43%) Percentage huge-30 90.17 ( 0.00%) 95.17 ( 5.54%) Percentage huge-32 89.72 ( 0.00%) 93.59 ( 4.32%) Compaction migrate scanned 54168306 25516488 Compaction free scanned 800530954 87603321 Migration scan rates are reduced by 52%. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 126 ++++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 99 insertions(+), 27 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 137e32e8a2f5..24e3a9db4b70 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -279,13 +279,52 @@ void reset_isolation_suitable(pg_data_t *pgdat) } } +/* + * Sets the pageblock skip bit if it was clear. Note that this is a hint as + * locks are not required for read/writers. Returns true if it was already set. + */ +static bool test_and_set_skip(struct compact_control *cc, struct page *page, + unsigned long pfn) +{ + bool skip; + + /* Do no update if skip hint is being ignored */ + if (cc->ignore_skip_hint) + return false; + + if (!IS_ALIGNED(pfn, pageblock_nr_pages)) + return false; + + skip = get_pageblock_skip(page); + if (!skip && !cc->no_set_skip_hint) + set_pageblock_skip(page); + + return skip; +} + +static void update_cached_migrate(struct compact_control *cc, unsigned long pfn) +{ + struct zone *zone = cc->zone; + + pfn = pageblock_end_pfn(pfn); + + /* Set for isolation rather than compaction */ + if (cc->no_set_skip_hint) + return; + + if (pfn > zone->compact_cached_migrate_pfn[0]) + zone->compact_cached_migrate_pfn[0] = pfn; + if (cc->mode != MIGRATE_ASYNC && + pfn > zone->compact_cached_migrate_pfn[1]) + zone->compact_cached_migrate_pfn[1] = pfn; +} + /* * If no pages were isolated then mark this pageblock to be skipped in the * future. The information is later cleared by __reset_isolation_suitable(). */ static void update_pageblock_skip(struct compact_control *cc, - struct page *page, unsigned long nr_isolated, - bool migrate_scanner) + struct page *page, unsigned long nr_isolated) { struct zone *zone = cc->zone; unsigned long pfn; @@ -304,16 +343,8 @@ static void update_pageblock_skip(struct compact_control *cc, pfn = page_to_pfn(page); /* Update where async and sync compaction should restart */ - if (migrate_scanner) { - if (pfn > zone->compact_cached_migrate_pfn[0]) - zone->compact_cached_migrate_pfn[0] = pfn; - if (cc->mode != MIGRATE_ASYNC && - pfn > zone->compact_cached_migrate_pfn[1]) - zone->compact_cached_migrate_pfn[1] = pfn; - } else { - if (pfn < zone->compact_cached_free_pfn) - zone->compact_cached_free_pfn = pfn; - } + if (pfn < zone->compact_cached_free_pfn) + zone->compact_cached_free_pfn = pfn; } #else static inline bool isolation_suitable(struct compact_control *cc, @@ -328,10 +359,19 @@ static inline bool pageblock_skip_persistent(struct page *page) } static inline void update_pageblock_skip(struct compact_control *cc, - struct page *page, unsigned long nr_isolated, - bool migrate_scanner) + struct page *page, unsigned long nr_isolated) +{ +} + +static void update_cached_migrate(struct compact_control *cc, unsigned long pfn) { } + +static bool test_and_set_skip(struct compact_control *cc, struct page *page, + unsigned long pfn) +{ + return false; +} #endif /* CONFIG_COMPACTION */ /* @@ -570,7 +610,7 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, /* Update the pageblock-skip if the whole pageblock was scanned */ if (blockpfn == end_pfn) - update_pageblock_skip(cc, valid_page, total_isolated, false); + update_pageblock_skip(cc, valid_page, total_isolated); cc->total_free_scanned += nr_scanned; if (total_isolated) @@ -705,6 +745,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, unsigned long start_pfn = low_pfn; bool skip_on_failure = false; unsigned long next_skip_pfn = 0; + bool skip_updated = false; /* * Ensure that there are not too many pages isolated from the LRU @@ -771,8 +812,19 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, page = pfn_to_page(low_pfn); - if (!valid_page) + /* + * Check if the pageblock has already been marked skipped. + * Only the aligned PFN is checked as the caller isolates + * COMPACT_CLUSTER_MAX at a time so the second call must + * not falsely conclude that the block should be skipped. + */ + if (!valid_page && IS_ALIGNED(low_pfn, pageblock_nr_pages)) { + if (!cc->ignore_skip_hint && get_pageblock_skip(page)) { + low_pfn = end_pfn; + goto isolate_abort; + } valid_page = page; + } /* * Skip if free. We read page order here without zone lock @@ -860,8 +912,19 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, if (!locked) { locked = compact_trylock_irqsave(zone_lru_lock(zone), &flags, cc); - if (!locked) + + /* Allow future scanning if the lock is contended */ + if (!locked) { + clear_pageblock_skip(page); break; + } + + /* Try get exclusive access under lock */ + if (!skip_updated) { + skip_updated = true; + if (test_and_set_skip(cc, page, low_pfn)) + goto isolate_abort; + } /* Recheck PageLRU and PageCompound under lock */ if (!PageLRU(page)) @@ -939,15 +1002,20 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, if (unlikely(low_pfn > end_pfn)) low_pfn = end_pfn; +isolate_abort: if (locked) spin_unlock_irqrestore(zone_lru_lock(zone), flags); /* - * Update the pageblock-skip information and cached scanner pfn, - * if the whole pageblock was scanned without isolating any page. + * Updated the cached scanner pfn if the pageblock was scanned + * without isolating a page. The pageblock may not be marked + * skipped already if there were no LRU pages in the block. */ - if (low_pfn == end_pfn) - update_pageblock_skip(cc, valid_page, nr_isolated, true); + if (low_pfn == end_pfn && !nr_isolated) { + if (valid_page && !skip_updated) + set_pageblock_skip(valid_page); + update_cached_migrate(cc, low_pfn); + } trace_mm_compaction_isolate_migratepages(start_pfn, low_pfn, nr_scanned, nr_isolated); @@ -1332,8 +1400,6 @@ static unsigned long fast_find_migrateblock(struct compact_control *cc) nr_scanned++; free_pfn = page_to_pfn(freepage); if (free_pfn < high_pfn) { - update_fast_start_pfn(cc, free_pfn); - /* * Avoid if skipped recently. Move to the tail * of the list so it will not be found again @@ -1355,9 +1421,9 @@ static unsigned long fast_find_migrateblock(struct compact_control *cc) /* Reorder to so a future search skips recent pages */ move_freelist_tail(freelist, freepage); + update_fast_start_pfn(cc, free_pfn); pfn = pageblock_start_pfn(free_pfn); cc->fast_search_fail = 0; - set_pageblock_skip(freepage); break; } @@ -1427,7 +1493,6 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone, low_pfn = block_end_pfn, block_start_pfn = block_end_pfn, block_end_pfn += pageblock_nr_pages) { - /* * This can potentially iterate a massively long zone with * many pageblocks unsuitable, so periodically check if we @@ -1442,8 +1507,15 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone, if (!page) continue; - /* If isolation recently failed, do not retry */ - if (!isolation_suitable(cc, page)) + /* + * If isolation recently failed, do not retry. Only check the + * pageblock once. COMPACT_CLUSTER_MAX causes a pageblock + * to be visited multiple times. Assume skip was checked + * before making it "skip" so other compaction instances do + * not scan the same block. + */ + if (IS_ALIGNED(low_pfn, pageblock_nr_pages) && + !isolation_suitable(cc, page)) continue; /* From patchwork Fri Jan 4 12:49:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748319 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3F71613B5 for ; Fri, 4 Jan 2019 12:52:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2FA3B283A5 for ; Fri, 4 Jan 2019 12:52:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2429628434; Fri, 4 Jan 2019 12:52:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 56A6E283A5 for ; Fri, 4 Jan 2019 12:52:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 646C78E00DF; Fri, 4 Jan 2019 07:52:37 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5CC3B8E00AE; Fri, 4 Jan 2019 07:52:37 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4BBA18E00DF; Fri, 4 Jan 2019 07:52:37 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by kanga.kvack.org (Postfix) with ESMTP id E0DCC8E00AE for ; Fri, 4 Jan 2019 07:52:36 -0500 (EST) Received: by mail-ed1-f70.google.com with SMTP id y35so35122397edb.5 for ; Fri, 04 Jan 2019 04:52:36 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=jifFj7D6gH+DpLBXWxa6HUYG65aeecDFM0kq3SfEeOs=; b=pnAWAQH5gDixieX5DLDxfGtRVpVNx+ajG/WMG8SSIIhDoSFFuYsuBxOVpdjWLjUq8F FDw5lLzqHEsLJj2b3WbiKNe1sq1VuHm6p7c+qnhkUr8t6GUgSycgr9jktZjaEsGJTDsF QmVlqiq8O+MhfwnSofaelIYovkLGlplXoY85Zi8OchZfsCemY0gFSfL/9OL8t5rvaoI3 yRlfVyScC1AsEedwr4viD7TbG3zL5VFQvo9UKZMf3VQXBlqY6B3yF+QtFE1ipxdMs8j2 QcthfdXqPtmi+RI9dqtHkMT6ttKs/AnzMZ4EaPfWUhb6bW3uARktLWdR4VF6iOCOZSly VIWQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWbGBgU6WbdSFaXpZuB+9jQjSqFPgeGXMnrxYaMy5saHcMyGLWkU X5b/5rOKcSffGb/15cmmWNcZ7WHurK+jtuarxqCHlWZb7th+I5/bnLj8yP3I3edM7F4D/+wOKXd bcJ+gTnVFmmhBWRwa/bF9divvOlVjUnhJ9CYVYhkxOsSbAs5zREwwAsBAJf/hIvYo9A== X-Received: by 2002:a17:906:6c9:: with SMTP id v9-v6mr38137513ejb.203.1546606356366; Fri, 04 Jan 2019 04:52:36 -0800 (PST) X-Google-Smtp-Source: AFSGD/X+rkC39Kn+6ScpWoMhbmmXS5vB4J46tFHbz1dHOTJ5K3TV4iiNz6No4Gv68kN61puDInSO X-Received: by 2002:a17:906:6c9:: with SMTP id v9-v6mr38137468ejb.203.1546606355125; Fri, 04 Jan 2019 04:52:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606355; cv=none; d=google.com; s=arc-20160816; b=QoSm8J6efLbpC62TGz8toJUYYevN8c+TGDnGLyqyBzV0UD30pchEUYwez+OvibsI7K U6kn5ctJz4VSpgiaJalIA1SCqpNCR+uRzKdAAI37fPBNK40bkedWamvLmLMod6bQMHxc M824pUywiuzDq25jCfvlPte8q9nnw1k/gmlRcr2xLhV3OAjh/RzThQLJQoIiu5UvZQvU O9IMyip8Z3AVnQX5aMvkPLfqDP2V23azjYaqLR8qQvkg9ye9E+lXW++LvsMsYxIhYU9U 2bGSM2e0GcAWUK9t/wbYZhnl8VrFcJd7WlmIZH7efEw6x3gwD6QsN5/1E0SwtfeAyEE8 Vv/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=jifFj7D6gH+DpLBXWxa6HUYG65aeecDFM0kq3SfEeOs=; b=FW9/Fysqm2BmXWOICaK+VNhyGuI9SeCAYA9pxFVg+YQq4A1KcoDC6P3xhEmWF3p094 OdfyO+hp1Kem5R+KR7PKuj3GA4nUMNsLO2JOH/b/ZqhpqRvv3XUX7KoQdDm7XZEd9fIV dQTQvYNerJ1BDKcDN4aWXdQE6KWn4ba6GpKAoefvL6fmr9zRmy8eZMNhtXsBlYtF6b6V JWcOcWfUa2EVI52rKNSQWYpUTa5FdSskplqHnI0xdz6NHd53/8p3dRqWfwNv/h15f1na oSPfZ2KHZBIG1xG/ntrFP69VIWZ+GwN/McFOfbZtVzJ9ng/DWjsMWNvHmzaP4hmTLrpz u4pA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp16.blacknight.com (outbound-smtp16.blacknight.com. [46.22.139.233]) by mx.google.com with ESMTPS id g1si444531edh.399.2019.01.04.04.52.34 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:52:35 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) client-ip=46.22.139.233; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.233 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp16.blacknight.com (Postfix) with ESMTPS id BEF921C1F5B for ; Fri, 4 Jan 2019 12:52:34 +0000 (GMT) Received: (qmail 4428 invoked from network); 4 Jan 2019 12:52:34 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:52:34 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 13/25] mm, compaction: Use free lists to quickly locate a migration target Date: Fri, 4 Jan 2019 12:49:59 +0000 Message-Id: <20190104125011.16071-14-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Similar to the migration scanner, this patch uses the free lists to quickly locate a migration target. The search is different in that lower orders will be searched for a suitable high PFN if necessary but the search is still bound. This is justified on the grounds that the free scanner typically scans linearly much more than the migration scanner. If a free page is found, it is isolated and compaction continues if enough pages were isolated. For SYNC* scanning, the full pageblock is scanned for any remaining free pages so that is can be marked for skipping in the near future. 1-socket thpfioscale 4.20.0 4.20.0 isolmig-v2r15 findfree-v2r15 Amean fault-both-1 0.00 ( 0.00%) 0.00 * 0.00%* Amean fault-both-3 3066.68 ( 0.00%) 2884.51 ( 5.94%) Amean fault-both-5 4298.49 ( 0.00%) 4419.70 ( -2.82%) Amean fault-both-7 5986.99 ( 0.00%) 6039.04 ( -0.87%) Amean fault-both-12 9324.85 ( 0.00%) 9992.34 ( -7.16%) Amean fault-both-18 13350.05 ( 0.00%) 12690.05 ( 4.94%) Amean fault-both-24 13491.77 ( 0.00%) 14393.93 ( -6.69%) Amean fault-both-30 15630.86 ( 0.00%) 16894.08 ( -8.08%) Amean fault-both-32 17428.50 ( 0.00%) 17813.68 ( -2.21%) The impact on latency is variable but the search is optimistic and sensitive to the exact system state. Success rates are similar but the major impact is to the rate of scanning 4.20.0-rc6 4.20.0-rc6 isolmig-v1r4findfree-v1r8 Compaction migrate scanned 25516488 28324352 Compaction free scanned 87603321 56131065 The free scan rates are reduced by 35%. The 2-socket reductions for the free scanner are more dramatic which is a likely reflection that the machine has more memory. Signed-off-by: Mel Gorman --- mm/compaction.c | 203 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 198 insertions(+), 5 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 24e3a9db4b70..9438f0564ed5 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1136,7 +1136,7 @@ static inline bool compact_scanners_met(struct compact_control *cc) /* Reorder the free list to reduce repeated future searches */ static void -move_freelist_tail(struct list_head *freelist, struct page *freepage) +move_freelist_head(struct list_head *freelist, struct page *freepage) { LIST_HEAD(sublist); @@ -1147,6 +1147,193 @@ move_freelist_tail(struct list_head *freelist, struct page *freepage) } } +static void +move_freelist_tail(struct list_head *freelist, struct page *freepage) +{ + LIST_HEAD(sublist); + + if (!list_is_last(freelist, &freepage->lru)) { + list_cut_before(&sublist, freelist, &freepage->lru); + if (!list_empty(&sublist)) + list_splice_tail(&sublist, freelist); + } +} + +static void +fast_isolate_around(struct compact_control *cc, unsigned long pfn, unsigned long nr_isolated) +{ + unsigned long start_pfn, end_pfn; + struct page *page = pfn_to_page(pfn); + + /* Do not search around if there are enough pages already */ + if (cc->nr_freepages >= cc->nr_migratepages) + return; + + /* Minimise scanning during async compaction */ + if (cc->direct_compaction && cc->mode == MIGRATE_ASYNC) + return; + + /* Pageblock boundaries */ + start_pfn = pageblock_start_pfn(pfn); + end_pfn = min(start_pfn + pageblock_nr_pages, zone_end_pfn(cc->zone)); + + /* Scan before */ + if (start_pfn != pfn) { + isolate_freepages_block(cc, &start_pfn, pfn, &cc->freepages, false); + if (cc->nr_freepages >= cc->nr_migratepages) + return; + } + + /* Scan after */ + start_pfn = pfn + nr_isolated; + if (start_pfn != end_pfn) + isolate_freepages_block(cc, &start_pfn, end_pfn, &cc->freepages, false); + + /* Skip this pageblock in the future as it's full or nearly full */ + if (cc->nr_freepages < cc->nr_migratepages) + set_pageblock_skip(page); +} + +static unsigned long +fast_isolate_freepages(struct compact_control *cc) +{ + unsigned int limit = min(1U, freelist_scan_limit(cc) >> 1); + unsigned int order_scanned = 0, nr_scanned = 0; + unsigned long low_pfn, min_pfn, high_pfn = 0, highest = 0; + unsigned long nr_isolated = 0; + unsigned long distance; + struct page *page = NULL; + bool scan_start = false; + int order; + + /* + * If starting the scan, use a deeper search and use the highest + * PFN found if a suitable one is not found. + */ + if (cc->free_pfn == pageblock_start_pfn(zone_end_pfn(cc->zone) - 1)) { + limit = pageblock_nr_pages >> 1; + scan_start = true; + } + + /* + * Preferred point is in the top quarter of the scan space but take + * a pfn from the top half if the search is problematic. + */ + distance = (cc->free_pfn - cc->migrate_pfn); + low_pfn = pageblock_start_pfn(cc->free_pfn - (distance >> 2)); + min_pfn = pageblock_start_pfn(cc->free_pfn - (distance >> 1)); + + if (WARN_ON_ONCE(min_pfn > low_pfn)) + low_pfn = min_pfn; + + for (order = cc->order - 1; + order >= 0 && !page; + order--) { + struct free_area *area = &cc->zone->free_area[order]; + struct list_head *freelist; + struct page *freepage; + unsigned long flags; + + if (!area->nr_free) + continue; + + spin_lock_irqsave(&cc->zone->lock, flags); + freelist = &area->free_list[MIGRATE_MOVABLE]; + list_for_each_entry_reverse(freepage, freelist, lru) { + unsigned long pfn; + + order_scanned++; + nr_scanned++; + pfn = page_to_pfn(freepage); + + if (pfn >= highest) + highest = pageblock_start_pfn(pfn); + + if (pfn >= low_pfn) { + cc->fast_search_fail = 0; + page = freepage; + break; + } + + if (pfn >= min_pfn && pfn > high_pfn) { + high_pfn = pfn; + + /* Shorten the scan if a candidate is found */ + limit >>= 1; + } + + if (order_scanned >= limit) + break; + } + + /* Use a minimum pfn if a preferred one was not found */ + if (!page && high_pfn) { + page = pfn_to_page(high_pfn); + + /* Update freepage for the list reorder below */ + freepage = page; + } + + /* Reorder to so a future search skips recent pages */ + move_freelist_head(freelist, freepage); + + /* Isolate the page if available */ + if (page) { + if (__isolate_free_page(page, order)) { + set_page_private(page, order); + nr_isolated = 1 << order; + cc->nr_freepages += nr_isolated; + list_add_tail(&page->lru, &cc->freepages); + count_compact_events(COMPACTISOLATED, nr_isolated); + } else { + /* If isolation fails, abort the search */ + order = -1; + page = NULL; + } + } + + spin_unlock_irqrestore(&cc->zone->lock, flags); + + /* + * Smaller scan on next order so the total scan ig related + * to freelist_scan_limit. + */ + if (order_scanned >= limit) + limit = min(1U, limit >> 1); + } + + if (!page) { + cc->fast_search_fail++; + if (scan_start) { + /* + * Use the highest PFN found above min. If one was + * not found, be pessemistic for direct compaction + * and use the min mark. + */ + if (highest) { + page = pfn_to_page(highest); + cc->free_pfn = highest; + } else { + if (cc->direct_compaction) { + page = pfn_to_page(min_pfn); + cc->free_pfn = min_pfn; + } + } + } + } + + if (highest && highest > cc->zone->compact_cached_free_pfn) + cc->zone->compact_cached_free_pfn = highest; + + cc->total_free_scanned += nr_scanned; + if (!page) + return cc->free_pfn; + + low_pfn = page_to_pfn(page); + fast_isolate_around(cc, low_pfn, nr_isolated); + return low_pfn; +} + /* * Based on information in the current compact_control, find blocks * suitable for isolating free pages from and then isolate them. @@ -1161,6 +1348,11 @@ static void isolate_freepages(struct compact_control *cc) unsigned long low_pfn; /* lowest pfn scanner is able to scan */ struct list_head *freelist = &cc->freepages; + /* Try a small search of the free lists for a candidate */ + isolate_start_pfn = fast_isolate_freepages(cc); + if (cc->nr_freepages) + goto splitmap; + /* * Initialise the free scanner. The starting point is where we last * successfully isolated from, zone-cached value, or the end of the @@ -1173,7 +1365,7 @@ static void isolate_freepages(struct compact_control *cc) * is using. */ isolate_start_pfn = cc->free_pfn; - block_start_pfn = pageblock_start_pfn(cc->free_pfn); + block_start_pfn = pageblock_start_pfn(isolate_start_pfn); block_end_pfn = min(block_start_pfn + pageblock_nr_pages, zone_end_pfn(zone)); low_pfn = pageblock_end_pfn(cc->migrate_pfn); @@ -1237,9 +1429,6 @@ static void isolate_freepages(struct compact_control *cc) } } - /* __isolate_free_page() does not map the pages */ - split_map_pages(freelist); - /* * Record where the free scanner will restart next time. Either we * broke from the loop and set isolate_start_pfn based on the last @@ -1247,6 +1436,10 @@ static void isolate_freepages(struct compact_control *cc) * and the loop terminated due to isolate_start_pfn < low_pfn */ cc->free_pfn = isolate_start_pfn; + +splitmap: + /* __isolate_free_page() does not map the pages */ + split_map_pages(freelist); } /* From patchwork Fri Jan 4 12:50:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748321 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1CF2214E5 for ; Fri, 4 Jan 2019 12:52:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0DEE9283C9 for ; Fri, 4 Jan 2019 12:52:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 01E012844C; Fri, 4 Jan 2019 12:52:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 50C50283C9 for ; Fri, 4 Jan 2019 12:52:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 691DF8E00E0; Fri, 4 Jan 2019 07:52:47 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 61A4E8E00AE; Fri, 4 Jan 2019 07:52:47 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 509908E00E0; Fri, 4 Jan 2019 07:52:47 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by kanga.kvack.org (Postfix) with ESMTP id E5C0D8E00AE for ; Fri, 4 Jan 2019 07:52:46 -0500 (EST) Received: by mail-ed1-f70.google.com with SMTP id b3so35181023edi.0 for ; Fri, 04 Jan 2019 04:52:46 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=/hdAGFXBogCDzBnIyWLlp9WDxwYv3s4mVWOn70vkZJ0=; b=oPZ7b+WAcwu1l9Mom+qavfTwK7T24rGH15hGpsu9//Cy6f8nwjfNAlAeLUXft0CMKP NJMC41siHz/OCV/bqqawIE1jiU+QQVI0IlzVgvSBpaYZdguJ6SloaI2Fx6ytZCo4WIHo hHOBdVrLGZMVbMRYpKEAZc8o/Pv6ZieNHHJ2t/adkMGhessv/HJmR9VzEZQ/S7/IBPbU zqqgfhVCFsWT3VxkS7S+r9lvoNX3CDNub7KSi+vMe9VzrPpd1MhT8M1ar8fKP/u2H3C9 jquzbrj7H68D/5a8BxdclGuBNlPG7Uxf4+ljV0zL8SuwMphyHsxD3B6KKgpRXZf4IHED JErg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWZ0gkBmXHcvTC3I0g+gDCaAKlH4TwD83jj+BMTkWUN/AfFHywsX BJRBI6HKXAr6TxG7VkrogbYxlE0x0ilt1CCRMcqjEaRfNQoPUkn+ybij8RQE3BcHopNitBx2X9B BD0xL+/CNa0fGzuNqyISUGNFIgkT6MckD1SaQVmuQTf4rEhyVqbSbjPDlA7qJXR+lEQ== X-Received: by 2002:a50:98a6:: with SMTP id j35mr45802620edb.141.1546606366414; Fri, 04 Jan 2019 04:52:46 -0800 (PST) X-Google-Smtp-Source: AFSGD/WDVAlvoS+CGMG4iMlGpPECxfOjAiZxXQdnzueQT6gxB0zXswBsA7zEcHF2Lanz0n+QrmTy X-Received: by 2002:a50:98a6:: with SMTP id j35mr45802576edb.141.1546606365315; Fri, 04 Jan 2019 04:52:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606365; cv=none; d=google.com; s=arc-20160816; b=SoxajhyN+5acztqaSXU5Ojns7lExyHuiAP/nD56zmK3kWlPX+BdpHPXYokumEEjF1C ko90Q7r9lJkmsDwrHypympv6XGJNCA+Nw3xX3DkPQaj7/ikmerDBu9nfPXUTXiOQPCzC A3lVz1RNmNagtlXG3NJEmItFidUXKla6SiLVbvItHKkpLB+yhptf+zubA9dvr2/g+kAp xr+dtzmFze+anVKqPaZjUu5atczv4upaOXJ2To1sNrBk62s8YE/QIn/A1Af/m2QQpdms GP8/w5yBMs2SRtpkltkByi+B+KsOfi26PGqLPaUoyLCncbtpd5KS5l5xVrWSm/SBWqsf nYBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=/hdAGFXBogCDzBnIyWLlp9WDxwYv3s4mVWOn70vkZJ0=; b=PybZ2NQFnphGUaPm5++IomNckUkl6THvA5IT08N68WiOZa5kiJ80aD8Ge+vYFcTYlz FX69D4CrLpPEmQ4HkNJGwtEEhgzI+vEmI59rrsWvOGzdxLdEG5z23TqqXTZeRBA6Hf0g hqFMRmXAUKCZ0jaxn/L2fm+hPWEmB/qtf56cQ77dIOOAnFB6ymra9BAAm/ZEal1GZH9W 3rCC9AvGJf2YSuyWtGvWrxbznH9fO2U5moqiPuf326ZHCp5GMdIIjMoNUI9YHosYNT8E IVk1PHqaMg6dCpTZFw89gdLyS/Will0zdUZKEKDMwDpgr+1TW56riO3QXQEvcSVUBLFH TAZQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp10.blacknight.com (outbound-smtp10.blacknight.com. [46.22.139.15]) by mx.google.com with ESMTPS id u22-v6si2809239ejb.233.2019.01.04.04.52.45 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:52:45 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) client-ip=46.22.139.15; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp10.blacknight.com (Postfix) with ESMTPS id EA6291C1CDA for ; Fri, 4 Jan 2019 12:52:44 +0000 (GMT) Received: (qmail 4922 invoked from network); 4 Jan 2019 12:52:44 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:52:44 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 14/25] mm, compaction: Avoid rescanning the same pageblock multiple times Date: Fri, 4 Jan 2019 12:50:00 +0000 Message-Id: <20190104125011.16071-15-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Pageblocks are marked for skip when no pages are isolated after a scan. However, it's possible to hit corner cases where the migration scanner gets stuck near the boundary between the source and target scanner. Due to pages being migrated in blocks of COMPACT_CLUSTER_MAX, pages that are migrated can be reallocated before the pageblock is complete. The pageblock is not necessarily skipped so it can be rescanned multiple times. Similarly, a pageblock with some dirty/writeback pages may fail to isolate and be rescanned until writeback completes which is wasteful. This patch tracks if a pageblock is being rescanned. If so, then the entire pageblock will be migrated as one operation. This narrows the race window during which pages can be reallocated during migration. Secondly, if there are pages that cannot be isolated then the pageblock will still be fully scanned and marked for skipping. On the second rescan, the pageblock skip is set and the migration scanner makes progress. 4.20.0 4.20.0 finishscan-v2r15 norescan-v2r15 Amean fault-both-3 3729.80 ( 0.00%) 2872.13 * 23.00%* Amean fault-both-5 5148.49 ( 0.00%) 4330.56 * 15.89%* Amean fault-both-7 7393.24 ( 0.00%) 6496.63 ( 12.13%) Amean fault-both-12 11709.32 ( 0.00%) 10280.59 ( 12.20%) Amean fault-both-18 16626.82 ( 0.00%) 11079.19 * 33.37%* Amean fault-both-24 19944.34 ( 0.00%) 17207.80 * 13.72%* Amean fault-both-30 23435.53 ( 0.00%) 17736.13 * 24.32%* Amean fault-both-32 23948.70 ( 0.00%) 18509.41 * 22.71%* 4.20.0 4.20.0 finishscan-v2r15 norescan-v2r15 Percentage huge-1 0.00 ( 0.00%) 0.00 ( 0.00%) Percentage huge-3 88.39 ( 0.00%) 96.87 ( 9.60%) Percentage huge-5 92.07 ( 0.00%) 94.63 ( 2.77%) Percentage huge-7 91.96 ( 0.00%) 93.83 ( 2.03%) Percentage huge-12 93.38 ( 0.00%) 92.65 ( -0.78%) Percentage huge-18 91.89 ( 0.00%) 93.66 ( 1.94%) Percentage huge-24 91.37 ( 0.00%) 93.15 ( 1.95%) Percentage huge-30 92.77 ( 0.00%) 93.16 ( 0.42%) Percentage huge-32 87.97 ( 0.00%) 92.58 ( 5.24%) The fault latency reduction is large and while the THP allocation success rate is only slightly higher, it's already high at this point of the series. Compaction migrate scanned 60718343.00 31772603.00 Compaction free scanned 933061894.00 63267928.00 Migration scan rates are reduced by 48% and free scan rates are also reduced as the same migration source block is not being selected multiple times. The corner case where migration scan rates go through the roof due to a dirty/writeback pageblock located at the boundary of the migration/free scanner did not happen in this case. When it does happen, the scan rates multiple by factors measured in the hundreds and would be misleading to present. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 32 ++++++++++++++++++++++++++------ mm/internal.h | 1 + 2 files changed, 27 insertions(+), 6 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 9438f0564ed5..9c2cc7955446 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -959,8 +959,11 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, cc->nr_migratepages++; nr_isolated++; - /* Avoid isolating too much */ - if (cc->nr_migratepages == COMPACT_CLUSTER_MAX) { + /* + * Avoid isolating too much unless this block is being + * rescanned (e.g. dirty/writeback pages, parallel allocation). + */ + if (cc->nr_migratepages == COMPACT_CLUSTER_MAX && !cc->rescan) { ++low_pfn; break; } @@ -1007,11 +1010,14 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, spin_unlock_irqrestore(zone_lru_lock(zone), flags); /* - * Updated the cached scanner pfn if the pageblock was scanned - * without isolating a page. The pageblock may not be marked - * skipped already if there were no LRU pages in the block. + * Updated the cached scanner pfn once the pageblock has been scanned + * Pages will either be migrated in which case there is no point + * scanning in the near future or migration failed in which case the + * failure reason may persist. The block is marked for skipping if + * there were no pages isolated in the block or if the block is + * rescanned twice in a row. */ - if (low_pfn == end_pfn && !nr_isolated) { + if (low_pfn == end_pfn && (!nr_isolated || cc->rescan)) { if (valid_page && !skip_updated) set_pageblock_skip(valid_page); update_cached_migrate(cc, low_pfn); @@ -2031,6 +2037,20 @@ static enum compact_result compact_zone(struct compact_control *cc) int err; unsigned long start_pfn = cc->migrate_pfn; + /* + * Avoid multiple rescans which can happen if a page cannot be + * isolated (dirty/writeback in async mode) or if the migrated + * pages are being allocated before the pageblock is cleared. + * The first rescan will capture the entire pageblock for + * migration. If it fails, it'll be marked skip and scanning + * will proceed as normal. + */ + cc->rescan = false; + if (pageblock_start_pfn(last_migrated_pfn) == + pageblock_start_pfn(start_pfn)) { + cc->rescan = true; + } + switch (isolate_migratepages(cc->zone, cc)) { case ISOLATE_ABORT: ret = COMPACT_CONTENDED; diff --git a/mm/internal.h b/mm/internal.h index b25b33c5dd80..e5ca2a10b8ad 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -204,6 +204,7 @@ struct compact_control { bool direct_compaction; /* False from kcompactd or /proc/... */ bool whole_zone; /* Whole zone should/has been scanned */ bool contended; /* Signal lock or sched contention */ + bool rescan; /* Rescanning the same pageblock */ }; unsigned long From patchwork Fri Jan 4 12:50:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748325 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D4D3214E5 for ; Fri, 4 Jan 2019 12:52:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C5074283C9 for ; Fri, 4 Jan 2019 12:52:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B96172844C; Fri, 4 Jan 2019 12:52:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E37E728437 for ; Fri, 4 Jan 2019 12:52:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 050BB8E00E1; Fri, 4 Jan 2019 07:52:58 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id F1A828E00AE; Fri, 4 Jan 2019 07:52:57 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E0A658E00E1; Fri, 4 Jan 2019 07:52:57 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 814868E00AE for ; Fri, 4 Jan 2019 07:52:57 -0500 (EST) Received: by mail-ed1-f72.google.com with SMTP id f31so34824976edf.17 for ; Fri, 04 Jan 2019 04:52:57 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=gTcL3Zn2fYDut6yLa/gKEZRVuDmmBu1i0RhJNaSDP/Y=; b=Aq+XKUhQs53x0f4m9UJzlTJHuoIAGl6VbN8Zp2wce1jvJtVv1r60Xxpnb9xU8tcZxl DIPP0+R0JqwjIdsrmwx5m5xskGNRTqn9Qr3IsdAKA1+JTuJd3vw+GAQXUUoHiJTIyMSJ kVww+aI8N8+ILnZSg8Sb9tm5keKuDk3K1e6d4vDH3bI58MdwudXYaxUyCCXNGMrZDNSX JtKjNalf6KW8DVivuGUWbxVi4kSh3VYJY3Pc4t5ZXAzXYfATV8BWWharLtfhntCpJRI6 g4Ux0MefbXjvvIIu7fHP1LrVenugOamAYOIp1c/krfJNSMDeTYuLWAWXM1a8BKWnoghH QP4Q== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AJcUukfrUm5CDFdDcvNoi8rc1P7BNYBysiZLrU5JaSR8ZZ6ior6Dj4oQ pb0Q40ObkST7PaxQiLEIA3X52cyYIhn3LeipHV+v6kL/yvsFAgTN73RhNvjM5kaUB+A16GGAqwz 8z2AJIEHMy/XgCXau2+BleVE3nQ+26A6xBNq7ysaNnBIltcehZ2QsF19YtaDTJRWeFg== X-Received: by 2002:a17:906:2cd3:: with SMTP id r19-v6mr30389828ejr.38.1546606376975; Fri, 04 Jan 2019 04:52:56 -0800 (PST) X-Google-Smtp-Source: ALg8bN7BxxGC1L/RtvHVbXbM9716QX3qXZHPS/eiN27SyBcGPSV+nDrICqNq0nfaxaicf1UdRIqO X-Received: by 2002:a17:906:2cd3:: with SMTP id r19-v6mr30389773ejr.38.1546606375494; Fri, 04 Jan 2019 04:52:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606375; cv=none; d=google.com; s=arc-20160816; b=1EgIeKUXxwScTYlekpF4oS5/sENZ5XmG8YDoLe2PHbwoZ8UqoOwRY6ntFJNkdeHPY1 JoQVa+lUUhy8H4HIu2krQo71I44zw95qrmqsh3B46maa4EzfOaMUBYyz1mfxYD61v/Tb Ws1BI/J2Fg37xvtmVCX6G2mrf1HOtjabUXp8UfgkuszvCXNl/6RT2zjL8WjbwUeX+I9F E6S8TlVHm9R1N03HZ5Q5V/jtV2w8gsbXihx8fnhPzq9wxw7OEFZOeGIsOI26FlJKzVaR HepqHZy6I8W4T7iY+cpArZDIhstCZd0jLO25uN30iM4raBYVvzUn1vfW4JjN4YcHE4O7 GhGA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=gTcL3Zn2fYDut6yLa/gKEZRVuDmmBu1i0RhJNaSDP/Y=; b=hXxBu9CtUVd7iK2BiVikJ1ORHRafAeoKtD0f1F8zx2yfmy2dpIYShBTuetEKiwnQyL x5EjcVXL21x6FmqV6tVp1ZkdL87V9kr3C6X4YcnUdVTiVQD7s2vZi3XAYQOjYrGzPFtr pJ82tO7PbT6JGKjC5zrJxecZg4xcyKwByMvwxUarQV1Mpi9NjzKtjfqIuykU2g6gs5ia MuihsVZKh7R3BMUl7KbOsrohrAT9UmwVATcrB5I/25wi1k3FIaN5CNyrkJKd3A6a2jYg yS6dkLqDQvmvC/8hWVMiljjQZVsHnbSWCYPCv5OF6MSLs8DFPCxXuATGmFYx2/7s8qqj C50A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp08.blacknight.com (outbound-smtp08.blacknight.com. [46.22.139.13]) by mx.google.com with ESMTPS id r18-v6si2062593ejf.218.2019.01.04.04.52.55 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:52:55 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) client-ip=46.22.139.13; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp08.blacknight.com (Postfix) with ESMTPS id 221F01C1C4F for ; Fri, 4 Jan 2019 12:52:55 +0000 (GMT) Received: (qmail 5515 invoked from network); 4 Jan 2019 12:52:55 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:52:55 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 15/25] mm, compaction: Finish pageblock scanning on contention Date: Fri, 4 Jan 2019 12:50:01 +0000 Message-Id: <20190104125011.16071-16-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Async migration aborts on spinlock contention but contention can be high when there are multiple compaction attempts and kswapd is active. The consequence is that the migration scanners move forward uselessly while still contending on locks for longer while leaving suitable migration sources behind. This patch will acquire the lock but track when contention occurs. When it does, the current pageblock will finish as compaction may succeed for that block and then abort. This will have a variable impact on latency as in some cases useless scanning is avoided (reduces latency) but a lock will be contended (increase latency) or a single contended pageblock is scanned that would otherwise have been skipped (increase latency). 4.20.0 4.20.0 norescan-v2r15 finishcontend-v2r15 Amean fault-both-1 0.00 ( 0.00%) 0.00 * 0.00%* Amean fault-both-3 2872.13 ( 0.00%) 2973.08 ( -3.51%) Amean fault-both-5 4330.56 ( 0.00%) 3870.19 ( 10.63%) Amean fault-both-7 6496.63 ( 0.00%) 6580.50 ( -1.29%) Amean fault-both-12 10280.59 ( 0.00%) 9527.40 ( 7.33%) Amean fault-both-18 11079.19 ( 0.00%) 13395.86 * -20.91%* Amean fault-both-24 17207.80 ( 0.00%) 14936.94 * 13.20%* Amean fault-both-30 17736.13 ( 0.00%) 16748.46 ( 5.57%) Amean fault-both-32 18509.41 ( 0.00%) 18521.30 ( -0.06%) 4.20.0 4.20.0 norescan-v2r15 finishcontend-v2r15 Percentage huge-1 0.00 ( 0.00%) 0.00 ( 0.00%) Percentage huge-3 96.87 ( 0.00%) 97.57 ( 0.72%) Percentage huge-5 94.63 ( 0.00%) 96.88 ( 2.39%) Percentage huge-7 93.83 ( 0.00%) 95.47 ( 1.74%) Percentage huge-12 92.65 ( 0.00%) 98.64 ( 6.47%) Percentage huge-18 93.66 ( 0.00%) 98.33 ( 4.98%) Percentage huge-24 93.15 ( 0.00%) 98.88 ( 6.15%) Percentage huge-30 93.16 ( 0.00%) 97.09 ( 4.21%) Percentage huge-32 92.58 ( 0.00%) 96.20 ( 3.92%) As expected, a variable impact on latency while allocation success rates are slightly higher. System CPU usage is reduced by about 10% but scan rate impact is mixed Compaction migrate scanned 31772603 19980216 Compaction free scanned 63267928 120381828 Migration scan rates are reduced 37% which is expected as a pageblock is used by the async scanner instead of skipped but the free scanning is increased. This can be partially accounted for by the increased success rate but also by the fact that the scanners do not meet for longer when pageblocks are actually used. Overall this is justified and completing a pageblock scan is very important for later patches. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 95 +++++++++++++++++++++++---------------------------------- 1 file changed, 39 insertions(+), 56 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 9c2cc7955446..608d274f9880 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -376,24 +376,25 @@ static bool test_and_set_skip(struct compact_control *cc, struct page *page, /* * Compaction requires the taking of some coarse locks that are potentially - * very heavily contended. For async compaction, back out if the lock cannot - * be taken immediately. For sync compaction, spin on the lock if needed. + * very heavily contended. For async compaction, trylock and record if the + * lock is contended. The lock will still be acquired but compaction will + * abort when the current block is finished regardless of success rate. + * Sync compaction acquires the lock. * - * Returns true if the lock is held - * Returns false if the lock is not held and compaction should abort + * Always returns true which makes it easier to track lock state in callers. */ -static bool compact_trylock_irqsave(spinlock_t *lock, unsigned long *flags, +static bool compact_lock_irqsave(spinlock_t *lock, unsigned long *flags, struct compact_control *cc) { - if (cc->mode == MIGRATE_ASYNC) { - if (!spin_trylock_irqsave(lock, *flags)) { - cc->contended = true; - return false; - } - } else { - spin_lock_irqsave(lock, *flags); + /* Track if the lock is contended in async mode */ + if (cc->mode == MIGRATE_ASYNC && !cc->contended) { + if (spin_trylock_irqsave(lock, *flags)) + return true; + + cc->contended = true; } + spin_lock_irqsave(lock, *flags); return true; } @@ -426,10 +427,8 @@ static bool compact_unlock_should_abort(spinlock_t *lock, } if (need_resched()) { - if (cc->mode == MIGRATE_ASYNC) { + if (cc->mode == MIGRATE_ASYNC) cc->contended = true; - return true; - } cond_resched(); } @@ -449,10 +448,8 @@ static inline bool compact_should_abort(struct compact_control *cc) { /* async compaction aborts if contended */ if (need_resched()) { - if (cc->mode == MIGRATE_ASYNC) { + if (cc->mode == MIGRATE_ASYNC) cc->contended = true; - return true; - } cond_resched(); } @@ -538,18 +535,8 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, * recheck as well. */ if (!locked) { - /* - * The zone lock must be held to isolate freepages. - * Unfortunately this is a very coarse lock and can be - * heavily contended if there are parallel allocations - * or parallel compactions. For async compaction do not - * spin on the lock and we acquire the lock as late as - * possible. - */ - locked = compact_trylock_irqsave(&cc->zone->lock, + locked = compact_lock_irqsave(&cc->zone->lock, &flags, cc); - if (!locked) - break; /* Recheck this is a buddy page under lock */ if (!PageBuddy(page)) @@ -910,15 +897,9 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, /* If we already hold the lock, we can skip some rechecking */ if (!locked) { - locked = compact_trylock_irqsave(zone_lru_lock(zone), + locked = compact_lock_irqsave(zone_lru_lock(zone), &flags, cc); - /* Allow future scanning if the lock is contended */ - if (!locked) { - clear_pageblock_skip(page); - break; - } - /* Try get exclusive access under lock */ if (!skip_updated) { skip_updated = true; @@ -961,9 +942,12 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, /* * Avoid isolating too much unless this block is being - * rescanned (e.g. dirty/writeback pages, parallel allocation). + * rescanned (e.g. dirty/writeback pages, parallel allocation) + * or a lock is contended. For contention, isolate quickly to + * potentially remove one source of contention. */ - if (cc->nr_migratepages == COMPACT_CLUSTER_MAX && !cc->rescan) { + if (cc->nr_migratepages == COMPACT_CLUSTER_MAX && + !cc->rescan && !cc->contended) { ++low_pfn; break; } @@ -1411,12 +1395,8 @@ static void isolate_freepages(struct compact_control *cc) isolate_freepages_block(cc, &isolate_start_pfn, block_end_pfn, freelist, false); - /* - * If we isolated enough freepages, or aborted due to lock - * contention, terminate. - */ - if ((cc->nr_freepages >= cc->nr_migratepages) - || cc->contended) { + /* Are enough freepages isolated? */ + if (cc->nr_freepages >= cc->nr_migratepages) { if (isolate_start_pfn >= block_end_pfn) { /* * Restart at previous pageblock if more @@ -1458,13 +1438,8 @@ static struct page *compaction_alloc(struct page *migratepage, struct compact_control *cc = (struct compact_control *)data; struct page *freepage; - /* - * Isolate free pages if necessary, and if we are not aborting due to - * contention. - */ if (list_empty(&cc->freepages)) { - if (!cc->contended) - isolate_freepages(cc); + isolate_freepages(cc); if (list_empty(&cc->freepages)) return NULL; @@ -1729,7 +1704,7 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone, low_pfn = isolate_migratepages_block(cc, low_pfn, block_end_pfn, isolate_mode); - if (!low_pfn || cc->contended) + if (!low_pfn) return ISOLATE_ABORT; /* @@ -1759,9 +1734,7 @@ static enum compact_result __compact_finished(struct compact_control *cc) { unsigned int order; const int migratetype = cc->migratetype; - - if (cc->contended || fatal_signal_pending(current)) - return COMPACT_CONTENDED; + int ret; /* Compaction run completes if the migrate and free scanner meet */ if (compact_scanners_met(cc)) { @@ -1796,6 +1769,7 @@ static enum compact_result __compact_finished(struct compact_control *cc) return COMPACT_CONTINUE; /* Direct compactor: Is a suitable page free? */ + ret = COMPACT_NO_SUITABLE_PAGE; for (order = cc->order; order < MAX_ORDER; order++) { struct free_area *area = &cc->zone->free_area[order]; bool can_steal; @@ -1835,11 +1809,15 @@ static enum compact_result __compact_finished(struct compact_control *cc) return COMPACT_SUCCESS; } - return COMPACT_CONTINUE; + ret = COMPACT_CONTINUE; + break; } } - return COMPACT_NO_SUITABLE_PAGE; + if (cc->contended || fatal_signal_pending(current)) + ret = COMPACT_CONTENDED; + + return ret; } static enum compact_result compact_finished(struct compact_control *cc) @@ -1981,6 +1959,7 @@ static enum compact_result compact_zone(struct compact_control *cc) unsigned long end_pfn = zone_end_pfn(cc->zone); unsigned long last_migrated_pfn; const bool sync = cc->mode != MIGRATE_ASYNC; + unsigned long a, b, c; cc->migratetype = gfpflags_to_migratetype(cc->gfp_mask); ret = compaction_suitable(cc->zone, cc->order, cc->alloc_flags, @@ -2026,6 +2005,10 @@ static enum compact_result compact_zone(struct compact_control *cc) cc->whole_zone = true; } + a = cc->migrate_pfn; + b = cc->free_pfn; + c = (cc->free_pfn - cc->migrate_pfn) / pageblock_nr_pages; + last_migrated_pfn = 0; trace_mm_compaction_begin(start_pfn, cc->migrate_pfn, From patchwork Fri Jan 4 12:50:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748327 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 332BB14E5 for ; Fri, 4 Jan 2019 12:53:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 22EAF283C9 for ; Fri, 4 Jan 2019 12:53:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 177792844C; Fri, 4 Jan 2019 12:53:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 96E37283C9 for ; Fri, 4 Jan 2019 12:53:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B72E08E00E2; Fri, 4 Jan 2019 07:53:07 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AFC0D8E00AE; Fri, 4 Jan 2019 07:53:07 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9C43A8E00E2; Fri, 4 Jan 2019 07:53:07 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id 39E728E00AE for ; Fri, 4 Jan 2019 07:53:07 -0500 (EST) Received: by mail-ed1-f69.google.com with SMTP id c3so35319862eda.3 for ; Fri, 04 Jan 2019 04:53:07 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=mqc/z7vuvN6xwwTBO71f8UjlpUOyDJMtrxPbpJW/JW8=; b=USZJwGC8yjtxq7WN7chVeJ5/R/ZuX3hwre2vH192OzM0qZNrL5XfYIsolvqrXhgl7C YKstq/9SumDGC8o/UYax67zN3b8dyIbURKynimKkyOlsRt8E4cmuHeOAZ4IN8myXsfB8 iN1pDaEFHWXASVt7vq3h0S2qaFNGNCyWdmWFcZq2xpQd4DXnQnvpk1wKuhqAvW48iShn SrfMBtQjLfSNHEqHguMw86cxAfUBq4nb/rJcWLdb33JdF144RjtloujDdEIYrYf0VM72 OffUvZK0bRxYMo7/bsIBr2AQ7OqLehHRJrkaBNge95opCsZeeFmVl3nqmR9m1PjZDfVF vTeQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWYupPYnPBz9QZ5X/yK7qIzmNj8DRd2ZH0/2MtW4pDGpphwT8nVk 61OiYZL+fOtYmtEVl5btistpPaSZQjy6ALz6QP5pXQwNJjx+zxghm5TEfzA4foQj15tuk9YmqC3 ruZrLTlSRvpumPHLlIz4Hq7VPB4ScUw6oJn/bVAeaK+y50YDVUmmdC4/gFfo7DkSUjg== X-Received: by 2002:a50:8907:: with SMTP id e7mr47254216ede.252.1546606386720; Fri, 04 Jan 2019 04:53:06 -0800 (PST) X-Google-Smtp-Source: AFSGD/WEapednZ25arxRUbIfM2R+FVKxXwTNCsJGvEE1jhOGZ1TdveEAwmye5uBRhwAEkMk28SE2 X-Received: by 2002:a50:8907:: with SMTP id e7mr47254161ede.252.1546606385614; Fri, 04 Jan 2019 04:53:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606385; cv=none; d=google.com; s=arc-20160816; b=wc8ypkB/5DAsA+D2GwTo0NoqojAYW673stA/W1k47QNmNAxj7ZM7f7aRFYlTuLnmXF nwVpV9rRWkC0vjY2nHshwZgJG5peuSDI+Z33YuXEC1j7caKwO8yb+UeO/EPeYAZuK7YJ jKx7Zy19ICQ983zJTsU9e0IAqzEVwCCCHOvMU60pbnQieqIFkW8yS5lEB0kB3IPSCIT7 YWoz3zl+bA+WQeDAM2f6PT6y6WczSqGyg4WDzuSQh/FVmRuxZ7FN6hEr7T/iUKUD4yUV IpxU+5mhKlgd8pQOdFdbCmWN1sPFOgM4z37k8Ej8uGX8UGGsdDGNPpTeNM3qL9ra/2JZ 8Rvw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=mqc/z7vuvN6xwwTBO71f8UjlpUOyDJMtrxPbpJW/JW8=; b=Ac66/d9U/9iVZ/B9eQzbWoOitjpWNANNR81IFwOHHMeIceVSwknZ/TaJENE6ICfee7 /h77eqBzB+yUgP2S3rzPxlaxCNsYq/jSbKhqFKhGB1Xs6QCrUxFjto3IkM679L7jO+M4 rpn+ZQdeVBBaSNhem5KMYZsh/WRwWiKnkyoiUmWjDmCxRdhUVL/jeNfAt7KnfBMTEuJ6 SP/YsK2ZeKUppZnjbtt2b16J0Vn69S2OOO0LEE1DR2AQE8/sh2wxsOvhYv4KKagBCLwP fRm3041NLBJvo27QNpsBI1MZDF07rimguINVkbW8BhguzJqPSl4ugPaikt6cS0UiCLoo EIIg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp04.blacknight.com (outbound-smtp04.blacknight.com. [81.17.249.35]) by mx.google.com with ESMTPS id k7si7138254edx.327.2019.01.04.04.53.05 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:53:05 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) client-ip=81.17.249.35; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp04.blacknight.com (Postfix) with ESMTPS id 4A8B1988BA for ; Fri, 4 Jan 2019 12:53:05 +0000 (UTC) Received: (qmail 6206 invoked from network); 4 Jan 2019 12:53:05 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:53:05 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 16/25] mm, compaction: Check early for huge pages encountered by the migration scanner Date: Fri, 4 Jan 2019 12:50:02 +0000 Message-Id: <20190104125011.16071-17-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP When scanning for sources or targets, PageCompound is checked for huge pages as they can be skipped quickly but it happens relatively late after a lot of setup and checking. This patch short-cuts the check to make it earlier. It might still change when the lock is acquired but this has less overhead overall. The free scanner advances but the migration scanner does not. Typically the free scanner encounters more movable blocks that change state over the lifetime of the system and also tends to scan more aggressively as it's actively filling its portion of the physical address space with data. This could change in the future but for the moment, this worked better in practice and incurred fewer scan restarts. The impact on latency and allocation success rates is marginal but the free scan rates are reduced by 32% and system CPU usage is reduced by 2.6%. The 2-socket results are not materially different. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 608d274f9880..921720f7a416 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1071,6 +1071,9 @@ static bool suitable_migration_source(struct compact_control *cc, { int block_mt; + if (pageblock_skip_persistent(page)) + return false; + if ((cc->mode != MIGRATE_ASYNC) || !cc->direct_compaction) return true; @@ -1693,12 +1696,17 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone, continue; /* - * For async compaction, also only scan in MOVABLE blocks. - * Async compaction is optimistic to see if the minimum amount - * of work satisfies the allocation. + * For async compaction, also only scan in MOVABLE blocks + * without huge pages. Async compaction is optimistic to see + * if the minimum amount of work satisfies the allocation. + * The cached PFN is updated as it's possible that all + * remaining blocks between source and target are suitable + * and the compaction scanners fail to meet. */ - if (!suitable_migration_source(cc, page)) + if (!suitable_migration_source(cc, page)) { + update_cached_migrate(cc, block_end_pfn); continue; + } /* Perform the isolation */ low_pfn = isolate_migratepages_block(cc, low_pfn, From patchwork Fri Jan 4 12:50:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748329 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4790D13B5 for ; Fri, 4 Jan 2019 12:53:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 37AB22844C for ; Fri, 4 Jan 2019 12:53:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2B3BB28471; Fri, 4 Jan 2019 12:53:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B2D5E2844C for ; Fri, 4 Jan 2019 12:53:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D47A88E00E3; Fri, 4 Jan 2019 07:53:17 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CD0A08E00AE; Fri, 4 Jan 2019 07:53:17 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B9AE28E00E3; Fri, 4 Jan 2019 07:53:17 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by kanga.kvack.org (Postfix) with ESMTP id 5A02A8E00AE for ; Fri, 4 Jan 2019 07:53:17 -0500 (EST) Received: by mail-ed1-f70.google.com with SMTP id t7so34967632edr.21 for ; Fri, 04 Jan 2019 04:53:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=DVHlWS1AJ7m4k2BsTyIoPk8zEJulUYCQzTwyj8ztOYY=; b=QrmAxGIk/rnsJFd9s0rsLOekER/eYdx4nOV8/6xt3r8X+3PiQs9yWhCcfjAkBSzjEV H5++wvmHOKxY+bSLvp4BkCAebA7iIIfmcuXubrCS88vt8xyew/ZANXabvqTPzepsP8xq hPUzGSzgCC+QSSElbe0uPIfTdqLNyyZxCicc3SvQVf2rIP5z7CPkNpvjnSEYeqXd0Y+N uL/w4hbn5q5Ob3tJdSqKA0cy9mvCVi3TGhXVRt896tCv9cPFVBkZjnoTW7L6jDF45Pyc /dGZMDxYcHephfIZfE8eAniqzh0H3+jZHmWqGDMXsCr3vPAjAZUVe99IzUo9qLFB6O3Y 3Jrg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWaxRbqp3venzXDFaFg8D2WwpCr5CsIqNuIN0RUrQ6UKNdvoiqdD 8Xw+UQSaxpedpVOvB5mKpAkUeVbrN06iAfm/wxBQlf6BgGvMEOmqhRGjkIsxd1hzO5Ur+vsfG5C XR94THeywpdl44Je+c/GM3x+PYWq6jElfjPMK5f07VK9mAR8e8YWrtdu6FxjHsdZygg== X-Received: by 2002:a50:9977:: with SMTP id l52mr43863958edb.277.1546606396841; Fri, 04 Jan 2019 04:53:16 -0800 (PST) X-Google-Smtp-Source: AFSGD/VjsbvCG+znR2MoveV/U8a8177TyIDYkljMipU0PlIjydoPX5oiRo2Z1ZVa1/uCjjl39XTf X-Received: by 2002:a50:9977:: with SMTP id l52mr43863924edb.277.1546606395927; Fri, 04 Jan 2019 04:53:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606395; cv=none; d=google.com; s=arc-20160816; b=sXR4y+KRYVBCwe4YviQy1vJg8yGdQwpFWVBvY2tApqHlFk72+1gZpYodnTx8GJ0NRX 1+UEH8GAbvC+BlUOv30HZCKV45xXYyjhb9njFQ4xaOu/ke5kKRws9KRY0jk21So5Gd/R Jv59pyGWyexa8hQSQXilRmkHPsaMLEXJp2EA60pX34n5th7Fe/zlZVgoLMBZ/B/AB0RR 7KOSBVLlKJdSyIr9Ye13Y9KYvzU5PvSUR8ximWJhei9VU2ycggpJtpHP9tFmePyBk/lL U2Eoczgg0DbVsL2IE9RaHc9+b/oel41uZNaWQKy8DMxNpJK6mW9iCi4eAVgYHP6QoMV8 Mujw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=DVHlWS1AJ7m4k2BsTyIoPk8zEJulUYCQzTwyj8ztOYY=; b=cXiEXfHMmIRJ5Dz29pvHFxZOnhiy2G+BbgnO29ggS2IPDLzsVmg/knQ5962L2fL41Y UzASM0IL2fufSnVq41vByLcQ2zOvSDGeTySHQG5BYg4Y3q/uYgGtHUeIIBKM6JF25ui1 JuJseRxdcOJqj30gTkNLC6DvPf/fbt1D+qEBF3UixDuB1qh9Voq5CACymLMBZqUQ9y32 gbikojb8VMIXZbqBCUJO234V87nxi3h+5RDUcXd+hHdDzU05cXCcVOhwLHOAF8cIazVI y6gu5C2dasYrUrIgtt3jNiDEAEGHbNZUgf8xVtAI4Mqj8Ea0Zdftigw5GTqo0RwrENhz 5VxA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp13.blacknight.com (outbound-smtp13.blacknight.com. [46.22.139.230]) by mx.google.com with ESMTPS id h13si2217209edf.24.2019.01.04.04.53.15 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:53:15 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) client-ip=46.22.139.230; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (unknown [81.17.254.16]) by outbound-smtp13.blacknight.com (Postfix) with ESMTPS id 8096A1C1788 for ; Fri, 4 Jan 2019 12:53:15 +0000 (GMT) Received: (qmail 6693 invoked from network); 4 Jan 2019 12:53:15 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:53:15 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 17/25] mm, compaction: Keep cached migration PFNs synced for unusable pageblocks Date: Fri, 4 Jan 2019 12:50:03 +0000 Message-Id: <20190104125011.16071-18-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Migrate has separate cached PFNs for ASYNC and SYNC* migration on the basis that some migrations will fail in ASYNC mode. However, if the cached PFNs match at the start of scanning and pageblocks are skipped due to having no isolation candidates, then the sync state does not matter. This patch keeps matching cached PFNs in sync until a pageblock with isolation candidates is found. The actual benefit is marginal given that the sync scanner following the async scanner will often skip a number of pageblocks but it's useless work. Any benefit depends heavily on whether the scanners restarted recently so overall the reduction in scan rates is a mere 2.8% which is borderline noise. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/mm/compaction.c b/mm/compaction.c index 921720f7a416..be27e4fa1b40 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1967,6 +1967,7 @@ static enum compact_result compact_zone(struct compact_control *cc) unsigned long end_pfn = zone_end_pfn(cc->zone); unsigned long last_migrated_pfn; const bool sync = cc->mode != MIGRATE_ASYNC; + bool update_cached; unsigned long a, b, c; cc->migratetype = gfpflags_to_migratetype(cc->gfp_mask); @@ -2019,6 +2020,17 @@ static enum compact_result compact_zone(struct compact_control *cc) last_migrated_pfn = 0; + /* + * Migrate has separate cached PFNs for ASYNC and SYNC* migration on + * the basis that some migrations will fail in ASYNC mode. However, + * if the cached PFNs match and pageblocks are skipped due to having + * no isolation candidates, then the sync state does not matter. + * Until a pageblock with isolation candidates is found, keep the + * cached PFNs in sync to avoid revisiting the same blocks. + */ + update_cached = !sync && + cc->zone->compact_cached_migrate_pfn[0] == cc->zone->compact_cached_migrate_pfn[1]; + trace_mm_compaction_begin(start_pfn, cc->migrate_pfn, cc->free_pfn, end_pfn, sync); @@ -2050,6 +2062,11 @@ static enum compact_result compact_zone(struct compact_control *cc) last_migrated_pfn = 0; goto out; case ISOLATE_NONE: + if (update_cached) { + cc->zone->compact_cached_migrate_pfn[1] = + cc->zone->compact_cached_migrate_pfn[0]; + } + /* * We haven't isolated and migrated anything, but * there might still be unflushed migrations from @@ -2057,6 +2074,7 @@ static enum compact_result compact_zone(struct compact_control *cc) */ goto check_drain; case ISOLATE_SUCCESS: + update_cached = false; last_migrated_pfn = start_pfn; ; } From patchwork Fri Jan 4 12:50:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748331 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A7F8913B5 for ; Fri, 4 Jan 2019 12:53:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 999A12844C for ; Fri, 4 Jan 2019 12:53:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8CFB828471; Fri, 4 Jan 2019 12:53:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 09AA52844C for ; Fri, 4 Jan 2019 12:53:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 224088E00E4; Fri, 4 Jan 2019 07:53:28 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1D3068E00AE; Fri, 4 Jan 2019 07:53:28 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C3C98E00E4; Fri, 4 Jan 2019 07:53:28 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by kanga.kvack.org (Postfix) with ESMTP id A28578E00AE for ; Fri, 4 Jan 2019 07:53:27 -0500 (EST) Received: by mail-ed1-f70.google.com with SMTP id o21so35178681edq.4 for ; Fri, 04 Jan 2019 04:53:27 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=cMRlI2NWPacNnN5U7NKMzhDaWWCj0Q44SgLJB2fJ6H4=; b=KvnrGH9WbB8FC/hJyeo+uMZRMukH7bnnthZOz0rB41MmHpGF+HRBgyM46MgBrwdI/b ubgRWusyLduP+mHHef6eeymW7REWxe6KDC3kSES/ssemdpc0IB5wFj7/72Ys4wHz9Eqn dedkhzrMaunzzwyOncNldnCnZVYDrka0lsOFH7umGVDgUQ770Hrpcqt4PJYqaD969adf mc/8t7eyblfQxJdb2kWNrBDS8HpYbUT/aLKQQGw75Lz0EzdECweMeN00qwlbNr4yokEU pDDmWIAUeqvxdcR8X5IrmWrhrtcS/LTO3TOMsFH0XsBAeff/yj4aA28CigT/4u6XJXGD hw4Q== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWbdZsewQ++Jwm4GZhZt68rmE76i+ClEyPFh/xGUct6z3s/7jJDD gyD2tE/065TRbwFNn+gqklfHOsn64uELI8oVyJfpzDWwTy9sdM8WqGGlcF7dZ9SBhZ5K/nSJ3YG LiN0hltiG38rhvltqF8Epdxp2eVFPBm6dSgDsCEUKDOpDK9nBCKU/kaRNtnSqo+ZuJQ== X-Received: by 2002:a50:e3cb:: with SMTP id c11mr47122384edm.80.1546606407154; Fri, 04 Jan 2019 04:53:27 -0800 (PST) X-Google-Smtp-Source: AFSGD/WBHoXjaSmhwpIm6Zox/ZWsuCCWATCU1kZkZdJpSO39NC2C6mKCXpcdkyI/9qQlbg1/sNaC X-Received: by 2002:a50:e3cb:: with SMTP id c11mr47122337edm.80.1546606406006; Fri, 04 Jan 2019 04:53:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606405; cv=none; d=google.com; s=arc-20160816; b=JU4L8qx0+dP7DZ1KWgzgGgpkFizK/QnjgjIj64v3WviFFbeVIG7X8bbdSf5NehJjx2 9DF57o7mkU6zjRdnV7Sk/lb6O3GvQ3KX0z76xC+BszRa8+kJV6D7YHOvtDLW36zeLOnf 8s0SgWP038fNp8qymU4pc6tDWSTeRaTiUHgNHoBnYTL4GSiZxfhOt8bJp3fwbzj5nK+h JFpRhZjoObCr35WbE67x9bMTqTZiTwA2lc08x3ogjIPPOCRibK6/FqxNpLcvP8z012Ds d5V8fZ04S7mwMlEu7Gi2BMoxbCXfdNDUP7tLK1qs5w9m8PrrCRgp7cwvDyPFxh6n2e+I IWNQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=cMRlI2NWPacNnN5U7NKMzhDaWWCj0Q44SgLJB2fJ6H4=; b=K7aD5jhpWS2ZNfbS69VUDzGh03E5g/kIz5PTFK3Ytidwy56ubQ1/wq7P3n2lSaqEfO LSup8CS1E4bbLYfOGRIGV9G4xM6Y5gt4LAlfRhPUdCGsbFDdQQV2zQpI4pgzqJirqmto GzbuSWsYmy9V/84KLFPn+dPbPaJMHTUxsDNybkXwg9Ske8AF5iAV9mkBZxyo/xLbvfL+ cHMl26y1V0N6KhWYTtkDMSZEQEOS3sJx7PZga1xYNNeSYSfABxyNroIAeSNZR1QM14Xo 6Ihft/+1wtZnqpg8ljaaqGKGEgxctUhAFUZ6yKylKz1oXtpDaI5ZJohw6alSTFJoELqa LAoA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp26.blacknight.com (outbound-smtp26.blacknight.com. [81.17.249.194]) by mx.google.com with ESMTPS id g14si7194628edy.160.2019.01.04.04.53.25 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:53:25 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) client-ip=81.17.249.194; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.194 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp26.blacknight.com (Postfix) with ESMTPS id A9632B87AC for ; Fri, 4 Jan 2019 12:53:25 +0000 (GMT) Received: (qmail 7595 invoked from network); 4 Jan 2019 12:53:25 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:53:25 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 18/25] mm, compaction: Rework compact_should_abort as compact_check_resched Date: Fri, 4 Jan 2019 12:50:04 +0000 Message-Id: <20190104125011.16071-19-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP With incremental changes, compact_should_abort no longer makes any documented sense. Rename to compact_check_resched and update the associated comments. There is no benefit other than reducing redundant code and making the intent slightly clearer. It could potentially be merged with earlier patches but it just makes the review slightly harder. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 61 ++++++++++++++++++++++----------------------------------- 1 file changed, 23 insertions(+), 38 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index be27e4fa1b40..1a41a2dbff24 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -398,6 +398,21 @@ static bool compact_lock_irqsave(spinlock_t *lock, unsigned long *flags, return true; } +/* + * Aside from avoiding lock contention, compaction also periodically checks + * need_resched() and records async compaction as contended if necessary. + */ +static inline void compact_check_resched(struct compact_control *cc) +{ + /* async compaction aborts if contended */ + if (need_resched()) { + if (cc->mode == MIGRATE_ASYNC) + cc->contended = true; + + cond_resched(); + } +} + /* * Compaction requires the taking of some coarse locks that are potentially * very heavily contended. The lock should be periodically unlocked to avoid @@ -426,33 +441,7 @@ static bool compact_unlock_should_abort(spinlock_t *lock, return true; } - if (need_resched()) { - if (cc->mode == MIGRATE_ASYNC) - cc->contended = true; - cond_resched(); - } - - return false; -} - -/* - * Aside from avoiding lock contention, compaction also periodically checks - * need_resched() and either schedules in sync compaction or aborts async - * compaction. This is similar to what compact_unlock_should_abort() does, but - * is used where no lock is concerned. - * - * Returns false when no scheduling was needed, or sync compaction scheduled. - * Returns true when async compaction should abort. - */ -static inline bool compact_should_abort(struct compact_control *cc) -{ - /* async compaction aborts if contended */ - if (need_resched()) { - if (cc->mode == MIGRATE_ASYNC) - cc->contended = true; - - cond_resched(); - } + compact_check_resched(cc); return false; } @@ -750,8 +739,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, return 0; } - if (compact_should_abort(cc)) - return 0; + compact_check_resched(cc); if (cc->direct_compaction && (cc->mode == MIGRATE_ASYNC)) { skip_on_failure = true; @@ -1374,12 +1362,10 @@ static void isolate_freepages(struct compact_control *cc) isolate_start_pfn = block_start_pfn) { /* * This can iterate a massively long zone without finding any - * suitable migration targets, so periodically check if we need - * to schedule, or even abort async compaction. + * suitable migration targets, so periodically check resched. */ - if (!(block_start_pfn % (SWAP_CLUSTER_MAX * pageblock_nr_pages)) - && compact_should_abort(cc)) - break; + if (!(block_start_pfn % (SWAP_CLUSTER_MAX * pageblock_nr_pages))) + compact_check_resched(cc); page = pageblock_pfn_to_page(block_start_pfn, block_end_pfn, zone); @@ -1673,11 +1659,10 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone, /* * This can potentially iterate a massively long zone with * many pageblocks unsuitable, so periodically check if we - * need to schedule, or even abort async compaction. + * need to schedule. */ - if (!(low_pfn % (SWAP_CLUSTER_MAX * pageblock_nr_pages)) - && compact_should_abort(cc)) - break; + if (!(low_pfn % (SWAP_CLUSTER_MAX * pageblock_nr_pages))) + compact_check_resched(cc); page = pageblock_pfn_to_page(block_start_pfn, block_end_pfn, zone); From patchwork Fri Jan 4 12:50:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748337 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A9F1513B5 for ; Fri, 4 Jan 2019 12:53:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 95E9C2844C for ; Fri, 4 Jan 2019 12:53:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 89F9228471; Fri, 4 Jan 2019 12:53:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EB1962844C for ; Fri, 4 Jan 2019 12:53:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 162ED8E00E5; Fri, 4 Jan 2019 07:53:38 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0EA718E00AE; Fri, 4 Jan 2019 07:53:38 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F1D378E00E5; Fri, 4 Jan 2019 07:53:37 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id 9465D8E00AE for ; Fri, 4 Jan 2019 07:53:37 -0500 (EST) Received: by mail-ed1-f71.google.com with SMTP id z10so35109614edz.15 for ; Fri, 04 Jan 2019 04:53:37 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=i8TN/NXWm8K+VVHHC6gi+xYMTRXd4mPWvACQisViumU=; b=F5BpFXBmfAF6/AgRYxdjIINTXE9jd+cfgxJG9yxLcUbdNU3hDAG3OzoaPj3PAMbIwi MA7hbv2/ojbuY6JMFWgLmBnmifKJvLgNu7YXQPcK9AsMiJk7pwwaIEk4herIvXtXxfqd qdBlFGwZ4dQuAT/LC3YQp9XwPPo/pFJneX4D36YAz8zHsKJI+igMKnWQ5ZMc5ayzOQLV UiK7dbDTPVfLyldxiKwkoBbcQpmw7tMUE6nANGJ9Xi2WX+4VvZcMXC0a3SHzNE3Nr2Jj 52GGwfSVZ1pffW4e+CJygK2P+nu7jWRZp+50A4w12WwMNO4jwY8he2VS5fcIJSRgIFtI GCiQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.193 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWYoCYPAplK5dTD5Kuo/8Rf7YzqwFx1Ne6LKuRws9xpkiS8+5c/Y xyztN2Ej0LmHXQUPGCfUA8mkM2nFZl+mtCKpigPT8JbQBWludPOYTtH0zupP6RqLQRZjwr7Mnx7 w7yf5qNH6TOpj1QS+IerZ/0Hr20EKLGD21YVJSB8Kn0UVm9GtFXRecFuiSjWFptGIUw== X-Received: by 2002:a50:9315:: with SMTP id m21mr45578790eda.58.1546606417090; Fri, 04 Jan 2019 04:53:37 -0800 (PST) X-Google-Smtp-Source: AFSGD/XZTlPxVrSklqVr229yJkFlGsc1eQy1q2BToMoqgZAi6Bju/p+BvSyBiKeXVPqIgCr0gCrW X-Received: by 2002:a50:9315:: with SMTP id m21mr45578755eda.58.1546606416210; Fri, 04 Jan 2019 04:53:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606416; cv=none; d=google.com; s=arc-20160816; b=mcgAA0f/Ly8iAML6A/vYB9KmZ5wWBGYxvgcgCDBEubYTi15qWTLRVqPE28gcyr/72/ r6xJXPzZ/sHQPZicl/mX+0YxcHN7nweFbGexWHVHwXOmXEwVUBBSWj8WfhPvermh1nBN 892vKdu54uGw8RZ982JELyFj5bthSeoph6c9dsodditl6kAZvsm4dFR5T9odJFehf4q5 rb47IA6VWJVUbfRtyERMjFfRSwNmJwSAq+zj/0Gappq0bnk6kaZ3MUpWJuPLe8Glxo9Q rO57cjlE2EmbXyPnPhTqpLy99jaaLeRZc+lY1g/cRN2lA4DB057Q8U0WRnwF+igLA0Il t7Cw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=i8TN/NXWm8K+VVHHC6gi+xYMTRXd4mPWvACQisViumU=; b=XwuIUmnL2mFvivtFV+tIJ+r1228qU8LzM6tP0SGZ+Iush8XP2prZt7H+OgqmbfQQ1R CnFtD92HAQyoaXM+7lEho5YKO9mrNJvSm59G7lYCjtAp7zKdTpTqHIeDAsiQjeVFjdt/ d/vyb5kVug4F9VB2opu4cu01rLI5CSv7OKpvQPOtBM+qeYi5XAR2xBbelJx/SSO92sEa JJZI+cAztFIqUdwX2u+gudsp96pd1fI7M0EwhAsCvgpTIvQcuibr3FTJWKSwzzq2NRLj 9DtBHMlHAuMKafIHZ9YbBIOZ2uDL7Ro72P72URhO+VXcAEL4AakhlJbJbuONkjHKO90B 1W4A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.193 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp25.blacknight.com (outbound-smtp25.blacknight.com. [81.17.249.193]) by mx.google.com with ESMTPS id i46si3118602eda.288.2019.01.04.04.53.36 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:53:36 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.193 as permitted sender) client-ip=81.17.249.193; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 81.17.249.193 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp25.blacknight.com (Postfix) with ESMTPS id D7616B87A3 for ; Fri, 4 Jan 2019 12:53:35 +0000 (GMT) Received: (qmail 8538 invoked from network); 4 Jan 2019 12:53:35 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:53:35 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 19/25] mm, compaction: Do not consider a need to reschedule as contention Date: Fri, 4 Jan 2019 12:50:05 +0000 Message-Id: <20190104125011.16071-20-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Scanning on large machines can take a considerable length of time and eventually need to be rescheduled. This is treated as an abort event but that's not appropriate as the attempt is likely to be retried after making numerous checks and taking another cycle through the page allocator. This patch will check the need to reschedule if necessary but continue the scanning. The main benefit is reduced scanning when compaction is taking a long time or the machine is over-saturated. It also avoids an unnecessary exit of compaction that ends up being retried by the page allocator in the outer loop. 4.20.0 4.20.0 synccached-v2r15 noresched-v2r15 Amean fault-both-3 2655.55 ( 0.00%) 2736.50 ( -3.05%) Amean fault-both-5 4580.67 ( 0.00%) 4133.70 ( 9.76%) Amean fault-both-7 5740.50 ( 0.00%) 5738.61 ( 0.03%) Amean fault-both-12 9237.55 ( 0.00%) 9392.82 ( -1.68%) Amean fault-both-18 12899.51 ( 0.00%) 13257.15 ( -2.77%) Amean fault-both-24 16342.47 ( 0.00%) 16859.44 ( -3.16%) Amean fault-both-30 20394.26 ( 0.00%) 16249.30 * 20.32%* Amean fault-both-32 17450.76 ( 0.00%) 14904.71 * 14.59%* Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 1a41a2dbff24..75eb0d40d4d7 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -398,19 +398,11 @@ static bool compact_lock_irqsave(spinlock_t *lock, unsigned long *flags, return true; } -/* - * Aside from avoiding lock contention, compaction also periodically checks - * need_resched() and records async compaction as contended if necessary. - */ +/* Avoid soft-lockups due to long scan times */ static inline void compact_check_resched(struct compact_control *cc) { - /* async compaction aborts if contended */ - if (need_resched()) { - if (cc->mode == MIGRATE_ASYNC) - cc->contended = true; - + if (need_resched()) cond_resched(); - } } /* From patchwork Fri Jan 4 12:50:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748339 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1C55E14E5 for ; Fri, 4 Jan 2019 12:53:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 09C852810E for ; Fri, 4 Jan 2019 12:53:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EE5852836F; Fri, 4 Jan 2019 12:53:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 62AB02810E for ; Fri, 4 Jan 2019 12:53:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 77B828E00E6; Fri, 4 Jan 2019 07:53:48 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 72B258E00AE; Fri, 4 Jan 2019 07:53:48 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 61ADA8E00E6; Fri, 4 Jan 2019 07:53:48 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 057328E00AE for ; Fri, 4 Jan 2019 07:53:48 -0500 (EST) Received: by mail-ed1-f72.google.com with SMTP id b3so35181738edi.0 for ; Fri, 04 Jan 2019 04:53:47 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=qvPONPnRj0V1IpypwsEizwqGPfeoFo2DW9REbulI7jQ=; b=SHwLHK5MAEmuJEAJytR2uQD5maUZDNWuNsM8VT6wR+r5IPaArxUz24fsGTM+Ec2b4t PZe700cVqi5LYRTT5f8Vk9GdcLpSu9FD8Y43oFelcmtm4x76n1pwqXkyNKgvPu/iMKyh zkkOASDb+8+FfRI/QEW0i6Uk34LlqXL9rz1P+MK0mjcBD4/jKtaXgcXRAMwKk2MJ6pVM hhVjRmT2pDo2+4m6fyCc62XVfalZP6BNZgI+0raN+rCiaBj7mfQrdhNYpLob9AdQGAwd Ta1c3heXAgzlTE4waRLJ0ZlYNsf9swO5WbJUtTa2pxh62W6tJxH7iLMvT7liAYbwgmYW Ntrg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWZYV2yf0hl3pb8v65zH637bpyJkaOb8z+SgcnFboSco9KJ//hqO kVAuEm7klMRVjwWuUcmMKP13qHW6maSeeBpXgr5o6XIzNZmux7DavwMpCsrEO0tK3KHIQo5Sil+ qhz6O42E4EG8XqcI9bTWdX97WIDJ+N4KX0d30Sl9qGKDfs/DvrMm9JURb+3rXD11rFg== X-Received: by 2002:a50:b902:: with SMTP id m2mr44510589ede.108.1546606427503; Fri, 04 Jan 2019 04:53:47 -0800 (PST) X-Google-Smtp-Source: AFSGD/XYnlOnU4oDRSPptjj/+qXI5VacQO9ua6pW1lcXJznrey22hngZiBPcU2WsuMWQDWm6NVeR X-Received: by 2002:a50:b902:: with SMTP id m2mr44510547ede.108.1546606426448; Fri, 04 Jan 2019 04:53:46 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606426; cv=none; d=google.com; s=arc-20160816; b=rKhLkO8DfAr/q3Th70MtfLwYrbZqE4M65jKn//bNo0VwY1V58mjuhOiVomTiZl2Kll XQgx9zh1KVyFeuly8YtuIwGflWzLqSzMWjepD5RmO6NooTBNNbxPE+eo4Q6nnmbHM9jB s6qbAgF2EaEgJD3u3fOfm9hExtrT9QpeEuInEV7bpXSZ6Spgx2mp0lgQOyN1sWNMaexY TFByYi9RXExzIgb7AnyxEVoBE1AXBvqk/uuU83blM7jj+hXMkZN95AbGkSXWmH93NwI/ F1eT/cLnBx7IFYYlXDcrxNxZGrDwlPUIf3oPvFgD/XdYk/gE1/o5Mt3CxOT8tlDliAFY r9rw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=qvPONPnRj0V1IpypwsEizwqGPfeoFo2DW9REbulI7jQ=; b=Q/ji2aOdY4XFe3JK/60TLxpD1jKoM+oe9ivQEWg+r3T6KdcB+BFC4xIqIaIdvGp7pc 8zJIhEC+aFB6C//UjwNglqQbyejtngsfpic03hrXGyopVmxPVI2TOwe7ebqmwhi1MH0e ZxtFyrdBxu28rjUKnEkxnF0eVL7HoLOdvBDLpomIg+/wOktH/vfNekRWPwT/P5o+L5KB IpX0KYiDSgaV3AxeNZSvKzolfvdtchOJrcBfg/+4rz+YK2XfpRBrat8x3xNkXAYU8baf jM2c+G7upgTFPEUJA6E1LsTBdnyvQTcU0QjyfCnzccqegY2R/h890P9dKGYpb7YqY7/D 4osA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp10.blacknight.com (outbound-smtp10.blacknight.com. [46.22.139.15]) by mx.google.com with ESMTPS id b56si4122501eda.336.2019.01.04.04.53.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:53:46 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) client-ip=46.22.139.15; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp10.blacknight.com (Postfix) with ESMTPS id 17B421C18D7 for ; Fri, 4 Jan 2019 12:53:46 +0000 (GMT) Received: (qmail 9216 invoked from network); 4 Jan 2019 12:53:46 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:53:45 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 20/25] mm, compaction: Reduce unnecessary skipping of migration target scanner Date: Fri, 4 Jan 2019 12:50:06 +0000 Message-Id: <20190104125011.16071-21-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The fast isolation of pages can move the scanner faster than is necessary depending on the contents of the free list. This patch will only allow the fast isolation to initialise the scanner and advance it slowly. The primary means of moving the scanner forward is via the linear scanner to reduce the likelihood the migration source/target scanners meet prematurely triggering a rescan. 4.20.0 4.20.0 noresched-v2r15 slowfree-v2r15 Amean fault-both-1 0.00 ( 0.00%) 0.00 * 0.00%* Amean fault-both-3 2736.50 ( 0.00%) 2512.53 ( 8.18%) Amean fault-both-5 4133.70 ( 0.00%) 4159.43 ( -0.62%) Amean fault-both-7 5738.61 ( 0.00%) 5950.15 ( -3.69%) Amean fault-both-12 9392.82 ( 0.00%) 8674.38 ( 7.65%) Amean fault-both-18 13257.15 ( 0.00%) 12850.79 ( 3.07%) Amean fault-both-24 16859.44 ( 0.00%) 17242.86 ( -2.27%) Amean fault-both-30 16249.30 ( 0.00%) 19404.18 * -19.42%* Amean fault-both-32 14904.71 ( 0.00%) 16200.79 ( -8.70%) The impact to latency, success rates and scan rates is marginal but avoiding unnecessary restarts is important. It helps later patches that are more careful about how pageblocks are treated as earlier iterations of those patches hit corner cases where the restarts were punishing and very visible. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 27 ++++++++++----------------- 1 file changed, 10 insertions(+), 17 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 75eb0d40d4d7..6c5552c6d8f9 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -324,10 +324,9 @@ static void update_cached_migrate(struct compact_control *cc, unsigned long pfn) * future. The information is later cleared by __reset_isolation_suitable(). */ static void update_pageblock_skip(struct compact_control *cc, - struct page *page, unsigned long nr_isolated) + struct page *page, unsigned long pfn) { struct zone *zone = cc->zone; - unsigned long pfn; if (cc->no_set_skip_hint) return; @@ -335,13 +334,8 @@ static void update_pageblock_skip(struct compact_control *cc, if (!page) return; - if (nr_isolated) - return; - set_pageblock_skip(page); - pfn = page_to_pfn(page); - /* Update where async and sync compaction should restart */ if (pfn < zone->compact_cached_free_pfn) zone->compact_cached_free_pfn = pfn; @@ -359,7 +353,7 @@ static inline bool pageblock_skip_persistent(struct page *page) } static inline void update_pageblock_skip(struct compact_control *cc, - struct page *page, unsigned long nr_isolated) + struct page *page, unsigned long pfn) { } @@ -450,7 +444,7 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, bool strict) { int nr_scanned = 0, total_isolated = 0; - struct page *cursor, *valid_page = NULL; + struct page *cursor; unsigned long flags = 0; bool locked = false; unsigned long blockpfn = *start_pfn; @@ -477,9 +471,6 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, if (!pfn_valid_within(blockpfn)) goto isolate_fail; - if (!valid_page) - valid_page = page; - /* * For compound pages such as THP and hugetlbfs, we can save * potentially a lot of iterations if we skip them at once. @@ -576,10 +567,6 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, if (strict && blockpfn < end_pfn) total_isolated = 0; - /* Update the pageblock-skip if the whole pageblock was scanned */ - if (blockpfn == end_pfn) - update_pageblock_skip(cc, valid_page, total_isolated); - cc->total_free_scanned += nr_scanned; if (total_isolated) count_compact_events(COMPACTISOLATED, total_isolated); @@ -1295,8 +1282,10 @@ fast_isolate_freepages(struct compact_control *cc) } } - if (highest && highest > cc->zone->compact_cached_free_pfn) + if (highest && highest >= cc->zone->compact_cached_free_pfn) { + highest -= pageblock_nr_pages; cc->zone->compact_cached_free_pfn = highest; + } cc->total_free_scanned += nr_scanned; if (!page) @@ -1376,6 +1365,10 @@ static void isolate_freepages(struct compact_control *cc) isolate_freepages_block(cc, &isolate_start_pfn, block_end_pfn, freelist, false); + /* Update the skip hint if the full pageblock was scanned */ + if (isolate_start_pfn == block_end_pfn) + update_pageblock_skip(cc, page, block_start_pfn); + /* Are enough freepages isolated? */ if (cc->nr_freepages >= cc->nr_migratepages) { if (isolate_start_pfn >= block_end_pfn) { From patchwork Fri Jan 4 12:50:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748341 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3F22814E5 for ; Fri, 4 Jan 2019 12:54:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2E3A92810E for ; Fri, 4 Jan 2019 12:54:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1E7A62836F; Fri, 4 Jan 2019 12:54:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 909E62810E for ; Fri, 4 Jan 2019 12:53:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9ED1D8E00E7; Fri, 4 Jan 2019 07:53:58 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 99BEA8E00AE; Fri, 4 Jan 2019 07:53:58 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 88B718E00E7; Fri, 4 Jan 2019 07:53:58 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id 2B0078E00AE for ; Fri, 4 Jan 2019 07:53:58 -0500 (EST) Received: by mail-ed1-f69.google.com with SMTP id c3so35320440eda.3 for ; Fri, 04 Jan 2019 04:53:58 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=7NMy9m+YgJH0m/HLzWMXEowcaGcqNh8354CNSDiq+Cs=; b=WWfJP/uWwYSVcu/kPCr1ByHiS/ezCds/YJ0Q8Ggc1Hc80YP0FDTJux3xIwgQIkaLpC m750wxjiTfHEF6hwiij6+bN7JlTdMa5dX8cRS3Xl+aY/E58JkkUuVc9X74Clzux5GXJU zEyVJTmsQIqH0135lFFoC7l5llyQnkHBW1vlr2KksC1deOzcKIlSx4PP3xXFsPrYFa9m H5X/K+r6oLDDVmJnPV+ducHi1nYNIE1N2dNhaP6tTMROMqVi3aVYiTfWeYra53uiEbWB 3pjPDjcY6shYlP2QNXuR30QHZXJrOi5jW6sRBv5aWTMJ+bjj3xW8SKqu13NuYNz/Ybe1 4T5A== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWYYrlpUzIln/FA1akLq14g61LLQYh375E/0ZZfpUObOAuEaB69Z Iy91xb2LkRl3PIy2KZ9tWgL9vXtwjSOr9AkTfvT7WiRmNXJ1kJ3kvqv8rbEsbYuDIdks8TPpP4A 8JtpX80hOjzLKHusOkxPuF7jMVhcS6rT6ucnEXZHbf0Dh+ptzhSnkD4CStjAZtNrqKA== X-Received: by 2002:aa7:d29a:: with SMTP id w26mr46997385edq.30.1546606437624; Fri, 04 Jan 2019 04:53:57 -0800 (PST) X-Google-Smtp-Source: AFSGD/VJ7ZWgNWpR20YpWk8kqhedBmE7scs6rp+N3SHfsDvJxoMOALeV7ODANrSrwoO4OJAp52tY X-Received: by 2002:aa7:d29a:: with SMTP id w26mr46997347edq.30.1546606436655; Fri, 04 Jan 2019 04:53:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606436; cv=none; d=google.com; s=arc-20160816; b=g9kGyyMsO98zcs8kG8dEV9wqQxdF0FW3lxDNvEM+nZcZbkjILwHmYzGHksutMQF79+ 0FTUwU5obU+Kz+5eYPSoaFqT4GIRVxhM+SHpb2RP/gvoAgyk3ZXuEDWPQ9RyVfuocFNE bOfrZD8HgD/WMa5PxNvu2Akkpwlux+xcMiv1WbF2ZOkv7M2QfqnkJe9HKB7G/J1l7flq G8XCP2WxywfJbQA8OPmyIQbPHpgc7jigZJQ1MoL7MvVifdKidSEgW1FoQJKRB9SDUnjD DPIOTca0Rnf+JDht8mAIyZFTb+ZGLNWeltI+6Xi44BhD0kgndybiq4rQ7EE5SgGb36Aw 19Hw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=7NMy9m+YgJH0m/HLzWMXEowcaGcqNh8354CNSDiq+Cs=; b=u82VPZSgZhpqUGAOXr5b2C+bNhvMjhSDwtOFP8QovgbumqhZke1RvQDOunLWmQPkjT HXJRf7f1cQrR0ISWUSdPunMxYEy2TO5Q/k0/8EzbBY1lC0oK3F7CoOV6k2FefS0cNQT+ O8mRxQmp+fQpSrcppP9vSSssx0T5iQjmVWphQNeK/jTUadaXi9O99/ilUd545DZyVZF0 xVUB4B+c075NX5QVfRjfADe/znK+iNiIYTQiGkqO1Md6ljyybX3ikSWiac0GrbF8vaTc W1NT+5URsqqpXUU3gfgkCmCiYNVKA0m4CVNpAYpm9DG/2vvRmUjCDez1a69AvNt7wLNe Rq9A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp08.blacknight.com (outbound-smtp08.blacknight.com. [46.22.139.13]) by mx.google.com with ESMTPS id l16-v6si1298091ejq.174.2019.01.04.04.53.56 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:53:56 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) client-ip=46.22.139.13; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp08.blacknight.com (Postfix) with ESMTPS id 499061C1BE5 for ; Fri, 4 Jan 2019 12:53:56 +0000 (GMT) Received: (qmail 10079 invoked from network); 4 Jan 2019 12:53:56 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:53:56 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 21/25] mm, compaction: Round-robin the order while searching the free lists for a target Date: Fri, 4 Jan 2019 12:50:07 +0000 Message-Id: <20190104125011.16071-22-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP As compaction proceeds and creates high-order blocks, the free list search gets less efficient as the larger blocks are used as compaction targets. Eventually, the larger blocks will be behind the migration scanner for partially migrated pageblocks and the search fails. This patch round-robins what orders are searched so that larger blocks can be ignored and find smaller blocks that can be used as migration targets. The overall impact was small on 1-socket but it avoids corner cases where the migration/free scanners meet prematurely or situations where many of the pageblocks encountered by the free scanner are almost full instead of being properly packed. Previous testing had indicated that without this patch there were occasional large spikes in the free scanner without this patch. By co-incidence, the 2-socket results showed a 54% reduction in the free scanner but will not be universally true. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 33 ++++++++++++++++++++++++++++++--- mm/internal.h | 3 ++- 2 files changed, 32 insertions(+), 4 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 6c5552c6d8f9..652e249168b1 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1154,6 +1154,24 @@ fast_isolate_around(struct compact_control *cc, unsigned long pfn, unsigned long set_pageblock_skip(page); } +/* Search orders in round-robin fashion */ +static int next_search_order(struct compact_control *cc, int order) +{ + order--; + if (order < 0) + order = cc->order - 1; + + /* Search wrapped around? */ + if (order == cc->search_order) { + cc->search_order--; + if (cc->search_order < 0) + cc->search_order = cc->order - 1; + return -1; + } + + return order; +} + static unsigned long fast_isolate_freepages(struct compact_control *cc) { @@ -1186,9 +1204,15 @@ fast_isolate_freepages(struct compact_control *cc) if (WARN_ON_ONCE(min_pfn > low_pfn)) low_pfn = min_pfn; - for (order = cc->order - 1; - order >= 0 && !page; - order--) { + /* + * Search starts from the last successful isolation order or the next + * order to search after a previous failure + */ + cc->search_order = min_t(unsigned int, cc->order - 1, cc->search_order); + + for (order = cc->search_order; + !page && order >= 0; + order = next_search_order(cc, order)) { struct free_area *area = &cc->zone->free_area[order]; struct list_head *freelist; struct page *freepage; @@ -1211,6 +1235,7 @@ fast_isolate_freepages(struct compact_control *cc) if (pfn >= low_pfn) { cc->fast_search_fail = 0; + cc->search_order = order; page = freepage; break; } @@ -2146,6 +2171,7 @@ static enum compact_result compact_zone_order(struct zone *zone, int order, .total_migrate_scanned = 0, .total_free_scanned = 0, .order = order, + .search_order = order, .gfp_mask = gfp_mask, .zone = zone, .mode = (prio == COMPACT_PRIO_ASYNC) ? @@ -2385,6 +2411,7 @@ static void kcompactd_do_work(pg_data_t *pgdat) struct zone *zone; struct compact_control cc = { .order = pgdat->kcompactd_max_order, + .search_order = pgdat->kcompactd_max_order, .total_migrate_scanned = 0, .total_free_scanned = 0, .classzone_idx = pgdat->kcompactd_classzone_idx, diff --git a/mm/internal.h b/mm/internal.h index e5ca2a10b8ad..d028abd8a8f3 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -191,7 +191,8 @@ struct compact_control { struct zone *zone; unsigned long total_migrate_scanned; unsigned long total_free_scanned; - unsigned int fast_search_fail; /* failures to use free list searches */ + unsigned short fast_search_fail;/* failures to use free list searches */ + unsigned short search_order; /* order to start a fast search at */ const gfp_t gfp_mask; /* gfp mask of a direct compactor */ int order; /* order a direct compactor needs */ int migratetype; /* migratetype of direct compactor */ From patchwork Fri Jan 4 12:50:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748343 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8346213B5 for ; Fri, 4 Jan 2019 12:54:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 742EF2810E for ; Fri, 4 Jan 2019 12:54:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 682622836F; Fri, 4 Jan 2019 12:54:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CEAA42810E for ; Fri, 4 Jan 2019 12:54:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C5E748E00E8; Fri, 4 Jan 2019 07:54:08 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C333C8E00AE; Fri, 4 Jan 2019 07:54:08 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD9098E00E8; Fri, 4 Jan 2019 07:54:08 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id 4EFEB8E00AE for ; Fri, 4 Jan 2019 07:54:08 -0500 (EST) Received: by mail-ed1-f71.google.com with SMTP id c34so34720212edb.8 for ; Fri, 04 Jan 2019 04:54:08 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=nBi3kpmLXtB+/L42OygzYTjrh03AFHn26ukynBWzS6c=; b=Pj8zjsLxfH+eu9NpEA16wN5Fg7E8Bu2JV3wjbd7qNA311GRyey425bvn8b31CEQqze N9posbLxsvJWAYH56JyEmn9H0c7Nr/PHflMfvhzjqWEYDcWiohoIbUg25FqeZ09LXdJt peqHi558WxSsEXXijaxqDOSJ5V6na5Zx+RqDA+IJAPL4VOrTgd0oYI58o5M5WQntSFTw 6qwReX+EsDu7WZaL34Wpz92hD3irLsSLXnM1wknf2B8BobSJdmsGriUZdAzDtrId3v1O WRg8yvFAEI5On1ZvM656XKzpJjxF5F2WOKQhUMGVud5/BFgGRxnvOJEcbvFSnWpBOdGS Q0rQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWaxpgVQVFvBhIgfhQA9gDio57bcbdf+CV4QRmpXY5bM9r+i/OyR 6+634V2tHhKmWU19S5ZTdXCaf9drdZeW3tJoqebrBkNVrBUdIjoRisFG1uRFFQJ49foc64ZTjf4 yinnnUeVVimvY4GVxwDw27im4BAiiLCTbeiaFkTmSh8936lzRsDQhU4s5+fzR+YDB8A== X-Received: by 2002:a17:906:753:: with SMTP id z19-v6mr38800139ejb.175.1546606447804; Fri, 04 Jan 2019 04:54:07 -0800 (PST) X-Google-Smtp-Source: AFSGD/WWeqYmykrh1Rwxnoh61OwFm/OESBGfmBt0clagtE0lZUqR7Ho6NE+OYx58mLtwkSCrXlno X-Received: by 2002:a17:906:753:: with SMTP id z19-v6mr38800103ejb.175.1546606446817; Fri, 04 Jan 2019 04:54:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606446; cv=none; d=google.com; s=arc-20160816; b=lw/xE6Mful1UGo4naJ03VOeIQ/iXipMvvHl38nYeQ3M2+qfRURg0FdQa4D/SUuCViX 8VMmDKePvjA2xKw6hnTCKufp2BbvMxheXNzoupz6+QeP+jf/DrGb2Fj1DU47wdKrg5DC SbQBBYOe9+zEEcQzM2N/wgtN1nhRImDSIlRp39kpGnDhqbTjZM/+pxEhhShg5S6E8Z2y QKQ2v79CacWuwj/3Jcne6vNiAWEfamLchsgQ0KpLNtWzmQwO6hGZBVN9rE9GdA4bqAhb w/5vaqc5IvbIniWJhbSPdpnT4Rp+AxIMX3VjG1vb7WLJJzeC+k++uvtkwAGXgd0GHIx4 IFSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=nBi3kpmLXtB+/L42OygzYTjrh03AFHn26ukynBWzS6c=; b=w9bKwaBelO0v+9BjyKsyTj43J3snIPCIWFuAwWFFBCqgDDrMcgpSu7LxYiHUl4bzfo 1lQvByRIZSshBswTjjB6GLgU1rU54tm5mw51vPG1wYnEwnMhge29hscg+9gD/mOKn95t iqN1Fkja4Qento/bmDnhOVB5tqPhvZogi4smmX6WlAM1hm/LKij/sQNL5Y1b1Jnfa8CN KV7RqLfhUct87VHnqwWxQpdX/faSbbjeNnxDw+EspYSjmmgFvV6I2i4BaFSGg86LlFmA 42ytYhtBmBBuwdF34FINLtlmTYgKYIIpxYhtZCEkIi+b0UNIa6SNYGPPOAlvLH20dSBS NIHA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp13.blacknight.com (outbound-smtp13.blacknight.com. [46.22.139.230]) by mx.google.com with ESMTPS id p18si6154495edi.197.2019.01.04.04.54.06 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:54:06 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) client-ip=46.22.139.230; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (unknown [81.17.254.16]) by outbound-smtp13.blacknight.com (Postfix) with ESMTPS id 7509D1C1CE7 for ; Fri, 4 Jan 2019 12:54:06 +0000 (GMT) Received: (qmail 10824 invoked from network); 4 Jan 2019 12:54:06 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:54:06 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 22/25] mm, compaction: Sample pageblocks for free pages Date: Fri, 4 Jan 2019 12:50:08 +0000 Message-Id: <20190104125011.16071-23-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Once fast searching finishes, there is a possibility that the linear scanner is scanning full blocks found by the fast scanner earlier. This patch uses an adaptive stride to sample pageblocks for free pages. The more consecutive full pageblocks encountered, the larger the stride until a pageblock with free pages is found. The scanners might meet slightly sooner but it is an acceptable risk given that the search of the free lists may still encounter the pages and adjust the cached PFN of the free scanner accordingly. In terms of latency and success rates, the impact is not obvious but the free scan rate is reduced by 87% on a 1-socket machine and 92% on a 2-socket machine. It's also the first time in the series where the number of pages scanned by the migration scanner is greater than the free scanner due to the increased search efficiency. Signed-off-by: Mel Gorman --- mm/compaction.c | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 652e249168b1..cc532e81a7b7 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -441,6 +441,7 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, unsigned long *start_pfn, unsigned long end_pfn, struct list_head *freelist, + unsigned int stride, bool strict) { int nr_scanned = 0, total_isolated = 0; @@ -450,10 +451,14 @@ static unsigned long isolate_freepages_block(struct compact_control *cc, unsigned long blockpfn = *start_pfn; unsigned int order; + /* Strict mode is for isolation, speed is secondary */ + if (strict) + stride = 1; + cursor = pfn_to_page(blockpfn); /* Isolate free pages. */ - for (; blockpfn < end_pfn; blockpfn++, cursor++) { + for (; blockpfn < end_pfn; blockpfn += stride, cursor += stride) { int isolated; struct page *page = cursor; @@ -624,7 +629,7 @@ isolate_freepages_range(struct compact_control *cc, break; isolated = isolate_freepages_block(cc, &isolate_start_pfn, - block_end_pfn, &freelist, true); + block_end_pfn, &freelist, 0, true); /* * In strict mode, isolate_freepages_block() returns 0 if @@ -1139,7 +1144,7 @@ fast_isolate_around(struct compact_control *cc, unsigned long pfn, unsigned long /* Scan before */ if (start_pfn != pfn) { - isolate_freepages_block(cc, &start_pfn, pfn, &cc->freepages, false); + isolate_freepages_block(cc, &start_pfn, pfn, &cc->freepages, 1, false); if (cc->nr_freepages >= cc->nr_migratepages) return; } @@ -1147,7 +1152,7 @@ fast_isolate_around(struct compact_control *cc, unsigned long pfn, unsigned long /* Scan after */ start_pfn = pfn + nr_isolated; if (start_pfn != end_pfn) - isolate_freepages_block(cc, &start_pfn, end_pfn, &cc->freepages, false); + isolate_freepages_block(cc, &start_pfn, end_pfn, &cc->freepages, 1, false); /* Skip this pageblock in the future as it's full or nearly full */ if (cc->nr_freepages < cc->nr_migratepages) @@ -1333,7 +1338,9 @@ static void isolate_freepages(struct compact_control *cc) unsigned long isolate_start_pfn; /* exact pfn we start at */ unsigned long block_end_pfn; /* end of current pageblock */ unsigned long low_pfn; /* lowest pfn scanner is able to scan */ + unsigned long nr_isolated; struct list_head *freelist = &cc->freepages; + unsigned int stride; /* Try a small search of the free lists for a candidate */ isolate_start_pfn = fast_isolate_freepages(cc); @@ -1356,6 +1363,7 @@ static void isolate_freepages(struct compact_control *cc) block_end_pfn = min(block_start_pfn + pageblock_nr_pages, zone_end_pfn(zone)); low_pfn = pageblock_end_pfn(cc->migrate_pfn); + stride = cc->mode == MIGRATE_ASYNC ? COMPACT_CLUSTER_MAX : 1; /* * Isolate free pages until enough are available to migrate the @@ -1387,8 +1395,8 @@ static void isolate_freepages(struct compact_control *cc) continue; /* Found a block suitable for isolating free pages from. */ - isolate_freepages_block(cc, &isolate_start_pfn, block_end_pfn, - freelist, false); + nr_isolated = isolate_freepages_block(cc, &isolate_start_pfn, + block_end_pfn, freelist, stride, false); /* Update the skip hint if the full pageblock was scanned */ if (isolate_start_pfn == block_end_pfn) @@ -1412,6 +1420,13 @@ static void isolate_freepages(struct compact_control *cc) */ break; } + + /* Adjust stride depending on isolation */ + if (nr_isolated) { + stride = 1; + continue; + } + stride = min_t(unsigned int, COMPACT_CLUSTER_MAX, stride << 1); } /* From patchwork Fri Jan 4 12:50:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748345 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D56C913B5 for ; Fri, 4 Jan 2019 12:54:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C54B628450 for ; Fri, 4 Jan 2019 12:54:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B910D2847E; Fri, 4 Jan 2019 12:54:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0FE9428450 for ; Fri, 4 Jan 2019 12:54:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0781D8E00E9; Fri, 4 Jan 2019 07:54:19 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 04CBD8E00AE; Fri, 4 Jan 2019 07:54:19 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E333D8E00E9; Fri, 4 Jan 2019 07:54:18 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by kanga.kvack.org (Postfix) with ESMTP id 87EC78E00AE for ; Fri, 4 Jan 2019 07:54:18 -0500 (EST) Received: by mail-ed1-f69.google.com with SMTP id 39so35263097edq.13 for ; Fri, 04 Jan 2019 04:54:18 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=O9RGkFe8csXG+P7/vbPud3vVSE6KR+MXgQLqDX1fNis=; b=Rky5L6Y7Egxndta7mo6YV10IcXz6LHP3rpB4o7TVIqFHYjSDtRZPtj75guvE65vSvY 1e5uLej94ZiBsvX4k2Q+M/5AeGf1x958y830V32Jyy0RE911f0F09kE87d1jRBOilJZh AdtQYPUuEVo9Pj16cCukKDPOLk64Hc+nSJJ1nTxrjEBrf4bVSEG1Mzi9EV23CiPkbkRl wk64em/oeiKelbXbObBUhD7Lce9tMSSFFaNw2xIbJ84G5bbMAYMAOJi+/5aO/2zNOqTT HcGgcQ1t5dVLUWdSVYSH1tuAGVXeH/5Aj3gArOnF/h9haH+7NYhR187dFB3sDyoKF905 W6pA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWZU4bMG5tmGXnyrcEShJro896ezbQ0mYb9qCsFFJKwM+JKl2Kue uvyT8qxTs7EYe3tKqx9dRPGQAkrPgTNowGhYj7emO0a3/J986cGWjPwyRyEP33dUMSYWhQiumEM 7TLWuHjHzq1iSLRT2iNsSXajLXzRgMFB68SsyIJxyGEaZN9oDlf7848xfNHGNjKggJA== X-Received: by 2002:a50:8907:: with SMTP id e7mr47257398ede.252.1546606458030; Fri, 04 Jan 2019 04:54:18 -0800 (PST) X-Google-Smtp-Source: AFSGD/WRvEyAN7EOjXpwxnWK9/2fifLqQrhPj6sziqLqWF0ms7RZkTR0vfDrmUZmSF/psmQDH/T+ X-Received: by 2002:a50:8907:: with SMTP id e7mr47257352ede.252.1546606457014; Fri, 04 Jan 2019 04:54:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606456; cv=none; d=google.com; s=arc-20160816; b=ra3qwirDNLOEQOWCidGysc4cRR/2a8Jbjwbody/eaBrDwyTQ732mkt4D/q8qOngNGk Thc8p64NeglzPjspiXGYnPfGIzUpqgWMM4VxD1BbLFOEoHrfKFTg8Pq9TcB6tqJHb/PA 7jA6q5a3DMiQR7IgdXjgy+ut8xuMtlbHv9+F+bykgMwQeTE777oFgS1SDeDzwJ+fnk4i 3leUVdLwZZ+M1uEkY+97ARHU5L/l6Zk0eEynp8E1FMP+g70Q9mPJlXvwSH9Bl0TYrC6m r6XJjieSw1dDawZeI6iAsU5EDthexu08npZzQ6f/uD6i/HZ/NeDLvv/6QEIU4z+1TGsS uqRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=O9RGkFe8csXG+P7/vbPud3vVSE6KR+MXgQLqDX1fNis=; b=ibgW3sN19Q7Y+y9MyJmXCgxbEbwkOyV3F9Iubw3aicrq0QDflRaNtMwJ/BaXW5Lq8F S9uJXgQ3uTOHZvshmA/XIFrYKieUAZqqRYhRV1ESNt2Yv0CDZNp/JfAcxFN7PJEvl4Gs Qso1IXBB6VIZx8F0rBrc/iUojmeF3sP+al/HoeOCy3aQGmu3hP79y6Em7FcTdihz//3s VSV5sI3rTj1XfgeBEc1x2LAul499yO8zB6YfT/8vB+oZFu+3FyYsRHzANEXXmE1hqXuz ctqy7MmLqe/vF4IBWunv7sK6pClJgkfXDjjufITen/JHwCi9dmB48beLP/tUlnkrqSxY +SBQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp08.blacknight.com (outbound-smtp08.blacknight.com. [46.22.139.13]) by mx.google.com with ESMTPS id r24si206942edp.187.2019.01.04.04.54.16 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:54:16 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) client-ip=46.22.139.13; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.13 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp08.blacknight.com (Postfix) with ESMTPS id A3EF61C1C34 for ; Fri, 4 Jan 2019 12:54:16 +0000 (GMT) Received: (qmail 11447 invoked from network); 4 Jan 2019 12:54:16 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:54:16 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 23/25] mm, compaction: Be selective about what pageblocks to clear skip hints Date: Fri, 4 Jan 2019 12:50:09 +0000 Message-Id: <20190104125011.16071-24-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Pageblock hints are cleared when compaction restarts or kswapd makes enough progress that it can sleep but it's over-eager in that the bit is cleared for migration sources with no LRU pages and migration targets with no free pages. As pageblock skip hint flushes are relatively rare and out-of-band with respect to kswapd, this patch makes a few more expensive checks to see if it's appropriate to even clear the bit. Every pageblock that is not cleared will avoid 512 pages being scanned unnecessarily on x86-64. The impact is variable with different workloads showing small differences in latency, success rates and scan rates. This is expected as clearing the hints is not that common but doing a small amount of work out-of-band to avoid a large amount of work in-band later is generally a good thing. Signed-off-by: Mel Gorman --- include/linux/mmzone.h | 2 + mm/compaction.c | 119 +++++++++++++++++++++++++++++++++++++++++-------- 2 files changed, 102 insertions(+), 19 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index cc4a507d7ca4..faa1e6523f49 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -480,6 +480,8 @@ struct zone { unsigned long compact_cached_free_pfn; /* pfn where async and sync compaction migration scanner should start */ unsigned long compact_cached_migrate_pfn[2]; + unsigned long compact_init_migrate_pfn; + unsigned long compact_init_free_pfn; #endif #ifdef CONFIG_COMPACTION diff --git a/mm/compaction.c b/mm/compaction.c index cc532e81a7b7..7f316e1a7275 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -231,6 +231,62 @@ static bool pageblock_skip_persistent(struct page *page) return false; } +static bool +__reset_isolation_pfn(struct zone *zone, unsigned long pfn, bool check_source, + bool check_target) +{ + struct page *page = pfn_to_online_page(pfn); + struct page *end_page; + + if (!page) + return false; + if (zone != page_zone(page)) + return false; + if (pageblock_skip_persistent(page)) + return false; + + /* + * If skip is already cleared do no further checking once the + * restart points have been set. + */ + if (check_source && check_target && !get_pageblock_skip(page)) + return true; + + /* + * If clearing skip for the target scanner, do not select a + * non-movable pageblock as the starting point. + */ + if (!check_source && check_target && + get_pageblock_migratetype(page) != MIGRATE_MOVABLE) + return false; + + /* + * Only clear the hint if a sample indicates there is either a + * free page or an LRU page in the block. One or other condition + * is necessary for the block to be a migration source/target. + */ + page = pfn_to_page(pageblock_start_pfn(pfn)); + if (zone != page_zone(page)) + return false; + end_page = page + pageblock_nr_pages; + + do { + if (check_source && PageLRU(page)) { + clear_pageblock_skip(page); + return true; + } + + if (check_target && PageBuddy(page)) { + clear_pageblock_skip(page); + return true; + } + + page += (1 << PAGE_ALLOC_COSTLY_ORDER); + } while (page < end_page); + + return false; +} + /* * This function is called to clear all cached information on pageblocks that * should be skipped for page isolation when the migrate and free page scanner @@ -238,30 +294,54 @@ static bool pageblock_skip_persistent(struct page *page) */ static void __reset_isolation_suitable(struct zone *zone) { - unsigned long start_pfn = zone->zone_start_pfn; - unsigned long end_pfn = zone_end_pfn(zone); - unsigned long pfn; + unsigned long migrate_pfn = zone->zone_start_pfn; + unsigned long free_pfn = zone_end_pfn(zone); + unsigned long reset_migrate = free_pfn; + unsigned long reset_free = migrate_pfn; + bool source_set = false; + bool free_set = false; - zone->compact_blockskip_flush = false; + if (!zone->compact_blockskip_flush) + return; - /* Walk the zone and mark every pageblock as suitable for isolation */ - for (pfn = start_pfn; pfn < end_pfn; pfn += pageblock_nr_pages) { - struct page *page; + zone->compact_blockskip_flush = false; + /* + * Walk the zone and update pageblock skip information. Source looks + * for PageLRU while target looks for PageBuddy. When the scanner + * is found, both PageBuddy and PageLRU are checked as the pageblock + * is suitable as both source and target. + */ + for (; migrate_pfn < free_pfn; migrate_pfn += pageblock_nr_pages, + free_pfn -= pageblock_nr_pages) { cond_resched(); - page = pfn_to_online_page(pfn); - if (!page) - continue; - if (zone != page_zone(page)) - continue; - if (pageblock_skip_persistent(page)) - continue; + /* Update the migrate PFN */ + if (__reset_isolation_pfn(zone, migrate_pfn, true, source_set) && + migrate_pfn < reset_migrate) { + source_set = true; + reset_migrate = migrate_pfn; + zone->compact_init_migrate_pfn = reset_migrate; + zone->compact_cached_migrate_pfn[0] = reset_migrate; + zone->compact_cached_migrate_pfn[1] = reset_migrate; + } - clear_pageblock_skip(page); + /* Update the free PFN */ + if (__reset_isolation_pfn(zone, free_pfn, free_set, true) && + free_pfn > reset_free) { + free_set = true; + reset_free = free_pfn; + zone->compact_init_free_pfn = reset_free; + zone->compact_cached_free_pfn = reset_free; + } } - reset_cached_positions(zone); + /* Leave no distance if no suitable block was reset */ + if (reset_migrate >= reset_free) { + zone->compact_cached_migrate_pfn[0] = migrate_pfn; + zone->compact_cached_migrate_pfn[1] = migrate_pfn; + zone->compact_cached_free_pfn = free_pfn; + } } void reset_isolation_suitable(pg_data_t *pgdat) @@ -1193,7 +1273,7 @@ fast_isolate_freepages(struct compact_control *cc) * If starting the scan, use a deeper search and use the highest * PFN found if a suitable one is not found. */ - if (cc->free_pfn == pageblock_start_pfn(zone_end_pfn(cc->zone) - 1)) { + if (cc->free_pfn >= cc->zone->compact_init_free_pfn) { limit = pageblock_nr_pages >> 1; scan_start = true; } @@ -1338,7 +1418,6 @@ static void isolate_freepages(struct compact_control *cc) unsigned long isolate_start_pfn; /* exact pfn we start at */ unsigned long block_end_pfn; /* end of current pageblock */ unsigned long low_pfn; /* lowest pfn scanner is able to scan */ - unsigned long nr_isolated; struct list_head *freelist = &cc->freepages; unsigned int stride; @@ -1374,6 +1453,8 @@ static void isolate_freepages(struct compact_control *cc) block_end_pfn = block_start_pfn, block_start_pfn -= pageblock_nr_pages, isolate_start_pfn = block_start_pfn) { + unsigned long nr_isolated; + /* * This can iterate a massively long zone without finding any * suitable migration targets, so periodically check resched. @@ -2020,7 +2101,7 @@ static enum compact_result compact_zone(struct compact_control *cc) cc->zone->compact_cached_migrate_pfn[1] = cc->migrate_pfn; } - if (cc->migrate_pfn == start_pfn) + if (cc->migrate_pfn <= cc->zone->compact_init_migrate_pfn) cc->whole_zone = true; } From patchwork Fri Jan 4 12:50:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748347 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B7D4313B5 for ; Fri, 4 Jan 2019 12:54:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A777A2847E for ; Fri, 4 Jan 2019 12:54:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9B34828485; Fri, 4 Jan 2019 12:54:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A63422847E for ; Fri, 4 Jan 2019 12:54:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9289B8E00EA; Fri, 4 Jan 2019 07:54:29 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8D7CC8E00AE; Fri, 4 Jan 2019 07:54:29 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7A1308E00EA; Fri, 4 Jan 2019 07:54:29 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by kanga.kvack.org (Postfix) with ESMTP id 1DEFB8E00AE for ; Fri, 4 Jan 2019 07:54:29 -0500 (EST) Received: by mail-ed1-f72.google.com with SMTP id b3so35182266edi.0 for ; Fri, 04 Jan 2019 04:54:29 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=0CM+nS0n9I31CQXaBzYLcsaGQg1JfmRFqq+n+yIuoSA=; b=fqn4aYGYn/gHdqdhZmEvikMV5eAbOE8jnCjwP4qQ2GRk3t/kU/QmpKmOs1zsn3NDhN Tfb6yvDHHNkUmhOt5TQn+ljPrPHqYjphiB8O53rdz9j99oVOmq5vnvCp8w57wbnV+h1Q sN1zLNTP6urSjOIGAGP1/LwFgZUSjeOoNOc21MANuYMYjLUsRC55497OfiguSSuws+jW zp93YuMpawMtUJf+yLJMop8qma0JTqHerBJSYcGZ71t25oR3RHL0KMJEZYEhz0u5V9Ff +JMKWJfBEaBtSFF9lbcomBJMmNX1bASWH5p/UJhbUOFqcBub5MRindKEYhqafzGxgN0k d0Lw== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWbuxvxWoliwPybfC6VA2AXn/x1i0fWhfkRdqDOWTK3XRusBaoDE 7fOKgiUfEA9XENb/RYH7dauK7k+BQDyScILoxyL1Z7uNj7wDSeP2ARXiA2vA7tuwfNuL/VZYdn4 +y0ieLn5sr4G9XdEVIJl6t8HqeqidQyQJe67Qfvrk+ugl7OCtsk6tN/+qE8MrdgU9AA== X-Received: by 2002:a50:fe15:: with SMTP id f21mr44247025edt.116.1546606468582; Fri, 04 Jan 2019 04:54:28 -0800 (PST) X-Google-Smtp-Source: AFSGD/XURqev+jLv7xS1CH9eJM09ZXItVZ4HS5wQwMNM85kK3htZi2kY5P47V6YJXIFWzTBVQ/OA X-Received: by 2002:a50:fe15:: with SMTP id f21mr44246979edt.116.1546606467232; Fri, 04 Jan 2019 04:54:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606467; cv=none; d=google.com; s=arc-20160816; b=V9TAC16XGcj0MQ3npWVLyleAv7rUeMLul1CaWvG0nifUJmtBngrBxrg/5Db3BpJW0v lnDGAn5ZJoJ4BAVqbIKyuVsKlcBvw6DZ5L0MVJ8Z76Rh80O1GRRWLkS/zzFG8AhhZLpB TNFE9poR8/DmGgFdYz9ZZ79H9+LseIConKhzr8if7Crr8o2F1ztNUMM2tPqU/lki8KCc zUqI6NclNbys0CggC/UEqGkDALZ3uxSSqJzsIL8+vILq/CFwHDE4phhN4BMI0Hbz+jDF eaOf8aSUeVl/fwZA3HWdAqoKbvZdeKyrgbPkJCb7K70QEpjhoEvn09f1hgLBpOnokRa3 GjQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=0CM+nS0n9I31CQXaBzYLcsaGQg1JfmRFqq+n+yIuoSA=; b=KoVGUqAfQ+nda8IsoTGv/BAD3Utq98zjhNusxu/hmEwzm79ewN0BQwq6xHYhYdLizd o2oT84hYb+Zuua5Io4MD/8SwkJy5/ofGnOuOvhh45f9PR9zd0134NPA+Fjo0RaHkiSzE 2+SqY1FFW4yvN+UaglTO7f8D5neEB3uRnCS8gZhLzQVb7oygTHX7oRhF5AOWP+CgfxHw 6yDdtzsKeEoHFGu80P0q5+nJdOM3igOr7TA/lW1ifF82JDGJDSy40mqxgt+arVkFQB6i hdyLtBfolQkf1IVd+umzPlon1Mvm7TWVXTdf3b9two3VEaCfjUWGSBp7uEzI1tbB95f9 N/Fg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp10.blacknight.com (outbound-smtp10.blacknight.com. [46.22.139.15]) by mx.google.com with ESMTPS id r18-v6si5649388ejh.206.2019.01.04.04.54.27 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:54:27 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) client-ip=46.22.139.15; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.15 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp10.blacknight.com (Postfix) with ESMTPS id D214C1C1CB6 for ; Fri, 4 Jan 2019 12:54:26 +0000 (GMT) Received: (qmail 12178 invoked from network); 4 Jan 2019 12:54:26 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:54:26 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 24/25] mm, compaction: Capture a page under direct compaction Date: Fri, 4 Jan 2019 12:50:10 +0000 Message-Id: <20190104125011.16071-25-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Compaction is inherently race-prone as a suitable page freed during compaction can be allocated by any parallel task. This patch uses a capture_control structure to isolate a page immediately when it is freed by a direct compactor in the slow path of the page allocator. The intent is to avoid redundant scanning. 4.20.0 4.20.0 selective-v2r15 capture-v2r15 Amean fault-both-1 0.00 ( 0.00%) 0.00 * 0.00%* Amean fault-both-3 2624.85 ( 0.00%) 2594.49 ( 1.16%) Amean fault-both-5 3842.66 ( 0.00%) 4088.32 ( -6.39%) Amean fault-both-7 5459.47 ( 0.00%) 5936.54 ( -8.74%) Amean fault-both-12 9276.60 ( 0.00%) 10160.85 ( -9.53%) Amean fault-both-18 14030.73 ( 0.00%) 13908.92 ( 0.87%) Amean fault-both-24 13298.10 ( 0.00%) 16819.86 * -26.48%* Amean fault-both-30 17648.62 ( 0.00%) 17901.74 ( -1.43%) Amean fault-both-32 19161.67 ( 0.00%) 18621.32 ( 2.82%) Latency is only moderately affected but the devil is in the details. A closer examination indicates that base page fault latency is much reduced but latency of huge pages is increased as it takes creater care to succeed. Part of the "problem" is that allocation success rates are close to 100% even when under pressure and compaction gets harder 4.20.0 4.20.0 selective-v2r15 capture-v2r15 Percentage huge-1 0.00 ( 0.00%) 0.00 ( 0.00%) Percentage huge-3 99.95 ( 0.00%) 99.98 ( 0.03%) Percentage huge-5 98.83 ( 0.00%) 98.01 ( -0.84%) Percentage huge-7 96.78 ( 0.00%) 98.30 ( 1.58%) Percentage huge-12 98.85 ( 0.00%) 97.76 ( -1.10%) Percentage huge-18 97.52 ( 0.00%) 99.05 ( 1.57%) Percentage huge-24 97.07 ( 0.00%) 99.34 ( 2.35%) Percentage huge-30 96.59 ( 0.00%) 99.08 ( 2.58%) Percentage huge-32 95.94 ( 0.00%) 99.03 ( 3.22%) And scan rates are reduced as expected by 10% for the migration scanner and 37% for the free scanner indicating that there is less redundant work. Compaction migrate scanned 20338945.00 18133661.00 Compaction free scanned 12590377.00 7986174.00 The impact on 2-socket is much larger albeit not presented. Under a different workload that fragments heavily, the allocation latency is reduced by 26% while the success rate goes from 63% to 80% Signed-off-by: Mel Gorman --- include/linux/compaction.h | 3 ++- include/linux/sched.h | 4 ++++ kernel/sched/core.c | 3 +++ mm/compaction.c | 31 +++++++++++++++++++------ mm/internal.h | 9 +++++++ mm/page_alloc.c | 58 ++++++++++++++++++++++++++++++++++++++++++---- 6 files changed, 96 insertions(+), 12 deletions(-) diff --git a/include/linux/compaction.h b/include/linux/compaction.h index 68250a57aace..b0d530cf46d1 100644 --- a/include/linux/compaction.h +++ b/include/linux/compaction.h @@ -95,7 +95,8 @@ extern int sysctl_compact_unevictable_allowed; extern int fragmentation_index(struct zone *zone, unsigned int order); extern enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order, unsigned int alloc_flags, - const struct alloc_context *ac, enum compact_priority prio); + const struct alloc_context *ac, enum compact_priority prio, + struct page **page); extern void reset_isolation_suitable(pg_data_t *pgdat); extern enum compact_result compaction_suitable(struct zone *zone, int order, unsigned int alloc_flags, int classzone_idx); diff --git a/include/linux/sched.h b/include/linux/sched.h index 89541d248893..f5ac0cf9cc32 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -47,6 +47,7 @@ struct pid_namespace; struct pipe_inode_info; struct rcu_node; struct reclaim_state; +struct capture_control; struct robust_list_head; struct sched_attr; struct sched_param; @@ -964,6 +965,9 @@ struct task_struct { struct io_context *io_context; +#ifdef CONFIG_COMPACTION + struct capture_control *capture_control; +#endif /* Ptrace state: */ unsigned long ptrace_message; kernel_siginfo_t *last_siginfo; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index f66920173370..ef478b0daa45 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2177,6 +2177,9 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) INIT_HLIST_HEAD(&p->preempt_notifiers); #endif +#ifdef CONFIG_COMPACTION + p->capture_control = NULL; +#endif init_numa_balancing(clone_flags, p); } diff --git a/mm/compaction.c b/mm/compaction.c index 7f316e1a7275..ae70be023b21 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -2051,7 +2051,8 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, return false; } -static enum compact_result compact_zone(struct compact_control *cc) +static enum compact_result +compact_zone(struct compact_control *cc, struct capture_control *capc) { enum compact_result ret; unsigned long start_pfn = cc->zone->zone_start_pfn; @@ -2225,6 +2226,11 @@ static enum compact_result compact_zone(struct compact_control *cc) } } + /* Stop if a page has been captured */ + if (capc && capc->page) { + ret = COMPACT_SUCCESS; + break; + } } out: @@ -2258,7 +2264,8 @@ static enum compact_result compact_zone(struct compact_control *cc) static enum compact_result compact_zone_order(struct zone *zone, int order, gfp_t gfp_mask, enum compact_priority prio, - unsigned int alloc_flags, int classzone_idx) + unsigned int alloc_flags, int classzone_idx, + struct page **capture) { enum compact_result ret; struct compact_control cc = { @@ -2279,14 +2286,24 @@ static enum compact_result compact_zone_order(struct zone *zone, int order, .ignore_skip_hint = (prio == MIN_COMPACT_PRIORITY), .ignore_block_suitable = (prio == MIN_COMPACT_PRIORITY) }; + struct capture_control capc = { + .cc = &cc, + .page = NULL, + }; + + if (capture) + current->capture_control = &capc; INIT_LIST_HEAD(&cc.freepages); INIT_LIST_HEAD(&cc.migratepages); - ret = compact_zone(&cc); + ret = compact_zone(&cc, &capc); VM_BUG_ON(!list_empty(&cc.freepages)); VM_BUG_ON(!list_empty(&cc.migratepages)); + *capture = capc.page; + current->capture_control = NULL; + return ret; } @@ -2304,7 +2321,7 @@ int sysctl_extfrag_threshold = 500; */ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order, unsigned int alloc_flags, const struct alloc_context *ac, - enum compact_priority prio) + enum compact_priority prio, struct page **capture) { int may_perform_io = gfp_mask & __GFP_IO; struct zoneref *z; @@ -2332,7 +2349,7 @@ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order, } status = compact_zone_order(zone, order, gfp_mask, prio, - alloc_flags, ac_classzone_idx(ac)); + alloc_flags, ac_classzone_idx(ac), capture); rc = max(status, rc); /* The allocation should succeed, stop compacting */ @@ -2400,7 +2417,7 @@ static void compact_node(int nid) INIT_LIST_HEAD(&cc.freepages); INIT_LIST_HEAD(&cc.migratepages); - compact_zone(&cc); + compact_zone(&cc, NULL); VM_BUG_ON(!list_empty(&cc.freepages)); VM_BUG_ON(!list_empty(&cc.migratepages)); @@ -2543,7 +2560,7 @@ static void kcompactd_do_work(pg_data_t *pgdat) if (kthread_should_stop()) return; - status = compact_zone(&cc); + status = compact_zone(&cc, NULL); if (status == COMPACT_SUCCESS) { compaction_defer_reset(zone, cc.order, false); diff --git a/mm/internal.h b/mm/internal.h index d028abd8a8f3..6b1e5e313855 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -208,6 +208,15 @@ struct compact_control { bool rescan; /* Rescanning the same pageblock */ }; +/* + * Used in direct compaction when a page should be taken from the freelists + * immediately when one is created during the free path. + */ +struct capture_control { + struct compact_control *cc; + struct page *page; +}; + unsigned long isolate_freepages_range(struct compact_control *cc, unsigned long start_pfn, unsigned long end_pfn); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 05c9a81d54ed..83ea34d8dbe2 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -789,6 +789,41 @@ static inline int page_is_buddy(struct page *page, struct page *buddy, return 0; } +#ifdef CONFIG_COMPACTION +static inline struct capture_control *task_capc(struct zone *zone) +{ + struct capture_control *capc = current->capture_control; + + return capc && + !(current->flags & PF_KTHREAD) && + !capc->page && + capc->cc->zone == zone && + capc->cc->direct_compaction ? capc : NULL; +} + +static inline bool +compaction_capture(struct capture_control *capc, struct page *page, int order) +{ + if (!capc || order != capc->cc->order) + return false; + + capc->page = page; + return true; +} + +#else +static inline struct capture_control *task_capc(struct zone *zone) +{ + return NULL; +} + +static inline bool +compaction_capture(struct capture_control *capc, struct page *page, int order) +{ + return false; +} +#endif /* CONFIG_COMPACTION */ + /* * Freeing function for a buddy system allocator. * @@ -822,6 +857,7 @@ static inline void __free_one_page(struct page *page, unsigned long uninitialized_var(buddy_pfn); struct page *buddy; unsigned int max_order; + struct capture_control *capc = task_capc(zone); max_order = min_t(unsigned int, MAX_ORDER, pageblock_order + 1); @@ -837,6 +873,12 @@ static inline void __free_one_page(struct page *page, continue_merging: while (order < max_order - 1) { + if (compaction_capture(capc, page, order)) { + if (likely(!is_migrate_isolate(migratetype))) + __mod_zone_freepage_state(zone, -(1 << order), + migratetype); + return; + } buddy_pfn = __find_buddy_pfn(pfn, order); buddy = page + (buddy_pfn - pfn); @@ -3700,7 +3742,7 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, unsigned int alloc_flags, const struct alloc_context *ac, enum compact_priority prio, enum compact_result *compact_result) { - struct page *page; + struct page *page = NULL; unsigned long pflags; unsigned int noreclaim_flag; @@ -3711,13 +3753,15 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, noreclaim_flag = memalloc_noreclaim_save(); *compact_result = try_to_compact_pages(gfp_mask, order, alloc_flags, ac, - prio); + prio, &page); memalloc_noreclaim_restore(noreclaim_flag); psi_memstall_leave(&pflags); - if (*compact_result <= COMPACT_INACTIVE) + if (*compact_result <= COMPACT_INACTIVE) { + WARN_ON_ONCE(page); return NULL; + } /* * At least in one zone compaction wasn't deferred or skipped, so let's @@ -3725,7 +3769,13 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, */ count_vm_event(COMPACTSTALL); - page = get_page_from_freelist(gfp_mask, order, alloc_flags, ac); + /* Prep a captured page if available */ + if (page) + prep_new_page(page, order, gfp_mask, alloc_flags); + + /* Try get a page from the freelist if available */ + if (!page) + page = get_page_from_freelist(gfp_mask, order, alloc_flags, ac); if (page) { struct zone *zone = page_zone(page); From patchwork Fri Jan 4 12:50:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 10748349 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A2F7C1575 for ; Fri, 4 Jan 2019 12:54:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 943312847E for ; Fri, 4 Jan 2019 12:54:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 88CB228481; Fri, 4 Jan 2019 12:54:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2A8B428488 for ; Fri, 4 Jan 2019 12:54:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3D14F8E00EB; Fri, 4 Jan 2019 07:54:39 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 380B38E00AE; Fri, 4 Jan 2019 07:54:39 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 270D08E00EB; Fri, 4 Jan 2019 07:54:39 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by kanga.kvack.org (Postfix) with ESMTP id BA91E8E00AE for ; Fri, 4 Jan 2019 07:54:38 -0500 (EST) Received: by mail-ed1-f70.google.com with SMTP id t2so34862056edb.22 for ; Fri, 04 Jan 2019 04:54:38 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=bdIUtOAOt9coFkqKltJJ1JNZAbLhv6qY/uLZwv5FejQ=; b=L+nNzqV4JI5SUnY3IflpBsbRau+cOJdN8EqeXFKthDk0K7fk7/Uu09MHwKiiewuWz/ 2zwRQybZDVZ6ifomyGMbAEtIgo6s0aUjgBzz6htRWPywkWtS2W/ethRRVucXeNaVjxK9 HPPpKlKK+nHylpki4pC6GgCG4v2imwTje6MG0Vxg6BaCMfdLL2Cju9MF46fhsukGiGLd z9vYNyw21m7AMlumoml7dQa7TSbPhLc2qN33d6eKBYHu8K+QBAiEXRVyZcRbrNNZE0Y5 JQNakaETVr7lBHo25saERty6rQYeh3GnU5xbskKFs8yf155NsmP7zqBHq31yH0ocV4LI vmJg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Gm-Message-State: AA+aEWZZM65emyUb2CB6bJN7k8Jh0x7GbpeFovmLSxETARGLFysI/CNU hSdEvLYLcRX1aGzoDhf2d9EPOn7sl8HkBsIKEGNCuMba9L636+Ew4sxDFstj7jzcRx1GlZN3UoT OVBAMDk9xqZ+vTsqF0W5GcnW9FiCoXb5mQVfbeMtoS7MfI6EZse7IpdGyP8uqtvfbEQ== X-Received: by 2002:a50:ee1a:: with SMTP id g26mr45332536eds.266.1546606478264; Fri, 04 Jan 2019 04:54:38 -0800 (PST) X-Google-Smtp-Source: AFSGD/UtN6GzvQhAjcuXza8mAoQvsiGkQQqJWhyebHz2INGy+F7t1fOpTvRG2mAs4m9uTKU+28Co X-Received: by 2002:a50:ee1a:: with SMTP id g26mr45332501eds.266.1546606477386; Fri, 04 Jan 2019 04:54:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1546606477; cv=none; d=google.com; s=arc-20160816; b=za8PCZE+sliIYzqQ8KjNLTh4BEuiGqOrE9TxQpXTLJAoS89IibhDeO5NbnwKEVNq3w FdKx1tAY0Byb7NviBshEm6N0LhLfD0I106eSnIZwzFa04R/hE9mIoJdWvR/ZfxCWNhZu +wJdWBPUgoclTyBdtXvvJgevQEE8EdYBh00rL5wT1PzXHNDqzQN0VRISsyT4AVDj1iwE Y0vNbhUF6tK58SJ1kZQq6PV13fD62LXzZJNApbP5f4BS0cnNXbfDnqGa9n8S9Wc2LPeY 9QrlNuhm83xOX7gdhSLqG4Xa2NIBo35oTEH5TvqqWnJbe+A37uS0lbQqWHNA7LxMojxd uwxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=bdIUtOAOt9coFkqKltJJ1JNZAbLhv6qY/uLZwv5FejQ=; b=ZuYVS2TTxZVMPikkDdJfzrFEOZrtmerD6PvgdCz2huuUhse3kdKqoWdG9rt7QWVWf5 cKQodLo1cmn3xACcXffKLiqJZfogArImopKE+lIzh+pkbxatm2U1Ilui2HPtd05xllMH ctL7GGbI2NpKrBXPVyAR0UjWhHl2ceZzLop0WgedvbD2SOjcUIZryQba7Xkzv4pd7wfY dAZyQtKGr6kBDLfrANOSVb2HYA1tQIgDY2f6UjK5LqGvqeT2ifoqYW28AuEuswRIbKsA yUqg68RtauxAAeDBAmURmdkcQLIjy8HNfqDM0lFyqxYPUMP37Conb+ANdivZq1/Hrvyk QpPw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from outbound-smtp13.blacknight.com (outbound-smtp13.blacknight.com. [46.22.139.230]) by mx.google.com with ESMTPS id l1si937018edc.252.2019.01.04.04.54.37 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 04 Jan 2019 04:54:37 -0800 (PST) Received-SPF: pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) client-ip=46.22.139.230; Authentication-Results: mx.google.com; spf=pass (google.com: domain of mgorman@techsingularity.net designates 46.22.139.230 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net Received: from mail.blacknight.com (unknown [81.17.254.16]) by outbound-smtp13.blacknight.com (Postfix) with ESMTPS id 0A7B71C1C22 for ; Fri, 4 Jan 2019 12:54:37 +0000 (GMT) Received: (qmail 12608 invoked from network); 4 Jan 2019 12:54:36 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 4 Jan 2019 12:54:36 -0000 From: Mel Gorman To: Linux-MM Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , ying.huang@intel.com, kirill@shutemov.name, Andrew Morton , Linux List Kernel Mailing , Mel Gorman Subject: [PATCH 25/25] mm, compaction: Do not direct compact remote memory Date: Fri, 4 Jan 2019 12:50:11 +0000 Message-Id: <20190104125011.16071-26-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190104125011.16071-1-mgorman@techsingularity.net> References: <20190104125011.16071-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Remote compaction is expensive and possibly counter-productive. Locality is expected to often have better performance characteristics than remote high-order pages. For small allocations, it's expected that locality is generally required or fallbacks are possible. For larger allocations such as THP, they are forbidden at the time of writing but if __GFP_THISNODE is ever removed, then it would still be preferable to fallback to small local base pages over remote THP in the general case. kcompactd is still woken via kswapd so compaction happens eventually. While this patch potentially has both positive and negative effects, it is best to avoid the possibility of remote compaction given the cost relative to any potential benefit. Signed-off-by: Mel Gorman --- mm/compaction.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/mm/compaction.c b/mm/compaction.c index ae70be023b21..cc17f0c01811 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -2348,6 +2348,16 @@ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order, continue; } + /* + * Do not compact remote memory. It's expensive and high-order + * small allocations are expected to prefer or require local + * memory. Similarly, larger requests such as THP can fallback + * to base pages in preference to remote huge pages if + * __GFP_THISNODE is not specified + */ + if (zone_to_nid(zone) != zone_to_nid(ac->preferred_zoneref->zone)) + continue; + status = compact_zone_order(zone, order, gfp_mask, prio, alloc_flags, ac_classzone_idx(ac), capture); rc = max(status, rc);