From patchwork Wed Aug 19 04:27:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 11722495 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 990C1739 for ; Wed, 19 Aug 2020 04:27:11 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 65AA22083B for ; Wed, 19 Aug 2020 04:27:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="pO3zqnqL" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 65AA22083B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9CC9B8D0002; Wed, 19 Aug 2020 00:27:10 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9A2BD6B0036; Wed, 19 Aug 2020 00:27:10 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8B8A98D0002; Wed, 19 Aug 2020 00:27:10 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0237.hostedemail.com [216.40.44.237]) by kanga.kvack.org (Postfix) with ESMTP id 7661A6B0033 for ; Wed, 19 Aug 2020 00:27:10 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 15BE2180AD817 for ; Wed, 19 Aug 2020 04:27:10 +0000 (UTC) X-FDA: 77166033420.05.bear22_391568727025 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id CEA3618016858 for ; Wed, 19 Aug 2020 04:27:09 +0000 (UTC) X-Spam-Summary: 1,0,0,d63d7752cbc6c78c,d41d8cd98f00b204,alexander.duyck@gmail.com,,RULES_HIT:41:69:152:355:379:960:973:988:989:1260:1277:1311:1313:1314:1345:1359:1431:1437:1515:1516:1518:1535:1542:1593:1594:1711:1730:1747:1777:1792:2194:2198:2199:2200:2393:2559:2562:2693:2731:2899:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3872:3874:4117:4250:4321:5007:6119:6120:6261:6653:6742:7576:7903:8957:9010:9413:9592:10004:10400:11026:11232:11658:11914:12043:12114:12297:12438:12517:12519:12555:12679:12683:12760:12986:13255:14096:14097:14181:14687:14721:21080:21444:21451:21627:21666:21939:21990:30054:30056:30069:30070,0,RBL:209.85.219.67:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.50.0.100;04yfby981yzso4mytaq1qxgddnutsypw1dnc9mxk4oyanffmq8dmb7ncr9sa96c.8qcawtoucu53se8ej3e1wo6xrqf17id6xsfh3f1jbqfb65cfy99bmwjmssxexo6.s-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules: 0:0:0,LF X-HE-Tag: bear22_391568727025 X-Filterd-Recvd-Size: 6126 Received: from mail-qv1-f67.google.com (mail-qv1-f67.google.com [209.85.219.67]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Wed, 19 Aug 2020 04:27:09 +0000 (UTC) Received: by mail-qv1-f67.google.com with SMTP id dd12so10712159qvb.0 for ; Tue, 18 Aug 2020 21:27:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=O0GPbeNx4pE5Cj/la6lYT0zllz6aGU2XNC8iLf9uFA4=; b=pO3zqnqL4c/mDAjAfh4ZWDwEb7d57VhBDoidBgKwr3U5coFnxDOXwnl2WrCcEy2ytc vPoW0tq9uKas+aNj5L2ZSate75S8hark40KC/B1fHTbUYPsjwNaa6uOzkoMyUQDpmlBL dOt7/SWEQPgLq23TAemQobbfIE2Jpi9Y99z4Ne9S+gMTKMvQKs3YapyvSlEmI2di09pJ dHjRXJ2gP38hCNvvOZk4UqKmcWaQRFnTLqeFlhyGet0x8XXRiPSc12oSao2IOrQonhDN 0qNWU11jMey8c9BEr8FQP9SCHMrSyMK8EtIwBEOpavLMBYW6DnNK/t44D+cr7nXdgrJv 7tUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=O0GPbeNx4pE5Cj/la6lYT0zllz6aGU2XNC8iLf9uFA4=; b=uMGp8gkk7oUHStXvOV5Ak6PdUi0ux6w04SjUAZEfhdKlogbArWLrwEL6KlSQbEX9S7 3DjQXMMZ6tpFkgdBaCC1eCc38SASKo2qQE3UHNxb7kYc2Xu9AVyjpcgucNMs77yX0T68 /hJnYeM2xk+vU0iADI+PJAoWcTitZ0dn9A83yTlLeWWLiQv/Mc8XRui8hDjpTMlRHRyR WKJgk/qOrtcVAhC7rPls4szFk/R9DDRieEgBg3lF78Bni0LbsS5E3Ei/SQsWQinW/FOg AmvTQRYebgAJpv3BiSAQLbazTqlqtqMJAwgK6lp+Jys1pwgUyk8/1eTh+5MXR50rd0Lg a0Iw== X-Gm-Message-State: AOAM531BxqRkT8crk5cmDa2QFjxuo0SL22IPIQYfTdfZ92dj35o/DG4A TpQQI3+Qps6Ub3vEWorUH64= X-Google-Smtp-Source: ABdhPJxW+1uPbr69HlT8zdk5DCZHvushfLGH5Z8cxaaxbXoq04RhdHu82fDyOZeUTBoieLU6jFhc1g== X-Received: by 2002:ad4:4b0b:: with SMTP id r11mr22115044qvw.94.1597811228761; Tue, 18 Aug 2020 21:27:08 -0700 (PDT) Received: from localhost.localdomain ([2001:470:b:9c3:9e5c:8eff:fe4f:f2d0]) by smtp.gmail.com with ESMTPSA id p17sm21999894qkj.69.2020.08.18.21.27.06 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Aug 2020 21:27:08 -0700 (PDT) Subject: [RFC PATCH v2 1/5] mm: Identify compound pages sooner in isolate_migratepages_block From: Alexander Duyck To: alex.shi@linux.alibaba.com Cc: yang.shi@linux.alibaba.com, lkp@intel.com, rong.a.chen@intel.com, khlebnikov@yandex-team.ru, kirill@shutemov.name, hughd@google.com, linux-kernel@vger.kernel.org, alexander.duyck@gmail.com, daniel.m.jordan@oracle.com, linux-mm@kvack.org, shakeelb@google.com, willy@infradead.org, hannes@cmpxchg.org, tj@kernel.org, cgroups@vger.kernel.org, akpm@linux-foundation.org, richard.weiyang@gmail.com, mgorman@techsingularity.net, iamjoonsoo.kim@lge.com Date: Tue, 18 Aug 2020 21:27:05 -0700 Message-ID: <20200819042705.23414.84098.stgit@localhost.localdomain> In-Reply-To: <20200819041852.23414.95939.stgit@localhost.localdomain> References: <20200819041852.23414.95939.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Rspamd-Queue-Id: CEA3618016858 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Alexander Duyck Since we are holding a reference to the page much sooner in isolate_migratepages_block we can move the PageCompound check out of the LRU locked section and instead just place it after get_page_unless_zero. By doing this we can allow any of the items that might trigger a failure to trigger a failure for the compound page rather than the order 0 page and as a result we should be able to process the pageblock faster. In addition by testing for PageCompound sooner we can avoid having the LRU flag cleared and then reset in the exception case. As a result this should prevent possible races where another thread might be attempting to pull the LRU pages from the list. Signed-off-by: Alexander Duyck Reviewed-by: Alex Shi --- mm/compaction.c | 33 ++++++++++++++++++--------------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index d3f87f759773..88c7b950f676 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -984,6 +984,24 @@ static bool too_many_isolated(pg_data_t *pgdat) if (unlikely(!get_page_unless_zero(page))) goto isolate_fail; + /* + * Page is compound. We know the order before we know if it is + * on the LRU so we cannot assume it is THP. However since the + * page will have the LRU validated shortly we can use the value + * to skip over this page for now or validate the LRU is set and + * then isolate the entire compound page if we are isolating to + * generate a CMA page. + */ + if (PageCompound(page)) { + const unsigned int order = compound_order(page); + + if (likely(order < MAX_ORDER)) + low_pfn += (1UL << order) - 1; + + if (!cc->alloc_contig) + goto isolate_fail_put; + } + if (__isolate_lru_page_prepare(page, isolate_mode) != 0) goto isolate_fail_put; @@ -1009,23 +1027,8 @@ static bool too_many_isolated(pg_data_t *pgdat) if (test_and_set_skip(cc, page, low_pfn)) goto isolate_abort; } - - /* - * Page become compound since the non-locked check, - * and it's on LRU. It can only be a THP so the order - * is safe to read and it's 0 for tail pages. - */ - if (unlikely(PageCompound(page) && !cc->alloc_contig)) { - low_pfn += compound_nr(page) - 1; - SetPageLRU(page); - goto isolate_fail_put; - } } - /* The whole page is taken off the LRU; skip the tail pages. */ - if (PageCompound(page)) - low_pfn += compound_nr(page) - 1; - /* Successfully isolated */ del_page_from_lru_list(page, lruvec, page_lru(page)); mod_node_page_state(page_pgdat(page), From patchwork Wed Aug 19 04:27:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 11722497 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CF051138C for ; Wed, 19 Aug 2020 04:27:19 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8E8842083B for ; Wed, 19 Aug 2020 04:27:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="o+etTA4y" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8E8842083B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C8BBE8D0005; Wed, 19 Aug 2020 00:27:18 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BEE226B0036; Wed, 19 Aug 2020 00:27:18 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A8E778D0005; Wed, 19 Aug 2020 00:27:18 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 8D6266B0033 for ; Wed, 19 Aug 2020 00:27:18 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 4A47A181AEF09 for ; Wed, 19 Aug 2020 04:27:18 +0000 (UTC) X-FDA: 77166033756.21.suit78_5a177cb27025 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin21.hostedemail.com (Postfix) with ESMTP id 1A501180442C0 for ; Wed, 19 Aug 2020 04:27:18 +0000 (UTC) X-Spam-Summary: 1,0,0,e39efb698f9125f3,d41d8cd98f00b204,alexander.duyck@gmail.com,,RULES_HIT:41:69:152:273:355:379:960:973:988:989:1260:1277:1311:1313:1314:1345:1359:1431:1437:1515:1516:1518:1535:1544:1593:1594:1605:1711:1730:1747:1777:1792:2393:2559:2562:2693:2895:2901:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4118:4250:5007:6119:6120:6261:6653:6742:7576:7903:8660:8957:9413:9592:10004:11026:11232:11658:11914:12043:12296:12297:12438:12517:12519:12555:12679:12683:12760:13141:13148:13161:13229:13230:13255:14096:14097:14181:14687:14721:21063:21080:21444:21451:21627:21666:21740:21795:21939:21990:30012:30051:30054:30056:30070,0,RBL:209.85.160.195:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100;04y8e9x89wh3mc8m9h56y8apycq7eycmixc4eomoooa89bwa3smu95tzcmxqxp7.ma3hsducyhdcin54thk6m31ka4zihkpk83qcapjg5y1x87ze8rpxnfyy5riwxw6.a-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL :neutral X-HE-Tag: suit78_5a177cb27025 X-Filterd-Recvd-Size: 7545 Received: from mail-qt1-f195.google.com (mail-qt1-f195.google.com [209.85.160.195]) by imf39.hostedemail.com (Postfix) with ESMTP for ; Wed, 19 Aug 2020 04:27:17 +0000 (UTC) Received: by mail-qt1-f195.google.com with SMTP id v22so16908971qtq.8 for ; Tue, 18 Aug 2020 21:27:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=nJQwZMk0HVg1Xi5MdVbpw2haA8xPFp0DOao/4ztkzvc=; b=o+etTA4y6IyxOSsiGgtDeP6nsslFfCsdsqI/ytWlD0PrEUbBiCogFNkpkPYaJS2uSV 5Jrtw6H7e8g3+b0ajpM/n1TB1IoHygoMchrjdnR0ctTVyV0u7c9ejlKYhU6tpaFAg8lf cxEjRIH6aarpICQ3D40Jep09AZ1RKdcrXs8Hqoqzr4SdDacFmBkgE/O0LUZAi6SWO5TI 6GtbeZRf+u/eU7YRn+F6n6QC9lvwgq7CpKoibwq4XfQwRL7JOtdwY79HY6VrXMs2V0Lm faOt3XiyhX3ezawcRskNnct6vHVkrJ6FnE5Eo4JnELvFe2nfcGZXW8LJgF6OIMBWdvfL d8wA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=nJQwZMk0HVg1Xi5MdVbpw2haA8xPFp0DOao/4ztkzvc=; b=CvB5NLTl7iYyOmtlH58ea5mIWp4SzFBKcRfQ/q7vgDiLOnwxA/Ilwadjs4rbulGpqe ut/xt9t7KbKAfEnN4BnIy824gOEZhTy2xBXvZr3SBu67KeyekGxM43+lqx79U4Vw1Jgv hGRk0ETSIHSgzJUaod9WfueTvWFourSfIRxtYEvteAf5LZbJ4b526CK3bHPN8fyPKDA0 UsBhi49BseLwb3C1LV4HnwpO3HQVi5MNKvW9i857RVel3AxlVwGquHPqIjy+HCftVLzK 2nGplK2C/KXYuWo1BPMx2TRkNkLqclJTn1dRyP90lkhWF8F6jg3EYMrXPCLFI++lgHhJ /xqQ== X-Gm-Message-State: AOAM530eAFoyOMKL7vTuhLFR6yGLQ9p5oCrCbSHB/YAfify2ebjHADOQ D3ttcl7vV6ehnTf05ropbtQ= X-Google-Smtp-Source: ABdhPJxoECmzGuSI1+0DQ9CGQygdSLOyI+slhpLtdVXP5IOP+KyGcOaAEGCSGUdabuqBc1XlA2TaKQ== X-Received: by 2002:aed:3e0c:: with SMTP id l12mr21600150qtf.179.1597811236964; Tue, 18 Aug 2020 21:27:16 -0700 (PDT) Received: from localhost.localdomain ([2001:470:b:9c3:9e5c:8eff:fe4f:f2d0]) by smtp.gmail.com with ESMTPSA id p33sm27941856qtp.49.2020.08.18.21.27.14 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Aug 2020 21:27:16 -0700 (PDT) Subject: [RFC PATCH v2 2/5] mm: Drop use of test_and_set_skip in favor of just setting skip From: Alexander Duyck To: alex.shi@linux.alibaba.com Cc: yang.shi@linux.alibaba.com, lkp@intel.com, rong.a.chen@intel.com, khlebnikov@yandex-team.ru, kirill@shutemov.name, hughd@google.com, linux-kernel@vger.kernel.org, alexander.duyck@gmail.com, daniel.m.jordan@oracle.com, linux-mm@kvack.org, shakeelb@google.com, willy@infradead.org, hannes@cmpxchg.org, tj@kernel.org, cgroups@vger.kernel.org, akpm@linux-foundation.org, richard.weiyang@gmail.com, mgorman@techsingularity.net, iamjoonsoo.kim@lge.com Date: Tue, 18 Aug 2020 21:27:14 -0700 Message-ID: <20200819042713.23414.5084.stgit@localhost.localdomain> In-Reply-To: <20200819041852.23414.95939.stgit@localhost.localdomain> References: <20200819041852.23414.95939.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Rspamd-Queue-Id: 1A501180442C0 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Alexander Duyck The only user of test_and_set_skip was isolate_migratepages_block and it was using it after a call that was testing and clearing the LRU flag. As such it really didn't need to be behind the LRU lock anymore as it wasn't really fulfilling its purpose. Since it is only possible to be able to test and set the skip flag if we were able to obtain the LRU bit for the first page in the pageblock the use of the test_and_set_skip becomes redundant as the LRU flag now becomes the item that limits us to only one thread being able to perform the operation and there being no need for a test_and_set operation. With that being the case we can simply drop the bit and instead directly just call the set_pageblock_skip function if the page we are working on is the valid_page at the start of the pageblock. Then any other threads that enter this pageblock should see the skip bit set on the first valid page in the pageblock. Since we have dropped the late abort case we can drop the code that was clearing the LRU flag and calling page_put since the abort case will now not be holding a reference to a page now. Signed-off-by: Alexander Duyck Reviewed-by: Alex Shi --- mm/compaction.c | 53 +++++++++++++---------------------------------------- 1 file changed, 13 insertions(+), 40 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 88c7b950f676..f986c67e83cc 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -399,29 +399,6 @@ void reset_isolation_suitable(pg_data_t *pgdat) } } -/* - * Sets the pageblock skip bit if it was clear. Note that this is a hint as - * locks are not required for read/writers. Returns true if it was already set. - */ -static bool test_and_set_skip(struct compact_control *cc, struct page *page, - unsigned long pfn) -{ - bool skip; - - /* Do no update if skip hint is being ignored */ - if (cc->ignore_skip_hint) - return false; - - if (!IS_ALIGNED(pfn, pageblock_nr_pages)) - return false; - - skip = get_pageblock_skip(page); - if (!skip && !cc->no_set_skip_hint) - set_pageblock_skip(page); - - return skip; -} - static void update_cached_migrate(struct compact_control *cc, unsigned long pfn) { struct zone *zone = cc->zone; @@ -480,12 +457,6 @@ static inline void update_pageblock_skip(struct compact_control *cc, static void update_cached_migrate(struct compact_control *cc, unsigned long pfn) { } - -static bool test_and_set_skip(struct compact_control *cc, struct page *page, - unsigned long pfn) -{ - return false; -} #endif /* CONFIG_COMPACTION */ /* @@ -895,7 +866,6 @@ static bool too_many_isolated(pg_data_t *pgdat) if (!valid_page && IS_ALIGNED(low_pfn, pageblock_nr_pages)) { if (!cc->ignore_skip_hint && get_pageblock_skip(page)) { low_pfn = end_pfn; - page = NULL; goto isolate_abort; } valid_page = page; @@ -1021,11 +991,20 @@ static bool too_many_isolated(pg_data_t *pgdat) lruvec_memcg_debug(lruvec, page); - /* Try get exclusive access under lock */ - if (!skip_updated) { + /* + * Indicate that we want exclusive access to the + * rest of the pageblock. + * + * The LRU flag prevents simultaneous access to the + * first PFN, and the LRU lock helps to prevent + * simultaneous update of multiple pageblocks shared + * in the same bitmap. + */ + if (page == valid_page) { + if (!cc->ignore_skip_hint && + !cc->no_set_skip_hint) + set_pageblock_skip(page); skip_updated = true; - if (test_and_set_skip(cc, page, low_pfn)) - goto isolate_abort; } } @@ -1098,15 +1077,9 @@ static bool too_many_isolated(pg_data_t *pgdat) if (unlikely(low_pfn > end_pfn)) low_pfn = end_pfn; - page = NULL; - isolate_abort: if (lruvec) unlock_page_lruvec_irqrestore(lruvec, flags); - if (page) { - SetPageLRU(page); - put_page(page); - } /* * Updated the cached scanner pfn once the pageblock has been scanned From patchwork Wed Aug 19 04:27:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 11722499 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CD24F739 for ; Wed, 19 Aug 2020 04:27:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 90A5020772 for ; Wed, 19 Aug 2020 04:27:27 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Eoxng6/B" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 90A5020772 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CDBB28D0008; Wed, 19 Aug 2020 00:27:26 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C633A6B0036; Wed, 19 Aug 2020 00:27:26 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B523E8D0008; Wed, 19 Aug 2020 00:27:26 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0213.hostedemail.com [216.40.44.213]) by kanga.kvack.org (Postfix) with ESMTP id 9A0F76B0033 for ; Wed, 19 Aug 2020 00:27:26 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 60F9F824805A for ; Wed, 19 Aug 2020 04:27:26 +0000 (UTC) X-FDA: 77166034092.28.store63_520f85d27025 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id 2E9F76C3B for ; Wed, 19 Aug 2020 04:27:26 +0000 (UTC) X-Spam-Summary: 1,0,0,5e884b3373084492,d41d8cd98f00b204,alexander.duyck@gmail.com,,RULES_HIT:41:152:355:379:960:973:988:989:1260:1277:1311:1313:1314:1345:1359:1431:1437:1515:1516:1518:1534:1541:1593:1594:1711:1730:1747:1777:1792:2198:2199:2393:2553:2559:2562:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3870:3871:3872:3874:4250:4321:5007:6120:6261:6653:6742:7576:8660:8957:9413:10004:10128:10400:11026:11658:11914:12043:12296:12297:12438:12517:12519:12555:12679:12760:13069:13148:13161:13229:13230:13255:13311:13357:14096:14097:14181:14687:14721:21080:21444:21451:21627:21666:21939:30054:30090,0,RBL:209.85.222.193:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100;04yfa4p8q1mcyhc5dom79n1yyzj3gyc51gk6hknm6wko4dffkpf7m4r48q3zcqg.8h9kuqauh7szb1s5gdkw659db3p7xj93ew6gc79gnccsp8ghas8i7gk9tmcw3b6.c-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:no ne X-HE-Tag: store63_520f85d27025 X-Filterd-Recvd-Size: 4999 Received: from mail-qk1-f193.google.com (mail-qk1-f193.google.com [209.85.222.193]) by imf01.hostedemail.com (Postfix) with ESMTP for ; Wed, 19 Aug 2020 04:27:25 +0000 (UTC) Received: by mail-qk1-f193.google.com with SMTP id b14so20519729qkn.4 for ; Tue, 18 Aug 2020 21:27:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=T7hUYa5UogPv6sBfCV0FhXNPQBfOPqefw0rv85aPgto=; b=Eoxng6/BeOd5C+j6fxh7qVKG8VAHJChogDv9BN+ih1vUiDUYa428vI1Qr+RZbHjb+F z3PDkhrTAEWMFXQqqS47dXG8H6GuVtMQfIdV1JByxC7hytaIGD/THiPJiKiGDitv9Mu4 SF3IEcMaQ/mNQsTxJY6Sja8V162q2FroarImRxmS5HP5urBApgHmPQxX4t9Ai+lsXxSG OYKZmN1fuUCZoYCc/WUjYKRxEgh+BdWg8jQZdErGjvVfvxnfHHrBPVrpqC0lZKb0eTR+ n58O8F+j9uYedxQzKWsfe2axi/nAxCbseEV93iRLNiPzCtPxbpaHLF8Aslyd2pM2RZa+ 82Mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=T7hUYa5UogPv6sBfCV0FhXNPQBfOPqefw0rv85aPgto=; b=NOBZia7cifbGdS0HMrnC98pDiCKKhFRGA2RRa3uGLKWovpWIY26JZ/Pqnpz2/LECU/ FzOr4e6mzXu7dopx2mO/R+9RbXa5FetsxfWN2j68B6wKkK2nhhWATRag8LQjJd1ryKNR ziFykp00CQ0bclemnCX6KD2cRUqFcnCLAaoftfjXIZPaKVYc039V9olet5jpSuECwm7s bF6YzNjPPyqArOWkmnvoXqXkxtnencj+e+/kgBDnt/SbbcYdrKu2dQBK++TFC5P8jqND G6C8nyb67kuHS0dmnFJbugWHjnEt0rhCjQWDtYURh8UhLV+MstM2k/YIf/VHPcZQ2eFY SEsg== X-Gm-Message-State: AOAM530l3mtH940u5FvHfMYspeUYqKP69ePQQxUYhhHCfS4xBgrGN7jE 2oraxhh9NJTUwoygil2QPTY= X-Google-Smtp-Source: ABdhPJwjewQd6K+YoPWRo0i16OTkUs6Abew8umKdDVBaSIhn0FBlk9jhcWGGDVzESBeD13fateK8hQ== X-Received: by 2002:a37:bd46:: with SMTP id n67mr20703006qkf.190.1597811245192; Tue, 18 Aug 2020 21:27:25 -0700 (PDT) Received: from localhost.localdomain ([2001:470:b:9c3:9e5c:8eff:fe4f:f2d0]) by smtp.gmail.com with ESMTPSA id w58sm27335636qth.95.2020.08.18.21.27.23 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Aug 2020 21:27:24 -0700 (PDT) Subject: [RFC PATCH v2 3/5] mm: Add explicit page decrement in exception path for isolate_lru_pages From: Alexander Duyck To: alex.shi@linux.alibaba.com Cc: yang.shi@linux.alibaba.com, lkp@intel.com, rong.a.chen@intel.com, khlebnikov@yandex-team.ru, kirill@shutemov.name, hughd@google.com, linux-kernel@vger.kernel.org, alexander.duyck@gmail.com, daniel.m.jordan@oracle.com, linux-mm@kvack.org, shakeelb@google.com, willy@infradead.org, hannes@cmpxchg.org, tj@kernel.org, cgroups@vger.kernel.org, akpm@linux-foundation.org, richard.weiyang@gmail.com, mgorman@techsingularity.net, iamjoonsoo.kim@lge.com Date: Tue, 18 Aug 2020 21:27:22 -0700 Message-ID: <20200819042722.23414.2654.stgit@localhost.localdomain> In-Reply-To: <20200819041852.23414.95939.stgit@localhost.localdomain> References: <20200819041852.23414.95939.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Rspamd-Queue-Id: 2E9F76C3B X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Alexander Duyck In isolate_lru_pages we have an exception path where if we call get_page_unless_zero and that succeeds, but TestClearPageLRU fails we call put_page. Normally this would be problematic but due to the way that the calls are ordered and the fact that we are holding the LRU lock we know that the caller must be holding another reference for the page. Since we can assume that we can replace the put_page with a call to put_page_testzero contained within a WARN_ON. By doing this we should see if we ever leak a page as a result of the reference count somehow hitting zero when it shouldn't, and can avoid the overhead and confusion of using the full put_page call. Signed-off-by: Alexander Duyck Reviewed-by: Alex Shi --- mm/vmscan.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 5bc0c2322043..3ebe3f9b653b 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1688,10 +1688,13 @@ static unsigned long isolate_lru_pages(unsigned long nr_to_scan, if (!TestClearPageLRU(page)) { /* - * This page may in other isolation path, - * but we still hold lru_lock. + * This page is being isolated in another + * thread, but we still hold lru_lock. The + * other thread must be holding a reference + * to the page so this should never hit a + * reference count of 0. */ - put_page(page); + WARN_ON(put_page_testzero(page)); goto busy; } From patchwork Wed Aug 19 04:27:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 11722501 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 48701138C for ; Wed, 19 Aug 2020 04:27:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 09FB820772 for ; Wed, 19 Aug 2020 04:27:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="uTztFXSy" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 09FB820772 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 365E38D0009; Wed, 19 Aug 2020 00:27:35 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2EE916B0036; Wed, 19 Aug 2020 00:27:35 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1B4108D0009; Wed, 19 Aug 2020 00:27:35 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0101.hostedemail.com [216.40.44.101]) by kanga.kvack.org (Postfix) with ESMTP id F25E16B0033 for ; Wed, 19 Aug 2020 00:27:34 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id A6F531F0A for ; Wed, 19 Aug 2020 04:27:34 +0000 (UTC) X-FDA: 77166034428.21.nut02_2d0888a27025 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin21.hostedemail.com (Postfix) with ESMTP id 728AF180442C0 for ; Wed, 19 Aug 2020 04:27:34 +0000 (UTC) X-Spam-Summary: 1,0,0,f1c4460db4a64058,d41d8cd98f00b204,alexander.duyck@gmail.com,,RULES_HIT:2:41:69:152:355:379:421:960:966:968:973:988:989:1260:1277:1311:1313:1314:1345:1359:1431:1437:1515:1516:1518:1535:1593:1594:1605:1606:1730:1747:1777:1792:2194:2196:2198:2199:2200:2201:2393:2559:2562:2693:2731:2898:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4119:4250:4321:4385:5007:6119:6120:6261:6653:6742:7576:7875:7903:8531:8957:9010:9413:9592:9707:10004:11026:11232:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12679:12683:12760:13161:13229:14096:14097:14687:21080:21444:21451:21627:21666:21740:21796:21987:21990:30034:30036:30054:30070:30091,0,RBL:209.85.222.194:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100;04yfkgsz7rm4uywwt4zc3gudm1pfxoco3oyuzomeg8o6amjn7uu61ekxfsq6us1.pa93cwrhaa6hrrnudd1pmbtuzwxbbzwgbbd1nkz19phymq9izh4961ix8k6qutg.w-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MS F:not bu X-HE-Tag: nut02_2d0888a27025 X-Filterd-Recvd-Size: 8974 Received: from mail-qk1-f194.google.com (mail-qk1-f194.google.com [209.85.222.194]) by imf25.hostedemail.com (Postfix) with ESMTP for ; Wed, 19 Aug 2020 04:27:33 +0000 (UTC) Received: by mail-qk1-f194.google.com with SMTP id n129so20516414qkd.6 for ; Tue, 18 Aug 2020 21:27:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=5RYocxNjH/JuNLjY5mlxGXA5aUyyMf0iKZRCbtOnwVc=; b=uTztFXSyFhR+OqlR3qfbXh972QyhBGIQS57dnpJlCe8O8USD/B0vecgu2PDPSYgKLQ FH7u9ZEn/yjCqyfcU9tMUiZcj6IkFFDIauBIO7M3DLoyzlkO4UXhA/d89sksieOAIfUp OvPZZ/6AcVwrB6TF20v/UNVjUQyiGVpJNIVrSh7uxtzJUfn4Axb7rsU3cnSlphprgL86 3mHT3A7y41n9L2sPVhTkqxGdOYHHqYUZfAzECqj2uDykgGUC0HHiWRWoQyG4fdBMezQp vPZxkPPpBxkPwUyrxX59CVv2keRH9keUcRf/hmOc+e4Kixb1yZdEgkaQ0fjWLae+b99P Xslg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=5RYocxNjH/JuNLjY5mlxGXA5aUyyMf0iKZRCbtOnwVc=; b=FEKLVF+dDQ5vWbaVd0dZ3e8k0mDVYINyjIPTIBluvMBCjHhj6sBXU4ChML0i9XRTMc 9Z7sXVh8DIQO/hYBQslhufkHnHjhIq342WhGcGRKfVmSEbMVPOgXHKHVyL8Ozr2ostFV sRFLJCroWLSXGjV+9K3t/Kq1thxXIgKc+fqPB7bR6augKBnRmV0TAgGZfTi629VgVpt2 +G//KKa3ud4J6ofBMgdPM3nqAmETclkpAwLKtH4a3AW8riJWgD5rbBbaKCgFGN6dMrvA 3JLYPcV5X/ZauCge+R8Lw7ycbOAkDv/hTODAujgZea75Xfu6igZcO5iYmMGa90KkfVBu sayA== X-Gm-Message-State: AOAM532OlC7aM9GCxnjBtQy1+cOJEUUSrZPchGwryk2r7HltZfr+5cJx PnOqXvko/dii0pXQ6lVxbu4= X-Google-Smtp-Source: ABdhPJxUUKPy8cTPbdHxNKJ42tPwCQ6Lf76LRyVq5ZadiUVAip+EDc39Tjp9mkEub1FXGbbEuTk89g== X-Received: by 2002:a05:620a:1429:: with SMTP id k9mr19720876qkj.273.1597811253360; Tue, 18 Aug 2020 21:27:33 -0700 (PDT) Received: from localhost.localdomain ([2001:470:b:9c3:9e5c:8eff:fe4f:f2d0]) by smtp.gmail.com with ESMTPSA id c33sm26752405qtk.40.2020.08.18.21.27.31 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Aug 2020 21:27:32 -0700 (PDT) Subject: [RFC PATCH v2 4/5] mm: Split release_pages work into 3 passes From: Alexander Duyck To: alex.shi@linux.alibaba.com Cc: yang.shi@linux.alibaba.com, lkp@intel.com, rong.a.chen@intel.com, khlebnikov@yandex-team.ru, kirill@shutemov.name, hughd@google.com, linux-kernel@vger.kernel.org, alexander.duyck@gmail.com, daniel.m.jordan@oracle.com, linux-mm@kvack.org, shakeelb@google.com, willy@infradead.org, hannes@cmpxchg.org, tj@kernel.org, cgroups@vger.kernel.org, akpm@linux-foundation.org, richard.weiyang@gmail.com, mgorman@techsingularity.net, iamjoonsoo.kim@lge.com Date: Tue, 18 Aug 2020 21:27:30 -0700 Message-ID: <20200819042730.23414.41309.stgit@localhost.localdomain> In-Reply-To: <20200819041852.23414.95939.stgit@localhost.localdomain> References: <20200819041852.23414.95939.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Rspamd-Queue-Id: 728AF180442C0 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Alexander Duyck The release_pages function has a number of paths that end up with the LRU lock having to be released and reacquired. Such an example would be the freeing of THP pages as it requires releasing the LRU lock so that it can be potentially reacquired by __put_compound_page. In order to avoid that we can split the work into 3 passes, the first without the LRU lock to go through and sort out those pages that are not in the LRU so they can be freed immediately from those that can't. The second pass will then go through removing those pages from the LRU in batches as large as a pagevec can hold before freeing the LRU lock. Once the pages have been removed from the LRU we can then proceed to free the remaining pages without needing to worry about if they are in the LRU any further. The general idea is to avoid bouncing the LRU lock between pages and to hopefully aggregate the lock for up to the full page vector worth of pages. Signed-off-by: Alexander Duyck --- mm/swap.c | 109 +++++++++++++++++++++++++++++++++++++------------------------ 1 file changed, 67 insertions(+), 42 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index fe53449fa1b8..b405f81b2c60 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -795,6 +795,54 @@ void lru_add_drain_all(void) } #endif +static void __release_page(struct page *page, struct list_head *pages_to_free) +{ + if (PageCompound(page)) { + __put_compound_page(page); + } else { + /* Clear Active bit in case of parallel mark_page_accessed */ + __ClearPageActive(page); + __ClearPageWaiters(page); + + list_add(&page->lru, pages_to_free); + } +} + +static void __release_lru_pages(struct pagevec *pvec, + struct list_head *pages_to_free) +{ + struct lruvec *lruvec = NULL; + unsigned long flags = 0; + int i; + + /* + * The pagevec at this point should contain a set of pages with + * their reference count at 0 and the LRU flag set. We will now + * need to pull the pages from their LRU lists. + * + * We walk the list backwards here since that way we are starting at + * the pages that should be warmest in the cache. + */ + for (i = pagevec_count(pvec); i--;) { + struct page *page = pvec->pages[i]; + + lruvec = relock_page_lruvec_irqsave(page, lruvec, &flags); + VM_BUG_ON_PAGE(!PageLRU(page), page); + __ClearPageLRU(page); + del_page_from_lru_list(page, lruvec, page_off_lru(page)); + } + + unlock_page_lruvec_irqrestore(lruvec, flags); + + /* + * A batch of pages are no longer on the LRU list. Go through and + * start the final process of returning the deferred pages to their + * appropriate freelists. + */ + for (i = pagevec_count(pvec); i--;) + __release_page(pvec->pages[i], pages_to_free); +} + /** * release_pages - batched put_page() * @pages: array of pages to release @@ -806,32 +854,24 @@ void lru_add_drain_all(void) void release_pages(struct page **pages, int nr) { int i; + struct pagevec pvec; LIST_HEAD(pages_to_free); - struct lruvec *lruvec = NULL; - unsigned long flags; - unsigned int lock_batch; + pagevec_init(&pvec); + + /* + * We need to first walk through the list cleaning up the low hanging + * fruit and clearing those pages that either cannot be freed or that + * are non-LRU. We will store the LRU pages in a pagevec so that we + * can get to them in the next pass. + */ for (i = 0; i < nr; i++) { struct page *page = pages[i]; - /* - * Make sure the IRQ-safe lock-holding time does not get - * excessive with a continuous string of pages from the - * same lruvec. The lock is held only if lruvec != NULL. - */ - if (lruvec && ++lock_batch == SWAP_CLUSTER_MAX) { - unlock_page_lruvec_irqrestore(lruvec, flags); - lruvec = NULL; - } - if (is_huge_zero_page(page)) continue; if (is_zone_device_page(page)) { - if (lruvec) { - unlock_page_lruvec_irqrestore(lruvec, flags); - lruvec = NULL; - } /* * ZONE_DEVICE pages that return 'false' from * put_devmap_managed_page() do not require special @@ -848,36 +888,21 @@ void release_pages(struct page **pages, int nr) if (!put_page_testzero(page)) continue; - if (PageCompound(page)) { - if (lruvec) { - unlock_page_lruvec_irqrestore(lruvec, flags); - lruvec = NULL; - } - __put_compound_page(page); + if (!PageLRU(page)) { + __release_page(page, &pages_to_free); continue; } - if (PageLRU(page)) { - struct lruvec *prev_lruvec = lruvec; - - lruvec = relock_page_lruvec_irqsave(page, lruvec, - &flags); - if (prev_lruvec != lruvec) - lock_batch = 0; - - VM_BUG_ON_PAGE(!PageLRU(page), page); - __ClearPageLRU(page); - del_page_from_lru_list(page, lruvec, page_off_lru(page)); + /* record page so we can get it in the next pass */ + if (!pagevec_add(&pvec, page)) { + __release_lru_pages(&pvec, &pages_to_free); + pagevec_reinit(&pvec); } - - /* Clear Active bit in case of parallel mark_page_accessed */ - __ClearPageActive(page); - __ClearPageWaiters(page); - - list_add(&page->lru, &pages_to_free); } - if (lruvec) - unlock_page_lruvec_irqrestore(lruvec, flags); + + /* flush any remaining LRU pages that need to be processed */ + if (pagevec_count(&pvec)) + __release_lru_pages(&pvec, &pages_to_free); mem_cgroup_uncharge_list(&pages_to_free); free_unref_page_list(&pages_to_free); From patchwork Wed Aug 19 04:27:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 11722503 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8D738739 for ; Wed, 19 Aug 2020 04:27:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4DB3C207BB for ; Wed, 19 Aug 2020 04:27:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Z+DDR3Mm" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4DB3C207BB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7F64A6B0033; Wed, 19 Aug 2020 00:27:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7811E8D000C; Wed, 19 Aug 2020 00:27:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 621F48D000A; Wed, 19 Aug 2020 00:27:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0080.hostedemail.com [216.40.44.80]) by kanga.kvack.org (Postfix) with ESMTP id 45BF16B0033 for ; Wed, 19 Aug 2020 00:27:43 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id EF284362C for ; Wed, 19 Aug 2020 04:27:42 +0000 (UTC) X-FDA: 77166034764.19.trip06_1007adc27025 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin19.hostedemail.com (Postfix) with ESMTP id B35221ACEBC for ; Wed, 19 Aug 2020 04:27:42 +0000 (UTC) X-Spam-Summary: 1,0,0,f324cb7f535cc72e,d41d8cd98f00b204,alexander.duyck@gmail.com,,RULES_HIT:41:69:152:355:379:960:966:973:981:988:989:1260:1277:1311:1313:1314:1345:1359:1431:1437:1515:1516:1518:1535:1544:1593:1594:1605:1711:1730:1747:1777:1792:1801:2194:2196:2199:2200:2393:2559:2562:2691:2693:2898:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4118:4250:4385:4605:5007:6120:6261:6653:6742:7576:8660:8957:9010:9413:9592:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12679:12683:12760:13148:13161:13172:13229:13230:13869:14096:14097:14181:14687:14721:21080:21444:21451:21627:21666:21740:21796:21939:21987:21990:30005:30012:30036:30054:30070,0,RBL:209.85.160.195:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.50.0.100;04y86diep1i4tg4gzcwwjhxdpg8qqocnjkc7oj5rua1tbsep7w6ijj38zcfij9m.qdh7mk3nn3u4sbmrt16aeygpziapoa5qcb6ksf9p8byo31trp9tqnku3opcuf96.r-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,Domain Cache:0, X-HE-Tag: trip06_1007adc27025 X-Filterd-Recvd-Size: 7842 Received: from mail-qt1-f195.google.com (mail-qt1-f195.google.com [209.85.160.195]) by imf15.hostedemail.com (Postfix) with ESMTP for ; Wed, 19 Aug 2020 04:27:42 +0000 (UTC) Received: by mail-qt1-f195.google.com with SMTP id v22so16909390qtq.8 for ; Tue, 18 Aug 2020 21:27:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=taoylAfZ50RvbeR+qDxRIBM5RKgvtzNS5VKzJspUyB8=; b=Z+DDR3MmkmAOpsWWDpyXeYNgIGzTgK9Ddb7UhBT/vJrDAZuGC7IlYskyQk03tzEygi hg0yvwJORukjN33wkxhvlIHH+zvaP6U6lypz2wcfStTBHY8CmTz2F7TgzrpOR/fJm1M7 EFODaa5/wquuu3op5FFYJRBLf5S9RBmRBvBOB54DJqk3ofigrbivIJ1NXhgI5tnxk9EX VX77DTOK8ANaygFl9DWT0sOHN+dtc1RaB7H+Xn9h5/7jP/Epn+84BTlG6LRCCUFBnKKo Ikfqq6/wFEvz43BPxSR6OKSjvmmyvtPFx/O6FVNMW3b4ApIC1B4o5rqHt+Vy/0CELP6F R4ug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=taoylAfZ50RvbeR+qDxRIBM5RKgvtzNS5VKzJspUyB8=; b=O6UyTHps0V0kk8juDpQX6ruKjxtT326szLTJ3fxjWe0pPLHqgIzlcPvdBFmZF8rh5t UWeIbVZEmnNkbylWIpVnyBfciBu0p3RthUKl2FSLNwyJnSWgPHRMiWfakXV02H+/8+ET 28yG851/n3wVpK5T2L1V17ZwPkZmKWDDspHYYxPVQ7jEzh/mplv/bNhl8dO4Bkv6LPaI NKbuAQzzkEsyarMcIysOlTby99eznz2ojqtaFdR5/ZzXN0cXln+glggPMZxzcMVPGnlW SbGGT8uyeb6OPx+4csUZJiybqic5YZzq6kc9cRBWWFIwhkWvsJUjdJ5yHQRCfIRT3ohE 5z3A== X-Gm-Message-State: AOAM531nFmZw+q/vuqrg7uYln4+9GTHM/CtEFx58Uu4f1c3QfXefpLd2 u5VmcbH29XwX8fUcQNqTqp4= X-Google-Smtp-Source: ABdhPJyDanWGHvp3JCIJhebJK8Z+qpciCv8gSBa8JKVjUZllBLCq+m9VK5Y9O/7JNh540/o4GxDybA== X-Received: by 2002:ac8:5146:: with SMTP id h6mr21244001qtn.290.1597811261609; Tue, 18 Aug 2020 21:27:41 -0700 (PDT) Received: from localhost.localdomain ([2001:470:b:9c3:9e5c:8eff:fe4f:f2d0]) by smtp.gmail.com with ESMTPSA id g129sm24061413qkb.39.2020.08.18.21.27.39 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Aug 2020 21:27:41 -0700 (PDT) Subject: [RFC PATCH v2 5/5] mm: Split move_pages_to_lru into 3 separate passes From: Alexander Duyck To: alex.shi@linux.alibaba.com Cc: yang.shi@linux.alibaba.com, lkp@intel.com, rong.a.chen@intel.com, khlebnikov@yandex-team.ru, kirill@shutemov.name, hughd@google.com, linux-kernel@vger.kernel.org, alexander.duyck@gmail.com, daniel.m.jordan@oracle.com, linux-mm@kvack.org, shakeelb@google.com, willy@infradead.org, hannes@cmpxchg.org, tj@kernel.org, cgroups@vger.kernel.org, akpm@linux-foundation.org, richard.weiyang@gmail.com, mgorman@techsingularity.net, iamjoonsoo.kim@lge.com Date: Tue, 18 Aug 2020 21:27:38 -0700 Message-ID: <20200819042738.23414.60815.stgit@localhost.localdomain> In-Reply-To: <20200819041852.23414.95939.stgit@localhost.localdomain> References: <20200819041852.23414.95939.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Rspamd-Queue-Id: B35221ACEBC X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Alexander Duyck The current code for move_pages_to_lru is meant to release the LRU lock every time it encounters an unevictable page or a compound page that must be freed. This results in a fair amount of code bulk because the lruvec has to be reacquired every time the lock is released and reacquired. Instead of doing this I believe we can break the code up into 3 passes. The first pass will identify the pages we can move to LRU and move those. In addition it will sort the list out leaving the unevictable pages in the list and moving those pages that have dropped to a reference count of 0 to pages_to_free. The second pass will return the unevictable pages to the LRU. The final pass will free any compound pages we have in the pages_to_free list before we merge it back with the original list and return from the function. The advantage of doing it this way is that we only have to release the lock between pass 1 and 2, and then we reacquire the lock after pass 3 after we merge the pages_to_free back into the original list. As such we only have to release the lock at most once in an entire call instead of having to test to see if we need to relock with each page. Signed-off-by: Alexander Duyck Reviewed-by: Alex Shi --- mm/vmscan.c | 68 ++++++++++++++++++++++++++++++++++------------------------- 1 file changed, 39 insertions(+), 29 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 3ebe3f9b653b..6a2bdbc1a9eb 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1850,22 +1850,21 @@ static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec, { int nr_pages, nr_moved = 0; LIST_HEAD(pages_to_free); - struct page *page; - struct lruvec *orig_lruvec = lruvec; + struct page *page, *next; enum lru_list lru; - while (!list_empty(list)) { - page = lru_to_page(list); + list_for_each_entry_safe(page, next, list, lru) { VM_BUG_ON_PAGE(PageLRU(page), page); - list_del(&page->lru); - if (unlikely(!page_evictable(page))) { - if (lruvec) { - spin_unlock_irq(&lruvec->lru_lock); - lruvec = NULL; - } - putback_lru_page(page); + + /* + * if page is unevictable leave it on the list to be returned + * to the LRU after we have finished processing the other + * entries in the list. + */ + if (unlikely(!page_evictable(page))) continue; - } + + list_del(&page->lru); /* * The SetPageLRU needs to be kept here for list intergrity. @@ -1878,20 +1877,14 @@ static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec, * list_add(&page->lru,) * list_add(&page->lru,) */ - lruvec = relock_page_lruvec_irq(page, lruvec); SetPageLRU(page); if (unlikely(put_page_testzero(page))) { __ClearPageLRU(page); __ClearPageActive(page); - if (unlikely(PageCompound(page))) { - spin_unlock_irq(&lruvec->lru_lock); - lruvec = NULL; - destroy_compound_page(page); - } else - list_add(&page->lru, &pages_to_free); - + /* defer freeing until we can release lru_lock */ + list_add(&page->lru, &pages_to_free); continue; } @@ -1904,16 +1897,33 @@ static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec, if (PageActive(page)) workingset_age_nonresident(lruvec, nr_pages); } - if (orig_lruvec != lruvec) { - if (lruvec) - spin_unlock_irq(&lruvec->lru_lock); - spin_lock_irq(&orig_lruvec->lru_lock); - } - /* - * To save our caller's stack, now use input list for pages to free. - */ - list_splice(&pages_to_free, list); + if (unlikely(!list_empty(list) || !list_empty(&pages_to_free))) { + spin_unlock_irq(&lruvec->lru_lock); + + /* return any unevictable pages to the LRU list */ + while (!list_empty(list)) { + page = lru_to_page(list); + list_del(&page->lru); + putback_lru_page(page); + } + + /* + * To save our caller's stack use input + * list for pages to free. + */ + list_splice(&pages_to_free, list); + + /* free any compound pages we have in the list */ + list_for_each_entry_safe(page, next, list, lru) { + if (likely(!PageCompound(page))) + continue; + list_del(&page->lru); + destroy_compound_page(page); + } + + spin_lock_irq(&lruvec->lru_lock); + } return nr_moved; }