From patchwork Thu Aug 13 04:02:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 11711655 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 37931618 for ; Thu, 13 Aug 2020 04:02:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 04D86207DA for ; Thu, 13 Aug 2020 04:02:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="c4tk2gd6" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 04D86207DA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 409178D0008; Thu, 13 Aug 2020 00:02:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 391EC8D0001; Thu, 13 Aug 2020 00:02:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 25A2E8D0008; Thu, 13 Aug 2020 00:02:45 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0246.hostedemail.com [216.40.44.246]) by kanga.kvack.org (Postfix) with ESMTP id 102E08D0001 for ; Thu, 13 Aug 2020 00:02:45 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id CDF32364D for ; Thu, 13 Aug 2020 04:02:44 +0000 (UTC) X-FDA: 77144199048.12.beds81_3105bc626ff1 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id 9F76A1801A74B for ; Thu, 13 Aug 2020 04:02:44 +0000 (UTC) X-Spam-Summary: 1,0,0,5990dc2a4b994bfb,d41d8cd98f00b204,alexander.duyck@gmail.com,,RULES_HIT:41:69:152:355:379:960:973:988:989:1260:1277:1311:1313:1314:1345:1359:1431:1437:1515:1516:1518:1535:1542:1593:1594:1711:1730:1747:1777:1792:2194:2198:2199:2200:2393:2559:2562:2693:2731:2899:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3872:3874:4117:4250:5007:6119:6120:6261:6653:6742:7576:7903:8957:9010:9413:9592:10004:10400:11026:11232:11658:11914:12043:12114:12297:12438:12517:12519:12555:12679:12683:12760:13161:13229:13255:14096:14097:14181:14687:14721:21080:21444:21451:21627:21666:21939:21990:30054:30056:30069:30070,0,RBL:209.85.222.194:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100;04ygf5jdtwdtf7mttiofuturh1js6opw1dnc9mxk43qg5hyf3idmb7ncr9s4yon.8jzpc9yucu53se8ej3e1wo6xrert8a99fazh3f1jbqfb65cfy99bmwjmssxexo6.s-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rule s:0:0:0, X-HE-Tag: beds81_3105bc626ff1 X-Filterd-Recvd-Size: 6120 Received: from mail-qk1-f194.google.com (mail-qk1-f194.google.com [209.85.222.194]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Thu, 13 Aug 2020 04:02:44 +0000 (UTC) Received: by mail-qk1-f194.google.com with SMTP id d14so4118578qke.13 for ; Wed, 12 Aug 2020 21:02:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=w5SvE1fHI6P73p4d/Ave7K8xWurioQmoju2x6tyh2NI=; b=c4tk2gd6nfdYlP84ldnqXqtP9SL2coSrqYEmasUe0gQZjJ9C0qFxsRqvyD5YIjTIIy xy++8GAuvZBRqgofWHFgrqTguHhoglS5fxYbhD3S3hvBiiusYCIulc73UJJpWoRXNx2H naAhGT9cveMS7FMK01PpSjGi/n3QaC/mEfygXXhEM5Ry8Cm/3wlooFXW+una3QuRDRkO mLIxbRvO8+xXgT5VxrkpJtiEL35TzuV2kQ8SfPbcM+nYBVtnVWPF1th3BoG4zc3dclcI OjI3eBp5/pnZOyMFt7I5xh0/XNv+Sgy2qAVnFDvmydFQk9kFsCcJFfQhriOYNWzDDbvG k18Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=w5SvE1fHI6P73p4d/Ave7K8xWurioQmoju2x6tyh2NI=; b=ccqdh3hmGykuORTyxcuOovIc4BZsl6RT8nlr8KEX+ej0siJJ9KprKRqD70AlX/5Emz WHEQFKNpPG670dBdtUi6ciVIK5FVY/8i9TTFzix2wlDaLx6L9E+1qiy/4CPCKDa2JqRo wx6fH1o9jMeUq3cPEiHLx0ulAtP9lP8oar9lGgsi5Dtcr+pEHfaOVD3/1WieosBOkwih NkGnbli5L06d+Tywx6LbPijY9S6B6uBXa134RhqPrWZP1sji7XdT+OcNXYMev7x/0lAI eR5RON4uKpsPMA5h4SaaAsigt4oc7U5lnurYX7UVRzgoVi+CXOZkwn1M8HyufWtJQykt 2uEg== X-Gm-Message-State: AOAM533ag6rmhaJGxug0gdlQ5imVWyZt3BL1g4HHNl3KFCDfOhXf/L45 EIS3w/zYPeW8hxZ8BRT/sCi+O4v1uqs= X-Google-Smtp-Source: ABdhPJyPijbO2JJyKyc4Hd9Lkz09TGdRjKGbofrL3sAQNVTDFk9hoq+hNTNwhwCNKnBbKpoyYYqOiQ== X-Received: by 2002:a37:8d0:: with SMTP id 199mr2876941qki.335.1597291363538; Wed, 12 Aug 2020 21:02:43 -0700 (PDT) Received: from localhost.localdomain ([2001:470:b:9c3:9e5c:8eff:fe4f:f2d0]) by smtp.gmail.com with ESMTPSA id n128sm4396126qke.8.2020.08.12.21.02.41 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 12 Aug 2020 21:02:43 -0700 (PDT) Subject: [RFC PATCH 3/3] mm: Identify compound pages sooner in isolate_migratepages_block From: Alexander Duyck To: alex.shi@linux.alibaba.com Cc: yang.shi@linux.alibaba.com, lkp@intel.com, rong.a.chen@intel.com, khlebnikov@yandex-team.ru, kirill@shutemov.name, hughd@google.com, linux-kernel@vger.kernel.org, alexander.duyck@gmail.com, daniel.m.jordan@oracle.com, linux-mm@kvack.org, shakeelb@google.com, willy@infradead.org, hannes@cmpxchg.org, tj@kernel.org, cgroups@vger.kernel.org, akpm@linux-foundation.org, richard.weiyang@gmail.com, mgorman@techsingularity.net, iamjoonsoo.kim@lge.com Date: Wed, 12 Aug 2020 21:02:40 -0700 Message-ID: <20200813040240.13054.76770.stgit@localhost.localdomain> In-Reply-To: <20200813035100.13054.25671.stgit@localhost.localdomain> References: <20200813035100.13054.25671.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Rspamd-Queue-Id: 9F76A1801A74B X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Alexander Duyck Since we are holding a reference to the page much sooner in isolate_migratepages_block we could move the PageCompound check out of the LRU locked section and instead just place it after get_page_unless_zero. By doing this we can allow any of the items that might trigger a failure to trigger a failure for the compound page rather than the order 0 page and as a result we should be able to process the pageblock faster. In addition by testing for PageCompound sooner we can avoid having the LRU flag cleared and then reset in the exception case. As a result this should prevent any possible races where another thread might be attempting to pull the LRU pages from the list. Signed-off-by: Alexander Duyck --- mm/compaction.c | 33 ++++++++++++++++++--------------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index c1e9918f9dd4..3803f129fd6a 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -954,6 +954,24 @@ static bool too_many_isolated(pg_data_t *pgdat) if (unlikely(!get_page_unless_zero(page))) goto isolate_fail; + /* + * Page is compound. We know the order before we know if it is + * on the LRU so we cannot assume it is THP. However since the + * page will have the LRU validated shortly we can use the value + * to skip over this page for now or validate the LRU is set and + * then isolate the entire compound page if we are isolating to + * generate a CMA page. + */ + if (PageCompound(page)) { + const unsigned int order = compound_order(page); + + if (likely(order < MAX_ORDER)) + low_pfn += (1UL << order) - 1; + + if (!cc->alloc_contig) + goto isolate_fail_put; + } + if (__isolate_lru_page_prepare(page, isolate_mode) != 0) goto isolate_fail_put; @@ -978,23 +996,8 @@ static bool too_many_isolated(pg_data_t *pgdat) rcu_read_unlock(); lruvec_memcg_debug(lruvec, page); - - /* - * Page become compound since the non-locked check, - * and it's on LRU. It can only be a THP so the order - * is safe to read and it's 0 for tail pages. - */ - if (unlikely(PageCompound(page) && !cc->alloc_contig)) { - low_pfn += compound_nr(page) - 1; - SetPageLRU(page); - goto isolate_fail_put; - } } - /* The whole page is taken off the LRU; skip the tail pages. */ - if (PageCompound(page)) - low_pfn += compound_nr(page) - 1; - /* Successfully isolated */ del_page_from_lru_list(page, lruvec, page_lru(page)); mod_node_page_state(page_pgdat(page),