From patchwork Thu Aug 13 04:02:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 11711651 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DCD82618 for ; Thu, 13 Aug 2020 04:02:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9B01520771 for ; Thu, 13 Aug 2020 04:02:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="p7aKYuuO" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9B01520771 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D7F508D0003; Thu, 13 Aug 2020 00:02:28 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D2DFE8D0001; Thu, 13 Aug 2020 00:02:28 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C1DF68D0003; Thu, 13 Aug 2020 00:02:28 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0121.hostedemail.com [216.40.44.121]) by kanga.kvack.org (Postfix) with ESMTP id A901F8D0001 for ; Thu, 13 Aug 2020 00:02:28 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 69E4A5DCD for ; Thu, 13 Aug 2020 04:02:28 +0000 (UTC) X-FDA: 77144198376.14.sound09_5e0f2dd26ff1 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin14.hostedemail.com (Postfix) with ESMTP id 39D8318229818 for ; Thu, 13 Aug 2020 04:02:28 +0000 (UTC) X-Spam-Summary: 1,0,0,3d989c0059484599,d41d8cd98f00b204,alexander.duyck@gmail.com,,RULES_HIT:41:69:152:355:379:960:966:973:988:989:1260:1277:1311:1313:1314:1345:1359:1431:1437:1515:1516:1518:1535:1543:1593:1594:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3355:3865:3866:3868:3870:3871:3872:4118:4250:4321:4385:5007:6120:6261:6653:6742:7576:7903:8957:9413:9592:10004:10400:11026:11473:11658:11914:12043:12114:12291:12296:12297:12438:12517:12519:12555:12679:12683:12760:12986:13153:13228:13868:14096:14097:14181:14687:14721:21080:21444:21451:21611:21627:21666:21987:30054,0,RBL:209.85.222.195:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100;04yf6ubrjrzztoi3h8cdi36cr899rypwhjqxncq8cgxfzky9w6xyo6znxfhbx46.aendubqkpcuh9c7rrndkwbsfx71xy3kkxtydwzbzi78ytexqnqr84dkmrbxrire.h-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:n one X-HE-Tag: sound09_5e0f2dd26ff1 X-Filterd-Recvd-Size: 7369 Received: from mail-qk1-f195.google.com (mail-qk1-f195.google.com [209.85.222.195]) by imf41.hostedemail.com (Postfix) with ESMTP for ; Thu, 13 Aug 2020 04:02:27 +0000 (UTC) Received: by mail-qk1-f195.google.com with SMTP id 62so4151271qkj.7 for ; Wed, 12 Aug 2020 21:02:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=XtxT1MTxM5ARJ+08eWr2bJ37iwug5sulaz41RNmMU7k=; b=p7aKYuuOYtZex8EmSHqns2ZVG6wtpk4+BJY6vTjOWkw9WYj2eijkenTtU70ZLdXWyE GQrZFDJrpyqGeq6kIv3KfpWbtPJ0thdDxLNyrRk32/WBCYNISKcCW1gOEnlUnynVrMY2 unbACAKixzD21x1yKI8PZY26drlVM4slXcE4ZnTDk6FPDyy+BNskGWk2LNwClTmLWpAu UjT5Bc1jXlMXNnfJWxGeJA8NmE7vREmRF9Qo3Ym76m44t8HmrT4uKcB10SnYiSsdX/cE iy1vSkyE7G/EC5PmTx5uC0TRNuQGC48FnnYA5zPE0xjrCLLM1NZltN3VJwV6S0aaW5+/ OkVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=XtxT1MTxM5ARJ+08eWr2bJ37iwug5sulaz41RNmMU7k=; b=mxnl0kAu5xB/qt2hGMDZ030AlOKaWCCnpXUROruesVgbqfXfNIXL7Lva4vQOgnwT+M jEobkvyZrV7turnnVji2G4RVnqwunbZo/eitlsF/xGJMiTzeyHtt9XFWFz5pL42sKol2 vpgd4hoD+I1zfbJy8hkzydK1ER6T0eZyCH62sbucAELbkPUgn0Y+DbUSnm7PFZLyY50l Oaw0nDU3MEG8Q3J0FjezylvMaxXMri6MfosazqolR6vgbJvzzO6kj3ZPXvUzY69aQ9yC asLmb2PQET5PuGe2NBigYwF87YV+EQOHlCeAQJet+qmpXCAbiv2+p7rx5nEUXMGbnhwt PpSA== X-Gm-Message-State: AOAM532535O+Ph//bHe2N+dL97ltu6AC/UNNc6lryNdA3hkuQAm8eXDM TFaUaXLHwRdbhh9QL9QiIM0= X-Google-Smtp-Source: ABdhPJx3h8V1BGcujSWy01LeHbHKLGD8NBYz+86psVF7eNMM6naaE4zznE3jh+HZ/CMxH6oUQbTbFQ== X-Received: by 2002:a05:620a:10ab:: with SMTP id h11mr2890878qkk.219.1597291347120; Wed, 12 Aug 2020 21:02:27 -0700 (PDT) Received: from localhost.localdomain ([2001:470:b:9c3:9e5c:8eff:fe4f:f2d0]) by smtp.gmail.com with ESMTPSA id s4sm4959859qtn.34.2020.08.12.21.02.24 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 12 Aug 2020 21:02:26 -0700 (PDT) Subject: [RFC PATCH 1/3] mm: Drop locked from isolate_migratepages_block From: Alexander Duyck To: alex.shi@linux.alibaba.com Cc: yang.shi@linux.alibaba.com, lkp@intel.com, rong.a.chen@intel.com, khlebnikov@yandex-team.ru, kirill@shutemov.name, hughd@google.com, linux-kernel@vger.kernel.org, alexander.duyck@gmail.com, daniel.m.jordan@oracle.com, linux-mm@kvack.org, shakeelb@google.com, willy@infradead.org, hannes@cmpxchg.org, tj@kernel.org, cgroups@vger.kernel.org, akpm@linux-foundation.org, richard.weiyang@gmail.com, mgorman@techsingularity.net, iamjoonsoo.kim@lge.com Date: Wed, 12 Aug 2020 21:02:24 -0700 Message-ID: <20200813040224.13054.96724.stgit@localhost.localdomain> In-Reply-To: <20200813035100.13054.25671.stgit@localhost.localdomain> References: <20200813035100.13054.25671.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Rspamd-Queue-Id: 39D8318229818 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Alexander Duyck We can drop the need for the locked variable by making use of the lruvec_holds_page_lru_lock function. By doing this we can avoid some rcu locking ugliness for the case where the lruvec is still holding the LRU lock associated with the page. Instead we can just use the lruvec and if it is NULL we assume the lock was released. Signed-off-by: Alexander Duyck --- mm/compaction.c | 45 ++++++++++++++++++++------------------------- 1 file changed, 20 insertions(+), 25 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index b99c96c4862d..5021a18ef722 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -803,9 +803,8 @@ static bool too_many_isolated(pg_data_t *pgdat) { pg_data_t *pgdat = cc->zone->zone_pgdat; unsigned long nr_scanned = 0, nr_isolated = 0; - struct lruvec *lruvec; + struct lruvec *lruvec = NULL; unsigned long flags = 0; - struct lruvec *locked = NULL; struct page *page = NULL, *valid_page = NULL; unsigned long start_pfn = low_pfn; bool skip_on_failure = false; @@ -866,9 +865,9 @@ static bool too_many_isolated(pg_data_t *pgdat) * a fatal signal is pending. */ if (!(low_pfn % SWAP_CLUSTER_MAX)) { - if (locked) { - unlock_page_lruvec_irqrestore(locked, flags); - locked = NULL; + if (lruvec) { + unlock_page_lruvec_irqrestore(lruvec, flags); + lruvec = NULL; } if (fatal_signal_pending(current)) { @@ -949,9 +948,9 @@ static bool too_many_isolated(pg_data_t *pgdat) */ if (unlikely(__PageMovable(page)) && !PageIsolated(page)) { - if (locked) { - unlock_page_lruvec_irqrestore(locked, flags); - locked = NULL; + if (lruvec) { + unlock_page_lruvec_irqrestore(lruvec, flags); + lruvec = NULL; } if (!isolate_movable_page(page, isolate_mode)) @@ -992,16 +991,13 @@ static bool too_many_isolated(pg_data_t *pgdat) if (!TestClearPageLRU(page)) goto isolate_fail_put; - rcu_read_lock(); - lruvec = mem_cgroup_page_lruvec(page, pgdat); - /* If we already hold the lock, we can skip some rechecking */ - if (lruvec != locked) { - if (locked) - unlock_page_lruvec_irqrestore(locked, flags); + if (!lruvec || !lruvec_holds_page_lru_lock(page, lruvec)) { + if (lruvec) + unlock_page_lruvec_irqrestore(lruvec, flags); + lruvec = mem_cgroup_page_lruvec(page, pgdat); compact_lock_irqsave(&lruvec->lru_lock, &flags, cc); - locked = lruvec; rcu_read_unlock(); lruvec_memcg_debug(lruvec, page); @@ -1023,8 +1019,7 @@ static bool too_many_isolated(pg_data_t *pgdat) SetPageLRU(page); goto isolate_fail_put; } - } else - rcu_read_unlock(); + } /* The whole page is taken off the LRU; skip the tail pages. */ if (PageCompound(page)) @@ -1057,9 +1052,9 @@ static bool too_many_isolated(pg_data_t *pgdat) isolate_fail_put: /* Avoid potential deadlock in freeing page under lru_lock */ - if (locked) { - unlock_page_lruvec_irqrestore(locked, flags); - locked = NULL; + if (lruvec) { + unlock_page_lruvec_irqrestore(lruvec, flags); + lruvec = NULL; } put_page(page); @@ -1073,9 +1068,9 @@ static bool too_many_isolated(pg_data_t *pgdat) * page anyway. */ if (nr_isolated) { - if (locked) { - unlock_page_lruvec_irqrestore(locked, flags); - locked = NULL; + if (lruvec) { + unlock_page_lruvec_irqrestore(lruvec, flags); + lruvec = NULL; } putback_movable_pages(&cc->migratepages); cc->nr_migratepages = 0; @@ -1102,8 +1097,8 @@ static bool too_many_isolated(pg_data_t *pgdat) page = NULL; isolate_abort: - if (locked) - unlock_page_lruvec_irqrestore(locked, flags); + if (lruvec) + unlock_page_lruvec_irqrestore(lruvec, flags); if (page) { SetPageLRU(page); put_page(page); From patchwork Thu Aug 13 04:02:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 11711653 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 552CC618 for ; Thu, 13 Aug 2020 04:02:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 13CA92076C for ; Thu, 13 Aug 2020 04:02:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="SASn4JQn" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 13CA92076C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4CC958D0006; Thu, 13 Aug 2020 00:02:37 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 47CB88D0001; Thu, 13 Aug 2020 00:02:37 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 36BFF8D0006; Thu, 13 Aug 2020 00:02:37 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0027.hostedemail.com [216.40.44.27]) by kanga.kvack.org (Postfix) with ESMTP id 216C48D0001 for ; Thu, 13 Aug 2020 00:02:37 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id D96FD8245578 for ; Thu, 13 Aug 2020 04:02:36 +0000 (UTC) X-FDA: 77144198712.25.bell29_0d15c5c26ff1 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id 9B72E1804E3A1 for ; Thu, 13 Aug 2020 04:02:36 +0000 (UTC) X-Spam-Summary: 1,0,0,5904bd10103952e9,d41d8cd98f00b204,alexander.duyck@gmail.com,,RULES_HIT:41:69:152:273:355:379:960:973:988:989:1260:1277:1311:1313:1314:1345:1359:1431:1437:1515:1516:1518:1535:1544:1593:1594:1605:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:2693:2731:2895:2901:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4118:4250:4321:5007:6119:6120:6261:6653:6742:7576:7903:8660:8957:9413:9592:10004:11026:11232:11658:11914:12043:12296:12297:12438:12517:12519:12555:12679:12683:12760:13141:13148:13161:13229:13230:14096:14097:14181:14687:14721:21063:21080:21444:21451:21611:21627:21666:21740:21795:21939:21990:30012:30051:30054:30056:30070,0,RBL:209.85.222.196:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100;04y8p31pdozxbcw4u8455dby77qhpoc7u7efahfqjuxkyn1wetup17mk1txsqe5.j4xpje95b48zf3b9n6f91by3j4wjxzowz8y6wdy4gbnpnqinohzhu635uwhaxuq.4-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk ,SPF:fp, X-HE-Tag: bell29_0d15c5c26ff1 X-Filterd-Recvd-Size: 7553 Received: from mail-qk1-f196.google.com (mail-qk1-f196.google.com [209.85.222.196]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Thu, 13 Aug 2020 04:02:35 +0000 (UTC) Received: by mail-qk1-f196.google.com with SMTP id m7so4126572qki.12 for ; Wed, 12 Aug 2020 21:02:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=g1Efyw0jo1iBHDaFGnCxXDBpkzz2KS4HPptzP2dpWOE=; b=SASn4JQnilStqGrDLOtz2BFduofMDuiSagfrx9tnHZ71TnmRd67ZdWrSRGdfw9IAGq m9yIwMMEh0ZnxBlWWdbePMh6dEcgXTiWIPxC6fGrDTGYo7BGwbIWQwzY9Jhaq1h4RGIO 0Arw+y0bGAEeM1aYyHrS3Z9BHtyk9Pr+2pPT2rEJCnIlPqpxaCX+EcEyt7FFs1Zxo3Xo fy/zMPyLB9PA4P+zHE4VNKdiRgum8g6zhIjBRU+lJwJ8BAiAlg75lMljBWqa4qUbT9l4 1TyLlqB6Kq8Gfn4JyGDzb0CJ/iU7Cu1i1QISLdmN/pZIL9Mz33xB9MB15pN+tZnGiLAO RQNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=g1Efyw0jo1iBHDaFGnCxXDBpkzz2KS4HPptzP2dpWOE=; b=gpK7Q/OUUdBOU+RGkdFJV7musxvNcttiZD6/c1TXcw7KMvBOZU7EL+x25F3TQMkLeS 3UGX7pnzt7X7QPcrfvNYuSuJ8POtVvVWeIOJEJodQoHkmeo/TTjqoSnNU3T/nYXKr0gD 6z97ipKl4nlc1HjHPP/0QjyJhR4ayPEnqUh0287YnAH2fw2MfouBgUbcQkcbw7jgo2l/ 5BTo5tOrsUk+UVjPA6fgr9V7Emm3eYQ/VDETxKAx1ur+yVy0rtDj1SNzPFgI83J1APN8 eisgrQiQYEs76HUPE7xIeHrJ4M2zdAaqMwr8UX8hxHpr/zL8tIvUaeAOYmjQqQBuG8Ox QK8Q== X-Gm-Message-State: AOAM531QFCNq45DEDc5+nhdQLMFSTYLkzgPgAE2L6SyTQVy/J8kt1Wsc yNFo5arR6Uhl7hEwhg/aPdg= X-Google-Smtp-Source: ABdhPJy/12XScIZblZWf8AE4TR/BGGcbA7se3/6sHFcRW8yveFpWOs/DZ0MnaggImp0MSomoB06k7g== X-Received: by 2002:a05:620a:13c4:: with SMTP id g4mr3081984qkl.142.1597291355359; Wed, 12 Aug 2020 21:02:35 -0700 (PDT) Received: from localhost.localdomain ([2001:470:b:9c3:9e5c:8eff:fe4f:f2d0]) by smtp.gmail.com with ESMTPSA id u39sm5526609qtc.54.2020.08.12.21.02.33 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 12 Aug 2020 21:02:34 -0700 (PDT) Subject: [RFC PATCH 2/3] mm: Drop use of test_and_set_skip in favor of just setting skip From: Alexander Duyck To: alex.shi@linux.alibaba.com Cc: yang.shi@linux.alibaba.com, lkp@intel.com, rong.a.chen@intel.com, khlebnikov@yandex-team.ru, kirill@shutemov.name, hughd@google.com, linux-kernel@vger.kernel.org, alexander.duyck@gmail.com, daniel.m.jordan@oracle.com, linux-mm@kvack.org, shakeelb@google.com, willy@infradead.org, hannes@cmpxchg.org, tj@kernel.org, cgroups@vger.kernel.org, akpm@linux-foundation.org, richard.weiyang@gmail.com, mgorman@techsingularity.net, iamjoonsoo.kim@lge.com Date: Wed, 12 Aug 2020 21:02:32 -0700 Message-ID: <20200813040232.13054.82417.stgit@localhost.localdomain> In-Reply-To: <20200813035100.13054.25671.stgit@localhost.localdomain> References: <20200813035100.13054.25671.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Rspamd-Queue-Id: 9B72E1804E3A1 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Alexander Duyck The only user of test_and_set_skip was isolate_migratepages_block and it was using it after a call that was testing and clearing the LRU flag. As such it really didn't need to be behind the LRU lock anymore as it wasn't really fulfilling its purpose. With that being the case we can simply drop the bit and instead directly just call the set_pageblock_skip function if the page we are working on is the valid_page at the start of the pageblock. It shouldn't be possible for us to encounter the bit being set since we obtained the LRU flag for the first page in the pageblock which means we would have exclusive access to setting the skip bit. As such we don't need to worry about the abort case since no other thread will be able to call what used to be test_and_set_skip. Since we have dropped the late abort case we can drop the code that was clearing the LRU flag and calling page_put since the abort case will now not be holding a reference to a page. Signed-off-by: Alexander Duyck --- mm/compaction.c | 50 +++++++------------------------------------------- 1 file changed, 7 insertions(+), 43 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 5021a18ef722..c1e9918f9dd4 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -399,29 +399,6 @@ void reset_isolation_suitable(pg_data_t *pgdat) } } -/* - * Sets the pageblock skip bit if it was clear. Note that this is a hint as - * locks are not required for read/writers. Returns true if it was already set. - */ -static bool test_and_set_skip(struct compact_control *cc, struct page *page, - unsigned long pfn) -{ - bool skip; - - /* Do no update if skip hint is being ignored */ - if (cc->ignore_skip_hint) - return false; - - if (!IS_ALIGNED(pfn, pageblock_nr_pages)) - return false; - - skip = get_pageblock_skip(page); - if (!skip && !cc->no_set_skip_hint) - skip = !set_pageblock_skip(page); - - return skip; -} - static void update_cached_migrate(struct compact_control *cc, unsigned long pfn) { struct zone *zone = cc->zone; @@ -480,12 +457,6 @@ static inline void update_pageblock_skip(struct compact_control *cc, static void update_cached_migrate(struct compact_control *cc, unsigned long pfn) { } - -static bool test_and_set_skip(struct compact_control *cc, struct page *page, - unsigned long pfn) -{ - return false; -} #endif /* CONFIG_COMPACTION */ /* @@ -895,7 +866,6 @@ static bool too_many_isolated(pg_data_t *pgdat) if (!valid_page && IS_ALIGNED(low_pfn, pageblock_nr_pages)) { if (!cc->ignore_skip_hint && get_pageblock_skip(page)) { low_pfn = end_pfn; - page = NULL; goto isolate_abort; } valid_page = page; @@ -991,6 +961,13 @@ static bool too_many_isolated(pg_data_t *pgdat) if (!TestClearPageLRU(page)) goto isolate_fail_put; + /* Indicate that we want exclusive access to this pageblock */ + if (page == valid_page) { + skip_updated = true; + if (!cc->ignore_skip_hint) + set_pageblock_skip(page); + } + /* If we already hold the lock, we can skip some rechecking */ if (!lruvec || !lruvec_holds_page_lru_lock(page, lruvec)) { if (lruvec) @@ -1002,13 +979,6 @@ static bool too_many_isolated(pg_data_t *pgdat) lruvec_memcg_debug(lruvec, page); - /* Try get exclusive access under lock */ - if (!skip_updated) { - skip_updated = true; - if (test_and_set_skip(cc, page, low_pfn)) - goto isolate_abort; - } - /* * Page become compound since the non-locked check, * and it's on LRU. It can only be a THP so the order @@ -1094,15 +1064,9 @@ static bool too_many_isolated(pg_data_t *pgdat) if (unlikely(low_pfn > end_pfn)) low_pfn = end_pfn; - page = NULL; - isolate_abort: if (lruvec) unlock_page_lruvec_irqrestore(lruvec, flags); - if (page) { - SetPageLRU(page); - put_page(page); - } /* * Updated the cached scanner pfn once the pageblock has been scanned From patchwork Thu Aug 13 04:02:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Duyck X-Patchwork-Id: 11711655 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 37931618 for ; Thu, 13 Aug 2020 04:02:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 04D86207DA for ; Thu, 13 Aug 2020 04:02:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="c4tk2gd6" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 04D86207DA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 409178D0008; Thu, 13 Aug 2020 00:02:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 391EC8D0001; Thu, 13 Aug 2020 00:02:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 25A2E8D0008; Thu, 13 Aug 2020 00:02:45 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0246.hostedemail.com [216.40.44.246]) by kanga.kvack.org (Postfix) with ESMTP id 102E08D0001 for ; Thu, 13 Aug 2020 00:02:45 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id CDF32364D for ; Thu, 13 Aug 2020 04:02:44 +0000 (UTC) X-FDA: 77144199048.12.beds81_3105bc626ff1 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id 9F76A1801A74B for ; Thu, 13 Aug 2020 04:02:44 +0000 (UTC) X-Spam-Summary: 1,0,0,5990dc2a4b994bfb,d41d8cd98f00b204,alexander.duyck@gmail.com,,RULES_HIT:41:69:152:355:379:960:973:988:989:1260:1277:1311:1313:1314:1345:1359:1431:1437:1515:1516:1518:1535:1542:1593:1594:1711:1730:1747:1777:1792:2194:2198:2199:2200:2393:2559:2562:2693:2731:2899:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3872:3874:4117:4250:5007:6119:6120:6261:6653:6742:7576:7903:8957:9010:9413:9592:10004:10400:11026:11232:11658:11914:12043:12114:12297:12438:12517:12519:12555:12679:12683:12760:13161:13229:13255:14096:14097:14181:14687:14721:21080:21444:21451:21627:21666:21939:21990:30054:30056:30069:30070,0,RBL:209.85.222.194:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100;04ygf5jdtwdtf7mttiofuturh1js6opw1dnc9mxk43qg5hyf3idmb7ncr9s4yon.8jzpc9yucu53se8ej3e1wo6xrert8a99fazh3f1jbqfb65cfy99bmwjmssxexo6.s-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rule s:0:0:0, X-HE-Tag: beds81_3105bc626ff1 X-Filterd-Recvd-Size: 6120 Received: from mail-qk1-f194.google.com (mail-qk1-f194.google.com [209.85.222.194]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Thu, 13 Aug 2020 04:02:44 +0000 (UTC) Received: by mail-qk1-f194.google.com with SMTP id d14so4118578qke.13 for ; Wed, 12 Aug 2020 21:02:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=w5SvE1fHI6P73p4d/Ave7K8xWurioQmoju2x6tyh2NI=; b=c4tk2gd6nfdYlP84ldnqXqtP9SL2coSrqYEmasUe0gQZjJ9C0qFxsRqvyD5YIjTIIy xy++8GAuvZBRqgofWHFgrqTguHhoglS5fxYbhD3S3hvBiiusYCIulc73UJJpWoRXNx2H naAhGT9cveMS7FMK01PpSjGi/n3QaC/mEfygXXhEM5Ry8Cm/3wlooFXW+una3QuRDRkO mLIxbRvO8+xXgT5VxrkpJtiEL35TzuV2kQ8SfPbcM+nYBVtnVWPF1th3BoG4zc3dclcI OjI3eBp5/pnZOyMFt7I5xh0/XNv+Sgy2qAVnFDvmydFQk9kFsCcJFfQhriOYNWzDDbvG k18Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=w5SvE1fHI6P73p4d/Ave7K8xWurioQmoju2x6tyh2NI=; b=ccqdh3hmGykuORTyxcuOovIc4BZsl6RT8nlr8KEX+ej0siJJ9KprKRqD70AlX/5Emz WHEQFKNpPG670dBdtUi6ciVIK5FVY/8i9TTFzix2wlDaLx6L9E+1qiy/4CPCKDa2JqRo wx6fH1o9jMeUq3cPEiHLx0ulAtP9lP8oar9lGgsi5Dtcr+pEHfaOVD3/1WieosBOkwih NkGnbli5L06d+Tywx6LbPijY9S6B6uBXa134RhqPrWZP1sji7XdT+OcNXYMev7x/0lAI eR5RON4uKpsPMA5h4SaaAsigt4oc7U5lnurYX7UVRzgoVi+CXOZkwn1M8HyufWtJQykt 2uEg== X-Gm-Message-State: AOAM533ag6rmhaJGxug0gdlQ5imVWyZt3BL1g4HHNl3KFCDfOhXf/L45 EIS3w/zYPeW8hxZ8BRT/sCi+O4v1uqs= X-Google-Smtp-Source: ABdhPJyPijbO2JJyKyc4Hd9Lkz09TGdRjKGbofrL3sAQNVTDFk9hoq+hNTNwhwCNKnBbKpoyYYqOiQ== X-Received: by 2002:a37:8d0:: with SMTP id 199mr2876941qki.335.1597291363538; Wed, 12 Aug 2020 21:02:43 -0700 (PDT) Received: from localhost.localdomain ([2001:470:b:9c3:9e5c:8eff:fe4f:f2d0]) by smtp.gmail.com with ESMTPSA id n128sm4396126qke.8.2020.08.12.21.02.41 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 12 Aug 2020 21:02:43 -0700 (PDT) Subject: [RFC PATCH 3/3] mm: Identify compound pages sooner in isolate_migratepages_block From: Alexander Duyck To: alex.shi@linux.alibaba.com Cc: yang.shi@linux.alibaba.com, lkp@intel.com, rong.a.chen@intel.com, khlebnikov@yandex-team.ru, kirill@shutemov.name, hughd@google.com, linux-kernel@vger.kernel.org, alexander.duyck@gmail.com, daniel.m.jordan@oracle.com, linux-mm@kvack.org, shakeelb@google.com, willy@infradead.org, hannes@cmpxchg.org, tj@kernel.org, cgroups@vger.kernel.org, akpm@linux-foundation.org, richard.weiyang@gmail.com, mgorman@techsingularity.net, iamjoonsoo.kim@lge.com Date: Wed, 12 Aug 2020 21:02:40 -0700 Message-ID: <20200813040240.13054.76770.stgit@localhost.localdomain> In-Reply-To: <20200813035100.13054.25671.stgit@localhost.localdomain> References: <20200813035100.13054.25671.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Rspamd-Queue-Id: 9F76A1801A74B X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Alexander Duyck Since we are holding a reference to the page much sooner in isolate_migratepages_block we could move the PageCompound check out of the LRU locked section and instead just place it after get_page_unless_zero. By doing this we can allow any of the items that might trigger a failure to trigger a failure for the compound page rather than the order 0 page and as a result we should be able to process the pageblock faster. In addition by testing for PageCompound sooner we can avoid having the LRU flag cleared and then reset in the exception case. As a result this should prevent any possible races where another thread might be attempting to pull the LRU pages from the list. Signed-off-by: Alexander Duyck --- mm/compaction.c | 33 ++++++++++++++++++--------------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index c1e9918f9dd4..3803f129fd6a 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -954,6 +954,24 @@ static bool too_many_isolated(pg_data_t *pgdat) if (unlikely(!get_page_unless_zero(page))) goto isolate_fail; + /* + * Page is compound. We know the order before we know if it is + * on the LRU so we cannot assume it is THP. However since the + * page will have the LRU validated shortly we can use the value + * to skip over this page for now or validate the LRU is set and + * then isolate the entire compound page if we are isolating to + * generate a CMA page. + */ + if (PageCompound(page)) { + const unsigned int order = compound_order(page); + + if (likely(order < MAX_ORDER)) + low_pfn += (1UL << order) - 1; + + if (!cc->alloc_contig) + goto isolate_fail_put; + } + if (__isolate_lru_page_prepare(page, isolate_mode) != 0) goto isolate_fail_put; @@ -978,23 +996,8 @@ static bool too_many_isolated(pg_data_t *pgdat) rcu_read_unlock(); lruvec_memcg_debug(lruvec, page); - - /* - * Page become compound since the non-locked check, - * and it's on LRU. It can only be a THP so the order - * is safe to read and it's 0 for tail pages. - */ - if (unlikely(PageCompound(page) && !cc->alloc_contig)) { - low_pfn += compound_nr(page) - 1; - SetPageLRU(page); - goto isolate_fail_put; - } } - /* The whole page is taken off the LRU; skip the tail pages. */ - if (PageCompound(page)) - low_pfn += compound_nr(page) - 1; - /* Successfully isolated */ del_page_from_lru_list(page, lruvec, page_lru(page)); mod_node_page_state(page_pgdat(page),