From patchwork Fri Sep 4 15:14:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11757719 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BB4CF618 for ; Fri, 4 Sep 2020 15:14:56 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7E0692074D for ; Fri, 4 Sep 2020 15:14:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=soleen.com header.i=@soleen.com header.b="VHJ5Cez3" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7E0692074D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CB2336B0080; Fri, 4 Sep 2020 11:14:53 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C62BF6B0081; Fri, 4 Sep 2020 11:14:53 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE33D6B0082; Fri, 4 Sep 2020 11:14:53 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0072.hostedemail.com [216.40.44.72]) by kanga.kvack.org (Postfix) with ESMTP id 94C926B0081 for ; Fri, 4 Sep 2020 11:14:53 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 569A61E00 for ; Fri, 4 Sep 2020 15:14:53 +0000 (UTC) X-FDA: 77225726466.08.face13_381233e270b3 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin08.hostedemail.com (Postfix) with ESMTP id 04EE41819E76B for ; Fri, 4 Sep 2020 15:14:52 +0000 (UTC) X-Spam-Summary: 1,0,0,8ba072d0d9e74cac,d41d8cd98f00b204,pasha.tatashin@soleen.com,,RULES_HIT:41:355:379:541:800:960:966:968:973:988:989:1260:1311:1314:1345:1359:1381:1437:1515:1535:1543:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2553:2559:2562:2693:2898:3138:3139:3140:3141:3142:3355:3865:3866:3867:3868:3870:3871:3872:3874:4117:4321:4385:4605:5007:6119:6261:6653:7875:7903:8660:8784:8957:9010:9121:10004:11026:11658:11914:12043:12048:12295:12296:12297:12438:12517:12519:12555:12679:12895:12986:13148:13161:13229:13230:13255:13894:14093:14096:14181:14394:14721:21080:21094:21323:21444:21451:21627:21740:21939:21987:21990:30034:30054:30070:30090,0,RBL:209.85.222.196:@soleen.com:.lbl8.mailshell.net-66.100.201.201 62.2.0.100;04yrucoxzu444ihk3cf81ad18x7p1opzs68dayw88w81kusngx13dis4e1fgkrm.m46xky97razaoqbn66c8xtda3j7s81p1mzjktkwdx87wozt3wapeu9idnwnkiaf.w-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL :none,Cu X-HE-Tag: face13_381233e270b3 X-Filterd-Recvd-Size: 6801 Received: from mail-qk1-f196.google.com (mail-qk1-f196.google.com [209.85.222.196]) by imf37.hostedemail.com (Postfix) with ESMTP for ; Fri, 4 Sep 2020 15:14:52 +0000 (UTC) Received: by mail-qk1-f196.google.com with SMTP id b14so6607694qkn.4 for ; Fri, 04 Sep 2020 08:14:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=JZfJMp3zvHnMX4z50lv6bmh7AfNeSTMMAnF/ge8iWQ4=; b=VHJ5Cez3Zh2443E1mfYsjZ+8MpLSXNAKfoc4e2M5DNliw2MC18n71/lyCPHipV0WMD Ju0AhOf++WJ/c2/0sL1Wyk1NFOL8JLg2iC4XAc7y/+D0WKJSG5KubnRTJUm/RZZ4jKhg hmKyq4iVz9XzMzOCqYwZV0qaKNryVhmI5C0XfDZciaeHEFcDEG8Phnu285EfE5Yga+IU +jOgKaJgUKiokKz1AWf60SWM+7A0rWYlbP7qzVvCfe6FGhl1P5L4lcmsjnRuujazrRF2 1b3O9908i0yMQVL6vQPX1kyHjrNdp5iy/TFE7u7jMOFq48ulwI/JQH7xh9BJqj3RnFtV NqeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JZfJMp3zvHnMX4z50lv6bmh7AfNeSTMMAnF/ge8iWQ4=; b=iAc3U1EnOQhOrzwZcZpH9Zf2Ui5EFfJinLAr3xhzG3dC7aVQAXC5Ku5qQEQCoFH2VS vHtWLx7URlQHaM7KGgneJ0+pwshZTCq6P0cmTGIy8SBvgw8EDAhe+Yfes45I1TnBbgND xLV3iu8CQ3Tc9FY3wFpF97LsQsPqntJ1mqP1I0K0DBKt1458hisqtLJE1frNuYG3tiVk hTZNNU9tzeYSjbX3ne8K1x5rGuz6o9xZPR4J5qrbOMimIoOaeSf3FR+91vyP7HN6Lruz 4rT4O6Myq+YU4yTKD8DIa8VrDR9neYfKcYnva7X0/ZX+L8DIK9sV7heqh6dKzLIdy9Ke rjmw== X-Gm-Message-State: AOAM531jinHP7kIK52VV/lynEapH5OwidTSMZHsfbw6m+vjt88rY4f4o yQvB8BTgS42opImoedq5XBbWkQ== X-Google-Smtp-Source: ABdhPJw7MDic8qSUBtr4IY+EdB+g1H8j4jBm7crUPhFYDelqxCCMjuL6rAaxQt68VIsp9qLj91vyRw== X-Received: by 2002:a05:620a:145:: with SMTP id e5mr8215251qkn.479.1599232491857; Fri, 04 Sep 2020 08:14:51 -0700 (PDT) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id r21sm4802727qtj.80.2020.09.04.08.14.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Sep 2020 08:14:51 -0700 (PDT) From: Pavel Tatashin To: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, mhocko@suse.com, linux-mm@kvack.org, pasha.tatashin@soleen.com, osalvador@suse.de, richard.weiyang@gmail.com, david@redhat.com, vbabka@suse.cz, rientjes@google.com Subject: [PATCH v3 1/2] mm/memory_hotplug: drain per-cpu pages again during memory offline Date: Fri, 4 Sep 2020 11:14:47 -0400 Message-Id: <20200904151448.100489-2-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200904151448.100489-1-pasha.tatashin@soleen.com> References: <20200904151448.100489-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 04EE41819E76B X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There is a race during page offline that can lead to infinite loop: a page never ends up on a buddy list and __offline_pages() keeps retrying infinitely or until a termination signal is received. Thread#1 - a new process: load_elf_binary begin_new_exec exec_mmap mmput exit_mmap tlb_finish_mmu tlb_flush_mmu release_pages free_unref_page_list free_unref_page_prepare set_pcppage_migratetype(page, migratetype); // Set page->index migration type below MIGRATE_PCPTYPES Thread#2 - hot-removes memory __offline_pages start_isolate_page_range set_migratetype_isolate set_pageblock_migratetype(page, MIGRATE_ISOLATE); Set migration type to MIGRATE_ISOLATE-> set drain_all_pages(zone); // drain per-cpu page lists to buddy allocator. Thread#1 - continue free_unref_page_commit migratetype = get_pcppage_migratetype(page); // get old migration type list_add(&page->lru, &pcp->lists[migratetype]); // add new page to already drained pcp list Thread#2 Never drains pcp again, and therefore gets stuck in the loop. The fix is to try to drain per-cpu lists again after check_pages_isolated_cb() fails. Fixes: c52e75935f8d ("mm: remove extra drain pages on pcp list") Signed-off-by: Pavel Tatashin Cc: stable@vger.kernel.org Acked-by: David Rientjes Acked-by: Vlastimil Babka Acked-by: Michal Hocko Acked-by: David Hildenbrand --- mm/memory_hotplug.c | 14 ++++++++++++++ mm/page_isolation.c | 8 ++++++++ 2 files changed, 22 insertions(+) diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index e9d5ab5d3ca0..b11a269e2356 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1575,6 +1575,20 @@ static int __ref __offline_pages(unsigned long start_pfn, /* check again */ ret = walk_system_ram_range(start_pfn, end_pfn - start_pfn, NULL, check_pages_isolated_cb); + /* + * per-cpu pages are drained in start_isolate_page_range, but if + * there are still pages that are not free, make sure that we + * drain again, because when we isolated range we might + * have raced with another thread that was adding pages to pcp + * list. + * + * Forward progress should be still guaranteed because + * pages on the pcp list can only belong to MOVABLE_ZONE + * because has_unmovable_pages explicitly checks for + * PageBuddy on freed pages on other zones. + */ + if (ret) + drain_all_pages(zone); } while (ret); /* Ok, all of our target is isolated. diff --git a/mm/page_isolation.c b/mm/page_isolation.c index 242c03121d73..63a3db10a8c0 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -170,6 +170,14 @@ __first_valid_page(unsigned long pfn, unsigned long nr_pages) * pageblocks we may have modified and return -EBUSY to caller. This * prevents two threads from simultaneously working on overlapping ranges. * + * Please note that there is no strong synchronization with the page allocator + * either. Pages might be freed while their page blocks are marked ISOLATED. + * In some cases pages might still end up on pcp lists and that would allow + * for their allocation even when they are in fact isolated already. Depending + * on how strong of a guarantee the caller needs drain_all_pages might be needed + * (e.g. __offline_pages will need to call it after check for isolated range for + * a next retry). + * * Return: the number of isolated pageblocks on success and -EBUSY if any part * of range cannot be isolated. */ From patchwork Fri Sep 4 15:14:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11757721 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 054D6618 for ; Fri, 4 Sep 2020 15:14:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B8B1B2074D for ; Fri, 4 Sep 2020 15:14:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=soleen.com header.i=@soleen.com header.b="DJh1krJ2" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B8B1B2074D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 49C8D6B0081; Fri, 4 Sep 2020 11:14:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 402796B0082; Fri, 4 Sep 2020 11:14:55 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2A2A76B0083; Fri, 4 Sep 2020 11:14:55 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0162.hostedemail.com [216.40.44.162]) by kanga.kvack.org (Postfix) with ESMTP id 0F56E6B0081 for ; Fri, 4 Sep 2020 11:14:55 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id CF2CF1730858 for ; Fri, 4 Sep 2020 15:14:54 +0000 (UTC) X-FDA: 77225726508.11.basin50_2e01e2d270b3 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id 9F7AF180F8B81 for ; Fri, 4 Sep 2020 15:14:54 +0000 (UTC) X-Spam-Summary: 1,0,0,c0d41609e17f2de2,d41d8cd98f00b204,pasha.tatashin@soleen.com,,RULES_HIT:2:41:69:355:379:541:800:960:966:968:973:988:989:1260:1311:1314:1345:1359:1381:1437:1515:1535:1605:1606:1730:1747:1777:1792:2196:2199:2393:2553:2559:2562:2693:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4119:4250:4321:4385:5007:6119:6261:6653:7875:7903:8660:8957:9010:9592:10004:11026:11473:11658:11914:12043:12048:12114:12296:12297:12438:12517:12519:12555:12679:12683:12895:12986:13148:13230:13894:14096:14394:21080:21324:21444:21627:21740:21939:21990:30034:30054:30070:30090,0,RBL:209.85.222.195:@soleen.com:.lbl8.mailshell.net-62.2.0.100 66.100.201.201;04yf7zu4bwdenn6g678dhuhpjxqghyc4xazjkjqcukqj1txbumkwgjr97mua5jn.e9xnqugo66soaiocm3y3zsq4ej1uuu6ytn7cs4ou5n9yecx43cc4krbarbpaa6r.c-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:26,LUA_SUMMARY:none X-HE-Tag: basin50_2e01e2d270b3 X-Filterd-Recvd-Size: 8252 Received: from mail-qk1-f195.google.com (mail-qk1-f195.google.com [209.85.222.195]) by imf48.hostedemail.com (Postfix) with ESMTP for ; Fri, 4 Sep 2020 15:14:54 +0000 (UTC) Received: by mail-qk1-f195.google.com with SMTP id g72so6559323qke.8 for ; Fri, 04 Sep 2020 08:14:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=3Ax7bx8bGQgBf4enQQ+1dSxLeyQMe6jS9zSWW4q6utY=; b=DJh1krJ2Yk6Rnt67V78KVKnqi+1eEnz6nxzYBjVPlmoJNPjfglga67CAGD8foPoNcP DUN6UAYvOXud1X7dC+OAACIUO60+e0GIMXtX1EoWf8KBbXksIuRaWKDfbsfeML9+zkQQ UMhsekrqJZqGsYUalJGd3cDwGFpCVqbnTYej6GWGgf7G77U5dNDBuHt212oLfxN6952t 4m6SfUV1JbFW8evros+nv7Gq+WhsQDMdQ1/nN6GIltGD/YeVgARi/5w/q16jWqXKwOSQ c/6MOdJbtKfOvbdqA1C8qiJZRJNvaDGJszysR7LyuLgKdJuo6xFzxz3gw5Xn9YkGoSPS yZtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3Ax7bx8bGQgBf4enQQ+1dSxLeyQMe6jS9zSWW4q6utY=; b=P9PJRJVrqdCXdi/D57f5cfYOutJUD+ecqSj+WYAJLaUnXkV9FtWn1AIyEiTQWx6xHR OK2lyj/s9VnZP4hAWXYdu0WxxUvx8O626Dgku2b2dLXiOvZRkojyrroW53Ajn3IP3EJa uGxklejJ2/eMfRSqkaTPYpTNm5Ikm8t3fLGrGW1jWOsLx+GjvZB+DHRI1esDAcQ8z550 RQfA3ZHt4YAKt/5NflXXMIZfWcvoxjL2ixO4QHBwHhbRV2oAQTzdo5rbF54YBahAvVBH CVp6kL3AduOE2eKMFd+/OsUwJU+4Tzeo1xaljdvui9W4SU6eoIcwpSGN48L+6AVU/YXG bzIw== X-Gm-Message-State: AOAM532SCG3d80G/HxzS+zXeZMgOWAH8xhiPqU6LJpbSacl0mTswYEjQ /XdYRcuA6GAr1gKC3G6REAcpJw== X-Google-Smtp-Source: ABdhPJyssxk1AMfs3g1XHj9xTwmyKkhKlBfUHsqyKTMI5WB3Vrx23rk1+SPiZoJMG1wZ1J735094kQ== X-Received: by 2002:ae9:edc8:: with SMTP id c191mr7988285qkg.226.1599232493403; Fri, 04 Sep 2020 08:14:53 -0700 (PDT) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id r21sm4802727qtj.80.2020.09.04.08.14.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Sep 2020 08:14:52 -0700 (PDT) From: Pavel Tatashin To: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, mhocko@suse.com, linux-mm@kvack.org, pasha.tatashin@soleen.com, osalvador@suse.de, richard.weiyang@gmail.com, david@redhat.com, vbabka@suse.cz, rientjes@google.com Subject: [PATCH v3 2/2] mm: drain per-cpu pages outside of isolate_migratepages_range Date: Fri, 4 Sep 2020 11:14:48 -0400 Message-Id: <20200904151448.100489-3-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200904151448.100489-1-pasha.tatashin@soleen.com> References: <20200904151448.100489-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 9F7AF180F8B81 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It is expensive to drain per-cpu page lists as a thread is started on each CPU, and we are waiting for them to complete. Currently, we drain on every block of pages that is isolated. Instead lets drain once after pages are isolated. For example, when 2G of memory is hot-removed drain is called 16 times, with this change it will be done only one time on average. Signed-off-by: Pavel Tatashin --- mm/memory_hotplug.c | 15 +-------------- mm/page_alloc.c | 2 ++ mm/page_isolation.c | 40 ++++++++++++---------------------------- 3 files changed, 15 insertions(+), 42 deletions(-) diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index b11a269e2356..5a2ed1a94555 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1536,6 +1536,7 @@ static int __ref __offline_pages(unsigned long start_pfn, } do { + drain_all_pages(zone); pfn = start_pfn; do { if (signal_pending(current)) { @@ -1575,20 +1576,6 @@ static int __ref __offline_pages(unsigned long start_pfn, /* check again */ ret = walk_system_ram_range(start_pfn, end_pfn - start_pfn, NULL, check_pages_isolated_cb); - /* - * per-cpu pages are drained in start_isolate_page_range, but if - * there are still pages that are not free, make sure that we - * drain again, because when we isolated range we might - * have raced with another thread that was adding pages to pcp - * list. - * - * Forward progress should be still guaranteed because - * pages on the pcp list can only belong to MOVABLE_ZONE - * because has_unmovable_pages explicitly checks for - * PageBuddy on freed pages on other zones. - */ - if (ret) - drain_all_pages(zone); } while (ret); /* Ok, all of our target is isolated. diff --git a/mm/page_alloc.c b/mm/page_alloc.c index fab5e97dc9ca..6d6a501a103e 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -8462,6 +8462,8 @@ int alloc_contig_range(unsigned long start, unsigned long end, if (ret < 0) return ret; + drain_all_pages(cc.zone); + /* * In case of -EBUSY, we'd like to know which page causes problem. * So, just fall through. test_pages_isolated() has a tracepoint diff --git a/mm/page_isolation.c b/mm/page_isolation.c index 63a3db10a8c0..8dfa6c6c668d 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -19,8 +19,8 @@ static int set_migratetype_isolate(struct page *page, int migratetype, int isol_ { struct page *unmovable = NULL; struct zone *zone; - unsigned long flags; - int ret = -EBUSY; + unsigned long flags, nr_pages; + int ret = -EBUSY, mt; zone = page_zone(page); @@ -39,24 +39,18 @@ static int set_migratetype_isolate(struct page *page, int migratetype, int isol_ * We just check MOVABLE pages. */ unmovable = has_unmovable_pages(zone, page, migratetype, isol_flags); - if (!unmovable) { - unsigned long nr_pages; - int mt = get_pageblock_migratetype(page); - - set_pageblock_migratetype(page, MIGRATE_ISOLATE); - zone->nr_isolate_pageblock++; - nr_pages = move_freepages_block(zone, page, MIGRATE_ISOLATE, - NULL); - - __mod_zone_freepage_state(zone, -nr_pages, mt); - ret = 0; - } + if (unmovable) + goto out; + mt = get_pageblock_migratetype(page); + set_pageblock_migratetype(page, MIGRATE_ISOLATE); + zone->nr_isolate_pageblock++; + nr_pages = move_freepages_block(zone, page, MIGRATE_ISOLATE, NULL); + __mod_zone_freepage_state(zone, -nr_pages, mt); + ret = 0; out: spin_unlock_irqrestore(&zone->lock, flags); - if (!ret) { - drain_all_pages(zone); - } else { + if (ret) { WARN_ON_ONCE(zone_idx(zone) == ZONE_MOVABLE); if ((isol_flags & REPORT_FAILURE) && unmovable) @@ -170,14 +164,6 @@ __first_valid_page(unsigned long pfn, unsigned long nr_pages) * pageblocks we may have modified and return -EBUSY to caller. This * prevents two threads from simultaneously working on overlapping ranges. * - * Please note that there is no strong synchronization with the page allocator - * either. Pages might be freed while their page blocks are marked ISOLATED. - * In some cases pages might still end up on pcp lists and that would allow - * for their allocation even when they are in fact isolated already. Depending - * on how strong of a guarantee the caller needs drain_all_pages might be needed - * (e.g. __offline_pages will need to call it after check for isolated range for - * a next retry). - * * Return: the number of isolated pageblocks on success and -EBUSY if any part * of range cannot be isolated. */ @@ -192,9 +178,7 @@ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, BUG_ON(!IS_ALIGNED(start_pfn, pageblock_nr_pages)); BUG_ON(!IS_ALIGNED(end_pfn, pageblock_nr_pages)); - for (pfn = start_pfn; - pfn < end_pfn; - pfn += pageblock_nr_pages) { + for (pfn = start_pfn; pfn < end_pfn; pfn += pageblock_nr_pages) { page = __first_valid_page(pfn, pageblock_nr_pages); if (page) { if (set_migratetype_isolate(page, migratetype, flags)) {