From patchwork Thu Jan 14 10:35:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12019189 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01602C433DB for ; Thu, 14 Jan 2021 10:38:33 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 36D0123A3B for ; Thu, 14 Jan 2021 10:38:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 36D0123A3B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AA33C8D00CF; Thu, 14 Jan 2021 05:38:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A547A8D008E; Thu, 14 Jan 2021 05:38:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 969C38D00CF; Thu, 14 Jan 2021 05:38:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0141.hostedemail.com [216.40.44.141]) by kanga.kvack.org (Postfix) with ESMTP id 7F0108D008E for ; Thu, 14 Jan 2021 05:38:31 -0500 (EST) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 4D4281EE6 for ; Thu, 14 Jan 2021 10:38:31 +0000 (UTC) X-FDA: 77704031622.23.brush45_3f03a7827526 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin23.hostedemail.com (Postfix) with ESMTP id 2F78C3760C for ; Thu, 14 Jan 2021 10:38:31 +0000 (UTC) X-HE-Tag: brush45_3f03a7827526 X-Filterd-Recvd-Size: 4571 Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) by imf45.hostedemail.com (Postfix) with ESMTP for ; Thu, 14 Jan 2021 10:38:30 +0000 (UTC) Received: by mail-pj1-f51.google.com with SMTP id y12so2861980pji.1 for ; Thu, 14 Jan 2021 02:38:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=wusSjLYakm6+Z6kCpNRodcZYujbXCjpbVczSV3qSR3o=; b=TeiTKCMIOIJRw17QVnoSIfYza5zvzuJdQpuxOI/464cijUjiNyu9mYgydAdL2B4UTe iQsMS6mFYA/efN+W5hnMZNG8aMj4JvuHTV7zw6AQKNhNvs0V4QgCd2+KxV625KInYCvg sav6A4SdKh+/IXh4Rr0xQlxguGV5TgalxHSfo1UKCd/mt4Y7kPluOLu2gC/6KBSu9ApL S5w6wcKuli5Jq3UOCcpxq8d2jeKdawel/SjY9qoL8gnZVy9R03OXV5Atof1m2f/Grey+ crisfJEUxwDLr5TjQ2mNuvgbbRKT1TnEkyPaxMMF9HlTXV63JXWnd0Ci+vemx6gbqoME 27Bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wusSjLYakm6+Z6kCpNRodcZYujbXCjpbVczSV3qSR3o=; b=Jx2eb3kzEuoO9jkS1Wn7rSYJdx6ZonTt1xdtX35KCFc7XxOHuUIPM0IvM4G0u2Ufr2 KsxVOFBaLJied2vrYztZq+l/kPjS+FqnfRYXHnDU6t54XQMvprXrAJjWe9MeSllqXiFq /YkPHjgJA8hDRJ20HJnNpk9z4C+xRFDNdhK14ND9lteDzaRuEJXYBcDCxSinc7pyIRlT tshEtSt/J+FXED1ObXUxrgHcZYmrKrVgogUgW5gS+VKXEx4AqtM/aPSKev58cIDB7gKF ONWnSay4FVbElHpfeOr6xql3B8oSZH2qQnknQJnyst3In2WyROMKpb7MyUMVIujIuDye iVJQ== X-Gm-Message-State: AOAM533EK28T80aXuzcDu7olPv7T5BTzVNiCNv4zh3LSFvYBosAXNIps tmQDQ3qII1MAsNBLXo/oSTy4oA== X-Google-Smtp-Source: ABdhPJxZ8FcQdJ5BlixmjzNl7BHRg+e95EAUD1OOhclQznhvVWpOnMMnx+m4cvx0MnAtDvcX89YDGQ== X-Received: by 2002:a17:90a:6c22:: with SMTP id x31mr1617999pjj.196.1610620709633; Thu, 14 Jan 2021 02:38:29 -0800 (PST) Received: from localhost.localdomain ([139.177.225.247]) by smtp.gmail.com with ESMTPSA id l12sm4970112pjq.7.2021.01.14.02.38.25 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Jan 2021 02:38:29 -0800 (PST) From: Muchun Song To: mike.kravetz@oracle.com, akpm@linux-foundation.org Cc: n-horiguchi@ah.jp.nec.com, ak@linux.intel.com, mhocko@suse.cz, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Yang Shi , Michal Hocko , David Hildenbrand , Oscar Salvador Subject: [PATCH v5 1/5] mm: migrate: do not migrate HugeTLB page whose refcount is one Date: Thu, 14 Jan 2021 18:35:11 +0800 Message-Id: <20210114103515.12955-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210114103515.12955-1-songmuchun@bytedance.com> References: <20210114103515.12955-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: All pages isolated for the migration have an elevated reference count and therefore seeing a reference count equal to 1 means that the last user of the page has dropped the reference and the page has became unused and there doesn't make much sense to migrate it anymore. This has been done for regular pages and this patch does the same for hugetlb pages. Although the likelyhood of the race is rather small for hugetlb pages it makes sense the two code paths in sync. Signed-off-by: Muchun Song Reviewed-by: Mike Kravetz Acked-by: Yang Shi Acked-by: Michal Hocko Reviewed-by: David Hildenbrand Reviewed-by: Oscar Salvador --- mm/migrate.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/migrate.c b/mm/migrate.c index 4385f2fb5d18..a6631c4eb6a6 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1279,6 +1279,12 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, return -ENOSYS; } + if (page_count(hpage) == 1) { + /* page was freed from under us. So we are done. */ + putback_active_hugepage(hpage); + return MIGRATEPAGE_SUCCESS; + } + new_hpage = get_new_page(hpage, private); if (!new_hpage) return -ENOMEM; From patchwork Thu Jan 14 10:35:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12019191 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 152F7C433E6 for ; Thu, 14 Jan 2021 10:38:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4A86523A50 for ; Thu, 14 Jan 2021 10:38:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4A86523A50 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AA4298D00D0; Thu, 14 Jan 2021 05:38:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A79E78D008E; Thu, 14 Jan 2021 05:38:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9696F8D00D0; Thu, 14 Jan 2021 05:38:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0160.hostedemail.com [216.40.44.160]) by kanga.kvack.org (Postfix) with ESMTP id 7CEA58D008E for ; Thu, 14 Jan 2021 05:38:36 -0500 (EST) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 3360E181AC9CB for ; Thu, 14 Jan 2021 10:38:36 +0000 (UTC) X-FDA: 77704031832.12.fork42_020b67627526 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id 06CE718027F8D for ; Thu, 14 Jan 2021 10:38:36 +0000 (UTC) X-HE-Tag: fork42_020b67627526 X-Filterd-Recvd-Size: 5406 Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by imf36.hostedemail.com (Postfix) with ESMTP for ; Thu, 14 Jan 2021 10:38:35 +0000 (UTC) Received: by mail-pf1-f179.google.com with SMTP id x126so3089631pfc.7 for ; Thu, 14 Jan 2021 02:38:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=en8qVXfeyVDqIIP8px6ejCNcDJPI30a78TjFLPR7Emo=; b=Bb7SJUlOhKJC1ZzQyZRdMOsjlGvbnabYiGk73Uj/BFVF0fB4N9cHrQZRz2Ef0yCHtz E0ImuUx5/qPYjt+BO6+mt+i77HasP85vbfStHl0AMkjYZqvfQSCxTA3TCeXYPLtQ4PdS wXUz0Vic6sDxivwltEBJrz0GNebiJ66Ifr5RSYWWTDZRSXY1mLyQS2MVg9LAIwfSmSIB ybUqD2JIN8GSVmVAYNSBQysheqB+R0Jvdgx6W74b6s7lDL0K76LNty3kEJEGvFMEPns6 PWIZ0FhmKAbsQgfRtHki3p7AhVx6ukdkvuhjAEY2WIARmoHDLMLmhZAA8T6SGZVfeqmz JyqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=en8qVXfeyVDqIIP8px6ejCNcDJPI30a78TjFLPR7Emo=; b=L2k22u+88Hp6DYfbfmbk7130vlsFHlL1O3nsH60GKuf4ae90i6IKhsj6XBuz/fWYUC HI8wGnKhhdN51TWYiC1Ck9AzZJc+WA5wLgbHTYQe07ADTMSrh0bl3Uki2jjo8qavB9eq R5lFxq2nonbdPMYzikDlvtIhhEB0TG9gNrajLLZ+xauMV7WePmoG8NLTv9UgR6YCyBse Um5DHDSOtQ6qKwtTftCtojwotyWVy4nMmYjaJflZASwPf36ufgyUqpgvQtz3PTV781HT 6py6aOBEbNYgCbTRRLJebJ11/pZf0zAb0AM+uJQespmO+j/+X517idKpxtrX7yJZCwvV cb4g== X-Gm-Message-State: AOAM5337cxC18SdsEdqwrP7njNschPBhpJt1s4cZ9ffkQ4h0IIAVU+oH axwIusFilOLtpp1XFn0F+ilW7w== X-Google-Smtp-Source: ABdhPJzYhufPXUdWQ/iuHCLrwoMhWfXjHdW3SAj4YRCzZdLGSVm2HIBKmnA9RKGGOzKXrViUuR2j/w== X-Received: by 2002:a63:7904:: with SMTP id u4mr6863861pgc.41.1610620714482; Thu, 14 Jan 2021 02:38:34 -0800 (PST) Received: from localhost.localdomain ([139.177.225.247]) by smtp.gmail.com with ESMTPSA id l12sm4970112pjq.7.2021.01.14.02.38.30 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Jan 2021 02:38:33 -0800 (PST) From: Muchun Song To: mike.kravetz@oracle.com, akpm@linux-foundation.org Cc: n-horiguchi@ah.jp.nec.com, ak@linux.intel.com, mhocko@suse.cz, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Michal Hocko , Oscar Salvador , stable@vger.kernel.org Subject: [PATCH v5 2/5] mm: hugetlbfs: fix cannot migrate the fallocated HugeTLB page Date: Thu, 14 Jan 2021 18:35:12 +0800 Message-Id: <20210114103515.12955-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210114103515.12955-1-songmuchun@bytedance.com> References: <20210114103515.12955-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If a new hugetlb page is allocated during fallocate it will not be marked as active (set_page_huge_active) which will result in a later isolate_huge_page failure when the page migration code would like to move that page. Such a failure would be unexpected and wrong. Only export set_page_huge_active, just leave clear_page_huge_active as static. Because there are no external users. Fixes: 70c3547e36f5 (hugetlbfs: add hugetlbfs_fallocate()) Signed-off-by: Muchun Song Acked-by: Michal Hocko Reviewed-by: Mike Kravetz Reviewed-by: Oscar Salvador Cc: stable@vger.kernel.org --- fs/hugetlbfs/inode.c | 3 ++- include/linux/hugetlb.h | 2 ++ mm/hugetlb.c | 2 +- 3 files changed, 5 insertions(+), 2 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index b5c109703daa..21c20fd5f9ee 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -735,9 +735,10 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, mutex_unlock(&hugetlb_fault_mutex_table[hash]); + set_page_huge_active(page); /* * unlock_page because locked by add_to_page_cache() - * page_put due to reference from alloc_huge_page() + * put_page() due to reference from alloc_huge_page() */ unlock_page(page); put_page(page); diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index ebca2ef02212..b5807f23caf8 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -770,6 +770,8 @@ static inline void huge_ptep_modify_prot_commit(struct vm_area_struct *vma, } #endif +void set_page_huge_active(struct page *page); + #else /* CONFIG_HUGETLB_PAGE */ struct hstate {}; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 1f3bf1710b66..4741d60f8955 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1348,7 +1348,7 @@ bool page_huge_active(struct page *page) } /* never called for tail page */ -static void set_page_huge_active(struct page *page) +void set_page_huge_active(struct page *page) { VM_BUG_ON_PAGE(!PageHeadHuge(page), page); SetPagePrivate(&page[1]); From patchwork Thu Jan 14 10:35:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12019193 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DB60C433E0 for ; Thu, 14 Jan 2021 10:38:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C8E4A23A3B for ; Thu, 14 Jan 2021 10:38:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C8E4A23A3B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 59A198D00D1; Thu, 14 Jan 2021 05:38:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 54CF98D008E; Thu, 14 Jan 2021 05:38:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 411948D00D1; Thu, 14 Jan 2021 05:38:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0127.hostedemail.com [216.40.44.127]) by kanga.kvack.org (Postfix) with ESMTP id 2B8DF8D008E for ; Thu, 14 Jan 2021 05:38:42 -0500 (EST) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id E32AB1EE6 for ; Thu, 14 Jan 2021 10:38:41 +0000 (UTC) X-FDA: 77704032042.12.beds52_110827627526 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id BED081802B54C for ; Thu, 14 Jan 2021 10:38:41 +0000 (UTC) X-HE-Tag: beds52_110827627526 X-Filterd-Recvd-Size: 7099 Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Thu, 14 Jan 2021 10:38:41 +0000 (UTC) Received: by mail-pf1-f177.google.com with SMTP id a188so3084846pfa.11 for ; Thu, 14 Jan 2021 02:38:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xjSQX/DBq/dGDHDtI/eYRgdlVUdpJ6hklm+hnz97cjg=; b=EvUu9erDPTKjFMQD3ILkX2jgicEwa6KcpRq1JdSNIbVE1ROOWJ8eAvMmG2oczZokaA BDy8DaYvMSLKYCsF8qHjNPQg1UYIqEby4XLJrII5shLRtOR6/2ZEFEIkC1NClk7Wkqlw 2cFMt5Cqwq5hILU44hNKzKCCpS2pr7faUepyWgoMhhd5wYjLAAwuXX6fScFUkqLERxNK LPnOPS9afJYFuW3gu6hjFsMjgaC7H8sk/TRwOB3awiY0PXJsQphqAabSLRRWV8pPjJca 9ab7sQEpz6wb7RroZWJAgIZ3hoQ9H78mGwKbb3UPuP6nwGZ7TwAMyNR4pn9yplaRHjsS me/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xjSQX/DBq/dGDHDtI/eYRgdlVUdpJ6hklm+hnz97cjg=; b=RSIhW1GqArKLdo2K4BxGNsMcBTIAQ/8U0kckPZDJ28/ibTOV8nqQNLq2jAyoWwqanW Qjk2D1DWlwAELhuFr07LAarTXNGTxT4LYbnPg3NQiPkeYK6YzZA5MCc/u2GK/RxXQq6L CfNf0OJWx4zWY1U2lNEYf/DM8ZmtHqGLM7DGKxiMDxkarOKMapt2vb3rtv728ZtoWxO4 jpFfqk6NJv5dspkWrGuKiDHERXXhQ2AACw4Bv8GzYxFKZRfOaj52gZKQSJ39o6AVX5I2 vxrxu446U9gY/gRmxt9t15n3q8V2BuKsI7eRkBal1pMCFrjvmoY3GlraWBqYxelh+2GS 12IQ== X-Gm-Message-State: AOAM533wsWB/QRBvS+iqpdhmcDsLRJt9DSsC/9RSy90HqqYCava/MGXK MRZsiMqOR+7erp8IntNa4XYH/g== X-Google-Smtp-Source: ABdhPJzETpjNokcaByGnWQuYbXi9vblyTMKU1rIC+MG3Lq3WcuLgGvyZScNVoVyzG4ltBJHwt445WA== X-Received: by 2002:a63:50a:: with SMTP id 10mr6899209pgf.273.1610620718661; Thu, 14 Jan 2021 02:38:38 -0800 (PST) Received: from localhost.localdomain ([139.177.225.247]) by smtp.gmail.com with ESMTPSA id l12sm4970112pjq.7.2021.01.14.02.38.34 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Jan 2021 02:38:38 -0800 (PST) From: Muchun Song To: mike.kravetz@oracle.com, akpm@linux-foundation.org Cc: n-horiguchi@ah.jp.nec.com, ak@linux.intel.com, mhocko@suse.cz, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , stable@vger.kernel.org Subject: [PATCH v5 3/5] mm: hugetlb: fix a race between freeing and dissolving the page Date: Thu, 14 Jan 2021 18:35:13 +0800 Message-Id: <20210114103515.12955-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210114103515.12955-1-songmuchun@bytedance.com> References: <20210114103515.12955-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There is a race condition between __free_huge_page() and dissolve_free_huge_page(). CPU0: CPU1: // page_count(page) == 1 put_page(page) __free_huge_page(page) dissolve_free_huge_page(page) spin_lock(&hugetlb_lock) // PageHuge(page) && !page_count(page) update_and_free_page(page) // page is freed to the buddy spin_unlock(&hugetlb_lock) spin_lock(&hugetlb_lock) clear_page_huge_active(page) enqueue_huge_page(page) // It is wrong, the page is already freed spin_unlock(&hugetlb_lock) The race windows is between put_page() and dissolve_free_huge_page(). We should make sure that the page is already on the free list when it is dissolved. Fixes: c8721bbbdd36 ("mm: memory-hotplug: enable memory hotplug to handle hugepage") Signed-off-by: Muchun Song Reviewed-by: Mike Kravetz Cc: stable@vger.kernel.org --- mm/hugetlb.c | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 4741d60f8955..1b789d1fd06b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -79,6 +79,21 @@ DEFINE_SPINLOCK(hugetlb_lock); static int num_fault_mutexes; struct mutex *hugetlb_fault_mutex_table ____cacheline_aligned_in_smp; +static inline bool PageHugeFreed(struct page *head) +{ + return page_private(head + 4) == -1UL; +} + +static inline void SetPageHugeFreed(struct page *head) +{ + set_page_private(head + 4, -1UL); +} + +static inline void ClearPageHugeFreed(struct page *head) +{ + set_page_private(head + 4, 0); +} + /* Forward declaration */ static int hugetlb_acct_memory(struct hstate *h, long delta); @@ -1028,6 +1043,7 @@ static void enqueue_huge_page(struct hstate *h, struct page *page) list_move(&page->lru, &h->hugepage_freelists[nid]); h->free_huge_pages++; h->free_huge_pages_node[nid]++; + SetPageHugeFreed(page); } static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) @@ -1044,6 +1060,7 @@ static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) list_move(&page->lru, &h->hugepage_activelist); set_page_refcounted(page); + ClearPageHugeFreed(page); h->free_huge_pages--; h->free_huge_pages_node[nid]--; return page; @@ -1504,6 +1521,7 @@ static void prep_new_huge_page(struct hstate *h, struct page *page, int nid) spin_lock(&hugetlb_lock); h->nr_huge_pages++; h->nr_huge_pages_node[nid]++; + ClearPageHugeFreed(page); spin_unlock(&hugetlb_lock); } @@ -1754,6 +1772,7 @@ int dissolve_free_huge_page(struct page *page) { int rc = -EBUSY; +retry: /* Not to disrupt normal path by vainly holding hugetlb_lock */ if (!PageHuge(page)) return 0; @@ -1770,6 +1789,28 @@ int dissolve_free_huge_page(struct page *page) int nid = page_to_nid(head); if (h->free_huge_pages - h->resv_huge_pages == 0) goto out; + + /* + * We should make sure that the page is already on the free list + * when it is dissolved. + */ + if (unlikely(!PageHugeFreed(head))) { + spin_unlock(&hugetlb_lock); + + /* + * Theoretically, we should return -EBUSY when we + * encounter this race. In fact, we have a chance + * to successfully dissolve the page if we do a + * retry. Because the race window is quite small. + * If we seize this opportunity, it is an optimization + * for increasing the success rate of dissolving page. + */ + while (PageHeadHuge(head) && !PageHugeFreed(head)) + cond_resched(); + + goto retry; + } + /* * Move PageHWPoison flag from head page to the raw error page, * which makes any subpages rather than the error page reusable. From patchwork Thu Jan 14 10:35:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12019195 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92B89C433DB for ; Thu, 14 Jan 2021 10:38:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1D2E623A3B for ; Thu, 14 Jan 2021 10:38:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1D2E623A3B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AA2B68D00D2; Thu, 14 Jan 2021 05:38:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A52968D008E; Thu, 14 Jan 2021 05:38:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 968F98D00D2; Thu, 14 Jan 2021 05:38:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0140.hostedemail.com [216.40.44.140]) by kanga.kvack.org (Postfix) with ESMTP id 81E078D008E for ; Thu, 14 Jan 2021 05:38:46 -0500 (EST) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 4CAAA181AC9CB for ; Thu, 14 Jan 2021 10:38:46 +0000 (UTC) X-FDA: 77704032252.26.walk89_2d01e0727526 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin26.hostedemail.com (Postfix) with ESMTP id 287DF1804A333 for ; Thu, 14 Jan 2021 10:38:46 +0000 (UTC) X-HE-Tag: walk89_2d01e0727526 X-Filterd-Recvd-Size: 5122 Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by imf15.hostedemail.com (Postfix) with ESMTP for ; Thu, 14 Jan 2021 10:38:45 +0000 (UTC) Received: by mail-pf1-f171.google.com with SMTP id c12so3088719pfo.10 for ; Thu, 14 Jan 2021 02:38:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0cYwNXoL62ufowFhR51LMghCmA8txi2r5XxTjm0v6q4=; b=WAckzXA3NYKMKzPO7d3fxmhmFa8XPvuFtpLdTcYJt0gP2l6AyanNdZetxWYn6M6gh8 EKuAhkJ6BYRwelX6ez+H0e0UwT09Rsu1xybpU2VtIhSNwzQ4q564A5sHSawFcgoY9os5 aRZU6FJl2PWqX/JDf7WGYUMv6bB57KPfWNlRn5mAAwJ4TbBZjzwFRwyh+fKQFrlN9HMc AvQvmaLzm4gORcjnaSbXJ8kNWYLsxPdqFhs/H/MNrGp7PBz5QrXC8MhkbZ1saUcpzFhT 28eZaFI5C/HuYjZZMBu5/YFIV5rta7rOuHR6JMNWnCGh2KIUhh/mpFuG8LKNOD1QzfZ+ KVZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0cYwNXoL62ufowFhR51LMghCmA8txi2r5XxTjm0v6q4=; b=bgJ22kut04meg6WLpn9qCrKw5YJyxoRBZ49PpT0EMWvf3J9hqMiPeP8K/ClZNLMBbJ g7NipXoxeI914ysT/0ZgejNsENjPb+fE/LeSBlavsbofpkltbLoM28AT27iQrfcZAmf5 0MXKdJOVUNzzjfXa8657m0ZPd2hiym4Y6nDo+MpfXSM6AvjHBLy46VMyOzvggaMHysAU 0nfF99YEHvRRGU+rMkctjcyL4dRy5q0ViR335JjJJSlc6bDPSEX+R8Nhmx4JJhkryCoH TMb9pw/dI2n0yrBCsxTUXsqoH7AwyRZXUGqP1HqNdFGQPXV4ADhbrRHCyipsGwr3ixa9 W+Xw== X-Gm-Message-State: AOAM533tA4cX/cmbLz0j0FakMVOq+Rguc0GRI228vWFy/ndhpytpWk1h w90wMgrIaC4v4Qhaq3TlDxitNg== X-Google-Smtp-Source: ABdhPJxXNYBJ+M1mhSEdHS3f5xG3SEV5dk5DG3dTkVKcMRSTqqZO27UWtQjHKdl0bxjQxbvpmqv9PA== X-Received: by 2002:a63:1c09:: with SMTP id c9mr6887350pgc.185.1610620723225; Thu, 14 Jan 2021 02:38:43 -0800 (PST) Received: from localhost.localdomain ([139.177.225.247]) by smtp.gmail.com with ESMTPSA id l12sm4970112pjq.7.2021.01.14.02.38.39 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Jan 2021 02:38:42 -0800 (PST) From: Muchun Song To: mike.kravetz@oracle.com, akpm@linux-foundation.org Cc: n-horiguchi@ah.jp.nec.com, ak@linux.intel.com, mhocko@suse.cz, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Michal Hocko , stable@vger.kernel.org Subject: [PATCH v5 4/5] mm: hugetlb: fix a race between isolating and freeing page Date: Thu, 14 Jan 2021 18:35:14 +0800 Message-Id: <20210114103515.12955-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210114103515.12955-1-songmuchun@bytedance.com> References: <20210114103515.12955-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There is a race between isolate_huge_page() and __free_huge_page(). CPU0: CPU1: if (PageHuge(page)) put_page(page) __free_huge_page(page) spin_lock(&hugetlb_lock) update_and_free_page(page) set_compound_page_dtor(page, NULL_COMPOUND_DTOR) spin_unlock(&hugetlb_lock) isolate_huge_page(page) // trigger BUG_ON VM_BUG_ON_PAGE(!PageHead(page), page) spin_lock(&hugetlb_lock) page_huge_active(page) // trigger BUG_ON VM_BUG_ON_PAGE(!PageHuge(page), page) spin_unlock(&hugetlb_lock) When we isolate a HugeTLB page on CPU0. Meanwhile, we free it to the buddy allocator on CPU1. Then, we can trigger a BUG_ON on CPU0. Because it is already freed to the buddy allocator. Fixes: c8721bbbdd36 ("mm: memory-hotplug: enable memory hotplug to handle hugepage") Signed-off-by: Muchun Song Reviewed-by: Mike Kravetz Acked-by: Michal Hocko Cc: stable@vger.kernel.org --- mm/hugetlb.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 1b789d1fd06b..43b7a044f248 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5574,9 +5574,9 @@ bool isolate_huge_page(struct page *page, struct list_head *list) { bool ret = true; - VM_BUG_ON_PAGE(!PageHead(page), page); spin_lock(&hugetlb_lock); - if (!page_huge_active(page) || !get_page_unless_zero(page)) { + if (!PageHeadHuge(page) || !page_huge_active(page) || + !get_page_unless_zero(page)) { ret = false; goto unlock; } From patchwork Thu Jan 14 10:35:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12019197 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05D4BC433DB for ; Thu, 14 Jan 2021 10:38:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8D70923A3B for ; Thu, 14 Jan 2021 10:38:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8D70923A3B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 23DE08D00D3; Thu, 14 Jan 2021 05:38:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1EE328D008E; Thu, 14 Jan 2021 05:38:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 103F48D00D3; Thu, 14 Jan 2021 05:38:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0197.hostedemail.com [216.40.44.197]) by kanga.kvack.org (Postfix) with ESMTP id EDCEB8D008E for ; Thu, 14 Jan 2021 05:38:49 -0500 (EST) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id BB0DF181AC9CB for ; Thu, 14 Jan 2021 10:38:49 +0000 (UTC) X-FDA: 77704032378.27.rice44_0c11be527526 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin27.hostedemail.com (Postfix) with ESMTP id A18813D668 for ; Thu, 14 Jan 2021 10:38:49 +0000 (UTC) X-HE-Tag: rice44_0c11be527526 X-Filterd-Recvd-Size: 4327 Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Thu, 14 Jan 2021 10:38:49 +0000 (UTC) Received: by mail-pj1-f42.google.com with SMTP id v1so2863483pjr.2 for ; Thu, 14 Jan 2021 02:38:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4WYNxXgkqgGvh7TvTAzouOPetrx1LoimOg0RJns9SBk=; b=ZBD5gnOQj7KeBVofomR1qkSaECGUOhQdp1osQN6ZOThgdZogDsFTBtZ/UPy/Fvhg/Y e4y1J16wFNiDZD0xLv/xzUuEu/Uy2Qf2zxHfYcnGBANteZOYViau06OYWLZ8Y7nvNS2+ A+oSHmPV/G0UHoMjsWYib5rSXrmvJ2PJXUdgpMZKuCiMg/Ypi6YUEbmLiHSdolK65Wix MllM9G07BsivId819gXQKwHrDg3oT7skErXB7KzDU2X8hlExHeC4FMyn0d9oOO/cVnQm tgw5v56H0Tq8HN9pE1YFYF8daohNuRNHVQbP+hIiX6umTwGYRAurPbU4wFA8nWXeg921 aGDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4WYNxXgkqgGvh7TvTAzouOPetrx1LoimOg0RJns9SBk=; b=S4AKOQYteqLd2OPrQiZyrtr7Q1nNIP+FNd52luY1DQ3Fnxj6wWdMqa7Mp2HH83BGxL 4/XfzQfUTnciHxgAMvNCPrMGuo1bbltSLpHdIPVTbDyVfPuYXZoY/6LvsqY+HcP0izfz z53x61i4yCOifLxPQYxwJUS4aaAxo2XNcp7G4RQ+4zfUaF1YHWR7ACUYtkDB7fmvVbcH 5+rbrq3QdU0ynM6kALyghTKqc4JjMarPzaeoE+rkjpvWN7dXqUb0nRAS2iDbkl80AGoI p42Z5YSoVK7XbMyipYSIvP4NFIsE9bWXrzM0BHempZkfejdKj+LyxDKZzhuPeD8UZx4w 4Xfg== X-Gm-Message-State: AOAM532rn0KYUKPtcUopXVQBVtQtCfXqdBJ1KtWYcw/wwQKP0B1ZuhGE qeqgIUILMslvCRciONkfmPy9Bw== X-Google-Smtp-Source: ABdhPJxyztuJ0uI2Xtsn9clsCA2I9Y1fjJWQwyTIuJw00n1p1nR234xsKV8aRN3YjRK92vlFEBAmxw== X-Received: by 2002:a17:902:8a88:b029:dc:f3:6db2 with SMTP id p8-20020a1709028a88b02900dc00f36db2mr6991795plo.2.1610620728126; Thu, 14 Jan 2021 02:38:48 -0800 (PST) Received: from localhost.localdomain ([139.177.225.247]) by smtp.gmail.com with ESMTPSA id l12sm4970112pjq.7.2021.01.14.02.38.43 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Jan 2021 02:38:47 -0800 (PST) From: Muchun Song To: mike.kravetz@oracle.com, akpm@linux-foundation.org Cc: n-horiguchi@ah.jp.nec.com, ak@linux.intel.com, mhocko@suse.cz, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Michal Hocko , stable@vger.kernel.org Subject: [PATCH v5 5/5] mm: hugetlb: remove VM_BUG_ON_PAGE from page_huge_active Date: Thu, 14 Jan 2021 18:35:15 +0800 Message-Id: <20210114103515.12955-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210114103515.12955-1-songmuchun@bytedance.com> References: <20210114103515.12955-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The page_huge_active() can be called from scan_movable_pages() which do not hold a reference count to the HugeTLB page. So when we call page_huge_active() from scan_movable_pages(), the HugeTLB page can be freed parallel. Then we will trigger a BUG_ON which is in the page_huge_active() when CONFIG_DEBUG_VM is enabled. Just remove the VM_BUG_ON_PAGE. Fixes: 7e1f049efb86 ("mm: hugetlb: cleanup using paeg_huge_active()") Signed-off-by: Muchun Song Reviewed-by: Mike Kravetz Acked-by: Michal Hocko Cc: stable@vger.kernel.org --- mm/hugetlb.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 43b7a044f248..6c5dfef682cc 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1360,8 +1360,7 @@ struct hstate *size_to_hstate(unsigned long size) */ bool page_huge_active(struct page *page) { - VM_BUG_ON_PAGE(!PageHuge(page), page); - return PageHead(page) && PagePrivate(&page[1]); + return PageHeadHuge(page) && PagePrivate(&page[1]); } /* never called for tail page */