From patchwork Sun Jan 10 12:40:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12009091 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28CE3C433DB for ; Sun, 10 Jan 2021 12:41:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id ABC7C22B45 for ; Sun, 10 Jan 2021 12:41:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ABC7C22B45 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3A00A8D001A; Sun, 10 Jan 2021 07:41:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 34E8D8D0019; Sun, 10 Jan 2021 07:41:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2663E8D001A; Sun, 10 Jan 2021 07:41:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0226.hostedemail.com [216.40.44.226]) by kanga.kvack.org (Postfix) with ESMTP id 0970D8D0019 for ; Sun, 10 Jan 2021 07:41:30 -0500 (EST) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id C269D824556B for ; Sun, 10 Jan 2021 12:41:29 +0000 (UTC) X-FDA: 77689826298.11.bird80_0d1296b27504 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id A5C8E180F8B80 for ; Sun, 10 Jan 2021 12:41:29 +0000 (UTC) X-HE-Tag: bird80_0d1296b27504 X-Filterd-Recvd-Size: 4074 Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Sun, 10 Jan 2021 12:41:29 +0000 (UTC) Received: by mail-pj1-f42.google.com with SMTP id l23so8788013pjg.1 for ; Sun, 10 Jan 2021 04:41:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KaGXltvZsrp8CrKlla29akS0Y9Nr6dS/klR3W0Q0I3k=; b=mhH1+ONodj1jrEp44EgwTPMDpRYALjgkxKo7IAU4FLxQhOx+cQ7xJTKK4ST4aKUFRL ImjG14r00vwqLOEq1LtuX0K3jAyJxt+FAGBLG6xSPQWaEb0GzCchNO2AFzctpgzZ0w0N iCF6uzYn00XWOQ9HPQS5RLFG3MaGJRTQXR4N3ALAiP6H3G+3D749HLAXv0a7tultO+Cg 4fhgUb1bXnGmov2qI8bEo52OmWqtCnADVZxxeSBaOD0zWwREBMFTumxjJSj63Z+nx5VD CtruF9wExES4ZgYpMBfxycWgp6K8d8TV0GUOwyVxdCxHSqmPeCO5CAdI1K8YLaUE/FJ4 b3aQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KaGXltvZsrp8CrKlla29akS0Y9Nr6dS/klR3W0Q0I3k=; b=CmYNjQ7cYXyZcbis0GMpsbPUDBs46ZeI9t0fycYOp3UjKuzFdSpiOtsRhUbYF93klo nJUVjriGDYnK/9mBov842llEBi+ABMtS4b2xcvZlGRSQxPKjLoPgGQItTrEiUT05V7hL DR7NRfbrRLKSy1F6fOjB5/64P4ZwnvVImvf2dmp1Yeof25H3s9/jMXYqZahA08tg++6W kNYisuOCSf60Icn+KLtx7v2HoIhnx8iavbCoUFXErU5/u8tBviu/DQp8g0Ors+m8InEZ bCMn61e2xq3jdGQ2ZugRDSd5HHpu9RMqGhOD49MEQf38QmcID1/l+Yf/IHfqKZpFgbEP Ft8A== X-Gm-Message-State: AOAM5331tGG0Ep/EMrWdxBjN2OB27ZxQYB1VmLrEwWfe4lEe1KQo6XJo pyo86HDkmoPiZ9egp3gczoFcmQ== X-Google-Smtp-Source: ABdhPJwhzqxNy3uyJAZp111LPgXCrGrRbpbuCwkaaygJaHeHyHHUUioTRzwIzRdKi5SsZNGm87Hxhw== X-Received: by 2002:a17:90a:b395:: with SMTP id e21mr12896195pjr.197.1610282488011; Sun, 10 Jan 2021 04:41:28 -0800 (PST) Received: from localhost.localdomain ([139.177.225.247]) by smtp.gmail.com with ESMTPSA id p9sm16176960pfq.136.2021.01.10.04.41.24 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 10 Jan 2021 04:41:27 -0800 (PST) From: Muchun Song To: mike.kravetz@oracle.com, akpm@linux-foundation.org Cc: n-horiguchi@ah.jp.nec.com, ak@linux.intel.com, mhocko@suse.cz, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Yang Shi Subject: [PATCH v3 1/6] mm: migrate: do not migrate HugeTLB page whose refcount is one Date: Sun, 10 Jan 2021 20:40:12 +0800 Message-Id: <20210110124017.86750-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210110124017.86750-1-songmuchun@bytedance.com> References: <20210110124017.86750-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If the refcount is one when it is migrated, it means that the page was freed from under us. So we are done and do not need to migrate. This optimization is consistent with the regular pages, just like unmap_and_move() does. Signed-off-by: Muchun Song Reviewed-by: Mike Kravetz Acked-by: Yang Shi Acked-by: Michal Hocko --- mm/migrate.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/migrate.c b/mm/migrate.c index 4385f2fb5d18..a6631c4eb6a6 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1279,6 +1279,12 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, return -ENOSYS; } + if (page_count(hpage) == 1) { + /* page was freed from under us. So we are done. */ + putback_active_hugepage(hpage); + return MIGRATEPAGE_SUCCESS; + } + new_hpage = get_new_page(hpage, private); if (!new_hpage) return -ENOMEM; From patchwork Sun Jan 10 12:40:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12009093 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66702C433DB for ; Sun, 10 Jan 2021 12:41:35 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D4CA522ADF for ; Sun, 10 Jan 2021 12:41:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D4CA522ADF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 748938D001B; Sun, 10 Jan 2021 07:41:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6FB718D0019; Sun, 10 Jan 2021 07:41:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 60EB28D001B; Sun, 10 Jan 2021 07:41:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0082.hostedemail.com [216.40.44.82]) by kanga.kvack.org (Postfix) with ESMTP id 4B6DA8D0019 for ; Sun, 10 Jan 2021 07:41:34 -0500 (EST) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 0B283181AEF15 for ; Sun, 10 Jan 2021 12:41:34 +0000 (UTC) X-FDA: 77689826508.22.wood06_4a163c427504 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin22.hostedemail.com (Postfix) with ESMTP id DD3E518038E60 for ; Sun, 10 Jan 2021 12:41:33 +0000 (UTC) X-HE-Tag: wood06_4a163c427504 X-Filterd-Recvd-Size: 5244 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf32.hostedemail.com (Postfix) with ESMTP for ; Sun, 10 Jan 2021 12:41:33 +0000 (UTC) Received: by mail-pl1-f176.google.com with SMTP id s15so8046313plr.9 for ; Sun, 10 Jan 2021 04:41:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ffJDakXCDHXSBit9htKSivl1vC+lnq1/j37kP5/245Q=; b=cUxJC0PDgtphFqteTncK1MMWPA0tvaGir94in2c2J8OS1SoLVX445ms53I6WXgvgZt omcPb99wACh3cWgOjXnBmpbdfO6TGUoKvW5FkIAjWnK2WaGBlkaGuWe3fil1eko5nnNz 0fEHHF+vbssaLNNJZQ4OERgj1TxSq9doOTw9wn6N/E1c9kvgMXSW8khp42cxCiEKOYZc YIT82Ffe2Dm5QVfgBXtzGRcUq66thL3sVDK8CB/y+caCXc3MflE+iN6IsmWSE3r9Wr0Q dmMPD1A8lozfusvXZFMUOYroviV/u8043Bi2YD3mEM6yJnOAy79Fj1HAx7kIXVvsPX6M uFLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ffJDakXCDHXSBit9htKSivl1vC+lnq1/j37kP5/245Q=; b=tPeMGql0QztZB8ojkMF+w/NM8IP1PR795ik4HTO/ZMEGxnuDYL2zbLc4Xy5q4eSlcO dCo2ouEaULsxmxt3uXMxreRg9vVq/012ppsyNplF1FUdV8pAKknLI96jz7w7j3uer7aN KPFWk3VvWbH9/4OTv5BpKZhy6UUhfl0M0zxseYS72ZScABj/4OUlu24olNLM3bF21BhB vhhQAxnb3dzig3S+ZiSyEA1WiTFkd963NpvKEB9PovKoSTACxk7GqpH8JKEJTgQ1qPZO 6A9MXFq0gzlZRH2FtWe2tHsqguaoHtmmul30byptz5Gpp8L/cUTjY/RQXATkRBCvGGX+ yJQQ== X-Gm-Message-State: AOAM532jKr1BQ2auV3q9gR+5UD++iIqmBMOigww94u8R8czvsOEO4qgp tcWMrj2KvEjMxwd2f9/7ZqLN/Q== X-Google-Smtp-Source: ABdhPJyZ047vfaPcWkl/qH5UFf8rFpsGc+XXjNkX+M8hjJ33pepyB/KnD8+n6in94OwlD32paX1+4g== X-Received: by 2002:a17:902:b782:b029:dc:251:423f with SMTP id e2-20020a170902b782b02900dc0251423fmr12395450pls.30.1610282492291; Sun, 10 Jan 2021 04:41:32 -0800 (PST) Received: from localhost.localdomain ([139.177.225.247]) by smtp.gmail.com with ESMTPSA id p9sm16176960pfq.136.2021.01.10.04.41.28 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 10 Jan 2021 04:41:31 -0800 (PST) From: Muchun Song To: mike.kravetz@oracle.com, akpm@linux-foundation.org Cc: n-horiguchi@ah.jp.nec.com, ak@linux.intel.com, mhocko@suse.cz, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , stable@vger.kernel.org Subject: [PATCH v3 2/6] mm: hugetlbfs: fix cannot migrate the fallocated HugeTLB page Date: Sun, 10 Jan 2021 20:40:13 +0800 Message-Id: <20210110124017.86750-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210110124017.86750-1-songmuchun@bytedance.com> References: <20210110124017.86750-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If a new hugetlb page is allocated during fallocate it will not be marked as active (set_page_huge_active) which will result in a later isolate_huge_page failure when the page migration code would like to move that page. Such a failure would be unexpected and wrong. Only export set_page_huge_active, just leave clear_page_huge_active as static. Because there are no external users. Fixes: 70c3547e36f5 (hugetlbfs: add hugetlbfs_fallocate()) Signed-off-by: Muchun Song Cc: stable@vger.kernel.org Reviewed-by: Mike Kravetz Acked-by: Michal Hocko --- fs/hugetlbfs/inode.c | 3 ++- include/linux/hugetlb.h | 2 ++ mm/hugetlb.c | 2 +- 3 files changed, 5 insertions(+), 2 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index b5c109703daa..21c20fd5f9ee 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -735,9 +735,10 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, mutex_unlock(&hugetlb_fault_mutex_table[hash]); + set_page_huge_active(page); /* * unlock_page because locked by add_to_page_cache() - * page_put due to reference from alloc_huge_page() + * put_page() due to reference from alloc_huge_page() */ unlock_page(page); put_page(page); diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index ebca2ef02212..b5807f23caf8 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -770,6 +770,8 @@ static inline void huge_ptep_modify_prot_commit(struct vm_area_struct *vma, } #endif +void set_page_huge_active(struct page *page); + #else /* CONFIG_HUGETLB_PAGE */ struct hstate {}; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 1f3bf1710b66..4741d60f8955 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1348,7 +1348,7 @@ bool page_huge_active(struct page *page) } /* never called for tail page */ -static void set_page_huge_active(struct page *page) +void set_page_huge_active(struct page *page) { VM_BUG_ON_PAGE(!PageHeadHuge(page), page); SetPagePrivate(&page[1]); From patchwork Sun Jan 10 12:40:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12009095 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE2E0C433E0 for ; Sun, 10 Jan 2021 12:41:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 72E2122ADF for ; Sun, 10 Jan 2021 12:41:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 72E2122ADF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 06D2E8D001C; Sun, 10 Jan 2021 07:41:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0419A8D0019; Sun, 10 Jan 2021 07:41:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E73C08D001C; Sun, 10 Jan 2021 07:41:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0201.hostedemail.com [216.40.44.201]) by kanga.kvack.org (Postfix) with ESMTP id D0CCC8D0019 for ; Sun, 10 Jan 2021 07:41:37 -0500 (EST) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 9AF22181AEF15 for ; Sun, 10 Jan 2021 12:41:37 +0000 (UTC) X-FDA: 77689826634.18.owl89_28156af27504 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin18.hostedemail.com (Postfix) with ESMTP id 734CA100ED0FE for ; Sun, 10 Jan 2021 12:41:37 +0000 (UTC) X-HE-Tag: owl89_28156af27504 X-Filterd-Recvd-Size: 6343 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf20.hostedemail.com (Postfix) with ESMTP for ; Sun, 10 Jan 2021 12:41:36 +0000 (UTC) Received: by mail-pl1-f175.google.com with SMTP id v3so8041776plz.13 for ; Sun, 10 Jan 2021 04:41:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=I3gDuLjEjqpY5hJrgoP0zbm1oa1HSY0dQY30oFlgAYQ=; b=rWeFaHhdx26QIcu4oFq9jt4fdoiwTTEMHiX0GcDFnvXCNSEGXKNUOhSowItykhSP+s MU2lB7VGPKOpR26dagIJbSrZ0z4DQUseP8TI1Hr8218icPt2CE59nrthBPoGc5+2L0py GldzLHOZ/qQ22eoxfirNB/sX7sTFkGnYQS4NQwdHXLZ95tD8fo3K17CN+WvUYYuirXhr TpF+IMj8yyO/hRNwe/nRuTb4oq+CLbVTwC4ZMsGRxI8qPFxdbhi6TNcVuZv0oM+ZLqNg 4jK9h5RHhQ6APoWqIDlwAfx9/2k67z/4jNjM/HkPqSz2S5XOK+/rLaWpDAKaMrYgYVJT m2bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=I3gDuLjEjqpY5hJrgoP0zbm1oa1HSY0dQY30oFlgAYQ=; b=Ks7BOxjb2LhGL+pSmzoJe+m/NHFMinlV/jPc++yitwHNxk0ENIvOKlYRMOcGR5ffaU t25BuIcKyuFDCOgz7wNuCIRnx9NWnuJ+LiId5WWVbtNcRFQYg73desL1e0/aMc79DgEO Ph2/xXnV4KJQON/0OZZyzxvK6ejS7mENpFkTNrAe+Ps5NaB3ePsIMYJUXJ2JJritB1jM wyLY+hT23NmjvLw0grLNDxIugvcQELANBoH1c/Zu6AG75cpqKH7lIkI0ktTp/TabLEEF /tA/4DPCnYwbtD6IkyeGPedPFZ1TEqgORl72mpArO7JAikes8vz58xYXZkiJ2r4FQXju /y6w== X-Gm-Message-State: AOAM532EiIQTR30rVSWgm9WQtNH236Wa0vo+cD0L0FOsbS++11MxipZu VA07/8+fWVYpPka+wzSz3FWCYQ== X-Google-Smtp-Source: ABdhPJzWrZDYsSoIxdfRxNA612sIivazrBh4X8Socv1b7/yltfVgFXyC4/ghwU7KuryYYJvTKH3CpQ== X-Received: by 2002:a17:90a:4402:: with SMTP id s2mr12942473pjg.37.1610282495977; Sun, 10 Jan 2021 04:41:35 -0800 (PST) Received: from localhost.localdomain ([139.177.225.247]) by smtp.gmail.com with ESMTPSA id p9sm16176960pfq.136.2021.01.10.04.41.32 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 10 Jan 2021 04:41:35 -0800 (PST) From: Muchun Song To: mike.kravetz@oracle.com, akpm@linux-foundation.org Cc: n-horiguchi@ah.jp.nec.com, ak@linux.intel.com, mhocko@suse.cz, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , stable@vger.kernel.org Subject: [PATCH v3 3/6] mm: hugetlb: fix a race between freeing and dissolving the page Date: Sun, 10 Jan 2021 20:40:14 +0800 Message-Id: <20210110124017.86750-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210110124017.86750-1-songmuchun@bytedance.com> References: <20210110124017.86750-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There is a race condition between __free_huge_page() and dissolve_free_huge_page(). CPU0: CPU1: // page_count(page) == 1 put_page(page) __free_huge_page(page) dissolve_free_huge_page(page) spin_lock(&hugetlb_lock) // PageHuge(page) && !page_count(page) update_and_free_page(page) // page is freed to the buddy spin_unlock(&hugetlb_lock) spin_lock(&hugetlb_lock) clear_page_huge_active(page) enqueue_huge_page(page) // It is wrong, the page is already freed spin_unlock(&hugetlb_lock) The race windows is between put_page() and dissolve_free_huge_page(). We should make sure that the page is already on the free list when it is dissolved. Fixes: c8721bbbdd36 ("mm: memory-hotplug: enable memory hotplug to handle hugepage") Signed-off-by: Muchun Song Cc: stable@vger.kernel.org Reviewed-by: Mike Kravetz --- mm/hugetlb.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 4741d60f8955..4a9011e12175 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -79,6 +79,21 @@ DEFINE_SPINLOCK(hugetlb_lock); static int num_fault_mutexes; struct mutex *hugetlb_fault_mutex_table ____cacheline_aligned_in_smp; +static inline bool PageHugeFreed(struct page *head) +{ + return page_private(head + 4) == -1UL; +} + +static inline void SetPageHugeFreed(struct page *head) +{ + set_page_private(head + 4, -1UL); +} + +static inline void ClearPageHugeFreed(struct page *head) +{ + set_page_private(head + 4, 0); +} + /* Forward declaration */ static int hugetlb_acct_memory(struct hstate *h, long delta); @@ -1028,6 +1043,7 @@ static void enqueue_huge_page(struct hstate *h, struct page *page) list_move(&page->lru, &h->hugepage_freelists[nid]); h->free_huge_pages++; h->free_huge_pages_node[nid]++; + SetPageHugeFreed(page); } static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) @@ -1044,6 +1060,7 @@ static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) list_move(&page->lru, &h->hugepage_activelist); set_page_refcounted(page); + ClearPageHugeFreed(page); h->free_huge_pages--; h->free_huge_pages_node[nid]--; return page; @@ -1504,6 +1521,7 @@ static void prep_new_huge_page(struct hstate *h, struct page *page, int nid) spin_lock(&hugetlb_lock); h->nr_huge_pages++; h->nr_huge_pages_node[nid]++; + ClearPageHugeFreed(page); spin_unlock(&hugetlb_lock); } @@ -1770,6 +1788,14 @@ int dissolve_free_huge_page(struct page *page) int nid = page_to_nid(head); if (h->free_huge_pages - h->resv_huge_pages == 0) goto out; + + /* + * We should make sure that the page is already on the free list + * when it is dissolved. + */ + if (unlikely(!PageHugeFreed(head))) + goto out; + /* * Move PageHWPoison flag from head page to the raw error page, * which makes any subpages rather than the error page reusable. From patchwork Sun Jan 10 12:40:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12009097 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F25B0C433E0 for ; Sun, 10 Jan 2021 12:41:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9F6E022ADF for ; Sun, 10 Jan 2021 12:41:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9F6E022ADF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 378AA8D001D; Sun, 10 Jan 2021 07:41:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 34E1E8D0019; Sun, 10 Jan 2021 07:41:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 23DA08D001D; Sun, 10 Jan 2021 07:41:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0114.hostedemail.com [216.40.44.114]) by kanga.kvack.org (Postfix) with ESMTP id 0E28B8D0019 for ; Sun, 10 Jan 2021 07:41:41 -0500 (EST) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id C8BD7180AD806 for ; Sun, 10 Jan 2021 12:41:40 +0000 (UTC) X-FDA: 77689826760.23.legs54_430ccd327504 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin23.hostedemail.com (Postfix) with ESMTP id AD3FB37606 for ; Sun, 10 Jan 2021 12:41:40 +0000 (UTC) X-HE-Tag: legs54_430ccd327504 X-Filterd-Recvd-Size: 6031 Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by imf47.hostedemail.com (Postfix) with ESMTP for ; Sun, 10 Jan 2021 12:41:40 +0000 (UTC) Received: by mail-pf1-f179.google.com with SMTP id w2so1424079pfc.13 for ; Sun, 10 Jan 2021 04:41:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=HhlvFEw8jdfKSbqCegVc8DBdoL7Xly+0YXsiaFK68GM=; b=nKa0zwCZ2svXtZUVz4u93tOukfQ1rtXnodZzLZhzNw1D0tYe9JtTZFDbpoY1bK9Ae4 p0r4U3LV61u3OUIaXpKBU6s8bKiOAbz3MGr/sYXcISksN4RvmQJYhO96VW2igbvtWCZO tuweADoElAyPvb5rZaYt+9AC0vLhUxyFP3/0SPCeUHpsigUZLZO6gDn0Tczo8q3OUNax JACLi7hxFufRkJs4tq2LpErAFIoyJuRMX4fhVybpmp1T5siBX/mHgdkb87tEpWSyPtFV 1HFdOc6YrluZJzk2TcViU0uhqtJOv1KOVP4gFNFblSzqJoyQua1kNOFf+reaUu8xZLwS DUsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HhlvFEw8jdfKSbqCegVc8DBdoL7Xly+0YXsiaFK68GM=; b=qlO6QsMx4GBw1E1dYzEjEfCyV4sKn0V7w5u9+HxRDqbdHokwYemVQt0+IfNlk9Pce5 YUTE0xm+vNOCcZypfJzgxajuHWlYDZZUeXBeUxNiJWgZE0MfCkvTdR4yeD9LbuSp8aeY asfcJGa89rO9JBxeiLITGAE6cZsX+uESeHK0DsaiclEpSVPzr3shlUSHRKLGh+G0TvqX L/trT7DpP8DiPOe391d/I5ep3/H28PtVy4r2wbb7nvEZUsl3YPx+UmiZHE0GIGZsb5D6 +61G1POSAJkWKaXUD6L6pTrzEsF6rO+6BaoH3Egh2GUPUtUgm8iLNFDAKzPuNRlZpu+3 pbmw== X-Gm-Message-State: AOAM530VQGJvW+iEsdc5UDh79fzh+voE0xJRdg0giFUbsdtVsX0zTl00 G6/rY56iSmoRACRpruiD1Oax/Q== X-Google-Smtp-Source: ABdhPJz16FQS6hGSFtI9gh/TUdc46pPEDeMctnvCGgRCVHKmu4bKwdmuvDxjYQLBqeHMn1ZWiX3K6Q== X-Received: by 2002:aa7:8ad0:0:b029:1a9:3a46:78d1 with SMTP id b16-20020aa78ad00000b02901a93a4678d1mr12335934pfd.77.1610282499413; Sun, 10 Jan 2021 04:41:39 -0800 (PST) Received: from localhost.localdomain ([139.177.225.247]) by smtp.gmail.com with ESMTPSA id p9sm16176960pfq.136.2021.01.10.04.41.36 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 10 Jan 2021 04:41:38 -0800 (PST) From: Muchun Song To: mike.kravetz@oracle.com, akpm@linux-foundation.org Cc: n-horiguchi@ah.jp.nec.com, ak@linux.intel.com, mhocko@suse.cz, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song Subject: [PATCH v3 4/6] mm: hugetlb: add return -EAGAIN for dissolve_free_huge_page Date: Sun, 10 Jan 2021 20:40:15 +0800 Message-Id: <20210110124017.86750-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210110124017.86750-1-songmuchun@bytedance.com> References: <20210110124017.86750-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There is a race between dissolve_free_huge_page() and put_page(), and the race window is quite small. Theoretically, we should return -EBUSY when we encounter this race. In fact, we have a chance to successfully dissolve the page if we do a retry. Because the race window is quite small. If we seize this opportunity, it is an optimization for increasing the success rate of dissolving page. If we free a HugeTLB page from a non-task context, it is deferred through a workqueue. In this case, we need to flush the work. The dissolve_free_huge_page() can be called from memory hotplug, the caller aims to free the HugeTLB page to the buddy allocator so that the caller can unplug the page successfully. Signed-off-by: Muchun Song --- mm/hugetlb.c | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 4a9011e12175..a176ceed55f1 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1763,10 +1763,11 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, * nothing for in-use hugepages and non-hugepages. * This function returns values like below: * - * -EBUSY: failed to dissolved free hugepages or the hugepage is in-use - * (allocated or reserved.) - * 0: successfully dissolved free hugepages or the page is not a - * hugepage (considered as already dissolved) + * -EAGAIN: race with __free_huge_page() and can do a retry + * -EBUSY: failed to dissolved free hugepages or the hugepage is in-use + * (allocated or reserved.) + * 0: successfully dissolved free hugepages or the page is not a + * hugepage (considered as already dissolved) */ int dissolve_free_huge_page(struct page *page) { @@ -1793,8 +1794,10 @@ int dissolve_free_huge_page(struct page *page) * We should make sure that the page is already on the free list * when it is dissolved. */ - if (unlikely(!PageHugeFreed(head))) + if (unlikely(!PageHugeFreed(head))) { + rc = -EAGAIN; goto out; + } /* * Move PageHWPoison flag from head page to the raw error page, @@ -1813,6 +1816,14 @@ int dissolve_free_huge_page(struct page *page) } out: spin_unlock(&hugetlb_lock); + + /* + * If the freeing of the HugeTLB page is put on a work queue, we should + * flush the work before retrying. + */ + if (unlikely(rc == -EAGAIN)) + flush_work(&free_hpage_work); + return rc; } @@ -1835,7 +1846,12 @@ int dissolve_free_huge_pages(unsigned long start_pfn, unsigned long end_pfn) for (pfn = start_pfn; pfn < end_pfn; pfn += 1 << minimum_order) { page = pfn_to_page(pfn); +retry: rc = dissolve_free_huge_page(page); + if (rc == -EAGAIN) { + cpu_relax(); + goto retry; + } if (rc) break; } From patchwork Sun Jan 10 12:40:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12009099 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0578C433DB for ; Sun, 10 Jan 2021 12:41:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 72F7822ADF for ; Sun, 10 Jan 2021 12:41:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 72F7822ADF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0E0A18D001E; Sun, 10 Jan 2021 07:41:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0B77F8D0019; Sun, 10 Jan 2021 07:41:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F36148D001E; Sun, 10 Jan 2021 07:41:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0096.hostedemail.com [216.40.44.96]) by kanga.kvack.org (Postfix) with ESMTP id DF4AB8D0019 for ; Sun, 10 Jan 2021 07:41:44 -0500 (EST) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id A9476181AEF15 for ; Sun, 10 Jan 2021 12:41:44 +0000 (UTC) X-FDA: 77689826928.14.cow62_2112e6d27504 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin14.hostedemail.com (Postfix) with ESMTP id 8B2481822987B for ; Sun, 10 Jan 2021 12:41:44 +0000 (UTC) X-HE-Tag: cow62_2112e6d27504 X-Filterd-Recvd-Size: 5125 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) by imf11.hostedemail.com (Postfix) with ESMTP for ; Sun, 10 Jan 2021 12:41:44 +0000 (UTC) Received: by mail-pg1-f177.google.com with SMTP id g15so10779508pgu.9 for ; Sun, 10 Jan 2021 04:41:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9zrsneMtVXeg4b61BTwnRw2mnUrcLzSuxYCbbaoWav8=; b=ECYcoVG48fFW0nt6MIsiwAkaHi2G4EDJ1+2KCQwkSOhBIwXu3PyJGG87/xSzWSefKi GMu2prQfnPbOkZ7nBodB5rtI7PrCFTSTz91iSsDjCafbMs3bC8p9jygCAXY4MntDNnf+ w5mjBkib5igUCLB+g9Qbn6YwzpJ/Y3BBS+snBV5Yg2rKIUa+6Y8dSR6qGkKVUN67djYu U5nByE8TlftiWoYt4e0/KTKQDllHu8OY1XLX48S87Wt5Y07PqlxC0Xkub4H2Dwph6M3E w+3M9JgCl6t+/ClUI4RwKPrtcCU+taxqJgsjAi9e1bwMv328pODcFhXWYltAsR+3G9sB +phQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9zrsneMtVXeg4b61BTwnRw2mnUrcLzSuxYCbbaoWav8=; b=swuxCAVS0QXYtdzPHgbDv+0qKVWNSJUUiz8az1Q7X2XdomwgrmsNKcQ0hQMP4gpHo7 no3qUGUktljvzlTtiQUverg5XLZlI0xzXwxgIfaE9ZHyFwpkeSnER1+2EwfnxXSVOtNU HbTxB2Au+tOYp2i2CS+YHoZeXUA3CUSdCxhmN4tsWtHG4erqSYgEIJZVoW+hNdg6+NFC MmhBHuyb9Ha7s2bxkE1d7FNLKUyBvbVO0iJ0zSJ/p3mMPFyiW3cU70mF16kqZVWtNNrR v54kUDDC4oGtnhgo/DDMfNpD70jyO1cECIRgWnh866RfHkBBuQzb6WvjBJqGepzsl8mw e4KA== X-Gm-Message-State: AOAM531HEUHHSSepMCuBnvBjbvgwiVWBQSVgtpKCXuoQ+S7SEnXhDmAl +QNQdVrPjomvD3uY8g9D/id6kQ== X-Google-Smtp-Source: ABdhPJz8VammpoTwnjyKnNe7opIwIAMtj9XxNyBPNM5DGwpJ+k8YsAYnqUEYhJWs/tGCae7TfB+toA== X-Received: by 2002:a63:ea01:: with SMTP id c1mr15452769pgi.138.1610282503368; Sun, 10 Jan 2021 04:41:43 -0800 (PST) Received: from localhost.localdomain ([139.177.225.247]) by smtp.gmail.com with ESMTPSA id p9sm16176960pfq.136.2021.01.10.04.41.39 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 10 Jan 2021 04:41:42 -0800 (PST) From: Muchun Song To: mike.kravetz@oracle.com, akpm@linux-foundation.org Cc: n-horiguchi@ah.jp.nec.com, ak@linux.intel.com, mhocko@suse.cz, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Michal Hocko , stable@vger.kernel.org Subject: [PATCH v3 5/6] mm: hugetlb: fix a race between isolating and freeing page Date: Sun, 10 Jan 2021 20:40:16 +0800 Message-Id: <20210110124017.86750-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210110124017.86750-1-songmuchun@bytedance.com> References: <20210110124017.86750-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There is a race between isolate_huge_page() and __free_huge_page(). CPU0: CPU1: if (PageHuge(page)) put_page(page) __free_huge_page(page) spin_lock(&hugetlb_lock) update_and_free_page(page) set_compound_page_dtor(page, NULL_COMPOUND_DTOR) spin_unlock(&hugetlb_lock) isolate_huge_page(page) // trigger BUG_ON VM_BUG_ON_PAGE(!PageHead(page), page) spin_lock(&hugetlb_lock) page_huge_active(page) // trigger BUG_ON VM_BUG_ON_PAGE(!PageHuge(page), page) spin_unlock(&hugetlb_lock) When we isolate a HugeTLB page on CPU0. Meanwhile, we free it to the buddy allocator on CPU1. Then, we can trigger a BUG_ON on CPU0. Because it is already freed to the buddy allocator. Fixes: c8721bbbdd36 ("mm: memory-hotplug: enable memory hotplug to handle hugepage") Signed-off-by: Muchun Song Reviewed-by: Mike Kravetz Acked-by: Michal Hocko Cc: stable@vger.kernel.org --- mm/hugetlb.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index a176ceed55f1..e7ed30afbb8f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5575,9 +5575,9 @@ bool isolate_huge_page(struct page *page, struct list_head *list) { bool ret = true; - VM_BUG_ON_PAGE(!PageHead(page), page); spin_lock(&hugetlb_lock); - if (!page_huge_active(page) || !get_page_unless_zero(page)) { + if (!PageHeadHuge(page) || !page_huge_active(page) || + !get_page_unless_zero(page)) { ret = false; goto unlock; } From patchwork Sun Jan 10 12:40:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12009101 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB9A7C433DB for ; Sun, 10 Jan 2021 12:41:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9289422ADF for ; Sun, 10 Jan 2021 12:41:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9289422ADF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 32BC68D001F; Sun, 10 Jan 2021 07:41:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2E34D8D0019; Sun, 10 Jan 2021 07:41:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1C87F8D001F; Sun, 10 Jan 2021 07:41:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 07E788D0019 for ; Sun, 10 Jan 2021 07:41:49 -0500 (EST) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id CCB043629 for ; Sun, 10 Jan 2021 12:41:48 +0000 (UTC) X-FDA: 77689827096.08.year43_29081cd27504 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin08.hostedemail.com (Postfix) with ESMTP id B6CB91819E76B for ; Sun, 10 Jan 2021 12:41:48 +0000 (UTC) X-HE-Tag: year43_29081cd27504 X-Filterd-Recvd-Size: 4287 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf27.hostedemail.com (Postfix) with ESMTP for ; Sun, 10 Jan 2021 12:41:48 +0000 (UTC) Received: by mail-pl1-f177.google.com with SMTP id 4so8058357plk.5 for ; Sun, 10 Jan 2021 04:41:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Q2v5mrrSjESraNqHDwGVpPTSDpsByCOvtLmUTzZt6g0=; b=YVg4gth6O1BgEX5TXwXspQm27uCViGy7IU0/RmDe2i8CR9p4G0fXQW217eoFau69+I bcl/d5PDS+T1DBTddzhv7AVOPPvxtHJuE4NLYnjqVW/0n5js6XOdAT8nIdO3C2dc8emE O9WiwNqcJ6vM1ZfRlDiOocrbOoBMp2DVvN5kIPA6BATPRJ/CcI24qXY+NoBib7jrcsVA ap5aPeNdarr1Mp7cskUk3Lym7v7p0xERycNHqBeTH8TJsXETmlV23rndVerXw3okmV7O wvHn4OVoHXtrtEjhk/2bjHta7Ae9R3C8ZittrFvRrBGa9EvFuTjPnot7FHpfbZgaSWLf Jpng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Q2v5mrrSjESraNqHDwGVpPTSDpsByCOvtLmUTzZt6g0=; b=Lm3hFfhKa5dWkixBsXimXRy3viaR+yKmJxopA7D36qP80mx9JUO9s6VMT8QUPwaQxh zURFxbMtp6bzeEFN86SYH09ql+QRrwyOv1c5fPq4QropNzPge5tcZu/wA0Tl/v6qWZXC dZbu6Hnc79nETuQ1WBHk6oMp9N2CXs00l4kj/Y9ktIyQg18Zezd4yZoz3fx6IVHsvJ8h xc69QwKIYnOZFLaQgzWKSbwfqQRxJLHQjWK3j30RH94GOLd9sDQUIZ/dsHuNk7cu+R6l i8hrl3M4KG3lp1S7EaDy5jfH3VOgalPtJERzinP+RWrfnYCAO5IXiFF3D73rw+9/vfhA +WBA== X-Gm-Message-State: AOAM533AWJofM8Ds7qycSt8BrEE8b2wfqAUpE/AyoDXEHp41umILEOz1 9l5rBXbakwUmFIqqMCISYoq1Iw== X-Google-Smtp-Source: ABdhPJzl17pIIRw/wPQcY4D0IaWNRYplQXvDOcoTJCykKdxrbXO1sPO/EXGH5SNXKhzi6llVHPNJ5g== X-Received: by 2002:a17:90a:cb84:: with SMTP id a4mr13213004pju.50.1610282507382; Sun, 10 Jan 2021 04:41:47 -0800 (PST) Received: from localhost.localdomain ([139.177.225.247]) by smtp.gmail.com with ESMTPSA id p9sm16176960pfq.136.2021.01.10.04.41.43 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 10 Jan 2021 04:41:46 -0800 (PST) From: Muchun Song To: mike.kravetz@oracle.com, akpm@linux-foundation.org Cc: n-horiguchi@ah.jp.nec.com, ak@linux.intel.com, mhocko@suse.cz, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song , Michal Hocko , stable@vger.kernel.org Subject: [PATCH v3 6/6] mm: hugetlb: remove VM_BUG_ON_PAGE from page_huge_active Date: Sun, 10 Jan 2021 20:40:17 +0800 Message-Id: <20210110124017.86750-7-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210110124017.86750-1-songmuchun@bytedance.com> References: <20210110124017.86750-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The page_huge_active() can be called from scan_movable_pages() which do not hold a reference count to the HugeTLB page. So when we call page_huge_active() from scan_movable_pages(), the HugeTLB page can be freed parallel. Then we will trigger a BUG_ON which is in the page_huge_active() when CONFIG_DEBUG_VM is enabled. Just remove the VM_BUG_ON_PAGE. Fixes: 7e1f049efb86 ("mm: hugetlb: cleanup using paeg_huge_active()") Signed-off-by: Muchun Song Reviewed-by: Mike Kravetz Acked-by: Michal Hocko Cc: stable@vger.kernel.org --- mm/hugetlb.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index e7ed30afbb8f..5940bf0c49b9 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1360,8 +1360,7 @@ struct hstate *size_to_hstate(unsigned long size) */ bool page_huge_active(struct page *page) { - VM_BUG_ON_PAGE(!PageHuge(page), page); - return PageHead(page) && PagePrivate(&page[1]); + return PageHeadHuge(page) && PagePrivate(&page[1]); } /* never called for tail page */