From patchwork Wed Jul 21 23:05:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12392473 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B9C7C6377C for ; Wed, 21 Jul 2021 23:05:40 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D101861029 for ; Wed, 21 Jul 2021 23:05:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D101861029 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1AE026B0073; Wed, 21 Jul 2021 19:05:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 09CA76B0074; Wed, 21 Jul 2021 19:05:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D90DD6B0075; Wed, 21 Jul 2021 19:05:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0095.hostedemail.com [216.40.44.95]) by kanga.kvack.org (Postfix) with ESMTP id BABE86B0073 for ; Wed, 21 Jul 2021 19:05:37 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 4E4C718591F5A for ; Wed, 21 Jul 2021 23:05:37 +0000 (UTC) X-FDA: 78388128714.28.05BF695 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf15.hostedemail.com (Postfix) with ESMTP id AB2FAD008954 for ; Wed, 21 Jul 2021 23:05:36 +0000 (UTC) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16LMvG6u009329; Wed, 21 Jul 2021 23:05:32 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=g+6VZGRQSNnCGSvjXcCWDjj7JZh/wgzNtXrvPGSgtPs=; b=gEIPkGCuq/xbqaf9dusP7NleUhO0ZwjPAjMTluxHAW2OcsX4IyHXty0Onz8MAxgl3eKL HQYM2pidcraQ/dq2sE8vwfhgdNTkU/ffgUknlwC9arNNibrXp0NJADuM3YcTyQzUSbbg TdHs26GypPzoKkHtoIfsxEStsshbqH0t1qe++5PMkg17HLofcgv6U5O/PF0x1P6HRxFW i77nT6JuxPMMoe5UZm16RcBIfigjMrOMFyjt4qojafxKGquLCZbcIiZxNzIwcbpMY6zG tu7KAV80ZidQRdW1vFD2SUV9trcVElItCSOfgGhthkyEEeBhrWgnyb1zEKNsAeY3FZO2 uA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=g+6VZGRQSNnCGSvjXcCWDjj7JZh/wgzNtXrvPGSgtPs=; b=qtwxDFJa8pC1mN+WwY/aAzYNhgbAscZ8bl6zcGRBJHJqiAcg0FdY970QYG1UaAeaYIJl F9YMQdLW5PC/WrzRM1iFnOonlFSEPV5+oHe7cUWGDEI3VdQVljIcCxzNUU9j5WPoJIEy mkU4hcEzFzxB/ssYkhupzMN9BsCPCUY/BnBhMvaSs/WfnOWnH77GMdKkWGussiQGOfyp XsjRT2Kyt78F8c6WqoVVucXMzoDA4y91U0RfOVhZnf3N9hVSNErLPaM5raxC77M5txYP qJ6/alqcI+uzRijyD9QiSOeZNL+EuX64PqxwP551ceZUwfALiTzQ3DXqneblBVfMi8mp PA== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 39xu2fr5wh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 21 Jul 2021 23:05:31 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16LN0Z0h143558; Wed, 21 Jul 2021 23:05:30 GMT Received: from nam04-dm6-obe.outbound.protection.outlook.com (mail-dm6nam08lp2047.outbound.protection.outlook.com [104.47.73.47]) by userp3030.oracle.com with ESMTP id 39umb3qdx8-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 21 Jul 2021 23:05:30 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nKMNnt03KUZrlLaoursapZoUeP67Ys3e4e+F7VbRp9EcRRSgYkv991E5nDJFylXbb1nWjhfyNWxou2knIMvQfjzB1rxDZZ62mwoWgjW8Xbc+7mjKh09J0RKvmTUFSAChJOKji5aKd3ZJ3mR5i8jLNS89b1Jn8eXvtC/61AofaP/qZqFRSxPUZE76GUbP8n4/5x2uj46L07XppDMVYVSNqTrel/hIu3s5R1Rroa0BffeB2TbrQRFlTECkTraWF8pBAR/tWWX6OV/6EwpyOvoXhMT94HGOTySTOb6i+aPg1oHcb/bzBA2Ryqf6qUtAjmxdRZmBH7VGQY9t0TiXjiO8jQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=g+6VZGRQSNnCGSvjXcCWDjj7JZh/wgzNtXrvPGSgtPs=; b=OzsXu64JkzcGteUh2n0BYK6LdtVkUXKP899JNafYItKwBEkwg6Fn38OBrI0tpQJkksKjc9jfZIB6CHDCLJGHoI31g5GlGNFmuHfSeP5jZwd9FPHr4nI6A3xBGdKhQPlXxKPE5dBkz5l00xWFEHPBnkMdgJX+U1BP4+EgBuhm3U4fSNYkRPC0ArU53et4bq6oqoUGN1895E9xUWqVMPKU1NFyywiz4h1pyAUtIC8P0oYg9ruYezAsnCbJVlf9x2Oonvw9JOVJBo5TmNT3KyjpIxTd6aTa4ia5GD+ohTlUFao8P65CUCIua0sBW5uheZKk3bMXijT87Jxy27HU+qUapg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=g+6VZGRQSNnCGSvjXcCWDjj7JZh/wgzNtXrvPGSgtPs=; b=S8ZMVtdNBmGupWrR4i7l9CESLX0BxizUKZETR5m/YX3qJEjfKghJ7mGzOD16qkYZnBD8gGrZpL8WfbVK46Pkb9ZbgjPRQCbaswhJCKhsVx+noonLj8HdS+AzpR96UTOdUf3O/mg6k4WreYo73LrrwGtuDEOM1TfNTecqJtexrXM= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3735.namprd10.prod.outlook.com (2603:10b6:a03:11e::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.26; Wed, 21 Jul 2021 23:05:28 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a%5]) with mapi id 15.20.4331.034; Wed, 21 Jul 2021 23:05:28 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Michal Hocko , Oscar Salvador , Zi Yan , Muchun Song , Naoya Horiguchi , David Rientjes , Andrew Morton , Mike Kravetz Subject: [PATCH 3/8] hugetlb: add demote bool to gigantic page routines Date: Wed, 21 Jul 2021 16:05:06 -0700 Message-Id: <20210721230511.201823-4-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210721230511.201823-1-mike.kravetz@oracle.com> References: <20210721230511.201823-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MWHPR1201CA0021.namprd12.prod.outlook.com (2603:10b6:301:4a::31) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR1201CA0021.namprd12.prod.outlook.com (2603:10b6:301:4a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22 via Frontend Transport; Wed, 21 Jul 2021 23:05:27 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3cdbd765-48b1-41d3-79df-08d94c9c080a X-MS-TrafficTypeDiagnostic: BYAPR10MB3735: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1RBMTsoYgi8Nd19JOyby0SZZQicMAfFpVYuLblLtexl+J9nGS+gqFpRZxJR1/eBD2dWnTpiZPjBayxlbhkORNrYyqpMYvXN9AawRKOcqqLKeUiHf3180si+86b7iGdh37G3HAPQVb6o71LYg85aH4uGgMjn2iSvPT8Qcvts7dWMEip8KXDDTWryh+qnUPn0colkSPX1zglLrbOk+i1bsij0/poGaxw9MGFfrzNFU+v1X9dI67COb/yr+fNI+QNIqcOUo8420931uAxf+WI6lejFVaB0he3fC+BWTMcX0ynq/9EqCtdsbGkCOb3YUdjOoGwxSmss9S1c9+86AuhUlGzgMjjCykKL5AXmLME867+hpt1s+2XLyPwREiSsXbUucp4Gv3aJrSgMIaoftvuLZ/Pg6uoZ5fbcpCF6KPSU4HneITnTkz4jEDsd2dgnAdN3826n4o+S382x0zktfnR7hQ9vrrvnv0nTZwFZafmwZ6srzIEbN9RpZG7LypZqABh8S9MWJHipGAzP13AgNUO6S9LV27XQcGjoxQ1bDK3m0rUXiZSC4pScM4ifHu3YLqlce+KHOCpJFC7Jw52fF33Hwm7ecc8hYnfMverSPRr4cKkWL4XZ0/b0CtO7eNur6qz54xycPCQleny4CpythW6bgQAKxuf/99nJCujM3WBQZdgVAVR67Yd/2VDZBLapF+CU7Hdy886+4kehZr93dV9qVGg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(366004)(39860400002)(396003)(136003)(376002)(186003)(66556008)(478600001)(52116002)(86362001)(2616005)(107886003)(8936002)(5660300002)(1076003)(4326008)(6486002)(2906002)(66946007)(66476007)(956004)(6666004)(26005)(316002)(38350700002)(38100700002)(7416002)(36756003)(83380400001)(7696005)(54906003)(8676002)(44832011);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: IqjJ+cySHOcDSQE39ZeJASOLHxhQjyQYQTs3SnpEwyHF/jjAezjSw+S+KqtpU30MaXoRzsPkNIPg2dpF4qrwan9m7x20drtNc+jxUEPbrw6A+4KJRFsBqLFfUeDxfcs0VDQ4fLZ0zK2IlmL2KEloSDER2y3n/m7Elx/DIRtURampJtsw6tsakKHAvoxgYy87AWHw9FAswBb7Zj1nuyxw85sizqOpCLSSrvqn0f4rN3MvRTNAUMEOvm7w1FFXBZoJv7eR07VYAICEu28mWxPDWG+McZwv4yY1hTm21PakaUazqDbViyNqcRsGkcXGmAhpovOnJCHLR73DPV/h/CJrrrDwSbM/HyR9v7/iugi+A5ytlPCMKDT+MSDnPFn1CxbxF3swBa7FF2ucjN+TCFTyH5i7ges8IlMEeOA0t0l+NFxJz5itx3mRrfbxqSoukLLBd3X6IGYC1czx1hnWC+/d/Nlsf2DkVEHH9UNYRjKv4eDSrUwg0wlC3JCYyPX3LzY+xLz8xnjyQDs9O6H1mwRB8rM3wQZSCj4rVkQqvqKyMSjfaG0Z016P9YA3ttUq8QW9nfYTJ4eRY0gRQGG345sw1wBiL1iOpsSkNiopCp66Kq//HvY1WZp1HgG3dIRP/VTYGxbBixF7Xr1nDm+e45M4S2hARReyMlHJfNB2HcgYyZmHqUyQn0eK023uMbdrQsHpvc1DeWxoTz52V8/EUJG3FklgTPrF5s+lO9NXynad8nNnRidu0WT5nCwB50jMIf94moZwSlycl7TTpNPySjwS0K9freQ/yguJ1ESj+oQXcTt4AOc5c4abd7z8npb+8nqNXN2fTctSNQwRWQtBHZoxQBLCpcpbictZ1NDFBXelKD+7bbooSsQdiXrx4IoXakGD08F99uiZ5N2aFrTlR0t9QaNFEJb6K7O/E2fwiGEsqEqw3GFZwF2gcRm61P3Diqm/VcB5ruKkFv8T+o7lY7k3Ux4z09fmbiPdcR4mBqPURJdlxZuvsf6nDENpOPs/YSk27hD4lrPuCz3uyaYPpj5JvoD1Y9FEwG85BogQy+hebm28VZTEf9g06KDnBjhyGuXNj5zlDSxHFq/rdQztGwI1A7c+FphcH7GG4A5acbsnQk5yFYqrVPKRNTYWuxJz8sipubzsLgyUKXcEoGmbbG1m0IVn+ZQntmZdUUS3cKE3zcqFNR3YAlFLCoG/nPN/Gvcmdurc46ajTt3K3hriGUFes4VNtW8NYkjUZ14qeKz1P51zvYa1wiXhApVfv5a9Oj4VSn/pwkQ5Vied4FYIE84HxVPj8d4JnqK6a2Bf8QC00g5hhZmF7PB6IqMnVwIrIvKZ X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3cdbd765-48b1-41d3-79df-08d94c9c080a X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jul 2021 23:05:28.5862 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: aqL8FYER7HvI3f+Mtk+wFB3t8B8UKR8lCBjHrjmkBKypU2kBxRtkcXs0t/90EOetuie+lmE/mPEyOrNwPd1g2Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3735 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10052 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 spamscore=0 mlxscore=0 adultscore=0 mlxlogscore=999 suspectscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107210137 X-Proofpoint-ORIG-GUID: CL-P7M6HLSveF38KOGyW2bRvyl444ucQ X-Proofpoint-GUID: CL-P7M6HLSveF38KOGyW2bRvyl444ucQ Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=gEIPkGCu; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=qtwxDFJa; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=S8ZMVtdN; spf=none (imf15.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam02 X-Stat-Signature: myem4fc8s3yb14wtyn4za7rzwtt5fhkt X-Rspamd-Queue-Id: AB2FAD008954 X-HE-Tag: 1626908736-148589 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The routines remove_hugetlb_page and destroy_compound_gigantic_page will remove a gigantic page and make the set of base pages ready to be returned to a lower level allocator. In the process of doing this, they make all base pages reference counted. The routine prep_compound_gigantic_page creates a gigantic page from a set of base pages. It assumes that all these base pages are reference counted. During demotion, a gigantic page will be split into huge pages of a smaller size. This logically involves use of the routines, remove_hugetlb_page, and destroy_compound_gigantic_page followed by prep_compound*_page for each smaller huge page. When pages are reference counted (ref count >= 0), additional speculative ref counts could be taken. This could result in errors while demoting a huge page. Quite a bit of code would need to be created to handle all possible issues. Instead of dealing with the possibility of speculative ref counts, avoid the possibility by keeping ref counts at zero during the demote process. Add a boolean 'demote' to the routines remove_hugetlb_page, destroy_compound_gigantic_page and prep_compound_gigantic_page. If the boolean is set, the remove and destroy routines will not reference count pages and the prep routine will not expect reference counted pages. '*_for_demote' wrappers of the routines will be added in a subsequent patch where this functionality is used. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 55 ++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 43 insertions(+), 12 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index d034a223d5d7..0ad75e7775dd 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1251,8 +1251,8 @@ static int hstate_next_node_to_free(struct hstate *h, nodemask_t *nodes_allowed) nr_nodes--) #ifdef CONFIG_ARCH_HAS_GIGANTIC_PAGE -static void destroy_compound_gigantic_page(struct page *page, - unsigned int order) +static void __destroy_compound_gigantic_page(struct page *page, + unsigned int order, bool demote) { int i; int nr_pages = 1 << order; @@ -1264,7 +1264,8 @@ static void destroy_compound_gigantic_page(struct page *page, for (i = 1; i < nr_pages; i++, p = mem_map_next(p, page, i)) { p->mapping = NULL; clear_compound_head(p); - set_page_refcounted(p); + if (!demote) + set_page_refcounted(p); } set_compound_order(page, 0); @@ -1272,6 +1273,12 @@ static void destroy_compound_gigantic_page(struct page *page, __ClearPageHead(page); } +static void destroy_compound_gigantic_page(struct page *page, + unsigned int order) +{ + __destroy_compound_gigantic_page(page, order, false); +} + static void free_gigantic_page(struct page *page, unsigned int order) { #ifdef CONFIG_CMA @@ -1344,12 +1351,15 @@ static inline void destroy_compound_gigantic_page(struct page *page, /* * Remove hugetlb page from lists, and update dtor so that page appears - * as just a compound page. A reference is held on the page. + * as just a compound page. + * + * A reference is held on the page, except in the case of demote. * * Must be called with hugetlb lock held. */ -static void remove_hugetlb_page(struct hstate *h, struct page *page, - bool adjust_surplus) +static void __remove_hugetlb_page(struct hstate *h, struct page *page, + bool adjust_surplus, + bool demote) { int nid = page_to_nid(page); @@ -1387,8 +1397,12 @@ static void remove_hugetlb_page(struct hstate *h, struct page *page, * * This handles the case where more than one ref is held when and * after update_and_free_page is called. + * + * In the case of demote we do not ref count the page as it will soon + * be turned into a page of smaller size. */ - set_page_refcounted(page); + if (!demote) + set_page_refcounted(page); if (hstate_is_gigantic(h)) set_compound_page_dtor(page, NULL_COMPOUND_DTOR); else @@ -1398,6 +1412,12 @@ static void remove_hugetlb_page(struct hstate *h, struct page *page, h->nr_huge_pages_node[nid]--; } +static void remove_hugetlb_page(struct hstate *h, struct page *page, + bool adjust_surplus) +{ + __remove_hugetlb_page(h, page, adjust_surplus, false); +} + static void add_hugetlb_page(struct hstate *h, struct page *page, bool adjust_surplus) { @@ -1669,7 +1689,8 @@ static void prep_new_huge_page(struct hstate *h, struct page *page, int nid) spin_unlock_irq(&hugetlb_lock); } -static bool prep_compound_gigantic_page(struct page *page, unsigned int order) +static bool __prep_compound_gigantic_page(struct page *page, unsigned int order, + bool demote) { int i, j; int nr_pages = 1 << order; @@ -1705,12 +1726,17 @@ static bool prep_compound_gigantic_page(struct page *page, unsigned int order) * the ref count to zero if count is currently 1. If count * is not 1, we return an error and caller must discard the * pages. + * + * In the case of demote, the ref count will be zero. */ - if (!page_ref_freeze(p, 1)) { - pr_warn("HugeTLB page can not be used due to unexpected inflated ref count\n"); - goto out_error; + if (!demote) { + if (!page_ref_freeze(p, 1)) { + pr_warn("HugeTLB page can not be used due to unexpected inflated ref count\n"); + goto out_error; + } + } else { + VM_BUG_ON_PAGE(page_count(p), p); } - set_page_count(p, 0); set_compound_head(p, page); } atomic_set(compound_mapcount_ptr(page), -1); @@ -1733,6 +1759,11 @@ static bool prep_compound_gigantic_page(struct page *page, unsigned int order) return false; } +static bool prep_compound_gigantic_page(struct page *page, unsigned int order) +{ + return __prep_compound_gigantic_page(page, order, false); +} + /* * PageHuge() only returns true for hugetlbfs pages, but not for normal or * transparent huge pages. See the PageTransHuge() documentation for more