From patchwork Tue Jun 22 02:14:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12336305 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9AAEC48BC2 for ; Tue, 22 Jun 2021 02:15:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4E8E0611C1 for ; Tue, 22 Jun 2021 02:15:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4E8E0611C1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E2FD76B0062; Mon, 21 Jun 2021 22:15:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DE05F6B006C; Mon, 21 Jun 2021 22:15:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C32986B0070; Mon, 21 Jun 2021 22:15:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0133.hostedemail.com [216.40.44.133]) by kanga.kvack.org (Postfix) with ESMTP id 8DE8F6B0062 for ; Mon, 21 Jun 2021 22:15:09 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 31972180AD838 for ; Tue, 22 Jun 2021 02:15:09 +0000 (UTC) X-FDA: 78279742338.01.0E61354 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf10.hostedemail.com (Postfix) with ESMTP id 927414006F01 for ; Tue, 22 Jun 2021 02:15:08 +0000 (UTC) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 15M2CFWx002119; Tue, 22 Jun 2021 02:14:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=mbv2Nd6UumpVMG3C4bZmcm9fxjOu05OSYSd0BAAzfR4=; b=F1KRaT5gpWkaFFeHj4gDAzcmink1fqGBfn2GE3Xv2IYAVFKGyRcBGlOlu2MykbNxx5FW oLyXMN4CYjI0tNdI2CS93rvxTLtw9afYYDdzXEANoeHQ7iVqsTdu4lrcIPirD2vqekMA YStm5dnqv6Nt+liI0jpDDbovCgcryHYf3NRtV0SCH7vab2UWi1KxZmJVMjn6RMnxstHZ r9NcH9SlG6B3fh1yjt8VY345i0epG/vxF3myiFvN0NqP2FAnrnl5iYHiT/686w1MEosR WW84YOUbuxsiphMdpRG8b07eBZnYJ1K09cKIefrA0MgyOwhZpBSwdH6tVKIpnSqReQx9 Rw== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 39acyqandc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 22 Jun 2021 02:14:44 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 15M29aL0033420; Tue, 22 Jun 2021 02:14:43 GMT Received: from nam02-dm3-obe.outbound.protection.outlook.com (mail-dm3nam07lp2041.outbound.protection.outlook.com [104.47.56.41]) by aserp3030.oracle.com with ESMTP id 3996mcqnbh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 22 Jun 2021 02:14:43 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EOtierj9P5S7Hhwb3Y0BZQMct1Wv8Jn6SpLvDPqZr29IoSOU3GRrWp08G1dCM9SzBDLGkZHT66W6ickvEiBkG+p9lyfEWlwq5dBrDSVMoJmmoRgef9g5XJftfw2F2vb1CnGF3/YxB9u3JNG1ngTgdrKnGNLLQSZZ3DF45FORfAo2YRurh47HuM2g5yVIlSvZlpUOU7RUMh46AGns1k+NP2r+m+7yFSK8xL7wPRZLWY8i5O+S4TE589fe1iBwVq30gI2+9XjSqn8GCHCiVetq5T6s4qyW/IRtjoGq4+eZArM3cAcm+wQUItuNfKk+FE3Mcdop/9r5CClbBN+5Ifsn8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mbv2Nd6UumpVMG3C4bZmcm9fxjOu05OSYSd0BAAzfR4=; b=nG4vyKnxt9XmhBgmALXUSmq+aoTyznVgDS9uLyNhQLUI9IFFiJ3R8NUGG1tQcqMuGN6lS+RUMwAjdsjH323Smlh6Ap8sa8u4gQY4QzO6hg7DqS6DNsGYCoE0KPhvkg5QO/fY9jdE+bhKRxrofDsjN6+7HqwTJZNIgP2hwzuHaX3Jlmm8oDtdQGvXYggare9R1bVuMfNbGGWga+hTitclL8FYLRrM3eWiQTL+dyJYbHcSAzp9k7y1eq0OwMpr4N5N56L6dn+rs734WPlT63e6L7O93jd5Ba8sCTKT82jIXNq9SKPIqeX4MCGlaU6XbXAyZJLYthQ9AfTImHiJ4AzRCQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mbv2Nd6UumpVMG3C4bZmcm9fxjOu05OSYSd0BAAzfR4=; b=lloIpo3yvm3yb0z3qZsRuaLI0FQbzw094FUPccCB01Ro+T+BDKXPfRnGTuzkhgsHmTQh1YGcbHjKKo7ivRfeSEDlEEH11+GD8t2KXl+M35hXeg1zy7c22WWdLGKJJMUZBoXOVIApM41HmJkQTwWpZ56mG2lUD7ABQGCfnxVRFRk= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB2664.namprd10.prod.outlook.com (2603:10b6:a02:b7::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.16; Tue, 22 Jun 2021 02:14:40 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::5d11:ae6d:24b6:e838]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::5d11:ae6d:24b6:e838%4]) with mapi id 15.20.4242.023; Tue, 22 Jun 2021 02:14:40 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Jann Horn , Youquan Song , Andrea Arcangeli , Jan Kara , John Hubbard , "Kirill A . Shutemov" , Matthew Wilcox , Michal Hocko , Andrew Morton , Mike Kravetz Subject: [PATCH 1/2] hugetlb: remove prep_compound_huge_page cleanup Date: Mon, 21 Jun 2021 19:14:22 -0700 Message-Id: <20210622021423.154662-2-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210622021423.154662-1-mike.kravetz@oracle.com> References: <20210622021423.154662-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MW4PR03CA0072.namprd03.prod.outlook.com (2603:10b6:303:b6::17) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MW4PR03CA0072.namprd03.prod.outlook.com (2603:10b6:303:b6::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.19 via Frontend Transport; Tue, 22 Jun 2021 02:14:40 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: faa9ffb4-9b44-4605-dad4-08d935237e27 X-MS-TrafficTypeDiagnostic: BYAPR10MB2664: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: O3rp7unXtC3bFcY1KxIDhojrgTXFW6SowawVG8l3cWoShs1hfxNFQyJTWP2vEDVtFT/Ubw3RyNTP2cla20PIUlkh9nMs8IbDhc0CqdpMvA4simjyjx38YeBsUckiJDccxjcU0nkvCKl5dAgHa+9PnK7+to7yDinD5RZfCfizpPokXR9Pdu9+uomCJPzXNRLU7X/JwplNq9Bifm+mABJBwcAzmY3IcwUmni5XbomA+sgOSjDhAgC1UR2ec0SQdN4OCl+9yPFAdowx1qjpoLmndFyrF8zq1UVmP8AhwoiHgHsk/eM4U41xhyMkzsxCG1WCfSEJ8ySQaFTKsHqBfnywDZ/c4eUWEoM6N1CKRQTbgo5KsqSOXzlKrIKtpHEiZnrIiMA95JKDsAr5+VPpJkIkfsEcclQJ6eBVE9WcnpgE7RWy7wIEGC6cfakk3zSiSz3dJUJfNNh3bdbPiv03rnUYqVsxm6DeY9c//nvVp9JlAuJE72r5CsTn6iOjTQ+Rwdp9kAB/4HFvMgM2WkVE4Tfg/6J0+bYc6qXnEe5grCpPMwQz8tqAYUW9NUxZdE33fFK1tgzAc69fYqb17q07/wB4Up3cnz43ODXSLGt8loRbX2ygXKiMmDAtgLxt79dJ/20ToPBOUUeemCx6QjzvkuEBxBgAq1Lg+GFlG5JE9qCaN7k= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(376002)(396003)(136003)(39860400002)(366004)(346002)(52116002)(16526019)(186003)(66946007)(86362001)(66556008)(5660300002)(66476007)(8936002)(7696005)(2906002)(7416002)(1076003)(26005)(8676002)(36756003)(478600001)(44832011)(107886003)(38350700002)(38100700002)(54906003)(316002)(6486002)(2616005)(4326008)(83380400001)(956004)(6666004)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Ma1Eb6OlMV5TC/AH1DAFvIG3Sn/xwKwNmYGHgLQ9wspkbLxVTjV+cNVE1eDeaNODVT1nsWtHpYgVtiEdWPaGzdXpBnpuvbyQKhx2RJdd0qlcL5zFUmktJI8E8Y+Wt/c64rdfSz4HudpIGk8KJngfPnQYB5R9Ditk0Vhh1XCeXLqvf1f9cW7CP2mlSuo85dvQwwJeanTr9L+ug35GH5MHlC9J9gu6AaMdvPaTRQ5IOc09cZ0nIHTTBcVAeCDqba4GcnZD+5W5SfDZjh9Xxe1yr4hASyCrYqfdwZ1RUSOw7vRdJGf7JNxR8quYfjwQsY4hQ77OB747WOBMtW/MR8s9a3hWFX+UjzJpdDGFVA/hbE/dymP4T/qPuRmf/0QSGgwWiPllV3CvVEWFqdEeEhP2UOAAA4Zbr5sW77iPdov9U25HJAp3C1R9mXkOB/dCOBMObB5Fk21gHFoAt98RZaguYMdZENTa+reaII8ftkmFnKK+23eJ5JSDxxVw9Ar+IIgArelGWPcKURJJ1VJ7tZbtfo+PdqKiCXe7pDsrijTaRgIqrtCyiC34KCdMd4OzyDPwDVTSqLruyn16CcQrssTGbvUOgmxQmNnDAUhT2qMfUUjYX7eYIibQlVlHocG6dZJZH3AwWnc4C/D9qVVmMCFd3Uf3vh7ZfL0mWmQCVtZjZ3tFqwbzbwxeOCm8gJCOy/pfmRHIWWWG6SfEAh5uFOW8GsMBbhrPgz/hjDsFd18tATafWnWQF7lBsU5wKTJ+qlsjk6xDxufIIPl59mzzEx8EvXe+GQueBTwK2wWwFYr/DikEBUWPl5Ebw+9QRFH9xOGvx75EVXbAGqxLpEo0R/mexk1tM5561t6cphdNL/sry3rjcLZSDGVdxR9HMQ7OeJv9XHbE3vK68Mp9ptBBXGKvDwu5m6cTf1qyQFKIxTqsTLQxIyZpRkDdrBkCJhnlbKaFAhVDtzINNdXIeOgVAYSeaogO+g6yNDb5A8kV+pycDQk5dwub0SUnj/TBZ2lv09y+jC+Km3+YfjTfOLHvzRaCu7qDCtUyy2IiWIthseDP4pwUVCsp/wHutV/rSHBLlDOdNUUp7uR7d+pTjgb2sjm7wXclFocDSx9kndDhx6YfDOz+KqY5ekJ6IpOYawvhX3JQn1Pal2VwXMIOkRz5d7Rv518PP1df9umyAoACD9RBLQU7isCbxNifELldJkSlUTa4TldRZIJh+D19/QN4Hf6ygmWeZz48RQIlz7c8yrc2l8o/0CkxABA/KTuF192UmZa7/Yz4K7KrF8jpyfpg8tEBRmBJIkyzw2e7fZzRZN6RElh6Sdopigakim29a73bnmyD X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: faa9ffb4-9b44-4605-dad4-08d935237e27 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jun 2021 02:14:40.7837 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: YtGxJ11Q4S8m5O0rRsTWVtg99LWdDhe07bYKaBOfZKggApy7jUJLNoqBwL0VcSTlsNwrm4oZ5pdA4zTRwEImVA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB2664 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10022 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 mlxscore=0 mlxlogscore=999 adultscore=0 phishscore=0 spamscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2106220012 X-Proofpoint-GUID: BMyGzsjoWissqy57CebtcF3mvCvYrhgn X-Proofpoint-ORIG-GUID: BMyGzsjoWissqy57CebtcF3mvCvYrhgn X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 927414006F01 X-Stat-Signature: utirr7iq3iozz536kwbsqetkujz7bmx8 Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=F1KRaT5g; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=lloIpo3y; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf10.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=mike.kravetz@oracle.com X-HE-Tag: 1624328108-960231 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The routine prep_compound_huge_page is a simple wrapper to call either prep_compound_gigantic_page or prep_compound_page. However, it is only called from gather_bootmem_prealloc which only processes gigantic pages. Eliminate the routine and call prep_compound_gigantic_page directly. Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song --- mm/hugetlb.c | 29 ++++++++++------------------- 1 file changed, 10 insertions(+), 19 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 760b5fb836b8..50596b7d6da9 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1320,8 +1320,6 @@ static struct page *alloc_gigantic_page(struct hstate *h, gfp_t gfp_mask, return alloc_contig_pages(nr_pages, gfp_mask, nid, nodemask); } -static void prep_new_huge_page(struct hstate *h, struct page *page, int nid); -static void prep_compound_gigantic_page(struct page *page, unsigned int order); #else /* !CONFIG_CONTIG_ALLOC */ static struct page *alloc_gigantic_page(struct hstate *h, gfp_t gfp_mask, int nid, nodemask_t *nodemask) @@ -2759,16 +2757,10 @@ int __alloc_bootmem_huge_page(struct hstate *h) return 1; } -static void __init prep_compound_huge_page(struct page *page, - unsigned int order) -{ - if (unlikely(order > (MAX_ORDER - 1))) - prep_compound_gigantic_page(page, order); - else - prep_compound_page(page, order); -} - -/* Put bootmem huge pages into the standard lists after mem_map is up */ +/* + * Put bootmem huge pages into the standard lists after mem_map is up. + * Note: This only applies to gigantic (order > MAX_ORDER) pages. + */ static void __init gather_bootmem_prealloc(void) { struct huge_bootmem_page *m; @@ -2777,20 +2769,19 @@ static void __init gather_bootmem_prealloc(void) struct page *page = virt_to_page(m); struct hstate *h = m->hstate; + VM_BUG_ON(!hstate_is_gigantic(h)); WARN_ON(page_count(page) != 1); - prep_compound_huge_page(page, huge_page_order(h)); + prep_compound_gigantic_page(page, huge_page_order(h)); WARN_ON(PageReserved(page)); prep_new_huge_page(h, page, page_to_nid(page)); put_page(page); /* free it into the hugepage allocator */ /* - * If we had gigantic hugepages allocated at boot time, we need - * to restore the 'stolen' pages to totalram_pages in order to - * fix confusing memory reports from free(1) and another - * side-effects, like CommitLimit going negative. + * We need to restore the 'stolen' pages to totalram_pages + * in order to fix confusing memory reports from free(1) and + * other side-effects, like CommitLimit going negative. */ - if (hstate_is_gigantic(h)) - adjust_managed_page_count(page, pages_per_huge_page(h)); + adjust_managed_page_count(page, pages_per_huge_page(h)); cond_resched(); } } From patchwork Tue Jun 22 02:14:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12336307 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7BBA9C4743C for ; Tue, 22 Jun 2021 02:15:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 139C9611C1 for ; Tue, 22 Jun 2021 02:15:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 139C9611C1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AD5D76B006C; Mon, 21 Jun 2021 22:15:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AACBE6B0070; Mon, 21 Jun 2021 22:15:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8D7A56B0072; Mon, 21 Jun 2021 22:15:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0117.hostedemail.com [216.40.44.117]) by kanga.kvack.org (Postfix) with ESMTP id 5AAA66B006C for ; Mon, 21 Jun 2021 22:15:14 -0400 (EDT) Received: from smtpin36.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 02F92C5B2 for ; Tue, 22 Jun 2021 02:15:14 +0000 (UTC) X-FDA: 78279742548.36.961B6A4 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf21.hostedemail.com (Postfix) with ESMTP id 44990E0009BD for ; Tue, 22 Jun 2021 02:15:13 +0000 (UTC) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 15M2AjCw017377; Tue, 22 Jun 2021 02:14:45 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=dYQZpm9qlCJHQql8M5RmjibYwCoUzVft18eztzISBS8=; b=R0yYlhN5/MlbzXb99wpeFToOmHFox7jd+rteBjrRpHB4Rrj+0cU6S5H/5W+z28eouzl3 nTeGPBmawSdmYjT61Kbs7D4DvsFEOl1eYbRQNp2Q0eUbjn/NGHh2GM1xps/uENQYOGrc WB1M/AGDnHlSn29ql+5/ek5OXCaznX05AIVB6DvaOnLhmZhhpcC/oUI6RMsqodU34t0J YKdDVMHMY+JNDvfI6bz2vrNmAfTeY9JM0hChbAEkyKx7VmO2CdxAlqqOmenWOMd5/xcY /eIsdc2/fIfKv6iLxSpcqsyFwMa82jjY9yD2VcAwvUVSLcaIZk4q0u53jQBFnGYjaL8c 4g== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 39anput4k5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 22 Jun 2021 02:14:45 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 15M29aL1033420; Tue, 22 Jun 2021 02:14:44 GMT Received: from nam02-dm3-obe.outbound.protection.outlook.com (mail-dm3nam07lp2041.outbound.protection.outlook.com [104.47.56.41]) by aserp3030.oracle.com with ESMTP id 3996mcqnbh-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 22 Jun 2021 02:14:44 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FTiPJ1Aek0yibIykvzq6nWgv+CHhXrowdLgCnL2xB4XvA/yCjdSurku8/OXZW63HEzvyyrwIIjQungH2rTdvOpRhDYzYKqDOOATkx9pGtCyEreyLD4KiGmKXJwdJeWK0b3f6LtY5mriTawLP33A0MkZM0iZSL3tlhrr826YN17++n4QlZcqRbAnq+0fLDK02riK5ib7zKscgT+4/Afv3Tspap3ru2dFlKrsjdBhyozOWaCB71QT6cRMADIu5s02QYJ369cJBRWYQ3j2PdsaUbwdbdsQVefcP3c2fGDeCPwG0GF56wpxcXb2/Vh4i7PBiae4vnpFHs12YYBhy3HWsmA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dYQZpm9qlCJHQql8M5RmjibYwCoUzVft18eztzISBS8=; b=MUTfCWOHVFdu6ZvpDB/W1UETAXqMqEU9GQXrW8Puysix4ceNO+ApyTs5OKKMhX5/iCMVfrxyKKI0jVbK/+f5bht5fqwTsdB0lYtphIA+DqLB8U5fdEyTaHH5qGs3twlQd9T8D69CX+gHcYe+RX7D/+RRfpTUjtBw0OBrqHJBd7wxs27RLMCudC/0r+7IgvuHbP7VpAH6ppp/wsMKiiYQrUkq0an6b4wqwbjc2UTMcgVUFlY17iKQfQOLwX/sDETw6+Y/bDBgbiETimb0xIoJ7fNlu+Y2I7DadpAZ8yaXY+iWXyXlHW9snBfXQNtSkMvDVa4yhazeqD5TkZQjr9IhSA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dYQZpm9qlCJHQql8M5RmjibYwCoUzVft18eztzISBS8=; b=qR6Xa6SQOgvBBPLZ7dPhKsh4uiVJW1MpHGlk2YRHte375cWvqWlKDoFTmigNA1RtH3+s2Y5BJrmVy8HzBVzIegEyj54nQ7WUfCcqRkMKaL60iEjLyS1tcXbWC8u+GHEn4H9xXxaOTA28ipRIv78LL1Qh98/lV9bb2+6CQLRtr4g= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB2664.namprd10.prod.outlook.com (2603:10b6:a02:b7::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.16; Tue, 22 Jun 2021 02:14:42 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::5d11:ae6d:24b6:e838]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::5d11:ae6d:24b6:e838%4]) with mapi id 15.20.4242.023; Tue, 22 Jun 2021 02:14:42 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Jann Horn , Youquan Song , Andrea Arcangeli , Jan Kara , John Hubbard , "Kirill A . Shutemov" , Matthew Wilcox , Michal Hocko , Andrew Morton , Mike Kravetz Subject: [PATCH 2/2] hugetlb: address ref count racing in prep_compound_gigantic_page Date: Mon, 21 Jun 2021 19:14:23 -0700 Message-Id: <20210622021423.154662-3-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210622021423.154662-1-mike.kravetz@oracle.com> References: <20210622021423.154662-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MW4PR03CA0072.namprd03.prod.outlook.com (2603:10b6:303:b6::17) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MW4PR03CA0072.namprd03.prod.outlook.com (2603:10b6:303:b6::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.19 via Frontend Transport; Tue, 22 Jun 2021 02:14:41 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: df7594ba-0947-48c0-0e27-08d935237f1d X-MS-TrafficTypeDiagnostic: BYAPR10MB2664: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 6iOZVCgj25ztiAxgY44OYzazlbRAqYQAekqKdrS6Ylgq0UXzAA2EBkRrezMO5Ma+cARxobcuCcEiX+vgFANs/hcFspLz6bxtRx0EhDHsKdh708CS/LMOqurw/ke51xqc0psoY/+D5TIg7+KmPrCsUVSpaKFSup9hCQ1RRTSieSCR0LrnmE6cKyAMH08RmnkPxdaspVDVwXD2nXGhxucdzcylwS8kA0ECeUYn02bYLQwz48OFt/gWKsL3G31G9XjAqqFue6gWxnopAoDbYQaKcTSq75CjZUP1HIoTuDWVyswzrziFuj//B/DzzhPOBGkbXO+B1A2JrHY+wXpUd9Rs2tMqF2j9A8RxTSaRQsojD+TpDHySsjconYnW5g3tzBUcxpjcjoqVpwzNfFNcdmQ5Hceau7pIfWtUCLlEmB0K1ZPvSqnaf8xW6m3WBPHxbJ609mPzLUKbQ2Oc+ziyMK2YIem3wpTEXx/lgGNBGd/NV7pjq+w4aG1RXYJQu4B+yTJGPlyM09+yeX3rVhFRr0Z4roLBpMObEIu7TZR0I9g5nop0ZmHosSh8a2NbKydxfYCK0LgFNhg4GOVQnhtF5WnI0E64ok3j3Y2tE6v12KDSr/6VJSAFy+AuFb4KaAE3eDTSMmADB5jSyVI9+Xz8rN60e1TSE4G0EGV5UTWnru7slmqFBGiwKhinEYI2ecmFNbjrBAabHCvRezkW1qxCRZNM2CTn6ynV6goUIIeTmq2jBaUmQmeLhuWIhZKAVtmenD+bq4S46cngJ62SOjZCwCjW6g== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(376002)(396003)(136003)(39860400002)(366004)(346002)(52116002)(16526019)(186003)(66946007)(86362001)(66556008)(5660300002)(66476007)(8936002)(7696005)(2906002)(7416002)(1076003)(26005)(8676002)(966005)(36756003)(478600001)(44832011)(107886003)(38350700002)(38100700002)(54906003)(316002)(6486002)(2616005)(4326008)(83380400001)(956004)(6666004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 9+MFWhlqxhBnlwUdXFlQSwnNNmlSKdA+GFbXdOkkXhy6snNyWuEQDn1CHsT5wWrbicMZxaHm+ukp86s6/2hvftrnQ6TocXnhbk137al0ANk4A0v/ONoDZlzj9MHbm0o7Fwtaaf5xHvxIX2sYEk2yLrsWWre1+uVhggh4EYesjxJZZIKZZrZH0O3Qdtudvbds7HN4hjr6Ejp+FGkiiU4crjxbdPdMgMZvF0cm+eMpJ/XW4eX8Z0mygTDzBcO9PH65TsxVwham+gXvOJBYbUBiC0rwRrUs6UeAGKlfDr8iGSpEAL9VvOEDIqIf13heIV4TG4y1kR6MUmTc40noymRse0TvlL8dqhu2WKKYT29dbjHyQ42Vj6HkZQFmJx9y2fhKy0hZAo7/YIP4vyIDcPojR46cKsu5PO1DX1By0LR+ht0w/oGY+TrgTTj7pvRVTmKfQhoX1zxAB1DDXL1mNQNAzCZ5g2XjzwyKWbvGVyVL9OSn91a5wGsQHLGnM3atQMtgUohg6Yjgdhdgofkkxvf2KDqxd/29Gd5NebJV6svGgLqE0CGo722uOrEOC5+YRZjHhHYSGYVKJAbtDwHNbOip7FHJHPpgumIQw4g7to8Nwn6H3ZrTyDhdG8HhWe1VGTA9QtIzjLZeTra2kRlnWFP6mK2kPNmyJpziTVjNOdRGqhE01pkKJnqDVLUiPgosJq/EJ634pwH4aeekpU62etMFpb0PcJZbvcVSBDL0LsWk7SW9TqWQz+7dS2QrtviUZiYEWqcvfYaq7vm0GSXB9zMkIi4oMJnajS7ZbRoizHmRdF7tUzNZYllIPGMDSjrspqOXj2tsOD+M/NnT7volAUcL3GLyFSO2KYYFx0QV737RCpFtIw8r0d2Y2MVAMNmDBrvvV4nfW73EM/7USvWz5UhTr5zH8L4UXpdMVuVb8pXlav6BC5qneWVTwVMo4M5hXKo7GXIXk/kiLSgWaYTZKRBTxXHa+M3Rv1GwEDfKODL/dDjKpvLt3w3vGtzNHLCCs/CsglrigriFRwmrlqi7iGJefdzWn0IoReXQNV/HIe6/0Ka7/A1+80LuUEuwbEbW0IijxrSLO434oNpRnkEzIhDj+s/hW2PUQvLRyl775H4KAuPHlR7qqL2Q7XwsNYuFejP09AklzmxPU833Vo8IDMGuleJF9hJ4sHbC6RkB7Ess6nDbaucIxWnOUQ4EIkVrUc8u+B+m4tuuTbK/WetcL1cLcAGgVlkeKMRhIV1bXUAxQiqiHCF1ZjtEOJQx9GAH4HOaFAhdzRayZQ4EJwn+Rf+O4NQxB27Cg3hUdvPkgitN8KZgckUPruZWrvM9xRKE326g X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: df7594ba-0947-48c0-0e27-08d935237f1d X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jun 2021 02:14:42.4185 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: vrNheXdAIiOS+Wc/pVtVTIOMCW6OymXQ+127fTnXEeVK0tQRl1Uep1p6UBT/HyLwmasnKor6grNj3K8s39D8vg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB2664 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10022 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 mlxscore=0 mlxlogscore=999 adultscore=0 phishscore=0 spamscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2106220012 X-Proofpoint-GUID: NGsjaUlOblN-h6p5tPNC7arWFCKyIPpS X-Proofpoint-ORIG-GUID: NGsjaUlOblN-h6p5tPNC7arWFCKyIPpS X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 44990E0009BD Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=R0yYlhN5; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=qR6Xa6SQ; spf=none (imf21.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Stat-Signature: sqcrwkkm1mz7ebywptxcyu4j5tczkmiz X-HE-Tag: 1624328113-20384 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In [1], Jann Horn points out a possible race between prep_compound_gigantic_page and __page_cache_add_speculative. The root cause of the possible race is prep_compound_gigantic_page uncondittionally setting the ref count of pages to zero. It does this because prep_compound_gigantic_page is handed a 'group' of pages from an allocator and needs to convert that group of pages to a compound page. The ref count of each page in this 'group' is one as set by the allocator. However, the ref count of compound page tail pages must be zero. The potential race comes about when ref counted pages are returned from the allocator. When this happens, other mm code could also take a reference on the page. __page_cache_add_speculative is one such example. Therefore, prep_compound_gigantic_page can not just set the ref count of pages to zero as it does today. Doing so would lose the reference taken by any other code. This would lead to BUGs in code checking ref counts and could possibly even lead to memory corruption. There are two possible ways to address this issue. 1) Make all allocators of gigantic groups of pages be able to return a properly constructed compound page. 2) Make prep_compound_gigantic_page be more careful when constructing a compound page. This patch takes approach 2. In prep_compound_gigantic_page, use cmpxchg to only set ref count to zero if it is one. If the cmpxchg fails, call synchronize_rcu() in the hope that the extra ref count will be driopped during a rcu grace period. This is not a performance critical code path and the wait should be accceptable. If the ref count is still inflated after the grace period, then undo any modifications made and return an error. Currently prep_compound_gigantic_page is type void and does not return errors. Modify the two callers to check for and handle error returns. On error, the caller must free the 'group' of pages as they can not be used to form a gigantic page. After freeing pages, the runtime caller (alloc_fresh_huge_page) will retry the allocation once. Boot time allocations can not be retried. The routine prep_compound_page also unconditionally sets the ref count of compound page tail pages to zero. However, in this case the buddy allocator is constructing a compound page from freshly allocated pages. The ref count on those freshly allocated pages is already zero, so the set_page_count(p, 0) is unnecessary and could lead to confusion. Just remove it. [1] https://lore.kernel.org/linux-mm/CAG48ez23q0Jy9cuVnwAe7t_fdhMk2S7N5Hdi-GLcCeq5bsfLxw@mail.gmail.com/ Fixes: 58a84aa92723 ("thp: set compound tail page _count to zero") Reported-by: Jann Horn Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 72 +++++++++++++++++++++++++++++++++++++++++++------ mm/page_alloc.c | 1 - 2 files changed, 64 insertions(+), 9 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 50596b7d6da9..924553aa8f78 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1623,9 +1623,9 @@ static void prep_new_huge_page(struct hstate *h, struct page *page, int nid) spin_unlock_irq(&hugetlb_lock); } -static void prep_compound_gigantic_page(struct page *page, unsigned int order) +static bool prep_compound_gigantic_page(struct page *page, unsigned int order) { - int i; + int i, j; int nr_pages = 1 << order; struct page *p = page + 1; @@ -1647,11 +1647,48 @@ static void prep_compound_gigantic_page(struct page *page, unsigned int order) * after get_user_pages(). */ __ClearPageReserved(p); + /* + * Subtle and very unlikely + * + * Gigantic 'page allocators' such as memblock or cma will + * return a set of pages with each page ref counted. We need + * to turn this set of pages into a compound page with tail + * page ref counts set to zero. Code such as speculative page + * cache adding could take a ref on a 'to be' tail page. + * We need to respect any increased ref count, and only set + * the ref count to zero if count is currently 1. If count + * is not 1, we call synchronize_rcu in the hope that a rcu + * grace period will cause ref count to drop and then retry. + * If count is still inflated on retry we return an error and + * must discard the pages. + */ + if (!page_ref_freeze(p, 1)) { + pr_info("HugeTLB unexpected inflated ref count on freshly allocated page\n"); + synchronize_rcu(); + if (!page_ref_freeze(p, 1)) + goto out_error; + } set_page_count(p, 0); set_compound_head(p, page); } atomic_set(compound_mapcount_ptr(page), -1); atomic_set(compound_pincount_ptr(page), 0); + return true; + +out_error: + /* undo tail page modifications made above */ + p = page + 1; + for (j = 1; j < i; j++, p = mem_map_next(p, page, j)) { + clear_compound_head(p); + set_page_refcounted(p); + } + /* need to clear PG_reserved on remaining tail pages */ + for (; j < nr_pages; j++, p = mem_map_next(p, page, j)) + __ClearPageReserved(p); + set_compound_order(page, 0); + page[1].compound_nr = 0; + __ClearPageHead(page); + return false; } /* @@ -1771,7 +1808,9 @@ static struct page *alloc_fresh_huge_page(struct hstate *h, nodemask_t *node_alloc_noretry) { struct page *page; + bool retry = false; +retry: if (hstate_is_gigantic(h)) page = alloc_gigantic_page(h, gfp_mask, nid, nmask); else @@ -1780,8 +1819,21 @@ static struct page *alloc_fresh_huge_page(struct hstate *h, if (!page) return NULL; - if (hstate_is_gigantic(h)) - prep_compound_gigantic_page(page, huge_page_order(h)); + if (hstate_is_gigantic(h)) { + if (!prep_compound_gigantic_page(page, huge_page_order(h))) { + /* + * Rare failure to convert pages to compound page. + * Free pages and try again - ONCE! + */ + free_gigantic_page(page, huge_page_order(h)); + if (!retry) { + retry = true; + goto retry; + } + pr_warn("HugeTLB page can not be used due to unexpected inflated ref count\n"); + return NULL; + } + } prep_new_huge_page(h, page, page_to_nid(page)); return page; @@ -2771,10 +2823,14 @@ static void __init gather_bootmem_prealloc(void) VM_BUG_ON(!hstate_is_gigantic(h)); WARN_ON(page_count(page) != 1); - prep_compound_gigantic_page(page, huge_page_order(h)); - WARN_ON(PageReserved(page)); - prep_new_huge_page(h, page, page_to_nid(page)); - put_page(page); /* free it into the hugepage allocator */ + if (prep_compound_gigantic_page(page, huge_page_order(h))) { + WARN_ON(PageReserved(page)); + prep_new_huge_page(h, page, page_to_nid(page)); + put_page(page); /* add to the hugepage allocator */ + } else { + free_gigantic_page(page, huge_page_order(h)); + pr_warn("HugeTLB page can not be used due to unexpected inflated ref count\n"); + } /* * We need to restore the 'stolen' pages to totalram_pages diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 8836e54721ae..a672d9c85118 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -749,7 +749,6 @@ void prep_compound_page(struct page *page, unsigned int order) __SetPageHead(page); for (i = 1; i < nr_pages; i++) { struct page *p = page + i; - set_page_count(p, 0); p->mapping = TAIL_MAPPING; set_compound_head(p, page); }