From patchwork Mon Aug 9 18:48:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12427111 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B85DC4338F for ; Mon, 9 Aug 2021 18:53:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9DEF861040 for ; Mon, 9 Aug 2021 18:53:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 9DEF861040 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 0D8D26B0071; Mon, 9 Aug 2021 14:53:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 087956B0072; Mon, 9 Aug 2021 14:53:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E6A176B0073; Mon, 9 Aug 2021 14:52:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0248.hostedemail.com [216.40.44.248]) by kanga.kvack.org (Postfix) with ESMTP id CA1D46B0071 for ; Mon, 9 Aug 2021 14:52:59 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 6489C1D980 for ; Mon, 9 Aug 2021 18:52:59 +0000 (UTC) X-FDA: 78456439278.26.FD9D307 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf08.hostedemail.com (Postfix) with ESMTP id C8F8A30000BA for ; Mon, 9 Aug 2021 18:52:58 +0000 (UTC) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 179IqYGC031886; Mon, 9 Aug 2021 18:52:51 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=UG/aAiHICWCxjsvFC7O7lICWLzo8EYGo1APQM/mI0yc=; b=HBAdwgcj1IGuLQ9yqo+WxT7qMzUBJKfFtoM5gTSXK66epbmE19hktu+GULaBbfI+OjlU Spg8GrHAe9tblerkCg1yjDE/xtBYYyS9jHPU1FQpSjX1lPzpce5h38AiFEveWKiKUYYa IYdEHqUYbR6jGJG24zveAKa1Mb2zWzNF89YLo0CmvNfDx0Vz34FgHqUm1eDLRoKtvdzK dYsvSmmE/1etpbKbVsbtTQ9NH7dk1L8VPl7shX5bU8QThGBOEPkYsmZKts6S6GmPt9qx 9Yc3t8P9Tc7GgTyZhrV/RMi808EabU5MOg6gcVWsu8t+r2d2LYPklrfQRVSyxK+7PKF1 vw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=UG/aAiHICWCxjsvFC7O7lICWLzo8EYGo1APQM/mI0yc=; b=r78fHa1hzLFIoAm7erYB2E7w/gZ7hRAmNYZLvPylH3JJOBA8HTSkIrzTbh62QiVkH/jq jHK1bIAQo1+LfrrTQHusiWA7TutwZ/NMXPEZ/nP3AEr6UTNrAUG5gpN17UAlcvTPDLtO BQ7DG8gnNeMCcCEwg8FqfIPHpBvjnJGuMYCW7kltkCp0S7AjhjVPmLkswhEDNendDBLa rbIG2rpv/baPCSjxd9SvuwRX9sAfrfRc1NM+ZGEf2MqrHD4FdBKkj/ubxhm/gMvNMTVq C0Lu53pGD/AgWenf96vulV+TBI24gJydu5Z++bh2LZEGi2Ji7S9Z9Fh+1hJphEbzndU1 OQ== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 3ab01r9kx1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 09 Aug 2021 18:52:50 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 179IqBUC152733; Mon, 9 Aug 2021 18:52:46 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2177.outbound.protection.outlook.com [104.47.55.177]) by userp3020.oracle.com with ESMTP id 3aa3xrrm9r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 09 Aug 2021 18:52:45 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KgxW5+ycBOSlNxuaFpxJUD7vKMmSVxcVO24ojjlxkLcIlJLi93t2/ee8nkHBw0d7LCDiYw9EufoAWUtSfj8gkDXVmMMtIrCS3qCtI89YqllfgMHiQjrydgyNEMHJWHiwInwM0Ldn1ANlk9So9NtAXC0eunpQ+2SDLlycfYKY4UjRnSuD46RKYlQx9yNHPpcq/1P3nnD7lVNFa9vL6oXm5kC0ZcUt7Gtzm+LtKInV2ZMxTaNYm3fF381s8+Tn4YWNHorxp2R0L5CsoTb8yKol1JpJRt83jxW/rpJtkSFwjFdLmfIcJZ9jFFSQpxbL9tX1enDP+KAZzcpuv0o1ZNlbWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UG/aAiHICWCxjsvFC7O7lICWLzo8EYGo1APQM/mI0yc=; b=kKVYLjj8oIyUbQVFPlIs6j/0Nm8gPiHSzPpsErkRPAAXDAYE+xS/+lMkJ+06iLZsBodOaK6FJv2mVRo1zj5EE86/f2X7Qg+UthEKE2j/qmML/ISEKUFOTT01CISUngSD5h3R/iuL3bkxe8iqxw5hxti/cY/Ey+28Yd/Er7tdR2DCpPI0gnNJEcnF8EKY6qw8eYvj8rRktDnmxM21OJXX8URRRU1qnyekT7HrB4wYvGBONXGMd1muV07XBYbQOtBK9KB7jWNMZLpZVYiLZAqF6olH7m/6ifndMVJeSfzhoLGZ+qa2eG5BJNCnY3ynI2/UZUZ0k6rUYifkAt72bw7s7A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UG/aAiHICWCxjsvFC7O7lICWLzo8EYGo1APQM/mI0yc=; b=jc5gxLnrqTTNPiVg2qSeqvqYgV28ptRw/fJHMvaoWZQerpxRWdMF62T9qspjrPTLwo/ayTAv+9ZlHDVYzdbbCaCnRYVsdOFN8DMuZG29NPdNmNr/zofcDkhUE8egArH05iunI7vGnzmGGpsQC8eJ6hxkDpoB4g+zxzcVM2KswSM= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB2952.namprd10.prod.outlook.com (2603:10b6:a03:89::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4394.21; Mon, 9 Aug 2021 18:48:49 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a%6]) with mapi id 15.20.4394.023; Mon, 9 Aug 2021 18:48:49 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Michal Hocko , Oscar Salvador , David Hildenbrand , Matthew Wilcox , Naoya Horiguchi , Mina Almasry , Andrew Morton , Mike Kravetz Subject: [PATCH v2 1/3] hugetlb: simplify prep_compound_gigantic_page ref count racing code Date: Mon, 9 Aug 2021 11:48:30 -0700 Message-Id: <20210809184832.18342-2-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210809184832.18342-1-mike.kravetz@oracle.com> References: <20210809184832.18342-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR04CA0341.namprd04.prod.outlook.com (2603:10b6:303:8a::16) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MW4PR04CA0341.namprd04.prod.outlook.com (2603:10b6:303:8a::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4394.15 via Frontend Transport; Mon, 9 Aug 2021 18:48:48 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f7d36706-a30d-4694-404a-08d95b665319 X-MS-TrafficTypeDiagnostic: BYAPR10MB2952: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8882; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ge0ZBJ7nbs8EPZRTAXL0mo2jYvobD5rIegF3Ajs+i76YSANngh7AMiCGHP4KT4ZWqsMhu+DBHxDYsWtiTGF63podv0gUJWpvY7a3TpeDNKYjso0hZgA46QOZuJxPl55agpZh0CdDScV6LifSTq7wkCbksUaBGmrmNycyUvK48D1ZgYUsFyHMzM9f2J4Enn6DC7J5sf0Km4/ShQaH6lizGePGWmirbGDBe5Tw7Ugn/QGiLBhcJaJhIc7nRDPLFC1qWAwtcwmyV8JKvI5vfOO7pBDCsn4/RRyQ4GnYV0hl4wspCCJ1+THIydVkNiaT9yJejNVbhTES2yZf0scE5z4SxN+QuecdKBq/zXEuAoDxJ3df1k00d/mwtOrDH6vw6VGD7+jK8XYKVU/6Ry6E0BCOiA0myb+S4BUZ1tDLzVGxIVEiU76iStlwhriDiEWDVgygAQII5PG7HS+bThEvYN4mgjCD9EkFJMtx8WU/xEVAaHCBys8mC1RyR3Wp9fJN7VkzkoBJ5p7cnnYeqMXVUR7CfD0y3ExwmvSUal+jqs3mwqzKSl715XRqTLvCq28NeuEdF6zo257cqXfKC0J7kbwyLOF3/zl6OiY96KTHC6GVlBUnxSp0Kk2IhVTq+VekJVboF7sBKHXTIDBxu3JGUdJOJYQZEFex7pkZPFCArk5SCa6Vk7IkrBwHTBe0251+y/r8EAySteZKZlNPv/8l+W64lA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(396003)(136003)(376002)(346002)(366004)(39860400002)(4326008)(6666004)(54906003)(7696005)(52116002)(8936002)(5660300002)(66556008)(66476007)(478600001)(83380400001)(107886003)(956004)(38350700002)(36756003)(26005)(316002)(7416002)(186003)(8676002)(86362001)(38100700002)(66946007)(1076003)(44832011)(6486002)(2906002)(2616005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: o2V9JutTlBEWKN36zhySBtEzbyBPGja4rfJrg2PHZRRxLvoz/KGU68DFCpWQBUqeZYLKwbaFTEMZhtSzKMjGOI6Rf9SYFv8yBTsDvRT+EciHFBRXdEr+0J9vGgoot/LFHqLRZTkNObJqHXWrvnXw1mQcXYxZHnbiKFU8flPy49vM8vcB9iI3ILpXGselZgsbC4n0mqdWpkojQyMtfKe9pmILGShIt694iH3HUlBrRcO+vMwjeKwgLY/VNtJFXgTxemb/aRtZLBYYqk9jU/a3ilIL6q2LvaF4fSwtITq9eJQulgwtoVUYxJUo5TBUYNpyt3guxRQJ5vUgQV3NIDsnUgy2lUkHg9F7kWZnZepHXJnS02DYk8pH5cUpieRuZGN4GRLlcC6QQOBpFxLNPI1AjZh9AdEFC2hTvLGsYL2qcWPDlYWKc8Ao29jzg6Edb9nBS10GmVzBA4n+dLmtWOk8kEfRl7EkGbybzYv4z1c0/JC3jjyq0u90k11a5DNNDrF+GwyGbUUtRq6cf7ajFrQHi0QZXR8K9VckTOcbxwHL7gunqA061RE+j1F0v+iXtnNh2F2rxfdbP+t7Ec8WO2r8OiGWl1uSmCwmkHXbL4FB0azwpvw2Fk6y1lmvSRbVxjoJoxTWcCyqtOJY8b5Zbpo7Cj6/Ez2l0+bXa1NC4WrcNunZFKt4TNVFFx23cJvuJ2XBNs2KNHv38tjDzL6Hy8UfpxKvfyUWqCCYUcHeNAZQPLcG4TeYNtDT/DxvLPmiZDmXOQEuqUmTD80008VHF1Sp5hs/orun4N8eCJPv+W7DL0DcrpTIedzrjFnpwIlLmhMLjRXoPtKl2Qr+kyb20lqeqP/6OMSY6KEmIkMKL/lHHWHKOy5RXnfrcNNdVXMloNjAmSMYNnGH0LhafjPMzRkE6WJ1XL6ISycbd0wv/tKw7dw3/NNZIYCl4s1D6nyHxKRrIS3yiiDCUzHQCH6Het8jKJ1OuXX+IzFOBkz1A9DPGRhW1xSayeizoWan4SbOJLk2Gx7YQj3zh+s2Cf8iabPpkPtLvX1iRmVHE7Y8vgZ3hesQTiWfVPWKiirZ5TKU41KZt5Izv8jV2BE645KJOoFfUleTBSsiGIu8R5rWUYwtytaXEo982GAQy1I1ozyKRx1rYmS/ZLTvj8gXKhucxsSbg1Q8e9C+wFrOg0B+ueX/TcNCAxoAz47KD+q1mi/gEDtkBPsxcBWTsbnA0p3cNUZJdV8edTnQmYncoi4bqMLEalgB3y3vKSTouhZ2mFKw9rUaqWr4C9XhdtJ+RKao2Tj3q5F8TVGZ943jZjQTZSVxZCNJWBcNSPYcfJ1KA6ZAnYcL X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: f7d36706-a30d-4694-404a-08d95b665319 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Aug 2021 18:48:48.9397 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 9iMSjMKRXuJiVvZqPAlH3RAwdVkvgsXPu1e0Y9X8f7GJDDwqRQ9h5o8RuEq9G8o4LzQ4X0HfcRPxlbWB7LsFhA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB2952 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10071 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 mlxscore=0 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 spamscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108090133 X-Proofpoint-ORIG-GUID: 5tsxqjfx-WbWefhT2ly-6g4KQ-5uFSPe X-Proofpoint-GUID: 5tsxqjfx-WbWefhT2ly-6g4KQ-5uFSPe Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=HBAdwgcj; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=r78fHa1h; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=jc5gxLnr; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf08.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=mike.kravetz@oracle.com X-Stat-Signature: 6k5h7sy7ax7ktz9kctr54hju5oqxnej6 X-Rspamd-Queue-Id: C8F8A30000BA X-Rspamd-Server: rspam01 X-HE-Tag: 1628535178-204481 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Code in prep_compound_gigantic_page waits for a rcu grace period if it notices a temporarily inflated ref count on a tail page. This was due to the identified potential race with speculative page cache references which could only last for a rcu grace period. This is overly complicated as this situation is VERY unlikely to ever happen. Instead, just quickly return an error. Also, only print a warning in prep_compound_gigantic_page instead of multiple callers. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index dfc940d5221d..791ee699d635 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1657,16 +1657,14 @@ static bool prep_compound_gigantic_page(struct page *page, unsigned int order) * cache adding could take a ref on a 'to be' tail page. * We need to respect any increased ref count, and only set * the ref count to zero if count is currently 1. If count - * is not 1, we call synchronize_rcu in the hope that a rcu - * grace period will cause ref count to drop and then retry. - * If count is still inflated on retry we return an error and - * must discard the pages. + * is not 1, we return an error. An error return indicates + * the set of pages can not be converted to a gigantic page. + * The caller who allocated the pages should then discard the + * pages using the appropriate free interface. */ if (!page_ref_freeze(p, 1)) { - pr_info("HugeTLB unexpected inflated ref count on freshly allocated page\n"); - synchronize_rcu(); - if (!page_ref_freeze(p, 1)) - goto out_error; + pr_warn("HugeTLB page can not be used due to unexpected inflated ref count\n"); + goto out_error; } set_page_count(p, 0); set_compound_head(p, page); @@ -1830,7 +1828,6 @@ static struct page *alloc_fresh_huge_page(struct hstate *h, retry = true; goto retry; } - pr_warn("HugeTLB page can not be used due to unexpected inflated ref count\n"); return NULL; } } @@ -2828,8 +2825,8 @@ static void __init gather_bootmem_prealloc(void) prep_new_huge_page(h, page, page_to_nid(page)); put_page(page); /* add to the hugepage allocator */ } else { + /* VERY unlikely inflated ref count on a tail page */ free_gigantic_page(page, huge_page_order(h)); - pr_warn("HugeTLB page can not be used due to unexpected inflated ref count\n"); } /* From patchwork Mon Aug 9 18:48:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12427117 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69010C4338F for ; Mon, 9 Aug 2021 18:53:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DDE3C61051 for ; Mon, 9 Aug 2021 18:53:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org DDE3C61051 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 4597E6B0074; Mon, 9 Aug 2021 14:53:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 409806B0075; Mon, 9 Aug 2021 14:53:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2AA6F6B0078; Mon, 9 Aug 2021 14:53:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0100.hostedemail.com [216.40.44.100]) by kanga.kvack.org (Postfix) with ESMTP id 0CAF06B0074 for ; Mon, 9 Aug 2021 14:53:24 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id A48011809A334 for ; Mon, 9 Aug 2021 18:53:23 +0000 (UTC) X-FDA: 78456440286.03.C709ACE Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf05.hostedemail.com (Postfix) with ESMTP id 46239502BC77 for ; Mon, 9 Aug 2021 18:53:23 +0000 (UTC) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 179IpvcP000446; Mon, 9 Aug 2021 18:53:18 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=cjzHVGiPOd0xqdqMQdU4VkgDFiyhVm5RZJGYPVRWmXc=; b=cmuL7DIFsP0dtqzEhee4uOpVkuZ+Yv/IZw05P7UyvFkwX1co4Hmr+yL7+I9w/ERvkir7 GIfRN51QXmdYae7A7h660jxVx1YpmDWyD94WzUTjrE7nVAsAADbKgT2e93VmX710W9xe sFxwd/leR8RGiT7gqmWdQtOwMlb3JmSh93o5gPIq+fjX7qSvLWqBaut5VdLKSBwIYbWN DDf1V0A133JzzAWmsh9UHYk6W7/fH8peEwf0ET/QP8TWv0eoPmeudCQfkP9P79ANW9F2 C9q+hMBqTeCSoasxGmu4rLjXhH2lk+vJ4kd0P0Es820f2/KwikquJfTt3bB1B7Yn1x+8 iw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=cjzHVGiPOd0xqdqMQdU4VkgDFiyhVm5RZJGYPVRWmXc=; b=CjQBqHNFpuNBWTxIxWGhUqPRV6cQNDBttguYkx8+7Z1K1+AJLpwtY+Xwart2CEGxfKJN LKdrxV77SuozTuUmbF4zuX/PUQPydDzyXXgHQW+/Sz3Y8cX671XqaW2aeMyphpjktw9t txEDM7chQPharoeWUOROK3cEYObAz54ipXNI0zL9GHQMP26mKIOX/FIj/OMDnB5k148t vQxKEGdGyK0wZiCQWIRuKKweI6J4M3Kcd2v2+IhafaajyTs6XKLjbZ8ajZo206CyHQ6W sq75951la7zrBr293t0aJDbU45s86v67pz8ipFwX4Vzm/ATy8pz94+La4unadZaZhRzK kA== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 3aay0fsrv5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 09 Aug 2021 18:53:17 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 179IqEcJ152833; Mon, 9 Aug 2021 18:53:16 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2173.outbound.protection.outlook.com [104.47.58.173]) by userp3020.oracle.com with ESMTP id 3aa3xrrmpt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 09 Aug 2021 18:53:16 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DRT/6lkGksSfOP1kPsdL8DUCaV/ClvwkZgTDAGe+vgISJfbWKguvvUCVwxcesiO5eeeuKYpPQqk9Kamy1kZcW6ZuqolKF7oo2LwE48qYqJCVfaZ4aOvNRR6uZa9Xr5jtC+R8Rh7hk5tisw1nffzfQN5hcQA7P1WJmcRQILGb46D11tcrGV9RKk1Dy/U5QGKPBczLWjrSJBNgVsXgZK0bwYaJrUUXzMkanwlZ1QNetFwaH+r0GFKaXRpUsXN+AgAprEL+BAbYkxdcljxoEW6MO+YKosgwA5C4GpR/tS7MXnaO8zXTF/RPpakqtl/BuwUatMIf8YCcnGBqTl0z9O2o1A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cjzHVGiPOd0xqdqMQdU4VkgDFiyhVm5RZJGYPVRWmXc=; b=IR3/exgb1kQ2wSDV1mwl+Qkhl7YxDwPiSoVGSNH6YoXOwwydXaI8WcBYXyroYT+D+APvJsgdUfrtgMxegWgoTaesyHFHbqUbNEfiSR6j8B/29brV3KfvxLXwFXsnBjktEjZeQhzcxi0k90/c+u3UBh8GJYElVSeoua+fvjrPUCvs+n1AS7h/Y+JhOJhuVoGnpEaUkHKjDL41rI/ozfPePc+Q2P/Fe1UPpPlSr9cNLBL9j/pV3Y0UA8vTXLgbpjlslNDx62avW0Vglq95rI4eFrsYTDeT0naBLRYhI7UdakVCPAD4KphJtB9C3d62382wWBZKCosguCH3gYYoRDFERw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cjzHVGiPOd0xqdqMQdU4VkgDFiyhVm5RZJGYPVRWmXc=; b=twfsq3plMK29PUMvdpscuY29YZON14pBmAqUod8SEXxtojGjP6LqZrSSrNXXQsHYMuckWDTi7pTw1U94R8Q0b51xH1TRvbKcTM9SzvQ7HHotFSHID6pdBnlZkqiE20f6XqNeAj7+1INmWRb9GIh+OXhaPQ+FD+bOYejBQKmykkw= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by SJ0PR10MB5455.namprd10.prod.outlook.com (2603:10b6:a03:3b9::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4394.16; Mon, 9 Aug 2021 18:48:51 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a%6]) with mapi id 15.20.4394.023; Mon, 9 Aug 2021 18:48:51 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Michal Hocko , Oscar Salvador , David Hildenbrand , Matthew Wilcox , Naoya Horiguchi , Mina Almasry , Andrew Morton , Mike Kravetz Subject: [PATCH v2 2/3] hugetlb: drop ref count earlier after page allocation Date: Mon, 9 Aug 2021 11:48:31 -0700 Message-Id: <20210809184832.18342-3-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210809184832.18342-1-mike.kravetz@oracle.com> References: <20210809184832.18342-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR04CA0341.namprd04.prod.outlook.com (2603:10b6:303:8a::16) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MW4PR04CA0341.namprd04.prod.outlook.com (2603:10b6:303:8a::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4394.15 via Frontend Transport; Mon, 9 Aug 2021 18:48:50 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: aacf4ff0-6e5b-438f-6b26-08d95b66542f X-MS-TrafficTypeDiagnostic: SJ0PR10MB5455: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: WcBv8UGH850LwGBUGC+0rg1odTb5a7J2U6OFOjLjVhjyhn5RP/DQKWeAsc3d5UX1MRX0nDekx+3aXTXDGdLE4wgMaXXK+LhWprDL7WkrbDZqYDpmICspqrjnKwPnmTjUUwH6c49MFHzrzEZDSFXtZA0n6XNH2pmSyEuy+cF6OH+a5IpOD3GrhPQTCMtk/NGMv/6MMbM4RIidWis31j3kx/kuq3XinR1bU6UcZ1QseFvptf6H0+K3FkDX546JrX/CIq0FRPHVLNlILMdMhUPevUKKpx6vU/70NRVMR1zlv5uyI4xXqY2cQFogmChclLF2MEqGvi1KidAg7uOEkovwyoT4xneKJMnWOoDYs2fLUgLUWvt7wBzYHM6PhHfAP02iAoiljpIBnIS32OT8bfOvE8d2cWIhqtcnIFgInghzAp8/FgpdTBD2KmpemizWIuQDUbZxA6hknglnj7/7uX9usAkpo3m58We/TgN9c+YAHIAceGHPXcPaVfkaW2aFmtzxWGuEA5sHb9WqCKM4LfxHINrsg+HdAm7otFqDN4KBaixCtZiMJrA85KrWfNfDupGl9FHKuN27ePJDLJO6VsjBDYlcELE5lHdwtRNXBieSKotlYHl8ONgbgTAChiQ56aY4JF/CR1qyK9UBwPPfOQv0B3JRLgrHFLFgvONA8r1DdnbZk3/lITxC8/j0GUdNMYRyuMQltMf1N3ega4ZTgIhUfXa8WzWCYAu9qW4RBPfmALay9hggSqljnSi3z8+jDidNzdcUz3ILHAypguNVjcUZnw441xJjh3v/xmww6Xl9qB4= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(366004)(376002)(396003)(136003)(346002)(7416002)(83380400001)(86362001)(316002)(956004)(2616005)(5660300002)(8936002)(1076003)(38350700002)(6666004)(38100700002)(8676002)(186003)(6486002)(966005)(66476007)(66556008)(107886003)(66946007)(36756003)(4326008)(478600001)(7696005)(52116002)(2906002)(44832011)(26005)(54906003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 8wVbDK9zNdzb+WDR+clGFMq9UTHNVX3M8aSzdDPBbuHmGi596kcnDLkN0hXu0WSo/roUc+3wIhtGmMfz/Wtz41yAtwdfArrsFQQPAk21mTaMzBZNNNLDJCNlC7KLQO9VaIppspTQIhOy+HmVcxnDKKzYyET20F23ghLWNJz6biMz0mLVcTTd3wedb3pro4Eq8Lrquvwn8uZ4yxHHNUb+V2Lvc1Jg9k275GLmZk8rMUGQwKdBG/Dtf6lFePIpxk6Thr2JlHMs1HI2qQ/PP4WAMFYTMgPGVZo7MbbznnUNupaN3Wj48vaN6v90fQ5P4ublVH6OULzz3oNKtTi9ffK520uj6SueCDy3HJ/MT6NEbu6C4e8JzGQIbxeLpeSsbuZmptqF1n+2ExndT1tVYKe/Ky6yCSUzjLVvVZLgg9T/PTnePu4ioCdx3yT8Crug0w5MFw+BNRTs4sF2e44HdPZcraTqpzA2FmtQg9S3QpOgkLktpClDhQ/yeVMnKdT2kBzMXchgGy4hMW6NCX/Bsaf+dHUPXd156koiOyT5HhIuIrouwT18k4OKsCwbMikLyikdN+MAVPtd54+hlf665Ux9gzNRWxo2Kb5ZFqE2Eb14nOXWcuXXNefOdJiErLiMW824zJpc8lOlEHbJ6UcJCI64453yA/5x97W2iUPORZvg8H+vtaumLFRCZM1F/v1yIW+LI2iu9c8WVPLK7KLPVrI1PYw5eY20pptkWn3nYZu+bCEeyn9OtpaeS1HhipqjrC6Nhi5iLTrYCoETbSIxXJzVKQO3xs2ILjm70tke8/2XDAG3cGw4KtQa4gHKk5e72AMV6uw4limnsj24PeG6Jm9zqfxnzchH9VYN8N/QA1aCcAPHux1FRCz7YnFiEDyLkK1Uj+6EtPVyK3TmK8pP8fb0oZD18BwQbft5sxikEkUQ2dBDqOe11wRrUV2aQVihWYbY8VzEIUlFYzTgO7YU2WNKmS41XakWUp7JpWzxONM3rkJaEItvdZ5t6f9cg7AHiDbAPEQmfkJsLCJCRuL8ag9AzspHiUOlZSqtLVzlm1hRHnVQYLU18x3bW57YkU++QrEgAdKDzjdPdXVrHBgVSbJBlKMe1KVucshOrArsJ2CLnwBX+cGi+j+Qi+b1VEZmWggzAirpXulcC7sz1YMNEqqtYfJwePTOYS7nckY6ai9hFhte37CvFfsNAFsNc+8wj6qEDnwJLe13x7o4fDmqEY+TVs3YlAHrvOHPjyQ3L8CnjobalQyXGK+r8nmYI2txZ+r2Kp9DZeQalgERaQGJzcdr8zEmcJTR8NV+BhaRMjomRwIbP7telSxrNIaEuEaVBAnd X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: aacf4ff0-6e5b-438f-6b26-08d95b66542f X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Aug 2021 18:48:51.1769 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: MtM3OfArdxef9pdt1Ylmr4fi5KUwpx+bNRQF2wGH1Rd7VjZibEbUXfkp1Di9vpvb3yAMdziac/+ciCsSQtonng== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB5455 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10071 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 mlxscore=0 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 spamscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108090133 X-Proofpoint-ORIG-GUID: USjvolypyGEZqu_jZM87M8sImSlAsDnw X-Proofpoint-GUID: USjvolypyGEZqu_jZM87M8sImSlAsDnw X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 46239502BC77 Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=cmuL7DIF; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=CjQBqHNF; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=twfsq3pl; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf05.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=mike.kravetz@oracle.com X-Stat-Signature: igfzi5nhb5zwf6dwwpj4aqciqf9gwf6s X-HE-Tag: 1628535203-666719 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When discussing the possibility of inflated page ref counts, Muuchun Song pointed out this potential issue [1]. It is true that any code could potentially take a reference on a compound page after allocation and before it is converted to and put into use as a hugetlb page. Specifically, this could be done by any users of get_page_unless_zero. There are three areas of concern within hugetlb code. 1) When adding pages to the pool. In this case, new pages are allocated added to the pool by calling put_page to invoke the hugetlb destructor (free_huge_page). If there is an inflated ref count on the page, it will not be immediately added to the free list. It will only be added to the free list when the temporary ref count is dropped. This is deemed acceptable and will not be addressed. 2) A page is allocated for immediate use normally as a surplus page or migration target. In this case, the user of the page will also hold a reference. There is no issue as this is just like normal page ref counting. 3) A page is allocated and MUST be added to the free list to satisfy a reservation. One such example is gather_surplus_pages as pointed out by Muchun in [1]. More specifically, this case covers callers of enqueue_huge_page where the page reference count must be zero. This patch covers this third case. Three routines call enqueue_huge_page when the page reference count could potentially be inflated. They are: gather_surplus_pages, alloc_and_dissolve_huge_page and add_hugetlb_page. add_hugetlb_page is called on error paths when a huge page can not be freed due to the inability to allocate vmemmap pages. In this case, the temporairly inflated ref count is not an issue. When the ref is dropped the appropriate action will be taken. Instead of VM_BUG_ON if the ref count does not drop to zero, simply return. In gather_surplus_pages and alloc_and_dissolve_huge_page the caller expects a page (or pages) to be put on the free lists. In this case we must ensure there are no temporary ref counts. We do this by calling put_page_testzero() earlier and not using pages without a zero ref count. The temporary page flag (HPageTemporary) is used in such cases so that as soon as the inflated ref count is dropped the page will be freed. [1] https://lore.kernel.org/linux-mm/CAMZfGtVMn3daKrJwZMaVOGOaJU+B4dS--x_oPmGQMD=c=QNGEg@mail.gmail.com/ Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 100 +++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 78 insertions(+), 22 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 791ee699d635..bde324e69276 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1072,6 +1072,8 @@ static void enqueue_huge_page(struct hstate *h, struct page *page) int nid = page_to_nid(page); lockdep_assert_held(&hugetlb_lock); + VM_BUG_ON_PAGE(page_count(page), page); + list_move(&page->lru, &h->hugepage_freelists[nid]); h->free_huge_pages++; h->free_huge_pages_node[nid]++; @@ -1399,11 +1401,20 @@ static void add_hugetlb_page(struct hstate *h, struct page *page, SetHPageVmemmapOptimized(page); /* - * This page is now managed by the hugetlb allocator and has - * no users -- drop the last reference. + * This page is about to be managed by the hugetlb allocator and + * should have no users. Drop our reference, and check for others + * just in case. */ zeroed = put_page_testzero(page); - VM_BUG_ON_PAGE(!zeroed, page); + if (!zeroed) + /* + * It is VERY unlikely soneone else has taken a ref on + * the page. In this case, we simply return as the + * hugetlb destructor (free_huge_page) will be called + * when this other ref is dropped. + */ + return; + arch_clear_hugepage_flags(page); enqueue_huge_page(h, page); } @@ -2017,9 +2028,10 @@ int dissolve_free_huge_pages(unsigned long start_pfn, unsigned long end_pfn) * Allocates a fresh surplus page from the page allocator. */ static struct page *alloc_surplus_huge_page(struct hstate *h, gfp_t gfp_mask, - int nid, nodemask_t *nmask) + int nid, nodemask_t *nmask, bool zero_ref) { struct page *page = NULL; + bool retry = false; if (hstate_is_gigantic(h)) return NULL; @@ -2029,6 +2041,7 @@ static struct page *alloc_surplus_huge_page(struct hstate *h, gfp_t gfp_mask, goto out_unlock; spin_unlock_irq(&hugetlb_lock); +retry: page = alloc_fresh_huge_page(h, gfp_mask, nid, nmask, NULL); if (!page) return NULL; @@ -2046,11 +2059,35 @@ static struct page *alloc_surplus_huge_page(struct hstate *h, gfp_t gfp_mask, spin_unlock_irq(&hugetlb_lock); put_page(page); return NULL; - } else { - h->surplus_huge_pages++; - h->surplus_huge_pages_node[page_to_nid(page)]++; } + if (zero_ref) { + /* + * Caller requires a page with zero ref count. + * We will drop ref count here. If someone else is holding + * a ref, the page will be freed when they drop it. Abuse + * temporary page flag to accomplish this. + */ + SetHPageTemporary(page); + if (!put_page_testzero(page)) { + /* + * Unexpected inflated ref count on freshly allocated + * huge. Retry once. + */ + pr_info("HugeTLB unexpected inflated ref count on freshly allocated page\n"); + spin_unlock_irq(&hugetlb_lock); + if (retry) + return NULL; + + retry = true; + goto retry; + } + ClearHPageTemporary(page); + } + + h->surplus_huge_pages++; + h->surplus_huge_pages_node[page_to_nid(page)]++; + out_unlock: spin_unlock_irq(&hugetlb_lock); @@ -2092,7 +2129,7 @@ struct page *alloc_buddy_huge_page_with_mpol(struct hstate *h, nodemask_t *nodemask; nid = huge_node(vma, addr, gfp_mask, &mpol, &nodemask); - page = alloc_surplus_huge_page(h, gfp_mask, nid, nodemask); + page = alloc_surplus_huge_page(h, gfp_mask, nid, nodemask, false); mpol_cond_put(mpol); return page; @@ -2164,7 +2201,7 @@ static int gather_surplus_pages(struct hstate *h, long delta) spin_unlock_irq(&hugetlb_lock); for (i = 0; i < needed; i++) { page = alloc_surplus_huge_page(h, htlb_alloc_mask(h), - NUMA_NO_NODE, NULL); + NUMA_NO_NODE, NULL, true); if (!page) { alloc_ok = false; break; @@ -2205,24 +2242,20 @@ static int gather_surplus_pages(struct hstate *h, long delta) /* Free the needed pages to the hugetlb pool */ list_for_each_entry_safe(page, tmp, &surplus_list, lru) { - int zeroed; - if ((--needed) < 0) break; - /* - * This page is now managed by the hugetlb allocator and has - * no users -- drop the buddy allocator's reference. - */ - zeroed = put_page_testzero(page); - VM_BUG_ON_PAGE(!zeroed, page); + /* Add the page to the hugetlb allocator */ enqueue_huge_page(h, page); } free: spin_unlock_irq(&hugetlb_lock); - /* Free unnecessary surplus pages to the buddy allocator */ + /* + * Free unnecessary surplus pages to the buddy allocator. + * Pages have no ref count, call free_huge_page directly. + */ list_for_each_entry_safe(page, tmp, &surplus_list, lru) - put_page(page); + free_huge_page(page); spin_lock_irq(&hugetlb_lock); return ret; @@ -2531,6 +2564,7 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, { gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE; int nid = page_to_nid(old_page); + bool alloc_retry = false; struct page *new_page; int ret = 0; @@ -2541,9 +2575,30 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, * the pool. This simplifies and let us do most of the processing * under the lock. */ +alloc_retry: new_page = alloc_buddy_huge_page(h, gfp_mask, nid, NULL, NULL); if (!new_page) return -ENOMEM; + /* + * If all goes well, this page will be directly added to the free + * list in the pool. For this the ref count needs to be zero. + * Attempt to drop now, and retry once if needed. It is VERY + * unlikely there is another ref on the page. + * + * If someone else has a reference to the page, it will be freed + * when they drop their ref. Abuse temporary page flag to accomplish + * this. Retry once if there is an inflated ref count. + */ + SetHPageTemporary(new_page); + if (!put_page_testzero(new_page)) { + if (alloc_retry) + return -EBUSY; + + alloc_retry = true; + goto alloc_retry; + } + ClearHPageTemporary(new_page); + __prep_new_huge_page(h, new_page); retry: @@ -2583,11 +2638,10 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, remove_hugetlb_page(h, old_page, false); /* - * Reference count trick is needed because allocator gives us - * referenced page but the pool requires pages with 0 refcount. + * Ref count on new page is already zero as it was dropped + * earlier. It can be directly added to the pool free list. */ __prep_account_new_huge_page(h, nid); - page_ref_dec(new_page); enqueue_huge_page(h, new_page); /* @@ -2601,6 +2655,8 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, free_new: spin_unlock_irq(&hugetlb_lock); + /* Page has a zero ref count, but needs a ref to be freed */ + set_page_refcounted(new_page); update_and_free_page(h, new_page, false); return ret; From patchwork Mon Aug 9 18:48:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12427109 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8944AC4338F for ; Mon, 9 Aug 2021 18:50:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 09BAC61051 for ; Mon, 9 Aug 2021 18:50:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 09BAC61051 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 40FF36B0072; Mon, 9 Aug 2021 14:50:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3BED26B0073; Mon, 9 Aug 2021 14:50:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2387D8D0002; Mon, 9 Aug 2021 14:50:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0167.hostedemail.com [216.40.44.167]) by kanga.kvack.org (Postfix) with ESMTP id 069476B0072 for ; Mon, 9 Aug 2021 14:50:27 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 99066204B8 for ; Mon, 9 Aug 2021 18:50:26 +0000 (UTC) X-FDA: 78456432852.01.DE3DBFB Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf23.hostedemail.com (Postfix) with ESMTP id 314B890000BD for ; Mon, 9 Aug 2021 18:50:26 +0000 (UTC) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 179IkMkA006791; Mon, 9 Aug 2021 18:50:09 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=oKJUDB2Wy7QvwHTcImdp43Gkc6hzN7+WTJmshclrgtk=; b=Sxx5PAq7LobREs4zI+LZMukepvmIuZ2Gpc7WF2eZTNlaYYF2gxxgNUIG6ray3AvJMoRy O+qrJWRLODdFGkEP/d+Fg8fb1Oik620TQ11bJ9xp1pvblgXLgTXbImDvQPpm2slxL9Gd f9dJuyjEWmu13kyZNMXuRaiWYGXnuik/itAJI9a8tgLbNdJrCCfGBX2FwJRavJiT0m3u 3FJw1Zhp4LZ5r9g15WLpPeJbFmhUN6AyC45xMbOYiN448EtGUDxewEqgoAS+77oW/lNT 3yeiIc9PH3OzpWcGmq6fj/nx1ctU4mUYokNeGXTaYoKYq+r2xvzp1dfIsUEcPqz6CHN6 JQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=oKJUDB2Wy7QvwHTcImdp43Gkc6hzN7+WTJmshclrgtk=; b=YbvjlL8PasGqp2hhtn03L/4ehZvfnad+BX4isuack49EWWetw0K+Eb0ZHWI4llFuVmFT LIaAn2baVuE7bWnT5ylKn5frlRzwbHSCLq6Zwu8aEBXDedzdWj6yvulxgQdJMhuys5Zk p4NxI1HahM2OpzLiYB7E9d/fhKUR3siBIOck3Cfnn8r5swwoWJUBVBc+C4IBhehN2yXC g+bPWHZbvNqZaKR3S0giv5BAX8DydYh/tO9GX6ypl2f5xZ9D4IGv4V2qI9Z1XlYprEqE 8b4w6Ftz0HJMtV8S1V3iOO9Krr3+Rpy0nhkR8TCWorX3TN2fnWc1pMlv0WJ1mPkp1LzW yQ== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 3ab17dsd7k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 09 Aug 2021 18:50:08 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 179Io53r164818; Mon, 9 Aug 2021 18:50:07 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2169.outbound.protection.outlook.com [104.47.58.169]) by userp3030.oracle.com with ESMTP id 3a9f9vgf74-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 09 Aug 2021 18:50:07 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TZDsdu+PPvKm/WK7qzkXSbirFKALscQPbcfzI+TjwJLnZ3ohdjl63sUMcvBAX+Cyop7q9476EQWDSrrEg2OarU9bXYcPV2SkAYZql01uKEXq657cS4wAKLSb+pWXFBt6rURM6KWH+gwHI5IsEeH8UupKAKnuFHzrAx1sFrXxpo2bdxR8xlwyAljec8MnCxW1NVakQcE8Rb+Y7b35EPdZPAVqfiF+2YRyf+mbbk26lYXH5Oy5SasUsFTsTmJiZf7WXfvJls1bPZmENMiLYRpMNt17jGP2mqtDDOz1fpEL8OPlVK7PWYqQpLheRGSu5/O/P+omxsyF7ZSWA5xbGpAEhw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=oKJUDB2Wy7QvwHTcImdp43Gkc6hzN7+WTJmshclrgtk=; b=JRljBX6+KURMAWJHAKdfkZa1SdFVYEparLvmT7lBzeb7HWr4gJQoSfTzv6GWlOxW7Hp63wAV1N0W+uJZqIHLatJ2gkhK12CP5CbhWg9sqO6sMrbLttA91S8+lfYOo/ZjvHxeRugTlTV9hno+n2r+EMJHG/aCb58ELhubm8daKL/hXelpOxED4b1jLqLpoGlFOSjHEyzrI8UHTo9itZnuoI2ykRCLFb4TnoQIpqnPAuwlkrfEB2BAdhxKeXDyURnO+9HxCtvTc09qQwuPQ0AGYhZxCumZUZkd7/p2S2QJlXOjmJJvRTrVUkyHttAD4l03UyGFP6kRlC+9GYfUMgYZvQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=oKJUDB2Wy7QvwHTcImdp43Gkc6hzN7+WTJmshclrgtk=; b=mK2fzljQkr3RzAFFENaWnGOxfETIr2MByA2tuV5vWpYzDATcUKVs2s26g+YtWhSaBZJHsAzXE2Mu5SWZQVQNU0Ny+bPzcjRGWHunHmtBTS6j3GYmU+kB9OOYoNlZUeWL+65641BgB0D/4XDToVNqcexyX0h5C/QKzCH7CYxDANk= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by SJ0PR10MB5455.namprd10.prod.outlook.com (2603:10b6:a03:3b9::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4394.16; Mon, 9 Aug 2021 18:48:52 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a%6]) with mapi id 15.20.4394.023; Mon, 9 Aug 2021 18:48:52 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Michal Hocko , Oscar Salvador , David Hildenbrand , Matthew Wilcox , Naoya Horiguchi , Mina Almasry , Andrew Morton , Mike Kravetz Subject: [PATCH v2 3/3] hugetlb: before freeing hugetlb page set dtor to appropriate value Date: Mon, 9 Aug 2021 11:48:32 -0700 Message-Id: <20210809184832.18342-4-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210809184832.18342-1-mike.kravetz@oracle.com> References: <20210809184832.18342-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR04CA0341.namprd04.prod.outlook.com (2603:10b6:303:8a::16) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MW4PR04CA0341.namprd04.prod.outlook.com (2603:10b6:303:8a::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4394.15 via Frontend Transport; Mon, 9 Aug 2021 18:48:52 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6a3f5e15-3f32-4235-967a-08d95b665551 X-MS-TrafficTypeDiagnostic: SJ0PR10MB5455: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: EjxX1bSSYlvmilhtvdsA5fpdnvFksYd4IfGGyfxdsiBUyGTcMP9ddehJzM2lSZXBXbRwB+00sHxdeSYsZzewDuMEyY1AYLs+Vy5DQ+1M/gOlR3XUCVjjF1khpGKpgtNWQV/p1cB60RhWLFFnxoJ/6UbJw1n7Qgp8XWQ4w2EOfGzqyMwpFizmduDFJJWAgJawtMDLgtkk5mzarSNT9BS3JBgLU5rR2RToNadmWblkepw1CdQwvR2gnZyPbMHePQb2RxP2xyHvbKxnYP5+BKfInIM7a5QN/8yhR2+umbqmuwi7qNoBAg4dAQ3l1KTcmZ0vZY9rqLydbYHenEGj6rC4E6lMflEGSLG1RcPN2iogW7rDeP/zoX7SmHE4yhx+9JVuAtm6qQQN8LYQAF0HcQkQRq+nlLQCEbZTKBx0bG4A+9W+5V8i9Jy2DaGY8m95nKIrrtRN0ItxZ+vobHf/vUl/zcQ8jzSKU3BuJpNG3JIs0v3TO4wD4ob1PUd77m6gf88c5jLeS5pmlWMSWXEf5yVXg3ydZZofOyCYyYYuSaKgrg8QRO2H8QODh8nUIaAQPBHXfFPkWFICEeaA0MT6Obimfzr19BQfY1HC6VIvyu9gsGa4G3lkO8RKO8W8387G4nlcxRyD1A6TX7t4f1C9eqPQIq940NjzZg6QFoR7EX/8HW+A+wqxfhJHLSCpYRumXnKn X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(366004)(376002)(396003)(136003)(346002)(7416002)(83380400001)(86362001)(316002)(956004)(2616005)(5660300002)(8936002)(1076003)(38350700002)(6666004)(38100700002)(8676002)(186003)(6486002)(66476007)(66556008)(107886003)(66946007)(36756003)(4326008)(478600001)(7696005)(52116002)(2906002)(44832011)(26005)(54906003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: m49t77PVQq2xSCHAN17BZMRQHYyNhtXt+3K1LbYZCrEJz5I/RXRV6jG8XfcuPm5fMjmsN0ws2UcSegVyxl9ycS1pVqyvFbWdBcZfGLlwqQSzIJUHluoZGG9B75ZUf/LpOlCEX8RzWzg9sXfsuXrP+BTUDO7ggQkfEsMkxFrzUCns+RV8jvT2CvdTCfmGloMbV+7tb29QZnXIm+vuAdes05jgTOdGa6OLPyZwueBTrb2gtUuAHYqY1W95pcDCRhGmsaW6UkmOD0BuOamSvfDERivTR4IHZDAVm7oI9gatwIggjwkmU6cDp/jY8sBSJ2oRYM+DeEoY1dv+YRZR3/9ZxtdAyMn8FFlkRc2uitFl0Jy4qKGaIuPycz2qRl8aN8ZUzDWfiVvMqefaDPOdUTFaDJJTuCmUY02UagCwlD67j1IMlNbILxGerqPR67aq5wszOuJsaQtQO1xorVR0b/3K+j20C+GCxgxxZ3w3bzfxmcnejbsYhnH6qnEVTPXqQShAtjelKiy9XxXpKG0Ajp+7rBpH+Gl3VE5pdLuXZArraSLR0JZ76RNgz8vyn/D2Lex+6PyOHzWN2kiSNXRyHEWo6ZVPgW3pMwU342SfG870t49NNMcVhdSPQ8GFuMxOU2wllg+S+BpQirrl2WD3qUiJMcaVBQjMo+kQPjMfN6vDXcYJOBp/DKr7OVWa8/jh1EfcXzo4F3weGIYt9vst39NdHk3Gtu4kkRG6Za1znZNWIbvycBjn/V6Fn+6pi+DRYob5tCqETXVYUg4Auz99vmqAWwpSXdtD5QaKJIhJL+ASDwAtENrGxGZd0UDAMs81DVRw+9+KADqD9OHVdnd25fbKGMbZR51tkSJ+51xm2MfcTeg9TN/dIJtTGCAAYWLpX3oI04zewxflNlJcV4RFDk3qADxtijcz+v2mQKVxIptNS356kwGh6QNjxEYV5r/kRHHJi74cUgsXCrgCdzZK9Kte94hbuTXW1vpp08xpO3VD1zN5asC753oa2MfB1EO3gv+E812XUzwZTcyuKda4gAQiPg91vv0Z99htg19JJsDF6a/tUEJxpkBdVhBEGhNKk+Cne9rXUMiY1d96qbcWEQ51bmZRHRPsGOfaHxV8idp1T3hkIdSZyyB8Vgu3RUVLg0yyBQx8v89pqAl8fkBMF8v0SKxyFBPTrcD5TDVjBPFAo5+NWNPEydKtM51yKFbJCePLnPHKiDBA8Qr6txKrKMGCfaivviJ+BeXsPBiwl/vNT+HrsNaSwBnT8ukcQv8Lgi2Z0/Ps8h7F9binDuqpHI9hu+S2FN5Fm15F2fYG750UMuyRVYp0iFzUOzeNq8jG2k3+ X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6a3f5e15-3f32-4235-967a-08d95b665551 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Aug 2021 18:48:52.6763 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 0fsA5fgxFnZ2I7haR3QKr3ixJJknvDONB5Y3gu6fGAmpPs8+bKP+eOo5JrJ7Ok+x4T/WSK1fQo12rob6Tr/zSg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB5455 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10071 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 phishscore=0 adultscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=999 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108090133 X-Proofpoint-GUID: JqyZyoyrVN_yv7KLKO_i3um5mEjHYuhm X-Proofpoint-ORIG-GUID: JqyZyoyrVN_yv7KLKO_i3um5mEjHYuhm X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 314B890000BD Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=Sxx5PAq7; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=YbvjlL8P; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=mK2fzljQ; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf23.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=mike.kravetz@oracle.com X-Stat-Signature: 1u4dczthanta51q98ibbfq7qhd9us456 X-HE-Tag: 1628535026-590035 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When removing a hugetlb page from the pool the ref count is set to one (as the free page has no ref count) and compound page destructor is set to NULL_COMPOUND_DTOR. Since a subsequent call to free the hugetlb page will call __free_pages for non-gigantic pages and free_gigantic_page for gigantic pages the destructor is not used. However, consider the following race with code taking a speculative reference on the page: Thread 0 Thread 1 -------- -------- remove_hugetlb_page set_page_refcounted(page); set_compound_page_dtor(page, NULL_COMPOUND_DTOR); get_page_unless_zero(page) __update_and_free_page __free_pages(page, huge_page_order(h)); /* Note that __free_pages() will simply drop the reference to the page. */ put_page(page) __put_compound_page() destroy_compound_page NULL_COMPOUND_DTOR BUG: kernel NULL pointer dereference, address: 0000000000000000 To address this race, set the dtor to the normal compound page dtor for non-gigantic pages. The dtor for gigantic pages does not matter as gigantic pages are changed from a compound page to 'just a group of pages' before freeing. Hence, the destructor is not used. Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song --- mm/hugetlb.c | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index bde324e69276..214a02232d6d 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1370,8 +1370,28 @@ static void remove_hugetlb_page(struct hstate *h, struct page *page, h->surplus_huge_pages_node[nid]--; } + /* + * Very subtle + * + * For non-gigantic pages set the destructor to the normal compound + * page dtor. This is needed in case someone takes an additional + * temporary ref to the page, and freeing is delayed until they drop + * their reference. + * + * For gigantic pages set the destructor to the null dtor. This + * destructor will never be called. Before freeing the gigantic + * page destroy_compound_gigantic_page will turn the compound page + * into a simple group of pages. After this the destructor does not + * apply. + * + * This handles the case where more than one ref is held when and + * after update_and_free_page is called. + */ set_page_refcounted(page); - set_compound_page_dtor(page, NULL_COMPOUND_DTOR); + if (hstate_is_gigantic(h)) + set_compound_page_dtor(page, NULL_COMPOUND_DTOR); + else + set_compound_page_dtor(page, COMPOUND_PAGE_DTOR); h->nr_huge_pages--; h->nr_huge_pages_node[nid]--;