From patchwork Thu Mar 25 00:28:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12162587 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70DF0C433E0 for ; Thu, 25 Mar 2021 00:29:19 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C33A161A14 for ; Thu, 25 Mar 2021 00:29:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C33A161A14 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A325B6B0070; Wed, 24 Mar 2021 20:29:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9E25B6B0071; Wed, 24 Mar 2021 20:29:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7723F6B0072; Wed, 24 Mar 2021 20:29:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0072.hostedemail.com [216.40.44.72]) by kanga.kvack.org (Postfix) with ESMTP id 4B3B46B0070 for ; Wed, 24 Mar 2021 20:29:16 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 0CDC8B7B9 for ; Thu, 25 Mar 2021 00:29:16 +0000 (UTC) X-FDA: 77956512312.15.736D89C Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) by imf02.hostedemail.com (Postfix) with ESMTP id 0A162407F8F8 for ; Thu, 25 Mar 2021 00:29:12 +0000 (UTC) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12P0O4dx143453; Thu, 25 Mar 2021 00:28:57 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=FNlo68BHX6lihDW55viDAfC/BJhdtoMlsCflcTEe3wU=; b=s0ngk8ShMOjVtmvBO08UkCG21FiSDil7J1Ld/nG+2XMM8AdOtgAvLqQRIyYR4xvp+W3H MCeBzi/IVAvqvyEWz/GdYRfd2hgFU1as/aldoGpAUr0MAevRggF+B/GeL30sAo0YIZJZ syH4cspKCgh91Zz8rJN57ufhch//weB34mXhX7sTxD1tGYAAVYeGVlNVG15BXQz3DXnw tAzlTEPqo90CWx5tZrjg0DessMsRrY5Ahldo8EqT7PyGU5tdJG2LwVCqRk0c0W0rHnay DQqU/OjiQWVqQ77dlDwJV0S4ZxTi1EHQZnkrEDr715MxPXu9gsgBqbjMomRLl6iRGsg3 iQ== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by userp2120.oracle.com with ESMTP id 37d9pn4k6u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 00:28:57 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12P0PEnf162585; Thu, 25 Mar 2021 00:28:56 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2109.outbound.protection.outlook.com [104.47.58.109]) by aserp3030.oracle.com with ESMTP id 37dtmrgq90-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 00:28:56 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WmIndY/47P4lO11WnSMojfhUcH7p9Nk5gruYzlEvYWBt+QMjzSlgRGSI1iTupEH21K8PpgPeoqAZNsiJuWFFr6FFmDT5pivfmUx8Z6dyOiHk14PYLcKkzuetZwWlaSnT4GMZ1jpVz+K3iA+FKSquNRGcSv0I0yeudUMYx414n2HO48g9VmPhpXuNA6UYNrEMEK6f3gdt/vhwdmaINTFTbDA8qPusAU4JnqOAX/i5avGW3gNaAs9j78EQwwGs0aGLHL8wfH8ELFz/aWZD8l+IXC46ESG3kTqJvB3qivRxMzFHFXkxnrkBwVruHJCYxgXBzKPI0eYaNpKzCoej23goTQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FNlo68BHX6lihDW55viDAfC/BJhdtoMlsCflcTEe3wU=; b=YfVwzHTevwBB1XSziMN0OcDs4xF0LiyzaxGNb1PC0QcUoD2o6f10HHJ/Y5buEM7ji3lRRwqvGGm5XRq+uv2qYa1SfJR7CzRyQYsOYn4TXsSqMToj20CDuctlj9NQVqxFmJ5k7HSZGh+saJobeqNFStLjc6aI652d+LB5Vm789L239aV8NT3f92r5TPY9yj38ks151StacC8sKZtygF0liOOTrDk8GORWQ63DFW3d28pFUCmC6hweTXs5JNGcIoC4QXhqRbbocHpbVpe0xSw2jBsoCNVc/3bPyLWbL0lEZq3mGL16wUEa3Na21Uy1rBgbZk10QgI0nMrmMuZomYB43Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FNlo68BHX6lihDW55viDAfC/BJhdtoMlsCflcTEe3wU=; b=OgfiSvLXX6AB/VYLxjYgHi1sBU59LGsfGRI6VeP19qsWY4+4mrRYjbLrrZCQCNcRsw6ClYmBDE3ClNZ+9rNzo4uwai0Hx5BbOa5Bp6v8aD8Mh/rLUE0RGKifEB2iDCbMPKR0BMqIwiQFWQ4bcto7GO7dyuFLSLkasPYEOMS9qFs= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by SJ0PR10MB4462.namprd10.prod.outlook.com (2603:10b6:a03:2d7::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.25; Thu, 25 Mar 2021 00:28:54 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3955.027; Thu, 25 Mar 2021 00:28:53 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Roman Gushchin , Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Hillf Danton , Andrew Morton , Mike Kravetz Subject: [PATCH 1/8] mm: cma: introduce cma_release_nowait() Date: Wed, 24 Mar 2021 17:28:28 -0700 Message-Id: <20210325002835.216118-2-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210325002835.216118-1-mike.kravetz@oracle.com> References: <20210325002835.216118-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR15CA0042.namprd15.prod.outlook.com (2603:10b6:300:ad::28) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR15CA0042.namprd15.prod.outlook.com (2603:10b6:300:ad::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.24 via Frontend Transport; Thu, 25 Mar 2021 00:28:52 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 12e4c502-05a0-43f9-4080-08d8ef24f816 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4462: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6790; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ueEutneeFVMGPO4y2kLV7l5YM0mJMydzmdLBuwlHuck0L04d9/b+tbZzs2GDL5TEefBR6rv5DpRJb8k2TEav2mIKWgJGJAJ8Ttl505tJ+xcmSzdaTgde4U5t67hwWqu0K6erVxXWpC2ZzMw+YHwLyhPOD4V5AJ82nwVS3iRU9dcaFf/N4QXT6EAMCxuB7eXwLl+rjDmKxKjqIJWFf8Qdm/HxtXb2gX2ru/5umuKOkKAUimi+704WKhJvs85RCe2SQfbeJPhzO5RN8K8d/yiTrKlCZ+w1hPK/7f7zZef4XfPoIr3T3MZqoL9xZ3xbqbWfVWrf63fzdSZvCXF5mdDAw48G4QV0SXgFvYrfG/3VAOd1ac7U4F1xaKFHkkqDrZ9jt9ggbAEV4JxcUzM7t8R7O1vK3YSKTDc95CK6YhbZ7GNONXYe2/b+S0rMF7PECtDE82wHMieWLHDtVGiz3zBUWq9zP2T45Ftbwhx4ejlmlXJDYKFsHq7XtDx2m3Y7ErnDhr5egapcF4X0qZMMhxMIzOfoU/jqf36qNvxG7fTusDkGp01BvbPQ3CB33uTcVlb/qCwRscfjcC8k+BmICZ2N93pY1f6ZQ1dJJSVUC2FYbqz6zYx51PTQIBTnDkOARANwH8lSifS4dfWxkw/dljjLJr+1wLNFi4h6Hv7OOl5rm3U= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(346002)(136003)(396003)(376002)(39860400002)(8676002)(66476007)(66556008)(86362001)(6666004)(4326008)(66946007)(956004)(8936002)(107886003)(36756003)(44832011)(5660300002)(1076003)(2616005)(316002)(54906003)(38100700001)(7696005)(26005)(478600001)(186003)(83380400001)(2906002)(6486002)(52116002)(7416002)(16526019);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: AqLl24HnfidMAma5c5Ic2ehjFNPoZ6UdhM8XKvo3eX03G6cBksIVPkS2mvlizmD00NoamUomVG0Gzgcpit/lzEHiQuovFAEJxheMepDOI3b/gp5obIw+rt6TC9xdVJr7k7RldkRRyDz/y3hc3av+Mx3NtXuKFPaVTiSXDK61p7kOl4aKfsyfliy0HLzor1hhmx1lGroSLajqwOpr6C3jWmkODFs0S9OUOA025FkKRCE7VPliqh5yzdpB3fns0ijr1oYsW+ULceO1TnOF7TgjaY2lL8WV9wTDI5Lz44+mHSn28CTaUaww6QTaJmC1OFpW7QklEoC1XDUgMsAN9ibU3uHhzv7lt9gCd9yNPsZyvF7b/vj+QK/7RF9/C3fojulUzoNO+ncnpXJN9RANHW0mbjHllgkJMYu1ZsswJrLuQxC+V7pOQN3K0MRysWyCVYBzs9Ixk+XPTbNglD6391MJvmjnalFjSOa7E8MQMUYcYrjhF3rRdpeaRanm91gUopfORA5Lm1g3cywUf59Kj3yTmDOjzxQ58KYEnKeQheOTO532OsWUejBcBLu2X3lPQI0fmI+/zeAiqVGe1EP/R1iXaMmXAwnRxqOPv8b+F5TzzDVrMth1MdnkOP7pB6sJjZfCw5UlSlOK2kfoVGQ1ZSukIK8S2O6FG2be0z6RemL12k/D55Rl1c+1ufWhzwI5coi1Z9mF1kEz1W2iJWDMPMRUvE8iboIKUexn96Iu7Zl9bncoFSgfDmo+8TLcR4gE/atavNW7yEsrtBHew4KBaptoFL3wfdAKEosVSqjozoK0mvFit7VQUFHUJzpnJSw02Qg227uNZQ+8NA26Dw3IBnASzcF8RFlXkAsCOdKRDlBk5gqd7EELx2N6W1emjQMIKUNW04o08GlfkEidsEaTKAX4tO8lrA+abGTbaZwUtyWRchbY59gipdlO+qzhPJddgrrBtIwILFVTIIWzWa7kjOrUKHv9lnNYQTV2HcjOMZRyinZaW8yuBuQahKtMJgXh9/KzCe0v9QQBEbTo2cQTI3dgq6etvU2n31tUMAoscKW6CFQ3S+8K5E3nb4dPkOTng5Fh8UeKckm2ZRgbhr2gHG/9qiJPaEJWsSWs3SeU/VCsaOK35cuSittJzKC1hj6HzFnlOTyVtJD8MQgfBdi1rzrJJmPXkZU/DZLhHGAS/e7rST3GgV8kKV9/ws+xGJm+4LwLzJ+a3jHA+UNjDswEeUDAs9FplpPH+rp8B00vi1ZT9ptyd3C2urM1ouwOqw63bJQEjmSXSGEq5hKUpTvdWsmEvNRCqnTAo5mbfxL/KYV4wP23A9OssqbsdGNVJStgwTsH X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 12e4c502-05a0-43f9-4080-08d8ef24f816 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2021 00:28:53.6966 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: LnJSQRiM+w5ysyrhzCbzZonCKAXpfgVd76xcycixHLPkEcTzhIU+3TkP9FCeDn61bJMTdgh1cYYZWW/+gbO48A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4462 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9933 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 mlxscore=0 phishscore=0 bulkscore=0 suspectscore=0 malwarescore=0 mlxlogscore=999 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103250000 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9933 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 lowpriorityscore=0 phishscore=0 mlxlogscore=999 priorityscore=1501 impostorscore=0 bulkscore=0 spamscore=0 adultscore=0 clxscore=1015 malwarescore=0 mlxscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103250000 X-Stat-Signature: nqs85am7omfkagijxaqqer7upnnffnwr X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 0A162407F8F8 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf02; identity=mailfrom; envelope-from=""; helo=userp2120.oracle.com; client-ip=156.151.31.85 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616632152-931955 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Roman Gushchin cma_release() has to lock the cma_lock mutex to clear the cma bitmap. It makes it a blocking function, which complicates its usage from non-blocking contexts. For instance, hugetlbfs code is temporarily dropping the hugetlb_lock spinlock to call cma_release(). This patch introduces a non-blocking cma_release_nowait(), which postpones the cma bitmap clearance. It's done later from a work context. The first page in the cma allocation is used to store the work struct. Because CMA allocations and de-allocations are usually not that frequent, a single global workqueue is used. To make sure that subsequent cma_alloc() call will pass, cma_alloc() flushes the cma_release_wq workqueue. To avoid a performance regression in the case when only cma_release() is used, gate it by a per-cma area flag, which is set by the first call of cma_release_nowait(). Signed-off-by: Roman Gushchin [mike.kravetz@oracle.com: rebased to v5.12-rc3-mmotm-2021-03-17-22-24] Signed-off-by: Mike Kravetz --- include/linux/cma.h | 2 + mm/cma.c | 93 +++++++++++++++++++++++++++++++++++++++++++++ mm/cma.h | 5 +++ 3 files changed, 100 insertions(+) diff --git a/include/linux/cma.h b/include/linux/cma.h index 217999c8a762..497eca478c2f 100644 --- a/include/linux/cma.h +++ b/include/linux/cma.h @@ -47,6 +47,8 @@ extern int cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, extern struct page *cma_alloc(struct cma *cma, size_t count, unsigned int align, bool no_warn); extern bool cma_release(struct cma *cma, const struct page *pages, unsigned int count); +extern bool cma_release_nowait(struct cma *cma, const struct page *pages, + unsigned int count); extern int cma_for_each_area(int (*it)(struct cma *cma, void *data), void *data); #endif diff --git a/mm/cma.c b/mm/cma.c index 90e27458ddb7..14cc8e901703 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -36,9 +36,18 @@ #include "cma.h" +struct cma_clear_bitmap_work { + struct work_struct work; + struct cma *cma; + unsigned long pfn; + unsigned int count; +}; + struct cma cma_areas[MAX_CMA_AREAS]; unsigned cma_area_count; +struct workqueue_struct *cma_release_wq; + phys_addr_t cma_get_base(const struct cma *cma) { return PFN_PHYS(cma->base_pfn); @@ -146,6 +155,10 @@ static int __init cma_init_reserved_areas(void) for (i = 0; i < cma_area_count; i++) cma_activate_area(&cma_areas[i]); + cma_release_wq = create_workqueue("cma_release"); + if (!cma_release_wq) + return -ENOMEM; + return 0; } core_initcall(cma_init_reserved_areas); @@ -203,6 +216,7 @@ int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, cma->base_pfn = PFN_DOWN(base); cma->count = size >> PAGE_SHIFT; + cma->flags = 0; cma->order_per_bit = order_per_bit; *res_cma = cma; cma_area_count++; @@ -452,6 +466,14 @@ struct page *cma_alloc(struct cma *cma, size_t count, unsigned int align, goto out; for (;;) { + /* + * If the CMA bitmap is cleared asynchronously after + * cma_release_nowait(), cma release workqueue has to be + * flushed here in order to make the allocation succeed. + */ + if (test_bit(CMA_DELAYED_RELEASE, &cma->flags)) + flush_workqueue(cma_release_wq); + mutex_lock(&cma->lock); bitmap_no = bitmap_find_next_zero_area_off(cma->bitmap, bitmap_maxno, start, bitmap_count, mask, @@ -552,6 +574,77 @@ bool cma_release(struct cma *cma, const struct page *pages, unsigned int count) return true; } +static void cma_clear_bitmap_fn(struct work_struct *work) +{ + struct cma_clear_bitmap_work *w; + + w = container_of(work, struct cma_clear_bitmap_work, work); + + cma_clear_bitmap(w->cma, w->pfn, w->count); + + __free_page(pfn_to_page(w->pfn)); +} + +/** + * cma_release_nowait() - release allocated pages without blocking + * @cma: Contiguous memory region for which the allocation is performed. + * @pages: Allocated pages. + * @count: Number of allocated pages. + * + * Similar to cma_release(), this function releases memory allocated + * by cma_alloc(), but unlike cma_release() is non-blocking and can be + * called from an atomic context. + * It returns false when provided pages do not belong to contiguous area + * and true otherwise. + */ +bool cma_release_nowait(struct cma *cma, const struct page *pages, + unsigned int count) +{ + struct cma_clear_bitmap_work *work; + unsigned long pfn; + + if (!cma || !pages) + return false; + + pr_debug("%s(page %p)\n", __func__, (void *)pages); + + pfn = page_to_pfn(pages); + + if (pfn < cma->base_pfn || pfn >= cma->base_pfn + cma->count) + return false; + + VM_BUG_ON(pfn + count > cma->base_pfn + cma->count); + + /* + * Set CMA_DELAYED_RELEASE flag: subsequent cma_alloc()'s + * will wait for the async part of cma_release_nowait() to + * finish. + */ + if (unlikely(!test_bit(CMA_DELAYED_RELEASE, &cma->flags))) + set_bit(CMA_DELAYED_RELEASE, &cma->flags); + + /* + * To make cma_release_nowait() non-blocking, cma bitmap is cleared + * from a work context (see cma_clear_bitmap_fn()). The first page + * in the cma allocation is used to store the work structure, + * so it's released after the cma bitmap clearance. Other pages + * are released immediately as previously. + */ + if (count > 1) + free_contig_range(pfn + 1, count - 1); + + work = (struct cma_clear_bitmap_work *)page_to_virt(pages); + INIT_WORK(&work->work, cma_clear_bitmap_fn); + work->cma = cma; + work->pfn = pfn; + work->count = count; + queue_work(cma_release_wq, &work->work); + + trace_cma_release(pfn, pages, count); + + return true; +} + int cma_for_each_area(int (*it)(struct cma *cma, void *data), void *data) { int i; diff --git a/mm/cma.h b/mm/cma.h index 95d1aa2d808a..2063fb5bc985 100644 --- a/mm/cma.h +++ b/mm/cma.h @@ -17,6 +17,7 @@ struct cma_stat { struct cma { unsigned long base_pfn; unsigned long count; + unsigned long flags; unsigned long *bitmap; unsigned int order_per_bit; /* Order of pages represented by one bit */ struct mutex lock; @@ -31,6 +32,10 @@ struct cma { #endif }; +enum cma_flags { + CMA_DELAYED_RELEASE, /* cma bitmap is cleared asynchronously */ +}; + extern struct cma cma_areas[MAX_CMA_AREAS]; extern unsigned cma_area_count; From patchwork Thu Mar 25 00:28:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12162585 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FCF0C433C1 for ; Thu, 25 Mar 2021 00:29:17 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E821E61A12 for ; Thu, 25 Mar 2021 00:29:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E821E61A12 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 502D36B006E; Wed, 24 Mar 2021 20:29:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 466926B0070; Wed, 24 Mar 2021 20:29:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F7776B0072; Wed, 24 Mar 2021 20:29:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0101.hostedemail.com [216.40.44.101]) by kanga.kvack.org (Postfix) with ESMTP id E8D436B0070 for ; Wed, 24 Mar 2021 20:29:14 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id AF53C18028E83 for ; Thu, 25 Mar 2021 00:29:14 +0000 (UTC) X-FDA: 77956512228.23.1EBE54B Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf20.hostedemail.com (Postfix) with ESMTP id D035B132 for ; Thu, 25 Mar 2021 00:29:12 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12P0OnEe093913; Thu, 25 Mar 2021 00:28:58 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=DmpR3IHpRf0BF79O8TyIEbdtG2PrSinAJXwQxxRNiOA=; b=QRgSFcnU7E/nEQw+H5ueF2YQKTytUKW+6FI81EhJ7nY/q62LxtgP9df8s7qnRXCwcvp8 GxDEzjk9rfX88n8SeJQcnOPFU4kggDOi2bmptJ8vHkIRcx5JwmytBHmY2I3qU+nj5ci4 4Zr6t/mlRgjZ/M3AK0t8EITvIpjdkFxaYV8Ef/zzBywzMJASn+3bHj8ad2ZLVSmbwt+c 6CdxpSKhb6Loly5SYTqsa1exFlkPYmb/P2Pv3rKWtC08ztqhrMMmSn/DvE8mcFAyrnIE OPiZ+8NYiSBwIwU4m46wLXIYpjOZ7w0pNqNA4MOtz3JADkpPYfR7tMwZUGOEqjX4LKtS iw== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by userp2130.oracle.com with ESMTP id 37d8frcn7n-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 00:28:58 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12P0PEng162585; Thu, 25 Mar 2021 00:28:57 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2109.outbound.protection.outlook.com [104.47.58.109]) by aserp3030.oracle.com with ESMTP id 37dtmrgq90-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 00:28:57 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ihfpiSisCL2vSG75DSjtcLY+0L4R1kfNPipxFZKjC1rJ8wRx8L5p1kwbIdfRRATAa/KfMedWm2QmPMLXvbrZDXhZkNs8EwEi1CpVF9Bgt0IVTXIFri7+d7LU204jCfy/Zja5aFKUfRb7JuM+QxuttM+VwlA+oLTzrUOS89NCa6n9lxOkUj4z611ySuZaC/H8oFUzYxD9Q/UlDgN4KNZFtqs6ybDNACBJFgubWNEps6CgcZVzxWoS2XPTB1hkYum6FGJXZDc5b5aXPrzaj+2W4Al30Zgvoq0mEAayOKys/TpTuGibuloSZ0aiLseuF2/jCDOgHmTgAtrSEEjQovBWLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DmpR3IHpRf0BF79O8TyIEbdtG2PrSinAJXwQxxRNiOA=; b=DdyT64prTQ1pyu5+yQHIeNhUvq1HGC05vICeCYorDzqYqjRBK2nBFSDUuIkdhmCZC7p4BV9ddEaA13/Zk8VLYVg+HxxFSY+/OOP5VNUEZXEZrvKNwIvOJ8O28JuinAI25eDsWOsb4uBjFOC1glLy6MI7ZKGg7a4E2CSJZSQrMZ2hRrgLUbh184ytlOsdu4WFe9e5YqjaB0Bs1WwwkjlxLLpbvkCPYTO5sP5axC6d5CZxDv1qZqHYpyw5HJyOHElZFD5B82MdwEtPTzaI0sMovfaUeZXrJ3y+oudo9HD0pHUmSsE407HfMdBsdj8ltWyyFmxrVHcrGe4piEY+8t2B9Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DmpR3IHpRf0BF79O8TyIEbdtG2PrSinAJXwQxxRNiOA=; b=ojiEW73JqEiT/G/GCT3IplaHGYGOc01+EqkIjM0Ps3iAfJhGFsnK1Gx4wXNq5ZYKpT9U5kSV3cKYAiQ0cWQD3vz9wE6x511SGc+KTZHYwVjkfiMkY2MCL79n8rZAp8y0jhI/XOCVwQHGABQPaKLZgAiK8mjRaT0jRplBGtHMlk0= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by SJ0PR10MB4462.namprd10.prod.outlook.com (2603:10b6:a03:2d7::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.25; Thu, 25 Mar 2021 00:28:56 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3955.027; Thu, 25 Mar 2021 00:28:56 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Roman Gushchin , Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Hillf Danton , Andrew Morton , Mike Kravetz Subject: [PATCH 2/8] mm: hugetlb: don't drop hugetlb_lock around cma_release() call Date: Wed, 24 Mar 2021 17:28:29 -0700 Message-Id: <20210325002835.216118-3-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210325002835.216118-1-mike.kravetz@oracle.com> References: <20210325002835.216118-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR15CA0042.namprd15.prod.outlook.com (2603:10b6:300:ad::28) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR15CA0042.namprd15.prod.outlook.com (2603:10b6:300:ad::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.24 via Frontend Transport; Thu, 25 Mar 2021 00:28:55 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8cb70555-4e50-4bc9-8c78-08d8ef24f995 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4462: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3826; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: iiidFO7zLcG2+s0XtvSDrnKwt4M/XFhdKOLECLwhinAPpN+LY44bpqqK255HwkgEbQNiFYWFH2+eLO0cPRkOBz67dTY/Qg7yVdNvDDjg7aJBTiKt1t+nYWf8NXhQeeQtZDVc2kbKZAL31S7ARkpytzvB72+paFFIXbvwsCpGy5TNn9KDS3ePxb5zXwcBfDuJv5EJm3jcPPVz2rt1WqFUFmVJqRbAUARgU0lZerB6VAQg4IBKN1uQ7Qgoni5+3q+iFMMy7gPOi7LN9eGJPUkRXoVcevkrSrWoIBQIugYY2YKDcWbb6jbyGSuZMU9c6jPLG/jAXD0rMJuhS7jHuwTym2dmo5VrM90H9Ax3iVrlKZaSd+JZWya3IROFPIdctGZkjQzSjDlDNXH0IKbBx5JsIelDW7CWSDUMfWYqRJcqdoWi3mR4gI1GmZwcMTAqg85vUniEnK1QDuBs4y+b1jU0luQo4JH55rNJ0DIxiFio9ji664kDbTX/DyVQnr50c54Vh5+GG52/OQMlM6Qy7TBiZVtYtWHZ0zZ477EBOU6mO2LkaYbQ+/qGG2TlVwVDkRncoX6QxOB5bwvD1rhhdiJL/TXbw+mebDaBT2SDRl6c/Lca4LXm8NerOKgpxlhJNps22oOafphtbudBh+tTLYKAAg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(346002)(136003)(396003)(376002)(39860400002)(8676002)(66476007)(66556008)(86362001)(6666004)(4326008)(66946007)(956004)(8936002)(107886003)(36756003)(44832011)(5660300002)(1076003)(2616005)(316002)(54906003)(38100700001)(7696005)(26005)(478600001)(186003)(83380400001)(2906002)(6486002)(52116002)(7416002)(16526019);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: vAcymb4fQTlIePyEYxNJT1ixGhLASInvei4/DBMhStNi7YG/eOZ863qk8pLdINh/jGEjuC9NfneJMqMPVtmMQbZ4ZrFRdIUNE5GAJAiXk7NRI53Jv7g1cn+yV4yQoZQTsMHFixHlkDRG0DPz8LOUARoTZ4AsxRA6BIB47spngh2dVoElm2ArVJ1a8YxT/vKOMoQV5vi8V9SgDREZINwU8CSfBZBCGmKuxXM8zcuDY7t8h1WfW63EDU8Cn0q0fAaIUMsTVh7fnzio3WO0jVUI8YjY72CEUGW0bA//8ooPEJl2qeBVWsoEA2uvPfJ1rl80E1rSBKKrMai254DYp/ttkf4imCjMVi+qx04Y6GMrr1hoOfMVjk8pK3bC8nRI7bnFncs6/JoEO+9ZcB/OUevXmtm2aF1uUIfuAXO/K2N+TGLqnIzuc+ScT8BmJZ7m7h5t8+/jl6gNSFZLrmMsCN2CTuQ1lGyoaztc8xOmWKlAiTSRDs8QIxmYsl05gHvzEHDvMis4MxRYHBwd/RlhWw2iGj6r5x93Q6TMxt+i6yQYW6sQfKetsPasUnBuDxcrgWV/Sz7cUMZRHu4WCGgEbPn1bb/Gsh8T6Wp1eguRRRc4JZLZUoNj7HbpiQlUW6oNASRCgRPqxbaMThnBQBuxhUA47jds+kp6i1xCKt41qKJYY+R9v+m0PzEAaVwz+httse/yetH0DKhGqgjGsWGrbRHwaUJdQ3S97QVaXkvWm/jAzcsWShQeNJrGsgJl3KW87/qAumOWLCuaP1KIJrodYp9MwIu7YM6MmtXNa5frwnIsUuONs+tIJpZS4SQAGdCE0AgjdoK8OSKDZ0QjdO6m6MpH9LEbSyT1Ej4njuQFRuBGu2Qa//3PvBVyKrZ0v38qxRt7PQQXJ/XLphLxsTCDUV3lqePSMWqaFiAsNrQ5SZqhA2CmUauVnkmvzy61QnX70Io8BMuklfJoV8m8YqMbN1mTfeh+B4hyoe7eqssQqTVITfJItul+pPeuFRDtGQ2zg7uDUnBjCS7kz8fR4+Se2Cciuj7tnT4KG84o2sLqM/cDJJMHHVqGSIAeLP3BYWKYdcOpA8w11GrYmRZXI387vaCUc1T+tmUn/GcCLaL89IUqUU+xRJ9Uyyb4sw2e5lxISREYqP3qCP6jqLELCZJpJZ4O6kJLtwlIHDijIO+5jhD6QNTO920tKNkG86uyNTKRNC659Ac+qx2Elgz4GIavzGqapAh7nOkR/MgxKVJAb5/+KVE+tBc1DjWFDo1j6/27bogNiXeqeALhSvYC4/F2N9TMrHciagMs5TDHLiY64RLkMlWvYC4Wco5a9weurAnN35++ X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8cb70555-4e50-4bc9-8c78-08d8ef24f995 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2021 00:28:55.9497 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: knY4k15ZQKAPj2nb7+j3+ol3F0IJpjDtbxWrTwXWAJlXlVSN4K7citLydJXmZ6KH2X+p4bOyCkQO6CGMX3b+HQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4462 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9933 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 mlxscore=0 phishscore=0 bulkscore=0 suspectscore=0 malwarescore=0 mlxlogscore=999 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103250000 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9933 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 lowpriorityscore=0 priorityscore=1501 impostorscore=0 spamscore=0 mlxscore=0 suspectscore=0 mlxlogscore=999 phishscore=0 bulkscore=0 adultscore=0 malwarescore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103250000 X-Stat-Signature: 8qjirfy4gawez4nhkna91xqgrn5bmuyt X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: D035B132 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf20; identity=mailfrom; envelope-from=""; helo=userp2130.oracle.com; client-ip=156.151.31.86 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616632152-119834 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Roman Gushchin Replace blocking cma_release() with a non-blocking cma_release_nowait() call, so there is no more need to temporarily drop hugetlb_lock. Signed-off-by: Roman Gushchin Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 408dbc08298a..f9ba63fc1747 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1258,10 +1258,11 @@ static void free_gigantic_page(struct page *page, unsigned int order) { /* * If the page isn't allocated using the cma allocator, - * cma_release() returns false. + * cma_release_nowait() returns false. */ #ifdef CONFIG_CMA - if (cma_release(hugetlb_cma[page_to_nid(page)], page, 1 << order)) + if (cma_release_nowait(hugetlb_cma[page_to_nid(page)], page, + 1 << order)) return; #endif @@ -1348,14 +1349,8 @@ static void update_and_free_page(struct hstate *h, struct page *page) set_compound_page_dtor(page, NULL_COMPOUND_DTOR); set_page_refcounted(page); if (hstate_is_gigantic(h)) { - /* - * Temporarily drop the hugetlb_lock, because - * we might block in free_gigantic_page(). - */ - spin_unlock(&hugetlb_lock); destroy_compound_gigantic_page(page, huge_page_order(h)); free_gigantic_page(page, huge_page_order(h)); - spin_lock(&hugetlb_lock); } else { __free_pages(page, huge_page_order(h)); } From patchwork Thu Mar 25 00:28:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12162583 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F8D4C433E1 for ; Thu, 25 Mar 2021 00:29:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D090F61A1F for ; Thu, 25 Mar 2021 00:29:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D090F61A1F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5ED356B006C; Wed, 24 Mar 2021 20:29:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 526F26B006E; Wed, 24 Mar 2021 20:29:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2B6FB6B0070; Wed, 24 Mar 2021 20:29:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0178.hostedemail.com [216.40.44.178]) by kanga.kvack.org (Postfix) with ESMTP id 0C6CE6B006C for ; Wed, 24 Mar 2021 20:29:14 -0400 (EDT) Received: from smtpin37.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id C8458AF6B for ; Thu, 25 Mar 2021 00:29:13 +0000 (UTC) X-FDA: 77956512186.37.18B6FC9 Received: from aserp2120.oracle.com (aserp2120.oracle.com [141.146.126.78]) by imf27.hostedemail.com (Postfix) with ESMTP id 17A6680192D8 for ; Thu, 25 Mar 2021 00:29:10 +0000 (UTC) Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12P0P3VG005354; Thu, 25 Mar 2021 00:29:00 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=fDy69tQqeQmirM8UdgckDtRnyaLn8eOm9TxH2BQksXs=; b=Y4u1q1GG769GDBTJD0OJg/tuLvIIphmp5QLGRNmhSXRqez2wJyR54y7ZmiHrt7leGHSt /gfX/LJs1mCde8InzKp4M9AVp7UJWpXhL+p7gpgqcApdN60GHfFXzxNyhuEEMtTfXHZE mkha6bNs4pny+XwnJuoWfxqkEPRp+BBFMPpDS28SkYD2yvE1Qeitm2ykGBTq6NBu4cU0 aPJqUwdYnc1ulmO/vOlt5KRjoWvJqH2LgZRZEo7ovVgwZnaU7x8T8eBZYuvFO2T+PycS mkVr5CprpKNuckP6btLRCIjiI011trpsnDCu5VzPwhWlz5MqBQ+8mYIwR+1mbnbCOxY7 Rw== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by aserp2120.oracle.com with ESMTP id 37d90mmmcb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 00:29:00 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12P0QTQ3040269; Thu, 25 Mar 2021 00:29:00 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2171.outbound.protection.outlook.com [104.47.55.171]) by aserp3020.oracle.com with ESMTP id 37dty18nys-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 00:29:00 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=I/iM2mX7SvELFRsS4l7HCS2KRliJl5cChw7/ss/VChvxmZM7lePHSkbqQ52A863SMhsbGyARkrJeC0zZma1EGI+k68Sl7KtqRm2SU6cGbxrcaPBOZrJgqztU73V5yqENfm5XPdYGN+nDPeyycI06PGMeeByuh9l4COVwPs305fb4X9+B+Ahb/ia70tgAhpODyJ2I22Qi3uV3vhGFHx53zz+gyUQs34EshhbNjFZMNzNsPc86s1Mi6Ta++3W63PRZ3PlrzDK8RzRQqBLqfUx6DErc2BBRTLrciYJJmnNwFi7esNWrAhp+B5ZnKaUaoR1SvT4xkJrBm4URCp2FZld4DA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fDy69tQqeQmirM8UdgckDtRnyaLn8eOm9TxH2BQksXs=; b=QnPzXWaS5W7176LvkKkSo9fjjmPQN7pnHGGzUO8JBnJeDeah40hsmrNMpkMPBSrwvIQmu1A+5yYa7Tx6nue+L0E98ufXaI0FJ3vW2T7FMDUNAw0vXL5n4flYX6ETjkRyErFtiNnXbe/gej4S/VTBHxlOoEDuKiHSdbxtxZ9ilBORxTC5vLbYnqxp+XP03nrIiLZuRAw97QMNCJfymeRfUFwHpedwqsqmrPvBZv5xs73Sf9CHEUmW4J9cHaq4rvLaYb1XRqkCLLg5gY7nY/wr0hfJnrSwKwCh6xtEieXPk/YZG7NrJCqcj2G0kPrg7B2BByeh2eLp/5ocBzSsyzApIg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fDy69tQqeQmirM8UdgckDtRnyaLn8eOm9TxH2BQksXs=; b=j4Xw4hXqo4cgs+O2UiAnVJACItZJYRzFkYxIpE+h9Aa5DMU58Ul/5p5MaT1Nm5WFiQha3z0kGJYtD17WZlU0C2HoHwhZ0JHNF8kJuXepmgosC+1o6XI44E3PCIRP13f1fZfnp3gzd/LBHdRMz6v/ROJnASvmRVo1QUya3QV1Q/U= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BY5PR10MB3908.namprd10.prod.outlook.com (2603:10b6:a03:1b0::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Thu, 25 Mar 2021 00:28:57 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3955.027; Thu, 25 Mar 2021 00:28:57 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Roman Gushchin , Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Hillf Danton , Andrew Morton , Mike Kravetz Subject: [PATCH 3/8] hugetlb: add per-hstate mutex to synchronize user adjustments Date: Wed, 24 Mar 2021 17:28:30 -0700 Message-Id: <20210325002835.216118-4-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210325002835.216118-1-mike.kravetz@oracle.com> References: <20210325002835.216118-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR15CA0042.namprd15.prod.outlook.com (2603:10b6:300:ad::28) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR15CA0042.namprd15.prod.outlook.com (2603:10b6:300:ad::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.24 via Frontend Transport; Thu, 25 Mar 2021 00:28:56 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 02fbb528-6d82-4e48-b0e7-08d8ef24faad X-MS-TrafficTypeDiagnostic: BY5PR10MB3908: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Sc3vEtUHW6QOC8qoD/TuAkyWoZ8nsq1icCGXG6crzlBVEwL818EcN7a8nLtah3wsmjkCvrBDi6xC3vOmqnDTNmeBYmPOw88BONV7kkYL2s1Hm15XGJsDELVCvY2N2D3naP3TNRO1p+KeAN6FPXC8NWxUbDSFA42Ik9D1XbnsrU42osj71ZC1TNaVmtxSJHpEu/Q0/WDxM2a3v3Pp0lL/DbNCvQwG00i/cidVUBLtLYlbfL7C2YVif2aMITv6H3QQFa6ItvIEIWIfWcM7Ykw6sn0Z/AsmPinBB7bTjuykS175Snq5BLEXOx63imTvueXbgC2b/rLH25kaCjQ2AkqEpEjVKf8Dpmc0I1gFKimnstSFLZx/Yd/utvDwJGY17TcpXxDLSMuPuH3YDNKVlCAc8eiT7bmjFQxu+dtlE7VOWAx0732FO7PYHVIHV2gpOyhqiFM3zBG+VlrLqRi1I+GdRroQFoe7+v9gao5bEuHxX9y6aaowps36IfdUJNQ0+1VwnJvf5Rb3tRl0tkfiesEGyyteu6H4vKt4gMKmbeRDQQka7CdaHFVKfVpVlKfaa66x81LldpxQBQsNaJP6dV6yPcbh0V6kax2fHEJ8897LNECj99vIFTYSewdreg778g3h4y4Piu9GUOfVsXrtJlpmDg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(376002)(136003)(39860400002)(346002)(366004)(396003)(86362001)(6486002)(38100700001)(7416002)(6666004)(107886003)(8676002)(4326008)(2616005)(2906002)(956004)(186003)(16526019)(316002)(1076003)(36756003)(66476007)(66946007)(8936002)(478600001)(44832011)(83380400001)(7696005)(5660300002)(66556008)(54906003)(52116002)(26005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: /Bbl3kn2t6mPWwzPDyFxLlMuQ8NK71w/OApSASbAbHUNAbIltOyL6zT4nbQIcpWFWU3Zgn/C2iEJUVaZKCmsSWAHZT8ltIzsTMKYf37exRG96OJt3sUXViZJHlWR+6Ia9QEkJ5DiGfw4Q5rWPZ4LGUGUqk3aythc7YQDoMHHtfk6AvM3mrlPKkj3O8/tA9LRuVXp08zIZvvh9qhj04F4MKQ0Yyt6L/WtJ8yRnRpC7hoUwuKIAXmxPzWpkdqR3hi9r2Hx3B1dHFhhAGJ8U+pDXVUToFxvjUyeix5qAY7GKkiaTe2Tif0oWHRBp+dkyQdkijcG/x+lE9xZLOpxgZmMte/+sdDdIedWm3p1XrYmWKDiTZYBDtxV4qM5XIDMYcbfiPzszTa0aW7MO0g+l8MK8AC2lW0/WDLErIsR0cD5VKGGSxjejCYjDBAzENcjcIZErZ28qb42sc1lPVrnubQc/Co5Rg8z/EafTRQTFVqwxAHaD/WULQK913XMpVEV+PNEaKOUevIPMmuRJPr9wwM/mNZXezbXMHO8lBp/y4n2X/oFdRpKVCZeF348S0t+UE2a2gykCQGCAdA25rB9RjxpZq8HjdEDSLUlF2qhuFkN7BPzqJwtFHnJXnYkhu+P4kTutsJiFoK16XNMXiQvo+YmSh7LxsmNYbAkx59Q2bqb4ufKwIHFOM3rvY1VCEjhXKY10gz4NzeKsIfAWN9db5yfFSBI8TCGXmFnUjFaevftR9SiZH4z2Kqf4zk/CkdyFQ86hp/q1RtMgKb+LjKpBzz1/2GtLSsn7Q0yCv+ZwKCzdaYcGzXDqlqrl0vhsd4kc5bFIcZujfyioj2MAOmUmk79byUUD8pKCBKw2LD6q+G4kjEVP5oKmpuPWHu/iqkjTZW6+HqSlEuQ3CE/+pwpQFSvh0Kx+7jM+kEJlgGt4H9PeSx8JjCbINqGzOLs90P64bFr0sILqkYsgdItxfbHfMslgdzevQB24qK0wULgNBYsPw3fCgr/ALzFjfwNzKPDxrTe/tDkTwfsXfdlidOa3kwy6qkBN87FBHzpNWYrT+FPlNzMEmsEu2YTLOEn6UO6+FlXVASxuEapFHdtRRSWgZIi+PzhRL79PpPlB8IntZpT6YGqp7I/jUMKH1BBs09YDFtvpUNuTORrmiHZTJAdE/XxEfz2d6vW2MMY0WJI/gvE+h6eWZpXUSGkEapvccA4MINxbZKhuXTZtImObuNge+lfAqiUmbs3qqycxXJyhO8zcL2R3xkv1wKCFpnIufB3IFWNAEgAXXObfmlUWwjZk+4FMgW+CedI26J1FpVCYe/dXm0Lx4CllLE1Bhp5jEilIMgI X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 02fbb528-6d82-4e48-b0e7-08d8ef24faad X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2021 00:28:57.7767 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: kC5BdZ9RbJQwwYPAGj3KS0GGD0qyq9XHtqFO1KofOUaacb2xRsknJM517ZToUSn+9VcK+Y6YZpA8fxLpQx9Quw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB3908 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9933 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxlogscore=999 malwarescore=0 phishscore=0 bulkscore=0 mlxscore=0 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103250000 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9933 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 malwarescore=0 mlxscore=0 priorityscore=1501 bulkscore=0 impostorscore=0 lowpriorityscore=0 phishscore=0 mlxlogscore=999 suspectscore=0 clxscore=1015 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103250000 X-Stat-Signature: 3x6w658jngzsfhwe6xgx5utusicfb5gq X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 17A6680192D8 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf27; identity=mailfrom; envelope-from=""; helo=aserp2120.oracle.com; client-ip=141.146.126.78 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616632150-552880 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The helper routine hstate_next_node_to_alloc accesses and modifies the hstate variable next_nid_to_alloc. The helper is used by the routines alloc_pool_huge_page and adjust_pool_surplus. adjust_pool_surplus is called with hugetlb_lock held. However, alloc_pool_huge_page can not be called with the hugetlb lock held as it will call the page allocator. Two instances of alloc_pool_huge_page could be run in parallel or alloc_pool_huge_page could run in parallel with adjust_pool_surplus which may result in the variable next_nid_to_alloc becoming invalid for the caller and pages being allocated on the wrong node. Both alloc_pool_huge_page and adjust_pool_surplus are only called from the routine set_max_huge_pages after boot. set_max_huge_pages is only called as the reusult of a user writing to the proc/sysfs nr_hugepages, or nr_hugepages_mempolicy file to adjust the number of hugetlb pages. It makes little sense to allow multiple adjustment to the number of hugetlb pages in parallel. Add a mutex to the hstate and use it to only allow one hugetlb page adjustment at a time. This will synchronize modifications to the next_nid_to_alloc variable. Signed-off-by: Mike Kravetz Acked-by: Michal Hocko Reviewed-by: Oscar Salvador Reviewed-by: Miaohe Lin --- include/linux/hugetlb.h | 1 + mm/hugetlb.c | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index a7f7d5f328dc..8817ec987d68 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -566,6 +566,7 @@ HPAGEFLAG(Freed, freed) #define HSTATE_NAME_LEN 32 /* Defines one hugetlb page size */ struct hstate { + struct mutex mutex; int next_nid_to_alloc; int next_nid_to_free; unsigned int order; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index f9ba63fc1747..404b0b1c5258 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2616,6 +2616,8 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, else return -ENOMEM; + /* mutex prevents concurrent adjustments for the same hstate */ + mutex_lock(&h->mutex); spin_lock(&hugetlb_lock); /* @@ -2648,6 +2650,7 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, if (hstate_is_gigantic(h) && !IS_ENABLED(CONFIG_CONTIG_ALLOC)) { if (count > persistent_huge_pages(h)) { spin_unlock(&hugetlb_lock); + mutex_unlock(&h->mutex); NODEMASK_FREE(node_alloc_noretry); return -EINVAL; } @@ -2722,6 +2725,7 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, out: h->max_huge_pages = persistent_huge_pages(h); spin_unlock(&hugetlb_lock); + mutex_unlock(&h->mutex); NODEMASK_FREE(node_alloc_noretry); @@ -3209,6 +3213,7 @@ void __init hugetlb_add_hstate(unsigned int order) BUG_ON(hugetlb_max_hstate >= HUGE_MAX_HSTATE); BUG_ON(order == 0); h = &hstates[hugetlb_max_hstate++]; + mutex_init(&h->mutex); h->order = order; h->mask = ~(huge_page_size(h) - 1); for (i = 0; i < MAX_NUMNODES; ++i) From patchwork Thu Mar 25 00:28:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12162589 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42C87C433C1 for ; Thu, 25 Mar 2021 00:29:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9B0BC61A13 for ; Thu, 25 Mar 2021 00:29:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9B0BC61A13 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 21A676B0071; Wed, 24 Mar 2021 20:29:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A5536B0072; Wed, 24 Mar 2021 20:29:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC37D6B0073; Wed, 24 Mar 2021 20:29:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0237.hostedemail.com [216.40.44.237]) by kanga.kvack.org (Postfix) with ESMTP id C4B246B0071 for ; Wed, 24 Mar 2021 20:29:20 -0400 (EDT) Received: from smtpin40.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 853E33ABE for ; Thu, 25 Mar 2021 00:29:20 +0000 (UTC) X-FDA: 77956512480.40.33D9858 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf01.hostedemail.com (Postfix) with ESMTP id 69E1F500152A for ; Thu, 25 Mar 2021 00:29:19 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12P0Ou8Q091759; Thu, 25 Mar 2021 00:29:10 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=CeG/KlUR9a0yxn1POE/snKgA7hl8ZIq7hVyqToL0PmA=; b=uDWVDlo1yEmJC5DSA/IvdRArU7EsYMrPM6c3E2WloEN3AfFZg0eb33JI31S6a3FJudQd KaErDhDr4thkseDmDe6Qe/OFoiU6Cx8IThV5YB0593Gt0HEL/heZpjcN7C3lIoo9B6qD Xq7i8QDGLMGAhZFN19t66bLJqJo0KhdaqCfYAiLzGxAvLhnjnfW3H69JtBlM4KSrJIh4 Th2ndddgwmTQhTT29prUm5NvaGlD0dLnGvSyk9SIPOcS7ViuTZhLXXpa9HojR8gV4jAQ McnzVgKJQeulZXMwzHfMXKpXPNYr2fF8ig1xGBQ5dFqbI1wYUU1FgHjBLhePopmEmuFN kA== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by aserp2130.oracle.com with ESMTP id 37d6jbmrs7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 00:29:09 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12P0QTDQ040331; Thu, 25 Mar 2021 00:29:09 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2043.outbound.protection.outlook.com [104.47.66.43]) by aserp3020.oracle.com with ESMTP id 37dty18p2t-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 00:29:09 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=N+okZx3IVQcBDINOF6VhEzwCnPLSTT5iRLMBExQnvlFmn/Zy3PEpOiLrJegM60FzHnGjPiMQvAUpfl91iAz4g6RieWkqN0qe7OP7qCC0FKNPkt7pP61hLRORWz1qeTTOgFjVv+67pX0Xqr8PXXOBpamLKmu1AgWFZd0gRkqp6p1XfPBD6m8dAcw7zBAz0PBjQ7USAxzER5tPb6ZEs7zQE00S3WpIOjHWik6qsvNfegBIpqurTCjjBnStCvFsDTJaKEu9o6yEg/mmLt5brXLoXnyOmA5uAbGCPWIWtRU8oY80F088YjJDzIrSim1CBpyKNk9KfiXcGdn61R/HMlOYvQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CeG/KlUR9a0yxn1POE/snKgA7hl8ZIq7hVyqToL0PmA=; b=WxNAIo8pJUabRG69cuf41cNfDrFogHlWR6GPcli+aM+fGvCULUqHV9B+Ukke9db5aqflLVwjkoqTyhew8swZ2Qvfv3jc0xyJWDYtgVWJrr6/S5NwynBtTbIvfgHSqBaZB6hWL2q5KSoDduoe6L4Z2YtClFORM8oFs03drW/SlCqjAvCY0wZPHwaRQ5L/JN3JrEUQNzTwlgahmTNaU15ogPBrLstNzGsomB3lL9n5VJSCZUfVM+A85cmVIDHOLF1TmZKfAfbv4s+kG6J6FNeW9U+Pb19X2D2JVFeZTemPgzj0dhN/MfMvTL1W+e57R26rvqcwMlbJhc1fHHprHCXPrA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CeG/KlUR9a0yxn1POE/snKgA7hl8ZIq7hVyqToL0PmA=; b=rYSMe/FhMFIKy7RdLPsrtoPLY1ZZVE0PCBIN4RGdwE0a4XgvHUVX0hGo+K082sGuK3b7vFHt9w9NIW32Wm87Ph+zn9DXqhIFrAzdeglrOylbZXesQbeCbeinpbJYHKpkky/SRFh0ORdTXISvhxFKPZ31tUCt/xntNgltA2ZK9Pw= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BY5PR10MB3875.namprd10.prod.outlook.com (2603:10b6:a03:1f9::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.25; Thu, 25 Mar 2021 00:29:05 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3955.027; Thu, 25 Mar 2021 00:28:59 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Roman Gushchin , Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Hillf Danton , Andrew Morton , Mike Kravetz Subject: [PATCH 4/8] hugetlb: create remove_hugetlb_page() to separate functionality Date: Wed, 24 Mar 2021 17:28:31 -0700 Message-Id: <20210325002835.216118-5-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210325002835.216118-1-mike.kravetz@oracle.com> References: <20210325002835.216118-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR15CA0042.namprd15.prod.outlook.com (2603:10b6:300:ad::28) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR15CA0042.namprd15.prod.outlook.com (2603:10b6:300:ad::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.24 via Frontend Transport; Thu, 25 Mar 2021 00:28:58 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2071e5e3-0d6e-4ed3-f265-08d8ef24fbc2 X-MS-TrafficTypeDiagnostic: BY5PR10MB3875: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7219; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8EitNTb1PGBVMt1FNx1Q0IQj9OQU9houARzDjVDeVCxpwEHbETanWn56f/Av+QiL58mXfj+BuxDGl2U1jNtSiZes/XxWFmtPChDDNx81x1NoJUYI2BGuVcog8EE7C1WklkG7erguU3FpQ6mB6wtNmkhSta8cL5KSSZbz088dfGashYq9mOvU+ar4wws7G6vl3QlakBOwTzU9sttrsWiSI+UcaC/u3JPocvw6h+wCzb4N7ou2M6uzGd5Q27skPl2Qk0k3QW1NbmCQ9U8DNv4fw0htADtTcyVUIzUhRlJRztU0efnUsc9TM49g2lYKQr1AMFmEfMOctoOmb3KbIfOSz4XpDFj1+/Zf04cNe0OoQLWrT+4acyBkyzo//4qTYvnisYnAulBH9Wn3LGzNeiEbhdCu/OPWnTAAZ5CAdCAISV995OkULmzcQmY96wnEg9aLpunZvCtvVIaYv/lmavzrV4p1nS7JbGLhb0KzazyhHcQ311dej2soKKt0jKwZ2wrEEybsTdzpbRR//+xUqIrsJhPk1ZKHdgyqrpj7jI2Z2uS18k6cNKeI8kvU1mJJ1FiENgQzFTxmTnmV/d9Fz2ZlXZCLjq0o/LKIJRoN6XLIJgzGYHsbd/nQYXMeWEb1aAeqhwfbghrMNQDytZ7aXXxroA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(396003)(39860400002)(376002)(366004)(136003)(83380400001)(6486002)(52116002)(186003)(4326008)(66946007)(7696005)(2906002)(26005)(16526019)(66556008)(6666004)(5660300002)(66476007)(8936002)(44832011)(36756003)(316002)(478600001)(54906003)(2616005)(8676002)(1076003)(86362001)(38100700001)(956004)(7416002)(107886003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: kz1vvBB4MbZLoi7ZP9yFwB7b2DXjFcNYscMJiufv/5NPS/jCGhtEZMhuCbNoPlvydZTqUWCnngXTWp5kStyE2pPEbpm0U7bDg19nmjRZxKfaMO2WukZC9i0Hx9KRG11tsZtb6X9DZbqeWXQgn75EsmCGlB+449SKPlPcVPOdzfd6pN1RPTrT824oatDdU4WCv0faYKx9fM1A9dIYifXKZUn9bxUDt3+B9tUyPNP/gUnGfxJbzK9B5Prnbi7m/qcF2UXKmAi5Z1lmWcANbHaVGwhJsTkCYVqXlb/zm10E+hl7F/fo6SZm61jmH4fmeVU1JVRfOTB5n/2D1C5g8V4Al9FIi+DSBPWHiQ6H8MZNfOyGqaIVsX792U+HM+Tj7JOGHI6Vn8geh6PpMfBODDsEallyoJnPQ44rCfuegJ1W7M49IRPVPpWzRE4Up1vaDMcxFZxYcvixZebHEKJy1SQApEkMOsd3KeZ8gufBQOnPLuiT32qj5vGY5iSA5hGTjgp8CY+1wLL4qh4bYPNEynJRpdnDOV6bxwJe9VRFTGd4CmFv540R5pQo7ZIV5RJ///pfvxGKZhtBiTqojLOcMkfvJWBlucRtbGiMk/d2f34BxmlV8tNUAV3HbxV4aIBL+0G3zfF0d+dqC5GElcgLzUcji1aOg5Q+Om/TlI8kVnOd76t1/Zo+ebGNAGEn5X40snttJMhbPo0OAbrFlHjonSwJfTigEm1E6oY4vJYV5EwXKQ4YbhcVn/W/6MX+haPV1urDdyXkpcEJbRa/LwsLXrYEP+2uIqrfWrjCR/GJEXsWFXoXn6N2lfH8KJsrSclKpaxYZSBQuKy4LljjLSAD4aSZyzDzVezAkXSNUmZUa9V+WEgZWnIUZMujPo2CC0xTOyAEEJ9X/GGOi8uTPwEWw0PQb9C+I9qDGwQ2B2vkR/EdMpYylGAcM9KP2rqN6+GM19tDmNIcZeMWe6ch/zPiA7u1+XOmH4Q2+INawQGCAjnPw4itASao4Goerp33ScsqIXjMIEdcbmypAg9xgaPNPSHUraGoKRF351A1kOiScJydIKJSU/AHGITVGYH2GEppcvlCiAkyZXJVn0dHnk1DqzWDUbEKqObGVP2Mks9w+7fKILV6Y2whZuJwkQnvyAiJqFPgNFk6/kO62fU5hjB3qEdIrrGCq0xm+GNKHVrqmIfFkhz8l/iv6qKcDGTLfx7FOQ1tWtLkHsT8e7eYhWBGV6T+jP9c4SUPY88+emErUN7zcWYh9Dwj/X/Z6h+ZQZtadG+8CfAL3STWReT1MBg9YH8NYe7zLuVqNzRHbUujVYYISCje2pPKl/AtC23W2+X8vySA X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2071e5e3-0d6e-4ed3-f265-08d8ef24fbc2 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2021 00:28:59.6684 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: b+ws5csyBI4C9xKRlOr+evtMNli+seJvRCdpSOtj/iDocNs7rSRy7Mpk5OnQb6NjB7xSqNQJ5WNuCJ+LHm0zzA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB3875 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9933 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxlogscore=999 malwarescore=0 phishscore=0 bulkscore=0 mlxscore=0 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103250000 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9933 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 phishscore=0 lowpriorityscore=0 suspectscore=0 clxscore=1015 priorityscore=1501 spamscore=0 adultscore=0 impostorscore=0 mlxlogscore=999 mlxscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103250000 X-Stat-Signature: 1tdy744zzb485y7i5mijsgfgcrdtwjna X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 69E1F500152A Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf01; identity=mailfrom; envelope-from=""; helo=aserp2130.oracle.com; client-ip=141.146.126.79 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616632159-666645 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The new remove_hugetlb_page() routine is designed to remove a hugetlb page from hugetlbfs processing. It will remove the page from the active or free list, update global counters and set the compound page destructor to NULL so that PageHuge() will return false for the 'page'. After this call, the 'page' can be treated as a normal compound page or a collection of base size pages. remove_hugetlb_page is to be called with the hugetlb_lock held. Creating this routine and separating functionality is in preparation for restructuring code to reduce lock hold times. Signed-off-by: Mike Kravetz Acked-by: Michal Hocko Reviewed-by: Miaohe Lin Reviewed-by: Muchun Song --- mm/hugetlb.c | 70 +++++++++++++++++++++++++++++++++------------------- 1 file changed, 45 insertions(+), 25 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 404b0b1c5258..3938ec086b5c 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1327,6 +1327,46 @@ static inline void destroy_compound_gigantic_page(struct page *page, unsigned int order) { } #endif +/* + * Remove hugetlb page from lists, and update dtor so that page appears + * as just a compound page. A reference is held on the page. + * NOTE: hugetlb specific page flags stored in page->private are not + * automatically cleared. These flags may be used in routines + * which operate on the resulting compound page. + * + * Must be called with hugetlb lock held. + */ +static void remove_hugetlb_page(struct hstate *h, struct page *page, + bool adjust_surplus) +{ + int nid = page_to_nid(page); + + if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported()) + return; + + list_del(&page->lru); + + if (HPageFreed(page)) { + h->free_huge_pages--; + h->free_huge_pages_node[nid]--; + ClearHPageFreed(page); + } + if (adjust_surplus) { + h->surplus_huge_pages--; + h->surplus_huge_pages_node[nid]--; + } + + VM_BUG_ON_PAGE(hugetlb_cgroup_from_page(page), page); + VM_BUG_ON_PAGE(hugetlb_cgroup_from_page_rsvd(page), page); + + ClearHPageTemporary(page); + set_page_refcounted(page); + set_compound_page_dtor(page, NULL_COMPOUND_DTOR); + + h->nr_huge_pages--; + h->nr_huge_pages_node[nid]--; +} + static void update_and_free_page(struct hstate *h, struct page *page) { int i; @@ -1335,8 +1375,6 @@ static void update_and_free_page(struct hstate *h, struct page *page) if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported()) return; - h->nr_huge_pages--; - h->nr_huge_pages_node[page_to_nid(page)]--; for (i = 0; i < pages_per_huge_page(h); i++, subpage = mem_map_next(subpage, page, i)) { subpage->flags &= ~(1 << PG_locked | 1 << PG_error | @@ -1344,10 +1382,6 @@ static void update_and_free_page(struct hstate *h, struct page *page) 1 << PG_active | 1 << PG_private | 1 << PG_writeback); } - VM_BUG_ON_PAGE(hugetlb_cgroup_from_page(page), page); - VM_BUG_ON_PAGE(hugetlb_cgroup_from_page_rsvd(page), page); - set_compound_page_dtor(page, NULL_COMPOUND_DTOR); - set_page_refcounted(page); if (hstate_is_gigantic(h)) { destroy_compound_gigantic_page(page, huge_page_order(h)); free_gigantic_page(page, huge_page_order(h)); @@ -1415,15 +1449,12 @@ static void __free_huge_page(struct page *page) h->resv_huge_pages++; if (HPageTemporary(page)) { - list_del(&page->lru); - ClearHPageTemporary(page); + remove_hugetlb_page(h, page, false); update_and_free_page(h, page); } else if (h->surplus_huge_pages_node[nid]) { /* remove the page from active list */ - list_del(&page->lru); + remove_hugetlb_page(h, page, true); update_and_free_page(h, page); - h->surplus_huge_pages--; - h->surplus_huge_pages_node[nid]--; } else { arch_clear_hugepage_flags(page); enqueue_huge_page(h, page); @@ -1708,13 +1739,7 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, struct page *page = list_entry(h->hugepage_freelists[node].next, struct page, lru); - list_del(&page->lru); - h->free_huge_pages--; - h->free_huge_pages_node[node]--; - if (acct_surplus) { - h->surplus_huge_pages--; - h->surplus_huge_pages_node[node]--; - } + remove_hugetlb_page(h, page, acct_surplus); update_and_free_page(h, page); ret = 1; break; @@ -1752,7 +1777,6 @@ int dissolve_free_huge_page(struct page *page) if (!page_count(page)) { struct page *head = compound_head(page); struct hstate *h = page_hstate(head); - int nid = page_to_nid(head); if (h->free_huge_pages - h->resv_huge_pages == 0) goto out; @@ -1783,9 +1807,7 @@ int dissolve_free_huge_page(struct page *page) SetPageHWPoison(page); ClearPageHWPoison(head); } - list_del(&head->lru); - h->free_huge_pages--; - h->free_huge_pages_node[nid]--; + remove_hugetlb_page(h, page, false); h->max_huge_pages--; update_and_free_page(h, head); rc = 0; @@ -2553,10 +2575,8 @@ static void try_to_free_low(struct hstate *h, unsigned long count, return; if (PageHighMem(page)) continue; - list_del(&page->lru); + remove_hugetlb_page(h, page, false); update_and_free_page(h, page); - h->free_huge_pages--; - h->free_huge_pages_node[page_to_nid(page)]--; } } } From patchwork Thu Mar 25 00:28:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12162593 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE88BC433C1 for ; Thu, 25 Mar 2021 00:29:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 67CF861A12 for ; Thu, 25 Mar 2021 00:29:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 67CF861A12 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AA6886B0073; Wed, 24 Mar 2021 20:29:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A08EE6B0078; Wed, 24 Mar 2021 20:29:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5EA4F6B0074; Wed, 24 Mar 2021 20:29:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0147.hostedemail.com [216.40.44.147]) by kanga.kvack.org (Postfix) with ESMTP id 525FD6B0073 for ; Wed, 24 Mar 2021 20:29:22 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 09B68181AEF15 for ; Thu, 25 Mar 2021 00:29:22 +0000 (UTC) X-FDA: 77956512564.15.A106B72 Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) by imf30.hostedemail.com (Postfix) with ESMTP id 0C093E0011D9 for ; Thu, 25 Mar 2021 00:29:18 +0000 (UTC) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12P0O4e0143453; Thu, 25 Mar 2021 00:29:09 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=hPG9oS1yYoYLCZVPq76cD+6Ob4xYB8uw+7PBpGF3Plo=; b=IJ9xfV0FbH6JdbwDevke5vrAKD7HlzcfcgMbP9TvJjHWZ+EnyBqxKmEyDSl7Z8G7HtyK 0iCrOEKYhn4bqzk6RqNIwZp6OHUPHgVUP1RCnObgSRg7yML4Pgw/nPtEr+AkiYN2nF3e a5+l0EQEIKtyOCV+iYj8ugbvcsirc/NnqYkJhh4wnlaPmlGkDJ1JcXbuNgCRuyCG5K/L 7s4TfLjuZywnUp1AhEJUmHUQepxKadhXOf7fBzy14vAEYx8dfLkvrSsu16A8ktZKd9co wonlfQrumaqfYLlndoox2OpjkWHVYG70HqnoZfDSxWi9cWVA+NLjWAnkTDre6Q6mAXDz fQ== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by userp2120.oracle.com with ESMTP id 37d9pn4k75-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 00:29:09 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12P0QTDP040331; Thu, 25 Mar 2021 00:29:08 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2043.outbound.protection.outlook.com [104.47.66.43]) by aserp3020.oracle.com with ESMTP id 37dty18p2t-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 00:29:08 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Wqjb7gLp1EX4k0feSmoJa9MBLd7DMye2Jfl8eaiucyBUZ7TJXm3E0RdwRA0Gzu2NE2SW/CE0TwQxgH+nqSFF6th2kNpK+iTze3RD9FRLX73IsNzwdZ3omhwUxhV8PWboCsT45DqVciRtcfyldaP5EWdSVQUWqcYeGFKTZVVx+Xoph/e2GD7gpqv1cGTGpztI9OI0rIUMELqnYw4pj4teCaFJHLwEcq5ggF0c9krEWOQYF4/SCDLLGjj48czriV/CC/VUtZcgQqlgwDjius+CRoUNWC1XPXsEzpyL7Da4MGRLafaqk9JcialnvziFnVuZ19o6daP+eNMcbhnERku8XQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hPG9oS1yYoYLCZVPq76cD+6Ob4xYB8uw+7PBpGF3Plo=; b=a7B9Ck/KrSGgs6gyR25ON+TWPJdfEpFURoBkUoyTWR+BeNbzHrAzdoR9vcXO++swPatNmBsAzkdaPVFeCwCSAPVrrFeA+4R+QmISkV9OKjSpag7w0Jdczs68JE2+LzD/mm2ow6zIY6XLyFk6vvWH5kCsiFU9kmDQns4l07swFOrvsrhNgZ7FuEOsS+W5E2HUj/6w8knwHGfOMpn2cEvnoEDjC7M5epYIgL85RvKwqDXP9HTSVExlg+Tt+qn3uN/to5QP7E+VJKFrV19c6XKTSd0BUYK9v6Mvxt7LuHpjL1iEUUqelwP3OhL/Ifl3UVCu1pCptLD2LwUebR4ec4+8cg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hPG9oS1yYoYLCZVPq76cD+6Ob4xYB8uw+7PBpGF3Plo=; b=H0mqz+pmFsSjr/ClhUfHzHDUj1CagIbF4QKD0e7qGjlYYu3KnCvo/tu+0ytwpletIX+lkgnjHanXLJXZDyoxjp6eO3DKo6SY/BkvtYv3i8OxnOki4wymH5O9xd0NiQWG5X/Ie8rPRpvcqj/a9JP4ItygcuoKw+cTOac4XNK7QMI= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BY5PR10MB3875.namprd10.prod.outlook.com (2603:10b6:a03:1f9::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.25; Thu, 25 Mar 2021 00:29:06 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3955.027; Thu, 25 Mar 2021 00:29:06 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Roman Gushchin , Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Hillf Danton , Andrew Morton , Mike Kravetz Subject: [PATCH 5/8] hugetlb: call update_and_free_page without hugetlb_lock Date: Wed, 24 Mar 2021 17:28:32 -0700 Message-Id: <20210325002835.216118-6-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210325002835.216118-1-mike.kravetz@oracle.com> References: <20210325002835.216118-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR15CA0042.namprd15.prod.outlook.com (2603:10b6:300:ad::28) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR15CA0042.namprd15.prod.outlook.com (2603:10b6:300:ad::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.24 via Frontend Transport; Thu, 25 Mar 2021 00:29:00 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 5c70f952-fd41-4b62-58aa-08d8ef24fcf7 X-MS-TrafficTypeDiagnostic: BY5PR10MB3875: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3826; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: KCKbYXBIPx9z7RZ8nK00WJQMFBbGN9GKGLj5hf19JIdlos/I07vW+jzF2zfB5V93Gwh+cYutcTRdt41KcPlQ30ORhgNUEV8aqiQeflPyBzUOKHbrzcp6WJ8d/6KlaaPGsUkbRS/NKD9ycFa1qf6xzbkWg9zNK4wtnZ0V8CuuxBl8t5pQcrBlM6hXVhuAEgswJyJL+dh4cqACSk3717nnui1BwGr5tFhkJHd8EjaYVzZp9g7oiT8MsXbi5mu1xcj2w/SdSP4grnuKJ3yksJq+RpP7r7J7aWi4s9yiUGsAcLHArI5GkuiN9xzySKfm8YD1n0YIgknN9fpM6gQ+CQ01/66dkRJbqLs4PZejJm+92qs5s1dJ3y9/1EJJavl8deQkvop0xgLTGmuy9VeBi33s0ilKu8NESZstqWipoTd1rHPScKBk1u506fVbynZtmrMmk2+cNIfdq91qVkplrCEAaP9dbPQtd1MzVXV3DhhZHkQK0K9QustnVPvq2876S+E+FLmFFzLk7PKkOSjiBaTLyi9ue43OdO06S3S+e6ZVyZZyEFsKzAi+rgk8v3dINJvKsp9f3k6ae7v1KZjLvnVU+n/O5zFCiVXo7+/3Js9bSLIsAj9Pa9hj/JL3d6Jl/jMybjUHdORXQHmzmRhVO1S88A== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(396003)(39860400002)(376002)(366004)(136003)(83380400001)(6486002)(52116002)(186003)(4326008)(66946007)(7696005)(2906002)(26005)(16526019)(66556008)(6666004)(5660300002)(66476007)(8936002)(44832011)(36756003)(316002)(478600001)(54906003)(2616005)(8676002)(1076003)(15650500001)(86362001)(38100700001)(956004)(7416002)(107886003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: kbmSrKtqTCb2u+JX1AJJoP0PtzDlSJazY5VoW9Zqzivx6DYhPWqQAOwk9uYjQiY/HarZChDbDwNB1f/EKqN/BXgi3ScelvihUnPuL3BULbf4B1UeiPD11egCVaHljPVKbJtBXqUxCJmXE8KW0xBa8N8dUyCLnHXM7OMuiizVbdTpoo9Yzieblkzm6vU6VpUuBoG42/OF8//JemreUeeLFtBB2ryNgyey8D0dk9Cp8JguFMF5tejcfIE0oUkaSJj3bZ4w+4zxx/iR+M+mm3B3eVkbhDkDhXuHd6jFR4McB+ywSa9x4kav+TCEEuxDoKSuUxxEletB2ffStUrCfFaV+lzCxXeAIM57DIKpX2FBG+avo+S3CkpFt361ts4WGFYXp53uRatk+tdrvriCH9Ik/zaH3XWbGuZbOnIQLxxi3iOEhGeCnjBtchbrmO/4D6nflPESr8DbmPgyBKYhPBTaUxfpnypOCC0dRsd3wYUT063QprKpK9oudG1z22pKH3vXeVOSKRluadYQfdTnWYW0HVJkeqMQbX3bgUI4yNw3aRV8uBzOw/3Lj4Iy5oyqovDUEet4FYzcnqFbQjPTZbffUPPxBQbICKLkCipVYrFoD6VEui/M2qZQyE2DcgKsq/VESYGFcNoAyKkXzVkphBSZJEl/++Pgp/dajYQSG37FxEwrnK7WriMI7go6jABlqf2JMoDfJ27uucTFqBRaRcwDnElHmOboVVPkpJD/r5NOmw/k+mLqOVmrJgD/+Rh4FDcGQbQQtL0BO3R7emL68rf0O3ogMTPSI72uakc2smMGac13PgtRE4HIYdoXQhSe5Zq2xj7AeBnN7Bfr5thJbDhbuomr9lxd1DBACZ7U0CAn3yTDkT8MWl7rMgEPPg8urbkA0m7RWvh2Q3UxLjvKys09N3WWihAoTfhlKf5XvIKx6jfZEvV3WVFDOeDoayFTerbJ2gbwbWOWYUJkWID2uBn1Cr82T/mK3tOZF/andK1QZp0P5hDs60M5NmKdqcNzg1GmECMSwEL9Qs+kQ2JqbHLmHxQQ2q2BNndo4Qhbt0UNCawjabtSQkegbnZpk0qtAM2gVUpVYcnOzhNuK9yFGVBmIXwWna/+FAq2AgXL5wPyw7/dNXuJQRjD/0Eg5TvNm17XPuw8XDg3/8GOUUcVZnT2qfCEkI4Zivg9I2e38nf2LpIjaXXgR5aBn78Ys3PgoSfq1Nt7AIgrUwyprdq4o+nTlgd3v8Pk5J4KOD6zJaT64EmBD+GRevh8iDUiBezchfWi6bTXUlkgeACyfYxRDGpQxUfV9dTv8r5g38pYMWVqk4zB1p9ip8uBx9rZnkBahj+M X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5c70f952-fd41-4b62-58aa-08d8ef24fcf7 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2021 00:29:01.7055 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: nJM2qJ1CbnxQ3VIh1+2A6k42ViMKNno2gDT7HV0WfgwExGsVP6KLZaEF5s4RFuT7WzFCRQJxKO1lbe1gC/HOKA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB3875 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9933 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxlogscore=999 malwarescore=0 phishscore=0 bulkscore=0 mlxscore=0 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103250000 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9933 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 lowpriorityscore=0 phishscore=0 mlxlogscore=999 priorityscore=1501 impostorscore=0 bulkscore=0 spamscore=0 adultscore=0 clxscore=1015 malwarescore=0 mlxscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103250000 X-Stat-Signature: or5bcq5iu1qg5bd4853z4dw65oh1q6wa X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 0C093E0011D9 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf30; identity=mailfrom; envelope-from=""; helo=userp2120.oracle.com; client-ip=156.151.31.85 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616632158-934591 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With the introduction of remove_hugetlb_page(), there is no need for update_and_free_page to hold the hugetlb lock. Change all callers to drop the lock before calling. With additional code modifications, this will allow loops which decrease the huge page pool to drop the hugetlb_lock with each page to reduce long hold times. The ugly unlock/lock cycle in free_pool_huge_page will be removed in a subsequent patch which restructures free_pool_huge_page. Signed-off-by: Mike Kravetz Acked-by: Michal Hocko Reviewed-by: Muchun Song --- mm/hugetlb.c | 34 +++++++++++++++++++++++++++++----- 1 file changed, 29 insertions(+), 5 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 3938ec086b5c..fae7f034d1eb 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1450,16 +1450,18 @@ static void __free_huge_page(struct page *page) if (HPageTemporary(page)) { remove_hugetlb_page(h, page, false); + spin_unlock(&hugetlb_lock); update_and_free_page(h, page); } else if (h->surplus_huge_pages_node[nid]) { /* remove the page from active list */ remove_hugetlb_page(h, page, true); + spin_unlock(&hugetlb_lock); update_and_free_page(h, page); } else { arch_clear_hugepage_flags(page); enqueue_huge_page(h, page); + spin_unlock(&hugetlb_lock); } - spin_unlock(&hugetlb_lock); } /* @@ -1740,7 +1742,13 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, list_entry(h->hugepage_freelists[node].next, struct page, lru); remove_hugetlb_page(h, page, acct_surplus); + /* + * unlock/lock around update_and_free_page is temporary + * and will be removed with subsequent patch. + */ + spin_unlock(&hugetlb_lock); update_and_free_page(h, page); + spin_lock(&hugetlb_lock); ret = 1; break; } @@ -1809,8 +1817,9 @@ int dissolve_free_huge_page(struct page *page) } remove_hugetlb_page(h, page, false); h->max_huge_pages--; + spin_unlock(&hugetlb_lock); update_and_free_page(h, head); - rc = 0; + return 0; } out: spin_unlock(&hugetlb_lock); @@ -2563,22 +2572,37 @@ static void try_to_free_low(struct hstate *h, unsigned long count, nodemask_t *nodes_allowed) { int i; + struct list_head page_list; + struct page *page, *next; if (hstate_is_gigantic(h)) return; + /* + * Collect pages to be freed on a list, and free after dropping lock + */ + INIT_LIST_HEAD(&page_list); for_each_node_mask(i, *nodes_allowed) { - struct page *page, *next; struct list_head *freel = &h->hugepage_freelists[i]; list_for_each_entry_safe(page, next, freel, lru) { if (count >= h->nr_huge_pages) - return; + goto out; if (PageHighMem(page)) continue; remove_hugetlb_page(h, page, false); - update_and_free_page(h, page); + INIT_LIST_HEAD(&page->lru); + list_add(&page->lru, &page_list); } } + +out: + spin_unlock(&hugetlb_lock); + list_for_each_entry_safe(page, next, &page_list, lru) { + list_del(&page->lru); + update_and_free_page(h, page); + cond_resched(); + } + spin_lock(&hugetlb_lock); } #else static inline void try_to_free_low(struct hstate *h, unsigned long count, From patchwork Thu Mar 25 00:28:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12162931 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7BDFC433E0 for ; Thu, 25 Mar 2021 04:53:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 572F461A1E for ; Thu, 25 Mar 2021 04:53:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 572F461A1E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D268D6B0036; Thu, 25 Mar 2021 00:53:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CB18A6B006C; Thu, 25 Mar 2021 00:53:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A8F4B6B006E; Thu, 25 Mar 2021 00:53:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0142.hostedemail.com [216.40.44.142]) by kanga.kvack.org (Postfix) with ESMTP id 821F96B0036 for ; Thu, 25 Mar 2021 00:53:09 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 3D0CEA2A2 for ; Thu, 25 Mar 2021 04:53:09 +0000 (UTC) X-FDA: 77957177298.02.1B69045 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf03.hostedemail.com (Postfix) with ESMTP id 6F4CDC0007C9 for ; Thu, 25 Mar 2021 04:53:07 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12P0OOMj091690; Thu, 25 Mar 2021 00:29:10 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=xEcmDoGONJyuiiursH+GNkDrIZRdKSd50fOaTwQBeaQ=; b=nL+EHjvowspZ8/2u9eMD0FIr3HTyAD3gWq3tCUQRWpr3AXeWnirYrkDj4L9L8YA91xYU BFI5SfN7zCL8h80HtLZbeMMfMkXjvIFJyflHR4OLN7KIjntj3z2sa6gzSBAMhBNHSuRQ Gv5gySb0po84o8ICIGNr7/rCUTVe1IV5PYCRelScUzzH37mcDAC+von7O58eBFgRpSns oIMIhVSRUNK2egSr0Xlg0sk7DAL3j1+ozug+34NoxXVdzgC40F5EbchOE4KmGR1bevV6 8t4IZM4Q7Jfni038lDCPsqXoped1qZ/BjgIDOnk5/1RjgGUVSgHlZ+NUusz/77zO+JDD bQ== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by aserp2130.oracle.com with ESMTP id 37d6jbmrs6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 00:29:09 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12P0Ovoo009237; Thu, 25 Mar 2021 00:29:09 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2176.outbound.protection.outlook.com [104.47.57.176]) by userp3030.oracle.com with ESMTP id 37du00hg3t-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 00:29:08 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HRQgxtxzAY93oa5gOg9AIhgQhDGBaCRDWNw4W/LgitaUQWvs/HKvI8dmI+fjFFZ85qyWLoCIhWHsQUx50Dn5ZzPPZbdNmVNz76fD9g9DYrmz0NsP1Do7aTD69tAtWceJQ22HndLmIcyGyFsJgKpBDmkKLvlIByIS7HOGgVo8tN6oaZDjmY6qGspzeUvtP72O3yrMz0VAsh8UnnExiTpNhWFx/RM6XFZyFcxgv/7iSRbY9yw/vpcO93BP/GE+En5/Vle0m69EYkLOu3mJV6psJ5C+hD5tXL6uY08gNyhvntw0/75uJ9TdjD1ZhvQVgu8YJMT2ktYTtPNv9ST/LWkpww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xEcmDoGONJyuiiursH+GNkDrIZRdKSd50fOaTwQBeaQ=; b=PLA2uLlFhxh9tTN3Jp1bTUIR7mGApsv77+UIQop9AEgYv01eyeqa4eQoYGZbkRR8XTtywGTHsOg2/OfzR9NtL9ksT3/MlykEQXf/nVs1JUd1ri4ze8qE18NyUjO8IdCqOOh/d/X3XLF1O223vzFp/jhTkWjQGdAWm9dg9z+qhGMSTR7Esi9ArGAEpikL/U7j3H0j2MZc6ThPl0iOdRFfgEVWxYzd3QKlRPt4wOyk/+e3S67GN+JNXW3VC7Y5Jmzugm5ywzMb0GtRJ9H2lgUY3N+SB8tGiEDNygenn+0hEBXwXT5VID1UrW5EGnXVM0AHxjo+FLszVhPjhxiU5c5oTA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xEcmDoGONJyuiiursH+GNkDrIZRdKSd50fOaTwQBeaQ=; b=KfLQogzQYqnwwweeRPwGW3JrnpZgckbv7KDAuECS7UWueG/CrKui3lhjC+s4lYBNBnETxF15PGN5rfMEJgrvFVwEHzQfUoaMurwFtxW7FgEm7auzCiLmpEU9Gx4CucyxPS3XaE+oQsQpUhkkUrs0yIfRGcNF4aDER/tsDd1jybE= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by SJ0PR10MB4592.namprd10.prod.outlook.com (2603:10b6:a03:2d8::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.25; Thu, 25 Mar 2021 00:29:06 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3955.027; Thu, 25 Mar 2021 00:29:06 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Roman Gushchin , Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Hillf Danton , Andrew Morton , Mike Kravetz Subject: [PATCH 6/8] hugetlb: change free_pool_huge_page to remove_pool_huge_page Date: Wed, 24 Mar 2021 17:28:33 -0700 Message-Id: <20210325002835.216118-7-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210325002835.216118-1-mike.kravetz@oracle.com> References: <20210325002835.216118-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR15CA0042.namprd15.prod.outlook.com (2603:10b6:300:ad::28) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR15CA0042.namprd15.prod.outlook.com (2603:10b6:300:ad::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.24 via Frontend Transport; Thu, 25 Mar 2021 00:29:02 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: cc42d7b9-8187-473f-15ba-08d8ef24fe17 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4592: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 7cnD56LAFx44kjTVKSTjCnKnp+DBZ4Q/cT2OXTh+7CKwQW18q+kcrTVTqmPgwm1lHmX97gf73QZuc5+NoDNVnNqwGtJzJ2h9EL52bJYIdJsVwnDMc1QqpKK29w3GUBMdJm//PEVS3krsABRCDm8vJHMqDlVXR8cvWTGa4hzev3UC2mnshM6AT5kdluoZMsIzQtdJtMroZyQbtEuHtHSk4JO0kyO71NnA7GjvGbQae1/HiaEhatTTAs2h9ThNXvzjqPt566+tCPDGld7eU32F+nhja3hjivD2d4A8Tw3AVdva7qWNMi2wAlZ7N6ZFGNkxjiqw260nqtZVxRFFY4Fd646F3p2N+nvaio2gyLyKJElCecI2c1P+OYiRgWlslUPp2DqqW0gNJviM3TXwNW7LU3viggL0x0s3BvISbNNWgL0Vmt1VgOmOdHBnAEEalnGBkBVXC3HKgso51T3dwQHpbhvHUASDcvbo7gllYEPQFQJUIsP8cHCvq+b0QryYT+45+PJUpmitwvSEyhs04JO6pTTWMyuBJe8kfgiOZs5S4gF7o0PkwZM9ncd+TNgvn5UNa8M9WvjwaI1vWlAf4NwtqjCObd9Qw1kakeZyIVbkifkb25Y8WprwDZQnr1g26yiuYTeVztBiWiGMr5uIm4XIKiT71tThqkKTjWFGSy2fgaM= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(396003)(136003)(366004)(346002)(39860400002)(376002)(956004)(38100700001)(7696005)(52116002)(5660300002)(6486002)(2616005)(1076003)(316002)(478600001)(44832011)(2906002)(83380400001)(86362001)(8676002)(6666004)(4326008)(107886003)(8936002)(26005)(36756003)(7416002)(66556008)(54906003)(66946007)(16526019)(66476007)(186003)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 4Cu0vUN8m7QXUIBWA6u7/uciXci99DhPjLCLgvlN+xno2EYUt5aDsxWFQouQc707WPt7Oae4TRLUWvuX1tuyyhN3NhXUd5eULW/SwLK4VWrX/riMzm+G8JOpZHeVrYWN7tJlD2KlisLajIXc51sTT9yJThx72aLcAJp9nhVo1IaTf0mWWVtwFmffITps72AVTsBRGfMoEuWJbTeZ5WHM0pvsXcTF9YR/vOvPpjbwyONIMvA56Xs+zh1Haf7saqdKuqCqEf64QoGX1bPI2BpJfd1RfxRy7Mohyh5WP5za45IMV1pzBkDbjtR88SIR2aB6kdBEReeosUcnzCeLiJ1pS0Kmgl3Tsjc8GoPf3JCvqC234/iOYRgawon8+JoJfPmv9Mk0jZmZkhQE3KCY2ZcaIVlfjOJ+bHwK79V9oUcszPpr2JzTerVmrWW60HTjKiw771JfgR0pDlVnLq0up718vpjFYQ7zGQmw2Y0Dq49oku74ng2sTq1YDhB46CwxG6Kr//iJcpTyMw/oGb9aFUtbC8RmysZoxPH+UypCeyD+QOgRcgLBbzPijh/o4CSN13cnDflKNq/+uW4FyhI/5zfa3WrUBuL45aiRLOQsq7O+ei+mk1OG2BngavcThs5Lx4NCy8rEgOZZ2+mavagJjE3w38soO8S9E3grUoXYjUCfsP5GkchHVm9YzXvmQYW2HnuEuCfUG3Eyd5ThFh+EB0DPT2cUlRQu/f6f+KNVYjaqX/E9kJNZQJJB4SpWSn3ZW2s7FhwK7/mfGat0bM63r7yoOEEsyZtnTuFbFbh+BKdm7cu0l1l6Z3lqBp1oAD2LGrcd2zpkLmtatd779LNN+KILD9Sce8ZUDlfhBeA33rQN8HRzUKWzRBw+NQ9xg31Xogd/kKYeHQPwrrtyNTnG9aaL/xt5lpJDfGTRrXOmvICvLZj+jDQCXEoCxr6SueMiXN+gIfu9M6PHvmyu1UV+DHhzUAENuf93W23zOQ6dOxefFcy1TdiSWJqSgKerI12bAsO+i9EgMgaTYb7ootiLZWFRvGV2I4dqTVMbLKumxMC0zE7K1VBr/prpAplLb9oyVCvS+9jbpJJJ8A7r2en0jcpiQeLvYTJAv1yxeENRlYQdNu7KBdLgQNu3z4jQXu3q+89/sgwrUfK6bQDwDMn6BEFGgNedv6hjFFN9ZDEPF6rk2MkwQeUjCaTc7gLAAjKt9eW1iFKA4tQ2pqsjp1ZKqq14/xYRkIX4/l/1UBMGRk3vjPGrEbgYgMiN4qm6cUbPNxFdxAYR+KqEqyIRxFuJgIQGX1SzH23JucvCXxbRyNxJH9Ht8moDr262inoMhnt0H9Ct X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: cc42d7b9-8187-473f-15ba-08d8ef24fe17 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2021 00:29:03.5633 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: XXFw/VIQxGIRPP9qdCiHb9IiieXq+WjQxmRVpzAe/Jk9RPEy6uYKQ1IZ6qkMQF7GtIYnb4dpOgANK5T6kwKKLw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4592 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9933 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 phishscore=0 mlxlogscore=999 suspectscore=0 spamscore=0 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103250000 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9933 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 phishscore=0 lowpriorityscore=0 suspectscore=0 clxscore=1015 priorityscore=1501 spamscore=0 adultscore=0 impostorscore=0 mlxlogscore=999 mlxscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103250000 X-Stat-Signature: kw54pkqqw73rap9rnk9m89w8r4f6s55x X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 6F4CDC0007C9 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf03; identity=mailfrom; envelope-from=""; helo=aserp2130.oracle.com; client-ip=141.146.126.79 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616647987-631028 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: free_pool_huge_page was called with hugetlb_lock held. It would remove a hugetlb page, and then free the corresponding pages to the lower level allocators such as buddy. free_pool_huge_page was called in a loop to remove hugetlb pages and these loops could hold the hugetlb_lock for a considerable time. Create new routine remove_pool_huge_page to replace free_pool_huge_page. remove_pool_huge_page will remove the hugetlb page, and it must be called with the hugetlb_lock held. It will return the removed page and it is the responsibility of the caller to free the page to the lower level allocators. The hugetlb_lock is dropped before freeing to these allocators which results in shorter lock hold times. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 88 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 51 insertions(+), 37 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index fae7f034d1eb..a9785e73379f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1204,7 +1204,7 @@ static int hstate_next_node_to_alloc(struct hstate *h, } /* - * helper for free_pool_huge_page() - return the previously saved + * helper for remove_pool_huge_page() - return the previously saved * node ["this node"] from which to free a huge page. Advance the * next node id whether or not we find a free huge page to free so * that the next attempt to free addresses the next node. @@ -1720,16 +1720,18 @@ static int alloc_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, } /* - * Free huge page from pool from next node to free. - * Attempt to keep persistent huge pages more or less - * balanced over allowed nodes. + * Remove huge page from pool from next node to free. Attempt to keep + * persistent huge pages more or less balanced over allowed nodes. + * This routine only 'removes' the hugetlb page. The caller must make + * an additional call to free the page to low level allocators. * Called with hugetlb_lock locked. */ -static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, - bool acct_surplus) +static struct page *remove_pool_huge_page(struct hstate *h, + nodemask_t *nodes_allowed, + bool acct_surplus) { int nr_nodes, node; - int ret = 0; + struct page *page = NULL; for_each_node_mask_to_free(h, nr_nodes, node, nodes_allowed) { /* @@ -1738,23 +1740,14 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, */ if ((!acct_surplus || h->surplus_huge_pages_node[node]) && !list_empty(&h->hugepage_freelists[node])) { - struct page *page = - list_entry(h->hugepage_freelists[node].next, + page = list_entry(h->hugepage_freelists[node].next, struct page, lru); remove_hugetlb_page(h, page, acct_surplus); - /* - * unlock/lock around update_and_free_page is temporary - * and will be removed with subsequent patch. - */ - spin_unlock(&hugetlb_lock); - update_and_free_page(h, page); - spin_lock(&hugetlb_lock); - ret = 1; break; } } - return ret; + return page; } /* @@ -2074,17 +2067,16 @@ static int gather_surplus_pages(struct hstate *h, long delta) * to the associated reservation map. * 2) Free any unused surplus pages that may have been allocated to satisfy * the reservation. As many as unused_resv_pages may be freed. - * - * Called with hugetlb_lock held. However, the lock could be dropped (and - * reacquired) during calls to cond_resched_lock. Whenever dropping the lock, - * we must make sure nobody else can claim pages we are in the process of - * freeing. Do this by ensuring resv_huge_page always is greater than the - * number of huge pages we plan to free when dropping the lock. */ static void return_unused_surplus_pages(struct hstate *h, unsigned long unused_resv_pages) { unsigned long nr_pages; + struct page *page, *t_page; + struct list_head page_list; + + /* Uncommit the reservation */ + h->resv_huge_pages -= unused_resv_pages; /* Cannot return gigantic pages currently */ if (hstate_is_gigantic(h)) @@ -2101,24 +2093,27 @@ static void return_unused_surplus_pages(struct hstate *h, * evenly across all nodes with memory. Iterate across these nodes * until we can no longer free unreserved surplus pages. This occurs * when the nodes with surplus pages have no free pages. - * free_pool_huge_page() will balance the freed pages across the + * remove_pool_huge_page() will balance the freed pages across the * on-line nodes with memory and will handle the hstate accounting. - * - * Note that we decrement resv_huge_pages as we free the pages. If - * we drop the lock, resv_huge_pages will still be sufficiently large - * to cover subsequent pages we may free. */ + INIT_LIST_HEAD(&page_list); while (nr_pages--) { - h->resv_huge_pages--; - unused_resv_pages--; - if (!free_pool_huge_page(h, &node_states[N_MEMORY], 1)) + page = remove_pool_huge_page(h, &node_states[N_MEMORY], 1); + if (!page) goto out; - cond_resched_lock(&hugetlb_lock); + + INIT_LIST_HEAD(&page->lru); + list_add(&page->lru, &page_list); } out: - /* Fully uncommit the reservation */ - h->resv_huge_pages -= unused_resv_pages; + spin_unlock(&hugetlb_lock); + list_for_each_entry_safe(page, t_page, &page_list, lru) { + list_del(&page->lru); + update_and_free_page(h, page); + cond_resched(); + } + spin_lock(&hugetlb_lock); } @@ -2648,6 +2643,8 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, nodemask_t *nodes_allowed) { unsigned long min_count, ret; + struct page *page, *t_page; + struct list_head page_list; NODEMASK_ALLOC(nodemask_t, node_alloc_noretry, GFP_KERNEL); /* @@ -2757,11 +2754,28 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, min_count = h->resv_huge_pages + h->nr_huge_pages - h->free_huge_pages; min_count = max(count, min_count); try_to_free_low(h, min_count, nodes_allowed); + + /* + * Collect pages to be removed on list without dropping lock + */ + INIT_LIST_HEAD(&page_list); while (min_count < persistent_huge_pages(h)) { - if (!free_pool_huge_page(h, nodes_allowed, 0)) + page = remove_pool_huge_page(h, nodes_allowed, 0); + if (!page) break; - cond_resched_lock(&hugetlb_lock); + + INIT_LIST_HEAD(&page->lru); + list_add(&page->lru, &page_list); } + /* free the pages after dropping lock */ + spin_unlock(&hugetlb_lock); + list_for_each_entry_safe(page, t_page, &page_list, lru) { + list_del(&page->lru); + update_and_free_page(h, page); + cond_resched(); + } + spin_lock(&hugetlb_lock); + while (count < persistent_huge_pages(h)) { if (!adjust_pool_surplus(h, nodes_allowed, 1)) break; From patchwork Thu Mar 25 00:28:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12162595 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99F74C433C1 for ; Thu, 25 Mar 2021 00:29:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1772561A14 for ; Thu, 25 Mar 2021 00:29:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1772561A14 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A59FA6B0074; Wed, 24 Mar 2021 20:29:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9E0C26B0075; Wed, 24 Mar 2021 20:29:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 798D46B0078; Wed, 24 Mar 2021 20:29:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0252.hostedemail.com [216.40.44.252]) by kanga.kvack.org (Postfix) with ESMTP id 533A46B0074 for ; Wed, 24 Mar 2021 20:29:28 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 145CD181AEF15 for ; Thu, 25 Mar 2021 00:29:28 +0000 (UTC) X-FDA: 77956512816.17.EE3ACD3 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf07.hostedemail.com (Postfix) with ESMTP id C4B37A0000FC for ; Thu, 25 Mar 2021 00:29:26 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12P0OviV093935; Thu, 25 Mar 2021 00:29:11 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=r/8OLNAJDeRx2ccXwm1+xJZ3IzsTZihzPyH6mnbq5ro=; b=GyN2zOZ2XOTpyotA0ewWqpcQ7JDzkFAfxHLwhq5jqllM2iPdIJRbM852zAO57Pgbqzke YVG81KGecYNmdENQwq3CdoUezCCUGuzTymA3uDU4qpD7uoUyoAcgeIYjd1P1h0xC6C0t cCdNluL/F6TH5YJlFgVgrCTZZcOp50Ovm6/2yEPVH5LUfPxR8RpmABUwgGKVZZAvD92o soIFGxrNJ179b3TkgYqTd6upzgwkKNMx8xsmPFnKyQz33JC9UhuwucnHlAGmVIPZFISu m3A+ZWglly2I8ABfMEeXO+B5sWoQwC+TzogU239pUnTG8u7WQqZoF+84CoH0Mu9vfNeh Hg== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by userp2130.oracle.com with ESMTP id 37d8frcn7u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 00:29:11 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12P0QTDS040331; Thu, 25 Mar 2021 00:29:10 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2043.outbound.protection.outlook.com [104.47.66.43]) by aserp3020.oracle.com with ESMTP id 37dty18p2t-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 00:29:10 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=H7Y0DtsLKuf0QXu4QLa9yEguXWO4eYtg7aGz8PTXh+Nkz9t8u1kmctoiynRleIpVSY3XqqSCzvKfUrFkVBrgnfNDAMg0u5dbEg1aWyxTGdF1uBJups+zJx1jNnXKc7vmce/xmH3KL5RpZca0aVKp10y4CfTVOAAZEG5TaLF1XH/xIuDUpm+qdfnE7QLvtXP9zHMQhatj5wq+W8h3on3zh+QSOm1d9GV6zfFGxvu0kvzNTmX6XBnagGS6bB5iZPWMptP3+Zun01OpjD9jZI/tGxtMD/I9qEwo+xZ14O1FS+fxB2lwxmaQdqmUMCtUhXP3DhbecXt8Lz1NKSs31cc4Mg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=r/8OLNAJDeRx2ccXwm1+xJZ3IzsTZihzPyH6mnbq5ro=; b=oMPke/CnyOwUOOt3gRF7VDcTpkAGGQQjVpcw58TCwmARXHZ6Qf0y/xK+8SLdPIlNK4xmb+JxbyYqM8a6ev69frcNwQVTXCIZRirDR6hKWGD1nCMeuMzCOiqV8Hut4sa3wF1FE4z251G+0az7r6kNxtBCHc7OF65gQm1BBPrd6fiGzpV72oIUQmk44Bl81ahspy6gVlXQCL3eULwXK4H4QOH9PBtRUJKdpjdu8GZUFkdrGEsO+K2tua/XSUGsVdQKariyGzCmhGeimQ5hBDGoPvI7BUbpNoBpkoQdkwOnkCe4YBSiv6OG0+ydF3MR5Esw2PuwicriAUoY3hol5rd/Dg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=r/8OLNAJDeRx2ccXwm1+xJZ3IzsTZihzPyH6mnbq5ro=; b=W6Z7ShqWVlLitsu9XxN780FjOv+auATr0vCI9q1kWr5N5jOH0a4aYAh0RXrY8TROT0p7WK21qUylX9s0YLiSi25IxBufFLmjLTpn/HuDMgG01qYX+RreJOMV2+qUSIupFMk5LTjEIVhXssaHA8ULZYKQSRAMkKut/GRqcxfAZDE= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BY5PR10MB3875.namprd10.prod.outlook.com (2603:10b6:a03:1f9::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.25; Thu, 25 Mar 2021 00:29:07 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3955.027; Thu, 25 Mar 2021 00:29:07 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Roman Gushchin , Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Hillf Danton , Andrew Morton , Mike Kravetz Subject: [PATCH 7/8] hugetlb: make free_huge_page irq safe Date: Wed, 24 Mar 2021 17:28:34 -0700 Message-Id: <20210325002835.216118-8-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210325002835.216118-1-mike.kravetz@oracle.com> References: <20210325002835.216118-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR15CA0042.namprd15.prod.outlook.com (2603:10b6:300:ad::28) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR15CA0042.namprd15.prod.outlook.com (2603:10b6:300:ad::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.24 via Frontend Transport; Thu, 25 Mar 2021 00:29:05 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d49743ef-cf85-4e0a-ace2-08d8ef24ffa9 X-MS-TrafficTypeDiagnostic: BY5PR10MB3875: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1728; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: r0hDwuSaNHAxfA0ki67Qp32oOR4G0IH096T8jCmnNA9IDSEZQaD6ItLcnF5l8d+XDsSzUFnLyM15YtxAoVZT538SksjTu/xCkBKVLUj7vmjjzmiXNvUTbM16bAuZUoKCAHZmphEGGA1uwEFNiQC47ySCNzbDqWW1vawG0M1Cr21wNwRhswjm0fvE+iwuLI3SyQAwWn+eBtr58HXQpVXeGaYMUrpWjq/TPlDqkcuOotlJRiKnP8iDqm+u/noeZFZGRPt5hrTgpihCVgDlqBoGxy9WIg5383k/p2qdSJPsthiO1d8IrT3ItD9IBsrBmY5eGFsrLSDp5YObCkS2/SromRGP9WNFUFt+UwTOT/W45Z1TY6mViyW2QqAGfiWHjuTuikUhul+VeLlRu/C3Z8GPQRuCP5e27QYMFGqInbO8Bzx664VYb5Y3zZHlL3JaMO/4s4F80QeXnhLDeInz8aTVBQDU4+25k+YF2HtmYnem7UcFBtcj3vn8l10WaQiV7jKwQHzJnwmTlv6S+rZF8CmImDLXCpuIpS3WJYfqCSBxZAplVmZtokl5zjD1rFczhlH2YNTKIfZ7clUXKP9+xvXVKhgWdKQYgN4aJjRtGJZ8Y+kHy/Ep3gW6WgxkvGC/vM7UHCrHsQgQoyigML72qN+cp03/E1VFROfFutXPC2iBRF3LC3WBmv2VA43x86u0zpxAkiCqWkvw6o508tfCQHfO4I+Iv1bX5nGkLtxNfQY4BHuhlnSTUAQSN/9Ph3JjTWDi X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(396003)(39860400002)(376002)(366004)(136003)(83380400001)(6486002)(52116002)(186003)(4326008)(66946007)(7696005)(2906002)(26005)(16526019)(66556008)(6666004)(5660300002)(66476007)(8936002)(44832011)(36756003)(316002)(478600001)(30864003)(54906003)(2616005)(8676002)(1076003)(86362001)(38100700001)(956004)(966005)(7416002)(107886003)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: xOXSlXxTktwBszMTT5G8Hy7k41S85cf4MAll7PoSh+7x3ewesM7uUlwZ46i6WLsQaqlUy4fmI0k5trzjdcZS4h3p1JzogsNFwMLCfu7++8G3WTWHBrvQqJHtWHoWcpO+z8E4AXBJvjq8ymr8MdBjp+j2CmnjFlbmAHdr7sbNlF4vI8MZoO2m/Eg+gGtFTww0W5PBR2OMd8OFP92E0rEBm2+RFKCM28HQjLPbh1Mpa2y2HEg/PFv60ushDmAgmYeendFaWhM2qYIKE787izr8vkJ41jupTJ9cyjtUhHvgPOWdkFalhHTPmSXA+qwSl3xI8aU/YHX9H98X0xAjDCKjc/dnLuOcAqOX7NH4NOO/CBPbUD4eRzV3/QBOomXD2f1YKHrh6PpwfqfXnAZpvQ/9tcmFO8rso3Ffhbi8z8iwjAmBZchXcXdpl18Tu7pGGS7cMojhjGy8YMEaN8tsLXfQklMBiIT+JCrwVeVD4oL2/bUj5ZzNg82ApTMJGMp3AvxSffSnPo7B4aawYL3mA+S9w3kQEPnEq2gWRPmqGWQRjUKU5q5mHuPDP/vb8Z5um/aW6D+OtxR8al56bLgmS/vn8LOnO1PxcJepIhuO4FFmGouJfQOlWkDMa8/iDRLDig99S0j4AIUyVvw6N4hOeVLcVRwqIvoizUPo/Bpe9sqMNwqSUpCio8A2qsQYsIrV0yFtkorI0ZQXVubG4RhKrwLK7YoK0NENOrvZYU70oSaN2o+3NwfMHKulfa/e049ypxNAEcqKRzqL825Q6CbbCPjnJ/neEtId8aKkTal9ipQuCy04JdTO0kc4jzRE14n+ooKvexyTdmRhaj4p/uKJumu0oQt6lJk5OVYCdMiZt1OpQkenhpFKiOeFBb/p3GzU4KM/2uYjXGabvBj/wkZWG7zQI2CoBDL8NeZo512jjnotn7bAT1mUTk01LaGNiKgHWytiXkhCezk78qu6UkANbiCUkpJBClbWf4Z7VyHdmSxdw07FulcaL44du/vtm0JE0yoW1DcZhE+fWiu6exphe0DK1zpOiIu4nxDCaGZrLcviFd2Xc44tP+uizlLUsGaZW2ZCzvcfWwB1yP/rw6Ia6UkrXbuyhJMiw65krSOPSWd0HOB+sal9nMoCXOFXRBBmjyyFCbBEyq8F14b0Jx/RkmAXFGzQSur8TR349Yf4uSmScQiuFFpH4WFrp4NVFualxoGqBE4GeKdAkKOSPH21c9hhjBp+Xl1jsY3wKqLggYssI845OGewxTiFTK+xOajeflYcyzOe2sJXXQbarFqxtfNl7MpSbPo4IylGyAtD8snLJrW3ktR9BrSmT2ifVIHqDBX2 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: d49743ef-cf85-4e0a-ace2-08d8ef24ffa9 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2021 00:29:06.4496 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ndNQZ29Aepf6PmTRtDQBTi5yw3Xh38devvc2qnhE13yZoBJgFGBGrD0Pk7hI8dJOJKWBJB2i5JHkJcyIR0E0LA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB3875 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9933 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxlogscore=999 malwarescore=0 phishscore=0 bulkscore=0 mlxscore=0 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103250000 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9933 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 lowpriorityscore=0 priorityscore=1501 impostorscore=0 spamscore=0 mlxscore=0 suspectscore=0 mlxlogscore=999 phishscore=0 bulkscore=0 adultscore=0 malwarescore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103250000 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: C4B37A0000FC X-Stat-Signature: e9uhy59aed3zz179djm5n55raot1bnu3 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf07; identity=mailfrom; envelope-from=""; helo=userp2130.oracle.com; client-ip=156.151.31.86 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616632166-542849 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Commit c77c0a8ac4c5 ("mm/hugetlb: defer freeing of huge pages if in non-task context") was added to address the issue of free_huge_page being called from irq context. That commit hands off free_huge_page processing to a workqueue if !in_task. However, as seen in [1] this does not cover all cases. Instead, make the locks taken in the free_huge_page irq safe. This patch does the following: - Make hugetlb_lock irq safe. This is mostly a simple process of changing spin_*lock calls to spin_*lock_irq* calls. - Make subpool lock irq safe in a similar manner. - Revert the !in_task check and workqueue handoff. [1] https://lore.kernel.org/linux-mm/000000000000f1c03b05bc43aadc@google.com/ Signed-off-by: Mike Kravetz Acked-by: Michal Hocko Reviewed-by: Muchun Song --- mm/hugetlb.c | 169 +++++++++++++++++--------------------------- mm/hugetlb_cgroup.c | 8 +-- 2 files changed, 67 insertions(+), 110 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index a9785e73379f..e4c441b878f2 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -93,9 +93,10 @@ static inline bool subpool_is_free(struct hugepage_subpool *spool) return true; } -static inline void unlock_or_release_subpool(struct hugepage_subpool *spool) +static inline void unlock_or_release_subpool(struct hugepage_subpool *spool, + unsigned long irq_flags) { - spin_unlock(&spool->lock); + spin_unlock_irqrestore(&spool->lock, irq_flags); /* If no pages are used, and no other handles to the subpool * remain, give up any reservations based on minimum size and @@ -134,10 +135,12 @@ struct hugepage_subpool *hugepage_new_subpool(struct hstate *h, long max_hpages, void hugepage_put_subpool(struct hugepage_subpool *spool) { - spin_lock(&spool->lock); + unsigned long flags; + + spin_lock_irqsave(&spool->lock, flags); BUG_ON(!spool->count); spool->count--; - unlock_or_release_subpool(spool); + unlock_or_release_subpool(spool, flags); } /* @@ -156,7 +159,7 @@ static long hugepage_subpool_get_pages(struct hugepage_subpool *spool, if (!spool) return ret; - spin_lock(&spool->lock); + spin_lock_irq(&spool->lock); if (spool->max_hpages != -1) { /* maximum size accounting */ if ((spool->used_hpages + delta) <= spool->max_hpages) @@ -183,7 +186,7 @@ static long hugepage_subpool_get_pages(struct hugepage_subpool *spool, } unlock_ret: - spin_unlock(&spool->lock); + spin_unlock_irq(&spool->lock); return ret; } @@ -197,11 +200,12 @@ static long hugepage_subpool_put_pages(struct hugepage_subpool *spool, long delta) { long ret = delta; + unsigned long flags; if (!spool) return delta; - spin_lock(&spool->lock); + spin_lock_irqsave(&spool->lock, flags); if (spool->max_hpages != -1) /* maximum size accounting */ spool->used_hpages -= delta; @@ -222,7 +226,7 @@ static long hugepage_subpool_put_pages(struct hugepage_subpool *spool, * If hugetlbfs_put_super couldn't free spool due to an outstanding * quota reference, free it now. */ - unlock_or_release_subpool(spool); + unlock_or_release_subpool(spool, flags); return ret; } @@ -1401,7 +1405,7 @@ struct hstate *size_to_hstate(unsigned long size) return NULL; } -static void __free_huge_page(struct page *page) +void free_huge_page(struct page *page) { /* * Can't pass hstate in here because it is called from the @@ -1411,6 +1415,7 @@ static void __free_huge_page(struct page *page) int nid = page_to_nid(page); struct hugepage_subpool *spool = hugetlb_page_subpool(page); bool restore_reserve; + unsigned long flags; VM_BUG_ON_PAGE(page_count(page), page); VM_BUG_ON_PAGE(page_mapcount(page), page); @@ -1439,7 +1444,7 @@ static void __free_huge_page(struct page *page) restore_reserve = true; } - spin_lock(&hugetlb_lock); + spin_lock_irqsave(&hugetlb_lock, flags); ClearHPageMigratable(page); hugetlb_cgroup_uncharge_page(hstate_index(h), pages_per_huge_page(h), page); @@ -1450,66 +1455,18 @@ static void __free_huge_page(struct page *page) if (HPageTemporary(page)) { remove_hugetlb_page(h, page, false); - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); update_and_free_page(h, page); } else if (h->surplus_huge_pages_node[nid]) { /* remove the page from active list */ remove_hugetlb_page(h, page, true); - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); update_and_free_page(h, page); } else { arch_clear_hugepage_flags(page); enqueue_huge_page(h, page); - spin_unlock(&hugetlb_lock); - } -} - -/* - * As free_huge_page() can be called from a non-task context, we have - * to defer the actual freeing in a workqueue to prevent potential - * hugetlb_lock deadlock. - * - * free_hpage_workfn() locklessly retrieves the linked list of pages to - * be freed and frees them one-by-one. As the page->mapping pointer is - * going to be cleared in __free_huge_page() anyway, it is reused as the - * llist_node structure of a lockless linked list of huge pages to be freed. - */ -static LLIST_HEAD(hpage_freelist); - -static void free_hpage_workfn(struct work_struct *work) -{ - struct llist_node *node; - struct page *page; - - node = llist_del_all(&hpage_freelist); - - while (node) { - page = container_of((struct address_space **)node, - struct page, mapping); - node = node->next; - __free_huge_page(page); - } -} -static DECLARE_WORK(free_hpage_work, free_hpage_workfn); - -void free_huge_page(struct page *page) -{ - /* - * Defer freeing if in non-task context to avoid hugetlb_lock deadlock. - */ - if (!in_task()) { - /* - * Only call schedule_work() if hpage_freelist is previously - * empty. Otherwise, schedule_work() had been called but the - * workfn hasn't retrieved the list yet. - */ - if (llist_add((struct llist_node *)&page->mapping, - &hpage_freelist)) - schedule_work(&free_hpage_work); - return; + spin_unlock_irqrestore(&hugetlb_lock, flags); } - - __free_huge_page(page); } static void prep_new_huge_page(struct hstate *h, struct page *page, int nid) @@ -1519,11 +1476,11 @@ static void prep_new_huge_page(struct hstate *h, struct page *page, int nid) hugetlb_set_page_subpool(page, NULL); set_hugetlb_cgroup(page, NULL); set_hugetlb_cgroup_rsvd(page, NULL); - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); h->nr_huge_pages++; h->nr_huge_pages_node[nid]++; ClearHPageFreed(page); - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); } static void prep_compound_gigantic_page(struct page *page, unsigned int order) @@ -1769,7 +1726,7 @@ int dissolve_free_huge_page(struct page *page) if (!PageHuge(page)) return 0; - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); if (!PageHuge(page)) { rc = 0; goto out; @@ -1786,7 +1743,7 @@ int dissolve_free_huge_page(struct page *page) * when it is dissolved. */ if (unlikely(!HPageFreed(head))) { - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); cond_resched(); /* @@ -1810,12 +1767,12 @@ int dissolve_free_huge_page(struct page *page) } remove_hugetlb_page(h, page, false); h->max_huge_pages--; - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); update_and_free_page(h, head); return 0; } out: - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); return rc; } @@ -1857,16 +1814,16 @@ static struct page *alloc_surplus_huge_page(struct hstate *h, gfp_t gfp_mask, if (hstate_is_gigantic(h)) return NULL; - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); if (h->surplus_huge_pages >= h->nr_overcommit_huge_pages) goto out_unlock; - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); page = alloc_fresh_huge_page(h, gfp_mask, nid, nmask, NULL); if (!page) return NULL; - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); /* * We could have raced with the pool size change. * Double check that and simply deallocate the new page @@ -1876,7 +1833,7 @@ static struct page *alloc_surplus_huge_page(struct hstate *h, gfp_t gfp_mask, */ if (h->surplus_huge_pages >= h->nr_overcommit_huge_pages) { SetHPageTemporary(page); - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); put_page(page); return NULL; } else { @@ -1885,7 +1842,7 @@ static struct page *alloc_surplus_huge_page(struct hstate *h, gfp_t gfp_mask, } out_unlock: - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); return page; } @@ -1935,17 +1892,17 @@ struct page *alloc_buddy_huge_page_with_mpol(struct hstate *h, struct page *alloc_huge_page_nodemask(struct hstate *h, int preferred_nid, nodemask_t *nmask, gfp_t gfp_mask) { - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); if (h->free_huge_pages - h->resv_huge_pages > 0) { struct page *page; page = dequeue_huge_page_nodemask(h, gfp_mask, preferred_nid, nmask); if (page) { - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); return page; } } - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); return alloc_migrate_huge_page(h, gfp_mask, preferred_nid, nmask); } @@ -1993,7 +1950,7 @@ static int gather_surplus_pages(struct hstate *h, long delta) ret = -ENOMEM; retry: - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); for (i = 0; i < needed; i++) { page = alloc_surplus_huge_page(h, htlb_alloc_mask(h), NUMA_NO_NODE, NULL); @@ -2010,7 +1967,7 @@ static int gather_surplus_pages(struct hstate *h, long delta) * After retaking hugetlb_lock, we need to recalculate 'needed' * because either resv_huge_pages or free_huge_pages may have changed. */ - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); needed = (h->resv_huge_pages + delta) - (h->free_huge_pages + allocated); if (needed > 0) { @@ -2050,12 +2007,12 @@ static int gather_surplus_pages(struct hstate *h, long delta) enqueue_huge_page(h, page); } free: - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); /* Free unnecessary surplus pages to the buddy allocator */ list_for_each_entry_safe(page, tmp, &surplus_list, lru) put_page(page); - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); return ret; } @@ -2107,13 +2064,13 @@ static void return_unused_surplus_pages(struct hstate *h, } out: - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); list_for_each_entry_safe(page, t_page, &page_list, lru) { list_del(&page->lru); update_and_free_page(h, page); cond_resched(); } - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); } @@ -2348,7 +2305,7 @@ struct page *alloc_huge_page(struct vm_area_struct *vma, if (ret) goto out_uncharge_cgroup_reservation; - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); /* * glb_chg is passed to indicate whether or not a page must be taken * from the global free pool (global change). gbl_chg == 0 indicates @@ -2356,7 +2313,7 @@ struct page *alloc_huge_page(struct vm_area_struct *vma, */ page = dequeue_huge_page_vma(h, vma, addr, avoid_reserve, gbl_chg); if (!page) { - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); page = alloc_buddy_huge_page_with_mpol(h, vma, addr); if (!page) goto out_uncharge_cgroup; @@ -2364,7 +2321,7 @@ struct page *alloc_huge_page(struct vm_area_struct *vma, SetHPageRestoreReserve(page); h->resv_huge_pages--; } - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); list_add(&page->lru, &h->hugepage_activelist); /* Fall through */ } @@ -2377,7 +2334,7 @@ struct page *alloc_huge_page(struct vm_area_struct *vma, h_cg, page); } - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); hugetlb_set_page_subpool(page, spool); @@ -2591,13 +2548,13 @@ static void try_to_free_low(struct hstate *h, unsigned long count, } out: - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); list_for_each_entry_safe(page, next, &page_list, lru) { list_del(&page->lru); update_and_free_page(h, page); cond_resched(); } - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); } #else static inline void try_to_free_low(struct hstate *h, unsigned long count, @@ -2659,7 +2616,7 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, /* mutex prevents concurrent adjustments for the same hstate */ mutex_lock(&h->mutex); - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); /* * Check for a node specific request. @@ -2690,7 +2647,7 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, */ if (hstate_is_gigantic(h) && !IS_ENABLED(CONFIG_CONTIG_ALLOC)) { if (count > persistent_huge_pages(h)) { - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); mutex_unlock(&h->mutex); NODEMASK_FREE(node_alloc_noretry); return -EINVAL; @@ -2720,14 +2677,14 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, * page, free_huge_page will handle it by freeing the page * and reducing the surplus. */ - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); /* yield cpu to avoid soft lockup */ cond_resched(); ret = alloc_pool_huge_page(h, nodes_allowed, node_alloc_noretry); - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); if (!ret) goto out; @@ -2768,13 +2725,13 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, list_add(&page->lru, &page_list); } /* free the pages after dropping lock */ - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); list_for_each_entry_safe(page, t_page, &page_list, lru) { list_del(&page->lru); update_and_free_page(h, page); cond_resched(); } - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); while (count < persistent_huge_pages(h)) { if (!adjust_pool_surplus(h, nodes_allowed, 1)) @@ -2782,7 +2739,7 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, } out: h->max_huge_pages = persistent_huge_pages(h); - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); mutex_unlock(&h->mutex); NODEMASK_FREE(node_alloc_noretry); @@ -2938,9 +2895,9 @@ static ssize_t nr_overcommit_hugepages_store(struct kobject *kobj, if (err) return err; - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); h->nr_overcommit_huge_pages = input; - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); return count; } @@ -3527,9 +3484,9 @@ int hugetlb_overcommit_handler(struct ctl_table *table, int write, goto out; if (write) { - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); h->nr_overcommit_huge_pages = tmp; - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); } out: return ret; @@ -3625,7 +3582,7 @@ static int hugetlb_acct_memory(struct hstate *h, long delta) if (!delta) return 0; - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); /* * When cpuset is configured, it breaks the strict hugetlb page * reservation as the accounting is done on a global variable. Such @@ -3664,7 +3621,7 @@ static int hugetlb_acct_memory(struct hstate *h, long delta) return_unused_surplus_pages(h, (unsigned long) -delta); out: - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); return ret; } @@ -5727,7 +5684,7 @@ bool isolate_huge_page(struct page *page, struct list_head *list) { bool ret = true; - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); if (!PageHeadHuge(page) || !HPageMigratable(page) || !get_page_unless_zero(page)) { @@ -5737,16 +5694,16 @@ bool isolate_huge_page(struct page *page, struct list_head *list) ClearHPageMigratable(page); list_move_tail(&page->lru, list); unlock: - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); return ret; } void putback_active_hugepage(struct page *page) { - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); SetHPageMigratable(page); list_move_tail(&page->lru, &(page_hstate(page))->hugepage_activelist); - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); put_page(page); } @@ -5780,12 +5737,12 @@ void move_hugetlb_state(struct page *oldpage, struct page *newpage, int reason) */ if (new_nid == old_nid) return; - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); if (h->surplus_huge_pages_node[old_nid]) { h->surplus_huge_pages_node[old_nid]--; h->surplus_huge_pages_node[new_nid]++; } - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); } } diff --git a/mm/hugetlb_cgroup.c b/mm/hugetlb_cgroup.c index 726b85f4f303..5383023d0cca 100644 --- a/mm/hugetlb_cgroup.c +++ b/mm/hugetlb_cgroup.c @@ -204,11 +204,11 @@ static void hugetlb_cgroup_css_offline(struct cgroup_subsys_state *css) do { idx = 0; for_each_hstate(h) { - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); list_for_each_entry(page, &h->hugepage_activelist, lru) hugetlb_cgroup_move_parent(idx, h_cg, page); - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); idx++; } cond_resched(); @@ -784,7 +784,7 @@ void hugetlb_cgroup_migrate(struct page *oldhpage, struct page *newhpage) if (hugetlb_cgroup_disabled()) return; - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); h_cg = hugetlb_cgroup_from_page(oldhpage); h_cg_rsvd = hugetlb_cgroup_from_page_rsvd(oldhpage); set_hugetlb_cgroup(oldhpage, NULL); @@ -794,7 +794,7 @@ void hugetlb_cgroup_migrate(struct page *oldhpage, struct page *newhpage) set_hugetlb_cgroup(newhpage, h_cg); set_hugetlb_cgroup_rsvd(newhpage, h_cg_rsvd); list_move(&newhpage->lru, &h->hugepage_activelist); - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); return; } From patchwork Thu Mar 25 00:28:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12162591 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68E99C433DB for ; Thu, 25 Mar 2021 00:29:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F28F661A1D for ; Thu, 25 Mar 2021 00:29:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F28F661A1D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 80DFB6B0072; Wed, 24 Mar 2021 20:29:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 796FE6B0073; Wed, 24 Mar 2021 20:29:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 52BF66B0075; Wed, 24 Mar 2021 20:29:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0197.hostedemail.com [216.40.44.197]) by kanga.kvack.org (Postfix) with ESMTP id 448556B0072 for ; Wed, 24 Mar 2021 20:29:22 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 07D23824C45A for ; Thu, 25 Mar 2021 00:29:22 +0000 (UTC) X-FDA: 77956512564.23.655DBD3 Received: from aserp2120.oracle.com (aserp2120.oracle.com [141.146.126.78]) by imf21.hostedemail.com (Postfix) with ESMTP id 791BBE0011EE for ; Thu, 25 Mar 2021 00:29:20 +0000 (UTC) Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12P0PS9u005752; Thu, 25 Mar 2021 00:29:11 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=WdVUnI2JmHN0jJBwfh7qNh6wERVSNwNKKiV03MJuxNo=; b=ISg0hkO9p8fJmKgE/ESCD6bls0pCYuJjtPJQ0riMd+VwBLH00+5L3o3hLnq9G81O3AWj FZ8zpfQ1jdbxibVAEyzWWygdCU6hw6UcAdbbdHofA3lxzT+todNhPTeXD3zENE+LNSGU qUg7hVIKnTrziWbwAVcea9ue43r4dpGG+GmCSySUytY7FXMtsT8PyzR5E4rKnLN4p1FA Jd5tNQxaa0pMjXHT2qgT7TpFlXE0ooZxgG/POTcDOVsJH16Igc4kybvebWWX5txdeXYb ASGLkLtkZc837KpsSXoY23zPN6ksm6udAR5kGwwSuPTBADk21NSawSMZFrZ8oBkjTP1m kA== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by aserp2120.oracle.com with ESMTP id 37d90mmmch-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 00:29:11 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12P0QTDT040331; Thu, 25 Mar 2021 00:29:11 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2043.outbound.protection.outlook.com [104.47.66.43]) by aserp3020.oracle.com with ESMTP id 37dty18p2t-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 00:29:11 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KqwQBKmZ5+h6U6WiUWdBJlrJ+el6Sq/zBlwUORbMOAezeb/UFhXqhunKaQAihOmOvKivbc+a1ShxDxUx00wp3Z88Jsntmx/04R9KfSQoaMiCwA1L6s8EpzpcliDNnKsWxZJMnuzQsOCv8lDi9mpCHIylr1bXXU8o2Olvt2/nBJxu/kDpesXZS3FuZ7P3kJCS4Iigf47Mwb/dm6A/4IHbIvZYjf/xoKEkEQciuMRwfgrb14V49bNV6w6inPfAYCzmSztPvrVzulKV7SnVLgk9nE90plxB7/Xz148QJChB6sd3YEAEsPWgNIKj4evTUmXffx7Z/UIPr/ZXL0dk0wC1qA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WdVUnI2JmHN0jJBwfh7qNh6wERVSNwNKKiV03MJuxNo=; b=Y0Ee67R7r3r2GjG8NFZSLUrS1tbyMHKEY/QhKTy7OYMJlkwN/6oGWTFXdtXFuTQ21YAQhwqc9ks1jj6jkWupZPMoJz1o5cZlNFXQNgFwnT/zJ/pmZ2SsglskzIUrBY/ONYJpaMIe0b3cSqB+IWosDVaVW+wZlEeqDFPJy9uW+E7Gb6bbhqCamiWWvV7xKgU8q3dbOt0s2efZWcaD+AA9DlyWsfLPFRPp6DiKWxcx7gMBCUV2BRu3gRNMk9fqktrS70r78uba2x5Haf6GLoec2HcPJpeQFSuQZSUN0p5RNsvbDgH0qTSuG3osM469e4fO7Lx0qJKlqtNk1zrtarawjg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WdVUnI2JmHN0jJBwfh7qNh6wERVSNwNKKiV03MJuxNo=; b=wDpU8BuueW4DLvy1xUrDa6A8VJcmhCnBJXhlsqu3HovVUKynZJ6I2j/iW4RP5q2SqA9b0Z73mz+ZbGZHeYyzFxQAvPCBcGiLqa/aZJ9TEwZ1HWNRiOY3J06jiid6xeekRuk2aF/6cX8L6xsyVTYDItfGZwY21iaIPsDBclC5rmc= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BY5PR10MB3875.namprd10.prod.outlook.com (2603:10b6:a03:1f9::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.25; Thu, 25 Mar 2021 00:29:08 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3955.027; Thu, 25 Mar 2021 00:29:08 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Roman Gushchin , Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Hillf Danton , Andrew Morton , Mike Kravetz Subject: [PATCH 8/8] hugetlb: add lockdep_assert_held() calls for hugetlb_lock Date: Wed, 24 Mar 2021 17:28:35 -0700 Message-Id: <20210325002835.216118-9-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210325002835.216118-1-mike.kravetz@oracle.com> References: <20210325002835.216118-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR15CA0042.namprd15.prod.outlook.com (2603:10b6:300:ad::28) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR15CA0042.namprd15.prod.outlook.com (2603:10b6:300:ad::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.24 via Frontend Transport; Thu, 25 Mar 2021 00:29:07 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d4f3e426-7179-4b85-4eb3-08d8ef2500fc X-MS-TrafficTypeDiagnostic: BY5PR10MB3875: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2887; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: j46+KVww0v0VvHqpZeGON/JC1r4AruZkh/QLJIrYPu721W3rwgMoJQZimE91aUqOHcXPKYm50Y5zYqveg4iFxcNdMTyGCuT6jxJ4EJjqiSiEMy+h7L46Uk7ujW/pu+TWBgUXApHnENmX/lyqAn91LZbBTqTZVg3IMvCBREXRYeEHLEsT4h8Q9isA4ODD3C75N01HoYXVxrQpxzN14t6B0tCWlhi8XYNzHmA7/QL4f9RMehT46IoFEO/a3uMrD3NH0e6Q89MaHQvs1pKC3FB7nZKnO2+NXG3wx9PV5xVMDifvLYritNocFy6v0ax1UQLycq0ZL4X0TDgls7n8LuueqhhAtpv6HUwyBAI2k8dnp7LU2b3E2K4eeq2XY7OOA1T9Tenfpa+8Bw3fgmtn0mu2iOyAwQSrtt77WsswYEn2SUxkTcP/3g+VOGoU2LUv0p1otET0P//XsnreO9/Ogffqmq2di8tZB4y/kgBA0vCx+PDAklNNP9hlQ3kkT4MBuCE8mh/U2pUv/v38q5U4+ZovuQej5HhLmJGQTA9vT+dmKcsZ4WWhTEKhfejmmYLI5m00DuJwRGf8+W42/g5z7aK8HEbjGCHRuOjbBKaej7sbOypCBMHjE9TV6Y1ZwDJUN4KLDQa4H0UwlcFFkZHFjVDc3A== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(396003)(39860400002)(376002)(366004)(136003)(83380400001)(6486002)(52116002)(186003)(4326008)(66946007)(7696005)(2906002)(26005)(16526019)(66556008)(6666004)(5660300002)(66476007)(8936002)(44832011)(36756003)(316002)(478600001)(54906003)(2616005)(8676002)(1076003)(86362001)(38100700001)(956004)(7416002)(107886003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: fEPmPW3F5YG768ywihtIV06UaULC18iLIYJPtxNYfQoyVQsBEV07og0TreJxS2djzzaWTql3zOiuMJ42vlCzJTopFgoIJuqe80wthK1Gcl3tUj3rnPxmUPPoBfWcV4faA7d4v6mzrNBJYvqk0fKyPFbvmARpXUg7O0fu7jz//SKG8wwnQOQq3Z7EEUbi5ffLWvvsdSrXSXfzrepEIKFCybL0CcE/Ts6OvrOKbPMcA3udC/izlzQ1nmU8Mqifgx25ZUbQuLXCDh0Mxgr9NNA59C31BjQM3zMMTxMxTcxDMrKJxZFjWUNcNRzdNNRkMn0CH7iI9ifRDH6qvXqwGtJ8YXJJQR+jVp+dz9zPglVUiwEp9w0ZIUZl3vy9z3t0AAs+Br0R0Ds5rTspKjEql6kx5dSlppXKpGhICTaoFGi148kv79UMVfJMEmlmVC3S1S9x0Ep9pguII5lEPWriJjJWPOqLL47vprNCVFsZ6Y4Wx4Ks1J/4wiqh25SeikO31GZuBaGMSm/s1DNUldouvKVXEQopwA3zUzqUfaevNBJ7yp038HY7NMSMVPtjb9p5Dzf1ejNBBaSbCB8lkxy7Q1wmSNCGaMs0XCF3NQozYbJYtGtTSDPZFUqZh2188+cw9o9ofrvKeR1mUDNBCsTF7Qg4eovekcFUEi2hq9ZcOAQAANu8zJLSHjXy3NJcBGjoLjeLl5E1Cv7TFmVaQrdwLxxV3x/oDLZhG1z/KDPH3NzdIlLdzMy03IMFLtr+oC75t/GqdOeK6ffXu+3rNieCs/l1c/IubUiouIe0zBTogOH/MSRV9eHYgXhUSjeM8o0V1AgHGH7UEgD4dM9b83loA9y4JDC+McdYimWXQ1lZRqq8EvhqOPDP51ruMuGY81xE7EtA8qWEsVOcjW2sDGXYy/TwZuoLpK0JQeF69GE1It5qK9QgREf8d4LfTOkcJfpMLRMX1WhrSXNQGzvEKLoyJVwmUo9NAl6QyefwJ0jZEz9SSAu2rNIm8YcwdDKYOe0XdhQmvuGwraOkPFHCPoyV5pA9QTk/aCMO+InP3VNkzcjhW64aa3ZLJOdCRbB/yTFiw38FOkoARANO/HFO3TezlSXeW0TMnRd9fd5QcpwRgocnd09TMjV096y0eWGMFlvHq+Y/UepMpWU//0ixaI4/rW48ffyesg+FGBJ7FOdyjeQ0IIpAli+uA6RA1kqXXH77ewGxNV0whtkyYUwt91pzx+mgE9doF06u5arBUxZG2oOkJtVOzgkCMUiC1iYrDZS86JfCl6dY8emxWeOLWDhZkju9HjiBpxIhfV6W9p03lu4WzNsz8C7IxOQMCyyiaowBSFq0 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: d4f3e426-7179-4b85-4eb3-08d8ef2500fc X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2021 00:29:08.3553 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: From6nv2s4d2pGj0MItZTFRoP7yqQTU9lLkGLIIMWpotS5ByfSzuJI95M8hXNpFihsb39/Fzy1KW3LfQJvhcmQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB3875 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9933 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxlogscore=999 malwarescore=0 phishscore=0 bulkscore=0 mlxscore=0 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103250000 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9933 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 malwarescore=0 mlxscore=0 priorityscore=1501 bulkscore=0 impostorscore=0 lowpriorityscore=0 phishscore=0 mlxlogscore=999 suspectscore=0 clxscore=1015 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103250000 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 791BBE0011EE X-Stat-Signature: a3jgdyf8fu195mjctf3cpbk7hhayyjr3 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf21; identity=mailfrom; envelope-from=""; helo=aserp2120.oracle.com; client-ip=141.146.126.78 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616632160-815336 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: After making hugetlb lock irq safe and separating some functionality done under the lock, add some lockdep_assert_held to help verify locking. Signed-off-by: Mike Kravetz Acked-by: Michal Hocko Reviewed-by: Miaohe Lin Reviewed-by: Muchun Song --- mm/hugetlb.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index e4c441b878f2..de5b3cf4a155 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1062,6 +1062,8 @@ static bool vma_has_reserves(struct vm_area_struct *vma, long chg) static void enqueue_huge_page(struct hstate *h, struct page *page) { int nid = page_to_nid(page); + + lockdep_assert_held(&hugetlb_lock); list_move(&page->lru, &h->hugepage_freelists[nid]); h->free_huge_pages++; h->free_huge_pages_node[nid]++; @@ -1073,6 +1075,7 @@ static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) struct page *page; bool pin = !!(current->flags & PF_MEMALLOC_PIN); + lockdep_assert_held(&hugetlb_lock); list_for_each_entry(page, &h->hugepage_freelists[nid], lru) { if (pin && !is_pinnable_page(page)) continue; @@ -1345,6 +1348,7 @@ static void remove_hugetlb_page(struct hstate *h, struct page *page, { int nid = page_to_nid(page); + lockdep_assert_held(&hugetlb_lock); if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported()) return; @@ -1690,6 +1694,7 @@ static struct page *remove_pool_huge_page(struct hstate *h, int nr_nodes, node; struct page *page = NULL; + lockdep_assert_held(&hugetlb_lock); for_each_node_mask_to_free(h, nr_nodes, node, nodes_allowed) { /* * If we're returning unused surplus pages, only examine @@ -1939,6 +1944,7 @@ static int gather_surplus_pages(struct hstate *h, long delta) long needed, allocated; bool alloc_ok = true; + lockdep_assert_held(&hugetlb_lock); needed = (h->resv_huge_pages + delta) - h->free_huge_pages; if (needed <= 0) { h->resv_huge_pages += delta; @@ -2032,6 +2038,7 @@ static void return_unused_surplus_pages(struct hstate *h, struct page *page, *t_page; struct list_head page_list; + lockdep_assert_held(&hugetlb_lock); /* Uncommit the reservation */ h->resv_huge_pages -= unused_resv_pages; @@ -2527,6 +2534,7 @@ static void try_to_free_low(struct hstate *h, unsigned long count, struct list_head page_list; struct page *page, *next; + lockdep_assert_held(&hugetlb_lock); if (hstate_is_gigantic(h)) return; @@ -2573,6 +2581,7 @@ static int adjust_pool_surplus(struct hstate *h, nodemask_t *nodes_allowed, { int nr_nodes, node; + lockdep_assert_held(&hugetlb_lock); VM_BUG_ON(delta != -1 && delta != 1); if (delta < 0) {