From patchwork Mon Apr 5 23:00:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12183811 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76246C43460 for ; Mon, 5 Apr 2021 23:02:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EEF2C613BC for ; Mon, 5 Apr 2021 23:02:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EEF2C613BC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B3A176B007B; Mon, 5 Apr 2021 19:01:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B10E86B007D; Mon, 5 Apr 2021 19:01:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8ED2A6B007E; Mon, 5 Apr 2021 19:01:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 683956B007B for ; Mon, 5 Apr 2021 19:01:58 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 242488249980 for ; Mon, 5 Apr 2021 23:01:58 +0000 (UTC) X-FDA: 77999837916.16.D76358B Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) by imf18.hostedemail.com (Postfix) with ESMTP id 3E5592000263 for ; Mon, 5 Apr 2021 23:01:57 +0000 (UTC) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 135MxKqT191525; Mon, 5 Apr 2021 23:01:24 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=afCUay6aDA1HVi46U0n4bQgnXO/c96gibkJTrCcAfaA=; b=E42zgQ+N9kcnDU/ruYrAlIgSR5ATGMMUBnnoyIdbu5w9NloeHRfTn5ryrnNZt8fDj3eq J47jJE58bv/jX1KIiHpkGE2HgGrDjO1rkWQELc7UNffw6ZxzVcCHIkacr+a+OXn8RZts 8ECX1w3v3UcuSJ2cc7PnFvzzCI2/Ygh1KZ5AsC3bb+1ynws9PXK/o1Ktdb/qdTZz0A9z 4Me75aJIg30/PhIinoz+dodNXZtQeyQSJzMzDwfGCoo4AWos+OJsACvoWeSKhKavhUK/ XlsFn7r1+ZJPtRdIPjBrCzGbGLfwR2GOdcpgcf4qEh2oWQviv+Xl9drGkAzZkx3cTlsD 8A== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by userp2120.oracle.com with ESMTP id 37q3f2bkn4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 05 Apr 2021 23:01:24 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 135MtBZa020545; Mon, 5 Apr 2021 23:01:23 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2106.outbound.protection.outlook.com [104.47.55.106]) by aserp3020.oracle.com with ESMTP id 37q2pwt0wf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 05 Apr 2021 23:01:23 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mwipTAEmJ0Qst+KK8ZAU9ub+ERWLEFDO/+UC937Lec9lDHo5geVtKapYyDWCpuIxoQkco25Z5fDXDsQZAzL4S6eig5hw9PFGEbe9KjSiqnYrfoSuCUBdF5zbslfd7O/KR5ZYnRek84g5y/w2uOcJ2ogZJxA0gbrbKBCout3qvxLUM67VXRRB2ps5OcTBbYD3GvhBYmi9vX+oKCbYQMNxdlwqrbKIWDwnex1HNEJBB6Kex5yKQXfaK9QzAwMfnf79s+cuSPntlzNTyqcfBv9tKRMQrw9DlnGoPBHD4mzC+2AtQ+qWF9lJ8d+4j8sSFMWTrBNkNti2I3/Zo6M6lXWnzQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=afCUay6aDA1HVi46U0n4bQgnXO/c96gibkJTrCcAfaA=; b=RY828NNuWh/UzkJvEY8egszw1OyETPxhxruQ5sxeG383lPPJoDAcd6ScW2mZiBck+wevVeOEChtc5tBShb+ZWW0UeoQAnW79QFLaktoTp/SiZYCPYgzitqb0sQszym3MyHAaudsoBY89jvaiwP4mI97dLLAkRWslXglV409Pckz0fkbJ2FvTA+7jh2ip4L8iMg+RwmFHLS52RIJaDDp68ebesTwGacj2o4d7CDQR3MtpCSbfERe5mAu2b5M86NYS6mPIyW3NlBFylVoBJ9w8h+sa8jx9fr2BD6uQtsqojxi64lRdC6L9cR/uJi10vcpuOfylo0/Mk4gQWEUeBLsH1w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=afCUay6aDA1HVi46U0n4bQgnXO/c96gibkJTrCcAfaA=; b=fMCCpHz6yxZf2Wg3rOQ7v/WPPsFoSq9DN1deD+GRA7ic8mNuIdg1gcX0t1Y1/QaWgMlaNXtNI3u1GslV7o1y3JO8pWmGbfLicW8XXfxfzOnhOtNOvB7Q8TVHqj0WNQRmWeWiaJfSpbJSqDdg8d7jk+BYHg3pRTT4balsVFHSs1g= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by SJ0PR10MB5424.namprd10.prod.outlook.com (2603:10b6:a03:3bc::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.28; Mon, 5 Apr 2021 23:01:21 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3999.032; Mon, 5 Apr 2021 23:01:21 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Roman Gushchin , Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Hillf Danton , Joonsoo Kim , Barry Song , Will Deacon , Andrew Morton , Mike Kravetz Subject: [PATCH v4 1/8] mm/cma: change cma mutex to irq safe spinlock Date: Mon, 5 Apr 2021 16:00:36 -0700 Message-Id: <20210405230043.182734-2-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210405230043.182734-1-mike.kravetz@oracle.com> References: <20210405230043.182734-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR18CA0031.namprd18.prod.outlook.com (2603:10b6:320:31::17) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR18CA0031.namprd18.prod.outlook.com (2603:10b6:320:31::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.27 via Frontend Transport; Mon, 5 Apr 2021 23:01:19 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 48e65f73-890b-438b-3a2a-08d8f886b9e0 X-MS-TrafficTypeDiagnostic: SJ0PR10MB5424: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7691; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qyq5xlZS/N0GDE1HJQskH9F3iISArqNd8Wiqe4tHzEQMgboUEGNsA3edEK6XczIwsphrYnuZu4zEx0ZyJs3/PiNyU6NZ0/+mKKR96MfvbJPklOVQycbT7OYgwEZbTMdSOvGapfXvL2WQLOaYkW//hUw8uy1xI4Zxl4Ty7BWFvD81HrqTwtxm+3RK+OZOxKDCWKfvIh3goczn5PQY4i7vZsrvw4L1gfbvdfuwSeKzWs29SANYctPYlYH0hkw/V/CSZt6Po0KV8xm00Xv6ELg4AGjhnkcIQQPk10X4q6kLOlxTgDcxGUP2wSZhUTgn1CSzRDy7Yvvq/6y+RU+5XIS/XNgq22ATSRLTkvna+lXKyPSmkCsQAj2eKJ/CZAz2h04V37u6N9dKCFCwVld+3ZtIw8PLka2G252L/bXaH0O4r6vD9lwAwqo0QQQAsCLIY7wnCmSqCXVpkCq2U/zLFD595Fn0EYigy2r7NBle+80nNbXImHjUiwWnmL8uEMHpM4Ay6yIWsvZyNcWL3lFA2jQmFN04xtzOP3HHYZY6zUATaw81xjLVPXMfmo6TeiolOiLZ+5bN7AiFvtVKuBBUbGur/kCuG8kR0Ek7AlD8TdQWjSmJ94o+igG863xtZtikykTEjl311aPAY8lxfyZfx33D3/QkjUhwOCmqIyzgomGCMXaArTEAkn8MqDHRWHg0WOvm X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(376002)(366004)(346002)(39860400002)(396003)(66556008)(478600001)(66476007)(6486002)(66946007)(8936002)(52116002)(316002)(54906003)(7416002)(38350700001)(83380400001)(7696005)(8676002)(1076003)(107886003)(956004)(186003)(2906002)(26005)(86362001)(6666004)(38100700001)(5660300002)(44832011)(16526019)(4326008)(36756003)(2616005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: VcJ52AaC7yetz9arMy4yYWovaK+yNeuVOeL6NN5HzMVhC6jXAiIPJpqDqetJJX4311fOymSF3WajhPE3+6J0AgFlWekqeYaTlEeiJoW+fBU1BatZWSecsZhPaGzkPYFcc23YYqRqUMEDw1UoYfnjz+riCQxfVFrdz61I/XXlo6Cf/7zvzTw58eoLWbIi2T7IzNP/kn+v9KwMODcttAmTgn02Lm3qjloHwLkGO7DJtv6LX5g5cOvhqaKmwRGDRBApYNtl+jcVUwnlQMji535xZ4BYwDgWqORg9JNxmEaAWyUZeK5LdgEWH+RYJs/5B+yxXUKihs6EzHnB5S4DCpPOgIkk5pKTkVo6mcFcmAtU+L3IokQW+3lpey1Jd3VqwIfW8365jBVMqJgp6wXrVeTsvkiLQkNgJmTN5UXwChBKQmWjbRWq+xnhaUtzBVB4euLPoo1J2xUDHyLmY6SjYLplOryg9LZkEpuG6Uh9c/OkNFogh/R4VCqj1n/MPZIHS8oVTq/eLQ1ra6svDpEcBPRSbHb363uYx8xyypdB4ivzfwcNpCH9pkLlW4lFGRjNeOm+sFbdxGFO27Vf9F7bRsuSKdpPXH+8hnnXWtqXjoYWdhmuULAHRHPm8L0dDm7u4uqeVDTL8PTUhNKMQ0bnP+CRGCUo7oTr+5adQ8Meu7BIvVgAoAdbpf6BEPs18SqgN7xSHTtCR1SM0TouexrixOIP04unNywkPRi11+BccAu7Ms0HEZLC3b3muUwr+97rOwL63dPuZgWNh4BzkfWKBbIauhMs7xZpYEThHNKNgD37G0kK0P5SJZsiu5BN6gD0u/ZH8o5Iub56eLO1secN/8O0DOvpNyRNoPSQ8L9jPOMgRbNhbGYydZBQnoBc5aTN/lRhT1JCglpDjFRgf+TVAPCInDeJynJ0zKOa9xUdZW7YdWgrDgyo/p2BmBbctP75pJerLjcbTRPgtnvJs9n9bZoy7YncPVVBi4xh7X+M5Ih18kGxa1ZVP/8rZhkx/yHCGc9wJMspqLX1TQWK+CpPqc3qp5Bn3xz6hAIa5M9P7AdbAVdZTFn4mmYPnyU7+kD+aeAXglKXWI+6KKN8LeUI08mNO/X3O5nRH4aAD3R3ybAocho40Mlze1+dQnMW8ySi4dJfkv6wTyURBpmPk5u9mij/AUm7MCfhs+QizaWdBmPX7sCaMtVLkpZ/vibxdsm7ZSKfaEOEJ5JsPsUvFQKvBCNC81Dt241q7UnGR3BRDXGO9NSrkRo/b19fTIATy1097YQqrbye/5o+rKy917mDrvBSR1r7Y/p75HywycqLnfavo6iC+Cye7UnGk/QGi0L5Q9Lc X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 48e65f73-890b-438b-3a2a-08d8f886b9e0 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Apr 2021 23:01:21.1543 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Ss47ZGb9MIcCziTA2NqUdoOQ0NKrx4EZxALF9PY2qfJRxcgUrt0sJucDCapBU3Xnrngrkx33HgrG/16pSZuZ6w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB5424 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9945 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 suspectscore=0 adultscore=0 spamscore=0 bulkscore=0 mlxlogscore=999 phishscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104030000 definitions=main-2104050152 X-Proofpoint-GUID: Akm4HZn2E_Q23IrbxnkgX83-r7olEswK X-Proofpoint-ORIG-GUID: Akm4HZn2E_Q23IrbxnkgX83-r7olEswK X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9945 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxscore=0 suspectscore=0 spamscore=0 impostorscore=0 mlxlogscore=999 malwarescore=0 adultscore=0 lowpriorityscore=0 clxscore=1015 priorityscore=1501 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104030000 definitions=main-2104050152 X-Stat-Signature: n4mkn1mnw9qb8tx8ymuy8qg1igeiun4m X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 3E5592000263 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf18; identity=mailfrom; envelope-from=""; helo=userp2120.oracle.com; client-ip=156.151.31.85 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617663717-587783 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: cma_release is currently a sleepable operatation because the bitmap manipulation is protected by cma->lock mutex. Hugetlb code which relies on cma_release for CMA backed (giga) hugetlb pages, however, needs to be irq safe. The lock doesn't protect any sleepable operation so it can be changed to a (irq aware) spin lock. The bitmap processing should be quite fast in typical case but if cma sizes grow to TB then we will likely need to replace the lock by a more optimized bitmap implementation. Signed-off-by: Mike Kravetz Acked-by: Michal Hocko Reviewed-by: David Hildenbrand Acked-by: Roman Gushchin --- mm/cma.c | 18 +++++++++--------- mm/cma.h | 2 +- mm/cma_debug.c | 8 ++++---- 3 files changed, 14 insertions(+), 14 deletions(-) diff --git a/mm/cma.c b/mm/cma.c index f3bca4178c7f..995e15480937 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -24,7 +24,6 @@ #include #include #include -#include #include #include #include @@ -83,13 +82,14 @@ static void cma_clear_bitmap(struct cma *cma, unsigned long pfn, unsigned long count) { unsigned long bitmap_no, bitmap_count; + unsigned long flags; bitmap_no = (pfn - cma->base_pfn) >> cma->order_per_bit; bitmap_count = cma_bitmap_pages_to_bits(cma, count); - mutex_lock(&cma->lock); + spin_lock_irqsave(&cma->lock, flags); bitmap_clear(cma->bitmap, bitmap_no, bitmap_count); - mutex_unlock(&cma->lock); + spin_unlock_irqrestore(&cma->lock, flags); } static void __init cma_activate_area(struct cma *cma) @@ -118,7 +118,7 @@ static void __init cma_activate_area(struct cma *cma) pfn += pageblock_nr_pages) init_cma_reserved_pageblock(pfn_to_page(pfn)); - mutex_init(&cma->lock); + spin_lock_init(&cma->lock); #ifdef CONFIG_CMA_DEBUGFS INIT_HLIST_HEAD(&cma->mem_head); @@ -392,7 +392,7 @@ static void cma_debug_show_areas(struct cma *cma) unsigned long nr_part, nr_total = 0; unsigned long nbits = cma_bitmap_maxno(cma); - mutex_lock(&cma->lock); + spin_lock_irq(&cma->lock); pr_info("number of available pages: "); for (;;) { next_zero_bit = find_next_zero_bit(cma->bitmap, nbits, start); @@ -407,7 +407,7 @@ static void cma_debug_show_areas(struct cma *cma) start = next_zero_bit + nr_zero; } pr_cont("=> %lu free of %lu total pages\n", nr_total, cma->count); - mutex_unlock(&cma->lock); + spin_unlock_irq(&cma->lock); } #else static inline void cma_debug_show_areas(struct cma *cma) { } @@ -454,12 +454,12 @@ struct page *cma_alloc(struct cma *cma, unsigned long count, goto out; for (;;) { - mutex_lock(&cma->lock); + spin_lock_irq(&cma->lock); bitmap_no = bitmap_find_next_zero_area_off(cma->bitmap, bitmap_maxno, start, bitmap_count, mask, offset); if (bitmap_no >= bitmap_maxno) { - mutex_unlock(&cma->lock); + spin_unlock_irq(&cma->lock); break; } bitmap_set(cma->bitmap, bitmap_no, bitmap_count); @@ -468,7 +468,7 @@ struct page *cma_alloc(struct cma *cma, unsigned long count, * our exclusive use. If the migration fails we will take the * lock again and unmark it. */ - mutex_unlock(&cma->lock); + spin_unlock_irq(&cma->lock); pfn = cma->base_pfn + (bitmap_no << cma->order_per_bit); ret = alloc_contig_range(pfn, pfn + count, MIGRATE_CMA, diff --git a/mm/cma.h b/mm/cma.h index 68ffad4e430d..2c775877eae2 100644 --- a/mm/cma.h +++ b/mm/cma.h @@ -15,7 +15,7 @@ struct cma { unsigned long count; unsigned long *bitmap; unsigned int order_per_bit; /* Order of pages represented by one bit */ - struct mutex lock; + spinlock_t lock; #ifdef CONFIG_CMA_DEBUGFS struct hlist_head mem_head; spinlock_t mem_head_lock; diff --git a/mm/cma_debug.c b/mm/cma_debug.c index d5bf8aa34fdc..2e7704955f4f 100644 --- a/mm/cma_debug.c +++ b/mm/cma_debug.c @@ -36,10 +36,10 @@ static int cma_used_get(void *data, u64 *val) struct cma *cma = data; unsigned long used; - mutex_lock(&cma->lock); + spin_lock_irq(&cma->lock); /* pages counter is smaller than sizeof(int) */ used = bitmap_weight(cma->bitmap, (int)cma_bitmap_maxno(cma)); - mutex_unlock(&cma->lock); + spin_unlock_irq(&cma->lock); *val = (u64)used << cma->order_per_bit; return 0; @@ -53,7 +53,7 @@ static int cma_maxchunk_get(void *data, u64 *val) unsigned long start, end = 0; unsigned long bitmap_maxno = cma_bitmap_maxno(cma); - mutex_lock(&cma->lock); + spin_lock_irq(&cma->lock); for (;;) { start = find_next_zero_bit(cma->bitmap, bitmap_maxno, end); if (start >= bitmap_maxno) @@ -61,7 +61,7 @@ static int cma_maxchunk_get(void *data, u64 *val) end = find_next_bit(cma->bitmap, bitmap_maxno, start); maxchunk = max(end - start, maxchunk); } - mutex_unlock(&cma->lock); + spin_unlock_irq(&cma->lock); *val = (u64)maxchunk << cma->order_per_bit; return 0; From patchwork Mon Apr 5 23:00:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12183817 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48EA0C433ED for ; Mon, 5 Apr 2021 23:02:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DC36461205 for ; Mon, 5 Apr 2021 23:02:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DC36461205 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 609C56B0080; Mon, 5 Apr 2021 19:02:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5E3A06B0081; Mon, 5 Apr 2021 19:02:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C2616B0082; Mon, 5 Apr 2021 19:02:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0143.hostedemail.com [216.40.44.143]) by kanga.kvack.org (Postfix) with ESMTP id 18A036B0080 for ; Mon, 5 Apr 2021 19:02:04 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id C2F64180ACEE6 for ; Mon, 5 Apr 2021 23:02:03 +0000 (UTC) X-FDA: 77999838126.21.1FBA997 Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) by imf08.hostedemail.com (Postfix) with ESMTP id 2BB8680192DA for ; Mon, 5 Apr 2021 23:01:57 +0000 (UTC) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 135MxsqS005021; Mon, 5 Apr 2021 23:01:27 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=ztO9NuDcGbIfIeVAyvt8JiBahWGkgMKLgC08KIhDAXk=; b=JqWF5zvhYEjSd9Hmm+kNRCZK3ZiZFGsptHVY1VCkX7XBtVcfvnwO+iRCbGz4sab94nhe olrhLDZq521FNHzokDsj74Io6yiRlbHqTaT7UnehzaLRov2+jbc+8Cp4sECPMO89ajUR UFubPMQGWv50YWlGkwId+0tWMtJvZ3amQjEQBKB56YBmxr0GrqGQC7BkkbhI1e8Xo0q6 xLEngBmO4b3gnKlDsqjqLZR3cdRpAA7HKHgjNIOreXQ2FQfBTukg0kbDZ7w0sOOab9o8 ZAMisL/E5kzkCWOeDP2BxZto2kfwBCgcJVqPEIxcyYyLjJfEzEf25ddvVOymDP/m6mav eg== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by userp2120.oracle.com with ESMTP id 37q3f2bkn6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 05 Apr 2021 23:01:27 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 135MsxPn038332; Mon, 5 Apr 2021 23:01:27 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2047.outbound.protection.outlook.com [104.47.66.47]) by userp3020.oracle.com with ESMTP id 37q2ptgmh1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 05 Apr 2021 23:01:27 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BqIP3AleReQgNvq/miwPwTnGjlb3lPZakP9dT5v2UF1i3rIsIJnjb6fkzic0Og9fx/RA4G2of6Ur3zrDcP49sYR434E2d3LDyqTtH+r+SKKdBqO2LGHfEDjI7p9m9aluZBLADSuNl699gQCOvfHHcKCccBUo0IjIg30XUxooGl6XKrRESYVx2kJV8OEcjPGOFMLEIjVzR3D1ZoSUBbMRMtpDfaei6H5qqOPuS5k/r8zssJRJG16ju1avf3Kum9I/TIuy58dKkj7s78c4svSg7EMd852HwFAp8tHKJqJVa/TyWDmej0FODbSKjoD4OVILVJhuUv3phg/GFkpKqOlc8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ztO9NuDcGbIfIeVAyvt8JiBahWGkgMKLgC08KIhDAXk=; b=Yb8V5/y6fMoaFj1H7qiaJnSAjXKUvhYd8e6SNidWgQqIyCGyc1SdtABat+ZNH/rHFJ52X99KBjm/xX+BaqoQ5X/Nw6YHggIbgoBqZ/MrUXBYD3nS1OdXnPPzi5lzXSwjNXPg8Qn8TO5+y92/5pOikOKpF0ritTCJ9N9QLGSk/LjKEYen0RqeUn/i/VToNnRQ7lVouHC+vuBn9Nxvc7rdIw8ElzEUBuu/9luqisYwZB/qv3uMOEDK5861q274gQzmwazoe8eSaF83fjgfj+Wbj8gaELHPuV9B0o4JDthQ4NBKrpzWb9q58U7c3cTAmHWNBLO8ZuopSFMTqawJ48a8Jw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ztO9NuDcGbIfIeVAyvt8JiBahWGkgMKLgC08KIhDAXk=; b=kC9MuvLkmzZvqxLEoLsheWD0ooEUP5zNgMBXoffpIFzRtm9UzB3/cqjgQuw67hR4CwGJAXMt9LnheQsqnjmpQSuFRYE5XN8h2HriqZ383dWKYhHx4wdoKw75FRWVhEMTDnZToBImJIEi7gc6X+o1jhoSBLQxwGoiJvbpKTbbAh4= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by SJ0PR10MB5424.namprd10.prod.outlook.com (2603:10b6:a03:3bc::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.28; Mon, 5 Apr 2021 23:01:25 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3999.032; Mon, 5 Apr 2021 23:01:25 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Roman Gushchin , Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Hillf Danton , Joonsoo Kim , Barry Song , Will Deacon , Andrew Morton , Mike Kravetz Subject: [PATCH v4 2/8] hugetlb: no need to drop hugetlb_lock to call cma_release Date: Mon, 5 Apr 2021 16:00:37 -0700 Message-Id: <20210405230043.182734-3-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210405230043.182734-1-mike.kravetz@oracle.com> References: <20210405230043.182734-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR18CA0031.namprd18.prod.outlook.com (2603:10b6:320:31::17) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR18CA0031.namprd18.prod.outlook.com (2603:10b6:320:31::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.27 via Frontend Transport; Mon, 5 Apr 2021 23:01:22 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e5c31151-acfe-4462-5a5f-08d8f886bbe0 X-MS-TrafficTypeDiagnostic: SJ0PR10MB5424: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4125; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 6TJaV6WiTW0VN/4iUrdN0y/zOjd1tMfF99gTxgtucwEePEOqEJg73uXV6U6SmUHqCod/mnVeH6ofBBS7BD2DPL4AQbKD4noSPRGH4JZaDt1thxOJE6mArtF6Oq/HRD/Zm/iD+M0rNH68ZvgWi6lcBi6Klr2Pb+5nfH+Pe49d7xNpVeXZsny2uy+js5EENdsu/zk4jRn8c5mA9gwzZhULjswRBAc19EmdiGIq2rNk625gVl1qP8zUl61LX5/+/vZ/jDSAuNkNmYNdd02F9glDEzgSVPms+VVP4QboENNqxf9WeZS3QwWNo2T0B1uIMzTqftJlsdG5T7mG7tXnPDo3ZSOocxAYkqJeFOOHwyrprLLqm2zH2lg6Kz9RL/NcMsF3XVJ7B8PWev82ds74QiBDEGPzVe3pO5zjrCh/NHJCnw1lxtPIe6vL48w9DfT+iiTBJp1oR9zyocOQJIiNFwCX3vxCoLJ+ziQMmg33TfnpOAX60P16/ZWiesEZNKT1mJaOh1jAP15u/FZYEvPHdEcsqIG35IvtPn2Ya+Cbktp0bSPIhGZcFz11i08LfO5JGoLjd6SbapgmA5YuIdkL9fCKU0NvKLyp9LK7sQICw91hZukU6MFRoMKTt6LkrfTV7nT9PDhbNadNGi2l3LSLGe//sZsHQ3lMOTiH/eQmGnB5RJBIEn21r/iA1lpmovIyoIL8 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(376002)(366004)(346002)(39860400002)(396003)(66556008)(478600001)(66476007)(6486002)(66946007)(8936002)(52116002)(316002)(54906003)(4744005)(7416002)(38350700001)(83380400001)(7696005)(8676002)(1076003)(107886003)(956004)(186003)(2906002)(26005)(86362001)(6666004)(38100700001)(5660300002)(44832011)(16526019)(4326008)(36756003)(2616005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: z4be6VyDXc0Er3fTN6P39+6T3LC+CO9r+hitB9Jo2bWZxPfk164GAW27nXJ2ycOZ4eJSD1GHyREcYQwEhopYgg+7+qe2sX79tOlW4qpo7HOmwYXuMzrXplDrhF6AZ0//MQNMHwQ3kWm2/43bgxYUcp1gnSVFTAGdF9i0fBPJwwQs4ioZXIZHRjy5bffk7yi8Wc90psPUANCaOXFTg7bpqCCZerCSxMGFVXHUvP0LWKZ/SB26kb2hAFsJV+7wQNv76ZIHE0XY130fWf3EgAOAMzb/GEMUFsh4rEPEgifFc0110z0xU+gh4Z3gFsP3+DBxRTS5yF2I2xs9g7QRHoccuGmb+KdRKZiiol0dz2KF/KL4cRjUYIdb0CPJJqWsFAWYlFkh/yfJtHjL5L9TrljUlPgFrTJ3XAF4k1/O2CZZao5dvf4waW5A49OKLj1FuoaDjLmjUfpTw9jvERM1Arw28MQBMlOcPHvsnyV5buvDEnsDv8gj7eiEjb84/D20ItScSqAPD2iVgCJsaFFSuOWn17Kui/mPu/3+Pq0sc75kZpmW2eiBOAHKaZLwu+V+Y5XY6vyM1KDpgOik4TFEr1NGzRkUOpSYeDYR5ZTludI2KavtHByNCY+XydnmeteSUuFGZTfUa7Lrp4WgemW4MDtD0emwP5kYYRU3QQtVwh16CwZJpwBDXG/Zbsnc01NHpCoSMoOsr7Rx2tqr8h2qbwy12VeQIBMPr4dvgaLDePyE1T5EN+e8vEliwp90tIAyyPfcF2cZ7Me1QMztrPZGSeSKu9ludo0W9C05cDdMDlicJ7s8aHHo46bZc7zaq/Gmaw66nFDKpz/g76z1NZX8+Qc46f+KQQhuZzwyOF66kmuB5c1ZCMpdy1HhVi1BrNhI3vMwwQQcEVcEy4hZm8R2VO8pzGYCqldzbMdTF3M4FHvJSpJIGHrLHNBDHkuDcmk+rhsOI8eA2DYIh9ExLj/CMpWtTqwhKZVTy2VmvFdgCPgnJlJVqC8GON8SX4EkHNTiKlwSTcW9S0UO7BCYMtJymdf1lBQKfK8DyGZkaRTvmMy1VneYuS+uoTHSt050B9BTNulejS5Q3EpVvsfBZr3t+Hg3exxsGTsDDJ7y/sBE1CiYDLIOjBEYJlUKPIQvzv72GKsxY9c1SBMOh02LdwEbO1jr+Jvafiu1kTsK2sZH5ryth7mAKXgfTNWjZNS+tvPOiCy09uOZOljAOe8N2XxyOs/zXRFOMmhEaVYbUCgQ/CTK9R144Q/X9accr+zJDOHLJBi/tXYv99TXg65UMh2H4G3xG79hgoMbffwMp1dKjos+MStTgSsuV00qXu3apCflBc1g X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: e5c31151-acfe-4462-5a5f-08d8f886bbe0 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Apr 2021 23:01:25.0094 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: iRJ0DA6pdq8ynLklDDKubBAgYyVv2ipDdWhxGisNbI2zVEKZtcDLTAlbhSOwgIqodnczcLDlwYP964/lsvt72Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB5424 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9945 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 mlxscore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104030000 definitions=main-2104050152 X-Proofpoint-GUID: N-4FeMF-y2OmR0sLQ0ti77JjdjJ3hZ0b X-Proofpoint-ORIG-GUID: N-4FeMF-y2OmR0sLQ0ti77JjdjJ3hZ0b X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9945 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxscore=0 suspectscore=0 spamscore=0 impostorscore=0 mlxlogscore=999 malwarescore=0 adultscore=0 lowpriorityscore=0 clxscore=1015 priorityscore=1501 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104030000 definitions=main-2104050152 X-Rspamd-Queue-Id: 2BB8680192DA X-Stat-Signature: aj5fomtttdejb9bu4z3xtent46psad1t X-Rspamd-Server: rspam02 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf08; identity=mailfrom; envelope-from=""; helo=userp2120.oracle.com; client-ip=156.151.31.85 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617663717-331021 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that cma_release is non-blocking and irq safe, there is no need to drop hugetlb_lock before calling. Signed-off-by: Mike Kravetz Acked-by: Roman Gushchin Acked-by: Michal Hocko Reviewed-by: Oscar Salvador Reviewed-by: David Hildenbrand --- mm/hugetlb.c | 6 ------ 1 file changed, 6 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 3c3e4baa4156..1d62f0492e7b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1353,14 +1353,8 @@ static void update_and_free_page(struct hstate *h, struct page *page) set_compound_page_dtor(page, NULL_COMPOUND_DTOR); set_page_refcounted(page); if (hstate_is_gigantic(h)) { - /* - * Temporarily drop the hugetlb_lock, because - * we might block in free_gigantic_page(). - */ - spin_unlock(&hugetlb_lock); destroy_compound_gigantic_page(page, huge_page_order(h)); free_gigantic_page(page, huge_page_order(h)); - spin_lock(&hugetlb_lock); } else { __free_pages(page, huge_page_order(h)); } From patchwork Mon Apr 5 23:00:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12183805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05172C43460 for ; Mon, 5 Apr 2021 23:02:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 75791613BD for ; Mon, 5 Apr 2021 23:01:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 75791613BD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D6F786B0075; Mon, 5 Apr 2021 19:01:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D0AE56B0073; Mon, 5 Apr 2021 19:01:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 987D16B0075; Mon, 5 Apr 2021 19:01:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0251.hostedemail.com [216.40.44.251]) by kanga.kvack.org (Postfix) with ESMTP id 7BD5C6B0073 for ; Mon, 5 Apr 2021 19:01:57 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 26A00180275A4 for ; Mon, 5 Apr 2021 23:01:57 +0000 (UTC) X-FDA: 77999837874.10.B118241 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf18.hostedemail.com (Postfix) with ESMTP id 3C56E2000380 for ; Mon, 5 Apr 2021 23:01:56 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 135MxSsq005148; Mon, 5 Apr 2021 23:01:31 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=M3ZNCNlNGxs+XGnmLc2aGc0QhYl50TNtgEytgMXfYic=; b=O/XRvjMbzvSwJzLqEdkFfcbq5xH6n5p5/yyxExTHKcuG8nttQU0v/fHSpICNklsWky9n XnUUeAOIIWvKRkYNwo+Us9i0chy6RCKsFJO5Bn8IolOsmhRmmwrvVCixMueRd7xYRO/q VUWeY6xHCqFQPRMMiJ95YEGby0G20TAjqRqilfgKHm037PaACJKFYnZ7Cd/fxf3SZd9e qv1lK3XxpDmATaMDD+Mii8EPC57N1lt1LgKK55rKNUc8v5wJgUoVHYl0dYYsRCoHZegv +Ut2Zu8TRhTjIhw4mDn7p1n2x2CPKmC27M1hb7AjLkgxL7YKAIkziRnahPUxHxuMMyLj cg== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by aserp2130.oracle.com with ESMTP id 37q38mkkb1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 05 Apr 2021 23:01:31 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 135MtBJt020353; Mon, 5 Apr 2021 23:01:30 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2168.outbound.protection.outlook.com [104.47.58.168]) by aserp3020.oracle.com with ESMTP id 37q2pwt11c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 05 Apr 2021 23:01:30 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TCOYGVM7qDQsnvXmMHDAE/DiT55B135GYQuhQD9aS8EQHNSoj8n1wP84J4yvCz3Wbq1iHAkfgI6LQR9UO4IFWhRkz7MMapOE4je4d8J8Jcn5+YJ9OE/geeOeaq+Ee0Ab/GRqopiOVrNvwWnIfI85/CzvrNjbqWL0TBG8yiDRL+3xRP4Mn1ykNfckWgVSCbTgIBVzpxbLB9L8npxebSTS+T4yi9yTC3ZD2urgPReZy2dPQ8idLm4P9cpJOKyj/HcKBLoxYNL1HDM06HlHagMVmCOjouBLuxJUhN33Xy4wszE+zHi50h0znEM+GePgBLGAzpRMmXOn2WpRbqQbkdO6nw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=M3ZNCNlNGxs+XGnmLc2aGc0QhYl50TNtgEytgMXfYic=; b=fr0i8LERb6DL+biRKZ4p3VeEHvCmaEQtuVpvb45LHolRlwwVjnOxIvWRsrMgM2LG94TH9EtvPKmVo6D3FwlDdloCpXLlRYMu419Td8xoKPsMWMX1+TrYFJwrqBNJBlCseyi+22w1oDfUYfZbPT9VQHx/TViPY/f9bZi2ZZPy78ZvqcM+IK9vW5uXdd9B/JRF1vq8kcxHRyeUR5wDhDFs+I+wmtEWtsAhvmLyw1ypTZQqJ65bufDma+YTpeKGD9F+nfvHKzT6OQP3TAYLTE/hJsaDwLK/VQTcCwkryme6slBHm0AsE6IwmIK0A/e+fZjkz2mtCYpkXJp4acTQhIuB8Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=M3ZNCNlNGxs+XGnmLc2aGc0QhYl50TNtgEytgMXfYic=; b=Vky81WxyBzPfq5vWHRnQTjR5d88sPE5OK9F1hD0QQZZFbFGgrFCNBevNWqPbwnweF6KxrLZdj57+u+/e98H9NcTYpcTgFxRMuZZ3NlK2VqDdNADPaet0teKHpholnuhbqaVYXyafHPRgq44yeL4/i4k7VOC0DLj+sa9KDwqGxF0= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3544.namprd10.prod.outlook.com (2603:10b6:a03:121::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.26; Mon, 5 Apr 2021 23:01:28 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3999.032; Mon, 5 Apr 2021 23:01:28 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Roman Gushchin , Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Hillf Danton , Joonsoo Kim , Barry Song , Will Deacon , Andrew Morton , Mike Kravetz Subject: [PATCH v4 3/8] hugetlb: add per-hstate mutex to synchronize user adjustments Date: Mon, 5 Apr 2021 16:00:38 -0700 Message-Id: <20210405230043.182734-4-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210405230043.182734-1-mike.kravetz@oracle.com> References: <20210405230043.182734-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR18CA0031.namprd18.prod.outlook.com (2603:10b6:320:31::17) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR18CA0031.namprd18.prod.outlook.com (2603:10b6:320:31::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.27 via Frontend Transport; Mon, 5 Apr 2021 23:01:27 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 630bb9f4-5b6a-441b-8741-08d8f886be86 X-MS-TrafficTypeDiagnostic: BYAPR10MB3544: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 0wZlFccx04Lvetmi6PijuDNgoRIGEahFT3lhXw4S9mj+4n0R7tBTBhbiqYqbyhcbaaEEuG3XLY7mFNRtjrnQ+XJ1dxvumP7sFKDUnrgM1rh0xfM8Oxg5UX8Dyso7EfUYTJbtFDSq+JfzexCYs82oIAeRZE7LYCTVU5E9UdEhh59Sm+DO3C9ADobsE9irxgSbb1UYFyOkbL1c6sqHubX3SBNLiDLbDaRfUOle37Q58vdIrhjvYl05yXfiXMBDySG60f+0171Ll2TjqhUrFmM88y96H97rDejfzSSTdD2VWEplY9pNouThs/Tv+I685s8MoZIAKcyPzgmDQGY3/9gkgjuhBYvWaeu+uboDg7Y3jCJoNRfWwVU2mfBcCIOW4x8BUMeab3O25hRjz+PVIHctKoPjQez5kGDuJfccFioACAirYG+0us5TrW3Bcd97eOk4dBVyhIOkAhcxCpPwbOT/arw8itbeXdxzIoPDhSAjHciYy+7Fh+2TxZnU+CbF9B5/Nck5gYUCIJGoSSOi00oZ/m+KTVlNUqH+P++MBYr3eAxKeCm1wmBQqYPLeb4Z/nCQzGNvrFB916KnGZ+f0VeQzf7+XX4m0hg2IzLLfO/cgbHZ0ESjd/gu+3vDKuCDDqaMNy/aE5y/Xp59eYO8muZk3PXBC5ZhIYnxVTw+C9bSfWrGD2xWBV9ZpFrVvTfvaG0Q X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(396003)(376002)(366004)(39860400002)(346002)(2616005)(44832011)(66476007)(5660300002)(107886003)(8676002)(86362001)(26005)(6486002)(66946007)(16526019)(38350700001)(1076003)(2906002)(54906003)(83380400001)(4326008)(52116002)(7696005)(186003)(7416002)(478600001)(38100700001)(8936002)(36756003)(316002)(66556008)(6666004)(956004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 1+DD4bLsEC/X1g+vQOhz958tT9Zm/CK/VpxRGWxq67FAXLjbPwNYuCB+D5JSsG279vhEO5ZeWsghcZvwhSh75TcQlixKuhfHaIJPRqqo9WfG3qFeKh89nlLL6hKXJF6g8oXdYUKGk1IGjYtLJjGkHq36ER4JuizzH9Z3lU3wwLrIxJG4o8flvEifmnVadmPKsvDxsIiGQiunyPnhfSahsHoE911d88rTya82YMIeSTIqI6a9PgWv73i338PmQmjUBmSRTS2zEtZrJTXxfb6ost3Z8D4+HKLjMUEma1bW709Xy1xfLtOkQeQucFp6KgpprdhYqN6tIDXTWKIL5hlbpBkG27g4bbZtjAonJSCBmtw1FTB37aORq2GAaK2WqlpbZ+591rZiLEbO5mSd84cWlwpQ1mD9SYImBMkMe/fvF3KoFibvElG6RVeHm3VsvmP8M2330HY3TQ7URiz+V3c54LrmBr6VTOAUhCOmzI1JHbHNYbYySRWYsDY1V5Ub1ny0KPxX5g1rmV5x108bH97gsJT0pH+JNk+ULgYCoAEJxeKwHorS+tn+BTJ2HwIMFpigCfJVYeuNRTSEc3pay2flmEdrrQq5PoRpQWkDIMsj6xDcViuBewWr+x+1tnwiaZSlkS3yQ0FeKw8caX60sIvDbBXaHPoQiZcf3VcxwBLlQGe31mopZfNlqDAO8S0znL+9NSpdEgS5rQ0ttT34nfSAm3mYb23FHv/zo8MAL5WA/SA7CSpjxuzqZMlQ2GX0zgvJQw4HI+ltvFxcYbLAG0oKjZFouScr0k6QFbRrHQ2loqrMTDwk/vyJFpoYIht4fa3LSAh+48Rh20NaUwsd/nt4AKUfpxQBLVCcCC2lw4wMURs3EQ00AEdgEXgQbqwnnl4gwRZbQO4HujWAQifrDT2wS4BFjBINDz8Th7pbh0QGRwH20jClXbIZLkdHJYlzhKNhEzzIuzQN/cTH9273ONqP9gampNj7e9GoEbkNm9ckGDwFIV6S5d+7KHnS/ASYnbISkHpdmy2A0Jxa5nxO8x+hKsiyTw6wOuKOzyEjrKgMyNQNLOouYFYMzE1+TUlHRErw4753Ks/7JExtQinJl3uhK9Z9jXqk8YrxqKntHmvPvQNo3Opsl4Qjlbu66WKEVAESBMHHsmh6f6Mb5q1Qg4x71Hb3bkeiXacgoRU45XAlG0Pw28I8F/0BjDEMHvoN/hHCR2UP4qQKGd7d7L7N/TiL0uI0qQnn6KJI/jgVWjETAS75ffaPune7I2x95k0atwsQAn4nFBhLA2CFG/Rb+tABBJU3w2WjdUoOZqTLSjcq7K8PWyeQDG7jSXLBoX/3k7A9 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 630bb9f4-5b6a-441b-8741-08d8f886be86 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Apr 2021 23:01:28.0381 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: t1Yj43KlRK+Q0WPh6xjsxbcuZsatxwrY/1pmfiLZwxcinqrtp8Gk80YI+fU5KlSidEOvVIUlmCosPs8XBWlhxA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3544 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9945 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 suspectscore=0 adultscore=0 spamscore=0 bulkscore=0 mlxlogscore=999 phishscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104030000 definitions=main-2104050152 X-Proofpoint-GUID: LM4wrkCTCJTbQIi_vM2T_y-vUYRofiGK X-Proofpoint-ORIG-GUID: LM4wrkCTCJTbQIi_vM2T_y-vUYRofiGK X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9945 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 clxscore=1015 suspectscore=0 impostorscore=0 malwarescore=0 priorityscore=1501 spamscore=0 mlxscore=0 bulkscore=0 lowpriorityscore=0 adultscore=0 phishscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104030000 definitions=main-2104050152 X-Stat-Signature: teqp57sbraa75r8bg55zg4b8hbqsq47y X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 3C56E2000380 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf18; identity=mailfrom; envelope-from=""; helo=aserp2130.oracle.com; client-ip=141.146.126.79 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617663716-170219 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The helper routine hstate_next_node_to_alloc accesses and modifies the hstate variable next_nid_to_alloc. The helper is used by the routines alloc_pool_huge_page and adjust_pool_surplus. adjust_pool_surplus is called with hugetlb_lock held. However, alloc_pool_huge_page can not be called with the hugetlb lock held as it will call the page allocator. Two instances of alloc_pool_huge_page could be run in parallel or alloc_pool_huge_page could run in parallel with adjust_pool_surplus which may result in the variable next_nid_to_alloc becoming invalid for the caller and pages being allocated on the wrong node. Both alloc_pool_huge_page and adjust_pool_surplus are only called from the routine set_max_huge_pages after boot. set_max_huge_pages is only called as the reusult of a user writing to the proc/sysfs nr_hugepages, or nr_hugepages_mempolicy file to adjust the number of hugetlb pages. It makes little sense to allow multiple adjustment to the number of hugetlb pages in parallel. Add a mutex to the hstate and use it to only allow one hugetlb page adjustment at a time. This will synchronize modifications to the next_nid_to_alloc variable. Signed-off-by: Mike Kravetz Acked-by: Michal Hocko Reviewed-by: Oscar Salvador Reviewed-by: Miaohe Lin Reviewed-by: Muchun Song Reviewed-by: David Hildenbrand --- include/linux/hugetlb.h | 1 + mm/hugetlb.c | 8 ++++++++ 2 files changed, 9 insertions(+) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index d9b78e82652f..b92f25ccef58 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -566,6 +566,7 @@ HPAGEFLAG(Freed, freed) #define HSTATE_NAME_LEN 32 /* Defines one hugetlb page size */ struct hstate { + struct mutex resize_lock; int next_nid_to_alloc; int next_nid_to_free; unsigned int order; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 1d62f0492e7b..8497a3598c86 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2730,6 +2730,11 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, else return -ENOMEM; + /* + * resize_lock mutex prevents concurrent adjustments to number of + * pages in hstate via the proc/sysfs interfaces. + */ + mutex_lock(&h->resize_lock); spin_lock(&hugetlb_lock); /* @@ -2762,6 +2767,7 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, if (hstate_is_gigantic(h) && !IS_ENABLED(CONFIG_CONTIG_ALLOC)) { if (count > persistent_huge_pages(h)) { spin_unlock(&hugetlb_lock); + mutex_unlock(&h->resize_lock); NODEMASK_FREE(node_alloc_noretry); return -EINVAL; } @@ -2836,6 +2842,7 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, out: h->max_huge_pages = persistent_huge_pages(h); spin_unlock(&hugetlb_lock); + mutex_unlock(&h->resize_lock); NODEMASK_FREE(node_alloc_noretry); @@ -3323,6 +3330,7 @@ void __init hugetlb_add_hstate(unsigned int order) BUG_ON(hugetlb_max_hstate >= HUGE_MAX_HSTATE); BUG_ON(order == 0); h = &hstates[hugetlb_max_hstate++]; + mutex_init(&h->resize_lock); h->order = order; h->mask = ~(huge_page_size(h) - 1); for (i = 0; i < MAX_NUMNODES; ++i) From patchwork Mon Apr 5 23:00:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12183813 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E105BC433ED for ; Mon, 5 Apr 2021 23:02:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3FE77613BC for ; Mon, 5 Apr 2021 23:02:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3FE77613BC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 830F16B007D; Mon, 5 Apr 2021 19:02:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7681B6B0081; Mon, 5 Apr 2021 19:02:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 56C006B0080; Mon, 5 Apr 2021 19:02:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0104.hostedemail.com [216.40.44.104]) by kanga.kvack.org (Postfix) with ESMTP id 31E686B007D for ; Mon, 5 Apr 2021 19:02:03 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id E570A81DC for ; Mon, 5 Apr 2021 23:02:02 +0000 (UTC) X-FDA: 77999838084.20.95AE791 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf30.hostedemail.com (Postfix) with ESMTP id 8A5C3E00011E for ; Mon, 5 Apr 2021 23:01:57 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 135Mwvh1004925; Mon, 5 Apr 2021 23:01:35 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=u6d7WLtJ7GClBEpNkul0k+BxSZweslX8faM0um+cJiI=; b=qLgp8+21S5AJF8RE5fv1Y5P7+/hh9nyfXJUxEsHYYqCnAQwLf8pBtMzUf0D4yvZFxAuK AxbksRvQ0xvcmXz8/EekuaSqJ71rjOca+O5GkK22m5igN+LeThZhWi6TiJF7u3ga88vu cxlbcUm7KJgjVbK1irkA3xpixvEEsvBLPKuCacbuBv43efRc7WJC/zoHEDM/JAndJNTs vpxZHN3vJ+M4yVix1u8YIIBbuWijD8Kplv3W1Qomz70HUw+zCB6JwK0sVNyjtYKt6srv SUaM59lMW9NynoxiLsTJk3aw2Hdz1sTuMF2gx0M7b2dOTnCW0c3y6NQ1r+LMK9ihWqhn LA== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by aserp2130.oracle.com with ESMTP id 37q38mkkb5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 05 Apr 2021 23:01:35 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 135Msxpd038292; Mon, 5 Apr 2021 23:01:34 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2173.outbound.protection.outlook.com [104.47.58.173]) by userp3020.oracle.com with ESMTP id 37q2ptgmm9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 05 Apr 2021 23:01:34 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MbBAlpvhJzf6iic13BPvWKUH+RxEqmvSpCIephOs8axKY4S7JAXBJxYnZ40+7xvwkqZ53WsLw/7JH8AUx3MN0YNg6QGqjr3gQAQ/WysvXCSwn2tWqo7VhtK+fmgeumu6TEd41wSv7QPAeFKwBUgVcQGfLfvQdBvqfEDwc48ySXQRakflM/MGfsgOLEY5zpdDUstfJ3MBatB6m9A3FlmRVOb9ioTfTd4mKz62Cbr0F59aM3IPvYguSHkNmOqbbYjaJ93fnpdY3KGXI3u0530V+9k1KE7G0T15AxJCHHdhOoiXEeMYQNzScWKCcSi2yE8wavYwNHyvxGiK7bafgOB4JQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=u6d7WLtJ7GClBEpNkul0k+BxSZweslX8faM0um+cJiI=; b=KrJYbmOJUqS7WopbxxcCrGIw21JoCMrIPNNyz6dSCdfLTnM3RMnZ2cwXy2ndBJip3aCQ35hPnlwd07eAT5ehYPnLGD9oVbV0kxbEu3WjCyWG9j3KCmMG3oTph3tr8JVe0iQJbSG79MfYCN62I0QG/MaDfpkKhYjKpi3+DxVY/vG60VxZbmRnI+CeKTA74NbhgNNw4qDAfe7dkb5txEBjShynps4edGAknI1nqevd2tb2cqekC5IBfVHghuKlXklUvzUbOWUqFh/dW47PVelO9iOW+rdUJsDBvEPIu9FjRBrvvnizPt2Wm6lMK3A20UTWvm6EN2fh99HiNnPMLI2acA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=u6d7WLtJ7GClBEpNkul0k+BxSZweslX8faM0um+cJiI=; b=q1gnMVfrGcaTxnH22Wo5LwjfkSvDxEpxGBSr1gUpjRMHRvF6XsBrxAyUYaL69a15+cowtgYDqArP5G9YMIQcOtP5vMV4wJ71edJrLsPBSCYkeZir0nYtXOv7MoOOHUmxVAIOYrumZVY5M0vs+yXrHZXj/DI5TIiXRbo5SeIuHGY= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3544.namprd10.prod.outlook.com (2603:10b6:a03:121::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.26; Mon, 5 Apr 2021 23:01:31 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3999.032; Mon, 5 Apr 2021 23:01:31 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Roman Gushchin , Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Hillf Danton , Joonsoo Kim , Barry Song , Will Deacon , Andrew Morton , Mike Kravetz Subject: [PATCH v4 4/8] hugetlb: create remove_hugetlb_page() to separate functionality Date: Mon, 5 Apr 2021 16:00:39 -0700 Message-Id: <20210405230043.182734-5-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210405230043.182734-1-mike.kravetz@oracle.com> References: <20210405230043.182734-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR18CA0031.namprd18.prod.outlook.com (2603:10b6:320:31::17) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR18CA0031.namprd18.prod.outlook.com (2603:10b6:320:31::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.27 via Frontend Transport; Mon, 5 Apr 2021 23:01:29 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 00019ecc-7411-4732-fd82-08d8f886c00a X-MS-TrafficTypeDiagnostic: BYAPR10MB3544: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:37; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: S/+sHOgvi40j0pk5/QUdf0Exy3QKYogFQYeSO1hiG1OB1pR91B1t7RZOnD2rlEx5PyGew4nOjhT6mjbGdvLygsAYVRPyN6gxboxnqsQl72CgNA+DIXr8IbyBrfZck/0IrfJ6Ta+LJ+2X0McGLGC97PY0aYFt0qF8vXFVCgTVAZDol/c7pZpbcjF7i8GOMJVTleMwD4fmYR4ULHFZV5B0CG7lgew3IeTMabH/9HAsUXq8e4rR8W6E2JNIWAMQz4JfF19f8qIeNtknSi9Q0VO2VLbSIuByi77K2J2FTmc09t9IrQ0VfLTQU4U4223bAgfvXetP0UdTpzXK0oTSYt+fbEHCpHBNhm+4db1tc660l1f5DjWKDZYqT/dNFPmmkRNsykeNpl2b6z6xp1dzIhLUzY3FATPVoUH9j0UMlZkut5ii9xxpM+q2gZu9LcJun2L4BV7x2xZVcIycrj7ruhVAHex1Lg4GNgd+Tu1oLjj3UbhAc+8mWuUxscdk3TSj7l7Av6Tb//X3zdkKxKtmP037oR8ZNFdvY0uWsRqKO8TePx6zkH/f+aRk0sFX/4y1APL7+YvItVv1EvWx/Dur2A1VLL7XG07NYq7QYDyPcpvDKc8wW234k/eUbLVwy80sscWPV38A4EKZcdbDVFtbmdl7pGd91jmer6d/k87afMDjDX8N86orhd1XAibWFIeZxUHO X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(396003)(376002)(366004)(39860400002)(346002)(2616005)(44832011)(66476007)(5660300002)(107886003)(8676002)(86362001)(26005)(6486002)(66946007)(16526019)(38350700001)(1076003)(2906002)(54906003)(83380400001)(4326008)(52116002)(7696005)(186003)(7416002)(478600001)(38100700001)(8936002)(36756003)(316002)(66556008)(6666004)(956004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: /Mpy6+53BEQ27LC/yZPZe8UulK3bOGThAaT4xYMGL6P9vkzGCe7sn/qL4ScRFHGObZ2t9/9hXNCkAUy7CSOjfZP9vO12XsCkVpc6wX8UYhX66aR56SZdRNxmUiD/k5eX9Sns1uq6Disl0tsxt5lVeEuOKpuvVezoMVvL6Fbp+tVoUeJ5FVfT6hsXINU0RFakTcLeQFjvxmD7KkBcPeMp3W7YTWU4wIcOiaRxNRtJdPCTvOKI0nNaZfdhRZeNQwbDE/eFum4gm8ORV7i8AjzQxT8hL3jKedvdL2MnNFwpAL/6ZvbGORG/MIHvQG4NvfAMtFLb2hdQjsjBI8qa6N0kcNplYOoqPNcGr41vvvuw0G2yYnl+9PNjX1KJjns1w5cbpKBrjvkUnDBS4Ts2mkeKMi/3vlOFC2hlXwUK2n426S3zOszhEJ+G8v3iVLhwYt23TrOZ+YkF5WFKa+9LLXnLXycXbWB23n/JAmcaOVNz4sQi6snT9xUtUyAOdTV/WPzAUPGYDjqfEJ5Fcc8u/PaWKGvTeo0anu5cWzU1gJvzr6kqiDu1zZeo+i1/fnrWJ8lKmGBdokldV86693wlCcrqRHsquN5nxYxPJ3vkuVGDWJoz8F6tyzNmvAGisfig9O9pIKY+5diFos+VVOHLHpDd1ONM9A9U0YOb+0jOp8mpNeS/0aMnBJVK8vOvYMYiMHAX82RMVLw2zpovOFpzxRp+Pn+PUWuXhgEvKgICJy8jEB8dDkkT+/lOmgZYrmKy5o0TlGS0Dfy61FSLRSmMH/wsIGHpJJKLWCQwRXyi/w473CrbCbl92f4UDbYAdNJbCVt+gPI9Q34B/pDCz2BcrcikeCqgmhODTlVTIRuMOsFFS1ZXg+RRpwAblQEkURh6cmw528mTHnNWTObgKyBpG+l5ka1d5JBewlH8qPxpi0z+VdVw6wcNJ6p4QyrBDdS8Sg2f4cvZOwnF7a+kRYvwiplZGHw8Qs11GPNJTSHM8hCrwSeDcwHOJsWZ3l+50YOiYK2m7JY6gvTudv97UvVXihpipQqsznQPsfg+Cu5Do/Vd5gp5dsc9dxDly5FEpVp1edwPpYRPqoVjyOAkC5K5+oi2llJ58zJ0hri5bq9hKr45B4ECUOCQsUjHt2ruWDcCG8i4QlWoezs8bjeo23KgIpGo2BL/p16Q71kO+bv1oFi0qSDN73TfIwPFS9NBebZyyAXZRBJwZ+s763mwlzATvGkE+blbPxDItW5Tu1y3ctq5prAlGan2UEF4Yn6yVo7qCpOGKXGpFCUOYq+tkepuIfssPYw9AeKNTAvex4Wk3SKKTyQEp67a3UZp/Mjp3W0EPGjO X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 00019ecc-7411-4732-fd82-08d8f886c00a X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Apr 2021 23:01:31.2371 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 6ezYx3Pd6VNAiQnNe6WreSh7VoGyYZj1PFiODq5zADCHCTkabqUugWSar4lCNKVya96vDKQUVjdEBbkjEH9Olw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3544 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9945 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 mlxscore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104030000 definitions=main-2104050152 X-Proofpoint-GUID: Anhw9X2tXfRnmDTDmHBcVee3z_rqUne9 X-Proofpoint-ORIG-GUID: Anhw9X2tXfRnmDTDmHBcVee3z_rqUne9 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9945 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 clxscore=1015 suspectscore=0 impostorscore=0 malwarescore=0 priorityscore=1501 spamscore=0 mlxscore=0 bulkscore=0 lowpriorityscore=0 adultscore=0 phishscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104030000 definitions=main-2104050152 X-Rspamd-Queue-Id: 8A5C3E00011E X-Stat-Signature: dooky9uq6ypah75raderm76sjp1zk4ru X-Rspamd-Server: rspam02 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf30; identity=mailfrom; envelope-from=""; helo=aserp2130.oracle.com; client-ip=141.146.126.79 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617663717-327874 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The new remove_hugetlb_page() routine is designed to remove a hugetlb page from hugetlbfs processing. It will remove the page from the active or free list, update global counters and set the compound page destructor to NULL so that PageHuge() will return false for the 'page'. After this call, the 'page' can be treated as a normal compound page or a collection of base size pages. update_and_free_page no longer decrements h->nr_huge_pages{_node} as this is performed in remove_hugetlb_page. The only functionality performed by update_and_free_page is to free the base pages to the lower level allocators. update_and_free_page is typically called after remove_hugetlb_page. remove_hugetlb_page is to be called with the hugetlb_lock held. Creating this routine and separating functionality is in preparation for restructuring code to reduce lock hold times. This commit should not introduce any changes to functionality. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 88 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 51 insertions(+), 37 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 8497a3598c86..df2a3d1f632b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1055,18 +1055,13 @@ static bool vma_has_reserves(struct vm_area_struct *vma, long chg) return false; } -static void __enqueue_huge_page(struct list_head *list, struct page *page) -{ - list_move(&page->lru, list); - SetHPageFreed(page); -} - static void enqueue_huge_page(struct hstate *h, struct page *page) { int nid = page_to_nid(page); - __enqueue_huge_page(&h->hugepage_freelists[nid], page); + list_move(&page->lru, &h->hugepage_freelists[nid]); h->free_huge_pages++; h->free_huge_pages_node[nid]++; + SetHPageFreed(page); } static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) @@ -1331,6 +1326,43 @@ static inline void destroy_compound_gigantic_page(struct page *page, unsigned int order) { } #endif +/* + * Remove hugetlb page from lists, and update dtor so that page appears + * as just a compound page. A reference is held on the page. + * + * Must be called with hugetlb lock held. + */ +static void remove_hugetlb_page(struct hstate *h, struct page *page, + bool adjust_surplus) +{ + int nid = page_to_nid(page); + + if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported()) + return; + + list_del(&page->lru); + + if (HPageFreed(page)) { + h->free_huge_pages--; + h->free_huge_pages_node[nid]--; + ClearHPageFreed(page); + } + if (adjust_surplus) { + h->surplus_huge_pages--; + h->surplus_huge_pages_node[nid]--; + } + + VM_BUG_ON_PAGE(hugetlb_cgroup_from_page(page), page); + VM_BUG_ON_PAGE(hugetlb_cgroup_from_page_rsvd(page), page); + + ClearHPageTemporary(page); + set_page_refcounted(page); + set_compound_page_dtor(page, NULL_COMPOUND_DTOR); + + h->nr_huge_pages--; + h->nr_huge_pages_node[nid]--; +} + static void update_and_free_page(struct hstate *h, struct page *page) { int i; @@ -1339,8 +1371,6 @@ static void update_and_free_page(struct hstate *h, struct page *page) if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported()) return; - h->nr_huge_pages--; - h->nr_huge_pages_node[page_to_nid(page)]--; for (i = 0; i < pages_per_huge_page(h); i++, subpage = mem_map_next(subpage, page, i)) { subpage->flags &= ~(1 << PG_locked | 1 << PG_error | @@ -1348,10 +1378,6 @@ static void update_and_free_page(struct hstate *h, struct page *page) 1 << PG_active | 1 << PG_private | 1 << PG_writeback); } - VM_BUG_ON_PAGE(hugetlb_cgroup_from_page(page), page); - VM_BUG_ON_PAGE(hugetlb_cgroup_from_page_rsvd(page), page); - set_compound_page_dtor(page, NULL_COMPOUND_DTOR); - set_page_refcounted(page); if (hstate_is_gigantic(h)) { destroy_compound_gigantic_page(page, huge_page_order(h)); free_gigantic_page(page, huge_page_order(h)); @@ -1419,15 +1445,12 @@ static void __free_huge_page(struct page *page) h->resv_huge_pages++; if (HPageTemporary(page)) { - list_del(&page->lru); - ClearHPageTemporary(page); + remove_hugetlb_page(h, page, false); update_and_free_page(h, page); } else if (h->surplus_huge_pages_node[nid]) { /* remove the page from active list */ - list_del(&page->lru); + remove_hugetlb_page(h, page, true); update_and_free_page(h, page); - h->surplus_huge_pages--; - h->surplus_huge_pages_node[nid]--; } else { arch_clear_hugepage_flags(page); enqueue_huge_page(h, page); @@ -1712,13 +1735,7 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, struct page *page = list_entry(h->hugepage_freelists[node].next, struct page, lru); - list_del(&page->lru); - h->free_huge_pages--; - h->free_huge_pages_node[node]--; - if (acct_surplus) { - h->surplus_huge_pages--; - h->surplus_huge_pages_node[node]--; - } + remove_hugetlb_page(h, page, acct_surplus); update_and_free_page(h, page); ret = 1; break; @@ -1756,7 +1773,6 @@ int dissolve_free_huge_page(struct page *page) if (!page_count(page)) { struct page *head = compound_head(page); struct hstate *h = page_hstate(head); - int nid = page_to_nid(head); if (h->free_huge_pages - h->resv_huge_pages == 0) goto out; @@ -1787,9 +1803,7 @@ int dissolve_free_huge_page(struct page *page) SetPageHWPoison(page); ClearPageHWPoison(head); } - list_del(&head->lru); - h->free_huge_pages--; - h->free_huge_pages_node[nid]--; + remove_hugetlb_page(h, page, false); h->max_huge_pages--; update_and_free_page(h, head); rc = 0; @@ -2298,6 +2312,7 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, /* * Freed from under us. Drop new_page too. */ + remove_hugetlb_page(h, new_page, false); update_and_free_page(h, new_page); goto unlock; } else if (page_count(old_page)) { @@ -2305,6 +2320,7 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, * Someone has grabbed the page, try to isolate it here. * Fail with -EBUSY if not possible. */ + remove_hugetlb_page(h, new_page, false); update_and_free_page(h, new_page); spin_unlock(&hugetlb_lock); if (!isolate_huge_page(old_page, list)) @@ -2323,13 +2339,13 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, /* * Ok, old_page is still a genuine free hugepage. Replace it * with the new one. + * Note: h->free_huge_pages{_node} counters are decremented + * in remove_hugetlb_page for old_page and incremented in + * enqueue_huge_page for new page. Net result is no change. */ - list_del(&old_page->lru); + remove_hugetlb_page(h, old_page, false); update_and_free_page(h, old_page); - /* - * h->free_huge_pages{_node} counters do not need to be updated. - */ - __enqueue_huge_page(&h->hugepage_freelists[nid], new_page); + enqueue_huge_page(h, new_page); } unlock: spin_unlock(&hugetlb_lock); @@ -2667,10 +2683,8 @@ static void try_to_free_low(struct hstate *h, unsigned long count, return; if (PageHighMem(page)) continue; - list_del(&page->lru); + remove_hugetlb_page(h, page, false); update_and_free_page(h, page); - h->free_huge_pages--; - h->free_huge_pages_node[page_to_nid(page)]--; } } } From patchwork Mon Apr 5 23:00:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12183807 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54794C433ED for ; Mon, 5 Apr 2021 23:02:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CAA53613D6 for ; Mon, 5 Apr 2021 23:02:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CAA53613D6 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 08CB16B0073; Mon, 5 Apr 2021 19:01:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F2CEF6B007D; Mon, 5 Apr 2021 19:01:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C20616B007E; Mon, 5 Apr 2021 19:01:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0112.hostedemail.com [216.40.44.112]) by kanga.kvack.org (Postfix) with ESMTP id 988396B007B for ; Mon, 5 Apr 2021 19:01:57 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 56A8583F6 for ; Mon, 5 Apr 2021 23:01:57 +0000 (UTC) X-FDA: 77999837874.23.BFCF4BD Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf28.hostedemail.com (Postfix) with ESMTP id CEC0A200027D for ; Mon, 5 Apr 2021 23:01:56 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 135MxNOB023313; Mon, 5 Apr 2021 23:01:38 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=DwfHARgXSwAHwerjLcKAXizpQAo9ZwGoNO+AVwhn8b0=; b=rORKI4G07YuYotesOZYRWBOBZJQtFu6KxTbFKWlnlo2rtzQOoM+gaPehloDAIjRQ3tRp gpLAGnb9WpjWoWQyTVXI4O24Sm28r/qES40jWm3+mr3UI7KOjBz/JrhmDc5wdfKJWLHR 6JLsC/V+PdEn3y1gd2lLJ0a3aA05uDHDDOB9COihiOAGUzKUz52LuH8tlXO0ejzA9nSK xT14kg4BD1J/DvwEL3L5JjK73HE97/YWy+wh9F8lkI3hTTsQRTxYyP+z3N0s/43fRq+2 4nZzpPqHzYYDSy+u0EFigCM5D57plk0LFu+SqReamGkSef4Gf108vQ6dsBWzuQk29IKA LQ== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by userp2130.oracle.com with ESMTP id 37qfuxawxy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 05 Apr 2021 23:01:38 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 135MtBgp020405; Mon, 5 Apr 2021 23:01:37 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2171.outbound.protection.outlook.com [104.47.58.171]) by aserp3020.oracle.com with ESMTP id 37q2pwt153-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 05 Apr 2021 23:01:37 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nhBpqxRPLrMN/1hHAgNk4+CVotC8AI08GUpAzKbk+b5FKUiNU6hVdw6NZPKsX1w2YuEm6wAhRKp84dTu/5x28phnb8Ho4SaR0EbGakHo3uQBn2nFAX54jAr1gu4RmDCzfIzJzf0mwFvtkOWpsFSY5usR3tp8A3Id8AwT/ULGQMwIe1yFfLegMODMXmLTK0nnStxo7KgJeGkEx9V3pjjH718kruYe9Tb60sdFiVvCyT3lNXDs32eXUvYT8uRLlv8SX46+HrqbBKpN7vg0BXdzi+otpNLvQf9tzD1LoKjQOv41Y4hE7TQn9gQg6hZS7xt6Axzt+0AvxBEYRhTIKmzMvw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DwfHARgXSwAHwerjLcKAXizpQAo9ZwGoNO+AVwhn8b0=; b=lrPYmxFu2uncFgrAe7KTdsj12UiVtVVoMbQp8XQLM+10g/jF6ofhCDi3Ldfxf+bSR3Y6KbnnHl0Re6foGp7ywar0m9fCks97PXtvwi0pMmm/KN8oK6KDeKxubJzbseLBku752ohfs5+4v7lNoHfVNde+OEjYNKbs23YpAiJFRN4F952SuGqYcs0OT/UoAI1Tlz5/OY6HodEW0Bui3NK9ik2YYJQeLLyKRQs/isaTGDwt0fqxGDMWAEXMBIEe9/o86/iT6PHhP/77hEV7v6/ncw1QvqyxF9wDv5ZB7i/WLS46yxlKROSGTSUjfY1nkLiadptKKaRBUNu7BhaxkuTH/Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DwfHARgXSwAHwerjLcKAXizpQAo9ZwGoNO+AVwhn8b0=; b=fKuKpGmspksTJbJj/2JzKA4J93xhGTpnGxCOvFATRW+9W9EPs/+AYD4BQimt4zG+/8b6k+PnU6tIxI4jRl37Qd0Z0IWtoW22PWTp48agGVrPyAUMrHNvNWi08Z0L2LDzko4LRdCVgONg7C+NrFcK1YOf2WcE36af9hBqKmsNCDw= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3544.namprd10.prod.outlook.com (2603:10b6:a03:121::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.26; Mon, 5 Apr 2021 23:01:34 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3999.032; Mon, 5 Apr 2021 23:01:34 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Roman Gushchin , Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Hillf Danton , Joonsoo Kim , Barry Song , Will Deacon , Andrew Morton , Mike Kravetz Subject: [PATCH v4 5/8] hugetlb: call update_and_free_page without hugetlb_lock Date: Mon, 5 Apr 2021 16:00:40 -0700 Message-Id: <20210405230043.182734-6-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210405230043.182734-1-mike.kravetz@oracle.com> References: <20210405230043.182734-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR18CA0031.namprd18.prod.outlook.com (2603:10b6:320:31::17) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR18CA0031.namprd18.prod.outlook.com (2603:10b6:320:31::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.27 via Frontend Transport; Mon, 5 Apr 2021 23:01:33 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 0f674453-a4a5-49f9-46d9-08d8f886c247 X-MS-TrafficTypeDiagnostic: BYAPR10MB3544: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:86; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: kkA8iodh7P7PReh9uFyaMFdEh7ymioOXi1V1UeU2HgTD8u9Tg2XFQZL9V3sDXrByRaDdVmZy4jQ3GHq5bSgAvoQA5AYNWkpe2sjJQIztTihFdFHF2NNxrwJWQZLtzd0Pqvc76rPQzCQH2RpVaxmRS9uOS5T8TzNFuM3xCGclBTr6+Os7C1q+vph1Mn2KKCeuMofG26xjWis6mdCGo9oVgvTL8yNEW5MAIddIJ2RwWTKKwBMUehJp/1pKhaPddBVF1SwHE1CZeRNtQ8gv0hhVh+CFsn2tpDrDdqVqCksOvkz+dh5a3+9T5cAnHeGB4NxvoGHWRXUximefcT/eWXYL6LqONi3uydko8LJGO0muTnynB1CMNVaLobYzJYz5dQ1H8FNT5I32sNcwwHzBaHrhkRCAf3ZySAhoTjrAuG0V+AnLmwGLEgIpRpYhXXHdVcC9PNSwccSneL6LIDV9e3uwy7aCKFfvxpC6OXptahuivTJienpoqW7TEEOCmyQmfYUbee3h/TO2sGekp9TmP+Fth0/rVh2/P56p3yyCAUYlEfAMyEOSUBykGFWeXwbz+AEpW/s9n6a8SbFhGUlKN/Lv+l7V1LYNOax/7OtZavhViU343SQXydPEeXRo1r71DbuUt9sz5uZJIyX//Ggc/VLPiJmg1n26dwiU6QiYBdYJ2l/ELwtL5GytYGQWmaRIcobi X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(396003)(376002)(366004)(39860400002)(346002)(2616005)(44832011)(66476007)(5660300002)(107886003)(8676002)(86362001)(26005)(6486002)(66946007)(16526019)(38350700001)(1076003)(2906002)(54906003)(83380400001)(4326008)(52116002)(7696005)(186003)(7416002)(478600001)(38100700001)(8936002)(36756003)(316002)(66556008)(6666004)(15650500001)(956004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: VeTCCU7ovjHaq/kk9EC33W1q4zbsDTL7ceNRmVNFUQEcj9yHF+6VQ8JxNfxy7hDE0WkEXTCbcr0H4EdY0fmp2NQq90RQIC7WI2YsB7Cu4NOnypWjyAYWMwoZZRL/OA9CNUJO/JorQyYaUMjbeJCHXzhr547EHU09kjxPqhmy5w4GjVBR+R9V4uhusO1u9keyseyrvLHY4dSp/yX/uJ4dgCTHy4ROrcTaFsk58x+lzWPdGu+uSTP3u7yTF8XkrFYlYwMOWuaunAcNMUGjFM8xCOmX4sK7og+aQSzt3JssDd3xhzIEo41UeAs4ww1WsSzEP6dhA66kAlI8w66FZnVh6RElHKTr7c2RcBShtnlVMiYhs/6GhiaXV1vXT9OIuJ+hW4/ZTJ8fOo+ZEMdoHG0Cn6I2ATFTHTK5QnRB4Y08VJLzepc6Hq/CCtdSrOTMeMU8m4cs96QoylW7XNaYnEwQLWEM3kWtu2Mtg1GFatg4HEVEdkAzuzN00uptTmsqxuPSo9L7naJ5QpKbT9oNqnTh6a5sXttb12MD06sga4ErReyumB/EpiEahUY7Lu1J9imPuQGFuOd6Q1/Q3KxjXdoJU6gcXAwH0ykLJ69xdyQ3FB33AQdpKBWc1ZOpIIXVMqyvIRJi1jRGh8l6WCV/MQH7giS+h+lX/3nG22KqjdnAPy9Bf5mjqk3zqQBLPWF/xI3I4Hd6QDe3ie7aaOr/8eYR+dK9+g+BK0upvA8kVizuBYl2eGc8gHTtYRjGuqdP3SXlnIvIWRV/Tsc2ttpBmM14eQW1BjhwVK+0kezc+9bI+H79hmeEbbcvHeR0Ec/eDLS3sSr9E7DH+98RkeXURUZWiePcHTZ92jRz2ppkMfYi7s+fpriLgoAt+YHrzYk07Mov35GcCKPJ5DqInp7HiNm5d38sHiT2HE78gMR7fmDSrR9KjA9H1PUw5pNdQzN6e6oq3APOoB1ML4P1NLSQfa8r0+mJM1Ra4WvfOXv0N82STh35O+AfjnwPA7ZCZPq3VNP2L91/QnCZ5H085n24SVhWLyozxItoG9BgVjN8Uw1buoZx3pcxYnamEAmhVUNYRfLDVPdT+Mu1WNgN/bFXKoi8us4HbpjsKLtsSBRNrR3UL9FNAOA72PB0Ow+1J5p7CVKpsyEqgX6u0ORYUuBgUoXLlIJ+NofkjUs0bj6Z61YtYX+ZffK3HdyiV98nNu8ft2lAoC+8elxJQGxd4nuBw3Q6oSVE8oG5cmME7TLqEjefqSuk32Rt0lQ9x9CHTcQfVwAjNxSGG2zaOg1cPeN5Q9y19B+Acm5XlXU1ZoWlX6mCX+HmscUgsseUnA+oZ7wwTvDT X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0f674453-a4a5-49f9-46d9-08d8f886c247 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Apr 2021 23:01:34.7168 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 9zwcPbDvRRqRUUtS3hzQthed+mfjIKyCR1nQHZLv1+yNjecPrkIqq33R5Jg8b+s2+JjESgowQTkdQl/1fl8vsw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3544 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9945 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 suspectscore=0 adultscore=0 spamscore=0 bulkscore=0 mlxlogscore=999 phishscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104030000 definitions=main-2104050152 X-Proofpoint-GUID: Wqg87V0OSoN5nvBJGKl9UfO9QoHEKc-i X-Proofpoint-ORIG-GUID: Wqg87V0OSoN5nvBJGKl9UfO9QoHEKc-i X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9945 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 clxscore=1015 malwarescore=0 suspectscore=0 impostorscore=0 mlxscore=0 priorityscore=1501 mlxlogscore=999 bulkscore=0 spamscore=0 adultscore=0 phishscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104030000 definitions=main-2104050152 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: CEC0A200027D X-Stat-Signature: dkm175p98nqjxcewe5gtp9asqo5ktc7p Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf28; identity=mailfrom; envelope-from=""; helo=userp2130.oracle.com; client-ip=156.151.31.86 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617663716-261151 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With the introduction of remove_hugetlb_page(), there is no need for update_and_free_page to hold the hugetlb lock. Change all callers to drop the lock before calling. With additional code modifications, this will allow loops which decrease the huge page pool to drop the hugetlb_lock with each page to reduce long hold times. The ugly unlock/lock cycle in free_pool_huge_page will be removed in a subsequent patch which restructures free_pool_huge_page. Signed-off-by: Mike Kravetz Acked-by: Michal Hocko Reviewed-by: Oscar Salvador --- mm/hugetlb.c | 43 +++++++++++++++++++++++++++++++++---------- 1 file changed, 33 insertions(+), 10 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index df2a3d1f632b..be6031a8e2a9 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1446,16 +1446,18 @@ static void __free_huge_page(struct page *page) if (HPageTemporary(page)) { remove_hugetlb_page(h, page, false); + spin_unlock(&hugetlb_lock); update_and_free_page(h, page); } else if (h->surplus_huge_pages_node[nid]) { /* remove the page from active list */ remove_hugetlb_page(h, page, true); + spin_unlock(&hugetlb_lock); update_and_free_page(h, page); } else { arch_clear_hugepage_flags(page); enqueue_huge_page(h, page); + spin_unlock(&hugetlb_lock); } - spin_unlock(&hugetlb_lock); } /* @@ -1736,7 +1738,13 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, list_entry(h->hugepage_freelists[node].next, struct page, lru); remove_hugetlb_page(h, page, acct_surplus); + /* + * unlock/lock around update_and_free_page is temporary + * and will be removed with subsequent patch. + */ + spin_unlock(&hugetlb_lock); update_and_free_page(h, page); + spin_lock(&hugetlb_lock); ret = 1; break; } @@ -1805,8 +1813,9 @@ int dissolve_free_huge_page(struct page *page) } remove_hugetlb_page(h, page, false); h->max_huge_pages--; + spin_unlock(&hugetlb_lock); update_and_free_page(h, head); - rc = 0; + return 0; } out: spin_unlock(&hugetlb_lock); @@ -2291,6 +2300,7 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE; int nid = page_to_nid(old_page); struct page *new_page; + struct page *page_to_free; int ret = 0; /* @@ -2313,16 +2323,16 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, * Freed from under us. Drop new_page too. */ remove_hugetlb_page(h, new_page, false); - update_and_free_page(h, new_page); - goto unlock; + page_to_free = new_page; + goto unlock_free; } else if (page_count(old_page)) { /* * Someone has grabbed the page, try to isolate it here. * Fail with -EBUSY if not possible. */ remove_hugetlb_page(h, new_page, false); - update_and_free_page(h, new_page); spin_unlock(&hugetlb_lock); + update_and_free_page(h, new_page); if (!isolate_huge_page(old_page, list)) ret = -EBUSY; return ret; @@ -2344,11 +2354,12 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, * enqueue_huge_page for new page. Net result is no change. */ remove_hugetlb_page(h, old_page, false); - update_and_free_page(h, old_page); enqueue_huge_page(h, new_page); + page_to_free = old_page; } -unlock: +unlock_free: spin_unlock(&hugetlb_lock); + update_and_free_page(h, page_to_free); return ret; } @@ -2671,22 +2682,34 @@ static void try_to_free_low(struct hstate *h, unsigned long count, nodemask_t *nodes_allowed) { int i; + struct page *page, *next; + LIST_HEAD(page_list); if (hstate_is_gigantic(h)) return; + /* + * Collect pages to be freed on a list, and free after dropping lock + */ for_each_node_mask(i, *nodes_allowed) { - struct page *page, *next; struct list_head *freel = &h->hugepage_freelists[i]; list_for_each_entry_safe(page, next, freel, lru) { if (count >= h->nr_huge_pages) - return; + goto out; if (PageHighMem(page)) continue; remove_hugetlb_page(h, page, false); - update_and_free_page(h, page); + list_add(&page->lru, &page_list); } } + +out: + spin_unlock(&hugetlb_lock); + list_for_each_entry_safe(page, next, &page_list, lru) { + update_and_free_page(h, page); + cond_resched(); + } + spin_lock(&hugetlb_lock); } #else static inline void try_to_free_low(struct hstate *h, unsigned long count, From patchwork Mon Apr 5 23:00:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12183803 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E2C7C433B4 for ; Mon, 5 Apr 2021 23:01:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 545E2613BC for ; Mon, 5 Apr 2021 23:01:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 545E2613BC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B1C976B0071; Mon, 5 Apr 2021 19:01:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AE7FE6B0073; Mon, 5 Apr 2021 19:01:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8EBB36B0078; Mon, 5 Apr 2021 19:01:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0109.hostedemail.com [216.40.44.109]) by kanga.kvack.org (Postfix) with ESMTP id 68D2E6B0071 for ; Mon, 5 Apr 2021 19:01:57 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 32D5383F1 for ; Mon, 5 Apr 2021 23:01:57 +0000 (UTC) X-FDA: 77999837874.21.9C1A8D1 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf14.hostedemail.com (Postfix) with ESMTP id B98FBC0007E7 for ; Mon, 5 Apr 2021 23:01:52 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 135N07Sl014878; Mon, 5 Apr 2021 23:01:42 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=bjJ8BITTlbisNR/LOqpXpU5wYr8TGaUYDvW6QoK/vq8=; b=dIthAsTC9r8xaBRAxTOax+cum4Qi7Ai1HHrXZyTf0bgg6a0Yp9d7uN9YY/upbYgvx8kR Qqx8AhiN25/5wjd/AUTU5RS7j6clvyNGbcGhc7AWXr7T/nhlwuGwtMKonSDryFGwWubE 2BKbzZCmCbatdrj8lArTO6vYie/1MhP8aJzzcuIim+hOVeZJpaVdmnaFbOLyKqgAjgYG HwuQF3mJkkuxJaQYBBAQy42buTWYbUXzBstjTIDitMbBOn4NPhUzKQ2EInTcPto9vjfC vp872pjcKKvfF8OU1rPtnTAUt78AZSN+B9qM/kD9UV5opJ0ZPksaPG1UutUblE4bt5lY bg== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by aserp2130.oracle.com with ESMTP id 37q38mkkba-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 05 Apr 2021 23:01:42 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 135MtqFj089438; Mon, 5 Apr 2021 23:01:41 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2170.outbound.protection.outlook.com [104.47.58.170]) by userp3030.oracle.com with ESMTP id 37qa3hp3ex-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 05 Apr 2021 23:01:41 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AbNz2d+qYFjtWBicp+eWOhq88bkAP95p7wtI9VWZ9YS25nvY3H+Sap036xplCjL4RxcoKm4xwzpM8h8CqCsoTcjXPHNkR7acDxQvWvZLDy5q+AU6onhYeK7ZJWzJak1SP+oL8WAoWx2lr1KrDAr525sSx5MqU86XPeleUhJV5LKXTAUYQ+Pi0Ji3Ovoch8s4qk264WfYZFxh/WWu2yoUp9IiZHKlYhd4Rqs721Rwmgvpj50NMdqV+JCWlAikzbRWWMIEF6GLjH5E7BbYqbD4d8YXmTNOflsxY23EN9aHhlaVunxo0uS0k03FXaIOXJS1uQB3dmXq2vXoVLfMpcxI1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bjJ8BITTlbisNR/LOqpXpU5wYr8TGaUYDvW6QoK/vq8=; b=HHHZrCEyUcPw6+lIwbnvdUf6WGEORyN0Yb67yghnVSL1/eFRckv/UpD1RiJuJPbO8No6S4RLtWSJKWQSI97mkACnevVASeybyFzzsczhd6o6E39AVEDXxYuCZx4NJ5OPaFwElK/c26cP7Gc4sMLDYd+X8IQbcIMVoTAl3d/oS9uhgkHhWTkSu3wdya3sX5DRmU+ixwh8bVSi81WD2UinMYwkf/8Uf2PfrLu5UGs9mVoDlNCAHSjP8HnHwZVTB58cX/WzJYGcUj+0Mviy2bA8m0wf0EA+GI/ClPu+GufHlJhGaX6vQEdQViT4qLJ6JdfbUleZMfetgbL6x5g5HKIt1g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bjJ8BITTlbisNR/LOqpXpU5wYr8TGaUYDvW6QoK/vq8=; b=G3mFkMoer+T6gcqDSn5y9eZPCybnpGeQpU3AwvPeojAON5IX/CxQBKZqzTSFVukunMzZr9R8RA4FFtqRQV8WaTwIVY+JYcBMgED01f5jqVhlB/SNkGyWIFT5wFyWsCnip85+jZUDgP5s2Y8XH1iZJR4d2/YD8ZuR1TDMtkiAyBA= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3544.namprd10.prod.outlook.com (2603:10b6:a03:121::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.26; Mon, 5 Apr 2021 23:01:37 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3999.032; Mon, 5 Apr 2021 23:01:37 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Roman Gushchin , Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Hillf Danton , Joonsoo Kim , Barry Song , Will Deacon , Andrew Morton , Mike Kravetz Subject: [PATCH v4 6/8] hugetlb: change free_pool_huge_page to remove_pool_huge_page Date: Mon, 5 Apr 2021 16:00:41 -0700 Message-Id: <20210405230043.182734-7-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210405230043.182734-1-mike.kravetz@oracle.com> References: <20210405230043.182734-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR18CA0031.namprd18.prod.outlook.com (2603:10b6:320:31::17) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR18CA0031.namprd18.prod.outlook.com (2603:10b6:320:31::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.27 via Frontend Transport; Mon, 5 Apr 2021 23:01:36 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f9daf939-9e2e-4fe2-5329-08d8f886c3e6 X-MS-TrafficTypeDiagnostic: BYAPR10MB3544: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8882; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: udS3Xf8Z6OOW9Nhht2MG77DdVeyT6277pfQ6zrppmFNWsghgqHFJIzZPCAUz9yroT2ykyj7dztfVkw3JxmhG/9pejpV1Jnov5euq4r9l3sYiP6c2UTqEu7Pp0K9rpfi1tsqoBVvyRaeq0NxVbseJMBImqqL+hNFlIZs5OyDyZnHPSx5q1kQPmW8dHhZScpwWsWNrwoZPuUIiGHLt9mZFCTCDD4GT3kxgJV59/FQ+AEkV0RwUke3EF0srdJqZOuFEJeTx+vZwJ7mzMgmONDElMAfpKJ5QXCnoLMoZNwAih1JxiZ1ycOiV1FboQkM3JESBqDwoLUaZ/Qv0f/uMBjFdQxSO7n6pa4NIQOuTAGeq2pNvpHx3WlCCaQIom8PY8cqoe7rpncw9I8s0fcOiMESgOhQYZFVNk9hTkUD0zsdgNbJNcCJIZX0CInZJIh+jOgOoV/owmFFQTeZdVcPQLJqFmdx+pU6R50KZBAZZvAABF1o4masEN8cmZm+KgZDiumAtsX3kznjL8M6J7CY85EcBDPHiUfX3uR2Z6LTBwqMDzsOSV8CdsSdRx0ZEPvyroGC7OII7CO3A9CsJaOXHRqN0TkGYQfwVE17+oYuHdwZ9+OlfR9sWSHjYldOV2DsMcCmutJQReAeUXJojL0aqJVwJyRoCRd+u99QVnin0buN42YfCjkiBKbZ9RS84BjKJI3rRIChEx/SKTBVb6JlaAzfl/g== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(396003)(376002)(366004)(39860400002)(346002)(2616005)(44832011)(66476007)(5660300002)(107886003)(8676002)(86362001)(26005)(6486002)(66946007)(16526019)(38350700001)(1076003)(2906002)(54906003)(83380400001)(4326008)(52116002)(7696005)(186003)(7416002)(478600001)(38100700001)(8936002)(36756003)(316002)(66556008)(6666004)(956004)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: +UJqpB42TlONBETEKU3YXHW/apKDjTO0c5yS8g6DVc7CCpZfz2/gGvkNH39L4eCv/EqgYDlUwoM5xHsblxIVGFhC/9SQCOyA1un7Eij2bd9NZdRZnzszbcQVzvIO+4kSMPtco6JdO38PyP9MUgMdO2zsPxuXD3ZePVjpmZPL3kJ2JflCaUdULEWN1e0vZoc7UhH5Lth6md/lAqrh3df3nFOx+g/DeiIL0G1xjwMpwNr58kmaTCuRrK/c3ivNwSACAX5Mi0ofVmDmmcoRnf/S2/PekgIKggGn/wmNvH9yZs7ruZFFu2cV0iGx5k13+pvY2z1hp2ooT3vNt8Qh1ruZHgfGRjtNXkI+yyAUKBwHqDnWCYKXPEeMcKoKtSFS0heOYbl3FxlO6YNIkFhMyutZ0yBul12DPc28p/DDIV9NVcNsLDPpoQvd+rczxr6KT4IiOeM4/qelev+49MEUCLhcIJ8UnnXwlC8RXuVJlxHtS2SDxwFZOqD/xRBfWUqXfLvAhxswI0EGfjWtBbObrQ1QVqrYXs2d7WaDUAE8Dx7Hzf0fMZi9IhQPf5AlV9CdHV6nOx9utigHyjC+SwsrTNQSO1koO+fTeJKlzfkXmPJ2xlo/MDNUpamnY7BtuC17usMXTdyxKAQBtlZTHkLM99MD6A9isWH7ZsnvpMt6EWP5yq04YT7ZcMhswTLi/Iv6+zcarz061ogLZgUuCjklRGSeqkGdtofxBI0zSGdkOUP/YLeJgt76GBuqE8FS0aYCUKWxcj9GIjsxgmD2fCc3Zk+pTROi9sF7c97XFD+UvaEvXlfkhLa8+QxFfcGoXoC6BxzhyyXN3gRtT9PKtV9zeEwvpQAXf8sm4bsbFjtpHKO57PcsPh9XE61pE4CvxWCuZfj/C1a98sD7cV/PPkoFlAov1rAjVTUzrSvE7zXCRUY2L8TDMnm2hyf5bqC13kDzeOnebyzpPX6W4W7NNet1lnwIS3sdmqLtXuuj8qbZLzSj0o/4LHCXNvRTOeDjJMqv3p+9VXvIBjrBzAf5cD1LdyZOLQI9p1IHUA5/WZcNbrPu+9XkS9FWk1VauxX8eU8pP6yRrKwkcCsE3hE61iMDcfZt4kMgPWARIO4s++fvlMvFCBdwbs/a7oP6X7iMtEs/N/C3KSkveMVpw/cfdgOtznks99pDS1wfOFBWCzUIv1MXUvTb3jQWQAkIk7hDQ57Fi6ewNGczl3g+NlfTu7UpY+PVp99FpJsqcVZAE2Wlaa85Ts6N+w5gBu0o1SsTvIlgrbv+uByyYmbuQ8QUS75MyEJe32EPKJLmEDjlIdi5HQ7Cw46acBG2wpA1/REEwURYmvX1 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: f9daf939-9e2e-4fe2-5329-08d8f886c3e6 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Apr 2021 23:01:37.6678 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: +dkupVK7za+Eaj6fUlsM/Hr3gB2IK2W3g3mfGiQCsEGApRv69MIBqtc7MsQHU/7q06CKSQiWG9/Z56mFwQ3ecw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3544 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9945 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=0 malwarescore=0 mlxscore=0 spamscore=0 phishscore=0 adultscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104030000 definitions=main-2104050152 X-Proofpoint-GUID: smRMt5xKuNbAVAUTDkFDsi0ItUVRceh- X-Proofpoint-ORIG-GUID: smRMt5xKuNbAVAUTDkFDsi0ItUVRceh- X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9945 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 clxscore=1015 suspectscore=0 impostorscore=0 malwarescore=0 priorityscore=1501 spamscore=0 mlxscore=0 bulkscore=0 lowpriorityscore=0 adultscore=0 phishscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104030000 definitions=main-2104050152 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: B98FBC0007E7 X-Stat-Signature: 9mqq3q8jrhzcfs58bnoxtgddphigr5gm Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf14; identity=mailfrom; envelope-from=""; helo=aserp2130.oracle.com; client-ip=141.146.126.79 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617663712-629295 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: free_pool_huge_page was called with hugetlb_lock held. It would remove a hugetlb page, and then free the corresponding pages to the lower level allocators such as buddy. free_pool_huge_page was called in a loop to remove hugetlb pages and these loops could hold the hugetlb_lock for a considerable time. Create new routine remove_pool_huge_page to replace free_pool_huge_page. remove_pool_huge_page will remove the hugetlb page, and it must be called with the hugetlb_lock held. It will return the removed page and it is the responsibility of the caller to free the page to the lower level allocators. The hugetlb_lock is dropped before freeing to these allocators which results in shorter lock hold times. Add new helper routine to call update_and_free_page for a list of pages. Note: Some changes to the routine return_unused_surplus_pages are in need of explanation. Commit e5bbc8a6c992 ("mm/hugetlb.c: fix reservation race when freeing surplus pages") modified this routine to address a race which could occur when dropping the hugetlb_lock in the loop that removes pool pages. Accounting changes introduced in that commit were subtle and took some thought to understand. This commit removes the cond_resched_lock() and the potential race. Therefore, remove the subtle code and restore the more straight forward accounting effectively reverting the commit. Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song Acked-by: Michal Hocko Reviewed-by: Oscar Salvador --- mm/hugetlb.c | 93 ++++++++++++++++++++++++++++------------------------ 1 file changed, 51 insertions(+), 42 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index be6031a8e2a9..93a2a11b9376 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1204,7 +1204,7 @@ static int hstate_next_node_to_alloc(struct hstate *h, } /* - * helper for free_pool_huge_page() - return the previously saved + * helper for remove_pool_huge_page() - return the previously saved * node ["this node"] from which to free a huge page. Advance the * next node id whether or not we find a free huge page to free so * that the next attempt to free addresses the next node. @@ -1386,6 +1386,16 @@ static void update_and_free_page(struct hstate *h, struct page *page) } } +static void update_and_free_pages_bulk(struct hstate *h, struct list_head *list) +{ + struct page *page, *t_page; + + list_for_each_entry_safe(page, t_page, list, lru) { + update_and_free_page(h, page); + cond_resched(); + } +} + struct hstate *size_to_hstate(unsigned long size) { struct hstate *h; @@ -1716,16 +1726,18 @@ static int alloc_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, } /* - * Free huge page from pool from next node to free. - * Attempt to keep persistent huge pages more or less - * balanced over allowed nodes. + * Remove huge page from pool from next node to free. Attempt to keep + * persistent huge pages more or less balanced over allowed nodes. + * This routine only 'removes' the hugetlb page. The caller must make + * an additional call to free the page to low level allocators. * Called with hugetlb_lock locked. */ -static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, - bool acct_surplus) +static struct page *remove_pool_huge_page(struct hstate *h, + nodemask_t *nodes_allowed, + bool acct_surplus) { int nr_nodes, node; - int ret = 0; + struct page *page = NULL; for_each_node_mask_to_free(h, nr_nodes, node, nodes_allowed) { /* @@ -1734,23 +1746,14 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, */ if ((!acct_surplus || h->surplus_huge_pages_node[node]) && !list_empty(&h->hugepage_freelists[node])) { - struct page *page = - list_entry(h->hugepage_freelists[node].next, + page = list_entry(h->hugepage_freelists[node].next, struct page, lru); remove_hugetlb_page(h, page, acct_surplus); - /* - * unlock/lock around update_and_free_page is temporary - * and will be removed with subsequent patch. - */ - spin_unlock(&hugetlb_lock); - update_and_free_page(h, page); - spin_lock(&hugetlb_lock); - ret = 1; break; } } - return ret; + return page; } /* @@ -2070,17 +2073,16 @@ static int gather_surplus_pages(struct hstate *h, long delta) * to the associated reservation map. * 2) Free any unused surplus pages that may have been allocated to satisfy * the reservation. As many as unused_resv_pages may be freed. - * - * Called with hugetlb_lock held. However, the lock could be dropped (and - * reacquired) during calls to cond_resched_lock. Whenever dropping the lock, - * we must make sure nobody else can claim pages we are in the process of - * freeing. Do this by ensuring resv_huge_page always is greater than the - * number of huge pages we plan to free when dropping the lock. */ static void return_unused_surplus_pages(struct hstate *h, unsigned long unused_resv_pages) { unsigned long nr_pages; + struct page *page; + LIST_HEAD(page_list); + + /* Uncommit the reservation */ + h->resv_huge_pages -= unused_resv_pages; /* Cannot return gigantic pages currently */ if (hstate_is_gigantic(h)) @@ -2097,24 +2099,21 @@ static void return_unused_surplus_pages(struct hstate *h, * evenly across all nodes with memory. Iterate across these nodes * until we can no longer free unreserved surplus pages. This occurs * when the nodes with surplus pages have no free pages. - * free_pool_huge_page() will balance the freed pages across the + * remove_pool_huge_page() will balance the freed pages across the * on-line nodes with memory and will handle the hstate accounting. - * - * Note that we decrement resv_huge_pages as we free the pages. If - * we drop the lock, resv_huge_pages will still be sufficiently large - * to cover subsequent pages we may free. */ while (nr_pages--) { - h->resv_huge_pages--; - unused_resv_pages--; - if (!free_pool_huge_page(h, &node_states[N_MEMORY], 1)) + page = remove_pool_huge_page(h, &node_states[N_MEMORY], 1); + if (!page) goto out; - cond_resched_lock(&hugetlb_lock); + + list_add(&page->lru, &page_list); } out: - /* Fully uncommit the reservation */ - h->resv_huge_pages -= unused_resv_pages; + spin_unlock(&hugetlb_lock); + update_and_free_pages_bulk(h, &page_list); + spin_lock(&hugetlb_lock); } @@ -2682,7 +2681,6 @@ static void try_to_free_low(struct hstate *h, unsigned long count, nodemask_t *nodes_allowed) { int i; - struct page *page, *next; LIST_HEAD(page_list); if (hstate_is_gigantic(h)) @@ -2692,6 +2690,7 @@ static void try_to_free_low(struct hstate *h, unsigned long count, * Collect pages to be freed on a list, and free after dropping lock */ for_each_node_mask(i, *nodes_allowed) { + struct page *page, *next; struct list_head *freel = &h->hugepage_freelists[i]; list_for_each_entry_safe(page, next, freel, lru) { if (count >= h->nr_huge_pages) @@ -2705,10 +2704,7 @@ static void try_to_free_low(struct hstate *h, unsigned long count, out: spin_unlock(&hugetlb_lock); - list_for_each_entry_safe(page, next, &page_list, lru) { - update_and_free_page(h, page); - cond_resched(); - } + update_and_free_pages_bulk(h, &page_list); spin_lock(&hugetlb_lock); } #else @@ -2755,6 +2751,8 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, nodemask_t *nodes_allowed) { unsigned long min_count, ret; + struct page *page; + LIST_HEAD(page_list); NODEMASK_ALLOC(nodemask_t, node_alloc_noretry, GFP_KERNEL); /* @@ -2867,11 +2865,22 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, min_count = h->resv_huge_pages + h->nr_huge_pages - h->free_huge_pages; min_count = max(count, min_count); try_to_free_low(h, min_count, nodes_allowed); + + /* + * Collect pages to be removed on list without dropping lock + */ while (min_count < persistent_huge_pages(h)) { - if (!free_pool_huge_page(h, nodes_allowed, 0)) + page = remove_pool_huge_page(h, nodes_allowed, 0); + if (!page) break; - cond_resched_lock(&hugetlb_lock); + + list_add(&page->lru, &page_list); } + /* free the pages after dropping lock */ + spin_unlock(&hugetlb_lock); + update_and_free_pages_bulk(h, &page_list); + spin_lock(&hugetlb_lock); + while (count < persistent_huge_pages(h)) { if (!adjust_pool_surplus(h, nodes_allowed, 1)) break; From patchwork Mon Apr 5 23:00:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12183815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C10DC433B4 for ; Mon, 5 Apr 2021 23:02:11 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8E8B8613BC for ; Mon, 5 Apr 2021 23:02:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8E8B8613BC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AE4EB6B007E; Mon, 5 Apr 2021 19:02:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A4DE26B0082; Mon, 5 Apr 2021 19:02:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 82CE96B007E; Mon, 5 Apr 2021 19:02:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0202.hostedemail.com [216.40.44.202]) by kanga.kvack.org (Postfix) with ESMTP id 5DA9C6B007D for ; Mon, 5 Apr 2021 19:02:03 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 0EA9E8249980 for ; Mon, 5 Apr 2021 23:02:03 +0000 (UTC) X-FDA: 77999838126.02.2391234 Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) by imf29.hostedemail.com (Postfix) with ESMTP id B8AD2FC for ; Mon, 5 Apr 2021 23:02:00 +0000 (UTC) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 135MwvFr191382; Mon, 5 Apr 2021 23:01:46 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=vQe8pObsRjtowWnNhpKWUIDjEKSCCdNT1PtA7AxP1E8=; b=KC580ox951z29QqBISpbYtcbHcXwlM43CHZUUyMevL4RAd0VpEo7phepEcmeGh4GESZe kYKWDFIrcWJFOqhYhBb5x2hoHz7uyn4YuJGWV+k+hB79nlg0KcF6aSz49gBUuIg5xVYi mzPWLF4a/vD3H/a0e4AszdDh5yAIQErp6ZpwcFnE2U07rKOzFIV4HDM0Ovv+hnAXVgts w6w54Ms97O21QC7uq7l8jHxKCVCEkGCUZhv09HHDiWouK7smvl7T/K6/Kg/XXCPvajMF sj3Im/5LcEXebomOUZk4gbYcAAiaehlq5q1T/+s/crxRNLXs14w73SNc5jv2bE0RnPkR bQ== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by userp2120.oracle.com with ESMTP id 37q3f2bknf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 05 Apr 2021 23:01:46 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 135Mtx75178827; Mon, 5 Apr 2021 23:01:45 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2173.outbound.protection.outlook.com [104.47.58.173]) by aserp3030.oracle.com with ESMTP id 37q2pbsq1w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 05 Apr 2021 23:01:44 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=eFumLqRl//FqfbOpKOoaakFOutJ/EqrivGUFAMioo0Ipu8KJvsq6brKVwCzRXuNhJ9WASx29G5szx61RhvyqaDJadr/OSVoihBisj0E6Wr/BwzbvNKWYbF8WQQrobNEU4Wq1fKXSk5tcojOpJnxZmqpJ+rjxyR01zl+uQ/PJg5hlKD8G8u1jmqcJZUr401HynfSI6p95bbSPtBOQaTk4wQ8P9AQkyXON3ZKe/oR/qSP+CVNd/Q23aBOO9GC352ylgzFtStQ1rmXF2YsRrzpiEgH/L1m2kxeGPT9yt07Qv02pz9nRi0U63D9uhQ7sJxhsjUdgJW40STnUyugdBXwZyw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vQe8pObsRjtowWnNhpKWUIDjEKSCCdNT1PtA7AxP1E8=; b=dncfLChiATRv22jNqxoL0nnzum/QXWGGLqAupK4DXHNJN5Rua2UKCu/TvOKkPYbBnF08QSkFkQFZLW0w2UqmxMrTDidzONmDo2oxMOVHKJ0SOLYvtrCzyEj8ze8SVeLNtSWf7lGchAqbZ9ASxcyCC3Dg0eNStYDUh2U7vJRscbonK3GZyFA/Y4j+3RorEhL4fNOrrYErJih2jSEUHfetpGOJoWf+XnYLoTwiEXRAH0hAMVOBTUJhuyD6/L55VqODE23bDk5R4CpRPnD1WyQ5DxnFLC6fgmZofosmaJmNkXPXcPHcJGr+HDl9YW5aKV/+7XHt6V3XBQc5CecNI5QvjA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vQe8pObsRjtowWnNhpKWUIDjEKSCCdNT1PtA7AxP1E8=; b=E1C4PJax9pV/oTUxwPceq2Dm5yj2f0FzZtavZf5zq0zCnvmHJyaytb5/9v6N8NZhNTk6Vn8Na51gvwTmBLJyLpNnWbKhk9p+VzyAXP3Vc+OT3vogVQiHCJWu4HwGYf+JvjiLgU+i5rwpuGm4C4GMHeFVxksoGFxI3LmkIz+FRZs= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3544.namprd10.prod.outlook.com (2603:10b6:a03:121::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.26; Mon, 5 Apr 2021 23:01:42 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3999.032; Mon, 5 Apr 2021 23:01:42 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Roman Gushchin , Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Hillf Danton , Joonsoo Kim , Barry Song , Will Deacon , Andrew Morton , Mike Kravetz Subject: [PATCH v4 7/8] hugetlb: make free_huge_page irq safe Date: Mon, 5 Apr 2021 16:00:42 -0700 Message-Id: <20210405230043.182734-8-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210405230043.182734-1-mike.kravetz@oracle.com> References: <20210405230043.182734-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR18CA0031.namprd18.prod.outlook.com (2603:10b6:320:31::17) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR18CA0031.namprd18.prod.outlook.com (2603:10b6:320:31::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.27 via Frontend Transport; Mon, 5 Apr 2021 23:01:40 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e9e91f62-cfd1-447c-ae49-08d8f886c6a2 X-MS-TrafficTypeDiagnostic: BYAPR10MB3544: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:82; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Jj8Jk0YE96ykJ8GsSAU794BAoK9ZzLPWcQEJI7nATkmCo3KenKwjVrES0HKnJ8/kDHlhZkRWFFgZRz6T4lYX1WYDlbsnQJjLlOEZyjPmFwb9sQs/yFZn0Joqng3QrSsMyrxox3Tm8hKLzeQYllvObag8NccHRpqgJgFBI4XXmdtWzWMvBtUwFRqcrwBsDaeCgIV8UZTQpl3cdZDEQw+8PibShzH2skLKD/rHTMtjEctFhCD8CRbMiWJMCy9ynOAjguAjJqPqIWiL1rJ6VbLosKcZMClaReOpraKz9qXDjIg0FqavE2brC6x7nyxhS8Oj7b8K7k9BVZSovIRDT6rVya0iB0QhEXASOpBD6oF9TddIzvP8T5PQJYai8K6oD6YByrcUPkXW3WgKKTvbPUKwOb0CS2a0oPpEYgboh4HCFhtEnUqDA3XQsSd2Xq1wBnlAlf8kvcBVfhHcN6T2y0VNGkGHeAFgG2YL02NTtzC08I8d8ixwu25z90SpDFj3XKwqqvJfKlXODjici+UUZSdVbNod2P4BSgnsGGE5R6rPWtcYnaFUkrGKxj7xZRJYJ3yqGwKo5r1IRyLimPaaGjbDnHzg6dWrxWUQMPPMQX3zSq8hLeSctWlmizBW8W35VZ8+u3YgKYact+B+ERCoqhNoTfzzahIwHM5rUllZsz05MGQyt2SKxLKHvwrPFYwNb8IMDJZmcvm7YwioMXeLvDU6SXjqMfpKvgaiz7/0uj0TUrWsJXeQdyM4CX/83jz0K5xoxW54wPFbgDvoEP8HT+uNvzPKtPTg0ruN5Z7mVvXsceE= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(396003)(376002)(366004)(39860400002)(346002)(2616005)(44832011)(66476007)(5660300002)(107886003)(8676002)(86362001)(26005)(30864003)(6486002)(966005)(66946007)(16526019)(38350700001)(1076003)(2906002)(54906003)(83380400001)(4326008)(52116002)(7696005)(186003)(7416002)(478600001)(38100700001)(8936002)(36756003)(316002)(66556008)(956004)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: I4WcCFVZu7ZLO1abra4pl7xBhW2g+WpPTgm6S+UyG9Oa8XnqfUb6ZDdAIulPIWghK73BaCB7MuBqUs1c6aZPW1av5WuelfOtbVuNiTUGJTZT9dkvhWugFklba55zrCdMW7uKlQnheYkHXmmi64wUOXA8S1hCiorWH6GUj+C/5GbHzpmGmYeZTS8+hYRJfeY4R1qDa6tlmSL5X5jc+bNqVwfISvVn6qd+FsIPdQzubtUSjcNCvnpiNTIU0MGkWsaDraRM18h1KZ3ooUDRZjN2SwAlUpps20VtJw+EPv+PibvGsGP234Rzumyslgdwd9S86tC9VUKwRMy27xZSvv6kQMjyaGbz7+BAPdQyKqW70GJBYaRSs/RWqdn4MdQLUOwSv47EtxU00E/98r7DKBq1oiTjbEGbych81rsMvrzLuL8bjH7whzxmI2L0sXjTK1I528gAys/nQuBqxlm9Ev+jN61EcqgbMredJpQSoLtpcUnd9i9hvzk6kXA6B5grKtl05mKe5LKfrjQXJJ5WmTEbh8ak9yf5+N+Oi1fQjovERpnyKy4zipXBT68CjXX3LFezRTW7UI79TOKkdtvUhVvsDLo+hy0kR7SQ2fjS4NAqsmXngK9BcCuTSe4IZ0BshNlZ3eHaiPd0NwHH3hodX3XixxZF7WGupqSam1GoekyD2B1CB4yZ2Lhd74o7MHc7npPc4bqsHTA+TDk8V/W7/1esDDpUo20W3y1BNs4y+Ichq/RD/c5qoYS9RPoQFKLUlsfXkYZrH8LpkZ0oYrClESEK97UsXt4ihMnFlKH/ukh/96ANQASxewsYOHeLbbOe0dwkVNWuvWQ8XF3MwOOA8B1d8gllPleLzn6HFZKNvEopcTnh/GKc0Eogj0WjruIG8KfN0Fu/Mcfei/YgqGTCobVtHTPfhdLuEDkm6DjR7bbOeLgg1w3FtkxPk/+UN2RRd0D0VvxugEuMfMAf0og5DSjGw5+HCdLYMnrD9gJWI8IT5lUy721g7AKHJGQBGI9u4Fc0EmzIS9HXNwf3zLqE/bRLOoyDqkPib5pOfzopSMy5tq0NXLmfkCh0nJDOsV6t8HdTycw/m73qmNfFIGHnXcWBuxuTwIFdwyi+cDIns7zvDQ9wnAWX1IMEtqxYGJEkAlARKxdEprUErPOpjAbdziMiabVsfGl/3zsXKs6KRJPN+0SfSDrBi4i4ss1Xt/jR4PxA7x8Wb6CIyT9kr7sMEegit17/F3pCXJvurH2lANk2+cpZyWe743xqixFXI/gILVSmSJWdNgt7ALXzOa35II9L51YhNQxSnVQT5FNI2eJKumUoe8ZxEQzyX+wbAjd+a/g3 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: e9e91f62-cfd1-447c-ae49-08d8f886c6a2 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Apr 2021 23:01:41.9162 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: mtRvPt4cH6jTGYsw9t2W/BNP0FOT9kXls4avrOoAF4zNNkY6wEUf3v3r33zW8XKvmYSVwvl9SIove7Qk7qlmDA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3544 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9945 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 malwarescore=0 mlxscore=0 phishscore=0 bulkscore=0 spamscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104030000 definitions=main-2104050152 X-Proofpoint-GUID: YZ7Cz9YH4rU8R-bfabM-52I6uuAJ1o_p X-Proofpoint-ORIG-GUID: YZ7Cz9YH4rU8R-bfabM-52I6uuAJ1o_p X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9945 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxscore=0 suspectscore=0 spamscore=0 impostorscore=0 mlxlogscore=999 malwarescore=0 adultscore=0 lowpriorityscore=0 clxscore=1015 priorityscore=1501 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104030000 definitions=main-2104050152 X-Rspamd-Queue-Id: B8AD2FC X-Stat-Signature: 8r5fcgkjs6e5s49rm9g736oicy98a6i8 X-Rspamd-Server: rspam02 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf29; identity=mailfrom; envelope-from=""; helo=userp2120.oracle.com; client-ip=156.151.31.85 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617663720-823382 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Commit c77c0a8ac4c5 ("mm/hugetlb: defer freeing of huge pages if in non-task context") was added to address the issue of free_huge_page being called from irq context. That commit hands off free_huge_page processing to a workqueue if !in_task. However, this doesn't cover all the cases as pointed out by 0day bot lockdep report [1]. : Possible interrupt unsafe locking scenario: : : CPU0 CPU1 : ---- ---- : lock(hugetlb_lock); : local_irq_disable(); : lock(slock-AF_INET); : lock(hugetlb_lock); : : lock(slock-AF_INET); Shakeel has later explained that this is very likely TCP TX zerocopy from hugetlb pages scenario when the networking code drops a last reference to hugetlb page while having IRQ disabled. Hugetlb freeing path doesn't disable IRQ while holding hugetlb_lock so a lock dependency chain can lead to a deadlock. This commit addresses the issue by doing the following: - Make hugetlb_lock irq safe. This is mostly a simple process of changing spin_*lock calls to spin_*lock_irq* calls. - Make subpool lock irq safe in a similar manner. - Revert the !in_task check and workqueue handoff. [1] https://lore.kernel.org/linux-mm/000000000000f1c03b05bc43aadc@google.com/ Signed-off-by: Mike Kravetz Acked-by: Michal Hocko Reviewed-by: Muchun Song Reviewed-by: Oscar Salvador --- mm/hugetlb.c | 183 +++++++++++++++++--------------------------- mm/hugetlb_cgroup.c | 8 +- 2 files changed, 74 insertions(+), 117 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 93a2a11b9376..15b6e7aadb52 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -93,9 +93,10 @@ static inline bool subpool_is_free(struct hugepage_subpool *spool) return true; } -static inline void unlock_or_release_subpool(struct hugepage_subpool *spool) +static inline void unlock_or_release_subpool(struct hugepage_subpool *spool, + unsigned long irq_flags) { - spin_unlock(&spool->lock); + spin_unlock_irqrestore(&spool->lock, irq_flags); /* If no pages are used, and no other handles to the subpool * remain, give up any reservations based on minimum size and @@ -134,10 +135,12 @@ struct hugepage_subpool *hugepage_new_subpool(struct hstate *h, long max_hpages, void hugepage_put_subpool(struct hugepage_subpool *spool) { - spin_lock(&spool->lock); + unsigned long flags; + + spin_lock_irqsave(&spool->lock, flags); BUG_ON(!spool->count); spool->count--; - unlock_or_release_subpool(spool); + unlock_or_release_subpool(spool, flags); } /* @@ -156,7 +159,7 @@ static long hugepage_subpool_get_pages(struct hugepage_subpool *spool, if (!spool) return ret; - spin_lock(&spool->lock); + spin_lock_irq(&spool->lock); if (spool->max_hpages != -1) { /* maximum size accounting */ if ((spool->used_hpages + delta) <= spool->max_hpages) @@ -183,7 +186,7 @@ static long hugepage_subpool_get_pages(struct hugepage_subpool *spool, } unlock_ret: - spin_unlock(&spool->lock); + spin_unlock_irq(&spool->lock); return ret; } @@ -197,11 +200,12 @@ static long hugepage_subpool_put_pages(struct hugepage_subpool *spool, long delta) { long ret = delta; + unsigned long flags; if (!spool) return delta; - spin_lock(&spool->lock); + spin_lock_irqsave(&spool->lock, flags); if (spool->max_hpages != -1) /* maximum size accounting */ spool->used_hpages -= delta; @@ -222,7 +226,7 @@ static long hugepage_subpool_put_pages(struct hugepage_subpool *spool, * If hugetlbfs_put_super couldn't free spool due to an outstanding * quota reference, free it now. */ - unlock_or_release_subpool(spool); + unlock_or_release_subpool(spool, flags); return ret; } @@ -1407,7 +1411,7 @@ struct hstate *size_to_hstate(unsigned long size) return NULL; } -static void __free_huge_page(struct page *page) +void free_huge_page(struct page *page) { /* * Can't pass hstate in here because it is called from the @@ -1417,6 +1421,7 @@ static void __free_huge_page(struct page *page) int nid = page_to_nid(page); struct hugepage_subpool *spool = hugetlb_page_subpool(page); bool restore_reserve; + unsigned long flags; VM_BUG_ON_PAGE(page_count(page), page); VM_BUG_ON_PAGE(page_mapcount(page), page); @@ -1445,7 +1450,7 @@ static void __free_huge_page(struct page *page) restore_reserve = true; } - spin_lock(&hugetlb_lock); + spin_lock_irqsave(&hugetlb_lock, flags); ClearHPageMigratable(page); hugetlb_cgroup_uncharge_page(hstate_index(h), pages_per_huge_page(h), page); @@ -1456,66 +1461,18 @@ static void __free_huge_page(struct page *page) if (HPageTemporary(page)) { remove_hugetlb_page(h, page, false); - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); update_and_free_page(h, page); } else if (h->surplus_huge_pages_node[nid]) { /* remove the page from active list */ remove_hugetlb_page(h, page, true); - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); update_and_free_page(h, page); } else { arch_clear_hugepage_flags(page); enqueue_huge_page(h, page); - spin_unlock(&hugetlb_lock); - } -} - -/* - * As free_huge_page() can be called from a non-task context, we have - * to defer the actual freeing in a workqueue to prevent potential - * hugetlb_lock deadlock. - * - * free_hpage_workfn() locklessly retrieves the linked list of pages to - * be freed and frees them one-by-one. As the page->mapping pointer is - * going to be cleared in __free_huge_page() anyway, it is reused as the - * llist_node structure of a lockless linked list of huge pages to be freed. - */ -static LLIST_HEAD(hpage_freelist); - -static void free_hpage_workfn(struct work_struct *work) -{ - struct llist_node *node; - struct page *page; - - node = llist_del_all(&hpage_freelist); - - while (node) { - page = container_of((struct address_space **)node, - struct page, mapping); - node = node->next; - __free_huge_page(page); - } -} -static DECLARE_WORK(free_hpage_work, free_hpage_workfn); - -void free_huge_page(struct page *page) -{ - /* - * Defer freeing if in non-task context to avoid hugetlb_lock deadlock. - */ - if (!in_task()) { - /* - * Only call schedule_work() if hpage_freelist is previously - * empty. Otherwise, schedule_work() had been called but the - * workfn hasn't retrieved the list yet. - */ - if (llist_add((struct llist_node *)&page->mapping, - &hpage_freelist)) - schedule_work(&free_hpage_work); - return; + spin_unlock_irqrestore(&hugetlb_lock, flags); } - - __free_huge_page(page); } static void prep_new_huge_page(struct hstate *h, struct page *page, int nid) @@ -1525,11 +1482,11 @@ static void prep_new_huge_page(struct hstate *h, struct page *page, int nid) hugetlb_set_page_subpool(page, NULL); set_hugetlb_cgroup(page, NULL); set_hugetlb_cgroup_rsvd(page, NULL); - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); h->nr_huge_pages++; h->nr_huge_pages_node[nid]++; ClearHPageFreed(page); - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); } static void prep_compound_gigantic_page(struct page *page, unsigned int order) @@ -1775,7 +1732,7 @@ int dissolve_free_huge_page(struct page *page) if (!PageHuge(page)) return 0; - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); if (!PageHuge(page)) { rc = 0; goto out; @@ -1792,7 +1749,7 @@ int dissolve_free_huge_page(struct page *page) * when it is dissolved. */ if (unlikely(!HPageFreed(head))) { - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); cond_resched(); /* @@ -1816,12 +1773,12 @@ int dissolve_free_huge_page(struct page *page) } remove_hugetlb_page(h, page, false); h->max_huge_pages--; - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); update_and_free_page(h, head); return 0; } out: - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); return rc; } @@ -1863,16 +1820,16 @@ static struct page *alloc_surplus_huge_page(struct hstate *h, gfp_t gfp_mask, if (hstate_is_gigantic(h)) return NULL; - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); if (h->surplus_huge_pages >= h->nr_overcommit_huge_pages) goto out_unlock; - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); page = alloc_fresh_huge_page(h, gfp_mask, nid, nmask, NULL); if (!page) return NULL; - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); /* * We could have raced with the pool size change. * Double check that and simply deallocate the new page @@ -1882,7 +1839,7 @@ static struct page *alloc_surplus_huge_page(struct hstate *h, gfp_t gfp_mask, */ if (h->surplus_huge_pages >= h->nr_overcommit_huge_pages) { SetHPageTemporary(page); - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); put_page(page); return NULL; } else { @@ -1891,7 +1848,7 @@ static struct page *alloc_surplus_huge_page(struct hstate *h, gfp_t gfp_mask, } out_unlock: - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); return page; } @@ -1941,17 +1898,17 @@ struct page *alloc_buddy_huge_page_with_mpol(struct hstate *h, struct page *alloc_huge_page_nodemask(struct hstate *h, int preferred_nid, nodemask_t *nmask, gfp_t gfp_mask) { - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); if (h->free_huge_pages - h->resv_huge_pages > 0) { struct page *page; page = dequeue_huge_page_nodemask(h, gfp_mask, preferred_nid, nmask); if (page) { - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); return page; } } - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); return alloc_migrate_huge_page(h, gfp_mask, preferred_nid, nmask); } @@ -1999,7 +1956,7 @@ static int gather_surplus_pages(struct hstate *h, long delta) ret = -ENOMEM; retry: - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); for (i = 0; i < needed; i++) { page = alloc_surplus_huge_page(h, htlb_alloc_mask(h), NUMA_NO_NODE, NULL); @@ -2016,7 +1973,7 @@ static int gather_surplus_pages(struct hstate *h, long delta) * After retaking hugetlb_lock, we need to recalculate 'needed' * because either resv_huge_pages or free_huge_pages may have changed. */ - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); needed = (h->resv_huge_pages + delta) - (h->free_huge_pages + allocated); if (needed > 0) { @@ -2056,12 +2013,12 @@ static int gather_surplus_pages(struct hstate *h, long delta) enqueue_huge_page(h, page); } free: - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); /* Free unnecessary surplus pages to the buddy allocator */ list_for_each_entry_safe(page, tmp, &surplus_list, lru) put_page(page); - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); return ret; } @@ -2111,9 +2068,9 @@ static void return_unused_surplus_pages(struct hstate *h, } out: - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); update_and_free_pages_bulk(h, &page_list); - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); } @@ -2316,7 +2273,7 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, */ page_ref_dec(new_page); retry: - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); if (!PageHuge(old_page)) { /* * Freed from under us. Drop new_page too. @@ -2330,7 +2287,7 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, * Fail with -EBUSY if not possible. */ remove_hugetlb_page(h, new_page, false); - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); update_and_free_page(h, new_page); if (!isolate_huge_page(old_page, list)) ret = -EBUSY; @@ -2341,7 +2298,7 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, * freelist yet. Race window is small, so we can succed here if * we retry. */ - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); cond_resched(); goto retry; } else { @@ -2357,7 +2314,7 @@ static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *old_page, page_to_free = old_page; } unlock_free: - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); update_and_free_page(h, page_to_free); return ret; @@ -2374,15 +2331,15 @@ int isolate_or_dissolve_huge_page(struct page *page, struct list_head *list) * to carefully check the state under the lock. * Return success when racing as if we dissolved the page ourselves. */ - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); if (PageHuge(page)) { head = compound_head(page); h = page_hstate(head); } else { - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); return 0; } - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); /* * Fence off gigantic pages as there is a cyclic dependency between @@ -2462,7 +2419,7 @@ struct page *alloc_huge_page(struct vm_area_struct *vma, if (ret) goto out_uncharge_cgroup_reservation; - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); /* * glb_chg is passed to indicate whether or not a page must be taken * from the global free pool (global change). gbl_chg == 0 indicates @@ -2470,7 +2427,7 @@ struct page *alloc_huge_page(struct vm_area_struct *vma, */ page = dequeue_huge_page_vma(h, vma, addr, avoid_reserve, gbl_chg); if (!page) { - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); page = alloc_buddy_huge_page_with_mpol(h, vma, addr); if (!page) goto out_uncharge_cgroup; @@ -2478,7 +2435,7 @@ struct page *alloc_huge_page(struct vm_area_struct *vma, SetHPageRestoreReserve(page); h->resv_huge_pages--; } - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); list_add(&page->lru, &h->hugepage_activelist); /* Fall through */ } @@ -2491,7 +2448,7 @@ struct page *alloc_huge_page(struct vm_area_struct *vma, h_cg, page); } - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); hugetlb_set_page_subpool(page, spool); @@ -2703,9 +2660,9 @@ static void try_to_free_low(struct hstate *h, unsigned long count, } out: - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); update_and_free_pages_bulk(h, &page_list); - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); } #else static inline void try_to_free_low(struct hstate *h, unsigned long count, @@ -2770,7 +2727,7 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, * pages in hstate via the proc/sysfs interfaces. */ mutex_lock(&h->resize_lock); - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); /* * Check for a node specific request. @@ -2801,7 +2758,7 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, */ if (hstate_is_gigantic(h) && !IS_ENABLED(CONFIG_CONTIG_ALLOC)) { if (count > persistent_huge_pages(h)) { - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); mutex_unlock(&h->resize_lock); NODEMASK_FREE(node_alloc_noretry); return -EINVAL; @@ -2831,14 +2788,14 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, * page, free_huge_page will handle it by freeing the page * and reducing the surplus. */ - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); /* yield cpu to avoid soft lockup */ cond_resched(); ret = alloc_pool_huge_page(h, nodes_allowed, node_alloc_noretry); - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); if (!ret) goto out; @@ -2877,9 +2834,9 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, list_add(&page->lru, &page_list); } /* free the pages after dropping lock */ - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); update_and_free_pages_bulk(h, &page_list); - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); while (count < persistent_huge_pages(h)) { if (!adjust_pool_surplus(h, nodes_allowed, 1)) @@ -2887,7 +2844,7 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, } out: h->max_huge_pages = persistent_huge_pages(h); - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); mutex_unlock(&h->resize_lock); NODEMASK_FREE(node_alloc_noretry); @@ -3043,9 +3000,9 @@ static ssize_t nr_overcommit_hugepages_store(struct kobject *kobj, if (err) return err; - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); h->nr_overcommit_huge_pages = input; - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); return count; } @@ -3632,9 +3589,9 @@ int hugetlb_overcommit_handler(struct ctl_table *table, int write, goto out; if (write) { - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); h->nr_overcommit_huge_pages = tmp; - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); } out: return ret; @@ -3730,7 +3687,7 @@ static int hugetlb_acct_memory(struct hstate *h, long delta) if (!delta) return 0; - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); /* * When cpuset is configured, it breaks the strict hugetlb page * reservation as the accounting is done on a global variable. Such @@ -3769,7 +3726,7 @@ static int hugetlb_acct_memory(struct hstate *h, long delta) return_unused_surplus_pages(h, (unsigned long) -delta); out: - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); return ret; } @@ -5833,7 +5790,7 @@ bool isolate_huge_page(struct page *page, struct list_head *list) { bool ret = true; - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); if (!PageHeadHuge(page) || !HPageMigratable(page) || !get_page_unless_zero(page)) { @@ -5843,16 +5800,16 @@ bool isolate_huge_page(struct page *page, struct list_head *list) ClearHPageMigratable(page); list_move_tail(&page->lru, list); unlock: - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); return ret; } void putback_active_hugepage(struct page *page) { - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); SetHPageMigratable(page); list_move_tail(&page->lru, &(page_hstate(page))->hugepage_activelist); - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); put_page(page); } @@ -5886,12 +5843,12 @@ void move_hugetlb_state(struct page *oldpage, struct page *newpage, int reason) */ if (new_nid == old_nid) return; - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); if (h->surplus_huge_pages_node[old_nid]) { h->surplus_huge_pages_node[old_nid]--; h->surplus_huge_pages_node[new_nid]++; } - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); } } diff --git a/mm/hugetlb_cgroup.c b/mm/hugetlb_cgroup.c index 726b85f4f303..5383023d0cca 100644 --- a/mm/hugetlb_cgroup.c +++ b/mm/hugetlb_cgroup.c @@ -204,11 +204,11 @@ static void hugetlb_cgroup_css_offline(struct cgroup_subsys_state *css) do { idx = 0; for_each_hstate(h) { - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); list_for_each_entry(page, &h->hugepage_activelist, lru) hugetlb_cgroup_move_parent(idx, h_cg, page); - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); idx++; } cond_resched(); @@ -784,7 +784,7 @@ void hugetlb_cgroup_migrate(struct page *oldhpage, struct page *newhpage) if (hugetlb_cgroup_disabled()) return; - spin_lock(&hugetlb_lock); + spin_lock_irq(&hugetlb_lock); h_cg = hugetlb_cgroup_from_page(oldhpage); h_cg_rsvd = hugetlb_cgroup_from_page_rsvd(oldhpage); set_hugetlb_cgroup(oldhpage, NULL); @@ -794,7 +794,7 @@ void hugetlb_cgroup_migrate(struct page *oldhpage, struct page *newhpage) set_hugetlb_cgroup(newhpage, h_cg); set_hugetlb_cgroup_rsvd(newhpage, h_cg_rsvd); list_move(&newhpage->lru, &h->hugepage_activelist); - spin_unlock(&hugetlb_lock); + spin_unlock_irq(&hugetlb_lock); return; } From patchwork Mon Apr 5 23:00:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12183819 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3185C433ED for ; Mon, 5 Apr 2021 23:02:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5896561205 for ; Mon, 5 Apr 2021 23:02:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5896561205 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B38EF6B0081; Mon, 5 Apr 2021 19:02:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B11AB6B0082; Mon, 5 Apr 2021 19:02:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 93B8A6B0083; Mon, 5 Apr 2021 19:02:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0035.hostedemail.com [216.40.44.35]) by kanga.kvack.org (Postfix) with ESMTP id 6331C6B0082 for ; Mon, 5 Apr 2021 19:02:04 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 11625180ACEE8 for ; Mon, 5 Apr 2021 23:02:04 +0000 (UTC) X-FDA: 77999838168.28.20C5770 Received: from aserp2120.oracle.com (aserp2120.oracle.com [141.146.126.78]) by imf29.hostedemail.com (Postfix) with ESMTP id EDA93F4 for ; Mon, 5 Apr 2021 23:02:01 +0000 (UTC) Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 135N09Em118942; Mon, 5 Apr 2021 23:01:49 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=rN1Ph4/WEiFUVkJO1x0Thnt6ZIpHDaZzFrTDFQd0u4o=; b=beXQvoP+z4PmUOdp2WAEYhHtuRzj8yDTClMyT/WB9XWUXnvtaLweH0myq5FVyq2X0mMM PvfOgtjObP0c9KYh4sQnUYd9XxwI1jS3xxK/BmF1VrF09loshynlgG0Z00mxvLSstUl2 dolLooVcW9fx2nPfaLAKXBX1wpYhnUXgw1P7dQvituhgrBWkrxm7GcgH9t+RSUPajE7c rpY3P/qMzqFdQoQfUdeqYh4+VjtnW7pfDP4P9y7R+YOiZr8sVJoHXzHAWZeCXJuGhmi1 8jAfPCse9gxR5Z87+pY+ecrQcHs2t4ESi1UEW4ZjmNzOYkkEJdFLg/ll5CB/l0PiiO+S Bw== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by aserp2120.oracle.com with ESMTP id 37q3bwkk9d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 05 Apr 2021 23:01:49 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 135MtpU6089329; Mon, 5 Apr 2021 23:01:48 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2172.outbound.protection.outlook.com [104.47.58.172]) by userp3030.oracle.com with ESMTP id 37qa3hp3j0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 05 Apr 2021 23:01:48 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=a71MKINqrnIzXx26JKOuSPwqx8sVBrxMWqtQkmi1pIP+XwDcHa8TFcsKpob7A4tPD9oe+cLyt1repTGtKiI0/01u2WolP9vgf4zlDbsDjgvGraEQvKt3gKQsv/0QiJ1iUBb7pR4v55xPyQMqaX0jRLVgLFTUtXfkBbSQWrkKZllr553UVyjswTw3lFTlEWUIqSJ6OIMKIfAAMf0tVGvv3x/MvOocQkGizJwWjDnhSiqR6xQBbB4U9z6dQ/a/yWOVZCR6+fyFWJidoNjnp0pNrTRttFWxhRPm6eHMxz/R4CiD98oioIKaBOdXlZZbwuU3hb/vMt/QZsSR1pB3BvI2VQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rN1Ph4/WEiFUVkJO1x0Thnt6ZIpHDaZzFrTDFQd0u4o=; b=mFSwssSVfyeWiriEredcL/Crr/NaRIQsdlWnW84mJ1ywE7st4mX509NYd518lfMTHdl+AdYESFPSFp/++2yXGxDt2cJ5aCkHaPNIjfGw1xcnNqEr/r4Q3kdZi4KjN429Othu9YYeN3w0spV/bibiMW6D3QdJMX5AXdAF0VqeMNKA3ASSWNhGOOQq7KhVCXekvFvZCrU9BfQ+kwXJJrxhzRnL3AbEGqwoGAK4+utzx6B4DMhiEXhGmGnRW3lF/l2AIn/Z9ko0O+Xu5QQBfmfFtmX7Qst8sKHLAsiuDJQzr6+enWD4pjP1Ri2q1I7dTp9x7UZejej2Jtusyt1mD/DuKA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rN1Ph4/WEiFUVkJO1x0Thnt6ZIpHDaZzFrTDFQd0u4o=; b=F2y0ipls3Q+dfujDVoQvA+3P2wwHNyTv4sQNAmMrcoVQ1e+owC/vEOWQtOiXc8atI5uyTQU6bw3TDdjLxd/ojqoBQhicxR+K6Cj99RFTGfcq/P31ejZciIsJtOmMJl7ZgrqQ8ElPWFuwC/2G0xGOrp/h+0ZsTlNFb/gDaSWQ7MI= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3544.namprd10.prod.outlook.com (2603:10b6:a03:121::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.26; Mon, 5 Apr 2021 23:01:45 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3999.032; Mon, 5 Apr 2021 23:01:44 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Roman Gushchin , Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Hillf Danton , Joonsoo Kim , Barry Song , Will Deacon , Andrew Morton , Mike Kravetz Subject: [PATCH v4 8/8] hugetlb: add lockdep_assert_held() calls for hugetlb_lock Date: Mon, 5 Apr 2021 16:00:43 -0700 Message-Id: <20210405230043.182734-9-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210405230043.182734-1-mike.kravetz@oracle.com> References: <20210405230043.182734-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR18CA0031.namprd18.prod.outlook.com (2603:10b6:320:31::17) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR18CA0031.namprd18.prod.outlook.com (2603:10b6:320:31::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.27 via Frontend Transport; Mon, 5 Apr 2021 23:01:43 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: bdedd394-b14a-4dff-6156-08d8f886c869 X-MS-TrafficTypeDiagnostic: BYAPR10MB3544: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3826; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 7+jQ0DfLALoQKk+ZpYs5HUCbQWFbNwR+uN2ONDIe5FPn+NZ2KI3LxTdwa9hfB+SenzlqfBvvmyJcZmluyE2w83vaSsZClF6AUwR5JZysG1NA0ZaNgeaEJXwY9u9l89FQqYhxEPxCUd/OxZhtZM859G7Yu4iMAR7nHbQV7whaGX6LdaJfVBMVmckYZMaqQIwrzp/nVYnTm435D3MGRnEOdYUAYxjNffQhiMFQJ8176tm7amYbny5GjI8CdaIXJjs5uo5GPK5nNnGQspLty+p0mLxlcyGPqt1Q0KppqRVh0sMcHjQSdGMHN8XjdDSgN140oT6lKioZceCPUg6eUWDAv0xUHoNk13q17k3UME6B0FV5TuKjOQ6qQiiIrj3oQDp3MNc01rDFEYYIX8HxxmL3rjKa8yi5tgHi0m7PVWAdfnh+8VErNUs5W9TQUx+l7QqguxBaPTLSom+H++pKIu1esdpVu5AgUAiYGI1xen4uAEQeOVcIx7lkz6UeMrfvurjGbIdC1G2Kl+A5K/BwkB3IU3ycAnnS8DtZzTA9Im39gVFXcl9ffjkRQ4vBcrX0DnZ567svzETcE/6JhpZdElg9ujPumD6w6qNBc/aZSh7OT3oUzqL7bvetRA7ZgMrS5bgHUHBXv3FDKKsGE+YqzXAMp7Lk4VApjhJPVby3LBZSN8r0WGYoAdEqxi8q7qGvZm8H X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(396003)(376002)(366004)(39860400002)(346002)(2616005)(44832011)(66476007)(5660300002)(107886003)(8676002)(86362001)(26005)(6486002)(66946007)(16526019)(38350700001)(1076003)(2906002)(54906003)(83380400001)(4326008)(52116002)(7696005)(186003)(7416002)(478600001)(38100700001)(8936002)(36756003)(316002)(66556008)(956004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: GOZlXb0PQrB9U2r/Uxws8KrRZ0ckgpjlNVIcLsRmpAItlHq5TwI8ptEL+B964+cGcyY1ECZCmm489YDNTFnV+CHtrnUubV61nnt+ex0R6ppuBEtSQJ8Zqswaw7XjVgnYkjFs8V70e839cVV8h/Tx86KMAtB1EzXj1LQ1ITaRBrngN+lfvTfbUJ3NtMCdeMU8BxxS2FMShLHBta6a/7d0Vn1l/YQ0NjacPjpi5osdE6keQ6wjq4Vc8gFVizuqFhr8pT4aHZwUpZvYZIObzv1pT6jKgA5lfEXGeHCicWvHr+WGYkPz1JO3loUDuGTexSwcJduS7jPeMoVMDRjlgZnRRuo5MX5NZXgD0BXJmas45T2hT4TnEICPjP6x5RRnaulwTV2Y8zs20JkVY1yeKz4R2dnDeZhP1oWcaxYuFXv7PZ0O2lpKUzFYHKiTksytOQ1fTw5yj7HUFU3rT/WSCXA6LMsCMQsxbO7GIfI/A0VFJiOa5Nodzipfq2H3/M20AR0D3BMOB4RcLVEpaD8x4B2oBnpNfgWb2JtBTAva+++VC16FYBDOElUL038HKBTdS25L12xqIWHQSLHRSHq6TPf3kwRif+pz6zyKPvNolxd1ekUcwwPiOOhAocbQ2qieBzoQr5eBVu4qCzuWA3jVBb2VSp6QLPLaceFc3WFbHBokV9yHM3XEY2FvMmsxIMWr3l8/9iDR9spvD43+Uq+j57LxpZv6iIEN3UqWGfEJ5Ac3vt7HKIPTK3U5c9IqClDyaeg3GCUy3CfCdPJ4i4zy4V+6S3OipNzPtmXYjwbWn50GQtDAEQgeBioLP9oNGZ15m4V5V6ZNc7/wRB3YgAJVDS3qX8qMJmdL3PN3EzGUNHScahfDFw3yNe2ayghdJRDzvw+hY07wnW7/4ZglP7NxTfOSg6HPr4TE/KpbnRaiirv0CH2Ndo5uB/CsNsi/YK3hUTURc6QRyNfnK66BiV/ztdcM8ZuIq1gCB+hPKl8VBw9j+TZFHHt57om5l7s0biG+FIC1hoO3xOu+Nts6V8iFI3xBeRdXWcTj956nBdnxjHzk8GAh3pTMoT8+5aLVXz5AhLSbCm9iK8ozZw2UgCTEtm64Oa1qb2fg95OrjK2xwD1Fvz/TsVmjWwP/0u1NSah7hzrluxyUPwRE5TVDUfNHwC9Ot6HxIj3tqV/mFdMkeD3n6D0SRU5V34ZrhR+8c0qXrjpyjHgdyNhxPXRQJmFdnI/9RCNaw4HIgSVTYVgvntadmbsq3NCWYYalSjTyNZu4bE6lqcB+j6n3aIZVkIfJ+kh3JcAalKUTkPShNPfxAxl+YWO4y23uaAvQW8GZuGYiTwXK X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: bdedd394-b14a-4dff-6156-08d8f886c869 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Apr 2021 23:01:44.8742 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 9WlS1JCoBKM70EyETrON0/YlRYTWAWs2RiWAID3ASBSrJpCveZMTsXS89HZrjfDOPTzMCLSCEdZamtwMfMWIvQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3544 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9945 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=0 malwarescore=0 mlxscore=0 spamscore=0 phishscore=0 adultscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104030000 definitions=main-2104050152 X-Proofpoint-GUID: JmmVlhfpD62OQrU6c3vtPwg2ciht1v65 X-Proofpoint-ORIG-GUID: JmmVlhfpD62OQrU6c3vtPwg2ciht1v65 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9945 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 spamscore=0 lowpriorityscore=0 suspectscore=0 adultscore=0 phishscore=0 mlxscore=0 priorityscore=1501 clxscore=1015 impostorscore=0 bulkscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104030000 definitions=main-2104050152 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: EDA93F4 X-Stat-Signature: eyt3kkb8mabtpzugmn8hojufxgcur9u7 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf29; identity=mailfrom; envelope-from=""; helo=aserp2120.oracle.com; client-ip=141.146.126.78 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617663721-24623 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: After making hugetlb lock irq safe and separating some functionality done under the lock, add some lockdep_assert_held to help verify locking. Signed-off-by: Mike Kravetz Acked-by: Michal Hocko Reviewed-by: Miaohe Lin Reviewed-by: Muchun Song Reviewed-by: Oscar Salvador --- mm/hugetlb.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 15b6e7aadb52..5d9f74e2b963 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1062,6 +1062,8 @@ static bool vma_has_reserves(struct vm_area_struct *vma, long chg) static void enqueue_huge_page(struct hstate *h, struct page *page) { int nid = page_to_nid(page); + + lockdep_assert_held(&hugetlb_lock); list_move(&page->lru, &h->hugepage_freelists[nid]); h->free_huge_pages++; h->free_huge_pages_node[nid]++; @@ -1073,6 +1075,7 @@ static struct page *dequeue_huge_page_node_exact(struct hstate *h, int nid) struct page *page; bool pin = !!(current->flags & PF_MEMALLOC_PIN); + lockdep_assert_held(&hugetlb_lock); list_for_each_entry(page, &h->hugepage_freelists[nid], lru) { if (pin && !is_pinnable_page(page)) continue; @@ -1341,6 +1344,7 @@ static void remove_hugetlb_page(struct hstate *h, struct page *page, { int nid = page_to_nid(page); + lockdep_assert_held(&hugetlb_lock); if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported()) return; @@ -1696,6 +1700,7 @@ static struct page *remove_pool_huge_page(struct hstate *h, int nr_nodes, node; struct page *page = NULL; + lockdep_assert_held(&hugetlb_lock); for_each_node_mask_to_free(h, nr_nodes, node, nodes_allowed) { /* * If we're returning unused surplus pages, only examine @@ -1945,6 +1950,7 @@ static int gather_surplus_pages(struct hstate *h, long delta) long needed, allocated; bool alloc_ok = true; + lockdep_assert_held(&hugetlb_lock); needed = (h->resv_huge_pages + delta) - h->free_huge_pages; if (needed <= 0) { h->resv_huge_pages += delta; @@ -2038,6 +2044,7 @@ static void return_unused_surplus_pages(struct hstate *h, struct page *page; LIST_HEAD(page_list); + lockdep_assert_held(&hugetlb_lock); /* Uncommit the reservation */ h->resv_huge_pages -= unused_resv_pages; @@ -2640,6 +2647,7 @@ static void try_to_free_low(struct hstate *h, unsigned long count, int i; LIST_HEAD(page_list); + lockdep_assert_held(&hugetlb_lock); if (hstate_is_gigantic(h)) return; @@ -2681,6 +2689,7 @@ static int adjust_pool_surplus(struct hstate *h, nodemask_t *nodes_allowed, { int nr_nodes, node; + lockdep_assert_held(&hugetlb_lock); VM_BUG_ON(delta != -1 && delta != 1); if (delta < 0) {