From patchwork Fri Mar 19 22:42:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12152069 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E4A3C433E0 for ; Fri, 19 Mar 2021 22:43:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D82A96197E for ; Fri, 19 Mar 2021 22:43:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D82A96197E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7CA4B6B0074; Fri, 19 Mar 2021 18:43:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7A3586B0075; Fri, 19 Mar 2021 18:43:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5CC786B0078; Fri, 19 Mar 2021 18:43:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0144.hostedemail.com [216.40.44.144]) by kanga.kvack.org (Postfix) with ESMTP id 3ED2A6B0074 for ; Fri, 19 Mar 2021 18:43:14 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 010C3180373F9 for ; Fri, 19 Mar 2021 22:43:14 +0000 (UTC) X-FDA: 77938101108.28.87FB8CA Received: from aserp2120.oracle.com (aserp2120.oracle.com [141.146.126.78]) by imf26.hostedemail.com (Postfix) with ESMTP id 5EB2D40002C1 for ; Fri, 19 Mar 2021 22:43:13 +0000 (UTC) Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12JMZBiI128000; Fri, 19 Mar 2021 22:42:30 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=cUKW1HQalMJtxv8VJaWBtV7q/POIHFn8Vai9tFs6JPo=; b=MG/WCngVoZq9GLq1hevdAxA1qJcimW5Ki5urtJ1+J7wCZcBb0DYIQ2fdg7hFuiySk/kA IGRGCq1mUvttY+zwckQ8cC0oFjFUTlK7UsQu76BzMAa1jh7sklnnlHFV7NLCR3FL4+mG nsJyhEuFVm2NA6aZgek74UWfSY0e6hnrC4fI15xeQuSz8kU3+wnnksp8KdKLpyDUdur/ KWaVz81PquqlKHjp6sLRKI3Oi5PasrTaC+gy477iZ+eWyAw8dIoTCYl6DBpIOEc9oyPV EE2aVNnYtr9ymczg5adL4uoo9IrI2/JmhjtHskcQdR1z/FPuDL9ZqQ17rkyYoxrUOy/1 qA== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by aserp2120.oracle.com with ESMTP id 378nbmmg7q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 19 Mar 2021 22:42:30 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12JMeGb7155442; Fri, 19 Mar 2021 22:42:29 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2048.outbound.protection.outlook.com [104.47.66.48]) by userp3030.oracle.com with ESMTP id 3797b4sr31-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 19 Mar 2021 22:42:29 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BkiqgLELfct9cxf+6PqO8bS0bSD3NA0qj20UhzZTO3FmHZf01RzIUjn+8OvQqUrz5IkRbTpV/sft2esmbTSABPD9diwXY+JGJTnj0y1ofkyIA7kUdsenE/dja33SLNspwPUvdjFQz+V2gDZ1CCvwN/kUieloTwSWf5+jwHpyZ/NUdr+sVm7jFuxaDZhn0Pce8TMHb3hsPzcedXHXu/5FztREXnnrmo7nej9ueF3H0ml4EMHXYRHLVJeJHpKewmSXpWJ7tLSudCIJqyIg75wXA2/rvJMxe+s1oTvxLisroKjHx17se9rcSB29b4ISuYRtvfBA8SldKD08J/qxY6EooA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cUKW1HQalMJtxv8VJaWBtV7q/POIHFn8Vai9tFs6JPo=; b=kA/SaXljtyBCExoSWHn4C26zWRjoRzdiAOJCGN7iDeFrLsh/V8e48oZafj1CKlIsatcMbu92vy9gK4U06sg/7Tty318yfycVja/5YlPvqIlrwPzhA7wrFdBmn1f3U7JBkOhb4EkJbXtJfcXEQxcsONh8O5zF5icJIwHmZNmlEegv2zNonKOfmTRG7eYRT8S10FVWdgDRETSQCwAN9ejmagS1DPusRyr4Ws0HxMKLEF1eLRs5i4agfffYZT6VzALtKxt3uz94TAH079uZUlKVi2cTp4riZD95VFVhqVsm+vO9QG+rS60hlXAapKRV9hQMQDaA7ODfxKkoKj1k5ERkIA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cUKW1HQalMJtxv8VJaWBtV7q/POIHFn8Vai9tFs6JPo=; b=sfOZ27xN4Se6bxGCzVNU89My46pezxRcWeTZkgLsE7B0+v2CPeUwT66DjxxFj5dIAnAHvpWora+3LrOl5AXEaStSdWPB0OHMYsa82ZOFC1q6qt9PidrJSqE2fVy80eg1v+6AQH+elKj1CxMi8i6eNJU5NdaVTWqSMIQP6Apm3p4= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3160.namprd10.prod.outlook.com (2603:10b6:a03:151::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.23; Fri, 19 Mar 2021 22:42:26 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3955.024; Fri, 19 Mar 2021 22:42:26 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Andrew Morton , Mike Kravetz Subject: [RFC PATCH 1/8] hugetlb: add per-hstate mutex to synchronize user adjustments Date: Fri, 19 Mar 2021 15:42:02 -0700 Message-Id: <20210319224209.150047-2-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210319224209.150047-1-mike.kravetz@oracle.com> References: <20210319224209.150047-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR07CA0013.namprd07.prod.outlook.com (2603:10b6:300:116::23) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR07CA0013.namprd07.prod.outlook.com (2603:10b6:300:116::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18 via Frontend Transport; Fri, 19 Mar 2021 22:42:25 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 31d705ab-dea5-4eae-d482-08d8eb284532 X-MS-TrafficTypeDiagnostic: BYAPR10MB3160: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5236; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4e4WXJcbT4RmiKO7tQpewKR56jMVnHgEszWSjRgJGRDJLbebo+x0HqbLmZBwfbZ6cyFrNbRpasthz6HZcYvUoD0UVzEV5tzR5mmkQzaDoOqh3w6ElDhYK/PSHlQRuHEV00SPuQ75rHXvj4xVv9SdrezCpMeEKYye/y2jJGHqnUnjVptMti/YuDp9MvZCGgNI2IAzUkewADhrTOYukRA7DHMKPBSdSvKxMpBcnZkeRdUN1bMjXWue4xEMm2/ClCRO4WdnXwpbAPBVtwNrF1/tMx5QIbsdFXjR8OoJ0wwYWEW+OppXcpAjnLd/Op7w430y7ES7iMjCySEipu+2J61eUmHpOavorFqAQjxSdqQxeWPCpTtU7u94NGFhJB0iKZH6OFwPECp97XdcoWUxwKrC7zhAt/f2BIFw1835aVKKjHY7wPiW1LjmkJEUVMJoDKNG6THMcqjkCFFobQGbxIckT1SBJdQu43auxLH3vNLYFDE4SRHksM05Gkzx71Ofddu0beZDPEHhlb3kFON0ZgxSsx18qUEylJfUX3uszqSI1IThcfj576zlKmD88iLcezUErrs6AJZhRy9nUkmiUKq/3AsWS2U1n1Ttei7wmqA2skbhuHu2e0zIkFQMAHAJ24JJuOCk66wrweFAcfybd/jTcOgPpuStHXgcdMECycwMVFY= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(396003)(39860400002)(346002)(136003)(366004)(376002)(8936002)(316002)(5660300002)(66946007)(36756003)(4326008)(186003)(66556008)(26005)(66476007)(54906003)(6486002)(16526019)(52116002)(7696005)(83380400001)(8676002)(956004)(44832011)(107886003)(7416002)(1076003)(2616005)(38100700001)(86362001)(478600001)(2906002)(6666004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: B0nbODihgoaokyVDxOoN3Lry6cPs3o9pSrdALCz0aSS4e9sr17mHZN0D6JILJFcYVDkkLqc5hD3ad6LUYEc/ykis09OF///yMjWFS7FpwuXLUu3N08K1emPSY6G/IhxnD0Ozu6KK4RtuZCCU+jbB7xK8NAQID21uaJP0UDSOSpveBEMHES4HZ1dW6vpwGfWctIkv2VuLWKgGt61ANU0n3U+x20e284YDdKPy62urMjR5VPzdykupIaIAvuvkG58ihiKDVFfVocMiiRsMeFE9hgeuukniP6ZnpZVOWVkAJ68DbkWCvBtzSpNfhhgpDFQFrquotiJ+wChXIuQ9rcuZJZPjxj3zlNEFBa0KVFB2rW/qhRDwn/uuQyKFfB8NTZblRVM10o2RQRUsF1q+qV5NsrwltsSvUScvKGScB2w1sxeltq1EKGgQFMc4Y4p7fteS5CaM1iHfdrZiMTuXnThj7ClJTryk85xz+YQyElUva2FWyKlx6bSBKLNEJBgcsVLj/2aJocLMNnj+TaFg05ReDbD2vV8VPOS/jHyWf1h7aQAAxRQdwPh3+4fLRVAr6n4MG1Jyh+A7D2rlY659MdE5+J9J/gb/kFG90z+nUIVd2wspiIl5qXa5XNc/ATb3eMgBpPXoVtJWEuJkwFkk5WYf8ZWghn1sojFNVRDcIykvUBP+rFnGs2EUFx7t9M/qvwR9F0arJJ45DUSM5ZSbf7Fxf/fIzmrGeYG3wBxzgC7VsNE+stWW658obZskrYQnq3OZwNVU2R0Fmx7vQFvbK63psLF/39QsPwOb22SKUr5GmRcj5doo7n0Tim1zzPkG1WyVGacgaPzzUjLj+nEPqv6/6Mqsp1fgS8TjE1bFm6utgIWP8w6LiSpByaT6oPDAV+955lUysCjFI8Qf7tmYC/iMC1CxssnOXE5wzjGrBMIVgiVHUtuBXkpHOfpi0n+4q9ZjlTQH3lxZHHGFm1CBRom+OcNQujBrQOBOif+rwq8CzlZktx1BwkaOnD9yhM+0DMaw0UVjHL0VlvdQbjeaJyRk+ji+YyOwjL76HeCJ06v3tHMDkMzsT0GHJgrOUCA2E/zBcK/EtudWsBM8w+dK3P2laYnfRz3xTdUuyMyoH8fWqe90IJRjyHXaenV1umte8Sy5BBUDk9SbRCl8MQQvDLkYloaFTeyvHPqPNxC2SZyg3oQq1xxorISJXtD8SDg13DtIaOyXA34kkr32lxoa9CTahOvGch1Wj2O8hlbRSgoRtzI1peS5loD3XNyJp5+IoGnqCOFJZOJCgK4A1ufcXZ0kKIoiNVf7m3lUnPTAJG3F8+awJJqQltT2r4D1YNiSVgBm X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 31d705ab-dea5-4eae-d482-08d8eb284532 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2021 22:42:26.5780 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: FY9bAJwQUYfY/W+dz2hQTq5/NHldNowSM2Y4aUeZ8iV/FjvXYL24MEZMBI+tg5lVMaVDhqxq7YCZwbFKig65ug== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3160 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9928 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 spamscore=0 bulkscore=0 malwarescore=0 adultscore=0 mlxscore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103190156 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9928 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 impostorscore=0 malwarescore=0 adultscore=0 mlxscore=0 clxscore=1015 mlxlogscore=999 lowpriorityscore=0 phishscore=0 priorityscore=1501 spamscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103190155 X-Stat-Signature: thgi56bausjxmjc79iiemstf661bd3se X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 5EB2D40002C1 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf26; identity=mailfrom; envelope-from=""; helo=aserp2120.oracle.com; client-ip=141.146.126.78 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616193793-740110 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The number of hugetlb pages can be adjusted by writing to the sysps/proc files nr_hugepages, nr_hugepages_mempolicy or nr_overcommit_hugepages. There is nothing to prevent two concurrent modifications via these files. The underlying routine set_max_huge_pages() makes assumptions that only one occurrence is running at a time. Specifically, alloc_pool_huge_page uses a hstate specific variable without any synchronization. Add a mutex to the hstate and use it to only allow one hugetlb page adjustment at a time. Signed-off-by: Mike Kravetz --- include/linux/hugetlb.h | 1 + mm/hugetlb.c | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index cccd1aab69dd..f42d44050548 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -555,6 +555,7 @@ HPAGEFLAG(Freed, freed) #define HSTATE_NAME_LEN 32 /* Defines one hugetlb page size */ struct hstate { + struct mutex mutex; int next_nid_to_alloc; int next_nid_to_free; unsigned int order; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 5b1ab1f427c5..d5be25f910e8 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2601,6 +2601,8 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, else return -ENOMEM; + /* mutex prevents concurrent adjustments for the same hstate */ + mutex_lock(&h->mutex); spin_lock(&hugetlb_lock); /* @@ -2633,6 +2635,7 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, if (hstate_is_gigantic(h) && !IS_ENABLED(CONFIG_CONTIG_ALLOC)) { if (count > persistent_huge_pages(h)) { spin_unlock(&hugetlb_lock); + mutex_unlock(&h->mutex); NODEMASK_FREE(node_alloc_noretry); return -EINVAL; } @@ -2707,6 +2710,7 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, out: h->max_huge_pages = persistent_huge_pages(h); spin_unlock(&hugetlb_lock); + mutex_unlock(&h->mutex); NODEMASK_FREE(node_alloc_noretry); @@ -3194,6 +3198,7 @@ void __init hugetlb_add_hstate(unsigned int order) BUG_ON(hugetlb_max_hstate >= HUGE_MAX_HSTATE); BUG_ON(order == 0); h = &hstates[hugetlb_max_hstate++]; + mutex_init(&h->mutex); h->order = order; h->mask = ~(huge_page_size(h) - 1); for (i = 0; i < MAX_NUMNODES; ++i) From patchwork Fri Mar 19 22:42:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12152067 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD0FAC433DB for ; Fri, 19 Mar 2021 22:43:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6050F61981 for ; Fri, 19 Mar 2021 22:43:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6050F61981 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EF6B76B0073; Fri, 19 Mar 2021 18:43:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ECD806B0074; Fri, 19 Mar 2021 18:43:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD10A6B0075; Fri, 19 Mar 2021 18:43:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0033.hostedemail.com [216.40.44.33]) by kanga.kvack.org (Postfix) with ESMTP id B2A8E6B0073 for ; Fri, 19 Mar 2021 18:43:11 -0400 (EDT) Received: from smtpin39.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 708C7180269F6 for ; Fri, 19 Mar 2021 22:43:11 +0000 (UTC) X-FDA: 77938100982.39.D0B8AE9 Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) by imf12.hostedemail.com (Postfix) with ESMTP id A3D2E132 for ; Fri, 19 Mar 2021 22:43:10 +0000 (UTC) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12JMaeWS138519; Fri, 19 Mar 2021 22:42:30 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=T3dFsB+WFB+MLZZ9FRwUsTao9HIlRMf6SCY4VgMzVAs=; b=u08eU0AKT3dVXzoW0ter1V8nFot3W6gzBcqGmLW5sJyd2ZGvDLCcPGabGyrrpytuV1qp MJNM/vTHKdRLL7P1Z2mUwk1ke86DwQVIp3OmZbCuW5S05GplCUanlU++tSPX+GDTETkk yIp0W0WTVeGWN6nMJcrRrnb4oL1aJuOJ7sjoswNmsK54HD1UVA74+g4+/R20bvATjm/d p3UeJ2JVd+UhDGO7YjJRCmyzuFoyfdZodRifj4v5TbGw3BM6fpv4JYG13X8aKA/xTbzn CNyY0Wwad+iAlgcOF3H1T2VQ5HtMw7hv1hjY8wcm8W+arlW+W/CLUMryPLlVyKHHK4UR yg== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by userp2120.oracle.com with ESMTP id 378p1p4dpc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 19 Mar 2021 22:42:30 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12JMeGb8155442; Fri, 19 Mar 2021 22:42:30 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2048.outbound.protection.outlook.com [104.47.66.48]) by userp3030.oracle.com with ESMTP id 3797b4sr31-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 19 Mar 2021 22:42:29 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Yh92HkoaNBj48hePh2c8QWH6g964WoqD6u8yz8SOxPWMFnAZlc1IvJdHB6mf390bu3u0KXvgdPktc43bjcaueHXqpet6Rka3Bs7scbLEZVJggq0/VmKDOosNBQgl1di/gbbNOI6LFX74pVGh9BLcrbGtksnp1JOfZ4OiidTgoORc+4kN44ytd7GEGOUlAPaLuDZvNnlNVZUVIiiCcIDYAkaS7Ld8QJMp3GyihKq8Kftx6Hjg0Va77lVhy42l6hNDoNGvuMwUwT2uu4JqTCCTAqSU/1OzmL2jlLSgCR0P3+yiDN0g0a09tnCtxQ1mrFeO6+BfdCwQVq56oGbwQDjIyw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=T3dFsB+WFB+MLZZ9FRwUsTao9HIlRMf6SCY4VgMzVAs=; b=AorXE/vZ8EpmumJBHYa0/wMfpIifknLWypOjNIXM8XQ2pnf8lUhZnCO6BoGKY33yQ6wXvxCg1i+axdE8+c1Ydf2/UjxPIwgGZT6T9CCNGBFyQWyIc5qUVphUNjCbO9rmEhr1iXyKz3WGQ2ZlY11i6wmnn9+SCO1xng2HBekG0Qn303JltbB+DJuM28RSM0A37uc4dnI61h3xni3y4LlPmxcnTtKLTcrS4kCIN+1W9X8RDf1NbdrN+ZC6Ko7j3iOsGFKyKJiViCfDtfZ2t/+LIUA3/A1Cj4RLPdlxlDVO48ffjHsj2U+4UhUjKmD/6YkP3ONMsG7u2Abi9S1Vk9RcAQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=T3dFsB+WFB+MLZZ9FRwUsTao9HIlRMf6SCY4VgMzVAs=; b=W+U3tbgPHxJcEP4gLYfNrpfFIve8FA62+LkwoiVJPfvSLxDeF1x43gPqzXpTuj2hK+ey4oVF2BC2RLrayNM2iOM3HWDtToyeb9ritDQ8DroY2s+Vugfueocd3/mAspP25x26ilHepQUucjU7wJqX27MfDj074GWAlPjKuHn+0qY= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3160.namprd10.prod.outlook.com (2603:10b6:a03:151::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.23; Fri, 19 Mar 2021 22:42:28 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3955.024; Fri, 19 Mar 2021 22:42:28 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Andrew Morton , Mike Kravetz Subject: [RFC PATCH 2/8] hugetlb: recompute min_count when dropping hugetlb_lock Date: Fri, 19 Mar 2021 15:42:03 -0700 Message-Id: <20210319224209.150047-3-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210319224209.150047-1-mike.kravetz@oracle.com> References: <20210319224209.150047-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR07CA0013.namprd07.prod.outlook.com (2603:10b6:300:116::23) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR07CA0013.namprd07.prod.outlook.com (2603:10b6:300:116::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18 via Frontend Transport; Fri, 19 Mar 2021 22:42:27 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: cf984369-efef-43da-678f-08d8eb284659 X-MS-TrafficTypeDiagnostic: BYAPR10MB3160: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7219; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: lt2cHZm447l9rY0BDb8Butcwk4Rg4x0CZJNgmYmTuPLdN4OyS8DfKxsDiMxnpuCQOYnBN+9A2A4vw0HIhDr8YAhf+pXhyJvs5wGG2DF5fwXCChOLUhLGJ5V1THVSNxB1JNGu14bUa4DPWhWIVofMaZbzpk/8MU3zAymHdXrzPp7rtsd/qbEY7xG1qBKSoHbRn0vJ4O86NcabRPSK4/GevLTtICHx+/lYTdrVMUbHgwDOL/zx8vLhgrHVw57VdW2xn1IUv0mG7YvjDehkCS3WJBmykLIKwitxhV1ysdDImlExLxsOVDqtz3ZnD5Vh3ZUpnhID5njQoDmRQuSWHIEYwVr5BTrXrOS+xfk7QN2y103iTnAZuQwX0ajHKBwzMaikparZlrfY37RbsNn1vZXZdc1fHrmUUeHpuVl2K+onbVsopQZ2AX9Zae4Y0bwKZFIRvmh/nOp3K3buav4PeBonPBGYi/FNSfsgTivrKI5igNWbW5N2yIFLw+WVFrGW8fTSkpcKyfeWudWSMyZAlxVij4ChIuxWf3L4/vXGeWfq74zcizLIi6RkGD0mQSN6FvhSYovdoz1alpDcN2aM7zRMODWrhQVHJqWZXK+cJ8OnjVj6wBY+852zlomUFFQ9c+t6zaJ56g31zFGWpasEQt1fRA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(396003)(39860400002)(346002)(136003)(366004)(376002)(8936002)(316002)(5660300002)(66946007)(36756003)(4326008)(186003)(66556008)(26005)(66476007)(54906003)(6486002)(16526019)(52116002)(7696005)(83380400001)(8676002)(956004)(44832011)(107886003)(7416002)(1076003)(2616005)(38100700001)(86362001)(478600001)(2906002)(6666004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: f/WjJX7+4wbr3kVb4Qih2HoEhwve2Rzb132WT1yKH96WDIwx78I9FuQP/tkAqywBPPg/gAzE+Wz96ZGBK5/qPxySOvAJxghq2OW3y5c3jtKgC0ZEBUVuDAytfFE81yIbrQQMO9mpP3QVbqfESQCNfzrg5Ev/pFl5jcE0tz1N8+yeg184xjoWZymgu/E2sJAXj0LL2k5AHNOjTdQde3aY0nmA+ehIwfC+N3gI8zzAScYwkKcNqTYSRVgAgWsmzLIjMHTT2uENVLH9NBTj9zlMcd+FCnPPOjamMKQgL/D1tinSug9pdgDqxueQs830ktjV0v37CxV/HcQVsNInlJmA4ZSB96WfchTP1T978xzoy89wfqafcz6X1iedf1nGCb4tVnOVnS5KM2+d8VEqYf04ox5HbAfJwWseeall/d0uRCdKni1FZleXOdPpBkWXLMQBUQ+imN5FBiYJqc7iSdEZDwsdJTwhKc0KCS8PUjw5+BgouMn6j7xfgvRfL9aCk7zVkd1jUHIwVxHWQaBgQ4kTZrAbSp1qMo0NSZKx8xPa4jxbI3YIHGI7F9gM/go2xqFcqay0dCoLQEatG/Z4ja9LICZx7d8MLmwvmvhwhC/pZPfcEoG9OuCiz1PPHcKqlIjCHJvgGBkHt0xDgf4qazN5zj/PLWhf0ymlt3VJCF+mBkxM7t3mHSM8pRpdmR9SrJdwz+uxw21KMZTFlaQumpwFb1slli51oRVJeSO8/C/vrAQTSsvWJABZqzORzwvRcRdkaH7DGLG2nvb3ZZk312UgYx03dmGYeQPbRQv3S7EfNDqIoGqUoXWnWVeuvC+5GruHOT1P7AmthpL1/LuSDw96T1EpeT8MI7huON1tjrvg8M9ce/YHzycJgf32IMt/TMoaTLB1UPdStCehu3wIFm52/tAtSGT8xTTcsTwgYKir9m8oBr7iftmin7J92/ZYFnbKswmpA0SxnJ6Y6y0y8v2O2mP8mQ09TKU7TaKa20b38i+Pt1EmpL5acYS2Yur6Oh9G80K8ljK6XTemuQ2nMSiEpB1A5sIMJ1wF3sYhl5f+TqDNgq0W5tAJ8p6438o48Zz5Szldd2N0e0K3s6J090RTV0EryrrD3DopIJOBm3XVIGGzfNZItn1lBcDPrfXAEnC2StUY9iuFMJ1Vu35xE7rf92V7UwlvZSnbCF9uwvV/hsjHFqC3TEPXQeR7ul1f954b24Oeji5ZZYTAKTHgC5v27zhIqZnT2uYyYDNCd/qYZNt00eVvu/ChsreSu0IlTs/bM5QWJpbsEFk8kckK5Sy/UOq2LAAPh+6MLKSOydLAOSSn+BsIZl5pfExl1ZVvw1nD X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: cf984369-efef-43da-678f-08d8eb284659 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2021 22:42:28.5703 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: wwn+RoP9kheDfDfJj6Xwddz4X3B2uRzAXru56f8qvW6Y/aGkgs54kS2BpzL39AZotMxqr9NAtAmf+LcmgIVxbQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3160 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9928 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 spamscore=0 bulkscore=0 malwarescore=0 adultscore=0 mlxscore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103190156 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9928 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 suspectscore=0 adultscore=0 spamscore=0 clxscore=1015 phishscore=0 malwarescore=0 priorityscore=1501 bulkscore=0 mlxlogscore=999 lowpriorityscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103190155 X-Stat-Signature: g7jfkt8prqa8icqgm8co85b44niw6qy8 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: A3D2E132 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf12; identity=mailfrom; envelope-from=""; helo=userp2120.oracle.com; client-ip=156.151.31.85 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616193790-685070 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The routine set_max_huge_pages reduces the number of hugetlb_pages, by calling free_pool_huge_page in a loop. It does this as long as persistent_huge_pages() is above a calculated min_count value. However, this loop can conditionally drop hugetlb_lock and in some circumstances free_pool_huge_page can drop hugetlb_lock. If the lock is dropped, counters could change the calculated min_count value may no longer be valid. The routine try_to_free_low has the same issue. Recalculate min_count in each loop iteration as hugetlb_lock may have been dropped. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index d5be25f910e8..c537274c2a38 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2521,11 +2521,20 @@ static void __init report_hugepages(void) } } +static inline unsigned long min_hp_count(struct hstate *h, unsigned long count) +{ + unsigned long min_count; + + min_count = h->resv_huge_pages + h->nr_huge_pages - h->free_huge_pages; + return max(count, min_count); +} + #ifdef CONFIG_HIGHMEM static void try_to_free_low(struct hstate *h, unsigned long count, nodemask_t *nodes_allowed) { int i; + unsigned long min_count = min_hp_count(h, count); if (hstate_is_gigantic(h)) return; @@ -2534,7 +2543,7 @@ static void try_to_free_low(struct hstate *h, unsigned long count, struct page *page, *next; struct list_head *freel = &h->hugepage_freelists[i]; list_for_each_entry_safe(page, next, freel, lru) { - if (count >= h->nr_huge_pages) + if (min_count >= h->nr_huge_pages) return; if (PageHighMem(page)) continue; @@ -2542,6 +2551,12 @@ static void try_to_free_low(struct hstate *h, unsigned long count, update_and_free_page(h, page); h->free_huge_pages--; h->free_huge_pages_node[page_to_nid(page)]--; + + /* + * update_and_free_page could have dropped lock so + * recompute min_count. + */ + min_count = min_hp_count(h, count); } } } @@ -2695,13 +2710,15 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, * and won't grow the pool anywhere else. Not until one of the * sysctls are changed, or the surplus pages go out of use. */ - min_count = h->resv_huge_pages + h->nr_huge_pages - h->free_huge_pages; - min_count = max(count, min_count); - try_to_free_low(h, min_count, nodes_allowed); + min_count = min_hp_count(h, count); + try_to_free_low(h, count, nodes_allowed); while (min_count < persistent_huge_pages(h)) { if (!free_pool_huge_page(h, nodes_allowed, 0)) break; cond_resched_lock(&hugetlb_lock); + + /* Recompute min_count in case hugetlb_lock was dropped */ + min_count = min_hp_count(h, count); } while (count < persistent_huge_pages(h)) { if (!adjust_pool_surplus(h, nodes_allowed, 1)) From patchwork Fri Mar 19 22:42:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12152065 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A12B2C433E0 for ; Fri, 19 Mar 2021 22:43:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2C1506197E for ; Fri, 19 Mar 2021 22:43:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2C1506197E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EA9C96B0072; Fri, 19 Mar 2021 18:43:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E58E86B0073; Fri, 19 Mar 2021 18:43:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C5AE66B0074; Fri, 19 Mar 2021 18:43:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0044.hostedemail.com [216.40.44.44]) by kanga.kvack.org (Postfix) with ESMTP id AB8E26B0072 for ; Fri, 19 Mar 2021 18:43:06 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 67E00824999B for ; Fri, 19 Mar 2021 22:43:06 +0000 (UTC) X-FDA: 77938100772.21.4CA52A9 Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) by imf19.hostedemail.com (Postfix) with ESMTP id A73A090009DE for ; Fri, 19 Mar 2021 22:43:05 +0000 (UTC) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12JMZPjL137221; Fri, 19 Mar 2021 22:42:33 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=CfnRtUVJM27gw2iz+CQbyD7wcXFAxcVVPvq0KxvEoZ8=; b=mDrLJTB8RQl4luYmvHV7SQL/Z2uEPbjBmRpwOs30yEMMr6vOFs1Y5zWaoUJjvwcdECSZ 19saxJ9kgIK5ABrKHXFQNu7h59ii2kxwTZl0jbsbeqozkVK6EHIZsu+sqQUmFGV+yVqo 2kgki8XBZPOuS+ZivIoT0ArIQWU/xsmZ+IWlM5uBqJkY0uNx+O/A3OQzvqJ9g2fzGxI9 fpSqTnIrWuTiMH0Yx4mf0Y0vlEU8qwrVYQWbkafs1yuoVp+uQ5XTo9qiFRWEbo06+khA HfqKEMvGSlP9RrbdNXnE3hTT0WtOE2J8E27uGmvHRE0OEH+4KH2p8jZm13GLmjvPWNzU CA== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by userp2120.oracle.com with ESMTP id 378p1p4dpe-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 19 Mar 2021 22:42:33 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12JMeFA2155350; Fri, 19 Mar 2021 22:42:33 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2176.outbound.protection.outlook.com [104.47.58.176]) by userp3030.oracle.com with ESMTP id 3797b4sr44-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 19 Mar 2021 22:42:32 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ThwQ6SNvGG1QS0QJ5iONj1RxtzMSS8melT3KQrmSHbmPjSP6DobntySJ5i0BDvPKNu8qdNE4kbcUz7hYO5s8CFHRRYXNpRCIFQkpRt1Jhes3OT5P+DOiwSO+2eIhZthvJs5vAOLc0od7l0n0k4yNG+Z3psnIqwuFE64/qhf7aw20pb2JYcQOnLgi6sORnqWRfPfFh14AGMCi9deEkEgqAPpea3n8zLojJnwb077L8mlXs0bq+PQ0jPWgjnpBSEtggtfkvWdDITxOafg5AQpm39LfA1KD/GrjoEFbGc/nMseM7Yomdt2x61FoAgBj6Qixx3m0jwcyPFMMSPR/Po6PnQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CfnRtUVJM27gw2iz+CQbyD7wcXFAxcVVPvq0KxvEoZ8=; b=fapX5zzbwsx+WWx49BPEYG5ed70Kwozw8VtE2yDArVlTcgbS+SX/yg0sKUqjj551npaaWmrt8rzeyT1XRY2lp5shfvArOV/9faBT5FURn84TuKXbeLdzeUvH9zZLPBWXYW0kJcYO0es13cvlNmV+n8FVd74KFsWF9umh5r1cxeUBHnnK3SX3gt6HFAbyQnmqNqztkxjtsnNTUFUKqUr+xAMesnHizi8KeY8qLnZpyLnnM0F340m1gFWFXWgkRiCD47J8wVvyakb6KUCuQas+gKiBpJ72xHDuHmqJ14DRnK3qes6MCLoH3ijziluj3xhc93+obhadaQDYmvdhbbPJFw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CfnRtUVJM27gw2iz+CQbyD7wcXFAxcVVPvq0KxvEoZ8=; b=H6eeWzsIeHz47kaQ6dUaoescQiqo5SXoVod/LdApLhXI8WxFp2EYG1XXI4VMJ1ZF70fcCfY2xxsr3ikO3xZXFAOf4M7YJMd5zWZawkWm6WYH97uNDvjRFHDzEkJbUW1EkF6C38fYZptrOIIiPOiAvnJpTT3wRO7pIem3Hp+Pqpc= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3160.namprd10.prod.outlook.com (2603:10b6:a03:151::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.23; Fri, 19 Mar 2021 22:42:30 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3955.024; Fri, 19 Mar 2021 22:42:30 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Andrew Morton , Mike Kravetz Subject: [RFC PATCH 3/8] hugetlb: create remove_hugetlb_page() to separate functionality Date: Fri, 19 Mar 2021 15:42:04 -0700 Message-Id: <20210319224209.150047-4-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210319224209.150047-1-mike.kravetz@oracle.com> References: <20210319224209.150047-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR07CA0013.namprd07.prod.outlook.com (2603:10b6:300:116::23) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR07CA0013.namprd07.prod.outlook.com (2603:10b6:300:116::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18 via Frontend Transport; Fri, 19 Mar 2021 22:42:29 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d4ab2d41-d998-45b3-4862-08d8eb284780 X-MS-TrafficTypeDiagnostic: BYAPR10MB3160: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7691; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: dbBxhu6/NfZii0fIOELe+7yTZyBlLpCX+2RzuNGnbu4r9ywITV3c6ytPjkANIMdU+opOQVrUf7jTquv5/YYTkE1ItOOQD1aFJNJL6CukdVyetSc93ssP6mDF9SmrtWsviLfzv2lbYLlS4KNv7NiAxXk1t2sVK0a6QSAtUm2FoXYrtN0iRXAtmeCsxSfZk9Ue5YpIozltuQGAJp/GSow8hh7EoBPmxpmIr5NKiXt2Mi0TZmR1z0Dimnqs5W/IRcB+6PpfsBi6/UTLpJGhRzCdMG9DWonSd0IkYBje2E+QjKZ6Rx8Ke6PRe1HmYKC6pp6zXqJNYJVcA82LMqvR3qBeRzFqA8UwywKiP5AF28hZyBAdibfPOAl4VCe6f/9VL9yioJ8rQTo2/cz69wE+6DWhdkBTJkPV8D0/44GeQFev+9zq+svu90fD+dJi0tVL4wYWRn7I+zONpbbKPfFkN8Mki+h1JqkG/eRORuBeUl1xR64uwEcu+ZEvRq7jkwVPg1KQdKuK5hpcBzZz/z7EbsJvJx0gVPRaGC3/5mxi7Pb2sX9FrL+JkXW3g9E+cpSsDy0Dyx0wCLuKIunmiZKqBL26Krqt1im9ib5VF12lby/YYVlIaj2QMV+lJzKSyJ2EC1zbxkBNPtnW5IqmzEM2NAp3lg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(396003)(39860400002)(346002)(136003)(366004)(376002)(8936002)(316002)(5660300002)(66946007)(36756003)(4326008)(186003)(66556008)(26005)(66476007)(54906003)(6486002)(16526019)(52116002)(7696005)(83380400001)(8676002)(956004)(44832011)(107886003)(7416002)(1076003)(2616005)(38100700001)(86362001)(478600001)(2906002)(6666004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: kGfIr0TvOHXscYwRTdmu9ikeIDPlemrR4B0tKQoPw+diMb6LDSew5+AxWsNomwWhqyrjVOyEazYI2Pc54auwAIqBs2drSlctSQPgFloDIzZE8Og43vBhTS47ByRuAlgkpKG9vY69MI6OB+pWgCTPcKKSqtS2PsyhMFthTMAFZ609TU/JEGxHcQ+kIuT9/MhZwbbJ+fyAJDlE6x4puokXJJZAFklUv08dEfnn0T4OreOtcFmDXZ73NQtk8+dVGAZDFOMkmU2bnkQo8BJ63nphaSWQ+6TL0u75eHRg1q6+M93c2Lwdopj7+Yy1jv4Sqi65cFcXrg5Rya2vB0HUH1CgScNZvUZIQ719ZFjd/nKt3slKD62BJ/a1IFJMmGF56xusqjgNHrqsQ9Xe9jMBuCKIp/NQ9aK0cv33z/CasiW2QH7TLBoA5I9Fj8gdQ4Nqj9A4WvqmyGFn98HIrcUSlLiLx1Kvw+GTSui0W6TDgajkFfDNN8C5yO5addNbVcpSdGk9y0vOzovIbHvV1q1TkJXHywzQocjbml9L5anqs9iq3MppDw0Sjr35oJlW/vstDEABDYjTU56GZl+qXaSEO0X/KM2a+J8XIjKC6UIEv6QksfpT6J2KJxEoT5m6G7Z+7Zi4wJy2DkCiJP4lk9ah8ppq3nQeGlWIa5MO4KqgfNEqt8iSRLlzb2cBmD82CFeKCgH29wIgDxWZ69xNK4wo2UA9euzPYk3wWANYXPtK4JFpE6m15m1nMAumABklDGV0H8/FXd5N0mlG2DW92dzsdOSn+M+6rWqiDfM19DzTGarQ2z4A/Uxs5r4xeqdB0Z8ve9pfE0aNXzi2+BR76P68A1U2G0JskyvpRzdcZkcPbmtMMdXcJFS7wdY41eSbRI0ucO7yo5GGXFnlV6FIdgpLtIBpKr8e4ZAwACS6L5Q5tol3IzuoHg733CCNSwc246+NsOlosOpFdx4sUzXSATfQPj2I7pfcSVjGQjOiijgZVEHJGk6UC1GFyWODhWzWw99ieIxZ4cLIQariwl/sy5rs/XzRUM6+PIGa3Z59KfXXrXDWcIoFFgsKaJG7Cz7cLvtAyCaKbG/2lv2//7BGjdEuasnXFnt8jZN933Q54QEddlvMSg1LCODgqLDASgFG+fK85efHcvh4Bt6NPufUWC33XAEiZkypckJwn0soTqtyLc/eGgqKrERIKJ3l/tTulb6hWh4PwHhWInfB3M7/jsW3FSsaqu6OGYsVhVkYWCdisnnowu/9Pc7NJIE9yhOjykMSh+G9aaSBRRz5ZjTj5+Pe8GVDK8C3Eyv7Jsckasj1wW7coTy/bLwRnVWeo3WaZUolEqxp X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: d4ab2d41-d998-45b3-4862-08d8eb284780 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2021 22:42:30.4709 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: CzA3Jhr2nrxwALYBuNz7IkB9goMxah+cvcGG11T0/I1GJuOCFnGSJriptlHNtLJRK79FAQJj0JVVMBTLDibooA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3160 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9928 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 spamscore=0 bulkscore=0 malwarescore=0 adultscore=0 mlxscore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103190156 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9928 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 suspectscore=0 adultscore=0 spamscore=0 clxscore=1015 phishscore=0 malwarescore=0 priorityscore=1501 bulkscore=0 mlxlogscore=999 lowpriorityscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103190155 X-Stat-Signature: qdnkghhsgpee93k7tbkn8bsuf4ph8x3e X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: A73A090009DE Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf19; identity=mailfrom; envelope-from=""; helo=userp2120.oracle.com; client-ip=156.151.31.85 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616193785-207053 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The new remove_hugetlb_page() routine is designed to remove a hugetlb page from hugetlbfs processing. It will remove the page from the active or free list, update global counters and set the compound page destructor to NULL so that PageHuge() will return false for the 'page'. After this call, the 'page' can be treated as a normal compound page or a collection of base size pages. remove_hugetlb_page is to be called with the hugetlb_lock held. Creating this routine and separating functionality is in preparation for restructuring code to reduce lock hold times. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 70 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 46 insertions(+), 24 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c537274c2a38..ae185d3315e0 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1306,6 +1306,46 @@ static inline void destroy_compound_gigantic_page(struct page *page, unsigned int order) { } #endif +/* + * Remove hugetlb page from lists, and update dtor so that page appears + * as just a compound page. A reference is held on the page. + * NOTE: hugetlb specific page flags stored in page->private are not + * automatically cleared. These flags may be used in routines + * which operate on the resulting compound page. + * + * Must be called with hugetlb lock held. + */ +static void remove_hugetlb_page(struct hstate *h, struct page *page, + bool adjust_surplus) +{ + int nid = page_to_nid(page); + + if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported()) + return; + + list_del(&page->lru); + + if (HPageFreed(page)) { + h->free_huge_pages--; + h->free_huge_pages_node[nid]--; + ClearHPageFreed(page); + } + if (adjust_surplus) { + h->surplus_huge_pages--; + h->surplus_huge_pages_node[nid]--; + } + + VM_BUG_ON_PAGE(hugetlb_cgroup_from_page(page), page); + VM_BUG_ON_PAGE(hugetlb_cgroup_from_page_rsvd(page), page); + + ClearHPageTemporary(page); + set_page_refcounted(page); + set_compound_page_dtor(page, NULL_COMPOUND_DTOR); + + h->nr_huge_pages--; + h->nr_huge_pages_node[nid]--; +} + static void update_and_free_page(struct hstate *h, struct page *page) { int i; @@ -1314,8 +1354,6 @@ static void update_and_free_page(struct hstate *h, struct page *page) if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported()) return; - h->nr_huge_pages--; - h->nr_huge_pages_node[page_to_nid(page)]--; for (i = 0; i < pages_per_huge_page(h); i++, subpage = mem_map_next(subpage, page, i)) { subpage->flags &= ~(1 << PG_locked | 1 << PG_error | @@ -1323,10 +1361,6 @@ static void update_and_free_page(struct hstate *h, struct page *page) 1 << PG_active | 1 << PG_private | 1 << PG_writeback); } - VM_BUG_ON_PAGE(hugetlb_cgroup_from_page(page), page); - VM_BUG_ON_PAGE(hugetlb_cgroup_from_page_rsvd(page), page); - set_compound_page_dtor(page, NULL_COMPOUND_DTOR); - set_page_refcounted(page); if (hstate_is_gigantic(h)) { /* * Temporarily drop the hugetlb_lock, because @@ -1400,15 +1434,12 @@ static void __free_huge_page(struct page *page) h->resv_huge_pages++; if (HPageTemporary(page)) { - list_del(&page->lru); - ClearHPageTemporary(page); + remove_hugetlb_page(h, page, false); update_and_free_page(h, page); } else if (h->surplus_huge_pages_node[nid]) { /* remove the page from active list */ - list_del(&page->lru); + remove_hugetlb_page(h, page, true); update_and_free_page(h, page); - h->surplus_huge_pages--; - h->surplus_huge_pages_node[nid]--; } else { arch_clear_hugepage_flags(page); enqueue_huge_page(h, page); @@ -1693,13 +1724,7 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, struct page *page = list_entry(h->hugepage_freelists[node].next, struct page, lru); - list_del(&page->lru); - h->free_huge_pages--; - h->free_huge_pages_node[node]--; - if (acct_surplus) { - h->surplus_huge_pages--; - h->surplus_huge_pages_node[node]--; - } + remove_hugetlb_page(h, page, acct_surplus); update_and_free_page(h, page); ret = 1; break; @@ -1737,7 +1762,6 @@ int dissolve_free_huge_page(struct page *page) if (!page_count(page)) { struct page *head = compound_head(page); struct hstate *h = page_hstate(head); - int nid = page_to_nid(head); if (h->free_huge_pages - h->resv_huge_pages == 0) goto out; @@ -1768,9 +1792,7 @@ int dissolve_free_huge_page(struct page *page) SetPageHWPoison(page); ClearPageHWPoison(head); } - list_del(&head->lru); - h->free_huge_pages--; - h->free_huge_pages_node[nid]--; + remove_hugetlb_page(h, page, false); h->max_huge_pages--; update_and_free_page(h, head); rc = 0; @@ -2547,10 +2569,10 @@ static void try_to_free_low(struct hstate *h, unsigned long count, return; if (PageHighMem(page)) continue; - list_del(&page->lru); - update_and_free_page(h, page); + remove_hugetlb_page(h, page, false); h->free_huge_pages--; h->free_huge_pages_node[page_to_nid(page)]--; + update_and_free_page(h, page); /* * update_and_free_page could have dropped lock so From patchwork Fri Mar 19 22:42:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12152073 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6221C433DB for ; Fri, 19 Mar 2021 22:43:19 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3634B6197E for ; Fri, 19 Mar 2021 22:43:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3634B6197E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D21E26B007D; Fri, 19 Mar 2021 18:43:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CF79D6B007B; Fri, 19 Mar 2021 18:43:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE9C86B007B; Fri, 19 Mar 2021 18:43:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0154.hostedemail.com [216.40.44.154]) by kanga.kvack.org (Postfix) with ESMTP id 8EB736B007B for ; Fri, 19 Mar 2021 18:43:15 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 574731803A816 for ; Fri, 19 Mar 2021 22:43:15 +0000 (UTC) X-FDA: 77938101150.02.819306D Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf22.hostedemail.com (Postfix) with ESMTP id C0266C0007CA for ; Fri, 19 Mar 2021 22:43:14 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12JMa8UN027312; Fri, 19 Mar 2021 22:42:37 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=T5h08E48aTHqHAs7oSFnQWTyUDdovMoQpW+XQm58DBI=; b=ZMkTUmIhPom2Zt+sswil7yaW+l/cOjSaO8ekkCZid/3V+BkplOsfbt15AO4rJc8OlbpS mBSoDpgw0+STc5aSApKE8DY+wSBcas4v0DOeIYHmd+39ynJTSsnPWrlqLtJzSQHns68O Fi215ueQPGlTQBkIvx7592HRv7iJgQ9BhwG7nV70KVH52qGJpw0HJB+ISilC33s9GDfn JNp0mg0x0voVr8FI1U9zWboPIE/I1hZqB0BKU0JgzsDZmYmZXsV/ld4ArYK13I804PEA B3+ri9KT1a8WzUTUchlr/A+v8Cu57u9p5ZyWKHI5EAbJXI3KAlg458rtZOmGRpxSXGO5 KA== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by aserp2130.oracle.com with ESMTP id 378jwbvhm6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 19 Mar 2021 22:42:36 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12JMdl6A044897; Fri, 19 Mar 2021 22:42:36 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2175.outbound.protection.outlook.com [104.47.58.175]) by userp3020.oracle.com with ESMTP id 37cf2ca49n-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 19 Mar 2021 22:42:35 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kzwm2B08K54TDqL3A3hsIaEUzc6m2KPpFRYKtTLrCmcmPb+YQN/J1AzYyUPyRr8vN20HdgJkbDFYxpwnMLWHRW9Zj2cFral+dS7fEdQhhJdYLRhwcuusO2fpvPj5IY52XQ8BlqBg1ohG9rQX2Nv27bwjB1aO9hfuFDR7ZPVC+/lPLKElHvsSAF/oVPL7Gbi9NHk/MF49ywvbypnTPFe7zZs+RcKHw8Nf71Z0JZLhaP5zIB2z44ZATAMtq7shXHM5JJxrcif/NSR3QMsqE++/OChIBBcP1a9FAyNGRB2quwEK0MPI1gMZ2Ai18szZGTEw+ISF9rxL9g0j24Q8jaOwaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=T5h08E48aTHqHAs7oSFnQWTyUDdovMoQpW+XQm58DBI=; b=KtubUU1lx8VXu6ifcu+bHXvdWrMvvAmFKDUorG/bNmfqsnc/pGs93pOOkW4n2k5JEEhnyKhFiLaQfehYhkvYOiJTurI1wWw/N3PjRRHn+KBMDrInQnvrEFAcmNuL1NYEKB3VMdwmK0QmibY71hsP0pUa66IZC0EzPmIjAZ/X7uqn+dk5OEhaIte53sa4eO+SEIF/I3DwS3SYZIw1BAEj9bRxC7iSVKkgV6PLabrzboSAnKowtnfz/+ywHnNXskwzYVMLYdKvT1fmVWT5sMJ2xUbAynerLX80mGA9Bm/ruzHclNVkRgtQoQpyfldnMgqzm+NLpGGlmZOVHwCNcVNjhQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=T5h08E48aTHqHAs7oSFnQWTyUDdovMoQpW+XQm58DBI=; b=IBVUagw3XJ0qus9A08x9MyuhGoSfPYb5VfAMdiiJA4SEN4CsoN5+6m1H0Wnbq8AN50PC3RSsqmV41fchMK6NbARex0vbhb/H1dc9IjqI8NaUFXQvUsfIl0dCNURI5EiR2Du3mfVaUphHflDyni42qkve3S2CHWd0Hy/Hz0GSdS0= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3160.namprd10.prod.outlook.com (2603:10b6:a03:151::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.23; Fri, 19 Mar 2021 22:42:33 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3955.024; Fri, 19 Mar 2021 22:42:33 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Andrew Morton , Mike Kravetz Subject: [RFC PATCH 4/8] hugetlb: call update_and_free_page without hugetlb_lock Date: Fri, 19 Mar 2021 15:42:05 -0700 Message-Id: <20210319224209.150047-5-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210319224209.150047-1-mike.kravetz@oracle.com> References: <20210319224209.150047-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR07CA0013.namprd07.prod.outlook.com (2603:10b6:300:116::23) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR07CA0013.namprd07.prod.outlook.com (2603:10b6:300:116::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18 via Frontend Transport; Fri, 19 Mar 2021 22:42:32 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3585576b-7cb5-4aee-63f8-08d8eb284908 X-MS-TrafficTypeDiagnostic: BYAPR10MB3160: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5236; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8QhvduPRB5Ak5GzB56D8GPF3wLx2ZVqw8fxYm5cTFViBxQ2OR0nD73LrHdhqDphAwDCQBRUE2ePnKPZzd1e/vtI4WooOTFjH4zUL+UuSqPOuHht/J3argiCjjmrVozWCpOAU1Wq4HyJ+Q0ZdmsdW+uuV/GixYiOXvc+PTbQmgW1Sl4UwzxqFj1CKG6u+QKy6JcEclqW+hLA7aitVgkOJch/taMhlJ7pze4dGiC9m5wn6Fzv+bU8XPyL6CEBsyFimApTRfHKkTgWzbbe8yyBMHaRsGT2N1/P2KQ+prgXWeBFWE2E2JfpbirWbf7d544dLqp8sILr8V4SVKFgsgUvqix8asghufacq3Em6LgP+/qZ9nFL/Ucfv6QjIar/lPFFeJLVukVfLtAPTJg8q/0bRR3Va740UynqrGTA1tAMAwrO2v0FQ4KC6ADAMj5KErICgnhw/S1asuGf3tZgOxRcT0EPYCGuutMhB52XEfKhopC45PCNWgvn+EoHNrrfE5I40YORwbfQNr82q//xSAVOnGu/DiBHifcb8aNXkbYD2NOT0oCegjhCaQqgsP7Qi1QN9zlBaPWqmIUjjSCMNzfi4L+iEblTqdCk5T1LYaVsY+3YK9/qBtORqO06wW+JaLT4y4bDtEaYaQnQ6VoZTTj5hlg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(396003)(39860400002)(346002)(136003)(366004)(376002)(15650500001)(8936002)(316002)(5660300002)(66946007)(36756003)(4326008)(186003)(66556008)(26005)(66476007)(54906003)(6486002)(16526019)(52116002)(7696005)(83380400001)(8676002)(956004)(44832011)(107886003)(7416002)(1076003)(2616005)(38100700001)(86362001)(478600001)(2906002)(6666004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: dDVwFc6ZzErNoM67RGhEASNe7rekNTFXjdzzfVqZoNXe1yLjA7Qs52CQeq295ADe8C65Oblw2YcEIQnaDWtSkiCUos+RIRFwYDneiBQK4ciTihqmzheYOXFusP7aSxyEsYBgfqK+OgjelWqI5qxvApZlAIUD0mEN6a4/fQFk/yEsgZHVUBZrw7F4o2Smkip/4cC4/LFbrW8Oljfpsq5ZTc9hRoCpIgiZkwdgWp0FQwIVyrRhD8SMANHctuYu8I15bWPdm5FgXSbYrJY9e1QdgWXnlSTQ5g/WbAlzT7/bJEulNKaWOwmZvMXdF1Gtfb9qKlXQGbRuy/E/WnRcnR+PudHuJm07hXbkOa4yBpgatNboVeGlcmZ6s9cLqI0Vb60epRT1oKpljDIoYonH1ePwKw5zumkxoSTBeZ6CCl//PYgXL4Ghex6lW2+fgjnDySAWXwJdQo8FwZld587E31cXuweXLdl34v/hMFOswepzgeDHh5IC4RUDi5ssfSRA489sdZI+U/tc+8q03h3lIDARh9bvAZIowvOcrO0jfIVPKJ4n9UDvZzevO7ShClE6OmUII5CiT5QgL0+u1tI1ZEkFIYDVsbCTZKux65PCQ7mJ3mYdSYSwe1PXgIjxjkPG9bBbRK8CVrWdDYWZibUqRl+Hu+sbYPSgk+/GHDBGpJg5cUdCB5hULq+nWB/TJDJw9pdyhqTgz41cBFrV313qHsNKATpfIVl3Kj8hXqK5SQN3y4mRaE8FyRtbF1s9nBVpSH+aERMdJEYPp/vQhPclGxXRJ2xBnb9NZRYjdnrFrq+PLv16ximvQDpF0/SJx5CaWAk7Q5XLfoN0r799C55MKPwii+hHAkdUUl0iscle0ridm54j2xfvY/ezjDPZbHSOFezRfof4EHw7YAsQH4LOswrrOSChery6nbCrd5RBCTbqgwOBgc6LrqOtAQ1ANAJmutfhagoBpHtQBwnYTLHUgn/VwhPLBlaIIduIbE9lKRwfm47nIEuwFPtkY/wVE8VBTS+DKKrJmB/Ib/xSMjhDTeZMON8aMDvjRqN0bfHD0eTpeX7Ce3vADDnu6QjPBUIwOXgrkn6BVmCTeEHMZa1uiXNaizffq6/tsCytZwSA/HaYIP1OTExivYzbYSvqdsBTITA9gANrS+TdD5VS8ZgUezY59zKLPsQStlTseo+6gKTgchs/NfMt7zHDthTym7B/h0ilLK/aC1Wfe0u5tYzGhzWJU+06hQGRMKekQFdGHNm5WkOTuwOcHKLhz9LDMvRnz8J5rLBm0FATEjOk2IogobPyl/trKDgoEVEGGZgr6zm6VqidZax1zrQi+fqvU51eOsnE X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3585576b-7cb5-4aee-63f8-08d8eb284908 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2021 22:42:33.1133 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: KbW4oi7HnF4+Y26Ag9XTqW4ZUqRjK1nMEr6TUdI3cQi8mdl72cHRp71sLzAyW3DQv+Z4JyRM32mWO9TBpjLNEg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3160 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9928 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 suspectscore=0 adultscore=0 bulkscore=0 mlxlogscore=943 phishscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103190156 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9928 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxlogscore=999 spamscore=0 mlxscore=0 bulkscore=0 suspectscore=0 priorityscore=1501 lowpriorityscore=0 clxscore=1015 adultscore=0 phishscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103190155 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: C0266C0007CA X-Stat-Signature: 1a6iurn7cgn7m5ih7i1h3sha3p9zs1fi Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf22; identity=mailfrom; envelope-from=""; helo=aserp2130.oracle.com; client-ip=141.146.126.79 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616193794-422730 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: With the introduction of remove_hugetlb_page(), there is no need for update_and_free_page to hold the hugetlb lock. Change all callers to drop the lock before calling. With additional code modifications, this will allow loops which decrease the huge page pool to drop the hugetlb_lock with each page to reduce long hold times. The ugly unlock/lock cycle in free_pool_huge_page will be removed in a subsequent patch which restructures free_pool_huge_page. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index ae185d3315e0..3028cf10d504 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1362,14 +1362,8 @@ static void update_and_free_page(struct hstate *h, struct page *page) 1 << PG_writeback); } if (hstate_is_gigantic(h)) { - /* - * Temporarily drop the hugetlb_lock, because - * we might block in free_gigantic_page(). - */ - spin_unlock(&hugetlb_lock); destroy_compound_gigantic_page(page, huge_page_order(h)); free_gigantic_page(page, huge_page_order(h)); - spin_lock(&hugetlb_lock); } else { __free_pages(page, huge_page_order(h)); } @@ -1435,16 +1429,18 @@ static void __free_huge_page(struct page *page) if (HPageTemporary(page)) { remove_hugetlb_page(h, page, false); + spin_unlock(&hugetlb_lock); update_and_free_page(h, page); } else if (h->surplus_huge_pages_node[nid]) { /* remove the page from active list */ remove_hugetlb_page(h, page, true); + spin_unlock(&hugetlb_lock); update_and_free_page(h, page); } else { arch_clear_hugepage_flags(page); enqueue_huge_page(h, page); + spin_unlock(&hugetlb_lock); } - spin_unlock(&hugetlb_lock); } /* @@ -1725,7 +1721,13 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, list_entry(h->hugepage_freelists[node].next, struct page, lru); remove_hugetlb_page(h, page, acct_surplus); + /* + * unlock/lock around update_and_free_page is temporary + * and will be removed with subsequent patch. + */ + spin_unlock(&hugetlb_lock); update_and_free_page(h, page); + spin_lock(&hugetlb_lock); ret = 1; break; } @@ -1794,8 +1796,9 @@ int dissolve_free_huge_page(struct page *page) } remove_hugetlb_page(h, page, false); h->max_huge_pages--; + spin_unlock(&hugetlb_lock); update_and_free_page(h, head); - rc = 0; + return 0; } out: spin_unlock(&hugetlb_lock); @@ -2572,7 +2575,9 @@ static void try_to_free_low(struct hstate *h, unsigned long count, remove_hugetlb_page(h, page, false); h->free_huge_pages--; h->free_huge_pages_node[page_to_nid(page)]--; + spin_unlock(&hugetlb_lock); update_and_free_page(h, page); + spin_lock(&hugetlb_lock); /* * update_and_free_page could have dropped lock so From patchwork Fri Mar 19 22:42:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12152071 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2727AC433C1 for ; Fri, 19 Mar 2021 22:43:17 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B26326197F for ; Fri, 19 Mar 2021 22:43:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B26326197F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 55A936B0078; Fri, 19 Mar 2021 18:43:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 533656B007B; Fri, 19 Mar 2021 18:43:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 337A56B007D; Fri, 19 Mar 2021 18:43:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0044.hostedemail.com [216.40.44.44]) by kanga.kvack.org (Postfix) with ESMTP id 11C1A6B0078 for ; Fri, 19 Mar 2021 18:43:15 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id C3A1D1803A816 for ; Fri, 19 Mar 2021 22:43:14 +0000 (UTC) X-FDA: 77938101108.07.389843D Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf13.hostedemail.com (Postfix) with ESMTP id 17FD8E0011E2 for ; Fri, 19 Mar 2021 22:43:13 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12JMYVBG026034; Fri, 19 Mar 2021 22:42:38 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=7zLi0DBtPFvGN/R70Qix3ZkOT4iteU2T9DCf5/hbITE=; b=BCcmqmo1GO2yLc6iMFLlLWnajsmiQZssFVwyFwg+PCB1QSdnRcMYOQBDfJcat/hfeYBi Gsg/uMAmJ5bl+cqWMaWiNBtHuv3ILUvlR3Jdf1SlCNYXqr9K0jmXguICbwLKgWnjGmvN gcz8i/Afd+MSFLgPf6Ko3CHgUxH3R9HQMeKzYfD7VpyjFVk5LRrrlW3mo9tDeDjN9emv NumJMBZSk1H5W1hbC+ccb2wv8RCv+LkCkcVlDrfWy9Cb6o6c+VloPj0SR8OfvVhrx5Di GS2rtDxtjUHKSWYCtE9g4HDNJ5WAZVW1V01bfUbXQqGfjloM9qyUBIjpLvQJYG/965oW 3w== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by aserp2130.oracle.com with ESMTP id 378jwbvhm8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 19 Mar 2021 22:42:38 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12JMdl6B044897; Fri, 19 Mar 2021 22:42:37 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2175.outbound.protection.outlook.com [104.47.58.175]) by userp3020.oracle.com with ESMTP id 37cf2ca49n-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 19 Mar 2021 22:42:37 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=c/QLs2vSRL6vMUJohbj5lIOfcOn4kTYeb73irseb4b4UCAgacfTOqyBwEs0wSuUYy5khtK2+6+XTMmApBMNNMXTmRNOqfCCjE1ubVRts/YpIcEsUz//E4v1CSeUtxmB66XZFQkAveASdoxW2jgOysvw76VQiYo8bhGdEzE4/x+E2e2Od4IXHI0VV9FFEnF0NhjShLiBXiZ60b6QO2+CljmqMYNtaxCReOfOCsFlRzzMhyq+gO+TbKNxlm0dtgd7DNQ27nTjyarazts+hD7mWYyb414ySfB8zA24CNFhpo9nEb1LByHraBxt/b1Tsu4p3oD7PPvNmB+fZL4nDTcMP2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7zLi0DBtPFvGN/R70Qix3ZkOT4iteU2T9DCf5/hbITE=; b=K9dljNJvqjmQAXSTyK2LHj3mdBb7LHVHIM7CwM/JGrDPNOFNUpz2B9w9eXLnpFU2FHoLprZ/+Vn3T3JwTahKgCM8+c1xyBtT2mhSh7cF/DfOqUM1nOSDqag0LiVy7MZgEmU4k0yahQ/phFqkcHLnmG4dNxMPvghO6zSLWUPpnFZRgcquC8t1j+uL3W0wsqIljjeaEQFOl1trST3APLYtvvG+ZrVFUeL00XFvsGXza4sJTQUaYK9l3UBjI4eXSaYvoFhJXYuuKP5iKqQEeKvxi7ksQWxMT/mCljHwpE0pXwpI9CQ3dJpRoSN/X4ZPOCpZhr3PagKaf0y+AqlAFPpt3Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7zLi0DBtPFvGN/R70Qix3ZkOT4iteU2T9DCf5/hbITE=; b=Hf+g6QOTeW0HpKUYcJAPkRWJNARC0ol/mmyP3wEuhS6U6iHFZg+9lR3I9CEiiZ+ALpTkvMyw+Uod3FXlNzf4LUaG/YJfLeKJrQKXxE52gBx7q2m/4BM6wn9QuSAQc8YKYSoovRt7S2e9A4AL+XZ7IYQqAgD0mvwTZu1IYFUf/NE= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3160.namprd10.prod.outlook.com (2603:10b6:a03:151::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.23; Fri, 19 Mar 2021 22:42:35 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3955.024; Fri, 19 Mar 2021 22:42:35 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Andrew Morton , Mike Kravetz Subject: [RFC PATCH 5/8] hugetlb: change free_pool_huge_page to remove_pool_huge_page Date: Fri, 19 Mar 2021 15:42:06 -0700 Message-Id: <20210319224209.150047-6-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210319224209.150047-1-mike.kravetz@oracle.com> References: <20210319224209.150047-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR07CA0013.namprd07.prod.outlook.com (2603:10b6:300:116::23) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR07CA0013.namprd07.prod.outlook.com (2603:10b6:300:116::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18 via Frontend Transport; Fri, 19 Mar 2021 22:42:34 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 48cec62a-3a2a-4288-ee0f-08d8eb284a43 X-MS-TrafficTypeDiagnostic: BYAPR10MB3160: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: NJsubLE/sl2sIAz7koNpW8nyo5u8y9vm0CTTtKkXNwB2XA5VX+2ZUqtuEWNLGH1ckY79lSeHD5QRRD/Sd1a25ptAfzZpUYTwNYldSpzDhJfY+I53k8lbxPu2C792IYoNwfOR5ioeqrTxUhxqas6A6R06dZnHm3aHiP0jAZ5l3ctyfU3lb7hck91uZ5Xtwc23+ZnVd8YksobtS+d1ibKf9DlGdfzbnWrpM6/6dpM0ewlmfeqVi2fD9mTYgrkLDAASGv96mZpto/Dms0cNke5XgH95CRvt/lPc16xBS6nVCiXS2t0wIzd+ljUJ/uOTSkju1M2jR6S4jzHdZwkhW7WH5TuVsAv3vMx9zbG/KhbfQNs4yqRTUUHtoE50xiCRbil5UTU6xpx5f7CbUaT+0pYjP0AWh79Qg0ZVQYGleL7JnyMx/2O15VsopaCo3A0ZXfDFel0IoonOHomg9r0cVa24p22Bd1/Py9x80BMDzmbFMz5Hv4pq/FbCPweb5gvEwNiljSbnujBFEqV/61CAA8pUQZ3uCGsA11Cqmq2Ucms0k69ElSZk6oNoIPTMCDVbarjZ8+YjJXYa9O/16del9xbHiZE8T5fil0NVMm5FVp3YFrye32RITng4HYlGOPnhlAnfKaKMt/8yJ16xWGjXaFwl1qD+ypFrm0A4HthAzQz4PUU= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(396003)(39860400002)(346002)(136003)(366004)(376002)(8936002)(316002)(5660300002)(66946007)(36756003)(4326008)(186003)(66556008)(26005)(66476007)(54906003)(6486002)(16526019)(52116002)(7696005)(83380400001)(8676002)(956004)(44832011)(107886003)(7416002)(1076003)(2616005)(38100700001)(86362001)(478600001)(2906002)(6666004)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: HPDtJkCXd5rtV53kAeWu+Vn7eG2rJtuXdrl5rILt2kesyC877VBQmiH3bTHHXtcyVgJ+XSYMoUYgG/kS9YTri5xEhIuKOPm2DUQn6euUMfQZo4q84bq6K6TIYk2ej0DBsAhlNJ8J9tbIXaRdSpZAq1lGt8qW19ND8F46T1j23HU4Qzhz376YNNh1/5L1NZ00KUG1g2sBGizOoe+bIm4GNz7cVYoe7eXdY5kEZHsSEkhIFL96C6r8+4Rn5QAhehApkojUgljoJYEDDOAXBF7NvJFiP2lUBmfUlsE16KbEiYs2cYPHf46hzhHuQknOh6q2uYfB+sxmx7UIxLWCdLYOadVTXQlIb89t/erSax3lqbj86Y2DK+7vsw13f0t1D36XV8XEX106BP/1WG9dtCF9zHPpn6zco1awl8i0KBYO7kwUTF1lQmpVKQUJrvRl94XHXvLsFbz2zbCjv1dpJITy4x/ZtfPagHa9XyrHT6vGfN6meyYR9xzgkHAGxR9ehGDvX3Ogtk+3oetmDFQnniQA1y3dSkKmiQhzqz6pCdv0OM5OkE6LS0ZFQzHkpGg89lopWrh4ESwV+dIjRR3rWfTTcAP4pfbDOAVIOeVOZlDPgFDqnP4bMcC3zKbawwKavhdJP0naQqQ+ctCRPpfQZPACF2ual6iPLe4yYIO6zbXn73p5bclUfvsqPfy3eol+agbu0kMuov++NOEhfn60mI6oGDmunwNHWNf5+c+iCRHcAe69Feh3cawUqM/tSzPOhbBRBc4U1eTVmt+54e58AVlxZ+ehY2YbLlon6eqmA+rJsN+8+5BP5pkJ6pt5oZw/kWWQbYoXofTjXGrefWzirPGTPTKAJFauBeOykyEOq1rnCksPOoIOeFhat9hDO+TbDAbwwEXqpdyXW1JCw458fQFTfl85FCye15h/x/6QyLp8XE0XwopZCK0rKbSlnhltE5BHHkN7WvlrEU32vdxc56E+7zTAf6NHoT0haw3cIqZ+xjp7wDq0jdnusUoTV/aIT2KGtUu917YdAqjbHrJqGWpseJ761yZAq+ZGmxokBGiXv1rYu9q4uQ0Atc9cUsZjufCI1Jqz99tKaVvI3m34JlLsPi+sNNFwJnBm1PSEN6TrAhKdm5+J7jKF9F84H1XH7hm2r+mFrVM8F0lVuddnnTyw3EATax/9isOPMP4jP/SxC0/Q/5J/8lJG2ACQVmsN6JYNuHIVUPxGsmcwZEeK64ToqLRYXp8COZO5NipvnhS92TQYAPiWWztQ7R2kgn3m1v/Tjuknt1s8pnXjOxKVrTbmKhydkwVc8KeFWMC96xULnFuduJMaNcK+FjCTb7dQqBOK X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 48cec62a-3a2a-4288-ee0f-08d8eb284a43 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2021 22:42:35.0837 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: faIkNy8ZYBorNmrFqt4QOL3iydYEp3GvLkVpL43aH27nwNj6tNyLeE1hG4kXP+w74z+GIMWY0nvj/NT7jgwa1A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3160 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9928 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 suspectscore=0 adultscore=0 bulkscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103190156 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9928 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxlogscore=999 spamscore=0 mlxscore=0 bulkscore=0 suspectscore=0 priorityscore=1501 lowpriorityscore=0 clxscore=1015 adultscore=0 phishscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103190155 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 17FD8E0011E2 X-Stat-Signature: n4ofyj6wqnm1qyxed44pczgyyinjoruw Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf13; identity=mailfrom; envelope-from=""; helo=aserp2130.oracle.com; client-ip=141.146.126.79 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616193793-137816 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: free_pool_huge_page was called with hugetlb_lock held. It would remove a hugetlb page, and then free the corresponding pages to the lower level allocators such as buddy. free_pool_huge_page was called in a loop to remove hugetlb pages and these loops could hold the hugetlb_lock for a considerable time. Create new routine remove_pool_huge_page to replace free_pool_huge_page. remove_pool_huge_page will remove the hugetlb page, and it must be called with the hugetlb_lock held. It will return the removed page and it is the responsibility of the caller to free the page to the lower level allocators. The hugetlb_lock is dropped before freeing to these allocators which results in shorter lock hold times. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 53 +++++++++++++++++++++++++++++----------------------- 1 file changed, 30 insertions(+), 23 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 3028cf10d504..f60a24e326c2 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1184,7 +1184,7 @@ static int hstate_next_node_to_alloc(struct hstate *h, } /* - * helper for free_pool_huge_page() - return the previously saved + * helper for remove_pool_huge_page() - return the previously saved * node ["this node"] from which to free a huge page. Advance the * next node id whether or not we find a free huge page to free so * that the next attempt to free addresses the next node. @@ -1699,16 +1699,18 @@ static int alloc_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, } /* - * Free huge page from pool from next node to free. - * Attempt to keep persistent huge pages more or less - * balanced over allowed nodes. + * Remove huge page from pool from next node to free. Attempt to keep + * persistent huge pages more or less balanced over allowed nodes. + * This routine only 'removes' the hugetlb page. The caller must make + * an additional call to free the page to low level allocators. * Called with hugetlb_lock locked. */ -static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, - bool acct_surplus) +static struct page *remove_pool_huge_page(struct hstate *h, + nodemask_t *nodes_allowed, + bool acct_surplus) { int nr_nodes, node; - int ret = 0; + struct page *page = NULL; for_each_node_mask_to_free(h, nr_nodes, node, nodes_allowed) { /* @@ -1717,23 +1719,14 @@ static int free_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, */ if ((!acct_surplus || h->surplus_huge_pages_node[node]) && !list_empty(&h->hugepage_freelists[node])) { - struct page *page = - list_entry(h->hugepage_freelists[node].next, + page = list_entry(h->hugepage_freelists[node].next, struct page, lru); remove_hugetlb_page(h, page, acct_surplus); - /* - * unlock/lock around update_and_free_page is temporary - * and will be removed with subsequent patch. - */ - spin_unlock(&hugetlb_lock); - update_and_free_page(h, page); - spin_lock(&hugetlb_lock); - ret = 1; break; } } - return ret; + return page; } /* @@ -2064,6 +2057,7 @@ static void return_unused_surplus_pages(struct hstate *h, unsigned long unused_resv_pages) { unsigned long nr_pages; + struct page *page; /* Cannot return gigantic pages currently */ if (hstate_is_gigantic(h)) @@ -2080,7 +2074,7 @@ static void return_unused_surplus_pages(struct hstate *h, * evenly across all nodes with memory. Iterate across these nodes * until we can no longer free unreserved surplus pages. This occurs * when the nodes with surplus pages have no free pages. - * free_pool_huge_page() will balance the freed pages across the + * remove_pool_huge_page() will balance the freed pages across the * on-line nodes with memory and will handle the hstate accounting. * * Note that we decrement resv_huge_pages as we free the pages. If @@ -2090,9 +2084,15 @@ static void return_unused_surplus_pages(struct hstate *h, while (nr_pages--) { h->resv_huge_pages--; unused_resv_pages--; - if (!free_pool_huge_page(h, &node_states[N_MEMORY], 1)) + page = remove_pool_huge_page(h, &node_states[N_MEMORY], 1); + if (!page) goto out; - cond_resched_lock(&hugetlb_lock); + + /* Drop lock and free page to buddy as it could sleep */ + spin_unlock(&hugetlb_lock); + update_and_free_page(h, page); + cond_resched(); + spin_lock(&hugetlb_lock); } out: @@ -2631,6 +2631,7 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, nodemask_t *nodes_allowed) { unsigned long min_count, ret; + struct page *page; NODEMASK_ALLOC(nodemask_t, node_alloc_noretry, GFP_KERNEL); /* @@ -2740,9 +2741,15 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, min_count = min_hp_count(h, count); try_to_free_low(h, count, nodes_allowed); while (min_count < persistent_huge_pages(h)) { - if (!free_pool_huge_page(h, nodes_allowed, 0)) + page = remove_pool_huge_page(h, nodes_allowed, 0); + if (!page) break; - cond_resched_lock(&hugetlb_lock); + + /* Drop lock as free routines may sleep */ + spin_unlock(&hugetlb_lock); + update_and_free_page(h, page); + cond_resched(); + spin_lock(&hugetlb_lock); /* Recompute min_count in case hugetlb_lock was dropped */ min_count = min_hp_count(h, count); From patchwork Fri Mar 19 22:42:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12152079 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DFD8C433DB for ; Fri, 19 Mar 2021 22:43:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 98A896197E for ; Fri, 19 Mar 2021 22:43:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 98A896197E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B9C726B007B; Fri, 19 Mar 2021 18:43:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A65076B0082; Fri, 19 Mar 2021 18:43:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 83DF38D0001; Fri, 19 Mar 2021 18:43:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0121.hostedemail.com [216.40.44.121]) by kanga.kvack.org (Postfix) with ESMTP id 43F266B007B for ; Fri, 19 Mar 2021 18:43:17 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 0AEB58249980 for ; Fri, 19 Mar 2021 22:43:17 +0000 (UTC) X-FDA: 77938101234.21.8798949 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf06.hostedemail.com (Postfix) with ESMTP id 3DDB4C0001EA for ; Fri, 19 Mar 2021 22:43:16 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12JMYU62026030; Fri, 19 Mar 2021 22:42:40 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=1JUOu8sJE85O3bodR4CCM+u8Q8W0RyCN8K/iW9k5u70=; b=Au+77cJ96sFNe/iVUfWY02aoLmpjF7tcQuocm8p/+xtRQeNyVPZlXfjVOFyh+fXwG6cN dnIbrsBuwapGDIvhwC/HT1vBY1FK+V2R3hXuFLNcaG8tGrLM7PQdl2hCTvJGrt8ZLKMj MoS/0U3xuAejmp0cfV93LunCduUTHK90RSUPcqEguL8jHT8jG8I8MTkIPfZ/y3N0pgr9 seTIX15sEIqz6TGbGQo7KyLEGmdPtPIn/jxWS5BXmAWp2midXQLu5fchLjz99pxoc6Uc fCmNwV06cNZVNEpdykfHj216EHFYhMR4NrB9hJ5S3A0dvthHJHgNy4RWM6pVZrhhW6kZ gw== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by aserp2130.oracle.com with ESMTP id 378jwbvhmc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 19 Mar 2021 22:42:40 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12JMesZo001873; Fri, 19 Mar 2021 22:42:40 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2177.outbound.protection.outlook.com [104.47.58.177]) by aserp3030.oracle.com with ESMTP id 3796yxynfu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 19 Mar 2021 22:42:39 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PXJuCvF9ZyLFbAjKL4SlSFLHY49EZKbVc9jWPXd90dAS6UkqW06gaP8wwyRYISYcGTImObSggInb8nOwWBjF0DyoBKlCH/ioYkO6Fuy1Jc3LKsIbR+siIIphlt86nQc1V48aQMNkgfGYYWNAzG81I068wGEp/L9CQHtNiKniB06goYUck1Pi+fkJbGdjqxJxr8NazL9sSsHKZ1EFiTzbwQKP6qUrHCnGB1Pr9+E/ijxUAL84so/fYIFTmXRvMxiHuOkdYZnUtP1ZzUBVjUFMph1aqIAOKjIMqDvs3VGn3IWGO51szmajx63SegqrO9zj3YE6zl2FY6cZAdnHDJc9Zw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1JUOu8sJE85O3bodR4CCM+u8Q8W0RyCN8K/iW9k5u70=; b=WksGwmxkeMiTcV/EVJmd7LQTzslRutCZuLKslJvWiRqY41l0jmNbbYF5fSv/2qDI1w3ULNfpZCt+/Vfquydx82TIm5b6RhM7HuNeclmGts9DiZDKO7bNQSzYlEdPC9yor2aeMuvQn8zxtuQEerqUSU8w//0fKrkd8osJy/g9a2yad3+sKCsXwaJd8eJm2Yna+7/HisTzjuCRUDMimUchWUsZdrw/sd2vt4xsCCo9FS6Wz5fgrdQiyO1GgrrD7Zd8u8RgfHCQiNQu6B778++pDB2WqCcVO/3IhEytU2NDli4sGIj8MNDcKaQA9l/WFi+GK6Nb4aNt1eE0RXVmDOzytQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1JUOu8sJE85O3bodR4CCM+u8Q8W0RyCN8K/iW9k5u70=; b=StMb/wbUUYY8rR8CGqVg99sNZrJb+XQRkCscM6Cr8H9eRfBi4qcbaJUUmvJiC7lw3F9DwNIxcDH3WTZruk9jwpuQbIF5AR2RXzlgS8PcJptOKr1F9d8NYpaja8edAlxaz0pOglyu6F4EgfTMROO8VwzKJl82F4af5W6ZhvdR0uM= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3160.namprd10.prod.outlook.com (2603:10b6:a03:151::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.23; Fri, 19 Mar 2021 22:42:37 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3955.024; Fri, 19 Mar 2021 22:42:37 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Andrew Morton , Mike Kravetz Subject: [RFC PATCH 6/8] hugetlb: make free_huge_page irq safe Date: Fri, 19 Mar 2021 15:42:07 -0700 Message-Id: <20210319224209.150047-7-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210319224209.150047-1-mike.kravetz@oracle.com> References: <20210319224209.150047-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR07CA0013.namprd07.prod.outlook.com (2603:10b6:300:116::23) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR07CA0013.namprd07.prod.outlook.com (2603:10b6:300:116::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18 via Frontend Transport; Fri, 19 Mar 2021 22:42:36 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6ca269bf-a230-4846-2979-08d8eb284b67 X-MS-TrafficTypeDiagnostic: BYAPR10MB3160: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1728; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4qWAJZGz3Au512eGKeyklBzuD/lkBu798gk+JbkShG4mwhmYiL0qG3DqDT1gnAPll0wkWepmj6tK7XLih7aa3aAxygIrq54yF6rGno7WRvoJgXpfjggyBGTLwTA0FEf/Q7YJrisfqTpTTc9SqdSxWSkTdAuEhoEMZmDcBdsGi1Sdln93Cqg3zkmkTLeiHQBAPogS3hQsSebSz5E5tH76ymLPDl/VKrXJOq8mn5jpXr8srdJsMy/TNvoC/b900Xg8Se/IRY7gzuRgseHnZ0/Kqo483SphvPc0knhKdg3Jrdc4RKdFOyJUu6yMVYavl6dGGfIkCzRiC6ekV7ddYSxvG6btGUep9s1MTRs3wkGPp/Uhhgv4J4V+3ipWsCx5DWN6+Msv726vOMsL6kvwFeWU+CqjEcxhpH8TZkXb1UVfYOoUjt515z9rRl50fgc/ZzwLVSayzR6BVmtMxxLDNqMinBei0hCiM9Ex2R/NQ1qzCQ1zD9yAJ6BaftqH6TUetiJW/B3pWTdiA3ICQR1sK2MaNtMSFGv+NM2Ov/GOGHnQUaK0fDtWnaBQfYvOf0dnRaG/6J3X8VPoxOamz3AWnc1yT/Y+ZUIRRf/8Tqn4Pj247xCowGCqom/2BPePzkD1a6gm8tPjQgk90Yp5acXNRM8UjBras3l2C8W0R631xmrJyh8Y/J/pxBNYt2ly32ZROJDGRerAk35wkM4UamPWzlB+MHtWnBei6Ly41HFr5b/vy+kji8bulQ3K5koNZJJTZqw3 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(396003)(39860400002)(346002)(136003)(366004)(376002)(8936002)(316002)(5660300002)(66946007)(36756003)(4326008)(186003)(66556008)(26005)(66476007)(54906003)(6486002)(16526019)(30864003)(52116002)(7696005)(83380400001)(8676002)(956004)(966005)(44832011)(107886003)(7416002)(1076003)(2616005)(38100700001)(86362001)(478600001)(2906002)(6666004)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: paFHojV0twthsyDxj1+P7mh8ffPCYr6AKFLNzvnMnY7rzzf1+kfwwH3U7iz3xuz/3eqttBZczCwhy3XvqnhfRp1DvZxZi88cfi5lrzi7A23D/fEmShUv/kGhFJ382107E7aULCbmrbPDd6S7QxBeJ4Vn9VwuBkiG4I5JiDLwKdKj8AE4llPiZTDqdLf4fkK6ckwhIA4Qb2/x29us/357DOHg0rLxJYsYUDRHOB4lARkH7XuP8crkedv9tpt1jsWFWDH2bWl0NGy0LOvfHZP7QQSWmHtmxdYNTTJ5IeCM/TV5dE7GfzIVKQRfD9YLJhI/7tfMqmOvt/r8H0+wVtUK/N9azI6ELFoMmwN1d1kQ2kRgMqIORHJWY6TSPfMmB8US51hJrCFCuMJfRotAtJKAmvrucWGQde/H04/N5K4LQ80h5A8/3NOWmfn1DlDtFdaAWkbP+MeeECRnrUFM29Z5PQExcofv8U2apBTR1vZymhc+RjuUqLeJQYa75UE3PnWHtPiSh+TQeU4miQLtrFfSEJXEFzh/3ZVPQP773dbwsDNC3jw7GpO59Otm1QnFZYbmVxpWVBwFptvF16GnAETG4faAAfnO4Z0UY/3gMEtOaEFcVhmxbh4szdd21wCYOGaaowANqgeFdqx5X/R2Z4ehUxdSOGO1Rle0FKNZ0/YvyJ0cxbX3lVj1Y0x/iOKQRnS8cdjC4omacjdeCqK2YHmzhG9FU9R8LtDwyFRW9PU+6BNGYlLuBYrE2toCB1eVps98cuQ21YU3XHLhi35W0kLVcLQpj5ZYnLghNAhwwf0E0WbOjH0t6Qr0VN49gUd0+Aq8CDY1wU7eZxQt5Iq66HqySP00oeDYJGlieb0AetNCezeLEvTIo0r4Upq2TG1JV44cCHjsJSFNa7GfU8ewZEzs+FQtayPvA1PI440LvrwgSoV67TZC5ByGmXeRZSOGIbJFeGBjnUt2H7Wenv8ly6NtqW2ElyD3m5n0V0OpFyYf8dgRmEotjATkOcxjqkGZuTSvqYLsLckR5c9WImenIzbg+NOmWFet6qUXgWBpi+tWXvNxEDQDylcwjpEnqxdjH5c1W2VB5J+5gQbWqmJ0KI3TmTI6uIlHcoz+U+LdHENVCMNGfzxiLJ9cdsqZJFCQDKeYiEs0hneKXexV1CeSMaHr/WMeagpPKpSQ1c4RRJw2TUkPVnwMKJMA/NghoPA82kT7lum7U1CP+WwXGPQtvPSIyQ2e0MRx7WckYbSLJnnnzRJRJmbV7kRE4YfarXxpji/lt+nPYlG4gc1R0R6Ayp71mcSt0UC/H+dos0ynI/SFEn6W30VtDqT1omZ8mmf6h5ID X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6ca269bf-a230-4846-2979-08d8eb284b67 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2021 22:42:37.0261 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: c+C9rmx50i1+XZxKDtmxdziV/m1kElwBmEnzmXDX/z/33ajNEriJjJGP7CfCOZE/lO679hxrgSG8zQyeF+eKbw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3160 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9928 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 spamscore=0 bulkscore=0 mlxlogscore=999 mlxscore=0 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103190156 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9928 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxlogscore=999 spamscore=0 mlxscore=0 bulkscore=0 suspectscore=0 priorityscore=1501 lowpriorityscore=0 clxscore=1015 adultscore=0 phishscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103190155 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 3DDB4C0001EA X-Stat-Signature: hajbhk5p9xe4h1zudjmnfoypkxfic7og Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf06; identity=mailfrom; envelope-from=""; helo=aserp2130.oracle.com; client-ip=141.146.126.79 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616193796-68887 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Commit c77c0a8ac4c5 ("mm/hugetlb: defer freeing of huge pages if in non-task context") was added to address the issue of free_huge_page being called from irq context. That commit hands off free_huge_page processing to a workqueue if !in_task. However, as seen in [1] this does not cover all cases. Instead, make the locks taken in the free_huge_page irq safe. This patch does the following: - Make hugetlb_lock irq safe. This is mostly a simple process of changing spin_*lock calls to spin_*lock_irq* calls. - Make subpool lock irq safe in a similar manner. - Revert the !in_task check and workqueue handoff. [1] https://lore.kernel.org/linux-mm/000000000000f1c03b05bc43aadc@google.com/ Signed-off-by: Mike Kravetz Reported-by: kernel test robot --- mm/hugetlb.c | 206 ++++++++++++++++++++------------------------ mm/hugetlb_cgroup.c | 10 ++- 2 files changed, 100 insertions(+), 116 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index f60a24e326c2..82614bbe7bb9 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -94,9 +94,10 @@ static inline bool subpool_is_free(struct hugepage_subpool *spool) return true; } -static inline void unlock_or_release_subpool(struct hugepage_subpool *spool) +static inline void unlock_or_release_subpool(struct hugepage_subpool *spool, + unsigned long irq_flags) { - spin_unlock(&spool->lock); + spin_unlock_irqrestore(&spool->lock, irq_flags); /* If no pages are used, and no other handles to the subpool * remain, give up any reservations based on minimum size and @@ -135,10 +136,12 @@ struct hugepage_subpool *hugepage_new_subpool(struct hstate *h, long max_hpages, void hugepage_put_subpool(struct hugepage_subpool *spool) { - spin_lock(&spool->lock); + unsigned long flags; + + spin_lock_irqsave(&spool->lock, flags); BUG_ON(!spool->count); spool->count--; - unlock_or_release_subpool(spool); + unlock_or_release_subpool(spool, flags); } /* @@ -153,11 +156,12 @@ static long hugepage_subpool_get_pages(struct hugepage_subpool *spool, long delta) { long ret = delta; + unsigned long flags; if (!spool) return ret; - spin_lock(&spool->lock); + spin_lock_irqsave(&spool->lock, flags); if (spool->max_hpages != -1) { /* maximum size accounting */ if ((spool->used_hpages + delta) <= spool->max_hpages) @@ -184,7 +188,7 @@ static long hugepage_subpool_get_pages(struct hugepage_subpool *spool, } unlock_ret: - spin_unlock(&spool->lock); + spin_unlock_irqrestore(&spool->lock, flags); return ret; } @@ -198,11 +202,12 @@ static long hugepage_subpool_put_pages(struct hugepage_subpool *spool, long delta) { long ret = delta; + unsigned long flags; if (!spool) return delta; - spin_lock(&spool->lock); + spin_lock_irqsave(&spool->lock, flags); if (spool->max_hpages != -1) /* maximum size accounting */ spool->used_hpages -= delta; @@ -223,7 +228,7 @@ static long hugepage_subpool_put_pages(struct hugepage_subpool *spool, * If hugetlbfs_put_super couldn't free spool due to an outstanding * quota reference, free it now. */ - unlock_or_release_subpool(spool); + unlock_or_release_subpool(spool, flags); return ret; } @@ -1380,7 +1385,7 @@ struct hstate *size_to_hstate(unsigned long size) return NULL; } -static void __free_huge_page(struct page *page) +void free_huge_page(struct page *page) { /* * Can't pass hstate in here because it is called from the @@ -1390,6 +1395,7 @@ static void __free_huge_page(struct page *page) int nid = page_to_nid(page); struct hugepage_subpool *spool = hugetlb_page_subpool(page); bool restore_reserve; + unsigned long flags; VM_BUG_ON_PAGE(page_count(page), page); VM_BUG_ON_PAGE(page_mapcount(page), page); @@ -1418,7 +1424,7 @@ static void __free_huge_page(struct page *page) restore_reserve = true; } - spin_lock(&hugetlb_lock); + spin_lock_irqsave(&hugetlb_lock, flags); ClearHPageMigratable(page); hugetlb_cgroup_uncharge_page(hstate_index(h), pages_per_huge_page(h), page); @@ -1429,80 +1435,34 @@ static void __free_huge_page(struct page *page) if (HPageTemporary(page)) { remove_hugetlb_page(h, page, false); - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); update_and_free_page(h, page); } else if (h->surplus_huge_pages_node[nid]) { /* remove the page from active list */ remove_hugetlb_page(h, page, true); - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); update_and_free_page(h, page); } else { arch_clear_hugepage_flags(page); enqueue_huge_page(h, page); - spin_unlock(&hugetlb_lock); - } -} - -/* - * As free_huge_page() can be called from a non-task context, we have - * to defer the actual freeing in a workqueue to prevent potential - * hugetlb_lock deadlock. - * - * free_hpage_workfn() locklessly retrieves the linked list of pages to - * be freed and frees them one-by-one. As the page->mapping pointer is - * going to be cleared in __free_huge_page() anyway, it is reused as the - * llist_node structure of a lockless linked list of huge pages to be freed. - */ -static LLIST_HEAD(hpage_freelist); - -static void free_hpage_workfn(struct work_struct *work) -{ - struct llist_node *node; - struct page *page; - - node = llist_del_all(&hpage_freelist); - - while (node) { - page = container_of((struct address_space **)node, - struct page, mapping); - node = node->next; - __free_huge_page(page); + spin_unlock_irqrestore(&hugetlb_lock, flags); } } -static DECLARE_WORK(free_hpage_work, free_hpage_workfn); - -void free_huge_page(struct page *page) -{ - /* - * Defer freeing if in non-task context to avoid hugetlb_lock deadlock. - */ - if (!in_task()) { - /* - * Only call schedule_work() if hpage_freelist is previously - * empty. Otherwise, schedule_work() had been called but the - * workfn hasn't retrieved the list yet. - */ - if (llist_add((struct llist_node *)&page->mapping, - &hpage_freelist)) - schedule_work(&free_hpage_work); - return; - } - - __free_huge_page(page); -} static void prep_new_huge_page(struct hstate *h, struct page *page, int nid) { + unsigned long flags; + INIT_LIST_HEAD(&page->lru); set_compound_page_dtor(page, HUGETLB_PAGE_DTOR); hugetlb_set_page_subpool(page, NULL); set_hugetlb_cgroup(page, NULL); set_hugetlb_cgroup_rsvd(page, NULL); - spin_lock(&hugetlb_lock); + spin_lock_irqsave(&hugetlb_lock, flags); h->nr_huge_pages++; h->nr_huge_pages_node[nid]++; ClearHPageFreed(page); - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); } static void prep_compound_gigantic_page(struct page *page, unsigned int order) @@ -1742,13 +1702,14 @@ static struct page *remove_pool_huge_page(struct hstate *h, int dissolve_free_huge_page(struct page *page) { int rc = -EBUSY; + unsigned long flags; retry: /* Not to disrupt normal path by vainly holding hugetlb_lock */ if (!PageHuge(page)) return 0; - spin_lock(&hugetlb_lock); + spin_lock_irqsave(&hugetlb_lock, flags); if (!PageHuge(page)) { rc = 0; goto out; @@ -1765,7 +1726,12 @@ int dissolve_free_huge_page(struct page *page) * when it is dissolved. */ if (unlikely(!HPageFreed(head))) { - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); + + /* + * ??? Does this retry make any sense now that + * hugetlb_lock is held with irqs disabled ??? + */ cond_resched(); /* @@ -1789,12 +1755,12 @@ int dissolve_free_huge_page(struct page *page) } remove_hugetlb_page(h, page, false); h->max_huge_pages--; - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); update_and_free_page(h, head); return 0; } out: - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); return rc; } @@ -1832,20 +1798,21 @@ static struct page *alloc_surplus_huge_page(struct hstate *h, gfp_t gfp_mask, int nid, nodemask_t *nmask) { struct page *page = NULL; + unsigned long flags; if (hstate_is_gigantic(h)) return NULL; - spin_lock(&hugetlb_lock); + spin_lock_irqsave(&hugetlb_lock, flags); if (h->surplus_huge_pages >= h->nr_overcommit_huge_pages) goto out_unlock; - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); page = alloc_fresh_huge_page(h, gfp_mask, nid, nmask, NULL); if (!page) return NULL; - spin_lock(&hugetlb_lock); + spin_lock_irqsave(&hugetlb_lock, flags); /* * We could have raced with the pool size change. * Double check that and simply deallocate the new page @@ -1855,7 +1822,7 @@ static struct page *alloc_surplus_huge_page(struct hstate *h, gfp_t gfp_mask, */ if (h->surplus_huge_pages >= h->nr_overcommit_huge_pages) { SetHPageTemporary(page); - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); put_page(page); return NULL; } else { @@ -1864,7 +1831,7 @@ static struct page *alloc_surplus_huge_page(struct hstate *h, gfp_t gfp_mask, } out_unlock: - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); return page; } @@ -1914,17 +1881,19 @@ struct page *alloc_buddy_huge_page_with_mpol(struct hstate *h, struct page *alloc_huge_page_nodemask(struct hstate *h, int preferred_nid, nodemask_t *nmask, gfp_t gfp_mask) { - spin_lock(&hugetlb_lock); + unsigned long flags; + + spin_lock_irqsave(&hugetlb_lock, flags); if (h->free_huge_pages - h->resv_huge_pages > 0) { struct page *page; page = dequeue_huge_page_nodemask(h, gfp_mask, preferred_nid, nmask); if (page) { - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); return page; } } - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); return alloc_migrate_huge_page(h, gfp_mask, preferred_nid, nmask); } @@ -1951,7 +1920,8 @@ struct page *alloc_huge_page_vma(struct hstate *h, struct vm_area_struct *vma, * Increase the hugetlb pool such that it can accommodate a reservation * of size 'delta'. */ -static int gather_surplus_pages(struct hstate *h, long delta) +static int gather_surplus_pages(struct hstate *h, long delta, + unsigned long *irq_flags) __must_hold(&hugetlb_lock) { struct list_head surplus_list; @@ -1972,7 +1942,7 @@ static int gather_surplus_pages(struct hstate *h, long delta) ret = -ENOMEM; retry: - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, *irq_flags); for (i = 0; i < needed; i++) { page = alloc_surplus_huge_page(h, htlb_alloc_mask(h), NUMA_NO_NODE, NULL); @@ -1989,7 +1959,7 @@ static int gather_surplus_pages(struct hstate *h, long delta) * After retaking hugetlb_lock, we need to recalculate 'needed' * because either resv_huge_pages or free_huge_pages may have changed. */ - spin_lock(&hugetlb_lock); + spin_lock_irqsave(&hugetlb_lock, *irq_flags); needed = (h->resv_huge_pages + delta) - (h->free_huge_pages + allocated); if (needed > 0) { @@ -2029,12 +1999,12 @@ static int gather_surplus_pages(struct hstate *h, long delta) enqueue_huge_page(h, page); } free: - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, *irq_flags); /* Free unnecessary surplus pages to the buddy allocator */ list_for_each_entry_safe(page, tmp, &surplus_list, lru) put_page(page); - spin_lock(&hugetlb_lock); + spin_lock_irqsave(&hugetlb_lock, *irq_flags); return ret; } @@ -2054,7 +2024,8 @@ static int gather_surplus_pages(struct hstate *h, long delta) * number of huge pages we plan to free when dropping the lock. */ static void return_unused_surplus_pages(struct hstate *h, - unsigned long unused_resv_pages) + unsigned long unused_resv_pages, + unsigned long *irq_flags) { unsigned long nr_pages; struct page *page; @@ -2089,10 +2060,10 @@ static void return_unused_surplus_pages(struct hstate *h, goto out; /* Drop lock and free page to buddy as it could sleep */ - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, *irq_flags); update_and_free_page(h, page); cond_resched(); - spin_lock(&hugetlb_lock); + spin_lock_irqsave(&hugetlb_lock, *irq_flags); } out: @@ -2281,6 +2252,7 @@ struct page *alloc_huge_page(struct vm_area_struct *vma, int ret, idx; struct hugetlb_cgroup *h_cg; bool deferred_reserve; + unsigned long flags; idx = hstate_index(h); /* @@ -2332,7 +2304,7 @@ struct page *alloc_huge_page(struct vm_area_struct *vma, if (ret) goto out_uncharge_cgroup_reservation; - spin_lock(&hugetlb_lock); + spin_lock_irqsave(&hugetlb_lock, flags); /* * glb_chg is passed to indicate whether or not a page must be taken * from the global free pool (global change). gbl_chg == 0 indicates @@ -2340,7 +2312,7 @@ struct page *alloc_huge_page(struct vm_area_struct *vma, */ page = dequeue_huge_page_vma(h, vma, addr, avoid_reserve, gbl_chg); if (!page) { - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); page = alloc_buddy_huge_page_with_mpol(h, vma, addr); if (!page) goto out_uncharge_cgroup; @@ -2348,7 +2320,7 @@ struct page *alloc_huge_page(struct vm_area_struct *vma, SetHPageRestoreReserve(page); h->resv_huge_pages--; } - spin_lock(&hugetlb_lock); + spin_lock_irqsave(&hugetlb_lock, flags); list_add(&page->lru, &h->hugepage_activelist); /* Fall through */ } @@ -2361,7 +2333,7 @@ struct page *alloc_huge_page(struct vm_area_struct *vma, h_cg, page); } - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); hugetlb_set_page_subpool(page, spool); @@ -2556,7 +2528,8 @@ static inline unsigned long min_hp_count(struct hstate *h, unsigned long count) #ifdef CONFIG_HIGHMEM static void try_to_free_low(struct hstate *h, unsigned long count, - nodemask_t *nodes_allowed) + nodemask_t *nodes_allowed, + unsigned long *irq_flags) { int i; unsigned long min_count = min_hp_count(h, count); @@ -2575,9 +2548,9 @@ static void try_to_free_low(struct hstate *h, unsigned long count, remove_hugetlb_page(h, page, false); h->free_huge_pages--; h->free_huge_pages_node[page_to_nid(page)]--; - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, *irq_flags); update_and_free_page(h, page); - spin_lock(&hugetlb_lock); + spin_lock_irqsave(&hugetlb_lock, *irq_flags); /* * update_and_free_page could have dropped lock so @@ -2589,7 +2562,8 @@ static void try_to_free_low(struct hstate *h, unsigned long count, } #else static inline void try_to_free_low(struct hstate *h, unsigned long count, - nodemask_t *nodes_allowed) + nodemask_t *nodes_allowed, + unsigned long *irq_flags) { } #endif @@ -2633,6 +2607,7 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, unsigned long min_count, ret; struct page *page; NODEMASK_ALLOC(nodemask_t, node_alloc_noretry, GFP_KERNEL); + unsigned long flags; /* * Bit mask controlling how hard we retry per-node allocations. @@ -2646,7 +2621,7 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, /* mutex prevents concurrent adjustments for the same hstate */ mutex_lock(&h->mutex); - spin_lock(&hugetlb_lock); + spin_lock_irqsave(&hugetlb_lock, flags); /* * Check for a node specific request. @@ -2677,7 +2652,7 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, */ if (hstate_is_gigantic(h) && !IS_ENABLED(CONFIG_CONTIG_ALLOC)) { if (count > persistent_huge_pages(h)) { - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); mutex_unlock(&h->mutex); NODEMASK_FREE(node_alloc_noretry); return -EINVAL; @@ -2707,14 +2682,14 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, * page, free_huge_page will handle it by freeing the page * and reducing the surplus. */ - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); /* yield cpu to avoid soft lockup */ cond_resched(); ret = alloc_pool_huge_page(h, nodes_allowed, node_alloc_noretry); - spin_lock(&hugetlb_lock); + spin_lock_irqsave(&hugetlb_lock, flags); if (!ret) goto out; @@ -2739,7 +2714,7 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, * sysctls are changed, or the surplus pages go out of use. */ min_count = min_hp_count(h, count); - try_to_free_low(h, count, nodes_allowed); + try_to_free_low(h, count, nodes_allowed, &flags); while (min_count < persistent_huge_pages(h)) { page = remove_pool_huge_page(h, nodes_allowed, 0); if (!page) @@ -2760,7 +2735,7 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, } out: h->max_huge_pages = persistent_huge_pages(h); - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); mutex_unlock(&h->mutex); NODEMASK_FREE(node_alloc_noretry); @@ -2908,6 +2883,7 @@ static ssize_t nr_overcommit_hugepages_store(struct kobject *kobj, int err; unsigned long input; struct hstate *h = kobj_to_hstate(kobj, NULL); + unsigned long flags; if (hstate_is_gigantic(h)) return -EINVAL; @@ -2916,9 +2892,9 @@ static ssize_t nr_overcommit_hugepages_store(struct kobject *kobj, if (err) return err; - spin_lock(&hugetlb_lock); + spin_lock_irqsave(&hugetlb_lock, flags); h->nr_overcommit_huge_pages = input; - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); return count; } @@ -3490,6 +3466,7 @@ int hugetlb_overcommit_handler(struct ctl_table *table, int write, struct hstate *h = &default_hstate; unsigned long tmp; int ret; + unsigned long flags; if (!hugepages_supported()) return -EOPNOTSUPP; @@ -3505,9 +3482,9 @@ int hugetlb_overcommit_handler(struct ctl_table *table, int write, goto out; if (write) { - spin_lock(&hugetlb_lock); + spin_lock_irqsave(&hugetlb_lock, flags); h->nr_overcommit_huge_pages = tmp; - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); } out: return ret; @@ -3599,11 +3576,12 @@ unsigned long hugetlb_total_pages(void) static int hugetlb_acct_memory(struct hstate *h, long delta) { int ret = -ENOMEM; + unsigned long flags; if (!delta) return 0; - spin_lock(&hugetlb_lock); + spin_lock_irqsave(&hugetlb_lock, flags); /* * When cpuset is configured, it breaks the strict hugetlb page * reservation as the accounting is done on a global variable. Such @@ -3628,21 +3606,21 @@ static int hugetlb_acct_memory(struct hstate *h, long delta) * above. */ if (delta > 0) { - if (gather_surplus_pages(h, delta) < 0) + if (gather_surplus_pages(h, delta, &flags) < 0) goto out; if (delta > allowed_mems_nr(h)) { - return_unused_surplus_pages(h, delta); + return_unused_surplus_pages(h, delta, &flags); goto out; } } ret = 0; if (delta < 0) - return_unused_surplus_pages(h, (unsigned long) -delta); + return_unused_surplus_pages(h, (unsigned long) -delta, &flags); out: - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); return ret; } @@ -5654,8 +5632,9 @@ follow_huge_pgd(struct mm_struct *mm, unsigned long address, pgd_t *pgd, int fla bool isolate_huge_page(struct page *page, struct list_head *list) { bool ret = true; + unsigned long flags; - spin_lock(&hugetlb_lock); + spin_lock_irqsave(&hugetlb_lock, flags); if (!PageHeadHuge(page) || !HPageMigratable(page) || !get_page_unless_zero(page)) { @@ -5665,22 +5644,25 @@ bool isolate_huge_page(struct page *page, struct list_head *list) ClearHPageMigratable(page); list_move_tail(&page->lru, list); unlock: - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); return ret; } void putback_active_hugepage(struct page *page) { - spin_lock(&hugetlb_lock); + unsigned long flags; + + spin_lock_irqsave(&hugetlb_lock, flags); SetHPageMigratable(page); list_move_tail(&page->lru, &(page_hstate(page))->hugepage_activelist); - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); put_page(page); } void move_hugetlb_state(struct page *oldpage, struct page *newpage, int reason) { struct hstate *h = page_hstate(oldpage); + unsigned long flags; hugetlb_cgroup_migrate(oldpage, newpage); set_page_owner_migrate_reason(newpage, reason); @@ -5702,12 +5684,12 @@ void move_hugetlb_state(struct page *oldpage, struct page *newpage, int reason) SetHPageTemporary(oldpage); ClearHPageTemporary(newpage); - spin_lock(&hugetlb_lock); + spin_lock_irqsave(&hugetlb_lock, flags); if (h->surplus_huge_pages_node[old_nid]) { h->surplus_huge_pages_node[old_nid]--; h->surplus_huge_pages_node[new_nid]++; } - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); } } diff --git a/mm/hugetlb_cgroup.c b/mm/hugetlb_cgroup.c index f68b51fcda3d..b3b5759e44b4 100644 --- a/mm/hugetlb_cgroup.c +++ b/mm/hugetlb_cgroup.c @@ -200,15 +200,16 @@ static void hugetlb_cgroup_css_offline(struct cgroup_subsys_state *css) struct hstate *h; struct page *page; int idx; + unsigned long flags; do { idx = 0; for_each_hstate(h) { - spin_lock(&hugetlb_lock); + spin_lock_irqsave(&hugetlb_lock, flags); list_for_each_entry(page, &h->hugepage_activelist, lru) hugetlb_cgroup_move_parent(idx, h_cg, page); - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); idx++; } cond_resched(); @@ -774,12 +775,13 @@ void hugetlb_cgroup_migrate(struct page *oldhpage, struct page *newhpage) struct hugetlb_cgroup *h_cg; struct hugetlb_cgroup *h_cg_rsvd; struct hstate *h = page_hstate(oldhpage); + unsigned long flags; if (hugetlb_cgroup_disabled()) return; VM_BUG_ON_PAGE(!PageHuge(oldhpage), oldhpage); - spin_lock(&hugetlb_lock); + spin_lock_irqsave(&hugetlb_lock, flags); h_cg = hugetlb_cgroup_from_page(oldhpage); h_cg_rsvd = hugetlb_cgroup_from_page_rsvd(oldhpage); set_hugetlb_cgroup(oldhpage, NULL); @@ -789,7 +791,7 @@ void hugetlb_cgroup_migrate(struct page *oldhpage, struct page *newhpage) set_hugetlb_cgroup(newhpage, h_cg); set_hugetlb_cgroup_rsvd(newhpage, h_cg_rsvd); list_move(&newhpage->lru, &h->hugepage_activelist); - spin_unlock(&hugetlb_lock); + spin_unlock_irqrestore(&hugetlb_lock, flags); return; } From patchwork Fri Mar 19 22:42:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12152077 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F31DCC433C1 for ; Fri, 19 Mar 2021 22:43:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8B49D61981 for ; Fri, 19 Mar 2021 22:43:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8B49D61981 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 531A96B0081; Fri, 19 Mar 2021 18:43:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4C05E6B0083; Fri, 19 Mar 2021 18:43:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 275106B0082; Fri, 19 Mar 2021 18:43:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0106.hostedemail.com [216.40.44.106]) by kanga.kvack.org (Postfix) with ESMTP id EAC136B007B for ; Fri, 19 Mar 2021 18:43:16 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id B1F69AF8C for ; Fri, 19 Mar 2021 22:43:16 +0000 (UTC) X-FDA: 77938101192.16.FEDEF3C Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf05.hostedemail.com (Postfix) with ESMTP id E8836E0011F4 for ; Fri, 19 Mar 2021 22:43:15 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12JMYWrg111010; Fri, 19 Mar 2021 22:42:43 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=F0xSyHo4sNncbJblqG4BraCgk5CJapCgb4Ww1gZK4J0=; b=vq2+76+mQW/EweTzuhNykmGsZM/SfiA3nJ202CXeDimecEKlYGd/t3lkV8ZzUqHIdNtN x2Al91fidpxUeyacCmHhetlwu/IX+is83T3i+GdKckHqqZwwu4dqSgreSdg4M7tRoi1Q xk6TT8MCiITxgBJAXDa8xzIUmwabnJQAsbVfPAhtirefpktTnoeDY3KOnCibZ2apMbZ7 mWgQ9juEstY+CxdM3WJ7NOtAkLLx8+MwclQGnp3tltK8AyBo/KA871aAcl+6afM99KW8 956fGVy3lAgm7DmFB8yefy9vAZGIZEwxYUoiMw9c65PoOANtxGf99S4dtDfBAUqOFWMi mQ== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by userp2130.oracle.com with ESMTP id 37a4em1cjd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 19 Mar 2021 22:42:43 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12JMer2x001735; Fri, 19 Mar 2021 22:42:42 GMT Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2104.outbound.protection.outlook.com [104.47.70.104]) by aserp3030.oracle.com with ESMTP id 3796yxyng7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 19 Mar 2021 22:42:42 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WeJCAKWEhc9LNqR2YqpPRfoqXk4nSgL1rv0QsEUQQVGw89s5Ha0n8Mj/gAYE/ztSDqjE9kwbfAsw+m6s1d88aMDlREgYXx88bSKR1qUnQeU02KBCgokMuxI8q0ZN6oMNAmi+TFP3KybAz1zahKyvFgGzCnV9YEt+VJDOSpeJ7nLq0HMLK5Aph7NNqBlolVWiPbZKYtdm3/p/wgn62kx+k7lAWcwmGflmQ51BEBwjMt6PO96bI9Yi2+ZijXjOAsgdZgFd7Kr30B3HVcEccZpQVJYInzysg4/9YVX4//wYtXMTJHvmzveCEFoEmU/AfdLyeNmjDHsH7JEYV9RWVdZiAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=F0xSyHo4sNncbJblqG4BraCgk5CJapCgb4Ww1gZK4J0=; b=dkp4dQlHQOaYuoYqIu58iZfMZZX0zodrXIfpsB4uz7bI7oHwRz2s0DgWNz163oZoOJQgro6zn8aly3Lz/GzDjqcL9MlZMJgyYE6ty5xA/YuTSoqTnHqgT3JgaEUwPL69VXQ0KZc2grw/JW8z+enm9LGbc2I3FsVSlmX1jdvwUEsNKhmR6g2/qlrGCE3dpmyyW0CmXqPIC9CIMn2oMHX6NugzLK28lqxg1wSr+Qi1/E3R5FNZAlFKGvVW/TsOI/Q9+uH4inGqp1+GMI5XbEdX3xfq207JyrWbnTOkqbhbSzUQAbNu+FLwBIby7lPntHfmR7cO9p9QgNaFEDDmlnVQjQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=F0xSyHo4sNncbJblqG4BraCgk5CJapCgb4Ww1gZK4J0=; b=qS9yi8xV46DnqMfmTKlkbG9q0+NCHugEdKA7V1Q1vfsYTqvWCyFdyvGweCdaGIGXPcoJeV0twDg3nLBq6QDb7Ucgf5cUROhY8f3u5M4cM9jE/nGh8qxfoSb9nwNwBdwF22dMfUf14yhb6EKN8GovzF90zaNFsS1wD8lAoMt8FwM= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BY5PR10MB4211.namprd10.prod.outlook.com (2603:10b6:a03:20c::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3933.31; Fri, 19 Mar 2021 22:42:39 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3955.024; Fri, 19 Mar 2021 22:42:39 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Andrew Morton , Mike Kravetz Subject: [RFC PATCH 7/8] hugetlb: add update_and_free_page_no_sleep for irq context Date: Fri, 19 Mar 2021 15:42:08 -0700 Message-Id: <20210319224209.150047-8-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210319224209.150047-1-mike.kravetz@oracle.com> References: <20210319224209.150047-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR07CA0013.namprd07.prod.outlook.com (2603:10b6:300:116::23) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR07CA0013.namprd07.prod.outlook.com (2603:10b6:300:116::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18 via Frontend Transport; Fri, 19 Mar 2021 22:42:38 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 86e9ce0d-8429-42f3-a226-08d8eb284cd8 X-MS-TrafficTypeDiagnostic: BY5PR10MB4211: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4NbMZvWZ6mleUJLLiAZHKXBUsZO6uR9zN1adCGY7voG+qkJui/d1YPL47CTaArF7Gu2x1yvtlZxH3pn1hr/HgLZf8lEbb4Z+5gORhyIDs/CroV+3t8uFong5LxujUJQ5dTFiILQhCbycj2A6E8asu3zgYX0OCA7j27/Rf0plVWxo4Kg/x9uWCNnuVhfje+Go02dr2Be+fZQQZM7D/yLrd7Z5gmKhGQnaMdH7EmteYFIWlttVp+suuyIqYEo43A2alv60X20whDCsEdl/gIsryuHTEEPa4S08lFz+2bple43k9iQL0047amP8g+gCESe7y+3eBHyqLDlETqToInHtqzkuK0TAht2PToCxmNtceh0g0wN1uKFitaxnyZ6bUAqr0MSgLHSJ1FTN3mJDW3RDStZjBH2jBe4R4xWwBpO3UOo4oHUY9dzkf0kn1Tze55B6u2z/HCJo+gDgH2MMp8zk89Cw57iYlNxnC8Y6FNHOS0XuSozudEi1AXGEoCCWZNXSuROKQArb4OERQA8wZTeczmUoPkgUjKmcafLN0JnVYBK54kWPHXTiGBgxZRYDhK1R8pmgPrn3bJPX0qeIbsqFdm71AveSEuDs/6Qag+UWdGK5PUFiKB5eZUo8lKcwY4q7kSq3wdfGMiar7NFwYSmJ0A== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(376002)(346002)(39860400002)(396003)(366004)(66476007)(66946007)(36756003)(86362001)(6486002)(16526019)(6666004)(15650500001)(26005)(2616005)(83380400001)(107886003)(316002)(52116002)(7696005)(956004)(7416002)(2906002)(1076003)(66556008)(8676002)(186003)(4326008)(44832011)(54906003)(478600001)(5660300002)(38100700001)(8936002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 4d/ZBaMLhga4T6+UOaZ1RJBf+5vWs3dfPVe8T0SAkuotfkt1LL6/xJlGHXj7I589heTKaax+D6gY7O51nvt7rWmTQoDCMelmcVkC0rXXjul5Srb8gDdz+aMXQmhTTADRp5bw79mOXn8Fv5EN3BZSMmuk1kSfpuhvmPycpgRvKKLyseBF/MJMIbexDyjjKf58L5a7nQ82k+VJIX+WuBwTSCcPkGjylP+EV5Ta4Xnzc72h8FQcsMFYPca15X9hBxjjGL2n6XMHB/eqKxhtQX9c6zDvsRZmljG3d+OWNt8A4Hcj6NEKjVvA9+cBQ5RCHxxjNubaRXl+1BCNaRwELlcxcSiXqBDndpux12D1n460hHiIOHwVAmSqpX5NWd7K0mVe1D/FpzCBcMpLcjW+jjpS4WmCodONYS/X6vlMNSowajKlB8Im7IrsVG1OswZrl16Dqh8SdiyT6RRfXuL/Eb2h/bHRHL/O3cUddV0zvvavMGZpbtWvB2pbrrYNWCMRDD3TFW+8gS0l2GcLXHDd1yjSbgL3IB1y0RrPwL28V7/mu6vLbWr1IWaPCB9/nHJRBr2KLTNzbqfITKasDvDGDDAjGpZXKG9WYIvoY8vrAQ8xq2wRRG4lOIDZTrK1z0i+/xvZpf5K3rckp8KymzZsavLhDYAhkWMHgVtLDJ7bevBAcxQnHwQHKFcqBzL381kuyWnXTBgnrPV2jNlD4XoBsyvtHqqJGL8R918r+TXtpofbNwesy5/0airk51G19mIvwzw9d9mF1HT91vWxZ9IjP1/lTtWWLNWRXQDn4gTMwzAiDXIeAG4cu8b88eoR7lqoX0pAgtMJs2klCkHskZa0eWj4Qdjao+u65zNTFHlGSY5DXZqUxFWXVKvgiEMVDCO+NRTz6cGWZwiWnSrogIxw1Xju1OND3x12Jh3g7TEqy5SxnG8Chdax6Sc+rMjp2xEcy+Sp0or5yv8zv3sM2RgLGPH1TFTDq3WjThEyvr/CqimXqQCrjr6EJqZLPiJJGRoqEVVIOsYihzfnWm9Hw0pgs8sAe44TWnzsYgxh2htLkEFBBC1e2pm0EZZNTh4nQZunD3UpoFlBst2H4GnZdGklFnwZ4yftnyBi6hI/m6rcIkcgKz+xjmFIq5fIIRSW50maZWQy568l+xRkJ1tfsNBUVb7pWl+YeB+1gJW1Bm21AIXEAJ1/3POEiwSbzrlB6bUyqsmG1XpcjWolo48Pz58kKUXsffGGGShzh5dHS9yIaMMwUT6+AoTldzirmwu97FZWEa+dfADgaoUvoS0DCsXNkdVSuNdUPJN7dsehuT+CRseU9wgEHC0rctDUkW+MCq16Nv4d X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 86e9ce0d-8429-42f3-a226-08d8eb284cd8 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2021 22:42:39.5799 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Ma/QjEKl3iOtPvV3Uvig9zz7YBvanUi/SzN8WfsoJ1AGc3cqT/TOHRIAQqyYUvnOIsvJyom9iz/HhmlOsZwJDg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB4211 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9928 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 spamscore=0 bulkscore=0 mlxlogscore=999 mlxscore=0 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103190156 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9928 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 malwarescore=0 bulkscore=0 impostorscore=0 lowpriorityscore=0 mlxlogscore=999 spamscore=0 priorityscore=1501 adultscore=0 phishscore=0 suspectscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103190155 X-Stat-Signature: eqgog9zbzioisozfhzi6wsx97aduy8uh X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: E8836E0011F4 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf05; identity=mailfrom; envelope-from=""; helo=userp2130.oracle.com; client-ip=156.151.31.86 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616193795-356176 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The locks acquired in free_huge_page are irq safe. However, in certain circumstances the routine update_and_free_page could sleep. Since free_huge_page can be called from any context, it can not sleep. Use a waitqueue to defer freeing of pages if the operation may sleep. A new routine update_and_free_page_no_sleep provides this functionality and is only called from free_huge_page. Note that any 'pages' sent to the workqueue for deferred freeing have already been removed from the hugetlb subsystem. What is actually deferred is returning those base pages to the low level allocator. Signed-off-by: Mike Kravetz --- include/linux/hugetlb.h | 12 +++++- mm/hugetlb.c | 86 +++++++++++++++++++++++++++++++++++++++-- 2 files changed, 94 insertions(+), 4 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index f42d44050548..a81ca39c06be 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -666,9 +666,14 @@ static inline unsigned huge_page_shift(struct hstate *h) return h->order + PAGE_SHIFT; } +static inline bool order_is_gigantic(unsigned int order) +{ + return order >= MAX_ORDER; +} + static inline bool hstate_is_gigantic(struct hstate *h) { - return huge_page_order(h) >= MAX_ORDER; + return order_is_gigantic(huge_page_order(h)); } static inline unsigned int pages_per_huge_page(struct hstate *h) @@ -942,6 +947,11 @@ static inline unsigned int huge_page_shift(struct hstate *h) return PAGE_SHIFT; } +static inline bool order_is_gigantic(unsigned int order) +{ + return false; +} + static inline bool hstate_is_gigantic(struct hstate *h) { return false; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 82614bbe7bb9..b8304b290a73 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1351,7 +1351,60 @@ static void remove_hugetlb_page(struct hstate *h, struct page *page, h->nr_huge_pages_node[nid]--; } -static void update_and_free_page(struct hstate *h, struct page *page) +/* + * free_huge_page() can be called from any context. However, the freeing + * of a hugetlb page can potentially sleep. If freeing will sleep, defer + * the actual freeing to a workqueue to prevent sleeping in contexts where + * sleeping is not allowed. + * + * Use the page->mapping pointer as a llist_node structure for the lockless + * linked list of pages to be freeed. free_hpage_workfn() locklessly + * retrieves the linked list of pages to be freed and frees them one-by-one. + * + * The page passed to __free_huge_page is technically not a hugetlb page, so + * we can not use interfaces such as page_hstate(). + */ +static void __free_huge_page(struct page *page) +{ + unsigned int order = compound_order(page); + + if (order_is_gigantic(order)) { + destroy_compound_gigantic_page(page, order); + free_gigantic_page(page, order); + } else { + __free_pages(page, order); + } +} + +static LLIST_HEAD(hpage_freelist); + +static void free_hpage_workfn(struct work_struct *work) +{ + struct llist_node *node; + struct page *page; + + node = llist_del_all(&hpage_freelist); + + while (node) { + page = container_of((struct address_space **)node, + struct page, mapping); + node = node->next; + __free_huge_page(page); + } +} +static DECLARE_WORK(free_hpage_work, free_hpage_workfn); + +static bool free_page_may_sleep(struct hstate *h, struct page *page) +{ + /* freeing gigantic pages in CMA may sleep */ + if (hstate_is_gigantic(h)) + return true; + + return false; +} + +static void __update_and_free_page(struct hstate *h, struct page *page, + bool can_sleep) { int i; struct page *subpage = page; @@ -1366,6 +1419,21 @@ static void update_and_free_page(struct hstate *h, struct page *page) 1 << PG_active | 1 << PG_private | 1 << PG_writeback); } + + if (!can_sleep && free_page_may_sleep(h, page)) { + /* + * Send page freeing to workqueue + * + * Only call schedule_work() if hpage_freelist is previously + * empty. Otherwise, schedule_work() had been called but the + * workfn hasn't retrieved the list yet. + */ + if (llist_add((struct llist_node *)&page->mapping, + &hpage_freelist)) + schedule_work(&free_hpage_work); + return; + } + if (hstate_is_gigantic(h)) { destroy_compound_gigantic_page(page, huge_page_order(h)); free_gigantic_page(page, huge_page_order(h)); @@ -1374,6 +1442,18 @@ static void update_and_free_page(struct hstate *h, struct page *page) } } +static void update_and_free_page_no_sleep(struct hstate *h, struct page *page) +{ + /* can not sleep */ + return __update_and_free_page(h, page, false); +} + +static void update_and_free_page(struct hstate *h, struct page *page) +{ + /* can sleep */ + return __update_and_free_page(h, page, true); +} + struct hstate *size_to_hstate(unsigned long size) { struct hstate *h; @@ -1436,12 +1516,12 @@ void free_huge_page(struct page *page) if (HPageTemporary(page)) { remove_hugetlb_page(h, page, false); spin_unlock_irqrestore(&hugetlb_lock, flags); - update_and_free_page(h, page); + update_and_free_page_no_sleep(h, page); } else if (h->surplus_huge_pages_node[nid]) { /* remove the page from active list */ remove_hugetlb_page(h, page, true); spin_unlock_irqrestore(&hugetlb_lock, flags); - update_and_free_page(h, page); + update_and_free_page_no_sleep(h, page); } else { arch_clear_hugepage_flags(page); enqueue_huge_page(h, page); From patchwork Fri Mar 19 22:42:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12152063 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91E73C433DB for ; Fri, 19 Mar 2021 22:43:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EE0E76197F for ; Fri, 19 Mar 2021 22:43:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EE0E76197F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7A2296B0071; Fri, 19 Mar 2021 18:43:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 752146B0072; Fri, 19 Mar 2021 18:43:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 556776B0073; Fri, 19 Mar 2021 18:43:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3B5A56B0071 for ; Fri, 19 Mar 2021 18:43:06 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id EF314AF91 for ; Fri, 19 Mar 2021 22:43:05 +0000 (UTC) X-FDA: 77938100730.11.AECE164 Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) by imf23.hostedemail.com (Postfix) with ESMTP id 13AEFA000282 for ; Fri, 19 Mar 2021 22:43:04 +0000 (UTC) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12JMdmun139912; Fri, 19 Mar 2021 22:42:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=mLbWxPk+DV9MUgdTJ++7u5WZineUPSEO95Aqx3RVqk8=; b=XmejMrCbbNFncO7VBrul0eoRREDiTEWiDwFyXgla3a1mTWALuyXINiZZv8fw+gqFVILN LPli9MNck3FRHIk6E6wq8WhtbaYSCPROmAi24nJC2u35pJcCjA9CML4TvBz9NZiKIn0E k+g+VONl7/rS52JA5Zdwf1O3sI4C+ccCkuKUUaKoqjedX3tTU5vx6mofyCkZx7e2Oncv +vBuwk8WxrIjxToLbzwjTP/S3uUpxJuS5Rd+SveRN1kgroKCylllrU5IHLBDH3ts5KLn v37rB+6RrD0Xy5o0qViPbcuUD0YcsMYgaAlFXDnpxxbfBb6uHZx1flC/VhckkSTbhFtY CA== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by userp2120.oracle.com with ESMTP id 378p1p4dpm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 19 Mar 2021 22:42:44 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12JMer30001735; Fri, 19 Mar 2021 22:42:43 GMT Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2104.outbound.protection.outlook.com [104.47.70.104]) by aserp3030.oracle.com with ESMTP id 3796yxyng7-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 19 Mar 2021 22:42:43 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jVdV8/UabqagBSYUVvdHi8rOlJDktJ7nrKNychstyXAd3cOZUCEEU197NidmSF6Z6TfUig4m1m6z4g2f92dvo7cYWEGsY5fC3xhpQWp8/FewHhYXqTZx4jqdSWatcF1BiwN0fKqZhmZArUDZ8LuCNfaRD0C5ajWwuUwryOZIlrj7sMPtcxRs8P24gJb2sFSBakAgYDOLSkB27MKLxOv+kH6b1JEfYBnWsSNLNH7fb56M3t6VjfIRRzyuSCJoDQq+ZqqPs0xl5m3B9BgYBYufqHoY0nyCMX1uISeHtPy//i7Mo8bFhtwo6Kpc5TIUw5bLFL2swEWYSridz8pW3gIF8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mLbWxPk+DV9MUgdTJ++7u5WZineUPSEO95Aqx3RVqk8=; b=WRR9KESANNR971elv7wOUkKibzNoSoD/6DY1RwavDzAIcddRZU8CCUdZdeT2J4f1eGq9wDvbWw1nsrma75QEV8x9i7ICQHi0ylcwKVY/xsUge26hVu+UIuxSmgRSD369Opp+R7h8+Qn6BFmigfwtwctDYPIOznyE1OHLaoTr98ddknfsmVZAVgTwZ2/QKO8IQ5Plv4YgQMu+dfD0PVtkdvsI6cyIe+tMeoygvV85hnBJIf/plYaeDp2wf1ByQtugrUuI4VbCncwnBjJa42a1vHqy7IHdDz60/KeAg2DwfWcajjXGrrk5X5o8wfyuJuwtHKcPyYPFN3GgQWIP1ZRV/w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mLbWxPk+DV9MUgdTJ++7u5WZineUPSEO95Aqx3RVqk8=; b=si6E6KxU2UiqGFd8zumeYx36WwHrLAYwSsaV0iz0fuDDmDR+dKcd6wdL8J66OvV1dw538xGudHtYCiVmJ04ZdlW6iZ1wPUOiI82TBfq3Zccyjl3hbR4QQQyB4ZF30j1YdEYKtZsgnnGx29yOWXC49nZokX5SyrhqAi9zLqyKaIE= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BY5PR10MB4211.namprd10.prod.outlook.com (2603:10b6:a03:20c::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3933.31; Fri, 19 Mar 2021 22:42:41 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::980e:61ba:57d2:47ee%6]) with mapi id 15.20.3955.024; Fri, 19 Mar 2021 22:42:41 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Michal Hocko , Shakeel Butt , Oscar Salvador , David Hildenbrand , Muchun Song , David Rientjes , Miaohe Lin , Peter Zijlstra , Matthew Wilcox , HORIGUCHI NAOYA , "Aneesh Kumar K . V" , Waiman Long , Peter Xu , Mina Almasry , Andrew Morton , Mike Kravetz Subject: [RFC PATCH 8/8] hugetlb: track hugetlb pages allocated via cma_alloc Date: Fri, 19 Mar 2021 15:42:09 -0700 Message-Id: <20210319224209.150047-9-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20210319224209.150047-1-mike.kravetz@oracle.com> References: <20210319224209.150047-1-mike.kravetz@oracle.com> X-Originating-IP: [50.38.35.18] X-ClientProxiedBy: MWHPR07CA0013.namprd07.prod.outlook.com (2603:10b6:300:116::23) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR07CA0013.namprd07.prod.outlook.com (2603:10b6:300:116::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18 via Frontend Transport; Fri, 19 Mar 2021 22:42:40 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 96c9ce35-23cc-4096-a371-08d8eb284e01 X-MS-TrafficTypeDiagnostic: BY5PR10MB4211: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: D+Kt/obVxyrx0x7kBf9c0k8AtpX1TC+76rVOGw5NdSbFPLwL83Qkh08c+mYK46GYbW/sy1mexShCLGi5tdzO29UiuCXAS9U3ClJQxYiMNwhq0beAj27xyNFp9d1U9DLHYXuul6we3bBNG91/JwDWwWL58i/lEC+yJOxLN1G2WfTI6/5twgm3vKFrmcCikZR7qjJwHEAshFFslkM8F/1+UhyWkfyJidWZO6XzN5uPZu7zUglPR2V/GmJnWHdxjm5B/41ToXXWQxc9e7ADFfnaU4Z9jdivwNoxXPh7yHDnKRp1iQMiV9PGNdNYqr0oGJAPVfGDr7X+1S0j4O3s12e9VE7CvAVU1mPmLQvh2pQ3GsxeULcpkrGFlklhq2k49OT5IkkqiOEyygjLoK0CGj8trZNzyU2pmiBiUROJ7CRgGdUqSKUxGKGz6AQctFJ9s7qOGpuElbIrcyA0RhD+uYz+0UKhSR4Dup2dUm9PUjlNk4Yd8b4sMTEkkO+sBpYLifm7aW1Z1SZo5RgZj4vuBvgTNoO3GUyHJxru7F2feiCnzWR2QJY3VgUNcyKOyqyqfGjL0XzptkaBl423YITT+4FbZU9QQ82Kh/Cn2JtRrWf2k4H9lE6BPx/U6mExXnrG8PNo3FZynlM5Glb8n5U7bovWMA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(376002)(346002)(39860400002)(396003)(366004)(66476007)(66946007)(36756003)(86362001)(6486002)(16526019)(6666004)(26005)(2616005)(83380400001)(107886003)(316002)(52116002)(7696005)(956004)(7416002)(2906002)(1076003)(66556008)(8676002)(186003)(4326008)(44832011)(54906003)(478600001)(5660300002)(38100700001)(8936002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: EDPuktvNO6Z/QQrqlDcNkT7n+Huqa1rqLbDm9wgNID0KREFH3ZGMraitU8aMng7V4n7fuHjhSymhSsDzufJxLdXt71pUTOvVbpYfIba8/KjKlHwAi2lHRCS20ktwXZnGsN/1UquEAAqmvnhzgKWJbgbjXYMqJyhzIMeY5GUR1fna8LkJVzdd1IeJ3lq3hLyZl96DTvXAF1urJVITfPws1pKk85bGdmC2nEs7jeSqWh7G8keDb0vYNaLGaeGKVYwy35UVL1ODFFYMXYQNBiT2/Z0ZEsdyXRceJ7+1M8jPG0BKPhyyadRFXjkBTYYrBZ5Uz0hZN/piqcMhdlLE04eLas7swvFYw/3e9jAbUrqpsOv87VXuIXo7C5mrqeE5c1d/sRpz8dPlAsI4PDVWHapkU24iOxFIyzuvY51839kcRnKl1NR6dpj2UX0M9eGoTLtxl2RUYU95+KDwBnf/ZuS54G8y0qj4/3u3eEMuTGkIw8ikxIuWfofWtCRk8vmEKd78Y1/APdULd7iI4CQcXpuLXA0ulwcipeY7hBpxq3p2rH8nuZ8YWpmag6VggoEU45NkKgUy9bTiEYhJhjUWJLyt8/KHSe3LShHKb5HpTiCSszlq2c0EthQlMAZ+y0szHtejtgQPh/vM3GZ5wSFiKqSsiGafVIUHOLFEhWMwLukMgJYGbbeRLfnDTIUwG08NvBEkcwb3enOmo9b1n9/hjpCVYWRkKMBWxOqJ0gZSk4nTziajY1I6CxswT30Y0F8OJkNRwdYRM/eTlzPi8OuqLQuw13OgFvNFwdkJWOwxjcry7qSCPcdObyPcdkxtSa7qTeyhovNWr3Cs//+ozde2cbU1d0JM3I9ewDnkTfQsCu6zMWCytIbPrjmZqjGj/LMLIDJ2Wq21FGPd3upAZuhdPdhUIK3OoqPtjTrcctnu5MeDCeWXgH8tb/9Tv232WphdsmlbE9u5olMHSydWusL0plKjd/PiXNepkx1i11QWv+M7qFRl7eEM07WtdpQ5gY6YYrvx2iPZlqiEe5o/5TnFtqA48AElcrIkWU/OXS6pl3INmKEnW72fxw8w8pKnCIjJHx6SzsSj26m5GBSJJFyzuZP4X3oqluGAwYUlH5Xx5+vgbxi/USyAr5zf8V1hFetQ3TmcyI9ympy1ThyYuvhv4lZR1Yu+qUShtx3N4NB6u7S1x6+KgJanMQ9/9ssCh/j1+Kxdw1yRys9ntsHw9kET7IGdd3Gc3Va4NdwQtccZH/i3m6r8rJdRkhLR6dIL/4V2ShRYMl6baqfHGMPrVdBpzjE+h/hcjfy9to1q5pKF/4CMFOHXakalRWUtPRcvs6GsB/pL X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 96c9ce35-23cc-4096-a371-08d8eb284e01 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2021 22:42:41.4487 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 4iTcXTyoM9FocI3pZW4qXKfvikj+mDPfPDmvFeURcfLjHL4pPD9/xIX3g8uEFqP+ar1CWugtVogw14H8kOj9EA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB4211 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9928 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 spamscore=0 bulkscore=0 mlxlogscore=999 mlxscore=0 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103190156 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9928 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 suspectscore=0 adultscore=0 spamscore=0 clxscore=1015 phishscore=0 malwarescore=0 priorityscore=1501 bulkscore=0 mlxlogscore=999 lowpriorityscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103190155 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 13AEFA000282 X-Stat-Signature: i9hkmbb9tjf3d6j7ijtn7585w3hz7ps3 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf23; identity=mailfrom; envelope-from=""; helo=userp2120.oracle.com; client-ip=156.151.31.85 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616193784-68701 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a hugetlb page allocated via cma_alloc is freed via cma_release the cma_release call may sleep. For now, only gigantic pages can be allocated via cma_alloc. The routine free_huge_page can not sleep, so it defers freeing of all gigantic pages to a workqueue. Use a new hugetlb page specific flag HPageCma to indicate the page was allocated via cma_alloc. This flag can be used so that only gigantic pages allocated via cma_alloc will have deferred freeing. Signed-off-by: Mike Kravetz --- include/linux/hugetlb.h | 7 +++++++ mm/hugetlb.c | 18 ++++++++++-------- 2 files changed, 17 insertions(+), 8 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index a81ca39c06be..0aba6957a73a 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -498,12 +498,18 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr, * modifications require hugetlb_lock. * HPG_freed - Set when page is on the free lists. * Synchronization: hugetlb_lock held for examination and modification. + * HPG_cma - Set if huge page was directly allocated from CMA area via + * cma_alloc. Initially set for gigantic page cma allocations, but can + * be set in non-gigantic pages if gigantic pages are demoted. + * Synchronization: Only accessed or modified when there is only one + * reference to the page at allocation or free time. */ enum hugetlb_page_flags { HPG_restore_reserve = 0, HPG_migratable, HPG_temporary, HPG_freed, + HPG_cma, __NR_HPAGEFLAGS, }; @@ -549,6 +555,7 @@ HPAGEFLAG(RestoreReserve, restore_reserve) HPAGEFLAG(Migratable, migratable) HPAGEFLAG(Temporary, temporary) HPAGEFLAG(Freed, freed) +HPAGEFLAG(Cma, cma) #ifdef CONFIG_HUGETLB_PAGE diff --git a/mm/hugetlb.c b/mm/hugetlb.c index b8304b290a73..5efff5ce337f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1241,13 +1241,11 @@ static void destroy_compound_gigantic_page(struct page *page, static void free_gigantic_page(struct page *page, unsigned int order) { - /* - * If the page isn't allocated using the cma allocator, - * cma_release() returns false. - */ #ifdef CONFIG_CMA - if (cma_release(hugetlb_cma[page_to_nid(page)], page, 1 << order)) + if (HPageCma(page)) { + cma_release(hugetlb_cma[page_to_nid(page)], page, 1 << order); return; + } #endif free_contig_range(page_to_pfn(page), 1 << order); @@ -1269,8 +1267,10 @@ static struct page *alloc_gigantic_page(struct hstate *h, gfp_t gfp_mask, if (hugetlb_cma[nid]) { page = cma_alloc(hugetlb_cma[nid], nr_pages, huge_page_order(h), true); - if (page) + if (page) { + SetHPageCma(page); return page; + } } if (!(gfp_mask & __GFP_THISNODE)) { @@ -1280,8 +1280,10 @@ static struct page *alloc_gigantic_page(struct hstate *h, gfp_t gfp_mask, page = cma_alloc(hugetlb_cma[node], nr_pages, huge_page_order(h), true); - if (page) + if (page) { + SetHPageCma(page); return page; + } } } } @@ -1397,7 +1399,7 @@ static DECLARE_WORK(free_hpage_work, free_hpage_workfn); static bool free_page_may_sleep(struct hstate *h, struct page *page) { /* freeing gigantic pages in CMA may sleep */ - if (hstate_is_gigantic(h)) + if (HPageCma(page)) return true; return false;