From patchwork Thu Oct 7 18:19:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12542597 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5500DC433FE for ; Thu, 7 Oct 2021 18:19:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EA74261263 for ; Thu, 7 Oct 2021 18:19:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org EA74261263 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 838D7900003; Thu, 7 Oct 2021 14:19:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7B3646B0078; Thu, 7 Oct 2021 14:19:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 606046B0075; Thu, 7 Oct 2021 14:19:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0062.hostedemail.com [216.40.44.62]) by kanga.kvack.org (Postfix) with ESMTP id 528F86B0072 for ; Thu, 7 Oct 2021 14:19:43 -0400 (EDT) Received: from smtpin32.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 001C52702B for ; Thu, 7 Oct 2021 18:19:42 +0000 (UTC) X-FDA: 78670454646.32.7539975 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf29.hostedemail.com (Postfix) with ESMTP id 83783900259F for ; Thu, 7 Oct 2021 18:19:42 +0000 (UTC) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 197I5bc8025160; Thu, 7 Oct 2021 18:19:40 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=ztRI9qgc0t3v+U02wSqBcenJLvI0kf9QUfmuM/9rRZg=; b=cz9TvFrZKFxVCKNXnlxPtP6sEXubOpyWpBC3WJBW6R5jfGT6/kr2Pn5559gsu5oMeCjK yU+blKTr9+DYeF0WqciU+YrfQHkueQYJdIywRvwe9R5pdgDCkWAybnGyzqVbSUzRthwp z+LJBln4Jjj33H4Qr0y5eZ3cN3e2vnRfHN8lspKd8dyKn8dufrr+wrkD3TXB9oHGuKTs T/vK/d6O5+m8qJcU7wMXUP11/KoOfCEKtQdmamuUuOr1V9mTTdWEOuq3DDW/JxqxrQnR aGVmLJvuWWC9vb+djf1/dBH8T0y9iio2jGUUxrpTI9AvWXAauqgaZz9iYTpR+BS71loX dw== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 3bj0pw2vfa-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Oct 2021 18:19:39 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.1.2/8.16.1.2) with SMTP id 197IA8BP047007; Thu, 7 Oct 2021 18:19:39 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2042.outbound.protection.outlook.com [104.47.66.42]) by aserp3030.oracle.com with ESMTP id 3bev7ww3ea-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Oct 2021 18:19:38 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=On3VyucICEzczI4ZX+Tr85bWNvqFceqkdwHYqdoNfuM8g+HLJgIUNq/4SCF1PG/FuA0yUvEAsctz8DrGm+/bVaQkH6QGQg7hXM/6IriCyfaORnSg0iw0i5Dm+zXAjq33BCzASXSGhLmO9/k2W0oTXu+9n3CYkZ3nRaleUpJ58IrK94RShHSodeUaLhKsCiekhDafP84p9IfiQv/d3HPIeqfEU3oEWXYYNHRlsgcXWwCpmTAt0nFWkNjWxO9jmzh4hLNkTwGqlqodG+lbHRCHDTZxsRFs/xP3jjjO+IBF/FJXJb4F6O3Hu0UI/h9EVo5ECaTKmsYtkYWEp81dh/bMrw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ztRI9qgc0t3v+U02wSqBcenJLvI0kf9QUfmuM/9rRZg=; b=dnbmizEcDEX9cij/r3IDe7y9vq8/+59/Ll9CvggU2ErVOdLrbu2DnT+2tuM4cEDDivuzBHK+BRrWEWMRcwmNmNvaccdirg08AjHj2xOdE7/fFKyoWibkunvgDpga0NH/jHZiX6lzT6Fbo8nX8jwyZVggZ0/whX+Oh1EjaKJ37tddgGI0p7xmVCvD186s5qCbbC1yExgzNi2ao06gE7UnqtINLSCpo7a3FEirrhbgRF+qTjej7VCJ+iv90Be1V4NGRwRuTwyGOX15tZLmrXckIy6Q8qF25UHxvuKHtlPg4gKD8AX079bamTrd5KgZrC46xDneNYcYZ3ycSI97hOnIcA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ztRI9qgc0t3v+U02wSqBcenJLvI0kf9QUfmuM/9rRZg=; b=oqADCcqSsTP7SJPFRJs0TOq8KYP42fOrrgLcZKEiBg3TD4npy7kvqV1caHZIl+70C51MEZ9wfJj6jeIzF4VBBzLeoWmjWrbYNXnFgb10QHOSUR9kOV8OwBo+W/ZjvSkNbVcO4W+O1GgZFjwH1a8pdwTjx8JCMU2Oe5j3CzKut1A= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB2552.namprd10.prod.outlook.com (2603:10b6:a02:ad::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.18; Thu, 7 Oct 2021 18:19:36 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::bc59:71de:1590:cbf5]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::bc59:71de:1590:cbf5%7]) with mapi id 15.20.4587.020; Thu, 7 Oct 2021 18:19:36 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Michal Hocko , Oscar Salvador , Zi Yan , Muchun Song , Naoya Horiguchi , David Rientjes , "Aneesh Kumar K . V" , Nghia Le , Andrew Morton , Mike Kravetz Subject: [PATCH v4 1/5] hugetlb: add demote hugetlb page sysfs interfaces Date: Thu, 7 Oct 2021 11:19:14 -0700 Message-Id: <20211007181918.136982-2-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211007181918.136982-1-mike.kravetz@oracle.com> References: <20211007181918.136982-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MWHPR1701CA0012.namprd17.prod.outlook.com (2603:10b6:301:14::22) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 Received: from monkey.oracle.com (50.38.35.18) by MWHPR1701CA0012.namprd17.prod.outlook.com (2603:10b6:301:14::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.19 via Frontend Transport; Thu, 7 Oct 2021 18:19:35 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f9e528f3-5d95-40cb-191a-08d989bf04bb X-MS-TrafficTypeDiagnostic: BYAPR10MB2552: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: JY9zgc9p5ZQ4MGU+H7SWhlPpI0FPHRgIENLz1dS7k3ufW6kMXfdfm09JNVMn8i1suffa/6qZT2B3DY2ZHHHy7mDdYmK54PPmP4PzwgRGtlrEF/pkYqjKyOLVtZG4bWxcP0Y6Ddbo2bjkzKFNLrxECdJwXIfj1jtA2Am1J2FeGRSey3x59vJCLOdaGrb7s6lmp2eJ2WeuYcIjZUMYRaUsvW4Ys1TVVwAWHU5vs9dKiinTJSdaYSVTPs1CAyS8lWv5Ka6ias4Ycde8hDXf9rxhdv51MFjOG9Lmc6sv8buOD2haPZV3HfRNZmJ2Zphz03WWhtcIyv/tbhCTv1uM3Ib/E8mNHYaTzb+AlTn9yKQ44nX5W4daiguJlOxxzSiqQOqyldMvCG7TlmdKhxYpfkkkie+yYjHWp8t4mgUsBU5pekPvhm3MPjRYQ7tG0XyhxUvhSv5j+nGTfJvC3X40kaxMYJ6cQS0uhEyacPhjtZNT3/WX2ipGw3M9bLR+ppJFP+7Ie8+xGuaAmjBuU0z+8u5lzTesXSjuLNkzyq+vtitJvXzSQL+9nqg44ZNG6jwb/wUiOFGR4SNsxjmup4J0+Ql0AM4jfWiEg06ng7TDRGOWuw/udazRwFHNZJZOn7J8hDTHXW29Ku4mlipFg7rNcX8HNvdOXzUmOJ671HGHp+cm2bdTBRf3bVkjDIaCoajog7saYjdPT6QoMSE4By19wC7iUw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(86362001)(38350700002)(316002)(7416002)(44832011)(5660300002)(6486002)(107886003)(2906002)(1076003)(186003)(508600001)(54906003)(38100700002)(6666004)(66476007)(36756003)(66946007)(4326008)(66556008)(52116002)(7696005)(956004)(2616005)(26005)(8676002)(83380400001)(8936002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: bYSrn1gd/h+jzt4EN4md/QoAF2ZYI6WwcTsb+ZWSPe3UqYQaQpShaR3hfLqhUZZrw1FqL91mjwauCH1vIbLKBsbo2KqTzBc1j6/qKiHb4B/i5oZTEBpTDxTU9+KBFnOGLheA/CXezAMBgkXa0poHRJjnq1IzRT0ruq9Sit69jc60LGASs2qQz0t9UsRNiuzYcuZZQ/LeKSJFlB5mEVOU+9iVR9NzvbAxTv876CHNpF1fw/08oDZP3AE/lBwZIesE+6EbLHT1dS9bN+H3b8+9uXTOFwfcGbMx8zqNhSZ44lT+H4u7VFQlUWUYiRXSup2edZ9Kvrm/6dNO2wRGpB6Zpi6J1kVZX0qYjinC8GWyDEEkRlC6OdSLBJE06rXpuygz5unl6hs55lPHZWkqilikOmZyuSjR14YkgFf3lG67FU2H0kXk0K3hXoZsBqCDd8lY/F7LtdYh2iNz/r+vaBPYi0y84tGsNDFgAWlPmRaGLJe2oOrv62Hb9HBRoh16mFv3EKQc0rToVCcISWPma3QBh4WRECrFonduqRIw5jxNIIQ3NSpHWiw4okzZOrkXjMjlZIe5Nhnp3VcaVwOMPqeeN0nsScMANOGx6Lix3Ut7YaTqJ7ckqf9vmXdJ63otzSYTRF6lVcegyFW6cPRrWjZQsmtsbtZHKLi1WcQbaheF31YsPadFnQBIOkbbwjYpK45hglAXeZuhEiWGpody57fC7I4XdCacYTflKGs4NRDsP+dXq/QGjJPtvUj00szHARPE3XLA0R7/aiLObHvpvrhm/ehcMcIelYRNwm37zlFgv4G0OamAlDVf86sKELBQEoq5h9pEkka59nWKazAGNKpvLdFfw4D3YJeFMTu1xq+ec/RsWKXIvRy9VnjHOk4xMh7zk3lo/pD2bEK6UBYdWPPXAvT3XeNHgXuTElKLCJw1Sd/CbTG7t3hMWY1gni+I5cCucCQtJ0jphGPEFZxyFJ8WYx97ZvSZo49Cor85uaTbtOe8gbDWPMbog2Jg2BCSBLP2JxgdCpvSpJNqG9/yiJ/3KO0+zyVZD/si6ozMb5cDVjVOK79rWWRLvXVVdAyyb/ACbwhCotPqGQe592Ek7Xs0buA1buRg7mzX+2HnIUQxB10lRRDnfMt2MmNdiiBv1kIRkWB467+zOEC/vAAVYoaTOsBrmFo0DFnRsM0lr+0Hex+ZB7ii2woBw1/iQdx5BjaXxjhi6sBjWIpnJjZmyJJzP065YSHQ9bdpewPepRy4mDIScem3Wk07fNn3iDxPjok59c0aOikCx/nKvxTE/Ht2R3H7eKIOaI/sAoDssdGIsFtMk7fHQQ/iwXABPh/j9zKx X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: f9e528f3-5d95-40cb-191a-08d989bf04bb X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Oct 2021 18:19:36.1770 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: jPzxH/JerXjmRxFkQkeDtfnoQfYQkOOOd+G23IZzvrr5rRVEZ/HsYC97B/Vwv5cPne9AqMnkiTTG4pXa+J/JVw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB2552 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10130 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 mlxlogscore=999 adultscore=0 bulkscore=0 suspectscore=0 malwarescore=0 spamscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2109230001 definitions=main-2110070118 X-Proofpoint-ORIG-GUID: SNwzrWpT0vRQtK854lBRoxbbgw5hLTEx X-Proofpoint-GUID: SNwzrWpT0vRQtK854lBRoxbbgw5hLTEx Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=cz9TvFrZ; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=oqADCcqS; spf=none (imf29.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 83783900259F X-Stat-Signature: ag4icn5j1fdz16tueu46nf1owjb3ffdb X-HE-Tag: 1633630782-530725 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Two new sysfs files are added to demote hugtlb pages. These files are both per-hugetlb page size and per node. Files are: demote_size - The size in Kb that pages are demoted to. (read-write) demote - The number of huge pages to demote. (write-only) By default, demote_size is the next smallest huge page size. Valid huge page sizes less than huge page size may be written to this file. When huge pages are demoted, they are demoted to this size. Writing a value to demote will result in an attempt to demote that number of hugetlb pages to an appropriate number of demote_size pages. NOTE: Demote interfaces are only provided for huge page sizes if there is a smaller target demote huge page size. For example, on x86 1GB huge pages will have demote interfaces. 2MB huge pages will not have demote interfaces. This patch does not provide full demote functionality. It only provides the sysfs interfaces. It also provides documentation for the new interfaces. Signed-off-by: Mike Kravetz Reviewed-by: Oscar Salvador Signed-off-by: Mike Kravetz Reviewed-by: Oscar Salvador --- Documentation/admin-guide/mm/hugetlbpage.rst | 30 +++- include/linux/hugetlb.h | 1 + mm/hugetlb.c | 155 ++++++++++++++++++- 3 files changed, 183 insertions(+), 3 deletions(-) diff --git a/Documentation/admin-guide/mm/hugetlbpage.rst b/Documentation/admin-guide/mm/hugetlbpage.rst index 8abaeb144e44..bb90de3885d1 100644 --- a/Documentation/admin-guide/mm/hugetlbpage.rst +++ b/Documentation/admin-guide/mm/hugetlbpage.rst @@ -234,8 +234,12 @@ will exist, of the form:: hugepages-${size}kB -Inside each of these directories, the same set of files will exist:: +Inside each of these directories, the set of files contained in ``/proc`` +will exist. In addition, two additional interfaces for demoting huge +pages may exist:: + demote + demote_size nr_hugepages nr_hugepages_mempolicy nr_overcommit_hugepages @@ -243,7 +247,29 @@ Inside each of these directories, the same set of files will exist:: resv_hugepages surplus_hugepages -which function as described above for the default huge page-sized case. +The demote interfaces provide the ability to split a huge page into +smaller huge pages. For example, the x86 architecture supports both +1GB and 2MB huge pages sizes. A 1GB huge page can be split into 512 +2MB huge pages. Demote interfaces are not available for the smallest +huge page size. The demote interfaces are: + +demote_size + is the size of demoted pages. When a page is demoted a corresponding + number of huge pages of demote_size will be created. By default, + demote_size is set to the next smaller huge page size. If there are + multiple smaller huge page sizes, demote_size can be set to any of + these smaller sizes. Only huge page sizes less than the current huge + pages size are allowed. + +demote + is used to demote a number of huge pages. A user with root privileges + can write to this file. It may not be possible to demote the + requested number of huge pages. To determine how many pages were + actually demoted, compare the value of nr_hugepages before and after + writing to the demote interface. demote is a write only interface. + +The interfaces which are the same as in ``/proc`` (all except demote and +demote_size) function as described above for the default huge page-sized case. .. _mem_policy_and_hp_alloc: diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 1faebe1cd0ed..f2c3979efd69 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -596,6 +596,7 @@ struct hstate { int next_nid_to_alloc; int next_nid_to_free; unsigned int order; + unsigned int demote_order; unsigned long mask; unsigned long max_huge_pages; unsigned long nr_huge_pages; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 95dc7b83381f..44d3c9477635 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2986,7 +2986,7 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) static void __init hugetlb_init_hstates(void) { - struct hstate *h; + struct hstate *h, *h2; for_each_hstate(h) { if (minimum_order > huge_page_order(h)) @@ -2995,6 +2995,22 @@ static void __init hugetlb_init_hstates(void) /* oversize hugepages were init'ed in early boot */ if (!hstate_is_gigantic(h)) hugetlb_hstate_alloc_pages(h); + + /* + * Set demote order for each hstate. Note that + * h->demote_order is initially 0. + * - We can not demote gigantic pages if runtime freeing + * is not supported, so skip this. + */ + if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported()) + continue; + for_each_hstate(h2) { + if (h2 == h) + continue; + if (h2->order < h->order && + h2->order > h->demote_order) + h->demote_order = h2->order; + } } VM_BUG_ON(minimum_order == UINT_MAX); } @@ -3235,9 +3251,31 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, return 0; } +static int demote_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed) + __must_hold(&hugetlb_lock) +{ + int rc = 0; + + lockdep_assert_held(&hugetlb_lock); + + /* We should never get here if no demote order */ + if (!h->demote_order) { + pr_warn("HugeTLB: NULL demote order passed to demote_pool_huge_page.\n"); + return -EINVAL; /* internal error */ + } + + /* + * TODO - demote fucntionality will be added in subsequent patch + */ + return rc; +} + #define HSTATE_ATTR_RO(_name) \ static struct kobj_attribute _name##_attr = __ATTR_RO(_name) +#define HSTATE_ATTR_WO(_name) \ + static struct kobj_attribute _name##_attr = __ATTR_WO(_name) + #define HSTATE_ATTR(_name) \ static struct kobj_attribute _name##_attr = \ __ATTR(_name, 0644, _name##_show, _name##_store) @@ -3433,6 +3471,105 @@ static ssize_t surplus_hugepages_show(struct kobject *kobj, } HSTATE_ATTR_RO(surplus_hugepages); +static ssize_t demote_store(struct kobject *kobj, + struct kobj_attribute *attr, const char *buf, size_t len) +{ + unsigned long nr_demote; + unsigned long nr_available; + nodemask_t nodes_allowed, *n_mask; + struct hstate *h; + int err = 0; + int nid; + + err = kstrtoul(buf, 10, &nr_demote); + if (err) + return err; + h = kobj_to_hstate(kobj, &nid); + + /* Synchronize with other sysfs operations modifying huge pages */ + mutex_lock(&h->resize_lock); + + if (nid != NUMA_NO_NODE) { + init_nodemask_of_node(&nodes_allowed, nid); + n_mask = &nodes_allowed; + } else { + n_mask = &node_states[N_MEMORY]; + } + + spin_lock_irq(&hugetlb_lock); + while (nr_demote) { + /* + * Check for available pages to demote each time thorough the + * loop as demote_pool_huge_page will drop hugetlb_lock. + * + * NOTE: demote_pool_huge_page does not yet drop hugetlb_lock + * but will when full demote functionality is added in a later + * patch. + */ + if (nid != NUMA_NO_NODE) + nr_available = h->free_huge_pages_node[nid]; + else + nr_available = h->free_huge_pages; + nr_available -= h->resv_huge_pages; + if (!nr_available) + break; + + err = demote_pool_huge_page(h, n_mask); + if (err) + break; + + nr_demote--; + } + + spin_unlock_irq(&hugetlb_lock); + mutex_unlock(&h->resize_lock); + + if (err) + return err; + return len; +} +HSTATE_ATTR_WO(demote); + +static ssize_t demote_size_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + int nid; + struct hstate *h = kobj_to_hstate(kobj, &nid); + unsigned long demote_size = (PAGE_SIZE << h->demote_order) / SZ_1K; + + return sysfs_emit(buf, "%lukB\n", demote_size); +} + +static ssize_t demote_size_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + struct hstate *h, *demote_hstate; + unsigned long demote_size; + unsigned int demote_order; + int nid; + + demote_size = (unsigned long)memparse(buf, NULL); + + demote_hstate = size_to_hstate(demote_size); + if (!demote_hstate) + return -EINVAL; + demote_order = demote_hstate->order; + + /* demote order must be smaller than hstate order */ + h = kobj_to_hstate(kobj, &nid); + if (demote_order >= h->order) + return -EINVAL; + + /* resize_lock synchronizes access to demote size and writes */ + mutex_lock(&h->resize_lock); + h->demote_order = demote_order; + mutex_unlock(&h->resize_lock); + + return count; +} +HSTATE_ATTR(demote_size); + static struct attribute *hstate_attrs[] = { &nr_hugepages_attr.attr, &nr_overcommit_hugepages_attr.attr, @@ -3449,6 +3586,16 @@ static const struct attribute_group hstate_attr_group = { .attrs = hstate_attrs, }; +static struct attribute *hstate_demote_attrs[] = { + &demote_size_attr.attr, + &demote_attr.attr, + NULL, +}; + +static const struct attribute_group hstate_demote_attr_group = { + .attrs = hstate_demote_attrs, +}; + static int hugetlb_sysfs_add_hstate(struct hstate *h, struct kobject *parent, struct kobject **hstate_kobjs, const struct attribute_group *hstate_attr_group) @@ -3466,6 +3613,12 @@ static int hugetlb_sysfs_add_hstate(struct hstate *h, struct kobject *parent, hstate_kobjs[hi] = NULL; } + if (h->demote_order) { + if (sysfs_create_group(hstate_kobjs[hi], + &hstate_demote_attr_group)) + pr_warn("HugeTLB unable to create demote interfaces for %s\n", h->name); + } + return retval; } From patchwork Thu Oct 7 18:19:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12542599 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 823F6C4332F for ; Thu, 7 Oct 2021 18:19:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3085A61040 for ; Thu, 7 Oct 2021 18:19:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 3085A61040 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id CE6306B0072; Thu, 7 Oct 2021 14:19:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C955A6B0074; Thu, 7 Oct 2021 14:19:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 828E1900002; Thu, 7 Oct 2021 14:19:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0043.hostedemail.com [216.40.44.43]) by kanga.kvack.org (Postfix) with ESMTP id 6EA536B0072 for ; Thu, 7 Oct 2021 14:19:43 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 32650181CBDB6 for ; Thu, 7 Oct 2021 18:19:43 +0000 (UTC) X-FDA: 78670454646.08.7658631 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf28.hostedemail.com (Postfix) with ESMTP id C01F89001B69 for ; Thu, 7 Oct 2021 18:19:42 +0000 (UTC) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 197I2r7R025138; Thu, 7 Oct 2021 18:19:40 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=+yy9K+4Qv5Wn6Gj8K3ENqioNUdsnU/Fa18h9VDvRMdU=; b=C1HM24rs39NX9XWL50JM5dnq9gsBXnuU1nQ7rvDwNxImJGf5d/tBgzKvTm5LI3Xo2upc bbJ52n/YDeWlvSSh/XKm/hAtSulJa4hzhzTB9N32775vQwN58A1267RU8Zwt5Un3TCrI OuLcxu5lCTtunpXdhQ2lX5oM2b+Uyh82a4dJC1qjvqznhdNfCy6Uuez0zk3kwYKHYTKN txCBCsxYsLKW5EYp4xmrvoQcRRTfvoHHvfUQvpl2WGpMNeAhpLN9bUlLpLfgEZLD9+LM SgHm+wXUHoX+UJPSNI9Ylz7wYmfQ2lIx9WSAUqZItMUaBfXjlw9I8dPiYayFrTglXQOp 6w== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 3bj0pw2vfd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Oct 2021 18:19:40 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.1.2/8.16.1.2) with SMTP id 197IA8BQ047007; Thu, 7 Oct 2021 18:19:39 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2042.outbound.protection.outlook.com [104.47.66.42]) by aserp3030.oracle.com with ESMTP id 3bev7ww3ea-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Oct 2021 18:19:39 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SIjH9MzJzM4TuT5x6IeCK59pgI/rgDE0BBqqSCXaAcq99Oi/cR1TpVK2lRUwnDeLceQcgaKt7yXxMXEhLjeSXCeIIPDwCm8WjOEUo5MvikqJN/CzI1uGLWq/pnwIJWASvKmKuJCknJ1DgyEZYqexef1epFh13Mqa2uc0/O6qZKs0+NVWRjz2w4sx32a5yq2T53jGyo+dDyz2pvDCnTJPSba3qBeD9VSZ3LZ9jkwoa7NF+bohyAy+WaID792wx0s9ejL8s36HVTTLoa2F6ONL4PxG/5g4oi7ZQg///wPt8d3a0oR7Twpjvkl++VTJK3ZIneRE17d2XASxeWpW9Tsc0Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+yy9K+4Qv5Wn6Gj8K3ENqioNUdsnU/Fa18h9VDvRMdU=; b=Mghyy+8IKgiFgg6bCxzWsbu+g6AuORSpgxft7RnfSQn0mLRAH2XYUe4i82WcFFTFBKuxJiSxXRa6RuqUZ1cYIBI/G/uVnnMyDtbLUEwJRF0sJfwS4etja/ibRt11Eqxp3b+JbPQ4eszKNzxzH4HTk381weUXJgY1ee7uXmaYhTCGLheivy8Evd3hIJlvRI/Vkft1lLjLZ5rJ75P4SL3qYqAt1a1wfFf34chJbfT8rtaGFWWqRlW5kEqXsR4HtUK0VoWhMy9nnUpfqNF60apBA9WLA6K6lzgxQ5SX7jhFxKUrl19MJh8jW1yxTH8NLIxJ1i8ddGnoSgfzGesVCz3VSw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+yy9K+4Qv5Wn6Gj8K3ENqioNUdsnU/Fa18h9VDvRMdU=; b=z5+5r2VSY5j544ZwraZTTdvdHYnTx26SYkIOd1cxuZwpm700f1GT+ewuD45UdXLvMr/k2FZGhuHJ5rpnhw4x3cHmBVNeZx8lvq19bePg9BPoj/hUm5AkeiV5zLz5Jy9a9z7zDR8njtWEGrcg/udXqwu/vZyeRgSBIY2gZPmHYDE= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB2552.namprd10.prod.outlook.com (2603:10b6:a02:ad::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.18; Thu, 7 Oct 2021 18:19:37 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::bc59:71de:1590:cbf5]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::bc59:71de:1590:cbf5%7]) with mapi id 15.20.4587.020; Thu, 7 Oct 2021 18:19:37 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Michal Hocko , Oscar Salvador , Zi Yan , Muchun Song , Naoya Horiguchi , David Rientjes , "Aneesh Kumar K . V" , Nghia Le , Andrew Morton , Mike Kravetz Subject: [PATCH v4 2/5] mm/cma: add cma_pages_valid to determine if pages are in CMA Date: Thu, 7 Oct 2021 11:19:15 -0700 Message-Id: <20211007181918.136982-3-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211007181918.136982-1-mike.kravetz@oracle.com> References: <20211007181918.136982-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MWHPR1701CA0012.namprd17.prod.outlook.com (2603:10b6:301:14::22) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 Received: from monkey.oracle.com (50.38.35.18) by MWHPR1701CA0012.namprd17.prod.outlook.com (2603:10b6:301:14::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.19 via Frontend Transport; Thu, 7 Oct 2021 18:19:37 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 07e5530b-df52-4dab-39d8-08d989bf05c2 X-MS-TrafficTypeDiagnostic: BYAPR10MB2552: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1923; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: vAIltT4E8H9OS+YmBaISTbzDbwwN3EF6O3kJLeStloVNCuqPTuWy+XDL7Am492LjU8DA6IfB4iorDtT/FSmhjgqEtjadG5Kq/uRp717aqqDzo+jdh2QW5IwKNO6O52YryNvITL3LRoYS8MjuBhMQkInverksypSg0lhIpzNCJGkLzXTUq0U7ZxYetIWtPPw1vjub+/7DCoR4rnNaLZNgPxrUA6AxLcILSRg2fBXa7z+WLRGqQZzCTBx3CQZGnA8NAKnrbgvu6A0za3EkG8UCvtZ4OoMO2tffJ13Qlyn2v80sL1cyWEMDXpwK+8SGPNeDKsPmXM000xI6jkbe0Q0zoG2UvmCD2IyMu1OWoug0wPR8jIg8W/3w6fC0cqSSDz6CfMMauckuItnDWLrzBOKV/L2DdlzRFg2hZNJNcO2cYD1tnO8Cw8WUkE4Sxkvpu7Dl+ucDkDsg/wu2xbUu4MMJkA6S5gODwCc2OBaFS15lzJKVNspBbUV6UftPcbFNQyVl1zrL05iHjxIWS+Qh7Y+viyfMB/kz7z9KnfzrlGoZ8u/h2VO18r9uypqTklfhutwh4boGGF2hv1xLNRdT/h0OhFKiNZk3X+yZ4n9zFLY8pC3/fuMR5lxPAKJYJ8tSYtWX72HOMSzk5AIklAfZ8jZE8zmqHIgn2ChtbFmUWIoM/tcXG6qiJdhL7CELt8NPGkocDvo5YrAhrOGQkBSNLcA2Ng== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(86362001)(38350700002)(316002)(7416002)(44832011)(5660300002)(6486002)(107886003)(2906002)(1076003)(186003)(508600001)(54906003)(38100700002)(6666004)(66476007)(36756003)(66946007)(4326008)(66556008)(52116002)(7696005)(956004)(2616005)(26005)(8676002)(83380400001)(8936002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: plesN2WI2FBg2Wi7JdwkRzo+9eCaxiAMoM4kvp5dlkqgK4GDlsgEB/lb9yD6EjhgSZxYea6fF3hxCFH2k+yDhJMgzgRVPWRoCh4djtHl/sfc4EOuraz563iISFbmfgo4xms9WJmyXqKl4WIXAtLYtVn7bHWclhig7iJZov8L85Mt0lcHCU/PpvcWlMDQ1aEejcWoDLatZ4qD7lpVh250zcWqj9Ir0/Kh0SFAjLiJMplGFKsKuQsBElJwygwnID+NohJtxFU01IXJ52CQ8bDfgTGfGPTCQCQCro0Z4gCYuTRMJVMEmDh8ur8uKoIjdtYpggsDJfhEqhAJYbGTQqYqWALWgVIqEz4UlG9OaY/WUY9EkxKXGVtpq6fTgjBW32N+NKiBpGYMHpzQs/KCTHEIcCzE6mP+r3wRgdoj3rgrCJAZ5bt39YrONOas5LwTIaTLNb3QctUiWztncnX1pchd+KAiRjCL6A+Bm+E3iTSB+L5htRUDS3O8QjRWQ3ir4cjH5lyGFjsaefhqb8rcSELKPR3YrMvH1ExOxkfXFTI1cnroagetDpoWc6//RTGHyVdx48vuhbMXOvnE5KPMLs35dV6oMGnbRP2lm2TB5GrDyuKMmgr41t577EPwjDO9IM1sshX+31jPdh82woMPnV2KfaCqIAYNJ8b2S+PaGrw3rcIkCrPuyV4OLVO/b1PhJ8lZTll3voZLX4ZBdqxSuNeDqn2zur6AfOEqWUzKQyZ0LgDex5ofbHnvIyogWy/EmYXL99pj4WNtzxiq+OxCga3FzApwEgW8HSv9b8x4YEpIAEaopvxXx44oXQYVi2Z3PDQ8Q8pv0QL5MtNWpvSMaILkrRSvp7Dl5Czl2vv+vr6MMTxK9C720U1HLU+5KHG3Vs+SHbgL/rGDVYFpWg3Me8Or3tWJtzkd9i9lyMBOYpjWcef0OG+vCHGDHGvrrEXE++05Kzid2TTnmUrVdT0XAE9BaNV3x3oUw//DkXUW07rARV11aNgJ6d59tWxXkeqNlXPk5W0RUP30nvkJbSnU2dQMvXACZRuuYkDIsvO01qZKC8t3K8XpQ9CB8AnVWQ7CGqkRv7M4zYLS+P2bFCSUVXzu3buE/TDqQBEk7uiHW2Dwf/l6FLwR9vg2WNHm7nqx2foOws7MHKi++uIemNZW9s2CYzQlgCp8iHykJUyV0YR7s2B+DjCRlXt4QUMHF/3/A7AHcjiJcrydlVeHvcYl+bBOEUjw/wrIbK+Od2y8gxZ016+GH/bE/c9Sm4XUZaK0D826XTCjY0ps8p78VvTvWng8fn1d1CjYc6q3BRLrUKKdTWnCsRvJIft7zzDTLHbv8IlN X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 07e5530b-df52-4dab-39d8-08d989bf05c2 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Oct 2021 18:19:37.8895 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: RWMJXJM4Vmx+jpmr1SneTFfQUIH9DvwvELkJQk4eTVDN5ttakF0PLUgQV/UJT3N5xd+1CPHYbq2Rjl3KzM//VA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB2552 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10130 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 mlxlogscore=999 adultscore=0 bulkscore=0 suspectscore=0 malwarescore=0 spamscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2109230001 definitions=main-2110070118 X-Proofpoint-ORIG-GUID: I_sqvM63uolDTOObbGsKPwvXGi6X6JGA X-Proofpoint-GUID: I_sqvM63uolDTOObbGsKPwvXGi6X6JGA X-Rspamd-Queue-Id: C01F89001B69 X-Stat-Signature: zb5mx8kuqxtkmhh8e7fnwxy4pr63jjya Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=C1HM24rs; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=z5+5r2VS; spf=none (imf28.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam06 X-HE-Tag: 1633630782-79447 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add new interface cma_pages_valid() which indicates if the specified pages are part of a CMA region. This interface will be used in a subsequent patch by hugetlb code. In order to keep the same amount of DEBUG information, a pr_debug() call was added to cma_pages_valid(). In the case where the page passed to cma_release is not in cma region, the debug message will be printed from cma_pages_valid as opposed to cma_release. Signed-off-by: Mike Kravetz Acked-by: David Hildenbrand Reviewed-by: OScar Salvador --- include/linux/cma.h | 1 + mm/cma.c | 24 ++++++++++++++++++++---- 2 files changed, 21 insertions(+), 4 deletions(-) diff --git a/include/linux/cma.h b/include/linux/cma.h index 53fd8c3cdbd0..bd801023504b 100644 --- a/include/linux/cma.h +++ b/include/linux/cma.h @@ -46,6 +46,7 @@ extern int cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, struct cma **res_cma); extern struct page *cma_alloc(struct cma *cma, unsigned long count, unsigned int align, bool no_warn); +extern bool cma_pages_valid(struct cma *cma, const struct page *pages, unsigned long count); extern bool cma_release(struct cma *cma, const struct page *pages, unsigned long count); extern int cma_for_each_area(int (*it)(struct cma *cma, void *data), void *data); diff --git a/mm/cma.c b/mm/cma.c index 995e15480937..11152c3fb23c 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -524,6 +524,25 @@ struct page *cma_alloc(struct cma *cma, unsigned long count, return page; } +bool cma_pages_valid(struct cma *cma, const struct page *pages, + unsigned long count) +{ + unsigned long pfn; + + if (!cma || !pages) + return false; + + pfn = page_to_pfn(pages); + + if (pfn < cma->base_pfn || pfn >= cma->base_pfn + cma->count) { + pr_debug("%s(page %p, count %lu)\n", __func__, + (void *)pages, count); + return false; + } + + return true; +} + /** * cma_release() - release allocated pages * @cma: Contiguous memory region for which the allocation is performed. @@ -539,16 +558,13 @@ bool cma_release(struct cma *cma, const struct page *pages, { unsigned long pfn; - if (!cma || !pages) + if (!cma_pages_valid(cma, pages, count)) return false; pr_debug("%s(page %p, count %lu)\n", __func__, (void *)pages, count); pfn = page_to_pfn(pages); - if (pfn < cma->base_pfn || pfn >= cma->base_pfn + cma->count) - return false; - VM_BUG_ON(pfn + count > cma->base_pfn + cma->count); free_contig_range(pfn, count); From patchwork Thu Oct 7 18:19:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12542601 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E82BC433F5 for ; Thu, 7 Oct 2021 18:19:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7E4EC61040 for ; Thu, 7 Oct 2021 18:19:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 7E4EC61040 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 1C29F940007; Thu, 7 Oct 2021 14:19:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 14BA4900002; Thu, 7 Oct 2021 14:19:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EDF34940007; Thu, 7 Oct 2021 14:19:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0178.hostedemail.com [216.40.44.178]) by kanga.kvack.org (Postfix) with ESMTP id DDE57900002 for ; Thu, 7 Oct 2021 14:19:49 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 97E8782499B9 for ; Thu, 7 Oct 2021 18:19:49 +0000 (UTC) X-FDA: 78670454898.10.73D4ECF Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf01.hostedemail.com (Postfix) with ESMTP id 1B3BA50771B8 for ; Thu, 7 Oct 2021 18:19:48 +0000 (UTC) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 197I5Bu6009665; Thu, 7 Oct 2021 18:19:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=Ke4NJ37Bas3PZDEjWRZOHB+0MmguX7q3UWkBCvvRwU4=; b=SfNZHhCj86jCrpFeKMnWluDfdgvKC3Fyz8YUp7lF29EqzkzcJ6vBqckIBNlNU2WRedT1 bs16xB/qD1eGRJs6XmLe2GDkNuxmx9MLT4G/T0D1EfTTzCAn3u3IM3CglZcu1bOM76vF fx9TldBsyXlzQW5osAgEoEv5jiGdVQtnxe2K2HoUEw5qC2YxQ8qYhMtpX+n0jfddWVF/ CZ3IN3sUAW/s1ul8Mr5VZNHnr+dA4H8wkzrL5NlRpzCvrkV44w3tVpBCbA1/WaOw82Di swboOfUsLnWY/y59t26iOSm4wKenn0d+zmBny4zDBuKoZSJoDIKbj/wLTl8aW/6/9CvH zg== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 3bhxbs3rxx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Oct 2021 18:19:44 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.1.2/8.16.1.2) with SMTP id 197IAnqg177633; Thu, 7 Oct 2021 18:19:42 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2045.outbound.protection.outlook.com [104.47.66.45]) by userp3020.oracle.com with ESMTP id 3bf16x9114-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Oct 2021 18:19:42 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lK++3nCbOdfcuj5bkQgc2/Zf9HSNNxmzzZAyztee/zN498+jOA9wgeIzfkVjzWSkwwPl/8oL7mbrh30eSafm9QMcbTGjgZFpO0QVZXFyTOxxKSuWo8B4aCnxIPeGHVt5qbHFLVcVCUcrWScd1spWvrbGPhxN2f8biSkM8cqNOEQT1h/Ortf8sNZ9gAH1AfxPi0JxXt59PhnNgU5WphjyC3wmOk3sQ8SiwrsSP2nHHbkihHnKnZg7nl7bXhjgQOGQPvF3W76C1eukBsOHyjHcwL6l13vMMmOiwlCKSEKB/MbBjAfimeND0JcJA7hAk5kzPsYiVOHPehCkPKqb1lrHyw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Ke4NJ37Bas3PZDEjWRZOHB+0MmguX7q3UWkBCvvRwU4=; b=ixCSQSVfcTycSdvByfykJbP/hFJmqmRrz8vlf84HVascKraRceswJ0TaYTVUrLu5os8yiF6ZvHIRnJrNgdP3jrTxqgjmG5cyNHlGXVnPxS24QVVyzXGUEPhjqytABf+ZPVCO5YTN5ifFqdGpnP2o2pq8ZPNpQXjHYSQCYYSiePmcPJk6wB47aUfyCwpWaKb0mHb4h8oZyjgzLOJh36xanAlircomCt3XQF8Q6JN2CqOxv+JHFjfeZihLOKtJaIjkNTGkZGEb4GrH15+qlU6wCXLIYvov09TplLNcdrPdMOINA+skNnJIG0yuTEsLUd/uU9rKYobniSE/bXv0o5n96Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ke4NJ37Bas3PZDEjWRZOHB+0MmguX7q3UWkBCvvRwU4=; b=u9g7MYrm+z/BBDj1pbsUkd2ONlOeAt+oGRlsP5xTMqb29fAx3+ePi2hW5vxcMNxRWK0f8p43+YBcOt7APdQ5muVcuD9FDU/qbg5DAENs1AEWh9vZHqgjlBIKMvEdgnzMOgxadu5YYiclLw1QBr/iyPKOmb9dT45DRyxSGncvrFQ= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB2552.namprd10.prod.outlook.com (2603:10b6:a02:ad::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.18; Thu, 7 Oct 2021 18:19:39 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::bc59:71de:1590:cbf5]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::bc59:71de:1590:cbf5%7]) with mapi id 15.20.4587.020; Thu, 7 Oct 2021 18:19:39 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Michal Hocko , Oscar Salvador , Zi Yan , Muchun Song , Naoya Horiguchi , David Rientjes , "Aneesh Kumar K . V" , Nghia Le , Andrew Morton , Mike Kravetz Subject: [PATCH v4 3/5] hugetlb: be sure to free demoted CMA pages to CMA Date: Thu, 7 Oct 2021 11:19:16 -0700 Message-Id: <20211007181918.136982-4-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211007181918.136982-1-mike.kravetz@oracle.com> References: <20211007181918.136982-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MWHPR1701CA0012.namprd17.prod.outlook.com (2603:10b6:301:14::22) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 Received: from monkey.oracle.com (50.38.35.18) by MWHPR1701CA0012.namprd17.prod.outlook.com (2603:10b6:301:14::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.19 via Frontend Transport; Thu, 7 Oct 2021 18:19:38 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 5270edf1-4e36-40d2-3c9b-08d989bf06b4 X-MS-TrafficTypeDiagnostic: BYAPR10MB2552: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5516; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qg+/9WQUpsvY1pOKhXGRzvBfUIA3788sv1FFY36GDtnPD+puxrnbRxrJxkdiS8YNHkbsGQw0KymC3UoIeVkrONUAXmv+T59V7JVmWqZ7+Lym5axTHq/o25LznmosNzMgJrK9YMnWte2NMjez/h81Sif+nfj0QF41jcs7IT5UEeP1AeO30cKWr74dT+/NRyAMCF16cr0Pw59CTqd1GLQq+MSWeFT8O181pyzamAexYnu49WpUq6UloA1ZMUEcucGjZmtxCJ9ANN3CxioT+AI90nQk3XZx+wEAQ8k81UydcGkvrLMuHIqvun1K5AHRXIiYbSloVgTX/xz9uvITHEL8fHG8843vVRu/IEIq6WwuzUw/rKsBQB2ZgIGa+DVvCtRE18IZ6dxkcm096jWHhx99LvNX4xv4ZkHXbg2SFWlBe0ZsdSfP7IQZliLk8XMYEnoqYRwmQ+1PloOVcKRvtvtANG6KTfArTa2M9dvFLW+ggZgflEAMBjEXFK07EUBDzxa43mloI5kVAESmKCSiEyfmkYCIojiAg9AWUOvl84arUKSQxZIpD4FRXrakpAnbgR/mK5y+ilsHvBtpgRv+mMhT0IYYVl5ZWVXbYtzesECT11Hyb0rSrfBPSePUYlS5VF2GGS/ob7PcPoa3d7n9XK87sAHlF8TCLRtrCtnWUir+rKh/AlejVp8qfGnTjT3ndXj6iZMvnqQq6+IAT2LH0fU0FQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(86362001)(38350700002)(316002)(7416002)(44832011)(5660300002)(6486002)(107886003)(2906002)(1076003)(186003)(508600001)(54906003)(38100700002)(6666004)(66476007)(36756003)(66946007)(4326008)(66556008)(52116002)(7696005)(956004)(2616005)(26005)(8676002)(83380400001)(8936002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: /nssMMxVHzHBJgUuM/DtRAwL8rx/0axWnqs4eQ1gO06GuoE2dmlVMytpe9yyn1MG6EXC0pEpxaXCw4m8ITt8+vWUylNkHy6Pd4rNAQjlUbZlFkK4XDU2QK9jhgFg/9SuRkIDxJ5B5o2vOHZ0ocmkDGXmF6oCOr7Ve/OCzPutyh/LJ/uYSq+KUU61MmpmpbuXYEeidaQ1jv5Wubt/VZmyMli8sWVBG3MjbpooezD+z8O9kYAn05IBuryzXzsLVbQGh1CFZTE2Tq2c8j3w3t8Xau0jJRPx515p8GKtyoocAB1xUFBO36Q2EBMQzccYzVOZyRavTXDV9b9fqeDr+LMZ1uuKqXiUupEOe/Td9graXs8MwuZyode+47RJuJPs9Xxa2dfYmqsXbDIFgwu9t7MEMH94l4W7c/T6Sp2UXKKl64DAIYlXWuNijtwWgLUhASGsmX1IJhDujvBzKvOIwRMx8rLplL6A5zu41jBprLgn1swlWQvyY45LBq9iYu94OvhmVrwCSa+srBpogPRU//+2TRRQsHbivh8e5cD2Xxg8aEepwo0K1VQ/3zN7/9eVWpfs6zM9EIejYmIYpn1+VHlyrCGe0kha60b+21BEfZHHDGEhiPn3mVDptdrNv78PX0lfb/hwFL/ykbH3N2KcAFdpVQdfcV3MV192eaepY1sa6JaoWUAr7g6YbDB0B8l7WGMCAPVVrfXm6TpHvepsXwrPQCw5qnjmDI0zQNhNG39L8jmsBfY44vqVPx2MLMAB2Z6yxaTGxu8wXyNdJY0J0urvmV8Y3WrhOqzXqm6EsZsOr+EnJlOAfk3WLMnSuZMzIq5yaB39KJFCpFlIzrSRAvYu04ot0wy9cW4Mndx0nY8/3yC5pBknI77q33ACaGj/EndH32QQ0DvvZOcNhdPZ5jM/1UEJ+KKq17VAdK96j9lkwcygPmls0UFnhgR0W+maXO3IIBsAx1l2MJf1Baayty3kym5/VpUvp40s7aBeyXjTGdQ5QfxU0HQzWYwytFguhHC0Q2KuiakvlCIZnTHYnQUlLkGjhgJjulCIy2o/x1C+WU7rpz8FKcbCfBzQcXnJi4f4VNZq/ybuDFFTk91qsvKam9Ict66Fti0/B1pXcHtkNUKJYFNCpHM4tAlekZ0br4rtvydXvcBl3E9CpePTG2/8YY9rE1KT56KrieI4LZitxPuP4EEOP3YQwC6mGAKhtgaLRNT10lpEilpUPFaOm89rFtvHkM8y8+tg7CYjtSvFi/3v4PcMPhP7GgmtiprT9TOcCegQqR0Rvmkyul1JxcWqMfyLI5fV2EjMuas4bbRKtQr1Ltpb3h+JEB/ZXkvFj2I3 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5270edf1-4e36-40d2-3c9b-08d989bf06b4 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Oct 2021 18:19:39.6070 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: vIp0yEOL/K3qBaxh6HGEkwbQ6f9/L8MiXv2rzt5wpj72JdSLud3+Echoxwf6aBdYGdsBvJakBS28ghB74B8nbA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB2552 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10130 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0 phishscore=0 malwarescore=0 bulkscore=0 mlxlogscore=999 mlxscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2109230001 definitions=main-2110070118 X-Proofpoint-GUID: 6CtEoH-PluYfsDRJVY6SpD-4Ndk-JwhW X-Proofpoint-ORIG-GUID: 6CtEoH-PluYfsDRJVY6SpD-4Ndk-JwhW X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 1B3BA50771B8 X-Stat-Signature: mbgx6nwadpfoorgneozt1a6b54u5gjbc Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=SfNZHhCj; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=u9g7MYrm; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf01.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=mike.kravetz@oracle.com X-HE-Tag: 1633630788-459360 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When huge page demotion is fully implemented, gigantic pages can be demoted to a smaller huge page size. For example, on x86 a 1G page can be demoted to 512 2M pages. However, gigantic pages can potentially be allocated from CMA. If a gigantic page which was allocated from CMA is demoted, the corresponding demoted pages needs to be returned to CMA. Use the new interface cma_pages_valid() to determine if a non-gigantic hugetlb page should be freed to CMA. Also, clear mapping field of these pages as expected by cma_release. This also requires a change to CMA region creation for gigantic pages. CMA uses a per-region bit map to track allocations. When setting up the region, you specify how many pages each bit represents. Currently, only gigantic pages are allocated/freed from CMA so the region is set up such that one bit represents a gigantic page size allocation. With demote, a gigantic page (allocation) could be split into smaller size pages. And, these smaller size pages will be freed to CMA. So, since the per-region bit map needs to be set up to represent the smallest allocation/free size, it now needs to be set to the smallest huge page size which can be freed to CMA. Unfortunately, we set up the CMA region for huge pages before we set up huge pages sizes (hstates). So, technically we do not know the smallest huge page size as this can change via command line options and architecture specific code. Therefore, at region setup time we use HUGETLB_PAGE_ORDER as the smallest possible huge page size that can be given back to CMA. It is possible that this value is sub-optimal for some architectures/config options. If needed, this can be addressed in follow on work. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 41 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 44d3c9477635..563338f4dbc4 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -50,6 +50,16 @@ struct hstate hstates[HUGE_MAX_HSTATE]; #ifdef CONFIG_CMA static struct cma *hugetlb_cma[MAX_NUMNODES]; +static bool hugetlb_cma_page(struct page *page, unsigned int order) +{ + return cma_pages_valid(hugetlb_cma[page_to_nid(page)], page, + 1 << order); +} +#else +static bool hugetlb_cma_page(struct page *page, unsigned int order) +{ + return false; +} #endif static unsigned long hugetlb_cma_size __initdata; @@ -1272,6 +1282,7 @@ static void destroy_compound_gigantic_page(struct page *page, atomic_set(compound_pincount_ptr(page), 0); for (i = 1; i < nr_pages; i++, p = mem_map_next(p, page, i)) { + p->mapping = NULL; clear_compound_head(p); set_page_refcounted(p); } @@ -1476,7 +1487,13 @@ static void __update_and_free_page(struct hstate *h, struct page *page) 1 << PG_active | 1 << PG_private | 1 << PG_writeback); } - if (hstate_is_gigantic(h)) { + + /* + * Non-gigantic pages demoted from CMA allocated gigantic pages + * need to be given back to CMA in free_gigantic_page. + */ + if (hstate_is_gigantic(h) || + hugetlb_cma_page(page, huge_page_order(h))) { destroy_compound_gigantic_page(page, huge_page_order(h)); free_gigantic_page(page, huge_page_order(h)); } else { @@ -3001,9 +3018,13 @@ static void __init hugetlb_init_hstates(void) * h->demote_order is initially 0. * - We can not demote gigantic pages if runtime freeing * is not supported, so skip this. + * - If CMA allocation is possible, we can not demote + * HUGETLB_PAGE_ORDER or smaller size pages. */ if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported()) continue; + if (hugetlb_cma_size && h->order <= HUGETLB_PAGE_ORDER) + continue; for_each_hstate(h2) { if (h2 == h) continue; @@ -3555,6 +3576,8 @@ static ssize_t demote_size_store(struct kobject *kobj, if (!demote_hstate) return -EINVAL; demote_order = demote_hstate->order; + if (demote_order < HUGETLB_PAGE_ORDER) + return -EINVAL; /* demote order must be smaller than hstate order */ h = kobj_to_hstate(kobj, &nid); @@ -6543,6 +6566,7 @@ void __init hugetlb_cma_reserve(int order) if (hugetlb_cma_size < (PAGE_SIZE << order)) { pr_warn("hugetlb_cma: cma area should be at least %lu MiB\n", (PAGE_SIZE << order) / SZ_1M); + hugetlb_cma_size = 0; return; } @@ -6563,7 +6587,13 @@ void __init hugetlb_cma_reserve(int order) size = round_up(size, PAGE_SIZE << order); snprintf(name, sizeof(name), "hugetlb%d", nid); - res = cma_declare_contiguous_nid(0, size, 0, PAGE_SIZE << order, + /* + * Note that 'order per bit' is based on smallest size that + * may be returned to CMA allocator in the case of + * huge page demotion. + */ + res = cma_declare_contiguous_nid(0, size, 0, + PAGE_SIZE << HUGETLB_PAGE_ORDER, 0, false, name, &hugetlb_cma[nid], nid); if (res) { @@ -6579,6 +6609,13 @@ void __init hugetlb_cma_reserve(int order) if (reserved >= hugetlb_cma_size) break; } + + if (!reserved) + /* + * hugetlb_cma_size is used to determine if allocations from + * cma are possible. Set to zero if no cma regions are set up. + */ + hugetlb_cma_size = 0; } void __init hugetlb_cma_check(void) From patchwork Thu Oct 7 18:19:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12542603 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD776C433EF for ; Thu, 7 Oct 2021 18:19:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6D58B61040 for ; Thu, 7 Oct 2021 18:19:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 6D58B61040 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 0FF54900002; Thu, 7 Oct 2021 14:19:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0D6C8940008; Thu, 7 Oct 2021 14:19:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E43F1900004; Thu, 7 Oct 2021 14:19:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0117.hostedemail.com [216.40.44.117]) by kanga.kvack.org (Postfix) with ESMTP id CD7EA900002 for ; Thu, 7 Oct 2021 14:19:51 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 8BDE9181A2565 for ; Thu, 7 Oct 2021 18:19:51 +0000 (UTC) X-FDA: 78670454982.26.2ECB48E Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf23.hostedemail.com (Postfix) with ESMTP id 0D24A9001055 for ; Thu, 7 Oct 2021 18:19:50 +0000 (UTC) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 197I450W009645; Thu, 7 Oct 2021 18:19:46 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=V3c6Vzy/AeULLYVxMtzlAeTECPEIEraba5pN5XoNpVk=; b=NIYFrSdeUdkS5axxwnwLtz6XSKrU/HJ++Ohi6e8VsSdjkB9jrdmSrVBuTrcIaIB0WGAM 2THVFykK+9hx0I7C1LdFrqzBQHbXSAimqJH+U0bqg6u/A44mscp4LPKYYj3XU8T3PnDY QSALX0QgMbqIxrLalUpu2w4bhNO9AFc9HB3acZfgk8+QH1r7B1GScfrCHfGYzrYwIIoH oD9GlRkjhER0fJ2VywDs5XR3kB6KhmteHIywerK/3ygXc+w3K7+rxMgLkNupy5H7ytlg arV8DNhioMA68snxDxqBbW8MzB5KFFQRZ3F6DDSwar5H+jvWri0NqJbJPtsLBJ3lYRKk 6A== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 3bhxbs3ry8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Oct 2021 18:19:45 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.1.2/8.16.1.2) with SMTP id 197IAnqi177633; Thu, 7 Oct 2021 18:19:43 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2045.outbound.protection.outlook.com [104.47.66.45]) by userp3020.oracle.com with ESMTP id 3bf16x9114-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Oct 2021 18:19:43 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dY/aTTpsJZ1Nq1vYXTTnvmrk2I9Bip4M9NjlwOxlDJun+TyAPwSmxLdV9ncuIhSSKKcJnbo7wkciQ+O5sqDFWi6nb6/yeawZe6ucVk0RYaMlDgqAOW9dlSATpV2agQs+n+G6bu3HIKl/2dG7xcU8aCWTV99TtNGga2LtWtIVebAPuFfk5PW52o1a729GuT5D9XRO39BumPU52e/WRkSZWi4EvuYNAshHx47OOvtbsuJ9uAGajDYAIa91TbNjrfl+fOxRjPQVCB0lAEtMhFdtYxpeokBlAHvO2F76cdAgteroptrGJIDvTk0XTxXbECe5/EQ+a8OWHWmBscStdhgiEg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=V3c6Vzy/AeULLYVxMtzlAeTECPEIEraba5pN5XoNpVk=; b=deJBw55B8wbOwcVLKaLpXVqlGYJQDU+J74vSecoFsGloTTo2Q/jyR3lZDy0m3mQ/5tZpBdulU2ZoOCO40tmKjwZY++HuI1DiiZHV4L7bplaAebYE9LwjEJCtb/GkrN/ae2iOifRBk2NgQ2pzVIG3QdSN87guOSyn7QR1jaC7NSY8d4Pn6V3EwO2wI3Fngocu5NzfC7EnW+Vwh8GQg1rcsgW/FSfRRHKtEMb4uuD8h2PSY732Wg/3uEBXTZVBDyaC7Xhy51p/8yPRkhheQOInnaGL7pnXiuwI+xjyCJ8VpB2/Zzd1gmBDgL7lHCKkWw1dW4aAywFlM7KRpxkPkU3Ddw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=V3c6Vzy/AeULLYVxMtzlAeTECPEIEraba5pN5XoNpVk=; b=y8s+o8tRElAA0yp2QHDr+6kUDii0v6CyXAPqPMPiu0V8cNhDdspDDBcMD5d5XI8S8rwYODhtKBPBaY4hHqewSH1LLzlfYu5G64Vayjy9oypj0Rw3xz+Q7LrgJgypmiJQ2Rn3/N7T8wsP9h+mIMiyzdOjWejJMANuHvb3eMznH1A= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB2552.namprd10.prod.outlook.com (2603:10b6:a02:ad::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.18; Thu, 7 Oct 2021 18:19:41 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::bc59:71de:1590:cbf5]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::bc59:71de:1590:cbf5%7]) with mapi id 15.20.4587.020; Thu, 7 Oct 2021 18:19:41 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Michal Hocko , Oscar Salvador , Zi Yan , Muchun Song , Naoya Horiguchi , David Rientjes , "Aneesh Kumar K . V" , Nghia Le , Andrew Morton , Mike Kravetz Subject: [PATCH v4 4/5] hugetlb: add demote bool to gigantic page routines Date: Thu, 7 Oct 2021 11:19:17 -0700 Message-Id: <20211007181918.136982-5-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211007181918.136982-1-mike.kravetz@oracle.com> References: <20211007181918.136982-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MWHPR1701CA0012.namprd17.prod.outlook.com (2603:10b6:301:14::22) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 Received: from monkey.oracle.com (50.38.35.18) by MWHPR1701CA0012.namprd17.prod.outlook.com (2603:10b6:301:14::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.19 via Frontend Transport; Thu, 7 Oct 2021 18:19:40 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: afb97c48-61bf-4062-bbfd-08d989bf07c8 X-MS-TrafficTypeDiagnostic: BYAPR10MB2552: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: FyYOo1i0YzPExBwo97Mbb2WeqXXeBo1+VXLbUti9+6cIjqwyz8+ER4XF/clVCDK2U8RodvIRVeFPYt55zy2fAzAYHCqtHAiesel+0C44kdRlHj6qiiYDzlE+VNj2yKJTVhNYo2W/C4XBJmRfzjk7PTPyExoK5zXb4bEz6ysoLkiVVPCyfCE5GDfBnjQax0qjKl+W0KS12fC1hfYfMNtYL4BnbSH4mEkITWw8kXNgSjFO37V8ckImogJZADRWPQN3BIhfhyyo3rjrTCicKyQUvKO2LWvvXcB+HLvvfSpShMp4PY0CX29aVJPIBQovwOiKeVk31V76/JHMhNA1O6eb2B0bZlA32Y5D9NsMhOkMmwSHarzLWcMB7uPo0LlOTISqh6hEJNz2kiYGi6vMpCrpKwt+65UYmLsV81jVOFi3mY7wwtltK4/yLC846EzTNq3vmbP45OaJ8j+julEGgUhND8XOIrL581pLXQARJBGCVKABAV6z0SgaQGEdvXoIUQ3Vs1bSeVrsEaOvDAJlGtLNcl7IcmILU8OIafSamHV3ZpsBTQjAfjFoilCBEnSlN4bb6e2Yn6Vk35FJSbz02xVQLhuqCdIZ9jd2lBkDCYhYrgU0+owpVXoEhIEWFd9TNf3SubHihvi2n/qxtJ7JViyjIQ+0tGQ7AyHnJICyc+78ZwyIF7WOGaD4V63GnWe8HMZAh6mCCQuxPkyjOZVZByCG/w== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(86362001)(38350700002)(316002)(7416002)(44832011)(5660300002)(6486002)(107886003)(2906002)(1076003)(186003)(508600001)(54906003)(38100700002)(6666004)(66476007)(36756003)(66946007)(4326008)(66556008)(52116002)(7696005)(956004)(2616005)(26005)(8676002)(83380400001)(8936002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Q6oKAZgR5eczfhf48LzePNXUP3PUHy1PAByYHz35zdtQ3vBffNxojje+hSfkcWNY7XiuZChyNRQpYVhhk0fAtQRQtB/kgGjtGwCL7qnXiR9A9ywcxD55DOVvn4P7p1SO5sFtYb+zVFQQQTx3hCZVv0fXF4WVkCAAO5a6MCQIjRVfz7iIkvP1akmO6JN+8kZggACzAfUB4AyGqorTJkipnyIniQXSFIYQQO9Tovb9ZU/DCmAJDXnhN06ujdjf+vksRfSael0iTkH6SL/W6qX/59JTSwBn/maib+xO/XBl4rRqn2xCkGtLf29WzgK/J9LvgYTw9I87erM2ole4bXlJW2qiUmEwh16O2uyYD03SPNPrsAb+JUDweLofrE9pBxAwKSrFKVNnKCkJEsJ+twnUsMq2P4nJhz7A0YxZGCk4sqHhAQLdS6o6L0e0bANlrV3e3LA0cXuGvCFvvnGuR5e7JB7aRbdFi7ywOTCgPbCikCEVlqYahXBat+lxEMEMjVLVWScDcFp+/ufS8cquZBZ24qCiLCAYWDyEOHXfeScwASM0jYkGvKkUdbhG91I7Lwj3nFvkETcnq0hrXoNTkK7OzE2Qf6vfvMxHYyi50isI8FDEfdh27ttJsKEGnDUdgLw8QOUNalAhOD3zB8xFNnesBN9rWgw5Mqvzfbp0XtYpaj7RqrYRbA9rtvTPHCEHQQZORp6Kf9GmsCEwv6et1pWcEmD4Y261ULRHKskDI4RbBTB2I0dFelTepGTCR7IavW5neOT3klEzfdhcoosxsEIaLF/JVZsJcK2deaXLuxWBMsP4MaeUIk0PoeUpl8XXhHNyTsWevNP6zU/RNaa+kcTWiO8vUFhUXQqg2jJG/iIjudmwvqdBJSveY0alGSxgoy7eXMY43fLoEfe/WoxhyieZU96KFFTcKCHeSJPl5pepFGX4CIVzaO451RbIHKDoa+5FyIAm2xbxVshYa8WhuTqP132Od5GMTnh5BeUKDJ4O2SFU8HdDJOw1ry9D5Jls8r5S8VJVUl2qWmCN+imqvE46yGyZm7FQY6N64ZaXXn5oq1DttrXk5OR5DILl8M2enc9eONYORj8ifXLNvMt7AKqwPp55mFouNqnRt2faoZPuJ0L2PYVy+fSNs7jmyiB1PEqizSLyHW0cy7UOJmQ+CyeNjo+631gmqajL6hVkoHADJ5Rh36q6W/3QvLv5VSYeMgh/a4lbj2MZ1aYM8ugyYP+MfdWgmZP7/LsHvoCtG3dqUtwY/GjSdar624SGvltpTtA9N6NY0oNV1UlG3ArrtkNohntmEvXV7RBlwq9hqbsXO5Hh7AzrCwcdRHhkPMUCxcp7 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: afb97c48-61bf-4062-bbfd-08d989bf07c8 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Oct 2021 18:19:41.3165 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: jtYdt6kiGFywvreNtwuwkPFRdw17kxX4GHhopvh4KL0cWG6SpIhuXugphZO+c6nM5cZ+Zq/VQnffyJUtX/KXKA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB2552 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10130 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0 phishscore=0 malwarescore=0 bulkscore=0 mlxlogscore=999 mlxscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2109230001 definitions=main-2110070118 X-Proofpoint-GUID: Ig5iUroze3oKPTKexSaLSL63zx7RCC_2 X-Proofpoint-ORIG-GUID: Ig5iUroze3oKPTKexSaLSL63zx7RCC_2 Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=NIYFrSde; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=y8s+o8tR; spf=none (imf23.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 0D24A9001055 X-Stat-Signature: ct1gj7ztaohtt1gnurdecjoh66j43js7 X-HE-Tag: 1633630790-231650 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The routines remove_hugetlb_page and destroy_compound_gigantic_page will remove a gigantic page and make the set of base pages ready to be returned to a lower level allocator. In the process of doing this, they make all base pages reference counted. The routine prep_compound_gigantic_page creates a gigantic page from a set of base pages. It assumes that all these base pages are reference counted. During demotion, a gigantic page will be split into huge pages of a smaller size. This logically involves use of the routines, remove_hugetlb_page, and destroy_compound_gigantic_page followed by prep_compound*_page for each smaller huge page. When pages are reference counted (ref count >= 0), additional speculative ref counts could be taken. This could result in errors while demoting a huge page. Quite a bit of code would need to be created to handle all possible issues. Instead of dealing with the possibility of speculative ref counts, avoid the possibility by keeping ref counts at zero during the demote process. Add a boolean 'demote' to the routines remove_hugetlb_page, destroy_compound_gigantic_page and prep_compound_gigantic_page. If the boolean is set, the remove and destroy routines will not reference count pages and the prep routine will not expect reference counted pages. '*_for_demote' wrappers of the routines will be added in a subsequent patch where this functionality is used. Signed-off-by: Mike Kravetz Reviewed-by: Oscar Salvador Signed-off-by: Mike Kravetz Reviewed-by: Oscar Salvador --- mm/hugetlb.c | 54 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 43 insertions(+), 11 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 563338f4dbc4..794e0c4c1b3c 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1271,8 +1271,8 @@ static int hstate_next_node_to_free(struct hstate *h, nodemask_t *nodes_allowed) nr_nodes--) #ifdef CONFIG_ARCH_HAS_GIGANTIC_PAGE -static void destroy_compound_gigantic_page(struct page *page, - unsigned int order) +static void __destroy_compound_gigantic_page(struct page *page, + unsigned int order, bool demote) { int i; int nr_pages = 1 << order; @@ -1284,7 +1284,8 @@ static void destroy_compound_gigantic_page(struct page *page, for (i = 1; i < nr_pages; i++, p = mem_map_next(p, page, i)) { p->mapping = NULL; clear_compound_head(p); - set_page_refcounted(p); + if (!demote) + set_page_refcounted(p); } set_compound_order(page, 0); @@ -1292,6 +1293,12 @@ static void destroy_compound_gigantic_page(struct page *page, __ClearPageHead(page); } +static void destroy_compound_gigantic_page(struct page *page, + unsigned int order) +{ + __destroy_compound_gigantic_page(page, order, false); +} + static void free_gigantic_page(struct page *page, unsigned int order) { /* @@ -1364,12 +1371,15 @@ static inline void destroy_compound_gigantic_page(struct page *page, /* * Remove hugetlb page from lists, and update dtor so that page appears - * as just a compound page. A reference is held on the page. + * as just a compound page. + * + * A reference is held on the page, except in the case of demote. * * Must be called with hugetlb lock held. */ -static void remove_hugetlb_page(struct hstate *h, struct page *page, - bool adjust_surplus) +static void __remove_hugetlb_page(struct hstate *h, struct page *page, + bool adjust_surplus, + bool demote) { int nid = page_to_nid(page); @@ -1407,8 +1417,12 @@ static void remove_hugetlb_page(struct hstate *h, struct page *page, * * This handles the case where more than one ref is held when and * after update_and_free_page is called. + * + * In the case of demote we do not ref count the page as it will soon + * be turned into a page of smaller size. */ - set_page_refcounted(page); + if (!demote) + set_page_refcounted(page); if (hstate_is_gigantic(h)) set_compound_page_dtor(page, NULL_COMPOUND_DTOR); else @@ -1418,6 +1432,12 @@ static void remove_hugetlb_page(struct hstate *h, struct page *page, h->nr_huge_pages_node[nid]--; } +static void remove_hugetlb_page(struct hstate *h, struct page *page, + bool adjust_surplus) +{ + __remove_hugetlb_page(h, page, adjust_surplus, false); +} + static void add_hugetlb_page(struct hstate *h, struct page *page, bool adjust_surplus) { @@ -1681,7 +1701,8 @@ static void prep_new_huge_page(struct hstate *h, struct page *page, int nid) spin_unlock_irq(&hugetlb_lock); } -static bool prep_compound_gigantic_page(struct page *page, unsigned int order) +static bool __prep_compound_gigantic_page(struct page *page, unsigned int order, + bool demote) { int i, j; int nr_pages = 1 << order; @@ -1719,10 +1740,16 @@ static bool prep_compound_gigantic_page(struct page *page, unsigned int order) * the set of pages can not be converted to a gigantic page. * The caller who allocated the pages should then discard the * pages using the appropriate free interface. + * + * In the case of demote, the ref count will be zero. */ - if (!page_ref_freeze(p, 1)) { - pr_warn("HugeTLB page can not be used due to unexpected inflated ref count\n"); - goto out_error; + if (!demote) { + if (!page_ref_freeze(p, 1)) { + pr_warn("HugeTLB page can not be used due to unexpected inflated ref count\n"); + goto out_error; + } + } else { + VM_BUG_ON_PAGE(page_count(p), p); } set_page_count(p, 0); set_compound_head(p, page); @@ -1747,6 +1774,11 @@ static bool prep_compound_gigantic_page(struct page *page, unsigned int order) return false; } +static bool prep_compound_gigantic_page(struct page *page, unsigned int order) +{ + return __prep_compound_gigantic_page(page, order, false); +} + /* * PageHuge() only returns true for hugetlbfs pages, but not for normal or * transparent huge pages. See the PageTransHuge() documentation for more From patchwork Thu Oct 7 18:19:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12542605 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03F16C433EF for ; Thu, 7 Oct 2021 18:19:56 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9681260F4C for ; Thu, 7 Oct 2021 18:19:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 9681260F4C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 36F1C940009; Thu, 7 Oct 2021 14:19:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F847940008; Thu, 7 Oct 2021 14:19:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 14C28940009; Thu, 7 Oct 2021 14:19:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0162.hostedemail.com [216.40.44.162]) by kanga.kvack.org (Postfix) with ESMTP id F32D2940008 for ; Thu, 7 Oct 2021 14:19:54 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id B237D2DD7E for ; Thu, 7 Oct 2021 18:19:54 +0000 (UTC) X-FDA: 78670455108.01.0A0BF65 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf20.hostedemail.com (Postfix) with ESMTP id 32FABD003E10 for ; Thu, 7 Oct 2021 18:19:54 +0000 (UTC) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 197IBips020977; Thu, 7 Oct 2021 18:19:50 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=k1gSGNFr2IAFYHqvcKNZGTQZUar5E01BP6TqHAFUhKg=; b=R8G2awi2pyf7eq6KbL9W638FQJRD8/0hrVKJjWsPLOqgpL4Eb0BiE6Nl5xbd3OszurLY 9CDF+0qZ8J5RKHjFNlQyDMFX8LlU55QieSNUeyJ+RDYG8c/Sv2ztvhx2SnMCOUht2y3p zQYFMxrdMj+ZtwsN7XOlrRPzs6rOQrccXqvQHqn3Za3uQoOtfQ3qV8k3CDavY/SaJqIx r+tDwzd/SV9iAKHWJdbYSitLIIS8tDrx7BJ/u/uc5DxLEBBjkBehYQmdibsM0pwTo7do 2+fcw6jm16tiu3TUg6adhHEZBmPh+jEIOiG1htm2KHFJekcWoZqs6fi3xvNwl3QGe910 ag== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 3bhwfdkuje-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Oct 2021 18:19:49 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.1.2/8.16.1.2) with SMTP id 197IAnt5177657; Thu, 7 Oct 2021 18:19:46 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2041.outbound.protection.outlook.com [104.47.66.41]) by userp3020.oracle.com with ESMTP id 3bf16x914v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 07 Oct 2021 18:19:46 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FfYQDNkeRF/HgkM/yvAjTxU+6tmMZKG5SB24jaFewXiR/QJkg8VcyoMz6heJaDAnK7ciMdyfQ7yNi0IBJmUjtsWld3hFblP45pyyqsxvtNi229dcVggxfOfsv0IOTzHBcZaIKQuAALfjHlTtLEGiF9sVa7e0fSEe6y/nhOcxDNExyQNVBmKGX0kyCLa0ISCK6FNu+UwskP3h3xopl0rd+N1jk1JLMTcsPIRIRpTT1UotGw8pIvaKuQyoQh/jw5DPjkEitbxbQZCzF8ToqcYvV2GGPEeM4gqqmJOn8e/J63y090KMlCPVdWXVHVYOrMVUiRWDwWQhCrOljRpwTHEcJw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=k1gSGNFr2IAFYHqvcKNZGTQZUar5E01BP6TqHAFUhKg=; b=j03xYWgWsiI/B1ivnoFB6RiooO5Z/MER+0kp6qnH/TJCGufA1s0WeXsdEwuH+sT5MEu+7kfB3wouDlO/foZ8HqRmP0cMc1w4zwYRuFxS+IMNsAbpT51lOo6/4g55TGbLJmh4/9JuV57aqZGybjcOOndGykXL7fnOGRv5h2wWmirNKYs7JumrVFovuJS+DgnJnvcV7vUXtA79Y4FCV7VIx51DCuqGzwVLipmfkznYvciXLyokutbFZpW7K2Rj8NHVmBPvB0ZeR6fsAWasFFQjmMX9uZwJtm/+LmwCoXfkMOj5dvbXFOaG5KXsm7yYFpEWKR+4KDuUY1aQtT6kvkDfcg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=k1gSGNFr2IAFYHqvcKNZGTQZUar5E01BP6TqHAFUhKg=; b=L/4fXMgsx2x7u0XI/fBM1bAGAtP+ng3EFBC8RARQ3QJipS+qMFas+yIf+JvlvzTSbBw8oE+zbZ6nTJZLFRuNTNATzJnzs3Hpsr/n853spw2k5VFt15YXMfSfPkDn7pYufdPS6b4OE5pjVdA9Op/J7qbx2Xr//lrf9F/hLJIdQHk= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB2552.namprd10.prod.outlook.com (2603:10b6:a02:ad::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.18; Thu, 7 Oct 2021 18:19:44 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::bc59:71de:1590:cbf5]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::bc59:71de:1590:cbf5%7]) with mapi id 15.20.4587.020; Thu, 7 Oct 2021 18:19:44 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Michal Hocko , Oscar Salvador , Zi Yan , Muchun Song , Naoya Horiguchi , David Rientjes , "Aneesh Kumar K . V" , Nghia Le , Andrew Morton , Mike Kravetz Subject: [PATCH v4 5/5] hugetlb: add hugetlb demote page support Date: Thu, 7 Oct 2021 11:19:18 -0700 Message-Id: <20211007181918.136982-6-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211007181918.136982-1-mike.kravetz@oracle.com> References: <20211007181918.136982-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MWHPR1701CA0012.namprd17.prod.outlook.com (2603:10b6:301:14::22) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 Received: from monkey.oracle.com (50.38.35.18) by MWHPR1701CA0012.namprd17.prod.outlook.com (2603:10b6:301:14::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.19 via Frontend Transport; Thu, 7 Oct 2021 18:19:43 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 70f09238-7147-4cd0-812e-08d989bf0965 X-MS-TrafficTypeDiagnostic: BYAPR10MB2552: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6108; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Oc0E0bfM3q1W+NSAY8MnGMuLkm4FExQmaaj9Iueifn0EO5GvLCRdArP8zyKUK+cnkR3aIYLBVUs2cAE0kvk0lf7f0BlM5r8d34BVlIKiJ1VBY2ueUnR+sLsBQbmAQlqvDTbJizIk/EssD9F8HxHy/MbA07EFrV+fLCcuPaBtH7uGpA7dRYgoNmQ9mzNNbnADDmvxhEbTVoYKyXX1G6nXnRtpOpK+7fUCG0GudlKaLV+hRiAZbGmBbloMO3AP6qjM1FRy7+Dwj3GUGHUSZY0+CldzM2ZU85p3Jg6BCoP5lS85dKlvedS8CBdNDMtceqkL0CCV0IjvAd+BrJw3fmXlt8a9vzCxWx4Dy003h41g6tgOBig+aGDVQsEVHdDAQ33BLTZA/GNDAGH2l9SHMZVq6nOCs1XEH+RhV0VPQTlE2ipH8N0odR4mcSancCbKPCN8crVKKySjczLFhZapUGVyBzd/qK7mQzKwRfgdQgdkYSF2Cp2eQvTqFGK/3uP3mrURGjst1xYNV74oLeh/08Lk0zpZs7IQAgGy8TzPrkYrM0Sgfrqyxh7tm5LiDc8KQmRgn3rqXeFYhH5zd0vJYZVbC0oERlYA0J4Dt3xPIt8tat2Wxroo+kvPL+tsjn848StsuOpyclvfEzaTGamN5sZf3OyNDeV/pCee5OD3pW8yiR7z5fUAeF/QpHAN3MNNcTd0yK+EMyyeOm/EQC0JiWV5sA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(86362001)(38350700002)(316002)(7416002)(44832011)(5660300002)(6486002)(107886003)(2906002)(1076003)(186003)(508600001)(54906003)(38100700002)(6666004)(66476007)(36756003)(66946007)(4326008)(66556008)(52116002)(7696005)(956004)(2616005)(26005)(8676002)(83380400001)(8936002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: PIdGzAlDSagakaHvy0fnguLaoOhpp9kLYyAa6Dkk67jRohiZ/aVMYjy83ZamHLPoCJ3jxP8NJkpUeuA2IPCeSK49vNqLev6W113vzyvbBL7Aj33T1TPpQFjv72Y/8QUOUKSiol1Ads9yeUWOjRBQdIQQwlPMyphqGxgl1zUHMv3k+l1RfYGFJwxP7XP6pVjkeRDX06EHiRUShXDcQvJKR+teAdptmm11lkKnUb5h163FsktnHxfYK7s5olt+21d6vvF8bQnAfQShoaFLVt0nPYSrUQbXEbWuIMGInKaoSihxsjhbxEvUiFCZZKOdwY22VNAabkNw2plEdScCffQ2KgwTuhExLJwbRskCJnNmQjlJ0vgpnLwyywROSHzRmTrRAd3DER23/LrAHE4Lu7gb7k2EVYbuNJ+HZsGumo3KqQEjf0qN7XyMC3FkY5StlKpekoAklZcRDXrSx9SYV3UxJWjztC/Fdi3DVYuEJKpkN7WUKM6fD1Ns0WH0aKbT/cWrBvAeQk53ru/JPSJrNAUViwhsAHNDRaTKoVQ5mPzmIB1VSLEugaDWtnKirco2YdmtQdUUueervjsOBCGbv6bUeM1rFrCnNCVllVWGxlvHGI7f6/TMnxuVZEOMS7qdxzSUcDZXmTFm3j3d7Wlg0fNmQggUWsoGwE3ExzsB4vi8aA+uX8/SPLy3n/UWrzndTg5lmsJ4yNeNh+g4cripOSs+91DAIRJnU7EgHJk3ETJzjkBJFXmKCu8ckKBkzGRbtVgT/45yv9mLdoedhn4kLfAtYcFamfsm5sgoSnwCSGwGuApFVF44MaXAAgzNIMs/EEdn3Vp6MwCcaR2DwL9Fb2J5eblJz3Hm321/QnyBnR2FIZl03vf7d8fpnvQm+W2yg9wM63q9qrDRkN+hu+nX6cQ0mi1g9i7a2Lp88QJe8exnN3fwcbgHSQsInqPxShmuWfnIkI1HubsK9QutoJi682EeXLIBGICleXK62fjZrhQKYp+d8+JMBRXWOIyZgKRuBNMVMnMiMdymfnXUmN8NfMFB7SDaByB8ofeq4mKKacG8IggyundkYPjeLXsxFAAz0FzYotKVL4I9j/3/s6zmIvEEw9PMIGcm03Rgw2OB2+gzC69iSuzgJGpZcgZFkYgmCQI+T5Adxkw1nSFHTQmAs7uzGucOCK9pZUUAulP4UnnjKddhiwBwYfvOzbqUJPpzCqGRaTs3UacwqzrlFuBTp5QRWv02+fy6QO5rIT13ikHDf993/Nl/kZuD3e73KxcScWmGR7GjU9kndXqYoIUQffBlkgJ1dj53CT5JVM+evQ52q6N8fv459eny0vEvO3z7JT36 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 70f09238-7147-4cd0-812e-08d989bf0965 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Oct 2021 18:19:44.0485 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 8rOi98N9NhmFC9RqR05NgojdS0jqk4Hhk1cnDZ4XH3ZXOCHXV2iZMi0Z2iKtCjAiQU+HSf4RwLwigY4qvFK7BA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB2552 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10130 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0 phishscore=0 malwarescore=0 bulkscore=0 mlxlogscore=999 mlxscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2109230001 definitions=main-2110070118 X-Proofpoint-ORIG-GUID: hhoGVzZ_e14Tb0pHbGJ8gUhQ--30tzDK X-Proofpoint-GUID: hhoGVzZ_e14Tb0pHbGJ8gUhQ--30tzDK X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 32FABD003E10 X-Stat-Signature: ypk7nb9f4p5xozw8yxxxt4k75ja6iayc Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=R8G2awi2; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b="L/4fXMgs"; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf20.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=mike.kravetz@oracle.com X-HE-Tag: 1633630794-783170 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Demote page functionality will split a huge page into a number of huge pages of a smaller size. For example, on x86 a 1GB huge page can be demoted into 512 2M huge pages. Demotion is done 'in place' by simply splitting the huge page. Added '*_for_demote' wrappers for remove_hugetlb_page, destroy_compound_gigantic_page and prep_compound_gigantic_page for use by demote code. Signed-off-by: Mike Kravetz Signed-off-by: Mike Kravetz Reviewed-by: Oscar Salvador --- mm/hugetlb.c | 82 +++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 74 insertions(+), 8 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 794e0c4c1b3c..22cb6126be7e 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1270,7 +1270,7 @@ static int hstate_next_node_to_free(struct hstate *h, nodemask_t *nodes_allowed) ((node = hstate_next_node_to_free(hs, mask)) || 1); \ nr_nodes--) -#ifdef CONFIG_ARCH_HAS_GIGANTIC_PAGE +/* used to demote non-gigantic_huge pages as well */ static void __destroy_compound_gigantic_page(struct page *page, unsigned int order, bool demote) { @@ -1293,6 +1293,13 @@ static void __destroy_compound_gigantic_page(struct page *page, __ClearPageHead(page); } +static void destroy_compound_hugetlb_page_for_demote(struct page *page, + unsigned int order) +{ + __destroy_compound_gigantic_page(page, order, true); +} + +#ifdef CONFIG_ARCH_HAS_GIGANTIC_PAGE static void destroy_compound_gigantic_page(struct page *page, unsigned int order) { @@ -1438,6 +1445,12 @@ static void remove_hugetlb_page(struct hstate *h, struct page *page, __remove_hugetlb_page(h, page, adjust_surplus, false); } +static void remove_hugetlb_page_for_demote(struct hstate *h, struct page *page, + bool adjust_surplus) +{ + __remove_hugetlb_page(h, page, adjust_surplus, true); +} + static void add_hugetlb_page(struct hstate *h, struct page *page, bool adjust_surplus) { @@ -1779,6 +1792,12 @@ static bool prep_compound_gigantic_page(struct page *page, unsigned int order) return __prep_compound_gigantic_page(page, order, false); } +static bool prep_compound_gigantic_page_for_demote(struct page *page, + unsigned int order) +{ + return __prep_compound_gigantic_page(page, order, true); +} + /* * PageHuge() only returns true for hugetlbfs pages, but not for normal or * transparent huge pages. See the PageTransHuge() documentation for more @@ -3304,9 +3323,54 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, return 0; } +static int demote_free_huge_page(struct hstate *h, struct page *page) +{ + int i, nid = page_to_nid(page); + struct hstate *target_hstate; + int rc = 0; + + target_hstate = size_to_hstate(PAGE_SIZE << h->demote_order); + + remove_hugetlb_page_for_demote(h, page, false); + spin_unlock_irq(&hugetlb_lock); + + rc = alloc_huge_page_vmemmap(h, page); + if (rc) { + /* Allocation of vmemmmap failed, we can not demote page */ + spin_lock_irq(&hugetlb_lock); + set_page_refcounted(page); + add_hugetlb_page(h, page, false); + return rc; + } + + /* + * Use destroy_compound_hugetlb_page_for_demote for all huge page + * sizes as it will not ref count pages. + */ + destroy_compound_hugetlb_page_for_demote(page, huge_page_order(h)); + + for (i = 0; i < pages_per_huge_page(h); + i += pages_per_huge_page(target_hstate)) { + if (hstate_is_gigantic(target_hstate)) + prep_compound_gigantic_page_for_demote(page + i, + target_hstate->order); + else + prep_compound_page(page + i, target_hstate->order); + set_page_private(page + i, 0); + set_page_refcounted(page + i); + prep_new_huge_page(target_hstate, page + i, nid); + put_page(page + i); + } + + spin_lock_irq(&hugetlb_lock); + return rc; +} + static int demote_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed) __must_hold(&hugetlb_lock) { + int nr_nodes, node; + struct page *page; int rc = 0; lockdep_assert_held(&hugetlb_lock); @@ -3317,9 +3381,15 @@ static int demote_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed) return -EINVAL; /* internal error */ } - /* - * TODO - demote fucntionality will be added in subsequent patch - */ + for_each_node_mask_to_free(h, nr_nodes, node, nodes_allowed) { + if (!list_empty(&h->hugepage_freelists[node])) { + page = list_entry(h->hugepage_freelists[node].next, + struct page, lru); + rc = demote_free_huge_page(h, page); + break; + } + } + return rc; } @@ -3554,10 +3624,6 @@ static ssize_t demote_store(struct kobject *kobj, /* * Check for available pages to demote each time thorough the * loop as demote_pool_huge_page will drop hugetlb_lock. - * - * NOTE: demote_pool_huge_page does not yet drop hugetlb_lock - * but will when full demote functionality is added in a later - * patch. */ if (nid != NUMA_NO_NODE) nr_available = h->free_huge_pages_node[nid];