From patchwork Mon Aug 16 22:49:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12439439 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A11B1C4338F for ; Mon, 16 Aug 2021 22:50:19 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2D4C160F35 for ; Mon, 16 Aug 2021 22:50:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 2D4C160F35 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id CAF6D6B0092; Mon, 16 Aug 2021 18:50:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C39428D0014; Mon, 16 Aug 2021 18:50:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A8BC56B0095; Mon, 16 Aug 2021 18:50:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0178.hostedemail.com [216.40.44.178]) by kanga.kvack.org (Postfix) with ESMTP id 8DDC66B0092 for ; Mon, 16 Aug 2021 18:50:18 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 25EBC824999B for ; Mon, 16 Aug 2021 22:50:18 +0000 (UTC) X-FDA: 78482438916.26.344D3EB Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf07.hostedemail.com (Postfix) with ESMTP id 87EDB1008A24 for ; Mon, 16 Aug 2021 22:50:17 +0000 (UTC) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GMm00K013214; Mon, 16 Aug 2021 22:50:12 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=+J7Zw52+JZpunur+6wO3AoVPGRNPuU6huCAySr+7gcI=; b=JklqlkKeoKslzO44l2dMHVa00X4A1xNoOCfP6Ib3l9ErveBw5Uydn7lFwZCinG7H/luL 8JarjC7n5UNecSM0uGwkwWvgQXow91BU1ImgtswaMSfxBp00zkd+DMvY4ZBfF17G6TAE +bA3q9fSfSYjLVs9GieXfDU9CGHuI/iXNgCZhR8U6WiBIYYnH3xWkngihZL2uKcKnFo3 +bq8Va2PEMdjPS4vQY8MZf1J+SZ7t6cxTI1yslR9NVLHEiTitjNzlWsJ3JE4yEnM4If0 GdjzSZRJPjiCWSa2NGYgMxH0mCRm2NFCH+nXS/TrnhbWMqUHu+7lEHmdyes7os6gurFl Fw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=+J7Zw52+JZpunur+6wO3AoVPGRNPuU6huCAySr+7gcI=; b=fe8KICibS7Qyns3Aiwp3VuVaAGX/TuzCs8KfeMcNRNK656oQCOn21oagctgOK3WBga3q ifTOGqihlifxlCb8eAV/XM2PHIfIN2445G/RdTFavIuHSWEtwSYy225EfHgct9ImMUg3 oGa4tA4O9REIHYN+/l1Phv+XmmzGmPIIVtfnXkH0qT1WzsH2E4KEQ+rTlawFlihIhzzz et+W91N0a+1lxPvNUu/5k+1jQzRwE8yhXPC37xCL1bzAKaWuEW2WLiJud4h/67xe+olz sgFdu2giuxh9JwQ54FEUOegCVDnpHHXyc8IpDag3qWOcWynOVzTOkN2yFrbrTDStYpZi bw== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by mx0b-00069f02.pphosted.com with ESMTP id 3afdbd2nd8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 22:50:11 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GMihdJ152058; Mon, 16 Aug 2021 22:50:10 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2168.outbound.protection.outlook.com [104.47.59.168]) by aserp3020.oracle.com with ESMTP id 3ae5n6fxx6-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 22:50:10 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=eFf14vFtHkZ48YIrwY7omcFo9zDsT6cAPaNgCtGFOrBzv+VgscJN2Md4vO56fvxXd/tNOLSFyymBkv0KFiJdAN3xLrOBwE+UTnbQnGuO1hmJUuCb8JgH8oQ9RfPWmiFImByqXzl34t3tY9yJIXJ1iGA37+2DXhU+tuIiaBoE7MmQ++IttWEwvsrZe6OaP7KmZvvkxOBTJRYGHP5cYp5172j0+veMRM3ymnhMDjfNWemyTr4aNflJcnuvSaw5NFAwFwH+AcCH6jfwUX+XU8wWfjU4Gzf61jpjFLRVIxcABYLzARBfWhqAsVek0Nv57stTX6l5ZxxgBT3ibu1q/a3u0A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+J7Zw52+JZpunur+6wO3AoVPGRNPuU6huCAySr+7gcI=; b=V6FH0Nxt/ccPmTihhTSSZU8ej4TGG+IJh/xfuuLaafOKyimE12M3kaQp6U3X84SM8qV+jjftHy4ukk1SvZ9Ovb15LDhoOj2dnhIwHY4+e1mL3lUKttUQBvwYi0Kg8Qnt04dHk1CxgzQy0u4leTASY88b7tKPPBX/GX6NM8JXn/WB5ulQOJSHKxJdCt8/CdQb0RndV8S4Uz3QdbAtpqVOIVg127DEMNLEQfL6ut0Xzg3l4K1v1PHAhOR28JBPqjK1gU7bNoenVAb2thdNF+ZDdT618Qsg3NyDx8Dyya6LefKX8Z8YnJisA7ak264zL7Mf2WXgxLhpS+hJQMZvl3f9DQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+J7Zw52+JZpunur+6wO3AoVPGRNPuU6huCAySr+7gcI=; b=V3cr7cCg4JdSv9ri/Wan5AsmpqAy7KMtZ1cOxyDF2uqaVRLs7fNOMCcxi9eFdSQI0F8xjRJFgv+X1ZPjeEbww/pZQpSn2lu0rew47Lea66DtqEN0aUjsehUUJlsBO0kZhH7lNIkqmrf6b5HBruJTt/35SU7XDIGP9ZE7rskFcU4= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3397.namprd10.prod.outlook.com (2603:10b6:a03:154::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.14; Mon, 16 Aug 2021 22:50:08 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 22:50:08 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Michal Hocko , Oscar Salvador , Zi Yan , Muchun Song , Naoya Horiguchi , David Rientjes , Andrew Morton , Mike Kravetz Subject: [PATCH 1/8] hugetlb: add demote hugetlb page sysfs interfaces Date: Mon, 16 Aug 2021 15:49:46 -0700 Message-Id: <20210816224953.157796-2-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210816224953.157796-1-mike.kravetz@oracle.com> References: <20210816224953.157796-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW3PR06CA0030.namprd06.prod.outlook.com (2603:10b6:303:2a::35) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MW3PR06CA0030.namprd06.prod.outlook.com (2603:10b6:303:2a::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.14 via Frontend Transport; Mon, 16 Aug 2021 22:50:07 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ac8491b0-c9a8-49ed-735c-08d96108327e X-MS-TrafficTypeDiagnostic: BYAPR10MB3397: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8882; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 6Fte0Drq8ij0Ec3wkDgKzHzPC7ZOiBokNhSoouyU+YW6ZjaX7bgPcGxsiXykwO05VX/62morB3VBLb9lA6v/5SU78S2S/cAS4ibdrJWJVyzjETx/w4mJ/2X84R0K1L/PrDUuVjrzbci1SiaURXhblOLS7dmYOfuzuoP5PlHaYepMyMpFybAEoa1+mAeAAlidCcvS1Go9PuPASV8+Afx8U8Iwy7XvDee88Nj0xs60+FGDvWPLqC/Ct8Fl/l8A6Owf1V2OesBtpTmHyD7fPKwrhWRerKEGeQQmS7FJ/ohF3gkjWuuORjHhRUpAUtmSn5RLBcQho5ZEfOpWvNKLUKD9VIPZUCprzWVstdgo8t1/krg1GXpsJ8Vh487dOPZFfT7LVKkfKCT487fC/rMufl4LyRTAmlIx6NoBMTXpqoXcZTFqpb6z21jj/moqxlU4rWznwOScS/XP8UG53Fit1lsKO1FaSKThXbehVwowUJgyFPLsfXKOqMjBYN1ODlKVBotlkotDvHdxUI4E5JTlXe4YdznRg9qSqh0hk+MSUmU+PjWXDqcNxJAIj3ETwx0xkLvli/PpN52vaeBbK4Wh1vqOVkQ2BMKBttHs3f+rWmNKRDyyoZlcw0D1FHsC5zdQePza63g2HD9RKJe78KgPAg/4N4xGTRLszmnVUL4KszptMqWJXTl7v5fhjs6oHoD36mGi5iRfIIwutOcYtK5on6jX0g== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(366004)(346002)(396003)(376002)(39860400002)(4326008)(54906003)(6486002)(956004)(2616005)(107886003)(44832011)(316002)(7696005)(8936002)(8676002)(52116002)(1076003)(83380400001)(38350700002)(38100700002)(2906002)(6666004)(86362001)(26005)(66556008)(66476007)(478600001)(186003)(36756003)(5660300002)(7416002)(66946007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: djeyfESZ5OGWzXrbaKGIn0hyWv4BcX1d+NSERlvhTidsTAXtAPZm1XJQpUEfSEWPj8p1OYG9UY+sdvWe7wWZ7YyYFzhrEWNlUvpcwifWOXkRMIL+xfLeNBPg2g5A++fmsWnup+pQSeXkfT1bELYl9oZpazCRoJKN0fRfNcIi4ZcXF3A5ZyfjVQV+dY5fK60L1m9rZju8ioXOWvEuM5kiHbC0dAor5UcSjEYnkQ0AqHY0oqDIMXyMDBycT9g7+R2Yijc7R12ySH1jJkFS3AbzZJh0iNMUhkhRnkcX+Og9ceTSqRXtsXcy91nQhiRszHmzC5TYk8cA4upIesKt7IP1YbCJ2mz06k2yigMzUZcakSuQsXb9fyNRaEwK7CWqYGd8gCFOxmz2ETMVQzK2ujFtd9h0zSjtvaHd3DhdLxq1I5Dqk8bo8aGMo+pPPzLU4cSzkpXy7UfLc7ALvS6jSVb92DENK2Vf99tOWi5P/5FcsEH+XB/qdDLuV6pAG18eJTVmVn5Q2bUBhrDaJsIbDLwJb6U5yw3863AXJuF+7PQwENnmWBN3lS43IWeW8E0EHE/sKxyyKHO8rppKIt2/gnsGXJXPrUu8HiLhgl4LD5hh2yrmEHq79Q9NBb1NVbOz0ORP5294dEA53BuaqqIN/QWKKmxqJahY6Mlr8nhlxKkyWcN00h/JB1mM8qC4kiM1VyxUzbag3CQTCDQYS/J+HNFii7pcCMVzxRJMDdHTEFSgw3lXpiJYCWQpoONDIK+ELNizKoI5wb7O4aU7cyJzQa3bpvkvklRKz7PxHYPtMiJpmkvaUPKMz5ZPVmDMCEO6S37atof+PC8NgL8AUdmB5rjEjVbBSaOsG9gyoSbM/REgQ8MA9Kce9VXzHFtbRSvTVzXejDz6+8Pee1Uspn6oWjkAAm1a9DFriLgS9SLH+nkWqS+a1RlRrjTR/XLuNXyKkS+wylEoN0zk5xWKWvoKjhBjVI7Te7D5ZYUX5+R7Z8/DEznLCV6ciU9PwWGvKz4kO5DuxPZ/a0nJNMqz0dI9JgnZfgOZxdT3Ry4tvNx5vJVvEemAnOK0DRzbBdU6Bkp1o4T+FV7aitKb5lfGCyVX0u1gpha9TOsG/vj73gGD/bERDDDOc6cjl5bFB1yV3VrlXOsOxEOCFYx40AGtNcaJrB2emQYK9p51uc9YX9DGJUbcUj//+jtN9l0/pnddwY2V7ozrQc9JgM4riM139eHYItYgcQOrSyu7BwRRT/Hr5wGcQo3eA4psme+PyaojMEFXbp707HlvcjMylnoVjiygig1C2Dafb9Inw3kOwyHXoW0AYy3oxIje36ZDwh6xNACwrxST X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: ac8491b0-c9a8-49ed-735c-08d96108327e X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 22:50:08.5800 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: JiC3ydqsdXAq6tv+VbhCpaRU6EqHQmoDK6B+pdul10ux/2kkUH99UJV/CoypXlWk0bck19/UME0PcZzU+tMRFQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3397 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 bulkscore=0 suspectscore=0 mlxlogscore=999 phishscore=0 mlxscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160142 X-Proofpoint-ORIG-GUID: MPAjqycOSwBiKR1tvLvLhbNlNnuUc7n7 X-Proofpoint-GUID: MPAjqycOSwBiKR1tvLvLhbNlNnuUc7n7 Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=JklqlkKe; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=fe8KICib; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=V3cr7cCg; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf07.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=mike.kravetz@oracle.com X-Stat-Signature: ru3jimofhfcnedia6r1f93zfdmy8wsru X-Rspamd-Queue-Id: 87EDB1008A24 X-Rspamd-Server: rspam05 X-HE-Tag: 1629154217-307951 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Two new sysfs files are added to demote hugtlb pages. These files are both per-hugetlb page size and per node. Files are: demote_size - The size in Kb that pages are demoted to. (read-only) demote - The number of huge pages to demote. (write-only) Writing a value to demote will result in an attempt to demote that number of hugetlb pages to an appropriate number of demote_size pages. This patch does not provide full demote functionality. It only provides the sysfs interfaces and uses existing code to free pages to the buddy allocator if demote_size == PAGESIZE. Signed-off-by: Mike Kravetz --- include/linux/hugetlb.h | 1 + mm/hugetlb.c | 121 +++++++++++++++++++++++++++++++++++++++- 2 files changed, 121 insertions(+), 1 deletion(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index f7ca1a3870ea..d96e11ce986c 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -596,6 +596,7 @@ struct hstate { int next_nid_to_alloc; int next_nid_to_free; unsigned int order; + unsigned int demote_order; unsigned long mask; unsigned long max_huge_pages; unsigned long nr_huge_pages; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 6337697f7ee4..0f16306993b3 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2986,7 +2986,7 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) static void __init hugetlb_init_hstates(void) { - struct hstate *h; + struct hstate *h, *h2; for_each_hstate(h) { if (minimum_order > huge_page_order(h)) @@ -2995,6 +2995,17 @@ static void __init hugetlb_init_hstates(void) /* oversize hugepages were init'ed in early boot */ if (!hstate_is_gigantic(h)) hugetlb_hstate_alloc_pages(h); + + /* + * Set demote order for each hstate. Note that + * h->demote_order is initially 0. + */ + for_each_hstate(h2) { + if (h2 == h) + continue; + if (h2->order < h->order && h2->order > h->demote_order) + h->demote_order = h2->order; + } } VM_BUG_ON(minimum_order == UINT_MAX); } @@ -3235,9 +3246,36 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, return 0; } +static int demote_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed) + __must_hold(&hugetlb_lock) +{ + int rc = 0; + + lockdep_assert_held(&hugetlb_lock); + /* If no demote order, free to buddy */ + if (!h->demote_order) { + struct page *page = remove_pool_huge_page(h, nodes_allowed, 0); + + if (!page) + return rc; + spin_unlock_irq(&hugetlb_lock); + update_and_free_page(h, page, false); + spin_lock_irq(&hugetlb_lock); + return 1; + } + + /* + * TODO - demote fucntionality will be added in subsequent patch + */ + return rc; +} + #define HSTATE_ATTR_RO(_name) \ static struct kobj_attribute _name##_attr = __ATTR_RO(_name) +#define HSTATE_ATTR_WO(_name) \ + static struct kobj_attribute _name##_attr = __ATTR_WO(_name) + #define HSTATE_ATTR(_name) \ static struct kobj_attribute _name##_attr = \ __ATTR(_name, 0644, _name##_show, _name##_store) @@ -3433,12 +3471,91 @@ static ssize_t surplus_hugepages_show(struct kobject *kobj, } HSTATE_ATTR_RO(surplus_hugepages); +static ssize_t demote_store(struct kobject *kobj, + struct kobj_attribute *attr, const char *buf, size_t len) +{ + unsigned long nr_demote; + unsigned long nr_available; + nodemask_t nodes_allowed, *n_mask; + struct hstate *h; + int err; + int nid; + + err = kstrtoul(buf, 10, &nr_demote); + if (err) + return err; + h = kobj_to_hstate(kobj, &nid); + + /* Synchronize with other sysfs operations modifying huge pages */ + mutex_lock(&h->resize_lock); + + spin_lock_irq(&hugetlb_lock); + if (nid != NUMA_NO_NODE) { + nr_available = h->free_huge_pages_node[nid]; + init_nodemask_of_node(&nodes_allowed, nid); + n_mask = &nodes_allowed; + } else { + nr_available = h->free_huge_pages; + n_mask = &node_states[N_MEMORY]; + } + nr_available -= h->resv_huge_pages; + if (nr_available <= 0) + goto out; + nr_demote = min(nr_available, nr_demote); + + while (nr_demote) { + if (!demote_pool_huge_page(h, n_mask)) + break; + + /* + * We may have dropped the lock in the routines to + * demote/free a page. Recompute nr_demote as counts could + * have changed and we want to make sure we do not demote + * a reserved huge page. + */ + nr_demote--; + if (nid != NUMA_NO_NODE) + nr_available = h->free_huge_pages_node[nid]; + else + nr_available = h->free_huge_pages; + nr_available -= h->resv_huge_pages; + if (nr_available <= 0) + nr_demote = 0; + else + nr_demote = min(nr_available, nr_demote); + } + +out: + spin_unlock_irq(&hugetlb_lock); + mutex_unlock(&h->resize_lock); + + return len; +} +HSTATE_ATTR_WO(demote); + +static ssize_t demote_size_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct hstate *h; + unsigned long demote_size; + int nid; + + h = kobj_to_hstate(kobj, &nid); + demote_size = h->demote_order; + + return sysfs_emit(buf, "%lukB\n", + (unsigned long)(PAGE_SIZE << h->demote_order) / SZ_1K); +} +HSTATE_ATTR_RO(demote_size); + static struct attribute *hstate_attrs[] = { &nr_hugepages_attr.attr, &nr_overcommit_hugepages_attr.attr, &free_hugepages_attr.attr, &resv_hugepages_attr.attr, &surplus_hugepages_attr.attr, + &demote_size_attr.attr, + &demote_attr.attr, #ifdef CONFIG_NUMA &nr_hugepages_mempolicy_attr.attr, #endif @@ -3508,6 +3625,8 @@ static struct attribute *per_node_hstate_attrs[] = { &nr_hugepages_attr.attr, &free_hugepages_attr.attr, &surplus_hugepages_attr.attr, + &demote_size_attr.attr, + &demote_attr.attr, NULL, }; From patchwork Mon Aug 16 22:49:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12439443 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74520C4338F for ; Mon, 16 Aug 2021 22:50:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 10AEE60F22 for ; Mon, 16 Aug 2021 22:50:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 10AEE60F22 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 647C48D0016; Mon, 16 Aug 2021 18:50:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5A8B78D0014; Mon, 16 Aug 2021 18:50:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3AD708D0016; Mon, 16 Aug 2021 18:50:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0094.hostedemail.com [216.40.44.94]) by kanga.kvack.org (Postfix) with ESMTP id 1D4AE8D0014 for ; Mon, 16 Aug 2021 18:50:22 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id AC74E182AD9AD for ; Mon, 16 Aug 2021 22:50:21 +0000 (UTC) X-FDA: 78482439042.05.DFE44F4 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf28.hostedemail.com (Postfix) with ESMTP id 28B1F9009F2A for ; Mon, 16 Aug 2021 22:50:20 +0000 (UTC) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GMmfrh026744; Mon, 16 Aug 2021 22:50:15 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=juKGQFig8fUy1uHmmnuGMRjyuaZcj6mUOPoOpkcnplk=; b=fDiT/w+duKfxzmJFw05kRsZjdGTu/MnOaXlOoGhNTgD1aGcceBQjWs/gGLOSa3+E4Pxl 7+DKbJipCmuOtHfCHjAi3gaDimJKtvxiEIZ0bvlEBhLGrzkesEK8ng/vIXaN4rvpIcvJ 4u9VkFWE/hfbZNycYrfAPDOEPaoPTy381dTum1T+rPoD82tk85HnQ+TqTCoR6LH/+DaC QuXWo/12dMCjYY5KgsFdzbTlzOSltoJhi1hP4/o1G7NOC1of5g2Q1AAwz4VuxXl5xoOy VgOnhCQXiFm2+xOt10ZYHQU+szGVsF6aJHKOc1s0f3HXThFIuF95plkheusuIbBQRx6I LA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=juKGQFig8fUy1uHmmnuGMRjyuaZcj6mUOPoOpkcnplk=; b=YofYpZ2jf9Floc3dgsSf88VxFVKlYyoWbj6J1jPZfyZ4TUwAAGN3zuiR/NRcwZ3CA7yD cMznG+ixPc3pedyeojvKBMmQgy9Twukhqj2kmtldk674G/QUIqL0ZOQ1r1wPJFqy3JFV pxRhtjeAO1RL4EoO0zs+XuC3RceNbSNKyWH5yUKcydYLYkUjUbMKhFZV1/WRsL2H22HT TCqIfaE30iFqi3ZQXr4Lu8e0ufbD7HgmJc6Q4lZCPAruin9Fk79SNae3Jz3CA+M0lwWs EtO3rKbHnHBUulc3cJH1i2Fx1kRpaG2IdFBsxz65jVsiyUyqtGfgyZCo7RFsaXuVKHoE 8Q== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 3af830322x-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 22:50:15 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GMiu6H040104; Mon, 16 Aug 2021 22:50:14 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2172.outbound.protection.outlook.com [104.47.59.172]) by userp3030.oracle.com with ESMTP id 3ae2xy13wd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 22:50:14 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DwRkPl86gejsP/mOAG+yRELt2FyhnoDzd1fikFwzInqr+WSi8hPx1wvByqrw/70uPtmb9ukk5nJ0aFfE2LI9iHBdMrPUkXDAijCWtvBtBfGnmsz40hBiFQB5h9DhB4Kb7suJ8oUXvfaq4X4lakzlR1hU3PyQmGFI+NnGdceLzOTsTdYkuDFwbY7CO/+52SA7tfy8lvT6HkR/OwIf1Bkc9eU4FX6OYQ0sVfxGtQrdm+9dH1Mukl4dnQV/vKV2GaMQctmmKZJmuIIwKPfU3jakxpMDphk4+mO+HqpbA6fn8imvSTbrn860GQ4ewj1idgTLzjW0MsPyBKH60SffwpLgYw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=juKGQFig8fUy1uHmmnuGMRjyuaZcj6mUOPoOpkcnplk=; b=Geo6ayWz7Ihd35Rx67BN399zJF4bMtK1SQe0OIvo4K+N4KzmMsYOrMuySDbm6VQBBmshQZjoYh7QWXhYGMHByhcIydY2P/9rUU+1+wNB8kn4Ya3OBWPz9GNid/DlMQaS9De3CUotBVX5zkikQ1tmttPextDY2Ry23Bg3A4mjCY1bnyIx529aK1coKCNjPrw3cZUyiDyfrx3bP2DeHgDU/sHHpPP2SqdyaOG0H/1ynI9hv+2dRnMWx06O9G/w+uVDaX8/v7v2Nczu7LJ6lK+tmrIDg7v3NIYAxOPSzfuknziYqzsH3TbjZIcUpYyidq38wFYpFdDFvYN/x89RR65F5w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=juKGQFig8fUy1uHmmnuGMRjyuaZcj6mUOPoOpkcnplk=; b=MBCwc0HC6ZIAyMFcxh6SmQ8uxBa6TltpOCZmpEuUoS8u8rBqP6CU+5cKE/glbvNFxtmW/+57o2au8jdbrO3sztP0GLYL3+bV06i08XhRXwfp0QUYV5cAktWGlnqGncE098DYYCggoT1vOh8VNayiw0XKe7YHuVwVrkR2cP3HCIU= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3397.namprd10.prod.outlook.com (2603:10b6:a03:154::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.14; Mon, 16 Aug 2021 22:50:10 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 22:50:10 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Michal Hocko , Oscar Salvador , Zi Yan , Muchun Song , Naoya Horiguchi , David Rientjes , Andrew Morton , Mike Kravetz Subject: [PATCH 2/8] hugetlb: add HPageCma flag and code to free non-gigantic pages in CMA Date: Mon, 16 Aug 2021 15:49:47 -0700 Message-Id: <20210816224953.157796-3-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210816224953.157796-1-mike.kravetz@oracle.com> References: <20210816224953.157796-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW3PR06CA0030.namprd06.prod.outlook.com (2603:10b6:303:2a::35) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MW3PR06CA0030.namprd06.prod.outlook.com (2603:10b6:303:2a::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.14 via Frontend Transport; Mon, 16 Aug 2021 22:50:09 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: eb809ec4-95a5-4853-9b36-08d96108339b X-MS-TrafficTypeDiagnostic: BYAPR10MB3397: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8882; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: QVqq88YllSJtC+gGUESh/fv04nmxk+YNsvC05KC8wRQz4d+z8q1+URyvJ8u0ZSVups6BuFU9nk+6AZIu8Y7WM8jNOHRWL8ER1QbsQ4Gr5iG/Nbdxzqfbh7kZ+trGI6S0jxWauiB0klDnmwl5O566n33TliidAeFkzFF36FDN4dFx1G19He1KPjcjBNdA7bIpJgZG2wCBYcPNQNo5r/ke6Yti3EIskhUZOHPUZYJUYIlsB4tp2yhblfJJr1MJC/jbv2FRWRxvE6gJExJhhB/ef8nyIR01RLaoLSRZifb9AL4SIUG/oIPgmqW4ZbHQHTwh1AI3xAI/K+FlLKfOE95GD3Zjaxk95nhqf5Tbj0xqpNMqoKPLj4ucYOGhuuJ7ZBzHFucSkxpHuMmAZOe/yGlg8/xtVX1owWvoBH0tguLnrL4OZDSLcbx08SSjgQf1Pd1ap14CNrvOXQ+eAWy9Y9qqc2rJutGujs364KM+yyiTN53i1jLBPFyQ6h8XBcnHoRR2dLSfebowpAbbAWTFitVJKB+cGQy18FrkSSSZfJ02rUghJ7JKM1MVEAMP1EF0FD90mvnoRfoBEiWSbQ12ZNB8tcB2TptY6JqVEg0c8BVxC+ZUhn+QDn5bDxYhCyE97LnttzrkUgN4PL1fV0b3JhqWd6pxbWuv8q6pSNMXVZtmSJxlxvInP1DDDiCaSJlEZuKVKeL3EUl7IAbT2XpVror5Sg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(366004)(346002)(396003)(376002)(39860400002)(4326008)(54906003)(6486002)(956004)(2616005)(107886003)(44832011)(316002)(7696005)(8936002)(8676002)(52116002)(1076003)(83380400001)(38350700002)(38100700002)(2906002)(6666004)(86362001)(26005)(66556008)(66476007)(478600001)(186003)(36756003)(5660300002)(7416002)(66946007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: WKscPD08VhHhd9fFzezPG1Usy+Eu+7QZe3BOtFnI/uolCufdMK7yuoF66C/K05QCC3g+HnWJCFTYvS6wdaop+jI0H1+dDAkj2pZadz4Lje1XEgcUjsH6ecLxew4E5lW+CztWRtm7sQ6DzOOgdO9D9m0Q//sVd/FGRB6uk0SXigxEAHxDGQ412FEqiTOzjQQk1RaHk00uFvuWZgtC8imBmtW0/28UJJq/Rw7IOxN8XMnjc+ulyx1CD2XZqrmsNzuEx63qUHdWoJ2qjlew/wMJMHJd/UQNY7s0izWyNYJ03OKEdq3Idto5a+/JN1/ruMdKcGRHtyeXBGCEgzCcujyIHWy4cpFKdYIy/anhuFftbpYi7zrZeck5U8liCtOBQPTjT98Bfmb1+d7Hfn566zIrrZfYb8xrt/5N1WgL/dpPWJOIKOVZvUXhewmA37lePrVDMbW3AL7i7x7b+hOJI1bBAq6Og9LHn0zFQfzqdF/UHEZiv9gPp5gOVzZGyNlhomEIUU71xsZURZt4ukYfVHO21UWuuG1JRvNJ57Xp2kQNh0WiOIKC+6C6drELGz7vbM7JwjRdd6b4e6uX+GvVOAHWnXpRCtWf3Hg8WvA7g7v+SazAX2mJnxZFtDuct83/ZnhdM51Y8DJWQG+Bac1gMh5hXPjpwS51nzOjD5OwBXfe3Rzv40ydoTftXp6veo2gPpXW0wN/YOFzocA72QgSmzhlWDqnjsbL2QEabH7j8Sl+9nVC+svhj7FatUCyq9EowFdM15rJpf8v5VAYB5PCR1kh0D4vDBZyjJs2re35kczSKqmr1I6UATAVAovj4Pfo8l9wndfA8N6h7cEjZNqm52hSh4Z/P80rPLfmZwUekmThjDyGEIg82241Avo/liDutoJG6UXNDnNsLzZup/yvbYBMmoaRRV2x9UtQhUEu7Pk3vynh1dFX0nn251NALWjs+/cMJaxQuTNdziM8fW9ThEaSyC8+S8/8ymzTG7AEmSTEzDG/KETZs2w6LOzP0ddn2GRJyKF4w/71rY+NPvHbl82aegvsXpqI6exA06D3VSLwbBdIvni1VDbMXp9FXznL8pLqEJ9CrBeUvBMAh/ttk0fOnwskHC82BQZyw5t8Bc1ju00z7XXfrMP1aFoz1kGpHlBwyW4r7JR6Fk3vJMf55tEwv+RGRQk4WRlZI7rg8SoCUw2IJ4UxqbbtYtfhvUdHF78PknlKGT7mw3wpcQ27SJ8UhNgURHV/2PshTUYQnQS4S8oAwu7fd6BanDg8Qjv05qePXn1Tp6dCr7cEWaEoTBsp+E6G8KYW7Ri2EKTitgu6O2J0bDpIZMSVhUv1BEl83wEA X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: eb809ec4-95a5-4853-9b36-08d96108339b X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 22:50:10.5453 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: h9TZeCp7RxkIGe6LmyyfOZ41yhmbWpmwkdAAiFcBDVkls3LaBToPXsFP+t3BUNPpECmOYzTIPIovM+9kSB8pog== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3397 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 spamscore=0 adultscore=0 suspectscore=0 phishscore=0 mlxlogscore=999 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160142 X-Proofpoint-ORIG-GUID: ZtD13qfB-bA9x4zeCv2VKMjSS1joOJWt X-Proofpoint-GUID: ZtD13qfB-bA9x4zeCv2VKMjSS1joOJWt Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b="fDiT/w+d"; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=YofYpZ2j; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=MBCwc0HC; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf28.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=mike.kravetz@oracle.com X-Stat-Signature: hpdx9wyb478rcw5j56896jm17co5ek7f X-Rspamd-Queue-Id: 28B1F9009F2A X-Rspamd-Server: rspam05 X-HE-Tag: 1629154220-190698 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When huge page demotion is fully implemented, gigantic pages can be demoted to a smaller huge page size. For example, on x86 a 1G page can be demoted to 512 2M pages. However, gigantic pages can potentially be allocated from CMA. If a gigantic page which was allocated from CMA is demoted, the corresponding demoted pages needs to be returned to CMA. In order to track hugetlb pages that need to be returned to CMA, add the hugetlb specific flag HPageCma. Flag is set when a huge page is allocated from CMA and transferred to any demoted pages. Non-gigantic huge page freeing code checks for the flag and takes appropriate action. This also requires a change to CMA reservations for gigantic pages. Currently, the 'order_per_bit' is set to the gigantic page size. However, if gigantic pages can be demoted this needs to be set to the order of the smallest huge page. At CMA reservation time we do not know the size of the smallest huge page size, so use HUGETLB_PAGE_ORDER. Also, prohibit demotion to huge page sizes smaller than HUGETLB_PAGE_ORDER. Signed-off-by: Mike Kravetz --- include/linux/hugetlb.h | 7 +++++++ mm/hugetlb.c | 46 ++++++++++++++++++++++++++++++----------- 2 files changed, 41 insertions(+), 12 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index d96e11ce986c..60aa7e9fe2b9 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -533,6 +533,11 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr, * HPG_freed - Set when page is on the free lists. * Synchronization: hugetlb_lock held for examination and modification. * HPG_vmemmap_optimized - Set when the vmemmap pages of the page are freed. + * HPG_cma - Set if huge page was directly allocated from CMA area via + * cma_alloc. Initially set for gigantic page cma allocations, but can + * be set in non-gigantic pages if gigantic pages are demoted. + * Synchronization: Only accessed or modified when there is only one + * reference to the page at allocation, free or demote time. */ enum hugetlb_page_flags { HPG_restore_reserve = 0, @@ -540,6 +545,7 @@ enum hugetlb_page_flags { HPG_temporary, HPG_freed, HPG_vmemmap_optimized, + HPG_cma, __NR_HPAGEFLAGS, }; @@ -586,6 +592,7 @@ HPAGEFLAG(Migratable, migratable) HPAGEFLAG(Temporary, temporary) HPAGEFLAG(Freed, freed) HPAGEFLAG(VmemmapOptimized, vmemmap_optimized) +HPAGEFLAG(Cma, cma) #ifdef CONFIG_HUGETLB_PAGE diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 0f16306993b3..47b4b4d1a8f9 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1272,6 +1272,7 @@ static void destroy_compound_gigantic_page(struct page *page, atomic_set(compound_pincount_ptr(page), 0); for (i = 1; i < nr_pages; i++, p = mem_map_next(p, page, i)) { + p->mapping = NULL; clear_compound_head(p); set_page_refcounted(p); } @@ -1283,16 +1284,12 @@ static void destroy_compound_gigantic_page(struct page *page, static void free_gigantic_page(struct page *page, unsigned int order) { - /* - * If the page isn't allocated using the cma allocator, - * cma_release() returns false. - */ #ifdef CONFIG_CMA - if (cma_release(hugetlb_cma[page_to_nid(page)], page, 1 << order)) - return; + if (HPageCma(page)) + cma_release(hugetlb_cma[page_to_nid(page)], page, 1 << order); + else #endif - - free_contig_range(page_to_pfn(page), 1 << order); + free_contig_range(page_to_pfn(page), 1 << order); } #ifdef CONFIG_CONTIG_ALLOC @@ -1311,8 +1308,10 @@ static struct page *alloc_gigantic_page(struct hstate *h, gfp_t gfp_mask, if (hugetlb_cma[nid]) { page = cma_alloc(hugetlb_cma[nid], nr_pages, huge_page_order(h), true); - if (page) + if (page) { + SetHPageCma(page); return page; + } } if (!(gfp_mask & __GFP_THISNODE)) { @@ -1322,8 +1321,10 @@ static struct page *alloc_gigantic_page(struct hstate *h, gfp_t gfp_mask, page = cma_alloc(hugetlb_cma[node], nr_pages, huge_page_order(h), true); - if (page) + if (page) { + SetHPageCma(page); return page; + } } } } @@ -1480,6 +1481,20 @@ static void __update_and_free_page(struct hstate *h, struct page *page) destroy_compound_gigantic_page(page, huge_page_order(h)); free_gigantic_page(page, huge_page_order(h)); } else { +#ifdef CONFIG_CMA + /* + * Could be a page that was demoted from a gigantic page + * which was allocated in a CMA area. + */ + if (HPageCma(page)) { + destroy_compound_gigantic_page(page, + huge_page_order(h)); + if (!cma_release(hugetlb_cma[page_to_nid(page)], page, + 1 << huge_page_order(h))) + VM_BUG_ON_PAGE(1, page); + return; + } +#endif __free_pages(page, huge_page_order(h)); } } @@ -3003,7 +3018,8 @@ static void __init hugetlb_init_hstates(void) for_each_hstate(h2) { if (h2 == h) continue; - if (h2->order < h->order && h2->order > h->demote_order) + if (h2->order >= HUGETLB_PAGE_ORDER && + h2->order < h->order && h2->order > h->demote_order) h->demote_order = h2->order; } } @@ -6518,7 +6534,13 @@ void __init hugetlb_cma_reserve(int order) size = round_up(size, PAGE_SIZE << order); snprintf(name, sizeof(name), "hugetlb%d", nid); - res = cma_declare_contiguous_nid(0, size, 0, PAGE_SIZE << order, + /* + * Note that 'order per bit' is based on smallest size that + * may be returned to CMA allocator in the case of + * huge page demotion. + */ + res = cma_declare_contiguous_nid(0, size, 0, + PAGE_SIZE << HUGETLB_PAGE_ORDER, 0, false, name, &hugetlb_cma[nid], nid); if (res) { From patchwork Mon Aug 16 22:49:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12439441 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AAEB9C432BE for ; Mon, 16 Aug 2021 22:50:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 286C460C3F for ; Mon, 16 Aug 2021 22:50:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 286C460C3F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id C3A6C8D0015; Mon, 16 Aug 2021 18:50:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B9F0C8D0014; Mon, 16 Aug 2021 18:50:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A0E18D0015; Mon, 16 Aug 2021 18:50:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0204.hostedemail.com [216.40.44.204]) by kanga.kvack.org (Postfix) with ESMTP id 7A6368D0014 for ; Mon, 16 Aug 2021 18:50:20 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 2DD832195D for ; Mon, 16 Aug 2021 22:50:20 +0000 (UTC) X-FDA: 78482439000.15.3D9FE4F Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf05.hostedemail.com (Postfix) with ESMTP id 9532A5044C8F for ; Mon, 16 Aug 2021 22:50:19 +0000 (UTC) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GMllYs031286; Mon, 16 Aug 2021 22:50:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=cNHjsKEETFPtt238FV1F88IbK5MVdUjYRpFBkGQe7mM=; b=PJZFu1rnUCUcwtPQMvAulSh3Yqp7U6E4kxqwiTn2//fL3Mvh4SwK8dh27+telaWDVlgY 5MbE5ciJiDUZ3kv/jzMNgW0rceqi/soLMaIGKkA1o6oKHX2OQzlPa4KKpe3YTdDFpT7O aA1Z0FNqCRyeqUb88WgCwUkOdTfEptOxHmy+iHnraZtFJI+YTnSma9X6iFnR/h50p5FT IWG3ZO1AtwStkuP3SiLIJuEVhv/1JjTPcbJ6198ApwuWMxlGTPR5u+PoJexTS4UBc3HX qC0L904tOxxH/ws5k4ZmeUSaZR1uxTZ9kW7Di/tyZr0HM3m9WEwfCtrwZ8dU/UxNUipU TA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=cNHjsKEETFPtt238FV1F88IbK5MVdUjYRpFBkGQe7mM=; b=E79QKqgnScqeX5r72k3r0E+MKjTz8BoST+5XXZ2UG312ECXN+XhhSW2HRRC7AfGFIkXm a3+DT5Yz/PDjQPzA7OsOczIYqojsnY3iQno1A0SJgy8LFIZ1wbKVVc10hQYKyk984xVT mWULhHd7bULie2kBlWkOnFNpR7UIGw3RUm3/Jnsc0Vxc91VL8RIGswkp9nHGPLyUNy3J qJleU0+uwRDDWm1mZjRAF+EcWz7/MHi0zjRiUVTUDm0UCNSm0jPbcz4QzfUC62e55Cza r8c69N787+M46UbObE86L48C0V891i9DrRYESFh6iX3kN/SIZIcIyc8ny15soHHu8Ba7 Uw== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 3afgpgjb8b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 22:50:17 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GMiu6I040104; Mon, 16 Aug 2021 22:50:15 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2172.outbound.protection.outlook.com [104.47.59.172]) by userp3030.oracle.com with ESMTP id 3ae2xy13wd-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 22:50:14 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kFFzBn/KtJcnc/eFyG9y2apOzBCk/PzqgipIHfxu0zTUad+/K3AATSv4HKH42t45bmIOPMWkGzJr59wmd6bdwoJeHzxfueZ8GhWqyl/ntP8GFLU94fYPcCf/Xd06HWrhV2b8aCtXco9SRr0jToMMmiiz0n8S42DDHPJm5a9nTIWEDIUzQMR2qOvklRFexZrAUP/uJPJoGBilAO/qtgj1m5aWlNvAdOocWqtgbntuUxbDfWrjdDUS+8Fn3nTaEoJhWb0XbaHXYxIxJr+mbG+LMmnZKsZ3ckj0zxOA9rXOIBuBMpWo0qip8SSR5JIQXpMnCA33VAFhYAIp9L8G8Io/uw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cNHjsKEETFPtt238FV1F88IbK5MVdUjYRpFBkGQe7mM=; b=EpRJHaLDb/TzQVU03ERv2I6CpghJu+rTygdfXgZ/OHCeHHOzUGNCozN5Ae3zqDOAvxa6YEMPIiHP5oRvLAcv1eQl6Q/wDEd0ogEdHqKd7yuB06tKTZwHO72fR6Wn1QG+ieLX6RtyrBb/wVl0AKp9IK/M0eK0Fej5p4PLjOaxd89DkvqcWxMS/wXsO/5aleLEzS+nN5cPp9GG3+48KvVPkHByXkNOSdze2yx17fc6fCVqEaf9X0x3zlQiki2Ze4czELK+5re9wG2UrAS/Rrx4SKkvEXlKpsO+w7nU8SSm80Ge0rvcoYKewddIHKW4XV886pWhKtVvlRi4WUNWXrW3lw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cNHjsKEETFPtt238FV1F88IbK5MVdUjYRpFBkGQe7mM=; b=DmPL7HXenVxaO2gJ0BUIP07SkKG1IyW03/vWSK0xt4KW+EXZlDMgMJopOFxQCmLw+h1IXjVMmlc7syysxe8KtAfOrctouFRAqaJ4ATntbHTtqVvac01ihD9WBUcsIzjWZcLCGexTVRRV8qWd88vquYMS+6RaEKdMJAg4Fxzcqjc= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3397.namprd10.prod.outlook.com (2603:10b6:a03:154::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.14; Mon, 16 Aug 2021 22:50:12 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 22:50:12 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Michal Hocko , Oscar Salvador , Zi Yan , Muchun Song , Naoya Horiguchi , David Rientjes , Andrew Morton , Mike Kravetz Subject: [PATCH 3/8] hugetlb: add demote bool to gigantic page routines Date: Mon, 16 Aug 2021 15:49:48 -0700 Message-Id: <20210816224953.157796-4-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210816224953.157796-1-mike.kravetz@oracle.com> References: <20210816224953.157796-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW3PR06CA0030.namprd06.prod.outlook.com (2603:10b6:303:2a::35) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MW3PR06CA0030.namprd06.prod.outlook.com (2603:10b6:303:2a::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.14 via Frontend Transport; Mon, 16 Aug 2021 22:50:11 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 0e1fe214-46a1-47e2-8211-08d9610834b4 X-MS-TrafficTypeDiagnostic: BYAPR10MB3397: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: AmeihMVwqPB4M6deD4qBJIU1KyOgwyoWbMbHQL4BKuT2xi701YO++HFLgYXXs7i1egnMUL69FUdrjwhUOhPjJ0UQx8imfWjp5S2las68hJCMxRirZewMqsindl6U9vZbioUtBrl4/7Jpf+85y/DpFvjFsjgN98x3wyrUYkwdislsPVTX/X2kTgAT2nYzKLlwePzyqX62p7mMvjE4ygdZ0vWm3UJTP8jb4kRICpZB3Euoh815C2S3V2Td8makyxcGgbIfazYfDykur6amenJT1g+pnoJ4Xb/5wYldIWoqaYNV3k+XaZiPdD6E7BEx+0+zXp1oDNSIerQ5ZTKeVNfOW6kpyGGK1cNXmdb0mgvqqHBeUmsvoRjUb38Lj8pi5z/pwOngJI42fvxvTys3qORS7gNErgfvSl1SXBDHFSV8NLyC4PJC/SRYgLr4BJuYWpaijasXoDruM6r7bagqQK0b+ejeLbY7zU2dnuxOxKsZM+l/moertPd0kanFVT4ju/nvrfdSX2fVIa/AE2CMyNIim0M2Sey/VPEjDKpULk6P3/Y5uStq9iX4+7tTXK67tspBDKxP0rRCyDRBsyWgMx0+9+wE80GS4tKKJIFATS+uUsOXXBscFZxR16CYrm3H+yrlA/wFpP3s09EgXgZRitmNrnbWvUscAS3XwnP2fATlAvbfypUpxUkge07De2HPlx1UEHPemJ1ahFwMRdkW8VOfmw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(366004)(346002)(396003)(376002)(39860400002)(4326008)(54906003)(6486002)(956004)(2616005)(107886003)(44832011)(316002)(7696005)(8936002)(8676002)(52116002)(1076003)(83380400001)(38350700002)(38100700002)(2906002)(6666004)(86362001)(26005)(66556008)(66476007)(478600001)(186003)(36756003)(5660300002)(7416002)(66946007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: PTW3jsx2INRAWbKO/G4vosir8kd4lfwSO6to2g/MR7Gmryhw6ydICGrF8krkhLCHEUS00p/TOHbdL846LL7uG6jtZFwknATTgIKcYgQ1TXXo+n7SE6JgMRJYm9oEix8XhNdwONKnwwkBg4oesKLWBZPujJp3ItHqCGDsqMyDlpbq3qC6dtsQJWlR4N84gNTWiHa95VrEEl2Qo6kAGsTHspac8yi1alLnryJrQ0ED0aNBn4uxA82mAYj+eVWtydl5V6hh1iqGBiCvaQSxetYnFd5+YXQpUB0gH4yGW7OWkQG23c+l9mEq/HE3+f8b1sIQtsuTi0sedYjXrQJiyvaNaapp9SYKhCQNriB7G9d0HaG9QtP4V3qJHcDB0u6JdZHcGTCFhKKAq41gM58XaAJaZH1EQEQJCwFdTJkGYeJRlT5bbuUWm10dlFkmw9IbdWYKk6HkR936M5muJ09Utc0pO/hyDF145KU3KKfyG2AXBEUghPGsfeGdpxIG3WGf9gZDX5BtO2avJQK08LfUksHpitWQXf8hJbydSVUqTD0JKxyVl0cU+zGQ7XZ40m4P4SK3Pm4hObC+WDafs2apO6EjWPf8hMCAf70DxhFaHlcRMdsUKEkBVqQBFxudmTPrB0s76YQza9dRmGgZAD1X+2pyFnGuW3x0jjczbOgOh8SowblMnaUQjfJXufBScJzWSIJZp8MhOxnNJSHbY93J7FBybpHLhY9W48qoD773+ATnzOYQGCFB0X/mnLHd9OlzsVCQcU+4nI963U1ewPzDhFu+C2HhMXd1N0a/UG8crmAFxb/gg3SpxXHkiq6aeWQGy1BagtbRfm54G4E+az+ZGCRDe5hC2yV7lF6Rz5b88w0715TNVSFhk2fOqNMNWSx7niPRJpdRGTMsmH/4Mr+Ua9R2BegMXa5ewtAKja6iV5VeLFAssoqH/uE/lXTXHWpDHBhwSbzOildlmU2WxCXjKcjAY4w3RBYJVgHCvZ8agNL0ZQs0pLSxPd+Tip3+prg0ni2y4VO4aPYahbwgBuImLN6h9jzsLk0bfWR3YlibkUTRcDDgExPMEdFhqblaZe+vuEBd6QeJKMdFD2NkYsayKySyMfHwcpnuN5EReb3i4UbPtMoN4PiRGmBfVjkpfRlOBmku+zUPw9AHqO1iLJjuD5SdLo2W0tasIY9KnpPwbZeSqbyozkSZoCaxorzwOdhPwMq3/Wgzl6XPcHNx23ZqKlEKs9aKA/k7Vxh9ZuTOIRkwZPiZd+WdfO63M/SkhXBa3SPYeg/vTjSrifbD7LoA3mfbfwO9stNwqqr5XKc1rCbzqUiRfuU/W4BqYi4WocpDRaY/ X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0e1fe214-46a1-47e2-8211-08d9610834b4 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 22:50:12.2937 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: P8B1UsXFm4DRf4b8s4ZAE85oMMBVWzUTKM1WE/OcUr/ZxZT2BTH76Ce8/c8XxPSNj5aGW7twTYEuH/yasRvaGA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3397 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 spamscore=0 adultscore=0 suspectscore=0 phishscore=0 mlxlogscore=999 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160142 X-Proofpoint-GUID: ZDqPjs5Lb4l7sSJ7rYUm0iqv7m6VnApE X-Proofpoint-ORIG-GUID: ZDqPjs5Lb4l7sSJ7rYUm0iqv7m6VnApE X-Rspamd-Queue-Id: 9532A5044C8F Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=PJZFu1rn; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=E79QKqgn; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=DmPL7HXe; spf=none (imf05.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam01 X-Stat-Signature: ew7bu8egnx7bqy7bf8hkj1dwmqocxkf5 X-HE-Tag: 1629154219-791444 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The routines remove_hugetlb_page and destroy_compound_gigantic_page will remove a gigantic page and make the set of base pages ready to be returned to a lower level allocator. In the process of doing this, they make all base pages reference counted. The routine prep_compound_gigantic_page creates a gigantic page from a set of base pages. It assumes that all these base pages are reference counted. During demotion, a gigantic page will be split into huge pages of a smaller size. This logically involves use of the routines, remove_hugetlb_page, and destroy_compound_gigantic_page followed by prep_compound*_page for each smaller huge page. When pages are reference counted (ref count >= 0), additional speculative ref counts could be taken. This could result in errors while demoting a huge page. Quite a bit of code would need to be created to handle all possible issues. Instead of dealing with the possibility of speculative ref counts, avoid the possibility by keeping ref counts at zero during the demote process. Add a boolean 'demote' to the routines remove_hugetlb_page, destroy_compound_gigantic_page and prep_compound_gigantic_page. If the boolean is set, the remove and destroy routines will not reference count pages and the prep routine will not expect reference counted pages. '*_for_demote' wrappers of the routines will be added in a subsequent patch where this functionality is used. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 54 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 43 insertions(+), 11 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 47b4b4d1a8f9..2f2d5002fe73 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1261,8 +1261,8 @@ static int hstate_next_node_to_free(struct hstate *h, nodemask_t *nodes_allowed) nr_nodes--) #ifdef CONFIG_ARCH_HAS_GIGANTIC_PAGE -static void destroy_compound_gigantic_page(struct page *page, - unsigned int order) +static void __destroy_compound_gigantic_page(struct page *page, + unsigned int order, bool demote) { int i; int nr_pages = 1 << order; @@ -1274,7 +1274,8 @@ static void destroy_compound_gigantic_page(struct page *page, for (i = 1; i < nr_pages; i++, p = mem_map_next(p, page, i)) { p->mapping = NULL; clear_compound_head(p); - set_page_refcounted(p); + if (!demote) + set_page_refcounted(p); } set_compound_order(page, 0); @@ -1282,6 +1283,12 @@ static void destroy_compound_gigantic_page(struct page *page, __ClearPageHead(page); } +static void destroy_compound_gigantic_page(struct page *page, + unsigned int order) +{ + __destroy_compound_gigantic_page(page, order, false); +} + static void free_gigantic_page(struct page *page, unsigned int order) { #ifdef CONFIG_CMA @@ -1354,12 +1361,15 @@ static inline void destroy_compound_gigantic_page(struct page *page, /* * Remove hugetlb page from lists, and update dtor so that page appears - * as just a compound page. A reference is held on the page. + * as just a compound page. + * + * A reference is held on the page, except in the case of demote. * * Must be called with hugetlb lock held. */ -static void remove_hugetlb_page(struct hstate *h, struct page *page, - bool adjust_surplus) +static void __remove_hugetlb_page(struct hstate *h, struct page *page, + bool adjust_surplus, + bool demote) { int nid = page_to_nid(page); @@ -1397,8 +1407,12 @@ static void remove_hugetlb_page(struct hstate *h, struct page *page, * * This handles the case where more than one ref is held when and * after update_and_free_page is called. + * + * In the case of demote we do not ref count the page as it will soon + * be turned into a page of smaller size. */ - set_page_refcounted(page); + if (!demote) + set_page_refcounted(page); if (hstate_is_gigantic(h)) set_compound_page_dtor(page, NULL_COMPOUND_DTOR); else @@ -1408,6 +1422,12 @@ static void remove_hugetlb_page(struct hstate *h, struct page *page, h->nr_huge_pages_node[nid]--; } +static void remove_hugetlb_page(struct hstate *h, struct page *page, + bool adjust_surplus) +{ + __remove_hugetlb_page(h, page, adjust_surplus, false); +} + static void add_hugetlb_page(struct hstate *h, struct page *page, bool adjust_surplus) { @@ -1679,7 +1699,8 @@ static void prep_new_huge_page(struct hstate *h, struct page *page, int nid) spin_unlock_irq(&hugetlb_lock); } -static bool prep_compound_gigantic_page(struct page *page, unsigned int order) +static bool __prep_compound_gigantic_page(struct page *page, unsigned int order, + bool demote) { int i, j; int nr_pages = 1 << order; @@ -1717,10 +1738,16 @@ static bool prep_compound_gigantic_page(struct page *page, unsigned int order) * the set of pages can not be converted to a gigantic page. * The caller who allocated the pages should then discard the * pages using the appropriate free interface. + * + * In the case of demote, the ref count will be zero. */ - if (!page_ref_freeze(p, 1)) { - pr_warn("HugeTLB page can not be used due to unexpected inflated ref count\n"); - goto out_error; + if (!demote) { + if (!page_ref_freeze(p, 1)) { + pr_warn("HugeTLB page can not be used due to unexpected inflated ref count\n"); + goto out_error; + } + } else { + VM_BUG_ON_PAGE(page_count(p), p); } set_page_count(p, 0); set_compound_head(p, page); @@ -1745,6 +1772,11 @@ static bool prep_compound_gigantic_page(struct page *page, unsigned int order) return false; } +static bool prep_compound_gigantic_page(struct page *page, unsigned int order) +{ + return __prep_compound_gigantic_page(page, order, false); +} + /* * PageHuge() only returns true for hugetlbfs pages, but not for normal or * transparent huge pages. See the PageTransHuge() documentation for more From patchwork Mon Aug 16 22:49:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12439445 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF3A5C4320A for ; Mon, 16 Aug 2021 22:50:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6AF1D60C3F for ; Mon, 16 Aug 2021 22:50:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 6AF1D60C3F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 9CA9D8D0017; Mon, 16 Aug 2021 18:50:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9557C8D0014; Mon, 16 Aug 2021 18:50:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 708FE8D0017; Mon, 16 Aug 2021 18:50:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0027.hostedemail.com [216.40.44.27]) by kanga.kvack.org (Postfix) with ESMTP id 52C8A8D0014 for ; Mon, 16 Aug 2021 18:50:23 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id EDAED1829912D for ; Mon, 16 Aug 2021 22:50:22 +0000 (UTC) X-FDA: 78482439084.02.A8DA633 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf24.hostedemail.com (Postfix) with ESMTP id 65F00B007B11 for ; Mon, 16 Aug 2021 22:50:22 +0000 (UTC) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GMm00M013214; Mon, 16 Aug 2021 22:50:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=8QTQAqWYQLsqdQLgAUxfRRVN6/xbU+jI70PG4q8RsTk=; b=fvrDcqy3FiV1qvSbD7Vn8yEi4fQ/QLe52HUWFEZPCcxtTriCw2+VPiSoiYK3PaaGlo/W cXZHgFH318EcaQCSkuCpYKDCg1N2SLh1atgqEWQ8LCNS2C3Icwq/VMIENINyBltxolmz R0Qwo87D3pjTscAsvdiDsjaoEaQTT8pE153LQdnFScy/u2PHEkBbp0vijJVsah2Ra8PX CtYH2xMFHU6OZdaWCn+0eUkjCcOvPgtUAH6kyKjC2avO+d9ZGhiQ95g24OgdklSJ1P+4 qRnNcl40aI8Ocj7xe45xpjboUlkY/xrLX1wB9JXruq6SIsueONdt2v+LR8LsE9KaMFso Xg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=8QTQAqWYQLsqdQLgAUxfRRVN6/xbU+jI70PG4q8RsTk=; b=w8NuNn/BMUXLJ0b0fVpKdh0GmUeSdEeu0Qs0p7qFW2DkZGqsXjR7XZO87Mojrok0G1nd uHzk8f1SxHwr5X/X0QUUZgXZth5Em4baiv03DsKWwFSBs+XnBRUnw2v/qoaRpiYMn/n8 8KypX8LzavTde6WIuA2L54Z4tiVxiro6iDueZtFuG5Y+sLjJdQyUIkLyquLZZQnjzGGG dPeTRqyYXqLihwnxJJ+LuoLPWUiPrfc8xTfhmRQeoM+7UiWE5+XK2nhpldQpGVzCqdFj ENxuO0Tjs1IdKemzncKYx8j75Wz5wz49z/JSrKMmwBh7f88fzH/8A2bNAEh7qempvSL6 mw== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 3afdbd2ndc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 22:50:16 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GMiu6J040104; Mon, 16 Aug 2021 22:50:15 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2172.outbound.protection.outlook.com [104.47.59.172]) by userp3030.oracle.com with ESMTP id 3ae2xy13wd-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 22:50:15 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UgWy65du2xEr0T6O7cKQj1cgr/yWjhbgU6y6if7JSlY9UEyszjvFct3owpi9oqlZ7gz0Em6pBzEojhBWy1eUK1RXIq5o2dOibasX7VfnM+tO3004mbLYImJTJBcl4xOqkD2a7nlwmeSV0KpxabIhIH+gnB1nltd+IRxD8GoPAAwODi2mv2/aJdjIFdJZ5xxs2dE/LW0zDWMp8KBO2CZLnzDA4Cz7X9NNXs8tFsL19UddcMmPGaxXEMuE4lz8m7DYpjfKzHaK6hBMJ3jqJ3PgfSu39yNC53PpaHkW8ZSu2Nr++BM2fvXypf5Otk0pHoa30+n5qI9zxt5zBdgWBvyiLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8QTQAqWYQLsqdQLgAUxfRRVN6/xbU+jI70PG4q8RsTk=; b=OG8KoI2v7PeBNoNc7nakJi7MeMpg00RnM7aMR3Slz8lGZe+mpyLbAynhuWJEEe52/haBvrj6L4VbZLGIZg6g+KAsenFadvo1i1b60lcLH4XZQ+fCc9dydj/ZvwMsPZbJf36vDeIzJ0n5fycY7MnzRbX+qqSvB0XQrQw54FMMmmYQ83TqOYsGD/jfIQB/nUp512jqQsEM5tDgNlA5sV01mH+3GvJtsMVjkVWFL5x2Ud5qUkGT9e0VdskToH6kahH2Q3xndNevaythwZwJ/yJhNN3+IQedVf5GIf29gS+3zj+u+izqR2YbWfz4nXTUGJpeQlNR6GZKCJwZxbdIQI4BSA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8QTQAqWYQLsqdQLgAUxfRRVN6/xbU+jI70PG4q8RsTk=; b=JZT4yHSURXgfH17mX1eFxZCp9SdsDuRWd3UzJoH2Cvd74YrAdPrfR7zf+Cj1QlLhaaEThmRNYTlbt2x4lKnSXYyc+lgBUcKJQMPtTejlhpElxtW2rCIqdYwTv7YSrzMnEPiEjfD3aau2wxMYoIQs9jQkQ7G+H/8UdB7LgmTUZW8= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3397.namprd10.prod.outlook.com (2603:10b6:a03:154::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.14; Mon, 16 Aug 2021 22:50:13 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 22:50:13 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Michal Hocko , Oscar Salvador , Zi Yan , Muchun Song , Naoya Horiguchi , David Rientjes , Andrew Morton , Mike Kravetz Subject: [PATCH 4/8] hugetlb: add hugetlb demote page support Date: Mon, 16 Aug 2021 15:49:49 -0700 Message-Id: <20210816224953.157796-5-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210816224953.157796-1-mike.kravetz@oracle.com> References: <20210816224953.157796-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW3PR06CA0030.namprd06.prod.outlook.com (2603:10b6:303:2a::35) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MW3PR06CA0030.namprd06.prod.outlook.com (2603:10b6:303:2a::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.14 via Frontend Transport; Mon, 16 Aug 2021 22:50:13 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ffdf9579-f3ea-4064-6d20-08d96108358f X-MS-TrafficTypeDiagnostic: BYAPR10MB3397: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6108; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: M9MwYREgKg3V1a+BKzGaj1n0Wqa6dcnwRi4ZV7zfdQGMGMStw7XtZthAIq1WHoq31UB6xcHvsSdBQubF7DJNUosyrXMeUM4ATumW4NgDt1DeGZDiad6Lat26j3e1TlLq5vv0gs+sGAODf5dLfucgYgUEZ59MVPPluthfryA1DqF21B4fIzFztoMqE1gRUrSR5Lh0hV5GvkmyuGfh+pNYV1yI7Rf7B14Jx/9I1gPh9gJUpFySqL0aaT+TRmKKwn0sVJ2XwijYF8onG0hsNsxvUMshNAE4adgDZfHq6hm83ueBvQGklkE46deNFbqYzNO4i3x8RmXWBkGOvVvQDzNbzOBq2VYP7uQmbkpp7YOm4unCEtTI1fY+oF+A2gvhI5gzrrkvUrkscKjfTTOuqmgYHqRPb3tPBrQcEvdL4+I8BPFItl+yrIJIxsUV+pg2RK+RkbuOGUC2+BOyE1Bj27PeH+b05/dnlaBoU/RXSSzyo+l2erDn1k1hG21QZR7AhfcdmbCjbUtSifXE0KWlszW/qrLw3D/nxXVOryS371ethgIEucZbwoDYr8+wQcGXQ7+X7ZHiVbNceF3jqZ0liC2+2VOsEjB5gYEpq6985d0oEb6qEZxsolRyI6N4S2J85HQJ11wd/UkNzAtbzYDVzg9ek2dmZucL4lkUoe+Nas8HLM1E5u8ft9ZVFY/92Lx2DD6Q32O9VqBXGmYy7k7wYUe5lA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(366004)(346002)(396003)(376002)(39860400002)(4326008)(54906003)(6486002)(956004)(2616005)(107886003)(44832011)(316002)(7696005)(8936002)(8676002)(52116002)(1076003)(83380400001)(38350700002)(38100700002)(2906002)(6666004)(86362001)(26005)(66556008)(66476007)(478600001)(186003)(36756003)(5660300002)(7416002)(66946007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: wo+uFOuTNX6Wc3HsZkWDoLdNyeQ0hxjxLIUzK4WP0nqH/YKDUxP8sBccTaiF6g0pNsJglIX7hRsrQoXp0MMO4hx0xZ8WbTDWgQc/KkM9Vy1N9pj/rieyNZI2mE44ORaoHdb33dcCkU2nzLLawngPDe2PoEb5yUgh1mIaQ7r027UeEltWPgUXmJ2eZgFjTLCbuFFPUoc88+Z2ap1VuJrHXcDe+c1IXeFWVgwQGUyLdJrTMpiweaUhXYDBxyUq6WdfZU3E3TE0nK9aM9lGyqfmT8fk4zkxDD+XCyIIk3oQVy2ZALQecXN9wD9R378kV0nj6CggTc0XQZcQQZ3a3zN0WJrfziKmMEnHo//ODpWwuyGF7t9lwwVZDBmTqIwnlFYQHp8EELOXsHqZTEAwlI43dYpJrDMvdhtZeI+s1QnuJOipXVFd94PcrLEQST8SYlP+jwY40BpvGyXyFD0somWRfg02xXn55Xu6ZwilNgNlBze9kFSYiXgG5g38lrfFq9vK77uXwjN/4a+jnAwON/MS37oFxLqin8inXt9g9GYreBoh0iWlwEc/d3AmCSqsjojzSLZp8O04w9Nf+XMhF8j5Q61WhIqG/+Esw5/0BndFkZqIr2R2lrBREpj5wBnLsWxuac74WLbxagbyXZofpqeukso4eKpS8auOENj+BapZ9nu/MRY+xBn0Ffl5RGArL8ooN04hcPs13PGPIHlhRh00epbMM2DCC8BNX+WTiK3Ik71KWrXyTwu7nk/pZwHgE+dCjEJ8GpRH1RPpMxc00fvL+rFZKo0mvdeTrjz2R5W/zoDPGT8zszsP7de5ryCDDnW+uELantSQ7YK2KROlkXy0qFjcNaBfas7FhdDj6j8oRu70yf2mNFgPo2t4LQ8mzeokqT8031iiaVoRhWrKkgfvnzNrEvfrCuXJMQzMOilJNnfwCwkAp81EQBrQbb5YO91gH5rh0dfo5ycFMtOkqpEeGLfOiB5MTD2AkQOeMlNwPh1d7BB+2f/zc42bqq73wf9cIWT3TJ9kCoVhR3b0u3bbyidtyFg/4/tyIlj+vvfvBomd0+IOCLqW8KPgsUxuEG7kE7ZdvHfFc6wyBAMlY1pz6zMt/hjTxxsqWXBh/5Qm2ae/8ygjPgb1HnzZnHDE7C7DgArW9POPmgTP1MrE6GRoHJojnKUEhVJypD7b0BtPZ/0PHveex55Y7T2daYs+EqbFftGfqb2VTXWS3nmA4WIw1NgUBiYdfru0rNgOYuj8KH7OcXJHYd0mHHw1En5Z9iNfFVDyH9t0IGPUhElm7/43GGY404BnxCeVXEV1YE/BYxeStR7S9Uer/Gon7t5qACJV X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: ffdf9579-f3ea-4064-6d20-08d96108358f X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 22:50:13.7264 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: /xtcAeH0RSRgRgfZm4LutbQ2sZOHUb1AgAX60quE5LSlcmdvAlG8cy63jbnRT6d07ktfI+RD2N6jD2jXZd3CEA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3397 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 spamscore=0 adultscore=0 suspectscore=0 phishscore=0 mlxlogscore=999 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160142 X-Proofpoint-ORIG-GUID: oWhduWpMv38hWC9bwwpcWjx_8jlquBiP X-Proofpoint-GUID: oWhduWpMv38hWC9bwwpcWjx_8jlquBiP X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 65F00B007B11 X-Stat-Signature: spnxgtunxkri6xjsoqtphb1ssxfd5umn Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=fvrDcqy3; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b="w8NuNn/B"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=JZT4yHSU; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf24.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=mike.kravetz@oracle.com X-HE-Tag: 1629154222-597429 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Demote page functionality will split a huge page into a number of huge pages of a smaller size. For example, on x86 a 1GB huge page can be demoted into 512 2M huge pages. Demotion is done 'in place' by simply splitting the huge page. Added '*_for_demote' wrappers for remove_hugetlb_page, destroy_compound_gigantic_page and prep_compound_gigantic_page for use by demote code. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 76 +++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 73 insertions(+), 3 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 2f2d5002fe73..6d43523a1a4d 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1289,6 +1289,12 @@ static void destroy_compound_gigantic_page(struct page *page, __destroy_compound_gigantic_page(page, order, false); } +static void destroy_compound_gigantic_page_for_demote(struct page *page, + unsigned int order) +{ + __destroy_compound_gigantic_page(page, order, true); +} + static void free_gigantic_page(struct page *page, unsigned int order) { #ifdef CONFIG_CMA @@ -1428,6 +1434,12 @@ static void remove_hugetlb_page(struct hstate *h, struct page *page, __remove_hugetlb_page(h, page, adjust_surplus, false); } +static void remove_hugetlb_page_for_demote(struct hstate *h, struct page *page, + bool adjust_surplus) +{ + __remove_hugetlb_page(h, page, adjust_surplus, true); +} + static void add_hugetlb_page(struct hstate *h, struct page *page, bool adjust_surplus) { @@ -1777,6 +1789,12 @@ static bool prep_compound_gigantic_page(struct page *page, unsigned int order) return __prep_compound_gigantic_page(page, order, false); } +static bool prep_compound_gigantic_page_for_demote(struct page *page, + unsigned int order) +{ + return __prep_compound_gigantic_page(page, order, true); +} + /* * PageHuge() only returns true for hugetlbfs pages, but not for normal or * transparent huge pages. See the PageTransHuge() documentation for more @@ -3294,9 +3312,55 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, return 0; } +static int demote_free_huge_page(struct hstate *h, struct page *page) +{ + int i, nid = page_to_nid(page); + struct hstate *target_hstate; + bool cma_page = HPageCma(page); + + target_hstate = size_to_hstate(PAGE_SIZE << h->demote_order); + + remove_hugetlb_page_for_demote(h, page, false); + spin_unlock_irq(&hugetlb_lock); + + if (alloc_huge_page_vmemmap(h, page)) { + /* Allocation of vmemmmap failed, we can not demote page */ + spin_lock_irq(&hugetlb_lock); + set_page_refcounted(page); + add_hugetlb_page(h, page, false); + return 1; + } + + /* + * Use destroy_compound_gigantic_page_for_demote for all huge page + * sizes as it will not ref count pages. + */ + destroy_compound_gigantic_page_for_demote(page, huge_page_order(h)); + + for (i = 0; i < pages_per_huge_page(h); + i += pages_per_huge_page(target_hstate)) { + if (hstate_is_gigantic(target_hstate)) + prep_compound_gigantic_page_for_demote(page + i, + target_hstate->order); + else + prep_compound_page(page + i, target_hstate->order); + set_page_private(page + i, 0); + set_page_refcounted(page + i); + prep_new_huge_page(target_hstate, page + i, nid); + if (cma_page) + SetHPageCma(page + i); + put_page(page + i); + } + + spin_lock_irq(&hugetlb_lock); + return 0; +} + static int demote_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed) __must_hold(&hugetlb_lock) { + int nr_nodes, node; + struct page *page; int rc = 0; lockdep_assert_held(&hugetlb_lock); @@ -3312,9 +3376,15 @@ static int demote_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed) return 1; } - /* - * TODO - demote fucntionality will be added in subsequent patch - */ + for_each_node_mask_to_free(h, nr_nodes, node, nodes_allowed) { + if (!list_empty(&h->hugepage_freelists[node])) { + page = list_entry(h->hugepage_freelists[node].next, + struct page, lru); + rc = !demote_free_huge_page(h, page); + break; + } + } + return rc; } From patchwork Mon Aug 16 22:49:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12439449 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 581F1C4320A for ; Mon, 16 Aug 2021 22:50:30 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 03F4860ED5 for ; Mon, 16 Aug 2021 22:50:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 03F4860ED5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 008138D0019; Mon, 16 Aug 2021 18:50:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ED3DF8D0014; Mon, 16 Aug 2021 18:50:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C885D8D0019; Mon, 16 Aug 2021 18:50:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0101.hostedemail.com [216.40.44.101]) by kanga.kvack.org (Postfix) with ESMTP id AB8608D0014 for ; Mon, 16 Aug 2021 18:50:25 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 4CA59182AD9B2 for ; Mon, 16 Aug 2021 22:50:25 +0000 (UTC) X-FDA: 78482439210.13.4BF6F3F Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf10.hostedemail.com (Postfix) with ESMTP id B8FD06011E8E for ; Mon, 16 Aug 2021 22:50:24 +0000 (UTC) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GMmh8b026759; Mon, 16 Aug 2021 22:50:19 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=T7aTzh/od8/z0na96515K7P39K5dkQ+t5If8qBwYzOo=; b=edybYuwKLsJcrHvoByUDAeW0vJ6YYB1kJyahImddiYC15m74wZxiTQxDPry+iMHvMcG/ 2A7mh0zhqan7MCGSAaYlqGw5IYJJ0WPRfnADa5hLhkSoy7G+GAVi+caek5c6JbLgpseT fcVcVFdw+YlkpTgCpmjQDBjM3uxSgz5k5sS381A7a45QFX+SNRpMph0/0VoeVAlnOST2 wUHqYWJzduMIbRIsTSZbYPt7d6LhJj0diqcmgyUmLXk3Wh8RArTEZdbGCkG31gpfDK94 9NQc8pWqp0UWlCOAfA+6mGn4ONbnRKk6vYd5B5BF4lr9f1AfFcBZtS6qFO5cQJZ8hOmu wQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=T7aTzh/od8/z0na96515K7P39K5dkQ+t5If8qBwYzOo=; b=Sw2ML0E57kLi6DKyz3Sanuo9uqKvZvyccZSFgxr8l9S9QfqWLpifWy8MqpX5//UZNZ5r o7V3XACI8yNtA2RS/jpLEG08r63cQ8iDKIisiVN/TAKQl3qiHhQXYvBYsMhi6ABUxOM9 0WFkWnYP0y1qURieH9kPi5x+SXjiwMSiu2OUvw6pcbI4+psWWbB8SycjYv2t4OO/s5zH 8/mgTnkZmHzULAbIENdGL65EDSeO8DzaJoKf66xHxeL238OMq3x5QwpqXIt6UAHv2nkC /kTMorsfgMpw/WB52PJDX2E/cysTIH71jv9pp5SRPjnaNw5ro96DtG2y/Hxiw5IclTBh Yw== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 3af8303230-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 22:50:19 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GMium5171583; Mon, 16 Aug 2021 22:50:18 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2176.outbound.protection.outlook.com [104.47.59.176]) by userp3020.oracle.com with ESMTP id 3aeqkt0pdq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 22:50:17 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EWCALtwRpId2oZMwZXesu/AG9pf9Tnl4EEmBAMItKUNZxUQxjFSoLGm7//fJ0A7/sulFqcAyWB4OnhluNHWM8Y1dfz0oVA0wHLxvCcqvXt45EKJc60RvvpRKoxWvWqr/vGfd+PE3wTiJ6FeMQnByOv475hhyBHya48b4YdBiAua12KlmS33NhZms1Ia8Fr6BhALHFh+eWijZn5L65nqL8zV0+3Y37N/BhptOZHVhfZQJeF6qoQGGlNSgn3DMxYhFLQrMXt6bflhNaBSp4gmnYiEX2FZXOizKeM68Mmg3IeYias1imOUlopLpQCsbP1fSVxYoCirclEfM3D1xwioD7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=T7aTzh/od8/z0na96515K7P39K5dkQ+t5If8qBwYzOo=; b=hQQeK2zW1mfsbyBcJE+5kShuMheKMTyQMDUWpgheEEBb0vyJItMFM7uSeMJmLWDRlt/3U1sWkqyWSrgEW//Am/+HntsA8KULlebj/BG/Yd1DZaNpsEz6oBocJSmqCBJFeum50tdXAM/gwrKbS6+179ZV7ORy1a0eJ13R/TOPxL6uWu0buQJckfnsTR+9yDUsFhGYnPelk4CwN7QOIU6Tn9pCv14yeVHJqlkKBPkeCQUwyNgldqUS2e2j2Q5xjuXZhq4Hxp/pQQOTyr4qWnGGrMhzwPrd//zx/Di/PxYKC5L+KstUN4rIId6b/1rnwEAUKC+OGticqYCr20WgPXpHDg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=T7aTzh/od8/z0na96515K7P39K5dkQ+t5If8qBwYzOo=; b=AxDhJaaSmzT2IiMVzRxn/qyGn6otG1x9VfJ7W3Hwscopa8dT0/Lbv7SXYx7s+a+3xcusl9+oKx/VtwX3hM6f2e2cF0cNc4zV6QzXZAToG7GAmeHWaelY3rrZ6X/MPNMLwA5iV5m3tGnin2Td70UVAZvQK88ucKnL7vUqMFgQncU= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3397.namprd10.prod.outlook.com (2603:10b6:a03:154::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.14; Mon, 16 Aug 2021 22:50:15 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 22:50:15 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Michal Hocko , Oscar Salvador , Zi Yan , Muchun Song , Naoya Horiguchi , David Rientjes , Andrew Morton , Mike Kravetz Subject: [PATCH 5/8] hugetlb: document the demote sysfs interfaces Date: Mon, 16 Aug 2021 15:49:50 -0700 Message-Id: <20210816224953.157796-6-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210816224953.157796-1-mike.kravetz@oracle.com> References: <20210816224953.157796-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW3PR06CA0030.namprd06.prod.outlook.com (2603:10b6:303:2a::35) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MW3PR06CA0030.namprd06.prod.outlook.com (2603:10b6:303:2a::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.14 via Frontend Transport; Mon, 16 Aug 2021 22:50:14 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ab0beb7b-ceca-4e84-a3e0-08d961083689 X-MS-TrafficTypeDiagnostic: BYAPR10MB3397: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qLLHsMKs32gTY5t9RdyqVDAiSiE+VFEaKaARm5cwZqrxN37qsNOEvog3pgfrWegj9bQ4R7jVNoYr3oL+7oWpMPDpLOHylwzLWxAI7zJeyUTbiT5/yo6Y/v5tp5niYkIwgizWm4dKPuEl7u+FlWipjCWuPGUUy4DNNWIA71f+C8bakM/1uwZ6/giw8oI0gKtcvz1yOrdPsUfJhLLDbah7v8+Fv/iIK22TvrMO0MPUxG6bqsIJ1y7ga7iv6wzIRkXui7C5SB+ZROF3uN36B6w8Z9Z5th6yfdiu2Uv8NSwVC//KwVZzPsYp60VVoIMzqstrOnd8HrbK1KvdI2UnI46LSvZgtuf0UjndkWyr4F/4nGL2d6RjHGsqCA5HTdf7kzxXtS+JeNtKn/f0DI7am6FKdCnC9rdkOQ99t2vj97Mfd/vjfcusH9fzqGnqTUP4qytnFYy29dFstxYtb4Dp5R61zjvPROrOyXZ5Djf38IOufO3CzISYnhcNRD7vzrTNO3kFOUw+jtImO254iI7yHrGVNOdyJyJDXZek/VfWpff3bib/lUg+i9HcUv/oqhYw2DXgZC7LTXuKBqezLbib4r+sa5oz8vT2kZUrn1pnzVfW8twAZwuEPm57cyxwoALC8yogMOpKftpKzezOkWimC9OXdNSnRwLeHt54yG/jAMTEguv5tYZLJhzFdz0p/kK749aBdlF/ns5WbqRDDIihEl1YnA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(366004)(346002)(396003)(376002)(39860400002)(4326008)(54906003)(6486002)(956004)(2616005)(107886003)(44832011)(316002)(7696005)(8936002)(8676002)(52116002)(1076003)(83380400001)(38350700002)(38100700002)(2906002)(6666004)(86362001)(26005)(66556008)(66476007)(478600001)(186003)(36756003)(5660300002)(7416002)(66946007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: S1uk+sCyC7hlX2XXiWeFbcP510jfYXVQLd6maRFkURO1C8Ifr30lv91QK/CKGNfdb3L2eDUaZ4vwsSpKDCRSk3sV2y5fI872+a3yOFOj/qLn5SwPp5RLGufVmytcOPPgx17tZvkijCk/t1CskVTMp++8U8HLN+dR6T0Adrlte5BnMMGIV4L9/uuGnwKI2KgSxmXI7X3L5Wa8tD6gXRsnwuo2FhS/CYMQfR/IGN3GlWcbl9rGI4zzKo+tp4Ov+TN0sUQZhmw+xorMNxCPGREB0fIBSRJsfjbUWro3k6PWypwwAbtpTW7bnzF3MYZIKZo4/zJu4wJvD56S+D+YNyLkP2nuPmPN5a4vk6L2xbC8CFaKScJjlUoI9KNCUN+uqHe6Vj/dDOyN1EDsD7dg3E0Ym4vbXfx5iLPWRM87iM8natJSApEvx3FyrUEu6CH97e0wCcEd9QFdbNge1qA5eJp91Xh/fzPJBnf7s0Uy0j8NT2L6VP79K+xrHgq7DmAzdo0+fEGFVrZMp04CrIsLOBQEhrgfgGTdLSpD2lSSuHjU9QIRjdDsmhmyTroaith+4J4NIfE/YZb35gDbvk+Ucw3a69E1T1cOeQFe2NKieaJ7lDlOtmCCZe6msdUILQ/MBAa1XAsdXt14QVWoMqS76Y+vRakEUYeeukVLC56WAaIsrb6jn1AKr8epCFW8TqGYXKrw5H2WN+5RklWMdEX+lWOfCv/18fY4Q25BySIc8S9BMteAN4LD+/xp0ho+7t/Nl6bWnqfrXjIC2jG4elW0pjObjwoOrk9DBnatb7fOkQxb58SpQmPItefEsmk+xM/98j9qWTWqPl74MEOIilze65sDMUwdXQkaaMzpGTrg5yo0NUzECU5ivEOdnv4YaIsBOltp7x8ME/NisnJxesu6ogwjd/CxGASYQLVtnu0jnUApW8vPmpCVdB/dT1TJu7C0Q5UhDzWJmoTe3otDyNdqiy2lNN/LsuwCSP8nYFZ618Ho13iqEpoc/7MQ2YlPMiz3gf1KuidviiakBsgYQV2G2wcm/GF+N2bMAVAMQUnM6lSFXvXyHL0e5d/M5xl/fs0fcPu02gmSdURXiXyAg/7ZkbFnQOOZ9tPXrtPQWbSxWEBe/rvT58KzT7CzXRe2VgIR0p1l9PQLViUz2Qk8Mcb20NJYZQuFDqAted2y6bqOdBAsm9L8YY5oW0ZclIm4XZJR8FVk/g+tel8EtANEWsDO6bFtiLNJ7JySZZ2E7ZM4GeuLmmq5Rf+j1pzmZ6o4bEkdnpCmNI495t2JrNJGpaIJ0JEu4H0oiWTp0xf1kX2zaW6zbrqH9U50xvJKAqqItgilbbxd X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: ab0beb7b-ceca-4e84-a3e0-08d961083689 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 22:50:15.3761 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: +EyTKzbcCLcFY+v/rxOCwRLgrPE0GvAwOmKnuqdJaRxp/NIXsKBydPtASgpe5+GmC3h0+jiQ4zuZbwNREh1h4A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3397 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 malwarescore=0 mlxscore=0 spamscore=0 suspectscore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160142 X-Proofpoint-ORIG-GUID: 2I9iKlqgijKvhs5LA6_8Mt-ygu8mJ1wm X-Proofpoint-GUID: 2I9iKlqgijKvhs5LA6_8Mt-ygu8mJ1wm Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=edybYuwK; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=Sw2ML0E5; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=AxDhJaaS; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf10.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=mike.kravetz@oracle.com X-Stat-Signature: cukm6a6shtqjhuu8ozyyuh868mmod6h1 X-Rspamd-Queue-Id: B8FD06011E8E X-Rspamd-Server: rspam05 X-HE-Tag: 1629154224-788114 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Describe demote and demote_size interfaces. Signed-off-by: Mike Kravetz --- Documentation/admin-guide/mm/hugetlbpage.rst | 29 ++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/Documentation/admin-guide/mm/hugetlbpage.rst b/Documentation/admin-guide/mm/hugetlbpage.rst index 8abaeb144e44..902059a0257b 100644 --- a/Documentation/admin-guide/mm/hugetlbpage.rst +++ b/Documentation/admin-guide/mm/hugetlbpage.rst @@ -234,8 +234,12 @@ will exist, of the form:: hugepages-${size}kB -Inside each of these directories, the same set of files will exist:: +Inside each of these directories, the set of files contained in ``/proc`` +will exist. In addition, two additional interfaces for demoting huge +pages will exist:: + demote + demote_size nr_hugepages nr_hugepages_mempolicy nr_overcommit_hugepages @@ -243,7 +247,28 @@ Inside each of these directories, the same set of files will exist:: resv_hugepages surplus_hugepages -which function as described above for the default huge page-sized case. +The demote interfaces provide the ability to split a huge page into +smaller huge pages. For example, the x86 architecture supports both +1GB and 2MB huge pages sizes. A 1GB huge page can be split into 512 +2MB huge pages. The demote interfaces are: + +demote_size + is the size of demoted pages. When a page is demoted a corresponding + number of huge pages of demote_size will be created. For huge pages + of the smallest supported size (2MB on x86), demote_size will be the + system page size (PAGE_SIZE). If demote_size is the system page size + then demoting a page will simply free the huge page. demote_size is + a read only interface. + +demote + is used to demote a number of huge pages. A user with root privileges + can write to this file. It may not be possible to demote the + requested number of huge pages. To determine how many pages were + actually demoted, compare the value of nr_hugepages before and after + writing to the demote interface. demote is a write only interface. + +The interfaces which are the same as in ``/proc`` function as described +above for the default huge page-sized case. .. _mem_policy_and_hp_alloc: From patchwork Mon Aug 16 22:49:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12439447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 157CEC4338F for ; Mon, 16 Aug 2021 22:50:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9551460C3F for ; Mon, 16 Aug 2021 22:50:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 9551460C3F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 5AE6C8D0018; Mon, 16 Aug 2021 18:50:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4E6A18D0014; Mon, 16 Aug 2021 18:50:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 24EDD8D0018; Mon, 16 Aug 2021 18:50:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0139.hostedemail.com [216.40.44.139]) by kanga.kvack.org (Postfix) with ESMTP id F22DD8D0014 for ; Mon, 16 Aug 2021 18:50:23 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 84528824999B for ; Mon, 16 Aug 2021 22:50:23 +0000 (UTC) X-FDA: 78482439126.25.8BF67E0 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf12.hostedemail.com (Postfix) with ESMTP id 0C2221008107 for ; Mon, 16 Aug 2021 22:50:22 +0000 (UTC) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GMmFhE020124; Mon, 16 Aug 2021 22:50:20 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=dHD2WuM/ybdU4ii+bYUQxuX3QoRCL/c9D3+uW5FKocQ=; b=kcrNUF8X7EXZoJqbFN2a6XkMrj052tQScnUOFI3x444Y5RnouO+37Hb3VZeK8Kg9ug4a 5Bskd9Q4f6PGNv0oFMiIfEtrbLr1h9pPhRVFk9vwUGkWziSeV7EZMAV+gNVVC5Lg1Dk5 6X3f44ndfSzmc+ca6IWLYR2ChBAuAoG2G+azmriJyimC2lvYQxtU2yeQ0Bn3PKuXkP7p yw5DoSW6aXsVrrH6FGvU3R3KHy6SxUtlo31TBhU5KGh98bouEdnJhn6WhVlymIKWn5fZ k8gMfjC5DtXOV+szyCyHvQHO6Px6kq9VFsuV/0i9R/H7DVFervbPeUAjeFKTYPbDxWRa +A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=dHD2WuM/ybdU4ii+bYUQxuX3QoRCL/c9D3+uW5FKocQ=; b=cQrweU5IEqpR7sX3V8i7gIaa0zSzVsypyYwmjiIctN3himJHcCaI2miRSGAWCROxTvck f4LFjLTttn2XieeW2mWBXUIsv1tHbHMVME16zX7XvZZ2w2y7PBUv6VPJOplPcIEANOE+ rI0KPxq7JZvw5oMcB0EpuEwnmFNxeqS3//skm3OG5/phcKuf6g8NyfBOHicBY8sc0sky BRI5zrghFRzE+5JICIqhA0PWEo2mtkKHfP4lrarR6vGzYEzdUw/3e+hz0sRPI7fMTDrk 8Z87KakCsvGpg4F+Dutslbum3aDOFNlK7Mzinek+EdI2NwVUVH7yASclQ/2obGfLJlTC xQ== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 3af1q9bd75-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 22:50:20 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GMium6171583; Mon, 16 Aug 2021 22:50:18 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2176.outbound.protection.outlook.com [104.47.59.176]) by userp3020.oracle.com with ESMTP id 3aeqkt0pdq-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 22:50:18 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ANEs9hcY2LHTHg7mdH1/DIbLkOcdMvW8bHZexrYHgc5WQHdvGJoah8Se4bevsmqkbY9tB1/By+G+gFvNP3BfMqhE2MdkjmlV3Um4Y3ZSmuVhHjVabOZZestKpyJiWkxVm9RakoztXtqGykBUfVaJtxWpzc0zIkvUwXzFmYJ3mwywaqH2px1gupKwpJOMr/QzYS7xSl5NpCW9zoH7i50wtRtqym7l3PyGYC552iidNPZSJoVSckV+csk1ltDjD0ALQ/E2TUZUausbj41zqJsZ+4gBJ75OZA1rN1KgU2TqjneOaVQXVj2BBji3D7e6ilQ8rpYWmoToMIIE2VxBP0RFrg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dHD2WuM/ybdU4ii+bYUQxuX3QoRCL/c9D3+uW5FKocQ=; b=D17hzT0WfqqfBF++7XYTpH7CkuEHg/1vp1TCpeMc2Qcwp1wLFFrLSPe1BZsjzYnWj3pGycOhbko5TS5Smc7KIp23aFfydZ7hvC4jipTBvQ3KQvRO4NTvQfWwChgpxMcTfUa30dwhDlTVDcfAVe3cQ/tnCDLsBKlWEUdAAFWgbrp5T2glr4wGL9LaH3syDdwRdN1x//ze/rikRlaCONObw/z3oUq/TfPGEAk+vBRM2/a9OeVYcmy8WoUaRGsZ7zGZl+J9OH9/LmEulggpPAwNHNdyFaw1w17yizCtQfKXxveajvmp0PNv+wjCz+K4/xO4q/cQKdzBKSf0Ymbt/W728g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dHD2WuM/ybdU4ii+bYUQxuX3QoRCL/c9D3+uW5FKocQ=; b=jgpQOEgMjMxtCiTvYqqzoVnSh4QHOJR2O1hqfnqGYv05c/pjrNnPvVsT5qA2OXfN8HEACoPN1GrEQGLL55JN3qbs6Y/ZOlAR1kSqhX4UgLfQxReP4fqsZcg2j6rz53IeBGxDU523dTY7eHYFpFAMvlJo49AwZ6WF/0HUlP9ik5E= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3397.namprd10.prod.outlook.com (2603:10b6:a03:154::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.14; Mon, 16 Aug 2021 22:50:17 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 22:50:16 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Michal Hocko , Oscar Salvador , Zi Yan , Muchun Song , Naoya Horiguchi , David Rientjes , Andrew Morton , Mike Kravetz Subject: [PATCH 6/8] hugetlb: vmemmap optimizations when demoting hugetlb pages Date: Mon, 16 Aug 2021 15:49:51 -0700 Message-Id: <20210816224953.157796-7-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210816224953.157796-1-mike.kravetz@oracle.com> References: <20210816224953.157796-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW3PR06CA0030.namprd06.prod.outlook.com (2603:10b6:303:2a::35) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MW3PR06CA0030.namprd06.prod.outlook.com (2603:10b6:303:2a::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.14 via Frontend Transport; Mon, 16 Aug 2021 22:50:16 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: afea18e2-526c-419a-1aba-08d961083770 X-MS-TrafficTypeDiagnostic: BYAPR10MB3397: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7219; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: /1DBnWohQPUyQfYv+XTyk6K7LZOAjsWhFrGeVzhaAow9mdTfY9BW1eYDBJg5cx00+nap6gHGfafDAOcuXakjYr7lgvwvavX7fezKSsnkwbiFtklmENFvEEz99Kh1jKi4anjCRF21hzo2h7vQ6CVtyzalaIFzNkELcnyI6S1D5Z/YlsS+BrZES6nWVE6to67h9p9TPwrYvsJ+XwuUGXuVdVJDwLt8mDKKNgCxKXbrijYElPD40NNfOJ/FVayEv0B1sdPZmBjZZ3FxClK9Vw2xhRK8mUXggJa7UZAAQMvdO8OiRcJRbCrqyyZdEjA5Qm8wy0EK2kst3Yk4p1GmWco+08qxvb/4K8SLE35DuBKwTqWwhGGLOW5f6ARnolGQMmasjvu/nPT0AnXW937ez0wywWMWGh4Y+eYG7tAvNHB7Yjnzn27c0wUaiR/8+03NsBK2oaq/aYZjMwz++NCe+F3D3KpKL8/tucy7ZxN5qtEJoYErU3Syic6N4zIPvUskjYvHULK6nh5XGCxXGyV9NtFu7tUZT6SH1WPgi1bDcK5RSvgQP6Nzu1GOPEVkLBGb6AI1bW3zgc/l9CTgu0GJybuQDm/ieG4lihP7TUyjX8BXSCdFFLRAcGL0oMbUB29++fHxHml9TsiEGBiSXo8WFKLC3kYUS8RxChXSzI5S9efBw068jrK8AzHCsNmLsb2mol2kuNA3yDqnX9vTM/KwHrT2UQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(366004)(346002)(396003)(376002)(39860400002)(4326008)(54906003)(6486002)(956004)(2616005)(107886003)(44832011)(316002)(7696005)(8936002)(8676002)(30864003)(52116002)(1076003)(83380400001)(38350700002)(38100700002)(2906002)(6666004)(86362001)(26005)(66556008)(66476007)(478600001)(186003)(36756003)(5660300002)(7416002)(66946007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: CQEyHA3MYpQpaIU2adEk9Xi2QLGUsbTMdLNxjweD5xDeJGCJRi9+SdKBnBXnFyAxTfTSZ4g8lV+wND9sn8iJkcruMXfuddFfs4XHB4KExCmWTLHdHJKQr1QxZp3qNkdjODyEAd0HCSCq82KS2neWATUkx2oDdnAbudpseSvn+mv6NcX0GdcoRiGw46Xo0ddLkytktffJ2/bSgVY8JdHQJb39ccy0PRkCVZgpCGixiBSWzVLBT3Bz9fHMwqSH/MiRiWMl+v9ofm9MsDTc/IOdl7YRaqdSNf2EtDlLxfVO0oztZtFroSSNuG31YNZQBWXsyKsHlHmd3pKUs2LD7749NCPsUHJMTAVrRtQfYGk5j4wmWM7RwTra1nHpzDAS6bKFMwN145CWXzCeWm0j8SzQpJU4crWwa1blGiMW16hLzJiDqHcyn/3FYShlJtdLxymjacUTsdAPbfxhLH+sYlrmTt6WfvT2ibORG4LLsAJcMcwPl5CaVzoDzwcMjToQ2trWWqk0LvCcaUyJpQnvjqH+bbbrS+NLaFYYAFQ5gNjws61n7lHI8OCm1DFLctpXngIDDENcr2BE714z7c0CefvOVemvlOTB8lShis05psADPvIaI0E0VeW9ZHBaw5YhUekyd2AbaufLWQJQgHSFqXx4y4OzeLtCujqCDpeW1Am2x/veb9b86EMfjvinLngFZbcWsq+SSCItg55DEyg3Xh3cT4JJZVI/lk7ARSZgKwZGjG/n4m5SxL9tIUnzaA/gnoplvhf40P6h8MFt94upu7QHYN2ZJUPnXR1p4ryFxMGXNuxcHfaTGPTesBpUCikJx3AewQcNPNrpFh5clZUvBqekTP3ojptS2vrpQLsExh1ulh+/ht5uDeTBtxjdC/iSqBL+ge3Mf0HleeOFgpbAG/VQMd8a1d6T+4ZKzp2503EIkX3x6U/+IWIJr7EdE01n7NwnsDghgaEfllj23TGvjFjzGEfQrTYbgrTNs9tzYHSYUtEb6wWIEKXZ5a8vLP++HnoQIMh7sBIe6rLAHBf4v26KAX6JRXAlKT3sgZmuIGmCOn5zS4S5twDxk0Nqw0tQTQOaGZXX6kYS/1ZtFiW2gySbJtbLp3Y4Blh6B4pM6TGe2UaHMTn4wUo6cCi6vLldWLsO9uFNgm0CrBYraDJlAGhtsgtE4jm/KsHJqKb/Rlqnei26oJWF1aXrZufgo40PV9pHvhPHzl5N07eiFFXmx60hwZei+f7IX9L2hZ6dcMzhrDD0Ot640gxxcBA910jXHs+N91PH5UUay+uLY3vsVY/4pLQ4geKWVEQb66BaQfVb62uvUx9as2l6c8p3YmNrv/Ic X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: afea18e2-526c-419a-1aba-08d961083770 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 22:50:16.9005 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: touCsIhMSYOZ9V9IpV0c1ySDn6hj/aVKlY1xRyJA/MGTDrGB2hK5xoX99TqjpEQHsEipJJ/sXz16rCRLTPHYBQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3397 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 malwarescore=0 mlxscore=0 spamscore=0 suspectscore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160142 X-Proofpoint-GUID: 6B_eUD11eZZvXd_QVFBrQDecUXYN6USp X-Proofpoint-ORIG-GUID: 6B_eUD11eZZvXd_QVFBrQDecUXYN6USp X-Rspamd-Queue-Id: 0C2221008107 Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=kcrNUF8X; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=cQrweU5I; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=jgpQOEgM; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf12.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=mike.kravetz@oracle.com X-Rspamd-Server: rspam04 X-Stat-Signature: tg4teg75kxrbmzff74d333afbiwi7i4a X-HE-Tag: 1629154222-896753 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When demoting a hugetlb page with optimized vmemmap, we allocate vmemmap for the entire address range represented by the huge page. Then, we split the huge page into huge pages of a smaller size. When preparing these smaller huge pages for use, we optimize vmemmap for their associated address range and free vmemmap pages. It is sub-optimal to allocate all the vmemmap pages associated with the original page and then free most of those pages when preparing the smaller huge pages after the split. Instead, calculate the number of vmemmap pages needed after the split and just allocate that number of pages. Then, only populate those areas of the vmmap required for the smaller huge pages. Introduce two new routines: - demote_huge_page_vmemmap - This has knowledge of the hugetlb demote process and will calculate the number of huge pages needed for the smaller huge pages after the split. It also creates a 'demote_mask' that is used to indicate where within the address range of the original huge page the smaller huge pages will reside. This mask is used to map vmemmap pages within the range. - vmemmap_remap_demote - This is the routine which actually allocates the vmemmap pages and performs vmemmap manipulations based on the passed demote_mask. These routines will be used in a subsequent patch. Signed-off-by: Mike Kravetz --- include/linux/mm.h | 4 ++ mm/hugetlb_vmemmap.c | 60 +++++++++++++++++++++ mm/hugetlb_vmemmap.h | 6 +++ mm/sparse-vmemmap.c | 123 +++++++++++++++++++++++++++++++++++++++++-- 4 files changed, 189 insertions(+), 4 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 4df7b0a437a8..5302ab4aa260 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3172,6 +3172,10 @@ int vmemmap_remap_free(unsigned long start, unsigned long end, unsigned long reuse); int vmemmap_remap_alloc(unsigned long start, unsigned long end, unsigned long reuse, gfp_t gfp_mask); +int vmemmap_remap_demote(unsigned long start, unsigned long end, + unsigned long reuse, unsigned long demote_nr_pages, + unsigned long demote_mask, + unsigned long demote_map_pages, gfp_t gfp_mask); void *sparse_buffer_alloc(unsigned long size); struct page * __populate_section_memmap(unsigned long pfn, diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index c540c21e26f5..c82d60398c16 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -264,6 +264,66 @@ void free_huge_page_vmemmap(struct hstate *h, struct page *head) SetHPageVmemmapOptimized(head); } +/* + * vmammap pages will be allocated and mapped such that this range which + * previously represented a single huge page will now represent a set of + * pages of a smaller size. + */ +int demote_huge_page_vmemmap(struct hstate *h, struct page *head) +{ + int ret; + unsigned long vmemmap_addr = (unsigned long)head; + unsigned long vmemmap_end, vmemmap_reuse; + unsigned long demote_mask; + unsigned long demote_nr_pages; + struct hstate *target; + + VM_BUG_ON(!h->demote_order); + if (!HPageVmemmapOptimized(head)) + return 0; + + target = size_to_hstate(PAGE_SIZE << h->demote_order); + + /* Number of vmemmap pages required to demote page */ + demote_nr_pages = pages_per_huge_page(h) / pages_per_huge_page(target); + demote_nr_pages *= RESERVE_VMEMMAP_NR; + demote_nr_pages -= RESERVE_VMEMMAP_NR; /* pages currently present */ + + /* + * mask to identify where within the range new smaller pages will + * reside. This will be used to map new vmemmap pages. + */ + demote_mask = ((unsigned long) pages_per_huge_page(target) * + sizeof(struct page)) - 1; + + vmemmap_addr += RESERVE_VMEMMAP_SIZE; + vmemmap_end = vmemmap_addr + free_vmemmap_pages_size_per_hpage(h); + vmemmap_reuse = vmemmap_addr - PAGE_SIZE; + /* + * The range [@vmemmap_addr, @vmemmap_end) represents a single huge + * page of size h->order. It is vmemmap optimized and is only mapped + * with RESERVE_VMEMMAP_NR pages. The huge page will be split into + * multiple pages of a smaller size (h->demote_order). vmemmap pages + * must be callocated for each of these smaller size pages and + * appropriately mapped. + */ + ret = vmemmap_remap_demote(vmemmap_addr, vmemmap_end, vmemmap_reuse, + demote_nr_pages, demote_mask, + RESERVE_VMEMMAP_NR, + GFP_KERNEL | __GFP_NORETRY | __GFP_THISNODE); + + if (!ret) { + int i; + + for (i = pages_per_huge_page(target); + i < pages_per_huge_page(h); + i += pages_per_huge_page(target)) + SetHPageVmemmapOptimized(head + i); + } + + return ret; +} + void __init hugetlb_vmemmap_init(struct hstate *h) { unsigned int nr_pages = pages_per_huge_page(h); diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index cb2bef8f9e73..44382504efc3 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -13,6 +13,7 @@ #ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP int alloc_huge_page_vmemmap(struct hstate *h, struct page *head); void free_huge_page_vmemmap(struct hstate *h, struct page *head); +int demote_huge_page_vmemmap(struct hstate *h, struct page *head); void hugetlb_vmemmap_init(struct hstate *h); /* @@ -33,6 +34,11 @@ static inline void free_huge_page_vmemmap(struct hstate *h, struct page *head) { } +static inline int demote_huge_page_vmemmap(struct hstate *h, struct page *head) +{ + return 0; +} + static inline void hugetlb_vmemmap_init(struct hstate *h) { } diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index bdce883f9286..ac2681bf006b 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -43,6 +43,15 @@ * @reuse_addr: the virtual address of the @reuse_page page. * @vmemmap_pages: the list head of the vmemmap pages that can be freed * or is mapped from. + * @demote_mask demote specific. mask to know virtual address of + * where to start mapping pages during a demote operation. + * @demote_map_pages demote specific. number of pages which mapped for + * each demoted page. + * @demote_tmp_count demote specific. counter for the number of pages + * mapped per page. + * @demote_tmp_addr demote specific. when more then one page must be + * mapped for each demoted size page, virtual address + * of the next page to be mapped. */ struct vmemmap_remap_walk { void (*remap_pte)(pte_t *pte, unsigned long addr, @@ -51,6 +60,10 @@ struct vmemmap_remap_walk { struct page *reuse_page; unsigned long reuse_addr; struct list_head *vmemmap_pages; + unsigned long demote_mask; + unsigned long demote_map_pages; + unsigned long demote_tmp_count; + unsigned long demote_tmp_addr; }; static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, @@ -262,6 +275,51 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); } +static void vmemmap_demote_pte(pte_t *pte, unsigned long addr, + struct vmemmap_remap_walk *walk) +{ + pgprot_t pgprot = PAGE_KERNEL; + struct page *page; + void *to; + + if (!(addr & walk->demote_mask)) { + /* head page */ + page = list_first_entry(walk->vmemmap_pages, struct page, lru); + list_del(&page->lru); + to = page_to_virt(page); + copy_page(to, (void *)walk->reuse_addr); + set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); + /* + * after mapping head page, set demote_tmp_reuse for + * the following tail page to be mapped (if any). + */ + walk->demote_tmp_count = walk->demote_map_pages; + if (--walk->demote_tmp_count) + walk->demote_tmp_addr = addr + PAGE_SIZE; + } else if (addr == walk->demote_tmp_addr) { + /* first tall page */ + page = list_first_entry(walk->vmemmap_pages, struct page, lru); + list_del(&page->lru); + to = page_to_virt(page); + copy_page(to, (void *)walk->reuse_addr); + set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); + if (--walk->demote_tmp_count) { + walk->demote_tmp_addr = addr + PAGE_SIZE; + } else { + walk->demote_tmp_addr = 0UL; + /* remaining tail pages mapped to this page */ + walk->reuse_page = page; + } + } else { + /* remaining tail pages */ + pgprot_t pgprot = PAGE_KERNEL_RO; + pte_t entry = mk_pte(walk->reuse_page, pgprot); + + page = pte_page(*pte); + set_pte_at(&init_mm, addr, pte, entry); + } +} + /** * vmemmap_remap_free - remap the vmemmap virtual address range [@start, @end) * to the page which @reuse is mapped to, then free vmemmap @@ -327,11 +385,9 @@ int vmemmap_remap_free(unsigned long start, unsigned long end, return ret; } -static int alloc_vmemmap_page_list(unsigned long start, unsigned long end, +static int __alloc_vmemmap_pages(unsigned long nr_pages, int nid, gfp_t gfp_mask, struct list_head *list) { - unsigned long nr_pages = (end - start) >> PAGE_SHIFT; - int nid = page_to_nid((struct page *)start); struct page *page, *next; while (nr_pages--) { @@ -348,6 +404,21 @@ static int alloc_vmemmap_page_list(unsigned long start, unsigned long end, return -ENOMEM; } +static int alloc_vmemmap_pages(unsigned long nr_pages, int nid, + gfp_t gfp_mask, struct list_head *list) +{ + return __alloc_vmemmap_pages(nr_pages, nid, gfp_mask, list); +} + +static int alloc_vmemmap_range(unsigned long start, unsigned long end, + gfp_t gfp_mask, struct list_head *list) +{ + unsigned long nr_pages = (end - start) >> PAGE_SHIFT; + int nid = page_to_nid((struct page *)start); + + return __alloc_vmemmap_pages(nr_pages, nid, gfp_mask, list); +} + /** * vmemmap_remap_alloc - remap the vmemmap virtual address range [@start, end) * to the page which is from the @vmemmap_pages @@ -374,7 +445,51 @@ int vmemmap_remap_alloc(unsigned long start, unsigned long end, /* See the comment in the vmemmap_remap_free(). */ BUG_ON(start - reuse != PAGE_SIZE); - if (alloc_vmemmap_page_list(start, end, gfp_mask, &vmemmap_pages)) + if (alloc_vmemmap_range(start, end, gfp_mask, &vmemmap_pages)) + return -ENOMEM; + + mmap_read_lock(&init_mm); + vmemmap_remap_range(reuse, end, &walk); + mmap_read_unlock(&init_mm); + + return 0; +} + +/** + * vmemmap_remap_demote - remap the optimized vmemmap virtual address range + * for a huge page to accommodate splitting that huge + * page into pages of a smaller size. That smaller size + * is specified by demote_mask. + * @start: start address of the vmemmap virtual address range to remap + * for smaller pages. + * @end: end address of the vmemmap virtual address range to remap. + * @reuse: reuse address. + * @demote_nr_pages: number of vmammap pages to allocate for remapping. + * @demote_mask: mask specifying where to perform remapping within the passed + * range. + * @demote_map_pages: number of pages to map for each demoted page + * @gfp_mask: GFP flag for allocating vmemmap pages. + * + * Return: %0 on success, negative error code otherwise. + */ +int vmemmap_remap_demote(unsigned long start, unsigned long end, + unsigned long reuse, unsigned long demote_nr_pages, + unsigned long demote_mask, + unsigned long demote_map_pages, gfp_t gfp_mask) +{ + LIST_HEAD(vmemmap_pages); + int nid = page_to_nid((struct page *)start); + struct vmemmap_remap_walk walk = { + .remap_pte = vmemmap_demote_pte, + .reuse_addr = reuse, + .vmemmap_pages = &vmemmap_pages, + .demote_mask = demote_mask, + .demote_map_pages = demote_map_pages, + }; + + might_sleep_if(gfpflags_allow_blocking(gfp_mask)); + + if (alloc_vmemmap_pages(demote_nr_pages, nid, gfp_mask, &vmemmap_pages)) return -ENOMEM; mmap_read_lock(&init_mm); From patchwork Mon Aug 16 22:49:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12439453 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CAB69C4338F for ; Mon, 16 Aug 2021 22:50:34 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 67AAA60ED5 for ; Mon, 16 Aug 2021 22:50:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 67AAA60ED5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 33A3C8D001B; Mon, 16 Aug 2021 18:50:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2C5FE8D0014; Mon, 16 Aug 2021 18:50:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 02E908D001B; Mon, 16 Aug 2021 18:50:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0020.hostedemail.com [216.40.44.20]) by kanga.kvack.org (Postfix) with ESMTP id D8D7D8D0014 for ; Mon, 16 Aug 2021 18:50:29 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 71320824999B for ; Mon, 16 Aug 2021 22:50:29 +0000 (UTC) X-FDA: 78482439378.03.6458738 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf20.hostedemail.com (Postfix) with ESMTP id E79E7D0075A8 for ; Mon, 16 Aug 2021 22:50:28 +0000 (UTC) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GMlZKG030789; Mon, 16 Aug 2021 22:50:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=OXjOjDBb4PYJyOxgsU6x+jt+sMCOoNtKWtWK4nozLmU=; b=rNKOhnjNDlDBrwLQXJ468eWZIX2RRQDicrZsU6lQt9vizYAGTNT9p7bqXEH/bEIySUru 0G5xg9S+tgYJhfuBlyBwK75o9ZFrdNLPofXdVLKJLj/HNB/GBIdCrLhU6ktjH/OOdLII e+v+3EhCUVwmOYF8I0WRg2jws1O5Z7b+W1txh8kPMFebMPw/7jDcI2nZEC35nPsSTio6 cK/GDJOZiabCW50q58B5/M7Q9hoOZ8PVpHl11jDm5D3vtBR002+xG6opBzgfDRYN1dbs fdfqU1Ajzk1EfbajvvgJvFX+gdgCKjuTW+rZBe12irIK3CtGZwGmK2dKANHdPcHefwj0 tA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=OXjOjDBb4PYJyOxgsU6x+jt+sMCOoNtKWtWK4nozLmU=; b=WTtAyH9iwkgTp7jNpQ4jYL01ySM2trr6T72fkTYjbE7NM8VaToLdu6iIlMcxsQFpv7Z4 ffjHJ/EZHbMzcMczHHoyrmeje34NYbt5mnJgBjGITTatlhEXTI3vFBkZQpSx+evzBL1S 2lRAT07zagIDQ5cLA4zMRSzESYhx5B2XIuaUxbMbpyiWbSuNZoqEarRvOSQaEudbzm3x uTmORAx8elgkQMDn/ugY0xew2aVbvswFuLl3E0u2ijDer36b4ZSWteNPnBYzr8WUG2+b DrB3pVoGb4v5Q1e7Xj4lB6O/Tph5rBNi/ChTLozP+0T6W9cDygXu5X0ehIGLYZmNTfF1 Rg== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 3af3kxu6vk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 22:50:23 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GMiu5w040103; Mon, 16 Aug 2021 22:50:22 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2168.outbound.protection.outlook.com [104.47.59.168]) by userp3030.oracle.com with ESMTP id 3ae2xy1427-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 22:50:21 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Pz048eLHeMhQgUdiIEf5G2TzGa8TOU5elFeiIIQwTjWAOFd9KfpNRxfYzvRtMTmohjrXxCI+UZhscdnPWNu6XvPdmQkQEqCwHE6UUdvg2ylOCaWa5z3MOk2+0yJsXCITyhsq9CtV8sQ4ZsNjUGgwjNbIg+zADu0WGMn9+9cSfGFDKEwlGBGFa8gPgpIN1GDgKgW03gk0EH5fsFO7L4HO0fI5IKXe/Kc86vePLEgpiGCNTwuwGRQxMMyTKWbB6LyGdnlMcjzGvdXHujKp6CI5tUrF/92YFxXqDqOaqEhGSEU9jpMLDtppOSWLTOFTM7DUrcH4ejHlY/P3b0B11+4Exg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OXjOjDBb4PYJyOxgsU6x+jt+sMCOoNtKWtWK4nozLmU=; b=hu8JMwbv0/JKYUCITQChNNjF0bpa841QKJythCijGiXaFQT0x1UhG6F5GNkp7miiLRCRzdqrGkhOyW4bqsa65NikDy7uS1Klzz1dOdwF9tWPBPoJycRIMoDoJKtPv4eBecHcVVVFJl2eiF9G+5sFkQdM6wcYrBPJWz8RdfmgJ4+/zRmW9GjjY2TIq9lbfFYvjKOcW1EdzWtmgIpJyHVw2/ofQi+tUJmFc4IPeAjZheFBtfK3uURdixXlJ0pXcoxlZx9/9vxJEZp/WbZ0o3/Wag+kEsq01RhsFEdVf/VbGHGmC19wQZoorR5Gx3dNM2Mnx+omVWDExO1OWzzJunkEDw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OXjOjDBb4PYJyOxgsU6x+jt+sMCOoNtKWtWK4nozLmU=; b=bBJSKgBm0Gf0pBwaSaPhFxsfXm18LqJkzlsq7DBujeK6TQDbw6rsLzMwbHju/Z2s/M84FaMwwmxr8fobhOar8Babo79DWFhiuK+z0fW7WdfVllN+nCwf6xcJ4jU48O292WCKsAearEw/yUGJdnhp6YLg0EARcickwjPIuh9hI5s= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3397.namprd10.prod.outlook.com (2603:10b6:a03:154::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.14; Mon, 16 Aug 2021 22:50:19 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 22:50:19 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Michal Hocko , Oscar Salvador , Zi Yan , Muchun Song , Naoya Horiguchi , David Rientjes , Andrew Morton , Mike Kravetz Subject: [PATCH 7/8] hugetlb: prepare destroy and prep routines for vmemmap optimized pages Date: Mon, 16 Aug 2021 15:49:52 -0700 Message-Id: <20210816224953.157796-8-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210816224953.157796-1-mike.kravetz@oracle.com> References: <20210816224953.157796-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW3PR06CA0030.namprd06.prod.outlook.com (2603:10b6:303:2a::35) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MW3PR06CA0030.namprd06.prod.outlook.com (2603:10b6:303:2a::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.14 via Frontend Transport; Mon, 16 Aug 2021 22:50:18 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 5bc5d216-7c56-402e-4f19-08d9610838d8 X-MS-TrafficTypeDiagnostic: BYAPR10MB3397: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1091; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4fGfopAN/0KUaFb+JL7A7mx2TkJ5eyvVq0iaU4mf5QwKmYwVBhvq17QyqjFF/yoZLbXP4mH6+5IeljVj9j1y+g4fP4xqjxo2rlV8Czp7yRWmQ0Y3Kk0LfSdznCPobLI1WBK+gi3LVzsHT3GnuAsaLRSE1uTmu1adyWwEjaTv1P9Clh08ivSHUNbQfjmVaqJDv3tfRfjLivd5e4Bo0l52Ph4+xAM5ZX0/tOhSgz+NXSBZgtf0u44plinMPFG7XKV4+wZdH6UpISFo4w8cNhMRB+BCepQr+MjgH+xC1/TYKneIRf6V3bIsSUcnySFDl3Hc67vJfydfW5T1BQX1s1/+7lTISgTBKOEiGaLLWBs38LEko75376zz6Pp+WCJU/HM22v/juSM2FxzvJHcgB8RpW3SXBiPaOFiWnfYqYybSjbhL7RSKLMZU6MTM7F3Sb7kkiXLtEb+6o1szP0DMTRT/VaOhxJEEbt1l+Y2XxD1EtYzebK1JR6cmupPrX+1wkltmISrrWrLgPJPmXpaGggA9jjAf9kJ3cRFRoCTdTXv5b0HxI6HBnrq11AujZTXzqIYT3PP/w1TLZyJAFqZoPriV8aEazz324aFAILXpSshBNXWB7YGte8/+r4DuTGXfjGKDwNz2YusaKRJH3Y5pPlXwnQNLB8jiVI/50hDRqlC8cXzenCqO3GhbPRWfqMaMhYVJxywV1B49AIiP5fP2bSgcHQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(366004)(346002)(396003)(376002)(39860400002)(4326008)(54906003)(6486002)(956004)(2616005)(107886003)(44832011)(316002)(7696005)(8936002)(8676002)(52116002)(1076003)(83380400001)(38350700002)(38100700002)(2906002)(6666004)(86362001)(26005)(66556008)(66476007)(478600001)(186003)(36756003)(5660300002)(7416002)(66946007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: cATY3WoJU8cmkh1Urghq0u46z0ORuaomntxo/+/fL8faEWchM1gTHVIqRUrf/86U9uP7IxQ41Op4OavNTqvM/ZdxaIt22SDu+KuuIdwohVBxDWP0pu2RxiQvOBh1LGbghtDinXxUn7G7yxtcCmw/iFNH5zQM+blW5q6eQr7OknUG8i4zfF0R3yfsk1l1CoXeFpD5WeL3SAIccqSOJo34698NsilZzgiPjsKNY3Z7C2sCukjiS4QZ879X3r19+8iHwdWv2DW58ZDrtcVeFqBrjo3O+7eCL1k6PJjd6wQUt6g4hVQkz2rr5MSXYwli/TgSAWmiDd9r3ogOgAycU1f7+dknZzxxgqggndERiIE2TjkgBD5sYzXLpNYWgGcea/LY2w6s3YrUbzNCwG0Jox9qjxHGpJqq7jT0hRBWL81TEKKqk/ErNeqWGV/9G9w6WabexuZx3/q1PJBB3v5WQYRz23Ye9ANN+GhzpH1S7l+k9d3lRAdhKm7Q0+R151ob/WVD4AFv7pk7T1FiGpAOMScktvKWoXuwKMQZ/M7lp1vPykTTH1+tslkpmheJvWHDD77XhyZMKq7TRxfN6bFp6jEW0lAJZyZR9foEzPo44OiFcCI1iVdSNeD93xxtH7Syc0krFpiEScte07t+rgk45SwJ3XHQODjuPMsCQGQ+6y27A3BVaMKbxxhAprnWKI7JlwKWVulNzyDST+QiuZOuQPzBUBeVGX2g/ZJcdhgccTWgOOx4jTtOt/FfRUDEvDxTaIHYn57B5gjLh+0fzRctDzceCrTqnGJSuWKxeiME8wLV9VJe62xm78Smh9Kwl4ja/bOb55SZ3VIswQcYqUOOX3e+L4FLXwjprsFk6B6T1il5RWtejTSUrBrEsyY9ypQwNPzRf9AYXqCxHAhXMhidURQJ+O2eUb8S9RmHX2WyQOuwOOhBGy7Hu60c688D6zMjW0dqSPpjLi9vQdSkuFzorecj8E0yVWipbL4K/gvmTgCJ5fSHYpNZWRy+Aj+RnvsvHrdbCBmPnbWU/qW/cJUaDCf2iMY/N5vOPn5eS7XOeWTgWm5bKY667q9Wh4I8Aw3IILv1ebjNtzfuNvJg8eXQBMiiIA5eSpZ1sSiRh384A8cwtXiVXIBbfKOMuOXDzIOo7NxDt9G5pv6AGASD+owEccjnLe6xZHV/xLclKlrCdrhvulWZBt2NcmUmC0Id0c3VdvuF7nyW+FtEluO1eKoWNy8Xq0mpyFD1icqzZvlLU7caqn0N/JzKhGb0Pt3cgxvHD9Sfko0TjynLKMb38CZWMG2RBr/FzHaYUI99S6csYtMl9AgP7Dr5Sj7CLiJUilpmqByR X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5bc5d216-7c56-402e-4f19-08d9610838d8 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 22:50:19.3348 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: q2n8cIDgU7S6dpjMtaG8QR7Dx2x991oFBe54qeWvim1847zWOfmV3yoHE3kz1cvMSjOkaUZQ2Snq4RAjIbr4nw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3397 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 spamscore=0 adultscore=0 suspectscore=0 phishscore=0 mlxlogscore=999 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160142 X-Proofpoint-GUID: 19BOXqtOCKFp4HB3XFBG71uDl4-g7Mw_ X-Proofpoint-ORIG-GUID: 19BOXqtOCKFp4HB3XFBG71uDl4-g7Mw_ Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=rNKOhnjN; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=WTtAyH9i; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=bBJSKgBm; spf=none (imf20.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: E79E7D0075A8 X-Stat-Signature: 5bkh4tgok5yzxm71zohmsb5uy774i5or X-HE-Tag: 1629154228-108185 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When optimally demoting huge pages with vmemmap optimizations, the routines which destroy and prep hugetlb pages need to be modified. Currently, these routines expect all vmemmap pages to be present as they will write to all page structs for tail pages. To optimallly handle demotion of huge pages not all vmemmap pages will be present. Only those pages required for the demoted pages will be present. Therefore, the destroy and prep routines must only write to struct pages for which vmammap pages are present. Modify destroy_compound_gigantic_page_for_demote and prep_compound_gigantic_page_for_demote to take vmemmap optimized pages into account. Use the hugetlb specific flag HPageVmemmapOptimized to determine if this special processing is needed. These modifications will be used in subsequent patches where vmemmap optimizations for demote are fully enabled. Also modify the routine free_huge_page_vmemmap to immediately return if the passed page is already optimized. With demotion, prep_new_huge_page can be called for vmemmap optimized pages. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 17 +++++++++++++++-- mm/hugetlb_vmemmap.c | 12 ++---------- mm/hugetlb_vmemmap.h | 10 ++++++++++ 3 files changed, 27 insertions(+), 12 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 6d43523a1a4d..259b840718f1 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1265,12 +1265,18 @@ static void __destroy_compound_gigantic_page(struct page *page, unsigned int order, bool demote) { int i; - int nr_pages = 1 << order; + int nr_pages; struct page *p = page + 1; atomic_set(compound_mapcount_ptr(page), 0); atomic_set(compound_pincount_ptr(page), 0); + if (demote && HPageVmemmapOptimized(page)) { + clear_compound_head(page); + nr_pages = RESERVE_VMEMMAP_SIZE / sizeof(struct page); + } else + nr_pages = 1 << order; + for (i = 1; i < nr_pages; i++, p = mem_map_next(p, page, i)) { p->mapping = NULL; clear_compound_head(p); @@ -1527,6 +1533,7 @@ static void __update_and_free_page(struct hstate *h, struct page *page) return; } #endif + prep_compound_page(page, huge_page_order(h)); __free_pages(page, huge_page_order(h)); } } @@ -1715,9 +1722,14 @@ static bool __prep_compound_gigantic_page(struct page *page, unsigned int order, bool demote) { int i, j; - int nr_pages = 1 << order; + int nr_pages; struct page *p = page + 1; + if (demote && HPageVmemmapOptimized(page)) + nr_pages = RESERVE_VMEMMAP_SIZE / sizeof(struct page); + else + nr_pages = 1 << order; + /* we rely on prep_new_huge_page to set the destructor */ set_compound_order(page, order); __ClearPageReserved(page); @@ -1761,6 +1773,7 @@ static bool __prep_compound_gigantic_page(struct page *page, unsigned int order, } else { VM_BUG_ON_PAGE(page_count(p), p); } + p->mapping = TAIL_MAPPING; set_page_count(p, 0); set_compound_head(p, page); } diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index c82d60398c16..01c2cc959824 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -172,16 +172,6 @@ #include "hugetlb_vmemmap.h" -/* - * There are a lot of struct page structures associated with each HugeTLB page. - * For tail pages, the value of compound_head is the same. So we can reuse first - * page of tail page structures. We map the virtual addresses of the remaining - * pages of tail page structures to the first tail page struct, and then free - * these page frames. Therefore, we need to reserve two pages as vmemmap areas. - */ -#define RESERVE_VMEMMAP_NR 2U -#define RESERVE_VMEMMAP_SIZE (RESERVE_VMEMMAP_NR << PAGE_SHIFT) - bool hugetlb_free_vmemmap_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON); static int __init early_hugetlb_free_vmemmap_param(char *buf) @@ -250,6 +240,8 @@ void free_huge_page_vmemmap(struct hstate *h, struct page *head) if (!free_vmemmap_pages_per_hpage(h)) return; + if (HPageVmemmapOptimized(head)) /* possible for demote */ + return; vmemmap_addr += RESERVE_VMEMMAP_SIZE; vmemmap_end = vmemmap_addr + free_vmemmap_pages_size_per_hpage(h); diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 44382504efc3..36274bf0256c 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -10,6 +10,16 @@ #define _LINUX_HUGETLB_VMEMMAP_H #include +/* + * There are a lot of struct page structures associated with each HugeTLB page. + * For tail pages, the value of compound_head is the same. So we can reuse first + * page of tail page structures. We map the virtual addresses of the remaining + * pages of tail page structures to the first tail page struct, and then free + * these page frames. Therefore, we need to reserve two pages as vmemmap areas. + */ +#define RESERVE_VMEMMAP_NR 2U +#define RESERVE_VMEMMAP_SIZE (RESERVE_VMEMMAP_NR << PAGE_SHIFT) + #ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP int alloc_huge_page_vmemmap(struct hstate *h, struct page *head); void free_huge_page_vmemmap(struct hstate *h, struct page *head); From patchwork Mon Aug 16 22:49:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12439451 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90965C4338F for ; Mon, 16 Aug 2021 22:50:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3E5A160C3F for ; Mon, 16 Aug 2021 22:50:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 3E5A160C3F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 391408D001A; Mon, 16 Aug 2021 18:50:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 311B38D0014; Mon, 16 Aug 2021 18:50:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0EEDF8D001A; Mon, 16 Aug 2021 18:50:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0144.hostedemail.com [216.40.44.144]) by kanga.kvack.org (Postfix) with ESMTP id E65D58D0014 for ; Mon, 16 Aug 2021 18:50:26 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 9754D22873 for ; Mon, 16 Aug 2021 22:50:26 +0000 (UTC) X-FDA: 78482439252.29.09801AD Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf23.hostedemail.com (Postfix) with ESMTP id 2705490090E6 for ; Mon, 16 Aug 2021 22:50:25 +0000 (UTC) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GMmKIa029214; Mon, 16 Aug 2021 22:50:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=Zw5ULP+Lfflz7uyYnSO4JR8S10OYQiow+gaHUf8TpWo=; b=jfPdtbnttw2JRlDgkvCtNRCwBbEL9FzmDYW/d16k/rJz8EqHtcSVEm4pc1iBShneX8Do e9YRVlAwDks+rwNDO98vU+OuKX0lJLmZl7RPWrBt/KHAY5xwYExw9VBwmLAWcjfwwW2G MvJQcc+zvCkGhoq8bEaFawX2cNkn7tiVu9VgGHb76AMF8O3edNkIMzmN+l7yM2/7xVs+ kN4hh73ufkpDJj9xwOczzAv5MHN3cRoO5BEgF97j2VPIHeD0GIjtBnMnzGWsPiYV7uw1 dDJ7fBMQHt7MYz3ikKf4d2AfRCD70Q7pOi03dSwoledsC+gdoDGCrFSRFs4FI06ouBfo hw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=Zw5ULP+Lfflz7uyYnSO4JR8S10OYQiow+gaHUf8TpWo=; b=pYRof05wu6XTDPCsJ74jfze1JE9sPSh+maqThC/+TvAtHwK4hFMOhpWL4sdtEWNxYowG RWATKtSZYaWPCRVH82tAV/f3Dyt58yocArLo0Q7y7sjleIiX09GgHAe9/KlTlyMaY0GP cAmCkUuVgWW0AmlMLb5A+eDIs12kkL0tSWiU17P9kdoSYUzurXCOziZ792G9kNAnfqfC OJtgEVGaTJe4bFKio5L9mZbMPAAS4MfOvfCWYkSpb1BQcbCG3e/G7gljAZ0p2qydEEdt PDGnszygehBPqgYivGns+2zeSQTdFV5Nyg4rLHI4T6NCqWwLNQTYuhuaDgZwxVwI1qZg 6w== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 3afgmbabbd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 22:50:23 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GMiu5x040103; Mon, 16 Aug 2021 22:50:22 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2168.outbound.protection.outlook.com [104.47.59.168]) by userp3030.oracle.com with ESMTP id 3ae2xy1427-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 22:50:22 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OHTRMVw0oltdYQfl7PrrAUECYM38lT96ScvCuwb3rR1NlZw27emjruxLUYKBgf1HQUPrnZLoUBlgDDkzE+JLLLZWI+ViiI5LUVVt9pT3cgeNiv+6iWGdkQsRASmCOWfGwxFgY1yGYHLQ3xUOCErcJojvBow3uKtp6fNw2cmzCdRvBYd8xyROCqdD4tjggqBakSMEH526hGBIkLXtEZpogdzldRSxCYXrywbU/h8FO9pCNr81BO09wGl+dJvyNVg7j3xeh99ZGPzeuDoLv3dJz6Y+1a80tCBCHiZypDCzz5q4tQvqGUNXusSQOvMQf7WeIeeF825gAQinNktuaio+MQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Zw5ULP+Lfflz7uyYnSO4JR8S10OYQiow+gaHUf8TpWo=; b=cVhKQYaX0yW+gP4NzPViUM1Bwypzl9AY7aSWlNeo1AcKQMIV50KG3gFIHmC3PRKPv8RdWDelx44XqOehH3DGjYakVRIMsZ+ESb/1/bpnD5wDj/UfsfajNYUecLy1mDEO4sxzfbD06zQ5Sm6SL/NCwwNj2IbiNIkiyMpyqEOAG4wAjcFK89kxmZWB9DqvGkLLxvHo0x9hBas1r3cNlcas7hS0AFqLTfweIhw9Yv1xa00Bj9zMMWgq63D4BAp7TaAJKicaPa6HrRDpqIsI9+GMJnxDDFyS9JmvppdBuAQN+++FT2kvFsSC7f5xRbELH48xTHWG3NfRVkpXd+HayK4/2w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Zw5ULP+Lfflz7uyYnSO4JR8S10OYQiow+gaHUf8TpWo=; b=SRwhje2Iwv+Yd/wELhY9XMrvT8hcpqQet8f666pcn3ad9FGLcDNPAE1FgghyIYUFLIvUePExwMDWdrymgMPIRFWBVpcyGT2F5CMK4AMeGoosL+kxT78Ffy6er3tuWbo0MRL4+Jl7mjRa7KfJPyiJt8E//OodpqRNxHNwUZ1847I= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3397.namprd10.prod.outlook.com (2603:10b6:a03:154::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.14; Mon, 16 Aug 2021 22:50:21 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 22:50:21 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Michal Hocko , Oscar Salvador , Zi Yan , Muchun Song , Naoya Horiguchi , David Rientjes , Andrew Morton , Mike Kravetz Subject: [PATCH 8/8] hugetlb: Optimized demote vmemmap optimizatized pages Date: Mon, 16 Aug 2021 15:49:53 -0700 Message-Id: <20210816224953.157796-9-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210816224953.157796-1-mike.kravetz@oracle.com> References: <20210816224953.157796-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW3PR06CA0030.namprd06.prod.outlook.com (2603:10b6:303:2a::35) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MW3PR06CA0030.namprd06.prod.outlook.com (2603:10b6:303:2a::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.14 via Frontend Transport; Mon, 16 Aug 2021 22:50:20 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4dd8e10f-70be-482a-e7c6-08d9610839e4 X-MS-TrafficTypeDiagnostic: BYAPR10MB3397: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4303; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4eGgo0DEXmJ/T2rD7uymOcSDMEKS18xpED/LIDPwy6NozDelWr7O6n9bLpOr/YB74mEpi0LoBWZZ4RZz7tuScsc/+lvNeeamH69314LQXpumQpRnU9zkZZlsDVDJ9zvsESULvHcY9ccZT509l0Wg2hvSHs9cDEoxPEtL8P2ywo633F8KnDqd+kATgyA6UAVSPIa4HbccD/s3osi+kbQN4JNnLO+e6u3HfBSPj31Bdddynen8OIDEqDAIl0W0Q6FUsD6fi9p+t4T4a4ba8UoT8K+pEDu+kM87VxzFdDaxWXqXFjC9TPV15j3CYLoz3F8HGb2/MekYhjJM2QatdtkCsVoaLqAR9RBt5yL9pry5AGajm7nB80f6d1w5zdGJgk1O8Dm7x4kiPh/y6/gr5Dvrs1Gu8bX8NARFocuNBUXRcu+pYljHpQ9Xv6Vptb5fTmBHZZ4ctKdoX7TXL9MZhtqgSJqikf5vOibzOdQorUHbINir7SefaNJsu6SN8pfpbCqHmQToW0xrXjO6HGKzUV7YWvEFJTPwxXDQQPbMusynoiFYKERmZOil7VcJ18Sg+NaRmZoKx09kVV5868M4cK1MnpKUs3a+FF0WqA2benCfeoqztkJ5xslXnE8fIeVEjGIAEBa5NTCc6e9ij64yW/hc8UhmgW4m4JSUmPQu+/jQzgwXsVvvkqUX/cuOkDqgpeBywnjpDoB+NiHe9dfXpSuaRQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(366004)(346002)(396003)(376002)(39860400002)(4326008)(54906003)(6486002)(956004)(2616005)(107886003)(44832011)(316002)(7696005)(8936002)(8676002)(52116002)(1076003)(83380400001)(38350700002)(38100700002)(2906002)(6666004)(86362001)(26005)(66556008)(66476007)(478600001)(186003)(36756003)(5660300002)(7416002)(66946007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: AUzY9/XDSB804j84EDZJ3VVT4eSG06eKyFBAsIYOA9B9NXrdzcSb3OQllAXGLtNGiKjNLE8qR1E6d468rHYTlTP59I/wcEZhtOmdMEh7i+T7VZ/itq2rfr96RFtoCja1F5T7UIhMBFY0V0h+zML6c7L259HudPV552ChPANDV/W3E3AQHPBvZbITBQGIExBqQf0U/JozpLPF8uU5yYvMaR/toasvA0gR2g5vFgGWOKfY9QKlx/CNkCVGa7OJvbsv5O7fu4tZ/MjaLqkTS/LQ27v4XySfdlnTLG5gKooHizryN3Y0NEqVJXE3H89OksbltdfwEq2/Kj5B9H24RwIuaRIW+QFRsBEDB25JWM5CZLllPYjOTo1Ed6tBFhM2JAEFd1IU4ASwnLJ5EpIi6BHft0eB83CCiDsVqgjY0hjEMXN85NxUvnf+/otF5A7NqMTfp3vPR7zJsd4PdNCuOZiDNR3PMthG6fGvIalmdNYVQ0tyEkdb3EKKGfjT9XeR9TmOzW0Sdjp9fGMm8s5B7fLg8S+KQsQLPO6rGX3H7wSZmShaFdHe68sI0+5f3Slbd54Im1VaYzlPt9JzswkXtmAipgdxhawmpZ8iOdomeFjc+E5A9OjzNdOrGvLV9/3ZlHUKoEUWrYUkTm+q5tmfyhXew3e5c59ewvrTnIXPva0Ag4mNyGwVhdVaSuMTYWAUiMZVNgfRxPNwnM518S5rbN1zJrlAd0ykL9qV19v4DbnO8kA0opD8mXLkIbO37P0fHnL5E86mB7Wu12QyUNECbqf/qkblKRGTQpt6Q0Z70yvhWfAnbwJjkUfY7M1vtxs4HN0CA3f5kfrsyrSjGEfUWbiHTgAC93LcGHLC9A21neVQ8IBIEAKsYXxxlfNbfJVtwa9nAxBKnxNpgwYGWEVv9NFMrx0laqssVxpbIJ9CfWkIGN5tmNDcVWN8WkZDchblzC2QEinxgrc8dfYysP0YTrHPZOQ/A+V0ZulU6UbRl5aWOBN1gnDoARi2qE8ytyJG3vJB9bbTPAwTtovvPZtTTiEn9A3IYdE91IBYn/DMadRR2bW0cUIFMX8IQIKuveV1Be74ZL7zaYGKgBi4YFvJz3+zqw9+CBgLPizoy4fR5okUN/e418xsEflK+b+M+dD5wckhqoHoh0PQKm8iyDuMPJSu1DsNdeogX7JQmXq4K1XP0Sdg3D3LJMICMwgdfedn7Gcw2tl4aABM9EnBfUD2xM9hTl+yFt+E5xcVPLTtPLFNDItyDwMgimYv7CYujJUi7QWuYVAMkoUf6KOAYmCUiBjeL8lLYayQbjbYPjfas2QxoIkIDrwz+07VtLCpDxJdF7YF X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4dd8e10f-70be-482a-e7c6-08d9610839e4 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 22:50:21.1229 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: +wlji959zyBnJC2S8Ds0TDwKk5iyHM/xZonHskNGKTckPNvFMbl4uWWjR0lCjNbFQ1cyPijJ2MFz7My09USUvQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3397 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 spamscore=0 adultscore=0 suspectscore=0 phishscore=0 mlxlogscore=999 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160142 X-Proofpoint-ORIG-GUID: lJQNXD2hfqatuX9hzDjJb_zPMwVy9Br9 X-Proofpoint-GUID: lJQNXD2hfqatuX9hzDjJb_zPMwVy9Br9 X-Rspamd-Queue-Id: 2705490090E6 Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=jfPdtbnt; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=pYRof05w; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=SRwhje2I; spf=none (imf23.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam01 X-Stat-Signature: sw9ptdzc6kpiesnnydphxm9ieyeuonxr X-HE-Tag: 1629154225-207828 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Put all the pieces together to optimize the process of demoting vmemmap optimized pages. Instead of allocating all vmemmap pages for a page to be demoted, use the demote_huge_page_vmemmap routine which will only allocate/map pages needed for the demoted pages. For vmemmap optimized pages, use the destroy_compound_gigantic_page and prep_compound_gigantic_page routines during demote. These routines can deal with vmemmap optimized pages, and know which page structs are writable. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 259b840718f1..77052ab464b1 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3330,13 +3330,14 @@ static int demote_free_huge_page(struct hstate *h, struct page *page) int i, nid = page_to_nid(page); struct hstate *target_hstate; bool cma_page = HPageCma(page); + bool vmemmap_optimized = HPageVmemmapOptimized(page); target_hstate = size_to_hstate(PAGE_SIZE << h->demote_order); remove_hugetlb_page_for_demote(h, page, false); spin_unlock_irq(&hugetlb_lock); - if (alloc_huge_page_vmemmap(h, page)) { + if (demote_huge_page_vmemmap(h, page)) { /* Allocation of vmemmmap failed, we can not demote page */ spin_lock_irq(&hugetlb_lock); set_page_refcounted(page); @@ -3348,16 +3349,36 @@ static int demote_free_huge_page(struct hstate *h, struct page *page) * Use destroy_compound_gigantic_page_for_demote for all huge page * sizes as it will not ref count pages. */ - destroy_compound_gigantic_page_for_demote(page, huge_page_order(h)); + if (vmemmap_optimized) + /* + * If page is vmemmmap optimized, then demote_huge_page_vmemmap + * added vmammap for each smaller page of target order size. + * We must update/destroy all each of these smaller pages. + */ + for (i = 0; i < pages_per_huge_page(h); + i += pages_per_huge_page(target_hstate)) + destroy_compound_gigantic_page_for_demote(page + i, + huge_page_order(target_hstate)); + else + destroy_compound_gigantic_page_for_demote(page, + huge_page_order(h)); for (i = 0; i < pages_per_huge_page(h); i += pages_per_huge_page(target_hstate)) { - if (hstate_is_gigantic(target_hstate)) + /* + * Use gigantic page prep for vmemmap_optimized pages of + * all sizes as it has special vmemmap logic. The generic + * prep routine does not and should not know about hugetlb + * vmemmap optimizations. + */ + if (hstate_is_gigantic(target_hstate) || vmemmap_optimized) prep_compound_gigantic_page_for_demote(page + i, target_hstate->order); else prep_compound_page(page + i, target_hstate->order); set_page_private(page + i, 0); + if (vmemmap_optimized) + SetHPageVmemmapOptimized(page + i); set_page_refcounted(page + i); prep_new_huge_page(target_hstate, page + i, nid); if (cma_page)