From patchwork Wed Jul 21 23:05:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12392469 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECD22C6377B for ; Wed, 21 Jul 2021 23:05:34 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8652261029 for ; Wed, 21 Jul 2021 23:05:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8652261029 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3355C6B006C; Wed, 21 Jul 2021 19:05:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2E30C6B0071; Wed, 21 Jul 2021 19:05:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 026386B0074; Wed, 21 Jul 2021 19:05:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0129.hostedemail.com [216.40.44.129]) by kanga.kvack.org (Postfix) with ESMTP id CC4E16B006C for ; Wed, 21 Jul 2021 19:05:32 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 7C733269AD for ; Wed, 21 Jul 2021 23:05:32 +0000 (UTC) X-FDA: 78388128504.08.8BA2EE6 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf08.hostedemail.com (Postfix) with ESMTP id 47F6F3000D82 for ; Wed, 21 Jul 2021 23:05:31 +0000 (UTC) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16LMuRYr020157; Wed, 21 Jul 2021 23:05:29 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=LAfpmG+F4us2CJXJQgOM8oeTtf/v302JFO/6wDULhy8=; b=InY94sd0DVNL0QRotgCDT7zT18UoHMh1nfP85IlCuTF/A1ujr0uIpZYrUzAsl+zAZpJy aktVu9TH6pskdoyyoLfRc2JPqsgjVCLW36JTfvXYF4ggYj0Lp0VVniOrKJvOJoYIfXbA nBqbKBwix8hzVps1fxwIyAZDCz7gY9ChRBCd3WxoemoKnS33Aeb+7AO1o9ee4me7F16p 7mlALPvkUAQP8gxpHam6g4afs8YgBGicm7t9LYCGu8Ruw+mtsrwywcI9rYFJi3qnauYC Ut9F8baiIgr5SZmr0y2cvKZCtgYV/jm7+nFNzrYT2685TYlwkz4LLRGlLZ6Woz7k/ytr wg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=LAfpmG+F4us2CJXJQgOM8oeTtf/v302JFO/6wDULhy8=; b=hT2lwzHrXxkNOf0h3r1JvvBEdfvcTUyFGxsyNpLZUqybnUYblrnMDBcSdL13wCJZpDve lB4igUD4SUG1uY+oXmXqmKkOBzKMcizqQVdktjZ6+8qoEORYDI/yYoJQkakVFF6V1yQr vj2AW47taHQvSSS1SV3uLOxQg60n0ReXgcb3veGK5mq0B/pzJKG0AV7cDYOubx5T+Cvm 5MZ7oj6w6o64RFLczP4Vgl8tFw8/iuLHpgIRH1ET8kOeMBAa0zOnTRZ3tEnBZfllYIc5 VciEgYtehFViiOjKwoppHRWr+QzlR+IuIsqFnUTJyxpxpfzjTpYYCIsbNvPdBhMhfmtB pw== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 39xvm7r0ua-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 21 Jul 2021 23:05:28 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16LN5NPp103248; Wed, 21 Jul 2021 23:05:28 GMT Received: from nam04-dm6-obe.outbound.protection.outlook.com (mail-dm6nam08lp2042.outbound.protection.outlook.com [104.47.73.42]) by aserp3030.oracle.com with ESMTP id 39wunmv2ae-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 21 Jul 2021 23:05:27 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FpA3Pz5yiVOe9Nr+EJGnoAVUdwpodQwPzh4BQgOxMnLAOroHditMm/Ps+Kazd9zLxMbfWBWH/ww8Tgqmvy3gmYqLgRzaavjHWr1lZ95o1jJtwwY5YTDZD3GWWikDnF9njn8LROJ504m0bHG+I4Em8N0KH0rxFcWAcnViqNg201uB1TYtBahia+CPDzq2OqLg6uR/0ApmRaBooM4mSuwPUgD7KArDzNGtvHzS7sST7iLVc7/XKWg3D/bUKnkk74kX5y8aYLvwCwex9NxdElWefa7/qosuxhqXm5gipAPNYPdys4y7DaZPYaLZ4uNURF/KORcOBlWWh9YvbcoAAPmzvw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LAfpmG+F4us2CJXJQgOM8oeTtf/v302JFO/6wDULhy8=; b=CxDYhdJhOJE2Mt0gbVXhwtRBE7BKnXxVQbjDPIvGkWWFgeW1+h6v6Sz+NdLF0AavSkrzMzznfQqiF5Yrlo+VYSuNyYJJmh8+3OdxQmp+dMRlvRDRDGqZIWUCma3lDucoU9lbZX9LTYINwDl1AL27FPn8FosJpnEJIrlMVuGlpDnY85YuFavgDIvxDav8ddh6+75U4gi7CXqjvvsC7pmSzIcNEqJ5oOEIDQRgcTuMhGbb8W+gBOTnTyRehe0pUAx9lLCGC0BdFZZqVTNpwsEZOGyoivqQTfxbUZrHz5CgAvx5J7yWLuQOP0NnjGdolQ54kUDwKCk/FDvNDrufrIju3w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LAfpmG+F4us2CJXJQgOM8oeTtf/v302JFO/6wDULhy8=; b=ex709avpsBBEMQJRkDINVfMTfeV+oz7bqveysAIjlE4yvkpZggCYWA11fg8I16OR7R3PUmhxElsSJM//zlbGQNdGD4dN9m2luCw8KQHEEZ8UKf6m12CXNOtI9G2NRPjOjc7CN1ECH4AzlachDGFm1CQXcr+qjr0vTjmxcG6ZNGI= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3735.namprd10.prod.outlook.com (2603:10b6:a03:11e::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.26; Wed, 21 Jul 2021 23:05:24 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a%5]) with mapi id 15.20.4331.034; Wed, 21 Jul 2021 23:05:24 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Michal Hocko , Oscar Salvador , Zi Yan , Muchun Song , Naoya Horiguchi , David Rientjes , Andrew Morton , Mike Kravetz Subject: [PATCH 1/8] hugetlb: add demote hugetlb page sysfs interfaces Date: Wed, 21 Jul 2021 16:05:04 -0700 Message-Id: <20210721230511.201823-2-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210721230511.201823-1-mike.kravetz@oracle.com> References: <20210721230511.201823-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MWHPR1201CA0021.namprd12.prod.outlook.com (2603:10b6:301:4a::31) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR1201CA0021.namprd12.prod.outlook.com (2603:10b6:301:4a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22 via Frontend Transport; Wed, 21 Jul 2021 23:05:23 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 5b2acb7b-2193-48ad-9faa-08d94c9c059f X-MS-TrafficTypeDiagnostic: BYAPR10MB3735: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8882; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 7RGQkEWE11VI9MtMO2rurkrJJbbskmNDJFIf/4DpJFW0U5Nf4RMUN8peaRBNZFlMebRYV69EB1IgyMBBxUp20ylfHMP+pgImvG5VJxthWYW0BNCYw3wsTzQavXTpjw4WWObuhq8CHAtZgwCeZI/kNmYXqXWxrmE1UXzG2L/LwSKb3IxDzNtU0Zl5KZzoG8soJh8rBicDxmLa83ibPM2Z7+8/EoZjvrVgP+yfwBWoFUGm0NjAFzM+xfW5m3+49I8p3qytR81W2sdYKK8bMOKuLpa2U+S0t0zfRlzc1kk6QesoSM3lrNobdy1BNa3yVa0PwI5gueNdYNPB6rfD/Z0LxxhrPmm0IUrYKLZ4ttZAkle42HMLDN49zx3dyJuZw6fOd5jHZQAnLoj0J6qJqlKrtPRcMQ8eObw2+Zd2lMeqhIkCvaW4REvwg3/y3A93uaShkzVQvrFPe+3UcPR+7bJGEO3LXBCNsen1jfuEHvWLqxIGUzF8gXNjs2AinRNzkzj/pcx/tC8sESrTN2/JvTw+VlCh9VaLYcsjag4rg/P8IWLO5t/8/TCl28nmQsRVCVEE2nwZU4mfegpigf7uUEzzEWHp1E2aRjexuMyAGJW/1snsM65DBuemei/pHMylLifKny8A36zSQLD8IFLHi/COrW8RDNf8Q+E4+UrjlT8wHwwB7Bm4JJbdBqgDaFvueRLyI3aaOvl4HiIp3tnTntGYug== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(366004)(39860400002)(396003)(136003)(376002)(186003)(66556008)(478600001)(52116002)(86362001)(2616005)(107886003)(8936002)(5660300002)(1076003)(4326008)(6486002)(2906002)(66946007)(66476007)(956004)(6666004)(26005)(316002)(38350700002)(38100700002)(7416002)(36756003)(83380400001)(7696005)(54906003)(8676002)(44832011);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: /tl+uW4UPMZ0nGaNx88oX1ksfSotmXuSrFndNoYAr/uvddpBdaMvnv/N4cWQQFpHpt7eX2zMGt1HewrIIJOGl2qf9YS+Wa0DS5m/HsbEuVr1fRsPd5KgYl+aHopsIfrLg4N8cdQSdToO2YpGBD39fyqXvzZW2pihdZSi9nPP+4g2XNo0dNnO3Diy6dmH9zAc4xvSPmNfVAMbVorkcnZJQzUT5Jd8h9WCpgZ4t98YI8Jpztt9/kKqVaNksjVPI7a88RlzfVrtMjrNBWquLWLUD1jKACEaxroxZA5kGF2ihAmhJE1OaLyy1jyHoxtjyQHdhmR8Ze/L+GKvnoC9ocoRldvWtcxWfwSJ3u43GA3syFNJZ6lStnNhJubDjrHLF/p8QG83k6acJmgeRF3jqXNfI919fRy+HAiofUQqmQ07rOGhvOG5CLq+c4Qk3WfJknNIiFVTuhAIXI6z9+vCmhGDoABtNQWzKPSikr/LCFZKt2vjcLPeFYZdXpaImlit95G62oAC6Xf7LfNyjYoe7l7m5QFWXKpbAQgfH4BMhIkzkTqr28Q07kEg9JWCW7xzcZClKD/IdiqbMFXL1LC1xh8SqhzfuBY0RT8VmnKMIXmCXE/41KEHjCj1zgXSd5TGSKrLY6ohrjVGR+SqqBMGarET+ZJq64NX68KQAeEFddA4kwqbKhMcnuQPhrFEvNDxam4PSYeo7uoHRpWD18UOzd8n4UuOE0dk3DwUlmsF4Ulbf8ByPYFic7QeEVpUMNJa/eDI+yXRjWQbEgQ4WytsPLvVrYtG2yBmgQ0nup0sRp6QQMo4WZqOvlws1sWn0hXxryZ6sx1b2mgz1T1p8atPGgb5HjZB5229P0Hwrgd8H2aDycDOAtUWbo2HHBBzeY9znsBNGLYxnJ8k015ngT3FIJPLzy5AfAU50z24Y6W9Qi5iFLrfXdX4OFRELw557KStAIpv2m07UUYG2VAfXO8b30PMQ/zhWcvcqguelqEzvi3Qn9gN4qG2IKfxRBIKsQkSzX8rGgIogF5IDkcgOrXZYezU3o54CdxBFplKH9J25LUX1yvrH6y74eFgIV8A+RSPvvsZ92N7BsSxKFtvoqYuetM5ChdxFbDSZMyKmxJe+/bsVL2Wj9AZBjjoUenou3wFx93C1ugUuKf/PggwVqTn0sMSfmRf/ErWwb0tyNNzPoQxDcA5DSdiQUtKHC4+KyhGLXXiCsZa/8G+5wtn5aUBw2ovCvu+8RzwQ0BYPZl8oS+iQwRQzjMs4W60P/+Tf/uLEBzbKDX17QPZjrteHTSknibP3H+fPVJr5HGX3kqqsre7fw7b/acPw1ksaWsw/pcxreGm X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5b2acb7b-2193-48ad-9faa-08d94c9c059f X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jul 2021 23:05:24.4010 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: dQPwlUi0UC104ZZXUhmFPFjxleZvs4C/Gi3ajgah+bxkCIaNYzBskHlApdT6MnQ6iK2V5QfaOsT/r0sh1zVojA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3735 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10052 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 phishscore=0 spamscore=0 bulkscore=0 mlxlogscore=999 adultscore=0 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107210138 X-Proofpoint-ORIG-GUID: xzlZ6P8GUtB2xmJqjx6e37xyDn5rPiTD X-Proofpoint-GUID: xzlZ6P8GUtB2xmJqjx6e37xyDn5rPiTD Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=InY94sd0; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=hT2lwzHr; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=ex709avp; spf=none (imf08.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Stat-Signature: qhzyxeda1xac8uo8ycrxas3e6uo1eezj X-Rspamd-Queue-Id: 47F6F3000D82 X-Rspamd-Server: rspam01 X-HE-Tag: 1626908731-646153 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Two new sysfs files are added to demote hugtlb pages. These files are both per-hugetlb page size and per node. Files are: demote_size - The size in Kb that pages are demoted to. (read-only) demote - The number of huge pages to demote. (write-only) Writing a value to demote will result in an attempt to demote that number of hugetlb pages to an appropriate number of demote_size pages. This patch does not provide full demote functionality. It only provides the sysfs interfaces and uses existing code to free pages to the buddy allocator if demote_size == PAGESIZE. Signed-off-by: Mike Kravetz --- include/linux/hugetlb.h | 1 + mm/hugetlb.c | 121 +++++++++++++++++++++++++++++++++++++++- 2 files changed, 121 insertions(+), 1 deletion(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index f7ca1a3870ea..d96e11ce986c 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -596,6 +596,7 @@ struct hstate { int next_nid_to_alloc; int next_nid_to_free; unsigned int order; + unsigned int demote_order; unsigned long mask; unsigned long max_huge_pages; unsigned long nr_huge_pages; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 95714fb28150..cebc6dc353f3 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2964,7 +2964,7 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) static void __init hugetlb_init_hstates(void) { - struct hstate *h; + struct hstate *h, *h2; for_each_hstate(h) { if (minimum_order > huge_page_order(h)) @@ -2973,6 +2973,17 @@ static void __init hugetlb_init_hstates(void) /* oversize hugepages were init'ed in early boot */ if (!hstate_is_gigantic(h)) hugetlb_hstate_alloc_pages(h); + + /* + * Set demote order for each hstate. Note that + * h->demote_order is initially 0. + */ + for_each_hstate(h2) { + if (h2 == h) + continue; + if (h2->order < h->order && h2->order > h->demote_order) + h->demote_order = h2->order; + } } VM_BUG_ON(minimum_order == UINT_MAX); } @@ -3213,9 +3224,36 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, return 0; } +static int demote_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed) + __must_hold(&hugetlb_lock) +{ + int rc = 0; + + lockdep_assert_held(&hugetlb_lock); + /* If no demote order, free to buddy */ + if (!h->demote_order) { + struct page *page = remove_pool_huge_page(h, nodes_allowed, 0); + + if (!page) + return rc; + spin_unlock_irq(&hugetlb_lock); + update_and_free_page(h, page, false); + spin_lock_irq(&hugetlb_lock); + return 1; + } + + /* + * TODO - demote fucntionality will be added in subsequent patch + */ + return rc; +} + #define HSTATE_ATTR_RO(_name) \ static struct kobj_attribute _name##_attr = __ATTR_RO(_name) +#define HSTATE_ATTR_WO(_name) \ + static struct kobj_attribute _name##_attr = __ATTR_WO(_name) + #define HSTATE_ATTR(_name) \ static struct kobj_attribute _name##_attr = \ __ATTR(_name, 0644, _name##_show, _name##_store) @@ -3411,12 +3449,91 @@ static ssize_t surplus_hugepages_show(struct kobject *kobj, } HSTATE_ATTR_RO(surplus_hugepages); +static ssize_t demote_store(struct kobject *kobj, + struct kobj_attribute *attr, const char *buf, size_t len) +{ + unsigned long nr_demote; + unsigned long nr_available; + nodemask_t nodes_allowed, *n_mask; + struct hstate *h; + int err; + int nid; + + err = kstrtoul(buf, 10, &nr_demote); + if (err) + return err; + h = kobj_to_hstate(kobj, &nid); + + /* Synchronize with other sysfs operations modifying huge pages */ + mutex_lock(&h->resize_lock); + + spin_lock_irq(&hugetlb_lock); + if (nid != NUMA_NO_NODE) { + nr_available = h->free_huge_pages_node[nid]; + init_nodemask_of_node(&nodes_allowed, nid); + n_mask = &nodes_allowed; + } else { + nr_available = h->free_huge_pages; + n_mask = &node_states[N_MEMORY]; + } + nr_available -= h->resv_huge_pages; + if (nr_available <= 0) + goto out; + nr_demote = min(nr_available, nr_demote); + + while (nr_demote) { + if (!demote_pool_huge_page(h, n_mask)) + break; + + /* + * We may have dropped the lock in the routines to + * demote/free a page. Recompute nr_demote as counts could + * have changed and we want to make sure we do not demote + * a reserved huge page. + */ + nr_demote--; + if (nid != NUMA_NO_NODE) + nr_available = h->free_huge_pages_node[nid]; + else + nr_available = h->free_huge_pages; + nr_available -= h->resv_huge_pages; + if (nr_available <= 0) + nr_demote = 0; + else + nr_demote = min(nr_available, nr_demote); + } + +out: + spin_unlock_irq(&hugetlb_lock); + mutex_unlock(&h->resize_lock); + + return len; +} +HSTATE_ATTR_WO(demote); + +static ssize_t demote_size_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct hstate *h; + unsigned long demote_size; + int nid; + + h = kobj_to_hstate(kobj, &nid); + demote_size = h->demote_order; + + return sysfs_emit(buf, "%lukB\n", + (unsigned long)(PAGE_SIZE << h->demote_order) / SZ_1K); +} +HSTATE_ATTR_RO(demote_size); + static struct attribute *hstate_attrs[] = { &nr_hugepages_attr.attr, &nr_overcommit_hugepages_attr.attr, &free_hugepages_attr.attr, &resv_hugepages_attr.attr, &surplus_hugepages_attr.attr, + &demote_size_attr.attr, + &demote_attr.attr, #ifdef CONFIG_NUMA &nr_hugepages_mempolicy_attr.attr, #endif @@ -3486,6 +3603,8 @@ static struct attribute *per_node_hstate_attrs[] = { &nr_hugepages_attr.attr, &free_hugepages_attr.attr, &surplus_hugepages_attr.attr, + &demote_size_attr.attr, + &demote_attr.attr, NULL, }; From patchwork Wed Jul 21 23:05:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12392471 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 715A9C6377A for ; Wed, 21 Jul 2021 23:05:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DB48460BD3 for ; Wed, 21 Jul 2021 23:05:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DB48460BD3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7A2A66B0071; Wed, 21 Jul 2021 19:05:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 63E4E6B0073; Wed, 21 Jul 2021 19:05:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 443116B0074; Wed, 21 Jul 2021 19:05:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0083.hostedemail.com [216.40.44.83]) by kanga.kvack.org (Postfix) with ESMTP id 2341E6B0071 for ; Wed, 21 Jul 2021 19:05:37 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id C28841858C1A3 for ; Wed, 21 Jul 2021 23:05:36 +0000 (UTC) X-FDA: 78388128672.29.B4F8442 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf27.hostedemail.com (Postfix) with ESMTP id 1C86470148C8 for ; Wed, 21 Jul 2021 23:05:35 +0000 (UTC) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16LMvG6t009329; Wed, 21 Jul 2021 23:05:31 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=Q1yPM7aeSKO8g+X+N9Ys7mqaY/V7l224/sXdLF0nqpo=; b=RCcMkO+CZo+3ynZUkq/ZgBqljz+G+9U+EkzqVkT18Zo3Hj58go0gO/WYTQC+ab+9HYP0 Upj+Py892NmsfDLcSH7YgYM68ZgLcNxfL5geaYPR0xczspeq4QnKQpYDju+xEW0knsAg zZ32YH4veIj1QLFYlkptneZG/eC7CtsD+VWCURPW8DfejLGnrBSYUl+dg+9VZvXLBunZ QrnoJ1PRRQRo+uViVo4UlOL+yNZom0E5b64/LqgFItxMCDo8INBpcofpgg3t7hhOdFeV 0ONWdqni95+mC+P2pnllcyrmsviC27eVTavy8GwhDYPu3BuaAHMd0XJELY8ay+Zz7MTL Xw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=Q1yPM7aeSKO8g+X+N9Ys7mqaY/V7l224/sXdLF0nqpo=; b=R+QvXtu+gEkMZA8upJpCa7/SFu/Wx2hdim+FvF8HQrqxkCMqAt5IzVSm3tJlHFLCejVa l8vTxoNy3IGdtWKEYgwCSSCGyw9RSWIQcvG/0LM4ioBupTETaKqng9HRr2MjPP1B24ac Cv5Qr0GSk+YuEwUVE5QbBGeFHsPPwUXOt7n/UGxM6UrFkx4Xvq5e1mhkhINl0qw5d9nW vZeZedGGTiISjf3eO2nGzd4VPA0FWkulbqzYK95fH0kb7HHHVQjI4g0eHFpwJAFi9ytP nAMZgv4bcOHoUfq5netgtDBpt6PIlPHmv84Fa5I3SNKqmynpYHtbMyeLYX6RZz7NHLRN Cw== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 39xu2fr5wg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 21 Jul 2021 23:05:31 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16LN0Z0g143558; Wed, 21 Jul 2021 23:05:29 GMT Received: from nam04-dm6-obe.outbound.protection.outlook.com (mail-dm6nam08lp2047.outbound.protection.outlook.com [104.47.73.47]) by userp3030.oracle.com with ESMTP id 39umb3qdx8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 21 Jul 2021 23:05:29 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ABtm6yq56IA2Vb3XFF5TqoDpPuhepYuMAX8YHJ6eKoskflY/IptfU82N0ZrxtOPIZV8EwNB/LpmFFeyKz4w9hzOb9VKnI4TWuWJDc49uE+9uNbdmZA4JGArClfKqJPI98NbwHyEE8eMocbbpk2XvINWpIhf4XIyy/TsAniutN3jPddpipkzzMbTRtHGagQoXY/bbpDyL55eIWBQlKUv2dmyPqpu7NXThb78hdtMD9pjbI2fceUeNQW0ph1Q8iSQSsFnQMpC3UrbgvJhSSnw00WknB2VbcMPsMP1gW27aDitkwMYAvc2tBaJnet5dHnc56BnZrXL8VozQILpCmT7LoQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Q1yPM7aeSKO8g+X+N9Ys7mqaY/V7l224/sXdLF0nqpo=; b=nt/gzVdBauONFalx6dqMKJyTARzbV9NwE0AsBsK0N66xpFSmPcGPscQgKo5scfewEuhLH6O6yjjn1BGeix1vQYNzoVd19givzRTa9cJwJCHsFcxQex2Vywt3mWE3QfYq/fE97UCNwsjG5YXd+68NXQNBL034UZGeW6KFL/GNwJTwj6ARtH3MN/qYlznuJi80TwwItnqbvNux8cv9sZBcr/7IqqILZDgIjNcLqw6Tk16ad+bkSnJDILUxwl624ziYryFpfRZ4O3Xy1z4xiZTxB2r+YcjWLAJW9r7MZUCc+W7AjcUgK6nBTZipopXLei4ds7eOg+GtQPksy7HcLKLoTw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Q1yPM7aeSKO8g+X+N9Ys7mqaY/V7l224/sXdLF0nqpo=; b=Cxj+Dn1lMuz8F6yQZbt6rVKJLM/AsL/nXjySieufFvjU8gIn0Ss3N1XCFo+uCrjAXAcHiuQyg0hJiOgse+FbHB7vPmdjEa9axXgvM1lc9wEJ7XOVcH5do3ilYE8ZcRrZhKUQ2jRMNNQc4qXXREtgg6IApZzqo9b18tPFRPK80W4= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3735.namprd10.prod.outlook.com (2603:10b6:a03:11e::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.26; Wed, 21 Jul 2021 23:05:27 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a%5]) with mapi id 15.20.4331.034; Wed, 21 Jul 2021 23:05:26 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Michal Hocko , Oscar Salvador , Zi Yan , Muchun Song , Naoya Horiguchi , David Rientjes , Andrew Morton , Mike Kravetz Subject: [PATCH 2/8] hugetlb: add HPageCma flag and code to free non-gigantic pages in CMA Date: Wed, 21 Jul 2021 16:05:05 -0700 Message-Id: <20210721230511.201823-3-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210721230511.201823-1-mike.kravetz@oracle.com> References: <20210721230511.201823-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MWHPR1201CA0021.namprd12.prod.outlook.com (2603:10b6:301:4a::31) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR1201CA0021.namprd12.prod.outlook.com (2603:10b6:301:4a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22 via Frontend Transport; Wed, 21 Jul 2021 23:05:26 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3b493156-02da-40bf-44ed-08d94c9c06f1 X-MS-TrafficTypeDiagnostic: BYAPR10MB3735: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8882; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: WqY+U2o6+6wg5BU9l/R8f7KEItPIQiYNxyJms9LJkZJs30lcDO+LcCGh/+TFFRIJN49d4vCiSQz/MHSXYj5q82Stwl0p72CA/8weJ0NuI6kf2qpzkACOpalv4L7UFAG1rZ5FcX3PJTQhE3zezXV0RbaIvutQyuUYE0K6FbXo5ADkGB44lC4EYjXcPN5/b4Ialx7WD3ErZlmqopcivrydi32rIvMewzosi1dn3SqG04X+piSMIDDrfhnLlyy3yOBQNiSZbshy+nhChxcSI7mg1tW+eii8p3lfjRtKCtFcc71dLvXJV6oaNnNuIF9P0SK1bFeRgMCawpzd5BLP3KMf06TmfaMy9e6mHYwAEl4id8v4yX4YezAQlC9KFN572uZMgv5mxuhOGTlYusQQNMI1iFqnn+7Xzy2FaPXTUA2hzF+7xJSGM3T5GDUd29v1grU8yd3uG4vGith6h5MjNb7dRjEeNIHugLTw4KqAvN1Ft9CZZVU2yzDTwhdfcCALSIDFhN49TtrwXcBHaCaep1XbuXpURMpEVx3n6teO9NHKa6hI6WScDgaA0ikyJ1qvz/AY7Re4hM90TrFromq3F9uoyNJwDOB9ZF6ic6TXYdp+2ux7PitJ6sIm7C9Cul+0roMPOUFZ9k0uisYF+8uD/WJIndy7qDqF2XyFaaoJ4Wox932gHQgjc0QqxURiJPSPHPC5xWoAiRO/KQq5dRWrCvqGyw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(366004)(39860400002)(396003)(136003)(376002)(186003)(66556008)(478600001)(52116002)(86362001)(2616005)(107886003)(8936002)(5660300002)(1076003)(4326008)(6486002)(2906002)(66946007)(66476007)(956004)(6666004)(26005)(316002)(38350700002)(38100700002)(7416002)(36756003)(83380400001)(7696005)(54906003)(8676002)(44832011);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: bu2790WppkZgY+Bt3ON0a2uwBSx/F/5Jv1pAnwMP/Xf0RbTQtJM2+4o3Cl+7zDxJHEoFJqG+5GwmEQs1DKRLOehqkup8RDJa8VmdRyK8qRUf1dwf30/lv2pVZcBL86wvSvGfN8nl6hn5i0oTI2/8lyzxe1B6oIvpMTj/BFnK4J4FlmsjmWXw+Zho85Q9GqeQY0WfSmW9Eiyy264ZB15XPAebvFXH4xAErtJdPlPocnhvlwMlsjZzOv7KWO3Gwik7xRgS3km8bh1da4TAZd1Mv/gpOwndAWRjhlTpwUpPLBLKLy07brlrBAwTtymbg0UhQdWB4zKNKx1IsP/l0q86mJWWmOaIs6PaWDrC96fTtoXO/zPcEGrlUuxCOV3tbMC9ncuMKJxRpsanFBuPRNvunqqW6Z8h3cQ1kmE5xUPm7ckwAM2HWWRSNEqE+C3T/8O6YFVCPGgUh+VDQkryCc3vx4pE7OtnSYhRXAzETTUsIIQtb2Ick+w7MKUvy7KMW+1zLqEJPtU2k6NT13oETy27RDHT2IRrqz8gCv3nFnzpnJjNUViyUmxpdb2JDDu1CdGnzZ9OWTahSRnsnpXldPbPFHzNrWAoo0tQBUKMQFwwT+83em7W9FD4duGVB/O/dpQoBF+KXjqPZJ8UEE8E9BMxgkPGF6XjC3jLzsm7mQqmts/nLAHE4hwlGP2nzalimBHQkmHFoHP2o68HDlna9Gh2ZnUO582sAQUM5eld0t4UJD2T1+lUy3ejYPbKdwXuBVftooWBL0p50yo1cPuppF0J0y2GvrseoItoZW8Q0WOZSfx9DZRFAmRaEKLC6G/P+L/2TrE9khrbVrxGBDZ2IgYJEAqNZGXrQ9T9wXAgXy+hzB87eixrq4ao04iPGr73jJ+rMpg8qDqq3jTiST8MDqAkZTwzCf2swLDO2bBQ6JpWyP7dnS6BpX6Bz379oQ2HdxGrbeCBvkNroAm44VSLOIEZao1WvnLhhKWmko4n0gabmHa8h7TyQdPPrlH5+08jLNXjRTLn86mJAHOZjSmHct3RwxsoGPFDEptWs7k1y1T2+1YtpqhvMhnospHbNOf4oJ7CjfGRcNNvu8zWgMMLkIOxHbEXfkL64f3qmXQ4kDGYSJPzyMFfwv7j0m+38NvE7uwFJypJ+acjLnaQQkaTBmaruVT4bBjjlCvB4JkqKIGVFeeXCPmjZ+tN6UqbkSOlkz7+vk+iUlcSAxF/irOrQznfcal19dvgtXa4hTWasU9TAWf/UX1soUWb4BA+SieoAz3wVENgkQ3qHz4Smu9yopqmUW5+rE1U/5U2XGrctCCfPLZuBGugvzmB+AOaaGktEJ8V X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3b493156-02da-40bf-44ed-08d94c9c06f1 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jul 2021 23:05:26.7224 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: mPHZc6T0KUtWSzjtplku1+r6OGrS+XA7PEagv6btRDtcfjA5yZ6rLX6KhrTFb5fRtVKnrSGuC0CcqEPSXln/ag== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3735 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10052 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 spamscore=0 mlxscore=0 adultscore=0 mlxlogscore=999 suspectscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107210137 X-Proofpoint-ORIG-GUID: KMrpNXp0Hhay1jtszOXLZBQM89i1M4uZ X-Proofpoint-GUID: KMrpNXp0Hhay1jtszOXLZBQM89i1M4uZ X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 1C86470148C8 X-Stat-Signature: 44ut4an6krwgqnfwo87unjxt66rgfuai Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=RCcMkO+C; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=R+QvXtu+; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=Cxj+Dn1l; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf27.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=mike.kravetz@oracle.com X-HE-Tag: 1626908735-910904 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When huge page demotion is fully implemented, gigantic pages can be demoted to a smaller huge page size. For example, on x86 a 1G page can be demoted to 512 2M pages. However, gigantic pages can potentially be allocated from CMA. If a gigantic page which was allocated from CMA is demoted, the corresponding demoted pages need to be returned to CMA. In order to track hugetlb pages that need to be returned to CMA, add the hugetlb specific flag HPageCma. Flag is set when a huge page is allocated from CMA and transferred to any demoted pages. Non-gigantic huge page freeing code checks for the flag and takes appropriate action. This also requires a change to CMA reservations for gigantic pages. Currently, the 'order_per_bit' is set to the gigantic page size. However, if gigantic pages can be demoted this needs to be set to the order of the smallest huge page. At CMA reservation time we do not know the size of the smallest huge page size, so use HUGETLB_PAGE_ORDER. Also, prohibit demotion to huge page sizes smaller than HUGETLB_PAGE_ORDER. Signed-off-by: Mike Kravetz --- include/linux/hugetlb.h | 7 +++++++ mm/hugetlb.c | 46 ++++++++++++++++++++++++++++++----------- 2 files changed, 41 insertions(+), 12 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index d96e11ce986c..60aa7e9fe2b9 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -533,6 +533,11 @@ unsigned long hugetlb_get_unmapped_area(struct file *file, unsigned long addr, * HPG_freed - Set when page is on the free lists. * Synchronization: hugetlb_lock held for examination and modification. * HPG_vmemmap_optimized - Set when the vmemmap pages of the page are freed. + * HPG_cma - Set if huge page was directly allocated from CMA area via + * cma_alloc. Initially set for gigantic page cma allocations, but can + * be set in non-gigantic pages if gigantic pages are demoted. + * Synchronization: Only accessed or modified when there is only one + * reference to the page at allocation, free or demote time. */ enum hugetlb_page_flags { HPG_restore_reserve = 0, @@ -540,6 +545,7 @@ enum hugetlb_page_flags { HPG_temporary, HPG_freed, HPG_vmemmap_optimized, + HPG_cma, __NR_HPAGEFLAGS, }; @@ -586,6 +592,7 @@ HPAGEFLAG(Migratable, migratable) HPAGEFLAG(Temporary, temporary) HPAGEFLAG(Freed, freed) HPAGEFLAG(VmemmapOptimized, vmemmap_optimized) +HPAGEFLAG(Cma, cma) #ifdef CONFIG_HUGETLB_PAGE diff --git a/mm/hugetlb.c b/mm/hugetlb.c index cebc6dc353f3..d034a223d5d7 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1262,6 +1262,7 @@ static void destroy_compound_gigantic_page(struct page *page, atomic_set(compound_pincount_ptr(page), 0); for (i = 1; i < nr_pages; i++, p = mem_map_next(p, page, i)) { + p->mapping = NULL; clear_compound_head(p); set_page_refcounted(p); } @@ -1273,16 +1274,12 @@ static void destroy_compound_gigantic_page(struct page *page, static void free_gigantic_page(struct page *page, unsigned int order) { - /* - * If the page isn't allocated using the cma allocator, - * cma_release() returns false. - */ #ifdef CONFIG_CMA - if (cma_release(hugetlb_cma[page_to_nid(page)], page, 1 << order)) - return; + if (HPageCma(page)) + cma_release(hugetlb_cma[page_to_nid(page)], page, 1 << order); + else #endif - - free_contig_range(page_to_pfn(page), 1 << order); + free_contig_range(page_to_pfn(page), 1 << order); } #ifdef CONFIG_CONTIG_ALLOC @@ -1301,8 +1298,10 @@ static struct page *alloc_gigantic_page(struct hstate *h, gfp_t gfp_mask, if (hugetlb_cma[nid]) { page = cma_alloc(hugetlb_cma[nid], nr_pages, huge_page_order(h), true); - if (page) + if (page) { + SetHPageCma(page); return page; + } } if (!(gfp_mask & __GFP_THISNODE)) { @@ -1312,8 +1311,10 @@ static struct page *alloc_gigantic_page(struct hstate *h, gfp_t gfp_mask, page = cma_alloc(hugetlb_cma[node], nr_pages, huge_page_order(h), true); - if (page) + if (page) { + SetHPageCma(page); return page; + } } } } @@ -1470,6 +1471,20 @@ static void __update_and_free_page(struct hstate *h, struct page *page) destroy_compound_gigantic_page(page, huge_page_order(h)); free_gigantic_page(page, huge_page_order(h)); } else { +#ifdef CONFIG_CMA + /* + * Could be a page that was demoted from a gigantic page + * which was allocated in a CMA area. + */ + if (HPageCma(page)) { + destroy_compound_gigantic_page(page, + huge_page_order(h)); + if (!cma_release(hugetlb_cma[page_to_nid(page)], page, + 1 << huge_page_order(h))) + VM_BUG_ON_PAGE(1, page); + return; + } +#endif __free_pages(page, huge_page_order(h)); } } @@ -2981,7 +2996,8 @@ static void __init hugetlb_init_hstates(void) for_each_hstate(h2) { if (h2 == h) continue; - if (h2->order < h->order && h2->order > h->demote_order) + if (h2->order >= HUGETLB_PAGE_ORDER && + h2->order < h->order && h2->order > h->demote_order) h->demote_order = h2->order; } } @@ -6496,7 +6512,13 @@ void __init hugetlb_cma_reserve(int order) size = round_up(size, PAGE_SIZE << order); snprintf(name, sizeof(name), "hugetlb%d", nid); - res = cma_declare_contiguous_nid(0, size, 0, PAGE_SIZE << order, + /* + * Note that 'order per bit' is based on smallest size that + * may be returned to CMA allocator in the case of + * huge page demotion. + */ + res = cma_declare_contiguous_nid(0, size, 0, + PAGE_SIZE << HUGETLB_PAGE_ORDER, 0, false, name, &hugetlb_cma[nid], nid); if (res) { From patchwork Wed Jul 21 23:05:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12392473 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B9C7C6377C for ; Wed, 21 Jul 2021 23:05:40 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D101861029 for ; Wed, 21 Jul 2021 23:05:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D101861029 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1AE026B0073; Wed, 21 Jul 2021 19:05:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 09CA76B0074; Wed, 21 Jul 2021 19:05:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D90DD6B0075; Wed, 21 Jul 2021 19:05:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0095.hostedemail.com [216.40.44.95]) by kanga.kvack.org (Postfix) with ESMTP id BABE86B0073 for ; Wed, 21 Jul 2021 19:05:37 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 4E4C718591F5A for ; Wed, 21 Jul 2021 23:05:37 +0000 (UTC) X-FDA: 78388128714.28.05BF695 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf15.hostedemail.com (Postfix) with ESMTP id AB2FAD008954 for ; Wed, 21 Jul 2021 23:05:36 +0000 (UTC) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16LMvG6u009329; Wed, 21 Jul 2021 23:05:32 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=g+6VZGRQSNnCGSvjXcCWDjj7JZh/wgzNtXrvPGSgtPs=; b=gEIPkGCuq/xbqaf9dusP7NleUhO0ZwjPAjMTluxHAW2OcsX4IyHXty0Onz8MAxgl3eKL HQYM2pidcraQ/dq2sE8vwfhgdNTkU/ffgUknlwC9arNNibrXp0NJADuM3YcTyQzUSbbg TdHs26GypPzoKkHtoIfsxEStsshbqH0t1qe++5PMkg17HLofcgv6U5O/PF0x1P6HRxFW i77nT6JuxPMMoe5UZm16RcBIfigjMrOMFyjt4qojafxKGquLCZbcIiZxNzIwcbpMY6zG tu7KAV80ZidQRdW1vFD2SUV9trcVElItCSOfgGhthkyEEeBhrWgnyb1zEKNsAeY3FZO2 uA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=g+6VZGRQSNnCGSvjXcCWDjj7JZh/wgzNtXrvPGSgtPs=; b=qtwxDFJa8pC1mN+WwY/aAzYNhgbAscZ8bl6zcGRBJHJqiAcg0FdY970QYG1UaAeaYIJl F9YMQdLW5PC/WrzRM1iFnOonlFSEPV5+oHe7cUWGDEI3VdQVljIcCxzNUU9j5WPoJIEy mkU4hcEzFzxB/ssYkhupzMN9BsCPCUY/BnBhMvaSs/WfnOWnH77GMdKkWGussiQGOfyp XsjRT2Kyt78F8c6WqoVVucXMzoDA4y91U0RfOVhZnf3N9hVSNErLPaM5raxC77M5txYP qJ6/alqcI+uzRijyD9QiSOeZNL+EuX64PqxwP551ceZUwfALiTzQ3DXqneblBVfMi8mp PA== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 39xu2fr5wh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 21 Jul 2021 23:05:31 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16LN0Z0h143558; Wed, 21 Jul 2021 23:05:30 GMT Received: from nam04-dm6-obe.outbound.protection.outlook.com (mail-dm6nam08lp2047.outbound.protection.outlook.com [104.47.73.47]) by userp3030.oracle.com with ESMTP id 39umb3qdx8-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 21 Jul 2021 23:05:30 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nKMNnt03KUZrlLaoursapZoUeP67Ys3e4e+F7VbRp9EcRRSgYkv991E5nDJFylXbb1nWjhfyNWxou2knIMvQfjzB1rxDZZ62mwoWgjW8Xbc+7mjKh09J0RKvmTUFSAChJOKji5aKd3ZJ3mR5i8jLNS89b1Jn8eXvtC/61AofaP/qZqFRSxPUZE76GUbP8n4/5x2uj46L07XppDMVYVSNqTrel/hIu3s5R1Rroa0BffeB2TbrQRFlTECkTraWF8pBAR/tWWX6OV/6EwpyOvoXhMT94HGOTySTOb6i+aPg1oHcb/bzBA2Ryqf6qUtAjmxdRZmBH7VGQY9t0TiXjiO8jQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=g+6VZGRQSNnCGSvjXcCWDjj7JZh/wgzNtXrvPGSgtPs=; b=OzsXu64JkzcGteUh2n0BYK6LdtVkUXKP899JNafYItKwBEkwg6Fn38OBrI0tpQJkksKjc9jfZIB6CHDCLJGHoI31g5GlGNFmuHfSeP5jZwd9FPHr4nI6A3xBGdKhQPlXxKPE5dBkz5l00xWFEHPBnkMdgJX+U1BP4+EgBuhm3U4fSNYkRPC0ArU53et4bq6oqoUGN1895E9xUWqVMPKU1NFyywiz4h1pyAUtIC8P0oYg9ruYezAsnCbJVlf9x2Oonvw9JOVJBo5TmNT3KyjpIxTd6aTa4ia5GD+ohTlUFao8P65CUCIua0sBW5uheZKk3bMXijT87Jxy27HU+qUapg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=g+6VZGRQSNnCGSvjXcCWDjj7JZh/wgzNtXrvPGSgtPs=; b=S8ZMVtdNBmGupWrR4i7l9CESLX0BxizUKZETR5m/YX3qJEjfKghJ7mGzOD16qkYZnBD8gGrZpL8WfbVK46Pkb9ZbgjPRQCbaswhJCKhsVx+noonLj8HdS+AzpR96UTOdUf3O/mg6k4WreYo73LrrwGtuDEOM1TfNTecqJtexrXM= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3735.namprd10.prod.outlook.com (2603:10b6:a03:11e::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.26; Wed, 21 Jul 2021 23:05:28 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a%5]) with mapi id 15.20.4331.034; Wed, 21 Jul 2021 23:05:28 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Michal Hocko , Oscar Salvador , Zi Yan , Muchun Song , Naoya Horiguchi , David Rientjes , Andrew Morton , Mike Kravetz Subject: [PATCH 3/8] hugetlb: add demote bool to gigantic page routines Date: Wed, 21 Jul 2021 16:05:06 -0700 Message-Id: <20210721230511.201823-4-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210721230511.201823-1-mike.kravetz@oracle.com> References: <20210721230511.201823-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MWHPR1201CA0021.namprd12.prod.outlook.com (2603:10b6:301:4a::31) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR1201CA0021.namprd12.prod.outlook.com (2603:10b6:301:4a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22 via Frontend Transport; Wed, 21 Jul 2021 23:05:27 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3cdbd765-48b1-41d3-79df-08d94c9c080a X-MS-TrafficTypeDiagnostic: BYAPR10MB3735: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1RBMTsoYgi8Nd19JOyby0SZZQicMAfFpVYuLblLtexl+J9nGS+gqFpRZxJR1/eBD2dWnTpiZPjBayxlbhkORNrYyqpMYvXN9AawRKOcqqLKeUiHf3180si+86b7iGdh37G3HAPQVb6o71LYg85aH4uGgMjn2iSvPT8Qcvts7dWMEip8KXDDTWryh+qnUPn0colkSPX1zglLrbOk+i1bsij0/poGaxw9MGFfrzNFU+v1X9dI67COb/yr+fNI+QNIqcOUo8420931uAxf+WI6lejFVaB0he3fC+BWTMcX0ynq/9EqCtdsbGkCOb3YUdjOoGwxSmss9S1c9+86AuhUlGzgMjjCykKL5AXmLME867+hpt1s+2XLyPwREiSsXbUucp4Gv3aJrSgMIaoftvuLZ/Pg6uoZ5fbcpCF6KPSU4HneITnTkz4jEDsd2dgnAdN3826n4o+S382x0zktfnR7hQ9vrrvnv0nTZwFZafmwZ6srzIEbN9RpZG7LypZqABh8S9MWJHipGAzP13AgNUO6S9LV27XQcGjoxQ1bDK3m0rUXiZSC4pScM4ifHu3YLqlce+KHOCpJFC7Jw52fF33Hwm7ecc8hYnfMverSPRr4cKkWL4XZ0/b0CtO7eNur6qz54xycPCQleny4CpythW6bgQAKxuf/99nJCujM3WBQZdgVAVR67Yd/2VDZBLapF+CU7Hdy886+4kehZr93dV9qVGg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(366004)(39860400002)(396003)(136003)(376002)(186003)(66556008)(478600001)(52116002)(86362001)(2616005)(107886003)(8936002)(5660300002)(1076003)(4326008)(6486002)(2906002)(66946007)(66476007)(956004)(6666004)(26005)(316002)(38350700002)(38100700002)(7416002)(36756003)(83380400001)(7696005)(54906003)(8676002)(44832011);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: IqjJ+cySHOcDSQE39ZeJASOLHxhQjyQYQTs3SnpEwyHF/jjAezjSw+S+KqtpU30MaXoRzsPkNIPg2dpF4qrwan9m7x20drtNc+jxUEPbrw6A+4KJRFsBqLFfUeDxfcs0VDQ4fLZ0zK2IlmL2KEloSDER2y3n/m7Elx/DIRtURampJtsw6tsakKHAvoxgYy87AWHw9FAswBb7Zj1nuyxw85sizqOpCLSSrvqn0f4rN3MvRTNAUMEOvm7w1FFXBZoJv7eR07VYAICEu28mWxPDWG+McZwv4yY1hTm21PakaUazqDbViyNqcRsGkcXGmAhpovOnJCHLR73DPV/h/CJrrrDwSbM/HyR9v7/iugi+A5ytlPCMKDT+MSDnPFn1CxbxF3swBa7FF2ucjN+TCFTyH5i7ges8IlMEeOA0t0l+NFxJz5itx3mRrfbxqSoukLLBd3X6IGYC1czx1hnWC+/d/Nlsf2DkVEHH9UNYRjKv4eDSrUwg0wlC3JCYyPX3LzY+xLz8xnjyQDs9O6H1mwRB8rM3wQZSCj4rVkQqvqKyMSjfaG0Z016P9YA3ttUq8QW9nfYTJ4eRY0gRQGG345sw1wBiL1iOpsSkNiopCp66Kq//HvY1WZp1HgG3dIRP/VTYGxbBixF7Xr1nDm+e45M4S2hARReyMlHJfNB2HcgYyZmHqUyQn0eK023uMbdrQsHpvc1DeWxoTz52V8/EUJG3FklgTPrF5s+lO9NXynad8nNnRidu0WT5nCwB50jMIf94moZwSlycl7TTpNPySjwS0K9freQ/yguJ1ESj+oQXcTt4AOc5c4abd7z8npb+8nqNXN2fTctSNQwRWQtBHZoxQBLCpcpbictZ1NDFBXelKD+7bbooSsQdiXrx4IoXakGD08F99uiZ5N2aFrTlR0t9QaNFEJb6K7O/E2fwiGEsqEqw3GFZwF2gcRm61P3Diqm/VcB5ruKkFv8T+o7lY7k3Ux4z09fmbiPdcR4mBqPURJdlxZuvsf6nDENpOPs/YSk27hD4lrPuCz3uyaYPpj5JvoD1Y9FEwG85BogQy+hebm28VZTEf9g06KDnBjhyGuXNj5zlDSxHFq/rdQztGwI1A7c+FphcH7GG4A5acbsnQk5yFYqrVPKRNTYWuxJz8sipubzsLgyUKXcEoGmbbG1m0IVn+ZQntmZdUUS3cKE3zcqFNR3YAlFLCoG/nPN/Gvcmdurc46ajTt3K3hriGUFes4VNtW8NYkjUZ14qeKz1P51zvYa1wiXhApVfv5a9Oj4VSn/pwkQ5Vied4FYIE84HxVPj8d4JnqK6a2Bf8QC00g5hhZmF7PB6IqMnVwIrIvKZ X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3cdbd765-48b1-41d3-79df-08d94c9c080a X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jul 2021 23:05:28.5862 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: aqL8FYER7HvI3f+Mtk+wFB3t8B8UKR8lCBjHrjmkBKypU2kBxRtkcXs0t/90EOetuie+lmE/mPEyOrNwPd1g2Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3735 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10052 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 spamscore=0 mlxscore=0 adultscore=0 mlxlogscore=999 suspectscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107210137 X-Proofpoint-ORIG-GUID: CL-P7M6HLSveF38KOGyW2bRvyl444ucQ X-Proofpoint-GUID: CL-P7M6HLSveF38KOGyW2bRvyl444ucQ Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=gEIPkGCu; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=qtwxDFJa; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=S8ZMVtdN; spf=none (imf15.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam02 X-Stat-Signature: myem4fc8s3yb14wtyn4za7rzwtt5fhkt X-Rspamd-Queue-Id: AB2FAD008954 X-HE-Tag: 1626908736-148589 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The routines remove_hugetlb_page and destroy_compound_gigantic_page will remove a gigantic page and make the set of base pages ready to be returned to a lower level allocator. In the process of doing this, they make all base pages reference counted. The routine prep_compound_gigantic_page creates a gigantic page from a set of base pages. It assumes that all these base pages are reference counted. During demotion, a gigantic page will be split into huge pages of a smaller size. This logically involves use of the routines, remove_hugetlb_page, and destroy_compound_gigantic_page followed by prep_compound*_page for each smaller huge page. When pages are reference counted (ref count >= 0), additional speculative ref counts could be taken. This could result in errors while demoting a huge page. Quite a bit of code would need to be created to handle all possible issues. Instead of dealing with the possibility of speculative ref counts, avoid the possibility by keeping ref counts at zero during the demote process. Add a boolean 'demote' to the routines remove_hugetlb_page, destroy_compound_gigantic_page and prep_compound_gigantic_page. If the boolean is set, the remove and destroy routines will not reference count pages and the prep routine will not expect reference counted pages. '*_for_demote' wrappers of the routines will be added in a subsequent patch where this functionality is used. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 55 ++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 43 insertions(+), 12 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index d034a223d5d7..0ad75e7775dd 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1251,8 +1251,8 @@ static int hstate_next_node_to_free(struct hstate *h, nodemask_t *nodes_allowed) nr_nodes--) #ifdef CONFIG_ARCH_HAS_GIGANTIC_PAGE -static void destroy_compound_gigantic_page(struct page *page, - unsigned int order) +static void __destroy_compound_gigantic_page(struct page *page, + unsigned int order, bool demote) { int i; int nr_pages = 1 << order; @@ -1264,7 +1264,8 @@ static void destroy_compound_gigantic_page(struct page *page, for (i = 1; i < nr_pages; i++, p = mem_map_next(p, page, i)) { p->mapping = NULL; clear_compound_head(p); - set_page_refcounted(p); + if (!demote) + set_page_refcounted(p); } set_compound_order(page, 0); @@ -1272,6 +1273,12 @@ static void destroy_compound_gigantic_page(struct page *page, __ClearPageHead(page); } +static void destroy_compound_gigantic_page(struct page *page, + unsigned int order) +{ + __destroy_compound_gigantic_page(page, order, false); +} + static void free_gigantic_page(struct page *page, unsigned int order) { #ifdef CONFIG_CMA @@ -1344,12 +1351,15 @@ static inline void destroy_compound_gigantic_page(struct page *page, /* * Remove hugetlb page from lists, and update dtor so that page appears - * as just a compound page. A reference is held on the page. + * as just a compound page. + * + * A reference is held on the page, except in the case of demote. * * Must be called with hugetlb lock held. */ -static void remove_hugetlb_page(struct hstate *h, struct page *page, - bool adjust_surplus) +static void __remove_hugetlb_page(struct hstate *h, struct page *page, + bool adjust_surplus, + bool demote) { int nid = page_to_nid(page); @@ -1387,8 +1397,12 @@ static void remove_hugetlb_page(struct hstate *h, struct page *page, * * This handles the case where more than one ref is held when and * after update_and_free_page is called. + * + * In the case of demote we do not ref count the page as it will soon + * be turned into a page of smaller size. */ - set_page_refcounted(page); + if (!demote) + set_page_refcounted(page); if (hstate_is_gigantic(h)) set_compound_page_dtor(page, NULL_COMPOUND_DTOR); else @@ -1398,6 +1412,12 @@ static void remove_hugetlb_page(struct hstate *h, struct page *page, h->nr_huge_pages_node[nid]--; } +static void remove_hugetlb_page(struct hstate *h, struct page *page, + bool adjust_surplus) +{ + __remove_hugetlb_page(h, page, adjust_surplus, false); +} + static void add_hugetlb_page(struct hstate *h, struct page *page, bool adjust_surplus) { @@ -1669,7 +1689,8 @@ static void prep_new_huge_page(struct hstate *h, struct page *page, int nid) spin_unlock_irq(&hugetlb_lock); } -static bool prep_compound_gigantic_page(struct page *page, unsigned int order) +static bool __prep_compound_gigantic_page(struct page *page, unsigned int order, + bool demote) { int i, j; int nr_pages = 1 << order; @@ -1705,12 +1726,17 @@ static bool prep_compound_gigantic_page(struct page *page, unsigned int order) * the ref count to zero if count is currently 1. If count * is not 1, we return an error and caller must discard the * pages. + * + * In the case of demote, the ref count will be zero. */ - if (!page_ref_freeze(p, 1)) { - pr_warn("HugeTLB page can not be used due to unexpected inflated ref count\n"); - goto out_error; + if (!demote) { + if (!page_ref_freeze(p, 1)) { + pr_warn("HugeTLB page can not be used due to unexpected inflated ref count\n"); + goto out_error; + } + } else { + VM_BUG_ON_PAGE(page_count(p), p); } - set_page_count(p, 0); set_compound_head(p, page); } atomic_set(compound_mapcount_ptr(page), -1); @@ -1733,6 +1759,11 @@ static bool prep_compound_gigantic_page(struct page *page, unsigned int order) return false; } +static bool prep_compound_gigantic_page(struct page *page, unsigned int order) +{ + return __prep_compound_gigantic_page(page, order, false); +} + /* * PageHuge() only returns true for hugetlbfs pages, but not for normal or * transparent huge pages. See the PageTransHuge() documentation for more From patchwork Wed Jul 21 23:05:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12392475 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0400C6377A for ; Wed, 21 Jul 2021 23:05:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6A33F60BD3 for ; Wed, 21 Jul 2021 23:05:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6A33F60BD3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1FBF06B0074; Wed, 21 Jul 2021 19:05:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1866A6B0075; Wed, 21 Jul 2021 19:05:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E56796B0078; Wed, 21 Jul 2021 19:05:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0057.hostedemail.com [216.40.44.57]) by kanga.kvack.org (Postfix) with ESMTP id C54F16B0074 for ; Wed, 21 Jul 2021 19:05:39 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 73A5C18579018 for ; Wed, 21 Jul 2021 23:05:39 +0000 (UTC) X-FDA: 78388128798.13.9D4B185 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf24.hostedemail.com (Postfix) with ESMTP id E092FB00518F for ; Wed, 21 Jul 2021 23:05:38 +0000 (UTC) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16LMvJWF031212; Wed, 21 Jul 2021 23:05:34 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=TEWj0W18GtlHbGvGCGpiQIhisDaq0wLV7EXt77Rlnq8=; b=RQBylKXzOUqWSqM04ItRNNcuUR/aTg76ikqbbk5cotojqATZRIZTVilMSDKCxZvCNdrn npBVVye9Fw90v6vBishipXXkNUJnnEXB+J6uDbzyi1i8OjmGdCksL3FI3NMV6hh37CRy 6lBU/a6e6UVAVKMdv3W8ynjK9SHRk45ZgafFGkx2whLTJ1WVFEBQhW+lLYBnpaA1yU4t jQFY2izr3k0751iiTKTcHdgN+yJCTt1sBAH23OSsOasaD6UUkreP8DZrtizASjaycP4a ieQi5XAqqKbuy9GbmkWaxpCt9eSnai4d6VGgv64oy07ZuIGWtQqkPvZpzs3sC7VVuh+N Sw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=TEWj0W18GtlHbGvGCGpiQIhisDaq0wLV7EXt77Rlnq8=; b=WTRZXYI8e1cgSN+mKEL54E9Hh4OGU4h0QwLeZK+2n1pEAOiJp4VLjTFRxGzzp+NV98hr qJKFA5LiomSxEkUVhETU4EF0Gof3CVmxU/nyz6XyecidlD9HrJ/0tLgnQfn4+kg2yANt RckJPgtuhAvblMaTDJj+77E0a12RtWO1yLOLPJPvzc7db8EbiBOM2/XLO+yvCgdGd8sB +406AHUok19JyFvSQy060x/a+aJTRbe/UDMJ6FdYAd0Nx/1LjVbTEOhLvWhbkMc/IeyA AJj/cD9MS96cp68nEjdx26Rc/EVY0WZwmEZySCcn6TVJDeKryFEEtz78btTZd9MUN6yI vg== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 39wwmtuxqs-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 21 Jul 2021 23:05:33 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16LN0Umt098951; Wed, 21 Jul 2021 23:05:32 GMT Received: from nam04-dm6-obe.outbound.protection.outlook.com (mail-dm6nam08lp2043.outbound.protection.outlook.com [104.47.73.43]) by userp3020.oracle.com with ESMTP id 39v8yycrjm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 21 Jul 2021 23:05:32 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CX+9uRPbecGY+UZ1fsQGAGbPrnZxLmkEjGl2EK2o4DQlJiYnxaMZyRyOxXYj25ttbJ9jyfhgAq9LTJY4tvQV9BGSzvucjNITfMkfIPc03TW4MTGGmNhJpaymVmX8n7YW9x8+1YIa8a0rxmOda2NIFrDC4ey4Zx0ckiYHO0dqwZWweNOTBGamFPmkdAq/MX9G9G/UaD8G3fPWtM/XpCXrpOyALwZ/mxG0mATOTzjxTp4OKJXKnMp/NPfpyV93mjo50alDoy8gtW/C4vilBLV3rePxjsxM1Dy9Jd3sUkqhvwP6+WaQWJWUTw4IbwplgI/eUjcMhSedSnpfctl4VGJ/WQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TEWj0W18GtlHbGvGCGpiQIhisDaq0wLV7EXt77Rlnq8=; b=G0iEaDZmDMxmABAI7Rv6D6OPAt5aGGRLGQ+OvrA9dJBgjvJR3OVVU3wQFJqHE2OCWjLKJXHjRVqR/ez3aewYfWf/ja7twfJfR1025vJ97aBQE2oQAxHLcsM/yLxeUtwdQJvB/FVM9LT/omxrWN0HLI80GTYwBIyUY0SP3ltsIVzVkkcoAy/AJb9ovX243MYyOHoAAUCl90hXytFTNzElrpoE1XAfaHUUde7K7ZIcSvRyfS8leYrgjz7lu3E86w52TN/M3Yksym3TUGAQwkzeU0HmhuuODzu1hPq8h6BAqLQpHG5N3/RpnlvIsS55lOwSgUnfzaRc7wibXv9mW1oUrA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TEWj0W18GtlHbGvGCGpiQIhisDaq0wLV7EXt77Rlnq8=; b=P/JZRA/0ipRIQA6mXb9pIsY+EekCAZaDyinr5QztmvghcHC+6Z55joz1YVPzPevrKxWLn5f9BXPl8LZloKrYyOSTpJS5My/Rvk2wkZjWS9915NKc1UmXRBVD3ByWFVmDyMWSxp8g9wN8yV1sAJj1+T7lzUeAfmR2/sLacDwS4to= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3735.namprd10.prod.outlook.com (2603:10b6:a03:11e::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.26; Wed, 21 Jul 2021 23:05:30 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a%5]) with mapi id 15.20.4331.034; Wed, 21 Jul 2021 23:05:30 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Michal Hocko , Oscar Salvador , Zi Yan , Muchun Song , Naoya Horiguchi , David Rientjes , Andrew Morton , Mike Kravetz Subject: [PATCH 4/8] hugetlb: add hugetlb demote page support Date: Wed, 21 Jul 2021 16:05:07 -0700 Message-Id: <20210721230511.201823-5-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210721230511.201823-1-mike.kravetz@oracle.com> References: <20210721230511.201823-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MWHPR1201CA0021.namprd12.prod.outlook.com (2603:10b6:301:4a::31) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR1201CA0021.namprd12.prod.outlook.com (2603:10b6:301:4a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22 via Frontend Transport; Wed, 21 Jul 2021 23:05:29 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4de894ae-193e-4d92-dbe7-08d94c9c091b X-MS-TrafficTypeDiagnostic: BYAPR10MB3735: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6108; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: OHGYBRyUF94uGhmRsAbv1jPmaypVdTIIbc1mJEk1KmwlkFShPv/RpyfqF4gM5QZbdfbmhMie7L0JnFRcuwhhh9bCmBfaskZqcV71ZctlXTFRwhNW06rCZ1kTr40d9Zv5PbYWV6Qy6cVhWlE+LplwzMIeMemiHe/8Syr24HCZB7+u1XlHR5vzRt1+CR6v6W2inotqgJQnq/fu4e5sekG33WfbBgc6HomOTIKoMX+aq/1ni/LLN3g2zY4NN77lL1oU48I7Rh/GTbdDafSo0r2kh3MRSbH4rphFLqrr659hMbtQdoqO1CO6IVucI1Nv+/2NJWrl6c5f+mmGNylR3x5/D6IeEnHiFiyniSvt3mDrDw6jbmZ2ppUe5CWKuSIodhEG2atk16eAz051ExqHAvrj2PQd3fFoX+o+Xt5OF52NIDTbn8vXGjArY0juaYFigFZ7FDfA0pSL7uDWlXSfV0IN6COkbmB3ywtHaAdbui8Bgmg+B4TqsISajXQJ2WszSsQ55IiLJWwwnG7wWwd0v9HdZfnKP14gaTeJbZwFAPa6SCJ52ZpfTbCbq5jkHz3+P/2fb2hj+a00wGToDsEIcKV4xmkyzNgDAp8XDKYe3Lng46xaJA8Uo1htjjougQws4Ms/bZyalV+KbhEEZM7ifO471PxlRB/Srz9Mx5iAz4ORUTMc02PvMr4PEYYf/ysgQcPP5enGI396tmNsWORfv0jISQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(366004)(39860400002)(396003)(136003)(376002)(186003)(66556008)(478600001)(52116002)(86362001)(2616005)(107886003)(8936002)(5660300002)(1076003)(4326008)(6486002)(2906002)(66946007)(66476007)(956004)(6666004)(26005)(316002)(38350700002)(38100700002)(7416002)(36756003)(83380400001)(7696005)(54906003)(8676002)(44832011);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: SJvcJzjcdKTOX7QqR2YNNQZ8RgrEbGFHJIxGE36CHR6Mt4PFKd2uua0sa1FCGgCGzaZLzn9SObJ4QgOfOZ9BoxhOo/c2WM/nqZyRJY0ae6hJ9KtXjFnAS2e/OYZJu5RbXHjlkthgNQPIF+ukbuqWnNiwhmM+QQK6EJtY5epBdgxR9xRUvQDJ6E/Tvu0JTVzT+9hmGNJV2QtgMQ8m4JLSsW2NrhznD6XMNhm2Ax4qNRCmuGt5AqYILo6YmA1ttUiGpDxc4Ou3onSSRF9WdYF9f0SrWWwUsz1c4C2j7wIJMEnIqm914dmlv+QTsMIKk3S5zF8OukeuxKxmHf1jNs1iY0vMQlv3bo4HMm9QoOwv8ckIZ7oDNcxgiVK4SGjR21MGs7jmSyMMYEtDBws5rmhsRuAM3ysQGXY5dxOgKZ3+V6F3MwnKE9AsyWTB2RqVXjb0H02WUTBInCjlGFptRU33B0p61f4y5EUVcihRvWz73ipMocIrYHKqd/mSr+hCeDO3csuto1+IiG37mP04nZicAfpZ8/1hBBkaZFVT75ET9ZOkqLMJJCVl9x7bsQCsOj3X2HStzvJSJ+oM0gmp34U3P1oIMiK5ktoFBX3Yw7l+KE9JkHdcfItnsyC9X2w0ppjBpcgu+SxtWTMac1pmNht9WxPdmG2PekHxJ8eFhcb++6BaH8pfjU0XBGPowiAVs2MswI2XmP9flIoWC6SCgfmmkzBnGamwZY+Z87Xbpv0oiJDwLgePs6LTgzTcTuyszvIlNHqVcQ6FnLEjcNdPttJj+L+nmh0fcoUhEPnQR9jFwzG2IbHHG1TQYAEmZCfR1/+1DxLx4tvj83y89EOS36CC2Ln8sMomqgvL+NrixKes19HbjB/0JK4oqSMG6Nei6qwW18ZMB9aFJuS2CFYp4bJx6d7ElAjZ40xyE268QSBPSc00ig8ke+PA5zV+64RnFV7+SPFQhG0ZxOzWv6z2uk0hjkNEm1Gj9hazkpptfdi6haAMlFBzE+spZitWbxTdZeGlLSCAyEdbtqnAioSr7Lg3q2nX14GpoHJDbdJGSTa5gbRpPTVf0dWa8mUCAAUKS/OKTiikcA6Ey8+AhIu9MBJvKVToaj9Jk0xhI73yiSl41ydxZsX8D+/p3jvpdGkWCGlJTyMyZ5zwU9BSRGtRzwcXMisYaa7M1/RVVajLsldPFjNmmUXgF6+3d27EaQG1LC/z5A5m+g3vcIxL3qELrtaPdlwut+//dWOhmaZCUsWq4FPdsEOnBqwLYmDw0QPDK1WfYwswBvJQ/vsWf7AvjadZMMjetd8VspRyLSD22X++KiekI1Mu0wWNx/eVwFMb3eVp X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4de894ae-193e-4d92-dbe7-08d94c9c091b X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jul 2021 23:05:30.3151 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: zTTSB7cXXKv/a7RwlE0OM4kR23bDOKa2EzMoSxGzWJlcUYr1xtMmmLT/CXGm8LqkEx0IYjf0Aqe/nikBHRMFrA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3735 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10052 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 malwarescore=0 adultscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107210137 X-Proofpoint-ORIG-GUID: dBhsLCujzss-5qDBFDXxlv_qxCySU_wp X-Proofpoint-GUID: dBhsLCujzss-5qDBFDXxlv_qxCySU_wp Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=RQBylKXz; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=WTRZXYI8; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b="P/JZRA/0"; spf=none (imf24.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: E092FB00518F X-Stat-Signature: aopqux16he6so9bhqxeitw7ictufh5dd X-HE-Tag: 1626908738-372264 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Demote page functionality will split a huge page into a number of huge pages of a smaller size. For example, on x86 a 1GB huge page can be demoted into 512 2M huge pages. Demotion is done 'in place' by simply splitting the huge page. Added '*_for_demote' wrappers for remove_hugetlb_page, destroy_compound_gigantic_page and prep_compound_gigantic_page for use by demote code. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 76 +++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 73 insertions(+), 3 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 0ad75e7775dd..9bbfeba3e1ae 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1279,6 +1279,12 @@ static void destroy_compound_gigantic_page(struct page *page, __destroy_compound_gigantic_page(page, order, false); } +static void destroy_compound_gigantic_page_for_demote(struct page *page, + unsigned int order) +{ + __destroy_compound_gigantic_page(page, order, true); +} + static void free_gigantic_page(struct page *page, unsigned int order) { #ifdef CONFIG_CMA @@ -1418,6 +1424,12 @@ static void remove_hugetlb_page(struct hstate *h, struct page *page, __remove_hugetlb_page(h, page, adjust_surplus, false); } +static void remove_hugetlb_page_for_demote(struct hstate *h, struct page *page, + bool adjust_surplus) +{ + __remove_hugetlb_page(h, page, adjust_surplus, true); +} + static void add_hugetlb_page(struct hstate *h, struct page *page, bool adjust_surplus) { @@ -1764,6 +1776,12 @@ static bool prep_compound_gigantic_page(struct page *page, unsigned int order) return __prep_compound_gigantic_page(page, order, false); } +static bool prep_compound_gigantic_page_for_demote(struct page *page, + unsigned int order) +{ + return __prep_compound_gigantic_page(page, order, true); +} + /* * PageHuge() only returns true for hugetlbfs pages, but not for normal or * transparent huge pages. See the PageTransHuge() documentation for more @@ -3271,9 +3289,55 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, return 0; } +static int demote_free_huge_page(struct hstate *h, struct page *page) +{ + int i, nid = page_to_nid(page); + struct hstate *target_hstate; + bool cma_page = HPageCma(page); + + target_hstate = size_to_hstate(PAGE_SIZE << h->demote_order); + + remove_hugetlb_page_for_demote(h, page, false); + spin_unlock_irq(&hugetlb_lock); + + if (alloc_huge_page_vmemmap(h, page)) { + /* Allocation of vmemmmap failed, we can not demote page */ + spin_lock_irq(&hugetlb_lock); + set_page_refcounted(page); + add_hugetlb_page(h, page, false); + return 1; + } + + /* + * Use destroy_compound_gigantic_page_for_demote for all huge page + * sizes as it will not ref count pages. + */ + destroy_compound_gigantic_page_for_demote(page, huge_page_order(h)); + + for (i = 0; i < pages_per_huge_page(h); + i += pages_per_huge_page(target_hstate)) { + if (hstate_is_gigantic(target_hstate)) + prep_compound_gigantic_page_for_demote(page + i, + target_hstate->order); + else + prep_compound_page(page + i, target_hstate->order); + set_page_private(page + i, 0); + set_page_refcounted(page + i); + prep_new_huge_page(target_hstate, page + i, nid); + if (cma_page) + SetHPageCma(page + i); + put_page(page + i); + } + + spin_lock_irq(&hugetlb_lock); + return 0; +} + static int demote_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed) __must_hold(&hugetlb_lock) { + int nr_nodes, node; + struct page *page; int rc = 0; lockdep_assert_held(&hugetlb_lock); @@ -3289,9 +3353,15 @@ static int demote_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed) return 1; } - /* - * TODO - demote fucntionality will be added in subsequent patch - */ + for_each_node_mask_to_free(h, nr_nodes, node, nodes_allowed) { + if (!list_empty(&h->hugepage_freelists[node])) { + page = list_entry(h->hugepage_freelists[node].next, + struct page, lru); + rc = !demote_free_huge_page(h, page); + break; + } + } + return rc; } From patchwork Wed Jul 21 23:05:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12392477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78EF7C6377A for ; Wed, 21 Jul 2021 23:05:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1E6A06121E for ; Wed, 21 Jul 2021 23:05:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1E6A06121E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A3B426B0075; Wed, 21 Jul 2021 19:05:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9746C6B0078; Wed, 21 Jul 2021 19:05:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6DCCE6B007B; Wed, 21 Jul 2021 19:05:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0086.hostedemail.com [216.40.44.86]) by kanga.kvack.org (Postfix) with ESMTP id 458776B0075 for ; Wed, 21 Jul 2021 19:05:41 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id F19E61844C6AB for ; Wed, 21 Jul 2021 23:05:40 +0000 (UTC) X-FDA: 78388128840.02.0C8A838 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf06.hostedemail.com (Postfix) with ESMTP id 5C0FA802BE31 for ; Wed, 21 Jul 2021 23:05:40 +0000 (UTC) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16LMvacR031904; Wed, 21 Jul 2021 23:05:35 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=T7aTzh/od8/z0na96515K7P39K5dkQ+t5If8qBwYzOo=; b=l2DgWEUGsDftZi2+AvHeAXZ0GlFmkXdr7x2qivj9sFrKGbQTEEzSkx5QxFwIcKJ1Pd2u iIbVeYYwWpFXiwlBl1lTD6HbctyggALYr9X71KxRmE/qZztSWZmgEhqh3uVIjCDu820P XNcswRvdv2f+jgJLngnWTd73l80AXkk+Rh40Hfip1FfZxGNuyXgjv174/7NPcaUe29Zp N8fOqh2r9dYt+4KdjPUHdQdvnb3Q9k+6ciaRaPfVgSejKJisPZJDqDwtsb7J+4okf9dD FyAUCW0v/HFzI3uS07W4iRMfOmy0oQb7JVcalB2sED4SRIh9IMO9G5VPDN5LYo6kW8U2 TQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=T7aTzh/od8/z0na96515K7P39K5dkQ+t5If8qBwYzOo=; b=y2B8ZyqN2z2z+Bx72kQc2gowmGMMUmnGWGKvWKtcw+1vFR+4Nbrf/2/SyfuTmJEputNo MA8dzUesXd7IP7yOHKpKXFYMyyNkT2lFFJwsyEivJqnz0MnM3mIKcytYOq+uhRVxBHVl GKLZLamgvyXJitC3CioEJKrhxZGSM3wqRtfPL0SJVNEjnvn/fUQfz3mRSleA+y6Xrw5B /4lad5cmFJ6EZeGMUt/LzTUAPcYsMbIxWlY/VBAGxKsifmZKmB9yRqSD/pDs1Xs1kjl1 8nrH7rlRQvMs4zQG2rQWm/MMl+M+pGFeQrD55DkdYoiHaf+MZd4BnccMeOysLKwI/ZYM PA== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 39wwmtuxqt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 21 Jul 2021 23:05:35 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16LN5OwM103297; Wed, 21 Jul 2021 23:05:34 GMT Received: from nam04-dm6-obe.outbound.protection.outlook.com (mail-dm6nam08lp2047.outbound.protection.outlook.com [104.47.73.47]) by aserp3030.oracle.com with ESMTP id 39wunmv2mm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 21 Jul 2021 23:05:34 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Dk6LEmq6skipAO7lyZPV3T15RcBrOAEifWI4QN+POSMklnp1b98FGmoTKgSYSag7+awa6IX691h0CxKILJCUbggR8DBMWnoio4Hn77kSVlPwns6z80rZKq8pUN63f3EFeK1fCWBd4hMkhwsOLoJaSlgSzBbwOy7/SYG0Kdi4puogtm99l3jj18IDKGdAkxbHzfbmZ2wgekNpNELmQIzPCzibZfy55CQS12GFJG+cmidSMrOP5DXRwjD8zNETnh+MduTOWw+i2jfvISBqwsgRm7+dks+wMi6Epx47CD8Rvm+aHiS1NUWc0L9yxocOmqM+ZRrS0/r/hDmVmulmieFBPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=T7aTzh/od8/z0na96515K7P39K5dkQ+t5If8qBwYzOo=; b=MajbTL3/NBPKqR0u9+SZwWXB64bFBLp4vvdBcQFR/kfQECxjEgb1xwSeD1q56hEJlKGeHIF4rRbNSKyVK4M3zrm81UTIIyfg16ClkNdQ0ADslmVlXhE2Y1XIydHiTjL8N3HzAly9Q9JbdQ4p0GzzyHx1BzMYNxoXPE60344L9voY3bKzfyVM41K+s1+14IvH7+nQXEF9PsInFdTqOfJPJRXsLjdEdNIRgmv6MwSceZau20zDOGrU0q/5Q50VhhforRXOgPZ2cwXYYIOUJzgNCTP6NseJt45fsV6VcobZlgZk6EcNYpEgRNF7YiBnd0LgTcrtCrcsmDhfravJL/Ereg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=T7aTzh/od8/z0na96515K7P39K5dkQ+t5If8qBwYzOo=; b=T+K+dX3+OMjox3JWLan0iaN3YS/B6TI44IX+wXSbrxm7ttXcu2zcPfzGmJbTSDZTYupPdtBdH8iOhcezI2k53CXbNzfgeN+7Y6LPqM1oTGMsF06GP6gRZjeM9fn0GtGvAsbXpmTMLlnhNgb+CsT6fy7+ok/6EW3ahfkuD74wqXU= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3735.namprd10.prod.outlook.com (2603:10b6:a03:11e::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.26; Wed, 21 Jul 2021 23:05:32 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a%5]) with mapi id 15.20.4331.034; Wed, 21 Jul 2021 23:05:32 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Michal Hocko , Oscar Salvador , Zi Yan , Muchun Song , Naoya Horiguchi , David Rientjes , Andrew Morton , Mike Kravetz Subject: [PATCH 5/8] hugetlb: document the demote sysfs interfaces Date: Wed, 21 Jul 2021 16:05:08 -0700 Message-Id: <20210721230511.201823-6-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210721230511.201823-1-mike.kravetz@oracle.com> References: <20210721230511.201823-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MWHPR1201CA0021.namprd12.prod.outlook.com (2603:10b6:301:4a::31) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR1201CA0021.namprd12.prod.outlook.com (2603:10b6:301:4a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22 via Frontend Transport; Wed, 21 Jul 2021 23:05:31 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 91539aab-2700-47df-b3ac-08d94c9c0a4b X-MS-TrafficTypeDiagnostic: BYAPR10MB3735: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: CWrHud3LUDXWF5QOGbZkp2ega3iSG7YPU/edzIA0TQOinnDUdAG+jzKAGY+ZxjWEi7qmKyyYUdzJVdL/b+gC0lEMZuqv8xVS8Kcaj9S+ZWnHYv5Q/6Xlw5ZdhvUF1/SEjOp/msoFcn29JpdOFZFuvRWtpWROZg9ZnfeVa2zsqhMzeJwMppAxkdDEJXflGFgaoNSv8JWx5nCjpjj4yIHK5q+8kfVOFEBkP1YI98eosya+0AUgLJqgLedFHSLA/oZcoBYiuZPPCzNbbNrCFCwYMQts5ffCiMA441eeDiiPDZ/82HayyQgFlr0RzQC4vjdGgDLaeJZp4KD+ocIJBQIrzP9CDIXKEYtmUUWM+1gJBxoPOqWawGzKY3v9p4dnLVBZTVe/ioZo2lpBGroEhw1KcvhDs4iqJ9NF5J4xhyhqQC/M8kGuJkra9ssMQvDYCY3SgQeo34l0P0QBj6gMMJiuBbLnvRFCCNfu3Y42Th/X7e8XPbQLgRjZxqI3LJfdISphozWLNzoQtr5ifDh2J8xucmSblZGb4GSXBFwLEkexcuQBOWJWtYIqgNbQ49YPLWe9gDda89TqVzY6bDBtNee7P+NR7OmhOwt3QXR+xJ6jvR82+k2wK+c7rbKlMhq245c79zTXKWsR1TkHdMu+mtzO1Bh0r8Yj3lZBd8lQiZ2yQ4yMITfENwvOXFVBqnviC5izDJJz4eGFNKwGo4mEqcYVcQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(366004)(39860400002)(396003)(136003)(376002)(186003)(66556008)(478600001)(52116002)(86362001)(2616005)(107886003)(8936002)(5660300002)(1076003)(4326008)(6486002)(2906002)(66946007)(66476007)(956004)(6666004)(26005)(316002)(38350700002)(38100700002)(7416002)(36756003)(83380400001)(7696005)(54906003)(8676002)(44832011);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ITnFFS1lmTO5fZvXzuxZIVw+aNyPb4WiW0NYUOK/qhXUqGKq8A7Cuoff5+2LI9O0DwbvFRw0bl/e1XVv1/R4lg3fYyv47RVJg839CMnxG7nONaHA5SloKjX1jefQ0D/HVDJ4xW2v7K8XaZW2qqA8uBvyJ9p8mwXaFRFv2JGOSJFaw/GHdOfz/u4xjr/ArfUsAWZW5Fw93KRbeQwzELIAyIwnJk67FuhvzTpKYkqJHkba7s3jAowxjHCqoxKxpgkMrmcbF64sTXD2Ylub47xFmnHHS5TXQfHkTjYYod8Zc1KZ/3lGSjKW192NGSBA6napIrSqiLUKnvCIE66SWSM96B+fgVflBrmaHrbekaCnJmRBqabXov3eVJOvtxAsRZmj0xlU5pNgXIHCFvfW2UFZnZitMPrwqu7D0fnhsTu13k99M3YGAeezySxFVB9Ns3sVqxhfKQ7zGdVqEZeEKQIhs+ZasbiZjGht5+mYjWOaS2txG4M39q8IhUWSIrCidBFKn6F3tTJVjPjOe7O/28LAmi8WvQF1sLBcGYoLY6X3cV3oi6fiuFuQxyn//3GUgxTycEl31EYc0hLE6AsxkYKaFwUQqK3RdTNOfyj2WPXSC+LN5kri8y5ZHPjoQPIbLy0vmt86VYSgv4s5iKyi/PC8wbZfIJY0tTo6LHW+0HKzUuvydNbiUXuD0Itz8AKw/FnM5lWccKbZkRRBNcsYq5W+SyRgccVxotbFfKiuzjcnimJrs6sTnXsJ7lmrT8MXhw96A9/DaBHs4I3utKCvj2SbZ2c1Hsq6MEJ3qJPto3bqm01gMNRCRN+234ow+Xmht2anksCAA2/UcE2bd/6fJPRb8mYGg9s/8viGX/1MD8FtgkWbSYnLAUITQnV2T9wt3brNQjgjThNbrewH4ork80GS/XO2K9skR5FtyNPSgbjlGHw2ZREjl+3gV0+utszPPiXD65wO6DuJ7Vxi6bS/XwCSmEjMB9xBJ36Nn4kqa1tu1rEZAxYd+XkNqZxfNjJLgML+v2Gei9NGkgcVj/DdWVqYm7tBkORV17He6BgNDY5R0Pb3talYh5QlLc8kW5UY/bmaLsF69H/O41byMvjywKQAVJAWCBzJyC0yTctt5HUI1uWLGnIDJYRkH0IYaLY3Jm3mqxUneby/BE9T6GZEZJy89QzWOSN9QzaKX5Bvv80uIJuYJKVKgYxqVtjKdLIK5/EHrHMQ+4LQynv4l28CgOkfdT3W6/v6JxiIe57EMOwOlxOcSeTlNxw9vffVyiUbCHhiV11BbtH+ugri7AH/CSqn7CiJkwlw+Um/8l5RGglcz8ChFe2BcI5wyz2B/TEI5KLv X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 91539aab-2700-47df-b3ac-08d94c9c0a4b X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jul 2021 23:05:32.3004 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ZLzxgg6XlU+8LOF+Pgt5O/YfY/IjLWZsU6l99c/fK8l83wvVvZNSNlfyueb7crODUHrw+JZpmpLE8I26U0T3AQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3735 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10052 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 phishscore=0 spamscore=0 bulkscore=0 mlxlogscore=999 adultscore=0 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107210138 X-Proofpoint-ORIG-GUID: IAT_FspoB-uF-p2PsaCpeUZfGz5AJhnP X-Proofpoint-GUID: IAT_FspoB-uF-p2PsaCpeUZfGz5AJhnP Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=l2DgWEUG; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=y2B8ZyqN; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=T+K+dX3+; spf=none (imf06.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam05 X-Stat-Signature: j8wdoeq5rytcg18d15ntgof346nq57od X-Rspamd-Queue-Id: 5C0FA802BE31 X-HE-Tag: 1626908740-662982 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Describe demote and demote_size interfaces. Signed-off-by: Mike Kravetz --- Documentation/admin-guide/mm/hugetlbpage.rst | 29 ++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/Documentation/admin-guide/mm/hugetlbpage.rst b/Documentation/admin-guide/mm/hugetlbpage.rst index 8abaeb144e44..902059a0257b 100644 --- a/Documentation/admin-guide/mm/hugetlbpage.rst +++ b/Documentation/admin-guide/mm/hugetlbpage.rst @@ -234,8 +234,12 @@ will exist, of the form:: hugepages-${size}kB -Inside each of these directories, the same set of files will exist:: +Inside each of these directories, the set of files contained in ``/proc`` +will exist. In addition, two additional interfaces for demoting huge +pages will exist:: + demote + demote_size nr_hugepages nr_hugepages_mempolicy nr_overcommit_hugepages @@ -243,7 +247,28 @@ Inside each of these directories, the same set of files will exist:: resv_hugepages surplus_hugepages -which function as described above for the default huge page-sized case. +The demote interfaces provide the ability to split a huge page into +smaller huge pages. For example, the x86 architecture supports both +1GB and 2MB huge pages sizes. A 1GB huge page can be split into 512 +2MB huge pages. The demote interfaces are: + +demote_size + is the size of demoted pages. When a page is demoted a corresponding + number of huge pages of demote_size will be created. For huge pages + of the smallest supported size (2MB on x86), demote_size will be the + system page size (PAGE_SIZE). If demote_size is the system page size + then demoting a page will simply free the huge page. demote_size is + a read only interface. + +demote + is used to demote a number of huge pages. A user with root privileges + can write to this file. It may not be possible to demote the + requested number of huge pages. To determine how many pages were + actually demoted, compare the value of nr_hugepages before and after + writing to the demote interface. demote is a write only interface. + +The interfaces which are the same as in ``/proc`` function as described +above for the default huge page-sized case. .. _mem_policy_and_hp_alloc: From patchwork Wed Jul 21 23:05:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12392479 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C8D1C6377B for ; Wed, 21 Jul 2021 23:05:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AA9BA61029 for ; Wed, 21 Jul 2021 23:05:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AA9BA61029 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 556C06B0078; Wed, 21 Jul 2021 19:05:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4B9DC6B007B; Wed, 21 Jul 2021 19:05:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 296C78D0001; Wed, 21 Jul 2021 19:05:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0249.hostedemail.com [216.40.44.249]) by kanga.kvack.org (Postfix) with ESMTP id 01D9F6B0078 for ; Wed, 21 Jul 2021 19:05:43 -0400 (EDT) Received: from smtpin33.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id A74201BCC2 for ; Wed, 21 Jul 2021 23:05:43 +0000 (UTC) X-FDA: 78388128966.33.E2739E8 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf21.hostedemail.com (Postfix) with ESMTP id 049F5D01FE5F for ; Wed, 21 Jul 2021 23:05:42 +0000 (UTC) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16LMvDcM009325; Wed, 21 Jul 2021 23:05:38 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=MIxRDmB8nzh0QgtoXqdcIgpuGwLwYMo24Dc9amAPQRU=; b=GOxLJ5Uy8KAS0ltujil3lUmhkt9DLfbwkfuyz9HSqEtSamy4Hd1bRKLHYRF0c8C1havw FIjocxvusNChP4/LDK4leorzU/d8gEcwHZBcGmLNH+okYzTcJdWSH4GHASkM35fIpmnj QS+E/uBeRCp5TjA5eNnMJb1l+ox4/iFiiNA4uAkAnpgexFDFK9E30zDPt/vkL9HiX5Hq VIxAy+Y6lf/MMtorCq1ur7s//kPYjz8Mcc47yFyF5/NfLlt96zqTEOEWif+bwUoe9B/J z51Kk11xHG1JJegzDrrcjWQ0gHzgkLDc+Ka2FHMLZULfkGNMNfyS3BToU0x/1+s0yDoU WA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=MIxRDmB8nzh0QgtoXqdcIgpuGwLwYMo24Dc9amAPQRU=; b=sF+mi86m7x+7klhRkbAiTsU7KXFVZmk8gXSXF98W3uxX/ZN8PSnwaB3BHpaA6/3xNEa0 GnI4oLcFHDXcnIuS5LJi+bLHhA1jImcEhR1iVW+R212nEQ1eWlNKGZLsjIN8vKGoJvLh T6X7IvA9dSeOL+KYeLDKA8HmcNQWVnqKlrkWNa0hmJxvLUllcxLLL5C6km/BDILkvPt3 tHZL+WVyQ3f2YI1Gjdof56ydFMs7vfk6rBgWJwcXgCLbmpA/gc2lwAZusmOVHS/B0RAT 7Q1DOWFdDEMlZbyT5jgbzRM9GzQ6ac8A/291H9Bili21c+p/viXuSIkCRH4eVsmbusZw lw== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 39xu2fr5wr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 21 Jul 2021 23:05:37 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16LN0ZqG143586; Wed, 21 Jul 2021 23:05:36 GMT Received: from nam04-dm6-obe.outbound.protection.outlook.com (mail-dm6nam08lp2042.outbound.protection.outlook.com [104.47.73.42]) by userp3030.oracle.com with ESMTP id 39umb3qe17-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 21 Jul 2021 23:05:36 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TKX++ncLI05qY2xt2tYwGhZoVkocEzGyL6v+MRLKI7pIStAa50FbZU/r15B/Tmhastf0yHr/pM9ROAdLSkcMMRGbx82jbZtdmvMXK4WnrYqU7yi14S/MoYdfA0kMdbb287eEWhOqjR5RdY8DHVGIduNMTfcV9R5+HLty9vNxiKmo8XZe7rBXbaVRcEB5sjyWvdj0vvC2ca/zYw6LVk6pWL1Bn7QammDOKjecbRqKcULHfjgUH80hK8hlLgdjOEuU8T4t80TmKR2xrw1JSQcbV52UABGoSPd9BBQ0+jy9rpiuNZwGtiN9OKTSpEoh9cZblAHxozWUwRisHqXFtV9vZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MIxRDmB8nzh0QgtoXqdcIgpuGwLwYMo24Dc9amAPQRU=; b=dSh3VwM4o9nH9dvwiyS2LklbEjIaZNSq2P1Y0K7+F5LliEwhvu4mRVcYwEiY0pVNaagevvJD7mM2OrDOdY4hpMiiJUT0lfdzSF08SiTz8V9gKylmAbH+fp2SuZGKWc/pfzRuA9ALYw+7g4ZvkJ3Re1ntnlSFbz2SyoB7ioxQytJZb6XJIsB5mIQ37SUUY5T5FGZ7Uq6tDgL3lleRht7lJ9QT4VPJhx6zwopUO2qJQbczLNwrOc439fIv87DDmamzmUccJBwkR+wRO3vZFqYS7uBIo423EQfxdgipODNayPchd7/9WOtxaHdTYjO9giv629noNOl7sZHxXBA86BxKRQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MIxRDmB8nzh0QgtoXqdcIgpuGwLwYMo24Dc9amAPQRU=; b=dihowa4tBlV0ceufeFGsKj29cZ0flQ/uHoTtoxph44cowHrTJQ/3OEhlkZ/4rvVNDyIquP1ysG1JIC3RTKE5Rw0zhXGQpYFmAhNeZCW8RmVDySwmYabKkLdEJWMf7yk8espE+SxhlFIXj9QYefl7odA3rVn7i8373nkOao3e1+s= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3735.namprd10.prod.outlook.com (2603:10b6:a03:11e::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.26; Wed, 21 Jul 2021 23:05:34 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a%5]) with mapi id 15.20.4331.034; Wed, 21 Jul 2021 23:05:34 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Michal Hocko , Oscar Salvador , Zi Yan , Muchun Song , Naoya Horiguchi , David Rientjes , Andrew Morton , Mike Kravetz Subject: [PATCH 6/8] hugetlb: vmemmap optimizations when demoting hugetlb pages Date: Wed, 21 Jul 2021 16:05:09 -0700 Message-Id: <20210721230511.201823-7-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210721230511.201823-1-mike.kravetz@oracle.com> References: <20210721230511.201823-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MWHPR1201CA0021.namprd12.prod.outlook.com (2603:10b6:301:4a::31) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR1201CA0021.namprd12.prod.outlook.com (2603:10b6:301:4a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22 via Frontend Transport; Wed, 21 Jul 2021 23:05:33 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: abf508e1-0f52-4e57-cdd9-08d94c9c0b4e X-MS-TrafficTypeDiagnostic: BYAPR10MB3735: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7219; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: UtSVaq29R2H7e6Lhe416hhMcRakCjyUwt+8B6nX+pWXV6PpHEoiG73TPdwZ1a6x5gIkuX6fW8jibHmxCmjx3vyLsfpJ0kmRb0sRwbplVb+pG5V5UUuZ0X2ppH15CuyZF1kLbko5OLcWtyIR8opHbLHbwkvHMiDNbDdaQ4WwlTha2JxAG0vEgxOjhNVC0ZwPOfkIRncxdjbOo93t4sS2d/DmlJ2twaZ6O48j4XaVsrjMCJykrPNSMRthQi3+tg/PJYEA9RTiRrAJ7ESoLjOIAmSwMKedypUFTGVZPLxKM5iLCV4YaxAUAiXDhcCWW9iSZPO+i+mw75Riwg/5ZQXBAU9ITOMpT5Bi45YxXnj8TLwF9aWahfAG49Oh6CkrRNfqCGGEAu/DBSGFgEHYRwmW5rB6tHXDMnHG9eoPzQcarywdos7wKSyw+3bDEdaAhX99XIpvA6/CY9YDHEdFKF9tnP+ufW+P4nStPBtdOB48F1Z6vjLtv9JvA5kP0BgJqjC4sPcI94utqOUh9lqY2MrGMrTTmiPMI/ntrvoiOmki20qbjzM9/+uhzbfJrs8J5tVsDcHcRbGzbdWOY390fmykK4Yg/PgTf//90s4d3cS+d+8tXr4OE1nxwmcWck0zrqeDsLV0COWg7tgL+RlYHvFk1PlWGSSfDin7gybIt3NcMRuhlscfAqCu3/UF85vGXeMXtUfdC+Rv2QJLjx1QdExImaA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(366004)(39860400002)(396003)(136003)(376002)(186003)(66556008)(478600001)(52116002)(86362001)(2616005)(107886003)(8936002)(5660300002)(1076003)(4326008)(6486002)(2906002)(66946007)(66476007)(956004)(6666004)(26005)(316002)(38350700002)(38100700002)(7416002)(36756003)(83380400001)(7696005)(54906003)(30864003)(8676002)(44832011);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: XIVWhSapFP0sga6CMAaCIqCGZhnWqy7Gw2/qWdtCbfsQ5JSJU8sbckPokcasPx3lrX0DtVp9VDpPGxqF71YHbD0loxBS2hulDB1UIk4GgiU46pHXnJpjXItvhZWi7HNS2FNhC4Bv189oR17es63IW9us/ApLUppgntEC0CiaJZvmk3+b9akDFgx+ztwj6PM+lMcbwTaZVtWQ/iXbQR6jBfQRh90QmWYsUxPjAa/86w+HVzyzI/JRowq1QLdjwyttwkvMiah0LgCwEhSy53iOH00wzzVKyEdtimLgjbajfr4c6Dlasyv9+5PyrFgxpMty+qMh3+aL/0UaWcqVqWMqi0cljs1fjXVomXELA/4d86Z9Nvvjr8c/bARvoqkZcqwTw606kxn6Dt0g34R4mbPl/g+2VtG3E8uJ5kVmIki0dGI1i5eJ5wx7ZhAJcQDrKejLB1y/Yc20toYVL8SfXhIZQvG+V73+IbPk1moa2oUD4QAopOZyYxyNDWT1fWwzbFc0kqp1lYOX1Hj8BYvdA57vyi3BOSXNkj2ylPuzgX7pZJ4WXU/SPo4jLP/+Ks9xL6+PTnw+q4z+FT4hG/jnOYBk6qyF54MC5NBl5aFQbn/SkLITmFI4JERvtYtpXKHYeU5B29MXaE23CvM1whvpl6YTKDKIWgPWkRTgvLyCRltrgDgx8PxF6DzzkdY2lNnb5efLAbv9Bfy9r9W1VaNJ8o1ctxe7Lgku/imseVoNCOLFPWD1x0mOjDr8glg7t7Dd2vlbQ4Wf2s3AYjBr79j/+FzkU9ddEcK7P4aeMwFEOtJ0chR8VcRm6a0GNMG6fLmSmoxe+GR8zQ/+VZCWeF7ZHseQFAEBtpILIn4XZXyUE2iR1xvVIrKKJ7CwmVGp1pBVIi5Cud8oqUvOqSxxtod5jdG17pSm9G2sLNGtpDJQCMTt9tyyBQ4NoFw3p9AWpB/0M/t+BSFhrkIHDnfg5vhR4St0iQav5tlH0tyweEVyz6kfO1rWYNA4t96IgQB9JKXew8jeKqb5wa9xNKFq4GcuQEqRS3c1aLn3SVcYH1VtFoiWUSgqRNAbBNSoXeGYWzJpKb1tncmYtDMcXr82Ecd1TD8DSS4iRfB6KwfIfjpI1POKuTnFBc8hsOn7nQTtnxk5rIaY7sOY4kymWIJXnna0ir2oyne+d2O/QdeCfeR9mIVi4co5/tkbh3eSRQAv5y/LVWUqwfTPhchC73gzAfbTXxvYRB4ce6JwXdmJE7SxVWQ8MyuztWJMfgbigQDcXUcUrQLNppUNU5v8QcFFzOZhroYjPR+htZbZDfJvxd7kLsZ2kHD9uJh8JF0M/030QlXrDiXJ X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: abf508e1-0f52-4e57-cdd9-08d94c9c0b4e X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jul 2021 23:05:34.0751 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: K5fW3Ys/lwaf+eytBHiz20vs84O6dlkvTbXklTYGzYwxdXhBcVK6kzlmLBq5U/Q71foeIzRgNGEtZmen5aFICg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3735 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10052 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 spamscore=0 mlxscore=0 adultscore=0 mlxlogscore=999 suspectscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107210137 X-Proofpoint-ORIG-GUID: SI4Vrv7gCgMFVQL_iyJnfJ7rX6OI_OLH X-Proofpoint-GUID: SI4Vrv7gCgMFVQL_iyJnfJ7rX6OI_OLH Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=GOxLJ5Uy; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=sF+mi86m; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=dihowa4t; spf=none (imf21.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 049F5D01FE5F X-Stat-Signature: oeyjpnii353gzzws84jehfp67ubekmjo X-HE-Tag: 1626908742-47253 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When demoting a hugetlb page with optimized vmemmap, we allocate vmemmap for the entire address range represented by the huge page. Then, we split the huge page into huge pages of a smaller size. When preparing these smaller huge pages for use, we optimize vmemmap for their associated address range and free vmemmap pages. It is sub-optimal to allocate all the vmemmap pages associated with the original page and then free most of those pages when preparing the smaller huge pages after the split. Instead, calculate the number of vmemmap pages needed after the split and just allocate that number of pages. Then, only populate those areas of the vmmap required for the smaller huge pages. Introduce two new routines: - demote_huge_page_vmemmap - This has knowledge of the hugetlb demote process and will calculate the number of huge pages needed for the smaller huge pages after the split. It also creates a 'demote_mask' that is used to indicate where within the address range of the original huge page the smaller huge pages will reside. This mask is used to map vmemmap pages within the range. - vmemmap_remap_demote - This is the routine which actually allocates the vmemmap pages and performs vmemmap manipulations based on the passed demote_mask. These routines will be used in a subsequent patch. Signed-off-by: Mike Kravetz --- include/linux/mm.h | 4 ++ mm/hugetlb_vmemmap.c | 60 +++++++++++++++++++++ mm/hugetlb_vmemmap.h | 6 +++ mm/sparse-vmemmap.c | 123 +++++++++++++++++++++++++++++++++++++++++-- 4 files changed, 189 insertions(+), 4 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7ca22e6e694a..269b135489e6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3080,6 +3080,10 @@ int vmemmap_remap_free(unsigned long start, unsigned long end, unsigned long reuse); int vmemmap_remap_alloc(unsigned long start, unsigned long end, unsigned long reuse, gfp_t gfp_mask); +int vmemmap_remap_demote(unsigned long start, unsigned long end, + unsigned long reuse, unsigned long demote_nr_pages, + unsigned long demote_mask, + unsigned long demote_map_pages, gfp_t gfp_mask); void *sparse_buffer_alloc(unsigned long size); struct page * __populate_section_memmap(unsigned long pfn, diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index c540c21e26f5..c82d60398c16 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -264,6 +264,66 @@ void free_huge_page_vmemmap(struct hstate *h, struct page *head) SetHPageVmemmapOptimized(head); } +/* + * vmammap pages will be allocated and mapped such that this range which + * previously represented a single huge page will now represent a set of + * pages of a smaller size. + */ +int demote_huge_page_vmemmap(struct hstate *h, struct page *head) +{ + int ret; + unsigned long vmemmap_addr = (unsigned long)head; + unsigned long vmemmap_end, vmemmap_reuse; + unsigned long demote_mask; + unsigned long demote_nr_pages; + struct hstate *target; + + VM_BUG_ON(!h->demote_order); + if (!HPageVmemmapOptimized(head)) + return 0; + + target = size_to_hstate(PAGE_SIZE << h->demote_order); + + /* Number of vmemmap pages required to demote page */ + demote_nr_pages = pages_per_huge_page(h) / pages_per_huge_page(target); + demote_nr_pages *= RESERVE_VMEMMAP_NR; + demote_nr_pages -= RESERVE_VMEMMAP_NR; /* pages currently present */ + + /* + * mask to identify where within the range new smaller pages will + * reside. This will be used to map new vmemmap pages. + */ + demote_mask = ((unsigned long) pages_per_huge_page(target) * + sizeof(struct page)) - 1; + + vmemmap_addr += RESERVE_VMEMMAP_SIZE; + vmemmap_end = vmemmap_addr + free_vmemmap_pages_size_per_hpage(h); + vmemmap_reuse = vmemmap_addr - PAGE_SIZE; + /* + * The range [@vmemmap_addr, @vmemmap_end) represents a single huge + * page of size h->order. It is vmemmap optimized and is only mapped + * with RESERVE_VMEMMAP_NR pages. The huge page will be split into + * multiple pages of a smaller size (h->demote_order). vmemmap pages + * must be callocated for each of these smaller size pages and + * appropriately mapped. + */ + ret = vmemmap_remap_demote(vmemmap_addr, vmemmap_end, vmemmap_reuse, + demote_nr_pages, demote_mask, + RESERVE_VMEMMAP_NR, + GFP_KERNEL | __GFP_NORETRY | __GFP_THISNODE); + + if (!ret) { + int i; + + for (i = pages_per_huge_page(target); + i < pages_per_huge_page(h); + i += pages_per_huge_page(target)) + SetHPageVmemmapOptimized(head + i); + } + + return ret; +} + void __init hugetlb_vmemmap_init(struct hstate *h) { unsigned int nr_pages = pages_per_huge_page(h); diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index cb2bef8f9e73..44382504efc3 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -13,6 +13,7 @@ #ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP int alloc_huge_page_vmemmap(struct hstate *h, struct page *head); void free_huge_page_vmemmap(struct hstate *h, struct page *head); +int demote_huge_page_vmemmap(struct hstate *h, struct page *head); void hugetlb_vmemmap_init(struct hstate *h); /* @@ -33,6 +34,11 @@ static inline void free_huge_page_vmemmap(struct hstate *h, struct page *head) { } +static inline int demote_huge_page_vmemmap(struct hstate *h, struct page *head) +{ + return 0; +} + static inline void hugetlb_vmemmap_init(struct hstate *h) { } diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index bdce883f9286..ac2681bf006b 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -43,6 +43,15 @@ * @reuse_addr: the virtual address of the @reuse_page page. * @vmemmap_pages: the list head of the vmemmap pages that can be freed * or is mapped from. + * @demote_mask demote specific. mask to know virtual address of + * where to start mapping pages during a demote operation. + * @demote_map_pages demote specific. number of pages which mapped for + * each demoted page. + * @demote_tmp_count demote specific. counter for the number of pages + * mapped per page. + * @demote_tmp_addr demote specific. when more then one page must be + * mapped for each demoted size page, virtual address + * of the next page to be mapped. */ struct vmemmap_remap_walk { void (*remap_pte)(pte_t *pte, unsigned long addr, @@ -51,6 +60,10 @@ struct vmemmap_remap_walk { struct page *reuse_page; unsigned long reuse_addr; struct list_head *vmemmap_pages; + unsigned long demote_mask; + unsigned long demote_map_pages; + unsigned long demote_tmp_count; + unsigned long demote_tmp_addr; }; static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, @@ -262,6 +275,51 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); } +static void vmemmap_demote_pte(pte_t *pte, unsigned long addr, + struct vmemmap_remap_walk *walk) +{ + pgprot_t pgprot = PAGE_KERNEL; + struct page *page; + void *to; + + if (!(addr & walk->demote_mask)) { + /* head page */ + page = list_first_entry(walk->vmemmap_pages, struct page, lru); + list_del(&page->lru); + to = page_to_virt(page); + copy_page(to, (void *)walk->reuse_addr); + set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); + /* + * after mapping head page, set demote_tmp_reuse for + * the following tail page to be mapped (if any). + */ + walk->demote_tmp_count = walk->demote_map_pages; + if (--walk->demote_tmp_count) + walk->demote_tmp_addr = addr + PAGE_SIZE; + } else if (addr == walk->demote_tmp_addr) { + /* first tall page */ + page = list_first_entry(walk->vmemmap_pages, struct page, lru); + list_del(&page->lru); + to = page_to_virt(page); + copy_page(to, (void *)walk->reuse_addr); + set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); + if (--walk->demote_tmp_count) { + walk->demote_tmp_addr = addr + PAGE_SIZE; + } else { + walk->demote_tmp_addr = 0UL; + /* remaining tail pages mapped to this page */ + walk->reuse_page = page; + } + } else { + /* remaining tail pages */ + pgprot_t pgprot = PAGE_KERNEL_RO; + pte_t entry = mk_pte(walk->reuse_page, pgprot); + + page = pte_page(*pte); + set_pte_at(&init_mm, addr, pte, entry); + } +} + /** * vmemmap_remap_free - remap the vmemmap virtual address range [@start, @end) * to the page which @reuse is mapped to, then free vmemmap @@ -327,11 +385,9 @@ int vmemmap_remap_free(unsigned long start, unsigned long end, return ret; } -static int alloc_vmemmap_page_list(unsigned long start, unsigned long end, +static int __alloc_vmemmap_pages(unsigned long nr_pages, int nid, gfp_t gfp_mask, struct list_head *list) { - unsigned long nr_pages = (end - start) >> PAGE_SHIFT; - int nid = page_to_nid((struct page *)start); struct page *page, *next; while (nr_pages--) { @@ -348,6 +404,21 @@ static int alloc_vmemmap_page_list(unsigned long start, unsigned long end, return -ENOMEM; } +static int alloc_vmemmap_pages(unsigned long nr_pages, int nid, + gfp_t gfp_mask, struct list_head *list) +{ + return __alloc_vmemmap_pages(nr_pages, nid, gfp_mask, list); +} + +static int alloc_vmemmap_range(unsigned long start, unsigned long end, + gfp_t gfp_mask, struct list_head *list) +{ + unsigned long nr_pages = (end - start) >> PAGE_SHIFT; + int nid = page_to_nid((struct page *)start); + + return __alloc_vmemmap_pages(nr_pages, nid, gfp_mask, list); +} + /** * vmemmap_remap_alloc - remap the vmemmap virtual address range [@start, end) * to the page which is from the @vmemmap_pages @@ -374,7 +445,51 @@ int vmemmap_remap_alloc(unsigned long start, unsigned long end, /* See the comment in the vmemmap_remap_free(). */ BUG_ON(start - reuse != PAGE_SIZE); - if (alloc_vmemmap_page_list(start, end, gfp_mask, &vmemmap_pages)) + if (alloc_vmemmap_range(start, end, gfp_mask, &vmemmap_pages)) + return -ENOMEM; + + mmap_read_lock(&init_mm); + vmemmap_remap_range(reuse, end, &walk); + mmap_read_unlock(&init_mm); + + return 0; +} + +/** + * vmemmap_remap_demote - remap the optimized vmemmap virtual address range + * for a huge page to accommodate splitting that huge + * page into pages of a smaller size. That smaller size + * is specified by demote_mask. + * @start: start address of the vmemmap virtual address range to remap + * for smaller pages. + * @end: end address of the vmemmap virtual address range to remap. + * @reuse: reuse address. + * @demote_nr_pages: number of vmammap pages to allocate for remapping. + * @demote_mask: mask specifying where to perform remapping within the passed + * range. + * @demote_map_pages: number of pages to map for each demoted page + * @gfp_mask: GFP flag for allocating vmemmap pages. + * + * Return: %0 on success, negative error code otherwise. + */ +int vmemmap_remap_demote(unsigned long start, unsigned long end, + unsigned long reuse, unsigned long demote_nr_pages, + unsigned long demote_mask, + unsigned long demote_map_pages, gfp_t gfp_mask) +{ + LIST_HEAD(vmemmap_pages); + int nid = page_to_nid((struct page *)start); + struct vmemmap_remap_walk walk = { + .remap_pte = vmemmap_demote_pte, + .reuse_addr = reuse, + .vmemmap_pages = &vmemmap_pages, + .demote_mask = demote_mask, + .demote_map_pages = demote_map_pages, + }; + + might_sleep_if(gfpflags_allow_blocking(gfp_mask)); + + if (alloc_vmemmap_pages(demote_nr_pages, nid, gfp_mask, &vmemmap_pages)) return -ENOMEM; mmap_read_lock(&init_mm); From patchwork Wed Jul 21 23:05:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12392481 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1B82C6377A for ; Wed, 21 Jul 2021 23:05:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6FD4E61029 for ; Wed, 21 Jul 2021 23:05:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6FD4E61029 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 076A96B007B; Wed, 21 Jul 2021 19:05:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EF2DB6B007D; Wed, 21 Jul 2021 19:05:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C810C6B007E; Wed, 21 Jul 2021 19:05:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0237.hostedemail.com [216.40.44.237]) by kanga.kvack.org (Postfix) with ESMTP id 9F24F6B007B for ; Wed, 21 Jul 2021 19:05:45 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 55C718248047 for ; Wed, 21 Jul 2021 23:05:45 +0000 (UTC) X-FDA: 78388129050.03.79F7565 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf13.hostedemail.com (Postfix) with ESMTP id B0C08102AFFD for ; Wed, 21 Jul 2021 23:05:44 +0000 (UTC) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16LMvPNG003370; Wed, 21 Jul 2021 23:05:39 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=JvqKnCWQpghkClE9kLMTl3jAzhYt2NXfJ3KJieza1Sk=; b=qWynysMrLY2cpevrKmwk2vrQFNEFKGtrSbNdwhi5XeYAfGMG3qjw4k3A6hQbSzgprsbg /ruorWdvTtJxdX012E/d/L+clYrgehYn0xXb+mp3VrhzBrkxicotLNWmd3ePuF7xZpbS 7wHeZmOQ1BH6ALhs8M7aEM6HznR2tHXtwnzvc3PK4/dWzBXy4gFhaTWJ9b1Wueee9HQy EWD1kW5ryR4nqT1ZNTYVrp8WoL7uOEb38RsVMD/UCtm4h6kmY3SC5vA8K060pj/79Sz1 yrW76GA43FugLd1wSsBMSIPWwRkvX/qqGZk7mtho6WSruSWoJZFPZhm4A452AWQN3dn+ zg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=JvqKnCWQpghkClE9kLMTl3jAzhYt2NXfJ3KJieza1Sk=; b=qOFKH6Dja0B86MXTz3RJPLmIluRHnLm5wK24wg6gVzs6TqfcA+Ag45b1yMFEWVxrWmhB pfYC53eI6hoDNgDoCCwP8LypLyYoDHJX7laHHE9HIKHHhrYzD8zDPQAaGxmkEf5U5ndX pPLOgKEirsBajvVxme0EJ44Kpa4oCGEOWWwJJLgxe/Blww+Ufdtmo+/2P8t7euLSvyKX MFm1SeATYujEjLUhLY0LkeNe4d/IDtVdtLSwckBpTpeAsPxQ59kmPlmNIrL0Gux+EUym j2lhulUuve/URebS5buNROv7gVg8EFds/nHZqv0dff2vr2tdTnAIllqQJDXJ3b87bVsZ XQ== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by mx0b-00069f02.pphosted.com with ESMTP id 39wyq0uqqh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 21 Jul 2021 23:05:39 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16LN59eQ058644; Wed, 21 Jul 2021 23:05:38 GMT Received: from nam04-dm6-obe.outbound.protection.outlook.com (mail-dm6nam08lp2041.outbound.protection.outlook.com [104.47.73.41]) by aserp3020.oracle.com with ESMTP id 39uq1a3wgp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 21 Jul 2021 23:05:37 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hbmXM0GCvfhlDm1zsMxXsePWR7lL5lTjK7yX7m3RzzmsGWvOiOPonwKRffgLR2rLYPyzf8fs0PmAZx/1Eczt1EE33DxPm6i+zqrlDPEndBs6R+MSvvPQTRcUxJkKeMeDOJ/th67um0Cs56eqxTczFXst3R7arYYfYdvN36l6rTq/vmMrLHbp24Cn+MtQV5CGGoUhJ64ngZD/neC4Kks7GmK7FHBc3JtfINSJ+Ws9Zaks814fGH1GOXoasv1MgrIC3bNovReZ2h+ECJI47XApbanYczrz+BARknobimFOeP8VBHi/5EYgblpnjAaRnjVk4eQG5/Xyipw69TdyFmNWCg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JvqKnCWQpghkClE9kLMTl3jAzhYt2NXfJ3KJieza1Sk=; b=Rls9mM0IDLY5m9IyNXrS92wANgxqnoLZk/+ei3n6IF4AQbWJbGvaw7hHBbZItPmKWmAFKJtcbEcNib+76Nm5Y4Xvk+J5sUYpjpTsPgOS8X6mUim2/ggJGYhF8v0/sFx2BgfYPia5OK2c5roydJNtYyOiMUtCeJChnTyvbC1XLdkDQ6QBnFFxse/rfsGgQTixs7WpJo/Le3mjyow381OB/yIiWrvgKIVmC1fh1FHv3facD0HiQHR5/1h1QwqpAJjRRU53wzhkTz/Nr3qJQ+wBZpeN1v59rIIek/KrSuLbGi9kJjrktYwCyf3RjyG1dp7otIARAt3I2Gvz4pY+2hCN/g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JvqKnCWQpghkClE9kLMTl3jAzhYt2NXfJ3KJieza1Sk=; b=jl9y8U0QS9w6HnBbK4FAATki+pxbAxGhspEmQpIF1NGM2YbNOnlEkFmiBBLEnjy+ykKeoUZ/8jkIEkkrU1k9sjf2H1W/j55QLtPpTpVdXKdOx6F6Ty7sQb6oJBnA1VYB49N7HdZju5oJdWYNmFrg+yd4Mcro6H3lXzPLJ3TVgzY= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3735.namprd10.prod.outlook.com (2603:10b6:a03:11e::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.26; Wed, 21 Jul 2021 23:05:36 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a%5]) with mapi id 15.20.4331.034; Wed, 21 Jul 2021 23:05:36 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Michal Hocko , Oscar Salvador , Zi Yan , Muchun Song , Naoya Horiguchi , David Rientjes , Andrew Morton , Mike Kravetz Subject: [PATCH 7/8] hugetlb: prepare destroy and prep routines for vmemmap optimized pages Date: Wed, 21 Jul 2021 16:05:10 -0700 Message-Id: <20210721230511.201823-8-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210721230511.201823-1-mike.kravetz@oracle.com> References: <20210721230511.201823-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MWHPR1201CA0021.namprd12.prod.outlook.com (2603:10b6:301:4a::31) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR1201CA0021.namprd12.prod.outlook.com (2603:10b6:301:4a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22 via Frontend Transport; Wed, 21 Jul 2021 23:05:35 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c96bdbdb-4815-488b-ec5f-08d94c9c0c6e X-MS-TrafficTypeDiagnostic: BYAPR10MB3735: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1091; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8sU5BlFyuYBRECT8s60d5X5P857ud51bIAgm2W/Y5yt8TMTXpJFSWyYaFl1+aSh67+n988glZFZc6a7b2a/J2LFMtDfEsiMbGLRQ5h0Tc6NB8p7N8UCs7G0TgoNvSZnAvttv4kykZK0ekSw77zpGTdOjaeCi7TCRpStRtPXrkJcCqf9yBhPq5MpFyb2VDVomQTXEZcr9hiQVKQ4Wi6XUvSAu9naMsafHADLVDz5ITxy+Ax7R5nhoKpCOQzdZWlE+N7utnR86rqz2tWu6TGifrxRYwq45EnktV6TEzXaNKnQQR0OoE76nMAaNWnvBgap32I9U4tyDnAwfzWUR8r4cr9Og5vOnkHTssa+HcX/5pQhpJgqbpbQP9+BDOwiTQ8+9WW1DbTNmnBsBrBSak9RrleC2abqHui/EAjbfu1YDjCImwyOoq/TQkUZ9RfHBzNOiXSCLi5QrK6kDG1Z29hghPejnYmPusiAN3WMGOaDJORKyOsA1wltd8UEwFBA9aytYmGr04UZ1lz+vmi3MAvrf7e75cb9G04bkHmW8DZKuXvcOCzaacuBIUti5gCNc4bB1cZLUkhDDLchap70TdDpXvJCaetnCD8+IvUQaZ8QI8xFbZnYCbBMI+s6QJv1lbpo/X1j81SX7mgTk4j0LYoTusbA+bobfKNLsmXrQ+sHJZbutgRPzU8DVxhD14CehNheWsQOXUIexxqyxbklVt+APYA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(366004)(39860400002)(396003)(136003)(376002)(186003)(66556008)(478600001)(52116002)(86362001)(2616005)(107886003)(8936002)(5660300002)(1076003)(4326008)(6486002)(2906002)(66946007)(66476007)(956004)(6666004)(26005)(316002)(38350700002)(38100700002)(7416002)(36756003)(83380400001)(7696005)(54906003)(8676002)(44832011);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 28tmU5xpXiYvCMJgwQK5H14eXlyvRM53eEDBL+YrJRM5JXDB+9cxJBMzBybwVRgk8I832KFW1qzapryEs5F0Z5zF5l2xmPbyM4ZChlC5xUPH3vSJikqMudfviuXZabah3JQyMXXcJ7ZyQgnFXVqMdXjtYiHL3SFIijvSXfwxuzd4qyl9kgq4kTBAKwVpNZSgfowXA+OJ3Ov8OlyjoPL3WHMDiNcwjp0OUrJmZQP5bTVOY5xrU/PA+HRpVwSmtfN7JSuxIWA+wVl9+3vfnSSR3xfsmVEl6a8tX4mwyYKSQftnqzwJc5nXDOzO8J+hxFJEoUaIqSS2exACvdZDPyoqmJvsi71tLvolgCA87qMTRj7Ap0nUmzFuUOxXLRHqiZhS/ISZ/VJzK6UfNdhtpjeo+KblWGT517sw5izjVviZPddokZQkt+l0XdQCoiYmsGnsIs6l0N4OcKdPMBymtEhOdsZs3H1dqG3hGWpHY9amBZghmPslb9owKfkL9BlPv4dv9uxUOc25/Vn4vDIy28c3k+NDR91O9prncbohWs8aPx8PWBFmHaEi77Z7zkVhYLyGDCmpHfCMf0sWre1uxseshRJEbwPe7wAfPYy1jOCbyvbGiL0IbXcSj7ne3GFHE6n1zTephT/uvYFFIbMSRxOrKGMLijiXBtSve+QlxXZ4efqf+3xa4eZGScwAyVCGyagg8E8+iOfvPf1bs+/+rNMvIkvCWuNWQG3MngRglotlzfyKjGQ8NBxU4P0/GQRJGKZfamG3/Lr9KVwmGNHhJghqrjokJgf5+DjHa7INJZf9lfEyYPrfEqJtxgn0tfO8zjGMsagMz5913Vkr/h8/i8gaOFk5sprWPHttuj9MAvDxBzf3gMJiB5o/Mc2UeBvXb7NDkubcNkQJSDN31iBVfKoWqIRnGXHib9ZlESPM28ieg3dHhM/Hg0Usu6Vq//LIVTs744dojVUKy7uSe27a01+uxrQYpC8FTLX/kuJH2OV8VScTn0G3J+B9VFcTiHD17SuaLzZ/ICvsDHfkKtO7C8dkKb4iKwV/YiKBxDhQK2hXbfFtauncpiIYuvhPw1u/juD14o8xfylRzw4P/Y8uGBP1eA6wsAXBhAnb7xqRiQRElUypyJpvUqiS7J6OMwk71oQ3R8pw9vnZ6FUFSs4+IP75qFsC/OdwXjy5X7IHSUgldgemmfzNRn4jJdqUAyEWkJEMBnQkMhvcWogAw2Hjyx4rrFA13fSjfE63lGs+QpMM1a6nk+57yu+OHYzwgYFhKJPjSiKVNQiQgAyxhpjk7HqmzOjDyo0C3PAwK20NmS1bGtf2xjui7rnaPT+3kCHf2B5H X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: c96bdbdb-4815-488b-ec5f-08d94c9c0c6e X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jul 2021 23:05:36.1525 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: uC3bd4u0O4hLDGSWtROcCiOx0XuO4alBofrtwXWI+CKvOihTtNYG6H2xS1I99UhE6Zz6WX2eyu4nmOI2jxJOuw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3735 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10052 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 malwarescore=0 mlxlogscore=999 suspectscore=0 bulkscore=0 spamscore=0 phishscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107210138 X-Proofpoint-GUID: YPvNgFB6bkiZhgYIUnFiOWh-9rQVDfoZ X-Proofpoint-ORIG-GUID: YPvNgFB6bkiZhgYIUnFiOWh-9rQVDfoZ X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: B0C08102AFFD X-Stat-Signature: khcz3tzcogou6dgobgdfidnxr93i6nw1 Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=qWynysMr; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=qOFKH6Dj; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=jl9y8U0Q; spf=none (imf13.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-HE-Tag: 1626908744-783136 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When optimally demoting huge pages with vmemmap optimizations, the routines which destroy and prep hugetlb pages need to be modified. Currently, these routines expect all vmemmap pages to be present as they will write to all page structs for tail pages. To optimallly handle demotion of huge pages not all vmemmap pages will be present. Only those pages required for the demoted pages will be present. Therefore, the destroy and prep routines must only write to struct pages for which vmammap pages are present. Modify destroy_compound_gigantic_page_for_demote and prep_compound_gigantic_page_for_demote to take vmemmap optimized pages into account. Use the hugetlb specific flag HPageVmemmapOptimized to determine of this special processing is needed. These modifications will be used in subsequent patches where vmemmap optimizations for demote are fully enabled. Also modify the routine free_huge_page_vmemmap to immediately return if the passed page is already optimized. With demotion, prep_new_huge_page can be called for vmemmap optimized pages. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 17 +++++++++++++++-- mm/hugetlb_vmemmap.c | 12 ++---------- mm/hugetlb_vmemmap.h | 10 ++++++++++ 3 files changed, 27 insertions(+), 12 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 9bbfeba3e1ae..4040dd3c6fe3 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1255,12 +1255,18 @@ static void __destroy_compound_gigantic_page(struct page *page, unsigned int order, bool demote) { int i; - int nr_pages = 1 << order; + int nr_pages; struct page *p = page + 1; atomic_set(compound_mapcount_ptr(page), 0); atomic_set(compound_pincount_ptr(page), 0); + if (demote && HPageVmemmapOptimized(page)) { + clear_compound_head(page); + nr_pages = RESERVE_VMEMMAP_SIZE / sizeof(struct page); + } else + nr_pages = 1 << order; + for (i = 1; i < nr_pages; i++, p = mem_map_next(p, page, i)) { p->mapping = NULL; clear_compound_head(p); @@ -1517,6 +1523,7 @@ static void __update_and_free_page(struct hstate *h, struct page *page) return; } #endif + prep_compound_page(page, huge_page_order(h)); __free_pages(page, huge_page_order(h)); } } @@ -1705,9 +1712,14 @@ static bool __prep_compound_gigantic_page(struct page *page, unsigned int order, bool demote) { int i, j; - int nr_pages = 1 << order; + int nr_pages; struct page *p = page + 1; + if (demote && HPageVmemmapOptimized(page)) + nr_pages = RESERVE_VMEMMAP_SIZE / sizeof(struct page); + else + nr_pages = 1 << order; + /* we rely on prep_new_huge_page to set the destructor */ set_compound_order(page, order); __ClearPageReserved(page); @@ -1749,6 +1761,7 @@ static bool __prep_compound_gigantic_page(struct page *page, unsigned int order, } else { VM_BUG_ON_PAGE(page_count(p), p); } + p->mapping = TAIL_MAPPING; set_compound_head(p, page); } atomic_set(compound_mapcount_ptr(page), -1); diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index c82d60398c16..01c2cc959824 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -172,16 +172,6 @@ #include "hugetlb_vmemmap.h" -/* - * There are a lot of struct page structures associated with each HugeTLB page. - * For tail pages, the value of compound_head is the same. So we can reuse first - * page of tail page structures. We map the virtual addresses of the remaining - * pages of tail page structures to the first tail page struct, and then free - * these page frames. Therefore, we need to reserve two pages as vmemmap areas. - */ -#define RESERVE_VMEMMAP_NR 2U -#define RESERVE_VMEMMAP_SIZE (RESERVE_VMEMMAP_NR << PAGE_SHIFT) - bool hugetlb_free_vmemmap_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON); static int __init early_hugetlb_free_vmemmap_param(char *buf) @@ -250,6 +240,8 @@ void free_huge_page_vmemmap(struct hstate *h, struct page *head) if (!free_vmemmap_pages_per_hpage(h)) return; + if (HPageVmemmapOptimized(head)) /* possible for demote */ + return; vmemmap_addr += RESERVE_VMEMMAP_SIZE; vmemmap_end = vmemmap_addr + free_vmemmap_pages_size_per_hpage(h); diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 44382504efc3..36274bf0256c 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -10,6 +10,16 @@ #define _LINUX_HUGETLB_VMEMMAP_H #include +/* + * There are a lot of struct page structures associated with each HugeTLB page. + * For tail pages, the value of compound_head is the same. So we can reuse first + * page of tail page structures. We map the virtual addresses of the remaining + * pages of tail page structures to the first tail page struct, and then free + * these page frames. Therefore, we need to reserve two pages as vmemmap areas. + */ +#define RESERVE_VMEMMAP_NR 2U +#define RESERVE_VMEMMAP_SIZE (RESERVE_VMEMMAP_NR << PAGE_SHIFT) + #ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP int alloc_huge_page_vmemmap(struct hstate *h, struct page *head); void free_huge_page_vmemmap(struct hstate *h, struct page *head); From patchwork Wed Jul 21 23:05:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12392483 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 932D8C6377C for ; Wed, 21 Jul 2021 23:05:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4595061241 for ; Wed, 21 Jul 2021 23:05:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4595061241 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6DE866B007D; Wed, 21 Jul 2021 19:05:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 668246B007E; Wed, 21 Jul 2021 19:05:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3F5FB6B0080; Wed, 21 Jul 2021 19:05:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0046.hostedemail.com [216.40.44.46]) by kanga.kvack.org (Postfix) with ESMTP id 1AC8B6B007D for ; Wed, 21 Jul 2021 19:05:50 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id C27A91E067 for ; Wed, 21 Jul 2021 23:05:49 +0000 (UTC) X-FDA: 78388129218.15.B2A7F4B Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf23.hostedemail.com (Postfix) with ESMTP id 4DF959001B24 for ; Wed, 21 Jul 2021 23:05:49 +0000 (UTC) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16LMvacT031904; Wed, 21 Jul 2021 23:05:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=buzV1+L2Y2ONcY0CLrJ1pQYw1k0vHhcGd6Zvf9Ttln0=; b=mxnXyuhN62yrDfddRWd9RR0nP0x6tUa9Yc2JPiiZajurwL2/47YWcXLNankF7lH4OnqM X8tie+ltDX4xw+lBV7A+PpNHZSFMv7j6ziEWb5wTZe/DP3pYY5ulyqUUhEVwFhBW67iy H/fSm3cWoC0tQzrK89DVtLlMXcJmOMrhGcRRlxeDQ13JhDnj8Mwsc2ooTPcAm0O7u2ex KRTbKJLuaunqcLz9emJDh8MEr6xq2GH3z0VTqnrcwXyfgKgkp0B2gnzcx0/90aujJRjo SPcYCtENvVv0Y1go0tS6aVBtz98Hv5u+xstKPuMBYq28egHs3dPuk2qOeoqkk3JgbH56 5g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=buzV1+L2Y2ONcY0CLrJ1pQYw1k0vHhcGd6Zvf9Ttln0=; b=ozoiLXdhbc1Qfpsm6TDWSv1b2mJoSBTqQNMuKiarmsn/YWJS9jmHu48M6sbKD83BOvbW lGhkgEHd2f+F+aWy8OGRBUw215OiAcFEq7llvNeRO/SjFLZ/0xsvBMmDVhgHoYLHQ1Go A1AQn2CpxQ5T+yGQsbD6ISbLWAveIu4sYuB9Z3RlA0vnEvzcgCa/+KS8VHzbGFspxSjl zyEjNa/2E0XHhVx3Th0iKuxPKVo7TAzC4qmOuqY6i6JHFhOguJewWENHGq1jQLauFb2P MxKYy++mEMyTB5UpmRUMal0srUxTX7PzJR6z8fsitFs9tDHlktSujUJd9nvf9mHivZzV 0g== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 39wwmtuxr1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 21 Jul 2021 23:05:43 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16LN0T2E098889; Wed, 21 Jul 2021 23:05:42 GMT Received: from nam04-dm6-obe.outbound.protection.outlook.com (mail-dm6nam08lp2041.outbound.protection.outlook.com [104.47.73.41]) by userp3020.oracle.com with ESMTP id 39v8yycrpn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 21 Jul 2021 23:05:42 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=am/K3UDp7yai39RL8eW7PO3rj0GxfjJWFttpwm/I2/nh70Qb1F+gt/6m29ZjY+52dXfA7E7y8SA5kJoT0yKTnc28Q0UzR1Ksjca/sz7jQTdcbFG69uM/4IXnVfIJJB76VA53SzWmR50D2HojESKVqzJhNqxUEOg/faszO0XeHC+EgecUZIFWy6HXAhIk1g4cUsQWeYA76fvR/l3SN6LeIzFpAL6LoVUytXm7fMvDO6ji1gyJj5W+OI3Kex4CdconOGgqCaJK+0GiUhOIPMw4hV48WiMXH5h1NROuEwniSUpjrPThd3YgcVkSCik8MVAxSsV0eRTfnwfbpH+KJtiXmg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=buzV1+L2Y2ONcY0CLrJ1pQYw1k0vHhcGd6Zvf9Ttln0=; b=lhDhgJeX2n3NXzxd1ajrus0HIYxcuItHMPLstC1JU/MO80cQgrT/z4ODIAqM/ijARlymbwZ9qFW8BYm1CucL86IZCS8clLGrMH5LvhuSoK5DglurcAzsFhEawq2PHkoTeH891EXo+WyxdYtOPVpUeWCrXu69wTka9hCWUsXFNQEdpNL6ehnlhD/nYFVQoZfq4CmF12XrDRjgnIGXocWzEVmXC4A+lrcUeuzihruCUjP1d0FHF+hc4TU+tGvYC1qTXBk1+ZUmV4V+XciU9Q4YOVlFRSVgC892ZFf2tilnnLAHU50z3/9t3OeiNjx+dnmCpWNZdR94qHwKEI/oJKO3Cg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=buzV1+L2Y2ONcY0CLrJ1pQYw1k0vHhcGd6Zvf9Ttln0=; b=Adc7CCaT/us9PeC6py7QOkk1XPelw2y1baQ9KBT+3uoj4y+s6m/4mZbRn5PJKiWR+tq3p1NZ9MIDACDyOeortw8c4OExT+pVMuh4XsNwowGSwiPk9MZ4y1/rfoZdddpxXe82s6n5EZE5ZRhsqSVRJEW92LNoI4mjYnsRuJ3rxiA= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by BYAPR10MB3735.namprd10.prod.outlook.com (2603:10b6:a03:11e::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.26; Wed, 21 Jul 2021 23:05:38 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::8d2a:558d:ab4a:9c2a%5]) with mapi id 15.20.4331.034; Wed, 21 Jul 2021 23:05:38 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: David Hildenbrand , Michal Hocko , Oscar Salvador , Zi Yan , Muchun Song , Naoya Horiguchi , David Rientjes , Andrew Morton , Mike Kravetz Subject: [PATCH 8/8] hugetlb: Optimized demote vmemmap optimizatized pages Date: Wed, 21 Jul 2021 16:05:11 -0700 Message-Id: <20210721230511.201823-9-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210721230511.201823-1-mike.kravetz@oracle.com> References: <20210721230511.201823-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MWHPR1201CA0021.namprd12.prod.outlook.com (2603:10b6:301:4a::31) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from monkey.oracle.com (50.38.35.18) by MWHPR1201CA0021.namprd12.prod.outlook.com (2603:10b6:301:4a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22 via Frontend Transport; Wed, 21 Jul 2021 23:05:37 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6631d343-6ae6-42ce-80e4-08d94c9c0db3 X-MS-TrafficTypeDiagnostic: BYAPR10MB3735: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4303; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ev/9vW5jbGZZbZnjYqRRZksDR8kUh+gccRkvsNV71AjAclRMka3re6xW3JwAV7DX+KeCyagchIwy2aJtkjTKkOxIA5HCyCps8zkm2NV3FM42PKnoocMhRp2+GP8OcCnEBAmzDugi/uSL0CLo9Nob220D58VO5rj84cTbmK4+cpyC75ZvcADKJzFHUiSY9RJUX/r/YP2RZjcVQMT4wkTNkZg0T1YMk9qTsn4RxXZnn2PnW8sBq8OHzrRvIY6t3a8w/joPPtA89kdi8kenr6AnyvqLDYfgkvKpqqZ0AHHvs318gA874nuhe+mN4nyXGl45+MiEq4a2hoiE/rgME53Vbl2881U5KIlO/jR23I5pFnX79dKek5CBixZDrUQgJxs2bCfH2INScAXzu8i/RVFtd2vyLZ9niaCmz6xAycwxyNeUCgSO2iqT8DJ150VQCKKel7wgpLeCqEvFd8ZmI9OLalke1Wi/6ed0nRTu5HDEbXE8xWzLI2QY+MTWk35mXTN2C4Z1XhGPaqXqhd8Su+AqxrzvG1iBcGtxFQ6AGe4iLDvKc05vPfL8JCirjPoUsL8/fYstT5n7l45mQf230uL8P77asGu9jK1UGs3OeQZV/X/GoB19z97uE8hl1cziGBcpYK9xyLRrud3mvInbFNDnEqv+O8ZFs2rKYRU8lnlQhCUyUY3CR48ySp3hou56CBghoWfMFjxE1fFSJyDBgiqHpA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(366004)(39860400002)(396003)(136003)(376002)(186003)(66556008)(478600001)(52116002)(86362001)(2616005)(107886003)(8936002)(5660300002)(1076003)(4326008)(6486002)(2906002)(66946007)(66476007)(956004)(6666004)(26005)(316002)(38350700002)(38100700002)(7416002)(36756003)(83380400001)(7696005)(54906003)(8676002)(44832011);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: WggjKh0Tyxy+hgAqmEFfNi7WvLPcNhSIswkoQ9W6NS50EDmunJWGkbe3H0Wdo1Pl/6jr9/qnccasKXrIVdpTpk1Jd2aM+XR2Usc8uonhrvVgzIDkImkTG4kdXBFqHR8mKi8OvWU1BGwTsVKPrrQOx0oVkDGUFytVn6Fg/cGjaFbbIyvTlH6e944/KXJHbMNKjOXC8mpdW/Ls3f6Njr+TlWy6nyQj9OwBKoro1d4+3pfMK5P+cqVS6+2B+6SLNGKbgjkeKIUtzYj6QdpZQXdR0OwQ5EhqrXO2kQ29UwQGm9o5sPh2ary8Y2/1eUu0PhwydGi6LgDLYhp0HddJHLhls2HG3PrS0vZ0AXkKP+gOzCtNcUT/tfhCTmGjMfrzh5n/mAySrqPgFoq9L1Y/KA5hlfjtuUCJ/yM6qIxaUnGzt8p3wiJ+C1D8EuqjR7UgH7lI7SQrDF4ncXmtfSU0yM+JgAtuCdpKNbIwzFeENVOwNuegoPHgCXb/+8zB3iY2LiH0pgBSgEQZxPcDgX/mvsFHAPdJpl0vYv9dIMAEDvj0gomEhQ3E/PwLvgHo7WljfDSeZMtbHOTT83xYJtRCwIm7R+WEe93JExGunCiqpujqGGQFNJ1LtB9D6XypubDVCGydBuAqjpk7pOZH0EdcPwP88q3ULVQ7NOllbElhM3Nzi24emZfhOGRt4LGIFuV8N70KJ/FIIRupIsXBy06tA1sLXx8IIUPgYVUTscfzsWO+hQgDSFydClzzINeTH6za4V90eXJDEPz79tMBltqNf4W8s/2UylelpsAFWdbj4e7yCCeoBeGIgNlGF1XX8cbZYo6vlIT1m2MzaL9rT2s7Hkk3urrl6lu6bHUvqXMhZr3XNN1/LbblSFbg2yX/ZV9XQm/MZ+LKQ0w3I5wMZFtGCmBzWUJpKk64V/CudXeSFQLQpEEM+zPbS2PtXe9058zo5DNB0H/K/lZ3UhKLVCPIZ2wEdxisGeZGtUi8NS4YnCR6XytIyFS2M9ExBRPg0Dh0TUgXgXGNuZT59UU7ElWToNhFbxs9oITdQSsZCi+OAJD7QQPBECPDqaLoHOnz5SHcPunYKFlxN9Vdbl9aLTJ3fIF0CA37L7Pw+8ndNSW7O0cQtlBpasS816MaakbAm4Hb1ZWewNrsSgi+ccKE/TQqWMFDGQAIzi5qSJC2gT9dseCoW62vOnQLb3bX9Q+uk+NKEVReaLlnx7X+I1xvIyLjT5lRCz1CdTbcIHsWOTUIip5aigiPGMFkl469NSPI77bq92oU/YYQs8Gs51loxRGs58BDIWWs3jZ/vFfV+KpStJtvwD8TCCQWsIgL6cMzfGLs++6I X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6631d343-6ae6-42ce-80e4-08d94c9c0db3 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jul 2021 23:05:38.2503 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: rict4WaCU1DQ5NYurgeD2xrzvaMpIFR014t0P6zaPRtp9hXGeexPS8zU1L3QspKzhz0XCYSpXQIgSyYhcnWWGQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3735 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10052 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 malwarescore=0 adultscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107210137 X-Proofpoint-ORIG-GUID: FN1_UjeeCocJbyzeKN9SE_DOFfM_c1oG X-Proofpoint-GUID: FN1_UjeeCocJbyzeKN9SE_DOFfM_c1oG Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=mxnXyuhN; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=ozoiLXdh; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=Adc7CCaT; spf=none (imf23.hostedemail.com: domain of mike.kravetz@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 4DF959001B24 X-Stat-Signature: nqc5yrkxfh8eafd7koi7ys16xhai9jgh X-HE-Tag: 1626908749-821942 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Put all the pieces together to optimize dthe process of demoting vmemmap optimized pages. Instead of allocating all vmemmap pages for a page to be demoted, use the demote_huge_page_vmemmap routine which will only allocate/map pages needed for the demoted pages. For vmemmap optimized pages, use the destroy_compound_gigantic_page and prep_compound_gigantic_page routines during demote. These routines can deal with vmemmap optimized pages, and know which page structs are writable. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 4040dd3c6fe3..893406282c53 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3307,13 +3307,14 @@ static int demote_free_huge_page(struct hstate *h, struct page *page) int i, nid = page_to_nid(page); struct hstate *target_hstate; bool cma_page = HPageCma(page); + bool vmemmap_optimized = HPageVmemmapOptimized(page); target_hstate = size_to_hstate(PAGE_SIZE << h->demote_order); remove_hugetlb_page_for_demote(h, page, false); spin_unlock_irq(&hugetlb_lock); - if (alloc_huge_page_vmemmap(h, page)) { + if (demote_huge_page_vmemmap(h, page)) { /* Allocation of vmemmmap failed, we can not demote page */ spin_lock_irq(&hugetlb_lock); set_page_refcounted(page); @@ -3325,16 +3326,36 @@ static int demote_free_huge_page(struct hstate *h, struct page *page) * Use destroy_compound_gigantic_page_for_demote for all huge page * sizes as it will not ref count pages. */ - destroy_compound_gigantic_page_for_demote(page, huge_page_order(h)); + if (vmemmap_optimized) + /* + * If page is vmemmmap optimized, then demote_huge_page_vmemmap + * added vmammap for each smaller page of target order size. + * We must update/destroy all each of these smaller pages. + */ + for (i = 0; i < pages_per_huge_page(h); + i += pages_per_huge_page(target_hstate)) + destroy_compound_gigantic_page_for_demote(page + i, + huge_page_order(target_hstate)); + else + destroy_compound_gigantic_page_for_demote(page, + huge_page_order(h)); for (i = 0; i < pages_per_huge_page(h); i += pages_per_huge_page(target_hstate)) { - if (hstate_is_gigantic(target_hstate)) + /* + * Use gigantic page prep for vmemmap_optimized pages of + * all sizes as it has special vmemmap logic. The generic + * prep routine does not and should not know about hugetlb + * vmemmap optimizations. + */ + if (hstate_is_gigantic(target_hstate) || vmemmap_optimized) prep_compound_gigantic_page_for_demote(page + i, target_hstate->order); else prep_compound_page(page + i, target_hstate->order); set_page_private(page + i, 0); + if (vmemmap_optimized) + SetHPageVmemmapOptimized(page + i); set_page_refcounted(page + i); prep_new_huge_page(target_hstate, page + i, nid); if (cma_page)