From patchwork Mon Sep 18 23:01:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13390556 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8BABCD3421 for ; Mon, 18 Sep 2023 23:03:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0AD276B045E; Mon, 18 Sep 2023 19:03:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 063CF6B045D; Mon, 18 Sep 2023 19:03:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C6BF06B045E; Mon, 18 Sep 2023 19:03:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B38986B045C for ; Mon, 18 Sep 2023 19:03:02 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 7C02040B50 for ; Mon, 18 Sep 2023 23:03:02 +0000 (UTC) X-FDA: 81251245404.12.6A30AA1 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf04.hostedemail.com (Postfix) with ESMTP id 0247040013 for ; Mon, 18 Sep 2023 23:02:57 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=CDAgZj71; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=YvoVIsxS; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf04.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695078178; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/kABIFJps6jWnNmoklKd/IHG2yjhKjdt00SHy+x3QPE=; b=nl2DxTqc19R2UbaD1mOUwCnbdVyd1zul7oZxVK/ol4NajPA1/qufPjms8fRR0rOJbCTthD NQ+AVM1+d77A4I4SDhfMaJxNazuK2D9Thypff/N0avw/0CaWUiarShdaHLtA85nrcyy2hi jLotqHHwV6jVjUqVBcDpOTrr3ZcJFzs= ARC-Authentication-Results: i=2; imf04.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=CDAgZj71; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=YvoVIsxS; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf04.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1695078178; a=rsa-sha256; cv=pass; b=qMsDExx+kmkgRFwF05yyA4q4JOY+xkb1KTuvAvkvsB8xw7J4MrwIVnIB6553zDuFot4/RS YKJzKV02D5hIP9gmqsAKzBYLlryQ5qW6WgOgZtYH6P9OCTepu+J0Zb/TQKAOwgh69ccWv5 kenPSbPL5LNm1tponVJBCFIfD3nQ++Q= Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38IK4uFN015809; Mon, 18 Sep 2023 23:02:13 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=/kABIFJps6jWnNmoklKd/IHG2yjhKjdt00SHy+x3QPE=; b=CDAgZj71ErOiKLHRyFQulnHdliV+PjkNj4nygM9iCkFiTVmSGj94EV/wYwc0PFcn8b6Y M/X/OmXBJKInm/vy1zj2Ay4J/hpoOwmtla+L0c+uWt2D0yEmhclN5ZZqVtW6yl0NczEW 1yMum2JWs1I4x0AK5K0rEUyBYVwf0Ua7rUj+c8/YdjL7Xd4Jsycj+gT4xbrYu+ESNSRS KMFHRJc46QD1zlZJPYWWB0dnpG5z8nt1UeA57dz1Xy4GXE/HSN+NflBRmNKBmFBQHihF 8u4/MV6VXFW/sPVHEAhKgF4LuBzI6jJbk2B9b9jKUhxM8ciHYP2Ms+rLq8Lzq8rMAJmK ig== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t54wukkma-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 18 Sep 2023 23:02:13 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38ILqIbJ012157; Mon, 18 Sep 2023 23:02:12 GMT Received: from nam02-sn1-obe.outbound.protection.outlook.com (mail-sn1nam02lp2041.outbound.protection.outlook.com [104.47.57.41]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3t52t559w9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 18 Sep 2023 23:02:12 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Qi5QzkES+XrJq3qT+8OuUnB3kyFoDqOF+hzEJDUbnIv3/XlVK4c2x4Vbklh0twZW8av0NTAPvv+c/lnHeBMbqdckNYOzKUENBqOsfJLzx5yoTG7P/8fO6QQsUL+sJhVimeO2e2LhXUguBn8vf/9B6i7bN5h71T7k3ethT42SRK3bvRyXtNYggJv3+cgg+GqsjB6dXGXBFyrbpBV+96jvFVJUnHCRp9X+FQYulN4Gb+zayUonicefTqyRgoBJfVoPuOgTEAQ0TlC9Po4p5XlvC2p6a4hpD/Zgym7sMcUlwxx3BwdaSoeOsHLMpF+m7sfA7KHQXOjIPDEhkGEkbutClg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/kABIFJps6jWnNmoklKd/IHG2yjhKjdt00SHy+x3QPE=; b=lLB7F+CdYYMESX4V+7je+Rcw0NMJcLjdQZrH4BLR0kfa2NrMu21RQwieKUOvzTtDMUgXiiWLMvwJi7EJNk7PnXcIdD/F0c4gTSKh5X0/zIu6axfmFNUBlOc+uk4chvP7XzjU94yLLmOTKUvmRpI3wZFJyX1eX17+cYG40dwKYM70HvzQOJG+iuFtOfUIUeGBnsEc/Und8UrAhgE2mUOb4E+fpTPuciGsX163e1YDH5qozrXuSoTmbxfrfnOvy+Oemc7jXl/+YjKYa6eRuCyi2zsJYtocbDtx+WgS6WOmqTf3p7fmLjm67C9vo3rbu5EQlsLuZ36O4r0q6yR/3qsCQQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/kABIFJps6jWnNmoklKd/IHG2yjhKjdt00SHy+x3QPE=; b=YvoVIsxSjSqbaFj+ktzGHYM9TKs/5CEKqyV0x7r7I4yeyZs3KFaK+vLbjt6Np3sFs7gnVj8ND6Hq0InnbChiIH3vXkhk22TeTn1BYliDh72/Yi5zvsQaCjhaOFPTYkI/5wpLrKycDtTOXBdLCwwZSR4opycmUqQuGZzj8wVjmxk= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by DS0PR10MB7204.namprd10.prod.outlook.com (2603:10b6:8:f3::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.24; Mon, 18 Sep 2023 23:02:09 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6792.026; Mon, 18 Sep 2023 23:02:09 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz , James Houghton Subject: [PATCH v4 1/8] hugetlb: optimize update_and_free_pages_bulk to avoid lock cycles Date: Mon, 18 Sep 2023 16:01:53 -0700 Message-ID: <20230918230202.254631-2-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230918230202.254631-1-mike.kravetz@oracle.com> References: <20230918230202.254631-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR04CA0198.namprd04.prod.outlook.com (2603:10b6:303:86::23) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|DS0PR10MB7204:EE_ X-MS-Office365-Filtering-Correlation-Id: ace0880d-45ab-4a19-8c80-08dbb89b494d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: sRMQzPpi0Iic5phrplDDjWnPBUoNuxee5doIL0gR1xS18MoXnnc+Vyvbe+bPLMDgArHLW1+FidYpDP62cDnO08S/6XTruNvX+aSHV30plrO5DOQxPY2ok4opSXQ957xlMyylV0rY30+PInRpkprotih0/2agpliPE68UIEWzLfb6/jmjJWFQQNp8WfNKD7mia22UoppF9iKvHbdy6DjZKNBeIebhF4W7djtWmv9q4h4ODhHUxXbhI7UEgSCNnEIyPeX8C1a3k41QBJBIv9koAqW3D+LrP79dE5+cWSBR29WUtbd8cPqztIML+mQbp905WEtM+8kVHT3dB7mnMEQu4r9s3APBTLBtd+San/q4HJBxVImCJMENajPom2hz3uvOX0nT6+lM9LnESXP1aUhjY3bTnGQoGpgdyuyNI2QRfO+6Ij/XWe9wOUl2zC0HBM/PNMSfZvQEQZ1e2FuabrLiXYE8NMQptL7RsG2hmEnAOD69FSsKDKcIPvdXLOGwa2URrK/x9jFovBpJpLaQRE8Ag8L1kqAJBh2EZHL8y8yQtHnasFOcwAD/yhJkeHOu2Kex X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(39860400002)(376002)(136003)(366004)(396003)(451199024)(1800799009)(186009)(6486002)(5660300002)(44832011)(6506007)(86362001)(6512007)(54906003)(316002)(38100700002)(66946007)(66556008)(41300700001)(66476007)(6666004)(478600001)(8936002)(2616005)(26005)(2906002)(36756003)(8676002)(1076003)(4326008)(83380400001)(7416002)(15650500001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: utPZ8mDndgzqmnKPjbFrv3zI4L6D/qKFVuvBMMWJb/5meaRK3/9H6jHC35Pl5hIjTSlzX2SkpW9aH8EeQXubPqLN2dfyZswAT7pQnS9SBUtXFF+dF8esrqOLnBxfIYYOUgHDKfuKp8iNHFlnzd2R9fphEbYwgNsT7tguy31yAUvm0rf5vKBeaMRmLKiW/DcVbeqN/G6jkxskgZY4N3SKu69guWLbwXX2MDYO0hsp/dYkO3UnyJlAiSaxt6KTCOSEaKbCVJ5bo2vZxSdtl9CO6jWjsnLM9x3D3ohESq+Jtg5nvkIvUBUbY+pUgM/d53NeUGD+BT5DlhAvXLZCSBKsXEhA8QzuQm0DT59S1xRoQkWhuzbgrZlfYmX/qrhEOl1IEh8oqQ4RysHPGSriV8lPqD1lTaH7+bz55ZaUqPKtn14f+opsOLfa53ZkvSZV55Lw/urxosY5esl4dZZ6HznFu0TUbZ9NIZ5hl74T2ImUIvjKAeN4h05vr4DJfIrBIpPf2Fx31LhbnIhoA92coeGf6Vukl0aeZdFRHnnbmH2I+klqlThEy0aE4rIhg+3ITgGNPKq0PHInrU3IWeZKy+YP0pDFAljaXSmQA6yFnHOOOZSXi28TkHasF0V01a2my7Sj80w5QSh53nh+JUMYsgUf9iNC6hDgeh/Vvh35GcRcvALbo4XwWSS2MUmfHKO6i3d1RpdqXEhK6ZmABm0OauHl3us4Bl00sscoLW+Hl4+8JUaT7G8EiM6r7Zos+mRGCXY4BYgeWUnLrHFHnK0/Y3QZsH7C7fmpRTuoVDEzDztzUtDb3djMTQbFXw9bY57E63T5dFojgh98222eD+TRytu2/2RWz8Y8CMxSAS6OmcK3vITD9dIoufFnkPaiPjbxxKbA4EJ+FL5ytomjZiCG1HIySLWHMYevfrAWTMcqRdYIWPmrQaHYLNMZVA5z+6yY6qwStE1LJkmJLH+f7hHxV0/4yWMV01THUtES1erNn+ywzS8hf85NyZT4td8kX6SaG2FkrLkt8aqFuSVoc8D2LP9DKz4fk25JpkGk4UQoYjSfA9E46eUZ8j1oj4ExXEzxRCvb92gDe1cWU5q0kBivcJiq51kb8XYBNw6BnhFy9xIoyoU0VQqgfUl8NA2HTu1tWH96DkkkCOOMHC4SlDfluFVVdGXLZVZp69Ub6fpMAb1Sad/eaE1XX9KBRlPfif0c36DfpqKVVlPigj7Re/jaTRsgha1UKeRGxpgFutyky278WwZkZO9fw8Tsp9d2HS3QLybvKkT+uiScW1X+bQt25z0Bg+u7llbPaHHtVkUfNrBcsJTUO5dJET4UqsDCBoYAyPzHcdHSMCAoa18cB3mRnJHWu87N7P4pBewolWZp0Xs4eIbWKo7jbVi//aiS3m9/lj621hVnSeGArZbFFGdlsqgNIyaDbjgAsjYlLyJLPJdGVlKiUxP56zH3jdSHnFBvqLeu6k8gpfY06Y/JVJHGq5bb5nVUW7BOM6cbFYLsLvPBSQc69AHbOQvFyts69WetIfiuVDvMsG34Mx6PqhI7GxVmMcdF4Ffi1CzyLkwovIaMiguyawq9ff65aF6ghgHrdH2y5xA26FcjfAP0ahxHfJXhhw== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: ChchMA4CjwvCFCXHn7Sd5VLl/8qin3KO0+gH6glUSKit1gUwa81Gyh3S8x2Wm/cBrE3v+gWeMg+OEtYUq3SBiWpBoFz5BulRD6mvBzbCsEyqdvaIgljjJnHQ7T7PO/To8kZ03yO3IWQ//Aqymib8feE8AZ9KrWAeoBB94yo+JkZq94qgdJY7KeN9ASBr2CaOJJs7hrexRor13gyaIynHgzOrgIBaOyELaky3ftyRsOPzvLUTU3Opa57ei5/m/GH8D1a/qT83+vd/CbURt+Bzr8j9CZhM918bXB2KPNHAAQrSNpDkt6Vxq4VhWh4/B2wiDE2AhciBqPp3VI7CoSnkEaRalJRCR7yVHf3FGZbcfWFnwOwHNDNTNoNSXUMxRM1cC0bep9NQyl1rH/9OfHg2gkXiWOmFSbPPGF9qNvP2XpfOzMw1i60kDp986r+TyfkKxA0CnIiYW42frx5LiJ7q1A8t9ppeIYV6g9LBKv0lQU9g7auDv78pLXjH/nlpf6HHiAcTS5IUOaHOkZV2qMYglgOfKAzzpjpG1t08AegarZiDTIMs0mWkI620i71P0Ohr2y6Wuek+3KZdvskkFzd3nJEqt6PNx5cgVCxVcDMQSrMTAEzUbA+kp6ZGUqYewzHVeh+2JHLcsVNmst47A+Rn2Mulmn00RLq+8Kve5K9cYCINVQeBwdfd3MJd6zcVocxslCtD79vxZFK82sZ+Itka2zksxlOL/nZp54qA2aQOQjYvub9/DOP1ia9qlZaRHrQu/yAghD/O6FKnEtJXal2T7EERtMEQk1DytwaYHBJM0yoyyjpB8Ro96W+4hcQMATNNekT7xJMMuy/SpuBX6k6bWvk7kV7E4xJ3M0UQ7K9JACiEwmP0FPN2C6cfyCdOTmCvp3U8hRrraYPBwZJOLjWP/RrzFdBFuW8ESHhfXzFV8Q+UMmOcikOsfdvpMM5kMvixOw2mP1R/JB7XWNwceO2DTJUxW5bBUFjxhBQF0UM0xra5TJK3GU1x8nkOx65kEQRoCnlCdMx4HfZNEincv82EbaoX/yJTx16a9DeLDt/9UgxdGsDvY6fCj/x7wUvyvUlY2k3yJRNfIE+zLaz/thTghL8KWiZ6Jrx2pDuzW0NUfb7W6U6Eon1Tuj16I8Sc8cHB X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: ace0880d-45ab-4a19-8c80-08dbb89b494d X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Sep 2023 23:02:09.3575 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Wnetk6kIeZuYV7bNlfMaG/6YFxrO8oyxjHIo5rlcXzuBWGTGjlcc/w908gW+wHCcHRX1hWN7gDOJFGq2hFtM/g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB7204 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-18_11,2023-09-18_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxscore=0 malwarescore=0 adultscore=0 suspectscore=0 spamscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309180199 X-Proofpoint-GUID: dzPeiEgd2R5qJePP3nKocY-jYQo986Oz X-Proofpoint-ORIG-GUID: dzPeiEgd2R5qJePP3nKocY-jYQo986Oz X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 0247040013 X-Stat-Signature: erm5yksi5rekoz1obd38k3zpehjbas68 X-Rspam-User: X-HE-Tag: 1695078177-635555 X-HE-Meta: U2FsdGVkX1/Jynxbz/fN3U4pJVAD8kK7dfPxfaUQ3Ukvh3E+dgpTG8cG/ynjKdWyDfg4V2EqSHe7yu5ZodEaRJFrLlpKf57+7EqUfC/I58jtdDhNvkJR7PUBohkmKKf/G1Sy2/eelIes4eurzVjLavUJV26GqnAeF7K0gAi/Xy9cWvaa0KTOkd+STkipj6eW0X/3RrZH0XP08mraLFB9vefK9aV7adqS/RUgoJ6FKosoGPW8ssELg3DImAhq9QTTIFtUpQM12VGSJyuhLzsxO+AcyQ8GYpITWqnQGFeiaz+KCcUmwOHUzHMLXjznQvQvN+Pg3+JUQFj9TkaFsIrD6AnHE79hnsDvWzJsvMPJiSFRdcK50ZUQ0fTvrJJcfyyfAUKpAEIkFLeLZpYUYPidoRyUmd6KPYgejzj2QQ33BNeJG7KR8dXcsLio99flgknrG9QG5eoEC+/HtdvXkA7KXuB7cNsHmlWA/9a7674STT0ZaNbg05TNnhaQlZlOtJOm/T0hSSFjjstuDU6cnArAIR1mI38fmR7HZYcQiTe2IjSmCvJcioSEOwvxTGeQF/4YBLZ9ZgDtlaJTrHzpO2njCC+5UzqhtkGgaTQnEdIULVjJqCRQk1Z8Fprqj5yvEee0IIMMp6v1uDdqDpYrkjPbcgH6WEb4Qp3kjGBHVU3WMGJR30zAV8uurP5dKmUq8EDw4CP4ZN1jXuQTwVI0hYnnbBmhgA746VgFpuu1fmscZWQIjyfj4KK9B7zOoWlkXn6BAn/Vm5x4G0pyfzNwJthDtae7Xd0gKtvRMK8NLQE+IjAENkyRmVyHyqzkte+bJatMX7rPJ3OPsH8sEGZSZKf8HqeETW1fvT8ne2Y5wsn3mcHtK/vZVYx43tifFoYV7lmbo75tEDsmdsn/PlPzlEcbsErUAvX+vs5oGqb/OFWxLfmnqQFJj1wdpoUHpNLN/lwQYa7Pm/QA34DOQXFNhaA skZKWtYr 8O+WOHPxuqmSYOhkyP2OOlv/zAJm77oUYoDP19wF7h+UloaqX9i0e2xPTomPdADnZMEH0th0DJ892Qwn96ak/pTkh8drIDFgzFHl5Qobw7cVfFat9BcM6V2odMIwk9bZR8vkaV50rkFaggsLSuizsX5B1CYtBJU19YbA6Gc08yi4S8AgUwNoTdnEpUjQsiP+VyGDcrwFhQY+cGROyUpJUUGrXYkT1gLRfH0H2+Exj65i7bfyj5KHY5odWSpgqfdTg0H6WCYmzpF7gNwuQKejCC23eQInRMJ7sUL9Q0sX81MrLr57uxArAIEa75gkSFFMAko0CHmsB0je1lnHCqEMJoBZbct6Nx4nqiZA4YYRL2Y05k+caR0FvJgET7miZydmsH0NJFngWfch2rP8BBJN09SQPua+MmfH7eXoXyXfUS6HBlqQCRxxLUBg9eSbsvEYiR4QB+HExKk6wLCcGYtfBqrUAavO1i6AjDgStftLYCquMqfk15hTJvbtjAQyEJAlDzPznf2uBWD240yYy3XczlWuzOK/0K1LnCPh3hF3yFizlY2VOo36Om/ALf+2zNKitsdmPWlls9ip8jAPhEG0hneFT0s1fXcaFdTLHTt1Ah8dXn78ChNikDRlR7woRYAj+bv1W X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: update_and_free_pages_bulk is designed to free a list of hugetlb pages back to their associated lower level allocators. This may require allocating vmemmmap pages associated with each hugetlb page. The hugetlb page destructor must be changed before pages are freed to lower level allocators. However, the destructor must be changed under the hugetlb lock. This means there is potentially one lock cycle per page. Minimize the number of lock cycles in update_and_free_pages_bulk by: 1) allocating necessary vmemmap for all hugetlb pages on the list 2) take hugetlb lock and clear destructor for all pages on the list 3) free all pages on list back to low level allocators Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song Acked-by: James Houghton --- mm/hugetlb.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index e2123d1bb4a2..1ca0377c62ab 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1837,7 +1837,46 @@ static void update_and_free_hugetlb_folio(struct hstate *h, struct folio *folio, static void update_and_free_pages_bulk(struct hstate *h, struct list_head *list) { struct folio *folio, *t_folio; + bool clear_dtor = false; + /* + * First allocate required vmemmmap (if necessary) for all folios on + * list. If vmemmap can not be allocated, we can not free folio to + * lower level allocator, so add back as hugetlb surplus page. + * add_hugetlb_folio() removes the page from THIS list. + * Use clear_dtor to note if vmemmap was successfully allocated for + * ANY page on the list. + */ + list_for_each_entry_safe(folio, t_folio, list, lru) { + if (folio_test_hugetlb_vmemmap_optimized(folio)) { + if (hugetlb_vmemmap_restore(h, &folio->page)) { + spin_lock_irq(&hugetlb_lock); + add_hugetlb_folio(h, folio, true); + spin_unlock_irq(&hugetlb_lock); + } else + clear_dtor = true; + } + } + + /* + * If vmemmmap allocation was performed on any folio above, take lock + * to clear destructor of all folios on list. This avoids the need to + * lock/unlock for each individual folio. + * The assumption is vmemmap allocation was performed on all or none + * of the folios on the list. This is true expect in VERY rare cases. + */ + if (clear_dtor) { + spin_lock_irq(&hugetlb_lock); + list_for_each_entry(folio, list, lru) + __clear_hugetlb_destructor(h, folio); + spin_unlock_irq(&hugetlb_lock); + } + + /* + * Free folios back to low level allocators. vmemmap and destructors + * were taken care of above, so update_and_free_hugetlb_folio will + * not need to take hugetlb lock. + */ list_for_each_entry_safe(folio, t_folio, list, lru) { update_and_free_hugetlb_folio(h, folio, false); cond_resched(); From patchwork Mon Sep 18 23:01:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13390561 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85E13CD3423 for ; Mon, 18 Sep 2023 23:03:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2AD9B6B045D; Mon, 18 Sep 2023 19:03:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 199C96B0461; Mon, 18 Sep 2023 19:03:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD6926B0465; Mon, 18 Sep 2023 19:03:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 98C316B045D for ; Mon, 18 Sep 2023 19:03:03 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 5806914049B for ; Mon, 18 Sep 2023 23:03:03 +0000 (UTC) X-FDA: 81251245446.01.FA04847 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf14.hostedemail.com (Postfix) with ESMTP id D58E7100002 for ; Mon, 18 Sep 2023 23:02:59 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=Io9M3ipD; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=eiULz5Vb; spf=pass (imf14.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695078180; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9QgPmeLLE6Ld0BB2W1wJ/WRXNP2Ic8blIblUcqvJ17E=; b=C5sQxSAFhjRl8AHgp0GTSZslBsS3TMfSvCcIWivAwmcHrPfsLfSDZPW4R/W7FL75Ht9NhD kbv3HAYgV3CbNHM2h2kFdSpFALZYogGLUoyFYRUJJQsDr0BGig4hPieDFae1oOx8UIUSLs QLy3DpeCXT+m8TJH+1QWhbNYyH/DHb0= ARC-Authentication-Results: i=2; imf14.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=Io9M3ipD; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=eiULz5Vb; spf=pass (imf14.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1695078180; a=rsa-sha256; cv=pass; b=0hPolquTnIA9lfCrWxaPtczQj0t8HDQ5YAzPyA+isOOVQYa1q7ZIQ4atYKvTP1klffpS4O 4u4UWB73MjYbxjjvU7UYeQljAyueXdfVbpSU/G0kOo3iR065xsJrySRE6KCSkbNIci20nV AhNZFGxFHem7pwxrX9cFIB4mvZ6/tQM= Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38IK4vGV015829; Mon, 18 Sep 2023 23:02:22 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=9QgPmeLLE6Ld0BB2W1wJ/WRXNP2Ic8blIblUcqvJ17E=; b=Io9M3ipDha8sntlnBYcpAiN2zw6oc99hf+oW9rD0GUJOu9FMlebDAMALDtXr2Pl7Klji ejCW6BL+x0IiUj+iKb4LlwQnCq4jw/qnMqTryhjmG5o/GkDkVYFT/KDHbuwyCikO1tGX /JIMxeR55vLn/ZLqlhyBtzQE2gfR529HwUZVAmVhfwaYlk684OcE8+6rWvBh6cb3DC+D TFI1ByXxuVfKUV936CYUru5/DwVcx1fSrdUc4lztDInjjBcC5CeLZJFg17RXcM0odkAX 7T5v8Lx0S3PoqrnH0vFto0HilTmVeU7ONOwz8iHX7YYj2lW3abrJ3ljrQ9PJcAIzqg2I /Q== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t54wukkme-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 18 Sep 2023 23:02:22 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38ILkqos012129; Mon, 18 Sep 2023 23:02:21 GMT Received: from nam02-sn1-obe.outbound.protection.outlook.com (mail-sn1nam02lp2040.outbound.protection.outlook.com [104.47.57.40]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3t52t55a0b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 18 Sep 2023 23:02:20 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SJb3PUskidDPxM61y7DqRnqOptoSvOo9qdXj+NJmLR881DWyYS1RapchRgmbdMTieN5WxDfOWRogQvNYKiDMnnwJ2XMp35+wJx9WZbSf1xk0nnUCblOunShoELe3YGljMbUSFpHanKt0Rf425cyzUToDz0D4SeK+hRBeTMaE9nFD3f2VHouORKt0zFfFSE9yhFWpv2/5D7LRMWBe1UmT9Do3rxCd4tm4s1dETzb749N4B/lafa5X+s4Wj6cITPhp/zgz/JOG77tsaNcHQWWbIcipnfl+/8jT08pqRyIJfDpyYmSxQsFefxvbsRy1u/vAcudb8a11uesnMvt1SoOR0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9QgPmeLLE6Ld0BB2W1wJ/WRXNP2Ic8blIblUcqvJ17E=; b=GkuXan3oZ5wT9Pqtagja8cgn3MLRVS0MEgQKWnPsfbtjZO+Mj8XB5W6yEdkxMTmwv3J7NCA3Id7m+SmiBtUzRrx2VYN34Ul7iaa/OZ8mfF1n8EOqjFmBii4c88GMHe5pPTe4zqEWbV1yMeqeE2AlZu3bvnfz75v+nmSh/R/M6cg348UUtLHVAl4s7BLPW/vl+EC4F/tdFKd+sqOLu89KLHMcDhAJIk2zH5sUdx/RD12YUfCq16Ys47OzXcKSOW5bPelESPbRnUM8t2Ov3/C3pn1G4BF6+vnDppkmb5jVdmLQd6KrQU3zZ+OAqjQmbTuDM3ui4iheT/zzg10AogDxBA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9QgPmeLLE6Ld0BB2W1wJ/WRXNP2Ic8blIblUcqvJ17E=; b=eiULz5VbVq1agH/02/xHbaQ2yUWz2VkRI1uhTAXxEvlFgdqMILk+5I3RSWHtwoYjW7VMYb8no4eHGRT2ClQ0Lxra7fJ00Z3lecG3ztQgkjKkGwfB301Mm4zFC1QQB2SfmTP/AizX9Gs2OjiVpu05VWFXZLs3Yb7VSLu5xxjGsxw= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by DS0PR10MB7204.namprd10.prod.outlook.com (2603:10b6:8:f3::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.24; Mon, 18 Sep 2023 23:02:12 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6792.026; Mon, 18 Sep 2023 23:02:12 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v4 2/8] hugetlb: restructure pool allocations Date: Mon, 18 Sep 2023 16:01:54 -0700 Message-ID: <20230918230202.254631-3-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230918230202.254631-1-mike.kravetz@oracle.com> References: <20230918230202.254631-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR04CA0197.namprd04.prod.outlook.com (2603:10b6:303:86::22) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|DS0PR10MB7204:EE_ X-MS-Office365-Filtering-Correlation-Id: c5e78614-1406-4213-a25f-08dbb89b4adb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8DbDChEAMM24ur0zuDg8Stt/Nr9x1B1HB4JBa19KPg6fDMkPHxwm/49C/La2xaGq5FKjDABTUcmdk5J1VUBryf3pymyDq54nX72q2UlVeOt7O1oitZrqe2iYJRZHP/GsLoGTXuyPGiat/95khtOmOem4/4cgjWDa5y0nARqZ9F2gE85hRyWevPU9XXM9En0TIasYwZREIKsXNLfQJRTm/loaujyRdKIOUZBSc2K+e+25fjeImEzz1JNq/selQKX3gfoEzsm9F0s15OpoR7GiUsyxqRmx8+sad+rTl2psAmlIrqDgB78cEK2sh94bab6UrpnfPtM1OwtG8KVoPO6531WooBiHCpYIhaFvTUPdmEzrceZJek3MIdDSK3xzW6a+tm6UwHdJkxmp2a+X+XhJYapUD01hq/o7vXj+vyia3zEDg6A+PX5EpAmZDP6lTv51HTeo57oz7sBK8IwRlpA+p/gI10cFnJLH2VMwy5RUwHzh7bUbKAbR/j96GdVvn+inL/t2Usmee2G5qqloO7uMXJT2VNJs5DJtgIxIIyy1Bz8Zba/6dzm8/t1nB5/CqSGD X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(39860400002)(376002)(136003)(366004)(396003)(451199024)(1800799009)(186009)(6486002)(5660300002)(44832011)(6506007)(86362001)(6512007)(54906003)(316002)(38100700002)(66946007)(66556008)(41300700001)(66476007)(6666004)(478600001)(8936002)(2616005)(26005)(2906002)(36756003)(30864003)(107886003)(8676002)(1076003)(4326008)(83380400001)(7416002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: IEt/LQVJM1JUAwUW27fzmPg0Wa1zUHDfDrr7oD4d7W006VCZWIy8NcN303um4daO5YhqmQb2WHdLrUL0bSN7Tp+tN+aTBaGn5cgbbLj5fg2FZCduipb2JdHYudi5F7l35ZPpTzxpgYMGArnwlZaxP7rBXrgXFNmibXq4FllNJw/L6GTxnPndDEUJdj5R2h9MK/XSD80LKwNnfdwoxC9EnLwWEjencJt485nf/44kEytkzM+LycEnowlUH1otq+LMmvxmEIUB1xJhmvhR1vvLwUZpFplaeoHfV/YSF1xs98ptYtpFUIVD6rFf19zcjEGMzG1dIv0TauVLBy/brkMbkx+pw9dzPOzZZ6D0wdHMdR6cUzwyuShfrUESGNpVX0j2z2iXYQjSftBxAxujjE2RYXi04scMTbKvODju4Wk5Y8bo6Yh6pKQx5247Y5FiUoW2dshS+sefFQz3ol6jWjIIKGLC8/vk5tD/aIQsZsuwEunlSwyLs6st0k4em0B63z3bDXe9G4N6ebd7MbzSqhur0klbt4surEkzV8rGD2E253OCjl0JX0ODM+OhFgZ+t5cpUIXkP1ww4cPZfYef6PWHGFEvKRasnH9fUlWV77zaJhsPqC5/kP3NShDgmk3KjawazPqZ89cSy61rYJXrHxW72YS2L9FxUOTkryAJW+Pz3gxMYNKQsnCU/OoWJS0H7RC1ih/YI/9+9QG7a/oJNFfXg0NUEhpnwgYoaKoHHybzUiG0Rw40oAU4Kq+hrOpz8BIhMweu3xrEGahVim6Xd9ezpEJZXGDjXAfrK8nKB38CBm7E8uSVNgBkv7sdELblzPprv8wVvNH2SC/Q6S7wAOCU1wZdB4D3qgS8r/Vw7gRw7YxWgRxRfLxiTQBEZ+ed6+eTvYciDuNfmW3QANOfeTFpc66n9OjaAe/cZUGOBj2vVckriIYfn14as1gMWPRcacLf4ud0YyL6AVbBz2I7g6/kP+WFJZKI8XEFyKXnFE7maLgzdTtF2ajNgvUZnneKbq+p3Kc5lrEu8oOn2qL/HbyxhurmZfkiYJxndoIgvxlH3gkB8M8XAOOmKyiXpZ14oxT6XyCzOKG8wvUuSPTs1NED9bbOioiqfzRZZeVzEY40CZ+AsmAzd2EzzzzLwtCgCHpDtweH0s1zPucIuLrpAWDXMhGqCUekGatN938ODupM+fd7xoJBOO91gR6X7uUMnxcAtzq6/Ig2lND8xLT/xrSGpucW+0S/UQVheg125fR5wpZTf0wrHbCn4yur+Q6SVR+T0Qt2lfW46r0fbRHtTHiT7bFJKJZ9JApeL447yCzkPklhHtPh/pt/ONkUcpcjRstf2wxcLl2pjRQkCzIhYNPzkxY7y5c/hGEB7J+KmqJ8JHel61LIZqSe4yRMdFewyxNtVus7920MvsvY1fKKuKaJ7TxKtsGI7sRh/yh4i2k/1mY5KXUsy/axNQcyA26GgckX7ib6lWBdWGW6/TA/ITKqKgoYNKjgdo7rV5X5QpziGOFlVTrsN+3912xZlx+sSX9Uf1BDZPvtltcBwFP6vQ6eWjFk+MNAo69OmrX43ryuICAAfXAioEptG04pR7MKj6/XY7acevmSg7jBYyYKv8RkPw== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: dQvt6K1IY7v3YCyFavqYFh872u4U0KK5F9599no9bjZmhGPbY3wZLtpvvauVxfjJTDrvnDO/cR139YNdh/HfsRKteZJBpZ6ksn1HUPmFWtiviwB2OfCcDxmIzDwxWWs9Ykn4ZskU/JaCTaUv6NnqZDPHWQAgbxbT1QaXybU6fkmLk1AinQXxcx1VxGAKcyp1qgINKg5Wyg8OlrdzTgBYgoIO7SN/vMLccR2VTO2sVTTZ5umWf0LJNTODv7HTtgHSF5kYp4Ux8EysHEf59NFekyvZN5K0wFlfhSh0uyRzP8furENvEtT9TaGper50PGalqmV+nKARHIXCCYCg42ddNhbGj5tdoiKAZK/NQo5EZg25gv3cSqbUSsqfUTwctxaEja06U0TQFPwWYpxtJ2cWpgVIayt40ShGkwhimoR0K49JYMP/Hu9bs4hrmT52pQ3pCbq9KnZOJEYAkHcJeegAcq97oCTq74MADlBr/dSkG3qRJLbOeSQeZMLNCicUwdX+NilVmM83qGfFiCasJJ0Y6ypU0KB9efMJJEtjkhD0rR2p1eGPadOBqArrnTdd3sJ91zv+mjL+L+lhbWVoiZIZwxKLz8gU/OYNW1kmgFUxziaj+tPPQ8yrgTb5FVB/3FJRifdx7+8thE1l5dYbKMcmAu+As8rmUBQDNswsbrUdhRfCuMSIef0BjYxGsgjZyBZ7UyL/iTxSu4L6sG3xsMqTVLbZDi6gnl6yse8qCeoX3QfbDqOx39qi08p88I269Pc04LJnWZrk61x19Htg763Bsli1JO/45ghD3S9hhvbHS65fk0F0+atYxT42b1Ehg/LAxgXXq5RhryncFlsrMBZsY+7eEAaO9HqYUq1wCQvgSNTCw7xVeMo+v1AkiCANTmPXJMvqLnZjfFyg9K1uJWoF4+Gr50ZH1iNBv76b4T83n99HO9kEmd+dDn75Deo8yTC6dF+JNDRSajGPDPQwiKR5bV544DaK3o6tHPKMPEjinhzxvMQMgzLx9Hs0E08mJyLFXcPvzwVfMw1XclBlOh6VTIWyQEYgjxnCVrt1w96e+Ss6sdP9Cz7pq/gMXkV+aQ3h3HYdUW5boBKPPiFAPOWWOocFXSyhQDaKwyE4Eio2uF1JiN4qfk4Af3ByF57JbjIk X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: c5e78614-1406-4213-a25f-08dbb89b4adb X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Sep 2023 23:02:11.9849 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: hQeqc8TXBJJwnEsdZSLYDzaJfmRkwpKiU57+EHxmnhkWjxoZU0C/rKXHS32VdIHMy+K7WNLhWUFhRD99mYuJ0A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB7204 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-18_11,2023-09-18_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxscore=0 malwarescore=0 adultscore=0 suspectscore=0 spamscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309180199 X-Proofpoint-GUID: -HE2HcyGlqKCGdy9spGJRkPu0YBs-Pmh X-Proofpoint-ORIG-GUID: -HE2HcyGlqKCGdy9spGJRkPu0YBs-Pmh X-Rspamd-Queue-Id: D58E7100002 X-Rspam-User: X-Stat-Signature: stwx8hesgzykybguzxtccpuxqud5835c X-Rspamd-Server: rspam01 X-HE-Tag: 1695078179-936678 X-HE-Meta: U2FsdGVkX19SHNjlu9WIGXp/eYB/36K5+WzSj0bWYqH4T9nhk2oiKqxGBEsHFgK/7hAyfChAnAl5fACc/FlKhNMG0WRqz+PavD4IX4GDA6EX1Mr67RdapSXYAx/d/3GwFl3AMNjbchp60R4TiwgWaSJlD/aKSaPOZezFOCwh6HOsf/FwXba/JKIp3zmdORZhHwOOzX8MLLksa8z4z5lo+U9RhautK734I19Sp5lu/w+VFTpewX8LICl/TufPmESeKepU2jSeXGgg42rG5Ah7gU8O136eDQbVTulS7L+A+ou+NA6FfMNevi13lw5j7DP5iOpXuBcBAR8xByv1KBZLf5Vk4SApf4YcTk+UeEaMlyNIvsK7J+zr/iC/GbMcoWNxqjj8B/v59dEfuewtnsWdN+N577nT1u/E5OwIvCUpRts368U6Aviv7Po5xjhW/2nTAcrzwx2E5FcAFwsJe6CPz28PAbl73QgvN3I7AEJgVSDGi38BIs5C8bcikrQOQZr0sP4jp4v+iQtW7naGOod7Om6IVl/zIX24z2KjoPGwCJwDyBL1KEpRXKDkmvbDCZEzPqdt9BUGt3//QfEY3Ttq0EwveOU7PnZIvBFQPT1tIlZXlaJ7tWT8T37PgtIksVWe/3GyWWcHuonSHhW+y7hVWw1TBbj9RWL5h1mK1bK1FV8I+1OqWpPyZpHZPCB2KELNi6eUGh+wdd9Jkw46Itk65+r7BW27eQPJXNzUQjsvKHr4w85ZCod3+/UB0sMdBnmYzMFiIGmytnfaib6VlyJk/qxFOlsB0M02buWAGbfNpPPwxxhbUEvApmERjFbr9C6Tcvv6EK4YHrvMz7leKqzZcujM/rC63gVjlVSkNRYgFpAvVRNbZEKblRSzlzNVp8zjtKfelJbYDsHm9xmFiD35VY0L9lW7DFf+jvKAgwe/qYl5KHBPSz4WrE7k7PkzTA+S9t3KuTLX/1MKRhVQmVq 5wy7cNKc ha6jBZsCkr+iTyNmBfPtCrxd133cogJrgcDJGYX6rMQ7IwuR8nIJjvQo0YCQcf6TMfMkapY2bot8T2i5YPpdsIjBEzEBl/ZWtAN4E4q5C6Q6Ds7Jgrw/sOpfWeH9u5z71cUBoAW8/FmZWvZTfzCGdrYydnSy5SB2lPcJtIQl45gMC3rsUrx8iNWu1eBQZK2tSOFp1Vme2z0nfqx/DLVbP8glm6MIg10Jq1vXcWjHXdEcHlwvTbsX4yxQGFR1Q9y0men1xzmDzinaVt9DLAXJLub41pEQP7cV5Vq2in7ajhiJcBPrASqd54m8cHoN1Z6NgtKUqSREOioTrTtLp3zneAUEDY8eOjRGPmCGUDZ1OOuu8oKdbrneCL9A1BqMqqCJIhsxvG7r0BowlU7dE+rzCUXQ5TeNVhavzLwfrU6ZDxhn4nxH3MIEPOvkdt9/E/RvCLEDq6J7izv95fELhGjwWHuw5lF2tsxdk1r+W9dmnP+jl/EgBFlU/RElibr0jVbkFcxY7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allocation of a hugetlb page for the hugetlb pool is done by the routine alloc_pool_huge_page. This routine will allocate contiguous pages from a low level allocator, prep the pages for usage as a hugetlb page and then add the resulting hugetlb page to the pool. In the 'prep' stage, optional vmemmap optimization is done. For performance reasons we want to perform vmemmap optimization on multiple hugetlb pages at once. To do this, restructure the hugetlb pool allocation code such that vmemmap optimization can be isolated and later batched. The code to allocate hugetlb pages from bootmem was also modified to allow batching. No functional changes, only code restructure. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 183 ++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 144 insertions(+), 39 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 1ca0377c62ab..8624286be273 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1970,16 +1970,21 @@ static void __prep_account_new_huge_page(struct hstate *h, int nid) h->nr_huge_pages_node[nid]++; } -static void __prep_new_hugetlb_folio(struct hstate *h, struct folio *folio) +static void init_new_hugetlb_folio(struct hstate *h, struct folio *folio) { folio_set_hugetlb(folio); - hugetlb_vmemmap_optimize(h, &folio->page); INIT_LIST_HEAD(&folio->lru); hugetlb_set_folio_subpool(folio, NULL); set_hugetlb_cgroup(folio, NULL); set_hugetlb_cgroup_rsvd(folio, NULL); } +static void __prep_new_hugetlb_folio(struct hstate *h, struct folio *folio) +{ + init_new_hugetlb_folio(h, folio); + hugetlb_vmemmap_optimize(h, &folio->page); +} + static void prep_new_hugetlb_folio(struct hstate *h, struct folio *folio, int nid) { __prep_new_hugetlb_folio(h, folio); @@ -2190,16 +2195,9 @@ static struct folio *alloc_buddy_hugetlb_folio(struct hstate *h, return page_folio(page); } -/* - * Common helper to allocate a fresh hugetlb page. All specific allocators - * should use this function to get new hugetlb pages - * - * Note that returned page is 'frozen': ref count of head page and all tail - * pages is zero. - */ -static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, - gfp_t gfp_mask, int nid, nodemask_t *nmask, - nodemask_t *node_alloc_noretry) +static struct folio *__alloc_fresh_hugetlb_folio(struct hstate *h, + gfp_t gfp_mask, int nid, nodemask_t *nmask, + nodemask_t *node_alloc_noretry) { struct folio *folio; bool retry = false; @@ -2212,6 +2210,7 @@ static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, nid, nmask, node_alloc_noretry); if (!folio) return NULL; + if (hstate_is_gigantic(h)) { if (!prep_compound_gigantic_folio(folio, huge_page_order(h))) { /* @@ -2226,32 +2225,84 @@ static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, return NULL; } } - prep_new_hugetlb_folio(h, folio, folio_nid(folio)); return folio; } +static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h, + gfp_t gfp_mask, int nid, nodemask_t *nmask, + nodemask_t *node_alloc_noretry) +{ + struct folio *folio; + + folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, + node_alloc_noretry); + if (folio) + init_new_hugetlb_folio(h, folio); + return folio; +} + /* - * Allocates a fresh page to the hugetlb allocator pool in the node interleaved - * manner. + * Common helper to allocate a fresh hugetlb page. All specific allocators + * should use this function to get new hugetlb pages + * + * Note that returned page is 'frozen': ref count of head page and all tail + * pages is zero. */ -static int alloc_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, - nodemask_t *node_alloc_noretry) +static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, + gfp_t gfp_mask, int nid, nodemask_t *nmask, + nodemask_t *node_alloc_noretry) { struct folio *folio; - int nr_nodes, node; + + folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, + node_alloc_noretry); + if (!folio) + return NULL; + + prep_new_hugetlb_folio(h, folio, folio_nid(folio)); + return folio; +} + +static void prep_and_add_allocated_folios(struct hstate *h, + struct list_head *folio_list) +{ + struct folio *folio, *tmp_f; + + /* + * Add all new pool pages to free lists in one lock cycle + */ + spin_lock_irq(&hugetlb_lock); + list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { + __prep_account_new_huge_page(h, folio_nid(folio)); + enqueue_hugetlb_folio(h, folio); + } + spin_unlock_irq(&hugetlb_lock); + + INIT_LIST_HEAD(folio_list); +} + +/* + * Allocates a fresh hugetlb page in a node interleaved manner. The page + * will later be added to the appropriate hugetlb pool. + */ +static struct folio *alloc_pool_huge_folio(struct hstate *h, + nodemask_t *nodes_allowed, + nodemask_t *node_alloc_noretry) +{ gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE; + int nr_nodes, node; for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) { - folio = alloc_fresh_hugetlb_folio(h, gfp_mask, node, + struct folio *folio; + + folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, node, nodes_allowed, node_alloc_noretry); - if (folio) { - free_huge_folio(folio); /* free it into the hugepage allocator */ - return 1; - } + if (folio) + return folio; } - return 0; + return NULL; } /* @@ -3264,25 +3315,35 @@ static void __init hugetlb_folio_init_vmemmap(struct folio *folio, */ static void __init gather_bootmem_prealloc(void) { + LIST_HEAD(folio_list); struct huge_bootmem_page *m; + struct hstate *h, *prev_h = NULL; list_for_each_entry(m, &huge_boot_pages, list) { struct page *page = virt_to_page(m); struct folio *folio = (void *)page; - struct hstate *h = m->hstate; + + h = m->hstate; + /* + * It is possible to have multiple huge page sizes (hstates) + * in this list. If so, process each size separately. + */ + if (h != prev_h && prev_h != NULL) + prep_and_add_allocated_folios(prev_h, &folio_list); + prev_h = h; VM_BUG_ON(!hstate_is_gigantic(h)); WARN_ON(folio_ref_count(folio) != 1); hugetlb_folio_init_vmemmap(folio, h, HUGETLB_VMEMMAP_RESERVE_PAGES); - prep_new_hugetlb_folio(h, folio, folio_nid(folio)); + __prep_new_hugetlb_folio(h, folio); /* If HVO fails, initialize all tail struct pages */ if (!HPageVmemmapOptimized(&folio->page)) hugetlb_folio_init_tail_vmemmap(folio, HUGETLB_VMEMMAP_RESERVE_PAGES, pages_per_huge_page(h)); - free_huge_folio(folio); /* add to the hugepage allocator */ + list_add(&folio->lru, &folio_list); /* * We need to restore the 'stolen' pages to totalram_pages @@ -3292,6 +3353,8 @@ static void __init gather_bootmem_prealloc(void) adjust_managed_page_count(page, pages_per_huge_page(h)); cond_resched(); } + + prep_and_add_allocated_folios(h, &folio_list); } static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) @@ -3325,9 +3388,22 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) h->max_huge_pages_node[nid] = i; } +/* + * NOTE: this routine is called in different contexts for gigantic and + * non-gigantic pages. + * - For gigantic pages, this is called early in the boot process and + * pages are allocated from memblock allocated or something similar. + * Gigantic pages are actually added to pools later with the routine + * gather_bootmem_prealloc. + * - For non-gigantic pages, this is called later in the boot process after + * all of mm is up and functional. Pages are allocated from buddy and + * then added to hugetlb pools. + */ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) { unsigned long i; + struct folio *folio; + LIST_HEAD(folio_list); nodemask_t *node_alloc_noretry; bool node_specific_alloc = false; @@ -3369,14 +3445,25 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) for (i = 0; i < h->max_huge_pages; ++i) { if (hstate_is_gigantic(h)) { + /* + * gigantic pages not added to list as they are not + * added to pools now. + */ if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE)) break; - } else if (!alloc_pool_huge_page(h, - &node_states[N_MEMORY], - node_alloc_noretry)) - break; + } else { + folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY], + node_alloc_noretry); + if (!folio) + break; + list_add(&folio->lru, &folio_list); + } cond_resched(); } + + /* list will be empty if hstate_is_gigantic */ + prep_and_add_allocated_folios(h, &folio_list); + if (i < h->max_huge_pages) { char buf[32]; @@ -3510,7 +3597,9 @@ static int adjust_pool_surplus(struct hstate *h, nodemask_t *nodes_allowed, static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, nodemask_t *nodes_allowed) { - unsigned long min_count, ret; + unsigned long min_count; + unsigned long allocated; + struct folio *folio; LIST_HEAD(page_list); NODEMASK_ALLOC(nodemask_t, node_alloc_noretry, GFP_KERNEL); @@ -3587,7 +3676,8 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, break; } - while (count > persistent_huge_pages(h)) { + allocated = 0; + while (count > (persistent_huge_pages(h) + allocated)) { /* * If this allocation races such that we no longer need the * page, free_huge_folio will handle it by freeing the page @@ -3598,15 +3688,32 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, /* yield cpu to avoid soft lockup */ cond_resched(); - ret = alloc_pool_huge_page(h, nodes_allowed, + folio = alloc_pool_huge_folio(h, nodes_allowed, node_alloc_noretry); - spin_lock_irq(&hugetlb_lock); - if (!ret) + if (!folio) { + prep_and_add_allocated_folios(h, &page_list); + spin_lock_irq(&hugetlb_lock); goto out; + } + + list_add(&folio->lru, &page_list); + allocated++; /* Bail for signals. Probably ctrl-c from user */ - if (signal_pending(current)) + if (signal_pending(current)) { + prep_and_add_allocated_folios(h, &page_list); + spin_lock_irq(&hugetlb_lock); goto out; + } + + spin_lock_irq(&hugetlb_lock); + } + + /* Add allocated pages to the pool */ + if (!list_empty(&page_list)) { + spin_unlock_irq(&hugetlb_lock); + prep_and_add_allocated_folios(h, &page_list); + spin_lock_irq(&hugetlb_lock); } /* @@ -3632,8 +3739,6 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, * Collect pages to be removed on list without dropping lock */ while (min_count < persistent_huge_pages(h)) { - struct folio *folio; - folio = remove_pool_hugetlb_folio(h, nodes_allowed, 0); if (!folio) break; From patchwork Mon Sep 18 23:01:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13390562 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29740CD3424 for ; Mon, 18 Sep 2023 23:03:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 875FB6B0461; Mon, 18 Sep 2023 19:03:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7FF796B0462; Mon, 18 Sep 2023 19:03:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B6856B0464; Mon, 18 Sep 2023 19:03:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 318D76B0462 for ; Mon, 18 Sep 2023 19:03:04 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id F172A140B36 for ; Mon, 18 Sep 2023 23:03:03 +0000 (UTC) X-FDA: 81251245446.20.839B0F4 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf08.hostedemail.com (Postfix) with ESMTP id AA618160002 for ; Mon, 18 Sep 2023 23:03:00 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b="H/2MiC8t"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=ro9iDMxE; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf08.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695078180; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aGyfKWmz6L5+jK4Oa5XR9qAiw4qVE4/FVGtZT6ufHEc=; b=w4FOBeuSQtAJgrwyWIOCHqkQ1TsHfW4HtPACboGDfuyfGyo68zsCA+SSiEnPcRZX3ouB4Y xG0M1Vs1r76q1f0DCaKe+t+8QQGXmBAJdvSAyNqVFIz++EyhjoDH2rQA6QgUnhE9jFX/eh 9bcEbjDaFayR7m077gEAPtFbw7kpt8w= ARC-Authentication-Results: i=2; imf08.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b="H/2MiC8t"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=ro9iDMxE; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf08.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1695078180; a=rsa-sha256; cv=pass; b=1iXxj7DS6VPWZWiW4Em8HpsNe6Vi6hLEeR6dE4a8XRdu03sof7Qc0nS4ptwJceSYG6VLwx yASiIwS7kkuf1C/8uQm4ohCmh+JpmbpT4nFIfBVQZAX5Ppg/os84dPUrEQnOOsT7nni6e5 1CT8051BrNoGp3rHVnXTm8nUcmGPI7k= Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38IK3xs7031273; Mon, 18 Sep 2023 23:02:20 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=aGyfKWmz6L5+jK4Oa5XR9qAiw4qVE4/FVGtZT6ufHEc=; b=H/2MiC8t2yAXPqKgu5r5jjRUgDy/WJLA0V1y0TzCx++s8Ef1crtwNY2PzKpedYr8CXH0 BeACQkZM5lle23qp7JLwBTvu2osgdAu/uQ5M0EOh1OuA3FDSxD7aZCfhgezpXnlaU0uY /7tC/8v7sM829+wS1KfJLtPzzbRzPWpDyIK/xbq3lvmxVBKVLCkiGCuvOVx+S66x3i4q N5EqSBC2h5+jehEZdhRGRHq/jz0l1sB14vzSF9lhWZJDHPbtjvZNOkjK6UPEZxu4B+Rk tG2ajwER+aD5zWPLdV4TfpdbwV8Ja14yK3scHKxRaqydcX5/VUmMX5CHo3nV4MKpDStL 8w== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t5352upry-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 18 Sep 2023 23:02:20 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38ILV2Hj002042; Mon, 18 Sep 2023 23:02:19 GMT Received: from nam02-sn1-obe.outbound.protection.outlook.com (mail-sn1nam02lp2040.outbound.protection.outlook.com [104.47.57.40]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3t52t4nvs2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 18 Sep 2023 23:02:19 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=l2eVagItC1t6taDapo+wBiH/fJOOgFc9ijxF4DbZ0F0nrtjDiqY3iw4YWWuuxYdtODW6izMQ4Lm43ooCcDOHzMq6dZq0qr9F7OTqOJb8S//vA5zZZTdcn93EjK1oka4mgcW1ik9W3Q/aES4pfO5kKZH8hjY3hMgyQ6qClh7j/b/6oOHMiy7ebza1416zKh/w4QiZMwVmddrVJQJG1Ntc8viX7/ocz2pfAOeb2Pj0LLUSIaAOnh851tYutRV1R+95rfoJJJZ4HU+2BQLJfx+TJ6nSiUsQpKLF6cEfViktAnuFZptRYYI4IC+Y9ji/DzQ3IdsYgex2qBZyT8EgyY3GFQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=aGyfKWmz6L5+jK4Oa5XR9qAiw4qVE4/FVGtZT6ufHEc=; b=fo7AffVd0B69gwRti465K2Ls7BQVoEL0eybIciERbDHQ5cDMRa3YhuwXwFXZewyXFbOZ+j0Vx0zxTj0JKEq+Xv8NTNt1VHl7eCa217JuxyIm96dn+nivHbyNbXXogzZl5KmNsa3EXDtfIYwKkdIubOUxFYns3A3WivfB2+zj9Gi+6ZrlQyRKFxMkDWdzEVLoHzQw7gueC8MDp/6k/sU59lyjK54SFJbAcVKOuizeODrDuZhaAvQqm7cTChbY68Jn1KxpVRG0SxbLiZ4THUYguQ/xpRuWfx6ULxdqZ3nadGypIHUQX3g4+Nvm/K6MD9+pIbafi/J722a2+f6xfgPycQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=aGyfKWmz6L5+jK4Oa5XR9qAiw4qVE4/FVGtZT6ufHEc=; b=ro9iDMxEqtmzqV2tG+f3REUu2cHIPmQjdJYVQFIkBMfXH4DSZK5mKIKp2Nkbetui4Huu6N3xmg4wKyydkR4GOjjS4K1XogWIomLR0lFdBYtYl+QziwvY88l69gtqFdaOeHOCVlpugptCj+HEEy8/OQvtzNzLZSB1nOkKzYdZsXo= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by DS0PR10MB7204.namprd10.prod.outlook.com (2603:10b6:8:f3::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.24; Mon, 18 Sep 2023 23:02:14 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6792.026; Mon, 18 Sep 2023 23:02:14 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v4 3/8] hugetlb: perform vmemmap optimization on a list of pages Date: Mon, 18 Sep 2023 16:01:55 -0700 Message-ID: <20230918230202.254631-4-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230918230202.254631-1-mike.kravetz@oracle.com> References: <20230918230202.254631-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW3PR05CA0018.namprd05.prod.outlook.com (2603:10b6:303:2b::23) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|DS0PR10MB7204:EE_ X-MS-Office365-Filtering-Correlation-Id: cf4c3a7d-1c99-46e9-dc3c-08dbb89b4c8f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: FuKXB8Zs31XdTsAKVrof9dhZl/6e4NvWEaAK/badTca5Q6q0BYnBtq+QdLlQYaunDMOnIl8W432ctcm7FDVp8LanDbO3BaMvXuh90KNpCMtx8DAmCSS8csI09THh5Bs7uEltO38uMLoVFBeRePhAa4pJN6nyP3+KIMQ+LlxgoOkhUR8TFzxnrmyXQhtxMeywhEIOO+/8lLHEl3au21xgUZ7o9fz4gq33t3HNFnVoLFdrr/mLY2rM/r6Nfy+rBteiftiCIuMBuZCDKGwYc7nfYfd8KdEjEfR4ZPYdLQ0fepJN9cqg/EPIV+eRtIt4qEhG3GDjTyRJ4vm5sNQkgpfuEOcuMWgm4o/EyhWQ/xzzux6o7Ekj1nXCHjetVIf8JD4p0GgydHpVin/zX4hqjR/7gLyhsKm1v1GcIGu+TDyALuN8b8JwBVL77jea/7WTyIE0rgU/gkXiCZxt+NA9tPyGN6UmIuyIHptRB23xieoCKniRPObLR/FibXk3xnCK4lGujNsP+zvpSJ+qTtt7qXCyluYUiFRKkwTU8zFmdKqLZsolyvAUkYWKqqI9WB3C9w2X X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(39860400002)(376002)(136003)(366004)(396003)(451199024)(1800799009)(186009)(6486002)(5660300002)(44832011)(6506007)(86362001)(6512007)(54906003)(316002)(38100700002)(66946007)(66556008)(41300700001)(66476007)(6666004)(478600001)(8936002)(2616005)(26005)(2906002)(36756003)(107886003)(8676002)(1076003)(4326008)(83380400001)(7416002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: eSKkpDq7NJZ4bvnqsFWMSIO7S4oL2wMHn6qIyxcpJfBHiIj0Z1991Yffpy/R6bLOOi109vK+5oWUnHcRS9QqWQQ77G4lxdYtdfCz6PWCkX580fVQ7LiVrQoq7Fpo2Ke16SdrV74DCoEvAUDVYemzpXOaG7yXqMGldFSpEMvRgWMHzABpG5bOief45E/GY9fa/+83Eoa3V3eEHCExq0Q5biS90rm301dCW0S6W3zTTJ8FXv/Q0vZWKSSqPD/btmRRd3gdYxgBIj0+hhWH/Bq2WKV/VAR2M6C+Ky3B3NH9XUVcJiXY3HY5x1nC4EH+r77wZTi4SWQp4F1MADXiJbe2K4gZKsrxrg0Pp50yq1ytYz3O0R82Gspn0usK9p36A0j1WSYPVGmWw2DSyfwzd3GjXM43R7wFlfB9wULF0aNNk1bk+d0XehlFXryBQPPCJSE5qe8fzyH67mUybpMVvXWoo3IT1IsWMB04GJqQ4S2tG2Em2sGK3gr27jG1zsuHz50q/7uLY7Zn/4W2UJlviCUR2XQucYJzLKWhIHIyUC3Kh1uSRW+rQxQKRRQ5BOkoNlbko8YgEQsSKXRGDQvfHoYnSRCkH1ZHr+d16/pL3sQv2eSArXRnKGsGxtKjZKgVg3CP4O56C7OjQy1jA1tjbewUbaEYkzvKMZJYk2Zbu+tkRYz2q5U6Wuq3L0x809+D3uAqJ95eyt7BRWTXB63DGRAcTQdODoQ7KG0PMiwrgHRPwV+QqLGUmMp7qOVZ56J55bKwaTmdvBth2YYzNydRlXDUuMKjbAhyEcNBu5Gq8Mv1FMGKbZxkuNxJQ5zw+pZLh3bCRoM+nsvkOaI2U0zu0R950RW/yTLurQ4nlub6E/0cJGaefOBf6YSr8momZ1lo7VfWfjvqtyKeCmRiMXh1pLgsiTrbcOAjhHbvKfDs83DKkvLrKoTSwX3M1DBQoSybs7bpHvWS7iMKLXNo+4rmlDfAsHkty+OQgdXY+Rf69EhkVBdFgmnsNeOcnToi8YzFl9kf3wftMlSf9pyn7gyEoZLk70Wi6FAs0rd+CJ40CWZTIH/JLZ7wsalDu4MBEE5INlmCpH1EMG4Zc9gKjvHDot1sN62bC45J9fH8MkLroWCNdNQV44BDXtQDsJHsxl+RpArssaDROO1Ai2JImRcGx7GYG4be2EWLkvSeVsGWORD6bY3luz8V/5t84UMVwfwH0xCMuwB0WH4D9N2NIqzNAghq6AHU4AAFnjiPoDd8BXU0Dg4sgnCeTTCeSaLYFUOQLAhgB/5fep/MWXkYnEglBD9+5KetpVtSsEP4F7e3i3BQL85M0we4dlSBSUqZYQjp6JFCvqoI9vy3Owy8hLGm5EC2Ks6z0qk9BGQdmBcuAuBz3mAiNJ8c6IKA7kOHV3t8yPu4SmIYYpE/kQwur5DVzFRQDhrQIkOTlYiUPNIArB5u4ZZ1sEQdkc7jGu1TfK0rxD6iO4CGslD/sO7Yh8hEP9/19ziiStSAlXFNe72yNcSunPq/etfjmhlw0xDb8V7xnxmGiwuc0LPMj4lRIXIn8Cu0efoyPFwCB0+SPNMgtbawwDUs2iJGHKuMWOPJMvGL+tsM X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: /OXf8B4pm+LE1lDf+QICMnVvhZIiKBqdO4AHwo52mys4yqpEOSKRtX5Hvj0pGk/pbeQqzISpIMpwk/c+o9IpEo5fUVAdqIx/o+frDchsRFIoBehtpepqNcGuM9RbXJraKU3FYdxjO9DGSxZkMViNLezBDmcEKcPz5i7nr2Q8t7AlMWw5T4OeHJrQDgkgKxLSfXFAtD8VRYmRALZ5i6Hl1/xxG5pnyMmP20rhKdp2VTviuqU0sPKg3OfI8USM83Sc8IM7TdCXnuARwaZg591AnsC+Pwaoay2mfHNnILVo4k1YYyj7gWG2sz/mnTeUygSVvmjLbbKg5VgfUARHoiCNFjMcHrBio7L1Rmyhix8felomBY5FB1jg7Dkfw/zyPA2yOTIZGmY5i0qc3+TRxIeY+KlCYFjDYKM6+Uuz5ANtLDDqTrWMGH/hj6uLthpMagXDIum/oyL3ylamzgM+drOC3ilsaOVBRZVELsrdV+gPaW+xiYR8QDzWKMCxnRedWKQfvYY3CBbK8Xm7BS3IY1VIoARvIoDskyPmwvibShU46cUZ/mZ6Tt6EBPD5oisk2OrjIc0iLwp17t8RQi4j5tAvSA7BR7gqOSshbjxqfZEzk9IKFTn7blHsJ0IeaW2EjGwipCUPcP3EM2Q2R+MSVhWT/d47iJe7YuF5vEkJ6SkdxJYN9j6U/ZOKRQfLI8luZQgCCm9tPLZXB0/8JkrpHlXQqc9R6BqXGJ+3wA6KhMi6bnhLbATRMQo+PY5FPzHIhcSHpNAY5EKzQEyAYRLZoqckPDXopStZ3eL8e9L0I+wUgrdYjqrMiX5JIWQw65FhYPfUpMkIDnbb/0dtsX10OJTWTRqhB/hhwF0Y0jOextURHdPF8faxh/qXvsM/H7Aw/IhXHuoxvZm4i8qLfZr89IBc/bbHK4q7W9EtMNbovI9kwjLWRu9HUwK218jDabjDa18de+MlNGs5mAu6syFjgWk86G4xTFLWNUKtFqKkbfwLdD8eUesozOlarxwQFKTklfIL/PlWsmUD1YKWQQHSdvU2RBLd0Trgs3zMY/ySLuye35LqQ0NtOwpri6bZfweO+v6exakdXSuirQi8VnjpmdWORlnW+cAnRlO6rCYGQrzBI6vNqYa4YbyEa8gjnu7w+5a1 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: cf4c3a7d-1c99-46e9-dc3c-08dbb89b4c8f X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Sep 2023 23:02:14.8375 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: AqMO9WmiE9jboqjbZET893o76n1IfVpoe0NRkFgmbfNYt+APNYll5eSe+7dSPS5yfSUyw/KkSNzYIfrn5ygNUA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB7204 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-18_11,2023-09-18_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 adultscore=0 bulkscore=0 suspectscore=0 phishscore=0 malwarescore=0 mlxscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309180199 X-Proofpoint-ORIG-GUID: I1HVvo5hbKKAlkoc8CyrG-Rl11FwEtQr X-Proofpoint-GUID: I1HVvo5hbKKAlkoc8CyrG-Rl11FwEtQr X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: AA618160002 X-Stat-Signature: 7g6m5p4ki44b4qb31jm3bxpmbq8fe96d X-HE-Tag: 1695078180-262058 X-HE-Meta: U2FsdGVkX190ZEdB8F1Kyl3ZqTQRitL7bf1EcqUqhg+OkrMQjnvutbRfefoeDpJEUU9IPbTH16vM6UNIVtw+58Z7s7/nDHxdutsdjgvruwPBRta5B+6G1fc6pj+iIOEge1v9XPyO9X7fUEbCHiwnJgBOM0AxaVNX9THoUCIX8tAlhWiEJj2KAwMCwOE4mgWXWTUPE+/jAZF2p8fXWERyhEXg0LRHNzXxOuei3svqAP5ux457Db/oIFxoWCq5gpz583wcshtX+dOrlCkGZ54huKIr1AcCTp/CM3P+ae0IZzWIUDNDDbv3y3QsvA0/IjCRkLtEfikrizkro6/Iz82OTLtCMjSY0iGuBs6I1HiqKXQCqZAk50f7rax4n/7S7MEtSzd/NxbwmJ8RKC0MZZYlKAiT8UOSKF1KNDgWlM1daj5TYXJDnA4+DroDF2z7S+W+tsYfgBfnIzGe9e0TznmuocbHDkGqf/qmAPnXOmagPBVdE84fT5N963XSdmUONVo5EXVtUBO9YcVRQ9QmGLNSYBcaBDRnL0q7kWd/ubDdutiQIbknTvl3kMtrfURGQyPQ0jqSPSQ/XTQinL/EccakWktENvi0mVFvnUO5p6QCGYftNML3RR26BaBV0pha7Kh4qKetchXgHtlHFGkbTKTmOXanTf5cGNDSA1cqmABeqJdt4kv74HCHSaNQQ8noZb2ZS5Q0s8bxbjzD8IbNxWAPFYrtbLaLKNsGaZ2YOzggU5xOAdk8YlQaPFghBOlMQVkhaMQlX3u9WLY29arAH5YqHZzPfab/MFHNA7xuFq1y9Ylegd6O2JGCyQoAe5rCnjgdEdylhjqaFRz6SiNaTe2IDorUluG9T7vmOm87pYxVD8RDeOyD5QU+mamwPJpJMlklkfjAWGCpxdDlDkuWgsuyvaGWtOYygTSFYyq/61FuhPadlO9HqY+2p3i/FY4pMf9KkP76xa6tw9fhm1OWu83 pfJlpyqu nmgpe4mRiw0Oi7Ln7yzubgyij6TQ4xxFnTBPudtptH2t2rcS2VkW4eo8E68702261KstR8nf3qJT3/5ISbGBz2Akz6xqZKa8Pg1IuyPEfF++KY9yMmW+iYKG8OOuO2G8cKDhRrS0PKCPVqur7SCA0veyCmaR9rPq5iF95qBnONPWfqgZ53wTHE526tcXb2nX4/swM6uLEZGWW1gZVQvwSQgjB+MESUn4ikuYlH0HdirUrIhg2cuiM4gYdbDYjzryrBwzU2UnviCyADBGYkWyJueup6sNQJRdCvO5Ke1Xp40QhSTl9aWnM7J1q8ksBH2XxYCdl2XhBDF3MNPFOuU6Bxj1nsLI7Au+8I6e/q6QdD6pwrcYRpknTr1nkr6hhIN3T4wXkt3BIu8PKgN66DzCleEY5abyQCNektZkEBWMtc+MQNmCE3sE1hJ5UVMD5omSmgJQPhT26E0r4D9DieF3pW7kGtAP2BtCyyfP87Io7Q3DHuBENNpv9Uy9jHYG0mxAIIYzg X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When adding hugetlb pages to the pool, we first create a list of the allocated pages before adding to the pool. Pass this list of pages to a new routine hugetlb_vmemmap_optimize_folios() for vmemmap optimization. Due to significant differences in vmemmmap initialization for bootmem allocated hugetlb pages, a new routine prep_and_add_bootmem_folios is created. We also modify the routine vmemmap_should_optimize() to check for pages that are already optimized. There are code paths that might request vmemmap optimization twice and we want to make sure this is not attempted. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 50 +++++++++++++++++++++++++++++++++++++------- mm/hugetlb_vmemmap.c | 11 ++++++++++ mm/hugetlb_vmemmap.h | 5 +++++ 3 files changed, 58 insertions(+), 8 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 8624286be273..d6f3db3c1313 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2269,6 +2269,11 @@ static void prep_and_add_allocated_folios(struct hstate *h, { struct folio *folio, *tmp_f; + /* + * Send list for bulk vmemmap optimization processing + */ + hugetlb_vmemmap_optimize_folios(h, folio_list); + /* * Add all new pool pages to free lists in one lock cycle */ @@ -3309,6 +3314,40 @@ static void __init hugetlb_folio_init_vmemmap(struct folio *folio, prep_compound_head((struct page *)folio, huge_page_order(h)); } +static void __init prep_and_add_bootmem_folios(struct hstate *h, + struct list_head *folio_list) +{ + struct folio *folio, *tmp_f; + + /* + * Send list for bulk vmemmap optimization processing + */ + hugetlb_vmemmap_optimize_folios(h, folio_list); + + /* + * Add all new pool pages to free lists in one lock cycle + */ + spin_lock_irq(&hugetlb_lock); + list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { + if (!folio_test_hugetlb_vmemmap_optimized(folio)) { + /* + * If HVO fails, initialize all tail struct pages + * We do not worry about potential long lock hold + * time as this is early in boot and there should + * be no contention. + */ + hugetlb_folio_init_tail_vmemmap(folio, + HUGETLB_VMEMMAP_RESERVE_PAGES, + pages_per_huge_page(h)); + } + __prep_account_new_huge_page(h, folio_nid(folio)); + enqueue_hugetlb_folio(h, folio); + } + spin_unlock_irq(&hugetlb_lock); + + INIT_LIST_HEAD(folio_list); +} + /* * Put bootmem huge pages into the standard lists after mem_map is up. * Note: This only applies to gigantic (order > MAX_ORDER) pages. @@ -3329,7 +3368,7 @@ static void __init gather_bootmem_prealloc(void) * in this list. If so, process each size separately. */ if (h != prev_h && prev_h != NULL) - prep_and_add_allocated_folios(prev_h, &folio_list); + prep_and_add_bootmem_folios(prev_h, &folio_list); prev_h = h; VM_BUG_ON(!hstate_is_gigantic(h)); @@ -3337,12 +3376,7 @@ static void __init gather_bootmem_prealloc(void) hugetlb_folio_init_vmemmap(folio, h, HUGETLB_VMEMMAP_RESERVE_PAGES); - __prep_new_hugetlb_folio(h, folio); - /* If HVO fails, initialize all tail struct pages */ - if (!HPageVmemmapOptimized(&folio->page)) - hugetlb_folio_init_tail_vmemmap(folio, - HUGETLB_VMEMMAP_RESERVE_PAGES, - pages_per_huge_page(h)); + init_new_hugetlb_folio(h, folio); list_add(&folio->lru, &folio_list); /* @@ -3354,7 +3388,7 @@ static void __init gather_bootmem_prealloc(void) cond_resched(); } - prep_and_add_allocated_folios(h, &folio_list); + prep_and_add_bootmem_folios(h, &folio_list); } static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 76682d1d79a7..4558b814ffab 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -483,6 +483,9 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) { + if (HPageVmemmapOptimized((struct page *)head)) + return false; + if (!READ_ONCE(vmemmap_optimize_enabled)) return false; @@ -572,6 +575,14 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) SetHPageVmemmapOptimized(head); } +void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) +{ + struct folio *folio; + + list_for_each_entry(folio, folio_list, lru) + hugetlb_vmemmap_optimize(h, &folio->page); +} + static struct ctl_table hugetlb_vmemmap_sysctls[] = { { .procname = "hugetlb_optimize_vmemmap", diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 4573899855d7..c512e388dbb4 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -20,6 +20,7 @@ #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); +void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list); static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) { @@ -48,6 +49,10 @@ static inline void hugetlb_vmemmap_optimize(const struct hstate *h, struct page { } +static inline void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) +{ +} + static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate *h) { return 0; From patchwork Mon Sep 18 23:01:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13390555 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73E24CD3423 for ; Mon, 18 Sep 2023 23:03:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0AD776B045B; Mon, 18 Sep 2023 19:03:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 05DC56B045C; Mon, 18 Sep 2023 19:03:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF1AE6B045D; Mon, 18 Sep 2023 19:03:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C75C26B045B for ; Mon, 18 Sep 2023 19:03:01 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 881C1C0516 for ; Mon, 18 Sep 2023 23:03:01 +0000 (UTC) X-FDA: 81251245362.23.B8C8C28 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf20.hostedemail.com (Postfix) with ESMTP id 0146C1C0032 for ; Mon, 18 Sep 2023 23:02:57 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=mNZLYlDz; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=rxY4KSbX; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf20.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695078178; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BWivTdclf8bs60R4j/VxalnwEUHa8h8JPsQCF9s8r64=; b=q2fdV0PkJnORPrRoFyQJaG7IPbhG+vvUwR/opbLDKMyCID1IUNswFxTmW8BRSubcY07R4u 0XgJcQw8S7JKpN37eZEQvfnbKa3z1DjZsFCnTlOf5ib4HLlTv2NXSQqXiq5tbS1v8yZC0V LjDAuHmZ1mGbgJYMQO6xan8PkEudbxI= ARC-Authentication-Results: i=2; imf20.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=mNZLYlDz; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=rxY4KSbX; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf20.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1695078178; a=rsa-sha256; cv=pass; b=RyUgU/FvRm2W97SeEttk5qMxg2P+ZQdSxgrcWMRzrNUpqGxryqgnUqTn3WrwZOq332GpbD HiFOdPOvVsWqIdfMpFFRW+obDnv+qeDgi7sO5qAAZmf8xKWuNu9uml3kr0+acnewiYrVmA xy7s9iAu+kPWr1vcPWYM1juSPg4/42U= Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38IK54so016895; Mon, 18 Sep 2023 23:02:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=BWivTdclf8bs60R4j/VxalnwEUHa8h8JPsQCF9s8r64=; b=mNZLYlDzSNdNUymitkNgZIaEXQj0/ENQKa2WPxxXL0Flk7dePnuGw6LB57IJpmKZLwnY X53EhVQAbmLsWA9cL++VjNU7C3mxTbFu7l9ADLMxHYOYy3rJji6u7FccrHvVSwKQJLF+ 9v2cXhkOZ3IzO8OAc/lHAW326GjdkMkjgxihiJ9LXDmGDCSEIY9FDXDsGiQYpN8fXVA0 mX/DOwgRIDIM98GYvO3a6vOtwb/Ljx7NEx7URKSCfhk3VGZ8sHVznvEbnEsUWBZCa+sR 85XXFOgWegej4Eww3r+Qiu+2fuhZEl8EgQHLnsxVDc0mZowHEHSrTKyS9UIvLAUVPHMr oA== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t539ckqdr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 18 Sep 2023 23:02:23 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38ILd3Hp002093; Mon, 18 Sep 2023 23:02:22 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2168.outbound.protection.outlook.com [104.47.58.168]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3t52t4nvst-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 18 Sep 2023 23:02:22 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ApJVODDq9BDEokprNzEYZo6sy+iR4Axlbw0Czqc115d5XRqbo/4by8HnQ1ncJH2yH0GjGNAIliOUIioLxie2jcDaVKGbA0pmZ6VuxLUcjki/WYdnK8NNRTmjvJGLbutAdycgw1Jui/iPaSHFyv6OWXu95IOxTl35rZSPOhIrEiiCh2CPZHYn6NUwf8CsuH+3gFFRz3ItiAmZa8kqrqqkAQucXCSX5P1/EJusCOobjOgGcuBpYJNsxvMrGoqKkSTdgNavmO6LuMmDnWoK5i1Ad1WVhgsC8pbCIuEhbuputiXIKzqad0MBO9fyl/fyzbqnSPSQxJtRikfkQyYFsx8ZvA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BWivTdclf8bs60R4j/VxalnwEUHa8h8JPsQCF9s8r64=; b=CfgI0pbtzXVFoj775F88Iz4oc6cgTgBfU1TwRbe/acgHGi6JXps7lQg+//M1f4KcMBRE/1eyimzH2D0tROC6e2hcbQfKOAui5OH/3OvPLySVoIlihDJQTgmxIqicNF9BeU36sjbwVjJI6W+/I/OnfDiTJ3vtA9AYOZMNkIz86STtxiYLUq5dcIKRo7rTR7l4W4i4gtUGz+qie8tK4HDCxe7vg8vGPNI/iAByHjRwksIoCX/bknDjL10vSj8frOdgaTIs0JVQLCrIVJzo6hE37feOReBO5nr1bB9PBFA+ow9ybCXjoFXVf4ywtes2EmabU19tctcwxcUIag5XGckYZQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BWivTdclf8bs60R4j/VxalnwEUHa8h8JPsQCF9s8r64=; b=rxY4KSbXLPw8ThuEE8P9mWy1aJCfHDvceCuqa7unZ6RMBv4YwLm7fMQNctIf8/CvEYFNj5+gue9YYdvM0GOguSo/uLRAxDRGnK72MSM6ODiaOen0VJ8dWgKmZmheBbbufItUBox4fXDIfzG/Z/Y1Yr2jO8GH7z28WctOI8l4utU= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by CH3PR10MB7494.namprd10.prod.outlook.com (2603:10b6:610:163::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.26; Mon, 18 Sep 2023 23:02:18 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6792.026; Mon, 18 Sep 2023 23:02:18 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v4 4/8] hugetlb: perform vmemmap restoration on a list of pages Date: Mon, 18 Sep 2023 16:01:56 -0700 Message-ID: <20230918230202.254631-5-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230918230202.254631-1-mike.kravetz@oracle.com> References: <20230918230202.254631-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR02CA0022.namprd02.prod.outlook.com (2603:10b6:303:16d::15) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|CH3PR10MB7494:EE_ X-MS-Office365-Filtering-Correlation-Id: 2a6d6849-bf11-4b80-39ea-08dbb89b4e6a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: bI6NVzmRyDzBo/+DxRqI4bUUFGxcIZIG28cZgS/SXLoafxyySJXxUDN3K1xZZbZNPA17Q9r6lXIuEeS3Jtk0Wh9qVFcMVhUJlxMa6mbt/NWQOE+QDKyl5+a0wLTwk/2GoqVRqXMYTssUk7InqEszkf8Fww2YPBlqpapunmeB9vf8//7wjGaqPMouGe9lIzcNRTu4vpcXmLyrgo/qdchxEyzjbA1naMAWbVHT5SjzWpO7AeeVNYWniT2z238xt3KuEXXPZ+g/egL/TOVtOII7xBGxwoiEw9jWSUCeTko75THNxVv+JYGiZkxaqr9qLQnxS1wUIWZj+PWb0jOhB417olX+udSG+/+L8NxP5A7rl00LtnLMHmVacnFDMw5eZgPEB5kn1DXswzg1WPrcTJAsMxIS4A76CVj0byMshvPiiLJVS7wahpFm0nZavsIXjYUxpm6DyYc7ZuID7s0FxzakO9u7i2RvvDQ63OUdAk9+IZ9sy/d9XeWBYhEsTA+b3PEN46dNIZOWWpA90iJhJjxtWUUkgtQ57S9g/+vhEVxx4gZSru/kmYk77DQO+1mk+gv9 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(346002)(366004)(376002)(39860400002)(136003)(451199024)(186009)(1800799009)(41300700001)(316002)(66476007)(66556008)(54906003)(66946007)(478600001)(6666004)(38100700002)(7416002)(2906002)(86362001)(36756003)(44832011)(5660300002)(8676002)(4326008)(8936002)(83380400001)(107886003)(2616005)(26005)(1076003)(6512007)(6506007)(6486002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: woXV4yv4uaHRgvkrkBmRJsRElebY1wp3kKfNQy118eckfvGwzIJPVsc92WI8wscCu8oIn5db+VYT1Y12OZl39Ek1boQmEd9W/B5NMRBokPGCgHBCcDpG/HgRUVVezfAwEpW0t1ePy1ERtPzz6+RYpO5gX33bqI+lpn6aONv+ey0noLHDJ6YYi0GMf0OsEGE+LWwc64Jiv81yn2LVzAnn+rmUPx0OR1mwrha3BkMMYj7RvExlkWBr5CuNDOkHxUKWv5mCH34ZZ+jHdkT3W1LEsLFtcPKA00JyBnKzz0s6Tymdg0OTS/Ggu42Fa8CtHkEasIzkEG3JgorB0X6bRlvs9GS0ToAK0wPKSAR1TFxIxZMgaae0PlhnRwfBQF4MDBeO4hLFKOK5CjgxzqOVFftYcdlf3UOnAwsuhF+N9Kp1783LpwVBOxB2RMortv5BWW+scHyqXzKOinCqbsWlYuPpf9/d4pyUejF/CWxBxahiObDiNkHimWZgbtQRTZ7Jede3P3Vap9gUniyoLIrptLabRfKNXl3pkwZKymq4vJgOommICqNeKvxZK4rJI4xXN9BGe0mIR/5SuVH1o1X3YrFUIGsbsSQ7urlFWoteSHt4amGImiafLOEKa50BAXnDals7G8AFcebZuLFV5Hxi1LOGUblJD3G2IT4b+H9L53vYFakjJsmAa0pw+RROuARdFioCSlwSg1BY3v6XAAfOA/3kampe6SpKCPQuXptysS8RuvWa/r1LZJLa2x/4IfDMFsEfs7JtoBLP9asKXTeWXER6YxNHcGWjB+MAkWxj/KYMHxDFufc4gc8O+q/eZ+lBpk216Q215zYdZ+AAN5wiQQR12BVqwGKKHegwzUh/3pxbrAA6q0hUDVxcCbnLwSwoZkDWaAKr9uQdp317M3t4VpaAbLFpF9wR2b6vp62K9KAil5p4jPy9bAlNo/dpIduEiwJ+NAcV+N34yjIxojwen0TKzSFoL7/F0ILXRVOEORKRKqLtT4yDvmuiY2+ixQr2GO79v6wSZx8EhhR0TEaKSAxN4eqyy4Ydh2qDKEVCHCBDOUryXiIYV1J+vYkKRauNr7ozwo8sWB4/Duh+qSI/rsg0g26SnmsfDTIRNpMFjnIKnR+j5CKkV+d2IbFnY4FJjglyDj5CP89XJDLO3ZSHxStKudJTl4OwjhKbPrSzWFYrO61ouqdUfCSFSgtH0hwdNUIbhioRV2Y9sdr2DJ1LOn2lvQ9reubXm43OOezq/0uqQKJNxsJHhyeXlqKNIIlVn79zN86xRYn4OnDMo1kjVQNrCWTo07Upz3ZbTyey3/FonMp72bq5SiomiN1T5bg3UqFb9bXrga77h/E1BMUriZKhNCZl5eTY0PsDUYoyM8Y6kI6Qdd7reB+f0lEyciKWXwdvEzo1DRuCnrxWf2GVSex9Eo7mnOLquICYmFwro3iJ85D/+b8n/NzNWGlS4wLS7YJnPwqhfL32ZVDGv14z3cVAuJLLIbhxEl/W3Rp42APVtAhbBdn44HMPPwFFoWNC96KbXLbrAy212Ec71JRh/2pz6DM62RZNvrxH38eAt4gOdkwAZyWhUGJyCBUBb+foNLyH X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 0urzaT/DJ4Lx5TDjMfxpkIiYwFPE+Y9pZP9BmVIDRcp/iOZp0usNRraK7GXBtylYFua8IXPxWUFgIA/7H1KiER1UDiD7jvfHloqbvqZDsjlp84yaEI7+4QiSjzUNItiOJfdtSbDpZjCWHwxV6xGj0FB6DO8E0+RYJ5ZsGbviaLuqOpN93C2I1++mibabUrDO8giPS3A9QSg1NIA5EssaVDTCxCs9CNVfcJizCCBvONhZMhv8GSyBLJmnzWATJ81I7GKBhhYliEMCWgLWLfm4+K/c11aaIy8nyQzi4+TVBUQ+eGkxPzkw/lN7nQ65mP4GYABjgMZkzHKxWk4VWZThiZxIVRUGJ7j1pMLZeWiCiznJ8oSR3oMdJUg0hRnYFFMOMzjmO+rkbudw4F1Uiugb2ixJr41JK20Ovgeti3Us8i6LaDWjTi6Rwq0sjp6uUivva5EuzaycFok0rw/dYHAlE6n3p/18MEZh8sPo77RUYChgqenx7c+67vd/5FBmet1vTv61tfvObsOrbLgHGlP0/lPoaXfXo5n4XJywUafbpzrIZlGIV5R1ixbWjAL6SmzwdywoWodSjFbUJdaIUbiOr0QhoIWefn9GT9oDwBCatzaip2xXL0uX0VQcL/02TUoAgJTAwIu5eyccnfo7uYxVyhXtTWhmzINPzdd5WwkzABgGzhuWY+xfKOMxytedxB8A7lyFR00MeBRsH83fn+Sxbo5wNBpQJC4RdcolpSJ0btAKNRkt5XyT/qUmO6jhR+d/tNKOTOh9jZj7DWIOnG6XG3wZMrIrosSOku2u6LnQ8F+e5FvkHqJouYvxr9ZE9eAuYUWkzDEAR4L3VIsG/D9NSp9odUl3O7p0g1mCY7MXy+vz91PuzBYDlrt04KzfoPuvhaOp9PDqavlQ2j6w2a1uYJ3JGTViCH6M67yykv9vJBtkO/5WK03P5o8rt29NWkFY8jpvovVQpKK5hA0MZOGrG2hsgWvVPF83KNb9SXJ+E1LwBnF631gM3yjLrhaZSfJnX/rsMvbdu3rpIg0j4La2l3Dt57q65gNx6BR2E/Q5CoU0yWq8fZenwaYRkKPQQc5vHte43k2szbxwyHDTThRHmrLPm9egxeIwiyfydWUxF67mJuHy5B3RFbvLZoU4845D X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2a6d6849-bf11-4b80-39ea-08dbb89b4e6a X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Sep 2023 23:02:17.9285 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 4IhPQtPXc7uVqajX8pyOLnU+TMYJ+xWKDi4HqaGWbi5m9j/2xYNElt2LKg5fwJXsBr8o5vmpzPjWPL2gW9hhpA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR10MB7494 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-18_11,2023-09-18_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 adultscore=0 bulkscore=0 suspectscore=0 phishscore=0 malwarescore=0 mlxscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309180199 X-Proofpoint-ORIG-GUID: uHuxZ57rwM1eJSTTK4ZDA4gQneMnvgY6 X-Proofpoint-GUID: uHuxZ57rwM1eJSTTK4ZDA4gQneMnvgY6 X-Rspamd-Queue-Id: 0146C1C0032 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 9ogfyytzrxiapxq5ygtcjperj77bbckh X-HE-Tag: 1695078177-831613 X-HE-Meta: U2FsdGVkX19w3BKm4YvGl3tx/o1+DBOn5oxp1WsgYoYe/kQgyW7Ba0F1gYSaeyfhMdyy1PDjr4nBM4iNglfEuIqlEZyHIV/mCF8p02NNpVbZhPu/rw7i+6QzQdQ8aWicSkO6DbmUOsg8iScXz6MerWB/SW3Ag84dFStg++j1myZ7XU5snCRyVVXKumk9l6GjQxcAkeNR2eXeo+7nkJkCAVae9yRbB5FYB7koKtHeFHdWaLZVBz4lylbcRpRe0GuXeCShCtLqi65Kstb5rSQ1ZvuUo2HO6VvyjA1QiR20EFl9WgtrWtF0nHPJwuhcsyf7MfmWvsk1YSsy6v5ctOSeXhgxv09NKxKFiI2P/7Cp6z4zgiTKRhCKSkmu9fb79QzWDUKl9P4T9+QB8qetIakT+M90aVLk1LRLVD9s4ZqWgvFX4fx7uPGdT7aaLN+8qD+pPixsbvE5u59JM/+Zf5/sg227Fi9wGPd9cc2U/HoU0OYBgkHG8In795FhSVM5bC705r3OXIoZQFd80x/SxjkK6X9UDi8sxDtipZNgptZoztbiaDAorqQuw1LXtVzKH08fvqNb0r9i53S2NxRhITDgmkw1NP0aCDzz5hlRQu1gkSmRNkizSps1SFBSOhLQHsMvj0KXM65E+m/tFM3IoyjZRono10Pzj71T1eYRQnYpU1hNJzkupQRBgEvnXhyGXK2nR+al3k+lIShoHcndA56RUu4X5iQmfrWB8n1twzUqnTGVtSUyXnlNuKQTFGCcw7yQUjmcmlb2obX8MKWSzbctoySEhTM3eQMVVSDiyMs2zCIbbTCNN/v+5MhIjiv8EKCFZnaIFO4iMq2EtF4xtuFcgPgt4doccoMCOCSt42w9QePE77Iz9VV2qKvWTGCLtHGlrni1AuVrJTcMkri3xQw8QyTwXkONJ18XpLIbxJ3mzJ5udqrF/EH8e58tTLE5TXQTF79Pmm8bqxDq+kdWaUE X6WpJGOr oypkYxdX9QIA9tDPNKg8JFb3z7zvgQhz7RNIxyODV+CkctgcO2U30bRloUbzAPYSngeqaqoellKM7h0pRN44/hqwLEUW4A6IUMnDACN1r3F4c6dXOUx2NAkK+OwqVvKHpz0YoUhwy/P+pj3b4/AB7pmtuflPgnEUDsmgUnGVc08GdyejCHwkjDz4+X/TfqOIiKXJRWgi1tbsCu/lRBnTotaNV2tLtL/thi3M1FsOhFMJtJVtLAI34L3+WDO/DkvRANIwA/K5ll0VddcuQaMKkyqS1RcgkhifIYRNnpRAm6sk4p3GEie/aaoZ+fDatJPKk1d1z4W/pVz9SYtj1URCon5xhCfXD8gP2MOxl03pT4QCC0dcQu+agdmW293znf6tRB/gM8axvZY5F+dlAhpw8HfN6l6TijoEsBryfOkxzw2+lFV2/KfRw7ZsA8CGs4k/P1Nz/lU/yGjoKHp9Ylnoy0/qBntSCSAb1ihxF43YLJSLOs4Q+RENYVdmaG62JSzG+jFpf X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The routine update_and_free_pages_bulk already performs vmemmap restoration on the list of hugetlb pages in a separate step. In preparation for more functionality to be added in this step, create a new routine hugetlb_vmemmap_restore_folios() that will restore vmemmap for a list of folios. This new routine must provide sufficient feedback about errors and actual restoration performed so that update_and_free_pages_bulk can perform optimally. Signed-off-by: Mike Kravetz Signed-off-by: Mike Kravetz Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 36 ++++++++++++++++++------------------ mm/hugetlb_vmemmap.c | 37 +++++++++++++++++++++++++++++++++++++ mm/hugetlb_vmemmap.h | 11 +++++++++++ 3 files changed, 66 insertions(+), 18 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index d6f3db3c1313..814bb1982274 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1836,36 +1836,36 @@ static void update_and_free_hugetlb_folio(struct hstate *h, struct folio *folio, static void update_and_free_pages_bulk(struct hstate *h, struct list_head *list) { + int ret; + unsigned long restored; struct folio *folio, *t_folio; - bool clear_dtor = false; /* - * First allocate required vmemmmap (if necessary) for all folios on - * list. If vmemmap can not be allocated, we can not free folio to - * lower level allocator, so add back as hugetlb surplus page. - * add_hugetlb_folio() removes the page from THIS list. - * Use clear_dtor to note if vmemmap was successfully allocated for - * ANY page on the list. + * First allocate required vmemmmap (if necessary) for all folios. */ - list_for_each_entry_safe(folio, t_folio, list, lru) { - if (folio_test_hugetlb_vmemmap_optimized(folio)) { - if (hugetlb_vmemmap_restore(h, &folio->page)) { - spin_lock_irq(&hugetlb_lock); + ret = hugetlb_vmemmap_restore_folios(h, list, &restored); + + /* + * If there was an error restoring vmemmap for ANY folios on the list, + * add them back as surplus hugetlb pages. add_hugetlb_folio() removes + * the folio from THIS list. + */ + if (ret < 0) { + spin_lock_irq(&hugetlb_lock); + list_for_each_entry_safe(folio, t_folio, list, lru) + if (folio_test_hugetlb_vmemmap_optimized(folio)) add_hugetlb_folio(h, folio, true); - spin_unlock_irq(&hugetlb_lock); - } else - clear_dtor = true; - } + spin_unlock_irq(&hugetlb_lock); } /* - * If vmemmmap allocation was performed on any folio above, take lock - * to clear destructor of all folios on list. This avoids the need to + * If vmemmmap allocation was performed on ANY folio , take lock to + * clear destructor of all folios on list. This avoids the need to * lock/unlock for each individual folio. * The assumption is vmemmap allocation was performed on all or none * of the folios on the list. This is true expect in VERY rare cases. */ - if (clear_dtor) { + if (restored) { spin_lock_irq(&hugetlb_lock); list_for_each_entry(folio, list, lru) __clear_hugetlb_destructor(h, folio); diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 4558b814ffab..463a4037ec6e 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -480,6 +480,43 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) return ret; } +/** + * hugetlb_vmemmap_restore_folios - restore vmemmap for every folio on the list. + * @h: struct hstate. + * @folio_list: list of folios. + * @restored: Set to number of folios for which vmemmap was restored + * successfully if caller passes a non-NULL pointer. + * + * Return: %0 if vmemmap exists for all folios on the list. If an error is + * encountered restoring vmemmap for ANY folio, an error code + * will be returned to the caller. It is then the responsibility + * of the caller to check the hugetlb vmemmap optimized flag of + * each folio to determine if vmemmap was actually restored. + */ +int hugetlb_vmemmap_restore_folios(const struct hstate *h, + struct list_head *folio_list, + unsigned long *restored) +{ + unsigned long num_restored; + struct folio *folio; + int ret = 0, t_ret; + + num_restored = 0; + list_for_each_entry(folio, folio_list, lru) { + if (folio_test_hugetlb_vmemmap_optimized(folio)) { + t_ret = hugetlb_vmemmap_restore(h, &folio->page); + if (t_ret) + ret = t_ret; + else + num_restored++; + } + } + + if (*restored) + *restored = num_restored; + return ret; +} + /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) { diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index c512e388dbb4..bb58453c3cc0 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -19,6 +19,8 @@ #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); +int hugetlb_vmemmap_restore_folios(const struct hstate *h, + struct list_head *folio_list, unsigned long *restored); void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list); @@ -45,6 +47,15 @@ static inline int hugetlb_vmemmap_restore(const struct hstate *h, struct page *h return 0; } +static inline int hugetlb_vmemmap_restore_folios(const struct hstate *h, + struct list_head *folio_list, + unsigned long *restored) +{ + if (restored) + *restored = 0; + return 0; +} + static inline void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) { } From patchwork Mon Sep 18 23:01:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13390559 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9A7FCD3421 for ; Mon, 18 Sep 2023 23:03:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B5D4D6B0463; Mon, 18 Sep 2023 19:03:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B32CD6B0461; Mon, 18 Sep 2023 19:03:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6563F6B0462; Mon, 18 Sep 2023 19:03:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 2FF536B045D for ; Mon, 18 Sep 2023 19:03:03 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 0160F120460 for ; Mon, 18 Sep 2023 23:03:02 +0000 (UTC) X-FDA: 81251245446.07.96ED98E Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf07.hostedemail.com (Postfix) with ESMTP id 8D17540007 for ; Mon, 18 Sep 2023 23:02:59 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=3x48z9Xg; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=wW+yoqML; spf=pass (imf07.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1695078179; a=rsa-sha256; cv=pass; b=wXCPLCadTP/cF++aBXyNP4qRlhg3mFmGeNw2CPLgwCZGfG/orIJPM+4JD7zEJvjkTXrlIZ gfZsJoXUd4P7qZgvvrO0d0wYlRMJePwyDCUQUrBZ78TUpG6TCT7SAeoFIn7l4CgvSdixPL UjE46fnlIfR5hTzS6C0Tvx4wDg9z84I= ARC-Authentication-Results: i=2; imf07.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=3x48z9Xg; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=wW+yoqML; spf=pass (imf07.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695078179; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Y16y6xPVgvI7a8L2xgbgzpDrXOvAdzxTZsCIzk2fyVw=; b=uUy7e9kjfw9D7P9Z7V5ho6lxlHqUUbOzbO7IR2gkDHmmgbnFZcYt9T66xBi/ZwQf6eNDwu Hk3kEdw127zYxePKqiWgxlS8neCwELONnO4BE11tzO3chogWlcu+fPMMm9BmQrR3nAVmdu 2gbONWSBQUExw3lSlXGVpW1iisoy4Ek= Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38IK54UD005659; Mon, 18 Sep 2023 23:02:26 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=Y16y6xPVgvI7a8L2xgbgzpDrXOvAdzxTZsCIzk2fyVw=; b=3x48z9XgPar73/CgIkpbWSMGfqKQUl3y1yn7I8wh01CQ5gF8vmLLlSxRfde4ANsKBGx8 gR31H+ADpLFtDhzQTmCm2qu2YlQecFpAAHFj8s8qTAxBeuDV4wkP8O1X73PD7CwcDqYP Wy4mdv4DQRH/Z8eHWiJ/dvOIpNCbhcZqml0+BweCt9zT+4EqjvwsG57Cne30TRbqXWHt PKap0+n43wb+3GG6ospP9Ozwx7XojBAbli2f1/s3xFYdzjSjYw36PdWjjAAhIegc58zg nQ2R8ZC0Xon0L2ctuGOYEqHK2hd6Uj3RgqZDfmI+GvJWpaUpSIiuOiRnWcdrKD1yvOSE LQ== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t52y1upr0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 18 Sep 2023 23:02:26 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38ILubHU016052; Mon, 18 Sep 2023 23:02:24 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2168.outbound.protection.outlook.com [104.47.58.168]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3t52t4nder-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 18 Sep 2023 23:02:24 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gJ97kMFysiMp9zSD2wkpOXM6R/sZW9PQA2LnVE4dkn5bDE7JjLMY8MtanVM+1ZQLUFrTEOURLQKGZuUHQOGg2i2zQipV6aXPCrzyHOY6BvFATPzo94VzEJ3eAloNB12WZcbGmeHk/GSioTDFWd6iSi3uvyNxb4KsvXalGaLwXPm+fPKbcT8icpuOFu2p/CU2Rczi5qdAc8eKJnV3Mo4w+uacQy32SbuHHcMv1HSI0ehG1M5PjqT0WPgltMnJxuhxGIn5rpVvXVphTBDtExVcDBDjAltdntrRxEMFDvLCx/mp4QvC1BYWuXayg5VnHFnWbAKITf2juXCQWQVXO9tTbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Y16y6xPVgvI7a8L2xgbgzpDrXOvAdzxTZsCIzk2fyVw=; b=fdyuzvF4X2Fk7NlCPWy4cQt2X6ZV5/CfqPd9uqbksqr93j7ZmV80HZjc0UQRF0zj5YMORfLBW3m/YrPn4X3nfuiMFIuYkBYI2edlr1sw6yQiCkCFqOC8JeGAtEiNpaaM4pyUZ5fuAzQtg/DNyaTrD6hG8u1ZZ/Mknoa2+lM0UFhcvBsvP4zDIBw3Ex3tPjzOpacv76Pn1GLn3MH2BgNlNjwrpsMlujrKNGsxpr2tbP5CTdirJcRl2m8Va1FoBObE0rmRw9cPut+MPVzSx+uiPlAmj7A5+eIvF+49VKey2VY1+dRNmkBrH3vwJxgoeYw236cWXeauA03w48gjG6u/Pg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Y16y6xPVgvI7a8L2xgbgzpDrXOvAdzxTZsCIzk2fyVw=; b=wW+yoqMLzOftf+0SXOS94q4bNo8xgEURYGhmGksBI5A92+L6vC/tM6D87ZAVJMrUVqfWz8LLgrr0T66jjNhPmgwXvW4wYn1+lo4W2ftu/g2BmtVYRfSue5jNsvYR5hjUup5HpsuiQfakHyivZi9zdA/lyf6RHIw9DUu6p5xOdKw= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by CH3PR10MB7494.namprd10.prod.outlook.com (2603:10b6:610:163::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.26; Mon, 18 Sep 2023 23:02:20 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6792.026; Mon, 18 Sep 2023 23:02:20 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v4 5/8] hugetlb: batch freeing of vmemmap pages Date: Mon, 18 Sep 2023 16:01:57 -0700 Message-ID: <20230918230202.254631-6-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230918230202.254631-1-mike.kravetz@oracle.com> References: <20230918230202.254631-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW2PR16CA0049.namprd16.prod.outlook.com (2603:10b6:907:1::26) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|CH3PR10MB7494:EE_ X-MS-Office365-Filtering-Correlation-Id: 1d3de23c-99c0-40e1-9a80-08dbb89b5025 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: h9qFBw9tL1ZNaMQ3gUxzkE6jX8P1w0+gx4cGr7Z1kUnG+2Us+0+qY8Nm8TsxyM71EHU1VEdfj3yOl7I/FHKgitdUDR2N6/q0bzuucK79pnAIwHe/vZOF17UUHmWb5aSdVgLv5KhRF7GyifwNqFwDhkDJ1HqMVmsjs5QWmM5UauyCRaw/k9zALcma9/2K9BQ/utN2iXQVwZ92hPF+ruAkclyqTVsvTERnhlx4QNSK5mOHIwGTsvs1SJo7u1uhzXyquGJPYtsGrlq2FnXNkgEaSWWzxI6mWRya8x3zU/e4+uzG/FUaxeCsyZjeXEKErliM+aQ1MpuY9oChyJwHGyuJDySFWNQ5pWP1Y6PxmY/aiojwnwDvzykCRX5c+edAL0KW+wCm8EOzYPQpn9AwuoTs3ZkEBEPsfHEkCbf0Uh2N3uyd/NkbuoNur8n7wyDYkJp7e8U9oAfsQ8tlPJ1yi221bnWN3mkLUgDHycskXL19C/mN1WyCj2R2hD9w8AMkr8KN0yNBW8kVZHNrCbf3n2Mjgq4kpSjVDovNwjZgpWi2ZTr8ex3btOaz+m4/d/Wsh1hE X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(346002)(366004)(376002)(39860400002)(136003)(451199024)(186009)(1800799009)(41300700001)(316002)(66476007)(66556008)(54906003)(66946007)(478600001)(6666004)(38100700002)(7416002)(2906002)(86362001)(36756003)(44832011)(5660300002)(8676002)(4326008)(8936002)(83380400001)(107886003)(2616005)(26005)(1076003)(6512007)(6506007)(6486002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: miGrKbcNZpW5j9hgE8fPdwljQ0DRmW1/XJOTaU2FLv5UirNhYK6M2lfsfQEQ32lZvv6kwg/YQkZ8+kYf3iCcMPm2MSvfVee6nXxTlE2ZTFJ65/5DepQ8aYRLwSAGIyWjLayrIDlUNgzLXvgqYhw/EWL0Xb5Rb50JeKaRNTvAhK3SdkawH/O/uywBIyA1SQiD240gYpabYI65KleibyG67y6qacDMYhurZXBoRhUOb/fmidhCuaKTvbO5WYqiqexTBAD+NdzJoa7PQ8MGHoX/+qtqXx9eu5Ome6y+CTxthlexSFPS018DIv20xHxyK40/8jxjlqm32Fhe0oofhTGMWjy9abkJcQ2MQtVOVds159ucDXjzk3KGIyi0f/z6K+H098ZYD1A0L3L5yOVTlN/oGrGUFApeyedAX9J1J1qUZcbaK6zG8jl+B0/0f6bez454XzSgyw6BIJXFV7bKZF4eWPyTgZFz1YGdinlb2jkMN0XHXAIUGhJOfbmHtw8CCbFJqRH6J6I/NN/SK7oPl7g6rWCLowx3b01HF5PAVUBUbEgRdYgBSs7EE3HNXLE7yS54QmW5QJf4xcHsotQ/H6CbYCecgYSLOcaJjz6BOQMBlyKUkxIHa/3InS/8c6q27JcgngyAKeMiKBEaCgknf+sz7ut4R51KU1ksftcHxoavZ0uDg3I6+q5SYj3KIF8cySJjJfk+VcMwGoMzV6WL2JhsN+yTFaILCLySX/fG6z0xXG2HuaP2hp2ZQQaSj7UJbX4czETSQZKH9d6AaFal1mNHoKplN8ki5m2ptOsWaRMGFHGmNrFVzIoyyGjrY5P5YE9l1/6xkoAPt/iGKjTpc+nQkjHxxNg01QOlmilYqvOSX997u/mqFQ3/H2a9OkNtMcZyYgzbpM9UvQXA3l9HFeijiGTzFOnbwqay+NeVUNOx/jKXUSRIiApv1sLfNJyKleBqjxA04ED/oQaMf+0HSC0d//BcPfWE/JJh+YbBnuBc8QlOAurmexM07p7CL8G3cm+na0me/libWP3r6LOG/RRY9BjJX7zzuuyU+ARrHVAKPAqWWO6vBOPd9RO/up/kji5yLOwc4XJTcYVN3Kf1HuqNd+SYmoyyIvD2PFz9jmqqgV8f3T36orsCOf9E/7M5LsCeR5XHq6slkgrTQ4NchTApHmxROkgZrls4k5x4rLfDL/ay8M19EE3pOwm9qP2kC/lrPBeivtMmvNEZHP1KS7Mim0kB+l5zW27hVvzlHxYCAfClWTZx+VfrNnmwRl2UiEyGO1D21HH5Z/rdTVVvTYAwKN2Q6Lt1qPwtr1lO+EJ8AzI0OIffNPeAH/qjSnbNKnH46M8ABM4gVkhb1PuxqTArLakOriui+eXSeh90mHoSb0/2cRjJv4LzQ5yio52KEBm33cidu/YYhcAQHFiWzGE5NgpkJZEX3CVYgg/8lTkPSmvalk8+gYfldu7DUh9boNPOsMEhkegA8XOBJqBCt3Dov2ftTkXM9LeJc/oK/VtzaQ0w4QM79AMGG45/twW03WOy6w4+weX5PGHKO+Wpl7r9aVBldPe04yHFEYgNkMhO/bWB10DNWmq8nVWkBjNWuKMe X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: L0Bwz1RY7TakwgtaALdP1mLCehayJnJ5os+FKGmg+wfUIDWxbRUv6il69v6z8+BBrbB8EUFEZ1krUUka3YAvpfv4RgEMbmNVpI/9hTh4EvjMflxRMgLApjak3dVuQbOHIHuUynlUlksPU8LD2TP+tp5fia8aEDlblbHZEddeNrAdLAUeI28XuoheLEAZSKiJx5+ZWSCNVpoJ7YODPCru//+0I2nxJk7kxo/7OwuG48HGf+P9R0kjn4jLLuC83dxjfvV6NQxv5D9KQMAPpjdB3s/8OXVnoYqI2wx4Od3eWl0p5HgLdRDvZI6HQBD4eiblQAvk4ph2FGPwPrj1Qysp4VBLIWIZmzNGS/B4ax02XMWYpGFFjlbnMuVueAMdAK0x1jIv4E2VI6i35bEpYwJYOfOGWUFzug7fK95v7WEtdtH2yPvb3dTFrLQ6T5CG0SUrLFz+M3QZihl9TLa6tmG9xPpBF4m0xynw6hoGSpWljZEeqVJShs6OhEs6wW8kgf6WPTgUZiXelOGCnz8y7IelpbfBTrvCQHARpkZwajyN7NrvA/VMXjnatzC9Le+nwEHZSc+ctZ6Kxu7ff1wq3nnpmq2H0z8KcbSN4QDA0E5S0OlGpNzw2V8WnodTewbt2/tKgH2LJsfe2cnYuXtIr1q74EBgS82m3kgq9IGOUD2XjL9E6ulyPuOQPAljCB2RQgu502JD+efJ4L/V8AeLavIA1gJEb17KmWaib+zaNMdbSourPXV9S87dUdD96UO5xFNDq0xmKR3NkEB4p8nByZzz2Qyms1YCg1jxfD2136v4xF0KZIwnL4V1xX9V4U5Uo36W4stf3aplxfRnea8rNy5phtwiKTQLjD6Tt3KpBemwdR2g8CjBSle1EYFPD6V2X574habkqzxeg31HVk/K/Q3YnCq5G79lgH07+Dteg4aUtK63Y+Q8sJuoL6Cqwt464aTkvCKp1GbYvlsiOdzrYsy1SHBx3+n8aScMQJbHrWW8/lyKwQZx4XK+f7PUstr4hHusGMHoy6/wD657NPjJvnhbNdb3h1l7dmmTXVOLg9MyE1knxxwsSdJRytxXdK1HjMmsu6QcRlSVnFYF08ShDZJVvA+GbvMciPypNK29LKy0qUNNp12gOpwDKCB5GzIqR9gl X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1d3de23c-99c0-40e1-9a80-08dbb89b5025 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Sep 2023 23:02:20.8263 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: tlgvJR8qAn3balXj3gRSu3wCu7RmFuLS41Toijz6B121tSZ8Vv9A+lkIkmqqbD0PH8xASRbI8YVYtYX71Ss/Yg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR10MB7494 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-18_11,2023-09-18_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=999 adultscore=0 mlxscore=0 spamscore=0 malwarescore=0 suspectscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309180199 X-Proofpoint-GUID: 5R7xdpdnCPEO44ZY8QGEPxpw2FGnSMps X-Proofpoint-ORIG-GUID: 5R7xdpdnCPEO44ZY8QGEPxpw2FGnSMps X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 8D17540007 X-Stat-Signature: 7pi4p5oht5o44n9hkaw1commxd1zfrss X-Rspam-User: X-HE-Tag: 1695078179-833714 X-HE-Meta: U2FsdGVkX1+HF/3B2fiOIN/haYVxGvEB0WL+XAZbwHKRYUyTQoRbJhP7Mi3Ao3TJqsFOQYvfn6r9hSGzImXgYzuLvb+bjvpY/oATqRqx/Bg4pHiVdKTX36MnsF7R9SZ4weS/gz5KwSwch3QMXGzMIaGvBobN5oCaIRAAKxYmUAueiW8Ln74/qIKjwRPi9FXUhmqae1kFTOf2hyOW+d/Z9YNku7XLVbjF4PoqXf8qu+bnHECAypKbbIiDEaFqn54t9dlU52E2VkxbKJN2vnoQqMWi1eqhizx8w+9uKiSCijOd2qUiHSF1HW2JjejHBe/0TsqMlvlr1FBg65NOyA9k410HnCRJ5BIPwWOYyAvrtXhCUxjK/3pSWOX5h4Ct8/UykHdWYuKhyxVAWKS6vYGEWakQyxXM8H3oOGC/xAFQsNZI6zA5mRt+DKd/ahPg3RB8cdit0w0OhJDZzSgfy7xXh74PcvGcZsKOiPs4TbvUCTmXKzW7XhYG3CTLev5ox3Np/3d6SbNtrWDTwJEBf4tGr2ovIpTqA50VBHvs6FviPJGvI5Su7WvpzuS5Q0qlHlbH8H/Fpd9xeUNkkoMDdcyrMorpnO12lxk4NWJ9q/Cgtc+tVS2zWOggN4B5XdZJ29xP+9WaRxpsdJl5BsmxHvq5qWH5qA7Dku22UiXGbObsTgp2GQ59kvOOePNdM2WOqVV3/bT6xBjBgVZK52EA6wtxMUkTIJpfMC6Yt/j2SWBK+bd6YhDcxh7hObYGY19q5UjBG0u8yh2fJ8Ldtj8/Z6w7fvcCWTp5jBZc2/OXEWK4/Gp2LB/82zFd8i/dlBx8y+Ha9aTRkLm5l/BZoua+XUEKKdeDvb3NtA5hwaJTRVUMTe98QUiNm7fOp/udhD7dIk67siWK6FXfH2sf4f5lsGzr2vN7QxAoHadJ3ag7RFltQ/61L2Vp9hSPWvu1ZvRGXWMxndrRluBp+1+DQ3FJm1R m22Jct4+ ErfzlXKPwHolvEQKgwvTZS2RlXE4lHqX03SFBlqgjiseTPQVAF/PV7LWhcs/XvbRn7OZlr7Oxf2C22YCfKoSpt3y6OQib2U8huJq6tWrD5T4i/2pz+jx0fT20jgwwQ4aDU15KuAcW3RjKTPoeEyhVkjWXEf4+ssT89SU0SBBKYQUvNbT4tlkRB0EWJlSt2NYaJyOq7eulInCQJHmrlvyOGUulwLHq68+Feq97zwdleRmlNy3fw6QqhrAzUzYQPZITEYEYVG6oDzr6ZqFhsENzBANAGxYVwBpCRI8OpGgwG6brpj0IdFYHHGsjA5nh1U9Lr5qxxkD22toGE4mDHyRmoQA04xofu/FoTIDZ6cSvGXWDbPYQhgr1Tv4xk9/ebVDKyhO4pWTmOeWpu1remNdQzlF992GstBRBhqplUxZXhZmY9WbAAgnTP4cUuh8ZUcP7Sw3uE9UCGN9fQn8EbphUgEGL90UEL52L/Umx X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that batching of hugetlb vmemmap optimization processing is possible, batch the freeing of vmemmap pages. When freeing vmemmap pages for a hugetlb page, we add them to a list that is freed after the entire batch has been processed. This enhances the ability to return contiguous ranges of memory to the low level allocators. Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song --- mm/hugetlb_vmemmap.c | 85 ++++++++++++++++++++++++++++++-------------- 1 file changed, 59 insertions(+), 26 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 463a4037ec6e..147ed15bcae4 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -222,6 +222,9 @@ static void free_vmemmap_page_list(struct list_head *list) { struct page *page, *next; + if (list_empty(list)) + return; + list_for_each_entry_safe(page, next, list, lru) free_vmemmap_page(page); } @@ -251,7 +254,7 @@ static void vmemmap_remap_pte(pte_t *pte, unsigned long addr, } entry = mk_pte(walk->reuse_page, pgprot); - list_add_tail(&page->lru, walk->vmemmap_pages); + list_add(&page->lru, walk->vmemmap_pages); set_pte_at(&init_mm, addr, pte, entry); } @@ -306,18 +309,20 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, * @end: end address of the vmemmap virtual address range that we want to * remap. * @reuse: reuse address. + * @vmemmap_pages: list to deposit vmemmap pages to be freed. It is callers + * responsibility to free pages. * * Return: %0 on success, negative error code otherwise. */ static int vmemmap_remap_free(unsigned long start, unsigned long end, - unsigned long reuse) + unsigned long reuse, + struct list_head *vmemmap_pages) { int ret; - LIST_HEAD(vmemmap_pages); struct vmemmap_remap_walk walk = { .remap_pte = vmemmap_remap_pte, .reuse_addr = reuse, - .vmemmap_pages = &vmemmap_pages, + .vmemmap_pages = vmemmap_pages, }; int nid = page_to_nid((struct page *)reuse); gfp_t gfp_mask = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN; @@ -334,7 +339,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, if (walk.reuse_page) { copy_page(page_to_virt(walk.reuse_page), (void *)walk.reuse_addr); - list_add(&walk.reuse_page->lru, &vmemmap_pages); + list_add(&walk.reuse_page->lru, vmemmap_pages); } /* @@ -365,15 +370,13 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, walk = (struct vmemmap_remap_walk) { .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, - .vmemmap_pages = &vmemmap_pages, + .vmemmap_pages = vmemmap_pages, }; vmemmap_remap_range(reuse, end, &walk); } mmap_read_unlock(&init_mm); - free_vmemmap_page_list(&vmemmap_pages); - return ret; } @@ -389,7 +392,7 @@ static int alloc_vmemmap_page_list(unsigned long start, unsigned long end, page = alloc_pages_node(nid, gfp_mask, 0); if (!page) goto out; - list_add_tail(&page->lru, list); + list_add(&page->lru, list); } return 0; @@ -576,24 +579,17 @@ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *h return true; } -/** - * hugetlb_vmemmap_optimize - optimize @head page's vmemmap pages. - * @h: struct hstate. - * @head: the head page whose vmemmap pages will be optimized. - * - * This function only tries to optimize @head's vmemmap pages and does not - * guarantee that the optimization will succeed after it returns. The caller - * can use HPageVmemmapOptimized(@head) to detect if @head's vmemmap pages - * have been optimized. - */ -void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) +static int __hugetlb_vmemmap_optimize(const struct hstate *h, + struct page *head, + struct list_head *vmemmap_pages) { + int ret = 0; unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; unsigned long vmemmap_reuse; VM_WARN_ON_ONCE(!PageHuge(head)); if (!vmemmap_should_optimize(h, head)) - return; + return ret; static_branch_inc(&hugetlb_optimize_vmemmap_key); @@ -603,21 +599,58 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) /* * Remap the vmemmap virtual address range [@vmemmap_start, @vmemmap_end) - * to the page which @vmemmap_reuse is mapped to, then free the pages - * which the range [@vmemmap_start, @vmemmap_end] is mapped to. + * to the page which @vmemmap_reuse is mapped to. Add pages previously + * mapping the range to vmemmap_pages list so that they can be freed by + * the caller. */ - if (vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse)) + ret = vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse, vmemmap_pages); + if (ret) static_branch_dec(&hugetlb_optimize_vmemmap_key); else SetHPageVmemmapOptimized(head); + + return ret; +} + +/** + * hugetlb_vmemmap_optimize - optimize @head page's vmemmap pages. + * @h: struct hstate. + * @head: the head page whose vmemmap pages will be optimized. + * + * This function only tries to optimize @head's vmemmap pages and does not + * guarantee that the optimization will succeed after it returns. The caller + * can use HPageVmemmapOptimized(@head) to detect if @head's vmemmap pages + * have been optimized. + */ +void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) +{ + LIST_HEAD(vmemmap_pages); + + __hugetlb_vmemmap_optimize(h, head, &vmemmap_pages); + free_vmemmap_page_list(&vmemmap_pages); } void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) { struct folio *folio; + LIST_HEAD(vmemmap_pages); - list_for_each_entry(folio, folio_list, lru) - hugetlb_vmemmap_optimize(h, &folio->page); + list_for_each_entry(folio, folio_list, lru) { + int ret = __hugetlb_vmemmap_optimize(h, &folio->page, + &vmemmap_pages); + + /* + * Pages to be freed may have been accumulated. If we + * encounter an ENOMEM, free what we have and try again. + */ + if (ret == -ENOMEM && !list_empty(&vmemmap_pages)) { + free_vmemmap_page_list(&vmemmap_pages); + INIT_LIST_HEAD(&vmemmap_pages); + __hugetlb_vmemmap_optimize(h, &folio->page, &vmemmap_pages); + } + } + + free_vmemmap_page_list(&vmemmap_pages); } static struct ctl_table hugetlb_vmemmap_sysctls[] = { From patchwork Mon Sep 18 23:01:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13390563 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3AC57CD3425 for ; Mon, 18 Sep 2023 23:03:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 088A86B0462; Mon, 18 Sep 2023 19:03:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F2BDE6B0464; Mon, 18 Sep 2023 19:03:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE0446B0465; Mon, 18 Sep 2023 19:03:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B69626B0462 for ; Mon, 18 Sep 2023 19:03:04 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 94F8C1A03C9 for ; Mon, 18 Sep 2023 23:03:04 +0000 (UTC) X-FDA: 81251245488.12.BB87E92 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf30.hostedemail.com (Postfix) with ESMTP id 1E5F980019 for ; Mon, 18 Sep 2023 23:03:00 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=CNk8QpNW; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=V8aByWVj; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf30.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1695078181; a=rsa-sha256; cv=pass; b=yShX6WmGxtFd6MOwZZ+/2FvWlcFR3Dzuvqo3zf5WEeuuQXPBc/8yPRLeOj9RFYA8BoORxF V/JqTGL1lBaBsp368D8Q7gk9lc9Yv3oV4U2Jcr9ilV28AZT3OlTGbiEMuxInYox6aGA8py oAo2RNTa4xyR8PAE6O4JAERgNzfi3ps= ARC-Authentication-Results: i=2; imf30.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=CNk8QpNW; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=V8aByWVj; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf30.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695078181; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HVizojME0l9i5OjsslwdvJakCgkz9Uqrfyq+3ppeWTg=; b=sGYDlmuX5bt5smU9uC2EXzE/+1lC+nFGHxipPA00Qc/+SRgu563YXPH3wIZPGiaaEoD4oc CsrjDnRGbof2MtO3TlfMNSh7x9IeEkqAH3BkXvHw2kf1m2Nqc321JJdh1sh3Uqya/YxnrY Fk/kwuYc4QORspfZV7cNTzWTj1cjguM= Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38IK54UE005659; Mon, 18 Sep 2023 23:02:29 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=HVizojME0l9i5OjsslwdvJakCgkz9Uqrfyq+3ppeWTg=; b=CNk8QpNWwIMa4cb39CSbIf6Zyhvmn23vHkxlrGIliFgqjZUbVwtr64RNpDYWTiwD9FEm 6gzzJXCtU8IhfCx1sPdWXCGPLn0fFnkmvho1gKF4aAA/1Vypdyf8UQ7u6NkwtVVY7fi0 r8xQ8vi3BXhrANwXPQYZVhjVNX5Tw3cxI8F5Q2P7DrhnCmZu5Q2Fgr55zMjD22YDCiFI nVAb56NKfgsmDLQsWXby8VV7gTrpcgCR7E3xqAdXHjwZdrtch8zvHvbFkYEs6O+jkPc/ v5zvDa27FBIkTiItNErnk7I0hu0XNcePtiTPOVt2qOQ71nCNQST5teTYiZ/ei2vAIq12 Rw== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t52y1upr1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 18 Sep 2023 23:02:29 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38ILWfWc012091; Mon, 18 Sep 2023 23:02:27 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2169.outbound.protection.outlook.com [104.47.58.169]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3t52t55a3q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 18 Sep 2023 23:02:27 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cWMZlAwm554txPFGG4lPxDFq/USj/Q2ohagxmoRYuPQwEiqre8D0T5ugchT5oiu02Noqx459D+gN1FrcXHgbAUTIRf2Ohl3xmWMi5feH2RsHO+xFR1Y4A0NF15BvvVhEkLXfIfMGn/jrpksmHFftoIGFXCXPhoKycBNeMASg3Funye7io0xe8W3SV2fgdJcpd0HbwSmjYVLLQgBAoXDEnu9CHU32DtcYZaPIMRrFUrQ7ZhG0/YWKu8dSrKx6x//6R2aqwdThNGxToyubsbytEp4OiFDOJGxtP0upxBbQYbP/sEhXb0fxt/7SVUzaV/AkoXxT0Hhtkoa3Gpd8RG5oMQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=HVizojME0l9i5OjsslwdvJakCgkz9Uqrfyq+3ppeWTg=; b=m2/z1BiSljymn1lb6qytTNy0WQew1F82ON+lQdyTfNVvWBgh/1s//mrZlfES8Eh/s+KAL8Yy76c73wcyvhv6gWYME9a8Sb+qDXKnHwLVNXuzqLGQcQ+yXcLAoThDrUZ5YYtQOtHJuXloF0IQYw9a7qvIPdSU1oeOqUQX/Y2Ogzvb1VefkJM+pt5dD8gJlKTklaZNXHgmb6j0w/MfoDvvzhmTGUwRV3mSJvT9B1Tp+XMS/BGtr7P8Pbfa+qZObmzixf6dKE+P8ZKHMFAKy2Wyx5p6kd56hJ/7QqECl2JHJEdEmcVAvLgZ5hFYRn9RHWTMUsFAyTMAMIq/wIiR9cPX3w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HVizojME0l9i5OjsslwdvJakCgkz9Uqrfyq+3ppeWTg=; b=V8aByWVj+QA94h8D6eEKV5WA69JHW1hGdvTixQ83NCjRxUu4V9wOX1s3b3mC717Odnp/ni9gjxn9ensNFMpmuh9BCuwZhsyJtE+qYtTe7bv+ejm0Yv19MINcp65Qbuec7ABH7e1pLmV5mzf5ptW9qvAwV8YNxP2QFPg6F0+sEhE= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by CH3PR10MB7494.namprd10.prod.outlook.com (2603:10b6:610:163::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.26; Mon, 18 Sep 2023 23:02:24 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6792.026; Mon, 18 Sep 2023 23:02:24 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v4 6/8] hugetlb: batch PMD split for bulk vmemmap dedup Date: Mon, 18 Sep 2023 16:01:58 -0700 Message-ID: <20230918230202.254631-7-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230918230202.254631-1-mike.kravetz@oracle.com> References: <20230918230202.254631-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW2PR16CA0046.namprd16.prod.outlook.com (2603:10b6:907:1::23) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|CH3PR10MB7494:EE_ X-MS-Office365-Filtering-Correlation-Id: 6b4d9536-0c1a-4fc0-4297-08dbb89b522d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: MOZf/LfFxG5nO3hBzUzgFULJXxGSCuFexZoe4Ib7XBvVRoXwj45xWVQYhLYHtKtcXoXxboREJRW+VqxCGNWqKbr6bXK6cZRhZkVJUFHyUL4wbreq8U+jkouKB1ONuz4d/eqeV1qA0eAQ+NFnJHV7rg6LMBYGc8iPzx1AA9td/+EFTxK/jaOmf0pS/rgLwdfDQ+7SoZCKP/xFboFprFhVsUvB/beV1D2p3J3oqpKMqB+Cl3fy2Z1nrtnhM6uHLxHmQfwKjUE7qQEnevhGzS/eUIse8lkD2TERmyqiIpcjFY00Xsg+jQR6pdgD19IGJmnqra/Em03y4rz/n6PyXiaJn6Ucz9c7FYXw45N4UiHOCbgQo3KoXuvqnAvECj6vNFEZmZoomP2Ghy1/Vr0vrW5ShErH2cykWvcMOPqLdhOZZtlEKwmzmn9o7w1lVPthOXuF4mYBwb/1ufAk0VAvZ5PnpsUerP0iDWArlVsFc0JLZIxmC0J83PHVIZEA++461oKijNRy52LRK+ttJq/1cto5NPGyQW5jL6MDf/MAjD3YCgA7KVqLHHl7TTOe0gPuQg0f X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(346002)(366004)(376002)(39860400002)(136003)(451199024)(186009)(1800799009)(41300700001)(316002)(66476007)(66556008)(54906003)(66946007)(478600001)(6666004)(38100700002)(7416002)(2906002)(86362001)(36756003)(44832011)(5660300002)(8676002)(4326008)(8936002)(83380400001)(107886003)(2616005)(26005)(1076003)(6512007)(6506007)(6486002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: tRuQOWgnDO7mt50p3A82lgh6EodceLn9Vu5OvmugdKGCMey42fHoINv8Bf2u3g0p3hBjszSYSZhvp2Vt1fyAV7okZbaHZTOB4nYAZK1XanFI+jRiaOPPhaLwbyiuqQktW6HP3uGwa+YcmVlMjJ7p517KygHJp9lDGhaFY5MKqqvypLamjPVc4qUh/LrW6xhTqXMN8IEdIscxdXOJ+SlqNPQJ4qe6FbVaAs6/OCeG/yOdcY0p5gcyztBmP33CXrhe3jDaBrGcdA0wrN4OTRbKPg8YwQzcuuWij6a6P7VkIOvALcf4rx92Hhj0PmCe/eiVBEY3aWZ58Pbse3ADBhDy085K/+2wDJ2pz+78eUwVVC2ogjwLiyqKVtQ8rYR+DJf8b61HfTNphYU+lHrJDgTk0+5yYX5TqzQ1O3IEIGxi5wTlDzvrhvqOSiFuqqufXPsdseuVLyjgBKuRkSKIHL8wsU7bn/AigSQGpdjvVv6/aTjidriZYTMzc9iIVO9ya2OviQN4f5WwmAfWgLQYBIsHOjjGO2CLMmEqETXWOU1tsaHvdnxHp6/7o94PcKkXrw0tkwXaoLE2wMERAGsbcYSBi/7X2k8vr6N43iv+OSnICUlEIdtoCjiJ0f59SgGU1Oj1PYJpPJIIRXrEkTGkTNgr8H+DPu1UR1J7zawJDOKStlM1xXOnvkf5QyA7IGNC/CrHPJz98SEqTu6U5kkGkAjQmo/TsVUMmxWqOrbfRmK/77JjDyN+HelW93n0ODjLzb2+HHdvy6gDBNfxUURVPbCyIZoOltttBBi3PusWimise39vQwLgxSoBXs8Qyv2INXtPlYwfLTTNyNqWeVC/BoJvocCEU3Pgq4ZdMJ65XPTnJ3TvKhvl+GaWjDwDxtq+/Um2t1rwF+StuAUF5jUcVS815UACq1DWl3oRPzq4tqwcEIhkiW+ZpkRxBEHCbAjLG+GkGpzGUFL0yJWVeXda/VfM2gEyMW3Hzms0AMF5hkog0M/L6zROm9qbUqxZtsGgoXqY3iKbrHO2Bhf9Dmy1c8SY6RdA0LMX+L2+0xNGZGcjdDuU//FB1wlJYa3cdBuDm5DDRTQ01A2gUfhu3z0WnC4wI66B71zVwcN4I0ylGcse3W4By7KQFRqu+AvX6kJsISrfI/dnsi7vLJLIFRpcZQaeHFps/WjTwkBaxONzb1+R+EnWHZaBmgu+CIM8cWPD8ClwAuNwRkgPJU+yriXehaHyQ3jZt2WzW3CsVN6BxFzWS+M93mEdzxnI/d7DA6DUSqp5VOBDXscH1oHA+GVi38Ehnze0yNsFKE66nfTHKvVSLCEmM6Vq6WsW3cqrakAS66HopJf5WC3VmFWt7YhvjJRDatQfqyO2RD94u656ubD7X+c7cbwyXRZSB4apV0RVheDphyazshZJWxc+uD3oYZjpuCEAT8pSDNPBdIFudIiZ41CUWyNjfgj/iuSwxevUepAUiLyc8dGkSlxzx7lHlTHfSLzZxr0KAGuAnkdrLXkxZVrdfbpzmlwu9cSSIgh+djI2A1L5nMumfvXNhRge3WfXlnbeJytJMTZKsPlCCLbtxHHt5hAL/nqjKF0uxr9pEb/l X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: ZH1+//BY1BGiyqpcWTnWYTAMXxOX4WY/auQOsb4F52HgiFy/nakCAOZ/lrMjnw3VKkh3/zYmpr8PNEY5Na3nQwUbkUmHyhJg1XXfnizh9te+EQuPCKey/YpMf3AAbvHrids64zNGB4F7sZOWEOzu7Iw5oIlSFZnGwAE8laIdpVQBh6NwFZsOFWHa3ZP30gvoj6dl/GC711YziMtq1qVXsuwl81/grcOUCig8aCRIX7qRVYZ+Tp4ijarzR3tJbhr9BC4sU3DaWXho6i64T7tpjpC1BnFOSveCFW4emtl9xRQcTTqUnaM1Q4mTJQuZ7EOuFBcbO5AFmCrU50bswfbHQ+QTFDpAn980qI/udKZJ1rOJW1WjEhw5RVc4BEMInxT2TOuz/ctsDjcY28lRR7+FGV15neFzIh4ffCtennmfHglbbQYrv95v5CmH6uRgIKId/aAhZYpl8W6LouT4t2AJgDcCmgnXl0KZUroG88BJyErjCTokcQkerPsNfLhQ9yQp8ajw0glpUHiHc1Q2kIFIY6YmWiWLZYO7Ctviu7uo2ZkH5c2kl1VnOgHboVkdHDz8HcNgrGk/7Me0EdeeLFXABaTlLFzZiqKQcTU1t/hd62lfVo0CZq1jj8yxaqi3XJQAM9tYnHUtfv9JUr0aUst6i+LnbtuNr90Hafp+RUNRCSGP+1mamGVdVRY3NZNeaXn/dPm+iUbRSIZoYiPdSRZW3KkIQ1xvL/LhdL/UcOFwkHenOIc2K+2b9dOr9tWqF+nu4PUNUWCARRQZHfLVmbX02Hs5US3NsrSB+LaA+/PQgVdD/6HTwbd/8Gz4UdLtWzM7bk89nD0GfxVsIIF1jWTSucwMBToutBC5jpdvi4HC7je+kPSbgrn/6yXJbjKuPNfWHKDQ4hnLfwBRX1WbitGra9Pqv7VVh4y9A0sw+YXHhc08zuppEmIyaYpn8wwgCwfEobpLoAzH4ZobGIdMZ6JPMt80ahZfPrzgLiykbnT2s/0X1nyVfQoGgp1FyfqKbtNzD+DS4QcPCu8kQwvEEgtq76cbX+CC5fOkWEamxLhU0pAUeE3xU2bLFJaG1pnOdKVDaXcCZ9wf5q4avMUmygZpigjj/bwmtdCBdKGh5wLzrJFV8I+JmSMWaTIOc2fiPunB X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6b4d9536-0c1a-4fc0-4297-08dbb89b522d X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Sep 2023 23:02:24.2419 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: rVRBlDS3h6nPX7iXYNvpty+/kiy1dVO0vgHLi4Hr4Qp8tkUj4AMmALOJ7R7qU03r9ba/0v1IvdvYuCw9a5mr2Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR10MB7494 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-18_11,2023-09-18_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxscore=0 malwarescore=0 adultscore=0 suspectscore=0 spamscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309180199 X-Proofpoint-GUID: x6y18A2y5YUln9Kv7X7kti_xq7whMdX5 X-Proofpoint-ORIG-GUID: x6y18A2y5YUln9Kv7X7kti_xq7whMdX5 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 1E5F980019 X-Stat-Signature: xa17kudxb9oje433tq3yp1wajpc6rg4n X-HE-Tag: 1695078180-149791 X-HE-Meta: U2FsdGVkX18u4qRb0wKzZa4ELffDhMzv/v/GSxzMkdeOtWhlaAJVh3E5Q3lfTAJYZDU+uR1EimyQdwtz1eptJI7uFEEQ+2vDQcCl/62okE+nkWaKGOrfYDxZoPdJamUB5xfGyspsEzXDnZt42cj573odcE1Fa8BKnLn/N+9cI6S6xEF5PWAaYCawS/c6uGJOsYFsGgTGw1xFCuDytsjpksYXZV/VUCjuZBPi+VGdQZ6MVJsFAj3jp5wOJNahilldqSPxz8UcWU1M4V06fQIE9JKv7Nmp1rPEtDRfApVIR1ecytCkP5K2+TmmypEtguLYsHL9HOCydGkBd6RkxuOACgrve4PCYQom0x0TMywuuR9Ruqt7zPwU1U4cTFEUKe6bvAIMNxNJ07yrU5gj2hDUj2FWvv1+WDdMqSWWPohNVgeDuR88qy5m7x0GfLkIfhe0UvgAPzT327QgAJWr8Rb+vDVsyp6mG+7kPiak2p6xD5H10hv5yrx4IsK9i4HnkF8mzF92RlYxVvf4MBTLrKWOVj9IsLEqro7d4SlAX7jp1Rqacw1boVPlPQvk94Tc9U2Rot+OFvLIzQzjZ5LKcjd0xkTPLZPpUbL1XxJvGlrfzgdAvIBBF8XzSqGnDqp3261bky5zqYqqefkB5h5fofkpl1nAoE24PKl6zQ/yTox6rgNh7hv+ZXW7JdRRVS64LFfcqP+PCldZpDPtbjdq2qj8a32uxiKHu1we0KNSwJ+gdK7ONBabpWroaH0d4QJiT7BkIi2Tm9e+wUxn++qnA4qFuow0NTQTNNwqnQbn6l9pa0igNQ7dcLI2Z4aiEgnYAFeooAbw1teSkRzhug0yEAuFh4xdBsLurSyuS/l3E2v0bbnYVFoA134/ZNXYe2soHG7rj/1bpewUmQiSxIVVgEdsroaVzLLPT5aJNOHYOtbQB709M+YbJ29cMaqUlCx3VtoTYgAnnmVqCCFMvnXxEX5 a4+QdDRx 51fAxID090Gwf55LsIHFZGdW47U1Zj4cXAHc2ftMHG1C0eaZSC3KFqGtRAaOQ6s/AbJJHOk/NLJaEF3nuv841oDBGpRX/jtBw7jpUCczDxyY2A49hACi7cT5kzkY8S0deaNa19ontYtWP3/KmSbHp6Luqs+728FOMrMVobIZXF7a8cpen3VueaQa0RlO5/h/Th6CNxojPDXmC/LLIFQdv7rmoIYKKJsDS2BexIaNX6CtOEDvXgb2/hcNtCqUCoWdD7pOFTxfh+dNaJQ1vTn4dmQ45mEsUbseyAx1gAgGXs+iWo/xgsa357HcTlOohk9JCHoqNVSH9sRog7DFwtphcmrpIGkZNrx+DduIO6deKlPsTVZoxVUOWAEpnfLtHi9ETjG1x6LqGF5i/+zCMelC95hYugwEAB7aRVBbavgWW8u816qDyrIeOgpnIytVlM1Z0pRGuiIxLM03wrUkPEW3i8NesbuOzVVPYfyCIpV6tFiU0GItCvw5sIC4C5lPZkLd3aKNe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Joao Martins In an effort to minimize amount of TLB flushes, batch all PMD splits belonging to a range of pages in order to perform only 1 (global) TLB flush. Add a flags field to the walker and pass whether it's a bulk allocation or just a single page to decide to remap. First value (VMEMMAP_SPLIT_NO_TLB_FLUSH) designates the request to not do the TLB flush when we split the PMD. Rebased and updated by Mike Kravetz Signed-off-by: Joao Martins Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song --- mm/hugetlb_vmemmap.c | 79 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 75 insertions(+), 4 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 147ed15bcae4..e8bc2f7567db 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -27,6 +27,7 @@ * @reuse_addr: the virtual address of the @reuse_page page. * @vmemmap_pages: the list head of the vmemmap pages that can be freed * or is mapped from. + * @flags: used to modify behavior in bulk operations */ struct vmemmap_remap_walk { void (*remap_pte)(pte_t *pte, unsigned long addr, @@ -35,9 +36,11 @@ struct vmemmap_remap_walk { struct page *reuse_page; unsigned long reuse_addr; struct list_head *vmemmap_pages; +#define VMEMMAP_SPLIT_NO_TLB_FLUSH BIT(0) + unsigned long flags; }; -static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start) +static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, bool flush) { pmd_t __pmd; int i; @@ -80,7 +83,8 @@ static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start) /* Make pte visible before pmd. See comment in pmd_install(). */ smp_wmb(); pmd_populate_kernel(&init_mm, pmd, pgtable); - flush_tlb_kernel_range(start, start + PMD_SIZE); + if (flush) + flush_tlb_kernel_range(start, start + PMD_SIZE); } else { pte_free_kernel(&init_mm, pgtable); } @@ -127,11 +131,20 @@ static int vmemmap_pmd_range(pud_t *pud, unsigned long addr, do { int ret; - ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK); + ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK, + walk->flags & VMEMMAP_SPLIT_NO_TLB_FLUSH); if (ret) return ret; next = pmd_addr_end(addr, end); + + /* + * We are only splitting, not remapping the hugetlb vmemmap + * pages. + */ + if (!walk->remap_pte) + continue; + vmemmap_pte_range(pmd, addr, next, walk); } while (pmd++, addr = next, addr != end); @@ -198,7 +211,8 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, return ret; } while (pgd++, addr = next, addr != end); - flush_tlb_kernel_range(start, end); + if (walk->remap_pte) + flush_tlb_kernel_range(start, end); return 0; } @@ -300,6 +314,36 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); } +/** + * vmemmap_remap_split - split the vmemmap virtual address range [@start, @end) + * backing PMDs of the directmap into PTEs + * @start: start address of the vmemmap virtual address range that we want + * to remap. + * @end: end address of the vmemmap virtual address range that we want to + * remap. + * @reuse: reuse address. + * + * Return: %0 on success, negative error code otherwise. + */ +static int vmemmap_remap_split(unsigned long start, unsigned long end, + unsigned long reuse) +{ + int ret; + struct vmemmap_remap_walk walk = { + .remap_pte = NULL, + .flags = VMEMMAP_SPLIT_NO_TLB_FLUSH, + }; + + /* See the comment in the vmemmap_remap_free(). */ + BUG_ON(start - reuse != PAGE_SIZE); + + mmap_read_lock(&init_mm); + ret = vmemmap_remap_range(reuse, end, &walk); + mmap_read_unlock(&init_mm); + + return ret; +} + /** * vmemmap_remap_free - remap the vmemmap virtual address range [@start, @end) * to the page which @reuse is mapped to, then free vmemmap @@ -323,6 +367,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .remap_pte = vmemmap_remap_pte, .reuse_addr = reuse, .vmemmap_pages = vmemmap_pages, + .flags = 0, }; int nid = page_to_nid((struct page *)reuse); gfp_t gfp_mask = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN; @@ -371,6 +416,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, .vmemmap_pages = vmemmap_pages, + .flags = 0, }; vmemmap_remap_range(reuse, end, &walk); @@ -422,6 +468,7 @@ static int vmemmap_remap_alloc(unsigned long start, unsigned long end, .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, .vmemmap_pages = &vmemmap_pages, + .flags = 0, }; /* See the comment in the vmemmap_remap_free(). */ @@ -630,11 +677,35 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) free_vmemmap_page_list(&vmemmap_pages); } +static void hugetlb_vmemmap_split(const struct hstate *h, struct page *head) +{ + unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; + unsigned long vmemmap_reuse; + + if (!vmemmap_should_optimize(h, head)) + return; + + vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h); + vmemmap_reuse = vmemmap_start; + vmemmap_start += HUGETLB_VMEMMAP_RESERVE_SIZE; + + /* + * Split PMDs on the vmemmap virtual address range [@vmemmap_start, + * @vmemmap_end] + */ + vmemmap_remap_split(vmemmap_start, vmemmap_end, vmemmap_reuse); +} + void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) { struct folio *folio; LIST_HEAD(vmemmap_pages); + list_for_each_entry(folio, folio_list, lru) + hugetlb_vmemmap_split(h, &folio->page); + + flush_tlb_all(); + list_for_each_entry(folio, folio_list, lru) { int ret = __hugetlb_vmemmap_optimize(h, &folio->page, &vmemmap_pages); From patchwork Mon Sep 18 23:01:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13390560 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82361CD3423 for ; Mon, 18 Sep 2023 23:03:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EDAEC6B045C; Mon, 18 Sep 2023 19:03:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E1BFC6B0462; Mon, 18 Sep 2023 19:03:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 89B256B045C; Mon, 18 Sep 2023 19:03:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 31B126B045F for ; Mon, 18 Sep 2023 19:03:03 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 01BA2A0B15 for ; Mon, 18 Sep 2023 23:03:02 +0000 (UTC) X-FDA: 81251245404.09.A44A10E Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf15.hostedemail.com (Postfix) with ESMTP id 8E5A1A0046 for ; Mon, 18 Sep 2023 23:02:59 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b="oVsF/U9V"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=kuWXSbhy; spf=pass (imf15.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695078179; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=o88C+/1Bq84z9T8MC/ceJWbxnd4tBQq1TxgeUuB9kNU=; b=IeFDF00o38LDnagUC9dUcFujPnKT0QT8yLtoiXU8H7UlOd8SvMltgIsqGthgzbjxjkL+8e gbGvTB9Ab8ZIwjBypY4QUZhksDRVEZKQuNckRftkMMK8k4p4q5g3GaeHCFFuZDnzpQNKLC 7MzGelX89AEMbRtwkN73FvsjxzoRGTU= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1695078179; a=rsa-sha256; cv=pass; b=Gia1eTuMfaeLfF4hJYxvqUm8gyLyfdTnelaY405gJWezgkb8vfyrRT/we09Urzv6xir4O1 piBWU2t0nBkbV1QnnDiXnm+OCpZ2E9WrHhAwMURU84Vc6vIWaGhYpWvUpVJheV8S7vLAlm 6CWR3fdvMyOyvQWjsmbdNE40XjA705o= ARC-Authentication-Results: i=2; imf15.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b="oVsF/U9V"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=kuWXSbhy; spf=pass (imf15.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38IK3olX005656; Mon, 18 Sep 2023 23:02:32 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=o88C+/1Bq84z9T8MC/ceJWbxnd4tBQq1TxgeUuB9kNU=; b=oVsF/U9VZwtP3lYbcqVRXFsQz9sbzqkjy9z2+/JhsGBnYcfh85rXSHCKA5b31bGuK4Yk 984//fQ3DCAXVgAxnu2T500GOwKmmsHDqWFZl/f448gQa8OwjC3vnRv/Etw0JUXmFpbY cZDLrMwF17Pmz2BCwpuPYjqES51v7hijv54+YxzQtT2viZlq8NuADQzyMyr1rK0W/Qya MrRiQmHSGPI3jQMWaO4LfSjYdqDToKhPghTqOGBjMBYnyhSAy9+zJr3Oj0x49sH7wlf/ B3LsX0e99AiBE0z45rBYEx35RZ3KqxyTqGkprIYugIJeusLINqJt+GGwotXpx8bmPDAk gQ== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t54dd3mu8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 18 Sep 2023 23:02:32 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38ILxwTM027003; Mon, 18 Sep 2023 23:02:31 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2169.outbound.protection.outlook.com [104.47.58.169]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3t52t55y5g-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 18 Sep 2023 23:02:30 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FR3O223Wy0xQadkDrR2S0eRwZvkqwGD35YQnQJk6dMocWiGZOub3yloN3aGRiPrgtOcjbMYMox3l1SKv5EgB+zbIMXh24VzqnDXZwqkfB6xBk1LNmEgJw7WR+qfHfjgjUVGI+OnDdfyLbye5eHa5DSexvFmw5zaBI/DjNrue9ETO+6eo0ldmup0qNrCvNl5E8cZFnq3X68RPmrnhYOWg48gLh4nVJC4YTJSpePytFzzTTl8gMFuFSscQ8VbSdzAYXRUPimMdgBVgvXizRKltxQGGCyXEimH1SGgkCQoQDRV/nxQIO7K127t0GXFN1TM+wbBUsso28Hx+jfKn5+CDVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=o88C+/1Bq84z9T8MC/ceJWbxnd4tBQq1TxgeUuB9kNU=; b=L0/o6iMuhgyeaZZhenEdDLUvb8m0+nh9kSsQLAMnPHgOO++1vPdR36N96//B2WDQsVhyydPmgE1fC+WLUwpduP+oEhpRpp99yie2BniK7+ZaMFRam6VF0qZCkb+PHB98EWnDpP+G9absf0qJGHqGVMpuNrJv6QPiWy8tE2MADQ2CgXdDmRj6TxnUo7tU38JfIuo9z0qC6cES9ItaGkqCDOFgWpob/vKdys5ypJBoKfe1LyNhZIBc9NHLuCVNiVjOMPujNVea+cbIughnZpUDXEmPLHf9A63QVdlQ0B0oAIFDAEJ9qKFh1mHKIyxC3ZenWTAmkcDafT4rgBHgkt+1Iw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=o88C+/1Bq84z9T8MC/ceJWbxnd4tBQq1TxgeUuB9kNU=; b=kuWXSbhyPvoer7Iv6GndCuyfQch6RUQ9d+s6RY9gsqJdns9/kDsWaQV7tgCyIpl93dS9nuRbDFEXl+TVD0Wwc79B+owqI/Kn1hhRRcDoCFFVqbjR1cprK/XebKhGLCOATx+PCWu1T7CsN2y1NmFL6Bd1FUeBDF+OwLKYeDFndz4= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by CH3PR10MB7494.namprd10.prod.outlook.com (2603:10b6:610:163::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.26; Mon, 18 Sep 2023 23:02:27 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6792.026; Mon, 18 Sep 2023 23:02:27 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v4 7/8] hugetlb: batch TLB flushes when freeing vmemmap Date: Mon, 18 Sep 2023 16:01:59 -0700 Message-ID: <20230918230202.254631-8-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230918230202.254631-1-mike.kravetz@oracle.com> References: <20230918230202.254631-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR03CA0114.namprd03.prod.outlook.com (2603:10b6:303:b7::29) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|CH3PR10MB7494:EE_ X-MS-Office365-Filtering-Correlation-Id: ad7cb897-318a-42b1-2ef6-08dbb89b53d3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: dw8EuLiUp0MY0HbtlBI+E6qxsmQAMY4+s10eSYSi55ZM5bJ0NNdsiXjyGGcFPPI909PTaiNu06BUxVmP1N9AK4c9euhWRrNgI/CNoMAKoiHgiP6kjbhY4sGpxb61EFSI+yMmN5axcX6ixTgo+nrFsqtcVzvJVYA2YcOOVscDwhAd2mrnho2CwkBTgZAEflLcYHFHLgg//nGsCd55oYdyXNcFMN0Gj3m9CdKacV25oBHFHFsMkTiZm/iLjU/48tUyNX6DB7t2KSgdcXMvm1ck9kmZ26knK878HB4B/8cwtLuyvffyJhUs4A27RaYxqQUHF//afUnz1Bmcjq+5cS0vRUGe4eqEFtaYngCD8LKK4MEENhvvbZhBswgrhUD3OgXe33VTXOE6JHwU/9zR82A8FBWC+hLNIqueyXTvPLOnR0PEygySb7MKRTbQMNYA2c5fd40tlWp9O8Kk/NdIU32tephsTUJ/7NWCQQXdLACNKSFbraIrD8ThWXgHqfq30eLRCwtBG6Jvp4Li5Lv8MLV4pLAC7gyvaJhFWpZrlFnuKWIiwpFKQUZUvfNEpJY0nhaU X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(346002)(366004)(376002)(39860400002)(136003)(451199024)(186009)(1800799009)(41300700001)(316002)(66476007)(66556008)(54906003)(66946007)(478600001)(6666004)(38100700002)(7416002)(2906002)(86362001)(36756003)(44832011)(5660300002)(8676002)(4326008)(8936002)(83380400001)(107886003)(2616005)(26005)(1076003)(6512007)(6506007)(6486002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: AksU8VoJRZg9lzQp8HLUAlsXIoWwp+aVgZLPK000GU0MUq4RgQOlxDc+u+xD2hMneEuyGwSxL0VI15pDBLoeJJWJ921roKBA/RvdB8ViVLhEsTpQw9IhPzXP48885egdr5UYSecTs71uyjsdJNqS7Yea/iJuWku7cRKGegsl6wFAn282NU6SfFUccniAzwPYX8hMq0d08CLhrfveMjtwgxZaLvqvaLNl0gpya3/rSjwi6lqVeI/VCEnrwaFg5riGdqphXqOgVhSQhzDkjUax9jsG3v3HIU+2lFbMLnWxZQZWyjKAJWOkgbLQ9dziZ2qoKmZlm9Cok0Cy52MHFtmWhmlbb6kwJL8Vfn6/be1hMGlp+L6vr5Uut82Pu7MYIJKk6q1YAVJh+8+L16t9kQw03o/bU9nJlEaXIVzhQqPfOgTSOoJKydhn9SG17OzxFJskyra2Al2SstFKKVMMFmVs06sqCR+JEbDkwGlekiQXGd6nIOEEyuoiO/SXcWpHCNxKh3U7FNIx/pVNtFkAqNzncMmMMuPSK0YqbOmL15SrJkFKYSsKo6eZ7RwW0L047aJcmQuIztGWfwdChAm2YmOfolDSWLVx8eOGhKmiUdAZYG8m3Aog5PHoeGLEF4PGCIvYqow/xajcprSPO9zLXhebs3nwUWWvnpXUptoxSAEkp/VJB5oJGitc0LRWt9XTHVySczzRiSrO4J1nk1lWaj9CQxoLAWtnM+22gGwzUvKKl65ZDvAz41HC2bTMv64yLAELmEzAjPlQ3irGwKWiLwuA44YXUYeJJ9zblyJHgJsGn5jP8kmBpkGdsYmQBBtOioIdSmrsUkhgW/3QuqRlfRF++aW5RefflNyOl/PsToUnNX0sxQrgtGxW9JjAnRrg1Pn+wrRWLGIah5mpIEduiyr9TMX6F75nXdxGmIbIRGqgbqpjLPppgVPKB6DYGkV25En5TcNVF150H3trXpNRWXaLUAkwgJUibch0iPsvnEI7Yr1xP3f/6GuaIppohoknRBPWZsmKR+JB1I38+S7vUO7oy0sw6Jg8ezlzLjl1K3U1T11kJUWjwgydCaBC2QGhqV+Q4izXcAVoE0ZTewAOxO30b6zT3uLNU3zQbTTVPnAtDH2b3rBhR6YSDaZPJZdQXEHq7d5yhPA4xbXqS6xFyVTTpFmt1Mr94UrezsmQ304zR9DeRxUxQC+njM0JyL/9UUVdkIfEG/PNFvLg+H+gi0oqHTLuQGzpmBlEIjNrVtxwguVT7vtop3GnX8YlSYYIhcjjd7KmZ+kJxFYb0lHD2j+3znHG15m5andI79OTxViXMke1FO0xQmhjY9V0w8G+uSybE/2hoULEX3tXdt6F4XSShd6WK1iSbbViokXTEtFK5iVYO1cO5/9cyR2i5lCN7dFf4Qlfr6hyG57nlT8jbSbstNHyYXk3kRzHHql/yUV0jgPhbeDqpSBeFG4FJ39z+4ntxXOspnWYeN3dg5vJulf9WkXBc1uJkC9luhk9CBEPDqAQcQDPzWy4Oz4+9BOC1Gu6A7/b2SWaBRCh4cDZ74zEt8NS+OqkBJVsvfIXIFH3WSPeMQ4ipbo/tLb84l86Z32v X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: YjJqHFpPrtg2YzbqyJ2RvCeyMuqjyXOKlhXvgQcd6zh3tov8xcx9eq4qW73RcJwmnAALwr3coTL38Du77jlU+36ZwfXCD+RlF3C8RhDwviSGfLGG5gkQ9iUTqPzB9MKfULXhO9XoXM6FoaXMtLFvKv63nvjzNSHEcu2Jh7ZU+YmriLGB8dh8BX9f9wVguhaLSYAtLeNjaflRdx6UaAUoecJX4YpCkLVJKeyX429YHmtEmHKxvyAcJXjiicHH8tTn6yLspHsEiPeFeo7zUW2qnQzML4SX1mWu0ijsCiE+lTOvx0HsxzhzIZf+c3wgYBdkPfxNvccj4WM0XTlMrYWguImUCgofkLxBYQJ6jl0QHW+ysO92iKxweB6Ju89iAD/EQAs/xMO8gH04kFQRPA8u8zfGsqvhlynwQME2sMLYjLNFOwfzHXPm6LsLVGnzDWcnjQ0oIxrMOrRRwUp9gzQmcw2O30sC6Oqx2KNlXvJ6s2cbv4KB+asif26GiL7M9OkcxmD6LZwtn5CH/OqTeHB3aBQ/aZFurNOze061OFKhGO15J9Dz22HsNJ2mgJBkZK+7Wca/4Oj/oBNIXDCyQnsZKiStNE7I70lxfPq78ZMxwkLj3NIR0KU8CmVF3RWMqrUVclkG033vm+JuCk906jHkQENmkZi5eeWfagiDtmNmb119toPGPqKsf0/LFTSt2eFxjQREmsUBE3wLgmIsH47l0GURWA9HUPnNHDJV40sEsN4AgSUREYjEaYQiqGr6rIOY/7FZ2hZu3CE3Qo0gwOL2YBMeYbcp/qb+cKAaVDxWOjj+u94eeFgRYPp7j9/hAn+ullnMFmJ6v6zFSiKQodA2Y1l5mZU5NEURgmC9+DNWl0Jr/RilcUCpDzGwJbA27HRIktT82xLC1QL3VJB6ZBD+ogybGvEYolKJTBa1K3G/e9My5MGLK+AY28BJGc3Uc3ep1gEB2D4hb/tPoIGLJOqoRAvti0Q0fzJ509Ay7cg5g2ty5TlSnK283yisK06ErdwRiCnw8hIPjqJwGZeyUQstVNTb1VixZyjXCpkp5kzhGrsWgJM70FTRVrfFesoBfaDXtFJi6mOK/HZPawG0JuU31vAEIQhb8AbDBkYlCbNjMlwhf2XZ2CuKngV+h4DASqhe X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: ad7cb897-318a-42b1-2ef6-08dbb89b53d3 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Sep 2023 23:02:27.0168 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: fiqL7LWeeH9/lWKXM1bWewOjrLC5sWM77ZNGNwk6CJEy4kT9Kw1LS1PYw9VFkzX7OHSnE+BJ11t7yErU9k1PMA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR10MB7494 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-18_11,2023-09-18_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 spamscore=0 malwarescore=0 adultscore=0 phishscore=0 bulkscore=0 mlxscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309180199 X-Proofpoint-GUID: Gv-dJngzIin-_69pFzeyd21BP5yeilt9 X-Proofpoint-ORIG-GUID: Gv-dJngzIin-_69pFzeyd21BP5yeilt9 X-Rspamd-Queue-Id: 8E5A1A0046 X-Rspam-User: X-Stat-Signature: d3upixbxk3jn9jts1jpnyatk133mxned X-Rspamd-Server: rspam03 X-HE-Tag: 1695078179-94 X-HE-Meta: U2FsdGVkX19FE61O7163Y6aj7O1ZZe02LdYgoSiMlx9otMruEZYdZCN/V8ApsxtJ+WoF8N2vBtpkJg6KQr0pxd5MltQZFwUpO2pieQGOGUUUmKmwY85AuhLTOWEzkWx7lb4gMwNMyxuzj5Cr1Ex/J0Tq+EjMeGOLutypAGoUxnAIzntJ/Q1WgxNtZ84Xed9cpA3gneW0VMSPhcQsUR4yLVBggoyvPa2Q1rNbukcLcfozU13ehNS/D8LXxvtUAF9H8zkWOOJCq8el3AH/zM0xg+D1xLhLK8HQxqnSfD+V5DaAsAe2vblqRWyhGZGSNcacnstHXTpNRME/Z5CaNtLpHKhGzX8EGOndMvwTNBpvLeNIhal9MdZGiYxsPShYdvnb2/9zDeShD4j/IHxx82RygcYJ83UoQ/AD6JmNl/zOkVMAOLknWTpuhaHx8lb9xDmuWjYJwalaK1lGux9vqmytWlc0gl9++yB07KPM6WhLcL266tNWyxmzq4NCJlR1ldu1Z294vCdXu/6Asv8iJfB70gDCBYhI61y1EZ6QA0wGxL4WjAEAINjgIoZyqLZu18ql03qKCNFeT8XlqnTOHIn0t5stSnp5WY+yI2EGPLc+R0mOagCRF4MyXpT+YT1lQTe6EwbN9AloVA8Yp6foOATGvJ0/o7saMOKQ815fDhxPfC94xgzI/jOiLj99HyIn1z7A2hqptdk7+WRIqc0AciaQ3rJcuWPfyedjuwifC84zdQbvLUrryA+gYHgVxXbXWsmp5d6pqusTYxwNjp1Mx3QwUAKPn1oryrSg9mYV80J1O6+zeajoD+juCJ89QlFn6FpDK5z5W9SS/il0yogaLIKPkvpQhiiJvUGhMXrjIwjm27qddnSzCji3SPWMtO3BSnTVhVJyBBYhy9QqE06yaYGzNBrLQF9t+vJp2SzpNS8Xm092GPc57wZTBzjCE0InwR/BE1Qc9CX8qjgiOmcLFkw Vzm2PBlg iry5H8+Lrn545YRizLeDsDQhUHCCKfbQvi6YuYKjXiBC06gMU1V0de5CQnjs+tDxzf4i4jJOIwNizGBoP7rtgXv03dGogVHMl/++OXILnHYejRT3b1mJGTkuFsv+DQ+ke/iXyjuPRNoZSimGJfX9FLd7jchDPKoWGEub+QeAvxeTCWuh8dLi7NiiokhfQ87t2SSh/woPHCjRIEYbfKRoZv11XaHoblffwSC9lFQJ7xVg5c3P+lm2lpjS1gcUVFqBz+YT0VjCGssYrAhMz7GoSraYA4q1qWHM0Rb7OlX0lZd4WTxoL4td/1TGn+V71igEYBDEKMEaNrkMUYPf546d4HoDwmhGX/Ar9dC96d4sF2vgkQpVuAuuyHTdfYlQglXCoFlZT7+6p23HHo60JFHL6mCGgwvJ+dP2EpGzPoPD58viGEECyTwPFRqLdQmwKQBqmEL0v6S+fSkNv+K7m7kUWS71TGHZqHXSTnTb7bbC5GU2/O98O1SzSkFuMI7bZbdh3MaDS X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Joao Martins Now that a list of pages is deduplicated at once, the TLB flush can be batched for all vmemmap pages that got remapped. Expand the flags field value to pass whether to skip the TLB flush on remap of the PTE. The TLB flush is global as we don't have guarantees from caller that the set of folios is contiguous, or to add complexity in composing a list of kVAs to flush. Modified by Mike Kravetz to perform TLB flush on single folio if an error is encountered. Signed-off-by: Joao Martins Signed-off-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 44 +++++++++++++++++++++++++++++++++----------- 1 file changed, 33 insertions(+), 11 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index e8bc2f7567db..a6c356acb1fc 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -37,6 +37,7 @@ struct vmemmap_remap_walk { unsigned long reuse_addr; struct list_head *vmemmap_pages; #define VMEMMAP_SPLIT_NO_TLB_FLUSH BIT(0) +#define VMEMMAP_REMAP_NO_TLB_FLUSH BIT(1) unsigned long flags; }; @@ -211,7 +212,7 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, return ret; } while (pgd++, addr = next, addr != end); - if (walk->remap_pte) + if (walk->remap_pte && !(walk->flags & VMEMMAP_REMAP_NO_TLB_FLUSH)) flush_tlb_kernel_range(start, end); return 0; @@ -355,19 +356,21 @@ static int vmemmap_remap_split(unsigned long start, unsigned long end, * @reuse: reuse address. * @vmemmap_pages: list to deposit vmemmap pages to be freed. It is callers * responsibility to free pages. + * @flags: modifications to vmemmap_remap_walk flags * * Return: %0 on success, negative error code otherwise. */ static int vmemmap_remap_free(unsigned long start, unsigned long end, unsigned long reuse, - struct list_head *vmemmap_pages) + struct list_head *vmemmap_pages, + unsigned long flags) { int ret; struct vmemmap_remap_walk walk = { .remap_pte = vmemmap_remap_pte, .reuse_addr = reuse, .vmemmap_pages = vmemmap_pages, - .flags = 0, + .flags = flags, }; int nid = page_to_nid((struct page *)reuse); gfp_t gfp_mask = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN; @@ -628,7 +631,8 @@ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *h static int __hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head, - struct list_head *vmemmap_pages) + struct list_head *vmemmap_pages, + unsigned long flags) { int ret = 0; unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; @@ -639,6 +643,18 @@ static int __hugetlb_vmemmap_optimize(const struct hstate *h, return ret; static_branch_inc(&hugetlb_optimize_vmemmap_key); + /* + * Very Subtle + * If VMEMMAP_REMAP_NO_TLB_FLUSH is set, TLB flushing is not performed + * immediately after remapping. As a result, subsequent accesses + * and modifications to struct pages associated with the hugetlb + * page could be to the OLD struct pages. Set the vmemmap optimized + * flag here so that it is copied to the new head page. This keeps + * the old and new struct pages in sync. + * If there is an error during optimization, we will immediately FLUSH + * the TLB and clear the flag below. + */ + SetHPageVmemmapOptimized(head); vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h); vmemmap_reuse = vmemmap_start; @@ -650,11 +666,12 @@ static int __hugetlb_vmemmap_optimize(const struct hstate *h, * mapping the range to vmemmap_pages list so that they can be freed by * the caller. */ - ret = vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse, vmemmap_pages); - if (ret) + ret = vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse, + vmemmap_pages, flags); + if (ret) { static_branch_dec(&hugetlb_optimize_vmemmap_key); - else - SetHPageVmemmapOptimized(head); + ClearHPageVmemmapOptimized(head); + } return ret; } @@ -673,7 +690,7 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) { LIST_HEAD(vmemmap_pages); - __hugetlb_vmemmap_optimize(h, head, &vmemmap_pages); + __hugetlb_vmemmap_optimize(h, head, &vmemmap_pages, 0); free_vmemmap_page_list(&vmemmap_pages); } @@ -708,19 +725,24 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l list_for_each_entry(folio, folio_list, lru) { int ret = __hugetlb_vmemmap_optimize(h, &folio->page, - &vmemmap_pages); + &vmemmap_pages, + VMEMMAP_REMAP_NO_TLB_FLUSH); /* * Pages to be freed may have been accumulated. If we * encounter an ENOMEM, free what we have and try again. */ if (ret == -ENOMEM && !list_empty(&vmemmap_pages)) { + flush_tlb_all(); free_vmemmap_page_list(&vmemmap_pages); INIT_LIST_HEAD(&vmemmap_pages); - __hugetlb_vmemmap_optimize(h, &folio->page, &vmemmap_pages); + __hugetlb_vmemmap_optimize(h, &folio->page, + &vmemmap_pages, + VMEMMAP_REMAP_NO_TLB_FLUSH); } } + flush_tlb_all(); free_vmemmap_page_list(&vmemmap_pages); } From patchwork Mon Sep 18 23:02:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13390557 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 275F9CD3423 for ; Mon, 18 Sep 2023 23:03:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3FA5E6B0460; Mon, 18 Sep 2023 19:03:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 19BD76B045C; Mon, 18 Sep 2023 19:03:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E8F6A6B045F; Mon, 18 Sep 2023 19:03:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CE1CE6B045C for ; Mon, 18 Sep 2023 19:03:02 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id A9B01120460 for ; Mon, 18 Sep 2023 23:03:02 +0000 (UTC) X-FDA: 81251245404.09.FB38B81 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf21.hostedemail.com (Postfix) with ESMTP id 566E71C001E for ; Mon, 18 Sep 2023 23:02:59 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b="xfH/dLr4"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=LUjmyRYk; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf21.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695078179; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Tho8Epjh8mRVZoDoy8kl6Pit3DdeYADq24TrqH7Ykds=; b=Qq1g8MlAXSKPaiinvh9edq9zPKuZre0cFW+vCODimKN8W6z/8By8ccsuojhuW/OhU6RHlx 2+Sk9+o1BOOTJk4zzs22q2EdpMysygmioKcjT/bKsJNJAfRYVMXkFHjdxR2UXqwp5xApG+ 6ePJ+j229I53+Jz/NZ1h86VoN+M4gxU= ARC-Authentication-Results: i=2; imf21.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b="xfH/dLr4"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=LUjmyRYk; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf21.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1695078179; a=rsa-sha256; cv=pass; b=MISKpumcoaZ4+CDF8ha5sM7+N9+F3i7xMtp9rdg40RQf9/9qZRB2ZzZxeDOGUlqLH1k5ER 7zT9cuSHmyHKWT+mFn4Wy8fb33UVsa2MjII8Zgka1VW1ylCg5qsMe16iVEcqLFxZKHz9Ko ukrezgcFMLjaRl+tmjQooK+a4vFODPM= Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38IK3tGp022240; Mon, 18 Sep 2023 23:02:33 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=Tho8Epjh8mRVZoDoy8kl6Pit3DdeYADq24TrqH7Ykds=; b=xfH/dLr41eJPlIGv/2v9cb9frZrE5ij6+dFdRTWIA+WlD7bFDNwBcr3f2CbC/Ih4pQaO gdU4JkCAhzXE6xD33PU7Hqn490VOQXxeTa3yVaD+hNhIF96Srfhkk+qPI/xgsuIyZ7h+ bcu91M8zJk7FlNMbqbykjluHDboZqwK0zkg+jGWckMa6rK9VRsQ1BuAUzfnbSkD4NlPc qGw1WKmz/k9mWC/Snbgx6kxO8A2ldaL4N3Y5UQZ5nb2MYlittaEBdcCtusavHQ7WCxjV /9/XiEy1wkR20uAAV7CpKNz1vrUpPc7jWAgj38c17R9SpyXsU5JQDfFqPVA+qvBqeeqU OA== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3t53yu3m9a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 18 Sep 2023 23:02:33 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 38IME546030108; Mon, 18 Sep 2023 23:02:33 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2169.outbound.protection.outlook.com [104.47.58.169]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3t52t4w01k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 18 Sep 2023 23:02:33 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ImsJQcVpSPEZ9nMN6C/WQMG3N9M9xMY5Lb/0KlvFoLKU2hrtscIXingi87dkRJ2KKIbSqcabN9+cH6Brg2RHxV0h88eHMcCvtxvAl+qmGLJKVSIsFPKYOagL9eYbK+k0Q1aAp0Q4LtuWcdZ57q/2iSPbNybrRkmSC1oLp6nMBmuUF0Y9PVp/xStR1VDsaTet/AvhX9wTzz0GsxHOh352Q1QphI5rGmaHcibUA/RlEEZ0Vp4QyirpvrJe9QW1JBD/ZezCl5OsQvrRq7qcDcBBdoFzlIJwf9aKF+/rWFgaxX46rU46E9fkn5Jg9fxbb5KXaBj7f2Bm02u3EQilerB5ag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Tho8Epjh8mRVZoDoy8kl6Pit3DdeYADq24TrqH7Ykds=; b=fzG5TWBjbuZWFSkwehjU1GRnpLmso6NV8AzAfv/BIQ9xdoHNvMK/PLB7EGGacnioJVodJn96mg0KFZL76TglzjLOgHK+gV8DVKn/0Th6z7Nove5P47gJ+GkhIS+QPz2kk0aO1e5gkQEniGxm/nwmre4vkoiW7l7/CgDt4fZFLpCGgPjMjqnaIjbhMbUxWsutbeEiVYIx5nr4/h1YIkrQA/+FKQnQwV0OIKJplN4LAKWx/CGFbbJ7SbCCISJR5hO+RmYcwx3dXrpA/93oyqTO+NZqoBzdGGYU/Wk82NO0s1wcJvlN/AmmRwH/JlxRxfEQa2Dto4zl+L1Mdhz5u02wWg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Tho8Epjh8mRVZoDoy8kl6Pit3DdeYADq24TrqH7Ykds=; b=LUjmyRYksG6hzcPvab779EeUhkVuXA8Me401sQmvcBH1vzY8b3cY+7vigGxm0PO0OkO+PlMk4SLY5I57m6VOqZtj9kfL8ba3rcPwMMfxnoYGu0sTFc9/vfc+lkpK8V6MzBVs1tSK7XfJaK+9/fa1QExb0Qcqjq3rqAGWvEHeJYg= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by CH3PR10MB7494.namprd10.prod.outlook.com (2603:10b6:610:163::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.26; Mon, 18 Sep 2023 23:02:30 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::c621:12ca:ba40:9054%5]) with mapi id 15.20.6792.026; Mon, 18 Sep 2023 23:02:30 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v4 8/8] hugetlb: batch TLB flushes when restoring vmemmap Date: Mon, 18 Sep 2023 16:02:00 -0700 Message-ID: <20230918230202.254631-9-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230918230202.254631-1-mike.kravetz@oracle.com> References: <20230918230202.254631-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR03CA0095.namprd03.prod.outlook.com (2603:10b6:303:b7::10) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|CH3PR10MB7494:EE_ X-MS-Office365-Filtering-Correlation-Id: e119b49b-f7a5-497f-d71d-08dbb89b55da X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1ww9F8trYswgoyFAbQYH2FZPQljNDVjYBxjSUyvj2PtANnsS1vcX88tj2ZJo3Tg+LIpBirCUM/vx4OED2zqeqP2BFxjTsubg/pHfjLW+uSNYEMaX/8sW/Z5s0NpBSQXyXDcwm9pJT5zcHTBKAL5SQ7KhyTrdGJcCEaeppngtC5iIriSGtT9+NPUwEuI+9CqFmZ6E2OTcOclmvKuxC27m/BUgvoFuriD9f+3XXTZa28jsokVOq7sMQVaTxgw8ZEqZW7GXFHQiAJs/FSyACAV6ucRCEMlU4MzhyzdfeDr8mJhGQeNm4xxd9jM28ba43a/i3lf9Zm9VmkV1ZJgEFLz8EyohhqsZQxR4/h+ldxCaKrE1JRAwxtMSgML6K/Ch7Gl7o5RzcfLt7/judlrNZMJJ6dI9vCpyZBjuhhOTW5EOrblejkdnpYj2cx4K0CkjgamWM4pjMVdc2FWpoUt4EdoI+o+wih1ACX87e8QxOFHbHiNOBHnLMmjdBeyWFGVtuBNmp/AXuEiWielb5fmK7hFPESxkmAzs3q5lPSf9XLaY+6W0Fs2q5ePTQInT+pOJ7x8L X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(346002)(366004)(376002)(39860400002)(136003)(451199024)(186009)(1800799009)(41300700001)(316002)(66476007)(66556008)(54906003)(66946007)(478600001)(6666004)(38100700002)(7416002)(2906002)(86362001)(36756003)(44832011)(5660300002)(8676002)(4326008)(8936002)(83380400001)(107886003)(2616005)(26005)(1076003)(6512007)(6506007)(6486002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: A/3Zq+/zrbCtPuPYIH4+woZ/4ICems1T0AVkiTh7qX5aAIjvCFHxsk9O5f6kvGZ8ngySPN65GqAlNHVW18jI74/vtHUtA4gXjVo4sbu+GXJ4ioPKwYL+sEOypTcGlWdLA8a6YbJ1qvtyan35QGhJroNbc3nAN+HtgiXSnRNGNEMLmw43IGKiMFXtHE4xHq0v2sbalgZr0+1q/uhOtlrJtI7VBkxRLx1mAASShzNzfSD+Pt4k9TD1LR2mwSutcdH3wVnKxcIYism3rD9qWMBJyejYKvnJDAq3uRxEOu/SuUeLVcRGn1YT14J3+YNPUyJi2w2BjkNH5XLbwfJQxqRrxzDUDJTXHEk7CBlLcwgWGr2fb2VtmTgNdaPKrlqRdyRfwRkEgMqqviyFs4tPwU9nHqbHmzxWsHF8FY9j/UCiylNorpooYztd1TKJdtPcBdQSIM2SoWoyhez4Rbfs/lgH1NEA9o7gL0UY+GgR/RBKIJ4wnhOUim4s03lZ1BW/zHTPYxDDMWHNtq/DksDwEuhP7jAZsN5DpDGodeJ+G89D1njVIL3y4n01TtAj6TiDSXjwDSMevJYfk2fzlgmMzgTWwVOfAfdoRzYV5IAkQlXZG5otvBA+quIqQ6Ili+qdburF4l59Q3jnDYeR4gk/YJFm5VUvEwJuSgMNzwL+LNibVYcnGKpjGi+nrls6UioNXB5yvx3BnWvU3qYRluEb6NBH6nrrg3P551xpMDC6AYYVtnVYPMTwGUlRsm75k0DYuUBHM2Px+sHbkiFFhOfItyen9oL7sG4CVNKsTkh5xeh9UEASYK5DCy4ssSHrfpMFYcs8VB3dZ+IZc9TF/lVHBfk7ttFHWMo1nIgmLr1oDUwzT5yTF8osKDJeamc0KAxWviOtdlCmdmmMhohBKZk1vcVAOJFHkwTjzIej8yskd7uaEbQxpVEfqhtqf7Y+TdfigIBv6mzsnbPGCfEJM8UjFvQCMKLnzL8Hr7HxwGSIhSH0uOqUpO+OO2/U4xbU387hNBLeiVTOIN+Am8nC5XkRDrBtaBSRTBL2jUO+GGNDPldu/4oLi2WAoe1Wj7h10pEXsJuZvE8K/wfiV/0Y4umkCOBLC2FjAbdKUOoBVgV7dYZnmhv2c5OjcDa46UIiwd1ZBYbXy6vu0SOgEeQ2Klh4KNNE1ee2924R+tvwFdk423Y5hNM/B/d3x5IW83ClCbY3uJmfbiXrblGi5SKzRxJCaCL8uxAQTxVolpz9c1n8AjJSov1vbc8Z+QBWLqEZC30zcblF6cb+JvWTFPSi9dm4QTdiHB9up5wus7Zp0igYfhLN2skL9SsJ0gKogwsu6jxLm0Q5G0dvyqwDvwrJTIdvrS7NNM1hoxxS4uAvtmsDfkTl71PpJptNh+ihF8oPeD/+yiIX6NMg7+DYY7lPK6dA4JtG929a06VF6jkrWi5aS5X6Fr1IjguCnxAmyVgwIFQTKQ69SHRslcT2Sm8QZeezEPvlV88/CvyjF5IMXkVF4SCnZwUfzBeg61ErFz8D5W+pAuiXRqLg3WcwCYkFMtxRbrSF5T5nDl3n79G6wLl0kHmfmkfeeZkgCwjGHJRWUKaCSwCV X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: I7tNTb6N7ErbVk9GkwxwdQV62FGwnzrCdiKQ5ptc/s+/Kfd1bq1wMHqIPxyq3FcpvVA90w4AUCqAA4oUA5JD1q/FtPb99qABPap6hLGHtdF7MFCmSgJFzIWFORwoF4Qorp9sQqvjkowdGjng6KrcDGenpmwkWbqnnuxJt2zL7ymCle2QJlV1I5KTigHLMgqeGJZlDWkdnQCWj8BAZ9hou1zN4f41yKO/ITS974z7wh4fUk+u7OryS80Fz+Vf7mW+Em5nwdVKrsvyLTbad9HmD+eHneg5d6de20RVgdO7bkq6fS4BujxCjR7aC3GC2+TXVDrIjK79No3hpJxvRutIjagKYbTBpHxc2KHTlWzp/rCq45qwWYTBjqMfVoD73jxO0WQygLKZwrVKFwIUpGwq/RHkcijDvp7DE02wkXnOeBg1wukTmfxs5A2zOQqbq6OMTG1ONVUoLg8vzzlYdMJvF+TBMyfWdxMwFIlA3BBcZuX/RWs4MFi5uDAMi7UMpKhFdFzV/6446stPdJBtUorQTbEnHvg5x1N0WTN+MqBW2afW84sAj6POGDfU18fhUEqcjfN3Ro1W2hzL3FXbzA41iMILNu5XbtG5MR/OFkGoi5KYDBcaMyq1051KW8S91A3vvW7B7dUQUnfMWc6ZMJHsyWmZ7Nuzq5R/qubIsb1Fca0QR9JjtedbRtADqXqJGK+oTi4ClI9bNT7VruDLJtTNEgdCcDgXj9ZaKq/mjTH/Fu+jlnkvRwUwBIQatZy9fCsXYJVZeICUKiVuwwYeCXobyiO+JhdYU9nLEOPe8iXMkNUYZ7ZivUz5loTlDAY25ff2YnvjoMDwLWG1KSoAEWGwClNXvw0PZVJ00LFKn1i918bm60wfi8qTkQ7uNK43RemnORLljJ2Y/uaeKQ5S3spF3TU4V1VeIS5CSk42qIenYHh6qqzNnUGdaDBqnnsK+1bCliKrQANfCuc1BS8YzMmCwCRzD0/Gzq7/55Iej6iITQtXuBNWZhL1KW3diKBqytbIiQfSLwoIvNzd07veE3he/iCLIyPm8rwArBaDMEUbMPJv8vDbaD3z3hHH4biEenXrlLoQcDHYiji20SxcMvEDDtRUw1Zw1312b29U8qtmq/dVtg7eIIZJAOTfdNwRz2HL X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: e119b49b-f7a5-497f-d71d-08dbb89b55da X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Sep 2023 23:02:30.4115 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: CFbeR2xTi373hgAEcKQj1fjepMskh0Kxo3h325kLgCnKq00ETBk5Qmf2Jzihvy7Htv1BvW0JPgFN7oCFPwq9Kw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR10MB7494 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-18_11,2023-09-18_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 adultscore=0 phishscore=0 malwarescore=0 suspectscore=0 spamscore=0 bulkscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309180199 X-Proofpoint-GUID: dAHMXdPjXbZ81p97ODRhIZBD7Mdw8haq X-Proofpoint-ORIG-GUID: dAHMXdPjXbZ81p97ODRhIZBD7Mdw8haq X-Rspamd-Queue-Id: 566E71C001E X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: e59ed4k89ru9jxxt1odi4bbygh9zab49 X-HE-Tag: 1695078179-452639 X-HE-Meta: U2FsdGVkX18aBQoW2uQPJp8Cg7F6xZXzuRHd06wEzPhivNINmWTcPN/pMS+s3GefYCHGH4xki9WFVaNJvHgx3Oan970TFFl5uzxp3hdvDv7KcpFfsP5q6XWNtQrMz+peSeLbWPwSwpj8GomyelRFJJj+ko4Zlm5KFNq8e1QS6DqTlu4StILw119N1GoqgxdHfyMak17WsBkt6M+5iCQAHlWoLU0bcnUp6Q0JCTGk+RXtuo74rN5NLwL/ytfRLUUUYTKviIvtY56CXnLq/W4U+qLte6cJBSxsC8jty3dc6A9HjX5ty6/1wsk02ltZ/wlucbFY9eQxHGObfghTsrsBEPFrNh20gE+WLQRCpnCiO6H3aE+TibaRlcTHnJsByNdAS2h+Ea0x9iYsRiRpZFLBeZ2jMJPc9gzN5c0uy7WL4CHTmTuKVcSiqnearWpBAMNoDOQ5xBskgqg0Ew81SLW2iB5pgkG8ZOv363nmiwIVv+tEWL3m1KP+EEVHvmzdN170KCV17MKUMXTK52mYA1Yy/IUPQr2evq9DgxmCBfvmal/smy5rZ/N2vKfDIptCU+gnVFNAfhYNOUpxs/IMq9WsWz00Iy/GzwVFGhEnxBu33WezVE4A9tpXArDMdh5OXeiNUPteyqUEtxOMwp1e/1Yh5PA2+VBQcs0QS1M/2rrBrob06ESLmmY8ER1hWJTqSceV+z+U54M1uuIf29sXJTLFeQkrfbhVQn2qiT7Wy2u+kYKcUcsb0wqcO8rKz29Jd2TCT1Vy2TEOmbhgHtblIdoeA0H79+T49bOfKYhtvzupR1kYgC+p8dRGAhrdy0e70tZ5Tuof2jZB3DZLDXakBk3gU/tmw7/wpZHS7Xg1Us1XSw+VzprWZNPyuoaoz0LdmwGWnbFGx0vgs0oPPTn+opBC9z9gU9gyPFFSPPOS/xUuzo3PMLYI1GxnxFJiahoFIT1vSPAvgUsLUrL/WcbzhQ5 pTWIyAuf Jbly0Z2OoLqpFfJY8AD7iQs8B2v4mztWas3qkNRQAcvz7L5Fl9tI8wRUMKN3Kkdv1CZn65HErbT38x304sNflgTVfPpfiRSLOXekNquC7TRxGxRPEazP2GGy64JrJQZY9pxva+ZafvsYmmvWyUChblzqAuOmxKwTdD/bJjt7jr9nGuPZzMrfKuLRSt1CcvPwQnEdbbK0hAUMNBpIjQRP6MeQ0dhD+dOC7hwe/IHAGMoXE0PQfzvyEpwpFDU32K1hutEolnsEdOWmre2q11aB0BIxMobBPd6ePrVZl0yTAz52vIgYqg2FyXadpQ9jSeW5Bqjl0QbnZm34ZXQCZfZ7yPXDFr4U03/JPqLyCrw1HnUxPfHBmmsLb4A897BwmYlBuaWasH1nXwDJn3DQ3adu8+ao/zvmleCyQ/hpW4QI3CgEDwyPVuufa6m2FIsw9CT2YEwoI1KpP8psNJPKF7W5RO7AOnA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Update the internal hugetlb restore vmemmap code path such that TLB flushing can be batched. Use the existing mechanism of passing the VMEMMAP_REMAP_NO_TLB_FLUSH flag to indicate flushing should not be performed for individual pages. The routine hugetlb_vmemmap_restore_folios is the only user of this new mechanism, and it will perform a global flush after all vmemmap is restored. Signed-off-by: Joao Martins Signed-off-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 39 ++++++++++++++++++++++++--------------- 1 file changed, 24 insertions(+), 15 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index a6c356acb1fc..ae2229f19158 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -460,18 +460,19 @@ static int alloc_vmemmap_page_list(unsigned long start, unsigned long end, * @end: end address of the vmemmap virtual address range that we want to * remap. * @reuse: reuse address. + * @flags: modify behavior for bulk operations * * Return: %0 on success, negative error code otherwise. */ static int vmemmap_remap_alloc(unsigned long start, unsigned long end, - unsigned long reuse) + unsigned long reuse, unsigned long flags) { LIST_HEAD(vmemmap_pages); struct vmemmap_remap_walk walk = { .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, .vmemmap_pages = &vmemmap_pages, - .flags = 0, + .flags = flags, }; /* See the comment in the vmemmap_remap_free(). */ @@ -493,17 +494,7 @@ EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0); -/** - * hugetlb_vmemmap_restore - restore previously optimized (by - * hugetlb_vmemmap_optimize()) vmemmap pages which - * will be reallocated and remapped. - * @h: struct hstate. - * @head: the head page whose vmemmap pages will be restored. - * - * Return: %0 if @head's vmemmap pages have been reallocated and remapped, - * negative error code otherwise. - */ -int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) +static int __hugetlb_vmemmap_restore(const struct hstate *h, struct page *head, unsigned long flags) { int ret; unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; @@ -524,7 +515,7 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) * When a HugeTLB page is freed to the buddy allocator, previously * discarded vmemmap pages must be allocated and remapping. */ - ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse); + ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse, flags); if (!ret) { ClearHPageVmemmapOptimized(head); static_branch_dec(&hugetlb_optimize_vmemmap_key); @@ -533,6 +524,21 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) return ret; } +/** + * hugetlb_vmemmap_restore - restore previously optimized (by + * hugetlb_vmemmap_optimize()) vmemmap pages which + * will be reallocated and remapped. + * @h: struct hstate. + * @head: the head page whose vmemmap pages will be restored. + * + * Return: %0 if @head's vmemmap pages have been reallocated and remapped, + * negative error code otherwise. + */ +int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) +{ + return __hugetlb_vmemmap_restore(h, head, 0); +} + /** * hugetlb_vmemmap_restore_folios - restore vmemmap for every folio on the list. * @h: struct hstate. @@ -557,7 +563,8 @@ int hugetlb_vmemmap_restore_folios(const struct hstate *h, num_restored = 0; list_for_each_entry(folio, folio_list, lru) { if (folio_test_hugetlb_vmemmap_optimized(folio)) { - t_ret = hugetlb_vmemmap_restore(h, &folio->page); + t_ret = __hugetlb_vmemmap_restore(h, &folio->page, + VMEMMAP_REMAP_NO_TLB_FLUSH); if (t_ret) ret = t_ret; else @@ -565,6 +572,8 @@ int hugetlb_vmemmap_restore_folios(const struct hstate *h, } } + flush_tlb_all(); + if (*restored) *restored = num_restored; return ret;