From patchwork Fri Aug 25 19:04:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13366213 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E75AC3DA66 for ; Fri, 25 Aug 2023 19:05:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B9175280038; Fri, 25 Aug 2023 15:05:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B4115280002; Fri, 25 Aug 2023 15:05:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9BB31280038; Fri, 25 Aug 2023 15:05:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 8C21D280002 for ; Fri, 25 Aug 2023 15:05:38 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4C8BBC07A2 for ; Fri, 25 Aug 2023 19:05:38 +0000 (UTC) X-FDA: 81163555956.01.FE24493 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf09.hostedemail.com (Postfix) with ESMTP id D0CF5140010 for ; Fri, 25 Aug 2023 19:05:34 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=xGHrmpjR; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b="hI4Y/O0q"; dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf09.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692990335; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IaKcsxb0rORIYwEUogPfq5hXZgITbUVS8py0jJm+tS8=; b=QmNHvykeKRxrfXXChdHwV4dPv3ko/6fBO55EqUosBT0ixrSHmWxTiGvnB2VCBJcqLRuWO5 LMia6PUedzYZ9LwuQqq5uVnLAtJYXgCdq+Yz0DnaXs47ifpxeiYLWV8ICckc4L+vr6qmDg odphyL3VyXLRAnHr6a+fNNo91JV3CQw= ARC-Authentication-Results: i=2; imf09.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=xGHrmpjR; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b="hI4Y/O0q"; dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf09.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1692990335; a=rsa-sha256; cv=pass; b=AiSL0G5sBZDQU4/i8PjSWHKwo69PUY9qoLzbjcuo/8jevWFo9NHSnmvcbYd8/3D80tnl7a cyfSQTfXJYY1TPRADYksG3i+w8348kTdJAuMfgB30Bx7B6mOxnysIKmxlIsQKebST369nL VnMi5kYWsVPbY/Tsi4AMpkK+injvtqo= Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIEo51009220; Fri, 25 Aug 2023 19:05:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=IaKcsxb0rORIYwEUogPfq5hXZgITbUVS8py0jJm+tS8=; b=xGHrmpjRHuI4SZC9aEhUGA0vOyePfNynrPdGtR5O0oCvqwmF1se1t0bEu/rm77E7cJ0Q Dl2geZMi6P1Hndvel2G7wrecV1P/xxvrzo4gPLFm7d2lCjYP0O9/a5fBwl0qLzFMcnVr 46b5RqXXZvFvmjSVc2SvQcWWimQi5jC6tslA5TMFcsZEjcspRuiUWSQzWp8qhz7Ek803 MdFSacYj2DrrKolaH09A6XX82XG7eEVahUE/z3WPwmReyRy+GZSGPMPdlTWm6h3zhk0F 6HQZ1h0v5G/TVjo9Q+YnBSmRhoUNKJZl2NUox7KkpK+Jc7O6rTFhZ2Afr5cqT2ecLjx0 6Q== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn20df0ss-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:01 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PINZgD002154; Fri, 25 Aug 2023 19:04:49 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2045.outbound.protection.outlook.com [104.47.66.45]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yyg87c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:04:49 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iW5DcikoTA78u9+ccEbN2VYVGLJbVFfbRtNzag6Bn0qxGSX0BImdca8IAyt29D3U30Bzr5QdySfm6auaFFeATCnKcixEskHtCl7WrixtdLbW+PYUo8aHl1rlZmSGQEA5Koud39YBtzRwQdOYvXiHg3m0qhOU3Yuv3QFzG67f6u8EgE4jEuc8nmDFsYVuOtMivwYibViUsPYSRupTgPAuY0VynyLnCDhXoS8de8eHxJB1SuTrLuHHZWRXNcplOStxkx9fm0a2/zECzmPbpIMkMrfABVbG99cUde/XvRz3sUMfpHUpVecvyzj24SXF+MWKj0CvJNS/eWNF6tcV9QIEVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=IaKcsxb0rORIYwEUogPfq5hXZgITbUVS8py0jJm+tS8=; b=K1CkCPj7lh5kQlLA1ihj4um3wmpx9DY5RwYNs/zuxHCKE2EU4OBdr36QTgcnwghzqI1jESm0v3LE36BZkWZUrQM+h2LjIvavWiF3/iBt61voGUhvaokWvjb/UfVhWeaMESz+Irwu/2WAij4uhYPEjkHj2A9zfm9gJaPu7cyFWIfKDJ84gCPUNBkuuP/NQC66lK9C89jjRE6nxtiFuAJK0iC+hp7z9PU/DlWJVxTb2uuzCrkFdnyAgw3bG+vjwMeCycxAxCmdjxX1rHtc3ees235Dr0C71zapuWgA9UTQjfFzCDdyVQslOobyXYo/X9nQIPlLs0Um04Yi52JCg3dEmg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IaKcsxb0rORIYwEUogPfq5hXZgITbUVS8py0jJm+tS8=; b=hI4Y/O0qALFameayMJqP+ScXqE2PGpVnVUhrxPui+6KD7R+U5NFoSFrBxfluErZ7Wv9si+q6ElZiT5GQU3Revv2c9wMpi6uDWbHaL2+ttIJroKfAKItWVeCysoAQW8T9B7B7Oey132R0QqPrbqPyQl2alYiQwtJl6lMPvAeR05Y= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by DS0PR10MB6895.namprd10.prod.outlook.com (2603:10b6:8:131::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:04:47 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:04:47 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH 01/12] hugetlb: clear flags in tail pages that will be freed individually Date: Fri, 25 Aug 2023 12:04:21 -0700 Message-ID: <20230825190436.55045-2-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4P221CA0008.NAMP221.PROD.OUTLOOK.COM (2603:10b6:303:8b::13) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|DS0PR10MB6895:EE_ X-MS-Office365-Filtering-Correlation-Id: 95a40a7c-5363-4624-817b-08dba59e2671 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: yP3DnUyfzMA4oTljGSq/AmrA+/3u0aNjm14qs89BR7Mz6d3/040LqmaqMne8G1kl8fqgFqFXFGbiKTEBU2QmnQigbydAc6orUkFlIpGP9EAj7l+4Iuwem3lOc4xAxxorrdQQzqJ6T5gcSdHQxSTRO7xHaK344ZN1AALSCXXz1Pw+Ei8P/uCH3sDbNX92ODvGNgFq0cp7sez75JHfB38TFvIAJIypBO3GAvbmgZFnCIyX0Hh32Ih56pJCltPKbBVACituLkmPpIU0z+DTe4rRohWVMaGXN3yvkiwhbrEoUFk4cVmbNVdefEIHAlMFm5Ycxs3s9iOUKSRjFhoO3dFTXsv/bGJJHK71UYdq+RyBdid9+mjI0kn3BeTvfjLhVefFnXoJ7j8crNsvO9OOCRb3oDGQzoQVM6aTjZEVd/ZVKyPfYor98jnpfuMqibX/DiU5qtVDfJRVgpfn/jwGYUlPsXebI6hdrxa0Q5eNcWIOfxMfUyiwfrzF+9Oi+sb2M43gV+efRSRBiCkq/wcOctc9Em2Q9xyOtxQjO5kjPh9Ww7zHsRlm5KKL5rchdcPRYNKY X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(366004)(136003)(346002)(376002)(396003)(186009)(451199024)(1800799009)(54906003)(66476007)(66556008)(66946007)(316002)(478600001)(26005)(38100700002)(6666004)(41300700001)(6486002)(6506007)(86362001)(6512007)(2906002)(83380400001)(8676002)(8936002)(2616005)(4326008)(107886003)(5660300002)(7416002)(44832011)(1076003)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: gbYYzec3acpvkvv+xt1dCMgHomCJZpfIZ9VGWdtArL61Yi6jJbneOG3USNorKXbUK9foBJKcCUiL3nsL2AG9tTZuW/TxL/drExCxsnoD+OVtEy4sbz3FRGPdD9fur6M4DgiX1hbtGttf30n2T/LibvzrYHzqrZsz7AfyMGI1eilOpulOFKfHyObV4Hx11+1kJ2at6jpNBkfx2xskhHuY/ogzhGqofQz5/vpSYMrqUqU9NpdozrmBp5O57FvSaSdSenCQP11stXEbGgQnOiLZ7i7cOcKYsDFe4a0jant02TmLdO6Md1Bvih3r+jtuEKPdv9hdtw3ktxut/1HtZBhdfuIEzGDeDaR6BruASUgSz2io8peFRdzN8GL3QCDs+UDI5OP6IG5KTx6nI0h1yoc8dlOgBuBYHuM2iY0diGrmK1UbhbNCEcQNmDa6FkuPRA5PZuuI7BlW7dcaWGQHfJngoLUjQBbCOlzQbPGR5bkTjzkFWC5zb84mvBn1GoT0+CNFCgOBdRAr+LAkgc2PogTSYy7tiPJumPOR3t33wBUXEyDWrSi5sCm4JcIktkRf2HpcBj7r6lJoodgbabIX5wQOQ7pdc+3oThAHCLZOVOwrCzGOMNoRxcjSidfDZNilRzdZyXQ/rkxgO/tJ10HfXQPFbjQl10Otq/aJVWygzw/iKiSJOuAhRlX9nJhK6XpBPHiHs2PZSxAY7PrtqPzMqkmcAb0SkTbAUuXm6XJaDeLoTdnOPtMARBqMTqtxdhIWkIf7Mf3c8xXvsIGtNVjf8Q7ZwMPkiqqYDG9NbdtZkNldVRzgTbnS0nAia/3NWxUOchfyTr+BCWg70I3r0fQzAhShfO9ErJ7eLRn8H9LAaAn0K9SHXKpZHl2109AviA1MEac41D5DmCq2obtPe1yCo+gc7rXsXEFtQSmo3j3+QM+xLUx4QHx2/Gp/x7Ez10Gf1JNlDC2IPf3wDww/pZ537E8p29uEm4tDpNxUQkifIBHa3iE76GPuLsR6t/FGzuOqZwyG4jkABwZBYdJ8aCIfXQWxTDYZNieovcnhsPvsQim2G4Ano4MRcRlhvkmij40j2ljMmKmP1HhI7nNZhTmPmeh6uAPKqGGDhwrdH9hU2XiIZs0D8dxFvIRa7DkqWsLZcOQsuQQTQhT0MZehkZuttIYrsLS+beZWz+szbR6hfDSxbz/dePTyh+ry6CbrZiU9zcgeBU9W/l1DXLrHnyoMEWXDsrNrwa5NzNSzrW8Y9jSHf9LG2d20DsgTmWOMmYgBhdaWUZVYgKkjK2JFX4xclWNIx7308X/ki6KhG7g4uzVrYBssyJkaYUxux/QOZ7LNubBC65yKqcAtq6ANTu2DDPhLpJphMaDi/A0dG5VJXF8KdiA0zin8+4iPu3u0xDY9UZtlzDqNGUwqzyIj3kNwFSMwxQ6GhmSsS5MMBkBKEotfhKwzBlwiFxPYke7urjVRQWwQLHwcBAb0e5x9vHOgiv+UOj862HHUjbDW0D+G18ZUFXytcsHE2GmD2UVC0I0ZU222jlCCuCtM4PyUyV2yl/UeEeMLlGLk4lifIZLFcv7BYwxODEYRHTvasFRU/I7braoFRevoxFsDjT51rmB45zkKwg== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: mzIu7qk05iTRn6vT7c6E5B1QYvfsUZIyAeq6AUzsp1A7FLoTBCy+Ci3Sdb1SMaXhICJcm9tSUGGfsmnw8EQ0meMkoRvf6QWHSuLX66tsYjRqOgyVnT4LX+ZMfDe+lp9FMaNlueEWRiH+yW50WRyYbj1oQaQ9ZEZLcXHiM9eB13Uw5KNkSMCm5awsZmKnsCXJXWz8YzGn9VMdH6EPxM+Kf94+f0Lbb8MBI8r4gVKzlLwkoQ+hm0nXwU5TNfGP2T27XHmnQe/CnaRgLOilNcmeHEpsqR7WGNYkj3VsImd9FSA2yNMspvdkogETD0JHiVqczOdhMmYEsqQwpEZ2AcB1n6MpeEtmnbi6xV5aUTNHLMhOobmPh0LmpM3adiGG+WI8p18J4ytuf7rTHxFQ6v71Z+F/j5Yoh6AdZl6eI/3fYNyy2N0h4YTRt3Fc3CG9dqoKO+jtApOXQhXZYNp99zi3T3Vp/mggMn0nU5CnBKmEdT/QdhYnVuSwliTc1G601fvgXQl7UKUpN6viZf+piVxCd5AJOrLaVwGCIS3CIQjay4H+T2pVTVnvmTce69MbU9rO8x1+D4gAQPF3P3UEosKv03VgdrBoPH7tv9Sf+bPJSsNur4HdKj2uvXhzovBSBauEWJC+69Pmf75i0w6unMazqta6ZcrTg4VtvlL4YvOporNsSDpgZonCn33eT4MPvP/LiEjNeXDPTN8RpKCqGlEJVgdUPJMbpX2fmiEaitDFwGbAOdRsSRzPM1A7/EWy9QO+3X9SMlZBU0Ji+deY3IuTcfQSreSB0zC8kEliyRpE7kh5hufT4dyXFXsXaxsYFQ/BdxesP6NxPTtmSEbHHbDr3x6ZLndzISDaDGDYaPf6CcK+RjVCrrBeJIZ/OrYg9wHmroNeLRUhJpqurmdNUjAfCrVQLONAVqnCyp0mbMK/CdduXmCtxL31HuQYrfbRpfzNPeQ5rW8H7VLIQ1hbFS05NtIcWq1T3yTvK5iIUoqdlaKLFyirf96nAzfv8PffFGEQXszv3nDrUmOuDmJlsX/uCRzX2SSd9LemsO2ETPPNoBgxNAaXvdbnsikRxJCmkbBayfNUZ0ADrdgZISFEUS99EqW177+YUPL72RGcUFXeIq2NgS9c6xtgn68LCMGIOZNA X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 95a40a7c-5363-4624-817b-08dba59e2671 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:04:47.3441 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: tzXB29o68g/CAMe5RmHwZw+C5GlRk8ogcsrK+e4jDYuVSRtyyMgOp/18T1Q9LB12PbH+PndMRTKUD7AHYbqVZw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB6895 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 spamscore=0 mlxscore=0 bulkscore=0 malwarescore=0 phishscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-GUID: 3i6DDmuniE7-GNmQInzbcIkT2a_yHKJ7 X-Proofpoint-ORIG-GUID: 3i6DDmuniE7-GNmQInzbcIkT2a_yHKJ7 X-Rspamd-Queue-Id: D0CF5140010 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 9rf3uyemyuzw9ic9zskf4p38konofarq X-HE-Tag: 1692990334-981006 X-HE-Meta: U2FsdGVkX19bBJSBVU+QyTj/4UIH9KTddoX18sIpF19L9ecTKyInKhlLJMgv32SCVGWlLe/YpYf0Q2LQXhypZ4vHC1p21IFWwqHGZQcepsQMuXkY+yNql3aYjRLp0X4/PciBhLr5ChwpFQxSrdjMIzvMIkj/sMjNYctKecvUyMD3CFYXwJx9in3AfTM9EZgQjisqmFaHSBjRNRChc2/4jZ3Y+4f3/gaeGpcy5//SclLbIvYjcMWcNRkHSunCZ6uT3DEnlK/bVv7zMjte0ZXkMg8jKh3ZyPiouNPWjqqe+El1GoalkdsG7E8kv6y+s4+xOxCjBktRYef3YIekFG3kQHMkZlthUPlqY5pLuoILhHg4g05NZm7csbCE8qkZPJitUZYvoiKfBlZpy1FSHtiMWneVquMIuVkirCa1SeqpbV5xQVsBxqIF+g5BKmweoXp8RiImRJ6gBrMFddppa+Ok3a9GIAWgU4gJFImw/SasLKqH9okuQ9Kio1DTpQbqlMukXGT1wRdXV/DVcGkXok5SawHfMJb2TuVZJsALtlZ76dZpZVAnqbHDyklzIveg0tBqDKKeiyHka3lLy6YIz+YZX3G3r+7nOPK6tpr7eevhWbj5sg1JBoGt4gttR4qbd7YtAlwzXTzvoQ23wqdYsWU0A654aUE2tToH7VaCVSrr7/gnse5ZbLajjlD1by2uQaGLJhz0QIuxEeTZQgIAWxAFhAPiIxTksD5nZ7cx/BJBOAMgrnCg8wntdGurtqu056ihziEd9fPydN44YLyESA6wLWvL1gRUOqF92ocYuU4edilsGS8w0OyS82yrv4wAE94MaTqUMkRBRM0VkBAVW6f9o9QRoE1AmVaPpizZ144S47NTZFF5rmP82dDdsbkoIUfR0W0NL/OlX8hQaLs2Bokutr7zsK0PZrReypd0QE/cGcKg7JoHyKBmGJbWHAL+Ta0cvpkWtNUc4QvhQ2NEIVO PuUNeDCd thbbjEdFh64OgaAOa5ZrXFoqJ25jgKfmvQugxPgUbcX5IIfERYz4XLYZwjmY4v8VpU6m3K03HsKbBN7tshle+DT1yCfy8sy5ceznSj74Y2BhjDakIvhNWPNN1sQ+dwXdoqKlqZpii7IfcTfZTgUH0xTwoETuFRcye7SZX2RYFSU1Oo4oRamgxipgA4xS18/DZurDODMI+8DxpQrd7tTNm11qgev1WmO93PrEqBtO18Xr8RKx9FPU9TgjX1/NkSbQNvY7hD//lxsNcceFAaMUNe97eYdJ+Yvpp46CTH5xJQVjB9qTrzEtypqad+0NUj9l8Wp6JWGH1uWdg360I5MY1rpkSJ8BDYcl3IaG0119xNsP5VQifl0KOEHHfHa8vkXIOspvSgjvAw+SNqhhxulei4MwWsiLXrisAD+uN9RFMrsiRk2XI0G2OXjHQ+XPS5N2/mtpn7GXpHp/z+o5LWMLkeTPxscasXLLxiPHR3/zA8pIwfbcYKNMFoo8XHD4CjFbtnXKt X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 0bf70f27c1ba..ba6d39b71cb1 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1489,6 +1489,7 @@ static void __destroy_compound_gigantic_folio(struct folio *folio, for (i = 1; i < nr_pages; i++) { p = folio_page(folio, i); + p->flags &= ~PAGE_FLAGS_CHECK_AT_FREE; p->mapping = NULL; clear_compound_head(p); if (!demote) @@ -1707,8 +1708,6 @@ static void add_hugetlb_folio(struct hstate *h, struct folio *folio, static void __update_and_free_hugetlb_folio(struct hstate *h, struct folio *folio) { - int i; - struct page *subpage; bool clear_dtor = folio_test_hugetlb_vmemmap_optimized(folio); if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported()) @@ -1750,14 +1749,6 @@ static void __update_and_free_hugetlb_folio(struct hstate *h, spin_unlock_irq(&hugetlb_lock); } - for (i = 0; i < pages_per_huge_page(h); i++) { - subpage = folio_page(folio, i); - subpage->flags &= ~(1 << PG_locked | 1 << PG_error | - 1 << PG_referenced | 1 << PG_dirty | - 1 << PG_active | 1 << PG_private | - 1 << PG_writeback); - } - /* * Non-gigantic pages demoted from CMA allocated gigantic pages * need to be given back to CMA in free_gigantic_folio. From patchwork Fri Aug 25 19:04:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13366215 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D595C71133 for ; Fri, 25 Aug 2023 19:05:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E25CE280002; Fri, 25 Aug 2023 15:05:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D8C5D280055; Fri, 25 Aug 2023 15:05:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A08C2280002; Fri, 25 Aug 2023 15:05:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7DFE9280055 for ; Fri, 25 Aug 2023 15:05:39 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 56975120723 for ; Fri, 25 Aug 2023 19:05:39 +0000 (UTC) X-FDA: 81163555998.19.CFA7E47 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf11.hostedemail.com (Postfix) with ESMTP id A08D940002 for ; Fri, 25 Aug 2023 19:05:33 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b="ei0QlP/+"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b="So5lE5/0"; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf11.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692990335; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OlqrnYCf/SWJqiFiZ0S7D0zk7AgEaG4b4hT8Gbzo+AY=; b=wcDbU7JOLkDasq4dNBeRZ5O77DmvBUE3bJ6fi0ECiEcBgf6Ej90QRT1e/RfAAciSG422JV K5+87qFJFS4ZPVDgbTo5NJBEceC0mbDCBGQsScOQ+ldifFO5vC591+48H7dWVKP6WOea9+ dmLRlN5hIJxdu0uiAaMnjCqsw9ueIxU= ARC-Authentication-Results: i=2; imf11.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b="ei0QlP/+"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b="So5lE5/0"; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf11.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1692990335; a=rsa-sha256; cv=pass; b=marWcCUGPVaLf8iCzSXVH1Lm3s7d2/dVw79/76vzTY5cntndoMriC4z62+gtjrjU5FL9+j ohAjZnOnthcAaSf9+NiylLWLsUzEkKO9mvL9/M74qeO7DSbVQnI6B++wlwpoOrYO1f7pPn JA4bvWizop3MaPzAwbZv/gJAyYahycI= Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIEo4x009220; Fri, 25 Aug 2023 19:05:01 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=OlqrnYCf/SWJqiFiZ0S7D0zk7AgEaG4b4hT8Gbzo+AY=; b=ei0QlP/+liCl+Im+OJXOaqmMYdMvDMx5nDqljH96E061eLMvfbYZ/PxX+5QzzsjOgAAh JEEo+g/nFYdHKCRv+Z22gdLLEN+/CuNaLmdrvCumKWlTmBZHCl/9r/YXtNeq6LDQkShv fRoKJaWc48RsojFD1ktAWqKdr7oPkMYRjXbpdlZHLDO0k4MrY9Lgt4Wf/Uzut45LqmRs HqzaWMY5gqbVrec/PBgZ6tIhA+rBdDcl9XrfjxNS87kJuhJWYkEY8RSrdA9aGQHn0ezj evEZlbzEGTPY4KrPfZlHcIDzrpJ4kOp9QgnGImLSW/f/JdEVI27cppH2FUyV+Tojt6fF DA== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn20df0sx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:00 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PHGikX035838; Fri, 25 Aug 2023 19:04:54 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2044.outbound.protection.outlook.com [104.47.66.44]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yr77hf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:04:54 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VJte0QNpO8C7kl4nhXIJxycgoNmMOs5bFcppURIfA1qq3ml2kW600BkPUL0NxquaeXoEoy0VoB3ETUT91+rvScI0R+dwvaUWrDp3YD3qLqUGhyzQcEWWvyPx9MSJg2C5eq8JZ5ApGvJSpPYy8dNYGCJgHbuJlQgnPGuIjWdW2nsIMZcvt+So/Ej8SYg6M43ELZgRev//98BK+OMuaSXFD46tEIvjxiXY1i+aFMIf8YYmO7e/fUvMzLndV+u/LbX5J2cEyURGUOyRnV47uTZ/aINPMkUtdH2vV2pTVBpDoQtVKXxG367mnMzABVne6n/hwYn4fchUDa85BNXrMJ/SIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=OlqrnYCf/SWJqiFiZ0S7D0zk7AgEaG4b4hT8Gbzo+AY=; b=WO7uGu57sbCaKFXiUPYaF8rlHDkepte9SQHP+61H7kNq8E/WrSA1S384Zrc2E6nRt34jHSEblIkmtIjIVC9Y4GANvxi28QtXJoD2rJchDrQZkoo+EmZJjyzI5imFsB0Gk/Z4cFb30mifFRjgfONmPAlL4GGelD7CBV+pOjEXYcBDbG+aQdn5zfQfvshlpdHBB/tesgWiSefFMKwyLIqHUaurUt8rER94BvP2Q6X0MdfjGgr2lVTXgIQX+X4Pjv7y/7zeeQ9UJRT8Kdj1BBxCwnn5qagmNKc8x99UesmLCqtXT42JUxuLfMAQGbwnTxSjBpDVBcm+dX6ORHG822O5gg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OlqrnYCf/SWJqiFiZ0S7D0zk7AgEaG4b4hT8Gbzo+AY=; b=So5lE5/0YT4nQSyNfRykSNZfpIQqGHvkWD3KrdDYJ58r7cThwwmwY8Lz2hudWbZoKs6OMaNov3L6D3cgvKLT5+URnMv6GESizIA2osXpCsnnZSrsx+yl6YwBlKwtcSSY+LoeGqghopUYV+3MfGy4clwaXTl3858FWXARQRJLrJ8= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by DS0PR10MB6895.namprd10.prod.outlook.com (2603:10b6:8:131::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:04:52 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:04:50 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz , Sidhartha Kumar Subject: [PATCH 02/12] hugetlb: Use a folio in free_hpage_workfn() Date: Fri, 25 Aug 2023 12:04:22 -0700 Message-ID: <20230825190436.55045-3-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4P221CA0005.NAMP221.PROD.OUTLOOK.COM (2603:10b6:303:8b::10) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|DS0PR10MB6895:EE_ X-MS-Office365-Filtering-Correlation-Id: c2eb6754-8598-4f6f-141e-08dba59e2866 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: rFQROwA07L7FLFzZPKiExSpb8Yimk+ggo9TaCdjXGdCMBTV6yRktwGkVDOp9p6SV7QF1yc7VmAyfrYFDmXAgNyNqtZaS/Qz03ABJvxzd1puw8B9Y0hE+z4EOUWBcL9NvzMq1LtGA5SZknjUC4YfEy2eNTZQTIv6P8G4XgLJFJ9O10/x17kZdlBtjKO/KTe69ssRPebbJ5gfcOwCexcf+asUwoF/zCz9T+5g6DgoBFqOAWeueDdPK5I+SqMWqNUqv1DTwrhhlbvv5ft5pi52yfRUyOSI/afuitf2m3AR4bY3AOYvxqtuxuyhYKROYcU5jvgWJdpSmXmN1jvYm2ywU85M9P7uWyjMhaCFvIm2XPMhRDp9pbZSRB+UCmWdJ2HYMHrni/yC4S0xJ0L4eQqvtuAX/sNkiXstQp0mUi5Lato5zhpxEwPzXpmtRQftCh8v2GswbtHBgRunoYnHqxbwgOudOugJWgwhwxqODe5eMlJxo2+YESFCbIaL+547jh/i0UdCR8ChutF0zLGLTSi48sxfoz6/vmptiVDCUw0gmp79HhcJ2s3OvSEDdDd8kTPde X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(366004)(136003)(346002)(376002)(396003)(186009)(451199024)(1800799009)(54906003)(66476007)(66556008)(66946007)(316002)(478600001)(26005)(38100700002)(6666004)(41300700001)(6486002)(6506007)(86362001)(6512007)(2906002)(83380400001)(8676002)(8936002)(2616005)(4326008)(107886003)(5660300002)(7416002)(44832011)(1076003)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: OxeMLk9UYBeWy0q4+wFMYdUexxnmDfa9GZaIKj2Qx1EJ61G6G6pMIKkDiyI4vaTjGjAztj7EqgLQu6HnqDik7nmOH96ohcG0WTO2AZ9S/OS9L137Es4TW40/wYc4r21pCHX2uYNqWstK5FyT/1V06eMYMFSVfoH+3/+ebpeZHpkntYRTn2CWLhuKJVkHgTXtqKwDuIQtacaPYu9e/mynh1B142QKBBt1pdptd31UoRu906WhlkKqKT/BOeTbe/gjqZYup6R350SZgzNf6xYDgwcNLze89TbXoSZYo6DEunLgBl5vkLMRazqqrkIaElqWItKgl5QNlhAci6V7TMfa6/tjdnjBIb0J518flN0Bev5wbGf18fAos+Lv22geWIN5KKc9/H8+fEsvL3HQU7bAuzgQ2GjTeR+/FAXnCix8QvvGFxlCNICmuXGrhuPGSzq9Q83ldfIvj4zp0RhPQjI2kQb2XIvKL/z3ubTSwDAolX/VgT8WzNW95m9NUO/MvLjItliTr95FD4odvcZSWP/HnRffx3oUOqthoAkTUB6c+7RDusVKYIP+x54FJFjzvPY5r36Qd0MnvZmSyo3bkQR9vuvMxssAisngPn0uavhSMXTx4/yU70v4AnwFBGvpfHxpmfGh2GzZZDNJ+bzg8dEX+9gZuv/vogdSDwXRrtqejP9y0ILUggthfIns4jGp4pV5ExFh/4uJLAjv1M70PDs0l/8PpjmHXFbHpVxnnuOiFrCcvPN708lOcEmPR0KmuiHrrhuusnc1GDCZ3D+RwGpdnxlAF0/orhJnWhhsfPtEvm0txP1zSs3ix1MrvJ1vBqJ8RziVxuBfhpEbC++Je0zIzk92/17wG7FEOgMD9Qe48V/Y/+NP6Mu6Kpa3gWlItITjykmm8I4h3uaml6u1kYTSJBvRr/iZHsfvSulP1FSdKIILZC4jwvk/I89gS2cgYDr3SGc4chH3cCgQl6v89s0dU3X0xa782+DkRWcK89iIz5BbiCELtNRXlwzwak5YnWmzVFmTcP1xULD6Ni5ETOQrl4Ky8H7eQ1z+7Ru27bLxp8Q01fPXPHGVYhFaCV/kQhdd1Dx1SUCOGGgQLYQ+MFZodkSbs6ZCity6H5eI3Et4wrDCMd5qf6m1SYxmwOaIeQoYnh4aOaRlCLlR4ErCFGqfolGaTXbAFoM0hLpUlsH0LRGX+70xhlJCVca60nUlzldGIBWqhQ2xFj5KFApnB6KGwuhUQdI3Yix5EkaJOp0Oxilq8SXxcvyrXMHy/ehH350UnvSrGAGhD0MJRLFondI5RBYSpXPkSwcmx62VFz7IL2BaQtxSrDFJKx2DMABCphB19B51kItFSK+AaOrHyTzv1cbxVY7GVIzgsPh4/QicvifFL2O6pnlrVIW/f6k666h/IkIpl9PZsKvIvQf0cqmlO2fIc0+ClAr3VRPXSydntt7sJlHiKif+ehlGKQ53kwgTZuSvC+5Fe5Ok7xL50X7FnFCMtq1J47+fBGmnJdnbltjkRtDkOaoPiKp/LuqxqOgty5dOaSRVq/yQWx5fD+YjJ96cqZH88VSOm+BE7KMLPHRm933Qhb2PABPNtnWO9ajx X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: ccOTViSzUa9HtmR/xNh4+bgnKGrvIrBz9idEXsdlqDDDxGIBzTAgFMBuS6SqgPPsDPUTA3M1OJZE2RTNp/SJNePEDnFAZFTFZ5xjW7EDOsyB9xCRgr7/ghYH2DKVoJM+5hzU+tz0AxH6bTuz3wOb6qy9/A0uhGQefez7wGzWdMl40xIm7WmysrRgEIM1ZpvehAYSVsQxRRKkdkiLE0NdYEX6TSNF7z1PRsoRmftg+6Kj7YuhIA7iBmg3WCU13h7NSEKQkdz68V8nRgIW/aWwmy21lHjLzZhUemLaUQbiK4XVSwQCHrGRCurRctI3mC0omXiv3ELoPDNMqHMHPepB4Gm0ZsCgLbXl4b5mMmHNreFT6wKSAGyVNX7s/GyEkqZVy+pVQTsz99bq1eEzQbABPnSrxlfQou92FZnSMrAv57a/N9sWJjgjD+UtemGs+pYNMxNZfoXzdKz9PHHPxS31uAAiMwxV8Jyc4r75fWDajcDSMANPcusySccIu+hKTNNRlKbkL4AXcpKgRVdEsV00D1pMqMn4+fRlLugWjRgpRSePZYltUEFy3ANMB/N0ReQzxPIttXygZefLMf7qS9a3tTJZ+/kTsWoqKDzOAdABmLYVJMWccbLcUm1k0B2ksTKF6qxP34CNibINF1OQfJcQ5BJHNlOvKKAaDvv57Tc8hXbJhdNaRhyewuGRzhRdVUhG7NWihOe+pp5/Y6BntUpwmTOlmmq5sipcvTWcY46zQTK+Y8Z/zbdSiheLYI9PJtsd4P33aeUMh78bhnuX4KonpJb0EpSdLlfFg7fckw6KsDxRz6w8gDD2EfyaUeJJZLiW1m89x6O5IyOqnScoyvvY2dOrR3rnIQ2zvofhnOA65Y/BPWU9RpsV44zQMPIJt/A6/MGKG6bFVrHoNvGh9G83p/RCOpZS6t0JJJg26EGeEr0UPkmzvGnRrU8LaEH8kxbwNBCr+EsYjscOHip8dPjia/TuyQ4iTkzlSWY3IYAcgq+ji06Ss/IBnIPd2wEGUjnVQuQ8qn2KqERE9HgQ9PzZTGO2IEDCAaN5QGsdSJlYtJbZm/r+FRgdQ/ueBKK91T82BelwsspUavlcXBln/U3xWnDVzYx5SY2VW/kJTBZNjMEPFwNdfzKS8XQsQDGTl8Zh X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: c2eb6754-8598-4f6f-141e-08dba59e2866 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:04:50.8976 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 11krfbWWK/fRLYx4h8qG5XbsdxH8bVKXh/0SrgMH7+r5wvpd3KV2bb/vR0HEkLJxb+7rk+NMUsuNnBikrqP0Gg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB6895 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 malwarescore=0 adultscore=0 suspectscore=0 mlxscore=0 spamscore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-GUID: RjUyhRaDvbDSPDQPjxjvqQ7jGG8PRuKV X-Proofpoint-ORIG-GUID: RjUyhRaDvbDSPDQPjxjvqQ7jGG8PRuKV X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: A08D940002 X-Stat-Signature: b4g144dw18emfn6ixsj5w6nfh5e4snen X-HE-Tag: 1692990333-715613 X-HE-Meta: U2FsdGVkX1/eByoVPCVk0v4oIhPu1dzhbkbhKzt6YJfuEUqwPfE50IOJLTPr9mcrQ7Lqjb/isKj1UrViH3ZyAZw3jhraCOhFW4/1ipQlWRxnyc62y6ASHhOzmTy3cdUU3PpVLrNPOw08OpapHWFy0QhElfb3PUCkyCkaKg9PwmjR51xuBjdCbM9BJZtAlZeP+o+mfjLQ9Zgw12DZisCuS058Ta6K5D+G5FzwBuH80sI1//mwbL9uU28Fk7rgpQQp+E3j/ACgomZ5nF069kWMVY2Z+bEh60FWXYAJtqWnFooCC1tLTwKTR9D/SnGMKxCa8Ofgpg62G+LjWUzWNk7amOCJh9hdgarlAhxNZCpJHfZWaV/7ivKYQ4K7kMMoYHuDVG+wvK+FWQCVBmWkPgkUSgv463/OMiQ7SwvMKZkCSS4bMC0O4ueFD1eVdNV+eWnLEaplhiTCDHuHWc2IZFCKpwqYeFxn+/S+mSteH/VG4J4TFqx8wvNwL0LHYXA0m2czwfkwjWORpfJs/zvRwHUIn/g9G+e+m8Iv/qbUvNvOooJEK2MbqEj3P3A7tAxMOWiLH/FvUXvazeX5YeSE/9SvASeVmpzCssAobMRHUOZutq4sVALPY6TJc5lTsgS6HL7BdSIISDa0guxwrEhFGq0tLhKXp3Vqd99diw2+Av0tMaOa3dLqUxijuSMnn/Kx760B2JswuVNtwCB0Zrt66PGNN+ADg0fUwsO6Lca2Wd8R3ZGx5oYfSMgzyw6eOVI0ozBrmrVjc2Wg11UcWyCccDHcdXUgyRq0xDj0j2wH2wYblsKI4Ag7Y7sL4e+PlYCS/lYcYU/zWm1XC0W6VjeUvGFBEAIbMJqQTUbyd3zaq4aDccPi8dz9FJVZYtJDx4Kdvk+9P22vQUZ56O50gD41FbQl+pTNZB43dVJjF97fsqFb0H89OrWN6qVxjnBwo6inC1hEKwap39K/KEV2VowzEi5 5VGSQZBb D2usdo/4qTAmafPhLOcGhVYhVT2YoP0X28Pkhb6B44NEx1tg7gzIwc2du73hIpNH3SVbcRKFvNolcjzLmg6/lnIUGGF18jser3lei3wAnPBJMrBIYqnilOSvutiAgngK2MVQ3bWJrDRpQSclx3Twku8pfN4G7tLIzcwaXsCaDp0LH2SoT2KBCwyPI+TfDOeMMSZJxituR8gEESjR/hFY0wTPjaDNAiIjvq3z7znuYRQuMTkq8SYTJBOZScf2YATVFLm3dmQ11hKVb5XDgRN4SUy6gQbK8fs3nFIslhb3W1kDZJkQVqGU17ofQG6QjoZ6iPedlTXll6E965gw+gpLl7Qy+/pSI/ftMxCPUpDLDBFnMAtu5cKw69t6AhpDc7bNx284qJoy7ZrxOdp7jjyA2EvYfx89OH0N53Nr6WE+wh41OtVEIWUejinIHbk+YJIj/7Tb9DfcJb6wkXT04MBgQS5Fbm+gpnPyyhnAW++SYf5cJl/o6z5CfyWYj332P9OljZ0It85hzO20Y6luW6m8W7PmjMXW5TlDolqxuKsd/RxV90Mv/DrliKq6+c4ruBqp8GiUeaEItCGxMNIK8sLHv6RNVmsXFKmWxLCUmVwEU4g6e+67JrNnKNg+tF1AvECPeRg9G X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" update_and_free_hugetlb_folio puts the memory on hpage_freelist as a folio so we can take it off the list as a folio. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Mike Kravetz Reviewed-by: Muchun Song Cc: Sidhartha Kumar --- mm/hugetlb.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index ba6d39b71cb1..1a48a83846cb 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1782,22 +1782,22 @@ static void free_hpage_workfn(struct work_struct *work) node = llist_del_all(&hpage_freelist); while (node) { - struct page *page; + struct folio *folio; struct hstate *h; - page = container_of((struct address_space **)node, - struct page, mapping); + folio = container_of((struct address_space **)node, + struct folio, mapping); node = node->next; - page->mapping = NULL; + folio->mapping = NULL; /* * The VM_BUG_ON_FOLIO(!folio_test_hugetlb(folio), folio) in * folio_hstate() is going to trigger because a previous call to * remove_hugetlb_folio() will clear the hugetlb bit, so do * not use folio_hstate() directly. */ - h = size_to_hstate(page_size(page)); + h = size_to_hstate(folio_size(folio)); - __update_and_free_hugetlb_folio(h, page_folio(page)); + __update_and_free_hugetlb_folio(h, folio); cond_resched(); } From patchwork Fri Aug 25 19:04:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13366214 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7447FC6787B for ; Fri, 25 Aug 2023 19:05:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD1782800C6; Fri, 25 Aug 2023 15:05:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AAFBA280055; Fri, 25 Aug 2023 15:05:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F7D92800CD; Fri, 25 Aug 2023 15:05:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7B7A7280002 for ; Fri, 25 Aug 2023 15:05:39 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 5AA8D407DA for ; Fri, 25 Aug 2023 19:05:39 +0000 (UTC) X-FDA: 81163555998.20.EE250BA Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf29.hostedemail.com (Postfix) with ESMTP id D1F17120023 for ; Fri, 25 Aug 2023 19:05:34 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=2t+1O4bL; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=wV+6nZbB; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf29.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692990336; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zb6NKCA64ESqc3mqR+LOpdZU6XXh60EAfRdU796i4Ro=; b=B08hbdzNj2mps16LEk/9Iydwz0uaEYFgJjx9xj9xK90dx+BgB43VvgTQEAi5Q8t2n8UNAG MAiNoz0AHk6s/bC2fX4E/LjMPZWhASU7eBpa6WUlJfV5vKOMNoOUvuWiS+lRV4hPGs8SYp rum8bIkUh2ZACl7wd2TvJhkxp3B6NjM= ARC-Authentication-Results: i=2; imf29.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=2t+1O4bL; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=wV+6nZbB; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf29.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1692990336; a=rsa-sha256; cv=pass; b=kRd3KKg7mIbQ1s1eaZ2XEmU+pPxu+sMXwWqTiLtGphtSL/E5+vmlxYWIRKSUtf882oZFFQ WHfumeJpvzhOLu3BXVzY+GR0LinUTqt5Dee8KyJLWnZi2Fxkg8eySRVRH/YtZD8ebZX06t GqVk3Xd5jrgdOIoySPvCEvvRvEMQCCU= Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIESYv031124; Fri, 25 Aug 2023 19:05:01 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=zb6NKCA64ESqc3mqR+LOpdZU6XXh60EAfRdU796i4Ro=; b=2t+1O4bLN+k80Qt9bl8aoXFhuuMp9ZXlhGq2afXlv1gd9xeoJmb4MlG+dFYXOYomgqlh pd7rKQDWu4w9dOxZ/FZ7q27tc8uN3+WFx6gQTKk+WZYIJqtXEZQvQceCjOSwhTJqBSdR vEthCHAMekuHt6C65L48M7Qx/vFdBltTJdnkAHgwmoVYlviSM6H5EJXYJZrEMO93F6V/ MwTvuSuIFOL67XvKcpUheYgqurW3AjJ29fH8INRv0KYVxyQudGJVbvVLKLESEagaLAm2 aKkT/Atw1827JzDLQOl9FYVJ5Iriq2LnRGjLvLrzNUa6Axw6ZHPfVdvF0BsWVmRerCQG 2w== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn20cq0d3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:01 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIjPs7035753; Fri, 25 Aug 2023 19:04:56 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2041.outbound.protection.outlook.com [104.47.66.41]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yr77jj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:04:56 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BC1icQetrRL8CdwcQPbVd2yUqY7c8UF/mUdQ25eApLIfuiZ+U7QPW7uEUy+EPBxnVLIlZ67q0xlezRpjAPGNWS1+yuZ1eHgfiXBQGYlijdWbSc0HemBIVBGXEplqwq3fc6qR6brsdSeRiagUEU0yLXSf+nwOPyKPSfZ66p5DiY0LnW62pE+MQw7HHP3m4AIlNJubxR3PxqHnSVnX/2VfnrspbWpM2Krtw1oeCiU9J3wQA6Dd66n6BRnXdwWaQyeWic/RZ2lLSaEWp4cCBZOpbHFP/cP2IXbCcL8+P1qzAcmRf4OT2La+l4YSQSSv+BdIzh6xZKM63vZohtGKMCjBFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zb6NKCA64ESqc3mqR+LOpdZU6XXh60EAfRdU796i4Ro=; b=Dmb9c1nDCzFNX99cinMhXC1hhMxmciTIFfMr1U5AWErAdsbLIcY7C/GWMhMFHgAPrf+CtSyaxATo52ZP81fz6iYrp3TQCj4aTV/KEnSrUI3Y0uJ+N5UWlWt+SjkrO+TlFFgP2D5/1RMRsWviPPPhOkcly69E9Gt8hs4ki0LRuR99O+ywjqubgZy6CzYs75Mc0hZvUVph6FlemZD9xwGxxEwXUZJyPy/8LC6LuokKADNevspdtGXo/kZVR7i9xGIOpD+eiI65UIVW9uXwL2f5iM01fRQv1KIVO5gUymHEbhSiNxiq4rasXTJUtZhWXlmE2ZWeazYQ4QKDqepx706E2A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zb6NKCA64ESqc3mqR+LOpdZU6XXh60EAfRdU796i4Ro=; b=wV+6nZbBfTvov7/U74Whvg6bWvinIk/Wi8XNXGZUKPk1zZGKnM9pgoV7x7cVIXQnmkXByBllfoQO2zxH5RtOSPyrgsfphF7+97ARnRLmkZhzLlek1me1Scoe/cMP4uxyn+AUrGqEMNeJTRgvwY2wipLqgI7LH8mxSStdWrfz9Iw= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by DS0PR10MB6895.namprd10.prod.outlook.com (2603:10b6:8:131::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:04:54 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:04:54 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz , Sidhartha Kumar Subject: [PATCH 03/12] hugetlb: Remove a few calls to page_folio() Date: Fri, 25 Aug 2023 12:04:23 -0700 Message-ID: <20230825190436.55045-4-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR04CA0087.namprd04.prod.outlook.com (2603:10b6:303:6b::32) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|DS0PR10MB6895:EE_ X-MS-Office365-Filtering-Correlation-Id: f1c6fe78-d656-440d-cf30-08dba59e2a80 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 74PHp3nRoAJ6M9Fo0Pd0MN24HTz16HRlKL8Fq19+S5T8wbfui7T/x8rlt4ltVkoXMm0R5qBM9OXkIKLhHL4kRms+ltL0yg1Ds1Jm+IYFaXFBroO7dZ+AgRcaCjFWuc7va7kmcrUKi48KS6KlyrB7p5plz64QMsfdN++mIFjq7dP40Eb6Y1y4Ymx7WCJ4vNv+mYAcC3GPiAWV67la97jBaTeAreUIag6rM4hhYaqK4stz+L4SQLH+wW8TxjBY3hAIsBDYpJRjXOJjPww3U5JNcYpSBPF4eDqehcJh83FNvnb5wU7KseB9IcZWsaftf0eSlg0SPpa2TNGo6XgB5J/0bUeCHVWGrdollihAumO2iOe4IWTx+KxoddFxnpQWtJFpgRRlL2vupcDjsjyvU5A+eev6imKkwb0JVdun6yAvW7mt1O/YwKAG5iMXy8Vf4BOiKAAhS0eBZksfOlg7mwX5NU9yhNGpHly8gC/aDNHqENcJrgJuu/RkCmAvb/mSiAJIJOgTKpzanvmhNMubZbivC5KtEe8eaTw5njjGn/brCJPQTkBnP77HVoXH4bfv0yii X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(366004)(136003)(346002)(376002)(396003)(186009)(451199024)(1800799009)(54906003)(66476007)(66556008)(66946007)(316002)(478600001)(26005)(38100700002)(6666004)(41300700001)(6486002)(6506007)(86362001)(6512007)(2906002)(83380400001)(8676002)(8936002)(2616005)(4326008)(107886003)(5660300002)(7416002)(44832011)(1076003)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: B5TgNuPOV2zkV4qXRtkdJvrtWhUYCJxqgmbDZDF15l0LJeOkCUU84H7uWfSUiEtzqAxh+6PN1tFXeDjMeGDRJXH1yQmn61dErAzmTOgfD5PgAF+90thw4/7DB39q0EUvu+nquUgMlV0E5YhFmNQ3vrnBibq0RNHnKQZfzcRn5Y3XKRtz+tZp5QBqAT/vWPdWobDzVpRauIcJ5jok157r9lkxrrCNe+Ww5aoBgtSFfPsR+Z2eay80Wm81YXO2tNa7e0JFfmDI1wewaugPAYdV3Yn3uPnwwAKtUa52IxZoEJzeQdKzvgAMlI7uxmvdhAfwrtR4k9mTNwNx2XmzsTxSXaFOmBswLW6ePgKOq4c3zZ5Mmm+ls2KU+ooPqB3GgzVGwizOHVuH7WupObXX/g1Cy8btpjph25PJm+PWnSBA72rV7fMczza7VT3UCjcoJGc+3cKB/H74SW3Aa9kyma9Pmy9PyUI1Pcaxi4h+wviTLc8GSqCMSYVcIUxaNGaZAwc8YfZhEFoinMFZNP87x1v2y97k8TPgaDoXo4NGUtbACzlfQxes93pHmnJnbLcgjOXeXMLZyhUoN3D7cyjEXbVLVUNQ7HAEBE+yAaCDLk95zucl0/gzsOugqyD0jBp7VysehePNxYY50qAbnDzn/6k9AVoS101gFNC+6WsO7pE/7ooGqI4FHewPyFsbyns70U9ezzPysrLfa+UkOslbtvudgDzioTAh4XZUQuySu8Gt4hmFvo9BmUoReVn8FvU3c9d3hnFzaJWPhYYRF4A14mqRuMhq5P2EM/MO11yRFeBuxnetZqsUQWPVtD9z/3nR+3IiHvJVBg01ux1C97xmuDgggksJ9nfjjI+mFZr3e7KaJiKplZUN3j4+2CyOhsnuraoGiA0D9lUWxc29oOVcevIAYH9XcLnHjkuYAJE+cCnkkEl9aB9xnJE5hIo6e8IXmmQvVjU5z/xncWLeeXULz5hOdqOI9n96P3qXxEernNkbi8Pajes9bUFn57c38t5+b7fdSLg9ILh31yjMvaglfNh7vjC+g6bftvivu2ISXBfvHgZxt45tLd9sng22BAvCI7m5ThlR24KaZZKKJSw1/FDpPega3c38kmUgiVz9j+VccefXV/n4vp+BkKS8NUoytO5ytx0HNu9df+frR9SDkQ6VFufL0Hi/Y8iQV7ZdhmlU6Qjp8i/nGPKm9PGRgDbS9xvCJN7lR3uFHef0xhj+grXoBo4oFoS3trr5PY+VhiZSovGog7gVr4dFa62f/1sq/i9K7SSGQvFauUav/NLHqG8kjqW1+GyCVYHqzBF/w2l2iVqK4KN83jHHkegM/MTgc3EPbK9sPtTjXzzeFRR2qhp2fEux5nQAn5U2JH5GYjYQyw3fcdWj51/m4MPj/zqCTUSdrm7Yo0UCvqWNW9ZTY3rMa8RxckMuvQZs8RtWMA2eHNTWq7Bq5f3gKBEnq5bMmxnBdP7IeANeGwDPaiK0uhKIZGLV9blwSZoQwgx9hUqAauP8axFIgIbtWyosEc5J9Yv5v8/eiQ+E5ui5S0VFUA7xXoKAmQ1Fd0HhocwC/m79rSq/lXaqOdNAU9H96mPffK9L X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: XYNz8A4oew/fcpsNHxzr8c46jc11uMYyjGx9dpM86agOsu5ASLmyDCTYcB/pi9l+g/0XB9sXL4gTW8h/ee6P8zcPBUbMQgBK2S3olXI/s9QxKNoDUknXPSTnGvuATwrGUGW9LlPtFfnPd6tpGN20GGNEaDXbBVYr+4jB9bLamqfrFUvQm/5zfNqYiWxvzSbySjoR2GwtJBJoEg8Nx3H2+ezwYQMnE7Q2/fVfG/WoQ+PXcarBiefDG72ieXX38PW9aVu3K9tO5Hkioz9bKQbDmYaIE3839TxVxHlrkFDPnQeRJe0ZoXE13aZI0+W4Ur4VnfcAudNmBKRck2CyA3QpjugF9b1J1onGzWeM1W8tSK0Td8uko7d4P1n2W8dFXpqGpqOF96X7nzJj35oRjkyS7eMkIeJpd7IPQ5Sb2GBNiv27srNUZ9Y1nmet1VqadTT0Jo7APg3FNaqAATI32BQ1yg1jWEaoaB4A9ouJCgVeJSLJ91z98le8SzhObtlWaW/NAQAh34WSoFMt6uY89USofQ8i7AvemKKVTh5AaXkL70/DOkHaTkjnoggmiTo0z1aQlqF7EwWpy+ENOEYOgYplNIFvjomp/cG+67t4YKT1dQfRVNZ1vDbah6ufhbog79AaXnLtk72bzeQCaG1sDUa/741z0VkjCQE/NP1AvGoKUhrBKPZ6vrjmJNPDoj72GhkVLCsNHF+aN4i4nwo+KJSp2kS3/Mtdrsj3gcGJALuOc8VcrL/rEBuoAmX+dR7X4qD2HXWeUqAY4IyQdWWrYw5GeYRD78g+zn7T2JUZstBUgutEnVFGlx5ePEEPnpMKJaOAXW1WJsmMrfm7+iCGAYbatkHfKdzor/FUZBSIiT3CuUt6KLJKzeju/wPnEnjf7JbX9eoGQX60IFrBIc4JYfrf5tA0Jr7lyzXIS8jtLWCEwxi5HAm6i9nqzk7C6C85F6U9yQr4JIOG/gpJKbJp7fdrfTI9TBfq5uoABCOQYFxe2jy1rv/ZqmUQlAdMin4YnAOptoi+hmnwz2wwoOfFve3l9d64QoVDFqpyiDrFUAFBLkXfgWMt4vFe56PmbvW3uVYzeGW0LM04J8AFBnRKpLxh9LULIX/oAKCcH7mzUaC8kEO7jeexHsOHN1U7+nD+fVRB X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: f1c6fe78-d656-440d-cf30-08dba59e2a80 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:04:54.1109 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 4viqr2pRx2yxE1H0+MrvVG6QMsocN7EvySr8SP3OO1Lme1rMR87KC5iU+uskIteL3QVmMBimJyoWeqYvQuO8rg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB6895 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 malwarescore=0 adultscore=0 suspectscore=0 mlxscore=0 spamscore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-ORIG-GUID: 2OKrJiPSiCvmOy9oRGA8-kJw9qhUO_Pt X-Proofpoint-GUID: 2OKrJiPSiCvmOy9oRGA8-kJw9qhUO_Pt X-Rspam-User: X-Stat-Signature: ik3ywtbe7c4fdbcpzjyxg7eza7pskbo1 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: D1F17120023 X-HE-Tag: 1692990334-670314 X-HE-Meta: U2FsdGVkX1/nosqEdUlKgvVoIE0dcKAdh1B9ZmaJEFIgl+pqkW/ZBKl9ZpbYu4y3cWCZo/8nSwnivp+cngjSVfMvfdmJa+rfS3MHoPwAFPskDrsKeNYg5R2no5IlJfYgg8kjf2D/iru4sW8pBmNLDTnQlsoYH2c65bqPB8KKrMrrHi+B6HYJtNOs9/qSblruq9VjBt+hLUIParxrQlYkRwr1KOu6VCORkVYmIN1PN+7UtLjWQVoOT10wvfAQK+EHXg1KDWpm8cHtbtJ6sM8WFU2WwLKBPalzNr9v0+IkLPKLeUZaNi8JxH539eu79VpDPhU0oPkxWkpxajjQmOKYXAoYnvmPU9Hj4DEEDGOPNclWbck68BbaQPNJEBaSOsQwee22+E78juUzHE5AVFE52UR7Hh+W4+2HRjRwxvKIu3HV1afJ9096D+52iIIM9yaU0Q4w/QYec3yRFI8yTbIn8U61iFOL6iYkrNrUZIwCVqvGCX7jBl6Q3b46OHUqNt3IJl9oWhXbDSouIa0oAL8t08RrJAaO/VVDps63RBTEUsKrYarNGsTxVM1yxQAVovN58MaqiNj7QoxMWO+T9jaWC0rIDiF9Mm6jjc8nzI13+CY6voCKQBpeoJXCzlcUDAjG/uZcOkZJTDlFGSquL9JqkidGUAngxD/3gRgVS4s9p2W578qWtCHCnFAlzci72MnteSyvTiiqzO0ISM6yR+3eeBtmaqozKPWpEKCkMxfAae8TkcEysc2WvtQvyx4R5VafMoJX+Itq2usvyTvDHFGgpdubHOVXbDHi13ansLAxZCiASBTL6QB+n14rUUUT2lAFUv83hR64VGaoCNXjAMV3fVF/IJSS5v5C52SVekqzhRTKrKpMWZftDuLFjxswTF9tq3PEMqgWuSTcBxuuAeF4qW6mc+3DdcPpPWt5XXBinEurcA3s8V3qBmqbRANhv9riGLptI4K/zy5uhrZD5it mJckR5IR Q8xsBbLC5EIR3MhsgxaHvlRHx1Eutv7P4ViFf5phORlM9sz8LpFspL9glB0QPxUnTEx/YSmmsHwkkeDexlnQ9GkJf2bxyUXczkdLpjjkRIpbBZWIB72qGmEVJsMtIhWmEN0MXhxgvRnAbF0wmIvrIXADRIkvh4PMts272OfrBapwkjri3ND+5iMZs+vaqmnVqU01cBO8e6BcIsIyx1awM3qOrTbL1zg4LkITXt1sbw23pa7RO6BjAK4TiJZgiFEquc4JSgyFekkblgJI3pWDQfURnv0409KQTlAMjy4KYG7uau1Ms2seD4mCcaPosCzyHT4K6Pxcmo8hNcP9VsoVeCUOLZdzKTYj4sBLBmXX7PfjXI4Ja9yOJjA3vyzWznAD4Ohh6AeSAYHgznqlvKy8DBfEgtTU5NROzyGkp3EHMaM0Yz3CEs/JMx1JpEnaq7Q+GFo/4RtLI7J7f5XxkihMYxN/HqfqoJa78UVQNnKYcf6VAyJyVzBy6tHeVNzpY0Js6bf5PG6bi+/hbH9lhfc7f/WuFfxe0Lr0Gz6wqhfs3UdkJzfBti8Y9kOWP+KIryMKqPN1bC55+SZx/om+WOoRArMZEPGGxQWGmaf8fcWX6drHnlYeKsFYbn+dIKAhDVzg7nWW5iNtCyjVy0OBS+481x5Q01reGnvKmAcGjdMXohUAaseI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Anything found on a linked list threaded through ->lru is guaranteed to be a folio as the compound_head found in a tail page overlaps the ->lru member of struct page. So we can pull folios directly off these lists no matter whether pages or folios were added to the list. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Mike Kravetz Reviewed-by: Muchun Song Cc: Sidhartha Kumar --- mm/hugetlb.c | 26 +++++++++++--------------- 1 file changed, 11 insertions(+), 15 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 1a48a83846cb..a5348dfada89 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1831,11 +1831,9 @@ static void update_and_free_hugetlb_folio(struct hstate *h, struct folio *folio, static void update_and_free_pages_bulk(struct hstate *h, struct list_head *list) { - struct page *page, *t_page; - struct folio *folio; + struct folio *folio, *t_folio; - list_for_each_entry_safe(page, t_page, list, lru) { - folio = page_folio(page); + list_for_each_entry_safe(folio, t_folio, list, lru) { update_and_free_hugetlb_folio(h, folio, false); cond_resched(); } @@ -2224,8 +2222,7 @@ static struct page *remove_pool_huge_page(struct hstate *h, bool acct_surplus) { int nr_nodes, node; - struct page *page = NULL; - struct folio *folio; + struct folio *folio = NULL; lockdep_assert_held(&hugetlb_lock); for_each_node_mask_to_free(h, nr_nodes, node, nodes_allowed) { @@ -2235,15 +2232,14 @@ static struct page *remove_pool_huge_page(struct hstate *h, */ if ((!acct_surplus || h->surplus_huge_pages_node[node]) && !list_empty(&h->hugepage_freelists[node])) { - page = list_entry(h->hugepage_freelists[node].next, - struct page, lru); - folio = page_folio(page); + folio = list_entry(h->hugepage_freelists[node].next, + struct folio, lru); remove_hugetlb_folio(h, folio, acct_surplus); break; } } - return page; + return &folio->page; } /* @@ -3359,15 +3355,15 @@ static void try_to_free_low(struct hstate *h, unsigned long count, * Collect pages to be freed on a list, and free after dropping lock */ for_each_node_mask(i, *nodes_allowed) { - struct page *page, *next; + struct folio *folio, *next; struct list_head *freel = &h->hugepage_freelists[i]; - list_for_each_entry_safe(page, next, freel, lru) { + list_for_each_entry_safe(folio, next, freel, lru) { if (count >= h->nr_huge_pages) goto out; - if (PageHighMem(page)) + if (folio_test_highmem(folio)) continue; - remove_hugetlb_folio(h, page_folio(page), false); - list_add(&page->lru, &page_list); + remove_hugetlb_folio(h, folio, false); + list_add(&folio->lru, &page_list); } } From patchwork Fri Aug 25 19:04:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13366223 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C557C3DA66 for ; Fri, 25 Aug 2023 19:06:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 79A9A68000B; Fri, 25 Aug 2023 15:05:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7270468000F; Fri, 25 Aug 2023 15:05:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3A26568000B; Fri, 25 Aug 2023 15:05:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 17DC768000B for ; Fri, 25 Aug 2023 15:05:54 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id EE908A078E for ; Fri, 25 Aug 2023 19:05:53 +0000 (UTC) X-FDA: 81163556586.18.5D1B617 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf03.hostedemail.com (Postfix) with ESMTP id 5C4CA20030 for ; Fri, 25 Aug 2023 19:05:50 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=pACqANPj; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=gTWTMZna; spf=pass (imf03.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692990350; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9P3Cyny+mcYwz1jwFSE8K4E3sWShTXUh71C/hj9IWl4=; b=TKaPvI9dnpvYLZ4Bkb0Wo7L3YoDTDKHhVldJpcykMluuaOKG7eZSd/HSuhWhUMnJvCCtNn c9te5kXevOWvHZGAVusdttJdSFZ0N1l4gtTjPp8ao1TGe8NBe0aMtggIlQs9xhobn5a725 QiMfUxd4IKQfUonMwdhA5S+zpONkRz4= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1692990350; a=rsa-sha256; cv=pass; b=d9q3dMDK6JbeY/zL7DFg2h7T4OxpVF37P7VOPG722oMlksoDU/IpweLA13dW33JtGUzepV UGeIrB17NdJvVtaTEXo7yStV9BX7NYf1TYBEPz+hWvNVm2a1uV/s6tSwqFPqvQ/ROSpJCc Rd3bve/FjTw/R2dppsYbyeE+j41DaLM= ARC-Authentication-Results: i=2; imf03.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=pACqANPj; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=gTWTMZna; spf=pass (imf03.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIEepJ000972; Fri, 25 Aug 2023 19:05:06 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=9P3Cyny+mcYwz1jwFSE8K4E3sWShTXUh71C/hj9IWl4=; b=pACqANPjZGRlFsTq2VJ0FL80q/JHBrsJ+jUVMbK0ACpMXN/uDgnZeJhiGSHYWxUrJkDR ECeRIS9hkXx0lKBsRfSV5/JyzOYP1A8qeJoGwukYEewrnhQXpY81WLvebFEXjHmYjFp8 0plKVPUcYg9AWD07S+KRtz5AiKG5QBZ8iH2/esS36kT5bwkpLs4+t7pFk1I1dKViWSW1 G4ZIaE9f0LchEs7ztK5ZDEVsWutGn8/4y+o8yyjVMq97sbvjvLihwAsKH5PFM69551uL OHRYZXcD1YR8skBEqIk7SKOyjkr+QoUqPWNg5Olt1FOVYCRJdacxYMCuw0EjgapHEMTU xg== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn1yxq01b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:05 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PI417C005759; Fri, 25 Aug 2023 19:05:00 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2043.outbound.protection.outlook.com [104.47.66.43]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yuya0v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:00 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OVcPmGBj19r133Gy/VFs+hUe4ko9UR1s4FN2yjogf0RiIx8OMZDaSqBSvRrJ7ryKM5EbuLP7arZMMd7JC5RuRMkIjH0Pt44DiK4Y93qC3PA661SykBQ9yAHUyfD7iKAZ+jYvooN51mocNjeY29WPfqTYyBVLmSxWzDmQDc6hGgMpIFUnE2AugTxqwfXU03O+HCap/GXEjOsn0uxn97okBjynsqDNEzotA6VFJKVmFOA6IKNWDHRFyuc3ZpaMRbqnyMdEMFQoGLvxls6iBmDn5w1SDzCaco9V+yha3YyW413tScnN3tb9MhNILrQeIlJcgvFqDEtvcPwb8wDkpxBjxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9P3Cyny+mcYwz1jwFSE8K4E3sWShTXUh71C/hj9IWl4=; b=LVZZxllwW5bofTD1Zoh4lB9vxurlz+v0CVMuO6vlI4DP/lcWcgP01NDSdzBqwUIBLIKDJpYn6DdgAUSyU/T59xybNDwd+DPFJ36piPTdL7nL63kjv6fOUdB4D04eM2VtP+5TIYOcxGtwNcgD81Ua2eDzEVuHWbV0vG2msUnsU3c249wHKwELCt++6ToiMMm0Joh5zU56/Fh8xZa88XA/BR0qCrb31c8LhLosvi3SwMpAD2QkNifeuPq0a4mfV5kNaB8s+7zyQn45F1KMMNgt9mA843C7GtVOmnq6mlgxFJ2tQZ7HutdPv66NEF4RjuPu9ed5waT2HCS6KSdSGfBB2A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9P3Cyny+mcYwz1jwFSE8K4E3sWShTXUh71C/hj9IWl4=; b=gTWTMZna3DkMP2t2BbLfW+hTipfF0H8hIZuwzCpSEIMB4JGQxdGK+fZ9cVGrrbnbP0qFfzoWFCU90ONN/7BiMzuPCmtFYjsC/NgDVtARMq2zwH1HHJCHvxrayCCoQ9hYZpuNYch5Y0aqUi0sJPgKn9W5xg2vGOUgpVDFYL4/djU= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by DS0PR10MB6895.namprd10.prod.outlook.com (2603:10b6:8:131::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:04:58 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:04:58 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz , Sidhartha Kumar Subject: [PATCH 04/12] hugetlb: Convert remove_pool_huge_page() to remove_pool_hugetlb_folio() Date: Fri, 25 Aug 2023 12:04:24 -0700 Message-ID: <20230825190436.55045-5-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4P223CA0006.NAMP223.PROD.OUTLOOK.COM (2603:10b6:303:80::11) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|DS0PR10MB6895:EE_ X-MS-Office365-Filtering-Correlation-Id: 0b3179b0-4cfb-44ea-5855-08dba59e2d10 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: arA0fqYiuagTDTT5usXxz3l50Irt/cQTgoVY4737hdO3oI7Txl5cHL1H2J8PEPse8WtibGhylArf+DwIibQnVqPVbXra5zFm1mCdIQ+KH/u/ZMB55XE+VH5xA9xUj/G5BLU0xx4yrMwn1tsQ5sfL3YaaRcwZPO4Giwy5H2waqlwWT5jAK1Asm6NRIE76VBSmo+j9YKR3RsSvmPNc3dbyb6Aq4w01U1uyIPs9ZtC/WJacn0wpQ1tgrbTFsZoTtIQnT8H5WobGoothiN2yWGZEEsDh0gCuL0bcpgJQEZyF8ysWZMRm+8elP1BXH4sm1Wcgx6oCqkiEtCPj3sTqPQb+tam7eQLGZxa8/yku+A8QHi/dZr1Wbafu1rBvCG54lzBoIJJ/HHKMoUQ6PcTCQzwdN4MYLJw9k/e4z71stvZ8RXXbubMl9StFt/I9TB379iwl8JzuWWJfVsRGD317IYCMmkk5+JWvXZwz/6mlw12+bNHz1UL0MIOnaQ3xndfY+88DlsGdjTlp9oQuIPAAlLZQ6pyHJlzbsONjoo5ca3I4ze2Aua20CJpQXyujSf0fXVGYgo9tylck631RNkbMQz2swK+gXY7gU82S0BUjHXxTK+k= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(366004)(136003)(346002)(376002)(396003)(186009)(451199024)(1800799009)(54906003)(66476007)(66556008)(66946007)(316002)(478600001)(26005)(38100700002)(6666004)(41300700001)(6486002)(6506007)(86362001)(6512007)(2906002)(83380400001)(8676002)(8936002)(2616005)(4326008)(107886003)(5660300002)(7416002)(44832011)(1076003)(36756003)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: sDyjABoO2OnZ1+8Bc1YJhA4ZcA6z6Vq4x9xG00tC+QGE3DtXcS6VJgStCzzBE+2mvU8cNggcivRi00WDaQniUtM9njpPIi9LamKsoMHxZZhB1I1AntTDlHI5T0idpdXrhicXJZTNT3GIgHHRfTlxuCy9zbYEn15UnQ0WGp8HXgKL7cuotHJcD/QF6lIeyGAo/KE+33zLOJgV42rAgLr7XWZQDHLrhuBDyzMwF3YBUGP8G7BSJJ6rtFT+FkPKhKGsDLfhxAlbsyxsBkuIWvC2JiYZKZQLhGjJ5c7Wgpz9/OXw7c7MzeTpo1F5/lXbSalnrF98B1SW/+J8uVm0sklaJib37P1NJaAM4D/zaOEhr4dt/M7B/SQ9DqUHA+HYyLf1B9QwM1rEMJImy5nEmkWJZIpZrekX4KbyNRZ6n00hnEh+23l3KyWiWidlZx+Xr/P5zlIB007CSTpgfSlW+rHQMESlBdnEvEAAtxo3SDwqojCYHYac1pN28uVivurlp4aZRgkZgNPRkw/+/nTJswNQIsymqyqQFDlZsNmSiA3AqvPiGHMRUGbPAnDFORXvRSGMdmFKmQsUw06w77CM6cj3pD1U5hRhQpDYAfwkbUBzCR9qVe/xbfJigN+10pPmK3LzxgwqjVsBdnb698tlOvxslm9boiiwejxfjR0aXdgJyPmnvvDswvcIzqstFlGKBDnFH612EARUnfbKM1YmblRGPYzr6vKIl1f3b/BsotP4QNJaOfaQPjWaLa7zBq2Tr4QA3sEdgHlA7aLPLQZds4lm5+cjQR/joNvKgfMPL8nfVlTBkwORL4MbWzSmu+xPm6onO/YRDooSvO2PDDAs3E8h1WfUFsnzehQ3jg33Yzm24xgvdI3u+pO9asvcZ0vIDmOvPaLnyPDroe+dfm4DGWzEhanUwPCuysms7Gh/r2kpWm5gy4Sy8f+D7FQvVYysTqI4W4WTMNJZYOQXfBcyhdk9SM6jS9TTB5sLJ2tOyeDgE6NWZzoiaHXX6GhHw+EVWaQpQYrip8RI6RKE/mD2VsPkFf8DRrsE8uoo/Gj/fnwbvy9XRdb8ELgnRucpd1sQoif3AwmzhgACQWp8s+TmTH6e8+ceMtwqv+dt9PuSULHhYe0lC+hVTNDpovuz3Ay9zWY+21YGFbiE/ZwUG2NlHp9kQvKl8bdOyfQGgJuus/EPEK6N166k/nYKkXq+gMajsIXhjHs2Y9GMm/Os4y/OalltRKskNNOWr+FKHvw+Zyyjr5dVRK8iTSPnabTHpeCO0yHhrRqzBYAXHTEESzyu3P2MrJKsmFW3kf33UcWDdaFQFDRk7wrt9Il4mt8fS1T/l22T09CAQZW5fcdDKhDbFVFExZx+Xdv85cNkCdtbKOQud4PdmVJbcWPCiqoHDJjQ0x2lfP8HtdDbaFRVIziCWitvKGdH38gPpmks1v3V9RVEhtyQEijKnuxl8SW9khBCfvONj6kDBnfvqdtAIkQpsTTTDm/kkluNc3Vuq23WQvE0AX3D5YLTTTjsuPQTPD45S+J2lUdkZIWjycm396ueQkf7BJaU+diTfKCi5WnlzNRHtNGFT2eo8w7thasqW3QgNUapDWqCOG4zlXmY9N4ahuxBqQ== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: HoY4neH3eiGkzzi7sZjMmEKwYVrOQY2+tYoJdTx8GBqT3RwLeU0tZ01mbR88VxCHxATCy68hTAe67VrlfTQDN465oezv+ujqu/EJXRhi1AjUInRUkj0iJuZbTADUNvunp4E6cudQsuOLw/tXNUoTQCYrWSUZy58xV/KGbG5O0lNYfb67ruVpwioa8QIaLT9PVl0ZmMI7dXPFpWRq1IXeg7opRpwmbmTlUo3OdPdNW6f1Qz0tNUtIeV/BG7bEm3onRG4yeU9J+AWwP84wkYADdpMkIwftnpwScsNi9CHWmZURoydrd8WKBU39HhJ1an9S/efMsZ/I5qVMdzz/BOJNePUEwA7wXxjk5YFDBaKQwrgt7IBE7pWicuz3ETJVg4v5ZQGVavkA4wYKPNnuwPw0q/bTlJtoG9RYiadpFjjcwwWHQGNOO5YnJ2EVXM9stouhDQayiIs6DbTXmCDxrva15cNWuR7cpiyrirD4mWhyyCO3b+v8SOsHZdeL6RlJPVtogT4hVOe6+eWIKlWe8greUhFIzJulmpazst+tm3oH63JR0igQb6z/AvhDm85bj27vfJHtPTosZ0B9gTvzWSoh4GGeJa8XZlf1KPp7cVtz3IEtZEttL+RJMULp2lKp+/3SimetFcqtmg+58e5pktNygvG0mYBcBvO/mxjIlkEO/IE//dVc9HaxTHG+TeobGpUtVMjvMUrhKgh9iTsXnz+GSTmpZTRjCj9xL4qNUe+gOHs1sX+21mpNaH7aBwBsQ+0oeMoCGfR0V0OeK+Ko2qa7w94D09cm0PMV7tB1VFnfwCpyYfTUJD6Mfa7g3K7w7eKtXDFm/UgWCcR/W3c9s/G8TlctnY4SiIye9z1KY7gtde0ALU0Zi2H4paRK5quFl9nON7PcO/xCnQVNfRoU/QhixG1HpNYXV5SSaQ8J1416WgY3vRblVMGQTK7gTidcrtr2NMlmG0EkgB/4wXTYq9hEprojpE22BJ08zEMotSaaqlHgn+FHyjmp2KfTJdfUzvxM1gvJxVkQjIp4sOfraNBnEAXBtkra9lwsDOx8Vw296wK3Mo8Oc5QaKROfuLI1OqDWySmL629VUQk+bHuKIHMTHdHrGIQyYgnj0gHMwLnCz8a9r+FKJBCIvKyIYl7vVIFx X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0b3179b0-4cfb-44ea-5855-08dba59e2d10 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:04:58.4911 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: eWYBOH6ca0+EYdEoSF9dsZIJjYIR1CZqRiUJepR7Tb6DK8aaYFM/c5XwO4XAZPvTIUZ3vz9tEMi0We2WGhUXbA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB6895 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 bulkscore=0 mlxscore=0 adultscore=0 phishscore=0 mlxlogscore=999 malwarescore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-GUID: b0m3FrRmbHGtG0JsJ0bwj8eE5EcS9YFF X-Proofpoint-ORIG-GUID: b0m3FrRmbHGtG0JsJ0bwj8eE5EcS9YFF X-Rspamd-Queue-Id: 5C4CA20030 X-Rspam-User: X-Stat-Signature: gfp6bnnuxd4pajx4nbx1qp3dxp6fbhba X-Rspamd-Server: rspam03 X-HE-Tag: 1692990350-174461 X-HE-Meta: U2FsdGVkX1/XPrEZdXLi/CEP15sFKXL1VUFfM2+yJOH/8Q3VfO/U5oQ/IrFRqDpAZPObeNgRCl1J0TSibrBI2rIdo19BU1OKJyEGOYz6fr7V5uCssg3WaFjtOWsOVDDfcPFkqo1cIrFPIMFPZ4mGtwVF7A7L2n7ifNH1YFC1n1oya+2KMBncGWd1gZ00aM06czrqag15QDlNCPO9pHx0394VeAlkUYRTtpeB3zBe47OReKoc+YMtRi6knUYHd4MlcQSKiuWe4Pg+WUtGIdbHTB78SLEgNtXYgJLzCAz4DcGs6T/s3Rz0h1pJncQ7o6+7oS0yqZCdnptLv1p3yj6d7ZpePAlanhIz3XIx3ixcshoUd50ZQjOgXYkfK7qxBwdZ03tMPo/mwin6JfjGcrGMaN0qlFBCA7q4AJWB8/EPhfZoF0hj7aZQHgHr1KNBcysvr/goq7GtxXS+kzxYk9IHFiOyIr++t0EhdeaqFK2GxkhT/neKE+pvBSH2gi8XNPKrxjgiWtyCz8H8crFB6ToiiS+nE2VwycpPHMDVeW6jn4Osz2MbUsZXBGlopv+OgK8OQ4ZQh+Hq6SqKmx6jlm69Y7uWGC1U5NvFZ1yRfO2YsSyIuFagH/Tr7O7YTPagE/9x+3xNcS3N08v1rxKM5kQ7uOq4UNypueUTwcXAEY8rm7by0Gd/IbLpdOqXpDgeerYVZQJhoLE8Wf7/wLYfLii+dPwjfDdP92+YUrMigwkGev0bHYDojS+iHRvnJ/+l4zl73Xes8Cx6xsO2F9Dnm9tpwd8dYQ0vMuiltfEaYVdI+YeE+icgFXB0knMnyy9TNTEUlpOJ92a8gTOVbNMp4M0zr+EIrJ+atEBN6DHSz9T3HsRx2IQxIBbwQ/9kq4lbmHG+c2IpVGmGd9fgqVJjKnvy75P/wiO+F6Xasgni2ZS2Mb+ED2w02+GHHnFiGSkRqHtGSsVKnpGeIzOWSD83KyC yCmJzwbZ Gl0sZLA93UHOfrbSM/Em2eS3UDlEqOa8pA83Alj41vDJRMdumK7h0twNoyOWaucOzyhA0PhCZn2fxhI3ZPBzu4C0321nzogHuQVRz1PPFmZl6KRoNymHl2hTGWw3CLt4yslhHB1EkbBNMVJ8FUGUYZi/J/rvY+/2aL47asK0M0S9s2ZxJu/BiewcajYxmewB/RC7JLJIMLNUrZ0bQQB1hyTQy+UQDW0/oHErLtWq4SOmHrQE79NnulgATS/t4HQwZWA+lqrO8+JhWO9qrLKhQ6jKru6R6ypQLxzEqaofZishHa+i13jbipwjEKUJLhaFx3Iy7MuVTw7kaEqJz2tjgtxr3zsyWwOymgc1Jf/fpu0EAaTaID/bmYDsCCD2hv1eTbpTgP9RmupacIkgXcoIo194G+h1pFPyS/hXhcoKnGnv7TBwk7anUzAfawoU21YnTFBpcOoVNlRa3oA3jLawsIWTnNQWyUksTIBT+2HxLURwVKfBhgTyKj3tLynj1xezaa7/N93t+fB/zu19rK99d+wkljF/+S/axhLHNGIgfe2d536IbydnyvJYxv1fkJgtbtXu77A6A5N/tu0pZbZh9xiGIPPxkv1/42GX6UdEsWmYuSlk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Convert the callers to expect a folio and remove the unnecesary conversion back to a struct page. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Mike Kravetz Cc: Sidhartha Kumar --- mm/hugetlb.c | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index a5348dfada89..ec10e3804060 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1446,7 +1446,7 @@ static int hstate_next_node_to_alloc(struct hstate *h, } /* - * helper for remove_pool_huge_page() - return the previously saved + * helper for remove_pool_hugetlb_folio() - return the previously saved * node ["this node"] from which to free a huge page. Advance the * next node id whether or not we find a free huge page to free so * that the next attempt to free addresses the next node. @@ -2217,9 +2217,8 @@ static int alloc_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, * an additional call to free the page to low level allocators. * Called with hugetlb_lock locked. */ -static struct page *remove_pool_huge_page(struct hstate *h, - nodemask_t *nodes_allowed, - bool acct_surplus) +static struct folio *remove_pool_hugetlb_folio(struct hstate *h, + nodemask_t *nodes_allowed, bool acct_surplus) { int nr_nodes, node; struct folio *folio = NULL; @@ -2239,7 +2238,7 @@ static struct page *remove_pool_huge_page(struct hstate *h, } } - return &folio->page; + return folio; } /* @@ -2593,7 +2592,6 @@ static void return_unused_surplus_pages(struct hstate *h, unsigned long unused_resv_pages) { unsigned long nr_pages; - struct page *page; LIST_HEAD(page_list); lockdep_assert_held(&hugetlb_lock); @@ -2614,15 +2612,17 @@ static void return_unused_surplus_pages(struct hstate *h, * evenly across all nodes with memory. Iterate across these nodes * until we can no longer free unreserved surplus pages. This occurs * when the nodes with surplus pages have no free pages. - * remove_pool_huge_page() will balance the freed pages across the + * remove_pool_hugetlb_folio() will balance the freed pages across the * on-line nodes with memory and will handle the hstate accounting. */ while (nr_pages--) { - page = remove_pool_huge_page(h, &node_states[N_MEMORY], 1); - if (!page) + struct folio *folio; + + folio = remove_pool_hugetlb_folio(h, &node_states[N_MEMORY], 1); + if (!folio) goto out; - list_add(&page->lru, &page_list); + list_add(&folio->lru, &page_list); } out: @@ -3417,7 +3417,6 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, nodemask_t *nodes_allowed) { unsigned long min_count, ret; - struct page *page; LIST_HEAD(page_list); NODEMASK_ALLOC(nodemask_t, node_alloc_noretry, GFP_KERNEL); @@ -3537,11 +3536,13 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, * Collect pages to be removed on list without dropping lock */ while (min_count < persistent_huge_pages(h)) { - page = remove_pool_huge_page(h, nodes_allowed, 0); - if (!page) + struct folio *folio; + + folio = remove_pool_hugetlb_folio(h, nodes_allowed, 0); + if (!folio) break; - list_add(&page->lru, &page_list); + list_add(&folio->lru, &page_list); } /* free the pages after dropping lock */ spin_unlock_irq(&hugetlb_lock); From patchwork Fri Aug 25 19:04:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13366218 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F960C71133 for ; Fri, 25 Aug 2023 19:05:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 92666280055; Fri, 25 Aug 2023 15:05:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8ACAA2800D0; Fri, 25 Aug 2023 15:05:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 353712800CE; Fri, 25 Aug 2023 15:05:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 108CD2800D0 for ; Fri, 25 Aug 2023 15:05:40 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D305A807CB for ; Fri, 25 Aug 2023 19:05:39 +0000 (UTC) X-FDA: 81163555998.13.EA2EDAB Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf22.hostedemail.com (Postfix) with ESMTP id 8BF7CC002E for ; Fri, 25 Aug 2023 19:05:35 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=JcDyRPVG; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=KxVyXmUu; dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf22.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692990336; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PdmwOcXhjYj18bUXSFQSbkfcUgdNSVEPyPGP3ecmcJc=; b=Bv7WhQ59yw97HeBHltrPOJOxLFrvxley1Q9vhUuPR1TM42WtVczC4jE3x4yokERgJy8Zja gNiU/yv3O85i/QIy/RpOICujuujfhuO8C4lppn1jYys+2fx1R0WWyiK661gxpz3R3HYY2J Dp6i1NPbHTL/QGaW8YmuM/7P108/hus= ARC-Authentication-Results: i=2; imf22.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=JcDyRPVG; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=KxVyXmUu; dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf22.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1692990336; a=rsa-sha256; cv=pass; b=Zq8nSDUC3JypPdqm+6BVtysqqAOHCL7OuXUeJ4bHmixSrY/uFQF1/t4tUQk3sJxtumO2Pu lv/3SGoKbEkzhJQ+9mtCqydnMjRBrsivoPwGtGAuFw3LIqebIptrJFaUw31n0weAVuObMp EWhojudGrOkrdXV0DhPNr9vpCaXvdOM= Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIEOfe031083; Fri, 25 Aug 2023 19:05:07 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=PdmwOcXhjYj18bUXSFQSbkfcUgdNSVEPyPGP3ecmcJc=; b=JcDyRPVGUZFskL/nrSsBSF+TKgziEeoTD9ExBT20DkdP15qNym8OEoUB5dz+5tdMseIO 4eYLldh7godXnAuH/0YvNhGxX8sdWCZZtXVVOWe+sSgXrgMeGsFb58qtJOLm0aMMkf3M lL7mUtwzgLNYRr39om71T7zSbHrYHNmLe/OyPvPHEMHQee24pN0s6ZZEfP+C9Qd81pX5 ClkASfXYkg+5R6rC/n2HgZlYhqgzF2tNMCOUNEwoSR0XZjUpJAhPruZ7ev3LqHqL7bCm OxocR328SntIBj9TSHA5QXyt2A6MV3Z/54K/6Ak/AhB13jxCMBiFW+WHzPDRB9zBcHWv ag== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn20cq0dg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:07 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIUF4w002259; Fri, 25 Aug 2023 19:05:05 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2041.outbound.protection.outlook.com [104.47.66.41]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yyg8ht-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:05 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FweYzoQTcHTB8cM+kd0fH90FC8SUt5AY/KIlWnYBAQU+BreiEo+6iq+G9BTpgFBXg1MDPrz3ZnBs+gFdmvz0NGQoO/uqxg+i3nND7I8oLLxe8wJa073DpgUyyxD36cLWuHpcT16PGQ1+XN+FKftADhNcLhSkUFhGXuvSsL/6Zv8QEKcSMbMz8NiwZIZ364/bXDRrgNt/V05epfCCvukq51zS35rn5ULCl7I278cIqT/i9DCjK3Wh2ZM10EjHeLpmgJpAYIVQzWomiNvgiK9zvN15DBbVkNuzKQ2tHzDVPBCImhbb4vebFwTyqyxoXVYH+C1rAipInTjYXYtaDao1Xg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PdmwOcXhjYj18bUXSFQSbkfcUgdNSVEPyPGP3ecmcJc=; b=i52lHRk4dSBj2osfkp7MilIg2/ArAVRDXpbJGa/KttEnVaYnKzpPBhdM3lXlMNL5XU9HrvM/lBQ7+cNB+kz9QiLs97ZGUHBL8b3n5std3V2rjMEzwPoePjLBniePI1FR+TN02AFhCo02LvPL31u9SN+6l7kmdyJYNI4JI+dfP7bIGgSV/TN5lfPMfImvUXqwAz8lX+PK6/HyQaVRG27/Jctp88nqNFCjuSyUl/08DZE0UhcApTIP6mdpTfLQhsyZ3RBzu+Cj/JlRayOF/ee8kIWawtG9/S/hp6g4xx8LON6rZVnskuH5X2tP9Wy7yJ6x3rjZUtIZ9dkXOWwMBMd76w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PdmwOcXhjYj18bUXSFQSbkfcUgdNSVEPyPGP3ecmcJc=; b=KxVyXmUubOsX8Z/6xr8zo00aKswdVs/6KuXmnTK4+Nr92LZwCtjE8DIMg5yO+4ne14G9uTmVMyToCPm4JtfcY8jfUAR0k2uAxTMJ3+ude8WcOAJzgc5FJgOTK8pIfhMPVchw448vc+raFlnQKAi7JyuqllXQobICabkokb3OOa8= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by DS0PR10MB6895.namprd10.prod.outlook.com (2603:10b6:8:131::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:05:03 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:05:03 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH 05/12] hugetlb: restructure pool allocations Date: Fri, 25 Aug 2023 12:04:25 -0700 Message-ID: <20230825190436.55045-6-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW3PR06CA0029.namprd06.prod.outlook.com (2603:10b6:303:2a::34) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|DS0PR10MB6895:EE_ X-MS-Office365-Filtering-Correlation-Id: 08201307-93da-437a-b798-08dba59e2fb7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qlXVlevVXQK8HWBR+q/VThjL4jkvUXuSA+Qz49eVpz3fllsAJY3RHgrJioA1lp7BhpTBZVQYyZd1fX1g4iHgW6xhm68Qbn4eVEMZnNkcI+4eg7Boh3nowDqtp0iS/DQB7vMydTaus81UJBMXbfWlefiF9gZXRHx/g84FAE6airfP4MTH5vj/Oey0UaQ0f/lyLdefiFhWJu7ZHbpXc08eic+BkLozjbVXgIwaV33XzQRKFrrLb5Z+6evI+pdmp2RNj5gwHibbXZWnvimlZR91mZwfPUDePCyJliEKGcGdOl4kw17Lpy8v57fHdYyo7FhXQKwXbERE6by2YdgLh3qHYiIVIPgX28KwSYVsZqIr1S7ceXhoOnApP3MlSUvNTvIjCyUf02J6ZirWlTnTzH2HOTdOxTnSzDwAPM24/Nd2ZnCBAu3EK50Cc19NmrDybinZ2EBArUefRytJHx7pBWuaWcvQJNURyeZPvGWw/MHKxwsSUGtQHRsPsQTp6/KS/8iloBVD0tbik7twRCGjd7XVNA3E9V6fwmhzbDxMFt/wSofeUwR7Co4ueyOoQKQj7aAD X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(366004)(136003)(346002)(376002)(396003)(186009)(451199024)(1800799009)(54906003)(66476007)(66556008)(66946007)(316002)(478600001)(26005)(38100700002)(6666004)(41300700001)(6486002)(6506007)(86362001)(6512007)(2906002)(83380400001)(8676002)(8936002)(2616005)(4326008)(107886003)(5660300002)(7416002)(44832011)(1076003)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: FjD5kVpzf1H3W00Xod8M32RXSH7WXsHRVzv2CcbU0z27xWCxX8hXRswQQHUGoBluK5eGWR4lSwa09HgaH2H0V8FNMwHTVUX9tNyjgV9dRDrL97kDPQjtWwvxCNaD4YOxYsZQMMzBFrSgxotjBvanySTwx8C2rjpg9eXmNrEmkitoEFzw882chlAzpBe5L1mgWDd9x9nOV4/t/1v1y9p/3EUytPWh7vVgI/WlXUT1B8iV/LxXIjcKJN6IqNJUtEBALEhAr1Dh2rNFG29dxUzGWHuqzics8enP5J6NwursverX35EsxVtoMXPkkkHn196vkqjdnB+fYU0CkEh+3xcRk8PqzX9whOZWcaHm2A3FTsze4loj43x20uTH8D/Sk+aU8aOebepN63ZZNjtvuwRrtmE2i/p8Dj1n+AO2KUYRoZP13OYvFw7Bnm9sG/XT7ZqRxnoHIs2DYaJmD9AETuffupYrT7L08Due+QUNsFeb1PBWmUskhmqngAEju4acTUeEY1VijVNPZmHyFY+7lCyMZCVTsvqM1ZUPvuK2SuveIBdCzG5B5YgJ1/FOyZVXdGSCDb37RxpORfbQtBuYznnc2dCfYp8XVUcXWpUhnOSzbDkQ0EDqtvEl42P59BzaRkkkUnq1J9Zq/5gTcPR235t9vMxBC/ZsRc+BvsUwVxeSAYRpTxW5ppVTDiEV7+cGijCctgplWD5vhqX+cnqsxd/6UvvO+0A/0dFjZkhPz4nEaWnPXWcjoVmz7aTT9B1WU0kOPzYKSHhCpGA69j6l/KVd0hZBmdPPxOx7/zhHTBAqTsqral0Vh82xj5qWYpjPk0+WCI1X5DdgY0q6+dXOkORFuj7cQT7dF/f1gkJ3jEh46uF+Q1bKvjzmtLWBM+n3OHFFTV0HYLBBM275sQZvmsCy3z9uZAQozTeBfSIpytKSvxj9Q73/8L2sPaT7z57rP89ALXcIo3WSBpjHqd01jBkFvIywsg42IBxlFilwApZuQhggbsC2/2DLCmSLRll6BLU8N/doHN8Mrbc/9LyIFI3nIkXA6p0YL/AOk8xl2DnEbJxke80z1FWzSkbTCJ6fF4zl/pBN6flb/Z604TRXk8OyWeQDFjaaW2r+wISHpHvDGh7rrj37tm46KnatrzbqAV1qat4qyELIC11sU3zJDoq5uoyiXUUbwO5Kjk3b1tLwBz4vaZ4W0Sl3n+yYAowf7aRxx0vrSwYQGaLm4NRvNmLqes7+DUorCyca3c0UW7qLzpq+HYyTmstRZgzMY7j7poCe8x1IoaS9ddAnpeAbF9e+ny+JYLidhtAbwvKiYf7XY5pW/2kWIkJHBXKD6GEFJJyelIHnpKEkYTCYU5WtoAr4eAH8GXETvgYnTxOgEDlY+SvfiQ7VU+mlng6DzAo7qDEBvbL42LeaFOb3OlNyvtAB33+INPwviNfXBK2CbSsEEiILwBYOtpLkvwKDZ6kYQiftUoicbosN0rU7Sz3QknkHlUttk01qfHUQZuXu5O6bMAF5Nx0ZXy2uYdqH01oMYyn01dizUsrx62KJTHexCR6jr3LnvFE6Z7nt9R8F/fuMLYYfbLr0nkA085EivhdXxSDXdSMq0lJgB94hcQ+pHv21gw== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: tnGmb3xHPtClVn7B7yO/h0zVXfpXvaD92JMcMwbfVuJiDpsmG6/RcMKvKonfDTmSMj5E21jEE1/jtMntlX7sRNnasY8HkpH0NqrX26YhIlJ5Zl5TtIxxGgEHh/rIeEI+9yl+PheivwGJICbpMkl3jpo2Ucl54HE1+yH65r04yZ2T/drlmFv1o0jx0eHwa0QW+nQbT+Qx7/LMRize21YkSFCKs9ZWZmMGCdJ9Y0s27kJ+aUP8LQEYadwT9c1T5yJjRCRm6hQsGOM8cqIDlSCw+z9hpYuI5Gxuaorhu1Z5/mnJqi6pVib2k/z6z2qaw2jgEOhmqXB/CY8P4pp8LueVblek2erLVMyE5Z2LIzn3gyGoXTue+64XdtWMxIdwnPdmxrFs29VKRFWC5AcoEO78lCMo6hlMIeLP+HXaiEpLvmKvGxgVwDIRJ2y+nh46TzSfOtMPc98Oy6Va63UMWnCCY/xbo4rPhbb/Ja6fqyYVW/QJIDt2Bg+vU9AuzvP3Pe9+47ScZn9fCMi2bkN3UKByeRSXcxhRwmRKOq2HhK+iWGZ+bKzub9HPvXzz0ore7bnjr59+7AtN/jqUHmFG0jcJsPNx52lsdGtRzc/Vm0aL3LQb1pMN/LnwPrPFI5mNYH9Q4TBoXBok0Fr1DeQhqsfsqJllJKP7rhl1s7uj2glCYT1r30lrG1IvbD97heKMxFzySATmWiE4M/Mn0IXhQ/A58vHS4YXucrdP4RirP5Pj+9+fYWTLYTshM/2kBUT9luNLs/YMkuTIZmznD2zprLpnP3hidRJyGkdnQEAI9AE4Yx6AMIp8STqOwpDD8+PoxJB9u6XnxFUqOqGLglX7suUOREfvM2h9VD231p4ojc9uPWdyiYPC+Bv2Y/LSZYFdP2JY3b0685tg0Adjfv1f6DLPUhrPRVSr/SRKU6gpY+FAwXZueh1eJ40JodgWr1puo9sicby4/YWcyQXtfvOxbhM7dR3F1lzl8PW3fWTxNZI6hMfpjnjH7/ABaMVczWD4Mm3yJ7YItOwmIAl9Oaz5UmduHgod6CIUEq4FYpFbQFSrZbYXoThr+JTDoPZxfnWSGeu5/2kFcdcpZtRM96czouHSHD0z3svrCHZZZkdG3M2nMWSRPdWBf7qDCgm2BebBsoAx X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 08201307-93da-437a-b798-08dba59e2fb7 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:05:02.9823 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: tfwoHS2vp0n6TJd2zHXisUnCQd+G+e1IHJEkDxxq+9EctD9DNN/HVzJEhLDQLNfTcuDPja+Zefzq2QeYXGN8Uw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB6895 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 spamscore=0 mlxscore=0 bulkscore=0 malwarescore=0 phishscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-ORIG-GUID: hA9dqyJoFfetm6zs0B3tHWZsYX3wSaHo X-Proofpoint-GUID: hA9dqyJoFfetm6zs0B3tHWZsYX3wSaHo X-Rspamd-Queue-Id: 8BF7CC002E X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: a9ypk79hxkhnzt9qx56o4q31b5mf9jts X-HE-Tag: 1692990335-928087 X-HE-Meta: U2FsdGVkX19ZZkc/uG7LrOdTYMmAsqaoUo7MgCGfr+a22ORI6RLXd5ovawoTnTwXtcBaJ/2lXYPJVMrjg9UtMZfo3Sxv+ioqrIRK9DUCxu2+Ueq+FLBRPWg1y08MUcrHSNuWmOG0cf7CYo5Iptb6tpIoQG7SsI84hT9FYsJW1b+WdGZL7ev1MmSY4SJ3/mMLrnu7VXXRoeDYYXIT3KbqgLeXjKKQnr5pcyT6IsIKtIuqXNNGvMM2QXhFwn4BtsmJUvElNLQfR43XRnEzaXun4EbYd4Q+FHyGsh1AlDbavdn6z7lD9KPKzqlCgQXuK2HHBkHypcP332Cc/M13bMwiD7nMxuCqRcOP8lBifjkDKAxQk/8wbX9Ap70FyMhri5nEfJszEPhycTpRpF68MTX4sUzXnJ8M8iAIkTLCwRySt38nJ3S5dIdGwp10rEvK7n2JnL4ExoJcTAj+8cAokzBWmANRN4JOPYBI/sg1Hiu92aQlEa/TDT+F6+OTED2Tah4cQz+E53Z9jXFEBAU+OQ2AGbmndQGL+X113bVlPIICbEUiEohsZq7SlGlvSgq64sGu7WQeNFAjAFey4i+fVFfQjZfc9a2iXXVqd9oA7CatueXATXf07HJofKmf3czWO+mudtZRRjo7tdziyGUnoOdKb+fGF3SfEam9wjMTBIPqfTIJMgHpse4p0ub9i0TYrykMeeWwbdTkeozIEqbP5nDt4XJFj3784EW9oHGHdxLHh1IS/QEyOhtroouxR6GIvPJ41bDvAyhRUVcomXpxgsIDvk1R1kPmETHjITFLeyq4CG1pwIHTa4g1cF0T0hOVAFv9rOX1kIToVCpW2Zic2A92RmCVVmbDMdF1cpI5GyOda5Hlbg9MgHqkj6DxllIZib0G6ywpinvHVVX9PI0G/jR/nS/IIzuZg7dzNnepCvu+A0PiQozESZwMs9RNJn1kuHQngaVy+zVd/9Qw3Q2LP5b IbQmEutR KJg/Hqjrr3eJ6+t4P0G/9r+M3gInd9Va/92Tw+Ip0raLrrYY2HCH+NfpeQ4gu5ypkxDxpxDQHDSRwuFufFJ0GyxPPrLjjZadKszHQ5Fr7XLjKnqSJiPJQ4nMwQQAAHmDS8znKx3bseq+/28lKlkkV3p3zuNLbI4IpTgJk5sMx4phJGEdYpSKQMGpLhV/x+s50Y86mvP18r/Gg+roG9gDidhmd0ufABoVvd3z8ITJcxtN/g5T6jMyw3Lsz8zod9J26+q7WrT1CzSOkGlUKSHnPEQ2WhyTMK2iOsDSa5uqYE1Z6uwr7rKH4lMmeNndfjiq7+P79jeeRL1VtEeH4+EuRfkiLvJmUh0TworFvFVD8NrTrnUjl5qFHlbCdEf5ExtKiW1csVnVMBCWyWm5K7RyMAtK0XHdImmaaIK5zfL0r53Cij2ZjD5mVyArUG8kRTUwag2VN7QMg9EAZi8xrfVHfy6E1DIDN4HWdnKXBUDzIxQAzYTcSEYF8VXka1dgyKuU4/6PS X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allocation of a hugetlb page for the hugetlb pool is done by the routine alloc_pool_huge_page. This routine will allocate contiguous pages from a low level allocator, prep the pages for usage as a hugetlb page and then add the resulting hugetlb page to the pool. In the 'prep' stage, optional vmemmap optimization is done. For performance reasons we want to perform vmemmap optimization on multiple hugetlb pages at once. To do this, restructure the hugetlb pool allocation code such that vmemmap optimization can be isolated and later batched. No functional changes, only code restructure. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 155 +++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 119 insertions(+), 36 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index ec10e3804060..452e7ed6dc36 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2146,16 +2146,9 @@ static struct folio *alloc_buddy_hugetlb_folio(struct hstate *h, return page_folio(page); } -/* - * Common helper to allocate a fresh hugetlb page. All specific allocators - * should use this function to get new hugetlb pages - * - * Note that returned page is 'frozen': ref count of head page and all tail - * pages is zero. - */ -static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, - gfp_t gfp_mask, int nid, nodemask_t *nmask, - nodemask_t *node_alloc_noretry) +static struct folio *__alloc_fresh_hugetlb_folio(struct hstate *h, + gfp_t gfp_mask, int nid, nodemask_t *nmask, + nodemask_t *node_alloc_noretry) { struct folio *folio; bool retry = false; @@ -2168,6 +2161,7 @@ static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, nid, nmask, node_alloc_noretry); if (!folio) return NULL; + if (hstate_is_gigantic(h)) { if (!prep_compound_gigantic_folio(folio, huge_page_order(h))) { /* @@ -2182,32 +2176,80 @@ static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, return NULL; } } - prep_new_hugetlb_folio(h, folio, folio_nid(folio)); return folio; } +static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h, + gfp_t gfp_mask, int nid, nodemask_t *nmask, + nodemask_t *node_alloc_noretry) +{ + return __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, + node_alloc_noretry); +} + /* - * Allocates a fresh page to the hugetlb allocator pool in the node interleaved - * manner. + * Common helper to allocate a fresh hugetlb page. All specific allocators + * should use this function to get new hugetlb pages + * + * Note that returned page is 'frozen': ref count of head page and all tail + * pages is zero. */ -static int alloc_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, - nodemask_t *node_alloc_noretry) +static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, + gfp_t gfp_mask, int nid, nodemask_t *nmask, + nodemask_t *node_alloc_noretry) { struct folio *folio; - int nr_nodes, node; + + folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, + node_alloc_noretry); + if (!folio) + return NULL; + + prep_new_hugetlb_folio(h, folio, folio_nid(folio)); + return folio; +} + +static void prep_and_add_pool_folio(struct hstate *h, struct folio *folio) +{ + prep_new_hugetlb_folio(h, folio, folio_nid(folio)); + + /* add it to the appropriate hugepage pool */ + free_huge_folio(folio); +} + +static void prep_and_add_allocated_folios(struct hstate *h, + struct list_head *folio_list) +{ + struct folio *folio, *tmp_f; + + list_for_each_entry_safe(folio, tmp_f, folio_list, lru) + prep_and_add_pool_folio(h, folio); + + INIT_LIST_HEAD(folio_list); +} + +/* + * Allocates a fresh hugetlb page in a node interleaved manner. The page + * will later be added to the appropriate hugetlb pool. + */ +static struct folio *alloc_pool_huge_folio(struct hstate *h, + nodemask_t *nodes_allowed, + nodemask_t *node_alloc_noretry) +{ gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE; + int nr_nodes, node; for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) { - folio = alloc_fresh_hugetlb_folio(h, gfp_mask, node, + struct folio *folio; + + folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, node, nodes_allowed, node_alloc_noretry); - if (folio) { - free_huge_folio(folio); /* free it into the hugepage allocator */ - return 1; - } + if (folio) + return folio; } - return 0; + return NULL; } /* @@ -3184,8 +3226,7 @@ static void __init gather_bootmem_prealloc(void) WARN_ON(folio_ref_count(folio) != 1); if (prep_compound_gigantic_folio(folio, huge_page_order(h))) { WARN_ON(folio_test_reserved(folio)); - prep_new_hugetlb_folio(h, folio, folio_nid(folio)); - free_huge_folio(folio); /* add to the hugepage allocator */ + prep_and_add_pool_folio(h, folio); } else { /* VERY unlikely inflated ref count on a tail page */ free_gigantic_folio(folio, huge_page_order(h)); @@ -3231,9 +3272,22 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) h->max_huge_pages_node[nid] = i; } +/* + * NOTE: this routine is called in different contexts for gigantic and + * non-gigantic pages. + * - For gigantic pages, this is called early in the boot process and + * pages are allocated from memblock allocated or something similar. + * Gigantic pages are actually added to pools later with the routine + * gather_bootmem_prealloc. + * - For non-gigantic pages, this is called later in the boot process after + * all of mm is up and functional. Pages are allocated from buddy and + * then added to hugetlb pools. + */ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) { unsigned long i; + struct folio *folio; + LIST_HEAD(folio_list); nodemask_t *node_alloc_noretry; bool node_specific_alloc = false; @@ -3275,14 +3329,25 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) for (i = 0; i < h->max_huge_pages; ++i) { if (hstate_is_gigantic(h)) { + /* + * gigantic pages not added to list as they are not + * added to pools now. + */ if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE)) break; - } else if (!alloc_pool_huge_page(h, - &node_states[N_MEMORY], - node_alloc_noretry)) - break; + } else { + folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY], + node_alloc_noretry); + if (!folio) + break; + list_add(&folio->lru, &folio_list); + } cond_resched(); } + + /* list will be empty if hstate_is_gigantic */ + prep_and_add_allocated_folios(h, &folio_list); + if (i < h->max_huge_pages) { char buf[32]; @@ -3416,7 +3481,9 @@ static int adjust_pool_surplus(struct hstate *h, nodemask_t *nodes_allowed, static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, nodemask_t *nodes_allowed) { - unsigned long min_count, ret; + unsigned long min_count; + unsigned long allocated; + struct folio *folio; LIST_HEAD(page_list); NODEMASK_ALLOC(nodemask_t, node_alloc_noretry, GFP_KERNEL); @@ -3491,7 +3558,8 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, break; } - while (count > persistent_huge_pages(h)) { + allocated = 0; + while (count > (persistent_huge_pages(h) + allocated)) { /* * If this allocation races such that we no longer need the * page, free_huge_folio will handle it by freeing the page @@ -3502,15 +3570,32 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, /* yield cpu to avoid soft lockup */ cond_resched(); - ret = alloc_pool_huge_page(h, nodes_allowed, + folio = alloc_pool_huge_folio(h, nodes_allowed, node_alloc_noretry); - spin_lock_irq(&hugetlb_lock); - if (!ret) + if (!folio) { + prep_and_add_allocated_folios(h, &page_list); + spin_lock_irq(&hugetlb_lock); goto out; + } + + list_add(&folio->lru, &page_list); + allocated++; /* Bail for signals. Probably ctrl-c from user */ - if (signal_pending(current)) + if (signal_pending(current)) { + prep_and_add_allocated_folios(h, &page_list); + spin_lock_irq(&hugetlb_lock); goto out; + } + + spin_lock_irq(&hugetlb_lock); + } + + /* Add allocated pages to the pool */ + if (!list_empty(&page_list)) { + spin_unlock_irq(&hugetlb_lock); + prep_and_add_allocated_folios(h, &page_list); + spin_lock_irq(&hugetlb_lock); } /* @@ -3536,8 +3621,6 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, * Collect pages to be removed on list without dropping lock */ while (min_count < persistent_huge_pages(h)) { - struct folio *folio; - folio = remove_pool_hugetlb_folio(h, nodes_allowed, 0); if (!folio) break; From patchwork Fri Aug 25 19:04:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13366216 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DFBBAEE49A3 for ; Fri, 25 Aug 2023 19:05:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 17A5C2800D3; Fri, 25 Aug 2023 15:05:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 16DCB280055; Fri, 25 Aug 2023 15:05:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD19F2800CD; Fri, 25 Aug 2023 15:05:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 866D02800C6 for ; Fri, 25 Aug 2023 15:05:39 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 5BB9D12078A for ; Fri, 25 Aug 2023 19:05:39 +0000 (UTC) X-FDA: 81163555998.19.D92172C Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf13.hostedemail.com (Postfix) with ESMTP id 711CA20021 for ; Fri, 25 Aug 2023 19:05:35 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b="z/xyJX30"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=mbIFw+4F; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf13.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692990336; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hgjjLKzxatL2kLeVbyvUq9gKeYfImLdjnfaoRhKBwD0=; b=kFCGl7NzY7sO8M4Xq/JWzRi/7jOGt2BEiNiWqfok1hyvnZRXquHuWIFwkBM8X1DJV7ngpC UfhVejISYOcrec4OerdWM2DT5i4IZijeYBSVTTHvewc71wjLZUnIrQrV/cg4pg4b8No4P9 as+0VRnpH8SZpyeVXGw6wvk0UP65l0o= ARC-Authentication-Results: i=2; imf13.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b="z/xyJX30"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=mbIFw+4F; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf13.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1692990336; a=rsa-sha256; cv=pass; b=cxmRU/ycuziQbfjLnMvBoPIBtFFMKxR++R5T8GcJpDqlp/RabUW+MwZ3wNeu65y20NI8xA MD8JIOEoX+wvJAnKV6eZzViNEcc42gbcvH7DohM4K4uRezFOJoZLbGL0eFf87iVO4NxQEc GSm4viye7sTAB8DGPuzlQ1+WXdCxZBs= Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIEWaI031486; Fri, 25 Aug 2023 19:05:10 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=hgjjLKzxatL2kLeVbyvUq9gKeYfImLdjnfaoRhKBwD0=; b=z/xyJX30XZCI2BsNidlskUASj7IscFRb1fZd8bEPimLqCabKlc9o+Z+JooI5a5wclB+n ksqugs727M3bGNT3/SfFBaKJvvC9Ne3n6IurVD+hDZjT1zpiPdssKLWX93RFuaMEcTk+ 3gIgxqngdlIPApKKck0b72HKcSK3EEwLGdtjA+Sxnrx8YQnySl55z4Jn6IZOsaH49p1u MhZlOzXbLvGnTr/aPTE2I0GQ2Qlm6Ne/X5MxSrNDWSI/LjiTEFuA9cyFaEixPifWuwJY RtAxEzeNiPPGu7SqLvmqIrnvduPDbDHOM7qj74m/KImZWaJz2l2JI//nAwM8ondUTljO Fw== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn20cq0dn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:09 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIUF51002259; Fri, 25 Aug 2023 19:05:08 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2044.outbound.protection.outlook.com [104.47.66.44]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yyg8k5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:08 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jWrP9Qjf7um6WvLwmSCGrDhukrrmpgI1AcZTV5KSNEaeCiVIBXbzJLNzDqKyrB5ObIzKG0ij3ekKsiUqFwxg+awarjuunvAqOOB+yUBmUwRIWQikz2T3X/OdlkemQ3XVZATgOYoPPMchuxsITL12FangHZqIEtypOrh8wAMmCeZkpwDKj+5gdG9CKtyZ/sGvlwpZCaQ/YsY2ynYxPLhu9VbJAyOErVxfcZx0lBEOuCsx4VoslrVkXSKVktXRGz4juJ5zZAS7QhrwfcBjJlYtUvmwce4PIX45AnS1+uR+a5LyTCEXjnN75WwXJOMFCAgt1t/Hq3cGc9kUUTkGaBWCJw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=hgjjLKzxatL2kLeVbyvUq9gKeYfImLdjnfaoRhKBwD0=; b=THr6BKYBNTfAKpinJl98k1axbv0NvwhzQq9C05NTwdWkLUrlGFRgKwp0ITJmTGYh6ZLQZmKBM8Vlz7KCWi0hsQ7sNImQGqONnl0WiWX8OqxGnMzlvuPUTfdAB1r+Ntsvm+5kJCj9MPYxo7krlJAAoWPlO1R5AkW3VWnHclz41BWRHKOqwxlqmeMwcJNgcinDslSRhJJ7I44auv+ILaKPHOpjs7rj0B/o/YzSDXJ7HNx2KgWVDLUAprIu7F2hybSUwFP24pR0oHjxo+pn1JsybFXm2ADaFlOq1xHmMXoXQLstTosm5FEqgkbc7F76p+ME64JleiNbr+7ClK5fAeqYHQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hgjjLKzxatL2kLeVbyvUq9gKeYfImLdjnfaoRhKBwD0=; b=mbIFw+4FC5egAb6bkI13P7RO7pWnN315794cOe9Rp0cW06Ts7e9Vg2ukzD3OYJhVv6P6QWhPaFlFVc1Q3ULqbwqgmO0jjsXmYhT7xNPAJQrgqy9MenLTYgwlFa0w3xQgyeNDXTKy2Tmu1UVUYvQrnivZCm7hu6yNGNMomvURBao= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by DS0PR10MB6895.namprd10.prod.outlook.com (2603:10b6:8:131::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:05:06 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:05:06 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH 06/12] hugetlb: perform vmemmap optimization on a list of pages Date: Fri, 25 Aug 2023 12:04:26 -0700 Message-ID: <20230825190436.55045-7-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR04CA0208.namprd04.prod.outlook.com (2603:10b6:303:86::33) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|DS0PR10MB6895:EE_ X-MS-Office365-Filtering-Correlation-Id: 483753e0-17ac-4bfb-518d-08dba59e318d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: hnxJSfTikXoHKsRYQD9iT+K3jM6MTMtZxAy/XGK46jZ7HEIpBoWISR8uDWPFn8++wAZ8LYxx1KqAUwuCO4M23mY3qPJyLfc7Oi3/sHTjeHqVM6le0uk157eFpQ846/lBQ265kBLXJQ4Rr1qoxUJ0829D1zq8cDeDoceWXKEf0hd2DGXoVb7pHMI/UVusiF5oqsanZw1DDC6EZiNVkthRH+Yhr7L6hDf8oqPkKrgOtaXdsrGkyV9UjgLiptoVARfZqC4/4luEPBoAy4UVUrqwD32bVV95LQLCmthoq+k9MM3iJTQUkKbzaFUMR/ycdjDB1hX+oBnsMgJiY6r+syOmHquyysvm40VQkiR8aVV17QG6niCc5PQ+0nTtC1k+ujXc9HiBKf5bip+GKVGrbW7Njt4VrXEdqWezAqfGD/gXfBMukZU4MH2zpbiEGN148iE8xEBPIAqiJvmeDvAysspYSvptZ8WeEcLVTxhW9arqqPV+e6bnaTHsHhBTIv1yZBS4eksQPzoN3IKsPAp+iNnDl0IRUnvMlZsmRU61oVryVR4Kbxvll27+FcTruszQFnoj X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(366004)(136003)(346002)(376002)(396003)(186009)(451199024)(1800799009)(54906003)(66476007)(66556008)(66946007)(316002)(478600001)(26005)(38100700002)(6666004)(41300700001)(6486002)(6506007)(86362001)(6512007)(2906002)(83380400001)(8676002)(8936002)(2616005)(4326008)(107886003)(5660300002)(7416002)(44832011)(1076003)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 5rB4xzcCPyDfCyc5Hc3/aQ6oM98DT7rh2cRVOm85p7CdUdnjDz+cfVDhft0w93PBR9ANwIf6U1rK8D279FsQBRfgcPktOYnnY+1GcfXGNXBSk+27LRfc9MG/wCyskxJM4CDLcEN7OYNPlUQP6vOkPp3mpnOxRsvppWIrQvkdwFAhT7+TBo365HbOlEi3v3fh+piTzyR42VfuF0Pt4W3wWJC1HzvqgiVfkrNmzkUPF0jnb3EXO21nta2qItfQVCTrL+7WLnj9NuTsF3p6HNjMFu6vPz3yY4DyU2k0Cv8724Xr1iqs1572ItAygEOOGSeGAgmdWTwIWtD+KZrJsdw7uqG1Y8iFPrldBd1UZvKNkRdrcuYHzOhOmKlaA9RFqjPRI1PbjWCh7vpbfpAMYFxKHIwDd3X5QhMJR8xdXY7Fza0s1PGMrn7aF1YDntUmbEpFOEy7SBFo8I0fuyqG/BHV8fS34mQ41oCDxK1278cTXJkMBThIpK3to1IY7CaxNvOfkEB+CA0lYHudcyH2T4Duwq9DQJei9gS42KQrK/K3AwtwsS2fLHd8rxXcYzIanZ9ox2IaYhHj2mcHDgmaP8eAlm+Vd9525N5WTESOn9t0B3i9KaGopioDym1yidAuB38lIcijOIRKG3U85xH5gAIdQAtcrVHhJvEGngPN8SL0DJNNFbE0jiG6sUhqph6QsBi47twaxJOn2cPD0xEJAnSqtFIqHA0Yl0W/Jp4edzzYsYyHEKPbXy4o1YVdtaZfH2RsQ0eUHgrLsHuJoce2sj+IgVbXCHiQJdhOIlCm9cnkjOfFaOkigKxv4glwDInvMQyX3xqmg3UtBXpth+vXoub759RsC129ZwxKsEiLm/SWBfZqQxYMQXCqYitrH96XO8Tlp/hlXjfQAOnoFaaenHCIxQP5xRtSjoNt2okG/EUHIGaUo00At50r+aKhz2IZrVEEIfPXvEiE/6PQWAmtLAps2ynfcPgpMUMjbWom5EnEe7AO2xXzgJOTiZiQnp4QeCtCtbtUb98k+cmvdgVBzSsi6i1Eo1txescs+o3CGw6vJhgCXEzKZc5k8RDZfcf5Hzll5Cn3jh5qEqxQO9c/ki35QYxpKlzriVbvfySk9oI5rqrLQWLdZEg6gql3Cw8K1PRXIDQOUlNCCtR5EngpRsrWk/6NpNegLFCDCp5cDskXsTNTfgTJe+dEx552XulxmSZXpSaNWUcbVRIFr7Wh9fQGjdOhw9RWdB33WjgxssFERqtvai8iOwIKCy1pkhQaozTPsPoaVg8hXPoukDfK4YEGJD6va6dNqguQ+yeCNR2pDKexJ4Pvdv2USJwCdnxQ1j5Z5WHlvQtaHY2EVeJ1SF9UTRNYxhqsN4RjpPLRXQUFHslkNw2m3C2eDCBS9r1hGyaedAAv4DzeapPqrD5dUauSAzVo4O30Yq0DcaOmwrhMD9cUZKpQf2jMzPEn+x7UNHgSNNujXIWdwXLiVwURNLgaepwuluyMULG+N1vfdBiTxc/xfodwmpHAbXqDAXEy/sPFc8cbG2tDEl6N+KJselax2lK9bcf0+Jewv0rSvW9Zs7wOiBlb8YRGVMPDEdwGd4HWWrjj/Uh6TsedaJkB3RGR+w== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: wspp+Rq31laP+o+2y8YLGL0kHUIVl3+YPry7w5JVbxpZN+C4QHqWuvYAHYBwxlaqWbMULS8XxOTX/PTsu3rR4+KPRiCE/Mw8ZZdCdFqoXPdvLz9lnBqXT6Us1W11VBdkABKQ9wTfJRJnfUuIUtmiQ6hFj6MaycrBhg7s5c8uuwD39eO2pDgBdrioz/u2YKBkntQBsz+6DlUGSCigvxL8ZwfLKNGKxdQdbZs/npko62ns8Xpszzr+7twyPHHbho3PaUY0UAwiPHrTLne+Vkjpwe40wtBTNyRa6qt8tYdFzh4J5/6XJDGwe8GyoDBuo91YMvr7QO90Wxuw39JsoAD4PO47Ptp4yeIKlKiCM4bGU55l+U1QAFBjOibmsqaYlE6O5qiXovxEjNWj0hKA7cYKMYYxCCB+oJ3FJhk5128GI95mewG5mynlgR7sOl9PzWFiquA4UWUt3qZIB4C8TXqY8ojOiUWkmSJf0IC2wYuSRBIt+PEhpTFEIYehXsyk7CdRz4vnj0Z9mMpbrtI+xvrasYPzk8JN5D2QWGtDyFqpLc89EQOf2Ogf6aSKZUHreCkLJH7HH5pKq2izIGCfQGDj0gQrhrrt8e6QCTp1UDixYd/6LHkgfbQZXPLHeeruAsuGdN7plhfFwZCKUUtzLF6oOdXhWsy94fD5uWcRxxX77fNwX/ptIgylzdZ74K5oVElI/wIfWPd7oYwV0RngDfYQqHOaD4gOipnV7ILCGL2n5ZKYtm96aFjK/3ZInw7boeLaf3U2224uu3Ove6/TLxiacrU80NSpcuKTAm8TDYb4n10BwNjSVJr22PuIivRY4jWp/GJt1JZ5fLe2PWVkATOtjiZkrTREv6dsFn2saRHyXc9ea0lOivoxHThl4kXUafhBbTBcAxibRCuRYb/I4s/KWOUXo0oz5+Eoor3Luow+ecUFXpH0LdSZ6RGfi5mqkLGaCKQ1HroxFUr6EjFgxME/7hJ2FYlAOntvAKOoPCOG5n5TbvZEoXIB2mybVpbHGyOCk78At6aZb8FPjRaLuoNFps+gwCLVIXVclMNyB0WPmdHZLqqUY5ZUuf/lZ6OMe++KzIRgMsSF07fFKrPE0QRCrh3MBGmcPlgr3MRDUyADdkXV37XOuXqqGOtYiotx6LPQ X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 483753e0-17ac-4bfb-518d-08dba59e318d X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:05:05.9692 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: dX+NjcmF7KoWhK8wcA4xqAm5kCIA1KOFBpz4srQmQ5MoLXa43OPjiibUGHBzumcX5NvziaD4/g+aYTJPpR3Ykw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB6895 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 spamscore=0 mlxscore=0 bulkscore=0 malwarescore=0 phishscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-ORIG-GUID: f6SQZLyQGTPDeSD3UkeDiDl_BnIG-U0o X-Proofpoint-GUID: f6SQZLyQGTPDeSD3UkeDiDl_BnIG-U0o X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 711CA20021 X-Stat-Signature: tqot8kw7b5xoymfsz15trugkoriqyt7n X-Rspam-User: X-HE-Tag: 1692990335-376608 X-HE-Meta: U2FsdGVkX1+VaJiAahOQgDDP7jp9EVpuqKaUj4SU/qde3oSJDYNFdlBg5nPws8xtmY3mIpRWqd5opySFKDis1myk0JXj2besTyw1IeerYUsHQjJA5Uc7rEohCVATxVOYFG2XmYHEMqtPUuTVmcEEBfn73SK0sPr0o1d5UvrAXpiLOr0C26ayGicanm0EcadplY03HlP5pq9DNn3WmAG4JFeU7Dss1BrhMD607/k3ayHQ2e1PsL6JOByZXlnxtI9v7ZWSllR3khDcve982UwrY2r3FGdHitRVqmYqVbuZ22yzsFYmOvw17TMQJbc3smj5O3PcSCFZha6oJnGlRaeQE0NcXs7MvWRkEYByaTAYQP0kf+iIMWEuT+B3M6Uo8k6hibaUndCNOAAxmw6iAf4owNtH4TQ78qBgD+IJ8Vntsww6Sap+rroudqOWOdeHJAvpCl1GWzVW51csLnW5D9Q0Hf00A1tOJ5lpStpiprqj7cJQhcWBbUDilzK62bN/hBaeWTTVnkeYfc4DuPXoEZAjfPoYnMyYbxEEqhmmvh8wzN4fwlWF1az8WmgLYwriXQJkZyYxWgBtHlHoxH4TmBAWtA1wE30TVeeZrWr70B19yz8LSZUYAQqMpevtMkii2xgwNBYjye5JDnJCet/g7llX5epLoOzgpjvQBD+/oN7f4iIXYtpJFxSx2Gq+doGPynocbbunAkIFPq+s/nfivdQATz0SDt6iffrCkFkprSXS5lSxPd9F4U7ADdK667gUMByZlHA7UvDvaBfSTdfa7L7NePb0/0yb5zlpduchZf00ir8AzliKN6JX1kCzyjbem6AHp9PizLp8ZlctH+JpZsY+HCAP4cvr2eHdiSrsMSCzPelq22UtTM3DbW4d3/eooayscSZEdt31SXRD7SWHC9Lu+R21IU4vktvLvl5YF6X0PGVTEQ4F/XVW/20zryLooY2eZikp8lMjr4VrnccpD4x /zw/3hcn L+dI18j+PKvmAIeK3DrqfP8hYGiQNvixb9s8GwUeZ8ppwsoSDfREl4uPFuwTzoqFJ6jmnvI3+esVycWf0tAqyBOW/1YpWh3fVXwVtQ/Z/96lscOoIqb0iwHAyhfXSQ6npR2vS99JVdmiRAPNE3VFS4zxig03GIwei43tYd5uWB2Lttt8E/8DcP/9xjB3y7jynlmpp0aXOusnRO/EmOQciK2SRgNv/70Iudjpn53jvfUsWBpb2PWU3ccOCkz3ktfx+bfS1tpeLpuli9LHTfAV0Ab0UC42FXQLmcCQY8FrHjRi0JO+5SKRkkPf/nBa2RytFsNbJ5B4l1giLzbMetp4taVUa+7khsDl7kiVzO7HsXfOTve1bMvdq9EWWWXhQokrAyTBg02WNoZ6XRkdqKDqH0FrESmBRFIZmK6OD/3rC179UcvAXLLKYcdfpWVk+fTV6QPHMJ9kctEOFE3aK3J1pED8WE1Tp6onGEvr7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When adding hugetlb pages to the pool, we first create a list of the allocated pages before adding to the pool. Pass this list of pages to a new routine hugetlb_vmemmap_optimize_folios() for vmemmap optimization. We also modify the routine vmemmap_should_optimize() to check for pages that are already optimized. There are code paths that might request vmemmap optimization twice and we want to make sure this is not attempted. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 8 ++++++++ mm/hugetlb_vmemmap.c | 11 +++++++++++ mm/hugetlb_vmemmap.h | 5 +++++ 3 files changed, 24 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 452e7ed6dc36..3133dbd89696 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2223,6 +2223,14 @@ static void prep_and_add_allocated_folios(struct hstate *h, { struct folio *folio, *tmp_f; + /* + * Send list for bulk vmemmap optimization processing. + * prep_new_hugetlb_folio (called from prep_and_add_pool_folio) + * will also attempt to perform vmemmap optimization, but this + * is a NOOP if already done. + */ + hugetlb_vmemmap_optimize_folios(h, folio_list); + list_for_each_entry_safe(folio, tmp_f, folio_list, lru) prep_and_add_pool_folio(h, folio); diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 4b9734777f69..147018a504a6 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -482,6 +482,9 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) { + if (HPageVmemmapOptimized((struct page *)head)) + return false; + if (!READ_ONCE(vmemmap_optimize_enabled)) return false; @@ -570,6 +573,14 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) SetHPageVmemmapOptimized(head); } +void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) +{ + struct folio *folio; + + list_for_each_entry(folio, folio_list, lru) + hugetlb_vmemmap_optimize(h, &folio->page); +} + static struct ctl_table hugetlb_vmemmap_sysctls[] = { { .procname = "hugetlb_optimize_vmemmap", diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 25bd0e002431..036494e040ca 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -13,6 +13,7 @@ #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); +void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list); /* * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See @@ -47,6 +48,10 @@ static inline void hugetlb_vmemmap_optimize(const struct hstate *h, struct page { } +static inline void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) +{ +} + static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate *h) { return 0; From patchwork Fri Aug 25 19:04:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13366221 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7658DC71133 for ; Fri, 25 Aug 2023 19:05:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9247E68000D; Fri, 25 Aug 2023 15:05:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 85CB968000B; Fri, 25 Aug 2023 15:05:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 639E768000D; Fri, 25 Aug 2023 15:05:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 5052B68000B for ; Fri, 25 Aug 2023 15:05:51 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 2978412071A for ; Fri, 25 Aug 2023 19:05:51 +0000 (UTC) X-FDA: 81163556502.18.88FA394 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf06.hostedemail.com (Postfix) with ESMTP id 6AE71180014 for ; Fri, 25 Aug 2023 19:05:47 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=LCV6H6Gr; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b="OAmZc/4O"; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf06.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692990347; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3yqqBf3/EUJhnisgK8UhLekUfEAaatv4D5tHikWfcW4=; b=BwNm/8ki79SnH4vqHeJgJTY8/3i2WOW5yufaikUpDIbUVv3oGYjH7+dLkJHYQrgteG/GXI xecfUzpV7fh5I1VXsKygT+lV7+kbBWbQAUWKb6kfpA/0ehr2RGi9M2PD8vofubRQGM0xlp j3CwrqA4gIoSyfVJFUHygyCx4HuOBNk= ARC-Authentication-Results: i=2; imf06.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=LCV6H6Gr; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b="OAmZc/4O"; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf06.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1692990347; a=rsa-sha256; cv=pass; b=1BgWU50COR+boOfpf9MsuclGmUddOpUWOmkEfprf6WcbYw5F7Y2TIJAPqXAgSrafPqPmzs QVUYypv3nLWxDDSfdLEeOcDSy+PX7WYWamin2fd8f9DSmPvJvxhugrpQ79mKn4u1wghdLO d57H3GlrD2cBMh2oEceNXBJY2+MgZ4I= Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIEIrP010977; Fri, 25 Aug 2023 19:05:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=3yqqBf3/EUJhnisgK8UhLekUfEAaatv4D5tHikWfcW4=; b=LCV6H6GrK2qc8JPE7ZlMcsbBOPJrAw86cJ/5kRz2NgMYJ4wqofJhFm3JgM+zqxkBUD95 mThA4kFvetNXiUU3Am4VI21L4HsyyHNDcJJaNzCHE9789qiJ/1NsjYq4kWFENyRz+03W F8nhvEZbzPiSyk6XV9uMirZR9ITMHo6LLmNGFUmIODC1SijZGyX4jIiNA/YqxDfHj155 0sfGKJTJw7JV/yBzxE9D7Qa97852G3FFF2TUJi6MhOXUsyBpbwMMEWCZ/LgpPzE8zB15 FnRTXSDgbxoJ2BwRgiG40iJw7cjUzeV6+V/AWT5uhDuE5wHNxZJMt8ibs4i5WyQn1d0d VQ== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn1yv74ss-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:17 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIhRpw035774; Fri, 25 Aug 2023 19:05:16 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2043.outbound.protection.outlook.com [104.47.66.43]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yr77vn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:16 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AaMBVRSljPbIc0qe77chXvcGvfvJ0PYtMqt/JLEph5RfXd13Ue7f7WZq/ScN8NtIM9hRu0ifD/j8YUj8fte9Dsv0nt0hjISNhX8kNs0wS0tA415lwEpOw5NBlMRd/PRxM1ACRqwcR4EW8RvA02a7XpGoiwsHYwIVCiMasJgWBmDlLOwTYlQrr7uQM+ZN4nBShtAOuTjviVhqJF55lG45GJ3DsmudoL4/DtmqTiie8/9vMnZOZLTu3LkgExtT3FEGVaHsnknb9ytvl0s36jrADqznz2ziQxlZGXRNoNbfkO6PkKFcux966qCXgpzVmfDlmhKkB4vWBxlgSBmMGnD2Lw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3yqqBf3/EUJhnisgK8UhLekUfEAaatv4D5tHikWfcW4=; b=fxCkksHL0lvJwPGyBk3+6WT3mcB8zB3OXQHqPVLo3i2dG8b7ZJGEIQ8KZrDzZ4G9e86TIXeY2DOrOlnXfn4Pgk+cmG+6BtnhRxOXmjduQ5bp/0ybnGawAuL3AG2q5YA5+8TWSFXcEgOz2/8+HdG//aBZEG9Ue210WSDC0oDYFSt8XPuNzE5RdZ6Kl251nQC8U7RvnNiu5L7LJm9lW2EZpvS6bF/vdSEK92R/OFE3i5j5VDxoOarHC54tRq3Z8Xow1mPnqw95oYYr539TFZLWLB2Bea8hpKoPVEH/n+ZHdeTcbaiRiWw9kZXN024uENNbwcYE0H+Lj2W7PQSqEKs4DQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3yqqBf3/EUJhnisgK8UhLekUfEAaatv4D5tHikWfcW4=; b=OAmZc/4ORS3/ItrMeAzf5qWLA2nzpPHtj7R/GXXIf8LoO9VvP3b/0hAA7MBwwg16bBBQkJCJRczUQIfD4uiZ2YfLaTR6ATG0d5EBrkhno/iy33b8UR2yCjOipmOjJm07zOjnNSBg1ZxHGR5Wdq5hP6vNWp8SmmthCxE0hPhVjtI= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by DS0PR10MB6895.namprd10.prod.outlook.com (2603:10b6:8:131::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:05:09 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:05:09 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH 07/12] hugetlb: perform vmemmap restoration on a list of pages Date: Fri, 25 Aug 2023 12:04:27 -0700 Message-ID: <20230825190436.55045-8-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR03CA0179.namprd03.prod.outlook.com (2603:10b6:303:8d::34) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|DS0PR10MB6895:EE_ X-MS-Office365-Filtering-Correlation-Id: cd8336fd-8c41-481d-5fbf-08dba59e3378 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: gWBpvmOUKAZohKU7SVHXIsS6RXZ7jKx3FM9PbvhIn1rtfZEmLlhqKogWoDYG+tXqYki57aQpJa/ffxF0lig8Xb8zsk7EzRjqFEhiVg7wq6eq1Zrzn9qiAedNq89U9vUQYzJ1KCWk3Smn2pafZv/sXNakSs0Xp/L9Uy5pgf1Z0zDJoMI3YRvjhMO08hddkgyp2/5VJd4aaDNquAiMO4dsOYdKCf+cc5oNa2h7bi/pTPBRvxHTRZRy9RZTdCDVka6Xhwp0QgAcvGpd98wrOGQYUgsT04nb8DkKoY9yi5B7B+XwkBRUSNdHNyG00g1PRLQapyCWJCuigykNqs28l3QAhZKIjrUkT5FvXcMTJhB7daZt0zN4MNyU0xazVx2KiesPIH5IABnp/rDET9szruFhoAkMxgRkcAsKeEgsC10DpEInD3sA9+0jCf+TQgBukpHTyA1sSrMOOPKWZ//qdYovj+eqSu+x6JBRXRUrKsmIwHqzsjn86i48rF/lxHr8znEoAFtAS+cTnIJMWEEH/5U40SVOkTPK/JEEOhaoCTGxgHKXNxZP7eTObPvwcgxjtj0B X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(366004)(136003)(346002)(376002)(396003)(186009)(451199024)(1800799009)(54906003)(66476007)(66556008)(66946007)(316002)(478600001)(26005)(38100700002)(6666004)(41300700001)(6486002)(6506007)(86362001)(6512007)(2906002)(83380400001)(8676002)(8936002)(2616005)(4326008)(107886003)(5660300002)(7416002)(44832011)(1076003)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Guh3FDSOHYCmwWMJCnG0kxAbT7HhTPqf+R5FXkuhb0UULlrui9ro4MYLF5ssnpx+FKEH+Hqv4+/7b188P3yxY3kkFJ+RenF+PqSQRZxafL/AbvbWXgqp26BYjvxO26soLFmtX2VxICHEQJCDr54h/jlrvAHvlbNqVMhUoE+FaXK0TMvjdt/m2brGcdmHn0+3p6MLf2NQcmvDTe35206L1jcTNw09/FS1BmoiropyJORluh6Cxlc+trZXHncknypW6meGIXgI7tUhgL9vRwUU3fopFhAoKMYEL6nUfMPokqVWtZLfRwh8mZKXM3CBBU1SWNOIkJiwiz+QS7BY9AOPJJgSOdxtf99jnbswPDGMm7h1/oW384UmFvRSSbSNAk35cCgQ6n5VlFWd5QnRxlw+m0P7ySu9WJb749116QovDDh8IsQ0QBi9yz8f1+YG5xv/9Ru527+ZMUGXqAZgMtIwNgXBQXB0EOSdnxrlkTwp6+uSvpnkhKMdar9kEZJcNgLBFdh2TiNu3xzyFLMfGU8VI4i6LCzidDLKqMeiO3WR43DFp3QSjihOGo4GU69F8g7M1wnClAeZh7nQbOtpVQhc/r5z9NbedPz6XgZl035ih2tHmHt7cA16baM98dUGniB61T/kC6tuauveyESuqgleVpTKpQCTbIkYPKmrtnqJ6ecUXM41hOvVFEnep2GEZNep4X7r/QZ/mIYPC1KVH2EODr83B/bSk3M+GgHSKHL6Hjwo3RUVEcWTOAdObFk04q3cwggw/lV/ZiU8X8OIYLZeALlJ2u6Qsf3YXRNaPYh86sR+t6/FmYhlortrcQzzAakwCUZplm4ZpxH9snnQ2Y/ze7KNra+l6av5FbaibqeJ5c4k8LbISEi941NqvvsMote8Q+hG2q6FFs7Yyd/6xXzwGFhiTrr4oUy52ZmLvQYb109/THbLqyoci8ytif7jYStnRbMDFkHS7S8FLRd7C04lsbF6asMR3v/AbWPIKvEcgOzYT3Sq7d7Y2L8jVzjo7VVfXPwvtMqXzw47fHLT5pdfkFNXooSTIP8UEPDKH5SiaQnmvShnIPf8L+bYqr0zWPo/8m+VByAwDlNpAUKhCsuqYTKW9VYEpfSP/9Kigf6NlXvaut92n6sFMoUyvm9I8TBkcTKI/t6WaJPn9GWm7kaPadJ8Z0nV6itKT0Bp05HgbtEVSeoQofymR4mT/zAnDAX5CqbemGS1EMzAdmXfHaDooh129E75++r8Iz3vrgq9PiO7cMd9Q717RNT+TH/sdDzqop9Z/ut8+PBbqYpAg5c1di3x5B+gqvTUXf55brLKDRtfLB7S/8WDGvLXrWsH2r2az6YzH6N7qsJUEKyrTpPIEyTtAJ88RfMxjnn6XsuxOXib9td1i1oIUD73/Nqs8yx1gl+ROlOGrvHD0rVPajFxE8q9Q8BQX5A1kUdhmljcpW/ZhRCb5AvoLvUPQnZRqEaIK+4EYSzPexqep54NuG0DnXSGd+QGRhsdi2FLoKi/mERJlGm64EuPMLZDszdaJuJ2QBJKlCzVX/sDxw1m95pUYGzw8tygFZbNXuswqVdcJr+DxW+YCI+EETWrqyVcqLy960JVbLewAzWjtcDc3uG8tQ== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: DUXQQzwraCkXPAGLcbEU47Jvz7FPADnT6AKCtWvOgL8KBVKzlb+cLNA3E4qyKakqfCNRqnk30S46M+iAwjQxs4Lkx/SuKczsVpsOq9RBuxfQ2vhu7BRMMYiC3itjI3z7AuseH1QIabA+dznq6hz0QEh7c3YSNSyOQcI5mN8lw0NUjLLSKFF+MqXruYoqBji2xI9WRB8MdBBXAuPJRMcTEJO6UYsqTjT/5JHRP4Y11HLhgUBSZm6cOHyx+psl5qYa+Dh8sjkzr58x2iRXV5FUcrmmszSRvhZ9rfomC2XuYHulf+smiwxgQozGjJytA5awnumNoSPDPNtkSFZ+QOO4FvbZsop6+gsT0UKMeDlXnYE5/IvikS+BoVnaWbPdE+DpuwAhLNyhcRqBWo/vMhKezLlywH0q8FQfRWg06WnqI/1+pw0h9JCyxxOVajOmQbGc8VDwJ619RNNtBHzIydPx/5fRMyIsHbjeyecC6tHX6oRGEC18HPMcW5/J7m/nVGtZ7aO0xZWyK95CaPPubjIIQxjrVMYk4Q3lURg5ymGGHnqSh7AFppJ41kTOF5B42YO3I7Xmc+7c+YM1CQKWi5vEvsx07RMXIhMQszBBdc94KeBhsRv+NE2nVvXlsLjE9tKXsM5bnnx9BQwbK1Sswz+GTgpOPQMfBqxU3u8YJPxjFlOS4OGK0ekBYYq5mFCy1edvOovsknGrz17i8NEbAOif+oapbrAn7QRu6HvesDdehHQKFiPJBRWquD4XiF/uwJ9mYZTZ2t2jdu4zXiyvDAlCwbr5QVWlAFVepjHEuSRioGtwCnrDWvllLvjX7NRp3lzrxL+x60QgKDWvK8/sWrWKhrCwhBcT3bEnmkIrtcH/CZjWllIOtKyQrSA9Gxx4FQYNLkTIp47imrYzv/rV5rDTU4mjvaRi0FH30/Lpap91ijkzQPqFHxiNwDekXGER3zvW4oqriH4u8XeKJAd31EWe9WUf980ZyScRNa7FD9PDaCC+K53KeycvI3LUooMgmvdaNbkSFMs8/9u9nwbyCYp17CZnLXszQKgttRUNoHJ118i+lXL/xX2c4qHjq3u/VbQYcvcV47YWOww1ODsSwhvhYlikzcgCsZ4NixHBdbmY1rMIqhYCTsMMqnrZW2rIdtIg X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: cd8336fd-8c41-481d-5fbf-08dba59e3378 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:05:09.1211 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: x1VXRNSRdiJYI2eVKdL8GpoYyZ2GyfvXKMfWooyCQRAkeKdmZdD7gJb/3pMYFP/9uPxXHri+Q/HKH0bmgmchVw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB6895 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 malwarescore=0 adultscore=0 suspectscore=0 mlxscore=0 spamscore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-GUID: r70cOVOm7hPTyVVRs7esswLDi7Jym02f X-Proofpoint-ORIG-GUID: r70cOVOm7hPTyVVRs7esswLDi7Jym02f X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 6AE71180014 X-Stat-Signature: cw3gj3wdhbytwzqpbm847zspfq1ekakw X-HE-Tag: 1692990347-612368 X-HE-Meta: U2FsdGVkX1+qgVr6WIgTOmGbVRpZ/mNEirwuSBD/2cPwTIq7FGv4PebE/hRpkscdX6Y1c0o41ZdfdcW2//3lybq4+s2RzusiRhJ1U64ebPOgoZxWkBk3002z5UgFWSM+fwqA8TZv6ysladZmnIOX8kc9S3aU/IKPuTlU8g4mgMhrG8lLpoPEPDTIQAngarSA4ugDgxl2F3UdjQHsUmg0+/LX2/ceqx67M0vLizq2V1+oCwMpBrDNYnCziSu3Od9E7qiuJdfPXPieybUxi1H+/LHe7uXNvgpDPZqMCqTSmova3T1b9Z/zRgaOup54y30icB8eLV8nmj4PBRj+agU0cL+xsa+oudS5c8TU5plahj2uMMZQ+AHYEO4KWyQOqNwq9REwfB4z8aAW6mRsMNG57BxvqMp6QYBRbbZDIdZtsvYpH75XcoTq8MpspmTWDHLLftkAM4pFcwEdgg06NGiPlKsGh5eVAaGQx7EJe1Jeq/KvSGNrndtIfztYKgFRMdGm7oqx3xR40QiKHHlgJLXAR/ASfaP0AdwrK3p5frAqgpdY/WIvPAVYJJ99GiFZSrkm4ieXi8N0KvdCeY1J3QNjlH+2O3cMpNbJUt2paDx8Xf7AMZfnShEEF7U6WrYtqZ4G2E/ujDs49Mk1axNkvhMMAQguN7B6aZlL0klgoJGUer+tpZwpedYnukELkmjyoOrrwW46nP0O7bgGAQSz02OXmzqMKONMIMdP7G1WZPpIvhGY2YmcEP8Z/0YGOanyfON4K5RismUT65Ggnwhjnc/51U0tZTHHINsETdy5IDNFMPzGuBsDr8XScifHG5UKZVsklh+Oux/2OL1AjpYsX3MdumMO0fs6Ym7TCegwDr/G0bRyf6CH3y+UVTYwsIycEcwr2Y4LgRePZNAhsTtQyqlexKCgs0ae2wuDk9mh28Rr2vf1kPWMaMv79BDHWsekWc2cPAb5ayoNETmCBLb2n1A 5Hu4bEH/ oIIA1cE2wXsBg9ry0Jo9jIa580uvqpr+8odMMjeL44g7kHy6fTlho1+Zi5p4h52LpyXaBJIXxbQ56UBADflbcd8059Y6JGFGUSX7HKWJnmhWuFiE08FCTobZ9cd4Ywvln0nexDMqNHwrousKd9WKdfDolQ0aEvmgpEni0xWKv4ilJacnxlGL90JqDSE2mJ10nqzhIR2ZziSu5yog3DbC6uKPqY18FvoxqGjH4QomXK980VfdrOp9JbWaSRgJznmhNFbGVjWEuyg6wIW/ELxvplEQ6NZlALJfbux0JjmhAzAyabJWYkG9qT40NkKMwASnvKobdD83MYmYrKgvZavtHza25miWRSrfipZ3n8ZnmYniJj+S/xTyu/7boLK7hzaVcauy08qKshArI7VKmWLJwaBEgdu0538F8cN76lBZNmkT797tvoXbNasCDy5XGLY5U2slc4FY+ZWS/Yxhav63vS9MRAjA1JFofGoNfPJKQQVElagAhNMLaGl4zlHAVOARCz0m9 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When removing hugetlb pages from the pool, we first create a list of removed pages and then free those pages back to low level allocators. Part of the 'freeing process' is to restore vmemmap for all base pages if necessary. Pass this list of pages to a new routine hugetlb_vmemmap_restore_folios() so that vmemmap restoration can be performed in bulk. Signed-off-by: Mike Kravetz --- mm/hugetlb.c | 3 +++ mm/hugetlb_vmemmap.c | 8 ++++++++ mm/hugetlb_vmemmap.h | 6 ++++++ 3 files changed, 17 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 3133dbd89696..1bde5e234d5c 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1833,6 +1833,9 @@ static void update_and_free_pages_bulk(struct hstate *h, struct list_head *list) { struct folio *folio, *t_folio; + /* First restore vmemmap for all pages on list. */ + hugetlb_vmemmap_restore_folios(h, list); + list_for_each_entry_safe(folio, t_folio, list, lru) { update_and_free_hugetlb_folio(h, folio, false); cond_resched(); diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 147018a504a6..d5e6b6c76dce 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -479,6 +479,14 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) return ret; } +void hugetlb_vmemmap_restore_folios(const struct hstate *h, struct list_head *folio_list) +{ + struct folio *folio; + + list_for_each_entry(folio, folio_list, lru) + hugetlb_vmemmap_restore(h, &folio->page); +} + /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) { diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 036494e040ca..b7074672ceb2 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -12,6 +12,7 @@ #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); +void hugetlb_vmemmap_restore_folios(const struct hstate *h, struct list_head *folio_list); void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list); @@ -44,6 +45,11 @@ static inline int hugetlb_vmemmap_restore(const struct hstate *h, struct page *h return 0; } +static inline void hugetlb_vmemmap_restore_folios(const struct hstate *h, struct list_head *folio_list) +{ + return 0; +} + static inline void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) { } From patchwork Fri Aug 25 19:04:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13366219 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAC24C3DA66 for ; Fri, 25 Aug 2023 19:05:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 83827680001; Fri, 25 Aug 2023 15:05:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7C0862800D0; Fri, 25 Aug 2023 15:05:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 59C3F680001; Fri, 25 Aug 2023 15:05:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 43A8F2800D0 for ; Fri, 25 Aug 2023 15:05:46 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 117271C9B4F for ; Fri, 25 Aug 2023 19:05:46 +0000 (UTC) X-FDA: 81163556292.27.F7DAF03 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf10.hostedemail.com (Postfix) with ESMTP id 15150C0008 for ; Fri, 25 Aug 2023 19:05:41 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=IhTUKsod; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b="odZEN5/L"; spf=pass (imf10.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1692990342; a=rsa-sha256; cv=pass; b=x+4UUyA+J23LmTD7KxR2RCfn5iEmpIgg1EgPsE2QyeC3x9PpV1QQ32uB7YUIWlIGDGjkPM XqI/goNRrWOFtj/CzRq2WNOK36uFXlHaoaWTf91ECJYAZLYwtUGuakQdBZNYe8vTT8VIO9 cfjXAJf238+k0q0W7TacBA2aB4XGROs= ARC-Authentication-Results: i=2; imf10.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=IhTUKsod; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b="odZEN5/L"; spf=pass (imf10.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692990342; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JDqlank9XT0iTHl7uJPws1K8LVzypgRE+WIqntsMg+Y=; b=18LhS6qXyOnbnMzzg4SidLZgOJMDZGE++4dcLDzh5PM3AacyGKZIzr5UDoO/lDqPe6dWsh li5HT/4jKZx+eqIXa2pzP2HEZ9Y3uDfmHYkXblAihNmpV0s0EaEjlr8l9eqrGLJR6iOhNs jyXif1rW6mR9E8M1dQZD5Ps3/uZdEUc= Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIEPRi031105; Fri, 25 Aug 2023 19:05:19 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=JDqlank9XT0iTHl7uJPws1K8LVzypgRE+WIqntsMg+Y=; b=IhTUKsod5lykf5PsPX8PHVHBhgVkcRSXqe2IOxauGL1eW0e8iSLRsAMbWAmgk0pAYx7r P4DcvlpS1Um9Q5m31XR6xL9roKH0blg4Ul30NanmL2hKbA/Uem9pHJUULDR/OCuBsJmN HTwLRWLuR50eJc6K7+iu+Py/KvNfLfVfb9ookm38g5dMJDOIk51OSebsxG0EuH3djnmE 3+RC8zVwzU6o2hBYb5Cm09QEWD2atrSXcK4PZK9KI3uHyMLVHnKDhSKmaIJaTffeXcMH q1trYqQGGa+pBP+3rzUVg+ayRfmIAky3DmTEnVmjYmDN4ofb5l1vseYJ1KYkYVblIYLL Eg== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn20cq0e0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:18 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PINm49036073; Fri, 25 Aug 2023 19:05:17 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2046.outbound.protection.outlook.com [104.47.66.46]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yxqsmg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:17 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JwPgDGm+y4wgSJF6v1OViwtjzGfy0sZG9nhPBOZ2sSo1Vq0BCa1w8Q7tCBurLdXNdIJZQB9yCoIJdgT719z/82coz3hjW0mx1mru/Pr1cUw7RQsKA5bGpZIBNqsbzP2gdg/DP0SCUyJxsxeDdOPzwTa5e+pE1pgXLzYXJJAgzyhwBEUiZmNX0HPP4tgLIi4WfaPJzw9MtxyYlGlsE7Fl+ShuWR6bMVGcwi22McLbud3afMvZY5YZg81Rfsd/zX7dqoKV97rZ7v91iRA9uTgNjTAdqLnMNrg4Fa4Uiy5d1qhe0w8fSzqiTnkR4bc/548MOzGy+Ho97U28XqrCNmOdqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JDqlank9XT0iTHl7uJPws1K8LVzypgRE+WIqntsMg+Y=; b=E87cFLV7YQF957Afw5VM7jXbV+xKZD4a60FnVUZrYXWDLeH9AcQbrzkqxaroARh0wJT3yZQ2PZpxK9VQ13VIB+QgKn8Ng5KFTen1aRnw0TmEN53pL0k4/qNvS+gOwH+chCB9ptVG3dwpd+X7f+sYUIzX2VFnpw4O7SOIcuHSQvhZ55HdtWKlP05KZGApNFC3Ug6yAHKLQbjsCgMHshJ5swQQEH2qTFDWVsgZhXdgIDV38EXNJMnU11wYX1GRkqx+THwXK6+RqSwd/lXo/+WvBs/juv+Trhz5TnHiilYlKWA48BMbkoOVatoPGGwwzLjtyQAn2b5sbXoks8vKm3IGvQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JDqlank9XT0iTHl7uJPws1K8LVzypgRE+WIqntsMg+Y=; b=odZEN5/LwlJEIqxrW8QHKwNdm9YxXZKCR10GiklbtNjI1LEbAx+cJ0Txr32B3JUqnsuUmo6yceLD7dSuElqKCTOi6jMGwEHxoLt0/vTpdLvbrrS6AW+xotN0zpuLoKIa+yK6LFdF3QGQ57agnOm0iwNO0go/thg15JHedfl1gIA= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by PH7PR10MB6580.namprd10.prod.outlook.com (2603:10b6:510:207::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:05:13 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:05:12 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH 08/12] hugetlb: batch freeing of vmemmap pages Date: Fri, 25 Aug 2023 12:04:28 -0700 Message-ID: <20230825190436.55045-9-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR03CA0103.namprd03.prod.outlook.com (2603:10b6:303:b7::18) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|PH7PR10MB6580:EE_ X-MS-Office365-Filtering-Correlation-Id: faa1ee75-cf92-461f-6b21-08dba59e3582 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1G07+/q13KIfsIP4PVmRWyQ/Blx7hgWZDi4K0J8+tpF56E3KB3oLL0A13Znha3YDvnL9FqPNL91GtSxm/i95g22jfnawDeSwx4wDzoXP3SMpxnPerEN7IWh6545wXA0nCK1fTYTGnzELhjWYooacVKh3ldjbn3WbKoKKWMaPGyPpMW22+er79BosrZ7j0UG+AzBUjSj9xrtZQj26L049ekJjIBHAseqlf/8RgHFsabelj/+YZSkAmPW99qlQnbGLAECXnd3c8/ZXq/QkAJdK53MWmnb0XwEjZRd5tGT7Lyp/3LN0VUYN8rp4nfAXsY7HF/T9qF7HXIeni73y0QhcZQKyEJtI4wCEUOwoTCi2FiwTiknMfcEx1TsUuMbSsb+dOjIj1rf4eyL0PL4AfZxicZz3b5Deaot4EhOJjOKANFeakhBxuyg0AQlu4R9L5+j0mEMDdS8oYD1pP1e/JBRYf1XdnaTu5JPZHz/BmNpA3RAGzWcFvgoFDulNLLfmqqRdQDlWMJ/Ue7VmZEYNHmAxuplGjtXdTwTL25Rg7PhbqrWd1h0iGaxJrmrGN/Q9pOnQ X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366004)(136003)(346002)(39860400002)(376002)(396003)(451199024)(186009)(1800799009)(66946007)(54906003)(66556008)(66476007)(316002)(478600001)(26005)(44832011)(6666004)(38100700002)(41300700001)(86362001)(6486002)(6512007)(6506007)(2906002)(4326008)(8676002)(8936002)(1076003)(2616005)(107886003)(5660300002)(7416002)(83380400001)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Ew38Yk2Fh01yvjBlSgvmdHu6QAmLurm7gQXxTBKvYoLxUoObZx2Xgo9ReNtr/XVAfH+YZMDXRyOMaSFH8Y8VaI64x35cBMu33HVrGNKtViE0gieDHYrs9N9y1tv2ORfnOP8r0j0Rz6BroxDWU+BKLusiFA55gTlQTHDdmH8ZCKgg4udyBY2TcQySMNpV4NvzkFm0wCEQ2iUGUi+n/2VnLYtQsrZ2JPkjrz9IntHVpUtknvoviIa6QGLxCUgRtwMB3kmPQMBJhPa7wLuOIkEgOafm6A/l2GwzJVSljedCna1jpB5vVHz81VCEdYsReC5zfWmSopyI40TevfJIOWqdQcieMIGjnq4dvrx7QHuLicEX7lS9t7AH0z0GkjyqxjbXHgaimxL0JOBJ5/NAr3YW72Mt6agU+NKPUhl/HMbIDEQX53jlnJ/EzrzH5pn3dSqOmXZlfiydS+pDjsCUvxa6HVRzf5zy7/8pFia/52PSJ689rGZyM70yX4beg3YB+Mc0+lO8eJcosJw5+kJT62nl4/6dVDwi3KuorjZZYt2NNPGf9RVL7rO0BCMx4fDZaX/zmbc6C1rJwfloAIuJYUo5EK3uBzCVAyVxAL7ypzNEhmUfqUGbtLksHbu6hi6bu05x0kObpoVUF09tKvpF1BaRRbkfCB1qemALGzbyASUhYSFwskZoH5clKCCMqvjbAdbtE5S1PKY7NT/ucPihSOkpI07ZTAg4OBw0zH5fRCncMVqbagMM+k49kyaxwpIvWje70urCQzh1b6hqQWvcHf1VLEHj5IWzaRZ/d4LK0+0z5sXO7LtxPlHHcwvleM23SXnQTtshI2GZMuawA1G1JB/ECyT+XHM4n1zEwJ2vqJI8p407mchr9ztx6/GslXFpLzRFSMB+m+P7MwXa2gU/wUQSMVouPssE2TlTAYZ49f685sqOm/l5Xx1ZKaiAfUd0+kLOWr9A8d1aTXEll4HlfNLMmXWB74trGgeWWNNtniwJ8EKBMEkk1voXwC94mweEhYhjiyWKOUEdWF3bLb5w3N9xcyoH9m2uT20Y8gEyqsce38E/MBzpPy+xmaabI5Qmg502bPwP2dSyqOZjxqVBORAwyAMMvTxQBi4SbL+fXNADJzZVQgv80PT/TyNKstYLTl1p+rbhgZEthmoHcRj6TqRyiWlnkE+OqOmaTzPDI2XGHqWoSQ6rN4CHRaKUiduJKzsst0+UeVbGwNoRR8bN6i9NL7murJU16BmhllKbdS16HHgkm7Le2WOhkmv6uGtoaQ6ZzQOhDdyI1IEw6i+Lxd9hqQ3cbdceapdqqjEKwsOIwTzoKCCrrjXskgQzMhz263Kqprc584yGRL0CH+v3s8Qs7S1hOr7VBAkAw/Ou6RD/qtI3OJg4yffCM8y4oM2V00IdhYp9uVKdOW5NHk7TF+g4WKBM3nmO0OgAPdQOEueVaBywQvcnoYg8zNJQ0JdLPrZi5dCbpdP/cbYhHhuUnBu5nC5Pq5JRwchTF+0nGJ5OvZQFXHkRLBP67w5VXQTgSiUvITUsuHUnHO8qNxZwMWqcZMh5BJEJOrW8sl8qvHuWe6WI5kWHEeGDMgILPyn93kN+NT6GaRTunrs2fPS5V2tB8Q== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: u37lmZ0ynFClxHAdrhqbcrbYcI7Yg+/TTocAIB8ZgyRFwGg/yWXsMEEh5lqrl0Njbjxa/trFX6ehFhlgB4Z1UeEPgCX4rFtN3gR9uB1GCQEJ0mu3C0s8c0j6ETzs42IqHWOpyOnsbAxHWv+YOB3aD2/CmPDf25h7IEvSKv/ul2TrUZ3JPm7JYPBmG+Zfjz2PdTA/frwFuFmqRe8kRiKYr1UwUPUaWcai/v2dC6YGrDijWsyWq+59o4KkLPqcSnQdDhotTJ5i+7jIGZ2NeytMv3LA5V4JOw725tSReMIO8J3X+rm/+nk5dSycLBzZlyw8BASoZP0uXZ8NPzXrBxAXKhP+3BJnA0VnRBv5NuvAgr3ktSbePplFxDaEKjpbK7x7ZFNvIvgFE0/cH2Fwq+h3X0JMxjB/7PrOUhADgtYN2US7s0Pjno4+qMbZrKHcM+M8tEV4a/IOefUo570n6xYKzUTuvysAu50TNRyefBdrjZ07wGs8k2KDwCKs52I70j5qQI+NooNqMIuYn3GkphBR+FHIhSjoZFD1fngU92LaKT+fC1wdAf78tJJszEhpdd3RJ8FKwDiV1UaoLSz4a260nkvAJQcSP4zZ4ERqThbSU03z5pIq/SLESarsOC/7EEv97sXlQZYgy65X9QQK9OGwz7uBEZqMzX32ttw4eJkoSWsC/fjamMfT3PEwZWh60rN/rdTK9v9of56RycGZKRZHwpjeMCYm5ulJzFu8zpJBqZSBwk8/ZlXO6uK3pIQNH+rA7vNpLcdoqXQN486/O7f+aWlthiYCRlw60s3chQ09hdqVcolLjWmoYA8q/5ifbL2sB/l2GuowqSspEFICT5dKRdta7FU+BljOg6b0xw83KKcOZ9iE/gSGTwJWRwvnNnx/N+sQbhS36/yxvTLYNCnS+Mj/oNSelHRE/rdq5Puf6lC7aowPixsABuAXl8FIeoJODwJcINZ/rq/ReUnHd67RXuwnktVkrm1/sA99+ZHBvWPwOXSCbLD6KA/I+Apq9TGOyxcOmJi42Uqar7/ETQcIMr+Crntg/qhN2uqgdvCNADEgrwgL7mb+8XToVSCdI5COfdR6T5aLXKhS7EdOnO6pUGSjCpKAlzcXKw3iLQCDZ5LusWPhf7AUtV6DaIfYxcRJ X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: faa1ee75-cf92-461f-6b21-08dba59e3582 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:05:12.6238 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: n+8AA02CWG5p98oyE4adDLPTmNWr1BYcj6+itQsiN7REg0kxRt62bHIn8YVjDIIfcDTake+T13/OKoAGNgIZSQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR10MB6580 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=0 malwarescore=0 spamscore=0 phishscore=0 mlxlogscore=999 adultscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-ORIG-GUID: Ws-EtQvUChEPHTSo6S-b3jXxBZCOCRNT X-Proofpoint-GUID: Ws-EtQvUChEPHTSo6S-b3jXxBZCOCRNT X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 15150C0008 X-Stat-Signature: gqecuw1xbhcwbganouiiz6uwh39s3g6d X-Rspam-User: X-HE-Tag: 1692990341-372018 X-HE-Meta: U2FsdGVkX19bIhuwTdNrCW3uSF71wlAgvJP57wHjyTPp0i8kuYsaopShETkT7SSkLXYQL375f9FEiacIPKVD4H0/Z6eBz681rg7SLfE75a66To9IDJxSpesxiHLiI0QRWozuytaZ681a/Z73fR1p6I0ObPq9o5r53hHF9oUohWebIJRTYk37i/BL1mLInIeQ2sRYrS2J8x3eoFzn7qrQw/c+Fej/GIHWR+Y2h6it9W9+etdrhpBcI8DvQkJtxIL/A/3b796DpNJA9a8VgizVCpa19KMrDjr9+0QTD+ogEBZ8JuABHPxUeTqTtWy9NP61xIG/EhgVVKRnCv6YqnUfu+g9hCxS5wxCo9dh6USgQIoHcTdWyM8AkANM9RUmTmCnI7s2a2ipUJVHy6oTrn65yBijs1yPwwsyzCOtQIfwOUt2Om9ijRiGXmUKs0fPIlnisAo++P3jJCFd+hhw80JEvShJ8545buqvNyc+SJdWOzbtKJmSxaQvXwzgra1J+jsvNPUCaBGGXsmr3qBBTFUfTiCK2MTnEkz543xVUY7IxxYGDX+UNgHkSv3uSaeWFwMq4+lgo2PwJse4hgDzaYpVD4lvABro9Px6cl5WQmUuNwdybCTIjn+soOy4sdPFm4m928X4y0NnfPnPSAsueBSMfpad0FZWAX0ubK4mmBFJYAm3wkzqbOuglRI4AYccYqrfGNL8gvlWqUuJYlAcXfe5mbCXV/SQ4RfI9vssSXqlmHw2yvvFwH+Vw1RHTSyCjI9bmPvQe6Xh66Z058lzK3qwuy7Aao6pv1iobjlLdHFtwBzYMrPtRrcklZHoua/k5I9cVvUebQsIjwX/gZnu1ytqASHQj1UMCdJ0OxLWvttVTHlcZLFwy6DI+SUFjAgj6MBtoiR0MTpx8hV0XeReff/r4fJUeCkaOHWbIZdTZMrU1gooC5DKl1hB4K0ZTCOQ6McETh7q2rnBJsm5dM6lBC5 3RlhJ6bB Xh+zoQ3Mk8hUU4bhLUR3Zi/33DrTTD1CkNg0M88qb6ACpSLCJulWmZwhrlFoeP0POEplUWuMQCb2vClgDXnSmVCh5SYKuuWnQPJqyofJmGT+15YyeqUzBKMnvXKrXxr73iTr2IxB/UtvYm7YOGCCkf3XT9Bzu09anB6HsDYqt0eCxiMfUOHtcBYb35z9leP2T2h85DR7F3ovoce9OSh5N/zS1MsA8qgv4sX0RccALyJusNn+OxiKzMAH8H22qFEAt4vdqMoC7noJpl6E8hPyLKQctDYUdwYQ51jXOj8tuOR1dPiSuIMv80+sF00vNMos7WxlLkvsAvKl47yBfUnpxiYRMkOQsE4yFGlHFfVXD+mHT9Q2izZA7IGbOdlXCRvMGKMyPzJkgFrhMzXQ5j+DxeFKCJAoPOBYci6Ja85JAKVUZ9oIB0PFgpbNTuav+zwNTl6sixTujpxm6t6lBYt9x225/NxapAxvW/F0XqkNSQRYhbZhbSRdd+zEcZ9s6LdTyw5X7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that batching of hugetlb vmemmap optimization processing is possible, batch the freeing of vmemmap pages. When freeing vmemmap pages for a hugetlb page, we add them to a list that is freed after the entire batch has been processed. This enhances the ability to return contiguous ranges of memory to the low level allocators. Signed-off-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 56 ++++++++++++++++++++++++++++++++------------ 1 file changed, 41 insertions(+), 15 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index d5e6b6c76dce..e390170c0887 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -305,11 +305,14 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, * @end: end address of the vmemmap virtual address range that we want to * remap. * @reuse: reuse address. + * @bulk_pages: list to deposit vmemmap pages to be freed in bulk operations + * or NULL in non-bulk case; * * Return: %0 on success, negative error code otherwise. */ static int vmemmap_remap_free(unsigned long start, unsigned long end, - unsigned long reuse) + unsigned long reuse, + struct list_head *bulk_pages) { int ret; LIST_HEAD(vmemmap_pages); @@ -372,7 +375,14 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, } mmap_read_unlock(&init_mm); - free_vmemmap_page_list(&vmemmap_pages); + /* + * if performing bulk operation, do not free pages here. + * rather add them to the bulk list + */ + if (!bulk_pages) + free_vmemmap_page_list(&vmemmap_pages); + else + list_splice(&vmemmap_pages, bulk_pages); return ret; } @@ -546,17 +556,9 @@ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *h return true; } -/** - * hugetlb_vmemmap_optimize - optimize @head page's vmemmap pages. - * @h: struct hstate. - * @head: the head page whose vmemmap pages will be optimized. - * - * This function only tries to optimize @head's vmemmap pages and does not - * guarantee that the optimization will succeed after it returns. The caller - * can use HPageVmemmapOptimized(@head) to detect if @head's vmemmap pages - * have been optimized. - */ -void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) +static void __hugetlb_vmemmap_optimize(const struct hstate *h, + struct page *head, + struct list_head *bulk_pages) { unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; unsigned long vmemmap_reuse; @@ -575,18 +577,42 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) * to the page which @vmemmap_reuse is mapped to, then free the pages * which the range [@vmemmap_start, @vmemmap_end] is mapped to. */ - if (vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse)) + if (vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse, bulk_pages)) static_branch_dec(&hugetlb_optimize_vmemmap_key); else SetHPageVmemmapOptimized(head); } +/** + * hugetlb_vmemmap_optimize - optimize @head page's vmemmap pages. + * @h: struct hstate. + * @head: the head page whose vmemmap pages will be optimized. + * + * This function only tries to optimize @head's vmemmap pages and does not + * guarantee that the optimization will succeed after it returns. The caller + * can use HPageVmemmapOptimized(@head) to detect if @head's vmemmap pages + * have been optimized. + */ +void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) +{ + __hugetlb_vmemmap_optimize(h, head, NULL); +} + +void hugetlb_vmemmap_optimize_bulk(const struct hstate *h, struct page *head, + struct list_head *bulk_pages) +{ + __hugetlb_vmemmap_optimize(h, head, bulk_pages); +} + void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) { struct folio *folio; + LIST_HEAD(vmemmap_pages); list_for_each_entry(folio, folio_list, lru) - hugetlb_vmemmap_optimize(h, &folio->page); + hugetlb_vmemmap_optimize_bulk(h, &folio->page, &vmemmap_pages); + + free_vmemmap_page_list(&vmemmap_pages); } static struct ctl_table hugetlb_vmemmap_sysctls[] = { From patchwork Fri Aug 25 19:04:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13366220 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14A85C6787B for ; Fri, 25 Aug 2023 19:05:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E056468000C; Fri, 25 Aug 2023 15:05:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D662168000B; Fri, 25 Aug 2023 15:05:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B470C68000C; Fri, 25 Aug 2023 15:05:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A053668000B for ; Fri, 25 Aug 2023 15:05:47 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6607DB2440 for ; Fri, 25 Aug 2023 19:05:47 +0000 (UTC) X-FDA: 81163556334.12.38891FE Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf02.hostedemail.com (Postfix) with ESMTP id 81EFA8002A for ; Fri, 25 Aug 2023 19:05:43 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=bWEj0ckd; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=Ryzbc2cm; dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf02.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692990344; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0ahRRIn0ZNn8lfsDqjwet4c+B7wLB9LxrxIa3Q88Osc=; b=AvPbzZuUusS5tKX5hthx0+5Or7d8aVmIxu0b7bZHT+0QoR//eFGh3AKHypK/X5HGuqCTuq 8iDSGa3R0tab5wWdFe7ARkluPC0KDQf4nnf3PxObp3FyhHMhb3qCtxSrZoHgkKCE/AYxkG FZCYY0I69OuKSg5MXXd9TP7HL7+6jH4= ARC-Authentication-Results: i=2; imf02.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=bWEj0ckd; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=Ryzbc2cm; dmarc=pass (policy=none) header.from=oracle.com; spf=pass (imf02.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1692990344; a=rsa-sha256; cv=pass; b=URUA4pYQ1/QiZHe8jjC0VTFsrzzA0yRm9SQP9u4C9eLEydauKaBSrKDvwUbTBW1so6MQih IbGxcCwzlBoRGGmawId+KSayrMoWSW2X0KhNQyiqpUjCTZHsl3Hx8qAn+0wnNdnodkqq1U QNDO8Vjrvj3DvzFaPccGrUHTCRiOtig= Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIESZ0031124; Fri, 25 Aug 2023 19:05:19 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=0ahRRIn0ZNn8lfsDqjwet4c+B7wLB9LxrxIa3Q88Osc=; b=bWEj0ckdoCz9QYuzTiKhsyrwYkxPiRSZKmSCHTi+WDNv5P08bLhaNoeXFnR8oGQPVGjC Th43vBH5n+6Cmej3jr2Em274tWau3cMeFolIULCUo7V/EbrIH/lY/GAKIEzcprJNRU7B 0QhOkS1PUxwsCQ+s2ECIYLNvbiCDR0bjpCirh0eBo9QPpBBTQuVGCSghoNDWQNO5lVmq y2BT9DEFKRxeyMom9I19z7ecyQPPZd5I3aiUMV7h09QbBfvjMuVpb//+O/yo0+1gEKwP VBum4OnalPNBmB73AMpJdzFfu39gP458rJxkQpunG8AsOctY7gdXpRmwrpXiyd5QkVHj 9Q== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn20cq0e1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:19 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PINm4A036073; Fri, 25 Aug 2023 19:05:18 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2046.outbound.protection.outlook.com [104.47.66.46]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yxqsmg-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:17 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jcn6NGYkcVYZpeKZjTGReRDxBUoRAXWqkxWZMQf6EmwXrs53KjnINPbuiuSJUH8AEZcEjpjbDnN5ICRu7OVei6MLoBs1bSjtWPdHKAZpcugfXIMw/tJ1I5GMKjA33YjRrOR5KPV+0XZHgWH94zlKuLHCWRBxlZogF3L/wv175wZo+XYR+I1iTsNVpj9nTFRPis1YYkbCLyP/HGB+aCSVY42fP0gk1KjpLRp69VCK6gRaU+LG8w0rzfHkW5U6p+HeeYTiiIxbGtvj5opSt1Y3yd+S0QJW9isqJUQ0EjK1MLhk2jNINg9XkD0KC24dYqUm2lo+vX26wAi0+I3cl0eXpQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0ahRRIn0ZNn8lfsDqjwet4c+B7wLB9LxrxIa3Q88Osc=; b=gwKLHoQTXle6UGGbWPTFAN1uzJECRXyO9A54NsSPw9kk77ZUJaai2wEhXmmnJB35YwLRhbitFofkniTcERBmuyPUy/NQTrHrXzvIE8Wub0im9DtKfDuDChfpa01qqiu9OHaAb2celMPAEjnUeRT5/U3pQqFSi9erani0iYf0y1EGKNyHTS0kpT5avt9MLPuSip4pReHBZiPG75ww5eleExjBn+5ZTBy4WVoPFGoj2WKWaJEmlWIjW/yP3Zyn1UqZMQ09yMHNgX0AA4QLilFUzuScn11W//6geuKuWeE8xrun1xC9s79bPbJskLSsPADQqH37j3gY2cu9/iQkkgjzxQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0ahRRIn0ZNn8lfsDqjwet4c+B7wLB9LxrxIa3Q88Osc=; b=Ryzbc2cmjPfEqj1AbgMeVVvNOga7q7UjZerj0WKU+0OBq0Klr6U8eW9f2DRqQbAr+D6APIKBdcj5HPho7muw2yNMH6/7awft6U5Gi5B9uQ3GLRWZZ7lLQR9MCnuTveiSVHosC6cgkkw0wNg3KXu1lys7xYW9twU4wjv8bj7I2Sw= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by PH7PR10MB6580.namprd10.prod.outlook.com (2603:10b6:510:207::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:05:16 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:05:16 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH 09/12] hugetlb_vmemmap: Optimistically set Optimized flag Date: Fri, 25 Aug 2023 12:04:29 -0700 Message-ID: <20230825190436.55045-10-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR03CA0100.namprd03.prod.outlook.com (2603:10b6:303:b7::15) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|PH7PR10MB6580:EE_ X-MS-Office365-Filtering-Correlation-Id: abedd79b-f5f6-4695-4fc5-08dba59e3785 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: MAh1hHtCXEfiHswV0+4QMFeu7TMqA/19oyNn6ihMXY8ghVo8ZirDBie4BJ2ywESa4c1KaVct4cog5qWZ2sHtBLmhGD56cfC7At01ATEFtCWLHYWvINeVHIVLVe4T5HBeeIR7qRo6XgIUPHUJ2EnrGmTl4aTRljVlJLNOUTz0byfBEnF3gBJkuEkiv0D+73GbXJaZFXTCXtD5HeAlbIKIZiWYu5LtmRXMdC4BvV56yhHHxTHTXvnf6IIPXrHrKbSRZ81NA+xj2wjLeoGaCv/ovmVeCTRRAKYlyrHV1LikT9wbAR56O15o9vUhYwgHtBHXtCxB34jghBsz0I1ayhXcSbZbACiWUIUUcXFgrews5EATKbHSMAEqtMsf0x/TWtJQfdlx40YDPNxBxKzhWyOQbMORb6/RQQG9rjY0Xn8EXyqz/5YJ1BJOU8r+oXyH38WWJwTVQHj2xZ3c9mtGNZH5j+iHF7pNuix+/b661w26m5va1tUVIBYLb6s1k+L++T9ejycbrM2nOE5uujIoMxkvQuDhVxBkKZTORCLzYPkcJsA05VMb6TOZDEjonTxxasPL X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366004)(136003)(346002)(39860400002)(376002)(396003)(451199024)(186009)(1800799009)(66946007)(54906003)(66556008)(66476007)(316002)(478600001)(26005)(44832011)(6666004)(38100700002)(41300700001)(86362001)(6486002)(6512007)(6506007)(2906002)(4326008)(8676002)(8936002)(1076003)(2616005)(107886003)(5660300002)(7416002)(83380400001)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: keS5tgsj+yVcQgguZwbppEJXgXrGrv7II7O3LKR/Ul7jgok9W6lAvPshLHmNHVN1CgeUrlN62LxhoLJpFH1qXaRDbV7owVxXnjpsi3sNNjP5yF3LS/D3TUuxUmoXcnjRb82Oqu3tQvIFhkczhdAqesIeqJAxaa9aibzJRtcL5CTapOo5ipWhjVjqxJMWYD6ukkjOLu11hAicWB331q3Xb6r3eVat6u4ToxN6gIhBs2aXM48wVyHPKgdhDt+FUbdUOkjuQ+TLccA5QzQPl6/eYhnsQvTHu0U9ikxF15P7NxyYu+2jOvVjI07/kAdEuOSjxJF77dsjfpH+gZmcNpz5LdkANqfjvWCVHFo+bQmyVD2gIU8Cg6XY5kFGAqAUMpfJEOaNFp2CBTMOPvVsYKqeXt1o+ZPzcz4j+z3kfDHS6LWNx2IEMEw9K7Vv2U4rvGQrsqZL1rmBbOq5YQyeMk9x30QZ16T9itDbgeoDbeMhGSMF7GXm+5D16MuMzRUkGo0pT9DCJQkOxKO2CkTLt1pKGujihzxb2eY3yOdQczvoXROR14UgGon4hVnr1HHyFNzXZWBpYwibNKvSf6AqyfBN/jxWVLtCqQwC5ofM/hced/m/WAsYAEHqV/DqeP34tXYg6SIPj5ASuQg+xRdunrGf0Pvw+XGfPOJYi5XKQNLjQeU6+5nrV5BrMyGMd5Fs8OlErbO/iuRiO6GmWR+tM6blwPq1IG7bSFN78tZQiUdHBiPkgxfkIVMoUIFqkwrAmKwEKKc4TiJAcXVNynf2KmJU2H79s95/9pweAA3StJvVq8HEfHw3Z9O4pdH4YQregZESoId7ccz2/UjW2ZYy9HrrpFRd8EtcMZXtpb5OxKWxLKgn5Nq6+HcTMZB0VvwjjC5jfJ9gl+5avBTc+LGJY5Yum1GXZgMOHrk5t5Oqa7CLjpjLRW0NK1Y2jSgs3Wr8PD1cbniqeOBzqvXjlqDGo8Xe377K10BWPgdkTwptcb1+yVQ9Aw0CXai1F2w/850iNopxTlvB7CnPeZO/jZta9Ns+s2G+0ves3PtF3C8FuHofK3VgSrjs3LVRUojZUFP6tz3XdbpknNlKgvRkTkREPD6DsGc5aiWWcti0dGR+i8O3zseVC1L02Il791KOIsgfuhli3lxZpX1Qw0L6N+UyX8OD+NgFoRmXzINPxzx7ibgov4m9DaOIojbOPt77a42V/1oTPfE4rvtMURUPdeDxMBTTqgGcbgn9vFuoqoV5HTDyYiJWh3SsKIRcG9eBiAq5AE25Ye52Lsm/3CJp3xSYJXa78nI3uk1koQ3tQnsLk4bM0InDBDEjIiIDb5WZxebnFpCcIvsLrTfmsLpBABMGOpJSax8dX2m8us5kv4PuMxHkoNjXkG77OP8t5whuOa0HgDrCSteLkX/A8CNgz7GO6VIIHXLFOmaLnGuro6VUePHbaY5/+hz8nu51xdd90AAgOmNdIJuUxTPFvYyLDxdaAoZ2kTKvGR34785ZOZX83nOzYK7cZ9E/OBzLy4J8CH916onCskK1hUlYrgdESBRd31tPBXTyUnTKV8m/x7s9ZAdbeZll5tqnzuXgLd6AUCevT6+EopYqUOdLyZlhlocC15IUUA== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: EQogqAEtPqEPm1I6NwYwTlnZY/r0cinLfqoD1Ul2d5RDByWzbPJ2z7SyjhwDJqGDQVBczusYTbWXU/DU8DxCV+fOPk848vyu+B+kZpRwUfJ4wz8Sa7Wn2rMypJEf1PRsYjmt6djK2n1QLoDJ1opfRNes0cfmJ/cFZK1UVlFqcWbQ8rK8wivgcwSflqg5TeCNjwofFz/UNU8vLh6OHqVPA3a+sB/YnU1nlhXSdxT8NB/Mra2voZ50lNrHk2PHALTkNEwxNINtpqHPWEninTUAOSHiEQxQfMI/RPFsNUlMMwy0rM1rM76uPEhEuItWGVjOidbVdsrf609dh0sfgE15/EeUvDHy7Od7IStOC8vvB7GEFvBBxaoRkgS7iWoaTuUx9AqkQBHCGRuRcLq8rd39zDX+K+suAA0n3izVFS3SzeRpxWs3kndSas7HlVCl54RkpWV2Ysu/+YhpXGtBMOJtdGKDae5NuGcnRFPYzxJRi2X71UZf1nU8d4IonkSSJINcMG1775Z/xWNEmqNVwR7AI/91sBMz25DNjDTQlO7sdoWWDstsC03cUEGrAxPxgABQkZLCbylVqROvoPo3mUmOkehOomO5tborDX6i3x5lbBMc5//IV0+HdC+0j8WYfCn2F8nJKcNRBimBeDWptSIL3nzm+RBkERfiPGtl5Rq4Q9V3pxmvPSqo4WI5RYp5UtUhrJdoSWIQtMM+hje53btVCq6dmOMjivOL7Vl8dOc7QI5DkXX2EtdORO9c7pD8JjWJyP3C1taFoGPPfKpyzCjj7BcWYw0J2sdoE8wAJBDwG1OEnbXaSkg7a9uP9nMTx5aQjTEJoFBSWH/xDKufFsAQub5lm7WWxT3Q0tuuL1V+jQThoc19hvZ9dx8Rb1o14nUdwNdLEQFpzu8TDYWeGcv5beY4f9viAvd9TOZTe9zDe50zUxau89J62TVwdKdLSkzx7+dm9WESFaKy1jvVOYLEbEsPDywmof2gIQj1YJ7k+mpyhwNhlOZiiw/0aAsQEnPlfo2u6oWjeXBhORQH5bFjoSNVq0UbLixfHRGMou8X79zQOMFqaJjR9jDhay45tyYIwjH93pzgtLld/uoa7r+/nGbrbGkdWL0vBRsr0Ap0VPYHpbGSWVyAosTOtBu5fW8o X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: abedd79b-f5f6-4695-4fc5-08dba59e3785 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:05:15.9920 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: yiVlx8AVcL/crN6CQLHEuuXU578cJnz5XmyNZCXeAtewUYg0erIie81AWy0fSOL2jDY0T3ZDNGaWSV0hYjus4g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR10MB6580 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=0 malwarescore=0 spamscore=0 phishscore=0 mlxlogscore=999 adultscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-ORIG-GUID: _6aczlNKQatDKqsWD-YfwE6U4edUiDxf X-Proofpoint-GUID: _6aczlNKQatDKqsWD-YfwE6U4edUiDxf X-Rspamd-Queue-Id: 81EFA8002A X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 37gqzngnus1n1y313ep7no33hzz99wpx X-HE-Tag: 1692990343-604532 X-HE-Meta: U2FsdGVkX184kQ8UirUnRyO37GHsEr5dIDUu5kTulLecJcUjJQmvMSyqN4U3NqRhhiuB0GYd4LQajZM6qpItCJ2tC20uvOchmEqtNy7GvS1c0sghZ5q9nfqaWxBvtA7Jp5x5njrDZKBT1qyd+Toi/qh427kX68tvs+zGIJ05WqUWxqpnbCBI1wMmf4dhoh9Paxa8FyCuWKXB44EoExGYbnYeRZpa5obMy1h8klPbED8Kx3KeJRajR2tK+E8qp7AdQq0bZm77NzuHy6H075Kfc8I81a7nLCjLPkU8tPwqFkXz2HhAY9QIK0xThISZAkd9Xprm6RkKqHdz1suiMpowbvxDgd4s9koyaifyIrwvTtDHh0kjvC41htDCaRi2ZyUhtctKWuQVLaq/WT5siwEKq6zaYNJcYuUQSaSLHY1ZppjiD9ExGNQxRBQnuACkdk5Qb+bncVF537yH4HCdtPtj0DMDdu+fciNSjugQEOvchJgFbl+tGCN39uVrp13VLu2o3CTUPW7lBDD4R1yPnkYbF1hIwHPECA5Sslpg0w2OxitHaQDrniNc68E2HR199KgZyRGLgs9Vttpc7o59FiSzBGF5L7kR6yjduaBzaKE38no508TTQuP27Wlkiopx38E/6+7VU4xGoEP/d0jfkdlnr2IskMd9NE96b1a16nu/ISwS2EHvO0xYmrTR+4qOGG1ZvgMnt7QLOOwPhol8H0OgN/iWqfW5wsMONMT2yFXN4ug8qtLq9UoBYFZHpDuggVkv7AmJNcHdccoRycZjvVDEtJxHj/fWCuIloo0ta8V3nGrX5tPgOcLuQ3tVBW39hQML5/YDH1uqKVmipMGKraUd/gQgpG2PVfpSuTA7xFwl6Wy5VmbL++UCaWY+9a75MLwIqRfX0L0rcOz1CBK+sdb9ex+bXmJf7xt+2takW+zOnBxaAWLiNx4ro7nlanNEksBuJa/bOqlD65FNVVCZswB 75gCrfe8 bkHFDmoeffmbNAEUQBmX89NAtao45frYdMRNO8ySlbig3jUqjk9yLY3ZzM+/SW/RL/n77eYoBAsZ7mYS6A053o9rqS6kbxIg1JuvPjGQqbcRIJ5kxuRPyNNXyI5N9sXu83qikIQudnG1wnge8yHxB47/F87wScgAxwIHh3i3lXYX8x/O644bjmgM+f5GP7P+eOV29vyIpJOWArnOckY4KrPMFEkWFyc+I19TkqzvuuZi8JfL9roS0e/WnWjosq4Y3wDXq3JZlPyMftY/xFTBoyQPMQX6PUHU6mgu7GIAakHJrrbXwsanUAUVE2SjhY1KbnS3gigpwXwZ8bLruF6CC+8utjr6XiRmsXqnSWkk5pu66q0C4DknNY5h/TtDc3kGl7oaf8yvQcwVq42l8hRZdtHNvLS9l/m/g9AH3Z+q+KOCbKNMMnQPwCZOLnXR9AH5Mn4ScKaJe7OtxrDHITCpL6CXUohnFo6eazUZqIUgnT7ggPYrcXtj4PhIJRE+SL//EWOaA+jsnTfAh+5bedfJEMWIPmsMD1JzV30w7MXedgt+tuKE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: At the beginning of hugetlb_vmemmap_optimize, optimistically set the HPageVmemmapOptimized flag in the head page. Clear the flag if the operation fails. No change in behavior. However, this will become important in subsequent patches where we batch delay TLB flushing. We need to make sure the content in the old and new vmemmap pages are the same. Signed-off-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index e390170c0887..500a118915ff 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -566,7 +566,9 @@ static void __hugetlb_vmemmap_optimize(const struct hstate *h, if (!vmemmap_should_optimize(h, head)) return; + /* Optimistically assume success */ static_branch_inc(&hugetlb_optimize_vmemmap_key); + SetHPageVmemmapOptimized(head); vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h); vmemmap_reuse = vmemmap_start; @@ -577,10 +579,10 @@ static void __hugetlb_vmemmap_optimize(const struct hstate *h, * to the page which @vmemmap_reuse is mapped to, then free the pages * which the range [@vmemmap_start, @vmemmap_end] is mapped to. */ - if (vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse, bulk_pages)) + if (vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse, bulk_pages)) { static_branch_dec(&hugetlb_optimize_vmemmap_key); - else - SetHPageVmemmapOptimized(head); + ClearHPageVmemmapOptimized(head); + } } /** From patchwork Fri Aug 25 19:04:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13366222 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1349C6787B for ; Fri, 25 Aug 2023 19:05:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3A5B068000E; Fri, 25 Aug 2023 15:05:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 355B368000F; Fri, 25 Aug 2023 15:05:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 09A4568000E; Fri, 25 Aug 2023 15:05:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id DFF8068000B for ; Fri, 25 Aug 2023 15:05:53 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id ABC81407C6 for ; Fri, 25 Aug 2023 19:05:53 +0000 (UTC) X-FDA: 81163556586.22.30F45DA Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf29.hostedemail.com (Postfix) with ESMTP id 2885912002F for ; Fri, 25 Aug 2023 19:05:48 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=XGZnny1m; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=fN13S1eM; spf=pass (imf29.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692990349; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1D7XqVWhwIqkbRqvMvx7UBrASxcP9XTXO88KGrxsyNw=; b=KJZY74ZjExPHSrYYSrmtw/q14Qe7Mm5YtzQUmVu97iuGJnjXOBWcT5fY9R7C54IHa1aGzp ngCmbn77z8vrZFw6gXLM849rvenrm/tz7rkkDUDINbrIdKyIaaxk1sKFvU3SGV+SGr0lkF xBtI3IwDFTO/hr2P+4F6JtCuwRwQ2aw= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1692990349; a=rsa-sha256; cv=pass; b=KclaojUbkO6DvCAQABSQZok3K0VlR67qeR+qnyMaHxTECspR7SC1G0nzWRFoRhUEPgMEYV n+CkshXPdfyaVkdtiX+xTcRTHP7uGiYxtxZbBHTAsE6wqMe9sBfm6lOKk04lzaIhVdGN/1 0LDlP8ujWBCiCqh6LUtftBW2zJLlMVc= ARC-Authentication-Results: i=2; imf29.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=XGZnny1m; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=fN13S1eM; spf=pass (imf29.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIEMlu020647; Fri, 25 Aug 2023 19:05:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=1D7XqVWhwIqkbRqvMvx7UBrASxcP9XTXO88KGrxsyNw=; b=XGZnny1mk5Tp15Iqn/7NPm/bRsrvoj4w77pjtk8OTaNoExvFo8z6QgX77mKB96+PEJBX LG0zMeEwQTx92cGeDv/zn5EgswruNEn/lVUYhYsrcSRHOnHI+kIZfTl1H38tsYZEABEa jcO/Pbf47Phjr/aFWKjU1BXzSn/4jIxEY3qc2F6Z6FtOye6c3gYMkezHMgZSt56xXJ5h FIThXNkX6WkuM0nrLEXgEZSrBC2qXnANgJBdV9XmfGzvSEI/qF3ZSNDa67mff69krNF2 Ytlp8UZ965QKnH3mJn618QvWcxUxzua9RJ00qASMyrtw7GWvZXcIVvifn9J5l9DyESfi 0Q== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn1yu779r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:23 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIH5Qe000942; Fri, 25 Aug 2023 19:05:22 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2103.outbound.protection.outlook.com [104.47.55.103]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yv79m6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:22 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Cp46wsg6ySSmD8YL63Gb8ywxwTJGrLF9+PtVB8V4KNNT6TtPVewRflutN5DVyCZW+q28Mb5ucZu5SrGQAnBxXttoDXl4XmBQJWPPzAV1SCWrcJkL/BMc5owWlOhGFaWezM1l1ekKLtHx1HPe//neeF3yEEO0rZ441PPt/IqoQDhzkfxtQ8PmMTJBaL5KU+S2lzE5I2HCBIfl6gyT3kldYZvG2U0ukfbdH1Xw7ccn85vDJugyJ7ElxDZn8Sh1mYs3GonbKUVQQI/7sR8he8s8gwC4xgJ48eEwb6Ulne1Tm1odvHjB4jXaufdJrCHugxg/a0GyTaj3UBKWEi7t0f/bWA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1D7XqVWhwIqkbRqvMvx7UBrASxcP9XTXO88KGrxsyNw=; b=RfL6tOD5anylMFpNV7vG0zezUTUxLHBHtfkNO7NAFwgDy3x3m2mznNiiD37Nu428m1LGEar/nso4wt74tZ8FPicRbZP51zw5080JqdT6NtogdYVaTSdRQpxrL366FisJg+NfARf/ToFXpDezAXyZQMpo0L8D6jLVw9Vi6ZOMMc5NB2brgru2X9QJSSTcVfrPDJ897Xbp1hITMWbZjd4CWpY0LixfX6Lo9C8YcOlVa2+/yKWsBo2OoEngavumQ7s+daated3YUcxDVba7KibsIl8UzsOcx1Ftrwjp/ymX3A6VBT74gcwlaU3/FZrJYL23fpA1VcmBeIkbV4bMNvSxEA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1D7XqVWhwIqkbRqvMvx7UBrASxcP9XTXO88KGrxsyNw=; b=fN13S1eMfLh5RgTzNRCpLLiYU4avKxx22pJjV6GsnorZx4jQxgvBsbHhEs5ljBKKtWI4qijuctpPmdd00kbw7DzFPRA3FqLDvgiASVcIu7IfA8mkg45PGvEidsHA/MbIbm6mz5af+05DFByq6Psz5vw/WWrZqIIQ6R36rdlYhHc= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by PH7PR10MB6580.namprd10.prod.outlook.com (2603:10b6:510:207::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:05:19 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:05:19 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH 10/12] hugetlb: batch PMD split for bulk vmemmap dedup Date: Fri, 25 Aug 2023 12:04:30 -0700 Message-ID: <20230825190436.55045-11-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR04CA0226.namprd04.prod.outlook.com (2603:10b6:303:87::21) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|PH7PR10MB6580:EE_ X-MS-Office365-Filtering-Correlation-Id: e7cc4ce3-bc06-4cb6-c1d6-08dba59e3972 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: TH/mtV/sqimst11xZ31aIsahAJp+tuvMJLVDbkjRPL0pQ5JDV0GvQp6crCngSlJU43k+zJBqKukAxIMuk05Y/0gOzUrr9DlYv5LM9X6Ep3ILkikSshc6Ng1FtVdTFLAtaqvQ4/ND95dnThTnjyGMwkrdnNaFXonY+MmlFUPhs20nkBKF14N6k/O/jUmNIBtc5AxS3GKPSKGj/qwi3J6N6pjr4CuWF39m0nGS6BMZ25jef4Katzebih82o/mxDTPtjHAQOePbjmcBA+WeiK6mZOQUy2OdH6YxphjaE0KQ+8YkI6luexdNzShGBhx6KjmHo4UiCGdtcmQtuofN/2qROC5kDEKKV0sIxlewNIGUyp8JENypVmemKwEhAT18EcnaxWleu6UI+dZlQnNfyaQJx8LGT93NmOUL0uAYlQBqwWd/eapVqygOGUcQ9WjO/Vw5GKaGABS12EnRnPUB8TqgHXRf54AJeF9x2GMBsQ5ERpQ/Tnva2FNQeXx5OfX+n+QJ+INVbvUb9/Mo1jQH/ligPr4VA7xUDdVzrB4xxgHfTHBr9TYWO16Pa+Rq+plI3j4I X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366004)(136003)(346002)(39860400002)(376002)(396003)(451199024)(186009)(1800799009)(66946007)(54906003)(66556008)(66476007)(316002)(478600001)(26005)(44832011)(6666004)(38100700002)(41300700001)(86362001)(6486002)(6512007)(6506007)(2906002)(4326008)(8676002)(8936002)(1076003)(2616005)(107886003)(5660300002)(7416002)(83380400001)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 2qGuJyf9zoUbjmBndJ8sXknO5gcBXuP/SyL4yqQLkSDL3zXSEik7+We06wWcpTwRGbXYL4t0weeTLEsJrhs7LfhJTZkFjT6achXCMml8FZIs0SXdli5gZU3xrJqHsmpGfpFGLLOwctOM/xDliS323CmEOku5aEBCbgtB1lJKRsdu9sG0jy/IyPg4Rt4mfOYff0QLq4I4naYhdQQ584xoadEU5Sl2/siXSyZ0YqwBHAo2LefmBhg2zGbeiH4ZUIJA5apUKmRQp2MpO15XE0QrxK84nSmqxCxcTGWwUQT4d5H+VHY1fiY6xhll182NzWfFmbMku0sTQe8HLmRY653RuOe6fhkczxD/19qaQTXJUFRvRuZcYDojgW6V2ZkkO5fv5I3kywqTu31UbEsRXqh6/jDj1UA7/9WY+qDvbuu0b68lBVpDMtqm/lYOdB9mwN4/XbioQYdMJHpRBqLp334qFKARQQaZj0Ve2UyvOArEveJDKRe0yqTMCQ555y8zsfENrOtPKPw4yP2q5jsUXqtgI5bopWIiFUt221zgZn4X3DaYKXqkM7LYdacKDtEFLck3VdNxLCxwWu0huiCEG/AobZ4s6JU5oATsmV7fAMT40WpzTK0IWJ/9i6f9TuwUYYRQIwaObq6PMuD9Ohb1cv2YypCgt/kcL03W/nAOyoekn8TKNDjbKWWuLItwMgOIsdnb8OczJyZ2/n64DtWTrSpov/BB5hp+X0SBmASF865ct/xGXXHoCzZzcDK6lP5JzbrnQbZRbpRsziuPzuB9NR6Y8C7D5strDPZs2VhE+2rpvQjhFi0oxe3a3IR/wal2Lzca5sxNMIc+8DN+1HUzCWT1Zj7Vqq6dnOz0voohZ/PLbcA4hTwu0L0/ZYgyKIUsFCdn2OfMrzfDE59PPZO5x0HVqriOvm6v5R4WsWMIRDE+sPiiZJxwrLTvPdgr8bVC93mzBPmzukFB5a6s7L1n68ZOryPmxkJWBeA7Op/8LOCVvew+3LN7EiRa2kbyw7W5LJA6sGZb+A9Wa/NjX9O+bS58qPsEqYGs7cg2ug3MQjrqdZSLYZbheEhvV1SZ10fuW3lxNGRKabUmsY+sgXnYkjfWS6fWlOnBdQ3VIdtLBGQ4otPajOvd7nvnQ8bJYdzAsWAelYlnqlwywNMyo7kKbxrbaQj9AMzu8TRkBrP3jJCfEW1hE+bF8kx+valFj+107MV/YIsl5vrchITOtJ/eypnEhWyO+JUBOm8HAWqLug3zeBz8TfXBKD+eu2X3hDGMnuOGXR0RQJgM2UYCn+Fyd8hvghCvWtBG0V5Xp68sEZ1utOxGVQiTyJpayWKBvZDNtuGFK2Pc26pzbHr8Mk9htIyji+mlGXXlvwBqaSmHqoT/QFmdDjnTsdHK5MZQoQ4ndTR7tnmEY4ve66q4Ode8ETO5edy8BulUiM6Y/re44gx5EYi0GgDWe2xn3zZWI/Nd9YQG5LEmfYRqvKMU0QGqYcbWILvqDhMngIWUzvrBzlN0ma9DlMquHu8sF6kP2RFqC7MU+ObWK5lHgUTUCHKhZmYK2Fj37fCD7b3cA/fUGzgWfIQNIF8eoPOUq3hhMCHzlJjckEHQm40kPASqpk8xKM2xxQ== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 5Na0018J4TK22mjBFP1Awvmq1mHd8jb9g5N0RX+OzRPqVLhba/COFv4Eqd52fw7Zqa05B5IoV4hcnIKjumNvTjarLbBqM1OSkCEUHtmNdXx5dv7F2/w8T2AKdoIGXULOW3qD/TLfkfZdS1el05GJDoMwLDBqUkElpaSkKj0XOZsaf2fVSFDzUJowAquvRvGRHFeCgVGJAjlmZSvJEZca8mFzD8OCCaJQCbB4jqdlyaWNHvmqnn8fLsytev29Y2Arr6yPSbOd6qdWKFuZ8Q1hvZ+YOxd9Ad32WacTGO1D3ygoT4ozSlqSjOWZAyefFbBT8ILf4L+RShhWtcbBvYip0rtPEUKO6mu56qpWioJtxrssFj1lhPDkBCxT/B6S0EwmsWx6PrV79k4Vi21A6Oqfq8xyZThqKzD2a4psl60G/zRA7smYdfNW/Y0v9IMIof5pBE7t7fI+IFr9Vk5G+/C9Te1uvbaDjXWbJxoyqJcz0agtAYGhLn+4/gFLW3aa+nB6wbIMt+thHAKzYqvIUdOClXDi4bx0CFhjYc68RiqinU4HXXbw3XJvuqGr0oI/GFUDsWh1HgbkxRkkmVNb3AsPJqcM9n3QKldUdl1DaEpUczFLfBZZYvLg4MzqTDSkKIsF48M488ylQsrCCN8IPS2NkJ60QwOQVYQeQBiwfJLtYvvPTTI5Bc9r6VWUuSB8As3oMIjQhTdBi/hVfU/7/piR0w4cNh5gPam8n1zLRRHHr0EzLx7jgA7Fws7E80ktOqiTneLmZJF14y3AGTLxF6I+MNsVSnT9zMGoHPYjQzN9msq7Eei5Mrz67FOpnO8HseuJ/oDwhK0Da5u2dPdI341Mfx3kXA+kzQwZJIvS+9ufNQXYZLxYqBghfzQFUfwwBI2rErGUSPe7oze8q81bFJnisO/lefo31414hKQzLQoQ5qBMRpU2k+AfYJxR9kJlte5roBP3h2V/PK/AIec1swaEZb2OBf7nHLy3p2BT8ebY/feIHSCs5Q4cbAQB8zdFqMeoC3OWY/FzcyyEucJEXT/ELl6ZUvT1pwUD6AD0wK3spYHU8uyVLI998EVHbYUzIsGBEcFDxGGH7hTPKfym3+hO0ZUX/z1FWkwYzXiCnwmFlWoRuloOI9S2OZgqNUcvXVi3 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: e7cc4ce3-bc06-4cb6-c1d6-08dba59e3972 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:05:19.5089 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: e4NhHKJIv27o2YjGCnDkACiRmC91W8gezH2F/jZ0WqLFSi+QvNizHYenh3TzGddgjbyT3jDTlzjtrKmemz6eaA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR10MB6580 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 spamscore=0 bulkscore=0 malwarescore=0 phishscore=0 suspectscore=0 adultscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-GUID: H-ODefkFRKhDbLIp3WcUJaGTuSa0E1dP X-Proofpoint-ORIG-GUID: H-ODefkFRKhDbLIp3WcUJaGTuSa0E1dP X-Rspamd-Queue-Id: 2885912002F X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: kdd86n441ogyoff7ugsbaadznbffruid X-HE-Tag: 1692990348-354887 X-HE-Meta: U2FsdGVkX19B8Y/oAAbSCY3nVqdfFPU5RCb9P3qy4ekCZXXBHoi6Un8FfbkMzGx0YxvL/vLhz4/ETvBdGlDYJcqrF9ODMuvE6TM2jP8f8fCpE6oSBRtPj63oolVknrSG2EpM1dghw/NgvdZhcIEHTeK/NAKOL4mXaYUrc+8fPPff1wuALMFaY/ipmUHvHO0hfsRvoaPQ+hzy+/TlfSDR4bWrtNgKK/fKoBjFou5hg08doXnblsrhimZE8AwRxhjlgI9UaNmryyzRl2gUWbymSSiQRb2MYy1XYw9kLH/NH64LgbvSQo5ZcfstwI+qDO65Vvd2Glvd1iBnfinjWv0pMoM+zvG7ObltkozPs15TgoW0Z3A9GhjeSpfKC4FhiIFh77HdGPcwOqEpqNXKWGal108NqGikea8jlhvSheE0X+5PsuA/HFC8ugJDJZ435i5xdEJZixfiV6Ckl0ckipvh0/AYGVBewoH+VJMYlWbRjPyAGKUqjIyWMed8r3oNnXnW/BXTH+e6FK0+WZZYyXVt/n/EBx6Iznz4ekla8M7brejEwwYq39VJaZ05G3Gwbb5Km51+8PgVh+Bs47XboKdmtDHoq89BxZcsIXnczBsJt31CTdAMZqve3S5WnGItOqX2bfTNeGt33kB7F+Q/usYESkga46G9eoZNjT1mdv3Tu5SJharU5H8tweY0ZdrO8POeWQfknB/LELCKfc5qxpNyZY1RKDwNE+R7E7XpDz3HeLmKsShncpNeta14tX4//AKWA+/RxJvJWpQ/7Nhb+HuEBfGdXVvh/N61UuNfrr8U+XKXDxVdtR1j7NQnhxDCy7tj5IhyJOH7C+4QStZ43bW1/1ybOSxqd6NKGkvJmP6tRr2JPLSIszeDDvuxx9Y3nD/9v3Inf76FscqG1kUqTsOQsO2nVKQkhvfbQo0UbHOPJrsYQPWqP4nAlpj+vHdpm3kp7FksO+6Zr7UtsytfTJj p+SOVxTz wsQVb7CCHZApcZZM6J4HBZIxqF5lwM0erU9I78SYy8ifV11QWaoBUhQiL0+5G9yjKE8YlWDZyCSAU6+Hi4efNiY3vcCVNwRN2hAt/gffuixDmxQjb+ahR7ceMk0gaS6503i/noqyc7COrCVKwWzDkXlu2edwyi7EOPPzdwscJECAdE21Ohh069c9oyeQuVtIDTiDhBu4rYHPUXZk0dNGuJM1yuDJnpEX9pwOfPpVdTvMMTh5wstVcUijFXUmjs18GfZDdgGaWUYh+ImhhPh218sRJzLRjIPnz101zL0RR8v/TL2lV3nmP7gYGFfNFwj1QafEduGNvr1VSBXiPtoU+TnfmTRXnoEcT9KenvEseK+gd2Q70q8It0zsIJelOuTk5JDa+llmNtFLgh9PkalW1uvXuMpq4sceTQ9b/GKQaH7iJdNUH7PwBLTqmLZgYExd4OQSp1X7n+4Xad3ZhAkM4/m0uTUBC+EWy14MWxGmobnKhquItERMEM8/nkRNrNCFhrDW1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Joao Martins In an effort to minimize amount of TLB flushes, batch all PMD splits belonging to a range of pages in order to perform only 1 (global) TLB flush. This brings down from 14.2secs into 7.9secs a 1T hugetlb allocation. Rebased by Mike Kravetz Signed-off-by: Joao Martins Signed-off-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 94 ++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 90 insertions(+), 4 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 500a118915ff..904a64fe5669 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -26,6 +26,7 @@ * @reuse_addr: the virtual address of the @reuse_page page. * @vmemmap_pages: the list head of the vmemmap pages that can be freed * or is mapped from. + * @flags used to modify behavior in bulk operations */ struct vmemmap_remap_walk { void (*remap_pte)(pte_t *pte, unsigned long addr, @@ -34,9 +35,11 @@ struct vmemmap_remap_walk { struct page *reuse_page; unsigned long reuse_addr; struct list_head *vmemmap_pages; +#define VMEMMAP_REMAP_ONLY_SPLIT BIT(0) + unsigned long flags; }; -static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start) +static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, bool bulk) { pmd_t __pmd; int i; @@ -79,7 +82,8 @@ static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start) /* Make pte visible before pmd. See comment in pmd_install(). */ smp_wmb(); pmd_populate_kernel(&init_mm, pmd, pgtable); - flush_tlb_kernel_range(start, start + PMD_SIZE); + if (!bulk) + flush_tlb_kernel_range(start, start + PMD_SIZE); } else { pte_free_kernel(&init_mm, pgtable); } @@ -119,18 +123,28 @@ static int vmemmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, struct vmemmap_remap_walk *walk) { + bool bulk; pmd_t *pmd; unsigned long next; + bulk = walk->flags & VMEMMAP_REMAP_ONLY_SPLIT; pmd = pmd_offset(pud, addr); do { int ret; - ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK); + ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK, bulk); if (ret) return ret; next = pmd_addr_end(addr, end); + + /* + * We are only splitting, not remapping the hugetlb vmemmap + * pages. + */ + if (bulk) + continue; + vmemmap_pte_range(pmd, addr, next, walk); } while (pmd++, addr = next, addr != end); @@ -197,7 +211,8 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, return ret; } while (pgd++, addr = next, addr != end); - flush_tlb_kernel_range(start, end); + if (!(walk->flags & VMEMMAP_REMAP_ONLY_SPLIT)) + flush_tlb_kernel_range(start, end); return 0; } @@ -296,6 +311,48 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); } +/** + * vmemmap_remap_split - split the vmemmap virtual address range [@start, @end) + * backing PMDs of the directmap into PTEs + * @start: start address of the vmemmap virtual address range that we want + * to remap. + * @end: end address of the vmemmap virtual address range that we want to + * remap. + * @reuse: reuse address. + * + * Return: %0 on success, negative error code otherwise. + */ +static int vmemmap_remap_split(unsigned long start, unsigned long end, + unsigned long reuse) +{ + int ret; + LIST_HEAD(vmemmap_pages); + struct vmemmap_remap_walk walk = { + .flags = VMEMMAP_REMAP_ONLY_SPLIT, + }; + + /* + * In order to make remapping routine most efficient for the huge pages, + * the routine of vmemmap page table walking has the following rules + * (see more details from the vmemmap_pte_range()): + * + * - The range [@start, @end) and the range [@reuse, @reuse + PAGE_SIZE) + * should be continuous. + * - The @reuse address is part of the range [@reuse, @end) that we are + * walking which is passed to vmemmap_remap_range(). + * - The @reuse address is the first in the complete range. + * + * So we need to make sure that @start and @reuse meet the above rules. + */ + BUG_ON(start - reuse != PAGE_SIZE); + + mmap_read_lock(&init_mm); + ret = vmemmap_remap_range(reuse, end, &walk); + mmap_read_unlock(&init_mm); + + return ret; +} + /** * vmemmap_remap_free - remap the vmemmap virtual address range [@start, @end) * to the page which @reuse is mapped to, then free vmemmap @@ -320,6 +377,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .remap_pte = vmemmap_remap_pte, .reuse_addr = reuse, .vmemmap_pages = &vmemmap_pages, + .flags = 0, }; int nid = page_to_nid((struct page *)start); gfp_t gfp_mask = GFP_KERNEL | __GFP_THISNODE | __GFP_NORETRY | @@ -606,11 +664,39 @@ void hugetlb_vmemmap_optimize_bulk(const struct hstate *h, struct page *head, __hugetlb_vmemmap_optimize(h, head, bulk_pages); } +void hugetlb_vmemmap_split(const struct hstate *h, struct page *head) +{ + unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; + unsigned long vmemmap_reuse; + + if (!vmemmap_should_optimize(h, head)) + return; + + static_branch_inc(&hugetlb_optimize_vmemmap_key); + + vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h); + vmemmap_reuse = vmemmap_start; + vmemmap_start += HUGETLB_VMEMMAP_RESERVE_SIZE; + + /* + * Remap the vmemmap virtual address range [@vmemmap_start, @vmemmap_end) + * to the page which @vmemmap_reuse is mapped to, then free the pages + * which the range [@vmemmap_start, @vmemmap_end] is mapped to. + */ + if (vmemmap_remap_split(vmemmap_start, vmemmap_end, vmemmap_reuse)) + static_branch_dec(&hugetlb_optimize_vmemmap_key); +} + void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) { struct folio *folio; LIST_HEAD(vmemmap_pages); + list_for_each_entry(folio, folio_list, lru) + hugetlb_vmemmap_split(h, &folio->page); + + flush_tlb_kernel_range(0, TLB_FLUSH_ALL); + list_for_each_entry(folio, folio_list, lru) hugetlb_vmemmap_optimize_bulk(h, &folio->page, &vmemmap_pages); From patchwork Fri Aug 25 19:04:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13366225 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5345EC71133 for ; Fri, 25 Aug 2023 19:06:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CCC60680011; Fri, 25 Aug 2023 15:06:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BE28968000F; Fri, 25 Aug 2023 15:06:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9BC8D680011; Fri, 25 Aug 2023 15:06:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 872F968000F for ; Fri, 25 Aug 2023 15:06:00 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 5B9AA14061B for ; Fri, 25 Aug 2023 19:06:00 +0000 (UTC) X-FDA: 81163556880.04.EDC71A1 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf19.hostedemail.com (Postfix) with ESMTP id CEC001A0019 for ; Fri, 25 Aug 2023 19:05:56 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b="Ta2/LmhI"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=RUgChCt7; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf19.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692990357; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xPMxkRieDfK6Y92JL1GY6hu7Bt94+EbF/fBFQYFUe/8=; b=TtJLWFCbUaznDV298FVmnF2r+9JJ7yELvEJFF2Dsc92SP/xsE05B4nJJThJffPp0UZ+WZo YHGs62IlopSIawgKcE9KnBYnSdzx+J5j6Rir+zDRjZd2muA61tIdGxoioNonrwNQpeoWul 77+OBBmll+pswddUEmAtShudgsW0UCg= ARC-Authentication-Results: i=2; imf19.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b="Ta2/LmhI"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=RUgChCt7; dmarc=pass (policy=none) header.from=oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf19.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.165.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1692990357; a=rsa-sha256; cv=pass; b=oK0XDJ7CFhmCqz9mZpN/7rKlp3brc17ZaC3nwFDSP8GRQDgszPw/4BHBT16s0m59BQkgwF l2johVPkMCjV/n/Mz7AVzUL3aNw+ijCjxLSb1iGiFRkUnvYs514ldSixDVgdC2oB5jKbjr OnjpyDJkBj8o70VO2KYHOyAU7ZxTrSE= Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIERgZ031110; Fri, 25 Aug 2023 19:05:27 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=xPMxkRieDfK6Y92JL1GY6hu7Bt94+EbF/fBFQYFUe/8=; b=Ta2/LmhI3sn4f9S6TyPqeAD5fEekhWKQKEl6FmJs/iYJoPJBVK7j1OtsjOKIDU4KSpiw bOLVEc7TzNTOd3p3cUmyq9m5Ct5hHLWwQuB1cUJuqMcePCqIAasKmXYQiZ1jmPcPJFzE iSsWNXWZ0rMIOxZpOTjmItnlb6qx9AU9xH4kUZ668wGeRLUViC3KCHho+AN9fHp+RyYi mM+O9E1CJocPtJJWntxNSCHrXLUDOOCjtpyblcHFhBpYKRypcJgW4tgWDe46x3WcczWl Y5Yzs/35QGGoFn3r/4gEP09JudIxo36CEBd/7pLbj13vMN/81MWfGf6ecjhw5GNtMzuI oA== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn20cq0e8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:26 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PHPJ4x036043; Fri, 25 Aug 2023 19:05:25 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2046.outbound.protection.outlook.com [104.47.66.46]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yxqsqs-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:25 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FpWvEvbLYHXqObCK+tL2P6G+Ts4HUTvWOv0XGyahYl3/w78al/+YazILl3vn5pBCg3ZYDor88Dr0qlaeSXIVAMM+T9OvUdcc3J+Z8Y+q9MAOxvjGjFjeFOZisOQP+MLIGA/GIcFUiDj6dZLuY56tShtwXw5DyPiVc9+D8VWaR/xcX25Ca+PJMOtPHDhP6sMwKxs+j6rPDdm6EozrarrIWDn2qBOOz5Z8oIYGXWS16/lwVgP9ZRalkDvpxpzMhKiX3g34jy5bWUhsnW7LzZ9Yb4ppS8Yr/bdIOlZDa5Ggk7BaIcMLyg41enACt8W1CCVXHGA7ev4xkg88000ZWnZepg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xPMxkRieDfK6Y92JL1GY6hu7Bt94+EbF/fBFQYFUe/8=; b=Xk+UzjkhgGWBGV6lkx1DniVwrpBAw/wZ/rqYVsNs9EzhhI155Z5t00Oma1M+hWm3tOY2fsZIbN9g/DTJVPdJsrAMwywCKSdyb5zy8VO5Pj72cREqzZS7ezE9/7+CiJ1gkswFNvscBzM3ovO6jTsksWOZVyr7gQMiPDIfzi3Dt2aUgSBCQKGMCjrlv0fyNd/5lNGsLZ4jrX9uQUX7uDXR3uwO4HhK4H05dH8Cgh4Ho/MoZh1gLE5n0O7E7P2C0+HsH8yfkNj4xCmXBKVdjU2Dq+cYCYY/LPNU+83pkHJfS1OGWoiknzEDRQTAoQz//SV5Jb8A4m8GSkOAzPYtSk3F8w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xPMxkRieDfK6Y92JL1GY6hu7Bt94+EbF/fBFQYFUe/8=; b=RUgChCt7lPSGEwF3bSt2CCms9fWEcMVOynloujqyct8TdngGhvXZ+GCfZaQGCLT9MK73qdEGNGVZd1boPkwyDYSfptkhS2eOFg6aMcrz20wqsdln7hBglSUZXUdMdJPv0DWyLcxDoUKxGUAOXICLG3OTgMDRnYiWGVuGFKB93dA= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by PH7PR10MB6580.namprd10.prod.outlook.com (2603:10b6:510:207::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:05:22 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:05:22 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH 11/12] hugetlb: batch TLB flushes when freeing vmemmap Date: Fri, 25 Aug 2023 12:04:31 -0700 Message-ID: <20230825190436.55045-12-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR04CA0212.namprd04.prod.outlook.com (2603:10b6:303:87::7) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|PH7PR10MB6580:EE_ X-MS-Office365-Filtering-Correlation-Id: ece5d3c1-753a-4f34-2ddb-08dba59e3b7f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: WmUYLsQACR4xEZw2u+vRfLmnQobbCY9MVl4cusmasSibJ3YqlToRBBzNpvY8aVLmdvwTXyxfUOu0yufC0ASC0xCXwgOed6AFGjWgPWWDlv3hvLcB2orS8361MUMyn9d2+y0gEuLmJHb7Dt6rz/21nH5CanQguWJDIsHn9sVae99cwUt+viP3nr746rG3dtrzdVi5d+rM6akEMHKtgANI5003ZXq9+bqXmubQrsIgeFW0hBlNthyQBuOkN2OvHT+u11TZTIhI4Zk5olD9OBHqNgld98/GP9lQIAMzTUS2OQ9+aKBjjtdaqY6QoD8q9Hekv2W3JKcXknjCj2Wxo1B2jZ3AM4fd6w+QV8K1BL78bShBqcGo4UaWXuGao556J8VK2uQVGvLuc0FX5xlOQ+5BVnEQp5QgcjPTfHiiCVrROjUNZb8eWqs7t1WWHZWLPvfdBIKX5lIvFIaPubM3e1OGjmWt1hTOpU/mZvNVxLByISwXKB0W7XdoeVVh2nbCVcILVEV/361ANfHuwfkCZZ7gbp4NhN3OBKhYDRziz17AAwMwJLw5ZjaN+1w7aLEuyfZC X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366004)(136003)(346002)(39860400002)(376002)(396003)(451199024)(186009)(1800799009)(66946007)(54906003)(66556008)(66476007)(316002)(478600001)(26005)(44832011)(6666004)(38100700002)(41300700001)(86362001)(6486002)(6512007)(6506007)(2906002)(4326008)(8676002)(8936002)(1076003)(2616005)(107886003)(5660300002)(7416002)(83380400001)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: niBV8kWv0/DMoVpb6An6P+LVzGdCMb33+emMW1PmjKQZtEEj1wtaI1LfTN/EdleBnJhICX746oHocnmcyCXTWULUJk9sspItLG///47soMpOwD8G1wlaAoajkyzWsetB9aE5/3mDXtY4bp73OdAfnNHksKx9n8yXep6x59AgjZo0Dbu7MTOVCalylJybIeMzJrJ3QANKUxv7gRByf+EQ2bwn0USxCbcxqQfmVaGChFiyGrjEgcfTW/l8v5cDxHhCBb4VAHBjA9YsyijfVmXWyMtbKVrRebo07g+UjRX0YgZA+fCxoPIm01Lhuo79McQADfqi0+oxCmRqnvE5yVGnxMaTseH6RawNz4liWxX6qailXROvwc2PI6MhPeHu6eKiB3FZWSFdxU0QI0tCawDAYD7wViCDi78fbCcY7G5r4AaP0ijPVlvqM4CskOXWbGVjPPEPkK0s0IncnI/CCH3ScEuh7/XEU7xwCDRE8XhQLfvpMuZAh+fzwpoObb+2aRY/Linoc/D460omhsQXxhqj7VyYENeYQNhhoygc1lqwCZ7Du7ERoDL/aR/bA3N8MB0mpLOV6IZZJTj1Ztq40k8RtkAZsXb+lCtJEHciZ8Ra7L33lWTSv0Qt+RRk3CDgrrh2E7zqprnvftcB5PoeGGHCYDmJhzO9MZ+hMtOSzRFszUqIBDp5J1dZ/RUFfdYYqPIjg67b4U+Qb+eETSv6h5byKRJBBJiZPLjLD0/mABDhm6ANVNJCSxur1FtQnbBhNECb6oveAVkFGLJOB5/iKLOsWRfkzIYgtMz7HPP99Wa5wDi06EaRiMsZop3nNSlVNXqwaZyqz8GWpHIWX94tOfcfkwDyWPLnxpSWK0gFficb/bzrzeVTkmhwOP489uqhf/PZMWrAP8fyXdgGJqfHOfA70MGTFEcAjjiJSySUAFNe9AkwdJVNlVRzysIVXzkVz2IO9lUAUqF2BqsrA995GG7NufIqkfkjGBzsk/IA2q2yZQ9IZJJGtiLrNrMSaiBZ2YqacBZXRZtnH85nxIKwttGV4RxIHTQutFa5r/IyA+mcXj6LA2FfYaSHZIs14U6xZy/G4VH+ueMaqA3S0kDlQjM0wHYrDAvTDDAIDWO3vqH/8UErXSKKcUP/G6zBFEr6v2xW6GkJF4wppHBh0ziQNige05re5RlULdzwTX8Hh8RAM/gqyG0r0fdjDY1ZZ7s8ilQ/83l7mfDC6Igw4fUFi9NwXs4UChe3WJ4muszUvmfs+c1ejn6zLZ4WpJ7qGPPQ2vD8IaV5EdyMwiu8cl+BTrgpSYRAbcFKGi7SatYGaqCN1wXFOL1iTkuMNViSTjRKHCLAqAP8isjwJEprkab+xgY6QMBzu1YWBgDe7cmfkdBFxCp8ENi1NftzTObGPg6K5kDs9ABbUSiPQW0YqB8KFQseQnYxEijRG7sJN6mDiaJ82UlTixJQtGvVyl68HnuDy0K3SZbFUdAMte4nnvGwBCZi63BYjeNNcgo3zUjyZyVvcdrpzPCVclqq4gA9PBs3aHk2SB9GWkDmIwzFiRxGz7gQSgfqabAHZurf311NdGJnjht5Vtd7yudsltzUXqvTFEhIFX/Dllbu0cOWhT+UAM9pvA== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: U3uketIAvmWiYK2sHUD32dH+JcqP5a8nkG40h4BKlpF3LHoiq1EHMIEcrvFmi0EEoS47XljvK0H69Xy0s7+yamxZnv4LNGhto1It1HBcefNV1AhNXMhAa+7943JvOrs3s5NMcgWZRdGUkSso7iYdDlHR77i4cqPEx72wDUh6WTpbmOe3KPdfNHWvr7U+D6FzmKlM4bodBkyJbvtp/VsCUCyTH6gzJmPPArUSqtHU8Cxr3Y5xtwKR6Pp9kN/C33C9hkYWP9wMs9ecio8q59J9fcbi+GgLEm3bv3IL3IE+OKxIydBzokdHCbzYinXFr+XBYMMEg8oGCN7biJNo5TtHuIMPeGBsvyoCbKT08FWSJzpEZV4ra6pcUNeUk3ihM0GMoKLTN5oW5+zFgi5ZTpOAkD1saswvE4Go2cJIidkmCzeCNwt3S1k0vB9cToON091h9MJvj1PMG4V74Qac3gWWasHSsvqd9yml2RMkbNVgiV8K+zcCakuO8XWIAmcb/c6l9dUjCSan8pYyByNbHytt3d4xlRfMBoGcBh1q1j0i/HxIdMoCRNToh9ROFE9idDFmwO89ukedTxRJ/5OfbgSYsKBQiVDiqEEmFw+FzpmdZnA4p+oAdQmiuaMDB8b1eulAyy2VbWtcaj/nNDQYQn52kiwinJ+FFtJu2B4l7PAeLCI+Vct8ilLuA7ootm5iIXyr3wJeIlZidXc9xZ2eVsQuNEuM2UPEZ3l2xb6hYAnuuLcyNL8lEKTqU96V6T3kmxAeUxegRoDaMIuJdsrRSalYIDyvc/TE7HCDQ7QC43FZiiaQwLPZH5jD2cMM25Rzvk83p/ggA2qCbFnrKnqIZTjZ2fw7x6Lb4IG1ozM4iQJrIJd0KufL2mTOlqBV18jHaenBawRpyUbM5BQuInDxpCbUHJGBc5v0jRQcoz5z2MPWQyY4uJ/k9PeCTnP4HpONN4DGMxgYdjmaB+XL6+hh75oOvvZ/7GSf+woZ6aTQm3nmmJOAT6ty7y7K17nK7d/Uxo0kCcRZwbgm2LRK+0qxc5//4rrxFbdoK8uAgHBwy5tPqvIpNYSC+k2RD41SNoQYwQ6ib9kx6M1DjHigu/buAfNoAuAFhn97gLG7HCTzfk860AJpDuH9kp/Pyzk1lT6ChMTI X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: ece5d3c1-753a-4f34-2ddb-08dba59e3b7f X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:05:22.6665 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: UVKqhjj+Bfedxr1RlOSpOMV1U+XzrmeQXG8xQ8jiDMfcelCeaBIu0Zn4Ug3ByBw+FD0sgDjPJSwnL3SxqI1Sxw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR10MB6580 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=0 malwarescore=0 spamscore=0 phishscore=0 mlxlogscore=999 adultscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-ORIG-GUID: DUWcYynt5oije6WGFCohS6jt-asJt9Pu X-Proofpoint-GUID: DUWcYynt5oije6WGFCohS6jt-asJt9Pu X-Rspamd-Queue-Id: CEC001A0019 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: gz7xwwo97yqyu9j6wz6wm41hb55wxqaz X-HE-Tag: 1692990356-565924 X-HE-Meta: U2FsdGVkX1+KVSMlg7lYV2qXdfHUiPsMTPP4eGXicouTPqAUI2wFePBJoWWBqi4AOJLHmWoxDX9qj+2UplNEiyWMw2T9anNB0jzTcPXG7RajtuDboIcnFe+Ea5eRu1RrNltljPB6rsdcgcCgwxv9e+rTA2IoP7G22tE/81c0ajpfvNpC+bEnj4n0qM7tj0jPyPNp7A2mrLaftJhe7V5fxQJwAUtrnRFjmUQkCedol2+Zyv6Ay2D3MwsGvMsT+MJnO9BsvyZv7ZGfygH6RtBI3/ktUvMhnsr3Qq3HCHb6vLiT2zwnTkaDlNKQhYuBFwe3zG+HPHUSimxuuKk0eYETVYigG6QTIYpFZgxb/F+GSWigwzpeR0hOJMAYbfNzOP4EY/ZTPZWaWa8D1WagESSYX6sWyHqwBZ+PWdn1K4ey8zBUIFyK4B3vhYiZo1NPJVmYN1e2NGin+67QcVpZXh//EbaNDetqLcyi0zU7mDp0ZU1WGia94mQCwlCjlPFF5nQv7QsFlJ75oFDgTKO9uB05tgJ3Gfhf2OLGRlWRuWDtJkkRzaDMx4qi2IYkjatqDGDSc+dXxvQMQTTMF55dGXmtWgZ82OjrNmlnNF7TqxWWEljWqMI7qT4JxH4bTlcLH0WQe8D6YdWUXq22hsZVaupArDjwJ/NVdMRg3bLUrYVGFBQqx0ccAkTUSMz3kZrXCQUYyawCENnTUTBItFmGl/lueWQi7+nfTDm71qswuVCLBlK7JwZ4JfY0Y7X9dRm/1kTn0Es+7FOJKk3qfHF7VxhKxqQTmRVC1v1f4QnkexSFHXenIRLk0mfn1vWRSUSn7L994kXGDLWhsBj88eKMr+1QkhhN9m1sbUKql0f7v4yRrX+rtQN+LvB+fBeyHmMD2z3chqOJZXSF6XgzRxnsNKQykuYtJ+bvAsSxSTJddLVEuftRW/frH636Q6C6p7B6R24RaA280cHNjKOgDIDuH1m fPOBbndH 4QbHoT3hy+GbCAKSkqWysYXTIvVhiSOZtaQYUL8RA8FqMdINvHigWw32ZEI3jUYOJLV4tfuwzWYiROUh7Xc0tMYE24fiyBRc3cSLtJG/Kti/vcO29bkufBiKw8DYNxiYXt/hzQfwM9DB19r6ktWNKODY1mDWPS1bEJOyHpwoN0QhLXEMf9ZR7eZ7H9r+LTxhYxYhwFTdZzcBezQCPvZE99N5Me5dL/lMa6Sw5WGJ51BO+5YlOvH2nIFOV6StrNiFnN8C1wBAYSBPuHPwOF2B+/s7Mcq24y8gHaEBT06uu6dgM2LNWawK04pfJYMk8SZE7Od5KXKGevB6QR7nlSRdzAKMLtrSoPLD0PqC7mbuHY4+a6Uq1EoxC0GTMkuSZOOftoQiOgA7+L9eZmAcJUeqfidJJgZfLr4LpasTWI4NP7oM7iOy7SAxyoeUBsXv6Zs7779HIHIsm7zoedEKU87UCBqHgRpJrDqK3vIQJ9GncAgCSFjCT6UfSBuD29Q2UM4qkxZFe X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Joao Martins Now that a list of pages is deduplicated at once, the TLB flush can be batched for all vmemmap pages that got remapped. Add a flags field and pass whether it's a bulk allocation or just a single page to decide to remap. The TLB flush is global as we don't have guarantees from caller that the set of folios is contiguous, or to add complexity in composing a list of kVAs to flush. Modified by Mike Kravetz to perform TLB flush on single folio if an error is encountered. Signed-off-by: Joao Martins Signed-off-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 904a64fe5669..a2fc7b03ac6b 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -36,6 +36,7 @@ struct vmemmap_remap_walk { unsigned long reuse_addr; struct list_head *vmemmap_pages; #define VMEMMAP_REMAP_ONLY_SPLIT BIT(0) +#define VMEMMAP_REMAP_BULK_PAGES BIT(1) unsigned long flags; }; @@ -211,7 +212,8 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, return ret; } while (pgd++, addr = next, addr != end); - if (!(walk->flags & VMEMMAP_REMAP_ONLY_SPLIT)) + if (!(walk->flags & + (VMEMMAP_REMAP_ONLY_SPLIT | VMEMMAP_REMAP_BULK_PAGES))) flush_tlb_kernel_range(start, end); return 0; @@ -377,7 +379,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .remap_pte = vmemmap_remap_pte, .reuse_addr = reuse, .vmemmap_pages = &vmemmap_pages, - .flags = 0, + .flags = !bulk_pages ? 0 : VMEMMAP_REMAP_BULK_PAGES, }; int nid = page_to_nid((struct page *)start); gfp_t gfp_mask = GFP_KERNEL | __GFP_THISNODE | __GFP_NORETRY | @@ -427,6 +429,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, .vmemmap_pages = &vmemmap_pages, + .flags = 0, }; vmemmap_remap_range(reuse, end, &walk); @@ -700,6 +703,8 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l list_for_each_entry(folio, folio_list, lru) hugetlb_vmemmap_optimize_bulk(h, &folio->page, &vmemmap_pages); + flush_tlb_kernel_range(0, TLB_FLUSH_ALL); + free_vmemmap_page_list(&vmemmap_pages); } From patchwork Fri Aug 25 19:04:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 13366224 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59BB8C3DA66 for ; Fri, 25 Aug 2023 19:06:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 72CE9680010; Fri, 25 Aug 2023 15:05:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6D51968000F; Fri, 25 Aug 2023 15:05:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B362680010; Fri, 25 Aug 2023 15:05:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2E61168000F for ; Fri, 25 Aug 2023 15:05:57 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 11DECB27A8 for ; Fri, 25 Aug 2023 19:05:57 +0000 (UTC) X-FDA: 81163556754.10.4A17E24 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf26.hostedemail.com (Postfix) with ESMTP id 89FB7140010 for ; Fri, 25 Aug 2023 19:05:52 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=xOwe74aA; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=LneynFCx; spf=pass (imf26.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1692990353; a=rsa-sha256; cv=pass; b=pQI6z22KDhJdCAJlgSnbYAdvW5gGicDRPq0gmz4VhbU8H/Xwk5gWCebl+PGe3/Hko+srOO l0b08/P3tsA8PeA3OdGFeTawYEWsma89SMZqVbP4Zqz1zB7Kfk09euCM7i4g1bLcZqRsYY W66KS+Zdpicsg8Eg023oblOI32akyzY= ARC-Authentication-Results: i=2; imf26.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2023-03-30 header.b=xOwe74aA; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=LneynFCx; spf=pass (imf26.hostedemail.com: domain of mike.kravetz@oracle.com designates 205.220.177.32 as permitted sender) smtp.mailfrom=mike.kravetz@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=oracle.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692990353; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mxd9Mb3jlleYqr6X0jDbIBqXQT27sF2i7EgotUHdG6A=; b=BmS9qmQZF5Zib8EX18bd/TUk1DcOwopf1m6RkizErs6DQzZvMnY0xRpScBRmYkoAWEmYIS tC4uBRqvn5x7Svi3r00OW+GFEbEQ3U2a0R/ZT98UvWaRLwrcf1+5naQqlnycrDyxdslG/w FZ8kNuLP+zDv8zaxdLUK3MYbUtNnWCI= Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37PIEQpJ020689; Fri, 25 Aug 2023 19:05:30 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=mxd9Mb3jlleYqr6X0jDbIBqXQT27sF2i7EgotUHdG6A=; b=xOwe74aAcnCzf64hyZnqxAjcg+AXqq+OeJG+CoDtor7TYnSlOIkkwbPlkuZi0QJaaexe 3ObSk9ugi9fjyv/O7vLmS4NLyMAckNMbk/8GglSwIZ7Y+tJOS/x8r2NGZy+QgLay1jHi cEU2GsYzMfRwaY1ygJnyILz/VB5tG44cmfipLHFUTx/EwF3MNNYt1bF1ApYDfHt1I6H/ nJYBC3zSr42g3Zn1egdRKSs1a1aFO90bnLUjybpPjhQ7Q4GntzIBMtR8i9HUSWm5Lujn o/THO8g888hVHQofzCQcIE26p12XPTK3r478i3JoPYh7tYorIPhySidaz6wcojf468O+ tw== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3sn1yu779x-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:29 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37PHRtnk005844; Fri, 25 Aug 2023 19:05:28 GMT Received: from nam12-mw2-obe.outbound.protection.outlook.com (mail-mw2nam12lp2042.outbound.protection.outlook.com [104.47.66.42]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3sn1yuyadu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 25 Aug 2023 19:05:28 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iaN2eP/2WLk7YjpoOSctnwyKaeNH3sVW1dKcbcqQ2vSkklHDW59kcsVgoSpDlOoQDooZ/J4ZPLBDGcW8ucmDOctjUJDyqnOHwAkH3duRDFTMIwBVAm/zm0hzxG14037EY05jDLJ5txCH0iPGvKx5wFYb9Vs2NVr/oIMIuU9on1vN+ZXxP1aGeUPx2mXOlooaheynVA4gidNJSsG1P0JsQA21z6HAdZKY+0YNmaCHlg3s1EFaquZoCkf5tRvx3YIZG3EfY0zLud8DTxePUHDvy3ZzmBGhW9wQT1WxbdlZw8dZ8BrzY1Xapv3tZOqxWDYarYv7lpmXTjWpTiWhuolZ7w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mxd9Mb3jlleYqr6X0jDbIBqXQT27sF2i7EgotUHdG6A=; b=C1R9+vDAF0EdvTdlYZCjHQ2wTx/XUpmspAV5fcu13zbgE8MOTac/uQsUxLPkic26crE7YwKVDC71Cka1xlO2knnJrLqDz/ugviO0Rvnjf9dpSQ0C4SoUPwQ2D1NhuCoUQvoNjm5tABbCQkZoaBmmXhj4VseUkrHRuuE/HTAuWJHZAwpsmTDxrdwKsLV8Vv9icQWEcZd4DQctvkXSFtN5fFL9MBIotB3nfOycf/CQEctpfM1ZyQwjkbxUiul3pJceLkeIQUXPoVU/aFjZIxmNXvmr5gaNgMIdShh28ichaBtaQzOjG4Evejwz93kZTWp9T8p1tvcxOseJOLPclbC0dQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mxd9Mb3jlleYqr6X0jDbIBqXQT27sF2i7EgotUHdG6A=; b=LneynFCxOxCJeCTJiCzU05FedLicIJO/YFxNxo4sYKsLMpvhZzb/MAyZ+PHPqIInactm1SWmevBqwW3vuaXF0SikiPOcH0FbNvwsdzbbaC0dSWSNeCoueSSkdnNvUvJlFaFQA39Hcst8tSzihXSaHlPqU5kRWHQEZZJFuwWWIJk= Received: from DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) by PH7PR10MB6580.namprd10.prod.outlook.com (2603:10b6:510:207::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.27; Fri, 25 Aug 2023 19:05:26 +0000 Received: from DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a]) by DM6PR10MB4201.namprd10.prod.outlook.com ([fe80::8057:a3e9:cbd1:252a%4]) with mapi id 15.20.6699.022; Fri, 25 Aug 2023 19:05:26 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song , Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH 12/12] hugetlb: batch TLB flushes when restoring vmemmap Date: Fri, 25 Aug 2023 12:04:32 -0700 Message-ID: <20230825190436.55045-13-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230825190436.55045-1-mike.kravetz@oracle.com> References: <20230825190436.55045-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR04CA0240.namprd04.prod.outlook.com (2603:10b6:303:87::35) To DM6PR10MB4201.namprd10.prod.outlook.com (2603:10b6:5:216::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR10MB4201:EE_|PH7PR10MB6580:EE_ X-MS-Office365-Filtering-Correlation-Id: c4099dca-1ec6-45a7-88ba-08dba59e3d97 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: QaIVMGTQ5ax/Ee5aBYf85w5lv3QoggNv3UMcKCtofmGzdZwkQZ3BKwjMh7dBkuRdh/yS8qmOFim8OsdsvcYVXsm4A3TrZHQ21zlPuq80nk8SuxPHTaAv4JJqSnJjs/DIrYFU/xpzqH6mhKqcPo2NPqDu9tTEk/Y1jVgvjzpNpeKZWUAx0iUGhZk1P6IMc84koDgnTC6dQOjHGRakZ4G/09APNvA8kctk5mMtFnYpJHq7gWJIHLEIcdOGAjq2TAAQtYFnywrfowaAgTKebXv3vwnxBO3VP1QlI/Va58Dw4VPDQKB9tfeFY6hfcZCdhgqnBk+smqmMfKjwLxd3EZ2QGroVzFqlvadKLh3EoJ6X17t7l7Os4l2sVHPtZbZHByyEbLql9Q0RGhMQVJeOGlfq4+UCzKCSxgwKwmB27hZ3uvx8J8ZUZAogVM+VHdijj7Kqy4lor3JfP9vMyFGH+V0d28KZuc859sgEhD9tTMR1riW2JqzcZfk7XDcY/osiC460qG4B0euxYW02H91gj2Jc64gLDzu1MrAHtTp82l5I/s+eoG/MShngRk8CLlOKLA9G X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR10MB4201.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366004)(136003)(346002)(39860400002)(376002)(396003)(451199024)(186009)(1800799009)(66946007)(54906003)(66556008)(66476007)(316002)(478600001)(26005)(44832011)(6666004)(38100700002)(41300700001)(86362001)(6486002)(6512007)(6506007)(2906002)(4326008)(8676002)(8936002)(1076003)(2616005)(107886003)(5660300002)(7416002)(83380400001)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: gfHHhv5xJqa+q012OG1KsLGGkJslK5l8SHXdXHzDmBHuIy1BXxs9wm7hyBXLLtPxfQBo5dy1YMVGqnm4qAWsDq73YsZ8iLoUoFpZA/Ce1/LBI9q+jQrtfdFmJ4/gb0yTfWVyR9D1wYK889KaoYqZDz2kvf/SZLvIO33OuvBmarf0PzNTqbMbgSoAS0A48qC5oH79xhOOI3iKqVPPLdaKrFuyqPfKKfx21sgQlW15aGSgNSciQlXMGGcRjF1WXjlOlBWY+IsVwyj1FiYu4E0oKPiuz8W6FXoXKpZN8NsKY2jbIeyh7D1Ikmh/LH5hC/TP1OzVinoqIS2VklfzFcbJtDTYysVLuhZb0GHPpIURwMjXRP+pb0Z/y1E8zHuw6xDetLrs3zjB//VsQf2mpMKphUjEC5J8YIhBngHoODWsBmKn6ywGoGXkL0UM8t2fm5yBWXS3wvl0ATAmhEWe6CbM6HaEaqnN28N+Oop4GNXgJi6N5v6FjLJrSNMiB9hj55AV30jqYK3HsiZziK6bAm6HOZ9vWgQY9NueMq8y/dE5RJIvkD6sKcpfGrBPm7HWTbFnLxY+QFoyrIukt9Jk8MdtrbBijeNB3RjjrKIv13gnzYbqMi2L4n7ZophVg7xH1jkAfTxv2Qndw1g9ZAZEankKbx/xH+QLmrZAdkjUSp20V6lM9JGtnVUQuOg/ibAeYmeh1KKIyeZ/YQnLEmFL5HZbvw5rGPBkEsVjbQAFIB0aNS9VV8A4aLH/irrvJTW7smrnDQF852dFyY7rSr5hA6V5avwidixT9T7DL0AaDrpyEQ9SHlICsUlCF+rC59Lfe3Lnh+H9hFu57hhpVZHaJnWQAYZuzaYX6rjEB4b/9YbJKolSD4rZ5rk+Va1HV/MSMeatQPsMbw6F7MWRFsWTleMk8nDTuVrP8X/mlZyvfvRhvLKNi7Fp1dXXUOWf4QP83+dk9uUCAb3KcNZuswlpPwlCP0SAfrFW8EcrdnS/MT0QQZlA8WMe5A2z7a9mGvgfegXp2iMbJzCCBSd4bdVf60EiAwNdGrYH2cSNZXprIwXSNsYMm1wITJzzVy6amdgdIhxFvIbPUoO72ULs1ur1M8iuJ4NdiI7Tjp9kJBuG01Du+/xyRo0tNHZznLJSuSRzwMbnDO/xcTnud6WRMksbApEosROUoV7teSJX2OIBiUbzfvBW3JGB3ONu2Md3FbzdlYIiPAWtZBlqOvod/fsXk6jzGsQE1u+GlLAL2geXiaaBfD8ICmgKmhgCwg693ZAU7n7s2jI4KgL6r4yID8UF6dlqdU2UeD8Cz4Qkm77q3Am4+7L21mvk3j44YmXmAnnS8j7HpQGimqIEy1sGXhAjrDoWhv23deXJDLZhZBtmpgAhL4V+LN0kwETD9syf5dNykZmkZT0Jfao24uJ76irjxIuiA1rtQbGn+yU8rMSxXKHRmM+lfhbl8qh5fmYNZWXN6y3BrhD6b/xrejPMeQs7ZfzkvBBjXI/Wjb+E8kEfEGSnJz+GBiMoH4mXXC4BfvGGoydL1I5gcWrKV6yyugWwS03WiRmUfrDRDg09kMMmDXmZX3+eKCnzZgKvedbBjcaJGIf+CiEevBNC4AlqBFrtK8wzLg== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: /Eg/HPGvi5G2PVpkRR1okJp4pbbIMNOyL37Y6Li75xS0dicYwxkHafTFXTvK6+ek27KKdVv+3Ho5Kj4lpmiS3C/7crzzwUIxC6IL8lUx8thK7etIKYrs0kJVQ4dSVfIpn2UwR67rFTBpzdRm1rAUkPrTIDw6CUtbprbkowH+j6eC1ZJJ7ewSvGR8XcuDHoxC8fmJ8UzUoPYcANkIgvaV0AH1gGVqNmi+5LGuy2PuaE4IKqvYbgRLGk8obrR/3X+Jw8vBpJMVJoZcXS/uNgDgTKWQN0pjXkYsdYfYGiLELTj5bs9/TA9AgpU4zS1Rybc2sRTPnX76A+QszmcHBH5uuq3Ur5WGVS2yjOyf+ULroaebh5VMUfUidIKf5vtb4McKUBM2E1k0vTEwO26lie4/J/QPcj9ac0Eb20yOKEhcw+U7MYceNSH7EQutTwWSyhGHNaqKiMkZK+d4rA/28HMkw4ug2RqRCFcBiaIMZh4GF78vkkR3tRPBX9944qWH9tIiKc9tSx8WEdlUzYT5X83XeWw+m4szr7Lh+rq1Oek5C/7Fr0QuBKftrYUHk54YSz5QdOz4U9Czk1ELT1zhl4zysiz2AK6Qzs6fnc5RruKLWK6sNDAXpqJF05yuzfbSjh9rHMo4q/VV73EQZ9mSxaZsaV8zbMArQEhp1cqcw7WYwJUfLdvjJNKaxG18Wu2wICfo74ClZ4UxcK2LyPXXbNm1oEFFG1EUGnGdfGXvvWAnvatgqwAsTqtgnJp9T0m3CnUvx3Qb6EwsotR/RmPe3giHoY9DUGYfid75xmYO3jmj3D9IcDDrxPLtMyEdAKmDXf50gEu7LMWlNXWZZeeSnK254u6kK7xZQOIVlFu1mJlyozSWCZPZRijKTY1yDWQoZ9c/bekL31GflhMj+39UfL5lFVtgbGBaVcHkVwTG1sct2Jwo+G0cORV2vaVQg3dqF00rbE7/2QvodOM36jy3iVtveZnuGzQ/NeVgGYCJOsHIvxwdgj7DuA1CQ/EMDwsi9lrLAwMLijtyOAITcsuCRIk3eupt/4eAFo6gXEyJ0CfzSXBbxYrJzaQJqopxKHkW8eqEciX58a/SN4Y/8gpCdvwP3YX3zxeUW/AmBrFvaKFhA9ew6BHDVYD54Oh8GsrW2KB0 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: c4099dca-1ec6-45a7-88ba-08dba59e3d97 X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4201.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Aug 2023 19:05:26.4302 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: CYn6XHIEJMu4Sx/+dRtxXfeaG9eYX68aEHz+Z4dGfFqXhG3lPnWcZ6LI/UY6BFbiQp4Ci++WmXFmUGUdvr132A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR10MB6580 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-08-25_17,2023-08-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 bulkscore=0 mlxscore=0 adultscore=0 phishscore=0 mlxlogscore=999 malwarescore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2308250171 X-Proofpoint-GUID: 857681GrhwUbjGYotcHlduuiTyZWE8zi X-Proofpoint-ORIG-GUID: 857681GrhwUbjGYotcHlduuiTyZWE8zi X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 89FB7140010 X-Stat-Signature: gcxcmm17xihnfn6h4u3hjdskr3gq1qyj X-Rspam-User: X-HE-Tag: 1692990352-337252 X-HE-Meta: U2FsdGVkX1/uU0GGimTSnwU5vqwOklu9+zBFZvu8f7Q2j/7aWSbTQP5VCaJBQd2dd/Tqs4TOFXuW34i4ci9vAJUjENsndclyC/zc1nsS9th6vQ27eX9H9yXT2zNAiI4ePC7WU56Ryo6ZHCjQC0TV0D9kg4jtWJvu+eLq0vp/LunDSTjLMIUTCQCjEsxnjX+atYngxT8v+opnp5lA9xpogv4JFjmo8CLbmOcdgFdVvaC9Q10HOtDwg06FxAHMnx7ubZB/0ZNSBqSk9tNQNQXnmnduhe1RiQJ3wSMaW1BaWgaiHCub9t0z0DGVwAleA+N2BUeI3aQ+YK0cvtXL8ltQY7qe8XrTQjCsPnE7vxbTkJ1hNt37FNuMTYoZTxXBiNrG4a6UHerL0xD1FxdKXL3o3cvexSb7WkWHVxwxqB6Tw4q5yPLmnNgzE5iUA02BXQQgy/yyRv+gPqNn9J7vkyHDQLL2Ba9vn87uY28tu67ta4fp1KM0GqNPryjT9lEW/l1XHQs2M2+IlQaXEuJXmvk7mLftRayZd+8BG6aTVNH05vTmbZsw3FI4LYffTaDAu2Y5hhtIxs1CF7SvlZg4VWqmeV6c6y5c8bMfZhTNlKdsNt8S2wAu+omQLQqPLzR2czZnR4r3cusn20hM25s71iutv8jl/nhdo4zwEFz/43cpNlJufeW/5uXlxBMp+9QMhigFFzyLVe8w413WlSdffMFK8vwJ4qhZO2WTLXTQMgW3c9Fo+2jSFnXBVUfI4PEujhYa2U6Qh70E6Yb4UrW4sL/pl+PnB6PPAKAiVKwIo5oAnhT/K4VWL1+3YvKYf2un8K+4HL9W5JHiSeoujxY3y1EYs9akFqCwsTB/FHlbZFKA4bKtlS2KsdqAcx8YL7/2IjtVlrM0HxFlduk413tnfuLufn3nRWLh9PyzudUrisd/JFKs24HpsgFO0D6LL72j0QS6MIlj57HZX/06nsGcdo4 yN3dClLr E8uus8wGbzDcWezuvQOQX8AW7jyRANG2i9xJW2WlPh5tcFHjU9O5e7Y8O0OCIRCaFgV9Xd3A1y+04xUmZMTNaZWUxRc8DCy5Dt04yByNwuEq12uwJvOR7hTb7oJSaggerDpuXIMRO90vl1Hb9SUL/+nDC5eGS4yVT8iSIWtu1fdobr83SCct7/WL4IDDwdW+flIRpomTCKMADj7WVALrRUthEoriYeebhKDG9rEVttg211DFRqHuLIkqcmvAOidaoD396YElK8kQDNM0N9lE1lnaRU6yVn9NTm+2vru0L6tFz/5Jvmy28pZba7Cu7NhJnwsiY0fmM/gCklBz6ujTEgG+hvDJVZZodtBnL7sZbAYQfXyp6r0p+CaWFjcyk+vT8wZcE9swOkca+4+NXHqIlCvB7+ik087IXTb1i0L9pMIGi8Bd0NGygl976I25i4DoX7DPGshxRzG6IslIwLTAiCvg/UwTuUjA9J2Cs3mjSwd4+XKAXrPpr9z/LAbYoZrHzzD1Z X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Update the hugetlb_vmemmap_restore path to take a 'batch' parameter that indicates restoration is happening on a batch of pages. When set, use the existing mechanism (VMEMMAP_REMAP_BULK_PAGES) to delay TLB flushing. The routine hugetlb_vmemmap_restore_folios is the only user of this new batch parameter and it will perform a global flush after all vmemmap is restored. Signed-off-by: Mike Kravetz --- mm/hugetlb_vmemmap.c | 37 +++++++++++++++++++++++-------------- 1 file changed, 23 insertions(+), 14 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index a2fc7b03ac6b..d6e7440b9507 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -479,17 +479,19 @@ static int alloc_vmemmap_page_list(unsigned long start, unsigned long end, * @end: end address of the vmemmap virtual address range that we want to * remap. * @reuse: reuse address. + * @bulk: bulk operation, batch TLB flushes * * Return: %0 on success, negative error code otherwise. */ static int vmemmap_remap_alloc(unsigned long start, unsigned long end, - unsigned long reuse) + unsigned long reuse, bool bulk) { LIST_HEAD(vmemmap_pages); struct vmemmap_remap_walk walk = { .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, .vmemmap_pages = &vmemmap_pages, + .flags = !bulk ? 0 : VMEMMAP_REMAP_BULK_PAGES, }; /* See the comment in the vmemmap_remap_free(). */ @@ -511,17 +513,7 @@ EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0); -/** - * hugetlb_vmemmap_restore - restore previously optimized (by - * hugetlb_vmemmap_optimize()) vmemmap pages which - * will be reallocated and remapped. - * @h: struct hstate. - * @head: the head page whose vmemmap pages will be restored. - * - * Return: %0 if @head's vmemmap pages have been reallocated and remapped, - * negative error code otherwise. - */ -int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) +int __hugetlb_vmemmap_restore(const struct hstate *h, struct page *head, bool bulk) { int ret; unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; @@ -541,7 +533,7 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) * When a HugeTLB page is freed to the buddy allocator, previously * discarded vmemmap pages must be allocated and remapping. */ - ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse); + ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse, bulk); if (!ret) { ClearHPageVmemmapOptimized(head); static_branch_dec(&hugetlb_optimize_vmemmap_key); @@ -550,12 +542,29 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) return ret; } +/** + * hugetlb_vmemmap_restore - restore previously optimized (by + * hugetlb_vmemmap_optimize()) vmemmap pages which + * will be reallocated and remapped. + * @h: struct hstate. + * @head: the head page whose vmemmap pages will be restored. + * + * Return: %0 if @head's vmemmap pages have been reallocated and remapped, + * negative error code otherwise. + */ +int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) +{ + return __hugetlb_vmemmap_restore(h, head, false); +} + void hugetlb_vmemmap_restore_folios(const struct hstate *h, struct list_head *folio_list) { struct folio *folio; list_for_each_entry(folio, folio_list, lru) - hugetlb_vmemmap_restore(h, &folio->page); + (void)__hugetlb_vmemmap_restore(h, &folio->page, true); + + flush_tlb_kernel_range(0, TLB_FLUSH_ALL); } /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */