From patchwork Wed Jul 14 19:35:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12377733 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6365B2FAF for ; Wed, 14 Jul 2021 19:36:18 +0000 (UTC) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16EJVVjd010180; Wed, 14 Jul 2021 19:36:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=8lJ7EDg3tbbYqO7fNYtskGSK8ysuHvu4xJBz1wDuqS8=; b=pb3p4lDDgNBXmBNCgIWSQIIi1gAH25Zud6KNr6SSi/HBYOM1s4PxTRRGWtJpPoRHNIbG POJRW74XeZsw6hrbuvIGzd6qUvaOx8vfh40dgp8KpJEk9/jsB5iJM0MiEgNlcrsf0+Nm W/IAEuED00mPHJKU6I9gPQ+vPH1+XYlJFMGLde9k4d0E0se9lxXPNsk0HsASbc4eRH/7 xqQQ2RK+nlHIHbg8XeYkQTB8kspfXv0hH/Fd2CbqT9XnUN2GuggJegwpipRezpeDTdL0 DWffh7Ec8mEHWohVHwwJWTz2fUKw03IW+xMqZ7fZ04LaVyUPVArZcBNJAzTIBK7BFgqq KA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=8lJ7EDg3tbbYqO7fNYtskGSK8ysuHvu4xJBz1wDuqS8=; b=sAXUvwPHsaJB2dqyYmATcXocB1opgmdoQP4hYaY81nFjZDc5xOkf1MsrEaZJqt6wD2uX 5mCO5kEL0rrayuftJTeFHUs0AVrV/ZgKaCjQYlEt8yX7LtXqxnyVBciBXhDaU/yeXEX5 0we23+pdlxayFdoNigqDl15ulTDXciev4S04FOnQrXtrK35S1dh0/RL6AOhxO06o9w/e rhqsOWAmfRk/C860yUrHWhzA0Kb6+NeOZJeia4zqnfsYaS+kCDHvyD9+4ux6sYJvUTPk 2pZx56TglU5c1RiIPZGiLrO1w2lOqxOA/Cp15RYBGBBuhAGOPFrPNOwNzieeLFcnry1F EQ== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 39t2mc0fga-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:02 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16EJUwDl187323; Wed, 14 Jul 2021 19:36:01 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2174.outbound.protection.outlook.com [104.47.55.174]) by aserp3030.oracle.com with ESMTP id 39qyd1c5c6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:01 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Tkb5DNyy1XuZPaa07HmFdVkJ/YlWQt+wjH44X6TwAzzeMb5Oh4WKwAmHn+DXNxOa8W1K2S/qwPQv66S1fGV+uaUzj3BgXLM/ZD3CgrmosrNmhmh86+oD/2WrjZcnfPCW2kmn1IqNxV1tyHQu6wLNaYALUCdRliztcRJQge6MI0gzEYGupntZdwyYaX1BWqi4xMh9bHPo7mRfzyRlVsBNQQEQlbXd346FDmCPfnrWOPxXOaI3ECOiCF5MSUShvPErdRv6YnO/Q/rcs7Dg6dftO6K8rxkd4oA2I5lF6vF4tVba8zkPdyyB7Nm1h9n/jsFy+cLBX24A4090jFQGAAD+Zw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8lJ7EDg3tbbYqO7fNYtskGSK8ysuHvu4xJBz1wDuqS8=; b=FWfz8XuU6LNCDNjB5i8ctz0q1P1zVb7i8KpuuWpJaO3zKaLhgxM2iPuzqM7Z3ix4bJzjJVGxiFlr2YqHVa4P0oT4D1b4tpbCMostkMgosThior1gcAHjjeoT1eZlDVLnJK55gJtx1Sz34oEjHnIqG4w1HXQ20gCsy0KVC+gbkJNPi66oKS3ojqrA4R4aT2HoKWiLAfH7ySOs2cIoVVGeroRalcR8FQLAFbbf8Dehid6OHTb90Syi5Z2OGoZyQAzlxp2Pro65CQlWAtEKEs4katVNCprwzn5DjRXo/yf5+7THLanXu+Z530EFDovzgH3XU+P8rvvHtNPDT/idMTyDIg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8lJ7EDg3tbbYqO7fNYtskGSK8ysuHvu4xJBz1wDuqS8=; b=T7Nr1YRNIDghhVBGq+HBHqLkarpJ3O5FLIj8iNx60bdPMAPelrbWltKLvkVMwSmchsI321HY6Slv5eA4NhZhoa2TtgU+jjkvJYxGSxWya46jwk6CGjeNrnoVi02I+I9LmuFFil36btnn2GEnMyCfHf+FKFvBylIlePWay8FpKjo= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by MN2PR10MB4080.namprd10.prod.outlook.com (2603:10b6:208:185::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Wed, 14 Jul 2021 19:35:59 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360%9]) with mapi id 15.20.4331.022; Wed, 14 Jul 2021 19:35:59 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v3 01/14] memory-failure: fetch compound_head after pgmap_pfn_valid() Date: Wed, 14 Jul 2021 20:35:29 +0100 Message-Id: <20210714193542.21857-2-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210714193542.21857-1-joao.m.martins@oracle.com> References: <20210714193542.21857-1-joao.m.martins@oracle.com> X-ClientProxiedBy: AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4331.21 via Frontend Transport; Wed, 14 Jul 2021 19:35:57 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 36ddd624-a9d2-4e06-5a6d-08d946fe9b6f X-MS-TrafficTypeDiagnostic: MN2PR10MB4080: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: czqebY0y8Qi1zlKIYb3HVr/8JbmWw83aN1ag8Goln0gWCdBhmpOmT8KzFgwagRse7/HMWjyXWXwwK5memxOyd2Ud0m9kJbBB2xk3XOBknMHfZKtVgjSdzDia9fPk0zMaBRNI0nRM/ybpuVhiniKgSOTe8AVTly3GF8rzGyTL1He5pIr5IOZdSWlCtHTC1DJCzmIxQDmaaGy0XUAOuxuoShMWbMBDoB3hNfCJ9x7KMGA50/IkYE6B3iqM/R96k69LKRiglmw0R/a6kmjAsjchzm3BhVcyPKcsOcsa5qcB6Wkm799lYYzZzSLDtVGL/oHuCA6UP4GlxSWZSggedYi+tFa2JjDssopIUq2C+mePd5wle/Qi6vbikPN0NUEb6zD+vNJE4tbck+3Kyj+Gq5vxLZnONkx6GaMavwDOdwzekeJTDT9YITqyw+Xux5BhQRejQwsAW0ZnJz/WyMlBOBOdyuE5ZbtuJnXtCjR4wJi+LsxR/pxOD5P/HQaK81ijtmYEjIprRZV3GhdbI3sfUt0tc8g0YbWEus7I1+wRcjzUJAA256FwPYXEewE4NKbZFFcosSPGNDiSCE/FbcoAiDv2nfRxRecNv7Ac8dgpUh2Hm3gXL3kE15qdvEOZp2YfxNo1AORwGvTky3euXAbs/QhYVkQAr6pb3CG9pYDactbe9fCGg03ck7Q8FzMOALONcxCVzQGLNumbuV0m1Zzfwcm9NQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(396003)(346002)(366004)(376002)(39860400002)(136003)(6486002)(8676002)(54906003)(86362001)(5660300002)(36756003)(478600001)(186003)(6916009)(8936002)(4326008)(316002)(83380400001)(52116002)(107886003)(7696005)(66946007)(6666004)(1076003)(2906002)(26005)(38100700002)(66556008)(103116003)(956004)(66476007)(2616005)(38350700002)(7416002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: uNBqsHHiQUA5aMkW3m/X5ZBkwpRE1gpsa5lyjAMDhl6HT9xlQ7GFoQSi1H456J9esvY0avLT1pyQBKjoe2pfdTmgiLQ7ktXCUAngTDDNN+hE91eiI7lI+sz7IDk5Bw4Zz0QN43Gnq0fZA+fiJjsfp2mhgTfTtaTaNAY1SJNJ426/vOGsZZhDfSHkhF6h+VgjtBqMETZ8vtHtUgT8tFq7WzwOeKsHwJGQpsL5riO1EZr9D+fyiOF3go+3oECOGPOgAdbBTSUxew9F7vCeWi6ICNuQ9O4eAPCBdDo6s0kb2GxfOqhLuLDDo3qd+GKVB1U+9R36+G85ao7t3sqgmLdlcof3L4jNwzjLlNe/psb6FqvvurHoYAA6Of4ZjQLbpvHszdtG6YWBqqfh2QuLPOeEB6bhaLGN5d8gnyvmlmT/haXTaRt5R1MLS5wg4ldyYOO+9Mf2zXoL/1x/D3DzqeHjlvgxRDuPgRo/JZ9SLsZDLIc2g2OiZH5ZVbBV7g2/gMzGB23m+V5vaYqwxG9LhH5OsAwx73U/uM9MjCOlJzD5e7UOCsQb+tSjXvrzp2Vw4Z7+Ev09w9b7bDkLAsqjJUpv9cJmW2tEY+AWoKN7wXofnELZxAl/MQrZinIrkQOtEuF1uev7uDPhxVCR7tsQDiQMKJqjIfXo3Xb7lGmcQr/eNGY42MOLoPsPbuHT/lREBc2z3YS/vkYhraCdcWvbaeoawwLGBMt+0LV0NdfbNX0hrMaOk3zTInf19OZnUKXFnq0DcU2REpCYE79lwzq/eojEBvEgRYGa2D241PJi4q9LhOBk2cAdq88fcZi1jHjS6kZPq3mJsTO6LJCq/wfBiSe1BsD8Pi9pG6JSdXvhcuI4MBJu7XXTGy7WFsNWY4bjLTykgI4IkfEyrr0X9faNqh3dNYVt1LKr2TwZcBsTkrubPWI0a3APSGVNF2ueiJzC5xW52ln6UCFBQy753YbriahTEgXNr8bkCeNGbxQoEuJftzYtxKn/fAfHQs8bpLRU7S6tz9kS0VTAnS12JcIx6gKlzg+rs84FfQdIYS95p/lXNB+zg7q3m0oMQiysy0+tDEX04YVhp217VGTkixcYhpsCi9ZHP4TT0PIg61Rr5yoRly7PMtlU0J1pyqz4CQLsNdPwbH9Z1qdlYzK/jFJULDbL3pF9fqJ2wgZswbEA9q1/4GCsy3qT2mfwrW0BpDvpVvjM1r6jVL4SCWzPHkydJ2d3nnbOQ3q+llqcSd4Yy/A9oEshka8OAx46ui5+U9PCFDiNdGtM93/VkznSHdoHyFRW1QR2/iAAzzheE9d3RwvVyHTyJWBEnfjfulJ6rILrG4mt X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 36ddd624-a9d2-4e06-5a6d-08d946fe9b6f X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jul 2021 19:35:59.5847 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 2pp/3YptwSx5UhxTA5t7nCs1gywm50Z3MPd1r68mzTx6v4PlaROpyVhtDSFYQTH9Afv3ULKIqkFxqvlpTzx/oUQUXnRaBscZIoPRIRtbGjw= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR10MB4080 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10045 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=999 adultscore=0 malwarescore=0 bulkscore=0 mlxscore=0 suspectscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107140115 X-Proofpoint-ORIG-GUID: iOIbAJE7mU0kga6Z5JOSybjdCA6_YZjU X-Proofpoint-GUID: iOIbAJE7mU0kga6Z5JOSybjdCA6_YZjU memory_failure_dev_pagemap() at the moment assumes base pages (e.g. dax_lock_page()). For pagemap with compound pages fetch the compound_head in case a tail page memory failure is being handled. Currently this is a nop, but in the advent of compound pages in dev_pagemap it allows memory_failure_dev_pagemap() to keep working. Reported-by: Jane Chu Signed-off-by: Joao Martins Reviewed-by: Naoya Horiguchi Reviewed-by: Dan Williams Reviewed-by: Muchun Song --- mm/memory-failure.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/memory-failure.c b/mm/memory-failure.c index eefd823deb67..c40ea28a4677 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -1532,6 +1532,12 @@ static int memory_failure_dev_pagemap(unsigned long pfn, int flags, goto out; } + /* + * Pages instantiated by device-dax (not filesystem-dax) + * may be compound pages. + */ + page = compound_head(page); + /* * Prevent the inode from being freed while we are interrogating * the address_space, typically this would be handled by From patchwork Wed Jul 14 19:35:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12377739 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7285B6D14 for ; Wed, 14 Jul 2021 19:36:23 +0000 (UTC) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16EJVWTh022332; Wed, 14 Jul 2021 19:36:05 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=QF58l7lLOjz1kbBPHdHIdxCeV2ODkZdvt4bXLCnoySs=; b=g2SSLIdzkTeskzsE+vk0hGwkdn/Uwgx0pv23vOVgfZRY0EXCIALfLRB+YDv6Ffg41tfN iIjshxBh60Y9U91IX2z72Mj96yLpw11rSJwXWFXZutcvSCoeMyCg4Oz/TvvLs+jUVNHS ShN6VpPgBaHLwK4Q2lV9hPDy9+63XZsgaWfpmlSCN8JGmLcXY63gfKOE9UAFVMJd9E/I nGgzzpp21z2oVIE5Vb6Bznt8HaQlGMgdXalZvDrDwc09dkBZfYV2oGVHPhhydYECkfNm vqt6aW/oNvq1vn3VrL/N9hhKXHVn3GG40G0pyLy4SiippgYZF9M8tBomzR7My+d7qtf6 7w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=QF58l7lLOjz1kbBPHdHIdxCeV2ODkZdvt4bXLCnoySs=; b=F2b5PseRbQpjJ3+CnXBci+DtmywG3/E5w3hCb0zlVljndW9l50nJ+RSimX0YihB3mU2U HmXSEoAlO3humR2rXWMf3LBumTgUip5hPP7bYgE5Tyj+FZAqGInwJLOttsxOO1gBCWK5 C3lIMlqi51xx+Wqi3AxYbJZStM1FCEcXOpxjAXwVHXhx7OP7veotB8NKSdnf7iwuU6mQ n2NVOy7xX0t1FW4s5Un87F/T4do8UJqohEJFYioy24zRKCPuIQlY6FZhochn9ZSyxDTU NoPPN7od/esAU27mIEYLQOVfZ8nAyfHA0hJofoZoTrojDfdyQMn08nKAIq7u6PC8yz7o RQ== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 39suk8sej3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:05 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16EJV0dW187598; Wed, 14 Jul 2021 19:36:04 GMT Received: from nam02-bn1-obe.outbound.protection.outlook.com (mail-bn1nam07lp2040.outbound.protection.outlook.com [104.47.51.40]) by aserp3030.oracle.com with ESMTP id 39qyd1c5dv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:04 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NbaK+L8d3F2l4Ed8FhkWh5phuPHO+UiFgSG83imBVHpvbg9Ohfg6xIA+0zQVr/di3uwqEJwtDs5Pqd8kvyEaUmtHroMQNhiqrVlxiGHlF2iAvyAI2oXHe/Y6+T0O888DEgIR7jQaqCdJWotFBsSS7JduC9zBnyh+l+Q4mdLAHXtdc0BtgfB0xL52SUiiJhXW4mveb+DwF/YxV4tRnsGU9LQPrdESKyrhzaQfA3MM0r+1wVyYt+pvf8nzG3WFsEW9R3yV5dr5kctQcZ4SiQM0xcR9xIaCuyRQkbdUMSVRfDAWIfsaADYFfUq6cB8E/JzGZ1vApjFwVa+ULCAlGqHOLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QF58l7lLOjz1kbBPHdHIdxCeV2ODkZdvt4bXLCnoySs=; b=T/XA2C2NQtpolICafVe9PIO0h2FMa9Z1iF7JzdiB/dUykV2BAzDF+m1JEyCOCD68fsndoPTansLPw6wa0GF2zVtG4ExzuoxwiI+2P/bELTWzIRyBoEq2VAFhkC4X/Y3pG/QJSk+OKD4oG1ys+53yDaosqAy71x3/jgnPOpmdNU9twPmvyTXRszPz5YwknogVuJ7tQim50NXzp/Jxe71LhfHIpxHUo/JQH2KT4e45cjtQ1MTODcn4v+fQtr10RNcQOVXxO4Ik7NZ+b0pyAnT3UlojKTkyCjacqu74T8u2mWcyVFMt067lfhASUnPcz12HvCM2ZFZgX0X3BOBKh86Kiw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QF58l7lLOjz1kbBPHdHIdxCeV2ODkZdvt4bXLCnoySs=; b=JdjrS06DMzRd+IcKEiZKPBgKu5MR+6qXSPLlE8qgCl9JizgrScyH9r3zjKJAjcaMMaFmgb40GTaaydKNayF49K0hzwA5f8LvDyAjynsN7TSvn5u/pROi6nHO1dbPqbqkDkkqBMeA68MnQemqbEULPB9wFP5CVD0gAG4fsK342KQ= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BLAPR10MB5204.namprd10.prod.outlook.com (2603:10b6:208:328::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Wed, 14 Jul 2021 19:36:02 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360%9]) with mapi id 15.20.4331.022; Wed, 14 Jul 2021 19:36:02 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v3 02/14] mm/page_alloc: split prep_compound_page into head and tail subparts Date: Wed, 14 Jul 2021 20:35:30 +0100 Message-Id: <20210714193542.21857-3-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210714193542.21857-1-joao.m.martins@oracle.com> References: <20210714193542.21857-1-joao.m.martins@oracle.com> X-ClientProxiedBy: AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4331.21 via Frontend Transport; Wed, 14 Jul 2021 19:35:59 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 68f4687b-2fb3-40d0-15a3-08d946fe9d15 X-MS-TrafficTypeDiagnostic: BLAPR10MB5204: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1284; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: W4402u+lEF6pNB/KvYG/27tem37/Dz3TK4hf/6uOKB7ylMdEAL/7LDqemvBrZ818OLzZGnB3bQvqlZRZQfhVa3z270z8MiiVWjjYJqjeKbUrGbrvt0+jw+/cYwVTPTTD+QJFS+K/ZGsH4dv+f6XVsw9tuzai8P4ngjaTLsoyP2uicU4rlO06BW5qd7oClxCSFq5VrWjoKDIi5FYRjTQhrUvT0yffYkdEWFrr+KjIdkSzXR8l+cfMflzYsqIvRV73AlvrLjVkdECEKi34TTnytsV5tEY9B20Xkb9qT5etU6pScOulpiV2DGLZ/6FZRm3/O2yUIHK4oLCVUUEYGn1urgdaj+YqkHqoLGJ+XtmyEQaB6CN0qGMQGZsbP9H/p26zgLTUfOaISEnkbac3SlgIJs86pNhKJZasNZCX4AbRPaGHki6NjxlMR8ERZOmAcdV2whRhVyofJYfhwSvW3xGjJ5LqGSIO8izchpvGtx+smqriQpHY306O5RS0ltamdWFklZId+7wLBGPys9AirsmjvLT0dybjQGFAJh/Xii/BO4fKbuurpOMZ+ky5YOMyZZUcU5pE0Wk/x9Zix03nqnXGoqsYrTmLITK7m4nzx+fDTpH+PaK+DCnZ6JNO9C+i3AOQMPUanhvweNgxvK1USIispKiea+BAVDWXt8Jy8zna8Vr6XLBMbVS2torS7B/c9se2EXe7t6IDiShuLQ3QgbCTsQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(346002)(396003)(366004)(136003)(376002)(8676002)(4326008)(86362001)(6916009)(7416002)(2906002)(956004)(107886003)(2616005)(6486002)(1076003)(8936002)(66476007)(5660300002)(36756003)(103116003)(83380400001)(38350700002)(38100700002)(54906003)(6666004)(66556008)(316002)(66946007)(478600001)(186003)(52116002)(7696005)(26005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: rGiPgujiS2Ro6sxGUe0C1zPPklx8D9ELJ3+87mvbBmI/9v+Q6dTaFYeypRrVw0qJdXZuobP8I7XDXMuo/gcANy6+OUAzyPfAGyx0M3Ub6s1nUYqB4SR3SbuOEcX9hLqsZ499rO/+02yAv+kU/6w4sdCw9YmOQRPG2XtrELnCCLQlr0nIG8khhupngIuJaemXoNcMaiJW8oVmQshz/VFw8Qcwp6O+eHlSpHB24k84BFV7KcZeYDjxrXyfwtt3EdY7/2Q6+e/RKObiDRbp827O1droReTpkVdzsmx9ZMdiNFP8lxHoGWoQld0WZMFnyrtfRIJb1WvqgAhKHtQLn183vUnGwGoCH1YWMSRZm6qlWsY56Wzo57ohYKFrdKUcN3reRciPrsyNuxKvSKXqJcRcwnS9bPMLED+pMUoc1xMaD/ZpnYM8Ln184UJxn5JKfKQgJApP4vQcH/sTJjpa2bdoMoHK8r11LrRyaaLUxVHkm/HirJa21qj4+HqS7wR9HFFLJrx+4J7NltX8BxQLHatqQHP5hxcJ86/quWjuOjvQy03JN4kj+IglVKYkGk+9Tx1du7apoUhc7SQNASWcNU+OcUSw8WfWMU6Tk8btB2BlzqDFTtNJPjzSrpmMXfBj/ZE6e5AxaRecJip+3c3R7VaNRdvofnmq6TOlcLmvDVvQ0gM9VLHX9ZFSihMw5DdchT5uJifDyImeBu5ETXOwhIeORcnsJipiDMDtpHEdcd4A9v9JdrreUDU4H6FUc/Ow5Jb1sfwb01DY+DjC+y6hx7PYsLYKT4YGDdvs9O+CwTjiQR6OCkN3CbbXpjG9S3QVp51wX5vrGWosmbGcLatI422g/bhymzefzZuzNCrK6kuvqiJFge4wz++pZYC3ok4UFolgMMau6as0JuyTnXS6w8fYtJTAY2ulFNqSIXzf1sRmnE1xUwbPofF//LFfnUNqYLOhFd11O9B/yBdoLBBJjShDP8WwAv09T5x4bYkLXKhuBR0xGQ2mEX+tATR6+fTXBtkGx2FwaS0X7vzy0TybcjG/BhrCNkMFaMXYUGuXpkmLww4NsBaNHqF846G4G4JgVtjjyJk2X/2H613nWgt6YZ7S+scjVs6JCbt2CZEZVIOpCDTh5Jbvf3dnr0wPnYnxMLLQXwXp/E5A6M/Ybz/4vN06GB5qSK7M3kdEVX1UvMytYalr4xScVpcop12NdhPprDl3KeJWpGnoo45b7XziNwFw5fUaaVgS74pJl1VUTEhJo/WnzsFwDe62HNGn5u8YTExJcLbdIB2KCBqwQsV+wiFs/JVnXhUej8PmIVlMWJEIztFsWgPCTA1kr4sgbFHbVk+9 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 68f4687b-2fb3-40d0-15a3-08d946fe9d15 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jul 2021 19:36:02.2888 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: HEdCe539MzOzuQy6AG0upgUaUH3ZAzDNupiLO0V4Px571kd+OsY9aFafMrA6l1QDDmIvITDtdgrw1L56L3GcaFKPsQ7bFCvIy1s/cTvfD6c= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB5204 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10045 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=999 adultscore=0 malwarescore=0 bulkscore=0 mlxscore=0 suspectscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107140115 X-Proofpoint-ORIG-GUID: mpdWGPn4O0DOK9HHpSBtNXM6pFrP1MKR X-Proofpoint-GUID: mpdWGPn4O0DOK9HHpSBtNXM6pFrP1MKR Split the utility function prep_compound_page() into head and tail counterparts, and use them accordingly. This is in preparation for sharing the storage for / deduplicating compound page metadata. Signed-off-by: Joao Martins Acked-by: Mike Kravetz Reviewed-by: Dan Williams Reviewed-by: Muchun Song --- mm/page_alloc.c | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3b97e17806be..68b5591a69fe 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -730,23 +730,33 @@ void free_compound_page(struct page *page) free_the_page(page, compound_order(page)); } +static void prep_compound_head(struct page *page, unsigned int order) +{ + set_compound_page_dtor(page, COMPOUND_PAGE_DTOR); + set_compound_order(page, order); + atomic_set(compound_mapcount_ptr(page), -1); + if (hpage_pincount_available(page)) + atomic_set(compound_pincount_ptr(page), 0); +} + +static void prep_compound_tail(struct page *head, int tail_idx) +{ + struct page *p = head + tail_idx; + + p->mapping = TAIL_MAPPING; + set_compound_head(p, head); +} + void prep_compound_page(struct page *page, unsigned int order) { int i; int nr_pages = 1 << order; __SetPageHead(page); - for (i = 1; i < nr_pages; i++) { - struct page *p = page + i; - p->mapping = TAIL_MAPPING; - set_compound_head(p, page); - } + for (i = 1; i < nr_pages; i++) + prep_compound_tail(page, i); - set_compound_page_dtor(page, COMPOUND_PAGE_DTOR); - set_compound_order(page, order); - atomic_set(compound_mapcount_ptr(page), -1); - if (hpage_pincount_available(page)) - atomic_set(compound_pincount_ptr(page), 0); + prep_compound_head(page, order); } #ifdef CONFIG_DEBUG_PAGEALLOC From patchwork Wed Jul 14 19:35:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12377727 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D5AF2168 for ; Wed, 14 Jul 2021 19:36:16 +0000 (UTC) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16EJW1Cd025409; Wed, 14 Jul 2021 19:36:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=tqBpzLt289ReGCbu38csppoJJep4yCygFu6meR/8Gn0=; b=f7FvNdWAHglyx7cxDawO9ovBwYatMTzMDH8S0fClubGgRuvyPQghyEjCw3v4Po3Rg0oC EFpD0GN72cE/fkSGqsph6CbYNMnipCsHQACMXFx0vqYk8lLtAzD7qFhcLvr1qkkBFOyK LPzK30jq9z1FVnYTBlkjLoWSLj6Y6TcdbCrsHyLpfPye7FFIpg9DGNIEUTiSibk/3Pjq dSrqDEBSYD+8Hdo1+2J9PaX7OWDxmSHoK6g2EtVOhLlrgkPkN7ayTxKW3rhHrfyff/Mp VYRki3Dwj4PGoQXvxznoGP40gb1gGItJCN0zqSGjFaAQYgfiX3XRi2MGKkNky9X9ZGDZ oA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=tqBpzLt289ReGCbu38csppoJJep4yCygFu6meR/8Gn0=; b=0TNV1YTkzbNvmIYebkOYhZ0/PwssvIq1MWgk7wQTFBwaAtJ2CQL6Sw6cqGW8tSVd+cHy LdYvAQZYq1eoZEoH9K4Kr02HKKWKXs4tYVJnF5XTZAVlC0JyhXe00Yq3UrCbPxTkJMDY jZTtLBhzXdVFCOiF564fQBAOf6H7q4rfIMH1Ci+qQRjHBRDnoVvpDN7ENWevVi+otXY1 H1VwEpcs5SmaOwcGq2Q4BFs811DQSYJSn4z+eFQKBSM2d+0i7TxIy79/t0zOqLponMam cmjBegCo1zTyS0pb8VlUq2uwhrYRaDLrhax1rjB69jyOYL0wlOIxo9QuwcBmod3NVPWI xg== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 39t2tj0eyv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:08 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16EJUYrV021568; Wed, 14 Jul 2021 19:36:07 GMT Received: from nam02-bn1-obe.outbound.protection.outlook.com (mail-bn1nam07lp2047.outbound.protection.outlook.com [104.47.51.47]) by userp3030.oracle.com with ESMTP id 39q0p98kq3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:06 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZnQTlv8Rc0D3oZqLSGdzx5yVR04o6BMP1zsXWWzKGwBxALWnlXGWhoWtI/Y5cW9hBXY/beoQRpu9Y1QcydlLooVodUS7vBokfLI0R9UAegyOCMqGAJCfxDTXezhAKccBwaSew+2VB5LgJjM4paosXvp6WqNOC0QrJBU/gt5BKhPOM46+O6IFWphkVMCuIkE2+dboctv6HJbm5DRaoDhonQjeuefIgT+HMKX4b8MvaHnL+STZ3iYlcypzDFLhKqYVPuwRzdI8OpDOWkIWQNcFtQjEL9ALhXjq/8Z8Zwc6dxa4KL9NkdDuLuuSPxE62Nac84U2WtJReuzhRkw66JnltQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tqBpzLt289ReGCbu38csppoJJep4yCygFu6meR/8Gn0=; b=T0T7mB6ftIVvnSqjJDaaXk4nTccNP8kuEv58+Zk5PtBMhiOozkBNNF8l7mXo8W5i16Df28ECuU2RPxND7WPItZJFtAWF4FFONQF//W/fee/BdWe1Iqy28wGQLAclvXCOEEVw+18oZnVJpX1KHuwwFdwrxGz1qXumtV5amy5NqwJ1CxRGq4o0C8xRKjgZPdOf5P9rW5hnen0n1FXir7dstjk+FMR7IqhllBPpFhPnpyRP/6fLNsr9rYWA5ivt1EPMtuNNwaLT74nueArniKg7TZifyKeKeQfjjm5shBgJsB1lgapdeVhfak/egKrSeTq340VZ0HsKNDgUeq8ppN+awA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tqBpzLt289ReGCbu38csppoJJep4yCygFu6meR/8Gn0=; b=P7p+h0u07I2iOnbas8zZlxPHabqCVbHDXt8zn4OmRe1eBviZo7wqOsFX1R9JsFKuSbWNXSihTVDBTpn2Cf8i0KntvmfXVWnJtvw2mwhJKHwFzj8xnacxvBc41TbWFUMmdiCFwcP5SoZT0gu/1rwqVpLnWeKSXqUClh8b7TkIFXM= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BLAPR10MB5204.namprd10.prod.outlook.com (2603:10b6:208:328::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Wed, 14 Jul 2021 19:36:05 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360%9]) with mapi id 15.20.4331.022; Wed, 14 Jul 2021 19:36:05 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v3 03/14] mm/page_alloc: refactor memmap_init_zone_device() page init Date: Wed, 14 Jul 2021 20:35:31 +0100 Message-Id: <20210714193542.21857-4-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210714193542.21857-1-joao.m.martins@oracle.com> References: <20210714193542.21857-1-joao.m.martins@oracle.com> X-ClientProxiedBy: AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4331.21 via Frontend Transport; Wed, 14 Jul 2021 19:36:02 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: cf23e509-d98c-4032-146c-08d946fe9eac X-MS-TrafficTypeDiagnostic: BLAPR10MB5204: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4ONauQ1f66fd4QXLrk0hnN/FCJ64disUyMZPbbnws+Onln3iIscMxdRxpdOolh+dPuPRjVVvXDrpm0KlJHXFv1VYzd5R0IwjJi/pSK6xkiwWdJU0rRTEabv9tkdl0TCoF34Y6qtpK5pNV1DRpFA/9EhNB/Mm8v2+k8OPCvnr0po2/ZivRWAl+9Hhy2ZtxPr7e96FjF+qp3MEYXyLoWVlSWzFEDKxQ6OnFbpA0F+/nMxle9dJOpCxrb1jHxr8193N3IXqTMwrce8DMaS+kTsbw1VhIabOAYix+yaVRLjUpJBhFdO9ONYqjZ/ukP/IZKSlokxfy+W+9BGGvVIZz5WFrjkY+St+hb7D729SZGQg4mdvy0ULEdv9XRbtGS3pOhmG3wUuYvBAPECEOjmFH0/BWGgR8xSDDs21iFacf+qO+rLUZd9SmcbFFYN5TYh5z7AD0S5Xf+i0vuyojagzkgkJPYAPYEi/+vrvSi8hZvx6lJv4vEpXtgJtEADb76IlbGbv8oIZCeclR0tU6umLayUPQdcgrAKVWDoJF5zMSkBjtXJfh3ko7QfV2JVSEN3QcyIveBv4VF0m36mug02V1u8CPXpD/MJCPOf2btbN+yRzdroyCMfpidLo3CiflBtaoRKmn9TEtivYa+fNmywHf1wL7j7rKNZoxXsaAeewuGtKxTueYLe3kmCjA3z5zxst6j7vLU1elVWvIf33wbRZ34VcYA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(346002)(396003)(366004)(136003)(376002)(8676002)(4326008)(86362001)(6916009)(7416002)(2906002)(956004)(107886003)(2616005)(6486002)(1076003)(8936002)(66476007)(5660300002)(36756003)(103116003)(83380400001)(38350700002)(38100700002)(54906003)(6666004)(66556008)(316002)(66946007)(478600001)(186003)(52116002)(7696005)(26005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: cEcDA3dVwJwOu6YFq9efNVCIIrfy9oZ44PhY3Ggc8A6b9liWCAoxvzm4WKntw9rDYMc8S3TceFOvX374M2t4xqTNPBDiFLJmIPXJtQ1DI7/olDkSwINZtcEStNdJo7qc/6MjupZA/WRy9J3DlTj0rMkoRY6dCn7NHb0/NRxUPbvnbOQsGh4/1HqZMENazb0Qgd1ysEl8Wccvvp6Py+0eExV1+7eT761YzPlfBtv/cVyxpbL0pq6/Ln5dvny7KWqVDHkmPdtgrdXkMSUJhgdoNRZY7CPBgBI/KZn0fkqMnvamAkv1mWM5MoyfcQyh+9mmx14g75pWzFtxx/25G98Afpns4UK8wO5NANhCKaJ95YFp1asdd2S87Sz2Q1o97VJNlsN3pKtRSIF/4Y3xgVUOt4U4ng0jZIuIOKX0Pk33nYMIjjkuHF6D++15eEvwvLBVJhUEe9x7K8s8ek2mwikV4H4cS51FxmJSVU+4ezmAwKED02/ReZc/X4P9eN8VDpjpABs4w2FRzhtcie7UHPBz2AHt6/tH8UIEyV8He0dvJ3IG+qC9mnQ5Hpm4Zjnxpc5+DfUVJDMFREihIp0iPwBn3miGyD8AExPgvnKYI42Ah9Y+ZGwOsuz3lw39gbFYXqVhMAGTffM2MsO08SAcxldRofvOP7jhuJ34EP1vnbcsP0REBimNMDn80umrDPJRp+aaTnXABAJkk7wMCULSp3yHKv1BK35p0e74tp/2OIBc9XY5gTe220dTbXNRah84Kag0QTQTZgQS3iYAKz4dB4+dsLFUfMJHDPn1Rh8EBGdYWLSPYC0+EZVrCUcvACf17iY8VOHX7+Q9uxI9BP8uj10d4yF5nXtOGwfrOgGjof19qGE7+XSHg5r7GCBG/Q+4DcDjvrivB7LQBtt7D/qiLYeLmk2FzDVpMRwhwlMMD+Z+QwgmmFJHwO4LJYJQChTgbs72NXYh5p+ztR7/NFjmJFNB3pU7hV8oaveKt9WLKjh/S3488zrU/+EPDMGjQ8vYJXfSvuM4ZCAYo5Oe8FlFQCmcTzZmPYGjUy2Ho4K+m/Q1RZyxsCgjvSekAxioM5XKvtCdj/XzZ0ihhntuHcuOYra533TA9S3cUA6P8wFFCj9u2VH7xfj+4vNcBT+3BcEHrgds1ExO52IsJlYbj2npHP8PGySOH601Ez83wWheU5KgmMj3KnZ1F1yT5eYYmAQfeXqCUAybPXqhmpUL7yv4shu6ZlFX/ZHGjNSxIlEFQFS21viU4XnXtTcZXNJ6Oi8kVQJQXdTP0BcmwWpkjzpAATj2WAuTOa9HucXxbww61Vzeo5wkqHYtrN4ExUY68zNFfMhH X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: cf23e509-d98c-4032-146c-08d946fe9eac X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jul 2021 19:36:04.9242 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7QNljCBMEY+VuFdqjk7tK2nPZIKbNQiZGZura6OLaXkYN4U/OuwyxVzZLVWPgwE6QMa7VUsNwzSaaiB1QpwQZwG+LoBmO0kGBrpwVHqVi6o= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB5204 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10045 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 phishscore=0 spamscore=0 bulkscore=0 mlxlogscore=999 mlxscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107140115 X-Proofpoint-GUID: 1dw7y1K7D_lxIylc6_vxUfGqloHoJViq X-Proofpoint-ORIG-GUID: 1dw7y1K7D_lxIylc6_vxUfGqloHoJViq Move struct page init to an helper function __init_zone_device_page(). This is in preparation for sharing the storage for / deduplicating compound page metadata. Signed-off-by: Joao Martins Reviewed-by: Dan Williams --- mm/page_alloc.c | 74 +++++++++++++++++++++++++++---------------------- 1 file changed, 41 insertions(+), 33 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 68b5591a69fe..79f3b38afeca 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6557,6 +6557,46 @@ void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone } #ifdef CONFIG_ZONE_DEVICE +static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, + unsigned long zone_idx, int nid, + struct dev_pagemap *pgmap) +{ + + __init_single_page(page, pfn, zone_idx, nid); + + /* + * Mark page reserved as it will need to wait for onlining + * phase for it to be fully associated with a zone. + * + * We can use the non-atomic __set_bit operation for setting + * the flag as we are still initializing the pages. + */ + __SetPageReserved(page); + + /* + * ZONE_DEVICE pages union ->lru with a ->pgmap back pointer + * and zone_device_data. It is a bug if a ZONE_DEVICE page is + * ever freed or placed on a driver-private list. + */ + page->pgmap = pgmap; + page->zone_device_data = NULL; + + /* + * Mark the block movable so that blocks are reserved for + * movable at startup. This will force kernel allocations + * to reserve their blocks rather than leaking throughout + * the address space during boot when many long-lived + * kernel allocations are made. + * + * Please note that MEMINIT_HOTPLUG path doesn't clear memmap + * because this is done early in section_activate() + */ + if (IS_ALIGNED(pfn, pageblock_nr_pages)) { + set_pageblock_migratetype(page, MIGRATE_MOVABLE); + cond_resched(); + } +} + void __ref memmap_init_zone_device(struct zone *zone, unsigned long start_pfn, unsigned long nr_pages, @@ -6585,39 +6625,7 @@ void __ref memmap_init_zone_device(struct zone *zone, for (pfn = start_pfn; pfn < end_pfn; pfn++) { struct page *page = pfn_to_page(pfn); - __init_single_page(page, pfn, zone_idx, nid); - - /* - * Mark page reserved as it will need to wait for onlining - * phase for it to be fully associated with a zone. - * - * We can use the non-atomic __set_bit operation for setting - * the flag as we are still initializing the pages. - */ - __SetPageReserved(page); - - /* - * ZONE_DEVICE pages union ->lru with a ->pgmap back pointer - * and zone_device_data. It is a bug if a ZONE_DEVICE page is - * ever freed or placed on a driver-private list. - */ - page->pgmap = pgmap; - page->zone_device_data = NULL; - - /* - * Mark the block movable so that blocks are reserved for - * movable at startup. This will force kernel allocations - * to reserve their blocks rather than leaking throughout - * the address space during boot when many long-lived - * kernel allocations are made. - * - * Please note that MEMINIT_HOTPLUG path doesn't clear memmap - * because this is done early in section_activate() - */ - if (IS_ALIGNED(pfn, pageblock_nr_pages)) { - set_pageblock_migratetype(page, MIGRATE_MOVABLE); - cond_resched(); - } + __init_zone_device_page(page, pfn, zone_idx, nid, pgmap); } pr_info("%s initialised %lu pages in %ums\n", __func__, From patchwork Wed Jul 14 19:35:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12377731 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3788E17F for ; Wed, 14 Jul 2021 19:36:17 +0000 (UTC) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16EJVdnA009369; Wed, 14 Jul 2021 19:36:11 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=XoxHzHAMM9oJZqN+t6xe6mxgkfsozmJmybUdFrsYolY=; b=Htxd6b20o5qzVG2Ncdmka29c9xD98fim+2D83qUi6wyQM+9gkmkIUL2I31/3W0lTvWvp z4kglxuO/egd+CqIQTqgGKq6FqyJGuIb8FpazgmtYTkKBzA8ZnSB6c3CCvSkEPadVafW 6/eg5X7ktmKTQm6tv6c6hV36V0J5zgrh2lzQaSs7YqxnOYUqrfnDwfqnbWm3/GaLehhb x7dsZ8mFHZIwx6Y4/UidKe6jNAqaurI0LLjh0cNY5Em3BigNIdb5SdEavrZXHC0WfYIx EEJ7Tc3ztf1bwTJ8DvNBgk2LNSd55gd+6QhPSVhrogeiz9Cx1rs5glJZNVNImag2Ys/w kQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=XoxHzHAMM9oJZqN+t6xe6mxgkfsozmJmybUdFrsYolY=; b=CZUzielpMQEiZzcCla6ys3Iw6y9LDhsLAMIizkmU9Y3Agag7HXO+oJM3h2WjrMiHfmqa QR4Zig9zvNAy5zgoEDv8q8BfohK/VhHcJM9lHaWsKDi2h8kjlXUy5RIMbZPT/oeaMvii gk3zxu5KB25rhLZjLQLZ9FhZpvfzq2rpuyg7LHikjBVjoGLY6tOWHlotbM5aN81svCJ8 BsHIBs/1EYobHfiYosWqdNHkd7vlSfYS71YPGvNSys3Ic9OlumIZTK2aMePyKJI3XR3e 06fXyOwTT8QAZ7/04H+NbgJ13AIWcF6QHdQ9OfwUy/Dp+JibUUxjZVjN5j6uyRkn06Pu 3Q== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 39sbtuk7ue-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:11 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16EJUxOI079690; Wed, 14 Jul 2021 19:36:09 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2177.outbound.protection.outlook.com [104.47.55.177]) by userp3020.oracle.com with ESMTP id 39qnb428kk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:09 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=oXonmSdi8usYB0W8Tzp+hQJDQ9JP5S7Kz+mfmktYijuREE18ElqRe/KgguE+KRlLlCGkl1TV7bWH5GBPnnG8V/bub9e6B7tGQ/J2u4o+cSyB/qnqneLnBc9jpSRRfxFzBZvXyB2elr3kON+OXODpnJlcGCTvd9t5qb9ouaI4hcxjjLyBiO55mJOS67eS8HXVAcdwDOUi5ZNYoWZDVJ/zm79DDZwrma9SK8BV8rMilodhnk7quTlvBnrbJ4KyAsOtvhdMrSufrUFuQw2gK/Po530vcbzaJTNwkJ3xU5X708Fl8jutPCoieSId1iv+EstYf/V2qLAVXWWC7x37EHEaUw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XoxHzHAMM9oJZqN+t6xe6mxgkfsozmJmybUdFrsYolY=; b=GhnUUtwHGt91POhXB0WkwZIRYKu7ug96HHehatgvrvqAXEtkv1EipcqpJfESTYCxcZgxN13lxLfzsncsOkPuagE/gh/qHTXv+fWklDU3sVDIQYCB+EjIKYS5DiNLWaIp1ipL4gTovQQn7vDodW+1OeheGScR0NuPg61bXs2MOzlFbxFxtd1Zmk3RNYRySNlfdNSmTFHRhMrrNO/itBRXaD7RyWKk9aLG72e4cytygKZMixXJGfjOnmLfseKlGQ7NIKE0VbMkaaizzeDR9OWT4Ys5uIicgcN9G4GPWlwa6BLgCSyqLZ/m2Jryv08HhxQB9AT+GTNLFmGzAm8xinctIA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XoxHzHAMM9oJZqN+t6xe6mxgkfsozmJmybUdFrsYolY=; b=n/98cJFRvkr7vXDa0Dsv6sGEi3XvE5/QC3KnCQpkOU+E2LDLdTsnMIqXWeuliwo2UfWt4S2+f+fygL1gd7tySvfJm8wC0u0qau+s6ohDth5uPRACONd64rqhpHULAzAN2DjjV89GkG9P/BDRQYGGe6YrCxO3EXcTnNWzIpw2bLc= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by MN2PR10MB4080.namprd10.prod.outlook.com (2603:10b6:208:185::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Wed, 14 Jul 2021 19:36:07 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360%9]) with mapi id 15.20.4331.022; Wed, 14 Jul 2021 19:36:07 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v3 04/14] mm/memremap: add ZONE_DEVICE support for compound pages Date: Wed, 14 Jul 2021 20:35:32 +0100 Message-Id: <20210714193542.21857-5-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210714193542.21857-1-joao.m.martins@oracle.com> References: <20210714193542.21857-1-joao.m.martins@oracle.com> X-ClientProxiedBy: AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4331.21 via Frontend Transport; Wed, 14 Jul 2021 19:36:05 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2fd76329-1c9e-4ec9-b829-08d946fea03a X-MS-TrafficTypeDiagnostic: MN2PR10MB4080: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5236; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: NB0GI8L9bYp2M/UT2zgQwOV3EkVAz0iZShY8kgyWnnJaxa7L5rfelrI+jW/N5469rnxHNP8YKAaMaxONdCXnOHS5TnKxfA9Cb5NXr5/q14/x3qK+oUvGE0ain1fe/hBRvM1eNJC1Eb/hUbvSWy3deBs5vVg2psmOnmmsJXbeyipzAb3PPXJUPjG4ch/dEsACih8qmaO0VeHSyv7z5sRlFeG0oSS6YvB6p1JU9oO5qVpLQy3dmA3E8d09uHM3xvanLo9xkhrEt92AVOFoIsYxo00aDStbYA6ARGKqll0pSPyPecblenZj0ATEaxppb5JV/rfaRdFE1hQIawjx6qVOaP+UvJnxMA5bQaNfu4jVnDotkmjXn81CTjlcPNpliKgchs6Oni93YvXDeME4GaZTcHrnr/qaB3ZxRFDpMZ5PntzJMha5lL6ox6v+SMiXKqEAl2IGPG0GD+jaSoTEng0OpMaK7cNNTc4SVNDeuX8oOxIrDg/RT3MWT91bXD/KHXUiXAPFwvhiIVGuW4fnKSyvl5jU1I5niLJ7PHe3U1Qha5XD/zpw+AbVFX44R6rGr82U/2+SumzVXC0ukFLX5gijcki0zcLyjJDK4Su4FiHyqPpCKEsyXUHyZ97tMjVgIvyJHi2n2F5sD/XePyMQEmQHQbaOjye7l3dQjMtgoEISc7UTqjbXcWEwp7ZsWJ3CDq6DDtqbJvMRjiWOCtKbBCtz6A== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(396003)(346002)(366004)(376002)(39860400002)(136003)(6486002)(8676002)(54906003)(86362001)(5660300002)(36756003)(478600001)(186003)(6916009)(8936002)(4326008)(316002)(83380400001)(52116002)(107886003)(7696005)(66946007)(6666004)(1076003)(2906002)(26005)(38100700002)(66556008)(103116003)(956004)(66476007)(2616005)(38350700002)(7416002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: frx9lq9rMAZ1EEbT4EaZ9uVBb2AJeom6vqZrikauCM7Vd2wGvOX48LZKmUoeOUvVY7rIQIbCIAVjOwWte2a8BRodVMsmkgyNkON/ZGI6wrTnz0TGMGJZker70lqyes23HjSCcq/51UypgeDoN9GusyIam9ACEdwnb5x2r0aXQsUabeGEUosaWsT5H4Pydq2qL6F5dNa9lm32IKLlg3BPNmw8Ced7tTDuJYAHIj9JoL15q8OYp8uyaJb1JYJTRzuBYbVP+rGIHJTe/JnCh23PM6oQBgBvQybaQK0sUyrUIc/ugsVNEyhVoEDQd0+Vx8aBN8DDKbT2Hiw5TFvVgFQ5xeHq3efkW0PBRx/xEzVxot5ZFZBpB35v51kLIGkLk465qgNCaKy6CMgduZLPHNkr97Z+XR+c2a8lRdSKALSwuzpfbRhEBc/1d0tECgvoIeVXjJJrGt3obC+BNkKPPbFeE16oYeg/YtcavpJVzka8EwNjkJ5RfmF28XBHi1AT3qJePke5Q01gHNs5EfMEURRW8403TRqr/3v/0CaDP324KtkR1I1154jvr3FmjJlIRzytHQQstf87Af5c+9i10p/8bNb8L5LK/hjp7hDGc9Qgu0o0NX0J+bq9DK6x+GdpYjqGgMQHFqnBb7IAyIIatQkvCPNFZK0+AZIm4ANuw3WXaJ8nYyr1xH0pn3oZs7IqVBVtTb1uUl4pdvfFKf4in+cLfEWFzU7A2m4RZHDm7Hkrfl8Cs6GDH9LNXFg9PuLezox7Cw3nV480lqcaEB02cfIlm5BazyheH8tB/caeePwXU0Wv8OzsZccutUVRcdgV/L5r03tMU3zJtDSIqlOga8SMmpGTS8Pl4mgMi1U6hbmOKXmwbg1GG/9+CnTfne/+dkevwajoNNI/f14UuQoRTIFo0G0gQ7rgpgTkgCXDjPBKRPERMkNKnM5L0PloDFMD6cEKBgva7b8wE+LfhcJdzSUkb6QGrVs5w6+gTxQbaxLJnQwx61Cu4O2A26+8rZKULZOxTr3rR2wkllZWvGczM9UEmCv1vEbUMgmHk/Vihmcg0TjbTF/yR9mJD2jHQHEcESkXg7qn7GPiyqa6FlJJJw8fef7q+Zeho8pSArlWoRhNSXsw7P6sSMeLJ/AWM2KDCX6DVFM/cDD3PhCaC3GomKjQoDUEq78U1TvTdLHAqUrUXrT32Oa4h5VyVHL2fk3g2bRV6o4u3fLTzhGRK8gd0Xk8KFoTVROpWWxtqXRt3iN3xzAJST7H1VLKzROUXHxXRIfigx05NpSvMugFUm6Fkae6hEw+WCaMGS7EuCC8qrsHImAvFRcMY9zgJfZFez8Wszqg X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2fd76329-1c9e-4ec9-b829-08d946fea03a X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jul 2021 19:36:07.5925 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: VmhNzo3B5o0lIAjD4J5hnYkxxeahuH9psZvVjpcED8ajyMrzw3NM0pn2HblBSvYvFd+J/TQGbcolamKSrC+aLjgbr0+3Lmawa9o86E3JJMI= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR10MB4080 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10045 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 bulkscore=0 malwarescore=0 spamscore=0 suspectscore=0 adultscore=0 mlxlogscore=999 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107140115 X-Proofpoint-GUID: S9BiDl7ICR21jCS_uf07NHZ8H6WSXT6_ X-Proofpoint-ORIG-GUID: S9BiDl7ICR21jCS_uf07NHZ8H6WSXT6_ Add a new align property for struct dev_pagemap which specifies that a pagemap is composed of a set of compound pages of size @align, instead of base pages. When a compound page geometry is requested, all but the first page are initialised as tail pages instead of order-0 pages. For certain ZONE_DEVICE users like device-dax which have a fixed page size, this creates an opportunity to optimize GUP and GUP-fast walkers, treating it the same way as THP or hugetlb pages. Signed-off-by: Joao Martins Reviewed-by: Dan Williams --- include/linux/memremap.h | 17 +++++++++++++++++ mm/memremap.c | 8 ++++++-- mm/page_alloc.c | 34 +++++++++++++++++++++++++++++++++- 3 files changed, 56 insertions(+), 3 deletions(-) diff --git a/include/linux/memremap.h b/include/linux/memremap.h index 119f130ef8f1..e5ab6d4525c1 100644 --- a/include/linux/memremap.h +++ b/include/linux/memremap.h @@ -99,6 +99,10 @@ struct dev_pagemap_ops { * @done: completion for @internal_ref * @type: memory type: see MEMORY_* in memory_hotplug.h * @flags: PGMAP_* flags to specify defailed behavior + * @geometry: structural definition of how the vmemmap metadata is populated. + * A zero or PAGE_SIZE defaults to using base pages as the memmap metadata + * representation. A bigger value but also multiple of PAGE_SIZE will set + * up compound struct pages representative of the requested geometry size. * @ops: method table * @owner: an opaque pointer identifying the entity that manages this * instance. Used by various helpers to make sure that no @@ -114,6 +118,7 @@ struct dev_pagemap { struct completion done; enum memory_type type; unsigned int flags; + unsigned long geometry; const struct dev_pagemap_ops *ops; void *owner; int nr_range; @@ -130,6 +135,18 @@ static inline struct vmem_altmap *pgmap_altmap(struct dev_pagemap *pgmap) return NULL; } +static inline unsigned long pgmap_geometry(struct dev_pagemap *pgmap) +{ + if (!pgmap || !pgmap->geometry) + return PAGE_SIZE; + return pgmap->geometry; +} + +static inline unsigned long pgmap_pfn_geometry(struct dev_pagemap *pgmap) +{ + return PHYS_PFN(pgmap_geometry(pgmap)); +} + #ifdef CONFIG_ZONE_DEVICE bool pfn_zone_device_reserved(unsigned long pfn); void *memremap_pages(struct dev_pagemap *pgmap, int nid); diff --git a/mm/memremap.c b/mm/memremap.c index 805d761740c4..ffcb924eb6a5 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -318,8 +318,12 @@ static int pagemap_range(struct dev_pagemap *pgmap, struct mhp_params *params, memmap_init_zone_device(&NODE_DATA(nid)->node_zones[ZONE_DEVICE], PHYS_PFN(range->start), PHYS_PFN(range_len(range)), pgmap); - percpu_ref_get_many(pgmap->ref, pfn_end(pgmap, range_id) - - pfn_first(pgmap, range_id)); + if (pgmap_geometry(pgmap) > PAGE_SIZE) + percpu_ref_get_many(pgmap->ref, (pfn_end(pgmap, range_id) + - pfn_first(pgmap, range_id)) / pgmap_pfn_geometry(pgmap)); + else + percpu_ref_get_many(pgmap->ref, pfn_end(pgmap, range_id) + - pfn_first(pgmap, range_id)); return 0; err_add_memory: diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 79f3b38afeca..188cb5f8c308 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6597,6 +6597,31 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, } } +static void __ref memmap_init_compound(struct page *page, unsigned long pfn, + unsigned long zone_idx, int nid, + struct dev_pagemap *pgmap, + unsigned long nr_pages) +{ + unsigned int order_align = order_base_2(nr_pages); + unsigned long i; + + __SetPageHead(page); + + for (i = 1; i < nr_pages; i++) { + __init_zone_device_page(page + i, pfn + i, zone_idx, + nid, pgmap); + prep_compound_tail(page, i); + + /* + * The first and second tail pages need to + * initialized first, hence the head page is + * prepared last. + */ + if (i == 2) + prep_compound_head(page, order_align); + } +} + void __ref memmap_init_zone_device(struct zone *zone, unsigned long start_pfn, unsigned long nr_pages, @@ -6605,6 +6630,7 @@ void __ref memmap_init_zone_device(struct zone *zone, unsigned long pfn, end_pfn = start_pfn + nr_pages; struct pglist_data *pgdat = zone->zone_pgdat; struct vmem_altmap *altmap = pgmap_altmap(pgmap); + unsigned int pfns_per_compound = pgmap_pfn_geometry(pgmap); unsigned long zone_idx = zone_idx(zone); unsigned long start = jiffies; int nid = pgdat->node_id; @@ -6622,10 +6648,16 @@ void __ref memmap_init_zone_device(struct zone *zone, nr_pages = end_pfn - start_pfn; } - for (pfn = start_pfn; pfn < end_pfn; pfn++) { + for (pfn = start_pfn; pfn < end_pfn; pfn += pfns_per_compound) { struct page *page = pfn_to_page(pfn); __init_zone_device_page(page, pfn, zone_idx, nid, pgmap); + + if (pfns_per_compound == 1) + continue; + + memmap_init_compound(page, pfn, zone_idx, nid, pgmap, + pfns_per_compound); } pr_info("%s initialised %lu pages in %ums\n", __func__, From patchwork Wed Jul 14 19:35:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12377735 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 98A716D10 for ; Wed, 14 Jul 2021 19:36:20 +0000 (UTC) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16EJVU3Z022310; Wed, 14 Jul 2021 19:36:14 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=9gl7ahGRX4/CQWnfgqY84S2/HDEsgBnAUIfSo+Y+DPw=; b=1H0i8/vxW7KuvnDPeZAGYFrol0UXYauAfWPd8uxfG5LmOWeKEr4AT3QnjxskEvM/36iF /a7qcIWt2XRmCaywF+plYPpnhg+vtdMRHmHKAp+PJgtv8Egv57UKkqqtfVO+Mgx/Vfel FVGy6X3TYUsigru96MJnvE+Wj3l6reYBaUf9MkeZUi0yj8jNSnadf+sjLm3lZdgRr+vh Mur5qxlG1/y2z3A48ZFjC4QqhGSTbGO3zL9MwugP2qGNKm8EILwef4SAB+Xtsfp2sCyu ymliNYq2oZAx0PdmolGy04RGmB+bXo9kg3xwXMrs45J2q/hk6pznvEQ5fS/vlRhE6QYH aQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=9gl7ahGRX4/CQWnfgqY84S2/HDEsgBnAUIfSo+Y+DPw=; b=ylfLmsDO4s4M/gKtjTJQrR8YLNas/SVxU60FmID+5G8cAq1uqPnkVaxkE+V7xNECxG6U MuUtxAjAMys06vtGOGWDFiQyMooaUzH0GvSB5kx4v1NSmFNAQxswGnJXyQVVHpG6bWKc iGK3oW9Jg+RBF8j1zXYuVoa2JrCnKUkAen4CKiaczJFwf6DzLE8iPTOBcF9KpUD8oJqX yWOajPIaazlrRHCKEZxJ4J6i7M/uhSvkKFDxbqDTQfwaV7QU1q7ly16hE+8GsQKlDHWF spLDLr4U1vkbdIlwOsMzHIdjSKfbsMVmYuoET6QLiN1PLChLEh4rsIAvTKwNRG0f3H2q 4A== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by mx0b-00069f02.pphosted.com with ESMTP id 39suk8sejf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:13 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16EJUokv114218; Wed, 14 Jul 2021 19:36:12 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2173.outbound.protection.outlook.com [104.47.55.173]) by aserp3020.oracle.com with ESMTP id 39q3cfyytm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:12 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JuEi6WGxyZXcT3Ec72nxiFfd62EzIB88x1xEBXDfbW7GfjwMoG635Hajq+iAQPVbLQpgaINw95D5AiDbzZGsbWo0DDRo8rsQnKgDEKcflW2wc/QRUyoSTcgpoWtYRvTQqsPMdgqdjNU2NOrxHC7Q0y6iutr5Dk3ByXszsGqvdYfkpwNlgHmzOTzV9+ioW9aMgp8rHmQ0pUksACJ1HnnlgB41ubP1BU2h4VOwNWH+Vlq4ZKkdNrT1uqHf/sq57GcOXWQiY2TLHBlDJh7B+B34Ht2s9/ensAPgPjiRUuAYmXov0kqq88rVPGir4WXUXVefJGJFM0wQdmax7pxZMIYKBg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9gl7ahGRX4/CQWnfgqY84S2/HDEsgBnAUIfSo+Y+DPw=; b=Qs5mMWb8lyaQvsvZcRg+9gP3LXY09y4N7V3ik6tPVPJyJEeubeuoe3P8CfdJYKZcH/enVdU6/r/GQD3MsLgdbHuVHJA4s7eKiZvl2g8jMnbYfS6b869d9u7sruEUwpaXpBRcoinYDWfKnGvRVSgA7g0mUPjvjPbCNkSiZ9roJ05ws3z36x+lEI/TQAJ+KYAjdno0ugIDZs7WZrMnckjnGKiDub1twm1SM8UVTpRC1G0NVokYLX2+BejBdppnDUhO0b+tSvc+7FJvYSb8RslWFLFKGXUNYhDG4XTXwBpJa9qfXCiJto0gm3M7ZlxAYpAVh64tCjOJNwwGAWrTzgBoeQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9gl7ahGRX4/CQWnfgqY84S2/HDEsgBnAUIfSo+Y+DPw=; b=d5qn5U41ba2m/xWIqlkkaQLJPrd9cUOQw4smv+CCU4OTcIxYlOt1DoDG118OVJMp+DGnVmSX12nWZUh3MEKU3Y44X80CubQFCfPezDTVQKOE2n1di1eHQ0Cy4XEoXPx7hsmi7Dcn5vd/xu0ZVYvBngl0gNnSkGQeDEtPkt2Fzok= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by MN2PR10MB4080.namprd10.prod.outlook.com (2603:10b6:208:185::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Wed, 14 Jul 2021 19:36:10 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360%9]) with mapi id 15.20.4331.022; Wed, 14 Jul 2021 19:36:10 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v3 05/14] mm/sparse-vmemmap: add a pgmap argument to section activation Date: Wed, 14 Jul 2021 20:35:33 +0100 Message-Id: <20210714193542.21857-6-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210714193542.21857-1-joao.m.martins@oracle.com> References: <20210714193542.21857-1-joao.m.martins@oracle.com> X-ClientProxiedBy: AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4331.21 via Frontend Transport; Wed, 14 Jul 2021 19:36:07 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 61a46144-e51a-451f-7905-08d946fea221 X-MS-TrafficTypeDiagnostic: MN2PR10MB4080: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3513; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: xmgwBwQjObuPEQdR6B0Kyy3heMPop2zKbMk0MsWE/o+gw7yDDijOW6Qq7OG/+MFgNMFpGYAv133riloTYPsGnJK2Vb8NzwK/ACyPEJVLzOfTr2gdM29AW4mm1gfBeq+vD7B/y/ewAu+6DXNyGXMuobspLu4tszOqDKjVLNVVOZ+hHmIZc6bMIzj3loDnU0ac2hcNfynLL5FgW1uR2/rX4gj3KfMqJs7uQIm1ENBHfhDxVRXmKtTh0PrsDrq1XRTLi6tYglMaRTsqyiovWqytgg6RLotNQqE8S/U6/0d3U6HI3sw6jbZG2OU4TuDW1qjXWcAWkMfSxrazqUZZLLNXtbq93LYbnS+36UyKo8SXspk4tGXZ80s5swSg5eKKINmHTDx+ZZf4oa16Q87elYM0Ruakr6VAEGdZTKm96HfNzpTPQ88aVFOMNFfUlU9vXyIR6/XPrcBGu9FxekfF5TVf0CcjIKB62IYH0B9MDGSTKlYEiTfXUfvUChEGcgU8PMiL/29phkFvBueRWwj46C3IpwtjD0sNT19WYfXdZ0iJq3zM4rJRDD989qXpVOA74K5XNjiQEp9uMtpTItktB0Msir0NF7emz6+qScEh2EijcjHP3HRD1pa4vpj3DoIeyapkwHqePl9qx+SLTJkSE8X3iAx7AtllIbiuOvXklcFdly3fixZomz2e7c6bIHSykGrqiUAyoLw43UmretTkGes0zpenVFm66EQWRTVJ1CObpIquskyUp79QVDobrtIyKlhm3gVN4hSNQcrYq/w/aM84rtB03QdPpw3G/htv5xs/w0w= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(396003)(346002)(366004)(376002)(39860400002)(136003)(6486002)(8676002)(54906003)(86362001)(5660300002)(36756003)(478600001)(186003)(6916009)(966005)(8936002)(4326008)(316002)(83380400001)(52116002)(107886003)(7696005)(66946007)(6666004)(1076003)(2906002)(26005)(38100700002)(66556008)(103116003)(956004)(66476007)(2616005)(38350700002)(7416002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 0Im0LBEvm1yS38qPlYLo8zCfvgOcS5F+xWQdyzdZt80Wn4XovAIfK3PPEA1rk/hNDsrDgyZSTiqdvYd5wYbmUAeAZyjD55isuv9l/3G1ZLfv2ZiBGOdd0UnMh9aFvOwiIjgeY7iuz+SqVI9kQLZqovUNNOp7sg2f4oHIXAGU6Mh+LDV0Bl/bXtsfESkcm8llQ1gPuhmkJLokKHhXfgEsrtknBuv4FPSGhmub9/z3VTxGguYsjaiEU9n4smIFaEBeCiHbpq6vh8g0d9YU74c5FAxQz0RZvevHhCCAlBP9X8qYlvwiQUNQ1Dzjf98SUaMFV59jSf6LN5H+71PT52oK1aHWmAn66PJWXm7TWMmr2nVcaOi2cuwTVSE9iRSJWh6Og9PdmAJ6y0cumghpDKuC5cucRAqwCGt2/8yRsSsX0DwlMiV1ZWs3ca4GE7pOkJAn9sHggGTJqxui5SykRrU+LRg8nDy4j0lmWTPXF004elA3vQ1P6Oxlq/uPfG4cANobsJYZnq4BJsH/kqHUveSoj0PbkX4mZP/Tdet+rZXLFnb2b1RYCD7Okz6lnFBmstQCgjV9kLs+yuxHSVGIURqjp/js1alUHbYkLMFLNUNAxztz1y9xLFuTyJrjL0N9/QPOY5aW/XcNnXoPOIGdi6taVwTnBT0Yk4SqKBlBgEM68Ojf0sKKDL6o59gDJJ5sXcCDMQFNYuPIUQlqomN5ritvZ0xFbmfs36tQOj30SejKS8TTU5Lf89lAyJn2uDqoDGI46N7+X2SMjBZxGwaIodmxWgZqVyODya95/Hljs3BIZIdnf0Wc8XkiIc1lZYRFRj4kAYIG6BSR1fSen60uOcstVayJ0zBlnb5/8cW3l3VG04DaIgf6kpul9/n26Lyj5Qs0CtsPXeZSNPDXj16QHIgs8yBr9/7LCG9Vj4qUB1oRr8KTKYmjG2avDBGeFH7x4vtxj+U9ix3Si9tcLQ/hy5QJ+Xjt09Bb1wqUe2A/MQ3UmtJCHht12fBqqlfOeJs0kZf9ja90iQU/DZehTHk9t2wdGffKji1m4WN26knBRSO9jPB/7UzCB0NM33LShF1EGIjumZqhYpUG+nZxl0fGBO8GOJp+Urw0/r4Az61qiskLcwsdEnZHws77MPxdIu/+3awGO9Oe36lKMeEFvub0MlN0It5ErpH7pNDr+0LVfmrNfFpXW4IvgO1d7qAgQRRqADFUbdJrp6U90aPIRLHo+SfkfErIB5XkD7u1AYwsvyhPNPhhmo8sOTx46tFHY1sS3fBScT10jgF39+SfxwRJLfeZmQ0QerKifpU4auq9vSjOTTVRe7cXP6clBrQ8/8UQWQlb X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 61a46144-e51a-451f-7905-08d946fea221 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jul 2021 19:36:10.7237 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: EqN6kIli+EqIJGV4n3IhZO1tOhhYwbZemXLyVDHrpDK7yIQN+bg1GDIrQCAxGDc/KgLVN6rXLVQEngYETE2nosSLINh6uZEF9NuOWKvV+4k= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR10MB4080 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10045 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 malwarescore=0 mlxscore=0 suspectscore=0 phishscore=0 adultscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107140115 X-Proofpoint-ORIG-GUID: vTU9N2D7tfGNEsuemFJwZ34Wx82aiyen X-Proofpoint-GUID: vTU9N2D7tfGNEsuemFJwZ34Wx82aiyen In support of using compound pages for devmap mappings, plumb the pgmap down to the vmemmap_populate implementation. Note that while altmap is retrievable from pgmap the memory hotplug code passes altmap without pgmap[*], so both need to be independently plumbed. So in addition to @altmap, pass @pgmap to sparse section populate functions namely: sparse_add_section section_activate populate_section_memmap __populate_section_memmap Passing @pgmap allows __populate_section_memmap() to both fetch the geometry in which memmap metadata is created for and also to let sparse-vmemmap fetch pgmap ranges to co-relate to a given section and pick whether to just reuse tail pages from past onlined sections. [*] https://lore.kernel.org/linux-mm/20210319092635.6214-1-osalvador@suse.de/ Signed-off-by: Joao Martins Reviewed-by: Dan Williams --- include/linux/memory_hotplug.h | 5 ++++- include/linux/mm.h | 3 ++- mm/memory_hotplug.c | 3 ++- mm/sparse-vmemmap.c | 3 ++- mm/sparse.c | 24 +++++++++++++++--------- 5 files changed, 25 insertions(+), 13 deletions(-) diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h index a7fd2c3ccb77..9b1bca80224d 100644 --- a/include/linux/memory_hotplug.h +++ b/include/linux/memory_hotplug.h @@ -14,6 +14,7 @@ struct mem_section; struct memory_block; struct resource; struct vmem_altmap; +struct dev_pagemap; #ifdef CONFIG_MEMORY_HOTPLUG struct page *pfn_to_online_page(unsigned long pfn); @@ -60,6 +61,7 @@ typedef int __bitwise mhp_t; struct mhp_params { struct vmem_altmap *altmap; pgprot_t pgprot; + struct dev_pagemap *pgmap; }; bool mhp_range_allowed(u64 start, u64 size, bool need_mapping); @@ -333,7 +335,8 @@ extern void remove_pfn_range_from_zone(struct zone *zone, unsigned long nr_pages); extern bool is_memblock_offlined(struct memory_block *mem); extern int sparse_add_section(int nid, unsigned long pfn, - unsigned long nr_pages, struct vmem_altmap *altmap); + unsigned long nr_pages, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap); extern void sparse_remove_section(struct mem_section *ms, unsigned long pfn, unsigned long nr_pages, unsigned long map_offset, struct vmem_altmap *altmap); diff --git a/include/linux/mm.h b/include/linux/mm.h index 7ca22e6e694a..f244a9219ce4 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3083,7 +3083,8 @@ int vmemmap_remap_alloc(unsigned long start, unsigned long end, void *sparse_buffer_alloc(unsigned long size); struct page * __populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap); + unsigned long nr_pages, int nid, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap); pgd_t *vmemmap_pgd_populate(unsigned long addr, int node); p4d_t *vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node); pud_t *vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node); diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 8cb75b26ea4f..c728a8ff38ad 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -268,7 +268,8 @@ int __ref __add_pages(int nid, unsigned long pfn, unsigned long nr_pages, /* Select all remaining pages up to the next section boundary */ cur_nr_pages = min(end_pfn - pfn, SECTION_ALIGN_UP(pfn + 1) - pfn); - err = sparse_add_section(nid, pfn, cur_nr_pages, altmap); + err = sparse_add_section(nid, pfn, cur_nr_pages, altmap, + params->pgmap); if (err) break; cond_resched(); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index bdce883f9286..80d3ba30d345 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -603,7 +603,8 @@ int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, } struct page * __meminit __populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap) + unsigned long nr_pages, int nid, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { unsigned long start = (unsigned long) pfn_to_page(pfn); unsigned long end = start + nr_pages * sizeof(struct page); diff --git a/mm/sparse.c b/mm/sparse.c index 6326cdf36c4f..5310be6171f1 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -453,7 +453,8 @@ static unsigned long __init section_map_size(void) } struct page __init *__populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap) + unsigned long nr_pages, int nid, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { unsigned long size = section_map_size(); struct page *map = sparse_buffer_alloc(size); @@ -552,7 +553,7 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, break; map = __populate_section_memmap(pfn, PAGES_PER_SECTION, - nid, NULL); + nid, NULL, NULL); if (!map) { pr_err("%s: node[%d] memory map backing failed. Some memory will not be available.", __func__, nid); @@ -657,9 +658,10 @@ void offline_mem_sections(unsigned long start_pfn, unsigned long end_pfn) #ifdef CONFIG_SPARSEMEM_VMEMMAP static struct page * __meminit populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap) + unsigned long nr_pages, int nid, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { - return __populate_section_memmap(pfn, nr_pages, nid, altmap); + return __populate_section_memmap(pfn, nr_pages, nid, altmap, pgmap); } static void depopulate_section_memmap(unsigned long pfn, unsigned long nr_pages, @@ -728,7 +730,8 @@ static int fill_subsection_map(unsigned long pfn, unsigned long nr_pages) } #else struct page * __meminit populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap) + unsigned long nr_pages, int nid, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { return kvmalloc_node(array_size(sizeof(struct page), PAGES_PER_SECTION), GFP_KERNEL, nid); @@ -851,7 +854,8 @@ static void section_deactivate(unsigned long pfn, unsigned long nr_pages, } static struct page * __meminit section_activate(int nid, unsigned long pfn, - unsigned long nr_pages, struct vmem_altmap *altmap) + unsigned long nr_pages, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { struct mem_section *ms = __pfn_to_section(pfn); struct mem_section_usage *usage = NULL; @@ -883,7 +887,7 @@ static struct page * __meminit section_activate(int nid, unsigned long pfn, if (nr_pages < PAGES_PER_SECTION && early_section(ms)) return pfn_to_page(pfn); - memmap = populate_section_memmap(pfn, nr_pages, nid, altmap); + memmap = populate_section_memmap(pfn, nr_pages, nid, altmap, pgmap); if (!memmap) { section_deactivate(pfn, nr_pages, altmap); return ERR_PTR(-ENOMEM); @@ -898,6 +902,7 @@ static struct page * __meminit section_activate(int nid, unsigned long pfn, * @start_pfn: start pfn of the memory range * @nr_pages: number of pfns to add in the section * @altmap: device page map + * @pgmap: device page map object that owns the section * * This is only intended for hotplug. * @@ -911,7 +916,8 @@ static struct page * __meminit section_activate(int nid, unsigned long pfn, * * -ENOMEM - Out of memory. */ int __meminit sparse_add_section(int nid, unsigned long start_pfn, - unsigned long nr_pages, struct vmem_altmap *altmap) + unsigned long nr_pages, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { unsigned long section_nr = pfn_to_section_nr(start_pfn); struct mem_section *ms; @@ -922,7 +928,7 @@ int __meminit sparse_add_section(int nid, unsigned long start_pfn, if (ret < 0) return ret; - memmap = section_activate(nid, start_pfn, nr_pages, altmap); + memmap = section_activate(nid, start_pfn, nr_pages, altmap, pgmap); if (IS_ERR(memmap)) return PTR_ERR(memmap); From patchwork Wed Jul 14 19:35:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12377737 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D74A86D14 for ; Wed, 14 Jul 2021 19:36:21 +0000 (UTC) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16EJVU3b022310; Wed, 14 Jul 2021 19:36:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=+Di/bDAXE43OnWt5MOHanBvl+EkBqkE3LqLc4QmEmeY=; b=wFOywtjPqKULnebJSNbZW2Tus4Rvn5/DS2gHSu33OecJ3M917AmFmwbZ7+bjG0v12kTo 0dgIBHtCIT8b+fIhfvwyHBMPJmE7vYK7htDM3voxKvAo6DdqzTYKqvBuoPZZ8XzXDbiu HRlLrKKitoFLh5Dn+P0uPnMMtSgZuLXYpqF8D6Fj8cXl/+DnYgoUz8vx/Ar/EBX2dP4u 3ynceYltQEwUbHl/eTgVaJ1OfhkcZLdO2lQJpbnsVrKwc1Oa6c6agWZLXg1jdfMuH4S1 PpZI9JvjP6ePWUsEYS6u8vI0XcRo+NjvM+W39sFVUD4gBedCM66ZRJ94jrDqRk9Y6IA3 6g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=+Di/bDAXE43OnWt5MOHanBvl+EkBqkE3LqLc4QmEmeY=; b=tZHvYdHf2NqFuJDB+U+AfkCKm0uGA3hOb7V/7SfwToEgnvojuDoH3tWcnT1G42mR3MRP 8eh/8CFD39c/EipxUclj7Oe2a+0AKKuBIPuy0M1CGsVS1WorkQBE2JXkKNccax4kXCSk JgpMi8RnOupOfdDGajyWzqwB8y9pQ1vJQc9HPkTIhnH/16hU0NFf367wDKdSg48x6jr/ iUgIBjls2Qojo6zfx+8zBK7dv16FbTZxmnOWmj8sp5czOuhpEtNvgMuBucLd8zxSSUZd kghVlDqSf5cBTr91ygM0AUvv/+XWqUacp818akGTA6v9VWxUiIrAOB5mOiTC2J9U0jIk rg== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 39suk8sejh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:16 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16EJV3eE080003; Wed, 14 Jul 2021 19:36:15 GMT Received: from nam02-bn1-obe.outbound.protection.outlook.com (mail-bn1nam07lp2042.outbound.protection.outlook.com [104.47.51.42]) by userp3020.oracle.com with ESMTP id 39qnb428qn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:15 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=c8gq3pMR4XPBVigjP8RWjBKJpCxpy64yQYytxtzdSVg10maoAyyl+wlb5n2rZ1hEdunSofTovR5TgXc0BmH3SZq5o/2Rv8q1DaEwkhz/6LMNEdoz4gwPBn/wu6GouwA1vW6EA7SG+TAHF+o3MbwWh/j7yTVI7riiUBlpRBIHpADXlI4mlJx2WR/KsPNyjd3QZ5BlkPM1vmYkcESqHBiA6rj1wZTsmdIUekbnimbJr+NTwwTKAS7s+Tvb/e6+Axus852v77Hfs8O9ZYqdZhcBCbJ3ilN4f0YJR3+FS23sWZaPhelqPHXKNeo9bxprMjG9BI9RttDcJKhFU++1G/Iu4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+Di/bDAXE43OnWt5MOHanBvl+EkBqkE3LqLc4QmEmeY=; b=dr0IKNuUowF4TwibZa3Sc8yBPXiJ1ujOAn71p/L1V7bvVXt3nedTDkCfRkxdKypeYIZOFKbulDFf/ZvnJHNXl/3gpZJtwPHuxY9RkniK+hYfmDJxDuZHnh7rdcgyrNf87H7GHEVeWXJTlRSGLlnOss388OYex2hMVPCzibGomYCIXbNGf9OWWDq/HdZYOjjhEZ1JowJ7PN/HCd8rwXVmY6IzVKTy6cyZbc2S6FrZdq8H7sjgBlgGOpjLVC9zTov2xmEqO3mq3z2hSHRFvMNe3eCIOT5kgO0BjrA2IfCmdzw5ySda4cOfKDpbA8jeNOUDlL76MqVTDXCAVGcVO8r4xQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+Di/bDAXE43OnWt5MOHanBvl+EkBqkE3LqLc4QmEmeY=; b=qJTDfiRJLmWy0DOM6JZ4ypvwQKHn4kJBscWS1/1DDCa0pgwT5vm2MciTfXc0u6k9TH1pvVb8scVqR5FE+2T6Fl+GX74sIq6clZoYJVOhpNU4c31J+YZ3WQw+osyhBemf6ZnIKNTnGtmipgR8L6IIBOh8YgPGaH/MfJAf6r/bZCA= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BLAPR10MB5204.namprd10.prod.outlook.com (2603:10b6:208:328::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Wed, 14 Jul 2021 19:36:13 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360%9]) with mapi id 15.20.4331.022; Wed, 14 Jul 2021 19:36:13 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v3 06/14] mm/sparse-vmemmap: refactor core of vmemmap_populate_basepages() to helper Date: Wed, 14 Jul 2021 20:35:34 +0100 Message-Id: <20210714193542.21857-7-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210714193542.21857-1-joao.m.martins@oracle.com> References: <20210714193542.21857-1-joao.m.martins@oracle.com> X-ClientProxiedBy: AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4331.21 via Frontend Transport; Wed, 14 Jul 2021 19:36:10 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ebd03e5b-12d7-4cf9-b1e4-08d946fea3b2 X-MS-TrafficTypeDiagnostic: BLAPR10MB5204: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3631; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5a21q8lwVpwxcM9cW28/LOqiq0ibN6egHH2OzwFLzxPNz1enS/Pb9EDa0PeAuR2bsWOkS12rE541Dsjt8i8G9ybUIWbcTkawCas4Qbj38GyVDvw3xExboo+wVQji1lxDwpO12ZMZB8+ZWtoZnqi6xCi7QP1lLCUcgB4SXeMwVeErDbdWbXa7Y/FlNl3LGz3GROsjeCG0MhzRBXbMOmhRAm5tDJmIrOqq8WGHtOz4WNOF7ETo44oaNTTY6h/bMvhCXl0McBXdZWM4IlhdIigEfjfMDKQ33VT4YCdAeUZnrNs4kQGWNW4vRtsL9OpOqP81s8Dv3HfiIQVmlpddUU5c1HrTpDqIpmIH0S1T9rVrXbxI8LwnDFLpj1rd0N/2Rty9jdQOLLxw/0tks0AEk+wNOshx8cdDgr8pOwIMW66Ts185v4tNnUeQMicVrIKmfMETmUFgRPBROJmh63t65tof/KcNPvbqj9eNNbQ31OW22TZQ1j87E/jmpwJXtJULWiFgwYObXuNUkmqDJObp8CXHKkgGq3cKQf8hU0VlHEDDPbxvDnRz9XO69vn2Bn46I+OPDI9qSVLnUWDE5USTR5S0AfDpWBQjPWQmIASCrlSLa4iqqu72sCpR8G16GFdUu1K0vzqPU8pdTuozNxHQt8k6lN0u3wmfWyMfsHRjUOg8jWNt+n9wliEMK5DbmBTp54/p4TGYlF5fZsVWB7eiqVtsqA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(346002)(396003)(366004)(136003)(376002)(8676002)(4326008)(86362001)(6916009)(7416002)(2906002)(956004)(107886003)(2616005)(6486002)(1076003)(8936002)(66476007)(5660300002)(36756003)(103116003)(83380400001)(38350700002)(38100700002)(54906003)(6666004)(66556008)(316002)(66946007)(478600001)(186003)(52116002)(7696005)(26005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: EmojyKe0/nih23QRqbezz57OeFvcW1sE/OS7hzXZ1N/7GNr8YKfHP6rPc1upe6zVZ6Bu1xUWEUjWgZbG4tnOdEO2MoAWhSwsbR6lBBlcaqhDY3IVf9paiT4FHkMocq1Sq9sIHuC5EAqaw76CGUH09PlGR0NzDJsmdbtbQfCe89jZahpcOwRGDuG86rYyGo+d6rH9GhnDB4LiUDExadaTUGPqDiFF48sFT4dRoCir+WfgRO4ipgYVJmHOwm/CpfdtkNgNwXucc0CYkOU3DvVOvaubC2edBmF+ecrM41Axfvw56GscR6J29dYOHqiJk7+i5dlhM9vxuX1ur+WSm1n8yMo2B3ElAWWlk+lpBrGh5A5K23uEOpprhikU32UooLnwuWdiasxuASFgAbxwEIfNsSxvKB1fBCRaHxRTIj+e7gW6Iy2gbNP7t6IlMEW7/Z39atlVCtDzuAaelVODIgRvHUPJvgEaXgbd/btoFbrwkbecRuSqwwKhVJuXCki9THRRus3t7nZRSVph2RBba9GuMkYs0hsAIJv9WxmBwie/Z8FnEh4dyecup2+bWe5URrIUgmqQrTARIQFu09y785L6nke2kFAapUGnB3Q6OBVmEM2rGpBldLSAhiMoDdfQ4Ode4QNjB2pUAfrFhd2vx9rD1GjUtk7M7zny+uqhGF1HmvILoYJs9lyhDDt3Fva9C3OZFHdl9UHMgvwSskQq0Jf/TRJlelHbKvz8ESyZ2+KzIYPxYzqc1qH1ut6LYCehrAF6UJsvJ0+CL9CIb37O4i+gqiH6z94xd5lZRFbFkOJPFiDrsxottPU767GVi2t9q6nHgnqoH9ClUoBkBswOy7xEoQUqDdEevfhnpIdEBRNF/2hsSJM6AS5pDhpR4SDw/BaCmeJCvuIEYdEALY3p3w7/gDxVINvE2Z6KDRHB0Psmb12PsUR5Rl1qWgbW/lpXLFsIcJMDFMnlzbns7iSnQADsaVcgBoOBmndw8Ef1snlnvg6X/dKC493lHLMWzEtgS8n5G3qf8AQQ3cRgkgkSnUvBG9AlX9gx74sbGy+DMoKvJDRl9HwMoir3qEGEJMUryBMciQQthlzv1/5sLaJ0GK7aB9JW5mgmeDpwmPcBVMRML8wn+MsmRZbKad03CU9NsxvaJVy+KoWcBcV4AAzfe8j1QPGr1xB/k3vST9XUXZkwoBBoksbU68qcjVi8aeh757WJYgDqHsNV/SZ5eHQy+dBdK2dhOQo0hOMVvjY76PrrXPWXjrUH0vmR8FcdCCosSs5sKdB8BxKVUFqUDVr+HPlWRLNpYvZubDx1uk/gmECJhNsqaIAWiRkLrUTmGBdNYY9s X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: ebd03e5b-12d7-4cf9-b1e4-08d946fea3b2 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jul 2021 19:36:13.3452 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Qk7nSZm4iv6K/nCYRKGZywN6/4VcmhVkvf8zaEIE4zgiiFE2saJSxpDP8g5KazJajOlrBj+/cknqcoQR4wtTnOz0SThXuElzIEuz/iPJIuk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB5204 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10045 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 bulkscore=0 malwarescore=0 spamscore=0 suspectscore=0 adultscore=0 mlxlogscore=999 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107140115 X-Proofpoint-ORIG-GUID: _Cw1gc2gImGet65DfhBv8es3ivjGbjSZ X-Proofpoint-GUID: _Cw1gc2gImGet65DfhBv8es3ivjGbjSZ In preparation for describing a memmap with compound pages, move the actual pte population logic into a separate function vmemmap_populate_address() and have vmemmap_populate_basepages() walk through all base pages it needs to populate. Signed-off-by: Joao Martins --- mm/sparse-vmemmap.c | 44 ++++++++++++++++++++++++++------------------ 1 file changed, 26 insertions(+), 18 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 80d3ba30d345..76f4158f6301 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -570,33 +570,41 @@ pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) return pgd; } -int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, - int node, struct vmem_altmap *altmap) +static int __meminit vmemmap_populate_address(unsigned long addr, int node, + struct vmem_altmap *altmap) { - unsigned long addr = start; pgd_t *pgd; p4d_t *p4d; pud_t *pud; pmd_t *pmd; pte_t *pte; + pgd = vmemmap_pgd_populate(addr, node); + if (!pgd) + return -ENOMEM; + p4d = vmemmap_p4d_populate(pgd, addr, node); + if (!p4d) + return -ENOMEM; + pud = vmemmap_pud_populate(p4d, addr, node); + if (!pud) + return -ENOMEM; + pmd = vmemmap_pmd_populate(pud, addr, node); + if (!pmd) + return -ENOMEM; + pte = vmemmap_pte_populate(pmd, addr, node, altmap); + if (!pte) + return -ENOMEM; + vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); +} + +int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, + int node, struct vmem_altmap *altmap) +{ + unsigned long addr = start; + for (; addr < end; addr += PAGE_SIZE) { - pgd = vmemmap_pgd_populate(addr, node); - if (!pgd) - return -ENOMEM; - p4d = vmemmap_p4d_populate(pgd, addr, node); - if (!p4d) - return -ENOMEM; - pud = vmemmap_pud_populate(p4d, addr, node); - if (!pud) - return -ENOMEM; - pmd = vmemmap_pmd_populate(pud, addr, node); - if (!pmd) - return -ENOMEM; - pte = vmemmap_pte_populate(pmd, addr, node, altmap); - if (!pte) + if (vmemmap_populate_address(addr, node, altmap)) return -ENOMEM; - vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); } return 0; From patchwork Wed Jul 14 19:35:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12377741 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C5D696D14 for ; Wed, 14 Jul 2021 19:36:26 +0000 (UTC) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16EJVdnC009369; Wed, 14 Jul 2021 19:36:19 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=3HwTnj70uitZefbBY/DNOFlh8e/6/Pc33VEsDK3qRKY=; b=MHPxikysxK8uwug5cl62qMpiO5tQOYoS+EAKW1CQ+VxTGU3AkENBbSPIpczrZ8n3Juk0 kdqph3tAINO+S9afaZ8qg1A3Z6d1bmhZjwoLMUbAcbmJJrL+/Fkmo7pPdwA2bRcV8REI 3HrjijM4SOdsPmG3QwtflGDWyBaEzg5YuHMV+evlxFVQtqUGXIUk/c3BI8eFFwQS4LqB gNvgmapN6gM0S/MNUqorvrZxyZ+m8EQaO4o410g7fSmQXUmVG5XRwGEOiTVceEOPRz34 NrfCbF4rBlMWjR0cFoArCDVdQqwSLD8UDf7PeJR+W3KyFsCQZ6XyIjKNdITeDuvdSzM7 9A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=3HwTnj70uitZefbBY/DNOFlh8e/6/Pc33VEsDK3qRKY=; b=x8psN6KN8oHEhstgaB2eohEByzXpD/pRjgEYYi3rtTQEXHOwPAcTCA3PpOrBmaJoJ8zT fx0g9VGwqCiMbEvRf5gQ1RWGNOpXEssFPZJNQvwWhfFmHuQ16mB65Fc0Q/Pt0jvr3MBF SKDFYCuSxFZwDDOr58uvZLpFWd8BKB/Havys/T4MBdHY3dX3eO6VES0bdnRp43OgLz/x /M9XB2kLBZGzKyhLKGs1B3u9frDbBwQEZG+SLnlDzOUrlLzkLIn/0EprkpAcboHeVzoC k6loilQ+svnqjYCFI3reas1AcdA+7zpZ4PJ8SjvLi8IDEJaIm7xIBM+4JAJNs4Q98qMN PA== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 39sbtuk7uu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:19 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16EJV0gZ187587; Wed, 14 Jul 2021 19:36:18 GMT Received: from nam02-bn1-obe.outbound.protection.outlook.com (mail-bn1nam07lp2043.outbound.protection.outlook.com [104.47.51.43]) by aserp3030.oracle.com with ESMTP id 39qyd1c5qr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:18 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ivD/dQ5HO3suf64CXrjkYf+Q2AWWyWnVAcAeb3BFc3gzotB3Dp2hQurXWs9BJ7wEs2esVo8XLcv+GZkjWwu+hFXjNfYKxsxnA/ompyxbajvUkFGB0qY6dXwCfBnxp+zTMWE834qntXSLQ6fDGHmyvm4nk/eeT42jonx4Xvpl+fIinY5p1ODpaFyNGqJaEdaMUiLo9SzGaUaC3ivhBJ0AX7xRwuY4LENAiPCHEl+91+mDp/pASH23pCLAW9SDC8r72kXKUBOI74vCGVHKYBCXXs2qfP8C3DfeYiUrTPJW+6H05N1OrO4hbEFiwOYbZbwSWMLBItrKhjk0oIy3ueD7Bw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3HwTnj70uitZefbBY/DNOFlh8e/6/Pc33VEsDK3qRKY=; b=PRYo4tzXsD/V1inW3L5CzIbq/Vm7d0/dgOAl/uXR/tjv8obZmtXakjai1uBi5wjT1R6fu6lgOLGx6MwUAD4yBnc1SCBk3o6pTay5/uco0/uBhFuz51vSlqcIZvTJSgoH4u6RpBpz4gvWWNtPGJZp7nbTzXvyNvR2sFcTh9VTY15e7HnaSpMR5ogE80SfSSZ5c61fF94jKHTMEf49plq3uShHLjmq9iQYZwwo/qqGl0ioZfpH+YZJXc0Toy9SY25ELhPS8p85dukcugKoihNMElMjQoDVzLv6pE4IReFqD8qEhooCByRpl1mWBJji59mDfGeWI3afgGS1LfRqJNUlYQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3HwTnj70uitZefbBY/DNOFlh8e/6/Pc33VEsDK3qRKY=; b=e83udu1Sc3/wQQilagWmzP+B1exNAGjt3g6ic3LmwI23NgpT6dIxiWqAi3Y2XfrUgvXyRwk5j/jKukBs5LFfQCr4Gbajct4b67qaCyXudPHsghLZ2couSbwkaivlXu2ltAE9HL3tZQvQd7jDrIBaXljNtUDZ2+sIpEA/9oFBkN8= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BLAPR10MB5204.namprd10.prod.outlook.com (2603:10b6:208:328::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Wed, 14 Jul 2021 19:36:16 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360%9]) with mapi id 15.20.4331.022; Wed, 14 Jul 2021 19:36:16 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v3 07/14] mm/hugetlb_vmemmap: move comment block to Documentation/vm Date: Wed, 14 Jul 2021 20:35:35 +0100 Message-Id: <20210714193542.21857-8-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210714193542.21857-1-joao.m.martins@oracle.com> References: <20210714193542.21857-1-joao.m.martins@oracle.com> X-ClientProxiedBy: AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4331.21 via Frontend Transport; Wed, 14 Jul 2021 19:36:13 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3e8ec1b6-6c07-4bae-cd3e-08d946fea53a X-MS-TrafficTypeDiagnostic: BLAPR10MB5204: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: xc4C3wsZVkjgKE+GRsNY52H/yaDOeCWmfpqvdIYCjWKgWWVTI4QnB5OkTmr//b68eO+2sWXA41/r2QBD2SONj1DVumgKeOLHVGtUrQtU6Y0twc+P/5kXZhYHYTSUyMO+Dverrc0NA4PON/YWY7XSBIU6e1IUQ8sKH+OTjbNrD/U4t6vXzWZ6+TK1Nz50UgnSzHvQo2+PodF5IIJ1pOfc8R+DzDY2Xfj0KJQfSL0R6HjGEYDgIgoMTr1Yzq4RwN8xhLBaAxWtEOcDjDk4YCX2e6ypheTlpKTfIJfXK9EtmXpyCJoce5LKFxsDm5phbdnwvWvHaHKvYwA743gt3QHcim+NL2v4+RGjB+BU+S8xsnENsqWadeYzjFKEdAuVCUkYH0z/eSw11mTx5USo3Jli1zO7WoFP1vVZXnzbTDHybSNzkwZ3DwB3ESjXzMCrvhdwJX7F91TUY0rQbmMmSnf6Uz7k7fbJ12di+O85YcxEcP23Ee2Zj25Ozkrd8DHv2zD64/OKiuPd2JhH99G1Jyw6tbFK8kB90lVlhNeW/fg2nOa0L2Qz4dsKFwFJ9Q68VFpOluVHPfLMroOKrnrvRmwYbZBLmNhlxiTBCJwEVW4EVWi0Czz4Hhh5POK/h7+g2qNPfUKMuChy/eioBtwIM15YnrGGiHfa7Rg+nPj//9xTtyoA4zySSRKa7y2HEHQg0Rk4o9nJraQCdfQz7RqPycVG/g== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(346002)(396003)(366004)(136003)(376002)(8676002)(4326008)(86362001)(6916009)(7416002)(2906002)(956004)(107886003)(2616005)(6486002)(1076003)(8936002)(66476007)(30864003)(5660300002)(36756003)(103116003)(83380400001)(38350700002)(38100700002)(54906003)(6666004)(66556008)(316002)(66946007)(478600001)(186003)(52116002)(7696005)(26005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: +zI0nJExJo8l2rQg0eyfyhfZaLiuYJ65qhL4tP7JaV21NDHsZ0hZGiRqRvSVHVAxlwesJu3VI/r3rrNdv/Ms/XqiJVtqTOx0xtxDOoQhAGuEGV1mD8M63Y1iKbK+lkeiyJFheOkoRXWVf4RkvcaKbj5cm93QXsxuqRhttvtWAWCIT87hti3msWjok1Dke2L60vNNg56ve743b6+egzA/JpU75voeaorqrZ6DstHaC8XBrTu7bxcmRtzOTRk8hAmKV5mNitvJ27nt10rfHdAUYKRSpra/OOxaSxKNaDBpXquIe4laGbepk4SBz0Asb/mEjE7xO7yrzlsWbsDfPBknVVmf68dSUEuWNWrBn3KzqwDbAHyC6/++SejL6UycoVL6AysK5n8fkaRBs+d3Cr9NmRFSDtzXoxPuuqmZBg4nsnKcMgDrISv/ksuVVrmsQ8aftKZaWFiLUSkn2yGIQe3HfRsp3s3ONZXOXoCG9+WSkHk/U5+9I3rBEiQPtUB5pSqbze80mxB3muP/Lkdrvuvz0lHYblwmUtLRI/2eR1ZL1VNlisC21O3f8p/X9X4INe3Y54Nz8dBj+pOF+5zhx/yga2Ec2kx6YtUIw7zx+b4ic1ok96Ud7QfA+kfXN8hx6FQ/GQu5WWgm3JJ71KmyJ6ikRexiIc5QunUfA7hnz0+3SGgW5GhIjIBaHmov1KupTjyszkVsSuuT6+F17msmi06mEYqc0Lh96gkl5k7fLO02TcbzSPoOiI+2P2woMCkhrSi0MrO7U7Cj2u3WvEwcaHAQ5wCAwbFrelC/0qguFMJ3iyQhjLUhXRhZ0XMwWeWKeQwwDVwDwVVbcL0KOnVjKYVUzqI7UXi43XqymvtPKBfSIka4eob8DLXkONVEfKyoNmNWYAFVSqJMIJKL8YuLA4hm0u7kyOS7BaLdbWmxV5RAg8AtutHf/P5JAk0t6/to46InyIVzcL/EvHGimBTQYVb5UTsSspFVmriRc1l5tkjvFR5m5Lme7jaWwx52tjGbWDsTSFvwZyHfF3D0hIQXGVvflHjwbWHKQFxYZL7uJmaU992axisNdCYiZlfIrRMbBrrjyZvTK3vJLf+gffnZdUypfUhDRgHNX2oXksCpJLODaOgf7jNnFWsSPkFF9nJCMXfUsrZJ0nq5dlL5ENUg7zF4lELYGhnbsoF8VVPMfD6ciu/LjqZBbqZPqKgQWhZJNXblFTBfEMi9y5zBE1CaPdilrc4mxc3SYaq08um/sV3lL2Vq93TOmzDoWa5vLfcNdikNbqrU1561POlH8CFZG78bvqLcz16T1Bw1MviORTDAXjPbvfs9iMVlFwUgQ8ISM/Qp X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3e8ec1b6-6c07-4bae-cd3e-08d946fea53a X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jul 2021 19:36:15.9796 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: enWOJHummvgtIqaL0hpYLBQ5AXIcxbLhu0mKDc4MEc+w/qfl5QZIx8rYGRuqa4IV/dnxtsI8rEC4yMSABuVDQXZzws5vqoEKXSydAS1qSbY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB5204 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10045 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=999 adultscore=0 malwarescore=0 bulkscore=0 mlxscore=0 suspectscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107140115 X-Proofpoint-GUID: 5kWZcSiPQLkGrfwSDdhsCHqo0cLD73f6 X-Proofpoint-ORIG-GUID: 5kWZcSiPQLkGrfwSDdhsCHqo0cLD73f6 In preparation for device-dax for using hugetlbfs compound page tail deduplication technique, move the comment block explanation into a common place in Documentation/vm. Cc: Muchun Song Cc: Mike Kravetz Suggested-by: Dan Williams Signed-off-by: Joao Martins Reviewed-by: Muchun Song Reviewed-by: Dan Williams --- Documentation/vm/index.rst | 1 + Documentation/vm/vmemmap_dedup.rst | 170 +++++++++++++++++++++++++++++ mm/hugetlb_vmemmap.c | 162 +-------------------------- 3 files changed, 172 insertions(+), 161 deletions(-) create mode 100644 Documentation/vm/vmemmap_dedup.rst diff --git a/Documentation/vm/index.rst b/Documentation/vm/index.rst index eff5fbd492d0..edd690afd890 100644 --- a/Documentation/vm/index.rst +++ b/Documentation/vm/index.rst @@ -51,5 +51,6 @@ descriptions of data structures and algorithms. split_page_table_lock transhuge unevictable-lru + vmemmap_dedup z3fold zsmalloc diff --git a/Documentation/vm/vmemmap_dedup.rst b/Documentation/vm/vmemmap_dedup.rst new file mode 100644 index 000000000000..215ae2ef3bce --- /dev/null +++ b/Documentation/vm/vmemmap_dedup.rst @@ -0,0 +1,170 @@ +.. SPDX-License-Identifier: GPL-2.0 + +.. _vmemmap_dedup: + +================================== +Free some vmemmap pages of HugeTLB +================================== + +The struct page structures (page structs) are used to describe a physical +page frame. By default, there is a one-to-one mapping from a page frame to +it's corresponding page struct. + +HugeTLB pages consist of multiple base page size pages and is supported by +many architectures. See hugetlbpage.rst in the Documentation directory for +more details. On the x86-64 architecture, HugeTLB pages of size 2MB and 1GB +are currently supported. Since the base page size on x86 is 4KB, a 2MB +HugeTLB page consists of 512 base pages and a 1GB HugeTLB page consists of +4096 base pages. For each base page, there is a corresponding page struct. + +Within the HugeTLB subsystem, only the first 4 page structs are used to +contain unique information about a HugeTLB page. __NR_USED_SUBPAGE provides +this upper limit. The only 'useful' information in the remaining page structs +is the compound_head field, and this field is the same for all tail pages. + +By removing redundant page structs for HugeTLB pages, memory can be returned +to the buddy allocator for other uses. + +Different architectures support different HugeTLB pages. For example, the +following table is the HugeTLB page size supported by x86 and arm64 +architectures. Because arm64 supports 4k, 16k, and 64k base pages and +supports contiguous entries, so it supports many kinds of sizes of HugeTLB +page. + ++--------------+-----------+-----------------------------------------------+ +| Architecture | Page Size | HugeTLB Page Size | ++--------------+-----------+-----------+-----------+-----------+-----------+ +| x86-64 | 4KB | 2MB | 1GB | | | ++--------------+-----------+-----------+-----------+-----------+-----------+ +| | 4KB | 64KB | 2MB | 32MB | 1GB | +| +-----------+-----------+-----------+-----------+-----------+ +| arm64 | 16KB | 2MB | 32MB | 1GB | | +| +-----------+-----------+-----------+-----------+-----------+ +| | 64KB | 2MB | 512MB | 16GB | | ++--------------+-----------+-----------+-----------+-----------+-----------+ + +When the system boot up, every HugeTLB page has more than one struct page +structs which size is (unit: pages): + + struct_size = HugeTLB_Size / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE + +Where HugeTLB_Size is the size of the HugeTLB page. We know that the size +of the HugeTLB page is always n times PAGE_SIZE. So we can get the following +relationship. + + HugeTLB_Size = n * PAGE_SIZE + +Then, + + struct_size = n * PAGE_SIZE / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE + = n * sizeof(struct page) / PAGE_SIZE + +We can use huge mapping at the pud/pmd level for the HugeTLB page. + +For the HugeTLB page of the pmd level mapping, then + + struct_size = n * sizeof(struct page) / PAGE_SIZE + = PAGE_SIZE / sizeof(pte_t) * sizeof(struct page) / PAGE_SIZE + = sizeof(struct page) / sizeof(pte_t) + = 64 / 8 + = 8 (pages) + +Where n is how many pte entries which one page can contains. So the value of +n is (PAGE_SIZE / sizeof(pte_t)). + +This optimization only supports 64-bit system, so the value of sizeof(pte_t) +is 8. And this optimization also applicable only when the size of struct page +is a power of two. In most cases, the size of struct page is 64 bytes (e.g. +x86-64 and arm64). So if we use pmd level mapping for a HugeTLB page, the +size of struct page structs of it is 8 page frames which size depends on the +size of the base page. + +For the HugeTLB page of the pud level mapping, then + + struct_size = PAGE_SIZE / sizeof(pmd_t) * struct_size(pmd) + = PAGE_SIZE / 8 * 8 (pages) + = PAGE_SIZE (pages) + +Where the struct_size(pmd) is the size of the struct page structs of a +HugeTLB page of the pmd level mapping. + +E.g.: A 2MB HugeTLB page on x86_64 consists in 8 page frames while 1GB +HugeTLB page consists in 4096. + +Next, we take the pmd level mapping of the HugeTLB page as an example to +show the internal implementation of this optimization. There are 8 pages +struct page structs associated with a HugeTLB page which is pmd mapped. + +Here is how things look before optimization. + + HugeTLB struct pages(8 pages) page frame(8 pages) + +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ + | | | 0 | -------------> | 0 | + | | +-----------+ +-----------+ + | | | 1 | -------------> | 1 | + | | +-----------+ +-----------+ + | | | 2 | -------------> | 2 | + | | +-----------+ +-----------+ + | | | 3 | -------------> | 3 | + | | +-----------+ +-----------+ + | | | 4 | -------------> | 4 | + | PMD | +-----------+ +-----------+ + | level | | 5 | -------------> | 5 | + | mapping | +-----------+ +-----------+ + | | | 6 | -------------> | 6 | + | | +-----------+ +-----------+ + | | | 7 | -------------> | 7 | + | | +-----------+ +-----------+ + | | + | | + | | + +-----------+ + +The value of page->compound_head is the same for all tail pages. The first +page of page structs (page 0) associated with the HugeTLB page contains the 4 +page structs necessary to describe the HugeTLB. The only use of the remaining +pages of page structs (page 1 to page 7) is to point to page->compound_head. +Therefore, we can remap pages 2 to 7 to page 1. Only 2 pages of page structs +will be used for each HugeTLB page. This will allow us to free the remaining +6 pages to the buddy allocator. + +Here is how things look after remapping. + + HugeTLB struct pages(8 pages) page frame(8 pages) + +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ + | | | 0 | -------------> | 0 | + | | +-----------+ +-----------+ + | | | 1 | -------------> | 1 | + | | +-----------+ +-----------+ + | | | 2 | ----------------^ ^ ^ ^ ^ ^ + | | +-----------+ | | | | | + | | | 3 | ------------------+ | | | | + | | +-----------+ | | | | + | | | 4 | --------------------+ | | | + | PMD | +-----------+ | | | + | level | | 5 | ----------------------+ | | + | mapping | +-----------+ | | + | | | 6 | ------------------------+ | + | | +-----------+ | + | | | 7 | --------------------------+ + | | +-----------+ + | | + | | + | | + +-----------+ + +When a HugeTLB is freed to the buddy system, we should allocate 6 pages for +vmemmap pages and restore the previous mapping relationship. + +For the HugeTLB page of the pud level mapping. It is similar to the former. +We also can use this approach to free (PAGE_SIZE - 2) vmemmap pages. + +Apart from the HugeTLB page of the pmd/pud level mapping, some architectures +(e.g. aarch64) provides a contiguous bit in the translation table entries +that hints to the MMU to indicate that it is one of a contiguous set of +entries that can be cached in a single TLB entry. + +The contiguous bit is used to increase the mapping size at the pmd and pte +(last) level. So this type of HugeTLB page can be optimized only when its +size of the struct page structs is greater than 2 pages. + diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index c540c21e26f5..e2994e50ddee 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -6,167 +6,7 @@ * * Author: Muchun Song * - * The struct page structures (page structs) are used to describe a physical - * page frame. By default, there is a one-to-one mapping from a page frame to - * it's corresponding page struct. - * - * HugeTLB pages consist of multiple base page size pages and is supported by - * many architectures. See hugetlbpage.rst in the Documentation directory for - * more details. On the x86-64 architecture, HugeTLB pages of size 2MB and 1GB - * are currently supported. Since the base page size on x86 is 4KB, a 2MB - * HugeTLB page consists of 512 base pages and a 1GB HugeTLB page consists of - * 4096 base pages. For each base page, there is a corresponding page struct. - * - * Within the HugeTLB subsystem, only the first 4 page structs are used to - * contain unique information about a HugeTLB page. __NR_USED_SUBPAGE provides - * this upper limit. The only 'useful' information in the remaining page structs - * is the compound_head field, and this field is the same for all tail pages. - * - * By removing redundant page structs for HugeTLB pages, memory can be returned - * to the buddy allocator for other uses. - * - * Different architectures support different HugeTLB pages. For example, the - * following table is the HugeTLB page size supported by x86 and arm64 - * architectures. Because arm64 supports 4k, 16k, and 64k base pages and - * supports contiguous entries, so it supports many kinds of sizes of HugeTLB - * page. - * - * +--------------+-----------+-----------------------------------------------+ - * | Architecture | Page Size | HugeTLB Page Size | - * +--------------+-----------+-----------+-----------+-----------+-----------+ - * | x86-64 | 4KB | 2MB | 1GB | | | - * +--------------+-----------+-----------+-----------+-----------+-----------+ - * | | 4KB | 64KB | 2MB | 32MB | 1GB | - * | +-----------+-----------+-----------+-----------+-----------+ - * | arm64 | 16KB | 2MB | 32MB | 1GB | | - * | +-----------+-----------+-----------+-----------+-----------+ - * | | 64KB | 2MB | 512MB | 16GB | | - * +--------------+-----------+-----------+-----------+-----------+-----------+ - * - * When the system boot up, every HugeTLB page has more than one struct page - * structs which size is (unit: pages): - * - * struct_size = HugeTLB_Size / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE - * - * Where HugeTLB_Size is the size of the HugeTLB page. We know that the size - * of the HugeTLB page is always n times PAGE_SIZE. So we can get the following - * relationship. - * - * HugeTLB_Size = n * PAGE_SIZE - * - * Then, - * - * struct_size = n * PAGE_SIZE / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE - * = n * sizeof(struct page) / PAGE_SIZE - * - * We can use huge mapping at the pud/pmd level for the HugeTLB page. - * - * For the HugeTLB page of the pmd level mapping, then - * - * struct_size = n * sizeof(struct page) / PAGE_SIZE - * = PAGE_SIZE / sizeof(pte_t) * sizeof(struct page) / PAGE_SIZE - * = sizeof(struct page) / sizeof(pte_t) - * = 64 / 8 - * = 8 (pages) - * - * Where n is how many pte entries which one page can contains. So the value of - * n is (PAGE_SIZE / sizeof(pte_t)). - * - * This optimization only supports 64-bit system, so the value of sizeof(pte_t) - * is 8. And this optimization also applicable only when the size of struct page - * is a power of two. In most cases, the size of struct page is 64 bytes (e.g. - * x86-64 and arm64). So if we use pmd level mapping for a HugeTLB page, the - * size of struct page structs of it is 8 page frames which size depends on the - * size of the base page. - * - * For the HugeTLB page of the pud level mapping, then - * - * struct_size = PAGE_SIZE / sizeof(pmd_t) * struct_size(pmd) - * = PAGE_SIZE / 8 * 8 (pages) - * = PAGE_SIZE (pages) - * - * Where the struct_size(pmd) is the size of the struct page structs of a - * HugeTLB page of the pmd level mapping. - * - * E.g.: A 2MB HugeTLB page on x86_64 consists in 8 page frames while 1GB - * HugeTLB page consists in 4096. - * - * Next, we take the pmd level mapping of the HugeTLB page as an example to - * show the internal implementation of this optimization. There are 8 pages - * struct page structs associated with a HugeTLB page which is pmd mapped. - * - * Here is how things look before optimization. - * - * HugeTLB struct pages(8 pages) page frame(8 pages) - * +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ - * | | | 0 | -------------> | 0 | - * | | +-----------+ +-----------+ - * | | | 1 | -------------> | 1 | - * | | +-----------+ +-----------+ - * | | | 2 | -------------> | 2 | - * | | +-----------+ +-----------+ - * | | | 3 | -------------> | 3 | - * | | +-----------+ +-----------+ - * | | | 4 | -------------> | 4 | - * | PMD | +-----------+ +-----------+ - * | level | | 5 | -------------> | 5 | - * | mapping | +-----------+ +-----------+ - * | | | 6 | -------------> | 6 | - * | | +-----------+ +-----------+ - * | | | 7 | -------------> | 7 | - * | | +-----------+ +-----------+ - * | | - * | | - * | | - * +-----------+ - * - * The value of page->compound_head is the same for all tail pages. The first - * page of page structs (page 0) associated with the HugeTLB page contains the 4 - * page structs necessary to describe the HugeTLB. The only use of the remaining - * pages of page structs (page 1 to page 7) is to point to page->compound_head. - * Therefore, we can remap pages 2 to 7 to page 1. Only 2 pages of page structs - * will be used for each HugeTLB page. This will allow us to free the remaining - * 6 pages to the buddy allocator. - * - * Here is how things look after remapping. - * - * HugeTLB struct pages(8 pages) page frame(8 pages) - * +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ - * | | | 0 | -------------> | 0 | - * | | +-----------+ +-----------+ - * | | | 1 | -------------> | 1 | - * | | +-----------+ +-----------+ - * | | | 2 | ----------------^ ^ ^ ^ ^ ^ - * | | +-----------+ | | | | | - * | | | 3 | ------------------+ | | | | - * | | +-----------+ | | | | - * | | | 4 | --------------------+ | | | - * | PMD | +-----------+ | | | - * | level | | 5 | ----------------------+ | | - * | mapping | +-----------+ | | - * | | | 6 | ------------------------+ | - * | | +-----------+ | - * | | | 7 | --------------------------+ - * | | +-----------+ - * | | - * | | - * | | - * +-----------+ - * - * When a HugeTLB is freed to the buddy system, we should allocate 6 pages for - * vmemmap pages and restore the previous mapping relationship. - * - * For the HugeTLB page of the pud level mapping. It is similar to the former. - * We also can use this approach to free (PAGE_SIZE - 2) vmemmap pages. - * - * Apart from the HugeTLB page of the pmd/pud level mapping, some architectures - * (e.g. aarch64) provides a contiguous bit in the translation table entries - * that hints to the MMU to indicate that it is one of a contiguous set of - * entries that can be cached in a single TLB entry. - * - * The contiguous bit is used to increase the mapping size at the pmd and pte - * (last) level. So this type of HugeTLB page can be optimized only when its - * size of the struct page structs is greater than 2 pages. + * See Documentation/vm/vmemmap_dedup.rst */ #define pr_fmt(fmt) "HugeTLB: " fmt From patchwork Wed Jul 14 19:35:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12377743 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3CAA16D34 for ; Wed, 14 Jul 2021 19:36:29 +0000 (UTC) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16EJVdnD009369; Wed, 14 Jul 2021 19:36:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=0Pvcm+J+3I7j6BU9TtanyoN0Fmn8A133iSAWo+cf7Sk=; b=UDPRvio0pcN0G4drwXjniPMW29+4T6bo6fzs8HJncQgqR2dO2FVkaQaa5oe9y/khU+qv NaXtad/Rgqlfb8/hjaQfHS0In2ZswnAxSp8+9aW3ObAHhvYU3Ucpwi8BWoIfSCmz6YG0 o7j+TdzaJTOOPrY9v8qCBW1uWxXeidkOm1BRr94qHJQ5TzH5TKaIsMy3JQ5phYqQyqmM OaFahTIh5gI0wNSJgaTA0wcnDGCTFSO9lfB3wWyECdZ6ba2yKye2H7Nxpls0BR9WsKz/ Hu9cRwsC/iqXj+vn+EeCeY0gVUmr+1mLc8vryLlIN1N2GV5JeyRgmc2XCMjJikdrWgH3 ZA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=0Pvcm+J+3I7j6BU9TtanyoN0Fmn8A133iSAWo+cf7Sk=; b=sJfM91boMvQMi9txAMTfMBt9SNbPn3T+dN94FAZhJE5flIy+5Z2BnSyEmDv/tYFhiMFP 1ANr9q1WFdjmTqvM3eZsDsYVe44KjvMEo6GNkwYYBTQ5i5W9vfJHGdKB/p/CsheFHjsX oaQO6HKPtoSK46guDC0DQo2wmjATlMtFthCdXeRcghi0tbCNQIIZ1VqjTebesKKB90sl UX5FlWqHSERQ2WZbGKGoFLSA29JTYpfl6kbKuaaJcS554pP/lG1S1imYyxV0NJA8rxPD kEddzZf78DAbhp3X/+QCTFOK0J/U0EJqz022odK7L5m+eZ8dituZiB/wWMn7Hr4yZrf9 zw== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by mx0b-00069f02.pphosted.com with ESMTP id 39sbtuk7uw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:22 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16EJUnWR114139; Wed, 14 Jul 2021 19:36:21 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2171.outbound.protection.outlook.com [104.47.55.171]) by aserp3020.oracle.com with ESMTP id 39q3cfyyyt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:20 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SOKi8G9Ab4nmg1UW2LLUajBBzOq5A5/Of79/EC0ssXyUKfVnocLfp6CrVouO69lC+HGRJC+RHapub0HVXgQqT6zI5bPG4ZFhDA0rXby1rtBkUxiLvuhTcn2ejPq0UFiLWjdW9rZxf0bUjP8ELTWe1RjlJlYYcdSuMfjOy3Wb+5eH0DTnw5flpqqos6KoHenY/MT4INh3o83BMcrnturFJBqn1/hqUtLMBWnOZYlDUi3ESjCPXql5XwaaOm5uDyUDcFTONAGLQMfNF5eAvp1jQd313oCURonsCL5CjEfNl8TZGSH1E/NjAa5Hh9bZIm/9NMw3Vj+QPNA8z7decp3NGA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0Pvcm+J+3I7j6BU9TtanyoN0Fmn8A133iSAWo+cf7Sk=; b=nY/g6XrSFTjH69NcEQuBqZltg8ocA1s457GU5+/qY86WNxrlmOYZQZvQOOA7SFin2FuK1AaD3/EeyUxoAv/mxnuq+jZOxkB930PTtlp92bX+GMoevoqTSXDYyYVpt6yH95pJGWiKZ/fQPQo8fPMs/NQUXnEZ1KW6yAk8EoqQ8ELvyjozGN5FXqAqW4p804r+z5odRNRKlJlvYcS+sTMGOZycj9E3ujXXHdTERlfYgkDVT5vMpXW68nCpvvSVTQXAj+yLFFeerdLlbZzEq6gtboICwlwoEXuOE3tLwrTm5g3IvRx63EegmhrswxYJxA3QaMWZVXzfkg+rcq0xS7oNpA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0Pvcm+J+3I7j6BU9TtanyoN0Fmn8A133iSAWo+cf7Sk=; b=jVc1CoPFGv07XfjexZ4XA2uwtnAdVZJnvFYbECB5O9iELgQkYbGgA+IY0318JMGORlJW+39nh7p2R0mzc5elZobS7Y9u6MwQPt+pXzlK/15xZ/g970FhgnZMo28uQeXbC+2YaTvngyZRfCcU+AbCWB/hiEoETpkfXQvPzRwZ1i8= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BL0PR10MB3028.namprd10.prod.outlook.com (2603:10b6:208:77::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4308.24; Wed, 14 Jul 2021 19:36:18 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360%9]) with mapi id 15.20.4331.022; Wed, 14 Jul 2021 19:36:18 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v3 08/14] mm/sparse-vmemmap: populate compound pagemaps Date: Wed, 14 Jul 2021 20:35:36 +0100 Message-Id: <20210714193542.21857-9-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210714193542.21857-1-joao.m.martins@oracle.com> References: <20210714193542.21857-1-joao.m.martins@oracle.com> X-ClientProxiedBy: AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4331.21 via Frontend Transport; Wed, 14 Jul 2021 19:36:16 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: aa5be1c1-ed19-4c23-038d-08d946fea6df X-MS-TrafficTypeDiagnostic: BL0PR10MB3028: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: tKk1EsqBiwuNozwXy3RxnQvKYQBJgiZv0kUNv5bRo4sLBzx38ASuXI3/DRLT8earoJ14CZ6gFt/oSihObsC/0ELFu/6V1eLDuxA1o598nNZez9vk8a3IMveSpnaITd7agFZY1PlUdBXZzTj97itKGOQ38Kr2B4ErCzkDEKE1GAXvNvWb5xbFM65H8fId/EGl8nlpC5jhSOWqLUNpIHC43qpciFrx6o8u8mM1Em3RwV6EvPkeBfmkpJ9OJ6M5xa8xVOsJcbIUFbVgNeQefbr9N6OXt/2aF/M03/oDc/2KEZuEmgdZW9GlXgvfSlisDTOet3uKhyQ4UgE7cyUcBk6T7kWyIRBN7hjQLt3DB6t6QnVFbBpWi+fEcRpNm3i787FXPTE+dy6wZJSKSRTvp20gKkP/9wNusuDqRfostCjecN/DtJWolT51EMvjWEwfvY2dUN3dJ422Ka1cajXfH1BqQB8PVkizrSpHANdqsg0KLOZO81JGhGZOMTjLQTIZGL0LT9ec2g/8WBBZkXp6oARdL032wxLTCZDhnSxhjUp5ckweinGZYpop3PxxdABJJO2fBcqb36wrKn9oiJWoN+diwt710vInGLFM8MjX0c5CoGu6Me/rSYbJOfOFXc0agQjOOqz+wBdn8SWNXTMtn4gErkOKeu219rA1ptnwc1r3LatnYoj5UeHyz7M86mJEqLx1WQt1Uc/3TwcL5+tzjs43DA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(396003)(376002)(39860400002)(136003)(366004)(316002)(86362001)(83380400001)(54906003)(4326008)(5660300002)(8676002)(8936002)(956004)(1076003)(6916009)(2906002)(2616005)(107886003)(36756003)(38100700002)(66556008)(7416002)(38350700002)(26005)(478600001)(103116003)(66946007)(6666004)(66476007)(186003)(6486002)(52116002)(7696005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 0EkzP1tGnfz18yHK7MklY4KFqIQU4DMRpwuMKcssWJHJtkCh+sZxzsGn5EmAWjjV4Kv2ncEPs2GjO3e5EWc5/rI5lRQ1jBZGk0dDYXTPbxu81cQ3aC1QAYDU6MfExtJ+chczZI21zCLXdr96dkzlGMOCGoSieobq395R6HDTRZA7CZzC7ZSMBwIaKTNDaXxjEcNHqmbAIm4LppmJ/sIEtdrqInVzWBp6t62AtxTX2JJ8d1xEhwu9laDUTqjYCr24pfOBWIEDfURixGEIZOMaAygBPKF4Yxc57OVHzmtnKdJ1isYpfCcnQJmpFul8Pc0TnRGAM2BGsj2yl0qwqRa8BmsmfXzUJOgRcMbCs49HeYmfuBGrYRk5BMivIWs7s+O9qNT4/XfZak55yxEYOKx5urREZsl7dLQ9JaXwSBTu5U4OFkX20ItvkNXWtK/KTqVvnd+BYuDQAEjbNB7hTZeTzL6GhL9r2sw+k4KSgdN1qEXCsvACYuq3Yo/bzSqJ4NqIGnmjvZ2R9IDQ0mb9tg830gzuV+oY25/CXIqBwTIHIbRayolCRCxLpG5skWvE4OjovP07lVBu18CNQ4caNq7xS7d9l7icg1bnKtjLi2tC0Q3jPV4w2IkISH6NecUJgUhfpzq0m96zGhN/iQ/Ml+B7IV2uvZTD8j2Hes9VKKyxzxaWpjNbLzHH/l5ywPpQE3gIeESljG/mj1mWvtm7Xpi66kisaRXXS78FV7i/CbGaXAz6e2BS5GDW16PDXOPzcExx1BRQWwQALpjvU2zmsU9S2j7H+KRIyZaR3InuEiTfViJVPnm2kPHR5OnrtBewkNeduMEmf31jAPyRjygm4Owye0IQO1+W+GwvwAPbsdxve8hid9MErkglfaQLX5fuueOLeWHN4D/dfPpkqE4jwvnV6Dkh+QKZBHArBB6IxlCqyu1KOhDXxS07SZ/NU+XqmDqmN2lUO+OR2/NSy7l2SXqE/Z+3nENgKk/nhbsl9g7bElLsSqMhLnKHDIoF5htZStkmkHVtx3W8ez/atFPGF2Bloy+Cc6jzbvhsW467JhJ+vxxYKhDy0r3Io2A2hrAGWdaS12fr/sJNxrYJGwekqgJkEFM4NQNq+MUTsxbojo3EsI8hFV5FUHuOunx8ak/X5fccctxheaMxBwHWcbrp26C1XN8CNNaRGIdnuN5uqIa7ts6vdrNpVxG8PPQ/bw5mbQAnSJqLSynC9gcJJxzF71owcnuzw2XifcXIPXxi8sqn+DsQT3OimfUG3JdbqijKPh3dXvlp/mH4seAM44qRzaSo/xA8REPPwvTIA8MofuppukEOacElxHM0jf2l4sjb1/sR X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: aa5be1c1-ed19-4c23-038d-08d946fea6df X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jul 2021 19:36:18.7784 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 3v3xu1XFaBoUEa0NGsRC0uLtfr3XyN6EEa9TYu453ItJHyIPr1XNd0y1KVnXt6F/ubimXuUo7mBCm188I79t/mUTx3PwcBBf8WnCtIh110g= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR10MB3028 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10045 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 malwarescore=0 mlxscore=0 suspectscore=0 phishscore=0 adultscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107140115 X-Proofpoint-GUID: ioyksikqfgUczpMncODgLRGMpnRNdL39 X-Proofpoint-ORIG-GUID: ioyksikqfgUczpMncODgLRGMpnRNdL39 A compound pagemap is a dev_pagemap with @align > PAGE_SIZE and it means that pages are mapped at a given huge page alignment and utilize uses compound pages as opposed to order-0 pages. Take advantage of the fact that most tail pages look the same (except the first two) to minimize struct page overhead. Allocate a separate page for the vmemmap area which contains the head page and separate for the next 64 pages. The rest of the subsections then reuse this tail vmemmap page to initialize the rest of the tail pages. Sections are arch-dependent (e.g. on x86 it's 64M, 128M or 512M) and when initializing compound pagemap with big enough @align (e.g. 1G PUD) it will cross various sections. To be able to reuse tail pages across sections belonging to the same gigantic page, fetch the @range being mapped (nr_ranges + 1). If the section being mapped is not offset 0 of the @align, then lookup the PFN of the struct page address that precedes it and use that to populate the entire section. On compound pagemaps with 2M align, this mechanism lets 6 pages be saved out of the 8 necessary PFNs necessary to set the subsection's 512 struct pages being mapped. On a 1G compound pagemap it saves 4094 pages. Altmap isn't supported yet, given various restrictions in altmap pfn allocator, thus fallback to the already in use vmemmap_populate(). It is worth noting that altmap for devmap mappings was there to relieve the pressure of inordinate amounts of memmap space to map terabytes of pmem. With compound pages the motivation for altmaps for pmem gets reduced. Signed-off-by: Joao Martins --- Documentation/vm/vmemmap_dedup.rst | 27 +++++- include/linux/mm.h | 2 +- mm/memremap.c | 1 + mm/sparse-vmemmap.c | 133 +++++++++++++++++++++++++++-- 4 files changed, 151 insertions(+), 12 deletions(-) diff --git a/Documentation/vm/vmemmap_dedup.rst b/Documentation/vm/vmemmap_dedup.rst index 215ae2ef3bce..42830a667c2a 100644 --- a/Documentation/vm/vmemmap_dedup.rst +++ b/Documentation/vm/vmemmap_dedup.rst @@ -2,9 +2,12 @@ .. _vmemmap_dedup: -================================== -Free some vmemmap pages of HugeTLB -================================== +================================================= +Free some vmemmap pages of HugeTLB and Device DAX +================================================= + +HugeTLB +======= The struct page structures (page structs) are used to describe a physical page frame. By default, there is a one-to-one mapping from a page frame to @@ -168,3 +171,21 @@ The contiguous bit is used to increase the mapping size at the pmd and pte (last) level. So this type of HugeTLB page can be optimized only when its size of the struct page structs is greater than 2 pages. +Device DAX +========== + +The device-dax interface uses the same tail deduplication technique explained +in the previous chapter, except when used with the vmemmap in the device (altmap). + +The differences with HugeTLB are relatively minor. + +The following page sizes are supported in DAX: PAGE_SIZE (4K on x86_64), +PMD_SIZE (2M on x86_64) and PUD_SIZE (1G on x86_64). + +There's no remapping of vmemmap given that device-dax memory is not part of +System RAM ranges initialized at boot, hence the tail deduplication happens +at a later stage when we populate the sections. + +It only use 3 page structs for storing all information as opposed +to 4 on HugeTLB pages. This does not affect memory savings between both. + diff --git a/include/linux/mm.h b/include/linux/mm.h index f244a9219ce4..5e3e153ddd3d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3090,7 +3090,7 @@ p4d_t *vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node); pud_t *vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node); pmd_t *vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node); pte_t *vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, - struct vmem_altmap *altmap); + struct vmem_altmap *altmap, struct page *block); void *vmemmap_alloc_block(unsigned long size, int node); struct vmem_altmap; void *vmemmap_alloc_block_buf(unsigned long size, int node, diff --git a/mm/memremap.c b/mm/memremap.c index ffcb924eb6a5..9198fdace903 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -345,6 +345,7 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid) { struct mhp_params params = { .altmap = pgmap_altmap(pgmap), + .pgmap = pgmap, .pgprot = PAGE_KERNEL, }; const int nr_range = pgmap->nr_range; diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 76f4158f6301..a8de6c472999 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -495,16 +495,31 @@ void __meminit vmemmap_verify(pte_t *pte, int node, } pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, - struct vmem_altmap *altmap) + struct vmem_altmap *altmap, + struct page *block) { pte_t *pte = pte_offset_kernel(pmd, addr); if (pte_none(*pte)) { pte_t entry; void *p; - p = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); - if (!p) - return NULL; + if (!block) { + p = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); + if (!p) + return NULL; + } else { + /* + * When a PTE/PMD entry is freed from the init_mm + * there's a a free_pages() call to this page allocated + * above. Thus this get_page() is paired with the + * put_page_testzero() on the freeing path. + * This can only called by certain ZONE_DEVICE path, + * and through vmemmap_populate_compound_pages() when + * slab is available. + */ + get_page(block); + p = page_to_virt(block); + } entry = pfn_pte(__pa(p) >> PAGE_SHIFT, PAGE_KERNEL); set_pte_at(&init_mm, addr, pte, entry); } @@ -571,7 +586,8 @@ pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) } static int __meminit vmemmap_populate_address(unsigned long addr, int node, - struct vmem_altmap *altmap) + struct vmem_altmap *altmap, + struct page *reuse, struct page **page) { pgd_t *pgd; p4d_t *p4d; @@ -591,10 +607,14 @@ static int __meminit vmemmap_populate_address(unsigned long addr, int node, pmd = vmemmap_pmd_populate(pud, addr, node); if (!pmd) return -ENOMEM; - pte = vmemmap_pte_populate(pmd, addr, node, altmap); + pte = vmemmap_pte_populate(pmd, addr, node, altmap, reuse); if (!pte) return -ENOMEM; vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); + + if (page) + *page = pte_page(*pte); + return 0; } int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, @@ -603,7 +623,97 @@ int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, unsigned long addr = start; for (; addr < end; addr += PAGE_SIZE) { - if (vmemmap_populate_address(addr, node, altmap)) + if (vmemmap_populate_address(addr, node, altmap, NULL, NULL)) + return -ENOMEM; + } + + return 0; +} + +static int __meminit vmemmap_populate_range(unsigned long start, + unsigned long end, + int node, struct page *page) +{ + unsigned long addr = start; + + for (; addr < end; addr += PAGE_SIZE) { + if (vmemmap_populate_address(addr, node, NULL, page, NULL)) + return -ENOMEM; + } + + return 0; +} + +static inline int __meminit vmemmap_populate_page(unsigned long addr, int node, + struct page **page) +{ + return vmemmap_populate_address(addr, node, NULL, NULL, page); +} + +static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, + unsigned long start, + unsigned long end, int node, + struct dev_pagemap *pgmap) +{ + unsigned long offset, size, addr; + + /* + * For compound pages bigger than section size (e.g. x86 1G compound + * pages with 2M subsection size) fill the rest of sections as tail + * pages. + * + * Note that memremap_pages() resets @nr_range value and will increment + * it after each range successful onlining. Thus the value or @nr_range + * at section memmap populate corresponds to the in-progress range + * being onlined here. + */ + offset = PFN_PHYS(start_pfn) - pgmap->ranges[pgmap->nr_range].start; + if (!IS_ALIGNED(offset, pgmap_geometry(pgmap)) && + pgmap_geometry(pgmap) > SUBSECTION_SIZE) { + pte_t *ptep; + + addr = start - PAGE_SIZE; + + /* + * Sections are populated sequently and in sucession meaning + * this section being populated wouldn't start if the + * preceding one wasn't successful. So there is a guarantee that + * the previous struct pages are mapped when trying to lookup + * the last tail page. + */ + ptep = pte_offset_kernel(pmd_off_k(addr), addr); + if (!ptep) + return -ENOMEM; + + /* + * Reuse the page that was populated in the prior iteration + * with just tail struct pages. + */ + return vmemmap_populate_range(start, end, node, + pte_page(*ptep)); + } + + size = min(end - start, pgmap_pfn_geometry(pgmap) * sizeof(struct page)); + for (addr = start; addr < end; addr += size) { + unsigned long next = addr, last = addr + size; + struct page *block; + + /* Populate the head page vmemmap page */ + if (vmemmap_populate_page(addr, node, NULL)) + return -ENOMEM; + + /* Populate the tail pages vmemmap page */ + block = NULL; + next = addr + PAGE_SIZE; + if (vmemmap_populate_page(next, node, &block)) + return -ENOMEM; + + /* + * Reuse the previous page for the rest of tail pages + * See layout diagram in Documentation/vm/vmemmap_dedup.rst + */ + next += PAGE_SIZE; + if (vmemmap_populate_range(next, last, node, block)) return -ENOMEM; } @@ -616,12 +726,19 @@ struct page * __meminit __populate_section_memmap(unsigned long pfn, { unsigned long start = (unsigned long) pfn_to_page(pfn); unsigned long end = start + nr_pages * sizeof(struct page); + unsigned int geometry = pgmap_geometry(pgmap); + int r; if (WARN_ON_ONCE(!IS_ALIGNED(pfn, PAGES_PER_SUBSECTION) || !IS_ALIGNED(nr_pages, PAGES_PER_SUBSECTION))) return NULL; - if (vmemmap_populate(start, end, nid, altmap)) + if (geometry > PAGE_SIZE && !altmap) + r = vmemmap_populate_compound_pages(pfn, start, end, nid, pgmap); + else + r = vmemmap_populate(start, end, nid, altmap); + + if (r < 0) return NULL; return pfn_to_page(pfn); From patchwork Wed Jul 14 19:35:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12377745 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A87E06D14 for ; Wed, 14 Jul 2021 19:36:30 +0000 (UTC) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16EJVMMW031363; Wed, 14 Jul 2021 19:36:25 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=DgFa36vKxyDbO5Oq+1ov6YbXaB8wMCO9g9s3XnYQq5E=; b=C6KNc5jOO+paSUiDwOEPLvPr9wElDqMzZg5VF1a0d8Yz7TwZapaOL2M7OaMebXT0aVLA 1ZrSNPY+LOHenLm2pqFfj4jVW958BRsKhmKpesN+PQrieAbkR51P9rSkdEE/kpKaARqd 7DlEl3s1kbyA+JhDEXDkvkAokj5P4VC1ANUoXT8mQO1Fnvm275NC07cI0hl5dkLOL9o/ To5zIvWOPPXLmUrTaKDy4xxLlMvSETkBFXuUFVAxZwf/shRHnqJtpxvnOIA+GM6vGzht M9uTQ1fEMT0c4UtAXCWle/loxc7Gg7I1WfKE1U2TaX7wi1j/aloqxp1CRpO1PqA8DTT2 QQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=DgFa36vKxyDbO5Oq+1ov6YbXaB8wMCO9g9s3XnYQq5E=; b=Bx9OnzGP9tq2kdP/DNtAeRVQhEUEaII5ovcKu6i2k/g0xMzqInXgRV8VQgSaHoSkaQkn kgQxcRc3i/SalyqeHaegzn2rWFqvz6bJUeadFTMeopvLJoVJJ+CzoOc8CtLK4qHO2bYo ZNlSGcaqyFGy4KMI/4THPUrbeE3CHsuYCT97f/lQvQO0RHJLLw7GJ5ImkWCFkR+lOjw6 A3nHQbHF8/Brn+L1cO88C3EKJmu2wgu6bTkMNY3z7GLKGOT2PmCKZ2lmG2fUaHOodv0b 53eZCZ0tbX3GpjOqgiqfZ3pc062lsJqbam0YsWhCuUoKmT1dVIC9awN9XyLP2Ma/JlUT 0w== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 39rqkb5knd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:25 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16EJUXgJ021507; Wed, 14 Jul 2021 19:36:23 GMT Received: from nam02-bn1-obe.outbound.protection.outlook.com (mail-bn1nam07lp2047.outbound.protection.outlook.com [104.47.51.47]) by userp3030.oracle.com with ESMTP id 39q0p98max-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:23 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FGjcXfZ1rwCjNTNfi7hw1qwfatSBYGogAHnJ9wyiVacuRa3ybyIRDq9y653sURhsIBNJ88ebSZwzp1wZrngTfxWBMaE8GIaEyzUfPdCr889ltUBzKQNpL6rVWMrZVS7O0/y/qjCWo5dDZSXv6z5CHocgNpZjzMjUD+/B3FYB6nV2B7ODUIiBCUrUKU2JA90ULIIVybRNRQL9zaUg7KCa9KP5CHVhyM05oJJRbw8UlGVHTE7fyTih26P6bH4v4y7J4oMij7I91EfihyIr7rrHEjiYcq1zL63a0UNazs+bu1m9Q/wFy8GS9fXJess/xYRIe+ZcSi8dJvEncj03YCkGtA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DgFa36vKxyDbO5Oq+1ov6YbXaB8wMCO9g9s3XnYQq5E=; b=cAonrxfh2hbNGU/Ei5E3qDowR+Ws1LznVog6qugEhbPxzJ7+G3AdovYxPbnrov0DATg/S/uwHmTLino5b3YJezY/9OnhsOqLzlAvzdecPpoEzhumwJMzVDhfXplJObhWX2Nwd6CEHHIfxlUygnTze3ovZvv+a+snmNC6O4fVsEPyJKNL5N4tLYWpTFJRXPGAok21ImGm28tLAWH/tvPkg7h4yr5Gkwm18TdYdFA/v0u81TEh63a/whtCR4W1YAv8qcoz0a1UOSQHvUyx4ZnWSHsHm47XBL7d+TOxiN1WM44Y91fP25NbEDwO7qUjKStOgqzFk19J6N7X9Or3JzJlng== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DgFa36vKxyDbO5Oq+1ov6YbXaB8wMCO9g9s3XnYQq5E=; b=NGcJAWTIePrnribpIPZQz/bUPjoCqwvHUSkjxAROYfhLb/cp3EbiDpmZ1sAtgnA/LjsqbHY7GzTeHRUWqQbF9nBWytxIdzSWnnqZu5tI39wAYUZvqM6i/RDw0FAKV9PD2QfVYGfupyrGZoGVGCz6Kfw3I3CLPlzmAmTW8PTYEqo= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BLAPR10MB5204.namprd10.prod.outlook.com (2603:10b6:208:328::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Wed, 14 Jul 2021 19:36:21 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360%9]) with mapi id 15.20.4331.022; Wed, 14 Jul 2021 19:36:21 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v3 09/14] mm/page_alloc: reuse tail struct pages for compound pagemaps Date: Wed, 14 Jul 2021 20:35:37 +0100 Message-Id: <20210714193542.21857-10-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210714193542.21857-1-joao.m.martins@oracle.com> References: <20210714193542.21857-1-joao.m.martins@oracle.com> X-ClientProxiedBy: AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4331.21 via Frontend Transport; Wed, 14 Jul 2021 19:36:19 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2dc6b64f-a22f-4e25-1c62-08d946fea87a X-MS-TrafficTypeDiagnostic: BLAPR10MB5204: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7219; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Z6ExAPWA9gSVMUJZYbq3jwbgIPnTpH4doipXlA1f5AdpVvj+dFRdvGft7SrA4qkgapC6+9iGlbZQV7vNR1aaCxRLGItkG21t+TWdeEZ8npvyUYucqiQW1FUO4+nKr/nGaKZaBwvUZQimH1roqF+fDm7/yNRT1Wf0rIIlvn4VoHC3KXYC7TVp5wRlhu+JlNAOSTs6N8My3uiuzuwGgD8wjlgWxGkqcD1JjW6+cuJg8BMtysukfXraLznuINwznCDvacDkRx87+FDx3mnuH4ZKnqwdJthhw459NjkW7iwyI48fwJ+pk92/SVZOK7Mjv+WnE/aMRTNr5TZsFDGfmOZbeJrpYYLwarQYS9eYKi+BFVzTGWvAq3XwrojQsdVQJL/kWyi8Y5BtLAYvZ1G884RCi5BbKfbZtLTxcljY0Pe/TuncHvqPFuupxrOPlEHmFOCYQ9wAeydKrh8qv1GdhQDnHO7Vj1LiuK0MdGIZnTsHBjhZWqJhtYS28RAFtWRFFCzfKlG3PYaxA3QwVO2z2hrSJ25/5HEKNSMwoewnT6FUMOpITt9Od2Bve6g23u8HOJ4eCxxiwsO37B9aZbaTIfDUthdEPMP5AYKGyWkKj3E8Xg4KiSdhpJb7h1GOwlyOAIZYtTz0uagi9PiXl2Nq4LlnpcUroaMjZmSn5/7eGUxhdbs9thpWSnALNrqD0Gp02IcLBhI8rIvgYiMnZCeoitSziA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(346002)(396003)(366004)(136003)(376002)(8676002)(4326008)(86362001)(6916009)(7416002)(2906002)(956004)(107886003)(2616005)(6486002)(1076003)(8936002)(66476007)(5660300002)(36756003)(103116003)(83380400001)(38350700002)(38100700002)(54906003)(6666004)(66556008)(316002)(66946007)(478600001)(186003)(52116002)(7696005)(26005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: xCdSzq4kC4+LesmXrsbwWRIHsPdvuCJ9fP65dTlOSBSKKNemMDLcLJp/YhTRxWJme3KTK7eavRZ+P/hFigUDkXbr5v7IEur6FqmcfFI0hQJfnqb6hTvDmRbETDi68SqE3l7KJCrQQyCJPyAsBcik7aAfDHLgmzF0xaM3wGrkHCBEmQtsqBfBgV38lex+u9es1nILvybEgErMNGgUgcyBWCD9G9qooHz4VMrOW2HusKEhq8OKNYJ+++NpbCD41s/x+exE3CLnLAxVDA828V7AB5tfoaNXsc9MNliuycerQEl6/zT7O4pIuFGFtb/IJINLOSKozu6IHeL1IyvFsYM8GnPgJoNI9G5/5Ki1gwzfliihgRHF7hmd7e+/IDEBgrR9z53NplPlgmc2A5RtVl1YoRCafSQnojdD3sUPM+t7144xZNeaxYHRFxvnC+hhP6khJwxIkeCTWlLMZqembSON56UxOlqHI9DhMddWrA/OHgxJpf2i0AGgrbFOs53R2WrZM3N+SpqibZ1PQGEbX/gJBhunNJz7DDH9XBOdj16E8bW8x4plMJsNaoLNYxSQVpwpASym5ld6B8CPzKihzFOjy2uje0TpFH3b3Pb2FwFdOdcVGbrGKNXp5WRc62kW6mem8gCmw5LEEaOGv+QBS9IRsIDSTUhrY3TBS+ZQRmW+xGQYa/bhzM5nFsQgrTMyzxdZ3P3sTlsbRX3i+A5DiqUudH0vLbuSLXQuJNqZuozBF3u5PIySRuhsfvxl5j7NEPBa59veoeaDyRVFf4ANYFfIUOWNpNjBTvaxt/u56gSprd2sooQEBWtfmgKJriRhWgB2vqtre+oPA0QiZk3Rv9nAPLpihWgQbTTcto1sRWlUhNbdIkV1KSgN/eSvqhoOIu9SLp1+SGxCa26DOMG3uXHZ00ehH6l88zJzSEHzJxDvy3Gb6FGv1e0+wE0aALJCaFcjSoybwOPuAsvuFvU4GpTqlYT01Ju+FoTXPD3H5iq7tn5ZzyMqEdo7xiTNsutw7/LRAFd1U//WJahT3QFcXhh8I0Y7tWZtf7LKfP+/inte/ILqvSMXwrRIcAzaPmIm2+RxNVUk1FJOgtoWq4IQgxsCGAZww9XUyNT6o3tTVUFXV987dVuBrQU6iMjYJeFRTy/b1MztqK6vZpHuswO6biru5E2rrMTqJLdiqNNnEOdigrrT4Riwx87ppySNmTVqeMmOti/BUQfAHXfAOALmL8rhUjaXJWUABg6gWqU2Z+mw9mH4FLtlxaROdgw/GCpcGlTjedlLDn/5Hf9qYlH+MFw3w53AessotxYc2RPaDV21PQ4iz2ILhdCCbko19VmiG/lr X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2dc6b64f-a22f-4e25-1c62-08d946fea87a X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jul 2021 19:36:21.4178 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Sin5DG/cO6/Bfdlm/uP6T5eRg+O8IYknnIgCIM2kjAED9UAjq0r24LsHKeb0RTU5QezqHrllPRhNGkk1Nr9B5in5GLltp/bDBQQ1m/lxypw= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB5204 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10045 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 phishscore=0 spamscore=0 bulkscore=0 mlxlogscore=999 mlxscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107140115 X-Proofpoint-ORIG-GUID: PNMdpS0IzVgF6E6OfWgr2zg7oxkWjhQK X-Proofpoint-GUID: PNMdpS0IzVgF6E6OfWgr2zg7oxkWjhQK Currently memmap_init_zone_device() ends up initializing 32768 pages when it only needs to initialize 128 given tail page reuse. That number is worse with 1GB compound page geometries, 262144 instead of 128. Update memmap_init_zone_device() to skip redundant initialization, detailed below. When a pgmap @geometry is set, all pages are mapped at a given huge page alignment and use compound pages to describe them as opposed to a struct per 4K. With @geometry > PAGE_SIZE and when struct pages are stored in ram (!altmap) most tail pages are reused. Consequently, the amount of unique struct pages is a lot smaller that the total amount of struct pages being mapped. The altmap path is left alone since it does not support memory savings based on compound pagemap geometries. Signed-off-by: Joao Martins --- mm/page_alloc.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 188cb5f8c308..96975edac0a8 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6600,11 +6600,23 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, static void __ref memmap_init_compound(struct page *page, unsigned long pfn, unsigned long zone_idx, int nid, struct dev_pagemap *pgmap, + struct vmem_altmap *altmap, unsigned long nr_pages) { unsigned int order_align = order_base_2(nr_pages); unsigned long i; + /* + * With compound page geometry and when struct pages are stored in ram + * (!altmap) most tail pages are reused. Consequently, the amount of + * unique struct pages to initialize is a lot smaller that the total + * amount of struct pages being mapped. + * See vmemmap_populate_compound_pages(). + */ + if (!altmap) + nr_pages = min_t(unsigned long, nr_pages, + 2 * (PAGE_SIZE/sizeof(struct page))); + __SetPageHead(page); for (i = 1; i < nr_pages; i++) { @@ -6657,7 +6669,7 @@ void __ref memmap_init_zone_device(struct zone *zone, continue; memmap_init_compound(page, pfn, zone_idx, nid, pgmap, - pfns_per_compound); + altmap, pfns_per_compound); } pr_info("%s initialised %lu pages in %ums\n", __func__, From patchwork Wed Jul 14 19:35:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12377747 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 25F612FAF for ; Wed, 14 Jul 2021 19:36:33 +0000 (UTC) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16EJVWTn022332; Wed, 14 Jul 2021 19:36:28 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=o13ytpm6INie/NN14mwJls2enUk1cPtSh7jEJOPQtuw=; b=zpp5j1XcdhM8Zj7vUlH82Bos73EsMV3S8L3oPCZicN2NMMOtMAbt2eEy5x+i45cejJ2k Ph89HdFJ23GBK5vLTEi83YiT9XVOowdfEpf6a6+cPKv2O+XWpcdi1bJjuNPYiY+nuN6R 1xcHHzCOe49Al3bVW9xx+tmYuVd0ey8jrnKaGfwllXIgXgKPCIF2/RBGiILKi8PUNrCr mI7VNht6ihRZry2ikdQ5u+HuDRQ3h3L6LEKB7iQOrhUTKKntwmOcwrWhIKCEkYUXVMfd W2hhTyvPVjw96ybToSNs/OyRS0bjg6TqtbXykbmySa6+wdCR/LLVUXfksqMO6nWtwJqR BQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=o13ytpm6INie/NN14mwJls2enUk1cPtSh7jEJOPQtuw=; b=bkSXQJB0OZsu8nmv8hDpwjvOEOOImSAUjc2U2XWLJBjrfGG7KPGLUBIIQOwuuriy/mzn 9o96cQNpXZkgVrLqaK2UJNVEct2uSFmmCISUyGxBTUQQd3FfMIeqB4Lj1uKl+At7LCjz l2eLlEwckUY0zW4QcGsCDhbQUAjO2VaSENkW9vCsziZe36tNZYVJ69gHXs6wmTYamDtU MRPNDmqkEkQ+gqh19tUHgZidhVbfrDBWwFkN2cZ3RWt532Mn/Ggxeq3nhN5V1A5FcxEW PBzhYVM6HVkTS0syQA+j5CjfwUDiuwQl6ob6aZ0HZ5MGl+2InbPsJ/Idw118RA8CN4O0 eQ== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 39suk8sejy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:27 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16EJUXrX021532; Wed, 14 Jul 2021 19:36:26 GMT Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2104.outbound.protection.outlook.com [104.47.70.104]) by userp3030.oracle.com with ESMTP id 39q0p98mdn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:26 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IvnFMAcEFrr6fPjxzRepdruN3x+N67wWkSKK39Z37HFn9LbKPzJTr6xyyNrIU1KTwbRvJ89SpVDEwwQFsdYoK7kthZtzMfP4IRarmYkdyhBv0VyZ2xKgfASS5RMrZrHlC6XZO0chX1uy+1kHnjxUHSs3MnHKdZBf1/qqEW70ldIBGW2tmbbhJgWoY++7ieUK/41GUvkrUZ61jfgcxzzK8V80n+CuqoZxLs9CWLebgkNZPuHqy+ZHRLbB+DB2mP/W8WcubZVfhO0hOHYmiXiW0fcmLyMGH2jCs6PdP0lZHfEzSiIHwpU0Bv4SXQVF+3jjU9+9ml3jz65hHq3G03AeDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=o13ytpm6INie/NN14mwJls2enUk1cPtSh7jEJOPQtuw=; b=nH42uNyFVeIwn+obbTUHyne9cDZ6KN9eQVyZgaLLJOLtZ8wARrY5LR8J1GW3SwnD/44zXuulKJ7EX4W1wF+BpHpXIGBzvS9ML60PmXpPT1ZMEyz+bRlo+EZJNZCq78KaRtovc4cES/qdYPPPvfqkTQpaki5P+weulpRpSBYxxMh+SRk1irWMsaCnC4y1zmJcRktZxqCWq9XIL4HPlGlsF/X58uvR81wr2H+OieRYUmml+j2M8h6dQyjflYX+5Cng8EtR1zOONDCX7RjGyXbfO08bq6MN3J9JRV+3s1MZ64TulaVUj+GdNxHoa99Ip91Q0q0jC4OqWNt43wJI4TawEw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=o13ytpm6INie/NN14mwJls2enUk1cPtSh7jEJOPQtuw=; b=Mo1yYcQVM5ilTzGeo9du30enB3SNnmuqFO15iWiFL32jHEL32VL2iQ8E6jZhX8guo8tmUWVPd5p5QWtuDxxbCjFmzkS44SNjk4vAxK44HZDDl4qK4CG3c/h6dLBJ8Mql539HoL9R28hgPu3EYTSLeLkB1bkNPfb93tKJyR69moQ= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BL0PR10MB2899.namprd10.prod.outlook.com (2603:10b6:208:78::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21; Wed, 14 Jul 2021 19:36:24 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360%9]) with mapi id 15.20.4331.022; Wed, 14 Jul 2021 19:36:24 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v3 10/14] device-dax: use ALIGN() for determining pgoff Date: Wed, 14 Jul 2021 20:35:38 +0100 Message-Id: <20210714193542.21857-11-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210714193542.21857-1-joao.m.martins@oracle.com> References: <20210714193542.21857-1-joao.m.martins@oracle.com> X-ClientProxiedBy: AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4331.21 via Frontend Transport; Wed, 14 Jul 2021 19:36:21 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 05f4e851-1b4c-4086-866e-08d946feaa1a X-MS-TrafficTypeDiagnostic: BL0PR10MB2899: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2512; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: //DS5/blqvKD7K8715LxJxfSnmcnTA6Phl3CIICxXBxSjzd3pzSBHWhjWBAfypfROupSBQUanii0wIHGTgVswhPHAhZ/5sWoBAge/rDy5JghdNubJ+tA0DHGGCYBob/r+gdDbm/CmJzgsxX6htVgL0SWno+Q8aFZFAEro5wxGg00Xdi9mrBZQ3tZbt+Rn/PzjkTWJG7Duv8h3l7N+ISShHD4SwgIIkD+PdLHtKc+YpeFUZSl4lU/aCWo0ERi9T4tFLd/F5s81SxdWxxKEyFisYuOHD66UUTrnvkmUBs4TEAxUOyzrk6OhthFiFK09518ZInadAqMjjUPel1GVMqO9c9sA8zGTZhRaYodwbdbKQWxRlUH9ffWhCE0WWjwbbQN08mWZkMchjVD+G/8Oo6f0VZ0ekvw87bi4cTfx5/19Kl9vctB0gbwoC3L3uzy8ySbnofjbI++K4NkIXG5cpUxkWk0sN7QybOYcUmCo3G75+tOgiXCxQhwRs+LR5PpeA2YjUBVJ5v9XezPZWzb+/BbvwXdCa6X3gT78pRFqBkG8XLbcOMZWBq6PVSA40WglnAaKIhmlvJ9dSzJml7BJCHf6ouQrJ1xJ40Q/qpsnjq9TDGrnIxylyMEpdd2LtXw/4ZC8XDsEsVg1QYV0kIVD3yWIuzs5twUMsjLat21m2NSDK/ijAvIB90XO1LvHCROKJNRYRcHGAZepbYSvXpJqH7C0w== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(39860400002)(376002)(396003)(366004)(136003)(36756003)(6666004)(478600001)(2616005)(66556008)(956004)(1076003)(4744005)(8936002)(107886003)(8676002)(4326008)(5660300002)(186003)(26005)(52116002)(66476007)(54906003)(66946007)(103116003)(316002)(7696005)(6916009)(86362001)(7416002)(83380400001)(6486002)(38350700002)(2906002)(38100700002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: pfh6b43GqJPbmajU0h1ihY4rv969lRbaM3IAA8vUo9VihzAgeLFma38Q34HZpBuPSb5kxzGF+2TLWGIRqQRpX+9d1/HWdKyUM1ckHHl3l1uI9hncZhgPOlNy7s/dJgJc7BmIbmYZLa9npwqBiQFFRKc/tIhnVKjwZLZUKecjnq6H1vHXJXihgNW83XfAXvrOc8XyCeXy4lN4uh1SucEuEMRkDDntAuAHxDOowloAbjRyyKHyifjJb6ehHtEejufJ66wLN2QNCzOwo60eiKuJu91jx1So+Zg0uk1ILY3qzMR8PEwzF3xyZ60hpLTG8xrmYeVQ10uL2Fv+dcHZyFLhMfuKocdZTmWheb94hDkBzxYn2SPDObHm4RpQ08jOnBSAHU4SNBDMB6nG/zH2qpflbYIXI/Cryg1SsDOgevh9XeD+GxDady/BQThdTA+ByHcknbUv3nBlAc96M0kbe6DBp5gRaEDO9ChTplaRL/7PixpCJg5qOq0fpjJ5ouuw1Tn64XTfy1s2HidYn9/2nj31acPb+WXArvz9esAI2zeMvoFVwEdOnB8gjJb4frzmVQAedBnhbgJ5g+oLJRbQU+MQ6IVuc1gHLRsnU0cEgTEaZZbleF7Cu5Lyk5gvRFTnGm8WHWnIFO1DYmbUBouuljAT8QQMqKgBVpdtuAmXjbRRk+yeZdZ5jRoJwVqmTjaHWNG2R6JkhQ/g+wCFdZdmZ38ki51BdifkQXt/Dqqbv2lbmaVrh68qHS8y2R32XovabMA/q6fbSEho1wrI+Kmd9l1IErhsYup+S2LEsjGCz4Slof8LZE8f5TuZwOmLoR67kOKzqlIf+Re/oP1VNtKNdUPmevd7vaZF07bedGx8+rBmo9AvgfGbIyfax/T9Tl02MZSVNCauUy76TbwQKKwU1VcUn8HE8sowlW2/a6p1uH4WfokS3Z+xPkXQbH/XECI3DPlSldP/KKRh7JUQ7eOqYIka/UKJDJZPkF/H+F4N8yTw5MP5IIW4/cgj3GwAUtgHuoyZILfZFbzF5k7huR6yk137uKiX4klEAuSgV/jOvYUVcvpxwsfd13a9JEM6lUw6im6TYkUb/qvjMH57KU9cCqLDqz3e8F/pmVyBXrtXkwbSwBENZ+XvJPQOejL+cvlqJBUFlEts4eK/E/Borj7Sr7T0YRJtuyu7tAO1zOx35upd7mNb+HOtr0ObQaOV+SdkkVBUE/Yni7KadlJz93HvgFbKUzNQRLJtVktVjPBfN4PiuuOOWlEZigaQ/nclbDvAYENmsSysTUs2ljo4kPmyrLxDmXhq+0lyBtvaNALpaPXQVD7+FXC+mgJf3OX6YHUWQCuk X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 05f4e851-1b4c-4086-866e-08d946feaa1a X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jul 2021 19:36:24.2105 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: v+iRekKBqT1nkTObqk51KRpa5D6Q34pocR+alkjvH3j71c1EBC7Wrgdz+UTIQ7vxJqp845VdTQBLlzHw0c+P7pwKxe7pAPRWUk8dH1fDG0A= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR10MB2899 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10045 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 phishscore=0 spamscore=0 bulkscore=0 mlxlogscore=999 mlxscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107140115 X-Proofpoint-ORIG-GUID: 2VsLBtRtdvkao4YmgYE9NwFgmmx_7CFv X-Proofpoint-GUID: 2VsLBtRtdvkao4YmgYE9NwFgmmx_7CFv Rather than calculating @pgoff manually, switch to ALIGN() instead. Suggested-by: Dan Williams Signed-off-by: Joao Martins Reviewed-by: Dan Williams --- drivers/dax/device.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/dax/device.c b/drivers/dax/device.c index dd8222a42808..0b82159b3564 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -234,8 +234,8 @@ static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, * mapped. No need to consider the zero page, or racing * conflicting mappings. */ - pgoff = linear_page_index(vmf->vma, vmf->address - & ~(fault_size - 1)); + pgoff = linear_page_index(vmf->vma, + ALIGN(vmf->address, fault_size)); for (i = 0; i < fault_size / PAGE_SIZE; i++) { struct page *page; From patchwork Wed Jul 14 19:35:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12377749 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5C9762FAD for ; Wed, 14 Jul 2021 19:36:36 +0000 (UTC) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16EJVVkV022326; Wed, 14 Jul 2021 19:36:30 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=XS+vp3u3f2gS+4FaUrtKudUv4lyVIJUIhs/+PkjZjxU=; b=AmxLNzAEdnNmKH+8IwEJAK6E1Jxufo8dvp35I8pu6d8T6w9hXOa8xpSAVIpq6DvIZ4WQ qMYxpVxO6jqPv1kC3ROpLSHJZwmEs4Juq0mMntqp9SvJZtE9GnQA4beV9yEscjbZZvwB D88dtNZh7iFcG6H8UP2VCnscXFIhsXpeLHNFdhOrKxm9Ee7FZASmJkmA8c7kukLIA3Kl SDqcTc/XTPSuD93LnWhGQwP9WjVHUIBJKjVvuCPPN0asvbYaEl4jCYT/QM4ZpHGDZ3R5 UjBpZ46mMKMZrHpFhg+9jqKBAHWyyXpDtztpAJcc08udooMzZm93+d/g6eVD1tn2Cqvf +w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=XS+vp3u3f2gS+4FaUrtKudUv4lyVIJUIhs/+PkjZjxU=; b=QSond0ZPvY4kIjqW7/PgwWOO4QLdI3J/D4GaFnM5/mfZAlnmkYPr2PPw/skdkr/bn/H0 8oVjsLzutYv6NxygOFGpuHZoOkb84pskjWP9DljsW/4IFUm2adaSPMs2LPeMMWNo7362 W2Is6xIKpDP1TkEnG0zTF/RD12hJh1lBY9j4/tIhGrVNUf+VjjRFp8jd6hEuQqok3cUv 8nGJAR5mhRq6BRZmMqFy/05KGzmM0vkYqTR7RCykLquigU+HVhZkzUErGGBPVvGJzifB c1YxHm+X1TIHR7spO2Q7AfIFhMUdk8RUTSYEm32CC9lxY4JAIIeaw+KBifU/V1P7VdOK /w== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 39suk8sek1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:30 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16EJV0sQ079711; Wed, 14 Jul 2021 19:36:29 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2168.outbound.protection.outlook.com [104.47.59.168]) by userp3020.oracle.com with ESMTP id 39qnb4290t-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:28 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nb1QlTviaedz757qwIAn69TmEu7jpwwidxxLH8nwkz1FEPltXaeFkclUvHJuH6FHe6IHwkpea2zzzT9SkHoWFQBt0VzmqKMWucrOeHuQ+Dqm0BKFKvyjqnQRr239lZoZp8nk6DTQD6Q2pawCCEY+peGlVqmgN/4x018C8UN1AbckqvrzSEQvppt++JkHhcitCYPzgypXOo55co21SH6DfbEY53J4Ncm5k7XejTneeSlppGXRl/jCYYdqRkYIhIOsW5qrHApbUlEoucF8WKQ7dNlYfme+FCU4DRv0KMeFTAAkRTk5Ck4dGMHmjRj0ftLA8v+x0lRzoKROKvAT4crN4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XS+vp3u3f2gS+4FaUrtKudUv4lyVIJUIhs/+PkjZjxU=; b=m4Tm6+IZ2IHzp2WW1cglr5yrOp01nnKftGvAgRnhyMf3Zf+6dfurexkTQKLOxoddog19kdAYLSf9gf/1W4Ck2Hozl00yfGPlX8HY9hjsCkifGLVfaIK3AtK4NLCMzFKG08e2faWTNYvFJAouANuIDqhWVF1yyF59sheEujqBObXVbLMbTKhDpUZx3rghlcJKtXgdq67PvqWnIzSSEJ3DMVqFT/Bdf71p9LI4ufPqOLk/NoqUmt6kbaf1WjqGpgHiUh3qP2Ja/8sVReKjsypSp18MH0OHJbJl+ZEkIrqbxOnSJNF7Zq9vW/jrTdufTl5lSDqlgMC+pgEQ1YuGas/gyA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XS+vp3u3f2gS+4FaUrtKudUv4lyVIJUIhs/+PkjZjxU=; b=jJDfXoJHrCV4rEKAD06raLWD/Gr4yQavUq1f/yKUEHikEWtqI4W1uDG50dMZOclKh/wWrw+l00dIg6nulAM3mqVHGDqc09JCJRrOIrmVK07wGtqSxbyZlBAhdZ0RwfQXlVVIDUBWqGnIyyoQtMJ4gaIQCuJVIATE9jkpCSLbc0c= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BL0PR10MB2899.namprd10.prod.outlook.com (2603:10b6:208:78::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21; Wed, 14 Jul 2021 19:36:27 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360%9]) with mapi id 15.20.4331.022; Wed, 14 Jul 2021 19:36:27 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v3 11/14] device-dax: ensure dev_dax->pgmap is valid for dynamic devices Date: Wed, 14 Jul 2021 20:35:39 +0100 Message-Id: <20210714193542.21857-12-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210714193542.21857-1-joao.m.martins@oracle.com> References: <20210714193542.21857-1-joao.m.martins@oracle.com> X-ClientProxiedBy: AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4331.21 via Frontend Transport; Wed, 14 Jul 2021 19:36:24 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: a1e36061-a547-4984-37d8-08d946feabbd X-MS-TrafficTypeDiagnostic: BL0PR10MB2899: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:854; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Lt2OjV0LlOUze2VFsWs6yuZPPzHGfEtKx4vB4ldN/9i9Xj2O+AFY040fTMIJdKv9XjZvvBeUTYrXY1I2kWypq1hlPx+WJiYnoXsfWS1MBkfM0LoPNQ7IxdS60kfxIhk4VRnOGBeI9n7XNR9WNEoNdc6LRdA5YfjYrhfSNZTSM7P7QJ2tMH/CMLOpLuoVBVKUsjAf9xezMIlI+5wlsaTYkRP9cjSccANpkO4XtwyTtDPh4Dp11mYq07GBbQs4Vgp9agISlZSMGlAsZDOwM4YDbohGRsKmr6dIzvx1CtBVwMnbqPBWJ+M110nVxbV06Z9dPY6twqokbe0u5dkoyGGqwXK1V58xoN7l2SPGvVDxKmZiq/QCcFKawVHieTf2zp7C/yFgT28bGK9zxg8CK7mbdMAKZRd0LYGuSc3Az4Lw7+bZi/KKYVXM84Og/PlYci3Kpyh64xHJ2zGu6yZsNnTWmr09JbpOxAN6bEkaDTYxNgbP37nVmq/xTHTX/w3UdVyWctRgErvgAI94f6snswQOMWLuBiWPagz5ks2ajdP/wdfnudelneAnNiOWcL0oOz+y6nyJOWdk/5Qt+Q0AjyrJjXYHKbBeUTgPr+UCIPgGOWE1tOIl70ax0E47skALTiOy11djy9h47gVEBFAf7oT9ooS61Gz3b6ksaK3cx459K1lIuu1DTwk8XyNuZAjUg7RlpV6inT9uWA535VgCGPk/yA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(39860400002)(376002)(396003)(366004)(136003)(36756003)(6666004)(478600001)(2616005)(66556008)(956004)(1076003)(4744005)(8936002)(107886003)(8676002)(4326008)(5660300002)(186003)(26005)(52116002)(66476007)(54906003)(66946007)(103116003)(316002)(7696005)(6916009)(86362001)(7416002)(6486002)(38350700002)(2906002)(38100700002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 05I7u65vtoCp6i8Up9kz4IzQK87XJfwBBWd/ZWaWlTxqIKoTzW/IYFrBQURQ6u7E9n3YWSWLh2F7a+hasOjzNex8zA7acJbAa+8xMde0Loj9InPxCtqHngGhvCnss0o8xr+CQlf+ekwCEiVdd5UT4frpqHI34ABAkwqIYdPlM1O250i8IzDmWdp3aIu1j3fVdqP5IEzlTybfGL8cVTQkc5U+B9e6aD7Ke4ecvryYLgfQ9Q7Pck7fS5IPqbvOsD5rku6bAwKNnfTar78xaDVx5sEd0oeGp6OCTdDzU7tCsf1wrVXks7QuEfveiWP2H7eBh8NQEhznbuWROIZK5IVn/OdtIohvkpjkezWToQEKLpjVyZrIWxt7efk0aNfaR4cP1TjlcoEt4xpE/fL/CrunDfDBkkqNOribP1aACa7zKCTAwEpwXKyaXHjVLIKKVWCeKza957TFzTmoDWrdKsxa59aRvkrI1W6w44gzd9GqoOG8ThxJU4fYz4WM15HdJKC6XXBrm78h22n0NJ0MTyfB2bTz38V/bL6s5TtrM7oYAlet5jHvltI43cnzG7pSH6C/3N/Mrk7cwh8TGfC7Oh9MDvIAjCXJynIBocu2IMv8OimSks0MyjjIUo1GPcwVx/saDWJ4gLIfKBojDejA8RYC7WVZzSWvqvUTBtycWUSxGcLSKTibX/Ad8grTtgy+FtLtqwy29U8TgCu7xDnCPGWZ57dJiygW0DF8BZSdWXYEsD8MKTA+4i3qg94p1KctT3vl5A3WI0QRIbnq4ECj9vdL/4XfmYjFXwDIR+IGNMv5WwO8H0ZPCz4BOhD443tesOuB6Vl5c39Tgp/J+8xwqj8IeY/lLcBab92KxFgflzGEgNl1RQFCE3ntJBAJmn7uOqE7GKrPx4T7IczjJvcZWdkMQjB+0zKrnUrnrraUgLweQfn4VKIjxLFFWOFFySOSFacanFzEP/Ga7mKOesBrzsVt87/jq5rXwjkGlDVVIos3loeT3XHBsV2lp+sIXgu6oofnAmMLIdbz5tqdWQDj2cS1XBp/2yPoWwsQmo9PRvHCYsO3CouNkgt8+qT9T2UeUiGRqDrj3JWBhxDzYrGNuB/5G4lMpWBKFFu1RJZ3IyB7MS0yItpBKnV9kAiTXjLDcF3zv2STCIZ2UhadF59PAqSJuf1+TJ8KzPIv5pOycIKW7I7N+d5XcBVlakFeF6aaIOaM4FGxzpg0kVErsdj/I33Uvo6/m62cnDFV/r57byFJMdNkuPG0Jttdvjm/dLFhgnW1KMnEB+fMThbaF6wL5L5daRidkmXoKo4J9QVo9s36qv73NpYy7lWKd6zY5hTDFGLC X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: a1e36061-a547-4984-37d8-08d946feabbd X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jul 2021 19:36:26.9495 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: tFAbL7J3fkDoCcP7Voi4ZaGwiiMq4JGcCtRWhkP55cYMTNcm/2/F3FyBPWwVn/vVygEi5GWZQq0xvUs+xUPmkyOeslLAZyRcVvW7TSl0tHY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR10MB2899 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10045 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 bulkscore=0 malwarescore=0 spamscore=0 suspectscore=0 adultscore=0 mlxlogscore=999 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107140115 X-Proofpoint-ORIG-GUID: AaPNpLEq8BgQMAAJpnhK4_2Ask5k89g6 X-Proofpoint-GUID: AaPNpLEq8BgQMAAJpnhK4_2Ask5k89g6 Right now, only static dax regions have a valid @pgmap pointer in its struct dev_dax. Dynamic dax case however, do not. In preparation for device-dax compound pagemap support, make sure that dev_dax pgmap field is set after it has been allocated and initialized. Suggested-by: Dan Williams Signed-off-by: Joao Martins --- drivers/dax/device.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/dax/device.c b/drivers/dax/device.c index 0b82159b3564..6e348b5f9d45 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -426,6 +426,8 @@ int dev_dax_probe(struct dev_dax *dev_dax) } pgmap->type = MEMORY_DEVICE_GENERIC; + dev_dax->pgmap = pgmap; + addr = devm_memremap_pages(dev, pgmap); if (IS_ERR(addr)) return PTR_ERR(addr); From patchwork Wed Jul 14 19:35:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12377751 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 645D56D37 for ; Wed, 14 Jul 2021 19:36:39 +0000 (UTC) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16EJVdqI009390; Wed, 14 Jul 2021 19:36:33 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=eItdugg6FcfLnkv9zYSPCFwM/aIL01iwtGtFBoCYk6k=; b=h/M1lzJP8vCYG+OUcJDkQxIxl9xK62qMeo5/sdoyt6nYPHFtGV/+twJv7z41oqW7Bw5G Ug22y4Zqz5udngQ1+NrKN5g8ras9a3P/w2W7kkRhdz6zRhHxqDivxh2XXNA+RJ/QnYod NtVRhawOpUs7L5xvB3dcdaMpUhybi0rdmxOXTN1rLdt88XWD0G6i6S1RJqLl+JAheukT wotz+6l45XXezRbGe7rQhZw6XCDMX54B9FTqg0FiwvJM5wto++zUwgtSMbS2Ffw2QhXN JUcDUvmxz50+m6rigu6/ylCiuEnCFttlyYd6hw3jvdxpafswZYaaERJVCkI+5hW1wvJ6 jg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=eItdugg6FcfLnkv9zYSPCFwM/aIL01iwtGtFBoCYk6k=; b=GYmvTbBj/D3Ix768TwaFJtlAzQt2+Jvdbkr/Yjm1ypSuTlT2jqnp7zElAc40PZe6y8If 15KITnEf2b3hhVXzQW3J+N5osr7UjrMKHPtRA35lV5uPGC3kGQvw0xHHJxVXEPBDGuHa mm5aw6PZkJeoLrxVBspt2kD5j4ZxQhg8Okm2mWGjGdb81vHBsaEiQ5HjC9hPZCcHZILJ qsPtjsSDtabPGrkQXFzBd+iylycpHr5SyoohnJMrbxs7bcRu5MjB2yVapi/p6LXSOL5b 86mpAhD468Vl8cZ5WyfzWQgiua0w5knyPAe6BgVTC4EMp/Zy2aDgOVU0DvyufAoerqFQ 6g== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 39sbtuk7v7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:33 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16EJUYri021568; Wed, 14 Jul 2021 19:36:31 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2177.outbound.protection.outlook.com [104.47.59.177]) by userp3030.oracle.com with ESMTP id 39q0p98mju-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:31 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GCqsbbYVIljX4YD0GbxVBU3v0ZlPTGzxsjDLCpfEPl+xCvV5/SUzD1davla6hR0s4Rsp9J5xGZ3QockYIV7HZ1k9KFNe+J56N01K8GXNuGdXircHaqxya4h4YQ749a7QSExtzPDUmpTribWMaeoEjWIcrL2FEeE5Hy6glGpxK+u1+btgxtusILKd+wIn2sMVVt0banbIHc2lZHnexbCw/PE5bU3KddqrErGwqqlNfShNkusim//wVcusPSnlVBPuZgehqvZgSuAu6dkydnN4qyxLwQH39sqMZ0KjcPt5pQVCdklJcsv+Cv/1EixOKIOMEPhRK4smnmaiTbfvH/j34g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eItdugg6FcfLnkv9zYSPCFwM/aIL01iwtGtFBoCYk6k=; b=i+Fe5Xi5u1dFG72Wfu8BOI3mYfy0YK6W0rjiFC669z0QcKrQzd0jtVr2LozIhAM9GchHP6iBcsV5WDahrSZ0KC0WIPMy5dITDb/WPBsup0jvF5P35CrsQqiIABqefno5/Px9VUbSKoelsJTACo5ydvFbBQpH4Gk3Zv6NZZr0HhkWiNfr+P8JlU9orQCGvcDu8k0Bb3wC/0gQgZuWwJFEaDcHsNU6qNYOwiijG8w6J6AD/ZnTBTKVKShWGrAWfAvwr5Wexa8A+v7zrm5X+jnhwMrgyjNx+SmLohAL5KP+yJfsKmCnEsTwfbdKQzLovwxkRk74O+kaS5hoDq6UNevK6A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eItdugg6FcfLnkv9zYSPCFwM/aIL01iwtGtFBoCYk6k=; b=Lr44uyilPxB6BfYrFIYsAcxt9jMTzvlqTNcgQTrqKh7bJ1/A0l6CPYccEgDLlOjOPTAe3eiHt1HfkZiYVx0laiwwBWAMRDal5XwreQoc5sr8zyhttqjSVvqgN7aieDdFqdIhcLFp0OfZn1LCZlG7Vlsqyqi2wWhOJKjBtCunTeg= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BL0PR10MB2899.namprd10.prod.outlook.com (2603:10b6:208:78::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21; Wed, 14 Jul 2021 19:36:29 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360%9]) with mapi id 15.20.4331.022; Wed, 14 Jul 2021 19:36:29 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v3 12/14] device-dax: compound pagemap support Date: Wed, 14 Jul 2021 20:35:40 +0100 Message-Id: <20210714193542.21857-13-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210714193542.21857-1-joao.m.martins@oracle.com> References: <20210714193542.21857-1-joao.m.martins@oracle.com> X-ClientProxiedBy: AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4331.21 via Frontend Transport; Wed, 14 Jul 2021 19:36:27 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 558d8404-a7d8-4653-ae9c-08d946fead59 X-MS-TrafficTypeDiagnostic: BL0PR10MB2899: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4125; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: uTmjoVPgyVOk3o9Pr/FaC3T7y366Or3KBz5OC/LqicVSiF4iWrmiCYVyL1Uq3a2mrEbE7Mc9ElfXnwPM0W2aS8lkz5g2bcjiaDpeGGNAfCbvTbH1meEz1xev0jLRItIft+u/I0t0kLwK9H4Oo4eDjZVDrBcb7ecv0QBfc5skvyMc5B86qkA2Ok9R9KrNE3TcRlNk+9ZYWBIrWEE2zGmwxUV9hXpBPVgsavysMWJznAeL0ZhwnTl6QmUssHbtJlFsjmX2KvyFDO38u3xyDmG6+OBVZikAVl+L54Y6HaKqfIkCzsAsHvBPa1bvg9eqKqaA6+Nrmw+KxONqRg2ynucTRfr5by61gwGZaDc11EFCeFF/MB84c1k5UU2SaavAr/IO0eo3h0VvztuuwLLSJRDFglS8oHL/NTI6E2/gnP4CB63cqh6DHkaj9ijUFmvAQ6CcChASLlosijxqMApM+fl0LttWsW3pyQO1TTbGq4U2TWh22/uRP0PaDR7R+TBUgsNQeZQ8y9VifMHMnyhkESczidoaoZ/7PZ8uSEjcCY05gOWnf6xpWKf81Z1Ru97AQ0xZowPhBpEP16A6xdWW3JzdyA4tP+/3mUiZeYZs1mq1CjPUEW3lU1TlpHceF9F9QxrnoJn/HVe9DN7fdMxdRjACDv7BqhIkiSkvJ2bFQlwyk0gVufX54A1XW7xdXdcncivcW3x4aV7oefLeBLTbGcv6Fw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(39860400002)(376002)(396003)(366004)(136003)(36756003)(6666004)(478600001)(2616005)(66556008)(956004)(1076003)(8936002)(107886003)(8676002)(4326008)(5660300002)(186003)(26005)(52116002)(66476007)(54906003)(66946007)(103116003)(316002)(7696005)(6916009)(86362001)(7416002)(83380400001)(6486002)(38350700002)(2906002)(38100700002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: wnAm+Nt5Y1Adg8vg5EaL7sw8p1gLCEaW08evj5J61CR21WIVvK5RnvNE0k9xkwucoRaQeHtnDzkvgQAYt269m2MVXIHTr7ZPhTDmVRSst9h4DB0YyjALtEK4KJm9EdNUXdCgfLbUM7AjKFRbT3FtfrbM8yv1xacdrpGLhNbwzhT1MlT7rruzINGYAjrDtAKqXAzem2pzX+cj04EBKDkAo/9M2Ro489SnKp17uZ5meYj1OJZ0r5Uva40X8h33AVLDeTriCdJcqMbWD43g9ygaue91y3Pz2h9xmu6pVBUvfIU2TkJCGKDwFgMgB76gtEvMeV8z3jeDL39k5LbfsGRE2RXA2KqlUuiRMkofyxP7EDDhaqfuWec45BmQYg7xiSDsApFoN7rfk5bj4qHZpa0ggtPdocFf4QUDaeFV2oUsawmhsl9/CCKMbToGGMTl/8VrYPhXt9x1Ih3SEKKWchdxi0q0IDo8iWksKG/TyPSeE0bkEPCwDvDngqtotDuaPsv5+hyBlvmhelCA3mPofmLHRklEk1Fa1ylxhJkE7okrNqkJy26Qw981dZtkbyzuCFsjrTd4wi2Hu2Rx3pk64Vc6QAIqhI2W69qpeZaBvDOqoeNWZbnh7RDJAqLQxglkicr/JaMA2qIaHNler3H3zbwSlOOstBVW2YTCgmTOgDe+ffp6yimPNwfEZLYWFx9CQKtP8+nesitJNo6+Yag/iJJmO0orHKSuVetQUiJHpmAkXvma9OMgCUt9J2rMjlYXYx2fUwvLwSik0vETJ4E44+fqwY/abZ57c9gbl/OH/oXb5039ERloO0Z7pcDZm7xCbNUiLFSjHN4L0VY9Xn9676dK59vK4i2T48+NDaXqhAbGLzsx8Icp0k0maeQh7B2h7W7/CS0Ki6ohS1miTk/3RgPr+W5GsXwsJtXUNSOqsQQf6mFCb9DS3waLRci0c6kUPx12YGz1RsYhEcHXVtaYgDQe4mCC7ELsdJ1JMqr5GspY04cEyFObpDbaGgDOYJrtExoyFTbRaO9VRKpwenOJY+J5H845a7mqRvbhLdJ2fMK9MrQhS8NaJS0JaJ8qnVM+DfEGhCU3DpHPFMICrfzHvkPbGGc5lP4gKdGOCpDaTB7NUc2n/IdpWwrVmP+jS43llIuZbyHDNc6RyZy6tTBIUSntLP8VLYvnsn4uo9qJ3n7txYRQpBOrk08/duf5R/5QkAtESjFKvVScRAP7oAMhq6ISpuBg9VfLXOgrWMPjPR53C94H58i3aN8N5R0/MNY/2cQkO/MJBD52h88ICNJAu1I678/Cs+4SxVuON9bkgT1uveR36ZqJT+v/IyDRSeOT0YzI X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 558d8404-a7d8-4653-ae9c-08d946fead59 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jul 2021 19:36:29.5470 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: /Cu86l3rJLakhIe4cPDFRYBEAi8dmO9+X404t1Rkw1r5u8bbPFximFh2OICvOZM7BEYAfqUocmD/amGDdpaw90/xsMR1fL3MpJTAMMiC0dY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR10MB2899 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10045 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 phishscore=0 spamscore=0 bulkscore=0 mlxlogscore=999 mlxscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107140115 X-Proofpoint-GUID: 5fceide8n79df_Kr4Z6nsvPLTE_LhNtx X-Proofpoint-ORIG-GUID: 5fceide8n79df_Kr4Z6nsvPLTE_LhNtx Use the newly added compound pagemap facility which maps the assigned dax ranges as compound pages at a page size of @align. Currently, this means, that region/namespace bootstrap would take considerably less, given that you would initialize considerably less pages. On setups with 128G NVDIMMs the initialization with DRAM stored struct pages improves from ~268-358 ms to ~78-100 ms with 2M pages, and to less than a 1msec with 1G pages. dax devices are created with a fixed @align (huge page size) which is enforced through as well at mmap() of the device. Faults, consequently happen too at the specified @align specified at the creation, and those don't change through out dax device lifetime. MCEs poisons a whole dax huge page, as well as splits occurring at the configured page size. Signed-off-by: Joao Martins --- drivers/dax/device.c | 56 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 43 insertions(+), 13 deletions(-) diff --git a/drivers/dax/device.c b/drivers/dax/device.c index 6e348b5f9d45..149627c922cc 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -192,6 +192,42 @@ static vm_fault_t __dev_dax_pud_fault(struct dev_dax *dev_dax, } #endif /* !CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ +static void set_page_mapping(struct vm_fault *vmf, pfn_t pfn, + unsigned long fault_size, + struct address_space *f_mapping) +{ + unsigned long i; + pgoff_t pgoff; + + pgoff = linear_page_index(vmf->vma, ALIGN(vmf->address, fault_size)); + + for (i = 0; i < fault_size / PAGE_SIZE; i++) { + struct page *page; + + page = pfn_to_page(pfn_t_to_pfn(pfn) + i); + if (page->mapping) + continue; + page->mapping = f_mapping; + page->index = pgoff + i; + } +} + +static void set_compound_mapping(struct vm_fault *vmf, pfn_t pfn, + unsigned long fault_size, + struct address_space *f_mapping) +{ + struct page *head; + + head = pfn_to_page(pfn_t_to_pfn(pfn)); + head = compound_head(head); + if (head->mapping) + return; + + head->mapping = f_mapping; + head->index = linear_page_index(vmf->vma, + ALIGN(vmf->address, fault_size)); +} + static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, enum page_entry_size pe_size) { @@ -225,8 +261,7 @@ static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, } if (rc == VM_FAULT_NOPAGE) { - unsigned long i; - pgoff_t pgoff; + struct dev_pagemap *pgmap = dev_dax->pgmap; /* * In the device-dax case the only possibility for a @@ -234,17 +269,10 @@ static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, * mapped. No need to consider the zero page, or racing * conflicting mappings. */ - pgoff = linear_page_index(vmf->vma, - ALIGN(vmf->address, fault_size)); - for (i = 0; i < fault_size / PAGE_SIZE; i++) { - struct page *page; - - page = pfn_to_page(pfn_t_to_pfn(pfn) + i); - if (page->mapping) - continue; - page->mapping = filp->f_mapping; - page->index = pgoff + i; - } + if (pgmap_geometry(pgmap) > PAGE_SIZE) + set_compound_mapping(vmf, pfn, fault_size, filp->f_mapping); + else + set_page_mapping(vmf, pfn, fault_size, filp->f_mapping); } dax_read_unlock(id); @@ -426,6 +454,8 @@ int dev_dax_probe(struct dev_dax *dev_dax) } pgmap->type = MEMORY_DEVICE_GENERIC; + if (dev_dax->align > PAGE_SIZE) + pgmap->geometry = dev_dax->align; dev_dax->pgmap = pgmap; addr = devm_memremap_pages(dev, pgmap); From patchwork Wed Jul 14 19:35:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12377753 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9D15D2FAF for ; Wed, 14 Jul 2021 19:36:40 +0000 (UTC) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16EJVe8R009412; Wed, 14 Jul 2021 19:36:35 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=V4tl+Eg71XG44E4s8uD07Q6GrHzl1AUOqv3jJcurla0=; b=pcIKmdfIFwwBUlUlfSdHT0LwELVa6x+yhdizAA4peqwwu9hJEQKnZvFIw8ULHq1alg4R Lt/a6SxIjM/pesH5WUv2Qr7nJSQhMo5hbkdA97TNesEyR9cVCMNBxNBFRJF1PH9zA4nt 3aCnCoDwXsV0JBW/bopJNd/Ct9p4dVxM2/QQABSi+Jp2JXDIdGMk3oAFa/mKbzMSTBiV eU2eGnbD8zY403iMQje02M2YITZCV7rOJFpONzxP06tagyxA9NrvA/rEEn4a8Mw/lhZe cMbK+oNojDObP+77fbtXQGqkYlFBkPeq3717OiH3OE38edk+7sP4MmxGu6fjqK3V+AwR MA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=V4tl+Eg71XG44E4s8uD07Q6GrHzl1AUOqv3jJcurla0=; b=LFdg5Wp0g9iMXHetwOm8de4wgCp1V7RS2edBVa9VMCF/cJTQpzfbACM/ttfadtzL5Jjt tBNOidtj142ifuIbgnGDw1oncEhPQ1ChfPfyMROmIJkuq1P+YF3LGHAZv8PajT+fS9mX FIJ5Gml1bxyz98VwM/TUEE7/T2aejt4mS0U9XXJaGRMcw19blnyFVwNHGISMiiayas4Y QtHtEb+8CBACjn5lcX7H+kn1zH30N3uO00ndGjWJmdaKT6H6qkzjRAKlGai1+lVLok8K ysvPQ/AvNNllt2svGsg1w8fUMZr4QNP+wvr0qGqU7VIaIeVRqoxpy3i8dqSWNznuOqeC nA== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 39sbtuk7v9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:35 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16EJUwFG187358; Wed, 14 Jul 2021 19:36:34 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2169.outbound.protection.outlook.com [104.47.58.169]) by aserp3030.oracle.com with ESMTP id 39qyd1c60m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:34 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JPJ0+SgyWIK+J6xGTZAtg+hUR/Inll4KQiI7Q8smCzLsRu0vPnX1rZZN5MreK+1wIedc5iQteO6ilmNdy09kGcr8d0154sK1dT74FATAgX3OxsEIctOUpUDGigewO5Q9HzFWd3H6vg4dcUtADm2d8Gtv7fkQAHnu6Hdww8QzM1hxqMBAXEhyJwRzUpyVJh6FV9fCgBML0QjvY9AD50oxXyiePR38FSrJbYP+GKTNOuU76Wb0q3pr4/A6vV5MDXdouxbbDh9CprI9kY/4j7Vx/DMSeHbiYUca9SR4t8UANY0CL7s451xJiEm2wMJZO4qoxDMWFFZx/0hrWJew0q5pRw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=V4tl+Eg71XG44E4s8uD07Q6GrHzl1AUOqv3jJcurla0=; b=O2nSQKZ6w3lzSs3/yma/0EbRAZDlTkJcGIkqTxk7LCwL2GITlZHSwVN8xXFkEYdj6v98+AasSTNceznXR1NflQd3Xbtjc3Q2Yeuegl2K5Iv9+f5YDgCZxFcMubZ6HGncsBLQka6T6a4jMIeZsLcudEGyyYoCJG5jS0rfT8rjGWhBhG16Qf2pqwYWKP+an1J0FFTclsGA6wUyUa2HyAGr7MkTaxch8JwAr9jYEJP2bPnQTBXGib5IU0q+Zx8xkTEZ+lByCMkdgL/7OJUgrES9ZtYFjzxcN1RUC5gCZtvaaMmqhkodBZGbuWkN3EAR5i0baYmcIjMg1cKIyLDMOuP9kw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=V4tl+Eg71XG44E4s8uD07Q6GrHzl1AUOqv3jJcurla0=; b=ejvsZDR4MkGXHoRe4kNFAUKtvc12T96RV8a2tSDR+sR18dTGyLaqr8miIjypabMDRT2uc1kEXr8W/iYJgWpNdFZVc+Y2aw+TKNdFs3BNmJDGtq905NI8cwdfSj0+huOpMu98jEqq4BWeyTFK2HsZXv7YaWOYnlmD9o6SfwsfA30= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by MN2PR10MB4368.namprd10.prod.outlook.com (2603:10b6:208:1d6::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4308.22; Wed, 14 Jul 2021 19:36:32 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360%9]) with mapi id 15.20.4331.022; Wed, 14 Jul 2021 19:36:32 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v3 13/14] mm/gup: grab head page refcount once for group of subpages Date: Wed, 14 Jul 2021 20:35:41 +0100 Message-Id: <20210714193542.21857-14-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210714193542.21857-1-joao.m.martins@oracle.com> References: <20210714193542.21857-1-joao.m.martins@oracle.com> X-ClientProxiedBy: AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4331.21 via Frontend Transport; Wed, 14 Jul 2021 19:36:29 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2b0bd4e0-b5e5-4efe-cdd4-08d946feaef2 X-MS-TrafficTypeDiagnostic: MN2PR10MB4368: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2803; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: e8HRuVsY5GL5emEH27DYgZtX4kPTweEDwCa40TY3MMQJ4j4YKRB2NHBPTzQ84AYzLSPatkJroLT20owCEodnFd8xij/lFW9EBnGGUvfVB1JNTrj2uQw6ixpkQZ1kuazfJgvNcZD0ZPKc+n9vMAsDQ1LpKjlSXQcL0d5Xycr/WCNCkq/lGqvlEHGmPKT8aovUqY/TtgconOi8V70h/FBDDuLqH5/YK1mTni3RVZicK7TB8IyRXkZx2FFvC4oNxEvKX12bWu54dVUBgloIzxxtXbOMBBiA+e0GA34eUtgoqc1jF/2pmMvbQpd9Gq+U+rqUEfabdU4mtvKy8Kl/POojtjGrsEI3lRcBiWAjJQm6a3jMNAfvlsOfP5EAG4VrbCohWrpJJ3MjH/e0LR1DGZmKOVUE9B5Z5DyzNtUzLMXPPdUf5zxqrjTZSql6rAYL9op3oN+xA1JFFF+07KiXRnRmZ0kVM8dlYfl2WTz77r+olatEdhQ4BVAFRxeGP2GBqAap7ReZL0DqPGUDdvItHvuQojof4k9tBCSuJgYZgV0czwKWh8687hw1eVXBtn9PmDJ92priyED2Yd9KFEmT0dPaWVUkPJsF1J4k1dpYk57/lf1Alv8KW82/6W9DvptLBak3JIRBnDtA2YTtgyFEDnwaNbsulLvNqNEeGh7Ivzg+FpggPH7/95hLeby+zm7yKZL/hIotU1O7Jj6gHvlFLotOeA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(39860400002)(366004)(376002)(346002)(396003)(7416002)(26005)(6486002)(83380400001)(8676002)(107886003)(36756003)(6916009)(54906003)(103116003)(8936002)(38350700002)(2616005)(478600001)(6666004)(86362001)(956004)(316002)(186003)(4326008)(5660300002)(7696005)(2906002)(38100700002)(1076003)(52116002)(66556008)(66476007)(66946007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: jG3IFEMbikOydADuFxKSrCoBpZQt9bEDVhllKvYR5SVxV/jPoUTqBp/itwvJBloFdZ7zYUCbM4L7Rd/o3DTjJFRquWzH6bewG6NFtWjkTPG+NsrpX3LXI3dgfgd/5osMy6kExBwiu/If6we6ASrTVyxHMAof/FGhlQhhqJvsc5aKvV7o2i1AD6D6BQzkWMuUWnaPcplnQzUO47xA9awML0vjFjDcplFvlKaID1bvZ+2JlqmxbhNxkkiL29mdyox0PTwNLDQ6BJ4bPJNRePixkBrR3o4llYcSfbuBxE/AB3jj/3RlKMiD9ehYmECuuZIo6c/axwdKFE4NL/lbQNw+S/bAelNOPY0fyRU8fIMtexdJ/0NRXhAyPobaqOy28B62wjqziUlwJsc0/MKoxqpW9SDNMNLzTqWb2mdiYs+W/FEuJJspO4GmW/BYrQaN6LVz9m0UUNUhA3TC4I4tQFheBoV59RcwvW4pP0+koCFonTUpD5Xf1gWWpWZOf6XShw9EwlGV54PVYezLK+pg524Eer4V7Xnkn0HDPsZd3Hhvbg8Y7965vfumXA7swvMIJSip7janfktVVH3OVXmDtcU/boKeleFWKBbfYGQKnerWa65BOwzGOk7QdDsgs082dqYWvG7EsVn049RUsFDNhhUUJZ9BulBwCDlNdVL6Ag9JL9RoEjurnKVhEX/WCZmU40YiPx+ZLfi8b7FISbgLzWFflQ7VVcc7SC3uWx204loqS6BYtSWHkg5CUHA/y+QybbClLOYC/gTnPCZIINr/1p/dPdeImYIZOV53/LZxkw0Gm5gvAxOcOqGquKG8XFQUAcJET0BuhQI+iHHxlbfRKcejO5rI8bnoaQhhBDnVq5hObCiLZAiTETipqsL2jCST0pXZ9f1g7U1O4hv5zCBeRyGWPBrgShIfKxRBZqVJjEWA86kukjPLO7jv/OZC0YZG6i6m/pCQOwZTYQlaECv3GtPS+KdfwqKlJ9DO8p7Cz8tAQqTPu/YK/IgfRu0e3QxxyOQqgDWm5+H9edYQViOsrN1nYtU+ApRHxxP7q3oKEzyr5t+W6BcZRyndlf3pLi/GHrX4e8VURs4p0ORMEWBuci0lXT7CEj/Gqkz6vepIEKin3+jHpc6rfAPPm8RpJFt8Sq2McXgOBL7Fz/cSil1FcHeFTH8g865ufJGZAkri2k+DllkbaHL0spxnnHeoGLV0cHV4Z362J99TZtfJ5F+hmL6XciDn5wWxTr1BSX96f6u29HlisxtlpzREtfy7ab3B5T4xQjvqvQVvaKsylz+Kcm9nHriaGO5fxrg66Dfb/2g6170/Rdtr4OobbwnXJygUL89/ X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2b0bd4e0-b5e5-4efe-cdd4-08d946feaef2 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jul 2021 19:36:32.2820 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: eLgvzCOqrYVVBQ0OTizkTuuWv3cVbi/W4KSdByhneSOLnnQIpNQInGESJhnL3t4/6OOcnDaCYwjYRauXCi/M/vjsw69QQgS+C3rczVN0Jy0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR10MB4368 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10045 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=999 adultscore=0 malwarescore=0 bulkscore=0 mlxscore=0 suspectscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107140115 X-Proofpoint-GUID: Uc1CyPBQq8n_-8K54zU-beUugqfXnrk5 X-Proofpoint-ORIG-GUID: Uc1CyPBQq8n_-8K54zU-beUugqfXnrk5 Use try_grab_compound_head() for device-dax GUP when configured with a compound pagemap. Rather than incrementing the refcount for each page, do one atomic addition for all the pages to be pinned. Performance measured by gup_benchmark improves considerably get_user_pages_fast() and pin_user_pages_fast() with NVDIMMs: $ gup_test -f /dev/dax1.0 -m 16384 -r 10 -S [-u,-a] -n 512 -w (get_user_pages_fast 2M pages) ~59 ms -> ~6.1 ms (pin_user_pages_fast 2M pages) ~87 ms -> ~6.2 ms [altmap] (get_user_pages_fast 2M pages) ~494 ms -> ~9 ms (pin_user_pages_fast 2M pages) ~494 ms -> ~10 ms $ gup_test -f /dev/dax1.0 -m 129022 -r 10 -S [-u,-a] -n 512 -w (get_user_pages_fast 2M pages) ~492 ms -> ~49 ms (pin_user_pages_fast 2M pages) ~493 ms -> ~50 ms [altmap with -m 127004] (get_user_pages_fast 2M pages) ~3.91 sec -> ~70 ms (pin_user_pages_fast 2M pages) ~3.97 sec -> ~74 ms Signed-off-by: Joao Martins Reviewed-by: Dan Williams --- mm/gup.c | 53 +++++++++++++++++++++++++++++++++-------------------- 1 file changed, 33 insertions(+), 20 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 42b8b1fa6521..9baaa1c0b7f3 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2234,31 +2234,55 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, } #endif /* CONFIG_ARCH_HAS_PTE_SPECIAL */ + +static int record_subpages(struct page *page, unsigned long addr, + unsigned long end, struct page **pages) +{ + int nr; + + for (nr = 0; addr != end; addr += PAGE_SIZE) + pages[nr++] = page++; + + return nr; +} + #if defined(CONFIG_ARCH_HAS_PTE_DEVMAP) && defined(CONFIG_TRANSPARENT_HUGEPAGE) static int __gup_device_huge(unsigned long pfn, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { - int nr_start = *nr; + int refs, nr_start = *nr; struct dev_pagemap *pgmap = NULL; do { - struct page *page = pfn_to_page(pfn); + struct page *pinned_head, *head, *page = pfn_to_page(pfn); + unsigned long next; pgmap = get_dev_pagemap(pfn, pgmap); if (unlikely(!pgmap)) { undo_dev_pagemap(nr, nr_start, flags, pages); return 0; } - SetPageReferenced(page); - pages[*nr] = page; - if (unlikely(!try_grab_page(page, flags))) { - undo_dev_pagemap(nr, nr_start, flags, pages); + + head = compound_head(page); + /* @end is assumed to be limited at most one compound page */ + next = PageCompound(head) ? end : addr + PAGE_SIZE; + refs = record_subpages(page, addr, next, pages + *nr); + + SetPageReferenced(head); + pinned_head = try_grab_compound_head(head, refs, flags); + if (!pinned_head) { + if (PageCompound(head)) { + ClearPageReferenced(head); + put_dev_pagemap(pgmap); + } else { + undo_dev_pagemap(nr, nr_start, flags, pages); + } return 0; } - (*nr)++; - pfn++; - } while (addr += PAGE_SIZE, addr != end); + *nr += refs; + pfn += refs; + } while (addr += (refs << PAGE_SHIFT), addr != end); if (pgmap) put_dev_pagemap(pgmap); @@ -2318,17 +2342,6 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr, } #endif -static int record_subpages(struct page *page, unsigned long addr, - unsigned long end, struct page **pages) -{ - int nr; - - for (nr = 0; addr != end; addr += PAGE_SIZE) - pages[nr++] = page++; - - return nr; -} - #ifdef CONFIG_ARCH_HAS_HUGEPD static unsigned long hugepte_addr_end(unsigned long addr, unsigned long end, unsigned long sz) From patchwork Wed Jul 14 19:35:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12377755 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1845F3406 for ; Wed, 14 Jul 2021 19:36:44 +0000 (UTC) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16EJVCjN028294; Wed, 14 Jul 2021 19:36:38 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=u9glgLq7pr1iwNPx5tMgAhYw42aywrTzKkFbLrYBLvM=; b=RDz+SNmpXTOof7iusbYqoX+FYb2sSbs0TabV4M/o7imVw7lO/tzkeWrsFl4yqF9RL6An Ylf/68Gb3CANWKZCdtxwjbOELCq6kc/fLzOf9rIbR98pz2nufbrXS2JGSsBnL2wtxqlr BIYtJpXXCgcFWCYmBv7uJ0J4C/+xt+jQ2a5sSzGsuFu+9RsZlWebYW74MpmM5CD3jSXk lo+bZbCGnju4RAnYW6/HTYMxgtj9riSO0cO78d2nOySvdihgRqKIwNFo3q2oqt3+6BCM a++5nVflBDW9RSvZmNzSoIef4NZoTtIDGAPA0K6X8tkRrCBPpvdMBq+09Ptr9okc0LRz KQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=u9glgLq7pr1iwNPx5tMgAhYw42aywrTzKkFbLrYBLvM=; b=gfD+AJW3VEvs6k1s9ZfmtEAgvwkXrJpL6KW/RVXOXnSAA4Cgbp+rftAQ6GgC0Lhik9Ny wuH0iodv4UbT1FQMpwvMcCZvN+VLRnq124citf0GiaBy5ND4bQfqdl5J9JxyNJb/zw8p 1iwfLNuhPzMBvdyVYhnrg3OfPvS8dyNtO2nMsCMdNO6C/+wPsKlMzQbV//7I8MYOv1wd xKgyHzhCY1Ne8NMzP3LeLatZXqGiInL69L+iBeLKIRsWpwbs8dW8Akgi5FlmlKfvawxK AQO3Ap2kOeEeN/44vmHoN6MWwXoc/92VHBxXgITrymvBdFhU/XlCS5ZimzM2/mB5YOxP kA== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 39t2fcggtf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:38 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16EJV1WH079823; Wed, 14 Jul 2021 19:36:37 GMT Received: from nam11-bn8-obe.outbound.protection.outlook.com (mail-bn8nam11lp2176.outbound.protection.outlook.com [104.47.58.176]) by userp3020.oracle.com with ESMTP id 39qnb4295u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 14 Jul 2021 19:36:37 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TgHENo35TX3jpeNVQdLq0XMjnpfmikEylHhXhZQdom29NigUOIRi4Y/NRUdeKW723jfyhxwim9/Ed997AW7Otxclcbr2gFWTbzRDDUO0Lp7sZmiFs/3jhhooXlBwnB/TkMc0nLSU8Z/szQ9gd9qoWHHwDX+8PCmbtYpqF2TX9oI2L4zW+zRhqD1XBLB8QNHZt6M4/Gq/hPUhduSg69Iahwcz5Zl2rG5cKRnsMExEPE+ucKvZ5UAZcQ/4dJELcfDlHd14ED3B3iH0JjP4Tc/eWIlubE+RKU+ZqHwrB5MDu89IFQ2YNXveVmxUn1xLVarcP1IC4ejiCAROYwtiZY4DGA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=u9glgLq7pr1iwNPx5tMgAhYw42aywrTzKkFbLrYBLvM=; b=Qu43Pqp1reyK5Z7+uQqmJXWTjzqtI0xtFYy7CglODS6NcXwqsHktY1M2hILNCH1M0Nv4+kyo2OVFU5PI6aJOWz1P/Kb4O1Phx9Yh3iVI6tGrJ1fA6QllfgC11Jgsttmg/k88CHrQL/6KvHTSPShf9Ku1dOGjc+IrC9bLNMDdCIaEjkZiM+YgsEYbutxq3cq6iWe2T/+jJBpOdlkKRUH6gm+CawzDGPuUUX3yXihsNszCBNQ2PrYiDoeTD+r6KVZAanTuYFcYObwCrRQAi09l4yfpq58OsCefjeFbYKvAn85GRQ9d7aOHWiDo9GA3NSUEm8/HlvkfUxSj3489G9cdGA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=u9glgLq7pr1iwNPx5tMgAhYw42aywrTzKkFbLrYBLvM=; b=RpOx2DJbhaejp8nkKFiBfsC7uxhLthvJXBpp3R37gHmVoC+bbD9df08xnfS4bfGSelcPj5YzlJ38qW+5xgqZkdV/cy1ZalZYQJ3siVhvEyACbL+tRqtWDdC1Q6wuxZK0pAdScvbsOOol4/OXxRGpX7jWqja2DJjw5fHbmvq+TnU= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by MN2PR10MB4368.namprd10.prod.outlook.com (2603:10b6:208:1d6::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4308.22; Wed, 14 Jul 2021 19:36:35 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::5833:5ab2:944c:7360%9]) with mapi id 15.20.4331.022; Wed, 14 Jul 2021 19:36:35 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v3 14/14] mm/sparse-vmemmap: improve memory savings for compound pud geometry Date: Wed, 14 Jul 2021 20:35:42 +0100 Message-Id: <20210714193542.21857-15-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210714193542.21857-1-joao.m.martins@oracle.com> References: <20210714193542.21857-1-joao.m.martins@oracle.com> X-ClientProxiedBy: AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by AM4PR0101CA0048.eurprd01.prod.exchangelabs.com (2603:10a6:200:41::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4331.21 via Frontend Transport; Wed, 14 Jul 2021 19:36:32 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: a8d7434f-df70-485f-21a2-08d946feb0a0 X-MS-TrafficTypeDiagnostic: MN2PR10MB4368: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8882; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: h6cByBZUn4grEZYs/ArtHeOv29eJogEYe0601whyx0iWeXnHiqgshruV/pbW4ysnfmHi2itVbpaDOyWjJejtAh+SLoX9b4e/WfwoBWvZ+wISGgEWUslEtdfEi9jt5FcNtaE8AerjSjLxchpf+zgo6oXz7FM7dfOdDJb9qsfvn3jcb6yK5xxyi4dSUbcHqwo+5sCZLHK9Ib1NTS6ZbFnY35/yzPY9WZqaG405QQcF+XD0rdgxgTpXmDPIZW7YBnBWyXn3ps3MomHYy4p4oP5IqhyyYX0YklUHkpscEznlqUyG4RSywaaTpxByi4E9ZgtqnFEi/ExagYkaH8urb/wOie6BV51ZnVKj8YGPeym31RQMQNe0EFBthsBUlwmrJMURJddi9JKzn+R3HGx0A1Xq5/IYFszjw2Q2cPanvsLJykOIPqICgoU7dVib0ImJzww7v0G4B9fptsPIijIfekoqY+m7WMG4pozfZlffQbJazQXQWoNAlqN/xasv4rByubWYhJL6AK9Y5XGbty3XojWAMLt+277bpH7I7C3SGUxZsTCt/q5Iv3Jhk1xxvUyu2k0w4nhwoFirSyeJaP60UT7HZOxpdu8lqzZk8j+oO6pTZaUxio8llLLfVNNXWOI4BPovXnBRu1b57WPYXnDUL0s/cXEqTgCLQO3Gu7S629I5mYs7lQTmfg99xynDX8MIfgP0QHIdqGPr/eAnyYtP3SQHA5BuSdRc4v3p7HXPYQRZ0hk= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(136003)(39860400002)(366004)(376002)(346002)(396003)(7416002)(26005)(6486002)(83380400001)(8676002)(107886003)(36756003)(6916009)(54906003)(103116003)(8936002)(38350700002)(2616005)(478600001)(6666004)(86362001)(956004)(316002)(186003)(4326008)(5660300002)(30864003)(7696005)(2906002)(38100700002)(1076003)(52116002)(66556008)(66476007)(66946007)(25903002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: pWKYz+7skNHdxxv+AyGwwLQmLzBu773yNr8H24ziKuTiENaALbo20lt/St41da1WndDacAnUbfHTViwGTyKddi0lQ8UA74Z11oNho7BFtyA2jHHFcidG5GeKW5eDuk5TJ7OH2FMOr25s4eJT83nDHCrYj/k2XtGh3VZrBuwzWtxkHREh3xt9T/Qrw5H7HRsk7p9unQ2kksOWj3vmsZcVCORMmmq2rONAkxdJvDH6ZBjWRx0/Bslyu9eAycL9R7oVoxDQlnX1+0icad6tCEP0w7FYtXb9lyyQMooOJFSJBTpi5jsOiTUqNtmtGctR586ejZyIuoF9AReL9whEWpzhIZUmQjDfHTtjYCbjDkap3IywnHLlXztxisC30oLPHpsNwBxXheqD/icOpvLAq88FFwnkgn5UnSDNHmoWhqiwBpY8lpkH7y/0xgOHl2Sqy5Fr/eRt3QuRDHJwja+ONoObNBA+/uPo2ptuURN93bhBshWnyumaWYAeq7WMsHIAyBHhpjuQLZpDe9HqZyF/b6ALTAuKWC4Sa+i7XDd1HiAP4P5yO4yglOfoDwUAVrsEKr+UMaQpoO7+hA1lvoHjy8smkmJAZPepwHy61bI8BfhxtthHxWKFJJmBGUl1pwyaXmXcG2SKDt2WFNHkR4R7jFQK3088V8HW6gyqh7rGNs/2e+RZc/yEdYb1Jc6QMr2ht/3M+8PBaWg0xTBj2ZO4o29km+4If0+F+g63hYoo2Os7nv/QQPB+cVZ1KCMCXnTE3ZhD8WlgMa096NLVAuOylRECGVsRGw10zMukuNNTNujghS0r6su3e4YXQdkLeiHHoYwPDRlZpo/n2OWF+7fQ5SHJVvvIR+JctMLlZ8eEnDmFUCDsnUyY5t1EF2iaF6wOktlTYord0QookR6dq4jeRAd6goXGLBz73rRVtLrWKgtbrP+SBKMTEgJ8fyEUjIjUVZz4hHLtHmdFMhFMcmFNuK32wFkMBBUqzI60VlD/loPUpIV47rUfdGjkzuLgTqng45S3HkE2umsbvIE9f3O0CjP/fpljeq2UoSV6ZiOtQytbaTEDVM+rBGlSakU0GRZDzRkIKttI4axz7zSgqWRj09hoMeoXAhg8rho4wpDU9EHi8YY9oXXpzX0JT3q3r/3rnF+ipE2B2NJkY/PcyDO3DxVrBjCvdsnbFxVPOj4XwJ+XoGpKxR8itkazyB7MMwJzRv6+/nr3MvuXc2l7PzDqu8oirz8MGf07Q3OSpXzIJX3Xj9KGKwvh/jCZ9GvQi7dPQe8egkIvE6Q4bnk9k1X+SEH4nNDHvO/JREJJch6nRjyPHeUoigs9VF/VXdWJEpM7S2VD X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: a8d7434f-df70-485f-21a2-08d946feb0a0 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jul 2021 19:36:35.0658 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: nf6b1hhvxs45WOvvw8u+HOojSyxREvaMs6Fkr4puGWMIZSkEvAhw+UMLDJ6nEvVTyxN1L2xNTsgDheqi+lQKClxl9cNGU3taDZyOTTtc4x0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR10MB4368 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10045 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 bulkscore=0 malwarescore=0 spamscore=0 suspectscore=0 adultscore=0 mlxlogscore=999 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107140115 X-Proofpoint-GUID: Cps0hVHBooy3oufbcb3KKALHaJ2cbJZ1 X-Proofpoint-ORIG-GUID: Cps0hVHBooy3oufbcb3KKALHaJ2cbJZ1 Currently, for compound PUD mappings, the implementation consumes 40MB per TB but it can be optimized to 16MB per TB with the approach detailed below. Right now basepages are used to populate the PUD tail pages, and it picks the address of the previous page of the subsection that precedes the memmap being initialized. This is done when a given memmap address isn't aligned to the pgmap @geometry (which is safe to do because @ranges are guaranteed to be aligned to @geometry). For pagemaps with an align which spans various sections, this means that PMD pages are unnecessarily allocated for reusing the same tail pages. Effectively, on x86 a PUD can span 8 sections (depending on config), and a page is being allocated a page for the PMD to reuse the tail vmemmap across the rest of the PTEs. In short effecitvely the PMD cover the tail vmemmap areas all contain the same PFN. So instead of doing this way, populate a new PMD on the second section of the compound page (tail vmemmap PMD), and then the following sections utilize the preceding PMD previously populated which only contain tail pages). After this scheme for an 1GB pagemap aligned area, the first PMD (section) would contain head page and 32767 tail pages, where the second PMD contains the full 32768 tail pages. The latter page gets its PMD reused across future section mapping of the same pagemap. Besides fewer pagetable entries allocated, keeping parity with hugepages in the directmap (as done by vmemmap_populate_hugepages()), this further increases savings per compound page. Rather than requiring 8 PMD page allocations only need 2 (plus two base pages allocated for head and tail areas for the first PMD). 2M pages still require using base pages, though. Signed-off-by: Joao Martins --- Documentation/vm/vmemmap_dedup.rst | 109 +++++++++++++++++++++++++++++ include/linux/mm.h | 3 +- mm/sparse-vmemmap.c | 74 +++++++++++++++++--- 3 files changed, 174 insertions(+), 12 deletions(-) diff --git a/Documentation/vm/vmemmap_dedup.rst b/Documentation/vm/vmemmap_dedup.rst index 42830a667c2a..96d9f5f0a497 100644 --- a/Documentation/vm/vmemmap_dedup.rst +++ b/Documentation/vm/vmemmap_dedup.rst @@ -189,3 +189,112 @@ at a later stage when we populate the sections. It only use 3 page structs for storing all information as opposed to 4 on HugeTLB pages. This does not affect memory savings between both. +Additionally, it further extends the tail page deduplication with 1GB +device-dax compound pages. + +E.g.: A 1G device-dax page on x86_64 consists in 4096 page frames, split +across 8 PMD page frames, with the first PMD having 2 PTE page frames. +In total this represents a total of 40960 bytes per 1GB page. + +Here is how things look after the previously described tail page deduplication +technique. + + device-dax page frames struct pages(4096 pages) page frame(2 pages) + +-----------+ -> +----------+ --> +-----------+ mapping to +-------------+ + | | | 0 | | 0 | -------------> | 0 | + | | +----------+ +-----------+ +-------------+ + | | | 1 | -------------> | 1 | + | | +-----------+ +-------------+ + | | | 2 | ----------------^ ^ ^ ^ ^ ^ ^ + | | +-----------+ | | | | | | + | | | 3 | ------------------+ | | | | | + | | +-----------+ | | | | | + | | | 4 | --------------------+ | | | | + | PMD 0 | +-----------+ | | | | + | | | 5 | ----------------------+ | | | + | | +-----------+ | | | + | | | .. | ------------------------+ | | + | | +-----------+ | | + | | | 511 | --------------------------+ | + | | +-----------+ | + | | | + | | | + | | | + +-----------+ page frames | + +-----------+ -> +----------+ --> +-----------+ mapping to | + | | | 1 .. 7 | | 512 | ----------------------------+ + | | +----------+ +-----------+ | + | | | .. | ----------------------------+ + | | +-----------+ | + | | | .. | ----------------------------+ + | | +-----------+ | + | | | .. | ----------------------------+ + | | +-----------+ | + | | | .. | ----------------------------+ + | PMD | +-----------+ | + | 1 .. 7 | | .. | ----------------------------+ + | | +-----------+ | + | | | .. | ----------------------------+ + | | +-----------+ | + | | | 4095 | ----------------------------+ + +-----------+ +-----------+ + +Page frames of PMD 1 through 7 are allocated and mapped to the same PTE page frame +that contains stores tail pages. As we can see in the diagram, PMDs 1 through 7 +all look like the same. Therefore we can map PMD 2 through 7 to PMD 1 page frame. +This allows to free 6 vmemmap pages per 1GB page, decreasing the overhead per +1GB page from 40960 bytes to 16384 bytes. + +Here is how things look after PMD tail page deduplication. + + device-dax page frames struct pages(4096 pages) page frame(2 pages) + +-----------+ -> +----------+ --> +-----------+ mapping to +-------------+ + | | | 0 | | 0 | -------------> | 0 | + | | +----------+ +-----------+ +-------------+ + | | | 1 | -------------> | 1 | + | | +-----------+ +-------------+ + | | | 2 | ----------------^ ^ ^ ^ ^ ^ ^ + | | +-----------+ | | | | | | + | | | 3 | ------------------+ | | | | | + | | +-----------+ | | | | | + | | | 4 | --------------------+ | | | | + | PMD 0 | +-----------+ | | | | + | | | 5 | ----------------------+ | | | + | | +-----------+ | | | + | | | .. | ------------------------+ | | + | | +-----------+ | | + | | | 511 | --------------------------+ | + | | +-----------+ | + | | | + | | | + | | | + +-----------+ page frames | + +-----------+ -> +----------+ --> +-----------+ mapping to | + | | | 1 | | 512 | ----------------------------+ + | | +----------+ +-----------+ | + | | ^ ^ ^ ^ ^ ^ | .. | ----------------------------+ + | | | | | | | | +-----------+ | + | | | | | | | | | .. | ----------------------------+ + | | | | | | | | +-----------+ | + | | | | | | | | | .. | ----------------------------+ + | | | | | | | | +-----------+ | + | | | | | | | | | .. | ----------------------------+ + | PMD 1 | | | | | | | +-----------+ | + | | | | | | | | | .. | ----------------------------+ + | | | | | | | | +-----------+ | + | | | | | | | | | .. | ----------------------------+ + | | | | | | | | +-----------+ | + | | | | | | | | | 4095 | ----------------------------+ + +-----------+ | | | | | | +-----------+ + | PMD 2 | ----+ | | | | | + +-----------+ | | | | | + | PMD 3 | ------+ | | | | + +-----------+ | | | | + | PMD 4 | --------+ | | | + +-----------+ | | | + | PMD 5 | ----------+ | | + +-----------+ | | + | PMD 6 | ------------+ | + +-----------+ | + | PMD 7 | --------------+ + +-----------+ diff --git a/include/linux/mm.h b/include/linux/mm.h index 5e3e153ddd3d..e9dc3e2de7be 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3088,7 +3088,8 @@ struct page * __populate_section_memmap(unsigned long pfn, pgd_t *vmemmap_pgd_populate(unsigned long addr, int node); p4d_t *vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node); pud_t *vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node); -pmd_t *vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node); +pmd_t *vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node, + struct page *block); pte_t *vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, struct vmem_altmap *altmap, struct page *block); void *vmemmap_alloc_block(unsigned long size, int node); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index a8de6c472999..68041ca9a797 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -537,13 +537,22 @@ static void * __meminit vmemmap_alloc_block_zero(unsigned long size, int node) return p; } -pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node) +pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node, + struct page *block) { pmd_t *pmd = pmd_offset(pud, addr); if (pmd_none(*pmd)) { - void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node); - if (!p) - return NULL; + void *p; + + if (!block) { + p = vmemmap_alloc_block_zero(PAGE_SIZE, node); + if (!p) + return NULL; + } else { + /* See comment in vmemmap_pte_populate(). */ + get_page(block); + p = page_to_virt(block); + } pmd_populate_kernel(&init_mm, pmd, p); } return pmd; @@ -585,15 +594,14 @@ pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) return pgd; } -static int __meminit vmemmap_populate_address(unsigned long addr, int node, - struct vmem_altmap *altmap, - struct page *reuse, struct page **page) +static int __meminit vmemmap_populate_pmd_address(unsigned long addr, int node, + struct vmem_altmap *altmap, + struct page *reuse, pmd_t **ptr) { pgd_t *pgd; p4d_t *p4d; pud_t *pud; pmd_t *pmd; - pte_t *pte; pgd = vmemmap_pgd_populate(addr, node); if (!pgd) @@ -604,9 +612,24 @@ static int __meminit vmemmap_populate_address(unsigned long addr, int node, pud = vmemmap_pud_populate(p4d, addr, node); if (!pud) return -ENOMEM; - pmd = vmemmap_pmd_populate(pud, addr, node); + pmd = vmemmap_pmd_populate(pud, addr, node, reuse); if (!pmd) return -ENOMEM; + if (ptr) + *ptr = pmd; + return 0; +} + +static int __meminit vmemmap_populate_address(unsigned long addr, int node, + struct vmem_altmap *altmap, + struct page *reuse, struct page **page) +{ + pmd_t *pmd; + pte_t *pte; + + if (vmemmap_populate_pmd_address(addr, node, altmap, NULL, &pmd)) + return -ENOMEM; + pte = vmemmap_pte_populate(pmd, addr, node, altmap, reuse); if (!pte) return -ENOMEM; @@ -650,6 +673,20 @@ static inline int __meminit vmemmap_populate_page(unsigned long addr, int node, return vmemmap_populate_address(addr, node, NULL, NULL, page); } +static int __meminit vmemmap_populate_pmd_range(unsigned long start, + unsigned long end, + int node, struct page *page) +{ + unsigned long addr = start; + + for (; addr < end; addr += PMD_SIZE) { + if (vmemmap_populate_pmd_address(addr, node, NULL, page, NULL)) + return -ENOMEM; + } + + return 0; +} + static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, unsigned long start, unsigned long end, int node, @@ -670,6 +707,7 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, offset = PFN_PHYS(start_pfn) - pgmap->ranges[pgmap->nr_range].start; if (!IS_ALIGNED(offset, pgmap_geometry(pgmap)) && pgmap_geometry(pgmap) > SUBSECTION_SIZE) { + pmd_t *pmdp; pte_t *ptep; addr = start - PAGE_SIZE; @@ -681,11 +719,25 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, * the previous struct pages are mapped when trying to lookup * the last tail page. */ - ptep = pte_offset_kernel(pmd_off_k(addr), addr); - if (!ptep) + pmdp = pmd_off_k(addr); + if (!pmdp) + return -ENOMEM; + + /* + * Reuse the tail pages vmemmap pmd page + * See layout diagram in Documentation/vm/vmemmap_dedup.rst + */ + if (offset % pgmap_geometry(pgmap) > PFN_PHYS(PAGES_PER_SECTION)) + return vmemmap_populate_pmd_range(start, end, node, + pmd_page(*pmdp)); + + /* See comment above when pmd_off_k() is called. */ + ptep = pte_offset_kernel(pmdp, addr); + if (pte_none(*ptep)) return -ENOMEM; /* + * Populate the tail pages vmemmap pmd page. * Reuse the page that was populated in the prior iteration * with just tail struct pages. */