From patchwork Fri Aug 27 14:58:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12462211 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4F4DF3FC3 for ; Fri, 27 Aug 2021 14:59:22 +0000 (UTC) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 17RDWLhq024915; Fri, 27 Aug 2021 14:58:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=iywGmSUorxPiMyQNTM8KoFlTm/4AjDxqck3CR3VAoR4=; b=04uwW1Glv2DR7nGaAwn8vC2iH5XbUYqqy5lSo5vPN+/KM6XTca2huBl52H71V3RY9GHD P7zpLOISAVLyspAj/FcdjHIbk9+SltmuswoQ/P7JfNr5yBWw6x2hA5hpJt83sN2TeRIF 5Rm86sLfEgoCkKboxXbylwXZOVPdqgpV+Kf/TrFX6SBMq+3YlnJkWvy4Z61ZanUIPQTn woS/3r5WV4STNMAu4x9r6UeXmgbUIG0JGtfLFsL4G5teiXnXpEpp/Ej/Aak/4EE/k22V DhiCdquPUF6VQF9Rk2A5yTXJQprHeNoKEZubQGWjOLBtmNylWLDWBLT9gLQlq9MAWVVL 9w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=iywGmSUorxPiMyQNTM8KoFlTm/4AjDxqck3CR3VAoR4=; b=jMKIhew5M7zqMj9a+b+tkvxXmY4CMhWaAPa0J8eruxDUJVimRCM10nAFUifCd+bsIqHz xBc2Vfx/9i84SwO8wHZWteKlea/glIG70ScYr3QAq4C0B1GZBZxqvzd2b9NpP9m2sYLE t4MuHJShHRii5settOoGrkTRUCYZpTDsM5yQ1Kha1JzUSR26YDf/FejbhyLNnFZJxZq6 W6qtqLFtVWItyxyepQVOt5VJucZ0uKS00A8XwpUCgGXYeJpPaqLpSeCTsHc05+ZC8DlE pwL6MHUoz2JsVw5PKB6vPvWiqqdqmIBQAd4IN9jhJfMlY3/pQcX6PnGbAokV1o1Qowrz 1w== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by mx0b-00069f02.pphosted.com with ESMTP id 3ap4xv3s1x-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:58:44 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17REpQK4187244; Fri, 27 Aug 2021 14:58:43 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2175.outbound.protection.outlook.com [104.47.56.175]) by aserp3020.oracle.com with ESMTP id 3ajsabbq91-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:58:43 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nxW92oMgKSxRJnQxq6qooaJ/qN8YejJM+B0Cjc3aNMrWtdPnVo5RTpPEvRpEHHtdbRpXHi6+fXiOh+1hyk72kFWqv8V83KIv7yTitwgqBvP/aPYlA8OeDSQjQj3gAntASRS0cjz2kiHTCrVE2TnPZrOE5YYIKN+Vqec/pmwG3/TCC/lX8zkTmb3cok1f/zFhyIJ9aKSOLt7aiSoU9J+zHuc0diUjCLeNB76FmK/mtDJKfjdpicDqTdwyQtKWksu8hNzui4p1XQ370CcJSMQER/Y+9/aAFwYllkWtG6eR9nwytUpnYeGlE/Jp2eCK04qSrLmBtTu9kloKiCaanKaZkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iywGmSUorxPiMyQNTM8KoFlTm/4AjDxqck3CR3VAoR4=; b=fERn7xQ8oYET1xSFNZtE2TiCUrXnO0kg3pnFW8RnfwP8qJNc8b/VCuIDeHKgcYrmXFow8m0KSFC/Dkq/6bQkb4suH5UnNjsr1IpiJ9crDpYySbRwPVriyr8vDszZsX4WDEM+OUngvaVk9tullj7kgvxe1NTyB6yDSFrttBQGvimE8yxLOn4gc2CgKb46fqWOWVpGk3+Lc7NExx9QXoSKzGGJdbfnhqmgn3g9kHbtv1hRRRNAW28/obMyL1rQZqZq75AzocSrqsvod8YnUlNUByoh9lY8NTqRIYkcThakPfmCjo9VuLMoBiCv24ShrAOiv47abv0cD+U8dstzg+K/+Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iywGmSUorxPiMyQNTM8KoFlTm/4AjDxqck3CR3VAoR4=; b=fPgMlCSxCDFLzvXVNJ0cvCbS55q+fMus5oRtPOvBz8+F+krRtN16tPGZrBYTG8DmtFPIKB8FC+JNN81eA8F8qHIzBtijGb/GcnD0SK2xRrzSjYtpzL/zMi4lQQXHoKowoZxmzc7nlw6xa2iSxwAmPO7ouZ+sn4OpAFS4JXex/3w= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BLAPR10MB5025.namprd10.prod.outlook.com (2603:10b6:208:30d::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4436.22; Fri, 27 Aug 2021 14:58:41 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198%5]) with mapi id 15.20.4457.020; Fri, 27 Aug 2021 14:58:41 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , Christoph Hellwig , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v4 01/14] memory-failure: fetch compound_head after pgmap_pfn_valid() Date: Fri, 27 Aug 2021 15:58:06 +0100 Message-Id: <20210827145819.16471-2-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210827145819.16471-1-joao.m.martins@oracle.com> References: <20210827145819.16471-1-joao.m.martins@oracle.com> X-ClientProxiedBy: LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (138.3.204.8) by LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4457.20 via Frontend Transport; Fri, 27 Aug 2021 14:58:38 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2818fe2f-6bb1-495c-85f1-08d9696b2872 X-MS-TrafficTypeDiagnostic: BLAPR10MB5025: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: jbsI2vFMURiQEeu/HeiXNqCDrgpShE8IE84Or5xDOXyHYE2hbv7ht18YdrIcDSozVQ5j7/lqYVnkiCq96mkCtXPkd2XUuZUTOim6r7e+QWsjnoDDsfGSL0Ms0AtZ8QbkszA+BKcRUokEyhFSJYdKM5avNdfrqCveG+BBy5yYiZfPxm/6BEzjWyZa2BjJ7W9JaUxAIleNfldd+HiKJlycUSW08izDJyrnua1z6GKcFi+eBqKrLCwxwynuGYmg13VH7ILEle2vKuBhscXC9a7w94aS/TywshqU5RyQMIr1Yy8+1fE/bWSozAJA+pz4pFa6Y+RiU/Nwh8CC2BU2IrO/RipDGpy/fHRGL1KUc49998ZNF1ib+WyoA0TlMDlNlRC7mGTi8PfKRcAXI25c9qfo0eZkolZG/LFUj5bcdsPTrTOXJbJB34Na2kVEXZN5oQd0ZgW0K9aXhiBbBqxXFkGQUjTdQp58We1HQ71yr293ipip2EEm1F417OJJZW0La1Z0VWL0VGkYht8bQ1+xvHQbeBcyd3Ln4Zek6shJzr7ZfElqYxB8kJLOaP6cUJiBgWHP3KCwpt0vZA9ISnq6SkRGy92E6DyB9d4tbMjFlMKuzSLjS9F6rJZMHHMXPPnn+TJfojUj0xi3IYhdcfywyHm77XKe34misk1xVCGwdHdBmoGKq1pfhmpzMx+6bvdV8/minChzXtCDhM4ZQG1NDb2tIw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(316002)(508600001)(103116003)(66946007)(83380400001)(66476007)(86362001)(54906003)(6916009)(6666004)(6486002)(2906002)(1076003)(107886003)(8676002)(186003)(4326008)(38350700002)(5660300002)(956004)(26005)(66556008)(52116002)(7696005)(8936002)(36756003)(2616005)(7416002)(38100700002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: IBGZP44k9ImKNnhHRV8gIw1rKXiZB+YQwOlQiVb/5qcvO2WHHZBbwzvFqy7Qeh1NkGA2nC3dzjvmYNmwK7IEUa4E57OQx8INWMBfm2cSEacQ15Iw1nyBSP6QtsV4QAbznSK2LUP48kpSgsLo+VjYeHJungiq1ilY91C+rsceh8sEIAXx0rmZWPayT8HqR9AtVn8sTQuM3cit/qr+kKyNmmWTVSxIH6i0jI+7cNfB/PnDRqmWPGVDsIVPGY+I6HZUVHh+JNQbqxP300OK41vjrXaSwxJqqC8AFXXh9fJBuvXrJ/KaS/1A/GHS1s8QqeWikY+M9vZXOQJAPTSvg9jQ648vYt8bu7EsFqCgaFx0N89MYp1krOGgxV3a2wlFiOt2frSJQCItzUcE5HR1HJ1DIz7iJoA5Zj7qIUIL4SzIgW79fBjTBM/BoK64FKcZ3JcY2O56Qv9r+vH3IJ0XkK5I4bSwDDFwiswfSDijaZX0a3LlsoDgUTLQpz9mxuhKoSarm7e6j5uq1Ibn5ecLRMznFzyvc4/hbZGULJfvDR5FlWHRXaGMayZQjECXWUEZ9g2djY5nBEuUanm/gynavCr1vyXTlQyzcqMYXB+lmmhFRvTAcIKUlN89tQckRgOPm1AnRDBC1AJSrtaSs9TAmPDNz1F/UBEXlo/kMp3EwiqXgQ97U0XkyuwjCTqxP46x39BKUw2bBqoydCgiuCuqEvqffbMBUP+oGu6s4uPHUzI6cUb0aIwqLdHH6TapVdPIGFBSTGNPlS6ndtueKr0XZIVdDCIxsvW3BaadvDIJ1O2L7mqKRL43UBgZbsQz4xkSbM8bWBMUZxA/f4S5mO7GWbBw+whMs+GKWWZARPkon9JkuePHwBolsEidylyJOqWti3iLLhZmpCJqEJ5y4F2MGA07ZD5qyBDlHNsxej0KtisZqKPOUzo/wArDu9M3rYX2BD5HTuJ8pxr8aayulINhDeSat2rOBwxSaBCtGx2AlT4NEV56akp7DFZTqkViD6M+tFcPYm/ARGIvegBDoIyMoQrWqxOvoeafG1ygqs4xzH7EvV6viRZARfb/lBM4mOiR4YT+aDhvO50MZSDhw305QOgBTfIJ1xfmj1MlNXFq7JUXusICgzZRTdZqOyPb82iJrbq2xUDSwoC4B21KA66HwrIYbhyDeMdqbBg4lZDc6HmwnYsnRkzF2Pfbi5+ZUn5uqV8l6kBAV5hY0SSN0AgTVN0feXgrZNV2ZM8n14OO1JJmNIbl35SG54WYzCRxj/MkOWBnMA4l1N4aLd5cdXQnTUERb725mS8On1Buv0ufOy90kUsFXeja6EYJXwaTcEdopCG1 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2818fe2f-6bb1-495c-85f1-08d9696b2872 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Aug 2021 14:58:41.3126 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 0AC6WS3ORmXNTsiFuTP7QIsQBj0jPmHDJoqqPJ1hJ1AxaRRAwTVyklmRhnaZDJE7JHL6nWoruRPdn1VkVhgDZ+9iOBIROfLqvkNXlHLMnq8= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB5025 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10089 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 mlxlogscore=999 bulkscore=0 mlxscore=0 adultscore=0 malwarescore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108270092 X-Proofpoint-ORIG-GUID: r0ZnmbRoSOQ1xyHUHiZcjCrnuOrKJTP4 X-Proofpoint-GUID: r0ZnmbRoSOQ1xyHUHiZcjCrnuOrKJTP4 memory_failure_dev_pagemap() at the moment assumes base pages (e.g. dax_lock_page()). For devmap with compound pages fetch the compound_head in case a tail page memory failure is being handled. Currently this is a nop, but in the advent of compound pages in dev_pagemap it allows memory_failure_dev_pagemap() to keep working. Reported-by: Jane Chu Signed-off-by: Joao Martins Reviewed-by: Naoya Horiguchi Reviewed-by: Dan Williams Reviewed-by: Muchun Song --- mm/memory-failure.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/memory-failure.c b/mm/memory-failure.c index 60df8fcd0444..beee19a5aa0f 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -1538,6 +1538,12 @@ static int memory_failure_dev_pagemap(unsigned long pfn, int flags, goto out; } + /* + * Pages instantiated by device-dax (not filesystem-dax) + * may be compound pages. + */ + page = compound_head(page); + /* * Prevent the inode from being freed while we are interrogating * the address_space, typically this would be handled by From patchwork Fri Aug 27 14:58:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12462213 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4D1F03FC2 for ; Fri, 27 Aug 2021 14:59:22 +0000 (UTC) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 17RDWLfU024925; Fri, 27 Aug 2021 14:58:48 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=N1ejeuFkXnP7EXDM3V4VZS+JdAbvmrAUrphHaOs6m34=; b=NVuBGvSaVr/MMGvU/JylBbIjS1F0uOT+avaQSbo/dMZgxFHgc7J/cSGf4l1CNl8M5ORt /24yExRV0QH3Z6mtJvgFom3N6MhLfFU9rZATMob4/v2YknJzAIuqsjV+maCnPWLxkj3L 96ykiIkJp0Gh7sQF/0cyT7f7k0a2DoneyAQUydyGB/34mbqZpWIF0NuCRP2so0URmR37 NQ/ACJ5dQoep/5rNxgim5QrxB5AQ3pmYXL+2vBFgWF2+Qp+kY3uomqyh2Eak1kMv2J+x wJNWjTWCdbFCTbEdR8KzO2aycd18Q3s7lBiVjJKGxqtt6m7Ce4E7AvwqLoR5CRS0OXNc 2A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=N1ejeuFkXnP7EXDM3V4VZS+JdAbvmrAUrphHaOs6m34=; b=GlCVDRTfsXtxPupUM9gk8PVy6PbngV6p5GxhH20n+v828/TTL5eiOgJKe02OyPwZ3xLc npKeZ+CFCxV6ST1bUDweIRt8qqBJdekAe28T0g0TUXRDZ2Qf/pG1cLWFVLtTO3OkN1eN o68wPY1JR/jOgDvncDQsRHQrn0e/LrQiHrcoXJBkw5jBb4UCtzsjFFb50ASOtBwFRepz 4p/76ODj1VOfagBActRzx89DVJRGrwcvDCevVn+9rpskjDyghsQsw+qK0ol4bJg7n4i0 ERQCHveDC4WqaKzvdvdFSXLpcE1OXN42updBxqyIXUZ+gUyCNVGRd6g/jEbbf7TyU7C4 8A== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by mx0b-00069f02.pphosted.com with ESMTP id 3ap4xv3s22-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:58:47 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17REpPbf187140; Fri, 27 Aug 2021 14:58:47 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2169.outbound.protection.outlook.com [104.47.56.169]) by aserp3020.oracle.com with ESMTP id 3ajsabbqag-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:58:46 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=clpfgj0NO/fk009c6/cCBtOapp0B8/e72u/gvzyS7yoM4a8MpcksUOJjztqga5+NJI92LkioU7+fKDxFxYXh9JvVQbDCRQ6DFXEmqbYzBwEC2TFxq8Aq3BLXKe2g8YTfTW+0Ky0Nysv9ZpEmGWoF+wdb7WyRltr+zXrt8ZHoua9o8CrdHHW0jiOaUxF7yKuZ8NyCsIy8rsLxkF19qL0/vg1vLjpIumCLpqtqasIWS59IrHH0y/ko1+4kVKkEp5r9+091kqLPyULgAB1B3VOYPoeMzJQXkelLz9AwuZ/meHLM14zrqy6sC1kdvFDTaz+03Mdiu1ucvlFHBE8oiHGRgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=N1ejeuFkXnP7EXDM3V4VZS+JdAbvmrAUrphHaOs6m34=; b=eA1XFZDFkMKmVgo0ovOZJ1PnMlBLYLCFr+EvMNbGEXgvpG0qZkx3e5VTZZoPEZ/T4oXvvcxtxJfqKPGKcpG/YpN7Vn5+mqL3LJdni1WH6m+D8S5N5rhTiN9lnYfalCSXTa6+49BKcgw/wYZ6fekj+pn+TQrZFVtZr5ANEMbMkk1+1Tt9bbKX1tVbbqxKfCxey1a3r99GUFkHnVcHIuNLq+I3jwV0LznRx2mSSdERcsOHXHtY/3l0XwOciLV9WSr1ws0qqmG5W0QtmgbK4x5acGhbChyjrdyIpBjj82mt9XWAebUloQy6JzwXP+x73EjJ19m5nL51umPX0Kj+izKdTQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=N1ejeuFkXnP7EXDM3V4VZS+JdAbvmrAUrphHaOs6m34=; b=HLKy23LSaF78eNcNX0Y8yDNNfrjozLTMrvXuclF/741Vus+p6iXTcOeCMJ0eV2kmGOTk/pWZ6uyGk0F3A2S1jhoOuJEWUbjcqmSzIrmVZ4p4FvzA0qwrj7L6pHQLQPYNO0ls9GUOYtEP7XhOzuLw+MxSolXZwqpn22JOFBzT/tg= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BLAPR10MB5025.namprd10.prod.outlook.com (2603:10b6:208:30d::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4436.22; Fri, 27 Aug 2021 14:58:45 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198%5]) with mapi id 15.20.4457.020; Fri, 27 Aug 2021 14:58:45 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , Christoph Hellwig , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v4 02/14] mm/page_alloc: split prep_compound_page into head and tail subparts Date: Fri, 27 Aug 2021 15:58:07 +0100 Message-Id: <20210827145819.16471-3-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210827145819.16471-1-joao.m.martins@oracle.com> References: <20210827145819.16471-1-joao.m.martins@oracle.com> X-ClientProxiedBy: LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (138.3.204.8) by LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4457.20 via Frontend Transport; Fri, 27 Aug 2021 14:58:41 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c71d9cd8-ee23-4d62-59dc-08d9696b2aa2 X-MS-TrafficTypeDiagnostic: BLAPR10MB5025: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1284; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Tc+XQJytI1RY49zXPh2fZJqxEFoy9qHtfRjrLmNitqc2CAebkb+ltu71R8nbyur2dyJJ+PuvtWV99pYzxqb+K+Og7/Y3ZmRAwWRAHRcrHE7nOAHWRnzn6e9i7LCFXDsAA+DJaIwlxlW8+n+YkDKzs/bwdple2l8j/pg3ysk8UORnGHl1Kla57vgIy1tv2RHykwUhFlHZYXbm7ivTwahkyC0ux1jjnPK0gPdG/Da6nw3KH6Y6YwIJVkt3Enkg8oiJrWZuI6wHiZdPGsJWsno01J8t/qzdKHFV4oDDOOeCUeVffgnmE4zCT03RBB6JYWUPZzojgjpXkD7kCnPsOEnale3fJU3389+sfTBBPo84Of4h5wSX0pZOyIGqudzncVAL4jFIPXPOio+5zul5873+uB33kSiM2Ys4v+jzWRF1jifsOr+lqVHl2LUg8hlOwlVjJHzguWFb5RFqacOus2qm5tIffQH9wuUs4rcrEmStHTi2WzjuqdK+wbUsfZVrRfRfi3XidasJM3Xa3cqkyI54tTZYlKy7lVsbA+BKB6hdBLXijJHCC16NIzNg6LxOCKYv8vArEfcUGcpwAmt8w7Cy5btYBh1CdNK2JATHgjrcPrkCNc01x/YXVKoMRJg/WejY+d1HMcU/ZlLRZSRuoNNujyxVbMIw6uEeiklhbHqs/Nj4nPQtg1p/hv8ldPnl4wd6fSwbfeMNwbhygec30yjLzQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(316002)(508600001)(103116003)(66946007)(83380400001)(66476007)(86362001)(54906003)(6916009)(6666004)(6486002)(2906002)(1076003)(107886003)(8676002)(186003)(4326008)(38350700002)(5660300002)(956004)(26005)(66556008)(52116002)(7696005)(8936002)(36756003)(2616005)(7416002)(38100700002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: xHj7sspuGFk13y6MqNsB1yu7Fgp9dO23h740t+LqKa2luV142VD8d26vIaZPi7i2e+unqdc/udVhZGpoG9JfG20caJHOwBNpcc8AkzBscAPtYpfidxtxBSOMTe+wHdoUjO5xRjXmIjvDLQ32CGGn22OyqmuSMUjMg0yVZ4APQTaTnqv635oRRMeHvmtR6OiIvm7EUb4nnjJTlp4qZ0R3QyK7GhyQKQL2QcRZKP59XuL5akf1224FIDL4zzVM+5SnUM5jaBSN37Kv6GtE6Og7AU5pfk5lV22ATsglC8VUJlBl9zFNYqI5jmtgBTpQvefXhqDtcvHQ8A2brMN9tpuaeLMGxLvOWlS6yVIH0Hpy0qDvWfwlDleRB3QlNtFOpWt43+E9uywYuSu8o2b2POXBknZXYByDmGUt43e0pudiQYra+0nPa11ffEPpLPN+OI37SZ+pi4f0f1LhgNR5gqwmxjHLa27o2fUK/pMZpGoOvFg4uocm2eyNVeAdRxs7JRPlc4ZxlkuhkWRCRr3IpvGC3LlnhmgVwu7dyj0D6EVDEnjAAPgABUJ8noN+dVxGhsxjusUzJWobhcb0AGB8an5K1uyTsqZv2A02FZL772XuG0evT5qcefkCkL4UEx8Bp3o7oNhXk+5HfSsBGHscEEcj+oOBwqkfgUp3HQAlmsAfhlHta8FuQkpS4s4zQP9JCWC2vb0N3hrVUJUZnSVwnoMJdRpJ/qYtWqpiY/JUqWRt86sVRnBun28hMufw7qZnLrfeYf9VNX88zWcoUgikyNt9wE9bwyvlxpLTB6QD73icgBjZTCU6KS5YB9cFV8PwnS36EwEr4YJCmjvNlLMBatHPDFb2UplS4/R2ArYGZ3GFIyzmQfFFt6CkOnhNjoonqbxGn39PIDqBzWv1oinPidxPQYEmSGlC1PakCNIks0ThDI5Rm7ExrzGfEju17Mpvml+3nZIB/qBg+6a+bnwNDOnPro5a3UV2aXnuXF40ArbQwARsztooOpe7kQEO1QlEtCujeLH18StIcBFUg/op6I0kvyxbmURZRLpWZMxeu76Hq8vmgmFgAr7nt0apldQPaCH4/pcRGq0ZnKh7YYlk+pC5nWm8r9vMuEAw4OSHHQwpCRHtZU3D1biEZk0x8vqk/UqUyPwnnS70kCUNK8EdaT+cPS6ll2oz3Q8NDq8e8QJXCD7uEKgS0Ai2trUV6VA+jLSXLyjD6xVBJS74pMd755Z6W9usQac+jByR4cBBJXhXHnvwsLSBhb95SXzJsKiRH8g9gC4WxWFsVRiyLTEz5SMc3rgYOY3+WjD/UZp/XmdqTNeWK8PFLEJ/9xucjvVt81qY X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: c71d9cd8-ee23-4d62-59dc-08d9696b2aa2 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Aug 2021 14:58:44.9367 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 9IoQou7z3UI+n02+zMI7TkO1aWBnGpXN6h+9AXiVzJFXNrfh+ClDb+V64SRiTfBCtDlTE641iXfCPVILc/f3ce/T8cBG+IO7G7ce1svyNuk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB5025 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10089 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 mlxlogscore=999 bulkscore=0 mlxscore=0 adultscore=0 malwarescore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108270092 X-Proofpoint-ORIG-GUID: iQwjjTr-mUINkifZjE8XHLGpR1NVCqIr X-Proofpoint-GUID: iQwjjTr-mUINkifZjE8XHLGpR1NVCqIr Split the utility function prep_compound_page() into head and tail counterparts, and use them accordingly. This is in preparation for sharing the storage for / deduplicating compound page metadata. Signed-off-by: Joao Martins Acked-by: Mike Kravetz Reviewed-by: Dan Williams Reviewed-by: Muchun Song --- mm/page_alloc.c | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 91edb930b8ab..2f735b2ff417 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -728,23 +728,33 @@ void free_compound_page(struct page *page) free_the_page(page, compound_order(page)); } +static void prep_compound_head(struct page *page, unsigned int order) +{ + set_compound_page_dtor(page, COMPOUND_PAGE_DTOR); + set_compound_order(page, order); + atomic_set(compound_mapcount_ptr(page), -1); + if (hpage_pincount_available(page)) + atomic_set(compound_pincount_ptr(page), 0); +} + +static void prep_compound_tail(struct page *head, int tail_idx) +{ + struct page *p = head + tail_idx; + + p->mapping = TAIL_MAPPING; + set_compound_head(p, head); +} + void prep_compound_page(struct page *page, unsigned int order) { int i; int nr_pages = 1 << order; __SetPageHead(page); - for (i = 1; i < nr_pages; i++) { - struct page *p = page + i; - p->mapping = TAIL_MAPPING; - set_compound_head(p, page); - } + for (i = 1; i < nr_pages; i++) + prep_compound_tail(page, i); - set_compound_page_dtor(page, COMPOUND_PAGE_DTOR); - set_compound_order(page, order); - atomic_set(compound_mapcount_ptr(page), -1); - if (hpage_pincount_available(page)) - atomic_set(compound_pincount_ptr(page), 0); + prep_compound_head(page, order); } #ifdef CONFIG_DEBUG_PAGEALLOC From patchwork Fri Aug 27 14:58:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12462217 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50B7F3FC4 for ; Fri, 27 Aug 2021 14:59:22 +0000 (UTC) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 17RDWNnF015229; Fri, 27 Aug 2021 14:58:51 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=DJxhtf2UP32RKJtBZO6aYGJ0FEFIQg4NRMEDKTpC288=; b=AOpwsd+ySzxx6CSxxVCWQL1nStC8+NJ+DKxFYxG2eIwj1ZIw+AwiWsi9wee/6clSr505 MQkJhzjh8wC3uFP7Vr8vcNMMcF8jgVJMCr9JoAALTUx4yYlBwvxg+HvuYYB7W5ZE/bzx lm2vwPdIn32LKf+btUpBkll36pV2RYH7wHxJSB5nO3+7W/5uwIHVOkNBdooGteFymw+1 ejHiYUpH5FUWhLjqoo2rlMbXzeEdLB4eRWw88j5MpyYcXN2zUOCrKHSveBhcPjlGJ2Ku XURptiugO4g7+Tpgi7oDvC6Z1jyOlkNokpNB50OcsFl/6sbQGOLh5eH/38IsXmX73EoT fg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=DJxhtf2UP32RKJtBZO6aYGJ0FEFIQg4NRMEDKTpC288=; b=njvoaE409aSv52T8ngPgfVWUWKiLOsLpq4/ylpHLKxVR1Zo+2JB7XXhSMky5Ko9xfpdG FPy/W1A8xF4eWqRNC1zaKCEY1P2o+x+Y/bSWDqKKoyukE25JxYsK8vxySgfOWWi8e3CA CSOfcdMnWyBJVh4V4zIKvhLAp/Vmm9gw1yDiFGzsByr/J+d2kD82EQR/uugNp+XGP14B OyySFod8KjiKYOi5KTaehZqWF800JkWR81gctlf8yqHGtaLM9bkBzUtwrRQa3+vIG00J BVrgt4obHPDvX9Lppa32AtXtOQt3DCTPRWVZ05cRN4urOrkUdZyx3vEIdcxF41h1xoCa cg== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 3ap4ekuwq5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:58:51 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17REpph2034347; Fri, 27 Aug 2021 14:58:50 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2174.outbound.protection.outlook.com [104.47.56.174]) by userp3030.oracle.com with ESMTP id 3ajpm4u8av-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:58:50 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Rpk5j1EMeH/ByyXdOrDygTUtBdwTUJm4JgAYKwtqgrQYkE5Q/hvQEg+flj7Q4jnb2/GxlV0HIPzwz51uv4xEMVCUUokpM3dWWvR+WfFau0DFFJaQGXzQfBDxU7DQoEm573eG1TmR2npWEGUhF4Pk0A7qks7gizWhbjsbaorRZ+ugi2hMtaAcaV/OAX/NJ0A23abFz0bpW4HEGN6qBVLwho2Q2eZg/ELKUsS944RDGu9+3w166GMzcJ1rXcWrHzwr6SiIQzOddKDu5gwoFMh7wj2CIAFSluSLKQNq0S5tTRhVdmVGYeXQX8ZWdCQK+g4ZT8EAiBJo5abrkZfE+NyNwQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DJxhtf2UP32RKJtBZO6aYGJ0FEFIQg4NRMEDKTpC288=; b=jjttVF/qSETGyADOoL8uWEZcCNmsg7Oq+s52a8f56KC0h0AWAf2iFAzf2DAQOelRTaBX1KLSBWQrqAMt6QCIQ//qR6w1VfJgd4Qz5naVrbNNhZ9y4EMv5PmXJO2mQbFx5zPirx+ZJfJLu+VXpGzsWax7ixZlbMPBFfdtmyUVbcv1/KpgLL0T5kCiDZ2WsfhN12JOzKH/MGnhcxForHNx2YBJnEDyRTnrggJ9vjTzA6Ia899X0Y7GylUvPoRa1vxIWADrkbuvyhE2J98bzJ6GvcMkdtJs6gcHDlRAikUZE/SVs2Of4bO3twiZL33LarVIYSzTmzKbFu2UoCHabmocSA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DJxhtf2UP32RKJtBZO6aYGJ0FEFIQg4NRMEDKTpC288=; b=VdtDsPbNKNEB+2ekF97eOzHbDqszFhJ1N5uy2VlzEyWL8jJVmbhUKjbbKDXOt57B7MvYNF/IG3Kjp4FrNUYqBCQG6Z3b6FTBmQAem3IXZfPo/L9AhEC/ShRZhPmt2EjMawzbHoA1AStX9ZKFKI4kunXXsXJG/bfnmzufamGXE8U= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BLAPR10MB5025.namprd10.prod.outlook.com (2603:10b6:208:30d::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4436.22; Fri, 27 Aug 2021 14:58:47 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198%5]) with mapi id 15.20.4457.020; Fri, 27 Aug 2021 14:58:47 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , Christoph Hellwig , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v4 03/14] mm/page_alloc: refactor memmap_init_zone_device() page init Date: Fri, 27 Aug 2021 15:58:08 +0100 Message-Id: <20210827145819.16471-4-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210827145819.16471-1-joao.m.martins@oracle.com> References: <20210827145819.16471-1-joao.m.martins@oracle.com> X-ClientProxiedBy: LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (138.3.204.8) by LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4457.20 via Frontend Transport; Fri, 27 Aug 2021 14:58:45 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f763bd4d-e7a5-4ae8-ae53-08d9696b2c5b X-MS-TrafficTypeDiagnostic: BLAPR10MB5025: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: rO8I8YU1cWASEYR10rBb7Jpf3RIL7x5WCfc632W755QOPqynQLdpPuCD1bzlbmoxxBllydKfMK/VQu9lhESwpMv+It6jJZxXSQ18+u0YTScEmlEoEElcZxibYSjbsDSmFTeT/Ml/4A7ewXNs+0J+Molzqsj0PiH5HUJQiyo0ChOult3XvoRHC0YPCaT33d5T4tWqNjOkI2ommpELtfBHcq7lHOAkPs4WeMcQ3NU7OHLYylj3tQLz4uX2fKrPfrURCqI5W/fIvkwWz8b9AWICvh3YnV7sknsRR2Xm30VDbQ5VudY/Cnc6pbANX2Dn6fb7DmX7B8y6Gw7Oe2CV2Y7gwNDNdMB4SaRtdb1QQnf+psOlRwbbQ9O0jyk8Lf67sYmkTvWLvxDSN9ik8bIuW1Z9CuxpDt/RTpWIld4DHX58NyN+DzaI/pLMb4clkEG2ZvkKXTwa15hWSNGWj4nOrM178/aqT/5rf6JqtXapRpCeq3/rUZwUwoLSsmMS9PVUw9yawcb8ZJ1lLm5tNbl51/5SUemah6Ytz7Df12YOBhRFtOD4/ocuPLUAuK6VV9xenlfft7582YF4Yr/GoFIdTteQv/UkFbnJaOI3iv3E/FYlpPuT02Ssm5KsgKSpDuobul/jfoKYI0Rj1cr4B7Q7fYuI4uwg261sawgEoIzCmVAnvO4zOOzw0RfabDwI/dU3vUWJd/Q6U1hz3MfUplKgJB7UvQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(316002)(508600001)(103116003)(66946007)(83380400001)(66476007)(86362001)(54906003)(6916009)(6666004)(6486002)(2906002)(1076003)(107886003)(8676002)(186003)(4326008)(38350700002)(5660300002)(956004)(26005)(66556008)(52116002)(7696005)(8936002)(36756003)(2616005)(7416002)(38100700002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: SSWmmgPdPDXviN/K87m0ecGYzjz1+l6zy93T18C9W+GRKEIenIfAlvhE7+8G/3V6Rrrim6DZ9wrDYFUI+wqD95FbCORwkofATPEMT6Z+S+t3eBd9U6qvi5ElVKfS56dm2P40pHcosLxROxx9K5Gh7dsw8+s3CghgB7hFxbN7cQJBk3wza2EPkwf8wckWCfDziQi2p38nkWpb+YCxtXXLJNIsVqClIYXHdeDxe2I97BJqi4pgG/mHqRpNniRmArCq4cophU6Q1+3v5Xb7Ul8wUL2JO6WidlNg3sFoA9p4dYpxELQa+8Elb0fwsQ+SlZJdam7l8VNWC3aWhrv1mNp6RTaodX4AOH/bygccFRbQwFXDAfOxPPEI/O44JWx2wyyOQRhePFuO8a0nYEpQCRjdnxrijJJMbdN+H/YVHDiaqQyQ9t/q3/f1pgGxxWorhl0LwYEYjIJHx0lxPqZROit2Kf8mJ7eZeFBkOf7EkJcZ/NJPVEoSVgZhuyCXR6WDIQz5LsNzzgtWEWfqRlQDmhfnIzk7akfY425twZFq7H2ScTXiwfoVm+DWKY2x1QSnI1AvXt18bFZ6vsKIxB166umV8RQirpgme3xlfxGINzxnfU4+hgy0AZfz81TqQP6VaZ/hUFqKFf1+EH+pl3YmRgBl2oz7fFN5WqEi1x+hAY95jlOWs3NVFfADQxBZ0Pdmonw6dfUGcMDR2AB+9LD8zT/Z2bTY83lHLYXmn4ecPVTIeNWUWyx0qTIWuVyaYR6izYtzWHdQjlHza44GXvGkmEeATj5PP5+yPnkamPXAaOoWve8gAoaeMwRWS2oC5GEcy4YLUgGSQzBD05vz4fYqKdxUcEdk/RvOr1bzy27Ca6XC65j7MSA7D39pqYUVeIChuuZcUBDCntTnyQGKJkq5jLnWBgufiY4l1Hf1C4zQdrtoROsORieuR0cvA88fgw18bsepXCz1J0QofDlJpeSb1ZijyNwByKd0TJ+3XbytXAfMIamv2C02edOALfim1nWIsSsvnQF0bNPwpnIXtIoKqYEo1RrlS4BgasI5MxitfS9CiI131WZl42dskM3+304GvNCLRmDizIGwnRQqS8y1vxPtFklo0+4bHa6ECjx6GM+9/xuKLLNPSJkMzJePboGofS+HAFBJWAruhEUVNg4WlUiHMoMsqRP5gt04x+N0eVK+t6yzmkjkcX0pH/n76EOuzmSjkl11QRHuEs1bf12wbaLSouTXZfuIpJTXXix2Yw9cuRRmXZEFiBBA1D+w8aaOwMQRmKXGF+X7cZjYkak3wA7pyt6aEw4pi4y1tfkjpTjRlMVQpIx3asGZzFa1VAPf6L+I X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: f763bd4d-e7a5-4ae8-ae53-08d9696b2c5b X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Aug 2021 14:58:47.8051 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: P6aWHdLsE/NgOOYmS7Xs15NP6V3k1f/gRa7L5RdXMRfsfg32E1kGxXhjxO24nyTD2l2aLkxEQe3ROwkG2GUxNW8kmntkVEqVDy2wjp1gXxQ= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB5025 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10089 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 phishscore=0 malwarescore=0 mlxscore=0 bulkscore=0 mlxlogscore=999 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108270092 X-Proofpoint-GUID: Vn1QbcZUwBvuxadr5ikVqud0sA3FbdF_ X-Proofpoint-ORIG-GUID: Vn1QbcZUwBvuxadr5ikVqud0sA3FbdF_ Move struct page init to an helper function __init_zone_device_page(). This is in preparation for sharing the storage for / deduplicating compound page metadata. Signed-off-by: Joao Martins Reviewed-by: Dan Williams --- mm/page_alloc.c | 74 +++++++++++++++++++++++++++---------------------- 1 file changed, 41 insertions(+), 33 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 2f735b2ff417..57ef05800c06 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6570,6 +6570,46 @@ void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone } #ifdef CONFIG_ZONE_DEVICE +static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, + unsigned long zone_idx, int nid, + struct dev_pagemap *pgmap) +{ + + __init_single_page(page, pfn, zone_idx, nid); + + /* + * Mark page reserved as it will need to wait for onlining + * phase for it to be fully associated with a zone. + * + * We can use the non-atomic __set_bit operation for setting + * the flag as we are still initializing the pages. + */ + __SetPageReserved(page); + + /* + * ZONE_DEVICE pages union ->lru with a ->pgmap back pointer + * and zone_device_data. It is a bug if a ZONE_DEVICE page is + * ever freed or placed on a driver-private list. + */ + page->pgmap = pgmap; + page->zone_device_data = NULL; + + /* + * Mark the block movable so that blocks are reserved for + * movable at startup. This will force kernel allocations + * to reserve their blocks rather than leaking throughout + * the address space during boot when many long-lived + * kernel allocations are made. + * + * Please note that MEMINIT_HOTPLUG path doesn't clear memmap + * because this is done early in section_activate() + */ + if (IS_ALIGNED(pfn, pageblock_nr_pages)) { + set_pageblock_migratetype(page, MIGRATE_MOVABLE); + cond_resched(); + } +} + void __ref memmap_init_zone_device(struct zone *zone, unsigned long start_pfn, unsigned long nr_pages, @@ -6598,39 +6638,7 @@ void __ref memmap_init_zone_device(struct zone *zone, for (pfn = start_pfn; pfn < end_pfn; pfn++) { struct page *page = pfn_to_page(pfn); - __init_single_page(page, pfn, zone_idx, nid); - - /* - * Mark page reserved as it will need to wait for onlining - * phase for it to be fully associated with a zone. - * - * We can use the non-atomic __set_bit operation for setting - * the flag as we are still initializing the pages. - */ - __SetPageReserved(page); - - /* - * ZONE_DEVICE pages union ->lru with a ->pgmap back pointer - * and zone_device_data. It is a bug if a ZONE_DEVICE page is - * ever freed or placed on a driver-private list. - */ - page->pgmap = pgmap; - page->zone_device_data = NULL; - - /* - * Mark the block movable so that blocks are reserved for - * movable at startup. This will force kernel allocations - * to reserve their blocks rather than leaking throughout - * the address space during boot when many long-lived - * kernel allocations are made. - * - * Please note that MEMINIT_HOTPLUG path doesn't clear memmap - * because this is done early in section_activate() - */ - if (IS_ALIGNED(pfn, pageblock_nr_pages)) { - set_pageblock_migratetype(page, MIGRATE_MOVABLE); - cond_resched(); - } + __init_zone_device_page(page, pfn, zone_idx, nid, pgmap); } pr_info("%s initialised %lu pages in %ums\n", __func__, From patchwork Fri Aug 27 14:58:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12462229 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C6D903FD0 for ; Fri, 27 Aug 2021 14:59:23 +0000 (UTC) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.0.43) with SMTP id 17RDWLfq010418; Fri, 27 Aug 2021 14:58:54 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=gsbLElqDH1Mq88nZFfRxU9ecTofm1cuNy6qL+/DvlDo=; b=d81ENQTgXJMbcBdP1KcP+6apovx5FheFcureFUdVRlLS1qcBjUFeDFDQY6d4UI0hMDUl 0bJ5Io5WnMEAw0eRJ/zEw8+Gt3zILfCB0iBY0mqv2IR8SPQ7HewaRxW9jYDoNx01kwqu yYlU3iM9PvoGlZKEkmdlB5A1P5Wv0ZwojppnDWkqpq2PqnHwP4KUuBTA62mICzcNadRj MKaNybxY+Q3GiZABSPFJIScdRdofalhB4tc37vLLUOeXSAtqafqG01Ivx9wF8KKLFoeA N4jCVx4+hlif5PP4oDhlLpUSlMzLKTJEiNBHO2TyaeFJcrswwP3WNOmEqcameUrNk3CD WQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=gsbLElqDH1Mq88nZFfRxU9ecTofm1cuNy6qL+/DvlDo=; b=E8Qwbsd388014N3Td824hMaRGy14ZkXAuy+g4/rq+rkCqT0TVxH+lrIl2bsridyC1Jcc eOMGPlE7RY31/kGYv93WX6IJld0nfxH9T1jQRceUDVAiQW2MiqBDh5lTVLdrNM6iSfQz KjAc5lxhpE4GzxlBXG9V/hCM/xU+tO+SH0+LyCuUPonM8kDxDnyTrPd+ZsxZAmxuvDXx mj3EjBryp3t+GYRKApIoVNUVJAQPZaewC8SMh8ha7MtxlBPv6NNPDKMsF73tNh7FcrxV SFaMQbAGHURZrdlAVtrGyxwr/wQlq0/FByWjwvN0C6nx0isz48wEjw8FtV8TSOhr1Qi4 Yg== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 3ap3eauykt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:58:53 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17REpaoN152736; Fri, 27 Aug 2021 14:58:53 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2169.outbound.protection.outlook.com [104.47.56.169]) by aserp3030.oracle.com with ESMTP id 3ajqhmq1bb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:58:53 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JxqR57BPnY9W20Utfgm+Uhp4uuiEqO+1nKjWULMBnhQVGf9XrKehfDSCM/X1YjX4X+SN2kv8eX75TOj1i71qFfEph1gNSm3G9c6k4uuW5eHb8F3MKCS7yM78WwD7RSXiNni8NFuJn8zf58WNtr1gB6oC7HwtC4V5GijljNeM77taKJvykKBXFt2c4le5ku+zw0a7DcHA9VLQ0gWJf1QpaXdLOvnu/i9EzLqdUp5gRRPmm/Is4tfipnIwg4hUdLy++ZZgKP3zrDRRGtz7cqxVL9mEvYYjFq+5uarEniVG8IC4SAFCV10uudYQp7ifwh8OPv4uwPGhCsnSs7tejkkVXw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gsbLElqDH1Mq88nZFfRxU9ecTofm1cuNy6qL+/DvlDo=; b=inVfC5dmYkcV0vFg2i/oSyk5+xBLh+8S8jcakkhjlNkDSZ+oMAMYcGdxwVhL+vWv1c6a1QLgO4E9mwHc49J88I6AK8VpSxbUJJ1zrneSUITHrfl9yS2FN2/C2VzTc0EJuf3BwFcEtCVbG+tRD3qU5zTKs8+Py4nHfB/3jjhQaNE3J2veERbnjuYAiwRPkD0/nvPBlWUQkERV2TQBchgeVEQE+McyUJmwOqDKnoCM7SFCtpuQr8SBZNgrdKnpRzD3XSBc72S1riHfYIySw+dF6+Z2eeu1EdGWMJY77+lbeuo8M4Sg3eFAd9Ld+mpAwXlVNJJp74DVoPReP08mf4sKwg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gsbLElqDH1Mq88nZFfRxU9ecTofm1cuNy6qL+/DvlDo=; b=e1SRY0AcPxeZ5yz1jvUAhxzQlxt7wXVMI9YN+CP3F3vLbABuU33ejeEW1ks/DuDwaedPqOg68oRewVkTT5Y68/YlR6aUQDpsrASTi4SO8h/0qjmOq3ejA0f/W7LPo2P5t+rxHKVUEMona3PZc3IuUepfi48ai2URCHZEcrLrj9k= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BLAPR10MB5025.namprd10.prod.outlook.com (2603:10b6:208:30d::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4436.22; Fri, 27 Aug 2021 14:58:50 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198%5]) with mapi id 15.20.4457.020; Fri, 27 Aug 2021 14:58:50 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , Christoph Hellwig , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v4 04/14] mm/memremap: add ZONE_DEVICE support for compound pages Date: Fri, 27 Aug 2021 15:58:09 +0100 Message-Id: <20210827145819.16471-5-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210827145819.16471-1-joao.m.martins@oracle.com> References: <20210827145819.16471-1-joao.m.martins@oracle.com> X-ClientProxiedBy: LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (138.3.204.8) by LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4457.20 via Frontend Transport; Fri, 27 Aug 2021 14:58:48 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6a344fdc-cf6f-4d15-8741-08d9696b2e0a X-MS-TrafficTypeDiagnostic: BLAPR10MB5025: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:459; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: BO1NxLPrI6wbk+5lu8F8ZN3CtnAw1nykh3A+R1PxmbHtIBVU13fgxFuDkosJ8CKSzh6LSW9aoi+k29p9v3g07+y5KeawFdna1X8UemWzoQRWa0Vq/cSHObuYT5hPopjNmBbNjt5qfwxcDzStormgdCgTKsZAWlODR03kLoxnGV6DxjsiX7hRAeON+IeRVoE3Af2/GJAVrZKQ3pMe3u4zfr5Wk6nfS6J/4wbCVSm95kAx7JwrGsbPYZRYPV94VaNLA9Iogj4ZaFV5G4sJDQZMKEqJ+1YlbwVQZl/UeuslqcUhXv3N9aKg8D9LiSf85LQVHZWY5OkiIfnhH6UfhsWWlbiu3pBvEBRABT5zjczbBlvTpTbGUOnXxK0TULrxqnDmiQGJ3+L44t1jKPhupicAfJNV1zmLTDiai/KUb6HW5SDFqDH7rxHhYStP5DNB78iiq3N6KAe5N9lnMliCAEcKSIt7yE8b4AobY+whTADg3+KT62IVTIuN39pbdgesLV8RpsxSt9goFtaxzSay4zVeuh1kAefIZhggICG/HEXsyOisqTEU8HWMNPo0a/T5wWNBQhkTX+Er7Lcjp9qlop9yQcOXs5ylb528EozBSYz7mgzJQ40C2bbutKINkxOoppBW0sJjl+GFp2ZwdnHlMYEP+nJP/ma4pgT0gtdZzMo72XW+GRcPBgHDdLnouP3KkQAFGU9zLv7A3DU2WNbJebJWjg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(316002)(508600001)(103116003)(66946007)(83380400001)(66476007)(86362001)(54906003)(6916009)(6666004)(6486002)(2906002)(1076003)(107886003)(8676002)(186003)(4326008)(38350700002)(5660300002)(956004)(26005)(66556008)(52116002)(7696005)(8936002)(36756003)(2616005)(7416002)(38100700002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: r7aGzBJSB6fqOKRoUxdQTVDEJG2XK1HJXpU8PN6HkGaxzRQo2vAMdh0Wz7C40QoSr/zas23U4Ca2mdqbv4DfWdVMAQGchZ2WkZ9cujriKyLpM/wY/bgTxcs2zAIK+KFu3vPZXY78RXiHkPf7enufDUvfLa56FGrcaN5CFyorHrqffkmnhoUriUgxqdTbho5lK++055GCStQzcYaeTG3gCjkPVXHYUw9TcLiRfBVdpL79MCvdjcBs2+TBvvwLqDG7+NClwWgNmFMl/Hwv4z73PRVAT3CpoRMT2xXdu0CYIbI1yJE0zYC1kBnxT3c1Q5Cjd7PUXCCUl07KOFDy/ReZIg51t6pc0K6FN/aaoCvHoJtbJlYWfv5MN5ql3I+dPyDs5SKS5ZGErxT8+lStlRe1VXAppUcl3UTYud7fqIN9W/f/+Qpo+zun3hyQdDTlLvFyIWjLP+l4ibzojox4WyKXEupI1ZsWmri2YGWSMCAx2O320Mk7+LTbJ8HTBcrsDg1FejlT/DFFX/JhT3j1AxQLujN5i6EoU8aElQmRKnqjlL8TnOaMaZlAdome7b/SZPxPfsHlXtUl+21YGIYMnxnQRuljdQCmJkjasl/194LPtqcpdPxDfVrwLt5c6pERU1kL65AcHaiMmr7M7ywBvoPruia1vZ8x+olGOI2PGr641udc4/OUf54Nmp6bz27dNdQ2HDQqRzKB6712SY8ObWvKYnROkpGxgm42vCE+AQcqATTTiC1tPTYE6tYzNM7BoG++UblHU/3aNGHZo8oKrCtEV842OsvZQQMzJVxrX8ObtXrFT8Vj+B+2d4xcP9DcDc8Z50mgk4VwqkJW5FVwI1PCeYELXalg3kuOxQV+/ufqzUOyjV1T+Bbz/ZZ7JC8oiGW1MKFmPFZcHzmQo+RBUYnGMEtJyyNLGZMeJ2AFrLCpyNYz/UlEKLG9A2AgZS3FPUlbBkFGiA2BxlfIiWnqswTtMtg7VpUbD8HcORKzf94ONoZEL04ysiM5sQahSRAcALZ6s+h5ijd6MjIuv1vqvRvHIZ9t5U7RUttU3/e8hQt8OVTsr7fHoGL+vX/KF8eTjeIYbowTSe+DnmxmqgxmxcrYpqN6nAFCA88aLZvIBtyWTmN0kDiAdXDF+X2CEbHmwQ5arTAHUXLtClpobUOTYCvae84FOa9RYrisoQln0Phvx5bLVYUwYGXlwNVvKsZZJZO0JYlwA5R95xw0hlBhmAAiBk3y1hrVy59S353G+0NGdWDpQDVAPgw0H+tMUaVxSa593XLoYqskTAVP+dAuB0fbCz69R3+nHWvSv1tmX1yGsoCbYu3A2kM5vnvAUF9Ow3H8 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6a344fdc-cf6f-4d15-8741-08d9696b2e0a X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Aug 2021 14:58:50.7033 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: PCf+jcxrmw6WvyngflYSyKv1eN0RZEOcYIcjoiTbShYOMpY+DFBOPtGS3Yw9bXsX1Q5k2VKvXotVnxV2dJ0gdUJAGXNWKgHcG7Yi6dXGPlg= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB5025 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10089 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 bulkscore=0 malwarescore=0 spamscore=0 adultscore=0 mlxlogscore=999 suspectscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108270092 X-Proofpoint-ORIG-GUID: bysrfJRBWvgo6bNC7pJn1f-wY10uHaRm X-Proofpoint-GUID: bysrfJRBWvgo6bNC7pJn1f-wY10uHaRm Add a new @geometry property for struct dev_pagemap which specifies that a devmap is composed of a set of compound pages of size @geometry, instead of base pages. When a compound page geometry is requested, all but the first page are initialised as tail pages instead of order-0 pages. For certain ZONE_DEVICE users like device-dax which have a fixed page size, this creates an opportunity to optimize GUP and GUP-fast walkers, treating it the same way as THP or hugetlb pages. Additionally, commit 7118fc2906e2 ("hugetlb: address ref count racing in prep_compound_gigantic_page") removed set_page_count() because the setting of page ref count to zero was redundant. devmap pages don't come from page allocator though and only head page refcount is used for compound pages, hence initialize tail page count to zero. Signed-off-by: Joao Martins Reviewed-by: Dan Williams --- include/linux/memremap.h | 17 +++++++++++++++++ mm/memremap.c | 12 ++++++------ mm/page_alloc.c | 37 ++++++++++++++++++++++++++++++++++++- 3 files changed, 59 insertions(+), 7 deletions(-) diff --git a/include/linux/memremap.h b/include/linux/memremap.h index 119f130ef8f1..4b78d30c3987 100644 --- a/include/linux/memremap.h +++ b/include/linux/memremap.h @@ -99,6 +99,10 @@ struct dev_pagemap_ops { * @done: completion for @internal_ref * @type: memory type: see MEMORY_* in memory_hotplug.h * @flags: PGMAP_* flags to specify defailed behavior + * @geometry: structural definition of how the vmemmap metadata is populated. + * A zero or 1 defaults to using base pages as the memmap metadata + * representation. A bigger value will set up compound struct pages + * representative of the requested geometry size. * @ops: method table * @owner: an opaque pointer identifying the entity that manages this * instance. Used by various helpers to make sure that no @@ -114,6 +118,7 @@ struct dev_pagemap { struct completion done; enum memory_type type; unsigned int flags; + unsigned long geometry; const struct dev_pagemap_ops *ops; void *owner; int nr_range; @@ -130,6 +135,18 @@ static inline struct vmem_altmap *pgmap_altmap(struct dev_pagemap *pgmap) return NULL; } +static inline unsigned long pgmap_geometry(struct dev_pagemap *pgmap) +{ + if (pgmap && pgmap->geometry) + return pgmap->geometry; + return 1; +} + +static inline unsigned long pgmap_geometry_order(struct dev_pagemap *pgmap) +{ + return order_base_2(pgmap_geometry(pgmap)); +} + #ifdef CONFIG_ZONE_DEVICE bool pfn_zone_device_reserved(unsigned long pfn); void *memremap_pages(struct dev_pagemap *pgmap, int nid); diff --git a/mm/memremap.c b/mm/memremap.c index 84de22c14567..99646082436f 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -102,11 +102,11 @@ static unsigned long pfn_end(struct dev_pagemap *pgmap, int range_id) return (range->start + range_len(range)) >> PAGE_SHIFT; } -static unsigned long pfn_next(unsigned long pfn) +static unsigned long pfn_next(struct dev_pagemap *pgmap, unsigned long pfn) { - if (pfn % 1024 == 0) + if (pfn % (1024 << pgmap_geometry_order(pgmap))) cond_resched(); - return pfn + 1; + return pfn + pgmap_geometry(pgmap); } /* @@ -130,7 +130,7 @@ bool pfn_zone_device_reserved(unsigned long pfn) } #define for_each_device_pfn(pfn, map, i) \ - for (pfn = pfn_first(map, i); pfn < pfn_end(map, i); pfn = pfn_next(pfn)) + for (pfn = pfn_first(map, i); pfn < pfn_end(map, i); pfn = pfn_next(map, pfn)) static void dev_pagemap_kill(struct dev_pagemap *pgmap) { @@ -315,8 +315,8 @@ static int pagemap_range(struct dev_pagemap *pgmap, struct mhp_params *params, memmap_init_zone_device(&NODE_DATA(nid)->node_zones[ZONE_DEVICE], PHYS_PFN(range->start), PHYS_PFN(range_len(range)), pgmap); - percpu_ref_get_many(pgmap->ref, pfn_end(pgmap, range_id) - - pfn_first(pgmap, range_id)); + percpu_ref_get_many(pgmap->ref, (pfn_end(pgmap, range_id) + - pfn_first(pgmap, range_id)) / pgmap_geometry(pgmap)); return 0; err_add_memory: diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 57ef05800c06..c1497a928005 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6610,6 +6610,34 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, } } +static void __ref memmap_init_compound(struct page *head, unsigned long head_pfn, + unsigned long zone_idx, int nid, + struct dev_pagemap *pgmap, + unsigned long nr_pages) +{ + unsigned long pfn, end_pfn = head_pfn + nr_pages; + unsigned int order = pgmap_geometry_order(pgmap); + + __SetPageHead(head); + for (pfn = head_pfn + 1; pfn < end_pfn; pfn++) { + struct page *page = pfn_to_page(pfn); + + __init_zone_device_page(page, pfn, zone_idx, nid, pgmap); + prep_compound_tail(head, pfn - head_pfn); + set_page_count(page, 0); + + /* + * The first tail page stores compound_mapcount_ptr() and + * compound_order() and the second tail page stores + * compound_pincount_ptr(). Call prep_compound_head() after + * the first and second tail pages have been initialized to + * not have the data overwritten. + */ + if (pfn == head_pfn + 2) + prep_compound_head(head, order); + } +} + void __ref memmap_init_zone_device(struct zone *zone, unsigned long start_pfn, unsigned long nr_pages, @@ -6618,6 +6646,7 @@ void __ref memmap_init_zone_device(struct zone *zone, unsigned long pfn, end_pfn = start_pfn + nr_pages; struct pglist_data *pgdat = zone->zone_pgdat; struct vmem_altmap *altmap = pgmap_altmap(pgmap); + unsigned int pfns_per_compound = pgmap_geometry(pgmap); unsigned long zone_idx = zone_idx(zone); unsigned long start = jiffies; int nid = pgdat->node_id; @@ -6635,10 +6664,16 @@ void __ref memmap_init_zone_device(struct zone *zone, nr_pages = end_pfn - start_pfn; } - for (pfn = start_pfn; pfn < end_pfn; pfn++) { + for (pfn = start_pfn; pfn < end_pfn; pfn += pfns_per_compound) { struct page *page = pfn_to_page(pfn); __init_zone_device_page(page, pfn, zone_idx, nid, pgmap); + + if (pfns_per_compound == 1) + continue; + + memmap_init_compound(page, pfn, zone_idx, nid, pgmap, + pfns_per_compound); } pr_info("%s initialised %lu pages in %ums\n", __func__, From patchwork Fri Aug 27 14:58:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12462227 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 31ADB3FC3 for ; Fri, 27 Aug 2021 14:59:25 +0000 (UTC) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 17RDWMp0015214; Fri, 27 Aug 2021 14:58:58 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=wu8oqghcawDSfhMyWRMS/wTgRRsj91HDqE16jmGF5bo=; b=m858ETkKS6/bY3Ep/xON7A/rDutmLHHlilZzLLucQdRkOEaDyeYNspmEueb5IT0umSNq zH5k2o20CUSTQHU0sykD1p47ZX5yZ1vbLtcMp3IsHJ6562DmX09F4H15yw+lHCdmXde9 47fti6+rN6M2Of4piyHvxHeUV9r89ZQ0m5FtUpObcngLuakI4L5KHXYJk15Fe4lfO03q FtTZkRxFm6xmHqbH8xYdsjFBQc+HMkHzVbbEW8cEfpbsK/T/MPJVjQOsmt+lUsTSoWD1 go3oAq5ceJ8M9Y9ihk945KK/eEdix8IrIkChAzspa0nQGegl2nMSvXKAfchRRBFz8P9S cg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=wu8oqghcawDSfhMyWRMS/wTgRRsj91HDqE16jmGF5bo=; b=uEZE07Bod4DC+iC2PIQ8nO5PgtN/D9flGh3eeRPCHj9mujGOp9L/+hj2Aw4golQM0R9s q+5Q01ONkcx5nchiEIL+iSUlDBAcNo8rMtYmaHh+N6t4IPp+grrJaWQjAWjIK4W0gvJY ee32PuTVUGpobWMlR4nTjIa/iGDr5y0uNX7TWj9RfH1kFdf8N3P01OiVPpZ+c3BaMH5H OzO8DyxegOqo8YtnFJo+CdccbxKJlLHKZuwmECFq1a7yDxh5hbc+WqMwPbJkHFF5P88k S8SpunFVHkF4qlY2FA2Lso/9WSvUDn6ZHHQtGBzQsnhpOQrxZw/+vUA798TsrMCUYlHr 6Q== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 3ap4ekuwqf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:58:58 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17REpYWO152590; Fri, 27 Aug 2021 14:58:57 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2169.outbound.protection.outlook.com [104.47.56.169]) by aserp3030.oracle.com with ESMTP id 3ajqhmq1eb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:58:57 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PX8v6Tzz2jvjCFnMbTyaI2/sIfpv8x4m3jx+An2j/dZkqcR63DMpIjaYtlb9fP5Pu8Q5YLbn2QW1v2/Dy4GRNvLsk2S1GLGU8N+bD4uSXaOO04N8Kibv/JXEn0rhv8T+DikPteCGUoBWhozRMvYyV/lgef828OozKCmg1yG3nc0t5PSSDZTRhLCa82k4YDbFiXrsnS3Wnx0mlOmySAovojolLr3IlbYIXl88VDve1jJZU0r+vPDRxCKuKtd0QGAY2igRMEyZhCOZPbyMg4ssFNms1ssqK3YyPNV8C7ZsJyYXeQK5Tqkw8+qTch0eOEiTOG0alkYf6Sgt76AeXdmshQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wu8oqghcawDSfhMyWRMS/wTgRRsj91HDqE16jmGF5bo=; b=GHV/PgCdenLXaV6gM3E2GvbefhEWpCSn6BML1/8s+z2qqdFY/qpY9RNQHcXi3rXNLV1HlONEDUWJ/1UtjIijUxRoEvA51HrrENkd3JP3AiNuxItaHgJyqq4jLxqdQrw7p/R3oXgaj+mGTWmQ0OsXYiP1dAyXxhMjBXKEhO7XNzQz9zOzQys609PpT+GTxIdgRNkfVOrIA/dVkcdVTKqV+XMpqO0hAMIJCK8fAor1BH8L2WKTNP3aqM9l3Gk/BKXp6sZ+0lvDKONbwO+62R5RfoqCK3tXZrewlu03B4tMaUeRg/DbXdutxKDC/2Jhpx+ZssygwiVNB5fLAVisddVmsw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wu8oqghcawDSfhMyWRMS/wTgRRsj91HDqE16jmGF5bo=; b=N1anZRqZtM0u9YmO+W4IPLBW7xbLsgKEsVBshcux70RuDmTDqncji6uTaCwHp8y4X3JTUL7xhOmNpwo77fWrDA7pU9Dc7E12QtVE09b9aINrbZv18+CvFlDXTPayFr0PBhhKk9QWLfSMv/BfI+eYhf7RUL66s2Z7MMouVSdT0i8= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BLAPR10MB5025.namprd10.prod.outlook.com (2603:10b6:208:30d::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4436.22; Fri, 27 Aug 2021 14:58:55 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198%5]) with mapi id 15.20.4457.020; Fri, 27 Aug 2021 14:58:55 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , Christoph Hellwig , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v4 05/14] device-dax: use ALIGN() for determining pgoff Date: Fri, 27 Aug 2021 15:58:10 +0100 Message-Id: <20210827145819.16471-6-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210827145819.16471-1-joao.m.martins@oracle.com> References: <20210827145819.16471-1-joao.m.martins@oracle.com> X-ClientProxiedBy: LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (138.3.204.8) by LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4457.20 via Frontend Transport; Fri, 27 Aug 2021 14:58:51 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ee86bfcf-a81e-4212-a117-08d9696b30c2 X-MS-TrafficTypeDiagnostic: BLAPR10MB5025: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2512; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 9vzTNOlPea6nBL40OQFP94KLWLle2UmNZVC0dcyhgFI1gqpjVgwIaWTrV3oyQ4gYG/uMLii2Pds2fTRQZzBhH0o670HRMwSE4ukF5KwoNGLpYqlB1e4VXF8TE/2u5Xki8pyfmseTIFzyGOQU50/j2mlItOPPfm3AFbovLQRUvtfQf+uGWMkNL/NJnWbLNxWAH5gnwNb4fc9Xxyq36fzQineRTgdv9C+DV9w+1CaNnnWtkRgd5Jfn/tWS2QXpSdPQHdddzfvP9H4OOiU0KjvVcyNvYG/6vrQEgywsNtuOOuMqNvOF4YtFXmz+EfTy6YUIqFKbg2XwYygrZIFd1iK/O8nSlqWOMlozUn6qPhq/3CP65K6Hxy+52XZoK/nWkjiTNn7AicQILl8gyvskkw4jZVjrbp7IE0rd8TONIa+gyA55iFRyuRuO1ce+V5S4USEqQbe9w+b71lKhpTVbTzzTPCVBxAhg0JG392qlaH6Bhrcb4Es+7+s8U7PkVTRc/uuWoUdYp1qNF14ZYXYXWskUJs2Qcfuw2zkci8oodqDzXrRQthZHlGR8HYSLJkjT1pYHla0o1hhcvif70d8kDW8tXKxr4B4NDbmcSRz0Q3FMr5FexsmoeBxNDdeJWVK0/WQyyMH/bYZVbGt1F2XKhdAJ5wmb6tswgTVEGxfZEpXSl2+wmZ1SNxXHPWCaLb1b7zcySoZzIVPF0SqEVgsgwEQ8sg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(316002)(508600001)(103116003)(66946007)(83380400001)(4744005)(66476007)(86362001)(54906003)(6916009)(6666004)(6486002)(2906002)(1076003)(107886003)(8676002)(186003)(4326008)(38350700002)(5660300002)(956004)(26005)(66556008)(52116002)(7696005)(8936002)(36756003)(2616005)(7416002)(38100700002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 8j4plPIs6OqwbCGm1x1dFLAlUo2rK66/SySEqHRX9D+D5SwjWDZyC2mDOHgNUZBDngBBpHlDmGGH8AOOHlzZ5pG6OOKB66HkmQbDWdpr8UQYgGATN6jm1Td2cFfQNWeENsqW39m+dSF8g5xM9vXeJhkYJoxuXjkxQDnseduFQ6gYN4t7wqAXn/MJ5YZ8yPDRuDlgObFpwpL4BqSzbdcQL4ldsk0t9vhkvHkNBmYJlnI+TtILlyQAEtBsILzpAkxlBy/OR/qK0UJnsa2Oxer4VNIcau00ojApO7YIBsF7v95Fzx+uWhrtMv2yfMmnGYCUGuM4Y3oDy6J7DH8MTnH29UjsEeD3ozZnIZvC1AW230s63SZSPZcWSol+BOE5H1dmwR6i8yXX8vlPMvVrl9E9MvoZOExSf5Mq3HQGFnOYXmcw24x0Tb4O910gqLKzgyHWBYY1i/8hBERc7AkLnbecWq2k+j62lQfaNj2oHvNCMhK3RO7tFpJI9kKkQzndDOWhCfO3KfUsMyu3Hd7Sq21eC/KADZNZ4+Ww13MrHjchkkwzQqL0J1lxIbiTD3B5iD5jiXyRPWPlguuJJajivXHCtPNzwHB8+MOsIp73NYQnAU70pjZ/vvvV29LTtIrzEG3Yvke9dDsiOIlYxpXwQ0MVBKXcZgMy9VVtpAtsRkE+eExlQfN65mXcEeA9/+bEYmZA+Ho3EKo33UU5ZYUWKPqavmrZjvfJtTbUiWGZUaP3JS+xG5sgccy0bEqh9k7QVbEBzJ2DrixUuqpUXPjExods4Msr7Qn9cfj0/llHDZPuROJ4PZ/7BnEc3Ar+UEkhwn1pgRS0UHfCraRuP4yDwkE8Uo8WH3Q4KgClTXURV9sm1kQ93NsDYiI5E9054obd8XdzVcdlQEkYXBZOk+sOLMgYx3p+b3m1CRN6NjjjFQlTNNiGR6kTOcncx/nrkwnlEgESEy3Wy3LOJFVIRtDlVOpj+JLCfoAcAwoJQbmLZxN8fiC1CltTVL2/Mk7Hal5v/UInS6ynIbrqJSa7uHOI6DM8manz3faydgXplTLYPUxoRPAjB0LBFOPhmpe0WTFqdcQ3jEVNnNI/PM+PF4pMIlb6th4EgAhQfu+jHiaQ5FF9ETtZQg/1KcuBU5cRSqGr1KHKdQ6papQMS0OV4kkPz7XwZ3S7f86ROxj1E+t5oEDfZOe+8yL7Lrl9AQXjGIgqgFPeQDTzCukb8NU/2gBUlhxFIULHzM1IX2/5jJdm5UErvy/NLhoHB3EMWXzHZ1515Qclotp2aJfYN1CmOxRl5pdd6XoTnBiWNETc7SoOKnTI5zK7iVDUukul9uN1lX1O12x4 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: ee86bfcf-a81e-4212-a117-08d9696b30c2 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Aug 2021 14:58:55.2942 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: GYQ5upL47REGuuVNzIIP+feG2qJjZUBMO6CJ6NHSpC7uhc5iD7QS0IBnTlh4BMLWSuADbukN1xPxBvUbv3pftaacM8tvca5iRU6sKuQOA2c= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB5025 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10089 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 bulkscore=0 malwarescore=0 spamscore=0 adultscore=0 mlxlogscore=999 suspectscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108270092 X-Proofpoint-GUID: k2aCWWn_NvuxwQTuNKSh09KgxSKpSZST X-Proofpoint-ORIG-GUID: k2aCWWn_NvuxwQTuNKSh09KgxSKpSZST Rather than calculating @pgoff manually, switch to ALIGN() instead. Suggested-by: Dan Williams Signed-off-by: Joao Martins Reviewed-by: Dan Williams --- drivers/dax/device.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/dax/device.c b/drivers/dax/device.c index dd8222a42808..0b82159b3564 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -234,8 +234,8 @@ static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, * mapped. No need to consider the zero page, or racing * conflicting mappings. */ - pgoff = linear_page_index(vmf->vma, vmf->address - & ~(fault_size - 1)); + pgoff = linear_page_index(vmf->vma, + ALIGN(vmf->address, fault_size)); for (i = 0; i < fault_size / PAGE_SIZE; i++) { struct page *page; From patchwork Fri Aug 27 14:58:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12462239 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 28FCC3FC4 for ; Fri, 27 Aug 2021 15:00:17 +0000 (UTC) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 17RDWIJo002221; Fri, 27 Aug 2021 14:59:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=vleb8xbLm8Ca5bQPl0MUBtBg57DLCMTJ2NWP2oEleU4=; b=ClgSWWI/jybLUeb1e8dqFVSjd5Lu01UwmP/9BEVhoIx/1ZRjH56O2A63Sc5Bh9pECoul Zs+M4wHPLZHnSgZKp3a3oR4T0wLJTP5wrBIHv9Ze+yoCbU9YRn6EMIEogBB6z/RDVQtr +GfnR/TAG5YeXwtWzv9wDj9jB1oUHXgY0VzP/s7vF3lUx5KNqnBGYaxPLO3NxNlE3FUS RulJwcXtfRWcIchGZftYuCyPYiMq7rhnhI8MRo5KUF8DGCv5+580p1B6dKY3vROg9OT4 tqaMkWD7zZ12I3Sh5FfGZpI49jL1DNq842Hpk7uctcUANESiVwjeP96YBfjbbpRbTvcd 3g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=vleb8xbLm8Ca5bQPl0MUBtBg57DLCMTJ2NWP2oEleU4=; b=C9k31Njofgp8e69KVnervBuFxDWOn1O+b9jKizqPgRdNxrABMasWz5fTjYIDsi6WoREN gWp+TN62UbMyylFreogY0jwLD2um2XEVNqtBqnUOCN63douDQd+Shrgg65Vq923sxtxA 7r8Q49JrqW0ySL2zTKoOyxe3FXQau0UdmZ7p1ai3gwsggGPa8yKP+iZFQWGNhtV726vG P7XA6PYTsvObNPVfnMdjuz1nOPUWdyW1CvLM0GTRIH98CWdQjSJFXIA89dqVcVnEjNVW 0OmeFPUWijJSlniLYchCC8RCn1GGpWoC6Qb2ZzdyqWJMhPe7eIQzFesVLwvV/7iKdKhq Hw== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 3apvjr0vha-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:59:01 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17REpYJN152595; Fri, 27 Aug 2021 14:59:00 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2169.outbound.protection.outlook.com [104.47.56.169]) by aserp3030.oracle.com with ESMTP id 3ajqhmq1gk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:59:00 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fyLbwiT0foJTl1j/0/64bxs1bQTlBni3OvUgvJjy2NNCGso2rVC+X28lLIkGBx8BK687JGWTGVCqNC9rq3Gkp+zj7acaVe1nzr/wGfO/YDJURqtB9R4A7R9xZjCOrGUmgxO5UBa7CzdyxCS4bjbDHFaFl9+70TyxkDE97SvLHc9Idi9cnh0zgVn5JR1PuHT/VCYEp54e9U6BYmIzIaYu3iSX3kQf8Z5VFLQqfvruEVDDfeey+fhcz5ybHpmhnuECmKTsQK51lK6NcjfwQ+ZkBc8VXpy1c2XKxBkg4RqNzxz1HTj7kasYoaLrCRquDTlIkfEKjUazEvRTTWsMMX0ybQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vleb8xbLm8Ca5bQPl0MUBtBg57DLCMTJ2NWP2oEleU4=; b=mXYCA/jpNo2ZKFMVEy42h3cVUHl8ezSYf8MaM2wUjNdJuuZQ5rAQbxnFSQUy7dHRxterDmZoEc0oBQMWccf5iM1nqVecWFAHjI2NYOcNPL4jXX8hyzOJbGKLj4dJm5ogO+W7VKnKHk6MzL+cs4V1R704d1U8L34EB0u6SjkdcwNZnm35RP9rhaznkhWNm/XVHlo6CxTEjCA+AJqMRxLyHnsdbShjf8KKe0gsng7JEj8/TmJGNUCuOmL9U358kzp/ytykgLe7ndvW9E+m5VmBZrBTw0bS6uPJkY4O+lsy79VFw/aJAVAQqhiBeDyl6eQmDk5XV695hNH2Is2OHacpUg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vleb8xbLm8Ca5bQPl0MUBtBg57DLCMTJ2NWP2oEleU4=; b=yIBxWpgnjD7bP9ZAgxXwoo2BwQJ2q74lrh9nOh+qfHQSzfk8Sgvg2I03FflWb08tdIGYyzlhst24OoHiVWe/LImH5x9hGvd/X+zUm6mm9rnXB8qMIG/KuY2SYh8pLJOtxd6RQeA3LGmrYJKHuDita25Q0rhua5JBXRGk4KphSqQ= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BLAPR10MB5025.namprd10.prod.outlook.com (2603:10b6:208:30d::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4436.22; Fri, 27 Aug 2021 14:58:58 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198%5]) with mapi id 15.20.4457.020; Fri, 27 Aug 2021 14:58:58 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , Christoph Hellwig , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v4 06/14] device-dax: ensure dev_dax->pgmap is valid for dynamic devices Date: Fri, 27 Aug 2021 15:58:11 +0100 Message-Id: <20210827145819.16471-7-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210827145819.16471-1-joao.m.martins@oracle.com> References: <20210827145819.16471-1-joao.m.martins@oracle.com> X-ClientProxiedBy: LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (138.3.204.8) by LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4457.20 via Frontend Transport; Fri, 27 Aug 2021 14:58:55 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f784e285-90d9-4732-e201-08d9696b32d0 X-MS-TrafficTypeDiagnostic: BLAPR10MB5025: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4125; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: zA94Ks/uu9DvumACtUU1uny7LScnXQ0CLeaO/gU0DE9hABuk7sP+MeLFdV3yrX/2Tg7WYL3wlI7apXuZXU0tTrBe8gnarILkv6hbdWrwTMrpEfoRVdT64rJr5ZzU436oQAEAvwuH050UYI1ODpe0y/H9oVsrVgjxIs+UN560pVWASciifiNEzQ13117RCdBSn3ax2H85QpjCTql13d3X612vpcqqBrhk37h0NS/eH9vsvIohUQZg2k+QCOaY/uQj7CgPB5H0NezA7iWycsiaNuXKizAU8zADSP26AaM1Fzjbv9H1NcDa+6vRJiFWgRDYw01mAxhcaixV9X2ACcxkhaPmvX6BJAv9BD0FgIO12JtSrYuwYedEpm5Vqfs9E+Tk+o88z6mN51SZXB3Lijlfg4WpANFQ/2PLJ3HRtc8QlRn2fcsjFcFEpWf8AaSzTXRPjEILwzIKpMAdMgmT6HF+NVco/bn0ivUHOrtiJp98U0zDO2nOzJTqN/SHpx8LzQ9e4gDUcHXjsgTFdfr8OzjQRK6iKGNKlCgZQTVU/pECznicBvT7x4Q8VVntjvQG7wq5Cj9kKZiuJajzU0w9Jxy2PMd1TjdwhJdeiF/JEzBrewlkbwXams9AMxUkRNTPoEnMgOPagvwAJvdqR7Jr2OBx6ejBdJz+J1vIIXgy0xLpYl9FNu6iM+awKk/OJbTh/NwZazCglnRrc5KP7fNGTsDeUw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(316002)(508600001)(103116003)(66946007)(83380400001)(66476007)(86362001)(54906003)(6916009)(6666004)(6486002)(2906002)(1076003)(107886003)(8676002)(186003)(4326008)(38350700002)(5660300002)(956004)(26005)(66556008)(52116002)(7696005)(8936002)(36756003)(2616005)(7416002)(38100700002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 6w53Llmp6GSyFkkK2oJlqLjJ4fTKPz/aOHI98ZORis6tQ+0EwbKefH9Gkt/3AvHkMGB+RWjhcfxpuv7B++pIbVHyGOfovl49N0lA6S30FXj1DwD0D3Oxqgx9ElZsOQqEFWnKzgAiK/Cy4M91Qb/wOuntczjWBTBsCjr6QnJwfBZLvwn2vQ+qlYzYmcyYYy4jji6Tm7yVdyzfjowU24CfzuuGL6Z4n9aVVMu6nRr9mjw0JpshSAVfF9xjZe09jJLolXppxxqsIBByFe4xjdzWYZdQB9AtKGTjxxT1q5paweHP9q8nd7hjsWbgvn2iHINqhIyS1uHRdlU54ZGeu59R4n9pTCpdg+kHjmGR2nSGRrImLQeF7ayZWOOAtu7o8ENf4PsGIgN5auV9xHoVTcsQQGLpn9CshvyWiNv/ORfVyHV2B3w+ySGezPO/xK0rex2pzpwjN9l7uQIdpusMEq4gkIimMZFSXfZVdI0hEMD5Dzf8z35vtfaeBOPl08ZPmyevFT6905x4sU0iUMoC/1+6ol/VrpAuLJZN/Jsl2WFMg4AMM4BM1ggpNs3CmlbmIBueLSPPmowFzPVKThqMap9JKPsYk1YNL4IJIhzcsactDhjMab6BrhpUj/bRALrV7FuTPxf6ekTNsXW9cdbRt1zWi8sg/ikRUWhfJ8ToZF9M2GnAa9o9b1gpmrLwiM3UlcEWa88+corjzAYo4bDfIxBqlRoes95VvuajkV+c4h7T0iM/Iy83W3JcHCVDL5dgKSyXXYuktT2ddHBbr4dfYVkH50CX6XVkUhic572KHcdEQPGBJ8yB8GmAV4wC02RiehSOFtOBgCPGO4Y8EnNoS/oGPkJP89whVi+3DNvBZRuDtBMaJlIJrKAS+cPEkdRzKd5yKX5yE7elc/hXzye49lnIKv5Z5QkcM9dcvGmolBVpuaor52KdUyAZovaIBoTHp0FXVm60/fSJoogpgqgRXJ8onpOe3B3B+BzC4SHTDzdnkJIIlOwf2qPzesORr3FpQ2DNf1QBB0WlaA6k8qMSWbPlDYPsnYiqFJXZcylP/ijMudvcNbkdFtBeeVj72RTT95HW7MTbxafKV9T2GgfWwdx0BTGQxaJlxBFHPy9FTwBDEnfIohRu/P3IiStphFnkLaRzJsjNak88+TX7ezAYB1Dy/nYpt+5nJbeMkpXDWrhw0Z639nQmv1K/uCEMZ9X80Nwjnyq6b3aA4Ihhsi0EQjF6IEfT+Q7QbS8t2k+hKoepmCchn013VarcM05txYBaaqBh4C0IT4FGmExDnRx65wHrNkA0uNs4Dj74R+9xDbF7FJvuFPuzb3INmtohdgLveqTP X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: f784e285-90d9-4732-e201-08d9696b32d0 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Aug 2021 14:58:58.6992 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: +zpWT6dHB5G6hSsSRND8l/D7ZtJrkWiYVjFKJUN8ahPQo3i/p1b/nScisJtDE326k3bep3kChK0KOBA4AJ9/rJuzuNBB7xfOfrwomI9DoOM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB5025 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10089 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 bulkscore=0 malwarescore=0 spamscore=0 adultscore=0 mlxlogscore=999 suspectscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108270092 X-Proofpoint-GUID: R4utl5Dxk8nXHr60bbLetGzZUHLyPGb4 X-Proofpoint-ORIG-GUID: R4utl5Dxk8nXHr60bbLetGzZUHLyPGb4 Right now, only static dax regions have a valid @pgmap pointer in its struct dev_dax. Dynamic dax case however, do not. In preparation for device-dax compound devmap support, make sure that dev_dax pgmap field is set after it has been allocated and initialized. dynamic dax device have the @pgmap is allocated at probe() and it's managed by devm (contrast to static dax region which a pgmap is provided and dax core kfrees it). So in addition to ensure a valid @pgmap, clear the pgmap when the dynamic dax device is released to avoid the same pgmap ranges to be re-requested across multiple region device reconfigs. Suggested-by: Dan Williams Signed-off-by: Joao Martins --- drivers/dax/bus.c | 8 ++++++++ drivers/dax/device.c | 2 ++ 2 files changed, 10 insertions(+) diff --git a/drivers/dax/bus.c b/drivers/dax/bus.c index 6cc4da4c713d..49dbff9ba609 100644 --- a/drivers/dax/bus.c +++ b/drivers/dax/bus.c @@ -363,6 +363,14 @@ void kill_dev_dax(struct dev_dax *dev_dax) kill_dax(dax_dev); unmap_mapping_range(inode->i_mapping, 0, 0, 1); + + /* + * Dynamic dax region have the pgmap allocated via dev_kzalloc() + * and thus freed by devm. Clear the pgmap to not have stale pgmap + * ranges on probe() from previous reconfigurations of region devices. + */ + if (!is_static(dev_dax->region)) + dev_dax->pgmap = NULL; } EXPORT_SYMBOL_GPL(kill_dev_dax); diff --git a/drivers/dax/device.c b/drivers/dax/device.c index 0b82159b3564..6e348b5f9d45 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -426,6 +426,8 @@ int dev_dax_probe(struct dev_dax *dev_dax) } pgmap->type = MEMORY_DEVICE_GENERIC; + dev_dax->pgmap = pgmap; + addr = devm_memremap_pages(dev, pgmap); if (IS_ERR(addr)) return PTR_ERR(addr); From patchwork Fri Aug 27 14:58:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12462225 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6CD6D3FC2 for ; Fri, 27 Aug 2021 14:59:26 +0000 (UTC) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 17RDWEDL013579; Fri, 27 Aug 2021 14:59:06 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=keg8kqseuVixviAUTAEpC8IL/hzf/YKR4sC0T69lb54=; b=1BfrvPdoors89MrN5ZtwBLbPrl/Ka6VrbV3pF3lv5lQlnIMzxqPW9SVzK5cnWyHNb2+d pQ44qTG4lpW2AtZpQiY0WOr1CygXIROWlPcEiSeBmawucEp48OGb3F+EKo/3SP8F+nfk ZcYZU9qrk94gUlZGY9BUWGKuLjr3CyDoCrnmsxgYKpeq5qYUCbT7vIdRigf6slvTVahc iFla3KWwFOtjXX/+V8SA0NM8Gvn8k03ZjbziLgXs5ahYpb0rV2a4iyEXDaKt3xlGhhMG L8NL9c0PiZ926iKUXG2CQhH63dBIh5zJMq3gpJKmsf6nXdmMXLfixMbWUQsdtMmDAoq4 EQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=keg8kqseuVixviAUTAEpC8IL/hzf/YKR4sC0T69lb54=; b=cR4OKo1fNxvsU8CjDl9vAA1NM/tfyLkUUCgnbJyjXK0D1DydrM7YUwV7NieMxkkDvvBg owTJdcX/5DHclv7KBy2KwwwnjyzgIfPAX10e3NgYrxUc+dcfcg3eOXYbuUrmrCLSbbAz y4fhPGiUHMbOJI4cfLVa1YpKXjwCPdADKkSYJZlWkCPnOMRRTKxAYXsBLbtwCojLZUo0 mqbY4x5dYkGZMShxT8xcequ9qAYh8wCho+zh/rQDpOWlTmTnsH82ng3KaQwxSOr+16ZC Nw+rpTG6al+1DBfQ83VUOySnTEbnRZ9hsaTNQy8r9j/0NS1i1axlSdOL9CiXR1iVjwH3 dA== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 3ap552bst5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:59:06 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17REpbE3155999; Fri, 27 Aug 2021 14:59:05 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2169.outbound.protection.outlook.com [104.47.56.169]) by userp3020.oracle.com with ESMTP id 3akb9293cf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:59:05 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=I4m4pcd97Hhz+Z1KDdKn8j6k9niT19hQ0eXS3IU9hCNfOnyW+L5Hdf53jjJuYFnInHqKZyR2SvcrRPOdchNK3UFTlUP52JoKZhPUpH87e827hqHVx15YsfSRUP6/MzFKHmBoAtf1Ygt8XvIrz3YjVrLieYaF+LrPf4bv1irLK6uW+tZdylGZbUKhz3s5JlQMYhSW1D3PekL1HtoxH9Jbdvm1eDFwxQrKnIHmZbS5QNmWx5vmF0ZDGd6Hpu/7GNEmj4iOkYACPZhKsHji+xb9U7FMF9GgTxlp0OG54XFh6ZZ0axjVXW1svF9Ql74DJLVebibzeQEeJsf6un9vt0dBjQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=keg8kqseuVixviAUTAEpC8IL/hzf/YKR4sC0T69lb54=; b=H1jyQjkuQUfQbQcZtCTkLOeqZQARV0mu8DEMHrLIzEnlfHCa3GhwDkCTojxqTCgA03Hm4Eg/4gpJbr7aQs+ucJllF8rSS1j0n/saRqcwV6iH8lTF1UbNQAx+n7oLAw4/X8ebjYBUaW/XRQsiFYk2Y2Zvrk7z+t0xBJB+uWA1aTMWjWIDZDtRrzPcj+5bnCCwAepmReEFumas7PfHm3/OP7D5RH150wNNXWEGcbZtNQLlRMynZWxm3AluxMm7yTppkBART1jbWWzWnNcaEji/8JKo78GRqWkPX4T0YPc/dfP6vxfnpzBVJWY1qMqvXhkd6biSI6JwWmsEchb0ShSAaw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=keg8kqseuVixviAUTAEpC8IL/hzf/YKR4sC0T69lb54=; b=WpdQr3RtTcuj9P/0CfhUDE9jjVPwTi5DMa9ys+1C6UWkVyFWyAQK060TezC6/WJkPvdEGWTjnBLrCLzfuOC0QLdIjME6QrGqLNOVtw021+ZVvSkUnY6mIybEqP1mVIJjqK+Eynll86SOZtE/cyXgv+CuRvYHKwBzl7GnscQ7x0A= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BLAPR10MB4881.namprd10.prod.outlook.com (2603:10b6:208:327::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4436.24; Fri, 27 Aug 2021 14:59:02 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198%5]) with mapi id 15.20.4457.020; Fri, 27 Aug 2021 14:59:02 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , Christoph Hellwig , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v4 07/14] device-dax: compound devmap support Date: Fri, 27 Aug 2021 15:58:12 +0100 Message-Id: <20210827145819.16471-8-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210827145819.16471-1-joao.m.martins@oracle.com> References: <20210827145819.16471-1-joao.m.martins@oracle.com> X-ClientProxiedBy: LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (138.3.204.8) by LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4457.20 via Frontend Transport; Fri, 27 Aug 2021 14:58:59 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8c31d806-7d10-4fa4-bf9d-08d9696b34a8 X-MS-TrafficTypeDiagnostic: BLAPR10MB4881: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4125; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 3kOVmRjS/ruRpTayQ9UcfWngoCkh6iUMW/pXnSgYp0Kw5t5UXTGkC7zrlH14j1hSyqF/mQGp3OMA6kVOBmpG96RFuNXwdnV+jmODfA/7A5kmewvqmP32x87ZhcXVF4E1uJQOXAfseN0UkqLzijCFLOtRag0tvtz3mfKnRsjtJo1w2rBBjuccmzLeYpkTyDz0RPkBgnVxf4SDUk01ZYFj4Ig4LYmItMSHwtpboJNOGCTtR9JJZfHBG7Ya/RaMofQnbA1N8VNFmCQHktxdLH3dWCtmXRVHlPixkiroJpePRQsTRoECC3Dh65BXQlpvKeFgZ79JyAFkWYngbTQdNHLAXbrkv4t6sk5Rio6A6HzbFxlBafD1sCykOKSZyUx7iM9KwLi7j2XL0B2RHW7SyQdC0uYLliYRb+/d6njsE5opAGgzaERuQM1ib5v3TqwVyXBagSaPD89+BntJPeYIGduNS8yefxYCb6ZKyoP/mjfhO3/Xiccth1mz+C+34d+K2z087pJKk8lLUlr/HFIbeWuKRwrBk8tcTB6sEMNNq0Wv5ryqubl+I8hlFByfM2p8vENTklRPlF5NhuH4LFVr9ys0UdntA+mf2kY/2YUb0TyIWNiWPsRaq1X/e0w+qvSjgUPXCt4sOGVxDoRcUvQLCI1ZmY/SgM2nl+3iroFt2e73fGQApglYIQo92UW0cRjFFI/xxJddtKhVoN/sn029qCVFuA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(396003)(376002)(366004)(136003)(346002)(39860400002)(7696005)(2906002)(83380400001)(38350700002)(38100700002)(52116002)(6666004)(66946007)(8936002)(478600001)(7416002)(107886003)(86362001)(36756003)(6486002)(316002)(1076003)(8676002)(103116003)(26005)(54906003)(956004)(4326008)(2616005)(5660300002)(66476007)(66556008)(6916009)(186003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: y0V+nJZQTfgiZWFS2bekLA/nMVHodcX01mdhtwq8fdcgprIfnv7pxNDE39RnLKMeKqAE7G2t9bXe2sAtRMe4SL5S+1TgWzspicgyo5NYq7nIPWEnv7WDZmjppV61MTMRSsv2iTXN+/WT180FWaGhHo/khLkQvbbBHNojUq0r5Xjru3wiP5m0v1c681yKtFBd4Gw0REL8tAdBIAc02rHLyPuGt6mp7hm4mlrHgp8+HHdfGGStpGfRH9ZaQFNj8mSyBt+VkJjC6HB6fYhU5+c7DRWfzVroxaXiOIYorhBWMQSmXzijn16CcpdK25sc7e8KoYTWw0t4BVYufDVFVgvwYeP8um4JQf82DXrLTlld0w9/cYYgyfIagIrUHU0UJ70e8ml7iAUATfs9FU9im7yybUoZsOU1NfkaHyAvsDjmBgjs239QevpVg9nLl0Et1sa1JZzLzzrs7Xi6UaoT9KSqblQT1RhVir+43hkUJSeL5jQ7p8KnyfosngtJ5b4gVsUGSO2Mtt0qT2ZxX4LDkDTEof7ZMEW7bajvVW1HlylWyr56/cPnzzM6yarD2XZ023XPtEd32Zt0qOGKSKSsRMo8Ug/aFquLujF/Ap2GEAgJzJ1z+sdCT2v7UTYJJUFyC2IAI0LAXrJQ7R2yZxzkuWNyfb3Mzt+pzIE5BL/PULa/zPZl1n5v34LTmfi/pRBaMKanhlmb78NX0k0FMdE2Su8lkyG0zDpwQKkjaOO0bQt4OMyiCHXFuWwHNaQfVwnLTxYgTrucsnwd8oHaPVFxIaOICzApGsvB4tLfi/urSzRrsTTczzI3V0zMl/Ls+TQNF+YnN6cUW+jag5YlqaDeMsLBg/oJqqajYdECfbxdZibKc9qkYog5pKUUmMe1/Z04f1O2fGGNvs7OyLwjWObbbKekDOYU08j5waHnPatecOIeYMRomA01R5NzSRkdZqv+rhdzD0/k3iCASnd6x4F9Axbb4OH01jS5XRxE65BvhD2G7eK7yAdTpz7bq3pxYb8BuJvzuYEZDdoXvKv2HY2ti9fVnkYAv444n7PsqzAEmFVcVHssyiLtI1FmNYBdq79fI166OJqoJXTHi/f8HLFbyKJMQMdQ3R7yizVu+yw9pQp9v1GrOtFXb7kMEtoZvPBifb7aEqsCDMlxbqmY+oEW9tiGImNUUoNno98JzA9572mg3QXooDfkFkkePGdWUN0kdloizdQKmrfOiAYqdRm/I9crYNZCQ/WcFU1IOb75BsFbFCw7aJYxJ2p4afSCCjD0DGOiPr98YBqG47No0JF42wMizqA1yeLhSBIUFQqVii/ATWpk9jzmweqBbMlySCxE18LV X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8c31d806-7d10-4fa4-bf9d-08d9696b34a8 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Aug 2021 14:59:02.4000 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: qcEdRFm3PAwR87IM9LhBY5FpUFscXzVmg3R51/Qg1d2svtkQzzqVPHmhYuWAiTjjaEcNx24HwE8luqI2x29WR4k3xG/4jXQ5JHGXbUtfOUE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB4881 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10089 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 phishscore=0 spamscore=0 bulkscore=0 mlxlogscore=999 malwarescore=0 adultscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108270092 X-Proofpoint-GUID: 1040wEk4GCOOl66nXwvdIrKZl__XKND8 X-Proofpoint-ORIG-GUID: 1040wEk4GCOOl66nXwvdIrKZl__XKND8 Use the newly added compound devmap facility which maps the assigned dax ranges as compound pages at a page size of @align. Currently, this means, that region/namespace bootstrap would take considerably less, given that you would initialize considerably less pages. On setups with 128G NVDIMMs the initialization with DRAM stored struct pages improves from ~268-358 ms to ~78-100 ms with 2M pages, and to less than a 1msec with 1G pages. dax devices are created with a fixed @align (huge page size) which is enforced through as well at mmap() of the device. Faults, consequently happen too at the specified @align specified at the creation, and those don't change through out dax device lifetime. MCEs poisons a whole dax huge page, as well as splits occurring at the configured page size. Signed-off-by: Joao Martins Reviewed-by: Dan Williams --- drivers/dax/device.c | 56 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 43 insertions(+), 13 deletions(-) diff --git a/drivers/dax/device.c b/drivers/dax/device.c index 6e348b5f9d45..5d23128f9a60 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -192,6 +192,42 @@ static vm_fault_t __dev_dax_pud_fault(struct dev_dax *dev_dax, } #endif /* !CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ +static void set_page_mapping(struct vm_fault *vmf, pfn_t pfn, + unsigned long fault_size, + struct address_space *f_mapping) +{ + unsigned long i; + pgoff_t pgoff; + + pgoff = linear_page_index(vmf->vma, ALIGN(vmf->address, fault_size)); + + for (i = 0; i < fault_size / PAGE_SIZE; i++) { + struct page *page; + + page = pfn_to_page(pfn_t_to_pfn(pfn) + i); + if (page->mapping) + continue; + page->mapping = f_mapping; + page->index = pgoff + i; + } +} + +static void set_compound_mapping(struct vm_fault *vmf, pfn_t pfn, + unsigned long fault_size, + struct address_space *f_mapping) +{ + struct page *head; + + head = pfn_to_page(pfn_t_to_pfn(pfn)); + head = compound_head(head); + if (head->mapping) + return; + + head->mapping = f_mapping; + head->index = linear_page_index(vmf->vma, + ALIGN(vmf->address, fault_size)); +} + static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, enum page_entry_size pe_size) { @@ -225,8 +261,7 @@ static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, } if (rc == VM_FAULT_NOPAGE) { - unsigned long i; - pgoff_t pgoff; + struct dev_pagemap *pgmap = dev_dax->pgmap; /* * In the device-dax case the only possibility for a @@ -234,17 +269,10 @@ static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, * mapped. No need to consider the zero page, or racing * conflicting mappings. */ - pgoff = linear_page_index(vmf->vma, - ALIGN(vmf->address, fault_size)); - for (i = 0; i < fault_size / PAGE_SIZE; i++) { - struct page *page; - - page = pfn_to_page(pfn_t_to_pfn(pfn) + i); - if (page->mapping) - continue; - page->mapping = filp->f_mapping; - page->index = pgoff + i; - } + if (pgmap_geometry(pgmap) > 1) + set_compound_mapping(vmf, pfn, fault_size, filp->f_mapping); + else + set_page_mapping(vmf, pfn, fault_size, filp->f_mapping); } dax_read_unlock(id); @@ -426,6 +454,8 @@ int dev_dax_probe(struct dev_dax *dev_dax) } pgmap->type = MEMORY_DEVICE_GENERIC; + if (dev_dax->align > PAGE_SIZE) + pgmap->geometry = dev_dax->align >> PAGE_SHIFT; dev_dax->pgmap = pgmap; addr = devm_memremap_pages(dev, pgmap); From patchwork Fri Aug 27 14:58:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12462223 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 49B613FCE for ; Fri, 27 Aug 2021 14:59:23 +0000 (UTC) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 17RDWEfY013581; Fri, 27 Aug 2021 14:59:09 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=FmvbiBQe/MGVCFEPF/TYoEsiAbi2ZkHeMEfUAyib74g=; b=XHejN7e6jWAfukXW0b0eGvsvLknSJTT539FdpqVy8OkCeitTV9td1hCGWttg5vYORhj3 aSi8D3zgpm9O+IFgQmlmIDLOGetMsSZamOWw89AoRwR9KKjpnMexOCEHOp9CmXjntNhM UWumo2jowWBvc+IOQG+s5R25jER66FUBgM6cFHTUD03lGd9/zXmsDZbIw/q+PRDCGJVq nyU0i+W6KKHh4g6R1EqYdU9/Vt7uwN6amGhYqNo3t13V/8DQPHcDKbG5xbc8loKJ21Mh y93dy0MkuZ6kQTahUMpbY+IxF4kXSuaD5mu0BoDUquO0LlUNQLHL8Kvbbbxq40Z7+bbw 9w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=FmvbiBQe/MGVCFEPF/TYoEsiAbi2ZkHeMEfUAyib74g=; b=ha7DC0RTHYrU13pJSbA+7vE7JBW+pwXqLd6wQFjrQGLFerCwYiVjvtYhiNz+uwCc4p7Y jkwPtMvRecDvYk/9PLbcoGnnlSkoSWhiAq9oKkVUlL5lhQiz48OBEBvPnMxmNrgVKnQd F35M+Y+N70/GPq+sgfRRZnwe82KmMFiQN3qTctaHPhOViqluak9oYxjv4dTR2LLYZF/p MC9eAAmOTqdJw0ti2Q1txiI9aETWybewyEFeRlQAHEgolKALLS44wR7WZVFxSR2ZXNvN 42l/LbAekO8I44ez22ah8M9uluOXudwTZFHaQAIDXq7sC3Eizg+Ex+PWu7fm3eYaqzdN vQ== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 3ap552bsta-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:59:09 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17REpYnl152588; Fri, 27 Aug 2021 14:59:08 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2176.outbound.protection.outlook.com [104.47.56.176]) by aserp3030.oracle.com with ESMTP id 3ajqhmq1nb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:59:08 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=h50T0LGTYUO/96DQ71xiIS4QgX4Wbq2qo+7tokBfnkwION6MOHwpDdvsXAdn338S1Z/pVZpjqcNyKrKRZBfhIVsiSUx9oW9mE/lykOjSRO7J9+uDlcl0mG/GgSpsYdpDt1czjVEe1kVNmzc2koGdt0kayzCrnCdBPp+oRoC+TWrXZWBin2BJ3LkfGwPqVCjMX4U7XhZ3HCrECrPoYw61sjjCkE6mTihOKXpL/K+XxtV4Fb755ckcOAfpBBNXkjT4aq+wDjPotk6q9mRQe9LDk2xNPLF03hJuHdx+ThLB/dth9HxDZCD1RO7IB4IiKbLPynhfhDyjA1EfH+axQ56oZw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FmvbiBQe/MGVCFEPF/TYoEsiAbi2ZkHeMEfUAyib74g=; b=nBFMGoKQ+s/MyWnUb87oP0PFdI1NJzWdPfGB7rPTcMaxIm7cenIXrtxlOLQK4S3p1NaIT9CETEGGGufb0U0FNKKTgSgIEH8o7OS7OFhRT9SjZXz892cw62vC73ri92TiNsHPuZcVhIld2csvrzarJ0khkrUlfylMj5usZML7mPEQ8ZooVYbpyEDj1cgSgrkSrY0gKvDSjEHP6px1gv/4YDViyGmt9c/3k1c6oisxbDJe0GZlUZ4XnxI85sk/yxEzpgzvUG8wQaSC3/eV8lx5VD7cs8yd9vLiYm5VF1tP47whftbrQdRm/U1guhUoKXY6lHZ2Jz3uJfY8x5WQ6H+HNw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FmvbiBQe/MGVCFEPF/TYoEsiAbi2ZkHeMEfUAyib74g=; b=KHgJ4wz62mLtMB3gOXx1MhaFoWcXAZlHH5F/hb0Dot9bEklbEmORo3h878FVJoN5RczkeNpt18wOuonjTLvxDmL9ZuwP5lHaSCVJ6ttV5BezxFS8yNtL3eEoC8hmRYpfUlGDXOzP+4GpHp6zWSV3I4D/jm+qX5K0PGDJOnaa8Qc= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BLAPR10MB4881.namprd10.prod.outlook.com (2603:10b6:208:327::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4436.24; Fri, 27 Aug 2021 14:59:06 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198%5]) with mapi id 15.20.4457.020; Fri, 27 Aug 2021 14:59:06 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , Christoph Hellwig , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v4 08/14] mm/gup: grab head page refcount once for group of subpages Date: Fri, 27 Aug 2021 15:58:13 +0100 Message-Id: <20210827145819.16471-9-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210827145819.16471-1-joao.m.martins@oracle.com> References: <20210827145819.16471-1-joao.m.martins@oracle.com> X-ClientProxiedBy: LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (138.3.204.8) by LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4457.20 via Frontend Transport; Fri, 27 Aug 2021 14:59:03 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 7b2384b0-5692-4966-c863-08d9696b3747 X-MS-TrafficTypeDiagnostic: BLAPR10MB4881: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2803; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: GQNxzSKZVdWFOIUbfKt3nYBxtXT5kHgVCqn4n81A5dLRmdXyclokPRYxbabGO4dYQI9GXQzG5ibdMrNvPkA0jLldtuuYRamGe2DgRDGR56rHUw1CQXI7yJla/5mCQhrEsVGYILDNenaqVM3tckV+vK8yF39ydXT2Hph8BzVQbphKj+aXjC4cZ/wI49KPySuEpCMcxS4Hzi9gqDVWxi9qTcgm+D14H9dxJvmrKjH2iPHXn+z/oHzxeHVQuKPMQWdX4JiDcixYd7jLSWSxn39U+tuuy/kCi7Q/RnP29oD8Ak+rREo4oH9Pmxvlukn5V2mniMFMaMzJHU5TkWI+FaHrmJkeOpBXUBYC8JYlgs8d+QPTx7zUiJWNPtHqErOLZFhHgtH49QGcV+mSvm6anoLg8KDqm3e49qJg2FSPN/q/s0t5TBbY60upONymBkr/q1BKz64qFZinjWyOyItrJfYOIRQqYkX19N07kKuZHT5PdobUD1UQ+C0youXkZjBTzC7dZtU4ehVEssIvcAYhO2Zt8RiJMgCOq0qu8IQIPyutFawjaFBiVXkBpvhkX0P7/HYHl9Nro/K+p5sKlDLA7Gz+Ui/ZJVRGVo/H8AIuV/nfk5ZtIVVjhDa2CvA1zlyQIuzZjRJ2IvhUSvqFOOntOcif7R1qJkwQBuD1OfQLhKz5pRkGBIhz5ZAPBAkuNBByGrJ5HlRmjobKr6ZnNCUYNhD2fg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(396003)(376002)(366004)(136003)(346002)(39860400002)(7696005)(2906002)(83380400001)(38350700002)(38100700002)(52116002)(6666004)(66946007)(8936002)(478600001)(7416002)(107886003)(86362001)(36756003)(6486002)(316002)(1076003)(8676002)(103116003)(26005)(54906003)(956004)(4326008)(2616005)(5660300002)(66476007)(66556008)(6916009)(186003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: IRWUI3uqT81WINLx81LzxRnAZYnhJKuGfmzU2qQA5ScTBCXKG5TIKmfz7RWiMLsMI4nqnkt5PThTcGUTl1dpNggkiDn9vv6CjIwfrkratW8VUN630wxuDPC1CDjMIGXRX1jdU/q9rYcT2NoeuffPkCUXuLbJri8mm7W9b71QhI6fpz1OO0dbDnVaeug7FBaB4pKFuaS+mtTHJhziTV5KquvWPkuLhZ7Ust0YTMFdRX0Yt9Z8twrRyrsbb8ZVAW4Q/gXwJFhpzQ18bOSZydFqjlWYUF/R7C2Mr79+wldHEUcwTtZAIdsKXA0prta0pC1evjHY7z63lbtntEe1weJ+Ulh0v3SbZSsnkTxQWToCtlBfnR0WxcXJsvN3uwL9FAKAoUxSvnzGeYU8zIKsYLvWktWJQdrW4cZW0zqoQPVVV/s2hBzbSqS28+h4zRiWNU+1l0PRPLKHQfHfy1zukxvdcB1uCvP5HLN4IfH10NDk+nYyZtt1e/Ta328k6r2nFFNBiq6X2+7twIKBDoV52Q0V3ghvST/QgSH0tEGYjF2rbpBMIdOdq1Fj2dozBF2K9S8O0LD7z3FYZvutiEvHK8DrQM3vrp0PrcMv/lPKgFsMYTs/s3NJPir5rLYWmG8cwKJhpC3uqGTHGfkSy3sXwu6oCC9QWKJBXBJhhmHkmU7lFoM26OAeMkwxhGAJmEfabO+hQH8xDHGDVW8GPDsTvGSUiKfnw6S8LZyppCQu9gatxfcQtA/azzTK5IylpvHAK/0OtK4C3G7zdcfpfTuhpgNoUck/k9JiNdTABcGb9vra0t7v+kcAu62CLdvKObxVDMvUKzlPycJHDr0T0lUzt6e/8uEDiNyS2w92GaXjrsEOlm8NWZVUUz23RPLLRxGZRbR7sCjcg5A48+ehb3tXMdmV5wpz391zg+sujHLwZapQq07EhwV52uCZlJbLCFff7QFvyqZqN3mOld6ighMc2QcgaRROfzti0yYpAUMnl9y+sEqIwHVD/ss9vKHZdbBaQ7tJ/ENdcz+yQKQvwbErdpboWFdH7pTxcHiG5oC43ohK0bfMXAAjxs6D1tVTwmYDx9rG5JszQU+FO8ngiTgdp2eHcTvwr53hNb/C+aV4Ih1vDlBmNjJtTNVicze/UFrxCNaDZ/SX5ntborauidvQOvtdg104hGJYy4QOshHZ50itQe5BMQLNlQbHFQ9DTiClf5ooIclV2++EwD+rsi+GQEBpE0iOFE9w+cKLj3WA0LFXYsrUX9hEZrN7XPXtpMvjbYO5y8oS8hHxNeKQSwykMy7O7LP2O3fU5iG0GHu61Rb74Jy+GWL/kh3E/PywRCjwGKZc X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7b2384b0-5692-4966-c863-08d9696b3747 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Aug 2021 14:59:06.1794 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: uHaLM/zdCkbPOnD7fX8uxlAIvvw3M6FGyh7VjUwmeXdlmv+VtqNXIYP9pc8/7dOvFQizI31TYasY57DHZaGgfRb8BlBbGGHPbL0a0BCWJ/g= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB4881 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10089 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 bulkscore=0 malwarescore=0 spamscore=0 adultscore=0 mlxlogscore=999 suspectscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108270092 X-Proofpoint-GUID: 2GCEytDOWoXQKD-djmLOGrlIJMkQQyH8 X-Proofpoint-ORIG-GUID: 2GCEytDOWoXQKD-djmLOGrlIJMkQQyH8 Use try_grab_compound_head() for device-dax GUP when configured with a compound devmap. Rather than incrementing the refcount for each page, do one atomic addition for all the pages to be pinned. Performance measured by gup_benchmark improves considerably get_user_pages_fast() and pin_user_pages_fast() with NVDIMMs: $ gup_test -f /dev/dax1.0 -m 16384 -r 10 -S [-u,-a] -n 512 -w (get_user_pages_fast 2M pages) ~59 ms -> ~6.1 ms (pin_user_pages_fast 2M pages) ~87 ms -> ~6.2 ms [altmap] (get_user_pages_fast 2M pages) ~494 ms -> ~9 ms (pin_user_pages_fast 2M pages) ~494 ms -> ~10 ms $ gup_test -f /dev/dax1.0 -m 129022 -r 10 -S [-u,-a] -n 512 -w (get_user_pages_fast 2M pages) ~492 ms -> ~49 ms (pin_user_pages_fast 2M pages) ~493 ms -> ~50 ms [altmap with -m 127004] (get_user_pages_fast 2M pages) ~3.91 sec -> ~70 ms (pin_user_pages_fast 2M pages) ~3.97 sec -> ~74 ms Signed-off-by: Joao Martins Reviewed-by: Dan Williams --- mm/gup.c | 51 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 31 insertions(+), 20 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 7a406d79bd2e..0741d2c0ba5e 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2234,17 +2234,30 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, } #endif /* CONFIG_ARCH_HAS_PTE_SPECIAL */ + +static int record_subpages(struct page *page, unsigned long addr, + unsigned long end, struct page **pages) +{ + int nr; + + for (nr = 0; addr != end; addr += PAGE_SIZE) + pages[nr++] = page++; + + return nr; +} + #if defined(CONFIG_ARCH_HAS_PTE_DEVMAP) && defined(CONFIG_TRANSPARENT_HUGEPAGE) static int __gup_device_huge(unsigned long pfn, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { - int nr_start = *nr; + int refs, nr_start = *nr; struct dev_pagemap *pgmap = NULL; int ret = 1; do { - struct page *page = pfn_to_page(pfn); + struct page *head, *page = pfn_to_page(pfn); + unsigned long next = addr + PAGE_SIZE; pgmap = get_dev_pagemap(pfn, pgmap); if (unlikely(!pgmap)) { @@ -2252,16 +2265,25 @@ static int __gup_device_huge(unsigned long pfn, unsigned long addr, ret = 0; break; } - SetPageReferenced(page); - pages[*nr] = page; - if (unlikely(!try_grab_page(page, flags))) { - undo_dev_pagemap(nr, nr_start, flags, pages); + + head = compound_head(page); + /* @end is assumed to be limited at most one compound page */ + if (PageHead(head)) + next = end; + refs = record_subpages(page, addr, next, pages + *nr); + + SetPageReferenced(head); + if (unlikely(!try_grab_compound_head(head, refs, flags))) { + if (PageHead(head)) + ClearPageReferenced(head); + else + undo_dev_pagemap(nr, nr_start, flags, pages); ret = 0; break; } - (*nr)++; - pfn++; - } while (addr += PAGE_SIZE, addr != end); + *nr += refs; + pfn += refs; + } while (addr += (refs << PAGE_SHIFT), addr != end); put_dev_pagemap(pgmap); return ret; @@ -2320,17 +2342,6 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr, } #endif -static int record_subpages(struct page *page, unsigned long addr, - unsigned long end, struct page **pages) -{ - int nr; - - for (nr = 0; addr != end; addr += PAGE_SIZE) - pages[nr++] = page++; - - return nr; -} - #ifdef CONFIG_ARCH_HAS_HUGEPD static unsigned long hugepte_addr_end(unsigned long addr, unsigned long end, unsigned long sz) From patchwork Fri Aug 27 14:58:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12462219 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DF90C3FD1 for ; Fri, 27 Aug 2021 14:59:24 +0000 (UTC) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 17RDWEDN013579; Fri, 27 Aug 2021 14:59:15 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=jTz9Gcttrup59NqSwIQTfNduJVtZetbNHP5heiIiN90=; b=TAcAxocYU9q3cpG+mMWuHbMLVI8dPtMjjQLNMlIxoIUxCBgH5q9I/4z2NRvQ+6ZenR7w JsYPz1svxUu7s11rMXcCBSqZKTci1X4osbCGyyf6p3+Hd1nG0VQ+WAgjCpceKhQ/NlBm wfYFSTWiP0k28whHqPpqXdPVfPeTDjwjM8UUE0VOirB4kF738GwqjIHIlBRnY9+ZUZfN ek+xo/SybtAJbJkzuWYRuAF/W364A1LYAg+eootr3v5Xvrz5iVSTRubT6dZyOEqFMhcu HM6CIh0s87OWr1/JsnG7keGqiisMPlAyLvTJhs3Qy6LZ5Wug8yZsMn2v/CX7Piks/xDI YA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=jTz9Gcttrup59NqSwIQTfNduJVtZetbNHP5heiIiN90=; b=afZ49e46DljS0YTeWYQMcL6kYKBOo6S2oaAt1vMQTG/YZkDBslLRLR3xEYURLkNj1/DM 6fIRjN8ZJo3af72EhNEqg5ISg/EhSCaO/fkViWR09im4a8MHv/uBjPEOBssHfESCZ4UC z/R7RWYOnv9fkbDv5jpzfOuBfKzqxtTWf5eNsC69OGlkgEKhieSe6JjHitJDazXTFvEC UhHokc2VXED+rYGkdq8s3/rJUv4e+NTwyzADghiukPpyxj4rJQX+pi5lUA3Bk8zk6ixb yzE6lLKHuSDtL4TeTUURndWMSg76BmCDN95gk+Kxtsh0f4kBb7i8BxnN8k47Gg7xlM0e 6g== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 3ap552bsth-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:59:15 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17REpZPm152641; Fri, 27 Aug 2021 14:59:13 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2169.outbound.protection.outlook.com [104.47.56.169]) by aserp3030.oracle.com with ESMTP id 3ajqhmq1q8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:59:13 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CHUFiD3/fy00bi3MNQuHZrxyYGxX613fPzvKVAcj6BQrRtDLUQrMFkG0J2KXNDW9IRmpUQa3TmAzT3y+vLH2MXv1p5+7+VmikwWXp+3lGoGoSBujO/af8lAGeD7670QwJ/iIFJ1GUzQFIqp+tHLtCw7OUagFg9ryEru8G9s2T0/Bxa3wbkE/A966cVHzfan5Kb3GasPHP+KYtyndf8UzKar0guvXYCsmq6b3LpAPEermQ3FJY2xobIpYyWZXRtnMFuPjLJ7oqvKqhFeAwvIq9nYrIzRniv533kPiO5oavqWcCx0hwsOd4/I7xNNU+dR2WLyCty5YMMgT//K3jkxAKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jTz9Gcttrup59NqSwIQTfNduJVtZetbNHP5heiIiN90=; b=e39ZT16nN20FvSH//shT9q3MrBpTPzRP/NxncHC/oFKZosXaf3chhtVjzAXfLwI/02xOfgUfAsw/1zofeZg4Hc7y8OpR2y3ektWKRrFwbbiHo9BEtXJOTZuvLoQQrf9O97Kp+gSM6SXKcAbkYOBSYFdQ0/V2XXUJirdYH+HVy8lwpxyn1DuOon7D7SMxDlRlk/wfOoipjZy+VbXyv6opqnUgII/drK63PRQzrCsbdBgsSydbGuHXzN4UEW8ojAyPHhfxmTvr7ohp1PvLZ3Sq13BIqjLSvS86eUb93IJID+pg48YmKMjTyqWGeQzfPVZOa5nSON/O2ioA2JODsbGWpA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jTz9Gcttrup59NqSwIQTfNduJVtZetbNHP5heiIiN90=; b=GiaxpL1C+ArNittB3247YK+7Q6p2su8MYXPswI0VZgzrqah2pcLAmVGmnOWH0fXJw803G3Vg3KedjTF7ORLlC6StnZQeYxT3QSpfZF+twvPidA+L5nyuKkpeBae3MeFP51RwuywrbfTqhgkhx27gdFUAXJ+y+xs8aanLuiOkHX0= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BLAPR10MB4881.namprd10.prod.outlook.com (2603:10b6:208:327::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4436.24; Fri, 27 Aug 2021 14:59:09 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198%5]) with mapi id 15.20.4457.020; Fri, 27 Aug 2021 14:59:09 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , Christoph Hellwig , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v4 09/14] mm/sparse-vmemmap: add a pgmap argument to section activation Date: Fri, 27 Aug 2021 15:58:14 +0100 Message-Id: <20210827145819.16471-10-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210827145819.16471-1-joao.m.martins@oracle.com> References: <20210827145819.16471-1-joao.m.martins@oracle.com> X-ClientProxiedBy: LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (138.3.204.8) by LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4457.20 via Frontend Transport; Fri, 27 Aug 2021 14:59:06 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 74bfb882-4a08-4672-6883-08d9696b3920 X-MS-TrafficTypeDiagnostic: BLAPR10MB4881: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4714; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: gEyD3+Z+8WhaD9jeeEukl1ad1EzBfpsmiS2ECkciI4vSj6f0ejK+ahdEdXkBbIxOUvMDA5i8YClhZfzgzCXbSn4jSOh++y0m86XO5boJt1tFgdVAWma7laCIShOLwyYZ391/1oy/OHEHDWIgASR6+5/LIOhMgq+OQTZijT5LdY5JxWabHhTL+sSC4aPRQSDJg7uEqJS/OH6XNsxeyxqbEx7nUYQdZlrR3bu/16BmbmXf3c3S4IjRbA4OChVn4SJMtejvz0shElufem4hUGIWbRYlXgwEtWhx9YeRtcMWaRvYlko7e4qIFlzad8d5DvfVqPGQSKCAcQZWSkmO4eq2OhbCrTyP6gMmBmQoLQGmIkB1ecVOMG2KeUU64Iqg03xeyB3W9d7XGItPNQwq2IkL+SLlkQKUlNv1n+Nj4OQUK1xtMz6bkIHuOL7Qf/+YS2xfGB2WVX+17tOv1/pl57YLhN6sdx2kREeMMZtkJOyPvQkGWXTBSsNMtKSSt8I9owqZIon6vo1t4FPK/2K5SqwIZq5jGS8mMRpF8uIva9BLYF7mC9ps1CjD1mfqR5qbY//Ce5S7JdMZmQtaoaf7YPyC8aVhNSsp3zIzY9vExbamp/IBRYsF+2KSt/jfj4xW6eWaskixC9M/IyahIzlootxvrIwRzMle23+fgcLxIO+JEcsMZrK+a5RNbNq0kqN0lf5W/aV6sjydnuExmU7bhmulbUlp+unqfLO8MgwwIZC/2c619yfJ/WQLFZd2fPWFUbBrcbrQIRIXz6ElqvupzKqtK9Ud9AQzj2AvTTcRwQ7NItA= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(396003)(376002)(366004)(136003)(346002)(39860400002)(7696005)(2906002)(83380400001)(38350700002)(38100700002)(52116002)(966005)(6666004)(66946007)(8936002)(478600001)(7416002)(107886003)(86362001)(36756003)(6486002)(316002)(1076003)(8676002)(103116003)(26005)(54906003)(956004)(4326008)(2616005)(5660300002)(66476007)(66556008)(6916009)(186003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: E4+440vBJ+vuIGhYUwLx77Yg5aLSg5LGR9JizBPxIQqJlomTbmkmmBLLNGKEUgdLkGNovq1lAyRWRTQ2eqKoewa0kGSKwmATU83tqnZNbLiJqVK/A6H7ROXcvF+APoX0aAc8vjTJiTNDCTSViypBQmYopk03OECnZPMW6AWvBK/++8s4Sxcs5RrWgjWz74NxI2uC663iUkbDSbZXkkjMpisuEepGa9uO5AiKHm1u+QIOCPBDgh0ownE1aTfbGOB1lQdRlfsF1TIEm7AZbL6Ptnb1Hs9ef144c1DSTNg3FW+eny4nWfEWbcIgVSSR9eUBrnQfLcRswfoqO+xb8M7+SGlSJZpcq6JmtHFwxM/PPcAH7wup56WkfOYTHtjKk9BqiOoQLbY3JxBAJH+URABer+6xIKi5o8BkTi1kJ0kIuyA7GdFkkL6tutIF3Kbay9jJuxzdBRbd9BtG/XAwnHlgBmAqO6AkO7e3UmY4b8cg3bQ1ymNOtX5DId577dXZcAwIrM3kHp6FigqkCZ6JP4Oy+AvVonbcbcGEK11k50cHD9DKFXcZRvKp+40QvV0BFxZ5JgDFpfP3vfxwNmltFQF/84sPZ3ny0iFlvSAmPL4F9VeSVJF8+X+ZWAOkssS4LENEnDk4ZNeR4Gwe3P3X8oEBfjdQk0f4UmkXXPojwi1S3iy/VUgQvUj1oMPjhYZOWN6As6G57+6wT/vP/9cySOtbUTowbvCr15xzA+/G42CNyoDR7wMa+zDHetNctaaU+uRyP45563fE/qWM+mP3JBgQIlWk/O1pFoyAJD8AAZP8/FlAeUBYhhkwWRnxQhNbbap2OY1yYv4qImhHOb5VIdtE/UqEMISS3/EBZvqbOYWLYQgpwm+i31HiTdJrBYwlPPYxCqi2Mnt2FAXZZoDErSj6HD9QGlZ7JER6Butr7rmIgdew5DQAfXJAv0oVRlC8V2i8cCGbvdNFZI8tpjhjk3hkrZ4W7DB6gD6KzlP2Gzsm4zho5pQ/qnUGgmboabZOt7LU8An+poIcnnDHsAr0q8NuP0JjIIg84gPGesYzEj8yCZBk5opGMQWL9/3fgjez/xJzqaLC3PJ16yJy/9gtnMUUoqnkXhIDUbfj9BXhPy6FN1EZjNQc9MJQnMOUSBN9u7dK3WGBr+OCCc8cB3Gvj65OLWU9KxncgnIk9+SyHZFS0ubLqdKSKsDFlEaECyKQjtfg1R+kEQdwtEFW0/JnCwQk/Gaujm1jPT5wjFsxJ3hhH0F6RSs/ITRz+ZSQuTVJwO9QLJ5RukJQElr/p2HOYCeJq3RleNYJXP/q5SUoxJMKBrs7Ip0WsbQHZApLMa4NOQM1 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 74bfb882-4a08-4672-6883-08d9696b3920 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Aug 2021 14:59:09.4052 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 9Wy5H6KCNKoDjwYzdQIZGeATRolB0rzGesFrilpPNpb3EyHMOmeQFP7MugNDubFYW/bf83PCCVmhIbyIxAl2/msYifB7dAxiCyoL5aj36rY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB4881 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10089 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 bulkscore=0 malwarescore=0 spamscore=0 adultscore=0 mlxlogscore=999 suspectscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108270092 X-Proofpoint-GUID: EzsnpvNzBUapVABHvcnevMXOOidRwAT6 X-Proofpoint-ORIG-GUID: EzsnpvNzBUapVABHvcnevMXOOidRwAT6 In support of using compound pages for devmap mappings, plumb the pgmap down to the vmemmap_populate implementation. Note that while altmap is retrievable from pgmap the memory hotplug code passes altmap without pgmap[*], so both need to be independently plumbed. So in addition to @altmap, pass @pgmap to sparse section populate functions namely: sparse_add_section section_activate populate_section_memmap __populate_section_memmap Passing @pgmap allows __populate_section_memmap() to both fetch the geometry in which memmap metadata is created for and also to let sparse-vmemmap fetch pgmap ranges to co-relate to a given section and pick whether to just reuse tail pages from past onlined sections. While at it, fix the kdoc for @altmap for sparse_add_section(). [*] https://lore.kernel.org/linux-mm/20210319092635.6214-1-osalvador@suse.de/ Signed-off-by: Joao Martins Reviewed-by: Dan Williams --- include/linux/memory_hotplug.h | 5 ++++- include/linux/mm.h | 3 ++- mm/memory_hotplug.c | 3 ++- mm/sparse-vmemmap.c | 3 ++- mm/sparse.c | 26 ++++++++++++++++---------- 5 files changed, 26 insertions(+), 14 deletions(-) diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h index e5a867c950b2..a5e648ae86e9 100644 --- a/include/linux/memory_hotplug.h +++ b/include/linux/memory_hotplug.h @@ -15,6 +15,7 @@ struct memory_block; struct memory_group; struct resource; struct vmem_altmap; +struct dev_pagemap; #ifdef CONFIG_MEMORY_HOTPLUG struct page *pfn_to_online_page(unsigned long pfn); @@ -66,6 +67,7 @@ typedef int __bitwise mhp_t; struct mhp_params { struct vmem_altmap *altmap; pgprot_t pgprot; + struct dev_pagemap *pgmap; }; bool mhp_range_allowed(u64 start, u64 size, bool need_mapping); @@ -342,7 +344,8 @@ extern void remove_pfn_range_from_zone(struct zone *zone, unsigned long nr_pages); extern bool is_memblock_offlined(struct memory_block *mem); extern int sparse_add_section(int nid, unsigned long pfn, - unsigned long nr_pages, struct vmem_altmap *altmap); + unsigned long nr_pages, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap); extern void sparse_remove_section(struct mem_section *ms, unsigned long pfn, unsigned long nr_pages, unsigned long map_offset, struct vmem_altmap *altmap); diff --git a/include/linux/mm.h b/include/linux/mm.h index a3cc83d64564..4fca4942c0ab 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3167,7 +3167,8 @@ int vmemmap_remap_alloc(unsigned long start, unsigned long end, void *sparse_buffer_alloc(unsigned long size); struct page * __populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap); + unsigned long nr_pages, int nid, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap); pgd_t *vmemmap_pgd_populate(unsigned long addr, int node); p4d_t *vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node); pud_t *vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node); diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 0488eed3327c..81d36de86842 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -335,7 +335,8 @@ int __ref __add_pages(int nid, unsigned long pfn, unsigned long nr_pages, /* Select all remaining pages up to the next section boundary */ cur_nr_pages = min(end_pfn - pfn, SECTION_ALIGN_UP(pfn + 1) - pfn); - err = sparse_add_section(nid, pfn, cur_nr_pages, altmap); + err = sparse_add_section(nid, pfn, cur_nr_pages, altmap, + params->pgmap); if (err) break; cond_resched(); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index bdce883f9286..80d3ba30d345 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -603,7 +603,8 @@ int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, } struct page * __meminit __populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap) + unsigned long nr_pages, int nid, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { unsigned long start = (unsigned long) pfn_to_page(pfn); unsigned long end = start + nr_pages * sizeof(struct page); diff --git a/mm/sparse.c b/mm/sparse.c index 120bc8ea5293..122ac881f27b 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -427,7 +427,8 @@ static unsigned long __init section_map_size(void) } struct page __init *__populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap) + unsigned long nr_pages, int nid, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { unsigned long size = section_map_size(); struct page *map = sparse_buffer_alloc(size); @@ -524,7 +525,7 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, break; map = __populate_section_memmap(pfn, PAGES_PER_SECTION, - nid, NULL); + nid, NULL, NULL); if (!map) { pr_err("%s: node[%d] memory map backing failed. Some memory will not be available.", __func__, nid); @@ -629,9 +630,10 @@ void offline_mem_sections(unsigned long start_pfn, unsigned long end_pfn) #ifdef CONFIG_SPARSEMEM_VMEMMAP static struct page * __meminit populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap) + unsigned long nr_pages, int nid, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { - return __populate_section_memmap(pfn, nr_pages, nid, altmap); + return __populate_section_memmap(pfn, nr_pages, nid, altmap, pgmap); } static void depopulate_section_memmap(unsigned long pfn, unsigned long nr_pages, @@ -700,7 +702,8 @@ static int fill_subsection_map(unsigned long pfn, unsigned long nr_pages) } #else struct page * __meminit populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap) + unsigned long nr_pages, int nid, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { return kvmalloc_node(array_size(sizeof(struct page), PAGES_PER_SECTION), GFP_KERNEL, nid); @@ -823,7 +826,8 @@ static void section_deactivate(unsigned long pfn, unsigned long nr_pages, } static struct page * __meminit section_activate(int nid, unsigned long pfn, - unsigned long nr_pages, struct vmem_altmap *altmap) + unsigned long nr_pages, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { struct mem_section *ms = __pfn_to_section(pfn); struct mem_section_usage *usage = NULL; @@ -855,7 +859,7 @@ static struct page * __meminit section_activate(int nid, unsigned long pfn, if (nr_pages < PAGES_PER_SECTION && early_section(ms)) return pfn_to_page(pfn); - memmap = populate_section_memmap(pfn, nr_pages, nid, altmap); + memmap = populate_section_memmap(pfn, nr_pages, nid, altmap, pgmap); if (!memmap) { section_deactivate(pfn, nr_pages, altmap); return ERR_PTR(-ENOMEM); @@ -869,7 +873,8 @@ static struct page * __meminit section_activate(int nid, unsigned long pfn, * @nid: The node to add section on * @start_pfn: start pfn of the memory range * @nr_pages: number of pfns to add in the section - * @altmap: device page map + * @altmap: alternate pfns to allocate the memmap backing store + * @pgmap: alternate compound page geometry for devmap mappings * * This is only intended for hotplug. * @@ -883,7 +888,8 @@ static struct page * __meminit section_activate(int nid, unsigned long pfn, * * -ENOMEM - Out of memory. */ int __meminit sparse_add_section(int nid, unsigned long start_pfn, - unsigned long nr_pages, struct vmem_altmap *altmap) + unsigned long nr_pages, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { unsigned long section_nr = pfn_to_section_nr(start_pfn); struct mem_section *ms; @@ -894,7 +900,7 @@ int __meminit sparse_add_section(int nid, unsigned long start_pfn, if (ret < 0) return ret; - memmap = section_activate(nid, start_pfn, nr_pages, altmap); + memmap = section_activate(nid, start_pfn, nr_pages, altmap, pgmap); if (IS_ERR(memmap)) return PTR_ERR(memmap); From patchwork Fri Aug 27 14:58:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12462215 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9A67F3FCF for ; Fri, 27 Aug 2021 14:59:23 +0000 (UTC) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.0.43) with SMTP id 17RDWLfw010418; Fri, 27 Aug 2021 14:59:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=q84FdhJkLd7/8jzcasxBlvgfiKRkbuLEBL5wIpeF90U=; b=IVtFfwGmOn3Vr9SkzVDlYUOayTisZFwdE8lJF01ZlQ/jNc3iPvLaXxmrlg4/EAzHD1uX 2EfQ9GDAiHOdqei1cOv4lE+43H9V1mSMx7bQn1jRnEwBDyu9ZM/9/+cCqGrE4WLnyhc4 d+Q8lnBOYNUo0sncqRdAM9ggz+PBRnQLaEqsSmSik4PlZ8sAvJfWy9MQuQgmfQmbDMF9 4VZdv7jFJ/KfGMYr5ta9VEH1AMhBaL+H7VdA3h6bbd0EsE2ywfHFpufLcnyVHQO4QCjT K19TA+IkNH5wL14lWZvXQQwKrgHgoMxK7DR2WggFoppCNN918SahCWLt3n3UYrt+Xkr8 ng== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=q84FdhJkLd7/8jzcasxBlvgfiKRkbuLEBL5wIpeF90U=; b=SOT8url4BFkyWZzEBCPTgyeGq7uDwWsSfFqqTsRJrrYAejoEHtzMDHs1ro8+eUBsICyl 2mrmPuaAcm7I3d6//eEwEmbHTFj3F3XNTbN2LNl8vuoHWZKlX6iBMKS0Yoh36JwzxUhi 8JCtd+KjaFWpek+IeYfqA6fqrbv+ER1ckNpCoMEqNditZ7hQbxY2+c5fQ4Zhl6KEjQW2 PtQXagGLkAnTk0NWGfjJep8mkoUoph08aINDHqDLL7LpXGah4ORABAvMSkSwD+HInVFv o+fOLwI405u1DvJ10IQ6CSqI/D2CoXxIOznWSItH3tkBKXGqHjQUSNwkoKHDiBRK3u7E yg== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by mx0b-00069f02.pphosted.com with ESMTP id 3ap3eauyms-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:59:16 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17REpOPZ187123; Fri, 27 Aug 2021 14:59:16 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2176.outbound.protection.outlook.com [104.47.56.176]) by aserp3020.oracle.com with ESMTP id 3ajsabbqqn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:59:15 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gV4dxza5R3ZwNNLZlxd6ozE2RtQy3UE71t50Y8S1xu/ukVxzpKjdTL9vneGCubO1wTIxwE4t7G+pskbXLecMu33r8fuZJ8+J2qUoiv78cip4ImNoe2cH9QE22aaSMd97njp90oB/LQyiXPLNWBqq5BHr2y5Lftuu1e772WcnvPMAxFX0hausod+V5agl2DUE5UkQjc1lxCtrwXS47oKu8a7abBL+EDlw8nyNw8JKw3SDRrpAvP9aCFf7qt30EQ9BgbhG3KYaPWnUpotRaVbvxZawCbXkcb7oXbgHRd6FBaw6OpYlEpXROnlDSjqjS2jhd899Ll5W8UW7oQ19gpDoAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=q84FdhJkLd7/8jzcasxBlvgfiKRkbuLEBL5wIpeF90U=; b=hG1l3YLwNIpOoxR20YMXp3hiJfvrZ4WevTQwRqa6qm3FGE38oy21z5etjHX39036gr2ajBIjt9sQgWrY/0wp+yJzmIP4FPkHMb0Ezi1YrrtsR5/t90qaln6MoGCP+h96G7afz3PtwQ/d1EqQ1u3yl4h+U8e9Mql85Dcjc+cvusPM8+D6Ip6JFIfuVeaEvOt5OxHZ0WMoblFduEOZ0McZyfUbLM0trLt53J+iYpcmuHktey4m6U90IiSLBIIMtF3mFjNJ7pl5MSEhfi7aX49yxJfHpxx2DfL7l/umnq31xq2cNGQu/RJQ0BJtEfW4GSmJTT8c1yNf6oIB9WF/tyBZng== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=q84FdhJkLd7/8jzcasxBlvgfiKRkbuLEBL5wIpeF90U=; b=GNbqSeqWWVWo3navKtwNNwiaQ7YDpjWTkajiFo5f4qrBwCpd89JeGORp7XfMWMl41DSTDTnSIeSt7ZnESPU2suuayFmCSodYqj8jI71rWHSUPCJAF/ze8IRUJmtBZ6euvhbUDFm2DVf6uvMkE8tGXUO+cZ9fltKvph4zfQNU654= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BLAPR10MB4881.namprd10.prod.outlook.com (2603:10b6:208:327::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4436.24; Fri, 27 Aug 2021 14:59:12 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198%5]) with mapi id 15.20.4457.020; Fri, 27 Aug 2021 14:59:12 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , Christoph Hellwig , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v4 10/14] mm/sparse-vmemmap: refactor core of vmemmap_populate_basepages() to helper Date: Fri, 27 Aug 2021 15:58:15 +0100 Message-Id: <20210827145819.16471-11-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210827145819.16471-1-joao.m.martins@oracle.com> References: <20210827145819.16471-1-joao.m.martins@oracle.com> X-ClientProxiedBy: LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (138.3.204.8) by LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4457.20 via Frontend Transport; Fri, 27 Aug 2021 14:59:09 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d62540da-e90f-485b-134c-08d9696b3b14 X-MS-TrafficTypeDiagnostic: BLAPR10MB4881: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3631; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: iUFaFpeqd48GFEgFSrCWAo4QoPlhLFCGNfQW0so9dhHhPy2s1gDc5fLdx1qGNIQ93m1fk5dXxd0HKjDpuaFDpqEivg5FNtTj4o97jW6sJ33zRuXut2HLpHBuUMchqGokvlDrjJOg/M/DvmHIs171J6iUm6Lcy+ZD3XezXbV+kqzw3OrFmPFSF1Zp+1UxjYwDZjhfMUsD0+M2a0x3LVYUBBGTspVAwJNjWwmBQtWF8yZphNe73ae3ld87hTi6UWjxHj9FHw+pzeTPbP9Kvrcaj9fAI77F8699/gAP3vpC46b8GilR99qDHHW3dlVdyjDU9oSD54AOAoJtNlGSwpPTPZTjHWzpE1VJ4F1DlcFaHi6qxFUQeubQdYZZqNW5wsDTDx2nXElPn6o23eCK+LLPCejSeLcjACbzXQ/siXwpG7IaysHO0iTJiCAEhS/E059bvn9q+dn8RVb+y2cB6n/AzCXhiY/wtgUm2JRanxkQUtc7SLYTPoSLAuRIyEzU4FFOQLZ5TCxqWqFa9hLm74IvkLHkCeB7J9MWmt7+5FjQ4vojbL0786lc/baewKRrRBk9ivCLEnVzLwWl57UR8vrNlyhT9QOAORCw0gpdXxWR+TU252V8nF42cWO+tOUxHOqe7xhZCXryhiZ1soaaW84yhMd8tQwDd8JB7dEy3nmyLClvjzpW+Bae39lqQ6dT65cVMEkR4dg3DbJT5OK6F/CdtQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(396003)(376002)(366004)(136003)(346002)(39860400002)(7696005)(2906002)(83380400001)(38350700002)(38100700002)(52116002)(6666004)(66946007)(8936002)(478600001)(7416002)(107886003)(86362001)(36756003)(6486002)(316002)(1076003)(8676002)(103116003)(26005)(54906003)(956004)(4326008)(2616005)(5660300002)(66476007)(66556008)(6916009)(186003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 5nxZkwKelwuj2bJzaFkb8gxbB2FNd2t6ey2aJ0fvac0NyiBLnG0+SMutRkh7v3ZQFRNbQNbNhDWi9yoXfx3HmrPaNG4WRpvAB6H2uP8+ux4gBD51v9sfxgAitfTs5VFrlLcexgVZqaT7Z4cKstz/I/ND0g60Kyii48ktc/KigWQkuQ06dqiDdT90xIpW6tQTavs6S4rTDSG9EEQ3743QLTRHC7UJgJUvAqqq2yaP5HGn3q/z11P3CacpvVy9J6vvW8jJmWvH4r+nrf2lsC8/4Hp1qjSted0w1jifw/lwkCoL+wQArgB203eQkHZ83oeT77pxNKGd4gonyqlCGZRRiRhxC4DjsZMfuIpGyaHnJpbuXjW0PpnUyej706OnnYOvAAIS0/n9XpjyDxJuBsiIOVLBR6cNBXbDIUAT/2gl/YaFbOnvfV/Ryh/IQPW4ct7waA4zuHMa5eoDgSqdYfT47qNY86ddxLNNb7oPhVfSywzy8dFdtfVjHKj7qJaxG5rrcJRmcL7SMjsGHzJtvuHTNmuMwFGUm8Q7akfWV+prugSa1kgEkKUBqqdaZBeWwXoBcVCj1R3ZrHu3vlPmc/VUyLV5roceEW+3X/Fd8w4JNVBjUPOcYJZgK1k/1U7L53CsY3e1tmsKh7r8LoXtcyZJlS/zyStZAgHB7/gsUpkL6HkF6XPTROmLwwnFMXKXuL7mzfxZF8lhF1LLf3l5oQvzvjbZrqdjBbwHK2+I/T8kVTdumfLcr3klwR8QyVHgqR4sxEbncmcj+vtAUK1VgQ2hPju/c7pDAf7R6nn5y8XoqrzOHmh/hbwZbrX4vj3/PbKXEdC5J9HcsBBDTagvWp2++1XQeLjRXJqi0D+e1cex+K9tfbdc+iFi7P9Em0CrV66Ax9g+umpojP31nGXyW+jJmqtgCwfpH5EGJTxzJJTYMQYDnbLxsdYvm3d/pxIOQzI2su6vF3DdePv6Qkw9yljvVA6ZwhQ9+1os6KxVvmPWawYeMk5X9QRw6Ut9TtdkUJEsjvaiDmAClvm7vDofsGM1YT9G/JSb3ma0b4tnR+wNw5R8vSbAnUZJQc8Xv7tvwAz7Hka0QpL34SsTLuyckBbgLspWoTuOzXOzQ1XGkd0APBG0c3Y5C12SUHHm6SOnGNztpTctmANvMnAw/9SOy3slvrWbRG74LolPhLFjq79zv1vPoj5AmwU3DyBeaiveKpp1vn2ifICfjh7r5FSstFCx1XHuNzaTJ2c1eFkmZkOCEPPs+LTblxvTrNvCDcgygqOzBjCxG66TRc9yy/soDWo6WyMjSPZCpI1AmuoyOmfmDjppF1mtmyMryXukWGqenvOH X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: d62540da-e90f-485b-134c-08d9696b3b14 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Aug 2021 14:59:12.5524 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: GqnCklPomZUdlb8yUEpesxLUpOgVQJhmQtneQuw5KkAAfqVzQFdenj6xWWAsjManLmMuJyt84hzOB98YBDcRXXlsO6LInaKX4uYI06x4RKQ= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB4881 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10089 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 mlxlogscore=999 bulkscore=0 mlxscore=0 adultscore=0 malwarescore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108270092 X-Proofpoint-ORIG-GUID: pq2xjpxyrNCVuHZbn7Qwud3Py5nEtPcL X-Proofpoint-GUID: pq2xjpxyrNCVuHZbn7Qwud3Py5nEtPcL In preparation for describing a memmap with compound pages, move the actual pte population logic into a separate function vmemmap_populate_address() and have vmemmap_populate_basepages() walk through all base pages it needs to populate. Signed-off-by: Joao Martins --- mm/sparse-vmemmap.c | 51 ++++++++++++++++++++++++++++----------------- 1 file changed, 32 insertions(+), 19 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 80d3ba30d345..58e8e77bd5b5 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -570,33 +570,46 @@ pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) return pgd; } -int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, - int node, struct vmem_altmap *altmap) +static int __meminit vmemmap_populate_address(unsigned long addr, int node, + struct vmem_altmap *altmap) { - unsigned long addr = start; pgd_t *pgd; p4d_t *p4d; pud_t *pud; pmd_t *pmd; pte_t *pte; + pgd = vmemmap_pgd_populate(addr, node); + if (!pgd) + return -ENOMEM; + p4d = vmemmap_p4d_populate(pgd, addr, node); + if (!p4d) + return -ENOMEM; + pud = vmemmap_pud_populate(p4d, addr, node); + if (!pud) + return -ENOMEM; + pmd = vmemmap_pmd_populate(pud, addr, node); + if (!pmd) + return -ENOMEM; + pte = vmemmap_pte_populate(pmd, addr, node, altmap); + if (!pte) + return -ENOMEM; + vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); + + return 0; +} + +int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, + int node, struct vmem_altmap *altmap) +{ + unsigned long addr = start; + int rc; + for (; addr < end; addr += PAGE_SIZE) { - pgd = vmemmap_pgd_populate(addr, node); - if (!pgd) - return -ENOMEM; - p4d = vmemmap_p4d_populate(pgd, addr, node); - if (!p4d) - return -ENOMEM; - pud = vmemmap_pud_populate(p4d, addr, node); - if (!pud) - return -ENOMEM; - pmd = vmemmap_pmd_populate(pud, addr, node); - if (!pmd) - return -ENOMEM; - pte = vmemmap_pte_populate(pmd, addr, node, altmap); - if (!pte) - return -ENOMEM; - vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); + rc = vmemmap_populate_address(addr, node, altmap); + if (rc) + return rc; + } return 0; From patchwork Fri Aug 27 14:58:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12462231 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A7C8D3FE4 for ; Fri, 27 Aug 2021 14:59:27 +0000 (UTC) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.0.43) with SMTP id 17RDWLg1010418; Fri, 27 Aug 2021 14:59:20 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=h8zX5EwGpoSKxgJhw5pcbNisWYyaf2L4YDqtha8/xJ4=; b=tlOe1ZzwjfbZt9k7NmCglC3ezkA8D1ScBRhUcXR5VNvsdGdAl7R6shl/gfLw91wlIcgC nner4K2U/bsPd6hw9B/HuzBePezuvF/eskltvo2fN/0CGdHQz1y3jq8RHVZbInwK/Lqq Qk728nQ5CPiCtAUHLESuq2DcrpGJ2P5lsBck3Z09U5t+7N/wljQ1/c00IQjGPyfLshI7 bhZav1I3Fa7c4BXmoXSLtv4zLdZgw9gmznKkG+9CrZspFLMFxiAB9wc2IFWsk3FJuL4T SnjBbQPOF8qX94ZPxLI99Rjw3pAi8INrtjqWYjR3A+ZfbvyOm+vr6wmtwoLCvbEqbRU3 IA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=h8zX5EwGpoSKxgJhw5pcbNisWYyaf2L4YDqtha8/xJ4=; b=YobpENilg0pRjP75XsR7OZeYHGbPFRb5tPhxd+JGUVXif6VOw7Ojbkda+Ogak8AlPw7W zj1InWQVGKk+AjRuvxVn+E+cK3gwrr0vOaxiI/0N3dzPstjTaX3OPFJpXzG7StHHbKJF oxHw4IDIoBtGviAAwfZSCnc3bbF/JN97J+MpZAJc70JpkFAzuAmm5CSyxRzOnlfmN+C+ 8YXOj7rU62FGYJLuN7n+mj9rzjMgst2JaV/iJcvVPhNDLV43hanR7dHjw4ljf2i2/7XU /I/QewFL5vIQIyWiI1zAfoeCSmAp31aOklfD/fk0vVOzEMQm2O4x4ZHPtqu2v/vHFp+E CQ== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 3ap3eauymy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:59:20 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17REpbuL156012; Fri, 27 Aug 2021 14:59:19 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2170.outbound.protection.outlook.com [104.47.56.170]) by userp3020.oracle.com with ESMTP id 3akb9293n7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:59:19 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iFrJpan+g2WCLKFk+K1pra5N7I/61noCcp5lFVNzFTv0FJbHGbNScPY6VDB7V2JQiyZ5WvoUHG1quMH/mNGDQidTXA9wp1hhcOfCAKKFX3HINsdhCjNH+VulHiAA8hBqZoUHpFjbi/53ULyNyIMFMFyCCcpOHt0VKtXYlpErubYHjN3KUGtWmgr2MqhlqLQHiKqiYYAXz+b9WXC6iyNWfcuSJ19+9FVD3Yx4kVCvwOxtaxpWntRk+5acwwsueBngV9G+TBMLBvt/fI/MjWFXFikxSuyTd/jaF++uhlxDsd5gf6N68Nk8IK6XDsQSVKxt5+DPyFEU4YquQyFot5ZhcQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=h8zX5EwGpoSKxgJhw5pcbNisWYyaf2L4YDqtha8/xJ4=; b=WdvSL6fDY85Q6SR/xrqkXybms6VJP5JKPREpo4bsT5gD114uTo1yK+Cm3yKQXP3B8z8nm6CElH7GbGPUCBGE3N/4eZjxWo6Mem/JcrR2eXswXM/GTmp39fIO/oiI3tqg5Z6vpZ8f5/aBU6xUUokZ3U3ILbjmyZWDE8aHn/wEgpTDcSljV2a7Km45lvmUAIVcg8Ubia7G4qSgI+9Vh8I5ZyJ1pV9Wz3nkamPkMM0XgONwNIv7ziKMsYidXaAeiHcFmS/DHdtpxdjrV3UFPw7xMsZtgRySJYIdQcPPY+J01vtvEBPkUi/+b6uBy0CjeMr9NDYz6MUxLCygeoGRVNt8+A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=h8zX5EwGpoSKxgJhw5pcbNisWYyaf2L4YDqtha8/xJ4=; b=awHRuOcE592+2dptfovaqcbKz9VqfDPR9wA2b3NfvSIHaxjZ6F3YTwyDgNQLA5B9ByRxAuJEd8Cs/2Y+Jre55w/V7vUJSQaTlOGoesrbT4q3q/vSV/Ugfreoonun/0FZPLjTPynJUPST1VP2rRZD5IUDWrSyUBfJ3bfeuwK0SLc= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BLAPR10MB5025.namprd10.prod.outlook.com (2603:10b6:208:30d::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4436.22; Fri, 27 Aug 2021 14:59:16 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198%5]) with mapi id 15.20.4457.020; Fri, 27 Aug 2021 14:59:16 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , Christoph Hellwig , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v4 11/14] mm/hugetlb_vmemmap: move comment block to Documentation/vm Date: Fri, 27 Aug 2021 15:58:16 +0100 Message-Id: <20210827145819.16471-12-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210827145819.16471-1-joao.m.martins@oracle.com> References: <20210827145819.16471-1-joao.m.martins@oracle.com> X-ClientProxiedBy: LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (138.3.204.8) by LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4457.20 via Frontend Transport; Fri, 27 Aug 2021 14:59:12 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 53736d5a-7d54-463d-ed0e-08d9696b3cda X-MS-TrafficTypeDiagnostic: BLAPR10MB5025: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: DDtpwVtxxHsZeKdxN3DV7nZz1k29rZT0pAYX5Cqu1+Fz94FCrcuB5XJm7Y0np6Y7PxVHNgjP/mjD/zTlp11Fh48FNBhcOs6XdsBiWt54APVEz1E6lQOYgI6ODWeHeazO9UPzh+NlB+OSl0J1DbGE8kKMivI9n5EdtC8MXx6TEgpcDtYcM50C4LVwUWe78WwRzfpSV6DGaS4aQrRnJbdCKOsRyP0fYlty5LH9A5E6b87m1pJcfjmRY20xz1yIrWoPi98Zm/MVMBv8ufehjmb+NtVhL9Pz3U3AK9lHi/3r1NL1D1S3cFhWE2WWJGVvxGpAWhBGIQetgCpgoB9pVJLwLf8bNxzaRJYxE4K0v7gxbuzwo81kSOa1juNbO06yffOR0+xGVCu+EEC5ioG54yX4MptSEteZAlS9BvvXxcO53PukSH30X18EGaL32ubkNQpceaPJza8f68REhBUQv6ApivwOpfWFjb9iBY6z6QSq3KL74Ufa6x7bda8b63vMORdaJHZ5OwRIT5/uK48KmCh95cDj0TbdHu/U/mkXONYTuP8d8srUd9gyujp8OX/9i9qR2pT88gyT2QPVAFuDHcC/wVx/q+YEGMB7x9W7Log4kNyjmBuWq+QuG8QqR+wSDszbKkZybY4uHzuKYdJjzU+yu+GDzFle84XXS5MDui8ouwgHFFLRXxz7mDs7FzaA6qlXqPdu//se84ChI4TrMVjJOA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(316002)(508600001)(103116003)(66946007)(83380400001)(66476007)(86362001)(54906003)(6916009)(30864003)(6486002)(2906002)(1076003)(107886003)(8676002)(186003)(4326008)(38350700002)(5660300002)(956004)(26005)(66556008)(52116002)(7696005)(8936002)(36756003)(2616005)(7416002)(38100700002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 8GwWkVw6L23HAFPdVr1iqfUqFqE2Rkj5q9sCXjV776A8tNJbUk8deqEm0Mp+TFOJeWKikF2qgWfUd4LdcOExAaPkVeempO1tx2e4eA8ViJyaj2/CXc5tA0+WLlYVyZO31ttAWMgu+9B+mQk2JdckAdlfC9RXttRyRUvH7rR0MwreQ7y7jpJxjYlgISyuqIzahFVGxLj7KOEZuYAS4r/3y9IowHqWiwN91V+oUEg7Se/REbld3eThCkshZeIQZZBClyWm1oIe3NZgmj1pf+458dc/g6t+KKpZ75uRzfE1uPIL7UntXGNHC/7VwEK/CGzgi0fJQGhc0a8sQAe6ZK1910NC0qtuqGS/xMtBa8Rej/yTiKt4RmLOepMD11FCJnwvuhTZwsprIgYJrd50hkr6qxZffZPKoFLwxE7zaGGX2hW+lHZIdVh2TQJGje/guKC0y8Uy4tDmcLYEbt5o4SABjNfQg6qE421B8bW/4xQNMslgVVHDukTLx4Z2p0h+N9jtsxwSX/pS8IedQXIcqnFGsQz6Sgqhp9yTHLFmOV2xUAFpyxPK76NpJH/scjVTGIchZHxP1OfUIp55Tz3DmP27wzT/7ODzcDW7X2+gFk0OJ4rWUHbDB38yqwa1HlTbwXHTAZKabdor7IURdW2aP6G5kP8s4dr8YOFVe4hkTZZv5viNIj/GrsNreL45kt0BGJ3xgE0+MGUjaimXU5YoHNCMWJoe6jUhPXkaxql131BM3RPmExaISzLKq7j2OhxHMfclKBQwbEIamBJSNNL1geyCft6KPGTB8XY1eXOQ3w+HqkpgZvTT41nnxgR1q9O1dro4IzGiHrX/SBrtGSRg/UoFrTbySBVu3qQEmRlGivmySr6uJwBg3akNwC3oWmVMDFW7R5q/S+R1htezzqq/mXZgtoXNnDUWhJIiS1Qs2CjN73VsCThH57WjO2bo+QeHCz9vQZf97BIzv8Y1QVLmOPMgRF0yX/qSA7NwvOOj5jUbNZUZzyzGPqIMfhMsjifhnduAdECEqPRrVTdfOrd7kK2LuPobxGHHvCZA/OgEZ/QwFQKM6U79pmHcno/NONECjikqXBELwDaPogoFK8AIPXMFWVUASJt8OK3+G0jv02gAfF4btaWcCeMwjRBuixJxHxSsuoaJRRxeuLJQjVFSzjDTmHlTptYdnQ0I3IzTetltjFO4uGJC1m2B6mQY9qR0yOHjb4iD40ZLtLO8gji5aZ9CNNlccD9c6MOHKfa6jAe8lLFi8fo7LDSEeREHbIV54+2OfjM4lsItl2M152Y4yKo5x6L5hObAVlH263XPbgmVpUzc+dV1bMLHtLOwwS9aOjFI X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 53736d5a-7d54-463d-ed0e-08d9696b3cda X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Aug 2021 14:59:16.3626 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: mBmh0F0fiIRd9jDHTm0p+i1d73bJOPNrHH6m6IfXdZCHApd9Y0hEQuLSJLCluOcuHF8222NRdFkO+YqYgtdw0O+7SJ0r0VNLq3HlhQHn5eg= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB5025 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10089 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 phishscore=0 spamscore=0 bulkscore=0 mlxlogscore=999 malwarescore=0 adultscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108270092 X-Proofpoint-ORIG-GUID: 7Q5E0LC2cZZ9C88yIY7WfrzNPH8LOw6R X-Proofpoint-GUID: 7Q5E0LC2cZZ9C88yIY7WfrzNPH8LOw6R In preparation for device-dax for using hugetlbfs compound page tail deduplication technique, move the comment block explanation into a common place in Documentation/vm. Cc: Muchun Song Cc: Mike Kravetz Suggested-by: Dan Williams Signed-off-by: Joao Martins Reviewed-by: Muchun Song Reviewed-by: Dan Williams --- Documentation/vm/index.rst | 1 + Documentation/vm/vmemmap_dedup.rst | 170 +++++++++++++++++++++++++++++ mm/hugetlb_vmemmap.c | 162 +-------------------------- 3 files changed, 172 insertions(+), 161 deletions(-) create mode 100644 Documentation/vm/vmemmap_dedup.rst diff --git a/Documentation/vm/index.rst b/Documentation/vm/index.rst index b51f0d8992f8..68fe9b953b0a 100644 --- a/Documentation/vm/index.rst +++ b/Documentation/vm/index.rst @@ -52,5 +52,6 @@ descriptions of data structures and algorithms. split_page_table_lock transhuge unevictable-lru + vmemmap_dedup z3fold zsmalloc diff --git a/Documentation/vm/vmemmap_dedup.rst b/Documentation/vm/vmemmap_dedup.rst new file mode 100644 index 000000000000..215ae2ef3bce --- /dev/null +++ b/Documentation/vm/vmemmap_dedup.rst @@ -0,0 +1,170 @@ +.. SPDX-License-Identifier: GPL-2.0 + +.. _vmemmap_dedup: + +================================== +Free some vmemmap pages of HugeTLB +================================== + +The struct page structures (page structs) are used to describe a physical +page frame. By default, there is a one-to-one mapping from a page frame to +it's corresponding page struct. + +HugeTLB pages consist of multiple base page size pages and is supported by +many architectures. See hugetlbpage.rst in the Documentation directory for +more details. On the x86-64 architecture, HugeTLB pages of size 2MB and 1GB +are currently supported. Since the base page size on x86 is 4KB, a 2MB +HugeTLB page consists of 512 base pages and a 1GB HugeTLB page consists of +4096 base pages. For each base page, there is a corresponding page struct. + +Within the HugeTLB subsystem, only the first 4 page structs are used to +contain unique information about a HugeTLB page. __NR_USED_SUBPAGE provides +this upper limit. The only 'useful' information in the remaining page structs +is the compound_head field, and this field is the same for all tail pages. + +By removing redundant page structs for HugeTLB pages, memory can be returned +to the buddy allocator for other uses. + +Different architectures support different HugeTLB pages. For example, the +following table is the HugeTLB page size supported by x86 and arm64 +architectures. Because arm64 supports 4k, 16k, and 64k base pages and +supports contiguous entries, so it supports many kinds of sizes of HugeTLB +page. + ++--------------+-----------+-----------------------------------------------+ +| Architecture | Page Size | HugeTLB Page Size | ++--------------+-----------+-----------+-----------+-----------+-----------+ +| x86-64 | 4KB | 2MB | 1GB | | | ++--------------+-----------+-----------+-----------+-----------+-----------+ +| | 4KB | 64KB | 2MB | 32MB | 1GB | +| +-----------+-----------+-----------+-----------+-----------+ +| arm64 | 16KB | 2MB | 32MB | 1GB | | +| +-----------+-----------+-----------+-----------+-----------+ +| | 64KB | 2MB | 512MB | 16GB | | ++--------------+-----------+-----------+-----------+-----------+-----------+ + +When the system boot up, every HugeTLB page has more than one struct page +structs which size is (unit: pages): + + struct_size = HugeTLB_Size / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE + +Where HugeTLB_Size is the size of the HugeTLB page. We know that the size +of the HugeTLB page is always n times PAGE_SIZE. So we can get the following +relationship. + + HugeTLB_Size = n * PAGE_SIZE + +Then, + + struct_size = n * PAGE_SIZE / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE + = n * sizeof(struct page) / PAGE_SIZE + +We can use huge mapping at the pud/pmd level for the HugeTLB page. + +For the HugeTLB page of the pmd level mapping, then + + struct_size = n * sizeof(struct page) / PAGE_SIZE + = PAGE_SIZE / sizeof(pte_t) * sizeof(struct page) / PAGE_SIZE + = sizeof(struct page) / sizeof(pte_t) + = 64 / 8 + = 8 (pages) + +Where n is how many pte entries which one page can contains. So the value of +n is (PAGE_SIZE / sizeof(pte_t)). + +This optimization only supports 64-bit system, so the value of sizeof(pte_t) +is 8. And this optimization also applicable only when the size of struct page +is a power of two. In most cases, the size of struct page is 64 bytes (e.g. +x86-64 and arm64). So if we use pmd level mapping for a HugeTLB page, the +size of struct page structs of it is 8 page frames which size depends on the +size of the base page. + +For the HugeTLB page of the pud level mapping, then + + struct_size = PAGE_SIZE / sizeof(pmd_t) * struct_size(pmd) + = PAGE_SIZE / 8 * 8 (pages) + = PAGE_SIZE (pages) + +Where the struct_size(pmd) is the size of the struct page structs of a +HugeTLB page of the pmd level mapping. + +E.g.: A 2MB HugeTLB page on x86_64 consists in 8 page frames while 1GB +HugeTLB page consists in 4096. + +Next, we take the pmd level mapping of the HugeTLB page as an example to +show the internal implementation of this optimization. There are 8 pages +struct page structs associated with a HugeTLB page which is pmd mapped. + +Here is how things look before optimization. + + HugeTLB struct pages(8 pages) page frame(8 pages) + +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ + | | | 0 | -------------> | 0 | + | | +-----------+ +-----------+ + | | | 1 | -------------> | 1 | + | | +-----------+ +-----------+ + | | | 2 | -------------> | 2 | + | | +-----------+ +-----------+ + | | | 3 | -------------> | 3 | + | | +-----------+ +-----------+ + | | | 4 | -------------> | 4 | + | PMD | +-----------+ +-----------+ + | level | | 5 | -------------> | 5 | + | mapping | +-----------+ +-----------+ + | | | 6 | -------------> | 6 | + | | +-----------+ +-----------+ + | | | 7 | -------------> | 7 | + | | +-----------+ +-----------+ + | | + | | + | | + +-----------+ + +The value of page->compound_head is the same for all tail pages. The first +page of page structs (page 0) associated with the HugeTLB page contains the 4 +page structs necessary to describe the HugeTLB. The only use of the remaining +pages of page structs (page 1 to page 7) is to point to page->compound_head. +Therefore, we can remap pages 2 to 7 to page 1. Only 2 pages of page structs +will be used for each HugeTLB page. This will allow us to free the remaining +6 pages to the buddy allocator. + +Here is how things look after remapping. + + HugeTLB struct pages(8 pages) page frame(8 pages) + +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ + | | | 0 | -------------> | 0 | + | | +-----------+ +-----------+ + | | | 1 | -------------> | 1 | + | | +-----------+ +-----------+ + | | | 2 | ----------------^ ^ ^ ^ ^ ^ + | | +-----------+ | | | | | + | | | 3 | ------------------+ | | | | + | | +-----------+ | | | | + | | | 4 | --------------------+ | | | + | PMD | +-----------+ | | | + | level | | 5 | ----------------------+ | | + | mapping | +-----------+ | | + | | | 6 | ------------------------+ | + | | +-----------+ | + | | | 7 | --------------------------+ + | | +-----------+ + | | + | | + | | + +-----------+ + +When a HugeTLB is freed to the buddy system, we should allocate 6 pages for +vmemmap pages and restore the previous mapping relationship. + +For the HugeTLB page of the pud level mapping. It is similar to the former. +We also can use this approach to free (PAGE_SIZE - 2) vmemmap pages. + +Apart from the HugeTLB page of the pmd/pud level mapping, some architectures +(e.g. aarch64) provides a contiguous bit in the translation table entries +that hints to the MMU to indicate that it is one of a contiguous set of +entries that can be cached in a single TLB entry. + +The contiguous bit is used to increase the mapping size at the pmd and pte +(last) level. So this type of HugeTLB page can be optimized only when its +size of the struct page structs is greater than 2 pages. + diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index c540c21e26f5..e2994e50ddee 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -6,167 +6,7 @@ * * Author: Muchun Song * - * The struct page structures (page structs) are used to describe a physical - * page frame. By default, there is a one-to-one mapping from a page frame to - * it's corresponding page struct. - * - * HugeTLB pages consist of multiple base page size pages and is supported by - * many architectures. See hugetlbpage.rst in the Documentation directory for - * more details. On the x86-64 architecture, HugeTLB pages of size 2MB and 1GB - * are currently supported. Since the base page size on x86 is 4KB, a 2MB - * HugeTLB page consists of 512 base pages and a 1GB HugeTLB page consists of - * 4096 base pages. For each base page, there is a corresponding page struct. - * - * Within the HugeTLB subsystem, only the first 4 page structs are used to - * contain unique information about a HugeTLB page. __NR_USED_SUBPAGE provides - * this upper limit. The only 'useful' information in the remaining page structs - * is the compound_head field, and this field is the same for all tail pages. - * - * By removing redundant page structs for HugeTLB pages, memory can be returned - * to the buddy allocator for other uses. - * - * Different architectures support different HugeTLB pages. For example, the - * following table is the HugeTLB page size supported by x86 and arm64 - * architectures. Because arm64 supports 4k, 16k, and 64k base pages and - * supports contiguous entries, so it supports many kinds of sizes of HugeTLB - * page. - * - * +--------------+-----------+-----------------------------------------------+ - * | Architecture | Page Size | HugeTLB Page Size | - * +--------------+-----------+-----------+-----------+-----------+-----------+ - * | x86-64 | 4KB | 2MB | 1GB | | | - * +--------------+-----------+-----------+-----------+-----------+-----------+ - * | | 4KB | 64KB | 2MB | 32MB | 1GB | - * | +-----------+-----------+-----------+-----------+-----------+ - * | arm64 | 16KB | 2MB | 32MB | 1GB | | - * | +-----------+-----------+-----------+-----------+-----------+ - * | | 64KB | 2MB | 512MB | 16GB | | - * +--------------+-----------+-----------+-----------+-----------+-----------+ - * - * When the system boot up, every HugeTLB page has more than one struct page - * structs which size is (unit: pages): - * - * struct_size = HugeTLB_Size / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE - * - * Where HugeTLB_Size is the size of the HugeTLB page. We know that the size - * of the HugeTLB page is always n times PAGE_SIZE. So we can get the following - * relationship. - * - * HugeTLB_Size = n * PAGE_SIZE - * - * Then, - * - * struct_size = n * PAGE_SIZE / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE - * = n * sizeof(struct page) / PAGE_SIZE - * - * We can use huge mapping at the pud/pmd level for the HugeTLB page. - * - * For the HugeTLB page of the pmd level mapping, then - * - * struct_size = n * sizeof(struct page) / PAGE_SIZE - * = PAGE_SIZE / sizeof(pte_t) * sizeof(struct page) / PAGE_SIZE - * = sizeof(struct page) / sizeof(pte_t) - * = 64 / 8 - * = 8 (pages) - * - * Where n is how many pte entries which one page can contains. So the value of - * n is (PAGE_SIZE / sizeof(pte_t)). - * - * This optimization only supports 64-bit system, so the value of sizeof(pte_t) - * is 8. And this optimization also applicable only when the size of struct page - * is a power of two. In most cases, the size of struct page is 64 bytes (e.g. - * x86-64 and arm64). So if we use pmd level mapping for a HugeTLB page, the - * size of struct page structs of it is 8 page frames which size depends on the - * size of the base page. - * - * For the HugeTLB page of the pud level mapping, then - * - * struct_size = PAGE_SIZE / sizeof(pmd_t) * struct_size(pmd) - * = PAGE_SIZE / 8 * 8 (pages) - * = PAGE_SIZE (pages) - * - * Where the struct_size(pmd) is the size of the struct page structs of a - * HugeTLB page of the pmd level mapping. - * - * E.g.: A 2MB HugeTLB page on x86_64 consists in 8 page frames while 1GB - * HugeTLB page consists in 4096. - * - * Next, we take the pmd level mapping of the HugeTLB page as an example to - * show the internal implementation of this optimization. There are 8 pages - * struct page structs associated with a HugeTLB page which is pmd mapped. - * - * Here is how things look before optimization. - * - * HugeTLB struct pages(8 pages) page frame(8 pages) - * +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ - * | | | 0 | -------------> | 0 | - * | | +-----------+ +-----------+ - * | | | 1 | -------------> | 1 | - * | | +-----------+ +-----------+ - * | | | 2 | -------------> | 2 | - * | | +-----------+ +-----------+ - * | | | 3 | -------------> | 3 | - * | | +-----------+ +-----------+ - * | | | 4 | -------------> | 4 | - * | PMD | +-----------+ +-----------+ - * | level | | 5 | -------------> | 5 | - * | mapping | +-----------+ +-----------+ - * | | | 6 | -------------> | 6 | - * | | +-----------+ +-----------+ - * | | | 7 | -------------> | 7 | - * | | +-----------+ +-----------+ - * | | - * | | - * | | - * +-----------+ - * - * The value of page->compound_head is the same for all tail pages. The first - * page of page structs (page 0) associated with the HugeTLB page contains the 4 - * page structs necessary to describe the HugeTLB. The only use of the remaining - * pages of page structs (page 1 to page 7) is to point to page->compound_head. - * Therefore, we can remap pages 2 to 7 to page 1. Only 2 pages of page structs - * will be used for each HugeTLB page. This will allow us to free the remaining - * 6 pages to the buddy allocator. - * - * Here is how things look after remapping. - * - * HugeTLB struct pages(8 pages) page frame(8 pages) - * +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ - * | | | 0 | -------------> | 0 | - * | | +-----------+ +-----------+ - * | | | 1 | -------------> | 1 | - * | | +-----------+ +-----------+ - * | | | 2 | ----------------^ ^ ^ ^ ^ ^ - * | | +-----------+ | | | | | - * | | | 3 | ------------------+ | | | | - * | | +-----------+ | | | | - * | | | 4 | --------------------+ | | | - * | PMD | +-----------+ | | | - * | level | | 5 | ----------------------+ | | - * | mapping | +-----------+ | | - * | | | 6 | ------------------------+ | - * | | +-----------+ | - * | | | 7 | --------------------------+ - * | | +-----------+ - * | | - * | | - * | | - * +-----------+ - * - * When a HugeTLB is freed to the buddy system, we should allocate 6 pages for - * vmemmap pages and restore the previous mapping relationship. - * - * For the HugeTLB page of the pud level mapping. It is similar to the former. - * We also can use this approach to free (PAGE_SIZE - 2) vmemmap pages. - * - * Apart from the HugeTLB page of the pmd/pud level mapping, some architectures - * (e.g. aarch64) provides a contiguous bit in the translation table entries - * that hints to the MMU to indicate that it is one of a contiguous set of - * entries that can be cached in a single TLB entry. - * - * The contiguous bit is used to increase the mapping size at the pmd and pte - * (last) level. So this type of HugeTLB page can be optimized only when its - * size of the struct page structs is greater than 2 pages. + * See Documentation/vm/vmemmap_dedup.rst */ #define pr_fmt(fmt) "HugeTLB: " fmt From patchwork Fri Aug 27 14:58:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12462233 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 621973FD0 for ; Fri, 27 Aug 2021 14:59:31 +0000 (UTC) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 17RDWHDH002206; Fri, 27 Aug 2021 14:59:22 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=vAhtJAv1Ep8TEn/uiSYFII5C86zN825jI8yYzbOwAOM=; b=NnPNeTvAD1WjGZBkTFK54tKLTUyjuBB1DbSQiZYk5CgEhOtjJgYo5hE/f6TmWPQ0JL5D /frXQaHqVBO53QLkYKRg1UrbWJMM0MWBoZOJZqE6kgjqTsMUNpKhowuMstm4cQITSxy1 ZN6WBsU6yMmbj0x1EawpUqdlgVqO3cGQzyYuTp2dGS6wbF3W5VkinfMyY8/RpHItY51s cS1XIvFmJ1sFOpE7GggK3ZzswB7FO58xZXTOhOvpfTt5A4pXX44uoxzpqu5nBHyuLpWq B4ouIub3eWq4F7noCIVTogCFt3SMcDOTtkZaRfE2wIHShA/ZIlSXK02Kkc+B8CdHNsEN Mg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=vAhtJAv1Ep8TEn/uiSYFII5C86zN825jI8yYzbOwAOM=; b=k5xzmkgptY6eoVhkw+FeVGE9Xmzvx0XmaB92JPG/n1urUt4VE/XKlHOY5/8zAs5BS3F5 /CWij4ZbQwesS0By6viQBDdgC5tko41elbxkNBiFvW71JeAtv1qpyyUd4f/+tJPWwnTD A3vbX08rcjQ3KO8RzemhQRE3xCCmYqhYuwmmKExmHnQG3kpRSGrGGfVjg4wOqytIsmFm RmNdTZMCqD7XWoNHqAdknDyfLpBFdekSVqn72MEPOv0H7LMBiCyF5FWubm3LoX+OYiLu 9O/GWYE2Gse9IwWOwOGOhjj/2gbnKuafMtMTjbizbARO/hjQEuWLRn/2KLqQ4A5wLQ+e Cw== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by mx0b-00069f02.pphosted.com with ESMTP id 3apvjr0vj6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:59:22 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17REpP1T187148; Fri, 27 Aug 2021 14:59:21 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2176.outbound.protection.outlook.com [104.47.57.176]) by aserp3020.oracle.com with ESMTP id 3ajsabbqt0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:59:21 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=I7AhMQ50vTM9l4ldxLlWlQiTRyorHsxUDyYDVZPpsIGCByaQm8OSx78AybFTtdRx9vR0jw4zgqU2hulH4+DDG0YmB68+oZqzB1GpQ2Rsyhp6rJiOUfOKtd60ut3Az+cwUJhaR9UZV6NXQpDPFir67XwgcBl0WbsZ4ihzzW3XZQLLgJpkAo6yzhA2JVNtOLztA2fyowbeLjkF6tslOE3d9NPWR6MPt6k89EmNk6ZjdIcmQwCK1Rkw78JHYiBPdSs4X6pc9CmkWW+ZIS9l9RlJVoIAmn50kAa9yDWZYWOF2NQzuetj4OxsQoffw//E3Re2wiYgqMGyZvYN2+/IlOMBhQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vAhtJAv1Ep8TEn/uiSYFII5C86zN825jI8yYzbOwAOM=; b=BKfUOVUanZhdTMH5OJ7KOhsPSy01qGVj6k+D/YCFT35YbMZUY7gkIkOdm1AvJaKusfiTbWI9soCP83ZhHQZyicbOnXRBD4XJdJUTtoHJUP/k2tX0JRx/EyIlm3khNGQeWTNOk5hJZK1aVe6Y80iklyd/IQ88fcUFZz+SpW5FHwRjUXb6BATtO42bDmHo8vKFP1/WoZuh/XwnRFI8PxTttaNtj0BZ/iQjbTV2eW1CXDMWCf0qaI8OdfbpPLxt6huKfZDwoWwV2RONCj4YKMhed+zwsWTb8gRC5AozgF4UwFrgsFsqltD+XVDAhljPoGK3gYTh13sFYoM1Brlr53T/Vg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vAhtJAv1Ep8TEn/uiSYFII5C86zN825jI8yYzbOwAOM=; b=d8KpFDJPxyY3z5cT7C0/4r8DvRlgW2qyUd8RV9RQ6QrTc6Q5cwk01e7dspdr6lyxrxNPVfT6PjR/+4H/pTwM1flg8lo0VuFwrGdlDLC4Lt++poqGBHEISrZpyy2W4aEtCwsNg8qtEO01+VLq9M3TQeu1HA8aUlDoqHiOUWOm92g= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BLAPR10MB4946.namprd10.prod.outlook.com (2603:10b6:208:323::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4457.17; Fri, 27 Aug 2021 14:59:19 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198%5]) with mapi id 15.20.4457.020; Fri, 27 Aug 2021 14:59:19 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , Christoph Hellwig , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v4 12/14] mm/sparse-vmemmap: populate compound devmaps Date: Fri, 27 Aug 2021 15:58:17 +0100 Message-Id: <20210827145819.16471-13-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210827145819.16471-1-joao.m.martins@oracle.com> References: <20210827145819.16471-1-joao.m.martins@oracle.com> X-ClientProxiedBy: LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (138.3.204.8) by LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4457.20 via Frontend Transport; Fri, 27 Aug 2021 14:59:16 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3b3aad28-e758-41f6-9e03-08d9696b3f1b X-MS-TrafficTypeDiagnostic: BLAPR10MB4946: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: bo8OaTPUht27kA74sk6Lp9dhdr+Ja3YxNS9mRbcv6j6gUq1Ql5RwRXN9hTqU8g68tlOiiplRw7OkzXFliF7QPiKLSiI4MrR/i8D6darLSXcWKKuguNPZGWBzLu2k+Xkc6KCxw8MpTe3rGGcVrNid3xASuPJbQaYE/X/18UuscTD0aDLOYA8fddUcTzkxQ6hMQVUBXOE1OZWRR9loOUjbSQW5VE7h0slEXP9fyFUPHFgaTGXzLLxAG4/Swn7uD8FckSgG1kjGCS7IAJAoj0KDbWO9Pc6TB5Q5FICyrypfbqALkSLtqUNrk7vipw3YThaan0xgba69iuKtMS04O9NiQgpK6oe16x0nVJYmiXSRaSlEuXlUGQp1RHEp6yCTRKTqSyly0zxt+fRVj0huwL/GO4SXkGIJksRQYv3TBdMEap9BBjAQABrjqKh42anmG1lWmVTr45SRmpArgNKBPpNnZ9m67dEWCBV4XBpyUfYvzpNpHl2MwNmj+SWmhrdgfRhSkK02d0PEHqSPcwDBKMR9UxaS3QCyXXm2tKkn7oZBkWTcHhhtlbE1JvEx/TAYpr9bYrYkJ2P9U7yl/OdbAh72gdlV5yq37ojbuIRDHjxA9mruzpD8CpbqHU/9kz68DWesxaupTtrwuWywbbKU1Q/9y04yiLQy+8Vg4sKvXo71uesyKEfJ0KKls29x0kYh+OupEiFAMHWjNWFFXSeEhEgBqDmyIAJceJX8FYMWQBpDDII= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(136003)(376002)(346002)(396003)(366004)(316002)(86362001)(1076003)(2906002)(7416002)(107886003)(8936002)(66476007)(66556008)(186003)(66946007)(478600001)(38350700002)(36756003)(4326008)(7696005)(2616005)(8676002)(5660300002)(52116002)(956004)(103116003)(6486002)(38100700002)(6916009)(26005)(54906003)(83380400001)(334744004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: rdAwp7V+rvDVyc6WAC4OrcQ3PH/RykiEXZ22Sr2g6LEDN1w3cIRF4JlJZimL2t3TlZ4DKGDEBTIh6R5pWz/BCilRbS3c1nCr4z1Q742V0fjkJUe6cDV3sd2X6iz0ld4QiX4AM4KYUVZykfCbQCff4sz6EdE0lP/xOWJsFuhTfPDiR2f52Q16SiPWiIGmKDvZqnrs6xvmcPd8JVkDS3ywOIhvbP0tYV9cv1EPAXFd4tlP97zOEMRlEIeR8Z+AAqIODfFizIBM822FoHKeLzyVxr3uIAyX2U2TQK8qhl/JNJrilpMg60cV+M7sE7DQF4YgbsVCYWVnzcR5UuFAS23QpQaflH7tQiD/acOSPjJKDcdbHoL0PhU6ZU6z5h1ltaaSa6ipHbg18b0JPloJPDvFr2X4rhWsO0AsQI00kCPRDVjstlC4+pSN9HbnGKZ6vtYhPzRcZhVE8B751CKPh5mnDYmzXc5wfq9iabRssI62QtcbvEYDFETYqMBmeksVu0jQs56z0tC5bBAIfrAe5Q7gMsuDtHW095n7zCmAvFKJsx04JVsLByjYcdhaCeBcPpZyqBSF/JLYOumLgPPhgIGYty0OEphGPe86vx6PT6EN9qyGG0cIHVEneO0bSTBi8r2xrLDZ6DhYgpoV+FM1U2F03NDQv4yQnvPT13Z/qTDLIRrtd6mCPrjWXvwge13Cpw8+r4fC5KRn1Qj2g4LyHZlESWSpeE9WnnpuV8Qs08pTPpiZcwhVo8/QWZDluGNHmSZt9+z8ndq4/E/P5kUlX3llZ+YvSoBTqYhzieRZyqxctBWiPfBBiw7Bg54AjQ3RXcTiq7cyPF/LDGwQt/qhlqfCE8kUuLsn8y1GtTPD2Hb3XWMdA7ZnzvMI3wnGvv852xRur+Wj81QS2BZvHK42an8pCB8Qr9lRl9zBQkUKM3/M+NuhLewuFYs6Vl/tdDrRAp+VXfpG9J139KqO+2H0Qz/HLcOhJ345hu1yjfczdK3Yv9l5MzAaxXHOcC4HU5+apdENa23LmMxJbABkRVMriPlimqCrG+l2zEjKetpkAntiRnbS7JKGB7FtAy9PmkqJ8cOW2Q+QZq2fKZQGkwVIeu2L+6ZD3c0F6k5BP19Jow5dVtxjmxXOZhdk6PESReDv9jGcl2SKIlsvwIM0in6O59bs6wgLYllcZvIxsvbf3OZN+jQZtmup69KilVg1t6Ar2Z3N9QXDnur3qZVwV1X1OzY43tMGzeYb9M93GAqR2CqOsZbWsww4T8wjWCYq5L8W6oIahPvy4LrEe6Q+tZ+skywpJLR7j+KqmXBXeCf0zzVtc/Iwqgpfw7+h8fo4rLGty8+N X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3b3aad28-e758-41f6-9e03-08d9696b3f1b X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Aug 2021 14:59:19.3326 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 9/vEcUDS3YgVKdfxSdMmrNbdZzxj6++wlxgBCR0myY+TF0Z3UUJnIbUiS7DAXeXDlfAwVhzNFUVJgiNFh2dVsHk+ddVh1eUju5jymiN55mA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB4946 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10089 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 mlxlogscore=999 bulkscore=0 mlxscore=0 adultscore=0 malwarescore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108270092 X-Proofpoint-GUID: uiTuuas6BCndbBl9g2c3Uig-rb87hsgZ X-Proofpoint-ORIG-GUID: uiTuuas6BCndbBl9g2c3Uig-rb87hsgZ A compound devmap is a dev_pagemap with @geometry > PAGE_SIZE and it means that pages are mapped at a given huge page alignment and utilize uses compound pages as opposed to order-0 pages. Take advantage of the fact that most tail pages look the same (except the first two) to minimize struct page overhead. Allocate a separate page for the vmemmap area which contains the head page and separate for the next 64 pages. The rest of the subsections then reuse this tail vmemmap page to initialize the rest of the tail pages. Sections are arch-dependent (e.g. on x86 it's 64M, 128M or 512M) and when initializing compound devmap with big enough @geometry (e.g. 1G PUD) it may cross multiple sections. The vmemmap code needs to consult @pgmap so that multiple sections that all map the same tail data can refer back to the first copy of that data for a given gigantic page. On compound devmaps with 2M align, this mechanism lets 6 pages be saved out of the 8 necessary PFNs necessary to set the subsection's 512 struct pages being mapped. On a 1G compound devmap it saves 4094 pages. Altmap isn't supported yet, given various restrictions in altmap pfn allocator, thus fallback to the already in use vmemmap_populate(). It is worth noting that altmap for devmap mappings was there to relieve the pressure of inordinate amounts of memmap space to map terabytes of pmem. With compound pages the motivation for altmaps for pmem gets reduced. Signed-off-by: Joao Martins --- Documentation/vm/vmemmap_dedup.rst | 27 +++++- include/linux/mm.h | 2 +- mm/memremap.c | 1 + mm/sparse-vmemmap.c | 150 +++++++++++++++++++++++++++-- 4 files changed, 168 insertions(+), 12 deletions(-) diff --git a/Documentation/vm/vmemmap_dedup.rst b/Documentation/vm/vmemmap_dedup.rst index 215ae2ef3bce..faac78bef01c 100644 --- a/Documentation/vm/vmemmap_dedup.rst +++ b/Documentation/vm/vmemmap_dedup.rst @@ -2,9 +2,12 @@ .. _vmemmap_dedup: -================================== -Free some vmemmap pages of HugeTLB -================================== +========================================= +A vmemmap diet for HugeTLB and Device DAX +========================================= + +HugeTLB +======= The struct page structures (page structs) are used to describe a physical page frame. By default, there is a one-to-one mapping from a page frame to @@ -168,3 +171,21 @@ The contiguous bit is used to increase the mapping size at the pmd and pte (last) level. So this type of HugeTLB page can be optimized only when its size of the struct page structs is greater than 2 pages. +Device DAX +========== + +The device-dax interface uses the same tail deduplication technique explained +in the previous chapter, except when used with the vmemmap in the device (altmap). + +The differences with HugeTLB are relatively minor. + +The following page sizes are supported in DAX: PAGE_SIZE (4K on x86_64), +PMD_SIZE (2M on x86_64) and PUD_SIZE (1G on x86_64). + +There's no remapping of vmemmap given that device-dax memory is not part of +System RAM ranges initialized at boot, hence the tail deduplication happens +at a later stage when we populate the sections. + +It only use 3 page structs for storing all information as opposed +to 4 on HugeTLB pages. This does not affect memory savings between both. + diff --git a/include/linux/mm.h b/include/linux/mm.h index 4fca4942c0ab..77eaeae497f9 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3174,7 +3174,7 @@ p4d_t *vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node); pud_t *vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node); pmd_t *vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node); pte_t *vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, - struct vmem_altmap *altmap); + struct vmem_altmap *altmap, struct page *block); void *vmemmap_alloc_block(unsigned long size, int node); struct vmem_altmap; void *vmemmap_alloc_block_buf(unsigned long size, int node, diff --git a/mm/memremap.c b/mm/memremap.c index 99646082436f..0d4c98722c12 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -338,6 +338,7 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid) { struct mhp_params params = { .altmap = pgmap_altmap(pgmap), + .pgmap = pgmap, .pgprot = PAGE_KERNEL, }; const int nr_range = pgmap->nr_range; diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 58e8e77bd5b5..441bb95edd68 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -495,16 +495,31 @@ void __meminit vmemmap_verify(pte_t *pte, int node, } pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, - struct vmem_altmap *altmap) + struct vmem_altmap *altmap, + struct page *block) { pte_t *pte = pte_offset_kernel(pmd, addr); if (pte_none(*pte)) { pte_t entry; void *p; - p = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); - if (!p) - return NULL; + if (!block) { + p = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); + if (!p) + return NULL; + } else { + /* + * When a PTE/PMD entry is freed from the init_mm + * there's a a free_pages() call to this page allocated + * above. Thus this get_page() is paired with the + * put_page_testzero() on the freeing path. + * This can only called by certain ZONE_DEVICE path, + * and through vmemmap_populate_compound_pages() when + * slab is available. + */ + get_page(block); + p = page_to_virt(block); + } entry = pfn_pte(__pa(p) >> PAGE_SHIFT, PAGE_KERNEL); set_pte_at(&init_mm, addr, pte, entry); } @@ -571,7 +586,8 @@ pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) } static int __meminit vmemmap_populate_address(unsigned long addr, int node, - struct vmem_altmap *altmap) + struct vmem_altmap *altmap, + struct page *reuse, struct page **page) { pgd_t *pgd; p4d_t *p4d; @@ -591,11 +607,13 @@ static int __meminit vmemmap_populate_address(unsigned long addr, int node, pmd = vmemmap_pmd_populate(pud, addr, node); if (!pmd) return -ENOMEM; - pte = vmemmap_pte_populate(pmd, addr, node, altmap); + pte = vmemmap_pte_populate(pmd, addr, node, altmap, reuse); if (!pte) return -ENOMEM; vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); + if (page) + *page = pte_page(*pte); return 0; } @@ -606,10 +624,120 @@ int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, int rc; for (; addr < end; addr += PAGE_SIZE) { - rc = vmemmap_populate_address(addr, node, altmap); + rc = vmemmap_populate_address(addr, node, altmap, NULL, NULL); if (rc) return rc; + } + + return 0; +} + +static int __meminit vmemmap_populate_range(unsigned long start, + unsigned long end, + int node, struct page *page) +{ + unsigned long addr = start; + int rc; + for (; addr < end; addr += PAGE_SIZE) { + rc = vmemmap_populate_address(addr, node, NULL, page, NULL); + if (rc) + return rc; + } + + return 0; +} + +static inline int __meminit vmemmap_populate_page(unsigned long addr, int node, + struct page **page) +{ + return vmemmap_populate_address(addr, node, NULL, NULL, page); +} + +/* + * For compound pages bigger than section size (e.g. x86 1G compound + * pages with 2M subsection size) fill the rest of sections as tail + * pages. + * + * Note that memremap_pages() resets @nr_range value and will increment + * it after each range successful onlining. Thus the value or @nr_range + * at section memmap populate corresponds to the in-progress range + * being onlined here. + */ +static bool __meminit reuse_compound_section(unsigned long start_pfn, + struct dev_pagemap *pgmap) +{ + unsigned long geometry = pgmap_geometry(pgmap); + unsigned long offset = start_pfn - + PHYS_PFN(pgmap->ranges[pgmap->nr_range].start); + + return !IS_ALIGNED(offset, geometry) && geometry > PAGES_PER_SUBSECTION; +} + +static struct page * __meminit compound_section_tail_page(unsigned long addr) +{ + pte_t *ptep; + + addr -= PAGE_SIZE; + + /* + * Assuming sections are populated sequentially, the previous section's + * page data can be reused. + */ + ptep = pte_offset_kernel(pmd_off_k(addr), addr); + if (!ptep) + return NULL; + + return pte_page(*ptep); +} + +static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, + unsigned long start, + unsigned long end, int node, + struct dev_pagemap *pgmap) +{ + unsigned long size, addr; + + if (reuse_compound_section(start_pfn, pgmap)) { + struct page *page; + + page = compound_section_tail_page(start); + if (!page) + return -ENOMEM; + + /* + * Reuse the page that was populated in the prior iteration + * with just tail struct pages. + */ + return vmemmap_populate_range(start, end, node, page); + } + + size = min(end - start, pgmap_geometry(pgmap) * sizeof(struct page)); + for (addr = start; addr < end; addr += size) { + unsigned long next = addr, last = addr + size; + struct page *block; + int rc; + + /* Populate the head page vmemmap page */ + rc = vmemmap_populate_page(addr, node, NULL); + if (rc) + return rc; + + /* Populate the tail pages vmemmap page */ + block = NULL; + next = addr + PAGE_SIZE; + rc = vmemmap_populate_page(next, node, &block); + if (rc) + return rc; + + /* + * Reuse the previous page for the rest of tail pages + * See layout diagram in Documentation/vm/vmemmap_dedup.rst + */ + next += PAGE_SIZE; + rc = vmemmap_populate_range(next, last, node, block); + if (rc) + return rc; } return 0; @@ -621,12 +749,18 @@ struct page * __meminit __populate_section_memmap(unsigned long pfn, { unsigned long start = (unsigned long) pfn_to_page(pfn); unsigned long end = start + nr_pages * sizeof(struct page); + int r; if (WARN_ON_ONCE(!IS_ALIGNED(pfn, PAGES_PER_SUBSECTION) || !IS_ALIGNED(nr_pages, PAGES_PER_SUBSECTION))) return NULL; - if (vmemmap_populate(start, end, nid, altmap)) + if (pgmap_geometry(pgmap) > 1 && !altmap) + r = vmemmap_populate_compound_pages(pfn, start, end, nid, pgmap); + else + r = vmemmap_populate(start, end, nid, altmap); + + if (r < 0) return NULL; return pfn_to_page(pfn); From patchwork Fri Aug 27 14:58:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12462235 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5DF8D3FD0 for ; Fri, 27 Aug 2021 14:59:33 +0000 (UTC) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 17REC3PY025345; Fri, 27 Aug 2021 14:59:26 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=XpSpJm3nZ4qEUIa2z4eeiBY340ZMwnjwoC8TZYWrPsQ=; b=QhQ0dzfMiJruSv7quNu6+k5AUD7BC1TzNGw7P2CFEjw+ByyU1VACDe1ZJeHCC1nJXe5G IZCp4hclSo2SsKxUgIHHkLpsglJzAtBYDi1EUBTGe8/P4VmukdYgL8tHKf1vqicEcGN4 v536iUYBFxY7wbn8vBebnlczVNwL7DGXEwI6VjGqDwKyhkAwrr9zRggo+V30cD2d9IG9 QQqRjXO9BB1wxHDgQVaXLqFV40d+gAYqOEE5gF1NLq/v3mIcc+FCLyBm8LGNafLZ3gg+ xbvxmx1SnIv67GHoQXwjMSUnMaGxxgw4n9tJR15v3CEtRKQa1vDKiRAoEs3TLjZBYPWy MA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=XpSpJm3nZ4qEUIa2z4eeiBY340ZMwnjwoC8TZYWrPsQ=; b=LPW+xMyHMiG7hkic9ncpRtVnlwgoK9f1sOlr+LlSl7cF29BTtghd49fghkAHZfGcKeWq f/TNG3v7Rzi1AxMSC1Mc4Qs+A3ywZfleFYK7ksiECcFTMQPqTx/3HHooQVvRkMYintRe SODlCmC8jYq4W+xKiRcfQlDDSTz3WMb+nzKv09I76XdN3JE1IkBQ/26++LgnFt501WWf rlWPSZ1l8KAZIexTZJtqM089G1l79ceJxZicltZEUtNUpmEwJAy0LOyJWtcDkG2c/wxV KNPDbtKRE+IOa2r9Ej/n88COTRhluKfcB0487ghnLMWk7dQrqadMt1vJ9hnG9BeybWgZ KA== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 3aq1kvg4bp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:59:26 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17REpoFK034259; Fri, 27 Aug 2021 14:59:24 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2170.outbound.protection.outlook.com [104.47.56.170]) by userp3030.oracle.com with ESMTP id 3ajpm4u93h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:59:24 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Q5jdC6rODq2xPumNtsSyNyW4YPlE2qp1DMOpi1fsxSt+Xw6X9C6bYRMtvXqBaqGDi38i7yQMw8sKhO5kT0sPClq6il3aq84weF5l6dZ6whaakAM7Qiq5cMwrvYlSFiSwVkzsGQGwtX+uAUW8HiJv8UGtat1RdX40ZaO0fx8bVAWThWe2HF6U8IB7W5GU77uTbRp+b3LCBAWZoMYDCPiXUuJLQpzxphkPV9kbL2c4lg0j2svW/h6931iuJ8z9/GssVJteiK76GpIvBZsW7zAubwrl2gjT0k1gNbdEOYnTABbVlQxGzGrtkOb+xaGk7FR67CiJUEi4/ZK9a/DDNM+Rvg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XpSpJm3nZ4qEUIa2z4eeiBY340ZMwnjwoC8TZYWrPsQ=; b=Sls9STu3kHLeb9Pf7Rj3HPFK0NXTXoZypR5CK7+yiVz//X77W5WK2mMVdm16kYF6SuHLa5Cx51DWi5c/h/Y3Gayx+eDPTsBEYsUxmezsjs/MD9EsH7W8mj37YWOgFjejOrqc5iOmbo0HcsqiO+nNZgLGZBQfPSGWVrWF7/iR4WdO1cbtJicMkXQfIloVRRSX2rDrKlZMUaU9C34rLf/fl1NAjmD+Ie7NDCFoEAm9MR66zCzTiI5bYVqq6AIe6FuVhDZuFJvs6c+DR0EA+6+al/koe0n7FlEvi28APnMMQwt4wtnQhcGvfmcxPevXNbti8e/A9qi+1dS9KEgVNGWC6g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XpSpJm3nZ4qEUIa2z4eeiBY340ZMwnjwoC8TZYWrPsQ=; b=RK6lQMH4nmqxzJqXeuAxtkFcnd6Qyz161DBLSF0ZfTZKiubWRDeJpmFI7T03CIPLftxqL3op1TtTNboPknobXOHQF+A4ar8LgQY4XTzSlrIlp68erv8UrM72ZWh5X5HaFLRv15uJElSfHpfcwFtuOxiwva1xtheiV8QZHY44Zos= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BLAPR10MB4881.namprd10.prod.outlook.com (2603:10b6:208:327::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4436.24; Fri, 27 Aug 2021 14:59:22 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198%5]) with mapi id 15.20.4457.020; Fri, 27 Aug 2021 14:59:22 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , Christoph Hellwig , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v4 13/14] mm/page_alloc: reuse tail struct pages for compound devmaps Date: Fri, 27 Aug 2021 15:58:18 +0100 Message-Id: <20210827145819.16471-14-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210827145819.16471-1-joao.m.martins@oracle.com> References: <20210827145819.16471-1-joao.m.martins@oracle.com> X-ClientProxiedBy: LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (138.3.204.8) by LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4457.20 via Frontend Transport; Fri, 27 Aug 2021 14:59:19 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: a3c6c459-e426-4341-7ba9-08d9696b40f3 X-MS-TrafficTypeDiagnostic: BLAPR10MB4881: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5516; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: psU1ci5OGR/PO19YcNTKhiRCT1ExYGc4ZAIK114n4/8iGd6pePPrmImEOQDZ3dCnimMDzcmoTmu6H2a0ohX7w4A3hs/t8Pd/hINJID75rV8uXSOp3qhdPsfUlDqDZ1YqCMTSjhwmjHxq60Ox2Gm0FUdsNXd6xJY0LMnlYdLsC/UbpfOv+/wZZdiMXj2IEt2yj5lY4Na5t7cr4VE2VyULMuaEBE7nRvzSNN3Fi1FyewRf2FzZK59HoaAwCnmlz2AFCbvxf8V2i1G5JFilpGHaln9iKy0BleVQF027NOvD2lS5xXEjiDDpoMACqBsp6Sz5MDw0SmnqQdziwYX4ZN1KnZDl5zrBPEcvGtAcMryavap88+e7o6KAttvCTvRF7XUByCjvGBnXp53mmTl08rtBObo2EYH7V9s7BmkKbnM8dhsVaN6LLABekq3Gmte1epekjK57Ba8F4C5tuPvXC09w1LAKQtQI+czf57nfchd0kkzarAvB/7jzFS3uYgeexboAo0R19P6gOyFHBXxphDPSZXntIZ+GlqvyhzqJy/E33MgmU3Tltrk0mgOQAugvQhgsDMsSVlzylXeBTowTiYL2QvpYAIGeT6RcCbbdI1Omkuu6enkMVZYVBQs3Nt+v6PYYGedLybpHAfZXocgw11oplp8j3l3tA+9Ntdwp8fc5gpaRxr06ZYp+kUbXk/0dUQSTOXXeYt9N4vk/bCf/fnOokQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(396003)(376002)(366004)(136003)(346002)(39860400002)(7696005)(2906002)(83380400001)(38350700002)(38100700002)(52116002)(66946007)(8936002)(478600001)(7416002)(107886003)(86362001)(36756003)(6486002)(316002)(1076003)(8676002)(103116003)(26005)(54906003)(956004)(4326008)(2616005)(5660300002)(66476007)(66556008)(6916009)(186003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: T60hV4y8JopaAAQMTOZoIPh0Q7zI/PhjNuoKUFYv0wwnXsLz5aLlJpUV9REL6PBz7bczu1BoN9D9dPaArZ/MhZl4BSFf0Twar8FHhkS36MZDWbZICUcv5Z3M53AUcP0CiZ+nXuJiBXSS3pzC3KJExaY0F0rMWrE5GpAnGCgDjoZYY+N4dXvuQ7eG5AL8WFgKd4i4+xzqevaD4RlJhmY32+nXdu1/ITJKHEeCFr2DBqDRu6kBYjZqW1dWz2dzSOPHKbG/m+W6gBsAUab0/RsYPscKYOEA0y+vDo67YsN64CZoYl3JSIjBpcqa54KvhWma2IsRbadxwDP9D/qbPoDtYs+okyDNC59Q4lCib+FuJY1QpZjy1oukEat7h74F77txjBC72Pl8dXQUx9J2E15v1BE6BwhxO/GauF0QuzhyST9zoT1isCjrbLaSf714fSYpEV4TaBGEyfB1QtdYKqF8ZOFjNXJxaiajvem+JZQ4/2qiRDZxulKZjFfN2kSeRC7sCVa1F6sw/7O/v2uuMe97T0xYPYDlZQ87qZmJZlDlncnR66XGT+D6w/loJhnv/dA6l15+Niq7qt1Fv2qNTqq6yEX0ceFpJe4DJ+DpjRSj2aYhZnXIxbDhogdmT6VZ0GogWAiXWoUyccQVttZCVokxMmKK7xitOmYakbyfRxTCff5Ps54Cv8yhhLdy0Aw7jiEWpAkxOLAEyEoY7m/yzRRJg/HxzFLtrpV9Z2XUMadXNYQX42CSoeo3VoeBJ17BwBhNyFMKvgBW1gFhYKbeLqCFTqmEc//FEaP9mVPd+gCJzLRmtFrgkntDcwrkZ0w1FxY8mP0MAJANRu0ul7S88Sii/x5dJ+lO28fXG3A8oL4XJbcCvh+Qjg5Jf66WEyomynHEScO/J34iEQnVa0UIHLm6K7SrT/sSKgjZHyogAv9RYI8gp9fyh385DvnP98KltleQYMBjzWSQEB93kkQklVHqDK8kjosMrWdG34ZixJubzGrl0RNZnGGW3LAIaCED3ioPcaqO0645zWA9eX/1b+ZcPCnu33bcdWa/k98TEJCTBbd8cKTSPtt+gEvFAoOPPT20WDEAKi+QkY+fgtb/MLfNSSwQ7nCdpV8/jg3mGr/5/ybPrz2k9tBnajTRjJAfqprWlh9ToIgFx9GPNeVGsfwDPDgbEgQwjRgrvITGnInEuLyMQAWy88r8MH0D/4DL9ZGljE1TmZQKlCMmphEFeVr1bVNqd9btsUc8o0STFEIuafcLWQ+qCpxyr4hB40RcVdEVv4f9TcVztM4hQJvkKB7GJUij/0Bwu6GUXeq0RaZuqCyTXH9x+gGCno7VffLkM93s X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: a3c6c459-e426-4341-7ba9-08d9696b40f3 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Aug 2021 14:59:22.3334 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: pk7+OqImxE1v/vaSf2iSjZ+5gvLLJA0upkx+TdiEnhBIOlfQir0qKI0BG/ciSBV0KT/M+n4mpT50/Dt/kIC0PrHeTNnOoc1wsjYU8z6D2wE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB4881 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10089 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 phishscore=0 malwarescore=0 mlxscore=0 bulkscore=0 mlxlogscore=999 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108270092 X-Proofpoint-GUID: ZzlN6-x3mIww8QBdldqQCLmrC-hX6o8W X-Proofpoint-ORIG-GUID: ZzlN6-x3mIww8QBdldqQCLmrC-hX6o8W Currently memmap_init_zone_device() ends up initializing 32768 pages when it only needs to initialize 128 given tail page reuse. That number is worse with 1GB compound page geometries, 262144 instead of 128. Update memmap_init_zone_device() to skip redundant initialization, detailed below. When a pgmap @geometry is set, all pages are mapped at a given huge page alignment and use compound pages to describe them as opposed to a struct per 4K. With @geometry > PAGE_SIZE and when struct pages are stored in ram (!altmap) most tail pages are reused. Consequently, the amount of unique struct pages is a lot smaller that the total amount of struct pages being mapped. The altmap path is left alone since it does not support memory savings based on compound devmap geometries. Signed-off-by: Joao Martins --- mm/page_alloc.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c1497a928005..13464b4188b4 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6610,6 +6610,20 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, } } +/* + * With compound page geometry and when struct pages are stored in ram most + * tail pages are reused. Consequently, the amount of unique struct pages to + * initialize is a lot smaller that the total amount of struct pages being + * mapped. This is a paired / mild layering violation with explicit knowledge + * of how the sparse_vmemmap internals handle compound pages in the lack + * of an altmap. See vmemmap_populate_compound_pages(). + */ +static inline unsigned long compound_nr_pages(struct vmem_altmap *altmap, + unsigned long nr_pages) +{ + return !altmap ? 2 * (PAGE_SIZE/sizeof(struct page)) : nr_pages; +} + static void __ref memmap_init_compound(struct page *head, unsigned long head_pfn, unsigned long zone_idx, int nid, struct dev_pagemap *pgmap, @@ -6673,7 +6687,7 @@ void __ref memmap_init_zone_device(struct zone *zone, continue; memmap_init_compound(page, pfn, zone_idx, nid, pgmap, - pfns_per_compound); + compound_nr_pages(altmap, pfns_per_compound)); } pr_info("%s initialised %lu pages in %ums\n", __func__, From patchwork Fri Aug 27 14:58:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12462237 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BCF3D3FD0 for ; Fri, 27 Aug 2021 14:59:37 +0000 (UTC) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 17RDWHDJ002206; Fri, 27 Aug 2021 14:59:29 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=IxjwUQbC39PYrInE4g3Z9kpgnSdsw3Plbkz7I6s3XTo=; b=ob4cqpw0jMad1oicN2YFs84ng1W0m/BM26qvcZGBarMjJXYfnGXwl6Nlejr52V02188u YDU0mTOF8PMDOsmBV6V46T6W3e+Kib5sW5xFnhW/UvCwyNsoB1hUBRRshQ41POWwaAkn M/C/0GrS3bfYGXn+54IGriPaDIKyXffzmRe2+BGyG5Il5xRjC+T/tf8E0nfbBPqCtOCL d76X2l/vVBBVAkBM98FxRYWiHAWoyIqF2cOcp3mRbp4p4dsejrBGK7Mhks4vahBaep2U sba7rwpAz9kDT+1hlpobXoxi0LYK6WFYt3muAzuYbLXPP4fPjtSN/Fsu1YNnm+bAnLuT 3g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=IxjwUQbC39PYrInE4g3Z9kpgnSdsw3Plbkz7I6s3XTo=; b=MHJxPNN0DH06l9qnbkGYe/jCy5e7/qKaLmuzo8j9ZmHFYIeHnLZdnF5wnfdjqoHFfIw8 utnF1o9dgc05Q8XEV5PV6LHswOAcniFBz1RmMIaNQsdm4quAa9UuvuSpkFMaQ+OalOei dcsgjKB52NZK6jMPVbVyKShMCWGqxaU4mOnSLjFRLg0kG7S/bNttkSzlIbiDkklHumG7 fvzath6vlLh/JYOlFqDlh+KXB+qEVywqdCCJqaO1X/2Q0enhX0TMNpDgYUfr3HdYb4Hc FpTx53pKDlaU12b+ia9baFMsS+NOmblDCpeWPVWalt2/KsYqoy1xUXohnj1MFoITS0sO MQ== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by mx0b-00069f02.pphosted.com with ESMTP id 3apvjr0vje-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:59:29 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17REpOCI187113; Fri, 27 Aug 2021 14:59:28 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2176.outbound.protection.outlook.com [104.47.57.176]) by aserp3020.oracle.com with ESMTP id 3ajsabbqvx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 27 Aug 2021 14:59:27 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IyQjD868+t+Rs23AnWhJZjHCN5jxSpEexpAL478IVDE0D6Bq4q4e11x31Bn00CSIZp4OoGz2Gv5eLy1SOPgl6ndo1t735dQT7n54N7cM5Jqi0ma2RZqLOpfzZEvEGjjGM1T3ZT/3gAMc5t9Fnkr/fdIAI5nHjKR/k5dfnUqTsOaE+Ep9EaWq4t3dPJots9jexzSM4c63r/im3ergQXPi+OXTCeL6HRcESwbkknqv7d41EyQySC7lxn8qdCKlWDpDLEITqCvckbkyW30J+yWL7/Jt3DrBNwobUZtTNk0fPAnEGCOSY++T4XYTEnwWOuCNN6wH2fiQNg4/CWvNoT1spw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IxjwUQbC39PYrInE4g3Z9kpgnSdsw3Plbkz7I6s3XTo=; b=I82xcVkKqP3SXbaoNublxltALOJNrqBCmJ4tFK8KYPu6J75xlc30AxZaXAmRJUDBjKC6y26SWbDuPxsR2LskLxk3BP2J0oV18kRWLjjQOksDu/9bSmUrQaQx4vA6FWTwpD7i5wJYOSTN/ZQtCWtUTQcuxiU412MyZzwbrEh0GLOHeJ1N93Si32hzrwMZrzY9FwyzQwVmK/Yk9WlXInas1JW/3BLUnvv8e/2yYeGVuFoPyLgoZtZJX9+MFPmdoEjzfgF9ERu+WyeiO3eJqGFusTWOo80AkvmzCOWKhxmn9VOaSFUwRRXM7ZR8X2RjKny8zyb+qacEzcGploHEeOOVQw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IxjwUQbC39PYrInE4g3Z9kpgnSdsw3Plbkz7I6s3XTo=; b=BKIPz4CW2KYBHB6xECaFtJFotmQ1xqOL8KJT4qCBk4Lh+bipuBu7J7FuMLap005/gfVM609p4Ujiv9Leeriu3HY7CfHuwvV1c2pvfwKczN/836AFgtfYYLTc/eNCTjkI1CbU3Ju2/mDrPpptrpp6P9dCD2A64g6fOBAGzJo50rE= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BLAPR10MB4946.namprd10.prod.outlook.com (2603:10b6:208:323::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4457.17; Fri, 27 Aug 2021 14:59:25 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::c9e9:caf3:fa4a:198%5]) with mapi id 15.20.4457.020; Fri, 27 Aug 2021 14:59:25 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , Christoph Hellwig , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v4 14/14] mm/sparse-vmemmap: improve memory savings for compound pud geometry Date: Fri, 27 Aug 2021 15:58:19 +0100 Message-Id: <20210827145819.16471-15-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210827145819.16471-1-joao.m.martins@oracle.com> References: <20210827145819.16471-1-joao.m.martins@oracle.com> X-ClientProxiedBy: LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) Precedence: bulk X-Mailing-List: nvdimm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (138.3.204.8) by LO4P123CA0443.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1a9::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4457.20 via Frontend Transport; Fri, 27 Aug 2021 14:59:22 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ca1f022c-e285-4f82-2d6b-08d9696b42f1 X-MS-TrafficTypeDiagnostic: BLAPR10MB4946: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8882; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qt4cDl8azoue81zpiW0pG8LM+77oUyS6fy2EVzjon6EbLT7U0jSFDiTa4/tMyIHz7IuTnH+A0o0v0vbAayak9wDnoKaUNl9bxLj2mlUqoCcFIi8zsNTLWzqzjPzersQwtaEtFQKItMaIKEZ+zBzdf9o2bjewFmJxBISUgjDY8NxC8sHftYBWcGlZMPZ/cmWU90IuYSrFp/M9sVBl1muo1aFEdQDaToXDdgOWbEA+w20520kGZXXjjSJWdG4d/r3ziWaUzn7xSqcLxT37gBxQ5s1XXNNGLWTFLGiK4CcHWm3OJN2zIiTN0yhPKaBYq+koQi3B2EfPcY1IQPlX+wZ2INqeY5bygOW6yEEpleUfovw3C3fcEUYggrGfG8go8Rq2Dnbl9AZcPby64k7iR1/aQnuVW0iRxoQSyFBDqMgf8WqwesT3vzYMPpm0/nJ0/4g1rzy+T3RQAjAgd7MxrqwZ1LyhNCzkpQFxLPM5xzT2rQeMbGv+vbVXmR4QAn3d8xdwQcdb8FMrzTxbEYSdYJwJHxAM0Uh+W/B+Y/ML9cpNijE6yeOy1dxfVM5uqGAbRy5OuOdTQIG/XpyLCJd4a46Nr2dTy9pVBd+OfDYJqi3/c6Y5wgc40ujElzUqKoe2lvJMFqhe8FV78xry0CdSWLFkkiKNLhuRxItbFzga+y8NFkjOhKaRh+3m0CXoEre36gze8RT+8Wt2cP2pBquPUn4tyynTMKYNL9PjXqXYh5+9f0k= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(136003)(376002)(346002)(396003)(366004)(316002)(86362001)(1076003)(2906002)(7416002)(30864003)(107886003)(8936002)(66476007)(66556008)(186003)(66946007)(478600001)(38350700002)(36756003)(4326008)(7696005)(2616005)(8676002)(5660300002)(52116002)(956004)(103116003)(6486002)(6666004)(38100700002)(6916009)(26005)(54906003)(83380400001)(25903002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: VVV6e70C5iRq38ybStB8gAivSf1NgzMtyG6gDAtUpFzF8AyD+2Z3V0gsqXbrBruEj7Xb4Ce72VNY1pGcOszkQZzGBSyScTJs0ygMq5ZN47kPSkjvgLR4sp/dAQfA2UtfRmFGmM+RNUjYvY6MsPbTSDMhC+8abluVD7LXFlbkdlNdp7cK/BYaNkODWZ17GPhuyYHPvegfkdMNqfDu3QRQGZdwQZh1PP3yNcSVL9ZFz790yYzh3IAfv/svVNvDh4Ay9WGtGU8LfNwLgRgR2cbU6K9vLHd3MOoedxNJuL3WMd9chjbR6UtuY0Zq9EtwXgT4yiSjkSsDmDJgYpQvcwWXhe7U52wGuBOyFydX3Ybswcz8M1TfaNgWbEjKGm9bk8e3ivah1tiOHALOYuCLg0WPTlly4JygOETjMh912bEtAD4Prxh2erwq/WcDCByQCkOy9LX1sH6FP+OcR7A1xql19eY1g+ogJVRwCt4fCbqIIjk3V/pXtKE9HvcHcStEPc+qD1rVQwBbhLi3/Wp0qOJNC6dBbgSrKdspj07kpvg2LbVSDFXgkr7FDQnZfO75Pqz5vd830E3O4lOCrfGdgfZ2hFZ5v5eWQiu8NZMCUrq/RYxlGhhRTKRF23cDUhhkk9ZpBWpexxqtnYHi5TH+UQgnlwyKKZsPoJSXrQKj+roGC0GrquXRdXYrYjwd7qPkBOj9/fYzgWz3Nrbe5aN76QNrGLxip2BrDRbZ7mX4qrXRvCmiiR/i6Bj2BrIuFiORc4yQUGxXIKSDvxMwuywwUXCIlxyULwCeySSXI+xeQUbpDqMLxsLD4gcf4wqfctuSqj9O2vaS4xTQPqb4Wl1fVT1W+pKFGjFdWxIV7DNk44E8ufhT+QVgNDx5YHblEFHHnPcZ3/qFAKun6jtLjRdwNCQxrazS6TGQdivCfAWyL/cARry2jirYQ9zwdzdPs74jazcnHX92pbh7CzqIJBjbOCPqXBfHiPX4EO8B2vatbk3dizh8pM4MGbSflJKg+qs45DAWOXoV40cn5DxXDJdUeUfjNggNFyW22JH1GvW/oOxYPv7PrbJbf1W/qhMe/bdj4lNh+zJ3ZWmHGMNwXsaG+2w/UsFt5xJ59GfKiR9vve6+OZnf1cBCiLiwIIvyJJ1rHVmoh1naj5ZiNsvjCFhKgZCuDMcTeVknXNGFUEDA6/usPMR2e86BRc4A3w48IlVLF0k4fUSvoxkBETRwnzGOOYg3NYyH3W2ThYIuD29Tefygt1hPxfk5jFEwqGhzaSdqiQHzcQozN2teQGmZYqp25IrIPVTUAxwIsORZO9M1V2eSYoxW823WFBwa85zFMq0M30BQ X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: ca1f022c-e285-4f82-2d6b-08d9696b42f1 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Aug 2021 14:59:25.7733 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: s1m98ExqapkQ+of+bAwIZihJqNJil+MU9QzTouO2siszUbi+ou4sdQNSggUIJsqhIUjEbdrkpRDhMUJ8ODYx3RpuN3vI2VAQWDFh13EDa2w= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB4946 X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10089 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 mlxlogscore=999 bulkscore=0 mlxscore=0 adultscore=0 malwarescore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108270092 X-Proofpoint-GUID: D7_rhbCZ7hS-_r3n1jyu9sBJnblsDAEq X-Proofpoint-ORIG-GUID: D7_rhbCZ7hS-_r3n1jyu9sBJnblsDAEq Currently, for compound PUD mappings, the implementation consumes 40MB per TB but it can be optimized to 16MB per TB with the approach detailed below. Right now basepages are used to populate the PUD tail pages, and it picks the address of the previous page of the subsection that precedes the memmap being initialized. This is done when a given memmap address isn't aligned to the pgmap @geometry (which is safe to do because @ranges are guaranteed to be aligned to @geometry). For devmaps with an align which spans various sections, this means that PMD pages are unnecessarily allocated for reusing the same tail pages. Effectively, on x86 a PUD can span 8 sections (depending on config), and a page is being allocated a page for the PMD to reuse the tail vmemmap across the rest of the PTEs. In short effecitvely the PMD cover the tail vmemmap areas all contain the same PFN. So instead of doing this way, populate a new PMD on the second section of the compound page (tail vmemmap PMD), and then the following sections utilize the preceding PMD previously populated which only contain tail pages). After this scheme for an 1GB devmap aligned area, the first PMD (section) would contain head page and 32767 tail pages, where the second PMD contains the full 32768 tail pages. The latter page gets its PMD reused across future section mapping of the same devmap. Besides fewer pagetable entries allocated, keeping parity with hugepages in the directmap (as done by vmemmap_populate_hugepages()), this further increases savings per compound page. Rather than requiring 8 PMD page allocations only need 2 (plus two base pages allocated for head and tail areas for the first PMD). 2M pages still require using base pages, though. Signed-off-by: Joao Martins --- Documentation/vm/vmemmap_dedup.rst | 109 +++++++++++++++++++++++++++++ include/linux/mm.h | 3 +- mm/sparse-vmemmap.c | 108 +++++++++++++++++++++++----- 3 files changed, 203 insertions(+), 17 deletions(-) diff --git a/Documentation/vm/vmemmap_dedup.rst b/Documentation/vm/vmemmap_dedup.rst index faac78bef01c..65aabfa2ca0b 100644 --- a/Documentation/vm/vmemmap_dedup.rst +++ b/Documentation/vm/vmemmap_dedup.rst @@ -189,3 +189,112 @@ at a later stage when we populate the sections. It only use 3 page structs for storing all information as opposed to 4 on HugeTLB pages. This does not affect memory savings between both. +Additionally, it further extends the tail page deduplication with 1GB +device-dax compound pages. + +E.g.: A 1G device-dax page on x86_64 consists in 4096 page frames, split +across 8 PMD page frames, with the first PMD having 2 PTE page frames. +In total this represents a total of 40960 bytes per 1GB page. + +Here is how things look after the previously described tail page deduplication +technique. + + device-dax page frames struct pages(4096 pages) page frame(2 pages) + +-----------+ -> +----------+ --> +-----------+ mapping to +-------------+ + | | | 0 | | 0 | -------------> | 0 | + | | +----------+ +-----------+ +-------------+ + | | | 1 | -------------> | 1 | + | | +-----------+ +-------------+ + | | | 2 | ----------------^ ^ ^ ^ ^ ^ ^ + | | +-----------+ | | | | | | + | | | 3 | ------------------+ | | | | | + | | +-----------+ | | | | | + | | | 4 | --------------------+ | | | | + | PMD 0 | +-----------+ | | | | + | | | 5 | ----------------------+ | | | + | | +-----------+ | | | + | | | .. | ------------------------+ | | + | | +-----------+ | | + | | | 511 | --------------------------+ | + | | +-----------+ | + | | | + | | | + | | | + +-----------+ page frames | + +-----------+ -> +----------+ --> +-----------+ mapping to | + | | | 1 .. 7 | | 512 | ----------------------------+ + | | +----------+ +-----------+ | + | | | .. | ----------------------------+ + | | +-----------+ | + | | | .. | ----------------------------+ + | | +-----------+ | + | | | .. | ----------------------------+ + | | +-----------+ | + | | | .. | ----------------------------+ + | PMD | +-----------+ | + | 1 .. 7 | | .. | ----------------------------+ + | | +-----------+ | + | | | .. | ----------------------------+ + | | +-----------+ | + | | | 4095 | ----------------------------+ + +-----------+ +-----------+ + +Page frames of PMD 1 through 7 are allocated and mapped to the same PTE page frame +that contains stores tail pages. As we can see in the diagram, PMDs 1 through 7 +all look like the same. Therefore we can map PMD 2 through 7 to PMD 1 page frame. +This allows to free 6 vmemmap pages per 1GB page, decreasing the overhead per +1GB page from 40960 bytes to 16384 bytes. + +Here is how things look after PMD tail page deduplication. + + device-dax page frames struct pages(4096 pages) page frame(2 pages) + +-----------+ -> +----------+ --> +-----------+ mapping to +-------------+ + | | | 0 | | 0 | -------------> | 0 | + | | +----------+ +-----------+ +-------------+ + | | | 1 | -------------> | 1 | + | | +-----------+ +-------------+ + | | | 2 | ----------------^ ^ ^ ^ ^ ^ ^ + | | +-----------+ | | | | | | + | | | 3 | ------------------+ | | | | | + | | +-----------+ | | | | | + | | | 4 | --------------------+ | | | | + | PMD 0 | +-----------+ | | | | + | | | 5 | ----------------------+ | | | + | | +-----------+ | | | + | | | .. | ------------------------+ | | + | | +-----------+ | | + | | | 511 | --------------------------+ | + | | +-----------+ | + | | | + | | | + | | | + +-----------+ page frames | + +-----------+ -> +----------+ --> +-----------+ mapping to | + | | | 1 | | 512 | ----------------------------+ + | | +----------+ +-----------+ | + | | ^ ^ ^ ^ ^ ^ | .. | ----------------------------+ + | | | | | | | | +-----------+ | + | | | | | | | | | .. | ----------------------------+ + | | | | | | | | +-----------+ | + | | | | | | | | | .. | ----------------------------+ + | | | | | | | | +-----------+ | + | | | | | | | | | .. | ----------------------------+ + | PMD 1 | | | | | | | +-----------+ | + | | | | | | | | | .. | ----------------------------+ + | | | | | | | | +-----------+ | + | | | | | | | | | .. | ----------------------------+ + | | | | | | | | +-----------+ | + | | | | | | | | | 4095 | ----------------------------+ + +-----------+ | | | | | | +-----------+ + | PMD 2 | ----+ | | | | | + +-----------+ | | | | | + | PMD 3 | ------+ | | | | + +-----------+ | | | | + | PMD 4 | --------+ | | | + +-----------+ | | | + | PMD 5 | ----------+ | | + +-----------+ | | + | PMD 6 | ------------+ | + +-----------+ | + | PMD 7 | --------------+ + +-----------+ diff --git a/include/linux/mm.h b/include/linux/mm.h index 77eaeae497f9..ff0f7d40c6e6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3172,7 +3172,8 @@ struct page * __populate_section_memmap(unsigned long pfn, pgd_t *vmemmap_pgd_populate(unsigned long addr, int node); p4d_t *vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node); pud_t *vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node); -pmd_t *vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node); +pmd_t *vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node, + struct page *block); pte_t *vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, struct vmem_altmap *altmap, struct page *block); void *vmemmap_alloc_block(unsigned long size, int node); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 441bb95edd68..dc3a137ec768 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -537,13 +537,22 @@ static void * __meminit vmemmap_alloc_block_zero(unsigned long size, int node) return p; } -pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node) +pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node, + struct page *block) { pmd_t *pmd = pmd_offset(pud, addr); if (pmd_none(*pmd)) { - void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node); - if (!p) - return NULL; + void *p; + + if (!block) { + p = vmemmap_alloc_block_zero(PAGE_SIZE, node); + if (!p) + return NULL; + } else { + /* See comment in vmemmap_pte_populate(). */ + get_page(block); + p = page_to_virt(block); + } pmd_populate_kernel(&init_mm, pmd, p); } return pmd; @@ -585,15 +594,14 @@ pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) return pgd; } -static int __meminit vmemmap_populate_address(unsigned long addr, int node, - struct vmem_altmap *altmap, - struct page *reuse, struct page **page) +static int __meminit vmemmap_populate_pmd_address(unsigned long addr, int node, + struct vmem_altmap *altmap, + struct page *reuse, pmd_t **ptr) { pgd_t *pgd; p4d_t *p4d; pud_t *pud; pmd_t *pmd; - pte_t *pte; pgd = vmemmap_pgd_populate(addr, node); if (!pgd) @@ -604,9 +612,26 @@ static int __meminit vmemmap_populate_address(unsigned long addr, int node, pud = vmemmap_pud_populate(p4d, addr, node); if (!pud) return -ENOMEM; - pmd = vmemmap_pmd_populate(pud, addr, node); + pmd = vmemmap_pmd_populate(pud, addr, node, reuse); if (!pmd) return -ENOMEM; + if (ptr) + *ptr = pmd; + return 0; +} + +static int __meminit vmemmap_populate_address(unsigned long addr, int node, + struct vmem_altmap *altmap, + struct page *reuse, struct page **page) +{ + pmd_t *pmd; + pte_t *pte; + int rc; + + rc = vmemmap_populate_pmd_address(addr, node, altmap, NULL, &pmd); + if (rc) + return rc; + pte = vmemmap_pte_populate(pmd, addr, node, altmap, reuse); if (!pte) return -ENOMEM; @@ -654,6 +679,22 @@ static inline int __meminit vmemmap_populate_page(unsigned long addr, int node, return vmemmap_populate_address(addr, node, NULL, NULL, page); } +static int __meminit vmemmap_populate_pmd_range(unsigned long start, + unsigned long end, + int node, struct page *page) +{ + unsigned long addr = start; + int rc; + + for (; addr < end; addr += PMD_SIZE) { + rc = vmemmap_populate_pmd_address(addr, node, NULL, page, NULL); + if (rc) + return rc; + } + + return 0; +} + /* * For compound pages bigger than section size (e.g. x86 1G compound * pages with 2M subsection size) fill the rest of sections as tail @@ -665,13 +706,14 @@ static inline int __meminit vmemmap_populate_page(unsigned long addr, int node, * being onlined here. */ static bool __meminit reuse_compound_section(unsigned long start_pfn, - struct dev_pagemap *pgmap) + struct dev_pagemap *pgmap, + unsigned long *offset) { unsigned long geometry = pgmap_geometry(pgmap); - unsigned long offset = start_pfn - - PHYS_PFN(pgmap->ranges[pgmap->nr_range].start); - return !IS_ALIGNED(offset, geometry) && geometry > PAGES_PER_SUBSECTION; + *offset = start_pfn - PHYS_PFN(pgmap->ranges[pgmap->nr_range].start); + + return !IS_ALIGNED(*offset, geometry) && geometry > PAGES_PER_SUBSECTION; } static struct page * __meminit compound_section_tail_page(unsigned long addr) @@ -691,21 +733,55 @@ static struct page * __meminit compound_section_tail_page(unsigned long addr) return pte_page(*ptep); } +static struct page * __meminit compound_section_tail_huge_page(unsigned long addr, + unsigned long offset, struct dev_pagemap *pgmap) +{ + pmd_t *pmdp; + + addr -= PAGE_SIZE; + + /* + * Assuming sections are populated sequentially, the previous section's + * page data can be reused. + */ + pmdp = pmd_off_k(addr); + if (!pmdp) + return ERR_PTR(-ENOMEM); + + /* + * Reuse the tail pages vmemmap pmd page + * See layout diagram in Documentation/vm/vmemmap_dedup.rst + */ + if (offset % pgmap_geometry(pgmap) > PAGES_PER_SECTION) + return pmd_page(*pmdp); + + /* No reusable PMD, fallback to PTE tail page */ + return NULL; +} + static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, unsigned long start, unsigned long end, int node, struct dev_pagemap *pgmap) { - unsigned long size, addr; + unsigned long offset, size, addr; - if (reuse_compound_section(start_pfn, pgmap)) { - struct page *page; + if (reuse_compound_section(start_pfn, pgmap, &offset)) { + struct page *page, *hpage; + + hpage = compound_section_tail_huge_page(start, offset, pgmap); + if (IS_ERR(hpage)) + return -ENOMEM; + if (hpage) + return vmemmap_populate_pmd_range(start, end, node, + hpage); page = compound_section_tail_page(start); if (!page) return -ENOMEM; /* + * Populate the tail pages vmemmap pmd page. * Reuse the page that was populated in the prior iteration * with just tail struct pages. */