From patchwork Thu Jun 17 18:44:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12329259 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8A0DC2B9F4 for ; Thu, 17 Jun 2021 18:46:11 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6754F613E9 for ; Thu, 17 Jun 2021 18:46:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6754F613E9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CF34D6B0078; Thu, 17 Jun 2021 14:46:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C7C346B007B; Thu, 17 Jun 2021 14:46:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A2F866B007D; Thu, 17 Jun 2021 14:46:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 63EE96B0075 for ; Thu, 17 Jun 2021 14:46:07 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id EF388181AEF09 for ; Thu, 17 Jun 2021 18:46:06 +0000 (UTC) X-FDA: 78264095532.20.008BDA6 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf14.hostedemail.com (Postfix) with ESMTP id 75764C00F788 for ; Thu, 17 Jun 2021 18:46:06 +0000 (UTC) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 15HIawpr007633; Thu, 17 Jun 2021 18:45:41 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=Fs4CZCkv9VqQyCxw0vq4MCFN99w0gEVGwhskncAGQVw=; b=UiA2+DHiznrlVY/CJ4Sd9TiJYd7Ow7bP5HRM4AamgxIiEYi0KCAY/FjXPdg3y+IE9iWs 6j6yLi+vwpvPwJu6wnfVaMD42KrHG9wZUYj2KhsuPR/etJdozUEhKt1E9dVrVenBXtcp DhWM/wcV3z41vbiG6DmJIZfyOtng+wu3uiFtkIxC5mfBaBYXuLbsEI5qi1j03lm3Tlo+ SRFSGxtKtxmbRVPN+VypkmRq3dKhvfUcDhjXmkL/dSVUy76uUkRZVb34t89bxwMqWJa0 ZE9Tbrxz/MfCiOtfFaHNubsCSDjGA2xWOaMkcXi8AhRNQVrY66NjNpEN2ZqnO8JtqVk1 xg== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 397w1y1r5u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:45:40 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 15HIjaO5180356; Thu, 17 Jun 2021 18:45:39 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2174.outbound.protection.outlook.com [104.47.57.174]) by userp3020.oracle.com with ESMTP id 396waxy5wg-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:45:39 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OKxQOkYoIc/lByJh+rvBwfOZH/KPKokR35ZEgvY2+cNxWF4f5uG3Z7JU9iaxYGmFM4+axgUYX8//CqSDStyeFqtQLwYItwW7RVmPqjG8Xh3WgvfiyHreVcP+iSYeVxT/I9504ht0tvC0/ZysWnA8ZMv/ZVgQX2PUtczAZywt5kQUgZR8JHf0PP1FuWbHv+SFaVPJrJdQT+yBqAmxUvIN033Xa5At8IcYyFcOtncTgqeyYrmhz6L5RZAhudstQuFwkxKdFCZvt4sHdgi+X8HcxC8xipRE0eHazJk6A65PI/rG0XrlaZsqNIZkrN1y7D0jdutBC0uGsFdupb+iSAR9Jw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Fs4CZCkv9VqQyCxw0vq4MCFN99w0gEVGwhskncAGQVw=; b=FTpW6OduxUqJKHrfJkeIJaZWrhPAtR9JB8BSkYp7oN4i6dUir0L/sAisX7/lTAeeEM4ru+pP/fYKVanghIVvtiNVHCh90sbmScPQkYZ2ZPJ4YC4gVrKS2euTqMKqYOGqHwuJUGhe/AZhn4fqK/AGLf01fa8YvatDZMsEcVzFdSAeuB3zqwceCC7O0mXQ1v7FBxlYAyVQY9+WnWejCc8PW1IX5I9WAdBNXflRuijCt5h+osqrcyvrLlPW1NIEPfT59PcnrYkbIsSi0un/C35MaJmvp6NI0WhJgPXXTCkzCmR+4vyHihT9+V/P4VWwnjUf9BfzC4/W8hr6u09u48/7hw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Fs4CZCkv9VqQyCxw0vq4MCFN99w0gEVGwhskncAGQVw=; b=Hp/mU86YX7QNMADyyiFCkNPiD743evYAoh+DnKfgcUgvT2nqAeRYkt9BNEtvk/T7hGogwx0T5EGeH5nZ4L/KITHezNy6SrNJA13DOHkMxXM5lm0vvfwF5cS9vG6OONuFX0jyksI6D69swLYgDHH4VHdFVL2M+0+qWFSlCcR3d84= Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BLAPR10MB4852.namprd10.prod.outlook.com (2603:10b6:208:30f::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.15; Thu, 17 Jun 2021 18:45:27 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582%7]) with mapi id 15.20.4242.021; Thu, 17 Jun 2021 18:45:27 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v2 01/14] memory-failure: fetch compound_head after pgmap_pfn_valid() Date: Thu, 17 Jun 2021 19:44:54 +0100 Message-Id: <20210617184507.3662-2-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210617184507.3662-1-joao.m.martins@oracle.com> References: <20210617184507.3662-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4242.16 via Frontend Transport; Thu, 17 Jun 2021 18:45:24 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 50fb0ab1-ad29-4252-a2cf-08d931c012d0 X-MS-TrafficTypeDiagnostic: BLAPR10MB4852: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Ni6nB2tJDUusgNf4b1Ck+znYrf60ZR+3szEc5cgMWeclpvePsrUQz2bds9uVwvbDy2kaNt7D4hyHYmvByqpQXZVJqIq/07ulwmHxW1AioRpxcR+wiHt4fRnjzuQKLH2IeFv85eHiumRDJuJpik6et155TzeAe790PzqtfZFREmfTp9JqiSIVKhVbLEC1DHfdIqTUeXwsUI3qw1bcssUQU4YB+eXb7HPn2465WIb87wkkH1xCTnWxgJ5eX3Zqhz0Ak+iZsm0kSfbDjzfkJQI/IdL13Y+J0N5ME+2n/N/dTe5V6E8C7Qd6UQND8ias6gGTvS3Rm0du3JcaKBTgOjZo7GZCBeWDxj/rOo0ggnc1DuSJDQg4KfwStQZiiZT19YWYEPOvsj4kRJ4E6IbzLgGHpgmbV+cu05P9vLZ0FuZTlby0aFUsJudIoImZ8jsQh7qtWe2z9Yrtb2ksXiRbRMy29SRQ77ts+psbtEDLgFtRpy91/Gf29Uf36CYwr9zU3Jt9H2h5PNQx7wvOS15bHqclf+pLgqi1jOMTFL3K8BRJpo3aQj5CrXV3uV95AY4pWThnriEiA6URuyMdNdidkNoFrEHVtvh6ds3a90yA7NBrWUgNZxBwLgRJL5f+Lj2kETt5F4Yhtn543VBrpslCLS/Zsc7UT1fa6vTRUpv7FRtNt2s= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(396003)(376002)(136003)(366004)(346002)(83380400001)(6666004)(38350700002)(4326008)(103116003)(7696005)(5660300002)(186003)(16526019)(26005)(54906003)(478600001)(38100700002)(316002)(107886003)(1076003)(52116002)(8936002)(66476007)(66946007)(66556008)(36756003)(6486002)(6916009)(2906002)(86362001)(7416002)(8676002)(2616005)(956004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: +q+yDFW97GKJfFH0v/cqQaLUmRWK20ocxNrOC4o1ySy9anZHvRm23VRRq7CisnxVCbrcubxoia2yyycLWCyEn00cMDBOnzZzMvuRAxRzos1NeyAqR4eYOH28jTkCqBQwmDsPDCRig6yim+WICTidHTmWfLS5JChjuvijSJ0yxNWtEq3yaMvIKMNENOUjWggo6OOCjtmDQ+vaJW2naZUDAMwlm0wgZNb83SHSFgB5yFtDcnxRWilOZbdFxwIdj8fpjtamqOTRYNXhZytA11om6S8yJaJvdP33eO6tJURPDGGstcFDUobP6NxAi/Qr51duqw94MSqQ+elOijPWGZkR8aeZu2L28456SiYdEDFxzqSNgd7oQhhonzkqYXzxj6aCrqM6nkTAy1HYHHCkbNaQTTpvceZvyDtYbZsupxnDogh2bfBdI/zELyUz3rRlH3HLQHvB0BciKG1tD5qATH9yebBZBNGST8TJmO7szbyfiCMZ5xBVAggfxe74w7DJRzhCywF9qKqRUGeqGxiGPRC3pA0xBaUwQ/DZ5xSZgcv1vzfSy2SdF/4LwMnh2TS23jk8cW8cKFE20XJhAed4yHkqxYaHFW9DaoN4aOF4CeLU4y5oUr8i78x/hdfGKTqb3IbsKP0RzIsTd3DOEruVshphx+TrjuHJdcCetRl8lQ16rYPEjSDeI2sQGXgFCDJ6OwqLPUX+/OPP0qAhJ70KI/+Gsx707u1VCTbgfAgStGsCnpdYVI7dswmGAmXK7pNSe2cUQHX+h6bB9saxyPDzZBSHRRNbtHMs80fiKd55mfwyUjyEYf5L2c3ds1ym+cxTxIx+MmxdaNIY3nLBkksqkSwdejoaX+0kLKuV8gGPGoWgvygMj5WfsDr1HTnPLBgz4ua8txuKU83TwfOfveSW4zzhxyXsG2WpE4OE0dOuJzCJEqCrqjKjyR8MbdxSosYJsJbSbCk4hYsOeKnJN9Fk2pHGco4Xpm20YRKNbRbwRvspmSsnnnPnddbSbGWUle0KxmQurgHPqJDT1txsJcahgvl/KjvZflxpOGVOCg1RhSFIZUyuZal8Z1QKrpJOAAvXwFcZzViT+VJA9yhGio5XMtnMx5iWlmNiG5B/+Bagn6wYhQ/QyzgXa92QoebgvCW0O4tK3sp8Wnlw/LJ5tiWqnvPm+Jp1vHq4sh4fBA1AKtOxfTarsdjoz513dFhhvmPxaHCK0WeWowEFRi67LfPHIUhTT8NQEPTz0zq4BcU+ePLK7+7HUVbsjrQrKwg1k9l1l1re2UsVeLYHj0mDGd4Bn0YpVlS4zrqukh9jozVupxDJipaa213+Br7eSp/rt01AYEuH X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 50fb0ab1-ad29-4252-a2cf-08d931c012d0 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jun 2021 18:45:27.1040 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: PWx3Aj5ZoUTs9Q+Tg5Qw1Q7zsdt5MsjwaEn8siqhn6u42ss5Jwebavjjq8Gk41wA2emW0qvA2ReuBsjW5A66WLpoZnBB+5oiMDC9fM2I7CE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB4852 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10018 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 malwarescore=0 suspectscore=0 spamscore=0 bulkscore=0 mlxlogscore=999 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2106170117 X-Proofpoint-ORIG-GUID: FTJaQ_avhxf5AYXab3dmRo98HavP0URr X-Proofpoint-GUID: FTJaQ_avhxf5AYXab3dmRo98HavP0URr X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 75764C00F788 Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=UiA2+DHi; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b="Hp/mU86Y"; spf=none (imf14.hostedemail.com: domain of joao.m.martins@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=joao.m.martins@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Stat-Signature: nujpm4kcowbxcqicbni9h9qtbsqeuzaj X-HE-Tag: 1623955566-456421 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: memory_failure_dev_pagemap() at the moment assumes base pages (e.g. dax_lock_page()). For pagemap with compound pages fetch the compound_head in case a tail page memory failure is being handled. Currently this is a nop, but in the advent of compound pages in dev_pagemap it allows memory_failure_dev_pagemap() to keep working. Reported-by: Jane Chu Signed-off-by: Joao Martins Reviewed-by: Naoya Horiguchi --- mm/memory-failure.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/memory-failure.c b/mm/memory-failure.c index e684b3d5c6a6..f1be578e488f 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -1519,6 +1519,12 @@ static int memory_failure_dev_pagemap(unsigned long pfn, int flags, goto out; } + /* + * Pages instantiated by device-dax (not filesystem-dax) + * may be compound pages. + */ + page = compound_head(page); + /* * Prevent the inode from being freed while we are interrogating * the address_space, typically this would be handled by From patchwork Thu Jun 17 18:44:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12329265 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03315C2B9F4 for ; Thu, 17 Jun 2021 18:47:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A7217613ED for ; Thu, 17 Jun 2021 18:47:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A7217613ED Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 353336B007E; Thu, 17 Jun 2021 14:47:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 329216B0080; Thu, 17 Jun 2021 14:47:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 17D1F6B0081; Thu, 17 Jun 2021 14:47:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0246.hostedemail.com [216.40.44.246]) by kanga.kvack.org (Postfix) with ESMTP id CFD966B007E for ; Thu, 17 Jun 2021 14:47:01 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 5F173181AEF09 for ; Thu, 17 Jun 2021 18:47:01 +0000 (UTC) X-FDA: 78264097842.02.51BC52F Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf24.hostedemail.com (Postfix) with ESMTP id CC25AA00026F for ; Thu, 17 Jun 2021 18:46:00 +0000 (UTC) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 15HIawXC007626; Thu, 17 Jun 2021 18:45:41 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=HJOY4OK+hvH8kqzf+T/g3v3TOM3Ict48IL4jQnIfAIE=; b=RQ0ri5VCPX4LOdXlCZKsrtTr9jWkdKwyuHHQSn3uNJ4qBDw+MhfDTyXn5jK9YnEp2dk8 mP8OuPyDA1EXEAu4Qrj6kgDn/5ogcgGEeXllplibwl4242kZOT/jSAPLAnq5yPjjMxGm 7o69PwRVovHUBcTZt0yQUAb9ZKY5ygVtYupeBMvjh1MdIGVDe18fducUw1YdIS2+bGo+ 9FcyOrFM51XzkUGFi3ToOCQle0KG70uL+6s9i9QKo8BLyJfOi56JvFuc2zcz3EDB0loe B6GwXNojWMaxPiXWeZbRYKT/pLcK25+gzSQmuJ9asK5q+5YJHjHBZV65c0LvWWr0ZPg+ xg== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 397w1y1r61-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:45:40 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 15HIjaO6180356; Thu, 17 Jun 2021 18:45:39 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2174.outbound.protection.outlook.com [104.47.57.174]) by userp3020.oracle.com with ESMTP id 396waxy5wg-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:45:39 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=i4lvCYwH9tzkwZpuNxDgykiWkdNOMX3OMU8uaFmz99TB+beHH7l2QsEJq0aLuiZm/tV48alfOHsyrnck8ZmL5om7i1NjKbbAOb9bNKxt/uBVhv2n0ZfWH3VN3NAwVuOb4WNrS7uR3MOx9I7s3n88ErGuB5y5lWZ8NYKxIDury8uJAXiSVqOIwkmZ3BzqFYW5JeBVZLPzdl/pfeUnbaDWSYlgNTDAK96cQ5l38OQWJHlIMvWHtqbZwHtKLCOuRijcnevKg9fm5Dy7wbzhUnvA2m8w+EhJujCb/N8lBFuY8sozncj3SmXFhVh1EDkQ0xCQGAHJt8seNuLC3ngsNkFSzg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HJOY4OK+hvH8kqzf+T/g3v3TOM3Ict48IL4jQnIfAIE=; b=VDDxFjfqigrIHXbYnTH1Aoi+NWfiyc8JvypcXuxW+lfUtG3gMPq2NFBuARH/lxvjrDbGJhzI3fIi9SwsY7pEvdvvjFZD6i7Bb+Orh9ca2GM8IJPuDnvYsuPImjpNNMri5sAVkcHcWYAE0tO6sz5+XUFsCTLTxZm7Ra5yxggMXUvLfDTycBTOUo7GiSYd3p/EeUOWI9hK0p89ddYOBn+KddQRiJbZT21KITYOOcExoE9Bd33Pd/P5Fhi8TV/HRWM78nSQXeg7SOjqd+I8JXDcyuIGmnc34KOZfJyPenBJxTGl4Eo+K8614/gSj6dw/QrmzkFII7ElTeQLlqlvW/UjsQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HJOY4OK+hvH8kqzf+T/g3v3TOM3Ict48IL4jQnIfAIE=; b=lW32py9yWySvV+t+sSCT70lDFXAIbeYp/sEZTqzPvC+9B6cMpNm1CbNmL/4jZR+0MfFAs9t5iTTTcAjqZvC6uW5qUIhPqub97Q0zSY7I9vTjwVfNztOFmwuWhkYTW45HkSvAsX8jro8Ip8FetA/1vWOuKkmBc/daNPLxFEGK+/M= Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BLAPR10MB4852.namprd10.prod.outlook.com (2603:10b6:208:30f::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.15; Thu, 17 Jun 2021 18:45:29 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582%7]) with mapi id 15.20.4242.021; Thu, 17 Jun 2021 18:45:29 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v2 02/14] mm/page_alloc: split prep_compound_page into head and tail subparts Date: Thu, 17 Jun 2021 19:44:55 +0100 Message-Id: <20210617184507.3662-3-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210617184507.3662-1-joao.m.martins@oracle.com> References: <20210617184507.3662-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4242.16 via Frontend Transport; Thu, 17 Jun 2021 18:45:27 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 93704ef9-e252-4b7a-80e3-08d931c01459 X-MS-TrafficTypeDiagnostic: BLAPR10MB4852: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1284; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: nXOy3raysiL7Qc7utr41Ss0WWHP1922U+AR15FaeVANyp+9/9+rR7TE1bCHCHYrlB89SMEVPiXDKQixzqW5VxQMYo0YiZfGV7fWXg3sUqvKTIwf1hUvnOVUXRVGAjrp6zBXjiO4fsQu8Z5K2s3lGps9cbzvuE5iIOLsTjRCCEGbB7ZYBJSfsaezxwkN7xFqu3GmRCLNjB8lpwfX6ZTQIDqgMTSD8PGSr+izwAHB8m/nDetERMKtm/TrXt+LNIAstqTQj1KB9kJJ/svwAOPl1DsArDYY5GAMr2eGM2kOiFkPzF5Iha2gXRxYFczYjTUI5hCo5TG3FaILvwHCsH6QgYkOXNzrS3uWgY5y5EJHEHd2jw+SZ/M4WkPVZFgEcvT3xvVsHf5E8C38k2J26FvQENz+wPD02VbL9bxTRzEAR0lpYKE8TZYEGMinmemPyHbgSNlXlQkP8eF2h9+l7U2ds0KvdfooIUjOGLeD+oQxIAbLXORLzRSD5mH12nDQRDqcX9NuDgIrEodjrcD0U5QOJhvtbbMheEUxcTYDAJsJ8j7Vcw1PDFKzpJoZi/9bsVPjp4xVUPeIEWaMkvNDCWffFUbOlxyd1VNe6a0OXDYaivJg5Kc/Vf3qdXjsrC0MlKbucCSsHo2Qbay+8ff5e6bf79hW5wZiYhE8itN2XkWAf9QE= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(396003)(376002)(136003)(366004)(346002)(83380400001)(6666004)(38350700002)(4326008)(103116003)(7696005)(5660300002)(186003)(16526019)(26005)(54906003)(478600001)(38100700002)(316002)(107886003)(1076003)(52116002)(8936002)(66476007)(66946007)(66556008)(36756003)(6486002)(6916009)(2906002)(86362001)(7416002)(8676002)(2616005)(956004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ryJbLSn4gEtLPceemFhHUnfTcft7QX970jHa3vNNBpOcRMFCWHL/+F9KSJU3ki1Q95QFLqeZGDKfTX0H57n2zEl0ZkiS22koiLA/9Ag7ZYpFN+nOZufxxRhrV+KuI4pBELoTcfAa9fuqkyq5uOA1PyUiBPV5ITsy4HCMlE34vfH46Rr4V2X8fz5dS86UEUXZK4ANqjIfJmYORjLnCTQ4QLHLCSEh+fQ/2sNCIOuL72NDHppSNJ9AaMzVWoH3jkso7CWOAGwl7xBzaBKunFgawVzGlZ0aKKHd8e09goqCL5GuZPEaE1Bqz07QLi1+1Owj/aALhcUELXA7GWnfJSbCHMzdEAqoiwyP6jvYorL4yyCixk2UCzFecvP3BK/E/aHQO3oUDWG1PUo9M8v1vgFAAAG7LEJooWXp47OsK1fWpHaQKjIZ5sDBct1+H+V5jCrFiZGqZsykyeWtDLg5DAtvlpdDsre2w3PTwg1t4Y32fnoFRVEmYPfr1PWyv0NEUEiPxm3snk1pVooTwH5rYaXPU46dd7erDUbJAT48fvTkj0BWDnYGDT4YUG1csv2EMhjShNjixX1L8dTDaW2peCXcyiqcwoeBDZs3p2ILBS9plZjwuXebSQIKYIC1G31Lpn9UmaBE1Mnsf7RAp+2OWFz9a0QVzg6FXyCknUt97ghfJrMUGvHdOw95jzZEXCexOU2xx+uEl9OByYHsGhoGCXgOdPWkjKRSsf6pC0/4YJVdzidNMVvogWkUVOzOH/cTwvm55Li14xh0jW0RK8QH1t7Bbjw6ppX5LoPEQB3AYOnx+DdLZEA4f86Ely3aO6bnM0J9ZlgW/J1rot6sz2gYVMBx9cnvy2e1nfS+gfYenV6siy18pu/o5ROwEVhWF/ZxP8wrxxl3kWZR3u+SKuT/5hwcqIUQbhDrKu0d8dVvLQKbX1JZLRwAm03JASjY36J2liXtL6NsWQNqczZYSSfAMxOVPdRORSaPGVD0ip5C2bhNxe8DGPTd1kgGr01I3/ZdpxMx+aJH7ABNmTfRein1mIRuPcs34mC1vAxisaCywe4qEk1Hp++qAsH23WzQPm8vb0SaiFUdCpRvC4kKmg5YxfHPV3SKJU7A5Hn4NBRZ7YrzUQmk7xAufmhc6D1P6F0vBX68zzvQ2EdXG4IS81en8aeQ3kJsp+AHfG90K7F/C0HA1KxY3d+2M0m7Ub3PBkam5yePtAJkHZxEke9DTI559eqr3rhSzD83NNhe8kTqOBNrsNTeLU9C6VK8uEQclMwE+q+HEN2lCo4UywDrfKafGljHYxsYPFoKsRPOV5NUqMUkI69KdRc7DgvAzIKsYoVmH7gL X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 93704ef9-e252-4b7a-80e3-08d931c01459 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jun 2021 18:45:29.6021 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: I3uVl1AWjHbuY4uforHC8Tv2Tm1jik27LCnWDoFGyKlq5fmxPgBLQv0RSXS2U1ZTeAPC0OEqEmD5PtCfOeypjaVlM7fku0zK/ZheOTWqT8s= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB4852 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10018 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 malwarescore=0 suspectscore=0 spamscore=0 bulkscore=0 mlxlogscore=999 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2106170117 X-Proofpoint-ORIG-GUID: 4gqGtjdyWebzxqPPj-ahIklorBhLesJ3 X-Proofpoint-GUID: 4gqGtjdyWebzxqPPj-ahIklorBhLesJ3 Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=RQ0ri5VC; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=lW32py9y; spf=none (imf24.hostedemail.com: domain of joao.m.martins@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=joao.m.martins@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Stat-Signature: yxn98r57f8gtwewcfy6k5d84uh8n8jsy X-Rspamd-Queue-Id: CC25AA00026F X-Rspamd-Server: rspam06 X-HE-Tag: 1623955560-392240 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Split the utility function prep_compound_page() into head and tail counterparts, and use them accordingly. This is in preparation for sharing the storage for / deduplicating compound page metadata. Signed-off-by: Joao Martins Acked-by: Mike Kravetz --- mm/page_alloc.c | 32 +++++++++++++++++++++----------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 8836e54721ae..95967ce55829 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -741,24 +741,34 @@ void free_compound_page(struct page *page) free_the_page(page, compound_order(page)); } +static void prep_compound_head(struct page *page, unsigned int order) +{ + set_compound_page_dtor(page, COMPOUND_PAGE_DTOR); + set_compound_order(page, order); + atomic_set(compound_mapcount_ptr(page), -1); + if (hpage_pincount_available(page)) + atomic_set(compound_pincount_ptr(page), 0); +} + +static void prep_compound_tail(struct page *head, int tail_idx) +{ + struct page *p = head + tail_idx; + + set_page_count(p, 0); + p->mapping = TAIL_MAPPING; + set_compound_head(p, head); +} + void prep_compound_page(struct page *page, unsigned int order) { int i; int nr_pages = 1 << order; __SetPageHead(page); - for (i = 1; i < nr_pages; i++) { - struct page *p = page + i; - set_page_count(p, 0); - p->mapping = TAIL_MAPPING; - set_compound_head(p, page); - } + for (i = 1; i < nr_pages; i++) + prep_compound_tail(page, i); - set_compound_page_dtor(page, COMPOUND_PAGE_DTOR); - set_compound_order(page, order); - atomic_set(compound_mapcount_ptr(page), -1); - if (hpage_pincount_available(page)) - atomic_set(compound_pincount_ptr(page), 0); + prep_compound_head(page, order); } #ifdef CONFIG_DEBUG_PAGEALLOC From patchwork Thu Jun 17 18:44:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12329255 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12CAFC2B9F4 for ; Thu, 17 Jun 2021 18:46:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AF59C613E9 for ; Thu, 17 Jun 2021 18:46:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AF59C613E9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 920E76B0073; Thu, 17 Jun 2021 14:46:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8ADF56B0074; Thu, 17 Jun 2021 14:46:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 662876B0075; Thu, 17 Jun 2021 14:46:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0220.hostedemail.com [216.40.44.220]) by kanga.kvack.org (Postfix) with ESMTP id 2ED256B0073 for ; Thu, 17 Jun 2021 14:46:05 -0400 (EDT) Received: from smtpin36.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id B28F9180AE811 for ; Thu, 17 Jun 2021 18:46:04 +0000 (UTC) X-FDA: 78264095448.36.BACAE88 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf25.hostedemail.com (Postfix) with ESMTP id 254726000141 for ; Thu, 17 Jun 2021 18:46:03 +0000 (UTC) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 15HIawXD007626; Thu, 17 Jun 2021 18:45:41 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=VRH7+xHH70/n+QPNEPTu6FTZAS/XG+3Wl5/s+YYdgVU=; b=t8i7X8jo1KSljhOFzQ76xMURwomu5CI/k8jRO6oQNaYxrTtvBPiOBiMDBO3xEC+YEMka kHmyYyrdKkCfjsle+7/APlUc5JkEmd1Rp0tHiHuc+eEMc659pJ+qtlBJ8Od6krUqQF6N AoYAjLzPiivJQlPu7TIfBNiQjMnBY+0Nfk5jf3iHfaaop9Tz2R4MUmmXz6KHt45b+lZs W2m9cLShcMioQeyB/VX6p4qi8ua/2dhXfY+mGt3TeuoszVC/jnqgr7HdsTKpr9/zEyag K8FWrrbiIG4zK2kRsJ4YvY7pQTTs3MNCK6E5X65kQQhUA1TIQttdu6ADfhoN0PVgkvAL Qw== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 397w1y1r64-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:45:41 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 15HIjaO7180356; Thu, 17 Jun 2021 18:45:40 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2174.outbound.protection.outlook.com [104.47.57.174]) by userp3020.oracle.com with ESMTP id 396waxy5wg-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:45:40 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Ws2J1YJGkhWSC0GwxTJl8F6zXctXHSvWtX8QFx7dShGIp6whnXg+eQ0ifcKeyoropHGp8hVdp48Jkom8YkV8wWsmRq9rizLheI1NxpfW7I/d1CGRHr8qD7TqD9ab2XXgxQ6EC59t3f+fJpJbq/SisAmQqiwwIkz4IZAMIP0deRgdaNfU14LuRz0wA8WzYmm1BNvDfb54spp2jDDSVOThX55pQFQoRQ/n9g2NIZO64eB24UHSRkES+uH90Wcesawoz4lWoHmneZxxZ6W9wdDvmE1bqYlQx3avMEbqRHPwDEMQA2WHhe6QjPssNNLFiG9eWi16OblPSK9tLB71SrPVnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VRH7+xHH70/n+QPNEPTu6FTZAS/XG+3Wl5/s+YYdgVU=; b=I4nCPRnU0TYi/CDK3jk1NuQB5VcZs5LvCcu9aj7tEfR4chjWy+ruAuj64L8P3LMAYq6G9qWKR8mGk7MGEPqU0mjqHLUNu5UTR1pOnFutJnpNiuea0rc+XKAzrXOc/rCc0Newmh+IcvyhBPL1vu5KmyxbtlAo9nwrnHkrDBq7lvwaJNhiYplaymQt+dltpJejY0VnpkHh7xq/KgMPRrFhm24NVw9XTN8NKpXg9w9E9lh+m5i0BNp3/2QCmwXsnjzBNXFhBk6qMXhF36r0kVW8sGYaOsdF3mBbvvrb3c4+oSyCpnXYdxQUchGBihXzQFvTiWZYd6Bu/jYlEj9royQfPA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VRH7+xHH70/n+QPNEPTu6FTZAS/XG+3Wl5/s+YYdgVU=; b=eXhOAoq+Vn3DkMMvMEqOMlW3jtsDFDZJ6/Z6FTEDAKfcx80FEMWA/4fLKqicqZHW3E2bcJ024u+UbODPP9CT/yWIC9e5zeR43YTg/smuLAOtHDOan8918BA7jTrIk6Jgklscf2H3+AEHmfqt8dsWqCHvH60eX1S+/2l+WWmZLX8= Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BLAPR10MB4852.namprd10.prod.outlook.com (2603:10b6:208:30f::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.15; Thu, 17 Jun 2021 18:45:32 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582%7]) with mapi id 15.20.4242.021; Thu, 17 Jun 2021 18:45:32 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v2 03/14] mm/page_alloc: refactor memmap_init_zone_device() page init Date: Thu, 17 Jun 2021 19:44:56 +0100 Message-Id: <20210617184507.3662-4-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210617184507.3662-1-joao.m.martins@oracle.com> References: <20210617184507.3662-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4242.16 via Frontend Transport; Thu, 17 Jun 2021 18:45:29 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c680f4e4-fc8e-42e3-7c0f-08d931c015df X-MS-TrafficTypeDiagnostic: BLAPR10MB4852: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: UuJgS1vBlE8aWss5DWZZkbEEhYPbBJM6eIZtIUl5UWeMoTQIOSwfaQ18yqKbO3zlvzf1ALADmswGTAEb7hBtioWHRpmYiBXbhqhi+pe8/Y+F/o2V7shuWTYwvDsotMD95ZE5C0BRQbu8jNTFJYIe7t+OgV9pMGhLnUqLMBx+7ZeBIyWnzbj+UDQb654AbCu2AppE7bxaOSQyVcFA83vh/89JjUHwa1MeS+tcLHsY58m2m/hS1hvhSyIRCfRBfiGctLvV80ITC+QdVaCH/QBj+InwJ0D/zWfpe0TFBLWDS/i+dzpsh+qqQ1q3g+ho1xfSfp+2v7tbxq66lc1fH0CFlNjjNkNbA7ccKzVyn+u0UeYq/TtzaUM0u9l98oPplb69TN0gkOaJ/dofgmXSD0QfOwz14czusppeZ46p0KNoCgVoNDn6/JQwzWZ4hJPf58GXjVhILOvnzMQE7j3Fl0TmQqbGcaOHPnV3Qcrovwldkj1Rve13U/URpVcUpxeEqhnSurVyg2SI3WLDgZx6gBcJeeBd0UowkrxQp4GvBLfd2uGvF9Ufm8zuL2cbrbHMM2vYvaNRo9Dh4ZU7ea2ij5wIoNRrPh+LbYS3UnOKsLL+1mybz+TEu9O1qoQCrMD137Wl8zLg7MnufRHULxIurNi4Bw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(396003)(376002)(136003)(366004)(346002)(83380400001)(6666004)(38350700002)(4326008)(103116003)(7696005)(5660300002)(186003)(16526019)(26005)(54906003)(478600001)(38100700002)(316002)(107886003)(1076003)(52116002)(8936002)(66476007)(66946007)(66556008)(36756003)(6486002)(6916009)(2906002)(86362001)(7416002)(8676002)(2616005)(956004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: QzOp0MZwztt1B9ZsT1H8jJBimxSdcKyQRmilaEaszeG47hXbOeCpxSkDSE7yly89E2jFKYoQ6IUK20IoVhypAfkKUohTI7t+m7ejsZrJPgjwMi7v6dzv2mnkcTzTCgtkjqaLslnytkK3KtY8MLQ9e9n/7xW61QoUlyR9F4t7IwVs1OUY0HufF56gRbxaHQsUvEK9VYSmxqSOCbXTW7DVp3i1XCWjmK2URf/dsPEkJNJKPhJm4PfNq0VOkgwYdHx/sjuD8li6XdDi2+BwRlACbxQ66Wfa9H4HLkZSabkPrRHl2JiV2l/1XAFsdhHkTJLFePS93RW127zQu4yrI6v9ygRW3ZyzE1O+KzNB6PhHmBHMalFo/6a+g5OeMwR5j4YbS1+9cWxmOm0kzNyyJxu1EWjL/YDVBXIGp1lptm8LQOpa50sih6/b2wr7hkTboMAoIF/Nxm+jHfuz5sheW0BKoICtRPECoeN4P0ZHgnh+JrDOpq4Hdt15uYlvOHWPjrSwXdAJjo4nosBJ8W17JUd39UOGpZ0c5FJIc/fVN4pKEF6JmVrxbZykVZTDR1e+UGhgZd7ikOuCf6p0VMP1+SQOFY2tPNy0mnvLVzScJXJ/7H+PsX4ZmsGHnNPQ2jXetqCofPgB8V1ovRlJT793llBU4YmrOaHRI5pEvHejx5/C7mn0wcEhsS2tPNpjtIl5Mpomup2SqubYNsEobZrlxvwnEVMNOgOk0F+87XiHPlicsWY4uglhaC2B735TWgigQTy9wxgFiMwgQNeCzGYHFtM6F+k35OXAg8q90pmj8fKq1AJp9NTLishJtXjAbIHmnqCWJLp29LVKlhq7aRHn/qAw/93QYGgGRtxx2KwzOpACcx9G5y9MKnIojOGtClw9UDZXx6xSjVc0K66Bne/frnL9nCu39fkP4yUWHyWKVhhBlBsFA766XSktdIXai2dq0fRX2Vt6V2pzMMUOiQQeyMYmIBweeV5V1aGJJpjO6SGEEjAQE+M9RuX9xxKxS5q2ysoiZDzAHR8TIT7jWcaXuavVVknA8gAh118B9nGI4sZXJS29UIaIcgHO/eGra+bRytKCcMF/1J9289JV0Eszt2ABz94VD6onc9/QiLBXRJTTPluJc1AZZ3hdin2L2tpIER0iYCc3FC/5+hqqW0QaAZ31B+jqkVL/mKW8epGWIFG17/48rZOKlPGrptulEpNnqHHirRlrNlcoLzOPGw/b42y/dqo2q44HnItitnXdcYjDjlzr9XGboosN0Wymgy6xm00HhH4nb1p00nvHntD1p1xYByk+BZoS/XILOF/OpDvLVz/HwUq2p6WJLvMqOu/szRrb X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: c680f4e4-fc8e-42e3-7c0f-08d931c015df X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jun 2021 18:45:32.1578 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ir2Ezy5L0CawABvTIqf6M5YBzVrUSqfiwICgDFBFVF6W7ws8m9mRfxdlzSL9Dkl5WG3WJ56f0kzxvJckm5xUKLwZ2I4RHUDL+uGucJZ4gF8= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB4852 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10018 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 malwarescore=0 suspectscore=0 spamscore=0 bulkscore=0 mlxlogscore=999 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2106170117 X-Proofpoint-ORIG-GUID: UREPtd2OYLFeuVk7yBYV7mQ3-k4uXE-Q X-Proofpoint-GUID: UREPtd2OYLFeuVk7yBYV7mQ3-k4uXE-Q Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=t8i7X8jo; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=eXhOAoq+; spf=none (imf25.hostedemail.com: domain of joao.m.martins@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=joao.m.martins@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Stat-Signature: rwnhfc7urgzp1aag1e8wk4oib1b4zdyj X-Rspamd-Queue-Id: 254726000141 X-Rspamd-Server: rspam06 X-HE-Tag: 1623955563-445581 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move struct page init to an helper function __init_zone_device_page(). This is in preparation for sharing the storage for / deduplicating compound page metadata. Signed-off-by: Joao Martins --- mm/page_alloc.c | 74 +++++++++++++++++++++++++++---------------------- 1 file changed, 41 insertions(+), 33 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 95967ce55829..1264c025becb 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6565,6 +6565,46 @@ void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone } #ifdef CONFIG_ZONE_DEVICE +static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, + unsigned long zone_idx, int nid, + struct dev_pagemap *pgmap) +{ + + __init_single_page(page, pfn, zone_idx, nid); + + /* + * Mark page reserved as it will need to wait for onlining + * phase for it to be fully associated with a zone. + * + * We can use the non-atomic __set_bit operation for setting + * the flag as we are still initializing the pages. + */ + __SetPageReserved(page); + + /* + * ZONE_DEVICE pages union ->lru with a ->pgmap back pointer + * and zone_device_data. It is a bug if a ZONE_DEVICE page is + * ever freed or placed on a driver-private list. + */ + page->pgmap = pgmap; + page->zone_device_data = NULL; + + /* + * Mark the block movable so that blocks are reserved for + * movable at startup. This will force kernel allocations + * to reserve their blocks rather than leaking throughout + * the address space during boot when many long-lived + * kernel allocations are made. + * + * Please note that MEMINIT_HOTPLUG path doesn't clear memmap + * because this is done early in section_activate() + */ + if (IS_ALIGNED(pfn, pageblock_nr_pages)) { + set_pageblock_migratetype(page, MIGRATE_MOVABLE); + cond_resched(); + } +} + void __ref memmap_init_zone_device(struct zone *zone, unsigned long start_pfn, unsigned long nr_pages, @@ -6593,39 +6633,7 @@ void __ref memmap_init_zone_device(struct zone *zone, for (pfn = start_pfn; pfn < end_pfn; pfn++) { struct page *page = pfn_to_page(pfn); - __init_single_page(page, pfn, zone_idx, nid); - - /* - * Mark page reserved as it will need to wait for onlining - * phase for it to be fully associated with a zone. - * - * We can use the non-atomic __set_bit operation for setting - * the flag as we are still initializing the pages. - */ - __SetPageReserved(page); - - /* - * ZONE_DEVICE pages union ->lru with a ->pgmap back pointer - * and zone_device_data. It is a bug if a ZONE_DEVICE page is - * ever freed or placed on a driver-private list. - */ - page->pgmap = pgmap; - page->zone_device_data = NULL; - - /* - * Mark the block movable so that blocks are reserved for - * movable at startup. This will force kernel allocations - * to reserve their blocks rather than leaking throughout - * the address space during boot when many long-lived - * kernel allocations are made. - * - * Please note that MEMINIT_HOTPLUG path doesn't clear memmap - * because this is done early in section_activate() - */ - if (IS_ALIGNED(pfn, pageblock_nr_pages)) { - set_pageblock_migratetype(page, MIGRATE_MOVABLE); - cond_resched(); - } + __init_zone_device_page(page, pfn, zone_idx, nid, pgmap); } pr_info("%s initialised %lu pages in %ums\n", __func__, From patchwork Thu Jun 17 18:44:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12329249 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C110FC2B9F4 for ; Thu, 17 Jun 2021 18:46:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 20465613ED for ; Thu, 17 Jun 2021 18:46:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 20465613ED Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 947CE6B0070; Thu, 17 Jun 2021 14:46:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 91DE26B0071; Thu, 17 Jun 2021 14:46:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 749D86B0072; Thu, 17 Jun 2021 14:46:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0190.hostedemail.com [216.40.44.190]) by kanga.kvack.org (Postfix) with ESMTP id 406446B0070 for ; Thu, 17 Jun 2021 14:46:00 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id D7BD68783 for ; Thu, 17 Jun 2021 18:45:59 +0000 (UTC) X-FDA: 78264095238.06.32770BB Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf01.hostedemail.com (Postfix) with ESMTP id 4C2E65001708 for ; Thu, 17 Jun 2021 18:45:59 +0000 (UTC) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 15HIawXE007626; Thu, 17 Jun 2021 18:45:42 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=1jBM/jw+0Rr2nHTVvKm539T8Wk4XHEoGywUC7BkqfAs=; b=z9gq2l1vT0rqQL2UHOZtk/ejcLjlDSFXV6ybsXwOBlWYzicnV8nCEcgYatA0R9Pnznp2 NZXXsEpH1Wn7JzWtUZ/8GJqe6uWN4JJv86hddlmkG/wC1H96OCtl3hAiqhvihRX7yYs1 AkqZa5Pfdm42jZODSZRPNDJ70bYewdPSDMR+PFkLg6lpyNEzzDyUTPFuAPg2TCTxkHDd nX6bbcrDtvrTE+LzQwPRr1zSDuskuM05u5ylukuY9d7J1EDw+FN/SNusqF0pOW45TREZ u6c59ulzis49AUtQLhGz/3qmUVn/XL9mFeRRlKcU5W5rcK2v2CD0OkjT9zGZwiVBvxGL Zw== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 397w1y1r65-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:45:42 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 15HIjaO8180356; Thu, 17 Jun 2021 18:45:41 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2174.outbound.protection.outlook.com [104.47.57.174]) by userp3020.oracle.com with ESMTP id 396waxy5wg-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:45:41 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WkMSqOGJkZIhLoRydYS6EUeapLBMWYuf8XvNiTcGnAewLeI0N0A5bQqLUOE3kpeqHW1KAOXwH8oz5NsvO5Dnxaf9pkybqWIDLWF3U3p1dr1b3F46rhC7m7qzlp/ghyyUYIg0t0nKMx6QPfWEAgECHsZMHjFpH4nBP+mQ6h/h0yKk3hDedcycwp44EWfVBHAa8iaUfixmamN06yHhGxn2TWiUPkKwLHIi4NrkAzo4M/8XMIH+XRhq5VtaCliRPgjYaNM5lmpDJZdbYuDPolAqmJQQ0RuA/V6ZxsPOyo0lq/cAlFCWvM21NM5/cXRKT67AmGadUsLQxXm5RJYpqeH0ZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1jBM/jw+0Rr2nHTVvKm539T8Wk4XHEoGywUC7BkqfAs=; b=JyWrpiHIRKK62Kk8O1mWzjxSa7hH+yeeOKcmXXfh9/mgf1RijV5nrURpoDzub/Lk5lVM4ehVtjDNK39BjQNtuTnv5W2BwegaV9F94IpNJT8q1HH3VCxt3jBvoW2CwZNbNsTAavNemrdJPyFlFx5gzZWbrKXLPbl3fb3AeIzg38mxeW1vTJyz8fhN149NDPxgTa74TGGJUGd1oTv/7zFf+MVSlSE09n03snGDtw3ip0qk8n4SfNBffWgXmlnknT//EKuxKxB9T57m0vO4Wf2InSSqMgze3FhXw6CL/phHsLrdR70mDvbYX6ATjO1hSDLgXIdfNK4ANuwaa9353YR5kg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1jBM/jw+0Rr2nHTVvKm539T8Wk4XHEoGywUC7BkqfAs=; b=SnawXwGF0YyqGRh3e76DFEQ43A4KyqFOpcaxyviKfoeq2T1Nba+8YHl63XemGKNAaPdaOySVLlijDeQFm2rVy6VWanNcGYVV+1XJ+kfUaJh13GZB+GpNjADJrFss9LQVpVLgDS0ZBoqlcyJYfHNVFTcZJD4bYfK/wJOSH7Q3xCk= Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BLAPR10MB4852.namprd10.prod.outlook.com (2603:10b6:208:30f::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.15; Thu, 17 Jun 2021 18:45:34 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582%7]) with mapi id 15.20.4242.021; Thu, 17 Jun 2021 18:45:34 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v2 04/14] mm/memremap: add ZONE_DEVICE support for compound pages Date: Thu, 17 Jun 2021 19:44:57 +0100 Message-Id: <20210617184507.3662-5-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210617184507.3662-1-joao.m.martins@oracle.com> References: <20210617184507.3662-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4242.16 via Frontend Transport; Thu, 17 Jun 2021 18:45:32 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 009ce406-7c81-4adf-32d3-08d931c0175c X-MS-TrafficTypeDiagnostic: BLAPR10MB4852: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5236; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: AYWoufIoZLtWSyCFmy/WDL3l80/5cX+5Omynfms2N2uebTZS91l5ZZxBdDD+1fmphmvR+37I/qbaspgJwXJYPSBeqlu6OVS4FvkXKzzLsyexk83VD+7Pkrv1VidyW9d1tur8UNhUMlU9yqJPugaZvtbkWSbRqCA+47aGZ5rwUkZTbwdfgVfQcR9BU3EDB+XMyh2wjVcPXe/sWFxgAKHarkjqDETahIR3D5JGqG/1dXQSij8VDaYYwy84BsxfbM61gWmmrFxdoTwpGyNW1Im3Ya6F3uN5MdGQrw91tK4RHfomRQW9+6EcFHplWKIH9p3owFimhpTmrHjpGpiN+FOzMZLxBbtHVU4ZmTpEaoX3qVwqHeCqIwLGJnJtxYQOKpSrP5TtZUGGs7ftMLiTFB29/NS85ZdZrHXkeRITrs0ML6nnqRmlJwGMH4Gp1fICB0KgHo165F4SEIEiL2qrJvGfNogO11VhnbVqNjFwnwSjCozWgkBlSG3CTBHs1HdekqB9jU5G+hYIh4JSzkfVGj9c66xxzAWONFrVlRGnXbIfUxoJGmWa0xuPbCmyEhI2Dfkdd07v9Gj45/bE0BCsNTtsqg0QQ1D78goPRIoeO7gcInwMVQV/O00eyGV4/VKl7NdTbkuz6q7gx9XRZ95Tuejw2B6+QQ55VXPYecxSF5UhubA= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(396003)(376002)(136003)(366004)(346002)(83380400001)(6666004)(38350700002)(4326008)(103116003)(7696005)(5660300002)(186003)(16526019)(26005)(54906003)(478600001)(38100700002)(316002)(107886003)(1076003)(52116002)(8936002)(66476007)(66946007)(66556008)(36756003)(6486002)(6916009)(2906002)(86362001)(7416002)(8676002)(2616005)(956004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: uK/gsdVILvwelorGyY1z8M0fM72vqgCqFK7OWLVkMS4lJx+bbLlgHRUDEZvcvg7ax/FlGZZAAAqgbLmPIuM1vcs62rSLvhBPVMedCcmmlKNxmFaHxuwgeHnYoaD2gUNllI97GOsjyE4AN+kjXCKPBi7iBPqlIyU8Q5O77O9RqqtQ36lGWPJUCCCDh/w/jKh5MRMkj3LUvQoX7EVLwpLx2+3Rm0+1aZsUjZoSvUD0e2XR7zIcTQI07ny5Ti1XTUYsv4MYxV6Np/9pDnZ0OuPkz8zpNnQAbTXGG1zRXdjYua9+xOWeX+8xiDjB6xbTYAAAdk5f0pF8CjSJL0FFG1KGOy7p5r0FMl+1kvs+7helXSPKH9KeBGnCHHx1PfjwJMMD+TJT+KeS9tv4QzyNqSYMC/SlMoXUKbkzmexLWyVlAVi6FvzWbYYEpq3A80gUXbaDxjBivynvz+Jar/6GNnun/pCjWmNyOQFCOveZaoHhRcQOlST7zNU7j1N+5lNQq565NwHG4QipkxjoVkMPvXfeIeUVSMjeGZ7/dcp7G0j+oBkoYVeVKtovp637siYakxM0NBjGazA0jVJVazxBwjbx6p/RraV5/Sq407unFciVu06WyyBbk2j4zUNQoshTBIzL3eYmEx0vkgMAsqr18NeZpz2iUp9s1EgA7kYemNAV9xjdBnaXGNO+S5ewkjMC3SZRKDApHuP5P2eZbP8hRa+6O7HdTj+5XDupqbLbsZZwkQzyTxKWdwXzXZwPcKjOn4hNZ3a/jJOznyyicWpvBkQS2+6isH0w4VRAs7pLSpR4OkNd3YajwEmkP7sN/O13zhUhML1aGEfLCAcXjT8zpdoEY7d0A5jtdTS87QjgeL7iQCkwB9VBmMh6ndE7WeZbUuQ6UhyNmxkcazUU2kgxPM6InJo2aLt+lvoNSIg7Nwa3z/6vAp3R/HFyr18tjdN9y1NcCxDMHR5Nz5doMtUmlZ05lNsT+hZx9ktsbak2M5FC89NFMPpomeUeQtegJe9tmAKJvMmwr8w2cOE9+rVVGJ/8iNZT8dcbzW4cMsfSh+CXV9ctJwiIgZpCHFViFyYnoQhOWE7ebcMetf4IfBBxJc9J+07sUxBZfUwD6SBUhQVrfi6Exz5oFLtazXc+x26keFOCWahTbIjboeFQxffUdkJfoyiEmZi361Fy+oZspak/WzsQ0eh41kowZx5leWL9OXkl4bAwHgpslHYTpeRyun1NRUWw2zpzblXMO0BeN8oR0jzQ7Ge5MwMsinY70pLbYUqoca/UsDEjaoVvYPzKrALPOWpYLJeUQW+kLUmLHjvuZJEcxjgLD+4iHZ3uk4532av8 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 009ce406-7c81-4adf-32d3-08d931c0175c X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jun 2021 18:45:34.6688 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: rm14ccEkFivxFWTmz+g6uZQ5G2d9+0B/e0CF00RexL8/BYdu59iUeTezXG5BlmgOwkYJJSZ53vCz8c/whIhU+Cw8Q8UZ5NON7/7Ws6oo6+Y= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB4852 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10018 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 malwarescore=0 suspectscore=0 spamscore=0 bulkscore=0 mlxlogscore=999 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2106170117 X-Proofpoint-ORIG-GUID: gi4DWARM46z11_0f5r4zaVU21qvsQrG4 X-Proofpoint-GUID: gi4DWARM46z11_0f5r4zaVU21qvsQrG4 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 4C2E65001708 X-Stat-Signature: p3xh65mab5ek9bbcx4spwcjphu3g3wz1 Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=z9gq2l1v; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=SnawXwGF; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf01.hostedemail.com: domain of joao.m.martins@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=joao.m.martins@oracle.com X-HE-Tag: 1623955559-265755 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a new align property for struct dev_pagemap which specifies that a pagemap is composed of a set of compound pages of size @align, instead of base pages. When a compound page geometry is requested, all but the first page are initialised as tail pages instead of order-0 pages. For certain ZONE_DEVICE users like device-dax which have a fixed page size, this creates an opportunity to optimize GUP and GUP-fast walkers, treating it the same way as THP or hugetlb pages. Signed-off-by: Joao Martins --- include/linux/memremap.h | 17 +++++++++++++++++ mm/memremap.c | 8 ++++++-- mm/page_alloc.c | 34 +++++++++++++++++++++++++++++++++- 3 files changed, 56 insertions(+), 3 deletions(-) diff --git a/include/linux/memremap.h b/include/linux/memremap.h index 119f130ef8f1..e5ab6d4525c1 100644 --- a/include/linux/memremap.h +++ b/include/linux/memremap.h @@ -99,6 +99,10 @@ struct dev_pagemap_ops { * @done: completion for @internal_ref * @type: memory type: see MEMORY_* in memory_hotplug.h * @flags: PGMAP_* flags to specify defailed behavior + * @geometry: structural definition of how the vmemmap metadata is populated. + * A zero or PAGE_SIZE defaults to using base pages as the memmap metadata + * representation. A bigger value but also multiple of PAGE_SIZE will set + * up compound struct pages representative of the requested geometry size. * @ops: method table * @owner: an opaque pointer identifying the entity that manages this * instance. Used by various helpers to make sure that no @@ -114,6 +118,7 @@ struct dev_pagemap { struct completion done; enum memory_type type; unsigned int flags; + unsigned long geometry; const struct dev_pagemap_ops *ops; void *owner; int nr_range; @@ -130,6 +135,18 @@ static inline struct vmem_altmap *pgmap_altmap(struct dev_pagemap *pgmap) return NULL; } +static inline unsigned long pgmap_geometry(struct dev_pagemap *pgmap) +{ + if (!pgmap || !pgmap->geometry) + return PAGE_SIZE; + return pgmap->geometry; +} + +static inline unsigned long pgmap_pfn_geometry(struct dev_pagemap *pgmap) +{ + return PHYS_PFN(pgmap_geometry(pgmap)); +} + #ifdef CONFIG_ZONE_DEVICE bool pfn_zone_device_reserved(unsigned long pfn); void *memremap_pages(struct dev_pagemap *pgmap, int nid); diff --git a/mm/memremap.c b/mm/memremap.c index 805d761740c4..ffcb924eb6a5 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -318,8 +318,12 @@ static int pagemap_range(struct dev_pagemap *pgmap, struct mhp_params *params, memmap_init_zone_device(&NODE_DATA(nid)->node_zones[ZONE_DEVICE], PHYS_PFN(range->start), PHYS_PFN(range_len(range)), pgmap); - percpu_ref_get_many(pgmap->ref, pfn_end(pgmap, range_id) - - pfn_first(pgmap, range_id)); + if (pgmap_geometry(pgmap) > PAGE_SIZE) + percpu_ref_get_many(pgmap->ref, (pfn_end(pgmap, range_id) + - pfn_first(pgmap, range_id)) / pgmap_pfn_geometry(pgmap)); + else + percpu_ref_get_many(pgmap->ref, pfn_end(pgmap, range_id) + - pfn_first(pgmap, range_id)); return 0; err_add_memory: diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 1264c025becb..42611c206d0a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6605,6 +6605,31 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, } } +static void __ref memmap_init_compound(struct page *page, unsigned long pfn, + unsigned long zone_idx, int nid, + struct dev_pagemap *pgmap, + unsigned long nr_pages) +{ + unsigned int order_align = order_base_2(nr_pages); + unsigned long i; + + __SetPageHead(page); + + for (i = 1; i < nr_pages; i++) { + __init_zone_device_page(page + i, pfn + i, zone_idx, + nid, pgmap); + prep_compound_tail(page, i); + + /* + * The first and second tail pages need to + * initialized first, hence the head page is + * prepared last. + */ + if (i == 2) + prep_compound_head(page, order_align); + } +} + void __ref memmap_init_zone_device(struct zone *zone, unsigned long start_pfn, unsigned long nr_pages, @@ -6613,6 +6638,7 @@ void __ref memmap_init_zone_device(struct zone *zone, unsigned long pfn, end_pfn = start_pfn + nr_pages; struct pglist_data *pgdat = zone->zone_pgdat; struct vmem_altmap *altmap = pgmap_altmap(pgmap); + unsigned int pfns_per_compound = pgmap_pfn_geometry(pgmap); unsigned long zone_idx = zone_idx(zone); unsigned long start = jiffies; int nid = pgdat->node_id; @@ -6630,10 +6656,16 @@ void __ref memmap_init_zone_device(struct zone *zone, nr_pages = end_pfn - start_pfn; } - for (pfn = start_pfn; pfn < end_pfn; pfn++) { + for (pfn = start_pfn; pfn < end_pfn; pfn += pfns_per_compound) { struct page *page = pfn_to_page(pfn); __init_zone_device_page(page, pfn, zone_idx, nid, pgmap); + + if (pfns_per_compound == 1) + continue; + + memmap_init_compound(page, pfn, zone_idx, nid, pgmap, + pfns_per_compound); } pr_info("%s initialised %lu pages in %ums\n", __func__, From patchwork Thu Jun 17 18:44:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12329361 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9B0FC2B9F4 for ; Thu, 17 Jun 2021 19:29:56 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 03473610CD for ; Thu, 17 Jun 2021 19:29:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 03473610CD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8A0956B0070; Thu, 17 Jun 2021 15:29:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 87F746B0071; Thu, 17 Jun 2021 15:29:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6CA9F6B0072; Thu, 17 Jun 2021 15:29:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0240.hostedemail.com [216.40.44.240]) by kanga.kvack.org (Postfix) with ESMTP id 30EEA6B0070 for ; Thu, 17 Jun 2021 15:29:55 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id BBADA815E9BD for ; Thu, 17 Jun 2021 19:29:54 +0000 (UTC) X-FDA: 78264205908.28.DFDDCB1 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf18.hostedemail.com (Postfix) with ESMTP id 40F072001097 for ; Thu, 17 Jun 2021 19:29:54 +0000 (UTC) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 15HIZakw004683; Thu, 17 Jun 2021 18:45:42 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=pKYL0aGuXtxzTFSh6/IVah9AqDIYJZG0fVE5+ubXmuA=; b=Lsx5652pK8lSbUKM9GH95tZYlsY5FJx8qT6j0ZZsjFkCVIepVFBvxCkA3Gbc0Whlo3yq AhRNNOPp1fNybOnHljnxvEpeKVVJoGprwQHh/X8CSNFPPjdIzQ4Ad5OslOJFvWqhQsy7 qzQ9qswc0sQyr9FnkJhXuiumV0Ab/CHV1SBpaMIuqPw7bm/weX0rNLjzWDlac6UXVAuT coPwR0+eGJNrFcs+w85xSW+hj8hSkeXkQTkE2ZK342+lBINFZEvLntd49qvOIeXeKqsJ XPMq06j/pefXA9Ak8YPgj6wz1jHgYFxMT4zaOSEUR3LQJ+N4hsdq1ebUbCLD2MBavlyt JQ== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by mx0b-00069f02.pphosted.com with ESMTP id 39893qrcer-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:45:41 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 15HIjAlI109296; Thu, 17 Jun 2021 18:45:40 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2174.outbound.protection.outlook.com [104.47.55.174]) by aserp3020.oracle.com with ESMTP id 396waw6w0h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:45:40 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nuPkemGXjQlW3FrWbmV54px1S/+awdkWr+cHnoBVYs73BFCF5npxxHAdjZCrd5+8w9boDuQLbNsVRWkcapoIYgd8j20rZtRg+tOEXJDB9dpQUBPXzDGRtItQo7q6oTpRXiT0Wk5e5GIHsKbvVVz3gz4Mtd861rjBv4BE2FUM+ZXfFJzIAlK+OqL1D4HrmHvqzVd7C4F6X5ZBFt5Qq+NkEDLEtI1rW/mHNuu2R31Nv9SBow7JvyYML7C9wRmce5zMsKOJ9ywLDR062CwY7jBzjd8tU8MOzCNa/dqknOYxngWDrNGFgX2ikKAwskMI0fHs5HGxvRE8L23sgSNdeP4K6A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pKYL0aGuXtxzTFSh6/IVah9AqDIYJZG0fVE5+ubXmuA=; b=BzdJcbSQyYxd3spmcrLGr2hol28sZxb74kTUCQofSzolIEZBS0WP6EnzhTgTwjrJll0IDqHyGsh+VB+cCs26Exd0ISTbQZ8YI6U0zAftfNu5x5RjBA/y3XjmnbYyzEPU7LzbirYGu8hWufuyPZpAyI9jihk9QEMT3GrRkTIlwzFGaVw03nMkJMDwp+pW8I/5YlpdyG/kuIz3Dcob/3YVX3zqtmSq2b0kaacLUUPQFmua5tszU7U9jlgJDhuhgpelJmbgAhjn71XsCuojpE49rGorWkmOlVkUIzws6U45YLq/BNe0NaVtaNQDv5CEOC/of7u8ka5fkhx63l0ShJT7uw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pKYL0aGuXtxzTFSh6/IVah9AqDIYJZG0fVE5+ubXmuA=; b=PGHeh6ybMySdWQC32ctd/AbUtEja4sGdm5uKStg1z1En9+n2osZjQhK6RmK2YXfUxrrpiM0UwHIL0vod55TuPCX0h//jkYLhaspw+2X8lnj75CM4e+9XD8DOvV4SPLoCBj0hY8clF6GcOHmsbT7FL3bEIc6hzI32NLzDnout6aE= Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by MN2PR10MB3983.namprd10.prod.outlook.com (2603:10b6:208:1bd::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.16; Thu, 17 Jun 2021 18:45:38 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582%7]) with mapi id 15.20.4242.021; Thu, 17 Jun 2021 18:45:38 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v2 05/14] mm/sparse-vmemmap: add a pgmap argument to section activation Date: Thu, 17 Jun 2021 19:44:58 +0100 Message-Id: <20210617184507.3662-6-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210617184507.3662-1-joao.m.martins@oracle.com> References: <20210617184507.3662-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4242.16 via Frontend Transport; Thu, 17 Jun 2021 18:45:34 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8f4b0ca2-9c6a-44c5-5be2-08d931c01947 X-MS-TrafficTypeDiagnostic: MN2PR10MB3983: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3513; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: LeQvxhNsLSfo1v7NmhXXKNibWeYxEQD3jaXYbFMT/xXGTZKkSmywYFePSUc+uxQ93exj+5NMOA09V2X8n9QQ8cio4snRuRxYJ7V7Z7dGPo8S9gRYOH1CQc6oaN5ZXnzQnv4aL/h+1hJH+H+TKvXhwEzQdSmAQHJotC0e6my7osYZ+fcQ+yqQRAd6Ym/VZCszKyTEy2q88SASAtFiIuZH1se1MvXtyTOYPW2Uo8rfJTF4lruZOxPpdUFwFnXVSKsBQgvPDb61b9xV+ZIuT4ayBSFrNKuKVBeEy9nC/UVHoCCUv+NZgvpNyuH6cDvn0cB06X9ojxoLwGXtyljXe+gk5LQXLFOL8voELFo4pg5X8uXwzfe+R2lTJsw+EtIKNrq9sFvcOwp0Q9/BaPUqsTzJKdexp5eaQgYKjP5GzA/E9jYy0Aim0gUoNW6cWmZHPET6OZ1UA6Ry8JqN8PSACTfr7omS+jyhDcgI4at80VUVeYxbyFEyZ/bKffapvwQXhJsVYLFGESm2nNEpZuOmquoxpEtAi5FWuKYNwWzGvpl7NH7qq0RJLpBkrU6sKX8nkeOqVqzvi4U7NUxA0urcFFlKH3HOxV04/IOpREjJhjHlnt1WLDRtPYNrH0ZzqE4ax02M/GZaSjSsI2NSXnjUsYlZ3tPhGNa/rO9M0NzdlLyGUq2jdYAHyz3R1nNApRJIV5d0lMCJ1VXq+NpS2YZDx4QdAys9MF9Ng8yYcdAH1ENYukph750Nhlq3OTLh41Grx8MHrHVkDgfnJELkFqlCGuqNYQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(396003)(39860400002)(136003)(366004)(346002)(376002)(26005)(36756003)(16526019)(966005)(86362001)(7696005)(52116002)(4326008)(38350700002)(7416002)(1076003)(83380400001)(186003)(38100700002)(8936002)(54906003)(478600001)(2616005)(316002)(66476007)(5660300002)(6916009)(66556008)(66946007)(956004)(6486002)(6666004)(103116003)(107886003)(8676002)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: u05gB7T281fATkcKe+oXb8q968r9HsvuL0TuqmubLvhgc5EzZhoV1LXPI4trO4citw57KN01Gp98Tm6UHlA+9biGgOV9qzkjFChGAOclIVxBH/NHYxjNvS/1mfq0zmVAcxTubr8CshIzPU/SnupOsIesGeCIkZoQpp4b2x2AZMtlDoRd1JC3jb7x9/jjvxy/RJcdORafKF1PQvPFnTMNKnQ00ZcTgJobhUL9MwikDTZji4eagwFlZDSQnUq8fCGaUZaTQsn3xbDW0M2pnyj3gYZxCk4J5F9DrA8p1KGfCCyfe7ZzDL8FT/5nKyZhrwH2glzdwqqTgEV7B+QylsJ8wp9ZweSKVN6U1rvX1PKmrMTREp3h7xH+/NTR4ym/LDCiL7zlP1OM/eKI/1YefdOSYaez6ywpWal+M71nZTqtnVPO5wSeg6c6JK3U7dY+z4I5kn/Dbdd9j2E0HAVFr+glsD3A3Jk7nMtj0RDF5c/eqIF6YKbi2OpChWN12WXC8o2W4RCGC//atfq5zgYm3ByG3awl+nbujHyd8y6OLDppVNYQRcFaFuKEOSWU9mpMUdgKNm03neI0M8TRLvKB8KWO3WlfLjyu17G2UlKbjBeLlkQLAT/wOvTw2KAZFRJBmbOn0b3zcl46FSVgk1ujonPnhoQHOZUu5GMWH233k3DTF4fVSo8Et+YqfsaX9tji1ShcDpHjxcG/RqMbyeFYHjKaZOkza2Bfgh5GVqbtkioqNKQzoLKI0FbAnPki3v3y71CV8B8YcWhMjSd8CpeT4Y3a/CDbx8yKbsWLWdEZrzXHn20/ZMh8SUdhEx+wUu+1dWj5bvMMvL0Ldmcb4oG6d5qo5RVbudQZZTYFR0qFabFFvFPcwiOuYzolwh8DhMj/eteE64OPFPePL7nWAtAMbKZSvhKrmusa3KpnFzgg+5RRWdyO7z9e+emVfoSFKfx1XusDjsKPEO8xlklUrhP4knge2U54yQYbdFK0Q7x1S+vViANj4c92OWXiGFVKKtYjplPbgTHQenoxYyEQZoOfzilPmg8sfFrvhNktvuPsbx1bWuEzOdSqxO0FQcCO305UZ3qjbSatEz0sR757PebJ9Su1fezN0vdG5pmXNqpaUw/8O06gIFJodWpfUZbzlzi3DLO0wS5vRg1fPJ+GcLeoD5ltOv90Fq70XQq/zjNz2nWiZ0yAnGy7Wga/TzqiC5w+QIJH3Z4L/7D9yJUtXJEBaZcpXenKJO7YQmI+5mSlHT0cGTtxbDNLvXlZSZ9tKu9JYJJ9q1yJpC6XSXzDJDGrXagPTOVBcXQF6/IZK924hCfctcO1vHpLmohB1ruWm92QWLP8 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8f4b0ca2-9c6a-44c5-5be2-08d931c01947 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jun 2021 18:45:37.8797 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: z6209z0NfiuDeaCdUVktU2Qbpnq7ZkYEqwVraQ7eFDcDoAebezk1meGawMKrTLsuWufEE7HHEoFbGWbjgq46EJxPXgEBtHjXpwdrmHpwaKM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR10MB3983 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10018 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0 malwarescore=0 phishscore=0 spamscore=0 bulkscore=0 mlxscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2106170117 X-Proofpoint-ORIG-GUID: 9Xtekw7-PNB0vgxwDKWZ1Pp_2UGHcEac X-Proofpoint-GUID: 9Xtekw7-PNB0vgxwDKWZ1Pp_2UGHcEac Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=Lsx5652p; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=PGHeh6yb; spf=none (imf18.hostedemail.com: domain of joao.m.martins@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=joao.m.martins@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 40F072001097 X-Stat-Signature: exoz1rz9y3ri7jtjsnaz166ph5kbd56q X-HE-Tag: 1623958194-326280 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In support of using compound pages for devmap mappings, plumb the pgmap down to the vmemmap_populate implementation. Note that while altmap is retrievable from pgmap the memory hotplug code passes altmap without pgmap[*], so both need to be independently plumbed. So in addition to @altmap, pass @pgmap to sparse section populate functions namely: sparse_add_section section_activate populate_section_memmap __populate_section_memmap Passing @pgmap allows __populate_section_memmap() to both fetch the geometry in which memmap metadata is created for and also to let sparse-vmemmap fetch pgmap ranges to co-relate to a given section and pick whether to just reuse tail pages from past onlined sections. [*] https://lore.kernel.org/linux-mm/20210319092635.6214-1-osalvador@suse.de/ Signed-off-by: Joao Martins --- include/linux/memory_hotplug.h | 5 ++++- include/linux/mm.h | 3 ++- mm/memory_hotplug.c | 3 ++- mm/sparse-vmemmap.c | 3 ++- mm/sparse.c | 24 +++++++++++++++--------- 5 files changed, 25 insertions(+), 13 deletions(-) diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h index a7fd2c3ccb77..9b1bca80224d 100644 --- a/include/linux/memory_hotplug.h +++ b/include/linux/memory_hotplug.h @@ -14,6 +14,7 @@ struct mem_section; struct memory_block; struct resource; struct vmem_altmap; +struct dev_pagemap; #ifdef CONFIG_MEMORY_HOTPLUG struct page *pfn_to_online_page(unsigned long pfn); @@ -60,6 +61,7 @@ typedef int __bitwise mhp_t; struct mhp_params { struct vmem_altmap *altmap; pgprot_t pgprot; + struct dev_pagemap *pgmap; }; bool mhp_range_allowed(u64 start, u64 size, bool need_mapping); @@ -333,7 +335,8 @@ extern void remove_pfn_range_from_zone(struct zone *zone, unsigned long nr_pages); extern bool is_memblock_offlined(struct memory_block *mem); extern int sparse_add_section(int nid, unsigned long pfn, - unsigned long nr_pages, struct vmem_altmap *altmap); + unsigned long nr_pages, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap); extern void sparse_remove_section(struct mem_section *ms, unsigned long pfn, unsigned long nr_pages, unsigned long map_offset, struct vmem_altmap *altmap); diff --git a/include/linux/mm.h b/include/linux/mm.h index a127d93612fa..bb3b814e1860 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3083,7 +3083,8 @@ int vmemmap_remap_alloc(unsigned long start, unsigned long end, void *sparse_buffer_alloc(unsigned long size); struct page * __populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap); + unsigned long nr_pages, int nid, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap); pgd_t *vmemmap_pgd_populate(unsigned long addr, int node); p4d_t *vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node); pud_t *vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node); diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 8cb75b26ea4f..c728a8ff38ad 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -268,7 +268,8 @@ int __ref __add_pages(int nid, unsigned long pfn, unsigned long nr_pages, /* Select all remaining pages up to the next section boundary */ cur_nr_pages = min(end_pfn - pfn, SECTION_ALIGN_UP(pfn + 1) - pfn); - err = sparse_add_section(nid, pfn, cur_nr_pages, altmap); + err = sparse_add_section(nid, pfn, cur_nr_pages, altmap, + params->pgmap); if (err) break; cond_resched(); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index bdce883f9286..80d3ba30d345 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -603,7 +603,8 @@ int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, } struct page * __meminit __populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap) + unsigned long nr_pages, int nid, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { unsigned long start = (unsigned long) pfn_to_page(pfn); unsigned long end = start + nr_pages * sizeof(struct page); diff --git a/mm/sparse.c b/mm/sparse.c index 6326cdf36c4f..5310be6171f1 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -453,7 +453,8 @@ static unsigned long __init section_map_size(void) } struct page __init *__populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap) + unsigned long nr_pages, int nid, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { unsigned long size = section_map_size(); struct page *map = sparse_buffer_alloc(size); @@ -552,7 +553,7 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, break; map = __populate_section_memmap(pfn, PAGES_PER_SECTION, - nid, NULL); + nid, NULL, NULL); if (!map) { pr_err("%s: node[%d] memory map backing failed. Some memory will not be available.", __func__, nid); @@ -657,9 +658,10 @@ void offline_mem_sections(unsigned long start_pfn, unsigned long end_pfn) #ifdef CONFIG_SPARSEMEM_VMEMMAP static struct page * __meminit populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap) + unsigned long nr_pages, int nid, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { - return __populate_section_memmap(pfn, nr_pages, nid, altmap); + return __populate_section_memmap(pfn, nr_pages, nid, altmap, pgmap); } static void depopulate_section_memmap(unsigned long pfn, unsigned long nr_pages, @@ -728,7 +730,8 @@ static int fill_subsection_map(unsigned long pfn, unsigned long nr_pages) } #else struct page * __meminit populate_section_memmap(unsigned long pfn, - unsigned long nr_pages, int nid, struct vmem_altmap *altmap) + unsigned long nr_pages, int nid, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { return kvmalloc_node(array_size(sizeof(struct page), PAGES_PER_SECTION), GFP_KERNEL, nid); @@ -851,7 +854,8 @@ static void section_deactivate(unsigned long pfn, unsigned long nr_pages, } static struct page * __meminit section_activate(int nid, unsigned long pfn, - unsigned long nr_pages, struct vmem_altmap *altmap) + unsigned long nr_pages, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { struct mem_section *ms = __pfn_to_section(pfn); struct mem_section_usage *usage = NULL; @@ -883,7 +887,7 @@ static struct page * __meminit section_activate(int nid, unsigned long pfn, if (nr_pages < PAGES_PER_SECTION && early_section(ms)) return pfn_to_page(pfn); - memmap = populate_section_memmap(pfn, nr_pages, nid, altmap); + memmap = populate_section_memmap(pfn, nr_pages, nid, altmap, pgmap); if (!memmap) { section_deactivate(pfn, nr_pages, altmap); return ERR_PTR(-ENOMEM); @@ -898,6 +902,7 @@ static struct page * __meminit section_activate(int nid, unsigned long pfn, * @start_pfn: start pfn of the memory range * @nr_pages: number of pfns to add in the section * @altmap: device page map + * @pgmap: device page map object that owns the section * * This is only intended for hotplug. * @@ -911,7 +916,8 @@ static struct page * __meminit section_activate(int nid, unsigned long pfn, * * -ENOMEM - Out of memory. */ int __meminit sparse_add_section(int nid, unsigned long start_pfn, - unsigned long nr_pages, struct vmem_altmap *altmap) + unsigned long nr_pages, struct vmem_altmap *altmap, + struct dev_pagemap *pgmap) { unsigned long section_nr = pfn_to_section_nr(start_pfn); struct mem_section *ms; @@ -922,7 +928,7 @@ int __meminit sparse_add_section(int nid, unsigned long start_pfn, if (ret < 0) return ret; - memmap = section_activate(nid, start_pfn, nr_pages, altmap); + memmap = section_activate(nid, start_pfn, nr_pages, altmap, pgmap); if (IS_ERR(memmap)) return PTR_ERR(memmap); From patchwork Thu Jun 17 18:44:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12329251 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21F6CC48BE5 for ; Thu, 17 Jun 2021 18:46:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A0933613E9 for ; Thu, 17 Jun 2021 18:46:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A0933613E9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 43DD26B0071; Thu, 17 Jun 2021 14:46:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4183A6B0072; Thu, 17 Jun 2021 14:46:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2DE4D6B0073; Thu, 17 Jun 2021 14:46:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0087.hostedemail.com [216.40.44.87]) by kanga.kvack.org (Postfix) with ESMTP id EF21E6B0071 for ; Thu, 17 Jun 2021 14:46:01 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 81AFF10F02 for ; Thu, 17 Jun 2021 18:46:01 +0000 (UTC) X-FDA: 78264095322.23.8F64338 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf20.hostedemail.com (Postfix) with ESMTP id 013AF552 for ; Thu, 17 Jun 2021 18:46:00 +0000 (UTC) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 15HIam6B020296; Thu, 17 Jun 2021 18:45:45 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=+Di/bDAXE43OnWt5MOHanBvl+EkBqkE3LqLc4QmEmeY=; b=fTwQvBPq7I4wQxxeWN2dyis1BOck+IbHvpnBjwppe6weHWyTrmO1IV8M+S0nUsSxaZyb aEuGEK1yCT6YJ8geTmJkecsAy3gaeJSv0/zg1hUy+F9J5E/Me/zNUqqfeNwMuUkjrtk3 /qVCjH9ksyEkvUkpSIiB20S8m32mVMGvLcP2Mez0jg+IGRXzi9Cx8fDw5bs/fxIA8wgq jt5dcVBKgNUoPJNamE6LEg0/lemy7FzsMkjcc4wwqYwRzWTPBStQfao//mPP3XFl2+PS PDDc7gfgTg8i91oUHJGlDEyijpP3S+HEa55QlU25XrDHocyWfb2UZ6szd6+ehB6Ow5Kz PQ== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 39770hby05-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:45:45 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 15HIjMmu165499; Thu, 17 Jun 2021 18:45:44 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2109.outbound.protection.outlook.com [104.47.58.109]) by aserp3030.oracle.com with ESMTP id 396wavvr6m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:45:44 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mL2qrYm/8rSFAGQAUGVv0ejZB82qdfebNhG352xVnEZcurpW/BZGcsIEvYXq+jroj9J+a44Wy11l+smnwMOLGo3ZTLK5XoYB7E2ksJpP49Zyjek0lT7qKuBiHBxScuro5PuDVyTBmtbIEFuU1I3t1DFABpPy5ZMUTQLezpquVSlduAntLH6+0rmg1KpEt899YfBp8lUgbVZhxiN3w8OyE5kzETiHDUpdpAHnyBVMEmgWfdofnaF1F8DWyDmwz1QSVz9IBtKZ40s1BcoN0KmGK3keZGFcx8t6swj+L5efs7XM3nlo8iY63AMBlT6idlha0/HeRSfKwncW+HXEzlLnmg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+Di/bDAXE43OnWt5MOHanBvl+EkBqkE3LqLc4QmEmeY=; b=JzVX7YunTqtlKwh9sZeKYHlW9opNJrG95ZoPvfMbdL7lqOc0oAAQSpB/XKFlorCo/GoS8k7ZaJW5w6k0lfzCeNQuYWSfZINgJady4sz8+8OXYgbrAxBHsYnCDeul6WqDcAdGgD1gQPYYGDhhA5O0vtqtqbQY/RjFQrW/DnYKYFTjWQEHECjewD1DeXP9sCjuCaViBNdjHMuwq0qt7ouVzhMZJ546PM0FChBKzG2rXZaCJeYbdniWycQbTqNWsD052FeyRo8mfKpmqIiS9vZygiHuZbj52xWM045HDOkBKg+R3U4ZUzstzIKByt3U7Sz2nrjMgZvRC2aBqWFABayG1A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+Di/bDAXE43OnWt5MOHanBvl+EkBqkE3LqLc4QmEmeY=; b=teU9lVbwLHL8fccoSqDDZByE87FUi7xl70Lm5dpEY8qUvxlkTT/fp8TGJzrPgM3wsYzp9dQ84soFA69Zf6yhVNfphBRi8sSF+ci40ixNxdGRiPPNwLLxRiiOuEWZNQlokcpDotMXeJMk0ajH9dorIL5lvrWvlikPWhAWtMazWtI= Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by MN2PR10MB4365.namprd10.prod.outlook.com (2603:10b6:208:199::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.18; Thu, 17 Jun 2021 18:45:40 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582%7]) with mapi id 15.20.4242.021; Thu, 17 Jun 2021 18:45:40 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v2 06/14] mm/sparse-vmemmap: refactor core of vmemmap_populate_basepages() to helper Date: Thu, 17 Jun 2021 19:44:59 +0100 Message-Id: <20210617184507.3662-7-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210617184507.3662-1-joao.m.martins@oracle.com> References: <20210617184507.3662-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4242.16 via Frontend Transport; Thu, 17 Jun 2021 18:45:38 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 13f03156-e50d-4f1c-c178-08d931c01abe X-MS-TrafficTypeDiagnostic: MN2PR10MB4365: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3631; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: vQk4kcpMYlyXwaHABt+Rtdy2YtYvUk5V53SKDI6+himylDBpQTP9LZY0zODNyNCA8aTxI7ygD/IsBPyqA8pG3CHeWLEuyvK3wVR44KMxDSlEPCnXJVkGN2Nnbhnh+78oidWgWlV4IHfsM3qUGamYHDHNS7zrDRWCyLmW+L0YYbK+KxebotWFX6Qf1j6tKxvB1wy0VhzcZUyzNIW4Eha9UTAgSIrGYS6djU2ZI3YBVa33bGrQ5VeH8GTs0h87YdQ9D4Q1VDPGvYjZnTPxab/wUdgZ75qav3+FLtFshEf07jqMd3DQjlKEiucxKcYkziT1Eez488BXZQyCkkF6RkNz/QsN1EqOJkfTtL8qyNf8a4jOxhaTufLjX4psKWUM2zhOPX1q0IekXTGyIew0AaYPfS69MB1dmCKt2PpOSgYCM9JT8V2LjIy5Uw58+fkMW2WDk9GVgLiAFmkSWUIf4a6HRH9/vbodOaQ0LRoMQgFR3d5HAo2r0kR4mGQOLiXxled9plQU0yn9xjPYqv3HezK3TGLJ3ndeCcbsA7n1e1lCq/RBcXxmcXFM6Trt0jRirWQ58pn4EHM+r0PuotAltDB28vxqy4Pm5pdaq2+QZjwIbQ6ATs35zimAIS+rheH78rXVisD/eV2W7A+T6G/qhCYzLFYrRWymQbrIwGqTa81SeFU= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(396003)(366004)(346002)(39860400002)(136003)(376002)(2906002)(16526019)(316002)(186003)(86362001)(26005)(83380400001)(1076003)(956004)(66476007)(66946007)(8676002)(103116003)(6916009)(8936002)(5660300002)(38100700002)(7416002)(66556008)(54906003)(38350700002)(7696005)(52116002)(6486002)(107886003)(6666004)(4326008)(478600001)(36756003)(2616005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 8Y/JqGwER7veMjI4Ba4xouIC1Q+jWsAv8B7iziHRr8Gho4YLPNB2+E8GsqE0qRA/8cpd7ZpW/8RsfwjvbHRkAMTODZEcWQNj0kCMorHAWMLvwi3gpCcsnw6bNheF/fgLEDylwVRwfkfUwJt2Vf6FGLq6HPsLO7lvPwXTEi2dQbvIchsFHTEyTrZf64ow41pUy0Tko7R0xgLI2xOzXsSGDTGvsw4NK3I2xGrdVAR2tk9luqf+CKYcSDg9yPhv5Qj5H1PC50v+2He6sRjOVPPYFRd3VolWdwQc/futQVGh6gnmTvX1zOEEHRMgCLwQTBtrgo4R3JEOkSmTIDdoQQeceSE9DqRcSR22YbKR2YuUX6bYMtETiiVGdeNOEWCAlYDhyA5xhGfPMg0GNE+tgr7WEKA42WMyYnrFIeSPknS7cLa1UsTK7sLpRiPtJLWswJ580DId49DRATAr5/6WWUrRYTgQlQjhZkdp1XdtyHrx7bXpGUAYHqHyRNBtTj9im8PN/7+dmrul5MD3NPvYsyt3q3isrgqYnnOiLiHjT8rDJpfn8MNrBHiomy6do1Mlo8BZQiWTi+UoZKuxwNpnm89U9QmTHzpU7/uqHke9B1i9bfrpeRpEmtlhbWyu/lTZnNTceNTP8KbaRcf6l2ABUmmtxUzWnstEyQqr5ix8vaoEHHCTIRqy9525XFeb4oI37c6k20L/RI+PpxzK6sxg8Omoi1ZHaXkma5epYTO4tznd1ZhTE3j7VoyVnbaJwWOjQJpppFPlbFp/KeqLl9HSgnLldnmoI7XpLUC9qob4Ap7ZTZ811aouGUftVkMsl1iv5cJ7rZ7tg8iQab8DElLEiwjFVXBq/BuA95rUYBJSuSsZy9D59uQu0GbsIF33IBEvRoaBImI0lhcFmiEkiAG+3U/paF/xfnBtgl9LGD51N0VOsHJrUOz+jbk9XfThaPK711S+KcwWrgEH8Enl83LYREGGHXx1TdPSwB5tHbGsaHYJknhUGy6r3nV7HZ3dOVUXc9+N6fkkbKGi7we4S8TqKhuof/3m6w0LsBAvbl2DAK1LpZSl+ziCuxGqDrfM0qm1ZN/rUSuJII/TY7iJdbOGL0FnG7cbULAIJenlifhSr3+D1lPT8zDwxFyIiJpUf4Ycq9udovmsuM8jQ3CIyCbScqF1+/7RRrcYfiOnH/UBe370EMToHeZ/QGnW2X66vGsdgQi/I4idUxm2zDadJkVrAC9eTGzVUp0P58MSRJ/gS9RxN4+PeDcY4shduAXqSBSE3TXY7Qrrw1X4FDDyyusSroDVw3iLtJhYHMVucCbG2yQfPFnjPrf8oq7ApyjIqlQ+R9NI X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 13f03156-e50d-4f1c-c178-08d931c01abe X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jun 2021 18:45:40.3160 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: fAOlPGBv8USXstf49nD0FgXgZ7M6ofqUlBrcQLHNGpeuAcVrzA2Ku0rdUj8w4jxr1/JMkLsjpVen4KRucH/SLWLz6zfMN9PuZ6e7xF35azc= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR10MB4365 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10018 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 mlxscore=0 spamscore=0 adultscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2106170117 X-Proofpoint-GUID: 6a6TrCDbiXdyIACpoxPDiHzUZhAgiP4X X-Proofpoint-ORIG-GUID: 6a6TrCDbiXdyIACpoxPDiHzUZhAgiP4X X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 013AF552 X-Stat-Signature: sz4kjj1f6fbiw6qi1xpcy3bhg38j3g18 Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=fTwQvBPq; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=teU9lVbw; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf20.hostedemail.com: domain of joao.m.martins@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=joao.m.martins@oracle.com X-HE-Tag: 1623955560-286769 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for describing a memmap with compound pages, move the actual pte population logic into a separate function vmemmap_populate_address() and have vmemmap_populate_basepages() walk through all base pages it needs to populate. Signed-off-by: Joao Martins --- mm/sparse-vmemmap.c | 44 ++++++++++++++++++++++++++------------------ 1 file changed, 26 insertions(+), 18 deletions(-) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 80d3ba30d345..76f4158f6301 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -570,33 +570,41 @@ pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) return pgd; } -int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, - int node, struct vmem_altmap *altmap) +static int __meminit vmemmap_populate_address(unsigned long addr, int node, + struct vmem_altmap *altmap) { - unsigned long addr = start; pgd_t *pgd; p4d_t *p4d; pud_t *pud; pmd_t *pmd; pte_t *pte; + pgd = vmemmap_pgd_populate(addr, node); + if (!pgd) + return -ENOMEM; + p4d = vmemmap_p4d_populate(pgd, addr, node); + if (!p4d) + return -ENOMEM; + pud = vmemmap_pud_populate(p4d, addr, node); + if (!pud) + return -ENOMEM; + pmd = vmemmap_pmd_populate(pud, addr, node); + if (!pmd) + return -ENOMEM; + pte = vmemmap_pte_populate(pmd, addr, node, altmap); + if (!pte) + return -ENOMEM; + vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); +} + +int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, + int node, struct vmem_altmap *altmap) +{ + unsigned long addr = start; + for (; addr < end; addr += PAGE_SIZE) { - pgd = vmemmap_pgd_populate(addr, node); - if (!pgd) - return -ENOMEM; - p4d = vmemmap_p4d_populate(pgd, addr, node); - if (!p4d) - return -ENOMEM; - pud = vmemmap_pud_populate(p4d, addr, node); - if (!pud) - return -ENOMEM; - pmd = vmemmap_pmd_populate(pud, addr, node); - if (!pmd) - return -ENOMEM; - pte = vmemmap_pte_populate(pmd, addr, node, altmap); - if (!pte) + if (vmemmap_populate_address(addr, node, altmap)) return -ENOMEM; - vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); } return 0; From patchwork Thu Jun 17 18:45:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12329263 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 08EA9C49361 for ; Thu, 17 Jun 2021 18:46:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8018F613E9 for ; Thu, 17 Jun 2021 18:46:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8018F613E9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 85F216B007B; Thu, 17 Jun 2021 14:46:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 79A9D6B007D; Thu, 17 Jun 2021 14:46:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C3FC6B007E; Thu, 17 Jun 2021 14:46:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0232.hostedemail.com [216.40.44.232]) by kanga.kvack.org (Postfix) with ESMTP id 136486B007B for ; Thu, 17 Jun 2021 14:46:11 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 978321B414 for ; Thu, 17 Jun 2021 18:46:10 +0000 (UTC) X-FDA: 78264095700.16.3A4549B Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf23.hostedemail.com (Postfix) with ESMTP id EC80CA000274 for ; Thu, 17 Jun 2021 18:46:09 +0000 (UTC) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 15HIavBb016877; Thu, 17 Jun 2021 18:46:04 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=3mj1lwLN0qcqsdTuV65uEdmx7WahVMI775k4DK6dPXk=; b=E98V7SwhQ5RAjmAWFTCpG2qaRleuaLYLFyM9RugH+qpkkfINdtiRG66Ek7BtC4RM5Rq5 HIU7qgAJEh8u6q0su6rCVglv3MGhYR0Pn1Ibq5Zmt1f6SHRU0iZGj0saKLY69XgPXr/O joKB2H+U7wtGb7rYUrv/nSq0NZfa4szKsJ1cT8CyuaCg2w5tgeXiN7xI6zBZUdmGsFp8 JQcME8AHhECHU4jbNAWS17nysRWf2DmM9rWq55usKx39dLAs6WK/uu5kDxuLc6+VDrs5 zJi2s0n3+j10HX0obKtFebocwuJVnGr/yEGL2Q7ys3yIn6kAgpgnLIctdkAWotW2crqE wg== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 397mptjh4b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:46:03 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 15HIjZ0x180178; Thu, 17 Jun 2021 18:46:02 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2174.outbound.protection.outlook.com [104.47.55.174]) by userp3020.oracle.com with ESMTP id 396waxy64a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:46:02 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mWTvd00DcZWDbhmmvD2O+ZHwY0kvUcvJ+ht80oPjESE3A97lBkBBJ6YD/z179IDKYONbzqEoHBVim5qXxaeWzTDxDR9s2PIUBqlyv3rRN1TPLoTji461qwMk/NDcYArdslB9AZUGFWPgfkfUJ2UcYqtmaW/2/jgmL5wVoYngeM8YKIpdIjMhcZwBLfjsthFXtZ5tzh+p+c6MzEPtAtQaq/xbgWNnmBfoY/E4KdRlGWhfe9XOpp5vhhr2dw6OvWlHIQZOuJ2p4qPQjrE7sDk8YkljiDPvBg+4lPlI/ioMmKEHaN13bjBafyXfVBKGZ7CPAFYljVHWArATjub1CTIqlA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3mj1lwLN0qcqsdTuV65uEdmx7WahVMI775k4DK6dPXk=; b=k61XBk7XsJlc4RSOK7Wb3OtdXDn2Xqr/piiNWWLf0ESVbqKWjlYQBe1dO4FH/GJsMX+5611EiUFjmi7iPW+pxLVIjhf22/g31088PNmjmygoRrSeIAcCHEoRS2gt6pDi+tdqqH4iw1NIuZLQYCqrONfRzvKo4Alkngujof7xMjCRDNloKeg1whuedpQQQg9tLM0rsjSsnikh/m7h/hucixDBc4tDgcxzvxM46mcqm8rbCHTmjosf+r/0XPJ387M/vpueAWdVogcJegyG9M8j3NDsOqXEvo5FoR6XVVqfHnFdHC7OhteQOa9ytOndzPY69dUcnCINATYoS3lu6YRFtA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3mj1lwLN0qcqsdTuV65uEdmx7WahVMI775k4DK6dPXk=; b=ouqAoR8ly5EYR59/YKDVOIRVZnopbixUSRufpVQ/R9j1n4Mwjmaaj5lUGYRxb3QlyrSZW/fQhWoI1Xnc6rsRyeOiHsUjD4bnOGrkgb0Yp6ZFLIcnp4mp+bXh4HdAKQgcV2e8NZMCXG+frp56SQe3h1WakVO09nbjeUVOcNfNvq8= Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by MN2PR10MB4206.namprd10.prod.outlook.com (2603:10b6:208:1df::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.15; Thu, 17 Jun 2021 18:45:43 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582%7]) with mapi id 15.20.4242.021; Thu, 17 Jun 2021 18:45:43 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v2 07/14] mm/hugetlb_vmemmap: move comment block to Documentation/vm Date: Thu, 17 Jun 2021 19:45:00 +0100 Message-Id: <20210617184507.3662-8-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210617184507.3662-1-joao.m.martins@oracle.com> References: <20210617184507.3662-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4242.16 via Frontend Transport; Thu, 17 Jun 2021 18:45:40 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 47a416bd-32f2-48e5-4f3b-08d931c01c5a X-MS-TrafficTypeDiagnostic: MN2PR10MB4206: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: z/jnCqKIejaZrNM1Zlx/P/TmwV/cz1b9BUAjVxUj3MIgCtNJZ8wOUST+FLKOS08Ql71JeiJIUpb58/5La3bmNG1Lkie6ODPxyapyMJNmobv0LsXhiReS0bh1M/I1FgWUC46fbTuU+IchBUZ7qLElmvpau3QdcrPHBgDb6LmG5d1V5HeFmx3yOs93qdG50Y/xYWb0z3MWAdsVa7Q+lb6l7UShBI7a3yJFoP4dOYJqxia+qluabAbWupJJa1SOkuJ34CXp93pkPZiK58biwrGMC/KtTABnO1HQZIBlgoS5Lge7/bbOBxllETFZM6ARTnEUZC8sN6GfT2lWNpDk76nTH87dDXc078dxoSjh0R+oV71fDFiEfqWHFGxxlkMZ/2rz1WcFMz/PsjQNI4ot9+GhUTv1HvlrwAMdTopZ/aDoYPwcZyi8vZGjPJqpiA4m7iKIIHt19f5yBa7NIQecR2zAoP4vWQYtbNh70LC0mxUr2LFCFTRlfna+XMQ2aOxlcqfE8CCfKzvNNsppqVySZl9LvAXBDzqwwxOo8qnGv9TjS+9Q0V/7oCJCyuhwkMx6x8G/WkR+zN2r88TNMavb2MtlMrl8+FbGXINfqkJ92qwyE18GOlFQflGJzFCwYHIjJ48m/t/SESrx5/VNvCT6gZLfsFhg+IHSOcdl/8eL6hLz3Ik= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(136003)(346002)(376002)(39860400002)(396003)(2616005)(956004)(478600001)(52116002)(38350700002)(7696005)(8676002)(5660300002)(38100700002)(186003)(107886003)(36756003)(6916009)(4326008)(2906002)(16526019)(66556008)(66476007)(316002)(6666004)(86362001)(6486002)(66946007)(30864003)(54906003)(83380400001)(1076003)(7416002)(26005)(103116003)(8936002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: dZW/IRghpUxSb/+ApmKTbIfvnoZxkn1sNN/iBHT1UoRX0Dq3TKme8OxtVB2shKm1FxkisBhkirY3nt5c6Dyn6rcdjVpJxkWP5lSLO/3HrOTY2eljNzgoqy9zas4SA27D5te3RyXlAGinmaZ62Rm/M4HImr+tXuUr4r5WdCmx0uodu0dNeyW47HrzrVTcojb2TTbvtGu7dZLdvm/6qcbEYnhcE13oYe1tBdsTattkKa5MMlKoGmpTXhDbyIvILHXPwyBA3py9BuXAZwKpPznZR8OoYXduIX2IUuPLBitDeoJtMU64cfKb5GkbN1FKEFdLvKZr3RwpgQjZRJ1jcq9j4f1PGpy894cwJlkAk4aOI+07tMrG4ZKcck6k0/bnVcocOCd5xwyZZJIXKBCglTmRi0piuNoLQqdJgVU4QN+n0WPbp1a5pMVNMyJq9wc+sMmubEMPR4HFCixVH9gBn7ewYToyuWH2Ze5Jz3cIGAvBtaaUKqH7yewFm7TKd363w//uQLAe9Xokl4B+RU1HuwUSLgcRYTu0VrgfV3mPqLdSserqvk47ichmZunv86FW4HuZKq4PaFQiqJBXKDEylh8rykg2mW2ogGC4n9SBL7nwkitxTE15eoGEluPPaF1SflBOAEXnJe5NNI/Bgm/iB0HoWSpqFSzkwLaC+0lgqlzDka5bwuo8HpfecBBFQGY/XxgRObX3d2S96Z8tiNqGXIiIN3cYn0NPvw/8HwqtKfXbn6qRkmqWake2UoH4bhT4MzR7omDGdJlw8WTU/yLysZqK9pPyyqwisNG9wA1HRV7avQhJoK5YvRR8b1rhvXOGhdfXp0uy7+ekjkop0gxCFn0JHxscRXsyhHyPEqjkwFKswrpZRtftbWltkbTmJMQCoStqr1up8nwvuh4hGbxGnxX+1d0IAa2tBqFG/U27mkAqIjLd+CAaq3uv3LyS0udiqDTxfSQ3iuX6jA0+heujL66B+Ct2t7Xph0EpHvqJvmQGn0Occ5K8Edf3m5DLzZBqP7wk1A8LnWBpT+KfAKQcOD0l68CLBy2H6V273tqkO0t6RnaGL38PcZcrL6tuDMRQq8f7d8sGFNhg1+wRnLnjbe6Cy+tM0RrbJNn/p3z2oMCGVrwv8b0C9gxxS0K4zs9NsDAk8p/RXzbzjFFmmfmCrUaMl9zFX92YMAhsjH3Q3Murkp89paF9Og6XKEqTkKlaz3tkZqaEMqP0VsebSTLpdMWi7b4jq61VaXsKd7Tbt/wTYDL+nPdGlk/0QaoGYnBPH1FLrO/FJP4V12byne5bC6tWHOiXRrEvV6Q2IdLjSRoEMtNa+Bv3PeKhC7q7nTMgckPb X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 47a416bd-32f2-48e5-4f3b-08d931c01c5a X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jun 2021 18:45:43.1147 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 2Cim6kK80vwKXAPepWPXLrtNEOZXI27FfPCSSu0cbust0S4vXh/qA4nciE3FEFU2sx8/7wSOfJOebKOso4MEtR+2hq2v8qzBM59qlmpXW90= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR10MB4206 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10018 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 malwarescore=0 suspectscore=0 spamscore=0 bulkscore=0 mlxlogscore=999 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2106170117 X-Proofpoint-GUID: RIwiOldhSB3jbstGN3wyY7wTycuroupr X-Proofpoint-ORIG-GUID: RIwiOldhSB3jbstGN3wyY7wTycuroupr Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=E98V7Swh; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=ouqAoR8l; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf23.hostedemail.com: domain of joao.m.martins@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=joao.m.martins@oracle.com X-Stat-Signature: njrr59o8iojikq7m3onrecmqdocannpq X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: EC80CA000274 X-HE-Tag: 1623955569-745340 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In preparation for device-dax for using hugetlbfs compound page tail deduplication technique, move the comment block explanation into a common place in Documentation/vm. Cc: Muchun Song Cc: Mike Kravetz Suggested-by: Dan Williams Signed-off-by: Joao Martins --- Documentation/vm/compound_pagemaps.rst | 170 +++++++++++++++++++++++++ Documentation/vm/index.rst | 1 + mm/hugetlb_vmemmap.c | 162 +---------------------- 3 files changed, 172 insertions(+), 161 deletions(-) create mode 100644 Documentation/vm/compound_pagemaps.rst diff --git a/Documentation/vm/compound_pagemaps.rst b/Documentation/vm/compound_pagemaps.rst new file mode 100644 index 000000000000..6b1af50e8201 --- /dev/null +++ b/Documentation/vm/compound_pagemaps.rst @@ -0,0 +1,170 @@ +.. SPDX-License-Identifier: GPL-2.0 + +.. _commpound_pagemaps: + +================================== +Free some vmemmap pages of HugeTLB +================================== + +The struct page structures (page structs) are used to describe a physical +page frame. By default, there is a one-to-one mapping from a page frame to +it's corresponding page struct. + +HugeTLB pages consist of multiple base page size pages and is supported by +many architectures. See hugetlbpage.rst in the Documentation directory for +more details. On the x86-64 architecture, HugeTLB pages of size 2MB and 1GB +are currently supported. Since the base page size on x86 is 4KB, a 2MB +HugeTLB page consists of 512 base pages and a 1GB HugeTLB page consists of +4096 base pages. For each base page, there is a corresponding page struct. + +Within the HugeTLB subsystem, only the first 4 page structs are used to +contain unique information about a HugeTLB page. __NR_USED_SUBPAGE provides +this upper limit. The only 'useful' information in the remaining page structs +is the compound_head field, and this field is the same for all tail pages. + +By removing redundant page structs for HugeTLB pages, memory can be returned +to the buddy allocator for other uses. + +Different architectures support different HugeTLB pages. For example, the +following table is the HugeTLB page size supported by x86 and arm64 +architectures. Because arm64 supports 4k, 16k, and 64k base pages and +supports contiguous entries, so it supports many kinds of sizes of HugeTLB +page. + ++--------------+-----------+-----------------------------------------------+ +| Architecture | Page Size | HugeTLB Page Size | ++--------------+-----------+-----------+-----------+-----------+-----------+ +| x86-64 | 4KB | 2MB | 1GB | | | ++--------------+-----------+-----------+-----------+-----------+-----------+ +| | 4KB | 64KB | 2MB | 32MB | 1GB | +| +-----------+-----------+-----------+-----------+-----------+ +| arm64 | 16KB | 2MB | 32MB | 1GB | | +| +-----------+-----------+-----------+-----------+-----------+ +| | 64KB | 2MB | 512MB | 16GB | | ++--------------+-----------+-----------+-----------+-----------+-----------+ + +When the system boot up, every HugeTLB page has more than one struct page +structs which size is (unit: pages): + + struct_size = HugeTLB_Size / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE + +Where HugeTLB_Size is the size of the HugeTLB page. We know that the size +of the HugeTLB page is always n times PAGE_SIZE. So we can get the following +relationship. + + HugeTLB_Size = n * PAGE_SIZE + +Then, + + struct_size = n * PAGE_SIZE / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE + = n * sizeof(struct page) / PAGE_SIZE + +We can use huge mapping at the pud/pmd level for the HugeTLB page. + +For the HugeTLB page of the pmd level mapping, then + + struct_size = n * sizeof(struct page) / PAGE_SIZE + = PAGE_SIZE / sizeof(pte_t) * sizeof(struct page) / PAGE_SIZE + = sizeof(struct page) / sizeof(pte_t) + = 64 / 8 + = 8 (pages) + +Where n is how many pte entries which one page can contains. So the value of +n is (PAGE_SIZE / sizeof(pte_t)). + +This optimization only supports 64-bit system, so the value of sizeof(pte_t) +is 8. And this optimization also applicable only when the size of struct page +is a power of two. In most cases, the size of struct page is 64 bytes (e.g. +x86-64 and arm64). So if we use pmd level mapping for a HugeTLB page, the +size of struct page structs of it is 8 page frames which size depends on the +size of the base page. + +For the HugeTLB page of the pud level mapping, then + + struct_size = PAGE_SIZE / sizeof(pmd_t) * struct_size(pmd) + = PAGE_SIZE / 8 * 8 (pages) + = PAGE_SIZE (pages) + +Where the struct_size(pmd) is the size of the struct page structs of a +HugeTLB page of the pmd level mapping. + +E.g.: A 2MB HugeTLB page on x86_64 consists in 8 page frames while 1GB +HugeTLB page consists in 4096. + +Next, we take the pmd level mapping of the HugeTLB page as an example to +show the internal implementation of this optimization. There are 8 pages +struct page structs associated with a HugeTLB page which is pmd mapped. + +Here is how things look before optimization. + + HugeTLB struct pages(8 pages) page frame(8 pages) + +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ + | | | 0 | -------------> | 0 | + | | +-----------+ +-----------+ + | | | 1 | -------------> | 1 | + | | +-----------+ +-----------+ + | | | 2 | -------------> | 2 | + | | +-----------+ +-----------+ + | | | 3 | -------------> | 3 | + | | +-----------+ +-----------+ + | | | 4 | -------------> | 4 | + | PMD | +-----------+ +-----------+ + | level | | 5 | -------------> | 5 | + | mapping | +-----------+ +-----------+ + | | | 6 | -------------> | 6 | + | | +-----------+ +-----------+ + | | | 7 | -------------> | 7 | + | | +-----------+ +-----------+ + | | + | | + | | + +-----------+ + +The value of page->compound_head is the same for all tail pages. The first +page of page structs (page 0) associated with the HugeTLB page contains the 4 +page structs necessary to describe the HugeTLB. The only use of the remaining +pages of page structs (page 1 to page 7) is to point to page->compound_head. +Therefore, we can remap pages 2 to 7 to page 1. Only 2 pages of page structs +will be used for each HugeTLB page. This will allow us to free the remaining +6 pages to the buddy allocator. + +Here is how things look after remapping. + + HugeTLB struct pages(8 pages) page frame(8 pages) + +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ + | | | 0 | -------------> | 0 | + | | +-----------+ +-----------+ + | | | 1 | -------------> | 1 | + | | +-----------+ +-----------+ + | | | 2 | ----------------^ ^ ^ ^ ^ ^ + | | +-----------+ | | | | | + | | | 3 | ------------------+ | | | | + | | +-----------+ | | | | + | | | 4 | --------------------+ | | | + | PMD | +-----------+ | | | + | level | | 5 | ----------------------+ | | + | mapping | +-----------+ | | + | | | 6 | ------------------------+ | + | | +-----------+ | + | | | 7 | --------------------------+ + | | +-----------+ + | | + | | + | | + +-----------+ + +When a HugeTLB is freed to the buddy system, we should allocate 6 pages for +vmemmap pages and restore the previous mapping relationship. + +For the HugeTLB page of the pud level mapping. It is similar to the former. +We also can use this approach to free (PAGE_SIZE - 2) vmemmap pages. + +Apart from the HugeTLB page of the pmd/pud level mapping, some architectures +(e.g. aarch64) provides a contiguous bit in the translation table entries +that hints to the MMU to indicate that it is one of a contiguous set of +entries that can be cached in a single TLB entry. + +The contiguous bit is used to increase the mapping size at the pmd and pte +(last) level. So this type of HugeTLB page can be optimized only when its +size of the struct page structs is greater than 2 pages. + diff --git a/Documentation/vm/index.rst b/Documentation/vm/index.rst index eff5fbd492d0..19f981a73a54 100644 --- a/Documentation/vm/index.rst +++ b/Documentation/vm/index.rst @@ -31,6 +31,7 @@ descriptions of data structures and algorithms. active_mm arch_pgtable_helpers balance + commpound_pagemaps cleancache free_page_reporting frontswap diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index c540c21e26f5..69d1f0a90e02 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -6,167 +6,7 @@ * * Author: Muchun Song * - * The struct page structures (page structs) are used to describe a physical - * page frame. By default, there is a one-to-one mapping from a page frame to - * it's corresponding page struct. - * - * HugeTLB pages consist of multiple base page size pages and is supported by - * many architectures. See hugetlbpage.rst in the Documentation directory for - * more details. On the x86-64 architecture, HugeTLB pages of size 2MB and 1GB - * are currently supported. Since the base page size on x86 is 4KB, a 2MB - * HugeTLB page consists of 512 base pages and a 1GB HugeTLB page consists of - * 4096 base pages. For each base page, there is a corresponding page struct. - * - * Within the HugeTLB subsystem, only the first 4 page structs are used to - * contain unique information about a HugeTLB page. __NR_USED_SUBPAGE provides - * this upper limit. The only 'useful' information in the remaining page structs - * is the compound_head field, and this field is the same for all tail pages. - * - * By removing redundant page structs for HugeTLB pages, memory can be returned - * to the buddy allocator for other uses. - * - * Different architectures support different HugeTLB pages. For example, the - * following table is the HugeTLB page size supported by x86 and arm64 - * architectures. Because arm64 supports 4k, 16k, and 64k base pages and - * supports contiguous entries, so it supports many kinds of sizes of HugeTLB - * page. - * - * +--------------+-----------+-----------------------------------------------+ - * | Architecture | Page Size | HugeTLB Page Size | - * +--------------+-----------+-----------+-----------+-----------+-----------+ - * | x86-64 | 4KB | 2MB | 1GB | | | - * +--------------+-----------+-----------+-----------+-----------+-----------+ - * | | 4KB | 64KB | 2MB | 32MB | 1GB | - * | +-----------+-----------+-----------+-----------+-----------+ - * | arm64 | 16KB | 2MB | 32MB | 1GB | | - * | +-----------+-----------+-----------+-----------+-----------+ - * | | 64KB | 2MB | 512MB | 16GB | | - * +--------------+-----------+-----------+-----------+-----------+-----------+ - * - * When the system boot up, every HugeTLB page has more than one struct page - * structs which size is (unit: pages): - * - * struct_size = HugeTLB_Size / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE - * - * Where HugeTLB_Size is the size of the HugeTLB page. We know that the size - * of the HugeTLB page is always n times PAGE_SIZE. So we can get the following - * relationship. - * - * HugeTLB_Size = n * PAGE_SIZE - * - * Then, - * - * struct_size = n * PAGE_SIZE / PAGE_SIZE * sizeof(struct page) / PAGE_SIZE - * = n * sizeof(struct page) / PAGE_SIZE - * - * We can use huge mapping at the pud/pmd level for the HugeTLB page. - * - * For the HugeTLB page of the pmd level mapping, then - * - * struct_size = n * sizeof(struct page) / PAGE_SIZE - * = PAGE_SIZE / sizeof(pte_t) * sizeof(struct page) / PAGE_SIZE - * = sizeof(struct page) / sizeof(pte_t) - * = 64 / 8 - * = 8 (pages) - * - * Where n is how many pte entries which one page can contains. So the value of - * n is (PAGE_SIZE / sizeof(pte_t)). - * - * This optimization only supports 64-bit system, so the value of sizeof(pte_t) - * is 8. And this optimization also applicable only when the size of struct page - * is a power of two. In most cases, the size of struct page is 64 bytes (e.g. - * x86-64 and arm64). So if we use pmd level mapping for a HugeTLB page, the - * size of struct page structs of it is 8 page frames which size depends on the - * size of the base page. - * - * For the HugeTLB page of the pud level mapping, then - * - * struct_size = PAGE_SIZE / sizeof(pmd_t) * struct_size(pmd) - * = PAGE_SIZE / 8 * 8 (pages) - * = PAGE_SIZE (pages) - * - * Where the struct_size(pmd) is the size of the struct page structs of a - * HugeTLB page of the pmd level mapping. - * - * E.g.: A 2MB HugeTLB page on x86_64 consists in 8 page frames while 1GB - * HugeTLB page consists in 4096. - * - * Next, we take the pmd level mapping of the HugeTLB page as an example to - * show the internal implementation of this optimization. There are 8 pages - * struct page structs associated with a HugeTLB page which is pmd mapped. - * - * Here is how things look before optimization. - * - * HugeTLB struct pages(8 pages) page frame(8 pages) - * +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ - * | | | 0 | -------------> | 0 | - * | | +-----------+ +-----------+ - * | | | 1 | -------------> | 1 | - * | | +-----------+ +-----------+ - * | | | 2 | -------------> | 2 | - * | | +-----------+ +-----------+ - * | | | 3 | -------------> | 3 | - * | | +-----------+ +-----------+ - * | | | 4 | -------------> | 4 | - * | PMD | +-----------+ +-----------+ - * | level | | 5 | -------------> | 5 | - * | mapping | +-----------+ +-----------+ - * | | | 6 | -------------> | 6 | - * | | +-----------+ +-----------+ - * | | | 7 | -------------> | 7 | - * | | +-----------+ +-----------+ - * | | - * | | - * | | - * +-----------+ - * - * The value of page->compound_head is the same for all tail pages. The first - * page of page structs (page 0) associated with the HugeTLB page contains the 4 - * page structs necessary to describe the HugeTLB. The only use of the remaining - * pages of page structs (page 1 to page 7) is to point to page->compound_head. - * Therefore, we can remap pages 2 to 7 to page 1. Only 2 pages of page structs - * will be used for each HugeTLB page. This will allow us to free the remaining - * 6 pages to the buddy allocator. - * - * Here is how things look after remapping. - * - * HugeTLB struct pages(8 pages) page frame(8 pages) - * +-----------+ ---virt_to_page---> +-----------+ mapping to +-----------+ - * | | | 0 | -------------> | 0 | - * | | +-----------+ +-----------+ - * | | | 1 | -------------> | 1 | - * | | +-----------+ +-----------+ - * | | | 2 | ----------------^ ^ ^ ^ ^ ^ - * | | +-----------+ | | | | | - * | | | 3 | ------------------+ | | | | - * | | +-----------+ | | | | - * | | | 4 | --------------------+ | | | - * | PMD | +-----------+ | | | - * | level | | 5 | ----------------------+ | | - * | mapping | +-----------+ | | - * | | | 6 | ------------------------+ | - * | | +-----------+ | - * | | | 7 | --------------------------+ - * | | +-----------+ - * | | - * | | - * | | - * +-----------+ - * - * When a HugeTLB is freed to the buddy system, we should allocate 6 pages for - * vmemmap pages and restore the previous mapping relationship. - * - * For the HugeTLB page of the pud level mapping. It is similar to the former. - * We also can use this approach to free (PAGE_SIZE - 2) vmemmap pages. - * - * Apart from the HugeTLB page of the pmd/pud level mapping, some architectures - * (e.g. aarch64) provides a contiguous bit in the translation table entries - * that hints to the MMU to indicate that it is one of a contiguous set of - * entries that can be cached in a single TLB entry. - * - * The contiguous bit is used to increase the mapping size at the pmd and pte - * (last) level. So this type of HugeTLB page can be optimized only when its - * size of the struct page structs is greater than 2 pages. + * See Documentation/vm/compound_pagemaps.rst */ #define pr_fmt(fmt) "HugeTLB: " fmt From patchwork Thu Jun 17 18:45:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12329261 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6228C2B9F4 for ; Thu, 17 Jun 2021 18:46:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6FD67613E9 for ; Thu, 17 Jun 2021 18:46:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6FD67613E9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0C6CE6B0075; Thu, 17 Jun 2021 14:46:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EEB326B007E; Thu, 17 Jun 2021 14:46:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF06B6B0075; Thu, 17 Jun 2021 14:46:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0183.hostedemail.com [216.40.44.183]) by kanga.kvack.org (Postfix) with ESMTP id 8BE706B0078 for ; Thu, 17 Jun 2021 14:46:07 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 25ACC1804DF9A for ; Thu, 17 Jun 2021 18:46:07 +0000 (UTC) X-FDA: 78264095574.18.E88A60F Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf25.hostedemail.com (Postfix) with ESMTP id ABB02600014A for ; Thu, 17 Jun 2021 18:46:06 +0000 (UTC) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 15HIawq0007633; Thu, 17 Jun 2021 18:45:49 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=5/EPvxLoSzQ6jqxssx/JQu9Lz9ChgjF1ZDvVmPGJq5g=; b=UwWbrL60mZ4XbySCxIpJ80KUZIluJ61g0HT+J51XG0WbxtOSnyH263gpTk1y3smgF9+X DBd2WNoHkbi0EA4TpjvBe+f2bZrcbOw7yiTuofhzrPSPv8JEhDVF7Nzhg5Lh4DbAUekv meyB//xwxB6bG5KNXSiSpK0otiua10qZ+bsQ2yl41XwfIEuqQp/VJ2MJaT4hD4ndAusp Tn46QT+EayXF2fDDQxwhieBcEomy+SSLBzUe4jJiv8LzmQFkD8nQBTSiXN2CZ1hw2/Tl /L8EDgMgLK+xj9urhEeyY1JO/cyLdbLGjrKjjSbPZjlIYefvkmwZ6pXD/0yECCnQmjC4 MQ== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 397w1y1r6j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:45:49 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 15HIjNxx165547; Thu, 17 Jun 2021 18:45:48 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2109.outbound.protection.outlook.com [104.47.55.109]) by aserp3030.oracle.com with ESMTP id 396wavvr85-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:45:48 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=oZU6Kf3xaMVtPpfHdWtPAfeI7lQSRwxw16Fm4DppWaj0QiXrduknA4DX557UPgVfN0q0GlSKZ6KtQfjG81iyTe4yDjg0EzsHcu55eROiOLL70dc5kfFIiRDNfx8vnI0WePWLssg8noHh25y38QA3UKQQLscNNamzKFS1g8CnZP0GdNF/sUey7Edy0seBC69MEph+oQ/onhUb7gmXTVv716oTwERgOpkTPAWypNCbNnTGsaDOdIbq3asybtLFqvlI2WO+YZnqDY02Y//blTuhGXEFQhjPThgcUGhphEUM85CF4M9N3u/QbxaRriAYDdG0sHEz5b2L1jI3mYsuEgYDQw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5/EPvxLoSzQ6jqxssx/JQu9Lz9ChgjF1ZDvVmPGJq5g=; b=BHp71CfFHVC7hU0pR5ycICEPvGatYPTZA4/MGeV8O/h8n65USEmQz9YvdHXnY9sUdIOj2yWedeK7V6pHoyGLDnqoSxVksQU89JHfX6/sBI1GKKurNwDLSUsAOsZ2Gr/UXxmLFpPAaml5X5tERaTTjm8PESrFgiTemhLtzfw0Lu7jQW5UJ3Foyl9CfbNh8dX+3jT415q0nrny1jomQgTh+L/Hr5DT5likOzd/6NvlqK/5s4JJrovWPGgYYZRyLNklrufoCe14+I0mpr5m5VtMGppu+lFvRxniMmog9jnaddI3ymArs+KZc0+AYOTICb1qJTThsmseYGQ9hnIRuYtZVA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5/EPvxLoSzQ6jqxssx/JQu9Lz9ChgjF1ZDvVmPGJq5g=; b=Zj09Qt9mWVGprLGGNpktlsFWD26uFyrS+Dkur24ljfNeJBDF88OAvE9J4BxHzSX0ahHekMauRLIZWdkY2WmakPavURQqKnLl0ERPYOXe5zIrniMOpO8NhGC5Gu1XU/mJJ7iRst4HDP6iFUXpm8U2HvN5GgdB+NSCxgKa7hSGAlg= Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by MN2PR10MB3983.namprd10.prod.outlook.com (2603:10b6:208:1bd::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4242.16; Thu, 17 Jun 2021 18:45:45 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582%7]) with mapi id 15.20.4242.021; Thu, 17 Jun 2021 18:45:45 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v2 08/14] mm/sparse-vmemmap: populate compound pagemaps Date: Thu, 17 Jun 2021 19:45:01 +0100 Message-Id: <20210617184507.3662-9-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210617184507.3662-1-joao.m.martins@oracle.com> References: <20210617184507.3662-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4242.16 via Frontend Transport; Thu, 17 Jun 2021 18:45:43 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 053bc3fe-8d08-47ae-8ec8-08d931c01de9 X-MS-TrafficTypeDiagnostic: MN2PR10MB3983: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: sL3Wo6XHpaO79N2IMi+2VrDungi4QtockDK3NAnOKuKPq1874/mXmsoEJYepPEYts/zR+SRwfB24l4835V4bVMiPslgBG6v+VM94q4zImT8svFT8ED77ubgMv65IbbUBHgkYJGrer3GzcTRlpzPUum+gLQFZG7uHZi36FGAQAcPxJ5W1I39omHQCpL/sOrbCtPzrFeofvNCbhOsgxCkH5Jkh6HekB9o9bYvLCYjjWya1NT9B+pc0TJRVqVZZz+rFkSnRMbo+rLOBE45SCZlpzBqr516yvd2pTXEHhEvm6+8MQfobs0YRa08urac0wpqJdSVOsvOMean678HI8/VV9KuMlP6UtXht40T4RIRV62vTKeriHnRZ6xf/FKftfhe8e6PGJ4d2X5kRKIK+n7O611fj5FgYM6TWD1bL2Orm8iinE5LvJRvXd6NRq6x+P1WUKcV/3DQTqK8W/ggYK8W12GhgU6nNQIx9ycnQcPZ3fPaDBD23j6Fj6iZ5qUQwXXO3XsPwMwik+Fy0vqy+sM6IqvsSgLD5Vo1zUBNIuJSPX/TEri91WQMF4YIDh1Lwyu4kCoBN5D1pYG7hLAtI7W39xEz7IomXNQplBvq7q9sHFDPaIQBjMtHxRNLhJCK9avJMm+DQ1U8eRolVZA2gJ251MDJ7g/a641CyQGQnXKWXzZw= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(396003)(39860400002)(136003)(366004)(346002)(376002)(26005)(36756003)(16526019)(86362001)(7696005)(52116002)(4326008)(38350700002)(7416002)(1076003)(83380400001)(186003)(38100700002)(8936002)(54906003)(478600001)(2616005)(316002)(66476007)(5660300002)(6916009)(66556008)(66946007)(956004)(6486002)(6666004)(103116003)(107886003)(8676002)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Hz/EDZM/Hi7VjRGjdigHm4vXI4v3Xj0cYS6OY2ZnGu37SatAXTFVBz6QOMxbf2gMtwCLvRQJRWPACw/OzmWsHcjGx1a2rDnmK27X1+A8WwEa5PcKmXr0gnOfui6CXGUzGKBo/6pLajlSkPJLkd9M2xJMTbhIBYhf4yzZrPkyOpIdnUI876raydPcu/RkJjDbR/KGHxi6GekO+LZ5q0brwhIfCWboPOIzr0Qeca+O0xPGPKLmGwRABS7up+UxqENyh9s9vLybykUlTZA/E7lkn90q640cdpuxBssGVcgFSKTrl48MZhW9ZEykmPtlA91CgIU+x5wswlNbma0SAIpY0NUtI6OFHcB41x7Ru+/T8v4DRGh4376GYIeIEVf6TNqRDVsiZwjcjN2Eg974vWxS5KVQSbxXf1O4nBFb1nHhhI0dLq4iOfXHJLHq4muPfDocfVXSKtpVoEyMA3ZqDQf5MmY+PHUjbS8c+/eghHT3ntolFI+l6A45CppAPImtm1bvNtRnvEPLNcZkNdrugvQbDUuMnNliFoZr2ArTFyUS0fA9sqso0348RqGlQXPrOdHDzl6MA0JDdmxEgOgMNU8/fXXXh4zCnSOx3i8Lz3nNIJIYgKR36BqVslQzNF3BvQRW09KfHqOKVSfZE65tQL94HR9rxDlpm+bGV6FXztq/3o5eGvuAAaVjEtPEadsFJWcbmTwiN31MZSeprDB/IXmB+urqvmxFnsZFcml5jLWdIen0Ma37oaHs+k9popczlCEfSuEoi1ORKpNtz8lkS09/q+EsnAio58bj3wp5h9oe6fACgZq7quGrfZ4n+XKHXNs1MZZ3+wX8zznIHw4P9sA3gr/+cR0F1gsuCrhyVKUN97+2zW7w+qiBvb8CMytDD2c23d+UXQ/kc9lUyO70d3bsTlCKtvHKwmwXYLSa1KS4qyZasOV1QbkhV6TgCHuqjdpuTCS40TFU/EKZJePeQSKby/Hyl16GAXrExyHbyVf0oTgvgSp5LwHe5tdl88QBaHoZF0R91ZTwbVtOk55YJghdxsp9anMYyBe7vDB19PWb8QO5a8YOg9njvl+suasXTJ5HaxbUNsAG7BipVm8Rg/oKT7OeG/NSOOdQn/qSz0lwFr11jwd1wfcI0UnCDgDGMxm1vZvmAYCHbQVBVAds+ZcjhHK5nqnT+pRu1j0lRsBxm3SRElI+Ril/IgCIKamwArQATTeGPg6+A1Bp1yOlALNRvLeflzhHHJDmL/qCCmmEjGVbQlyFoUL4ZQxxOWT4QiY1l4HKDpGy78tw3JneR2QoxkSrpBPBYAwHWKnNxgsjfM0mlfvKC/OdIMI+JeDNiab+ X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 053bc3fe-8d08-47ae-8ec8-08d931c01de9 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jun 2021 18:45:45.6636 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: T9yaj5hzazF1xn2p0OATHa8o0z2Q83H3lj0+dPJldCuw7M65ovSbGD3FXYR78V44aO57dd4TqLd24kA9dfBIlVkGEPRgOAJpmedgvM869M4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR10MB3983 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10018 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 mlxscore=0 spamscore=0 adultscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2106170117 X-Proofpoint-ORIG-GUID: WanCUzMmarJPVCx2-eXlyREwxQx7x4ia X-Proofpoint-GUID: WanCUzMmarJPVCx2-eXlyREwxQx7x4ia Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=UwWbrL60; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=Zj09Qt9m; spf=none (imf25.hostedemail.com: domain of joao.m.martins@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=joao.m.martins@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: ABB02600014A X-Stat-Signature: eq8khjjj3uh7afcrazwpt6dr36gfu4ef X-HE-Tag: 1623955566-977365 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A compound pagemap is a dev_pagemap with @align > PAGE_SIZE and it means that pages are mapped at a given huge page alignment and utilize uses compound pages as opposed to order-0 pages. Take advantage of the fact that most tail pages look the same (except the first two) to minimize struct page overhead. Allocate a separate page for the vmemmap area which contains the head page and separate for the next 64 pages. The rest of the subsections then reuse this tail vmemmap page to initialize the rest of the tail pages. Sections are arch-dependent (e.g. on x86 it's 64M, 128M or 512M) and when initializing compound pagemap with big enough @align (e.g. 1G PUD) it will cross various sections. To be able to reuse tail pages across sections belonging to the same gigantic page, fetch the @range being mapped (nr_ranges + 1). If the section being mapped is not offset 0 of the @align, then lookup the PFN of the struct page address that precedes it and use that to populate the entire section. On compound pagemaps with 2M align, this mechanism lets 6 pages be saved out of the 8 necessary PFNs necessary to set the subsection's 512 struct pages being mapped. On a 1G compound pagemap it saves 4094 pages. Altmap isn't supported yet, given various restrictions in altmap pfn allocator, thus fallback to the already in use vmemmap_populate(). It is worth noting that altmap for devmap mappings was there to relieve the pressure of inordinate amounts of memmap space to map terabytes of pmem. With compound pages the motivation for altmaps for pmem gets reduced. Signed-off-by: Joao Martins --- Documentation/vm/compound_pagemaps.rst | 27 ++++- include/linux/mm.h | 2 +- mm/memremap.c | 1 + mm/sparse-vmemmap.c | 133 +++++++++++++++++++++++-- 4 files changed, 151 insertions(+), 12 deletions(-) diff --git a/Documentation/vm/compound_pagemaps.rst b/Documentation/vm/compound_pagemaps.rst index 6b1af50e8201..c81123327eea 100644 --- a/Documentation/vm/compound_pagemaps.rst +++ b/Documentation/vm/compound_pagemaps.rst @@ -2,9 +2,12 @@ .. _commpound_pagemaps: -================================== -Free some vmemmap pages of HugeTLB -================================== +================================================= +Free some vmemmap pages of HugeTLB and Device DAX +================================================= + +HugeTLB +======= The struct page structures (page structs) are used to describe a physical page frame. By default, there is a one-to-one mapping from a page frame to @@ -168,3 +171,21 @@ The contiguous bit is used to increase the mapping size at the pmd and pte (last) level. So this type of HugeTLB page can be optimized only when its size of the struct page structs is greater than 2 pages. +Device DAX +========== + +The device-dax interface uses the same tail deduplication technique explained +in the previous chapter, except when used with the vmemmap in the device (altmap). + +The differences with HugeTLB are relatively minor. + +The following page sizes are supported in DAX: PAGE_SIZE (4K on x86_64), +PMD_SIZE (2M on x86_64) and PUD_SIZE (1G on x86_64). + +There's no remapping of vmemmap given that device-dax memory is not part of +System RAM ranges initialized at boot, hence the tail deduplication happens +at a later stage when we populate the sections. + +It only use 3 page structs for storing all information as opposed +to 4 on HugeTLB pages. This does not affect memory savings between both. + diff --git a/include/linux/mm.h b/include/linux/mm.h index bb3b814e1860..f1454525f4a8 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3090,7 +3090,7 @@ p4d_t *vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node); pud_t *vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node); pmd_t *vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node); pte_t *vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, - struct vmem_altmap *altmap); + struct vmem_altmap *altmap, struct page *block); void *vmemmap_alloc_block(unsigned long size, int node); struct vmem_altmap; void *vmemmap_alloc_block_buf(unsigned long size, int node, diff --git a/mm/memremap.c b/mm/memremap.c index ffcb924eb6a5..9198fdace903 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -345,6 +345,7 @@ void *memremap_pages(struct dev_pagemap *pgmap, int nid) { struct mhp_params params = { .altmap = pgmap_altmap(pgmap), + .pgmap = pgmap, .pgprot = PAGE_KERNEL, }; const int nr_range = pgmap->nr_range; diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 76f4158f6301..aacc6148aec3 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -495,16 +495,31 @@ void __meminit vmemmap_verify(pte_t *pte, int node, } pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, - struct vmem_altmap *altmap) + struct vmem_altmap *altmap, + struct page *block) { pte_t *pte = pte_offset_kernel(pmd, addr); if (pte_none(*pte)) { pte_t entry; void *p; - p = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); - if (!p) - return NULL; + if (!block) { + p = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); + if (!p) + return NULL; + } else { + /* + * When a PTE/PMD entry is freed from the init_mm + * there's a a free_pages() call to this page allocated + * above. Thus this get_page() is paired with the + * put_page_testzero() on the freeing path. + * This can only called by certain ZONE_DEVICE path, + * and through vmemmap_populate_compound_pages() when + * slab is available. + */ + get_page(block); + p = page_to_virt(block); + } entry = pfn_pte(__pa(p) >> PAGE_SHIFT, PAGE_KERNEL); set_pte_at(&init_mm, addr, pte, entry); } @@ -571,7 +586,8 @@ pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) } static int __meminit vmemmap_populate_address(unsigned long addr, int node, - struct vmem_altmap *altmap) + struct vmem_altmap *altmap, + struct page *reuse, struct page **page) { pgd_t *pgd; p4d_t *p4d; @@ -591,10 +607,14 @@ static int __meminit vmemmap_populate_address(unsigned long addr, int node, pmd = vmemmap_pmd_populate(pud, addr, node); if (!pmd) return -ENOMEM; - pte = vmemmap_pte_populate(pmd, addr, node, altmap); + pte = vmemmap_pte_populate(pmd, addr, node, altmap, reuse); if (!pte) return -ENOMEM; vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); + + if (page) + *page = pte_page(*pte); + return 0; } int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, @@ -603,7 +623,97 @@ int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, unsigned long addr = start; for (; addr < end; addr += PAGE_SIZE) { - if (vmemmap_populate_address(addr, node, altmap)) + if (vmemmap_populate_address(addr, node, altmap, NULL, NULL)) + return -ENOMEM; + } + + return 0; +} + +static int __meminit vmemmap_populate_range(unsigned long start, + unsigned long end, + int node, struct page *page) +{ + unsigned long addr = start; + + for (; addr < end; addr += PAGE_SIZE) { + if (vmemmap_populate_address(addr, node, NULL, page, NULL)) + return -ENOMEM; + } + + return 0; +} + +static inline int __meminit vmemmap_populate_page(unsigned long addr, int node, + struct page **page) +{ + return vmemmap_populate_address(addr, node, NULL, NULL, page); +} + +static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, + unsigned long start, + unsigned long end, int node, + struct dev_pagemap *pgmap) +{ + unsigned long offset, size, addr; + + /* + * For compound pages bigger than section size (e.g. x86 1G compound + * pages with 2M subsection size) fill the rest of sections as tail + * pages. + * + * Note that memremap_pages() resets @nr_range value and will increment + * it after each range successful onlining. Thus the value or @nr_range + * at section memmap populate corresponds to the in-progress range + * being onlined here. + */ + offset = PFN_PHYS(start_pfn) - pgmap->ranges[pgmap->nr_range].start; + if (!IS_ALIGNED(offset, pgmap_geometry(pgmap)) && + pgmap_geometry(pgmap) > SUBSECTION_SIZE) { + pte_t *ptep; + + addr = start - PAGE_SIZE; + + /* + * Sections are populated sequently and in sucession meaning + * this section being populated wouldn't start if the + * preceding one wasn't successful. So there is a guarantee that + * the previous struct pages are mapped when trying to lookup + * the last tail page. + */ + ptep = pte_offset_kernel(pmd_off_k(addr), addr); + if (!ptep) + return -ENOMEM; + + /* + * Reuse the page that was populated in the prior iteration + * with just tail struct pages. + */ + return vmemmap_populate_range(start, end, node, + pte_page(*ptep)); + } + + size = min(end - start, pgmap_pfn_geometry(pgmap) * sizeof(struct page)); + for (addr = start; addr < end; addr += size) { + unsigned long next = addr, last = addr + size; + struct page *block; + + /* Populate the head page vmemmap page */ + if (vmemmap_populate_page(addr, node, NULL)) + return -ENOMEM; + + /* Populate the tail pages vmemmap page */ + block = NULL; + next = addr + PAGE_SIZE; + if (vmemmap_populate_page(next, node, &block)) + return -ENOMEM; + + /* + * Reuse the previous page for the rest of tail pages + * See layout diagram in Documentation/vm/compound_pagemaps.rst + */ + next += PAGE_SIZE; + if (vmemmap_populate_range(next, last, node, block)) return -ENOMEM; } @@ -616,12 +726,19 @@ struct page * __meminit __populate_section_memmap(unsigned long pfn, { unsigned long start = (unsigned long) pfn_to_page(pfn); unsigned long end = start + nr_pages * sizeof(struct page); + unsigned int geometry = pgmap_geometry(pgmap); + int r; if (WARN_ON_ONCE(!IS_ALIGNED(pfn, PAGES_PER_SUBSECTION) || !IS_ALIGNED(nr_pages, PAGES_PER_SUBSECTION))) return NULL; - if (vmemmap_populate(start, end, nid, altmap)) + if (geometry > PAGE_SIZE && !altmap) + r = vmemmap_populate_compound_pages(pfn, start, end, nid, pgmap); + else + r = vmemmap_populate(start, end, nid, altmap); + + if (r < 0) return NULL; return pfn_to_page(pfn); From patchwork Thu Jun 17 18:45:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12329365 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3138CC2B9F4 for ; Thu, 17 Jun 2021 19:32:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A14C16113C for ; Thu, 17 Jun 2021 19:32:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A14C16113C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3468C6B0070; Thu, 17 Jun 2021 15:32:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F7B76B0071; Thu, 17 Jun 2021 15:32:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D4146B0072; Thu, 17 Jun 2021 15:32:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0202.hostedemail.com [216.40.44.202]) by kanga.kvack.org (Postfix) with ESMTP id CB5BB6B0070 for ; Thu, 17 Jun 2021 15:32:30 -0400 (EDT) Received: from smtpin32.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 704501E4E9 for ; Thu, 17 Jun 2021 19:32:30 +0000 (UTC) X-FDA: 78264212460.32.86F285C Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf01.hostedemail.com (Postfix) with ESMTP id EDC5D5001700 for ; Thu, 17 Jun 2021 19:32:29 +0000 (UTC) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 15HIZal2004683; Thu, 17 Jun 2021 18:45:51 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=h4oUpIC84MOA73I32hlt5/11IyNT7LrSD6rSEUoSg5w=; b=En1HLGr1VhFocaQMWA6TjNtLXKCXrEG7QmfRDl2iIZxNlIgQDqmq1/9DoxQsh1qsFK5C mVROdP3IY4J2U8MdGUt0S8u5+ESkleH9VT915TT9m2FMcDYJiwrdF7ubbwmAetYc0BuZ RZR8u2fe0Wn352Ly9JE5mRLCiIRZg4go3VzrTCKedMSK4cd9ai8a7KK2BuVRARnQZ1K6 SnbJwgF1XcSrZldi5GfDNBNSxdWSVhhAN5Bfe8Xgj2wzbDjl1k8NNwi0k0um//BHvAVB H7fRasiNgUHe328uxcqUkXizWfuulUyC9eF4eJ1AfbvPDfgg9ZjUTrY/ZhyGH5We/g0N oQ== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 39893qrcfd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:45:51 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 15HIiudu160858; Thu, 17 Jun 2021 18:45:50 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2172.outbound.protection.outlook.com [104.47.59.172]) by userp3030.oracle.com with ESMTP id 396waqf6x4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:45:50 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hErx+VhZxcbZTjKbVwB47xOde/DFZ4LBzpiJ5X9do5PVXAWYqmSKreQ1CHVji+ADF2Pk/6TsEMyGBh4QaUXVBtcfu7gjxk/mOQFS1IPONhUUFnmsT7cXYDBWPraSOdoigq5/F+DH8LcSYxB+Hur0ljJMB/35/sVmhI5woRuu6Ts6xIYTUBIC2bwZXm0zpT4BhGmorNk7DwjJa+7H3Z0epTU96NLXE/tSGQzNy0TWJ994xSUFVyuc06y8v9mLob6LFPMrtELuS0NrctKpgKnsGFzvNM5xb+D7kzIUUx7X0e6Zt/Vhtv8QW0pQKE3we5Y6R7zdlSTxmckBYZaQ94yt1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=h4oUpIC84MOA73I32hlt5/11IyNT7LrSD6rSEUoSg5w=; b=GBHhS4gHEnUFpJGmY5ZbqjKSbRNOfnlv8hhIs0loclEq9z9C3ybnOUleJcTW5EccxPcpGYPA77976FXaB++/ZPVlgU6a/eriS2nk4YbrqZIKmujONZUctCiSB3rSUoLTxPWwgns/k5YHm9P62BrMoEfVjKIUFzNPzwrIW8dfFlp3sFIBFibMbfM0hdYzH1AfHzdXB8a4srrJO8ZU3SGDRzMM3oHpH5/iFbU5MjrXqAvXM9jlrZaF8ItZagkpocPcpKYKLX53F+XUvdAMOxGlQgtxLPsjtIdIC6LHWT1P2OklurJFum9Gb7N12MchThNG3Q4I3oxK6m5GTVgP2LKl+A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=h4oUpIC84MOA73I32hlt5/11IyNT7LrSD6rSEUoSg5w=; b=tL2CV437eQZpDCYFXI+2XfLK3GXtmck4KP8OLvixlIcAxwyEybb/w7Aj57appgvkuDXSNHhqhadnZJZFFR7IrxqNkYlD6t7rvlcVfg/0JrPyAqFYoGOcBGAkLsau73kjAD/8SccNgR7H2uZj/CYL5mqbulU3AwZvQNqSjTF+2ns= Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BL0PR10MB2913.namprd10.prod.outlook.com (2603:10b6:208:31::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.22; Thu, 17 Jun 2021 18:45:48 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582%7]) with mapi id 15.20.4242.021; Thu, 17 Jun 2021 18:45:48 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v2 09/14] mm/page_alloc: reuse tail struct pages for compound pagemaps Date: Thu, 17 Jun 2021 19:45:02 +0100 Message-Id: <20210617184507.3662-10-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210617184507.3662-1-joao.m.martins@oracle.com> References: <20210617184507.3662-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4242.16 via Frontend Transport; Thu, 17 Jun 2021 18:45:45 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d6cb6b75-86b0-4f9b-2b7b-08d931c01f62 X-MS-TrafficTypeDiagnostic: BL0PR10MB2913: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7219; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: M9lAJS0g7QhBnHWmtkPYjCyQyRkRiMsb2mVyBLzhsSOHJeT4gb12iHBGa30OU8KA8Ysm6KGW34z2QxqFyhIUv7w8A4ewv+RjAE6UZeVOyMGzhWquMHQgJnE1Ygh84LF2R/PnZLsrZZAV7KLTDaXbskAibXV9rICU2sjEn5k04Iu0uHU9SicN7E0uoiVVItT2SgkmFotHmjZTb1xs5gsfeeiRY5XrwRp6rsLae6i7n5QZRdMtr6grMm32JxPqMGqXX9k5NBML30LF/cFT7mkKjXnJ0N8C/+ShQa2zEc4y/iWQOUIlOXpl4f1fsyZjdOlahde1+BOom9z1Plvz2dmD6nTUu7r5BWlNoq9Ad5+3m90FkKdV0f9UyX62F7w6uyzOM+4Xy6orJIlkH4uPThdp8V0voxdbzijQ9Uru3BlaZ4vxMDrIzyLIpT3gp69TQH6Az91KIv6DxebnPFiIogEfGiWaEVUJyaIWxwK4G26IbcFGWPZc1DB52UlnBV57HpaAASGYLgSyow4xsW1jsip/dEFj9ge2HvqHAoUyu6cdEViTsdf3hVSZqofmn2H1YkaBLRNkeFaPItbtPP8dwNhbzDb1LKMN3kIHSMqp4k6HUi80nljIXaD4l4zDPRLltgLgcLwatNKQDXlBguiUfrXzng== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(376002)(346002)(39860400002)(396003)(366004)(136003)(478600001)(16526019)(66556008)(52116002)(66476007)(38350700002)(86362001)(4326008)(38100700002)(186003)(36756003)(7416002)(6486002)(54906003)(8936002)(26005)(83380400001)(2616005)(6666004)(8676002)(7696005)(5660300002)(103116003)(316002)(2906002)(956004)(6916009)(107886003)(1076003)(66946007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: oCJr+sIe4VvV8zzyuryMm94HDTSLsT3lvIoCNXe3vpk1KAvDk75QrHg72OFHJrUwahX+zlRZVWIkJHgDZVkhr5ZAosflHDN+IOBfrkT9K6fCqmvIws8mpmKG3+Vybb+SLrsPgvAzutvB8y8nXPcaIVSGOD3GUrmlvfWP3uMgG7AOMr1SR7yoC6NYWJmh9pCBIWNREaD3imox39iRnq2GATmVJPZ4laIzutwkEdQwO0YOMVWMBTWskjLWmSVPpFe8bwp7s5VrhHi6MWS1r+4YZI/WTl09dRc5xgHnX8R/7f5ChW1XGAbKsO4eYgtg/SHnUrC7NP6hdVWC/puLkjqAKDdI8n2dzb3nxgtwpgfoQAma9qLBROUyfRDe904uhUjesHIUc5bBUyqQ5nARCIetkfhb0NTP53UrM1EtOjj0hXT6RUQOfVpPECahuI1xhAMOw/SiXYhARPFPA10nRbEr5I9eibNaFO9v5dkAps35TkXSXXtl8dzWo//ctx/z4+oacmDxqLL9kRM62yuNcYJkRM9Hu0zRG3fJqNVCANBhD0VJk+5uQqUavfRAElTam/QqaPJqB5othZJLzHWANF77cwE3r0qF8VGtusF6EbxKMsBaGmbH4+WEAfqmMYMySHLm3nH8A9VQaVvb15CMKpQ/X15mWC89TMJeANhmYTvSz6mdz4yZOPuUhg97XLeW63ygvkPqVKPmcf7RVR4hkCK59YNoM63rc80FzduD0Z2CI0RJJJ4lXZiDqDkXUj73JUZsY6OMtP6+o1Fx2XkAJvPC9yXs8GJPkjuhcxLBlIVtwzg8aOVsZh1yz+nPocwTL/4stgYnh3YOq6LdX+Xa9aU5ZyXjrfaQaeJ5yNosyI1987luwwfA5GwQEuYErFua6WzVIwa3/ukwwS72PYvUe0+YL9h5DLTRiszhXsfsk6a7iKMzAgFtd0l8e487UPBp2iv57mBCLldC19N66PeIF6arpiybYu/CV5jbFvQP/DEp9leO58aSSiTa4SslrFZKqnnKgWxzUfDvcmNYpKOjJQwAJJrmlh/xZc4vIQHLaeXRGoz2UkEcdj3kFBF6hPDkpK80f+2nOTFuHiPPOpw2cRt56bgOvmAY1WMjoAWCqAbXpFQ2vaEBjlJ/4Cbz6dcTv7hqDDMQNlZCQbmBqxKW+Oq48iVTdejtpoDy0VL+krNUDfqFM2BZBf3FNg87Y5vTI9dKCmKLATEQIEvtNnyp8sW/ij/Aagra8JPbrgzUumpElhnpIdatlDmjJym4BEiVnO8+eFPVPv8tjF2oVQnWAgyYP9WBxpiCOwnzFl0NRKojWpNdI9fTtB4GhZWllSvq0Y0Z X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: d6cb6b75-86b0-4f9b-2b7b-08d931c01f62 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jun 2021 18:45:48.1218 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: r2jZYt5RQ+4gYruqoNdkDfYjbaAzA2AfySt3D5VWx3Do/67cV6nAuTq2WGmPo/ZW893Us0cxFagPUFJnR36wj7rETME1DvbtrWcyFYevWGA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR10MB2913 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10018 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 suspectscore=0 mlxlogscore=999 spamscore=0 adultscore=0 bulkscore=0 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2106170117 X-Proofpoint-ORIG-GUID: crXw_IrK-WZYqnhCWtvz-7I974ybnDTt X-Proofpoint-GUID: crXw_IrK-WZYqnhCWtvz-7I974ybnDTt Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=En1HLGr1; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=tL2CV437; spf=none (imf01.hostedemail.com: domain of joao.m.martins@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=joao.m.martins@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: EDC5D5001700 X-Stat-Signature: z1wbnz8ct6rsirmi3ec7a3kk7k5yrmtn X-HE-Tag: 1623958349-311560 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently memmap_init_zone_device() ends up initializing 32768 pages when it only needs to initialize 128 given tail page reuse. That number is worse with 1GB compound page geometries, 262144 instead of 128. Update memmap_init_zone_device() to skip redundant initialization, detailed below. When a pgmap @geometry is set, all pages are mapped at a given huge page alignment and use compound pages to describe them as opposed to a struct per 4K. With @geometry > PAGE_SIZE and when struct pages are stored in ram (!altmap) most tail pages are reused. Consequently, the amount of unique struct pages is a lot smaller that the total amount of struct pages being mapped. The altmap path is left alone since it does not support memory savings based on compound pagemap geometries. Signed-off-by: Joao Martins --- mm/page_alloc.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 42611c206d0a..cf4c2cd32874 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6608,11 +6608,23 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, static void __ref memmap_init_compound(struct page *page, unsigned long pfn, unsigned long zone_idx, int nid, struct dev_pagemap *pgmap, + struct vmem_altmap *altmap, unsigned long nr_pages) { unsigned int order_align = order_base_2(nr_pages); unsigned long i; + /* + * With compound page geometry and when struct pages are stored in ram + * (!altmap) most tail pages are reused. Consequently, the amount of + * unique struct pages to initialize is a lot smaller that the total + * amount of struct pages being mapped. + * See vmemmap_populate_compound_pages(). + */ + if (!altmap) + nr_pages = min_t(unsigned long, nr_pages, + 2 * (PAGE_SIZE/sizeof(struct page))); + __SetPageHead(page); for (i = 1; i < nr_pages; i++) { @@ -6665,7 +6677,7 @@ void __ref memmap_init_zone_device(struct zone *zone, continue; memmap_init_compound(page, pfn, zone_idx, nid, pgmap, - pfns_per_compound); + altmap, pfns_per_compound); } pr_info("%s initialised %lu pages in %ums\n", __func__, From patchwork Thu Jun 17 18:45:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12329321 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9F2CC2B9F4 for ; Thu, 17 Jun 2021 19:14:04 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6EDB06109D for ; Thu, 17 Jun 2021 19:14:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6EDB06109D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D1F3C6B0070; Thu, 17 Jun 2021 15:14:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CCFE56B0071; Thu, 17 Jun 2021 15:14:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD3B36B0072; Thu, 17 Jun 2021 15:14:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0236.hostedemail.com [216.40.44.236]) by kanga.kvack.org (Postfix) with ESMTP id 7B4FF6B0070 for ; Thu, 17 Jun 2021 15:14:03 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 0D72A82CE584 for ; Thu, 17 Jun 2021 19:14:03 +0000 (UTC) X-FDA: 78264165966.18.836F982 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf14.hostedemail.com (Postfix) with ESMTP id 77656C00F785 for ; Thu, 17 Jun 2021 19:14:02 +0000 (UTC) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 15HIZdKu004692; Thu, 17 Jun 2021 18:45:53 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=o13ytpm6INie/NN14mwJls2enUk1cPtSh7jEJOPQtuw=; b=yeRiOTka9gghx599O4IMxKCeJov663LZcljOt6wSRFVn/jjICFrvryr6nfQGKQt+PDBw NydyK05lY6gxQQcH4HGjtuu7cpDEuNkwyIYtSD1e7aAUFnC2nPhvpedtcRn62xlNW6mb xvbHEih9l8z/WiJpbOW3yxtpS6jJ7JRrTT5WZtEKU75dHGCjDukle7HEkENxXGbBlJMj d3y/MMSCkRWjsx3QxP9jBuDbirm7tpm0NNeKi+nykR31weANqstV+XoA6jAkjP/eG2pP i8dIowq4yoHM9KJOHteCH71aSpg5+Tby2VB2/2s0Zbs0v3zXmjQ5/ah30iEH94KLFWpm aQ== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 39893qrcfj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:45:53 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 15HIjNhL165530; Thu, 17 Jun 2021 18:45:52 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2177.outbound.protection.outlook.com [104.47.59.177]) by aserp3030.oracle.com with ESMTP id 396wavvrae-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:45:52 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OCNOnGd7ysqlmuOELnTTYd9YeNbBxhEqrSHdJOzKY+H6Bd9L3gg93yw4/ZUBk5zCZJHDC0zzFEwpQ4AhmEt2uw/KRBMXc1Mo2qPzVtxPZuhvKJxdsdXTEbxV7tPZyhyJ6tVK/MyjwnqUXK23KUiduiaiHj1JQ/VzvTbaSkL7NZBlESZl3eYhNaYdpsKifBMM2LppOFqresFDI3XRfIKh27cP23p5EBaQx0CHIXPO/3onoACOp6MsiSnvOMXLlZS2EXFHZxL+lpwOv3C/0wdw5w9pyxPe4fEurBoRgXv/0OpTiwBNI3GtuRCMaj1QHKNHGl6iKF+MtOQYH4D8yBJ6dQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=o13ytpm6INie/NN14mwJls2enUk1cPtSh7jEJOPQtuw=; b=eM8W0DyBTmFdICFdHzwWQi9oEcSuGtkzNgUBeoqhaLwTOrr4p/V82h2FXKNWpNo2mv6SQZ3a4vjWwOSHfbUZZAO0SQ958l/H9/c1tHo5go5eEsCtMMyOCi6yyPO3MCzT/3f+vm8Cs8lbvr9F2B1uITTfNI6Xx+/3dYj1FUnhQTmorsUSDeSWUjrnFu7kRuWCdxkl//C90qkb7pR403UxjOKBwBLrDj5UYGO9kouw7ryvQgx32X0dsNdVV3xz/XGLghPZAndiEDirzhZOVbuOACZmC8PmxAjMM+j2ULzYikRlb60tsiFGc/UM4ERs9xTaXo+OYwpS7rwq6UgWnZ3h7A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=o13ytpm6INie/NN14mwJls2enUk1cPtSh7jEJOPQtuw=; b=YPJvn/ebQyb0lcH7Nr+UIec7S7/8Kv4b3AO1q3nEIK99mO4WTRdjAREnhb+o7y/wI660zxRAc6X+aJufOyaFyExrft95Y8vFoMdyH/3A+S4OgJo2m2p3SjEx9Yu8BFLpaG0eTGjO4q79u/OhWBOC8bzMgQJRzmDe9rcJJ2p99vU= Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BL0PR10MB2913.namprd10.prod.outlook.com (2603:10b6:208:31::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.22; Thu, 17 Jun 2021 18:45:51 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582%7]) with mapi id 15.20.4242.021; Thu, 17 Jun 2021 18:45:50 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v2 10/14] device-dax: use ALIGN() for determining pgoff Date: Thu, 17 Jun 2021 19:45:03 +0100 Message-Id: <20210617184507.3662-11-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210617184507.3662-1-joao.m.martins@oracle.com> References: <20210617184507.3662-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4242.16 via Frontend Transport; Thu, 17 Jun 2021 18:45:48 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 1aefe263-47a2-4756-df35-08d931c020e5 X-MS-TrafficTypeDiagnostic: BL0PR10MB2913: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2512; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8aQi4nNRD5m6xmhhwETYTilR9ebhPYc++YripbFHCaMLVEeaNdhiKP7+xhOxy9wMrZRGL+RrkJI0ZPujGgw3hYdVUhBnSkr83I187/aTCkHNDvJCD+fKaiUxDvcA66w/Hq/34gfSxNaPQphTeU280+mfswv2WrCNvxForBa79uoF1LuyRlgGDXrgezYJFjhvajz+Cx7R1TFB21lIcjXwmyrsfHXgybFaJON4uQGlrDs4KCUVKQb4Kg6SGj4qFqA/NIM+HrAQ2CYjBy/dGR8sBye+zCwwz8GsYZYE116QvNpF4/rHQrzdHoAZuB+2X/tiE2qq3jknP4N3diqGLZ7EVOUDnlnAJWk0n6Ibh/KQ1qunuKA2V1XdlAcqGvHDfraDBslrU5xdCkik7UbkmxfSDWj11J78R4bZzdoRUz0WUoogrZ1FCHD6HNCG0W3JdtP1POQqeAH9KMLTyuYptE9Ikkyk+1qOK0MkZbVXOqJcsvgczg+ThKM8KGhS5cMLKAtQi7JXCf0fQYuTvfkiUQP9oKXEH8oWawHXDApu6Bv5auEVztdk2u/ZXLv71tvbY8eLJyw4/wAN7BRsALYMEuYsSBVaRbI4di/7V1wWnx8+wuVwi8/AqY8FA3BsShRXYAFVdYAETdO9xA7TsPtu1QVLxpxv70LuYx4dW0MOxVpsOzw= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(376002)(346002)(39860400002)(396003)(366004)(136003)(478600001)(16526019)(66556008)(52116002)(66476007)(38350700002)(86362001)(4326008)(38100700002)(186003)(36756003)(7416002)(6486002)(54906003)(8936002)(26005)(83380400001)(2616005)(6666004)(8676002)(7696005)(5660300002)(103116003)(316002)(2906002)(956004)(6916009)(4744005)(107886003)(1076003)(66946007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: tkbFtif5AOYyDz+GHWaPzqh8W5lTU4g9BGk+95LLqUTTgveSeGUiEcgbkI0DEmI00cDEOiTNaOOxuLdEgh7SbCXkIF6/w8H/tHTSW5E7/MYQL4yFwgjqUlw/xEWYFFysg88tvTGXaktbPqzMmVwm1pDTr8XtsF0FuEekoiB94bRr+XwcyZFmX4SqHaI2RPZ62DGeIv6+CivcLeMTJlnLShEziB4CrWMucCWjlrlJ0c98FhAzm/dZtdIwlAzFHZCxSnu8aIim8kGy4tMlGnj6G5fQB7kBJD4vMMK9qklTqFXbrTqoSoyK6yPCJEBHBWomf7Y/6+7EN5Rcx49O7TNckaPWXKFV/VynzCQMXAZy39XDchDH+ShHTkd74c7BSbrEmDTl7HjZZLXgl5B++P8WLk/QjZUwaDTV9pq3rGOZMwnCl7KgIGLwxxD8FeVFUebSCJuPz4JHLlXPB4qy465qAONKzoHHtojgiTPGMPCsOH6tdiB+F9g0RuNBMkEcgEqyrksrxF4PJGQUHXqwH2VCpLoqL0mVImInDTwVKUb4sloSPDINb7+1aW/1o8nWKWnKiNG+uXh3MYZ3yhLm0ulYuo6rsNBBlGY9PKRYB2tOIpmKQRAfSWXuYfmjdq8tQ2O9aoF1zWtz9cNgNnz4whWENjsx1hMyhYuQWWzvIjiMZeZ65brAJMAlZPkjpwqVuRCE07lO+h3bnvQftI0SI6JNi3dikaG3+j5DtVoHcbwktYiZgPRRgJqgd6OMAJ58GHrxZ/M8jUkPYtDiznTRO38PemoKc2bYlsv18o3D95atybWgv8zG0nlZ4Bjcef7TkTIoUYv90w45q9NP78yPE5tUc/gKAitDOaVQECFyRd9DUTdYJoVrbUGGkxhAMsTbyb/S40rsFPJZ3CrWs+N3Av4mRy5FmvH0N1/NvoMsDqSo8OaMikcs0qKzLgxSA1Iq1kGN0iwTkNiUgS3MRKwp1qthnxflyQjA4C7n/DyUIi9Hzu5nHa4uttjVSSwQNch1EUtxvoWljfeSm7zk+tqIbD8lpN7pLZXCnvhpvAL3HSGwxekMrXnqNq7jjLyrhx8pBhJ2NzYVc1pUlfY7N5Ka5wDUf1fVXs2kJm+3G03Ev26lGP6lSXB/MXAXO3F+cD4ixxfhHCH+c3imoevSxdczjBQcWn0kl3TMGbqi1/Gtfk8rpBIuxO/jPHNSxKOoiBsZ+bOrmLYU7xQXeOXWPwzZskJNYiBXitMoaP4PkxjZyfv5PJcgHj3WXPpzJR9ZLV6+nWuqfhwZILygW51Ga0wBW+jk0DH9pJS80NbOW3lk7ic4VDtbdDa6bIMSk83KrwmMMl0n X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1aefe263-47a2-4756-df35-08d931c020e5 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jun 2021 18:45:50.7144 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: EoPDRMNqoHnTc0fKOpTg0WMtZG+jaJx4PMJs4znnJqun93Zjoxm617qiZQsO02JEURoqJ9KYitN6/VsQOHFi1BzkGX9yHGd3B7G4LJUVedo= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR10MB2913 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10018 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 mlxscore=0 spamscore=0 adultscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2106170117 X-Proofpoint-ORIG-GUID: SkepR_0yNaOZY_O6w7se397tXSwlkcd5 X-Proofpoint-GUID: SkepR_0yNaOZY_O6w7se397tXSwlkcd5 Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=yeRiOTka; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b="YPJvn/eb"; spf=none (imf14.hostedemail.com: domain of joao.m.martins@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=joao.m.martins@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 77656C00F785 X-Stat-Signature: 47csbsd7z66muyh68shmarkxkmphdgrc X-HE-Tag: 1623957242-29286 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Rather than calculating @pgoff manually, switch to ALIGN() instead. Suggested-by: Dan Williams Signed-off-by: Joao Martins --- drivers/dax/device.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/dax/device.c b/drivers/dax/device.c index dd8222a42808..0b82159b3564 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -234,8 +234,8 @@ static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, * mapped. No need to consider the zero page, or racing * conflicting mappings. */ - pgoff = linear_page_index(vmf->vma, vmf->address - & ~(fault_size - 1)); + pgoff = linear_page_index(vmf->vma, + ALIGN(vmf->address, fault_size)); for (i = 0; i < fault_size / PAGE_SIZE; i++) { struct page *page; From patchwork Thu Jun 17 18:45:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12329253 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16F24C49EA2 for ; Thu, 17 Jun 2021 18:46:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9AF2B613E9 for ; Thu, 17 Jun 2021 18:46:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9AF2B613E9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B43DB6B0072; Thu, 17 Jun 2021 14:46:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF6E06B0073; Thu, 17 Jun 2021 14:46:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8D5036B0074; Thu, 17 Jun 2021 14:46:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0173.hostedemail.com [216.40.44.173]) by kanga.kvack.org (Postfix) with ESMTP id 57E606B0073 for ; Thu, 17 Jun 2021 14:46:02 -0400 (EDT) Received: from smtpin40.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id DA312824C44D for ; Thu, 17 Jun 2021 18:46:01 +0000 (UTC) X-FDA: 78264095322.40.444DC90 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf29.hostedemail.com (Postfix) with ESMTP id 69C2E562 for ; Thu, 17 Jun 2021 18:46:01 +0000 (UTC) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 15HIauX1020433; Thu, 17 Jun 2021 18:45:56 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=XS+vp3u3f2gS+4FaUrtKudUv4lyVIJUIhs/+PkjZjxU=; b=lqLiOCUmg1uNyDejtbY2IqLPPqP6LKhDglKlAbW1EMkCZv6Bd/KzwCcKVeDqVoLv64Zp v91sD8WArOq1D/nVz8orc9OtmkCwRCRfMFJJYHis8LDMR//sKt4uk/fhxtl6YZNxG7UW 9wsHUZi4FA39NMmShfCgx070RafXPz59KWYgyMkrOZRhFqLUToJtfwkbj/G1Jk2hxVcS JINZWAE/bcxnpkZgwA9D3Yvbo8o4TLUFBk8ft/LvccukNRQYNDTCS6ziDxa3vG2ftnlG ImCA733lHrko3FjG4H3vUP4jc/kN7tGsr7SkmuZTNPExehUDdY0AWCmULJ7bOgqwAXLl Xw== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 39770hby0s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:45:56 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 15HIiunJ160797; Thu, 17 Jun 2021 18:45:55 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2176.outbound.protection.outlook.com [104.47.59.176]) by userp3030.oracle.com with ESMTP id 396waqf704-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:45:55 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=oY/hCn1W5JBD0SCjGK5qbE+mxqSirRZV1f7nhjyWRWd0FB0vP/QWpH/qcfjXqdMNxgyNY3B1ToLFE8OjT6ZLWW6XLxDsxyGhKzNvGM7gVZr4ODE7Rfi5ihhJbPckzGM02OHgwz3XtCd7AuNmXAQh8ClbcB1QETYbMkzbLxv7/mYP5zIiQhnyN+dNFbqpCYr+81jUu3gowubbr/evRGGzeRZ8H5Ui3g3orcj5O4RqwTZH6zWgvz0wcLxb+pLs2IEIVU+Yo/c1/x72DlXq4ZU+mW6aj+syMb3IeDqsZgUEIQ4QYsVXLLvcCTULVhbCisy/f4zcZNI5z3lFjleFGWlRPA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XS+vp3u3f2gS+4FaUrtKudUv4lyVIJUIhs/+PkjZjxU=; b=LWv1mAXgo89T2VLUKLqmVzAKdUVTnnG4ZjPpvnKaV2XCiUt63JVf/oUTPuVNE88lNwMLCN2Akzc+/B+9k0iUKqg+f2VoDUbAriiT635La80Hasl87pKBBN9mJGewyHzlLa3DGM7XV5lcDBB+2sXXL2DPvTEr4Al5+b+rDARe5nsHO7ONr6nSe9e4sJVm0046DGqja2WFNJT2xnpUvpykzMeBiLnhisZCuCbxeWwSQ8pnHf24UWzWoRe16SzuwgKQS9fLY8D/FFrCgpe1YMFf8D92MJw88LPANDE8xGmUQgfuIOpoaVPU/SE9+cAcYK2jHUfuzdVbcaThrq19liTseQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XS+vp3u3f2gS+4FaUrtKudUv4lyVIJUIhs/+PkjZjxU=; b=VAyWbVe0UBdvsD163JvOkiSR9pQWpjFg2TpTfy/YLsS387mf5e0P0QgOLMmHGF9XhAWVLMCR+++DMPYxow/Y6pVsu6ZwlrKMG4jSu9xB03BNQQFoSm6v4OvYFrjFKd2uuLHfRvm9kdDWkEy25vWgZKz0CxKELZpGvZ81iUZx9fQ= Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BL0PR10MB2913.namprd10.prod.outlook.com (2603:10b6:208:31::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.22; Thu, 17 Jun 2021 18:45:53 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582%7]) with mapi id 15.20.4242.021; Thu, 17 Jun 2021 18:45:53 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v2 11/14] device-dax: ensure dev_dax->pgmap is valid for dynamic devices Date: Thu, 17 Jun 2021 19:45:04 +0100 Message-Id: <20210617184507.3662-12-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210617184507.3662-1-joao.m.martins@oracle.com> References: <20210617184507.3662-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4242.16 via Frontend Transport; Thu, 17 Jun 2021 18:45:51 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 99433f39-a075-409c-1269-08d931c02285 X-MS-TrafficTypeDiagnostic: BL0PR10MB2913: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:854; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: PEXAbDdNvwRB6KbEzdLl+5AKl6XIKQqp+9AsLza3H68DwUUSePzO48tn8rmmangqOvPmW0IZMGIBANeLkcI0VezkoVT5FJNlRMTLsYR21wCKAGE57YP7y7jLqocqvLu8VMB9KuduAVCXWpM8VZCUbApDZZhgv4IBWA4si9dEJGsNJUU0hvCkEQ85BM0UrJa3w1CBR8SWAVPW1lDuRl2eSSBHmTVOAE9ZlcrjSKx2WFHvnlIxWaUGkFJ/RPAkZoZQAmK6zHDVFk6FFknNZEv7sTqUhEb4oFt4IruIyDsM3INn9VsPVdJBo2urLCRw1Y7QgS0u39xkmnc2oX6x6f+OLn52fHol7M1ddSz18WxbjRT5dO6bObyn03D6srWZG60840HVfJUmPu8TNamZo1kH1GYwTXoW5sfuf2NCUjFplUk9XuxxKnET+hsHMZbROfTEWM58cL3zYHIXSK8ydTDNGl3DuITt66mIWq5+qRe6YaXyG63tZe95KyeJmfASZR8drPOgUdRS7YQs2SUHlAHjFTLfGYAYQRCikU7VQ7QbUj/monbhcnxKt75AUEue+9s+0dsx0myWh4ZD8negftea5qOyeXlpPmgfPAhjC26HjZ0pbwqeQjGQtv7ByrXY9wHqCPICIRT3DDsBXZaCy80Agw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(376002)(346002)(39860400002)(396003)(366004)(136003)(478600001)(16526019)(66556008)(52116002)(66476007)(38350700002)(86362001)(4326008)(38100700002)(186003)(36756003)(7416002)(6486002)(54906003)(8936002)(26005)(2616005)(6666004)(8676002)(7696005)(5660300002)(103116003)(316002)(2906002)(956004)(6916009)(4744005)(107886003)(1076003)(66946007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: g2k5up19auWhN2F8fBn+Nhs98UO085OfdUkrLTILpr0PC07EEqQrb50sgeaw1VFOewMVxJ7hhHRxB/Vjn27fRwbzoKfJEE5qaiAxrB6VCPjIIu5/eOqL1X6djPt6wlwBQdT8710PBo+xQz1DAFrV90DeOq0rwY39RFzlAcoQbgqlDFeCzhen09End3apzBhgtLBEwR2C6O3PFJWj8OYr/jryY3QlBrD8L3+9H+gNqcBbmZV47tZ3ymgGBoHxZ+BHwEh+iscCsbm9dlZrOTnemdQXMzAa/15Amno5q3ZdYCz8TlaIPVQR459HgbMOK9O4RZBxcOhkuqBjOZ/fJogb7Dwzu4mSdRu9rrPHgKIu0kGcaD4LFLjBCNw6txKUz5/IfnnruWt+xmERP5Ul6b+bD8NTKxbSVX5H7gW9ny29lQO+uI9zIOS/hSsK8rSJUg3oQIwRZZ8/swQKhYK553fch8pfX7dyX5mGJZAcCWrSATi7EwMvn7w+GHCrVSP50GrXcqsdjkWneaqlQgyp5/7Xw42xGBOnKTEoib9LT6FBkdulM8TKy6pcAI3mbOeH0hyi58J2kMuf0gjWnBUHkjfJ0vm7HK2PVSlkU7Z06lH2sNE9kyUoKQwwewdGATH7GdZqznrBHz3PlzN57/saEcfP+u/xevxZOsJOsmIRNwpbZDCdb3DS9HEUV7/SPtEQPorcPpgF3YCFxXSx86ygKajg8IQxwkNR+gUHKfL+PCKoJL2tFxMMsjLjnp03cLF7RCxvUrLjJJ1G61DmiAG9BO0+fiYUbg2ESMYjHxI4mVaUiFmthp8X+PlSinKb6QklKU0lJqlDXyHymts3nqGwO1+A/ImsmcLRvvNnLF+sNmgNp6ZiBCCmPO+AcPbppbMGmukkKdbv3JpNxsnQJG8w1i5hMVME2ymqWbWVPTv4bbD6sWEq7NS6k55In5ew6wL4X92ZrPznM/6sbpi5tUjpROat+jC3DQGr4xvYnZ6bsT/VrV2ySeAhsH8B+qSg2xHKaHPj5oEhb+Z1tejvuujlyq601gDWKD0HLLMlRXWCFyCKsJhSddQOplNJDMotR4rRKUTRTKfv2qN0FMqZOtn0tEvkuBH0+yneLLZzE03KibLzY0ScOWDv0G7SlcEeeJBSTUyZ7EtEnKVXJ6DfNj0rAdukMLRpo0xUiOJmAoP7Dkv6MFn66qc9LE10wNeLZOeKhWDFn3VeOS/oQuxcf5fK88woltxOjlMQMugyo9fv/VfDcB5gaKnLqIY3a+XKRXklYEg6yu/aURcC+iX3Rt2+n39yOh3Z9M1STXZSKm9+t/4imERYm7dxvOYZAUGPk95Ol2/1 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 99433f39-a075-409c-1269-08d931c02285 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jun 2021 18:45:53.5051 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cHotVGXCjKGjQE7eMWOa066LEv5zxpleGgbc9Z9WcLQ3VUAV/JxJiD2qSpoCmAZTGRKNwLGY1RfjL30FPTF9KgH43SP6MXaUNfQAgO0Ry7o= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR10MB2913 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10018 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 suspectscore=0 mlxlogscore=999 spamscore=0 adultscore=0 bulkscore=0 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2106170117 X-Proofpoint-GUID: N_LZLoVW262b8Nt8luDJDdwv0IteUJLm X-Proofpoint-ORIG-GUID: N_LZLoVW262b8Nt8luDJDdwv0IteUJLm Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=lqLiOCUm; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=VAyWbVe0; spf=none (imf29.hostedemail.com: domain of joao.m.martins@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=joao.m.martins@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 69C2E562 X-Stat-Signature: jys1u9wx5bay8bx7y9dekoj1hqmpcp7c X-HE-Tag: 1623955561-985972 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Right now, only static dax regions have a valid @pgmap pointer in its struct dev_dax. Dynamic dax case however, do not. In preparation for device-dax compound pagemap support, make sure that dev_dax pgmap field is set after it has been allocated and initialized. Suggested-by: Dan Williams Signed-off-by: Joao Martins --- drivers/dax/device.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/dax/device.c b/drivers/dax/device.c index 0b82159b3564..6e348b5f9d45 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -426,6 +426,8 @@ int dev_dax_probe(struct dev_dax *dev_dax) } pgmap->type = MEMORY_DEVICE_GENERIC; + dev_dax->pgmap = pgmap; + addr = devm_memremap_pages(dev, pgmap); if (IS_ERR(addr)) return PTR_ERR(addr); From patchwork Thu Jun 17 18:45:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12329257 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67134C48BE5 for ; Thu, 17 Jun 2021 18:46:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1333A613ED for ; Thu, 17 Jun 2021 18:46:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1333A613ED Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2936F6B0074; Thu, 17 Jun 2021 14:46:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 21AC66B0075; Thu, 17 Jun 2021 14:46:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F16EA6B0078; Thu, 17 Jun 2021 14:46:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0208.hostedemail.com [216.40.44.208]) by kanga.kvack.org (Postfix) with ESMTP id B0CB76B0074 for ; Thu, 17 Jun 2021 14:46:06 -0400 (EDT) Received: from smtpin39.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 50931181AEF07 for ; Thu, 17 Jun 2021 18:46:06 +0000 (UTC) X-FDA: 78264095532.39.8B28855 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf19.hostedemail.com (Postfix) with ESMTP id 8E7F59001E5E for ; Thu, 17 Jun 2021 18:46:05 +0000 (UTC) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 15HIav0Q007622; Thu, 17 Jun 2021 18:46:00 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=eItdugg6FcfLnkv9zYSPCFwM/aIL01iwtGtFBoCYk6k=; b=oGrTslbVnufsIXdP2rpPqBgSejm8a+wXnPRQakegOb3I8sDJnS/4fgTDhMAQo31qQlH0 ppc8yzM/lQLL97WZ3NsVzh9sdhrwWOtv3wwDq1G5HBOYdOlg2OjQ/37Wzttz0NGKZmfq aEYaMRGfo54Yyx+gJSin/ofSPW5rFUQoiWCcyA/kWwaq0s7qBMFrUNTbHm+/+6Wp1hbb SRA66w3ReLSmLaJzNC2nvdeMg6OYRkiHOdxbdDy3QqYb8UklRiQ1RQslp4lFdJ/PPmYj k8L3ymFq+B6hjfive+uTsuCWkpGN1BYmBS6xIfxAn4Vnm6wO/pOHykd2U2N2E8Fi5A7G dg== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 397w1y1r7g-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:45:59 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 15HIjbNk180431; Thu, 17 Jun 2021 18:45:58 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2170.outbound.protection.outlook.com [104.47.55.170]) by userp3020.oracle.com with ESMTP id 396waxy699-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:45:58 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jb9BdFmTqF2ythukq6ZrBLBHWfSu7ee0vqiNfi2kcVfid6Che2/uXltKv+tre1ryhQlCHm4EE7PSsB2WmtGbtqbYJqu15jR2jqX9v8GBlXvi0QmCCtrkwjHilq147wQn3+0wOBq1V85ZH2nGcBQ1BnJcIa11hDc3+bkWexAGqDvjK1341Tl8NvdPrVVtdj2qNUwkEnDt7z479vdiLm7KRcX/WgvFNC0Wy5UTaNdzpha4MISEBFKC699woBAsVyzZ+qej9z2wvZ76y4iop9jTxCpynJ+Fb0mT7LtKYsotgqLmrRGo+O7AlAX3XU3PAhumgyzn5m0Wy3/iHHzW6cqLsw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eItdugg6FcfLnkv9zYSPCFwM/aIL01iwtGtFBoCYk6k=; b=l88yNZrIn0VgQbw0WS9KgaHPBX5jA+g0z0xAtqRWG9qb1k/U0nAH8TV6V/4BMtYrDrB6n2e90mEJPxKgYxQV060AiOpu1VpjDEvpvHzjwjVvmEBf4/giy9ZLojlv6GFoYSZpPqN0Xl+JHfAo7kfhRH6nYwn3sbX0dQm66wu9juwYt272JP/slc2QewZs4jbs7qwVPsAgyJg5DoC+0L84T+Lou74lM3fywUr71FO9Z6GmUfSl4NCaxHF9PKcJdk6SiZUgN1Zjdovk8GquUntZxGsafzRj+5Wdj7iUd8W4gsXcN9EzakziypZpfSztKyWpxUcNn5gGMDFQhO7VVlYaZA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eItdugg6FcfLnkv9zYSPCFwM/aIL01iwtGtFBoCYk6k=; b=KreOZcZnJgIokuWfuBTOKDU7E2zQsQw2NswZH0hG1qvSGwhfdFSsgQoryu6iP6b8ctJtKXlP2P1PMyBtCRBWyAlQ4SbXuIfF+Y/z2Vr1+yTI8XtLpxcMy7VPBid2qUaRDqqIU/7jTosrpMOmJRpAsSPOtCrhmO06EfTdXPMeR5Q= Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BL0PR10MB2913.namprd10.prod.outlook.com (2603:10b6:208:31::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.22; Thu, 17 Jun 2021 18:45:56 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582%7]) with mapi id 15.20.4242.021; Thu, 17 Jun 2021 18:45:56 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v2 12/14] device-dax: compound pagemap support Date: Thu, 17 Jun 2021 19:45:05 +0100 Message-Id: <20210617184507.3662-13-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210617184507.3662-1-joao.m.martins@oracle.com> References: <20210617184507.3662-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4242.16 via Frontend Transport; Thu, 17 Jun 2021 18:45:53 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 18e6dd22-fc3e-476a-0287-08d931c0243f X-MS-TrafficTypeDiagnostic: BL0PR10MB2913: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4125; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: dJ99YG+/8N14x7c8trSoo9wBX+6lrxB9nIeV2lARI0yM1yHqnAOeVjK4oQLNoPfj+VdCIX8ve8ZGF4exxfZk1RniAehOVDXLqaEAmcype/uWfSwCGfFt1Z4vc8R+ufuqf5aJX7gQHN5D2pS384WRZtv0Fs4nPtcKxYCjBP7eR951prMN803AEtp31CDVK42ZBWRnCX4z4rb+dBhZ7g8r3gOUmjThHzE7d1LVe+Ktvwgwf6N2HPmndewlbLlzL0A+uiNjdveK6run/IYHH+MVZSfyIykeq3ZVSNn6sD+mj9u0HXh/WppD8KFY1AULvj8IfNCZzc5L3VdWCrskdCCfSgP8kHaZgPpnxqYdJ23BHs2bgwjSx9tbYu6yhNrNoosXIJUIO4s6cI/VDuM/Vdp7boQiQ2qhxsZsK5wCRsXYf92TLnGUNaQAVuov0Im3es948HxRz6c0MZ9bjgq7Sh13nAuYtbATH7bq/PtWPlNuuGQz3zZFDfPm6bliUvGwqk6GHDAfqY1V94G4DHaGzFVIPE0YtZsCnw+7fNNKFYYAG4Xa6fmhWex3+MvpMoz+DrS2ex2T0UK0Gk1X5QoWY+gldI589KlS3Ol6M5isGL1XsqP53saALnsplGa7opWi4uq0UCdFQoWNjmfEFhc+U1q1p1LAITtMtN6IzuUTxypLEvs= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(376002)(346002)(39860400002)(396003)(366004)(136003)(478600001)(16526019)(66556008)(52116002)(66476007)(38350700002)(86362001)(4326008)(38100700002)(186003)(36756003)(7416002)(6486002)(54906003)(8936002)(26005)(83380400001)(2616005)(6666004)(8676002)(7696005)(5660300002)(103116003)(316002)(2906002)(956004)(6916009)(107886003)(1076003)(66946007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: P3HHfNyXtvHsAzNruSlXj8i2REaCq0tj2PGesL/dhmWduEGLMwwo8XCL8HCDa3McGo1o9NfLXy8Bmb45SSkgS2iggaclljzcdMTzL8YzxO0BFM5dZgnBSqMkC51xhBZK/xehFQy0YDBq03MsDL8OgtykcGZLCveqIbsTV8YMVyI8PnuQr+uUWrHjDCEOSMI5huWgarnKh58Lco7oJxLmLE9NI5r0Pmd1twavwOLOdf6uY/RpbTFasBLzs8L2/7XtSGPSM1+mBTzu9/gQMwUma2m16HiDJZlroa2MUZfMHBLdNrDyED+TsCSbrqNN8YcrgtyJ+1KBSqiwz6Xwm8ydqADaKtaUFZ8A5MYAjSyxPIJfNv35FKS1nKrCX1cBhUEQNGLMFT+QmsOZlnPCbXF2o6BThrbtLrnFvKpcZ7yn5qn8XF/oDU21MmGQAfpyePbQbO6Mki9okznoV/teErrQTALowIy8nfL+aOYTh9woVYqeyWwwWAEIjf3Cu1taV9MqFLbhC98sz3+3RBHFjaWoNEMY4eI0YrquDS8gs645ETpr/AoOmXopqOlZ9TgHzlSaMPMCJynzLkTm0tWQ0nXE2Mml3RzJjQQw90jSTh0twCoXmyB6I6BIQJzxXw4GwUac074M1Nx6rtE7DB875oVnfPzMI7E6zCJKZ2b23Gbb1n2821BBAKrh/h0HpLUMyBUnCJ0qyJlrtppILrsh1hnfHpSF+6umqAsV8UI9lRvTx8xIH+nfQjPThuANZ5lM1+JR2hnQmiHkK3CUSL24uo5OBHL74qOj49ecYLVQD3RJTZLVB989+mBqVNTo9EAe/lXpWYEqnnlh1coZQSVZnbBulZeNI4PENCsYnpGCKPA6jppOCY6g/8c1OMZ3N9Wed5LkZLPPJN9f7arjI/PVmP6jn9iipS9EUxjGiviWSt9i9io+jmzTHgX5e6kwRcSsr9bPT30fS+bnp9DSCDL77n979x+1VJoEm5lRZbiEo2eUSN8/3PWNVck2grcYvfftQGCm9n8UGfaYxypb5DXdV+PwGub5h0p411R5v+vA4rJPoi2CQa/sgtZiGBaUlU2uTz+JGVXvHf8MuaJXMbOmlCofIL7zV9U4UsPsKPdE94WV2FfC/vVEUJ80UvThV8kypZ69KkLbVxdeyhIrhf+qmvMNVcsK/4rYoLlAOVgV4X5bEMekCKU6yaOxKrd+I6xpn15KL2z70Jq+i0lE6BVfZgETV+J3cB+gbolwAhvDguKbfiXcxsM9BITM82xEXFjYLGvCaXcI8Q1ELHtNl3F1WimkRmNXXmg42LNtKTtorkUUlpj3m+AgnpqgXK7HgMwCxFgT X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 18e6dd22-fc3e-476a-0287-08d931c0243f X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jun 2021 18:45:56.3238 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 2G6pX6PXxSZC2L8A6KBDa4mRaMsP54nV0ea8JpjUfsPT+CA81aV+miAFrcZ6KA0OLV+NskzZ9cBHlyebJAkEgXNlHhL8TjXrM2l0316kJpI= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR10MB2913 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10018 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 malwarescore=0 suspectscore=0 spamscore=0 bulkscore=0 mlxlogscore=999 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2106170117 X-Proofpoint-ORIG-GUID: -c3ovvukEtzqwovRSNE02twDLNcx_1_9 X-Proofpoint-GUID: -c3ovvukEtzqwovRSNE02twDLNcx_1_9 Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=oGrTslbV; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=KreOZcZn; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf19.hostedemail.com: domain of joao.m.martins@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=joao.m.martins@oracle.com X-Stat-Signature: swoojrzb1fis43j96dzs3nc81wsf8kaq X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 8E7F59001E5E X-HE-Tag: 1623955565-519157 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use the newly added compound pagemap facility which maps the assigned dax ranges as compound pages at a page size of @align. Currently, this means, that region/namespace bootstrap would take considerably less, given that you would initialize considerably less pages. On setups with 128G NVDIMMs the initialization with DRAM stored struct pages improves from ~268-358 ms to ~78-100 ms with 2M pages, and to less than a 1msec with 1G pages. dax devices are created with a fixed @align (huge page size) which is enforced through as well at mmap() of the device. Faults, consequently happen too at the specified @align specified at the creation, and those don't change through out dax device lifetime. MCEs poisons a whole dax huge page, as well as splits occurring at the configured page size. Signed-off-by: Joao Martins --- drivers/dax/device.c | 56 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 43 insertions(+), 13 deletions(-) diff --git a/drivers/dax/device.c b/drivers/dax/device.c index 6e348b5f9d45..149627c922cc 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -192,6 +192,42 @@ static vm_fault_t __dev_dax_pud_fault(struct dev_dax *dev_dax, } #endif /* !CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ +static void set_page_mapping(struct vm_fault *vmf, pfn_t pfn, + unsigned long fault_size, + struct address_space *f_mapping) +{ + unsigned long i; + pgoff_t pgoff; + + pgoff = linear_page_index(vmf->vma, ALIGN(vmf->address, fault_size)); + + for (i = 0; i < fault_size / PAGE_SIZE; i++) { + struct page *page; + + page = pfn_to_page(pfn_t_to_pfn(pfn) + i); + if (page->mapping) + continue; + page->mapping = f_mapping; + page->index = pgoff + i; + } +} + +static void set_compound_mapping(struct vm_fault *vmf, pfn_t pfn, + unsigned long fault_size, + struct address_space *f_mapping) +{ + struct page *head; + + head = pfn_to_page(pfn_t_to_pfn(pfn)); + head = compound_head(head); + if (head->mapping) + return; + + head->mapping = f_mapping; + head->index = linear_page_index(vmf->vma, + ALIGN(vmf->address, fault_size)); +} + static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, enum page_entry_size pe_size) { @@ -225,8 +261,7 @@ static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, } if (rc == VM_FAULT_NOPAGE) { - unsigned long i; - pgoff_t pgoff; + struct dev_pagemap *pgmap = dev_dax->pgmap; /* * In the device-dax case the only possibility for a @@ -234,17 +269,10 @@ static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, * mapped. No need to consider the zero page, or racing * conflicting mappings. */ - pgoff = linear_page_index(vmf->vma, - ALIGN(vmf->address, fault_size)); - for (i = 0; i < fault_size / PAGE_SIZE; i++) { - struct page *page; - - page = pfn_to_page(pfn_t_to_pfn(pfn) + i); - if (page->mapping) - continue; - page->mapping = filp->f_mapping; - page->index = pgoff + i; - } + if (pgmap_geometry(pgmap) > PAGE_SIZE) + set_compound_mapping(vmf, pfn, fault_size, filp->f_mapping); + else + set_page_mapping(vmf, pfn, fault_size, filp->f_mapping); } dax_read_unlock(id); @@ -426,6 +454,8 @@ int dev_dax_probe(struct dev_dax *dev_dax) } pgmap->type = MEMORY_DEVICE_GENERIC; + if (dev_dax->align > PAGE_SIZE) + pgmap->geometry = dev_dax->align; dev_dax->pgmap = pgmap; addr = devm_memremap_pages(dev, pgmap); From patchwork Thu Jun 17 18:45:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12329289 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7D75C49361 for ; Thu, 17 Jun 2021 19:03:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 710AD613BA for ; Thu, 17 Jun 2021 19:03:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 710AD613BA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0A9516B0071; Thu, 17 Jun 2021 15:03:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 081956B0072; Thu, 17 Jun 2021 15:03:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DED1B6B0073; Thu, 17 Jun 2021 15:03:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0186.hostedemail.com [216.40.44.186]) by kanga.kvack.org (Postfix) with ESMTP id AA9D06B0071 for ; Thu, 17 Jun 2021 15:03:08 -0400 (EDT) Received: from smtpin38.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 4DE5C5DFC for ; Thu, 17 Jun 2021 19:03:08 +0000 (UTC) X-FDA: 78264138456.38.C41BCDF Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf28.hostedemail.com (Postfix) with ESMTP id 27711200109D for ; Thu, 17 Jun 2021 19:03:06 +0000 (UTC) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 15HIZdL1004692; Thu, 17 Jun 2021 18:46:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=V4tl+Eg71XG44E4s8uD07Q6GrHzl1AUOqv3jJcurla0=; b=utaPcTAhQR4O6TQH5Zj/A2XvnIjaB/20Pbcolw9paMpXp9H7LJYAw9Rqoh9jfgN1D1Ud QlU4eHju5oPobKyZP9XJ8zFk6ww1TkFzt1CPxhE87RLODsqgk3ffn3x880DnzXOCTU66 oWdpj/VzMxY/mrh2ax+/ebc88PECYWT91adedGne6WE2sLEmbllOdO7FgKwYYJQON/cH 3sJYqXMOYeBSGVtTSQbUlp171xvH2YuktELcOHOrnvYeD7ys4lb6i9iVCxXXmckuqq2C b89IFTSD68xULzAJIhLmCD0+shVlXApc2ERCNhuWkpi2134DIDQQqD6yCSSJM19SubrN Fw== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by mx0b-00069f02.pphosted.com with ESMTP id 39893qrcga-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:46:02 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 15HIjCPJ109688; Thu, 17 Jun 2021 18:46:01 GMT Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2173.outbound.protection.outlook.com [104.47.59.173]) by aserp3020.oracle.com with ESMTP id 396waw6wk3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:46:01 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OBBpDkqvo4MyexW1tojkxHpxjggj4yWQDijmGcYbddhBHkxrywhcLY8NKcResUQEoztb8/BguJoWztnn5ndXzVDeiCQ2W2syHjRpXX1kA5j9gWhBiC9bGA4NKSEqJ6WjM5FnyaeP7LLpcmCuu4XQuuHcb5wCXMJIQZC9LojHFpFOqQgHP/QBHBwPl6s/Ob36Ucb8AQzxOSgOOA9RIFHsZkeemFUfg/AHLqUqCsVnG3wbr9faqZZo3+GEUhRqtCgKPROWcka2nBavQwabU86qSyV5oY97l7iykTI2zY9hWtPi4c9YTpX1/lJ3jlSJCGFpGwh5/o820ZcO2VXmbl1llA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=V4tl+Eg71XG44E4s8uD07Q6GrHzl1AUOqv3jJcurla0=; b=LRzZxwpqwRACs1hFZU6BaOBVJ6rlk42pVkhmULl3h4YoGbQ08wD2XVPESqB/DTV38gCwLQF18MjezwlPxdi8egzsy2tv3Yu0AWAocN7N3zdZAMu+J05yNumGdUzI1SEF/D/1JS86uqY/ImJ0EZXEoM7ukKQ4bgY7wtfM8D+YFV+TjgZuAvLjo50Tn9LM6W3QWRgFrh7j8DPosqYopayymuqx7usYQUpfa49FUSnwvof7bFqcR4062jWlWih9Ji63k/7T8bfzU9fWfkH5GEo9jTIfxDrHqf7I29sq+rpH95ibt+ZwC0b8NS2HbJXxcHCRfYx04jkTarYzXaEMvsHjIw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=V4tl+Eg71XG44E4s8uD07Q6GrHzl1AUOqv3jJcurla0=; b=rzxGsa6wyuwZAe0VMv3JA5XNBI4u+adGU3is5ZT1rYjU6VTG1uwMgNsFHj9O7diXc9CLE99SdwoY6DDYYwAk8nVdEoxYQ8Rudg2VQz4Lx3lpW6Ab0VCOXqW01uIIg9MrLetRBfBaof3AbqmvSYJmGu47qPgPCnGLqKVhd3O9sR8= Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BL0PR10MB2913.namprd10.prod.outlook.com (2603:10b6:208:31::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.22; Thu, 17 Jun 2021 18:45:59 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582%7]) with mapi id 15.20.4242.021; Thu, 17 Jun 2021 18:45:59 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v2 13/14] mm/gup: grab head page refcount once for group of subpages Date: Thu, 17 Jun 2021 19:45:06 +0100 Message-Id: <20210617184507.3662-14-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210617184507.3662-1-joao.m.martins@oracle.com> References: <20210617184507.3662-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4242.16 via Frontend Transport; Thu, 17 Jun 2021 18:45:56 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e79354d4-c052-4741-e2e1-08d931c02603 X-MS-TrafficTypeDiagnostic: BL0PR10MB2913: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2803; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: bgOfRXcpe5KqJbKhZ0ayLb2lSkPaMjV3GEQbzCK/8SM35Y5hxV6YK+f84OIpNnnnZYoqbKl4lAaUN6tCZ5WEloEMIgPutL6Ms5N8AF5R/Uy1vip8J7hWD0SpDUyF/MAJMbL9Dnq8SXhk0oOQZs+cOQw9xX6ArfxkLqvNgvMYXqr+lJZLFfZBOVdUC6OWiGh1x/Ol7zD3qEvIbHilDLUOnaAJm4XX2aJ9zCDySp2osGcybZvda6uNGlwcXAfstF5LLw7DMxKKtWwr9imL0Tx6zWUYitZfPUHWIZBIfWt+OV2QorAYeDVlgmZCZizGwO1Beuh/UNt60A6mz9dbEGMtNF5AaMtiRw5qIXvXG8DANqIpWmfr9Tn+I7dlO+FCAmZ34BMaVQn2V47kyvDUJN59YKxgG8buUyc9DgrktRC//slm30PB7bIEG6+Ra/U/SY+4MkcioYZXtayqbr9XprU/Hz3Zwzwib9gHai+GV1EARc7b+cdzEjimW6cz0nwiKV1Db7J7pbMHZeLvoEwpTXENF2P47c9b75zR2HTxgZaoXCanF0Cv1wu+BaANdJqsse6mdiCpEn7rIU7RRWMsuETSOQn18zXA02bCnRY91Uh3EeoMuKKCAcwkFb+eTtnPDiNJZN/hKZXQB8Nw/qJ5okqwyNPHxB8WmNSpEn7W8T7Xqz4= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(376002)(346002)(39860400002)(396003)(366004)(136003)(478600001)(16526019)(66556008)(52116002)(66476007)(38350700002)(86362001)(4326008)(38100700002)(186003)(36756003)(7416002)(6486002)(54906003)(8936002)(26005)(83380400001)(2616005)(6666004)(8676002)(7696005)(5660300002)(103116003)(316002)(2906002)(956004)(6916009)(107886003)(1076003)(66946007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: qeBETB634nkvJfLcBu3Ozy2Ho9ULpjoQdAbBybeK8Ysl616RakCCsD9VLG/Gl/OzabjsBKUoUF++NCA8o24Ds5/z0phauU+gQEcD/9B/VBLsZrF8CKZjMFHrnXS9/VJcaC1PoJxlY4/qCJAcFTuClZkBTZUuwIyz5uj5xmplhxYmH/nKfoe7Q2n3RCSsnKBC+jza1hOAcdudR/9EcDVQNAvIcQQifTcHacYgbmYIC+mLueSiWczFUhsgmkH+XICKvxpJFdqSDObZNlrRRzB86dFYh3EsFcONj9SPOE7knxL8ATH5600Ev0zgqXwtEEbN5YNl7wuPhn1ZE3ElludGvqFXmBfYcrV76T1RpmbqJojMKeb7k/QpMgycIDSccgo2ZMvYSI9yqhGZamMehmGpYX/D2KO58BGOzPm0t+AdVovAMVJxmjQtYx9jp47W+j2+XzoJmZFiLvJWy19FeDeQGJFDzhDqGcJMeLw5BffQ6N/pyBLKg9WyuFM1C2CUdjyJ1aRKRDr+wN4P1uhYjIZauJpgc97KCB1gvDcugxiwXIy68LRSith0ZNdmCMu54fnNjNVfhE4MPKO94YJUV86dRoWThc6Nqh51wWbG/LvakqwOxMD2AYQadIHkUNBeA16lz5hrMMMYunNdPVx46Jhiv6u/LAW7BGWfXLYI6FDDdATywT6rTlJIqnsxIVPFOhEoVnQeto81z/80brQvcsAHNvWalhL6SD9A8bzUuHpQVRtduQ/LHJK2dX13n2iX8Pp+/f4S8jbEwiHoCA9QiHLQQqgruRXvoZwgdbAdoanXB3+NJ3Sij7tjy4EgmAXOk4t5j69i3tJAOswm8rHDHeIyDBbBJHiKALWdzWBOx9WS+UbbgKF6MX+tj4u/cyEDrN+YmaxEgUtwiUTwmSC+wwKVpHXJR7yvSNtqOpASsfXXodRCRaHaQ2bd7+u01OYG0Vh1VLjKlcaOVipRBq97JsSn7WoALrkzI1l/17THi7hMzA6ke1FRAsqypgOoR39D9lizduUtMJz1ox+h4n3J7lRqCDu24WlCw9NoaLc/kpl0Yw+8RBMrMwl0e834/T879iAz8vEnF1ooRrHb6ptW7ecbsuy1IoCBIFnCEt5lP1sNqZKOTTy/HjoVZXPCNDkLeKeg0nXzhFVSpOOKcsu3GtUDheK2GbkLSbx2ClMq9mL3sZZ/0GdeRoWRV3BeTrNOBAUndb51EGGbp3dBaByqzWXNH4B1shbE4OTsXG2SA3YBwXHf0BCJESfp0EWd7p8R3Nu0UsC7Dh6gzuvsYuSiYpY4wtkH3hrvI7JQE6z7Gu29YrbyjqNnqvsk4xOcRL7Wmd+I X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: e79354d4-c052-4741-e2e1-08d931c02603 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jun 2021 18:45:59.3335 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 1YBbpZU9aVBnxjg+EKzwdgQK/o9rhlbq6GuGPDSzFoTfDFwfE68sOswzTPWl7pI4TWK+L8pR/g8Oga0ADnRb43pN+vc8LHX6cPo/+V8Wvkk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR10MB2913 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10018 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0 malwarescore=0 phishscore=0 spamscore=0 bulkscore=0 mlxscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2106170117 X-Proofpoint-ORIG-GUID: znkNf4zdkR9Gm8KzZ7KiIksF4Pe16kiC X-Proofpoint-GUID: znkNf4zdkR9Gm8KzZ7KiIksF4Pe16kiC X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 27711200109D Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=utaPcTAh; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=rzxGsa6w; spf=none (imf28.hostedemail.com: domain of joao.m.martins@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=joao.m.martins@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Stat-Signature: 9pew6n3cqonnoxuu3hwt3k3pfcgrrga8 X-HE-Tag: 1623956586-988161 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use try_grab_compound_head() for device-dax GUP when configured with a compound pagemap. Rather than incrementing the refcount for each page, do one atomic addition for all the pages to be pinned. Performance measured by gup_benchmark improves considerably get_user_pages_fast() and pin_user_pages_fast() with NVDIMMs: $ gup_test -f /dev/dax1.0 -m 16384 -r 10 -S [-u,-a] -n 512 -w (get_user_pages_fast 2M pages) ~59 ms -> ~6.1 ms (pin_user_pages_fast 2M pages) ~87 ms -> ~6.2 ms [altmap] (get_user_pages_fast 2M pages) ~494 ms -> ~9 ms (pin_user_pages_fast 2M pages) ~494 ms -> ~10 ms $ gup_test -f /dev/dax1.0 -m 129022 -r 10 -S [-u,-a] -n 512 -w (get_user_pages_fast 2M pages) ~492 ms -> ~49 ms (pin_user_pages_fast 2M pages) ~493 ms -> ~50 ms [altmap with -m 127004] (get_user_pages_fast 2M pages) ~3.91 sec -> ~70 ms (pin_user_pages_fast 2M pages) ~3.97 sec -> ~74 ms Signed-off-by: Joao Martins --- mm/gup.c | 53 +++++++++++++++++++++++++++++++++-------------------- 1 file changed, 33 insertions(+), 20 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 42b8b1fa6521..9baaa1c0b7f3 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2234,31 +2234,55 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, } #endif /* CONFIG_ARCH_HAS_PTE_SPECIAL */ + +static int record_subpages(struct page *page, unsigned long addr, + unsigned long end, struct page **pages) +{ + int nr; + + for (nr = 0; addr != end; addr += PAGE_SIZE) + pages[nr++] = page++; + + return nr; +} + #if defined(CONFIG_ARCH_HAS_PTE_DEVMAP) && defined(CONFIG_TRANSPARENT_HUGEPAGE) static int __gup_device_huge(unsigned long pfn, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { - int nr_start = *nr; + int refs, nr_start = *nr; struct dev_pagemap *pgmap = NULL; do { - struct page *page = pfn_to_page(pfn); + struct page *pinned_head, *head, *page = pfn_to_page(pfn); + unsigned long next; pgmap = get_dev_pagemap(pfn, pgmap); if (unlikely(!pgmap)) { undo_dev_pagemap(nr, nr_start, flags, pages); return 0; } - SetPageReferenced(page); - pages[*nr] = page; - if (unlikely(!try_grab_page(page, flags))) { - undo_dev_pagemap(nr, nr_start, flags, pages); + + head = compound_head(page); + /* @end is assumed to be limited at most one compound page */ + next = PageCompound(head) ? end : addr + PAGE_SIZE; + refs = record_subpages(page, addr, next, pages + *nr); + + SetPageReferenced(head); + pinned_head = try_grab_compound_head(head, refs, flags); + if (!pinned_head) { + if (PageCompound(head)) { + ClearPageReferenced(head); + put_dev_pagemap(pgmap); + } else { + undo_dev_pagemap(nr, nr_start, flags, pages); + } return 0; } - (*nr)++; - pfn++; - } while (addr += PAGE_SIZE, addr != end); + *nr += refs; + pfn += refs; + } while (addr += (refs << PAGE_SHIFT), addr != end); if (pgmap) put_dev_pagemap(pgmap); @@ -2318,17 +2342,6 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr, } #endif -static int record_subpages(struct page *page, unsigned long addr, - unsigned long end, struct page **pages) -{ - int nr; - - for (nr = 0; addr != end; addr += PAGE_SIZE) - pages[nr++] = page++; - - return nr; -} - #ifdef CONFIG_ARCH_HAS_HUGEPD static unsigned long hugepte_addr_end(unsigned long addr, unsigned long end, unsigned long sz) From patchwork Thu Jun 17 18:45:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12329363 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3527C49361 for ; Thu, 17 Jun 2021 19:29:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 38F1E60FE8 for ; Thu, 17 Jun 2021 19:29:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 38F1E60FE8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4F8426B0071; Thu, 17 Jun 2021 15:29:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4CF996B0072; Thu, 17 Jun 2021 15:29:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2F8D56B0073; Thu, 17 Jun 2021 15:29:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0164.hostedemail.com [216.40.44.164]) by kanga.kvack.org (Postfix) with ESMTP id E2ED76B0071 for ; Thu, 17 Jun 2021 15:29:55 -0400 (EDT) Received: from smtpin40.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 76D061DE7A for ; Thu, 17 Jun 2021 19:29:55 +0000 (UTC) X-FDA: 78264205950.40.07A8547 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf18.hostedemail.com (Postfix) with ESMTP id E90A62001096 for ; Thu, 17 Jun 2021 19:29:54 +0000 (UTC) Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 15HIZal9004683; Thu, 17 Jun 2021 18:46:07 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=FSIlMjqFrKWQi7mpEYHw63HXjrqwPmYnSQoXBdSH1v0=; b=vtKEsbEdMuQaX8xtLKfd9jWbFT0uWVX5RPOnp0QYZFKGD2QsZ57aZO/J40luzon2ckS1 IvgIXAnNXh+Wf6PxsA+E8cOQjp0BHcU8zRt/Nlh/tsuaKlhWMir3jaRGes3+quiyiwUc Ik0Atl+aAWd9AaAqctqNHc1pyG9ACw+UPwkOEQVRmBT+RM7kx7UMK3eTNtzGjdGaOWqj mDqBItM7faN1EwNdrPBwM6QajZ6Nl53P04miQlEj9kJ/ZLRjYmwmPAJXb8sI+7zPTRVj tOgs6fzrsCSRlLckMgz4ZqGu7npTx2psR3nLBrDohbdTB5l97BkJNvXtRFbLzrwagrmH Kw== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 39893qrcgc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:46:07 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 15HIjZKK180135; Thu, 17 Jun 2021 18:46:06 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2171.outbound.protection.outlook.com [104.47.55.171]) by userp3020.oracle.com with ESMTP id 396waxy6ca-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 17 Jun 2021 18:46:05 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kAr5Wc2eN/nZYvxHnjGrp+i7eoRXArv1Umqvg+TYGeB+Z5VLPRDH38z6Qw5nRAOXEQpwuaPv+p6gn2hyo87GGylzMGO90dmSaCl+RMF5zZNN+0MunR/S5s89VbRcGYjcP2OjD1DpUhyyrIwtC6IZmgiImVGmrb9bMMVyPwR/1dO3xgi1AuZWH2onUBy8Gmb6ho9tcBdKzlSMzkE+NkES7pxhp+b8fvzh52IaHZEBJM/h/gMaAW1pmXb+VGBaq12xUfpo6veD2EaNr2xUzRlij9z5MgUxrFPECPb2kH2gYi78E35Cxba9a5YzqM2eYM1RD3bC7uW1AOvLVbq4JEVGgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FSIlMjqFrKWQi7mpEYHw63HXjrqwPmYnSQoXBdSH1v0=; b=QPyt0UKsJJLrDGH9QzaiyeV657lBhAACsjKP1GsDVG/XJe9vYrnewY16sTIWNfY/FZA12QrCdfzeWL0AqIPzlEYZ9eAPpQhX7ReG6bS5+2ibGa+uBhqiR1j2ofe8Ygz3qIcgQ7Tc4LdFbCXa/NnmKec/ZG7WKs59rCKdw1EiI3y2QzsQHRZxKdn+AIbhpNS7hJfL3t7r0isoJ7MJWLZJZWP4xoRq+Xmt836hRaFEMC3zsbebdcu+AQtTPKKoOsk2fflrKM2XUQY2tWCpj/blrzG/nnFiuRrKKkSd5WvQrtF31JEXb3kp2Bt0RyEzFGuqz3bzXYAygXgVuj7cjMkc/w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FSIlMjqFrKWQi7mpEYHw63HXjrqwPmYnSQoXBdSH1v0=; b=cgw58Ik9t1UYzXOccdyiQVn+AEHbXwUUQFsHS+7Efz5jAbavafz8DYdL/1BGGJoXODDjzmWL3c3D/4PYRzZN1qVnukWMBFiH26rA3mvjf67J2hWMBGX4bSexOJafCzmh8/Pe1sFNLGnnU1FxA0EOqF5YdCJFje14L8HPwe7iKnc= Received: from BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) by BL0PR10MB2913.namprd10.prod.outlook.com (2603:10b6:208:31::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4219.22; Thu, 17 Jun 2021 18:46:02 +0000 Received: from BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582]) by BLAPR10MB4835.namprd10.prod.outlook.com ([fe80::d875:3dd7:c053:6582%7]) with mapi id 15.20.4242.021; Thu, 17 Jun 2021 18:46:02 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Vishal Verma , Dave Jiang , Naoya Horiguchi , Matthew Wilcox , Jason Gunthorpe , John Hubbard , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Jonathan Corbet , nvdimm@lists.linux.dev, linux-doc@vger.kernel.org, Joao Martins Subject: [PATCH v2 14/14] mm/sparse-vmemmap: improve memory savings for compound pud geometry Date: Thu, 17 Jun 2021 19:45:07 +0100 Message-Id: <20210617184507.3662-15-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210617184507.3662-1-joao.m.martins@oracle.com> References: <20210617184507.3662-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) To BLAPR10MB4835.namprd10.prod.outlook.com (2603:10b6:208:331::11) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0113.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:c::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.4242.16 via Frontend Transport; Thu, 17 Jun 2021 18:45:59 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: a2b08e69-428d-447e-19ec-08d931c027b4 X-MS-TrafficTypeDiagnostic: BL0PR10MB2913: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8882; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: xDFbblCgdhDLtt3GWfWLtdyzGxKaE6kfGoAm8JXKbHFzg3drTzfsO9vg7jUa+/hlr4DqapMQrLcWluBL140b4Tnac8NlcOVqPwINCNFAdB1W1gjYBbhMsVzka6XLQeHUpy5MALvi6vVN3OJUejLPsDkXY2oMHmGQIqZv98YOHAIuLJ3VGr3x0oKrJP0pEf8Cj+CtABJrX9Z+OWS3uO6iiJDgRaEFqRbYAp47vFMVP4KhjUGC3Exjr2+MwovetIoV8dcrS2/9eRGu/s/CWguWzWDK7JjtBkxIgNv7rF75f+1BVihDOTI2Rk/+802X9JUQqW12T+N0o4sVD/rlwGaygcamkBFa/7SI1X9QjnVxxgZ4P8sXEi0tHgKpfJJyXL/m7UUxPCzEwnFEGDlZj+BakVgqOJwWee7usSAYcNm33ZHHMJBqNS/Iax7HfsaOWQRlK65acU2Oz0OhnIUhzto36Mpxb35AoyeB7bKicyfm0inY8V9lfvIoXHeYMN6J6i22SaPuCz7RmXdjpKQLEHkzVJlORHS7Yk1c+zENB/3owC1hav6868SmByKKPss3/6YkVoCa7re4SUHlG7u8LK+U13npFZJkJNYIMDWe8upHryxCgLMa1hpgJREFQlCfza3jTl300iBSgs9GX20WG3E+DMH24WssWRZ7R/7TlEv7vykP9sdqEzFX//G8h8G0z4Hf X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BLAPR10MB4835.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(376002)(346002)(39860400002)(396003)(366004)(136003)(478600001)(16526019)(66556008)(52116002)(66476007)(38350700002)(86362001)(4326008)(38100700002)(186003)(36756003)(30864003)(7416002)(6486002)(54906003)(8936002)(26005)(83380400001)(2616005)(6666004)(8676002)(7696005)(5660300002)(103116003)(316002)(2906002)(956004)(6916009)(107886003)(1076003)(66946007)(25903002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: pPulQz8ZDUlXI5l6TW0kMNrFYNIn2aCe13FvKU2uRXJLpprsMECNQUiB/7jqCRRgPIn2WR7hU30DorW4CJIRsakMIPjVGz5tbuzheZDXrnbcv/O+50M1rUch9/SOz4Bb8VBRBIFUX7YHNbmz45a6p6YzYwR+uxnzc+7fWY1dl9uexYN/yRVFY1y6RI3QZtKGsrfNx0y+ET/eIcw0MJmkINp+oEKqyPGvlI1GcneEgeboA+kyZVpSLVdPBxipf+wTuUy3np2YD2hgecQkgd8L8/gZ28fcd+NZoxRUp0rWhCPNC/CWKe8zJSq2ezvgXR81j6Z++tDQBvra9Gzq+alzUL7LB2EFokK5cnsGmP5Rwgqqx7DfybWsy6ruKNVL/I+ZtZLBsTaK4pEOprXfvfD+YgshV4/0r74S4Zs2PQhrM7OqLNh9AdUSUEy8Ey4H0MUitr9wGd8sNvOZC8RhHs8VPF9hQjg9XcST3bCWZmU0JydsGxzrcFIY2rhMORnS+3AolFny2FeSnkRqKMz/lYprOg9BOL25Tr4rF9LsK/FJ5lzdzm/HcpkaT7Yb+bJ7kssiKbQO3eNwexnEZOF07QPxCDhhXvdklzU0cjTkXFUMN1uuZXKf5DUPBM7UTwCFonVsfqIjH5R0etvHHG+aN3UYffUP1H4uOg5HtG0+KUqOrzM73EDPU6KtorKqtpXolujjbKwpA1FoCpy53ouFhrPd9YYp8WXB5qe1sRS4ic1FSwFBGeA/mvX+Yp14MAPNBtXjftDSvFnBIt+OlbPbgi8T/yGL8az46aaGpr7o5JMEiUohHYjx+tB/d5HeyhnGO/iL7/ByoBcyCG8LSy0vMOksv1MuJNNwwoBKmTcVLIG0eS4/tvwXevCIv5TLy/MFk7O5I3b9eD+WT0UdZHn8CuQu13ti6EhbCd2T7aQiszMyECI2SNMVfxWNdpfqWbP4V6WDx2NLlmnOiSAKB839s3lK3dxbKYAECHDRiP3CbPO22S2whnMNb8P4s+31BwthUT1ledq8gWV0hWNYpe+mT7FJMPmoP36+ub6IaRK2Kf0l6ltjdWLLf8dfyURYizG03Hxr5Rt8rUICQ3wdE3TUx/UDQw5X4Mt8Dw4h4HDk9kShKUdz53oquLaVs1COUpZ1ud4uLD4JWDOReAJj+ekHXPbVK8T08DiWg9mgUF+xrlr264ulAErMugd7zf9D1LIycJNtw85uk0w+fA4W5m6rdmAXqSmuFzYhR90fGndou1s/gTpsgOPyxfCiWvBIEKL97NWqutdWmpPhoHzRxNwXblSwRVdOpX2Z1QDkPPmdNZnT8KCoU9EHAbSDIjK3yzh/QIG3 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: a2b08e69-428d-447e-19ec-08d931c027b4 X-MS-Exchange-CrossTenant-AuthSource: BLAPR10MB4835.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jun 2021 18:46:02.1173 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: hDSm66bT+v0sF+XDCk/xF5a1N572KCBvKtu+bWFQrGaJbwzf/ktEsDl5C9wqIrJMy6qVp8qyOPtzG76Ohhq3VPveSy3So3cACsbHVtmX6to= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR10MB2913 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10018 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 malwarescore=0 suspectscore=0 spamscore=0 bulkscore=0 mlxlogscore=999 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2106170117 X-Proofpoint-ORIG-GUID: c9c3eTyYahqIiMqc6vnhZGJGr5M8FWxv X-Proofpoint-GUID: c9c3eTyYahqIiMqc6vnhZGJGr5M8FWxv Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2020-01-29 header.b=vtKEsbEd; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=cgw58Ik9; spf=none (imf18.hostedemail.com: domain of joao.m.martins@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=joao.m.martins@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: E90A62001096 X-Stat-Signature: grazxiurno4rw96roscde6rap6dxc7pg X-HE-Tag: 1623958194-387352 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, for compound PUD mappings, the implementation consumes 40MB per TB but it can be optimized to 16MB per TB with the approach detailed below. Right now basepages are used to populate the PUD tail pages, and it picks the address of the previous page of the subsection that precedes the memmap being initialized. This is done when a given memmap address isn't aligned to the pgmap @geometry (which is safe to do because @ranges are guaranteed to be aligned to @geometry). For pagemaps with an align which spans various sections, this means that PMD pages are unnecessarily allocated for reusing the same tail pages. Effectively, on x86 a PUD can span 8 sections (depending on config), and a page is being allocated a page for the PMD to reuse the tail vmemmap across the rest of the PTEs. In short effecitvely the PMD cover the tail vmemmap areas all contain the same PFN. So instead of doing this way, populate a new PMD on the second section of the compound page (tail vmemmap PMD), and then the following sections utilize the preceding PMD previously populated which only contain tail pages). After this scheme for an 1GB pagemap aligned area, the first PMD (section) would contain head page and 32767 tail pages, where the second PMD contains the full 32768 tail pages. The latter page gets its PMD reused across future section mapping of the same pagemap. Besides fewer pagetable entries allocated, keeping parity with hugepages in the directmap (as done by vmemmap_populate_hugepages()), this further increases savings per compound page. Rather than requiring 8 PMD page allocations only need 2 (plus two base pages allocated for head and tail areas for the first PMD). 2M pages still require using base pages, though. Signed-off-by: Joao Martins --- Documentation/vm/compound_pagemaps.rst | 109 +++++++++++++++++++++++++ include/linux/mm.h | 3 +- mm/sparse-vmemmap.c | 74 ++++++++++++++--- 3 files changed, 174 insertions(+), 12 deletions(-) diff --git a/Documentation/vm/compound_pagemaps.rst b/Documentation/vm/compound_pagemaps.rst index c81123327eea..a6603b7165f7 100644 --- a/Documentation/vm/compound_pagemaps.rst +++ b/Documentation/vm/compound_pagemaps.rst @@ -189,3 +189,112 @@ at a later stage when we populate the sections. It only use 3 page structs for storing all information as opposed to 4 on HugeTLB pages. This does not affect memory savings between both. +Additionally, it further extends the tail page deduplication with 1GB +device-dax compound pages. + +E.g.: A 1G device-dax page on x86_64 consists in 4096 page frames, split +across 8 PMD page frames, with the first PMD having 2 PTE page frames. +In total this represents a total of 40960 bytes per 1GB page. + +Here is how things look after the previously described tail page deduplication +technique. + + device-dax page frames struct pages(4096 pages) page frame(2 pages) + +-----------+ -> +----------+ --> +-----------+ mapping to +-------------+ + | | | 0 | | 0 | -------------> | 0 | + | | +----------+ +-----------+ +-------------+ + | | | 1 | -------------> | 1 | + | | +-----------+ +-------------+ + | | | 2 | ----------------^ ^ ^ ^ ^ ^ ^ + | | +-----------+ | | | | | | + | | | 3 | ------------------+ | | | | | + | | +-----------+ | | | | | + | | | 4 | --------------------+ | | | | + | PMD 0 | +-----------+ | | | | + | | | 5 | ----------------------+ | | | + | | +-----------+ | | | + | | | .. | ------------------------+ | | + | | +-----------+ | | + | | | 511 | --------------------------+ | + | | +-----------+ | + | | | + | | | + | | | + +-----------+ page frames | + +-----------+ -> +----------+ --> +-----------+ mapping to | + | | | 1 .. 7 | | 512 | ----------------------------+ + | | +----------+ +-----------+ | + | | | .. | ----------------------------+ + | | +-----------+ | + | | | .. | ----------------------------+ + | | +-----------+ | + | | | .. | ----------------------------+ + | | +-----------+ | + | | | .. | ----------------------------+ + | PMD | +-----------+ | + | 1 .. 7 | | .. | ----------------------------+ + | | +-----------+ | + | | | .. | ----------------------------+ + | | +-----------+ | + | | | 4095 | ----------------------------+ + +-----------+ +-----------+ + +Page frames of PMD 1 through 7 are allocated and mapped to the same PTE page frame +that contains stores tail pages. As we can see in the diagram, PMDs 1 through 7 +all look like the same. Therefore we can map PMD 2 through 7 to PMD 1 page frame. +This allows to free 6 vmemmap pages per 1GB page, decreasing the overhead per +1GB page from 40960 bytes to 16384 bytes. + +Here is how things look after PMD tail page deduplication. + + device-dax page frames struct pages(4096 pages) page frame(2 pages) + +-----------+ -> +----------+ --> +-----------+ mapping to +-------------+ + | | | 0 | | 0 | -------------> | 0 | + | | +----------+ +-----------+ +-------------+ + | | | 1 | -------------> | 1 | + | | +-----------+ +-------------+ + | | | 2 | ----------------^ ^ ^ ^ ^ ^ ^ + | | +-----------+ | | | | | | + | | | 3 | ------------------+ | | | | | + | | +-----------+ | | | | | + | | | 4 | --------------------+ | | | | + | PMD 0 | +-----------+ | | | | + | | | 5 | ----------------------+ | | | + | | +-----------+ | | | + | | | .. | ------------------------+ | | + | | +-----------+ | | + | | | 511 | --------------------------+ | + | | +-----------+ | + | | | + | | | + | | | + +-----------+ page frames | + +-----------+ -> +----------+ --> +-----------+ mapping to | + | | | 1 | | 512 | ----------------------------+ + | | +----------+ +-----------+ | + | | ^ ^ ^ ^ ^ ^ | .. | ----------------------------+ + | | | | | | | | +-----------+ | + | | | | | | | | | .. | ----------------------------+ + | | | | | | | | +-----------+ | + | | | | | | | | | .. | ----------------------------+ + | | | | | | | | +-----------+ | + | | | | | | | | | .. | ----------------------------+ + | PMD 1 | | | | | | | +-----------+ | + | | | | | | | | | .. | ----------------------------+ + | | | | | | | | +-----------+ | + | | | | | | | | | .. | ----------------------------+ + | | | | | | | | +-----------+ | + | | | | | | | | | 4095 | ----------------------------+ + +-----------+ | | | | | | +-----------+ + | PMD 2 | ----+ | | | | | + +-----------+ | | | | | + | PMD 3 | ------+ | | | | + +-----------+ | | | | + | PMD 4 | --------+ | | | + +-----------+ | | | + | PMD 5 | ----------+ | | + +-----------+ | | + | PMD 6 | ------------+ | + +-----------+ | + | PMD 7 | --------------+ + +-----------+ diff --git a/include/linux/mm.h b/include/linux/mm.h index f1454525f4a8..3f3a5c308939 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3088,7 +3088,8 @@ struct page * __populate_section_memmap(unsigned long pfn, pgd_t *vmemmap_pgd_populate(unsigned long addr, int node); p4d_t *vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node); pud_t *vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node); -pmd_t *vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node); +pmd_t *vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node, + struct page *block); pte_t *vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, struct vmem_altmap *altmap, struct page *block); void *vmemmap_alloc_block(unsigned long size, int node); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index aacc6148aec3..2eba2da31b91 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -537,13 +537,22 @@ static void * __meminit vmemmap_alloc_block_zero(unsigned long size, int node) return p; } -pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node) +pmd_t * __meminit vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node, + struct page *block) { pmd_t *pmd = pmd_offset(pud, addr); if (pmd_none(*pmd)) { - void *p = vmemmap_alloc_block_zero(PAGE_SIZE, node); - if (!p) - return NULL; + void *p; + + if (!block) { + p = vmemmap_alloc_block_zero(PAGE_SIZE, node); + if (!p) + return NULL; + } else { + /* See comment in vmemmap_pte_populate(). */ + get_page(block); + p = page_to_virt(block); + } pmd_populate_kernel(&init_mm, pmd, p); } return pmd; @@ -585,15 +594,14 @@ pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) return pgd; } -static int __meminit vmemmap_populate_address(unsigned long addr, int node, - struct vmem_altmap *altmap, - struct page *reuse, struct page **page) +static int __meminit vmemmap_populate_pmd_address(unsigned long addr, int node, + struct vmem_altmap *altmap, + struct page *reuse, pmd_t **ptr) { pgd_t *pgd; p4d_t *p4d; pud_t *pud; pmd_t *pmd; - pte_t *pte; pgd = vmemmap_pgd_populate(addr, node); if (!pgd) @@ -604,9 +612,24 @@ static int __meminit vmemmap_populate_address(unsigned long addr, int node, pud = vmemmap_pud_populate(p4d, addr, node); if (!pud) return -ENOMEM; - pmd = vmemmap_pmd_populate(pud, addr, node); + pmd = vmemmap_pmd_populate(pud, addr, node, reuse); if (!pmd) return -ENOMEM; + if (ptr) + *ptr = pmd; + return 0; +} + +static int __meminit vmemmap_populate_address(unsigned long addr, int node, + struct vmem_altmap *altmap, + struct page *reuse, struct page **page) +{ + pmd_t *pmd; + pte_t *pte; + + if (vmemmap_populate_pmd_address(addr, node, altmap, NULL, &pmd)) + return -ENOMEM; + pte = vmemmap_pte_populate(pmd, addr, node, altmap, reuse); if (!pte) return -ENOMEM; @@ -650,6 +673,20 @@ static inline int __meminit vmemmap_populate_page(unsigned long addr, int node, return vmemmap_populate_address(addr, node, NULL, NULL, page); } +static int __meminit vmemmap_populate_pmd_range(unsigned long start, + unsigned long end, + int node, struct page *page) +{ + unsigned long addr = start; + + for (; addr < end; addr += PMD_SIZE) { + if (vmemmap_populate_pmd_address(addr, node, NULL, page, NULL)) + return -ENOMEM; + } + + return 0; +} + static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, unsigned long start, unsigned long end, int node, @@ -670,6 +707,7 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, offset = PFN_PHYS(start_pfn) - pgmap->ranges[pgmap->nr_range].start; if (!IS_ALIGNED(offset, pgmap_geometry(pgmap)) && pgmap_geometry(pgmap) > SUBSECTION_SIZE) { + pmd_t *pmdp; pte_t *ptep; addr = start - PAGE_SIZE; @@ -681,11 +719,25 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, * the previous struct pages are mapped when trying to lookup * the last tail page. */ - ptep = pte_offset_kernel(pmd_off_k(addr), addr); - if (!ptep) + pmdp = pmd_off_k(addr); + if (!pmdp) + return -ENOMEM; + + /* + * Reuse the tail pages vmemmap pmd page + * See layout diagram in Documentation/vm/compound_pagemaps.rst + */ + if (offset % pgmap_geometry(pgmap) > PFN_PHYS(PAGES_PER_SECTION)) + return vmemmap_populate_pmd_range(start, end, node, + pmd_page(*pmdp)); + + /* See comment above when pmd_off_k() is called. */ + ptep = pte_offset_kernel(pmdp, addr); + if (pte_none(*ptep)) return -ENOMEM; /* + * Populate the tail pages vmemmap pmd page. * Reuse the page that was populated in the prior iteration * with just tail struct pages. */