From patchwork Thu Mar 25 23:09:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12165477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C561FC433DB for ; Fri, 26 Mar 2021 02:04:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5CA6661A3E for ; Fri, 26 Mar 2021 02:04:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5CA6661A3E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DCD4F6B0036; Thu, 25 Mar 2021 22:04:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D7C836B006E; Thu, 25 Mar 2021 22:04:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B5C056B0070; Thu, 25 Mar 2021 22:04:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0216.hostedemail.com [216.40.44.216]) by kanga.kvack.org (Postfix) with ESMTP id 9522C6B0036 for ; Thu, 25 Mar 2021 22:04:58 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 521CFA8CA for ; Fri, 26 Mar 2021 02:04:58 +0000 (UTC) X-FDA: 77960382276.23.1DF1542 Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) by imf05.hostedemail.com (Postfix) with ESMTP id 142D1E0011C5 for ; Fri, 26 Mar 2021 02:04:56 +0000 (UTC) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PNAHj5136830; Thu, 25 Mar 2021 23:10:25 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=IYSeFT3PKOSLK03Sr/oxNuTZOq3HJR8yrGxzOY6hTwI=; b=GEN96YFXtN/4IlQoktWOMyZoFCCRu1E2UVbMfcajqTgoDDKKTfPFo9nEpb/qMUDaXyrI 4F+USNKKaLsRESYdOyl8zFAFNKIwl7e7zyzTW/rK07kjiCsUqbgWXW/jvNbLnK50N18O EIccLjRRwHdF3h7PyNSvru7zcQpxNxmEWr8IRq9/eVtbzbgJCG+vmKdJ8NhO3NmEPMle Lt0ZH8VgOsj609pHjOF5C0+vz4w1UIN1TjGxvyiYSLkBIZRRf6l1WFzoNhr7HykbFFVD FsffFDBjvxVBxbOjxEIhE+3En88bxWJHh2qr27akljYcyIUkxkI2KlPCzR4KcrBxkxiA wg== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by userp2120.oracle.com with ESMTP id 37h13e8dxn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:10:25 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12PN1Ko7002948; Thu, 25 Mar 2021 23:10:24 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2177.outbound.protection.outlook.com [104.47.56.177]) by aserp3030.oracle.com with ESMTP id 37h140nhh8-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 25 Mar 2021 23:10:24 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=etm+xDll18SMSsVBEfeinS5EOTQ50wTmCvmaxnIe2sVKUHgGjP6NhioZYetCnq0TTUbkRyxSRRMOo3MeLwnHcObKwkpZW7hH5EVgPgJ9Ubiqgfq6MGRkQD/TVO+Nfb/4HYAu7dGxRTdfJ8UIb1uEiOYYi0+J0xMjr2haUe3RXKPYNUeROfLbWdIOuSUvw06A6GFL5JYPK1pbblD1elhZZjp4bkB/fr3rr7rQxeJ2JU7Adis/cDmiltDhraweziZ2ayH5+XlfMOJ2r/iYo8SuTKOQDovfVmsL4mMuWuTVHrjNfFh1Hi0o9MGbhEE8J9Bh+u2pR1ukRrypCSlVHG055Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IYSeFT3PKOSLK03Sr/oxNuTZOq3HJR8yrGxzOY6hTwI=; b=T85ZBa0Gz3qbf752pOT4FOu0vCkMUOvtqe/EcKYfCuj96JceLxvAhDcg+wUAeTZoTqI0AYe3GrUrT0FNvDxjD0y69bzHtC2IO+2QAsIozfOWaV75ZnRZKR2N6/trdGgSzp6fsANFx3c/ynwWwfNKgTX2AEedcxR6O5eJ1hHkfsfeZds0qm/QFcSVly9CJtTXbGQm4K/lchkd6dJ2DjocVbSUpHwZWWA+lwsVuEWnPmJj8vAcNvwMoTF0yL4vkVqP3nRBF3p/0oLC6AkrZNh6423HtHkSKaB8BGrLFMuA8Rg7sKB9JYIi0bxaSuWuSmYe9DCqvZ0zBHekOxWnfEuv6w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IYSeFT3PKOSLK03Sr/oxNuTZOq3HJR8yrGxzOY6hTwI=; b=YcwpZyuV2XyzZD+zkOO0EkfWtcSfTodB6hlDcCWRfAykzGswLlWpBLByihKP+nIqk6Iae2eufYjmdgBbGifqJjL2rIBI958XsWo35iWorU49D1KUXShkSCT6Yj271pyGvT6nLsym9E8owW2pwsmToLhOI2G7N7mFJakYQ0xHIxc= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by BYAPR10MB2853.namprd10.prod.outlook.com (2603:10b6:a03:92::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3977.25; Thu, 25 Mar 2021 23:10:23 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::507f:a466:3318:65db%7]) with mapi id 15.20.3955.027; Thu, 25 Mar 2021 23:10:23 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: Dan Williams , Ira Weiny , linux-nvdimm@lists.01.org, Matthew Wilcox , Jason Gunthorpe , Jane Chu , Muchun Song , Mike Kravetz , Andrew Morton , Joao Martins Subject: [PATCH v1 10/11] device-dax: compound pagemap support Date: Thu, 25 Mar 2021 23:09:37 +0000 Message-Id: <20210325230938.30752-11-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210325230938.30752-1-joao.m.martins@oracle.com> References: <20210325230938.30752-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO2P265CA0082.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3955.24 via Frontend Transport; Thu, 25 Mar 2021 23:10:20 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b525738f-d968-4734-015d-08d8efe32b25 X-MS-TrafficTypeDiagnostic: BYAPR10MB2853: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2733; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ybcPQ6fIy+mt9RcJtx0XBBAcpAvkrl4AbMoC/yLkVIKUg7+E4bMlBmf/Yc3+aKTgvMPYEuS9xtZnlY8OIiwmXY1EJtLuiAi2Kr3ld/Qq8R2K0gwPfYVz5Rjj1lphB5uqmTowrqFNrikzqiQLdh+e3N4vBaWSldy7SSQ3skdv4xK5NFMvXQBCdkvFWWqx3Tg62E7sipbErbsY+a/pu9U44BvGufQdEW/DjN5U9VcZYPAZY9BFD3Xqo7sTYskBN+MISzozLl42C8titX2I9FGvkJbgX/t6kMoYf+oPjPH38qBkMWnbhOb0eFutkFzcUwMmLUhSdjffU14zAKpsmwogbjxXdYBKu7d8RFfRHEI9CD31uytD8I/+2sRXbnnjNjRQusGIj7egWxv3scIKcWVK3GnZLioXMjhskTvAA4OPaGg/NPNATzWffnqQdQsfdfGNOEoF/A4V9wcawijRJP5OlJNRbUfANk+5M/FWU6C3AS3u72SkjQ7TYi1zHE9vVXOXx67ALHYts/PkT1b5DbEUepVKpm5LgRMuaOHowhDilcA84jv18YPXoIY7X/idVdtL6nFsQTUKAs0jUvgFHNgxHrExq9V+0F98dCl71BSctn4JNTLfuma0w+QgEOKzwbvPDWk99ssSYexkzTb8JbWX4Pv5DvxGeu8X2daIhm7A15w= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(346002)(39860400002)(376002)(396003)(366004)(136003)(38100700001)(4326008)(8676002)(1076003)(2616005)(8936002)(956004)(6666004)(6916009)(107886003)(66476007)(478600001)(103116003)(86362001)(186003)(83380400001)(7696005)(5660300002)(52116002)(316002)(66556008)(6486002)(36756003)(16526019)(2906002)(26005)(54906003)(66946007);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: OtgS/otz953nXBf8nYoWlr2cJWomyvpNjI2tsPyduyDTawTaducu1Yar0d1ROLCSJUC0SRkymJY6KHcZ2KvIbQVSebAB6TzcSdAqEwvliz0PIU/Cdm6m7dT8nnnms2GE5lEHn5OpaTCclx7e8kMnkrcM6kNcPMjdnyTee96WBQ6zreMIzo5WWPpPn+tSuoC9yx7TB1Utu+7iGfMfvmrifHbCVOcPcMEP3EGCLpA/el3JRlNXgJfdmQxjGX3Xm39KrXW2t/NBMcs9adCLMESDhuSFKz1NgPz01nMJF8GWI00L4PExa2jPVJ0/De8sGdQ2Kc6MtvtZz8cYeE4LNO+DCHXXvN6M837pAWrn5pfu6C52yL0WvBqArzBypCmwt3Ba1pXJaqznLA8C4Fbc6sTkoTsqYZi+kIRqoHtCqAfNzKyd+BZiFTFwRsGJhaayM4Gy3FPJfK1FLFM/aNppgf1YS6yH/YO/LvBfnfQfgLr38M6tFAB/3pwkTvEk2ks9MkceXh0S8AQCVj/wBDbn0Z6quAMUPq24tzI6PpLWIb/fZ9a3XMPttEO39LFsvIc6xH1HWBHar/p65wKmOeHtexnXprkh1GI/A/bhfbOQoEYkUaAsqzUX5uBR5kIVW2BToJJFNzyxghHm8u+vfEa4E5hc9EckMlSTzW+ogMstfb6x2Dc+OCI4bqgSdQz8u/FehkN5Iqy/2FC1p55WFEFky9i6qqSYJuDhm+JYH/QocvFzkNpLJvoXq6kOozSS1a7+cxle192ET3xRh5dlyWReTmXA/O90gN5jG6fJ2clYfxRlDIVPdghwXd8nOdgSL7Ql3moKdLJ/J7GPqizcbasxAIFwWJd5DLUd6TgPhMhDYVU1tr7G1X/Lh8ex37wmU/v1Yf+MIyWakMsDaefQ3Qq4+IG0nei+AE6pmSxlBODtgLZyJllCS327Cef6jYsoniPWVyyO3GbuAFjifaLbjk6cnVcvfEAR4Tv0Xj+R72Dlf6MhodzdgC5zy636e2aoP3B2Cmss+68R/CH7QLmFfwsJQXbzHmPv5VAqMMD5n3bK0L9vZ0XkA2PGaDFODXjJHHqhq+F0EniI1lkLarW+sneeUy6Qo74V1rNi5SDPlC/7TnkpRY4FCELOwfKibXMvgpVUfJk7fnOIopdXZ+wKudOBTgsU5EUtp6EJDrJe5UcOotIn1aUII5lk1nnyxxhB5HLikjFhI5kPO5YMrIUInexatpNbsGqHypQuQP2PxvnLlckflGn9gdD0A8SgCvrllbe36HTiVLpfJ8CrwZhRRl9pHdWpBZvYHBKpKa8roHxfjEqgZPsc7M34eUmNmPNrNlNCyogM X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: b525738f-d968-4734-015d-08d8efe32b25 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Mar 2021 23:10:23.5753 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: /t2V/bTRWG92xMai7Tdg4ewVrfHPUrAL9vBp2nLXV6BWP9YiruPBxPuWfQLdCcSYXpLPHWqIzrnQhTcKX1NpJjHNowQSuPq52QrNEh873ZY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB2853 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 malwarescore=0 mlxscore=0 spamscore=0 mlxlogscore=999 adultscore=0 phishscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250170 X-Proofpoint-ORIG-GUID: tuYdsBbc0DzjLg-8akuONTWhRb81AwUa X-Proofpoint-GUID: tuYdsBbc0DzjLg-8akuONTWhRb81AwUa X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9934 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 clxscore=1015 lowpriorityscore=0 bulkscore=0 malwarescore=0 priorityscore=1501 suspectscore=0 impostorscore=0 phishscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2103250000 definitions=main-2103250171 X-Stat-Signature: z8jbyjrsuqhx5muzwzughjuour9foxw4 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 142D1E0011C5 Received-SPF: none (oracle.com>: No applicable sender policy available) receiver=imf05; identity=mailfrom; envelope-from=""; helo=userp2120.oracle.com; client-ip=156.151.31.85 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616724296-145757 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: dax devices are created with a fixed @align (huge page size) which is enforced through as well at mmap() of the device. Faults, consequently happen too at the specified @align specified at the creation, and those don't change through out dax device lifetime. MCEs poisons a whole dax huge page, as well as splits occurring at at the configured page size. Use the newly added compound pagemap facility which maps the assigned dax ranges as compound pages at a page size of @align. Currently, this means, that region/namespace bootstrap would take considerably less, given that you would initialize considerably less pages. On setups with 128G NVDIMMs the initialization with DRAM stored struct pages improves from ~268-358 ms to ~78-100 ms with 2M pages, and to less than a 1msec with 1G pages. Signed-off-by: Joao Martins --- drivers/dax/device.c | 58 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 45 insertions(+), 13 deletions(-) diff --git a/drivers/dax/device.c b/drivers/dax/device.c index db92573c94e8..e3dcc4ad1727 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -192,6 +192,43 @@ static vm_fault_t __dev_dax_pud_fault(struct dev_dax *dev_dax, } #endif /* !CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ +static void set_page_mapping(struct vm_fault *vmf, pfn_t pfn, + unsigned long fault_size, + struct address_space *f_mapping) +{ + unsigned long i; + pgoff_t pgoff; + + pgoff = linear_page_index(vmf->vma, vmf->address + & ~(fault_size - 1)); + + for (i = 0; i < fault_size / PAGE_SIZE; i++) { + struct page *page; + + page = pfn_to_page(pfn_t_to_pfn(pfn) + i); + if (page->mapping) + continue; + page->mapping = f_mapping; + page->index = pgoff + i; + } +} + +static void set_compound_mapping(struct vm_fault *vmf, pfn_t pfn, + unsigned long fault_size, + struct address_space *f_mapping) +{ + struct page *head; + + head = pfn_to_page(pfn_t_to_pfn(pfn)); + head = compound_head(head); + if (head->mapping) + return; + + head->mapping = f_mapping; + head->index = linear_page_index(vmf->vma, vmf->address + & ~(fault_size - 1)); +} + static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, enum page_entry_size pe_size) { @@ -225,8 +262,7 @@ static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, } if (rc == VM_FAULT_NOPAGE) { - unsigned long i; - pgoff_t pgoff; + struct dev_pagemap *pgmap = pfn_t_to_page(pfn)->pgmap; /* * In the device-dax case the only possibility for a @@ -234,17 +270,10 @@ static vm_fault_t dev_dax_huge_fault(struct vm_fault *vmf, * mapped. No need to consider the zero page, or racing * conflicting mappings. */ - pgoff = linear_page_index(vmf->vma, vmf->address - & ~(fault_size - 1)); - for (i = 0; i < fault_size / PAGE_SIZE; i++) { - struct page *page; - - page = pfn_to_page(pfn_t_to_pfn(pfn) + i); - if (page->mapping) - continue; - page->mapping = filp->f_mapping; - page->index = pgoff + i; - } + if (pgmap->align > PAGE_SIZE) + set_compound_mapping(vmf, pfn, fault_size, filp->f_mapping); + else + set_page_mapping(vmf, pfn, fault_size, filp->f_mapping); } dax_read_unlock(id); @@ -426,6 +455,9 @@ int dev_dax_probe(struct dev_dax *dev_dax) } pgmap->type = MEMORY_DEVICE_GENERIC; + if (dev_dax->align > PAGE_SIZE) + pgmap->align = dev_dax->align; + addr = devm_memremap_pages(dev, pgmap); if (IS_ERR(addr)) return PTR_ERR(addr);