From patchwork Thu Jun 16 21:05:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12884845 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B1CA0C433EF for ; Thu, 16 Jun 2022 21:07:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=IUnHsP5YElJ/f51oNJtnh4kDD0Dc47oirzHF/XAnqJk=; b=AIvuPRqxXu7znp pW8Uf/kRG2Aisrxy4P6yyyzaSBEc2+BMG8uFTBZ8ut9v9/NjQUlF5SgtDDYm1VLj7V+RB8PXYMvKx BJQi3Jh46X2yLYXwJXMhcUZumriqxdIK1Alsx17/uqDpvgssHZukpZSc+GCcMkCKxW0C9sr2qIQPM /Mfl1R1kMVwF3RGVjFTI0nyMfYi1uh5qQyqCkjX30mZsPyMeejEzZMgp5uDvzZZYSLFA38tMeHtI8 ie0KWIfYf4IhsyiBOstVS1ABuOIMRGIYLlj7mA9f0mFtUH+ptgbeqNwsI3n+agiE9HGMF7B5dDGdk G416NgQMEg+Gq1EAcIhQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o1wgj-004Lc8-Ma; Thu, 16 Jun 2022 21:06:01 +0000 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o1wgW-004LYD-3L for linux-arm-kernel@lists.infradead.org; Thu, 16 Jun 2022 21:05:49 +0000 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25GIhs7r032716; Thu, 16 Jun 2022 21:05:30 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=w5cTVYOn/v+TXONP93tevxMeXQlxzwNQUwB1Q4Yx+As=; b=ubnNNyVj/FmQScLBo3gzCrM7s+5z4GQCUgF0hZ9DsxY7t1Bh4xdb+kaUXux+RAYiyvI7 PMyBg97abNugmZVgs0sFuKtoSW/H3EDj6P+7m9oJWKtFlXlVLDcF5ufSCGbDjeT26Nfr rzImrm6QCTRz2UBOtOTTIE8DYMAhNnW6bGAvgrHd789IKh7rfIbchQsOxe/tBzniGF8P zbIDtHFaDFJ7/MMKwAXakuAU8tQtH3opn/DIWj4rXCGrto9+eay28/hVi5xGJseOzepy FZwj825lXq2tQh8kGgTCkb1HCPP7LKGkaRoXr9o19UFSgVkWo67Oq1vBsocadopO6RAE xA== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3gmhu2vg2j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 16 Jun 2022 21:05:30 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 25GL0Go1015726; Thu, 16 Jun 2022 21:05:29 GMT Received: from nam02-dm3-obe.outbound.protection.outlook.com (mail-dm3nam02lp2044.outbound.protection.outlook.com [104.47.56.44]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3gpqq2tugm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 16 Jun 2022 21:05:29 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Izv8Ugm0+KThLmkna22uVj2SRLVVkjCT+k6sS+AraddbIQ5WDjpR3fwD9rmbUvRpPQNNJiclWkFU+2NEVf1997cJLUaS6QTH1iul9AZBrWI7pqfEHY8aoHv2Mjwh5yjHDZAr92TgphbVUKjU+wnHSNqiFUmpFD9Ql8vfYTMtnbFgnaeGOIa0Pseh2Tt1XxDwkYN+eTrNZPEgHbZx4jwn+AZnRWLA1qUwD/sXsy4TX4zOETP9KIITgxRqpMKAdhXR9T42PUfzKiMiuk/kz5FPS9krHP2Vhb+mNwhtFBfn3HXjiU5m5oyuWjlXilGuIiiOn/qnPI0nJKUXn2dKiX4DyA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=w5cTVYOn/v+TXONP93tevxMeXQlxzwNQUwB1Q4Yx+As=; b=cH5yA8qK6S2DDlk+MLicffDcud0SmpNwYwl6ytGrpTLQuecfjcMQcY4lAghdCgwHr4fZGmqozJ3RWqo9aWC8mC4zQW1QTnxhJ7Ev3zVciphcEUNCQ9icNTp1ChsTGa1klYYVNXdcuoTBIFUtJuhUXJ+wN82Pg7128uGeAD90cYawk0WOj1NsikM6QmeEdFN+Fn+QjNrvNcrVmIDBMlMBNb2Dwp4/Bjzq2Di/TBr8a/ndx7rxaM1oBkSjGFI/Ue14bpa/yG4MRKDhMW6ldrE1APFswGhhvVIHshoU5mXyky0QwU+F9VfbPvmhAwS069pN8WvJhqp7DxOQ0PD9H5tk6A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=w5cTVYOn/v+TXONP93tevxMeXQlxzwNQUwB1Q4Yx+As=; b=hvq4QmRycb+cUCmEev+An/mAQ8/skeUlCmwupqk0hbCnkqQVg6/RwUHfyqOIBnv8IZ1NwAG6CkLtJe3eSbcsIdyoPIRuIdEhb/RhQ1/BWVweojWlK1/wWeJM6LHiCEGUtceeUB30UrBQ7UUAa7Z1NKsJh4EW0bEALv8stcfK2Ag= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by DS0PR10MB6197.namprd10.prod.outlook.com (2603:10b6:8:c3::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5332.22; Thu, 16 Jun 2022 21:05:27 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::2125:9bb7:bfeb:81f9]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::2125:9bb7:bfeb:81f9%8]) with mapi id 15.20.5353.015; Thu, 16 Jun 2022 21:05:27 +0000 From: Mike Kravetz To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org Cc: Muchun Song , Baolin Wang , Michal Hocko , Peter Xu , Naoya Horiguchi , James Houghton , Mina Almasry , "Aneesh Kumar K . V" , Anshuman Khandual , Paul Walmsley , Christian Borntraeger , catalin.marinas@arm.com, will@kernel.org, Andrew Morton , Mike Kravetz Subject: [PATCH 1/4] hugetlb: skip to end of PT page mapping when pte not present Date: Thu, 16 Jun 2022 14:05:15 -0700 Message-Id: <20220616210518.125287-2-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20220616210518.125287-1-mike.kravetz@oracle.com> References: <20220616210518.125287-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR04CA0091.namprd04.prod.outlook.com (2603:10b6:303:83::6) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 11115864-9766-4f53-e5c7-08da4fdbeffe X-MS-TrafficTypeDiagnostic: DS0PR10MB6197:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: T0P0TduMZQce0mjwG/oSmKHWjE56l/J20JaJOVo/TJZDS9C4rqrxDh62ljtIlHc+Kj9phMJsigO9C/qoAn4B0QXp2S/WDMvvC59NkZnYq0ai6AKKFfyrMW4qIAidLkQB93f82YkUWF3BjuTis6+0bnzOO/RK5FTvewpiupAIvFxVCNPkzmP9IcTtwamR1tNB5ozs6YAErZTaqJJVHYOclPssZp8pWiB2XmKb9ne2NODCUR4rFbMC/6eBvGCe7IvhMp84eY9gJhjmABDwCNyU3ylPYLpFU/r062FUvNwHcN67RWU9EPVUKJfkZM3JEj/8Xzds5CYe5zMyc9qiuUJ1AO+u3hVGQZjgZJpnGkcHAr1FsZRN/Hy0s/OTivYn/dhUwKZFGOegvWXvUk+Yh02w+PBrjDuZ8tVzyhTzLZi/wzoqYaZLMvGrAcdB0ciOLdQJD6QHkPnx6VIcqyxhvBU2HSqwYobaOY7C2Vy8N1rD+TF5CzbQwpgzzD647sdO1uKSUchcJA1uUTGzxB6KXq4qiFg32865RjB+Xb1w6gJqMEj/9Y0S9dTriF5aAnBVKmWu6eEJjFUIgO8710hRfPTDBSiHs+CrkUYlffAkT3wsDeLFO9b7Z4/IqgsHmEEZ/fAGBl9Rn40OIQ+Z3nBaAFuU2e7+o9LfmMeWFB15K42NIipY6dpHzPLOOBkwPD+Cq3LOab18Pk9KKGxPQfocCX0OcQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(366004)(6486002)(38100700002)(6666004)(66946007)(7416002)(83380400001)(316002)(2906002)(66476007)(6506007)(4326008)(8676002)(54906003)(1076003)(498600001)(5660300002)(66556008)(86362001)(26005)(44832011)(36756003)(6512007)(2616005)(186003)(921005)(8936002)(107886003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: CSyJXNi5AP/1OLed91zMPmjo+mwlO3lTKcBO9WanfxCUtW9xnTn0j/Fs9GFoH7CXF/1h7cpr4xlU7n7XPIT45xAaKUNI14AtyxNnQJJBJ7EFwtZxmtna8at41DFXesNGVRDfxkRv8hkyONup8iGjCYgi3anBseU0BifuALXM1cUoNYX+xpHVxk91L8hvH3YujjU8PTKgRiYMtGIqo5em7/0iw0VBx96FCi5xdfedYPzA7Y4qOFzerp1Vb+VpZQ4xZpNXQ9Ei4nNGcZ3MhpN9vHlILaxt7gpgdhzNyjgPcw91QBWF5nQ0DxxYgvRpeCehzupUZ5eGnUxeorW3fyQCDRxZSyOwqt1MriuYJf8lIOxjay3WLSgN5Of6rGEkpcU3BVmyFmY2QLUXkQO1Uc3yBOOeLB4mHq2vxjN1IBYjja8Rb8t3jvw3E3cUDmRamDV600IUBikFRrLRCJRpDfRRLsanQ3Xhj5x/DMk2eKO1mE314AbWyJAOx5Xy9SaDhR7oASwjTI4UOyjHm1FMitHy61P3f64MllFgM4BvG4Ixxk25NDNyLMoTWPuHdHEr6tbeQETzFBPqPJFL3nZkr9DlOrcRkzAwbJ3oG4DVUyINj45StKdnUWJa8G50XER5umk6GFHJ5Ytt4l1z86qS3AGRLWT6jCCQ+Pgd9/M1qM2mVwdewfv9yokIHne/V1GImiueyKw1Sjsqu/chEo+v/+XPs7dulqoAZNxSIbbVgWfdTjNgpX+3jqh0PLdQamPiuSabMIAg6PA45/Jmo3r1dsORsROCS700tMm7fbjxzSq45cQiSbC9efdpj4Tn9qqkZ9jHgvu9uC4OqKUyuRffoTr9pTBXlDo4Xqaw/VIKQph3rLFyc7itCkPZXdY6ck9Lqo7uadJpCzO9EODESmxMhqmLbJ9tyk0/TlbBexhwTMO/OH3a9Y9tLnVD76xT6KQgKPR1sYXM/VKg2ofErcz1tfw63RnwNpefuHs9Gh4pDZcU0su8kvGsNqTIVgA/gJgw7bkSA8IvGeeiE1ggzVN6iJ6E0RM0gs1KbZ+nUANtvTfE9XT2e6g9/3ZaY9Lmi4HJ1i7xgp0Q851j+H0cwSuyfudfPQrhEQvFWQzASBPjwouXwqxRivGTtlUI0fqxRKAY9CPMc6MsLeQqT46S/405yfpB4AHHzFjhl+6zP3bH41J2123muw27i4m8lUXaZs5tyrODowXbeKoEsNaq91Z+joSblehDydvj7fFBOjVdSdxUGM1f21Yfy/O2uqUGg4IXuCuUpfzvEmKI2XXbsSLE8LbvRstA1c99uepdlOfBaRN4vh6mK+Ya4DN+h1Dpo+OdtvYQDSKK37VEKLs3IMtiTvVjZZQ6jFRv6hSoxpSZEmJM9w1mBZIJrSPYpa/WzW3BbQm88ii+GN/pU8EXqMW70XvYpVsLjLG1y80FpokFbRriqsAyOq7Bw6rHOwby2GS9ziC/L8VIdHiA1FJxRuLEQhQ8lnpG8luV6avFhBYWxzMBm9K/Xr7ed7alefEa6b9mdHCsqnq5i7zIabievmqMpNwnKpkiBpyLqqKvZQ7wm9ffJ73MAvP0UCmckzfP+qoHUHqdyLmbOhO0M/Vhk46vJX1URumKnIN7mufnE+qGxRuziMjsx9K/4m9lrXsoY9dVDrpoRFDWD8iBBPEw6fxlOYUD5uyuOqmYeAAVYoEjpBRV/ow2pWj3H+o3+X1YLtzTM/URswvFag0JPphtynbIcc4GNpN+aA4wSUa6sGl0E7zI+a8= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 11115864-9766-4f53-e5c7-08da4fdbeffe X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2022 21:05:27.1269 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: i8twpWPpnkkw8vEE1iSX3yw69oNM8xvvtQ/NhxxnuYiEhCIurFYFtSmlmvwGvh0nkOm8ksTainB0rKXTZE6mLg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB6197 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.883 definitions=2022-06-16_16:2022-06-16,2022-06-16 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxlogscore=999 bulkscore=0 spamscore=0 phishscore=0 adultscore=0 suspectscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206160084 X-Proofpoint-GUID: sxMWOdiHSyPh95NX-KsyJz11lNRhFtWb X-Proofpoint-ORIG-GUID: sxMWOdiHSyPh95NX-KsyJz11lNRhFtWb X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220616_140548_297245_4E5A5501 X-CRM114-Status: GOOD ( 25.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org HugeTLB address ranges are linearly scanned during fork, unmap and remap operations. If a non-present entry is encountered, the code currently continues to the next huge page aligned address. However, a non-present entry implies that the page table page for that entry is not present. Therefore, the linear scan can skip to the end of range mapped by the page table page. This can speed operations on large sparsely populated hugetlb mappings. Create a new routine hugetlb_mask_last_page() that will return an address mask. When the mask is ORed with an address, the result will be the address of the last huge page mapped by the associated page table page. Use this mask to update addresses in routines which linearly scan hugetlb address ranges when a non-present pte is encountered. hugetlb_mask_last_page is related to the implementation of huge_pte_offset as hugetlb_mask_last_page is called when huge_pte_offset returns NULL. This patch only provides a complete hugetlb_mask_last_page implementation when CONFIG_ARCH_WANT_GENERAL_HUGETLB is defined. Architectures which provide their own versions of huge_pte_offset can also provide their own version of hugetlb_mask_last_page. Signed-off-by: Mike Kravetz Tested-by: Baolin Wang Reviewed-by: Baolin Wang Acked-by: Muchun Song Reported-by: kernel test robot Reported-by: kernel test robot --- include/linux/hugetlb.h | 1 + mm/hugetlb.c | 62 +++++++++++++++++++++++++++++++++++++---- 2 files changed, 58 insertions(+), 5 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 642a39016f9a..e37465e830fe 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -197,6 +197,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, unsigned long sz); pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr, unsigned long sz); +unsigned long hugetlb_mask_last_page(struct hstate *h); int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long *addr, pte_t *ptep); void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 259b9c41892f..7c4a82848603 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4740,6 +4740,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, unsigned long npages = pages_per_huge_page(h); struct address_space *mapping = src_vma->vm_file->f_mapping; struct mmu_notifier_range range; + unsigned long last_addr_mask; int ret = 0; if (cow) { @@ -4759,11 +4760,14 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, i_mmap_lock_read(mapping); } + last_addr_mask = hugetlb_mask_last_page(h); for (addr = src_vma->vm_start; addr < src_vma->vm_end; addr += sz) { spinlock_t *src_ptl, *dst_ptl; src_pte = huge_pte_offset(src, addr, sz); - if (!src_pte) + if (!src_pte) { + addr |= last_addr_mask; continue; + } dst_pte = huge_pte_alloc(dst, dst_vma, addr, sz); if (!dst_pte) { ret = -ENOMEM; @@ -4780,8 +4784,10 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, * after taking the lock below. */ dst_entry = huge_ptep_get(dst_pte); - if ((dst_pte == src_pte) || !huge_pte_none(dst_entry)) + if ((dst_pte == src_pte) || !huge_pte_none(dst_entry)) { + addr |= last_addr_mask; continue; + } dst_ptl = huge_pte_lock(h, dst, dst_pte); src_ptl = huge_pte_lockptr(h, src, src_pte); @@ -4942,6 +4948,7 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, unsigned long sz = huge_page_size(h); struct mm_struct *mm = vma->vm_mm; unsigned long old_end = old_addr + len; + unsigned long last_addr_mask; unsigned long old_addr_copy; pte_t *src_pte, *dst_pte; struct mmu_notifier_range range; @@ -4957,12 +4964,16 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, flush_cache_range(vma, range.start, range.end); mmu_notifier_invalidate_range_start(&range); + last_addr_mask = hugetlb_mask_last_page(h); /* Prevent race with file truncation */ i_mmap_lock_write(mapping); for (; old_addr < old_end; old_addr += sz, new_addr += sz) { src_pte = huge_pte_offset(mm, old_addr, sz); - if (!src_pte) + if (!src_pte) { + old_addr |= last_addr_mask; + new_addr |= last_addr_mask; continue; + } if (huge_pte_none(huge_ptep_get(src_pte))) continue; @@ -5007,6 +5018,7 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct struct hstate *h = hstate_vma(vma); unsigned long sz = huge_page_size(h); struct mmu_notifier_range range; + unsigned long last_addr_mask; bool force_flush = false; WARN_ON(!is_vm_hugetlb_page(vma)); @@ -5027,11 +5039,14 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct end); adjust_range_if_pmd_sharing_possible(vma, &range.start, &range.end); mmu_notifier_invalidate_range_start(&range); + last_addr_mask = hugetlb_mask_last_page(h); address = start; for (; address < end; address += sz) { ptep = huge_pte_offset(mm, address, sz); - if (!ptep) + if (!ptep) { + address |= last_addr_mask; continue; + } ptl = huge_pte_lock(h, mm, ptep); if (huge_pmd_unshare(mm, vma, &address, ptep)) { @@ -6305,6 +6320,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long pages = 0, psize = huge_page_size(h); bool shared_pmd = false; struct mmu_notifier_range range; + unsigned long last_addr_mask; bool uffd_wp = cp_flags & MM_CP_UFFD_WP; bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE; @@ -6321,12 +6337,15 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, flush_cache_range(vma, range.start, range.end); mmu_notifier_invalidate_range_start(&range); + last_addr_mask = hugetlb_mask_last_page(h); i_mmap_lock_write(vma->vm_file->f_mapping); for (; address < end; address += psize) { spinlock_t *ptl; ptep = huge_pte_offset(mm, address, psize); - if (!ptep) + if (!ptep) { + address |= last_addr_mask; continue; + } ptl = huge_pte_lock(h, mm, ptep); if (huge_pmd_unshare(mm, vma, &address, ptep)) { /* @@ -6877,6 +6896,39 @@ pte_t *huge_pte_offset(struct mm_struct *mm, return (pte_t *)pmd; } +/* + * Return a mask that can be used to update an address to the last huge + * page in a page table page mapping size. Used to skip non-present + * page table entries when linearly scanning address ranges. Architectures + * with unique huge page to page table relationships can define their own + * version of this routine. + */ +unsigned long hugetlb_mask_last_page(struct hstate *h) +{ + unsigned long hp_size = huge_page_size(h); + + switch (hp_size) { + case P4D_SIZE: + return PGDIR_SIZE - P4D_SIZE; + case PUD_SIZE: + return P4D_SIZE - PUD_SIZE; + case PMD_SIZE: + return PUD_SIZE - PMD_SIZE; + default: + break; /* Should never happen */ + } + + return ~(0UL); +} + +#else + +/* See description above. Architectures can provide their own version. */ +__weak unsigned long hugetlb_mask_last_page(struct hstate *h) +{ + return ~(0UL); +} + #endif /* CONFIG_ARCH_WANT_GENERAL_HUGETLB */ /* From patchwork Thu Jun 16 21:05:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12884846 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B132EC43334 for ; Thu, 16 Jun 2022 21:07:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=/aLMrPpvKS9bceGozzPYcYBIXG/IUHvhOYYjtXLfynI=; b=nEAislXYSDrP66 xcAc79HCtmEGi+MWJE0IVQFShlZf/7sfsrN/XTqbIevds4wr4tAtOc4EPFZpQ+ytLMaaxmOljsSI4 k6w44RQ7dekJ66AxTWSkryURgnkc/zJMxqGxLNM+PhTMl2pRPC2HDeIcXaJQgwdCJ26JWo3y1Tb8g nGj/5laPN+vRYxHTLQuwG73+ThwhCo0BXy7L689djF98gJ0+FtR3bnS8AkH1nODxWV5fX2+wCerho IBgan3slg3StXbeEVk5FpjKW3ztqbbRkO1n0Bozo+A6o9ldzgjySjFVr7w/1WrSS7ICRqQniciwdz fpr/wVYl/ZeWo1nybziQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o1wgt-004Ldw-EC; Thu, 16 Jun 2022 21:06:11 +0000 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o1wgb-004LZj-Fl for linux-arm-kernel@lists.infradead.org; Thu, 16 Jun 2022 21:05:55 +0000 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25GIdgGG009844; Thu, 16 Jun 2022 21:05:33 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=2wMpgJCRVKlN/DcMr5p4CoglZow28y7ZBL3d5MPCx5Q=; b=ffXIt+X6MDAmQrz9LwSsFzyqb/lOc1PdTjr1MWBakliYJpb9SJpFJkJtA5cwoWBdXEmt OmISUvFk9CYM/GStZ/7fcpozs7tQFky78kyvFP3PbRl3WGoZzj44+Y84n1T5u6AmCOTo x02xqmiLCHs9oFWKlVx5LpeCwe3bM4wvv/v58ROLCjz3zq+LwlXz5W2KlmONQzw4xLfR vG3u5GIWUCiz6smH3NVqLfgwEvz+zGdSBapDQIalAU213+zbRJ83E25yjXN1/UwFyvnQ QdpeEOvjhjphTQfC3stPOYt/HYHYuNzf6Fv55Gv4qBjg7fdmaeASF/Q/8/sSi8lq3Xow +A== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3gmkktmd3w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 16 Jun 2022 21:05:33 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 25GL0HTU031914; Thu, 16 Jun 2022 21:05:31 GMT Received: from nam02-dm3-obe.outbound.protection.outlook.com (mail-dm3nam02lp2049.outbound.protection.outlook.com [104.47.56.49]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3gpr2bpscu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 16 Jun 2022 21:05:31 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dlqLmbqYjc1LT5P1oQ7gOduCcfdgpSwIP4p8QtOKDzgIX8Aji81Sywf71G3J/0sKA1Jhsh0lAcLh+Nb033qLFcYZw8hqYvrDOVVjhLVnKbhGgVUobVq7aGcxXTWVQOXYWRwQmNG9e6X6I7m9fuDsTwt7UVxFT+fvwURx3zfKdfExQgxiQeRE7ByokYhu6+NvTt54d3s2d0QWd9aGymjW07RFEtV1s4Vq2GCcFIWqVGnN5R6t4ByX785yTirKXwqFScuTZfJlIevP4HrUTYgGCrMNmBTQAwj2lRGMi9d+TBW16JFU6BQAam954recyi+QsAi0JJD5UHc3MlFRDJs/4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=2wMpgJCRVKlN/DcMr5p4CoglZow28y7ZBL3d5MPCx5Q=; b=dQKHSnp0Quo9SgLu6Ys+8mz+QKuVlNvzTIOhlRsvoCfUCqeFiAjUPa4mFlAHj2ZFVoAfv/Atpk6t4Vi077fLsSe0QVS3afrN2PomZcUD0iyJVhnNvCXdPNjYV4JFkh21P6w8ACZy/CppAbuHOmIMF8o49M9N7wLJpKICbHdZ7bEe8izRnOj0P39nlKuf1kCLaGsFt27TuZN/ZeRRPlE+WxN/JfDiP1zxPKdn6qaG84erA0pijtWpFBXiOTKhtyHv7xU3miIO83bp3qUK87M/giHyYippyDI7cYJeI5MSd0kbnr0BRSDlmyoz+cxkXEwjzR+DxkUB7bu0iRRU1q0FSA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2wMpgJCRVKlN/DcMr5p4CoglZow28y7ZBL3d5MPCx5Q=; b=sLB5BmLewU8AnpW7JYua8VHK4yQEcxWTPobjoJGsBjFiWoRoDcrGURCHKAVjsUcs1KnDecjRhKE9BADul4mCTSIomXuqCv0T99J1eVBExOdSSSO+PUlp6BaZ4GhZXDycz5AN/kslOgzyQPqVP4Z8NoWD7eOU8WVXXElko0zDOak= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by DS0PR10MB6197.namprd10.prod.outlook.com (2603:10b6:8:c3::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5332.22; Thu, 16 Jun 2022 21:05:29 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::2125:9bb7:bfeb:81f9]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::2125:9bb7:bfeb:81f9%8]) with mapi id 15.20.5353.015; Thu, 16 Jun 2022 21:05:29 +0000 From: Mike Kravetz To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org Cc: Muchun Song , Baolin Wang , Michal Hocko , Peter Xu , Naoya Horiguchi , James Houghton , Mina Almasry , "Aneesh Kumar K . V" , Anshuman Khandual , Paul Walmsley , Christian Borntraeger , catalin.marinas@arm.com, will@kernel.org, Andrew Morton , Mike Kravetz Subject: [PATCH 2/4] arm64/hugetlb: Implement arm64 specific hugetlb_mask_last_page Date: Thu, 16 Jun 2022 14:05:16 -0700 Message-Id: <20220616210518.125287-3-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20220616210518.125287-1-mike.kravetz@oracle.com> References: <20220616210518.125287-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR03CA0237.namprd03.prod.outlook.com (2603:10b6:303:b9::32) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 24855748-effa-4322-bc1e-08da4fdbf177 X-MS-TrafficTypeDiagnostic: DS0PR10MB6197:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: J5ac2N9HU2mqaLjIkIcFAdzhT18SHYd/qHgtZkWKt+f139++Oz+f3sr8e/6mvZ3EE+wt/Up1/gmG8ENykFu42WnQTCIgI84kUoJTBi1+4oSrQ8gao2/aSWq2BQ28ad+nXfEvpSIbTA8u8joMoMTrhVf/CyM1+s8CjrZfJENoiB7dEBjqxXyXK3dsmajUy0AG1QP6YLVxVEIvA1bJ71Jqmmyhd1exOHrJElrAAFVFpHDK6/yfX94cumzN0/p8y2TjFhos3kymmQNhtdAdf8r36Yla4ASlHcF5RPBwq+2JSs4IWpvxMLZJbDPyvb2haUsldbTnhDmG4b4h66WkmdhZNGbe1l5JwdCwX7EH94RIvT2t30i425LGpSzsuZYVS51nUCwqAB305RbEEgTs1c3iyiqlHJTELXDN0/N7McfOwF5i953O2CYJtw+lynaSOagKhEp2qUVcmxhySW9dnJek85OtIg5lOA7uD1kFcdcoS7Pfh/L66FPuzhTT9u5apFLbGg4Atoh5FXh/bDoFtNMIazeWvC1PIZFH0TTqrmNAp3sJaoh88FxO61oy+wx3wAaIpV0LVzbHXuV27lXvoyhRd2ggBlzCV/ewnXWJQPNAqliqBV6ylVQu1CM+ULr7fVMk1gkio4ioW20PRYwUPGa5RLXMs9Y/LLbbtitfVbJI3CLDGfE03z7OpMKpFexQqUq+F9s7C5+eMXINCvPMkDeY5nWNF5pYdtQMAWtbBYxnlH3oaNHGZHPWqbPIEzLpj1S/MSECuY8EsmvUL6IiUeonpw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(366004)(6486002)(38100700002)(6666004)(66946007)(7416002)(83380400001)(316002)(2906002)(66476007)(6506007)(4326008)(8676002)(54906003)(1076003)(498600001)(5660300002)(66556008)(86362001)(26005)(44832011)(36756003)(6512007)(966005)(2616005)(186003)(921005)(8936002)(107886003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: pQBTpR2kLhxeyBpvihVvNyoT8W83Ivs8o3K5tehfI04ZZ8O/pO/WYEUD+xNms0fVKt553i0/aOqV6rojo9n0NpWVwxj8qF44vEKSYF6oprXhosEzZew/9jYdyq15isAV0YrR9f5Tb1dtM4/Ll6QkbwzmVIjhQodfBjV2E+deuxah3kyDSt7VYmXBpOU5adVa+6JsYuqA/OCY8zy4517ffftCcISiPX6MQ2WC87wD7KH2z+OPY9tvAZT3elfTmbOd5iuCEZ2ICPUgJvfoVBply/XGLt/MfQgVFl4ihP/E7vWAEQboZTdSvUwF7/PawswsNAiCnf6iJbrF5D9USIKahK3SIgbrlNORMkQvMsoq3cA3KVUFley5kTA+wLbj5HYPZVioD1fgg5HYIkOlAzwpp9idO5UneO5n+ACDR5z5YdrYujTpaSfggcMBKoKj/XUf+8sxShSSFdiUNban7ExXQRQLGaaPHwRS+Id8j7l4kw7zcoCJrMt4EIOg/EOeDZ3Xsa7Eyz8KidYL+NQGn3MBckIUcOiMbWCSaFHeB3D+7Az1DimLXFwuhZ/ze5rohqnzxrfxd3+E32+vBg9+AZWDBUgMrTSYCR53ZFPb6+r1INA+5cpgmKr8f6uHOvHAlT94PQzHsEs2or6ddvvIhQ3uvGHjJ2jTc2py+y0KSPxQrmW/Yc6ejMSQwKfS5gq1xEjnafbHaGxYyqjhdnnSEBlsw8Ve50oAWnHBwzMnlxVN+iRvRqMkBY7yENgsu9ZtVSUIdfGwj7n8ryd8q8SVba1vcuXqLZkYvIRmC9Ibwaqqns4tDekvm4zwjZwxPGIfltNHBZ6XET0S20dNOPKjBP/G5khdprgnkLVEhwGNWqWii5GNfW743iA8wqSKJ/NVVMxT0bxqbZX/GM+MtlEWkeccsc+klSyiS+AfFs7qiby69giyu2bOYFf1DwN7eyEjQAbczBnJNLlPWMbxaRG884GrHLXBANcIBGdS+oee5FUsstgVL2MoB9ZFQgAKAFtXS/FbyqJhKzGjYbbJUEBIOCaeguuEFepiBFzZ2hlvYYUud6Koiq3gTiKX1X+OVwyziuFVq5bCpOfJbUxfgEWMUckJ7S94hvVs0/dcrR6iqpv0P0pnoU7YjXmul4PjybW/j4H47Jfbn5xy/siVZDoBWsWS8Wc7Z8b3FYIkWM+swopy8BjEo9z18pYRXFvv+flXxTR5bHnPjx97U4HFMJ4BIZqh6lX2zdr764w0sBH5cvm37Fk7GS3tbjJ9aorsAXrUScTuG/GM079QVCN7zJEBvjG3RWoC1I9afOFJMMR68spuN/DNc6Cae24sbBZ8W49KbF1XpyWfSx2sWW9hQhasTUmAI5jjjN0c7tafu03TSJBuN1u1EnGabGjHAgi5lCcIugs2WIoSxDFNEyw4w6X3Ka7YkGoyLJimExNZPN1Eb597AB15874mbAyyVTEdE43FbDCGMRocPQYcWSCj7ZMcYvoiilFL57jeQ+sZd1GVtILJGEfdZH9DmNhds24awEmPpUKsJab5v35az/KZFGdbJZd9GLhbaLNK5EDRLE7gq/3MdtwNqtnI1QwfBvDmVJD2I+ynkqnx3IpGQVbEh4AEbtJUrGkPktYNR6oAtckx/2cbIt9LD2JCqK5CWmCCyNGjvORrLpSugsKs+yHWOMbhJjV5gIWpizrSd22zOLGIA2KWX4uVmyTYLVPVIcy3xdFKauVX1Ipw6AiVt/2Q29Jz5aOiQinBp0O40ZilFukZqjeFbd8= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 24855748-effa-4322-bc1e-08da4fdbf177 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2022 21:05:29.6121 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: mLbVJ2YtEc3AfH5Ef+39eTX+FsfxeNKMJT+icbYwKClJuhyUXYphX2nNGBFPfZNlT28OR1JYy8dTIoKCQBxp/Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB6197 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.883 definitions=2022-06-16_16:2022-06-16,2022-06-16 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 spamscore=0 adultscore=0 mlxlogscore=999 phishscore=0 malwarescore=0 bulkscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206160084 X-Proofpoint-GUID: Hiy9ugxg1FCaekjr715ESZpCUD5DLwnY X-Proofpoint-ORIG-GUID: Hiy9ugxg1FCaekjr715ESZpCUD5DLwnY X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220616_140553_563366_6B4AF171 X-CRM114-Status: GOOD ( 18.71 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Baolin Wang The HugeTLB address ranges are linearly scanned during fork, unmap and remap operations, and the linear scan can skip to the end of range mapped by the page table page if hitting a non-present entry, which can help to speed linear scanning of the HugeTLB address ranges. So hugetlb_mask_last_page() is introduced to help to update the address in the loop of HugeTLB linear scanning with getting the last huge page mapped by the associated page table page[1], when a non-present entry is encountered. Considering ARM64 specific cont-pte/pmd size HugeTLB, this patch implemented an ARM64 specific hugetlb_mask_last_page() to help this case. [1] https://lore.kernel.org/linux-mm/20220527225849.284839-1-mike.kravetz@oracle.com/ Signed-off-by: Baolin Wang Signed-off-by: Mike Kravetz Acked-by: Muchun Song --- arch/arm64/mm/hugetlbpage.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index e2a5ec9fdc0d..ddeafee7c4de 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -368,6 +368,26 @@ pte_t *huge_pte_offset(struct mm_struct *mm, return NULL; } +unsigned long hugetlb_mask_last_page(struct hstate *h) +{ + unsigned long hp_size = huge_page_size(h); + + switch (hp_size) { + case PUD_SIZE: + return PGDIR_SIZE - PUD_SIZE; + case CONT_PMD_SIZE: + return PUD_SIZE - CONT_PMD_SIZE; + case PMD_SIZE: + return PUD_SIZE - PMD_SIZE; + case CONT_PTE_SIZE: + return PMD_SIZE - CONT_PTE_SIZE; + default: + break; + } + + return ~0UL; +} + pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) { size_t pagesize = 1UL << shift; From patchwork Thu Jun 16 21:05:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12884848 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 25874C43334 for ; Thu, 16 Jun 2022 21:08:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=XxXkrE/cB2P8e17falmmNvFlQC54hSqqJiIQOCOsJbE=; b=vfjxppz8ZTQ41B Iym2oECu3DgWxDVgEdHhvcOS0YyXzLlMfbQ3XHWGxOB0FntcDwvzEkEp/nFeTHlfgYW49RSFMu+Kg o3nvSPIhdD+xEDOIldpLCmsR4c39T8tmDosuS3oLBRhTuemtatabEfm6qGcnfJM0LCXLNjRO17X7R IMFOJRweE/vQS/6G5aklMAknRozdJZK6ExOn6J1LSMmbxZpcXYgR4/jfD75xV17xhPSV0Vm3xw5oD E+vGxjpRcbVqF7fUQ8KjvmkkcK8uQqeD1pMPWTIJW/Oo5TuD7zNqnOJQrSOzrC90050YKUtwN2O5v G3FV0nfI5fzRtGVLcISQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o1whL-004LoT-Jy; Thu, 16 Jun 2022 21:06:40 +0000 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o1wgi-004Lbj-Db for linux-arm-kernel@lists.infradead.org; Thu, 16 Jun 2022 21:06:02 +0000 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25GJUUCl022342; Thu, 16 Jun 2022 21:05:40 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=u8MqV7+YTOCa3XoVx1aCQg2YiiCNVTZOThdez4VjBHU=; b=vI+LJORBdDRUTcuygK9W7t+1O16FIqGo1fg8oP4O0HFbk/83kOkIO5uvo08eNklQz+Xw MWy9tH/ceXd6Qj4EnTqbOWWLrNo/TR1NbOa28ki7zbO+qfCa0xDAK6D6HamOquTnSc5z /b0FSjDtAlGa3ZeLes7boRTxEJHo6gIInDMostIW/AXLWxiGc+il7gxv8k6vM026nbD0 CFyd5+f/8tuKll0CzYcZHAfygbjAujc/fdqfOuCqzS7mf1hbBm2StlSBCJePBInZ5XC4 IF1Hs9AVXq5VJI9Wbs0lN7FUxLGk2Fpl4uPMYiBEsGfnEsmsckgxn55jZgZ9yiIvWUVV Eg== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3gmhfcv83v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 16 Jun 2022 21:05:40 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 25GL0GHu015694; Thu, 16 Jun 2022 21:05:39 GMT Received: from nam02-dm3-obe.outbound.protection.outlook.com (mail-dm3nam02lp2045.outbound.protection.outlook.com [104.47.56.45]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3gpqq2tujq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 16 Jun 2022 21:05:39 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=k0Wh3H2IUeXHq0fMu7k8931N0tRsCbETNiYKR2wHd0GkOg/PN7jQeQo5hA0G8mux6b9XBtRChkXkpLCozIkqQCQ/uIamW5z8g5a+ZemIgCCh44Pq2uZnxr2w3UZ/nfnmG/asrtlpoadzfmsYPcjdBZb/EGK1weH9sZENKe4eDpz5/LvOb8NGyR7SSJEs8dfM6WxXsFngdEaOw3JXfKJBgBbtY5rMTtOSKqv8kO0+YI72fMLuzhE+9gnF1OULAlJ3TIAyQBvORLvt3NiNqGqb9hjWocZEBoW0WQ0QKHZvpblp3ibV1x7ywDjZSLi6mEuAGwQ2bzykps5zqvpfZHdR7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=u8MqV7+YTOCa3XoVx1aCQg2YiiCNVTZOThdez4VjBHU=; b=NzdHcjadTSV2P5xaOWj9Kkgvv0tfPPVCZkimwGYVpq1lmec43kZr8ssPO/myo/w8xVt4aIPTqZLi9L7jsXl8TxToZCHZZEqqZqvYiy/WsIFuvJZKD+D1ipWUKQS6HbGxet9vwOsrbxrzZv2bM0/rMQUuWKailut86UYBAAiH43P8Kssrl/z6spJeyjPwlCgcCZ/ZyILwt8uVrvvbbv5MLccDnkZ683dnVToi0hGsAS+lZl3mozadg/Yvvz9ROK7kShgpJQZ9PwzWyhBkYqooPnZq+4RgrrVO+MKvT711ESc4fwOmyaBpxPRviCgPOmX/0W1e78KCIYNZKpEd4lRyew== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=u8MqV7+YTOCa3XoVx1aCQg2YiiCNVTZOThdez4VjBHU=; b=ShR35TXuO2VeHrr/p6KARqrrYYLUEGi8N5A6Ol853jjtBk9BBFZEUha0rYd2izqnvngyiiVlqRSdv2qoAAfwkVifNtZDxHQE46a3YAJG5vWCvaDeqVNI80czWX9rplAIzP9fhw66MOszkoLcfNys5pb3oO0n1iqY6q4TGjWkRZ4= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by DS0PR10MB6197.namprd10.prod.outlook.com (2603:10b6:8:c3::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5332.22; Thu, 16 Jun 2022 21:05:32 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::2125:9bb7:bfeb:81f9]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::2125:9bb7:bfeb:81f9%8]) with mapi id 15.20.5353.015; Thu, 16 Jun 2022 21:05:32 +0000 From: Mike Kravetz To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org Cc: Muchun Song , Baolin Wang , Michal Hocko , Peter Xu , Naoya Horiguchi , James Houghton , Mina Almasry , "Aneesh Kumar K . V" , Anshuman Khandual , Paul Walmsley , Christian Borntraeger , catalin.marinas@arm.com, will@kernel.org, Andrew Morton , Mike Kravetz Subject: [PATCH 3/4] hugetlb: do not update address in huge_pmd_unshare Date: Thu, 16 Jun 2022 14:05:17 -0700 Message-Id: <20220616210518.125287-4-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20220616210518.125287-1-mike.kravetz@oracle.com> References: <20220616210518.125287-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4P222CA0020.NAMP222.PROD.OUTLOOK.COM (2603:10b6:303:114::25) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 30d5d0e4-8fc1-4044-8a9f-08da4fdbf2f7 X-MS-TrafficTypeDiagnostic: DS0PR10MB6197:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: +670796Ud2uwDglHFfmynye2SLPsLm3W6pb/6SS13+zVi+1whFADCzBGpRu4cI0yH78SzzD4eikl0ETozLTGzzx40/KCMv6f4Ycmuw90NGuEPJEA/TfPtDNnQ30PMmq2dhYO7zBCvsd/RaL05PlnEiAfI7wvKzPOKAzzzZozQthpqO75KsCpI3QWD5E2cHOcuyURcYY4cyeiOlFVNcyMHYhxuBOGyDsk77TDYvIt1uOv2lOoIu6CyYvCXBbJnB8PnomrXp8KutO5+GfER2gqm+qZ35lkzq5AdiDNRCLXcwndYKp65nNJkjs+uy5X6whvohLuSkK97jGdIoFGSjS3Z3Yzxax/4vqklOy3bl5dHvs2npiSfZqj4NpmqXmusBpWobJaztmE2FtKIeeU3UhSXgGN8+vPfhZrVk0U1uQnmDKMThFdbJOKnweMVAUDO5JcF2NDLQBvfCwrgIBj/5QraiSGRoA7p+DRgtpk2lDn6nvOXOei/AYHI1SsNfCT7N1rKeUR1KWLuvBIpLULHeGIzHIlYfqhWtT7UGeX1YYX7HDSOJ5YPRO2LgiXQl8CZ2Dxrv4WbCisMcU1z2vkOLFAyGdUMzCvAzaC2fZHIiqcwmhMjLDiyAP1L+31sB04qt+KmSnmGbA2wKsb8agGnUL6QxLtGZJiMTzTeiE2fWXwuTCcULRzX1oTSld2/gxgivU46z5V9VLTD1g7kTkuIdctbw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(366004)(6486002)(38100700002)(6666004)(66946007)(7416002)(83380400001)(316002)(2906002)(66476007)(6506007)(4326008)(8676002)(54906003)(1076003)(498600001)(5660300002)(15650500001)(66556008)(86362001)(26005)(44832011)(36756003)(6512007)(2616005)(186003)(921005)(8936002)(107886003)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Ws714Hu6Rl9RQFWNWfI7HKXPqH7Y+MEiYmFw651CD1Oxe0VaOL/DvhLNEnomBWCXi2wrduyDad95Cn8hpxRT9B06FxlTb1IgbUEnPCaFuYl34MQuQrHVvjwesppvB5+vnhT1gZEJEEUHdP12X0KUBiuR05KKtVksubu6wSUQrQfJJ8Sr4Lw1XgTOBcaDOIIXG68Co5PyuAm7BaU+6ILNptGNCz7J8QsBq84hr9wPs4qjIEjE/jE6leklScG/FtH5el1+4PHAqvtQcxhUbYr6cc+xLpIVqohUlZz7529D0Ce8LVRHJcPxEXDg/sSukEQ4WhZy2cBSNMZFlh0n+SI25+P5gaFG/rT33DJOG30ZF7O3tv0uOIdG198x6qA3Z6zJT+twrK4n/TBuLe7mom1LnmIJXr/XRCi/C0jKBgEDFhLYmFdyokcQ4k0xiDFmFNTIjAE2e1gAzWu9cwP7iDPshNIRkzD77uki+z+TaMnBqCadCE7zgZbhhPhrz9TjtJDyx+gVCHLHoOh1IHV3tCb5Mz+ZzhSM3vYkERqPU8EhlQ+L+XKQ3S4di3Xe0Aj8k5zX/D31Nt9zQ76EBgbntcUfNwgOQ8gPdPw4VtPZf/slzp7KWJeZpbsRyxxIHY4sWUENCBk4ZBmREbiMriuF+XQnpoJVmuFfDxhNRY+GSg4UgUNQpsaqGmeuwZd1L56W85Atxjw/SXkFhUihWvVRaemh28WBaTUb1YakOE1lL0PUHRJKCjtBnTAHZzeuvNbmuovvAsKLzlhumASN98xl4XPPHbJT75ZyTh2+ww1LNip8esUGG9iBmnyiIJ/YrkQZmvn2BPBeSOkYTfN6VHs0AETfR2EkBBCNUt1+IxNeIRsWlln8G471/gU/13oBQilimECs7P5rI4ZTEOo8PAYNC9UFCk1WlclhkXJVbvGFbuaTB//JdLN85Qb7ydyS8eQfDAF570tUslwULI54/UQWsP7Z9JczSmWlTfd0ugR0Twaea8hKvqc6OdaicpDjME+2YfYfL16NvAm0H4FYCC8x3fYI77ZmzkHco06eQN0zvRcEfD2ClaljK50+p+yyy6XXWG33c5rnmAX8o+XdlLKUC5qlthgK+ymtXEju3P84YIfsn0xc/IfgYh26AvxmJM51U9uPh/bdSJhOhJkXZu91qcRd4i3Mil808qmFGAKIhvNL2AuwWiMQS2S4yNzvkP3R19HxDs4PEoQ5FKytn7EGin06ONjBmCtL9jQKSZ/mONOlvSCRgeh4UKfXyjNDoSx3pNUqZfVmwDjllhysvD3RBq/YYuRlZFlzeG1mtIzNT1cX4S2YbC/bdnTrYCSK2VD1KlBvOSsfCGb5L7QjRjjJayhF4QQQcZitMuB55eBQUR5Kcn9JENoJmJLyT0f+ejv9tJBxdixwZj2pDzR97MLAcUvooO2aWQGgFS+OlGlRhUfciSPG3Px5u3g2hKK/CIrc0eCt1+d5KjzuQ/SZEPOqStL2RTQG8p1BxPOFrv2VBe/sw48G6q5y7s7kocUGmpN+Yl24g9jt1XimocHLkgEFizQyWkmFo38O1BcG0M3sdG9Wl8NpcnlYp6SHm7W7UpIkVEa1JwCDzGPpz/WO0NC3v6PWpJ0cVlhJEJ71pImAihZOqB8+O0WaUSTfGVwF7YAXk661SjMOhSLBWIlBo/voJZjYfv/pJIqwOKnISanF5Dq5q7d8XFCBvAwpsBT85n06YBNIW5g3QdqZqu1tDLdStGYZPgU4BAwDr9gEbqB/Ns/jOdc= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 30d5d0e4-8fc1-4044-8a9f-08da4fdbf2f7 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2022 21:05:32.1431 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 8pYTf0QuDt9Lb0AGPq6ZtzSVmNpIrfC/LTQHiPAFaX0wV7GpRMvrtxrSvnUFtOL9dwThyY0CgLuOCnIXIVFmAA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB6197 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.883 definitions=2022-06-16_16:2022-06-16,2022-06-16 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxlogscore=999 bulkscore=0 spamscore=0 phishscore=0 adultscore=0 suspectscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206160084 X-Proofpoint-ORIG-GUID: 2VPGOjUixEPzgRp5YosTaMF77GoSxn7w X-Proofpoint-GUID: 2VPGOjUixEPzgRp5YosTaMF77GoSxn7w X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220616_140600_613655_220A0B0F X-CRM114-Status: GOOD ( 24.41 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org As an optimization for loops sequentially processing hugetlb address ranges, huge_pmd_unshare would update a passed address if it unshared a pmd. Updating a loop control variable outside the loop like this is generally a bad idea. These loops are now using hugetlb_mask_last_page to optimize scanning when non-present ptes are discovered. The same can be done when huge_pmd_unshare returns 1 indicating a pmd was unshared. Remove address update from huge_pmd_unshare. Change the passed argument type and update all callers. In loops sequentially processing addresses use hugetlb_mask_last_page to update address if pmd is unshared. Signed-off-by: Mike Kravetz Acked-by: Muchun Song Reviewed-by: Baolin Wang --- include/linux/hugetlb.h | 4 ++-- mm/hugetlb.c | 47 ++++++++++++++++++----------------------- mm/rmap.c | 4 ++-- 3 files changed, 24 insertions(+), 31 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index e37465e830fe..ee9a28ef26ee 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -199,7 +199,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr, unsigned long sz); unsigned long hugetlb_mask_last_page(struct hstate *h); int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma, - unsigned long *addr, pte_t *ptep); + unsigned long addr, pte_t *ptep); void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma, unsigned long *start, unsigned long *end); struct page *follow_huge_addr(struct mm_struct *mm, unsigned long address, @@ -246,7 +246,7 @@ static inline struct address_space *hugetlb_page_mapping_lock_write( static inline int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma, - unsigned long *addr, pte_t *ptep) + unsigned long addr, pte_t *ptep) { return 0; } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 7c4a82848603..f7da2d54ef39 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4949,7 +4949,6 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, struct mm_struct *mm = vma->vm_mm; unsigned long old_end = old_addr + len; unsigned long last_addr_mask; - unsigned long old_addr_copy; pte_t *src_pte, *dst_pte; struct mmu_notifier_range range; bool shared_pmd = false; @@ -4977,14 +4976,10 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, if (huge_pte_none(huge_ptep_get(src_pte))) continue; - /* old_addr arg to huge_pmd_unshare() is a pointer and so the - * arg may be modified. Pass a copy instead to preserve the - * value in old_addr. - */ - old_addr_copy = old_addr; - - if (huge_pmd_unshare(mm, vma, &old_addr_copy, src_pte)) { + if (huge_pmd_unshare(mm, vma, old_addr, src_pte)) { shared_pmd = true; + old_addr |= last_addr_mask; + new_addr |= last_addr_mask; continue; } @@ -5049,10 +5044,11 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct } ptl = huge_pte_lock(h, mm, ptep); - if (huge_pmd_unshare(mm, vma, &address, ptep)) { + if (huge_pmd_unshare(mm, vma, address, ptep)) { spin_unlock(ptl); tlb_flush_pmd_range(tlb, address & PUD_MASK, PUD_SIZE); force_flush = true; + address |= last_addr_mask; continue; } @@ -6347,7 +6343,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, continue; } ptl = huge_pte_lock(h, mm, ptep); - if (huge_pmd_unshare(mm, vma, &address, ptep)) { + if (huge_pmd_unshare(mm, vma, address, ptep)) { /* * When uffd-wp is enabled on the vma, unshare * shouldn't happen at all. Warn about it if it @@ -6357,6 +6353,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, pages++; spin_unlock(ptl); shared_pmd = true; + address |= last_addr_mask; continue; } pte = huge_ptep_get(ptep); @@ -6780,11 +6777,11 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, * 0 the underlying pte page is not shared, or it is the last user */ int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma, - unsigned long *addr, pte_t *ptep) + unsigned long addr, pte_t *ptep) { - pgd_t *pgd = pgd_offset(mm, *addr); - p4d_t *p4d = p4d_offset(pgd, *addr); - pud_t *pud = pud_offset(p4d, *addr); + pgd_t *pgd = pgd_offset(mm, addr); + p4d_t *p4d = p4d_offset(pgd, addr); + pud_t *pud = pud_offset(p4d, addr); i_mmap_assert_write_locked(vma->vm_file->f_mapping); BUG_ON(page_count(virt_to_page(ptep)) == 0); @@ -6794,14 +6791,6 @@ int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma, pud_clear(pud); put_page(virt_to_page(ptep)); mm_dec_nr_pmds(mm); - /* - * This update of passed address optimizes loops sequentially - * processing addresses in increments of huge page size (PMD_SIZE - * in this case). By clearing the pud, a PUD_SIZE area is unmapped. - * Update address to the 'last page' in the cleared area so that - * calling loop can move to first page past this area. - */ - *addr |= PUD_SIZE - PMD_SIZE; return 1; } @@ -6813,7 +6802,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, } int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma, - unsigned long *addr, pte_t *ptep) + unsigned long addr, pte_t *ptep) { return 0; } @@ -6926,6 +6915,13 @@ unsigned long hugetlb_mask_last_page(struct hstate *h) /* See description above. Architectures can provide their own version. */ __weak unsigned long hugetlb_mask_last_page(struct hstate *h) { + unsigned long hp_size = huge_page_size(h); + +#ifdef CONFIG_ARCH_WANT_HUGE_PMD_SHARE + /* Required if arch does pmd sharing */ + if (hp_size == PMD_SIZE) + return PUD_SIZE - PMD_SIZE; +#endif return ~(0UL); } @@ -7152,14 +7148,11 @@ void hugetlb_unshare_all_pmds(struct vm_area_struct *vma) mmu_notifier_invalidate_range_start(&range); i_mmap_lock_write(vma->vm_file->f_mapping); for (address = start; address < end; address += PUD_SIZE) { - unsigned long tmp = address; - ptep = huge_pte_offset(mm, address, sz); if (!ptep) continue; ptl = huge_pte_lock(h, mm, ptep); - /* We don't want 'address' to be changed */ - huge_pmd_unshare(mm, vma, &tmp, ptep); + huge_pmd_unshare(mm, vma, address, ptep); spin_unlock(ptl); } flush_hugetlb_tlb_range(vma, start, end); diff --git a/mm/rmap.c b/mm/rmap.c index 04fac1af870b..1c22e5e1219a 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1559,7 +1559,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, */ VM_BUG_ON(!(flags & TTU_RMAP_LOCKED)); - if (huge_pmd_unshare(mm, vma, &address, pvmw.pte)) { + if (huge_pmd_unshare(mm, vma, address, pvmw.pte)) { flush_tlb_range(vma, range.start, range.end); mmu_notifier_invalidate_range(mm, range.start, range.end); @@ -1923,7 +1923,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, */ VM_BUG_ON(!(flags & TTU_RMAP_LOCKED)); - if (huge_pmd_unshare(mm, vma, &address, pvmw.pte)) { + if (huge_pmd_unshare(mm, vma, address, pvmw.pte)) { flush_tlb_range(vma, range.start, range.end); mmu_notifier_invalidate_range(mm, range.start, range.end); From patchwork Thu Jun 16 21:05:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 12884847 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4FAA9CCA47A for ; Thu, 16 Jun 2022 21:07:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=DXZD/a/l5DoItwPkI6zHXSSL785lT7YbfwLFdCuVT/A=; b=L8+rzroeTfkit8 1BiTiU8WjWUuNKK8cWZcJcbN2jBYEvT1w0VzU/lREoMegX2VodcrDW8hM458HWgnFA3vGjXyBEDh2 70OosE3RObLqOndRddrb0XqmuGtzyhCGStOlb/UhkUFuOGH2F2sSmZ+uKak3xV5haE1bi/c0ieEmt ia0qDoHWY8tQYPdA3MweQtvArDNWUEYAC3FJK4zFZJ6RxGgTvphtOKUZ69BPtbRlweLcS32ERcDIg g0b9wcd89E0cOzgIaXUW7Iv3p+HW0mJM5OLFI9R0bWoF9q+OvG2MVZH/5TSdGfWkv751PcWpF3oZI 1m50bQMU0kHOb8RrHMTA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o1wh4-004Li6-T8; Thu, 16 Jun 2022 21:06:23 +0000 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o1wgg-004Lb7-2j for linux-arm-kernel@lists.infradead.org; Thu, 16 Jun 2022 21:05:59 +0000 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25GIdgGI009844; Thu, 16 Jun 2022 21:05:41 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=BfKSG0usVJqxG0ol28aIEYVmg+YEzrDIzL+kXD4E9wg=; b=o+e4cy7evmuGzoHea8BiIpW4hw9XzJqk6IOX0uWGNTUsTmjWc58Wm/BFyunqf2onpY7B Vuhl4KvbM+5elwvB0c8CGpMELNfAS1pp3m6YpaihTbgVhy0QJbR0Tro9VUFFi8OeQHhL A4PaDpbs1nfFjcXwqEi8T9mQEEEYlJ0YwaGGb1rZtkukugOcdWKuHkd7eMwVyFjqqW6/ FvFnz93pBUOC3wlIYuJhB9QBDK6NF2VJBHEFNOZ/tOrv4xJnehbtlfixRo21QWDfVnis Qc94eSSDiBypCNwruWvy+QLvw8agFENtQd0BcRwzLu6au0Ywugcpf+5F8Q8evXRqNdiv Mg== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3gmkktmd42-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 16 Jun 2022 21:05:40 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 25GL0GHv015694; Thu, 16 Jun 2022 21:05:40 GMT Received: from nam02-dm3-obe.outbound.protection.outlook.com (mail-dm3nam02lp2045.outbound.protection.outlook.com [104.47.56.45]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3gpqq2tujq-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 16 Jun 2022 21:05:40 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FKTFXw7iXw01dGjBElm4aANH6RFuX3uwODZx1FP/i2de7d3AxuGA8VcGSeChUI+dSeqHwpbsg+9OB69Rp5a70abSbU7kBoQXWOuSU3sLLBYY0rdKQUq9YgtrtuEt/iPPvJkHydYLc6F2NCNAQeqeyD9p/6mmi4SYleKL9nUikVSOzRAmkQNJdkHYIfmhJ889jXsPp7mnxRTEM37KDKTv2STKL58Bjzvlcjr8Z73FTRFVeU+XaWnmCqWSv36RgLqC9ujXUSChomUppQzZw7wwOZOheOwXvrtP4pmdRnDV/nMwa5HO4/MYjTKR92nDWkzzP4f+1F+y8WqJA/KQU1LcKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BfKSG0usVJqxG0ol28aIEYVmg+YEzrDIzL+kXD4E9wg=; b=F3Mv/fPtqwu5kdIRuFqrnW6yFuasr6zMJ2jGzsiFMxuSQgCZeH4NMxLf/dcfbX27IO67d3exk1SbL83H6Yrr7eEyyl4M7CMUyADBbx2qJLfSPNoZTLDDeVvviINHx8NRRO42GcYL6YknnMGS0tvmS/qMdfaIlEIr02V+0dy+FgvHYTbVacIm3ruAUJrIOyjrThmUQ6x6CusAl8/ouwmyGzCbFiv7urB6zqT5QF8qCHG59cl7QIrLyPBKoHu/L4pwipMUslWRIB/Zpk2kZhD6vCYDCQl4gGllNMvfvISLJ2vbjzYiohhF+jSKdNFdYmi1jyKRMT899YeNa0rZZoteQg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BfKSG0usVJqxG0ol28aIEYVmg+YEzrDIzL+kXD4E9wg=; b=F3kP2LWAXHtwZqfJ5yO2ae5dDscOnIFBbcTc82nICI/kslP9/FEncXRCFgiLRHBdWCeCHf1gLbgfwEicd/u1pgTNQKyqJ10jeJPWrPiYysvbhYRsrUQsd3E1fovWAAX/g/Gj04N80mjnbYY7Y60c+r+L4j8gTaiHVDe4RqfBxWo= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by DS0PR10MB6197.namprd10.prod.outlook.com (2603:10b6:8:c3::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5332.22; Thu, 16 Jun 2022 21:05:35 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::2125:9bb7:bfeb:81f9]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::2125:9bb7:bfeb:81f9%8]) with mapi id 15.20.5353.015; Thu, 16 Jun 2022 21:05:35 +0000 From: Mike Kravetz To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-ia64@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org Cc: Muchun Song , Baolin Wang , Michal Hocko , Peter Xu , Naoya Horiguchi , James Houghton , Mina Almasry , "Aneesh Kumar K . V" , Anshuman Khandual , Paul Walmsley , Christian Borntraeger , catalin.marinas@arm.com, will@kernel.org, Andrew Morton , Mike Kravetz , David Hildenbrand Subject: [PATCH 4/4] hugetlb: Lazy page table copies in fork() Date: Thu, 16 Jun 2022 14:05:18 -0700 Message-Id: <20220616210518.125287-5-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20220616210518.125287-1-mike.kravetz@oracle.com> References: <20220616210518.125287-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR04CA0109.namprd04.prod.outlook.com (2603:10b6:303:83::24) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: def82adf-ef2e-490c-63f2-08da4fdbf4ac X-MS-TrafficTypeDiagnostic: DS0PR10MB6197:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: u5Uo7dTWBxc2qtpx7rKB5IVITFajHc6Hgu0DY62jC9p9uZ4pe0AYe8b9hLoienWriDpEf0O7UWDBcpgI4uc0K9/AlslnSFuUaO6R/ptA+1ObNhBDaqIOUPIR74N2Es5/AqccVjEvUw2TZG2uyiheYXFs1Tfh6mUkvlpK3zkFH6RtVGWkS1+Ij7I6+OYdpdLObqre2wnBDpWgxolV9XdqabX3IMdWcdHtG6uvmrrqb8wsvDmjtCHEX48063m7RMmEBYm7ZkvPeLrfbhc+qHe/SK9KQODAChXOyOnOnuewXGs/PZ4ZF4k7myr+r7XrTYqBk7zGEO8Fogng59mS2TVgCDnlHZUJoOTJuCtjc7b2bFTsTapaj9M7I8lweHDn5AAJGgHYIwSk7tEhMtkl7R5ZdapiAY73AVvzYYAlk5E+rtJnlefAJXaojRSGR5VgVdlN0mqs0zlIMov/ctKMDopDWVKm2fFESaUi5fDaj59KPyImfVw25klbRcdNfQf1U7f68AmcKhHOQg4+fI0+VDRuzCrIqYlWIZFbKe7KoK+QsqRis8uX7MP15DDcFQtDtRAfxpxMZtoHjb7WP2Ljy4Qo2UXheNRcQiliYinCtLn25dqsTG9vn3tS9zXeRCfto9lAt5dIHwB5SBr+m+J3sx0aU9tf/LfKSWigb2RKJdBcKg12H8a+49R/McQGi22bm/uJ X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(366004)(6486002)(38100700002)(6666004)(66946007)(7416002)(83380400001)(316002)(2906002)(66476007)(6506007)(4326008)(8676002)(54906003)(1076003)(498600001)(5660300002)(66556008)(86362001)(26005)(44832011)(36756003)(6512007)(2616005)(186003)(921005)(8936002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 5oEVS7A04wOfzSwQdYAZAORfCOAPLfcal+VGyb50K6y0NcpCG8tI5U7kL7yiD6tNv2dCx1deRhaxnvWipU4VAW/UiLaONNh1WH9BZikpbfcHp2ykXse25T5VclordR5RKyvlTLnk1HCz/u0kX3rOAv7e7VYKCHzDyjFlQHYe8Hd1ewP9y/6rjuP1tbUNjm3H91u5aqZNgm0aNByvBEEtGqTicJTwj2WM78QiDC+dM46QY3PxvYxMfrdZV2TX1uLCLGpcCiGJYwBTqnG4nh6z8oIhCsEesaZdwlTdk9Q0Qom1isUUdKDhK6S/JGXM8/8EjdazVy0XLx5K2dm6sByTXBJZcSOKjOF2TNCJWcXVMaq7Zz2FyjaDJPcPlJy/X/G0eaUnfRqd7vvu2huSEn0GcJR7w/vFT3MJte3ww6HZezfIiYqyv+dLDfzcmfhAV2J7YjRirr+eiZKre5iwUnztVeVMltksFZnh0/xP/27YV85vEaKh7LsqdYcfjrwLbml5EuPTYdrKXn6ctndhXbTfHZgQE7iNWcWYdWn27FaOk3ErB53fAkh0v3hYP6QwyUoubZEbirJ00kgtvJhoeayp7x7O+oRSgcLwMCsZIGyolzon3JplOq6bRp31CBUtAWbKhmmdAmd6r/dj+iCf5IfjjGtytBgXM3NxvSB/Epreku3GxlIB9LaHbMg1VXfAsVWzmJw+5F2kJJ+hcVLvPTS76h18J9TQOwZUDl4mUCkkzPryOYgNhirFO2NdDEAB9RfAjDroOGFMCJCloN1OPm8xB7NRfhhtRl0uoGg4yLhl9QagJw/sSXqSng+ZTI07grteuuDfEIFcREm5ldfqjfYjCvw+IrS0RBd+hf4n1iE5xvnmUF2moEMatHns8elIE0J+XUEiMU1rS+gFNfAzkLm03cbAEcX18LNUTG7begwed9LNOKbXz7QCu+Xv2405Rb4vihYe36AOzN/tsNf7g5XlA1WcPeQOFNjY0MiyxZUQjgRiEhH4yMERrHz9FgTu7dFhauP4wPxBkBEaEzYSEMpmJtWmvPOWHFAALFM8bR+qVaA60rb5HLsTHD/Ek+4sA4j+ej/A+kntPLACqb0A2JzV4Hfi+xHuLXRk1oEvByzGL2uZ0C7SeA1j390/2JNieQ7dk1zwvos2CAYwfdmuGAX2dd4iVpFl1rXfZuMJUYYje1RWOac2u7BhKS6RqQS2/ik2EtuVVr1UFpRm8hNCF/eEK6aXHhNb8M1WkBOEJ6RW71wD5VL5G5RkaUqCNiDXOgDxG9CHjNztKWVThEwpbD7Sw/hjKQPEYZb+ZMSzXr77GCmyjoZLOmAPUtcvOxyQZYWrn//NQakWwsH80CT+kMErCrFIhD1M4UCuU7Xb+maKEys1qlalLxJsF/3RSsCcS++jOzQzP2xZTaKy5wYldOvRUrmyzLimxQ1l1IhvVYfmhvYs+zd4dyRlIRUlO6BLeV+d7+B7ngoJM6G7J0mP8Ve5FT93lxhyZFh18sIrv7NJgRmDLhb53LjNRprkPhEtrrVdlO2D9wzusIMi/ch9SqMEcQn1aMNKBOf03PuuvIrowpV6q3qEgy1hIl/1Vr2o8v0K/E29HQaoGuJ2E/lpSMJuSGxgTG2g9/2Z4ThETXyMXpkPWyNOgG0JOJayiF/Fn5OaEvlXIH/5dg8V0XfAvGM/YxK71mz9vVqu2ZyvY1FYt82yafJOlxaihVPQNY5xzMYqLDR+0rQCE46+M9s3HOrREQk5uQHUoDmg8XAlMjKU/jI= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: def82adf-ef2e-490c-63f2-08da4fdbf4ac X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2022 21:05:34.9591 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: yTuK49dGr1vktt9noofs7FvAhg4HMrm1GbucXyY+F617FjbRjFVPuGfPokCKcJNFkmbb8ELnU5x2TndlbzdXHw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB6197 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.883 definitions=2022-06-16_16:2022-06-16,2022-06-16 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxlogscore=999 bulkscore=0 spamscore=0 phishscore=0 adultscore=0 suspectscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206160084 X-Proofpoint-GUID: SUcO1w6rXXGWAt8ZQo27JFglZq9zzZfl X-Proofpoint-ORIG-GUID: SUcO1w6rXXGWAt8ZQo27JFglZq9zzZfl X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220616_140558_225761_C92E8475 X-CRM114-Status: GOOD ( 20.06 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Lazy page table copying at fork time was introduced with commit commit d992895ba2b2 ("[PATCH] Lazy page table copies in fork()"). At the time, hugetlb was very new and did not support page faulting. As a result, it was excluded. When full page fault support was added for hugetlb, the exclusion was not removed. Simply remove the check that prevents lazy copying of hugetlb page tables at fork. Of course, like other mappings this only applies to shared mappings. Lazy page table copying at fork will be less advantageous for hugetlb mappings because: - There are fewer page table entries with hugetlb - hugetlb pmds can be shared instead of copied In any case, completely eliminating the copy at fork time should speed things up. Signed-off-by: Mike Kravetz Acked-by: Muchun Song Acked-by: David Hildenbrand --- mm/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index fee2884481f2..90d2a614b2de 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1262,7 +1262,7 @@ vma_needs_copy(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma) if (userfaultfd_wp(dst_vma)) return true; - if (src_vma->vm_flags & (VM_HUGETLB | VM_PFNMAP | VM_MIXEDMAP)) + if (src_vma->vm_flags & (VM_PFNMAP | VM_MIXEDMAP)) return true; if (src_vma->anon_vma)