From patchwork Tue Apr 26 15:06:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Liam R. Howlett" X-Patchwork-Id: 12827375 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18A2BC433EF for ; Tue, 26 Apr 2022 15:07:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1945A6B009B; Tue, 26 Apr 2022 11:06:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EBB6F6B00A1; Tue, 26 Apr 2022 11:06:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 94EF96B009C; Tue, 26 Apr 2022 11:06:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.25]) by kanga.kvack.org (Postfix) with ESMTP id 5AB0E6B009B for ; Tue, 26 Apr 2022 11:06:46 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 3258D265EF for ; Tue, 26 Apr 2022 15:06:46 +0000 (UTC) X-FDA: 79399357212.10.3857E13 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by imf07.hostedemail.com (Postfix) with ESMTP id 6C58D40049 for ; Tue, 26 Apr 2022 15:06:43 +0000 (UTC) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 23QDT4aY025790; Tue, 26 Apr 2022 15:06:42 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : references : in-reply-to : content-type : content-transfer-encoding : mime-version; s=corp-2021-07-09; bh=HwAGCtpbwgLGsu1rCnMMsCQ2nMj2Dmh7jGriP+5uWhY=; b=Ob/8Gjzz/yvjMRVS3LsE+V2w9gkY0OkcBK15KITf1XDqx8aCf1+Adm+PsOfSflASAMHl OUCwSmS3Zg7sfZf+O9T453F4R/IE4LoyprZG5t+rAcAloWIHO1V89RZ1/sx7eO8XKJ+8 nj+P5u1kUTIAvsIKMZRuQRjB3BZQN3PxUzlKSiuRTEUWfdKHB9euYs2NNZ0GA0nPOROY EbI5pPRs90yXFKauq2NmFxLCfJb+ATaHH6mzmOjGEzgBObW8/+0RLMn97v+i1z9z7+Rz yiAzO4icMSIvjaLobw8Mq70EyAJpEw+X7KaFZqZG1W652RsWVezap0yzO1lUcusUyTnA Lg== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3fmb1mpdcc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 26 Apr 2022 15:06:41 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 23QF6UNt037819; Tue, 26 Apr 2022 15:06:40 GMT Received: from nam02-dm3-obe.outbound.protection.outlook.com (mail-dm3nam07lp2047.outbound.protection.outlook.com [104.47.56.47]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3fp5yjktqn-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 26 Apr 2022 15:06:40 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=X3t6wH0TspHDeUiCJScm7hn2Nd0UKmvC+hqAgawwrpAKni59eyHgOEluVTq3rGvrJjn56Bi29HQQb00RDVRbeq7zz4HGMLBUkT6o4LRp4mYivrJ0xcXmA0DKMSVoJ7Yrjf4c+4+bcIX0GJWf4f2JOtEJ/rbNNuZwDpfVH2HOPysb02gSSTxG+cb6n/xmv2JMzl6K/UF24kC2R/nagAMDRg2yjBSvFd4hW4oyEKcW1DrjD+O8J5X2lO94dxU2ZHi7OMBpD32pnbSDtxjQvW+k5c1VF/V539tDWQJNuGeSmF9pfHcpOrrTuU4UKIDuirPvOArtqhcxXL4cqdxbvEpRfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=HwAGCtpbwgLGsu1rCnMMsCQ2nMj2Dmh7jGriP+5uWhY=; b=cBwjWRqv20nmViVp2ohwG2c57qHVy21l4uYK0PwxQSLkxVKJcHirZZJSTNIgQ/Kv6v9CaeUFFbAzt925G2z/i3lPc9cGoYvjNjLmT98GzDQl/scfjdj7GieI+++BD9eA3vyE7506LHpA2ffbUl/JQ55XR0D/y/90ZUB+KT+8zcSqQfeP5FpiViLzCMlvazYe7UhHlbc9nSTApgyBzDnhZdUp8Rt1Qjbe0L97YYuldfMiuTD6K0hERAqLQSV3p0GerHk3XOSpg7gMeDxpZGc3J1WegDl50hrj7WfNzNv8JEeShEAK6y6xWuUgnS4KSegmlSTmVdDe/RAKnhY6wLoWpA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HwAGCtpbwgLGsu1rCnMMsCQ2nMj2Dmh7jGriP+5uWhY=; b=ZU9/w+ROrIn3Dwx3wIlgVM/F3qXe/Zf+e24TocpQqEyQ7fyCpD7cKZi+DKXLjKbCrrfn5LfsVAq3HHrnckXCV7nUHXNZM3w7Ef3Mw1ArK39GpRyuaDIVtmYyvFAIxYsE+fqSbRvzYEHqQ6xeqsJpYRLXihLQ3EqXVxmBTmo44gE= Received: from SN6PR10MB3022.namprd10.prod.outlook.com (2603:10b6:805:d8::25) by PH0PR10MB4679.namprd10.prod.outlook.com (2603:10b6:510:3c::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5186.14; Tue, 26 Apr 2022 15:06:37 +0000 Received: from SN6PR10MB3022.namprd10.prod.outlook.com ([fe80::318c:d02:2280:c2c]) by SN6PR10MB3022.namprd10.prod.outlook.com ([fe80::318c:d02:2280:c2c%7]) with mapi id 15.20.5186.021; Tue, 26 Apr 2022 15:06:37 +0000 From: Liam Howlett To: "maple-tree@lists.infradead.org" , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" , Andrew Morton , Yu Zhao Subject: [PATCH v8 25/70] mm/mmap: use advanced maple tree API for mmap_region() Thread-Topic: [PATCH v8 25/70] mm/mmap: use advanced maple tree API for mmap_region() Thread-Index: AQHYWX85FN9pJkEu4E25/Rd/2oQ9Mg== Date: Tue, 26 Apr 2022 15:06:36 +0000 Message-ID: <20220426150616.3937571-26-Liam.Howlett@oracle.com> References: <20220426150616.3937571-1-Liam.Howlett@oracle.com> In-Reply-To: <20220426150616.3937571-1-Liam.Howlett@oracle.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.35.1 x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 275a0f7c-4e21-4fc7-00d3-08da27965c20 x-ms-traffictypediagnostic: PH0PR10MB4679:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: P/aF3/0IodGZhCIxZ7kymDRSst0vWGalUdUIJzzUW4zy/n7uKDBbQ3DttcvFaTTLnePb/BGzZXz5CGbT3XyMxOYje2TvV72VoU1j+EgGmYQgQXRm/pL80QcleFS6S4pipwwVu6OR6Wb/3usJ31sgfI5uMs5SZUKrwPLJworaPhF26AqOLepGRBgFP/G7vNxEikRbu1hesxDPL/uIHwVUcZIOFTE1+P0lahDMiTzZ3eEJvFATwzG7jPNtHNv+3NIIbMWXDxxBaooWnCSZGxP0xfDNwItRFA2eG+GTXfhhy2ilJlmhPFi/bQTb45kEEov1JToL8upq3N2MvxA5/NwsaxNJd8tjY+VzF9V7wAB68atkmMqrzu7BldFTUdNJAjtR5hMImiyVthcDoEptvgtS6Oa25yod0lBArf80Svqk6k2fwEdCY+ilX9H64C4SA+J6vujoM4HZ1Z5ZjOpINP5TLdnPLL70fWkfyuAUc4l4vK7Dkgxa6AlrVg4Z34QDQmp0iZS8XGC7g9VnIUqBZhtCBtUXhAUiVvDlerBUhh8scl/cpKob2v2+6QIojLBo+jgXIKukXwqJsDvXXfGf5GIdAm1y3KYMdvxgFnLYbzsZIYRTqZEKRXsEf+g535QP+u+dPM13p/XpqgXQrcNNQ7U8d4Le37kertRm+sQqS6pLnIZ3YPh6xqOTq+ImfHVh4iln/mZqZtqQU13/IB37rOj6fA== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SN6PR10MB3022.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(366004)(86362001)(5660300002)(66476007)(38070700005)(316002)(2616005)(30864003)(122000001)(38100700002)(83380400001)(2906002)(66556008)(76116006)(66946007)(66446008)(64756008)(91956017)(8936002)(44832011)(6512007)(186003)(26005)(1076003)(71200400001)(110136005)(6506007)(6486002)(508600001)(36756003)(8676002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?q?sLCzdivVOH0yiv7XgJ24N8o?= =?iso-8859-1?q?ODjH8Dlexj+t/JCyReyur5z75PUxqc9HgGGMZhGz08kSrl0bHEOoErGNk1N6?= =?iso-8859-1?q?MHL0qPWtA6Pa7cM03w64q7akvsbQuSYfyaqUkV7j4uPXOBrvdSonBoRG+7BI?= =?iso-8859-1?q?xR4eYEsi2U9ats+KYJj8T0cQ8W3nTCBg52CfReuCYf1A5fNi5YcEMtX9c99k?= =?iso-8859-1?q?sRp5h/5DvlY7SWMjOXOaUg9o/c1oB/kDf4TExAcTORbqEPkMmkGd26QqQddX?= =?iso-8859-1?q?0gDqKlMZrNqdLkSN8nDJcPhnT105XuikCYPar3QqNNFBDIAlglNh40k6uOIl?= =?iso-8859-1?q?41+3m1sa8BfrQlRNCDIwgdllmo7X6ga+ujUHl5dvAgMcCO2BkKdf2f7zOWZs?= =?iso-8859-1?q?mOO/118DU9HYVcEjF/n1B60WbXvFTZ5cl07i09HUrV4QFCgiA8Fa7tZYV6Ra?= =?iso-8859-1?q?j5VTSX33uPDLM9O1JZBZ5GIq1pP/DQsnbSigSm0xklBM+FPgTWji8djeYxab?= =?iso-8859-1?q?UztvW3/xkCVfDqAYTj8YBodtmRWhrQIdrnCDWOUL7Px9EM0IIhsnX344vx01?= =?iso-8859-1?q?wEnZyc0vADyVIOBRvdxUudo8JMV5TbA4nQIRjCebHyVAgZp3xH+mBIJFfWTz?= =?iso-8859-1?q?TH+hmtmXIa5UmxdJHXv2lCQ/2I3i53Qn677zvxKaf5Vrtbabm1TDmL2CCAEw?= =?iso-8859-1?q?yn39cyqYkJymV6Iw55+qamY2sfFR7gt4oFeLaQ7DdpOyjirj675j6Yxcs+WT?= =?iso-8859-1?q?J5utUvW8txfvqqA9gG4t61I4c15wO/Hv4YtEzkl8812s2DCALAH4Qes3qwrr?= =?iso-8859-1?q?m/9gzYAEpnzbju+T13fuXpqQx0S1I31TviE99sWvbMJhT7zxoIGUePsVTAfk?= =?iso-8859-1?q?fatXnvhLHIC0FxX27iF5mFbH8axpdjchzKAI1IuwoMHCo6rnuOZZCz6Uaizi?= =?iso-8859-1?q?El0gXOznF5b+yaWAfhNtMQI4P3S7zKE/tqKSRPc8g5TRM762R6Qa2MWMwoI8?= =?iso-8859-1?q?pbGV4kDtMt/WqBFyhZjlq9zqQS/d8G6UnAbiaDfH585i06fGunJkdeGn20Ih?= =?iso-8859-1?q?jRujKAob7vuC9f45PdpJOVAcTjXB2KeAof/ENP7nrmOA2Irslr8LPyyWp+qR?= =?iso-8859-1?q?1O6Fl+K9Qzeim6xeaqPQ2J2wxP46luIivacPumptN6GlEj/I/hdxbmtaNCOF?= =?iso-8859-1?q?3vF2oFEbP1dXHD4MXd/iCiumWWchdYS0OHhIXJ97o6WUP8/2uURSu5acilZv?= =?iso-8859-1?q?ANsZp1dXHNTFmsps3H1r31okLbcFiJniaJ+e6EK2IpSH0XE28Xzk1R+tPLsH?= =?iso-8859-1?q?jerqntt8DN9p2gX5aF6veDSOxVAYgMrmM+goKGoaTuwWAoEXVTO8/B9MJTvW?= =?iso-8859-1?q?iBfnuapyV4ZxviuutGzl3gmgoS5rF43LguIdDCUH3rr1at/BHzHQ1MRT5mtR?= =?iso-8859-1?q?G+ioHr2m4vVAYxFT0WqkaFRh02GfBfQPt8djNs+aZYwsRA3NO3LxxmXGoJDt?= =?iso-8859-1?q?XOKzdRIiJTvTnSCWtoIIrPU1W9ZKNQ7reDHZxRwnpo/G7DRE8kKN8KjDMZCw?= =?iso-8859-1?q?5SKqqKAPaNKt5TKNd8vKemV3E0dDSQBfiqdsyfPgCLBHWTToeDBPPUkI9IkR?= =?iso-8859-1?q?544ImrnHFL1vJwxxtMxP9mMS0d9QhsRmiyhLjIrkWuszsfLvpnEw1ryvffim?= =?iso-8859-1?q?qMOEOMZk/xT5CldTwNnQR8M9P0FdHf9DvLO0ZRZEeQd2KFWGcyeGxxyHAW73?= =?iso-8859-1?q?ngv9rd3O3KAbCDzkByDjIWYaV7F0qXKxzG0sFbnYizC6REtkVQlbXmUrbbgU?= =?iso-8859-1?q?NbVcIEoc=3D?= MIME-Version: 1.0 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN6PR10MB3022.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 275a0f7c-4e21-4fc7-00d3-08da27965c20 X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Apr 2022 15:06:36.4744 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: PrK8siBa9HLpKMYBAcuz5w3GF2OrX3m8irRMdrNOBR4bhi4Kgb5I2pLmFpa5adLC1AKIc6BAORmRRLCJtBBkgA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR10MB4679 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.858 definitions=2022-04-26_04:2022-04-26,2022-04-26 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=999 mlxscore=0 suspectscore=0 malwarescore=0 spamscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2204260096 X-Proofpoint-GUID: CS2FehUSQFNwap1RME7l6PkkK68mRLFm X-Proofpoint-ORIG-GUID: CS2FehUSQFNwap1RME7l6PkkK68mRLFm X-Rspamd-Queue-Id: 6C58D40049 X-Stat-Signature: 8rcu5st3ntwytxmhimpoy6jfm4iusupa X-Rspam-User: Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b="Ob/8Gjzz"; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b="ZU9/w+RO"; spf=none (imf07.hostedemail.com: domain of liam.howlett@oracle.com has no SPF policy when checking 205.220.165.32) smtp.mailfrom=liam.howlett@oracle.com; dmarc=pass (policy=none) header.from=oracle.com X-Rspamd-Server: rspam09 X-HE-Tag: 1650985603-770716 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" Changing mmap_region() to use the maple tree state and the advanced maple tree interface allows for a lot less tree walking. This change removes the last caller of munmap_vma_range(), so drop this unused function. Add vma_expand() to expand a VMA if possible by doing the necessary hugepage check, uprobe_munmap of files, dcache flush, modifications then undoing the detaches, etc. Signed-off-by: Liam R. Howlett --- mm/mmap.c | 245 +++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 198 insertions(+), 47 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 843fddce316f..2d7d9c26c57c 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -519,28 +519,6 @@ static inline struct vm_area_struct *__vma_next(struct mm_struct *mm, return vma->vm_next; } -/* - * munmap_vma_range() - munmap VMAs that overlap a range. - * @mm: The mm struct - * @start: The start of the range. - * @len: The length of the range. - * @pprev: pointer to the pointer that will be set to previous vm_area_struct - * - * Find all the vm_area_struct that overlap from @start to - * @end and munmap them. Set @pprev to the previous vm_area_struct. - * - * Returns: -ENOMEM on munmap failure or 0 on success. - */ -static inline int -munmap_vma_range(struct mm_struct *mm, unsigned long start, unsigned long len, - struct vm_area_struct **pprev, struct list_head *uf) -{ - while (range_has_overlap(mm, start, start + len, pprev)) - if (do_munmap(mm, start, len, uf)) - return -ENOMEM; - return 0; -} - static unsigned long count_vma_pages_range(struct mm_struct *mm, unsigned long addr, unsigned long end) { @@ -649,6 +627,127 @@ static void __insert_vm_struct(struct mm_struct *mm, struct ma_state *mas, mm->map_count++; } +/* + * vma_expand - Expand an existing VMA + * + * @mas: The maple state + * @vma: The vma to expand + * @start: The start of the vma + * @end: The exclusive end of the vma + * @pgoff: The page offset of vma + * @next: The current of next vma. + * + * Expand @vma to @start and @end. Can expand off the start and end. Will + * expand over @next if it's different from @vma and @end == @next->vm_end. + * Checking if the @vma can expand and merge with @next needs to be handled by + * the caller. + * + * Returns: 0 on success + */ +inline int vma_expand(struct ma_state *mas, struct vm_area_struct *vma, + unsigned long start, unsigned long end, pgoff_t pgoff, + struct vm_area_struct *next) +{ + struct mm_struct *mm = vma->vm_mm; + struct address_space *mapping = NULL; + struct rb_root_cached *root = NULL; + struct anon_vma *anon_vma = vma->anon_vma; + struct file *file = vma->vm_file; + bool remove_next = false; + bool anon_cloned = false; + + if (next && (vma != next) && (end == next->vm_end)) { + remove_next = true; + if (next->anon_vma && !vma->anon_vma) { + int error; + + vma->anon_vma = next->anon_vma; + error = anon_vma_clone(vma, next); + if (error) + return error; + anon_cloned = true; + } + } + + /* Not merging but overwriting any part of next is not handled. */ + VM_BUG_ON(!remove_next && next != vma && end > next->vm_start); + /* Only handles expanding */ + VM_BUG_ON(vma->vm_start < start || vma->vm_end > end); + + if (mas_preallocate(mas, vma, GFP_KERNEL)) + goto nomem; + + vma_adjust_trans_huge(vma, start, end, 0); + + if (file) { + mapping = file->f_mapping; + root = &mapping->i_mmap; + uprobe_munmap(vma, vma->vm_start, vma->vm_end); + i_mmap_lock_write(mapping); + flush_dcache_mmap_lock(mapping); + vma_interval_tree_remove(vma, root); + } else if (anon_vma) { + anon_vma_lock_write(anon_vma); + anon_vma_interval_tree_pre_update_vma(vma); + } + + vma->vm_start = start; + vma->vm_end = end; + vma->vm_pgoff = pgoff; + /* Note: mas must be pointing to the expanding VMA */ + vma_mas_store(vma, mas); + + if (file) { + vma_interval_tree_insert(vma, root); + flush_dcache_mmap_unlock(mapping); + } + + /* Expanding over the next vma */ + if (remove_next) { + /* Remove from mm linked list - also updates highest_vm_end */ + __vma_unlink_list(mm, next); + + /* Kill the cache */ + vmacache_invalidate(mm); + + if (file) + __remove_shared_vm_struct(next, file, mapping); + + } else if (!next) { + mm->highest_vm_end = vm_end_gap(vma); + } + + if (anon_vma) { + anon_vma_interval_tree_post_update_vma(vma); + anon_vma_unlock_write(anon_vma); + } + + if (file) { + i_mmap_unlock_write(mapping); + uprobe_mmap(vma); + } + + if (remove_next) { + if (file) { + uprobe_munmap(next, next->vm_start, next->vm_end); + fput(file); + } + if (next->anon_vma) + anon_vma_merge(vma, next); + mm->map_count--; + mpol_put(vma_policy(next)); + vm_area_free(next); + } + + validate_mm(mm); + return 0; + +nomem: + if (anon_cloned) + unlink_anon_vmas(vma); + return -ENOMEM; +} + /* * We cannot adjust vm_start, vm_end, vm_pgoff fields of a vma that * is already present in an i_mmap tree without adjusting the tree. @@ -1649,9 +1748,15 @@ unsigned long mmap_region(struct file *file, unsigned long addr, struct list_head *uf) { struct mm_struct *mm = current->mm; - struct vm_area_struct *vma, *prev, *merge; - int error; + struct vm_area_struct *vma = NULL; + struct vm_area_struct *prev, *next; + pgoff_t pglen = len >> PAGE_SHIFT; unsigned long charged = 0; + unsigned long end = addr + len; + unsigned long merge_start = addr, merge_end = end; + pgoff_t vm_pgoff; + int error; + MA_STATE(mas, &mm->mm_mt, addr, end - 1); /* Check against address space limit. */ if (!may_expand_vm(mm, vm_flags, len >> PAGE_SHIFT)) { @@ -1661,16 +1766,17 @@ unsigned long mmap_region(struct file *file, unsigned long addr, * MAP_FIXED may remove pages of mappings that intersects with * requested mapping. Account for the pages it would unmap. */ - nr_pages = count_vma_pages_range(mm, addr, addr + len); + nr_pages = count_vma_pages_range(mm, addr, end); if (!may_expand_vm(mm, vm_flags, (len >> PAGE_SHIFT) - nr_pages)) return -ENOMEM; } - /* Clear old maps, set up prev and uf */ - if (munmap_vma_range(mm, addr, len, &prev, uf)) + /* Unmap any existing mapping in the area */ + if (do_munmap(mm, addr, len, uf)) return -ENOMEM; + /* * Private writable mapping: check memory availability */ @@ -1681,14 +1787,43 @@ unsigned long mmap_region(struct file *file, unsigned long addr, vm_flags |= VM_ACCOUNT; } - /* - * Can we just expand an old mapping? - */ - vma = vma_merge(mm, prev, addr, addr + len, vm_flags, - NULL, file, pgoff, NULL, NULL_VM_UFFD_CTX, NULL); - if (vma) - goto out; + next = mas_next(&mas, ULONG_MAX); + prev = mas_prev(&mas, 0); + if (vm_flags & VM_SPECIAL) + goto cannot_expand; + + /* Attempt to expand an old mapping */ + /* Check next */ + if (next && next->vm_start == end && !vma_policy(next) && + can_vma_merge_before(next, vm_flags, NULL, file, pgoff+pglen, + NULL_VM_UFFD_CTX, NULL)) { + merge_end = next->vm_end; + vma = next; + vm_pgoff = next->vm_pgoff - pglen; + } + + /* Check prev */ + if (prev && prev->vm_end == addr && !vma_policy(prev) && + (vma ? can_vma_merge_after(prev, vm_flags, vma->anon_vma, file, + pgoff, vma->vm_userfaultfd_ctx, NULL) : + can_vma_merge_after(prev, vm_flags, NULL, file, pgoff, + NULL_VM_UFFD_CTX , NULL))) { + merge_start = prev->vm_start; + vma = prev; + vm_pgoff = prev->vm_pgoff; + } + + + /* Actually expand, if possible */ + if (vma && + !vma_expand(&mas, vma, merge_start, merge_end, vm_pgoff, next)) { + khugepaged_enter_vma_merge(vma, vm_flags); + goto expanded; + } + mas.index = addr; + mas.last = end - 1; +cannot_expand: /* * Determine the object being mapped and call the appropriate * specific mapper. the address has already been validated, but @@ -1701,7 +1836,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, } vma->vm_start = addr; - vma->vm_end = addr + len; + vma->vm_end = end; vma->vm_flags = vm_flags; vma->vm_page_prot = vm_get_page_prot(vm_flags); vma->vm_pgoff = pgoff; @@ -1722,28 +1857,30 @@ unsigned long mmap_region(struct file *file, unsigned long addr, * * Answer: Yes, several device drivers can do it in their * f_op->mmap method. -DaveM - * Bug: If addr is changed, prev, rb_link, rb_parent should - * be updated for vma_link() */ WARN_ON_ONCE(addr != vma->vm_start); addr = vma->vm_start; + mas_reset(&mas); /* If vm_flags changed after call_mmap(), we should try merge vma again * as we may succeed this time. */ if (unlikely(vm_flags != vma->vm_flags && prev)) { - merge = vma_merge(mm, prev, vma->vm_start, vma->vm_end, vma->vm_flags, + next = vma_merge(mm, prev, vma->vm_start, vma->vm_end, vma->vm_flags, NULL, vma->vm_file, vma->vm_pgoff, NULL, NULL_VM_UFFD_CTX, NULL); - if (merge) { + if (next) { /* ->mmap() can change vma->vm_file and fput the original file. So * fput the vma->vm_file here or we would add an extra fput for file * and cause general protection fault ultimately. */ fput(vma->vm_file); vm_area_free(vma); - vma = merge; - /* Update vm_flags to pick up the change. */ + vma = prev; + /* Update vm_flags and possible addr to pick up the change. We don't + * warn here if addr changed as the vma is not linked by vma_link(). + */ + addr = vma->vm_start; vm_flags = vma->vm_flags; goto unmap_writable; } @@ -1767,7 +1904,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, goto free_vma; } - if (vma_link(mm, vma, prev)) { + if (mas_preallocate(&mas, vma, GFP_KERNEL)) { error = -ENOMEM; if (file) goto unmap_and_free_vma; @@ -1775,12 +1912,28 @@ unsigned long mmap_region(struct file *file, unsigned long addr, goto free_vma; } + if (vma->vm_file) + i_mmap_lock_write(vma->vm_file->f_mapping); + + vma_mas_store(vma, &mas); + __vma_link_list(mm, vma, prev); + mm->map_count++; + if (vma->vm_file) { + if (vma->vm_flags & VM_SHARED) + mapping_allow_writable(vma->vm_file->f_mapping); + + flush_dcache_mmap_lock(vma->vm_file->f_mapping); + vma_interval_tree_insert(vma, &vma->vm_file->f_mapping->i_mmap); + flush_dcache_mmap_unlock(vma->vm_file->f_mapping); + i_mmap_unlock_write(vma->vm_file->f_mapping); + } + /* Once vma denies write, undo our temporary denial count */ unmap_writable: if (file && vm_flags & VM_SHARED) mapping_unmap_writable(file->f_mapping); file = vma->vm_file; -out: +expanded: perf_event_mmap(vma); vm_stat_account(mm, vm_flags, len >> PAGE_SHIFT); @@ -1807,6 +1960,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, vma_set_page_prot(vma); + validate_mm(mm); return addr; unmap_and_free_vma: @@ -1823,6 +1977,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, unacct_error: if (charged) vm_unacct_memory(charged); + validate_mm(mm); return error; } @@ -2610,10 +2765,6 @@ int __do_munmap(struct mm_struct *mm, unsigned long start, size_t len, prev = vma->vm_prev; /* we have start < vma->vm_end */ - /* if it doesn't overlap, we have nothing.. */ - if (vma->vm_start >= end) - return 0; - /* * If we need to split any vma, do it now to save pain later. *