From patchwork Tue Jun 21 20:46:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Liam R. Howlett" X-Patchwork-Id: 12889915 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A305C43334 for ; Tue, 21 Jun 2022 23:27:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F23A58E0069; Tue, 21 Jun 2022 19:27:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ED2D88E0059; Tue, 21 Jun 2022 19:27:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D25A98E0069; Tue, 21 Jun 2022 19:27:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BF20F8E0059 for ; Tue, 21 Jun 2022 19:27:10 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8F52A32B0B for ; Tue, 21 Jun 2022 23:27:10 +0000 (UTC) X-FDA: 79603831020.07.017FF6F Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by imf18.hostedemail.com (Postfix) with ESMTP id F0AB91C000E for ; Tue, 21 Jun 2022 23:27:09 +0000 (UTC) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 25LJ29q2004726; Tue, 21 Jun 2022 20:47:14 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : references : in-reply-to : content-type : content-transfer-encoding : mime-version; s=corp-2021-07-09; bh=BzQ1A3niV148/8mjhkNKU+zXAywBC/GdDwVUKc+XAio=; b=XdWkfgfbdIMYAAi7/8Z5rxRmwebvhiAzoliGPHFxRiqEGztg5hJIAm/T5RgdOB6mJLC5 /v6sNaLbju6hj8L7B24SgWXBLIgbGSwA3M0i+cLXsv0XwE9gBG9ALZY3wQ30Q295apPP rXEt+BiUQdUsXuF7vVCkoChQSNb9tyoOkS3okEqeN6EBWzwoDCV+x8XbCFuFOatSjnzc H/qd8eRCMWHq9b2P7TqU/INt14pZ4vGX7RCqZl0KXAXiLCOKbwB3hNycqWqzewjPNEZK /74Jphxe9DNqn55m+TTePklB2jzTXvoQrM/cTHfxjWWU1kgadEfk11E8khXWiJGR4eRi FA== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3gs54cpnw3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 21 Jun 2022 20:47:14 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 25LKeUgq027828; Tue, 21 Jun 2022 20:47:13 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2176.outbound.protection.outlook.com [104.47.56.176]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3gth8wsp36-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 21 Jun 2022 20:47:13 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ipjHb7djJWab4u80LTFYBHemn07gQriToT5saiZ48tTgtV4VaxGEsDt7yoExG4TbmTBZxD4guC9pZhJKPjuzF9/yrihGThnanbzdjPBLCZ64QJZ/hFBprK8kDCNy2IuvRVpmCN72KF2xIeqDiecq/kAY9qJG+iLUQDi/2X8GfHIwozahBMDCb1bFzGFqUKQ55Kevfts+kn+C8nlJFECjFMci09sjn9OHkaaKacpwXfJ93eVCAera3zJGgADhmJZhTUb4Ed/WKTulZkhBQVcBGRzuZYQWRdBqqMAsjqcuFA7sMbVg6P9ilaNPvyFOvk0GcDSJcL9PTN4cL8QC0IAigw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BzQ1A3niV148/8mjhkNKU+zXAywBC/GdDwVUKc+XAio=; b=bC9VMNRMzsa/D20UCzUPE4vZUcjvBggjZHvzem+O8xlw/o4acEgw+pJGVuTABKRVsLsA22vhgtpzZtFfUUTJ0W+9cJ8n7IeT0cW/BKPR0kYXZ0NM8g8I4rVMqyrCqXYDh8zGRQ/VfWzFeaSY7yN3XNamjVY4yCaCg9TWd3F9fIlffAnHqNCTv0iEFcjiNa8+tFi+yvgRkYKYGFJpFtZGng52bwyjgrQvJExcZ0VZtuk/eJ792BZA8vmuf3NZi4qH/hBk3LPQXkHNFrbDw0XH1pDuPAv7nhe7OsoKCadC/T8RHjOuxZFmrNiEN/Dvk+Sqf//+cW4FUokYdPxHWf/New== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BzQ1A3niV148/8mjhkNKU+zXAywBC/GdDwVUKc+XAio=; b=Jn/rKPViKcJWFhXzPdolMh6OhP/x9M1TGbnc9bDdpKrgipv5+xgfUtU1E+Sx97bGTWL0QploKMCc0K+3BwCDgWbVwJvJNwCk+2ALKyJhHpd35oViZzIv14nbObyDUeVWqMX44VJuugDrb+KHR+onLW2lwdZw0RpJQO+H5MEKzV0= Received: from SN6PR10MB3022.namprd10.prod.outlook.com (2603:10b6:805:d8::25) by SN6PR10MB3085.namprd10.prod.outlook.com (2603:10b6:805:da::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5353.15; Tue, 21 Jun 2022 20:47:10 +0000 Received: from SN6PR10MB3022.namprd10.prod.outlook.com ([fe80::f59a:175d:d24:949c]) by SN6PR10MB3022.namprd10.prod.outlook.com ([fe80::f59a:175d:d24:949c%7]) with mapi id 15.20.5353.022; Tue, 21 Jun 2022 20:47:10 +0000 From: Liam Howlett To: "maple-tree@lists.infradead.org" , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" , Andrew Morton , "damon @ lists . linux . dev" , SeongJae Park , David Hildenbrand Subject: [PATCH v10 24/69] mm/mmap: use advanced maple tree API for mmap_region() Thread-Topic: [PATCH v10 24/69] mm/mmap: use advanced maple tree API for mmap_region() Thread-Index: AQHYhbAN+Sl6mjnOcEGnOIZ/LXvzQA== Date: Tue, 21 Jun 2022 20:46:59 +0000 Message-ID: <20220621204632.3370049-25-Liam.Howlett@oracle.com> References: <20220621204632.3370049-1-Liam.Howlett@oracle.com> In-Reply-To: <20220621204632.3370049-1-Liam.Howlett@oracle.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.35.1 x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 364f19d0-2e6a-4de4-c3a8-08da53c736a7 x-ms-traffictypediagnostic: SN6PR10MB3085:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: l7tdq0aOj5lazU9xmaPWIH4LK3SaIvH1OB+v2tUyYWBoB68DbXzVml0lnlJMHULXh4BROXmyL1ytmIBiJQf/IfRqy2JU102lNs91cP6FK+7WTp3N+ApZsZ1KH9GfZewlRU1bfU+nQd+3vEeetNE1ULi1wQNLoIWsRN6igv41xHH4mhLO4z6imBuPmQal5CN89NPh1y36K+UxdiGxIHIIKrJOYMI/t4MvOd3RZZRbevG3jTbg4YGNUmVnTLpcBINqUKvCVhd/1TqnVCa9JObyaChagCclHJWX20hfCp7ZZ4Q6BA97IF46bG3IQNuyViai7s/THZxnM+NwbQ+omwKLfaRBcPsKfeEIZOcqDTTNPx2nW0OFbmjEVOoe7YoZqT4KyklWL1tNziTLQZGp8jL3JGU29MAEgRSkHqyMLKtKOukHEB+k/5zOnGJjyqyPXmxaLUo8PcQiBTgRqh5vh0KTWg+WPoWYMtMEMwE6f0ofR2S3Uf50FMTPmR3IqlHBfth2PcCevygEe7hCzVOGukq3W6knCmfcWQ25GDsNf1A2Huo5AdWMZIo7YT2I2JP9LTSSvWm5WirMzc4lZOXQMG3XoIeQkcqv4rU/vp6av3EGvj6BrMnFZsJUybWFk1GHdjVpoLjR5Puqgx14YeiJvmXoOnNu6+xR0ooXX51q7aySXSWeGvOoCJDNp1JhjG55AxIQsLf5KOO2/oQi+xbmuxfHn5RU1TRJEjEO7IOLC4suVZlfdgBy+RYk0jR/RFxVLLmBPPINS7mkYRstI4OQRSOKB5ai0nqceumrRDldETG96TQWC7LzckikVH3REDs1tNaad7NrdmLGui7TshQ/nbIaQA== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SN6PR10MB3022.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(396003)(366004)(376002)(39860400002)(346002)(136003)(2616005)(66446008)(6486002)(76116006)(66556008)(966005)(36756003)(64756008)(83380400001)(478600001)(66476007)(1076003)(71200400001)(316002)(186003)(44832011)(5660300002)(8676002)(66946007)(6506007)(2906002)(30864003)(41300700001)(26005)(6666004)(86362001)(91956017)(6512007)(38070700005)(38100700002)(122000001)(110136005)(8936002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?q?Sm5umozM6aO56AN6mpSrmnG?= =?iso-8859-1?q?wqiK8JmEAJsR84mXPu4j3cLMBPW9p7AMdp/37hGl+1i96lGYQlRYAtDwAk6J?= =?iso-8859-1?q?HS8/PfwMJw2nHlfIkKMiYkZ6wRYGCuXHoACqJAWmhVL2/RPUQjiSylmE+yVe?= =?iso-8859-1?q?onwgDojsXYjOD+KommnXWyOKPUVtias9s5ZJpPAqWv3SdE3fDfv11pMOtD3z?= =?iso-8859-1?q?bAgJwt06g4N02jQS19sQhPZ8AgzPs6M2eAcwmmAtkRKvE+R+Jk3pIauorX54?= =?iso-8859-1?q?WPAoa8PQpSQ4kz3eBJmPBhp9x3Mp9r6YFHouiOcUGcxudu+RxoG/IXgmd44y?= =?iso-8859-1?q?mVlAwKgIyGFCQkLMP6klhFPr5Ru/hNl5CoD0IhmB1+erMs7Ea1hBbTBMsoe1?= =?iso-8859-1?q?mIk9r/yzrQCfumk2LjeVLPqg/SJmo/pM68kkZaIfrtEgWxnjkK4rPUE+4WE3?= =?iso-8859-1?q?KRaoiaOO3o9zMTMbBWTLYbytHdHIXrPHz4Ezxx56m95lfuplMpK8EjuTfIbG?= =?iso-8859-1?q?hNDPPwhtp9Utt3fZ6NgVTi4+XOTJxRYHaBemnPEeFasIExstLwdkdm43OqT7?= =?iso-8859-1?q?BNZbfbpkhEfew+/pGoVtRmLbETJOwHslFR6YGpen2EqK2G6+l/FIWa5eVfUD?= =?iso-8859-1?q?ovA16sBblUhiJEOOUQ6z/j9fY4x03MJN/x4ZckeCGHS+MwEWKOCk+ug4137x?= =?iso-8859-1?q?otrL374fyr7J5iFsJdA3aGgrbNYZScwdrCLBKAwJxZEZLR5HhQdv6SPFfApm?= =?iso-8859-1?q?QxnUMwXfEBhWGt68fu7L8zgTXaVlyP5eKyCztC6v/LYEj23MVRvcJBHGAYXz?= =?iso-8859-1?q?QSw6qLQ5mJ/GWhgR/a0IbOMyc+iSlxWCtEoF+b2aFzTn7NZoRmgHnoNs/lEM?= =?iso-8859-1?q?5jM2p32bBLsNOLjVXHxOegrnE8M+pcmA/pzs3WYYpdzcGpRFcaaY2Io0GGKm?= =?iso-8859-1?q?qukLE+UoNIGg41l0bhJqfER9UpCmrxxw1OpBBuSUkQ/E7XXQf/K51JlHdRXw?= =?iso-8859-1?q?ZJOgW+ByU+u55dwnXaHxnFQbXr8ftqUCf6rQ1SJGs+O8hXgWiPhxv+9O4wbW?= =?iso-8859-1?q?mYb/z+UgNVvACRWlZx6xJ/eDv+W8W8vHKRdwgtloORZe4UdWMDeSs/fVkoh8?= =?iso-8859-1?q?Df4ut0MtaJMc7RRlu1ne8n1LUrKzwU+oMSRIJ+mt8lrmUajmN2DSvE9GJGXA?= =?iso-8859-1?q?CIQ/+phfO8iIzsZOl2ikZfBDR2/G5sb8CcW62X69V6+btV9jzgKNmlNl2N9V?= =?iso-8859-1?q?vR4JtwS1Eltb6eOLCLU1g91O8nLur0SyXr6y517xFHpZrATlTFpaJOCKM+sV?= =?iso-8859-1?q?3qLcNcFQpjf/3sbLUCz7aFFYRRTHlYF/oYy6SQAM7qV0HgF0Jgo+abwewzh5?= =?iso-8859-1?q?eorwpQr6EIvM0CBxP+PoxGxh06osH9sR/X+HWbSNPaglq30/kvvFfOerb8lT?= =?iso-8859-1?q?H/oHZpcF3eEyk8Whi3J8IA7ZhxQybO0kjSaKO+QNbE54oHcOTR/brUD14aLk?= =?iso-8859-1?q?CPSRrwWnW2LOJHahD1Op4s4brmOkAYK80WRrIFkDIE2Fr9PME//6iUtn9Epv?= =?iso-8859-1?q?YpVLCa4FT7VKDawNBDkEU3iB3Kpb96vC6ZIYwLIQDa3JUpWhVMYmtbibLPps?= =?iso-8859-1?q?Cf/ShMdGeua0R89aBgQYstzj5tWEyGYutxjMWBjzfFOOlOG1HoFcdez3d94i?= =?iso-8859-1?q?3tF69y5i/cj/MuiTjCY67K/Sij2bKpV88SObw8EJL7qz4t7Pw3sbFesLfqTF?= =?iso-8859-1?q?fouR6nCqVyu10Z1mFIgFydB8tvTErB6324agK9o0mSvwVmCT9rR69aGN/i+Y?= =?iso-8859-1?q?A1epuDdk=3D?= MIME-Version: 1.0 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SN6PR10MB3022.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 364f19d0-2e6a-4de4-c3a8-08da53c736a7 X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Jun 2022 20:46:59.0509 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 7o2daqBcgS8+xfmyIfw3Xo4KMy73/4xMXHIUUdIyeWp6/8evS7ou7Ait8/Q5vhKKBshwnn2z30QniI+TD0WOXw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN6PR10MB3085 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.517,18.0.883 definitions=2022-06-21_09:2022-06-21,2022-06-21 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 malwarescore=0 mlxscore=0 phishscore=0 adultscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2204290000 definitions=main-2206210087 X-Proofpoint-GUID: FYJybcgbce8Y6aAr3Ubx4wjq7EyO_u0p X-Proofpoint-ORIG-GUID: FYJybcgbce8Y6aAr3Ubx4wjq7EyO_u0p ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1655854030; a=rsa-sha256; cv=pass; b=nDsIZd7g8RTXx3wvFa7jT7iSCyczeHofZyiouiUBUAuRCVCbPKueHLvvow+UCsP7P/RXWz 4Wu3DiDnPTQZIrNvDYDfpSTQ9P51rNkGnyxVYVcFWEBu0VsXA/I5CkQY0oRdOJo/1pSqzE NBXuAXT3QlrfCWo8CHKrYXezGDtrb/M= ARC-Authentication-Results: i=2; imf18.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=XdWkfgfb; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b="Jn/rKPVi"; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf18.hostedemail.com: domain of liam.howlett@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=liam.howlett@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655854030; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BzQ1A3niV148/8mjhkNKU+zXAywBC/GdDwVUKc+XAio=; b=I0dN2rq0lyHkRhEHVoszxaFHKpWMwmd9loSHE3jFPsoX3/5LM3XZ+njBNX4OWTQNARF+yx 7kI4Zec0V6TSXqMHGbUwljEtrEvQBTYt716HvTvD9itjyZf/OFuiYhg3Zi6CMJqwk5zBIJ WnQN5AC45YiiF6wsWZEXIE2xV+nR9WU= X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: F0AB91C000E X-Stat-Signature: oghiw8yu5p4k3krsnm4bbrzcydt5shik X-Rspam-User: Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=oracle.com header.s=corp-2021-07-09 header.b=XdWkfgfb; dkim=pass header.d=oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b="Jn/rKPVi"; dmarc=pass (policy=none) header.from=oracle.com; spf=none (imf18.hostedemail.com: domain of liam.howlett@oracle.com has no SPF policy when checking 205.220.177.32) smtp.mailfrom=liam.howlett@oracle.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") X-HE-Tag: 1655854029-232286 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" Changing mmap_region() to use the maple tree state and the advanced maple tree interface allows for a lot less tree walking. This change removes the last caller of munmap_vma_range(), so drop this unused function. Add vma_expand() to expand a VMA if possible by doing the necessary hugepage check, uprobe_munmap of files, dcache flush, modifications then undoing the detaches, etc. Link: https://lkml.kernel.org/r/20220504011345.662299-9-Liam.Howlett@oracle.com Link: https://lkml.kernel.org/r/20220519020341.rr3s6b4dr7o36cqb@revolver Signed-off-by: Liam R. Howlett Cc: Catalin Marinas Cc: David Howells Cc: "Matthew Wilcox (Oracle)" Cc: SeongJae Park Cc: Vlastimil Babka Cc: Will Deacon Cc: Davidlohr Bueso Signed-off-by: Andrew Morton --- mm/mmap.c | 252 +++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 204 insertions(+), 48 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 9afe51a7db6c..d6549a74e73e 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -516,28 +516,6 @@ static inline struct vm_area_struct *__vma_next(struct mm_struct *mm, return vma->vm_next; } -/* - * munmap_vma_range() - munmap VMAs that overlap a range. - * @mm: The mm struct - * @start: The start of the range. - * @len: The length of the range. - * @pprev: pointer to the pointer that will be set to previous vm_area_struct - * - * Find all the vm_area_struct that overlap from @start to - * @end and munmap them. Set @pprev to the previous vm_area_struct. - * - * Returns: -ENOMEM on munmap failure or 0 on success. - */ -static inline int -munmap_vma_range(struct mm_struct *mm, unsigned long start, unsigned long len, - struct vm_area_struct **pprev, struct list_head *uf) -{ - while (range_has_overlap(mm, start, start + len, pprev)) - if (do_munmap(mm, start, len, uf)) - return -ENOMEM; - return 0; -} - static unsigned long count_vma_pages_range(struct mm_struct *mm, unsigned long addr, unsigned long end) { @@ -664,6 +642,130 @@ static void __insert_vm_struct(struct mm_struct *mm, struct ma_state *mas, mm->map_count++; } +/* + * vma_expand - Expand an existing VMA + * + * @mas: The maple state + * @vma: The vma to expand + * @start: The start of the vma + * @end: The exclusive end of the vma + * @pgoff: The page offset of vma + * @next: The current of next vma. + * + * Expand @vma to @start and @end. Can expand off the start and end. Will + * expand over @next if it's different from @vma and @end == @next->vm_end. + * Checking if the @vma can expand and merge with @next needs to be handled by + * the caller. + * + * Returns: 0 on success + */ +inline int vma_expand(struct ma_state *mas, struct vm_area_struct *vma, + unsigned long start, unsigned long end, pgoff_t pgoff, + struct vm_area_struct *next) +{ + struct mm_struct *mm = vma->vm_mm; + struct address_space *mapping = NULL; + struct rb_root_cached *root = NULL; + struct anon_vma *anon_vma = vma->anon_vma; + struct file *file = vma->vm_file; + bool remove_next = false; + bool anon_cloned = false; + + if (next && (vma != next) && (end == next->vm_end)) { + remove_next = true; + if (next->anon_vma && !vma->anon_vma) { + int error; + + vma->anon_vma = next->anon_vma; + error = anon_vma_clone(vma, next); + if (error) + return error; + anon_cloned = true; + } + } + + /* Not merging but overwriting any part of next is not handled. */ + VM_BUG_ON(next && !remove_next && next != vma && end > next->vm_start); + /* Only handles expanding */ + VM_BUG_ON(vma->vm_start < start || vma->vm_end > end); + + if (mas_preallocate(mas, vma, GFP_KERNEL)) + goto nomem; + + vma_adjust_trans_huge(vma, start, end, 0); + + if (anon_vma) { + anon_vma_lock_write(anon_vma); + anon_vma_interval_tree_pre_update_vma(vma); + } + + if (file) { + mapping = file->f_mapping; + root = &mapping->i_mmap; + uprobe_munmap(vma, vma->vm_start, vma->vm_end); + i_mmap_lock_write(mapping); + flush_dcache_mmap_lock(mapping); + vma_interval_tree_remove(vma, root); + } + + vma->vm_start = start; + vma->vm_end = end; + vma->vm_pgoff = pgoff; + /* Note: mas must be pointing to the expanding VMA */ + vma_mas_store(vma, mas); + + if (file) { + vma_interval_tree_insert(vma, root); + flush_dcache_mmap_unlock(mapping); + } + + /* Expanding over the next vma */ + if (remove_next) { + /* Remove from mm linked list - also updates highest_vm_end */ + __vma_unlink_list(mm, next); + + /* Kill the cache */ + vmacache_invalidate(mm); + + if (file) + __remove_shared_vm_struct(next, file, mapping); + + } else if (!next) { + mm->highest_vm_end = vm_end_gap(vma); + } + + if (file) { + i_mmap_unlock_write(mapping); + uprobe_mmap(vma); + } + + if (anon_vma) { + anon_vma_interval_tree_post_update_vma(vma); + anon_vma_unlock_write(anon_vma); + } + + + if (remove_next) { + if (file) { + uprobe_munmap(next, next->vm_start, next->vm_end); + fput(file); + } + if (next->anon_vma) + anon_vma_merge(vma, next); + mm->map_count--; + mpol_put(vma_policy(next)); + vm_area_free(next); + } + + validate_mm(mm); + return 0; + +nomem: + if (anon_cloned) + unlink_anon_vmas(vma); + return -ENOMEM; +} + /* * We cannot adjust vm_start, vm_end, vm_pgoff fields of a vma that * is already present in an i_mmap tree without adjusting the tree. @@ -1665,9 +1767,15 @@ unsigned long mmap_region(struct file *file, unsigned long addr, struct list_head *uf) { struct mm_struct *mm = current->mm; - struct vm_area_struct *vma, *prev, *merge; - int error; + struct vm_area_struct *vma = NULL; + struct vm_area_struct *next, *prev, *merge; + pgoff_t pglen = len >> PAGE_SHIFT; unsigned long charged = 0; + unsigned long end = addr + len; + unsigned long merge_start = addr, merge_end = end; + pgoff_t vm_pgoff; + int error; + MA_STATE(mas, &mm->mm_mt, addr, end - 1); /* Check against address space limit. */ if (!may_expand_vm(mm, vm_flags, len >> PAGE_SHIFT)) { @@ -1677,16 +1785,17 @@ unsigned long mmap_region(struct file *file, unsigned long addr, * MAP_FIXED may remove pages of mappings that intersects with * requested mapping. Account for the pages it would unmap. */ - nr_pages = count_vma_pages_range(mm, addr, addr + len); + nr_pages = count_vma_pages_range(mm, addr, end); if (!may_expand_vm(mm, vm_flags, (len >> PAGE_SHIFT) - nr_pages)) return -ENOMEM; } - /* Clear old maps, set up prev and uf */ - if (munmap_vma_range(mm, addr, len, &prev, uf)) + /* Unmap any existing mapping in the area */ + if (do_munmap(mm, addr, len, uf)) return -ENOMEM; + /* * Private writable mapping: check memory availability */ @@ -1697,14 +1806,43 @@ unsigned long mmap_region(struct file *file, unsigned long addr, vm_flags |= VM_ACCOUNT; } - /* - * Can we just expand an old mapping? - */ - vma = vma_merge(mm, prev, addr, addr + len, vm_flags, - NULL, file, pgoff, NULL, NULL_VM_UFFD_CTX, NULL); - if (vma) - goto out; + next = mas_next(&mas, ULONG_MAX); + prev = mas_prev(&mas, 0); + if (vm_flags & VM_SPECIAL) + goto cannot_expand; + + /* Attempt to expand an old mapping */ + /* Check next */ + if (next && next->vm_start == end && !vma_policy(next) && + can_vma_merge_before(next, vm_flags, NULL, file, pgoff+pglen, + NULL_VM_UFFD_CTX, NULL)) { + merge_end = next->vm_end; + vma = next; + vm_pgoff = next->vm_pgoff - pglen; + } + + /* Check prev */ + if (prev && prev->vm_end == addr && !vma_policy(prev) && + (vma ? can_vma_merge_after(prev, vm_flags, vma->anon_vma, file, + pgoff, vma->vm_userfaultfd_ctx, NULL) : + can_vma_merge_after(prev, vm_flags, NULL, file, pgoff, + NULL_VM_UFFD_CTX , NULL))) { + merge_start = prev->vm_start; + vma = prev; + vm_pgoff = prev->vm_pgoff; + } + + + /* Actually expand, if possible */ + if (vma && + !vma_expand(&mas, vma, merge_start, merge_end, vm_pgoff, next)) { + khugepaged_enter_vma(vma, vm_flags); + goto expanded; + } + mas.index = addr; + mas.last = end - 1; +cannot_expand: /* * Determine the object being mapped and call the appropriate * specific mapper. the address has already been validated, but @@ -1717,7 +1855,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, } vma->vm_start = addr; - vma->vm_end = addr + len; + vma->vm_end = end; vma->vm_flags = vm_flags; vma->vm_page_prot = vm_get_page_prot(vm_flags); vma->vm_pgoff = pgoff; @@ -1738,28 +1876,32 @@ unsigned long mmap_region(struct file *file, unsigned long addr, * * Answer: Yes, several device drivers can do it in their * f_op->mmap method. -DaveM - * Bug: If addr is changed, prev, rb_link, rb_parent should - * be updated for vma_link() */ WARN_ON_ONCE(addr != vma->vm_start); addr = vma->vm_start; + mas_reset(&mas); - /* If vm_flags changed after call_mmap(), we should try merge vma again - * as we may succeed this time. + /* + * If vm_flags changed after call_mmap(), we should try merge + * vma again as we may succeed this time. */ if (unlikely(vm_flags != vma->vm_flags && prev)) { merge = vma_merge(mm, prev, vma->vm_start, vma->vm_end, vma->vm_flags, NULL, vma->vm_file, vma->vm_pgoff, NULL, NULL_VM_UFFD_CTX, NULL); if (merge) { - /* ->mmap() can change vma->vm_file and fput the original file. So - * fput the vma->vm_file here or we would add an extra fput for file - * and cause general protection fault ultimately. + /* + * ->mmap() can change vma->vm_file and fput + * the original file. So fput the vma->vm_file + * here or we would add an extra fput for file + * and cause general protection fault + * ultimately. */ fput(vma->vm_file); vm_area_free(vma); vma = merge; /* Update vm_flags to pick up the change. */ + addr = vma->vm_start; vm_flags = vma->vm_flags; goto unmap_writable; } @@ -1783,7 +1925,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, goto free_vma; } - if (vma_link(mm, vma, prev)) { + if (mas_preallocate(&mas, vma, GFP_KERNEL)) { error = -ENOMEM; if (file) goto unmap_and_free_vma; @@ -1791,6 +1933,22 @@ unsigned long mmap_region(struct file *file, unsigned long addr, goto free_vma; } + if (vma->vm_file) + i_mmap_lock_write(vma->vm_file->f_mapping); + + vma_mas_store(vma, &mas); + __vma_link_list(mm, vma, prev); + mm->map_count++; + if (vma->vm_file) { + if (vma->vm_flags & VM_SHARED) + mapping_allow_writable(vma->vm_file->f_mapping); + + flush_dcache_mmap_lock(vma->vm_file->f_mapping); + vma_interval_tree_insert(vma, &vma->vm_file->f_mapping->i_mmap); + flush_dcache_mmap_unlock(vma->vm_file->f_mapping); + i_mmap_unlock_write(vma->vm_file->f_mapping); + } + /* * vma_merge() calls khugepaged_enter_vma() either, the below * call covers the non-merge case. @@ -1802,7 +1960,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, if (file && vm_flags & VM_SHARED) mapping_unmap_writable(file->f_mapping); file = vma->vm_file; -out: +expanded: perf_event_mmap(vma); vm_stat_account(mm, vm_flags, len >> PAGE_SHIFT); @@ -1829,6 +1987,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, vma_set_page_prot(vma); + validate_mm(mm); return addr; unmap_and_free_vma: @@ -1845,6 +2004,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, unacct_error: if (charged) vm_unacct_memory(charged); + validate_mm(mm); return error; } @@ -2661,10 +2821,6 @@ int __do_munmap(struct mm_struct *mm, unsigned long start, size_t len, prev = vma->vm_prev; /* we have start < vma->vm_end */ - /* if it doesn't overlap, we have nothing.. */ - if (vma->vm_start >= end) - return 0; - /* * If we need to split any vma, do it now to save pain later. *