From patchwork Tue Jan 17 15:58:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13104843 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33369C3DA78 for ; Tue, 17 Jan 2023 15:58:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 012EF6B007D; Tue, 17 Jan 2023 10:58:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E8DB16B0081; Tue, 17 Jan 2023 10:58:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C44B36B007D; Tue, 17 Jan 2023 10:58:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id AEFCB6B007D for ; Tue, 17 Jan 2023 10:58:55 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 6064AC0853 for ; Tue, 17 Jan 2023 15:58:55 +0000 (UTC) X-FDA: 80364749430.22.E78E5D8 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2040.outbound.protection.outlook.com [40.107.236.40]) by imf09.hostedemail.com (Postfix) with ESMTP id CE8C114001C for ; Tue, 17 Jan 2023 15:58:52 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=BWOhDtMF; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf09.hostedemail.com: domain of jgg@nvidia.com designates 40.107.236.40 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673971132; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=B50ohIdvhDbFwyWVcAFTxgU9fHhVhqr+gvZ/3zGe7iI=; b=AGXYWsfMJNdd/iEdzGne6LJtGYlj8YxlrfBEBOkCKK6sw7e+FEnnU61Mcm0Xyof03w8zus nZZyn0lkYrO7dzJln6OvqNSA9NEK3ry0zg16WCVpwjBSC2kZrW5DpbrfxI/jPVHZ1Lzl3d E9mD4pQrA/00dwHe04lI55/phcEadvw= ARC-Authentication-Results: i=2; imf09.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=BWOhDtMF; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf09.hostedemail.com: domain of jgg@nvidia.com designates 40.107.236.40 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1673971132; a=rsa-sha256; cv=pass; b=gf12oXg73V05tlRGcYM+sHtu2RgY/OpxNuZicrkXdq2614MFHjDy/1RlQuQhNgopTEJ8m5 ennxTz1J6LG5d6nVLF1e9LbIGxwjZ5rOWXwaooKyjyPVtFasb4EpYhCaCZt5qRWcgiHP4D jIrYZorH+ai9Ta0XUdmvAytPwKWjvTc= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=B4JDQc60m8qbzJ9VGNJopz2cpsZmk1IclA6paeo4Me7zGgEppuEie2mx6aeB261KCasmVlh+/3KRi6NO7EAD1xyL7W6ercHWgM0r68PmP6qWmTbfROpQl4hwJHC8TZct69dVpjIuWJOZA848XsEN1yZUnShnz7NcZZ6EASvQUX0BmXKCaZkOSRlFoCv7ZHclGAL3jxAkbRBo+BcajYMjJlDrX6wkTNO/boDqpi+RwICfOUYA8HWbC4CqxTkBEGBXarY97MDWSELQkGluuy99m5ILT2yqGpk9M6msPsVP5CYEWvuYCcnP2Aztnn8I0dvmjxQPEK2Azna2XFPKoBNCyw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=B50ohIdvhDbFwyWVcAFTxgU9fHhVhqr+gvZ/3zGe7iI=; b=Eqc85Z7UjCXpQ0iNYT9GpkIqf0RzPxjpNiYzsPoxe8kaPOJJZRtDop9iXKorj++EyCHCvd4fgtGKdEkzvzz1fHTSrnt8KFIhVLRyn/svQiGjbtCRFR7r0L51mio70peWUYsWMPIots6qhLKqCiZkHAXQh96SpsJ+oB6/C0j7ScJkddW3//PpNKZ9LnuNKiYjXB0l/f2zoKAI5y1G2XToEVNrjqhF61osOewfJjbBDTE4MBRVCL+NpZz5xL6TrepjFdcC5vklyKKxPfruCztgn07UT3puJAVh8KckZTcZI6YzsJ1HxkAA0eK7tUkgqSaL0XJvWV+he3DcewgN3HGK9w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=B50ohIdvhDbFwyWVcAFTxgU9fHhVhqr+gvZ/3zGe7iI=; b=BWOhDtMFP/0ZmESvlRYXeWkqeSzKEqyVxmtc6WydzMV8S7QsGRzxW157gqgdHVxZ+dlYtVWyKTMHdYXxvXy8RHBVbkmjUhhOoEoO0epjDxM9sg1Q2FrJVQJoxLhrJLCB3BEvCG5Uvac3Egnh0l2xQApJAcvwbK2q2avpmhtdimqW984M9b+9P+m2ILQZ/JKWGwPjqe1QyCep6d0yLtxHFXZTB8ibfEuNeWPcV5CBSQuycZ76tpf9+ah1Y+NWlEU7Jfb/7/o6pVoZZCBdO49J6EXy+f3m7eCeLQJ7Y7SpKWBNquxHgkqnOjFKHJ7fsw3LNZo83kxn0XkBY232u56Mqw== Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8334.namprd12.prod.outlook.com (2603:10b6:208:3ff::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.23; Tue, 17 Jan 2023 15:58:44 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::f8b0:df13:5f8d:12a]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::f8b0:df13:5f8d:12a%9]) with mapi id 15.20.6002.013; Tue, 17 Jan 2023 15:58:44 +0000 From: Jason Gunthorpe To: Cc: Alistair Popple , John Hubbard , linux-mm@kvack.org Subject: [PATCH 1/8] mm/gup: have internal functions get the mmap_read_lock() Date: Tue, 17 Jan 2023 11:58:32 -0400 Message-Id: <1-v1-dd94f8f0d5ad+716-gup_tidy_jgg@nvidia.com> In-Reply-To: <0-v1-dd94f8f0d5ad+716-gup_tidy_jgg@nvidia.com> References: X-ClientProxiedBy: CP5P284CA0103.BRAP284.PROD.OUTLOOK.COM (2603:10d6:103:92::14) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8334:EE_ X-MS-Office365-Filtering-Correlation-Id: c0570b53-1ca8-45a5-c427-08daf8a3b61b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: tXi2v9BDQo4Esv0thXPcaq7D2sxSN+s6aVPJXhlzIp+406mqKePfhTlaDHG654X+DbrSeVLJuDBznj2ddW+CQwJ7hVFCEXTUrsZXClwjqwM24dqvUj/y4SxvsS68PPTrY7Kvb6CXARtOQ2WzrgUUzMU9zqTfY5bRBOujX3Ra+UmKjxh6EZN+0dMllM3907iuNt5hqCNXkNmj4mMhZG+V09daD1xVsn3btRIRgfjuz9jpK4VAIxHuxHa2JDIQYulU3dY7TNkojD/Q6U6mJrJSeqFxo2fZ4W+LAvfmDn3PbOzmuIkCVAILDeau1RQI2Na8uzCSrNI5QG6NqtyGBvgiwrvQPwqvlkgfE40PP/6K5l3IZ67f8AHbJhh2hRbJlemY/41l1jTKBzXN7eb56ZiVttfl6B3FvyJqIvyvF4ozxak7TLgBUHHiWXeTO9n4ZgOQI2MkT0pfB8x0yukIblFYjfnsFSFaDZQhpyTQFH6+mnpEmCZ5V/EdWFLLsEfbx7+sWblml0jmPcVrFf6MUUxAHO4uHCS6ar289rgOWlY3WyEfCF/Wb/bjSLF+20KJRMjQ+LGa0thhFV7GztNiPDndgUd1DmkpIovgXiGWw0sKSId5y/m1IMjxgib+t0yQsNaAZ9R3mhEBA1x3VgUHYFKqfWlVKxfgyHPWMg8cRWcXri78dw6ErkzIa8pUrit7+eBnSBQ7wvqPHPMxMRjdpRo9DW0kDlp+1/BUHKBjEYCceJw= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(396003)(366004)(346002)(136003)(376002)(39860400002)(109986013)(451199015)(36756003)(86362001)(2616005)(66556008)(4326008)(26005)(8676002)(6512007)(186003)(66946007)(66476007)(41300700001)(83380400001)(6666004)(6506007)(316002)(54906003)(478600001)(2906002)(6486002)(38100700002)(8936002)(5660300002)(4216001)(266003)(15583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: iftoO33dCsMtfFsEPctFtMgyEunWc+VaeYRwrfFnNT8NGdkVuwsC3bTl5SiQ+eB+GmXe+sK8vGg8NA+BzAX15UfjHEqmPwKz4FHLS/M3UkomWTgiX90GFyhMoC/cTtGoDKcrilFVcMrTJgmsH7btBZpz4/iGxiBKB+8K8EnTi7PXe2wKSoqykOYsXB7GboBpW15aCXEz66cduAYe9tSPxKAQDWsU+vJS3oNUIC48AeSorSsi0m2gmWxHEiO5K01aTij9ratv9rESt6xRI0nFTKqLy/p1Srh+SdsE7kDnnwA3GDk/7uHXpKfUIzbf9+yOvwRlFWHiMdC/jbZnTX5RoM/rq97nSiBtNbFkLWh/4YYXGQTE4sQPLO/tNFd/TsXbOeqdkiZ2edAiXvSudpSG5Prvmddd/y3o5uI5U4Gd1sp95IfOeXG7BFdqtPgQe+4gl2pSCR1bQFWafc/PFRTQCtdUziehADeQZdzLABpbg0HiR4Z2QJpoAn38C9Vm6lPbXT1MKVIkgFf5P1kN2/gV1e8e8FUhAHwONukTQTU0XiFBXgNibX16TH0YWx8VhpreUKD5URdypFDtn0MMAcWjPHwL2ShaUSXidBkKCsaytfipUudOY6FAtaSGbN7uSYs7F+63bzC8SQNwqahWyoURC2QXUp6+WuLdwJWT4GewpknQfbwZ+ErRaO2E93KGVF40gprVDDCAcSJo5clgX2339QRt7uEqHieSearKpZ60Fp9qmNK1Z1R0Ey8P8kiSgjH9runHrMzEQjcn5yKKPM/m5IanGrR3+ULY/LWKUjDHug/PIj5+QWUDN9Bs/TOslswtOzM4t2kxo0Ud0MHb7zWkZkUvSbjE8CnmC8BuW2Q6zYZgDyiSylY0byosC4aCR/q1NvrJZYc9r2/VlsoGAVxozhwFYCCeZZ+F0QKpqOLYpjIdhM//c/4O1Lu/IAXwX1UJli5X6FCc0y0RgKhPJ6zrcwvVzR/hHT6Z2la2MI4L9BHguIQXcbcFIxWrW/keTogZKvA9WpsxKoF9nz1gQPlfsmCNQQZSNXAXhTw0AXZXwuovIXs6AsjOFsjDFZZwT4tsxAO4qq1qCyYauMUNkJjlkYOFcyErZLSOL/unJ08JTPqxYkU+UeQq/QM/Ih5MNkKO1IWvjO14hH1h4Taef1Nbkm4myo17jeHUKkd9qmpW5DE8ptxlvS5g8LraI2rZZVhPeO6+GT2OlepzIcjUqA9H1PXGch5rCnqLnGRAZNpuutuAxfdB3+PiBShiEtlVhg1JyyARMCOO8kNB2JOX93fwQ/icJdC8jqU7H0VN6uBg0V13KUZYUBX4Uwjb4fJuv+idtT6vgWl17pORQJIPfp7ZmZznixHAS65JmHzmPcCPD8MkRcdX+h0YrFA78qq/ivHiQFazqxv/WHAi/zcGxBR7lf0opfN2ckyIwcNV/L+BM0iBQPY5D5lasf2nngEu4tQm0GvSEMWFbzf5eP1X0fjwXpejAVzgHJDVzqx6XWzIF2WQ8jnW0mby0pf0x/cvi8jmw0erUkwSAxreVlM/XtvwcJlUmDsM8HO+tps0IXEJhQUbLkYAS6R+pLlTuCwijL5A X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: c0570b53-1ca8-45a5-c427-08daf8a3b61b X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jan 2023 15:58:44.6395 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: utete+pJ9AiqKu2yVRSjB0K7KQ5fPNfym7ADgo4W4+6Asm/fuD3pIqHyHJfoLfIE X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8334 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: 75595jm4pzu9wtytypqrdsi6j8i1jwgi X-Rspamd-Queue-Id: CE8C114001C X-HE-Tag: 1673971132-901857 X-HE-Meta: U2FsdGVkX1+fDXPPHUd8kMQyzTfyCdjKSpBqHAYD66HMSQjtWguecljClT6Wr3lE2EjzgFQYgoJnXwLjIvRdhywPc053oTWVXZKX1D01g6xwkdk69yrJaGc6j6Z3rqgBzbFcHQ6c1LwfGa8UtpW8K+Qm2D0uOUkR6dZi7flBpAUysAuroXlCLbx7HRBCywOzVrvWZe+ZKU2808kkmU+FDlJDxyIEcCb+qNPtW8x02PPtixw6Ju6XlNnWYvi0fRbbCd4ksEHseYRDvcZaAuljhRJ/0gBXh1rpJ4syKVsJvfB/YW7yom0NEMWz+Ai39iyGftzFEdqfQquZ+ZGrSQBe1SotO/9B2atG6A5gWOTqUCxe2588TKg1GHt6AdpiYWGOlXSCvhpDhp/vXHibGAGPJoxoPeRqbjx510md+vgk3n4MiqHtJ/UPTs6+uOYZbDnTcy2pVuvOuNxhXTUoWrimEQxWJHYxHQR3HlzL6MG0NL/kHqT0hj9nVfo8D+N852DpvwYr2yUZYQnT2ytHYAtg26AhJBVPvBDdS9muLNRekWFuYIUnItv7qhBih9xTPllI6j42NSwbI5mh94ktBBmKAWGQ9PD8zuXHAxmnlIfJplmsqDAmF91wumL8QIO75SmGqOC0ckxJ0gZtRdLxEYnfW0VLs8pPTnEhV6wJz+GDNvorSlt9bkJ6USMPFxUjoA2qa2f6zUokUpUInWLbYk1yumH+BpSQ5+IPIbEf1C3cV2UELg3G8+Rr1IWG/+KBldIb9ch16PbUrF4rfm3Gm4PIf5SP9L6kD2dDrdAw/4oF8//ti6t1aqZnfb08oVvu8R93Wm18Xx5uzTAT9LK2FzW6emxERybjgKoOEUxWuO1X+QYlQCvMNfu+dnde/gYAK/KbukNNdvgY5fFVknH1dYNid5081LLFZ9qeYdB4EYxOvetM9rfERc5ejwns9xuBz/xEgoFIYZtl9u1IQG3PvtA F2NpwNno d7C1u93CSRGRhmIrSj2/3AaSp794xXJKxD8w+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: __get_user_pages_locked() and __gup_longterm_locked() both require the mmap lock to be held. They have a slightly unusual locked parameter that is used to allow these functions to unlock and relock the mmap lock and convey that fact to the caller. Several places wrapper these functions with a simple mmap_read_lock() just so they can follow the optimized locked protocol. Consolidate this internally to the functions. Allow internal callers to set locked = 0 to cause the functions to obtain and release the lock on their own. Reorganize __gup_longterm_locked() to use the autolocking in __get_user_pages_locked(). Replace all the places obtaining the mmap_read_lock() just to call __get_user_pages_locked() with the new mechanism. Replace all the internal callers of get_user_pages_unlocked() with direct calls to __gup_longterm_locked() using the new mechanism. A following patch will add assertions ensuring the external interface continues to always pass in locked = 1. Signed-off-by: Jason Gunthorpe --- mm/gup.c | 92 +++++++++++++++++++++++++++++--------------------------- 1 file changed, 47 insertions(+), 45 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index f45a3a5be53a48..3a9f764165f50b 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1343,13 +1343,22 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm, unsigned int flags) { long ret, pages_done; - bool lock_dropped; + bool lock_dropped = false; if (locked) { /* if VM_FAULT_RETRY can be returned, vmas become invalid */ BUG_ON(vmas); - /* check caller initialized locked */ - BUG_ON(*locked != 1); + } + + /* + * The internal caller expects GUP to manage the lock internally and the + * lock must be released when this returns. + */ + if (locked && !*locked) { + if (mmap_read_lock_killable(mm)) + return -EAGAIN; + lock_dropped = true; + *locked = 1; } if (flags & FOLL_PIN) @@ -1368,7 +1377,6 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm, flags |= FOLL_GET; pages_done = 0; - lock_dropped = false; for (;;) { ret = __get_user_pages(mm, start, nr_pages, flags, pages, vmas, locked); @@ -1659,9 +1667,24 @@ static long __get_user_pages_locked(struct mm_struct *mm, unsigned long start, unsigned int foll_flags) { struct vm_area_struct *vma; + bool must_unlock = false; unsigned long vm_flags; long i; + if (!nr_pages) + return 0; + + /* + * The internal caller expects GUP to manage the lock internally and the + * lock must be released when this returns. + */ + if (locked && !*locked) { + if (mmap_read_lock_killable(mm)) + return -EAGAIN; + must_unlock = true; + *locked = 1; + } + /* calculate required read or write permissions. * If FOLL_FORCE is set, we only require the "MAY" flags. */ @@ -1673,12 +1696,12 @@ static long __get_user_pages_locked(struct mm_struct *mm, unsigned long start, for (i = 0; i < nr_pages; i++) { vma = find_vma(mm, start); if (!vma) - goto finish_or_fault; + break; /* protect what we can, including chardevs */ if ((vma->vm_flags & (VM_IO | VM_PFNMAP)) || !(vm_flags & vma->vm_flags)) - goto finish_or_fault; + break; if (pages) { pages[i] = virt_to_page((void *)start); @@ -1690,9 +1713,11 @@ static long __get_user_pages_locked(struct mm_struct *mm, unsigned long start, start = (start + PAGE_SIZE) & PAGE_MASK; } - return i; + if (must_unlock && *locked) { + mmap_read_unlock(mm); + *locked = 0; + } -finish_or_fault: return i ? : -EFAULT; } #endif /* !CONFIG_MMU */ @@ -1861,17 +1886,13 @@ EXPORT_SYMBOL(fault_in_readable); #ifdef CONFIG_ELF_CORE struct page *get_dump_page(unsigned long addr) { - struct mm_struct *mm = current->mm; struct page *page; - int locked = 1; + int locked = 0; int ret; - if (mmap_read_lock_killable(mm)) - return NULL; - ret = __get_user_pages_locked(mm, addr, 1, &page, NULL, &locked, + ret = __get_user_pages_locked(current->mm, addr, 1, &page, NULL, + &locked, FOLL_FORCE | FOLL_DUMP | FOLL_GET); - if (locked) - mmap_read_unlock(mm); return (ret == 1) ? page : NULL; } #endif /* CONFIG_ELF_CORE */ @@ -2047,13 +2068,9 @@ static long __gup_longterm_locked(struct mm_struct *mm, int *locked, unsigned int gup_flags) { - bool must_unlock = false; unsigned int flags; long rc, nr_pinned_pages; - if (locked && WARN_ON_ONCE(!*locked)) - return -EINVAL; - if (!(gup_flags & FOLL_LONGTERM)) return __get_user_pages_locked(mm, start, nr_pages, pages, vmas, locked, gup_flags); @@ -2070,11 +2087,6 @@ static long __gup_longterm_locked(struct mm_struct *mm, return -EINVAL; flags = memalloc_pin_save(); do { - if (locked && !*locked) { - mmap_read_lock(mm); - must_unlock = true; - *locked = 1; - } nr_pinned_pages = __get_user_pages_locked(mm, start, nr_pages, pages, vmas, locked, gup_flags); @@ -2085,11 +2097,6 @@ static long __gup_longterm_locked(struct mm_struct *mm, rc = check_and_migrate_movable_pages(nr_pinned_pages, pages); } while (rc == -EAGAIN); memalloc_pin_restore(flags); - - if (locked && *locked && must_unlock) { - mmap_read_unlock(mm); - *locked = 0; - } return rc ? rc : nr_pinned_pages; } @@ -2242,16 +2249,10 @@ EXPORT_SYMBOL(get_user_pages); long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages, struct page **pages, unsigned int gup_flags) { - struct mm_struct *mm = current->mm; - int locked = 1; - long ret; + int locked = 0; - mmap_read_lock(mm); - ret = __gup_longterm_locked(mm, start, nr_pages, pages, NULL, &locked, - gup_flags | FOLL_TOUCH); - if (locked) - mmap_read_unlock(mm); - return ret; + return __gup_longterm_locked(current->mm, start, nr_pages, pages, NULL, + &locked, gup_flags | FOLL_TOUCH); } EXPORT_SYMBOL(get_user_pages_unlocked); @@ -2904,6 +2905,7 @@ static int internal_get_user_pages_fast(unsigned long start, { unsigned long len, end; unsigned long nr_pinned; + int locked = 0; int ret; if (WARN_ON_ONCE(gup_flags & ~(FOLL_WRITE | FOLL_LONGTERM | @@ -2932,8 +2934,9 @@ static int internal_get_user_pages_fast(unsigned long start, /* Slow path: try to get the remaining pages with get_user_pages */ start += nr_pinned << PAGE_SHIFT; pages += nr_pinned; - ret = get_user_pages_unlocked(start, nr_pages - nr_pinned, pages, - gup_flags); + ret = __gup_longterm_locked(current->mm, start, nr_pages - nr_pinned, + pages, NULL, &locked, + gup_flags | FOLL_TOUCH); if (ret < 0) { /* * The caller has to unpin the pages we already pinned so @@ -3180,14 +3183,13 @@ EXPORT_SYMBOL(pin_user_pages); long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages, struct page **pages, unsigned int gup_flags) { - /* FOLL_GET and FOLL_PIN are mutually exclusive. */ - if (WARN_ON_ONCE(gup_flags & FOLL_GET)) - return -EINVAL; + int locked = 0; if (WARN_ON_ONCE(!pages)) return -EINVAL; - gup_flags |= FOLL_PIN; - return get_user_pages_unlocked(start, nr_pages, pages, gup_flags); + gup_flags |= FOLL_PIN | FOLL_TOUCH; + return __gup_longterm_locked(current->mm, start, nr_pages, pages, NULL, + &locked, gup_flags); } EXPORT_SYMBOL(pin_user_pages_unlocked);