From patchwork Tue Jan 17 15:58:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13104843 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33369C3DA78 for ; Tue, 17 Jan 2023 15:58:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 012EF6B007D; Tue, 17 Jan 2023 10:58:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E8DB16B0081; Tue, 17 Jan 2023 10:58:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C44B36B007D; Tue, 17 Jan 2023 10:58:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id AEFCB6B007D for ; Tue, 17 Jan 2023 10:58:55 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 6064AC0853 for ; Tue, 17 Jan 2023 15:58:55 +0000 (UTC) X-FDA: 80364749430.22.E78E5D8 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2040.outbound.protection.outlook.com [40.107.236.40]) by imf09.hostedemail.com (Postfix) with ESMTP id CE8C114001C for ; Tue, 17 Jan 2023 15:58:52 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=BWOhDtMF; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf09.hostedemail.com: domain of jgg@nvidia.com designates 40.107.236.40 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673971132; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=B50ohIdvhDbFwyWVcAFTxgU9fHhVhqr+gvZ/3zGe7iI=; b=AGXYWsfMJNdd/iEdzGne6LJtGYlj8YxlrfBEBOkCKK6sw7e+FEnnU61Mcm0Xyof03w8zus nZZyn0lkYrO7dzJln6OvqNSA9NEK3ry0zg16WCVpwjBSC2kZrW5DpbrfxI/jPVHZ1Lzl3d E9mD4pQrA/00dwHe04lI55/phcEadvw= ARC-Authentication-Results: i=2; imf09.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=BWOhDtMF; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf09.hostedemail.com: domain of jgg@nvidia.com designates 40.107.236.40 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1673971132; a=rsa-sha256; cv=pass; b=gf12oXg73V05tlRGcYM+sHtu2RgY/OpxNuZicrkXdq2614MFHjDy/1RlQuQhNgopTEJ8m5 ennxTz1J6LG5d6nVLF1e9LbIGxwjZ5rOWXwaooKyjyPVtFasb4EpYhCaCZt5qRWcgiHP4D jIrYZorH+ai9Ta0XUdmvAytPwKWjvTc= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=B4JDQc60m8qbzJ9VGNJopz2cpsZmk1IclA6paeo4Me7zGgEppuEie2mx6aeB261KCasmVlh+/3KRi6NO7EAD1xyL7W6ercHWgM0r68PmP6qWmTbfROpQl4hwJHC8TZct69dVpjIuWJOZA848XsEN1yZUnShnz7NcZZ6EASvQUX0BmXKCaZkOSRlFoCv7ZHclGAL3jxAkbRBo+BcajYMjJlDrX6wkTNO/boDqpi+RwICfOUYA8HWbC4CqxTkBEGBXarY97MDWSELQkGluuy99m5ILT2yqGpk9M6msPsVP5CYEWvuYCcnP2Aztnn8I0dvmjxQPEK2Azna2XFPKoBNCyw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=B50ohIdvhDbFwyWVcAFTxgU9fHhVhqr+gvZ/3zGe7iI=; b=Eqc85Z7UjCXpQ0iNYT9GpkIqf0RzPxjpNiYzsPoxe8kaPOJJZRtDop9iXKorj++EyCHCvd4fgtGKdEkzvzz1fHTSrnt8KFIhVLRyn/svQiGjbtCRFR7r0L51mio70peWUYsWMPIots6qhLKqCiZkHAXQh96SpsJ+oB6/C0j7ScJkddW3//PpNKZ9LnuNKiYjXB0l/f2zoKAI5y1G2XToEVNrjqhF61osOewfJjbBDTE4MBRVCL+NpZz5xL6TrepjFdcC5vklyKKxPfruCztgn07UT3puJAVh8KckZTcZI6YzsJ1HxkAA0eK7tUkgqSaL0XJvWV+he3DcewgN3HGK9w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=B50ohIdvhDbFwyWVcAFTxgU9fHhVhqr+gvZ/3zGe7iI=; b=BWOhDtMFP/0ZmESvlRYXeWkqeSzKEqyVxmtc6WydzMV8S7QsGRzxW157gqgdHVxZ+dlYtVWyKTMHdYXxvXy8RHBVbkmjUhhOoEoO0epjDxM9sg1Q2FrJVQJoxLhrJLCB3BEvCG5Uvac3Egnh0l2xQApJAcvwbK2q2avpmhtdimqW984M9b+9P+m2ILQZ/JKWGwPjqe1QyCep6d0yLtxHFXZTB8ibfEuNeWPcV5CBSQuycZ76tpf9+ah1Y+NWlEU7Jfb/7/o6pVoZZCBdO49J6EXy+f3m7eCeLQJ7Y7SpKWBNquxHgkqnOjFKHJ7fsw3LNZo83kxn0XkBY232u56Mqw== Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8334.namprd12.prod.outlook.com (2603:10b6:208:3ff::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.23; Tue, 17 Jan 2023 15:58:44 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::f8b0:df13:5f8d:12a]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::f8b0:df13:5f8d:12a%9]) with mapi id 15.20.6002.013; Tue, 17 Jan 2023 15:58:44 +0000 From: Jason Gunthorpe To: Cc: Alistair Popple , John Hubbard , linux-mm@kvack.org Subject: [PATCH 1/8] mm/gup: have internal functions get the mmap_read_lock() Date: Tue, 17 Jan 2023 11:58:32 -0400 Message-Id: <1-v1-dd94f8f0d5ad+716-gup_tidy_jgg@nvidia.com> In-Reply-To: <0-v1-dd94f8f0d5ad+716-gup_tidy_jgg@nvidia.com> References: X-ClientProxiedBy: CP5P284CA0103.BRAP284.PROD.OUTLOOK.COM (2603:10d6:103:92::14) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8334:EE_ X-MS-Office365-Filtering-Correlation-Id: c0570b53-1ca8-45a5-c427-08daf8a3b61b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: tXi2v9BDQo4Esv0thXPcaq7D2sxSN+s6aVPJXhlzIp+406mqKePfhTlaDHG654X+DbrSeVLJuDBznj2ddW+CQwJ7hVFCEXTUrsZXClwjqwM24dqvUj/y4SxvsS68PPTrY7Kvb6CXARtOQ2WzrgUUzMU9zqTfY5bRBOujX3Ra+UmKjxh6EZN+0dMllM3907iuNt5hqCNXkNmj4mMhZG+V09daD1xVsn3btRIRgfjuz9jpK4VAIxHuxHa2JDIQYulU3dY7TNkojD/Q6U6mJrJSeqFxo2fZ4W+LAvfmDn3PbOzmuIkCVAILDeau1RQI2Na8uzCSrNI5QG6NqtyGBvgiwrvQPwqvlkgfE40PP/6K5l3IZ67f8AHbJhh2hRbJlemY/41l1jTKBzXN7eb56ZiVttfl6B3FvyJqIvyvF4ozxak7TLgBUHHiWXeTO9n4ZgOQI2MkT0pfB8x0yukIblFYjfnsFSFaDZQhpyTQFH6+mnpEmCZ5V/EdWFLLsEfbx7+sWblml0jmPcVrFf6MUUxAHO4uHCS6ar289rgOWlY3WyEfCF/Wb/bjSLF+20KJRMjQ+LGa0thhFV7GztNiPDndgUd1DmkpIovgXiGWw0sKSId5y/m1IMjxgib+t0yQsNaAZ9R3mhEBA1x3VgUHYFKqfWlVKxfgyHPWMg8cRWcXri78dw6ErkzIa8pUrit7+eBnSBQ7wvqPHPMxMRjdpRo9DW0kDlp+1/BUHKBjEYCceJw= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(396003)(366004)(346002)(136003)(376002)(39860400002)(109986013)(451199015)(36756003)(86362001)(2616005)(66556008)(4326008)(26005)(8676002)(6512007)(186003)(66946007)(66476007)(41300700001)(83380400001)(6666004)(6506007)(316002)(54906003)(478600001)(2906002)(6486002)(38100700002)(8936002)(5660300002)(4216001)(266003)(15583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: iftoO33dCsMtfFsEPctFtMgyEunWc+VaeYRwrfFnNT8NGdkVuwsC3bTl5SiQ+eB+GmXe+sK8vGg8NA+BzAX15UfjHEqmPwKz4FHLS/M3UkomWTgiX90GFyhMoC/cTtGoDKcrilFVcMrTJgmsH7btBZpz4/iGxiBKB+8K8EnTi7PXe2wKSoqykOYsXB7GboBpW15aCXEz66cduAYe9tSPxKAQDWsU+vJS3oNUIC48AeSorSsi0m2gmWxHEiO5K01aTij9ratv9rESt6xRI0nFTKqLy/p1Srh+SdsE7kDnnwA3GDk/7uHXpKfUIzbf9+yOvwRlFWHiMdC/jbZnTX5RoM/rq97nSiBtNbFkLWh/4YYXGQTE4sQPLO/tNFd/TsXbOeqdkiZ2edAiXvSudpSG5Prvmddd/y3o5uI5U4Gd1sp95IfOeXG7BFdqtPgQe+4gl2pSCR1bQFWafc/PFRTQCtdUziehADeQZdzLABpbg0HiR4Z2QJpoAn38C9Vm6lPbXT1MKVIkgFf5P1kN2/gV1e8e8FUhAHwONukTQTU0XiFBXgNibX16TH0YWx8VhpreUKD5URdypFDtn0MMAcWjPHwL2ShaUSXidBkKCsaytfipUudOY6FAtaSGbN7uSYs7F+63bzC8SQNwqahWyoURC2QXUp6+WuLdwJWT4GewpknQfbwZ+ErRaO2E93KGVF40gprVDDCAcSJo5clgX2339QRt7uEqHieSearKpZ60Fp9qmNK1Z1R0Ey8P8kiSgjH9runHrMzEQjcn5yKKPM/m5IanGrR3+ULY/LWKUjDHug/PIj5+QWUDN9Bs/TOslswtOzM4t2kxo0Ud0MHb7zWkZkUvSbjE8CnmC8BuW2Q6zYZgDyiSylY0byosC4aCR/q1NvrJZYc9r2/VlsoGAVxozhwFYCCeZZ+F0QKpqOLYpjIdhM//c/4O1Lu/IAXwX1UJli5X6FCc0y0RgKhPJ6zrcwvVzR/hHT6Z2la2MI4L9BHguIQXcbcFIxWrW/keTogZKvA9WpsxKoF9nz1gQPlfsmCNQQZSNXAXhTw0AXZXwuovIXs6AsjOFsjDFZZwT4tsxAO4qq1qCyYauMUNkJjlkYOFcyErZLSOL/unJ08JTPqxYkU+UeQq/QM/Ih5MNkKO1IWvjO14hH1h4Taef1Nbkm4myo17jeHUKkd9qmpW5DE8ptxlvS5g8LraI2rZZVhPeO6+GT2OlepzIcjUqA9H1PXGch5rCnqLnGRAZNpuutuAxfdB3+PiBShiEtlVhg1JyyARMCOO8kNB2JOX93fwQ/icJdC8jqU7H0VN6uBg0V13KUZYUBX4Uwjb4fJuv+idtT6vgWl17pORQJIPfp7ZmZznixHAS65JmHzmPcCPD8MkRcdX+h0YrFA78qq/ivHiQFazqxv/WHAi/zcGxBR7lf0opfN2ckyIwcNV/L+BM0iBQPY5D5lasf2nngEu4tQm0GvSEMWFbzf5eP1X0fjwXpejAVzgHJDVzqx6XWzIF2WQ8jnW0mby0pf0x/cvi8jmw0erUkwSAxreVlM/XtvwcJlUmDsM8HO+tps0IXEJhQUbLkYAS6R+pLlTuCwijL5A X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: c0570b53-1ca8-45a5-c427-08daf8a3b61b X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jan 2023 15:58:44.6395 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: utete+pJ9AiqKu2yVRSjB0K7KQ5fPNfym7ADgo4W4+6Asm/fuD3pIqHyHJfoLfIE X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8334 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: 75595jm4pzu9wtytypqrdsi6j8i1jwgi X-Rspamd-Queue-Id: CE8C114001C X-HE-Tag: 1673971132-901857 X-HE-Meta: U2FsdGVkX1+fDXPPHUd8kMQyzTfyCdjKSpBqHAYD66HMSQjtWguecljClT6Wr3lE2EjzgFQYgoJnXwLjIvRdhywPc053oTWVXZKX1D01g6xwkdk69yrJaGc6j6Z3rqgBzbFcHQ6c1LwfGa8UtpW8K+Qm2D0uOUkR6dZi7flBpAUysAuroXlCLbx7HRBCywOzVrvWZe+ZKU2808kkmU+FDlJDxyIEcCb+qNPtW8x02PPtixw6Ju6XlNnWYvi0fRbbCd4ksEHseYRDvcZaAuljhRJ/0gBXh1rpJ4syKVsJvfB/YW7yom0NEMWz+Ai39iyGftzFEdqfQquZ+ZGrSQBe1SotO/9B2atG6A5gWOTqUCxe2588TKg1GHt6AdpiYWGOlXSCvhpDhp/vXHibGAGPJoxoPeRqbjx510md+vgk3n4MiqHtJ/UPTs6+uOYZbDnTcy2pVuvOuNxhXTUoWrimEQxWJHYxHQR3HlzL6MG0NL/kHqT0hj9nVfo8D+N852DpvwYr2yUZYQnT2ytHYAtg26AhJBVPvBDdS9muLNRekWFuYIUnItv7qhBih9xTPllI6j42NSwbI5mh94ktBBmKAWGQ9PD8zuXHAxmnlIfJplmsqDAmF91wumL8QIO75SmGqOC0ckxJ0gZtRdLxEYnfW0VLs8pPTnEhV6wJz+GDNvorSlt9bkJ6USMPFxUjoA2qa2f6zUokUpUInWLbYk1yumH+BpSQ5+IPIbEf1C3cV2UELg3G8+Rr1IWG/+KBldIb9ch16PbUrF4rfm3Gm4PIf5SP9L6kD2dDrdAw/4oF8//ti6t1aqZnfb08oVvu8R93Wm18Xx5uzTAT9LK2FzW6emxERybjgKoOEUxWuO1X+QYlQCvMNfu+dnde/gYAK/KbukNNdvgY5fFVknH1dYNid5081LLFZ9qeYdB4EYxOvetM9rfERc5ejwns9xuBz/xEgoFIYZtl9u1IQG3PvtA F2NpwNno d7C1u93CSRGRhmIrSj2/3AaSp794xXJKxD8w+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: __get_user_pages_locked() and __gup_longterm_locked() both require the mmap lock to be held. They have a slightly unusual locked parameter that is used to allow these functions to unlock and relock the mmap lock and convey that fact to the caller. Several places wrapper these functions with a simple mmap_read_lock() just so they can follow the optimized locked protocol. Consolidate this internally to the functions. Allow internal callers to set locked = 0 to cause the functions to obtain and release the lock on their own. Reorganize __gup_longterm_locked() to use the autolocking in __get_user_pages_locked(). Replace all the places obtaining the mmap_read_lock() just to call __get_user_pages_locked() with the new mechanism. Replace all the internal callers of get_user_pages_unlocked() with direct calls to __gup_longterm_locked() using the new mechanism. A following patch will add assertions ensuring the external interface continues to always pass in locked = 1. Signed-off-by: Jason Gunthorpe --- mm/gup.c | 92 +++++++++++++++++++++++++++++--------------------------- 1 file changed, 47 insertions(+), 45 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index f45a3a5be53a48..3a9f764165f50b 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1343,13 +1343,22 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm, unsigned int flags) { long ret, pages_done; - bool lock_dropped; + bool lock_dropped = false; if (locked) { /* if VM_FAULT_RETRY can be returned, vmas become invalid */ BUG_ON(vmas); - /* check caller initialized locked */ - BUG_ON(*locked != 1); + } + + /* + * The internal caller expects GUP to manage the lock internally and the + * lock must be released when this returns. + */ + if (locked && !*locked) { + if (mmap_read_lock_killable(mm)) + return -EAGAIN; + lock_dropped = true; + *locked = 1; } if (flags & FOLL_PIN) @@ -1368,7 +1377,6 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm, flags |= FOLL_GET; pages_done = 0; - lock_dropped = false; for (;;) { ret = __get_user_pages(mm, start, nr_pages, flags, pages, vmas, locked); @@ -1659,9 +1667,24 @@ static long __get_user_pages_locked(struct mm_struct *mm, unsigned long start, unsigned int foll_flags) { struct vm_area_struct *vma; + bool must_unlock = false; unsigned long vm_flags; long i; + if (!nr_pages) + return 0; + + /* + * The internal caller expects GUP to manage the lock internally and the + * lock must be released when this returns. + */ + if (locked && !*locked) { + if (mmap_read_lock_killable(mm)) + return -EAGAIN; + must_unlock = true; + *locked = 1; + } + /* calculate required read or write permissions. * If FOLL_FORCE is set, we only require the "MAY" flags. */ @@ -1673,12 +1696,12 @@ static long __get_user_pages_locked(struct mm_struct *mm, unsigned long start, for (i = 0; i < nr_pages; i++) { vma = find_vma(mm, start); if (!vma) - goto finish_or_fault; + break; /* protect what we can, including chardevs */ if ((vma->vm_flags & (VM_IO | VM_PFNMAP)) || !(vm_flags & vma->vm_flags)) - goto finish_or_fault; + break; if (pages) { pages[i] = virt_to_page((void *)start); @@ -1690,9 +1713,11 @@ static long __get_user_pages_locked(struct mm_struct *mm, unsigned long start, start = (start + PAGE_SIZE) & PAGE_MASK; } - return i; + if (must_unlock && *locked) { + mmap_read_unlock(mm); + *locked = 0; + } -finish_or_fault: return i ? : -EFAULT; } #endif /* !CONFIG_MMU */ @@ -1861,17 +1886,13 @@ EXPORT_SYMBOL(fault_in_readable); #ifdef CONFIG_ELF_CORE struct page *get_dump_page(unsigned long addr) { - struct mm_struct *mm = current->mm; struct page *page; - int locked = 1; + int locked = 0; int ret; - if (mmap_read_lock_killable(mm)) - return NULL; - ret = __get_user_pages_locked(mm, addr, 1, &page, NULL, &locked, + ret = __get_user_pages_locked(current->mm, addr, 1, &page, NULL, + &locked, FOLL_FORCE | FOLL_DUMP | FOLL_GET); - if (locked) - mmap_read_unlock(mm); return (ret == 1) ? page : NULL; } #endif /* CONFIG_ELF_CORE */ @@ -2047,13 +2068,9 @@ static long __gup_longterm_locked(struct mm_struct *mm, int *locked, unsigned int gup_flags) { - bool must_unlock = false; unsigned int flags; long rc, nr_pinned_pages; - if (locked && WARN_ON_ONCE(!*locked)) - return -EINVAL; - if (!(gup_flags & FOLL_LONGTERM)) return __get_user_pages_locked(mm, start, nr_pages, pages, vmas, locked, gup_flags); @@ -2070,11 +2087,6 @@ static long __gup_longterm_locked(struct mm_struct *mm, return -EINVAL; flags = memalloc_pin_save(); do { - if (locked && !*locked) { - mmap_read_lock(mm); - must_unlock = true; - *locked = 1; - } nr_pinned_pages = __get_user_pages_locked(mm, start, nr_pages, pages, vmas, locked, gup_flags); @@ -2085,11 +2097,6 @@ static long __gup_longterm_locked(struct mm_struct *mm, rc = check_and_migrate_movable_pages(nr_pinned_pages, pages); } while (rc == -EAGAIN); memalloc_pin_restore(flags); - - if (locked && *locked && must_unlock) { - mmap_read_unlock(mm); - *locked = 0; - } return rc ? rc : nr_pinned_pages; } @@ -2242,16 +2249,10 @@ EXPORT_SYMBOL(get_user_pages); long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages, struct page **pages, unsigned int gup_flags) { - struct mm_struct *mm = current->mm; - int locked = 1; - long ret; + int locked = 0; - mmap_read_lock(mm); - ret = __gup_longterm_locked(mm, start, nr_pages, pages, NULL, &locked, - gup_flags | FOLL_TOUCH); - if (locked) - mmap_read_unlock(mm); - return ret; + return __gup_longterm_locked(current->mm, start, nr_pages, pages, NULL, + &locked, gup_flags | FOLL_TOUCH); } EXPORT_SYMBOL(get_user_pages_unlocked); @@ -2904,6 +2905,7 @@ static int internal_get_user_pages_fast(unsigned long start, { unsigned long len, end; unsigned long nr_pinned; + int locked = 0; int ret; if (WARN_ON_ONCE(gup_flags & ~(FOLL_WRITE | FOLL_LONGTERM | @@ -2932,8 +2934,9 @@ static int internal_get_user_pages_fast(unsigned long start, /* Slow path: try to get the remaining pages with get_user_pages */ start += nr_pinned << PAGE_SHIFT; pages += nr_pinned; - ret = get_user_pages_unlocked(start, nr_pages - nr_pinned, pages, - gup_flags); + ret = __gup_longterm_locked(current->mm, start, nr_pages - nr_pinned, + pages, NULL, &locked, + gup_flags | FOLL_TOUCH); if (ret < 0) { /* * The caller has to unpin the pages we already pinned so @@ -3180,14 +3183,13 @@ EXPORT_SYMBOL(pin_user_pages); long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages, struct page **pages, unsigned int gup_flags) { - /* FOLL_GET and FOLL_PIN are mutually exclusive. */ - if (WARN_ON_ONCE(gup_flags & FOLL_GET)) - return -EINVAL; + int locked = 0; if (WARN_ON_ONCE(!pages)) return -EINVAL; - gup_flags |= FOLL_PIN; - return get_user_pages_unlocked(start, nr_pages, pages, gup_flags); + gup_flags |= FOLL_PIN | FOLL_TOUCH; + return __gup_longterm_locked(current->mm, start, nr_pages, pages, NULL, + &locked, gup_flags); } EXPORT_SYMBOL(pin_user_pages_unlocked); From patchwork Tue Jan 17 15:58:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13104841 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 884FFC677F1 for ; Tue, 17 Jan 2023 15:58:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 827BB6B007B; Tue, 17 Jan 2023 10:58:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7B12B6B007D; Tue, 17 Jan 2023 10:58:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 602ED6B007E; Tue, 17 Jan 2023 10:58:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 4319F6B007D for ; Tue, 17 Jan 2023 10:58:53 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 126051A0615 for ; Tue, 17 Jan 2023 15:58:53 +0000 (UTC) X-FDA: 80364749346.09.7723190 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2040.outbound.protection.outlook.com [40.107.236.40]) by imf09.hostedemail.com (Postfix) with ESMTP id 46E10140003 for ; Tue, 17 Jan 2023 15:58:50 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=miKlORK8; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf09.hostedemail.com: domain of jgg@nvidia.com designates 40.107.236.40 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673971130; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vjkJ4CJY5l+CpLD2RdwgUCctGewL3b0/B+3EzpvAOKA=; b=tj/oqWjMlflf+CPQ+zUqZbd8Kj00gMxJj4ZtF4++59dJACAqziG6t9tYloH/MKF9Dl41Jl 6w2CdGg7heJwjI2zdAjmjRJDAZB3zm+jiBrGQ8bLOMrgIwjC+zo/2bXcQmHSNOQr12j4OL g8zUtjEmzqY/ACmQcpZjSZy7bO4frbM= ARC-Authentication-Results: i=2; imf09.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=miKlORK8; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf09.hostedemail.com: domain of jgg@nvidia.com designates 40.107.236.40 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1673971130; a=rsa-sha256; cv=pass; b=D/SM50F7bwhhxJc9TToFRL1Pye4aaH9H3KUVbK87R4GVe7UPu5wmFC/9V5gT4i8fpMbEhU LJp7ekRZaBMKsLpQMRThHYKkiWWvrS4kEPjdcz1AvynywqlK1coXpR7YTDd8bs3rY+H6UM Y0ZSRDCkmASk44tkgiF6nQVrq8gQMZA= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KLZVmfBNkstqRPWi1vw9vGK8oDAVTYqsAuzrssnZux5amZdXzvEX2miO6mEo/l+eBDG/pY+ZBIRaTtC73fIpn1RGQ8wnbhvJb9zc1P/prcAAe/rRcMZjQi9Y2t0sij8ZeI9vrKLwpVGWiIvG8ZG7Vd7o6T5tNWVTgcfO4OdtZogH9Mhy6+GfOh2wEnLiqrLj9DcLbslCya2ror9FSHNctzGPB9fgcR39xOoMWi8b7a6QH3qXP/WC5UOEKAoRnfL9W5Hyqc4dyhqkO7objPXm+UhMPlGEU3pV9SM5mpRLHXNS4C7T3O0MkKaGOV/wkl4M1hiQyiys0AEi7EDldU1yIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vjkJ4CJY5l+CpLD2RdwgUCctGewL3b0/B+3EzpvAOKA=; b=OySUAHuakPJOwE/aT22HlR7HCfKFJeWZZYriUbpPUnqRh62+zb7WhqyDbKLGITDQ3aR5HC34/xkq95NPtgQ1VZswJIuljdUMiQMZFB3U/1Fxv4vedyPVNoPdPrY2b1sL52DjwrF1+OVqknihXXAtV8TMBt1jpcRiRzyo2B/GX7SpsgStdlXFBzPdpTbNTl7f9/p//N5OM7UUTzsFz+h7Wmqy8ReYQD9h1yayrRQNetxZHNPKfmy/q5LSKB3ohFU7nyoirCwclKl3dklCpTd8DOzlCal2wBsJIhbV2MywMqbSKNH6VFwCTqhpSVzGTWBMNewVQgcH7Zvb47meLis0Hg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vjkJ4CJY5l+CpLD2RdwgUCctGewL3b0/B+3EzpvAOKA=; b=miKlORK8j9l1sJxEItte2pdG0sQmkwSffR9pY65/JnxXugf3Tn7JPnpz9dtTVO7gJ3ZtevQFkzfaf16KQ7Rb8XDojpEhf8Bqb2HzB0DUcuFif0Ry8Zm9qFHX8hyDcJBvCDFGbJyGgVKHG2/2UCHpLhT59bPxP01Vj539gFcxlUcnwK6fmOYnwrI7MTml1Ml0HWz3CtN3YbPRbY+TZV7hWHIF4lG/ranTDY07fDP+ccDJpq+hIZyWRxNSRUpq05jAfmLUmVjIM0xvZpVjKU7cSVm39Wp/Fdt8Et4i3LhVSXv3An5nJ6FgBNOUavt+gawvofMvLcD3T7w0sA9neWtbbg== Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8334.namprd12.prod.outlook.com (2603:10b6:208:3ff::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.23; Tue, 17 Jan 2023 15:58:41 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::f8b0:df13:5f8d:12a]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::f8b0:df13:5f8d:12a%9]) with mapi id 15.20.6002.013; Tue, 17 Jan 2023 15:58:41 +0000 From: Jason Gunthorpe To: Cc: Alistair Popple , John Hubbard , linux-mm@kvack.org Subject: [PATCH 2/8] mm/gup: don't call __gup_longterm_locked() if FOLL_LONGTERM cannot be set Date: Tue, 17 Jan 2023 11:58:33 -0400 Message-Id: <2-v1-dd94f8f0d5ad+716-gup_tidy_jgg@nvidia.com> In-Reply-To: <0-v1-dd94f8f0d5ad+716-gup_tidy_jgg@nvidia.com> References: X-ClientProxiedBy: BN9P221CA0013.NAMP221.PROD.OUTLOOK.COM (2603:10b6:408:10a::6) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8334:EE_ X-MS-Office365-Filtering-Correlation-Id: 721c3584-e34b-45d1-679f-08daf8a3b434 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: J8oeinBDx25F++/z2Ic9cLvO6e7gxluYmOt1CsZuPsw3xOcx+X4c/S1rl+PR69zLO6/CP+J1ur9PhVfGx22RaGcCdcZAHyNyUJFPqMV6mvm56yPzhAuc16rnKzzZVG7JExmzdNviWKm7C7I3ZV3dqHCC2dLkxD885yCl/uzFpam+eE7Nn4V0mzecqFxUPLN4XO+bQCIPkxuOMfq1xJjMVo2BPxZes5LMoYm8i4AG4Hr3/F/gi/SsIrkMu5SEGe4WlISWy1Ekv8Czl0fbS7dLBx8jqzds3Qw1oAA3Dqoqkwm5DKeAONarHWElhVI9oSyyaNk2/49IYBHPl7QlotaA9F56CEGJhZqjlOuXReoMjZiEZJLJoxt9AAfchoJwbU6e/7h+tQmRgOJCrpu6eOD1+RDKc13+P49ZIbFQS6U68u1BjVyD01IYyh90Sq+pTyKnYfp5mI2V+mpW/OX71q/++LowaScKcVeBZ6UpJyvU5nyoUaIESdL1tnkhaWoeju6YmjOU1T04GFVwM5AqC4yIIlfXK09d7X05QpVwkq7g1wkg6OQjG2MnOLqLwoRSMsuVw81ACXmRQugkOakZXGcRjke/f1UVkQl+/UwNT1RGdToFMTUgbJG5denAQwN+zaIyIlCIqYHcGhOAn1fmOgwt0t1pNLXKQKrhofA2M2wO49qpCo1VM8P+D+3Qwsj8rnBzXp2MCnLjwBAsEjM6Z0GchQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(396003)(366004)(346002)(136003)(376002)(39860400002)(109986013)(451199015)(36756003)(86362001)(2616005)(66556008)(4326008)(26005)(8676002)(6512007)(186003)(66946007)(66476007)(41300700001)(83380400001)(6666004)(6506007)(316002)(54906003)(478600001)(2906002)(6486002)(38100700002)(8936002)(5660300002)(4216001)(266003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Bs3eeTDwBH0+d6VczjCLkUVxU8S4WT+x38ulNR8sHUCDbGMFC2VxIeRYmhnjNBCHV05UpddDmFkyxLlq5Hekkm9VnNZE2iJj6VLAs4SI6kLRL3maZewhdnat8pu8KA3et7aM4NuapvXadtR8A5ZD2vHZVCI/dmn0/e2uzW8AIApP9kd6KDFGREvWMPIcb8V16kL57YOhf0whygAOltq8qCLfZOjodk5PQG3RiDDGgyQ6FrFDM2aGSUcEO2VpCP1aazvFQixn8Jm4YBxzGUz1X0f+0R7fdwa0dZBUtlx6SIanvlGflH8mjUID3HSr2N9yQnQO80awPhFtxeRnBffCelEsNL79c1F6Vgmo1NkGRw3Z5CPK0IGrlhPZLxfwmlQyZxXDbBXca2cbMk1wn4c472qCaGRvKX2ujtvpAuP68KVrDZ50mMm7lHbN2+FXZ1yt13KLChH1S9aKNZ+dDJG8VKKXqH0Jl2i3w80qucVrRtNssKW2H3phCJpcy486KOWwneircxQbioQlFoT422nH9u4UHJxoCpkBUG0Gg16iJ7MvtuTbl4eQa797Mtv0Qgu99++rrJ/bsUKlu29ABwItGwvT5ZCgW7e6XJmbYUMppMR5mEYpM7/adBycAcNg3OJinDGc7YM5Mz8mnBDXBIABicKMBnMFBGdFQG0O1W+LcDw2UX0ARS73fRb4lvQ7hsuMKRUGY4tBSKKFlCQw/XnECkRAMbnBBznKtQw3lmY8FKKBVtPhNFEHDgrPKh7DGugj15xPzo541O8fIkNVlrzp+14GyjMFpCEq+2G62nB9SDp/G7tX95VmKqImpQT5c6zCY3YlWl6xOd43mcs1mn6H1IWbcG+qDpUNLPCG3RkVrEcn0G6SiVB196NdUKRC7Ce+F4e3N/hUZGx9thHW3EmfqmQzw/gZL29hFDfmGLjHQnEg5tvBdtvf9AeaTEvUf7lpdOlAlhdmw8uNY5geA45FnfjOLREw5/2bZnNnjCdmHld4fTUFJLpeXHd/KdpFUnp8Gw0TG/w3+sd4b5SuKpqTEs2aPWPP6pPkdghT36N/xHXkWzMq5+BwI+JHIJ7atM9H5H8RPR0dVzxMZtPe0rTjIl4CdSqtYl/+GsJ2g5BLvKmDCWLaXQB45JMzi7zOLOb2ptQ0Y7bXAFchElS0MdR4fELC8JEcgG1uX/uvFqlWSyiKfj7kGm24uvE77QuUOAC/cmjbuqnenjA9xyLCSHtKoqRwY8H7bBn358ljUuNEEHq7Un3fiCj0aTFmOHofRd/SBgSTkOu9qO4gENOdHms/Lr+9VRk/FAJARLnxdwlysxuh8lgnXyqtwvh/XXV/eWJA23a7bYgTewOu8m/WKCfcMCCOWnqHw9TG1QBqImboSXxIKcgLmUdD1iGqw7NMEZxrS1N4grxCEhc+4dm1G/CBsqgF2UAGVMu5ZTnNUFVJKF3506Xiy/tiv09RjmX8wvU5Nuo74od34YCJDD54w9HKmG6H+3LqVs51ChSU8mcgZxWb2nDP+JiUahcodw1bdZL//PX0wl0Vf73K0ALiE2yfXbpIjQBcW8tT9uN8PO8UOlVEmIxVWmh56hbNzaO24B7Q X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 721c3584-e34b-45d1-679f-08daf8a3b434 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jan 2023 15:58:41.4287 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: wc1UgINiLELvSsCi7ajrbjSzgwWEYVHaAdCKOnnPVJMMZpeZt5Hd5cYPX2e/dE9E X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8334 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: nn5bs61crjrq17quw49m5y7u9ofqy7pa X-Rspamd-Queue-Id: 46E10140003 X-HE-Tag: 1673971130-568409 X-HE-Meta: U2FsdGVkX18Sy8qPAr3tpocxqtQjCZyfepWLxeF4RKqmRJBljNfeuTPpcMBxWiF33/QMLdLKt+VviX5kijpMLm1QYFrzGHxEvHQSwEkJ3CIPBFyyFQKFhPzEEmYW3uuLKyR3dxaAe22tscJRQqJDE8eYoaquLMQAX0CXeJIqgwJQtB+cvh6xT83IKK6pmTr/8OB1+61pMFm9Q9A0YAegH7oO4nrCRtKvg9JdNKRmYXaxAgLNobaghKnEHEc4R1hASmH0tOEP0kWJIVjSiypA6/w8/3WbD+Mc1H+5uyiZ4b+Bt71JVAkdgMKZAT9jc3gGC+q4/WOSo7CA3b1ffC3Q2WLfaVKGM1s7Y8VcMNImxz94RtEof0dJxYZ3FkjZuqUjN4nGwO5c/VjrRvRFQL9VDPxmfBJQmxu/Nt7IO2JlUCdly1ghPBO4U/d6pDTSYaVvseg5fWIXX/xHytoXgf8/xrauHrxVbvL9lvZs21ePyQU/SK+yYZky3SBGsw1WYKmGBNorgribHW+ekdrK/XU9ZyiHhMuSO/rf+F3dHvVUArkW2e4X3LtMcf0XmLi7C9+Bgysg5Wkcy+u2+ChIxeQ/4ZstTWYbMtGmfr2MsKLb8epaoKSg/DMT566gqd3a2sbEmJunEwkjw+etER4L4CrWTzWyKupe6EfEyqjet25Wx1QqwimpWv2hm2a3pY+RBgk7rmBixJbysl0sdOk7qK2zWqy1gt5BtJMjZi1X0OqALvi/HRiACYQ9h4/5EUsE6bf+gMqs0OBDEu46kOIenfca7O3S86MxNULl9qweQDdBoDPLa7mNjXMnMJNR5fj1ohB+PnhbKcEmSlFUkcR0QjpCfx/252p7ioBcTbXfCmb5bl5tlA8HNiQRxq+9/9/KRanz4pbbjGCe2AJBGW4wZZA8mQQrWuwHxlkeNJ7DFOXolJgksihmE5tVqXE+o5/rcEZ/AoGleG/S74jWttMQ1NN Vn65d3F+ qpcKfsAuajFx10qZiagoFfV1CiF9jmqbUpuLz X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: get_user_pages_remote(), get_user_pages_unlocked() and get_user_pages() are never called with FOLL_LONGTERM, so directly call __get_user_pages_locked() The next patch will add an assertion for this. Suggested-by: John Hubbard Signed-off-by: Jason Gunthorpe Reviewed-by: John Hubbard --- mm/gup.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 3a9f764165f50b..2c833f862d0354 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2188,8 +2188,8 @@ long get_user_pages_remote(struct mm_struct *mm, if (!is_valid_gup_flags(gup_flags)) return -EINVAL; - return __gup_longterm_locked(mm, start, nr_pages, pages, vmas, locked, - gup_flags | FOLL_TOUCH | FOLL_REMOTE); + return __get_user_pages_locked(mm, start, nr_pages, pages, vmas, locked, + gup_flags | FOLL_TOUCH | FOLL_REMOTE); } EXPORT_SYMBOL(get_user_pages_remote); @@ -2226,8 +2226,8 @@ long get_user_pages(unsigned long start, unsigned long nr_pages, if (!is_valid_gup_flags(gup_flags)) return -EINVAL; - return __gup_longterm_locked(current->mm, start, nr_pages, - pages, vmas, NULL, gup_flags | FOLL_TOUCH); + return __get_user_pages_locked(current->mm, start, nr_pages, pages, + vmas, NULL, gup_flags | FOLL_TOUCH); } EXPORT_SYMBOL(get_user_pages); @@ -2251,8 +2251,8 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages, { int locked = 0; - return __gup_longterm_locked(current->mm, start, nr_pages, pages, NULL, - &locked, gup_flags | FOLL_TOUCH); + return __get_user_pages_locked(current->mm, start, nr_pages, pages, + NULL, &locked, gup_flags | FOLL_TOUCH); } EXPORT_SYMBOL(get_user_pages_unlocked); From patchwork Tue Jan 17 15:58:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13104839 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5EB20C3DA78 for ; Tue, 17 Jan 2023 15:58:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F048D6B0075; Tue, 17 Jan 2023 10:58:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E8E256B0078; Tue, 17 Jan 2023 10:58:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE0856B007B; Tue, 17 Jan 2023 10:58:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id BEEA56B0075 for ; Tue, 17 Jan 2023 10:58:51 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 8BC0A1A0500 for ; Tue, 17 Jan 2023 15:58:51 +0000 (UTC) X-FDA: 80364749262.20.1DA9AB0 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2041.outbound.protection.outlook.com [40.107.236.41]) by imf08.hostedemail.com (Postfix) with ESMTP id E5755160018 for ; Tue, 17 Jan 2023 15:58:48 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=lLd8bQAk; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf08.hostedemail.com: domain of jgg@nvidia.com designates 40.107.236.41 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673971129; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GjALvm23kGUWqLHMO0zRo2Cdd4iLEKEi148r5kwH6Yo=; b=1koHwzgX/HstzO3ClbGu8Ri/VDDydMGvKTwM+Am5c/NUcQobLOmhvdVpdVK+TdAICygO8I jLOrKEVGtwY0ErxoMN4q7OtvH79RCPBOXbcP9sBoTYhJbp7QGraCvK87IbsAseCK5MoHtW yR7uI0gMPKsh6wxngVom0nZ4j6acrpA= ARC-Authentication-Results: i=2; imf08.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=lLd8bQAk; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf08.hostedemail.com: domain of jgg@nvidia.com designates 40.107.236.41 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1673971129; a=rsa-sha256; cv=pass; b=zlWIpryakWax5R8jq06lIEd/w0+mx2mjFPEbxcf/BqGeZtYF6AatYcsN+AhPVWpZpCSGeU YvAATyGhVKz9xwM/85baeew7OLathSk4qAb46ivKiQEoTR5KMoXO26mlhKktdrk/ursKE0 Q0jWPUUUw7yBEKO4TAzYEXOEieiCw4o= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gb+up6fGC55JHoz1QXt7iUCdIqExiMcnEPibXxW+TRudSwRr1IG7G54fng0WRZ56eq+B3cGO4fsBShhM5AwYIkIxOcTlriQR/QeZ07O4hE4kX/3YEaHDeBRZAdoM/u8p9tP0aR33rvhqpUHNqX8efVxSo+3rB6bUOBf7ubO+4PSjbxqtE822ReQO9TeB280uer0zdT1Jj+lfCVbh1StTwk6eNqnlqxUVemaoGcQ5kKJNlhggXHdN3GqXY2ZhQiCszGu8GudvW+O6cfI16HKV5nyiLAUsCOzI2EReVz3YgOpHqQ0U0AeXLIUfRmAixgyApCfvNrDW5OPklMRUo+e99g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=GjALvm23kGUWqLHMO0zRo2Cdd4iLEKEi148r5kwH6Yo=; b=Muce33/bHbX5FBEXEsz0WcneXmuB7b23kHlHMAhOyKDWNkWKwWUoe8lNGkP7w6uTFRodg63SKUGJBddXMMGvjN//Px87aWrMSIgQ7++ZCPxgmGYvR6nd7mQTQg4UDtzHHhNAKqS/zKmF8pB5IE7+//hNrAHEl3DxSxPVzfBQBh2CMuE5zNOUd08Nvmr3+nO4lY0ZFe6OZCZToyuh7egPIrGZEMSyR07+XJ4Gj9tzhiXu8L94X+iX6HNn6UmQLzURBwO1UzVnTPcJUpnPQjZpxQqkhWiiwdGqLLgoCbvkqUK4jxss0NI8Il69I7EWGwVXrWqXNBqgfVdkjttGYJXT+A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=GjALvm23kGUWqLHMO0zRo2Cdd4iLEKEi148r5kwH6Yo=; b=lLd8bQAkm2FCxrvJnFH8tQQ3m9BYvj8dvycwD09k5VSNVQ7IWrTcKmcGMZjV3Lsdk7wM4qXRd3lcB/UMJIW3Yt3ZJkvWFXWTKEW7crXbyMVKnc7/pmzomFFZHmW82k5IqP2w7QBfsojB2XP3zO6r6wAXqq/92DDp+Zfd8GPUFPfIB3CPgFV/2C4CBGQlbZnFdjpgyHcIcih9JukReVFLuoeKlYqxwkKyvLEk8cmOXpEwbIh7Hd/BDTJjGW9y1GlyiXAyYbZwmxQ2VoC7NjypM3E5q0RL5YTvUi6tLddKXoTCD50Srv9MkZCIPUlVa+yGyTZ95KfmIt/ivNBj0vSlZg== Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by SJ0PR12MB6783.namprd12.prod.outlook.com (2603:10b6:a03:44e::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.23; Tue, 17 Jan 2023 15:58:46 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::f8b0:df13:5f8d:12a]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::f8b0:df13:5f8d:12a%9]) with mapi id 15.20.6002.013; Tue, 17 Jan 2023 15:58:46 +0000 From: Jason Gunthorpe To: Cc: Alistair Popple , John Hubbard , linux-mm@kvack.org Subject: [PATCH 3/8] mm/gup: simplify the external interface functions and consolidate invariants Date: Tue, 17 Jan 2023 11:58:34 -0400 Message-Id: <3-v1-dd94f8f0d5ad+716-gup_tidy_jgg@nvidia.com> In-Reply-To: <0-v1-dd94f8f0d5ad+716-gup_tidy_jgg@nvidia.com> References: X-ClientProxiedBy: CP5P284CA0101.BRAP284.PROD.OUTLOOK.COM (2603:10d6:103:92::8) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|SJ0PR12MB6783:EE_ X-MS-Office365-Filtering-Correlation-Id: df0ecac8-7da6-48f4-89f0-08daf8a3b6eb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Gk/KrtHgb2qNeKyaa2eaN64SzygvsZPkqXon8eatVsLPffHK7ntKTULXErSjQ6JMZg9Sxw70b1Z9mL6zOr1ouQf2uG3s4kH1VzWsWczmwriSvtX1+y5fxIoPDSvP/zgeraGV0uu3NllpAi2AJO2OL33n/lG6ALi2t9aAy3kJRKaB0PtXFyEn8uzB0i5z84W6wjneKmsuzWzhzqPV4HOQOYLaLS3HCGxA4wg78sJz/pCpSZ+U/t9YwCeQqUkOYzwff97cS2/tYIs6MbL8uXJqo/6Xhy64gPdPxq48rnBg0uRVRb0iiG6VTcg4XWpPAcbmyzJhjSE8nH0QNODVVe99P9fGzjLbEMEK5ppnW74z0ZuO7tS1g1y10fJ7GHY7a3iU0purj/pGh0KyELXkHnLYf/6Lh20R29yi0yIq4ALTZQVMFSCVdHOyyjAgohtN4I7IWUHJBUFUobBuIFfDs4ZaN9wigKD64nZiyPEr6dHOaSe7bEUk8rImwioBLrVux7YZT6qpXPGVNCtsNCqvQ9aPVcIbjlbu2/Qi+fq1veIhgYg78JmQP7SirShmSsdymo1SUlxHZjqNHNU840281uTbIe39hhgo1BamCcoc1gHbnVQb0Su/Do9D75XzgXChnVhA/P5n6d+7TeFr2iwPDb9u3dc+5vJKpVQxbQybcaXYlCvk84KeRBvuSNoHVTNvspdVs1L5qtr1t5sR5EP74KdozA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(346002)(376002)(39860400002)(136003)(366004)(396003)(451199015)(109986013)(36756003)(4326008)(8936002)(66476007)(66556008)(8676002)(66946007)(30864003)(5660300002)(2906002)(83380400001)(38100700002)(6486002)(54906003)(6666004)(316002)(478600001)(86362001)(41300700001)(6512007)(2616005)(26005)(186003)(6506007)(4216001)(266003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: qs9aVaUBdlYFPL2mrPwCIPaN/eFSv1EVe2oiC0a0DqsKgdTpsJuAAAbp0xxf5n1K5JYfigSwx6xkoCnnJJdRKcNCfUPsKB3PWuVEiiSdL94UEvxu4I019eYgzJFk9615qP3cb2OC25+vkWdool7M2ck7k1K8KSCv+F77usRYNRYL3QfMQaJKGNHycDfnO2DUb8PvdWT+t+eWvYSr6drnBSRtEs9UBkGwmT/ylDpBrBkG45eTn/FMaM5D7DYe5EViw3HmZqN2F4R4cRZOH7wden+RCYz4WgKv0TlcFGoViiVtk4YnruX8qYw99knnYRMdbCUzjFvfWdfuIluPP3/k6+FIfVElGkxmzhLPsXvPtRJeCjuihvYqKIQe+BqwYsHnrtu/21p/kozFUtxZrly4Yei6sO5+rK7FQkD85j0DWo1vCcu4Ez51oPvVfyHY9lkbMB2qb0E+TnJ+nW34Cp+dq+JxhyBG/PX9SZVkHAb4sJh5ixN4ciZ3jJ68XHugv/RUtg/FWSSnAwDUq6VN1poMph8gZ2iIFxvmtQqh0NscZF5ovoHhmfOPrDm41NJrB4Bvf7gKFzWJD9FTYn9qEJPtXmStATmtE2tHXuVjqsKRTRY5TVuXAahHCUn1MT7sqGOelNiW+bhiMymA+Wghve4MPanV+0RtJthCwSg1aI7wTR9A2gZWT8+MP2jJsyWiPmGJSwwcneHUC1gve33uSo+iJKAXsxYfxP0zAUvi+Pw06KhnQ5/W2ya+qHCQ5H+S6QEqDfk86VVfseml0446YLbKG/q9NTez96WVzOclWjdOrwKWaBcfEZQEE7o01GsuwEnII+tNx624EYXWtKTewx3DiLl8WukDl20RUkXqwq2H3grop+SFu4IKt+IJU+Wf/HMyMiBbSHTWqF5EwCX/4AMWAL4ahM5+VYkanwUDd2MqZINBk2XqbfGxq9No2+Ps19y5GGbLhC9+TLlByagrlh1lpzxJ7MEnWmc7B1Ah3EC9ZGcmwUw2yAJBrEly8tjiq0rViurt6TmbpSZFI2MP0rWzKJYF8LQHXNX91PJH2i6ngKWgL8cOirJ+Xa1WqptrVoLXwbOB/FPkQIambFsQpkfd8ka/bpCQgqjfNBvnOmXhkIQ/Sj2r9ifwzHc1u28Y83C4UFrdaNzqpV4NPnbwtvYA4T0wcdoYffeyImP4P0q8Ko1eoR4lsaM6PPdxIrVIUnvLCqK+8VZbIbOuaVeGOuMjNDGdexDudUuxTLPp/Cd1gXyVXo3J0ftiB9Z4oLHvBY3DxvhmQ6ulJVi+12kjTQ0xbV9WkgFAB2KTSm3Zs+QjgHkH1Vkw6liGzmY+mvxr3E3IhR8I4e14fIRWWIfoqIk8RYfmJvOKtc5Sh5jT897Ob0ULyLS/LFicdthHXInDbp08bsoIoBMY8CGvrGp/xAysi175m7k+gW3upsNCyHqVXqoaqtv62cvv2W8Ph+UfFUPkYoz7Aa3KWNTWvibgBBJ7w94W5JF6dlMXn+zJgdLf8mO0hXPU0Pcka8cRVabzwvI67wMS2kk4KrUPzHi/uFBUTOl2PxWAL1dZo24AFaP8/QJ81CwvBY1QbKufMh1Dj3Il X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: df0ecac8-7da6-48f4-89f0-08daf8a3b6eb X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jan 2023 15:58:46.2499 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: pSzfHGrXUs+nM6PZRhHebAtB1tPZXp19DHtuK84tPofyx0u4CdV2LVk4u+3dzI4g X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB6783 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: E5755160018 X-Stat-Signature: 4z4jkpeggx3miw78n767jyt1hijqc85d X-Rspam-User: X-HE-Tag: 1673971128-565886 X-HE-Meta: U2FsdGVkX1/ViTxfKwtBzaIWU7UcIgDoest7iquUH/03ja7gX0bNyLLlHhIexZKEfHEKxpJMGC1e66nIOdzGOJvbO9SORyMz08IIqugE72pWmZDLsoptZoWQtNpmuLdeJ1mAE+cilqOvanEUdcO8iN5bo0xmkWb53xtQmuc68xAcXO3xn7qV/6UvbJX7Gms/jwKbOUs0NR2AQcEoRIXnGqL0QiwifipQDA0kLYmqFk5buqC5QrJ5VHuUUZuScDBI55YmcUjZD0AqugS6Tkbqws7PELKxyngl9t14HXEBwPZmJoRl7S2xUjX4TYa2PWC+hOgbOgQcw1NqNyRpW9B4v1c5TOxbU/V7GH3Dxtmk7D/RC99TT8EiFnsTQm2gyBbXHFPMkib2lx5VbEu+SJCSNVsWrnqqA4fVMqLVdkX6r7VwilrLve8rFl1tLZsv31AJDs4MhL8RG/mCZrQqwSQkSxyU77QJcJOjm2yGBBsBrpdY/sb17goP6eqapixVivHgjDlRFh8g7da/q7Dur31Smht2swC8reo4tqkYV7aPmIpj+l1dhsY+ERbp228Lh6yh6Qx4/5Jt0zecu8warw+jE+L7Wr4hNTt1p0UuIgYpXGCw5ZDas4HRQ/pBJyXak//5PWLLff5s7vNGmqGq5+UJdl6jeTeUx9ImYhwzCX8t1c3sACT4n5iW4zEwG+UrsJpAuq/0Wm/dd0JjrVnHmQxr77eSooDPobhOSdK53O5SlUBqfyofJivGsiGRzPLO2RtDxnHJZK4oeExNz0/9X8+kRXZG80PXzNPOAVpboDqJ5TRkew9HuxdiBOpBeVEBDe2SKH4PZNmdnxzpbsLD8/BQe6oXfBHEr4GQxbGOR8c/L2pd+aSDVdqGt4WlwDSwk6z5qAvMNWcC5uKTaYIAI+GhsuTQtprB0GNtBMaRi+1TYEBnmz9x+mg50wWA0FGib5cHjjtH6ikb/X4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The GUP family of functions have a complex, but fairly well defined, set of invariants for their arguments. Currently these are sprinkled about, sometimes in duplicate through many functions. Internally we don't follow all the invariants that the external interface has to follow, so place these checks directly at the exported interface. This ensures the internal functions never reach a violated invariant. Remove the duplicated invariant checks. The end result is to make these functions fully internal: __get_user_pages_locked() internal_get_user_pages_fast() __gup_longterm_locked() And all the other functions call directly into one of these. Suggested-by: John Hubbard Signed-off-by: Jason Gunthorpe --- mm/gup.c | 150 +++++++++++++++++++++++------------------------ mm/huge_memory.c | 10 ---- 2 files changed, 75 insertions(+), 85 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 2c833f862d0354..9e332e3f6ea8e2 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -215,7 +215,6 @@ int __must_check try_grab_page(struct page *page, unsigned int flags) { struct folio *folio = page_folio(page); - WARN_ON_ONCE((flags & (FOLL_GET | FOLL_PIN)) == (FOLL_GET | FOLL_PIN)); if (WARN_ON_ONCE(folio_ref_count(folio) <= 0)) return -ENOMEM; @@ -818,7 +817,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address, if (vma_is_secretmem(vma)) return NULL; - if (foll_flags & FOLL_PIN) + if (WARN_ON_ONCE(foll_flags & FOLL_PIN)) return NULL; page = follow_page_mask(vma, address, foll_flags, &ctx); @@ -975,9 +974,6 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags) if ((gup_flags & FOLL_LONGTERM) && vma_is_fsdax(vma)) return -EOPNOTSUPP; - if ((gup_flags & FOLL_LONGTERM) && (gup_flags & FOLL_PCI_P2PDMA)) - return -EOPNOTSUPP; - if (vma_is_secretmem(vma)) return -EFAULT; @@ -1345,11 +1341,6 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm, long ret, pages_done; bool lock_dropped = false; - if (locked) { - /* if VM_FAULT_RETRY can be returned, vmas become invalid */ - BUG_ON(vmas); - } - /* * The internal caller expects GUP to manage the lock internally and the * lock must be released when this returns. @@ -2075,16 +2066,6 @@ static long __gup_longterm_locked(struct mm_struct *mm, return __get_user_pages_locked(mm, start, nr_pages, pages, vmas, locked, gup_flags); - /* - * If we get to this point then FOLL_LONGTERM is set, and FOLL_LONGTERM - * implies FOLL_PIN (although the reverse is not true). Therefore it is - * correct to unconditionally call check_and_migrate_movable_pages() - * which assumes pages have been pinned via FOLL_PIN. - * - * Enforce the above reasoning by asserting that FOLL_PIN is set. - */ - if (WARN_ON(!(gup_flags & FOLL_PIN))) - return -EINVAL; flags = memalloc_pin_save(); do { nr_pinned_pages = __get_user_pages_locked(mm, start, nr_pages, @@ -2094,28 +2075,66 @@ static long __gup_longterm_locked(struct mm_struct *mm, rc = nr_pinned_pages; break; } + + /* FOLL_LONGTERM implies FOLL_PIN */ rc = check_and_migrate_movable_pages(nr_pinned_pages, pages); } while (rc == -EAGAIN); memalloc_pin_restore(flags); return rc ? rc : nr_pinned_pages; } -static bool is_valid_gup_flags(unsigned int gup_flags) +/* + * Check that the given flags are valid for the exported gup/pup interface, and + * update them with the required flags that the caller must have set. + */ +static bool is_valid_gup_args(struct page **pages, struct vm_area_struct **vmas, + int *locked, unsigned int *gup_flags_p, + unsigned int to_set) { + unsigned int gup_flags = *gup_flags_p; + /* - * FOLL_PIN must only be set internally by the pin_user_pages*() APIs, - * never directly by the caller, so enforce that with an assertion: + * These flags not allowed to be specified externally to the gup + * interfaces: + * - FOLL_PIN/FOLL_TRIED/FOLL_FAST_ONLY is internal only + * - FOLL_REMOTE is internal only and used on follow_page() */ - if (WARN_ON_ONCE(gup_flags & FOLL_PIN)) + if (WARN_ON_ONCE(gup_flags & (FOLL_PIN | FOLL_TRIED | + FOLL_REMOTE | FOLL_FAST_ONLY))) return false; + + gup_flags |= to_set; + + /* FOLL_GET and FOLL_PIN are mutually exclusive. */ + if (WARN_ON_ONCE((gup_flags & (FOLL_PIN | FOLL_GET)) == + (FOLL_PIN | FOLL_GET))) + return false; + + /* LONGTERM can only be specified when pinning */ + if (WARN_ON_ONCE(!(gup_flags & FOLL_PIN) && (gup_flags & FOLL_LONGTERM))) + return false; + + /* Pages input must be given if using GET/PIN */ + if (WARN_ON_ONCE((gup_flags & (FOLL_GET | FOLL_PIN)) && !pages)) + return false; + + /* At the external interface locked must be set */ + if (WARN_ON_ONCE(locked && *locked != 1)) + return false; + + /* We want to allow the pgmap to be hot-unplugged at all times */ + if (WARN_ON_ONCE((gup_flags & FOLL_LONGTERM) && + (gup_flags & FOLL_PCI_P2PDMA))) + return false; + /* - * FOLL_PIN is a prerequisite to FOLL_LONGTERM. Another way of saying - * that is, FOLL_LONGTERM is a specific case, more restrictive case of - * FOLL_PIN. + * Can't use VMAs with locked, as locked allows GUP to unlock + * which invalidates the vmas array */ - if (WARN_ON_ONCE(gup_flags & FOLL_LONGTERM)) + if (WARN_ON_ONCE(vmas && locked)) return false; + *gup_flags_p = gup_flags; return true; } @@ -2185,11 +2204,12 @@ long get_user_pages_remote(struct mm_struct *mm, unsigned int gup_flags, struct page **pages, struct vm_area_struct **vmas, int *locked) { - if (!is_valid_gup_flags(gup_flags)) + if (!is_valid_gup_args(pages, vmas, locked, &gup_flags, + FOLL_TOUCH | FOLL_REMOTE)) return -EINVAL; return __get_user_pages_locked(mm, start, nr_pages, pages, vmas, locked, - gup_flags | FOLL_TOUCH | FOLL_REMOTE); + gup_flags); } EXPORT_SYMBOL(get_user_pages_remote); @@ -2223,11 +2243,11 @@ long get_user_pages(unsigned long start, unsigned long nr_pages, unsigned int gup_flags, struct page **pages, struct vm_area_struct **vmas) { - if (!is_valid_gup_flags(gup_flags)) + if (!is_valid_gup_args(pages, vmas, NULL, &gup_flags, FOLL_TOUCH)) return -EINVAL; return __get_user_pages_locked(current->mm, start, nr_pages, pages, - vmas, NULL, gup_flags | FOLL_TOUCH); + vmas, NULL, gup_flags); } EXPORT_SYMBOL(get_user_pages); @@ -2251,8 +2271,11 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages, { int locked = 0; + if (!is_valid_gup_args(pages, NULL, NULL, &gup_flags, FOLL_TOUCH)) + return -EINVAL; + return __get_user_pages_locked(current->mm, start, nr_pages, pages, - NULL, &locked, gup_flags | FOLL_TOUCH); + NULL, &locked, gup_flags); } EXPORT_SYMBOL(get_user_pages_unlocked); @@ -2980,7 +3003,9 @@ int get_user_pages_fast_only(unsigned long start, int nr_pages, * FOLL_FAST_ONLY is required in order to match the API description of * this routine: no fall back to regular ("slow") GUP. */ - gup_flags |= FOLL_GET | FOLL_FAST_ONLY; + if (!is_valid_gup_args(pages, NULL, NULL, &gup_flags, + FOLL_GET | FOLL_FAST_ONLY)) + return -EINVAL; nr_pinned = internal_get_user_pages_fast(start, nr_pages, gup_flags, pages); @@ -3017,16 +3042,14 @@ EXPORT_SYMBOL_GPL(get_user_pages_fast_only); int get_user_pages_fast(unsigned long start, int nr_pages, unsigned int gup_flags, struct page **pages) { - if (!is_valid_gup_flags(gup_flags)) - return -EINVAL; - /* * The caller may or may not have explicitly set FOLL_GET; either way is * OK. However, internally (within mm/gup.c), gup fast variants must set * FOLL_GET, because gup fast is always a "pin with a +1 page refcount" * request. */ - gup_flags |= FOLL_GET; + if (!is_valid_gup_args(pages, NULL, NULL, &gup_flags, FOLL_GET)) + return -EINVAL; return internal_get_user_pages_fast(start, nr_pages, gup_flags, pages); } EXPORT_SYMBOL_GPL(get_user_pages_fast); @@ -3050,14 +3073,8 @@ EXPORT_SYMBOL_GPL(get_user_pages_fast); int pin_user_pages_fast(unsigned long start, int nr_pages, unsigned int gup_flags, struct page **pages) { - /* FOLL_GET and FOLL_PIN are mutually exclusive. */ - if (WARN_ON_ONCE(gup_flags & FOLL_GET)) - return -EINVAL; - - if (WARN_ON_ONCE(!pages)) + if (!is_valid_gup_args(pages, NULL, NULL, &gup_flags, FOLL_PIN)) return -EINVAL; - - gup_flags |= FOLL_PIN; return internal_get_user_pages_fast(start, nr_pages, gup_flags, pages); } EXPORT_SYMBOL_GPL(pin_user_pages_fast); @@ -3073,20 +3090,14 @@ int pin_user_pages_fast_only(unsigned long start, int nr_pages, { int nr_pinned; - /* - * FOLL_GET and FOLL_PIN are mutually exclusive. Note that the API - * rules require returning 0, rather than -errno: - */ - if (WARN_ON_ONCE(gup_flags & FOLL_GET)) - return 0; - - if (WARN_ON_ONCE(!pages)) - return 0; /* * FOLL_FAST_ONLY is required in order to match the API description of * this routine: no fall back to regular ("slow") GUP. */ - gup_flags |= (FOLL_PIN | FOLL_FAST_ONLY); + if (!is_valid_gup_args(pages, NULL, NULL, &gup_flags, + FOLL_PIN | FOLL_FAST_ONLY)) + return 0; + nr_pinned = internal_get_user_pages_fast(start, nr_pages, gup_flags, pages); /* @@ -3128,16 +3139,11 @@ long pin_user_pages_remote(struct mm_struct *mm, unsigned int gup_flags, struct page **pages, struct vm_area_struct **vmas, int *locked) { - /* FOLL_GET and FOLL_PIN are mutually exclusive. */ - if (WARN_ON_ONCE(gup_flags & FOLL_GET)) - return -EINVAL; - - if (WARN_ON_ONCE(!pages)) - return -EINVAL; - + if (!is_valid_gup_args(pages, vmas, locked, &gup_flags, + FOLL_PIN | FOLL_TOUCH | FOLL_REMOTE)) + return 0; return __gup_longterm_locked(mm, start, nr_pages, pages, vmas, locked, - gup_flags | FOLL_PIN | FOLL_TOUCH | - FOLL_REMOTE); + gup_flags); } EXPORT_SYMBOL(pin_user_pages_remote); @@ -3162,14 +3168,8 @@ long pin_user_pages(unsigned long start, unsigned long nr_pages, unsigned int gup_flags, struct page **pages, struct vm_area_struct **vmas) { - /* FOLL_GET and FOLL_PIN are mutually exclusive. */ - if (WARN_ON_ONCE(gup_flags & FOLL_GET)) - return -EINVAL; - - if (WARN_ON_ONCE(!pages)) - return -EINVAL; - - gup_flags |= FOLL_PIN; + if (!is_valid_gup_args(pages, vmas, NULL, &gup_flags, FOLL_PIN)) + return 0; return __gup_longterm_locked(current->mm, start, nr_pages, pages, vmas, NULL, gup_flags); } @@ -3185,10 +3185,10 @@ long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages, { int locked = 0; - if (WARN_ON_ONCE(!pages)) - return -EINVAL; + if (!is_valid_gup_args(pages, NULL, NULL, &gup_flags, + FOLL_PIN | FOLL_TOUCH)) + return 0; - gup_flags |= FOLL_PIN | FOLL_TOUCH; return __gup_longterm_locked(current->mm, start, nr_pages, pages, NULL, &locked, gup_flags); } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index abe6cfd92ffa0e..eaf879c835de44 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1039,11 +1039,6 @@ struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, assert_spin_locked(pmd_lockptr(mm, pmd)); - /* FOLL_GET and FOLL_PIN are mutually exclusive. */ - if (WARN_ON_ONCE((flags & (FOLL_PIN | FOLL_GET)) == - (FOLL_PIN | FOLL_GET))) - return NULL; - if (flags & FOLL_WRITE && !pmd_write(*pmd)) return NULL; @@ -1202,11 +1197,6 @@ struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr, if (flags & FOLL_WRITE && !pud_write(*pud)) return NULL; - /* FOLL_GET and FOLL_PIN are mutually exclusive. */ - if (WARN_ON_ONCE((flags & (FOLL_PIN | FOLL_GET)) == - (FOLL_PIN | FOLL_GET))) - return NULL; - if (pud_present(*pud) && pud_devmap(*pud)) /* pass */; else From patchwork Tue Jan 17 15:58:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13104845 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E916C677F1 for ; Tue, 17 Jan 2023 15:59:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A73196B0080; Tue, 17 Jan 2023 10:58:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9FC356B0082; Tue, 17 Jan 2023 10:58:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 73C146B0085; Tue, 17 Jan 2023 10:58:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 583026B0080 for ; Tue, 17 Jan 2023 10:58:58 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 23D351A0768 for ; Tue, 17 Jan 2023 15:58:58 +0000 (UTC) X-FDA: 80364749556.03.02F7940 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2050.outbound.protection.outlook.com [40.107.94.50]) by imf05.hostedemail.com (Postfix) with ESMTP id 7BDBE100007 for ; Tue, 17 Jan 2023 15:58:55 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=RLZ+3Z10; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf05.hostedemail.com: domain of jgg@nvidia.com designates 40.107.94.50 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673971135; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v5o2IKvIXUjZnA3k5FObffCFUYi7W0moro08HtJtOww=; b=yoi/uF0PBj9cRtyTH0MSM3R1uhM49aN5tXn02mTxrvUv/cKmVplbslHuxV/8kgx5pvaHHC JSy9FwGZbZwZA4mTE2Hfg1qt7faJJ9akdo9xgkhMQmiqVvQoWhj3lC771eyoefePF7LCMO 4t13sDrSILIIaTEZrwKYeMYNh0CXaPs= ARC-Authentication-Results: i=2; imf05.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=RLZ+3Z10; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf05.hostedemail.com: domain of jgg@nvidia.com designates 40.107.94.50 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1673971135; a=rsa-sha256; cv=pass; b=ZopJ3yvFFSBjroNKEuM0hZcZjRqBpscBKKC5a22ykfDRQvRwE4zKCwMBVTjXYdE3pBHxJI beBGERRXcywyCcCfnS6bJIZECHRaWnPaKpSa54O+cUTc6Ful1U3nEiSY/d1fVKNoM/cHUz 7aZAkisiooV/uTJRol+L2yqd3djRLyE= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Qsb+9XfgPnc4Cq4JiH4PAFF/8B/e1AAXNScpjigGDvqNOJO2twZFu7TPUVE33OcNCxIia8gLmXhRszA3gdf0aiRfKC8xvw4e+WtOllcbhboViXIDX4SwI68uUsgFedZBcS9XrQRMFmpKL20nanhSWgwuyXpaHr/0u3gcTcUFOQnzVUaxeVGRGhBvjsgluNwbx/Jyi5NYUu75+HqjYm3Kkbq5cpety9VER0zdiP5EckUfctovp1fRyzg3kuS3zpksH2n8gzq6LJdx5vpwxhmakFLnAFGytF1QBrjaQ0gVqsEjy0Hk7CLBhLsFkHDNeS2mz3fb0p8/eR0jaSA3Su7ATw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=v5o2IKvIXUjZnA3k5FObffCFUYi7W0moro08HtJtOww=; b=hcKiMlRXeMrdjwn0hj12hvoP8gL7kK4ntYq0m5eYod4wY5GhKIoK7YHXs2KwTiWqhLw8Dn0GM+jSnB8rccHVgp55Bj5mfCPCrecPOKkV3NwsI44ifSzhAjY8sy9cRuto1YWE2O+FQXXJ5kB50C/xV5FUccqgg94I3yX/MojEE1hcz88vbscpvBUm4+mJ3sj3cHYC/wJ4xW/RsD3rhi3XHFuONV8vKbs4Eer7c0gVt8ZUEt6QVdJ8SooE2V6egAIGF8dQFEPprHfCi5uigaNM3EJiA5/XljR9MKxir8Z3beRC/oYHWt4mGWWmWgOYmEwscuYXwYwcfFSjM7tSE1FFpg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=v5o2IKvIXUjZnA3k5FObffCFUYi7W0moro08HtJtOww=; b=RLZ+3Z10HEDBooQ9A7p2ZLkDruzu7DxBeVhWyJNDTTgZGgsv73jpyZBKSpsES0WhIyZfiD39SGYhu1y5M6PsvZh1lyLPEjcBP4//WchuGqOOJJyIXGeeJBIMB+AtcviuvRCkbzzVZAe2FMrrDLYuDb/ovpdYlp3Cz14Efm0yzDnEFrEdqO6tAhYlEZ4RAb5UAMnKEnxT/Fdq4B1N09ezi6J7u2Nkhkdg1cnpVMbkWBnIPujf+tpyI7XZquNP0eJ0kXJPbK6aMcuJTQlaqH4bBuHb96RBhbzZ8PFgbE53CfJhQH+mXvtpYDyEafCQl5eYceWvhJSb82P+nl+iDOvM6A== Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8334.namprd12.prod.outlook.com (2603:10b6:208:3ff::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.23; Tue, 17 Jan 2023 15:58:45 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::f8b0:df13:5f8d:12a]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::f8b0:df13:5f8d:12a%9]) with mapi id 15.20.6002.013; Tue, 17 Jan 2023 15:58:45 +0000 From: Jason Gunthorpe To: Cc: Alistair Popple , John Hubbard , linux-mm@kvack.org Subject: [PATCH 4/8] mm/gup: add an assertion that the mmap lock is locked Date: Tue, 17 Jan 2023 11:58:35 -0400 Message-Id: <4-v1-dd94f8f0d5ad+716-gup_tidy_jgg@nvidia.com> In-Reply-To: <0-v1-dd94f8f0d5ad+716-gup_tidy_jgg@nvidia.com> References: X-ClientProxiedBy: CP5P284CA0091.BRAP284.PROD.OUTLOOK.COM (2603:10d6:103:92::23) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8334:EE_ X-MS-Office365-Filtering-Correlation-Id: 2b2b031c-b3b0-4987-3769-08daf8a3b6a6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: PwJoXoAH7Mb39t+8eLwWdfJ0W27ReCJGkT+AK4CT0rOHArQUs/I2wDEkJic5/ds5V7afe6KGpaCQkuIdITNltUSxyuNCgItPsJLkIkuTHq63VKS4kbZAOytqu/HOtx/8/kMXpB1+fzkwbTs5dq/Ka7ULNn7x1ZCHsUHTqRD70JJDME6I8gmZVjix68lbS04eXJAVm5wV9/rkE2lYxLTnRZwEpfNpAAR0fTSI+sv64OR35UavS9OoC4RRVdlVRsoxzOBy0yNTs3eItyLqoQb6TMtYCiYtStrxucgEkFq3+zDrwnxmcAhma+LnSsF2r452ysI9wEFuZRYvMIbEqUEkfDMc6HITff9/XxUZeah9FTeyBd+SMx4BN31YyqDwvmU2YFz7bDbzK9i6xN7Bpm9PvTO6nCvRA40aNubw1o6nsE+GUwuKpOV7aIHKjn5b+4FM3nikZfpqL9QvBk7hgF38PjbHOaLdIdoouw1dp+n3SDUpV28oSSL4YR0cm6O6JnCE+cNTEUY/gHCMAISB3nEMreWut/v8E3J3m/R4uMBWnoZKPP+6mCjxjSvZi7uzcRKfrdrs+UVHubthZjBFx0QxV5/seKRYKmDMSs8uA69VDH8rxh9ulQ9hfhYE2OaZGQQlyY2KEQwFT+FduiHI0djT95VqSA2yGntoOPanPnUziqROzutEqdnByh9hqtSpAsQFrpspA/9WXfBp/8C1cnUNvg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(396003)(366004)(346002)(136003)(376002)(39860400002)(109986013)(451199015)(36756003)(86362001)(2616005)(66556008)(4326008)(26005)(8676002)(6512007)(186003)(66946007)(66476007)(41300700001)(83380400001)(6666004)(6506007)(316002)(54906003)(478600001)(4744005)(2906002)(6486002)(38100700002)(8936002)(5660300002)(4216001)(266003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: dubOv2ggdMLqo63/RD3ezEQ7PJAyQYTpNnJZ2ACXaFBl4Zg5OF8ZEtHM6pdMt3QEMNiUx7Hzi5kx2J7OSCuMau7KPPM1O98ziwnil12LqEKYtOWmPJ86dvz4cv+ce4Y6LdDFz12lMea5d8ZpE5lPpobNZU3argIVezUee9TrqbZj7BQk9AxnUyYGRBEV7NYcqLig5WaxjBYA3sf/vsuyJkjA+2XlCbqE3zhK/cROAnzHdP9qk73Q7NOiEQ8knoy9DK2FH45Ao8+Zc+Gl0srlopyZS4bTG3IzFEjV4TysJ/e/5MsKBUTmaJJFS65K6FMEx/O8Yzk9nQIE5+1VeQ+eqvz42UErEbVJ3UvF34KzoaSX+FUHCVjJdi853RubCf68C+gZsiyXcWa6W1P4GkButkrNxU01HjswJe9RdjCwD1IcbRKXdz9ojDggd/dKAf/w+D+TVdEJvYUM9oTrz+ewKy70Q/fs7OL2x+5e87rqBk35X6ydPIu1GW6BbajqsHuUBpZEyZgX87koHCcqS7Wv4A2uCWq3z0hUJ+4+3fF89g1m9C4VCaYtgXaHfxwvzoStQuEz2tNUrcc8PthuEDvmSrwu7AWQJbhpF5JYtydrEGWhrPeI6nKymKHmE2lMIe5aqlShF8PvG9tqrCkvDVj3ECG8oQk3S+Rn9+tSRE9XiVsXVGPesc9XAGWX0umLwZrktgCAyyKoWwi4HHl9ogA63BWuW6fgiu9lK6lNRE/Rm08eVetbaZN72soPvI5GIbmWXvfWqTygm1mac5Fd/jXzeYs6D93XniXraoIhTExtJnG8tB4/f9+2QCOQY/iObTUHDfe7K2ojQzrnIyCKttrKE7Jnsx8lGc7KiecijoXa1IOEdrdBdSGBnlivYPiejfpYSx7+KdNch/yIkZavQYOIpW8HnfrPuHto2i0NadjC1rpXmbm6OAau047MI1RmPQtJGHxvigOHKxBk5ZQfV482FswOSw1mp6erg8/BT1yJ+58/2EcwH1eHgTC0HDbHwEtCW1IJ9d1pv1jfBphvUFItgouG9Sk8LI6ZpWlxu6sfTgk50gfi9L/h0ZBrYcR5lmqFICoHPwIbmgdAc7c6U2UhdHhPtyYyBeq74URu++akKy/z3tw5SE4o8eKCWD+Z24FELi7srrd/zD6k/unebLa/TgyYzHk13ZSVdcU5ZR3dE+xLBc7do+9XchtA7t4NoXKfvz0mfibEbTP3lUDYKmfTFsNmdLPZ5GaGahYstkDFs//FRZsGs1k/CrU6IXht7czqQdVFyf/bMp0N6Is9bUB64oQgCg9QtKB9McEIyw5OQHGXWw43gDlzGe3674mj8Amy1Zw7QrCnVKUHjLGhgTevn+9DKjVRvr2H/jqXU76rdoY+u5lMFZ5fDqbGpB6Ky2XKqxabPHpgqV3xgK4UbdHp9pr+/pyBkA0KqJiix8sxU7uLPNOAw32gPINj1UlGmXNGPELvAKQNEEc3nTBynDgZLR8whnUX5jYDQxqA7R3BMR0chKXz1yfpJ4QuF8i93IXVmYfe2hLAyGZTSGqwUnwbSLVVzMhugvXED7ewzv0cs9oS4v7BCRSYO97Mz7i/IwgF X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2b2b031c-b3b0-4987-3769-08daf8a3b6a6 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jan 2023 15:58:45.5456 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: zJQ2d3OYSVEQFjzqzFdFXcWO2XD6VhTdnKZskBKhsrWoZVSPu2m7GHdnX52tVB2Q X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8334 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 7BDBE100007 X-Rspam-User: X-Stat-Signature: xh8s4wrww78inxmc5bhffxfh4h6k7abu X-HE-Tag: 1673971135-30500 X-HE-Meta: U2FsdGVkX18ht++9rosolemUdcusPJ1BOC2oFSYQu6aOI+U6FnUSt/ggOZytGoO7/tMvaCrsqeydkZ17MENrziq77u1eeCss3vJH/CZUVkRgoSrkM9+CL9A4Zaj7iyfmbmqJ0R4cCqApAGOJcLWXJb68jp+uGDDj+D2nxnTCAIhvFNtdcI8B5P5SKTLaTyMr2d2X7PL/A+TV3gDt1OPPwfcGWAmiKKPVYEVvru9CVIIP0NPxoRjTV2ka1M2v3OYe0+Sk4qYQ2QV+K2pakVhJqD3fTw572LcIzdgTP1hKHce9n/gWs/nT9Vg9CfKZh+wn4NtVJ55ZA2Jm15N5HZhxt+In7xPy5JiU42tUGA4npJfeTWN8Dy1x+Qw4i4KdT7DDBn82UiHI0mImYp55XxjpKUW5+n0kbAvrlapeBoZhpNh5kM+LIKL8FJei2finriWaDM5in2iRf8su1K3jAm1BxhpcNwvRh/Y44618v+xrMrOHe9Yb33nhCDTHnogSy0nUnmADYc1M30dGDQMOsqmnk8jAfxKcb+Fp1F920jQ/T9y70w2wWjoB4w3DeWl+OIydOFZ6vnejwB75EbZC3OuTFUcUz7wnH85kr+xv5IHK57xdLBx7fpgwq2BDvDcFXQe+9qS+o7ASPndXeSYYT++NuRnLLuC4WfcolQrKEPbXoblVzXJYXvZg/bqS3ljvGW61G05SWgermDjg+8++QA1olGTup0e0N2j8R0yIiOgb8J7uEF9tXL/Espii+Iy2+8RDRXCdnwAPuUAZDwGC9rThAzL7EU7k6Dafd258C8XGcfy4p1iTotgKlQtDohe8FgnYC2MAiFnQl4bmMNW6/I84kILuPtX6Bq1TGeZIU7f73+LDMVqV/W1CsHMlujRI7CAaonPDm1fnwVZqKn9d3sa7IGmEtMZPijSiFeKGKFVstcLoVAIVTp3fT2tb3vVGoESQEEidPUAOvL4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is always required, but we can't have a proper unguarded assertion because of a shortcut in fork. So, cover as much as we can for now. Signed-off-by: Jason Gunthorpe --- mm/gup.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/mm/gup.c b/mm/gup.c index 9e332e3f6ea8e2..d203e268793b9c 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1350,6 +1350,14 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm, return -EAGAIN; lock_dropped = true; *locked = 1; + } else if (flags & FOLL_PIN) { + /* + * The mmap lock must be held when calling this function. This + * is true even for non-pin modes, but due to a shortcut in fork + * not taking the lock for the new mm we cannot check this + * comprehensively. + */ + mmap_assert_locked(mm); } if (flags & FOLL_PIN) From patchwork Tue Jan 17 15:58:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13104842 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50EA9C6379F for ; Tue, 17 Jan 2023 15:58:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C53936B007E; Tue, 17 Jan 2023 10:58:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BF6A36B0082; Tue, 17 Jan 2023 10:58:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 936F46B0080; Tue, 17 Jan 2023 10:58:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 86E916B007D for ; Tue, 17 Jan 2023 10:58:55 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 55A291207E1 for ; Tue, 17 Jan 2023 15:58:55 +0000 (UTC) X-FDA: 80364749430.20.A2E5B87 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2050.outbound.protection.outlook.com [40.107.94.50]) by imf05.hostedemail.com (Postfix) with ESMTP id AB8B2100007 for ; Tue, 17 Jan 2023 15:58:52 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=U6rKZsXu; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf05.hostedemail.com: domain of jgg@nvidia.com designates 40.107.94.50 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673971132; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2tu7TiL3PlSe3N1ZwZ5IAdG9YwNZypbtwySgKrz0Cxg=; b=w7hZg93efIl6NhagnO2sgKfTHyrZo1tNtTPOKCipXY+IibdwjpaEchksFP16Z2RDCNmNcJ kDE0Kr9CPf/WWWYSLIQT35uwVFlRT8Z1lOJldeMdJuXr1gazShdKtD0Xm+V5vdaW9gwZBE nGpxE8gZcOMBHRaB31Qlw8N5Mv1r7Rw= ARC-Authentication-Results: i=2; imf05.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=U6rKZsXu; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf05.hostedemail.com: domain of jgg@nvidia.com designates 40.107.94.50 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1673971132; a=rsa-sha256; cv=pass; b=AzhqgXt/ccnbaTKzmLiP4+aBIc8RzoL4hBb/SnIoN6dDVpQZyk4TBF49fE9HtF9vkFXmz5 yOwHg7EAGV11mYgizTHuHSX2XPbTTJJp1WoiLdm29wQi2UR3V2VzZ+PQp0mj2lUYSQrJG3 GXXuy3PtZrf2UBBqxfAg5W1z/XcbpG0= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KD8L257cXofDqTtyBl0M8CpNOg+ujxZ7icv/Tynxdu13IGtLINAn7hsYtgwW+cOG6c7XPR/apddQ0i0cqHookfZNsp0n9f7x91wqwDcvKZ+o4JOF3J9CHIvOe2qQmhzdg1VqlqsxSG2YBsAqS548W/qLggvss+dbGn2qlokUO6czR7LGAuccLKVmWP5ye+U2tZwYS5gYJCuLM684WcNOSJb6eIx/kHrJQokJLEwXUaaRP+2VPwf7xWyOukVqztjYYTerYWQUraOWHh9CRLP5AFaZgzjrd2t8rfyxKoByPb3FM41bZQPFzPooCEc3kXrGvCNUvWFrPgLfMi2Q0i6Yuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=2tu7TiL3PlSe3N1ZwZ5IAdG9YwNZypbtwySgKrz0Cxg=; b=Q6iyMYfpgN5TIQY0zy8Zbu9DWoARUdis/4KF4ic01yhTGWWR1HhPTUNDL0W1fBeAxSnhZ0sgc+JYEVbqJGuak18FReAx1SV1XEwys350T+GPx2tI3C2ZT8BwFCDBqZXQLw019e60vi8sdrakdv3s6u0XYpoyPklwlQiDKLEbHt21V96FsDg8HPYTe6XZl199t/189S3YlMPHAuMefl9jUajGhz2eR4K4ZwREchBj9S0ONqzkDQTb2WzjXkcxbG60XWFTo2GLV5rheX6fe1jV0CwHUWr34edwSuNIA++dkIfUz5Usni+1z8dxadcwWw7SGxxsHpp8aiPhUltcOsP3zA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2tu7TiL3PlSe3N1ZwZ5IAdG9YwNZypbtwySgKrz0Cxg=; b=U6rKZsXuBZMBP4+ikNAB6pTkE6cwwvGF6s5PgNgCtirhKL6gnLRq+UZygpAroLB5jsoAexLCG1NVHjZHg4aHEWPgLbAjp6Cno0meTDTt6ZzJUfl2g46g4Gb7ALR9BYLWJQTFtmtnacea1fXtKWENZ8TGuCtPe49HALgKRj9ZXr7dhz7AhchQJkWXRHflWFGQg3s33tNO9wULL8mcT49JjfmmUfhm8ETD0hBe+4dQX7WiUfE7MtsOwDXBI3r5OAZVbQjGOx1+Eid2CX89pmQfSya7W9/2bs4dTfQ9LDNDaVn9hsEvZgiwUZv4Xw1TOJdzTOUz89iT4RdhJUaoJbXAow== Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8334.namprd12.prod.outlook.com (2603:10b6:208:3ff::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.23; Tue, 17 Jan 2023 15:58:42 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::f8b0:df13:5f8d:12a]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::f8b0:df13:5f8d:12a%9]) with mapi id 15.20.6002.013; Tue, 17 Jan 2023 15:58:42 +0000 From: Jason Gunthorpe To: Cc: Alistair Popple , John Hubbard , linux-mm@kvack.org Subject: [PATCH 5/8] mm/gup: add FOLL_UNLOCK Date: Tue, 17 Jan 2023 11:58:36 -0400 Message-Id: <5-v1-dd94f8f0d5ad+716-gup_tidy_jgg@nvidia.com> In-Reply-To: <0-v1-dd94f8f0d5ad+716-gup_tidy_jgg@nvidia.com> References: X-ClientProxiedBy: MW4PR04CA0062.namprd04.prod.outlook.com (2603:10b6:303:6b::7) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8334:EE_ X-MS-Office365-Filtering-Correlation-Id: 153a646d-dc6d-4355-0eef-08daf8a3b4b5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: LueFUuHQbM+AsloFekvN2tfOyhZIrU6pbYapQb97wTgwnIHP1NBFFUrgLDwlCCzPgnhmwspKn+MVk+cwZrpRrhf16f4bbfPJ5mL6gw20jy9/1OJEk3CwaczXZkZu3SkBqukoGSV83nEsVW+AZXI1eyJrlLSPPHDda1BQgV/DrBbEi/LCXRYj4L+uvfCaB//PXiHHT9Qx8LP5H+mEp4nPLjhNMv5njzFtNPT5cwTbVUdzii4hRY8yYzCXrutUDR3qkv0xNPbL5eM9v9gYgu7tm437kOOFkxp15jLSUdtdp9LzTSxfLHE/NN/BJ6i1qucXwX+v4ldIxnwWY6pTaahdONvfjWI31dIY3TJ2AAQQx8JYx90ZxCow50laFvF+3Y70CJ+NsItV4xsWBNIylYpzoiVXAIU9w8DC+rxfg7Y7lPw3pnEulwVun9drNyPD029JQ4T+3g6GycrNnfWpj+X39td174rRC1DYO3OUlAlWz9J3NZezPJhy+x3Sn4f7+Tccu0LjecNC2HMtpU/xmZ3YcTn+6RbznwIyFGTQv8bBmMLfJOwWzFSCiOD1Au/qsTBp/8lOf8rcIEaOzX+ViFErtZHBKqHIteXzg8+SUGcAjy7A/K4dIsFMnzY9Qz57711LFnjftGGdfLS4WW4o4NyBmMfawftK9QebXW4uoE+l0mjjB/iG8tGC/g/KRHRqGmprmr9GoW8jiqOE2P6qrQwbtw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(396003)(366004)(346002)(136003)(376002)(39860400002)(109986013)(451199015)(36756003)(86362001)(2616005)(66556008)(4326008)(26005)(8676002)(6512007)(186003)(66946007)(66476007)(41300700001)(83380400001)(6666004)(6506007)(316002)(54906003)(478600001)(2906002)(6486002)(38100700002)(8936002)(5660300002)(4216001)(266003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: vYXpwViVXkY9oWL1RirLcSOSbAOCTmT7MzsCEH+uFJYkIs/4nQiwArn0yHw+oVVD7tHAufK3eZ+vLZqVtAAgPDbDISEHbezlpK2Mh5OeanYoxk4/ecIbJLjN6s3FzRYi0k1mW4gP8vh4geoSpD+6HtmsqYRs8J/FJ325N2174rhV/2axMB62f8l9QkkEAP7QfwSrbqSSK3YF03T+stNTsgsW6MTTajM3+gVFnw4wTtyLfMHh9ObeoYwZ3DAbrW3srOaWiiG4PmxJzAN6G1lSuHWzHtqA4AnIY72MsFtwDkj9EZmnOUGGix6M8MiWFk31JdT4s9YBEfv1rkHO5ulsCxjC51VVod/wr/k2tB12F7oJhB8X3O+2ouXyNIem45SNSrYAmuAninycwZ3rbLnHmJ53HlzYwiwp44SHGvjW6kCI/fTGv3eT77pDVU4b2O6nMbkpz/z/yYrJkFNrmM/bN+/s7Ns0NfpJoM/FJoinXf4qpEnjHwtZo066chIuuvwmwWBN91qgRZaV/nJxX6+rpZ5xMoTK4BSF/K3ePYh4ffMTvM1vTcM/e53cm8bqTqcAzZw77aCD8mJQYZz68+L7RqdAzMkXGz8d9hwrlW7L03MFxBGfbFGQy1rAYaZGWRxWoP3qOSG/Mtm+Y28Vw9gky5QLCSZevK0LkrO19o1QA1nmcwIvsviXdmLBSllaLIsSIFQ7Nt8rlltXNg05amASTx5J+5r5Xh3HtBxbDBxx+4KrDDsEmpPRYHQv6CdJnAcWnAO4q0y396N8NcyR+bye9DSMzSpvKFHTHcrL4UloU+5zztyV5OI/e3ko6O7i+X3UzAtFRhgfpdGINKzSmC07u9FWncf6Zi1b2a//JCn/NVTn5YHr7WdkVoTtcTh19I4996n3sljCiLlOiPCdndoj1uNRAB3mYxkkotzoVQYBySF0/uoJxGy85sYUM+lYw1IXHR9HDbUppxD/KQZHdIInQHwkvM53V5OXdxXZu5gyT3Vs1KSUiLxaMvkKb5kp5LHP7EDnq1zaOSMnbQNpmhfYuIlCGpvGV/BBhVBGNvjVVdlwYqlJyjC39GF3H/pjYIRXzDSLhirvzbFOekilJoHjDL2rRJvjPdISa/Zd5SXe1dvbMILxTa6achs/bHeQ+y5ECuOy/TbECIXVHNIvWKCNqX0NyTJU9Tc1X+ElQASWdEq9A8GQ9D4A+P2zRaHrlFs97Tww6JTLI1R/Be7rFI4AJsh4nLd9LnxW+mZilYw3ro0WQF0cD/QaYonl8IYUKubovIGzeKVRDqQOH03/Wdd0ZWZWGnWQf23Lj+YGBsNOxuFiztYEALoKtJA0Cs7S2FNerGeKUmLnCmOmAfNe6CuewsYCzG+9ltpJGe2OHhupwnN7YNgYP4brKy3nil+Y5KLe9LtWq4VJaDVYhjpzdB8gFl04DfFxHSGOXY+jvGgHLqvjMXOE8rgKjAVkubuHRxTNpJvtXKcVIZ0X0EGsQ70aBG3uRLAHNBcWMfiOOSErk6J7Y+4YvUsUHuL+lgr96oVzTyN44nvp63EXiLGpGEMkKiGV1+AeS54ar3ss8YhfCNwWIGOgLLYz/38q97iKCyvw X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 153a646d-dc6d-4355-0eef-08daf8a3b4b5 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jan 2023 15:58:42.2562 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 3X8SedHJ7n5xxrnicnaYwgJ2WNQG1p8uTindOiVrKIsWgiUJ1a0u2dxgKvkEANuY X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8334 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: AB8B2100007 X-Rspam-User: X-Stat-Signature: 8tw7jmptzhd3cxtuhpyfwhwrptqpdwz6 X-HE-Tag: 1673971132-512409 X-HE-Meta: U2FsdGVkX1+qBW8WJU0YR5JMHhcMfoq1K4crEI+fV+oSfwduJ7rn4wmM/LHqj1qqBvyEnn3ggfKvEQ4U64ytUNDvWDLpkYm8WeFx47CjKydhYtehyyY1NvgvStbsuxXY2SH0mrwAgF8Yq2uICjA+i7KETJ8lrQ6hNen+pe9vouXczdTVFoJvCEemv1/S4/SB5rusRiuH/o06O9on6PH5DiDWWM4NggzFYouaWVVBGhPKs4UPIzu8ezyHl2q5L0jFoHuJrYWkz4i8q6UxMDLiSxvhNlz+F09hYLYK0BeSSD2MAUxvkRk8HjGB54s7evDlAXH5Y2o2aubmVJBbs7V70Fbs+A4IHMpF06b2obZlxvRRAccghGhrKkUSXKg2V+w8VfPoEL1d7MPAmiMUR71LNirNKxmA3+CSTshp/I+nrFQI2r4rNVXWTvl5ebBTcXyP7Ou3yXYJie5rrXgyK5N/OY+3OKPBfFvrbaRKzGB8p/3lOwvQFyvfto6Et9VGSZtPFXVQl5i1DZjP8tSvinJdWciiXWvc7tXcpN/o20i+kiJYLv3aupjGEKvCMVQLv3g7WhboUPcVBB0XRdlrwQ4Pp76RxDFT2IjFTEDC4tZH1tT8G9mBQ2D70jaY1akNwLFDe3NFb7V/FZwrDPLFLybP0HUm77XUuAiYA63Bw25rw0OOh9FQXA/c7C5FHaHQAQAR/2nO386ZpwbTbS+bhPlLtAa5J6hLB9vAdJsHRgz31E7DUVLJue+tFluEVFkFrzbu1lak+25pIPiqFGF5OYnlJeL1p7yqyvX9hkNZxALWV3SGSac3kSdBoGssi4Plu6S19jWb3Jy2UN3VCw+FboO0Kt0St7CX5U3F9uWLUifPDQ9aRerjM6eM8cwenSB+JR6NMa6wMVd+KXocQqwEfXKIjCi3K47PV5kaZFWk6rb6HaujJSpVgKRKm0UxkYLUSzanWNsXn/5J9rSmt8+bmr5 g4d/cMaq 8oN9cj1GQ7dxXfxhR+r0uDfrcMQcS+DAjudWE X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Setting FOLL_UNLOCK allows GUP to lock/unlock the mmap lock on its own. It is a more explicit replacement for locked != NULL. This clears the way for passing in locked = 1, without intending that the lock can be unlocked. Set the flag in all cases where it is used, eg locked is present in the external interface or locked is used internally with locked = 0. Signed-off-by: Jason Gunthorpe Reviewed-by: John Hubbard --- include/linux/mm.h | 1 + mm/gup.c | 31 +++++++++++++++++++------------ 2 files changed, 20 insertions(+), 12 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index f3f196e4d66d6f..7496a5c8acede1 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3089,6 +3089,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address, #define FOLL_FAST_ONLY 0x80000 /* gup_fast: prevent fall-back to slow gup */ #define FOLL_PCI_P2PDMA 0x100000 /* allow returning PCI P2PDMA pages */ #define FOLL_INTERRUPTIBLE 0x200000 /* allow interrupts from generic signals */ +#define FOLL_UNLOCK 0x400000 /* allow unlocking the mmap lock */ /* * FOLL_PIN and FOLL_LONGTERM may be used in various combinations with each diff --git a/mm/gup.c b/mm/gup.c index d203e268793b9c..4c360fb05cf3e4 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -896,7 +896,7 @@ static int faultin_page(struct vm_area_struct *vma, fault_flags |= FAULT_FLAG_WRITE; if (*flags & FOLL_REMOTE) fault_flags |= FAULT_FLAG_REMOTE; - if (locked) { + if (*flags & FOLL_UNLOCK) { fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; /* * FAULT_FLAG_INTERRUPTIBLE is opt-in. GUP callers must set @@ -1379,9 +1379,11 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm, for (;;) { ret = __get_user_pages(mm, start, nr_pages, flags, pages, vmas, locked); - if (!locked) + if (!(flags & FOLL_UNLOCK)) { /* VM_FAULT_RETRY couldn't trigger, bypass */ - return ret; + pages_done = ret; + break; + } /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */ if (!*locked) { @@ -2106,12 +2108,20 @@ static bool is_valid_gup_args(struct page **pages, struct vm_area_struct **vmas, * interfaces: * - FOLL_PIN/FOLL_TRIED/FOLL_FAST_ONLY is internal only * - FOLL_REMOTE is internal only and used on follow_page() + * - FOLL_UNLOCK is internal only and used if locked is !NULL */ - if (WARN_ON_ONCE(gup_flags & (FOLL_PIN | FOLL_TRIED | + if (WARN_ON_ONCE(gup_flags & (FOLL_PIN | FOLL_TRIED | FOLL_UNLOCK | FOLL_REMOTE | FOLL_FAST_ONLY))) return false; gup_flags |= to_set; + if (locked) { + /* At the external interface locked must be set */ + if (WARN_ON_ONCE(*locked != 1)) + return false; + + gup_flags |= FOLL_UNLOCK; + } /* FOLL_GET and FOLL_PIN are mutually exclusive. */ if (WARN_ON_ONCE((gup_flags & (FOLL_PIN | FOLL_GET)) == @@ -2126,10 +2136,6 @@ static bool is_valid_gup_args(struct page **pages, struct vm_area_struct **vmas, if (WARN_ON_ONCE((gup_flags & (FOLL_GET | FOLL_PIN)) && !pages)) return false; - /* At the external interface locked must be set */ - if (WARN_ON_ONCE(locked && *locked != 1)) - return false; - /* We want to allow the pgmap to be hot-unplugged at all times */ if (WARN_ON_ONCE((gup_flags & FOLL_LONGTERM) && (gup_flags & FOLL_PCI_P2PDMA))) @@ -2139,7 +2145,7 @@ static bool is_valid_gup_args(struct page **pages, struct vm_area_struct **vmas, * Can't use VMAs with locked, as locked allows GUP to unlock * which invalidates the vmas array */ - if (WARN_ON_ONCE(vmas && locked)) + if (WARN_ON_ONCE(vmas && (gup_flags & FOLL_UNLOCK))) return false; *gup_flags_p = gup_flags; @@ -2279,7 +2285,8 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages, { int locked = 0; - if (!is_valid_gup_args(pages, NULL, NULL, &gup_flags, FOLL_TOUCH)) + if (!is_valid_gup_args(pages, NULL, NULL, &gup_flags, + FOLL_TOUCH | FOLL_UNLOCK)) return -EINVAL; return __get_user_pages_locked(current->mm, start, nr_pages, pages, @@ -2967,7 +2974,7 @@ static int internal_get_user_pages_fast(unsigned long start, pages += nr_pinned; ret = __gup_longterm_locked(current->mm, start, nr_pages - nr_pinned, pages, NULL, &locked, - gup_flags | FOLL_TOUCH); + gup_flags | FOLL_TOUCH | FOLL_UNLOCK); if (ret < 0) { /* * The caller has to unpin the pages we already pinned so @@ -3194,7 +3201,7 @@ long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages, int locked = 0; if (!is_valid_gup_args(pages, NULL, NULL, &gup_flags, - FOLL_PIN | FOLL_TOUCH)) + FOLL_PIN | FOLL_TOUCH | FOLL_UNLOCK)) return 0; return __gup_longterm_locked(current->mm, start, nr_pages, pages, NULL, From patchwork Tue Jan 17 15:58:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13104844 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E95F9C6379F for ; Tue, 17 Jan 2023 15:59:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 67A536B0081; Tue, 17 Jan 2023 10:58:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 631126B0083; Tue, 17 Jan 2023 10:58:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3BB916B0082; Tue, 17 Jan 2023 10:58:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 2742E6B0080 for ; Tue, 17 Jan 2023 10:58:58 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 01A361A0763 for ; Tue, 17 Jan 2023 15:58:57 +0000 (UTC) X-FDA: 80364749514.04.D315FF5 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2040.outbound.protection.outlook.com [40.107.236.40]) by imf09.hostedemail.com (Postfix) with ESMTP id 62F8314001A for ; Tue, 17 Jan 2023 15:58:55 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="MePP/OOl"; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf09.hostedemail.com: domain of jgg@nvidia.com designates 40.107.236.40 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673971135; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fPljsBNcNtZRVUzV7zqDRBW99+5upHByOuilIVwbwAk=; b=2GPUoBmKG9nNSAUBcrLOhcyIAXxiCZziN1dJ0IvQn5CoarINOeG87KrmfCbp5LpAUJIVkl yCvzNEof2+qDDNLqoahV49mUtulF4Ju6aoYkNVFTG70lUhEDU/Q/s7zZIDHCWSP1S8nFxt VbL15qGiMdgDqa/4A5INdrV/rkhdzI0= ARC-Authentication-Results: i=2; imf09.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="MePP/OOl"; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf09.hostedemail.com: domain of jgg@nvidia.com designates 40.107.236.40 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1673971135; a=rsa-sha256; cv=pass; b=HW8+DbXDqdcZb40kKc8sAWgL3JrTwJ3fNfhYIDLz5TwtYTjdJayB613ZDsJgpyej1sN+Gf aWLnjQejUAjAjWAS9CoFBN8rcEA//vA1GSp1H71XjCh6vcPenuE71wbGuzq7NQQXK8pax6 8JIybJ9kFcrBW/ahEIQL1seRNgyNAYM= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=m/vmfteRePZOhaoT1GIBBr/V7/C46RCtSRf5THoPUszJVFGXh5twqJClGLntYRCJbp/koczbU6rh5lKFTyxy7XfBzR7bbFil1Dm/LiPsPX0D216lZD90n+RDf3/Qb9CY6nJEsuF0G4/Hni2cznIwvguAoRGmNIgITj49/tPiG/Bjgw/Zn0AtpGr4NADPQkne6IolghRb/SzZBaHlGEJOgJU6jmIgze9Acj7h1qIe48lrDBZZMX7vFaGS8qte/vQRdqkC3oUcBs1iZeMIqVlKwWJx2xT2D8VrvVKuSwts76DZBv3zhimddWsSL/Ax/cDXFcPEmOKvXd/TECN9uKOStA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fPljsBNcNtZRVUzV7zqDRBW99+5upHByOuilIVwbwAk=; b=GBQBTZk2lW2S9BycqMs619XN9GN5FU/dYXl9nMA4uLqch2Fa4Gsc5JVP+ibx4LIkKU096cvXRncMrV70b6A5l3DpjfkRisrMevi5UFPy9IDybOI0H3FJonU8mzew8BzHkO5kBlayG8lta8OmriGvNXWZvsPs3sIf3TetccA1RCYgD/ElaqtkA8nlhubWpzrdQ0jg3+mMziYV+nSKoLpdYMzraqNt2WfOym0mlJF7C+diC6ATJAfEXV0iln6j5EfzobBIXJZfiqTi1esMcxGR4JVfC9ZT/PjER/UM986+lqsmpxSNt8FqECZ3rXBlJTw+86IxQga+RdLzR1wPaRJ8PQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fPljsBNcNtZRVUzV7zqDRBW99+5upHByOuilIVwbwAk=; b=MePP/OOlkgpIlnqSgFJagen2SoAj6aH04nIRgQ+8zmun951ghAToCpQFr+Uja0bDDOX1yIi3d3figxtjqR7igww31bYyeaS3dyBd2Yqj6miJwm/P+3L11a5iRJ2zWTnJp9z+ojOgxLCa/II2+gac6IQB0u0Ski74A8vcwYq8rO0+DT3xrIYVJ4eHt/DY7R+kkDEjIyTvMUXzcLZr2vWFdeyGZLL16b165HWt3+1RbfyPglkqRlDlfA9L7AyJOXgFwaWQZX8N4gDkIDcJuQdcZUpim4xQx3vKJE6iQI4ZxPG0cPlEBFJOBjdGF3AMvfIoiQUWYYYe1IuX+UDkS+ZVeQ== Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8334.namprd12.prod.outlook.com (2603:10b6:208:3ff::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.23; Tue, 17 Jan 2023 15:58:45 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::f8b0:df13:5f8d:12a]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::f8b0:df13:5f8d:12a%9]) with mapi id 15.20.6002.013; Tue, 17 Jan 2023 15:58:45 +0000 From: Jason Gunthorpe To: Cc: Alistair Popple , John Hubbard , linux-mm@kvack.org Subject: [PATCH 6/8] mm/gup: make locked never NULL in the internal GUP functions Date: Tue, 17 Jan 2023 11:58:37 -0400 Message-Id: <6-v1-dd94f8f0d5ad+716-gup_tidy_jgg@nvidia.com> In-Reply-To: <0-v1-dd94f8f0d5ad+716-gup_tidy_jgg@nvidia.com> References: X-ClientProxiedBy: CP5P284CA0058.BRAP284.PROD.OUTLOOK.COM (2603:10d6:103:97::9) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8334:EE_ X-MS-Office365-Filtering-Correlation-Id: 5596f5c3-f22f-4272-5e12-08daf8a3b6ad X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: RuJxMSH7DrH+n3MnO01WYWBixn90x8rMBlso/YT4Ht//UriPcW0JgpOz8c7HTrLyifgmFnlzQFm8eoSdsTwBxr6blZuvgNIUC0/JHCmfQ2bZhyAs9cq+BeeVTJx5WW6WXChhQyYh8Nqba2Vm5kJwBPCskE5QVsg2vuSVz6K1SAoRwP1Wb/s2u8Wt0vF96iuen3C0mZMB6o4HdrOaluP9tzMXbqo+gLp/OqOCteYugqNbdvBDxvtgx5oKFuRrrVdgI8i4SOc8nkrcAkzhU/c4AA6GsxnkSCwRaB96RPZHN2KRRafV79MHn8041m17X1s1vSvA9HVPM4lRbW6VO3+dP+Qcr9Q7YJcViUcBF+YosYB6rf0Xx0ey5mdxpGLpleonup/EOpEd/feNplsOSRjLOIF1wQSj5Fp0/pXf5UYjTbyvbJCeiivQNnQFyLBraqEyT3Ttw7xrMLX7b0/7KxR7zsIYeBblU4RBtvKTR24htHOcOedLjzsmjjKmenZOp0zMNogfRpmunHrZJfMV/lYu8lPfbKoR6SdmRfXVplQXGM+AU3gC/DW/JKFOQX349HUJrlekhxclScb49hICAAcJmPitncSlgQDdpojr2vDwfATNBnL8Ikc+XkBYpYGVaFG984JESKpts0hDkiG/WLk58m4/CMwv+qrumU3QUCS30pfyAcLtFmcQBkvR0YYRZhggQHuFwDmnst2TYetqTxX27A== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(396003)(366004)(346002)(136003)(376002)(39860400002)(109986013)(451199015)(36756003)(86362001)(2616005)(66556008)(4326008)(26005)(8676002)(6512007)(186003)(66946007)(66476007)(41300700001)(83380400001)(6666004)(6506007)(316002)(54906003)(478600001)(2906002)(6486002)(38100700002)(8936002)(5660300002)(4216001)(266003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: INVF+ArVsM4lpntKtXKhnlUR6YLRUOKa8PT5fWxkWcueyr04Q3S8A394EHayv2GV8nihvR61GzLUrhTyRRG7BuSeH/u6K3yIQ2xpguqxgiz/BEG+hbMrAxlnrOfP1e+JaXGw/pX9xTnOQAe9RAOPPPbGqHyEjQcJ6TGB1SqGqLpkFD8F5pVgt7u/xjZ88Qz/aF3pQJ0Xl3qh6hBvlSEZ8y2sUP+YGHYQodO2QpdeebXEXg/+XYAy5I2yEO/rTdvc6hmuz5BaVNvnFdfjcq+o5obvp3JI7cTKsSD1L7HQgTjRBO73ppjpbepsLIajMf0vFQv6CKwGp9nK4/DAxdox4/W3oGZimyRTkkkgVZahx4hKRmaDFyn7P1X31XcDk8Qkewy78JfFymncfoOzjpX1k4WcHHNL2oXwiRhkIlQL5I15fUgcRglxttt9p//bcB/aQlZFvSetBieNWVYdOY0Agk9s4IDdWaLNPV+rKfXnzGtqqvN1B7tfTSkBKuUCET/ueX6XmS5kmIYpWwUv3JYSPM9mvesiyAz79R7BWLDvpAtGunpeQY46BwIG3I5ZT7JtyGI7eA2AgmOlzXonNpLBDIghMu3ebli+4aOKiAGxq9naAefbfaghmXjQZ9uDLOIkI/w2+Z4tllLdMqoxYnOn6pJE5k5BtGfAOJMTD6yA4Pfrh5DhpgwWhqQJrYGUyjVnMxKl3ZKdixzmZ3LfxgGdlPKZ38R4tq28SHUn3/M9LiA3qR5IFihRvVmwYFbfPDB02LorIU56wA19SP0kCcwWbXho/7u6uiMuA/mFz+whAPKKxv+Q9JKrsDneE908i7A4ekgOlhgPZYpAHcB7b8r5sBneBpUfWKuy5gIamX0RpqvieHyL1c3dpZg/iki3HnGgI93EEghzCBGL0yyj2vvwShomLePN/iZWjQUSq27e6rSM1HpvzBxPJTkk9uyeiKwkGvRKkLP6OR1m9vZPL0XxGRjBgrCQ5TApfH4MdxDCc48Bh+YDcz2MEC2PiqPJZRTEbS14QoPCjpAcshBHV4EwRgOEvmwnm99m0DzzDEbe7QiTHADvkmKXbnrTjLxv1R4BWrRcx2WAgspdgtCUgdDtqyk1NbO8gvs9ywxIgbcjhwxcZQHiBGAtSOpAxxcSSjS2J77xN3PjlGWEFClAB5QedmINsgnKF9RUFre7Dkcikra9nayQxRbIjateUA/Cnrsh2zkFQ8bHBloWlMw1aQJCebKrq1IAbDv4ImD7ddo3XchH2h1yVcUoQJ8XZ7npLuWmDbCQf0/8xrUFujAMTLemFBBnvX58MfW47+vG/MTQ14jBkBh70j9G+1b0VgFHRH8QBgU5YIxM/y+9q3qPd1KPxhNPKdK0OkwzQd869kIIGEpbBXRgCKkrijt8kT0WdfeTSynzJLU9v4lSmvOUijB9s03LG8LnXPOaTv8zfkaY7r5rUBoZV+3UOWJsYL63ahKeB6LyzyISbApQV7ogPDpvMKFO0X9KC/6umDx0z/taHL/MpktahUCbnCx0cP2z3UVwfOQw6LtcKxOscXeCdEgqUQBc8Et9tK5Yqwz+1lQXmAl5gdfOZAiLNZa3xAmyxVEI X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5596f5c3-f22f-4272-5e12-08daf8a3b6ad X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jan 2023 15:58:45.5612 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: kw1OURNpYudhOSCuejDAzgH4KVTFBWhvlLY+HhY8sYE26lkfnb0AMDFjmtg+h+3Z X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8334 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: dpw47h81xa9nbq1q5bbqsd6dtophyjzh X-Rspamd-Queue-Id: 62F8314001A X-HE-Tag: 1673971135-532469 X-HE-Meta: U2FsdGVkX19pn2mDJ2xvVSozt7GP5KLc5iUP/yM5OoBdWZsOemd/BMTTZKNgueZFhsmH1p3ro8CV7rJdqRlZbWdbulVrDGoXmWtxb/E+Qh2rdDmDHULv727SL+rpXehh2t6pwY4qsTGLZlPDXSbQk92s6yM7LT8IWNbI3beWGDUOltK9pg0W4WxwUi/mvahWzbn4z7KAdAhV2yBUvhndM4BkVwYcfrMQBiNPTHMqy3enGaUOM1E9kgOJBBYHbxQVTDBe+IbpPlbctIZjhBX1BesJ0DSQd8UH5NmreU2Dao/F1MYmANADXeEud5iCYfyv97mlCMdu5o4MI1Bih7VQpqo0FYlfuEh+Mi8uWnnAKlnRWTIKG7JHihpESOgukgwBHCNdHZKWrHezwKPkGFNmHrIVfagAQ3Ci84iZp+Yi6YYyZ1zfYhrefBZbLbyyIvT3eND1+198S1kOe1fKnxHl8F0btVaSXfSGZiMYuCsVBVhUloN45gnush9zlT0ckldcR6Q5Bf9i9qskrnwm6KyYSmVMBB32PqPc9b6j/NXbziU0/4E1MPRs9ybk70e8hQJCKWY08VyKKuFjk6ujl1ids9PMUiVAKMFwAFeZWtqqTeZViQ+85bQF5qP+mjc95La1bBH6Jhc+eDzCR0MjEtSsebVBmVMOeoQII3DUc4fAYfEDPZCIgqARyP6/gwgdKvSDEYFYfR9AGjr14TvUaNq0McLqJW9KjDkWea8T0E0AF8rpr86aaWZeBMP9eX428A9x2jzhbmCf3N34oKAiAhbu26ydFjUvj6c+pKY9OiwamqF0LA7ZJikBNAY9mLoilxKwSR5x3iUo5TK9KmWJrZGyrps3F/qAm3yb3Hyey74hcNJvZ0c/sdyspt2zKmsY+epoz0/AQWNhtzCQEcWNwt4xFR3LEzPUFYJCyEFYmbbxLZenoYDIz1RnbXD5+qIGSv7nMFH8Byfvm1E= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000008, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that NULL locked doesn't have a special meaning we can just make it non-NULL in all cases and remove the special tests. get_user_pages() and pin_user_pages() can safely pass in a locked = 1 get_user_pages_remote) and pin_user_pages_remote() can swap in a local variable for locked if NULL is passed. Remove all the NULL checks. Signed-off-by: Jason Gunthorpe --- mm/gup.c | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 4c360fb05cf3e4..1ccfff759a29eb 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -930,8 +930,8 @@ static int faultin_page(struct vm_area_struct *vma, * mmap lock in the page fault handler. Sanity check this. */ WARN_ON_ONCE(fault_flags & FAULT_FLAG_RETRY_NOWAIT); - if (locked) - *locked = 0; + *locked = 0; + /* * We should do the same as VM_FAULT_RETRY, but let's not * return -EBUSY since that's not reflecting the reality of @@ -951,7 +951,7 @@ static int faultin_page(struct vm_area_struct *vma, } if (ret & VM_FAULT_RETRY) { - if (locked && !(fault_flags & FAULT_FLAG_RETRY_NOWAIT)) + if (!(fault_flags & FAULT_FLAG_RETRY_NOWAIT)) *locked = 0; return -EBUSY; } @@ -1121,7 +1121,7 @@ static long __get_user_pages(struct mm_struct *mm, i = follow_hugetlb_page(mm, vma, pages, vmas, &start, &nr_pages, i, gup_flags, locked); - if (locked && *locked == 0) { + if (!*locked) { /* * We've got a VM_FAULT_RETRY * and we've lost mmap_lock. @@ -1345,7 +1345,7 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm, * The internal caller expects GUP to manage the lock internally and the * lock must be released when this returns. */ - if (locked && !*locked) { + if (!*locked) { if (mmap_read_lock_killable(mm)) return -EAGAIN; lock_dropped = true; @@ -1679,7 +1679,7 @@ static long __get_user_pages_locked(struct mm_struct *mm, unsigned long start, * The internal caller expects GUP to manage the lock internally and the * lock must be released when this returns. */ - if (locked && !*locked) { + if (!*locked) { if (mmap_read_lock_killable(mm)) return -EAGAIN; must_unlock = true; @@ -2218,11 +2218,14 @@ long get_user_pages_remote(struct mm_struct *mm, unsigned int gup_flags, struct page **pages, struct vm_area_struct **vmas, int *locked) { + int local_locked = 1; + if (!is_valid_gup_args(pages, vmas, locked, &gup_flags, FOLL_TOUCH | FOLL_REMOTE)) return -EINVAL; - return __get_user_pages_locked(mm, start, nr_pages, pages, vmas, locked, + return __get_user_pages_locked(mm, start, nr_pages, pages, vmas, + locked ? locked : &local_locked, gup_flags); } EXPORT_SYMBOL(get_user_pages_remote); @@ -2257,11 +2260,13 @@ long get_user_pages(unsigned long start, unsigned long nr_pages, unsigned int gup_flags, struct page **pages, struct vm_area_struct **vmas) { + int locked = 1; + if (!is_valid_gup_args(pages, vmas, NULL, &gup_flags, FOLL_TOUCH)) return -EINVAL; return __get_user_pages_locked(current->mm, start, nr_pages, pages, - vmas, NULL, gup_flags); + vmas, &locked, gup_flags); } EXPORT_SYMBOL(get_user_pages); @@ -3154,10 +3159,13 @@ long pin_user_pages_remote(struct mm_struct *mm, unsigned int gup_flags, struct page **pages, struct vm_area_struct **vmas, int *locked) { + int local_locked = 1; + if (!is_valid_gup_args(pages, vmas, locked, &gup_flags, FOLL_PIN | FOLL_TOUCH | FOLL_REMOTE)) return 0; - return __gup_longterm_locked(mm, start, nr_pages, pages, vmas, locked, + return __gup_longterm_locked(mm, start, nr_pages, pages, vmas, + locked ? locked : &local_locked, gup_flags); } EXPORT_SYMBOL(pin_user_pages_remote); @@ -3183,10 +3191,12 @@ long pin_user_pages(unsigned long start, unsigned long nr_pages, unsigned int gup_flags, struct page **pages, struct vm_area_struct **vmas) { + int locked = 1; + if (!is_valid_gup_args(pages, vmas, NULL, &gup_flags, FOLL_PIN)) return 0; return __gup_longterm_locked(current->mm, start, nr_pages, - pages, vmas, NULL, gup_flags); + pages, vmas, &locked, gup_flags); } EXPORT_SYMBOL(pin_user_pages); From patchwork Tue Jan 17 15:58:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13104840 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C73BC6379F for ; Tue, 17 Jan 2023 15:58:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 234C26B0078; Tue, 17 Jan 2023 10:58:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 16F686B007B; Tue, 17 Jan 2023 10:58:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EDC756B007D; Tue, 17 Jan 2023 10:58:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D6BE46B0078 for ; Tue, 17 Jan 2023 10:58:52 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 7D2C3C0243 for ; Tue, 17 Jan 2023 15:58:52 +0000 (UTC) X-FDA: 80364749304.14.7F12EB3 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2050.outbound.protection.outlook.com [40.107.94.50]) by imf05.hostedemail.com (Postfix) with ESMTP id D52D010000A for ; Tue, 17 Jan 2023 15:58:49 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=VwQs4iKF; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf05.hostedemail.com: domain of jgg@nvidia.com designates 40.107.94.50 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673971130; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=l9eqCimBunOkyhTHJiytaKg+utKbiKcfhXDDw9h2q5k=; b=I+ODxgB+1fj3q4Z+Vd5T0QKLi0osKTLTOmLHaH7oFJIC4cr8IF5Qhoeec/ZuPr8MX089nz BNT0L2N0RSZFxJu9sOkQoQNTuda6BlRyGQrbZ9fZgVQ+8PcEurlhd7VminO8M8mSv4Q4W0 fFWvnwyE172Zwo+7m3teNUl3nRsPqBI= ARC-Authentication-Results: i=2; imf05.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=VwQs4iKF; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf05.hostedemail.com: domain of jgg@nvidia.com designates 40.107.94.50 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1673971130; a=rsa-sha256; cv=pass; b=CY0vnv1m4Ljd3gQAEkd0qlVrfdQ+41panufcE0tUDujaLF65TL8Zps54ibx+mFNqDGAjsc jYpR8EmCALfmT0rFNdioNkLM853uUXq+jy7CyYc3/UUVqTdwyHF/PSOKdZ5QjNtfTKqdXo o6LClZIJP97Ycb/vXj0BgjBrb5tOdAQ= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kuQoVzYXWdilM/B5O4jfDXJQ0NeQrebzqVZPGLo5W8Yd7QYugqfcpG7DQYn65EwlUlg6IzrsYtco95zNRMfnxT4gJTiJR7gAV2WMEprgaK3gqc1IRG3kmDaVV20veGnZRQGpshHoErZ52sb9SATrscEpigk0jXmcYnwjDaNsHw9lqYSE7Wznoy1WC4lGY8Ym+YJef7JvD6FGVur/cDRYkX/wZ6nvDS4kykaAM2eOel0G+m2S132ZRrxUwcu+sh55xtoEaH09ifELIy8hguoCEFPZPxf24Vj+FDfL/E2f2Q1ZwXv8sRXmz9PZnhtncWSXQBCfFPVFVJcXMzYQgpAUlA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=l9eqCimBunOkyhTHJiytaKg+utKbiKcfhXDDw9h2q5k=; b=bvxYSXgI8e9KxF5LLbbgnCNw4hKWrwqoRgnz2viCZ+9FouOM41oYPmXp7fE4A6t4+0+bzNk7OJ6xMz5LJfewNgzyjVDhBUlhgrM11hPpLXQma14l5IjsN8Y6QKtpEW3P2XM6Ep5edIcLFkV/uJHx99txA5yPrejvH2PXCcwWw27gIryHTsWxbszChx/FpmzcaKzMsUw98DLO1SudcYhpHp52DtOPiAEcj5rH9sZckMclrnsfFeaUZ+XV77yIc9rsiNKoUyxVKA2oQwd/YdHBV1/z5ahDvWEQRCUbf+a3sqJCZ3dWZT0iCtrgTlBiyMQciBKEoyiE/t7XQJ+zhlu5eQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=l9eqCimBunOkyhTHJiytaKg+utKbiKcfhXDDw9h2q5k=; b=VwQs4iKF/q43xpTlSXRjiJuhmQmj5JGBqUhAfN+nJChRl2hVuZ9Q58CPBxxYpbnrmCDlIRf7MWh8K8UnsHQdLPMQdMmNAzsXv745PlEAxlfLH1BcSbcFSbfZMtgTHvBHaKa9bAYtTRMbJ5wtxxCghudGkXbvBH5TU9k9cHBvIJtHQiYvMA0f8yjl403nosqubvp0KHlgHFhRYo4yRF5KTMIaLlcS+AEQVo5oTQz4YhRbtK6vvVC62jym+hz+hNWqdSzHaxeyS5m/WXJPKy1FbudO2+LD+d1hfrP6o4qaI0Ij3iDUd5a9sS33ofG/GTjzzGkHmdQteMG8BclVa56m5w== Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8334.namprd12.prod.outlook.com (2603:10b6:208:3ff::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.23; Tue, 17 Jan 2023 15:58:42 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::f8b0:df13:5f8d:12a]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::f8b0:df13:5f8d:12a%9]) with mapi id 15.20.6002.013; Tue, 17 Jan 2023 15:58:42 +0000 From: Jason Gunthorpe To: Cc: Alistair Popple , John Hubbard , linux-mm@kvack.org Subject: [PATCH 7/8] mm/gup: remove pin_user_pages_fast_only() Date: Tue, 17 Jan 2023 11:58:38 -0400 Message-Id: <7-v1-dd94f8f0d5ad+716-gup_tidy_jgg@nvidia.com> In-Reply-To: <0-v1-dd94f8f0d5ad+716-gup_tidy_jgg@nvidia.com> References: X-ClientProxiedBy: MW4PR04CA0069.namprd04.prod.outlook.com (2603:10b6:303:6b::14) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8334:EE_ X-MS-Office365-Filtering-Correlation-Id: 20a92e2a-151e-452e-afd8-08daf8a3b493 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: B4UASNbHQFL/unB7e2jmQdlejx3sc3edjVxwiro63PjyaIdxJTluO0Fs5YhruMBYvLE/1kOdbkzXmxaGgEQY4oTOt9+I9sxxoG6gG/Bi/ZiSqamuNDlkeosKshEEW7CUyz/Rm36KRlb6Hzi6zaMrYx2gtgiZuEGsxVn0JZCSbIY+4I62NuIhFSbxijBSRs4hJgC5m6ieKPCEfSDaRtJVWtc/fZ9dseG9W4CHmeBq3zBHeefLqA503cuBOTeladZqcGWume8GQUqMlM57QOMu9O3UCyrgrohmnpiy0/5Zj2vVnbtt0zQWl61iWkHP04x+dls9XPmesk8WFTVHpxLUbO0q967so+JPcUCM9bArFTJjYjxTJxiKthwrNDom9BMPXMWt34VGrWHRvwdtqZSsp+18ZcnI4kihliLQnZqob2KZcSyr/BKNGiM1T82WKdCZjvDYGc5d4ZotH/TBg4osNGcOs7AoBy3fxz1odnx8+ZtDbJ5iuOSOucZEPIp4sECC5DGgpmdVJJ5oA4RBNtuxVtF97GVaS7lf/cDPg8i5f2c+YdBCjasdCdIS63hxmJn/1ddBgLR3nNXNlm5UbgkZ//kouEs9PI6E9h8Q6JgpO64+GjemU6QfnW1p5fEv5dorbWjkLUeWm+Vfx0Os6kZsRBLBv5xahEG5O2nHkfZQflLu70WBODBpaj56Rznvu85335zkTlqDNczWa6WviKGOew== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(396003)(366004)(346002)(136003)(376002)(39860400002)(109986013)(451199015)(36756003)(86362001)(2616005)(66556008)(4326008)(26005)(8676002)(6512007)(186003)(66946007)(66476007)(41300700001)(83380400001)(6666004)(6506007)(316002)(54906003)(478600001)(2906002)(6486002)(38100700002)(8936002)(5660300002)(4216001)(266003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 58j9noFZPHlQoaZNszfUd5jnLSakXR5K2D9CV0CmXqz7jpFq84hTd3cRulr8kesAqhvQSiW1uM9S6asNC0LmvXAVTlar6tUnjCaLpRrA/LULxbHAYvF6ka2vHVSsC+rN/ZvTSu/k1xwGliVTVNYs+HVkACBVDY/oJRJodgM72t0Q7k3AzjbKJgp75H6QMth7D9e9KVFs7r7lDzZa7f3ZTOJu4CvPZ+EjzGxJSZfj7998+mC43AnmSLAwIt/23QPFFhB9BQiK+qywAA83KgLdWKfiokipv3GMpJGIcKcqUEcN50+12nsc1S/FlLv08vYnMtO4Vgmm8V2OQNJXqz9I5v/05RJZozip4lv6mpw/8PKpATj646wXh6yyQ5/JsOFk/pQl+5jN62PxN+f4CrHsLZ2FmhJUhdpYXTbMJ7TGtzNhmyqXv8gtg1NR1CgIX/pNLu8BjnX5EGtiRr8fGAHLVKIEDh5uYRqR3KjLo/Nzln7XJpGk7agKnZ1jHSWM/uodBDobWXWuf0lPx/oRcrBTN9TQID1zTykbCvHc6QN6//9ZlNdz0/SunDhHp5jPIpE9ic8uPCD9ioefxtngUKS2BxnYGDpKTrUuNedj3TZ2d8p+sC+Ls+rJUWAw1Ogkcx2Wrx0pWyhLr03jTxKzZ7mtA91MadwbMJZU6sMmavhoSz2HgsIh5POCepy+8+4fz/VFm7sjERO1fZ1mVoMbF5PUXgYf+iX/yi+DrQm4cFw1aoO/CnSVXkQT7MhSvGYFnC01QPaRGrdPIbI9z2N93zMVqeJgjeV+/DgcEeCjyEb14eBF6ntW2F1D0lveL/uCSLPZmKBZ8SCGH119qxUXX9j5sPfkB62UIVyt46m6BbJaXv5MFRemrXx6kN8eL8rvtrErDKx3+K70dhQxXzYfW6BVU8sV5w8YIKOGaax+2w5QSFrLWBWyCkznnAxeXzB3NfYcOsRFlxmkq9okj9BKBlCj6ebVesZ9ep3cONnQUutmhLPtabu9gu+iH6RV5jAzn9mlfDcTUEgeSl2zx4gOP+lMkCi/OUgKZJBx7AGOkKaZknMHpaLsZYZp5wuzmF7kraJzv2NYtUa6kMl/elD8X9Y+1TQlGeIf+3cBIytqPFILpaMSyvzWvcsjjeGC4nj21PMhOtBmDlUaq86xW1PNKU7FkyUd7vPTLv16pzDV/WAvi6YS9+wY02O/84G2SqMZ9Hpeom2pDheMlanX0CHzbsU/OJDtcluuqXyhcgT6M/MW60gQtNSeld49a8KcwxE5FQ9Ta/M2ln1EWK/pIVdfI7pXjWx8SjPbM2H+mkMbMvDQI5++eLFEZApeq0EQD/Y0lhuPQ0wYjBvl8AFZ8VyVZOGKStQOMNowd9wNzU4dsNNYkZePAY0j5mvnGE36rr++U9tWy4JwIZSq1aYrLUh4HGyjm0VBVDYOqicgt5071T7PLuYU1nsZj6zR0TzcDLxVCBwoktE3H733aESVLQTQvc4WtLCcaYt+HOMFmlwM37Hw0YhgsAk//LAP1t5aoUIT5Bcw1TdKIPaOsYD23XMbS8UNPmpaMlJoUU2UMdaLPnakfQXkz2YD8HSOiZflvAkeZqBv X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 20a92e2a-151e-452e-afd8-08daf8a3b493 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jan 2023 15:58:42.0531 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: /e5IYhcBmZp13fy/wCG4x4c2Ato9+qz1n65cUk5SBPShWkhaejHHDFI9gpxjQBAf X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8334 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: D52D010000A X-Rspam-User: X-Stat-Signature: irje88rnqpomx3eyjkaa9t35pxj3i9uq X-HE-Tag: 1673971129-152025 X-HE-Meta: U2FsdGVkX19ysFTzXkPz/IDGT849jpSVvyjddees4kYgYdREfitQt3ClO1QZ+/Uzy1pnYYeX6rKHFV+DkSdSoTpNhZtl4zjJMh/qxc9AoqOCivylSJUwLvJIGjmcOnwhAT5XaBkmFGMeDojWC6Nw/US6UBJR8cS1HZJsRTC4cIOVQtKcV9K+y1H/jiGRbiLSvvMvO/8ZLJz3U2gM3ptJwTulBD/js5OSQwAhovGpHOqsPNKHmPPW3Yb5OqcqxeiS90QhjTDpePE9iyJK8HOi3LLxz9uzcL5JZ5sNwfc7cQ7h6zmVYMy58HMHaI/B9F1EqROfD1SP8F0XAC2O/0WISDhaC+G4qZ+nUwaoC4c+Qsj8EfA07esh2wHaI9CTXrggJIqjYl9QfAa57Eau9n2p7EOL1M8tWejESUS0ty5YVs+NnkhBijHOyuUIwwy+E2CidDPeYHkVKa1iv8oqI27wXHsIstQ4cn+fArJVysNIzTmAz/RjOv2e02qWMx4AIC5cL62blwvd3UQgzfszBDE4RwxHFEEjuyzLf5qZ1Dhqv81eTRqSFhV+ICNIlawH8mcps8RlvD6P/3o/PdSPe0FRfk4rOPcvSCdAShfmCH1gGvGFujtkzP/0e2NPuAJC8eErcYDLUInuhDNc+Sp2W+DNU+B/grQESzBz1+CU8zYT1/WO5tx0C4XRFBm0JnbeNUC5FgyP8d4Daz+Wo2WnR0IZlqd9+IzLF4yIxiAroHRsuSqGebthoFRkO4d6uLwIwVrhzTMYOO4E9VMf49iHPdX8UBCruxNB83lVWboNPavPBVIBnVcaWSCj+Rq1xiEK6UshaTBFfYHfOsBAIpLw+qtG1oqxZ/oDpe50y8n42g7qLojgjpnt2GremplfZaLjDT4EaqkBDyxaMKjcMFAKYzSW9ji3UhPxY5fcssjwxkOC739Uf2BJb5gAQkF240jXryia023ISQG9MIue78IJ5sb BqqoU4nV J/odEbtK60CKY/ZPOZj9DUwv3VbGf6l05cuTy X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Commit ed29c2691188 ("drm/i915: Fix userptr so we do not have to worry about obj->mm.lock, v7.") removed the only caller, remove this dead code too. Signed-off-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: David Hildenbrand --- include/linux/mm.h | 2 -- mm/gup.c | 33 --------------------------------- 2 files changed, 35 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7496a5c8acede1..de300b0de58e7d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2159,8 +2159,6 @@ extern int mprotect_fixup(struct mmu_gather *tlb, struct vm_area_struct *vma, */ int get_user_pages_fast_only(unsigned long start, int nr_pages, unsigned int gup_flags, struct page **pages); -int pin_user_pages_fast_only(unsigned long start, int nr_pages, - unsigned int gup_flags, struct page **pages); static inline bool get_user_page_fast_only(unsigned long addr, unsigned int gup_flags, struct page **pagep) diff --git a/mm/gup.c b/mm/gup.c index 1ccfff759a29eb..b260c182135587 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -3099,39 +3099,6 @@ int pin_user_pages_fast(unsigned long start, int nr_pages, } EXPORT_SYMBOL_GPL(pin_user_pages_fast); -/* - * This is the FOLL_PIN equivalent of get_user_pages_fast_only(). Behavior - * is the same, except that this one sets FOLL_PIN instead of FOLL_GET. - * - * The API rules are the same, too: no negative values may be returned. - */ -int pin_user_pages_fast_only(unsigned long start, int nr_pages, - unsigned int gup_flags, struct page **pages) -{ - int nr_pinned; - - /* - * FOLL_FAST_ONLY is required in order to match the API description of - * this routine: no fall back to regular ("slow") GUP. - */ - if (!is_valid_gup_args(pages, NULL, NULL, &gup_flags, - FOLL_PIN | FOLL_FAST_ONLY)) - return 0; - - nr_pinned = internal_get_user_pages_fast(start, nr_pages, gup_flags, - pages); - /* - * This routine is not allowed to return negative values. However, - * internal_get_user_pages_fast() *can* return -errno. Therefore, - * correct for that here: - */ - if (nr_pinned < 0) - nr_pinned = 0; - - return nr_pinned; -} -EXPORT_SYMBOL_GPL(pin_user_pages_fast_only); - /** * pin_user_pages_remote() - pin pages of a remote process * From patchwork Tue Jan 17 15:58:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13104838 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61796C6379F for ; Tue, 17 Jan 2023 15:58:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0380C6B0074; Tue, 17 Jan 2023 10:58:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F2A1F6B0075; Tue, 17 Jan 2023 10:58:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA3B46B0078; Tue, 17 Jan 2023 10:58:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CC2296B0074 for ; Tue, 17 Jan 2023 10:58:49 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9C48B140225 for ; Tue, 17 Jan 2023 15:58:49 +0000 (UTC) X-FDA: 80364749178.04.1947167 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2050.outbound.protection.outlook.com [40.107.94.50]) by imf05.hostedemail.com (Postfix) with ESMTP id 13888100009 for ; Tue, 17 Jan 2023 15:58:46 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=QPDKUQEE; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf05.hostedemail.com: domain of jgg@nvidia.com designates 40.107.94.50 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673971127; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8KuMywXBvp0scjsK2KcpwVq8rhkQLG+F9pe1b2uQyI8=; b=TfodS19LU1or+zgzOS5JOLpx4U0nhaW63KQnZfR2nS6g3nwjGc0DQNfKoHDA1tw9AxBYFC BUdlmNwkRKA6y0Kd6vt+XyjFsRliLpY8tjxD70c9k8fhMAK21usMmBJgQeQsVWDU6noWIS aw1PvAU20gVRlnIvTvdDJfpI6zgXOL4= ARC-Authentication-Results: i=2; imf05.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=QPDKUQEE; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf05.hostedemail.com: domain of jgg@nvidia.com designates 40.107.94.50 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1673971127; a=rsa-sha256; cv=pass; b=gnzsWQ7chaS9/Sd5aWwFDye42zvEpVjg25WKQ/6g5xWFPTae+qYWF7krK9ys1PpBYhh3g2 e5wLDXbr3cz5s4gDcNEw+wgXtyb2IeRqkOMYT5mZvRHh8/epU+YPTuDQBnvRsIU8zstnbs uxezqDIeEZQblXfgF1DfD2+uc4Srq7Q= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CasRme6km+nuUD7Cq8UySqIpFSCvwz/t9zfwcoyL4UxkqPeMO3ARpIxBHThc4KABGykw/zd713AHSQvmaPKfJsYEjErjVf1M0DQG71HxaGGY1rjV7xAbBxBImBn0HTxSXzGZKfMB6mO2im/d25fsuLi6ddjHXOMjNxwvw0dvuiM17dPzcJ0djK1f0koy1tpfi+JWiSLOgqmQY6aG4MVq8WZpRDKumapVTc4wqgA7rCuVl1Hw9ZEPynGsOlBtVZDkDdKFjKruza+N92EXCCwJBIw+yjP31m8Gnj9qrNq8MTnC4OSLwO0iG6MFngUU5xbDsNKFqeTNJAYMTmAd2INXng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=8KuMywXBvp0scjsK2KcpwVq8rhkQLG+F9pe1b2uQyI8=; b=APRIbNLW/69HxlxCGXWt0nAO4C5iN9rNz3J3sIWURSNwqEsLB75lcO8htU0H7I/Gn32pirwzI0VmauxmiY1DEjRwWmukIh4PB5gy3lMnqiUkb4GXrABjsO53SVcbrFs5BoNvAzLfXtQHRIyCEzex5LlnFHdSLWRadJ6R5iUQraBmWeGr0Kv4n6rTJRbUUtzRm/8+Kf0vx9SBs7AGULkrwStpHGjGRfK3GtkuEPHh0cZKSsXp9ZmZK2hWOxN7x8U0fcu8XGWW0a9qmO5LbBZv6AhfiSPON78KLnXA9RZzWS32Ml6iE/aFgKc5cfV+nzL/Ts/ZzO2AfrjIVSiRtFoPgQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8KuMywXBvp0scjsK2KcpwVq8rhkQLG+F9pe1b2uQyI8=; b=QPDKUQEE341giaz23RsOhZsrzO04df9BkhO6uN4DLLKtj0PVw3J1xK59u5RFs5lYQvaAI9/SbU4UqyJXCQe8jF54A9rJjgo1utlM5iX+urotZOm2YCoNKNG9zExCIuFR4qKuPiohtaI1I32OuVwhq/SIAPUc5hYQeXYT62h3qmlh1OLcBt19n3wdJMtLzC2GHAQOTeKvo2AR9N8MTKWTZZsvd5zmaqSKW5mWTf5l+bWPEP3IkN/M9Io9J3L9SglNWqB1azbE7u9bjFsC33mnOcFfHZS9sQvz3cBFJ77L02b0PY930XeXt0/7gXhrTi9IN8TPhWcVdP/MA4V60avOAg== Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8334.namprd12.prod.outlook.com (2603:10b6:208:3ff::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.23; Tue, 17 Jan 2023 15:58:41 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::f8b0:df13:5f8d:12a]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::f8b0:df13:5f8d:12a%9]) with mapi id 15.20.6002.013; Tue, 17 Jan 2023 15:58:41 +0000 From: Jason Gunthorpe To: Cc: Alistair Popple , John Hubbard , linux-mm@kvack.org Subject: [PATCH 8/8] mm/gup: make get_user_pages_fast_only() return the common return value Date: Tue, 17 Jan 2023 11:58:39 -0400 Message-Id: <8-v1-dd94f8f0d5ad+716-gup_tidy_jgg@nvidia.com> In-Reply-To: <0-v1-dd94f8f0d5ad+716-gup_tidy_jgg@nvidia.com> References: X-ClientProxiedBy: BL1PR13CA0079.namprd13.prod.outlook.com (2603:10b6:208:2b8::24) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8334:EE_ X-MS-Office365-Filtering-Correlation-Id: 614e1bd1-6d26-43b5-ea91-08daf8a3b3ae X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: EpuBz+ilGv0Ui1fQhqHiC8eYLrnDppoyo+5ocbrVKCH+NZOhTD4qBxuRKoPoEmwyi8saIqL5/fPUQmWu2jjj/Gn3teHOgQBKDfuZe+UinxinBM6dYdwf+qJ1xQbZzCU0is5xq8HAuDOtneZEh0NVUGsEa3v+JA1uLt29IgYdEMFvB6SG2Db6cIi7ZoVYjMeH62xsVXmImJsJ9NvKsMl+9GrqCYl6ghGJOgZYAng9DJwEdzKLmleLq4iseeiKbGUUHajRh8741otn6Nej7cZjH6psKUDuXkqagXSGTaZ9rg45f+qX7VppEfBbo6TFHNNynsPcMmxqgyAQan4BpifvlZgSbu9zTd4gQCdUvJpEedpWFcIjz2hbC/xDksC9D2yOe5D0YKSO7bz4zdZBXG2ggULTKYXN2tdOafTwaRpuLpePmlDHhJpVm8tBXul7KJnNon35CIIfY4p54TWaKyA/b3vN5u1LtWXhQ/hF618TrxRCcdaPm3j2Hux2u5clcLyMaGhfY1cOfH64EfHeV2hsVPvw4Vsc3bMrTr7FJgitgR/VolFh7fdKLlOBhrh/9cIhD/6uCbw91IL3onYaBKi9Dxc9Zf7ql18lY8vU5M3fsDSYKDWB6ov6LzzDDdEBC2dfPvp0D0vpSudMaNBwZNfF1asnQo39sPNAQDnUEyy92U2/jqYFk8GA+eYTezxhrZwi1khwAfPvyLV3308vfBHPRA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(396003)(366004)(346002)(136003)(376002)(39860400002)(109986013)(451199015)(36756003)(86362001)(2616005)(66556008)(4326008)(26005)(8676002)(6512007)(186003)(66946007)(66476007)(41300700001)(83380400001)(6506007)(316002)(54906003)(478600001)(2906002)(6486002)(38100700002)(8936002)(5660300002)(4216001)(266003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 7Jxf60Q90w6VEdfQI546S4fLogvNazAFH3Ar1dDHFLoSx20UVvYBWQVCVed/BxtzlT2omRFTVq1Oghps6G38zuSZ4Q2Urrdy8crk3ev5L3U3P+eUz2b1yvfvQFpZ0Iln8Oqfe81aBuJT/bI+ohxqG3CWVUrpMoYIr2GeMfWs/mTJ6bxmRGTxRG6bJHwyUQU3jqbkCr6mGg68E13+G3YpAjeVVeBjlMbqWc/8E1lNgXX50TTg6JR1T8uX2h9LzCWjBe5cMWJQMcf3V5eS98E1NqD1fddmx/sUVrbjCnp4VTGDVGGidDlipyBsWmLs+0/vszRrlAn3e09tEDrWRS+Xeb9ON94Nr1nNcig0BVi8Pew/NSPO1x9qr+ocKCQBmBtff9G4P1neVDjIzaIJiAgJ7fIQZJi4sgvJ8LufODDaIOiFkmegKuVHHH+z3klAZxq2bsg7UroxkIRYXDvSh/XaJvmsQigm/xeAHtIbzpBz4wK6NINQAHDTKtx/KMi+Vy0D1SWrvXBELmh6Sp9bliTYPBUT2pbQuauBjNVWW870lkpveBgWCcJcHUeREGdLHXt2TDdf3IqXJtVJHKxBPsHLoMEpaTEXgurSZsrWfgBpEOnkT3GBgiorqeWUI+rZsfM2h5JqXVvdD3PfC2Ztc/0rNfO/fWHsgznS48UFADNBQuLKnnor0O0sMldS4QI2aidcdNj4PyHlZ8MRexuX0dg9DuJxSsjBStbDlVQbtn0VBrNVfWKTx3SCn1T3lbhuLE6npD+RMAZr2saPc+yW7ghNTjTKV7xxCUgoctoaq+osi8wAnuGjIAeIKfuzLECUvB26jK0iRZdjTVRbLx0/mQO0oYsDpNLfQXioirWW2PNy7U9z9+0QbqwugyfTxVOCvdqgx+CqZXy6UbiDH87XHLGMWOf2e9VI3HvOZRIqqQ+m/cjLH1ZXx/vLQbNtNMyBcSxJNE7k650SIpgtYHOCrrsv0W9MhIfTUKkR2JiXMvxf62an7o01qIs20MR5kOef3SpwpYDG2J1hMwYBw8zlVs8b4v6jqmnO5iFRBi1oeXYW4IMwN8zXMa6B8pBRYYvcKuxkGTesStrwk5t+1tTiBdfuzU7bWag3vaMfWiqioJ+jAzOcWhWTrz5oHpzFNSCGS+gt3cSOl0bLlIz3srJv/+saoTs+Ba2FizNUH+wPjdDieEzRWGmzKoe23orPDPzgHvwaVu9egfvsgjsug7swGj6tpxftWAxQDtAxIqa86Vc1rsVViPfWwQvdUVyy72thWWA92ox2/O9WhU9lRZK4SrZYf65qc+WMjSaPbHAxNldY44fKu1ZT+pzhbImozYubY7ma4EH1KXR/xTZfxkWDVvZ4a2gHdIV6O/JwnZ4I6FhYVUpikDqif9ggb8Uh9HMfiw5CI+YcSSS6O0Z0wkS7cTENRDKj6ccRzZWRAPjUnY7FpMtt1NFHpzp7MH60H1WEl7oLgvjjIa7zZ99XLVND7qp30bDolsYeVA7x22O9if6DZlpTmR3TcZ3QYWjVRWxrlcwSVttA+ENSamp71yhSG+Sewq5Xavqbu9mQsoEuD6ReelBwG/Rsw/Df+U9M8enQRv0N X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 614e1bd1-6d26-43b5-ea91-08daf8a3b3ae X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jan 2023 15:58:40.5695 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: bJbW2DxYbI4WPYs8icI5Jg5kalhZ0HdNZy38VrVNAPkhgH7OVdtz27ozzbXPxRV+ X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8334 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 13888100009 X-Rspam-User: X-Stat-Signature: 8p5yp3bjr8zwcu7hm4yzn4ma4iyfnhoh X-HE-Tag: 1673971126-312603 X-HE-Meta: U2FsdGVkX1/+PZnMDJTZdFon3OFtLnk6SojBzSJsE1y8abyWVNmG+gcfV46NqqDzeU7q3X69dl4yjow+/XISKVea/YiH0LSaFHuSBESz2Hody7ucauQVzk3PFXSxnF8fiXC5gr1vVPl8qJNrR3EjYYJMR4N3NBVwU2uY3VvncHFRVuPUydvKozirTLNY6qPa4jGU4GQKbuoqET//84ZejeRcJtnvH8BFGUwwdcJ6JUueNVMhaXsKx1U5UKhZdpImWHGGfDP7BVbs/I2I3drPfQLpnySQtXIHYSfpUDzaMcSshIEZtUWVkHLfr04L3uOxFX+638HWaZAJmmjlBSVZLo1nEA/CRINxYBR+HMlONwzIeQNYER9l+p0nJ1scV0PhtLh2ctImsd9IZhDj/PtkLyL6cv3dgnqND2Hh93EXuthLSp4BfssrO/dip51i7dGJYxcXlTt3sN4+5Kt1yX4Sz4mBCkXxDJnvXEcePwS1c1oleg/ZtwxbroANq4PVwrAOd+LCuKfHcRutztEc4JlF6N7ptoyxT5qX0s1QPuVNUfWDXo5gEb+qOrmbHquduKrA0wbXOw5duRhB23yCcYwTFYoGgHN7t/1P5es5jY4GZ5bXFd5vfbW8Cej/g10miiuUAVZQRRzuicf9bG0MxPpvLq07Bny9678tDoszplNELgUjKn1HyiHLADd8QrZVsk1SSBiFYKM0gr/iPCj6A+JvXk57a8+8awuoZHnnfnxSx7OwvQ1CCkKXGMfHjmlihN5Z+zmyxwOaE1j8eBGO/NX8y+gXnA/2V000xAAjC9zgq4Yd/0g6++pd+KYLqhRHualEZ8k9N7/1K9OAe+s2MXoSiJax1/1Jj9bU7eUmQrVFc+6foN6/4m/75EaB+WQvsbqvpTdDXQ7pSC29+dEoQFQ3RpjgCiqJfzK0oDPPoWUhnKkbrdBeNTKM3/pqPa98mOGnFI1vsvxPZPltyyHZd6k FjVs9Pfc t0ojBvWg57Xz/GlLzK9zM2v7QdNvChbumx5/k X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There are only two callers, both can handle the common return code: - get_user_page_fast_only() checks == 1 - gfn_to_page_many_atomic() already returns -1, and the only caller checks for -ve Remove the restriction against returning negative values. Signed-off-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: David Hildenbrand --- mm/gup.c | 17 +---------------- 1 file changed, 1 insertion(+), 16 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index b260c182135587..20ebc4d1f0d719 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -3002,8 +3002,6 @@ static int internal_get_user_pages_fast(unsigned long start, * * Like get_user_pages_fast() except it's IRQ-safe in that it won't fall back to * the regular GUP. - * Note a difference with get_user_pages_fast: this always returns the - * number of pages pinned, 0 if no pages were pinned. * * If the architecture does not support this function, simply return with no * pages pinned. @@ -3015,7 +3013,6 @@ static int internal_get_user_pages_fast(unsigned long start, int get_user_pages_fast_only(unsigned long start, int nr_pages, unsigned int gup_flags, struct page **pages) { - int nr_pinned; /* * Internally (within mm/gup.c), gup fast variants must set FOLL_GET, * because gup fast is always a "pin with a +1 page refcount" request. @@ -3027,19 +3024,7 @@ int get_user_pages_fast_only(unsigned long start, int nr_pages, FOLL_GET | FOLL_FAST_ONLY)) return -EINVAL; - nr_pinned = internal_get_user_pages_fast(start, nr_pages, gup_flags, - pages); - - /* - * As specified in the API description above, this routine is not - * allowed to return negative values. However, the common core - * routine internal_get_user_pages_fast() *can* return -errno. - * Therefore, correct for that here: - */ - if (nr_pinned < 0) - nr_pinned = 0; - - return nr_pinned; + return internal_get_user_pages_fast(start, nr_pages, gup_flags, pages); } EXPORT_SYMBOL_GPL(get_user_pages_fast_only);