From patchwork Tue Jan 17 15:58:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13104842 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50EA9C6379F for ; Tue, 17 Jan 2023 15:58:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C53936B007E; Tue, 17 Jan 2023 10:58:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BF6A36B0082; Tue, 17 Jan 2023 10:58:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 936F46B0080; Tue, 17 Jan 2023 10:58:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 86E916B007D for ; Tue, 17 Jan 2023 10:58:55 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 55A291207E1 for ; Tue, 17 Jan 2023 15:58:55 +0000 (UTC) X-FDA: 80364749430.20.A2E5B87 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2050.outbound.protection.outlook.com [40.107.94.50]) by imf05.hostedemail.com (Postfix) with ESMTP id AB8B2100007 for ; Tue, 17 Jan 2023 15:58:52 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=U6rKZsXu; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf05.hostedemail.com: domain of jgg@nvidia.com designates 40.107.94.50 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673971132; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2tu7TiL3PlSe3N1ZwZ5IAdG9YwNZypbtwySgKrz0Cxg=; b=w7hZg93efIl6NhagnO2sgKfTHyrZo1tNtTPOKCipXY+IibdwjpaEchksFP16Z2RDCNmNcJ kDE0Kr9CPf/WWWYSLIQT35uwVFlRT8Z1lOJldeMdJuXr1gazShdKtD0Xm+V5vdaW9gwZBE nGpxE8gZcOMBHRaB31Qlw8N5Mv1r7Rw= ARC-Authentication-Results: i=2; imf05.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=U6rKZsXu; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf05.hostedemail.com: domain of jgg@nvidia.com designates 40.107.94.50 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1673971132; a=rsa-sha256; cv=pass; b=AzhqgXt/ccnbaTKzmLiP4+aBIc8RzoL4hBb/SnIoN6dDVpQZyk4TBF49fE9HtF9vkFXmz5 yOwHg7EAGV11mYgizTHuHSX2XPbTTJJp1WoiLdm29wQi2UR3V2VzZ+PQp0mj2lUYSQrJG3 GXXuy3PtZrf2UBBqxfAg5W1z/XcbpG0= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KD8L257cXofDqTtyBl0M8CpNOg+ujxZ7icv/Tynxdu13IGtLINAn7hsYtgwW+cOG6c7XPR/apddQ0i0cqHookfZNsp0n9f7x91wqwDcvKZ+o4JOF3J9CHIvOe2qQmhzdg1VqlqsxSG2YBsAqS548W/qLggvss+dbGn2qlokUO6czR7LGAuccLKVmWP5ye+U2tZwYS5gYJCuLM684WcNOSJb6eIx/kHrJQokJLEwXUaaRP+2VPwf7xWyOukVqztjYYTerYWQUraOWHh9CRLP5AFaZgzjrd2t8rfyxKoByPb3FM41bZQPFzPooCEc3kXrGvCNUvWFrPgLfMi2Q0i6Yuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=2tu7TiL3PlSe3N1ZwZ5IAdG9YwNZypbtwySgKrz0Cxg=; b=Q6iyMYfpgN5TIQY0zy8Zbu9DWoARUdis/4KF4ic01yhTGWWR1HhPTUNDL0W1fBeAxSnhZ0sgc+JYEVbqJGuak18FReAx1SV1XEwys350T+GPx2tI3C2ZT8BwFCDBqZXQLw019e60vi8sdrakdv3s6u0XYpoyPklwlQiDKLEbHt21V96FsDg8HPYTe6XZl199t/189S3YlMPHAuMefl9jUajGhz2eR4K4ZwREchBj9S0ONqzkDQTb2WzjXkcxbG60XWFTo2GLV5rheX6fe1jV0CwHUWr34edwSuNIA++dkIfUz5Usni+1z8dxadcwWw7SGxxsHpp8aiPhUltcOsP3zA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2tu7TiL3PlSe3N1ZwZ5IAdG9YwNZypbtwySgKrz0Cxg=; b=U6rKZsXuBZMBP4+ikNAB6pTkE6cwwvGF6s5PgNgCtirhKL6gnLRq+UZygpAroLB5jsoAexLCG1NVHjZHg4aHEWPgLbAjp6Cno0meTDTt6ZzJUfl2g46g4Gb7ALR9BYLWJQTFtmtnacea1fXtKWENZ8TGuCtPe49HALgKRj9ZXr7dhz7AhchQJkWXRHflWFGQg3s33tNO9wULL8mcT49JjfmmUfhm8ETD0hBe+4dQX7WiUfE7MtsOwDXBI3r5OAZVbQjGOx1+Eid2CX89pmQfSya7W9/2bs4dTfQ9LDNDaVn9hsEvZgiwUZv4Xw1TOJdzTOUz89iT4RdhJUaoJbXAow== Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by IA1PR12MB8334.namprd12.prod.outlook.com (2603:10b6:208:3ff::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.23; Tue, 17 Jan 2023 15:58:42 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::f8b0:df13:5f8d:12a]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::f8b0:df13:5f8d:12a%9]) with mapi id 15.20.6002.013; Tue, 17 Jan 2023 15:58:42 +0000 From: Jason Gunthorpe To: Cc: Alistair Popple , John Hubbard , linux-mm@kvack.org Subject: [PATCH 5/8] mm/gup: add FOLL_UNLOCK Date: Tue, 17 Jan 2023 11:58:36 -0400 Message-Id: <5-v1-dd94f8f0d5ad+716-gup_tidy_jgg@nvidia.com> In-Reply-To: <0-v1-dd94f8f0d5ad+716-gup_tidy_jgg@nvidia.com> References: X-ClientProxiedBy: MW4PR04CA0062.namprd04.prod.outlook.com (2603:10b6:303:6b::7) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|IA1PR12MB8334:EE_ X-MS-Office365-Filtering-Correlation-Id: 153a646d-dc6d-4355-0eef-08daf8a3b4b5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: LueFUuHQbM+AsloFekvN2tfOyhZIrU6pbYapQb97wTgwnIHP1NBFFUrgLDwlCCzPgnhmwspKn+MVk+cwZrpRrhf16f4bbfPJ5mL6gw20jy9/1OJEk3CwaczXZkZu3SkBqukoGSV83nEsVW+AZXI1eyJrlLSPPHDda1BQgV/DrBbEi/LCXRYj4L+uvfCaB//PXiHHT9Qx8LP5H+mEp4nPLjhNMv5njzFtNPT5cwTbVUdzii4hRY8yYzCXrutUDR3qkv0xNPbL5eM9v9gYgu7tm437kOOFkxp15jLSUdtdp9LzTSxfLHE/NN/BJ6i1qucXwX+v4ldIxnwWY6pTaahdONvfjWI31dIY3TJ2AAQQx8JYx90ZxCow50laFvF+3Y70CJ+NsItV4xsWBNIylYpzoiVXAIU9w8DC+rxfg7Y7lPw3pnEulwVun9drNyPD029JQ4T+3g6GycrNnfWpj+X39td174rRC1DYO3OUlAlWz9J3NZezPJhy+x3Sn4f7+Tccu0LjecNC2HMtpU/xmZ3YcTn+6RbznwIyFGTQv8bBmMLfJOwWzFSCiOD1Au/qsTBp/8lOf8rcIEaOzX+ViFErtZHBKqHIteXzg8+SUGcAjy7A/K4dIsFMnzY9Qz57711LFnjftGGdfLS4WW4o4NyBmMfawftK9QebXW4uoE+l0mjjB/iG8tGC/g/KRHRqGmprmr9GoW8jiqOE2P6qrQwbtw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(396003)(366004)(346002)(136003)(376002)(39860400002)(109986013)(451199015)(36756003)(86362001)(2616005)(66556008)(4326008)(26005)(8676002)(6512007)(186003)(66946007)(66476007)(41300700001)(83380400001)(6666004)(6506007)(316002)(54906003)(478600001)(2906002)(6486002)(38100700002)(8936002)(5660300002)(4216001)(266003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: vYXpwViVXkY9oWL1RirLcSOSbAOCTmT7MzsCEH+uFJYkIs/4nQiwArn0yHw+oVVD7tHAufK3eZ+vLZqVtAAgPDbDISEHbezlpK2Mh5OeanYoxk4/ecIbJLjN6s3FzRYi0k1mW4gP8vh4geoSpD+6HtmsqYRs8J/FJ325N2174rhV/2axMB62f8l9QkkEAP7QfwSrbqSSK3YF03T+stNTsgsW6MTTajM3+gVFnw4wTtyLfMHh9ObeoYwZ3DAbrW3srOaWiiG4PmxJzAN6G1lSuHWzHtqA4AnIY72MsFtwDkj9EZmnOUGGix6M8MiWFk31JdT4s9YBEfv1rkHO5ulsCxjC51VVod/wr/k2tB12F7oJhB8X3O+2ouXyNIem45SNSrYAmuAninycwZ3rbLnHmJ53HlzYwiwp44SHGvjW6kCI/fTGv3eT77pDVU4b2O6nMbkpz/z/yYrJkFNrmM/bN+/s7Ns0NfpJoM/FJoinXf4qpEnjHwtZo066chIuuvwmwWBN91qgRZaV/nJxX6+rpZ5xMoTK4BSF/K3ePYh4ffMTvM1vTcM/e53cm8bqTqcAzZw77aCD8mJQYZz68+L7RqdAzMkXGz8d9hwrlW7L03MFxBGfbFGQy1rAYaZGWRxWoP3qOSG/Mtm+Y28Vw9gky5QLCSZevK0LkrO19o1QA1nmcwIvsviXdmLBSllaLIsSIFQ7Nt8rlltXNg05amASTx5J+5r5Xh3HtBxbDBxx+4KrDDsEmpPRYHQv6CdJnAcWnAO4q0y396N8NcyR+bye9DSMzSpvKFHTHcrL4UloU+5zztyV5OI/e3ko6O7i+X3UzAtFRhgfpdGINKzSmC07u9FWncf6Zi1b2a//JCn/NVTn5YHr7WdkVoTtcTh19I4996n3sljCiLlOiPCdndoj1uNRAB3mYxkkotzoVQYBySF0/uoJxGy85sYUM+lYw1IXHR9HDbUppxD/KQZHdIInQHwkvM53V5OXdxXZu5gyT3Vs1KSUiLxaMvkKb5kp5LHP7EDnq1zaOSMnbQNpmhfYuIlCGpvGV/BBhVBGNvjVVdlwYqlJyjC39GF3H/pjYIRXzDSLhirvzbFOekilJoHjDL2rRJvjPdISa/Zd5SXe1dvbMILxTa6achs/bHeQ+y5ECuOy/TbECIXVHNIvWKCNqX0NyTJU9Tc1X+ElQASWdEq9A8GQ9D4A+P2zRaHrlFs97Tww6JTLI1R/Be7rFI4AJsh4nLd9LnxW+mZilYw3ro0WQF0cD/QaYonl8IYUKubovIGzeKVRDqQOH03/Wdd0ZWZWGnWQf23Lj+YGBsNOxuFiztYEALoKtJA0Cs7S2FNerGeKUmLnCmOmAfNe6CuewsYCzG+9ltpJGe2OHhupwnN7YNgYP4brKy3nil+Y5KLe9LtWq4VJaDVYhjpzdB8gFl04DfFxHSGOXY+jvGgHLqvjMXOE8rgKjAVkubuHRxTNpJvtXKcVIZ0X0EGsQ70aBG3uRLAHNBcWMfiOOSErk6J7Y+4YvUsUHuL+lgr96oVzTyN44nvp63EXiLGpGEMkKiGV1+AeS54ar3ss8YhfCNwWIGOgLLYz/38q97iKCyvw X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 153a646d-dc6d-4355-0eef-08daf8a3b4b5 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Jan 2023 15:58:42.2562 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 3X8SedHJ7n5xxrnicnaYwgJ2WNQG1p8uTindOiVrKIsWgiUJ1a0u2dxgKvkEANuY X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8334 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: AB8B2100007 X-Rspam-User: X-Stat-Signature: 8tw7jmptzhd3cxtuhpyfwhwrptqpdwz6 X-HE-Tag: 1673971132-512409 X-HE-Meta: U2FsdGVkX1+qBW8WJU0YR5JMHhcMfoq1K4crEI+fV+oSfwduJ7rn4wmM/LHqj1qqBvyEnn3ggfKvEQ4U64ytUNDvWDLpkYm8WeFx47CjKydhYtehyyY1NvgvStbsuxXY2SH0mrwAgF8Yq2uICjA+i7KETJ8lrQ6hNen+pe9vouXczdTVFoJvCEemv1/S4/SB5rusRiuH/o06O9on6PH5DiDWWM4NggzFYouaWVVBGhPKs4UPIzu8ezyHl2q5L0jFoHuJrYWkz4i8q6UxMDLiSxvhNlz+F09hYLYK0BeSSD2MAUxvkRk8HjGB54s7evDlAXH5Y2o2aubmVJBbs7V70Fbs+A4IHMpF06b2obZlxvRRAccghGhrKkUSXKg2V+w8VfPoEL1d7MPAmiMUR71LNirNKxmA3+CSTshp/I+nrFQI2r4rNVXWTvl5ebBTcXyP7Ou3yXYJie5rrXgyK5N/OY+3OKPBfFvrbaRKzGB8p/3lOwvQFyvfto6Et9VGSZtPFXVQl5i1DZjP8tSvinJdWciiXWvc7tXcpN/o20i+kiJYLv3aupjGEKvCMVQLv3g7WhboUPcVBB0XRdlrwQ4Pp76RxDFT2IjFTEDC4tZH1tT8G9mBQ2D70jaY1akNwLFDe3NFb7V/FZwrDPLFLybP0HUm77XUuAiYA63Bw25rw0OOh9FQXA/c7C5FHaHQAQAR/2nO386ZpwbTbS+bhPlLtAa5J6hLB9vAdJsHRgz31E7DUVLJue+tFluEVFkFrzbu1lak+25pIPiqFGF5OYnlJeL1p7yqyvX9hkNZxALWV3SGSac3kSdBoGssi4Plu6S19jWb3Jy2UN3VCw+FboO0Kt0St7CX5U3F9uWLUifPDQ9aRerjM6eM8cwenSB+JR6NMa6wMVd+KXocQqwEfXKIjCi3K47PV5kaZFWk6rb6HaujJSpVgKRKm0UxkYLUSzanWNsXn/5J9rSmt8+bmr5 g4d/cMaq 8oN9cj1GQ7dxXfxhR+r0uDfrcMQcS+DAjudWE X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Setting FOLL_UNLOCK allows GUP to lock/unlock the mmap lock on its own. It is a more explicit replacement for locked != NULL. This clears the way for passing in locked = 1, without intending that the lock can be unlocked. Set the flag in all cases where it is used, eg locked is present in the external interface or locked is used internally with locked = 0. Signed-off-by: Jason Gunthorpe Reviewed-by: John Hubbard --- include/linux/mm.h | 1 + mm/gup.c | 31 +++++++++++++++++++------------ 2 files changed, 20 insertions(+), 12 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index f3f196e4d66d6f..7496a5c8acede1 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3089,6 +3089,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address, #define FOLL_FAST_ONLY 0x80000 /* gup_fast: prevent fall-back to slow gup */ #define FOLL_PCI_P2PDMA 0x100000 /* allow returning PCI P2PDMA pages */ #define FOLL_INTERRUPTIBLE 0x200000 /* allow interrupts from generic signals */ +#define FOLL_UNLOCK 0x400000 /* allow unlocking the mmap lock */ /* * FOLL_PIN and FOLL_LONGTERM may be used in various combinations with each diff --git a/mm/gup.c b/mm/gup.c index d203e268793b9c..4c360fb05cf3e4 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -896,7 +896,7 @@ static int faultin_page(struct vm_area_struct *vma, fault_flags |= FAULT_FLAG_WRITE; if (*flags & FOLL_REMOTE) fault_flags |= FAULT_FLAG_REMOTE; - if (locked) { + if (*flags & FOLL_UNLOCK) { fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; /* * FAULT_FLAG_INTERRUPTIBLE is opt-in. GUP callers must set @@ -1379,9 +1379,11 @@ static __always_inline long __get_user_pages_locked(struct mm_struct *mm, for (;;) { ret = __get_user_pages(mm, start, nr_pages, flags, pages, vmas, locked); - if (!locked) + if (!(flags & FOLL_UNLOCK)) { /* VM_FAULT_RETRY couldn't trigger, bypass */ - return ret; + pages_done = ret; + break; + } /* VM_FAULT_RETRY or VM_FAULT_COMPLETED cannot return errors */ if (!*locked) { @@ -2106,12 +2108,20 @@ static bool is_valid_gup_args(struct page **pages, struct vm_area_struct **vmas, * interfaces: * - FOLL_PIN/FOLL_TRIED/FOLL_FAST_ONLY is internal only * - FOLL_REMOTE is internal only and used on follow_page() + * - FOLL_UNLOCK is internal only and used if locked is !NULL */ - if (WARN_ON_ONCE(gup_flags & (FOLL_PIN | FOLL_TRIED | + if (WARN_ON_ONCE(gup_flags & (FOLL_PIN | FOLL_TRIED | FOLL_UNLOCK | FOLL_REMOTE | FOLL_FAST_ONLY))) return false; gup_flags |= to_set; + if (locked) { + /* At the external interface locked must be set */ + if (WARN_ON_ONCE(*locked != 1)) + return false; + + gup_flags |= FOLL_UNLOCK; + } /* FOLL_GET and FOLL_PIN are mutually exclusive. */ if (WARN_ON_ONCE((gup_flags & (FOLL_PIN | FOLL_GET)) == @@ -2126,10 +2136,6 @@ static bool is_valid_gup_args(struct page **pages, struct vm_area_struct **vmas, if (WARN_ON_ONCE((gup_flags & (FOLL_GET | FOLL_PIN)) && !pages)) return false; - /* At the external interface locked must be set */ - if (WARN_ON_ONCE(locked && *locked != 1)) - return false; - /* We want to allow the pgmap to be hot-unplugged at all times */ if (WARN_ON_ONCE((gup_flags & FOLL_LONGTERM) && (gup_flags & FOLL_PCI_P2PDMA))) @@ -2139,7 +2145,7 @@ static bool is_valid_gup_args(struct page **pages, struct vm_area_struct **vmas, * Can't use VMAs with locked, as locked allows GUP to unlock * which invalidates the vmas array */ - if (WARN_ON_ONCE(vmas && locked)) + if (WARN_ON_ONCE(vmas && (gup_flags & FOLL_UNLOCK))) return false; *gup_flags_p = gup_flags; @@ -2279,7 +2285,8 @@ long get_user_pages_unlocked(unsigned long start, unsigned long nr_pages, { int locked = 0; - if (!is_valid_gup_args(pages, NULL, NULL, &gup_flags, FOLL_TOUCH)) + if (!is_valid_gup_args(pages, NULL, NULL, &gup_flags, + FOLL_TOUCH | FOLL_UNLOCK)) return -EINVAL; return __get_user_pages_locked(current->mm, start, nr_pages, pages, @@ -2967,7 +2974,7 @@ static int internal_get_user_pages_fast(unsigned long start, pages += nr_pinned; ret = __gup_longterm_locked(current->mm, start, nr_pages - nr_pinned, pages, NULL, &locked, - gup_flags | FOLL_TOUCH); + gup_flags | FOLL_TOUCH | FOLL_UNLOCK); if (ret < 0) { /* * The caller has to unpin the pages we already pinned so @@ -3194,7 +3201,7 @@ long pin_user_pages_unlocked(unsigned long start, unsigned long nr_pages, int locked = 0; if (!is_valid_gup_args(pages, NULL, NULL, &gup_flags, - FOLL_PIN | FOLL_TOUCH)) + FOLL_PIN | FOLL_TOUCH | FOLL_UNLOCK)) return 0; return __gup_longterm_locked(current->mm, start, nr_pages, pages, NULL,