From patchwork Tue Jan 24 20:34:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 13114789 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E778BC54E94 for ; Tue, 24 Jan 2023 20:35:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CCF346B0087; Tue, 24 Jan 2023 15:34:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BB5A36B008A; Tue, 24 Jan 2023 15:34:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 795E06B0088; Tue, 24 Jan 2023 15:34:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 5CF926B0085 for ; Tue, 24 Jan 2023 15:34:57 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 463FD4037E for ; Tue, 24 Jan 2023 20:34:57 +0000 (UTC) X-FDA: 80390846634.19.93ADED7 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2056.outbound.protection.outlook.com [40.107.100.56]) by imf28.hostedemail.com (Postfix) with ESMTP id 96ED1C0003 for ; Tue, 24 Jan 2023 20:34:54 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=N3pjiIOR; spf=pass (imf28.hostedemail.com: domain of jgg@nvidia.com designates 40.107.100.56 as permitted sender) smtp.mailfrom=jgg@nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674592494; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TandkHoeiGygDdfgH7afNs02tnvU4eGp9W/Vp5Be4Fs=; b=njSrt7jhThKmCgBUUZozkDFBjIJRyLL4T1joNBs55v0LfQTZkwEiYsfAP5cKUObjNZV37J onLzRtxCdxG679olpxYUn/j6B9PN0F0qQouSiCNrrm1nJWhIzJS8hnzFQDdfElsWBH9GxZ FgwY0HTC0CYsG62PEeAK88NdWTM0F8M= ARC-Authentication-Results: i=2; imf28.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=N3pjiIOR; spf=pass (imf28.hostedemail.com: domain of jgg@nvidia.com designates 40.107.100.56 as permitted sender) smtp.mailfrom=jgg@nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1674592494; a=rsa-sha256; cv=pass; b=iAaxvDs0vIJqRG3DqkRiFtOfzhe7w/cd5H1Ypp/OVDRXOGF0hvEMgqWyxSKRxt5dK1/0h/ NkOaKGpqUtPIYoJvjm/2GPddU3GarQ6LitmvMtlPmknkoLiQS0tfqwuD1Rc7Zuh46PjRnq PpRJUb1f5brvfMYBFdnDlSEHZdEaxCM= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=n2uPHriNVRb7gWHxJGdRI/Q/SSD/ShvKHCdehDtJB9D6Ie9zX4BMYOdeXGPrgF7xkhnVM/i1t3/pV13SCsxVs0P1eTAZPg/uiM/RF2KHDGE5E3gzlxeVvKGurx4W+ax8hVt0cmYvmXE6UGaa9EdQ2GwEcw4dW6qLU28DRMIXG1vTKpALQlslBDrEtn+aZrPnP2M/rzyNJKPWymrMsb9mjuyldpgidtvYGMJP4Vszm/ZS3SkYMlFxt8wa9beTwIpWSCY3JFKvhv/Xaik07l0nnPxw0FTg9JzZlfYm0FsJJ14KdB8w/W7VxUhcpjzRennsqjREwxI3hIl1glw7oPvccg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TandkHoeiGygDdfgH7afNs02tnvU4eGp9W/Vp5Be4Fs=; b=dw6+CIuO3tzopDF+9RYFZTbCKlkHSgjrs3m44vB+aGkmorgDSbewBjlzJXi0DcRBqHGDvJC0ouLZblxi7LdwTyOxiVPwhvxNsi5zSetmE74SUflwHXVwP8rkYhfvKfRA5xORMtWJ+dYg9biCPSZECx4x13/Fv4Lf1lg1VxPjBDzpx1DtHOeAsiqB5wHev+BuQxh7zD9U90LM8/7Vk7NV6LXx1zMWBW/MMtwkoQNO7XghDoGWWQ+rJbycOHPSVlnTLDybg+lED66rR9N6QC4TolT/8iYL7DX+68yGZNQV7pi1sNApL9PF/41lJcu+at5yYFWsh0SJHyQwD4gzFfNFEw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TandkHoeiGygDdfgH7afNs02tnvU4eGp9W/Vp5Be4Fs=; b=N3pjiIORRdhbgOrcuJ5oyHIO80XJRKsuYlVrkymTR+tEmHgedh+eEVxEMu6OLWVonkVTU6O+SB5V9KfSjF5+6grF8DwHaVMhqv0SgcCyVilbeLCy3kEsosK6PMFABVopY8GFd9jJ1DoVOMni14wu9R/rgmyH71bCGbqJHVU/+PyjuKSVLQvnmv85Xcwk2veEaJ88spWkbGOHkDt76yAQsMKHdyVTNbQp2wKYKSDzMkEgM7bB3r+zgS0xDnKVROyDoA2AFUQ3ZuqcDsExKdGqN+KFEwkv38agJ8QjBaV6WUW9BUJ622fQ0PQ/zt/+oNEsPsfwGxY4k0/+c0aQNoN+xA== Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by MN2PR12MB4095.namprd12.prod.outlook.com (2603:10b6:208:1d1::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.33; Tue, 24 Jan 2023 20:34:40 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::3cb3:2fce:5c8f:82ee]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::3cb3:2fce:5c8f:82ee%4]) with mapi id 15.20.6002.033; Tue, 24 Jan 2023 20:34:40 +0000 From: Jason Gunthorpe To: Cc: Alistair Popple , David Hildenbrand , David Howells , Christoph Hellwig , John Hubbard , linux-mm@kvack.org, "Mike Rapoport (IBM)" Subject: [PATCH v2 12/13] mm/gup: move gup_must_unshare() to mm/internal.h Date: Tue, 24 Jan 2023 16:34:33 -0400 Message-Id: <12-v2-987e91b59705+36b-gup_tidy_jgg@nvidia.com> In-Reply-To: <0-v2-987e91b59705+36b-gup_tidy_jgg@nvidia.com> References: X-ClientProxiedBy: BL0PR02CA0030.namprd02.prod.outlook.com (2603:10b6:207:3c::43) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|MN2PR12MB4095:EE_ X-MS-Office365-Filtering-Correlation-Id: 21fb699d-2fe1-4651-a0f6-08dafe4a6927 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: xRtf3/eUAAXURlxh2txpZ9NElLurjm9RC/C7Lq2sKRkxppxVVBgBfYria+OZR2oPBxMe0HhsoBFiJNioo8kcjSkbwpBBY+bzK95bJl2LW+dEKON+zYwd/fbhRwK7zZOIfjI6P+VguaD1UT5Ygf8ju9SeBfVS5RagNnd8XbKIGNAV/SOKZxKuXrwuGmmk3Lhrls8jUulJ11WwqtI6dZwFpRzh/brUCjFk6VhICbUzSNr2h9/mBcm19Q7DjPsRaFP+zsrmSk1TsLhi0ZRjVJSm2lR9M/Y49xUBiR5ffileU6NHwDvMHqwLk6tiCnQM8P4FJrF4aWGEbkP/7CJwYgQFWXYL+uAoNrxND1DFV7LDiY6jOqkwjAXVYs8uGSDQeTZuCqzIyCaIjcZDs07O2r6mpyhocOKOn6LquP1eGxPE27VmBoPUdBTed65x2KCbdqT4K12xrR8Ha3VWkKCo461eAwCPVeokOcMfGBGUeT4h2gYq8mMbfkO58gkR2KRJXsH81ENNw3qxIh3ffNE4/QAGmukIrtWgfNHyWu1vgu76iMeMVlGdiPJe48vE1833NQTLgp4RjkbHU3rSHcFB1CqcjDcPCqwdRu8MnmLH19+wF/oI7g03NHBV9nfz/7mhUtaJ8Z+RgQ2axhkjdgvbi7ZYNo3yNYPCR0grP7Tb4nxXlqam+h51K6mOzg3CUYLWEAxD X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(366004)(396003)(39860400002)(376002)(346002)(136003)(451199018)(109986016)(38100700002)(83380400001)(5660300002)(41300700001)(86362001)(2906002)(26005)(4326008)(8936002)(316002)(6666004)(6512007)(6506007)(8676002)(186003)(66476007)(66556008)(54906003)(2616005)(478600001)(6486002)(36756003)(66946007)(266003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: BXHFa6QrwJESZK6eiSEN08Wmm+Gsu/Omc3lFSgW02n5mDHHHamMQZwqG/0MWWCozlXxgSU9jM6UfwSdoTsuuDejdbNBycK0SZeja+G9OtxkwHZ6pPLFQh+4NHjew1SO3vYEzyHkvUKOzCoTzAsqGsGFoUl9qMKVuAy4xtDCO/Ca12I+eh9J6unZS3G3e601sZ9wV19sGvyf4DWJaNspQAxHacJy2a7VEb4/CbMDEFWL9CmD791sinwXkUG3k87xBkQ2K4jlS8WdfO7Q88emMCQQ8x66hLCNINe6KMEbUyVqJ5Dt2QqKMFGphGqyIyTg+P8q9zuol5ro9zjSnR7jLEUmXBjP4zTBjxlZoAka/vdk8GVel4Dy1DRQSHo+LaLp3bldq9DXCQWcRjqhPv8Dbxrfc3qQJ0pEGpMF4Is+AicH5/YIfBsFkwsj7GFDt/W11PzeaLu27D8m0QjvM6y7M7mC4aGQznwJH1WR4O+qT9GH4qxDZZo69+xRQ9fWspPV+9Kv9P5Q0x8lrP3bS/hT4Yhynzpcuz58aDaDB9/CoevMKQBM1VbrUic/GUqJYl9cZSbCE5XFflOtaEkoakswOfeJHR40e03uYwKKrpvSQ4ANwRaQyIc+aUB7pPTk4N6x4pf0p81G+YoDgn1AjC8iozG8jKuMqm13opSQ2lPm79T/ywlFfymtZZZYcm8N5EsmkuS1RXECfK/ZuQGXGCHpaezFBbs+6CFantSRSN47O8it1jm7AgTy0Uom3mDWM/YJKXedTMvaCaqsuvqT/cebeI9T8/zrZXmU4nG5AG1Yke8tnMbaFuS7+KdqISTlGt3Ekq/1giXUmUSJosPMrTDg4eksUo17BX53lV0RvgJU4+ihHq/ENQjsT3DvHHZUVs48ERHm/biRgV2xKL1XuIOo/yTUj2Sz3gxiJx5boaRxQiCJ0XjCZ/4+fnjuwI4fodcoMHwht1uhBs3Dh6uBEf6dR6CbGGwNdw6VfVpDiGHzcEOfPoTlXS4WiB/CV1iU7Bbb4q60IKOiMiyIR34LDTHycyRAYMPmlt633MW2YsmbmJG/4o6/tGdYyTnXvquTXINGAj+5557rqeSzvaOzk04j//AX2y23Wm2BRPi5OoBm++VLboMDQp4q4I2xMLhJMT5v4jeiWTHPodTymde5h+hG2bRpMgTe8d3AsRKTYtnIa75IaFwRF9XVKstuEqeFChN3OzZjBlKshQyAchW1EyJTaQAeWrOcoIthIsuCax7QrcwmeBuyjTREBj0KwRX8qHaiStk1CITLVhXtlLmWYDCjwFgdi304J9NXVT3sHONnaLFSgerAYdiJAIAnUdT+a/Zvx1pkR17ok310AboosmZ0XKAFShvBjK9JbBC5WA5yQDsZ/ljUnVqxSfX12KQoCEynWAlMLAKoEP5sJtAKUDmX96fXSqy7CZqQgUxrXJTi884SPffCN51L9TG1Lm+Cs0ucexRzrTZIPFnjcYF8smLJ8QlOA5p3hnsU5hKVxu8Gf4OX5hOTGr9PQRajL4/iyJwRg4OFpsw3QnY6JG0U1c4GpHbXtK98cC3KovDdPke7TV5ghX3ziIx+wTZ1rHoh6Pn2d X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 21fb699d-2fe1-4651-a0f6-08dafe4a6927 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jan 2023 20:34:37.3148 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: vZefzCyIe8TGWmoHVjhk4q87wdzKuvSFhYdpUCnyXSYRwDYtI6PccZt7Oyk60P7b X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4095 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 96ED1C0003 X-Stat-Signature: ca97bnkw8ex56ps3o3kxn3981maoarai X-HE-Tag: 1674592494-797855 X-HE-Meta: U2FsdGVkX1+4gxfo0H2KaexYRtYirGHXzawQj6Sk0bD5F1ten00JEbR9cR8/NG3ezOGIRWA1/ao26SsT+QbyhR5CUWOEyCugIr0bIcEc2Xv6gq8fYLv8gcQSkuseRP3156PzRcelUOK1/hTyZY+BKbfDAtPqnr4HOWuYPnABBbnrEd9ONoFNU6PQbe+KSwsMAogk/sG8ZR6L1Cw/RcTSc+bILSD5AV7XWEQT4/KsLC08LX2CH9CHGDa+i5vEz/dhKHTIudPvtsBcfLQ7Om7x/RxS8Bd/5jrn8mFXg7c7aOXUUvhPhiuOVzFy8qBs0N0MbP0Sux0M6NP0F69ij3zywS/aHbf1nMlqUGVAv6qgGcBTWYQXbi0xOd0/PKj3Rr40R8PMhymFA0FfzEigZCJy/UqfNFztKg7Q+QKMcbHE4nMoVoNEPNCrgkekPHRZEMNxxYF0H/0VkVqnm8v1q+NuJ0OyMyijUYdRvpLXwrRxe6nW3sEtpSj5Dhc3edyXAtpC8VQDfx+Pn5x1YvLUeHds9aXTS+UPB2xO6TCsAXOafkq3O6KdI56KkEmQq3Q7Sq88yVg47Ljmr7bB60OM2NpwVaHowb5Szbwot1RY02TTQ0soESc3r+Qd+gEhZJ3U/rrnYNywc3wsDcV7X5KL6YHd1xrQCWGKdHKeJBUkCG3D87XiJ2dqjtZqaASer6aSOQV0dkEYvtdrjGGi1d6CIf8I+zxwK2XyD6kqQdjv9u95fvKXnl00fF/WbIte4Brxjh3TR9G5NmO4LXrD6bVhAcI4qAKMFUe3ACHfHq2C7+GuxV/lHCzZbB/0hBRZr73vMp6Wczvj94vZ2fOue8Wfmn3GeMT+rhVdeZFRs+39QMjmAME5QAyV+l6F/TF2UboylCZEOVSIjI1BjbSmJB6fDQlQAw3ju6JQSLzWnd4oHq117MjZwKMCeuMFvUv/XA0RuigNepCT6Pkyqew2SSUo8mV nMlkCzYo s7eQ/X9G2qKQG+TP5OT5za6Rf8qkuctM28WwuJ4zDz+fpTNa85unyffb2HxIfwXLZbKDV0BhCJ4vyrINM1DMwEB8zBJys+fdiOFuabtCwaEpVLdbbSbDNAykamT9u1tjScwrkkU4dpm2bE9butZz+NHh/fj+v4Mvhd24vWWWr6Kw2cIFqS7njdntNOPlcdsdGq7+suTiucQRxM2GXRNoYpJKnZSbnlAWhEpmh79hx9mjCZqyaVSIquVjCcrvkbA3+HYi8XZKw0QiUGu8I3cDXH2A1mYUu9j00DiMP X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This function is only used in gup.c and closely related. It touches FOLL_PIN so it must be moved before the next patch. Signed-off-by: Jason Gunthorpe Reviewed-by: John Hubbard Reviewed-by: David Hildenbrand --- include/linux/mm.h | 65 ---------------------------------------------- mm/internal.h | 65 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 65 insertions(+), 65 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a47a6e8a9c78be..e0bacf9f2c5ebe 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3087,71 +3087,6 @@ static inline int vm_fault_to_errno(vm_fault_t vm_fault, int foll_flags) return 0; } -/* - * Indicates for which pages that are write-protected in the page table, - * whether GUP has to trigger unsharing via FAULT_FLAG_UNSHARE such that the - * GUP pin will remain consistent with the pages mapped into the page tables - * of the MM. - * - * Temporary unmapping of PageAnonExclusive() pages or clearing of - * PageAnonExclusive() has to protect against concurrent GUP: - * * Ordinary GUP: Using the PT lock - * * GUP-fast and fork(): mm->write_protect_seq - * * GUP-fast and KSM or temporary unmapping (swap, migration): see - * page_try_share_anon_rmap() - * - * Must be called with the (sub)page that's actually referenced via the - * page table entry, which might not necessarily be the head page for a - * PTE-mapped THP. - * - * If the vma is NULL, we're coming from the GUP-fast path and might have - * to fallback to the slow path just to lookup the vma. - */ -static inline bool gup_must_unshare(struct vm_area_struct *vma, - unsigned int flags, struct page *page) -{ - /* - * FOLL_WRITE is implicitly handled correctly as the page table entry - * has to be writable -- and if it references (part of) an anonymous - * folio, that part is required to be marked exclusive. - */ - if ((flags & (FOLL_WRITE | FOLL_PIN)) != FOLL_PIN) - return false; - /* - * Note: PageAnon(page) is stable until the page is actually getting - * freed. - */ - if (!PageAnon(page)) { - /* - * We only care about R/O long-term pining: R/O short-term - * pinning does not have the semantics to observe successive - * changes through the process page tables. - */ - if (!(flags & FOLL_LONGTERM)) - return false; - - /* We really need the vma ... */ - if (!vma) - return true; - - /* - * ... because we only care about writable private ("COW") - * mappings where we have to break COW early. - */ - return is_cow_mapping(vma->vm_flags); - } - - /* Paired with a memory barrier in page_try_share_anon_rmap(). */ - if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) - smp_rmb(); - - /* - * Note that PageKsm() pages cannot be exclusive, and consequently, - * cannot get pinned. - */ - return !PageAnonExclusive(page); -} - /* * Indicates whether GUP can follow a PROT_NONE mapped page, or whether * a (NUMA hinting) fault is required. diff --git a/mm/internal.h b/mm/internal.h index 0f035bcaf133f5..5c1310b98db64d 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -854,6 +854,71 @@ int migrate_device_coherent_page(struct page *page); struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags); int __must_check try_grab_page(struct page *page, unsigned int flags); +/* + * Indicates for which pages that are write-protected in the page table, + * whether GUP has to trigger unsharing via FAULT_FLAG_UNSHARE such that the + * GUP pin will remain consistent with the pages mapped into the page tables + * of the MM. + * + * Temporary unmapping of PageAnonExclusive() pages or clearing of + * PageAnonExclusive() has to protect against concurrent GUP: + * * Ordinary GUP: Using the PT lock + * * GUP-fast and fork(): mm->write_protect_seq + * * GUP-fast and KSM or temporary unmapping (swap, migration): see + * page_try_share_anon_rmap() + * + * Must be called with the (sub)page that's actually referenced via the + * page table entry, which might not necessarily be the head page for a + * PTE-mapped THP. + * + * If the vma is NULL, we're coming from the GUP-fast path and might have + * to fallback to the slow path just to lookup the vma. + */ +static inline bool gup_must_unshare(struct vm_area_struct *vma, + unsigned int flags, struct page *page) +{ + /* + * FOLL_WRITE is implicitly handled correctly as the page table entry + * has to be writable -- and if it references (part of) an anonymous + * folio, that part is required to be marked exclusive. + */ + if ((flags & (FOLL_WRITE | FOLL_PIN)) != FOLL_PIN) + return false; + /* + * Note: PageAnon(page) is stable until the page is actually getting + * freed. + */ + if (!PageAnon(page)) { + /* + * We only care about R/O long-term pining: R/O short-term + * pinning does not have the semantics to observe successive + * changes through the process page tables. + */ + if (!(flags & FOLL_LONGTERM)) + return false; + + /* We really need the vma ... */ + if (!vma) + return true; + + /* + * ... because we only care about writable private ("COW") + * mappings where we have to break COW early. + */ + return is_cow_mapping(vma->vm_flags); + } + + /* Paired with a memory barrier in page_try_share_anon_rmap(). */ + if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) + smp_rmb(); + + /* + * Note that PageKsm() pages cannot be exclusive, and consequently, + * cannot get pinned. + */ + return !PageAnonExclusive(page); +} + extern bool mirrored_kernelcore; static inline bool vma_soft_dirty_enabled(struct vm_area_struct *vma)