From patchwork Thu Jun 27 00:54:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 13713600 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F116C30659 for ; Thu, 27 Jun 2024 00:55:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C601F6B009F; Wed, 26 Jun 2024 20:55:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C10406B00A0; Wed, 26 Jun 2024 20:55:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A8A556B00A1; Wed, 26 Jun 2024 20:55:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 842CB6B009F for ; Wed, 26 Jun 2024 20:55:29 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 361A1A1CFF for ; Thu, 27 Jun 2024 00:55:29 +0000 (UTC) X-FDA: 82274850378.08.FBFDDFC Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2081.outbound.protection.outlook.com [40.107.220.81]) by imf20.hostedemail.com (Postfix) with ESMTP id 583A51C0007 for ; Thu, 27 Jun 2024 00:55:26 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=lUrLIacI; spf=pass (imf20.hostedemail.com: domain of apopple@nvidia.com designates 40.107.220.81 as permitted sender) smtp.mailfrom=apopple@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719449718; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LwTuajmqH/Dz9iq8vwzSLICD/nvk7k2MSnhgTVOPjdk=; b=68DtcjbbgtWZYGM8jemIt6ZxbHkl3RpjQ/OjKFU9K5DlcQu4QWm2VjZ2D2ACok5wZFtalY kbLF9Od686N0mhWUw7eEQg1TZI4Cpqd42ZVdYkMmOT/ZmXqR/7JXFsfRTMy65rUoJeEhJ+ Pgg6BeWokEBIqDMD7fit+VKu1U2zl9A= ARC-Authentication-Results: i=2; imf20.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=lUrLIacI; spf=pass (imf20.hostedemail.com: domain of apopple@nvidia.com designates 40.107.220.81 as permitted sender) smtp.mailfrom=apopple@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1719449718; a=rsa-sha256; cv=pass; b=7Lo+EP/ODLF4Re3Ei0dS+Q5+vSR/k5UzstmQnCV8/dPVRkHLl6FE/1Ipf0Q3CiuEPMgIkM uKQJcm9bsC5TBRrg+vICGvPGnbZch7ZIaj1hm5/AbRlOKwwVXqzYauQqqK/35TC9/Xkjls E7EMDz48aDPAzWMM7LJhSX5gg6RY6OY= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PRQ9UMZVvQJlGA1JyILEsfYbqGIwEwcdKAXQOgmUNiWA7P5AQt9oPzwXy5HujfzfKUarlk6coiYV6cLGmxLazadSEJvpmn+pqf4QJEt/8/pXTfg68uCcISI/yL5E1YsF3vaybffvplTJy8EkA4JJ+CO8fkS7AsToXc9Omzyh3N2qRvY0Yi6wZZH/Lsc2rjUHuDL/9Nkaj5doFnP1AG1wWP9RxejgaG3vhkkVgKt1NmJHAQvTOozBRD3IJusfadKlHe3PtjQ6pniG+jKVOgXOMhWzAZlRFUCqUT+UuLFqf0R2Tincz34zt5iKMKaoBF0FogEiG3tZnWuA7fYlDse6rw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LwTuajmqH/Dz9iq8vwzSLICD/nvk7k2MSnhgTVOPjdk=; b=SotSGtBw5bAhGNtC1Jzioabd3HF5PsxXFPJbKEL9RoX+3N0+VsGo7IL8Qe7r2aWxAp3ObpOBxcxIL0iPbCLTDP4CDRR2k5jSmAgX8xjxFb4uzAUp9ntweB/7NhtLO/ozD5oGBBZfY7OiX51JXc0tI624Kx1jVgpWxd4S9mYPG+ep2QS1FdGSQ65tHdRlPFLhiDSKH9ajsxxBku29Kjr4H16fJQjX14676H6p68anhXCpufLQyI/tMmZ1sfijJa43BfAF98kwqCLJTDNskUs5oNOioroeBFIpAMGZ6CjDdS73R4N/0ZqMC+9n0FxJW88TNDgGHipvP8uvyVHMtdi0Fw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LwTuajmqH/Dz9iq8vwzSLICD/nvk7k2MSnhgTVOPjdk=; b=lUrLIacIPI/ywe/gFyfRXydzEzVgb5vC+zFqv2bWT+0XpKyMlLqNiNEv8jyYDHVVFpF4V/N/MtcjeNh9Q/J17rsPMvSZCXsYMYLr0MwSv+XAk1xHRiU5WYC9vk8oYm9j1717bX6nCSKxYYKchAHIGzNGMljvpVEnnHTf9TZFQj88jGCTTOapOle7GsFWHTg32kyjJMdx3j7Fji3QfePoiU5dxwU9JPS4oEwgB4Ja2uy+pUUDae1moUe70tvba9NR/QGZ90Klvv/TIeWw05lYZkOVR2zR6dw99KPgl4ZjNmZAFzMY6IMzhfPHOiJ3Di2h4JCOjhxun4qKotOwseAp3A== Received: from DS0PR12MB7726.namprd12.prod.outlook.com (2603:10b6:8:130::6) by MW6PR12MB7071.namprd12.prod.outlook.com (2603:10b6:303:238::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7698.32; Thu, 27 Jun 2024 00:55:04 +0000 Received: from DS0PR12MB7726.namprd12.prod.outlook.com ([fe80::953f:2f80:90c5:67fe]) by DS0PR12MB7726.namprd12.prod.outlook.com ([fe80::953f:2f80:90c5:67fe%6]) with mapi id 15.20.7698.025; Thu, 27 Jun 2024 00:55:04 +0000 From: Alistair Popple To: dan.j.williams@intel.com, vishal.l.verma@intel.com, dave.jiang@intel.com, logang@deltatee.com, bhelgaas@google.com, jack@suse.cz, jgg@ziepe.ca Cc: catalin.marinas@arm.com, will@kernel.org, mpe@ellerman.id.au, npiggin@gmail.com, dave.hansen@linux.intel.com, ira.weiny@intel.com, willy@infradead.org, djwong@kernel.org, tytso@mit.edu, linmiaohe@huawei.com, david@redhat.com, peterx@redhat.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, nvdimm@lists.linux.dev, linux-cxl@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, jhubbard@nvidia.com, hch@lst.de, david@fromorbit.com, Alistair Popple Subject: [PATCH 06/13] mm/memory: Add dax_insert_pfn Date: Thu, 27 Jun 2024 10:54:21 +1000 Message-ID: <50013c1ee52b5bb1213571bff66780568455f54c.1719386613.git-series.apopple@nvidia.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: X-ClientProxiedBy: SY5P282CA0090.AUSP282.PROD.OUTLOOK.COM (2603:10c6:10:201::17) To DS0PR12MB7726.namprd12.prod.outlook.com (2603:10b6:8:130::6) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR12MB7726:EE_|MW6PR12MB7071:EE_ X-MS-Office365-Filtering-Correlation-Id: f829664a-b39b-46d3-4f65-08dc9643c831 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|7416014|1800799024; X-Microsoft-Antispam-Message-Info: lYXynZNeE0+XENjHJM0WN4qEdMZjjd6nLh0DgkAMDNpJSxN8LmPNHzTyQ0qYPIm9wx2aC06AlJYDXpYQ4qX4Ry0J+4AkQovojUMHpvdWESfdk48hRdEHNks6iI95PGkEHYkI4+/Q3nhyuao5pv1NwqOAUfHZSY+Q0XI2r1iDzRtS7CviPVwgjuWQVT0Syb05lkpi4jOhoUJkuauY4xmeRqeNpojheVzruW3NLmBo34szDisfF25nzo+s5j9o+H3g/NC8gb0coC/txPJ0x3MlbAx8xtvqLUQQP/J4q4NIy7AYUrXUiDZ1AL5M5kxRApYMvFVTbyz+c4O/SLF2U3FeU8K2CurPPMqJi5PU9mvfkluoBLzVrCSJrPJCwOEzT7yr/e5P9kp48EfKYe9cxsWZomvZZXmAwgvgkTHZNShcLgF9MPeaqR4L4qpRnS+nkYD9OaJrwjv7PWH+1E9V5RxjKYrOdFa4xwKAJscu4t0xyQQdNVSYynvm+2TKum24alwQewLK8xACj29y1ublCfTYBZ3NabhofNPs8021qR3+Lc4qFD6uVpDuzO+pMWvaytkSlJ64PutcyQhxP/Yiqz6JmbLbYKGAXpFTTjIano+35mundFPP/M9AMgNJaotCYWgJJQa3rLv0EgzL+ip7OEz3+KVBCHFOK+6TMkcwcEUYi5BQSNS9XslcXGTeVShovqfZ64bXogVe48bJUFd1Za/sKldN+QrcmtX3HjfoKZivSqZ0CeW3FRBImdy4c+z6NjGV/CXBqkn8DzMiwyVd/z3KKFG4K9dCFJv8uWt6/NXvECp3YZG5n8rzOTfT4zxrSzlFViiAcAnGFZ56QeuKrnK/bm4f1LURHPFbmlR9v2+wMMNT51gRmGw2JG/R53rUwMQdCuhwHofJx6pgsu+w07CznWWz4uG2+11TIDhIeAAx462U9cvXFcO8PErK3R+MedmhDxqw8ZUfMkVXCO8HGU1Q4IDFUeO7OolWYx5asTi0eIyvuHEbUwp2tQ4XMfxnF/mAn3B3mBs3y+GGVV3ba+RWR3Pe1lcxibb34ibnkphVNp/x2Drvu0nhFszz7Ksx4EFA7N6+4ksSsuN/edZwu0WfJaoGVibWUyWwEMUNeItsSShjKOzf1szjyrvgFFh71fBgN/kFCbFUNYVr3Aknfo/XQdfrxUX9bosPaXLY78ltdVm/YeoeOeZ6Jj2CheB7ci4lwVqRul42B5kyhBAQgQXulOccVUWqm0b1rXjNG2Pc4HqQAbtF0JScDW7Rl/6xCIZpJxHHxuAkfiw6mbuvwqd9+sjr3ywY6sy5116nb8W71evlxZo8qa3vNS3br4WTpvEX8gmYzdf97+ZvOYxAHwfZGw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS0PR12MB7726.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(7416014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: c49ryrsnGaEKkZ0paxxpf8ffnzOPK3NhW6sv9Z9P/3NjUpF8jPcWaewk2EMCbJKkmI4Lo9aTM5kPWJllBB0yaYEVZ7eXRHq7CfBSqfdYqoKk8EjAAurW0fpr3mzs48tY+Do7b4td9ae5z24KRN2EE6L9Z5sUwGG0keGYsCEXk3rdx5FP25lBPSAmJNqs5foW+atTN8PuQeSfYjZAXoaIGZLjWEmNWjkjSG7NTmNd4249hvPZKm4P1IuxG/3ENWj29PAY7gYqVX/jaM/EIZJz59f2K1ECnbHhluuCc6XkBTOeXbTYqo3L9GKH6iJIWMNxiqU3boQKigjJxQ3Tt/Ln84k3g550uhSwcYV7YQ0o4arjqnT+s1xiErJPM9ub71r176CPAy1mzfpH1I9UVzQCAM6TcWdV2scrw3GrpvWYGX2yNzk3zjA3Ofc6DF5E5vmpoz5FlTIvS5bhawMW9vOUkLm8i2bJoia6I20UPKcYRLRDsBXemmIkHvYXiJD1HmjVEfgiNAc5SWmR4ZxhXNIvBfTmvzkyAkOVLoGXclj1g6iu6KF17YtbhuSB0Zb4Q+87O1jSMTwrOs8Yk0Q3qW/EdaqPK9XjymTlxNIHc6HK5L+X6ouPfdCR1jJpykJ+r7q0XWvYE5E9hV4dqncixJMwDiz8HwJ9BLhsnR54vBHfaNgGAxElfQuI/+7MxvuJMnp4BntCGq0yBvC2h6pgy3jWpI9I0GWVnTbTE2Z/7PylfP70TLu5NfpyvL0gFRLeFiCxeRBQEPmDEOHgCgGBq9Zf2p90YHiwvpPi4HZeiu2w/y3tGbM5ew40WyoJsiDnavdb9VBlPaVX4P1ZE5cMhAgSfQcq5wd7q2pvKMrI3cGLvIhCmI0pkPrXEdZySeHV5eEEM72GK+J/bozOrj7dXJYvZqLHlxU7NYW646DFfviZ7I4RM8UBaBuRagksg7SDOwNy+RgfBDpp7esWbsSDu7y1W7aKiW1nVKSgDay9S820p6FORmaF9WEUnauRZilVOyl/iVkiK3nIxohSPGlMD2IM5a4d7yWl10sLQ2yXkGKUid7HySJXAidXf9ErQsuyryMPZWXRgdEm6ObzOudLTTvz93A8R2DnPFFY4nSLz6gTTIQ7WvlUZniqngsrbouQNZNvBM1ALPFj9Piv1DP/n5yelLFqd7o5+hx2zCDBBIDnCNPI7yPfPC8lZDuVfZkQE5Dfk1yOkSbw+lJwf96BDu9VuQjeP6cRL42Wx8TvESanzzVtj1W0YwAHDHnMm5tP1XS5raDJyYSY9zw6VuANTsYI9bcg7Xbbsuig8PKMDi90F6vSCz9pszOqX+u1rdirc8feRelHbHn6bQ1poL/RpZ1qizRoom0xo9Pwlt7NpoMX+DopxM35juBU38KNIcSvkeblPpwsjVakYztjLrR2Vg7bmmU10qG6xsWgTln6xnr/sSaj0rdNiBjn3X5Ej/AqDlowSfHzv65BOkIC5p58du503QpbkUH4UJH7EikznB9Iw1qSQTAxVVcwnrrMOsh98Orp7f0remHuNP30BOEz56U776vf8quyUGzvdO7tTDnoyLxIni5Nsv8IPTrS9z5GOWPm X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: f829664a-b39b-46d3-4f65-08dc9643c831 X-MS-Exchange-CrossTenant-AuthSource: DS0PR12MB7726.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jun 2024 00:55:04.8153 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: VP+IB+RrKOqGr43kzXBieunOvUrssGRIP9nq84UcCbyyH4uUIkcIvvcuTsr07JP1Uuf/DAVI/7xcJTTj2DABTA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW6PR12MB7071 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 583A51C0007 X-Stat-Signature: goyq8rmnzu41f5o7w4djkdsp3yx6dn4b X-HE-Tag: 1719449726-190210 X-HE-Meta: U2FsdGVkX199ObV8cl0SACJO42djffHtVK5zxIgzz1H+sEP5m2ORp8ND658CMoGTEEQ+ZySs+xi8U8/yNOqF29zgu0nPg1NF6GVY6grQi5OJYe6b+hqTHdt9cj+/Krd24AAfLT9Jnurh97O6mAkzrrynTEuy/i82vIkczHVOehyHpliE39oSC/jbGTsGX7Ub4hddkwrBlf9r92XvCjkYOoH/AVWn4mdSGEzo3GwxFPNIc3CUu8C44bmVjZon3WUQL/ZZXLx5omysj7pM4V/3SQWudk3Fb2TSkv7FcZZkPrVg42d6sBntHycAFcdUJ6OmrsCcshAnkWp1SIu5okE9fXMWM9yaIcYs+ALUEkQBkma+oOjv/xU+cb94ZRlNP4eNOTg8f7L+X+cuk9D8esEp72FKKnK1Jf1Zs6V/nNmgatZvs/zVUmzOpLiYBucjc6WaSWnujQndejB4UTAFHj/Yr4oEr9q7LfqRqTQ+nUuPCcJIxiLHLoEj5JT1Ff7sR65Xb7qaZoOvDidRZU2/mR9hs0Pn9Xb54uQfjXJiUrW7PZAlXkqCT245CfGqfMVMUdpYavi6KQvKI906aNNHDkmbzko8r30eLbg7RK66ecYFIAiLpFFdkBVRlmSDKeYEhdjmVd4nmLn2r8qvf+L02+dQEr3e6x6s9VnKBefccaffsW4+uoIkLkWuTP4n65fwtBCLUq6/ytadhIn2iH41wI8GXGKorRlQhm98w08QbsUXIfF5EAwfQKD/g6iqqJ3Mo4afPIKp6c4oEqoDIg9GiYVHEzu460e462wBtaEUJSfTP4ajSVWvqolTaUdkzJL4S9qYJ4fJIIKvzXzO5rVsw/MLmk0Ysx5ktOgLNqmy2oC5TXEvYa90cGDdx2Hj82TdKfyFz5kaRG9GCaVY1G9fdgUTTi1ckK7SFd4syvhFh2AcvFulJ7qk9J4WbE/n1qbSUoMNZhjgkt8fhVI8yUnfMdJ e2z+SjqA Zk6ON0n8etrIzA99QvHjsx8CAPxrDz0ntRa87M2Es27+FSywLZsw7m0C6VuWNR2cDhOwLeuf9dybBjbZYmxvvnjOa+9hrr6umTedV8Zj1L/Utu0Cus07TQk2zYTh01f8tN9HUqSbL9lsPYLhbVHRowxTDgq0v1KnOwES2k0sFGiiJEksl3cDUO03FR8jFJLtrxTyzRgWpjtElf0m+U/obIrMSxJbV6vio+Gv7vhvLzopMLhRWQbfbXKcCZArs7SSm2dyZmWz/XmhQBMuebuK9fZW2tBrD/6bYMJCHcd3CVoUR9vt9+OqtYrOmFHZ43qFbXvh5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently to map a DAX page the DAX driver calls vmf_insert_pfn. This creates a special devmap PTE entry for the pfn but does not take a reference on the underlying struct page for the mapping. This is because DAX page refcounts are treated specially, as indicated by the presence of a devmap entry. To allow DAX page refcounts to be managed the same as normal page refcounts introduce dax_insert_pfn. This will take a reference on the underlying page much the same as vmf_insert_page, except it also permits upgrading an existing mapping to be writable if requested/possible. Signed-off-by: Alistair Popple --- include/linux/mm.h | 4 ++- mm/memory.c | 79 ++++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 76 insertions(+), 7 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 9a5652c..b84368b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1080,6 +1080,8 @@ int vma_is_stack_for_current(struct vm_area_struct *vma); struct mmu_gather; struct inode; +extern void prep_compound_page(struct page *page, unsigned int order); + /* * compound_order() can be called without holding a reference, which means * that niceties like page_folio() don't work. These callers should be @@ -3624,6 +3626,8 @@ int vm_map_pages(struct vm_area_struct *vma, struct page **pages, unsigned long num); int vm_map_pages_zero(struct vm_area_struct *vma, struct page **pages, unsigned long num); +vm_fault_t dax_insert_pfn(struct vm_area_struct *vma, + unsigned long addr, pfn_t pfn, bool write); vm_fault_t vmf_insert_pfn(struct vm_area_struct *vma, unsigned long addr, unsigned long pfn); vm_fault_t vmf_insert_pfn_prot(struct vm_area_struct *vma, unsigned long addr, diff --git a/mm/memory.c b/mm/memory.c index ce48a05..4f26a1f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1989,14 +1989,42 @@ static int validate_page_before_insert(struct page *page) } static int insert_page_into_pte_locked(struct vm_area_struct *vma, pte_t *pte, - unsigned long addr, struct page *page, pgprot_t prot) + unsigned long addr, struct page *page, pgprot_t prot, bool mkwrite) { struct folio *folio = page_folio(page); + pte_t entry = ptep_get(pte); - if (!pte_none(ptep_get(pte))) + if (!pte_none(entry)) { + if (mkwrite) { + /* + * For read faults on private mappings the PFN passed + * in may not match the PFN we have mapped if the + * mapped PFN is a writeable COW page. In the mkwrite + * case we are creating a writable PTE for a shared + * mapping and we expect the PFNs to match. If they + * don't match, we are likely racing with block + * allocation and mapping invalidation so just skip the + * update. + */ + if (pte_pfn(entry) != page_to_pfn(page)) { + WARN_ON_ONCE(!is_zero_pfn(pte_pfn(entry))); + return -EFAULT; + } + entry = maybe_mkwrite(entry, vma); + entry = pte_mkyoung(entry); + if (ptep_set_access_flags(vma, addr, pte, entry, 1)) + update_mmu_cache(vma, addr, pte); + return 0; + } return -EBUSY; + } + /* Ok, finally just insert the thing.. */ folio_get(folio); + if (mkwrite) + entry = maybe_mkwrite(mk_pte(page, prot), vma); + else + entry = mk_pte(page, prot); inc_mm_counter(vma->vm_mm, mm_counter_file(folio)); folio_add_file_rmap_pte(folio, page, vma); set_pte_at(vma->vm_mm, addr, pte, mk_pte(page, prot)); @@ -2011,7 +2039,7 @@ static int insert_page_into_pte_locked(struct vm_area_struct *vma, pte_t *pte, * pages reserved for the old functions anyway. */ static int insert_page(struct vm_area_struct *vma, unsigned long addr, - struct page *page, pgprot_t prot) + struct page *page, pgprot_t prot, bool mkwrite) { int retval; pte_t *pte; @@ -2024,7 +2052,7 @@ static int insert_page(struct vm_area_struct *vma, unsigned long addr, pte = get_locked_pte(vma->vm_mm, addr, &ptl); if (!pte) goto out; - retval = insert_page_into_pte_locked(vma, pte, addr, page, prot); + retval = insert_page_into_pte_locked(vma, pte, addr, page, prot, mkwrite); pte_unmap_unlock(pte, ptl); out: return retval; @@ -2040,7 +2068,7 @@ static int insert_page_in_batch_locked(struct vm_area_struct *vma, pte_t *pte, err = validate_page_before_insert(page); if (err) return err; - return insert_page_into_pte_locked(vma, pte, addr, page, prot); + return insert_page_into_pte_locked(vma, pte, addr, page, prot, false); } /* insert_pages() amortizes the cost of spinlock operations @@ -2177,7 +2205,7 @@ int vm_insert_page(struct vm_area_struct *vma, unsigned long addr, BUG_ON(vma->vm_flags & VM_PFNMAP); vm_flags_set(vma, VM_MIXEDMAP); } - return insert_page(vma, addr, page, vma->vm_page_prot); + return insert_page(vma, addr, page, vma->vm_page_prot, false); } EXPORT_SYMBOL(vm_insert_page); @@ -2451,7 +2479,7 @@ static vm_fault_t __vm_insert_mixed(struct vm_area_struct *vma, * result in pfn_t_has_page() == false. */ page = pfn_to_page(pfn_t_to_pfn(pfn)); - err = insert_page(vma, addr, page, pgprot); + err = insert_page(vma, addr, page, pgprot, mkwrite); } else { return insert_pfn(vma, addr, pfn, pgprot, mkwrite); } @@ -2464,6 +2492,43 @@ static vm_fault_t __vm_insert_mixed(struct vm_area_struct *vma, return VM_FAULT_NOPAGE; } +vm_fault_t dax_insert_pfn(struct vm_area_struct *vma, + unsigned long addr, pfn_t pfn_t, bool write) +{ + pgprot_t pgprot = vma->vm_page_prot; + unsigned long pfn = pfn_t_to_pfn(pfn_t); + struct page *page = pfn_to_page(pfn); + int err; + + if (addr < vma->vm_start || addr >= vma->vm_end) + return VM_FAULT_SIGBUS; + + track_pfn_insert(vma, &pgprot, pfn_t); + + if (!pfn_modify_allowed(pfn, pgprot)) + return VM_FAULT_SIGBUS; + + /* + * We refcount the page normally so make sure pfn_valid is true. + */ + if (!pfn_t_valid(pfn_t)) + return VM_FAULT_SIGBUS; + + WARN_ON_ONCE(pfn_t_devmap(pfn_t)); + + if (WARN_ON(is_zero_pfn(pfn) && write)) + return VM_FAULT_SIGBUS; + + err = insert_page(vma, addr, page, pgprot, write); + if (err == -ENOMEM) + return VM_FAULT_OOM; + if (err < 0 && err != -EBUSY) + return VM_FAULT_SIGBUS; + + return VM_FAULT_NOPAGE; +} +EXPORT_SYMBOL_GPL(dax_insert_pfn); + vm_fault_t vmf_insert_mixed(struct vm_area_struct *vma, unsigned long addr, pfn_t pfn) {