From patchwork Tue Feb 18 03:55:32 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 13978908 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA5A6C02198 for ; Tue, 18 Feb 2025 03:57:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 42D642800CD; Mon, 17 Feb 2025 22:57:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3E1122800BF; Mon, 17 Feb 2025 22:57:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1E3972800CD; Mon, 17 Feb 2025 22:57:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id F26C62800BF for ; Mon, 17 Feb 2025 22:57:20 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B62E9817D2 for ; Tue, 18 Feb 2025 03:57:20 +0000 (UTC) X-FDA: 83131705440.20.76445CB Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2064.outbound.protection.outlook.com [40.107.220.64]) by imf30.hostedemail.com (Postfix) with ESMTP id C8FCA8000C for ; Tue, 18 Feb 2025 03:57:17 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=s+J0Si8h; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf30.hostedemail.com: domain of apopple@nvidia.com designates 40.107.220.64 as permitted sender) smtp.mailfrom=apopple@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1739851037; a=rsa-sha256; cv=pass; b=xBaBK4k0wtoG/ag0Ji/TbJwOXKNuGSeh1v7LL3I1bhpK/crlj6cBZ8RnTMEtocbN83JbiS JRZhUvw7gNtMxPyZDNRJlDXv8RejxGD3HZwz19ou/dmBA0MaUJP0YIcbF2nV44cLQdZqBi tk7P4MDxB4iPyg6cWfD4q6phNtyVCDM= ARC-Authentication-Results: i=2; imf30.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=s+J0Si8h; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf30.hostedemail.com: domain of apopple@nvidia.com designates 40.107.220.64 as permitted sender) smtp.mailfrom=apopple@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739851037; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BPkfSG+iXeKAyAjVqa1PJ1E1IonWgQrZ7MBYoYvRJTg=; b=Whgni77kDxIVuswEbE85BRQO/4B8prkxoxT+DUk0IGOleduBPnQYb5VnysiO3Lg/QmCGhy aikXRsI9TBPansRsWtW7gNbgccWSAZMXYwN7mA2r7enB9l1QULz5GOor0bpLPKhFgmUngB K/g/VXVWBxnubV8z5NcD3c1ZCaGhEoo= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=NfjMgE2tIi3xt1+1A5/gW9d37z8ipmfyCKh0V633/F4wQwzAvdVMmX37Yj3JmGJWXpUdo/kHhtIzDL7JO17V3LPtatYAhKqYfVJzXojPzRg/qNCeva1tqXvaDtPFDIXqi53RXJAMuTG0SZhaGHNuXGpLlU+T76WOeSaNIkfBkgxroqFyy7kIgn0qI2Ybssv+gyFUM6YiE4y1Pk3YJRna7+Wz+p0vwN25XgQ1tekYzPpeELHFWqsd/jbmU5Hes4mQ/EbqAlDDkZcNz4sv5jSwR/nCRgJE9YBvJKTW1IQ2Nxn1fGCOXl8BLHrqfV0ubMiD/AZHaFOkXSO/7KfLxaIRIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BPkfSG+iXeKAyAjVqa1PJ1E1IonWgQrZ7MBYoYvRJTg=; b=xOx+N4QI/Ah34rgng+rP9EZL1KNwTVRuuCUT/Ni3DICYo5b/46ewc/1vOMJ7toEU0VuMraC01GI78ntEoxpNlTdT3s6KPwujgQfbORkHtZ+Y1eeG+vQnKK8d2/OPq66WjPzxK2lstzWUNjaC6qfkrzzacDnmtumjrgGl+zL77Y2Hq724ZNtMUExxNTKHvsOoUZHGgTQwZyqk1UrFjQloP8On+cZv0wuBAU7nwMMzl8R0OHe8MBk5OzCXMDh3HeCAQegrPwPQkaYekNN0ozxon+lI27PXie04Qjy8lxbn2oc7mBEBS8umJNmqo6IyJU7gsQepFTnSX45lyRcwulo34A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BPkfSG+iXeKAyAjVqa1PJ1E1IonWgQrZ7MBYoYvRJTg=; b=s+J0Si8hNs+qazbX4Y2OQ0odNmFefpUABaFFEvI0NgmsdMqIWgPSJuLXGAsmiFVu3cCcLBU22zqdx3y4X9pggQ4H49UAfNYI+x1o9KzCEWfzy4dzxblazM0AdV/DC4oOtUYI4etC6NZqOIiFFKwJNX2EXbZNhcM3K7CEl0ovSecfOWx45QnuTFSzUS6/ht3upiELOPjpbYxkU069xUNUOeOis8BQcCU5+UWirase7leePKQokX/TPPMvXDWQYsClEg2hUkVLx3rD5wMgRdH5T2UNHHOAUKINvQXnqRvEKLbrZ2MKHkJb47DA1vj63yuMcH1X3iEbgfRvz40JEFlA5g== Received: from DS0PR12MB7726.namprd12.prod.outlook.com (2603:10b6:8:130::6) by CH3PR12MB7593.namprd12.prod.outlook.com (2603:10b6:610:141::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.19; Tue, 18 Feb 2025 03:57:15 +0000 Received: from DS0PR12MB7726.namprd12.prod.outlook.com ([fe80::953f:2f80:90c5:67fe]) by DS0PR12MB7726.namprd12.prod.outlook.com ([fe80::953f:2f80:90c5:67fe%7]) with mapi id 15.20.8445.017; Tue, 18 Feb 2025 03:57:15 +0000 From: Alistair Popple To: akpm@linux-foundation.org, dan.j.williams@intel.com, linux-mm@kvack.org Cc: Alistair Popple , Alison Schofield , lina@asahilina.net, zhang.lyra@gmail.com, gerald.schaefer@linux.ibm.com, vishal.l.verma@intel.com, dave.jiang@intel.com, logang@deltatee.com, bhelgaas@google.com, jack@suse.cz, jgg@ziepe.ca, catalin.marinas@arm.com, will@kernel.org, mpe@ellerman.id.au, npiggin@gmail.com, dave.hansen@linux.intel.com, ira.weiny@intel.com, willy@infradead.org, djwong@kernel.org, tytso@mit.edu, linmiaohe@huawei.com, david@redhat.com, peterx@redhat.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, nvdimm@lists.linux.dev, linux-cxl@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, jhubbard@nvidia.com, hch@lst.de, david@fromorbit.com, chenhuacai@kernel.org, kernel@xen0n.name, loongarch@lists.linux.dev Subject: [PATCH v8 16/20] mm/huge_memory: Add vmf_insert_folio_pmd() Date: Tue, 18 Feb 2025 14:55:32 +1100 Message-ID: <16b232f09b2e678ef5c21eb74f21794a06422ce4.1739850794.git-series.apopple@nvidia.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: X-ClientProxiedBy: SY6PR01CA0068.ausprd01.prod.outlook.com (2603:10c6:10:ea::19) To DS0PR12MB7726.namprd12.prod.outlook.com (2603:10b6:8:130::6) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR12MB7726:EE_|CH3PR12MB7593:EE_ X-MS-Office365-Filtering-Correlation-Id: 24c37faa-2354-478d-2056-08dd4fd054af X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: lBEMAxJCJ497ZANIjP+a2HbX2IEb6BbN9dgdBjcT0PPvpI9NENvgCKx/mXSO9bbBz28PSM6ADYOc8ea6pK8LIriubf/nHolk6Ngn8kDSg+/+JkpUc2+B++QRKi0TTPxiuL/iz4pW5K3kf/rZtVQ7ZiJRPJt/s1mnln3yDJA+bsPeI1ZjI2c+Ekb5RKP2Wq0YKi6dHMuIO9WT5sx+vXAtJzY2ZMburHLqT8fprkjyTlz8IvKghTfO/P72doHuwDybc58bmJA/94Zzb0fmpC5nsCE3Zqnx//l6dYoqm2QUroIG0unAIAm6EhCTsjQp3Wx3wAzGIc6ys6nYowpeoMg5sJ7flGExCD5+z2wpnqctzbVX1H2BSI+CEA9KmRWKaLlcHAl7cu8Y9vzxymzZqbsqNSiIs/rRcGjJD6Kk10Ar7KHIbXgQKDQ9F0QdcwSdYf8mu/tLAhBFb3gViBBHHbItmh1izRaXLzeNpJYBlRjnm8QZJrrBw2R0TWE0WV84wZIvH4CWSbXNMq4EmVVkf31rdG15aiKzowE07AupHcfUOqxSHLKDWTxPxcXnpK/KYWbbTg3+AUxx3uJLOYfWjhZ6pXS/tr3jbP/co5hgkryAryXz38ScdAnKr97EG2xZRPSLBtQoVMm/XVZjxTpZq10YgFKJeVtJJG4a33n0Wgz6sVOF3Va56nUcIJXPBpUeUbK14xulg4ZCa8QJ2iJ0m5Q+VmuFC7znL2j29eVMBtgxyLvSgOmbrWTVwqZwACBa51q8PE3OEGDF6SIBY5d1XF6N30O6DD07dVFDYLPCf9uC4REpXhsPQDkt2vSApQAomeQEcY1UHjpQ7tn+VIBRNc7CbUi3zxyfl3YtXViBSqnQflT91fDVWN7ALwRbBFg/xDvr5mc4eZNL8pFwcSydbQhgVGUnX+hbyTD+V6+yH08Dcok6DbBWRBgxsDZc4Jyx4reEZqzmayrnqkEMuJkvcttnn6bqUmFHDVHB3rsskVTc7O0AOzELjVO4wSBPGIvkYNsPM0qRc7UJGWiybWvUev2WdxJT85LuSBkdoAWy5E3LynyPR7r6FB9kY0F0pzoCl7TrzzsPaOfL4ckmFCWDXL+PcVU51mGcYyuxvd7K57TWfeC11lK6fJoU9KkHS0ZbPU6qJ8LwPsDRA9qG+TQliLheA/sTdj+aeB2BlNSMQHXs/qLh0zOTy56rN0tOFkjaFvT+B4Lvn+ti5X5BwQ+K42AQo6TW3kZHJdXshjdxChSnTQ27QyFSu45qQPjawvRB9Bt7ETG0FVYSxIDC7Kzg2Q5BiPtLHXHOurwGM6KJGRmhgew04tiQgyj0qvvWdxFRqKmAf1B+6omdIoDMASIJ0oakDNMsZZzIA/yHFhXjHDavm1IJhoS+7Lh9eSUHx9HtX1Ci X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DS0PR12MB7726.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: pY6qgrMIi3O2e2bkDzvGBDdgZThi3ZYUIWDqXIoW8ErVgcUaf3zPIN4vqvV6qrwtDz1ANvovOFLm8C76GwOESNG6VvMMvukH+RPd4vwkZDHbHjqGY5pIAZpCIzh6K3JWfR/kCsT0lX6IrAJudtOqBpGh7DrZ65upmPnX5TJnosuMeloeK9QOYW3HD35UwIhV6EDVzO9HqB5N+1mm2QTVzrLgGhvG+Flb4z0yBHb/b58TNLyeNAvUHzcokhyZIrKwFnJhwZTwKmKiX7wvbhEKGo2esuaBU+yoYlrc13hX4V9B+1KYsXbN140GmfhLIL5iTRD34X0VgDyd4C0JHEtlMrqUJDKhUenzDMOmHSWAjexbdnIb5Novzr5j6yJp68dYrkjZqtojDbAw78WkXI9kZS2SYLV+cm4v5E68+7z69XUnyuPPXGlP6rVn6k1qXiqTasBatR+h/MYn9A0QS3yBK2zjqIflfD79/TvqqBuPQKIrrsuyngcd2PlHvclzZlHbpI+YoXlOfdhpTTVjNKZsjvp9SCWDfrU5AlY1jy835WUSJUUI/wmdjLalPIP+qcVG3U/9x+5ROZ8Hcgjk8n2iDE3nu/114A+oYwHSkrTj7fTX0tngD0no7ANXBuIu0GKFZ9gYvuZ+oUnEpC7NZa61HiXCJkB3V2e35mHAHpzR8uu9fngZAgBwvIE7jgK7y+tYZBoxHW7VRA6HBPtvSSxoqx8ei5AgsU/8iUa3mccYsNgKQLvL9Y5EodDnnYVrsF+8ibocxWz17dfITk++YrubuSBQG7TDvt7WlbV5dAd6Umz/rbu6nA8QJw6qlbP4c4wxQBHRmytcfkkZ3dSAppMR+BtVDZ1TA4zAciO0bYbPj7FWTNDWYoytIJcZDSoP16d6khF5regEgIeZsi2xnIK8ZjJH2InwJRcU+8nFOOBQtt5nu7Ptmjyz55nAtw+ta5UUT7CxX3hjUmp7iwHULv9+cOT4PC67BSfcU6ueH9tQwvCmCpDKpe06soevWHwU7MSxefyxsj6rPwGeezM2HHHTWnxdG0kSLmQTrlaBW61FTbQ+iAqJ21nirjbTNHSUA2m/MKb54/qJ2WNiPtg9Fuj9Skhm36B1uVMS/kiBlTPSk840PEKx9aiRZZoYoTId8mAR49nnriLFzA56qFUqhKL7ZMHuJOPs2tezfcrOHYGKiNJR7w6zWyAihvKyViN3tOSDLd5eMry4nXlerItnbTdaYsFd4oGE4yEn/fCAhFhKs39lR58DAQ3KAD7uwp+4I0G4nYv07kr4+WU/4SJxiqQddyqageDdnAjevlyCEu6W17s222KDFLt4cx6nhZmXrNtdflh1639RWhE03mL6dMSqkMEQjRXJ/g4P29YfIcX52mbo1EB/HpMxYAZGk/ISFAGx/mW0N0MTvIsvDgVS5ZU3gVmlZXBvh1tN7UoFhQUUy1SJVy6LBSfsnFp+yax7P1i6bBwFV1aphnTxlFtO8OqaJP3ypS2o/DUDXKb84Oo/EqNvZ8RL1R+jlkkIf3eZ8yDMzKXbRE4DHMOx1USVhjaACCX55srSo/g6I0YRKPncbqGSDYxgTUQPMHT8O4A9cmmF X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 24c37faa-2354-478d-2056-08dd4fd054af X-MS-Exchange-CrossTenant-AuthSource: DS0PR12MB7726.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Feb 2025 03:57:15.2316 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 9tCb2iUkgYhRW08XmmxhM3SUU0/9GmLe9S/rscn3kAbzKGsZz6mZoldA5fdZKbEMWHtwIaVX22hrXBbTEzPynA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB7593 X-Rspamd-Queue-Id: C8FCA8000C X-Stat-Signature: 1qhrtokgyotc7x31bymx8b6qwf88zsk4 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1739851037-211148 X-HE-Meta: U2FsdGVkX1967c3mWxbf07lEC4wOOcy+tKlcY4fVE4W8VXZgnBbSZZ81h1lFyrZENehquIx3PApsbLLRuemFBLsJHrqd41a9veLvhFLsg4gldcGTi7NLe5mSmxTQlvS/ydfCLo1PpRutiGnCXx3457OuWLBocR4natncQERcFop4OriTEKzoOYEo2/+7dSn2wxaNqrULzO6SoaWUufr0Gzp7ND+SJN2dkIxzoQnSDurMSu9BJ160Cwc/r5U4Y74nKIyHeNr0U2nZ0hgwet4bMTeV+ViWUtTDN0oUzxHi7jlyhwzV64Wd+quEFkV6WTD9tvcRKKfEx883UbkNWw8cKneBrCJcL9LMwLC9RCcNrerEoUYFAStRD5SCfMxELxq/DFdqisIEtyV3SIFopoaBt04V1lXTd2lti/JqXUYOUZ8/rHa3VaumWdZtdbGE97dxMYtXUWt6Z49QVHZ3IDhAEawGSiVNdLnKC20HJlk9MhBZurcTX7kor2b0Hnt539V6egvtPZuwR6HjIygX64OTlR/b8lCZ59lb4KmRPj3PgIkMIwRUH2CzTMy5naCuCE1kRYF92Pvrcsm46F9+F9TNBWXBddnAH+E7CNEvz8e1BPldypSCuvOn14ObqTpsuFTKSVqBGxhV2Idmvo6q4eoARlvm2PdIoqyMEqef3nHtUYpLjCJXkzWnPznZyw18HI1fQ4BAE5jkiN7Hoaq/ZxRSfWUhTzPoOS+1XUXTkyzJM7hm2g1sy903WCgiQ/yPXMzwIeuJC1DIOZ3rHnNGMTfcG/tSoL3v4Ja0JQxnz5H0/zc5zhXTJzzgTpQKF/8Lre7Txvi11JZoFKtPI7tLwNS79G4Uo+JBoYLIWkMOGB29PUSj7xkZzCvcpV2oYjVOtv7xADGB7WuClwd3if0libd2L7Dwk6MkDhwOeSrHGnxU8aR/Ls4FUwqcnaSUIjGie0I9WayzV5NxQYYztmtZFxM 8qBCpIEP 1QvoiYvtP4EVt9+KuvLW+Av3lGA8/ffcaMI9CKrGQRD5KKJGr5xWYMasqKvbgG2uPd4P2y5WJpbvr8OfmI9QY2Hz+S/8LrNUt8RBsKlm2SGYptM1QfsG/WJMt7wIeq6CCBp9lekaZV4+J7ydg7OFGGTBI4m9MseA3PIciLcQbNvA7TqiR3cj7dvwQXrNcneIRQoz8kLOpcoJFDZevji2O/SemY+WB8OsfYz+j/srIEAl7ZdOaJZM7k2KaJrZIUJ0J1h/KZgHkYZ9qAbghDgrxu+NWO/RJOGzI+V8d1zPyAsbqZSX9ixx1z2BCeiPXVI+/taw+7lwD0JjrXYPmb1W9SoN6dL1/HswX0Agqi25orKt7V4hqQhhk825UUuse14gmCPNFj57IN0XUnSkwl/oHw27NsyAxaOEfVczOAyTqYrp8m6cZRQGARofAwsSnnIxXxHaIF38tU2j/2NqBhpxwsmyw5waF4B7Ei+QIf9mW8jv/xMM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently DAX folio/page reference counts are managed differently to normal pages. To allow these to be managed the same as normal pages introduce vmf_insert_folio_pmd. This will map the entire PMD-sized folio and take references as it would for a normally mapped page. This is distinct from the current mechanism, vmf_insert_pfn_pmd, which simply inserts a special devmap PMD entry into the page table without holding a reference to the page for the mapping. It is not currently useful to implement a more generic vmf_insert_folio() which selects the correct behaviour based on folio_order(). This is because PTE faults require only a subpage of the folio to be PTE mapped rather than the entire folio. It would be possible to add this context somewhere but callers already need to handle PTE faults and PMD faults separately so a more generic function is not useful. Signed-off-by: Alistair Popple Acked-by: David Hildenbrand --- Changes for v8: - Cleanup useless and confusing pgtable assignment. - Fix line lengths Changes for v7: - Fix bad pgtable handling for PPC64 (Thanks Dan and Dave) - Add lockdep_assert() to document locking requirements for insert_pfn_pmd() Changes for v5: - Minor code cleanup suggested by David --- include/linux/huge_mm.h | 2 +- mm/huge_memory.c | 65 ++++++++++++++++++++++++++++++++++-------- 2 files changed, 55 insertions(+), 12 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index b60e2d4..e893d54 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -39,6 +39,8 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, pfn_t pfn, bool write); vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, pfn_t pfn, bool write); +vm_fault_t vmf_insert_folio_pmd(struct vm_fault *vmf, struct folio *folio, + bool write); vm_fault_t vmf_insert_folio_pud(struct vm_fault *vmf, struct folio *folio, bool write); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 1da6047..d189826 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1375,20 +1375,20 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf) return __do_huge_pmd_anonymous_page(vmf); } -static void insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr, +static int insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, pfn_t pfn, pgprot_t prot, bool write, pgtable_t pgtable) { struct mm_struct *mm = vma->vm_mm; pmd_t entry; - spinlock_t *ptl; - ptl = pmd_lock(mm, pmd); + lockdep_assert_held(pmd_lockptr(mm, pmd)); + if (!pmd_none(*pmd)) { if (write) { if (pmd_pfn(*pmd) != pfn_t_to_pfn(pfn)) { WARN_ON_ONCE(!is_huge_zero_pmd(*pmd)); - goto out_unlock; + return -EEXIST; } entry = pmd_mkyoung(*pmd); entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); @@ -1396,7 +1396,7 @@ static void insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr, update_mmu_cache_pmd(vma, addr, pmd); } - goto out_unlock; + return -EEXIST; } entry = pmd_mkhuge(pfn_t_pmd(pfn, prot)); @@ -1412,16 +1412,11 @@ static void insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr, if (pgtable) { pgtable_trans_huge_deposit(mm, pmd, pgtable); mm_inc_nr_ptes(mm); - pgtable = NULL; } set_pmd_at(mm, addr, pmd, entry); update_mmu_cache_pmd(vma, addr, pmd); - -out_unlock: - spin_unlock(ptl); - if (pgtable) - pte_free(mm, pgtable); + return 0; } /** @@ -1440,6 +1435,8 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, pfn_t pfn, bool write) struct vm_area_struct *vma = vmf->vma; pgprot_t pgprot = vma->vm_page_prot; pgtable_t pgtable = NULL; + spinlock_t *ptl; + int error; /* * If we had pmd_special, we could avoid all these restrictions, @@ -1462,12 +1459,56 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, pfn_t pfn, bool write) } track_pfn_insert(vma, &pgprot, pfn); + ptl = pmd_lock(vma->vm_mm, vmf->pmd); + error = insert_pfn_pmd(vma, addr, vmf->pmd, pfn, pgprot, write, + pgtable); + spin_unlock(ptl); + if (error && pgtable) + pte_free(vma->vm_mm, pgtable); - insert_pfn_pmd(vma, addr, vmf->pmd, pfn, pgprot, write, pgtable); return VM_FAULT_NOPAGE; } EXPORT_SYMBOL_GPL(vmf_insert_pfn_pmd); +vm_fault_t vmf_insert_folio_pmd(struct vm_fault *vmf, struct folio *folio, + bool write) +{ + struct vm_area_struct *vma = vmf->vma; + unsigned long addr = vmf->address & PMD_MASK; + struct mm_struct *mm = vma->vm_mm; + spinlock_t *ptl; + pgtable_t pgtable = NULL; + int error; + + if (addr < vma->vm_start || addr >= vma->vm_end) + return VM_FAULT_SIGBUS; + + if (WARN_ON_ONCE(folio_order(folio) != PMD_ORDER)) + return VM_FAULT_SIGBUS; + + if (arch_needs_pgtable_deposit()) { + pgtable = pte_alloc_one(vma->vm_mm); + if (!pgtable) + return VM_FAULT_OOM; + } + + ptl = pmd_lock(mm, vmf->pmd); + if (pmd_none(*vmf->pmd)) { + folio_get(folio); + folio_add_file_rmap_pmd(folio, &folio->page, vma); + add_mm_counter(mm, mm_counter_file(folio), HPAGE_PMD_NR); + } + error = insert_pfn_pmd(vma, addr, vmf->pmd, + pfn_to_pfn_t(folio_pfn(folio)), vma->vm_page_prot, + write, pgtable); + spin_unlock(ptl); + if (error && pgtable) + pte_free(mm, pgtable); + + return VM_FAULT_NOPAGE; +} +EXPORT_SYMBOL_GPL(vmf_insert_folio_pmd); + #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD static pud_t maybe_pud_mkwrite(pud_t pud, struct vm_area_struct *vma) {