From patchwork Thu Mar 6 04:42:33 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Balbir Singh X-Patchwork-Id: 14003828 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2FDE6C282D1 for ; Thu, 6 Mar 2025 04:46:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 074846B00A0; Wed, 5 Mar 2025 23:46:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F3BEB6B00A1; Wed, 5 Mar 2025 23:46:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D912C6B00A2; Wed, 5 Mar 2025 23:46:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B96476B00A0 for ; Wed, 5 Mar 2025 23:46:35 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 24008121047 for ; Thu, 6 Mar 2025 04:46:36 +0000 (UTC) X-FDA: 83189890392.10.8DD5963 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2065.outbound.protection.outlook.com [40.107.243.65]) by imf21.hostedemail.com (Postfix) with ESMTP id 678F61C0003 for ; Thu, 6 Mar 2025 04:46:33 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=IjayQsof; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf21.hostedemail.com: domain of balbirs@nvidia.com designates 40.107.243.65 as permitted sender) smtp.mailfrom=balbirs@nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1741236393; a=rsa-sha256; cv=pass; b=IKOufKcfDKXlLS3e6aNuJEv/CofAwZUAMlrMCKIWwx8qD5JtNz4uJjFIU8D8eUe2qP/aMg hgkYV0FFhaqquD1/u6bglLmLMCQ1QHTmmJ7YtePWSqCmmja83ZhL84lDwbY9PA0uQzFSgX QfrsGJ49f3W4eb9p4d+Gm8EMO5YaXsU= ARC-Authentication-Results: i=2; imf21.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=IjayQsof; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf21.hostedemail.com: domain of balbirs@nvidia.com designates 40.107.243.65 as permitted sender) smtp.mailfrom=balbirs@nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741236393; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=r83BGccQ3T5HvdU2HP8phbbJ2DBl3Y42ADRff8m64ss=; b=opEg4VRWXoHwYHM8GHyXpegvLitz+oHr8bfjptY6VzlAfz7GhY0kpLW14cym5XDzh7echP mC1m2eOKNJog9LsGN9sDtwFD1J+N5hT91OXCIrfrUENmr9om2CjRSBoFCudfYuiGgilBQX +9CkxY6mbD/BrJiTK5Dh4NwnFa1Uol0= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=G9v9xXv9FMgfzJyp1HOBc2IUh1VpExzhBC71My7n+rsvliA3CWRwGUlMMkUiHV0ohQs5gh6d62Hb+Rdit+X/JIXR/RXA9HB7zV+t62JbZ/5NPR0rJZXloprIoA+NPyhgdEKGrVVD149ZMP/gHX82ap7BnrhTDDzC40GdoEueedSdYWoMFZAeDFsZY3K6QEm906FAZs0O7BzU++/DJPGSLjVQNQ2DD68Z3Q6o2LTqWBOxgLKtdbjT+GkNaChQu29j3F+96mDIcxiBsbPE0qNI6UndAjlEWV8tF6F5SuGKp4JWHuYwRtAJBApWT75g0Jjpk8WiASsDmYkLi0cqiG9zjQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=r83BGccQ3T5HvdU2HP8phbbJ2DBl3Y42ADRff8m64ss=; b=iMYs4VRHnpWRlS3js0B2/2NzcfY60Wjxie/HelrG/Ho/BUqb3gCrInPkSx2kLcaCnGim7bYcmMM4wxOhywyZs1gdVaeOh4C5mfRnJWGZ7yogq7N1Q0BTmNvmt+VqWoe+ws1IKjiimuwdvqlv2ZJqLPoU2hksEC9eDBZEmNEoK2ErRsqbGVBATtoYdwtFB5Vs/Fhno8dR3vSlSKDAKRyV/Rlu7Cz0xsPCrHW3bOnanhg488KFgm1xZ++P5dJtqxOBdBN1wovnUsnCQspg6NP4nUm3FeY2A8XxW/YPHuXtG5OGL8sAxQKPUffHaxyr3cKcGkp2SXBxVr0R1QXEvUfOQw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=r83BGccQ3T5HvdU2HP8phbbJ2DBl3Y42ADRff8m64ss=; b=IjayQsofRcyApBAQ1ObrtJ2ZZzKql9OQnQqAxRo2RGFM+Tsxb9q1G8YWYYzyo2lepr4HulmWnetduRAUmv/6YiVapgqZJMkQSfKOREHlxkU28kjOTUBQp4kuyBxDxLSBKRY3dptvw0Fol9vSzXEnu7CgnQaAKTa9qZIDU2B36AcYdIJcAunjdVJONlJyVLzpAEwM4eIP3nYJKeQZMb1eC+VMItn8mnjcqZ8Qy4QpzoIDgJz4ZVOzl/7nlexoH8Fomavt4xuxXNXfnK926HI21I8mfkoqbwsMs/wfXXoFfiGqT/pI/+8tL+aXNhHgQ+fMhprUZnC3N+DVfLZHDFQSBA== Received: from SA1PR12MB7272.namprd12.prod.outlook.com (2603:10b6:806:2b6::7) by DS0PR12MB7534.namprd12.prod.outlook.com (2603:10b6:8:139::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8489.25; Thu, 6 Mar 2025 04:43:50 +0000 Received: from SA1PR12MB7272.namprd12.prod.outlook.com ([fe80::a970:b87e:819a:1868]) by SA1PR12MB7272.namprd12.prod.outlook.com ([fe80::a970:b87e:819a:1868%7]) with mapi id 15.20.8489.028; Thu, 6 Mar 2025 04:43:50 +0000 From: Balbir Singh To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: dri-devel@lists.freedesktop.org, nouveau@lists.freedesktop.org, Balbir Singh , Karol Herbst , Lyude Paul , Danilo Krummrich , David Airlie , Simona Vetter , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Shuah Khan , David Hildenbrand , Barry Song , Baolin Wang , Ryan Roberts , Matthew Wilcox , Peter Xu , Zi Yan , Kefeng Wang , Jane Chu , Alistair Popple , Donet Tom Subject: [RFC 05/11] mm/memory/fault: Add support for zone device THP fault handling Date: Thu, 6 Mar 2025 15:42:33 +1100 Message-ID: <20250306044239.3874247-6-balbirs@nvidia.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250306044239.3874247-1-balbirs@nvidia.com> References: <20250306044239.3874247-1-balbirs@nvidia.com> X-ClientProxiedBy: SJ0PR03CA0273.namprd03.prod.outlook.com (2603:10b6:a03:39e::8) To SA1PR12MB7272.namprd12.prod.outlook.com (2603:10b6:806:2b6::7) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA1PR12MB7272:EE_|DS0PR12MB7534:EE_ X-MS-Office365-Filtering-Correlation-Id: a4549798-3d90-4651-a655-08dd5c697d94 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|366016|1800799024; X-Microsoft-Antispam-Message-Info: S3BI7T+LhfE63h2snhKOREZIHhldMBWT+G+fkjG6LErXeCpomJmB1Nx9vDoo6ulIRjo9k8bGIP+RQk/X42pyFCIlODXfiC85m43RUbEQb3KNUlBsnjF9VUCpVukvuayl/+OkMzkuxTCjGhP8+RHBEz86AhdqCBlVB3p25PGGp+ILEmX9do2XG0cp4StL18B6SMaYYidoEVnqeGAWibs9aCRb932RNP77sUes0xaUQ+dvUZBz/tdTBQEtZMhet7bJT4F/jMnA42advXLxCvf41D7ze26IYPl555rgGIxQt3t3A8PoxQerbI2MX7JUDT3UH/LohwwacOQoB9hn3zClw5thex8QRXbmd53nq+e71XeiwEFUVGIza4MtkVp3EHv0A+z/XVJWfeV6zAnIzN0SGqkQ542DwNX/aWyR9CegJrFYHE/egk94TklyXmlmdvzJ93pODmaYCtp//BEB4ojvQEf6ZWX/SnWR3uuOAzUHnMruuNvbwJjcwRiz9MpeGbq5AKYGulNPRPWHXB1CATNxVP2BVHpHvz+c8qh/Pdk5ejUaVEQAGPZbdjeCiUZi1r9NSu58aCtBxDJgZfHZfrxxRPLm1qd7RuoRHyrk6f4UcKjO9L5AndVjZWKZO0Xhw2TxHHNICfbMdngLXRWYrNDEhk5nIFVafZZOtKvkAlhoxN+P24mZspE7Er3FpVbwSVBj9ZfW+WJfCHtbWICixXLOCKWrM4kc+wy9cKofz5F/HXmG+ZOlSn7FFUOdwN48G2ZjnCzZmiNWNB9m+arjyxHdtFpb6tyblUKgBGhrUKCXpRxRycJljyfr3uF7XfTzPkmwATnnpleaiPIdgnbht4lrj62kL+wMyRwCrH78MFF7FPxhngLEV+P6Y0ZB8uWVYFZjYgsCtBzXUShcDI1xftr6fNIXFRHFmXUBBkeJ4UJCZLc0P/8bUXVeaRQSXgCa5OtGSiZk5+P7jnOunc+j6GYiM515KQCu1gja3QaoaXlT0XCsyBda/yQawTOx/+s/ZN5SB0amAes0pCRZw8hXy5kWSs/BT2xOD8s88fcnfzakljdcbNn2U3IF17Be99DPGFhFGvUR/F00+fLpgy3/jdv6BumAgxvr3e/4krLmLHruG9mmXu2r96X/hvgH3nXVu+dAyrA36Z75b6KFdpy2Q6wpydTtVGdSkkLE2LFuAqdCb8ceNkLBGdmRnn1nmnG8yqp3Ofwk3Aj5lAmR0NDD9hPiYfvtfwDBFwWMeHYz06UNL7Xmg0KPQH9kShzry1Gb4znWKPX+H5V3r9PxdtZp3ci6buUiazn9vpBUA0vy54eaGAqKAoE3If02KrkL52Lxq2lilMFy2wM2SRKaS7NzRlIHtow8k1VP7g4VtgvJjQdxOjr3K5xyWYIad2YtxN6GJFpK X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SA1PR12MB7272.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(7416014)(376014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: nbfIbJjpTzxjcxNvXlG/yxnZa7bTTepjwysH7PRBns1d47Dlebx4rZ/NbFXFlttzpNIrfMt+osx5AUEd3uz+jhGWAscnWtzJ5J7wvR3YY42nX5KE3I3ft0lJIdVxWB2IsogXoFb1OmMUfBtC7WnW3r9sKtpD3x26VOFgXVRQNGyvsN6AKD3cMRSALCxR6P3EgsUvulmbnIw8PXAhNj5frI5UB14H+GHTPnxMb8uVyYHaSEKmZ8CfNAd2mE3EFLVs59HuQT9nKpAcQ6FkFaA4+guJKAHrXSHgta5pUC3cNwtq1AGLzE5VO2NmKpT1zbi7/XgzveBIgnj+vBeCflAj06pyi6umMdof0QSPKrmTCfg3SqC/ed2sA3L7TCuApoOIdnOwazt/1CrkuW/HCcwuP+1BUZbkcfNKZIsCdu21h0A4hJ9ef3sfTWawhAMyjN4jFe5NMHQPss1KOeEsxaxYq90pI++KvWF7Hw4rCbiMP6Sd4aqIlbLQQ2lqh0u9rhCY5fw47PRVJV8eZLWzDiKNjeqF37Icvr/Ka/8IqDJQdGdOOW/UJ/tLBISeYIch2gVmQmR3RSDxGkOeq3evgRHmg0SjgvTnAPfT/R4pWltS/uqkpGZH4mtUXbeZp2DpdtogB8tct2GGUB6s0Z6tTzZKhSDnQVqzsJv3Ps/GIQdQ3G6OpbHFMl83fmzZkleciT71aUoOPLgCgylonX6ERBNv0eAMpvzN1nrHmHF/TVy0V0tyCwjrytqu0Tr8UCkx0HkqwdkfmllF5pifE6N3ujofhI169EiaQs6TfdCyFqMPiOi3AvFT/Iar50Va6cu70fIrJjwHItu4EszDJWzl1HdFOWOQEqs/wxuMhm4KVCBqITGogdfh3x8FM6jw1D8ESFi4re3PjPPwWerM9PX29sEAXSmtLOI2ETzXDH0ngUa49u5yDPrpAg+Yf2Tk4Rpo3Lpw0lRoVB3YqU+Mjjn6chay2Pt8O/yHpLFN6sOLwSRg1OL1h5cMi9gVt8Shm3Sci39VwbKB/dikrmS9fsvDNpRXACDloCrcylYcz0MZzZ0y76YwBJSohPaE9rEtE+DCB/rtb1vpk21NZvdbBFRg1Dgx5kcVYN2fT4wUWDPUUKrGUtzokk4rUfd3h4Ho0BRGRiz+TTdmRvYd/qDu3mzkf0IeDHOqYUvCwJlh9NdqjBAJuIAE8KIisYf0FUOOuwS2BqTME3qZaLp8T2FBk4ZPwPXx/5+WJyoj/i7MQCYR6Zej3f+VE9qj7jASaRfJW+/9A/LUmyywAu7JhExzqiqNO1GTx1YSZ26k9zbBCMdOyI5crxyrMqboZlhWUkoVmXRhJLn7rJC5GH68CkYT3HdD1GRHxovpesueffRztJdKekwO4+/pGKb5bf72JcmUCgdNrnk+y/otPcOvWG/DIMKGA6tC3KoJMD0SsJBa/Lo9RpkWnsBMUaJjmjyLDVsvy71ofT+r5v/m7agKHFxj0B7/dnI/qrFUnhnWDEo+g7b9nQNUGTmZyF5xK9ugQ0aA+Sf7sZWNiEBdoRnR8u5wNmKOfwHOReqH8x0O2tOr5yW2uVOnP3mzDzyiyFjoIBrw+48df2Iz X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: a4549798-3d90-4651-a655-08dd5c697d94 X-MS-Exchange-CrossTenant-AuthSource: SA1PR12MB7272.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Mar 2025 04:43:50.7071 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: X94RF78/N4jm1vGr6AFg8veX3pmjAwOo/GcpthyhFpRzmRWMVC2gB1gQkZbYLVIYqgRBKqh76PbsLhacn0yUqg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB7534 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 678F61C0003 X-Stat-Signature: 9gitqd34g3sewprzx1udqzay3gsuj5no X-Rspam-User: X-HE-Tag: 1741236393-284685 X-HE-Meta: U2FsdGVkX19+zPBGFglA2zLYiQUVw8QEUnOuI/EbkDDRjQI06seIMmJ/lPjCoxPkKYFraEoP/qkJ+icgCIalfeDGF2JfU9GuY4PhBPcF66ch4IigelwoEzuA//gV3DT3V5PucF1idyhRvtiM0AsQ8zCm1U8Zv2BLztZr38W/BGsdskXmfpjToRCb6tWG1CICFdYVnjqtAyUERVLfWRLtANpf/8f++BKRdYhVq9qWkEOOHzZWJPtLmwo/3ZMpChb0PHkvmOLGdAA7CNZ00/vXEpaM5/ZlqPrqplAwlmUpeek1BOaSKjJ0MeegO0wTE8w2UX8wxh6v+wejtyADklR3V2FQnkSq9OCHWGeU3Hz6X0VsF6xmMt9Jn+3BwMLr+ko7G65TLbvOPIK5kzChGuuaG1wi8boH7Zk5wRyO6QEuECYkDDizjBVKMyaiQ/PVZKu1iGjEVR5Xh+5kwe9ceJMLaXrX4/jZ6wOcRvlVyLih6t6tV0HLgl8aUi3rqCscB37WNNLJ4tnmXuO169SWFkc9QvIZVK4EbfVH7hCFD80Fea3BDaIkA6OJww2o7+5l/MKcmJhXEU9Oy2ayvqOlKoDchYmrUm6t9ZljQlyf+38biSocjJ+7UZTP/vHvK3KQwFdKYxrxiLh5HU+D1VyS5cdcfOlnH4d6DtMJumTNTQlPaUumH/QAgT0F4lVOwpomci9uWsrUjMHs2nM+J0Spv4zhMQrDAJXI23ujskwUSiEuVTsasLSeyk0kYhuUVbcsL4Aerju2Y4VNv2smWfctix7mm/LZE7zpVplH1v4XmpAKKIwbTPnMaoTWP6hb04vA0cneYCOZ9gVXqitUNJGuEBaX7dwtfQk2HdALNShc9XbUV+oXMg6+KuYUN5fxBlhSbsOLDIo2JCGc0InNmxQL3RCVT9FYOORbvOdZZvTd2pxG+d4XJRxBzunJ3MtGCVU4I/k/ytBDfSi7SYN9Ni1yiHV QkH7JFml Hd2MrA7oaXHFEt3aJPtbDorxCUtg9p8636xmlIhzH+vebtgcNrjILkxOzkImiSSyPHXx9hOWK7wa5VwCq95nq1C6+/qsdRXxdJOBheXKAvLkq4NqGCodlZsur667xai7wGq+jni/58xwEDNsTIc3bjtKCUbupNs27/+mj9/FmZz+2tmVlJFhk4cMnSYzDTYxoREmmD6kJBZU2Hm/sn1yDDReV7F6GcdpjkQYjmbfVln30kKnpGxLJGHklGR3t+FfnxuDAA6Q2AqR7cx1B7KeYTXs3bwgEeJ86LHHJKxqG4ngQ+b5Haby23m9KiX0Jghr9kS1YqZ92QJvXrZkN0K7p628h151R90X0FFfpV/4PEgvlMibKpFodNRnBHYvGd3/ObY32f76CFaCIK57g8ZxG2s4K5tqrzfiz7cqQMooi6B2csA8sk5K6tNcX9roeKVScTMcx30/dzVazB9FpSlr9K9V+xbhmhBlSYZMMhQgvQIJT5kv19JMk9loO4IAZlBs3KmPb X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When the CPU touches a zone device THP entry, the data needs to be migrated back to the CPU, call migrate_to_ram() on these pages via do_huge_pmd_device_private() fault handling helper. Signed-off-by: Balbir Singh --- include/linux/huge_mm.h | 7 +++++++ mm/huge_memory.c | 35 +++++++++++++++++++++++++++++++++++ mm/memory.c | 6 ++++-- 3 files changed, 46 insertions(+), 2 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index e893d546a49f..ad0c0ccfcbc2 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -479,6 +479,8 @@ struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf); +vm_fault_t do_huge_pmd_device_private(struct vm_fault *vmf); + extern struct folio *huge_zero_folio; extern unsigned long huge_zero_pfn; @@ -634,6 +636,11 @@ static inline vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) return 0; } +static inline vm_fault_t do_huge_pmd_device_private(struct vm_fault *vmf) +{ + return 0; +} + static inline bool is_huge_zero_folio(const struct folio *folio) { return false; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d8e018d1bdbd..995ac8be5709 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1375,6 +1375,41 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf) return __do_huge_pmd_anonymous_page(vmf); } +vm_fault_t do_huge_pmd_device_private(struct vm_fault *vmf) +{ + struct vm_area_struct *vma = vmf->vma; + unsigned long haddr = vmf->address & HPAGE_PMD_MASK; + vm_fault_t ret; + spinlock_t *ptl; + swp_entry_t swp_entry; + struct page *page; + + if (!thp_vma_suitable_order(vma, haddr, PMD_ORDER)) + return VM_FAULT_FALLBACK; + + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + vma_end_read(vma); + return VM_FAULT_RETRY; + } + + ptl = pmd_lock(vma->vm_mm, vmf->pmd); + if (unlikely(!pmd_same(*vmf->pmd, vmf->orig_pmd))) { + spin_unlock(ptl); + return 0; + } + + swp_entry = pmd_to_swp_entry(vmf->orig_pmd); + page = pfn_swap_entry_to_page(swp_entry); + vmf->page = page; + vmf->pte = NULL; + get_page(page); + spin_unlock(ptl); + ret = page_pgmap(page)->ops->migrate_to_ram(vmf); + put_page(page); + + return ret; +} + static int insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, pfn_t pfn, pgprot_t prot, bool write, pgtable_t pgtable) diff --git a/mm/memory.c b/mm/memory.c index a838c8c44bfd..deaa67b88708 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6149,8 +6149,10 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, vmf.orig_pmd = pmdp_get_lockless(vmf.pmd); if (unlikely(is_swap_pmd(vmf.orig_pmd))) { - VM_BUG_ON(thp_migration_supported() && - !is_pmd_migration_entry(vmf.orig_pmd)); + if (is_device_private_entry( + pmd_to_swp_entry(vmf.orig_pmd))) + return do_huge_pmd_device_private(&vmf); + if (is_pmd_migration_entry(vmf.orig_pmd)) pmd_migration_entry_wait(mm, vmf.pmd); return 0;