From patchwork Mon Jul 8 16:57:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Martin Oliveira X-Patchwork-Id: 13726871 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E040C3271E for ; Mon, 8 Jul 2024 16:57:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 60FD16B0093; Mon, 8 Jul 2024 12:57:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C0BA6B009A; Mon, 8 Jul 2024 12:57:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4115F6B009C; Mon, 8 Jul 2024 12:57:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 245116B0093 for ; Mon, 8 Jul 2024 12:57:47 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 663DD1C2E45 for ; Mon, 8 Jul 2024 16:57:46 +0000 (UTC) X-FDA: 82317192132.28.7A978EE Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2111.outbound.protection.outlook.com [40.107.93.111]) by imf11.hostedemail.com (Postfix) with ESMTP id 9A40540027 for ; Mon, 8 Jul 2024 16:57:43 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=eideticcom.onmicrosoft.com header.s=selector2-eideticcom-onmicrosoft-com header.b=EtvkJSjc; dmarc=none; spf=pass (imf11.hostedemail.com: domain of Martin.Oliveira@eideticom.com designates 40.107.93.111 as permitted sender) smtp.mailfrom=Martin.Oliveira@eideticom.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720457830; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=KwHU0+f3eZPAD6XFLlWvYxFwBJQpaASLsoXRhKff3Tc=; b=u6VOYLGSPi7KxswxyoB41zGMxJksjgZwJ2tR0YJlXUpLwoTlsEiitKfAbnDMc5iGrAvrCx 6t/A7iKB3oHL/OX855SwYwxd9OwZ+n7vMvLm9UHCinpuFz09oaNJPo+C/B3zBRwQgm8PEk zMihxqyCJhhopgZb1drPNgLQKx1rAnY= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1720457830; a=rsa-sha256; cv=pass; b=pnCYdKzseAC3g3vbBNDCNf+PYKGjMsma2jcj33Qe/rOu6hnDLE2EWs0xY84Eef1gVFHulS DY2UOOLndxlw18YTzK72SfeggEwQo8gNjQX/ONasnpXnMmYhZFU+BZnwqmHQzSNW6I29x7 suvfLUFBAalWLaUVckkGxrWrkV3s3Hg= ARC-Authentication-Results: i=2; imf11.hostedemail.com; dkim=pass header.d=eideticcom.onmicrosoft.com header.s=selector2-eideticcom-onmicrosoft-com header.b=EtvkJSjc; dmarc=none; spf=pass (imf11.hostedemail.com: domain of Martin.Oliveira@eideticom.com designates 40.107.93.111 as permitted sender) smtp.mailfrom=Martin.Oliveira@eideticom.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fPLy//ygiqduHkk4qnLRHBVvxi3RLaqW4U4+MmXysLJCHtBQ/casuMaas8HTChHS6kwMIxjCflHatpkULTSClEA/PPUM+uUXYtzxVgp2ZIC+vJEVeemOAuQTL/20ZtVE19ef6CP6by0AtVvpqrdGFAx+rraBkrjg+Erk9viHfxSvVj807R/hmzA/YyWTg5AzDaRUsr2T/db0M9cnlzNMp6kHyMSPhMmPgmEHnf4CYZgk1E32x/sS3wM49WvGp0b9/ma0XEjzSu5BjMw4VgQTXuYYRffxrrYBbLVHe/Il6l2l8Eb+XaJ2o/AtZLNYWSzvQV+dFQBqWMFyELTxSCPFLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=KwHU0+f3eZPAD6XFLlWvYxFwBJQpaASLsoXRhKff3Tc=; b=KGqn6kllFCrhOYsEmGslL4QURDiIU3nQQYHVxpl/G/C0kTYumqimFb0Ac7R3SVMumvbJs7A5Sh+82XUblqwTogxDMf40c7Us3+Hn3BrYvkD3oP/zscwkokxrlDYFmFn0fA5bx4NOgbUvT9WYLGUipjHt1DPeo5yd4npHJKc7dY4Ivdv2mvJvV7P3FD1PyAvmGV76QqVRFt59mE0GZ0q1yEUv3CaMnwupOx4yPdy3N17UvnaVgIYJg5ivUoCW2dsU411eo9ApJz/mVkqkCdS6KH0SrpLlamoTmLPGrDYvOymXcZnIzUDJDzT08LH7Mb3+Wb/3BjJYkagMzNcog5ULzw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=eideticom.com; dmarc=pass action=none header.from=eideticom.com; dkim=pass header.d=eideticom.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=eideticcom.onmicrosoft.com; s=selector2-eideticcom-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KwHU0+f3eZPAD6XFLlWvYxFwBJQpaASLsoXRhKff3Tc=; b=EtvkJSjcksnkP18K7BkU7ATbnH13B7bFBIGG7VuuQgInUtPLPSyViZTtyCfDhWH403xyNyZKCfv8XRPPbiBqKpRpAVRGGjZyKmR5NpJUMb2MXbntjnndK4/wqy66dEme7aNjjAvg9R38VDywHAHj2SIY62azhYgtjlPZxdOuQDE= Received: from DM6PR19MB4248.namprd19.prod.outlook.com (2603:10b6:5:2b0::11) by BY3PR19MB5028.namprd19.prod.outlook.com (2603:10b6:a03:361::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7741.35; Mon, 8 Jul 2024 16:57:39 +0000 Received: from DM6PR19MB4248.namprd19.prod.outlook.com ([fe80::d508:c71a:eb4f:7cf4]) by DM6PR19MB4248.namprd19.prod.outlook.com ([fe80::d508:c71a:eb4f:7cf4%6]) with mapi id 15.20.7741.033; Mon, 8 Jul 2024 16:57:39 +0000 From: Martin Oliveira To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-rdma@vger.kernel.org Cc: Andrew Morton , Artemy Kovalyov , Greg Kroah-Hartman , Jason Gunthorpe , Leon Romanovsky , Logan Gunthorpe , Martin Oliveira , Michael Guralnik , Mike Marciniszyn , Shiraz Saleem , Tejun Heo , John Hubbard , Dan Williams , David Sloan Subject: [PATCH v4 0/3] Enable P2PDMA in Userspace RDMA Date: Mon, 8 Jul 2024 10:57:11 -0600 Message-Id: <20240708165714.3401377-1-martin.oliveira@eideticom.com> X-Mailer: git-send-email 2.34.1 X-ClientProxiedBy: MW4PR03CA0252.namprd03.prod.outlook.com (2603:10b6:303:b4::17) To DM6PR19MB4248.namprd19.prod.outlook.com (2603:10b6:5:2b0::11) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR19MB4248:EE_|BY3PR19MB5028:EE_ X-MS-Office365-Filtering-Correlation-Id: 752ba691-ab09-407a-0a5f-08dc9f6f1369 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|52116014|366016|1800799024|38350700014; X-Microsoft-Antispam-Message-Info: Z3eR+bCmVg3+Z+y431Fe2kkGxtLBkT7allSdIS0Zu+Fusd197NKr5iYbZvf7ukwfYwYwqkRdK9xbGdX90aJ4Of3z9roYkH11avNInEU/IPYfEQYWimcPypMcKynYumdzAM/IvYGB/nNmPBaGt3ADIkZWgGtFnYhqT2Hd5HmLR8w6reRaYL3bQZNiXvLKHXCZ26/e6mVGL/r8OpKG5mrbI7KgSm/G/y1l4kowza+bnCuf/IMmEbPfKGWD5IM9af3Cf8pRyv6Jo8uRmlLCVcfmDFsjacXnCArgW2BGXqSksoRWZTDhklsn1MFLJfhOktOWUCbS3QeIoR/3kuv/MA2yGSUfccCHcdv2g2Dj2fkaTsaUVcr5xyY529rCmseqghVypMTnCQRbv3sv9tYhV8UhFMhMXHHfsklP3R/v96wR2CsuihI27nbiPRlLnpjhmZ99aJ/Vbf24ph+NrPTIuI/oaty09Nq/7iMK9Ui+nnmOVGZi+hFBNyJTnorf9+jxsI0ZUfsUnQgy1cX6riqCKZn+lUh0vq4BwJt3v7ZCJHTfVG6YEkEAhgNWwE6RtgYMpYPyuoZhGWsddj23lBqULGo7oE/vbruBiV6X1DSrYbTx6rxl7l6mzMtuDkBuMmfe4ENFwHWP+EOyGmb2lkngbbpDT0Y+nRdtHuEsmDdAqSEOHK3bExWF3i1Bq/oETPjOHdLNiGGeD7TwXpUZTGZKx87+ysAv21d+necbJMTWveySNpxS8d8ZGvbljNcKXpIG7342b9GCxnhLMeWS6JeH8sD/TaEWX1mz1B1zmgLiyd7KVUf7BazcdIDyxlhjiWWikiHadRPgMgEXkTxOI8a/Kmis6yA5QJ66RmykGDEiU6v6EmlW0L/Xzh6YgZ700PRXT259koWQinBgdAlmNOFjgnrKB0SoD4i7a+ErPFwkbuAzFUFn1BaiyxAuxM+fe+6lMGDrvRHmbJ2UVi+VviBpZ4+idhvU39qTifNghD/ZkGFDlduaJBhgfs5GVMWOHJ8Ovm052Fw8TwbgzSsu/VttBBUp9TN6qpXmclkm1ktAD7YiJPyEPMdSe+a8PYwWaFSa/bowj98LeMkuRl7FW6Mu8t/1wQyKo33P4pCSMhBshXYu1an3NYP8XvXba0AKsXeVj0bIV8sSuMEE6tNwbSQ/k9775LuKC3t15x4ThPugZELzG4NVe6Sh+mm5/UbfOZDrvsJUB32WTrrtaJulqW4fxIz3BfRVp6fIfG3WF3H7lL3dm482stqIHU+zUPRTXeyZW9olQAaBd8zid//i92cyy4b+YsfPqve6neyK30tvVH8ONKqXlPMZPZOT+A8/Z1U0e+WP7k0+pKoNzbLXokwYgLaBwFGd2JNZpdH1npuPMD/smxM= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR19MB4248.namprd19.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(52116014)(366016)(1800799024)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 44Q7Uuu20Vbs+vcdzN1AamvmILZGLCDcA5U9D2qzOFKnSh7KSn3BjKIbw8STogg59JpoWShdpN4V73tk5XI+fCZNX7C3uFDYRLB0p5Et3kizchLvAtdP6gXnk1tPfqXg6lx0Fo1zusOvak88bIPNMwKlO4hY+DJNHHRFkMYBMuXGQHX0+L93Z0wFr4G1PWSxklCm/vFIaI2IbAHF+mnHkfRx3AOwuhKcUtR+go9kVFpf3BueZuOkHTaTINv/NYRH6yRe1wNI/3DQl/f0FVhdrRm+9Qn687eMna/lYwmmtVVQq2XK1UW0QOpVB5R5gOXSJEg3yRF00Scb/obm+dfWbgbSGvL3sFCkmnUZfIN/XNvn1upBcpqRLIhJMatETcxxbEBK9DgRFuhv3H7PIcTqbhtSF+d/PMilx90+rwQewYRgxAGYHtIAIPggHP6jUrdmpvmZtrsChYYyG8KH+GiLLNfpf0O9OFvRhPezN2sU8kwY/5aP0PA5zYGWn7wvek0gm7bUx60Q6m4GGihLGksXdXSbmJQs0Z9HcI4J0GHa9G/i6mHgjynZ46y1F2w/7+jIj/GUfJFULDwATXQQw1x/+ZJ3/cBKDFX7cQnfdAVAtDKgfgDV57TULezUbdS94TrpIwpdtOZGnhNK3y+oXEo6d3+5L5QW1jqEm5j9Epm++rgYXmwmkA/gQRF2n4WiGz6rv0/Nmn1oQOMs7qQGhizktn8DRAPr3TY6iTw3daHLPbsfFhs5kTOQZdcudnLM4jxM/OijszBrz7ZdbXVfpPXXwfVTk0m2V3cQqmBU237XhUENgQ6zkW1Dp0+Jk1yANSgjgrD13X0eg6t3azPGAsTzBeQa+1xHKyN1ZTj02fp6tk9ipVz+QA5dUqtfIJPjQ9QSW6hmaCPhMKm935MaBFmKbV8NLGOUy1Tcfmb70stjV8D7rgOPCmogzo/T+1izIuX400qhzue/l4LnX3hhJwhe1bMDfvVOqzFtiX5a4Fy/7XhyBC2JyMaffOJhMBhHZWrGYbHVg6X8HtEdH6DU3pZZcySMYf6+IVSzZKm9U9wPFK5C8kt8TKewwvnoE01fniKvzM3ROks/nbmhp0/8jPU/TJDSJR8LbeQ52tv/ciE+R/sMM/i5Y4yGNyhki8/pqExQ/HEHGUcPIqHZjlcp1spUGHt3Pb00tI32mOhfiNbRVZM7Ys13016+JGGZX5KxqvoHcx+jdO4ax7tf37NN07UkUBxX0JCWOcIAXS87YXo7emLNAiSEpO8qyCAMdqyPnNkzMmtOnKVbKVNC5suTERmJ7OFUxvnIjFBpEQlGciDGX9J4QOZUodKJHInZlLKUs6OhPGANnMWPTzEYQ02lX30QYOMBe8DKbHId3YRtHng6lpfu87uROG8D1/2wmOznKuiFhUqod93HNPVPpLL1JpkO9w2YrTczW9wMuWnfXkvJExzL5ww2kHzPlKVK0sqvkvMvkTs2BfddQcxvnL3PkReKoZHaOH+CKsxm7nRYBa2ukACQ8CRRNfZQeEAKD7d2d/7jSPpr77yXGrYFYFa6dRj1DO1hYzRo5YPY/cf7sZMSncMCQUXm7XKHomffRnvW9n5d1r8A7YPxjnIo9zfZ3xPLDQ== X-OriginatorOrg: eideticom.com X-MS-Exchange-CrossTenant-Network-Message-Id: 752ba691-ab09-407a-0a5f-08dc9f6f1369 X-MS-Exchange-CrossTenant-AuthSource: DM6PR19MB4248.namprd19.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Jul 2024 16:57:39.6959 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3490cd4b-0360-4377-abb1-15f8c5af8fc2 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: y4R5uoC6PrRYOyxllpt1X7dzKkXIzrAZj+uBIMX92tf1mDcZyLl6/gmdgFxbqxFdbKd8Mqo0Lpb05BJbTcz0LPrQ0RWsBn9QiKc/lcgnTvI= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY3PR19MB5028 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 9A40540027 X-Stat-Signature: z6hq8pyfwpqq7ozd57ez5zrk3sjmizxx X-Rspam-User: X-HE-Tag: 1720457863-449851 X-HE-Meta: U2FsdGVkX18rsRyW4ei41mSirjrRQoJ7dFeHUOTdphR+Dz9kwzigcMnmSbVPvIszQWSw68a27Uk8b6ONY6hPKNGGR9+5nSQ0looNcHMKV9GPIkfdIZWFQHtI+kuxwE+7dnEHYE5S/YBcaKqDoQroWDh+2SPSOFrLwVpnECs1xzR0WGfHJklYot2Zv/71s2+bhMxp4xVlobvy8TQNIEu5HY/e5twstN7tWoPLluQLODR25WnEo1kTWgcZX33fcsbbf2pYvKOJVwbVQvXTWCj3QRKngXezX2NwP1SyzAyF/peWv8lGyVCMcrVlxM0gugooX/ZZgf8541Cs2JTxPfwAq1zsCkHrL8zqYvbnQC5oNwvxINhjTOqdZjoQsTpbMuETZp0pMMq5F979tbTfHw+a5uijGxLozkWwmiunyeBEfAYX79vGU3fPe6UJWCeX3Fm52Es4u1XTTmGSK/bdSI0dnNdreeCo7VYuHPC3Il+67Y3wSU0TxvDRcPPtNVJXgZzw3i2rreqeec7uniCwNtA7tVSQbcfQtCyQE9M+BPvl3Zb6U9CkWeBApMOYOJ+cE2u8IUrS7esVGOCjMINd/MsJFlYfZqdv65b4Th2yiGf9uyCGZo4nSka2bHRCxRRg41rYltrIkiHlntaUOLcUJtSDVxBW0Kz+CzAvrQpTtPKvh9pNX+VUeGXvOXQtZHL4sQnY0eMHXWnuX28thIflzkvBbi6x0VMP35AxC+Eco9rBGMLNrwWqgbofS5I6BU/R6+U7u07787nzUnNfmHwRNnP95mK7XIl1iuCzXIoMjtu/eivvLiwhUNz9VpyZA1PW1dtOrAXfFINqisFVFn37SZBSXTxf/lRjqNdNy7IjyOZGZWlU/C6TjEQRPYLAXzdPo8GmSGYus3jzoD3WU9lJbixMOzWfs9gja53boHmTnEOArUbjsx7J6YDSqKOIL6uvDHxbMIH51bWkm0xRbgEARoz efK0/vs9 0fzGsu34+x/wXPHaxS2NM71T2cl9cVHqvdSIA5eRjsRic6FJzHeqOn+HFe42qY2gbq+WV1b6pwpc/UNkUwIbRdOc3Uz8qCBn2L5r7Mbf+eb/R2yUO4UKfhl8OaLCBJ7yfhkj6rP0ckRzcZDyv1vqBrFZqbuelhLfzhU5fj2KzN5AEYL8jDkHzqWp80X6EKqgtkDYiopDnkLGWGBiAZbALhjzTlc/oOI8WQWOual4eUXhPpZ5Oikh6ctfWDjue5/lLTATfqVOA7x1B+Su/ZHJo894blrkulmSPgCNQ6DH5XgHzPDjAOhjNgfzKMSgkZXv5J0o0CooyEOli1v4Fy+R7jgy/xioumZEdzqYSYkDtV6954ps= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This version of this patch series fixes the unhandled WARN issue from v3. Everything else remains the same. Thanks to everyone who provided reviews and feedback! Martin Original cover letter: This patch series enables P2PDMA memory to be used in userspace RDMA transfers. With this series, P2PDMA memory mmaped into userspace (ie. only NVMe CMBs, at the moment) can then be used with ibv_reg_mr() (or similar) interfaces. This can be tested by passing a sysfs p2pmem allocator to the --mmap flag of the perftest tools. This requires addressing two issues: * Stop exporting kernfs VMAs with page_mkwrite, which is incompatible with FOLL_LONGTERM and is redudant since the default fault code has the same behavior as kernfs_vma_page_mkwrite() (i.e., call file_update_time()). * Remove the restriction on FOLL_LONGTREM with FOLL_PCI_P2PDMA which was initially put in place due to excessive caution with assuming P2PDMA would have similar problems to fsdax with unmap_mapping_range(). Seeing P2PDMA only uses unmap_mapping_range() on device unbind and immediately waits for all page reference counts to go to zero after calling it, it is actually believed to be safe from reuse and user access faults. See [1] for more discussion. This was tested using a Mellanox ConnectX-6 SmartNIC (MT28908 Family), using the mlx5_core driver, as well as an NVMe CMB. Thanks, Martin [1]: https://lore.kernel.org/linux-mm/87cypuvh2i.fsf@nvdebian.thelocal/T/ --- Changes in v4: - Actually handle the WARN if someone sets ->page_mkwrite Changes in v3: - Change to WARN_ON() if an implementaion of kernfs sets .page_mkwrite() (Suggested by Christoph) - Removed fast-gup patch Changes in v2: - Remove page_mkwrite() for all kernfs, instead of creating a different vm_ops for p2pdma. Martin Oliveira (3): kernfs: remove page_mkwrite() from vm_operations_struct mm/gup: allow FOLL_LONGTERM & FOLL_PCI_P2PDMA RDMA/umem: add support for P2P RDMA drivers/infiniband/core/umem.c | 3 +++ fs/kernfs/file.c | 25 ++----------------------- mm/gup.c | 5 ----- 3 files changed, 5 insertions(+), 28 deletions(-) base-commit: 22a40d14b572deb80c0648557f4bd502d7e83826