From patchwork Wed Oct 16 00:05:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bernd Schubert X-Patchwork-Id: 13837444 Received: from outbound-ip168a.ess.barracuda.com (outbound-ip168a.ess.barracuda.com [209.222.82.36]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E0A5BF50F for ; Wed, 16 Oct 2024 00:05:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=209.222.82.36 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729037148; cv=fail; b=kQ6rTbEscc7u/Bv7fAHS7K2m3OG56eRC545nK0yHlwUFITf0Zg8R3obEUIP+cswQsihWxkkxsmG0Zc2Lmd0jK7BzpNapxfm1SGYiVqv5a24u1vBdl4VICV0RG7USi2kSv/0n2tZmyJ9M3qo7qapQwpvndfsgiOd5PhDTDWU2RsE= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729037148; c=relaxed/simple; bh=LNRjk1BQzlDUsM9lv7ATvb3bUoxo9H0fgwLuky5tTO0=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=tRUvZk4GQEAZ8G+t+I8wRoB/zruQRIkQGKQiI98WfP463A1TQenOe6ONROe3wV4r6hHp4wB+qDsm0/FQ4UkswHVgNLf+6OxqDvirEJE07dqWnGqHOHbISnyCuyPx/nWy8pF4akGEvnTvZErue9BYCVakWvkYcMIJHZUw6R+k4Xg= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ddn.com; spf=pass smtp.mailfrom=ddn.com; dkim=pass (1024-bit key) header.d=ddn.com header.i=@ddn.com header.b=LqVTt29W; arc=fail smtp.client-ip=209.222.82.36 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=ddn.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ddn.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=ddn.com header.i=@ddn.com header.b="LqVTt29W" Received: from NAM04-DM6-obe.outbound.protection.outlook.com (mail-dm6nam04lp2046.outbound.protection.outlook.com [104.47.73.46]) by mx-outbound46-22.us-east-2c.ess.aws.cudaops.com (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 16 Oct 2024 00:05:43 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=XyI1gunq0EwOPmE+euQvvzlfCUTDcTvzxRig5E+4e+qpCkgxqsPdYgq81rIyYVmZ1wDzIQZAL7UqUMVxEqaOuiPFXbjqPjUXIddUHVJjOO7mibA+IPAuXZBudn0ZoFBCt9QsoYM7sT4A39YosJBtv5PYBXiE/WSug+iz3D0gNXbObHsBUPac6FTe4nZ6HcjIujFy2YhlEcmsEJ00fELFXNLc/JqvlNDhRdnvgCBOcZQMrBhg1H0TKvCoixQdCLavcS9V0AV1E2XyQg02Wev+nOivOzU/wMpmKO+v8S1FtGYZ4LMGe4Oer2d2gBeRzFSvoWaRwX5DtZoTSh/C2hsYOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=sJzM04F5auMPqTC4+F6onlxqr84QUG4DCTOySQU1uKA=; b=QJoVJwBO5+AgtCqplQktZd0X5VW/31/8Y7pjFXdG2TBLfHMQomXVyCZtW/U9OkygIAvOqXp40E21bU/2ACbytTLWe86DsKUfjUxKeDdOvwCZ2sgU2VMqbwRdj7Ow0o07UTIUsOeGvQLJyf5gcA32TFEzqjWMp8B9dj38FZtZ/j4UeYTPRqS/uRjPMyLsW9WiQEwfr76TE4byKxhIy35jau/b9TmbJFIXbDp3Sn7vBxVO8wf+bNdUAfHCh7hu6iIREuKakYn2S91IoO5Qif2zJJbIQiDPp+CGIDadtLDjetNaXp8zVaSwtewrSnlXjynzaPsMwUbYV5FUFMoiBUx3XQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 50.222.100.11) smtp.rcpttodomain=ddn.com smtp.mailfrom=ddn.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=ddn.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ddn.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sJzM04F5auMPqTC4+F6onlxqr84QUG4DCTOySQU1uKA=; b=LqVTt29WohiODgNuwFBdjwxdbzSTLU+s85OOPEqa/Zp/wTLMHaY71JmhmVGnYPUkEuVSjYTjs6Ca2pW1cENmJm3gQhAMh/N4xHjm4bDjp4zw+zCsW15kmq/DsCWzpmgHOZVdlLWrXc3A7kemLd29wUm5OhVr34qdnYXEJZQmdNI= Received: from SA9PR10CA0009.namprd10.prod.outlook.com (2603:10b6:806:a7::14) by DS7PR19MB6133.namprd19.prod.outlook.com (2603:10b6:8:87::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.26; Wed, 16 Oct 2024 00:05:39 +0000 Received: from SA2PEPF000015C9.namprd03.prod.outlook.com (2603:10b6:806:a7:cafe::f6) by SA9PR10CA0009.outlook.office365.com (2603:10b6:806:a7::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8069.18 via Frontend Transport; Wed, 16 Oct 2024 00:05:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 50.222.100.11) smtp.mailfrom=ddn.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=ddn.com; Received-SPF: Pass (protection.outlook.com: domain of ddn.com designates 50.222.100.11 as permitted sender) receiver=protection.outlook.com; client-ip=50.222.100.11; helo=uww-mrp-01.datadirectnet.com; pr=C Received: from uww-mrp-01.datadirectnet.com (50.222.100.11) by SA2PEPF000015C9.mail.protection.outlook.com (10.167.241.199) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8069.17 via Frontend Transport; Wed, 16 Oct 2024 00:05:39 +0000 Received: from localhost (unknown [10.68.0.8]) by uww-mrp-01.datadirectnet.com (Postfix) with ESMTP id 644F880; Wed, 16 Oct 2024 00:05:38 +0000 (UTC) From: Bernd Schubert Date: Wed, 16 Oct 2024 02:05:26 +0200 Subject: [PATCH RFC v4 14/15] fuse: {io-uring} Prevent mount point hang on fuse-server termination Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20241016-fuse-uring-for-6-10-rfc4-v4-14-9739c753666e@ddn.com> References: <20241016-fuse-uring-for-6-10-rfc4-v4-0-9739c753666e@ddn.com> In-Reply-To: <20241016-fuse-uring-for-6-10-rfc4-v4-0-9739c753666e@ddn.com> To: Miklos Szeredi Cc: Jens Axboe , Pavel Begunkov , linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Joanne Koong , Amir Goldstein , Ming Lei , Bernd Schubert X-Mailer: b4 0.15-dev-2a633 X-Developer-Signature: v=1; a=ed25519-sha256; t=1729037122; l=3263; i=bschubert@ddn.com; s=20240529; h=from:subject:message-id; bh=LNRjk1BQzlDUsM9lv7ATvb3bUoxo9H0fgwLuky5tTO0=; b=OSxHr7iE5Z9jpuEd/Hh4hKpj+WZY6v6a1lCErC19rnFjofITSCT53k5PGozETC9dAjChOkLTO 6aJ9zQx/6x8D2QdJHwaFwVObWLsitRFVH5igA8X+uj9j7VpqG0HAyTR X-Developer-Key: i=bschubert@ddn.com; a=ed25519; pk=EZVU4bq64+flgoWFCVQoj0URAs3Urjno+1fIq9ZJx8Y= X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF000015C9:EE_|DS7PR19MB6133:EE_ X-MS-Office365-Filtering-Correlation-Id: 07ccda9f-6189-4c09-3d69-08dced7644aa X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|376014|1800799024|36860700013; X-Microsoft-Antispam-Message-Info: =?utf-8?q?rVlXsNFp3VNM1j+92nAeQ5BSAewQtrC?= =?utf-8?q?pdO4YeAAkGB0HKf09ckNN4Q0pNH5Lxgi2IIG3uEme2UczcK9a3RB18R9LxagpHUY4?= =?utf-8?q?vCic5RyKPZaGEhCXgrLK9X+y8kubruvGIHeI5aVDKrqyFxjBXp1AccA3MMlvEyYeU?= =?utf-8?q?k26UZXrYUnHNqmjuZz6xHhc17+Yb01/WDaXIFCMK4/FqwatLjp+5rY0Q9S1tWuk96?= =?utf-8?q?6iHeKQjExqQouV5RZJiq9jIR5OU99klRqb5T9dMoX68vAXkSc8RFVK6xBnPKWCe/i?= =?utf-8?q?UWJAtonijgcVyDzJZa8kHCuyGUHQgHIBlPr+e4udYuEaGA5rs8wib8l9QLz/zTNzZ?= =?utf-8?q?+P/PjdcxDv18CS9OCGtFa2MprPYouDNdjjKcWDAVtWEm2fq2NrtsIn8bAZC46P3e4?= =?utf-8?q?L8NomHeTG+TItrWVZePycVaZRF5mt64p4EEUm4BqR6XuWKG5v6qlBSNOBK3KwMSzX?= =?utf-8?q?CA8+w8o7+I4dzjGN3bV0+eDNSjpVesWgTHRolKYlcSyEMWzipf8fGv9ogftYD8WWX?= =?utf-8?q?hSdVsWnYS6uOk2rbrYigGrVCE4vg6ABMtPLdOp00iiIrx24wrVBJHYJY0C9YAwA4w?= =?utf-8?q?ddZs7+mIGVl9lM3aAYuC+6ni12uPLgq5k023g9ln9faB6TAHjr/sY9LIDHlC9CBa8?= =?utf-8?q?aMH+kFzWCZ86aYL4KEYkVpMNu8biF8cCK9pPSCneXXv2i5EBNkPOv6NwkD9zDY9ET?= =?utf-8?q?xn91gnXnUNA+2Rtcj4ypFaBqdOgej8bsvlqs9PkABlCrSNKNLbGSfuxgVG1xXOZqH?= =?utf-8?q?6PBQ912E9ghOEj8l4zuriMinp2VE20ttDDNyGRVnqE+dlbi07VwU+fL9/mMjmFz1s?= =?utf-8?q?+25WCC1qnaRzX4+Rui95mDzpQF/YwGv+WQcyF9QeemoonUX9wBcvBM+w892nJNKNv?= =?utf-8?q?ezSEDpeVrrktdNsBzTqyzPdT94mexYbEOpQvpDwm28mN6aGRVomqKEUCT8u1/bAyM?= =?utf-8?q?VtM/UJSPO6e7CpxBbhjlHO/ACq/fZX0HSISjblTHJnoY6sVfuUENeHI7PKRW2Cjye?= =?utf-8?q?HxqmcR+F1qLHpYhygUmJl3fax9MdCya5lwoguIr+rlYYdMPYuAAAQL1agnJUJALRo?= =?utf-8?q?6CsXfC6UObbv1X4aqaHPjxGLV75jibwP3+m63wdiFIueAM7vdguzh3DsU77z6hrAH?= =?utf-8?q?QiMqlYzLIAAGHNjVBEzrbC1LxI76gfPFaE1S0Im2LUVL4MSQCatBiIkc092d4+o45?= =?utf-8?q?FQ+T/UVq6mDJNEd+IHOQnFiSv942m2lITvZa24PY9QUlGxBOddWBUZc1KDXrfYmuB?= =?utf-8?q?S9fA8TULmaAGux65p7UZ/GgDWK4YMAg4/Z84I3j2CykUIDVlS2BF9OK9jsZbltsGd?= =?utf-8?q?8Acq9D/X87bP?= X-Forefront-Antispam-Report: CIP:50.222.100.11;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:uww-mrp-01.datadirectnet.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(376014)(1800799024)(36860700013);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: Hcyr/Jz/8OIZxRWrfKtKDoEb9Y1b1kKn6WYuEr8Pk/jfaRFHhMUqf7gGMA7XGqJu6D96Mf05SE4cp0s9M+ufra3C+XWyjrv1MtJ94VQvHX8/2ezyV+jOxZuKg3MIwJ15bBTLszghE9k2og3yTBipMQzfm5u7EaFnW+zO+C1HYW+Rvqhdcl0yk//hcGLcMD879X3qB8R+933HFlUCd5RxA2Ek8+iZ3q3RFpwZO/Pr+5zYSsvvd6QTVZPg5yZPXIECDiQY/5Th2H4JveOZPsh3W2mMggxuMNTJUF6MkU7k/AsmjGdZiPdlRr5vIx4AKZxLG9PXbNy1QDL9/LsMctDmYrdzWSGQkVB7+LAaMEF4xqZFCoCfGaO5uCJ/+/8kw1DZD6Jd4nlHyFxlofVxCikp2E6UfGOrQH2QDkiBhNBeXqE+DYHtxw3DaMqkofnYUDjf8vGOkj5LW9Svyz5sq/F7tr/RJfpjHK18Xz+M+i0Z8/NEu3ewxzUGqX85mvaklyCU6RBORMelrPlFZ8td3HaaDUzwLd25P1kbpm4hGtfM8oDxL5h4bUdGy4IWWZ0lCzHHAXAQ4BpMyJtBxw3DOdscwQwXg4T4YtFvEyHGA7BLD6h3zbZnG1cHixxLbJMFs1lJ5GJll+aCXLh72CrLsJS8Pw== X-OriginatorOrg: ddn.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Oct 2024 00:05:39.2917 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 07ccda9f-6189-4c09-3d69-08dced7644aa X-MS-Exchange-CrossTenant-Id: 753b6e26-6fd3-43e6-8248-3f1735d59bb4 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=753b6e26-6fd3-43e6-8248-3f1735d59bb4;Ip=[50.222.100.11];Helo=[uww-mrp-01.datadirectnet.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF000015C9.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR19MB6133 X-BESS-ID: 1729037143-111798-10778-56326-1 X-BESS-VER: 2019.1_20241015.1627 X-BESS-Apparent-Source-IP: 104.47.73.46 X-BESS-Parts: H4sIAAAAAAACA4uuVkqtKFGyUioBkjpK+cVKVsaGpkZAVgZQMDkx0dTAzDzJ0D LRMC3NyMzUwNw4MdHQyMDSwsTUIDlFqTYWAKwjkepBAAAA X-BESS-Outbound-Spam-Score: 0.00 X-BESS-Outbound-Spam-Report: Code version 3.2, rules version 3.2.2.259752 [from cloudscan9-84.us-east-2a.ess.aws.cudaops.com] Rule breakdown below pts rule name description ---- ---------------------- -------------------------------- 0.00 BSF_BESS_OUTBOUND META: BESS Outbound X-BESS-Outbound-Spam-Status: SCORE=0.00 using account:ESS124931 scores of KILL_LEVEL=7.0 tests=BSF_BESS_OUTBOUND X-BESS-BRTS-Status: 1 When the fuse-server terminates while the fuse-client or kernel still has queued URING_CMDs, these commands retain references to the struct file used by the fuse connection. This prevents fuse_dev_release() from being invoked, resulting in a hung mount point. This patch addresses the issue by making queued URING_CMDs cancelable, allowing fuse_dev_release() to proceed as expected and preventing the mount point from hanging. Signed-off-by: Bernd Schubert --- fs/fuse/dev_uring.c | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c index 6632c9163b8a51c39e07258fea631cf9383ce538..5603831d490c64045ff402140c317019e69f8987 100644 --- a/fs/fuse/dev_uring.c +++ b/fs/fuse/dev_uring.c @@ -21,6 +21,7 @@ MODULE_PARM_DESC(enable_uring, struct fuse_uring_cmd_pdu { struct fuse_ring_ent *ring_ent; + struct fuse_ring_queue *queue; }; /* @@ -374,6 +375,61 @@ void fuse_uring_stop_queues(struct fuse_ring *ring) } } +/* + * Handle IO_URING_F_CANCEL, typically should come on on daemon termination + */ +static void fuse_uring_cancel(struct io_uring_cmd *cmd, + unsigned int issue_flags, struct fuse_conn *fc) +{ + struct fuse_uring_cmd_pdu *pdu = (struct fuse_uring_cmd_pdu *)cmd->pdu; + struct fuse_ring_queue *queue = pdu->queue; + struct fuse_ring_ent *ent; + bool found = false; + bool need_cmd_done = false; + + spin_lock(&queue->lock); + + /* XXX: This is cumbersome for large queues. */ + list_for_each_entry(ent, &queue->ent_avail_queue, list) { + if (pdu->ring_ent == ent) { + found = true; + break; + } + } + + if (!found) { + pr_info("qid=%d Did not find ent=%p", queue->qid, ent); + spin_unlock(&queue->lock); + return; + } + + if (ent->state == FRRS_WAIT) { + ent->state = FRRS_USERSPACE; + list_move(&ent->list, &queue->ent_in_userspace); + need_cmd_done = true; + } + spin_unlock(&queue->lock); + + if (need_cmd_done) + io_uring_cmd_done(cmd, -ENOTCONN, 0, issue_flags); + + /* + * releasing the last entry should trigger fuse_dev_release() if + * the daemon was terminated + */ +} + +static void fuse_uring_prepare_cancel(struct io_uring_cmd *cmd, int issue_flags, + struct fuse_ring_ent *ring_ent) +{ + struct fuse_uring_cmd_pdu *pdu = (struct fuse_uring_cmd_pdu *)cmd->pdu; + + pdu->ring_ent = ring_ent; + pdu->queue = ring_ent->queue; + + io_uring_cmd_mark_cancelable(cmd, issue_flags); +} + /* * Checks for errors and stores it into the request */ @@ -902,6 +958,7 @@ static int fuse_uring_fetch(struct io_uring_cmd *cmd, unsigned int issue_flags, goto err; atomic_inc(&ring->queue_refs); + fuse_uring_prepare_cancel(cmd, issue_flags, ring_ent); _fuse_uring_fetch(ring_ent, cmd, issue_flags); return 0; @@ -947,6 +1004,11 @@ int fuse_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_flags) if (fc->aborted) goto out; + if ((unlikely(issue_flags & IO_URING_F_CANCEL))) { + fuse_uring_cancel(cmd, issue_flags, fc); + return 0; + } + switch (cmd_op) { case FUSE_URING_REQ_FETCH: err = fuse_uring_fetch(cmd, issue_flags, fc);