From patchwork Wed Jul 24 07:11:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13740637 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 098DAD29E for ; Wed, 24 Jul 2024 07:12:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721805133; cv=none; b=sBKDbBjHbHjbAgGUce20mMYfZMhI1zSvo/1q/ocyIUO0xTY7tpxLjHm37zwenK401soGqZTDsD2qE957bPiukpLFUmh08T9iBK0e023EO7a/j7IQHwkOKE9qHz1lZ+wVjXNwfE9uNUsm+Q/C2NYjHodVxFMhT+Vj+xi3PmSeLBg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721805133; c=relaxed/simple; bh=7lKq5GKeCiAb86MZvhON7q/aQQ/Lt6mJp4CwAsbFps4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QImt6QFlBdWEN15d9yaA7AhKIx0SY/bYPszPd3EV8n6kvtT4gA9L14VIJYduZg7IRSYZ7uCYd1o+lDHeQKbUW9He+0DRqq19gfCCtCPmgNYZOxIt7xU9sXqga8qUlcOXEyw+qXlydDBVVfZpVEFQgI/a80x/eJp3L0YpeMewplk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=KQ+2zZ3O; arc=none smtp.client-ip=209.85.210.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="KQ+2zZ3O" Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-70d2d7e692eso2039662b3a.0 for ; Wed, 24 Jul 2024 00:12:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1721805131; x=1722409931; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fr/pmDtspgk05V3DznP5o69OQ8+jdC4T9yBHzGzK7S4=; b=KQ+2zZ3O2jtdNHh8DuWqz2Lm5L5+On6e+AGpazi0yj/lMIMzwRprQXZ22ey5Oa5byy fCiyHRfIz8eb7AGdh7CozLu32VbABAHcgO29N9dVYC8oAC4SgdXoGLwVI+e8OLLSUFcm r8epvdHGZXx/i4qajYfnOejavIVfVpLWQmh4k+yIhbfnW97/Fy329pxJHTXgdu/j4l6F ZXX/uCRzsEoqqBjgyccxoRwCwSjvgg7mwCVQwQIzRwEMBhhIyXwfSUYoHWFtnkMhtTWJ tRRhlJpKEbTXJ7oPUmoXQrvs/aUQQCAMVB8C6Qk2Tpb9LMGAqWVNKb2ZfyEegose/Ksz s4hQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721805131; x=1722409931; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fr/pmDtspgk05V3DznP5o69OQ8+jdC4T9yBHzGzK7S4=; b=EWkcC9wDeakF6oK9aXgA7thHoXdaM+k1VUrcf8MEYEcc0CXgjx6xEJ/DVP9t4IFsFu tGvxmi5bMALNCNfcLG11DF1sDxB688PE0jRC8i8b/UvffvqBUi/lQhzsPNTXM1A6PHC4 z86H+FL1qWw9e7m0GOvqpN9Lem5K67ZupY1/MW2BeFmnRwVL3Tk2CYhFCqNtvJBwjr4L gk7C773iHJfZ+TXdo1l89G5unTo4spfF2iUaQC+j2tXkfxL5wx/x+fIn+iPMXkA3rQjD b7Q47e6+ADzyhGxx/q3lxxkTRxjJWwqTKgbZ6A+Y/oLeVV2pQPcoVA+DvOVMSNFYzbSj 3WrA== X-Gm-Message-State: AOJu0YxOm6R26ia3bPvRwD8tnbJx3yO1LCkKk/MCb5IMwPtPtA4u230m JeGCLCaENcrO9Xh7ZK3coNQ70YPqmS9fp8UzyTRqqTO69j/JZVGdwN/ROapfF3w= X-Google-Smtp-Source: AGHT+IHUpsDbIZseD47umSPALrSWJE4Ica37wHhFexbrDeyAnw0SFI4kH9oAcH57CWliXN0EbWHjZA== X-Received: by 2002:a05:6a20:2583:b0:1c2:8f07:579e with SMTP id adf61e73a8af0-1c461a06f68mr1518550637.52.1721805131150; Wed, 24 Jul 2024 00:12:11 -0700 (PDT) Received: from localhost.localdomain ([117.136.120.216]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7a2163d7d3csm4096841a12.13.2024.07.24.00.12.07 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Jul 2024 00:12:10 -0700 (PDT) From: Yafang Shao To: miklos@szeredi.hu Cc: linux-fsdevel@vger.kernel.org, Yafang Shao Subject: [RFC PATCH 1/2] fuse: Add "timeout" sysfs attribute for each fuse connection Date: Wed, 24 Jul 2024 15:11:55 +0800 Message-Id: <20240724071156.97188-2-laoar.shao@gmail.com> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20240724071156.97188-1-laoar.shao@gmail.com> References: <20240724071156.97188-1-laoar.shao@gmail.com> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 A dedicated "timeout" sysfs attribute has been introduced for each fuse connection, empowering users to manage and control the timeout duration for individual connections. This is a preparation for the followup patch. Signed-off-by: Yafang Shao --- fs/fuse/control.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++- fs/fuse/fuse_i.h | 5 ++++- 2 files changed, 53 insertions(+), 2 deletions(-) diff --git a/fs/fuse/control.c b/fs/fuse/control.c index 97ac994ff78f..247fbec72cda 100644 --- a/fs/fuse/control.c +++ b/fs/fuse/control.c @@ -180,6 +180,44 @@ static ssize_t fuse_conn_congestion_threshold_write(struct file *file, return ret; } +static ssize_t fuse_conn_timeout_read(struct file *file, + char __user *buf, size_t len, + loff_t *ppos) +{ + struct fuse_conn *fc; + u32 val; + + fc = fuse_ctl_file_conn_get(file); + if (!fc) + return 0; + + val = READ_ONCE(fc->timeout); + fuse_conn_put(fc); + return fuse_conn_limit_read(file, buf, len, ppos, val); +} + +static ssize_t fuse_conn_timeout_write(struct file *file, + const char __user *buf, + size_t count, loff_t *ppos) +{ + struct fuse_conn *fc; + ssize_t ret; + u32 val; + + ret = fuse_conn_limit_write(file, buf, count, ppos, &val, + 3600); + if (ret <= 0) + goto out; + fc = fuse_ctl_file_conn_get(file); + if (!fc) + goto out; + + WRITE_ONCE(fc->timeout, val); + fuse_conn_put(fc); +out: + return ret; +} + static const struct file_operations fuse_ctl_abort_ops = { .open = nonseekable_open, .write = fuse_conn_abort_write, @@ -206,6 +244,13 @@ static const struct file_operations fuse_conn_congestion_threshold_ops = { .llseek = no_llseek, }; +static const struct file_operations fuse_conn_timeout_ops = { + .open = nonseekable_open, + .read = fuse_conn_timeout_read, + .write = fuse_conn_timeout_write, + .llseek = no_llseek, +}; + static struct dentry *fuse_ctl_add_dentry(struct dentry *parent, struct fuse_conn *fc, const char *name, @@ -274,7 +319,10 @@ int fuse_ctl_add_conn(struct fuse_conn *fc) 1, NULL, &fuse_conn_max_background_ops) || !fuse_ctl_add_dentry(parent, fc, "congestion_threshold", S_IFREG | 0600, 1, NULL, - &fuse_conn_congestion_threshold_ops)) + &fuse_conn_congestion_threshold_ops) || + !fuse_ctl_add_dentry(parent, fc, "timeout", + S_IFREG | 0600, 1, NULL, + &fuse_conn_timeout_ops)) goto err; return 0; diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h index f23919610313..367601bf7285 100644 --- a/fs/fuse/fuse_i.h +++ b/fs/fuse/fuse_i.h @@ -45,7 +45,7 @@ #define FUSE_NAME_MAX 1024 /** Number of dentries for each connection in the control filesystem */ -#define FUSE_CTL_NUM_DENTRIES 5 +#define FUSE_CTL_NUM_DENTRIES 6 /** List of active connections */ extern struct list_head fuse_conn_list; @@ -917,6 +917,9 @@ struct fuse_conn { /** IDR for backing files ids */ struct idr backing_files_map; #endif + + /* fuse request timeout */ + u32 timeout; }; /* From patchwork Wed Jul 24 07:11:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Yafang Shao X-Patchwork-Id: 13740638 Received: from mail-oa1-f48.google.com (mail-oa1-f48.google.com [209.85.160.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 66C60D29E for ; Wed, 24 Jul 2024 07:12:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721805136; cv=none; b=An0E+M6650/m7D5ZqToyD+26w/8brnYzmnR+6LgJKu8CQ+i9QjF1o2fpJ3n17/KlXB2yuWtduywPa2btGG7gIclxpK0Ytkde5SXcnTyr1x6VdUvd02CgJSUlTD7TkvocjVbjtDhlfN5hMOlXJVoZbbhou+GpLshs7Hs55rWgRRs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721805136; c=relaxed/simple; bh=aLiteNOyJdfK24UMxr9OGDyQfqCn+haD8SUOtGcYBj0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=MIfXkN7HsLY9PbR6Tzu3Amd1srWFDfXsoTvFb6BQZVyhGqpWOgC5q8RZO2nEGDAocyEFWLkhS7f+zr62JsX60yoYrMBp9SBIBh6QRgi4d5qNM1go5M6kOnhJ2Q8scxTmmwnxhSrs8IgOvBJfioJNVLpe9CbpXD6/o1knHC203vg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=bcAd1x7q; arc=none smtp.client-ip=209.85.160.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bcAd1x7q" Received: by mail-oa1-f48.google.com with SMTP id 586e51a60fabf-260f863109cso3358744fac.1 for ; Wed, 24 Jul 2024 00:12:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1721805134; x=1722409934; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vefBD6huXZ2/Z9w18nUiA7vgWJOXvgETJXelGXjG7eQ=; b=bcAd1x7qGt0v02U4U4koyyqpQaxWaMB8hQ+6NsrMhrw4xVBu1ADDo4HUi82ji99HX2 ouuGjtsnfrUUj6HpG5hIHbMS7A5gznrF9OO8IwrWsf5+vHz6TZy59AZanfFctNalGDHs Ts/CrOSivNFVHd1kH8ATsrSMScoNEIQLYWyM3DMlYXD1EbfYrzu+Or2W3cxGSNeDbVku qH0lj1h9HekcJm/2GjXS/JAKUlrfmnzOfS8ScE2abisafoJEyX+jMcmmdFWk8RYT7A1+ F8Jq7EUrYH77TmFGzo93EobE263dAhoJBDzzXzVRRiBnOeEMOym7Ju1gFmtBKls5b0bJ Hokw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721805134; x=1722409934; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vefBD6huXZ2/Z9w18nUiA7vgWJOXvgETJXelGXjG7eQ=; b=QGn4hb7IJhu+CbKK/JT77P/rU98MNtUp1lIYQj9ONeSgKQlXO2Vv83f68qScgrh1sH X9qK44R65JhaX6ggIagiMUB1Yql54A52DHEE4YOp0da1d419OSteiLBQl8y6oRKl2MBO vba56cXRaTXZIlVp2zVsIsMJxTZJejIFxgHvCn64uVJ+/PiX36stH39tzsiOCvIjbSEn 6B587xyT10vAzpwzzTjZSaIryaOU42RUmBi25LefmPvRUaPyZLNougTL8FzclIFDjRqi toU7KxtJ0BNt2WJRclLewq3ThNDmFeyx+tmKrwJLQ7yQe2ryUciP4okMI767RfNaN9Y1 +tuQ== X-Gm-Message-State: AOJu0YyFG53FKbTtKPlSnZOWIowlf6akmLcanWOuzQiTUA+7m9cLUqjF 5l5nXrVW3/Yi/QdtBS5ASpvCA2iN12J6QICih9Pcu3B2GIfzv3TcGu2QonAHtMM= X-Google-Smtp-Source: AGHT+IF3BYCQMqxESTgXz8ovBVrE8boRyJ5eqa1Z7Vf89ke2dqx/GrxeA5q/f0KpCDb+1uGazycjCg== X-Received: by 2002:a05:6870:ac2b:b0:260:e48b:62d8 with SMTP id 586e51a60fabf-2648cb0aaffmr1271111fac.29.1721805134317; Wed, 24 Jul 2024 00:12:14 -0700 (PDT) Received: from localhost.localdomain ([117.136.120.216]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7a2163d7d3csm4096841a12.13.2024.07.24.00.12.12 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Jul 2024 00:12:13 -0700 (PDT) From: Yafang Shao To: miklos@szeredi.hu Cc: linux-fsdevel@vger.kernel.org, Yafang Shao Subject: [RFC PATCH 2/2] fuse: Enhance each fuse connection with timeout support Date: Wed, 24 Jul 2024 15:11:56 +0800 Message-Id: <20240724071156.97188-3-laoar.shao@gmail.com> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20240724071156.97188-1-laoar.shao@gmail.com> References: <20240724071156.97188-1-laoar.shao@gmail.com> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In our experience with fuse.hdfs, we encountered a challenge where, if the HDFS server encounters an issue, the fuse.hdfs daemon—responsible for sending requests to the HDFS server—can get stuck indefinitely. Consequently, access to the fuse.hdfs directory becomes unresponsive. The current workaround involves manually aborting the fuse connection, which is unreliable in automatically addressing the abnormal connection issue. To alleviate this pain point, we have implemented a timeout mechanism that automatically handles such abnormal cases, thereby streamlining the process and enhancing reliability. The timeout value is configurable by the user, allowing them to tailor it according to their specific workload requirements. Signed-off-by: Yafang Shao --- fs/fuse/dev.c | 57 +++++++++++++++++++++++++++++++++++++++++------- fs/fuse/fuse_i.h | 2 ++ 2 files changed, 51 insertions(+), 8 deletions(-) diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c index 9eb191b5c4de..ff9c55bcfb3d 100644 --- a/fs/fuse/dev.c +++ b/fs/fuse/dev.c @@ -369,10 +369,27 @@ static void request_wait_answer(struct fuse_req *req) if (!fc->no_interrupt) { /* Any signal may interrupt this */ - err = wait_event_interruptible(req->waitq, - test_bit(FR_FINISHED, &req->flags)); - if (!err) - return; + if (!fc->timeout) { + err = wait_event_interruptible(req->waitq, + test_bit(FR_FINISHED, &req->flags)); + if (!err) + return; + } else { + err = wait_event_interruptible_timeout(req->waitq, + test_bit(FR_FINISHED, &req->flags), + (long)fc->timeout * HZ); + if (err > 0) + return; + + /* timeout */ + if (!err) { + req->out.h.error = -EAGAIN; + set_bit(FR_TIMEOUT, &req->flags); + /* matches barrier in fuse_dev_do_write() */ + smp_mb__after_atomic(); + return; + } + } set_bit(FR_INTERRUPTED, &req->flags); /* matches barrier in fuse_dev_do_read() */ @@ -383,10 +400,27 @@ static void request_wait_answer(struct fuse_req *req) if (!test_bit(FR_FORCE, &req->flags)) { /* Only fatal signals may interrupt this */ - err = wait_event_killable(req->waitq, - test_bit(FR_FINISHED, &req->flags)); - if (!err) - return; + if (!fc->timeout) { + err = wait_event_killable(req->waitq, + test_bit(FR_FINISHED, &req->flags)); + if (!err) + return; + } else { + err = wait_event_killable_timeout(req->waitq, + test_bit(FR_FINISHED, &req->flags), + (long)fc->timeout * HZ); + if (err > 0) + return; + + /* timeout */ + if (!err) { + req->out.h.error = -EAGAIN; + set_bit(FR_TIMEOUT, &req->flags); + /* matches barrier in fuse_dev_do_write() */ + smp_mb__after_atomic(); + return; + } + } spin_lock(&fiq->lock); /* Request is not yet in userspace, bail out */ @@ -1951,6 +1985,13 @@ static ssize_t fuse_dev_do_write(struct fuse_dev *fud, goto copy_finish; } + /* matches barrier in request_wait_answer() */ + smp_mb__after_atomic(); + if (test_and_clear_bit(FR_TIMEOUT, &req->flags)) { + spin_unlock(&fpq->lock); + goto copy_finish; + } + /* Is it an interrupt reply ID? */ if (oh.unique & FUSE_INT_REQ_BIT) { __fuse_get_request(req); diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h index 367601bf7285..c1467eb8c2e9 100644 --- a/fs/fuse/fuse_i.h +++ b/fs/fuse/fuse_i.h @@ -375,6 +375,7 @@ struct fuse_io_priv { * FR_FINISHED: request is finished * FR_PRIVATE: request is on private list * FR_ASYNC: request is asynchronous + * FR_TIMEOUT: request is timeout */ enum fuse_req_flag { FR_ISREPLY, @@ -389,6 +390,7 @@ enum fuse_req_flag { FR_FINISHED, FR_PRIVATE, FR_ASYNC, + FR_TIMEOUT, }; /**