From patchwork Mon Feb 14 13:58:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Greg Kurz X-Patchwork-Id: 12745732 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C1748C433F5 for ; Mon, 14 Feb 2022 15:29:44 +0000 (UTC) Received: from localhost ([::1]:40838 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nJdIN-00074S-KP for qemu-devel@archiver.kernel.org; Mon, 14 Feb 2022 10:29:43 -0500 Received: from eggs.gnu.org ([209.51.188.92]:40154) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nJbt8-0005fC-Hj for qemu-devel@nongnu.org; Mon, 14 Feb 2022 08:59:35 -0500 Received: from us-smtp-delivery-44.mimecast.com ([207.211.30.44]:35400) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nJbt1-00020e-V5 for qemu-devel@nongnu.org; Mon, 14 Feb 2022 08:59:34 -0500 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-653-fG9nwed8PmGaa5ZDna_Efw-1; Mon, 14 Feb 2022 08:59:15 -0500 X-MC-Unique: fG9nwed8PmGaa5ZDna_Efw-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 6D217802924; Mon, 14 Feb 2022 13:59:14 +0000 (UTC) Received: from bahia.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id B71D87DE22; Mon, 14 Feb 2022 13:58:44 +0000 (UTC) From: Greg Kurz To: qemu-devel@nongnu.org Subject: [PATCH v5 1/3] virtiofsd: Add support for FUSE_SYNCFS request with announce_submounts Date: Mon, 14 Feb 2022 14:58:18 +0100 Message-Id: <20220214135820.43897-2-groug@kaod.org> In-Reply-To: <20220214135820.43897-1-groug@kaod.org> References: <20220214135820.43897-1-groug@kaod.org> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kaod.org Received-SPF: softfail client-ip=207.211.30.44; envelope-from=groug@kaod.org; helo=us-smtp-delivery-44.mimecast.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_SOFTFAIL=0.665, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Sebastian Hasler , "Dr. David Alan Gilbert" , Greg Kurz , virtio-fs@redhat.com, Stefan Hajnoczi , Vivek Goyal Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Honor the expected behavior of syncfs() to synchronously flush all data and metadata to disk on linux systems. If virtiofsd is started with '-o announce_submounts', the client is expected to send a FUSE_SYNCFS request for each individual submount. In this case, we just create a new file descriptor on the submount inode with lo_inode_open(), call syncfs() on it and close it. The intermediary file is needed because O_PATH descriptors aren't backed by an actual file and syncfs() would fail with EBADF. If virtiofsd is started without '-o announce_submounts' or if the client doesn't have the FUSE_CAP_SUBMOUNTS capability, the client only sends a single FUSE_SYNCFS request for the root inode. The server would thus need to track submounts internally and call syncfs() on each of them. This will be implemented later. Note that syncfs() might suffer from a time penalty if the submounts are being hammered by some unrelated workload on the host. The only solution to prevent that is to avoid shared mounts. Signed-off-by: Greg Kurz --- tools/virtiofsd/fuse_lowlevel.c | 11 +++++++ tools/virtiofsd/fuse_lowlevel.h | 13 ++++++++ tools/virtiofsd/passthrough_ll.c | 45 +++++++++++++++++++++++++++ tools/virtiofsd/passthrough_seccomp.c | 1 + 4 files changed, 70 insertions(+) diff --git a/tools/virtiofsd/fuse_lowlevel.c b/tools/virtiofsd/fuse_lowlevel.c index e4679c73abc2..e02d8b25a5f6 100644 --- a/tools/virtiofsd/fuse_lowlevel.c +++ b/tools/virtiofsd/fuse_lowlevel.c @@ -1876,6 +1876,16 @@ static void do_lseek(fuse_req_t req, fuse_ino_t nodeid, } } +static void do_syncfs(fuse_req_t req, fuse_ino_t nodeid, + struct fuse_mbuf_iter *iter) +{ + if (req->se->op.syncfs) { + req->se->op.syncfs(req, nodeid); + } else { + fuse_reply_err(req, ENOSYS); + } +} + static void do_init(fuse_req_t req, fuse_ino_t nodeid, struct fuse_mbuf_iter *iter) { @@ -2280,6 +2290,7 @@ static struct { [FUSE_RENAME2] = { do_rename2, "RENAME2" }, [FUSE_COPY_FILE_RANGE] = { do_copy_file_range, "COPY_FILE_RANGE" }, [FUSE_LSEEK] = { do_lseek, "LSEEK" }, + [FUSE_SYNCFS] = { do_syncfs, "SYNCFS" }, }; #define FUSE_MAXOP (sizeof(fuse_ll_ops) / sizeof(fuse_ll_ops[0])) diff --git a/tools/virtiofsd/fuse_lowlevel.h b/tools/virtiofsd/fuse_lowlevel.h index c55c0ca2fc1c..b889dae4de0e 100644 --- a/tools/virtiofsd/fuse_lowlevel.h +++ b/tools/virtiofsd/fuse_lowlevel.h @@ -1226,6 +1226,19 @@ struct fuse_lowlevel_ops { */ void (*lseek)(fuse_req_t req, fuse_ino_t ino, off_t off, int whence, struct fuse_file_info *fi); + + /** + * Synchronize file system content + * + * If this request is answered with an error code of ENOSYS, + * this is treated as success and future calls to syncfs() will + * succeed automatically without being sent to the filesystem + * process. + * + * @param req request handle + * @param ino the inode number + */ + void (*syncfs)(fuse_req_t req, fuse_ino_t ino); }; /** diff --git a/tools/virtiofsd/passthrough_ll.c b/tools/virtiofsd/passthrough_ll.c index b3d0674f6d2f..2bf5d40df531 100644 --- a/tools/virtiofsd/passthrough_ll.c +++ b/tools/virtiofsd/passthrough_ll.c @@ -3357,6 +3357,50 @@ static void lo_lseek(fuse_req_t req, fuse_ino_t ino, off_t off, int whence, } } +static int lo_do_syncfs(struct lo_data *lo, struct lo_inode *inode) +{ + int fd, ret = 0; + + fuse_log(FUSE_LOG_DEBUG, "lo_do_syncfs(ino=%" PRIu64 ")\n", + inode->fuse_ino); + + fd = lo_inode_open(lo, inode, O_RDONLY); + if (fd < 0) { + return -fd; + } + + if (syncfs(fd) < 0) { + ret = errno; + } + + close(fd); + return ret; +} + +static void lo_syncfs(fuse_req_t req, fuse_ino_t ino) +{ + struct lo_data *lo = lo_data(req); + int err; + + if (lo->announce_submounts) { + struct lo_inode *inode; + + inode = lo_inode(req, ino); + if (!inode) { + fuse_reply_err(req, EBADF); + return; + } + + err = lo_do_syncfs(lo, inode); + lo_inode_put(lo, &inode); + } else { + /* Requires the sever to track submounts. Not implemented yet */ + err = ENOSYS; + } + + fuse_reply_err(req, err); +} + static void lo_destroy(void *userdata) { struct lo_data *lo = (struct lo_data *)userdata; @@ -3417,6 +3461,7 @@ static struct fuse_lowlevel_ops lo_oper = { .copy_file_range = lo_copy_file_range, #endif .lseek = lo_lseek, + .syncfs = lo_syncfs, .destroy = lo_destroy, }; diff --git a/tools/virtiofsd/passthrough_seccomp.c b/tools/virtiofsd/passthrough_seccomp.c index a3ce9f898d2d..3e9d6181dc69 100644 --- a/tools/virtiofsd/passthrough_seccomp.c +++ b/tools/virtiofsd/passthrough_seccomp.c @@ -108,6 +108,7 @@ static const int syscall_allowlist[] = { SCMP_SYS(set_robust_list), SCMP_SYS(setxattr), SCMP_SYS(symlinkat), + SCMP_SYS(syncfs), SCMP_SYS(time), /* Rarely needed, except on static builds */ SCMP_SYS(tgkill), SCMP_SYS(unlinkat), From patchwork Mon Feb 14 13:58:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Greg Kurz X-Patchwork-Id: 12745736 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0ACCEC433F5 for ; Mon, 14 Feb 2022 15:34:10 +0000 (UTC) Received: from localhost ([::1]:49122 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nJdMf-0004JL-Ov for qemu-devel@archiver.kernel.org; Mon, 14 Feb 2022 10:34:09 -0500 Received: from eggs.gnu.org ([209.51.188.92]:40342) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nJbtc-0005zT-7a for qemu-devel@nongnu.org; Mon, 14 Feb 2022 09:00:04 -0500 Received: from us-smtp-delivery-44.mimecast.com ([207.211.30.44]:24260) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nJbtW-00023y-UB for qemu-devel@nongnu.org; Mon, 14 Feb 2022 09:00:01 -0500 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-569-GsF5Hoj_NOSACRQYi6lHDA-1; Mon, 14 Feb 2022 08:59:52 -0500 X-MC-Unique: GsF5Hoj_NOSACRQYi6lHDA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 8C573802924; Mon, 14 Feb 2022 13:59:51 +0000 (UTC) Received: from bahia.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id DBEE66ABB9; Mon, 14 Feb 2022 13:59:14 +0000 (UTC) From: Greg Kurz To: qemu-devel@nongnu.org Subject: [PATCH v5 2/3] virtiofsd: Track submounts Date: Mon, 14 Feb 2022 14:58:19 +0100 Message-Id: <20220214135820.43897-3-groug@kaod.org> In-Reply-To: <20220214135820.43897-1-groug@kaod.org> References: <20220214135820.43897-1-groug@kaod.org> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kaod.org Received-SPF: softfail client-ip=207.211.30.44; envelope-from=groug@kaod.org; helo=us-smtp-delivery-44.mimecast.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_SOFTFAIL=0.665, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Sebastian Hasler , "Dr. David Alan Gilbert" , Greg Kurz , virtio-fs@redhat.com, Stefan Hajnoczi , Vivek Goyal Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" If Support for FUSE_SYNCFS requires the server to track submounts that may exist under the shared directory. lo_do_lookup() already knows how to detect them : it is a directory with a different device ID or mount ID than its parent. Use the same logic and record the information under the lo_inode structure. Signed-off-by: Greg Kurz --- tools/virtiofsd/passthrough_ll.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/tools/virtiofsd/passthrough_ll.c b/tools/virtiofsd/passthrough_ll.c index 2bf5d40df531..e94c4e6f8635 100644 --- a/tools/virtiofsd/passthrough_ll.c +++ b/tools/virtiofsd/passthrough_ll.c @@ -118,6 +118,7 @@ struct lo_inode { GHashTable *posix_locks; /* protected by lo_inode->plock_mutex */ mode_t filetype; + bool is_submount; }; struct lo_cred { @@ -1017,6 +1018,7 @@ static int lo_do_lookup(fuse_req_t req, fuse_ino_t parent, const char *name, struct lo_data *lo = lo_data(req); struct lo_inode *inode = NULL; struct lo_inode *dir = lo_inode(req, parent); + bool is_submount; if (inodep) { *inodep = NULL; /* in case there is an error */ @@ -1051,8 +1053,10 @@ static int lo_do_lookup(fuse_req_t req, fuse_ino_t parent, const char *name, goto out_err; } - if (S_ISDIR(e->attr.st_mode) && lo->announce_submounts && - (e->attr.st_dev != dir->key.dev || mnt_id != dir->key.mnt_id)) { + is_submount = S_ISDIR(e->attr.st_mode) && + (e->attr.st_dev != dir->key.dev || mnt_id != dir->key.mnt_id); + + if (is_submount && lo->announce_submounts) { e->attr_flags |= FUSE_ATTR_SUBMOUNT; } @@ -1079,6 +1083,7 @@ static int lo_do_lookup(fuse_req_t req, fuse_ino_t parent, const char *name, inode->key.ino = e->attr.st_ino; inode->key.dev = e->attr.st_dev; inode->key.mnt_id = mnt_id; + inode->is_submount = is_submount; if (lo->posix_lock) { pthread_mutex_init(&inode->plock_mutex, NULL); inode->posix_locks = g_hash_table_new_full( @@ -1100,8 +1105,9 @@ static int lo_do_lookup(fuse_req_t req, fuse_ino_t parent, const char *name, lo_inode_put(lo, &dir); - fuse_log(FUSE_LOG_DEBUG, " %lli/%s -> %lli\n", (unsigned long long)parent, - name, (unsigned long long)e->ino); + fuse_log(FUSE_LOG_DEBUG, " %lli/%s -> %lli%s\n", + (unsigned long long) parent, name, (unsigned long long) e->ino, + is_submount ? " (submount)" : ""); return 0; From patchwork Mon Feb 14 13:58:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Greg Kurz X-Patchwork-Id: 12745737 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6CB17C433EF for ; Mon, 14 Feb 2022 15:37:22 +0000 (UTC) Received: from localhost ([::1]:58152 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nJdPl-0001zo-Bm for qemu-devel@archiver.kernel.org; Mon, 14 Feb 2022 10:37:21 -0500 Received: from eggs.gnu.org ([209.51.188.92]:40624) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nJbu4-0006mD-3D for qemu-devel@nongnu.org; Mon, 14 Feb 2022 09:00:32 -0500 Received: from us-smtp-delivery-44.mimecast.com ([205.139.111.44]:32826) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nJbu2-0002QD-6f for qemu-devel@nongnu.org; Mon, 14 Feb 2022 09:00:31 -0500 Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-108-iYdiRZBtNpmmfswym7P7iA-1; Mon, 14 Feb 2022 09:00:25 -0500 X-MC-Unique: iYdiRZBtNpmmfswym7P7iA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 9E261189DF42; Mon, 14 Feb 2022 14:00:24 +0000 (UTC) Received: from bahia.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id E83266ABB9; Mon, 14 Feb 2022 13:59:51 +0000 (UTC) From: Greg Kurz To: qemu-devel@nongnu.org Subject: [PATCH v5 3/3] virtiofsd: Add support for FUSE_SYNCFS request without announce_submounts Date: Mon, 14 Feb 2022 14:58:20 +0100 Message-Id: <20220214135820.43897-4-groug@kaod.org> In-Reply-To: <20220214135820.43897-1-groug@kaod.org> References: <20220214135820.43897-1-groug@kaod.org> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=groug@kaod.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: kaod.org Received-SPF: softfail client-ip=205.139.111.44; envelope-from=groug@kaod.org; helo=us-smtp-delivery-44.mimecast.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_SOFTFAIL=0.665, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Sebastian Hasler , "Dr. David Alan Gilbert" , Greg Kurz , virtio-fs@redhat.com, Stefan Hajnoczi , Vivek Goyal Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This adds the missing bits to support FUSE_SYNCFS in the case submounts aren't announced to the client. Iterate over all inodes and call syncfs() on the ones marked as submounts. Since syncfs() can block for an indefinite time, we cannot call it with lo->mutex held as it would prevent the server to process other requests. This is thus broken down in two steps. First build a list of submounts with lo->mutex held, drop the mutex and finally process the list. A reference is taken on the inodes to ensure they don't go away when lo->mutex is dropped. Signed-off-by: Greg Kurz --- tools/virtiofsd/passthrough_ll.c | 38 ++++++++++++++++++++++++++++++-- 1 file changed, 36 insertions(+), 2 deletions(-) diff --git a/tools/virtiofsd/passthrough_ll.c b/tools/virtiofsd/passthrough_ll.c index e94c4e6f8635..7ce944bfe2a0 100644 --- a/tools/virtiofsd/passthrough_ll.c +++ b/tools/virtiofsd/passthrough_ll.c @@ -3400,8 +3400,42 @@ static void lo_syncfs(fuse_req_t req, fuse_ino_t ino) err = lo_do_syncfs(lo, inode); lo_inode_put(lo, &inode); } else { - /* Requires the sever to track submounts. Not implemented yet */ - err = ENOSYS; + g_autoptr(GSList) submount_list = NULL; + GSList *elem; + GHashTableIter iter; + gpointer key, value; + + pthread_mutex_lock(&lo->mutex); + + g_hash_table_iter_init(&iter, lo->inodes); + while (g_hash_table_iter_next(&iter, &key, &value)) { + struct lo_inode *inode = value; + + if (inode->is_submount) { + g_atomic_int_inc(&inode->refcount); + submount_list = g_slist_prepend(submount_list, inode); + } + } + + pthread_mutex_unlock(&lo->mutex); + + /* The root inode is always present and not tracked in the hash table */ + err = lo_do_syncfs(lo, &lo->root); + + for (elem = submount_list; elem; elem = g_slist_next(elem)) { + struct lo_inode *inode = elem->data; + int r; + + r = lo_do_syncfs(lo, inode); + if (r) { + /* + * Try to sync as much as possible. Only one error can be + * reported to the client though, arbitrarily the last one. + */ + err = r; + } + lo_inode_put(lo, &inode); + } } fuse_reply_err(req, err);