From patchwork Mon Dec 27 12:54:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12699741 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56CDCC4332F for ; Mon, 27 Dec 2021 12:54:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236767AbhL0Myt (ORCPT ); Mon, 27 Dec 2021 07:54:49 -0500 Received: from out4436.biz.mail.alibaba.com ([47.88.44.36]:37412 "EHLO out4436.biz.mail.alibaba.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233867AbhL0Mys (ORCPT ); Mon, 27 Dec 2021 07:54:48 -0500 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R311e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04394;MF=jefflexu@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0V.wXYT2_1640609685; Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0V.wXYT2_1640609685) by smtp.aliyun-inc.com(127.0.0.1); Mon, 27 Dec 2021 20:54:46 +0800 From: Jeffle Xu To: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org Cc: linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: [PATCH v1 01/23] cachefiles: add cachefiles_demand devnode Date: Mon, 27 Dec 2021 20:54:22 +0800 Message-Id: <20211227125444.21187-2-jefflexu@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211227125444.21187-1-jefflexu@linux.alibaba.com> References: <20211227125444.21187-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org fscache/cachefiles used to serve as a local cache for remote fs. The following patches will introduce a new use case, in which local read-only fs could implement demand reading with fscache. By then the user daemon needs to read and poll on the devnode, and thus the original cachefiles devnode can't be reused in this case. Thus create a new devnode specifically for the new mode. The following patches will add more file_operations. Signed-off-by: Jeffle Xu --- fs/cachefiles/daemon.c | 8 ++++++++ fs/cachefiles/internal.h | 1 + fs/cachefiles/main.c | 12 ++++++++++++ 3 files changed, 21 insertions(+) diff --git a/fs/cachefiles/daemon.c b/fs/cachefiles/daemon.c index 40a792421fc1..871f1e0f423d 100644 --- a/fs/cachefiles/daemon.c +++ b/fs/cachefiles/daemon.c @@ -56,6 +56,14 @@ const struct file_operations cachefiles_daemon_fops = { .llseek = noop_llseek, }; +const struct file_operations cachefiles_demand_fops = { + .owner = THIS_MODULE, + .open = cachefiles_daemon_open, + .release = cachefiles_daemon_release, + .write = cachefiles_daemon_write, + .llseek = noop_llseek, +}; + struct cachefiles_daemon_cmd { char name[8]; int (*handler)(struct cachefiles_cache *cache, char *args); diff --git a/fs/cachefiles/internal.h b/fs/cachefiles/internal.h index 421423819d63..e0ed811d628d 100644 --- a/fs/cachefiles/internal.h +++ b/fs/cachefiles/internal.h @@ -145,6 +145,7 @@ extern int cachefiles_has_space(struct cachefiles_cache *cache, * daemon.c */ extern const struct file_operations cachefiles_daemon_fops; +extern const struct file_operations cachefiles_demand_fops; /* * error_inject.c diff --git a/fs/cachefiles/main.c b/fs/cachefiles/main.c index 3f369c6f816d..0a423274d283 100644 --- a/fs/cachefiles/main.c +++ b/fs/cachefiles/main.c @@ -39,6 +39,12 @@ static struct miscdevice cachefiles_dev = { .fops = &cachefiles_daemon_fops, }; +static struct miscdevice cachefiles_demand_dev = { + .minor = MISC_DYNAMIC_MINOR, + .name = "cachefiles_demand", + .fops = &cachefiles_demand_fops, +}; + /* * initialise the fs caching module */ @@ -52,6 +58,9 @@ static int __init cachefiles_init(void) ret = misc_register(&cachefiles_dev); if (ret < 0) goto error_dev; + ret = misc_register(&cachefiles_demand_dev); + if (ret < 0) + goto error_demand_dev; /* create an object jar */ ret = -ENOMEM; @@ -68,6 +77,8 @@ static int __init cachefiles_init(void) return 0; error_object_jar: + misc_deregister(&cachefiles_demand_dev); +error_demand_dev: misc_deregister(&cachefiles_dev); error_dev: cachefiles_unregister_error_injection(); @@ -86,6 +97,7 @@ static void __exit cachefiles_exit(void) pr_info("Unloading\n"); kmem_cache_destroy(cachefiles_object_jar); + misc_deregister(&cachefiles_demand_dev); misc_deregister(&cachefiles_dev); cachefiles_unregister_error_injection(); } From patchwork Mon Dec 27 12:54:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12699742 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE369C433F5 for ; Mon, 27 Dec 2021 12:54:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236776AbhL0Myw (ORCPT ); Mon, 27 Dec 2021 07:54:52 -0500 Received: from out30-44.freemail.mail.aliyun.com ([115.124.30.44]:40946 "EHLO out30-44.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236770AbhL0Myu (ORCPT ); Mon, 27 Dec 2021 07:54:50 -0500 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R161e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04400;MF=jefflexu@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0V.xJoPB_1640609686; Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0V.xJoPB_1640609686) by smtp.aliyun-inc.com(127.0.0.1); Mon, 27 Dec 2021 20:54:47 +0800 From: Jeffle Xu To: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org Cc: linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: [PATCH v1 02/23] cachefiles: add mode command to distinguish modes Date: Mon, 27 Dec 2021 20:54:23 +0800 Message-Id: <20211227125444.21187-3-jefflexu@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211227125444.21187-1-jefflexu@linux.alibaba.com> References: <20211227125444.21187-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Add one flag bit to distinguish the new introduced demand-read mode from the original mode. User daemon could set the specified mode with 'mode' command. If user daemon doesn't ever explicitly set the mode, then the behaviour is the same with that prior this patch, i.e. cachefiles serves as the local cache for remote fs by default. Signed-off-by: Jeffle Xu --- fs/cachefiles/daemon.c | 32 ++++++++++++++++++++++++++++++++ fs/cachefiles/internal.h | 1 + 2 files changed, 33 insertions(+) diff --git a/fs/cachefiles/daemon.c b/fs/cachefiles/daemon.c index 871f1e0f423d..892a9bdba53f 100644 --- a/fs/cachefiles/daemon.c +++ b/fs/cachefiles/daemon.c @@ -41,6 +41,7 @@ static int cachefiles_daemon_dir(struct cachefiles_cache *, char *); static int cachefiles_daemon_inuse(struct cachefiles_cache *, char *); static int cachefiles_daemon_secctx(struct cachefiles_cache *, char *); static int cachefiles_daemon_tag(struct cachefiles_cache *, char *); +static int cachefiles_daemon_mode(struct cachefiles_cache *, char *); static int cachefiles_daemon_bind(struct cachefiles_cache *, char *); static void cachefiles_daemon_unbind(struct cachefiles_cache *); @@ -83,6 +84,7 @@ static const struct cachefiles_daemon_cmd cachefiles_daemon_cmds[] = { { "inuse", cachefiles_daemon_inuse }, { "secctx", cachefiles_daemon_secctx }, { "tag", cachefiles_daemon_tag }, + { "mode", cachefiles_daemon_mode }, { "", NULL } }; @@ -671,6 +673,36 @@ static int cachefiles_daemon_inuse(struct cachefiles_cache *cache, char *args) return -EINVAL; } +/* + * Set the cache mode + * - command: "mode cache|demand" + */ +static int cachefiles_daemon_mode(struct cachefiles_cache *cache, char *args) +{ + _enter(",%s", args); + + if (test_bit(CACHEFILES_READY, &cache->flags)) { + pr_err("Cache already started\n"); + return -EINVAL; + } + + if (!*args) { + pr_err("Empty mode specified\n"); + return -EINVAL; + } + + if (!strncmp(args, "cache", strlen("cache"))) { + clear_bit(CACHEFILES_DEMAND_MODE, &cache->flags); + } else if (!strncmp(args, "demand", strlen("demand"))) { + set_bit(CACHEFILES_DEMAND_MODE, &cache->flags); + } else { + pr_err("Invalid mode specified\n"); + return -EINVAL; + } + + return 0; +} + /* * Bind a directory as a cache */ diff --git a/fs/cachefiles/internal.h b/fs/cachefiles/internal.h index e0ed811d628d..a8e6500889d7 100644 --- a/fs/cachefiles/internal.h +++ b/fs/cachefiles/internal.h @@ -98,6 +98,7 @@ struct cachefiles_cache { #define CACHEFILES_DEAD 1 /* T if cache dead */ #define CACHEFILES_CULLING 2 /* T if cull engaged */ #define CACHEFILES_STATE_CHANGED 3 /* T if state changed (poll trigger) */ +#define CACHEFILES_DEMAND_MODE 4 /* T if works in demand read mode for read-only fs */ char *rootdirname; /* name of cache root directory */ char *secctx; /* LSM security context */ char *tag; /* cache binding tag */ From patchwork Mon Dec 27 12:54:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12699743 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 850FCC433FE for ; Mon, 27 Dec 2021 12:54:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236770AbhL0Myx (ORCPT ); Mon, 27 Dec 2021 07:54:53 -0500 Received: from out30-56.freemail.mail.aliyun.com ([115.124.30.56]:39213 "EHLO out30-56.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236771AbhL0Myu (ORCPT ); Mon, 27 Dec 2021 07:54:50 -0500 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R791e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04394;MF=jefflexu@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0V.w7GWF_1640609687; Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0V.w7GWF_1640609687) by smtp.aliyun-inc.com(127.0.0.1); Mon, 27 Dec 2021 20:54:48 +0800 From: Jeffle Xu To: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org Cc: linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: [PATCH v1 03/23] cachefiles: detect backing file size in demand-read mode Date: Mon, 27 Dec 2021 20:54:24 +0800 Message-Id: <20211227125444.21187-4-jefflexu@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211227125444.21187-1-jefflexu@linux.alibaba.com> References: <20211227125444.21187-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org When upper read-only fs uses fscache for demand reading, it has no idea on the size of the backed file. (You need to input the file size as the @object_size parameter when calling fscache_acquire_cookie().) In this using scenario, user daemon is responsible for preparing all backing files with correct file size (though they can be all sparse files), and the upper fs shall guarantee that past EOF access will never happen. Then with this precondition, cachefiles can detect the actual size of backing file, and set it as the size of backed file. Signed-off-by: Jeffle Xu --- fs/cachefiles/namei.c | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/fs/cachefiles/namei.c b/fs/cachefiles/namei.c index 074722c21522..54123b2693cd 100644 --- a/fs/cachefiles/namei.c +++ b/fs/cachefiles/namei.c @@ -511,9 +511,19 @@ struct file *cachefiles_create_tmpfile(struct cachefiles_object *object) */ static bool cachefiles_create_file(struct cachefiles_object *object) { + struct cachefiles_cache *cache = object->volume->cache; struct file *file; int ret; + /* + * Demand read mode requires that backing files have been prepared with + * correct file size under corresponding directory. We can get here when + * the backing file doesn't exist under corresponding directory, or the + * file size is unexpected 0. + */ + if (test_bit(CACHEFILES_DEMAND_MODE, &cache->flags)) + return false; + ret = cachefiles_has_space(object->volume->cache, 1, 0, cachefiles_has_space_for_create); if (ret < 0) @@ -530,6 +540,32 @@ static bool cachefiles_create_file(struct cachefiles_object *object) return true; } +/* + * Fs using fscache for demand reading may have no idea of the file size of + * backing files. Thus the demand read mode requires that backing files have + * been prepared with correct file size under corresponding directory. Then + * fscache backend is responsible for taking the file size of the backing file + * as the object size. + */ +static int cachefiles_recheck_size(struct cachefiles_object *object, + struct file *file) +{ + loff_t size; + struct cachefiles_cache *cache = object->volume->cache; + + if (!test_bit(CACHEFILES_DEMAND_MODE, &cache->flags)) + return 0; + + size = i_size_read(file_inode(file)); + if (size) { + object->cookie->object_size = size; + return 0; + } else { + return -EINVAL; + } + +} + /* * Open an existing file, checking its attributes and replacing it if it is * stale. @@ -569,6 +605,10 @@ static bool cachefiles_open_file(struct cachefiles_object *object, } _debug("file -> %pd positive", dentry); + ret = cachefiles_recheck_size(object, file); + if (ret < 0) + goto check_failed; + ret = cachefiles_check_auxdata(object, file); if (ret < 0) goto check_failed; From patchwork Mon Dec 27 12:54:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12699758 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0B22C433F5 for ; Mon, 27 Dec 2021 12:55:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232299AbhL0Mzm (ORCPT ); Mon, 27 Dec 2021 07:55:42 -0500 Received: from out4436.biz.mail.alibaba.com ([47.88.44.36]:5951 "EHLO out4436.biz.mail.alibaba.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236862AbhL0MzC (ORCPT ); Mon, 27 Dec 2021 07:55:02 -0500 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R161e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04394;MF=jefflexu@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0V.w7GWM_1640609689; Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0V.w7GWM_1640609689) by smtp.aliyun-inc.com(127.0.0.1); Mon, 27 Dec 2021 20:54:49 +0800 From: Jeffle Xu To: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org Cc: linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: [PATCH v1 04/23] netfs: make ops->init_rreq() optional Date: Mon, 27 Dec 2021 20:54:25 +0800 Message-Id: <20211227125444.21187-5-jefflexu@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211227125444.21187-1-jefflexu@linux.alibaba.com> References: <20211227125444.21187-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org There's already upper fs implementing empty .init_rreq() callback, and thus make it optional. Signed-off-by: Jeffle Xu --- fs/ceph/addr.c | 5 ----- fs/netfs/read_helper.c | 3 ++- 2 files changed, 2 insertions(+), 6 deletions(-) diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c index b3d9459c9bbd..c98e5238a1b6 100644 --- a/fs/ceph/addr.c +++ b/fs/ceph/addr.c @@ -297,10 +297,6 @@ static void ceph_netfs_issue_op(struct netfs_read_subrequest *subreq) dout("%s: result %d\n", __func__, err); } -static void ceph_init_rreq(struct netfs_read_request *rreq, struct file *file) -{ -} - static void ceph_readahead_cleanup(struct address_space *mapping, void *priv) { struct inode *inode = mapping->host; @@ -312,7 +308,6 @@ static void ceph_readahead_cleanup(struct address_space *mapping, void *priv) } static const struct netfs_read_request_ops ceph_netfs_read_ops = { - .init_rreq = ceph_init_rreq, .is_cache_enabled = ceph_is_cache_enabled, .begin_cache_operation = ceph_begin_cache_operation, .issue_op = ceph_netfs_issue_op, diff --git a/fs/netfs/read_helper.c b/fs/netfs/read_helper.c index 1659a55e9c1c..8c58cff420ba 100644 --- a/fs/netfs/read_helper.c +++ b/fs/netfs/read_helper.c @@ -55,7 +55,8 @@ static struct netfs_read_request *netfs_alloc_read_request( INIT_WORK(&rreq->work, netfs_rreq_work); refcount_set(&rreq->usage, 1); __set_bit(NETFS_RREQ_IN_PROGRESS, &rreq->flags); - ops->init_rreq(rreq, file); + if (ops->init_rreq) + ops->init_rreq(rreq, file); netfs_stat(&netfs_n_rh_rreq); } From patchwork Mon Dec 27 12:54:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12699745 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C0BBC433EF for ; Mon, 27 Dec 2021 12:55:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236838AbhL0My7 (ORCPT ); Mon, 27 Dec 2021 07:54:59 -0500 Received: from out30-43.freemail.mail.aliyun.com ([115.124.30.43]:41581 "EHLO out30-43.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236787AbhL0Myx (ORCPT ); Mon, 27 Dec 2021 07:54:53 -0500 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R781e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04426;MF=jefflexu@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0V.wXYTl_1640609690; Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0V.wXYTl_1640609690) by smtp.aliyun-inc.com(127.0.0.1); Mon, 27 Dec 2021 20:54:50 +0800 From: Jeffle Xu To: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org Cc: linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: [PATCH v1 05/23] netfs: add inode parameter to netfs_alloc_read_request() Date: Mon, 27 Dec 2021 20:54:26 +0800 Message-Id: <20211227125444.21187-6-jefflexu@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211227125444.21187-1-jefflexu@linux.alibaba.com> References: <20211227125444.21187-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org When working as the local cache, the @file parameter of netfs_alloc_read_request() represents the backed file inside netfs. It is for two use: 1) we can derive the corresponding inode from file, 2) works as the argument for ops->init_rreq(). In the new introduced demand-read mode, netfs_readpage() will be called by the upper fs to read from backing files. However in this new mode, the backed file may not be opened, and thus the @file argument is NULL in this case. For netfs_readpage(), @file parameter represents the backed file inside netfs, while @folio parameter represents one page cache inside the address space of this backed file. We can still derive the inode from the @folio parameter, even when @file parameter is NULL. Thus refactor netfs_alloc_read_request() somewhat for this change. Signed-off-by: Jeffle Xu --- fs/netfs/read_helper.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/fs/netfs/read_helper.c b/fs/netfs/read_helper.c index 8c58cff420ba..ca84918b6b5d 100644 --- a/fs/netfs/read_helper.c +++ b/fs/netfs/read_helper.c @@ -39,7 +39,7 @@ static void netfs_put_subrequest(struct netfs_read_subrequest *subreq, static struct netfs_read_request *netfs_alloc_read_request( const struct netfs_read_request_ops *ops, void *netfs_priv, - struct file *file) + struct inode *inode, struct file *file) { static atomic_t debug_ids; struct netfs_read_request *rreq; @@ -48,7 +48,7 @@ static struct netfs_read_request *netfs_alloc_read_request( if (rreq) { rreq->netfs_ops = ops; rreq->netfs_priv = netfs_priv; - rreq->inode = file_inode(file); + rreq->inode = inode; rreq->i_size = i_size_read(rreq->inode); rreq->debug_id = atomic_inc_return(&debug_ids); INIT_LIST_HEAD(&rreq->subrequests); @@ -870,6 +870,7 @@ void netfs_readahead(struct readahead_control *ractl, void *netfs_priv) { struct netfs_read_request *rreq; + struct inode *inode = file_inode(ractl->file); unsigned int debug_index = 0; int ret; @@ -878,7 +879,7 @@ void netfs_readahead(struct readahead_control *ractl, if (readahead_count(ractl) == 0) goto cleanup; - rreq = netfs_alloc_read_request(ops, netfs_priv, ractl->file); + rreq = netfs_alloc_read_request(ops, netfs_priv, inode, ractl->file); if (!rreq) goto cleanup; rreq->mapping = ractl->mapping; @@ -948,12 +949,13 @@ int netfs_readpage(struct file *file, void *netfs_priv) { struct netfs_read_request *rreq; + struct inode *inode = folio_file_mapping(folio)->host; unsigned int debug_index = 0; int ret; _enter("%lx", folio_index(folio)); - rreq = netfs_alloc_read_request(ops, netfs_priv, file); + rreq = netfs_alloc_read_request(ops, netfs_priv, inode, file); if (!rreq) { if (netfs_priv) ops->cleanup(folio_file_mapping(folio), netfs_priv); @@ -1122,7 +1124,7 @@ int netfs_write_begin(struct file *file, struct address_space *mapping, } ret = -ENOMEM; - rreq = netfs_alloc_read_request(ops, netfs_priv, file); + rreq = netfs_alloc_read_request(ops, netfs_priv, inode, file); if (!rreq) goto error; rreq->mapping = folio_file_mapping(folio); From patchwork Mon Dec 27 12:54:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12699744 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAF43C433FE for ; Mon, 27 Dec 2021 12:55:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236797AbhL0My7 (ORCPT ); Mon, 27 Dec 2021 07:54:59 -0500 Received: from out30-56.freemail.mail.aliyun.com ([115.124.30.56]:55794 "EHLO out30-56.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232184AbhL0Myy (ORCPT ); Mon, 27 Dec 2021 07:54:54 -0500 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R911e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04407;MF=jefflexu@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0V.wf8nA_1640609691; Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0V.wf8nA_1640609691) by smtp.aliyun-inc.com(127.0.0.1); Mon, 27 Dec 2021 20:54:52 +0800 From: Jeffle Xu To: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org Cc: linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: [PATCH v1 06/23] erofs: export erofs_map_blocks() Date: Mon, 27 Dec 2021 20:54:27 +0800 Message-Id: <20211227125444.21187-7-jefflexu@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211227125444.21187-1-jefflexu@linux.alibaba.com> References: <20211227125444.21187-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org ... so that it can be used in the following introduced fs/erofs/fscache.c. Signed-off-by: Jeffle Xu --- fs/erofs/data.c | 4 ++-- fs/erofs/internal.h | 2 ++ 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/fs/erofs/data.c b/fs/erofs/data.c index 0e35ef3f9f3d..477aaff0c832 100644 --- a/fs/erofs/data.c +++ b/fs/erofs/data.c @@ -77,8 +77,8 @@ static int erofs_map_blocks_flatmode(struct inode *inode, return err; } -static int erofs_map_blocks(struct inode *inode, - struct erofs_map_blocks *map, int flags) +int erofs_map_blocks(struct inode *inode, + struct erofs_map_blocks *map, int flags) { struct super_block *sb = inode->i_sb; struct erofs_inode *vi = EROFS_I(inode); diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h index 3265688af7f9..45fb6f5d11b5 100644 --- a/fs/erofs/internal.h +++ b/fs/erofs/internal.h @@ -447,6 +447,8 @@ struct page *erofs_get_meta_page(struct super_block *sb, erofs_blk_t blkaddr); int erofs_map_dev(struct super_block *sb, struct erofs_map_dev *dev); int erofs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo, u64 start, u64 len); +int erofs_map_blocks(struct inode *inode, + struct erofs_map_blocks *map, int flags); /* inode.c */ static inline unsigned long erofs_inode_hash(erofs_nid_t nid) From patchwork Mon Dec 27 12:54:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12699762 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB358C433EF for ; Mon, 27 Dec 2021 12:55:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236787AbhL0MzA (ORCPT ); Mon, 27 Dec 2021 07:55:00 -0500 Received: from out4436.biz.mail.alibaba.com ([47.88.44.36]:5085 "EHLO out4436.biz.mail.alibaba.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236808AbhL0Myz (ORCPT ); Mon, 27 Dec 2021 07:54:55 -0500 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R901e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e01424;MF=jefflexu@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0V.wXYUC_1640609692; Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0V.wXYUC_1640609692) by smtp.aliyun-inc.com(127.0.0.1); Mon, 27 Dec 2021 20:54:53 +0800 From: Jeffle Xu To: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org Cc: linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: [PATCH v1 07/23] erofs: add nodev mode Date: Mon, 27 Dec 2021 20:54:28 +0800 Message-Id: <20211227125444.21187-8-jefflexu@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211227125444.21187-1-jefflexu@linux.alibaba.com> References: <20211227125444.21187-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Until then erofs is exactly blockdev based filesystem. In other using scenarios (e.g. container image), erofs needs to run upon files. This patch introduces a new nodev mode, in which erofs could be mounted from a bootstrap blob file containing the complete erofs image. The following patch will introduce a new mount option "uuid", by which users could specify the bootstrap blob file. Signed-off-by: Jeffle Xu --- fs/erofs/data.c | 13 ++++++++--- fs/erofs/internal.h | 1 + fs/erofs/super.c | 56 +++++++++++++++++++++++++++++++++------------ 3 files changed, 53 insertions(+), 17 deletions(-) diff --git a/fs/erofs/data.c b/fs/erofs/data.c index 477aaff0c832..61fa431d0713 100644 --- a/fs/erofs/data.c +++ b/fs/erofs/data.c @@ -11,11 +11,18 @@ struct page *erofs_get_meta_page(struct super_block *sb, erofs_blk_t blkaddr) { - struct address_space *const mapping = sb->s_bdev->bd_inode->i_mapping; + struct address_space *mapping; struct page *page; - page = read_cache_page_gfp(mapping, blkaddr, - mapping_gfp_constraint(mapping, ~__GFP_FS)); + if (sb->s_bdev) { + mapping = sb->s_bdev->bd_inode->i_mapping; + page = read_cache_page_gfp(mapping, blkaddr, + mapping_gfp_constraint(mapping, ~__GFP_FS)); + } else { + /* TODO: data path in nodev mode */ + page = ERR_PTR(-EINVAL); + } + /* should already be PageUptodate */ if (!IS_ERR(page)) lock_page(page); diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h index 45fb6f5d11b5..c9ee8c247202 100644 --- a/fs/erofs/internal.h +++ b/fs/erofs/internal.h @@ -67,6 +67,7 @@ struct erofs_mount_opts { unsigned int max_sync_decompress_pages; #endif unsigned int mount_opt; + char *uuid; }; struct erofs_dev_context { diff --git a/fs/erofs/super.c b/fs/erofs/super.c index 6a969b1e0ee6..80c00c34eafc 100644 --- a/fs/erofs/super.c +++ b/fs/erofs/super.c @@ -304,15 +304,19 @@ static int erofs_init_devices(struct super_block *sb, } dis = ptr + erofs_blkoff(pos); - bdev = blkdev_get_by_path(dif->path, - FMODE_READ | FMODE_EXCL, - sb->s_type); - if (IS_ERR(bdev)) { - err = PTR_ERR(bdev); - goto err_out; + if (sb->s_bdev) { + bdev = blkdev_get_by_path(dif->path, + FMODE_READ | FMODE_EXCL, + sb->s_type); + if (IS_ERR(bdev)) { + err = PTR_ERR(bdev); + goto err_out; + } + dif->bdev = bdev; + dif->dax_dev = fs_dax_get_by_bdev(bdev); + } else { + /* TODO: multi devs in nodev mode */ } - dif->bdev = bdev; - dif->dax_dev = fs_dax_get_by_bdev(bdev); dif->blocks = le32_to_cpu(dis->blocks); dif->mapped_blkaddr = le32_to_cpu(dis->mapped_blkaddr); sbi->total_blocks += dif->blocks; @@ -337,7 +341,11 @@ static int erofs_read_superblock(struct super_block *sb) void *data; int ret; - page = read_mapping_page(sb->s_bdev->bd_inode->i_mapping, 0, NULL); + /* TODO: metadata path in nodev mode */ + if (sb->s_bdev) + page = read_mapping_page(sb->s_bdev->bd_inode->i_mapping, 0, NULL); + else + page = ERR_PTR(-EOPNOTSUPP); if (IS_ERR(page)) { erofs_err(sb, "cannot read erofs superblock"); return PTR_ERR(page); @@ -633,9 +641,12 @@ static int erofs_fc_fill_super(struct super_block *sb, struct fs_context *fc) sb->s_magic = EROFS_SUPER_MAGIC; - if (!sb_set_blocksize(sb, EROFS_BLKSIZ)) { + if (sb->s_bdev && !sb_set_blocksize(sb, EROFS_BLKSIZ)) { erofs_err(sb, "failed to set erofs blksize"); return -EINVAL; + } else { + sb->s_blocksize = EROFS_BLKSIZ; + sb->s_blocksize_bits = LOG_BLOCK_SIZE; } sbi = kzalloc(sizeof(*sbi), GFP_KERNEL); @@ -644,16 +655,22 @@ static int erofs_fc_fill_super(struct super_block *sb, struct fs_context *fc) sb->s_fs_info = sbi; sbi->opt = ctx->opt; - sbi->dax_dev = fs_dax_get_by_bdev(sb->s_bdev); sbi->devs = ctx->devs; ctx->devs = NULL; + if (sb->s_bdev) + sbi->dax_dev = fs_dax_get_by_bdev(sb->s_bdev); + else + sbi->dax_dev = NULL; + err = erofs_read_superblock(sb); if (err) return err; if (test_opt(&sbi->opt, DAX_ALWAYS) && - !dax_supported(sbi->dax_dev, sb->s_bdev, EROFS_BLKSIZ, 0, bdev_nr_sectors(sb->s_bdev))) { + (!sbi->dax_dev || + !dax_supported(sbi->dax_dev, sb->s_bdev, EROFS_BLKSIZ, 0, + bdev_nr_sectors(sb->s_bdev)))) { errorfc(fc, "DAX unsupported by block device. Turning off DAX."); clear_opt(&sbi->opt, DAX_ALWAYS); } @@ -701,6 +718,10 @@ static int erofs_fc_fill_super(struct super_block *sb, struct fs_context *fc) static int erofs_fc_get_tree(struct fs_context *fc) { + struct erofs_fs_context *ctx = fc->fs_private; + + if (ctx->opt.uuid) + return get_tree_nodev(fc, erofs_fc_fill_super); return get_tree_bdev(fc, erofs_fc_fill_super); } @@ -789,7 +810,10 @@ static void erofs_kill_sb(struct super_block *sb) WARN_ON(sb->s_magic != EROFS_SUPER_MAGIC); - kill_block_super(sb); + if (sb->s_bdev) + kill_block_super(sb); + else + generic_shutdown_super(sb); sbi = EROFS_SB(sb); if (!sbi) @@ -889,7 +913,11 @@ static int erofs_statfs(struct dentry *dentry, struct kstatfs *buf) { struct super_block *sb = dentry->d_sb; struct erofs_sb_info *sbi = EROFS_SB(sb); - u64 id = huge_encode_dev(sb->s_bdev->bd_dev); + u64 id = 0; + + /* TODO: fsid in nodev mode */ + if (sb->s_bdev) + id = huge_encode_dev(sb->s_bdev->bd_dev); buf->f_type = sb->s_magic; buf->f_bsize = EROFS_BLKSIZ; From patchwork Mon Dec 27 12:54:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12699761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A49ABC4332F for ; Mon, 27 Dec 2021 12:55:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236846AbhL0MzA (ORCPT ); Mon, 27 Dec 2021 07:55:00 -0500 Received: from out30-45.freemail.mail.aliyun.com ([115.124.30.45]:58404 "EHLO out30-45.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236813AbhL0My5 (ORCPT ); Mon, 27 Dec 2021 07:54:57 -0500 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R571e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04426;MF=jefflexu@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0V.vxTIo_1640609693; Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0V.vxTIo_1640609693) by smtp.aliyun-inc.com(127.0.0.1); Mon, 27 Dec 2021 20:54:54 +0800 From: Jeffle Xu To: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org Cc: linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: [PATCH v1 08/23] erofs: register global fscache volume Date: Mon, 27 Dec 2021 20:54:29 +0800 Message-Id: <20211227125444.21187-9-jefflexu@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211227125444.21187-1-jefflexu@linux.alibaba.com> References: <20211227125444.21187-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org All erofs instances will share one global fscache volume. In this using scenario, one erofs instance could be mounted from one (or multiple) blob files instead of blkdev. The number of blob files that each erofs instance could correspond to is limited, since these blob files are quite large in size. For example, when used for container image distribution, one erofs instance used for container image for node.js will correspond to ~20 blob files in total. Thus in densely employed environment, there could be as many as hundreds of containers and thus thousands of fscache cookies under one fscache volume. Then as for cachefiles backend, the hash table managing all cookies under one volume contains 32K slots. Thus the hashing functionality shall scale well in this case. Besides, cachefiles backend will scatter backing files under 256 fan sub-directoris, and thus the scalability of looking up backing files shall also not be an issue. Signed-off-by: Jeffle Xu --- fs/erofs/Makefile | 2 +- fs/erofs/fscache.c | 21 +++++++++++++++++++++ fs/erofs/internal.h | 5 +++++ fs/erofs/super.c | 7 +++++++ 4 files changed, 34 insertions(+), 1 deletion(-) create mode 100644 fs/erofs/fscache.c diff --git a/fs/erofs/Makefile b/fs/erofs/Makefile index 756fe2d65272..f9a3609625aa 100644 --- a/fs/erofs/Makefile +++ b/fs/erofs/Makefile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only obj-$(CONFIG_EROFS_FS) += erofs.o -erofs-objs := super.o inode.o data.o namei.o dir.o utils.o pcpubuf.o +erofs-objs := super.o inode.o data.o namei.o dir.o utils.o pcpubuf.o fscache.o erofs-$(CONFIG_EROFS_FS_XATTR) += xattr.o erofs-$(CONFIG_EROFS_FS_ZIP) += decompressor.o zmap.o zdata.o erofs-$(CONFIG_EROFS_FS_ZIP_LZMA) += decompressor_lzma.o diff --git a/fs/erofs/fscache.c b/fs/erofs/fscache.c new file mode 100644 index 000000000000..cdbbe3f15a20 --- /dev/null +++ b/fs/erofs/fscache.c @@ -0,0 +1,21 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2021, Alibaba Cloud + */ +#include "internal.h" + +static struct fscache_volume *volume; + +int __init erofs_fscache_init(void) +{ + volume = fscache_acquire_volume("erofs", NULL, NULL, 0); + if (!volume) + return -EINVAL; + + return 0; +} + +void erofs_fscache_cleanup(void) +{ + fscache_relinquish_volume(volume, NULL, false); +} diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h index c9ee8c247202..d464642e9d35 100644 --- a/fs/erofs/internal.h +++ b/fs/erofs/internal.h @@ -17,6 +17,7 @@ #include #include #include +#include #include "erofs_fs.h" /* redefine pr_fmt "erofs: " */ @@ -569,6 +570,10 @@ static inline int z_erofs_load_lzma_config(struct super_block *sb, } #endif /* !CONFIG_EROFS_FS_ZIP */ +/* fscache.c */ +int erofs_fscache_init(void); +void erofs_fscache_cleanup(void); + #define EFSCORRUPTED EUCLEAN /* Filesystem is corrupted */ #endif /* __EROFS_INTERNAL_H */ diff --git a/fs/erofs/super.c b/fs/erofs/super.c index 80c00c34eafc..517d74f3c303 100644 --- a/fs/erofs/super.c +++ b/fs/erofs/super.c @@ -876,6 +876,10 @@ static int __init erofs_module_init(void) if (err) goto zip_err; + err = erofs_fscache_init(); + if (err) + goto fscache_err; + err = register_filesystem(&erofs_fs_type); if (err) goto fs_err; @@ -883,6 +887,8 @@ static int __init erofs_module_init(void) return 0; fs_err: + erofs_fscache_cleanup(); +fscache_err: z_erofs_exit_zip_subsystem(); zip_err: z_erofs_lzma_exit(); @@ -901,6 +907,7 @@ static void __exit erofs_module_exit(void) /* Ensure all RCU free inodes / pclusters are safe to be destroyed. */ rcu_barrier(); + erofs_fscache_cleanup(); z_erofs_exit_zip_subsystem(); z_erofs_lzma_exit(); erofs_exit_shrinker(); From patchwork Mon Dec 27 12:54:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12699746 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 235A1C433EF for ; Mon, 27 Dec 2021 12:55:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236871AbhL0MzC (ORCPT ); Mon, 27 Dec 2021 07:55:02 -0500 Received: from out30-56.freemail.mail.aliyun.com ([115.124.30.56]:57114 "EHLO out30-56.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232241AbhL0My5 (ORCPT ); Mon, 27 Dec 2021 07:54:57 -0500 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R721e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04357;MF=jefflexu@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0V.wf8ne_1640609695; Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0V.wf8ne_1640609695) by smtp.aliyun-inc.com(127.0.0.1); Mon, 27 Dec 2021 20:54:55 +0800 From: Jeffle Xu To: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org Cc: linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: [PATCH v1 09/23] erofs: add cookie context helper functions Date: Mon, 27 Dec 2021 20:54:30 +0800 Message-Id: <20211227125444.21187-10-jefflexu@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211227125444.21187-1-jefflexu@linux.alibaba.com> References: <20211227125444.21187-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Introduce 'struct erofs_cookie_ctx' for managing cookie for backing file, and the following introduced API for reading from backing file. Besides, introduce two helper functions for initializing and cleaning up erofs_cookie_ctx. struct erofs_cookie_ctx * erofs_fscache_get_ctx(struct super_block *sb, char *path); void erofs_fscache_put_ctx(struct erofs_cookie_ctx *ctx); Signed-off-by: Jeffle Xu --- fs/erofs/fscache.c | 75 +++++++++++++++++++++++++++++++++++++++++++++ fs/erofs/internal.h | 8 +++++ 2 files changed, 83 insertions(+) diff --git a/fs/erofs/fscache.c b/fs/erofs/fscache.c index cdbbe3f15a20..15c5bb0f8ea5 100644 --- a/fs/erofs/fscache.c +++ b/fs/erofs/fscache.c @@ -6,6 +6,81 @@ static struct fscache_volume *volume; +static int erofs_fscache_init_cookie(struct erofs_cookie_ctx *ctx, char *path) +{ + struct fscache_cookie *cookie; + + /* + * @object_size shall be non-zero to avoid + * FSCACHE_COOKIE_NO_DATA_TO_READ. + */ + cookie = fscache_acquire_cookie(volume, 0, + path, strlen(path), + NULL, 0, -1); + if (!cookie) + return -EINVAL; + + fscache_use_cookie(cookie, false); + ctx->cookie = cookie; + return 0; +} + +static void erofs_fscache_cleanup_cookie(struct erofs_cookie_ctx *ctx) +{ + struct fscache_cookie *cookie = ctx->cookie; + + fscache_unuse_cookie(cookie, NULL, NULL); + fscache_relinquish_cookie(cookie, false); + ctx->cookie = NULL; +} + +static int erofs_fscahce_init_ctx(struct erofs_cookie_ctx *ctx, + struct super_block *sb, char *path) +{ + int ret; + + ret = erofs_fscache_init_cookie(ctx, path); + if (ret) { + erofs_err(sb, "failed to init cookie\n"); + return ret; + } + + return 0; +} + +static void erofs_fscache_cleanup_ctx(struct erofs_cookie_ctx *ctx) +{ + erofs_fscache_cleanup_cookie(ctx); +} + +struct erofs_cookie_ctx *erofs_fscache_get_ctx(struct super_block *sb, + char *path) +{ + struct erofs_cookie_ctx *ctx; + int ret; + + ctx = kzalloc(sizeof(*ctx), GFP_KERNEL); + if (!ctx) + return ERR_PTR(-ENOMEM); + + ret = erofs_fscahce_init_ctx(ctx, sb, path); + if (ret) { + kfree(ctx); + return ERR_PTR(ret); + } + + return ctx; +} + +void erofs_fscache_put_ctx(struct erofs_cookie_ctx *ctx) +{ + if (!ctx) + return; + + erofs_fscache_cleanup_ctx(ctx); + kfree(ctx); +} + int __init erofs_fscache_init(void) { volume = fscache_acquire_volume("erofs", NULL, NULL, 0); diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h index d464642e9d35..4179c3dcb7f9 100644 --- a/fs/erofs/internal.h +++ b/fs/erofs/internal.h @@ -91,6 +91,10 @@ struct erofs_sb_lz4_info { u16 max_pclusterblks; }; +struct erofs_cookie_ctx { + struct fscache_cookie *cookie; +}; + struct erofs_sb_info { struct erofs_mount_opts opt; /* options */ #ifdef CONFIG_EROFS_FS_ZIP @@ -574,6 +578,10 @@ static inline int z_erofs_load_lzma_config(struct super_block *sb, int erofs_fscache_init(void); void erofs_fscache_cleanup(void); +struct erofs_cookie_ctx *erofs_fscache_get_ctx(struct super_block *sb, + char *path); +void erofs_fscache_put_ctx(struct erofs_cookie_ctx *ctx); + #define EFSCORRUPTED EUCLEAN /* Filesystem is corrupted */ #endif /* __EROFS_INTERNAL_H */ From patchwork Mon Dec 27 12:54:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12699763 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6840FC433EF for ; Mon, 27 Dec 2021 12:56:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236792AbhL0Mz6 (ORCPT ); Mon, 27 Dec 2021 07:55:58 -0500 Received: from out30-45.freemail.mail.aliyun.com ([115.124.30.45]:46182 "EHLO out30-45.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236823AbhL0My6 (ORCPT ); Mon, 27 Dec 2021 07:54:58 -0500 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R461e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04395;MF=jefflexu@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0V.xJoQp_1640609696; Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0V.xJoQp_1640609696) by smtp.aliyun-inc.com(127.0.0.1); Mon, 27 Dec 2021 20:54:57 +0800 From: Jeffle Xu To: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org Cc: linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: [PATCH v1 10/23] erofs: add anonymous inode managing page cache of blob file Date: Mon, 27 Dec 2021 20:54:31 +0800 Message-Id: <20211227125444.21187-11-jefflexu@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211227125444.21187-1-jefflexu@linux.alibaba.com> References: <20211227125444.21187-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Introduce one anonymous inode for managing page cache of corresponding blob file. Then erofs could read directly from the address space of the anonymous inode when cache hit. Signed-off-by: Jeffle Xu --- fs/erofs/fscache.c | 35 +++++++++++++++++++++++++++++++++++ fs/erofs/internal.h | 1 + 2 files changed, 36 insertions(+) diff --git a/fs/erofs/fscache.c b/fs/erofs/fscache.c index 15c5bb0f8ea5..4dfca7c95710 100644 --- a/fs/erofs/fscache.c +++ b/fs/erofs/fscache.c @@ -34,6 +34,33 @@ static void erofs_fscache_cleanup_cookie(struct erofs_cookie_ctx *ctx) ctx->cookie = NULL; } +static const struct address_space_operations erofs_fscache_aops = { +}; + +static int erofs_fscache_get_inode(struct erofs_cookie_ctx *ctx, + struct super_block *sb) +{ + struct inode *const inode = new_inode(sb); + + if (!inode) + return -ENOMEM; + + set_nlink(inode, 1); + inode->i_size = OFFSET_MAX; + + inode->i_mapping->a_ops = &erofs_fscache_aops; + mapping_set_gfp_mask(inode->i_mapping, + GFP_NOFS | __GFP_HIGHMEM | __GFP_MOVABLE); + ctx->inode = inode; + return 0; +} + +static void erofs_fscache_put_inode(struct erofs_cookie_ctx *ctx) +{ + iput(ctx->inode); + ctx->inode = NULL; +} + static int erofs_fscahce_init_ctx(struct erofs_cookie_ctx *ctx, struct super_block *sb, char *path) { @@ -45,12 +72,20 @@ static int erofs_fscahce_init_ctx(struct erofs_cookie_ctx *ctx, return ret; } + ret = erofs_fscache_get_inode(ctx, sb); + if (ret) { + erofs_err(sb, "failed to get anonymous inode\n"); + erofs_fscache_cleanup_cookie(ctx); + return ret; + } + return 0; } static void erofs_fscache_cleanup_ctx(struct erofs_cookie_ctx *ctx) { erofs_fscache_cleanup_cookie(ctx); + erofs_fscache_put_inode(ctx); } struct erofs_cookie_ctx *erofs_fscache_get_ctx(struct super_block *sb, diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h index 4179c3dcb7f9..2e4f267b37e7 100644 --- a/fs/erofs/internal.h +++ b/fs/erofs/internal.h @@ -93,6 +93,7 @@ struct erofs_sb_lz4_info { struct erofs_cookie_ctx { struct fscache_cookie *cookie; + struct inode *inode; }; struct erofs_sb_info { From patchwork Mon Dec 27 12:54:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12699760 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB9C5C433FE for ; Mon, 27 Dec 2021 12:55:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237065AbhL0Mzo (ORCPT ); Mon, 27 Dec 2021 07:55:44 -0500 Received: from out30-131.freemail.mail.aliyun.com ([115.124.30.131]:50496 "EHLO out30-131.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236806AbhL0MzA (ORCPT ); Mon, 27 Dec 2021 07:55:00 -0500 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R131e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04395;MF=jefflexu@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0V.vZtXb_1640609697; Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0V.vZtXb_1640609697) by smtp.aliyun-inc.com(127.0.0.1); Mon, 27 Dec 2021 20:54:58 +0800 From: Jeffle Xu To: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org Cc: linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: [PATCH v1 11/23] erofs: register cookie context for bootstrap Date: Mon, 27 Dec 2021 20:54:32 +0800 Message-Id: <20211227125444.21187-12-jefflexu@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211227125444.21187-1-jefflexu@linux.alibaba.com> References: <20211227125444.21187-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Something worth mentioning about the cleanup routine. 1. The init routine is prior to when the root inode gets initialized, and thus the corresponding cleanup routine shall be placed under .kill_sb() callback. 2. The init routine will instantiate anonymous inodes under the super_block, and thus .put_super() callback shall also contain the cleanup routine. Or we'll get "VFS: Busy inodes after unmount." warning. Signed-off-by: Jeffle Xu --- fs/erofs/internal.h | 2 ++ fs/erofs/super.c | 13 +++++++++++++ 2 files changed, 15 insertions(+) diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h index 2e4f267b37e7..4ee4ff6774ba 100644 --- a/fs/erofs/internal.h +++ b/fs/erofs/internal.h @@ -141,6 +141,8 @@ struct erofs_sb_info { u8 volume_name[16]; /* volume name */ u32 feature_compat; u32 feature_incompat; + + struct erofs_cookie_ctx *bootstrap; }; #define EROFS_SB(sb) ((struct erofs_sb_info *)(sb)->s_fs_info) diff --git a/fs/erofs/super.c b/fs/erofs/super.c index 517d74f3c303..141cabd01d32 100644 --- a/fs/erofs/super.c +++ b/fs/erofs/super.c @@ -663,6 +663,16 @@ static int erofs_fc_fill_super(struct super_block *sb, struct fs_context *fc) else sbi->dax_dev = NULL; + if (!sb->s_bdev) { + struct erofs_cookie_ctx *bootstrap; + + bootstrap = erofs_fscache_get_ctx(sb, ctx->opt.uuid); + if (IS_ERR(bootstrap)) + return PTR_ERR(bootstrap); + + sbi->bootstrap = bootstrap; + } + err = erofs_read_superblock(sb); if (err) return err; @@ -820,6 +830,7 @@ static void erofs_kill_sb(struct super_block *sb) return; erofs_free_dev_context(sbi->devs); + erofs_fscache_put_ctx(sbi->bootstrap); fs_put_dax(sbi->dax_dev); kfree(sbi); sb->s_fs_info = NULL; @@ -837,6 +848,8 @@ static void erofs_put_super(struct super_block *sb) iput(sbi->managed_cache); sbi->managed_cache = NULL; #endif + erofs_fscache_put_ctx(sbi->bootstrap); + sbi->bootstrap = NULL; } static struct file_system_type erofs_fs_type = { From patchwork Mon Dec 27 12:54:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12699759 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43F50C4332F for ; Mon, 27 Dec 2021 12:55:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237061AbhL0Mzn (ORCPT ); Mon, 27 Dec 2021 07:55:43 -0500 Received: from out30-132.freemail.mail.aliyun.com ([115.124.30.132]:43159 "EHLO out30-132.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236849AbhL0MzB (ORCPT ); Mon, 27 Dec 2021 07:55:01 -0500 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R161e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04400;MF=jefflexu@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0V.vxTJW_1640609698; Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0V.vxTJW_1640609698) by smtp.aliyun-inc.com(127.0.0.1); Mon, 27 Dec 2021 20:54:59 +0800 From: Jeffle Xu To: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org Cc: linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: [PATCH v1 12/23] erofs: implement fscache-based metadata read Date: Mon, 27 Dec 2021 20:54:33 +0800 Message-Id: <20211227125444.21187-13-jefflexu@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211227125444.21187-1-jefflexu@linux.alibaba.com> References: <20211227125444.21187-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org This patch implements the data plane of reading metadata from bootstrap blob file over fscache. Be noted that currently it only supports the scenario where the backing file has no hole. Once it hits a hole of the backing file, erofs will fail the IO with -EOPNOTSUPP for now. The following patch will fix this issue, i.e. implementing the demand reading mode. Signed-off-by: Jeffle Xu Reported-by: kernel test robot Reported-by: kernel test robot --- fs/erofs/data.c | 4 +-- fs/erofs/fscache.c | 59 +++++++++++++++++++++++++++++++++++++++++++++ fs/erofs/internal.h | 3 +++ fs/erofs/super.c | 7 ++---- 4 files changed, 66 insertions(+), 7 deletions(-) diff --git a/fs/erofs/data.c b/fs/erofs/data.c index 61fa431d0713..f0857c285fac 100644 --- a/fs/erofs/data.c +++ b/fs/erofs/data.c @@ -19,8 +19,8 @@ struct page *erofs_get_meta_page(struct super_block *sb, erofs_blk_t blkaddr) page = read_cache_page_gfp(mapping, blkaddr, mapping_gfp_constraint(mapping, ~__GFP_FS)); } else { - /* TODO: data path in nodev mode */ - page = ERR_PTR(-EINVAL); + page = erofs_readpage_from_fscache(EROFS_SB(sb)->bootstrap, + blkaddr); } /* should already be PageUptodate */ diff --git a/fs/erofs/fscache.c b/fs/erofs/fscache.c index 4dfca7c95710..325f5663836b 100644 --- a/fs/erofs/fscache.c +++ b/fs/erofs/fscache.c @@ -6,6 +6,65 @@ static struct fscache_volume *volume; +static int erofs_begin_cache_operation(struct netfs_read_request *rreq) +{ + return fscache_begin_read_operation(&rreq->cache_resources, + rreq->netfs_priv); +} + +static void erofs_priv_cleanup(struct address_space *mapping, void *netfs_priv) +{ +} + +static void erofs_issue_op(struct netfs_read_subrequest *subreq) +{ + /* + * TODO: implement demand-read logic later. + * We rely on user daemon to prepare blob files under corresponding + * directory, and we can reach here if blob files don't exist. + */ + + netfs_subreq_terminated(subreq, -EOPNOTSUPP, false); +} + +const struct netfs_read_request_ops erofs_req_ops = { + .begin_cache_operation = erofs_begin_cache_operation, + .cleanup = erofs_priv_cleanup, + .issue_op = erofs_issue_op, +}; + +struct page *erofs_readpage_from_fscache(struct erofs_cookie_ctx *ctx, + pgoff_t index) +{ + struct folio *folio; + struct page *page; + struct super_block *sb = ctx->inode->i_sb; + int ret; + + page = find_or_create_page(ctx->inode->i_mapping, index, GFP_KERNEL); + if (unlikely(!page)) { + erofs_err(sb, "failed to allocate page"); + return ERR_PTR(-ENOMEM); + } + + /* The content is already buffered in the address space */ + if (PageUptodate(page)) { + unlock_page(page); + return page; + } + + /* Or a new page cache is created, then read the content from fscache */ + folio = page_folio(page); + + ret = netfs_readpage(NULL, folio, &erofs_req_ops, ctx->cookie); + if (unlikely(ret || !PageUptodate(page))) { + erofs_err(sb, "failed to read from fscache"); + return ERR_PTR(-EINVAL); + } + + return page; +} + static int erofs_fscache_init_cookie(struct erofs_cookie_ctx *ctx, char *path) { struct fscache_cookie *cookie; diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h index 4ee4ff6774ba..10fb7ef26ddf 100644 --- a/fs/erofs/internal.h +++ b/fs/erofs/internal.h @@ -585,6 +585,9 @@ struct erofs_cookie_ctx *erofs_fscache_get_ctx(struct super_block *sb, char *path); void erofs_fscache_put_ctx(struct erofs_cookie_ctx *ctx); +struct page *erofs_readpage_from_fscache(struct erofs_cookie_ctx *ctx, + pgoff_t index); + #define EFSCORRUPTED EUCLEAN /* Filesystem is corrupted */ #endif /* __EROFS_INTERNAL_H */ diff --git a/fs/erofs/super.c b/fs/erofs/super.c index 141cabd01d32..0e5964d8b24b 100644 --- a/fs/erofs/super.c +++ b/fs/erofs/super.c @@ -334,25 +334,22 @@ static int erofs_init_devices(struct super_block *sb, static int erofs_read_superblock(struct super_block *sb) { - struct erofs_sb_info *sbi; + struct erofs_sb_info *sbi = EROFS_SB(sb); struct page *page; struct erofs_super_block *dsb; unsigned int blkszbits; void *data; int ret; - /* TODO: metadata path in nodev mode */ if (sb->s_bdev) page = read_mapping_page(sb->s_bdev->bd_inode->i_mapping, 0, NULL); else - page = ERR_PTR(-EOPNOTSUPP); + page = erofs_readpage_from_fscache(sbi->bootstrap, 0); if (IS_ERR(page)) { erofs_err(sb, "cannot read erofs superblock"); return PTR_ERR(page); } - sbi = EROFS_SB(sb); - data = kmap(page); dsb = (struct erofs_super_block *)(data + EROFS_SUPER_OFFSET); From patchwork Mon Dec 27 12:54:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12699747 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38BE8C433F5 for ; Mon, 27 Dec 2021 12:55:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236890AbhL0MzI (ORCPT ); Mon, 27 Dec 2021 07:55:08 -0500 Received: from out30-132.freemail.mail.aliyun.com ([115.124.30.132]:39581 "EHLO out30-132.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236868AbhL0MzC (ORCPT ); Mon, 27 Dec 2021 07:55:02 -0500 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R191e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04395;MF=jefflexu@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0V.xJoRI_1640609699; Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0V.xJoRI_1640609699) by smtp.aliyun-inc.com(127.0.0.1); Mon, 27 Dec 2021 20:55:00 +0800 From: Jeffle Xu To: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org Cc: linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: [PATCH v1 13/23] erofs: implement fscache-based data read Date: Mon, 27 Dec 2021 20:54:34 +0800 Message-Id: <20211227125444.21187-14-jefflexu@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211227125444.21187-1-jefflexu@linux.alibaba.com> References: <20211227125444.21187-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org This patch implements the data plane of reading data from bootstrap blob file over fscache. Be noted that currently compressed layout is not supported yet. Signed-off-by: Jeffle Xu --- fs/erofs/fscache.c | 91 +++++++++++++++++++++++++++++++++++++++++++++ fs/erofs/inode.c | 6 ++- fs/erofs/internal.h | 1 + 3 files changed, 97 insertions(+), 1 deletion(-) diff --git a/fs/erofs/fscache.c b/fs/erofs/fscache.c index 325f5663836b..bfcec831d58a 100644 --- a/fs/erofs/fscache.c +++ b/fs/erofs/fscache.c @@ -65,6 +65,97 @@ struct page *erofs_readpage_from_fscache(struct erofs_cookie_ctx *ctx, return page; } +static inline void do_copy_page(struct page *from, struct page *to, + size_t offset, size_t len) +{ + char *vfrom, *vto; + + vfrom = kmap_atomic(from); + vto = kmap_atomic(to); + memcpy(vto, vfrom + offset, len); + kunmap_atomic(vto); + kunmap_atomic(vfrom); +} + +static int erofs_fscache_do_readpage(struct file *file, struct page *page) +{ + struct inode *inode = page->mapping->host; + struct erofs_inode *vi = EROFS_I(inode); + struct super_block *sb = inode->i_sb; + struct erofs_map_blocks map; + erofs_off_t o_la, pa; + size_t offset, len; + struct page *ipage; + int ret; + + if (erofs_inode_is_data_compressed(vi->datalayout)) { + erofs_info(sb, "compressed layout not supported yet"); + return -EOPNOTSUPP; + } + + o_la = page_offset(page); + map.m_la = o_la; + + ret = erofs_map_blocks(inode, &map, EROFS_GET_BLOCKS_RAW); + if (ret) + return ret; + + if (!(map.m_flags & EROFS_MAP_MAPPED)) { + zero_user(page, 0, PAGE_SIZE); + return 0; + } + + /* + * 1) For FLAT_PLAIN/FLAT_INLINE layout, the output map.m_la shall be + * equal to o_la, and the output map.m_pa is exactly the physical + * address of o_la. + * 2) For CHUNK_BASED layout, the output map.m_la is rounded down to the + * nearest chunk boundary, and the output map.m_pa is actually the + * physical address of this chunk boundary. So we need to recalculate + * the actual physical address of o_la. + */ + pa = map.m_pa + o_la - map.m_la; + + ipage = erofs_get_meta_page(sb, erofs_blknr(pa)); + if (IS_ERR(ipage)) + return PTR_ERR(ipage); + + /* + * @offset refers to the page offset inside @ipage. + * 1) Except for the inline layout, the offset shall all be 0, and @pa + * shall be aligned with EROFS_BLKSIZ in this case. Thus we can + * conveniently get the offset from @pa. + * 2) While for the inline layout, the offset may be non-zero. Since + * currently only flat layout supports inline, we can calculate the + * offset from the corresponding physical address. + */ + offset = erofs_blkoff(pa); + len = min_t(u64, map.m_llen, PAGE_SIZE); + + do_copy_page(ipage, page, offset, len); + + unlock_page(ipage); + return 0; +} + +static int erofs_fscache_readpage(struct file *file, struct page *page) +{ + int ret; + + ret = erofs_fscache_do_readpage(file, page); + if (ret) + SetPageError(page); + else + SetPageUptodate(page); + + unlock_page(page); + return ret; +} + +const struct address_space_operations erofs_fscache_access_aops = { + .readpage = erofs_fscache_readpage, +}; + static int erofs_fscache_init_cookie(struct erofs_cookie_ctx *ctx, char *path) { struct fscache_cookie *cookie; diff --git a/fs/erofs/inode.c b/fs/erofs/inode.c index 2345f1de438e..452d147277c4 100644 --- a/fs/erofs/inode.c +++ b/fs/erofs/inode.c @@ -299,7 +299,11 @@ static int erofs_fill_inode(struct inode *inode, int isdir) err = z_erofs_fill_inode(inode); goto out_unlock; } - inode->i_mapping->a_ops = &erofs_raw_access_aops; + + if (inode->i_sb->s_bdev) + inode->i_mapping->a_ops = &erofs_raw_access_aops; + else + inode->i_mapping->a_ops = &erofs_fscache_access_aops; out_unlock: unlock_page(page); diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h index 10fb7ef26ddf..f3a1aa429a93 100644 --- a/fs/erofs/internal.h +++ b/fs/erofs/internal.h @@ -357,6 +357,7 @@ struct page *erofs_grab_cache_page_nowait(struct address_space *mapping, extern const struct super_operations erofs_sops; extern const struct address_space_operations erofs_raw_access_aops; +extern const struct address_space_operations erofs_fscache_access_aops; extern const struct address_space_operations z_erofs_aops; /* From patchwork Mon Dec 27 12:54:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12699748 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2CBA5C433EF for ; Mon, 27 Dec 2021 12:55:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236956AbhL0MzJ (ORCPT ); Mon, 27 Dec 2021 07:55:09 -0500 Received: from out30-57.freemail.mail.aliyun.com ([115.124.30.57]:53854 "EHLO out30-57.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236901AbhL0MzE (ORCPT ); Mon, 27 Dec 2021 07:55:04 -0500 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R181e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04400;MF=jefflexu@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0V.w7GYF_1640609701; Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0V.w7GYF_1640609701) by smtp.aliyun-inc.com(127.0.0.1); Mon, 27 Dec 2021 20:55:01 +0800 From: Jeffle Xu To: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org Cc: linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: [PATCH v1 14/23] erofs: register cookie context for data blobs Date: Mon, 27 Dec 2021 20:54:35 +0800 Message-Id: <20211227125444.21187-15-jefflexu@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211227125444.21187-1-jefflexu@linux.alibaba.com> References: <20211227125444.21187-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Similar to the multi device mode, erofs could be mounted from multiple blob files (one bootstrap blob file and optional multiple data blob files). In this case, each device slot contains the path of corresponding data blob file. This patch registers corresponding cookie context for each data blob file. Similarly, the cleanup routine shall be contained in both .kill_sb() and .put_super() callback for the same reason. Signed-off-by: Jeffle Xu --- fs/erofs/internal.h | 1 + fs/erofs/super.c | 12 +++++++++++- 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h index f3a1aa429a93..f60577a7aade 100644 --- a/fs/erofs/internal.h +++ b/fs/erofs/internal.h @@ -52,6 +52,7 @@ struct erofs_device_info { char *path; struct block_device *bdev; struct dax_device *dax_dev; + struct erofs_cookie_ctx *ctx; u32 blocks; u32 mapped_blkaddr; diff --git a/fs/erofs/super.c b/fs/erofs/super.c index 0e5964d8b24b..ea56122f7a35 100644 --- a/fs/erofs/super.c +++ b/fs/erofs/super.c @@ -287,6 +287,7 @@ static int erofs_init_devices(struct super_block *sb, idr_for_each_entry(&sbi->devs->tree, dif, id) { erofs_blk_t blk = erofs_blknr(pos); struct block_device *bdev; + struct erofs_cookie_ctx *ctx; if (!page || page->index != blk) { if (page) { @@ -315,7 +316,12 @@ static int erofs_init_devices(struct super_block *sb, dif->bdev = bdev; dif->dax_dev = fs_dax_get_by_bdev(bdev); } else { - /* TODO: multi devs in nodev mode */ + ctx = erofs_fscache_get_ctx(sb, dif->path); + if (IS_ERR(ctx)) { + err = PTR_ERR(ctx); + goto err_out; + } + dif->ctx = ctx; } dif->blocks = le32_to_cpu(dis->blocks); dif->mapped_blkaddr = le32_to_cpu(dis->mapped_blkaddr); @@ -755,6 +761,7 @@ static int erofs_release_device_info(int id, void *ptr, void *data) { struct erofs_device_info *dif = ptr; + erofs_fscache_put_ctx(dif->ctx); fs_put_dax(dif->dax_dev); if (dif->bdev) blkdev_put(dif->bdev, FMODE_READ | FMODE_EXCL); @@ -845,6 +852,9 @@ static void erofs_put_super(struct super_block *sb) iput(sbi->managed_cache); sbi->managed_cache = NULL; #endif + erofs_free_dev_context(sbi->devs); + sbi->devs = NULL; + erofs_fscache_put_ctx(sbi->bootstrap); sbi->bootstrap = NULL; } From patchwork Mon Dec 27 12:54:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12699757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE92DC433EF for ; Mon, 27 Dec 2021 12:55:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236923AbhL0Mzh (ORCPT ); Mon, 27 Dec 2021 07:55:37 -0500 Received: from out30-56.freemail.mail.aliyun.com ([115.124.30.56]:47004 "EHLO out30-56.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236913AbhL0MzF (ORCPT ); Mon, 27 Dec 2021 07:55:05 -0500 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R111e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04423;MF=jefflexu@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0V.w7GYT_1640609702; Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0V.w7GYT_1640609702) by smtp.aliyun-inc.com(127.0.0.1); Mon, 27 Dec 2021 20:55:02 +0800 From: Jeffle Xu To: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org Cc: linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: [PATCH v1 15/23] erofs: implement fscache-based data read for data blobs Date: Mon, 27 Dec 2021 20:54:36 +0800 Message-Id: <20211227125444.21187-16-jefflexu@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211227125444.21187-1-jefflexu@linux.alibaba.com> References: <20211227125444.21187-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org This patch implements the data plane of reading data from data blob file over fscache. Signed-off-by: Jeffle Xu --- fs/erofs/data.c | 3 +++ fs/erofs/fscache.c | 25 +++++++++++++++++++++++-- fs/erofs/internal.h | 1 + 3 files changed, 27 insertions(+), 2 deletions(-) diff --git a/fs/erofs/data.c b/fs/erofs/data.c index f0857c285fac..621769ab5753 100644 --- a/fs/erofs/data.c +++ b/fs/erofs/data.c @@ -166,6 +166,7 @@ int erofs_map_dev(struct super_block *sb, struct erofs_map_dev *map) /* primary device by default */ map->m_bdev = sb->s_bdev; map->m_daxdev = EROFS_SB(sb)->dax_dev; + map->m_ctx = EROFS_SB(sb)->bootstrap; if (map->m_deviceid) { down_read(&devs->rwsem); @@ -176,6 +177,7 @@ int erofs_map_dev(struct super_block *sb, struct erofs_map_dev *map) } map->m_bdev = dif->bdev; map->m_daxdev = dif->dax_dev; + map->m_ctx = dif->ctx; up_read(&devs->rwsem); } else if (devs->extra_devices) { down_read(&devs->rwsem); @@ -192,6 +194,7 @@ int erofs_map_dev(struct super_block *sb, struct erofs_map_dev *map) map->m_pa -= startoff; map->m_bdev = dif->bdev; map->m_daxdev = dif->dax_dev; + map->m_ctx = dif->ctx; break; } } diff --git a/fs/erofs/fscache.c b/fs/erofs/fscache.c index bfcec831d58a..7d4f6682e521 100644 --- a/fs/erofs/fscache.c +++ b/fs/erofs/fscache.c @@ -77,12 +77,24 @@ static inline void do_copy_page(struct page *from, struct page *to, kunmap_atomic(vfrom); } +static struct page *erofs_fscache_get_page(struct erofs_cookie_ctx *ctx, + erofs_blk_t blkaddr) +{ + struct page *page; + + page = erofs_readpage_from_fscache(ctx, blkaddr); + if (!IS_ERR(page)) + lock_page(page); + return page; +} + static int erofs_fscache_do_readpage(struct file *file, struct page *page) { struct inode *inode = page->mapping->host; struct erofs_inode *vi = EROFS_I(inode); struct super_block *sb = inode->i_sb; struct erofs_map_blocks map; + struct erofs_map_dev mdev; erofs_off_t o_la, pa; size_t offset, len; struct page *ipage; @@ -105,6 +117,15 @@ static int erofs_fscache_do_readpage(struct file *file, struct page *page) return 0; } + mdev = (struct erofs_map_dev) { + .m_deviceid = map.m_deviceid, + .m_pa = map.m_pa, + }; + + ret = erofs_map_dev(inode->i_sb, &mdev); + if (ret) + return ret; + /* * 1) For FLAT_PLAIN/FLAT_INLINE layout, the output map.m_la shall be * equal to o_la, and the output map.m_pa is exactly the physical @@ -114,9 +135,9 @@ static int erofs_fscache_do_readpage(struct file *file, struct page *page) * physical address of this chunk boundary. So we need to recalculate * the actual physical address of o_la. */ - pa = map.m_pa + o_la - map.m_la; + pa = mdev.m_pa + o_la - map.m_la; - ipage = erofs_get_meta_page(sb, erofs_blknr(pa)); + ipage = erofs_fscache_get_page(mdev.m_ctx, erofs_blknr(pa)); if (IS_ERR(ipage)) return PTR_ERR(ipage); diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h index f60577a7aade..b21c3751e88d 100644 --- a/fs/erofs/internal.h +++ b/fs/erofs/internal.h @@ -447,6 +447,7 @@ static inline int z_erofs_map_blocks_iter(struct inode *inode, struct erofs_map_dev { struct block_device *m_bdev; struct dax_device *m_daxdev; + struct erofs_cookie_ctx *m_ctx; erofs_off_t m_pa; unsigned int m_deviceid; From patchwork Mon Dec 27 12:54:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12699756 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6948C433FE for ; Mon, 27 Dec 2021 12:55:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236887AbhL0Mzg (ORCPT ); Mon, 27 Dec 2021 07:55:36 -0500 Received: from out30-54.freemail.mail.aliyun.com ([115.124.30.54]:48501 "EHLO out30-54.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236926AbhL0MzH (ORCPT ); Mon, 27 Dec 2021 07:55:07 -0500 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R201e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04426;MF=jefflexu@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0V.wXYVp_1640609703; Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0V.wXYVp_1640609703) by smtp.aliyun-inc.com(127.0.0.1); Mon, 27 Dec 2021 20:55:04 +0800 From: Jeffle Xu To: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org Cc: linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: [PATCH v1 16/23] erofs: add 'uuid' mount option Date: Mon, 27 Dec 2021 20:54:37 +0800 Message-Id: <20211227125444.21187-17-jefflexu@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211227125444.21187-1-jefflexu@linux.alibaba.com> References: <20211227125444.21187-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Introduce 'uuid' mount option to enable the nodev mode, in which erofs could be mounted from blob files instead of blkdev. By then users could specify the path of bootstrap blob file containing the complete erofs image. Signed-off-by: Jeffle Xu --- fs/erofs/super.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/fs/erofs/super.c b/fs/erofs/super.c index ea56122f7a35..4f17aedc4acd 100644 --- a/fs/erofs/super.c +++ b/fs/erofs/super.c @@ -451,6 +451,7 @@ enum { Opt_dax, Opt_dax_enum, Opt_device, + Opt_uuid, Opt_err }; @@ -475,6 +476,7 @@ static const struct fs_parameter_spec erofs_fs_parameters[] = { fsparam_flag("dax", Opt_dax), fsparam_enum("dax", Opt_dax_enum, erofs_dax_param_enums), fsparam_string("device", Opt_device), + fsparam_string("uuid", Opt_uuid), {} }; @@ -570,6 +572,12 @@ static int erofs_fc_parse_param(struct fs_context *fc, } ++ctx->devs->extra_devices; break; + case Opt_uuid: + kfree(ctx->opt.uuid); + ctx->opt.uuid = kstrdup(param->string, GFP_KERNEL); + if (!ctx->opt.uuid) + return -ENOMEM; + break; default: return -ENOPARAM; } @@ -784,6 +792,7 @@ static void erofs_fc_free(struct fs_context *fc) struct erofs_fs_context *ctx = fc->fs_private; erofs_free_dev_context(ctx->devs); + kfree(ctx->opt.uuid); kfree(ctx); } From patchwork Mon Dec 27 12:54:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12699749 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32F89C433EF for ; Mon, 27 Dec 2021 12:55:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236999AbhL0MzM (ORCPT ); Mon, 27 Dec 2021 07:55:12 -0500 Received: from out30-42.freemail.mail.aliyun.com ([115.124.30.42]:35575 "EHLO out30-42.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236934AbhL0MzH (ORCPT ); Mon, 27 Dec 2021 07:55:07 -0500 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R191e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04395;MF=jefflexu@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0V.xJoSA_1640609704; Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0V.xJoSA_1640609704) by smtp.aliyun-inc.com(127.0.0.1); Mon, 27 Dec 2021 20:55:05 +0800 From: Jeffle Xu To: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org Cc: linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: [PATCH v1 17/23] netfs: support on demand read Date: Mon, 27 Dec 2021 20:54:38 +0800 Message-Id: <20211227125444.21187-18-jefflexu@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211227125444.21187-1-jefflexu@linux.alibaba.com> References: <20211227125444.21187-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Add demand_read() callback to netfs_cache_ops to implement demand reading. The precondition for implementing demand reading semantic is that, all blob files are sparse and have been placed under corresponding directory on the first beginning. When upper fs starts to access the blob file, it will "cache miss" (hit the hole) and then .issue_op() callback will be called to prepare the data. Then the working flow is as described below. The .issue_op() callback could be implemented by netfs_demand_read() helper, which will in turn call .demand_read() callback of corresponding fscache backend to prepare the data. The implementation of .demand_read() callback can be backend specific. The following patch will introduce an implementation of .demand_read() callback for cachefiles, which will notify user daemon the requested file range to read. The .demand_read() callback will get blocked until the user daemon has prepared the corresponding data. Then once .demand_read() callback returns with 0, it is guaranteed that the requested data has been ready. In this case, transform this IO request to NETFS_READ_FROM_CACHE state, initiate an incomplete completion and then retry to read from backing file. Signed-off-by: Jeffle Xu --- fs/netfs/read_helper.c | 26 ++++++++++++++++++++++++++ include/linux/netfs.h | 4 ++++ 2 files changed, 30 insertions(+) diff --git a/fs/netfs/read_helper.c b/fs/netfs/read_helper.c index ca84918b6b5d..8aac65132b67 100644 --- a/fs/netfs/read_helper.c +++ b/fs/netfs/read_helper.c @@ -1006,6 +1006,32 @@ int netfs_readpage(struct file *file, } EXPORT_SYMBOL(netfs_readpage); +void netfs_demand_read(struct netfs_read_subrequest *subreq) +{ + struct netfs_read_request *rreq = subreq->rreq; + struct netfs_cache_resources *cres = &rreq->cache_resources; + loff_t start_pos; + size_t len; + int ret; + + start_pos = subreq->start + subreq->transferred; + len = subreq->len - subreq->transferred; + + /* + * In success case (ret == 0), user daemon has downloaded data for us, + * thus transform to NETFS_READ_FROM_CACHE state and advertise that 0 + * byte readed, so that the request will enter into INCOMPLETE state and + * retry to read from backing file. + */ + ret = cres->ops->demand_read(cres, start_pos, len); + if (!ret) { + subreq->source = NETFS_READ_FROM_CACHE; + __clear_bit(NETFS_SREQ_WRITE_TO_CACHE, &subreq->flags); + } + + netfs_subreq_terminated(subreq, ret, false); +} + /* * Prepare a folio for writing without reading first * @folio: The folio being prepared diff --git a/include/linux/netfs.h b/include/linux/netfs.h index b46c39d98bbd..80a738762deb 100644 --- a/include/linux/netfs.h +++ b/include/linux/netfs.h @@ -244,6 +244,9 @@ struct netfs_cache_ops { int (*prepare_write)(struct netfs_cache_resources *cres, loff_t *_start, size_t *_len, loff_t i_size, bool no_space_allocated_yet); + + int (*demand_read)(struct netfs_cache_resources *cres, + loff_t start_pos, size_t len); }; struct readahead_control; @@ -259,6 +262,7 @@ extern int netfs_write_begin(struct file *, struct address_space *, void **, const struct netfs_read_request_ops *, void *); +extern void netfs_demand_read(struct netfs_read_subrequest *); extern void netfs_subreq_terminated(struct netfs_read_subrequest *, ssize_t, bool); extern void netfs_stats_show(struct seq_file *); From patchwork Mon Dec 27 12:54:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12699755 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DE01C433F5 for ; Mon, 27 Dec 2021 12:55:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237120AbhL0Mzb (ORCPT ); Mon, 27 Dec 2021 07:55:31 -0500 Received: from out30-56.freemail.mail.aliyun.com ([115.124.30.56]:46668 "EHLO out30-56.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232241AbhL0MzJ (ORCPT ); Mon, 27 Dec 2021 07:55:09 -0500 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R111e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04407;MF=jefflexu@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0V.vxTKl_1640609705; Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0V.vxTKl_1640609705) by smtp.aliyun-inc.com(127.0.0.1); Mon, 27 Dec 2021 20:55:06 +0800 From: Jeffle Xu To: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org Cc: linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: [PATCH v1 18/23] cachefiles: use idr tree managing pending demand read Date: Mon, 27 Dec 2021 20:54:39 +0800 Message-Id: <20211227125444.21187-19-jefflexu@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211227125444.21187-1-jefflexu@linux.alibaba.com> References: <20211227125444.21187-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org This is in prep for implementing .demand_read() callback for cachefiles. In the following patch, .demand_read() callback is responsible for notifying user daemon the requested file range (hole) to read. Then .demand_read() callback will get blocked until user daemon has prepared data and filled the hole. This patch introduces an idr tree to manage all these pending demand-read IO. Signed-off-by: Jeffle Xu --- fs/cachefiles/daemon.c | 3 +++ fs/cachefiles/internal.h | 4 ++++ 2 files changed, 7 insertions(+) diff --git a/fs/cachefiles/daemon.c b/fs/cachefiles/daemon.c index 892a9bdba53f..01496fa8c263 100644 --- a/fs/cachefiles/daemon.c +++ b/fs/cachefiles/daemon.c @@ -118,6 +118,8 @@ static int cachefiles_daemon_open(struct inode *inode, struct file *file) INIT_LIST_HEAD(&cache->volumes); INIT_LIST_HEAD(&cache->object_list); spin_lock_init(&cache->object_list_lock); + idr_init(&cache->reqs); + spin_lock_init(&cache->reqs_lock); /* set default caching limits * - limit at 1% free space and/or free files @@ -152,6 +154,7 @@ static int cachefiles_daemon_release(struct inode *inode, struct file *file) cachefiles_daemon_unbind(cache); /* clean up the control file interface */ + idr_destroy(&cache->reqs); cache->cachefilesd = NULL; file->private_data = NULL; cachefiles_open = 0; diff --git a/fs/cachefiles/internal.h b/fs/cachefiles/internal.h index a8e6500889d7..af8ac8107f77 100644 --- a/fs/cachefiles/internal.h +++ b/fs/cachefiles/internal.h @@ -15,6 +15,7 @@ #include #include #include +#include #define CACHEFILES_DIO_BLOCK_SIZE 4096 @@ -102,6 +103,9 @@ struct cachefiles_cache { char *rootdirname; /* name of cache root directory */ char *secctx; /* LSM security context */ char *tag; /* cache binding tag */ + + struct idr reqs; + spinlock_t reqs_lock; }; #include From patchwork Mon Dec 27 12:54:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12699751 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E202AC433EF for ; Mon, 27 Dec 2021 12:55:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237036AbhL0MzR (ORCPT ); Mon, 27 Dec 2021 07:55:17 -0500 Received: from out30-130.freemail.mail.aliyun.com ([115.124.30.130]:49200 "EHLO out30-130.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236957AbhL0MzK (ORCPT ); Mon, 27 Dec 2021 07:55:10 -0500 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R231e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04357;MF=jefflexu@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0V.w7GZ4_1640609706; Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0V.w7GZ4_1640609706) by smtp.aliyun-inc.com(127.0.0.1); Mon, 27 Dec 2021 20:55:07 +0800 From: Jeffle Xu To: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org Cc: linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: [PATCH v1 19/23] cachefiles: implement .demand_read() for demand read Date: Mon, 27 Dec 2021 20:54:40 +0800 Message-Id: <20211227125444.21187-20-jefflexu@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211227125444.21187-1-jefflexu@linux.alibaba.com> References: <20211227125444.21187-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Implement .demand_read() callback for cachefiels backend. .demand_read() callback is responsible for notifying user daemon the pending request to process, and will get blocked until user daemon has prepared data and filled the hole. Signed-off-by: Jeffle Xu Reported-by: kernel test robot Reported-by: kernel test robot --- fs/cachefiles/internal.h | 12 +++++++++ fs/cachefiles/io.c | 56 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 68 insertions(+) diff --git a/fs/cachefiles/internal.h b/fs/cachefiles/internal.h index af8ac8107f77..eeb6ad7dcd49 100644 --- a/fs/cachefiles/internal.h +++ b/fs/cachefiles/internal.h @@ -61,6 +61,18 @@ struct cachefiles_object { #define CACHEFILES_OBJECT_USING_TMPFILE 0 /* Have an unlinked tmpfile */ }; +struct cachefiles_req_in { + uint64_t id; + uint64_t off; + uint64_t len; + char path[NAME_MAX]; +}; + +struct cachefiles_req { + struct completion done; + struct cachefiles_req_in req_in; +}; + /* * Cache files cache definition */ diff --git a/fs/cachefiles/io.c b/fs/cachefiles/io.c index 60b1eac2ce78..376603e5ed99 100644 --- a/fs/cachefiles/io.c +++ b/fs/cachefiles/io.c @@ -539,12 +539,68 @@ static void cachefiles_end_operation(struct netfs_cache_resources *cres) fscache_end_cookie_access(fscache_cres_cookie(cres), fscache_access_io_end); } +static struct cachefiles_req *cachefiles_alloc_req(struct cachefiles_object *object, + loff_t start_pos, + size_t len) +{ + struct cachefiles_req *req; + struct cachefiles_req_in *req_in; + + req = kzalloc(sizeof(*req), GFP_KERNEL); + if (!req) + return NULL; + + req_in = &req->req_in; + + req_in->off = start_pos; + req_in->len = len; + strncpy(req_in->path, object->d_name, sizeof(req_in->path)); + + init_completion(&req->done); + + return req; +} + +int cachefiles_demand_read(struct netfs_cache_resources *cres, + loff_t start_pos, size_t len) +{ + struct cachefiles_object *object; + struct cachefiles_cache *cache; + struct cachefiles_req *req; + int ret; + + object = cachefiles_cres_object(cres); + cache = object->volume->cache; + + req = cachefiles_alloc_req(object, start_pos, len); + if (!req) + return -ENOMEM; + + spin_lock(&cache->reqs_lock); + ret = idr_alloc(&cache->reqs, req, 0, 0, GFP_KERNEL); + if (ret >= 0) + req->req_in.id = ret; + spin_unlock(&cache->reqs_lock); + if (ret < 0) { + kfree(req); + return -ENOMEM; + } + + wake_up_all(&cache->daemon_pollwq); + + wait_for_completion(&req->done); + kfree(req); + + return 0; +} + static const struct netfs_cache_ops cachefiles_netfs_cache_ops = { .end_operation = cachefiles_end_operation, .read = cachefiles_read, .write = cachefiles_write, .prepare_read = cachefiles_prepare_read, .prepare_write = cachefiles_prepare_write, + .demand_read = cachefiles_demand_read, }; /* From patchwork Mon Dec 27 12:54:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12699750 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C293C433EF for ; Mon, 27 Dec 2021 12:55:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233717AbhL0MzQ (ORCPT ); Mon, 27 Dec 2021 07:55:16 -0500 Received: from out4436.biz.mail.alibaba.com ([47.88.44.36]:6852 "EHLO out4436.biz.mail.alibaba.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236921AbhL0MzL (ORCPT ); Mon, 27 Dec 2021 07:55:11 -0500 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R131e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04394;MF=jefflexu@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0V.xJoSZ_1640609708; Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0V.xJoSZ_1640609708) by smtp.aliyun-inc.com(127.0.0.1); Mon, 27 Dec 2021 20:55:08 +0800 From: Jeffle Xu To: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org Cc: linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: [PATCH v1 20/23] cachefiles: implement .poll() for demand read Date: Mon, 27 Dec 2021 20:54:41 +0800 Message-Id: <20211227125444.21187-21-jefflexu@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211227125444.21187-1-jefflexu@linux.alibaba.com> References: <20211227125444.21187-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org User daemon needs to poll on the devnode, and will be notified once there's pending request to process. Signed-off-by: Jeffle Xu --- fs/cachefiles/daemon.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/fs/cachefiles/daemon.c b/fs/cachefiles/daemon.c index 01496fa8c263..311dcd911a85 100644 --- a/fs/cachefiles/daemon.c +++ b/fs/cachefiles/daemon.c @@ -29,6 +29,8 @@ static ssize_t cachefiles_daemon_write(struct file *, const char __user *, size_t, loff_t *); static __poll_t cachefiles_daemon_poll(struct file *, struct poll_table_struct *); +static __poll_t cachefiles_demand_poll(struct file *, + struct poll_table_struct *); static int cachefiles_daemon_frun(struct cachefiles_cache *, char *); static int cachefiles_daemon_fcull(struct cachefiles_cache *, char *); static int cachefiles_daemon_fstop(struct cachefiles_cache *, char *); @@ -62,6 +64,7 @@ const struct file_operations cachefiles_demand_fops = { .open = cachefiles_daemon_open, .release = cachefiles_daemon_release, .write = cachefiles_daemon_write, + .poll = cachefiles_demand_poll, .llseek = noop_llseek, }; @@ -319,6 +322,21 @@ static __poll_t cachefiles_daemon_poll(struct file *file, return mask; } +static __poll_t cachefiles_demand_poll(struct file *file, + struct poll_table_struct *poll) +{ + struct cachefiles_cache *cache = file->private_data; + __poll_t mask; + + poll_wait(file, &cache->daemon_pollwq, poll); + mask = 0; + + if (!idr_is_empty(&cache->reqs)) + mask |= EPOLLIN; + + return mask; +} + /* * Give a range error for cache space constraints * - can be tail-called From patchwork Mon Dec 27 12:54:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12699752 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 000FFC4332F for ; Mon, 27 Dec 2021 12:55:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237037AbhL0MzS (ORCPT ); Mon, 27 Dec 2021 07:55:18 -0500 Received: from out30-132.freemail.mail.aliyun.com ([115.124.30.132]:35702 "EHLO out30-132.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236978AbhL0MzM (ORCPT ); Mon, 27 Dec 2021 07:55:12 -0500 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R141e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04426;MF=jefflexu@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0V.xJoSm_1640609709; Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0V.xJoSm_1640609709) by smtp.aliyun-inc.com(127.0.0.1); Mon, 27 Dec 2021 20:55:10 +0800 From: Jeffle Xu To: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org Cc: linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: [PATCH v1 21/23] cachefiles: implement .read() for demand read Date: Mon, 27 Dec 2021 20:54:42 +0800 Message-Id: <20211227125444.21187-22-jefflexu@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211227125444.21187-1-jefflexu@linux.alibaba.com> References: <20211227125444.21187-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Once notified, user daemon need to get more details of the request by reading the devnode. The readed information includes the file range of the request, with which user daemon could somehow prepare corresponding data (e.g. download from remote through network) and fill the hole. Signed-off-by: Jeffle Xu --- fs/cachefiles/daemon.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/fs/cachefiles/daemon.c b/fs/cachefiles/daemon.c index 311dcd911a85..ce4cc5617dfc 100644 --- a/fs/cachefiles/daemon.c +++ b/fs/cachefiles/daemon.c @@ -29,6 +29,8 @@ static ssize_t cachefiles_daemon_write(struct file *, const char __user *, size_t, loff_t *); static __poll_t cachefiles_daemon_poll(struct file *, struct poll_table_struct *); +static ssize_t cachefiles_demand_read(struct file *, char __user *, size_t, + loff_t *); static __poll_t cachefiles_demand_poll(struct file *, struct poll_table_struct *); static int cachefiles_daemon_frun(struct cachefiles_cache *, char *); @@ -63,6 +65,7 @@ const struct file_operations cachefiles_demand_fops = { .owner = THIS_MODULE, .open = cachefiles_daemon_open, .release = cachefiles_daemon_release, + .read = cachefiles_demand_read, .write = cachefiles_daemon_write, .poll = cachefiles_demand_poll, .llseek = noop_llseek, @@ -322,6 +325,32 @@ static __poll_t cachefiles_daemon_poll(struct file *file, return mask; } +static ssize_t cachefiles_demand_read(struct file *file, char __user *_buffer, + size_t buflen, loff_t *pos) +{ + struct cachefiles_cache *cache = file->private_data; + struct cachefiles_req *req; + int n, id = 0; + + if (!test_bit(CACHEFILES_READY, &cache->flags)) + return 0; + + spin_lock(&cache->reqs_lock); + req = idr_get_next(&cache->reqs, &id); + spin_unlock(&cache->reqs_lock); + if (!req) + return 0; + + n = sizeof(req->req_in); + if (n > buflen) + return -EMSGSIZE; + + if (copy_to_user(_buffer, &req->req_in, n) != 0) + return -EFAULT; + + return n; +} + static __poll_t cachefiles_demand_poll(struct file *file, struct poll_table_struct *poll) { From patchwork Mon Dec 27 12:54:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12699753 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6E79C433F5 for ; Mon, 27 Dec 2021 12:55:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237058AbhL0MzT (ORCPT ); Mon, 27 Dec 2021 07:55:19 -0500 Received: from out30-42.freemail.mail.aliyun.com ([115.124.30.42]:45133 "EHLO out30-42.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237000AbhL0MzN (ORCPT ); Mon, 27 Dec 2021 07:55:13 -0500 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R161e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04357;MF=jefflexu@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0V.wXYWr_1640609710; Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0V.wXYWr_1640609710) by smtp.aliyun-inc.com(127.0.0.1); Mon, 27 Dec 2021 20:55:11 +0800 From: Jeffle Xu To: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org Cc: linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: [PATCH v1 22/23] cachefiles: add done command for demand read Date: Mon, 27 Dec 2021 20:54:43 +0800 Message-Id: <20211227125444.21187-23-jefflexu@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211227125444.21187-1-jefflexu@linux.alibaba.com> References: <20211227125444.21187-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Once has prepared data and filled the hole, user daemon could notify cachefiles backend by writing 'done' command to devnode. With done command, the id originally readed from devnode is transferred back to cachefiles backend. This id identifies the position of the completed request inside the idr tree, and thus cachefiles backend could find the corresponding completed request once gets notified. Signed-off-by: Jeffle Xu --- fs/cachefiles/daemon.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/fs/cachefiles/daemon.c b/fs/cachefiles/daemon.c index ce4cc5617dfc..36d0bcf5fc3f 100644 --- a/fs/cachefiles/daemon.c +++ b/fs/cachefiles/daemon.c @@ -48,6 +48,7 @@ static int cachefiles_daemon_tag(struct cachefiles_cache *, char *); static int cachefiles_daemon_mode(struct cachefiles_cache *, char *); static int cachefiles_daemon_bind(struct cachefiles_cache *, char *); static void cachefiles_daemon_unbind(struct cachefiles_cache *); +static int cachefiles_daemon_done(struct cachefiles_cache *, char *); static unsigned long cachefiles_open; @@ -91,6 +92,7 @@ static const struct cachefiles_daemon_cmd cachefiles_daemon_cmds[] = { { "secctx", cachefiles_daemon_secctx }, { "tag", cachefiles_daemon_tag }, { "mode", cachefiles_daemon_mode }, + { "done", cachefiles_daemon_done }, { "", NULL } }; @@ -604,6 +606,38 @@ static int cachefiles_daemon_tag(struct cachefiles_cache *cache, char *args) return 0; } +/* + * Request completion + * - command: "done " + */ +static int cachefiles_daemon_done(struct cachefiles_cache *cache, char *args) +{ + unsigned long id; + int ret; + struct cachefiles_req *req; + + _enter(",%s", args); + + if (!*args) { + pr_err("Empty id specified\n"); + return -EINVAL; + } + + ret = kstrtoul(args, 0, &id); + if (ret) + return ret; + + spin_lock(&cache->reqs_lock); + req = idr_remove(&cache->reqs, id); + spin_unlock(&cache->reqs_lock); + if (!req) + return -EINVAL; + + complete(&req->done); + + return 0; +} + /* * Request a node in the cache be culled from the current working directory * - command: "cull " From patchwork Mon Dec 27 12:54:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jingbo Xu X-Patchwork-Id: 12699754 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D42AC433FE for ; Mon, 27 Dec 2021 12:55:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237088AbhL0MzY (ORCPT ); Mon, 27 Dec 2021 07:55:24 -0500 Received: from out30-132.freemail.mail.aliyun.com ([115.124.30.132]:48603 "EHLO out30-132.freemail.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237029AbhL0MzQ (ORCPT ); Mon, 27 Dec 2021 07:55:16 -0500 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R631e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04426;MF=jefflexu@linux.alibaba.com;NM=1;PH=DS;RN=12;SR=0;TI=SMTPD_---0V.wXYX3_1640609711; Received: from localhost(mailfrom:jefflexu@linux.alibaba.com fp:SMTPD_---0V.wXYX3_1640609711) by smtp.aliyun-inc.com(127.0.0.1); Mon, 27 Dec 2021 20:55:12 +0800 From: Jeffle Xu To: dhowells@redhat.com, linux-cachefs@redhat.com, xiang@kernel.org, chao@kernel.org, linux-erofs@lists.ozlabs.org Cc: linux-fsdevel@vger.kernel.org, joseph.qi@linux.alibaba.com, bo.liu@linux.alibaba.com, tao.peng@linux.alibaba.com, gerry@linux.alibaba.com, eguan@linux.alibaba.com, linux-kernel@vger.kernel.org Subject: [PATCH v1 23/23] erofs: support on demand read Date: Mon, 27 Dec 2021 20:54:44 +0800 Message-Id: <20211227125444.21187-24-jefflexu@linux.alibaba.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20211227125444.21187-1-jefflexu@linux.alibaba.com> References: <20211227125444.21187-1-jefflexu@linux.alibaba.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org Implement .issue_op() callback to support demand reading. Signed-off-by: Jeffle Xu --- fs/erofs/fscache.c | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/fs/erofs/fscache.c b/fs/erofs/fscache.c index 7d4f6682e521..bd64dbe1b0be 100644 --- a/fs/erofs/fscache.c +++ b/fs/erofs/fscache.c @@ -18,13 +18,7 @@ static void erofs_priv_cleanup(struct address_space *mapping, void *netfs_priv) static void erofs_issue_op(struct netfs_read_subrequest *subreq) { - /* - * TODO: implement demand-read logic later. - * We rely on user daemon to prepare blob files under corresponding - * directory, and we can reach here if blob files don't exist. - */ - - netfs_subreq_terminated(subreq, -EOPNOTSUPP, false); + netfs_demand_read(subreq); } const struct netfs_read_request_ops erofs_req_ops = {