| Message ID | 20190815044155.88483-1-gaoxiang25@huawei.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | erofs: promote erofs from staging v8 | expand |
On Thu, Aug 15, 2019 at 12:41:31PM +0800, Gao Xiang wrote: > [I strip the previous cover letter, the old one can be found in v6: > https://lore.kernel.org/r/20190802125347.166018-1-gaoxiang25@huawei.com/] > > We'd like to submit a formal moving patch applied to staging tree > for 5.4, before that we'd like to hear if there are some ACKs, > suggestions or NAKs, objections of EROFS. Therefore, we can improve > it in this round or rethink about the whole thing. > > As related materials mentioned [1] [2], the goal of EROFS is to > save extra storage space with guaranteed end-to-end performance > for read-only files, which has better performance over exist Linux > compression filesystems based on fixed-sized output compression > and inplace decompression. It even has better performance in > a large compression ratio range compared with generic uncompressed > filesystems with proper CPU-storage combinations. And we think this > direction is correct and a dedicated kernel team is continuously / > actively working on improving it, enough testers and beta / end > users using it. > > EROFS has been applied to almost all in-service HUAWEI smartphones > (Yes, the number is still increasing by time) and it seems like > a success. It can be used in more wider scenarios. We think it's > useful for Linux / Android OS community and it's the time moving > out of staging. > > In order to get started, latest stable mkfs.erofs is available at > > git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git -b dev > > with README in the repository. > > We are still tuning sequential read performance for ultra-fast > speed NVME SSDs like Samsung 970PRO, but at least now you can > try on your PC with some data with proper compression ratio, > the latest Linux kernel, USB stick for convenience sake and > a not very old-fashioned CPU. There are also benchmarks available > in the above materials mentioned. > > EROFS is a self-contained filesystem driver. Although there are > still some TODOs to be more generic, we will actively keep on > developping / tuning EROFS with the evolution of Linux kernel > as the other in-kernel filesystems. > > As I mentioned before in LSF/MM 2019, in the future, we'd like > to generalize the decompression engine into a library for other > fses to use after the whole system is mature like fscrypt. > However, such metadata should be designed respectively for > each fs, and synchronous metadata read cost will be larger > than EROFS because of those ondisk limitation. Therefore EROFS > is still a better choice for read-only scenarios. > > EROFS is now ready for reviewing and moving, and the code is > already cleaned up as shiny floors... Please kindly take some > precious time, share your comments about EROFS and let us know > your opinion about this. It's really important for us since > generally speaking, we like to use Linux _in-tree_ stuffs rather > than lack of supported out-of-tree / orphan stuffs as well. I know everyone is busy, but given the length this has been in staging, and the constant good progress toward cleaning it all up that has been happening, I want to get this moved out of staging soon. So, unless there are any objections, I'll take this patchset in a week into my staging tree to move the filesystem into the "real" part of the kernel. thanks, greg k-h
On Thu, Aug 15, 2019 at 2:06 AM Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote: > > I know everyone is busy, but given the length this has been in staging, > and the constant good progress toward cleaning it all up that has been > happening, I want to get this moved out of staging soon. Since it doesn't touch anything outside of its own filesystem, I have no real objections. We've never had huge problems with odd filesystems. I read through the patches to look for syntactic stuff (ie very much *not* looking at actual code working or not), and had only one comment. It's not critical, but it would be nice to do as part of (or before) the "get it out of staging". Linus
Hi Linus, On Thu, Aug 15, 2019 at 09:18:12AM -0700, Linus Torvalds wrote: > On Thu, Aug 15, 2019 at 2:06 AM Greg Kroah-Hartman > <gregkh@linuxfoundation.org> wrote: > > > > I know everyone is busy, but given the length this has been in staging, > > and the constant good progress toward cleaning it all up that has been > > happening, I want to get this moved out of staging soon. > > Since it doesn't touch anything outside of its own filesystem, I have > no real objections. We've never had huge problems with odd > filesystems. > > I read through the patches to look for syntactic stuff (ie very much > *not* looking at actual code working or not), and had only one > comment. It's not critical, but it would be nice to do as part of (or > before) the "get it out of staging". Thanks for your kind reply! OK, I will submit a patch later to address your comment and a pending formal moving patch with a suggestion by Stephen earlier for Greg as well. Thanks, Gao Xiang > > Linus
Hi Greg, On Thu, Aug 15, 2019 at 11:06:03AM +0200, Greg Kroah-Hartman wrote: > On Thu, Aug 15, 2019 at 12:41:31PM +0800, Gao Xiang wrote: > > [I strip the previous cover letter, the old one can be found in v6: > > https://lore.kernel.org/r/20190802125347.166018-1-gaoxiang25@huawei.com/] > > > > We'd like to submit a formal moving patch applied to staging tree > > for 5.4, before that we'd like to hear if there are some ACKs, > > suggestions or NAKs, objections of EROFS. Therefore, we can improve > > it in this round or rethink about the whole thing. > > > > As related materials mentioned [1] [2], the goal of EROFS is to > > save extra storage space with guaranteed end-to-end performance > > for read-only files, which has better performance over exist Linux > > compression filesystems based on fixed-sized output compression > > and inplace decompression. It even has better performance in > > a large compression ratio range compared with generic uncompressed > > filesystems with proper CPU-storage combinations. And we think this > > direction is correct and a dedicated kernel team is continuously / > > actively working on improving it, enough testers and beta / end > > users using it. > > > > EROFS has been applied to almost all in-service HUAWEI smartphones > > (Yes, the number is still increasing by time) and it seems like > > a success. It can be used in more wider scenarios. We think it's > > useful for Linux / Android OS community and it's the time moving > > out of staging. > > > > In order to get started, latest stable mkfs.erofs is available at > > > > git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git -b dev > > > > with README in the repository. > > > > We are still tuning sequential read performance for ultra-fast > > speed NVME SSDs like Samsung 970PRO, but at least now you can > > try on your PC with some data with proper compression ratio, > > the latest Linux kernel, USB stick for convenience sake and > > a not very old-fashioned CPU. There are also benchmarks available > > in the above materials mentioned. > > > > EROFS is a self-contained filesystem driver. Although there are > > still some TODOs to be more generic, we will actively keep on > > developping / tuning EROFS with the evolution of Linux kernel > > as the other in-kernel filesystems. > > > > As I mentioned before in LSF/MM 2019, in the future, we'd like > > to generalize the decompression engine into a library for other > > fses to use after the whole system is mature like fscrypt. > > However, such metadata should be designed respectively for > > each fs, and synchronous metadata read cost will be larger > > than EROFS because of those ondisk limitation. Therefore EROFS > > is still a better choice for read-only scenarios. > > > > EROFS is now ready for reviewing and moving, and the code is > > already cleaned up as shiny floors... Please kindly take some > > precious time, share your comments about EROFS and let us know > > your opinion about this. It's really important for us since > > generally speaking, we like to use Linux _in-tree_ stuffs rather > > than lack of supported out-of-tree / orphan stuffs as well. > > I know everyone is busy, but given the length this has been in staging, > and the constant good progress toward cleaning it all up that has been > happening, I want to get this moved out of staging soon. > > So, unless there are any objections, I'll take this patchset in a week > into my staging tree to move the filesystem into the "real" part of the > kernel. It seem that the time is passed, as a brief conclusion, it seems we don't get "real" objection in the previous week. During these days, we have enhanced our robustness against corrupted images by our first fuzzer based on mkfs.erofs these days (since it's a RO fs, it will generate reproductable images). Although the original intended use case of EROFS is on the top of dm-verity for Android, we still want to gain more wider use so we quickly build a fuzzer and addresses them (yes, we will develop another independent fuzzer tools as well.) And thanks all people for all useful suggestions these days, and we think these wonderful fses (ext4/xfs/btrfs/...) have awesome rich tools, that's also our next step to address on, especially debugging tools. As a Newborn communities, we only have a few paid-job people working on that, but we are doing our best on EROFS, please kindly give us some time to grow up (I personally speed my all spare/working time on EROFS from its start), and apply EROFS to more wider use like what we did successfully for many many HUAWEI smartphones... As Greg said before [1], we have already proven the advantage of EROFS solutions, the next step is to develop it more actively... And we would also like to generalize the decompression engine into a library for other general fses to use (we're very happy to share our efforts), it seems interesting to other fs as well [2]. I sent several patchsets from July 4, 2019 (v1-v8), Cc most of fs people at the first beginning and get responses and suggestions from people (Ted, Pavel, Eric, Stephen, Amir, David, Jan, Richard, Linus...), so could you kindly consider the following moving patch [3] and let us join, contribute Linux "real" part community more actively, again, we have a steady stream of work on EROFS, and will do our best on it. Thank you all very much! Sorry about my English... [1] https://lore.kernel.org/r/20190618064523.GA6015@kroah.com/ [2] https://lore.kernel.org/r/20190820162510.GC10232@mit.edu/ [3] https://lore.kernel.org/r/20190817082313.21040-1-hsiangkao@aol.com/ Thanks, Gao Xiang > > thanks, > > greg k-h
on 2019/8/15 at 12:41, Gao Xiang wrote: > [I strip the previous cover letter, the old one can be found in v6: > https://lore.kernel.org/r/20190802125347.166018-1-gaoxiang25@huawei.com/] > > We'd like to submit a formal moving patch applied to staging tree > for 5.4, before that we'd like to hear if there are some ACKs, > suggestions or NAKs, objections of EROFS. Therefore, we can improve > it in this round or rethink about the whole thing. ACK since it is stable enough and doesn't affect vfs or other filesystems. And then, it could attract more users, we can get more feedback, and they can help us to further improve it. Thanks Miao > > As related materials mentioned [1] [2], the goal of EROFS is to > save extra storage space with guaranteed end-to-end performance > for read-only files, which has better performance over exist Linux > compression filesystems based on fixed-sized output compression > and inplace decompression. It even has better performance in > a large compression ratio range compared with generic uncompressed > filesystems with proper CPU-storage combinations. And we think this > direction is correct and a dedicated kernel team is continuously / > actively working on improving it, enough testers and beta / end > users using it. > > EROFS has been applied to almost all in-service HUAWEI smartphones > (Yes, the number is still increasing by time) and it seems like > a success. It can be used in more wider scenarios. We think it's > useful for Linux / Android OS community and it's the time moving > out of staging. > > In order to get started, latest stable mkfs.erofs is available at > > git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git -b dev > > with README in the repository. > > We are still tuning sequential read performance for ultra-fast > speed NVME SSDs like Samsung 970PRO, but at least now you can > try on your PC with some data with proper compression ratio, > the latest Linux kernel, USB stick for convenience sake and > a not very old-fashioned CPU. There are also benchmarks available > in the above materials mentioned. > > EROFS is a self-contained filesystem driver. Although there are > still some TODOs to be more generic, we will actively keep on > developping / tuning EROFS with the evolution of Linux kernel > as the other in-kernel filesystems. > > As I mentioned before in LSF/MM 2019, in the future, we'd like > to generalize the decompression engine into a library for other > fses to use after the whole system is mature like fscrypt. > However, such metadata should be designed respectively for > each fs, and synchronous metadata read cost will be larger > than EROFS because of those ondisk limitation. Therefore EROFS > is still a better choice for read-only scenarios. > > EROFS is now ready for reviewing and moving, and the code is > already cleaned up as shiny floors... Please kindly take some > precious time, share your comments about EROFS and let us know > your opinion about this. It's really important for us since > generally speaking, we like to use Linux _in-tree_ stuffs rather > than lack of supported out-of-tree / orphan stuffs as well. > > Thank you in advance, > Gao Xiang > > [1] https://kccncosschn19eng.sched.com/event/Nru2/erofs-an-introduction-and-our-smartphone-practice-xiang-gao-huawei > [2] https://www.usenix.org/conference/atc19/presentation/gao > > Changelog from v7: > o keep up with the latest staging tree in addition to > the latest staging patch: > https://lore.kernel.org/r/20190814103705.60698-1-gaoxiang25@huawei.com/ > - use EUCLEAN for fs corruption cases suggested by Pavel; > - turn EIO into EOPNOTSUPP for unsupported on-disk format; > - fix all misused ENOTSUPP into EOPNOTSUPP pointed out by Chao; > o update cover letter > > It can also be found in git at tag "erofs_2019-08-15" (will be shown later) at: > https://git.kernel.org/pub/scm/linux/kernel/git/xiang/linux.git/ > > and the latest fs code is available at: > https://git.kernel.org/pub/scm/linux/kernel/git/xiang/linux.git/tree/fs/erofs?h=erofs-outofstaging > > Changelog from v6: > o keep up with the latest staging patchset > https://lore.kernel.org/linux-fsdevel/20190813023054.73126-1-gaoxiang25@huawei.com/ > in order to fix the following cases: > - inline erofs_inode_is_data_compressed() in erofs_fs.h; > - remove incomplete cleancache; > - remove all BUG_ON in EROFS. > o Removing the file names from the comments at the top of the files > suggested by Stephen will be applied to the real moving patch later. > > Changelog from v5: > o keep up with "[PATCH v2] staging: erofs: updates according to erofs-outofstaging v4" > https://lore.kernel.org/lkml/20190731155752.210602-1-gaoxiang25@huawei.com/ > which mainly addresses review comments from Chao: > - keep the marco EROFS_IO_MAX_RETRIES_NOFAIL in internal.h; > - kill a redundant NULL check in "__stagingpage_alloc"; > - add some descriptions in document about "use_vmap"; > - rearrange erofs_vmap of "staging: erofs: kill CONFIG_EROFS_FS_USE_VM_MAP_RAM"; > > o all changes have been merged into staging tree, which are under staging-testing: > https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/log/?h=staging-testing > > Changelog from v4: > o rebase on Linux 5.3-rc1; > > o keep up with "staging: erofs: updates according to erofs-outofstaging v4" > in order to get main code bit-for-bit identical with staging tree: > https://lore.kernel.org/lkml/20190729065159.62378-1-gaoxiang25@huawei.com/ > > Changelog from v3: > o use GPL-2.0-only for SPDX-License-Identifier suggested by Stephen; > > o kill all kconfig cache strategies and turn them into mount options > "cache_strategy={disable|readahead|readaround}" suggested by Ted. > As the first step, cached pages can still be usable after cache is > disabled by remounting, and these pages will be fallen out over > time, which can be refined in the later version if some requirement > is needed. Update related document as well; > > o turn on CONFIG_EROFS_FS_SECURITY by default suggested by David; > > o kill CONFIG_EROFS_FS_IO_MAX_RETRIES and fold it into code; turn > EROFS_FS_USE_VM_MAP_RAM into a module parameter ("use_vmap") > suggested by David. > > Changelog from v2: > o kill sbi->dev_name and clean up all failure handling in > fill_super() suggested by Al. > Note that the initialzation of managed_cache is now moved > after s_root is assigned since it's more preferred to iput() > in .put_super() and all inodes should be evicted before > the end of generic_shutdown_super(sb); > > o fold in the following staging patches (and thanks): > staging: erofs:converting all 'unsigned' to 'unsigned int' > staging: erofs: Remove function erofs_kill_sb() > - However it was revoked due to erofs_kill_sb reused... > staging: erofs: avoid opened loop codes > staging: erofs: support bmap > > o move EROFS_SUPER_MAGIC_V1 from linux/fs/erofs/erofs_fs.h to > include/uapi/linux/magic.h for userspace utilities. > > Changelog from v1: > o resend the whole filesystem into a patchset suggested by Greg; > o code is more cleaner, especially for decompression frontend. > > Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> > Cc: Alexander Viro <viro@zeniv.linux.org.uk> > Cc: Andrew Morton <akpm@linux-foundation.org> > Cc: Stephen Rothwell <sfr@canb.auug.org.au> > Cc: Theodore Ts'o <tytso@mit.edu> > Cc: Pavel Machek <pavel@denx.de> > Cc: David Sterba <dsterba@suse.cz> > Cc: Amir Goldstein <amir73il@gmail.com> > Cc: Christoph Hellwig <hch@infradead.org> > Cc: Darrick J . Wong <darrick.wong@oracle.com> > Cc: Dave Chinner <david@fromorbit.com> > Cc: Jaegeuk Kim <jaegeuk@kernel.org> > Cc: Jan Kara <jack@suse.cz> > Cc: Richard Weinberger <richard@nod.at> > Cc: Chao Yu <yuchao0@huawei.com> > Cc: Miao Xie <miaoxie@huawei.com> > Cc: Li Guifu <bluce.liguifu@huawei.com> > Cc: Fang Wei <fangwei1@huawei.com> > Signed-off-by: Gao Xiang <gaoxiang25@huawei.com> > > > Gao Xiang (24): > erofs: add on-disk layout > erofs: add erofs in-memory stuffs > erofs: add super block operations > erofs: add raw address_space operations > erofs: add inode operations > erofs: support special inode > erofs: add directory operations > erofs: add namei functions > erofs: support tracepoint > erofs: update Kconfig and Makefile > erofs: introduce xattr & posixacl support > erofs: introduce tagged pointer > erofs: add compression indexes support > erofs: introduce superblock registration > erofs: introduce erofs shrinker > erofs: introduce workstation for decompression > erofs: introduce per-CPU buffers implementation > erofs: introduce pagevec for decompression subsystem > erofs: add erofs_allocpage() > erofs: introduce generic decompression backend > erofs: introduce LZ4 decompression inplace > erofs: introduce the decompression frontend > erofs: introduce cached decompression > erofs: add document > > Documentation/filesystems/erofs.txt | 225 +++++ > fs/Kconfig | 1 + > fs/Makefile | 1 + > fs/erofs/Kconfig | 98 ++ > fs/erofs/Makefile | 11 + > fs/erofs/compress.h | 62 ++ > fs/erofs/data.c | 425 ++++++++ > fs/erofs/decompressor.c | 360 +++++++ > fs/erofs/dir.c | 148 +++ > fs/erofs/erofs_fs.h | 316 ++++++ > fs/erofs/inode.c | 333 +++++++ > fs/erofs/internal.h | 555 +++++++++++ > fs/erofs/namei.c | 253 +++++ > fs/erofs/super.c | 666 +++++++++++++ > fs/erofs/tagptr.h | 110 +++ > fs/erofs/utils.c | 335 +++++++ > fs/erofs/xattr.c | 705 ++++++++++++++ > fs/erofs/xattr.h | 94 ++ > fs/erofs/zdata.c | 1405 +++++++++++++++++++++++++++ > fs/erofs/zdata.h | 195 ++++ > fs/erofs/zmap.c | 463 +++++++++ > fs/erofs/zpvec.h | 159 +++ > include/trace/events/erofs.h | 256 +++++ > include/uapi/linux/magic.h | 1 + > 24 files changed, 7177 insertions(+) > create mode 100644 Documentation/filesystems/erofs.txt > create mode 100644 fs/erofs/Kconfig > create mode 100644 fs/erofs/Makefile > create mode 100644 fs/erofs/compress.h > create mode 100644 fs/erofs/data.c > create mode 100644 fs/erofs/decompressor.c > create mode 100644 fs/erofs/dir.c > create mode 100644 fs/erofs/erofs_fs.h > create mode 100644 fs/erofs/inode.c > create mode 100644 fs/erofs/internal.h > create mode 100644 fs/erofs/namei.c > create mode 100644 fs/erofs/super.c > create mode 100644 fs/erofs/tagptr.h > create mode 100644 fs/erofs/utils.c > create mode 100644 fs/erofs/xattr.c > create mode 100644 fs/erofs/xattr.h > create mode 100644 fs/erofs/zdata.c > create mode 100644 fs/erofs/zdata.h > create mode 100644 fs/erofs/zmap.c > create mode 100644 fs/erofs/zpvec.h > create mode 100644 include/trace/events/erofs.h >
On 2019/8/15 12:41, Gao Xiang wrote: > [I strip the previous cover letter, the old one can be found in v6: > https://lore.kernel.org/r/20190802125347.166018-1-gaoxiang25@huawei.com/] > > We'd like to submit a formal moving patch applied to staging tree > for 5.4, before that we'd like to hear if there are some ACKs, > suggestions or NAKs, objections of EROFS. Therefore, we can improve > it in this round or rethink about the whole thing. > > As related materials mentioned [1] [2], the goal of EROFS is to > save extra storage space with guaranteed end-to-end performance > for read-only files, which has better performance over exist Linux > compression filesystems based on fixed-sized output compression > and inplace decompression. It even has better performance in > a large compression ratio range compared with generic uncompressed > filesystems with proper CPU-storage combinations. And we think this > direction is correct and a dedicated kernel team is continuously / > actively working on improving it, enough testers and beta / end > users using it. > > EROFS has been applied to almost all in-service HUAWEI smartphones > (Yes, the number is still increasing by time) and it seems like > a success. It can be used in more wider scenarios. We think it's > useful for Linux / Android OS community and it's the time moving > out of staging. > > In order to get started, latest stable mkfs.erofs is available at > > git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git -b dev > > with README in the repository. > > We are still tuning sequential read performance for ultra-fast > speed NVME SSDs like Samsung 970PRO, but at least now you can > try on your PC with some data with proper compression ratio, > the latest Linux kernel, USB stick for convenience sake and > a not very old-fashioned CPU. There are also benchmarks available > in the above materials mentioned. > > EROFS is a self-contained filesystem driver. Although there are > still some TODOs to be more generic, we will actively keep on > developping / tuning EROFS with the evolution of Linux kernel > as the other in-kernel filesystems. > > As I mentioned before in LSF/MM 2019, in the future, we'd like > to generalize the decompression engine into a library for other > fses to use after the whole system is mature like fscrypt. > However, such metadata should be designed respectively for > each fs, and synchronous metadata read cost will be larger > than EROFS because of those ondisk limitation. Therefore EROFS > is still a better choice for read-only scenarios. > > EROFS is now ready for reviewing and moving, and the code is > already cleaned up as shiny floors... Please kindly take some > precious time, share your comments about EROFS and let us know > your opinion about this. It's really important for us since > generally speaking, we like to use Linux _in-tree_ stuffs rather > than lack of supported out-of-tree / orphan stuffs as well. EROFS proposes its very unique fixed-sized output compression and inplace decompression framework joining into the ecosystem of compression filesystem, I think it will enrich diversity of compression filesystem, and bring healthy competition there. I do believe this is the right time to promote erofs to fs/ directory, let it be the formal member of filesystem clubhouse. Acked-by: Chao Yu <yuchao0@huawei.com> Thanks > > Thank you in advance, > Gao Xiang > > [1] https://kccncosschn19eng.sched.com/event/Nru2/erofs-an-introduction-and-our-smartphone-practice-xiang-gao-huawei > [2] https://www.usenix.org/conference/atc19/presentation/gao > > Changelog from v7: > o keep up with the latest staging tree in addition to > the latest staging patch: > https://lore.kernel.org/r/20190814103705.60698-1-gaoxiang25@huawei.com/ > - use EUCLEAN for fs corruption cases suggested by Pavel; > - turn EIO into EOPNOTSUPP for unsupported on-disk format; > - fix all misused ENOTSUPP into EOPNOTSUPP pointed out by Chao; > o update cover letter > > It can also be found in git at tag "erofs_2019-08-15" (will be shown later) at: > https://git.kernel.org/pub/scm/linux/kernel/git/xiang/linux.git/ > > and the latest fs code is available at: > https://git.kernel.org/pub/scm/linux/kernel/git/xiang/linux.git/tree/fs/erofs?h=erofs-outofstaging > > Changelog from v6: > o keep up with the latest staging patchset > https://lore.kernel.org/linux-fsdevel/20190813023054.73126-1-gaoxiang25@huawei.com/ > in order to fix the following cases: > - inline erofs_inode_is_data_compressed() in erofs_fs.h; > - remove incomplete cleancache; > - remove all BUG_ON in EROFS. > o Removing the file names from the comments at the top of the files > suggested by Stephen will be applied to the real moving patch later. > > Changelog from v5: > o keep up with "[PATCH v2] staging: erofs: updates according to erofs-outofstaging v4" > https://lore.kernel.org/lkml/20190731155752.210602-1-gaoxiang25@huawei.com/ > which mainly addresses review comments from Chao: > - keep the marco EROFS_IO_MAX_RETRIES_NOFAIL in internal.h; > - kill a redundant NULL check in "__stagingpage_alloc"; > - add some descriptions in document about "use_vmap"; > - rearrange erofs_vmap of "staging: erofs: kill CONFIG_EROFS_FS_USE_VM_MAP_RAM"; > > o all changes have been merged into staging tree, which are under staging-testing: > https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/log/?h=staging-testing > > Changelog from v4: > o rebase on Linux 5.3-rc1; > > o keep up with "staging: erofs: updates according to erofs-outofstaging v4" > in order to get main code bit-for-bit identical with staging tree: > https://lore.kernel.org/lkml/20190729065159.62378-1-gaoxiang25@huawei.com/ > > Changelog from v3: > o use GPL-2.0-only for SPDX-License-Identifier suggested by Stephen; > > o kill all kconfig cache strategies and turn them into mount options > "cache_strategy={disable|readahead|readaround}" suggested by Ted. > As the first step, cached pages can still be usable after cache is > disabled by remounting, and these pages will be fallen out over > time, which can be refined in the later version if some requirement > is needed. Update related document as well; > > o turn on CONFIG_EROFS_FS_SECURITY by default suggested by David; > > o kill CONFIG_EROFS_FS_IO_MAX_RETRIES and fold it into code; turn > EROFS_FS_USE_VM_MAP_RAM into a module parameter ("use_vmap") > suggested by David. > > Changelog from v2: > o kill sbi->dev_name and clean up all failure handling in > fill_super() suggested by Al. > Note that the initialzation of managed_cache is now moved > after s_root is assigned since it's more preferred to iput() > in .put_super() and all inodes should be evicted before > the end of generic_shutdown_super(sb); > > o fold in the following staging patches (and thanks): > staging: erofs:converting all 'unsigned' to 'unsigned int' > staging: erofs: Remove function erofs_kill_sb() > - However it was revoked due to erofs_kill_sb reused... > staging: erofs: avoid opened loop codes > staging: erofs: support bmap > > o move EROFS_SUPER_MAGIC_V1 from linux/fs/erofs/erofs_fs.h to > include/uapi/linux/magic.h for userspace utilities. > > Changelog from v1: > o resend the whole filesystem into a patchset suggested by Greg; > o code is more cleaner, especially for decompression frontend. > > Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> > Cc: Alexander Viro <viro@zeniv.linux.org.uk> > Cc: Andrew Morton <akpm@linux-foundation.org> > Cc: Stephen Rothwell <sfr@canb.auug.org.au> > Cc: Theodore Ts'o <tytso@mit.edu> > Cc: Pavel Machek <pavel@denx.de> > Cc: David Sterba <dsterba@suse.cz> > Cc: Amir Goldstein <amir73il@gmail.com> > Cc: Christoph Hellwig <hch@infradead.org> > Cc: Darrick J . Wong <darrick.wong@oracle.com> > Cc: Dave Chinner <david@fromorbit.com> > Cc: Jaegeuk Kim <jaegeuk@kernel.org> > Cc: Jan Kara <jack@suse.cz> > Cc: Richard Weinberger <richard@nod.at> > Cc: Chao Yu <yuchao0@huawei.com> > Cc: Miao Xie <miaoxie@huawei.com> > Cc: Li Guifu <bluce.liguifu@huawei.com> > Cc: Fang Wei <fangwei1@huawei.com> > Signed-off-by: Gao Xiang <gaoxiang25@huawei.com> > > > Gao Xiang (24): > erofs: add on-disk layout > erofs: add erofs in-memory stuffs > erofs: add super block operations > erofs: add raw address_space operations > erofs: add inode operations > erofs: support special inode > erofs: add directory operations > erofs: add namei functions > erofs: support tracepoint > erofs: update Kconfig and Makefile > erofs: introduce xattr & posixacl support > erofs: introduce tagged pointer > erofs: add compression indexes support > erofs: introduce superblock registration > erofs: introduce erofs shrinker > erofs: introduce workstation for decompression > erofs: introduce per-CPU buffers implementation > erofs: introduce pagevec for decompression subsystem > erofs: add erofs_allocpage() > erofs: introduce generic decompression backend > erofs: introduce LZ4 decompression inplace > erofs: introduce the decompression frontend > erofs: introduce cached decompression > erofs: add document > > Documentation/filesystems/erofs.txt | 225 +++++ > fs/Kconfig | 1 + > fs/Makefile | 1 + > fs/erofs/Kconfig | 98 ++ > fs/erofs/Makefile | 11 + > fs/erofs/compress.h | 62 ++ > fs/erofs/data.c | 425 ++++++++ > fs/erofs/decompressor.c | 360 +++++++ > fs/erofs/dir.c | 148 +++ > fs/erofs/erofs_fs.h | 316 ++++++ > fs/erofs/inode.c | 333 +++++++ > fs/erofs/internal.h | 555 +++++++++++ > fs/erofs/namei.c | 253 +++++ > fs/erofs/super.c | 666 +++++++++++++ > fs/erofs/tagptr.h | 110 +++ > fs/erofs/utils.c | 335 +++++++ > fs/erofs/xattr.c | 705 ++++++++++++++ > fs/erofs/xattr.h | 94 ++ > fs/erofs/zdata.c | 1405 +++++++++++++++++++++++++++ > fs/erofs/zdata.h | 195 ++++ > fs/erofs/zmap.c | 463 +++++++++ > fs/erofs/zpvec.h | 159 +++ > include/trace/events/erofs.h | 256 +++++ > include/uapi/linux/magic.h | 1 + > 24 files changed, 7177 insertions(+) > create mode 100644 Documentation/filesystems/erofs.txt > create mode 100644 fs/erofs/Kconfig > create mode 100644 fs/erofs/Makefile > create mode 100644 fs/erofs/compress.h > create mode 100644 fs/erofs/data.c > create mode 100644 fs/erofs/decompressor.c > create mode 100644 fs/erofs/dir.c > create mode 100644 fs/erofs/erofs_fs.h > create mode 100644 fs/erofs/inode.c > create mode 100644 fs/erofs/internal.h > create mode 100644 fs/erofs/namei.c > create mode 100644 fs/erofs/super.c > create mode 100644 fs/erofs/tagptr.h > create mode 100644 fs/erofs/utils.c > create mode 100644 fs/erofs/xattr.c > create mode 100644 fs/erofs/xattr.h > create mode 100644 fs/erofs/zdata.c > create mode 100644 fs/erofs/zdata.h > create mode 100644 fs/erofs/zmap.c > create mode 100644 fs/erofs/zpvec.h > create mode 100644 include/trace/events/erofs.h >
[I strip the previous cover letter, the old one can be found in v6: https://lore.kernel.org/r/20190802125347.166018-1-gaoxiang25@huawei.com/] We'd like to submit a formal moving patch applied to staging tree for 5.4, before that we'd like to hear if there are some ACKs, suggestions or NAKs, objections of EROFS. Therefore, we can improve it in this round or rethink about the whole thing. As related materials mentioned [1] [2], the goal of EROFS is to save extra storage space with guaranteed end-to-end performance for read-only files, which has better performance over exist Linux compression filesystems based on fixed-sized output compression and inplace decompression. It even has better performance in a large compression ratio range compared with generic uncompressed filesystems with proper CPU-storage combinations. And we think this direction is correct and a dedicated kernel team is continuously / actively working on improving it, enough testers and beta / end users using it. EROFS has been applied to almost all in-service HUAWEI smartphones (Yes, the number is still increasing by time) and it seems like a success. It can be used in more wider scenarios. We think it's useful for Linux / Android OS community and it's the time moving out of staging. In order to get started, latest stable mkfs.erofs is available at git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git -b dev with README in the repository. We are still tuning sequential read performance for ultra-fast speed NVME SSDs like Samsung 970PRO, but at least now you can try on your PC with some data with proper compression ratio, the latest Linux kernel, USB stick for convenience sake and a not very old-fashioned CPU. There are also benchmarks available in the above materials mentioned. EROFS is a self-contained filesystem driver. Although there are still some TODOs to be more generic, we will actively keep on developping / tuning EROFS with the evolution of Linux kernel as the other in-kernel filesystems. As I mentioned before in LSF/MM 2019, in the future, we'd like to generalize the decompression engine into a library for other fses to use after the whole system is mature like fscrypt. However, such metadata should be designed respectively for each fs, and synchronous metadata read cost will be larger than EROFS because of those ondisk limitation. Therefore EROFS is still a better choice for read-only scenarios. EROFS is now ready for reviewing and moving, and the code is already cleaned up as shiny floors... Please kindly take some precious time, share your comments about EROFS and let us know your opinion about this. It's really important for us since generally speaking, we like to use Linux _in-tree_ stuffs rather than lack of supported out-of-tree / orphan stuffs as well. Thank you in advance, Gao Xiang [1] https://kccncosschn19eng.sched.com/event/Nru2/erofs-an-introduction-and-our-smartphone-practice-xiang-gao-huawei [2] https://www.usenix.org/conference/atc19/presentation/gao Changelog from v7: o keep up with the latest staging tree in addition to the latest staging patch: https://lore.kernel.org/r/20190814103705.60698-1-gaoxiang25@huawei.com/ - use EUCLEAN for fs corruption cases suggested by Pavel; - turn EIO into EOPNOTSUPP for unsupported on-disk format; - fix all misused ENOTSUPP into EOPNOTSUPP pointed out by Chao; o update cover letter It can also be found in git at tag "erofs_2019-08-15" (will be shown later) at: https://git.kernel.org/pub/scm/linux/kernel/git/xiang/linux.git/ and the latest fs code is available at: https://git.kernel.org/pub/scm/linux/kernel/git/xiang/linux.git/tree/fs/erofs?h=erofs-outofstaging Changelog from v6: o keep up with the latest staging patchset https://lore.kernel.org/linux-fsdevel/20190813023054.73126-1-gaoxiang25@huawei.com/ in order to fix the following cases: - inline erofs_inode_is_data_compressed() in erofs_fs.h; - remove incomplete cleancache; - remove all BUG_ON in EROFS. o Removing the file names from the comments at the top of the files suggested by Stephen will be applied to the real moving patch later. Changelog from v5: o keep up with "[PATCH v2] staging: erofs: updates according to erofs-outofstaging v4" https://lore.kernel.org/lkml/20190731155752.210602-1-gaoxiang25@huawei.com/ which mainly addresses review comments from Chao: - keep the marco EROFS_IO_MAX_RETRIES_NOFAIL in internal.h; - kill a redundant NULL check in "__stagingpage_alloc"; - add some descriptions in document about "use_vmap"; - rearrange erofs_vmap of "staging: erofs: kill CONFIG_EROFS_FS_USE_VM_MAP_RAM"; o all changes have been merged into staging tree, which are under staging-testing: https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/log/?h=staging-testing Changelog from v4: o rebase on Linux 5.3-rc1; o keep up with "staging: erofs: updates according to erofs-outofstaging v4" in order to get main code bit-for-bit identical with staging tree: https://lore.kernel.org/lkml/20190729065159.62378-1-gaoxiang25@huawei.com/ Changelog from v3: o use GPL-2.0-only for SPDX-License-Identifier suggested by Stephen; o kill all kconfig cache strategies and turn them into mount options "cache_strategy={disable|readahead|readaround}" suggested by Ted. As the first step, cached pages can still be usable after cache is disabled by remounting, and these pages will be fallen out over time, which can be refined in the later version if some requirement is needed. Update related document as well; o turn on CONFIG_EROFS_FS_SECURITY by default suggested by David; o kill CONFIG_EROFS_FS_IO_MAX_RETRIES and fold it into code; turn EROFS_FS_USE_VM_MAP_RAM into a module parameter ("use_vmap") suggested by David. Changelog from v2: o kill sbi->dev_name and clean up all failure handling in fill_super() suggested by Al. Note that the initialzation of managed_cache is now moved after s_root is assigned since it's more preferred to iput() in .put_super() and all inodes should be evicted before the end of generic_shutdown_super(sb); o fold in the following staging patches (and thanks): staging: erofs:converting all 'unsigned' to 'unsigned int' staging: erofs: Remove function erofs_kill_sb() - However it was revoked due to erofs_kill_sb reused... staging: erofs: avoid opened loop codes staging: erofs: support bmap o move EROFS_SUPER_MAGIC_V1 from linux/fs/erofs/erofs_fs.h to include/uapi/linux/magic.h for userspace utilities. Changelog from v1: o resend the whole filesystem into a patchset suggested by Greg; o code is more cleaner, especially for decompression frontend. Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Alexander Viro <viro@zeniv.linux.org.uk> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Stephen Rothwell <sfr@canb.auug.org.au> Cc: Theodore Ts'o <tytso@mit.edu> Cc: Pavel Machek <pavel@denx.de> Cc: David Sterba <dsterba@suse.cz> Cc: Amir Goldstein <amir73il@gmail.com> Cc: Christoph Hellwig <hch@infradead.org> Cc: Darrick J . Wong <darrick.wong@oracle.com> Cc: Dave Chinner <david@fromorbit.com> Cc: Jaegeuk Kim <jaegeuk@kernel.org> Cc: Jan Kara <jack@suse.cz> Cc: Richard Weinberger <richard@nod.at> Cc: Chao Yu <yuchao0@huawei.com> Cc: Miao Xie <miaoxie@huawei.com> Cc: Li Guifu <bluce.liguifu@huawei.com> Cc: Fang Wei <fangwei1@huawei.com> Signed-off-by: Gao Xiang <gaoxiang25@huawei.com> Gao Xiang (24): erofs: add on-disk layout erofs: add erofs in-memory stuffs erofs: add super block operations erofs: add raw address_space operations erofs: add inode operations erofs: support special inode erofs: add directory operations erofs: add namei functions erofs: support tracepoint erofs: update Kconfig and Makefile erofs: introduce xattr & posixacl support erofs: introduce tagged pointer erofs: add compression indexes support erofs: introduce superblock registration erofs: introduce erofs shrinker erofs: introduce workstation for decompression erofs: introduce per-CPU buffers implementation erofs: introduce pagevec for decompression subsystem erofs: add erofs_allocpage() erofs: introduce generic decompression backend erofs: introduce LZ4 decompression inplace erofs: introduce the decompression frontend erofs: introduce cached decompression erofs: add document Documentation/filesystems/erofs.txt | 225 +++++ fs/Kconfig | 1 + fs/Makefile | 1 + fs/erofs/Kconfig | 98 ++ fs/erofs/Makefile | 11 + fs/erofs/compress.h | 62 ++ fs/erofs/data.c | 425 ++++++++ fs/erofs/decompressor.c | 360 +++++++ fs/erofs/dir.c | 148 +++ fs/erofs/erofs_fs.h | 316 ++++++ fs/erofs/inode.c | 333 +++++++ fs/erofs/internal.h | 555 +++++++++++ fs/erofs/namei.c | 253 +++++ fs/erofs/super.c | 666 +++++++++++++ fs/erofs/tagptr.h | 110 +++ fs/erofs/utils.c | 335 +++++++ fs/erofs/xattr.c | 705 ++++++++++++++ fs/erofs/xattr.h | 94 ++ fs/erofs/zdata.c | 1405 +++++++++++++++++++++++++++ fs/erofs/zdata.h | 195 ++++ fs/erofs/zmap.c | 463 +++++++++ fs/erofs/zpvec.h | 159 +++ include/trace/events/erofs.h | 256 +++++ include/uapi/linux/magic.h | 1 + 24 files changed, 7177 insertions(+) create mode 100644 Documentation/filesystems/erofs.txt create mode 100644 fs/erofs/Kconfig create mode 100644 fs/erofs/Makefile create mode 100644 fs/erofs/compress.h create mode 100644 fs/erofs/data.c create mode 100644 fs/erofs/decompressor.c create mode 100644 fs/erofs/dir.c create mode 100644 fs/erofs/erofs_fs.h create mode 100644 fs/erofs/inode.c create mode 100644 fs/erofs/internal.h create mode 100644 fs/erofs/namei.c create mode 100644 fs/erofs/super.c create mode 100644 fs/erofs/tagptr.h create mode 100644 fs/erofs/utils.c create mode 100644 fs/erofs/xattr.c create mode 100644 fs/erofs/xattr.h create mode 100644 fs/erofs/zdata.c create mode 100644 fs/erofs/zdata.h create mode 100644 fs/erofs/zmap.c create mode 100644 fs/erofs/zpvec.h create mode 100644 include/trace/events/erofs.h