| Message ID | cover.1619463858.git.osandov@fb.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | fs: interface for directly reading/writing compressed data | expand |
On Mon, Apr 26, 2021 at 12:06:03PM -0700, Omar Sandoval wrote: > From: Omar Sandoval <osandov@fb.com> > > This series adds an API for reading compressed data on a filesystem > without decompressing it as well as support for writing compressed data > directly to the filesystem. I have test cases (including fsstress > support) and example programs which I'll send up once the dust settles > [1]. > > The main use-case is Btrfs send/receive: currently, when sending data > from one compressed filesystem to another, the sending side decompresses > the data and the receiving side recompresses it before writing it out. > This is wasteful and can be avoided if we can just send and write > compressed extents. The patches implementing the send/receive support > were sent with the last submission of this series [2]. > > Patches 1-3 add the VFS support, UAPI, and documentation. Patches 4-7 > are Btrfs prep patches. Patch 8 adds Btrfs encoded read support and > patch 9 adds Btrfs encoded write support. > > These patches are based on Dave Sterba's Btrfs misc-next branch [3], > which is in turn currently based on v5.12-rc8. > > This is a resend of v9 [4], rebased on the latest kdave/misc-next > branch. > > Omar Sandoval (9): > iov_iter: add copy_struct_from_iter() > fs: add O_ALLOW_ENCODED open flag > fs: add RWF_ENCODED for reading/writing compressed data > btrfs: don't advance offset for compressed bios in > btrfs_csum_one_bio() > btrfs: add ram_bytes and offset to btrfs_ordered_extent > btrfs: support different disk extent size for delalloc > btrfs: optionally extend i_size in cow_file_range_inline() > btrfs: implement RWF_ENCODED reads > btrfs: implement RWF_ENCODED writes > > 1: https://github.com/osandov/xfstests/tree/rwf-encoded > 2: https://lore.kernel.org/linux-btrfs/cover.1615922753.git.osandov@fb.com/ > 3: https://github.com/kdave/btrfs-devel/tree/misc-next > 4: https://lore.kernel.org/linux-btrfs/cover.1617258892.git.osandov@fb.com/ > > Omar Sandoval (9): > iov_iter: add copy_struct_from_iter() > fs: add O_ALLOW_ENCODED open flag > fs: add RWF_ENCODED for reading/writing compressed data > btrfs: don't advance offset for compressed bios in > btrfs_csum_one_bio() > btrfs: add ram_bytes and offset to btrfs_ordered_extent > btrfs: support different disk extent size for delalloc > btrfs: optionally extend i_size in cow_file_range_inline() > btrfs: implement RWF_ENCODED reads > btrfs: implement RWF_ENCODED writes > > Documentation/filesystems/encoded_io.rst | 240 ++++++ > Documentation/filesystems/index.rst | 1 + > arch/alpha/include/uapi/asm/fcntl.h | 1 + > arch/parisc/include/uapi/asm/fcntl.h | 1 + > arch/sparc/include/uapi/asm/fcntl.h | 1 + > fs/btrfs/compression.c | 12 +- > fs/btrfs/compression.h | 6 +- > fs/btrfs/ctree.h | 9 +- > fs/btrfs/delalloc-space.c | 18 +- > fs/btrfs/file-item.c | 35 +- > fs/btrfs/file.c | 46 +- > fs/btrfs/inode.c | 929 +++++++++++++++++++++-- > fs/btrfs/ordered-data.c | 124 +-- > fs/btrfs/ordered-data.h | 25 +- > fs/btrfs/relocation.c | 4 +- > fs/fcntl.c | 10 +- > fs/namei.c | 4 + > fs/read_write.c | 168 +++- > include/linux/encoded_io.h | 17 + > include/linux/fcntl.h | 2 +- > include/linux/fs.h | 13 + > include/linux/uio.h | 1 + > include/uapi/asm-generic/fcntl.h | 4 + > include/uapi/linux/encoded_io.h | 30 + > include/uapi/linux/fs.h | 5 +- > lib/iov_iter.c | 91 +++ > 26 files changed, 1563 insertions(+), 234 deletions(-) > create mode 100644 Documentation/filesystems/encoded_io.rst > create mode 100644 include/linux/encoded_io.h > create mode 100644 include/uapi/linux/encoded_io.h Ping.
On Tue, May 04, 2021 at 10:15:51AM -0700, Omar Sandoval wrote: > On Mon, Apr 26, 2021 at 12:06:03PM -0700, Omar Sandoval wrote: > > From: Omar Sandoval <osandov@fb.com> > > > > This series adds an API for reading compressed data on a filesystem > > without decompressing it as well as support for writing compressed data > > directly to the filesystem. I have test cases (including fsstress > > support) and example programs which I'll send up once the dust settles > > [1]. > > > > The main use-case is Btrfs send/receive: currently, when sending data > > from one compressed filesystem to another, the sending side decompresses > > the data and the receiving side recompresses it before writing it out. > > This is wasteful and can be avoided if we can just send and write > > compressed extents. The patches implementing the send/receive support > > were sent with the last submission of this series [2]. > > > > Patches 1-3 add the VFS support, UAPI, and documentation. Patches 4-7 > > are Btrfs prep patches. Patch 8 adds Btrfs encoded read support and > > patch 9 adds Btrfs encoded write support. > > > > These patches are based on Dave Sterba's Btrfs misc-next branch [3], > > which is in turn currently based on v5.12-rc8. > > > > This is a resend of v9 [4], rebased on the latest kdave/misc-next > > branch. > > > > Omar Sandoval (9): > > iov_iter: add copy_struct_from_iter() > > fs: add O_ALLOW_ENCODED open flag > > fs: add RWF_ENCODED for reading/writing compressed data > > btrfs: don't advance offset for compressed bios in > > btrfs_csum_one_bio() > > btrfs: add ram_bytes and offset to btrfs_ordered_extent > > btrfs: support different disk extent size for delalloc > > btrfs: optionally extend i_size in cow_file_range_inline() > > btrfs: implement RWF_ENCODED reads > > btrfs: implement RWF_ENCODED writes > > > > 1: https://github.com/osandov/xfstests/tree/rwf-encoded > > 2: https://lore.kernel.org/linux-btrfs/cover.1615922753.git.osandov@fb.com/ > > 3: https://github.com/kdave/btrfs-devel/tree/misc-next > > 4: https://lore.kernel.org/linux-btrfs/cover.1617258892.git.osandov@fb.com/ > > > > Omar Sandoval (9): > > iov_iter: add copy_struct_from_iter() > > fs: add O_ALLOW_ENCODED open flag > > fs: add RWF_ENCODED for reading/writing compressed data > > btrfs: don't advance offset for compressed bios in > > btrfs_csum_one_bio() > > btrfs: add ram_bytes and offset to btrfs_ordered_extent > > btrfs: support different disk extent size for delalloc > > btrfs: optionally extend i_size in cow_file_range_inline() > > btrfs: implement RWF_ENCODED reads > > btrfs: implement RWF_ENCODED writes > > > > Documentation/filesystems/encoded_io.rst | 240 ++++++ > > Documentation/filesystems/index.rst | 1 + > > arch/alpha/include/uapi/asm/fcntl.h | 1 + > > arch/parisc/include/uapi/asm/fcntl.h | 1 + > > arch/sparc/include/uapi/asm/fcntl.h | 1 + > > fs/btrfs/compression.c | 12 +- > > fs/btrfs/compression.h | 6 +- > > fs/btrfs/ctree.h | 9 +- > > fs/btrfs/delalloc-space.c | 18 +- > > fs/btrfs/file-item.c | 35 +- > > fs/btrfs/file.c | 46 +- > > fs/btrfs/inode.c | 929 +++++++++++++++++++++-- > > fs/btrfs/ordered-data.c | 124 +-- > > fs/btrfs/ordered-data.h | 25 +- > > fs/btrfs/relocation.c | 4 +- > > fs/fcntl.c | 10 +- > > fs/namei.c | 4 + > > fs/read_write.c | 168 +++- > > include/linux/encoded_io.h | 17 + > > include/linux/fcntl.h | 2 +- > > include/linux/fs.h | 13 + > > include/linux/uio.h | 1 + > > include/uapi/asm-generic/fcntl.h | 4 + > > include/uapi/linux/encoded_io.h | 30 + > > include/uapi/linux/fs.h | 5 +- > > lib/iov_iter.c | 91 +++ > > 26 files changed, 1563 insertions(+), 234 deletions(-) > > create mode 100644 Documentation/filesystems/encoded_io.rst > > create mode 100644 include/linux/encoded_io.h > > create mode 100644 include/uapi/linux/encoded_io.h > > Ping. Ping. Al, I originally sent v9 over a month ago, and I've been sending out this series for over a year, so I'd appreciate it if you could take a look. It's fine if you hate it, but I'd like to know where to go from here. Thanks, Omar
From: Omar Sandoval <osandov@fb.com> This series adds an API for reading compressed data on a filesystem without decompressing it as well as support for writing compressed data directly to the filesystem. I have test cases (including fsstress support) and example programs which I'll send up once the dust settles [1]. The main use-case is Btrfs send/receive: currently, when sending data from one compressed filesystem to another, the sending side decompresses the data and the receiving side recompresses it before writing it out. This is wasteful and can be avoided if we can just send and write compressed extents. The patches implementing the send/receive support were sent with the last submission of this series [2]. Patches 1-3 add the VFS support, UAPI, and documentation. Patches 4-7 are Btrfs prep patches. Patch 8 adds Btrfs encoded read support and patch 9 adds Btrfs encoded write support. These patches are based on Dave Sterba's Btrfs misc-next branch [3], which is in turn currently based on v5.12-rc8. This is a resend of v9 [4], rebased on the latest kdave/misc-next branch. Omar Sandoval (9): iov_iter: add copy_struct_from_iter() fs: add O_ALLOW_ENCODED open flag fs: add RWF_ENCODED for reading/writing compressed data btrfs: don't advance offset for compressed bios in btrfs_csum_one_bio() btrfs: add ram_bytes and offset to btrfs_ordered_extent btrfs: support different disk extent size for delalloc btrfs: optionally extend i_size in cow_file_range_inline() btrfs: implement RWF_ENCODED reads btrfs: implement RWF_ENCODED writes 1: https://github.com/osandov/xfstests/tree/rwf-encoded 2: https://lore.kernel.org/linux-btrfs/cover.1615922753.git.osandov@fb.com/ 3: https://github.com/kdave/btrfs-devel/tree/misc-next 4: https://lore.kernel.org/linux-btrfs/cover.1617258892.git.osandov@fb.com/ Omar Sandoval (9): iov_iter: add copy_struct_from_iter() fs: add O_ALLOW_ENCODED open flag fs: add RWF_ENCODED for reading/writing compressed data btrfs: don't advance offset for compressed bios in btrfs_csum_one_bio() btrfs: add ram_bytes and offset to btrfs_ordered_extent btrfs: support different disk extent size for delalloc btrfs: optionally extend i_size in cow_file_range_inline() btrfs: implement RWF_ENCODED reads btrfs: implement RWF_ENCODED writes Documentation/filesystems/encoded_io.rst | 240 ++++++ Documentation/filesystems/index.rst | 1 + arch/alpha/include/uapi/asm/fcntl.h | 1 + arch/parisc/include/uapi/asm/fcntl.h | 1 + arch/sparc/include/uapi/asm/fcntl.h | 1 + fs/btrfs/compression.c | 12 +- fs/btrfs/compression.h | 6 +- fs/btrfs/ctree.h | 9 +- fs/btrfs/delalloc-space.c | 18 +- fs/btrfs/file-item.c | 35 +- fs/btrfs/file.c | 46 +- fs/btrfs/inode.c | 929 +++++++++++++++++++++-- fs/btrfs/ordered-data.c | 124 +-- fs/btrfs/ordered-data.h | 25 +- fs/btrfs/relocation.c | 4 +- fs/fcntl.c | 10 +- fs/namei.c | 4 + fs/read_write.c | 168 +++- include/linux/encoded_io.h | 17 + include/linux/fcntl.h | 2 +- include/linux/fs.h | 13 + include/linux/uio.h | 1 + include/uapi/asm-generic/fcntl.h | 4 + include/uapi/linux/encoded_io.h | 30 + include/uapi/linux/fs.h | 5 +- lib/iov_iter.c | 91 +++ 26 files changed, 1563 insertions(+), 234 deletions(-) create mode 100644 Documentation/filesystems/encoded_io.rst create mode 100644 include/linux/encoded_io.h create mode 100644 include/uapi/linux/encoded_io.h