| Message ID | 20210531085106.259490-1-wqu@suse.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | btrfs: add data write support for subpage | expand |
On Mon, May 31, 2021 at 4:52 AM Qu Wenruo <wqu@suse.com> wrote: > > This huge patchset can be fetched from github: > https://github.com/adam900710/linux/tree/subpage > > === Current stage === > The tests on x86 pass without new failure, and generic test group on > arm64 with 64K page size passes except known failure and defrag group. > > For btrfs test group, all pass except compression/raid56/defrag. > > For anyone who is interested in testing, please apply this patch for > btrfs-progs before testing. > https://patchwork.kernel.org/project/linux-btrfs/patch/20210420073036.243715-1-wqu@suse.com/ > Or there will be too many false alerts. > > === Limitation === > There are several limitations introduced just for subpage: > - No compressed write support > Read is no problem, but compression write path has more things left to > be modified. > Thus for current patchset, no matter what inode attribute or mount > option is, no new compressed extent can be created for subpage case. > > - No inline extent will be created > This is mostly due to the fact that filemap_fdatawrite_range() will > trigger more write than the range specified. > In fallocate calls, this behavior can make us to writeback which can > be inlined, before we enlarge the isize, causing inline extent being > created along with regular extents. > > - No support for RAID56 > There are still too many hardcoded PAGE_SIZE in raid56 code. > Considering it's already considered unsafe due to its write-hole > problem, disabling RAID56 for subpage looks sane to me. > > - No defrag support for subpage > The support for subpage defrag has already an initial version > submitted to the mail list. > Thus the correct support won't be included in this patchset. > > === Patchset structure === > > Patch 01~19: Make data write path to be subpage compatible > Patch 20~21: Make data relocation path to be subpage compatible > Patch 22~29: Various fixes for subpage corner cases > Patch 30: Enable subpage data write > > === Changelog === > v2: > - Rebased to latest misc-next > Now metadata write patches are removed from the series, as they are > already merged into misc-next. > > - Added new Reviewed-by/Tested-by/Reported-by tags > > - Use separate endio functions to subpage metadata write path > > - Re-order the patches, to make refactors at the top of the series > One refactor, the submit_extent_page() one, should benefit 4K page > size more than 64K page size, thus it's worthy to be merged early > > - New bug fixes exposed by Ritesh Harjani on Power > > - Reject RAID56 completely > Exposed by btrfs test group, which caused BUG_ON() for various sites. > Considering RAID56 is already not considered safe, it's better to > reject them completely for now. > > - Fix subpage scrub repair failure > Caused by hardcoded PAGE_SIZE > > - Fix free space cache inode size > Same cause as scrub repair failure > > v3: > - Rebased to remove write path prepration patches > > - Properly enable btrfs defrag > Previsouly, btrfs defrag is in fact just disabled. > This makes tons of tests in btrfs/defrag to fail. > > - More bug fixes for rare race/crashes > * Fix relocation false alert on csum mismatch > * Fix relocation data corruption > * Fix a rare case of false ASSERT() > The fix already get merged into the prepration patches, thus no > longer in this patchset though. > > Mostly reported by Ritesh from IBM. > > v4: > - Disable subpage defrag completely > As full page defrag can race with fsstress in btrfs/062, causing > strange ordered extent bugs. > The full subpage defrag will be submitted as an indepdent patchset. > > Qu Wenruo (30): > btrfs: pass bytenr directly to __process_pages_contig() > btrfs: refactor the page status update into process_one_page() > btrfs: provide btrfs_page_clamp_*() helpers > btrfs: only require sector size alignment for > end_bio_extent_writepage() > btrfs: make btrfs_dirty_pages() to be subpage compatible > btrfs: make __process_pages_contig() to handle subpage > dirty/error/writeback status > btrfs: make end_bio_extent_writepage() to be subpage compatible > btrfs: make process_one_page() to handle subpage locking > btrfs: introduce helpers for subpage ordered status > btrfs: make page Ordered bit to be subpage compatible > btrfs: update locked page dirty/writeback/error bits in > __process_pages_contig > btrfs: prevent extent_clear_unlock_delalloc() to unlock page not > locked by __process_pages_contig() > btrfs: make btrfs_set_range_writeback() subpage compatible > btrfs: make __extent_writepage_io() only submit dirty range for > subpage > btrfs: make btrfs_truncate_block() to be subpage compatible > btrfs: make btrfs_page_mkwrite() to be subpage compatible > btrfs: reflink: make copy_inline_to_page() to be subpage compatible > btrfs: fix the filemap_range_has_page() call in > btrfs_punch_hole_lock_range() > btrfs: don't clear page extent mapped if we're not invalidating the > full page > btrfs: extract relocation page read and dirty part into its own > function > btrfs: make relocate_one_page() to handle subpage case > btrfs: fix wild subpage writeback which does not have ordered extent. > btrfs: disable inline extent creation for subpage > btrfs: allow submit_extent_page() to do bio split for subpage > btrfs: reject raid5/6 fs for subpage > btrfs: fix a crash caused by race between prepare_pages() and > btrfs_releasepage() > btrfs: fix a use-after-free bug in writeback subpage helper > btrfs: fix a subpage false alert for relocating partial preallocated > data extents > btrfs: fix a subpage relocation data corruption > btrfs: allow read-write for 4K sectorsize on 64K page size systems > > fs/btrfs/ctree.h | 2 +- > fs/btrfs/disk-io.c | 13 +- > fs/btrfs/extent_io.c | 563 ++++++++++++++++++++++++++++------------ > fs/btrfs/file.c | 32 ++- > fs/btrfs/inode.c | 147 +++++++++-- > fs/btrfs/ioctl.c | 6 + > fs/btrfs/ordered-data.c | 5 +- > fs/btrfs/reflink.c | 14 +- > fs/btrfs/relocation.c | 287 ++++++++++++-------- > fs/btrfs/subpage.c | 156 ++++++++++- > fs/btrfs/subpage.h | 31 +++ > fs/btrfs/super.c | 7 - > fs/btrfs/sysfs.c | 5 + > fs/btrfs/volumes.c | 8 + > 14 files changed, 949 insertions(+), 327 deletions(-) > Could you please rebase your branch on 5.13-rc4? I'd rather test it on top of that release...
On 2021/5/31 下午5:47, Neal Gompa wrote: > On Mon, May 31, 2021 at 4:52 AM Qu Wenruo <wqu@suse.com> wrote: >> >> This huge patchset can be fetched from github: >> https://github.com/adam900710/linux/tree/subpage >> >> === Current stage === >> The tests on x86 pass without new failure, and generic test group on >> arm64 with 64K page size passes except known failure and defrag group. >> >> For btrfs test group, all pass except compression/raid56/defrag. >> >> For anyone who is interested in testing, please apply this patch for >> btrfs-progs before testing. >> https://patchwork.kernel.org/project/linux-btrfs/patch/20210420073036.243715-1-wqu@suse.com/ >> Or there will be too many false alerts. >> >> === Limitation === >> There are several limitations introduced just for subpage: >> - No compressed write support >> Read is no problem, but compression write path has more things left to >> be modified. >> Thus for current patchset, no matter what inode attribute or mount >> option is, no new compressed extent can be created for subpage case. >> >> - No inline extent will be created >> This is mostly due to the fact that filemap_fdatawrite_range() will >> trigger more write than the range specified. >> In fallocate calls, this behavior can make us to writeback which can >> be inlined, before we enlarge the isize, causing inline extent being >> created along with regular extents. >> >> - No support for RAID56 >> There are still too many hardcoded PAGE_SIZE in raid56 code. >> Considering it's already considered unsafe due to its write-hole >> problem, disabling RAID56 for subpage looks sane to me. >> >> - No defrag support for subpage >> The support for subpage defrag has already an initial version >> submitted to the mail list. >> Thus the correct support won't be included in this patchset. >> >> === Patchset structure === >> >> Patch 01~19: Make data write path to be subpage compatible >> Patch 20~21: Make data relocation path to be subpage compatible >> Patch 22~29: Various fixes for subpage corner cases >> Patch 30: Enable subpage data write >> >> === Changelog === >> v2: >> - Rebased to latest misc-next >> Now metadata write patches are removed from the series, as they are >> already merged into misc-next. >> >> - Added new Reviewed-by/Tested-by/Reported-by tags >> >> - Use separate endio functions to subpage metadata write path >> >> - Re-order the patches, to make refactors at the top of the series >> One refactor, the submit_extent_page() one, should benefit 4K page >> size more than 64K page size, thus it's worthy to be merged early >> >> - New bug fixes exposed by Ritesh Harjani on Power >> >> - Reject RAID56 completely >> Exposed by btrfs test group, which caused BUG_ON() for various sites. >> Considering RAID56 is already not considered safe, it's better to >> reject them completely for now. >> >> - Fix subpage scrub repair failure >> Caused by hardcoded PAGE_SIZE >> >> - Fix free space cache inode size >> Same cause as scrub repair failure >> >> v3: >> - Rebased to remove write path prepration patches >> >> - Properly enable btrfs defrag >> Previsouly, btrfs defrag is in fact just disabled. >> This makes tons of tests in btrfs/defrag to fail. >> >> - More bug fixes for rare race/crashes >> * Fix relocation false alert on csum mismatch >> * Fix relocation data corruption >> * Fix a rare case of false ASSERT() >> The fix already get merged into the prepration patches, thus no >> longer in this patchset though. >> >> Mostly reported by Ritesh from IBM. >> >> v4: >> - Disable subpage defrag completely >> As full page defrag can race with fsstress in btrfs/062, causing >> strange ordered extent bugs. >> The full subpage defrag will be submitted as an indepdent patchset. >> >> Qu Wenruo (30): >> btrfs: pass bytenr directly to __process_pages_contig() >> btrfs: refactor the page status update into process_one_page() >> btrfs: provide btrfs_page_clamp_*() helpers >> btrfs: only require sector size alignment for >> end_bio_extent_writepage() >> btrfs: make btrfs_dirty_pages() to be subpage compatible >> btrfs: make __process_pages_contig() to handle subpage >> dirty/error/writeback status >> btrfs: make end_bio_extent_writepage() to be subpage compatible >> btrfs: make process_one_page() to handle subpage locking >> btrfs: introduce helpers for subpage ordered status >> btrfs: make page Ordered bit to be subpage compatible >> btrfs: update locked page dirty/writeback/error bits in >> __process_pages_contig >> btrfs: prevent extent_clear_unlock_delalloc() to unlock page not >> locked by __process_pages_contig() >> btrfs: make btrfs_set_range_writeback() subpage compatible >> btrfs: make __extent_writepage_io() only submit dirty range for >> subpage >> btrfs: make btrfs_truncate_block() to be subpage compatible >> btrfs: make btrfs_page_mkwrite() to be subpage compatible >> btrfs: reflink: make copy_inline_to_page() to be subpage compatible >> btrfs: fix the filemap_range_has_page() call in >> btrfs_punch_hole_lock_range() >> btrfs: don't clear page extent mapped if we're not invalidating the >> full page >> btrfs: extract relocation page read and dirty part into its own >> function >> btrfs: make relocate_one_page() to handle subpage case >> btrfs: fix wild subpage writeback which does not have ordered extent. >> btrfs: disable inline extent creation for subpage >> btrfs: allow submit_extent_page() to do bio split for subpage >> btrfs: reject raid5/6 fs for subpage >> btrfs: fix a crash caused by race between prepare_pages() and >> btrfs_releasepage() >> btrfs: fix a use-after-free bug in writeback subpage helper >> btrfs: fix a subpage false alert for relocating partial preallocated >> data extents >> btrfs: fix a subpage relocation data corruption >> btrfs: allow read-write for 4K sectorsize on 64K page size systems >> >> fs/btrfs/ctree.h | 2 +- >> fs/btrfs/disk-io.c | 13 +- >> fs/btrfs/extent_io.c | 563 ++++++++++++++++++++++++++++------------ >> fs/btrfs/file.c | 32 ++- >> fs/btrfs/inode.c | 147 +++++++++-- >> fs/btrfs/ioctl.c | 6 + >> fs/btrfs/ordered-data.c | 5 +- >> fs/btrfs/reflink.c | 14 +- >> fs/btrfs/relocation.c | 287 ++++++++++++-------- >> fs/btrfs/subpage.c | 156 ++++++++++- >> fs/btrfs/subpage.h | 31 +++ >> fs/btrfs/super.c | 7 - >> fs/btrfs/sysfs.c | 5 + >> fs/btrfs/volumes.c | 8 + >> 14 files changed, 949 insertions(+), 327 deletions(-) >> > > Could you please rebase your branch on 5.13-rc4? I'd rather test it on > top of that release... > > It can be rebased on david's misc-next branch without any conflicts. Although misc-next is only on v5.13-rc3, I don't believe there will be anything btrfs related out of misc-next. Is there anything special only show up in -rc4? Thanks, Qu
On Mon, May 31, 2021 at 5:50 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote: > > > > On 2021/5/31 下午5:47, Neal Gompa wrote: > > On Mon, May 31, 2021 at 4:52 AM Qu Wenruo <wqu@suse.com> wrote: > >> > >> This huge patchset can be fetched from github: > >> https://github.com/adam900710/linux/tree/subpage > >> > >> === Current stage === > >> The tests on x86 pass without new failure, and generic test group on > >> arm64 with 64K page size passes except known failure and defrag group. > >> > >> For btrfs test group, all pass except compression/raid56/defrag. > >> > >> For anyone who is interested in testing, please apply this patch for > >> btrfs-progs before testing. > >> https://patchwork.kernel.org/project/linux-btrfs/patch/20210420073036.243715-1-wqu@suse.com/ > >> Or there will be too many false alerts. > >> > >> === Limitation === > >> There are several limitations introduced just for subpage: > >> - No compressed write support > >> Read is no problem, but compression write path has more things left to > >> be modified. > >> Thus for current patchset, no matter what inode attribute or mount > >> option is, no new compressed extent can be created for subpage case. > >> > >> - No inline extent will be created > >> This is mostly due to the fact that filemap_fdatawrite_range() will > >> trigger more write than the range specified. > >> In fallocate calls, this behavior can make us to writeback which can > >> be inlined, before we enlarge the isize, causing inline extent being > >> created along with regular extents. > >> > >> - No support for RAID56 > >> There are still too many hardcoded PAGE_SIZE in raid56 code. > >> Considering it's already considered unsafe due to its write-hole > >> problem, disabling RAID56 for subpage looks sane to me. > >> > >> - No defrag support for subpage > >> The support for subpage defrag has already an initial version > >> submitted to the mail list. > >> Thus the correct support won't be included in this patchset. > >> > >> === Patchset structure === > >> > >> Patch 01~19: Make data write path to be subpage compatible > >> Patch 20~21: Make data relocation path to be subpage compatible > >> Patch 22~29: Various fixes for subpage corner cases > >> Patch 30: Enable subpage data write > >> > >> === Changelog === > >> v2: > >> - Rebased to latest misc-next > >> Now metadata write patches are removed from the series, as they are > >> already merged into misc-next. > >> > >> - Added new Reviewed-by/Tested-by/Reported-by tags > >> > >> - Use separate endio functions to subpage metadata write path > >> > >> - Re-order the patches, to make refactors at the top of the series > >> One refactor, the submit_extent_page() one, should benefit 4K page > >> size more than 64K page size, thus it's worthy to be merged early > >> > >> - New bug fixes exposed by Ritesh Harjani on Power > >> > >> - Reject RAID56 completely > >> Exposed by btrfs test group, which caused BUG_ON() for various sites. > >> Considering RAID56 is already not considered safe, it's better to > >> reject them completely for now. > >> > >> - Fix subpage scrub repair failure > >> Caused by hardcoded PAGE_SIZE > >> > >> - Fix free space cache inode size > >> Same cause as scrub repair failure > >> > >> v3: > >> - Rebased to remove write path prepration patches > >> > >> - Properly enable btrfs defrag > >> Previsouly, btrfs defrag is in fact just disabled. > >> This makes tons of tests in btrfs/defrag to fail. > >> > >> - More bug fixes for rare race/crashes > >> * Fix relocation false alert on csum mismatch > >> * Fix relocation data corruption > >> * Fix a rare case of false ASSERT() > >> The fix already get merged into the prepration patches, thus no > >> longer in this patchset though. > >> > >> Mostly reported by Ritesh from IBM. > >> > >> v4: > >> - Disable subpage defrag completely > >> As full page defrag can race with fsstress in btrfs/062, causing > >> strange ordered extent bugs. > >> The full subpage defrag will be submitted as an indepdent patchset. > >> > >> Qu Wenruo (30): > >> btrfs: pass bytenr directly to __process_pages_contig() > >> btrfs: refactor the page status update into process_one_page() > >> btrfs: provide btrfs_page_clamp_*() helpers > >> btrfs: only require sector size alignment for > >> end_bio_extent_writepage() > >> btrfs: make btrfs_dirty_pages() to be subpage compatible > >> btrfs: make __process_pages_contig() to handle subpage > >> dirty/error/writeback status > >> btrfs: make end_bio_extent_writepage() to be subpage compatible > >> btrfs: make process_one_page() to handle subpage locking > >> btrfs: introduce helpers for subpage ordered status > >> btrfs: make page Ordered bit to be subpage compatible > >> btrfs: update locked page dirty/writeback/error bits in > >> __process_pages_contig > >> btrfs: prevent extent_clear_unlock_delalloc() to unlock page not > >> locked by __process_pages_contig() > >> btrfs: make btrfs_set_range_writeback() subpage compatible > >> btrfs: make __extent_writepage_io() only submit dirty range for > >> subpage > >> btrfs: make btrfs_truncate_block() to be subpage compatible > >> btrfs: make btrfs_page_mkwrite() to be subpage compatible > >> btrfs: reflink: make copy_inline_to_page() to be subpage compatible > >> btrfs: fix the filemap_range_has_page() call in > >> btrfs_punch_hole_lock_range() > >> btrfs: don't clear page extent mapped if we're not invalidating the > >> full page > >> btrfs: extract relocation page read and dirty part into its own > >> function > >> btrfs: make relocate_one_page() to handle subpage case > >> btrfs: fix wild subpage writeback which does not have ordered extent. > >> btrfs: disable inline extent creation for subpage > >> btrfs: allow submit_extent_page() to do bio split for subpage > >> btrfs: reject raid5/6 fs for subpage > >> btrfs: fix a crash caused by race between prepare_pages() and > >> btrfs_releasepage() > >> btrfs: fix a use-after-free bug in writeback subpage helper > >> btrfs: fix a subpage false alert for relocating partial preallocated > >> data extents > >> btrfs: fix a subpage relocation data corruption > >> btrfs: allow read-write for 4K sectorsize on 64K page size systems > >> > >> fs/btrfs/ctree.h | 2 +- > >> fs/btrfs/disk-io.c | 13 +- > >> fs/btrfs/extent_io.c | 563 ++++++++++++++++++++++++++++------------ > >> fs/btrfs/file.c | 32 ++- > >> fs/btrfs/inode.c | 147 +++++++++-- > >> fs/btrfs/ioctl.c | 6 + > >> fs/btrfs/ordered-data.c | 5 +- > >> fs/btrfs/reflink.c | 14 +- > >> fs/btrfs/relocation.c | 287 ++++++++++++-------- > >> fs/btrfs/subpage.c | 156 ++++++++++- > >> fs/btrfs/subpage.h | 31 +++ > >> fs/btrfs/super.c | 7 - > >> fs/btrfs/sysfs.c | 5 + > >> fs/btrfs/volumes.c | 8 + > >> 14 files changed, 949 insertions(+), 327 deletions(-) > >> > > > > Could you please rebase your branch on 5.13-rc4? I'd rather test it on > > top of that release... > > > > > It can be rebased on david's misc-next branch without any conflicts. > Although misc-next is only on v5.13-rc3, I don't believe there will be > anything btrfs related out of misc-next. > > Is there anything special only show up in -rc4? > Not particularly. I just want to layer it on top of what's shipping in Fedora Rawhide to test it there locally. Admittedly, Rawhide is still at ad9f25d33860, which is just a bit above rc3. Nevertheless, the patch doesn't apply right now, so it's a bit hard to test beyond the "I compiled it locally and it doesn't break my system" level.
On 2021/5/31 下午8:17, Neal Gompa wrote: > On Mon, May 31, 2021 at 5:50 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote: >> >> >> >> On 2021/5/31 下午5:47, Neal Gompa wrote: >>> On Mon, May 31, 2021 at 4:52 AM Qu Wenruo <wqu@suse.com> wrote: >>>> >>>> This huge patchset can be fetched from github: >>>> https://github.com/adam900710/linux/tree/subpage >>>> >>>> === Current stage === >>>> The tests on x86 pass without new failure, and generic test group on >>>> arm64 with 64K page size passes except known failure and defrag group. >>>> >>>> For btrfs test group, all pass except compression/raid56/defrag. >>>> >>>> For anyone who is interested in testing, please apply this patch for >>>> btrfs-progs before testing. >>>> https://patchwork.kernel.org/project/linux-btrfs/patch/20210420073036.243715-1-wqu@suse.com/ >>>> Or there will be too many false alerts. >>>> >>>> === Limitation === >>>> There are several limitations introduced just for subpage: >>>> - No compressed write support >>>> Read is no problem, but compression write path has more things left to >>>> be modified. >>>> Thus for current patchset, no matter what inode attribute or mount >>>> option is, no new compressed extent can be created for subpage case. >>>> >>>> - No inline extent will be created >>>> This is mostly due to the fact that filemap_fdatawrite_range() will >>>> trigger more write than the range specified. >>>> In fallocate calls, this behavior can make us to writeback which can >>>> be inlined, before we enlarge the isize, causing inline extent being >>>> created along with regular extents. >>>> >>>> - No support for RAID56 >>>> There are still too many hardcoded PAGE_SIZE in raid56 code. >>>> Considering it's already considered unsafe due to its write-hole >>>> problem, disabling RAID56 for subpage looks sane to me. >>>> >>>> - No defrag support for subpage >>>> The support for subpage defrag has already an initial version >>>> submitted to the mail list. >>>> Thus the correct support won't be included in this patchset. >>>> >>>> === Patchset structure === >>>> >>>> Patch 01~19: Make data write path to be subpage compatible >>>> Patch 20~21: Make data relocation path to be subpage compatible >>>> Patch 22~29: Various fixes for subpage corner cases >>>> Patch 30: Enable subpage data write >>>> >>>> === Changelog === >>>> v2: >>>> - Rebased to latest misc-next >>>> Now metadata write patches are removed from the series, as they are >>>> already merged into misc-next. >>>> >>>> - Added new Reviewed-by/Tested-by/Reported-by tags >>>> >>>> - Use separate endio functions to subpage metadata write path >>>> >>>> - Re-order the patches, to make refactors at the top of the series >>>> One refactor, the submit_extent_page() one, should benefit 4K page >>>> size more than 64K page size, thus it's worthy to be merged early >>>> >>>> - New bug fixes exposed by Ritesh Harjani on Power >>>> >>>> - Reject RAID56 completely >>>> Exposed by btrfs test group, which caused BUG_ON() for various sites. >>>> Considering RAID56 is already not considered safe, it's better to >>>> reject them completely for now. >>>> >>>> - Fix subpage scrub repair failure >>>> Caused by hardcoded PAGE_SIZE >>>> >>>> - Fix free space cache inode size >>>> Same cause as scrub repair failure >>>> >>>> v3: >>>> - Rebased to remove write path prepration patches >>>> >>>> - Properly enable btrfs defrag >>>> Previsouly, btrfs defrag is in fact just disabled. >>>> This makes tons of tests in btrfs/defrag to fail. >>>> >>>> - More bug fixes for rare race/crashes >>>> * Fix relocation false alert on csum mismatch >>>> * Fix relocation data corruption >>>> * Fix a rare case of false ASSERT() >>>> The fix already get merged into the prepration patches, thus no >>>> longer in this patchset though. >>>> >>>> Mostly reported by Ritesh from IBM. >>>> >>>> v4: >>>> - Disable subpage defrag completely >>>> As full page defrag can race with fsstress in btrfs/062, causing >>>> strange ordered extent bugs. >>>> The full subpage defrag will be submitted as an indepdent patchset. >>>> >>>> Qu Wenruo (30): >>>> btrfs: pass bytenr directly to __process_pages_contig() >>>> btrfs: refactor the page status update into process_one_page() >>>> btrfs: provide btrfs_page_clamp_*() helpers >>>> btrfs: only require sector size alignment for >>>> end_bio_extent_writepage() >>>> btrfs: make btrfs_dirty_pages() to be subpage compatible >>>> btrfs: make __process_pages_contig() to handle subpage >>>> dirty/error/writeback status >>>> btrfs: make end_bio_extent_writepage() to be subpage compatible >>>> btrfs: make process_one_page() to handle subpage locking >>>> btrfs: introduce helpers for subpage ordered status >>>> btrfs: make page Ordered bit to be subpage compatible >>>> btrfs: update locked page dirty/writeback/error bits in >>>> __process_pages_contig >>>> btrfs: prevent extent_clear_unlock_delalloc() to unlock page not >>>> locked by __process_pages_contig() >>>> btrfs: make btrfs_set_range_writeback() subpage compatible >>>> btrfs: make __extent_writepage_io() only submit dirty range for >>>> subpage >>>> btrfs: make btrfs_truncate_block() to be subpage compatible >>>> btrfs: make btrfs_page_mkwrite() to be subpage compatible >>>> btrfs: reflink: make copy_inline_to_page() to be subpage compatible >>>> btrfs: fix the filemap_range_has_page() call in >>>> btrfs_punch_hole_lock_range() >>>> btrfs: don't clear page extent mapped if we're not invalidating the >>>> full page >>>> btrfs: extract relocation page read and dirty part into its own >>>> function >>>> btrfs: make relocate_one_page() to handle subpage case >>>> btrfs: fix wild subpage writeback which does not have ordered extent. >>>> btrfs: disable inline extent creation for subpage >>>> btrfs: allow submit_extent_page() to do bio split for subpage >>>> btrfs: reject raid5/6 fs for subpage >>>> btrfs: fix a crash caused by race between prepare_pages() and >>>> btrfs_releasepage() >>>> btrfs: fix a use-after-free bug in writeback subpage helper >>>> btrfs: fix a subpage false alert for relocating partial preallocated >>>> data extents >>>> btrfs: fix a subpage relocation data corruption >>>> btrfs: allow read-write for 4K sectorsize on 64K page size systems >>>> >>>> fs/btrfs/ctree.h | 2 +- >>>> fs/btrfs/disk-io.c | 13 +- >>>> fs/btrfs/extent_io.c | 563 ++++++++++++++++++++++++++++------------ >>>> fs/btrfs/file.c | 32 ++- >>>> fs/btrfs/inode.c | 147 +++++++++-- >>>> fs/btrfs/ioctl.c | 6 + >>>> fs/btrfs/ordered-data.c | 5 +- >>>> fs/btrfs/reflink.c | 14 +- >>>> fs/btrfs/relocation.c | 287 ++++++++++++-------- >>>> fs/btrfs/subpage.c | 156 ++++++++++- >>>> fs/btrfs/subpage.h | 31 +++ >>>> fs/btrfs/super.c | 7 - >>>> fs/btrfs/sysfs.c | 5 + >>>> fs/btrfs/volumes.c | 8 + >>>> 14 files changed, 949 insertions(+), 327 deletions(-) >>>> >>> >>> Could you please rebase your branch on 5.13-rc4? I'd rather test it on >>> top of that release... >>> >>> >> It can be rebased on david's misc-next branch without any conflicts. >> Although misc-next is only on v5.13-rc3, I don't believe there will be >> anything btrfs related out of misc-next. >> >> Is there anything special only show up in -rc4? >> > > Not particularly. I just want to layer it on top of what's shipping in > Fedora Rawhide to test it there locally. Admittedly, Rawhide is still > at ad9f25d33860, which is just a bit above rc3. Nevertheless, the > patch doesn't apply right now, so it's a bit hard to test beyond the > "I compiled it locally and it doesn't break my system" level. The failure to apply is more or less expected, as this patchset is dependent on subpage data write preparation patchset. Which is not yet merged, but already in misc-next branch. (Which also reduces overhead for 4K page size) Thus it's recommended to grab the branch, other than applying it upon upstream commit. Thanks, Qu
On Mon, May 31, 2021 at 04:50:36PM +0800, Qu Wenruo wrote: > This huge patchset can be fetched from github: > https://github.com/adam900710/linux/tree/subpage > > === Current stage === > The tests on x86 pass without new failure, and generic test group on > arm64 with 64K page size passes except known failure and defrag group. > > For btrfs test group, all pass except compression/raid56/defrag. > > For anyone who is interested in testing, please apply this patch for > btrfs-progs before testing. > https://patchwork.kernel.org/project/linux-btrfs/patch/20210420073036.243715-1-wqu@suse.com/ > Or there will be too many false alerts. This patch has been merged to progs 5.12. > > === Limitation === > There are several limitations introduced just for subpage: > - No compressed write support > Read is no problem, but compression write path has more things left to > be modified. > Thus for current patchset, no matter what inode attribute or mount > option is, no new compressed extent can be created for subpage case. > > - No inline extent will be created > This is mostly due to the fact that filemap_fdatawrite_range() will > trigger more write than the range specified. > In fallocate calls, this behavior can make us to writeback which can > be inlined, before we enlarge the isize, causing inline extent being > created along with regular extents. > > - No support for RAID56 > There are still too many hardcoded PAGE_SIZE in raid56 code. > Considering it's already considered unsafe due to its write-hole > problem, disabling RAID56 for subpage looks sane to me. > > - No defrag support for subpage > The support for subpage defrag has already an initial version > submitted to the mail list. > Thus the correct support won't be included in this patchset. > > === Patchset structure === > > Patch 01~19: Make data write path to be subpage compatible We can chop the first series batch again as this is still preparatory and we need to make sure it does not interfere with other patches.
On 2021/5/31 下午10:09, David Sterba wrote: > On Mon, May 31, 2021 at 04:50:36PM +0800, Qu Wenruo wrote: >> This huge patchset can be fetched from github: >> https://github.com/adam900710/linux/tree/subpage >> >> === Current stage === >> The tests on x86 pass without new failure, and generic test group on >> arm64 with 64K page size passes except known failure and defrag group. >> >> For btrfs test group, all pass except compression/raid56/defrag. >> >> For anyone who is interested in testing, please apply this patch for >> btrfs-progs before testing. >> https://patchwork.kernel.org/project/linux-btrfs/patch/20210420073036.243715-1-wqu@suse.com/ >> Or there will be too many false alerts. > > This patch has been merged to progs 5.12. >> >> === Limitation === >> There are several limitations introduced just for subpage: >> - No compressed write support >> Read is no problem, but compression write path has more things left to >> be modified. >> Thus for current patchset, no matter what inode attribute or mount >> option is, no new compressed extent can be created for subpage case. >> >> - No inline extent will be created >> This is mostly due to the fact that filemap_fdatawrite_range() will >> trigger more write than the range specified. >> In fallocate calls, this behavior can make us to writeback which can >> be inlined, before we enlarge the isize, causing inline extent being >> created along with regular extents. >> >> - No support for RAID56 >> There are still too many hardcoded PAGE_SIZE in raid56 code. >> Considering it's already considered unsafe due to its write-hole >> problem, disabling RAID56 for subpage looks sane to me. >> >> - No defrag support for subpage >> The support for subpage defrag has already an initial version >> submitted to the mail list. >> Thus the correct support won't be included in this patchset. >> >> === Patchset structure === >> >> Patch 01~19: Make data write path to be subpage compatible > > We can chop the first series batch again as this is still preparatory > and we need to make sure it does not interfere with other patches. > Considering how many tests those patches haven been tested, and how little they get modified during development, I'm totally fine to get them merged first. Thanks, Qu
On 21/05/31 04:50PM, Qu Wenruo wrote: > This huge patchset can be fetched from github: > https://github.com/adam900710/linux/tree/subpage > > === Current stage === > The tests on x86 pass without new failure, and generic test group on > arm64 with 64K page size passes except known failure and defrag group. > > For btrfs test group, all pass except compression/raid56/defrag. > > For anyone who is interested in testing, please apply this patch for > btrfs-progs before testing. > https://patchwork.kernel.org/project/linux-btrfs/patch/20210420073036.243715-1-wqu@suse.com/ > Or there will be too many false alerts. > > === Limitation === > There are several limitations introduced just for subpage: > - No compressed write support > Read is no problem, but compression write path has more things left to > be modified. > Thus for current patchset, no matter what inode attribute or mount > option is, no new compressed extent can be created for subpage case. > > - No inline extent will be created > This is mostly due to the fact that filemap_fdatawrite_range() will > trigger more write than the range specified. > In fallocate calls, this behavior can make us to writeback which can > be inlined, before we enlarge the isize, causing inline extent being > created along with regular extents. > > - No support for RAID56 > There are still too many hardcoded PAGE_SIZE in raid56 code. > Considering it's already considered unsafe due to its write-hole > problem, disabling RAID56 for subpage looks sane to me. > > - No defrag support for subpage > The support for subpage defrag has already an initial version > submitted to the mail list. > Thus the correct support won't be included in this patchset. > > === Patchset structure === > > Patch 01~19: Make data write path to be subpage compatible > Patch 20~21: Make data relocation path to be subpage compatible > Patch 22~29: Various fixes for subpage corner cases > Patch 30: Enable subpage data write > > === Changelog === > v2: > - Rebased to latest misc-next > Now metadata write patches are removed from the series, as they are > already merged into misc-next. > > - Added new Reviewed-by/Tested-by/Reported-by tags > > - Use separate endio functions to subpage metadata write path > > - Re-order the patches, to make refactors at the top of the series > One refactor, the submit_extent_page() one, should benefit 4K page > size more than 64K page size, thus it's worthy to be merged early > > - New bug fixes exposed by Ritesh Harjani on Power > > - Reject RAID56 completely > Exposed by btrfs test group, which caused BUG_ON() for various sites. > Considering RAID56 is already not considered safe, it's better to > reject them completely for now. > > - Fix subpage scrub repair failure > Caused by hardcoded PAGE_SIZE > > - Fix free space cache inode size > Same cause as scrub repair failure > > v3: > - Rebased to remove write path prepration patches > > - Properly enable btrfs defrag > Previsouly, btrfs defrag is in fact just disabled. > This makes tons of tests in btrfs/defrag to fail. > > - More bug fixes for rare race/crashes > * Fix relocation false alert on csum mismatch > * Fix relocation data corruption > * Fix a rare case of false ASSERT() > The fix already get merged into the prepration patches, thus no > longer in this patchset though. > > Mostly reported by Ritesh from IBM. > > v4: > - Disable subpage defrag completely > As full page defrag can race with fsstress in btrfs/062, causing > strange ordered extent bugs. > The full subpage defrag will be submitted as an indepdent patchset. Hello Qu, I have completed another 2 full iterations of testing this v4 with "-g all" on Power. There are no failures reported (other than the known ones mostly due to defrag disabled) and no kernel warnings/errors reported with v4 of your patch series (other than a known one causing transaction abort -28 msg, but that seems to be triggering even w/o your patch series i.e. with 64k blocksize too). From the test perspective, please feel free to add below for your v4. Tested-by: Ritesh Harjani <riteshh@linux.ibm.com> [ppc64] I think since this patch series looks good, I can now start helping with defrag patch series of yours :) That should be on top of this v4 I guess. Thanks a lot for your effort and help!! -ritesh
On 2021/6/2 上午10:22, riteshh wrote: > On 21/05/31 04:50PM, Qu Wenruo wrote: >> This huge patchset can be fetched from github: >> https://github.com/adam900710/linux/tree/subpage >> >> === Current stage === >> The tests on x86 pass without new failure, and generic test group on >> arm64 with 64K page size passes except known failure and defrag group. >> >> For btrfs test group, all pass except compression/raid56/defrag. >> >> For anyone who is interested in testing, please apply this patch for >> btrfs-progs before testing. >> https://patchwork.kernel.org/project/linux-btrfs/patch/20210420073036.243715-1-wqu@suse.com/ >> Or there will be too many false alerts. >> >> === Limitation === >> There are several limitations introduced just for subpage: >> - No compressed write support >> Read is no problem, but compression write path has more things left to >> be modified. >> Thus for current patchset, no matter what inode attribute or mount >> option is, no new compressed extent can be created for subpage case. >> >> - No inline extent will be created >> This is mostly due to the fact that filemap_fdatawrite_range() will >> trigger more write than the range specified. >> In fallocate calls, this behavior can make us to writeback which can >> be inlined, before we enlarge the isize, causing inline extent being >> created along with regular extents. >> >> - No support for RAID56 >> There are still too many hardcoded PAGE_SIZE in raid56 code. >> Considering it's already considered unsafe due to its write-hole >> problem, disabling RAID56 for subpage looks sane to me. >> >> - No defrag support for subpage >> The support for subpage defrag has already an initial version >> submitted to the mail list. >> Thus the correct support won't be included in this patchset. >> >> === Patchset structure === >> >> Patch 01~19: Make data write path to be subpage compatible >> Patch 20~21: Make data relocation path to be subpage compatible >> Patch 22~29: Various fixes for subpage corner cases >> Patch 30: Enable subpage data write >> >> === Changelog === >> v2: >> - Rebased to latest misc-next >> Now metadata write patches are removed from the series, as they are >> already merged into misc-next. >> >> - Added new Reviewed-by/Tested-by/Reported-by tags >> >> - Use separate endio functions to subpage metadata write path >> >> - Re-order the patches, to make refactors at the top of the series >> One refactor, the submit_extent_page() one, should benefit 4K page >> size more than 64K page size, thus it's worthy to be merged early >> >> - New bug fixes exposed by Ritesh Harjani on Power >> >> - Reject RAID56 completely >> Exposed by btrfs test group, which caused BUG_ON() for various sites. >> Considering RAID56 is already not considered safe, it's better to >> reject them completely for now. >> >> - Fix subpage scrub repair failure >> Caused by hardcoded PAGE_SIZE >> >> - Fix free space cache inode size >> Same cause as scrub repair failure >> >> v3: >> - Rebased to remove write path prepration patches >> >> - Properly enable btrfs defrag >> Previsouly, btrfs defrag is in fact just disabled. >> This makes tons of tests in btrfs/defrag to fail. >> >> - More bug fixes for rare race/crashes >> * Fix relocation false alert on csum mismatch >> * Fix relocation data corruption >> * Fix a rare case of false ASSERT() >> The fix already get merged into the prepration patches, thus no >> longer in this patchset though. >> >> Mostly reported by Ritesh from IBM. >> >> v4: >> - Disable subpage defrag completely >> As full page defrag can race with fsstress in btrfs/062, causing >> strange ordered extent bugs. >> The full subpage defrag will be submitted as an indepdent patchset. > > Hello Qu, > > I have completed another 2 full iterations of testing this v4 with "-g all" on > Power. There are no failures reported (other than the known ones mostly due to > defrag disabled) and no kernel warnings/errors reported with v4 of your patch > series (other than a known one causing transaction abort -28 msg, but that seems > to be triggering even w/o your patch series i.e. with 64k blocksize too). > > From the test perspective, please feel free to add below for your v4. > > Tested-by: Ritesh Harjani <riteshh@linux.ibm.com> [ppc64] > > > I think since this patch series looks good, I can now start helping with defrag > patch series of yours :) > That should be on top of this v4 I guess. Exactly, and that has been just submitted. You can also fetch the branch from 'defrag_refactor' branch from my github repo. And since that branch only touches defrag ioctl, free feel to just test '-g defrag' test cases if you want a quick run. Thanks, Qu > > Thanks a lot for your effort and help!! > -ritesh > >
On 21/06/02 10:24AM, Qu Wenruo wrote: > > > On 2021/6/2 上午10:22, riteshh wrote: > > On 21/05/31 04:50PM, Qu Wenruo wrote: > > > This huge patchset can be fetched from github: > > > https://github.com/adam900710/linux/tree/subpage > > > > > > === Current stage === > > > The tests on x86 pass without new failure, and generic test group on > > > arm64 with 64K page size passes except known failure and defrag group. > > > > > > For btrfs test group, all pass except compression/raid56/defrag. > > > > > > For anyone who is interested in testing, please apply this patch for > > > btrfs-progs before testing. > > > https://patchwork.kernel.org/project/linux-btrfs/patch/20210420073036.243715-1-wqu@suse.com/ > > > Or there will be too many false alerts. > > > > > > === Limitation === > > > There are several limitations introduced just for subpage: > > > - No compressed write support > > > Read is no problem, but compression write path has more things left to > > > be modified. > > > Thus for current patchset, no matter what inode attribute or mount > > > option is, no new compressed extent can be created for subpage case. > > > > > > - No inline extent will be created > > > This is mostly due to the fact that filemap_fdatawrite_range() will > > > trigger more write than the range specified. > > > In fallocate calls, this behavior can make us to writeback which can > > > be inlined, before we enlarge the isize, causing inline extent being > > > created along with regular extents. > > > > > > - No support for RAID56 > > > There are still too many hardcoded PAGE_SIZE in raid56 code. > > > Considering it's already considered unsafe due to its write-hole > > > problem, disabling RAID56 for subpage looks sane to me. > > > > > > - No defrag support for subpage > > > The support for subpage defrag has already an initial version > > > submitted to the mail list. > > > Thus the correct support won't be included in this patchset. > > > > > > === Patchset structure === > > > > > > Patch 01~19: Make data write path to be subpage compatible > > > Patch 20~21: Make data relocation path to be subpage compatible > > > Patch 22~29: Various fixes for subpage corner cases > > > Patch 30: Enable subpage data write > > > > > > === Changelog === > > > v2: > > > - Rebased to latest misc-next > > > Now metadata write patches are removed from the series, as they are > > > already merged into misc-next. > > > > > > - Added new Reviewed-by/Tested-by/Reported-by tags > > > > > > - Use separate endio functions to subpage metadata write path > > > > > > - Re-order the patches, to make refactors at the top of the series > > > One refactor, the submit_extent_page() one, should benefit 4K page > > > size more than 64K page size, thus it's worthy to be merged early > > > > > > - New bug fixes exposed by Ritesh Harjani on Power > > > > > > - Reject RAID56 completely > > > Exposed by btrfs test group, which caused BUG_ON() for various sites. > > > Considering RAID56 is already not considered safe, it's better to > > > reject them completely for now. > > > > > > - Fix subpage scrub repair failure > > > Caused by hardcoded PAGE_SIZE > > > > > > - Fix free space cache inode size > > > Same cause as scrub repair failure > > > > > > v3: > > > - Rebased to remove write path prepration patches > > > > > > - Properly enable btrfs defrag > > > Previsouly, btrfs defrag is in fact just disabled. > > > This makes tons of tests in btrfs/defrag to fail. > > > > > > - More bug fixes for rare race/crashes > > > * Fix relocation false alert on csum mismatch > > > * Fix relocation data corruption > > > * Fix a rare case of false ASSERT() > > > The fix already get merged into the prepration patches, thus no > > > longer in this patchset though. > > > > > > Mostly reported by Ritesh from IBM. > > > > > > v4: > > > - Disable subpage defrag completely > > > As full page defrag can race with fsstress in btrfs/062, causing > > > strange ordered extent bugs. > > > The full subpage defrag will be submitted as an indepdent patchset. > > > > Hello Qu, > > > > I have completed another 2 full iterations of testing this v4 with "-g all" on > > Power. There are no failures reported (other than the known ones mostly due to > > defrag disabled) and no kernel warnings/errors reported with v4 of your patch > > series (other than a known one causing transaction abort -28 msg, but that seems > > to be triggering even w/o your patch series i.e. with 64k blocksize too). > > > > From the test perspective, please feel free to add below for your v4. > > > > Tested-by: Ritesh Harjani <riteshh@linux.ibm.com> [ppc64] > > > > > > I think since this patch series looks good, I can now start helping with defrag > > patch series of yours :) > > That should be on top of this v4 I guess. > > Exactly, and that has been just submitted. > > You can also fetch the branch from 'defrag_refactor' branch from my github > repo. > > And since that branch only touches defrag ioctl, free feel to just test '-g > defrag' test cases if you want a quick run. Great! Thanks for the quick response. Will start my testing. -ritesh
On Wed, Jun 02, 2021 at 07:52:39AM +0530, riteshh wrote: > I have completed another 2 full iterations of testing this v4 with "-g all" on > Power. There are no failures reported (other than the known ones mostly due to > defrag disabled) and no kernel warnings/errors reported with v4 of your patch > series (other than a known one causing transaction abort -28 msg, but that seems > to be triggering even w/o your patch series i.e. with 64k blocksize too). > > >From the test perspective, please feel free to add below for your v4. > > Tested-by: Ritesh Harjani <riteshh@linux.ibm.com> [ppc64] Thank you very much for testing, I've added the tag to the series.
On Mon, May 31, 2021 at 04:50:36PM +0800, Qu Wenruo wrote: > This huge patchset can be fetched from github: > https://github.com/adam900710/linux/tree/subpage > > v4: > - Disable subpage defrag completely > As full page defrag can race with fsstress in btrfs/062, causing > strange ordered extent bugs. > The full subpage defrag will be submitted as an indepdent patchset. I went through the whole series and now it's going through fstests (4k pages, x86) before I move it to misc-next. The write support is limiting some features but as it's still experimental I don't see a reason not to enable it. The defrag series has been posted, so that's the next step, compression won't be difficult to fix I hope and the inline support is not critical and will be addressed eventually. I will probably reorder the patches in misc-next so the whole series is in one batch in case we need to bisect it. As this is first implementation, there are some known things to clean up or refactor. I'd rather postpone that for next cycle so there are no conflicts when merging fixes. Thank you for taking the subpage support to a working state, even with the few limitations.
On 2021/6/3 上午1:57, David Sterba wrote: > On Mon, May 31, 2021 at 04:50:36PM +0800, Qu Wenruo wrote: >> This huge patchset can be fetched from github: >> https://github.com/adam900710/linux/tree/subpage >> >> v4: >> - Disable subpage defrag completely >> As full page defrag can race with fsstress in btrfs/062, causing >> strange ordered extent bugs. >> The full subpage defrag will be submitted as an indepdent patchset. > > I went through the whole series and now it's going through fstests (4k > pages, x86) before I move it to misc-next. The write support is > limiting some features but as it's still experimental I don't see a > reason not to enable it. Thank you very much! > The defrag series has been posted, so that's > the next step, compression won't be difficult to fix I hope and the > inline support is not critical and will be addressed eventually. Compression shouldn't be that complex, but I believe compression may have more hidden corners, thus I'm not in that a hurry. My main concern is the inline part. In fact, if applied my btrfs check patches, even on x86_64, we can have fsstress to create inline extent mixed with regular extent inside one inode. Although it doesn't cause any data corruption, if we really want to inline extent exclusive, we need to address that first. > > I will probably reorder the patches in misc-next so the whole series is > in one batch in case we need to bisect it. I'm OK with the re-order, but I don't believe we really do for bisect. As subpage is not enabled until the last patch, I doubt if we can get any useful info from bisect. > As this is first > implementation, there are some known things to clean up or refactor. I'd > rather postpone that for next cycle so there are no conflicts when > merging fixes. Yeah, sure. If you have any cleanups, I'm ready to learn from them and prevent the same problem from happening in incoming patches. Thanks, Qu > > Thank you for taking the subpage support to a working state, even with > the few limitations. >
On 31/5/21 4:50 pm, Qu Wenruo wrote: > This huge patchset can be fetched from github: > https://github.com/adam900710/linux/tree/subpage > > === Current stage === > The tests on x86 pass without new failure, and generic test group on > arm64 with 64K page size passes except known failure and defrag group. > > For btrfs test group, all pass except compression/raid56/defrag. > > For anyone who is interested in testing, please apply this patch for > btrfs-progs before testing. > https://patchwork.kernel.org/project/linux-btrfs/patch/20210420073036.243715-1-wqu@suse.com/ > Or there will be too many false alerts. > > === Limitation === > There are several limitations introduced just for subpage: > - No compressed write support > Read is no problem, but compression write path has more things left to > be modified. > Thus for current patchset, no matter what inode attribute or mount > option is, no new compressed extent can be created for subpage case. > > - No inline extent will be created > This is mostly due to the fact that filemap_fdatawrite_range() will > trigger more write than the range specified. > In fallocate calls, this behavior can make us to writeback which can > be inlined, before we enlarge the isize, causing inline extent being > created along with regular extents. > > - No support for RAID56 > There are still too many hardcoded PAGE_SIZE in raid56 code. > Considering it's already considered unsafe due to its write-hole > problem, disabling RAID56 for subpage looks sane to me. > > - No defrag support for subpage > The support for subpage defrag has already an initial version > submitted to the mail list. > Thus the correct support won't be included in this patchset. > I am confused about what supports as of now? /sys/fs/btrfs/features/supported_sectorsizes list just the pagesize. Thanks, Anand > === Patchset structure === > > Patch 01~19: Make data write path to be subpage compatible > Patch 20~21: Make data relocation path to be subpage compatible > Patch 22~29: Various fixes for subpage corner cases > Patch 30: Enable subpage data write > > === Changelog === > v2: > - Rebased to latest misc-next > Now metadata write patches are removed from the series, as they are > already merged into misc-next. > > - Added new Reviewed-by/Tested-by/Reported-by tags > > - Use separate endio functions to subpage metadata write path > > - Re-order the patches, to make refactors at the top of the series > One refactor, the submit_extent_page() one, should benefit 4K page > size more than 64K page size, thus it's worthy to be merged early > > - New bug fixes exposed by Ritesh Harjani on Power > > - Reject RAID56 completely > Exposed by btrfs test group, which caused BUG_ON() for various sites. > Considering RAID56 is already not considered safe, it's better to > reject them completely for now. > > - Fix subpage scrub repair failure > Caused by hardcoded PAGE_SIZE > > - Fix free space cache inode size > Same cause as scrub repair failure > > v3: > - Rebased to remove write path prepration patches > > - Properly enable btrfs defrag > Previsouly, btrfs defrag is in fact just disabled. > This makes tons of tests in btrfs/defrag to fail. > > - More bug fixes for rare race/crashes > * Fix relocation false alert on csum mismatch > * Fix relocation data corruption > * Fix a rare case of false ASSERT() > The fix already get merged into the prepration patches, thus no > longer in this patchset though. > > Mostly reported by Ritesh from IBM. > > v4: > - Disable subpage defrag completely > As full page defrag can race with fsstress in btrfs/062, causing > strange ordered extent bugs. > The full subpage defrag will be submitted as an indepdent patchset. > > Qu Wenruo (30): > btrfs: pass bytenr directly to __process_pages_contig() > btrfs: refactor the page status update into process_one_page() > btrfs: provide btrfs_page_clamp_*() helpers > btrfs: only require sector size alignment for > end_bio_extent_writepage() > btrfs: make btrfs_dirty_pages() to be subpage compatible > btrfs: make __process_pages_contig() to handle subpage > dirty/error/writeback status > btrfs: make end_bio_extent_writepage() to be subpage compatible > btrfs: make process_one_page() to handle subpage locking > btrfs: introduce helpers for subpage ordered status > btrfs: make page Ordered bit to be subpage compatible > btrfs: update locked page dirty/writeback/error bits in > __process_pages_contig > btrfs: prevent extent_clear_unlock_delalloc() to unlock page not > locked by __process_pages_contig() > btrfs: make btrfs_set_range_writeback() subpage compatible > btrfs: make __extent_writepage_io() only submit dirty range for > subpage > btrfs: make btrfs_truncate_block() to be subpage compatible > btrfs: make btrfs_page_mkwrite() to be subpage compatible > btrfs: reflink: make copy_inline_to_page() to be subpage compatible > btrfs: fix the filemap_range_has_page() call in > btrfs_punch_hole_lock_range() > btrfs: don't clear page extent mapped if we're not invalidating the > full page > btrfs: extract relocation page read and dirty part into its own > function > btrfs: make relocate_one_page() to handle subpage case > btrfs: fix wild subpage writeback which does not have ordered extent. > btrfs: disable inline extent creation for subpage > btrfs: allow submit_extent_page() to do bio split for subpage > btrfs: reject raid5/6 fs for subpage > btrfs: fix a crash caused by race between prepare_pages() and > btrfs_releasepage() > btrfs: fix a use-after-free bug in writeback subpage helper > btrfs: fix a subpage false alert for relocating partial preallocated > data extents > btrfs: fix a subpage relocation data corruption > btrfs: allow read-write for 4K sectorsize on 64K page size systems > > fs/btrfs/ctree.h | 2 +- > fs/btrfs/disk-io.c | 13 +- > fs/btrfs/extent_io.c | 563 ++++++++++++++++++++++++++++------------ > fs/btrfs/file.c | 32 ++- > fs/btrfs/inode.c | 147 +++++++++-- > fs/btrfs/ioctl.c | 6 + > fs/btrfs/ordered-data.c | 5 +- > fs/btrfs/reflink.c | 14 +- > fs/btrfs/relocation.c | 287 ++++++++++++-------- > fs/btrfs/subpage.c | 156 ++++++++++- > fs/btrfs/subpage.h | 31 +++ > fs/btrfs/super.c | 7 - > fs/btrfs/sysfs.c | 5 + > fs/btrfs/volumes.c | 8 + > 14 files changed, 949 insertions(+), 327 deletions(-) >
On 2021/6/8 下午4:23, Anand Jain wrote: > On 31/5/21 4:50 pm, Qu Wenruo wrote: >> This huge patchset can be fetched from github: >> https://github.com/adam900710/linux/tree/subpage >> >> === Current stage === >> The tests on x86 pass without new failure, and generic test group on >> arm64 with 64K page size passes except known failure and defrag group. >> >> For btrfs test group, all pass except compression/raid56/defrag. >> >> For anyone who is interested in testing, please apply this patch for >> btrfs-progs before testing. >> https://patchwork.kernel.org/project/linux-btrfs/patch/20210420073036.243715-1-wqu@suse.com/ >> >> Or there will be too many false alerts. >> >> === Limitation === >> There are several limitations introduced just for subpage: >> - No compressed write support >> Read is no problem, but compression write path has more things left to >> be modified. >> Thus for current patchset, no matter what inode attribute or mount >> option is, no new compressed extent can be created for subpage case. >> >> - No inline extent will be created >> This is mostly due to the fact that filemap_fdatawrite_range() will >> trigger more write than the range specified. >> In fallocate calls, this behavior can make us to writeback which can >> be inlined, before we enlarge the isize, causing inline extent being >> created along with regular extents. >> >> - No support for RAID56 >> There are still too many hardcoded PAGE_SIZE in raid56 code. >> Considering it's already considered unsafe due to its write-hole >> problem, disabling RAID56 for subpage looks sane to me. >> >> - No defrag support for subpage >> The support for subpage defrag has already an initial version >> submitted to the mail list. >> Thus the correct support won't be included in this patchset. >> > > I am confused about what supports as of now? From my latest subpage branch, everything work as expected. Defrag support is in another patchset, which can be applied independently. > /sys/fs/btrfs/features/supported_sectorsizes [adam@arm-btrfs linux]$ uname -a Linux arm-btrfs 5.13.0-rc2-custom+ #5 SMP Tue Jun 1 16:11:41 CST 2021 aarch64 GNU/Linux [adam@arm-btrfs linux]$ getconf PAGESIZE 65536 [adam@arm-btrfs linux]$ cat /sys/fs/btrfs/features/supported_sectorsizes 4096 65536 It still shows 4k as support sectorsize. What's your branch/HEAD? And are you using 64K page size? Thanks, Qu > list just the pagesize. > > Thanks, Anand > > >> === Patchset structure === >> >> Patch 01~19: Make data write path to be subpage compatible >> Patch 20~21: Make data relocation path to be subpage compatible >> Patch 22~29: Various fixes for subpage corner cases >> Patch 30: Enable subpage data write >> >> === Changelog === >> v2: >> - Rebased to latest misc-next >> Now metadata write patches are removed from the series, as they are >> already merged into misc-next. >> >> - Added new Reviewed-by/Tested-by/Reported-by tags >> >> - Use separate endio functions to subpage metadata write path >> >> - Re-order the patches, to make refactors at the top of the series >> One refactor, the submit_extent_page() one, should benefit 4K page >> size more than 64K page size, thus it's worthy to be merged early >> >> - New bug fixes exposed by Ritesh Harjani on Power >> >> - Reject RAID56 completely >> Exposed by btrfs test group, which caused BUG_ON() for various sites. >> Considering RAID56 is already not considered safe, it's better to >> reject them completely for now. >> >> - Fix subpage scrub repair failure >> Caused by hardcoded PAGE_SIZE >> >> - Fix free space cache inode size >> Same cause as scrub repair failure >> >> v3: >> - Rebased to remove write path prepration patches >> >> - Properly enable btrfs defrag >> Previsouly, btrfs defrag is in fact just disabled. >> This makes tons of tests in btrfs/defrag to fail. >> >> - More bug fixes for rare race/crashes >> * Fix relocation false alert on csum mismatch >> * Fix relocation data corruption >> * Fix a rare case of false ASSERT() >> The fix already get merged into the prepration patches, thus no >> longer in this patchset though. >> Mostly reported by Ritesh from IBM. >> >> v4: >> - Disable subpage defrag completely >> As full page defrag can race with fsstress in btrfs/062, causing >> strange ordered extent bugs. >> The full subpage defrag will be submitted as an indepdent patchset. >> >> Qu Wenruo (30): >> btrfs: pass bytenr directly to __process_pages_contig() >> btrfs: refactor the page status update into process_one_page() >> btrfs: provide btrfs_page_clamp_*() helpers >> btrfs: only require sector size alignment for >> end_bio_extent_writepage() >> btrfs: make btrfs_dirty_pages() to be subpage compatible >> btrfs: make __process_pages_contig() to handle subpage >> dirty/error/writeback status >> btrfs: make end_bio_extent_writepage() to be subpage compatible >> btrfs: make process_one_page() to handle subpage locking >> btrfs: introduce helpers for subpage ordered status >> btrfs: make page Ordered bit to be subpage compatible >> btrfs: update locked page dirty/writeback/error bits in >> __process_pages_contig >> btrfs: prevent extent_clear_unlock_delalloc() to unlock page not >> locked by __process_pages_contig() >> btrfs: make btrfs_set_range_writeback() subpage compatible >> btrfs: make __extent_writepage_io() only submit dirty range for >> subpage >> btrfs: make btrfs_truncate_block() to be subpage compatible >> btrfs: make btrfs_page_mkwrite() to be subpage compatible >> btrfs: reflink: make copy_inline_to_page() to be subpage compatible >> btrfs: fix the filemap_range_has_page() call in >> btrfs_punch_hole_lock_range() >> btrfs: don't clear page extent mapped if we're not invalidating the >> full page >> btrfs: extract relocation page read and dirty part into its own >> function >> btrfs: make relocate_one_page() to handle subpage case >> btrfs: fix wild subpage writeback which does not have ordered extent. >> btrfs: disable inline extent creation for subpage >> btrfs: allow submit_extent_page() to do bio split for subpage >> btrfs: reject raid5/6 fs for subpage >> btrfs: fix a crash caused by race between prepare_pages() and >> btrfs_releasepage() >> btrfs: fix a use-after-free bug in writeback subpage helper >> btrfs: fix a subpage false alert for relocating partial preallocated >> data extents >> btrfs: fix a subpage relocation data corruption >> btrfs: allow read-write for 4K sectorsize on 64K page size systems >> >> fs/btrfs/ctree.h | 2 +- >> fs/btrfs/disk-io.c | 13 +- >> fs/btrfs/extent_io.c | 563 ++++++++++++++++++++++++++++------------ >> fs/btrfs/file.c | 32 ++- >> fs/btrfs/inode.c | 147 +++++++++-- >> fs/btrfs/ioctl.c | 6 + >> fs/btrfs/ordered-data.c | 5 +- >> fs/btrfs/reflink.c | 14 +- >> fs/btrfs/relocation.c | 287 ++++++++++++-------- >> fs/btrfs/subpage.c | 156 ++++++++++- >> fs/btrfs/subpage.h | 31 +++ >> fs/btrfs/super.c | 7 - >> fs/btrfs/sysfs.c | 5 + >> fs/btrfs/volumes.c | 8 + >> 14 files changed, 949 insertions(+), 327 deletions(-) >> >
On 8/6/21 5:02 pm, Qu Wenruo wrote: > > > On 2021/6/8 下午4:23, Anand Jain wrote: >> On 31/5/21 4:50 pm, Qu Wenruo wrote: >>> This huge patchset can be fetched from github: >>> https://github.com/adam900710/linux/tree/subpage >>> >>> === Current stage === >>> The tests on x86 pass without new failure, and generic test group on >>> arm64 with 64K page size passes except known failure and defrag group. >>> >>> For btrfs test group, all pass except compression/raid56/defrag. >>> >>> For anyone who is interested in testing, please apply this patch for >>> btrfs-progs before testing. >>> https://patchwork.kernel.org/project/linux-btrfs/patch/20210420073036.243715-1-wqu@suse.com/ >>> >>> >>> Or there will be too many false alerts. >>> >>> === Limitation === >>> There are several limitations introduced just for subpage: >>> - No compressed write support >>> Read is no problem, but compression write path has more things >>> left to >>> be modified. >>> Thus for current patchset, no matter what inode attribute or mount >>> option is, no new compressed extent can be created for subpage case. >>> >>> - No inline extent will be created >>> This is mostly due to the fact that filemap_fdatawrite_range() will >>> trigger more write than the range specified. >>> In fallocate calls, this behavior can make us to writeback which can >>> be inlined, before we enlarge the isize, causing inline extent being >>> created along with regular extents. >>> >>> - No support for RAID56 >>> There are still too many hardcoded PAGE_SIZE in raid56 code. >>> Considering it's already considered unsafe due to its write-hole >>> problem, disabling RAID56 for subpage looks sane to me. >>> >>> - No defrag support for subpage >>> The support for subpage defrag has already an initial version >>> submitted to the mail list. >>> Thus the correct support won't be included in this patchset. >>> >> >> I am confused about what supports as of now? > > From my latest subpage branch, everything work as expected. > > Defrag support is in another patchset, which can be applied independently. > >> /sys/fs/btrfs/features/supported_sectorsizes > > [adam@arm-btrfs linux]$ uname -a > Linux arm-btrfs 5.13.0-rc2-custom+ #5 SMP Tue Jun 1 16:11:41 CST 2021 > aarch64 GNU/Linux > [adam@arm-btrfs linux]$ getconf PAGESIZE > 65536 > [adam@arm-btrfs linux]$ cat /sys/fs/btrfs/features/supported_sectorsizes > 4096 65536 > > It still shows 4k as support sectorsize. > > What's your branch/HEAD? And are you using 64K page size? > I am on misc-next (which contains all these 30 patches). I don't see subpages supported. Is there any patch I missed? $ cat /sys/fs/btrfs/features/supported_sectorsizes 65536 Thanks, Anand > Thanks, > Qu > >> list just the pagesize. >> >> Thanks, Anand >> >> >>> === Patchset structure === >>> >>> Patch 01~19: Make data write path to be subpage compatible >>> Patch 20~21: Make data relocation path to be subpage compatible >>> Patch 22~29: Various fixes for subpage corner cases >>> Patch 30: Enable subpage data write >>> >>> === Changelog === >>> v2: >>> - Rebased to latest misc-next >>> Now metadata write patches are removed from the series, as they are >>> already merged into misc-next. >>> >>> - Added new Reviewed-by/Tested-by/Reported-by tags >>> >>> - Use separate endio functions to subpage metadata write path >>> >>> - Re-order the patches, to make refactors at the top of the series >>> One refactor, the submit_extent_page() one, should benefit 4K page >>> size more than 64K page size, thus it's worthy to be merged early >>> >>> - New bug fixes exposed by Ritesh Harjani on Power >>> >>> - Reject RAID56 completely >>> Exposed by btrfs test group, which caused BUG_ON() for various sites. >>> Considering RAID56 is already not considered safe, it's better to >>> reject them completely for now. >>> >>> - Fix subpage scrub repair failure >>> Caused by hardcoded PAGE_SIZE >>> >>> - Fix free space cache inode size >>> Same cause as scrub repair failure >>> >>> v3: >>> - Rebased to remove write path prepration patches >>> >>> - Properly enable btrfs defrag >>> Previsouly, btrfs defrag is in fact just disabled. >>> This makes tons of tests in btrfs/defrag to fail. >>> >>> - More bug fixes for rare race/crashes >>> * Fix relocation false alert on csum mismatch >>> * Fix relocation data corruption >>> * Fix a rare case of false ASSERT() >>> The fix already get merged into the prepration patches, thus no >>> longer in this patchset though. >>> Mostly reported by Ritesh from IBM. >>> >>> v4: >>> - Disable subpage defrag completely >>> As full page defrag can race with fsstress in btrfs/062, causing >>> strange ordered extent bugs. >>> The full subpage defrag will be submitted as an indepdent patchset. >>> >>> Qu Wenruo (30): >>> btrfs: pass bytenr directly to __process_pages_contig() >>> btrfs: refactor the page status update into process_one_page() >>> btrfs: provide btrfs_page_clamp_*() helpers >>> btrfs: only require sector size alignment for >>> end_bio_extent_writepage() >>> btrfs: make btrfs_dirty_pages() to be subpage compatible >>> btrfs: make __process_pages_contig() to handle subpage >>> dirty/error/writeback status >>> btrfs: make end_bio_extent_writepage() to be subpage compatible >>> btrfs: make process_one_page() to handle subpage locking >>> btrfs: introduce helpers for subpage ordered status >>> btrfs: make page Ordered bit to be subpage compatible >>> btrfs: update locked page dirty/writeback/error bits in >>> __process_pages_contig >>> btrfs: prevent extent_clear_unlock_delalloc() to unlock page not >>> locked by __process_pages_contig() >>> btrfs: make btrfs_set_range_writeback() subpage compatible >>> btrfs: make __extent_writepage_io() only submit dirty range for >>> subpage >>> btrfs: make btrfs_truncate_block() to be subpage compatible >>> btrfs: make btrfs_page_mkwrite() to be subpage compatible >>> btrfs: reflink: make copy_inline_to_page() to be subpage compatible >>> btrfs: fix the filemap_range_has_page() call in >>> btrfs_punch_hole_lock_range() >>> btrfs: don't clear page extent mapped if we're not invalidating the >>> full page >>> btrfs: extract relocation page read and dirty part into its own >>> function >>> btrfs: make relocate_one_page() to handle subpage case >>> btrfs: fix wild subpage writeback which does not have ordered extent. >>> btrfs: disable inline extent creation for subpage >>> btrfs: allow submit_extent_page() to do bio split for subpage >>> btrfs: reject raid5/6 fs for subpage >>> btrfs: fix a crash caused by race between prepare_pages() and >>> btrfs_releasepage() >>> btrfs: fix a use-after-free bug in writeback subpage helper >>> btrfs: fix a subpage false alert for relocating partial preallocated >>> data extents >>> btrfs: fix a subpage relocation data corruption >>> btrfs: allow read-write for 4K sectorsize on 64K page size systems >>> >>> fs/btrfs/ctree.h | 2 +- >>> fs/btrfs/disk-io.c | 13 +- >>> fs/btrfs/extent_io.c | 563 ++++++++++++++++++++++++++++------------ >>> fs/btrfs/file.c | 32 ++- >>> fs/btrfs/inode.c | 147 +++++++++-- >>> fs/btrfs/ioctl.c | 6 + >>> fs/btrfs/ordered-data.c | 5 +- >>> fs/btrfs/reflink.c | 14 +- >>> fs/btrfs/relocation.c | 287 ++++++++++++-------- >>> fs/btrfs/subpage.c | 156 ++++++++++- >>> fs/btrfs/subpage.h | 31 +++ >>> fs/btrfs/super.c | 7 - >>> fs/btrfs/sysfs.c | 5 + >>> fs/btrfs/volumes.c | 8 + >>> 14 files changed, 949 insertions(+), 327 deletions(-) >>> >>
On 2021/6/8 下午5:45, Anand Jain wrote: > > > On 8/6/21 5:02 pm, Qu Wenruo wrote: >> >> >> On 2021/6/8 下午4:23, Anand Jain wrote: >>> On 31/5/21 4:50 pm, Qu Wenruo wrote: >>>> This huge patchset can be fetched from github: >>>> https://github.com/adam900710/linux/tree/subpage >>>> >>>> === Current stage === >>>> The tests on x86 pass without new failure, and generic test group on >>>> arm64 with 64K page size passes except known failure and defrag group. >>>> >>>> For btrfs test group, all pass except compression/raid56/defrag. >>>> >>>> For anyone who is interested in testing, please apply this patch for >>>> btrfs-progs before testing. >>>> https://patchwork.kernel.org/project/linux-btrfs/patch/20210420073036.243715-1-wqu@suse.com/ >>>> >>>> >>>> Or there will be too many false alerts. >>>> >>>> === Limitation === >>>> There are several limitations introduced just for subpage: >>>> - No compressed write support >>>> Read is no problem, but compression write path has more things >>>> left to >>>> be modified. >>>> Thus for current patchset, no matter what inode attribute or mount >>>> option is, no new compressed extent can be created for subpage case. >>>> >>>> - No inline extent will be created >>>> This is mostly due to the fact that filemap_fdatawrite_range() will >>>> trigger more write than the range specified. >>>> In fallocate calls, this behavior can make us to writeback which can >>>> be inlined, before we enlarge the isize, causing inline extent being >>>> created along with regular extents. >>>> >>>> - No support for RAID56 >>>> There are still too many hardcoded PAGE_SIZE in raid56 code. >>>> Considering it's already considered unsafe due to its write-hole >>>> problem, disabling RAID56 for subpage looks sane to me. >>>> >>>> - No defrag support for subpage >>>> The support for subpage defrag has already an initial version >>>> submitted to the mail list. >>>> Thus the correct support won't be included in this patchset. >>>> >>> >>> I am confused about what supports as of now? >> >> From my latest subpage branch, everything work as expected. >> >> Defrag support is in another patchset, which can be applied >> independently. >> >>> /sys/fs/btrfs/features/supported_sectorsizes >> >> [adam@arm-btrfs linux]$ uname -a >> Linux arm-btrfs 5.13.0-rc2-custom+ #5 SMP Tue Jun 1 16:11:41 CST 2021 >> aarch64 GNU/Linux >> [adam@arm-btrfs linux]$ getconf PAGESIZE >> 65536 >> [adam@arm-btrfs linux]$ cat /sys/fs/btrfs/features/supported_sectorsizes >> 4096 65536 >> >> It still shows 4k as support sectorsize. >> >> What's your branch/HEAD? And are you using 64K page size? >> > > I am on misc-next (which contains all these 30 patches). I don't see > subpages supported. Is there any patch I missed? misc-next doesn't have full support yet. It lacks: - Relocation support - Bio split support - Various subpage specific fixes Thus no subpage enabling patch. Thanks, Qu > > $ cat /sys/fs/btrfs/features/supported_sectorsizes > 65536 > > Thanks, Anand > >> Thanks, >> Qu >> >>> list just the pagesize. >>> >>> Thanks, Anand >>> >>> >>>> === Patchset structure === >>>> >>>> Patch 01~19: Make data write path to be subpage compatible >>>> Patch 20~21: Make data relocation path to be subpage compatible >>>> Patch 22~29: Various fixes for subpage corner cases >>>> Patch 30: Enable subpage data write >>>> >>>> === Changelog === >>>> v2: >>>> - Rebased to latest misc-next >>>> Now metadata write patches are removed from the series, as they are >>>> already merged into misc-next. >>>> >>>> - Added new Reviewed-by/Tested-by/Reported-by tags >>>> >>>> - Use separate endio functions to subpage metadata write path >>>> >>>> - Re-order the patches, to make refactors at the top of the series >>>> One refactor, the submit_extent_page() one, should benefit 4K page >>>> size more than 64K page size, thus it's worthy to be merged early >>>> >>>> - New bug fixes exposed by Ritesh Harjani on Power >>>> >>>> - Reject RAID56 completely >>>> Exposed by btrfs test group, which caused BUG_ON() for various >>>> sites. >>>> Considering RAID56 is already not considered safe, it's better to >>>> reject them completely for now. >>>> >>>> - Fix subpage scrub repair failure >>>> Caused by hardcoded PAGE_SIZE >>>> >>>> - Fix free space cache inode size >>>> Same cause as scrub repair failure >>>> >>>> v3: >>>> - Rebased to remove write path prepration patches >>>> >>>> - Properly enable btrfs defrag >>>> Previsouly, btrfs defrag is in fact just disabled. >>>> This makes tons of tests in btrfs/defrag to fail. >>>> >>>> - More bug fixes for rare race/crashes >>>> * Fix relocation false alert on csum mismatch >>>> * Fix relocation data corruption >>>> * Fix a rare case of false ASSERT() >>>> The fix already get merged into the prepration patches, thus no >>>> longer in this patchset though. >>>> Mostly reported by Ritesh from IBM. >>>> >>>> v4: >>>> - Disable subpage defrag completely >>>> As full page defrag can race with fsstress in btrfs/062, causing >>>> strange ordered extent bugs. >>>> The full subpage defrag will be submitted as an indepdent patchset. >>>> >>>> Qu Wenruo (30): >>>> btrfs: pass bytenr directly to __process_pages_contig() >>>> btrfs: refactor the page status update into process_one_page() >>>> btrfs: provide btrfs_page_clamp_*() helpers >>>> btrfs: only require sector size alignment for >>>> end_bio_extent_writepage() >>>> btrfs: make btrfs_dirty_pages() to be subpage compatible >>>> btrfs: make __process_pages_contig() to handle subpage >>>> dirty/error/writeback status >>>> btrfs: make end_bio_extent_writepage() to be subpage compatible >>>> btrfs: make process_one_page() to handle subpage locking >>>> btrfs: introduce helpers for subpage ordered status >>>> btrfs: make page Ordered bit to be subpage compatible >>>> btrfs: update locked page dirty/writeback/error bits in >>>> __process_pages_contig >>>> btrfs: prevent extent_clear_unlock_delalloc() to unlock page not >>>> locked by __process_pages_contig() >>>> btrfs: make btrfs_set_range_writeback() subpage compatible >>>> btrfs: make __extent_writepage_io() only submit dirty range for >>>> subpage >>>> btrfs: make btrfs_truncate_block() to be subpage compatible >>>> btrfs: make btrfs_page_mkwrite() to be subpage compatible >>>> btrfs: reflink: make copy_inline_to_page() to be subpage compatible >>>> btrfs: fix the filemap_range_has_page() call in >>>> btrfs_punch_hole_lock_range() >>>> btrfs: don't clear page extent mapped if we're not invalidating the >>>> full page >>>> btrfs: extract relocation page read and dirty part into its own >>>> function >>>> btrfs: make relocate_one_page() to handle subpage case >>>> btrfs: fix wild subpage writeback which does not have ordered >>>> extent. >>>> btrfs: disable inline extent creation for subpage >>>> btrfs: allow submit_extent_page() to do bio split for subpage >>>> btrfs: reject raid5/6 fs for subpage >>>> btrfs: fix a crash caused by race between prepare_pages() and >>>> btrfs_releasepage() >>>> btrfs: fix a use-after-free bug in writeback subpage helper >>>> btrfs: fix a subpage false alert for relocating partial preallocated >>>> data extents >>>> btrfs: fix a subpage relocation data corruption >>>> btrfs: allow read-write for 4K sectorsize on 64K page size systems >>>> >>>> fs/btrfs/ctree.h | 2 +- >>>> fs/btrfs/disk-io.c | 13 +- >>>> fs/btrfs/extent_io.c | 563 >>>> ++++++++++++++++++++++++++++------------ >>>> fs/btrfs/file.c | 32 ++- >>>> fs/btrfs/inode.c | 147 +++++++++-- >>>> fs/btrfs/ioctl.c | 6 + >>>> fs/btrfs/ordered-data.c | 5 +- >>>> fs/btrfs/reflink.c | 14 +- >>>> fs/btrfs/relocation.c | 287 ++++++++++++-------- >>>> fs/btrfs/subpage.c | 156 ++++++++++- >>>> fs/btrfs/subpage.h | 31 +++ >>>> fs/btrfs/super.c | 7 - >>>> fs/btrfs/sysfs.c | 5 + >>>> fs/btrfs/volumes.c | 8 + >>>> 14 files changed, 949 insertions(+), 327 deletions(-) >>>> >>>
On 8/6/21 5:50 pm, Qu Wenruo wrote: > > > On 2021/6/8 下午5:45, Anand Jain wrote: >> >> >> On 8/6/21 5:02 pm, Qu Wenruo wrote: >>> >>> >>> On 2021/6/8 下午4:23, Anand Jain wrote: >>>> On 31/5/21 4:50 pm, Qu Wenruo wrote: >>>>> This huge patchset can be fetched from github: >>>>> https://github.com/adam900710/linux/tree/subpage >>>>> >>>>> === Current stage === >>>>> The tests on x86 pass without new failure, and generic test group on >>>>> arm64 with 64K page size passes except known failure and defrag group. >>>>> >>>>> For btrfs test group, all pass except compression/raid56/defrag. >>>>> >>>>> For anyone who is interested in testing, please apply this patch for >>>>> btrfs-progs before testing. >>>>> https://patchwork.kernel.org/project/linux-btrfs/patch/20210420073036.243715-1-wqu@suse.com/ >>>>> >>>>> >>>>> >>>>> Or there will be too many false alerts. >>>>> >>>>> === Limitation === >>>>> There are several limitations introduced just for subpage: >>>>> - No compressed write support >>>>> Read is no problem, but compression write path has more things >>>>> left to >>>>> be modified. >>>>> Thus for current patchset, no matter what inode attribute or mount >>>>> option is, no new compressed extent can be created for subpage >>>>> case. >>>>> >>>>> - No inline extent will be created >>>>> This is mostly due to the fact that filemap_fdatawrite_range() will >>>>> trigger more write than the range specified. >>>>> In fallocate calls, this behavior can make us to writeback which >>>>> can >>>>> be inlined, before we enlarge the isize, causing inline extent >>>>> being >>>>> created along with regular extents. >>>>> >>>>> - No support for RAID56 >>>>> There are still too many hardcoded PAGE_SIZE in raid56 code. >>>>> Considering it's already considered unsafe due to its write-hole >>>>> problem, disabling RAID56 for subpage looks sane to me. >>>>> >>>>> - No defrag support for subpage >>>>> The support for subpage defrag has already an initial version >>>>> submitted to the mail list. >>>>> Thus the correct support won't be included in this patchset. >>>>> >>>> >>>> I am confused about what supports as of now? >>> >>> From my latest subpage branch, everything work as expected. >>> >>> Defrag support is in another patchset, which can be applied >>> independently. >>> >>>> /sys/fs/btrfs/features/supported_sectorsizes >>> >>> [adam@arm-btrfs linux]$ uname -a >>> Linux arm-btrfs 5.13.0-rc2-custom+ #5 SMP Tue Jun 1 16:11:41 CST 2021 >>> aarch64 GNU/Linux >>> [adam@arm-btrfs linux]$ getconf PAGESIZE >>> 65536 >>> [adam@arm-btrfs linux]$ cat /sys/fs/btrfs/features/supported_sectorsizes >>> 4096 65536 >>> >>> It still shows 4k as support sectorsize. >>> >>> What's your branch/HEAD? And are you using 64K page size? >>> >> >> I am on misc-next (which contains all these 30 patches). I don't see >> subpages supported. Is there any patch I missed? > > misc-next doesn't have full support yet. Thanks for clarifying. I wasted too much time figuring out what's wrong with my setup. It wasn't explicit in the cover letter. In that case, there is no regression in this set (plus with a bug fix patch as below). My Apologies to send this late: Tested-by: Anand Jain <anand.jain@oracle.com> [aarch64] My test setup config: Last commit on misc-next: ----- aecab60cef1a (HEAD -> misc-next) btrfs: fix embarrassing bugs in find_next_dirty_byte() 3f3a096a4d67 (origin/misc-next) btrfs: remove total_data_size variable in btrfs_batch_insert_items() ----- PLATFORM: Linux/aarch64 $ cpunr 32 $ cat /proc/meminfo | head -1 MemTotal: 16411904 kB $ getconf PAGESIZE 65536 >>> From my latest subpage branch, everything work as expected. Let me try to apply on the current misc-next. I am targeting to test this with sectorsize=4096 on a pagesize=64K. It will take some time as I need to get back to few other items as well. Thanks, Anand > > It lacks: > - Relocation support > - Bio split support > - Various subpage specific fixes > > Thus no subpage enabling patch. > > Thanks, > Qu >> >> $ cat /sys/fs/btrfs/features/supported_sectorsizes >> 65536 >> >> Thanks, Anand >> >>> Thanks, >>> Qu >>> >>>> list just the pagesize. >>>> >>>> Thanks, Anand >>>> >>>> >>>>> === Patchset structure === >>>>> >>>>> Patch 01~19: Make data write path to be subpage compatible >>>>> Patch 20~21: Make data relocation path to be subpage compatible >>>>> Patch 22~29: Various fixes for subpage corner cases >>>>> Patch 30: Enable subpage data write >>>>> >>>>> === Changelog === >>>>> v2: >>>>> - Rebased to latest misc-next >>>>> Now metadata write patches are removed from the series, as they are >>>>> already merged into misc-next. >>>>> >>>>> - Added new Reviewed-by/Tested-by/Reported-by tags >>>>> >>>>> - Use separate endio functions to subpage metadata write path >>>>> >>>>> - Re-order the patches, to make refactors at the top of the series >>>>> One refactor, the submit_extent_page() one, should benefit 4K page >>>>> size more than 64K page size, thus it's worthy to be merged early >>>>> >>>>> - New bug fixes exposed by Ritesh Harjani on Power >>>>> >>>>> - Reject RAID56 completely >>>>> Exposed by btrfs test group, which caused BUG_ON() for various >>>>> sites. >>>>> Considering RAID56 is already not considered safe, it's better to >>>>> reject them completely for now. >>>>> >>>>> - Fix subpage scrub repair failure >>>>> Caused by hardcoded PAGE_SIZE >>>>> >>>>> - Fix free space cache inode size >>>>> Same cause as scrub repair failure >>>>> >>>>> v3: >>>>> - Rebased to remove write path prepration patches >>>>> >>>>> - Properly enable btrfs defrag >>>>> Previsouly, btrfs defrag is in fact just disabled. >>>>> This makes tons of tests in btrfs/defrag to fail. >>>>> >>>>> - More bug fixes for rare race/crashes >>>>> * Fix relocation false alert on csum mismatch >>>>> * Fix relocation data corruption >>>>> * Fix a rare case of false ASSERT() >>>>> The fix already get merged into the prepration patches, thus no >>>>> longer in this patchset though. >>>>> Mostly reported by Ritesh from IBM. >>>>> >>>>> v4: >>>>> - Disable subpage defrag completely >>>>> As full page defrag can race with fsstress in btrfs/062, causing >>>>> strange ordered extent bugs. >>>>> The full subpage defrag will be submitted as an indepdent patchset. >>>>> >>>>> Qu Wenruo (30): >>>>> btrfs: pass bytenr directly to __process_pages_contig() >>>>> btrfs: refactor the page status update into process_one_page() >>>>> btrfs: provide btrfs_page_clamp_*() helpers >>>>> btrfs: only require sector size alignment for >>>>> end_bio_extent_writepage() >>>>> btrfs: make btrfs_dirty_pages() to be subpage compatible >>>>> btrfs: make __process_pages_contig() to handle subpage >>>>> dirty/error/writeback status >>>>> btrfs: make end_bio_extent_writepage() to be subpage compatible >>>>> btrfs: make process_one_page() to handle subpage locking >>>>> btrfs: introduce helpers for subpage ordered status >>>>> btrfs: make page Ordered bit to be subpage compatible >>>>> btrfs: update locked page dirty/writeback/error bits in >>>>> __process_pages_contig >>>>> btrfs: prevent extent_clear_unlock_delalloc() to unlock page not >>>>> locked by __process_pages_contig() >>>>> btrfs: make btrfs_set_range_writeback() subpage compatible >>>>> btrfs: make __extent_writepage_io() only submit dirty range for >>>>> subpage >>>>> btrfs: make btrfs_truncate_block() to be subpage compatible >>>>> btrfs: make btrfs_page_mkwrite() to be subpage compatible >>>>> btrfs: reflink: make copy_inline_to_page() to be subpage compatible >>>>> btrfs: fix the filemap_range_has_page() call in >>>>> btrfs_punch_hole_lock_range() >>>>> btrfs: don't clear page extent mapped if we're not invalidating the >>>>> full page >>>>> btrfs: extract relocation page read and dirty part into its own >>>>> function >>>>> btrfs: make relocate_one_page() to handle subpage case >>>>> btrfs: fix wild subpage writeback which does not have ordered >>>>> extent. >>>>> btrfs: disable inline extent creation for subpage >>>>> btrfs: allow submit_extent_page() to do bio split for subpage >>>>> btrfs: reject raid5/6 fs for subpage >>>>> btrfs: fix a crash caused by race between prepare_pages() and >>>>> btrfs_releasepage() >>>>> btrfs: fix a use-after-free bug in writeback subpage helper >>>>> btrfs: fix a subpage false alert for relocating partial >>>>> preallocated >>>>> data extents >>>>> btrfs: fix a subpage relocation data corruption >>>>> btrfs: allow read-write for 4K sectorsize on 64K page size systems >>>>> >>>>> fs/btrfs/ctree.h | 2 +- >>>>> fs/btrfs/disk-io.c | 13 +- >>>>> fs/btrfs/extent_io.c | 563 >>>>> ++++++++++++++++++++++++++++------------ >>>>> fs/btrfs/file.c | 32 ++- >>>>> fs/btrfs/inode.c | 147 +++++++++-- >>>>> fs/btrfs/ioctl.c | 6 + >>>>> fs/btrfs/ordered-data.c | 5 +- >>>>> fs/btrfs/reflink.c | 14 +- >>>>> fs/btrfs/relocation.c | 287 ++++++++++++-------- >>>>> fs/btrfs/subpage.c | 156 ++++++++++- >>>>> fs/btrfs/subpage.h | 31 +++ >>>>> fs/btrfs/super.c | 7 - >>>>> fs/btrfs/sysfs.c | 5 + >>>>> fs/btrfs/volumes.c | 8 + >>>>> 14 files changed, 949 insertions(+), 327 deletions(-) >>>>> >>>>
On Tue, Jun 08, 2021 at 07:11:03PM +0800, Anand Jain wrote: > > Thanks for clarifying. I wasted too much time figuring out what's wrong > with my setup. It wasn't explicit in the cover letter. > In that case, there is no regression in this set (plus with a bug fix > patch as below). My Apologies to send this late: > > Tested-by: Anand Jain <anand.jain@oracle.com> [aarch64] Thanks for testing, I've updated the patches with the tag.
This huge patchset can be fetched from github: https://github.com/adam900710/linux/tree/subpage === Current stage === The tests on x86 pass without new failure, and generic test group on arm64 with 64K page size passes except known failure and defrag group. For btrfs test group, all pass except compression/raid56/defrag. For anyone who is interested in testing, please apply this patch for btrfs-progs before testing. https://patchwork.kernel.org/project/linux-btrfs/patch/20210420073036.243715-1-wqu@suse.com/ Or there will be too many false alerts. === Limitation === There are several limitations introduced just for subpage: - No compressed write support Read is no problem, but compression write path has more things left to be modified. Thus for current patchset, no matter what inode attribute or mount option is, no new compressed extent can be created for subpage case. - No inline extent will be created This is mostly due to the fact that filemap_fdatawrite_range() will trigger more write than the range specified. In fallocate calls, this behavior can make us to writeback which can be inlined, before we enlarge the isize, causing inline extent being created along with regular extents. - No support for RAID56 There are still too many hardcoded PAGE_SIZE in raid56 code. Considering it's already considered unsafe due to its write-hole problem, disabling RAID56 for subpage looks sane to me. - No defrag support for subpage The support for subpage defrag has already an initial version submitted to the mail list. Thus the correct support won't be included in this patchset. === Patchset structure === Patch 01~19: Make data write path to be subpage compatible Patch 20~21: Make data relocation path to be subpage compatible Patch 22~29: Various fixes for subpage corner cases Patch 30: Enable subpage data write === Changelog === v2: - Rebased to latest misc-next Now metadata write patches are removed from the series, as they are already merged into misc-next. - Added new Reviewed-by/Tested-by/Reported-by tags - Use separate endio functions to subpage metadata write path - Re-order the patches, to make refactors at the top of the series One refactor, the submit_extent_page() one, should benefit 4K page size more than 64K page size, thus it's worthy to be merged early - New bug fixes exposed by Ritesh Harjani on Power - Reject RAID56 completely Exposed by btrfs test group, which caused BUG_ON() for various sites. Considering RAID56 is already not considered safe, it's better to reject them completely for now. - Fix subpage scrub repair failure Caused by hardcoded PAGE_SIZE - Fix free space cache inode size Same cause as scrub repair failure v3: - Rebased to remove write path prepration patches - Properly enable btrfs defrag Previsouly, btrfs defrag is in fact just disabled. This makes tons of tests in btrfs/defrag to fail. - More bug fixes for rare race/crashes * Fix relocation false alert on csum mismatch * Fix relocation data corruption * Fix a rare case of false ASSERT() The fix already get merged into the prepration patches, thus no longer in this patchset though. Mostly reported by Ritesh from IBM. v4: - Disable subpage defrag completely As full page defrag can race with fsstress in btrfs/062, causing strange ordered extent bugs. The full subpage defrag will be submitted as an indepdent patchset. Qu Wenruo (30): btrfs: pass bytenr directly to __process_pages_contig() btrfs: refactor the page status update into process_one_page() btrfs: provide btrfs_page_clamp_*() helpers btrfs: only require sector size alignment for end_bio_extent_writepage() btrfs: make btrfs_dirty_pages() to be subpage compatible btrfs: make __process_pages_contig() to handle subpage dirty/error/writeback status btrfs: make end_bio_extent_writepage() to be subpage compatible btrfs: make process_one_page() to handle subpage locking btrfs: introduce helpers for subpage ordered status btrfs: make page Ordered bit to be subpage compatible btrfs: update locked page dirty/writeback/error bits in __process_pages_contig btrfs: prevent extent_clear_unlock_delalloc() to unlock page not locked by __process_pages_contig() btrfs: make btrfs_set_range_writeback() subpage compatible btrfs: make __extent_writepage_io() only submit dirty range for subpage btrfs: make btrfs_truncate_block() to be subpage compatible btrfs: make btrfs_page_mkwrite() to be subpage compatible btrfs: reflink: make copy_inline_to_page() to be subpage compatible btrfs: fix the filemap_range_has_page() call in btrfs_punch_hole_lock_range() btrfs: don't clear page extent mapped if we're not invalidating the full page btrfs: extract relocation page read and dirty part into its own function btrfs: make relocate_one_page() to handle subpage case btrfs: fix wild subpage writeback which does not have ordered extent. btrfs: disable inline extent creation for subpage btrfs: allow submit_extent_page() to do bio split for subpage btrfs: reject raid5/6 fs for subpage btrfs: fix a crash caused by race between prepare_pages() and btrfs_releasepage() btrfs: fix a use-after-free bug in writeback subpage helper btrfs: fix a subpage false alert for relocating partial preallocated data extents btrfs: fix a subpage relocation data corruption btrfs: allow read-write for 4K sectorsize on 64K page size systems fs/btrfs/ctree.h | 2 +- fs/btrfs/disk-io.c | 13 +- fs/btrfs/extent_io.c | 563 ++++++++++++++++++++++++++++------------ fs/btrfs/file.c | 32 ++- fs/btrfs/inode.c | 147 +++++++++-- fs/btrfs/ioctl.c | 6 + fs/btrfs/ordered-data.c | 5 +- fs/btrfs/reflink.c | 14 +- fs/btrfs/relocation.c | 287 ++++++++++++-------- fs/btrfs/subpage.c | 156 ++++++++++- fs/btrfs/subpage.h | 31 +++ fs/btrfs/super.c | 7 - fs/btrfs/sysfs.c | 5 + fs/btrfs/volumes.c | 8 + 14 files changed, 949 insertions(+), 327 deletions(-)