| Message ID | 20191203064254.22683-1-wqu@suse.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | btrfs: Make balance cancelling response faster | expand |
On Tue, Dec 03, 2019 at 02:42:50PM +0800, Qu Wenruo wrote: > [PROBLEM] > There are quite some users reporting that 'btrfs balance cancel' slow to > cancel current running balance, or even doesn't work for certain dead > balance loop. > > With the following script showing how long it takes to fully stop a > balance: > #!/bin/bash > dev=/dev/test/test > mnt=/mnt/btrfs > > umount $mnt &> /dev/null > umount $dev &> /dev/null > > mkfs.btrfs -f $dev > mount $dev -o nospace_cache $mnt > > dd if=/dev/zero bs=1M of=$mnt/large & > dd_pid=$! > > sleep 3 > kill -KILL $dd_pid > sync > > btrfs balance start --bg --full $mnt & > sleep 1 > > echo "cancel request" >> /dev/kmsg > time btrfs balance cancel $mnt > umount $mnt > > It takes around 7~10s to cancel the running balance in my test > environment. > > [CAUSE] > Btrfs uses btrfs_fs_info::balance_cancel_req to record how many cancel > request are queued. > However that cancelling request is only checked after relocating a block > group. Yes that's the reason why it takes so long to cancel. Adding more cancellation points is fine, but I don't know what exactly happens when the block group relocation is not finished. There's code to merge the reloc inode and commit that, but that's only a high-level view of the thing.
On Tue, Dec 03, 2019 at 02:42:50PM +0800, Qu Wenruo wrote: > [PROBLEM] > There are quite some users reporting that 'btrfs balance cancel' slow to > cancel current running balance, or even doesn't work for certain dead > balance loop. > > With the following script showing how long it takes to fully stop a > balance: > #!/bin/bash > dev=/dev/test/test > mnt=/mnt/btrfs > > umount $mnt &> /dev/null > umount $dev &> /dev/null > > mkfs.btrfs -f $dev > mount $dev -o nospace_cache $mnt > > dd if=/dev/zero bs=1M of=$mnt/large & > dd_pid=$! > > sleep 3 > kill -KILL $dd_pid > sync > > btrfs balance start --bg --full $mnt & > sleep 1 > > echo "cancel request" >> /dev/kmsg > time btrfs balance cancel $mnt > umount $mnt > > It takes around 7~10s to cancel the running balance in my test > environment. > > [CAUSE] > Btrfs uses btrfs_fs_info::balance_cancel_req to record how many cancel > request are queued. > However that cancelling request is only checked after relocating a block > group. > > That behavior is far from optimal to provide a faster cancelling. > > [FIX] > This patchset will add more cancelling check points, to make cancelling > faster. Nice! I look forward to using this in the future! Does this cover device delete/resize as well? I think there needs to be a check added for fatal signals for those to work, as they don't respond to balance cancel. > And also, introduce a new error injection points to cover these newly > introduced and future check points. > > Qu Wenruo (4): > btrfs: relocation: Introduce error injection points for cancelling > balance > btrfs: relocation: Check cancel request after each data page read > btrfs: relocation: Check cancel request after each extent found > btrfs: relocation: Work around dead relocation stage loop > > fs/btrfs/ctree.h | 1 + > fs/btrfs/relocation.c | 23 +++++++++++++++++++++++ > fs/btrfs/volumes.c | 2 +- > 3 files changed, 25 insertions(+), 1 deletion(-) > > -- > 2.24.0 >
On 2019/12/5 上午10:58, Zygo Blaxell wrote: > On Tue, Dec 03, 2019 at 02:42:50PM +0800, Qu Wenruo wrote: >> [PROBLEM] >> There are quite some users reporting that 'btrfs balance cancel' slow to >> cancel current running balance, or even doesn't work for certain dead >> balance loop. >> >> With the following script showing how long it takes to fully stop a >> balance: >> #!/bin/bash >> dev=/dev/test/test >> mnt=/mnt/btrfs >> >> umount $mnt &> /dev/null >> umount $dev &> /dev/null >> >> mkfs.btrfs -f $dev >> mount $dev -o nospace_cache $mnt >> >> dd if=/dev/zero bs=1M of=$mnt/large & >> dd_pid=$! >> >> sleep 3 >> kill -KILL $dd_pid >> sync >> >> btrfs balance start --bg --full $mnt & >> sleep 1 >> >> echo "cancel request" >> /dev/kmsg >> time btrfs balance cancel $mnt >> umount $mnt >> >> It takes around 7~10s to cancel the running balance in my test >> environment. >> >> [CAUSE] >> Btrfs uses btrfs_fs_info::balance_cancel_req to record how many cancel >> request are queued. >> However that cancelling request is only checked after relocating a block >> group. >> >> That behavior is far from optimal to provide a faster cancelling. >> >> [FIX] >> This patchset will add more cancelling check points, to make cancelling >> faster. > > Nice! I look forward to using this in the future! > > Does this cover device delete/resize as well? Shrink also takes use of balance, so I see no reason why it won't work on such use cases. > I think there needs to be > a check added for fatal signals for those to work, as they don't respond > to balance cancel. That's a good extra idea. Since we have that wrapper, it would be easier to add in the future. Thanks, Qu > >> And also, introduce a new error injection points to cover these newly >> introduced and future check points. >> >> Qu Wenruo (4): >> btrfs: relocation: Introduce error injection points for cancelling >> balance >> btrfs: relocation: Check cancel request after each data page read >> btrfs: relocation: Check cancel request after each extent found >> btrfs: relocation: Work around dead relocation stage loop >> >> fs/btrfs/ctree.h | 1 + >> fs/btrfs/relocation.c | 23 +++++++++++++++++++++++ >> fs/btrfs/volumes.c | 2 +- >> 3 files changed, 25 insertions(+), 1 deletion(-) >> >> -- >> 2.24.0 >>
On 2019/12/5 上午12:39, David Sterba wrote: > On Tue, Dec 03, 2019 at 02:42:50PM +0800, Qu Wenruo wrote: >> [PROBLEM] >> There are quite some users reporting that 'btrfs balance cancel' slow to >> cancel current running balance, or even doesn't work for certain dead >> balance loop. >> >> With the following script showing how long it takes to fully stop a >> balance: >> #!/bin/bash >> dev=/dev/test/test >> mnt=/mnt/btrfs >> >> umount $mnt &> /dev/null >> umount $dev &> /dev/null >> >> mkfs.btrfs -f $dev >> mount $dev -o nospace_cache $mnt >> >> dd if=/dev/zero bs=1M of=$mnt/large & >> dd_pid=$! >> >> sleep 3 >> kill -KILL $dd_pid >> sync >> >> btrfs balance start --bg --full $mnt & >> sleep 1 >> >> echo "cancel request" >> /dev/kmsg >> time btrfs balance cancel $mnt >> umount $mnt >> >> It takes around 7~10s to cancel the running balance in my test >> environment. >> >> [CAUSE] >> Btrfs uses btrfs_fs_info::balance_cancel_req to record how many cancel >> request are queued. >> However that cancelling request is only checked after relocating a block >> group. > > Yes that's the reason why it takes so long to cancel. Adding more > cancellation points is fine, but I don't know what exactly happens when > the block group relocation is not finished. There's code to merge the > reloc inode and commit that, but that's only a high-level view of the > thing. When cancelled, we still merge the reloc roots with its source (if possible, as we still do the check for last_snapshot generation). That means, if balance is canceled halfway, we still merge what is relocated. Then do the regular cleanup (cleanup the reloc tree). I see no problem doing faster canceling here. Or do you have any extra concern? Thanks, Qu
On 2020/2/11 下午1:21, Qu Wenruo wrote: > > > On 2019/12/5 上午12:39, David Sterba wrote: >> On Tue, Dec 03, 2019 at 02:42:50PM +0800, Qu Wenruo wrote: >>> [PROBLEM] >>> There are quite some users reporting that 'btrfs balance cancel' slow to >>> cancel current running balance, or even doesn't work for certain dead >>> balance loop. >>> >>> With the following script showing how long it takes to fully stop a >>> balance: >>> #!/bin/bash >>> dev=/dev/test/test >>> mnt=/mnt/btrfs >>> >>> umount $mnt &> /dev/null >>> umount $dev &> /dev/null >>> >>> mkfs.btrfs -f $dev >>> mount $dev -o nospace_cache $mnt >>> >>> dd if=/dev/zero bs=1M of=$mnt/large & >>> dd_pid=$! >>> >>> sleep 3 >>> kill -KILL $dd_pid >>> sync >>> >>> btrfs balance start --bg --full $mnt & >>> sleep 1 >>> >>> echo "cancel request" >> /dev/kmsg >>> time btrfs balance cancel $mnt >>> umount $mnt >>> >>> It takes around 7~10s to cancel the running balance in my test >>> environment. >>> >>> [CAUSE] >>> Btrfs uses btrfs_fs_info::balance_cancel_req to record how many cancel >>> request are queued. >>> However that cancelling request is only checked after relocating a block >>> group. >> >> Yes that's the reason why it takes so long to cancel. Adding more >> cancellation points is fine, but I don't know what exactly happens when >> the block group relocation is not finished. There's code to merge the >> reloc inode and commit that, but that's only a high-level view of the >> thing. > > When cancelled, we still merge the reloc roots with its source (if > possible, as we still do the check for last_snapshot generation). > > That means, if balance is canceled halfway, we still merge what is > relocated. Then do the regular cleanup (cleanup the reloc tree). My bad, that's not the case. But it doesn't matter anyway. Since we error out by setting @err to -ECANCELD, we won't go through the merge part. We just mark all related reloc roots as orphan, then go through the cleanup path without doing the merge. That's the existing error handling code, and we should have experienced quite some of them for the random data reloc tree csum mismatch bug. At least the error handling part looks pretty solid. I'll add this explanation into the cover letter of next version. Thanks, Qu > > I see no problem doing faster canceling here. > > Or do you have any extra concern? > > Thanks, > Qu >
[PROBLEM] There are quite some users reporting that 'btrfs balance cancel' slow to cancel current running balance, or even doesn't work for certain dead balance loop. With the following script showing how long it takes to fully stop a balance: #!/bin/bash dev=/dev/test/test mnt=/mnt/btrfs umount $mnt &> /dev/null umount $dev &> /dev/null mkfs.btrfs -f $dev mount $dev -o nospace_cache $mnt dd if=/dev/zero bs=1M of=$mnt/large & dd_pid=$! sleep 3 kill -KILL $dd_pid sync btrfs balance start --bg --full $mnt & sleep 1 echo "cancel request" >> /dev/kmsg time btrfs balance cancel $mnt umount $mnt It takes around 7~10s to cancel the running balance in my test environment. [CAUSE] Btrfs uses btrfs_fs_info::balance_cancel_req to record how many cancel request are queued. However that cancelling request is only checked after relocating a block group. That behavior is far from optimal to provide a faster cancelling. [FIX] This patchset will add more cancelling check points, to make cancelling faster. And also, introduce a new error injection points to cover these newly introduced and future check points. Qu Wenruo (4): btrfs: relocation: Introduce error injection points for cancelling balance btrfs: relocation: Check cancel request after each data page read btrfs: relocation: Check cancel request after each extent found btrfs: relocation: Work around dead relocation stage loop fs/btrfs/ctree.h | 1 + fs/btrfs/relocation.c | 23 +++++++++++++++++++++++ fs/btrfs/volumes.c | 2 +- 3 files changed, 25 insertions(+), 1 deletion(-)