| Message ID | cover.1536331604.git.dsterba@suse.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Cleanup dev-replace locking | expand |
On Fri, Sep 07, 2018 at 04:54:59PM +0200, David Sterba wrote: > The series peels off the custom locking that's used for dev-replace and > uses read-write semaphore in the end. > > I've mainly focused on correctness and haven't measured the performance > effects. There should be none as the blocking and waiting was merely > open coding what the rw semaphore does, but without the fairness. The > overall number of locks taken is low, there's a lots of IO in between so > even if new scheme is slower, I don't expect any dramatic change. Mixed results. The btrfs/011 is a test that usually takes long time (around 1000 sec on my test box) and I have a long list of run times to compare. There this patchset does not show any problems. However on a VM with 8 CPUs, this goes from similar times (1000 sec) to several hours, without apparent reason. The is other low activity on the system, but so this is with testing other patchsets and the times are not that bad. So, I'm going to merge the first part of the patchset that does not switch the locking primitives to the semaphores, more analysis needed.
On 09/27/2018 07:06 PM, David Sterba wrote: > On Fri, Sep 07, 2018 at 04:54:59PM +0200, David Sterba wrote: >> The series peels off the custom locking that's used for dev-replace and >> uses read-write semaphore in the end. >> >> I've mainly focused on correctness and haven't measured the performance >> effects. There should be none as the blocking and waiting was merely >> open coding what the rw semaphore does, but without the fairness. The >> overall number of locks taken is low, there's a lots of IO in between so >> even if new scheme is slower, I don't expect any dramatic change. > > Mixed results. The btrfs/011 is a test that usually takes long time > (around 1000 sec on my test box) and I have a long list of run times to > compare. There this patchset does not show any problems. However on a VM > with 8 CPUs, this goes from similar times (1000 sec) to several hours, > without apparent reason. The is other low activity on the system, but so > this is with testing other patchsets and the times are not that bad. > > So, I'm going to merge the first part of the patchset that does not > switch the locking primitives to the semaphores, more analysis needed. > I have reviewed/tested all in this set and it looks good to me. So Reviewed-by: Anand Jain <anand.jain@oracle.com> I do observer btrfs/011 taking longer time to complete (200sec more) and Null pointer dereference even without this patch set, tracing back lead to conclusion, that v4.17 is good. v4.18-rc1 is bad. And in particular these commit ids.. 7a932516f55c Merge tag 'vfs-timespec64' of git://git.kernel.org/pub/scm/linux/kernel/git/arnd/playground e7655d2b2546 Merge tag 'for-4.18-part2-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux 15eefe2a99b2 Merge branch 'vfs_timespec64' of https://github.com/deepa-hub/vfs into vfs-timespec64 6396bb221514 treewide: kzalloc() -> kcalloc()
On Thu, Oct 04, 2018 at 09:06:01AM +0800, Anand Jain wrote: > > > On 09/27/2018 07:06 PM, David Sterba wrote: > > On Fri, Sep 07, 2018 at 04:54:59PM +0200, David Sterba wrote: > >> The series peels off the custom locking that's used for dev-replace and > >> uses read-write semaphore in the end. > >> > >> I've mainly focused on correctness and haven't measured the performance > >> effects. There should be none as the blocking and waiting was merely > >> open coding what the rw semaphore does, but without the fairness. The > >> overall number of locks taken is low, there's a lots of IO in between so > >> even if new scheme is slower, I don't expect any dramatic change. > > > > Mixed results. The btrfs/011 is a test that usually takes long time > > (around 1000 sec on my test box) and I have a long list of run times to > > compare. There this patchset does not show any problems. However on a VM > > with 8 CPUs, this goes from similar times (1000 sec) to several hours, > > without apparent reason. The is other low activity on the system, but so > > this is with testing other patchsets and the times are not that bad. > > > > So, I'm going to merge the first part of the patchset that does not > > switch the locking primitives to the semaphores, more analysis needed. > > > > I have reviewed/tested all in this set and it looks good to me. > > So Reviewed-by: Anand Jain <anand.jain@oracle.com> Thanks for testing, I'll add the tag to the patches. > I do observer btrfs/011 taking longer time to complete (200sec more) and > Null pointer dereference even without this patch set, tracing back lead > to conclusion, that The run times of 011 are quite wild and sometimes drop to 500 secs and go back to 1000+, on the same devices and basically same kernel. But replacing the locking would need further analysis as it's possible that the reader/writer fairness of the rwsem is acually not helping in the replace case that as currently implemented favors readers. I'm rarely observing a crash in 011, stacktrace attached, please let me know if it's the same as the one you mention. [ 738.344228] BTRFS info (device vdb): dev_replace from /dev/vdb (devid 1) to /dev/vdc started [ 739.585174] WARNING: CPU: 3 PID: 20788 at fs/btrfs/scrub.c:620 scrub_setup_ctx.isra.19+0x20c/0x210 [btrfs] [ 739.591953] Modules linked in: btrfs libcrc32c xor zstd_decompress zstd_compress xxhash raid6_pq loop [ 739.594926] CPU: 3 PID: 20788 Comm: btrfs Not tainted 4.19.0-rc5-default+ #271 [ 739.596350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.0.0-prebuilt.qemu-project.org 04/01/2014 [ 739.598304] RIP: 0010:scrub_setup_ctx.isra.19+0x20c/0x210 [btrfs] [ 739.603855] RSP: 0018:ffff914d076bbc30 EFLAGS: 00010246 [ 739.605120] RAX: 0000000000000001 RBX: 0000000000000040 RCX: 0000000000182000 [ 739.606206] RDX: ffffffffc04aa000 RSI: 0000000000000000 RDI: 0000000000000038 [ 739.607119] RBP: ffffffffc03c9640 R08: ffffffffc0328000 R09: ffffffff9a00abc0 [ 739.608025] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 [ 739.608950] R13: ffff9049f94d8000 R14: ffff9049fbc56800 R15: ffff9049f94daf30 [ 739.610321] FS: 00007f72e9a328c0(0000) GS:ffff9049fd200000(0000) knlGS:0000000000000000 [ 739.612299] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 739.613602] CR2: 00005645d94eb4f0 CR3: 0000000046de0000 CR4: 00000000000006e0 [ 739.615244] Call Trace: [ 739.615986] btrfs_scrub_dev+0x14c/0x5a0 [btrfs] [ 739.617122] ? start_transaction+0xa1/0x470 [btrfs] [ 739.618335] btrfs_dev_replace_start.cold.24+0x9a/0x144 [btrfs] [ 739.619680] btrfs_dev_replace_by_ioctl+0x35/0x60 [btrfs] [ 739.621003] btrfs_ioctl+0x2a94/0x31d0 [btrfs] [ 739.622123] ? filemap_map_pages+0x279/0x3d0 [ 739.623199] ? _raw_spin_unlock+0x24/0x40 [ 739.624225] ? __handle_mm_fault+0xb52/0xcb0 [ 739.625263] ? do_vfs_ioctl+0xa2/0x6b0 [ 739.626247] do_vfs_ioctl+0xa2/0x6b0 [ 739.627184] ksys_ioctl+0x3a/0x70 [ 739.628058] __x64_sys_ioctl+0x16/0x20 [ 739.628953] do_syscall_64+0x5a/0x1a0 [ 739.629893] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 739.631083] RIP: 0033:0x7f72e8efeaa7 [ 739.636109] RSP: 002b:00007ffd6d3695a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 739.637900] RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007f72e8efeaa7 [ 739.639508] RDX: 00007ffd6d3699e0 RSI: 00000000ca289435 RDI: 0000000000000003 [ 739.641116] RBP: 0000000000000000 R08: 00007f72e9a328c0 R09: 00007f72e93e72a0 [ 739.642780] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffd6d36c278 [ 739.644499] R13: 0000000000000000 R14: 0000000000000004 R15: 00005645d95b72a0 [ 739.646082] irq event stamp: 0 [ 739.646968] hardirqs last enabled at (0): [<0000000000000000>] (null) [ 739.648878] hardirqs last disabled at (0): [<ffffffff9905b9cd>] copy_process.part.66+0x84d/0x1db0 [ 739.651061] softirqs last enabled at (0): [<ffffffff9905b9cd>] copy_process.part.66+0x84d/0x1db0 [ 739.653042] softirqs last disabled at (0): [<0000000000000000>] (null) [ 739.654965] ---[ end trace 9e2151540055738e ]--- [ 740.471443] BTRFS info (device vdb): dev_replace from /dev/vdb (devid 1) to /dev/vdc canceled [ 740.614161] BUG: unable to handle kernel NULL pointer dereference at 00000000000000b0 [ 740.617332] PGD 0 P4D 0 [ 740.618575] Oops: 0000 [#1] PREEMPT SMP [ 740.620664] CPU: 0 PID: 12214 Comm: kworker/u16:4 Tainted: G W 4.19.0-rc5-default+ #271 [ 740.626680] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.0.0-prebuilt.qemu-project.org 04/01/2014 [ 740.629983] Workqueue: btrfs-scrub btrfs_scrub_helper [btrfs] [ 740.631239] RIP: 0010:scrub_write_page_to_dev_replace+0x86/0x2d0 [btrfs] [ 740.636572] RSP: 0018:ffff914d0b0fbd28 EFLAGS: 00010206 [ 740.637711] RAX: ffff9049931b0800 RBX: ffff9049e3103400 RCX: 0000000000000000 [ 740.639271] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9049931b0800 [ 740.640820] RBP: ffff9049fbc56800 R08: ffff9049931b0780 R09: ffff9049e3103400 [ 740.642359] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9049e30dbf00 [ 740.643876] R13: ffff9049fbc56ae8 R14: ffffffffc03c5ce0 R15: ffff9049c6d82a00 [ 740.645378] FS: 0000000000000000(0000) GS:ffff9049fcc00000(0000) knlGS:0000000000000000 [ 740.647291] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 740.648514] CR2: 00000000000000b0 CR3: 0000000038011000 CR4: 00000000000006f0 [ 740.650067] Call Trace: [ 740.650709] scrub_write_block_to_dev_replace+0x3b/0x60 [btrfs] [ 740.652083] scrub_bio_end_io_worker+0xee/0x500 [btrfs] [ 740.653241] normal_work_helper+0xcd/0x530 [btrfs] [ 740.654368] process_one_work+0x246/0x5c0 [ 740.655321] worker_thread+0x3c/0x390 [ 740.656233] ? rescuer_thread+0x360/0x360 [ 740.657271] kthread+0x116/0x130 [ 740.658138] ? kthread_create_worker_on_cpu+0x70/0x70 [ 740.659353] ret_from_fork+0x24/0x30 [ 740.660276] Modules linked in: btrfs libcrc32c xor zstd_decompress zstd_compress xxhash raid6_pq loop [ 740.662453] CR2: 00000000000000b0 [ 740.663402] ---[ end trace 9e2151540055738f ]--- [ 740.664622] RIP: 0010:scrub_write_page_to_dev_replace+0x86/0x2d0 [btrfs] [ 740.670545] RSP: 0018:ffff914d0b0fbd28 EFLAGS: 00010206 [ 740.671856] RAX: ffff9049931b0800 RBX: ffff9049e3103400 RCX: 0000000000000000 [ 740.673523] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9049931b0800 [ 740.675211] RBP: ffff9049fbc56800 R08: ffff9049931b0780 R09: ffff9049e3103400 [ 740.676876] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9049e30dbf00 [ 740.678582] R13: ffff9049fbc56ae8 R14: ffffffffc03c5ce0 R15: ffff9049c6d82a00 [ 740.680092] FS: 0000000000000000(0000) GS:ffff9049fcc00000(0000) knlGS:0000000000000000 [ 740.682002] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 740.683264] CR2: 00000000000000b0 CR3: 0000000038011000 CR4: 00000000000006f0 [ 740.684623] BUG: sleeping function called from invalid context at include/linux/cgroup-defs.h:711 [ 740.686144] in_atomic(): 0, irqs_disabled(): 1, pid: 12214, name: kworker/u16:4 [ 740.687471] INFO: lockdep is turned off. [ 740.688446] irq event stamp: 755514 [ 740.689212] hardirqs last enabled at (755513): [<ffffffff9960adf9>] _raw_spin_unlock_irq+0x29/0x50 [ 740.690933] hardirqs last disabled at (755514): [<ffffffff9960283e>] __schedule+0xae/0xaf0 [ 740.692750] softirqs last enabled at (755460): [<ffffffff99a0034d>] __do_softirq+0x34d/0x442 [ 740.694759] softirqs last disabled at (755439): [<ffffffff990648ba>] irq_exit+0xca/0xd0 [ 740.696621] CPU: 0 PID: 12214 Comm: kworker/u16:4 Tainted: G D W 4.19.0-rc5-default+ #271 [ 740.698340] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.0.0-prebuilt.qemu-project.org 04/01/2014 [ 740.700854] Workqueue: btrfs-scrub btrfs_scrub_helper [btrfs] [ 740.702106] Call Trace: [ 740.702695] dump_stack+0x85/0xc0 [ 740.703527] ___might_sleep.cold.79+0xac/0xbe [ 740.704539] exit_signals+0x30/0x130 [ 740.705425] do_exit+0xac/0xb90 [ 740.706178] ? rescuer_thread+0x360/0x360 [ 740.707018] ? kthread+0x116/0x130 [ 740.707748] rewind_stack_do_exit+0x17/0x20
>> I do observer btrfs/011 taking longer time to complete (200sec more) and >> Null pointer dereference even without this patch set, >> tracing back lead >> to conclusion, that Ah. My trace back was for the circular lock dependency warning which I reported here: Subject: btrfs/011 possible circular locking dependency detected. Which is just a warning. If possible we could fix it. But dereference is something we should try harder. Null deference doesn't happen consistently here, I couldn't capture the stack trace. > I'm rarely observing a crash in 011, stacktrace attached, Yep rarely. Thanks for the stacktrace. Per the stack trace.. replace started.. [1] [ 738.344228] BTRFS info (device vdb): dev_replace from /dev/vdb (devid 1) to /dev/vdc started [ 739.585174] WARNING: CPU: 3 PID: 20788 at fs/btrfs/scrub.c:620 scrub_setup_ctx.isra.19+0x20c/0x210 [btrfs] The replace was canceled 'before' (as below) the scrub_setup_ctx() was able to init with the replace target missing warning. ------------------- .. scrub_setup_ctx(..) 573 { :: 619 if (is_dev_replace) { 620 WARN_ON(!fs_info->dev_replace.tgtdev); :: ------------------- [2] [ 740.471443] BTRFS info (device vdb): dev_replace from /dev/vdb (devid 1) to /dev/vdc canceled As scrub_setup_ctx() was able to init scrub successfully with a warning as above [3] [ 740.614161] BUG: unable to handle kernel NULL pointer dereference at 00000000000000b0 [ 740.629983] Workqueue: btrfs-scrub btrfs_scrub_helper [btrfs] [ 740.631239] RIP: 0010:scrub_write_page_to_dev_replace+0x86/0x2d0 [btrfs] [ 740.650709] scrub_write_block_to_dev_replace+0x3b/0x60 [btrfs] We try to do bio to the target which is gone. I can reproduce with [4] (sorry for the very rudimentary diff) (without this patch-set.). [4] ---------------------- diff --git a/fs/btrfs/dev-replace.c b/fs/btrfs/dev-replace.c index dec01970d8c5..767cc380d08e 100644 --- a/fs/btrfs/dev-replace.c +++ b/fs/btrfs/dev-replace.c @@ -20,6 +20,9 @@ #include "rcu-string.h" #include "dev-replace.h" #include "sysfs.h" +#include <linux/delay.h> + +bool replace_cancel_done = false; static int btrfs_dev_replace_finishing(struct btrfs_fs_info *fs_info, int scrub_ret); @@ -488,6 +491,11 @@ int btrfs_dev_replace_start(struct btrfs_fs_info *fs_info, ret = btrfs_commit_transaction(trans); WARN_ON(ret); + while(replace_cancel_done == false) { + msleep(1000); + } + replace_cancel_done = false; + /* the disk copy procedure reuses the scrub code */ ret = btrfs_scrub_dev(fs_info, src_device->devid, 0, btrfs_device_get_total_bytes(src_device), @@ -829,6 +837,7 @@ int btrfs_dev_replace_cancel(struct btrfs_fs_info *fs_info) if (tgt_device) btrfs_destroy_dev_replace_tgtdev(tgt_device); + replace_cancel_done = true; leave: mutex_unlock(&dev_replace->lock_finishing_cancel_unmount); return result; ---------------------- Run: mkfs.btrfs -qf /dev/sdb && mount /dev/sdb /btrfs btrfs replace start -B 1 /dev/sdc /btrfs from another terminal btrfs replace cancel /btrfs And we see the panic [5] which is same as yours. The reason is the replace write lock does not encapsulate the scrub init, but if you do it now how to support cancel. Fix is WIP. Thanks, Anand [5] ----------------------------------------------- [22304.655493] BTRFS info (device sdb): dev_replace from /dev/sdb (devid 1) to /dev/sdc started [22343.645654] BTRFS info (device sdb): dev_replace from /dev/sdb (devid 1) to /dev/sdc canceled [22344.226987] WARNING: CPU: 0 PID: 7128 at fs/btrfs/scrub.c:620 scrub_setup_ctx.isra.19+0x220/0x230 [btrfs] [22344.226999] Modules linked in: btrfs xor zstd_decompress zstd_compress xxhash raid6_pq iptable_nat nf_nat_ipv4 nf_nat [last unloaded: xor] [22344.227026] CPU: 0 PID: 7128 Comm: btrfs Not tainted 4.19.0-rc6+ #9 [22344.227029] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [22344.227045] RIP: 0010:scrub_setup_ctx.isra.19+0x220/0x230 [btrfs] [22344.227048] Code: 00 00 49 8b 86 30 1e 00 00 5b 5d 49 89 84 24 50 03 00 00 41 c6 84 24 58 03 00 00 00 4c 89 e0 41 5c 41 5d 41 5e c3 0f 0b eb 8f <0f> 0b eb c8 66 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 41 [22344.227051] RSP: 0018:ffffadc3c0633c48 EFLAGS: 00010246 [22344.227054] RAX: ffff99e978bab800 RBX: 000000000000003f RCX: 0000000000000001 [22344.227057] RDX: 0000000000000000 RSI: 000000000015b000 RDI: 0000000000000000 [22344.227059] RBP: ffff99e978bab9f8 R08: 0000000000000066 R09: ffff99e97013dc00 [22344.227061] R10: 0000000000000001 R11: 00000000fcf2000b R12: ffff99e978bab800 [22344.227064] R13: 0000000000000001 R14: ffff99e9734d0000 R15: 0000000000000000 [22344.227067] FS: 00007f5f281a98c0(0000) GS:ffff99e97da00000(0000) knlGS:0000000000000000 [22344.227069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22344.227073] CR2: 000055ca99e26f30 CR3: 0000000078a38000 CR4: 00000000000406f0 [22344.227075] Call Trace: [22344.227092] btrfs_scrub_dev+0x19b/0x5b0 [btrfs] [22344.227110] btrfs_dev_replace_start+0x564/0x6a0 [btrfs] [22344.227126] btrfs_dev_replace_by_ioctl+0x3a/0x60 [btrfs] [22344.227140] btrfs_ioctl+0x201b/0x2ce0 [btrfs] [22344.227151] ? find_held_lock+0x2d/0x90 [22344.227161] do_vfs_ioctl+0xa9/0x6c0 [22344.227165] ? lockdep_hardirqs_on+0xed/0x1a0 [22344.227171] ksys_ioctl+0x60/0x90 [22344.227176] __x64_sys_ioctl+0x16/0x20 [22344.227179] do_syscall_64+0x50/0x180 [22344.227187] entry_SYSCALL_64_after_hwframe+0x49/0xbe [22344.227190] RIP: 0033:0x7f5f26fa7567 [22344.227194] Code: 44 00 00 48 8b 05 29 09 2d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d f9 08 2d 00 f7 d8 64 89 01 48 [22344.227196] RSP: 002b:00007fff72353c08 EFLAGS: 00000202 ORIG_RAX: 0000000000000010 [22344.227200] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007f5f26fa7567 [22344.227202] RDX: 00007fff72354a80 RSI: 00000000ca289435 RDI: 0000000000000003 [22344.227204] RBP: 0000000000000001 R08: 0000000000000000 R09: 00007fff72353b60 [22344.227206] R10: 0000000000000008 R11: 0000000000000202 R12: 0000000000000001 [22344.227208] R13: 0000000000000000 R14: 0000000000000003 R15: 00000000024f4050 [22344.227222] irq event stamp: 43742 [22344.227227] hardirqs last enabled at (43741): [<ffffffffa0a9518c>] __slab_alloc+0x52/0x61 [22344.227231] hardirqs last disabled at (43742): [<ffffffffa0801ad7>] trace_hardirqs_off_thunk+0x1a/0x1c [22344.227234] softirqs last enabled at (43244): [<ffffffffa180031c>] __do_softirq+0x31c/0x3a6 [22344.227238] softirqs last disabled at (43233): [<ffffffffa08e11d2>] irq_exit+0xb2/0xc0 [22344.227240] ---[ end trace 98bfe3316ee3a52a ]--- [22344.232735] BUG: unable to handle kernel NULL pointer dereference at 00000000000000a0 [22344.233165] PGD 0 P4D 0 [22344.233409] Oops: 0000 [#1] SMP PTI [22344.233641] CPU: 1 PID: 1091 Comm: kworker/u4:5 Tainted: G W 4.19.0-rc6+ #9 [22344.234036] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [22344.234429] Workqueue: btrfs-scrub btrfs_scrub_helper [btrfs] [22344.234757] RIP: 0010:scrub_write_page_to_dev_replace+0xb4/0x260 [btrfs] [22344.235149] Code: 43 28 48 8b 85 50 03 00 00 48 89 43 10 48 8b 43 18 48 85 c0 0f 84 20 01 00 00 48 89 58 50 48 c7 40 48 30 ee 1c c0 48 8b 53 10 <48> 8b 92 a0 00 00 00 48 8b 92 e0 00 00 00 48 39 50 08 74 18 66 81 [22344.236053] RSP: 0018:ffffadc3c0c27d20 EFLAGS: 00010246 [22344.236355] RAX: ffff99e97a65cd00 RBX: ffff99e97c197400 RCX: 0000000000000000 [22344.236707] RDX: 0000000000000000 RSI: ffff99e97a65cd00 RDI: ffff99e97a65cd00 [22344.237085] RBP: ffff99e978bab800 R08: 0000000000027970 R09: ffff99e97c197400 [22344.237424] R10: ffffffffa16001ff R11: 0000000000000000 R12: ffff99e978e12680 [22344.237780] R13: ffff99e978babac8 R14: 0000000000000000 R15: ffff99e978513838 [22344.238135] FS: 0000000000000000(0000) GS:ffff99e97db00000(0000) knlGS:0000000000000000 [22344.238629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [22344.239043] CR2: 00000000000000a0 CR3: 0000000078ca4000 CR4: 00000000000406e0 [22344.239516] Call Trace: [22344.239886] scrub_write_block_to_dev_replace+0x3f/0x60 [btrfs] [22344.240305] scrub_bio_end_io_worker+0x1af/0x490 [btrfs] [22344.240777] normal_work_helper+0xf0/0x4c0 [btrfs] [22344.241201] process_one_work+0x1f4/0x520 [22344.241792] ? process_one_work+0x16e/0x520 [22344.242106] worker_thread+0x46/0x3d0 [22344.242270] kthread+0xf8/0x130 [22344.242530] ? process_one_work+0x520/0x520 [22344.242824] ? kthread_delayed_work_timer_fn+0x80/0x80 [22344.243138] ret_from_fork+0x3a/0x50 [22344.243418] Modules linked in: btrfs xor zstd_decompress zstd_compress xxhash raid6_pq iptable_nat nf_nat_ipv4 nf_nat [last unloaded: xor] [22344.244014] CR2: 00000000000000a0 [22344.244266] ---[ end trace 98bfe3316ee3a52b ]---