From patchwork Wed Dec 20 06:20:32 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Qu Wenruo X-Patchwork-Id: 10125015 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 22C6160390 for ; Wed, 20 Dec 2017 06:20:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 302D5292EF for ; Wed, 20 Dec 2017 06:20:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 236A92969D; Wed, 20 Dec 2017 06:20:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_HI,T_TVD_MIME_EPI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 474D7292EF for ; Wed, 20 Dec 2017 06:20:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754380AbdLTGUj (ORCPT ); Wed, 20 Dec 2017 01:20:39 -0500 Received: from mout.gmx.net ([212.227.15.15]:54357 "EHLO mout.gmx.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752268AbdLTGUi (ORCPT ); Wed, 20 Dec 2017 01:20:38 -0500 Received: from [0.0.0.0] ([210.140.77.29]) by mail.gmx.com (mrgmx002 [212.227.17.184]) with ESMTPSA (Nemesis) id 0MK17F-1eSvBe29zh-001O3N; Wed, 20 Dec 2017 07:20:36 +0100 Subject: Re: Cannot balance, ENOSPC errors 4.14.2 vanilla kernel From: Qu Wenruo To: Adam Bahe Cc: linux-btrfs@vger.kernel.org References: <43ce8d2c-22cb-0b24-0735-c2c5177f0926@gmx.com> <5ad0ede6-72c5-f9a0-1ad4-95e0d17ebd7d@gmx.com> Message-ID: Date: Wed, 20 Dec 2017 14:20:32 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 In-Reply-To: <5ad0ede6-72c5-f9a0-1ad4-95e0d17ebd7d@gmx.com> X-Provags-ID: V03:K0:QbHqjEt6NATDKhiC7VBZydrxPL0pvtwiFX00e1XKJbCK/bkYBJj DCTtk7taDjlvAeaQjIRTUPJjysb1YU/EhGPxnjj3CCM2bRrA6xKPNEgtE4lGOp2xABzfvCo apEAqKbab3Qc++27RQF1Zq8Y9CbQ8ki7JGj2550yWy6wTArquyrDvLmHAJeTrWzzHAIC1Oy fEvkmzKMUF+oFDLNU4oDw== X-UI-Out-Filterresults: notjunk:1; V01:K0:K24kY6USGHE=:bamqwThIgmqwSxLCPye7s1 lXZ7Nr8XNSS95/slNXwElW5dUDbz3ODh3r6EPbT3s5wRiB1L2aHSvXqqhvMumwoRtQzIaoSef N4GnVy1JB6lgw2fEItkKJdg87GHQY7Kbdbcr8XSKL5wKPPc1fmKeyS73+3dfkBcN//8nRM1qV wDOWjTmR984z7g6TtHYF751NtRRE2t0riv2ozvJR/gJLViy3hY2P+KwK6CqYBW3Dl7a0B/kWF CusFw2qnFivMu4P5C/tp+Ejd5It+W0tARqRqlf4m6ZFWgbq/N8vKC8NI1B1HQHfa+mVkOyb6G 9ENghz9334JjbwMKM0fSPYniIU35snecZg/TXVOgaBreeb0xzakQ4gkX0uH0bbFsManvSO/0c r5LhJ7JpilQIw9KH2hPEqvGp0A13YDwm52pMzprkeZSwXp3Q91gCifSstwgXEswlnGxhfO0Ea R9fFgHjWwBrizUGLjwGEMeH5/jLIEMyXcokvAaw5BmZjMy2IgmNgg5VRXtmR/r5ngPQYKvx2X hSuR3Wd2/I4Sy+K70a4U/IV6QGbhOZqN84nfpbbdXPef7zKdhvEvQcUvx3ZpcNhbMkhyU0Jwq n/Zz6aNsd/RpAuL/+xDCoe2gIXDs68gCAn3X2RTDA4jbg55Hmb2J0FYVZh7vmpA3/NcZWV6Pp SFxrKhTt56/YNLoFg7HNiFJu15jODZz/1GfMMpwjsBW52mugZovHvQh49NqDcs1BsgQFQVprf bKxPrTvcwS5ijhn9iVVC0L28ZzLyQX9G4synx6yl+ndCOAas6YqqVjhr4uc5ODufdaypRl0eA 4DFwvfrgQtsqW+0uOhKB7HxTM03IQ== Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Also, if you're OK to compile kernel, would you please try the following } On 2017年12月20日 14:11, Qu Wenruo wrote: > > > On 2017年12月20日 13:00, Adam Bahe wrote: >> I'm using raid10. > > Pretty much the same. > > Raid10 is RAID1 first then RAID0. > > For your 20 disks (well, quite amazing) layout, btrfs will try to > allocate using all disks for RAID10. > > Any unfortunately, devid 7, 9, 12, 13, 14, 15 are already full. > > Normally btrfs should exclude them in chunk allocation, but I think > there is some small unaligned unallocated space making btrfs to choose > them for allocation. > > And causing no new chunk could be allocated. > > And considering the size of your fs, common method like adding temporary > small USB disk to allow convert doesn't work in your case. > > > You could try convert the profile from RAID10 to RAID1, which will > always use 2 disk to allocate space, so it would be OK to allocate new > chunks to start your convert. > > And final suggestion, don't use any profile with stripe with so many > uneven disks. > > Thanks, > Qu > >> >> On Tue, Dec 19, 2017 at 10:51 PM, Qu Wenruo wrote: >>> >>> >>> On 2017年12月20日 10:51, Adam Bahe wrote: >>>> Forgot to add, I should have plenty of space: >>>> >>>> Label: 'nas' uuid: 4fcd5725-b6c6-4d8a-9860-f2fc5474cbcb >>>> Total devices 20 FS bytes used 26.69TiB >>>> devid 1 size 3.64TiB used 3.28TiB path /dev/sdm >>>> devid 2 size 3.64TiB used 3.28TiB path /dev/sde >>>> devid 3 size 7.28TiB used 3.45TiB path /dev/sdt >>>> devid 4 size 9.03TiB used 2.51TiB path /dev/sdo >>>> devid 5 size 7.28TiB used 3.45TiB path /dev/sdi >>>> devid 6 size 7.28TiB used 3.45TiB path /dev/sdd >>>> devid 7 size 1.82TiB used 1.82TiB path /dev/sdp >>>> devid 9 size 1.82TiB used 1.82TiB path /dev/sdw >>>> devid 10 size 1.82TiB used 1.82TiB path /dev/sdk >>>> devid 11 size 9.03TiB used 1.82TiB path /dev/sdy >>>> devid 12 size 1.82TiB used 1.82TiB path /dev/sdg >>>> devid 13 size 1.82TiB used 1.82TiB path /dev/sdl >>>> devid 14 size 1.82TiB used 1.82TiB path /dev/sdr >>>> devid 15 size 1.82TiB used 1.82TiB path /dev/sdf >>>> devid 16 size 5.46TiB used 3.45TiB path /dev/sds >>>> devid 17 size 9.10TiB used 3.45TiB path /dev/sdn >>>> devid 18 size 9.10TiB used 3.45TiB path /dev/sdh >>>> devid 19 size 9.10TiB used 3.45TiB path /dev/sdc >>>> devid 20 size 9.10TiB used 3.45TiB path /dev/sdu >>>> devid 21 size 3.64TiB used 2.19TiB path /dev/sdj >>> >>> Depends on your profile. >>> >>> If using RAID0/5/6, the extra unallocated space means nothing if the >>> smallest disk is used up. >>> >>> Thanks, >>> Qu >>> >>>> >>>> On Tue, Dec 19, 2017 at 8:47 PM, Adam Bahe wrote: >>>>> I have been having ENOSPC errors on any btrfs device delete, btrfs >>>>> balance, btrfs device add actions for awhile now. How do I fix this? I >>>>> need to be able to remove devices and balance my filesystem again. >>>>> >>>>> [Tue Dec 19 15:25:26 2017] BTRFS info (device sdc): relocating block >>>>> group 190774812082176 flags system|raid10 >>>>> [Tue Dec 19 15:25:26 2017] BTRFS: Transaction aborted (error -27) >>>>> [Tue Dec 19 15:25:26 2017] ------------[ cut here ]------------ >>>>> [Tue Dec 19 15:25:26 2017] WARNING: CPU: 1 PID: 17036 at >>>>> fs/btrfs/extent-tree.c:10151 >>>>> btrfs_create_pending_block_groups+0x250/0x260 [btrfs] >>>>> [Tue Dec 19 15:25:26 2017] Modules linked in: dm_mod dax rpcrdma >>>>> ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi >>>>> ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm >>>>> ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core >>>>> x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass >>>>> crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel >>>>> crypto_simd glue_helper cryptd ext4 intel_cstate jbd2 intel_rapl_perf >>>>> mbcache iTCO_wdt iTCO_vendor_support ses mei_me lpc_ich pcspkr joydev >>>>> input_leds i2c_i801 mfd_core mei enclosure ioatdma sg wmi shpchp >>>>> ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter nfsd >>>>> auth_rpcgss nfs_acl lockd grace sunrpc ip_tables btrfs xor raid6_pq >>>>> mlx4_en sd_mod crc32c_intel ast i2c_algo_bit mlx4_core drm_kms_helper >>>>> ata_generic syscopyarea >>>>> [Tue Dec 19 15:25:26 2017] pata_acpi sysfillrect sysimgblt >>>>> fb_sys_fops ttm drm ata_piix ixgbe mdio ptp mpt3sas pps_core libata >>>>> raid_class dca scsi_transport_sas 8021q garp mrp >>>>> [Tue Dec 19 15:25:26 2017] CPU: 1 PID: 17036 Comm: btrfs Not tainted >>>>> 4.14.2-1.x86_64 #1 >>>>> [Tue Dec 19 15:25:26 2017] Hardware name: Supermicro Super >>>>> Server/X10DRi-T4+, BIOS 2.0 12/17/2015 >>>>> [Tue Dec 19 15:25:26 2017] task: ffff8804620416c0 task.stack: ffffc90026f64000 >>>>> [Tue Dec 19 15:25:26 2017] RIP: >>>>> 0010:btrfs_create_pending_block_groups+0x250/0x260 [btrfs] >>>>> [Tue Dec 19 15:25:26 2017] RSP: 0018:ffffc90026f679e8 EFLAGS: 00010246 >>>>> [Tue Dec 19 15:25:26 2017] RAX: 0000000000000026 RBX: 00000000ffffffe5 >>>>> RCX: 0000000000000000 >>>>> [Tue Dec 19 15:25:26 2017] RDX: 0000000000000000 RSI: ffff88046f84e108 >>>>> RDI: ffff88046f84e108 >>>>> [Tue Dec 19 15:25:26 2017] RBP: ffffc90026f67a68 R08: 0000000000000000 >>>>> R09: 0000000000003611 >>>>> [Tue Dec 19 15:25:26 2017] R10: 0000000000000004 R11: 0000000000003610 >>>>> R12: ffff8804679c38e8 >>>>> [Tue Dec 19 15:25:26 2017] R13: ffff880108578000 R14: ffff8804679c3828 >>>>> R15: ffff880108578128 >>>>> [Tue Dec 19 15:25:26 2017] FS: 00007f46ad2e48c0(0000) >>>>> GS:ffff88046f840000(0000) knlGS:0000000000000000 >>>>> [Tue Dec 19 15:25:26 2017] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>>> [Tue Dec 19 15:25:26 2017] CR2: 00007f2daae8a000 CR3: 000000048be96000 >>>>> CR4: 00000000001406e0 >>>>> [Tue Dec 19 15:25:26 2017] DR0: 0000000000000000 DR1: 0000000000000000 >>>>> DR2: 0000000000000000 >>>>> [Tue Dec 19 15:25:26 2017] DR3: 0000000000000000 DR6: 00000000fffe0ff0 >>>>> DR7: 0000000000000400 >>>>> [Tue Dec 19 15:25:26 2017] Call Trace: >>>>> [Tue Dec 19 15:25:26 2017] do_chunk_alloc+0x278/0x2f0 [btrfs] >>>>> [Tue Dec 19 15:25:26 2017] btrfs_force_chunk_alloc+0x30/0x40 [btrfs] >>>>> [Tue Dec 19 15:25:26 2017] relocate_block_group+0xd2/0x610 [btrfs] >>>>> [Tue Dec 19 15:25:26 2017] btrfs_relocate_block_group+0x187/0x240 [btrfs] >>>>> [Tue Dec 19 15:25:26 2017] btrfs_relocate_chunk+0x3b/0xc0 [btrfs] >>>>> [Tue Dec 19 15:25:26 2017] __btrfs_balance+0x854/0xc20 [btrfs] >>>>> [Tue Dec 19 15:25:26 2017] btrfs_balance+0x2f4/0x670 [btrfs] >>>>> [Tue Dec 19 15:25:26 2017] btrfs_ioctl_balance+0x439/0x560 [btrfs] >>>>> [Tue Dec 19 15:25:26 2017] btrfs_ioctl+0xf03/0x20f0 [btrfs] >>>>> [Tue Dec 19 15:25:26 2017] ? tty_write_unlock+0x31/0x40 >>>>> [Tue Dec 19 15:25:26 2017] ? tty_ldisc_deref+0x16/0x20 >>>>> [Tue Dec 19 15:25:26 2017] ? tty_write+0x1e4/0x2c0 >>>>> [Tue Dec 19 15:25:26 2017] ? process_echoes+0x70/0x70 >>>>> [Tue Dec 19 15:25:26 2017] ? __vfs_write+0x37/0x140 >>>>> [Tue Dec 19 15:25:26 2017] do_vfs_ioctl+0xa7/0x5f0 >>>>> [Tue Dec 19 15:25:26 2017] ? getnstimeofday64+0xe/0x20 >>>>> [Tue Dec 19 15:25:26 2017] ? __audit_syscall_entry+0xb3/0xf0 >>>>> [Tue Dec 19 15:25:26 2017] ? syscall_trace_enter+0x1d0/0x2b0 >>>>> [Tue Dec 19 15:25:26 2017] SyS_ioctl+0x79/0x90 >>>>> [Tue Dec 19 15:25:26 2017] do_syscall_64+0x67/0x150 >>>>> [Tue Dec 19 15:25:26 2017] entry_SYSCALL64_slow_path+0x25/0x25 >>>>> [Tue Dec 19 15:25:26 2017] RIP: 0033:0x7f46ac36b537 >>>>> [Tue Dec 19 15:25:26 2017] RSP: 002b:00007ffcceffa448 EFLAGS: 00000206 >>>>> ORIG_RAX: 0000000000000010 >>>>> [Tue Dec 19 15:25:26 2017] RAX: ffffffffffffffda RBX: 0000000000000003 >>>>> RCX: 00007f46ac36b537 >>>>> [Tue Dec 19 15:25:26 2017] RDX: 00007ffcceffa4e0 RSI: 00000000c4009420 >>>>> RDI: 0000000000000003 >>>>> [Tue Dec 19 15:25:26 2017] RBP: 00007ffcceffa4e0 R08: 00007f46ac639a00 >>>>> R09: 00007ffcceffa1c0 >>>>> [Tue Dec 19 15:25:26 2017] R10: 00007ffcceffa1d0 R11: 0000000000000206 >>>>> R12: 00007f46ac638870 >>>>> [Tue Dec 19 15:25:26 2017] R13: 00007ffcceffc893 R14: 0000000000000000 >>>>> R15: 0000000000000000 >>>>> [Tue Dec 19 15:25:26 2017] Code: 79 ff ff ff 49 8b 7c 24 60 89 da 48 >>>>> c7 c6 a8 55 57 a0 31 c0 e8 52 50 fe ff eb 9b 89 de 48 c7 c7 78 55 57 >>>>> a0 31 c0 e8 be 8a cd e0 <0f> ff eb 87 66 90 66 2e 0f 1f 84 00 00 00 00 >>>>> 00 0f 1f 44 00 00 >>>>> [Tue Dec 19 15:25:26 2017] ---[ end trace 7b033857aa29250b ]--- >>>>> [Tue Dec 19 15:25:26 2017] BTRFS: error (device sdc) in >>>>> btrfs_create_pending_block_groups:10151: errno=-27 unknown >>>>> [Tue Dec 19 15:25:26 2017] BTRFS info (device sdc): forced readonly >>>>> [Tue Dec 19 15:25:26 2017] BTRFS info (device sdc): 3388 enospc errors >>>>> during balance >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> >>> >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > diff to help us to further enhance btrfs chunk allocator? The chunk allocator itself is designed to handle your case, so it should pick up the remaining devices and allocate new RAID10 chunk with the unallocated space. But the truth is not the case. I'm wondering if it's the devs_increment and substripes calculation causing the problem. If you could help testing btrfs with the diff applied, dmesg would contain extra info when you try to do device remove. Thanks, Qu ================ diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c index 49810b70afd3..851ff13f5c29 100644 --- a/fs/btrfs/volumes.c +++ b/fs/btrfs/volumes.c @@ -4732,6 +4732,8 @@ static int __btrfs_alloc_chunk(struct btrfs_trans_handle *trans, if (ndevs < devs_increment * sub_stripes || ndevs < devs_min) { ret = -ENOSPC; + pr_info("ndevs=%d dev_increment=%d sub_stripes=%d devs_min=%d\n", + ndevs, devs_increment, sub_stripes, devs_min); goto error;