btrfs-progs: btrfs-convert: Add larger device support
diff mbox

Message ID 20170427100534.GA10404@giis.co.in
State New
Headers show

Commit Message

Lakshmipathi.G April 27, 2017, 10:05 a.m. UTC
Bug: Fail to convert 22TB EXT4 to BTRFS
https://bugzilla.kernel.org/show_bug.cgi?id=194795

Signed-off-by: Lakshmipathi.G <Lakshmipathi.G@giis.co.in>
---
 convert/source-ext2.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

Comments

Lakshmipathi.G April 28, 2017, 5:55 a.m. UTC | #1
Hi Qu,

Seems like its a known issue with older bitmaps.
http://linux-ext4.vger.kernel.narkive.com/wkNbJe0b/patch-debugfs-open-with-ext2-flag-64bits

Cheers.
Lakshmipathi.G

On Fri, Apr 28, 2017 at 09:48:10AM +0800, Qu Wenruo wrote:
> 
> 
> At 04/28/2017 09:27 AM, Qu Wenruo wrote:
> >At 04/27/2017 06:05 PM, Lakshmipathi.G wrote:
> >>Bug: Fail to convert 22TB EXT4 to BTRFS
> >>https://bugzilla.kernel.org/show_bug.cgi?id=194795
> >
> >Thanks for fixing it.
> >
> >While testing manually using LVM thin provision, although the convert
> >itself works, the converted image has something wrong on its size.
> >
> >The converted image is only 4T sized, not the correct size (20T) of my
> >original setup.
> >
> >This may not be the problem of your patch.
> >I'll double check the original code to ensure it works.
> 
> It's the method we get blocks count wrong.
> 
> We should do it just as dumpe2fs does:
> ------
> static __u64 e2p_blocks_count(struct ext2_super_block *super)
> {
> 	return super->s_blocks_count |
> 		(ext2fs_has_feature_64bit(super) ?
> 		(__u64) super->s_blocks_count_hi << 32 : 0);
> }
> ------
> 
> Current super->s_blocks_count can only handle U32_MAX, higher bits are in
> super->s_blocks_count_hi.
> 
> Thanks,
> Qu
> 
> >
> >Thanks,
> >Qu
> >
> >>
> >>Signed-off-by: Lakshmipathi.G <Lakshmipathi.G@giis.co.in>
> >>---
> >>  convert/source-ext2.c | 11 ++++++-----
> >>  1 file changed, 6 insertions(+), 5 deletions(-)
> >>
> >>diff --git a/convert/source-ext2.c b/convert/source-ext2.c
> >>index 1b0576b..275cb89 100644
> >>--- a/convert/source-ext2.c
> >>+++ b/convert/source-ext2.c
> >>@@ -34,8 +34,9 @@ static int ext2_open_fs(struct btrfs_convert_context
> >>*cctx, const char *name)
> >>      ext2_filsys ext2_fs;
> >>      ext2_ino_t ino;
> >>      u32 ro_feature;
> >>+    int open_flag = EXT2_FLAG_SOFTSUPP_FEATURES | EXT2_FLAG_64BITS;
> >>-    ret = ext2fs_open(name, 0, 0, 0, unix_io_manager, &ext2_fs);
> >>+    ret = ext2fs_open(name, open_flag, 0, 0, unix_io_manager, &ext2_fs);
> >>      if (ret) {
> >>          fprintf(stderr, "ext2fs_open: %s\n", error_message(ret));
> >>          return -1;
> >>@@ -148,7 +149,7 @@ static int ext2_read_used_space(struct
> >>btrfs_convert_context *cctx)
> >>          return -ENOMEM;
> >>      for (i = 0; i < fs->group_desc_count; i++) {
> >>-        ret = ext2fs_get_block_bitmap_range(fs->block_map, blk_itr,
> >>+        ret = ext2fs_get_block_bitmap_range2(fs->block_map, blk_itr,
> >>                          block_nbytes * 8, block_bitmap);
> >>          if (ret) {
> >>              error("fail to get bitmap from ext2, %s",
> >>@@ -353,7 +354,7 @@ static int ext2_create_symlink(struct
> >>btrfs_trans_handle *trans,
> >>      int ret;
> >>      char *pathname;
> >>      u64 inode_size = btrfs_stack_inode_size(btrfs_inode);
> >>-    if (ext2fs_inode_data_blocks(ext2_fs, ext2_inode)) {
> >>+    if (ext2fs_inode_data_blocks2(ext2_fs, ext2_inode)) {
> >>          btrfs_set_stack_inode_size(btrfs_inode, inode_size + 1);
> >>          ret = ext2_create_file_extents(trans, root, objectid,
> >>                  btrfs_inode, ext2_fs, ext2_ino,
> >>@@ -627,9 +628,9 @@ static int ext2_copy_extended_attrs(struct
> >>btrfs_trans_handle *trans,
> >>          ret = -ENOMEM;
> >>          goto out;
> >>      }
> >>-    err = ext2fs_read_ext_attr(ext2_fs, ext2_inode->i_file_acl, buffer);
> >>+    err = ext2fs_read_ext_attr2(ext2_fs, ext2_inode->i_file_acl,
> >>buffer);
> >>      if (err) {
> >>-        fprintf(stderr, "ext2fs_read_ext_attr: %s\n",
> >>+        fprintf(stderr, "ext2fs_read_ext_attr2: %s\n",
> >>              error_message(err));
> >>          ret = -1;
> >>          goto out;
> >>
> 
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
David Sterba May 2, 2017, 3:06 p.m. UTC | #2
On Thu, Apr 27, 2017 at 03:35:34PM +0530, Lakshmipathi.G wrote:
> Bug: Fail to convert 22TB EXT4 to BTRFS
> https://bugzilla.kernel.org/show_bug.cgi?id=194795

Thanks for working on the bug, the proposed fix looks good to me judging
by the use of extended API. This would need a test though. I'm trying
this on a sparse file, 3TB occupies 50G and is slow to create.
Conversion works and hasn't finished yet, so this is not suitable for
tests run by default.

Please write a more detailed changelog, a reference to bugzilla is not
really sufficient.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Lakshmipathi.G May 2, 2017, 3:28 p.m. UTC | #3
On Tue, May 02, 2017 at 05:06:01PM +0200, David Sterba wrote:
> On Thu, Apr 27, 2017 at 03:35:34PM +0530, Lakshmipathi.G wrote:
> > Bug: Fail to convert 22TB EXT4 to BTRFS
> > https://bugzilla.kernel.org/show_bug.cgi?id=194795
> 
> Thanks for working on the bug, the proposed fix looks good to me judging
> by the use of extended API. This would need a test though. I'm trying
> this on a sparse file, 3TB occupies 50G and is slow to create.
> Conversion works and hasn't finished yet, so this is not suitable for
> tests run by default.
> 
> Please write a more detailed changelog, a reference to bugzilla is not
> really sufficient.

Sure, will send a patch with better commit description about the bug and
proposed fix. 

Cheers.
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
David Sterba May 2, 2017, 3:35 p.m. UTC | #4
On Tue, May 02, 2017 at 05:06:01PM +0200, David Sterba wrote:
> On Thu, Apr 27, 2017 at 03:35:34PM +0530, Lakshmipathi.G wrote:
> > Bug: Fail to convert 22TB EXT4 to BTRFS
> > https://bugzilla.kernel.org/show_bug.cgi?id=194795
> 
> Thanks for working on the bug, the proposed fix looks good to me judging
> by the use of extended API. This would need a test though. I'm trying
> this on a sparse file, 3TB occupies 50G and is slow to create.
> Conversion works and hasn't finished yet, so this is not suitable for
> tests run by default.

So it took a few more minutes. I'm testing it again, not on NFS,  and
now the creation of empty filesystem took just a few seconds and empty
file is ~160MB. This sounds much better. So please write a test, using
something like

truncate -s3T image
mkfs.ext4 image
mount && write some data
convert && rollback

Thanks.  Later we might need to add some mkfs.ext4 option coverage.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Lakshmipathi.G May 3, 2017, 4:38 p.m. UTC | #5
> file is ~160MB. This sounds much better. So please write a test, using
> something like
> 
> truncate -s3T image
> mkfs.ext4 image
> mount && write some data
> convert && rollback
> 
> Thanks.  Later we might need to add some mkfs.ext4 option coverage.
Sure, will send the above test script in upcoming days, then look
into mkfs.ext4 options. thanks.

Cheers.
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Lakshmipathi.G May 9, 2017, 2:16 p.m. UTC | #6
created a test script but it fail to detect mk2fs. Test script 009
also produces:

# ./convert-tests/009-common-inode-flags/test.sh
    [TEST/conv]     common inode flags test, btrfs defaults
failed: mke2fs -t ext4 -b 4096 -F
/home/laks/centos/laks/BTRFS/btrfs-progs/tests/test.img

Seems like run_check fails to detect mke2fs command. Something wrong
with my setup or issue with run_check?
David Sterba May 9, 2017, 3:41 p.m. UTC | #7
On Tue, May 09, 2017 at 07:46:08PM +0530, Lakshmipathi.G wrote:
> created a test script but it fail to detect mk2fs. Test script 009
> also produces:
> 
> # ./convert-tests/009-common-inode-flags/test.sh
>     [TEST/conv]     common inode flags test, btrfs defaults
> failed: mke2fs -t ext4 -b 4096 -F
> /home/laks/centos/laks/BTRFS/btrfs-progs/tests/test.img
> 
> Seems like run_check fails to detect mke2fs command. Something wrong
> with my setup or issue with run_check?

Does the mke2fs command exist on your system? We maybe want to use
mkfs.ext4 directly.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Lakshmipathi.G May 9, 2017, 5:15 p.m. UTC | #8
>
> Does the mke2fs command exist on your system? We maybe want to use
> mkfs.ext4 directly.

Yes, mke2fs is there. (I'll check using mkfs.ext4 directly)

# ./convert-tests/009-common-inode-flags/test.sh
    [TEST/conv]     common inode flags test, btrfs defaults
failed: mke2fs -t ext4 -b 4096 -F
/home/laks/centos/laks/BTRFS/btrfs-progs/tests/test.img

After failure, I can run failed command manually.
# mke2fs -t ext4 -b 4096 -F
/home/laks/centos/laks/BTRFS/btrfs-progs/tests/test.imgmke2fs 1.42.13
(17-May-2015)
Discarding device blocks: done
Creating filesystem with 131072 4k blocks and 32768 inodes
Filesystem UUID: 659dbb16-ea5e-48e0-8b1a-30474f54c282
Superblock backups stored on blocks:
	32768, 98304

Allocating group tables: done
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

Patch
diff mbox

diff --git a/convert/source-ext2.c b/convert/source-ext2.c
index 1b0576b..275cb89 100644
--- a/convert/source-ext2.c
+++ b/convert/source-ext2.c
@@ -34,8 +34,9 @@  static int ext2_open_fs(struct btrfs_convert_context *cctx, const char *name)
 	ext2_filsys ext2_fs;
 	ext2_ino_t ino;
 	u32 ro_feature;
+	int open_flag = EXT2_FLAG_SOFTSUPP_FEATURES | EXT2_FLAG_64BITS;
 
-	ret = ext2fs_open(name, 0, 0, 0, unix_io_manager, &ext2_fs);
+	ret = ext2fs_open(name, open_flag, 0, 0, unix_io_manager, &ext2_fs);
 	if (ret) {
 		fprintf(stderr, "ext2fs_open: %s\n", error_message(ret));
 		return -1;
@@ -148,7 +149,7 @@  static int ext2_read_used_space(struct btrfs_convert_context *cctx)
 		return -ENOMEM;
 
 	for (i = 0; i < fs->group_desc_count; i++) {
-		ret = ext2fs_get_block_bitmap_range(fs->block_map, blk_itr,
+		ret = ext2fs_get_block_bitmap_range2(fs->block_map, blk_itr,
 						block_nbytes * 8, block_bitmap);
 		if (ret) {
 			error("fail to get bitmap from ext2, %s",
@@ -353,7 +354,7 @@  static int ext2_create_symlink(struct btrfs_trans_handle *trans,
 	int ret;
 	char *pathname;
 	u64 inode_size = btrfs_stack_inode_size(btrfs_inode);
-	if (ext2fs_inode_data_blocks(ext2_fs, ext2_inode)) {
+	if (ext2fs_inode_data_blocks2(ext2_fs, ext2_inode)) {
 		btrfs_set_stack_inode_size(btrfs_inode, inode_size + 1);
 		ret = ext2_create_file_extents(trans, root, objectid,
 				btrfs_inode, ext2_fs, ext2_ino,
@@ -627,9 +628,9 @@  static int ext2_copy_extended_attrs(struct btrfs_trans_handle *trans,
 		ret = -ENOMEM;
 		goto out;
 	}
-	err = ext2fs_read_ext_attr(ext2_fs, ext2_inode->i_file_acl, buffer);
+	err = ext2fs_read_ext_attr2(ext2_fs, ext2_inode->i_file_acl, buffer);
 	if (err) {
-		fprintf(stderr, "ext2fs_read_ext_attr: %s\n",
+		fprintf(stderr, "ext2fs_read_ext_attr2: %s\n",
 			error_message(err));
 		ret = -1;
 		goto out;