| Message ID | 20170427100534.GA10404@giis.co.in (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
Hi Qu, Seems like its a known issue with older bitmaps. http://linux-ext4.vger.kernel.narkive.com/wkNbJe0b/patch-debugfs-open-with-ext2-flag-64bits Cheers. Lakshmipathi.G On Fri, Apr 28, 2017 at 09:48:10AM +0800, Qu Wenruo wrote: > > > At 04/28/2017 09:27 AM, Qu Wenruo wrote: > >At 04/27/2017 06:05 PM, Lakshmipathi.G wrote: > >>Bug: Fail to convert 22TB EXT4 to BTRFS > >>https://bugzilla.kernel.org/show_bug.cgi?id=194795 > > > >Thanks for fixing it. > > > >While testing manually using LVM thin provision, although the convert > >itself works, the converted image has something wrong on its size. > > > >The converted image is only 4T sized, not the correct size (20T) of my > >original setup. > > > >This may not be the problem of your patch. > >I'll double check the original code to ensure it works. > > It's the method we get blocks count wrong. > > We should do it just as dumpe2fs does: > ------ > static __u64 e2p_blocks_count(struct ext2_super_block *super) > { > return super->s_blocks_count | > (ext2fs_has_feature_64bit(super) ? > (__u64) super->s_blocks_count_hi << 32 : 0); > } > ------ > > Current super->s_blocks_count can only handle U32_MAX, higher bits are in > super->s_blocks_count_hi. > > Thanks, > Qu > > > > >Thanks, > >Qu > > > >> > >>Signed-off-by: Lakshmipathi.G <Lakshmipathi.G@giis.co.in> > >>--- > >> convert/source-ext2.c | 11 ++++++----- > >> 1 file changed, 6 insertions(+), 5 deletions(-) > >> > >>diff --git a/convert/source-ext2.c b/convert/source-ext2.c > >>index 1b0576b..275cb89 100644 > >>--- a/convert/source-ext2.c > >>+++ b/convert/source-ext2.c > >>@@ -34,8 +34,9 @@ static int ext2_open_fs(struct btrfs_convert_context > >>*cctx, const char *name) > >> ext2_filsys ext2_fs; > >> ext2_ino_t ino; > >> u32 ro_feature; > >>+ int open_flag = EXT2_FLAG_SOFTSUPP_FEATURES | EXT2_FLAG_64BITS; > >>- ret = ext2fs_open(name, 0, 0, 0, unix_io_manager, &ext2_fs); > >>+ ret = ext2fs_open(name, open_flag, 0, 0, unix_io_manager, &ext2_fs); > >> if (ret) { > >> fprintf(stderr, "ext2fs_open: %s\n", error_message(ret)); > >> return -1; > >>@@ -148,7 +149,7 @@ static int ext2_read_used_space(struct > >>btrfs_convert_context *cctx) > >> return -ENOMEM; > >> for (i = 0; i < fs->group_desc_count; i++) { > >>- ret = ext2fs_get_block_bitmap_range(fs->block_map, blk_itr, > >>+ ret = ext2fs_get_block_bitmap_range2(fs->block_map, blk_itr, > >> block_nbytes * 8, block_bitmap); > >> if (ret) { > >> error("fail to get bitmap from ext2, %s", > >>@@ -353,7 +354,7 @@ static int ext2_create_symlink(struct > >>btrfs_trans_handle *trans, > >> int ret; > >> char *pathname; > >> u64 inode_size = btrfs_stack_inode_size(btrfs_inode); > >>- if (ext2fs_inode_data_blocks(ext2_fs, ext2_inode)) { > >>+ if (ext2fs_inode_data_blocks2(ext2_fs, ext2_inode)) { > >> btrfs_set_stack_inode_size(btrfs_inode, inode_size + 1); > >> ret = ext2_create_file_extents(trans, root, objectid, > >> btrfs_inode, ext2_fs, ext2_ino, > >>@@ -627,9 +628,9 @@ static int ext2_copy_extended_attrs(struct > >>btrfs_trans_handle *trans, > >> ret = -ENOMEM; > >> goto out; > >> } > >>- err = ext2fs_read_ext_attr(ext2_fs, ext2_inode->i_file_acl, buffer); > >>+ err = ext2fs_read_ext_attr2(ext2_fs, ext2_inode->i_file_acl, > >>buffer); > >> if (err) { > >>- fprintf(stderr, "ext2fs_read_ext_attr: %s\n", > >>+ fprintf(stderr, "ext2fs_read_ext_attr2: %s\n", > >> error_message(err)); > >> ret = -1; > >> goto out; > >> > > -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Apr 27, 2017 at 03:35:34PM +0530, Lakshmipathi.G wrote: > Bug: Fail to convert 22TB EXT4 to BTRFS > https://bugzilla.kernel.org/show_bug.cgi?id=194795 Thanks for working on the bug, the proposed fix looks good to me judging by the use of extended API. This would need a test though. I'm trying this on a sparse file, 3TB occupies 50G and is slow to create. Conversion works and hasn't finished yet, so this is not suitable for tests run by default. Please write a more detailed changelog, a reference to bugzilla is not really sufficient. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, May 02, 2017 at 05:06:01PM +0200, David Sterba wrote: > On Thu, Apr 27, 2017 at 03:35:34PM +0530, Lakshmipathi.G wrote: > > Bug: Fail to convert 22TB EXT4 to BTRFS > > https://bugzilla.kernel.org/show_bug.cgi?id=194795 > > Thanks for working on the bug, the proposed fix looks good to me judging > by the use of extended API. This would need a test though. I'm trying > this on a sparse file, 3TB occupies 50G and is slow to create. > Conversion works and hasn't finished yet, so this is not suitable for > tests run by default. > > Please write a more detailed changelog, a reference to bugzilla is not > really sufficient. Sure, will send a patch with better commit description about the bug and proposed fix. Cheers. Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, May 02, 2017 at 05:06:01PM +0200, David Sterba wrote: > On Thu, Apr 27, 2017 at 03:35:34PM +0530, Lakshmipathi.G wrote: > > Bug: Fail to convert 22TB EXT4 to BTRFS > > https://bugzilla.kernel.org/show_bug.cgi?id=194795 > > Thanks for working on the bug, the proposed fix looks good to me judging > by the use of extended API. This would need a test though. I'm trying > this on a sparse file, 3TB occupies 50G and is slow to create. > Conversion works and hasn't finished yet, so this is not suitable for > tests run by default. So it took a few more minutes. I'm testing it again, not on NFS, and now the creation of empty filesystem took just a few seconds and empty file is ~160MB. This sounds much better. So please write a test, using something like truncate -s3T image mkfs.ext4 image mount && write some data convert && rollback Thanks. Later we might need to add some mkfs.ext4 option coverage. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
> file is ~160MB. This sounds much better. So please write a test, using > something like > > truncate -s3T image > mkfs.ext4 image > mount && write some data > convert && rollback > > Thanks. Later we might need to add some mkfs.ext4 option coverage. Sure, will send the above test script in upcoming days, then look into mkfs.ext4 options. thanks. Cheers. Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
created a test script but it fail to detect mk2fs. Test script 009
also produces:
# ./convert-tests/009-common-inode-flags/test.sh
[TEST/conv] common inode flags test, btrfs defaults
failed: mke2fs -t ext4 -b 4096 -F
/home/laks/centos/laks/BTRFS/btrfs-progs/tests/test.img
Seems like run_check fails to detect mke2fs command. Something wrong
with my setup or issue with run_check?
On Tue, May 09, 2017 at 07:46:08PM +0530, Lakshmipathi.G wrote: > created a test script but it fail to detect mk2fs. Test script 009 > also produces: > > # ./convert-tests/009-common-inode-flags/test.sh > [TEST/conv] common inode flags test, btrfs defaults > failed: mke2fs -t ext4 -b 4096 -F > /home/laks/centos/laks/BTRFS/btrfs-progs/tests/test.img > > Seems like run_check fails to detect mke2fs command. Something wrong > with my setup or issue with run_check? Does the mke2fs command exist on your system? We maybe want to use mkfs.ext4 directly. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
> > Does the mke2fs command exist on your system? We maybe want to use > mkfs.ext4 directly. Yes, mke2fs is there. (I'll check using mkfs.ext4 directly) # ./convert-tests/009-common-inode-flags/test.sh [TEST/conv] common inode flags test, btrfs defaults failed: mke2fs -t ext4 -b 4096 -F /home/laks/centos/laks/BTRFS/btrfs-progs/tests/test.img After failure, I can run failed command manually. # mke2fs -t ext4 -b 4096 -F /home/laks/centos/laks/BTRFS/btrfs-progs/tests/test.imgmke2fs 1.42.13 (17-May-2015) Discarding device blocks: done Creating filesystem with 131072 4k blocks and 32768 inodes Filesystem UUID: 659dbb16-ea5e-48e0-8b1a-30474f54c282 Superblock backups stored on blocks: 32768, 98304 Allocating group tables: done Writing inode tables: done Creating journal (4096 blocks): done Writing superblocks and filesystem accounting information: done
diff --git a/convert/source-ext2.c b/convert/source-ext2.c index 1b0576b..275cb89 100644 --- a/convert/source-ext2.c +++ b/convert/source-ext2.c @@ -34,8 +34,9 @@ static int ext2_open_fs(struct btrfs_convert_context *cctx, const char *name) ext2_filsys ext2_fs; ext2_ino_t ino; u32 ro_feature; + int open_flag = EXT2_FLAG_SOFTSUPP_FEATURES | EXT2_FLAG_64BITS; - ret = ext2fs_open(name, 0, 0, 0, unix_io_manager, &ext2_fs); + ret = ext2fs_open(name, open_flag, 0, 0, unix_io_manager, &ext2_fs); if (ret) { fprintf(stderr, "ext2fs_open: %s\n", error_message(ret)); return -1; @@ -148,7 +149,7 @@ static int ext2_read_used_space(struct btrfs_convert_context *cctx) return -ENOMEM; for (i = 0; i < fs->group_desc_count; i++) { - ret = ext2fs_get_block_bitmap_range(fs->block_map, blk_itr, + ret = ext2fs_get_block_bitmap_range2(fs->block_map, blk_itr, block_nbytes * 8, block_bitmap); if (ret) { error("fail to get bitmap from ext2, %s", @@ -353,7 +354,7 @@ static int ext2_create_symlink(struct btrfs_trans_handle *trans, int ret; char *pathname; u64 inode_size = btrfs_stack_inode_size(btrfs_inode); - if (ext2fs_inode_data_blocks(ext2_fs, ext2_inode)) { + if (ext2fs_inode_data_blocks2(ext2_fs, ext2_inode)) { btrfs_set_stack_inode_size(btrfs_inode, inode_size + 1); ret = ext2_create_file_extents(trans, root, objectid, btrfs_inode, ext2_fs, ext2_ino, @@ -627,9 +628,9 @@ static int ext2_copy_extended_attrs(struct btrfs_trans_handle *trans, ret = -ENOMEM; goto out; } - err = ext2fs_read_ext_attr(ext2_fs, ext2_inode->i_file_acl, buffer); + err = ext2fs_read_ext_attr2(ext2_fs, ext2_inode->i_file_acl, buffer); if (err) { - fprintf(stderr, "ext2fs_read_ext_attr: %s\n", + fprintf(stderr, "ext2fs_read_ext_attr2: %s\n", error_message(err)); ret = -1; goto out;
Bug: Fail to convert 22TB EXT4 to BTRFS https://bugzilla.kernel.org/show_bug.cgi?id=194795 Signed-off-by: Lakshmipathi.G <Lakshmipathi.G@giis.co.in> --- convert/source-ext2.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)