From patchwork Tue May 7 03:22:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qu Wenruo X-Patchwork-Id: 13656182 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A02978821 for ; Tue, 7 May 2024 03:23:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715052200; cv=none; b=PlDcDnISNBMAbLhvcISTvB47+3TuiEizsCJtfeMyofKV477tJA43x4Q7in3WQjrwVNvmT0SennGbtSN1cyhT68jg+TUV6L8Z3hKn1fB5dMOlJFi4n2PfFykllXGsHneaTIU3musSSwmYzx2dgfZ87BJMqZ8uleoxMo7R6k+zm54= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715052200; c=relaxed/simple; bh=RIxTV7mBvhwf/3gcxCxgYpg9WhniAqMHrcnYBlPi6VI=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DZ6ZwO0fm1lH4TcO44qYzrV+ieCxxlvnOVd7iLzfiC1BPt/Rw2ibz2LKgfVVLGssdis8tIGe2ApmkVnCHxBahXtA9ydMvCI3rAqPvpkVeZuG5WY34/bCZ0DF13TdzBXUjU6q3wPPzH8zdX0MCKjAYrKIxolbsUWoRKMP60LY86Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com; spf=pass smtp.mailfrom=suse.com; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b=oMI8fHfs; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b=oMI8fHfs; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b="oMI8fHfs"; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b="oMI8fHfs" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 9044633846 for ; Tue, 7 May 2024 03:23:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1715052196; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=joVqXMBOrfXjDMM60D6a2mctyQCU98Fi2QuJwv3Hc+8=; b=oMI8fHfs4Fw3Zai3hbWjIB6mhro6oX+bYDa8ANUzB8UCKQ5qW74cN25un3E1OX/aGLpMqM kKCIbEu0zwgpOS8XAp8OEdbUHeBcQCc6TuL7NJYS6UP8ZlHCl66FXzTKKQFb4RasBkeYbp OOiYbruGeJqhp99a98xCs0iGs3z4nfo= Authentication-Results: smtp-out1.suse.de; dkim=pass header.d=suse.com header.s=susede1 header.b=oMI8fHfs DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1715052196; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=joVqXMBOrfXjDMM60D6a2mctyQCU98Fi2QuJwv3Hc+8=; b=oMI8fHfs4Fw3Zai3hbWjIB6mhro6oX+bYDa8ANUzB8UCKQ5qW74cN25un3E1OX/aGLpMqM kKCIbEu0zwgpOS8XAp8OEdbUHeBcQCc6TuL7NJYS6UP8ZlHCl66FXzTKKQFb4RasBkeYbp OOiYbruGeJqhp99a98xCs0iGs3z4nfo= Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id B28FE13A2D for ; Tue, 7 May 2024 03:23:15 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap1.dmz-prg2.suse.org with ESMTPSA id yJnHF6OeOWa9WgAAD6G6ig (envelope-from ) for ; Tue, 07 May 2024 03:23:15 +0000 From: Qu Wenruo To: linux-btrfs@vger.kernel.org Subject: [PATCH 1/3] Revert "btrfs-progs: convert: add raid-stripe-tree to allowed features" Date: Tue, 7 May 2024 12:52:53 +0930 Message-ID: <1557b1ea70084d913d33daa883325cce5b45172b.1715051693.git.wqu@suse.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-btrfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Level: X-Spamd-Result: default: False [-3.01 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; R_DKIM_ALLOW(-0.20)[suse.com:s=susede1]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; MIME_TRACE(0.00)[0:+]; RBL_SPAMHAUS_BLOCKED_OPENRESOLVER(0.00)[2a07:de40:b281:104:10:150:64:97:from]; DKIM_SIGNED(0.00)[suse.com:s=susede1]; RCVD_TLS_ALL(0.00)[]; DKIM_TRACE(0.00)[suse.com:+]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; SPAMHAUS_XBL(0.00)[2a07:de40:b281:104:10:150:64:97:from]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[linux-btrfs@vger.kernel.org]; DNSWL_BLOCKED(0.00)[2a07:de40:b281:104:10:150:64:97:from]; RCVD_VIA_SMTP_AUTH(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.com:dkim,suse.com:email,imap1.dmz-prg2.suse.org:helo,imap1.dmz-prg2.suse.org:rdns] X-Rspamd-Action: no action X-Rspamd-Queue-Id: 9044633846 X-Rspamd-Server: rspamd1.dmz-prg2.suse.org X-Spam-Flag: NO X-Spam-Score: -3.01 This reverts commit 346a3819237b319985ad6042d6302f67b040a803. The RST feature (at least for now) is mostly for zoned devices to support extra data profiles. Thus btrfs-convert is never designed to handle RST, because there is no way an ext4/reiserfs can even be created on a zoned device, or it would create the correct RST tree root at all Revert this unsupported feature to prevent false alerts. Signed-off-by: Qu Wenruo --- common/fsfeatures.h | 3 +- tests/common.convert | 44 ++++--------------- tests/convert-tests/001-ext2-basic/test.sh | 2 +- tests/convert-tests/003-ext4-basic/test.sh | 2 +- .../005-delete-all-rollback/test.sh | 6 +-- .../convert-tests/010-reiserfs-basic/test.sh | 2 +- .../011-reiserfs-delete-all-rollback/test.sh | 6 +-- tests/convert-tests/024-ntfs-basic/test.sh | 2 +- 8 files changed, 15 insertions(+), 52 deletions(-) diff --git a/common/fsfeatures.h b/common/fsfeatures.h index 5b241bdf9122..f636171f6bd9 100644 --- a/common/fsfeatures.h +++ b/common/fsfeatures.h @@ -68,8 +68,7 @@ static const struct btrfs_mkfs_features btrfs_convert_allowed_features = { BTRFS_FEATURE_INCOMPAT_BIG_METADATA | BTRFS_FEATURE_INCOMPAT_EXTENDED_IREF | BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA | - BTRFS_FEATURE_INCOMPAT_NO_HOLES | - BTRFS_FEATURE_INCOMPAT_RAID_STRIPE_TREE, + BTRFS_FEATURE_INCOMPAT_NO_HOLES, .runtime_flags = BTRFS_FEATURE_RUNTIME_QUOTA, }; diff --git a/tests/common.convert b/tests/common.convert index 06eec9b2a853..50d49a0d32da 100644 --- a/tests/common.convert +++ b/tests/common.convert @@ -214,47 +214,19 @@ convert_test_post_check_checksums() { grep -q 'FAILED' && _fail "file validation failed" } -# Check if convert can mount the image based on features -# $1: name of the feature -convert_can_mount() { - local features="$1" - - if [ "$features" = 'block-group-tree' ]; then - if ! [ -f "/sys/fs/btrfs/features/block_group_tree" ]; then - _log "Skip mount checks due to missing support of block-group-tree" - return 1 - fi - fi - if [ "$features" = 'raid-stripe-tree' ]; then - if ! [ -f "/sys/fs/btrfs/features/raid_stripe_tree" ]; then - _log "Skip mount checks due to missing support of raid-stripe-tree" - return 1 - fi - fi - - return 0 -} - # post conversion checks, all three in one call, on an unmounted image -# $1: features for mount compatibility checks -# $2: file with checksums -# $3: file with permissions. -# $4: file with acl entries. +# $1: file with checksums +# $2: file with permissions. +# $3: file with acl entries. convert_test_post_checks_all() { - local features="$1" - + _assert_path "$1" _assert_path "$2" _assert_path "$3" - _assert_path "$4" - - if ! convert_can_mount "$features"; then - return 0 - fi run_check_mount_test_dev - convert_test_post_check_checksums "$2" - convert_test_post_check_permissions "$3" - convert_test_post_check_acl "$4" + convert_test_post_check_checksums "$1" + convert_test_post_check_permissions "$2" + convert_test_post_check_acl "$3" # Create a large file to trigger data chunk allocation generate_dataset "large" @@ -313,7 +285,7 @@ convert_test() { run_check_umount_test_dev convert_test_do_convert "$features" "$nodesize" "$fstype" - convert_test_post_checks_all "$features" "$CHECKSUMTMP" "$EXT_PERMTMP" "$EXT_ACLTMP" + convert_test_post_checks_all "$CHECKSUMTMP" "$EXT_PERMTMP" "$EXT_ACLTMP" rm -- "$CHECKSUMTMP" rm -- "$EXT_PERMTMP" rm -- "$EXT_ACLTMP" diff --git a/tests/convert-tests/001-ext2-basic/test.sh b/tests/convert-tests/001-ext2-basic/test.sh index 461ff4a5f1a9..6dc105b55e27 100755 --- a/tests/convert-tests/001-ext2-basic/test.sh +++ b/tests/convert-tests/001-ext2-basic/test.sh @@ -12,7 +12,7 @@ check_kernel_support_acl # Iterate over defaults and options that are not tied to hardware capabilities # or number of devices -for feature in '' 'block-group-tree' 'raid-stripe-tree'; do +for feature in '' 'block-group-tree' ; do convert_test ext2 "$feature" "ext2 4k nodesize" 4096 mke2fs -b 4096 convert_test ext2 "$feature" "ext2 16k nodesize" 16384 mke2fs -b 4096 convert_test ext2 "$feature" "ext2 64k nodesize" 65536 mke2fs -b 4096 diff --git a/tests/convert-tests/003-ext4-basic/test.sh b/tests/convert-tests/003-ext4-basic/test.sh index 14637fc852db..8ae5264cb340 100755 --- a/tests/convert-tests/003-ext4-basic/test.sh +++ b/tests/convert-tests/003-ext4-basic/test.sh @@ -12,7 +12,7 @@ check_kernel_support_acl # Iterate over defaults and options that are not tied to hardware capabilities # or number of devices -for feature in '' 'block-group-tree' 'raid-stripe-tree' ; do +for feature in '' 'block-group-tree' ; do convert_test ext4 "$feature" "ext4 4k nodesize" 4096 mke2fs -t ext4 -b 4096 convert_test ext4 "$feature" "ext4 16k nodesize" 16384 mke2fs -t ext4 -b 4096 convert_test ext4 "$feature" "ext4 64k nodesize" 65536 mke2fs -t ext4 -b 4096 diff --git a/tests/convert-tests/005-delete-all-rollback/test.sh b/tests/convert-tests/005-delete-all-rollback/test.sh index 5603d3078bc8..fa56ca292bfc 100755 --- a/tests/convert-tests/005-delete-all-rollback/test.sh +++ b/tests/convert-tests/005-delete-all-rollback/test.sh @@ -38,10 +38,6 @@ do_test() { convert_test_do_convert "$features" "$nodesize" - if ! convert_can_mount "$features"; then - return 0 - fi - run_check_mount_test_dev convert_test_post_check_checksums "$CHECKSUMTMP" @@ -68,7 +64,7 @@ do_test() { # Iterate over defaults and options that are not tied to hardware capabilities # or number of devices -for feature in '' 'block-group-tree' 'raid-stripe-tree' ; do +for feature in '' 'block-group-tree' ; do do_test "$feature" "ext4 4k nodesize" 4096 mke2fs -t ext4 -b 4096 do_test "$feature" "ext4 16k nodesize" 16384 mke2fs -t ext4 -b 4096 do_test "$feature" "ext4 64k nodesize" 65536 mke2fs -t ext4 -b 4096 diff --git a/tests/convert-tests/010-reiserfs-basic/test.sh b/tests/convert-tests/010-reiserfs-basic/test.sh index 6ab02b31d176..86b1826783b3 100755 --- a/tests/convert-tests/010-reiserfs-basic/test.sh +++ b/tests/convert-tests/010-reiserfs-basic/test.sh @@ -15,7 +15,7 @@ prepare_test_dev # Iterate over defaults and options that are not tied to hardware capabilities # or number of devices -for feature in '' 'block-group-tree' 'raid-stripe-tree'; do +for feature in '' 'block-group-tree' ; do convert_test reiserfs "$feature" "reiserfs 4k nodesize" 4096 mkreiserfs -b 4096 convert_test reiserfs "$feature" "reiserfs 16k nodesize" 16384 mkreiserfs -b 4096 convert_test reiserfs "$feature" "reiserfs 64k nodesize" 65536 mkreiserfs -b 4096 diff --git a/tests/convert-tests/011-reiserfs-delete-all-rollback/test.sh b/tests/convert-tests/011-reiserfs-delete-all-rollback/test.sh index 688613c2c6ad..e1c3e02eb7a0 100755 --- a/tests/convert-tests/011-reiserfs-delete-all-rollback/test.sh +++ b/tests/convert-tests/011-reiserfs-delete-all-rollback/test.sh @@ -40,10 +40,6 @@ do_test() { convert_test_do_convert "$features" "$nodesize" - if ! convert_can_mount "$features"; then - return 0 - fi - run_check_mount_test_dev convert_test_post_check_checksums "$CHECKSUMTMP" @@ -70,7 +66,7 @@ do_test() { # Iterate over defaults and options that are not tied to hardware capabilities # or number of devices -for feature in '' 'block-group-tree' 'raid-stripe-tree'; do +for feature in '' 'block-group-tree' ; do do_test "$feature" "reiserfs 4k nodesize" 4096 mkreiserfs -b 4096 do_test "$feature" "reiserfs 16k nodesize" 16384 mkreiserfs -b 4096 do_test "$feature" "reiserfs 64k nodesize" 65536 mkreiserfs -b 4096 diff --git a/tests/convert-tests/024-ntfs-basic/test.sh b/tests/convert-tests/024-ntfs-basic/test.sh index a93f60070674..d2e1be79ef6b 100755 --- a/tests/convert-tests/024-ntfs-basic/test.sh +++ b/tests/convert-tests/024-ntfs-basic/test.sh @@ -17,6 +17,6 @@ prepare_test_dev # Iterate over defaults and options that are not tied to hardware capabilities # or number of devices. Test only 4K block size as minimum. -for feature in '' 'block-group-tree' 'raid-stripe-tree'; do +for feature in '' 'block-group-tree' ; do convert_test ntfs "$feature" "ntfs 4k nodesize" 4096 mkfs.ntfs -s 4096 done From patchwork Tue May 7 03:22:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qu Wenruo X-Patchwork-Id: 13656183 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4382C8F59 for ; Tue, 7 May 2024 03:23:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715052202; cv=none; b=o3C/J6EKNos1ke/djOwLGq1MRCScx37NcL0US0qJz+iYpM5cBWFEbXwTrUSD2YmE/lMb3FbkhqznOpYn06i1wGm7xnJLfrkw5/4WuOQkjdrqDnKnRBTpo87h320SfwURJPiHV2mXG/sFB6+tBwRQZ7gXzaWCHvbmM1ayhnUhQBQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715052202; c=relaxed/simple; bh=yyESevwEqkls5selzPLL/ElyiZlf3cGevu/9/m4wbuU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=T7ZGPbiIXRIE9fz1PawboR2begLTlfkY69FYKh42ieHjNYpa7KG6iv0mb946C2l86efiTsX3SHQqDDomOjJ/4pBaiPnNXxAauqVdp9Gfq88SBuMH6CND3oHw3TTjzdchADqVlUz6CgWxF4JafcA6iAyOldgy4pkEDS8uK2v0fO8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com; spf=pass smtp.mailfrom=suse.com; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b=N7ofmuZA; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b=N7ofmuZA; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b="N7ofmuZA"; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b="N7ofmuZA" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 65FCD33845; Tue, 7 May 2024 03:23:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1715052198; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7udLJszg5+2qHaaozIKSf2aS76+CQ58ZvIdclp+fgdw=; b=N7ofmuZAPxuTt97PT+amR4vywKvbQVOMD+8JUCFqeqOhCnYJadz1SQreZoCfHNATEdXqMk Ulr8NnINDSUID2V3mI+onL3kZzMRKJJa4KV0lYz7fIKIVwndHDUvdKd3FDs0/7GXB2UZ0z UX0whcKaWUT3e0xsvT9VcpUiIRcZsqg= Authentication-Results: smtp-out1.suse.de; dkim=pass header.d=suse.com header.s=susede1 header.b=N7ofmuZA DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1715052198; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7udLJszg5+2qHaaozIKSf2aS76+CQ58ZvIdclp+fgdw=; b=N7ofmuZAPxuTt97PT+amR4vywKvbQVOMD+8JUCFqeqOhCnYJadz1SQreZoCfHNATEdXqMk Ulr8NnINDSUID2V3mI+onL3kZzMRKJJa4KV0lYz7fIKIVwndHDUvdKd3FDs0/7GXB2UZ0z UX0whcKaWUT3e0xsvT9VcpUiIRcZsqg= Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 1EE0D139CB; Tue, 7 May 2024 03:23:16 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap1.dmz-prg2.suse.org with ESMTPSA id +BqhMqSeOWa9WgAAD6G6ig (envelope-from ); Tue, 07 May 2024 03:23:16 +0000 From: Qu Wenruo To: linux-btrfs@vger.kernel.org Cc: Yordan Subject: [PATCH 2/3] btrfs-progs: convert: rework file extent iteration to handle unwritten extents Date: Tue, 7 May 2024 12:52:54 +0930 Message-ID: X-Mailer: git-send-email 2.45.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-btrfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Flag: NO X-Spam-Score: -3.01 X-Rspamd-Action: no action X-Rspamd-Queue-Id: 65FCD33845 X-Spam-Level: X-Rspamd-Server: rspamd2.dmz-prg2.suse.org X-Spamd-Result: default: False [-3.01 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-1.000]; R_DKIM_ALLOW(-0.20)[suse.com:s=susede1]; MIME_GOOD(-0.10)[text/plain]; MX_GOOD(-0.01)[]; ARC_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; DKIM_SIGNED(0.00)[suse.com:s=susede1]; MIME_TRACE(0.00)[0:+]; FUZZY_BLOCKED(0.00)[rspamd.com]; TO_DN_SOME(0.00)[]; RBL_SPAMHAUS_BLOCKED_OPENRESOLVER(0.00)[2a07:de40:b281:104:10:150:64:97:from]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DNSWL_BLOCKED(0.00)[2a07:de40:b281:104:10:150:64:97:from]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.com:dkim,suse.com:email,imap1.dmz-prg2.suse.org:helo,imap1.dmz-prg2.suse.org:rdns]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[suse.com:+]; FREEMAIL_CC(0.00)[gmail.com]; RCVD_TLS_ALL(0.00)[]; SPAMHAUS_XBL(0.00)[2a07:de40:b281:104:10:150:64:97:from]; FREEMAIL_ENVRCPT(0.00)[gmail.com] [BUG] There is a bug report that btrfs-convert can not handle unwritten extents (EXT2_EXTENT_FLAGS_UNINIT set, which is pretty much the same as BTRFS_FILE_EXTENT_PREALLOC), which can cause the converted image to have incorrect contents. [CAUSE] Currently we use ext2fs_block_iterate2() to go through all data extents of an ext2 inode, but it doesn't provide the info on if the range is unwritten or not. Thus for unwritten extents, the results btrfs would just treat it as regular extents, and read the contents from disk other than setting the contents to zero. [FIX] Instead of the ext2fs_block_iterate2(), here we follow the debugfs' "dump_extents" command, to use ext2fs_extent_*() helpers to go through every data extent of the inode, that's if the inode support the EXT4_EXTENTS_FL flag. Now we can properly get the info of which extents are unwritten, and use holes to replace those unwritten extents. Reported-by: Yordan Link: https://lore.kernel.org/all/d34c7d77a7f00c93bea6a4d6e83c7caf.mailbg@mail.bg/ Signed-off-by: Qu Wenruo --- convert/source-ext2.c | 116 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 109 insertions(+), 7 deletions(-) diff --git a/convert/source-ext2.c b/convert/source-ext2.c index bba81e4034fd..029fa198fc24 100644 --- a/convert/source-ext2.c +++ b/convert/source-ext2.c @@ -303,6 +303,88 @@ static int ext2_block_iterate_proc(ext2_filsys fs, blk_t *blocknr, return 0; } +static int iterate_one_file_extent(struct blk_iterate_data *data, + u64 filepos, u64 len, u64 disk_bytenr, + bool prealloced) +{ + const int sectorsize = data->trans->fs_info->sectorsize; + const int sectorbits = ilog2(sectorsize); + int ret; + + UASSERT(len > 0); + for (int i = 0; i < len; i += sectorsize) { + /* + * Just treat preallocated extent as hole. + * + * As there is no way to utilize the preallocated space, since + * any file extent would also be shared by ext2 image. + */ + if (prealloced) + ret = block_iterate_proc(0, + (filepos + i) >> sectorbits, data); + else + ret = block_iterate_proc( + (disk_bytenr + i) >> sectorbits, + (filepos + i) >> sectorbits, data); + + if (ret < 0) + return ret; + } + return 0; +} + +static int iterate_file_extents(struct blk_iterate_data *data, + ext2_filsys ext2fs, ext2_ino_t ext2_ino, + u32 convert_flags) +{ + ext2_extent_handle_t handle = NULL; + struct ext2fs_extent extent; + const int sectorsize = data->trans->fs_info->sectorsize; + const int sectorbits = ilog2(sectorsize); + int op = EXT2_EXTENT_ROOT; + errcode_t errcode; + int ret = 0; + + errcode = ext2fs_extent_open(ext2fs, ext2_ino, &handle); + if (errcode) { + error("failed to open ext2 inode %u: %s", ext2_ino, + error_message(errcode)); + return -EIO; + } + while (1) { + u64 disk_bytenr; + u64 filepos; + u64 len; + + errcode = ext2fs_extent_get(handle, op, &extent); + if (errcode == EXT2_ET_EXTENT_NO_NEXT) + break; + if (errcode) { + data->errcode = errcode; + ret = -EIO; + goto out; + } + op = EXT2_EXTENT_NEXT; + + if (extent.e_flags & EXT2_EXTENT_FLAGS_SECOND_VISIT) + continue; + if (!(extent.e_flags & EXT2_EXTENT_FLAGS_LEAF)) + continue; + + filepos = extent.e_lblk << sectorbits; + len = extent.e_len << sectorbits; + disk_bytenr = extent.e_pblk << sectorbits; + + ret = iterate_one_file_extent(data, filepos, len, disk_bytenr, + extent.e_flags & EXT2_EXTENT_FLAGS_UNINIT); + if (ret < 0) + goto out; + } +out: + ext2fs_extent_free(handle); + return ret; +} + /* * traverse file's data blocks, record these data blocks as file extents. */ @@ -315,6 +397,7 @@ static int ext2_create_file_extents(struct btrfs_trans_handle *trans, int ret; char *buffer = NULL; errcode_t err; + struct ext2_inode ext2_inode = { 0 }; u32 last_block; u32 sectorsize = root->fs_info->sectorsize; u64 inode_size = btrfs_stack_inode_size(btrfs_inode); @@ -323,10 +406,32 @@ static int ext2_create_file_extents(struct btrfs_trans_handle *trans, init_blk_iterate_data(&data, trans, root, btrfs_inode, objectid, convert_flags & CONVERT_FLAG_DATACSUM); - err = ext2fs_block_iterate2(ext2_fs, ext2_ino, BLOCK_FLAG_DATA_ONLY, - NULL, ext2_block_iterate_proc, &data); - if (err) - goto error; + err = ext2fs_read_inode(ext2_fs, ext2_ino, &ext2_inode); + if (err) { + error("failed to read ext2 inode %u: %s", + ext2_ino, error_message(err)); + return -EIO; + } + /* + * For inodes without extent block maps, go with the older + * ext2fs_block_iterate2(). + * Otherwise use ext2fs_extent_*() based solution, as that can provide + * UNINIT extent flags. + */ + if ((ext2_inode.i_flags & EXT4_EXTENTS_FL) == 0) { + err = ext2fs_block_iterate2(ext2_fs, ext2_ino, + BLOCK_FLAG_DATA_ONLY, NULL, + ext2_block_iterate_proc, &data); + if (err) { + error("ext2fs_block_iterate2: %s", error_message(err)); + return -EIO; + } + } else { + ret = iterate_file_extents(&data, ext2_fs, ext2_ino, + convert_flags); + if (ret < 0) + goto fail; + } ret = data.errcode; if (ret) goto fail; @@ -366,9 +471,6 @@ static int ext2_create_file_extents(struct btrfs_trans_handle *trans, fail: free(buffer); return ret; -error: - error("ext2fs_block_iterate2: %s", error_message(err)); - return -1; } static int ext2_create_symlink(struct btrfs_trans_handle *trans, From patchwork Tue May 7 03:22:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qu Wenruo X-Patchwork-Id: 13656184 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AF7644EB4A for ; Tue, 7 May 2024 03:23:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715052203; cv=none; b=VuUpYBEARxVQExYNbLNyGmZCS1QuDbC7BjucCfKfF/Q6XfgTtkGeELJhkFq1dZ9Fi8TgSZJumF7W5XeOMjXqS7IOltgBSF/4xnO8DSogKytErnbjtkupMTzGvjrWbPgb13TJ9100nQO+sHshv27mzly3O4xo6mx8CtJPSAeR6NE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715052203; c=relaxed/simple; bh=PCoFlLJ27cjSK6eAFeTEROQ3vIQbz6bnZTz2arGJ8xo=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QIn0KFfze6EHrZR+h5eBbNECFTaLAJNroVQlZ3UROc4p5/46O0pHRIHQdlgQbPWYMV4EAehDy+sFViJ3PcLM3r5sxZjwBxKFXt9xrb0VtsfHtCiTTvCiwjgXbxENbjOcKifQ/gIMMx5nXcsINHOylSQrZTy4HvQAXBZ2D+a/u6U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com; spf=pass smtp.mailfrom=suse.com; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b=AwYhFaMN; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b=AwYhFaMN; arc=none smtp.client-ip=195.135.223.131 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b="AwYhFaMN"; dkim=pass (1024-bit key) header.d=suse.com header.i=@suse.com header.b="AwYhFaMN" Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id D8E612029F for ; Tue, 7 May 2024 03:23:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1715052199; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iTIG8f0cs7oIW2gVWp1CqxZ+4e2MIS/6XFmetDbygCM=; b=AwYhFaMNSNL1DsojYdxE54c3t8TzHCc5wX+I5T143s7dcRtfv4IaE/QoXjyveiKcJ7VUyn 7C0dYSX8AMzOIJqUuZpLqgnlUrVlPRL6VFzZWIiGaZWSIpJH0rFL/tZ7RG36347evXyLRk RK01FyfJ2cyvWk4G4+ksBDDMZSUDJd0= Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1715052199; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iTIG8f0cs7oIW2gVWp1CqxZ+4e2MIS/6XFmetDbygCM=; b=AwYhFaMNSNL1DsojYdxE54c3t8TzHCc5wX+I5T143s7dcRtfv4IaE/QoXjyveiKcJ7VUyn 7C0dYSX8AMzOIJqUuZpLqgnlUrVlPRL6VFzZWIiGaZWSIpJH0rFL/tZ7RG36347evXyLRk RK01FyfJ2cyvWk4G4+ksBDDMZSUDJd0= Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 02143139CB for ; Tue, 7 May 2024 03:23:18 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap1.dmz-prg2.suse.org with ESMTPSA id aCHFKaaeOWa9WgAAD6G6ig (envelope-from ) for ; Tue, 07 May 2024 03:23:18 +0000 From: Qu Wenruo To: linux-btrfs@vger.kernel.org Subject: [PATCH 3/3] btrfs-progs: tests/convert: add test case for ext4 unwritten extents Date: Tue, 7 May 2024 12:52:55 +0930 Message-ID: <542d0b559a21ae84202d2e7632dfab59019e4982.1715051693.git.wqu@suse.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-btrfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Level: X-Spamd-Result: default: False [-2.80 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MID_CONTAINS_FROM(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_MISSING_CHARSET(0.50)[]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; FUZZY_BLOCKED(0.00)[rspamd.com]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; DKIM_SIGNED(0.00)[suse.com:s=susede1]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:helo,imap1.dmz-prg2.suse.org:rdns,suse.com:email]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[linux-btrfs@vger.kernel.org]; RCVD_TLS_ALL(0.00)[] X-Spam-Score: -2.80 X-Spam-Flag: NO The test case would verify the behaivor of ext4 unwritten extents by: - Create a unwritten (preallocated) extent on ext4 - Fill the on-disk extent with random garbage This is to make sure if btrfs tries to read the on-disk data, it would definitely get some garbage. As I found sometimes mkfs.ext4 can fill the unused bg with zeros. - Fill the preallocated file range with some data This is to make sure btrfs-convert can handle mixed written and unwritten ranges. - Save the checksum of the file. - Convert the fs - verify the checksum For older btrfs-convert, there would be only one regular file extent, and reading the file would read out some garbage and cause checksum to mismatch. For the fixed btrfs-convert, we punch holes for unwritten extents, thus only the written part would be read out and match the checksum. Signed-off-by: Qu Wenruo --- .../025-ext4-uninit-written/test.sh | 53 +++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100755 tests/convert-tests/025-ext4-uninit-written/test.sh diff --git a/tests/convert-tests/025-ext4-uninit-written/test.sh b/tests/convert-tests/025-ext4-uninit-written/test.sh new file mode 100755 index 000000000000..cca8f69a201e --- /dev/null +++ b/tests/convert-tests/025-ext4-uninit-written/test.sh @@ -0,0 +1,53 @@ +#!/bin/bash +# Make sure btrfs is handling the ext4 uninit (preallocated) extent correctly + +source "$TEST_TOP/common" || exit +source "$TEST_TOP/common.convert" || exit + +setup_root_helper +prepare_test_dev 1G + +check_global_prereq mkfs.ext4 +check_global_prereq fallocate +check_global_prereq filefrag +check_global_prereq awk +check_global_prereq md5sum +check_prereq btrfs-convert +check_prereq btrfs + +convert_test_prep_fs ext4 mke2fs -t ext4 -b 4096 + +# Create a preallocated extent first. +run_check $SUDO_HELPER fallocate -l 32K "$TEST_MNT/file" +sync +# Get the real on-disk location and write some data into it. +physical=$(run_check_stdout $SUDO_HELPER filefrag -v "$TEST_MNT/file" | grep unwritten | awk '{print $4}' | grep -o "[[:digit:]]*") + +if [ -z "$physical" ]; then + _fail "unable to get the physical address of the file" +fi + +# Now fill the underlying range with non-zeros. +# For properly converted fs, we should not read the contents anyway +run_check $SUDO_HELPER dd if=/dev/urandom of=$TEST_DEV bs=4096 seek="$physical" conv=notrunc count=8 + +# Write some thing into the file range. +run_check $SUDO_HELPER dd if=/dev/zero of="$TEST_MNT/file" bs=4096 count=1 conv=notrunc +run_check $SUDO_HELPER dd if=/dev/zero of="$TEST_MNT/file" bs=4096 count=1 seek=3 conv=notrunc +sync +md5_before=$(md5sum "$TEST_MNT/file" | cut -f1 -d' ') +_log "md5sum before convert: $md5_before" +run_check_umount_test_dev + +# Btrfs-convert should handle the unwritten part correctly, either punching a hole +# or a proper preallocated extent, so that we won't read the on-disk data. +convert_test_do_convert + +run_check_mount_test_dev +md5_after=$(md5sum "$TEST_MNT/file" | cut -f1 -d' ') +_log "md5sum after convert: $md5_after" +run_check_umount_test_dev + +if [ "$md5_before" != "$md5_after" ]; then + _fail "contents mismatch" +fi